From nobody Tue Apr 7 17:16:24 2026 Received: from mail-oa1-f50.google.com (mail-oa1-f50.google.com [209.85.160.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D6B8F372EF6 for ; Wed, 11 Mar 2026 19:51:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.50 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773258719; cv=none; b=HIcRHMEfAmDT7ZoGHbycvLVTOlr4tiPM85Ea8V3F/Tnc+KsZZuhzCbyqY+I+tZRinyGUmvQAYPrQTQ5fyxiBfM/nHSEtq/UV0uRMisa2leACE3OKjhlhRkFiPRDItfdn+UT5AYpzx/nT/f8zcFjO3Xovvf7pxfLFayTLMV7a87Q= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773258719; c=relaxed/simple; bh=7eHlGuashuOEH83D5jEJW47valze4//FIdIm/BV1VkU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=HK65/359/GrrX76mfgQsG+6H21F/4GVjWgO39Y5kQbs2EgTEsoQJU8AyoWBJ33MXHyMWMVlUs2iYZC3TYX3fw5UFCRK15pRRpbBdNu0BkbeNiGcJ/vntJYVrcv+Gy4AO+LszVdf28y/E0isxcuzoty8ZvBchzajVsJyYJ+ttroc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=WVKXV1Uy; arc=none smtp.client-ip=209.85.160.50 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="WVKXV1Uy" Received: by mail-oa1-f50.google.com with SMTP id 586e51a60fabf-41706b23543so189954fac.1 for ; Wed, 11 Mar 2026 12:51:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1773258715; x=1773863515; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=FZdSVXkR6xOKSFPXSWZwyldzzATjLCewLPLvELCw7LY=; b=WVKXV1UygGIb/GC8VUbMVLt8juWBDtFT8uY2q7dBNgiz4de1hN1ofdERXMJNFgptPf CvUBZPK6KdlgNKlSaccMuCDFKjylH+sgR3nNe7PRvLtma3zq6yl5PUYKhMP3gtLLy4hd B2miNud0UcBzyyCFTaiqVlQDED4cGaZwOSuT1X/CuRRgsHAJkRJzwEWDjdhf6ayMHXKr TwXR47NEjdP3RaCUoaff7L1tDuJ+ypTfvtECcwG5zJbxvi9Fr2qenFrJhZHbLNsORWjx Md0cuK1df/gQ88AH0+N/w99i2aBify+aJdUNyDrl03dYLeUgB6fFjgoNaAjCNFvxmATu jq9g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1773258715; x=1773863515; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=FZdSVXkR6xOKSFPXSWZwyldzzATjLCewLPLvELCw7LY=; b=ghhOXtBvBdPgl6qWOLGqsfhOYKa3MwefcRLKN2zAYVJnaQJO9VODBPEgA/CbabmbBb Vjw1Bt6YcPvRlpQXDxCJeZuOFy2Sc9mzxsxDl9N6Bf/jV4HvEpJXXtsUZeE6I/ljSQK0 /YfS7XIPkg7eubIs8xqH7oDd/ngaac2lCBFtwENkkVr+meNwBidkfXysHkJO8IK45W/U 6J7R7NWfNbZixXXX8QKmmWR1aHVzu7PIu4uYVQ4snF97C+20OAm1ZznQIKJHRi1hTR27 dFdX9byNtZV01PFc1QqNpB+bG+csyzlWulTJ1+VXlqIk+TVRNHQQS3HcdkeKDrvCTv+G FKkw== X-Forwarded-Encrypted: i=1; AJvYcCVVs5umhAu+dwv5+NT17Yoj2B7lgHl8bF6s7mLIG3R40ADIgA9/sl1MONHNOdU5kq7xNWUWaOGdc5ItR+k=@vger.kernel.org X-Gm-Message-State: AOJu0Yw89TN6OOoLnnheeZvOwvUVOHyZM+Xt3kpV+bn8IcHcZCtJa7ch 6HW/ND0j/InWOLwhyYCPgTWoaHen4ZIUacwcMpUnFZcJkgIO+B0w/ksb1S2dTA== X-Gm-Gg: ATEYQzzLJfdUggyqbwk0gIwHaQSylLXmecGFwO2wTShTPQP2CQ2pNyndpok8jFkjYzU f3rqNxBCTRkcdDNvSo4WFNJ4zqcF+drS/GL09ZFxkblFgieSqOgs7Cs6XtbuGOFJeLd3NY80rRD 1f7HpV0ClsZNV5ALg1saIhmwNfFcepBvYNRoimKCUdK5kSTS82Jk0yMTneNgzA7jfJW3MRUQAT0 PYXdmcv5YDY1wROzO23EgVapME9NT6b3jbMD3KjuEzk/Wwg0p56xHWk8k6kULGufNk04N62VF8d D/S1diAJG+lnInGxPJYnN+AXk9OnkQ6sC7RzG4ux8GAlPB0hH3avE4TO/cwTuOjd6VQA44CZvSp q6LAkgJrdAKmfxFiXdsNxivXCtJHP9OHt1862szSs1z4oeOzG5TE7b7E921zYE56NqSnz42ISr0 VsikgHnSbbTdqiaY0Ps00Syuvbp/YOXqX3 X-Received: by 2002:a05:6870:a346:b0:3e8:9b72:5cda with SMTP id 586e51a60fabf-41798ee621fmr598211fac.11.1773258715531; Wed, 11 Mar 2026 12:51:55 -0700 (PDT) Received: from localhost ([2a03:2880:10ff:48::]) by smtp.gmail.com with ESMTPSA id 586e51a60fabf-4177e26e2c6sm3145231fac.7.2026.03.11.12.51.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 11 Mar 2026 12:51:55 -0700 (PDT) From: Joshua Hahn To: Minchan Kim , Sergey Senozhatsky Cc: Nhat Pham , Nhat Pham , Johannes Weiner , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@meta.com Subject: [PATCH 01/11] mm/zsmalloc: Rename zs_object_copy to zs_obj_copy Date: Wed, 11 Mar 2026 12:51:38 -0700 Message-ID: <20260311195153.4013476-2-joshua.hahnjy@gmail.com> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20260311195153.4013476-1-joshua.hahnjy@gmail.com> References: <20260311195153.4013476-1-joshua.hahnjy@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" All the zsmalloc functions that operate on a zsmalloc object (encoded location values) are named "zs_obj_xxx", except for zs_object_copy. Rename zs_object_copy to zs_obj_copy to conform to the pattern. No functional changes intended. Signed-off-by: Joshua Hahn Reviewed-by: Nhat Pham Reviewed-by: Yosry Ahmed --- mm/zsmalloc.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 2c1430bf8d57..7a9b8f55d529 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -1416,7 +1416,7 @@ void zs_free(struct zs_pool *pool, unsigned long hand= le) } EXPORT_SYMBOL_GPL(zs_free); =20 -static void zs_object_copy(struct size_class *class, unsigned long dst, +static void zs_obj_copy(struct size_class *class, unsigned long dst, unsigned long src) { struct zpdesc *s_zpdesc, *d_zpdesc; @@ -1537,7 +1537,7 @@ static void migrate_zspage(struct zs_pool *pool, stru= ct zspage *src_zspage, =20 used_obj =3D handle_to_obj(handle); free_obj =3D obj_malloc(pool, dst_zspage, handle); - zs_object_copy(class, free_obj, used_obj); + zs_obj_copy(class, free_obj, used_obj); obj_idx++; obj_free(class->size, used_obj); =20 --=20 2.52.0 From nobody Tue Apr 7 17:16:24 2026 Received: from mail-oi1-f181.google.com (mail-oi1-f181.google.com [209.85.167.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7ECFA32E692 for ; Wed, 11 Mar 2026 19:51:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773258720; cv=none; b=blLaG7E51WK5jIlueB8RcNrU3TVhvSjcN7i0Pv01Ex3+elxC3CcIoWL3ip5SJ/aKmB6nLgZNDPDBsppHduM9tmlkgZnLTo4ed1k3LJDzJtd9K8uH5I3jkzJPs1+kJQ3h8Zjui8s2Hw5ruITZVItli71Hw07fDEWvkFcaDHEmd44= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773258720; c=relaxed/simple; bh=DRVOmiVm25Vx0HkEpJHAm8C+2fTQ3k8yx6y2TT+WBVk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=gm7/6L/zFPj3gZu93brTX0rTAE+94tA8gRwi/iuWmPsHiGOOc1Msx6bb8YbFzQ6Dzsx8CI0d4UvQ0fVnlQOwijB5G8mW9T4zbiz5/YWvGVPZvTeBjdtael0NBB24MIXu/CytE3JATI+vqC01jrTVCobiyCzFhLBdu9QtQ4cE65U= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=HL6XpnkB; arc=none smtp.client-ip=209.85.167.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="HL6XpnkB" Received: by mail-oi1-f181.google.com with SMTP id 5614622812f47-466ebbf7ff7so119278b6e.1 for ; Wed, 11 Mar 2026 12:51:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1773258717; x=1773863517; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=66epcnsZKRJCgUvj1UpodhEZPhy6a4TqXH4bGO7dPGI=; b=HL6XpnkBUY3BCcwxkaM54fSMpLQoyRR515Y6ldGeQOaBX4FFu6m6dMXOC7B9NDNCru f82HYLy/UNWhewpaEggQWhRKYAo7kT0Wk3xtfvCAuqYBTqP3lbxD0jD46BNVdBN0jqnL Wtg9KN43XqXTm1Pvky6GLRbem4sjzsr7KWMpiKX2+K4isrSvNSsAYzqKNKhaX9IbI6Zh 7+fPipx1h4tineWTCh7cPvFPhp2uXUZ1MXFJFYV9f5RZm2VgGHhSzCqD00uGE/Bb5Mo4 zod7Y7QVGU6G0Y+jirHS+ICf1Sy12jDVaQklCAeNubfBqufr3YW4RdMBybMHLUJKBAsD ly1w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1773258717; x=1773863517; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=66epcnsZKRJCgUvj1UpodhEZPhy6a4TqXH4bGO7dPGI=; b=O/9JL5GblaLNLyJwbJp3i595z3dNWKePHZYrFe9E0SoG3vsuOVNdLPtnbHbYxGitVW BaD50QKEe54aJO0A4fS42Q6fQBN5Xi6pKkquy+UdXrRvn4dVHaszwX+lCJB37NcqFs+W ppREwXP6R92utpOqEh0iOsB9cDed+WGJ0RuNRMc/ELbWO4aKu2mjAPVSRFsR+MopzDWj VyrJA4nv0OElU6JuXN6GS3VXghXFlFg7ytIq0FkSWL4tIoaSQe5s0YMx9R/ngC1tEgIq B/mFByCHYGWdIAzmf0SQg8B50V9LYXC45I2+KlOGsuQt+HKj5xOAqqdRb4XmAz26zndf cVjA== X-Forwarded-Encrypted: i=1; AJvYcCWW3CSIycAE59ZO6VWvmIhTlHTojTxxMdHS6c6UO9wAuoNOY6KU3BrMgmfaxT+Z6Z9Lm5kFfbJ0u0N1PGs=@vger.kernel.org X-Gm-Message-State: AOJu0Yxyg7aCMHo7s3vcxkC2DoFiRnG9rjYWL/kgSPgR7CvD3R8WXzdr jVZ+B4hz3Z1ZeqwnAP36Q8KtlQMdm2lqz/eB+XkLNir16M8QAwFFBcun X-Gm-Gg: ATEYQzxyZsqpSWd5/QROEcJ9R9UAnfxsax7oYjuuJixGLQECelZal7SdXl9nW+38N5i o2WCpJql3t0d8A+HtQZebChS/JLo+g5n2zGass8NZt9HK36LP7OWBZOApglSFGxWt3Y9jq0nTUZ T/UE4Ck5huR5gqw/rTIy1RQo3OW4zGiS59/feunOLRTx2TGbkAhcEKTXrYAfkQcX+LJllfSRoeh ybgEJUG8MIU8vC2EPkePDWKmvIzR/HrepxMJT7Bd12RgjhMyi4uV7xCDeGVD2Mg+rpOd12MrLc+ b6QhUlJARGyJk0Sw4Ah064URlPywyzGIK0Feh3+fF0CumTG3yBVocl1qzN4rOGHIscoAI+EsuZi XM2G/kFU9XHL44tZqkcCm2RWBnnejHyjhEh4KBcUiKEJIIf6FuQvF6V0mI/FKthlLqf5z/GqDi/ 4EEfjwHvrqKsoSLVhsvxC0UA== X-Received: by 2002:a05:6808:250f:b0:450:275b:d942 with SMTP id 5614622812f47-46733425adamr2199782b6e.10.1773258716902; Wed, 11 Mar 2026 12:51:56 -0700 (PDT) Received: from localhost ([2a03:2880:10ff:48::]) by smtp.gmail.com with ESMTPSA id 586e51a60fabf-41791432d87sm1535210fac.1.2026.03.11.12.51.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 11 Mar 2026 12:51:56 -0700 (PDT) From: Joshua Hahn To: Minchan Kim , Sergey Senozhatsky Cc: Nhat Pham , Nhat Pham , Johannes Weiner , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@meta.com Subject: [PATCH 02/11] mm/zsmalloc: Make all obj_idx unsigned ints Date: Wed, 11 Mar 2026 12:51:39 -0700 Message-ID: <20260311195153.4013476-3-joshua.hahnjy@gmail.com> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20260311195153.4013476-1-joshua.hahnjy@gmail.com> References: <20260311195153.4013476-1-joshua.hahnjy@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" object indices, which describe the location of an object in a zspage, cannot be negative. To reflect this most helpers calculate and return these values as unsigned ints. Convert find_alloced_obj, the only function that calculates obj_idx as a signed int, to use an unsigned int as well. No functional change intended. Signed-off-by: Joshua Hahn Reviewed-by: Nhat Pham Reviewed-by: Yosry Ahmed --- mm/zsmalloc.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 7a9b8f55d529..7758486e1d06 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -1491,10 +1491,11 @@ static void zs_obj_copy(struct size_class *class, u= nsigned long dst, * return handle. */ static unsigned long find_alloced_obj(struct size_class *class, - struct zpdesc *zpdesc, int *obj_idx) + struct zpdesc *zpdesc, + unsigned int *obj_idx) { unsigned int offset; - int index =3D *obj_idx; + unsigned int index =3D *obj_idx; unsigned long handle =3D 0; void *addr =3D kmap_local_zpdesc(zpdesc); =20 @@ -1521,7 +1522,7 @@ static void migrate_zspage(struct zs_pool *pool, stru= ct zspage *src_zspage, { unsigned long used_obj, free_obj; unsigned long handle; - int obj_idx =3D 0; + unsigned int obj_idx =3D 0; struct zpdesc *s_zpdesc =3D get_first_zpdesc(src_zspage); struct size_class *class =3D pool->size_class[src_zspage->class]; =20 --=20 2.52.0 From nobody Tue Apr 7 17:16:24 2026 Received: from mail-oa1-f46.google.com (mail-oa1-f46.google.com [209.85.160.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 890F137C0F8 for ; Wed, 11 Mar 2026 19:51:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.46 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773258723; cv=none; b=Vod3Q6cbhW2UcY4CFaF9ELGaZIKFyJkV6RyyHVbbJJqLVebvu0UUMj7+Wm7Hk+9pNx93/26f97kDnh3Ezv2jvpU8KayQrgYozROMX+qrPlxAlKz4S5RnHrwu+aLBDevbe5JoXl/nFQQ/jjVQ8rVPjV0rrvPIf1oG9mMsdHhOBx8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773258723; c=relaxed/simple; bh=DZCEel+gASKs1YULBRoMAZRVMrRj9weWbG1mBFEBaxA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=lsCwx+V5KhfLOVYo9trkB2vO7qh6AzlKVrxequ5QEsrWKp1bnY5qFEBX+012wtesuvz+Z+T7IlkaGkgZaRt+mOaf3Nqh7hFAqEMjgHgERXXgGflT7qM/fvrl3AJwxApadOEo0d48OkU/izf2ZceQ/iO3NpxsJyifnWHOJtMntTE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=YnnNA+Dy; arc=none smtp.client-ip=209.85.160.46 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="YnnNA+Dy" Received: by mail-oa1-f46.google.com with SMTP id 586e51a60fabf-4171451e89aso174745fac.3 for ; Wed, 11 Mar 2026 12:51:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1773258718; x=1773863518; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=xHiXRPU0oj2HIsyVL3yHXql4isoiizU2/nOjnBQn0Ag=; b=YnnNA+DyqHV+7WGhKKZQpaWCFo2/y8o9ZrLvOKKtAKeYx5Y70QC6p7Dho6K+SP3uGr jdLaja0Dkk7rA/4nKXBO4ay5zzAMM3mphLO8xAFVY9W5hfXJ0Y81CZ4f3fiGxh+0KcJp 2lAhp/neawJfeOInaoyIQjl2cbPzGsCdRgUFDDuwwthVGI8keIC/jVHWQG6A5ikE+EWM bin1EuSzvpBEKnIa697x1nVWzRy5YJnnf6nVpdGhAYpdhrbgNc20mm6zvuqNx45zgxpD KFsXlFiKD+cEmVdQGV2GU9k/ml9Iw6qoCPXcxoI/p7/EK6T1l84qUUmdpk3dK5witaPI Qp2A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1773258718; x=1773863518; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=xHiXRPU0oj2HIsyVL3yHXql4isoiizU2/nOjnBQn0Ag=; b=DI9efQ0p8yjaMMnms3fcpBS+0X/k8A7ik/4nIfPRF76qQUe9Wchb6y4E5So/YRRTIe kWbBSzwGOWi3GfCjFgw9dx3V/XS7zBWsNk9rD5JfRIJnclHm4s9x72rjhyvAc3jvXVfF lPHzwT57Rswyf1vQglmLgNmT1Y++wuTIib4kc03zluS0VlOZgZk4dgV2cSUvIIp4h9wS K7dGx2Y75Yr6pwUNe2AGqflNnHL+UgGZWNodpJMGTxT4Zg7csQsRFb3vAmgPUJfjt5wh WUj25lGup5x9DxCDpdSor74XltxJRYmjmWE/eMh4W/Il1ZKIPUMjPpdoz7qgxQGi5SD3 KRkg== X-Forwarded-Encrypted: i=1; AJvYcCXs5/HI28iUg4cwvXT3iychAPxVGzQIFv4Pon+dbNf6eDAWfW5PRnMrYyy3WDLVsVi2u/JT+hj44OLufM0=@vger.kernel.org X-Gm-Message-State: AOJu0YxFBLgxPzH1NOer6tsC205jFuVtEhbugP6JfHVfjglZj3v9Yc8U JGccOJcWTGF6ut8HvJjeT2bMDoS5M1CSkVL8j/Co7SWl0/QMzrbSrMTU X-Gm-Gg: ATEYQzxWK9m8M1kE4NueXUK1z01DQn3xIFeZWVcbAu7vXXT7QhREObflAbYnCAM5kxE GOwUtBNNohni0RKycP1Uqq8Y4L9Gt5Bxc4L+enoRfcBEf+BtikP3I8VGpeqrbf+vOXkp/alPLL2 1VZ//3dcmWqSGCZSMZf/uGzn6u+SmPViRT0D6SfyQ1tN130FpFCoVpOU17GpXtzxT8pgC/8CVwK ky2rIt7Z/tAYpo1pdGWQjt+hjLPsFu5lhXU4LJ9on5STtVibk8yxs8dOWRPhe3WB8tVKrZLvGZi ggLkNKQoNgQRNYtrOrO5Acpo0gUZR0AToZ+kZEYPJINULqcR9rt2rZapAJ6bEXd56z81xabW1AD StseSiCrGesaf2TcRnJA66bbYA6Dyn6z0wfceObD1hYi/5vxP5CEAEg3Qu9JAi8JUior5zdaTcH sFfg/BjHmnKzyACHR32NObmBfe1apZOw3v X-Received: by 2002:a05:6870:3314:b0:3d9:2fe2:f5c8 with SMTP id 586e51a60fabf-4177c8bcaf6mr2475836fac.32.1773258718211; Wed, 11 Mar 2026 12:51:58 -0700 (PDT) Received: from localhost ([2a03:2880:10ff:55::]) by smtp.gmail.com with ESMTPSA id 586e51a60fabf-4177e64a931sm3100578fac.14.2026.03.11.12.51.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 11 Mar 2026 12:51:57 -0700 (PDT) From: Joshua Hahn To: Minchan Kim , Sergey Senozhatsky Cc: Johannes Weiner , Yosry Ahmed , Nhat Pham , Nhat Pham , Chengming Zhou , Andrew Morton , linux-mm@kvack.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@meta.com Subject: [PATCH 03/11] mm/zsmalloc: Introduce conditional memcg awareness to zs_pool Date: Wed, 11 Mar 2026 12:51:40 -0700 Message-ID: <20260311195153.4013476-4-joshua.hahnjy@gmail.com> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20260311195153.4013476-1-joshua.hahnjy@gmail.com> References: <20260311195153.4013476-1-joshua.hahnjy@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Introduce 3 new fields to struct zs_pool to allow individual zpools to be "memcg-aware": memcg_aware, compressed_stat, and uncompressed_stat. memcg_aware is used in later patches to determine whether memory should be allocated to keep track of per-compresed object objgs. compressed_stat and uncompressed_stat are enum indices that point into memcg (node) stats that zsmalloc will account towards. In reality, these fields help distinguish between the two users of zsmalloc, zswap and zram. The enum indices compressed_stat and uncompressed_stat are parametrized to minimize zswap-specific hardcoding in zsmalloc. Suggested-by: Yosry Ahmed Signed-off-by: Joshua Hahn Acked-by: Nhat Pham --- drivers/block/zram/zram_drv.c | 3 ++- include/linux/zsmalloc.h | 5 ++++- mm/zsmalloc.c | 13 ++++++++++++- mm/zswap.c | 3 ++- 4 files changed, 20 insertions(+), 4 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index bca33403fc8b..d1eae5c20df7 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1980,7 +1980,8 @@ static bool zram_meta_alloc(struct zram *zram, u64 di= sksize) if (!zram->table) return false; =20 - zram->mem_pool =3D zs_create_pool(zram->disk->disk_name); + /* zram does not support memcg accounting */ + zram->mem_pool =3D zs_create_pool(zram->disk->disk_name, false, 0, 0); if (!zram->mem_pool) { vfree(zram->table); zram->table =3D NULL; diff --git a/include/linux/zsmalloc.h b/include/linux/zsmalloc.h index 478410c880b1..24fb2e0fdf67 100644 --- a/include/linux/zsmalloc.h +++ b/include/linux/zsmalloc.h @@ -23,8 +23,11 @@ struct zs_pool_stats { =20 struct zs_pool; struct scatterlist; +enum memcg_stat_item; =20 -struct zs_pool *zs_create_pool(const char *name); +struct zs_pool *zs_create_pool(const char *name, bool memcg_aware, + enum memcg_stat_item compressed_stat, + enum memcg_stat_item uncompressed_stat); void zs_destroy_pool(struct zs_pool *pool); =20 unsigned long zs_malloc(struct zs_pool *pool, size_t size, gfp_t flags, diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 7758486e1d06..3f0f42b78314 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -214,6 +214,9 @@ struct zs_pool { #ifdef CONFIG_COMPACTION struct work_struct free_work; #endif + bool memcg_aware; + enum memcg_stat_item compressed_stat; + enum memcg_stat_item uncompressed_stat; /* protect zspage migration/compaction */ rwlock_t lock; atomic_t compaction_in_progress; @@ -2050,6 +2053,9 @@ static int calculate_zspage_chain_size(int class_size) /** * zs_create_pool - Creates an allocation pool to work from. * @name: pool name to be created + * @memcg_aware: whether the consumer of this pool will account memcg stats + * @compressed_stat: compressed memcontrol stat item to account + * @uncompressed_stat: uncompressed memcontrol stat item to account * * This function must be called before anything when using * the zsmalloc allocator. @@ -2057,7 +2063,9 @@ static int calculate_zspage_chain_size(int class_size) * On success, a pointer to the newly created pool is returned, * otherwise NULL. */ -struct zs_pool *zs_create_pool(const char *name) +struct zs_pool *zs_create_pool(const char *name, bool memcg_aware, + enum memcg_stat_item compressed_stat, + enum memcg_stat_item uncompressed_stat) { int i; struct zs_pool *pool; @@ -2071,6 +2079,9 @@ struct zs_pool *zs_create_pool(const char *name) rwlock_init(&pool->lock); atomic_set(&pool->compaction_in_progress, 0); =20 + pool->memcg_aware =3D memcg_aware; + pool->compressed_stat =3D compressed_stat; + pool->uncompressed_stat =3D uncompressed_stat; pool->name =3D kstrdup(name, GFP_KERNEL); if (!pool->name) goto err; diff --git a/mm/zswap.c b/mm/zswap.c index e6ec3295bdb0..ff9abaa8aa38 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -257,7 +257,8 @@ static struct zswap_pool *zswap_pool_create(char *compr= essor) =20 /* unique name for each pool specifically required by zsmalloc */ snprintf(name, 38, "zswap%x", atomic_inc_return(&zswap_pools_count)); - pool->zs_pool =3D zs_create_pool(name); + pool->zs_pool =3D zs_create_pool(name, true, MEMCG_ZSWAP_B, + MEMCG_ZSWAPPED); if (!pool->zs_pool) goto error; =20 --=20 2.52.0 From nobody Tue Apr 7 17:16:24 2026 Received: from mail-oi1-f178.google.com (mail-oi1-f178.google.com [209.85.167.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 23BD9314D18 for ; Wed, 11 Mar 2026 19:52:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773258724; cv=none; b=k3DPeR6Uii8dyRDrLOUj/m9VJnY6R8GHxI7RCVJvY62ilzcquUsWyeRZtsc92x9VxRZOrphRpmM0ya4oaszX4QnOeQpgBaXVExxHEGCrl7cR6LJIjUe1csUW2BJ2696C/J0W2YGY+93GtVRoJijqnwrcKhp6nOrHc7tmcbX7h3Y= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773258724; c=relaxed/simple; bh=FlX1JxtGQ4L9YMM6+aD5mJkMLFZBnO5Nl+spgsSmm9o=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=hN2istWOdLpKLC9FAhTnpSQ/xhU0coL0z14uROVxTqfAxBC+y0RsLdg5X2oX24ByP22tUsLDzFX4/IP/+wQtORBpahi60hkzZm4f2udYeKp7dWwPMcc6BJVeJG2zAaTxKSAPe1grQ7SDdZCZR5uUD2yHlFqDbJX1CPhytLH2XM4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=YbPANPoy; arc=none smtp.client-ip=209.85.167.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="YbPANPoy" Received: by mail-oi1-f178.google.com with SMTP id 5614622812f47-463f00cda04so192182b6e.2 for ; Wed, 11 Mar 2026 12:52:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1773258719; x=1773863519; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=mnVFn/QWUod1peerD4NOctgVgrZtb1oHuY7D76awS8Y=; b=YbPANPoyXW0LwO6glEfUA2Fpx9mVsjufWa/85vXlALMXac8oAWxKTspdMrtCR9gc5J CktD1x7RR/vjNVCGXyFDo1tjYh21yvl8rpXqKNhQf4LSdhJzlNaRVyPZDK2QHzM8lljj 9yw++01j1o3GYGLxic4jo9kNZE7WLN4AGht3IH+Rj2UfbTaCfZKFzvSPD+GCstaKoUji +cHjUCNBpNmuBShzEEF2OPvnGf7tgRJggFO006DwtCJnUUX9KBRpYzQQ4iX0bjCssiVf 59stEfMDFLUS6HtZsbL/i+wItGXOVUVsiKrQMX6r3c/Csdfv+2WrBLRl0n7TZAvdR26R WKwQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1773258719; x=1773863519; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=mnVFn/QWUod1peerD4NOctgVgrZtb1oHuY7D76awS8Y=; b=iMHPG6eBZnhGdMXuBVbYI5zEUP9xm7ibtuKxvO/eupHGukVMq9Epdea3awTpDYgwOx 89g9KpxEEQggm+vjB1F5zaicW208+dxlNiY0DC+aLGKtZxd9GCTwzfAkLW+/E9MdSE7U c7qxCmln1G9zTOLZnuwuAIq8ysyqNDOUtYBX+g4XqZIhC63YwqaBzDIhHCytRNa3WLNA tH+I3hwS7DZg8TNENPli/Ke6LBZmpFo4nyBhwXjpoEq/OrLUGId9er2bgpIlJPUDbTGj bzRh/VArNo89EZQRBpqFbWFxsuWQx3k6gDHp9zETbHL64NvSI8ow2KfWGf1PWY69mBM/ wHKg== X-Forwarded-Encrypted: i=1; AJvYcCVhzbwCEw8BoTJK3WA7Tjg6RefqejPLM/bI7tRY++DUB/ibRluplIwzoHz7cOPtNbkLs0CMmsxkzU6S1yU=@vger.kernel.org X-Gm-Message-State: AOJu0YzGMsrf+KU3Mrx4kK9K3/4vwLfLC/bx0bmqOVDHzINiyBpeEss2 W9a4Z6Tgg7V3bkFb06MvIMAvZX+WgMyEI97ISJsA78yuvjn5qbko4bIS X-Gm-Gg: ATEYQzzqb2zQAJG1dGWoAxf6vy7McSKo3M7BrY3lBOwKGAyHyFdzOVlyvZk6wqgX7gf nSFwFZa+mURHDKP7Z/OY31LY74f4yjkpHdt3l7w8YYvxxLxZ2xNS2g2Wofb4s+mz3XMKXy0jmkX V7yvTFXcMAkvh371ME7dvC8EztpFZn17i0oFlY74LQglczc1XUiXUW3ws8zop0YG+skumimiExF JDi34k4XS6jj6PUCJaV4MFtzccVgIdBs0stI0pXZhTFJphlEQS6X6mmXkSYXi8+rXNHb6VYcxf9 XNYxzPq39Z5I9ZBp+mcXU+TOsxHfCWZlwcsECVD2Xpd4fktY1wheUKa99h3Sdp1bYq9jQLBcOAt sBcv9xh+JkDo1LnyIyx0zoQTzdUSv9TdMLALVwweVIoBHhcv57Ea07XGUBYNTLCqT6aqplqipga 2vdSXSzBhuQ4spt3IGu17nFg== X-Received: by 2002:a05:6808:6a83:b0:462:d09a:cdca with SMTP id 5614622812f47-46733545ca6mr2042383b6e.32.1773258719499; Wed, 11 Mar 2026 12:51:59 -0700 (PDT) Received: from localhost ([2a03:2880:10ff:4d::]) by smtp.gmail.com with ESMTPSA id 586e51a60fabf-4177e5e94b4sm2862488fac.11.2026.03.11.12.51.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 11 Mar 2026 12:51:59 -0700 (PDT) From: Joshua Hahn To: Minchan Kim , Sergey Senozhatsky Cc: Johannes Weiner , Harry Yoo , Yosry Ahmed , Nhat Pham , Nhat Pham , Chengming Zhou , Andrew Morton , linux-mm@kvack.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@meta.com Subject: [PATCH 04/11] mm/zsmalloc: Introduce objcgs pointer in struct zspage Date: Wed, 11 Mar 2026 12:51:41 -0700 Message-ID: <20260311195153.4013476-5-joshua.hahnjy@gmail.com> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20260311195153.4013476-1-joshua.hahnjy@gmail.com> References: <20260311195153.4013476-1-joshua.hahnjy@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Introduce an array of struct obj_cgroup pointers to zspage to keep track of compressed objects' memcg ownership, if the zs_pool has been made to be memcg-aware at creation time. Move the error path for alloc_zspage to a jump label to simplify the growing error handling path for a failed zpdesc allocation. Suggested-by: Johannes Weiner Suggested-by: Harry Yoo Signed-off-by: Joshua Hahn --- mm/zsmalloc.c | 34 ++++++++++++++++++++++++++-------- 1 file changed, 26 insertions(+), 8 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 3f0f42b78314..dcf99516227c 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -39,6 +39,7 @@ #include #include #include +#include #include "zpdesc.h" =20 #define ZSPAGE_MAGIC 0x58 @@ -273,6 +274,7 @@ struct zspage { struct zpdesc *first_zpdesc; struct list_head list; /* fullness list */ struct zs_pool *pool; + struct obj_cgroup **objcgs; struct zspage_lock zsl; }; =20 @@ -825,6 +827,8 @@ static void __free_zspage(struct zs_pool *pool, struct = size_class *class, zpdesc =3D next; } while (zpdesc !=3D NULL); =20 + if (pool->memcg_aware) + kfree(zspage->objcgs); cache_free_zspage(zspage); =20 class_stat_sub(class, ZS_OBJS_ALLOCATED, class->objs_per_zspage); @@ -946,6 +950,16 @@ static struct zspage *alloc_zspage(struct zs_pool *poo= l, if (!IS_ENABLED(CONFIG_COMPACTION)) gfp &=3D ~__GFP_MOVABLE; =20 + if (pool->memcg_aware) { + zspage->objcgs =3D kcalloc(class->objs_per_zspage, + sizeof(struct obj_cgroup *), + gfp & ~__GFP_HIGHMEM); + if (!zspage->objcgs) { + cache_free_zspage(zspage); + return NULL; + } + } + zspage->magic =3D ZSPAGE_MAGIC; zspage->pool =3D pool; zspage->class =3D class->index; @@ -955,14 +969,8 @@ static struct zspage *alloc_zspage(struct zs_pool *poo= l, struct zpdesc *zpdesc; =20 zpdesc =3D alloc_zpdesc(gfp, nid); - if (!zpdesc) { - while (--i >=3D 0) { - zpdesc_dec_zone_page_state(zpdescs[i]); - free_zpdesc(zpdescs[i]); - } - cache_free_zspage(zspage); - return NULL; - } + if (!zpdesc) + goto err; __zpdesc_set_zsmalloc(zpdesc); =20 zpdesc_inc_zone_page_state(zpdesc); @@ -973,6 +981,16 @@ static struct zspage *alloc_zspage(struct zs_pool *poo= l, init_zspage(class, zspage); =20 return zspage; + +err: + while (--i >=3D 0) { + zpdesc_dec_zone_page_state(zpdescs[i]); + free_zpdesc(zpdescs[i]); + } + if (pool->memcg_aware) + kfree(zspage->objcgs); + cache_free_zspage(zspage); + return NULL; } =20 static struct zspage *find_get_zspage(struct size_class *class) --=20 2.52.0 From nobody Tue Apr 7 17:16:24 2026 Received: from mail-oi1-f179.google.com (mail-oi1-f179.google.com [209.85.167.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E5DBA37BE75 for ; Wed, 11 Mar 2026 19:52:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773258726; cv=none; b=QG0DF5lV/42uU8lMv8g4a0wUzFe1kQ91rZ9YtnvlyIYxKlrzc/nvKnGvxK9vWaozj9vtRb5S1H9Y+2KRevn4VGh6w/reKZKMcSQ4OpCwxPoj0coHPTzO0Y5O7YAMUFxt3aDYc2O0nPYquCD5Tz9MS+sqcCYtxFQyrp/4j9p8A1k= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773258726; c=relaxed/simple; bh=o47mybQcyMuEdhSSbqd2I6tPg73t5Jjugzv2/OoCmDY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=osvmuHgIHGzGHYrMk+qTKNH3XiEL1J0mJ9vwgmVreIWx1EqpjHC1QHo1rKcssPrEnpWLhpZsr3Rz9lby90J4AgI99xsqlNjiuQzqfF2Ha2a21QwRA4L8JHKRP8jUeUXluNacFv/Ula8+D4j9AHmiI71vj5DdE06CspANC+bdNBQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=l1KshGNW; arc=none smtp.client-ip=209.85.167.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="l1KshGNW" Received: by mail-oi1-f179.google.com with SMTP id 5614622812f47-46708149af2so204623b6e.0 for ; Wed, 11 Mar 2026 12:52:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1773258722; x=1773863522; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=bVMnNhXg4JHLBLwKWI7WYc2GrcGR/wasu1xpRjHo+OU=; b=l1KshGNWocIlDALDa9RXMfEeRVJb6vgJuAR38o9Sp4d5XHy49RLqPOIoiau9DIWi76 evU1A1JFkDAVjpWHA4CSblkAA81EDDeJ4Sr0bJCuNfxBfhCS165X8RZjyFtGhJlmZQue Gyko7KA0PWxoNMR9RKje4qnNrKZjUsZD8TT/v6DODKp29jpeF2SmOTItJughQ1t/xII3 71tmd2TrMI00rKgGAhQowcL9PR38yuEdzJMBbstmMeM5pvRj1ONVxQmnGAhYxXKsqeN8 dBWofphbdO4a29PxQbmmZ/Mnp/ArQLLp3DyyUEwOpjm1hWgu6zVrXg7sKyDXjzjjq+B4 jYMw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1773258722; x=1773863522; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=bVMnNhXg4JHLBLwKWI7WYc2GrcGR/wasu1xpRjHo+OU=; b=G7QrsaASS86m3xru7STczh2JCmmfPM6RYhQVEkiwBaNv9sjgsy0kIdWWruQYmJMKFW ggiEOVfcr1YJ/UVBIRJpBrJ2Nol64A3a6N1XQD9mfbbsx99RiVjMqy+xn13koN8t4Q+A FsU0nB46BcTIfTfT4vY716B6RpAZ31kSf/blsJPVYqGQzorX4deJTVMgr6oGad28RDAY j2XaFMjuHTiKhsVOrJZETCleQtksXM8Jn5rCiwc5Tl8CwfCDrFJ38Ys5lXeAupuRcPtF H0yvykn+P0tNGg/z9VjJexCDxJIABfD5HSCcYANUJFqmw+0rmAQX7lqjGeZPm/6GqQ9g 8FDw== X-Forwarded-Encrypted: i=1; AJvYcCUvwbC5bN3N7hCysiTe0Fr5aPfss35q4ILHZJxFnCUA7XhIzGP6cl1STFXZ21ZcaHVqEJrBieVtGMUXR9w=@vger.kernel.org X-Gm-Message-State: AOJu0YwoVi+CxzQFEK1vE9zSjLVQx+jZoVeQJ41mAckdwafbeJiuKTp7 Elo0gqSPgNb2qv7+RjQu2s10+qtkfuzN3ydCfFN7gUqIq1VHu44LTEJw X-Gm-Gg: ATEYQzzM0dfL9/ewose7SC96GwbxknNtYHNJdUqqvK4H8oXcuIhN1QfPRHg7kuFdvTH hOi2sNMpIN+NxyWBhTeq//KCVXoF9e8p3WI+tIdBLMBmonwicjwdOxpZd1CJxXH7jexgKTo13/P jLvYne7ZWLCGawLWQH+g6tcKwpzQPEou/vYBSJB7mAQPAFN48r+PhGxugZvN06nFpIg67gfUnSB Kkar4eM2/8qfcMjPAjCSW8lQeYjq7WhrEj5qXh8r6CjLcbEXkbpv/3DWwcCTD8vwpYJdHzm8oOo o3RzwNFk0acuTFgeyb7S2/sXedxG09UZjhxQ6aY+WkPg3NEaAKv9IXhltVDptuFCVekQZivy0Cq 3HyeEgq80mYRoCeOW8ClB6Qc3NMCzZTplCQ5ZQxnGy7R1cJSY5WhGLfp9tRj7l/1n+y3L1ttCuX 5D09PC8lgTnzi7vLV8Z5HtZwngaQi+ X-Received: by 2002:a05:6808:1b1e:b0:467:baa:ed5c with SMTP id 5614622812f47-467333f24demr2109453b6e.12.1773258721631; Wed, 11 Mar 2026 12:52:01 -0700 (PDT) Received: from localhost ([2a03:2880:10ff::]) by smtp.gmail.com with ESMTPSA id 5614622812f47-46733f23116sm1835915b6e.0.2026.03.11.12.52.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 11 Mar 2026 12:52:00 -0700 (PDT) From: Joshua Hahn To: Minchan Kim , Sergey Senozhatsky Cc: Johannes Weiner , Jens Axboe , Yosry Ahmed , Nhat Pham , Nhat Pham , Chengming Zhou , Andrew Morton , linux-mm@kvack.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@meta.com Subject: [PATCH 05/11] mm/zsmalloc: Store obj_cgroup pointer in zspage Date: Wed, 11 Mar 2026 12:51:42 -0700 Message-ID: <20260311195153.4013476-6-joshua.hahnjy@gmail.com> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20260311195153.4013476-1-joshua.hahnjy@gmail.com> References: <20260311195153.4013476-1-joshua.hahnjy@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" With each zspage now having an array of obj_cgroup pointers, plumb the obj_cgroup pointer from the zswap / zram layer down to zsmalloc. zram still sees no visible change from its end. For the zswap path, store the obj_cgroup pointer after compression when writing the object, and erase the pointer when the object gets freed. The lifetime and charging of the obj_cgroup is still handled in the zswap layer. Suggested-by: Johannes Weiner Signed-off-by: Joshua Hahn --- drivers/block/zram/zram_drv.c | 7 ++++--- include/linux/zsmalloc.h | 3 ++- mm/zsmalloc.c | 25 ++++++++++++++++++++++++- mm/zswap.c | 6 +++--- 4 files changed, 33 insertions(+), 8 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index d1eae5c20df7..e68e408992e7 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -2232,7 +2232,7 @@ static int write_incompressible_page(struct zram *zra= m, struct page *page, } =20 src =3D kmap_local_page(page); - zs_obj_write(zram->mem_pool, handle, src, PAGE_SIZE); + zs_obj_write(zram->mem_pool, handle, src, PAGE_SIZE, NULL); kunmap_local(src); =20 slot_lock(zram, index); @@ -2297,7 +2297,7 @@ static int zram_write_page(struct zram *zram, struct = page *page, u32 index) return -ENOMEM; } =20 - zs_obj_write(zram->mem_pool, handle, zstrm->buffer, comp_len); + zs_obj_write(zram->mem_pool, handle, zstrm->buffer, comp_len, NULL); zcomp_stream_put(zstrm); =20 slot_lock(zram, index); @@ -2521,7 +2521,8 @@ static int recompress_slot(struct zram *zram, u32 ind= ex, struct page *page, return PTR_ERR((void *)handle_new); } =20 - zs_obj_write(zram->mem_pool, handle_new, zstrm->buffer, comp_len_new); + zs_obj_write(zram->mem_pool, handle_new, zstrm->buffer, + comp_len_new, NULL); zcomp_stream_put(zstrm); =20 slot_free(zram, index); diff --git a/include/linux/zsmalloc.h b/include/linux/zsmalloc.h index 24fb2e0fdf67..645957a156c4 100644 --- a/include/linux/zsmalloc.h +++ b/include/linux/zsmalloc.h @@ -23,6 +23,7 @@ struct zs_pool_stats { =20 struct zs_pool; struct scatterlist; +struct obj_cgroup; enum memcg_stat_item; =20 struct zs_pool *zs_create_pool(const char *name, bool memcg_aware, @@ -51,7 +52,7 @@ void zs_obj_read_sg_begin(struct zs_pool *pool, unsigned = long handle, struct scatterlist *sg, size_t mem_len); void zs_obj_read_sg_end(struct zs_pool *pool, unsigned long handle); void zs_obj_write(struct zs_pool *pool, unsigned long handle, - void *handle_mem, size_t mem_len); + void *handle_mem, size_t mem_len, struct obj_cgroup *objcg); =20 extern const struct movable_operations zsmalloc_mops; =20 diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index dcf99516227c..d4735451c273 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -1195,7 +1195,7 @@ void zs_obj_read_sg_end(struct zs_pool *pool, unsigne= d long handle) EXPORT_SYMBOL_GPL(zs_obj_read_sg_end); =20 void zs_obj_write(struct zs_pool *pool, unsigned long handle, - void *handle_mem, size_t mem_len) + void *handle_mem, size_t mem_len, struct obj_cgroup *objcg) { struct zspage *zspage; struct zpdesc *zpdesc; @@ -1216,6 +1216,11 @@ void zs_obj_write(struct zs_pool *pool, unsigned lon= g handle, class =3D zspage_class(pool, zspage); off =3D offset_in_page(class->size * obj_idx); =20 + if (objcg) { + WARN_ON_ONCE(!pool->memcg_aware); + zspage->objcgs[obj_idx] =3D objcg; + } + if (!ZsHugePage(zspage)) off +=3D ZS_HANDLE_SIZE; =20 @@ -1388,6 +1393,9 @@ static void obj_free(int class_size, unsigned long ob= j) f_offset =3D offset_in_page(class_size * f_objidx); zspage =3D get_zspage(f_zpdesc); =20 + if (zspage->pool->memcg_aware) + zspage->objcgs[f_objidx] =3D NULL; + vaddr =3D kmap_local_zpdesc(f_zpdesc); link =3D (struct link_free *)(vaddr + f_offset); =20 @@ -1538,6 +1546,16 @@ static unsigned long find_alloced_obj(struct size_cl= ass *class, return handle; } =20 +static void zs_migrate_objcg(struct zspage *s_zspage, struct zspage *d_zsp= age, + unsigned long used_obj, unsigned long free_obj) +{ + unsigned int s_idx =3D used_obj & OBJ_INDEX_MASK; + unsigned int d_idx =3D free_obj & OBJ_INDEX_MASK; + + d_zspage->objcgs[d_idx] =3D s_zspage->objcgs[s_idx]; + s_zspage->objcgs[s_idx] =3D NULL; +} + static void migrate_zspage(struct zs_pool *pool, struct zspage *src_zspage, struct zspage *dst_zspage) { @@ -1560,6 +1578,11 @@ static void migrate_zspage(struct zs_pool *pool, str= uct zspage *src_zspage, used_obj =3D handle_to_obj(handle); free_obj =3D obj_malloc(pool, dst_zspage, handle); zs_obj_copy(class, free_obj, used_obj); + + if (pool->memcg_aware) + zs_migrate_objcg(src_zspage, dst_zspage, + used_obj, free_obj); + obj_idx++; obj_free(class->size, used_obj); =20 diff --git a/mm/zswap.c b/mm/zswap.c index ff9abaa8aa38..68b87c3cc326 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -852,7 +852,7 @@ static void acomp_ctx_put_unlock(struct crypto_acomp_ct= x *acomp_ctx) } =20 static bool zswap_compress(struct page *page, struct zswap_entry *entry, - struct zswap_pool *pool) + struct zswap_pool *pool, struct obj_cgroup *objcg) { struct crypto_acomp_ctx *acomp_ctx; struct scatterlist input, output; @@ -912,7 +912,7 @@ static bool zswap_compress(struct page *page, struct zs= wap_entry *entry, goto unlock; } =20 - zs_obj_write(pool->zs_pool, handle, dst, dlen); + zs_obj_write(pool->zs_pool, handle, dst, dlen, objcg); entry->handle =3D handle; entry->length =3D dlen; =20 @@ -1414,7 +1414,7 @@ static bool zswap_store_page(struct page *page, return false; } =20 - if (!zswap_compress(page, entry, pool)) + if (!zswap_compress(page, entry, pool, objcg)) goto compress_failed; =20 old =3D xa_store(swap_zswap_tree(page_swpentry), --=20 2.52.0 From nobody Tue Apr 7 17:16:24 2026 Received: from mail-ot1-f48.google.com (mail-ot1-f48.google.com [209.85.210.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 368113806A9 for ; Wed, 11 Mar 2026 19:52:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.48 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773258729; cv=none; b=ThoPdMz3vkG1SimjLBP+F/tHmA+7UbsdyIDExtoJszMMIT8KtzOpDPJ0U/4QjegfWNgWHitGQnbOeEErHrKz/cU8IZsgcXDYqYr27fyPSQNoDTLj5g+RtbFo1sOrUrWc3VcT48rSnkno/XPCNan6054gbpwJUp/J7H9jmm8ft+I= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773258729; c=relaxed/simple; bh=GNNyj9xjEQZ9BXC1LZPtQuJLWEE+GEda29JW9PPaxcg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=MCBNNa0g1yS7SJisTesynJM04HY0uI2YUS2E4o18z6SdsbzteQVsPKM8KzMyF/Z1D8D/LuVqR7sN26MYAnXkVV1EhaNNrUbqE0PoFwfp3vTD2yoAIfR8i/YX4dq6gAKmbPbmP6aG5z4YpwL5nJg3VtvQuZUPItLBEXs3qp+ouDE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=AOGgZXxD; arc=none smtp.client-ip=209.85.210.48 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="AOGgZXxD" Received: by mail-ot1-f48.google.com with SMTP id 46e09a7af769-7d4be94eeacso357763a34.2 for ; Wed, 11 Mar 2026 12:52:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1773258723; x=1773863523; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=yLnaCWh6dwryZ7JtF26SIxlcfvOnnhqkpWeywtzyEhI=; b=AOGgZXxDO3YVP8M1fKbK57HEVBE8xm713DOISqP0JUdsF2Qz5KXMIv10ZCp0vg5Li6 quPIIPBKggFjAZo2C7wsBtzdB3X2bITxc2ZwtOLcgwBRFAlgyDeE4VChjO0wTaWmZ0Li xg7YlEjCK2Ca82hlmRDkBU24oYN/LTuOw+NdUnOd+ThclodSMTSS2+RLvh3sg/BkibGh oECPL2+Mu+Gn/Zxsk59rWZODyeeTsq7XWt0fRGoGv8alsFdiZyO0+3EepN1y40lgEmdH hUNFKM9j0DkeKgATT2SbkoPePlcwBhVKYaYXM7cUcGMtfKJ9j1LgnqPga0004gm9spKI tlPQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1773258723; x=1773863523; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=yLnaCWh6dwryZ7JtF26SIxlcfvOnnhqkpWeywtzyEhI=; b=covqy+Az4kcfBNKdSMFdkRSy0ut10CoDjYMFphGDeXbj+89SI49t1CHx1/PLGTve0o BL4uNFEqyj3PuXbEVWrfvb6+tngAfT1cHBgN5Vcqhn0731aGjUgjHObzOEJUutNrqKvC eBmKt4ir4bFNZ+TkIQjgOCTAT/N7x68dx7nxH/jHqNJgYKj4hliz44JZcp6rXkkhe/rz HXhBXNyQecGZh8OqlaHoBM73TOei2Noyii5K0WUEUo3vq32netGDdeDfDiU6GK9muCtG 3hdaO34RBEmBQVP3T0w+3aUm+Ud3ncUGme7pefm8dwZbNML2hUjq7zkYuP14lODijWAG 8Kng== X-Forwarded-Encrypted: i=1; AJvYcCVKzuObseg/IsFGki5B8GITRWVYF+S1yOebPzRc5VRLye8bCFW2Im8hHy3WJwtO+vhzqg4UXvjaoKFnVVc=@vger.kernel.org X-Gm-Message-State: AOJu0YzHIuPY051Wgm6tECiZmIxDrVDYwhOoFSMoC5ePRl+x/bnS2X8E bZSw91+cnE9WSApxjpsvmwHYpR4ReK5LF+H2Mpolk1WeWFZ30ln0dpgq X-Gm-Gg: ATEYQzwVy7omfE4OUS8fPhop4TMM6Gdzp+bJ8EZWjHnCsL1L23HvcplgpCe3jQsQwF4 yMQt4D9X4Ds5YNSld+VJKMZrJApw8RK4XvJiK0wJQyY3h+RssReE59yZ1q9vS5rjIX6RfDlDO8y 9CPHiWP7BepadUD1FqKK9xM8I5Ju0Az5749vupRYIdT4Q5AT3gUtghcS+KyYz2cuw+56dWh45Ad V2I5AUf6MHqPtQ9LJT1XaqCEpJbvFoyG8Sw7vjVaFv8DXJVtmXjHNeIqi8x2hplTo6Jb53o28CL oEUftqRi2aTIF4Z+2ZqmAPt6/j6WywWZqCpskXB1LxyyemaMrTTsvUsSJ9gdObGnGS5ITTSJm/h Ki72J3Ut0fjTzZVjf0MaZQz3p+kNMOPRhfxrL/wF74oMwe1ZtZS8JZARfKPXa/2O0Sy/h2iv0vd OPLTeG+pgI4pyRj3BDxr0q3g== X-Received: by 2002:a05:6830:449f:b0:7d7:551b:aeeb with SMTP id 46e09a7af769-7d76a4c6880mr2823367a34.0.1773258722891; Wed, 11 Mar 2026 12:52:02 -0700 (PDT) Received: from localhost ([2a03:2880:10ff:42::]) by smtp.gmail.com with ESMTPSA id 46e09a7af769-7d7749f1f5csm680280a34.6.2026.03.11.12.52.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 11 Mar 2026 12:52:02 -0700 (PDT) From: Joshua Hahn To: Minchan Kim , Sergey Senozhatsky Cc: Johannes Weiner , Yosry Ahmed , Nhat Pham , Nhat Pham , Chengming Zhou , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@meta.com Subject: [PATCH 06/11] mm/zsmalloc, zswap: Redirect zswap_entry->objcg to zspage Date: Wed, 11 Mar 2026 12:51:43 -0700 Message-ID: <20260311195153.4013476-7-joshua.hahnjy@gmail.com> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20260311195153.4013476-1-joshua.hahnjy@gmail.com> References: <20260311195153.4013476-1-joshua.hahnjy@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Now that obj_cgroups are tracked in the zspage, redirect the zswap layer to use the pointer stored in the zspage and remove the pointer in struct zswap_entry. This offsets the temporary memory increase caused by the duplicate storage of the obj_cgroup pointer and results in a net zero memory footprint change (aside from the array pointer and flags in zspage). The lifetime and charging of the obj_cgroup is still handled in the zswap layer. Clean up mem_cgroup_from_entry, which has no remaining callers. Suggested-by: Johannes Weiner Signed-off-by: Joshua Hahn --- include/linux/memcontrol.h | 5 ++++ include/linux/zsmalloc.h | 1 + mm/zsmalloc.c | 25 +++++++++++++++++++ mm/zswap.c | 50 +++++++++++++++++--------------------- 4 files changed, 53 insertions(+), 28 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 70b685a85bf4..0652db4ff2d5 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1072,6 +1072,11 @@ static inline struct mem_cgroup *get_mem_cgroup_from= _objcg(struct obj_cgroup *ob return NULL; } =20 +static inline struct mem_cgroup *obj_cgroup_memcg(struct obj_cgroup *objcg) +{ + return NULL; +} + static inline bool folio_memcg_kmem(struct folio *folio) { return false; diff --git a/include/linux/zsmalloc.h b/include/linux/zsmalloc.h index 645957a156c4..6010d8dac9ff 100644 --- a/include/linux/zsmalloc.h +++ b/include/linux/zsmalloc.h @@ -41,6 +41,7 @@ unsigned long zs_get_total_pages(struct zs_pool *pool); unsigned long zs_compact(struct zs_pool *pool); =20 unsigned int zs_lookup_class_index(struct zs_pool *pool, unsigned int size= ); +struct obj_cgroup *zs_lookup_objcg(struct zs_pool *pool, unsigned long han= dle); =20 void zs_pool_stats(struct zs_pool *pool, struct zs_pool_stats *stats); =20 diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index d4735451c273..a94ca8c26ad9 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -1049,6 +1049,31 @@ unsigned int zs_lookup_class_index(struct zs_pool *p= ool, unsigned int size) } EXPORT_SYMBOL_GPL(zs_lookup_class_index); =20 +struct obj_cgroup *zs_lookup_objcg(struct zs_pool *pool, unsigned long han= dle) +{ + unsigned long obj; + struct zpdesc *zpdesc; + struct zspage *zspage; + struct obj_cgroup *objcg; + unsigned int obj_idx; + + if (!pool->memcg_aware) + return NULL; + + read_lock(&pool->lock); + obj =3D handle_to_obj(handle); + obj_to_location(obj, &zpdesc, &obj_idx); + + zspage =3D get_zspage(zpdesc); + zspage_read_lock(zspage); + read_unlock(&pool->lock); + + objcg =3D zspage->objcgs[obj_idx]; + zspage_read_unlock(zspage); + + return objcg; +} + unsigned long zs_get_total_pages(struct zs_pool *pool) { return atomic_long_read(&pool->pages_allocated); diff --git a/mm/zswap.c b/mm/zswap.c index 68b87c3cc326..436066965413 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -193,7 +193,6 @@ struct zswap_entry { bool referenced; struct zswap_pool *pool; unsigned long handle; - struct obj_cgroup *objcg; struct list_head lru; }; =20 @@ -602,25 +601,13 @@ static int zswap_enabled_param_set(const char *val, * lru functions **********************************/ =20 -/* should be called under RCU */ -#ifdef CONFIG_MEMCG -static inline struct mem_cgroup *mem_cgroup_from_entry(struct zswap_entry = *entry) -{ - return entry->objcg ? obj_cgroup_memcg(entry->objcg) : NULL; -} -#else -static inline struct mem_cgroup *mem_cgroup_from_entry(struct zswap_entry = *entry) -{ - return NULL; -} -#endif - static inline int entry_to_nid(struct zswap_entry *entry) { return page_to_nid(virt_to_page(entry)); } =20 -static void zswap_lru_add(struct list_lru *list_lru, struct zswap_entry *e= ntry) +static void zswap_lru_add(struct list_lru *list_lru, struct zswap_entry *e= ntry, + struct obj_cgroup *objcg) { int nid =3D entry_to_nid(entry); struct mem_cgroup *memcg; @@ -637,19 +624,20 @@ static void zswap_lru_add(struct list_lru *list_lru, = struct zswap_entry *entry) * Similar reasoning holds for list_lru_del(). */ rcu_read_lock(); - memcg =3D mem_cgroup_from_entry(entry); + memcg =3D objcg ? obj_cgroup_memcg(objcg) : NULL; /* will always succeed */ list_lru_add(list_lru, &entry->lru, nid, memcg); rcu_read_unlock(); } =20 -static void zswap_lru_del(struct list_lru *list_lru, struct zswap_entry *e= ntry) +static void zswap_lru_del(struct list_lru *list_lru, struct zswap_entry *e= ntry, + struct obj_cgroup *objcg) { int nid =3D entry_to_nid(entry); struct mem_cgroup *memcg; =20 rcu_read_lock(); - memcg =3D mem_cgroup_from_entry(entry); + memcg =3D objcg ? obj_cgroup_memcg(objcg) : NULL; /* will always succeed */ list_lru_del(list_lru, &entry->lru, nid, memcg); rcu_read_unlock(); @@ -717,12 +705,15 @@ static void zswap_entry_cache_free(struct zswap_entry= *entry) */ static void zswap_entry_free(struct zswap_entry *entry) { - zswap_lru_del(&zswap_list_lru, entry); + struct obj_cgroup *objcg =3D zs_lookup_objcg(entry->pool->zs_pool, + entry->handle); + + zswap_lru_del(&zswap_list_lru, entry, objcg); zs_free(entry->pool->zs_pool, entry->handle); zswap_pool_put(entry->pool); - if (entry->objcg) { - obj_cgroup_uncharge_zswap(entry->objcg, entry->length); - obj_cgroup_put(entry->objcg); + if (objcg) { + obj_cgroup_uncharge_zswap(objcg, entry->length); + obj_cgroup_put(objcg); } if (entry->length =3D=3D PAGE_SIZE) atomic_long_dec(&zswap_stored_incompressible_pages); @@ -995,6 +986,7 @@ static int zswap_writeback_entry(struct zswap_entry *en= try, struct mempolicy *mpol; bool folio_was_allocated; struct swap_info_struct *si; + struct obj_cgroup *objcg; int ret =3D 0; =20 /* try to allocate swap cache folio */ @@ -1044,8 +1036,9 @@ static int zswap_writeback_entry(struct zswap_entry *= entry, xa_erase(tree, offset); =20 count_vm_event(ZSWPWB); - if (entry->objcg) - count_objcg_events(entry->objcg, ZSWPWB, 1); + objcg =3D zs_lookup_objcg(entry->pool->zs_pool, entry->handle); + if (objcg) + count_objcg_events(objcg, ZSWPWB, 1); =20 zswap_entry_free(entry); =20 @@ -1464,11 +1457,10 @@ static bool zswap_store_page(struct page *page, */ entry->pool =3D pool; entry->swpentry =3D page_swpentry; - entry->objcg =3D objcg; entry->referenced =3D true; if (entry->length) { INIT_LIST_HEAD(&entry->lru); - zswap_lru_add(&zswap_list_lru, entry); + zswap_lru_add(&zswap_list_lru, entry, objcg); } =20 return true; @@ -1593,6 +1585,7 @@ int zswap_load(struct folio *folio) bool swapcache =3D folio_test_swapcache(folio); struct xarray *tree =3D swap_zswap_tree(swp); struct zswap_entry *entry; + struct obj_cgroup *objcg; =20 VM_WARN_ON_ONCE(!folio_test_locked(folio)); =20 @@ -1621,8 +1614,9 @@ int zswap_load(struct folio *folio) folio_mark_uptodate(folio); =20 count_vm_event(ZSWPIN); - if (entry->objcg) - count_objcg_events(entry->objcg, ZSWPIN, 1); + objcg =3D zs_lookup_objcg(entry->pool->zs_pool, entry->handle); + if (objcg) + count_objcg_events(objcg, ZSWPIN, 1); =20 /* * When reading into the swapcache, invalidate our entry. The --=20 2.52.0 From nobody Tue Apr 7 17:16:24 2026 Received: from mail-oa1-f52.google.com (mail-oa1-f52.google.com [209.85.160.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 89B5538237E for ; Wed, 11 Mar 2026 19:52:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.52 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773258730; cv=none; b=QTJVmawpxFPYMvR2/egMwQ6PKv2yDKD1cSIJDnPRCN9P6mTjO+/aRzWjJYCe3C9IKQFCK+zcSr5MQhDorCC/5wh9Yzu6GV3cOOjOmP6QfKYiRqSGvdOfGRC32pKbrjFNqEqd9tZ2Kh8DtTSxGlPTEyptKEcswAIDuG52DiUhazc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773258730; c=relaxed/simple; bh=ltWZnUnXeRVqYVTEBtTy0eYCoDdqibyFZCtqJ7qUlq8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=rIwnPe3uGMfqUsyXNOrdh9Z/gfEmmyXfQcOyGir5VXdo+f3ckjXhD/DFBGSnq39Ja+pl0zaySAwi6Tvw4gwpNqO2axMJQYQkbDKZBNBD0u3dS1CJvY7unZ+NhEdl8sFds014UvN+vSzsNDxSvrDpdHfyWyjopVtzxjDBcE3J6lo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=foAkeCW2; arc=none smtp.client-ip=209.85.160.52 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="foAkeCW2" Received: by mail-oa1-f52.google.com with SMTP id 586e51a60fabf-41708f6c3feso189564fac.3 for ; Wed, 11 Mar 2026 12:52:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1773258724; x=1773863524; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=8u5o092L3ULCpmoipnWOnXiIPpIcwoMHmpMXH/cA6Oo=; b=foAkeCW2rLG48Jt1YjeSANAiyq3bdHTJ6/zTw+Sp677+qvV0xiPJ9PKZgczYzHi8NC e6ZQtHTwqyOaL4gEXamXx/JzgULF/0zGxwKfmqo2iEQ48VEo4hvVzUnV2VnFSAvEX47+ 4eBiEIP00POoi2ATzv4xQDaVA3s+CdtbqqEBspnJM5se/YZ60Be158amBJHcE38W2/UI ZU4Rs/QbF5TIlNTeg/ugHB9ShVS5lUO4mDK4zoSCwYvXg2sgfQ44825xzs9rLrrh6pqm igf02XDUUHdA22HqjVXojxGx0mN99w+tu0gSD8RwQX+Nu/ld7HvLsbb7DUP8V23ZcDi6 FZWw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1773258724; x=1773863524; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=8u5o092L3ULCpmoipnWOnXiIPpIcwoMHmpMXH/cA6Oo=; b=NmNgTyh53kt1bTe5ZjipvqnqEYrxsG29bASfVOfkxyC5Qdo+9DABK6zbCkzHKsaq3T CP8edMk2WOGWCXJNx9YhCldAG37t9fetx4GJ05/NRHZ57IIcoqR65XnvhC5UJoaNEdyC JnMmHexCt9h1WqHq9M2WyCOP+qUASJfaVL+v++Mo9ZnF3RxsqUQO20i+ouDYg1x9bQXg 5NUWg/6NYTvIGwnr+rglphHxswulT82AHNHNReJhmx+1ey8BbBy7JEDi+hOsITxJ5qXa ZZ1JwNsifoauAoU89oRWELe4i+ORA6lrxmbVJSUF7hBvJtHxWVoZJok0ieYZKMlxf9zG RdjA== X-Forwarded-Encrypted: i=1; AJvYcCU9MOzwe9qebdp/Mi1eedjprj8Alt1ST2iqbi+Y8ygAJZF+O6FnJXwXfDTgx2KDUmBU8dJPWOM5ok87w0A=@vger.kernel.org X-Gm-Message-State: AOJu0YwXxq765xucR7CdHXeWZ2mIksYot1f8A7VialfbGgsmAqMAo892 rqvVGHJJivJ/64L9j0YePY3/zarHbRvTBABdOmkgyE8j5hGk6bJaa1t8 X-Gm-Gg: ATEYQzxUKCzo+7z0b//HIcHK80A3eijeHM0ogSyjaTE3YlXoS6knO5+mOEZrB9oYW/a iGtFBT9UITbI+s5CCTeaMphjkhTV8dziBLfFLw1TQjfJQ4bUaapupuofX30CkxriHYdn679hLUM VVv50+Dn85dqD0rbOjHthh3PgtjxKXU0Yuy+FQyY7lgKZ8+SDi4rYWvYkPewy0975JiMriZ+hlr d8VJyDKvKqthhD1xSbUlqh52BrkMRMETXMctSdihhmGTJkgFz0CPAkLJiefNuwYdpQNc3EixeHW 4c4S7wZ1QGX0xZ5d7aVT5+fWPj+I8SgSGloJOoNt5HCJT7e9eAWDvRACFMtoTxaReP3FeFEcSZr LyE+AI6wUBvHzPX7kDg+tMvf3w7RF3mL4kGZ7QuJRNui7oF3iXWusxtftcKZqqNyCZGo7QAj4y3 hAzYNFhSUp3wmwwkdIiF1PWQ== X-Received: by 2002:a05:6870:169d:b0:3fa:eb8:4c92 with SMTP id 586e51a60fabf-4177c9c1dd0mr2187825fac.50.1773258724198; Wed, 11 Mar 2026 12:52:04 -0700 (PDT) Received: from localhost ([2a03:2880:10ff:46::]) by smtp.gmail.com with ESMTPSA id 586e51a60fabf-4177e6e82cdsm3116391fac.18.2026.03.11.12.52.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 11 Mar 2026 12:52:03 -0700 (PDT) From: Joshua Hahn To: Minchan Kim , Sergey Senozhatsky Cc: Johannes Weiner , Yosry Ahmed , Nhat Pham , Nhat Pham , Chengming Zhou , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@meta.com Subject: [PATCH 07/11] mm/zsmalloc, zswap: Handle objcg charging and lifetime in zsmalloc Date: Wed, 11 Mar 2026 12:51:44 -0700 Message-ID: <20260311195153.4013476-8-joshua.hahnjy@gmail.com> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20260311195153.4013476-1-joshua.hahnjy@gmail.com> References: <20260311195153.4013476-1-joshua.hahnjy@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Now that zswap_entries do not directly track obj_cgroups of the entries, handle the lifetime management and charging of these entries into the zsmalloc layer. One functional change is that zswap entries are now no longer accounted by the size of the compressed object, but by the size of the size_class slot they occupy. This brings charging one step closer to an accurate representation of the memory consumed in the zpdesc; even if a compressed object doesn't consume the entirety of a obj slot, we should account the entirety of the compressed object slot that the object makes unusable. While at it, also remove an unnecessary newline in obj_free. Signed-off-by: Joshua Hahn --- include/linux/memcontrol.h | 10 ------ mm/memcontrol.c | 54 ++----------------------------- mm/zsmalloc.c | 65 ++++++++++++++++++++++++++++++++++++-- mm/zswap.c | 8 ----- 4 files changed, 66 insertions(+), 71 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 0652db4ff2d5..701d9ab6fef1 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1851,22 +1851,12 @@ static inline bool memcg_is_dying(struct mem_cgroup= *memcg) =20 #if defined(CONFIG_MEMCG) && defined(CONFIG_ZSWAP) bool obj_cgroup_may_zswap(struct obj_cgroup *objcg); -void obj_cgroup_charge_zswap(struct obj_cgroup *objcg, size_t size); -void obj_cgroup_uncharge_zswap(struct obj_cgroup *objcg, size_t size); bool mem_cgroup_zswap_writeback_enabled(struct mem_cgroup *memcg); #else static inline bool obj_cgroup_may_zswap(struct obj_cgroup *objcg) { return true; } -static inline void obj_cgroup_charge_zswap(struct obj_cgroup *objcg, - size_t size) -{ -} -static inline void obj_cgroup_uncharge_zswap(struct obj_cgroup *objcg, - size_t size) -{ -} static inline bool mem_cgroup_zswap_writeback_enabled(struct mem_cgroup *m= emcg) { /* if zswap is disabled, do not block pages going to the swapping device = */ diff --git a/mm/memcontrol.c b/mm/memcontrol.c index a52da3a5e4fd..68139be66a4f 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -716,6 +716,7 @@ void mod_memcg_state(struct mem_cgroup *memcg, enum mem= cg_stat_item idx, =20 put_cpu(); } +EXPORT_SYMBOL(mod_memcg_state); =20 #ifdef CONFIG_MEMCG_V1 /* idx can be of type enum memcg_stat_item or node_stat_item. */ @@ -3169,11 +3170,13 @@ int obj_cgroup_charge(struct obj_cgroup *objcg, gfp= _t gfp, size_t size) { return obj_cgroup_charge_account(objcg, gfp, size, NULL, 0); } +EXPORT_SYMBOL(obj_cgroup_charge); =20 void obj_cgroup_uncharge(struct obj_cgroup *objcg, size_t size) { refill_obj_stock(objcg, size, true, 0, NULL, 0); } +EXPORT_SYMBOL(obj_cgroup_uncharge); =20 static inline size_t obj_full_size(struct kmem_cache *s) { @@ -5488,57 +5491,6 @@ bool obj_cgroup_may_zswap(struct obj_cgroup *objcg) return ret; } =20 -/** - * obj_cgroup_charge_zswap - charge compression backend memory - * @objcg: the object cgroup - * @size: size of compressed object - * - * This forces the charge after obj_cgroup_may_zswap() allowed - * compression and storage in zswap for this cgroup to go ahead. - */ -void obj_cgroup_charge_zswap(struct obj_cgroup *objcg, size_t size) -{ - struct mem_cgroup *memcg; - - if (!cgroup_subsys_on_dfl(memory_cgrp_subsys)) - return; - - VM_WARN_ON_ONCE(!(current->flags & PF_MEMALLOC)); - - /* PF_MEMALLOC context, charging must succeed */ - if (obj_cgroup_charge(objcg, GFP_KERNEL, size)) - VM_WARN_ON_ONCE(1); - - rcu_read_lock(); - memcg =3D obj_cgroup_memcg(objcg); - mod_memcg_state(memcg, MEMCG_ZSWAP_B, size); - mod_memcg_state(memcg, MEMCG_ZSWAPPED, 1); - rcu_read_unlock(); -} - -/** - * obj_cgroup_uncharge_zswap - uncharge compression backend memory - * @objcg: the object cgroup - * @size: size of compressed object - * - * Uncharges zswap memory on page in. - */ -void obj_cgroup_uncharge_zswap(struct obj_cgroup *objcg, size_t size) -{ - struct mem_cgroup *memcg; - - if (!cgroup_subsys_on_dfl(memory_cgrp_subsys)) - return; - - obj_cgroup_uncharge(objcg, size); - - rcu_read_lock(); - memcg =3D obj_cgroup_memcg(objcg); - mod_memcg_state(memcg, MEMCG_ZSWAP_B, -size); - mod_memcg_state(memcg, MEMCG_ZSWAPPED, -1); - rcu_read_unlock(); -} - bool mem_cgroup_zswap_writeback_enabled(struct mem_cgroup *memcg) { /* if zswap is disabled, do not block pages going to the swapping device = */ diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index a94ca8c26ad9..291194572a09 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -1028,6 +1028,59 @@ static bool zspage_empty(struct zspage *zspage) return get_zspage_inuse(zspage) =3D=3D 0; } =20 +#ifdef CONFIG_MEMCG +static void zs_charge_objcg(struct zs_pool *pool, struct obj_cgroup *objcg, + int size) +{ + struct mem_cgroup *memcg; + + if (!cgroup_subsys_on_dfl(memory_cgrp_subsys)) + return; + + VM_WARN_ON_ONCE(!(current->flags & PF_MEMALLOC)); + WARN_ON_ONCE(!pool->memcg_aware); + + /* PF_MEMALLOC context, charging must succeed */ + if (obj_cgroup_charge(objcg, GFP_KERNEL, size)) + VM_WARN_ON_ONCE(1); + + rcu_read_lock(); + memcg =3D obj_cgroup_memcg(objcg); + mod_memcg_state(memcg, pool->compressed_stat, size); + mod_memcg_state(memcg, pool->uncompressed_stat, 1); + rcu_read_unlock(); +} + +static void zs_uncharge_objcg(struct zs_pool *pool, struct obj_cgroup *obj= cg, + int size) +{ + struct mem_cgroup *memcg; + + if (!cgroup_subsys_on_dfl(memory_cgrp_subsys)) + return; + + WARN_ON_ONCE(!pool->memcg_aware); + + obj_cgroup_uncharge(objcg, size); + + rcu_read_lock(); + memcg =3D obj_cgroup_memcg(objcg); + mod_memcg_state(memcg, pool->compressed_stat, -size); + mod_memcg_state(memcg, pool->uncompressed_stat, -1); + rcu_read_unlock(); +} +#else +static void zs_charge_objcg(struct zs_pool *pool, struct obj_cgroup *objcg, + int size) +{ +} + +static void zs_uncharge_objcg(struct zs_pool *pool, struct obj_cgroup *obj= cg, + int size) +{ +} +#endif + /** * zs_lookup_class_index() - Returns index of the zsmalloc &size_class * that hold objects of the provided size. @@ -1244,6 +1297,8 @@ void zs_obj_write(struct zs_pool *pool, unsigned long= handle, if (objcg) { WARN_ON_ONCE(!pool->memcg_aware); zspage->objcgs[obj_idx] =3D objcg; + obj_cgroup_get(objcg); + zs_charge_objcg(pool, objcg, class->size); } =20 if (!ZsHugePage(zspage)) @@ -1409,17 +1464,23 @@ static void obj_free(int class_size, unsigned long = obj) struct link_free *link; struct zspage *zspage; struct zpdesc *f_zpdesc; + struct zs_pool *pool; unsigned long f_offset; unsigned int f_objidx; void *vaddr; =20 - obj_to_location(obj, &f_zpdesc, &f_objidx); f_offset =3D offset_in_page(class_size * f_objidx); zspage =3D get_zspage(f_zpdesc); + pool =3D zspage->pool; + + if (pool->memcg_aware && zspage->objcgs[f_objidx]) { + struct obj_cgroup *objcg =3D zspage->objcgs[f_objidx]; =20 - if (zspage->pool->memcg_aware) + zs_uncharge_objcg(pool, objcg, class_size); + obj_cgroup_put(objcg); zspage->objcgs[f_objidx] =3D NULL; + } =20 vaddr =3D kmap_local_zpdesc(f_zpdesc); link =3D (struct link_free *)(vaddr + f_offset); diff --git a/mm/zswap.c b/mm/zswap.c index 436066965413..bca29a6e18f3 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -711,10 +711,6 @@ static void zswap_entry_free(struct zswap_entry *entry) zswap_lru_del(&zswap_list_lru, entry, objcg); zs_free(entry->pool->zs_pool, entry->handle); zswap_pool_put(entry->pool); - if (objcg) { - obj_cgroup_uncharge_zswap(objcg, entry->length); - obj_cgroup_put(objcg); - } if (entry->length =3D=3D PAGE_SIZE) atomic_long_dec(&zswap_stored_incompressible_pages); zswap_entry_cache_free(entry); @@ -1437,10 +1433,6 @@ static bool zswap_store_page(struct page *page, * when the entry is removed from the tree. */ zswap_pool_get(pool); - if (objcg) { - obj_cgroup_get(objcg); - obj_cgroup_charge_zswap(objcg, entry->length); - } atomic_long_inc(&zswap_stored_pages); if (entry->length =3D=3D PAGE_SIZE) atomic_long_inc(&zswap_stored_incompressible_pages); --=20 2.52.0 From nobody Tue Apr 7 17:16:24 2026 Received: from mail-oa1-f45.google.com (mail-oa1-f45.google.com [209.85.160.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EB665382F24 for ; Wed, 11 Mar 2026 19:52:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.45 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773258731; cv=none; b=QZak0TvAsHJsfB9g5vB2RznkWFPV099WBgsESf/cArdYdsI9Mv9HT7cMVt5R1oVQsXqJS5+SqMoOz6mpitVwviVj3WiVYInQy6zcnW1lOOuwqi8/4NkR6K1YE5efOXDPBiup4+4fkrKikFgauaiV93Yx05YQ8LDoeeG4X3PTHIU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773258731; c=relaxed/simple; bh=HF2J8eGf9ezx76WeuiLEbVDyfaZqYDKuQnnfjxWeg54=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Cuhbs+JUvWMq115gkukHdr7MKYGCFlrg47z/hssC3j09XMdQeTSS/SpifDg6hER8bs7+1T9MH0SyfSf3MzRKgF2mmWy+5w9n2nj0eMd9HCnHD/Lj2VIczSBY0rsuZqZ3A4UtHJLWy7aWgRw1DON0f7q3JkCOKlz7rPANIpx7Nt4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Hxnu4Tee; arc=none smtp.client-ip=209.85.160.45 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Hxnu4Tee" Received: by mail-oa1-f45.google.com with SMTP id 586e51a60fabf-415e568a7ecso127248fac.0 for ; Wed, 11 Mar 2026 12:52:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1773258725; x=1773863525; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=9uGTcZbp7r+H1UaYa68xNPMIKqxUKJCwYhvaf1Y6xf8=; b=Hxnu4Tee603RcE0HCPjo47X15Y6evAy+vlzl0CBtZSOCXXb+i9TjhKjEwShWI2QEDD kRN3zlACPOTzsw1lCK2TJrOPr+Ii8vlrgoLVgF1eupIa6KAIOMaiqvql+OVhOq/AP23r 3ZwQiRma01TTys1e1JKbFkewmnc0/u6vjkeiNoKOVQtYLgHl0Zzr2YS2hwqW5g0sXz4Y amrrm0fx+FdRDG1f6UmMmA+65xic+rmOe3enDorLXS49ZidXl8quPNQXelizEc7Z56DQ Rrbo219EO0KIEjxlVEivA8AMTNIumxHdXgrlIke+N3a+rhdz5ip9IIgSXmtKqMmxWPpE aspQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1773258725; x=1773863525; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=9uGTcZbp7r+H1UaYa68xNPMIKqxUKJCwYhvaf1Y6xf8=; b=G0obxtLBmyhowRQj5BqLqWogcNoHmuoN1CKuj2p2pdUaa+ZwS+CDokGF+avJ9BxtID uMygJ7haP5Rq+fOD4smyK/6i7sOt2tDED/0fYAc/g9sJZfpE0YvHRh+be971Sgy9WE73 lRSTjKmYnuel0tUSC3BPPSvdBnHz9vNbQICevFrdt3KklDCfw37IQstfsEnJKPq0tnFT Ik5Ghmz3zDAojghbUgTOzHo6hTAXi/Qk3aPOHhhSKLWTxwR9R7Ff0SWmJo0WJNBT+Sj9 A8YyJYK87I6/xoZMhD+qBOx1RvLx3vqxLzxyc5KXrK11dX579PJWMsMjynsH37lnFFwU 5fjw== X-Forwarded-Encrypted: i=1; AJvYcCUGQ69WZW8NTNXw5QEzCEGOpsDcsYfKOymieg2loFbhhdQjeuvXnDUwGY62YQ7t4LEH7wVPmO+r9VubZ5E=@vger.kernel.org X-Gm-Message-State: AOJu0YwblJ6Rl/DCVRElK4UG33gsWzYBd3R0O8eNvxivAEjv44IXd2Mr /6i7TU7iarl5kZr+3D40B865AiTkS62dVP6znB6dh68RjBRlCVDKuH8/ X-Gm-Gg: ATEYQzwiR55Xsg5a+Lyt3coTlqR1mPdZnrKe4KDfgKtXkc1QuVU7gWBEBiCAORmEMQA 9SsHaJ5TbQv+dINpvD61kyJq3P/C+OgXqFWEVt9u4bMjI69ky6cMd7jT8eVNq1EAdGDoTngwnJg ZlQCTx3Onh3IjvAkg1RTGdzZEZ5I5PyMrImUNximsASKfyI7eAWMIx2wpnuyN0g2J7wxqtWxbG8 SVU6gJq6PW+szZFdzt6ijILXoYXkTWpReLFxwngzthm98vNMIHMKV1eceaOWIEYRyAta09b/cMI lBfPX89iDfBKZpzdj1JL4LVPiZeY68mLERFg9daI+Pw96Kct2QVOFriqeM7a/eWWLcYVkqnW1qv 6Ol2ByKS3+qkLVNLw0y3j/2TABOrBEIZkp92xOB24mMtfGfzLAHBkwad6DGR4eFDsXZiwvZ8auY TKkzTOe7QuvZk56ApzyPNZ3g== X-Received: by 2002:a05:6870:3c8b:b0:417:31f3:5def with SMTP id 586e51a60fabf-4177ca59b45mr2303457fac.32.1773258725506; Wed, 11 Mar 2026 12:52:05 -0700 (PDT) Received: from localhost ([2a03:2880:10ff:46::]) by smtp.gmail.com with ESMTPSA id 586e51a60fabf-4177e64a931sm3100861fac.14.2026.03.11.12.52.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 11 Mar 2026 12:52:05 -0700 (PDT) From: Joshua Hahn To: Minchan Kim , Sergey Senozhatsky Cc: Johannes Weiner , Yosry Ahmed , Nhat Pham , Nhat Pham , Chengming Zhou , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@meta.com Subject: [PATCH 08/11] mm/memcontrol: Track MEMCG_ZSWAPPED in bytes Date: Wed, 11 Mar 2026 12:51:45 -0700 Message-ID: <20260311195153.4013476-9-joshua.hahnjy@gmail.com> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20260311195153.4013476-1-joshua.hahnjy@gmail.com> References: <20260311195153.4013476-1-joshua.hahnjy@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Zswap compresses and uncompresses in PAGE_SIZE units, which simplifies the accounting for how much memory it has compressed. However, when a compressed object is stored at the boundary of two zspages, accounting at a PAGE_SIZE granularity makes it difficult to fractionally charge each backing zspage with the ratio of memory it backs for the compressed object. To make sub-PAGE_SIZE granularity charging possible for MEMCG_ZSWAPPED, track the value in bytes and adjust its accounting accordingly. No functional changes intended. Signed-off-by: Joshua Hahn Reviewed-by: Nhat Pham --- include/linux/memcontrol.h | 2 +- mm/memcontrol.c | 5 +++-- mm/zsmalloc.c | 4 ++-- mm/zswap.c | 8 +++++--- 4 files changed, 11 insertions(+), 8 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 701d9ab6fef1..ce2e598b5963 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -38,7 +38,7 @@ enum memcg_stat_item { MEMCG_VMALLOC, MEMCG_KMEM, MEMCG_ZSWAP_B, - MEMCG_ZSWAPPED, + MEMCG_ZSWAPPED_B, MEMCG_NR_STAT, }; =20 diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 68139be66a4f..1cb02d2febe8 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -342,7 +342,7 @@ static const unsigned int memcg_stat_items[] =3D { MEMCG_VMALLOC, MEMCG_KMEM, MEMCG_ZSWAP_B, - MEMCG_ZSWAPPED, + MEMCG_ZSWAPPED_B, }; =20 #define NR_MEMCG_NODE_STAT_ITEMS ARRAY_SIZE(memcg_node_stat_items) @@ -1364,7 +1364,7 @@ static const struct memory_stat memory_stats[] =3D { { "shmem", NR_SHMEM }, #ifdef CONFIG_ZSWAP { "zswap", MEMCG_ZSWAP_B }, - { "zswapped", MEMCG_ZSWAPPED }, + { "zswapped", MEMCG_ZSWAPPED_B }, #endif { "file_mapped", NR_FILE_MAPPED }, { "file_dirty", NR_FILE_DIRTY }, @@ -1412,6 +1412,7 @@ static int memcg_page_state_unit(int item) switch (item) { case MEMCG_PERCPU_B: case MEMCG_ZSWAP_B: + case MEMCG_ZSWAPPED_B: case NR_SLAB_RECLAIMABLE_B: case NR_SLAB_UNRECLAIMABLE_B: return 1; diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 291194572a09..24665d7cd4a9 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -1047,7 +1047,7 @@ static void zs_charge_objcg(struct zs_pool *pool, str= uct obj_cgroup *objcg, rcu_read_lock(); memcg =3D obj_cgroup_memcg(objcg); mod_memcg_state(memcg, pool->compressed_stat, size); - mod_memcg_state(memcg, pool->uncompressed_stat, 1); + mod_memcg_state(memcg, pool->uncompressed_stat, PAGE_SIZE); rcu_read_unlock(); } =20 @@ -1066,7 +1066,7 @@ static void zs_uncharge_objcg(struct zs_pool *pool, s= truct obj_cgroup *objcg, rcu_read_lock(); memcg =3D obj_cgroup_memcg(objcg); mod_memcg_state(memcg, pool->compressed_stat, -size); - mod_memcg_state(memcg, pool->uncompressed_stat, -1); + mod_memcg_state(memcg, pool->uncompressed_stat, -(int)PAGE_SIZE); rcu_read_unlock(); } #else diff --git a/mm/zswap.c b/mm/zswap.c index bca29a6e18f3..d81e2db4490b 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -257,7 +257,7 @@ static struct zswap_pool *zswap_pool_create(char *compr= essor) /* unique name for each pool specifically required by zsmalloc */ snprintf(name, 38, "zswap%x", atomic_inc_return(&zswap_pools_count)); pool->zs_pool =3D zs_create_pool(name, true, MEMCG_ZSWAP_B, - MEMCG_ZSWAPPED); + MEMCG_ZSWAPPED_B); if (!pool->zs_pool) goto error; =20 @@ -1214,8 +1214,10 @@ static unsigned long zswap_shrinker_count(struct shr= inker *shrinker, */ if (!mem_cgroup_disabled()) { mem_cgroup_flush_stats(memcg); - nr_backing =3D memcg_page_state(memcg, MEMCG_ZSWAP_B) >> PAGE_SHIFT; - nr_stored =3D memcg_page_state(memcg, MEMCG_ZSWAPPED); + nr_backing =3D memcg_page_state(memcg, MEMCG_ZSWAP_B); + nr_backing >>=3D PAGE_SHIFT; + nr_stored =3D memcg_page_state(memcg, MEMCG_ZSWAPPED_B); + nr_stored >>=3D PAGE_SHIFT; } else { nr_backing =3D zswap_total_pages(); nr_stored =3D atomic_long_read(&zswap_stored_pages); --=20 2.52.0 From nobody Tue Apr 7 17:16:24 2026 Received: from mail-oi1-f178.google.com (mail-oi1-f178.google.com [209.85.167.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1CF1A383C83 for ; Wed, 11 Mar 2026 19:52:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773258735; cv=none; b=Q+pVNnB0AoSABwhszKYskZVSEqq0GBC9+CQ4HwjRPFl+UNn7vY+zAvfBD9/GxrgfX5+QXrNa4dP2gbtzGgfQDvMzec92YXp5/0MSFvXQXJGbXJyTPDIqsbYUwgvK8q1NjMJs5xdJGgzeBIOf7l1j/VMXoSUcpNl+ox5ugwHWxPc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773258735; c=relaxed/simple; bh=dAUgueIEZ0YXut4fS4lJiG6S2nqFrA7mA7DBgHTcaq0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=q+iNxWinI263PRj3yKecST6Sbd6B0X+0sUZNnzTROwgnmpJGcBJv1rzb9QOqKEGJbwpVhjGslWVHNSXAVBKG/kT2fD6WyuvSJcOOJStSWMssG1ZrW+k24jNW7tn1Fev0tNPenHX0GEp52JWsX179Vjjwe3JvSL8iRsXr6RI7CLI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=grVR1c4e; arc=none smtp.client-ip=209.85.167.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="grVR1c4e" Received: by mail-oi1-f178.google.com with SMTP id 5614622812f47-464bba3a9easo225000b6e.0 for ; Wed, 11 Mar 2026 12:52:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1773258727; x=1773863527; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=e4vnZsb9mXNhtrNGbaJxIgoiKf+xedfYV6/25GSFY7Q=; b=grVR1c4ey3ufRp/etWAguwvOJXDqU8ZisNM0pjrvsGywWK9mBs/ufSOVSGKFRrdwwl f4+vdHVtGtTE/mML7wD8lpt+X1I7PuKwAPMqstHR7EYaQsg2QZAMt1kRJAZ3FCvsufoj nWFp9SyGSWlFuzO2df0egZQpImM2WtaEM9jdsUbMWKgtRtnVbT+fPNct/sod00SFmUaI ko+QdkI7q3dozCQZYhvlK5Z9YBb2dulhbQY8Kd+62Mx6dUtS0fLvoOtVn5ed/NxPAjX9 WaOq0Ie7Zk5vgIyJybYzrYi6EYdbR35FYXo+4CqrtXE56oIWL0Oon9qbrhX3LGLoWrzL CisQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1773258727; x=1773863527; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=e4vnZsb9mXNhtrNGbaJxIgoiKf+xedfYV6/25GSFY7Q=; b=M94PdAcP2JV+XE06aEum7VQ15cob8tYjINdmHHUFsntPhv6ESpd4IUJdptoNJxQcNf UM+56tDP6A6hbVmOQ4NoDLfyrvXBo5JUY/y8Nb8KfUbK0MYCD89OF6Bvrl3ymjIS2GZV RJxI8G9RGm7y5v3zYOMC0oNjS1LlcP2L/NsObPOQUpjry+iE1WQsid8PItAKR4Pyi+S0 E4tGprC9MZsXnWonWGVP8H/Amf4IpoitSoRz94piHmqUs2CKWaWZASa7WfqEojLXRn1W 600SISwpd6EOS0rjXdPRdJqF2nV7izGrBjYwzR59Md8WwCDCDbZp25O0DwiogvF1376I rnRw== X-Forwarded-Encrypted: i=1; AJvYcCU1Pl4DMJf0ykslAnN7usk0MzWJHk2FS12az68WFFg50mJjxwnDEX5AbkBLZZljjlqk7KehYgLuKYhdErU=@vger.kernel.org X-Gm-Message-State: AOJu0YylbubjkRp+WyLhroPxEesBC/DXigRvn6WvnoXjI51S6WWXLuI7 udq18cIms0bVk+1wimiyzGuwwb+2N1s9lFZiRF1vH+qV79V0r85NwAeg X-Gm-Gg: ATEYQzz6TKhN6fVzUi2Xasan/E9VTnvy/R+AJTc/O955fr3bxM18EuY9aZDvaMTMPgJ gPYnKO+YPQrf7W//6oloSutY/yNiaLBV3jALhU8+9496QuaD/nEeVbFG02bem7APHnqN7udZT11 6u3TYvpkNN91+2GO1ij2sA7i98tkqogFA9kqj60qBnnyaRejaz28oVbAWIJlEEo0Sg+1jpmiiFx g5VIxg3c/JOmEYiJ/+w4t8NBjx01i7ibhFas04v1PCzS6FnGJLhKlTeFFFciABbsqGYX5zYtd/t WTLZxepJEky2e/GGj0/blgi6ft0WzEKF7nN25tunnStyEw88LL7N8SW0QkjAh/ec9LBoZxvBaCS 1e7SH96ec3ZbsrVa8Z1SJ6h1J8GnZaX9CpJoLbMIkP9ZE/uzBMC8b6jDFvs/Ym5jww0zwGtU+BP JWYBzPnIqwikQiNzsZzSGUCZrBi1ste+aC X-Received: by 2002:a05:6820:c92:b0:67b:af79:4c1c with SMTP id 006d021491bc7-67bc887c1e1mr2569350eaf.2.1773258726845; Wed, 11 Mar 2026 12:52:06 -0700 (PDT) Received: from localhost ([2a03:2880:10ff:47::]) by smtp.gmail.com with ESMTPSA id 006d021491bc7-67bc9354e59sm1922332eaf.16.2026.03.11.12.52.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 11 Mar 2026 12:52:06 -0700 (PDT) From: Joshua Hahn To: Minchan Kim , Sergey Senozhatsky Cc: Johannes Weiner , Yosry Ahmed , Nhat Pham , Nhat Pham , Chengming Zhou , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Axel Rasmussen , Yuanchu Xie , Wei Xu , David Hildenbrand , Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Andrew Morton , cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@meta.com Subject: [PATCH 09/11] mm/vmstat, memcontrol: Track ZSWAP_B, ZSWAPPED_B per-memcg-lruvec Date: Wed, 11 Mar 2026 12:51:46 -0700 Message-ID: <20260311195153.4013476-10-joshua.hahnjy@gmail.com> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20260311195153.4013476-1-joshua.hahnjy@gmail.com> References: <20260311195153.4013476-1-joshua.hahnjy@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Now that memcg charging happens in the zsmalloc layer where we have both objcg and page information, we can specify which node's memcg lruvec zswapped memory should be accounted to. Move MEMCG_ZSWAP_B and MEMCG_ZSWAPPED_B from enum memcg_stat_item to enum node_stat_item. Reanme their prefixes from MEMCG to NR to reflect this move as well. In addition, decouple the updates of node stats (vmstat) and memcg-lruvec stats, since node stats can only track values at a PAGE_SIZE granularity. As a result of tracking zswap statistics at a finer granularity, the charging from zsmalloc also gets more complicated to cover the cases when the compressed object spans two zpdescs, which both live on different nodes. In this case, the memcg-lruvec of both node-memcg combinations are partially charged. memcg-lruvec stats are now updated precisely and proportionally when compressed objects are split across pages. Unfortunately for node stats, only NR_ZSWAP_B can be kept accurate. NR_ZSWAPPED_B works as a good best-effort value, but cannot proportionally account for compressed objects split across nodes due to the coarse PAGE_SIZE granularity of node stats. For such objects, NR_ZSWAPPED_B is accounted to the first zpdesc's node stats. Note that this is not a new inaccuracy, but one that is simply left unable to be fixed as part of these changes. The small inaccuracy is accepted in place of invasive changes across all of vmstat infrastructure to begin tracking stats at byte granularity. Finally, note that handling of objcg migrations across zspages (and their subsequent migrations across nodes) are handled in the next patch. Suggested-by: Johannes Weiner Signed-off-by: Joshua Hahn --- include/linux/memcontrol.h | 5 +- include/linux/mmzone.h | 2 + include/linux/zsmalloc.h | 6 +-- mm/memcontrol.c | 22 ++++---- mm/vmstat.c | 2 + mm/zsmalloc.c | 104 +++++++++++++++++++++++++++---------- mm/zswap.c | 7 ++- 7 files changed, 102 insertions(+), 46 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index ce2e598b5963..b03501e0c09b 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -37,8 +37,6 @@ enum memcg_stat_item { MEMCG_PERCPU_B, MEMCG_VMALLOC, MEMCG_KMEM, - MEMCG_ZSWAP_B, - MEMCG_ZSWAPPED_B, MEMCG_NR_STAT, }; =20 @@ -927,6 +925,9 @@ struct mem_cgroup *mem_cgroup_get_oom_group(struct task= _struct *victim, struct mem_cgroup *oom_domain); void mem_cgroup_print_oom_group(struct mem_cgroup *memcg); =20 +void mod_memcg_lruvec_state(struct lruvec *lruvec, enum node_stat_item idx, + int val); + /* idx can be of type enum memcg_stat_item or node_stat_item */ void mod_memcg_state(struct mem_cgroup *memcg, enum memcg_stat_item idx, int val); diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 3e51190a55e4..ae16a90491ac 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -258,6 +258,8 @@ enum node_stat_item { #ifdef CONFIG_HUGETLB_PAGE NR_HUGETLB, #endif + NR_ZSWAP_B, + NR_ZSWAPPED_B, NR_BALLOON_PAGES, NR_KERNEL_FILE_PAGES, NR_VM_NODE_STAT_ITEMS diff --git a/include/linux/zsmalloc.h b/include/linux/zsmalloc.h index 6010d8dac9ff..fd79916c7740 100644 --- a/include/linux/zsmalloc.h +++ b/include/linux/zsmalloc.h @@ -24,11 +24,11 @@ struct zs_pool_stats { struct zs_pool; struct scatterlist; struct obj_cgroup; -enum memcg_stat_item; +enum node_stat_item; =20 struct zs_pool *zs_create_pool(const char *name, bool memcg_aware, - enum memcg_stat_item compressed_stat, - enum memcg_stat_item uncompressed_stat); + enum node_stat_item compressed_stat, + enum node_stat_item uncompressed_stat); void zs_destroy_pool(struct zs_pool *pool); =20 unsigned long zs_malloc(struct zs_pool *pool, size_t size, gfp_t flags, diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 1cb02d2febe8..d87bc4beff16 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -333,6 +333,8 @@ static const unsigned int memcg_node_stat_items[] =3D { #ifdef CONFIG_HUGETLB_PAGE NR_HUGETLB, #endif + NR_ZSWAP_B, + NR_ZSWAPPED_B, }; =20 static const unsigned int memcg_stat_items[] =3D { @@ -341,8 +343,6 @@ static const unsigned int memcg_stat_items[] =3D { MEMCG_PERCPU_B, MEMCG_VMALLOC, MEMCG_KMEM, - MEMCG_ZSWAP_B, - MEMCG_ZSWAPPED_B, }; =20 #define NR_MEMCG_NODE_STAT_ITEMS ARRAY_SIZE(memcg_node_stat_items) @@ -737,9 +737,8 @@ unsigned long memcg_page_state_local(struct mem_cgroup = *memcg, int idx) } #endif =20 -static void mod_memcg_lruvec_state(struct lruvec *lruvec, - enum node_stat_item idx, - int val) +void mod_memcg_lruvec_state(struct lruvec *lruvec, enum node_stat_item idx, + int val) { struct mem_cgroup_per_node *pn; struct mem_cgroup *memcg; @@ -766,6 +765,7 @@ static void mod_memcg_lruvec_state(struct lruvec *lruve= c, =20 put_cpu(); } +EXPORT_SYMBOL(mod_memcg_lruvec_state); =20 /** * mod_lruvec_state - update lruvec memory statistics @@ -1363,8 +1363,8 @@ static const struct memory_stat memory_stats[] =3D { { "vmalloc", MEMCG_VMALLOC }, { "shmem", NR_SHMEM }, #ifdef CONFIG_ZSWAP - { "zswap", MEMCG_ZSWAP_B }, - { "zswapped", MEMCG_ZSWAPPED_B }, + { "zswap", NR_ZSWAP_B }, + { "zswapped", NR_ZSWAPPED_B }, #endif { "file_mapped", NR_FILE_MAPPED }, { "file_dirty", NR_FILE_DIRTY }, @@ -1411,8 +1411,8 @@ static int memcg_page_state_unit(int item) { switch (item) { case MEMCG_PERCPU_B: - case MEMCG_ZSWAP_B: - case MEMCG_ZSWAPPED_B: + case NR_ZSWAP_B: + case NR_ZSWAPPED_B: case NR_SLAB_RECLAIMABLE_B: case NR_SLAB_UNRECLAIMABLE_B: return 1; @@ -5482,7 +5482,7 @@ bool obj_cgroup_may_zswap(struct obj_cgroup *objcg) =20 /* Force flush to get accurate stats for charging */ __mem_cgroup_flush_stats(memcg, true); - pages =3D memcg_page_state(memcg, MEMCG_ZSWAP_B) / PAGE_SIZE; + pages =3D memcg_page_state(memcg, NR_ZSWAP_B) / PAGE_SIZE; if (pages < max) continue; ret =3D false; @@ -5511,7 +5511,7 @@ static u64 zswap_current_read(struct cgroup_subsys_st= ate *css, struct mem_cgroup *memcg =3D mem_cgroup_from_css(css); =20 mem_cgroup_flush_stats(memcg); - return memcg_page_state(memcg, MEMCG_ZSWAP_B); + return memcg_page_state(memcg, NR_ZSWAP_B); } =20 static int zswap_max_show(struct seq_file *m, void *v) diff --git a/mm/vmstat.c b/mm/vmstat.c index 86b14b0f77b5..389ff986ceac 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1279,6 +1279,8 @@ const char * const vmstat_text[] =3D { #ifdef CONFIG_HUGETLB_PAGE [I(NR_HUGETLB)] =3D "nr_hugetlb", #endif + [I(NR_ZSWAP_B)] =3D "zswap", + [I(NR_ZSWAPPED_B)] =3D "zswapped", [I(NR_BALLOON_PAGES)] =3D "nr_balloon_pages", [I(NR_KERNEL_FILE_PAGES)] =3D "nr_kernel_file_pages", #undef I diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 24665d7cd4a9..ab085961b0e2 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -216,8 +216,8 @@ struct zs_pool { struct work_struct free_work; #endif bool memcg_aware; - enum memcg_stat_item compressed_stat; - enum memcg_stat_item uncompressed_stat; + enum node_stat_item compressed_stat; + enum node_stat_item uncompressed_stat; /* protect zspage migration/compaction */ rwlock_t lock; atomic_t compaction_in_progress; @@ -823,6 +823,9 @@ static void __free_zspage(struct zs_pool *pool, struct = size_class *class, reset_zpdesc(zpdesc); zpdesc_unlock(zpdesc); zpdesc_dec_zone_page_state(zpdesc); + if (pool->memcg_aware) + dec_node_page_state(zpdesc_page(zpdesc), + pool->compressed_stat); zpdesc_put(zpdesc); zpdesc =3D next; } while (zpdesc !=3D NULL); @@ -974,6 +977,9 @@ static struct zspage *alloc_zspage(struct zs_pool *pool, __zpdesc_set_zsmalloc(zpdesc); =20 zpdesc_inc_zone_page_state(zpdesc); + if (pool->memcg_aware) + inc_node_page_state(zpdesc_page(zpdesc), + pool->compressed_stat); zpdescs[i] =3D zpdesc; } =20 @@ -985,6 +991,9 @@ static struct zspage *alloc_zspage(struct zs_pool *pool, err: while (--i >=3D 0) { zpdesc_dec_zone_page_state(zpdescs[i]); + if (pool->memcg_aware) + dec_node_page_state(zpdesc_page(zpdescs[i]), + pool->compressed_stat); free_zpdesc(zpdescs[i]); } if (pool->memcg_aware) @@ -1029,10 +1038,48 @@ static bool zspage_empty(struct zspage *zspage) } =20 #ifdef CONFIG_MEMCG -static void zs_charge_objcg(struct zs_pool *pool, struct obj_cgroup *objcg, - int size) +static void __zs_mod_memcg_lruvec(struct zs_pool *pool, struct zpdesc *zpd= esc, + struct obj_cgroup *objcg, int size, + int sign, unsigned long offset) { struct mem_cgroup *memcg; + struct lruvec *lruvec; + int compressed_size =3D size, original_size =3D PAGE_SIZE; + int nid =3D page_to_nid(zpdesc_page(zpdesc)); + int next_nid =3D nid; + + if (offset + size > PAGE_SIZE) { + struct zpdesc *next_zpdesc =3D get_next_zpdesc(zpdesc); + + next_nid =3D page_to_nid(zpdesc_page(next_zpdesc)); + if (nid !=3D next_nid) { + compressed_size =3D PAGE_SIZE - offset; + original_size =3D (PAGE_SIZE * compressed_size) / size; + } + } + + rcu_read_lock(); + memcg =3D obj_cgroup_memcg(objcg); + lruvec =3D mem_cgroup_lruvec(memcg, NODE_DATA(nid)); + mod_memcg_lruvec_state(lruvec, pool->compressed_stat, + sign * compressed_size); + mod_memcg_lruvec_state(lruvec, pool->uncompressed_stat, + sign * original_size); + + if (nid !=3D next_nid) { + lruvec =3D mem_cgroup_lruvec(memcg, NODE_DATA(next_nid)); + mod_memcg_lruvec_state(lruvec, pool->compressed_stat, + sign * (size - compressed_size)); + mod_memcg_lruvec_state(lruvec, pool->uncompressed_stat, + sign * (PAGE_SIZE - original_size)); + } + rcu_read_unlock(); +} + +static void zs_charge_objcg(struct zs_pool *pool, struct zpdesc *zpdesc, + struct obj_cgroup *objcg, int size, + unsigned long offset) +{ =20 if (!cgroup_subsys_on_dfl(memory_cgrp_subsys)) return; @@ -1044,18 +1091,19 @@ static void zs_charge_objcg(struct zs_pool *pool, s= truct obj_cgroup *objcg, if (obj_cgroup_charge(objcg, GFP_KERNEL, size)) VM_WARN_ON_ONCE(1); =20 - rcu_read_lock(); - memcg =3D obj_cgroup_memcg(objcg); - mod_memcg_state(memcg, pool->compressed_stat, size); - mod_memcg_state(memcg, pool->uncompressed_stat, PAGE_SIZE); - rcu_read_unlock(); + __zs_mod_memcg_lruvec(pool, zpdesc, objcg, size, 1, offset); + + /* + * Node-level vmstats are charged in PAGE_SIZE units. As a best-effort, + * always charge the uncompressed stats to the first zpdesc. + */ + inc_node_page_state(zpdesc_page(zpdesc), pool->uncompressed_stat); } =20 -static void zs_uncharge_objcg(struct zs_pool *pool, struct obj_cgroup *obj= cg, - int size) +static void zs_uncharge_objcg(struct zs_pool *pool, struct zpdesc *zpdesc, + struct obj_cgroup *objcg, int size, + unsigned long offset) { - struct mem_cgroup *memcg; - if (!cgroup_subsys_on_dfl(memory_cgrp_subsys)) return; =20 @@ -1063,20 +1111,24 @@ static void zs_uncharge_objcg(struct zs_pool *pool,= struct obj_cgroup *objcg, =20 obj_cgroup_uncharge(objcg, size); =20 - rcu_read_lock(); - memcg =3D obj_cgroup_memcg(objcg); - mod_memcg_state(memcg, pool->compressed_stat, -size); - mod_memcg_state(memcg, pool->uncompressed_stat, -(int)PAGE_SIZE); - rcu_read_unlock(); + __zs_mod_memcg_lruvec(pool, zpdesc, objcg, size, -1, offset); + + /* + * Node-level vmstats are charged in PAGE_SIZE units. As a best-effort, + * always uncharged the uncompressed stats from the first zpdesc. + */ + dec_node_page_state(zpdesc_page(zpdesc), pool->uncompressed_stat); } #else -static void zs_charge_objcg(struct zs_pool *pool, struct obj_cgroup *objcg, - int size) +static void zs_charge_objcg(struct zs_pool *pool, struct zpdesc *zpdesc, + struct obj_cgroup *objcg, int size, + unsigned long offset) { } =20 -static void zs_uncharge_objcg(struct zs_pool *pool, struct obj_cgroup *obj= cg, - int size) +static void zs_uncharge_objcg(struct zs_pool *pool, struct zpdesc *zpdesc, + struct obj_cgroup *objcg, int size, + unsigned long offset) { } #endif @@ -1298,7 +1350,7 @@ void zs_obj_write(struct zs_pool *pool, unsigned long= handle, WARN_ON_ONCE(!pool->memcg_aware); zspage->objcgs[obj_idx] =3D objcg; obj_cgroup_get(objcg); - zs_charge_objcg(pool, objcg, class->size); + zs_charge_objcg(pool, zpdesc, objcg, class->size, off); } =20 if (!ZsHugePage(zspage)) @@ -1477,7 +1529,7 @@ static void obj_free(int class_size, unsigned long ob= j) if (pool->memcg_aware && zspage->objcgs[f_objidx]) { struct obj_cgroup *objcg =3D zspage->objcgs[f_objidx]; =20 - zs_uncharge_objcg(pool, objcg, class_size); + zs_uncharge_objcg(pool, f_zpdesc, objcg, class_size, f_offset); obj_cgroup_put(objcg); zspage->objcgs[f_objidx] =3D NULL; } @@ -2191,8 +2243,8 @@ static int calculate_zspage_chain_size(int class_size) * otherwise NULL. */ struct zs_pool *zs_create_pool(const char *name, bool memcg_aware, - enum memcg_stat_item compressed_stat, - enum memcg_stat_item uncompressed_stat) + enum node_stat_item compressed_stat, + enum node_stat_item uncompressed_stat) { int i; struct zs_pool *pool; diff --git a/mm/zswap.c b/mm/zswap.c index d81e2db4490b..2e9352b46693 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -256,8 +256,7 @@ static struct zswap_pool *zswap_pool_create(char *compr= essor) =20 /* unique name for each pool specifically required by zsmalloc */ snprintf(name, 38, "zswap%x", atomic_inc_return(&zswap_pools_count)); - pool->zs_pool =3D zs_create_pool(name, true, MEMCG_ZSWAP_B, - MEMCG_ZSWAPPED_B); + pool->zs_pool =3D zs_create_pool(name, true, NR_ZSWAP_B, NR_ZSWAPPED_B); if (!pool->zs_pool) goto error; =20 @@ -1214,9 +1213,9 @@ static unsigned long zswap_shrinker_count(struct shri= nker *shrinker, */ if (!mem_cgroup_disabled()) { mem_cgroup_flush_stats(memcg); - nr_backing =3D memcg_page_state(memcg, MEMCG_ZSWAP_B); + nr_backing =3D memcg_page_state(memcg, NR_ZSWAP_B); nr_backing >>=3D PAGE_SHIFT; - nr_stored =3D memcg_page_state(memcg, MEMCG_ZSWAPPED_B); + nr_stored =3D memcg_page_state(memcg, NR_ZSWAPPED_B); nr_stored >>=3D PAGE_SHIFT; } else { nr_backing =3D zswap_total_pages(); --=20 2.52.0 From nobody Tue Apr 7 17:16:24 2026 Received: from mail-oo1-f49.google.com (mail-oo1-f49.google.com [209.85.161.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6EC0C383C9C for ; Wed, 11 Mar 2026 19:52:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.161.49 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773258735; cv=none; b=CEgiJQjV/d2gvSNxynvc9WJnASz2nDuruvSQUZmbCtj1ww98Mq/gO7jabKPpmNIIEetbAaEapC9J6x04VvQod85TpVVhp6C9MBmpqPfwtV1VfWY++pymjUorTfaWoIqH5xjIMwtLNB8NsyQYaA0C/roMOgMq9ZyZbM4+x5Oh73Y= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773258735; c=relaxed/simple; bh=bHJG3+YkzZlvrN13LSWBugwv+mqzY0z+0JYsWV64PVA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Jo0dyoGb0nSUGV3Kir0jbrEn4aUs4ISJM3LI5nZ6oPqg8ECPuyHDjJLbAhx5hQteqU4lYIjMokbnH3qCtSF0PdYOrdeHerrCsbCe8liaT/N3MNoY7a386kZskhJJRESZNSZ1TifgZYnguHcegc5XgW8DzAn9akBvyw7jfUsXWto= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=m120IXrF; arc=none smtp.client-ip=209.85.161.49 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="m120IXrF" Received: by mail-oo1-f49.google.com with SMTP id 006d021491bc7-67bb39a1122so174634eaf.3 for ; Wed, 11 Mar 2026 12:52:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1773258728; x=1773863528; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=lInkB9M3FDEV05WLG9mmxw0Eg6pj3q7yDvK5tck5dXo=; b=m120IXrF6QXv0Gn2wdeZl59k+OAuq7Q9l2OzioGV+FMn7SGoxG/yVf1tPoWjgjUdfs QD6dIS2YRwPq0UhRbTk9qdSqEmMRsVuLs63tDdX+vF/OIxc1yDRYkWJPCaJaHxQKdOmc jXgpl6S9ekomKEDacVw19sr/woa9kbIlKob8uM3ubasgEemWDXNe7ZAL5iI+uJppCZAp A14mEPzBhj63rbn9vjmxylLBwt1QEM+YvIZsb1LSuSaAubzYZ4mx266ZHQdzOiQ24VR2 h+1qETCCYhSeVvjypQ+zQ/sWaCuZX18r1dJoJzALt7C6mQ/OXHwLONrueG2Tsmrn87yl sJyQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1773258728; x=1773863528; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=lInkB9M3FDEV05WLG9mmxw0Eg6pj3q7yDvK5tck5dXo=; b=qSht0JUTAzDvOZOfwCFtQ/uzg+QwVz06PYjVLYJm4qPHjJoeehBVU7QBcW9TadMYlf wdRiRvZm5Afgxkx7PHrIkO0SnSu5tEf1zfSAbRrHPfRPp/iPKgxJQmSq1tB21iC+QHVW l63816SdRbRSvhv6OGxdUIOCo/XJi5Eli6kXcS5xECchKAB9w1qft0ALxppwyiTflVSl w4bBQxwrxu+++3bJIchdpa3tnyztuaQIxGnG+kwVoxpaZ0QEd4ivHPoNn7O0gDkZwobS waOH92Ub5PAd9B8s7Uhtf5k/RqpDmy79ZuJOLIFZH020c1OpvLd2+f4j6fyeS/NT0Pva ZaGw== X-Forwarded-Encrypted: i=1; AJvYcCUpeN8aR12pRJSgxQ720c/EO/bF7zBWzWrQSksczis9LdseHsrfYkfCTZS/4bsH2Uq41szK6rY9lcZ8HGg=@vger.kernel.org X-Gm-Message-State: AOJu0YzwAuLxPZlgxdJ8MtqXyB8rWLXTWjQJc81wG59m4UIPIgESBVpM WqouO9plP1il1++HarYpf5O7HHrnTlZhKWP5zkAInq8jL91dd1gdMbxJ X-Gm-Gg: ATEYQzwxhYJAVCtO/LR0TystqQ8wHL4F1UjLYj4VStQFwbSh1WZbjtBYIKGqMb9HLB2 48PBM8C2FU5rj01wmZCM3vtbQTU9QVtXUTV3dlY10oP61Bsw7hiHK4rBKD5KYGm1CzoUEKTs8Rd np6AIMybGY6/1ZQYtie4n0dOgqZ4WsMcHKC92RloE5DsrMYKYaW6W7S782FLat1m55X7C45BDE0 J3NQzZPT9Gael+KqEIJuDCVAf4c6aIoN4j+k60ZEhRk1QEi6IsFl0mzQnPexP987o+cyqU3uObS vYHYgG9yB9qc9hrIP3P/6xoap3KtoKbnE4ZpegSR7ip+ql7OZrSSe8d9Y1Ta/JzbwZIqZp+/QoC xQ3iLLu0AjNsGhP6ff6VtquURBjZyM/Aqn/9ZQrWaUUdqGRekhyWruIUg0hHsjhQWHvl2wCpchv wkiUyzO04QkqK7T9DDBBIF0Q== X-Received: by 2002:a05:6820:1629:b0:67b:b627:349b with SMTP id 006d021491bc7-67bc8a92e0amr2511581eaf.64.1773258728114; Wed, 11 Mar 2026 12:52:08 -0700 (PDT) Received: from localhost ([2a03:2880:10ff:5e::]) by smtp.gmail.com with ESMTPSA id 006d021491bc7-67bc9118bc7sm2041227eaf.6.2026.03.11.12.52.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 11 Mar 2026 12:52:07 -0700 (PDT) From: Joshua Hahn To: Minchan Kim , Sergey Senozhatsky Cc: Johannes Weiner , Yosry Ahmed , Nhat Pham , Nhat Pham , Harry Yoo , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@meta.com Subject: [PATCH 10/11] mm/zsmalloc: Handle single object charge migration in migrate_zspage Date: Wed, 11 Mar 2026 12:51:47 -0700 Message-ID: <20260311195153.4013476-11-joshua.hahnjy@gmail.com> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20260311195153.4013476-1-joshua.hahnjy@gmail.com> References: <20260311195153.4013476-1-joshua.hahnjy@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In zsmalloc, there are two types of migrations: Migrations of single compressed objects from one zspage to another, and substitutions of zpdescs from zspages. In both of these migrations, memcg association for the compressed objects do not change. However, the physical location of the compressed objects may change, which alters their lruvec association. In this patch, handle the single compressed object migration and transfer lruvec and node statistics across the affected lruvecs / nodes. Zsmalloc compressed objects, like slab objects, can span two pages. When a spanning object is migrated, possibly to another zspage where it spans two zpdescs, up to 4 nodes can be touched. Instead of enumerating all possible combinations of node migrations, simply uncharge entirely from the source (1 or 2 nodes) and charge entirely to the destination (1 or 2 nodes). s_off d_off v v ----------+ +---- -----+ +--------- ... ooo ooo xx| |x oo ... --> ... ooo x| |xx ooo oo ... ----------+ +---- -----+ +--------- pg1 pg2 pg3 pg4 s_zspage d_zspage To do this, calculate how much of the compressed object lives on each page and perform up to 4 uncharge-charges. Note that these operations cannot call the existing zs_{charge, uncharge}_objcg functions we introduced, since we are holding the class spin lock and obj_cgroup_charge can sleep. Signed-off-by: Joshua Hahn --- mm/zsmalloc.c | 74 ++++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 70 insertions(+), 4 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index ab085961b0e2..f3508ff8b3ab 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -1684,15 +1684,81 @@ static unsigned long find_alloced_obj(struct size_c= lass *class, return handle; } =20 +#ifdef CONFIG_MEMCG static void zs_migrate_objcg(struct zspage *s_zspage, struct zspage *d_zsp= age, - unsigned long used_obj, unsigned long free_obj) + unsigned long used_obj, unsigned long free_obj, + struct zs_pool *pool, int size) { - unsigned int s_idx =3D used_obj & OBJ_INDEX_MASK; - unsigned int d_idx =3D free_obj & OBJ_INDEX_MASK; + struct zpdesc *s_zpdesc, *d_zpdesc; + struct obj_cgroup *objcg; + struct mem_cgroup *memcg; + struct lruvec *l; + unsigned int s_idx, d_idx; + unsigned int s_off, d_off; + int charges[4], nids[4], partial; + int s_bytes_in_page, d_bytes_in_page; + int i; + + if (!cgroup_subsys_on_dfl(memory_cgrp_subsys)) + goto out; + + obj_to_location(used_obj, &s_zpdesc, &s_idx); + obj_to_location(free_obj, &d_zpdesc, &d_idx); + + objcg =3D s_zspage->objcgs[s_idx]; + if (!objcg) + goto out; + + /* + * The object migration here can touch up to 4 nodes. + * Instead of breaking down all possible combinations of node changes, + * just uncharge entirely from the source and charge entirely to the + * destination, even if there is are node overlaps between src and dst. + */ + s_off =3D (s_idx * size) % PAGE_SIZE; + d_off =3D (d_idx * size) % PAGE_SIZE; + s_bytes_in_page =3D min_t(int, size, PAGE_SIZE - s_off); + d_bytes_in_page =3D min_t(int, size, PAGE_SIZE - d_off); + + charges[0] =3D -s_bytes_in_page; + nids[0] =3D page_to_nid(zpdesc_page(s_zpdesc)); + charges[1] =3D -(size - s_bytes_in_page); /* 0 if object doesn't span */ + if (charges[1]) + nids[1] =3D page_to_nid(zpdesc_page(get_next_zpdesc(s_zpdesc))); + + charges[2] =3D d_bytes_in_page; + nids[2] =3D page_to_nid(zpdesc_page(d_zpdesc)); + charges[3] =3D size - d_bytes_in_page; /* 0 if object doesn't span */ + if (charges[3]) + nids[3] =3D page_to_nid(zpdesc_page(get_next_zpdesc(d_zpdesc))); =20 + rcu_read_lock(); + memcg =3D obj_cgroup_memcg(objcg); + for (i =3D 0; i < 4; i++) { + if (!charges[i]) + continue; + + l =3D mem_cgroup_lruvec(memcg, NODE_DATA(nids[i])); + partial =3D (PAGE_SIZE * charges[i]) / size; + mod_memcg_lruvec_state(l, pool->compressed_stat, charges[i]); + mod_memcg_lruvec_state(l, pool->uncompressed_stat, partial); + } + rcu_read_unlock(); + + dec_node_page_state(zpdesc_page(s_zpdesc), pool->uncompressed_stat); + inc_node_page_state(zpdesc_page(d_zpdesc), pool->uncompressed_stat); + +out: d_zspage->objcgs[d_idx] =3D s_zspage->objcgs[s_idx]; s_zspage->objcgs[s_idx] =3D NULL; } +#else +static void zs_migrate_objcg(struct zspage *s_zspage, struct zspage *d_zsp= age, + unsigned long used_obj, unsigned long free_obj, + struct zs_pool *pool, int size) +{ +} +#endif =20 static void migrate_zspage(struct zs_pool *pool, struct zspage *src_zspage, struct zspage *dst_zspage) @@ -1719,7 +1785,7 @@ static void migrate_zspage(struct zs_pool *pool, stru= ct zspage *src_zspage, =20 if (pool->memcg_aware) zs_migrate_objcg(src_zspage, dst_zspage, - used_obj, free_obj); + used_obj, free_obj, pool, class->size); =20 obj_idx++; obj_free(class->size, used_obj); --=20 2.52.0 From nobody Tue Apr 7 17:16:24 2026 Received: from mail-ot1-f52.google.com (mail-ot1-f52.google.com [209.85.210.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id ABAC719E839 for ; Wed, 11 Mar 2026 19:52:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.52 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773258735; cv=none; b=jF4rI3aj7AEqJdNSF4gPwU+We+uZcqoXrwX3S0gNGgjXvmk0Uay2beMjKZ9jRjqjDoCKknGa7KaFQnkLscWFlVuDelp1y0/fYJM/2srErAo/VO7jXOt6CGdrSMo+aBJR9qh9U2M4OmiDVd4v7fgos+FZHM/g4k8CF41IDzuFa90= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773258735; c=relaxed/simple; bh=qc0p7mZmuKWxiOXsyTmmf2kiIbU+0g44NqEbl2N314s=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=W0K9Do+YucRr7vIkaweWkAOyZectPnJWtfFo15lu2eQE/yCyqAtNIRBooRwsqGid11K5Xfy/ct7HMopxTLbP373u22B4pWKQ8eUs44eSBllSsfEZhtirfkW1MCMTmNXGO5v5jBOvuq2kF4eOcRJpmke94FprxJGe92jBe7VvIcc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=GWG3LVyG; arc=none smtp.client-ip=209.85.210.52 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="GWG3LVyG" Received: by mail-ot1-f52.google.com with SMTP id 46e09a7af769-7d4c383f2fcso271816a34.0 for ; Wed, 11 Mar 2026 12:52:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1773258729; x=1773863529; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=WGJ9MJa4Q34Av92QC7yL+Kbu9MyB37hTyusSYsRh4jE=; b=GWG3LVyGGN5FFvuOlbq3qQ2fT7au4gMdCKW8zCj6knBE0xCbwkxQ0OyekQJUN/nMAk 9+MC2rKJ6b708q99n1lh+bd2CpgXz2I4cuaqh6NQITC/8b40Xuh2tWJdTVUS5bj6JUyu vMargvam6QLepxbTvycsGJHBuljNgIWHZtuqz4Fzohw+ffuRuIKn1lj89SM0Ia+wE2yT 4v6qxf4d/WcbZLZzmwA2WhAebPu2DJSTAsL9/wv5uyRtjkxSFaCrBMryF5iZtsrOt4Gb xNr02nwJUuJPGmedLeiBGOaafwrq1PW5clyOvZlt9+qcIDL6efhusW7D3Oa4m0dHq9uq t2ZQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1773258729; x=1773863529; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=WGJ9MJa4Q34Av92QC7yL+Kbu9MyB37hTyusSYsRh4jE=; b=n6SIdK68cSg3jPjEhjCW/V1aieDnWZDFfOO/bKDYdkhTDgBfV5MzF6lK8fc85aivrh IkgPoxmRYyrPao2mmKUwY2Chjsj1isQZfWnfzolm1wdVEIibrIQSMcMne27QsZiiQMsN bBFqvUP/WBJN/EmhAHlHY6KmULObPk0d/LqEfXY3raA8M+f/HTZ5DLgn+6VIDJ14YuK4 IVIbHKgIss+yAvnB2rgB8rXw174Hi0tF7IVtw5PdFc0XbuAcfsQ7E6Gsmzb4bgrxo5Sy E76UtXbmmJOoCKK1/PITEUTr01uusHqbPHroYPTV7dmjDPUuaNTKciP5AnAwzJlp9HvG BjEg== X-Forwarded-Encrypted: i=1; AJvYcCVbh/9bxQ2mSuqN5+GJ5UvbN6E95NzzDtKAv67oG90XYfWUm4vOXammuM+eV5CHP/Yq3vulEDx/03UipXo=@vger.kernel.org X-Gm-Message-State: AOJu0Yxs2EKacCwOTIb0gYdXaaLwuk67aooPfiwlbWUE9Trs6bNuGYqq DUkSxfI4gKxEXJEdUALGdrQnpbH/mibUnWCrcpLYCxjNUoGcaZcxhskq8J9LWg== X-Gm-Gg: ATEYQzxWr4nv8mNG1hNk4SFmJ1rQuvHOutSDgkyhbrE/c3eVZuHlLBUsHSqrWBul0+W 9eh0VDzh43ibDZ0RCgpPlZF9uBFXPWoRHCBF63iRz7gMu8/3ZURw0xlytO54bMJrtsN1W8GMYHk cbnDALH6/lC+8cjhg+sujNBc8YqI5Syt5TFawtTqPNlEq2hDnzR9aHumHnbVsCIzWv3Hzas2cPu CCulkzCSsK2OAlBGvAusgHtFsAEVMM12G+u09EBXDyF5Kcx65eVy8BifMLYJHZEIlOYZh9IJ2Zl W6tvMu5SDhe2MIk16Vp2O9SyXydFsxACFX1nLb4layAptgGFBZdZWXpGhxIRtEqkIBu4+3w//2o NnMlu/cJ+NTx63G1WOEbhernL8ZQWjSB2eHdtXv1g4yoKwD9+szec7JDn3oFQbHGoXOlaiEo06d r8vvuEYhVwUhCOlol/GvZlXuFmK/SU/I9l X-Received: by 2002:a05:6830:7004:b0:7c7:61e0:a4ee with SMTP id 46e09a7af769-7d76a73b581mr2550615a34.11.1773258729419; Wed, 11 Mar 2026 12:52:09 -0700 (PDT) Received: from localhost ([2a03:2880:10ff:4e::]) by smtp.gmail.com with ESMTPSA id 46e09a7af769-7d76ae39b39sm2746167a34.15.2026.03.11.12.52.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 11 Mar 2026 12:52:08 -0700 (PDT) From: Joshua Hahn To: Minchan Kim , Sergey Senozhatsky Cc: Johannes Weiner , Yosry Ahmed , Nhat Pham , Nhat Pham , Harry Yoo , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@meta.com Subject: [PATCH 11/11] mm/zsmalloc: Handle charge migration in zpdesc substitution Date: Wed, 11 Mar 2026 12:51:48 -0700 Message-ID: <20260311195153.4013476-12-joshua.hahnjy@gmail.com> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20260311195153.4013476-1-joshua.hahnjy@gmail.com> References: <20260311195153.4013476-1-joshua.hahnjy@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In zsmalloc, there are two types of migrations: Migrations of single compressed objects from one zspage to another, and substitutions of zpdescs from zspages. In both of these migrations, memcg association for the compressed objects do not change. However, the physical location of the compressed objects may change, which alters their lruvec association. In this patch, handle the substitution of zpdescs from zspages, which may change the node of all objects present (wholly or partially). Take special care to address the partial compressed object at the beginning of the swapped out zpdesc. "Ownership" of spanning objects are associated to the zpdesc it begins on. Thus, when handling the first compressed object, we must iterate through the (up to 4) zpdescs present in the zspage to find the previous zpdesc, then retrieve the object's zspage-wide index. For the same reason, pool->uncompressed_stat, which can only be accounted at PAGE_SIZE granularity for the node statistics, are accounted for objects beginning in the zpdesc. Likewise for the spanning object at the end of the replaced zpdesc, account only the amount that lives on the zpdesc. Note that these operations cannot call the existing zs_{charge, uncharge}_objcg functions we introduced, since we are holding the class spin lock and obj_cgroup_charge can sleep. Signed-off-by: Joshua Hahn --- mm/zsmalloc.c | 92 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 92 insertions(+) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index f3508ff8b3ab..a4c90447d28e 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -1932,6 +1932,94 @@ static bool zs_page_isolate(struct page *page, isola= te_mode_t mode) return page_zpdesc(page)->zspage; } =20 +#ifdef CONFIG_MEMCG +static void zs_migrate_lruvec(struct zs_pool *pool, struct obj_cgroup *obj= cg, + int old_nid, int new_nid, int charge, + int obj_size) +{ + struct mem_cgroup *memcg; + struct lruvec *old_lruvec, *new_lruvec; + int partial; + + if (old_nid =3D=3D new_nid || !objcg) + return; + + /* Proportional (partial) uncompressed share for this portion */ + partial =3D (PAGE_SIZE * charge) / obj_size; + + rcu_read_lock(); + memcg =3D obj_cgroup_memcg(objcg); + old_lruvec =3D mem_cgroup_lruvec(memcg, NODE_DATA(old_nid)); + new_lruvec =3D mem_cgroup_lruvec(memcg, NODE_DATA(new_nid)); + + mod_memcg_lruvec_state(old_lruvec, pool->compressed_stat, -charge); + mod_memcg_lruvec_state(new_lruvec, pool->compressed_stat, charge); + + mod_memcg_lruvec_state(old_lruvec, pool->uncompressed_stat, -partial); + mod_memcg_lruvec_state(new_lruvec, pool->uncompressed_stat, partial); + rcu_read_unlock(); +} + +/* + * Transfer per-lruvec and node-level stats when a zspage replaces a zpdesc + * with one from a different NUMA node. Must be called while old_zpdesc is + * still linked to the zspage. memcg-level charges are unchanged. + */ +static void zs_page_migrate_lruvec(struct zs_pool *pool, struct zspage *zs= page, + struct zpdesc *old_zpdesc, + struct zpdesc *new_zpdesc, + struct size_class *class) +{ + int size =3D class->size; + int old_nid =3D page_to_nid(zpdesc_page(old_zpdesc)); + int new_nid =3D page_to_nid(zpdesc_page(new_zpdesc)); + unsigned int off, first_obj_offset, page_offset =3D 0; + unsigned int idx; + struct zpdesc *cursor =3D zspage->first_zpdesc; + + if (old_nid =3D=3D new_nid) + return; + + while (cursor !=3D old_zpdesc) { + cursor =3D get_next_zpdesc(cursor); + page_offset +=3D PAGE_SIZE; + } + + first_obj_offset =3D get_first_obj_offset(old_zpdesc); + idx =3D (page_offset + first_obj_offset) / size; + + /* Boundary object spaning from the previous zpdesc*/ + if (idx > 0 && zspage->objcgs[idx - 1]) + zs_migrate_lruvec(pool, zspage->objcgs[idx - 1], + old_nid, new_nid, first_obj_offset, size); + + for (off =3D first_obj_offset; + off < PAGE_SIZE && idx < class->objs_per_zspage; + idx++, off +=3D size) { + struct obj_cgroup *objcg =3D zspage->objcgs[idx]; + int bytes_on_page =3D min_t(int, size, PAGE_SIZE - off); + + if (!objcg) + continue; + + zs_migrate_lruvec(pool, objcg, old_nid, new_nid, + bytes_on_page, size); + + dec_node_page_state(zpdesc_page(old_zpdesc), + pool->uncompressed_stat); + inc_node_page_state(zpdesc_page(new_zpdesc), + pool->uncompressed_stat); + } +} +#else +static void zs_page_migrate_lruvec(struct zs_pool *pool, struct zspage *zs= page, + struct zpdesc *old_zpdesc, + struct zpdesc *new_zpdesc, + struct size_class *class) +{ +} +#endif + static int zs_page_migrate(struct page *newpage, struct page *page, enum migrate_mode mode) { @@ -2004,6 +2092,10 @@ static int zs_page_migrate(struct page *newpage, str= uct page *page, } kunmap_local(s_addr); =20 + /* Transfer lruvec/node stats while old zpdesc is still linked */ + if (pool->memcg_aware) + zs_page_migrate_lruvec(pool, zspage, zpdesc, newzpdesc, class); + replace_sub_page(class, zspage, newzpdesc, zpdesc); /* * Since we complete the data copy and set up new zspage structure, --=20 2.52.0