From nobody Tue Apr 7 17:15:08 2026 Received: from mail-ot1-f41.google.com (mail-ot1-f41.google.com [209.85.210.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DE28144CF4E for ; Thu, 26 Feb 2026 19:29:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.41 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772134184; cv=none; b=SttkaIo/Mos5cxOBdV/VrrOHgtmh+7CkIGgrWrg3DuNpTj1MCLpUiU8+y05qgk7lR7KBGjHQWAOKK7DSIRSUGJcvY1RtSSVxoymZ3ManQqrzdtc9efMCDP1Vu6og14Q5U8yM+FG9dfv2WgQ2vrVqMd4SyQmTmh0WUvP5rJ7KbRQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772134184; c=relaxed/simple; bh=6oULxnjxBKraHfOSFODqOjtws7MQfoIUuo5SUlQgyzk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=s2V5gkfgSLOBkFEvCsKqaoTy0RzqCObJ9shHOOrtDADxlpNoch+88gqnFqV58FcMxD+OnW31LBoeocK80wXYmqEZo72x4sDSLeDY6pt8hTpZ2AO03NYUbYdntdkCIug7j+JbHOG6Szdvqwg1pWXsk9PS1wwsRBA/1yk8BQu/i6Y= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=RxFsBHn3; arc=none smtp.client-ip=209.85.210.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="RxFsBHn3" Received: by mail-ot1-f41.google.com with SMTP id 46e09a7af769-7d18c654458so550296a34.3 for ; Thu, 26 Feb 2026 11:29:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1772134182; x=1772738982; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=GxU9M0uqLwBLhcOAnkTTmqCLiEjdRi64cRGLMJGuVY4=; b=RxFsBHn3VQd7gCOjNtVx5WL8AYSUUu30LH1HoLVp+OHhymhRCHbSWxqQwiADh+AmsV Fgbq/5MtAwqwyHJhxzfEUswzKa+nmR73bPUTAoEbNy3UuLS4iNFAJObYd/O/rGvGi4+D w/NlvasJFgVZJU+/zjthKsChsUJKrdpj2fdaVbTkwDkxim8mIgS4bn9YXa/hkjds2Rsk K7xXOn78+vEvm/lo30Wh4VvfabqgUo43meP4Eg+nco3sAuGIkkOyCvikpy9s/r32gB6J nxsnbW9HLLdRF63XYrDy7usiFcHac0qwEal67umbEDl1c6rRHyK6X2sHELSgO+eD0TrS PxEg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1772134182; x=1772738982; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=GxU9M0uqLwBLhcOAnkTTmqCLiEjdRi64cRGLMJGuVY4=; b=XK6fSC2b169iljAA7dBJwW7Ugnx+3lW/p4yaH9Noj21xgr+uCz9W57y2sFVVIQLAx2 /2LdWgnknWTZ7nrDEHm7tshtELVljO7KihQ31nK3o+BV6rNK7UpQQDDFcbjk5R792m2W 98M9spI66i1vQlmkUuoRPb7Ht/IJ++aHvmeFLy6EOvoEpBRqaqvHmFJUjgexMahfJXWb aZzXfbwh8JUDCwtEAGI67+88eJSfmxtUMACT6KXC7g6CiPrAp1b7KT7HmVfKUbE9IUah w2/1emViPlx2c7Mmy0ASiSfKU7w0RrrI9dGCETJ1GNc2qQZqG/u3BvtrFEB7JRoOgm3X o6Vw== X-Forwarded-Encrypted: i=1; AJvYcCUYcGMHW7fIvw8ekV2bZjUZDm3tUttDwPACfuQ3AS8BlMpFAku2xXxRprdaGK8uKpNLgezaw80ON1qJaL0=@vger.kernel.org X-Gm-Message-State: AOJu0Yx1qfLzYivEejtmCX6HfIJhWSmz+v1kmF9mgK6QQc3e51pTJHSJ fMn7x64btWrajfwZCZGyo4gxXxMcGBqP3awo0DMcyuZ8AnHMjjwYnIDK7ccwkNFH X-Gm-Gg: ATEYQzy572+4d8nj4YCLatKqvzGTPFdHLvbrNoAvkb4ecTRfFjZSiW9ceCuwKsJYKKR pTsE2smci1Ck/unvsekARvuOVLbfCcbHn35Eu0TuQIVreQ88BzfAZWrDgPdG9YEmhM4cmrHecMl dvw2a5Tfh9SB7A2D2cIcWhTwJiUM4MAqs3DDO2STDLh0R5vM0grK8PcUlUVvzIIQ6Rpsyw4miGv QeJAcxVDQU/qg2iofutUH7JN70NIiK4yeyUpPzhaJattYl1UuczVQPfHEXzWE6/U54GGK/UfKaX NsAavaqYPyJcwQm0GXM+AOnwo7UxsiyxMFP5b4Xlha9kGyH0i6lATDhVLRC71yqSbtLx5bvp+7I MT8JiOlEMp6ewqt9gDIPYSaMlyqnKmWGt855uyElb/Qof8tgpqT0Q520Byhnfw0fWYxw/8bkB+L V2wa2aKCiKPvpTrlaz2hK4 X-Received: by 2002:a05:6830:81ce:b0:7cf:ddb7:8823 with SMTP id 46e09a7af769-7d591b34e79mr366803a34.11.1772134181770; Thu, 26 Feb 2026 11:29:41 -0800 (PST) Received: from localhost ([2a03:2880:10ff:3::]) by smtp.gmail.com with ESMTPSA id 46e09a7af769-7d586625e4esm2415605a34.15.2026.02.26.11.29.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 26 Feb 2026 11:29:41 -0800 (PST) From: Joshua Hahn To: Minchan Kim , Sergey Senozhatsky Cc: Johannes Weiner , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@meta.com Subject: [PATCH 1/8] mm/zsmalloc: Rename zs_object_copy to zs_obj_copy Date: Thu, 26 Feb 2026 11:29:24 -0800 Message-ID: <20260226192936.3190275-2-joshua.hahnjy@gmail.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20260226192936.3190275-1-joshua.hahnjy@gmail.com> References: <20260226192936.3190275-1-joshua.hahnjy@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" All the zsmalloc functions that operate on a zsmalloc object (encoded location values) are named "zs_obj_xxx", except for zs_object_copy. Rename zs_object_copy to zs_obj_copy to conform to the pattern. No functional changes intended. Signed-off-by: Joshua Hahn --- mm/zsmalloc.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index d5d1c27b3852..0ca2e94af5ad 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -1416,7 +1416,7 @@ void zs_free(struct zs_pool *pool, unsigned long hand= le) } EXPORT_SYMBOL_GPL(zs_free); =20 -static void zs_object_copy(struct size_class *class, unsigned long dst, +static void zs_obj_copy(struct size_class *class, unsigned long dst, unsigned long src) { struct zpdesc *s_zpdesc, *d_zpdesc; @@ -1537,7 +1537,7 @@ static void migrate_zspage(struct zs_pool *pool, stru= ct zspage *src_zspage, =20 used_obj =3D handle_to_obj(handle); free_obj =3D obj_malloc(pool, dst_zspage, handle); - zs_object_copy(class, free_obj, used_obj); + zs_obj_copy(class, free_obj, used_obj); obj_idx++; obj_free(class->size, used_obj); =20 --=20 2.47.3 From nobody Tue Apr 7 17:15:08 2026 Received: from mail-ot1-f45.google.com (mail-ot1-f45.google.com [209.85.210.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 31DCE44D013 for ; Thu, 26 Feb 2026 19:29:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.45 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772134185; cv=none; b=eO00o/pfdGK3mAsi9M0klIg7UUBNGZ8/l9rXXkw29CX7wMH6YEuo1fTv1OIWXm8HGRJfyz+yNjZxg5Ga2IZ8lkh0puN0k/wWSJjpG8lrFnVpp+aD3Vq4Zg8er37sdmS7w+acYlFRfZW+fAqpRrl/FyqoRk3q0y8pLQhKNMFkRCc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772134185; c=relaxed/simple; bh=alZfvIU72ngV1euNni8omcShAt5mQpGrR+lWK2GAbZQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Q9RrJVsB9pUOHKSvVh84WDsxY8L1Cgr+VSPKIc7Y2hQfuqNezoj1HcPnqSVz6b8D3dTBOlS//j/tfJIRNnpw2pXDDg7n26X9RmCgx/Mo2F0BfJaiXjVTE3/NFOq9mGdpqGvP1qhdNzNMW4yNdqP5liyFAigIWRuuuJY02XFSiv0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Q+BRFPjU; arc=none smtp.client-ip=209.85.210.45 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Q+BRFPjU" Received: by mail-ot1-f45.google.com with SMTP id 46e09a7af769-7d4c4cacf78so557314a34.0 for ; Thu, 26 Feb 2026 11:29:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1772134183; x=1772738983; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=xmGkLxxoqKOr20qSUXIJFsIMVgh1ZbOFwoeb0w7HC5g=; b=Q+BRFPjUHllZ3qPQyMDhWi/rU0nsz4D4wkTOSsql1kZdcb7h0aXDa6XRjBrvS0Z55m znUUvlglarEIPuQESDiIF6WuqHuu6wwfTYhX+1RVCuiJ3ItARCHO94uX6OtnEBLVNvxa nYrUbZ7n75IpERZf7g8yr1v4TgaP9Mc+sXYbrlgTt8zI0wLcjz1dsT1SE2/rXnuzcPPw oMpv6KuYrcpcI8kcvEml9C/JvZP/csF5tCNbAu6p1bGVltqGPvauAJdxVIENoiuSMH8P H5XwCmiKW6NraBnojtGEubADwyjvtXC1BHREqP4HCsRHVL8QrEP9F4OlvGB2eAFUnMUQ PCsQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1772134183; x=1772738983; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=xmGkLxxoqKOr20qSUXIJFsIMVgh1ZbOFwoeb0w7HC5g=; b=WyRulA5Z3alTeYAbkl6tFQIkRwPxCAUsIfoRjUy8pC+W+LMgpAmeDjwlymgzmw0pUg 8kyXovJEl9QSJ58Gjskwz+s2fqBAtQyJiCO/Ugs6j9HtdwVg2Vz5yviOdQAaU80LJEL/ KAxgkBTahKNyc2sFRMVy2ozLpBgPbY0a34opjbBpam0wu9fw8vkZgUNh6HOI3MH+7VgY f8ZCm5RXFLp9izUfzqsafxlXaU7nB7RV3VGAsoavKXyB8X1Er88u0aZ3YjVvh+eAjgO1 nv8DCGEejDU2HSeWd7DCRVwE7iyoTApfiadWq1kFMKUBnFRO6ogC9MRYikwwa+o0hKBI 0Smg== X-Forwarded-Encrypted: i=1; AJvYcCX77ktZSeq2jqOqpFQoPBOKp2XiHVTKrMf52u+/AFkDnDb7CjiK0q3MIWWQqU9Y0/unOSYQJe9qvspWX/A=@vger.kernel.org X-Gm-Message-State: AOJu0YzqPCraaIoIUImX6eW0Oe1bqjjUI+Nqxp47mmfC5hRfQTJrPAZ/ KafJy+OmdIP+M9QTlyUyZFdtb5OpX3AIx83sB2wynuy7PDOlOM9U8QYT X-Gm-Gg: ATEYQzx4hw7OPVQWPH9fe9QUxjsm5H5BuC42VL8cRV5HCuuK5D+EeSeT67ipcAPTBCu TegRgNOgAduJpedQKMLledeIZyJNOXOTlnxVmI0842P9qiccW7qKfuvBqA1LIP4O8qT6Yqq8/5g IHKdQyxEiKdRJ6w4TRHTwUK/lhLrDisM8HwIth2JahH2Mineq4QuNhCUP3vm17urQJ0e2dqrCKz OfGeAhTqxOSqqyViD2NW3D405UtaSi4B5LR6aggPcuaXXsb8EDOB8JBfAGcdQ6g2uVaNuLmBH/W 4A40FRy6SXrZvxuANxvGH1+42uO7exDNOFXLlGkMe+7yzAj1I9jcSD6/PtWNuDyS4oRUDbyOpPF 50LpVp30ggRktdUO+/QtrLdk6b1VE7gz4o+kF1SaVMycAPwWwLjAaxEg6uOrd+eSjLOq/SdEH2q 5tnUAu7hgG6N84TouJl0NotNUfWDCdOvlR X-Received: by 2002:a05:6830:25c5:b0:7d1:9df9:c916 with SMTP id 46e09a7af769-7d591bd0ef0mr210396a34.23.1772134183134; Thu, 26 Feb 2026 11:29:43 -0800 (PST) Received: from localhost ([2a03:2880:10ff:4e::]) by smtp.gmail.com with ESMTPSA id 46e09a7af769-7d5866541fcsm2414575a34.21.2026.02.26.11.29.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 26 Feb 2026 11:29:42 -0800 (PST) From: Joshua Hahn To: Minchan Kim , Sergey Senozhatsky Cc: Johannes Weiner , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@meta.com Subject: [PATCH 2/8] mm/zsmalloc: Make all obj_idx unsigned ints Date: Thu, 26 Feb 2026 11:29:25 -0800 Message-ID: <20260226192936.3190275-3-joshua.hahnjy@gmail.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20260226192936.3190275-1-joshua.hahnjy@gmail.com> References: <20260226192936.3190275-1-joshua.hahnjy@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" object indices, which describe the location of an object in a zspage, cannot be negative. To reflect this most helpers calculate and return these values as unsigned ints. Convert find_alloced_obj, the only function that calculates obj_idx as a signed int, to use an unsigned int as well. No functional change intended. Signed-off-by: Joshua Hahn --- mm/zsmalloc.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 0ca2e94af5ad..7846f31bcc8b 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -1491,10 +1491,11 @@ static void zs_obj_copy(struct size_class *class, u= nsigned long dst, * return handle. */ static unsigned long find_alloced_obj(struct size_class *class, - struct zpdesc *zpdesc, int *obj_idx) + struct zpdesc *zpdesc, + unsigned int *obj_idx) { unsigned int offset; - int index =3D *obj_idx; + unsigned int index =3D *obj_idx; unsigned long handle =3D 0; void *addr =3D kmap_local_zpdesc(zpdesc); =20 @@ -1521,7 +1522,7 @@ static void migrate_zspage(struct zs_pool *pool, stru= ct zspage *src_zspage, { unsigned long used_obj, free_obj; unsigned long handle; - int obj_idx =3D 0; + unsigned int obj_idx =3D 0; struct zpdesc *s_zpdesc =3D get_first_zpdesc(src_zspage); struct size_class *class =3D pool->size_class[src_zspage->class]; =20 --=20 2.47.3 From nobody Tue Apr 7 17:15:08 2026 Received: from mail-oi1-f182.google.com (mail-oi1-f182.google.com [209.85.167.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BBB7C3019D9 for ; Thu, 26 Feb 2026 19:29:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772134187; cv=none; b=BIyaRTwgElDAabDA4hKnOd5JXVorTGbQ/8CSLsiQEbdAfTAcKoypGU2Vh5IMJxeT0oCx30wN23M+7unv5YhrJfgtnDMjiojU3nX1FHuhYCbSGRBImPhNrTY7/vA0Q/P77Jc8ky6uIrHrHH1u3BXRpw2kd31m/OrIXiZBGfuMszE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772134187; c=relaxed/simple; bh=DT19TRdA5oyg3IgCUS/WqcyK8wzAoIu7HCAdavF1MoM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Fy00WD8+A5XeSXGv54MH+e0U+kO6gbU/Dld4mWXY0jVfeDItyZ2oKPiXKAtdt+sSKjfqcSaZ4cCgn+BDNmFBv2v3GqfuglNdxgGYU+dlVqSGmATPONdGXgC8CDolcQb69YEE4F0d7gG0forFyUUeXyL44pdizc9TH3JPwN7334I= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=cYq4izOy; arc=none smtp.client-ip=209.85.167.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="cYq4izOy" Received: by mail-oi1-f182.google.com with SMTP id 5614622812f47-4638e238094so738367b6e.3 for ; Thu, 26 Feb 2026 11:29:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1772134185; x=1772738985; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=gW6bnOf4saMn/cuTsdPZSzO5iSHkVVIBXAzg98Lfp08=; b=cYq4izOywjtK6YxQmfJTyS5s4kfpA6siY6kcbJ7BDt1caWXY6V2O3gEMRlPcrTiPRo HecR5FnpNKk6/2PN0YRXwWoE3e6AJx5RI00gtRLZTaGY+rLvOwjUw4kZWBFwFp6bFncT 4lRpQ5eCch0wqzEjgEMWirMd9NYXEU3MzQAICCYxVR2QXGM9Ib5qNuL/zG7LOXLdca/T 34caMtW2P8zAmTQykmWlNFfna13rh6mlP8ks2gQ2miWugTiztH5GnK/wlSgqF6J02vpE ipfClH38nGwxmqwlk1jhr4Q7eVsMaJH2AReDud4CRJqdoOuszH/KXxz1hKN6a1ATDdTs EsIA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1772134185; x=1772738985; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=gW6bnOf4saMn/cuTsdPZSzO5iSHkVVIBXAzg98Lfp08=; b=jsUoWLWi1Wzk2myG5y4d/A6YxvcDpKY1UjUNJ+3GxJqWMC7+fa/UGzcexHnKgfviK1 Hc5ijCmzcd5NeihC5mqbnd283epIWVWzjjPGY9LUFBSgoe31o5jAyks70VQAQebJfyG+ jxUtY2TQitXPbuB7+XOy3lILnnzif9g6AZHzOjJf8GjPoPva6037JVqa0VKKCBZPkBWE 9+ds2m/e/V7Qav5Vbtn4zSWH4gvIjOcapfWqFVEHgCpPaG7qGcelrnLRvTY4w/EXy/o3 2TWrC7D2XlyzL7s0kqXxcgQ+my5wzbkoakCDZ2C9gICsM3VMAVpoNq3923uXlR4EvvrL yJ4w== X-Forwarded-Encrypted: i=1; AJvYcCWke1LnNy8fd0QYJ8E6f8g30pZpxT3xIcX9E7TFsi3RvdkmCgctAABCerA1lKuTVN029Aa4z/FR1bX+pXc=@vger.kernel.org X-Gm-Message-State: AOJu0YzvD2zs6eMCiBw1fm5YjfoglTcBMEvIiTWmesN+H3VBjzNjTPec 2Z4HQfiXEZizDcktjxuJjn/Q1t45/N2/z0F4ggtQq1Mk300/BioBOoVe X-Gm-Gg: ATEYQzyMmyrxzSv57mF9d9Kur4HXBSegPiZglyBZQhGXy77jfvKo/jqh3OV/VaTab7y XsYGci3JOmvGl6L9+jEOvNlHoZBG4sgiKH9+dOB2RVuxoGIv7dzIhFakCvYBuyWWh7dc9scI7DH v0SJbw5/gmxJ5D302pGtr0QSxExXvKMhgKCVd7FKTK4FWDxQJ6WmftebIEwyQjVWP6ko0Vrs25u zzVl1mClKKPnexoZIUDR9PYLORNUO6jqH8N8ke3HW+fca/ExvB+1u6RGQZZYWlnCXdRyGQY/zAr Vy6L4iidDqfRLX7Bu6GKLzD9Ue85nULPT+fa15VHmb+e5xtO4O6WEqMgSNGp3HlLSbNVPb0EORT QA1/7PUuJsUu5+hL84OOVEdc2zroGLwQC90ATjUDDXlqi7phtFNSBzF3GZd6BmL82w/KiXgWU9/ BWvPZcv8MN4EmBJsS65tYI X-Received: by 2002:a05:6808:1907:b0:455:8400:f078 with SMTP id 5614622812f47-464be9d3740mr159972b6e.25.1772134184607; Thu, 26 Feb 2026 11:29:44 -0800 (PST) Received: from localhost ([2a03:2880:10ff:8::]) by smtp.gmail.com with ESMTPSA id 5614622812f47-464bb3ab494sm475167b6e.8.2026.02.26.11.29.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 26 Feb 2026 11:29:44 -0800 (PST) From: Joshua Hahn To: Minchan Kim , Sergey Senozhatsky Cc: Johannes Weiner , Jens Axboe , Yosry Ahmed , Nhat Pham , Nhat Pham , Chengming Zhou , Andrew Morton , linux-mm@kvack.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@meta.com Subject: [PATCH 3/8] mm/zsmalloc: Introduce objcgs pointer in struct zpdesc Date: Thu, 26 Feb 2026 11:29:26 -0800 Message-ID: <20260226192936.3190275-4-joshua.hahnjy@gmail.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20260226192936.3190275-1-joshua.hahnjy@gmail.com> References: <20260226192936.3190275-1-joshua.hahnjy@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Introduce an array of struct obj_cgroup pointers to zpdesc to keep track of compressed objects' memcg ownership. The 8 bytes required to add the array in struct zpdesc brings its size up from 56 bytes to 64 bytes. However, in the current implementation, struct zpdesc lays on top of struct page[1]. This allows the increased size to remain invisible to the outside, since 64 bytes are used for struct zpdesc anyways. The newly added obj_cgroup array pointer overlays page->memcg_data, which causes problems for functions that try to perform page charging by checking the zeroness of page->memcg_data. To make sure that the backing zpdesc's obj_cgroup ** is not interpreted as a mem_cgroup *, follow SLUB's lead and use the MEMCG_DATA_OBJEXTS bit to tag the pointer. Consumers of zsmalloc that do not perform memcg accounting (i.e. zram) are completely unaffected by this patch, as the array to track the obj_cgroup pointers are only allocated in the zswap path. This patch temporarily increases the memory used by zswap by 8 bytes per zswap_entry, since the obj_cgroup pointer is duplicated in the zpdesc and in zswap_entry. In the following patches, we will redirect memory charging operations to use the zpdesc's obj_cgroup instead, and remove the pointer from zswap_entry. This will leave no net memory usage increase for both zram and zswap. In this patch, allocate / free the objcg pointer array for the zswap path, and handle partial object migration and full zpdesc migration. [1] In the (near) future, struct zpdesc may no longer overlay struct page as we shift towards using memdescs. When this happens, the size increase of struct zpdesc will no longer free. With that said, the difference can be kept minimal. All the changes that are being implemented are currently guarded under CONFIG_MEMCG. We can optionally minimize the impact on zram users by guarding these changes in CONFIG_MEMCG && CONFIG_ZSWAP as well. Suggested-by: Johannes Weiner Signed-off-by: Joshua Hahn --- drivers/block/zram/zram_drv.c | 10 ++--- include/linux/zsmalloc.h | 2 +- mm/zpdesc.h | 25 +++++++++++- mm/zsmalloc.c | 74 +++++++++++++++++++++++++++++------ mm/zswap.c | 2 +- 5 files changed, 93 insertions(+), 20 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 61d3e2c74901..60ee85679730 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -2220,8 +2220,8 @@ static int write_incompressible_page(struct zram *zra= m, struct page *page, * like we do for compressible pages. */ handle =3D zs_malloc(zram->mem_pool, PAGE_SIZE, - GFP_NOIO | __GFP_NOWARN | - __GFP_HIGHMEM | __GFP_MOVABLE, page_to_nid(page)); + GFP_NOIO | __GFP_NOWARN | __GFP_HIGHMEM | + __GFP_MOVABLE, page_to_nid(page), false); if (IS_ERR_VALUE(handle)) return PTR_ERR((void *)handle); =20 @@ -2283,8 +2283,8 @@ static int zram_write_page(struct zram *zram, struct = page *page, u32 index) } =20 handle =3D zs_malloc(zram->mem_pool, comp_len, - GFP_NOIO | __GFP_NOWARN | - __GFP_HIGHMEM | __GFP_MOVABLE, page_to_nid(page)); + GFP_NOIO | __GFP_NOWARN | __GFP_HIGHMEM | + __GFP_MOVABLE, page_to_nid(page), false); if (IS_ERR_VALUE(handle)) { zcomp_stream_put(zstrm); return PTR_ERR((void *)handle); @@ -2514,7 +2514,7 @@ static int recompress_slot(struct zram *zram, u32 ind= ex, struct page *page, handle_new =3D zs_malloc(zram->mem_pool, comp_len_new, GFP_NOIO | __GFP_NOWARN | __GFP_HIGHMEM | __GFP_MOVABLE, - page_to_nid(page)); + page_to_nid(page), false); if (IS_ERR_VALUE(handle_new)) { zcomp_stream_put(zstrm); return PTR_ERR((void *)handle_new); diff --git a/include/linux/zsmalloc.h b/include/linux/zsmalloc.h index 478410c880b1..8ef28b964bb0 100644 --- a/include/linux/zsmalloc.h +++ b/include/linux/zsmalloc.h @@ -28,7 +28,7 @@ struct zs_pool *zs_create_pool(const char *name); void zs_destroy_pool(struct zs_pool *pool); =20 unsigned long zs_malloc(struct zs_pool *pool, size_t size, gfp_t flags, - const int nid); + const int nid, bool objcg); void zs_free(struct zs_pool *pool, unsigned long obj); =20 size_t zs_huge_class_size(struct zs_pool *pool); diff --git a/mm/zpdesc.h b/mm/zpdesc.h index b8258dc78548..d10a73e4a90e 100644 --- a/mm/zpdesc.h +++ b/mm/zpdesc.h @@ -20,10 +20,12 @@ * @zspage: Points to the zspage this zpdesc is a part of. * @first_obj_offset: First object offset in zsmalloc pool. * @_refcount: The number of references to this zpdesc. + * @objcgs: Array of objcgs pointers that the stored objs + * belong to. Overlayed on top of page->memcg_data, and + * will always have first bit set if it is a valid pointer. * * This struct overlays struct page for now. Do not modify without a good - * understanding of the issues. In particular, do not expand into the over= lap - * with memcg_data. + * understanding of the issues. * * Page flags used: * * PG_private identifies the first component page. @@ -47,6 +49,9 @@ struct zpdesc { */ unsigned int first_obj_offset; atomic_t _refcount; +#ifdef CONFIG_MEMCG + unsigned long objcgs; +#endif }; #define ZPDESC_MATCH(pg, zp) \ static_assert(offsetof(struct page, pg) =3D=3D offsetof(struct zpdesc, zp= )) @@ -59,6 +64,9 @@ ZPDESC_MATCH(__folio_index, handle); ZPDESC_MATCH(private, zspage); ZPDESC_MATCH(page_type, first_obj_offset); ZPDESC_MATCH(_refcount, _refcount); +#ifdef CONFIG_MEMCG +ZPDESC_MATCH(memcg_data, objcgs); +#endif #undef ZPDESC_MATCH static_assert(sizeof(struct zpdesc) <=3D sizeof(struct page)); =20 @@ -171,4 +179,17 @@ static inline bool zpdesc_is_locked(struct zpdesc *zpd= esc) { return folio_test_locked(zpdesc_folio(zpdesc)); } + +#ifdef CONFIG_MEMCG +static inline struct obj_cgroup **zpdesc_objcgs(struct zpdesc *zpdesc) +{ + return (struct obj_cgroup **)(zpdesc->objcgs & ~OBJEXTS_FLAGS_MASK); +} + +static inline void zpdesc_set_objcgs(struct zpdesc *zpdesc, + struct obj_cgroup **objcgs) +{ + zpdesc->objcgs =3D (unsigned long)objcgs | MEMCG_DATA_OBJEXTS; +} +#endif #endif diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 7846f31bcc8b..7d56bb700e11 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -39,6 +39,7 @@ #include #include #include +#include #include "zpdesc.h" =20 #define ZSPAGE_MAGIC 0x58 @@ -777,6 +778,10 @@ static void reset_zpdesc(struct zpdesc *zpdesc) ClearPagePrivate(page); zpdesc->zspage =3D NULL; zpdesc->next =3D NULL; +#ifdef CONFIG_MEMCG + kfree(zpdesc_objcgs(zpdesc)); + zpdesc->objcgs =3D 0; +#endif /* PageZsmalloc is sticky until the page is freed to the buddy. */ } =20 @@ -893,6 +898,43 @@ static void init_zspage(struct size_class *class, stru= ct zspage *zspage) set_freeobj(zspage, 0); } =20 +#ifdef CONFIG_MEMCG +static bool alloc_zspage_objcgs(struct size_class *class, gfp_t gfp, + struct zpdesc *zpdescs[]) +{ + /* + * Add 2 to objcgs_per_zpdesc to account for partial objs that may be + * stored at the beginning or end of the zpdesc. + */ + int objcgs_per_zpdesc =3D (PAGE_SIZE / class->size) + 2; + int i; + struct obj_cgroup **objcgs; + + for (i =3D 0; i < class->pages_per_zspage; i++) { + objcgs =3D kcalloc(objcgs_per_zpdesc, sizeof(struct obj_cgroup *), + gfp & ~__GFP_HIGHMEM); + if (!objcgs) { + while (--i >=3D 0) { + kfree(zpdesc_objcgs(zpdescs[i])); + zpdescs[i]->objcgs =3D 0; + } + + return false; + } + + zpdesc_set_objcgs(zpdescs[i], objcgs); + } + + return true; +} +#else +static bool alloc_zspage_objcgs(struct size_class *class, gfp_t gfp, + struct zpdesc *zpdescs[]) +{ + return true; +} +#endif + static void create_page_chain(struct size_class *class, struct zspage *zsp= age, struct zpdesc *zpdescs[]) { @@ -931,7 +973,7 @@ static void create_page_chain(struct size_class *class,= struct zspage *zspage, */ static struct zspage *alloc_zspage(struct zs_pool *pool, struct size_class *class, - gfp_t gfp, const int nid) + gfp_t gfp, const int nid, bool objcg) { int i; struct zpdesc *zpdescs[ZS_MAX_PAGES_PER_ZSPAGE]; @@ -952,24 +994,29 @@ static struct zspage *alloc_zspage(struct zs_pool *po= ol, struct zpdesc *zpdesc; =20 zpdesc =3D alloc_zpdesc(gfp, nid); - if (!zpdesc) { - while (--i >=3D 0) { - zpdesc_dec_zone_page_state(zpdescs[i]); - free_zpdesc(zpdescs[i]); - } - cache_free_zspage(zspage); - return NULL; - } + if (!zpdesc) + goto err; __zpdesc_set_zsmalloc(zpdesc); =20 zpdesc_inc_zone_page_state(zpdesc); zpdescs[i] =3D zpdesc; } =20 + if (objcg && !alloc_zspage_objcgs(class, gfp, zpdescs)) + goto err; + create_page_chain(class, zspage, zpdescs); init_zspage(class, zspage); =20 return zspage; + +err: + while (--i >=3D 0) { + zpdesc_dec_zone_page_state(zpdescs[i]); + free_zpdesc(zpdescs[i]); + } + cache_free_zspage(zspage); + return NULL; } =20 static struct zspage *find_get_zspage(struct size_class *class) @@ -1289,13 +1336,14 @@ static unsigned long obj_malloc(struct zs_pool *poo= l, * @size: size of block to allocate * @gfp: gfp flags when allocating object * @nid: The preferred node id to allocate new zspage (if needed) + * @objcg: Whether the zspage should track per-object memory charging. * * On success, handle to the allocated object is returned, * otherwise an ERR_PTR(). * Allocation requests with size > ZS_MAX_ALLOC_SIZE will fail. */ unsigned long zs_malloc(struct zs_pool *pool, size_t size, gfp_t gfp, - const int nid) + const int nid, bool objcg) { unsigned long handle; struct size_class *class; @@ -1330,7 +1378,7 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t = size, gfp_t gfp, =20 spin_unlock(&class->lock); =20 - zspage =3D alloc_zspage(pool, class, gfp, nid); + zspage =3D alloc_zspage(pool, class, gfp, nid, objcg); if (!zspage) { cache_free_handle(handle); return (unsigned long)ERR_PTR(-ENOMEM); @@ -1672,6 +1720,10 @@ static void replace_sub_page(struct size_class *clas= s, struct zspage *zspage, if (unlikely(ZsHugePage(zspage))) newzpdesc->handle =3D oldzpdesc->handle; __zpdesc_set_movable(newzpdesc); +#ifdef CONFIG_MEMCG + zpdesc_set_objcgs(newzpdesc, zpdesc_objcgs(oldzpdesc)); + oldzpdesc->objcgs =3D 0; +#endif } =20 static bool zs_page_isolate(struct page *page, isolate_mode_t mode) diff --git a/mm/zswap.c b/mm/zswap.c index af3f0fbb0558..dd083110bfa0 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -905,7 +905,7 @@ static bool zswap_compress(struct page *page, struct zs= wap_entry *entry, } =20 gfp =3D GFP_NOWAIT | __GFP_NORETRY | __GFP_HIGHMEM | __GFP_MOVABLE; - handle =3D zs_malloc(pool->zs_pool, dlen, gfp, page_to_nid(page)); + handle =3D zs_malloc(pool->zs_pool, dlen, gfp, page_to_nid(page), true); if (IS_ERR_VALUE(handle)) { alloc_ret =3D PTR_ERR((void *)handle); goto unlock; --=20 2.47.3 From nobody Tue Apr 7 17:15:08 2026 Received: from mail-ot1-f41.google.com (mail-ot1-f41.google.com [209.85.210.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 06FA444D6BA for ; Thu, 26 Feb 2026 19:29:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.41 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772134190; cv=none; b=roIk4rPXRFL787zhIA+9cMqlz1ZuO6EmmYAnfnw6vklpmYxTm4Cy1XeBH43/gHQKCPo5faPyepKcrHYYiaSYeVxeeKPPlra9K4V3KLsXaDqKDywu4fQhS7bl5PrrdZLKeELqjKnvy7fUwr/TLchQ5yEdQTlEmBkXCh3ZGDJJnCk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772134190; c=relaxed/simple; bh=wfhNq3yitHAZ8XpYXoCk88fkKDKZU9yo7nZb3MeoDNk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=UW6ycgj9v+JzpRf68o2WBkQjEYZgBljMrwo0VZPjzbL62rRUGKXHcZxt09L1OZGOWxRRhm6bxizodqJK8m2tZTd6NC9tusHjeq6UUfgiC5QbOgDN7JmUkL+CFr8wXHT9196MQ7qadcY0l4pBA61kSsycrclS0gRfwKPnFEKehdg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=lyU8G3QK; arc=none smtp.client-ip=209.85.210.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="lyU8G3QK" Received: by mail-ot1-f41.google.com with SMTP id 46e09a7af769-7d18d02af68so949581a34.2 for ; Thu, 26 Feb 2026 11:29:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1772134186; x=1772738986; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=MTFgJFJwcADBwm8xdyLgLzJR5Ju+0hDvZcOEI0tPhVw=; b=lyU8G3QKqXOrqG+pgD5g2k1/qy/eledBVYavkCp4YD3z5HAkZoQ0KtqH6PTCesqi0/ rJrjggHz2AGrIm8qig45DSzgskxQqZw5yGr5GF7mVNpxz8xJRcy7cXqfoHjXvNr7/irN PxpsnQYwoWsqheOTDzJ2B38B0oWKzeUT0kM/jqmZkFyai6w8PSslNjxTibWyOWf57ML4 YlimCqA0zprTFOLRt6RBMExF9EfzxOYQT5rCAOJWpE85eKkS3EeFluc4f9zcaYq6usc4 OLWD30HGWwKJ6Qn4YB3I49PlNQAmdSuGSkmpo7C68upaHCMNYkhcTzELRlKp1GsQq3bc HJew== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1772134186; x=1772738986; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=MTFgJFJwcADBwm8xdyLgLzJR5Ju+0hDvZcOEI0tPhVw=; b=pCRtIdo1QmywVEd2mAUQipphSJOPrClFRIMmJWskcYpkGFc2vC0Jl5LOUtbuQmpbQM +GplFW4S90ZlavBlJV1sllI5lu2kfFbGCn1FbTNv7XH4KzpjcdYHqj5KmXGVPCtsi9LL P22vzcUQYL0b96I2dfo/ko9Cv2wHnXeTPMZ8CRYuEJtMt6yFxsdjjxLSgUUSlziGBWZN fem47vt6bpmjuw84Uj89w7U2l5Z9uPHAfPZfJbXah/6xZCaeJNAeBOYJNe/NuzqRMdd2 r7ZaNh5+7x4DSfiIFBi1nQ/+zOmKaPiRMx8PnKNG+aNdNK8bEMev7GGxJM4UmLiHVRVa QMoA== X-Forwarded-Encrypted: i=1; AJvYcCXPQA+gIma0uxLVN8bQBE1yrjCLjxsMg78yyTnju4lMe5lBo+u1AtBIY2JlEjzCvQbVTRN0CqHhc4uafiU=@vger.kernel.org X-Gm-Message-State: AOJu0YxcVgEoxZ369Na9j879v7jl5qhCoPMt5ISpWFftpxaiji+bo8z3 P9PmM3aCgUTrJmDeoWH4tOsXZZuSq+XaMDgTGWls931AKR3cgg1uolvh X-Gm-Gg: ATEYQzwi9nH+Aa+RjJurIUPdYzbIG43gFoNIpUL51+K9bvkyZ5sRu5R5gMeo/QjoSZy 33S6bd3ZbTZy1Gk3KP/VhbSC7Zk4470SBaOEzaM6LJ89Txrr4pxdzq+TfXhZfkjsVetYPTXi7Cv Zx+Hq+63QHJQWLkCNnXYKnrYUZdgf35S6KjIHtszlX7doNikVhItyvQHqbHUOXKxd4dcjYidTPT mZ2vrabIT+fMXfivAfyw+klsTheEbXGcH1IhvqiV1pJFTHlYE14+9VFVuAjFnCC64DLx5nXIBbL bPVvt9oV4J2Pnuijb4TScxf3IezRLfXvsuNRng+kKq4twhkE/NjEUWwcOm2ZR2QmqfhT403rFWZ ANHobDmH4kuFqJrEDYQ0yLVoLbgVzV9VtF0Vh/AaUiimPcI+ON3L5kKro6XpRon/oxDYRquSEp3 pHIsZEz/myPMB6CsPz1jzX3w== X-Received: by 2002:a05:6871:7402:b0:40e:9550:3242 with SMTP id 586e51a60fabf-41626de807cmr252273fac.16.1772134186118; Thu, 26 Feb 2026 11:29:46 -0800 (PST) Received: from localhost ([2a03:2880:10ff:5f::]) by smtp.gmail.com with ESMTPSA id 586e51a60fabf-4160d21196csm2606670fac.12.2026.02.26.11.29.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 26 Feb 2026 11:29:45 -0800 (PST) From: Joshua Hahn To: Minchan Kim , Sergey Senozhatsky Cc: Johannes Weiner , Jens Axboe , Yosry Ahmed , Nhat Pham , Nhat Pham , Chengming Zhou , Andrew Morton , linux-mm@kvack.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@meta.com Subject: [PATCH 4/8] mm/zsmalloc: Store obj_cgroup pointer in zpdesc Date: Thu, 26 Feb 2026 11:29:27 -0800 Message-ID: <20260226192936.3190275-5-joshua.hahnjy@gmail.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20260226192936.3190275-1-joshua.hahnjy@gmail.com> References: <20260226192936.3190275-1-joshua.hahnjy@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" With each zswap-backing zpdesc now having an array of obj_cgroup pointers, plumb the obj_cgroup pointer from the zswap / zram layer down to zsmalloc. Introduce two helper functions zpdesc_obj_cgroup and zpdesc_set_obj_cgroup, which abstract the conversion of an object's zspage idx to its zpdesc idx and the retrieval of the obj_cgroup pointer from the zpdesc. From the zswap path, store the obj_cgroup pointer after compression when writing the object and free when the object gets freed. Also handle the migration of an object across zpdescs. The lifetime and charging of the obj_cgroup is still handled in the zswap layer. Suggested-by: Johannes Weiner Signed-off-by: Joshua Hahn --- drivers/block/zram/zram_drv.c | 7 ++-- include/linux/zsmalloc.h | 3 +- mm/zsmalloc.c | 71 ++++++++++++++++++++++++++++++++++- mm/zswap.c | 6 +-- 4 files changed, 79 insertions(+), 8 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 60ee85679730..209668b14428 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -2231,7 +2231,7 @@ static int write_incompressible_page(struct zram *zra= m, struct page *page, } =20 src =3D kmap_local_page(page); - zs_obj_write(zram->mem_pool, handle, src, PAGE_SIZE); + zs_obj_write(zram->mem_pool, handle, src, PAGE_SIZE, NULL); kunmap_local(src); =20 slot_lock(zram, index); @@ -2296,7 +2296,7 @@ static int zram_write_page(struct zram *zram, struct = page *page, u32 index) return -ENOMEM; } =20 - zs_obj_write(zram->mem_pool, handle, zstrm->buffer, comp_len); + zs_obj_write(zram->mem_pool, handle, zstrm->buffer, comp_len, NULL); zcomp_stream_put(zstrm); =20 slot_lock(zram, index); @@ -2520,7 +2520,8 @@ static int recompress_slot(struct zram *zram, u32 ind= ex, struct page *page, return PTR_ERR((void *)handle_new); } =20 - zs_obj_write(zram->mem_pool, handle_new, zstrm->buffer, comp_len_new); + zs_obj_write(zram->mem_pool, handle_new, zstrm->buffer, + comp_len_new, NULL); zcomp_stream_put(zstrm); =20 slot_free(zram, index); diff --git a/include/linux/zsmalloc.h b/include/linux/zsmalloc.h index 8ef28b964bb0..22f3baa13f24 100644 --- a/include/linux/zsmalloc.h +++ b/include/linux/zsmalloc.h @@ -15,6 +15,7 @@ #define _ZS_MALLOC_H_ =20 #include +#include =20 struct zs_pool_stats { /* How many pages were migrated (freed) */ @@ -48,7 +49,7 @@ void zs_obj_read_sg_begin(struct zs_pool *pool, unsigned = long handle, struct scatterlist *sg, size_t mem_len); void zs_obj_read_sg_end(struct zs_pool *pool, unsigned long handle); void zs_obj_write(struct zs_pool *pool, unsigned long handle, - void *handle_mem, size_t mem_len); + void *handle_mem, size_t mem_len, struct obj_cgroup *objcg); =20 extern const struct movable_operations zsmalloc_mops; =20 diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 7d56bb700e11..e5ae9a0fc78a 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -899,6 +899,41 @@ static void init_zspage(struct size_class *class, stru= ct zspage *zspage) } =20 #ifdef CONFIG_MEMCG +/* idx is indexed per-zspage, not per-zpdesc. */ +static inline struct obj_cgroup *zpdesc_obj_cgroup(struct zpdesc *zpdesc, + unsigned int idx, + int size) +{ + struct obj_cgroup **objcgs =3D zpdesc_objcgs(zpdesc); + unsigned int off =3D offset_in_page(size * idx); + unsigned int zpdesc_idx =3D DIV_ROUND_UP(off, size); + + if (!objcgs) + return NULL; + + return objcgs[zpdesc_idx]; +} + +/* idx is indexed per-zspage, not per-zpdesc. */ +static inline void zpdesc_set_obj_cgroup(struct zpdesc *zpdesc, + unsigned int idx, int size, + struct obj_cgroup *objcg) +{ + struct obj_cgroup **objcgs =3D zpdesc_objcgs(zpdesc); + unsigned int off =3D offset_in_page(size * idx); + unsigned int zpdesc_idx =3D DIV_ROUND_UP(off, size); + + if (!objcgs) + return; + + objcgs[zpdesc_idx] =3D objcg; + if (off + size > PAGE_SIZE) { + /* object spans two pages */ + objcgs =3D zpdesc_objcgs(get_next_zpdesc(zpdesc)); + objcgs[0] =3D objcg; + } +} + static bool alloc_zspage_objcgs(struct size_class *class, gfp_t gfp, struct zpdesc *zpdescs[]) { @@ -927,12 +962,40 @@ static bool alloc_zspage_objcgs(struct size_class *cl= ass, gfp_t gfp, =20 return true; } + +static void migrate_obj_objcg(unsigned long used_obj, unsigned long free_o= bj, + int size) +{ + unsigned int s_obj_idx, d_obj_idx; + struct zpdesc *s_zpdesc, *d_zpdesc; + struct obj_cgroup *objcg; + + obj_to_location(used_obj, &s_zpdesc, &s_obj_idx); + obj_to_location(free_obj, &d_zpdesc, &d_obj_idx); + objcg =3D zpdesc_obj_cgroup(s_zpdesc, s_obj_idx, size); + + zpdesc_set_obj_cgroup(d_zpdesc, d_obj_idx, size, objcg); + zpdesc_set_obj_cgroup(s_zpdesc, s_obj_idx, size, NULL); +} #else +static inline struct obj_cgroup *zpdesc_obj_cgroup(struct zpdesc *zpdesc, + unsigned int offset, + int size) +{ + return NULL; +} + +static inline void zpdesc_set_obj_cgroup(struct zpdesc *zpdesc, + unsigned int offset, int size, + struct obj_cgroup *objcg) {} static bool alloc_zspage_objcgs(struct size_class *class, gfp_t gfp, struct zpdesc *zpdescs[]) { return true; } + +static void migrate_obj_objcg(unsigned long used_obj, unsigned long free_o= bj, + int size) {} #endif =20 static void create_page_chain(struct size_class *class, struct zspage *zsp= age, @@ -1221,7 +1284,7 @@ void zs_obj_read_sg_end(struct zs_pool *pool, unsigne= d long handle) EXPORT_SYMBOL_GPL(zs_obj_read_sg_end); =20 void zs_obj_write(struct zs_pool *pool, unsigned long handle, - void *handle_mem, size_t mem_len) + void *handle_mem, size_t mem_len, struct obj_cgroup *objcg) { struct zspage *zspage; struct zpdesc *zpdesc; @@ -1242,6 +1305,9 @@ void zs_obj_write(struct zs_pool *pool, unsigned long= handle, class =3D zspage_class(pool, zspage); off =3D offset_in_page(class->size * obj_idx); =20 + if (objcg) + zpdesc_set_obj_cgroup(zpdesc, obj_idx, class->size, objcg); + if (!ZsHugePage(zspage)) off +=3D ZS_HANDLE_SIZE; =20 @@ -1415,6 +1481,8 @@ static void obj_free(int class_size, unsigned long ob= j) f_offset =3D offset_in_page(class_size * f_objidx); zspage =3D get_zspage(f_zpdesc); =20 + zpdesc_set_obj_cgroup(f_zpdesc, f_objidx, class_size, NULL); + vaddr =3D kmap_local_zpdesc(f_zpdesc); link =3D (struct link_free *)(vaddr + f_offset); =20 @@ -1587,6 +1655,7 @@ static void migrate_zspage(struct zs_pool *pool, stru= ct zspage *src_zspage, used_obj =3D handle_to_obj(handle); free_obj =3D obj_malloc(pool, dst_zspage, handle); zs_obj_copy(class, free_obj, used_obj); + migrate_obj_objcg(used_obj, free_obj, class->size); obj_idx++; obj_free(class->size, used_obj); =20 diff --git a/mm/zswap.c b/mm/zswap.c index dd083110bfa0..1e2d60f47919 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -851,7 +851,7 @@ static void acomp_ctx_put_unlock(struct crypto_acomp_ct= x *acomp_ctx) } =20 static bool zswap_compress(struct page *page, struct zswap_entry *entry, - struct zswap_pool *pool) + struct zswap_pool *pool, struct obj_cgroup *objcg) { struct crypto_acomp_ctx *acomp_ctx; struct scatterlist input, output; @@ -911,7 +911,7 @@ static bool zswap_compress(struct page *page, struct zs= wap_entry *entry, goto unlock; } =20 - zs_obj_write(pool->zs_pool, handle, dst, dlen); + zs_obj_write(pool->zs_pool, handle, dst, dlen, objcg); entry->handle =3D handle; entry->length =3D dlen; =20 @@ -1413,7 +1413,7 @@ static bool zswap_store_page(struct page *page, return false; } =20 - if (!zswap_compress(page, entry, pool)) + if (!zswap_compress(page, entry, pool, objcg)) goto compress_failed; =20 old =3D xa_store(swap_zswap_tree(page_swpentry), --=20 2.47.3 From nobody Tue Apr 7 17:15:08 2026 Received: from mail-ot1-f44.google.com (mail-ot1-f44.google.com [209.85.210.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8C28344D6AE for ; Thu, 26 Feb 2026 19:29:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.44 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772134190; cv=none; b=qc377HRLjGvXf2aYHHhepeSNqwbVPA6D4sPE7esXwVT2atsVkAH6Y475s398APUBSFfev8Bixd8nWkLNhU0AgG7GBjL5PC4PTWHaG9MER2opFh0SMxhLVLsus5FJt4lr2Hv49EGnRHu20k1boWG5GG8/Ibr5B3JqqUqi0XBjKJo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772134190; c=relaxed/simple; bh=hjkM1RjyarEiiaVEBUIJZvvAWI5jh2AlKz8A4/ebMNA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=LEWsPDE6hrrtYtfwnMdLAzN19ddiM6diwDWpjPLLmL8J24NBsJ1MdVGJCHHGcNmpk/wsnfl5Ooe4DNfK6OtB3ZSOys4EtXl+frVV0i0niJze1T+xrMxR7c1kz9nQTKlN0fYRLIpDhSeA8qLR3HrshzojlQgyxYnqjN8CgqmVzlw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=fhfVJxaw; arc=none smtp.client-ip=209.85.210.44 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="fhfVJxaw" Received: by mail-ot1-f44.google.com with SMTP id 46e09a7af769-7d196a2334fso1039646a34.1 for ; Thu, 26 Feb 2026 11:29:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1772134187; x=1772738987; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=oEfEA70JNXawqJbV01nYPcSs1BoCcq++8AXelGDLMgg=; b=fhfVJxaw0d0Mqlal0ER14frysDjcm22k5AEwRVIGu3ntFcSE5AdWi5RqUWuDJ9c+VK lmd8afrzW2GC3fBcpYipnW35kR5NbMqVolO4NjV3gqhKwejQxIQC7+gQkDAd44LfMEO4 UxmXbOXc/ipsWK3TqrJEhrNfQSKDGNFfxH3QNZMWOruHawrzzZZRYE/IShtPYwUHsJaq IIjFoxloOBqtl6DplaJcIZuY0aJHTo5fVnbjYV2miwrIVpqusDT2Rvh+YxuO2NA3/Qb1 IsjUB/Gt7rbAv+xs7iTAeT4x5TPydDQix+X0zqPSXFB9eGwH8HFvX0+Kc5gbF5/edn/h uILQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1772134187; x=1772738987; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=oEfEA70JNXawqJbV01nYPcSs1BoCcq++8AXelGDLMgg=; b=fJF62dPomJOiR5TKKGTsdWbuLZINWG9H6b1rX+g/+8GOFgD3iMurfi2PBaW8k0fJKi gLFDpXzZgxPgTmNwIuCQbPyO/IZLAh7EE1VIxxCHhPbnwMrytBE4xxXOjI0LPC/eszWV i+qYFGj32RbFNDiZNePRTaK2Zo/tIv1hvAxDbBiRYcvjAZ0RfzLYSuJFHD4NbvJ63QA4 Kf2crKkyFIEqWC+1dAx90f/oJ5vfBSSUqh56S9wzIr1oRfwGwbLaaPdiHCsU5E+ySIma abaAvr4Wnvd0iiOLh03x6VdX0me6Y1b15RFQ14EeliGmn6fPedEsm+ojnU/KV7f3CS6k 3UqQ== X-Forwarded-Encrypted: i=1; AJvYcCU/z0fv5mWNf2fnpRuvrKdYtDV12fAvPsACObqTNrMRbdZVtZyAWV7IiM5DALBZyaiVq+B6YnIPTJ1KfMs=@vger.kernel.org X-Gm-Message-State: AOJu0YxocTVKI3t05hxMqSVZiw+cFBNvtmbMOSC9yWAqkEm2Nmrz/HGN Ilx/NjTuOWdruGtuWcQkfuxP4g2okz9DqBqxZOfj2Li/29xlmbgNSp+F X-Gm-Gg: ATEYQzw3X4n2E+0yUXwRheOOgtod23J9la7NbLuJ+LD3dsneJ/hT9oLR03pKYfAWX6M YqyZ3qYPXv3QqMTIRbuS44yDdAvH3M/rRFQisSZ7sZdvEkkOp3yPVCw+9gRNtl1ktpP0p2L8Bp7 vBE73KtgsVU2gfrxXBEQmp+7PK8ybcHytoLnpv/fuH19nyI6JuX0visZ4Cy+0rGd/GiRh/JZeuO qVzbCXcxFXBZuVx2+pEpzvXyGozja40BLVch3I55XIrAlVfTGll3/P3G2ntf3IRwuaCicuDFhH1 g6SlXnCxay80QXd7PpXzTdu4YY7Z3WXMt+4KXiC0LTOvRu5Lk33Wcy8TUHKJ1jnHC7/TUvb6/dN rV7Iub2oOQKd0E0Sg8jktV6PRxQFdsBM5/pUJLPkFUCeGJwbgZr0IVgPObbIaWyDkCTg26VI0UH WPVOwllYG2mQrsMlZjpZuSTg== X-Received: by 2002:a05:6830:3986:b0:7c7:48b7:640a with SMTP id 46e09a7af769-7d5856d877bmr2357295a34.7.1772134187538; Thu, 26 Feb 2026 11:29:47 -0800 (PST) Received: from localhost ([2a03:2880:10ff:56::]) by smtp.gmail.com with ESMTPSA id 46e09a7af769-7d58644d426sm2404272a34.3.2026.02.26.11.29.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 26 Feb 2026 11:29:47 -0800 (PST) From: Joshua Hahn To: Minchan Kim , Sergey Senozhatsky Cc: Johannes Weiner , Yosry Ahmed , Nhat Pham , Nhat Pham , Chengming Zhou , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@meta.com Subject: [PATCH 5/8] mm/zsmalloc,zswap: Redirect zswap_entry->obcg to zpdesc Date: Thu, 26 Feb 2026 11:29:28 -0800 Message-ID: <20260226192936.3190275-6-joshua.hahnjy@gmail.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20260226192936.3190275-1-joshua.hahnjy@gmail.com> References: <20260226192936.3190275-1-joshua.hahnjy@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Now that obj_cgroups are tracked in zpdesc, redirect the zswap layer to use the pointer stored in the zpdesc and remove the pointer in struct zswap_entry. This offsets the temporary memory increase caused by the duplicate storage of the obj_cgroup pointer and results in a net zero memory footprint change. The lifetime and charging of the obj_cgroup is still handled in the zswap layer. Clean up mem_cgroup_from_entry, which has no more callers. Suggested-by: Johannes Weiner Signed-off-by: Joshua Hahn --- include/linux/zsmalloc.h | 1 + mm/zsmalloc.c | 29 +++++++++++++++++++++++ mm/zswap.c | 51 ++++++++++++++++++---------------------- 3 files changed, 53 insertions(+), 28 deletions(-) diff --git a/include/linux/zsmalloc.h b/include/linux/zsmalloc.h index 22f3baa13f24..05b2b163a427 100644 --- a/include/linux/zsmalloc.h +++ b/include/linux/zsmalloc.h @@ -38,6 +38,7 @@ unsigned long zs_get_total_pages(struct zs_pool *pool); unsigned long zs_compact(struct zs_pool *pool); =20 unsigned int zs_lookup_class_index(struct zs_pool *pool, unsigned int size= ); +struct obj_cgroup *zs_lookup_objcg(struct zs_pool *pool, unsigned long han= dle); =20 void zs_pool_stats(struct zs_pool *pool, struct zs_pool_stats *stats); =20 diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index e5ae9a0fc78a..067215a6ddcc 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -977,6 +977,30 @@ static void migrate_obj_objcg(unsigned long used_obj, = unsigned long free_obj, zpdesc_set_obj_cgroup(d_zpdesc, d_obj_idx, size, objcg); zpdesc_set_obj_cgroup(s_zpdesc, s_obj_idx, size, NULL); } + +struct obj_cgroup *zs_lookup_objcg(struct zs_pool *pool, unsigned long han= dle) +{ + unsigned long obj; + struct zpdesc *zpdesc; + struct zspage *zspage; + struct size_class *class; + struct obj_cgroup *objcg; + unsigned int obj_idx; + + read_lock(&pool->lock); + obj =3D handle_to_obj(handle); + obj_to_location(obj, &zpdesc, &obj_idx); + + zspage =3D get_zspage(zpdesc); + zspage_read_lock(zspage); + read_unlock(&pool->lock); + + class =3D zspage_class(pool, zspage); + objcg =3D zpdesc_obj_cgroup(zpdesc, obj_idx, class->size); + zspage_read_unlock(zspage); + + return objcg; +} #else static inline struct obj_cgroup *zpdesc_obj_cgroup(struct zpdesc *zpdesc, unsigned int offset, @@ -996,6 +1020,11 @@ static bool alloc_zspage_objcgs(struct size_class *cl= ass, gfp_t gfp, =20 static void migrate_obj_objcg(unsigned long used_obj, unsigned long free_o= bj, int size) {} + +struct obj_cgroup *zs_lookup_objcg(struct zs_pool *pool, unsigned long han= dle) +{ + return NULL; +} #endif =20 static void create_page_chain(struct size_class *class, struct zspage *zsp= age, diff --git a/mm/zswap.c b/mm/zswap.c index 1e2d60f47919..55161a5c9d4c 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -193,7 +193,6 @@ struct zswap_entry { bool referenced; struct zswap_pool *pool; unsigned long handle; - struct obj_cgroup *objcg; struct list_head lru; }; =20 @@ -601,25 +600,13 @@ static int zswap_enabled_param_set(const char *val, * lru functions **********************************/ =20 -/* should be called under RCU */ -#ifdef CONFIG_MEMCG -static inline struct mem_cgroup *mem_cgroup_from_entry(struct zswap_entry = *entry) -{ - return entry->objcg ? obj_cgroup_memcg(entry->objcg) : NULL; -} -#else -static inline struct mem_cgroup *mem_cgroup_from_entry(struct zswap_entry = *entry) -{ - return NULL; -} -#endif - static inline int entry_to_nid(struct zswap_entry *entry) { return page_to_nid(virt_to_page(entry)); } =20 -static void zswap_lru_add(struct list_lru *list_lru, struct zswap_entry *e= ntry) +static void zswap_lru_add(struct list_lru *list_lru, struct zswap_entry *e= ntry, + struct obj_cgroup *objcg) { int nid =3D entry_to_nid(entry); struct mem_cgroup *memcg; @@ -636,19 +623,20 @@ static void zswap_lru_add(struct list_lru *list_lru, = struct zswap_entry *entry) * Similar reasoning holds for list_lru_del(). */ rcu_read_lock(); - memcg =3D mem_cgroup_from_entry(entry); + memcg =3D objcg ? obj_cgroup_memcg(objcg) : NULL; /* will always succeed */ list_lru_add(list_lru, &entry->lru, nid, memcg); rcu_read_unlock(); } =20 -static void zswap_lru_del(struct list_lru *list_lru, struct zswap_entry *e= ntry) +static void zswap_lru_del(struct list_lru *list_lru, struct zswap_entry *e= ntry, + struct obj_cgroup *objcg) { int nid =3D entry_to_nid(entry); struct mem_cgroup *memcg; =20 rcu_read_lock(); - memcg =3D mem_cgroup_from_entry(entry); + memcg =3D objcg ? obj_cgroup_memcg(objcg) : NULL; /* will always succeed */ list_lru_del(list_lru, &entry->lru, nid, memcg); rcu_read_unlock(); @@ -716,12 +704,16 @@ static void zswap_entry_cache_free(struct zswap_entry= *entry) */ static void zswap_entry_free(struct zswap_entry *entry) { - zswap_lru_del(&zswap_list_lru, entry); + struct obj_cgroup *objcg =3D zs_lookup_objcg(entry->pool->zs_pool, + entry->handle); + + zswap_lru_del(&zswap_list_lru, entry, objcg); zs_free(entry->pool->zs_pool, entry->handle); zswap_pool_put(entry->pool); - if (entry->objcg) { - obj_cgroup_uncharge_zswap(entry->objcg, entry->length); - obj_cgroup_put(entry->objcg); + + if (objcg) { + obj_cgroup_uncharge_zswap(objcg, entry->length); + obj_cgroup_put(objcg); } if (entry->length =3D=3D PAGE_SIZE) atomic_long_dec(&zswap_stored_incompressible_pages); @@ -994,6 +986,7 @@ static int zswap_writeback_entry(struct zswap_entry *en= try, struct mempolicy *mpol; bool folio_was_allocated; struct swap_info_struct *si; + struct obj_cgroup *objcg; int ret =3D 0; =20 /* try to allocate swap cache folio */ @@ -1043,8 +1036,9 @@ static int zswap_writeback_entry(struct zswap_entry *= entry, xa_erase(tree, offset); =20 count_vm_event(ZSWPWB); - if (entry->objcg) - count_objcg_events(entry->objcg, ZSWPWB, 1); + objcg =3D zs_lookup_objcg(entry->pool->zs_pool, entry->handle); + if (objcg) + count_objcg_events(objcg, ZSWPWB, 1); =20 zswap_entry_free(entry); =20 @@ -1463,11 +1457,10 @@ static bool zswap_store_page(struct page *page, */ entry->pool =3D pool; entry->swpentry =3D page_swpentry; - entry->objcg =3D objcg; entry->referenced =3D true; if (entry->length) { INIT_LIST_HEAD(&entry->lru); - zswap_lru_add(&zswap_list_lru, entry); + zswap_lru_add(&zswap_list_lru, entry, objcg); } =20 return true; @@ -1592,6 +1585,7 @@ int zswap_load(struct folio *folio) bool swapcache =3D folio_test_swapcache(folio); struct xarray *tree =3D swap_zswap_tree(swp); struct zswap_entry *entry; + struct obj_cgroup *objcg; =20 VM_WARN_ON_ONCE(!folio_test_locked(folio)); =20 @@ -1620,8 +1614,9 @@ int zswap_load(struct folio *folio) folio_mark_uptodate(folio); =20 count_vm_event(ZSWPIN); - if (entry->objcg) - count_objcg_events(entry->objcg, ZSWPIN, 1); + objcg =3D zs_lookup_objcg(entry->pool->zs_pool, entry->handle); + if (objcg) + count_objcg_events(objcg, ZSWPIN, 1); =20 /* * When reading into the swapcache, invalidate our entry. The --=20 2.47.3 From nobody Tue Apr 7 17:15:08 2026 Received: from mail-oi1-f173.google.com (mail-oi1-f173.google.com [209.85.167.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1421A44DB89 for ; Thu, 26 Feb 2026 19:29:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772134196; cv=none; b=BKeY7qGWL2kzkffXwFyJbIq224ZtZHBwOcL2AE6GVBhWw1+AioFEsVjd8YGOwp/OUoMAsVzK+jwCYj32eVvicKrBr8joOAMrS5oelBGLn1YnSz8XYu+IsfbMGcRrnAyNAMmek5ZSAuLqvwLQbBZHIKlKJ6BMTpAE9c1Wq2mJuk8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772134196; c=relaxed/simple; bh=yJ8hDpck92sFoRQLA5ASgE0Grx5gw3Jrij4YOy9XvEA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=OtlA/t2Ursd6pVlNbu+3GTx0yZQQqSbKclPx3An2aTxsvSN8LP+73fkKCdmITCZtWZwm8NUasGBf7bafP7ghJTrTq8IpazJ2kUxtDx82kgAt723Xu5I6b2o3KrGQjBxogtv4upj0fq4uOr5jqS5ktKF+e9cwjaQtX0HFxx0oW8c= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=WL5z6dpN; arc=none smtp.client-ip=209.85.167.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="WL5z6dpN" Received: by mail-oi1-f173.google.com with SMTP id 5614622812f47-4648448e387so986045b6e.1 for ; Thu, 26 Feb 2026 11:29:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1772134190; x=1772738990; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=BQnJpZ9tlMxlAvSH4/u9efMTjBoGOkCwurLY+eANTp4=; b=WL5z6dpNtNOKuc28cuEB/g1Uxf7EYrfOUmTcSj7a3S5i9agIvvk0n92/iebzw/vqzR L0wZebi7eNeJPzDqLiNRYP2TTIyT/50ZQiDawAkc3+N/NBUQbOtYlHXaNZoWjbomcfIu GhsONQwAt0L5dEP2iTIbzixk/XfeXPXO8yLGQjy5vRbyZGG5lvIS/s1ziDbtB177eGRd +ODjomDPIB78K81ptkMsqhnPSDMJRKBbMvzD/Ndy6CkJQsiWrY3i7D0R7l8/ClVqEkS9 Z+7jXQsLigvOpbtEkP8LPceFcWS8d/kCtZBJ/Q/LJGHd4Q1Fx6wXxGvKlW1LakXqHgcf WwAw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1772134190; x=1772738990; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=BQnJpZ9tlMxlAvSH4/u9efMTjBoGOkCwurLY+eANTp4=; b=ekRcxIlNjHoJLNuiOxrNgI50rvTYOa/mvORGkB1otmtBYR3l20ergEI5FPUtFnoRxz whsTJbxk6ifyze3YXpib/Y20sZH/UGGXeG3aR++A10yaojyKDwDPgYiyQBPvvPDNZ+M/ rP7fsdC9aH1GNu3yGAPjl2/NBGZGqjCgMkdvkR2shg7h/hn8Wx6s+8utpEXGUlN61SkL Nwnx8+kxRa6D4trgqIuPvjh1tKhtZIkRin97AZNYN4GunS4VqTJiX5OPpMtNgIoIXaqI 9Rzz29P5vNDmumr0NVaH6YvBw81JEJu8p7lxbykoDiUBtgS5Gz6eMSIzMBt+y5WY775m Je0g== X-Forwarded-Encrypted: i=1; AJvYcCUwJn+hQ4n4a5FrpgDl1Kdau2XT+qaTMcU815WxRhiyVT5VpNLiHjc2e0Y+br6Y08n/ctkqWQ0EPEnHHC8=@vger.kernel.org X-Gm-Message-State: AOJu0YzbdXPBdK7Re4LqaN6Nw2pDmdt0aX/JodvmKTQrlOK2drBbbGHB LaxnQmpVgmjy80LtZP+DPMPQTtD6oqOUFQZnhsLa4tMBp9jO89FSdW0k X-Gm-Gg: ATEYQzwHNfwNrsaqQ+Vi1Sc9xxbi6X+dx5uAMctd1lI/7VfZUqnwamGo9RuiSgkEzBS E6L/ZEHU08EZcZScOaY9pJpBWpBXy1+4BIDii+i8wyJD+3d+z9YizP7MOgj5pzrbDfIfxAYnjXZ zlgw/ItB/yo8ixvO/Y8MXfn/Rg8+LjCm8lKCzI5ZYj6VocMvqG2CUu+YlTxu4hWrJrz7N2SnJVV ZvH5VJebKQxtnqkr4mQitYw5CTZbpki3nnhwKRLF+pb5bOi3nKVrBUI7mXAb9ddVAXztyK7EV9/ ikiMsXVZL+UMRyt+jz/+p/P1LrZPZTkVA+MLQ9Pn2wNnhj9RFtsk8oKdtvMCVeMZ0ExWcsmJCux +d4+GIneTEzzSnL8+I6rvHWp59bkkQzfb8VUYr7RO4ASXfJpMX8JvsEYVwIiMpYMO0aKlHaJPNs 1Mx2GsHdM5HOI2PelK8YPmhA== X-Received: by 2002:a05:6808:c1e2:b0:45c:8c23:12b3 with SMTP id 5614622812f47-464bec872bdmr142745b6e.61.1772134189833; Thu, 26 Feb 2026 11:29:49 -0800 (PST) Received: from localhost ([2a03:2880:10ff:4f::]) by smtp.gmail.com with ESMTPSA id 5614622812f47-464bb0991f5sm498612b6e.0.2026.02.26.11.29.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 26 Feb 2026 11:29:48 -0800 (PST) From: Joshua Hahn To: Minchan Kim , Sergey Senozhatsky Cc: Johannes Weiner , Yosry Ahmed , Nhat Pham , Nhat Pham , Chengming Zhou , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@meta.com Subject: [PATCH 6/8] mm/zsmalloc, zswap: Handle objcg charging and lifetime in zsmalloc Date: Thu, 26 Feb 2026 11:29:29 -0800 Message-ID: <20260226192936.3190275-7-joshua.hahnjy@gmail.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20260226192936.3190275-1-joshua.hahnjy@gmail.com> References: <20260226192936.3190275-1-joshua.hahnjy@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Now that zswap_entries do not directly track obj_cgroups of the entries, handle the lifetime management and charging of these entries into the zsmalloc layer. One functional change is that zswap entries are now no longer accounted by the size of the compressed object, but by the size of the size_class slot they occupy. This brings the charging one step closer to an accurate representation of the memory consumed in the zpdesc; even if a compressed object doesn't consume the entirety of a obj slot, the hole it creates between the objects is dead space the obj is accountable for. Thus, account the memory each object makes unusable, not the amount of memory each object takes up. Signed-off-by: Joshua Hahn --- include/linux/memcontrol.h | 10 ------- mm/memcontrol.c | 51 ---------------------------------- mm/zsmalloc.c | 57 ++++++++++++++++++++++++++++++++++++-- mm/zswap.c | 8 ------ 4 files changed, 55 insertions(+), 71 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index b6c82c8f73e1..dd4278b1ca35 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1824,22 +1824,12 @@ static inline bool memcg_is_dying(struct mem_cgroup= *memcg) =20 #if defined(CONFIG_MEMCG) && defined(CONFIG_ZSWAP) bool obj_cgroup_may_zswap(struct obj_cgroup *objcg); -void obj_cgroup_charge_zswap(struct obj_cgroup *objcg, size_t size); -void obj_cgroup_uncharge_zswap(struct obj_cgroup *objcg, size_t size); bool mem_cgroup_zswap_writeback_enabled(struct mem_cgroup *memcg); #else static inline bool obj_cgroup_may_zswap(struct obj_cgroup *objcg) { return true; } -static inline void obj_cgroup_charge_zswap(struct obj_cgroup *objcg, - size_t size) -{ -} -static inline void obj_cgroup_uncharge_zswap(struct obj_cgroup *objcg, - size_t size) -{ -} static inline bool mem_cgroup_zswap_writeback_enabled(struct mem_cgroup *m= emcg) { /* if zswap is disabled, do not block pages going to the swapping device = */ diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 007413a53b45..3432e1afc037 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -5433,57 +5433,6 @@ bool obj_cgroup_may_zswap(struct obj_cgroup *objcg) return ret; } =20 -/** - * obj_cgroup_charge_zswap - charge compression backend memory - * @objcg: the object cgroup - * @size: size of compressed object - * - * This forces the charge after obj_cgroup_may_zswap() allowed - * compression and storage in zswap for this cgroup to go ahead. - */ -void obj_cgroup_charge_zswap(struct obj_cgroup *objcg, size_t size) -{ - struct mem_cgroup *memcg; - - if (!cgroup_subsys_on_dfl(memory_cgrp_subsys)) - return; - - VM_WARN_ON_ONCE(!(current->flags & PF_MEMALLOC)); - - /* PF_MEMALLOC context, charging must succeed */ - if (obj_cgroup_charge(objcg, GFP_KERNEL, size)) - VM_WARN_ON_ONCE(1); - - rcu_read_lock(); - memcg =3D obj_cgroup_memcg(objcg); - mod_memcg_state(memcg, MEMCG_ZSWAP_B, size); - mod_memcg_state(memcg, MEMCG_ZSWAPPED, 1); - rcu_read_unlock(); -} - -/** - * obj_cgroup_uncharge_zswap - uncharge compression backend memory - * @objcg: the object cgroup - * @size: size of compressed object - * - * Uncharges zswap memory on page in. - */ -void obj_cgroup_uncharge_zswap(struct obj_cgroup *objcg, size_t size) -{ - struct mem_cgroup *memcg; - - if (!cgroup_subsys_on_dfl(memory_cgrp_subsys)) - return; - - obj_cgroup_uncharge(objcg, size); - - rcu_read_lock(); - memcg =3D obj_cgroup_memcg(objcg); - mod_memcg_state(memcg, MEMCG_ZSWAP_B, -size); - mod_memcg_state(memcg, MEMCG_ZSWAPPED, -1); - rcu_read_unlock(); -} - bool mem_cgroup_zswap_writeback_enabled(struct mem_cgroup *memcg) { /* if zswap is disabled, do not block pages going to the swapping device = */ diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 067215a6ddcc..88c7cd399261 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -963,6 +963,44 @@ static bool alloc_zspage_objcgs(struct size_class *cla= ss, gfp_t gfp, return true; } =20 +static void zs_charge_objcg(struct zpdesc *zpdesc, struct obj_cgroup *objc= g, + int size, unsigned long offset) +{ + struct mem_cgroup *memcg; + + if (!cgroup_subsys_on_dfl(memory_cgrp_subsys)) + return; + + VM_WARN_ON_ONCE(!(current->flags & PF_MEMALLOC)); + + /* PF_MEMALLOC context, charging must succeed */ + if (obj_cgroup_charge(objcg, GFP_KERNEL, size)) + VM_WARN_ON_ONCE(1); + + rcu_read_lock(); + memcg =3D obj_cgroup_memcg(objcg); + mod_memcg_state(memcg, MEMCG_ZSWAP_B, size); + mod_memcg_state(memcg, MEMCG_ZSWAPPED, 1); + rcu_read_unlock(); +} + +static void zs_uncharge_objcg(struct zpdesc *zpdesc, struct obj_cgroup *ob= jcg, + int size, unsigned long offset) +{ + struct mem_cgroup *memcg; + + if (!cgroup_subsys_on_dfl(memory_cgrp_subsys)) + return; + + obj_cgroup_uncharge(objcg, size); + + rcu_read_lock(); + memcg =3D obj_cgroup_memcg(objcg); + mod_memcg_state(memcg, MEMCG_ZSWAP_B, -size); + mod_memcg_state(memcg, MEMCG_ZSWAPPED, -1); + rcu_read_unlock(); +} + static void migrate_obj_objcg(unsigned long used_obj, unsigned long free_o= bj, int size) { @@ -1018,6 +1056,12 @@ static bool alloc_zspage_objcgs(struct size_class *c= lass, gfp_t gfp, return true; } =20 +static void zs_charge_objcg(struct zpdesc *zpdesc, struct obj_cgroup *objc= g, + int size, unsigned long offset) {} + +static void zs_uncharge_objcg(struct zpdesc *zpdesc, struct obj_cgroup *ob= jcg, + int size, unsigned long offset) {} + static void migrate_obj_objcg(unsigned long used_obj, unsigned long free_o= bj, int size) {} =20 @@ -1334,8 +1378,11 @@ void zs_obj_write(struct zs_pool *pool, unsigned lon= g handle, class =3D zspage_class(pool, zspage); off =3D offset_in_page(class->size * obj_idx); =20 - if (objcg) + if (objcg) { + obj_cgroup_get(objcg); + zs_charge_objcg(zpdesc, objcg, class->size, off); zpdesc_set_obj_cgroup(zpdesc, obj_idx, class->size, objcg); + } =20 if (!ZsHugePage(zspage)) off +=3D ZS_HANDLE_SIZE; @@ -1501,6 +1548,7 @@ static void obj_free(int class_size, unsigned long ob= j) struct link_free *link; struct zspage *zspage; struct zpdesc *f_zpdesc; + struct obj_cgroup *objcg; unsigned long f_offset; unsigned int f_objidx; void *vaddr; @@ -1510,7 +1558,12 @@ static void obj_free(int class_size, unsigned long o= bj) f_offset =3D offset_in_page(class_size * f_objidx); zspage =3D get_zspage(f_zpdesc); =20 - zpdesc_set_obj_cgroup(f_zpdesc, f_objidx, class_size, NULL); + objcg =3D zpdesc_obj_cgroup(f_zpdesc, f_objidx, class_size); + if (objcg) { + zs_uncharge_objcg(f_zpdesc, objcg, class_size, f_offset); + obj_cgroup_put(objcg); + zpdesc_set_obj_cgroup(f_zpdesc, f_objidx, class_size, NULL); + } =20 vaddr =3D kmap_local_zpdesc(f_zpdesc); link =3D (struct link_free *)(vaddr + f_offset); diff --git a/mm/zswap.c b/mm/zswap.c index 55161a5c9d4c..77d3c6516ed3 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -711,10 +711,6 @@ static void zswap_entry_free(struct zswap_entry *entry) zs_free(entry->pool->zs_pool, entry->handle); zswap_pool_put(entry->pool); =20 - if (objcg) { - obj_cgroup_uncharge_zswap(objcg, entry->length); - obj_cgroup_put(objcg); - } if (entry->length =3D=3D PAGE_SIZE) atomic_long_dec(&zswap_stored_incompressible_pages); zswap_entry_cache_free(entry); @@ -1437,10 +1433,6 @@ static bool zswap_store_page(struct page *page, * when the entry is removed from the tree. */ zswap_pool_get(pool); - if (objcg) { - obj_cgroup_get(objcg); - obj_cgroup_charge_zswap(objcg, entry->length); - } atomic_long_inc(&zswap_stored_pages); if (entry->length =3D=3D PAGE_SIZE) atomic_long_inc(&zswap_stored_incompressible_pages); --=20 2.47.3 From nobody Tue Apr 7 17:15:08 2026 Received: from mail-oi1-f173.google.com (mail-oi1-f173.google.com [209.85.167.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7512E44E023 for ; Thu, 26 Feb 2026 19:29:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772134199; cv=none; b=E4pJ1CAylJJBHrD2yXrfFRUW17ST+BURu1B+mOpBCAwOA8zqgW7fVYXxOuDuMI/KRo5VxI4r5HmkUbXM6WHJGwZAIkUxyGTAHKhiWfhF7+DX/DAc//3Uiqcb5uH95fxw8JoH4hJgHvbtDDydRDnAP7qhwqJcFlaQvadi4t1ISZ4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772134199; c=relaxed/simple; bh=I/6VfJFxIb4g0o8KWA7c8Glu4mFHW8x/F43F+Sqs7Rw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=KvD2DhPUgiNMXf6lyT2b9mKoRHD/Uc7OU/T5MU567//Drw2SCptKxSgo3ltRXGgNjKqVAKkjmQDRSCgA8hXR8SgmXdCmTvjWlO6h5HLvpcu+sh50CB2of4fnRbjniuDxHsOrmMKpbwUqfuZSCagUm+4JZ4+te1sMJKwffFIUgZM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Le1BRfQ7; arc=none smtp.client-ip=209.85.167.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Le1BRfQ7" Received: by mail-oi1-f173.google.com with SMTP id 5614622812f47-4638e238094so738476b6e.3 for ; Thu, 26 Feb 2026 11:29:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1772134192; x=1772738992; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=qPksRk4LAnsoFLQrd4UcyaPDU9XJzdmOlJT+foKyitQ=; b=Le1BRfQ7f7OzIrthjeVdUlhU6gRFJQWUrBHuMVsnV6j3mycFZ6REiUIw/BPMx1SG4/ 7EjOfvPLKpg/MzcL9CAgQ37r44Hk5Ku4mW60OwPdm+1J/0fzw7ygSI2xAlKy3jim+vL1 0alld4trryC6EEeXEr5JDHhoPSN6JF+7EmqzryDjiWEjqt6Zg+V+3qy1u+oVG5/lAZtq 7npqNj3/UR+LE0t+GMpRzExQFR/cm7tMcpRzkPbesfj2ZoICgWg8gq0clVEvgrAR8BFC AR9LOWXeMK4mgM8NyZy1a1LJgZRKAqYjtI2KussTEa1s7YdZtXimjR5ian1ziJKQkV5Q 3c9A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1772134192; x=1772738992; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=qPksRk4LAnsoFLQrd4UcyaPDU9XJzdmOlJT+foKyitQ=; b=YikoW888tyZBGQY3U2fRIWozck7fc/a4piYyunZakMfzut+N+3SvShBCzx1jyPOQrk o4Gng5SZy1XOWY7mxsjwRb8pAK0HJ/dRiz1jT0Ry5zpsxNNkU3ZqxFk+/Kz1f2kLUAcv rb9DpC3rU7Pztd7nqwF74FsVppxc3TsR+k43ZxBLyuvwP27zD8olBWfGkGofstSiq7GZ 6TDKiF2Gc5qwgYp30689QvG/7dXETgVo8aLr+5qJL6P3iLytAzTgiUTVmxzYO6cIpUFK WrV7+L40oTVDhXUeNk1P5dOgroC6Bz3f9iouOKC8lV8t2cO5by4ZPzBHjps6a78X6qFJ Zr5w== X-Forwarded-Encrypted: i=1; AJvYcCV6p77l0YSwXVWThDukg2mHV3uhHo1AJSA1/Xod+Lj30tJhltMIyPVApzHpMnp3gy7m+vG0nor0FXxH4Yc=@vger.kernel.org X-Gm-Message-State: AOJu0YzvVLkyU5jmVQ3BvLUeuDjqVP2RQ4tphdPdongGz7lsKqmfzabm QUvGZKYDoICh9Nuyaj6s8YiMnl/0QUQtArhsrqMRbsCSclRUogXEwLQF X-Gm-Gg: ATEYQzzjzTFFQeklhCyH9i7Uaqeq7Tdvs11JUQCzJNxBu0ff1OvQdxzTAE3D4l7BHtm PFN4dfLuV482y6Qb+nDwY9Zz14bcg85Lntjn1gLR3cL8yofaP/mYZCE3qJllNR+XrQtfqxRY/os bIqBWvLG+Y1BlBUyDTg2DUZF26c2tZfap/E5omMQ359F1eO1SWOrfBdnBMheJ7SG1Xkap9zUNQD e3RAethgLK8rAhU2lj/k5W7cS7zHIXS6L/r86KT+8hNr3Ml3uCSsrImezwtTW/1g1/8K0LD/i/9 A3hNt8nPqCe3tuzmeRKRRD3Y603mt5gRcqG4kjLAPBR2yHz1Of2ZYTrvv+fGoAlM4qXCmWpgN9D EXPOQrIPryJWgKKy51E5qlTjxOOBTP+TBCo4VD9hcoOOQiISbi1SMWro5qptwVohlBRfCkbIin8 PXeyKSW7zpLL+GHx40JtMyGA== X-Received: by 2002:a05:6808:4707:b0:45e:f0ae:bc0f with SMTP id 5614622812f47-464be9cbef3mr147302b6e.23.1772134192235; Thu, 26 Feb 2026 11:29:52 -0800 (PST) Received: from localhost ([2a03:2880:10ff:59::]) by smtp.gmail.com with ESMTPSA id 586e51a60fabf-4160cf9b24dsm2713097fac.7.2026.02.26.11.29.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 26 Feb 2026 11:29:50 -0800 (PST) From: Joshua Hahn To: Minchan Kim , Sergey Senozhatsky Cc: Johannes Weiner , Yosry Ahmed , Nhat Pham , Nhat Pham , Chengming Zhou , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@meta.com Subject: [PATCH 7/8] mm/memcontrol: Track MEMCG_ZSWAPPED in bytes Date: Thu, 26 Feb 2026 11:29:30 -0800 Message-ID: <20260226192936.3190275-8-joshua.hahnjy@gmail.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20260226192936.3190275-1-joshua.hahnjy@gmail.com> References: <20260226192936.3190275-1-joshua.hahnjy@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Zswap compresses and uncompresses in PAGE_SIZE units, which simplifies the accounting for how much memory it has compressed. However, when a compressed object is stored at the boundary of two zspages, accounting at PAGE_SIZE units makes it difficult to fractionally charge each backing zspage with the ratio of memory it backs for the compressed object. To make sub-PAGE_SIZE granularity charging possible for MEMCG_ZSWAPPED, track the value in bytes and adjust its accounting accordingly. No functional changes intended. Signed-off-by: Joshua Hahn --- include/linux/memcontrol.h | 2 +- mm/memcontrol.c | 5 +++-- mm/zsmalloc.c | 4 ++-- mm/zswap.c | 6 ++++-- 4 files changed, 10 insertions(+), 7 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index dd4278b1ca35..d3952c918fd4 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -38,7 +38,7 @@ enum memcg_stat_item { MEMCG_VMALLOC, MEMCG_KMEM, MEMCG_ZSWAP_B, - MEMCG_ZSWAPPED, + MEMCG_ZSWAPPED_B, MEMCG_NR_STAT, }; =20 diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 3432e1afc037..b662902d4e03 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -340,7 +340,7 @@ static const unsigned int memcg_stat_items[] =3D { MEMCG_VMALLOC, MEMCG_KMEM, MEMCG_ZSWAP_B, - MEMCG_ZSWAPPED, + MEMCG_ZSWAPPED_B, }; =20 #define NR_MEMCG_NODE_STAT_ITEMS ARRAY_SIZE(memcg_node_stat_items) @@ -1345,7 +1345,7 @@ static const struct memory_stat memory_stats[] =3D { { "shmem", NR_SHMEM }, #ifdef CONFIG_ZSWAP { "zswap", MEMCG_ZSWAP_B }, - { "zswapped", MEMCG_ZSWAPPED }, + { "zswapped", MEMCG_ZSWAPPED_B }, #endif { "file_mapped", NR_FILE_MAPPED }, { "file_dirty", NR_FILE_DIRTY }, @@ -1393,6 +1393,7 @@ static int memcg_page_state_unit(int item) switch (item) { case MEMCG_PERCPU_B: case MEMCG_ZSWAP_B: + case MEMCG_ZSWAPPED_B: case NR_SLAB_RECLAIMABLE_B: case NR_SLAB_UNRECLAIMABLE_B: return 1; diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 88c7cd399261..6794927c60fb 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -980,7 +980,7 @@ static void zs_charge_objcg(struct zpdesc *zpdesc, stru= ct obj_cgroup *objcg, rcu_read_lock(); memcg =3D obj_cgroup_memcg(objcg); mod_memcg_state(memcg, MEMCG_ZSWAP_B, size); - mod_memcg_state(memcg, MEMCG_ZSWAPPED, 1); + mod_memcg_state(memcg, MEMCG_ZSWAPPED_B, 1); rcu_read_unlock(); } =20 @@ -997,7 +997,7 @@ static void zs_uncharge_objcg(struct zpdesc *zpdesc, st= ruct obj_cgroup *objcg, rcu_read_lock(); memcg =3D obj_cgroup_memcg(objcg); mod_memcg_state(memcg, MEMCG_ZSWAP_B, -size); - mod_memcg_state(memcg, MEMCG_ZSWAPPED, -1); + mod_memcg_state(memcg, MEMCG_ZSWAPPED_B, -1); rcu_read_unlock(); } =20 diff --git a/mm/zswap.c b/mm/zswap.c index 77d3c6516ed3..97f38d0afa86 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -1214,8 +1214,10 @@ static unsigned long zswap_shrinker_count(struct shr= inker *shrinker, */ if (!mem_cgroup_disabled()) { mem_cgroup_flush_stats(memcg); - nr_backing =3D memcg_page_state(memcg, MEMCG_ZSWAP_B) >> PAGE_SHIFT; - nr_stored =3D memcg_page_state(memcg, MEMCG_ZSWAPPED); + nr_backing =3D memcg_page_state(memcg, MEMCG_ZSWAP_B); + nr_backing >>=3D PAGE_SHIFT; + nr_stored =3D memcg_page_state(memcg, MEMCG_ZSWAPPED_B); + nr_stored >>=3D PAGE_SHIFT; } else { nr_backing =3D zswap_total_pages(); nr_stored =3D atomic_long_read(&zswap_stored_pages); --=20 2.47.3 From nobody Tue Apr 7 17:15:08 2026 Received: from mail-ot1-f53.google.com (mail-ot1-f53.google.com [209.85.210.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E5D6A44D035 for ; Thu, 26 Feb 2026 19:29:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.53 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772134201; cv=none; b=I9oQ1lZghQf8uKSE4H4ehsEWS+arDnrPHuw9muHp262UahhO0oV6GDkYgnjKOPZwms7hGzcI5YNivXRVbqhJAOKqaF24CBoXALV81mmH850tipEUIgt3LVrwhblff6Qg7zUbvVcjXpD1crTGrcZDheD4wYx72w+gFYxRXUfTkao= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772134201; c=relaxed/simple; bh=edHifQhuD39SytJ2pdcEcAu31cCV6BrBABjFy6qdldM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=gSDf2baBg9jp4ddUrP2PwxubdbKSlpSgO6Wny4nDgQlDV+1nkHyVs/3sUYMOUsqwo2iTUIvb3PloskDnPw5KaVhlhmCHuR7NsS2TWMefbnMDdSgC/VCtAXJOVVkjjNt+mlEv262BEUsw9B+brU52GbJfFTyyB8qE28NnL1eInmQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=XhwA1atj; arc=none smtp.client-ip=209.85.210.53 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="XhwA1atj" Received: by mail-ot1-f53.google.com with SMTP id 46e09a7af769-7d1890f5cafso492695a34.1 for ; Thu, 26 Feb 2026 11:29:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1772134194; x=1772738994; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=DB+jMUpA86bxO+NhQL7ENpCNqUG0HUtrRjDNh//YUrM=; b=XhwA1atjD5hDDHc1pivvDyZRBKzL1yjyBZIUMS6ZVJ2pwpsrN6h5tDl9SW1dtl+Mlx zAXYX1j+slujQgZ8cVtRA4dnkTpyZn2hIa5KwbVqoB0z6zyccJFV1KxrSh1dIMC6c+Mk +eIamg5qCaXnijzUa/67kVrEWHncBsoRfgZKs3Wb2Gz5k55ubRL3pkwAx2BzUturpQxr LGHduzsAOMFnF/76rerVqPjAfbIyFNJ0sLsw+d0WpDjP/ijKMRSNx109wyA1PLP+uq/o Jd9Io6Y1/v+5hF9FuY5219XfiTTCM89/dfcZVEYjoP5Am2mf1Hc7G+cGtgVcnqea0slx 7A5A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1772134194; x=1772738994; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=DB+jMUpA86bxO+NhQL7ENpCNqUG0HUtrRjDNh//YUrM=; b=oBAJYA9ADCsfdeTsTsvI9pNHUbj3G9aicUxQXTvz4Jej6xjJ0MqiDXEHt9oCwDS6np oLxNHUPzjzOPmjOhaPiBgT2Sm/YZYOhGdGVcaJfXC00MFt4p7qVVysXsHPSO5FKSrT6X k5djp93lZJmqa6xZ0wpq4+2Vc8HJLhs9Z3WYJptQYgr1ZTbdv20jbQc+/R9JjqHH8SCT AdfZ0GP+lcMLNSuNtYSRuTcyZv17/b88aNeBNWr5bdwebVGDHGo3sxWVeC3qK0hLVMXT jMPL8V0u2RZ5vmm56e6QO2Uv2uxlQctFq3Mx8sAwsz2DoKgn4ikSIf5fF0gAsc3mLfMs v2cw== X-Forwarded-Encrypted: i=1; AJvYcCWNgryvp+/4/I9RkCKjl0FCGqykGQy89Xsh3BS1ubwEVU1ItS9/bfVamDlyyYHB+h664VVbljQwbvEKy14=@vger.kernel.org X-Gm-Message-State: AOJu0YzvfabD8K3ktiTqLc+8qpLRAjIKLdNdzbmqy9MkiTO63e2DObrl pAOmBNWX4IYE8/GxtyoA/XiBhPVp/4dUYFLpDAphRaQVu5aocpxhYbp4 X-Gm-Gg: ATEYQzyP6d56EaqnvnC6LQU9/VnlauYJHo87Js6/wKyegP/wGXas2UbYxPjtXnQOla2 JydKlFl+bN0ZIsknfoXc3DKsoUd+NbHq8XnLx0IuNtMbQLOU/idJnIHHE52rCVEy3xDCLRY9uyG a0177w5pl88sTyuQOagj5qPHe/+NI6OE+8yw53cJ/eagj7zLi2dhvRwBgNekvmc2sqTsIN4OAss MwDE7hI4W2Tlx7LEowTJjsblon8OfDL7wPf75QIsIZopEwejGgKPLLrjMnbdBgMSfbwMpuAS4/q Z+tgbfg3M60812zulnxG1nq20xciW26dwhtnNcEdDZTyMhlspoCU0Akd1/4sDiKRDTnza+wuQSp s9UIsnxopjupj0HDf3ub7DgoLwL5ZkbayKVK1iWRAY3ty8ybCdBbHz/PNGoVFlOaKb+MA+JICPa nMDqH+0tnAxdc2UtjXir2wmw== X-Received: by 2002:a05:6830:8388:b0:7cb:125d:2a43 with SMTP id 46e09a7af769-7d591be4a4dmr307474a34.28.1772134193631; Thu, 26 Feb 2026 11:29:53 -0800 (PST) Received: from localhost ([2a03:2880:10ff:5e::]) by smtp.gmail.com with ESMTPSA id 46e09a7af769-7d5866557c6sm2419781a34.24.2026.02.26.11.29.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 26 Feb 2026 11:29:53 -0800 (PST) From: Joshua Hahn To: Minchan Kim , Sergey Senozhatsky Cc: Johannes Weiner , Yosry Ahmed , Nhat Pham , Nhat Pham , Chengming Zhou , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Axel Rasmussen , Yuanchu Xie , Wei Xu , David Hildenbrand , Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Andrew Morton , cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@meta.com Subject: [PATCH 8/8] mm/vmstat, memcontrol: Track ZSWAP_B, ZSWAPPED_B per-memcg-lruvec Date: Thu, 26 Feb 2026 11:29:31 -0800 Message-ID: <20260226192936.3190275-9-joshua.hahnjy@gmail.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20260226192936.3190275-1-joshua.hahnjy@gmail.com> References: <20260226192936.3190275-1-joshua.hahnjy@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Now that memcg charging happens in the zsmalloc layer where we have both objcg and page information, we can specify which node's memcg lruvec zswapped memory should be accounted to. Move MEMCG_ZSWAP_B and MEMCG_ZSWAPPED_B from enum_node_stat_item to int memcg_node_stat_items. Rename their prefix from MEMCG to NR to reflect this move as well. In addition, decouple the updates of node stats (vmstat) and memcg-lruvec stats, since node stats can only track values at a PAGE_SIZE granularity. Finally, track the moving charges whenever a compressed object migrates from one zspage to another. memcg-lruvec stats are now updated precisely and proportionally when compressed objects are split across pages. Unfortunately for node stats, only NR_ZSWAP_B can be kept accurate. NR_ZSWAPPED_B works as a good best-effort value, but cannot proportionally account for compressed objects split across pages due to the coarse PAGE_SIZE granularity of node stats. For such objects, NR_ZSWAPPED_B is accounted to the first zpdesc's node stats. Note that this is not a new inaccuracy, but one that is simply left unable to be fixed as part of these changes. The small inaccuracy is accepted in place of invasive changes across all of vmstat infrastructure to begin tracking stats at byte granularity. Suggested-by: Johannes Weiner Signed-off-by: Joshua Hahn --- include/linux/memcontrol.h | 5 +-- include/linux/mmzone.h | 2 ++ mm/memcontrol.c | 18 +++++----- mm/vmstat.c | 2 ++ mm/zsmalloc.c | 72 ++++++++++++++++++++++++++++++-------- mm/zswap.c | 4 +-- 6 files changed, 76 insertions(+), 27 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index d3952c918fd4..ba97b86d9104 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -37,8 +37,6 @@ enum memcg_stat_item { MEMCG_PERCPU_B, MEMCG_VMALLOC, MEMCG_KMEM, - MEMCG_ZSWAP_B, - MEMCG_ZSWAPPED_B, MEMCG_NR_STAT, }; =20 @@ -932,6 +930,9 @@ void mem_cgroup_print_oom_group(struct mem_cgroup *memc= g); void mod_memcg_state(struct mem_cgroup *memcg, enum memcg_stat_item idx, int val); =20 +void mod_memcg_lruvec_state(struct lruvec *lruvec, enum node_stat_item idx, + int val); + static inline void mod_memcg_page_state(struct page *page, enum memcg_stat_item idx, int val) { diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 3e51190a55e4..ae16a90491ac 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -258,6 +258,8 @@ enum node_stat_item { #ifdef CONFIG_HUGETLB_PAGE NR_HUGETLB, #endif + NR_ZSWAP_B, + NR_ZSWAPPED_B, NR_BALLOON_PAGES, NR_KERNEL_FILE_PAGES, NR_VM_NODE_STAT_ITEMS diff --git a/mm/memcontrol.c b/mm/memcontrol.c index b662902d4e03..dc7cfff97296 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -331,6 +331,8 @@ static const unsigned int memcg_node_stat_items[] =3D { #ifdef CONFIG_HUGETLB_PAGE NR_HUGETLB, #endif + NR_ZSWAP_B, + NR_ZSWAPPED_B, }; =20 static const unsigned int memcg_stat_items[] =3D { @@ -339,8 +341,6 @@ static const unsigned int memcg_stat_items[] =3D { MEMCG_PERCPU_B, MEMCG_VMALLOC, MEMCG_KMEM, - MEMCG_ZSWAP_B, - MEMCG_ZSWAPPED_B, }; =20 #define NR_MEMCG_NODE_STAT_ITEMS ARRAY_SIZE(memcg_node_stat_items) @@ -726,7 +726,7 @@ unsigned long memcg_page_state_local(struct mem_cgroup = *memcg, int idx) } #endif =20 -static void mod_memcg_lruvec_state(struct lruvec *lruvec, +void mod_memcg_lruvec_state(struct lruvec *lruvec, enum node_stat_item idx, int val) { @@ -1344,8 +1344,8 @@ static const struct memory_stat memory_stats[] =3D { { "vmalloc", MEMCG_VMALLOC }, { "shmem", NR_SHMEM }, #ifdef CONFIG_ZSWAP - { "zswap", MEMCG_ZSWAP_B }, - { "zswapped", MEMCG_ZSWAPPED_B }, + { "zswap", NR_ZSWAP_B }, + { "zswapped", NR_ZSWAPPED_B }, #endif { "file_mapped", NR_FILE_MAPPED }, { "file_dirty", NR_FILE_DIRTY }, @@ -1392,8 +1392,8 @@ static int memcg_page_state_unit(int item) { switch (item) { case MEMCG_PERCPU_B: - case MEMCG_ZSWAP_B: - case MEMCG_ZSWAPPED_B: + case NR_ZSWAP_B: + case NR_ZSWAPPED_B: case NR_SLAB_RECLAIMABLE_B: case NR_SLAB_UNRECLAIMABLE_B: return 1; @@ -5424,7 +5424,7 @@ bool obj_cgroup_may_zswap(struct obj_cgroup *objcg) =20 /* Force flush to get accurate stats for charging */ __mem_cgroup_flush_stats(memcg, true); - pages =3D memcg_page_state(memcg, MEMCG_ZSWAP_B) / PAGE_SIZE; + pages =3D memcg_page_state(memcg, NR_ZSWAP_B) / PAGE_SIZE; if (pages < max) continue; ret =3D false; @@ -5453,7 +5453,7 @@ static u64 zswap_current_read(struct cgroup_subsys_st= ate *css, struct mem_cgroup *memcg =3D mem_cgroup_from_css(css); =20 mem_cgroup_flush_stats(memcg); - return memcg_page_state(memcg, MEMCG_ZSWAP_B); + return memcg_page_state(memcg, NR_ZSWAP_B); } =20 static int zswap_max_show(struct seq_file *m, void *v) diff --git a/mm/vmstat.c b/mm/vmstat.c index 99270713e0c1..4b10610bd999 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1279,6 +1279,8 @@ const char * const vmstat_text[] =3D { #ifdef CONFIG_HUGETLB_PAGE [I(NR_HUGETLB)] =3D "nr_hugetlb", #endif + [I(NR_ZSWAP_B)] =3D "zswap", + [I(NR_ZSWAPPED_B)] =3D "zswapped", [I(NR_BALLOON_PAGES)] =3D "nr_balloon_pages", [I(NR_KERNEL_FILE_PAGES)] =3D "nr_kernel_file_pages", #undef I diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 6794927c60fb..548e7f4b8bf6 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -810,6 +810,7 @@ static void __free_zspage(struct zs_pool *pool, struct = size_class *class, struct zspage *zspage) { struct zpdesc *zpdesc, *next; + bool objcg =3D !!zpdesc_objcgs(zspage->first_zpdesc); =20 assert_spin_locked(&class->lock); =20 @@ -823,6 +824,8 @@ static void __free_zspage(struct zs_pool *pool, struct = size_class *class, reset_zpdesc(zpdesc); zpdesc_unlock(zpdesc); zpdesc_dec_zone_page_state(zpdesc); + if (objcg) + dec_node_page_state(zpdesc_page(zpdesc), NR_ZSWAP_B); zpdesc_put(zpdesc); zpdesc =3D next; } while (zpdesc !=3D NULL); @@ -963,11 +966,45 @@ static bool alloc_zspage_objcgs(struct size_class *cl= ass, gfp_t gfp, return true; } =20 -static void zs_charge_objcg(struct zpdesc *zpdesc, struct obj_cgroup *objc= g, - int size, unsigned long offset) +static void __zs_mod_memcg_lruvec(struct zpdesc *zpdesc, + struct obj_cgroup *objcg, int size, + int sign, unsigned long offset) { struct mem_cgroup *memcg; + struct lruvec *lruvec; + int compressed_size =3D size, original_size =3D PAGE_SIZE; + int nid =3D page_to_nid(zpdesc_page(zpdesc)); + int next_nid =3D nid; + + if (offset + size > PAGE_SIZE) { + struct zpdesc *next_zpdesc =3D get_next_zpdesc(zpdesc); + + next_nid =3D page_to_nid(zpdesc_page(next_zpdesc)); + if (nid !=3D next_nid) { + compressed_size =3D PAGE_SIZE - offset; + original_size =3D (PAGE_SIZE * compressed_size) / size; + } + } + + rcu_read_lock(); + memcg =3D obj_cgroup_memcg(objcg); + lruvec =3D mem_cgroup_lruvec(memcg, NODE_DATA(nid)); + mod_memcg_lruvec_state(lruvec, NR_ZSWAP_B, sign * compressed_size); + mod_memcg_lruvec_state(lruvec, NR_ZSWAPPED_B, sign * original_size); + + if (nid !=3D next_nid) { + lruvec =3D mem_cgroup_lruvec(memcg, NODE_DATA(next_nid)); + mod_memcg_lruvec_state(lruvec, NR_ZSWAP_B, + sign * (size - compressed_size)); + mod_memcg_lruvec_state(lruvec, NR_ZSWAPPED_B, + sign * (PAGE_SIZE - original_size)); + } + rcu_read_unlock(); +} =20 +static void zs_charge_objcg(struct zpdesc *zpdesc, struct obj_cgroup *objc= g, + int size, unsigned long offset) +{ if (!cgroup_subsys_on_dfl(memory_cgrp_subsys)) return; =20 @@ -977,28 +1014,30 @@ static void zs_charge_objcg(struct zpdesc *zpdesc, s= truct obj_cgroup *objcg, if (obj_cgroup_charge(objcg, GFP_KERNEL, size)) VM_WARN_ON_ONCE(1); =20 - rcu_read_lock(); - memcg =3D obj_cgroup_memcg(objcg); - mod_memcg_state(memcg, MEMCG_ZSWAP_B, size); - mod_memcg_state(memcg, MEMCG_ZSWAPPED_B, 1); - rcu_read_unlock(); + __zs_mod_memcg_lruvec(zpdesc, objcg, size, 1, offset); + + /* + * Node-level vmstats are charged in PAGE_SIZE units. As a + * best-effort, always charge NR_ZSWAPPED_B to the first zpdesc. + */ + inc_node_page_state(zpdesc_page(zpdesc), NR_ZSWAPPED_B); } =20 static void zs_uncharge_objcg(struct zpdesc *zpdesc, struct obj_cgroup *ob= jcg, int size, unsigned long offset) { - struct mem_cgroup *memcg; - if (!cgroup_subsys_on_dfl(memory_cgrp_subsys)) return; =20 obj_cgroup_uncharge(objcg, size); =20 - rcu_read_lock(); - memcg =3D obj_cgroup_memcg(objcg); - mod_memcg_state(memcg, MEMCG_ZSWAP_B, -size); - mod_memcg_state(memcg, MEMCG_ZSWAPPED_B, -1); - rcu_read_unlock(); + __zs_mod_memcg_lruvec(zpdesc, objcg, size, -1, offset); + + /* + * Node-level vmstats are uncharged in PAGE_SIZE units. As a + * best-effort, always uncharge NR_ZSWAPPED_B to the first zpdesc. + */ + dec_node_page_state(zpdesc_page(zpdesc), NR_ZSWAPPED_B); } =20 static void migrate_obj_objcg(unsigned long used_obj, unsigned long free_o= bj, @@ -1135,6 +1174,8 @@ static struct zspage *alloc_zspage(struct zs_pool *po= ol, __zpdesc_set_zsmalloc(zpdesc); =20 zpdesc_inc_zone_page_state(zpdesc); + if (objcg) + inc_node_page_state(zpdesc_page(zpdesc), NR_ZSWAP_B); zpdescs[i] =3D zpdesc; } =20 @@ -1149,6 +1190,9 @@ static struct zspage *alloc_zspage(struct zs_pool *po= ol, err: while (--i >=3D 0) { zpdesc_dec_zone_page_state(zpdescs[i]); + if (objcg) + dec_node_page_state(zpdesc_page(zpdescs[i]), + NR_ZSWAP_B); free_zpdesc(zpdescs[i]); } cache_free_zspage(zspage); diff --git a/mm/zswap.c b/mm/zswap.c index 97f38d0afa86..9e845e1d7214 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -1214,9 +1214,9 @@ static unsigned long zswap_shrinker_count(struct shri= nker *shrinker, */ if (!mem_cgroup_disabled()) { mem_cgroup_flush_stats(memcg); - nr_backing =3D memcg_page_state(memcg, MEMCG_ZSWAP_B); + nr_backing =3D memcg_page_state(memcg, NR_ZSWAP_B); nr_backing >>=3D PAGE_SHIFT; - nr_stored =3D memcg_page_state(memcg, MEMCG_ZSWAPPED_B); + nr_stored =3D memcg_page_state(memcg, NR_ZSWAPPED_B); nr_stored >>=3D PAGE_SHIFT; } else { nr_backing =3D zswap_total_pages(); --=20 2.47.3