From nobody Thu Nov 28 13:37:13 2024 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.11]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 93AB518E02D for ; Tue, 1 Oct 2024 05:37:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.11 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727761029; cv=none; b=pQnwzQwIblgpfhuMtDgSyixgWijcONd13xCriGFu90JeowhctTex85HNfkl4VpfZSEcZg8pHjISNa+PoVWplqUXzRF10aZaZp89m9tRL2Rle+TySw6a3OLfSLDJgtAInGFkempDahy/yr745Dbe6jEBWept8SMJBWfZuEMSCLfM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727761029; c=relaxed/simple; bh=HXPnxaS0BG2kkNkoIrCoeh8TspMXuNCwKniqdIZtWWM=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Ls/IWK2Z9W7oGK4M7qX/TEx63FCQr2+PhIyQu1IJccMCSBx8UaosE7EeWU3UweyPeCQ250K+y6xhh19M7zIskANT8opqmVwOGFdNR+0Vt9jVkvljj0TnZDJgTLRgZbIzTv2KJ79xCA0RfVwEZSCHp0jjYcIZyzbVO6fmC2n+fLA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=XMZGQ65s; arc=none smtp.client-ip=192.198.163.11 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="XMZGQ65s" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1727761028; x=1759297028; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=HXPnxaS0BG2kkNkoIrCoeh8TspMXuNCwKniqdIZtWWM=; b=XMZGQ65smx+ALRQJjuRIqAoKyRPJ2tEbTkpCSt1+kFCsAsoU2sblWoIO 7gu1LBSYoHFURtdNHFaUk/Q0mDkspkv0+emsBR8oABMOVa3lB2ZLs0KMK fQv+rV18WMy3FxIBtewaUXBY0effQIhvpO8hKiw0CZQLeWUgmko5njczv DH+SE8c9XmOOkNTj02lOTSpBZ69/1L+6HeDOF6pRAL4DFVwhvSqYsHN4a SpHx5dBuN5K/nQRFNuIJDzGRBrYbiXSShCGVsvDltVbAtRkPCI2Z2Dgs3 6rZ7NooYVn3LqKeFGf+mcoPsRqhNnbza7OGNpuQIXzBQE5/ZnW+wJscYX w==; X-CSE-ConnectionGUID: NJAPLnU0SziO0NXTr3anww== X-CSE-MsgGUID: CxccQfA1Qj+TXDqqnRcrRw== X-IronPort-AV: E=McAfee;i="6700,10204,11211"; a="37465066" X-IronPort-AV: E=Sophos;i="6.11,167,1725346800"; d="scan'208";a="37465066" Received: from fmviesa007.fm.intel.com ([10.60.135.147]) by fmvoesa105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 30 Sep 2024 22:32:23 -0700 X-CSE-ConnectionGUID: 5FyCjXUQQY2zJJxclxHnuw== X-CSE-MsgGUID: 4DdRtYD0TvK3eIsp+yhRmA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.11,167,1725346800"; d="scan'208";a="73205793" Received: from jf5300-b11a338t.jf.intel.com ([10.242.51.6]) by fmviesa007.fm.intel.com with ESMTP; 30 Sep 2024 22:32:23 -0700 From: Kanchana P Sridhar To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, hannes@cmpxchg.org, yosryahmed@google.com, nphamcs@gmail.com, chengming.zhou@linux.dev, usamaarif642@gmail.com, shakeel.butt@linux.dev, ryan.roberts@arm.com, ying.huang@intel.com, 21cnbao@gmail.com, akpm@linux-foundation.org, willy@infradead.org Cc: nanhai.zou@intel.com, wajdi.k.feghali@intel.com, vinodh.gopal@intel.com, kanchana.p.sridhar@intel.com Subject: [PATCH v10 1/7] mm: Define obj_cgroup_get() if CONFIG_MEMCG is not defined. Date: Mon, 30 Sep 2024 22:32:16 -0700 Message-Id: <20241001053222.6944-2-kanchana.p.sridhar@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20241001053222.6944-1-kanchana.p.sridhar@intel.com> References: <20241001053222.6944-1-kanchana.p.sridhar@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This resolves an issue with obj_cgroup_get() not being defined if CONFIG_MEMCG is not defined. Before this patch, we would see build errors if obj_cgroup_get() is called from code that is agnostic of CONFIG_MEMCG. The zswap_store() changes for large folios in subsequent commits will require the use of obj_cgroup_get() in zswap code that falls into this category. Signed-off-by: Kanchana P Sridhar Reviewed-by: Nhat Pham Reviewed-by: Yosry Ahmed Reviewed-by: Chengming Zhou Acked-by: Johannes Weiner --- include/linux/memcontrol.h | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 34d2da05f2f1..15c2716f9aa3 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1282,6 +1282,10 @@ struct mem_cgroup *mem_cgroup_from_css(struct cgroup= _subsys_state *css) return NULL; } =20 +static inline void obj_cgroup_get(struct obj_cgroup *objcg) +{ +} + static inline void obj_cgroup_put(struct obj_cgroup *objcg) { } --=20 2.27.0 From nobody Thu Nov 28 13:37:13 2024 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.11]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6DB121A4F2B for ; Tue, 1 Oct 2024 05:37:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.11 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727761031; cv=none; b=BsI70Rr2KxVp3siraYVygpWlkl7bQEiFq/DxcKTyRJCYlf62nA13kGqRCbznEjcx7EiotD/ikv79/zvpMbRlNen+R8KhJ8CvHGBP4/MNGukFnC8QDuIJqw4+ywcXccOPMXNLM0mA3csk5RExBxvUJi2Nviy8Y/40XxzM6yJ2vhQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727761031; c=relaxed/simple; bh=efGW2zzH8VCItzcrEw7X5/h61cbLWY8/tK2pA3rWPZo=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=CGL3cs26rGj0cAnjuHwy7l6XT5eGndGSGWXcMmhbpffLmWEdKhXfR/0ncY6L9Bk6Dsw5ETooag023QQ8joCFZBI28H5Hp+7+WGrxD+Qc8fNGYVLM5Dwxm1r/R138kIAPgsEm7xmWFdIVZWI6K9aNWJ/ySIpekw/JG/QR00MsXKQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=cJKNjHB0; arc=none smtp.client-ip=192.198.163.11 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="cJKNjHB0" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1727761029; x=1759297029; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=efGW2zzH8VCItzcrEw7X5/h61cbLWY8/tK2pA3rWPZo=; b=cJKNjHB0jO3WFZkpOxM7cU5eYcbBhHg2tTg8sMJEpABVZbvT8wOrmAZD JQw/fDoFriDukXyKJDvUfvt/9DGP8TLsZE8ZP76WC7tJeGXH+sVrU575h na4+r8GifWZfhYbr+pDiSvdbUmytV6lgGTGNBRtOHPf5+bBlV5pORvMPh i93dws64e0CBiz9Bc43Wb9QMJM08qsDLWI/+dBFFBx3LQZmjRaITutkqe eTVUS1zqsOwImO1A1SAStX/YIqUoPwoJWD1fBVUYKD1mOpprPCWp841yw zEPnpEbQyeu6LPHgYpJeOaYXyxN13sDMOEOgcS6744a848pyImWHHhihQ Q==; X-CSE-ConnectionGUID: 2X0JVJSHQraN9g06VMD3UA== X-CSE-MsgGUID: qNvtyG6XRAGbnwdKGVUGRA== X-IronPort-AV: E=McAfee;i="6700,10204,11211"; a="37465079" X-IronPort-AV: E=Sophos;i="6.11,167,1725346800"; d="scan'208";a="37465079" Received: from fmviesa007.fm.intel.com ([10.60.135.147]) by fmvoesa105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 30 Sep 2024 22:32:24 -0700 X-CSE-ConnectionGUID: w49pgGxrSHOs/Gotpi9JWA== X-CSE-MsgGUID: AyNh3wxhT7C8wmJiQpdeOg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.11,167,1725346800"; d="scan'208";a="73205798" Received: from jf5300-b11a338t.jf.intel.com ([10.242.51.6]) by fmviesa007.fm.intel.com with ESMTP; 30 Sep 2024 22:32:24 -0700 From: Kanchana P Sridhar To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, hannes@cmpxchg.org, yosryahmed@google.com, nphamcs@gmail.com, chengming.zhou@linux.dev, usamaarif642@gmail.com, shakeel.butt@linux.dev, ryan.roberts@arm.com, ying.huang@intel.com, 21cnbao@gmail.com, akpm@linux-foundation.org, willy@infradead.org Cc: nanhai.zou@intel.com, wajdi.k.feghali@intel.com, vinodh.gopal@intel.com, kanchana.p.sridhar@intel.com Subject: [PATCH v10 2/7] mm: zswap: Modify zswap_compress() to accept a page instead of a folio. Date: Mon, 30 Sep 2024 22:32:17 -0700 Message-Id: <20241001053222.6944-3-kanchana.p.sridhar@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20241001053222.6944-1-kanchana.p.sridhar@intel.com> References: <20241001053222.6944-1-kanchana.p.sridhar@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" For zswap_store() to be able to store a large folio by compressing it one page at a time, zswap_compress() needs to accept a page as input. This will allow us to iterate through each page in the folio in zswap_store(), compress it and store it in the zpool. Signed-off-by: Kanchana P Sridhar Reviewed-by: Nhat Pham Reviewed-by: Chengming Zhou Acked-by: Johannes Weiner Acked-by: Yosry Ahmed --- mm/zswap.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index efad4e941e44..fd7a8c14457a 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -875,7 +875,7 @@ static int zswap_cpu_comp_dead(unsigned int cpu, struct= hlist_node *node) return 0; } =20 -static bool zswap_compress(struct folio *folio, struct zswap_entry *entry) +static bool zswap_compress(struct page *page, struct zswap_entry *entry) { struct crypto_acomp_ctx *acomp_ctx; struct scatterlist input, output; @@ -893,7 +893,7 @@ static bool zswap_compress(struct folio *folio, struct = zswap_entry *entry) =20 dst =3D acomp_ctx->buffer; sg_init_table(&input, 1); - sg_set_folio(&input, folio, PAGE_SIZE, 0); + sg_set_page(&input, page, PAGE_SIZE, 0); =20 /* * We need PAGE_SIZE * 2 here since there maybe over-compression case, @@ -1456,7 +1456,7 @@ bool zswap_store(struct folio *folio) mem_cgroup_put(memcg); } =20 - if (!zswap_compress(folio, entry)) + if (!zswap_compress(&folio->page, entry)) goto put_pool; =20 entry->swpentry =3D swp; --=20 2.27.0 From nobody Thu Nov 28 13:37:13 2024 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.11]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CAD711B9B50 for ; Tue, 1 Oct 2024 05:37:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.11 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727761031; cv=none; b=NpUVCZvctFk1CA/RgfNzb44dObFuOy9V1o6B8Qwqqjhown400sOinWhvzPxY1rkeDxqxVZJcXILwAY3G9F+8bKqgHNrsqZt9iYiJiDzg6jc3eQ359lTTXUVEcK0UT3xbD3xrseiNjfCvBkWs9ztkaRjOI8j9QMmBIuI35+oQ8Fc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727761031; c=relaxed/simple; bh=RXFigHVvqX66h0CgnEPMdbPUVdZNL13x+vDE/BJrhj8=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=FNEwB8iq34nHrDVZrMZIHJ6McR5+0u/Mj45yTyfgGOKLqJFiiCRAVq3SNldUI1KUTSf6XT357RsU9if9TU5XCPVP5VuIloYzw1DRaY0y4AbxmiS3qwsstk8PFcAk6Gg8m2ysKXaJ1vce0IiFIKEwZcw0t3nM8CURUtUjSxDSki8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=Be1+Yeju; arc=none smtp.client-ip=192.198.163.11 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="Be1+Yeju" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1727761030; x=1759297030; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=RXFigHVvqX66h0CgnEPMdbPUVdZNL13x+vDE/BJrhj8=; b=Be1+YejuQ6+pSY8wua0nZOPkZXH7Q1djpHUlVH/AsBhssl5zQ7a+CRCx fWEgSotZ3orib/LX5jK/0C42L011Ja009txFCqXksCr2UuJ7l/wA4S7Ag wxyl+s840haFz2WQFjv/qV94cUQHCVFOJbRJT+40hKTBXg/LjKclckteY i4S+2g71AJAyj6B7UlVbQzHaTGfuETDx0JAn3uDdPIvNi4w8WqpiFPV8q ali21ggtn71pEJQM21aIL3whLfz2w0mRjpRBupwgmrLHgBsfkjPjw5BKw aC3Kke2GnehHoiP/V6dpxgFhDPIDZ9NqZBaZxIbmuXZmPv1ujCHFrV23H g==; X-CSE-ConnectionGUID: JdQ0KIhgRNG+KRSRbiNzRQ== X-CSE-MsgGUID: fGE4abNsTkiBx0lM+fPkfg== X-IronPort-AV: E=McAfee;i="6700,10204,11211"; a="37465092" X-IronPort-AV: E=Sophos;i="6.11,167,1725346800"; d="scan'208";a="37465092" Received: from fmviesa007.fm.intel.com ([10.60.135.147]) by fmvoesa105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 30 Sep 2024 22:32:25 -0700 X-CSE-ConnectionGUID: 8AwPM96ESfa4/3bCeF/HBQ== X-CSE-MsgGUID: +JTWy565Q2O8jcvuOpSgUA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.11,167,1725346800"; d="scan'208";a="73205801" Received: from jf5300-b11a338t.jf.intel.com ([10.242.51.6]) by fmviesa007.fm.intel.com with ESMTP; 30 Sep 2024 22:32:24 -0700 From: Kanchana P Sridhar To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, hannes@cmpxchg.org, yosryahmed@google.com, nphamcs@gmail.com, chengming.zhou@linux.dev, usamaarif642@gmail.com, shakeel.butt@linux.dev, ryan.roberts@arm.com, ying.huang@intel.com, 21cnbao@gmail.com, akpm@linux-foundation.org, willy@infradead.org Cc: nanhai.zou@intel.com, wajdi.k.feghali@intel.com, vinodh.gopal@intel.com, kanchana.p.sridhar@intel.com Subject: [PATCH v10 3/7] mm: zswap: Rename zswap_pool_get() to zswap_pool_tryget(). Date: Mon, 30 Sep 2024 22:32:18 -0700 Message-Id: <20241001053222.6944-4-kanchana.p.sridhar@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20241001053222.6944-1-kanchana.p.sridhar@intel.com> References: <20241001053222.6944-1-kanchana.p.sridhar@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Modify the name of the existing zswap_pool_get() to zswap_pool_tryget() to be representative of the call it makes to percpu_ref_tryget(). A subsequent patch will introduce a new zswap_pool_get() that calls percpu_ref_get(). The intent behind this change is for higher level zswap API such as zswap_store() to call zswap_pool_tryget() to check upfront if the pool's refcount is "0" (which means it could be getting destroyed) and to handle this as an error condition. zswap_store() would proceed only if zswap_pool_tryget() returns success, and any additional pool refcounts that need to be obtained for compressing sub-pages in a large folio could simply call zswap_pool_get(). Signed-off-by: Kanchana P Sridhar Acked-by: Yosry Ahmed Reviewed-by: Chengming Zhou Acked-by: Johannes Weiner Reviewed-by: Nhat Pham --- mm/zswap.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index fd7a8c14457a..0f281e50a034 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -403,7 +403,7 @@ static void __zswap_pool_empty(struct percpu_ref *ref) spin_unlock_bh(&zswap_pools_lock); } =20 -static int __must_check zswap_pool_get(struct zswap_pool *pool) +static int __must_check zswap_pool_tryget(struct zswap_pool *pool) { if (!pool) return 0; @@ -441,7 +441,7 @@ static struct zswap_pool *zswap_pool_current_get(void) rcu_read_lock(); =20 pool =3D __zswap_pool_current(); - if (!zswap_pool_get(pool)) + if (!zswap_pool_tryget(pool)) pool =3D NULL; =20 rcu_read_unlock(); @@ -462,7 +462,7 @@ static struct zswap_pool *zswap_pool_find_get(char *typ= e, char *compressor) if (strcmp(zpool_get_type(pool->zpool), type)) continue; /* if we can't get it, it's about to be destroyed */ - if (!zswap_pool_get(pool)) + if (!zswap_pool_tryget(pool)) continue; return pool; } --=20 2.27.0 From nobody Thu Nov 28 13:37:13 2024 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.11]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 00C291BB6BF for ; Tue, 1 Oct 2024 05:37:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.11 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727761031; cv=none; b=HwWDuA9GyDbr5vVCkGlstWyaZsu+8Fwp41yYe7In+RvVfymj+84+Iw3TN/mCjfcHqN/znrGmxQ1X2O+jB09CTzy/LHHtuss9dV5qPlzE0UK93yWE3ODEKUsXOtvDZRgOgrRgj3st22AuC+VKdc15MtDtdvslNTETW+o4gdljVXw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727761031; c=relaxed/simple; bh=lB+hvyf3yxepm5Kmd6XwZLQVvnN7/oVBqCxfq4eIzGU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Gsgd2Fr25a5B9/hDbhN47XVJv7nWBq+WyqMfkp0d96VoPX2TSjpCO/pvQKnQFz9r2TmVMFgGkeAbUPpGOdHoXuEO1cLqWVZK6eeCy1LVvYpyGAqwp/Zz1lDMeNkpTBGTAZecX9SrkGcC75gPdXDdQMU7+NOUFFxq6kZUafNY7as= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=YjkEP2gv; arc=none smtp.client-ip=192.198.163.11 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="YjkEP2gv" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1727761030; x=1759297030; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=lB+hvyf3yxepm5Kmd6XwZLQVvnN7/oVBqCxfq4eIzGU=; b=YjkEP2gvbNTPZZMJBJGZgXVgzTu5b4GyYSwBJJmumeUYA88EWEfJ7qNv 5BSVXNbn78D0hT4HRxOi+w6vKrO9q76nkyBbAbD0ruFMn11kqP8bvZuFb b13WAZ57TKPqXnpJwwO77gdFMlm2P95x75+89Ovhb17fsilash+RTRR/1 /XHtFp42zajeL6EEmnP7cr4TDfNq9ix9HupwR3Pj+A5F7sXJNgwkg2Jp+ JaA326zxupv7ZQ8QqXoityeNCLxybUL2Ud6dIDiYU3V5LBFnOg8yh8SiE REyAOO0W0ZpqVIPG0xcu1ACRFykYk++t2uX1eydRiIgY3TwmIW4ps5Rb1 w==; X-CSE-ConnectionGUID: k8QomehVSZCltBO6DX9u6Q== X-CSE-MsgGUID: ck6UqRWFSGuaGlhOcnZBNA== X-IronPort-AV: E=McAfee;i="6700,10204,11211"; a="37465103" X-IronPort-AV: E=Sophos;i="6.11,167,1725346800"; d="scan'208";a="37465103" Received: from fmviesa007.fm.intel.com ([10.60.135.147]) by fmvoesa105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 30 Sep 2024 22:32:25 -0700 X-CSE-ConnectionGUID: PKBxrDh5Qx2Gv53SQPU5uQ== X-CSE-MsgGUID: wtC/7FAEQlSZaTIPojmcow== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.11,167,1725346800"; d="scan'208";a="73205804" Received: from jf5300-b11a338t.jf.intel.com ([10.242.51.6]) by fmviesa007.fm.intel.com with ESMTP; 30 Sep 2024 22:32:25 -0700 From: Kanchana P Sridhar To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, hannes@cmpxchg.org, yosryahmed@google.com, nphamcs@gmail.com, chengming.zhou@linux.dev, usamaarif642@gmail.com, shakeel.butt@linux.dev, ryan.roberts@arm.com, ying.huang@intel.com, 21cnbao@gmail.com, akpm@linux-foundation.org, willy@infradead.org Cc: nanhai.zou@intel.com, wajdi.k.feghali@intel.com, vinodh.gopal@intel.com, kanchana.p.sridhar@intel.com Subject: [PATCH v10 4/7] mm: Change count_objcg_event() to count_objcg_events() for batch event updates. Date: Mon, 30 Sep 2024 22:32:19 -0700 Message-Id: <20241001053222.6944-5-kanchana.p.sridhar@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20241001053222.6944-1-kanchana.p.sridhar@intel.com> References: <20241001053222.6944-1-kanchana.p.sridhar@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" With the introduction of zswap_store() swapping out large folios, we need to efficiently update the objcg's memcg events once per successfully stored folio. For instance, the 'ZSWPOUT' event needs to be incremented by folio_nr_pages(). To facilitate this, the existing count_objcg_event() API is modified to be count_objcg_events() that additionally accepts a count parameter. The only existing calls to count_objcg_event() are in zswap.c - these have been modified to call count_objcg_events() with a count of 1. Signed-off-by: Kanchana P Sridhar Acked-by: Yosry Ahmed Reviewed-by: Nhat Pham Acked-by: Johannes Weiner --- include/linux/memcontrol.h | 12 +++++++----- mm/zswap.c | 6 +++--- 2 files changed, 10 insertions(+), 8 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 15c2716f9aa3..524006313b0d 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1764,8 +1764,9 @@ static inline int memcg_kmem_id(struct mem_cgroup *me= mcg) =20 struct mem_cgroup *mem_cgroup_from_slab_obj(void *p); =20 -static inline void count_objcg_event(struct obj_cgroup *objcg, - enum vm_event_item idx) +static inline void count_objcg_events(struct obj_cgroup *objcg, + enum vm_event_item idx, + unsigned long count) { struct mem_cgroup *memcg; =20 @@ -1774,7 +1775,7 @@ static inline void count_objcg_event(struct obj_cgrou= p *objcg, =20 rcu_read_lock(); memcg =3D obj_cgroup_memcg(objcg); - count_memcg_events(memcg, idx, 1); + count_memcg_events(memcg, idx, count); rcu_read_unlock(); } =20 @@ -1829,8 +1830,9 @@ static inline struct mem_cgroup *mem_cgroup_from_slab= _obj(void *p) return NULL; } =20 -static inline void count_objcg_event(struct obj_cgroup *objcg, - enum vm_event_item idx) +static inline void count_objcg_events(struct obj_cgroup *objcg, + enum vm_event_item idx, + unsigned long count) { } =20 diff --git a/mm/zswap.c b/mm/zswap.c index 0f281e50a034..69b9c025fd47 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -1053,7 +1053,7 @@ static int zswap_writeback_entry(struct zswap_entry *= entry, =20 count_vm_event(ZSWPWB); if (entry->objcg) - count_objcg_event(entry->objcg, ZSWPWB); + count_objcg_events(entry->objcg, ZSWPWB, 1); =20 zswap_entry_free(entry); =20 @@ -1482,7 +1482,7 @@ bool zswap_store(struct folio *folio) =20 if (objcg) { obj_cgroup_charge_zswap(objcg, entry->length); - count_objcg_event(objcg, ZSWPOUT); + count_objcg_events(objcg, ZSWPOUT, 1); } =20 /* @@ -1576,7 +1576,7 @@ bool zswap_load(struct folio *folio) =20 count_vm_event(ZSWPIN); if (entry->objcg) - count_objcg_event(entry->objcg, ZSWPIN); + count_objcg_events(entry->objcg, ZSWPIN, 1); =20 if (swapcache) { zswap_entry_free(entry); --=20 2.27.0 From nobody Thu Nov 28 13:37:13 2024 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.11]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 819E11BBBDD for ; Tue, 1 Oct 2024 05:37:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.11 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727761033; cv=none; b=MRVy+jvuHDF0zBUXOcaq++06/2cz/67IfQOW/+h55vKxY0vaIpeqZVqDX+dKSgsEvSeSbFBD9UfRA1uZxaxD+AhEgs9gvaAAUpCmqYrLNpNgPOGF1hRTurWnDNG+vQ4BzariZihJTwQk4pFoEatu7iCNW1do5pfuiU/QP5yXtHk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727761033; c=relaxed/simple; bh=fh13koWj73Qksnj2l+56lXO4bpj9lkhBiLj2Y16CgXI=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=prvvdnQghSVMC4EF8eztfLAW9PTZGjrXOrsKglzFTOtKKQcokrUgNjstQV/TfHimYOd3phkMR3OB7oGMfjhr+L+KVwuqIrQwamSs0N1/8S+LSbNM/vTfFCH1TIsk5uivHhmjX66EigOIlYZcsIoC8rqcVBw63UPievu9aTXJwrs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=maPrlA2p; arc=none smtp.client-ip=192.198.163.11 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="maPrlA2p" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1727761031; x=1759297031; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=fh13koWj73Qksnj2l+56lXO4bpj9lkhBiLj2Y16CgXI=; b=maPrlA2pMMXpV55fpPB0+xNtn5nctEAuAQKtRlYuMXRJvC8bDXIC8trS M4CEaTftPiVIXF/kElFw4Q5zIUyJh5o96bHynFBBh0WN/6RTw/X6+w+0+ BJlhTjrvnQW9q7DGgdjgk+Ir4MKWVbyFGDQ+IH2aad0LJHkXb/uzlxQko A1+rvpZCjipDpGZOUhaKgHFD6dG+4w2yYyZ8kdqXQnQ4ffBmkyJZ2GrZE jVYf7NExAxtMg65pBFP/ooiqaNc0fXuVmSptmfQsfXrPPg1a2pRmBAX6R NOJGb93bivFsCJigaLdItO4Z5ZKpJl9EpoQ4wQc0z5/2TP7dU3VFjpkKj w==; X-CSE-ConnectionGUID: xZ2ej7T+RQOeP9FDG2yKow== X-CSE-MsgGUID: cihIhq6VQNKtum542kyyfg== X-IronPort-AV: E=McAfee;i="6700,10204,11211"; a="37465115" X-IronPort-AV: E=Sophos;i="6.11,167,1725346800"; d="scan'208";a="37465115" Received: from fmviesa007.fm.intel.com ([10.60.135.147]) by fmvoesa105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 30 Sep 2024 22:32:26 -0700 X-CSE-ConnectionGUID: kM2GQueiQaqyj/KDhjFPcA== X-CSE-MsgGUID: kqT1lRAiTy6kXLW9DrzM5g== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.11,167,1725346800"; d="scan'208";a="73205808" Received: from jf5300-b11a338t.jf.intel.com ([10.242.51.6]) by fmviesa007.fm.intel.com with ESMTP; 30 Sep 2024 22:32:25 -0700 From: Kanchana P Sridhar To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, hannes@cmpxchg.org, yosryahmed@google.com, nphamcs@gmail.com, chengming.zhou@linux.dev, usamaarif642@gmail.com, shakeel.butt@linux.dev, ryan.roberts@arm.com, ying.huang@intel.com, 21cnbao@gmail.com, akpm@linux-foundation.org, willy@infradead.org Cc: nanhai.zou@intel.com, wajdi.k.feghali@intel.com, vinodh.gopal@intel.com, kanchana.p.sridhar@intel.com Subject: [PATCH v10 5/7] mm: zswap: Modify zswap_stored_pages to be atomic_long_t. Date: Mon, 30 Sep 2024 22:32:20 -0700 Message-Id: <20241001053222.6944-6-kanchana.p.sridhar@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20241001053222.6944-1-kanchana.p.sridhar@intel.com> References: <20241001053222.6944-1-kanchana.p.sridhar@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" For zswap_store() to support large folios, we need to be able to do a batch update of zswap_stored_pages upon successful store of all pages in the folio. For this, we need to add folio_nr_pages(), which returns a long, to zswap_stored_pages. Signed-off-by: Kanchana P Sridhar Acked-by: Yosry Ahmed Acked-by: Johannes Weiner Reviewed-by: Nhat Pham --- fs/proc/meminfo.c | 2 +- include/linux/zswap.h | 2 +- mm/zswap.c | 19 +++++++++++++------ 3 files changed, 15 insertions(+), 8 deletions(-) diff --git a/fs/proc/meminfo.c b/fs/proc/meminfo.c index 245171d9164b..8ba9b1472390 100644 --- a/fs/proc/meminfo.c +++ b/fs/proc/meminfo.c @@ -91,7 +91,7 @@ static int meminfo_proc_show(struct seq_file *m, void *v) #ifdef CONFIG_ZSWAP show_val_kb(m, "Zswap: ", zswap_total_pages()); seq_printf(m, "Zswapped: %8lu kB\n", - (unsigned long)atomic_read(&zswap_stored_pages) << + (unsigned long)atomic_long_read(&zswap_stored_pages) << (PAGE_SHIFT - 10)); #endif show_val_kb(m, "Dirty: ", diff --git a/include/linux/zswap.h b/include/linux/zswap.h index 9cd1beef0654..d961ead91bf1 100644 --- a/include/linux/zswap.h +++ b/include/linux/zswap.h @@ -7,7 +7,7 @@ =20 struct lruvec; =20 -extern atomic_t zswap_stored_pages; +extern atomic_long_t zswap_stored_pages; =20 #ifdef CONFIG_ZSWAP =20 diff --git a/mm/zswap.c b/mm/zswap.c index 69b9c025fd47..2b8da50f6322 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -43,7 +43,7 @@ * statistics **********************************/ /* The number of compressed pages currently stored in zswap */ -atomic_t zswap_stored_pages =3D ATOMIC_INIT(0); +atomic_long_t zswap_stored_pages =3D ATOMIC_INIT(0); =20 /* * The statistics below are not protected from concurrent access for @@ -802,7 +802,7 @@ static void zswap_entry_free(struct zswap_entry *entry) obj_cgroup_put(entry->objcg); } zswap_entry_cache_free(entry); - atomic_dec(&zswap_stored_pages); + atomic_long_dec(&zswap_stored_pages); } =20 /********************************* @@ -1232,7 +1232,7 @@ static unsigned long zswap_shrinker_count(struct shri= nker *shrinker, nr_stored =3D memcg_page_state(memcg, MEMCG_ZSWAPPED); } else { nr_backing =3D zswap_total_pages(); - nr_stored =3D atomic_read(&zswap_stored_pages); + nr_stored =3D atomic_long_read(&zswap_stored_pages); } =20 if (!nr_stored) @@ -1501,7 +1501,7 @@ bool zswap_store(struct folio *folio) } =20 /* update stats */ - atomic_inc(&zswap_stored_pages); + atomic_long_inc(&zswap_stored_pages); count_vm_event(ZSWPOUT); =20 return true; @@ -1650,6 +1650,13 @@ static int debugfs_get_total_size(void *data, u64 *v= al) } DEFINE_DEBUGFS_ATTRIBUTE(total_size_fops, debugfs_get_total_size, NULL, "%= llu\n"); =20 +static int debugfs_get_stored_pages(void *data, u64 *val) +{ + *val =3D atomic_long_read(&zswap_stored_pages); + return 0; +} +DEFINE_DEBUGFS_ATTRIBUTE(stored_pages_fops, debugfs_get_stored_pages, NULL= , "%llu\n"); + static int zswap_debugfs_init(void) { if (!debugfs_initialized()) @@ -1673,8 +1680,8 @@ static int zswap_debugfs_init(void) zswap_debugfs_root, &zswap_written_back_pages); debugfs_create_file("pool_total_size", 0444, zswap_debugfs_root, NULL, &total_size_fops); - debugfs_create_atomic_t("stored_pages", 0444, - zswap_debugfs_root, &zswap_stored_pages); + debugfs_create_file("stored_pages", 0444, + zswap_debugfs_root, NULL, &stored_pages_fops); =20 return 0; } --=20 2.27.0 From nobody Thu Nov 28 13:37:13 2024 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.11]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BB4DF1BBBED for ; Tue, 1 Oct 2024 05:37:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.11 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727761033; cv=none; b=KvDflvHnjjzvXxrb6m5Lk+ekW+9eNwP+ZoSpMR6/IpshcwYULqVsuq0XmLAYLh7WaBZHBc+HFCzuUNhqL3zGSBDqTOZSSZn/TTDdFg3XpPrZjWrOUVnXUovcnW2xWcMcHchkTX9YduV6/3mXaNpoVRWZGXYpk5G8jboGsGAdu8A= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727761033; c=relaxed/simple; bh=ZqA5lLeXDY0x6sdJphTljhIJY7azBNvVtOD2DnEXQcU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=gTtV+JH+LNwtkZ+IERzT0JxWtbmV4334alNw9t98M4akIN5Rv9thaWAnSb4rUE9CHR1kQUUZALI77El5B4zL+IczYN3E4VpBdbCfrHVAqVQuNkMDfTOj7rUSetJ2LBt4tA3S5i99CBUJ3zuV9gbTl6+dyDLbCBJrYDhs/8c49z0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=V012ApDo; arc=none smtp.client-ip=192.198.163.11 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="V012ApDo" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1727761032; x=1759297032; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=ZqA5lLeXDY0x6sdJphTljhIJY7azBNvVtOD2DnEXQcU=; b=V012ApDo7XyIRRqFZELvr5w0bMahnWBmgmlPq0/L4aQLuGR0wY4+DUU8 LpgpjwdkBu//kU6EB4FlhoRZYZKUuFaXp9PPhMdKDTBgbcudik9kE2bBC xkfhE/CC+Wx1Q0QgfWm4ROfVG28CNU5GFP/miXyH6kIUkVQRxXSPm28rW iFoTTA+iZZRP6MVMGoB1nCSUNmu5ykiEhlm/lGTIun2MarC9z0RWk6x6P d2KEKVjsnQesb9i3OkGMTTWl6GCBmucB9SmjfYMujkyWm8QKPWITSSHWs pQqA0lzGlMSbbkl6y42rdcsytjjIovlzk2ZQB2/HzYAxOQyqW718W7Vr4 Q==; X-CSE-ConnectionGUID: sK8J4JJ0Tw+cxjPY9N3CxA== X-CSE-MsgGUID: YKGNolaHRn+MX64WZ+ZVaw== X-IronPort-AV: E=McAfee;i="6700,10204,11211"; a="37465128" X-IronPort-AV: E=Sophos;i="6.11,167,1725346800"; d="scan'208";a="37465128" Received: from fmviesa007.fm.intel.com ([10.60.135.147]) by fmvoesa105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 30 Sep 2024 22:32:27 -0700 X-CSE-ConnectionGUID: fGhzmQz2RFu8X+pw/8tNSA== X-CSE-MsgGUID: 7dEULFhAQP6ZmQjTLj2l8A== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.11,167,1725346800"; d="scan'208";a="73205815" Received: from jf5300-b11a338t.jf.intel.com ([10.242.51.6]) by fmviesa007.fm.intel.com with ESMTP; 30 Sep 2024 22:32:26 -0700 From: Kanchana P Sridhar To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, hannes@cmpxchg.org, yosryahmed@google.com, nphamcs@gmail.com, chengming.zhou@linux.dev, usamaarif642@gmail.com, shakeel.butt@linux.dev, ryan.roberts@arm.com, ying.huang@intel.com, 21cnbao@gmail.com, akpm@linux-foundation.org, willy@infradead.org Cc: nanhai.zou@intel.com, wajdi.k.feghali@intel.com, vinodh.gopal@intel.com, kanchana.p.sridhar@intel.com Subject: [PATCH v10 6/7] mm: zswap: Support large folios in zswap_store(). Date: Mon, 30 Sep 2024 22:32:21 -0700 Message-Id: <20241001053222.6944-7-kanchana.p.sridhar@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20241001053222.6944-1-kanchana.p.sridhar@intel.com> References: <20241001053222.6944-1-kanchana.p.sridhar@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" zswap_store() will store large folios by compressing them page by page. This patch provides a sequential implementation of storing a large folio in zswap_store() by iterating through each page in the folio to compress and store it in the zswap zpool. zswap_store() calls the newly added zswap_store_page() function for each page in the folio. zswap_store_page() handles compressing and storing each page. We check the global and per-cgroup limits once at the beginning of zswap_store(), and only check that the limit is not reached yet. This is racy and inaccurate, but it should be sufficient for now. We also obtain initial references to the relevant objcg and pool to guarantee that subsequent references can be acquired by zswap_store_page(). A new function zswap_pool_get() is added to facilitate this. If these one-time checks pass, we compress the pages of the folio, while maintaining a running count of compressed bytes for all the folio's pages. If all pages are successfully compressed and stored, we do the cgroup zswap charging with the total compressed bytes, and batch update the zswap_stored_pages atomic/zswpout event stats with folio_nr_pages() once, before returning from zswap_store(). If an error is encountered during the store of any page in the folio, all pages in that folio currently stored in zswap will be invalidated. Thus, a folio is either entirely stored in zswap, or entirely not stored in zswap. The most important value provided by this patch is it enables swapping out large folios to zswap without splitting them. Furthermore, it batches some operations while doing so (cgroup charging, stats updates). This patch also forms the basis for building compress batching of pages in a large folio in zswap_store() by compressing up to say, 8 pages of the folio in parallel in hardware using the Intel In-Memory Analytics Accelerator (Intel IAA). This change reuses and adapts the functionality in Ryan Roberts' RFC patch [1]: "[RFC,v1] mm: zswap: Store large folios without splitting" [1] https://lore.kernel.org/linux-mm/20231019110543.3284654-1-ryan.robert= s@arm.com/T/#u Co-developed-by: Ryan Roberts Signed-off-by: Signed-off-by: Kanchana P Sridhar Acked-by: Johannes Weiner Acked-by: Yosry Ahmed Reviewed-by: Nhat Pham --- mm/zswap.c | 189 ++++++++++++++++++++++++++++++++++------------------- 1 file changed, 121 insertions(+), 68 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index 2b8da50f6322..09aaf70f95c6 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -411,6 +411,12 @@ static int __must_check zswap_pool_tryget(struct zswap= _pool *pool) return percpu_ref_tryget(&pool->ref); } =20 +/* The caller must already have a reference. */ +static void zswap_pool_get(struct zswap_pool *pool) +{ + percpu_ref_get(&pool->ref); +} + static void zswap_pool_put(struct zswap_pool *pool) { percpu_ref_put(&pool->ref); @@ -1402,68 +1408,38 @@ static void shrink_worker(struct work_struct *w) /********************************* * main API **********************************/ -bool zswap_store(struct folio *folio) + +static ssize_t zswap_store_page(struct page *page, + struct obj_cgroup *objcg, + struct zswap_pool *pool) { - swp_entry_t swp =3D folio->swap; - pgoff_t offset =3D swp_offset(swp); - struct xarray *tree =3D swap_zswap_tree(swp); struct zswap_entry *entry, *old; - struct obj_cgroup *objcg =3D NULL; - struct mem_cgroup *memcg =3D NULL; - - VM_WARN_ON_ONCE(!folio_test_locked(folio)); - VM_WARN_ON_ONCE(!folio_test_swapcache(folio)); - - /* Large folios aren't supported */ - if (folio_test_large(folio)) - return false; - - if (!zswap_enabled) - goto check_old; - - /* Check cgroup limits */ - objcg =3D get_obj_cgroup_from_folio(folio); - if (objcg && !obj_cgroup_may_zswap(objcg)) { - memcg =3D get_mem_cgroup_from_objcg(objcg); - if (shrink_memcg(memcg)) { - mem_cgroup_put(memcg); - goto reject; - } - mem_cgroup_put(memcg); - } - - if (zswap_check_limits()) - goto reject; =20 /* allocate entry */ - entry =3D zswap_entry_cache_alloc(GFP_KERNEL, folio_nid(folio)); + entry =3D zswap_entry_cache_alloc(GFP_KERNEL, page_to_nid(page)); if (!entry) { zswap_reject_kmemcache_fail++; goto reject; } =20 - /* if entry is successfully added, it keeps the reference */ - entry->pool =3D zswap_pool_current_get(); - if (!entry->pool) - goto freepage; + /* zswap_store() already holds a ref on 'objcg' and 'pool' */ + if (objcg) + obj_cgroup_get(objcg); + zswap_pool_get(pool); =20 - if (objcg) { - memcg =3D get_mem_cgroup_from_objcg(objcg); - if (memcg_list_lru_alloc(memcg, &zswap_list_lru, GFP_KERNEL)) { - mem_cgroup_put(memcg); - goto put_pool; - } - mem_cgroup_put(memcg); - } + /* if entry is successfully added, it keeps the reference */ + entry->pool =3D pool; =20 - if (!zswap_compress(&folio->page, entry)) - goto put_pool; + if (!zswap_compress(page, entry)) + goto put_pool_objcg; =20 - entry->swpentry =3D swp; + entry->swpentry =3D page_swap_entry(page); entry->objcg =3D objcg; entry->referenced =3D true; =20 - old =3D xa_store(tree, offset, entry, GFP_KERNEL); + old =3D xa_store(swap_zswap_tree(entry->swpentry), + swp_offset(entry->swpentry), + entry, GFP_KERNEL); if (xa_is_err(old)) { int err =3D xa_err(old); =20 @@ -1480,11 +1456,6 @@ bool zswap_store(struct folio *folio) if (old) zswap_entry_free(old); =20 - if (objcg) { - obj_cgroup_charge_zswap(objcg, entry->length); - count_objcg_events(objcg, ZSWPOUT, 1); - } - /* * We finish initializing the entry while it's already in xarray. * This is safe because: @@ -1500,32 +1471,114 @@ bool zswap_store(struct folio *folio) zswap_lru_add(&zswap_list_lru, entry); } =20 - /* update stats */ - atomic_long_inc(&zswap_stored_pages); - count_vm_event(ZSWPOUT); - - return true; + /* + * We shouldn't have any possibility of failure after the entry is + * added in the xarray. The pool/objcg refs obtained here will only + * be dropped if/when zswap_entry_free() gets called. + */ + return entry->length; =20 store_failed: zpool_free(entry->pool->zpool, entry->handle); -put_pool: - zswap_pool_put(entry->pool); -freepage: +put_pool_objcg: + zswap_pool_put(pool); + obj_cgroup_put(objcg); zswap_entry_cache_free(entry); reject: + return -EINVAL; +} + +bool zswap_store(struct folio *folio) +{ + long nr_pages =3D folio_nr_pages(folio); + swp_entry_t swp =3D folio->swap; + struct obj_cgroup *objcg =3D NULL; + struct mem_cgroup *memcg =3D NULL; + struct zswap_pool *pool; + size_t compressed_bytes =3D 0; + bool ret =3D false; + long index; + + VM_WARN_ON_ONCE(!folio_test_locked(folio)); + VM_WARN_ON_ONCE(!folio_test_swapcache(folio)); + + if (!zswap_enabled) + goto check_old; + + objcg =3D get_obj_cgroup_from_folio(folio); + if (objcg && !obj_cgroup_may_zswap(objcg)) { + memcg =3D get_mem_cgroup_from_objcg(objcg); + if (shrink_memcg(memcg)) { + mem_cgroup_put(memcg); + goto put_objcg; + } + mem_cgroup_put(memcg); + } + + if (zswap_check_limits()) + goto put_objcg; + + pool =3D zswap_pool_current_get(); + if (!pool) + goto put_objcg; + + if (objcg) { + memcg =3D get_mem_cgroup_from_objcg(objcg); + if (memcg_list_lru_alloc(memcg, &zswap_list_lru, GFP_KERNEL)) { + mem_cgroup_put(memcg); + goto put_pool; + } + mem_cgroup_put(memcg); + } + + for (index =3D 0; index < nr_pages; ++index) { + struct page *page =3D folio_page(folio, index); + ssize_t bytes; + + bytes =3D zswap_store_page(page, objcg, pool); + if (bytes < 0) + goto put_pool; + compressed_bytes +=3D bytes; + } + + if (objcg) { + obj_cgroup_charge_zswap(objcg, compressed_bytes); + count_objcg_events(objcg, ZSWPOUT, nr_pages); + } + + atomic_long_add(nr_pages, &zswap_stored_pages); + count_vm_events(ZSWPOUT, nr_pages); + + ret =3D true; + +put_pool: + zswap_pool_put(pool); +put_objcg: obj_cgroup_put(objcg); - if (zswap_pool_reached_full) + if (!ret && zswap_pool_reached_full) queue_work(shrink_wq, &zswap_shrink_work); check_old: /* - * If the zswap store fails or zswap is disabled, we must invalidate the - * possibly stale entry which was previously stored at this offset. - * Otherwise, writeback could overwrite the new data in the swapfile. + * If the zswap store fails or zswap is disabled, we must invalidate + * the possibly stale entries which were previously stored at the + * offsets corresponding to each page of the folio. Otherwise, + * writeback could overwrite the new data in the swapfile. */ - entry =3D xa_erase(tree, offset); - if (entry) - zswap_entry_free(entry); - return false; + if (!ret) { + unsigned type =3D swp_type(swp); + pgoff_t offset =3D swp_offset(swp); + struct zswap_entry *entry; + struct xarray *tree; + + for (index =3D 0; index < nr_pages; ++index) { + tree =3D swap_zswap_tree(swp_entry(type, offset + index)); + entry =3D xa_erase(tree, offset + index); + if (entry) + zswap_entry_free(entry); + } + } + + return ret; } =20 bool zswap_load(struct folio *folio) --=20 2.27.0 From nobody Thu Nov 28 13:37:13 2024 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.11]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E2FD11BBBF4 for ; Tue, 1 Oct 2024 05:37:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.11 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727761033; cv=none; b=Lfy5DDLXF9h+gZH0HJ+Nx/XAi1DyGwa1kmm3N2HyOMTiupahAiNJicYLa3BUdo3/1QKG/ql8cFyqGo2BCuQ4tnxXhAZE8rjLqmO9A9iTvRqSDNJyp5FBYb7fZFc1czxAEqJ1+7wA+rFQmthRT1bkDUOH7zdJ9JcQ/+OedUyxf1s= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727761033; c=relaxed/simple; bh=EoDLTcNBMro+ufgrjaTKFoCYNP0vIAE8K0H1FKjnXKw=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=XTCpX04CWPLIZbfeJ8DKAiTUsbrDEIWyMOFtAzSPwPzdjMM4KhXJ1e+4jrrw42jSOQc858ZwqlMG3UogF4fnI5loEmdi+D3Cqwwzw4JnjX2snvU9zoVHlC1HROnZM+03OyB8Eh0PjU+msEHEb/K5xBoK8XGWfMIMrdbTPwKC9As= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=f08IuqiM; arc=none smtp.client-ip=192.198.163.11 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="f08IuqiM" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1727761032; x=1759297032; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=EoDLTcNBMro+ufgrjaTKFoCYNP0vIAE8K0H1FKjnXKw=; b=f08IuqiMPbTqOQM/tc735SQ1f7dAbqIHD/fdaAFQ4l9Z5M9ZQSM6skoX VjIBk8ymRC1b21A/lmoxjdZvVGc+d4SZWK+WCQ2Ag8A4FAeuFabykNqJ+ Q30FmZ4uf+k2608Tk5ABJ+jgqu/qHbGHSzi4beCnZZgixjNS51gn9jyed CQHuLhpgaPfqeJhHnbz4Q+Bie+SGP2/IOKyeIIkBdft3LoOGeXy8Qva5A /uuc08es8yUlQd92YpBnAjpDOHF70Km3Ondfg4fVToxPhFo1YqnBRl8cg 3Upnr9uk5Cx1TRHIoXgL4kN1R29dtqyarr5IFXeyOairgWRfl8NdWXFSn g==; X-CSE-ConnectionGUID: R5S1NpZ6TuySY8St/JBkJQ== X-CSE-MsgGUID: utJADqsHSxWtvBVKVqCTqA== X-IronPort-AV: E=McAfee;i="6700,10204,11211"; a="37465139" X-IronPort-AV: E=Sophos;i="6.11,167,1725346800"; d="scan'208";a="37465139" Received: from fmviesa007.fm.intel.com ([10.60.135.147]) by fmvoesa105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 30 Sep 2024 22:32:27 -0700 X-CSE-ConnectionGUID: yFJuADSqS4eTxH7oBtWwig== X-CSE-MsgGUID: HpaMM9bDQFOGdwAg5THVcQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.11,167,1725346800"; d="scan'208";a="73205821" Received: from jf5300-b11a338t.jf.intel.com ([10.242.51.6]) by fmviesa007.fm.intel.com with ESMTP; 30 Sep 2024 22:32:27 -0700 From: Kanchana P Sridhar To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, hannes@cmpxchg.org, yosryahmed@google.com, nphamcs@gmail.com, chengming.zhou@linux.dev, usamaarif642@gmail.com, shakeel.butt@linux.dev, ryan.roberts@arm.com, ying.huang@intel.com, 21cnbao@gmail.com, akpm@linux-foundation.org, willy@infradead.org Cc: nanhai.zou@intel.com, wajdi.k.feghali@intel.com, vinodh.gopal@intel.com, kanchana.p.sridhar@intel.com Subject: [PATCH v10 7/7] mm: swap: Count successful large folio zswap stores in hugepage zswpout stats. Date: Mon, 30 Sep 2024 22:32:22 -0700 Message-Id: <20241001053222.6944-8-kanchana.p.sridhar@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20241001053222.6944-1-kanchana.p.sridhar@intel.com> References: <20241001053222.6944-1-kanchana.p.sridhar@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Added a new MTHP_STAT_ZSWPOUT entry to the sysfs transparent_hugepage stats so that successful large folio zswap stores can be accounted under the per-order sysfs "zswpout" stats: /sys/kernel/mm/transparent_hugepage/hugepages-*kB/stats/zswpout Other non-zswap swap device swap-out events will be counted under the existing sysfs "swpout" stats: /sys/kernel/mm/transparent_hugepage/hugepages-*kB/stats/swpout Also, added documentation for the newly added sysfs per-order hugepage "zswpout" stats. The documentation clarifies that only non-zswap swapouts will be accounted in the existing "swpout" stats. Signed-off-by: Kanchana P Sridhar Reviewed-by: Nhat Pham --- Documentation/admin-guide/mm/transhuge.rst | 8 ++++++-- include/linux/huge_mm.h | 1 + mm/huge_memory.c | 3 +++ mm/page_io.c | 1 + 4 files changed, 11 insertions(+), 2 deletions(-) diff --git a/Documentation/admin-guide/mm/transhuge.rst b/Documentation/adm= in-guide/mm/transhuge.rst index cfdd16a52e39..2a171ed5206e 100644 --- a/Documentation/admin-guide/mm/transhuge.rst +++ b/Documentation/admin-guide/mm/transhuge.rst @@ -530,10 +530,14 @@ anon_fault_fallback_charge instead falls back to using huge pages with lower orders or small pages even though the allocation was successful. =20 -swpout - is incremented every time a huge page is swapped out in one +zswpout + is incremented every time a huge page is swapped out to zswap in one piece without splitting. =20 +swpout + is incremented every time a huge page is swapped out to a non-zswap + swap device in one piece without splitting. + swpout_fallback is incremented if a huge page has to be split before swapout. Usually because failed to allocate some continuous swap space diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index 5eb4b0376c7d..3eca60f3d512 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -119,6 +119,7 @@ enum mthp_stat_item { MTHP_STAT_ANON_FAULT_ALLOC, MTHP_STAT_ANON_FAULT_FALLBACK, MTHP_STAT_ANON_FAULT_FALLBACK_CHARGE, + MTHP_STAT_ZSWPOUT, MTHP_STAT_SWPOUT, MTHP_STAT_SWPOUT_FALLBACK, MTHP_STAT_SHMEM_ALLOC, diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 243c15912105..a7b05f4c2a5e 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -611,6 +611,7 @@ static struct kobj_attribute _name##_attr =3D __ATTR_RO= (_name) DEFINE_MTHP_STAT_ATTR(anon_fault_alloc, MTHP_STAT_ANON_FAULT_ALLOC); DEFINE_MTHP_STAT_ATTR(anon_fault_fallback, MTHP_STAT_ANON_FAULT_FALLBACK); DEFINE_MTHP_STAT_ATTR(anon_fault_fallback_charge, MTHP_STAT_ANON_FAULT_FAL= LBACK_CHARGE); +DEFINE_MTHP_STAT_ATTR(zswpout, MTHP_STAT_ZSWPOUT); DEFINE_MTHP_STAT_ATTR(swpout, MTHP_STAT_SWPOUT); DEFINE_MTHP_STAT_ATTR(swpout_fallback, MTHP_STAT_SWPOUT_FALLBACK); #ifdef CONFIG_SHMEM @@ -629,6 +630,7 @@ static struct attribute *anon_stats_attrs[] =3D { &anon_fault_fallback_attr.attr, &anon_fault_fallback_charge_attr.attr, #ifndef CONFIG_SHMEM + &zswpout_attr.attr, &swpout_attr.attr, &swpout_fallback_attr.attr, #endif @@ -659,6 +661,7 @@ static struct attribute_group file_stats_attr_grp =3D { =20 static struct attribute *any_stats_attrs[] =3D { #ifdef CONFIG_SHMEM + &zswpout_attr.attr, &swpout_attr.attr, &swpout_fallback_attr.attr, #endif diff --git a/mm/page_io.c b/mm/page_io.c index bc1183299a7d..4aa34862676f 100644 --- a/mm/page_io.c +++ b/mm/page_io.c @@ -269,6 +269,7 @@ int swap_writepage(struct page *page, struct writeback_= control *wbc) swap_zeromap_folio_clear(folio); } if (zswap_store(folio)) { + count_mthp_stat(folio_order(folio), MTHP_STAT_ZSWPOUT); folio_unlock(folio); return 0; } --=20 2.27.0