From nobody Fri Nov 29 05:51:58 2024 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 11D40184E for ; Tue, 24 Sep 2024 01:17:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.12 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727140635; cv=none; b=tUZhqD1hPaD8dakS0xtFBRfp9NoHgdhNq5JTZHpgJYA48fbrNNYgwdtE3IkXUTZMkzHqsWybDdbRYumAVN4S6kYeWwqBeX1gzheKXzxB25YPHpT1FomlYnbdLBbTuASwlee5Duz6H4R6Vgdc53KhfbQxxglB+bFheBTpunUtzvE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727140635; c=relaxed/simple; bh=j6MRHvvdb4Y/ecSm2gb8TOi+Txe+Ci0dZeAdkAYjrOU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Eb1eAxDGV4NavT6ze7kONrwpV+2WQ3hVORP8VBlAsmZvqyfG5swJJBjoYicPRJPYK5yC76dzVIm3Ei+6zHsUNX2UOZvAuC9OI7UCbt3m3YJSRI1ux0I8j1sCcORcEjaFzobLi1FzNnVsLU8ZaeJTuazrlO9pX52nh8zCgmxvNIM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=bkVO02xn; arc=none smtp.client-ip=192.198.163.12 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="bkVO02xn" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1727140634; x=1758676634; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=j6MRHvvdb4Y/ecSm2gb8TOi+Txe+Ci0dZeAdkAYjrOU=; b=bkVO02xnEKx4CKZa8XokIOwGGEOdu396tgEUmYpH8TnOJZrOYcNe3CzZ X4cfB4zIGWHU6hWGZXY98/wR8WIpvHqdUPkbpsYXxHbyv++yruG6Fx7fe p5QNqCjVzhes7Px0w8hV+LPM/zEGL2EWNOMDE3CUCOsFXIGOItuuSx37q C6qR4zYoKIpjYASbQ/LAXABw4/hksaChATfuc+HkOox+HSuyjgXGTTdd6 qytcHpR2lLWAbK7BfFCOKwjzu6jik7xosZtuOqMotI/Se9x/WDKDdXhJ3 Vm90jaJ08l9iO3VptpBSS7ijwWJ7k38WoQaQC2rA/xKjyPKgyrmEDrzlp A==; X-CSE-ConnectionGUID: f8gFNQdLR3qyH+SAqaclHA== X-CSE-MsgGUID: RDhfqr4YSKec2upX7LJ39w== X-IronPort-AV: E=McAfee;i="6700,10204,11204"; a="30001998" X-IronPort-AV: E=Sophos;i="6.10,253,1719903600"; d="scan'208";a="30001998" Received: from orviesa001.jf.intel.com ([10.64.159.141]) by fmvoesa106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Sep 2024 18:17:10 -0700 X-CSE-ConnectionGUID: im59xWO4QgiDbIGKgbJwwQ== X-CSE-MsgGUID: mRXKs5anS0CqWg+gbdX0Xw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.10,253,1719903600"; d="scan'208";a="108688443" Received: from jf5300-b11a338t.jf.intel.com ([10.242.51.6]) by orviesa001.jf.intel.com with ESMTP; 23 Sep 2024 18:17:10 -0700 From: Kanchana P Sridhar To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, hannes@cmpxchg.org, yosryahmed@google.com, nphamcs@gmail.com, chengming.zhou@linux.dev, usamaarif642@gmail.com, shakeel.butt@linux.dev, ryan.roberts@arm.com, ying.huang@intel.com, 21cnbao@gmail.com, akpm@linux-foundation.org Cc: nanhai.zou@intel.com, wajdi.k.feghali@intel.com, vinodh.gopal@intel.com, kanchana.p.sridhar@intel.com Subject: [PATCH v7 1/8] mm: Define obj_cgroup_get() if CONFIG_MEMCG is not defined. Date: Mon, 23 Sep 2024 18:17:02 -0700 Message-Id: <20240924011709.7037-2-kanchana.p.sridhar@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20240924011709.7037-1-kanchana.p.sridhar@intel.com> References: <20240924011709.7037-1-kanchana.p.sridhar@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This resolves an issue with obj_cgroup_get() not being defined if CONFIG_MEMCG is not defined. Before this patch, we would see build errors if obj_cgroup_get() is called from code that is agnostic of CONFIG_MEMCG. The zswap_store() changes for mTHP in subsequent commits will require the use of obj_cgroup_get() in zswap code that falls into this category. Signed-off-by: Kanchana P Sridhar Reviewed-by: Nhat Pham --- include/linux/memcontrol.h | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 34d2da05f2f1..15c2716f9aa3 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1282,6 +1282,10 @@ struct mem_cgroup *mem_cgroup_from_css(struct cgroup= _subsys_state *css) return NULL; } =20 +static inline void obj_cgroup_get(struct obj_cgroup *objcg) +{ +} + static inline void obj_cgroup_put(struct obj_cgroup *objcg) { } --=20 2.27.0 From nobody Fri Nov 29 05:51:58 2024 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 67D8615C0 for ; Tue, 24 Sep 2024 01:17:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.12 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727140633; cv=none; b=NICjFwWS76nXXksVuDTGUfHsfjGIYC5WphEuae+AV7DXSrCxjIVX4CpvgQYEbztLVvpwFITay8FBtNLBo6gQGJlyD/fh8a4hnss2pTrXD8VRXkwV8G/EiHtyHr47On+OiYOMD9yPD8pDL0/29Zss7yIj0l7+oPSMMYJ48GwbExI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727140633; c=relaxed/simple; bh=pUUdV92IiG6zC8GNm5xDbVEXW5oShMyaugmREkMxm7k=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=bufcfVTNO/yd/hPwsSBHgwgi9yTyPePic6yqGRYBgxlqFO56jZAtLmzYfxHzdPiQr+t4Z+b/91fP4lfOsp5VzKg6qKYS79b0rZ/FqTkMHjpch4ULkako0rGGZYfYnkC2rWIYO+0b5QqOyRox8mLT2FZgIA+lBeTXzJNreDMuSG4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=DH+sUafW; arc=none smtp.client-ip=192.198.163.12 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="DH+sUafW" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1727140632; x=1758676632; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=pUUdV92IiG6zC8GNm5xDbVEXW5oShMyaugmREkMxm7k=; b=DH+sUafWH9CysPvlcT8N0UO2cMJSIwIvzM7DQ1gVA/2D1snM6oZBvp9g yZ0lDs2P0dlcVfPqIGCelfNi9TynkxAbvAxiNaMfZXwuXJ9LJKcBWuNTu ikEH6MCc+jcc6GtkkUSsrYcBSN4H2lxE5vrmSp+UR6PiZogDgSa77B3wA OtLB+JOoIfD9xpF8/5nl/cB2c4q58/6XG+xNkXkgWjxB/+d5J7zqsnvoT bYXW0FSKg1l2Rk49SynAq31dfN6/x559UjdQ07xvlCSnKcU+UnDrmsoyj ohBAoFMi9qKi1F32DScQlevABYyPfHOfbCBqzb+aXQTrrLcQmsRJhiw6E Q==; X-CSE-ConnectionGUID: mxAZTHilTKibq7Nm/HTHLA== X-CSE-MsgGUID: GhNIty5dR12G4DpB9cx5ZQ== X-IronPort-AV: E=McAfee;i="6700,10204,11204"; a="30002006" X-IronPort-AV: E=Sophos;i="6.10,253,1719903600"; d="scan'208";a="30002006" Received: from orviesa001.jf.intel.com ([10.64.159.141]) by fmvoesa106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Sep 2024 18:17:10 -0700 X-CSE-ConnectionGUID: Ow60v4ymQFG6hfj4EMYAUA== X-CSE-MsgGUID: HuTf+8mJT82ukI2RZ+87ug== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.10,253,1719903600"; d="scan'208";a="108688447" Received: from jf5300-b11a338t.jf.intel.com ([10.242.51.6]) by orviesa001.jf.intel.com with ESMTP; 23 Sep 2024 18:17:10 -0700 From: Kanchana P Sridhar To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, hannes@cmpxchg.org, yosryahmed@google.com, nphamcs@gmail.com, chengming.zhou@linux.dev, usamaarif642@gmail.com, shakeel.butt@linux.dev, ryan.roberts@arm.com, ying.huang@intel.com, 21cnbao@gmail.com, akpm@linux-foundation.org Cc: nanhai.zou@intel.com, wajdi.k.feghali@intel.com, vinodh.gopal@intel.com, kanchana.p.sridhar@intel.com Subject: [PATCH v7 2/8] mm: zswap: Modify zswap_compress() to accept a page instead of a folio. Date: Mon, 23 Sep 2024 18:17:03 -0700 Message-Id: <20240924011709.7037-3-kanchana.p.sridhar@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20240924011709.7037-1-kanchana.p.sridhar@intel.com> References: <20240924011709.7037-1-kanchana.p.sridhar@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" For zswap_store() to be able to store an mTHP by compressing it one page at a time, zswap_compress() needs to accept a page as input. This will allow us to iterate through each page in the mTHP in zswap_store(), compress it and store it in the zpool. Signed-off-by: Kanchana P Sridhar Reviewed-by: Nhat Pham --- mm/zswap.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index 449914ea9919..59b7733a62d3 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -876,7 +876,7 @@ static int zswap_cpu_comp_dead(unsigned int cpu, struct= hlist_node *node) return 0; } =20 -static bool zswap_compress(struct folio *folio, struct zswap_entry *entry) +static bool zswap_compress(struct page *page, struct zswap_entry *entry) { struct crypto_acomp_ctx *acomp_ctx; struct scatterlist input, output; @@ -894,7 +894,7 @@ static bool zswap_compress(struct folio *folio, struct = zswap_entry *entry) =20 dst =3D acomp_ctx->buffer; sg_init_table(&input, 1); - sg_set_folio(&input, folio, PAGE_SIZE, 0); + sg_set_page(&input, page, PAGE_SIZE, 0); =20 /* * We need PAGE_SIZE * 2 here since there maybe over-compression case, @@ -1458,7 +1458,7 @@ bool zswap_store(struct folio *folio) mem_cgroup_put(memcg); } =20 - if (!zswap_compress(folio, entry)) + if (!zswap_compress(&folio->page, entry)) goto put_pool; =20 entry->swpentry =3D swp; --=20 2.27.0 From nobody Fri Nov 29 05:51:58 2024 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 75C0D1B85DA for ; Tue, 24 Sep 2024 01:17:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.12 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727140636; cv=none; b=FgwIxZwJO8ZOLVFQG3DqVyK1UusDnelUN7Gvl+pIecjEm+HvcEXQJYFBdK0jOZoDTSR+eeBv9zkSd5/UX9OzdzWeK/gO57lMo84hG/qPkPjpB6QISqzYv7LdLXmTn7lfriqSMPEPy9ujrmyFDK32d909K4WqZOsGQWg+90MaELY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727140636; c=relaxed/simple; bh=RHgitxCwk2O92AmxepTIkYLlPlxcPNy3AMLz8p6gNWk=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=dRQZbnlbcHkuq9h9jTwuy+p+FeykM4JyslJGetLcvvTckKUFf5ARTWafIvyLNv4O5xnKBeGSL83j2wfv8Yp4BGSvFxgZSMzuZbHqWr4B83HbYE9E0zp4ZTUoEvPaev2uGjAc8f6jOoUNKwY7xto9/mF/vxbmuZigmzdvAi+GPss= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=EwRYuZ7L; arc=none smtp.client-ip=192.198.163.12 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="EwRYuZ7L" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1727140634; x=1758676634; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=RHgitxCwk2O92AmxepTIkYLlPlxcPNy3AMLz8p6gNWk=; b=EwRYuZ7LSiKPpD0zQBy/fHApoRfncwr3prrlZJYv5yEfgMfaO9yjX/ua urwiMO4VQDHA8tKrf3BYB8Z9kuAFrvtyT3k1G+Kn1sRjMJafHWlkHBrCT Eyg7fnFHpIDevacTciWM/i1UQmc24/sd9f4qHh1YmZiEgf9GE7ThJ1F7R 1ocHYzX8xRLCW7C/DP39iRqZB/yJUDV4y//M+5qAliD8w1hJfSjAWRwQv DgKIH/+cT8Ujkfu2/4sobwGT15p4N9HFbCaYfVJbNzNXY9+lPJJdcfred MHfgR2FBQTuwhau8DT9PNSC3QHfD/ZM0TP2bbIExp5Fegi8RfO+k5E0wq g==; X-CSE-ConnectionGUID: C3d2sTaySme0YMDci4tuxw== X-CSE-MsgGUID: +gwGu9dBR4uvw72NRwhLZA== X-IronPort-AV: E=McAfee;i="6700,10204,11204"; a="30002014" X-IronPort-AV: E=Sophos;i="6.10,253,1719903600"; d="scan'208";a="30002014" Received: from orviesa001.jf.intel.com ([10.64.159.141]) by fmvoesa106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Sep 2024 18:17:10 -0700 X-CSE-ConnectionGUID: l8/bok0JQDenH8syzM42bQ== X-CSE-MsgGUID: 3+hniOHXQ32cgOpTEREr0A== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.10,253,1719903600"; d="scan'208";a="108688451" Received: from jf5300-b11a338t.jf.intel.com ([10.242.51.6]) by orviesa001.jf.intel.com with ESMTP; 23 Sep 2024 18:17:10 -0700 From: Kanchana P Sridhar To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, hannes@cmpxchg.org, yosryahmed@google.com, nphamcs@gmail.com, chengming.zhou@linux.dev, usamaarif642@gmail.com, shakeel.butt@linux.dev, ryan.roberts@arm.com, ying.huang@intel.com, 21cnbao@gmail.com, akpm@linux-foundation.org Cc: nanhai.zou@intel.com, wajdi.k.feghali@intel.com, vinodh.gopal@intel.com, kanchana.p.sridhar@intel.com Subject: [PATCH v7 3/8] mm: zswap: Refactor code to store an entry in zswap xarray. Date: Mon, 23 Sep 2024 18:17:04 -0700 Message-Id: <20240924011709.7037-4-kanchana.p.sridhar@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20240924011709.7037-1-kanchana.p.sridhar@intel.com> References: <20240924011709.7037-1-kanchana.p.sridhar@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Added a new procedure zswap_store_entry() that refactors the code currently in zswap_store() to store an entry in the zswap xarray. This will allow us to call this procedure for each storing the swap offset of each page in an mTHP in the xarray, as part of zswap_store() supporting mTHP. Also, made a minor edit in the comments for 'struct zswap_entry' to delete the description of the 'value' member that was deleted in commit 20a5532ffa53d6ecf41ded920a7b0ff9c65a7dcf ("mm: remove code to handle same filled pages"). Signed-off-by: Kanchana P Sridhar Reviewed-by: Nhat Pham --- mm/zswap.c | 51 ++++++++++++++++++++++++++++++++++----------------- 1 file changed, 34 insertions(+), 17 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index 59b7733a62d3..fd35a81b6e36 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -190,7 +190,6 @@ static struct shrinker *zswap_shrinker; * section for context. * pool - the zswap_pool the entry's data is in * handle - zpool allocation handle that stores the compressed page data - * value - value of the same-value filled pages which have same content * objcg - the obj_cgroup that the compressed memory is charged to * lru - handle to the pool's lru used to evict pages. */ @@ -1404,12 +1403,44 @@ static void shrink_worker(struct work_struct *w) /********************************* * main API **********************************/ + +/* + * Returns true if the entry was successfully + * stored in the xarray, and false otherwise. + */ +static bool zswap_store_entry(struct xarray *tree, + struct zswap_entry *entry) +{ + struct zswap_entry *old; + pgoff_t offset =3D swp_offset(entry->swpentry); + + old =3D xa_store(tree, offset, entry, GFP_KERNEL); + + if (xa_is_err(old)) { + int err =3D xa_err(old); + + WARN_ONCE(err !=3D -ENOMEM, "unexpected xarray error: %d\n", err); + zswap_reject_alloc_fail++; + return false; + } + + /* + * We may have had an existing entry that became stale when + * the folio was redirtied and now the new version is being + * swapped out. Get rid of the old. + */ + if (old) + zswap_entry_free(old); + + return true; +} + bool zswap_store(struct folio *folio) { swp_entry_t swp =3D folio->swap; pgoff_t offset =3D swp_offset(swp); struct xarray *tree =3D swap_zswap_tree(swp); - struct zswap_entry *entry, *old; + struct zswap_entry *entry; struct obj_cgroup *objcg =3D NULL; struct mem_cgroup *memcg =3D NULL; =20 @@ -1465,22 +1496,8 @@ bool zswap_store(struct folio *folio) entry->objcg =3D objcg; entry->referenced =3D true; =20 - old =3D xa_store(tree, offset, entry, GFP_KERNEL); - if (xa_is_err(old)) { - int err =3D xa_err(old); - - WARN_ONCE(err !=3D -ENOMEM, "unexpected xarray error: %d\n", err); - zswap_reject_alloc_fail++; + if (!zswap_store_entry(tree, entry)) goto store_failed; - } - - /* - * We may have had an existing entry that became stale when - * the folio was redirtied and now the new version is being - * swapped out. Get rid of the old. - */ - if (old) - zswap_entry_free(old); =20 if (objcg) { obj_cgroup_charge_zswap(objcg, entry->length); --=20 2.27.0 From nobody Fri Nov 29 05:51:58 2024 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C79FD4C92 for ; Tue, 24 Sep 2024 01:17:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.12 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727140636; cv=none; b=MPzsWCPhPEIjPwtaXPEOVBZ5f47vsbnNzDPyE9+1k8oo9nUkfgkyjHqbSNQn0r0wbHfpmMSinOaggzhzNdZK8YPr7hJPqb65QtwaAHBP36MNspaiVSZ2AYb5LIBnu9dXxpO0bA1rbojpV4n5pijJ0epD5EvrZi7nPci62GhI+Mg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727140636; c=relaxed/simple; bh=C86so49XKpWlsyU2wpioV+UDdpE1jz5mE0FAlDLAhFE=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=l17OQwm1wGcvdtNUbTa11BN5E5TMAdseALzJXrzTTCScE75ZDOCz7SEO1jcz7bYQmd/Ewdobu92seggWcREm5rWuSdXwwxS/7IN+SDTAZXy5BcBueuqoocF6k20EX/jgwVfgzuvSwEmi3ImsAlBqeq3dJ8fxMI+RUOac4L6yfHQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=PgF32FDr; arc=none smtp.client-ip=192.198.163.12 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="PgF32FDr" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1727140634; x=1758676634; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=C86so49XKpWlsyU2wpioV+UDdpE1jz5mE0FAlDLAhFE=; b=PgF32FDr0FJfHmTcIVOwoLtCxm0oUTa1cvNk8ZJ/Zsin5Z9mz0XuvGdj qanr/oe7vAkU6ZjA+iz/qCvyWRi4T0iTjILHDz0nAl9PKNO9s4aboEY5w p4TZFKuxSoOroOa9zWcdznZkJ6ZirQMM0/qKUb22xg+vxFl8U4ua63L2Z OKfbPp1Zn5v1Iv0XAiPLeSGPsAzWnaFU4QNdrOB4ryLaN7c3jBRbT3Ai1 xnPp8IEP7ZL1HHxFXDuLoIhVu+jXchKk8mV58rVVFwFPtBQ7/oR9+JJgq AKptFKr13KrfyoJlhixl7c62Acy9pw4gS4liGD5xai6b+Uta4PoA9PuqF w==; X-CSE-ConnectionGUID: OfNFwrFDRdWdsEmk6NjnQQ== X-CSE-MsgGUID: IRHQzJT/Q6yS31ZVE10WuQ== X-IronPort-AV: E=McAfee;i="6700,10204,11204"; a="30002021" X-IronPort-AV: E=Sophos;i="6.10,253,1719903600"; d="scan'208";a="30002021" Received: from orviesa001.jf.intel.com ([10.64.159.141]) by fmvoesa106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Sep 2024 18:17:11 -0700 X-CSE-ConnectionGUID: sEGsfpQ9TySxYR5zKcbTVw== X-CSE-MsgGUID: jSs+zcAeQrCa4Hib2fFedw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.10,253,1719903600"; d="scan'208";a="108688454" Received: from jf5300-b11a338t.jf.intel.com ([10.242.51.6]) by orviesa001.jf.intel.com with ESMTP; 23 Sep 2024 18:17:10 -0700 From: Kanchana P Sridhar To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, hannes@cmpxchg.org, yosryahmed@google.com, nphamcs@gmail.com, chengming.zhou@linux.dev, usamaarif642@gmail.com, shakeel.butt@linux.dev, ryan.roberts@arm.com, ying.huang@intel.com, 21cnbao@gmail.com, akpm@linux-foundation.org Cc: nanhai.zou@intel.com, wajdi.k.feghali@intel.com, vinodh.gopal@intel.com, kanchana.p.sridhar@intel.com Subject: [PATCH v7 4/8] mm: zswap: Refactor code to delete stored offsets in case of errors. Date: Mon, 23 Sep 2024 18:17:05 -0700 Message-Id: <20240924011709.7037-5-kanchana.p.sridhar@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20240924011709.7037-1-kanchana.p.sridhar@intel.com> References: <20240924011709.7037-1-kanchana.p.sridhar@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Added a new procedure zswap_delete_stored_offsets() that can be called to delete stored offsets in a folio in case zswap_store() fails or zswap is disabled. Refactored the code in zswap_store() that handles these cases, to call zswap_delete_stored_offsets(). Signed-off-by: Kanchana P Sridhar --- mm/zswap.c | 33 ++++++++++++++++++++++++++++++--- 1 file changed, 30 insertions(+), 3 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index fd35a81b6e36..9bea948d653e 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -1435,8 +1435,37 @@ static bool zswap_store_entry(struct xarray *tree, return true; } =20 +/* + * If the zswap store fails or zswap is disabled, we must invalidate the + * possibly stale entries which were previously stored at the offsets + * corresponding to each page of the folio. Otherwise, writeback could + * overwrite the new data in the swapfile. + * + * This is called after the store of an offset in a large folio has failed. + * All zswap entries in the folio must be deleted. This helps make sure + * that a swapped-out mTHP is either entirely stored in zswap, or entirely + * not stored in zswap. + * + * This is also called if zswap_store() is invoked, but zswap is not enabl= ed. + * All offsets for the folio are deleted from zswap in this case. + */ +static void zswap_delete_stored_offsets(struct xarray *tree, + pgoff_t offset, + long nr_pages) +{ + struct zswap_entry *entry; + long i; + + for (i =3D 0; i < nr_pages; ++i) { + entry =3D xa_erase(tree, offset + i); + if (entry) + zswap_entry_free(entry); + } +} + bool zswap_store(struct folio *folio) { + long nr_pages =3D folio_nr_pages(folio); swp_entry_t swp =3D folio->swap; pgoff_t offset =3D swp_offset(swp); struct xarray *tree =3D swap_zswap_tree(swp); @@ -1541,9 +1570,7 @@ bool zswap_store(struct folio *folio) * possibly stale entry which was previously stored at this offset. * Otherwise, writeback could overwrite the new data in the swapfile. */ - entry =3D xa_erase(tree, offset); - if (entry) - zswap_entry_free(entry); + zswap_delete_stored_offsets(tree, offset, nr_pages); return false; } =20 --=20 2.27.0 From nobody Fri Nov 29 05:51:58 2024 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C03378C1F for ; Tue, 24 Sep 2024 01:17:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.12 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727140637; cv=none; b=UEjpofydH7qggXRnjf0ZOZR+HGdLmSJ7+oxDqIv0/m2ycpnG3AIXabjw1K2GkoaTXMA7/cPXzGqHQpyYOY+rPRfN99EAQZ+Xxzzh3sa1pkrOd5EcAZ9ehSSEeKqTvbYQHU7NvzKnicc6mB4lnGaLjNGNNXZ+1E6cI5JtybVS6po= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727140637; c=relaxed/simple; bh=2mNeQTWJUlWcoSNPRsAXaDiBYPB5nqJS0eAJPgQB/Sk=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=dovguDgRYGgose89cXXwF+dXTIHrIWe1QCfhUaeas898xdz+n0pnupBC5X5wAN2Q9cSRwURWswZ74nV4XavHKTkRKDioJJ5+wIb/icmT03wOY5NC0icV6mHs6U+SaxK4+FqjUtl3v8OMeFcxsggoyXAFce8AcgsQCgcPqGEJ03Q= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=Y9cpEceA; arc=none smtp.client-ip=192.198.163.12 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="Y9cpEceA" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1727140635; x=1758676635; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=2mNeQTWJUlWcoSNPRsAXaDiBYPB5nqJS0eAJPgQB/Sk=; b=Y9cpEceAFeDdjyqpGqw6++5yVBVK1XGCFsM6N1F+Se5+Zq+jSfeSOV8z hrIGwTYj5HeutCYA1EkoCn9nSlcspgbOfEcmwAvQh6zvxrmeVXZYHzBld bUnNEprjNcV1cDQYAlS2tVB7nnlg2vEp2KAOG3AF5Iwv6WeQR0ZLS5LVs OXGqyQP1QBgcNjLx5YkD1ivmsQcSw3CZfBwzHyoICZUuBSNm1+gQbAkUc P3swWGLcyH9RyLevzYwUWRuaAqbtEumRGnzUEax1LZIMfFiUJwC92VoGU gXpO9lHosTbffnmuU+14cwCfBANAeh6vP6ieE7q01kvNm04/lXVXWz9zj A==; X-CSE-ConnectionGUID: MMaIldbxQvuTiZoBlkF0sw== X-CSE-MsgGUID: RAkJsktxTRGhow8uLSr1NQ== X-IronPort-AV: E=McAfee;i="6700,10204,11204"; a="30002030" X-IronPort-AV: E=Sophos;i="6.10,253,1719903600"; d="scan'208";a="30002030" Received: from orviesa001.jf.intel.com ([10.64.159.141]) by fmvoesa106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Sep 2024 18:17:11 -0700 X-CSE-ConnectionGUID: e+1Xguk5Qg+HYwyEEGg18Q== X-CSE-MsgGUID: zCaUK/6kQ9uT9pt7psyHcA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.10,253,1719903600"; d="scan'208";a="108688457" Received: from jf5300-b11a338t.jf.intel.com ([10.242.51.6]) by orviesa001.jf.intel.com with ESMTP; 23 Sep 2024 18:17:10 -0700 From: Kanchana P Sridhar To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, hannes@cmpxchg.org, yosryahmed@google.com, nphamcs@gmail.com, chengming.zhou@linux.dev, usamaarif642@gmail.com, shakeel.butt@linux.dev, ryan.roberts@arm.com, ying.huang@intel.com, 21cnbao@gmail.com, akpm@linux-foundation.org Cc: nanhai.zou@intel.com, wajdi.k.feghali@intel.com, vinodh.gopal@intel.com, kanchana.p.sridhar@intel.com Subject: [PATCH v7 5/8] mm: zswap: Compress and store a specific page in a folio. Date: Mon, 23 Sep 2024 18:17:06 -0700 Message-Id: <20240924011709.7037-6-kanchana.p.sridhar@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20240924011709.7037-1-kanchana.p.sridhar@intel.com> References: <20240924011709.7037-1-kanchana.p.sridhar@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" For zswap_store() to handle mTHP folios, we need to iterate through each page in the mTHP, compress it and store it in the zswap pool. This patch introduces an auxiliary function zswap_store_page() that provides this functionality. The function signature reflects the design intent, namely, for it to be invoked by zswap_store() per-page in an mTHP. Hence, the folio's objcg and the zswap_pool to use are input parameters for sake of efficiency and consistency. The functionality in zswap_store_page() is reused and adapted from Ryan Roberts' RFC patch [1]: "[RFC,v1] mm: zswap: Store large folios without splitting" [1] https://lore.kernel.org/linux-mm/20231019110543.3284654-1-ryan.robert= s@arm.com/T/#u Co-developed-by: Ryan Roberts Signed-off-by: Signed-off-by: Kanchana P Sridhar --- mm/zswap.c | 88 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 88 insertions(+) diff --git a/mm/zswap.c b/mm/zswap.c index 9bea948d653e..8f2e0ab34c84 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -1463,6 +1463,94 @@ static void zswap_delete_stored_offsets(struct xarra= y *tree, } } =20 +/* + * Stores the page at specified "index" in a folio. + * + * @folio: The folio to store in zswap. + * @index: Index into the page in the folio that this function will store. + * @objcg: The folio's objcg. + * @pool: The zswap_pool to store the compressed data for the page. + */ +static bool __maybe_unused zswap_store_page(struct folio *folio, long inde= x, + struct obj_cgroup *objcg, + struct zswap_pool *pool) +{ + swp_entry_t swp =3D folio->swap; + int type =3D swp_type(swp); + pgoff_t offset =3D swp_offset(swp) + index; + struct page *page =3D folio_page(folio, index); + struct xarray *tree =3D swap_zswap_tree(swp); + struct zswap_entry *entry; + + if (objcg) + obj_cgroup_get(objcg); + + if (zswap_check_limits()) + goto reject; + + /* allocate entry */ + entry =3D zswap_entry_cache_alloc(GFP_KERNEL, folio_nid(folio)); + if (!entry) { + zswap_reject_kmemcache_fail++; + goto reject; + } + + /* if entry is successfully added, it keeps the reference */ + if (!zswap_pool_get(pool)) + goto freepage; + + entry->pool =3D pool; + + if (!zswap_compress(page, entry)) + goto put_pool; + + entry->swpentry =3D swp_entry(type, offset); + entry->objcg =3D objcg; + entry->referenced =3D true; + + if (!zswap_store_entry(tree, entry)) + goto store_failed; + + if (objcg) { + obj_cgroup_charge_zswap(objcg, entry->length); + count_objcg_event(objcg, ZSWPOUT); + } + + /* + * We finish initializing the entry while it's already in xarray. + * This is safe because: + * + * 1. Concurrent stores and invalidations are excluded by folio lock. + * + * 2. Writeback is excluded by the entry not being on the LRU yet. + * The publishing order matters to prevent writeback from seeing + * an incoherent entry. + */ + if (entry->length) { + INIT_LIST_HEAD(&entry->lru); + zswap_lru_add(&zswap_list_lru, entry); + } + + /* update stats */ + atomic_inc(&zswap_stored_pages); + count_vm_event(ZSWPOUT); + + return true; + +store_failed: + zpool_free(entry->pool->zpool, entry->handle); +put_pool: + zswap_pool_put(pool); +freepage: + zswap_entry_cache_free(entry); +reject: + obj_cgroup_put(objcg); + if (zswap_pool_reached_full) + queue_work(shrink_wq, &zswap_shrink_work); + + return false; +} + bool zswap_store(struct folio *folio) { long nr_pages =3D folio_nr_pages(folio); --=20 2.27.0 From nobody Fri Nov 29 05:51:58 2024 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7200FB67A for ; Tue, 24 Sep 2024 01:17:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.12 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727140639; cv=none; b=DErHwAbTCDJWivyPzajeZerehKaeFuSRlOMZKEvzppkUVnCVU33Lumom1Rb29anB3MclOvGa0TMrHxZJ4lrw5l8oljpd0vVsou1U6jC1CsBC+gvc3YXlnktd4p6BrPccXvjHz9Yvwmbi28fCrKyLyr2CXO0I7M54VtFSrIX1ohI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727140639; c=relaxed/simple; bh=TC61NMZkVLT5VlFsRHxoHlVUivXlE7pDxlM31DB9mpg=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=ko8kpUgi4762tJmsP7oLrFmxizlDmWputihNKhwCXWEX4/CAtMs57zP6WYVnJeUALOjMsx5VSxIW9WKxZwrWw/0I3JW+S0cseHtkt5iClAae7PTtQLvg/XkFTs+1YjbQMrr5rfKj8vSpz8eakip5dP/KAp9edO/vSvDVfjkeJys= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=BxsxztKY; arc=none smtp.client-ip=192.198.163.12 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="BxsxztKY" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1727140636; x=1758676636; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=TC61NMZkVLT5VlFsRHxoHlVUivXlE7pDxlM31DB9mpg=; b=BxsxztKYayVmUCd/vbvFfPGYU9U8u74rRmC3hFmntWOFpDDHI/fCb8Ov tGMjrLed4s2ZuaWSomollQkyCRIKgZlg8WQgTriFoC7M3I31z7+e96JnA wnvovIw8rp1NHiS20eO6hY7ca4ThD3v4B/R7SGQvNWF0RwwTGoxlGrSwl JjGhMQ42C2PdBbmLelLLuEa+07huFN9vEwSMN2oWNulqOlf5d8O90beMf ZjljQt/EvPMEG7NO3b8ifoMdlEf506RQiiMrK+MJ1gXLLpOrawhrFvG2M 1vvbFuD581Zshp2PPehOhV503hLjtKlq7EWfwF6VBsqlA3OhDI0n0eRmo A==; X-CSE-ConnectionGUID: sCU8AjK3RV+gw6WNglq0sg== X-CSE-MsgGUID: J33hCYPgTF2Flk2XkGoP7w== X-IronPort-AV: E=McAfee;i="6700,10204,11204"; a="30002038" X-IronPort-AV: E=Sophos;i="6.10,253,1719903600"; d="scan'208";a="30002038" Received: from orviesa001.jf.intel.com ([10.64.159.141]) by fmvoesa106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Sep 2024 18:17:11 -0700 X-CSE-ConnectionGUID: h6kZ0n4KQd2ezP+USAgHWA== X-CSE-MsgGUID: 6ihG0j0jSjSdRCs1upV+/g== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.10,253,1719903600"; d="scan'208";a="108688460" Received: from jf5300-b11a338t.jf.intel.com ([10.242.51.6]) by orviesa001.jf.intel.com with ESMTP; 23 Sep 2024 18:17:10 -0700 From: Kanchana P Sridhar To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, hannes@cmpxchg.org, yosryahmed@google.com, nphamcs@gmail.com, chengming.zhou@linux.dev, usamaarif642@gmail.com, shakeel.butt@linux.dev, ryan.roberts@arm.com, ying.huang@intel.com, 21cnbao@gmail.com, akpm@linux-foundation.org Cc: nanhai.zou@intel.com, wajdi.k.feghali@intel.com, vinodh.gopal@intel.com, kanchana.p.sridhar@intel.com Subject: [PATCH v7 6/8] mm: zswap: Support mTHP swapout in zswap_store(). Date: Mon, 23 Sep 2024 18:17:07 -0700 Message-Id: <20240924011709.7037-7-kanchana.p.sridhar@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20240924011709.7037-1-kanchana.p.sridhar@intel.com> References: <20240924011709.7037-1-kanchana.p.sridhar@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" zswap_store() will now store mTHP and PMD-size THP folios by compressing them page by page. This patch provides a sequential implementation of storing an mTHP in zswap_store() by iterating through each page in the folio to compress and store it in the zswap zpool. Towards this goal, zswap_compress() is modified to take a page instead of a folio as input. Each page's swap offset is stored as a separate zswap entry. If an error is encountered during the store of any page in the mTHP, all previous pages/entries stored will be invalidated. Thus, an mTHP is either entirely stored in ZSWAP, or entirely not stored in ZSWAP. This forms the basis for building batching of pages during zswap store of large folios by compressing batches of up to say, 8 pages in an mTHP in parallel in hardware, with the Intel In-Memory Analytics Accelerator (Intel IAA). A new config variable CONFIG_ZSWAP_STORE_THP_DEFAULT_ON (off by default) will enable/disable zswap storing of (m)THP. The corresponding tunable zswap module parameter is "mthp_enabled". This change reuses and adapts the functionality in Ryan Roberts' RFC patch [1]: "[RFC,v1] mm: zswap: Store large folios without splitting" [1] https://lore.kernel.org/linux-mm/20231019110543.3284654-1-ryan.robert= s@arm.com/T/#u Also, addressed some of the RFC comments from the discussion in [1]. Co-developed-by: Ryan Roberts Signed-off-by: Signed-off-by: Kanchana P Sridhar --- mm/Kconfig | 8 ++++ mm/zswap.c | 122 +++++++++++++++++++++++++---------------------------- 2 files changed, 66 insertions(+), 64 deletions(-) diff --git a/mm/Kconfig b/mm/Kconfig index 09aebca1cae3..c659fb732ec4 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -59,6 +59,14 @@ config ZSWAP_SHRINKER_DEFAULT_ON reducing the chance that cold pages will reside in the zswap pool and consume memory indefinitely. =20 +config ZSWAP_STORE_THP_DEFAULT_ON + bool "Store mTHP and THP folios in zswap" + depends on ZSWAP + default n + help + If selected, zswap will process mTHP and THP folios by + compressing and storing each 4K page in the large folio. + choice prompt "Default compressor" depends on ZSWAP diff --git a/mm/zswap.c b/mm/zswap.c index 8f2e0ab34c84..16ab770546d6 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -127,6 +127,14 @@ static bool zswap_shrinker_enabled =3D IS_ENABLED( CONFIG_ZSWAP_SHRINKER_DEFAULT_ON); module_param_named(shrinker_enabled, zswap_shrinker_enabled, bool, 0644); =20 +/* + * Enable/disable zswap processing of mTHP folios. + * For now, only zswap_store will process mTHP folios. + */ +static bool zswap_mthp_enabled =3D IS_ENABLED( + CONFIG_ZSWAP_STORE_THP_DEFAULT_ON); +module_param_named(mthp_enabled, zswap_mthp_enabled, bool, 0644); + bool zswap_is_enabled(void) { return zswap_enabled; @@ -1471,9 +1479,9 @@ static void zswap_delete_stored_offsets(struct xarray= *tree, * @objcg: The folio's objcg. * @pool: The zswap_pool to store the compressed data for the page. */ -static bool __maybe_unused zswap_store_page(struct folio *folio, long inde= x, - struct obj_cgroup *objcg, - struct zswap_pool *pool) +static bool zswap_store_page(struct folio *folio, long index, + struct obj_cgroup *objcg, + struct zswap_pool *pool) { swp_entry_t swp =3D folio->swap; int type =3D swp_type(swp); @@ -1551,51 +1559,63 @@ static bool __maybe_unused zswap_store_page(struct = folio *folio, long index, return false; } =20 +/* + * Modified to store mTHP folios. Each page in the mTHP will be compressed + * and stored sequentially. + */ bool zswap_store(struct folio *folio) { long nr_pages =3D folio_nr_pages(folio); swp_entry_t swp =3D folio->swap; pgoff_t offset =3D swp_offset(swp); struct xarray *tree =3D swap_zswap_tree(swp); - struct zswap_entry *entry; struct obj_cgroup *objcg =3D NULL; struct mem_cgroup *memcg =3D NULL; + struct zswap_pool *pool; + bool ret =3D false; + long index; =20 VM_WARN_ON_ONCE(!folio_test_locked(folio)); VM_WARN_ON_ONCE(!folio_test_swapcache(folio)); =20 - /* Large folios aren't supported */ - if (folio_test_large(folio)) + /* Storing large folios isn't enabled */ + if (!zswap_mthp_enabled && folio_test_large(folio)) return false; =20 if (!zswap_enabled) - goto check_old; + goto reject; =20 - /* Check cgroup limits */ + /* + * Check cgroup limits: + * + * The cgroup zswap limit check is done once at the beginning of an + * mTHP store, and not within zswap_store_page() for each page + * in the mTHP. We do however check the zswap pool limits at the + * start of zswap_store_page(). What this means is, the cgroup + * could go over the limits by at most (HPAGE_PMD_NR - 1) pages. + * However, the per-store-page zswap pool limits check should + * hopefully trigger the cgroup aware and zswap LRU aware global + * reclaim implemented in the shrinker. If this assumption holds, + * the cgroup exceeding the zswap limits could potentially be + * resolved before the next zswap_store, and if it is not, the next + * zswap_store would fail the cgroup zswap limit check at the start. + */ objcg =3D get_obj_cgroup_from_folio(folio); if (objcg && !obj_cgroup_may_zswap(objcg)) { memcg =3D get_mem_cgroup_from_objcg(objcg); if (shrink_memcg(memcg)) { mem_cgroup_put(memcg); - goto reject; + goto put_objcg; } mem_cgroup_put(memcg); } =20 if (zswap_check_limits()) - goto reject; - - /* allocate entry */ - entry =3D zswap_entry_cache_alloc(GFP_KERNEL, folio_nid(folio)); - if (!entry) { - zswap_reject_kmemcache_fail++; - goto reject; - } + goto put_objcg; =20 - /* if entry is successfully added, it keeps the reference */ - entry->pool =3D zswap_pool_current_get(); - if (!entry->pool) - goto freepage; + pool =3D zswap_pool_current_get(); + if (!pool) + goto put_objcg; =20 if (objcg) { memcg =3D get_mem_cgroup_from_objcg(objcg); @@ -1606,60 +1626,34 @@ bool zswap_store(struct folio *folio) mem_cgroup_put(memcg); } =20 - if (!zswap_compress(&folio->page, entry)) - goto put_pool; - - entry->swpentry =3D swp; - entry->objcg =3D objcg; - entry->referenced =3D true; - - if (!zswap_store_entry(tree, entry)) - goto store_failed; - - if (objcg) { - obj_cgroup_charge_zswap(objcg, entry->length); - count_objcg_event(objcg, ZSWPOUT); - } - /* - * We finish initializing the entry while it's already in xarray. - * This is safe because: - * - * 1. Concurrent stores and invalidations are excluded by folio lock. - * - * 2. Writeback is excluded by the entry not being on the LRU yet. - * The publishing order matters to prevent writeback from seeing - * an incoherent entry. + * Store each page of the folio as a separate entry. If we fail to store + * a page, unwind by removing all the previous pages we stored. */ - if (entry->length) { - INIT_LIST_HEAD(&entry->lru); - zswap_lru_add(&zswap_list_lru, entry); + for (index =3D 0; index < nr_pages; ++index) { + if (!zswap_store_page(folio, index, objcg, pool)) + goto put_pool; } =20 - /* update stats */ - atomic_inc(&zswap_stored_pages); - count_vm_event(ZSWPOUT); - - return true; + ret =3D true; =20 -store_failed: - zpool_free(entry->pool->zpool, entry->handle); put_pool: - zswap_pool_put(entry->pool); -freepage: - zswap_entry_cache_free(entry); -reject: + zswap_pool_put(pool); +put_objcg: obj_cgroup_put(objcg); if (zswap_pool_reached_full) queue_work(shrink_wq, &zswap_shrink_work); -check_old: +reject: /* - * If the zswap store fails or zswap is disabled, we must invalidate the - * possibly stale entry which was previously stored at this offset. - * Otherwise, writeback could overwrite the new data in the swapfile. + * If the zswap store fails or zswap is disabled, we must invalidate + * the possibly stale entries which were previously stored at the + * offsets corresponding to each page of the folio. Otherwise, + * writeback could overwrite the new data in the swapfile. */ - zswap_delete_stored_offsets(tree, offset, nr_pages); - return false; + if (!ret) + zswap_delete_stored_offsets(tree, offset, nr_pages); + + return ret; } =20 bool zswap_load(struct folio *folio) --=20 2.27.0 From nobody Fri Nov 29 05:51:58 2024 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A54D0C13C for ; Tue, 24 Sep 2024 01:17:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.12 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727140638; cv=none; b=lVBaGGhTNN6MaXgsctpTj7nYgfnYm2e/iUkeAcRARl7cjg+fuKfubuN0huORGTJg284Pn7hWkbDlkAwCLryEVg2hg8vuSdJKwD9TIKA72ESSmIPWkusPyvId+/9I4U0CDo6TS4ALEh8lw5lKJjg0j2W2jcENB9fHsv/Tk71aFo4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727140638; c=relaxed/simple; bh=nZQ5WdYOudXFblQyI1xpFNONjVx0pWlV5kfygJBmOHg=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Sv5GWdCrI57sVZL9XAOvczcZ32VJUM+KtOqdqkaCCds+Vp4dD+kUdoOtvcCfW/5C/OZH9lX8A+G7EPrIuEcP5f6uvUHixYrUyxHOls17cpag8sg3ij+wCmHPDcx3YOquZUu+vowQcgoR0XkpOZjux5Sw1YgP8raxF3BQrBfdvTg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=mtA2eFYH; arc=none smtp.client-ip=192.198.163.12 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="mtA2eFYH" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1727140636; x=1758676636; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=nZQ5WdYOudXFblQyI1xpFNONjVx0pWlV5kfygJBmOHg=; b=mtA2eFYH2HaCeU2SemCor6gO8aJy9aXp6uMwlMjdZGVwD8UdX4L9yN/S bhvcqSM34ZaCRUpLbvHK2e6yAqrvM8MJoAAFpTmEMXzDqYIyPhUDHPdwo lciwf0WE6994Eq5DVdiEngueUApRCExm0CaUGYtUFdeGTJoI/363TtdFb 5cvYXqS0xvqW9qz+wwYlr66KU4bhYU9jZBNzi2JDGKOxlD6RJ3AieUn+l +dH5SbflYOAtRZ9kUhbuySASjva8+OmF624f9TqjxSobIKgoBp4jfwRdY o0nZnCuWABiSEPJFYuKPProzhWho/Yv+Lk5X41a1KZ9wwQp/0sGv4x4LX g==; X-CSE-ConnectionGUID: cvmXRArySz6FRWN5pDTx/A== X-CSE-MsgGUID: zxEUhqtXQuS3wYbfUbzezQ== X-IronPort-AV: E=McAfee;i="6700,10204,11204"; a="30002046" X-IronPort-AV: E=Sophos;i="6.10,253,1719903600"; d="scan'208";a="30002046" Received: from orviesa001.jf.intel.com ([10.64.159.141]) by fmvoesa106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Sep 2024 18:17:11 -0700 X-CSE-ConnectionGUID: udjgHgWqQzO/LxXYb+uyVQ== X-CSE-MsgGUID: lLNNKrXVTtWLEYgP3S0l6Q== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.10,253,1719903600"; d="scan'208";a="108688463" Received: from jf5300-b11a338t.jf.intel.com ([10.242.51.6]) by orviesa001.jf.intel.com with ESMTP; 23 Sep 2024 18:17:10 -0700 From: Kanchana P Sridhar To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, hannes@cmpxchg.org, yosryahmed@google.com, nphamcs@gmail.com, chengming.zhou@linux.dev, usamaarif642@gmail.com, shakeel.butt@linux.dev, ryan.roberts@arm.com, ying.huang@intel.com, 21cnbao@gmail.com, akpm@linux-foundation.org Cc: nanhai.zou@intel.com, wajdi.k.feghali@intel.com, vinodh.gopal@intel.com, kanchana.p.sridhar@intel.com Subject: [PATCH v7 7/8] mm: swap: Count successful mTHP ZSWAP stores in sysfs mTHP zswpout stats. Date: Mon, 23 Sep 2024 18:17:08 -0700 Message-Id: <20240924011709.7037-8-kanchana.p.sridhar@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20240924011709.7037-1-kanchana.p.sridhar@intel.com> References: <20240924011709.7037-1-kanchana.p.sridhar@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add a new MTHP_STAT_ZSWPOUT entry to the sysfs mTHP stats so that per-order mTHP folio ZSWAP stores can be accounted. If zswap_store() successfully swaps out an mTHP, it will be counted under the per-order sysfs "zswpout" stats: /sys/kernel/mm/transparent_hugepage/hugepages-*kB/stats/zswpout Other block dev/fs mTHP swap-out events will be counted under the existing sysfs "swpout" stats: /sys/kernel/mm/transparent_hugepage/hugepages-*kB/stats/swpout Signed-off-by: Kanchana P Sridhar --- include/linux/huge_mm.h | 1 + mm/huge_memory.c | 3 +++ mm/page_io.c | 1 + 3 files changed, 5 insertions(+) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index 0b0539f4ee1a..ab95b94e9627 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -118,6 +118,7 @@ enum mthp_stat_item { MTHP_STAT_ANON_FAULT_ALLOC, MTHP_STAT_ANON_FAULT_FALLBACK, MTHP_STAT_ANON_FAULT_FALLBACK_CHARGE, + MTHP_STAT_ZSWPOUT, MTHP_STAT_SWPOUT, MTHP_STAT_SWPOUT_FALLBACK, MTHP_STAT_SHMEM_ALLOC, diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 4e34b7f89daf..7d8ce7891ba8 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -612,6 +612,7 @@ static struct kobj_attribute _name##_attr =3D __ATTR_RO= (_name) DEFINE_MTHP_STAT_ATTR(anon_fault_alloc, MTHP_STAT_ANON_FAULT_ALLOC); DEFINE_MTHP_STAT_ATTR(anon_fault_fallback, MTHP_STAT_ANON_FAULT_FALLBACK); DEFINE_MTHP_STAT_ATTR(anon_fault_fallback_charge, MTHP_STAT_ANON_FAULT_FAL= LBACK_CHARGE); +DEFINE_MTHP_STAT_ATTR(zswpout, MTHP_STAT_ZSWPOUT); DEFINE_MTHP_STAT_ATTR(swpout, MTHP_STAT_SWPOUT); DEFINE_MTHP_STAT_ATTR(swpout_fallback, MTHP_STAT_SWPOUT_FALLBACK); #ifdef CONFIG_SHMEM @@ -630,6 +631,7 @@ static struct attribute *anon_stats_attrs[] =3D { &anon_fault_fallback_attr.attr, &anon_fault_fallback_charge_attr.attr, #ifndef CONFIG_SHMEM + &zswpout_attr.attr, &swpout_attr.attr, &swpout_fallback_attr.attr, #endif @@ -660,6 +662,7 @@ static struct attribute_group file_stats_attr_grp =3D { =20 static struct attribute *any_stats_attrs[] =3D { #ifdef CONFIG_SHMEM + &zswpout_attr.attr, &swpout_attr.attr, &swpout_fallback_attr.attr, #endif diff --git a/mm/page_io.c b/mm/page_io.c index bc1183299a7d..4aa34862676f 100644 --- a/mm/page_io.c +++ b/mm/page_io.c @@ -269,6 +269,7 @@ int swap_writepage(struct page *page, struct writeback_= control *wbc) swap_zeromap_folio_clear(folio); } if (zswap_store(folio)) { + count_mthp_stat(folio_order(folio), MTHP_STAT_ZSWPOUT); folio_unlock(folio); return 0; } --=20 2.27.0 From nobody Fri Nov 29 05:51:58 2024 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7F9C812E7E for ; Tue, 24 Sep 2024 01:17:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.12 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727140639; cv=none; b=a1G94LI+Y2Xneo6yBP4ScS8gJqWPGBoRz1kiyjx0TWeNKv3xASo3zBaQ+DASuuRU9/TzBH9Brd9StztY7k4rA4bT4ZC0+LwBBEqKG7fjR6/XwcIN114D6gV8irklw0Fd0aJ7ePtLDJzpjexlXn+N++yyxvC51WaTT4cy8JnpNUM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727140639; c=relaxed/simple; bh=6ALoeaC41g+MaGEwJSRyCvBiONdX4EAtPZdFIMUQEE0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=fO6so110cYewOt5ZLAbJXSEtvjHrEyaQ1joRTlOOP2Y01sFazpCM7NjsInE04lj9O1c1m7okRLUzfgkJ8Med93KCcRpQNBJz+xGrtOr8XlYWFbCf7BN+g2myqsIkOvnaaFflBpxdhqTwObiaGd+aq280Swxg1/HDIdHRcJoNQy4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=hQRg9t8M; arc=none smtp.client-ip=192.198.163.12 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="hQRg9t8M" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1727140637; x=1758676637; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=6ALoeaC41g+MaGEwJSRyCvBiONdX4EAtPZdFIMUQEE0=; b=hQRg9t8M655KJBkSTNGN0uhpt5rdlRLI+M0xBVSeWAnV4XR7Etn6a32s 5so8VZqfmRQDR3gO68ulEzLH2GW0/T9i2eKkDKMMEEFzzPCAsjbl1Krxg KkL2ysdb+5N44+ZiH47mbbqp80bKcI8vBdJThxZjgmRfQ7D4YYhmMzXNQ WkGucgLsh1dVBSqYSsZ9diC3ZPcc+Jnp90ylTUflroYn1yNx6dBkLLisK eXt2Cce9Rpc7XCLSFP2BEPG9rJytihFyO1kR+5yZNmci9MVeXY7132iFB qzCGCkE4ckF8F19jJ2M0we1/gn98sZyCKDtQuhYDh8EYLi8kDnN4GPgQ/ g==; X-CSE-ConnectionGUID: TkANmsP0RPa8ArRA4Tef9g== X-CSE-MsgGUID: XRAFMqICROGPjF39eo+sRA== X-IronPort-AV: E=McAfee;i="6700,10204,11204"; a="30002051" X-IronPort-AV: E=Sophos;i="6.10,253,1719903600"; d="scan'208";a="30002051" Received: from orviesa001.jf.intel.com ([10.64.159.141]) by fmvoesa106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Sep 2024 18:17:11 -0700 X-CSE-ConnectionGUID: XVBMt6QAQySzvitwOcWK+A== X-CSE-MsgGUID: EuABZkyHT6aA0zdKQwNlSA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.10,253,1719903600"; d="scan'208";a="108688466" Received: from jf5300-b11a338t.jf.intel.com ([10.242.51.6]) by orviesa001.jf.intel.com with ESMTP; 23 Sep 2024 18:17:10 -0700 From: Kanchana P Sridhar To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, hannes@cmpxchg.org, yosryahmed@google.com, nphamcs@gmail.com, chengming.zhou@linux.dev, usamaarif642@gmail.com, shakeel.butt@linux.dev, ryan.roberts@arm.com, ying.huang@intel.com, 21cnbao@gmail.com, akpm@linux-foundation.org Cc: nanhai.zou@intel.com, wajdi.k.feghali@intel.com, vinodh.gopal@intel.com, kanchana.p.sridhar@intel.com Subject: [PATCH v7 8/8] mm: Document the newly added mTHP zswpout stats, clarify swpout semantics. Date: Mon, 23 Sep 2024 18:17:09 -0700 Message-Id: <20240924011709.7037-9-kanchana.p.sridhar@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20240924011709.7037-1-kanchana.p.sridhar@intel.com> References: <20240924011709.7037-1-kanchana.p.sridhar@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Added documentation for the newly added sysfs mTHP "zswpout" stats. Clarified that only non-ZSWAP mTHP swapouts will be accounted in the mTHP "swpout" stats. Signed-off-by: Kanchana P Sridhar --- Documentation/admin-guide/mm/transhuge.rst | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/Documentation/admin-guide/mm/transhuge.rst b/Documentation/adm= in-guide/mm/transhuge.rst index cfdd16a52e39..a65f905e9ca7 100644 --- a/Documentation/admin-guide/mm/transhuge.rst +++ b/Documentation/admin-guide/mm/transhuge.rst @@ -530,10 +530,14 @@ anon_fault_fallback_charge instead falls back to using huge pages with lower orders or small pages even though the allocation was successful. =20 -swpout - is incremented every time a huge page is swapped out in one +zswpout + is incremented every time a huge page is swapped out to ZSWAP in one piece without splitting. =20 +swpout + is incremented every time a huge page is swapped out to a non-ZSWAP + swap entity in one piece without splitting. + swpout_fallback is incremented if a huge page has to be split before swapout. Usually because failed to allocate some continuous swap space --=20 2.27.0