From nobody Fri Dec 19 10:39:44 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4D93C745E2 for ; Wed, 3 Jul 2024 04:01:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719979272; cv=none; b=Ll5bAVLV6jhcSjHnhxc/tzxvwmP7GZUwp02yVyahfhAaxRTXIYCb5+1Wo8dK+4PcIdjvNSHLFv2qZH1J3WJO6/nkLgqJASjEZMYRg2s6JOf3UwVPXkUzvve2A12xj/Sfc1oZKtRHSB/ZMq+HnsYSZov3Qlaa8xIAl12e5bxbhzY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719979272; c=relaxed/simple; bh=81TBE+QYmpKV96rZHKgj6AZ1fYwMgRmX/nAJKqOjEgA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=NkwoznrOCi+yjHv+0/QSy1mDuofYiigcwM7vXFUY3RecH52ECx68096w7+4SZR5W402upkmjwvd1kupk4COAPvXkQe6QRY6x87A/K+i1ywKYSj3tonleQKjNsKfc8KgW68Bp4SExYyI/IpIDMrP2NsH6bkU/JKYRNHvLSLIFzsI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=NCzHmjUT; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="NCzHmjUT" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8B957C4AF0E; Wed, 3 Jul 2024 04:01:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1719979271; bh=81TBE+QYmpKV96rZHKgj6AZ1fYwMgRmX/nAJKqOjEgA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=NCzHmjUT1evH4SkSRoxBPGps9eH/wF60nNhj3DQNG4ErQlV8wVwU82WmzAWXzA5y5 tNO3t6uq+kDRXkCvQEm3UgnevwfPAZd+aDrqyXavCl02m57ofEk1Dt1hcPUzJtOjop gd0kTeggPQkI95uQvc6hqhxPWTNb4CN4a9PVkgKsf31qyp1EigX1Qoo7g/J6tCb+wo 8OVHJ4Vl1xXn6zVz8lgE4djN+vj7ZPBLDcQ0yFanitjDZl4YfGv8IK6ywhb2l0jFAU rApsfucEskaLl1ikSSmJ2Luh8OjWtfIst0jlXYgDc+lifp0qZ7yUhImyk/Ye/0C+br CLWA3EZfbooCA== From: alexs@kernel.org To: Vitaly Wool , Miaohe Lin , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, minchan@kernel.org, willy@infradead.org, senozhatsky@chromium.org, david@redhat.com, 42.hyeyoo@gmail.com, Yosry Ahmed , nphamcs@gmail.com Cc: Alex Shi Subject: [PATCH v2 01/20] mm/zsmalloc: add zpdesc memory descriptor for zswap.zpool Date: Wed, 3 Jul 2024 12:05:51 +0800 Message-ID: <20240703040613.681396-2-alexs@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240703040613.681396-1-alexs@kernel.org> References: <20240703040613.681396-1-alexs@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Alex Shi The 1st patch introduces new memory decriptor zpdesc and rename zspage.first_page to zspage.first_zpdesc, no functional change. We removed PG_owner_priv_1 since it was moved to zspage after commit a41ec880aa7b ("zsmalloc: move huge compressed obj from page to zspage"). And keep the memcg_data member, since as Yosry pointed out: "When the pages are freed, put_page() -> folio_put() -> __folio_put() will = call mem_cgroup_uncharge(). The latter will call folio_memcg() (which reads folio->memcg_data) to figure out if uncharging needs to be done. There are also other similar code paths that will check folio->memcg_data. It is currently expected to be present for all folios. So until we have custom code paths per-folio type for allocation/freeing/etc, we need to keep folio->memcg_data present and properly initialized." Originally-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Signed-off-by: Alex Shi --- mm/zpdesc.h | 64 +++++++++++++++++++++++++++++++++++++++++++++++++++ mm/zsmalloc.c | 21 ++++++++--------- 2 files changed, 74 insertions(+), 11 deletions(-) create mode 100644 mm/zpdesc.h diff --git a/mm/zpdesc.h b/mm/zpdesc.h new file mode 100644 index 000000000000..4455a663ee84 --- /dev/null +++ b/mm/zpdesc.h @@ -0,0 +1,64 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* zpdesc.h: zswap.zpool memory descriptor + * + * Written by Alex Shi + * Hyeonggon Yoo <42.hyeyoo@gmail.com> + */ +#ifndef __MM_ZPDESC_H__ +#define __MM_ZPDESC_H__ + +/* + * struct zpdesc - Memory descriptor for zpool memory, now is for zsmalloc + * @flags: Page flags, PG_private: identifies the first component page + * @lru: Indirectly used by page migration + * @mops: Used by page migration + * @next: Next zpdesc in a zspage in zsmalloc zpool + * @handle: For huge zspage in zsmalloc zpool + * @zspage: Pointer to zspage in zsmalloc + * @memcg_data: Memory Control Group data. + * + * This struct overlays struct page for now. Do not modify without a good + * understanding of the issues. + */ +struct zpdesc { + unsigned long flags; + struct list_head lru; + struct movable_operations *mops; + union { + /* Next zpdescs in a zspage in zsmalloc zpool */ + struct zpdesc *next; + /* For huge zspage in zsmalloc zpool */ + unsigned long handle; + }; + struct zspage *zspage; + unsigned long _zp_pad_1; +#ifdef CONFIG_SLAB_OBJ_EXT + unsigned long memcg_data; +#endif +}; +#define ZPDESC_MATCH(pg, zp) \ + static_assert(offsetof(struct page, pg) =3D=3D offsetof(struct zpdesc, zp= )) + +ZPDESC_MATCH(flags, flags); +ZPDESC_MATCH(lru, lru); +ZPDESC_MATCH(mapping, mops); +ZPDESC_MATCH(index, next); +ZPDESC_MATCH(index, handle); +ZPDESC_MATCH(private, zspage); +ZPDESC_MATCH(memcg_data, memcg_data); +#undef ZPDESC_MATCH +static_assert(sizeof(struct zpdesc) <=3D sizeof(struct page)); + +#define zpdesc_page(zp) (_Generic((zp), \ + const struct zpdesc *: (const struct page *)(zp), \ + struct zpdesc *: (struct page *)(zp))) + +#define zpdesc_folio(zp) (_Generic((zp), \ + const struct zpdesc *: (const struct folio *)(zp), \ + struct zpdesc *: (struct folio *)(zp))) + +#define page_zpdesc(p) (_Generic((p), \ + const struct page *: (const struct zpdesc *)(p), \ + struct page *: (struct zpdesc *)(p))) + +#endif diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 5d6581ab7c07..a532851025f9 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -13,20 +13,18 @@ =20 /* * Following is how we use various fields and flags of underlying - * struct page(s) to form a zspage. + * struct zpdesc(page) to form a zspage. * - * Usage of struct page fields: - * page->private: points to zspage - * page->index: links together all component pages of a zspage + * Usage of struct zpdesc fields: + * zpdesc->zspage: points to zspage + * zpdesc->next: links together all component pages of a zspage * For the huge page, this is always 0, so we use this field * to store handle. * page->page_type: PG_zsmalloc, lower 16 bit locate the first object * offset in a subpage of a zspage * - * Usage of struct page flags: + * Usage of struct zpdesc(page) flags: * PG_private: identifies the first component page - * PG_owner_priv_1: identifies the huge component page - * */ =20 #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt @@ -64,6 +62,7 @@ #include #include #include +#include "zpdesc.h" =20 #define ZSPAGE_MAGIC 0x58 =20 @@ -253,7 +252,7 @@ struct zspage { }; unsigned int inuse; unsigned int freeobj; - struct page *first_page; + struct zpdesc *first_zpdesc; struct list_head list; /* fullness list */ struct zs_pool *pool; rwlock_t lock; @@ -448,7 +447,7 @@ static inline void mod_zspage_inuse(struct zspage *zspa= ge, int val) =20 static inline struct page *get_first_page(struct zspage *zspage) { - struct page *first_page =3D zspage->first_page; + struct page *first_page =3D zpdesc_page(zspage->first_zpdesc); =20 VM_BUG_ON_PAGE(!is_first_page(first_page), first_page); return first_page; @@ -948,7 +947,7 @@ static void create_page_chain(struct size_class *class,= struct zspage *zspage, set_page_private(page, (unsigned long)zspage); page->index =3D 0; if (i =3D=3D 0) { - zspage->first_page =3D page; + zspage->first_zpdesc =3D page_zpdesc(page); SetPagePrivate(page); if (unlikely(class->objs_per_zspage =3D=3D 1 && class->pages_per_zspage =3D=3D 1)) @@ -1324,7 +1323,7 @@ static unsigned long obj_malloc(struct zs_pool *pool, link->handle =3D handle | OBJ_ALLOCATED_TAG; else /* record handle to page->index */ - zspage->first_page->index =3D handle | OBJ_ALLOCATED_TAG; + zspage->first_zpdesc->handle =3D handle | OBJ_ALLOCATED_TAG; =20 kunmap_atomic(vaddr); mod_zspage_inuse(zspage, 1); --=20 2.43.0 From nobody Fri Dec 19 10:39:44 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5D739745E2 for ; Wed, 3 Jul 2024 04:01:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719979276; cv=none; b=GVHACTazXOD0E7PW9e9bwCmq3sjix+/N68Ezq9GNYAWNt73hY9Dw69O/JIZExdrElNsguSg2Gfp+EQzYBfeF0o2JPBzPXOavRQ5V70obPR8feZDWSK41g+8S4ayCdzmzPV7wY9o1/FPsr0a5Ng4sxKVm0zYPfHBTbHHS7efaU/E= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719979276; c=relaxed/simple; bh=mgpKO63FhRCf6bCfWvth1dWCysiouyCKf660VxcU2X8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=QdlkYzL/5qRPKzytvYr+qr8DI+HJFEBHW6EjfTxd1joAj8G3YKOGTEey7yDVSDyKHlNynighfK8SaLuDjpNQ+5UJAQmT6NrDv979lX+L918qZpM6fwgLhVypWYCdbe6Rf94RkPviTV0cCeGcXOJk57AMYvW4qaOp6nBJiVwErzU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=M3fvr9YU; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="M3fvr9YU" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 81E71C4AF0C; Wed, 3 Jul 2024 04:01:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1719979275; bh=mgpKO63FhRCf6bCfWvth1dWCysiouyCKf660VxcU2X8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=M3fvr9YU3RoqDYqi7LepVhhyobK2kETlXMib4NHfNfY6Xg3gkhcZVMOUv3/ph7BuW FYbmXaLXttnr8mKSiGOtqWQCsfx4B9jfmUcCX3oooPG6ckNgLhLvFFhGujDl7UYG/m paouB5VQgKocBJwp7NyPedhmvhxN6Tz9jY3PwV7x56sVrli8XW0YcF4tZh7Tnh1SFl 6NVukRxT9usEljzgLjDhoVPOfEYQIRDXJesuIfvKKRdbh+vB59OBL8c4p2Du1xGNhv XAMnJJPn1Ygq4VTLpVP9TL5nRFRoczdwlPPIdyHOOy1gZx+qIsuZ3u4IVD4BP5/3gv OXB0uZFAiuBsg== From: alexs@kernel.org To: Vitaly Wool , Miaohe Lin , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, minchan@kernel.org, willy@infradead.org, senozhatsky@chromium.org, david@redhat.com, 42.hyeyoo@gmail.com, Yosry Ahmed , nphamcs@gmail.com Cc: Alex Shi Subject: [PATCH v2 02/20] mm/zsmalloc: use zpdesc in trylock_zspage/lock_zspage Date: Wed, 3 Jul 2024 12:05:52 +0800 Message-ID: <20240703040613.681396-3-alexs@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240703040613.681396-1-alexs@kernel.org> References: <20240703040613.681396-1-alexs@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Alex Shi To use zpdesc in trylock_zspage/lock_zspage funcs, we add couple of helpers: zpdesc_lock/zpdesc_unlock/zpdesc_trylock/zpdesc_wait_locked and zpdesc_get/zpdesc_put for this purpose. Here we use the folio series func in guts for 2 reasons, one zswap.zpool only get single page, and use folio could save some compound_head checking; two, folio_put could bypass devmap checking that we don't need. Originally-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Signed-off-by: Alex Shi --- mm/zpdesc.h | 30 ++++++++++++++++++++++++ mm/zsmalloc.c | 64 ++++++++++++++++++++++++++++++++++----------------- 2 files changed, 73 insertions(+), 21 deletions(-) diff --git a/mm/zpdesc.h b/mm/zpdesc.h index 4455a663ee84..8a359d228507 100644 --- a/mm/zpdesc.h +++ b/mm/zpdesc.h @@ -61,4 +61,34 @@ static_assert(sizeof(struct zpdesc) <=3D sizeof(struct p= age)); const struct page *: (const struct zpdesc *)(p), \ struct page *: (struct zpdesc *)(p))) =20 +static inline void zpdesc_lock(struct zpdesc *zpdesc) +{ + folio_lock(zpdesc_folio(zpdesc)); +} + +static inline bool zpdesc_trylock(struct zpdesc *zpdesc) +{ + return folio_trylock(zpdesc_folio(zpdesc)); +} + +static inline void zpdesc_unlock(struct zpdesc *zpdesc) +{ + folio_unlock(zpdesc_folio(zpdesc)); +} + +static inline void zpdesc_wait_locked(struct zpdesc *zpdesc) +{ + folio_wait_locked(zpdesc_folio(zpdesc)); +} + +static inline void zpdesc_get(struct zpdesc *zpdesc) +{ + folio_get(zpdesc_folio(zpdesc)); +} + +static inline void zpdesc_put(struct zpdesc *zpdesc) +{ + folio_put(zpdesc_folio(zpdesc)); +} + #endif diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index a532851025f9..243677a9c6d2 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -433,13 +433,17 @@ static __maybe_unused int is_first_page(struct page *= page) return PagePrivate(page); } =20 +static int is_first_zpdesc(struct zpdesc *zpdesc) +{ + return PagePrivate(zpdesc_page(zpdesc)); +} + /* Protected by class->lock */ static inline int get_zspage_inuse(struct zspage *zspage) { return zspage->inuse; } =20 - static inline void mod_zspage_inuse(struct zspage *zspage, int val) { zspage->inuse +=3D val; @@ -453,6 +457,14 @@ static inline struct page *get_first_page(struct zspag= e *zspage) return first_page; } =20 +static struct zpdesc *get_first_zpdesc(struct zspage *zspage) +{ + struct zpdesc *first_zpdesc =3D zspage->first_zpdesc; + + VM_BUG_ON_PAGE(!is_first_zpdesc(first_zpdesc), zpdesc_page(first_zpdesc)); + return first_zpdesc; +} + #define FIRST_OBJ_PAGE_TYPE_MASK 0xffff =20 static inline void reset_first_obj_offset(struct page *page) @@ -745,6 +757,16 @@ static struct page *get_next_page(struct page *page) return (struct page *)page->index; } =20 +static struct zpdesc *get_next_zpdesc(struct zpdesc *zpdesc) +{ + struct zspage *zspage =3D get_zspage(zpdesc_page(zpdesc)); + + if (unlikely(ZsHugePage(zspage))) + return NULL; + + return zpdesc->next; +} + /** * obj_to_location - get (, ) from encoded object value * @obj: the encoded object value @@ -815,11 +837,11 @@ static void reset_page(struct page *page) =20 static int trylock_zspage(struct zspage *zspage) { - struct page *cursor, *fail; + struct zpdesc *cursor, *fail; =20 - for (cursor =3D get_first_page(zspage); cursor !=3D NULL; cursor =3D - get_next_page(cursor)) { - if (!trylock_page(cursor)) { + for (cursor =3D get_first_zpdesc(zspage); cursor !=3D NULL; cursor =3D + get_next_zpdesc(cursor)) { + if (!zpdesc_trylock(cursor)) { fail =3D cursor; goto unlock; } @@ -827,9 +849,9 @@ static int trylock_zspage(struct zspage *zspage) =20 return 1; unlock: - for (cursor =3D get_first_page(zspage); cursor !=3D fail; cursor =3D - get_next_page(cursor)) - unlock_page(cursor); + for (cursor =3D get_first_zpdesc(zspage); cursor !=3D fail; cursor =3D + get_next_zpdesc(cursor)) + zpdesc_unlock(cursor); =20 return 0; } @@ -1658,7 +1680,7 @@ static int putback_zspage(struct size_class *class, s= truct zspage *zspage) */ static void lock_zspage(struct zspage *zspage) { - struct page *curr_page, *page; + struct zpdesc *curr_zpdesc, *zpdesc; =20 /* * Pages we haven't locked yet can be migrated off the list while we're @@ -1670,24 +1692,24 @@ static void lock_zspage(struct zspage *zspage) */ while (1) { migrate_read_lock(zspage); - page =3D get_first_page(zspage); - if (trylock_page(page)) + zpdesc =3D get_first_zpdesc(zspage); + if (zpdesc_trylock(zpdesc)) break; - get_page(page); + zpdesc_get(zpdesc); migrate_read_unlock(zspage); - wait_on_page_locked(page); - put_page(page); + zpdesc_wait_locked(zpdesc); + zpdesc_put(zpdesc); } =20 - curr_page =3D page; - while ((page =3D get_next_page(curr_page))) { - if (trylock_page(page)) { - curr_page =3D page; + curr_zpdesc =3D zpdesc; + while ((zpdesc =3D get_next_zpdesc(curr_zpdesc))) { + if (zpdesc_trylock(zpdesc)) { + curr_zpdesc =3D zpdesc; } else { - get_page(page); + zpdesc_get(zpdesc); migrate_read_unlock(zspage); - wait_on_page_locked(page); - put_page(page); + zpdesc_wait_locked(zpdesc); + zpdesc_put(zpdesc); migrate_read_lock(zspage); } } --=20 2.43.0 From nobody Fri Dec 19 10:39:44 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E6A4E13D282 for ; Wed, 3 Jul 2024 04:01:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719979280; cv=none; b=o2Mn2maVp2ud6DUaiEOPlIk6IM7sGKLGMN7zWwXdMT/ZZK5//GnCqdPhJhjhWEWledkyfNmlop7ONlqfVh42usJIh8VH/26yc1uuWsxanY7/OnVnZMMGhBmntHoouVXHx07v2SSoMWrLdA/c121syzDE1cpnCDX7N9QJx/VlwEU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719979280; c=relaxed/simple; bh=ezRZsoVNxvO9p/Es1oKbiCO8zRXU9uaLAm4pSSvGkg4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ltkKrivr8V9aDIM8I3kquJTolYgclSSbgKQSu6DJYSl82W0bCaJkDZqjULPJTZtQu1IvrmGj967d4WddlzhYm88Y4EbkKz6LFPBVvpjnMpKIqe6L0Q1Uvu/UwVP/YAgK+5LZlgGEoWSQKVrHtWQROSGn1lYOxdmR5bL3o161bJc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=TwygWuGt; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="TwygWuGt" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7A3EBC4AF0B; Wed, 3 Jul 2024 04:01:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1719979279; bh=ezRZsoVNxvO9p/Es1oKbiCO8zRXU9uaLAm4pSSvGkg4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=TwygWuGtgKvxgwB10GWVsnz/3isUS/MQMC9Xy3wCXH3LbpIlnFGgsLTVo3vhpbFji FiO84usSj+i0Ez8oI8g3tH5PlCJBj++GLnlkjKGdY3AV/8Dp+4wCx9PohogcDNjlzP b4ybo55igdo98aSZFekjv6ZYFaH4LyyZwRORJ+/EQx+nW04btD9BvbObiSkwN2RLn0 3YT1WOS/2mpQWRedGerzZOzYsLU5SXoRzJkQZ3Oqj52OdO3t8zXCsVNdp2h+YlWxZk NZvdf+VxMWPVrpM043z+zyIpla47SpmiLILg43uwi2uQvkL9w9ZffuwQu96pQye/yF uOlDeyIXvkq3Q== From: alexs@kernel.org To: Vitaly Wool , Miaohe Lin , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, minchan@kernel.org, willy@infradead.org, senozhatsky@chromium.org, david@redhat.com, 42.hyeyoo@gmail.com, Yosry Ahmed , nphamcs@gmail.com Cc: Alex Shi Subject: [PATCH v2 03/20] mm/zsmalloc: convert __zs_map_object/__zs_unmap_object to use zpdesc Date: Wed, 3 Jul 2024 12:05:53 +0800 Message-ID: <20240703040613.681396-4-alexs@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240703040613.681396-1-alexs@kernel.org> References: <20240703040613.681396-1-alexs@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Hyeonggon Yoo <42.hyeyoo@gmail.com> These two functions take pointer to an array of struct page. Introduce zpdesc_kmap_atomic() and make __zs_{map,unmap}_object() take pointer to an array of zpdesc instead of page. Add silly type casting when calling them. Casting will be removed late. Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Signed-off-by: Alex Shi --- mm/zsmalloc.c | 21 +++++++++++++-------- 1 file changed, 13 insertions(+), 8 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 243677a9c6d2..68a39c233d34 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -243,6 +243,11 @@ struct zs_pool { atomic_t compaction_in_progress; }; =20 +static inline void *zpdesc_kmap_atomic(struct zpdesc *zpdesc) +{ + return kmap_atomic(zpdesc_page(zpdesc)); +} + struct zspage { struct { unsigned int huge:HUGE_BITS; @@ -1061,7 +1066,7 @@ static inline void __zs_cpu_down(struct mapping_area = *area) } =20 static void *__zs_map_object(struct mapping_area *area, - struct page *pages[2], int off, int size) + struct zpdesc *zpdescs[2], int off, int size) { int sizes[2]; void *addr; @@ -1078,10 +1083,10 @@ static void *__zs_map_object(struct mapping_area *a= rea, sizes[1] =3D size - sizes[0]; =20 /* copy object to per-cpu buffer */ - addr =3D kmap_atomic(pages[0]); + addr =3D zpdesc_kmap_atomic(zpdescs[0]); memcpy(buf, addr + off, sizes[0]); kunmap_atomic(addr); - addr =3D kmap_atomic(pages[1]); + addr =3D zpdesc_kmap_atomic(zpdescs[1]); memcpy(buf + sizes[0], addr, sizes[1]); kunmap_atomic(addr); out: @@ -1089,7 +1094,7 @@ static void *__zs_map_object(struct mapping_area *are= a, } =20 static void __zs_unmap_object(struct mapping_area *area, - struct page *pages[2], int off, int size) + struct zpdesc *zpdescs[2], int off, int size) { int sizes[2]; void *addr; @@ -1108,10 +1113,10 @@ static void __zs_unmap_object(struct mapping_area *= area, sizes[1] =3D size - sizes[0]; =20 /* copy per-cpu buffer to object */ - addr =3D kmap_atomic(pages[0]); + addr =3D zpdesc_kmap_atomic(zpdescs[0]); memcpy(addr + off, buf, sizes[0]); kunmap_atomic(addr); - addr =3D kmap_atomic(pages[1]); + addr =3D zpdesc_kmap_atomic(zpdescs[1]); memcpy(addr, buf + sizes[0], sizes[1]); kunmap_atomic(addr); =20 @@ -1252,7 +1257,7 @@ void *zs_map_object(struct zs_pool *pool, unsigned lo= ng handle, pages[1] =3D get_next_page(page); BUG_ON(!pages[1]); =20 - ret =3D __zs_map_object(area, pages, off, class->size); + ret =3D __zs_map_object(area, (struct zpdesc **)pages, off, class->size); out: if (likely(!ZsHugePage(zspage))) ret +=3D ZS_HANDLE_SIZE; @@ -1287,7 +1292,7 @@ void zs_unmap_object(struct zs_pool *pool, unsigned l= ong handle) pages[1] =3D get_next_page(page); BUG_ON(!pages[1]); =20 - __zs_unmap_object(area, pages, off, class->size); + __zs_unmap_object(area, (struct zpdesc **)pages, off, class->size); } local_unlock(&zs_map_area.lock); =20 --=20 2.43.0 From nobody Fri Dec 19 10:39:44 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3D72413D603 for ; Wed, 3 Jul 2024 04:01:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719979284; cv=none; b=G4rIvDPeEBn6Fy5FvVH7oQYGmZ2Rt2EISQQbWJZ173Sg75CMmzPN7ocZQN81NG4L81gyj0eOEc5kRQ1DrySr/HtAMJiCT8GC6GQvHAH5LiUcQhdt2/KmJ5FPlNk76o+nrPS9y0xuqiVn1DaBxUP92umzkXNbZEWzTHyRF1R7kf4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719979284; c=relaxed/simple; bh=1lHtFt8ZtGmaai/qGPIEUZAmHxJr0sOHHO8xZgO8IhM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=CpFbAISQhCxf5hqJRZWvoiaxh1Wynhx3ECuREZspuqWHRMQuqdlSiOa7GqWwU+FIks+8Scsw0jAgAVTrZQ5ug1e/FNyZbUG6ogh+pNcCeXV+Rk9WNFmqquqHvYN+itlWKtyTD5bjU+HK4WqosTxh0iPaezebx7TIprrApj5Wo70= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=QNbHJxK4; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="QNbHJxK4" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7079CC4AF0A; Wed, 3 Jul 2024 04:01:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1719979283; bh=1lHtFt8ZtGmaai/qGPIEUZAmHxJr0sOHHO8xZgO8IhM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=QNbHJxK4D6ImP2YzB0FOoQm3eGnBS+mZMAHPctPm0Ee21F3ECLriusqxrK3WCcuBx j1uedLH/jQcFqFam1Pi73yNj+PUAvGE5uysX/xNPmPiDCguFqG4F7NL8MXbZMhehtH ujUyVDOrH7NOM/L8nNAhdmsn4JET++rVfKxdTdPe0uwEQewgTlPAJIgdESwevnP6iC nAukgNgU+uUI//YtUBEgMtJH4A2ICHYlwmj2s2uRNYdAJyBl1BfYuU/6imKmtohjpR w4zdLzppZBMQCbgWGjAKg+VLvML+I0IpPi7TS9qicMsbP+vF6q4waqCBsyR5CfSrtO xtz9bmUDuPS2A== From: alexs@kernel.org To: Vitaly Wool , Miaohe Lin , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, minchan@kernel.org, willy@infradead.org, senozhatsky@chromium.org, david@redhat.com, 42.hyeyoo@gmail.com, Yosry Ahmed , nphamcs@gmail.com Cc: Alex Shi Subject: [PATCH v2 04/20] mm/zsmalloc: add and use pfn/zpdesc seeking funcs Date: Wed, 3 Jul 2024 12:05:54 +0800 Message-ID: <20240703040613.681396-5-alexs@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240703040613.681396-1-alexs@kernel.org> References: <20240703040613.681396-1-alexs@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Hyeonggon Yoo <42.hyeyoo@gmail.com> Add pfn_zpdesc conversion, convert obj_to_location() to take zpdesc and also convert its users to use zpdesc. Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Signed-off-by: Alex Shi --- mm/zpdesc.h | 9 +++++++ mm/zsmalloc.c | 75 ++++++++++++++++++++++++++------------------------- 2 files changed, 47 insertions(+), 37 deletions(-) diff --git a/mm/zpdesc.h b/mm/zpdesc.h index 8a359d228507..96b9ce80ea49 100644 --- a/mm/zpdesc.h +++ b/mm/zpdesc.h @@ -91,4 +91,13 @@ static inline void zpdesc_put(struct zpdesc *zpdesc) folio_put(zpdesc_folio(zpdesc)); } =20 +static inline unsigned long zpdesc_pfn(struct zpdesc *zpdesc) +{ + return page_to_pfn(zpdesc_page(zpdesc)); +} + +static inline struct zpdesc *pfn_zpdesc(unsigned long pfn) +{ + return page_zpdesc(pfn_to_page(pfn)); +} #endif diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 68a39c233d34..149fe2b332cb 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -773,15 +773,15 @@ static struct zpdesc *get_next_zpdesc(struct zpdesc *= zpdesc) } =20 /** - * obj_to_location - get (, ) from encoded object value + * obj_to_location - get (, ) from encoded object value * @obj: the encoded object value - * @page: page object resides in zspage + * @zpdesc: zpdesc object resides in zspage * @obj_idx: object index */ -static void obj_to_location(unsigned long obj, struct page **page, +static void obj_to_location(unsigned long obj, struct zpdesc **zpdesc, unsigned int *obj_idx) { - *page =3D pfn_to_page(obj >> OBJ_INDEX_BITS); + *zpdesc =3D pfn_zpdesc(obj >> OBJ_INDEX_BITS); *obj_idx =3D (obj & OBJ_INDEX_MASK); } =20 @@ -1208,13 +1208,13 @@ void *zs_map_object(struct zs_pool *pool, unsigned = long handle, enum zs_mapmode mm) { struct zspage *zspage; - struct page *page; + struct zpdesc *zpdesc; unsigned long obj, off; unsigned int obj_idx; =20 struct size_class *class; struct mapping_area *area; - struct page *pages[2]; + struct zpdesc *zpdescs[2]; void *ret; =20 /* @@ -1227,8 +1227,8 @@ void *zs_map_object(struct zs_pool *pool, unsigned lo= ng handle, /* It guarantees it can get zspage from handle safely */ read_lock(&pool->migrate_lock); obj =3D handle_to_obj(handle); - obj_to_location(obj, &page, &obj_idx); - zspage =3D get_zspage(page); + obj_to_location(obj, &zpdesc, &obj_idx); + zspage =3D get_zspage(zpdesc_page(zpdesc)); =20 /* * migration cannot move any zpages in this zspage. Here, class->lock @@ -1247,17 +1247,17 @@ void *zs_map_object(struct zs_pool *pool, unsigned = long handle, area->vm_mm =3D mm; if (off + class->size <=3D PAGE_SIZE) { /* this object is contained entirely within a page */ - area->vm_addr =3D kmap_atomic(page); + area->vm_addr =3D zpdesc_kmap_atomic(zpdesc); ret =3D area->vm_addr + off; goto out; } =20 /* this object spans two pages */ - pages[0] =3D page; - pages[1] =3D get_next_page(page); - BUG_ON(!pages[1]); + zpdescs[0] =3D zpdesc; + zpdescs[1] =3D get_next_zpdesc(zpdesc); + BUG_ON(!zpdescs[1]); =20 - ret =3D __zs_map_object(area, (struct zpdesc **)pages, off, class->size); + ret =3D __zs_map_object(area, zpdescs, off, class->size); out: if (likely(!ZsHugePage(zspage))) ret +=3D ZS_HANDLE_SIZE; @@ -1269,7 +1269,7 @@ EXPORT_SYMBOL_GPL(zs_map_object); void zs_unmap_object(struct zs_pool *pool, unsigned long handle) { struct zspage *zspage; - struct page *page; + struct zpdesc *zpdesc; unsigned long obj, off; unsigned int obj_idx; =20 @@ -1277,8 +1277,8 @@ void zs_unmap_object(struct zs_pool *pool, unsigned l= ong handle) struct mapping_area *area; =20 obj =3D handle_to_obj(handle); - obj_to_location(obj, &page, &obj_idx); - zspage =3D get_zspage(page); + obj_to_location(obj, &zpdesc, &obj_idx); + zspage =3D get_zspage(zpdesc_page(zpdesc)); class =3D zspage_class(pool, zspage); off =3D offset_in_page(class->size * obj_idx); =20 @@ -1286,13 +1286,13 @@ void zs_unmap_object(struct zs_pool *pool, unsigned= long handle) if (off + class->size <=3D PAGE_SIZE) kunmap_atomic(area->vm_addr); else { - struct page *pages[2]; + struct zpdesc *zpdescs[2]; =20 - pages[0] =3D page; - pages[1] =3D get_next_page(page); - BUG_ON(!pages[1]); + zpdescs[0] =3D zpdesc; + zpdescs[1] =3D get_next_zpdesc(zpdesc); + BUG_ON(!zpdescs[1]); =20 - __zs_unmap_object(area, (struct zpdesc **)pages, off, class->size); + __zs_unmap_object(area, zpdescs, off, class->size); } local_unlock(&zs_map_area.lock); =20 @@ -1434,23 +1434,24 @@ static void obj_free(int class_size, unsigned long = obj) { struct link_free *link; struct zspage *zspage; - struct page *f_page; + struct zpdesc *f_zpdesc; unsigned long f_offset; unsigned int f_objidx; void *vaddr; =20 - obj_to_location(obj, &f_page, &f_objidx); + + obj_to_location(obj, &f_zpdesc, &f_objidx); f_offset =3D offset_in_page(class_size * f_objidx); - zspage =3D get_zspage(f_page); + zspage =3D get_zspage(zpdesc_page(f_zpdesc)); =20 - vaddr =3D kmap_atomic(f_page); + vaddr =3D zpdesc_kmap_atomic(f_zpdesc); link =3D (struct link_free *)(vaddr + f_offset); =20 /* Insert this object in containing zspage's freelist */ if (likely(!ZsHugePage(zspage))) link->next =3D get_freeobj(zspage) << OBJ_TAG_BITS; else - f_page->index =3D 0; + f_zpdesc->next =3D NULL; set_freeobj(zspage, f_objidx); =20 kunmap_atomic(vaddr); @@ -1495,7 +1496,7 @@ EXPORT_SYMBOL_GPL(zs_free); static void zs_object_copy(struct size_class *class, unsigned long dst, unsigned long src) { - struct page *s_page, *d_page; + struct zpdesc *s_zpdesc, *d_zpdesc; unsigned int s_objidx, d_objidx; unsigned long s_off, d_off; void *s_addr, *d_addr; @@ -1504,8 +1505,8 @@ static void zs_object_copy(struct size_class *class, = unsigned long dst, =20 s_size =3D d_size =3D class->size; =20 - obj_to_location(src, &s_page, &s_objidx); - obj_to_location(dst, &d_page, &d_objidx); + obj_to_location(src, &s_zpdesc, &s_objidx); + obj_to_location(dst, &d_zpdesc, &d_objidx); =20 s_off =3D offset_in_page(class->size * s_objidx); d_off =3D offset_in_page(class->size * d_objidx); @@ -1516,8 +1517,8 @@ static void zs_object_copy(struct size_class *class, = unsigned long dst, if (d_off + class->size > PAGE_SIZE) d_size =3D PAGE_SIZE - d_off; =20 - s_addr =3D kmap_atomic(s_page); - d_addr =3D kmap_atomic(d_page); + s_addr =3D zpdesc_kmap_atomic(s_zpdesc); + d_addr =3D zpdesc_kmap_atomic(d_zpdesc); =20 while (1) { size =3D min(s_size, d_size); @@ -1542,17 +1543,17 @@ static void zs_object_copy(struct size_class *class= , unsigned long dst, if (s_off >=3D PAGE_SIZE) { kunmap_atomic(d_addr); kunmap_atomic(s_addr); - s_page =3D get_next_page(s_page); - s_addr =3D kmap_atomic(s_page); - d_addr =3D kmap_atomic(d_page); + s_zpdesc =3D get_next_zpdesc(s_zpdesc); + s_addr =3D zpdesc_kmap_atomic(s_zpdesc); + d_addr =3D zpdesc_kmap_atomic(d_zpdesc); s_size =3D class->size - written; s_off =3D 0; } =20 if (d_off >=3D PAGE_SIZE) { kunmap_atomic(d_addr); - d_page =3D get_next_page(d_page); - d_addr =3D kmap_atomic(d_page); + d_zpdesc =3D get_next_zpdesc(d_zpdesc); + d_addr =3D zpdesc_kmap_atomic(d_zpdesc); d_size =3D class->size - written; d_off =3D 0; } @@ -1791,7 +1792,7 @@ static int zs_page_migrate(struct page *newpage, stru= ct page *page, struct zs_pool *pool; struct size_class *class; struct zspage *zspage; - struct page *dummy; + struct zpdesc *dummy; void *s_addr, *d_addr, *addr; unsigned int offset; unsigned long handle; --=20 2.43.0 From nobody Fri Dec 19 10:39:44 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0461713D88D for ; Wed, 3 Jul 2024 04:01:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719979288; cv=none; b=AmprXsSIPGog62Z/7wsh6KD1BCgc3sSleZAtNYRWo8RSYS/G02p39dWabLOKK4Rq3XbE7SJ1D13eOlbexhdzw7LflhArNikh+MaDQeDoWXnwCQmuzbueemhXotSBW2KHuEU55K4mgQS30Qe6TrtZQ1CPItO64+agVwKIeH7m13Y= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719979288; c=relaxed/simple; bh=OUw0L1+EEkxStexhrY6FwqktPPZ5HwiknqjqfzU6PXk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=i+uakoV44hfjSeVNRG4ol7IXRX3um7foebtV2OgJlKg4Z21lCRqH46jVMnoLeCi7ElFnLQJYQdoBsz6GuyyqxKAELsBkfqEhTAFKCDShyyip8ux8/PLsfRrRgXwzjHb7lbzFI0+4PM5FQ8MpTki1RR/WuXQtaDhtIKvD6WqNi5Q= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Ygl1r0eG; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Ygl1r0eG" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 69877C4AF0E; Wed, 3 Jul 2024 04:01:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1719979287; bh=OUw0L1+EEkxStexhrY6FwqktPPZ5HwiknqjqfzU6PXk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Ygl1r0eGWf9A8ASCCyxxwaRcR/wNniNavWp/6Cp1Fb7i1Ie3OgUnLu+kb/x83qU4R Rtf1OTjVxaWrDtnC9EgWP60cL+fwY6UmOlhSy0g4md3TDCs0I58Xpeo2fYMQQIC7oa mWGA7aDYGYwcXPiw7U3+CqO0HQFqXZbuFcR1HrgtLk0mCgqKSqgQeVqgpEC7iJDFyx Q4W+t234bRMiEvLqUICPWSBBwtKPv/pZtcYWjGAGrHvheKJd7nLq5kdFh/6V+jbw9b EBrXUKpPCHP3lHr26GXnoTiIoogZTVS7XgIzHuJjYkhr/Qv66r8sBvtyhi/jVwGPTl 5r1Wts9gy5n2w== From: alexs@kernel.org To: Vitaly Wool , Miaohe Lin , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, minchan@kernel.org, willy@infradead.org, senozhatsky@chromium.org, david@redhat.com, 42.hyeyoo@gmail.com, Yosry Ahmed , nphamcs@gmail.com Cc: Alex Shi Subject: [PATCH v2 05/20] mm/zsmalloc: convert obj_malloc() to use zpdesc Date: Wed, 3 Jul 2024 12:05:55 +0800 Message-ID: <20240703040613.681396-6-alexs@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240703040613.681396-1-alexs@kernel.org> References: <20240703040613.681396-1-alexs@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Hyeonggon Yoo <42.hyeyoo@gmail.com> Use get_first_zpdesc/get_next_zpdesc to replace get_first_page/get_next_page. no functional change. Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Signed-off-by: Alex Shi --- mm/zsmalloc.c | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 149fe2b332cb..bbc165cb587d 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -1322,12 +1322,12 @@ EXPORT_SYMBOL_GPL(zs_huge_class_size); static unsigned long obj_malloc(struct zs_pool *pool, struct zspage *zspage, unsigned long handle) { - int i, nr_page, offset; + int i, nr_zpdesc, offset; unsigned long obj; struct link_free *link; struct size_class *class; =20 - struct page *m_page; + struct zpdesc *m_zpdesc; unsigned long m_offset; void *vaddr; =20 @@ -1335,14 +1335,14 @@ static unsigned long obj_malloc(struct zs_pool *poo= l, obj =3D get_freeobj(zspage); =20 offset =3D obj * class->size; - nr_page =3D offset >> PAGE_SHIFT; + nr_zpdesc =3D offset >> PAGE_SHIFT; m_offset =3D offset_in_page(offset); - m_page =3D get_first_page(zspage); + m_zpdesc =3D get_first_zpdesc(zspage); =20 - for (i =3D 0; i < nr_page; i++) - m_page =3D get_next_page(m_page); + for (i =3D 0; i < nr_zpdesc; i++) + m_zpdesc =3D get_next_zpdesc(m_zpdesc); =20 - vaddr =3D kmap_atomic(m_page); + vaddr =3D zpdesc_kmap_atomic(m_zpdesc); link =3D (struct link_free *)vaddr + m_offset / sizeof(*link); set_freeobj(zspage, link->next >> OBJ_TAG_BITS); if (likely(!ZsHugePage(zspage))) @@ -1355,7 +1355,7 @@ static unsigned long obj_malloc(struct zs_pool *pool, kunmap_atomic(vaddr); mod_zspage_inuse(zspage, 1); =20 - obj =3D location_to_obj(m_page, obj); + obj =3D location_to_obj(zpdesc_page(m_zpdesc), obj); record_obj(handle, obj); =20 return obj; --=20 2.43.0 From nobody Fri Dec 19 10:39:44 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DE06E137773 for ; Wed, 3 Jul 2024 04:01:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719979291; cv=none; b=dzMXxzrYZAW3toppakzSQmjbKVLDC9rvPJFWNORXvVzFwRFT6OO7kU66rqNqyXndrNApGlpz0C3xUOaUrnBTSfSqkOSIuOYk2USeGB2bIT8zc8vXHP1cvsNTDEJQ4+4gCTrHRhkSdZJW3fxd6jaWe23WmO73BNQhidvBxRGC8dA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719979291; c=relaxed/simple; bh=m9lqRC5WWSzScStlN5X+B3diMObK1z5QGj5Hed/vEuU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=awntAYyG8GJOpVV+rj2eXZ4HsFP1mna7ZJlf9FIPReRbmTJBEZWnlXIGnHbsJ2FEAmpNFByIJ2so3xKnzSJPn1cEtPzh5AtWTML25cWXmXPe+EV7NngHCJwBjnF0X/+wTQEsgUnj4Mj4c2uQOuKTI+VUvwwXD3LkjTBcsvY10Ag= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=bQm+Zd9Y; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="bQm+Zd9Y" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 36A9CC32781; Wed, 3 Jul 2024 04:01:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1719979291; bh=m9lqRC5WWSzScStlN5X+B3diMObK1z5QGj5Hed/vEuU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=bQm+Zd9YsHA7+waQJ9X7qV3qEMjqCA8PuU6zGGx7EbaQLrn0Z+sWBiSbppZR2r5bD LZDiDUol0LFVFAj9f2gYKmMmmtQRa2wpvDhYnj8wo/VVVTm2pDoxJqY5AXqvRybYfp /kEvQmIraoSve7GzxVmDHK86iW15DsKP6dxmqedX/nC1gyopT0TfYbPlSnVRpLij98 zK7Vk450yQjq6BWnE+jvYCgwVw2NEV5p927PMkRf7B6bi+F3gXNc8S8tuywT89ogWu 8D3yW0XPH1bmSAbY+NsLVDPdVHF9X2oldOzo+EYZKFWL37gWVhFwFhsZYgwT5wkgvX aH2nAR6XEr3oA== From: alexs@kernel.org To: Vitaly Wool , Miaohe Lin , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, minchan@kernel.org, willy@infradead.org, senozhatsky@chromium.org, david@redhat.com, 42.hyeyoo@gmail.com, Yosry Ahmed , nphamcs@gmail.com Cc: Alex Shi Subject: [PATCH v2 06/20] mm/zsmalloc: convert create_page_chain() and its users to use zpdesc Date: Wed, 3 Jul 2024 12:05:56 +0800 Message-ID: <20240703040613.681396-7-alexs@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240703040613.681396-1-alexs@kernel.org> References: <20240703040613.681396-1-alexs@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Alex Shi Introduce a few helper functions for conversion to convert create_page_chai= n() to use zpdesc, then use zpdesc in replace_sub_page() too. Originally-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Signed-off-by: Alex Shi --- mm/zpdesc.h | 6 +++ mm/zsmalloc.c | 115 +++++++++++++++++++++++++++++++++----------------- 2 files changed, 82 insertions(+), 39 deletions(-) diff --git a/mm/zpdesc.h b/mm/zpdesc.h index 96b9ce80ea49..a2e320f24dd4 100644 --- a/mm/zpdesc.h +++ b/mm/zpdesc.h @@ -100,4 +100,10 @@ static inline struct zpdesc *pfn_zpdesc(unsigned long = pfn) { return page_zpdesc(pfn_to_page(pfn)); } + +static inline void __zpdesc_set_movable(struct zpdesc *zpdesc, + const struct movable_operations *mops) +{ + __SetPageMovable(zpdesc_page(zpdesc), mops); +} #endif diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index bbc165cb587d..a8f390beeab8 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -248,6 +248,41 @@ static inline void *zpdesc_kmap_atomic(struct zpdesc *= zpdesc) return kmap_atomic(zpdesc_page(zpdesc)); } =20 +static inline void zpdesc_set_zspage(struct zpdesc *zpdesc, + struct zspage *zspage) +{ + zpdesc->zspage =3D zspage; +} + +static inline void zpdesc_set_first(struct zpdesc *zpdesc) +{ + SetPagePrivate(zpdesc_page(zpdesc)); +} + +static inline void zpdesc_inc_zone_page_state(struct zpdesc *zpdesc) +{ + inc_zone_page_state(zpdesc_page(zpdesc), NR_ZSPAGES); +} + +static inline void zpdesc_dec_zone_page_state(struct zpdesc *zpdesc) +{ + dec_zone_page_state(zpdesc_page(zpdesc), NR_ZSPAGES); +} + +static inline struct zpdesc *alloc_zpdesc(gfp_t gfp) +{ + struct page *page =3D alloc_page(gfp); + + return page_zpdesc(page); +} + +static inline void free_zpdesc(struct zpdesc *zpdesc) +{ + struct page *page =3D zpdesc_page(zpdesc); + + __free_page(page); +} + struct zspage { struct { unsigned int huge:HUGE_BITS; @@ -954,35 +989,35 @@ static void init_zspage(struct size_class *class, str= uct zspage *zspage) } =20 static void create_page_chain(struct size_class *class, struct zspage *zsp= age, - struct page *pages[]) + struct zpdesc *zpdescs[]) { int i; - struct page *page; - struct page *prev_page =3D NULL; - int nr_pages =3D class->pages_per_zspage; + struct zpdesc *zpdesc; + struct zpdesc *prev_zpdesc =3D NULL; + int nr_zpdescs =3D class->pages_per_zspage; =20 /* * Allocate individual pages and link them together as: - * 1. all pages are linked together using page->index - * 2. each sub-page point to zspage using page->private + * 1. all pages are linked together using zpdesc->next + * 2. each sub-page point to zspage using zpdesc->zspage * - * we set PG_private to identify the first page (i.e. no other sub-page + * we set PG_private to identify the first zpdesc (i.e. no other zpdesc * has this flag set). */ - for (i =3D 0; i < nr_pages; i++) { - page =3D pages[i]; - set_page_private(page, (unsigned long)zspage); - page->index =3D 0; + for (i =3D 0; i < nr_zpdescs; i++) { + zpdesc =3D zpdescs[i]; + zpdesc_set_zspage(zpdesc, zspage); + zpdesc->next =3D NULL; if (i =3D=3D 0) { - zspage->first_zpdesc =3D page_zpdesc(page); - SetPagePrivate(page); + zspage->first_zpdesc =3D zpdesc; + zpdesc_set_first(zpdesc); if (unlikely(class->objs_per_zspage =3D=3D 1 && class->pages_per_zspage =3D=3D 1)) SetZsHugePage(zspage); } else { - prev_page->index =3D (unsigned long)page; + prev_zpdesc->next =3D zpdesc; } - prev_page =3D page; + prev_zpdesc =3D zpdesc; } } =20 @@ -994,7 +1029,7 @@ static struct zspage *alloc_zspage(struct zs_pool *poo= l, gfp_t gfp) { int i; - struct page *pages[ZS_MAX_PAGES_PER_ZSPAGE]; + struct zpdesc *zpdescs[ZS_MAX_PAGES_PER_ZSPAGE]; struct zspage *zspage =3D cache_alloc_zspage(pool, gfp); =20 if (!zspage) @@ -1004,25 +1039,25 @@ static struct zspage *alloc_zspage(struct zs_pool *= pool, migrate_lock_init(zspage); =20 for (i =3D 0; i < class->pages_per_zspage; i++) { - struct page *page; + struct zpdesc *zpdesc; =20 - page =3D alloc_page(gfp); - if (!page) { + zpdesc =3D alloc_zpdesc(gfp); + if (!zpdesc) { while (--i >=3D 0) { - dec_zone_page_state(pages[i], NR_ZSPAGES); - __ClearPageZsmalloc(pages[i]); - __free_page(pages[i]); + zpdesc_dec_zone_page_state(zpdescs[i]); + __ClearPageZsmalloc(zpdesc_page(zpdescs[i])); + free_zpdesc(zpdescs[i]); } cache_free_zspage(pool, zspage); return NULL; } - __SetPageZsmalloc(page); + __SetPageZsmalloc(zpdesc_page(zpdesc)); =20 - inc_zone_page_state(page, NR_ZSPAGES); - pages[i] =3D page; + zpdesc_inc_zone_page_state(zpdesc); + zpdescs[i] =3D zpdesc; } =20 - create_page_chain(class, zspage, pages); + create_page_chain(class, zspage, zpdescs); init_zspage(class, zspage); zspage->pool =3D pool; zspage->class =3D class->index; @@ -1753,26 +1788,28 @@ static void migrate_write_unlock(struct zspage *zsp= age) static const struct movable_operations zsmalloc_mops; =20 static void replace_sub_page(struct size_class *class, struct zspage *zspa= ge, - struct page *newpage, struct page *oldpage) + struct zpdesc *newzpdesc, struct zpdesc *oldzpdesc) { - struct page *page; - struct page *pages[ZS_MAX_PAGES_PER_ZSPAGE] =3D {NULL, }; + struct zpdesc *zpdesc; + struct zpdesc *zpdescs[ZS_MAX_PAGES_PER_ZSPAGE] =3D {NULL, }; + unsigned int first_obj_offset; int idx =3D 0; =20 - page =3D get_first_page(zspage); + zpdesc =3D get_first_zpdesc(zspage); do { - if (page =3D=3D oldpage) - pages[idx] =3D newpage; + if (zpdesc =3D=3D oldzpdesc) + zpdescs[idx] =3D newzpdesc; else - pages[idx] =3D page; + zpdescs[idx] =3D zpdesc; idx++; - } while ((page =3D get_next_page(page)) !=3D NULL); + } while ((zpdesc =3D get_next_zpdesc(zpdesc)) !=3D NULL); =20 - create_page_chain(class, zspage, pages); - set_first_obj_offset(newpage, get_first_obj_offset(oldpage)); + create_page_chain(class, zspage, zpdescs); + first_obj_offset =3D get_first_obj_offset(zpdesc_page(oldzpdesc)); + set_first_obj_offset(zpdesc_page(newzpdesc), first_obj_offset); if (unlikely(ZsHugePage(zspage))) - newpage->index =3D oldpage->index; - __SetPageMovable(newpage, &zsmalloc_mops); + newzpdesc->handle =3D oldzpdesc->handle; + __zpdesc_set_movable(newzpdesc, &zsmalloc_mops); } =20 static bool zs_page_isolate(struct page *page, isolate_mode_t mode) @@ -1845,7 +1882,7 @@ static int zs_page_migrate(struct page *newpage, stru= ct page *page, } kunmap_atomic(s_addr); =20 - replace_sub_page(class, zspage, newpage, page); + replace_sub_page(class, zspage, page_zpdesc(newpage), page_zpdesc(page)); /* * Since we complete the data copy and set up new zspage structure, * it's okay to release migration_lock. --=20 2.43.0 From nobody Fri Dec 19 10:39:44 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C9F5D13DBBD for ; Wed, 3 Jul 2024 04:01:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719979295; cv=none; b=c8rvm4IsKUEJHEy1AzGtRBLrKtx09ucmqBzuQnv09MAIrafVEWP+P4RuEfRFMPkGCLiq5ewh2HM4w118XqfFgnlFGimCLnBAqBqfc6IdlJF/RNBwXaAdpLRbOmc2VYy6ixQSWk4/rWmFCDSk3sb92lZPv1nS7XZqgTHSqUygmk8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719979295; c=relaxed/simple; bh=82zz9HxUIo/V08TGVJ1E1ZkMfjqggCMIVYH80TRsdU8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=gj4NoGx3ZxOQFAi7xU6SW1H27Ql+Rzz1jcZ81UrJwhXYWzzXNc4N96p3ZIDOgEi8buofL6cUczjfJarNAjr6PeEjED/WpSLLF27w+2LAY9cacEGQd3K8NPg91u6WrRWetK9ZHb5ZWWGPflbd4nc4B9geeZFSul05LSH8PmK5lNg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=PE3Mp3iK; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="PE3Mp3iK" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 2D144C32781; Wed, 3 Jul 2024 04:01:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1719979295; bh=82zz9HxUIo/V08TGVJ1E1ZkMfjqggCMIVYH80TRsdU8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=PE3Mp3iKrGg+MQDziwiQtLqamfGDcd2BwzU0p66UpdNRzzHqrtP5zJJ9n9wogEL/9 P7K8fhmXl4L80J6DnfajNfBWAagd+7MGFej4mxZtsPMEvifMWkC8+EZmgbWyOy0Cwr LLmeNC4OLQaUHi4I0If9evvI/v/cZiITs3kNJWrM8rqSVbiAYS+paY30MeeHGZieo1 teIsVLI7MRzFlZhYW9wQutQkrZ+zhLQHFHUYWEcVOqBnkcAUUmd+yvFVqr51dTSUZD bsX/vwbYMqctqwgH4SyV5nxPb1Mib2Lyir1b/Wxfm+yvAeRhSx1GO1BEpiQtU0UJDc oF+pN+0ejOKJg== From: alexs@kernel.org To: Vitaly Wool , Miaohe Lin , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, minchan@kernel.org, willy@infradead.org, senozhatsky@chromium.org, david@redhat.com, 42.hyeyoo@gmail.com, Yosry Ahmed , nphamcs@gmail.com Cc: Alex Shi Subject: [PATCH v2 07/20] mm/zsmalloc: convert obj_allocated() and related helpers to use zpdesc Date: Wed, 3 Jul 2024 12:05:57 +0800 Message-ID: <20240703040613.681396-8-alexs@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240703040613.681396-1-alexs@kernel.org> References: <20240703040613.681396-1-alexs@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Hyeonggon Yoo <42.hyeyoo@gmail.com> Convert obj_allocated(), and related helpers to take zpdesc. Also make its callers to cast (struct page *) to (struct zpdesc *) when calling them. The users will be converted gradually as there are many. Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Signed-off-by: Alex Shi --- mm/zsmalloc.c | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index a8f390beeab8..29b9fa5baa46 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -845,15 +845,15 @@ static unsigned long handle_to_obj(unsigned long hand= le) return *(unsigned long *)handle; } =20 -static inline bool obj_allocated(struct page *page, void *obj, +static inline bool obj_allocated(struct zpdesc *zpdesc, void *obj, unsigned long *phandle) { unsigned long handle; - struct zspage *zspage =3D get_zspage(page); + struct zspage *zspage =3D get_zspage(zpdesc_page(zpdesc)); =20 if (unlikely(ZsHugePage(zspage))) { - VM_BUG_ON_PAGE(!is_first_page(page), page); - handle =3D page->index; + VM_BUG_ON_PAGE(!is_first_zpdesc(zpdesc), zpdesc_page(zpdesc)); + handle =3D zpdesc->handle; } else handle =3D *(unsigned long *)obj; =20 @@ -1603,18 +1603,18 @@ static void zs_object_copy(struct size_class *class= , unsigned long dst, * return handle. */ static unsigned long find_alloced_obj(struct size_class *class, - struct page *page, int *obj_idx) + struct zpdesc *zpdesc, int *obj_idx) { unsigned int offset; int index =3D *obj_idx; unsigned long handle =3D 0; - void *addr =3D kmap_atomic(page); + void *addr =3D zpdesc_kmap_atomic(zpdesc); =20 - offset =3D get_first_obj_offset(page); + offset =3D get_first_obj_offset(zpdesc_page(zpdesc)); offset +=3D class->size * index; =20 while (offset < PAGE_SIZE) { - if (obj_allocated(page, addr + offset, &handle)) + if (obj_allocated(zpdesc, addr + offset, &handle)) break; =20 offset +=3D class->size; @@ -1638,7 +1638,7 @@ static void migrate_zspage(struct zs_pool *pool, stru= ct zspage *src_zspage, struct size_class *class =3D pool->size_class[src_zspage->class]; =20 while (1) { - handle =3D find_alloced_obj(class, s_page, &obj_idx); + handle =3D find_alloced_obj(class, page_zpdesc(s_page), &obj_idx); if (!handle) { s_page =3D get_next_page(s_page); if (!s_page) @@ -1871,7 +1871,7 @@ static int zs_page_migrate(struct page *newpage, stru= ct page *page, =20 for (addr =3D s_addr + offset; addr < s_addr + PAGE_SIZE; addr +=3D class->size) { - if (obj_allocated(page, addr, &handle)) { + if (obj_allocated(page_zpdesc(page), addr, &handle)) { =20 old_obj =3D handle_to_obj(handle); obj_to_location(old_obj, &dummy, &obj_idx); --=20 2.43.0 From nobody Fri Dec 19 10:39:44 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 77131142625 for ; Wed, 3 Jul 2024 04:01:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719979299; cv=none; b=LGcmVfH/u7VaNd3K9Z9nN5vgOX28oH136jxpMv2cFkWKcekC8LjG/JocdKZMD6YlM3YytVfrhTtDu8ceJa1nJkb58BqKNE864zBLP5lWMUSTK/eSsaDDEhzC9mloniwbjX4Vd3MZlCw8+Bme/ikrHydjQtTWJZ9qXJ8wqMY/Rug= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719979299; c=relaxed/simple; bh=hCA+Wp2koQFPErK04B2puEI5lWDR1Rh4V60AMISo3bA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=b15v7B7ButIqces3ZSeSzbX2DT3D0vrrO9gGtjBufr6+sYjVII4OTEDCg/MkAR0IzSmVty1kS3udmGt+j8zkqsN+UdCtj5eLmAfDZltAS48jNhKwNo7Jz2qIPXWd8sRysgzWoGgMCAPtqPXFEMWX3ThOBWTEULi9dNX6/41LqdM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=ssU8oOZS; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="ssU8oOZS" Received: by smtp.kernel.org (Postfix) with ESMTPSA id ED24BC32781; Wed, 3 Jul 2024 04:01:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1719979299; bh=hCA+Wp2koQFPErK04B2puEI5lWDR1Rh4V60AMISo3bA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ssU8oOZS8VnZ5kjtvdEP4lDcKyV/8A6M1yM0c3F1KIS8eUI6cNiQvdY87Bb6YlG1V 9b2Qoy4jCAjauQk9G45OXT/p5dW+Jpq9PeP5AI1GVkm+G5B19918JQ/pLqfAPCXsWg c93b4mZRWvKhOEAtTHlXnN8MLEBdsiyswP0TRu5NP202rpZDdQ7/eOVWygU8xreES0 uL4HDJpE4IYOajppi6Nr4yiyTp5ZzDwOjGh6ADzC54ITody68TXzrudEUTH1gWDv3i UwlI4ku7/0UD8QVHuSqhSnWNpJFCsL4RazQOmUsYJCjQWblxy1h9oW+ERZ86wiRePZ AzhbrwkhjIkLg== From: alexs@kernel.org To: Vitaly Wool , Miaohe Lin , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, minchan@kernel.org, willy@infradead.org, senozhatsky@chromium.org, david@redhat.com, 42.hyeyoo@gmail.com, Yosry Ahmed , nphamcs@gmail.com Cc: Alex Shi Subject: [PATCH v2 08/20] mm/zsmalloc: convert init_zspage() to use zpdesc Date: Wed, 3 Jul 2024 12:05:58 +0800 Message-ID: <20240703040613.681396-9-alexs@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240703040613.681396-1-alexs@kernel.org> References: <20240703040613.681396-1-alexs@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Hyeonggon Yoo <42.hyeyoo@gmail.com> Replace get_first/next_page func series and kmap_atomic to new helper, no functional change. Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Signed-off-by: Alex Shi --- mm/zsmalloc.c | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 29b9fa5baa46..d3558f3f8bc3 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -948,16 +948,16 @@ static void init_zspage(struct size_class *class, str= uct zspage *zspage) { unsigned int freeobj =3D 1; unsigned long off =3D 0; - struct page *page =3D get_first_page(zspage); + struct zpdesc *zpdesc =3D get_first_zpdesc(zspage); =20 - while (page) { - struct page *next_page; + while (zpdesc) { + struct zpdesc *next_zpdesc; struct link_free *link; void *vaddr; =20 - set_first_obj_offset(page, off); + set_first_obj_offset(zpdesc_page(zpdesc), off); =20 - vaddr =3D kmap_atomic(page); + vaddr =3D zpdesc_kmap_atomic(zpdesc); link =3D (struct link_free *)vaddr + off / sizeof(*link); =20 while ((off +=3D class->size) < PAGE_SIZE) { @@ -970,8 +970,8 @@ static void init_zspage(struct size_class *class, struc= t zspage *zspage) * page, which must point to the first object on the next * page (if present) */ - next_page =3D get_next_page(page); - if (next_page) { + next_zpdesc =3D get_next_zpdesc(zpdesc); + if (next_zpdesc) { link->next =3D freeobj++ << OBJ_TAG_BITS; } else { /* @@ -981,7 +981,7 @@ static void init_zspage(struct size_class *class, struc= t zspage *zspage) link->next =3D -1UL << OBJ_TAG_BITS; } kunmap_atomic(vaddr); - page =3D next_page; + zpdesc =3D next_zpdesc; off %=3D PAGE_SIZE; } =20 --=20 2.43.0 From nobody Fri Dec 19 10:39:44 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 90A95137C25 for ; Wed, 3 Jul 2024 04:01:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719979303; cv=none; b=BXO33PWvIhXYy5YrAK6pJmEFC1NYAaDhD7XW+04p8LAZsDiMiBzggnKtCS7WLShFKK3cd8SSQdXhc5uj70uKoYsK9GWeo5VmRF9MRi6ccmYdhlRCsNiJCboCFASHkD3SYbHJrmjSoGQRAN8H7AX8dopZIdBJFXL57raA8A+ozVk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719979303; c=relaxed/simple; bh=cpsRKinL3S7LbxZnzQMdHFXcR2a2bxLsePv8cUuJa+o=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=qx5sqftZzSuwfH3T6pIio0iIPSIPSUXUxRGyzXr+EAviP0+UPc3jlnxQSfXt/+Fiu7R/QHQU3fQg1kJDiU6C5Wv3/e0Ez/3UNJTtwvam8hu6iUV8Uw8zcAEk6UtPbM1ZIGS8i0+Ngx+mzdPlvNm0xcplt9bzQ9UmMgPAgrDU5uM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=h/vn2l9o; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="h/vn2l9o" Received: by smtp.kernel.org (Postfix) with ESMTPSA id B9464C4AF0E; Wed, 3 Jul 2024 04:01:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1719979303; bh=cpsRKinL3S7LbxZnzQMdHFXcR2a2bxLsePv8cUuJa+o=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=h/vn2l9o3YSdVOjafhW6qL2RUpxiwj6dLECE60UFuR7t8SZfG4edNba8A/qzaWYOu FtDjnMdg7TiMbLrqMf4E5P4XVWzIrH4IQ2A8brhdPHmObwWoKyZSngO9YkKZdafPBC VcUl09u2ChPvdiYH8WCn/LocRf/ff/wUNAE3SdawSUx+9HQEWHIUOw8LeF65Z15POI 1EmLkS2S5y55raDFiV3ohsvIKoFNH3jumJfY+f7TobhdeFJyrjTqOoF5oH6ZlIsSTt 3Jq7Xy/bhl2AnZ8am4wTsCkWnlp8TBUcPD/dlOh2lY3B5IaxKoCmRyBdS0hpGQz5od KnkYo94ZbqgjA== From: alexs@kernel.org To: Vitaly Wool , Miaohe Lin , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, minchan@kernel.org, willy@infradead.org, senozhatsky@chromium.org, david@redhat.com, 42.hyeyoo@gmail.com, Yosry Ahmed , nphamcs@gmail.com Cc: Alex Shi Subject: [PATCH v2 09/20] mm/zsmalloc: convert obj_to_page() and zs_free() to use zpdesc Date: Wed, 3 Jul 2024 12:05:59 +0800 Message-ID: <20240703040613.681396-10-alexs@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240703040613.681396-1-alexs@kernel.org> References: <20240703040613.681396-1-alexs@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Hyeonggon Yoo <42.hyeyoo@gmail.com> Rename obj_to_page() to obj_to_zpdesc() and also convert it and its user zs_free() to use zpdesc. Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Signed-off-by: Alex Shi --- mm/zsmalloc.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index d3558f3f8bc3..7aa4a4acaec9 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -820,9 +820,9 @@ static void obj_to_location(unsigned long obj, struct z= pdesc **zpdesc, *obj_idx =3D (obj & OBJ_INDEX_MASK); } =20 -static void obj_to_page(unsigned long obj, struct page **page) +static void obj_to_zpdesc(unsigned long obj, struct zpdesc **zpdesc) { - *page =3D pfn_to_page(obj >> OBJ_INDEX_BITS); + *zpdesc =3D pfn_zpdesc(obj >> OBJ_INDEX_BITS); } =20 /** @@ -1496,7 +1496,7 @@ static void obj_free(int class_size, unsigned long ob= j) void zs_free(struct zs_pool *pool, unsigned long handle) { struct zspage *zspage; - struct page *f_page; + struct zpdesc *f_zpdesc; unsigned long obj; struct size_class *class; int fullness; @@ -1510,8 +1510,8 @@ void zs_free(struct zs_pool *pool, unsigned long hand= le) */ read_lock(&pool->migrate_lock); obj =3D handle_to_obj(handle); - obj_to_page(obj, &f_page); - zspage =3D get_zspage(f_page); + obj_to_zpdesc(obj, &f_zpdesc); + zspage =3D get_zspage(zpdesc_page(f_zpdesc)); class =3D zspage_class(pool, zspage); spin_lock(&class->lock); read_unlock(&pool->migrate_lock); --=20 2.43.0 From nobody Fri Dec 19 10:39:44 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1F8F71384B9 for ; Wed, 3 Jul 2024 04:01:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719979307; cv=none; b=sXdxMch7TjB88OUpJkyBgNIDLwHo1lytYnC+kdXZGKYP9qHVzSgWWALu4KZaWYdB0pe0DPnFSXSLD/9vuUNbhIeW/lB3SW3sYas5dP1n8CLFeKxZ7aJgpP5cxtwu+Z2QSOPO1zEl3uEOiqPKeZyGUJtWGMsoCoeIWEbiKemc2Ms= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719979307; c=relaxed/simple; bh=X6q7HiV/JKEJWx8fJkY9r9rnNkPwseyxy5Uf/fZ/T2M=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=uHvyTkqJYiLl0Y1Qv0mS29A1R8QgnBuI7nLw+6YTf3BmUI8DUZpAmyRTwZqO5dhYopQrSbL5kObqL42h7cIA6BaymeolspcYP8XdYJX9ByDsqH4iEE4FRkbHRnIu8v+OSJZACGF3SXuNqGMP33TewSCkCaK/JWBRjNJw7CBQBqU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=G0B04OIt; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="G0B04OIt" Received: by smtp.kernel.org (Postfix) with ESMTPSA id AF65AC32781; Wed, 3 Jul 2024 04:01:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1719979307; bh=X6q7HiV/JKEJWx8fJkY9r9rnNkPwseyxy5Uf/fZ/T2M=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=G0B04OItwHRPyh+8sULIxpolV9zSD+VmOfgGiZ3Y3gonxy225OS8tS5EBr//Ixoi9 CGJLSxDUAUyX0z+bVgXNHpaQFluz/fuqNE+w2SNDereXSgVjKrHUl5iUM9F+xalXIg HUj4yZYZT2U2YtJvt7grh2npA+NoNsjvRWnII0qLazNTDES26JXJqTmwLz90DAg0G0 c/YrOuXaWyExREG3YPDdvo751cKKvV7kcj8kaFVEH/LpVWeNFJO0YWnOtY/qa7UjP+ z7d1o1kNYc+2puqu1lkayptxyVMKKn9z4kj7znaF0PH6/Ey8uz6qESrnX9LdeDE3NA JoLmzRg7/SuGw== From: alexs@kernel.org To: Vitaly Wool , Miaohe Lin , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, minchan@kernel.org, willy@infradead.org, senozhatsky@chromium.org, david@redhat.com, 42.hyeyoo@gmail.com, Yosry Ahmed , nphamcs@gmail.com Cc: Alex Shi Subject: [PATCH v2 10/20] mm/zsmalloc: add zpdesc_is_isolated/zpdesc_zone helper for zs_page_migrate Date: Wed, 3 Jul 2024 12:06:00 +0800 Message-ID: <20240703040613.681396-11-alexs@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240703040613.681396-1-alexs@kernel.org> References: <20240703040613.681396-1-alexs@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Hyeonggon Yoo <42.hyeyoo@gmail.com> To convert page to zpdesc in zs_page_migrate(), we added zpdesc_is_isolated/zpdesc_zone helpers. No functional change. Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Signed-off-by: Alex Shi --- mm/zpdesc.h | 11 +++++++++++ mm/zsmalloc.c | 30 ++++++++++++++++-------------- 2 files changed, 27 insertions(+), 14 deletions(-) diff --git a/mm/zpdesc.h b/mm/zpdesc.h index a2e320f24dd4..61d7ba0c626e 100644 --- a/mm/zpdesc.h +++ b/mm/zpdesc.h @@ -106,4 +106,15 @@ static inline void __zpdesc_set_movable(struct zpdesc = *zpdesc, { __SetPageMovable(zpdesc_page(zpdesc), mops); } + +static inline bool zpdesc_is_isolated(struct zpdesc *zpdesc) +{ + return PageIsolated(zpdesc_page(zpdesc)); +} + +static inline struct zone *zpdesc_zone(struct zpdesc *zpdesc) +{ + return page_zone(zpdesc_page(zpdesc)); +} + #endif diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 7aa4a4acaec9..9bc9b14187ed 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -1830,19 +1830,21 @@ static int zs_page_migrate(struct page *newpage, st= ruct page *page, struct size_class *class; struct zspage *zspage; struct zpdesc *dummy; + struct zpdesc *newzpdesc =3D page_zpdesc(newpage); + struct zpdesc *zpdesc =3D page_zpdesc(page); void *s_addr, *d_addr, *addr; unsigned int offset; unsigned long handle; unsigned long old_obj, new_obj; unsigned int obj_idx; =20 - VM_BUG_ON_PAGE(!PageIsolated(page), page); + VM_BUG_ON_PAGE(!zpdesc_is_isolated(zpdesc), zpdesc_page(zpdesc)); =20 /* We're committed, tell the world that this is a Zsmalloc page. */ - __SetPageZsmalloc(newpage); + __SetPageZsmalloc(zpdesc_page(newzpdesc)); =20 /* The page is locked, so this pointer must remain valid */ - zspage =3D get_zspage(page); + zspage =3D get_zspage(zpdesc_page(zpdesc)); pool =3D zspage->pool; =20 /* @@ -1859,30 +1861,30 @@ static int zs_page_migrate(struct page *newpage, st= ruct page *page, /* the migrate_write_lock protects zpage access via zs_map_object */ migrate_write_lock(zspage); =20 - offset =3D get_first_obj_offset(page); - s_addr =3D kmap_atomic(page); + offset =3D get_first_obj_offset(zpdesc_page(zpdesc)); + s_addr =3D zpdesc_kmap_atomic(zpdesc); =20 /* * Here, any user cannot access all objects in the zspage so let's move. */ - d_addr =3D kmap_atomic(newpage); + d_addr =3D zpdesc_kmap_atomic(newzpdesc); copy_page(d_addr, s_addr); kunmap_atomic(d_addr); =20 for (addr =3D s_addr + offset; addr < s_addr + PAGE_SIZE; addr +=3D class->size) { - if (obj_allocated(page_zpdesc(page), addr, &handle)) { + if (obj_allocated(zpdesc, addr, &handle)) { =20 old_obj =3D handle_to_obj(handle); obj_to_location(old_obj, &dummy, &obj_idx); - new_obj =3D (unsigned long)location_to_obj(newpage, + new_obj =3D (unsigned long)location_to_obj(zpdesc_page(newzpdesc), obj_idx); record_obj(handle, new_obj); } } kunmap_atomic(s_addr); =20 - replace_sub_page(class, zspage, page_zpdesc(newpage), page_zpdesc(page)); + replace_sub_page(class, zspage, newzpdesc, zpdesc); /* * Since we complete the data copy and set up new zspage structure, * it's okay to release migration_lock. @@ -1891,14 +1893,14 @@ static int zs_page_migrate(struct page *newpage, st= ruct page *page, spin_unlock(&class->lock); migrate_write_unlock(zspage); =20 - get_page(newpage); - if (page_zone(newpage) !=3D page_zone(page)) { - dec_zone_page_state(page, NR_ZSPAGES); - inc_zone_page_state(newpage, NR_ZSPAGES); + zpdesc_get(newzpdesc); + if (zpdesc_zone(newzpdesc) !=3D zpdesc_zone(zpdesc)) { + zpdesc_dec_zone_page_state(zpdesc); + zpdesc_inc_zone_page_state(newzpdesc); } =20 reset_page(page); - put_page(page); + zpdesc_put(zpdesc); =20 return MIGRATEPAGE_SUCCESS; } --=20 2.43.0 From nobody Fri Dec 19 10:39:44 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4585022EE8 for ; Wed, 3 Jul 2024 04:01:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719979311; cv=none; b=X+xltNvvDQ8G9GRAjHGxjpqiykt1V7/FR6XJhFAOEcUKcbwNZkTKFuv9BjSqC5MLX9Y7A+hgLWlzbR+QBNx+W9+vwwpTxyPRk2pMzhOt3ynx3voEsgJrGvPxQH1iHqWsZZ61/Ua/6QGhtO91I3nbmnUi4d9/C+k17uGsMU18nV8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719979311; c=relaxed/simple; bh=d7GxkLwHXdQyb6XsJEXUauGOBeH6/SyL3Iuz9vNzPwE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=nqom+MaH3znl6yIx2LsMYxE4Qj6jgsdBnP7wHbQ8Z8GERQ48vNZLu4U83NFqAE7ChovsfaOmwdXF8mzaawsaNZTX3bKDkERPzlXu2vqqcpXE/NmemaGltUTzJ+RDRls1e800Kojt2yEegMtws3be9gTRu/wU7hgUZWjXS4RYde8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=NpDb5OJz; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="NpDb5OJz" Received: by smtp.kernel.org (Postfix) with ESMTPSA id AF31EC4AF0B; Wed, 3 Jul 2024 04:01:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1719979310; bh=d7GxkLwHXdQyb6XsJEXUauGOBeH6/SyL3Iuz9vNzPwE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=NpDb5OJzKlBHlmXy9eSfpYJA66+cDPSqPT/zmcenz11JqH6X7j4xggFBVksX4hHDr hrjebT56ic9Us0wcGrhTL9GrCCBcw2YKhHWj5Y/X7PXOoNg8AJDlaRCV1/bY5oVTjW 7reZHFTfGlB5U+5CYeaRbolfCWAWin7oIgpneC1lVlHKzPLBeY8NnMzeV6hQm2+Uq+ dyrqduH1axDvKQqlOLJqHrP2wLCJgyLxxzsoUfUu5MC/7ZxEDhAjZdajB8AmIPr8yV YuDrR4c//Kevf275D86Y0FW1sKewkadQl9n708d6mCQmWAu/FzFHMBk7NsFvLKFIyz O1sJ+oFdqQ6xA== From: alexs@kernel.org To: Vitaly Wool , Miaohe Lin , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, minchan@kernel.org, willy@infradead.org, senozhatsky@chromium.org, david@redhat.com, 42.hyeyoo@gmail.com, Yosry Ahmed , nphamcs@gmail.com Cc: Alex Shi Subject: [PATCH v2 11/20] mm/zsmalloc: rename reset_page to reset_zpdesc and use zpdesc in it Date: Wed, 3 Jul 2024 12:06:01 +0800 Message-ID: <20240703040613.681396-12-alexs@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240703040613.681396-1-alexs@kernel.org> References: <20240703040613.681396-1-alexs@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Alex Shi zpdesc.zspage matches with page.private, zpdesc.next matches with page.index. They will be reset in reset_page() wich is called prior to free base pages of a zspage. Use zpdesc to replace page struct and rename it to reset_zpdesc(), few page helper still left since they are used too widely. Signed-off-by: Alex Shi --- mm/zsmalloc.c | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 9bc9b14187ed..6d1971836391 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -865,12 +865,14 @@ static inline bool obj_allocated(struct zpdesc *zpdes= c, void *obj, return true; } =20 -static void reset_page(struct page *page) +static void reset_zpdesc(struct zpdesc *zpdesc) { + struct page *page =3D zpdesc_page(zpdesc); + __ClearPageMovable(page); ClearPagePrivate(page); - set_page_private(page, 0); - page->index =3D 0; + zpdesc->zspage =3D NULL; + zpdesc->next =3D NULL; reset_first_obj_offset(page); __ClearPageZsmalloc(page); } @@ -910,7 +912,7 @@ static void __free_zspage(struct zs_pool *pool, struct = size_class *class, do { VM_BUG_ON_PAGE(!PageLocked(page), page); next =3D get_next_page(page); - reset_page(page); + reset_zpdesc(page_zpdesc(page)); unlock_page(page); dec_zone_page_state(page, NR_ZSPAGES); put_page(page); @@ -1899,7 +1901,7 @@ static int zs_page_migrate(struct page *newpage, stru= ct page *page, zpdesc_inc_zone_page_state(newzpdesc); } =20 - reset_page(page); + reset_zpdesc(zpdesc); zpdesc_put(zpdesc); =20 return MIGRATEPAGE_SUCCESS; --=20 2.43.0 From nobody Fri Dec 19 10:39:44 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E2825158879 for ; Wed, 3 Jul 2024 04:01:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719979315; cv=none; b=JbbmmHkJNZAS+/5vmALz4k0EQwowm37OJOu1tk8LPYt7fbIyfQ6AiHtjG7tCRvmo8vSVtgX/g8xIRJJb7fFCu3NnP4W/b0yQEgb375kkgw07Cejch+SsEPZwJfPUOj2vc2hUJKlLawiD3nG+ehC8L0D5bqfPFlELvM3GXyPZX/U= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719979315; c=relaxed/simple; bh=rKWZuiJgMM3PFhIFbKbki7RwNRU1BYQ1UAI5bUa3ql8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=GTvHQyLxrpzG7EicTVsayxnqFmozIfAdf3Q9lYy2Q1FDvDtWhVclCFeKqhKO3PjA1mc9Ldr3Opd8Nxe+BXbgtkdU8VZIKSAGIsegX18b7NMtmiJbJL9W4I4Ww0o5UgvicIHnLucmhR4Pexyd1qXHzw09OqGFijjBR22YhehIpts= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=f1t7hQXJ; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="f1t7hQXJ" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7B8A6C32781; Wed, 3 Jul 2024 04:01:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1719979314; bh=rKWZuiJgMM3PFhIFbKbki7RwNRU1BYQ1UAI5bUa3ql8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=f1t7hQXJQvhCEgu9b1n3nE1YlFOxjEC/c/pUUWyfXG22Tw8QUwfCTUD9VZwL94fSY jv3/8d/eHpn1e+fQuoItWy8ZyrfsKIGYgO1xWlKssWAy9Twg9jtJDGp5cvknh2HJWd W5YnUonQE4WtBpZK6/z/Pci54CDg8ZV0li79TV2nbsIfnaWFZcSmGtqrgFhEgf8bOO OgWHm/ztVeQ56/PIb+6/rkbf8o6xP5OOTqlSNjwL6lQI1m8k5SNG5ocMCkzRy7TSwM Nqt/bERDQCtXlJL3/j3SpvyzKvCox2IVzTHeBDk2M3v70gQMtckTQ2XRIaKeFXX8LZ FKT2rzHyQimFQ== From: alexs@kernel.org To: Vitaly Wool , Miaohe Lin , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, minchan@kernel.org, willy@infradead.org, senozhatsky@chromium.org, david@redhat.com, 42.hyeyoo@gmail.com, Yosry Ahmed , nphamcs@gmail.com Cc: Alex Shi Subject: [PATCH v2 12/20] mm/zsmalloc: convert __free_zspage() to use zdsesc Date: Wed, 3 Jul 2024 12:06:02 +0800 Message-ID: <20240703040613.681396-13-alexs@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240703040613.681396-1-alexs@kernel.org> References: <20240703040613.681396-1-alexs@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Hyeonggon Yoo <42.hyeyoo@gmail.com> Introduce zpdesc_is_locked() and convert __free_zspage() to use zpdesc. Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Signed-off-by: Alex Shi --- mm/zpdesc.h | 4 ++++ mm/zsmalloc.c | 20 ++++++++++---------- 2 files changed, 14 insertions(+), 10 deletions(-) diff --git a/mm/zpdesc.h b/mm/zpdesc.h index 61d7ba0c626e..6de3c5d095bd 100644 --- a/mm/zpdesc.h +++ b/mm/zpdesc.h @@ -117,4 +117,8 @@ static inline struct zone *zpdesc_zone(struct zpdesc *z= pdesc) return page_zone(zpdesc_page(zpdesc)); } =20 +static inline bool zpdesc_is_locked(struct zpdesc *zpdesc) +{ + return PageLocked(zpdesc_page(zpdesc)); +} #endif diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 6d1971836391..68fdea7b6e0d 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -901,23 +901,23 @@ static int trylock_zspage(struct zspage *zspage) static void __free_zspage(struct zs_pool *pool, struct size_class *class, struct zspage *zspage) { - struct page *page, *next; + struct zpdesc *zpdesc, *next; =20 assert_spin_locked(&class->lock); =20 VM_BUG_ON(get_zspage_inuse(zspage)); VM_BUG_ON(zspage->fullness !=3D ZS_INUSE_RATIO_0); =20 - next =3D page =3D get_first_page(zspage); + next =3D zpdesc =3D get_first_zpdesc(zspage); do { - VM_BUG_ON_PAGE(!PageLocked(page), page); - next =3D get_next_page(page); - reset_zpdesc(page_zpdesc(page)); - unlock_page(page); - dec_zone_page_state(page, NR_ZSPAGES); - put_page(page); - page =3D next; - } while (page !=3D NULL); + VM_BUG_ON_PAGE(!zpdesc_is_locked(zpdesc), zpdesc_page(zpdesc)); + next =3D get_next_zpdesc(zpdesc); + reset_zpdesc(zpdesc); + zpdesc_unlock(zpdesc); + zpdesc_dec_zone_page_state(zpdesc); + zpdesc_put(zpdesc); + zpdesc =3D next; + } while (zpdesc !=3D NULL); =20 cache_free_zspage(pool, zspage); =20 --=20 2.43.0 From nobody Fri Dec 19 10:39:44 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DA9502E646 for ; Wed, 3 Jul 2024 04:01:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719979318; cv=none; b=ILzx2qdtly49r6l3qBvzIZs+A/5SYGsi5hNuTbYZZ4vt3m3kaeMOuqJ2mjbonAnWImBazaG01J0qyAHp7HxAbCilwamGuHAYTypZOkHJ1I00MkXAklwXfcgnHNUvYfJkPh1Jm2XCKIXFdpgrOMscVklVcN/3Bm8i58NW83/C4C8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719979318; c=relaxed/simple; bh=wTDI0eYHYt7QQuqfYVSh3nHEsVuSO77v1MHzFFaeoFg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=tVrdTCwTzRoPifPLxCASEdXsscI+YU5m9bPfzzakdh6ZX72Al8MmYDul2znpfv87Ham/l2NhygiS7R1IK33eFFQqKPcSAPzNKFhtWKcB7oHXYC8l3D/y3FOgCVvKBp40BXIQqOedOIDslbxGG0SvXnHCVmQ20nC0Kiyqf11Ck/8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=gVOc+ikQ; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="gVOc+ikQ" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 74B10C32781; Wed, 3 Jul 2024 04:01:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1719979318; bh=wTDI0eYHYt7QQuqfYVSh3nHEsVuSO77v1MHzFFaeoFg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=gVOc+ikQ0e4uEf4lRf4VFTkicUepq4dYTbwtWmScFz/shEGC3tPJ2Tv3kp+jVx39J 02cNh0BzDrMvoMSkuZTHsBlPZovMNzYKU9VGCsNc17yGimfEVfQQ3CNb5lzuXAIK+c vLBE7IJcOkdN1BRsM4pIX8l88EuGTlaSwpUqFmKj6l44HiFqFjGNl68SmrkQ6JlwlG YEv7zNYxkcmEBnWJ62sBfJcYrEnUJwxuyTJf2Kb6E3+qL3UZzdAq666Nr3NlRFnIgS P+qbUaMur7AJrvXSuEPhRwv5psWkuwXQGkQc7n1tj5nhvojIeOMofVoEn+JGscY7nv qedKqn8ZvIBgg== From: alexs@kernel.org To: Vitaly Wool , Miaohe Lin , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, minchan@kernel.org, willy@infradead.org, senozhatsky@chromium.org, david@redhat.com, 42.hyeyoo@gmail.com, Yosry Ahmed , nphamcs@gmail.com Cc: Alex Shi Subject: [PATCH v2 13/20] mm/zsmalloc: convert location_to_obj() to take zpdesc Date: Wed, 3 Jul 2024 12:06:03 +0800 Message-ID: <20240703040613.681396-14-alexs@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240703040613.681396-1-alexs@kernel.org> References: <20240703040613.681396-1-alexs@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Hyeonggon Yoo <42.hyeyoo@gmail.com> As all users of location_to_obj() now use zpdesc, convert location_to_obj() to take zpdesc. Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Signed-off-by: Alex Shi --- mm/zsmalloc.c | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 68fdea7b6e0d..e291c7319485 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -826,15 +826,15 @@ static void obj_to_zpdesc(unsigned long obj, struct z= pdesc **zpdesc) } =20 /** - * location_to_obj - get obj value encoded from (, ) - * @page: page object resides in zspage + * location_to_obj - get obj value encoded from (, ) + * @zpdesc: zpdesc object resides in zspage * @obj_idx: object index */ -static unsigned long location_to_obj(struct page *page, unsigned int obj_i= dx) +static unsigned long location_to_obj(struct zpdesc *zpdesc, unsigned int o= bj_idx) { unsigned long obj; =20 - obj =3D page_to_pfn(page) << OBJ_INDEX_BITS; + obj =3D zpdesc_pfn(zpdesc) << OBJ_INDEX_BITS; obj |=3D obj_idx & OBJ_INDEX_MASK; =20 return obj; @@ -1392,7 +1392,7 @@ static unsigned long obj_malloc(struct zs_pool *pool, kunmap_atomic(vaddr); mod_zspage_inuse(zspage, 1); =20 - obj =3D location_to_obj(zpdesc_page(m_zpdesc), obj); + obj =3D location_to_obj(m_zpdesc, obj); record_obj(handle, obj); =20 return obj; @@ -1879,8 +1879,7 @@ static int zs_page_migrate(struct page *newpage, stru= ct page *page, =20 old_obj =3D handle_to_obj(handle); obj_to_location(old_obj, &dummy, &obj_idx); - new_obj =3D (unsigned long)location_to_obj(zpdesc_page(newzpdesc), - obj_idx); + new_obj =3D (unsigned long)location_to_obj(newzpdesc, obj_idx); record_obj(handle, new_obj); } } --=20 2.43.0 From nobody Fri Dec 19 10:39:44 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3DC2E376E1 for ; Wed, 3 Jul 2024 04:02:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719979323; cv=none; b=XLeIIa3Sq46Ywhf4K7N5oRsIJ8yCwjCWFsNe7KhfM8ECaZ3ITBh/J/PILza/QdoE3ma8W8+T0R8mOP+6lNZ0fmdzgti97eARE/oeZMR2tqGHg1UAsCVjEx/UIHles6ET895A+cKCvKgoDcazCUHsQrVvl/gDf/8ENSnThk5+7UE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719979323; c=relaxed/simple; bh=K3GUIiHemMLy8fa8RNH072Iq/pXEu/mTjv8z/3myXkU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Z/UmLZW3pXew+7T+vmIqVl7/+ceRbnqMV/tmlNsArXTwh5ywKPP35Cs1ih4pBMD0+T1hManL0B6Uj1UMM/+Pc8AAElqMRVXWCSwgwIquOCM7hr1TmCJ+TXdyxlD/tKUMrWXJtowfVqMXc7jUbtsg3oS96RPEOQfHQ/616fcnWhU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=UAi8VdfB; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="UAi8VdfB" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 6D940C32781; Wed, 3 Jul 2024 04:01:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1719979322; bh=K3GUIiHemMLy8fa8RNH072Iq/pXEu/mTjv8z/3myXkU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=UAi8VdfBMIZrg0juq8txVEnN9W0EIBUjwpsOIiZcXYiKs8oR980Yy+uHO+fcF89px gU3DFwUpEt4cZuy68LLq7XCBh43WC3tWvg/b4RPgShtXhVSLkiqtHB6Cw08GmHBaVy t6TqfuihXAZJUFK2cGbb3I5U/kDa7LC25OR2aSeCr9jAJietiU90ZjWEIYzzKKbxcN IWqeUfxpYZiX8g7ytLWs+EQNJ2gLMuYGRelGem7bQ9Fgma33GDoN8rvUrqdybqMNJr Ao86j0XjV9Ea2MaRgEYnIk5y5CDjM9z8tG8mHiz0IxMP9HeX7MTYQa268tc5x3zpae OkY1Nxw+l3/6Q== From: alexs@kernel.org To: Vitaly Wool , Miaohe Lin , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, minchan@kernel.org, willy@infradead.org, senozhatsky@chromium.org, david@redhat.com, 42.hyeyoo@gmail.com, Yosry Ahmed , nphamcs@gmail.com Cc: Alex Shi Subject: [PATCH v2 14/20] mm/zsmalloc: convert migrate_zspage() to use zpdesc Date: Wed, 3 Jul 2024 12:06:04 +0800 Message-ID: <20240703040613.681396-15-alexs@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240703040613.681396-1-alexs@kernel.org> References: <20240703040613.681396-1-alexs@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Hyeonggon Yoo <42.hyeyoo@gmail.com> Use get_first_zpdesc/get_next_zpdesc to replace get_first/next_page. No functional change. Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Signed-off-by: Alex Shi --- mm/zsmalloc.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index e291c7319485..7d039b0c66db 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -1636,14 +1636,14 @@ static void migrate_zspage(struct zs_pool *pool, st= ruct zspage *src_zspage, unsigned long used_obj, free_obj; unsigned long handle; int obj_idx =3D 0; - struct page *s_page =3D get_first_page(src_zspage); + struct zpdesc *s_zpdesc =3D get_first_zpdesc(src_zspage); struct size_class *class =3D pool->size_class[src_zspage->class]; =20 while (1) { - handle =3D find_alloced_obj(class, page_zpdesc(s_page), &obj_idx); + handle =3D find_alloced_obj(class, s_zpdesc, &obj_idx); if (!handle) { - s_page =3D get_next_page(s_page); - if (!s_page) + s_zpdesc =3D get_next_zpdesc(s_zpdesc); + if (!s_zpdesc) break; obj_idx =3D 0; continue; --=20 2.43.0 From nobody Fri Dec 19 10:39:44 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F0878383B1 for ; Wed, 3 Jul 2024 04:02:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719979327; cv=none; b=q7k6ZVKzGZaUyz29ka0rGmM3u2TMNArQFhv9Joa10AraKdYWOwdOQwbeoj9vB2idlJN0MbCIEjSCikvmuBprOcvn/fhV3rUQgdaY0iHdSjLwwV44FymqkjZqTbygcAVLimS0fGUpKPD3LtC7wjEuOV0DrbiQAC089RkPHVPEhAU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719979327; c=relaxed/simple; bh=0AvWtekA1fBYWEoFvuJjRv8WAqdxESdkBRdAezOabRE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=lzJrJBiTNU32CMB/AH1gOSWTwE8xYn4FwgZ0pnQgfEN7gnfUNTSybOnPnJD5ejKy7mLO4EtQfgh+ATU874cduK24lLndPIX/RMWGBv14ccz+y+4CTIU3eK1DHsGHdeFpLiKs/JolO+U1Xahjt97aWNt1OC6IHNE4CMxHYdeKbSo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=XVwqKhxq; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="XVwqKhxq" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 369CFC4AF0E; Wed, 3 Jul 2024 04:02:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1719979326; bh=0AvWtekA1fBYWEoFvuJjRv8WAqdxESdkBRdAezOabRE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=XVwqKhxqJJIYI4ImJo3+psdvnBbfEXhEGJDSfDKi962ESzTtF5/hbZ2P7VODY/3r5 4I4jG8LqmmhAtzostIdPRwMaxNugcJ7W3Lw+obRV2bqbBj41gU9MiaOVMBBEmkrvAP uUfWcaK1g6MOHtA4amW+Zy5h3GHC9ilT1fvyNp4vMZF045n4UCOPHLdeRlWk1ATdJf ZeKKavWcPN4rFm71clrnhy3cPlRVjb7FwjSU4mIeC/PyYBtQN2Mj0FXTUNviqAhJYi 2GI9UT/7KubfIvR8WPuJlMYEUEnCU4gkEmis4XUM0cBgldgN6W+/DIW3ub3foDmaZT kRSS3fBZLRMnw== From: alexs@kernel.org To: Vitaly Wool , Miaohe Lin , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, minchan@kernel.org, willy@infradead.org, senozhatsky@chromium.org, david@redhat.com, 42.hyeyoo@gmail.com, Yosry Ahmed , nphamcs@gmail.com Cc: Alex Shi Subject: [PATCH v2 15/20] mm/zsmalloc: convert get_zspage() to take zpdesc Date: Wed, 3 Jul 2024 12:06:05 +0800 Message-ID: <20240703040613.681396-16-alexs@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240703040613.681396-1-alexs@kernel.org> References: <20240703040613.681396-1-alexs@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Hyeonggon Yoo <42.hyeyoo@gmail.com> Now that all users except get_next_page() (which will be removed in later patch) use zpdesc, convert get_zspage() to take zpdesc instead of page. Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Signed-off-by: Alex Shi --- mm/zsmalloc.c | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 7d039b0c66db..458ad696b473 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -779,9 +779,9 @@ static int fix_fullness_group(struct size_class *class,= struct zspage *zspage) return newfg; } =20 -static struct zspage *get_zspage(struct page *page) +static struct zspage *get_zspage(struct zpdesc *zpdesc) { - struct zspage *zspage =3D (struct zspage *)page_private(page); + struct zspage *zspage =3D zpdesc->zspage; =20 BUG_ON(zspage->magic !=3D ZSPAGE_MAGIC); return zspage; @@ -789,7 +789,7 @@ static struct zspage *get_zspage(struct page *page) =20 static struct page *get_next_page(struct page *page) { - struct zspage *zspage =3D get_zspage(page); + struct zspage *zspage =3D get_zspage(page_zpdesc(page)); =20 if (unlikely(ZsHugePage(zspage))) return NULL; @@ -799,7 +799,7 @@ static struct page *get_next_page(struct page *page) =20 static struct zpdesc *get_next_zpdesc(struct zpdesc *zpdesc) { - struct zspage *zspage =3D get_zspage(zpdesc_page(zpdesc)); + struct zspage *zspage =3D get_zspage(zpdesc); =20 if (unlikely(ZsHugePage(zspage))) return NULL; @@ -849,7 +849,7 @@ static inline bool obj_allocated(struct zpdesc *zpdesc,= void *obj, unsigned long *phandle) { unsigned long handle; - struct zspage *zspage =3D get_zspage(zpdesc_page(zpdesc)); + struct zspage *zspage =3D get_zspage(zpdesc); =20 if (unlikely(ZsHugePage(zspage))) { VM_BUG_ON_PAGE(!is_first_zpdesc(zpdesc), zpdesc_page(zpdesc)); @@ -1265,7 +1265,7 @@ void *zs_map_object(struct zs_pool *pool, unsigned lo= ng handle, read_lock(&pool->migrate_lock); obj =3D handle_to_obj(handle); obj_to_location(obj, &zpdesc, &obj_idx); - zspage =3D get_zspage(zpdesc_page(zpdesc)); + zspage =3D get_zspage(zpdesc); =20 /* * migration cannot move any zpages in this zspage. Here, class->lock @@ -1315,7 +1315,7 @@ void zs_unmap_object(struct zs_pool *pool, unsigned l= ong handle) =20 obj =3D handle_to_obj(handle); obj_to_location(obj, &zpdesc, &obj_idx); - zspage =3D get_zspage(zpdesc_page(zpdesc)); + zspage =3D get_zspage(zpdesc); class =3D zspage_class(pool, zspage); off =3D offset_in_page(class->size * obj_idx); =20 @@ -1479,7 +1479,7 @@ static void obj_free(int class_size, unsigned long ob= j) =20 obj_to_location(obj, &f_zpdesc, &f_objidx); f_offset =3D offset_in_page(class_size * f_objidx); - zspage =3D get_zspage(zpdesc_page(f_zpdesc)); + zspage =3D get_zspage(f_zpdesc); =20 vaddr =3D zpdesc_kmap_atomic(f_zpdesc); link =3D (struct link_free *)(vaddr + f_offset); @@ -1513,7 +1513,7 @@ void zs_free(struct zs_pool *pool, unsigned long hand= le) read_lock(&pool->migrate_lock); obj =3D handle_to_obj(handle); obj_to_zpdesc(obj, &f_zpdesc); - zspage =3D get_zspage(zpdesc_page(f_zpdesc)); + zspage =3D get_zspage(f_zpdesc); class =3D zspage_class(pool, zspage); spin_lock(&class->lock); read_unlock(&pool->migrate_lock); @@ -1846,7 +1846,7 @@ static int zs_page_migrate(struct page *newpage, stru= ct page *page, __SetPageZsmalloc(zpdesc_page(newzpdesc)); =20 /* The page is locked, so this pointer must remain valid */ - zspage =3D get_zspage(zpdesc_page(zpdesc)); + zspage =3D get_zspage(zpdesc); pool =3D zspage->pool; =20 /* --=20 2.43.0 From nobody Fri Dec 19 10:39:44 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 92C38383B1 for ; Wed, 3 Jul 2024 04:02:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719979330; cv=none; b=itzv8LVVLYIcE6M/Avv4TtKNWxrH9PHIroTq1qN30xMARW6U1iuGY+/d1wRd2Wb3hfGd4Mhn7362T6P2O/BG+NMPUEF+RqE+L+1rcHkljYHLzsDJGBS2m/JijSdPjJTJxPDkdj+1wCKyPc/saBOjIfMhW7Mv6l7UonA3mTo5Myg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719979330; c=relaxed/simple; bh=uH0rZNnpHFMljEmSyKQuSSlLph2dNP0xTWvseJvNDaY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=HnS+4vlF/RwNK7DhdeNVgCB8KPKwpC+Z1lDIXstI8Rg5ggxDnpLaraxyrLRsqp/E1I6TsOa+n3m7WeTqAcgdSvI0Pe0+fZ3rMBE+ZJ+CyyRtlK0SoLfWJGaS3Yevx14lH5hhfqNs7Nnh/RFEkwfON6p0KriH8gK2tUkXOP8OF84= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=iPsY0Brd; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="iPsY0Brd" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 2E17BC4AF0A; Wed, 3 Jul 2024 04:02:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1719979330; bh=uH0rZNnpHFMljEmSyKQuSSlLph2dNP0xTWvseJvNDaY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=iPsY0Brd5KWtyIOk5fV/0otAMuF46d4KG5B5pddwxSDNlJ1E0cbaAnUHd208Zws0d 9GHVhVF88u6OFKosO7B9jELp3CHwrLGdn4h4C4xb34PtLyYwVbQV+IsjoAXADnV9j9 ZffY97ScQAwMreWa3NmKgcpzng+WKLp1IgAjr1hdxnh4yioFj0tdWX3klI0o5/wT3s XSD0p939k4/5df5GCsmd/HriGXYfyFLk0zojxelaHRW2zZ1egLtRTBjYri2bNAjJLG 3ywNEDpuwtzOPd+oo+1gxRI63Ppyldz7y+1CeuJAmASlFlF6pBtMSPY1r+x5RnQZg2 zkf3T4L1vs6PA== From: alexs@kernel.org To: Vitaly Wool , Miaohe Lin , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, minchan@kernel.org, willy@infradead.org, senozhatsky@chromium.org, david@redhat.com, 42.hyeyoo@gmail.com, Yosry Ahmed , nphamcs@gmail.com Cc: Alex Shi Subject: [PATCH v2 16/20] mm/zsmalloc: convert SetZsPageMovable and remove unused funcs Date: Wed, 3 Jul 2024 12:06:06 +0800 Message-ID: <20240703040613.681396-17-alexs@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240703040613.681396-1-alexs@kernel.org> References: <20240703040613.681396-1-alexs@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Alex Shi Convert SetZsPageMovable() to use zpdesc, and then remove unused funcs:get_next_page/get_first_page/is_first_page. Originally-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Signed-off-by: Alex Shi --- mm/zsmalloc.c | 33 +++++---------------------------- 1 file changed, 5 insertions(+), 28 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 458ad696b473..8b713ac03902 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -468,11 +468,6 @@ static DEFINE_PER_CPU(struct mapping_area, zs_map_area= ) =3D { .lock =3D INIT_LOCAL_LOCK(lock), }; =20 -static __maybe_unused int is_first_page(struct page *page) -{ - return PagePrivate(page); -} - static int is_first_zpdesc(struct zpdesc *zpdesc) { return PagePrivate(zpdesc_page(zpdesc)); @@ -489,14 +484,6 @@ static inline void mod_zspage_inuse(struct zspage *zsp= age, int val) zspage->inuse +=3D val; } =20 -static inline struct page *get_first_page(struct zspage *zspage) -{ - struct page *first_page =3D zpdesc_page(zspage->first_zpdesc); - - VM_BUG_ON_PAGE(!is_first_page(first_page), first_page); - return first_page; -} - static struct zpdesc *get_first_zpdesc(struct zspage *zspage) { struct zpdesc *first_zpdesc =3D zspage->first_zpdesc; @@ -787,16 +774,6 @@ static struct zspage *get_zspage(struct zpdesc *zpdesc) return zspage; } =20 -static struct page *get_next_page(struct page *page) -{ - struct zspage *zspage =3D get_zspage(page_zpdesc(page)); - - if (unlikely(ZsHugePage(zspage))) - return NULL; - - return (struct page *)page->index; -} - static struct zpdesc *get_next_zpdesc(struct zpdesc *zpdesc) { struct zspage *zspage =3D get_zspage(zpdesc); @@ -1970,13 +1947,13 @@ static void init_deferred_free(struct zs_pool *pool) =20 static void SetZsPageMovable(struct zs_pool *pool, struct zspage *zspage) { - struct page *page =3D get_first_page(zspage); + struct zpdesc *zpdesc =3D get_first_zpdesc(zspage); =20 do { - WARN_ON(!trylock_page(page)); - __SetPageMovable(page, &zsmalloc_mops); - unlock_page(page); - } while ((page =3D get_next_page(page)) !=3D NULL); + WARN_ON(!zpdesc_trylock(zpdesc)); + __zpdesc_set_movable(zpdesc, &zsmalloc_mops); + zpdesc_unlock(zpdesc); + } while ((zpdesc =3D get_next_zpdesc(zpdesc)) !=3D NULL); } #else static inline void zs_flush_migration(struct zs_pool *pool) { } --=20 2.43.0 From nobody Fri Dec 19 10:39:44 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 943B7174EDE for ; Wed, 3 Jul 2024 04:02:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719979334; cv=none; b=EZXzF4phwGXbhgQRjRUPmtRP4Bix299INMDOeirrfUr++FDuOl04NcwovXiJ8AUfWC0qQ7EIiX3pB9/y1miFCN8UpF5XxjoG1EGWv8rvhLzzjtnpbEw/C90iX08BOJc716mcMYC2+JmCD/XvIqHBN/jxYwB0/DI6Ga+MdFXDCb4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719979334; c=relaxed/simple; bh=vsXuVfMJsSuMD7UFdk0WVE3RDVFvIl+YscGm1B7bHVw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=PxxhtNqYKjqFUWgnZpO4ucqxNAuLBziZh36QobJyx3nlBlajixVffRLe9WTILGR0zZW4yBEDEwOgYf5oBFil4beuZS6dBsVplRdYSU1I7q5j5bob3Dlale3L88behm2jl4cp1WusQ1rbsLml6n/HC4xsi+wKLaOzD0o+ZzbNyVw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=TUNlomcQ; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="TUNlomcQ" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 2527FC4AF0C; Wed, 3 Jul 2024 04:02:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1719979334; bh=vsXuVfMJsSuMD7UFdk0WVE3RDVFvIl+YscGm1B7bHVw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=TUNlomcQNEckWzE3pkdaNY/jWZemVDlbi9bxsdknLLDnG7+ZhTC6n5f1YKOBHPscF BhxNJlC+MBOjvNvdjcvvweVfGrQI08RQxthpm10H45a59hqsrL4GTJMugS9+1KtZdn 4M+riFrBTzpbHYxAIryEuoUqrDZckEejZCNBzptjOWLMeGv1EBzgq2ScxBviaaWrQ6 QWZDoYbgUQhGonitWAa/cUrUYW0++s9TjxH7obnaIBdamvY460PdXvBVLA6X1x6nkS B4DRFGqaiG9bq5kwyikisFoKGiRznLujqLzzynay9P0uddufucPKK3vVj2EgfV4as3 vtHSH2B65Q+mg== From: alexs@kernel.org To: Vitaly Wool , Miaohe Lin , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, minchan@kernel.org, willy@infradead.org, senozhatsky@chromium.org, david@redhat.com, 42.hyeyoo@gmail.com, Yosry Ahmed , nphamcs@gmail.com Cc: Alex Shi Subject: [PATCH v2 17/20] mm/zsmalloc: convert get/set_first_obj_offset() to take zpdesc Date: Wed, 3 Jul 2024 12:06:07 +0800 Message-ID: <20240703040613.681396-18-alexs@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240703040613.681396-1-alexs@kernel.org> References: <20240703040613.681396-1-alexs@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Hyeonggon Yoo <42.hyeyoo@gmail.com> Now that all users of get/set_first_obj_offset() are converted to use zpdesc, convert them to take zpdesc. Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Signed-off-by: Alex Shi --- mm/zpdesc.h | 6 +++++- mm/zsmalloc.c | 32 ++++++++++++++++---------------- 2 files changed, 21 insertions(+), 17 deletions(-) diff --git a/mm/zpdesc.h b/mm/zpdesc.h index 6de3c5d095bd..296cb1ff944b 100644 --- a/mm/zpdesc.h +++ b/mm/zpdesc.h @@ -15,6 +15,8 @@ * @next: Next zpdesc in a zspage in zsmalloc zpool * @handle: For huge zspage in zsmalloc zpool * @zspage: Pointer to zspage in zsmalloc + * @first_obj_offset: First object offset in zsmalloc zpool + * @_refcount: Indirectly use by page migration * @memcg_data: Memory Control Group data. * * This struct overlays struct page for now. Do not modify without a good @@ -31,7 +33,8 @@ struct zpdesc { unsigned long handle; }; struct zspage *zspage; - unsigned long _zp_pad_1; + unsigned int first_obj_offset; + atomic_t _refcount; #ifdef CONFIG_SLAB_OBJ_EXT unsigned long memcg_data; #endif @@ -45,6 +48,7 @@ ZPDESC_MATCH(mapping, mops); ZPDESC_MATCH(index, next); ZPDESC_MATCH(index, handle); ZPDESC_MATCH(private, zspage); +ZPDESC_MATCH(page_type, first_obj_offset); ZPDESC_MATCH(memcg_data, memcg_data); #undef ZPDESC_MATCH static_assert(sizeof(struct zpdesc) <=3D sizeof(struct page)); diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 8b713ac03902..0493f953fcb5 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -494,26 +494,26 @@ static struct zpdesc *get_first_zpdesc(struct zspage = *zspage) =20 #define FIRST_OBJ_PAGE_TYPE_MASK 0xffff =20 -static inline void reset_first_obj_offset(struct page *page) +static inline void reset_first_obj_offset(struct zpdesc *zpdesc) { - VM_WARN_ON_ONCE(!PageZsmalloc(page)); - page->page_type |=3D FIRST_OBJ_PAGE_TYPE_MASK; + VM_WARN_ON_ONCE(!PageZsmalloc(zpdesc_page(zpdesc))); + zpdesc->first_obj_offset |=3D FIRST_OBJ_PAGE_TYPE_MASK; } =20 -static inline unsigned int get_first_obj_offset(struct page *page) +static inline unsigned int get_first_obj_offset(struct zpdesc *zpdesc) { - VM_WARN_ON_ONCE(!PageZsmalloc(page)); - return page->page_type & FIRST_OBJ_PAGE_TYPE_MASK; + VM_WARN_ON_ONCE(!PageZsmalloc(zpdesc_page(zpdesc))); + return zpdesc->first_obj_offset & FIRST_OBJ_PAGE_TYPE_MASK; } =20 -static inline void set_first_obj_offset(struct page *page, unsigned int of= fset) +static inline void set_first_obj_offset(struct zpdesc *zpdesc, unsigned in= t offset) { /* With 16 bit available, we can support offsets into 64 KiB pages. */ BUILD_BUG_ON(PAGE_SIZE > SZ_64K); - VM_WARN_ON_ONCE(!PageZsmalloc(page)); + VM_WARN_ON_ONCE(!PageZsmalloc(zpdesc_page(zpdesc))); VM_WARN_ON_ONCE(offset & ~FIRST_OBJ_PAGE_TYPE_MASK); - page->page_type &=3D ~FIRST_OBJ_PAGE_TYPE_MASK; - page->page_type |=3D offset & FIRST_OBJ_PAGE_TYPE_MASK; + zpdesc->first_obj_offset &=3D ~FIRST_OBJ_PAGE_TYPE_MASK; + zpdesc->first_obj_offset |=3D offset & FIRST_OBJ_PAGE_TYPE_MASK; } =20 static inline unsigned int get_freeobj(struct zspage *zspage) @@ -850,7 +850,7 @@ static void reset_zpdesc(struct zpdesc *zpdesc) ClearPagePrivate(page); zpdesc->zspage =3D NULL; zpdesc->next =3D NULL; - reset_first_obj_offset(page); + reset_first_obj_offset(zpdesc); __ClearPageZsmalloc(page); } =20 @@ -934,7 +934,7 @@ static void init_zspage(struct size_class *class, struc= t zspage *zspage) struct link_free *link; void *vaddr; =20 - set_first_obj_offset(zpdesc_page(zpdesc), off); + set_first_obj_offset(zpdesc, off); =20 vaddr =3D zpdesc_kmap_atomic(zpdesc); link =3D (struct link_free *)vaddr + off / sizeof(*link); @@ -1589,7 +1589,7 @@ static unsigned long find_alloced_obj(struct size_cla= ss *class, unsigned long handle =3D 0; void *addr =3D zpdesc_kmap_atomic(zpdesc); =20 - offset =3D get_first_obj_offset(zpdesc_page(zpdesc)); + offset =3D get_first_obj_offset(zpdesc); offset +=3D class->size * index; =20 while (offset < PAGE_SIZE) { @@ -1784,8 +1784,8 @@ static void replace_sub_page(struct size_class *class= , struct zspage *zspage, } while ((zpdesc =3D get_next_zpdesc(zpdesc)) !=3D NULL); =20 create_page_chain(class, zspage, zpdescs); - first_obj_offset =3D get_first_obj_offset(zpdesc_page(oldzpdesc)); - set_first_obj_offset(zpdesc_page(newzpdesc), first_obj_offset); + first_obj_offset =3D get_first_obj_offset(oldzpdesc); + set_first_obj_offset(newzpdesc, first_obj_offset); if (unlikely(ZsHugePage(zspage))) newzpdesc->handle =3D oldzpdesc->handle; __zpdesc_set_movable(newzpdesc, &zsmalloc_mops); @@ -1840,7 +1840,7 @@ static int zs_page_migrate(struct page *newpage, stru= ct page *page, /* the migrate_write_lock protects zpage access via zs_map_object */ migrate_write_lock(zspage); =20 - offset =3D get_first_obj_offset(zpdesc_page(zpdesc)); + offset =3D get_first_obj_offset(zpdesc); s_addr =3D zpdesc_kmap_atomic(zpdesc); =20 /* --=20 2.43.0 From nobody Fri Dec 19 10:39:44 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BAD7E4776A for ; Wed, 3 Jul 2024 04:02:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719979338; cv=none; b=GBHkR80Ov0TtJ2z1Z1Dgr/+DpENngp17UGuHctgW87FsZ1ou3fC3RCP4peqa5O2rd8xt2WLqy1RY8jd7u4idGqVbgEuU4nqbwvnm7pz8m8lh0DwA0J1nXNyudMZWtKZ6cZXenC6xqGnZzedP6+l+R0mmED9YjCIbAKHMcVaaA40= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719979338; c=relaxed/simple; bh=WADQG+KFeKRI6MYgRuTdfJ+b1YmyjJ4toh2DCgrdz3U=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=g2TqabVOpHsIzGX8n0ePmSvc2zs0ms20p8dHPHdceGRHlhTNKNnl4uNO03Cv9SHaaEqJ8K1hO1+v3rpx0IgYARwTc1QjNmlGrchLpQmp276KFDzGJsCP+oOsSXWjLX+fciXFE0ZpMSQN32WsaRchv654LtaUlHFqViO/+TIQ+10= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=SClcJWWi; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="SClcJWWi" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1BD1DC4AF0B; Wed, 3 Jul 2024 04:02:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1719979338; bh=WADQG+KFeKRI6MYgRuTdfJ+b1YmyjJ4toh2DCgrdz3U=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=SClcJWWiEa+5YwdOV0tQyYXItfPMvdIJTAnwNUIfop+4H0HQMX6ZlDZTA9hyInDUt n474nw2Xna1yNdCGIxwVeOL+AsXksKOajIh2vblG663lUPirNCunRmA1yrrPDSvY4t tAKnQr8OFrdsHf20r/qE1JXENe2kMo+E51gUSfzDg6a/UXszSkdWHn34wdnr1yKv1R kF12ECffDFlANRMIMJ2uT+QygpKzV52sLDqzBCGLDPV5aJ6Wt98lGqf5iak5vpEClN dcelnlI5gfXuQnyhUIPfGTQxg211qR96KWLBHvR4OwbGn8W4VGlyU6VOIxCFdv9j9B w2mMYKGx6JW7g== From: alexs@kernel.org To: Vitaly Wool , Miaohe Lin , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, minchan@kernel.org, willy@infradead.org, senozhatsky@chromium.org, david@redhat.com, 42.hyeyoo@gmail.com, Yosry Ahmed , nphamcs@gmail.com Cc: Alex Shi Subject: [PATCH v2 18/20] mm/zsmalloc: introduce __zpdesc_clear_movable Date: Wed, 3 Jul 2024 12:06:08 +0800 Message-ID: <20240703040613.681396-19-alexs@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240703040613.681396-1-alexs@kernel.org> References: <20240703040613.681396-1-alexs@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Alex Shi Add a helper __zpdesc_clear_movable() for __ClearPageMovable(), and use it in callers to make code clear. Signed-off-by: Alex Shi --- mm/zpdesc.h | 5 +++++ mm/zsmalloc.c | 2 +- 2 files changed, 6 insertions(+), 1 deletion(-) diff --git a/mm/zpdesc.h b/mm/zpdesc.h index 296cb1ff944b..4780d7015267 100644 --- a/mm/zpdesc.h +++ b/mm/zpdesc.h @@ -111,6 +111,11 @@ static inline void __zpdesc_set_movable(struct zpdesc = *zpdesc, __SetPageMovable(zpdesc_page(zpdesc), mops); } =20 +static inline void __zpdesc_clear_movable(struct zpdesc *zpdesc) +{ + __ClearPageMovable(zpdesc_page(zpdesc)); +} + static inline bool zpdesc_is_isolated(struct zpdesc *zpdesc) { return PageIsolated(zpdesc_page(zpdesc)); diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 0493f953fcb5..08ab22f0ab7f 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -846,7 +846,7 @@ static void reset_zpdesc(struct zpdesc *zpdesc) { struct page *page =3D zpdesc_page(zpdesc); =20 - __ClearPageMovable(page); + __zpdesc_clear_movable(zpdesc); ClearPagePrivate(page); zpdesc->zspage =3D NULL; zpdesc->next =3D NULL; --=20 2.43.0 From nobody Fri Dec 19 10:39:44 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5F66F178CC4 for ; Wed, 3 Jul 2024 04:02:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719979342; cv=none; b=btb4tj0gq3BlFGLGQApJ+4OJP4jKEHTMyg9hEnHirBlYN8A4iDZlB4rn8DwPuH7Y3DkQVJA8WmQukKAvGejghIoTL+BSLmCeTr0ODiuGs3lvrn8Mjl4yh2Eunwk1aBs1B7qne5bFkSYSyoeYOAcSHaaVV/vZRluRC+i4mU32luk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719979342; c=relaxed/simple; bh=hU+tJyMJYXqvt89j51XXi7aJ8UUcH7P41TCm8J5zHmE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=hKP1p6pthuqsXC68DC6Iz4mTJaA7xcoZPBpRNrWIbWjoJ2QCUHIwk6X3/UN8CH4bt+Kip/I09xe85pCZMSvFq6k45p0879hTlX0cSLZLcaNpb2EHNc2Qam4YwnACoo7WtMj9PL69T3yxlnwKAheR3/dTAIcBOXCHDVIvmmVpDjM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Rxj4KWb+; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Rxj4KWb+" Received: by smtp.kernel.org (Postfix) with ESMTPSA id DCA06C4AF0C; Wed, 3 Jul 2024 04:02:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1719979342; bh=hU+tJyMJYXqvt89j51XXi7aJ8UUcH7P41TCm8J5zHmE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Rxj4KWb+WvTzcDWLevWOhyIZvPre2S3FEfiFSoQHVdaD48MPqIaESXA0lOeqDWz4J +PV3qdO7ClN5HuEXB8ZUTc93SoSHRCXfeyZI6fUCocUSJvT7GQ3gYuhhwAhBmiA2gQ 7Yd7yqxS9s9YMDPR2GjZ+8cb7cxFRTqV08nofEfWWqVvquRaIH5UowQJOAm9rb6Gk6 EqDDetEeLI9t2PnL67Cb00PJ3IQcUEQU3tIyO/IsiBlD8O1p6pNaYiknY4a9ISb+wO E+VQvCq3i2xi4qwqy6BJzM1lJTJPnVc4iWDAyklxFyGuvDzcv8yzDdWvCLl69l39Ja +6L9lNmpQ87Pg== From: alexs@kernel.org To: Vitaly Wool , Miaohe Lin , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, minchan@kernel.org, willy@infradead.org, senozhatsky@chromium.org, david@redhat.com, 42.hyeyoo@gmail.com, Yosry Ahmed , nphamcs@gmail.com Cc: Alex Shi Subject: [PATCH v2 19/20] mm/zsmalloc: introduce __zpdesc_clear_zsmalloc Date: Wed, 3 Jul 2024 12:06:09 +0800 Message-ID: <20240703040613.681396-20-alexs@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240703040613.681396-1-alexs@kernel.org> References: <20240703040613.681396-1-alexs@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Alex Shi Add a helper __zpdesc_clear_zsmalloc() for __ClearPageZsmalloc(), and use it in callers to make code clear. Signed-off-by: Alex Shi --- mm/zpdesc.h | 5 +++++ mm/zsmalloc.c | 4 ++-- 2 files changed, 7 insertions(+), 2 deletions(-) diff --git a/mm/zpdesc.h b/mm/zpdesc.h index 4780d7015267..e6a71c7a3254 100644 --- a/mm/zpdesc.h +++ b/mm/zpdesc.h @@ -116,6 +116,11 @@ static inline void __zpdesc_clear_movable(struct zpdes= c *zpdesc) __ClearPageMovable(zpdesc_page(zpdesc)); } =20 +static inline void __zpdesc_clear_zsmalloc(struct zpdesc *zpdesc) +{ + __ClearPageZsmalloc(zpdesc_page(zpdesc)); +} + static inline bool zpdesc_is_isolated(struct zpdesc *zpdesc) { return PageIsolated(zpdesc_page(zpdesc)); diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 08ab22f0ab7f..9e87a17ed802 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -851,7 +851,7 @@ static void reset_zpdesc(struct zpdesc *zpdesc) zpdesc->zspage =3D NULL; zpdesc->next =3D NULL; reset_first_obj_offset(zpdesc); - __ClearPageZsmalloc(page); + __zpdesc_clear_zsmalloc(zpdesc); } =20 static int trylock_zspage(struct zspage *zspage) @@ -1024,7 +1024,7 @@ static struct zspage *alloc_zspage(struct zs_pool *po= ol, if (!zpdesc) { while (--i >=3D 0) { zpdesc_dec_zone_page_state(zpdescs[i]); - __ClearPageZsmalloc(zpdesc_page(zpdescs[i])); + __zpdesc_clear_zsmalloc(zpdescs[i]); free_zpdesc(zpdescs[i]); } cache_free_zspage(pool, zspage); --=20 2.43.0 From nobody Fri Dec 19 10:39:44 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 50725178CC4 for ; Wed, 3 Jul 2024 04:02:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719979346; cv=none; b=BMMkUXB8OO5a/mKwLAHohd0qYoLq/A0OCz6LgDA6tljGX0xiklcL2p4K4bwAv6tfSfEt9xr7Ejbt1ctDjdXu34Ag1Zdh9pfFaVUsyMhybvVtnwUK6I+LonUTmJHl2S8vYdBN9sqHAz7Ye9ArCXRxXZt4K/HfWlCyQQIJGE/JcGI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719979346; c=relaxed/simple; bh=xf3aTHyp0ikeJrBlj+XtoBw8uMm+YJ3YtskUpiov5H4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=C/huOJtxlIeJzmJIhbfPK1wTgs/3WmrFNeU2FdVUMd7yeYLvQmjUbcxEVf3020a9qQr7wGCtV2OyHJ6AFCVNwVURX/4pAgsQPxwBoEOmvT1RGfSVqxrhwoZcHX+RYFMIcxeK+RHoQ19Lm0apy7RKow3IUbjJAeWVhb6DIRF0gZw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=S95km11Y; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="S95km11Y" Received: by smtp.kernel.org (Postfix) with ESMTPSA id A954FC32781; Wed, 3 Jul 2024 04:02:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1719979346; bh=xf3aTHyp0ikeJrBlj+XtoBw8uMm+YJ3YtskUpiov5H4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=S95km11YpkihSJvZhp/uEBasmUDA3sdyzUblqTJ0PHexxmxxWnXUj5mtykzJHpNrL XD0RWT9AlXI5pqwVWqcqGazjfUItM50KJOKLYBKitIgobArYWP0maVhIJWPj3Yq0fS sybAhV09B9V1Q7sDjIyjXElsWNUCNqumEPPQgXAFRXYD4otlCHYNLmBIiEUCaOqA/k 2eWyrQRHBNjkRAGZQG7t7PJnxkokERgGGw3mKOEfwa19w9CEgKNLYpKeNNXg9xqttq 1XwiKETYPDACzuEAZsYk7Nw+G5Bg9RICoxyHVdnSsINHRH3Qe+11uFGhN0hZY5+La2 Xii7jQTm0oudA== From: alexs@kernel.org To: Vitaly Wool , Miaohe Lin , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, minchan@kernel.org, willy@infradead.org, senozhatsky@chromium.org, david@redhat.com, 42.hyeyoo@gmail.com, Yosry Ahmed , nphamcs@gmail.com Cc: Alex Shi Subject: [PATCH v2 20/20] mm/zsmalloc: introduce __zpdesc_set_zsmalloc() Date: Wed, 3 Jul 2024 12:06:10 +0800 Message-ID: <20240703040613.681396-21-alexs@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240703040613.681396-1-alexs@kernel.org> References: <20240703040613.681396-1-alexs@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Alex Shi Add a helper __zpdesc_set_zsmalloc() for __SetPageZsmalloc(), and use it in callers to make code clear. Signed-off-by: Alex Shi --- mm/zpdesc.h | 5 +++++ mm/zsmalloc.c | 4 ++-- 2 files changed, 7 insertions(+), 2 deletions(-) diff --git a/mm/zpdesc.h b/mm/zpdesc.h index e6a71c7a3254..d0e5929c1666 100644 --- a/mm/zpdesc.h +++ b/mm/zpdesc.h @@ -116,6 +116,11 @@ static inline void __zpdesc_clear_movable(struct zpdes= c *zpdesc) __ClearPageMovable(zpdesc_page(zpdesc)); } =20 +static inline void __zpdesc_set_zsmalloc(struct zpdesc *zpdesc) +{ + __SetPageZsmalloc(zpdesc_page(zpdesc)); +} + static inline void __zpdesc_clear_zsmalloc(struct zpdesc *zpdesc) { __ClearPageZsmalloc(zpdesc_page(zpdesc)); diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 9e87a17ed802..7bb85447de94 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -1030,7 +1030,7 @@ static struct zspage *alloc_zspage(struct zs_pool *po= ol, cache_free_zspage(pool, zspage); return NULL; } - __SetPageZsmalloc(zpdesc_page(zpdesc)); + __zpdesc_set_zsmalloc(zpdesc); =20 zpdesc_inc_zone_page_state(zpdesc); zpdescs[i] =3D zpdesc; @@ -1820,7 +1820,7 @@ static int zs_page_migrate(struct page *newpage, stru= ct page *page, VM_BUG_ON_PAGE(!zpdesc_is_isolated(zpdesc), zpdesc_page(zpdesc)); =20 /* We're committed, tell the world that this is a Zsmalloc page. */ - __SetPageZsmalloc(zpdesc_page(newzpdesc)); + __zpdesc_set_zsmalloc(newzpdesc); =20 /* The page is locked, so this pointer must remain valid */ zspage =3D get_zspage(zpdesc); --=20 2.43.0