From nobody Sun Feb 8 02:21:45 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A853F18643 for ; Mon, 8 Jul 2024 06:28:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720420125; cv=none; b=OlF4Ex144xnSu1PVz580SqgHUGSZhAzGrPSVpIuw4Q0n/I0RV/i8l7UwXq8PKcOPI4xPzDyWScRXTgBuRja8X8oBdFdwmdi8QM/A9Lo152vzYl+otcP91tz7seR87V1qbPfUR/N0EYz7Xas6RHXxy1Lj6qxoODVOXt8scM9oG/Y= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720420125; c=relaxed/simple; bh=QwfKZsyWKxmeoIlS8kXArrgVhPn5VQ2gqaeTqGCWsTw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=HM0WqXr0Rb0cto2sAawWqiPXGxZ8uPJboKtmtO2OHGpLuFMmC/0wsz8UMIbHzBGQEdWUTudK4Svczz4KHfF1ZdoyI0QskYwbna/eWkiq99Li6cf3StPiraYtQHT4zWsZSkcJZ67IvoF/hOY8B/H14tzhaz4GQLHuaqRfNqenrHo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=iCMJo9/v; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="iCMJo9/v" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 2B319C116B1; Mon, 8 Jul 2024 06:28:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1720420125; bh=QwfKZsyWKxmeoIlS8kXArrgVhPn5VQ2gqaeTqGCWsTw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=iCMJo9/vSPJSaeCJLkrFmQWg/h52GHkcjcL/5WtAlAGBH2Z4f7gCFaUwqdR2huO8c Hwmd96q7j0coqLb4+LdNtXrNHxHdguBuIYmjcIhrtNwx13cWhtxmySF5Qrz/+kUbxx 7TzYzdagLRPIChRZyMH0KvB7i5rlieVZ2Ru7AG1kQHyKqV9i6t4CIGb6XfjX8oFJ5Y J/TDVncXA3cM0o6WbE680pKILmn1EN73ArAVKswe8XIV6yt6Px/OwQejHwmv8tNvwH FJnQyCW7Vx36BZdU0k8z3rko9+zlTh6ZqXsDTpwqXafp80NJ8VnP+r0C/UxT+b0ZGD wYjHHXaqvBdIg== From: alexs@kernel.org To: Vitaly Wool , Miaohe Lin , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, minchan@kernel.org, willy@infradead.org, senozhatsky@chromium.org, david@redhat.com, 42.hyeyoo@gmail.com, Yosry Ahmed , nphamcs@gmail.com Cc: Alex Shi Subject: [PATCH v3 01/20] mm/zsmalloc: add zpdesc memory descriptor for zswap.zpool Date: Mon, 8 Jul 2024 14:33:22 +0800 Message-ID: <20240708063344.1096626-2-alexs@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240708063344.1096626-1-alexs@kernel.org> References: <20240708063344.1096626-1-alexs@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Alex Shi (Tencent) The 1st patch introduces new memory decriptor zpdesc and rename zspage.first_page to zspage.first_zpdesc, no functional change. We removed PG_owner_priv_1 since it was moved to zspage after commit a41ec880aa7b ("zsmalloc: move huge compressed obj from page to zspage"). And keep the memcg_data member, since as Yosry pointed out: "When the pages are freed, put_page() -> folio_put() -> __folio_put() will = call mem_cgroup_uncharge(). The latter will call folio_memcg() (which reads folio->memcg_data) to figure out if uncharging needs to be done. There are also other similar code paths that will check folio->memcg_data. It is currently expected to be present for all folios. So until we have custom code paths per-folio type for allocation/freeing/etc, we need to keep folio->memcg_data present and properly initialized." Originally-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Signed-off-by: Alex Shi (Tencent) --- mm/zpdesc.h | 66 +++++++++++++++++++++++++++++++++++++++++++++++++++ mm/zsmalloc.c | 21 ++++++++-------- 2 files changed, 76 insertions(+), 11 deletions(-) create mode 100644 mm/zpdesc.h diff --git a/mm/zpdesc.h b/mm/zpdesc.h new file mode 100644 index 000000000000..2dbef231f616 --- /dev/null +++ b/mm/zpdesc.h @@ -0,0 +1,66 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* zpdesc.h: zswap.zpool memory descriptor + * + * Written by Alex Shi (Tencent) + * Hyeonggon Yoo <42.hyeyoo@gmail.com> + */ +#ifndef __MM_ZPDESC_H__ +#define __MM_ZPDESC_H__ + +/* + * struct zpdesc - Memory descriptor for zpool memory, now is for zsmalloc + * @flags: Page flags, PG_private: identifies the first component page + * @lru: Indirectly used by page migration + * @mops: Used by page migration + * @next: Next zpdesc in a zspage in zsmalloc zpool + * @handle: For huge zspage in zsmalloc zpool + * @zspage: Pointer to zspage in zsmalloc + * @memcg_data: Memory Control Group data. + * + * This struct overlays struct page for now. Do not modify without a good + * understanding of the issues. + */ +struct zpdesc { + unsigned long flags; + struct list_head lru; + struct movable_operations *mops; + union { + /* Next zpdescs in a zspage in zsmalloc zpool */ + struct zpdesc *next; + /* For huge zspage in zsmalloc zpool */ + unsigned long handle; + }; + struct zspage *zspage; + unsigned long _zp_pad_1; +#ifdef CONFIG_MEMCG + unsigned long memcg_data; +#endif +}; +#define ZPDESC_MATCH(pg, zp) \ + static_assert(offsetof(struct page, pg) =3D=3D offsetof(struct zpdesc, zp= )) + +ZPDESC_MATCH(flags, flags); +ZPDESC_MATCH(lru, lru); +ZPDESC_MATCH(mapping, mops); +ZPDESC_MATCH(index, next); +ZPDESC_MATCH(index, handle); +ZPDESC_MATCH(private, zspage); +#ifdef CONFIG_MEMCG +ZPDESC_MATCH(memcg_data, memcg_data); +#endif +#undef ZPDESC_MATCH +static_assert(sizeof(struct zpdesc) <=3D sizeof(struct page)); + +#define zpdesc_page(zp) (_Generic((zp), \ + const struct zpdesc *: (const struct page *)(zp), \ + struct zpdesc *: (struct page *)(zp))) + +#define zpdesc_folio(zp) (_Generic((zp), \ + const struct zpdesc *: (const struct folio *)(zp), \ + struct zpdesc *: (struct folio *)(zp))) + +#define page_zpdesc(p) (_Generic((p), \ + const struct page *: (const struct zpdesc *)(p), \ + struct page *: (struct zpdesc *)(p))) + +#endif diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 5d6581ab7c07..a532851025f9 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -13,20 +13,18 @@ =20 /* * Following is how we use various fields and flags of underlying - * struct page(s) to form a zspage. + * struct zpdesc(page) to form a zspage. * - * Usage of struct page fields: - * page->private: points to zspage - * page->index: links together all component pages of a zspage + * Usage of struct zpdesc fields: + * zpdesc->zspage: points to zspage + * zpdesc->next: links together all component pages of a zspage * For the huge page, this is always 0, so we use this field * to store handle. * page->page_type: PG_zsmalloc, lower 16 bit locate the first object * offset in a subpage of a zspage * - * Usage of struct page flags: + * Usage of struct zpdesc(page) flags: * PG_private: identifies the first component page - * PG_owner_priv_1: identifies the huge component page - * */ =20 #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt @@ -64,6 +62,7 @@ #include #include #include +#include "zpdesc.h" =20 #define ZSPAGE_MAGIC 0x58 =20 @@ -253,7 +252,7 @@ struct zspage { }; unsigned int inuse; unsigned int freeobj; - struct page *first_page; + struct zpdesc *first_zpdesc; struct list_head list; /* fullness list */ struct zs_pool *pool; rwlock_t lock; @@ -448,7 +447,7 @@ static inline void mod_zspage_inuse(struct zspage *zspa= ge, int val) =20 static inline struct page *get_first_page(struct zspage *zspage) { - struct page *first_page =3D zspage->first_page; + struct page *first_page =3D zpdesc_page(zspage->first_zpdesc); =20 VM_BUG_ON_PAGE(!is_first_page(first_page), first_page); return first_page; @@ -948,7 +947,7 @@ static void create_page_chain(struct size_class *class,= struct zspage *zspage, set_page_private(page, (unsigned long)zspage); page->index =3D 0; if (i =3D=3D 0) { - zspage->first_page =3D page; + zspage->first_zpdesc =3D page_zpdesc(page); SetPagePrivate(page); if (unlikely(class->objs_per_zspage =3D=3D 1 && class->pages_per_zspage =3D=3D 1)) @@ -1324,7 +1323,7 @@ static unsigned long obj_malloc(struct zs_pool *pool, link->handle =3D handle | OBJ_ALLOCATED_TAG; else /* record handle to page->index */ - zspage->first_page->index =3D handle | OBJ_ALLOCATED_TAG; + zspage->first_zpdesc->handle =3D handle | OBJ_ALLOCATED_TAG; =20 kunmap_atomic(vaddr); mod_zspage_inuse(zspage, 1); --=20 2.43.0 From nobody Sun Feb 8 02:21:45 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1E6531C287 for ; Mon, 8 Jul 2024 06:28:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720420129; cv=none; b=BlgQyJCtF8RM3QiXiKcuiju6UM5Faf2QVU5bJTb7A75os/aEULW75HEN/5IGKNo7jvpuzbCj3jy4J7lKcF5n7ITPESkxr0ECk1KXJepk/NhMAfVUsHuMc4ufALW48HQ+CvzsnzXrsiINebDgEmuSNZ29fq1VwaQ0kc/46REcbvA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720420129; c=relaxed/simple; bh=4H8b1UkHtxXxZUO1ViFEqBmxLyLsVYUHjKLF8FeKQ/8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=cV5xdh7BGQs0/P3uE53YZQZ39NKIUUqUUC50Cm7cwhHBHcDTQDvty4g3a9splL+ha0irfwnmoMM+VxNWYg58afhpVG17lUGdplFHpiyQVwcT5gW3gmBWWjujsvnxKkmXTkPcHjkWDyCqA/8ZO+BRpo8uxSLvXCfOYhci8DxrZxk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=kpT8bdSg; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="kpT8bdSg" Received: by smtp.kernel.org (Postfix) with ESMTPSA id DF041C4AF0D; Mon, 8 Jul 2024 06:28:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1720420129; bh=4H8b1UkHtxXxZUO1ViFEqBmxLyLsVYUHjKLF8FeKQ/8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=kpT8bdSgT3bmJpQR4uQG2X4CW6Kx4jj3I+s0aC4vc9tFaJ2B68jJshgCE37tGoa5s 1CMTQ7jKY4sI9dbb1JydFGhBLT58ChokB7oWgM4nTNnG5pW4ZvRr4mPXTTVFCqH4SM 42Q4qfgkCHZmv8r8rXGFYMPzdDe8SnnwmEU6ya/yEGxrCNRKv/Xm1OGDtunKYiBzsI ULYN+GTR22haM6FRpvpVRyAAWMLglJZdTZ9SaYpdFHK+sK4XYgwRIk0y319HE13Wgm THOoR1bVe7ACwR/zn/gTiY6XvbopHfVbrXFA9RpfOvXLFHXR4EK31WpKiqwvXKpkcG Ye2YagdhzAvgw== From: alexs@kernel.org To: Vitaly Wool , Miaohe Lin , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, minchan@kernel.org, willy@infradead.org, senozhatsky@chromium.org, david@redhat.com, 42.hyeyoo@gmail.com, Yosry Ahmed , nphamcs@gmail.com Cc: Alex Shi Subject: [PATCH v3 02/20] mm/zsmalloc: use zpdesc in trylock_zspage/lock_zspage Date: Mon, 8 Jul 2024 14:33:23 +0800 Message-ID: <20240708063344.1096626-3-alexs@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240708063344.1096626-1-alexs@kernel.org> References: <20240708063344.1096626-1-alexs@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Alex Shi (Tencent) To use zpdesc in trylock_zspage/lock_zspage funcs, we add couple of helpers: zpdesc_lock/zpdesc_unlock/zpdesc_trylock/zpdesc_wait_locked and zpdesc_get/zpdesc_put for this purpose. Here we use the folio series func in guts for 2 reasons, one zswap.zpool only get single page, and use folio could save some compound_head checking; two, folio_put could bypass devmap checking that we don't need. Originally-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Signed-off-by: Alex Shi (Tencent) --- mm/zpdesc.h | 30 ++++++++++++++++++++++++ mm/zsmalloc.c | 64 ++++++++++++++++++++++++++++++++++----------------- 2 files changed, 73 insertions(+), 21 deletions(-) diff --git a/mm/zpdesc.h b/mm/zpdesc.h index 2dbef231f616..3b04197cec9d 100644 --- a/mm/zpdesc.h +++ b/mm/zpdesc.h @@ -63,4 +63,34 @@ static_assert(sizeof(struct zpdesc) <=3D sizeof(struct p= age)); const struct page *: (const struct zpdesc *)(p), \ struct page *: (struct zpdesc *)(p))) =20 +static inline void zpdesc_lock(struct zpdesc *zpdesc) +{ + folio_lock(zpdesc_folio(zpdesc)); +} + +static inline bool zpdesc_trylock(struct zpdesc *zpdesc) +{ + return folio_trylock(zpdesc_folio(zpdesc)); +} + +static inline void zpdesc_unlock(struct zpdesc *zpdesc) +{ + folio_unlock(zpdesc_folio(zpdesc)); +} + +static inline void zpdesc_wait_locked(struct zpdesc *zpdesc) +{ + folio_wait_locked(zpdesc_folio(zpdesc)); +} + +static inline void zpdesc_get(struct zpdesc *zpdesc) +{ + folio_get(zpdesc_folio(zpdesc)); +} + +static inline void zpdesc_put(struct zpdesc *zpdesc) +{ + folio_put(zpdesc_folio(zpdesc)); +} + #endif diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index a532851025f9..243677a9c6d2 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -433,13 +433,17 @@ static __maybe_unused int is_first_page(struct page *= page) return PagePrivate(page); } =20 +static int is_first_zpdesc(struct zpdesc *zpdesc) +{ + return PagePrivate(zpdesc_page(zpdesc)); +} + /* Protected by class->lock */ static inline int get_zspage_inuse(struct zspage *zspage) { return zspage->inuse; } =20 - static inline void mod_zspage_inuse(struct zspage *zspage, int val) { zspage->inuse +=3D val; @@ -453,6 +457,14 @@ static inline struct page *get_first_page(struct zspag= e *zspage) return first_page; } =20 +static struct zpdesc *get_first_zpdesc(struct zspage *zspage) +{ + struct zpdesc *first_zpdesc =3D zspage->first_zpdesc; + + VM_BUG_ON_PAGE(!is_first_zpdesc(first_zpdesc), zpdesc_page(first_zpdesc)); + return first_zpdesc; +} + #define FIRST_OBJ_PAGE_TYPE_MASK 0xffff =20 static inline void reset_first_obj_offset(struct page *page) @@ -745,6 +757,16 @@ static struct page *get_next_page(struct page *page) return (struct page *)page->index; } =20 +static struct zpdesc *get_next_zpdesc(struct zpdesc *zpdesc) +{ + struct zspage *zspage =3D get_zspage(zpdesc_page(zpdesc)); + + if (unlikely(ZsHugePage(zspage))) + return NULL; + + return zpdesc->next; +} + /** * obj_to_location - get (, ) from encoded object value * @obj: the encoded object value @@ -815,11 +837,11 @@ static void reset_page(struct page *page) =20 static int trylock_zspage(struct zspage *zspage) { - struct page *cursor, *fail; + struct zpdesc *cursor, *fail; =20 - for (cursor =3D get_first_page(zspage); cursor !=3D NULL; cursor =3D - get_next_page(cursor)) { - if (!trylock_page(cursor)) { + for (cursor =3D get_first_zpdesc(zspage); cursor !=3D NULL; cursor =3D + get_next_zpdesc(cursor)) { + if (!zpdesc_trylock(cursor)) { fail =3D cursor; goto unlock; } @@ -827,9 +849,9 @@ static int trylock_zspage(struct zspage *zspage) =20 return 1; unlock: - for (cursor =3D get_first_page(zspage); cursor !=3D fail; cursor =3D - get_next_page(cursor)) - unlock_page(cursor); + for (cursor =3D get_first_zpdesc(zspage); cursor !=3D fail; cursor =3D + get_next_zpdesc(cursor)) + zpdesc_unlock(cursor); =20 return 0; } @@ -1658,7 +1680,7 @@ static int putback_zspage(struct size_class *class, s= truct zspage *zspage) */ static void lock_zspage(struct zspage *zspage) { - struct page *curr_page, *page; + struct zpdesc *curr_zpdesc, *zpdesc; =20 /* * Pages we haven't locked yet can be migrated off the list while we're @@ -1670,24 +1692,24 @@ static void lock_zspage(struct zspage *zspage) */ while (1) { migrate_read_lock(zspage); - page =3D get_first_page(zspage); - if (trylock_page(page)) + zpdesc =3D get_first_zpdesc(zspage); + if (zpdesc_trylock(zpdesc)) break; - get_page(page); + zpdesc_get(zpdesc); migrate_read_unlock(zspage); - wait_on_page_locked(page); - put_page(page); + zpdesc_wait_locked(zpdesc); + zpdesc_put(zpdesc); } =20 - curr_page =3D page; - while ((page =3D get_next_page(curr_page))) { - if (trylock_page(page)) { - curr_page =3D page; + curr_zpdesc =3D zpdesc; + while ((zpdesc =3D get_next_zpdesc(curr_zpdesc))) { + if (zpdesc_trylock(zpdesc)) { + curr_zpdesc =3D zpdesc; } else { - get_page(page); + zpdesc_get(zpdesc); migrate_read_unlock(zspage); - wait_on_page_locked(page); - put_page(page); + zpdesc_wait_locked(zpdesc); + zpdesc_put(zpdesc); migrate_read_lock(zspage); } } --=20 2.43.0 From nobody Sun Feb 8 02:21:45 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E8BB01DA5F for ; Mon, 8 Jul 2024 06:28:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720420133; cv=none; b=LHoTIxxK90Bck9W8wz+tFYKEyiBGT6exUl1InySMAQXXPuFvEV4c5xH9ju4/LuWJvdEN4BtytO92IEt7ohXFsfGTIgr3rJCZ1dWY1cSE8/BL6d9cMPUwxdSra4CuKo0WFKc7sW15z7t/vFblWq9fnl5xuIJDI88QwrpGQVBx11M= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720420133; c=relaxed/simple; bh=tmhg+UoGuxLgiU/lnh0EuqzBHj5JgCLnTFMLR6vDXTI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=N68G53APX9bWuZjwa+s1poUgJg+NA/Xh9owGVM8cEr/H1gdwnQy7EzRZgo1U5tIyT8TU2pPXrbSXV5tM6iwhbajCQyY3S33I/Jhkz4LGAQiA56eLZDeLIpMXpbWU6ZuYQB5qPblmwnNFkKkobTMLcGfOGJjtXnJAnBGSuLqLARw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=sIFBrxAh; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="sIFBrxAh" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9EC9AC116B1; Mon, 8 Jul 2024 06:28:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1720420132; bh=tmhg+UoGuxLgiU/lnh0EuqzBHj5JgCLnTFMLR6vDXTI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=sIFBrxAh+/uE4yv8Zd7MjpS9ccK07cFXKmfpP6Zay+VljtAK9V1ULuV2jXwDkIedx V715o8OO3VX4wzi3m7mMJYZwfQ+fFFf4QqDMjdcW76M781nHLVnMdsWMQb7d9NaqW7 63O+xYhqtBjdx76BJ9dY662CFuFOfHPjOZV81/eUEa5g6yuL+st800m6rL/1u+wORn oVGzoxHA5x+orY+sjaOiWnp3opLoh388Qd+uKplMd6+eVHwHHbWp8cPq5+mzY4eK0w 5I98WraI3cj1UYwMg59uY3taov4j525VdhWGHymwEupsZOqr1iHiN88zcvjLqBSmoe nEpSVZXJv0MNw== From: alexs@kernel.org To: Vitaly Wool , Miaohe Lin , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, minchan@kernel.org, willy@infradead.org, senozhatsky@chromium.org, david@redhat.com, 42.hyeyoo@gmail.com, Yosry Ahmed , nphamcs@gmail.com Cc: Alex Shi Subject: [PATCH v3 03/20] mm/zsmalloc: convert __zs_map_object/__zs_unmap_object to use zpdesc Date: Mon, 8 Jul 2024 14:33:24 +0800 Message-ID: <20240708063344.1096626-4-alexs@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240708063344.1096626-1-alexs@kernel.org> References: <20240708063344.1096626-1-alexs@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Hyeonggon Yoo <42.hyeyoo@gmail.com> These two functions take pointer to an array of struct page. Introduce zpdesc_kmap_atomic() and make __zs_{map,unmap}_object() take pointer to an array of zpdesc instead of page. Add silly type casting when calling them. Casting will be removed late. Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Signed-off-by: Alex Shi (Tencent) --- mm/zsmalloc.c | 21 +++++++++++++-------- 1 file changed, 13 insertions(+), 8 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 243677a9c6d2..68a39c233d34 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -243,6 +243,11 @@ struct zs_pool { atomic_t compaction_in_progress; }; =20 +static inline void *zpdesc_kmap_atomic(struct zpdesc *zpdesc) +{ + return kmap_atomic(zpdesc_page(zpdesc)); +} + struct zspage { struct { unsigned int huge:HUGE_BITS; @@ -1061,7 +1066,7 @@ static inline void __zs_cpu_down(struct mapping_area = *area) } =20 static void *__zs_map_object(struct mapping_area *area, - struct page *pages[2], int off, int size) + struct zpdesc *zpdescs[2], int off, int size) { int sizes[2]; void *addr; @@ -1078,10 +1083,10 @@ static void *__zs_map_object(struct mapping_area *a= rea, sizes[1] =3D size - sizes[0]; =20 /* copy object to per-cpu buffer */ - addr =3D kmap_atomic(pages[0]); + addr =3D zpdesc_kmap_atomic(zpdescs[0]); memcpy(buf, addr + off, sizes[0]); kunmap_atomic(addr); - addr =3D kmap_atomic(pages[1]); + addr =3D zpdesc_kmap_atomic(zpdescs[1]); memcpy(buf + sizes[0], addr, sizes[1]); kunmap_atomic(addr); out: @@ -1089,7 +1094,7 @@ static void *__zs_map_object(struct mapping_area *are= a, } =20 static void __zs_unmap_object(struct mapping_area *area, - struct page *pages[2], int off, int size) + struct zpdesc *zpdescs[2], int off, int size) { int sizes[2]; void *addr; @@ -1108,10 +1113,10 @@ static void __zs_unmap_object(struct mapping_area *= area, sizes[1] =3D size - sizes[0]; =20 /* copy per-cpu buffer to object */ - addr =3D kmap_atomic(pages[0]); + addr =3D zpdesc_kmap_atomic(zpdescs[0]); memcpy(addr + off, buf, sizes[0]); kunmap_atomic(addr); - addr =3D kmap_atomic(pages[1]); + addr =3D zpdesc_kmap_atomic(zpdescs[1]); memcpy(addr, buf + sizes[0], sizes[1]); kunmap_atomic(addr); =20 @@ -1252,7 +1257,7 @@ void *zs_map_object(struct zs_pool *pool, unsigned lo= ng handle, pages[1] =3D get_next_page(page); BUG_ON(!pages[1]); =20 - ret =3D __zs_map_object(area, pages, off, class->size); + ret =3D __zs_map_object(area, (struct zpdesc **)pages, off, class->size); out: if (likely(!ZsHugePage(zspage))) ret +=3D ZS_HANDLE_SIZE; @@ -1287,7 +1292,7 @@ void zs_unmap_object(struct zs_pool *pool, unsigned l= ong handle) pages[1] =3D get_next_page(page); BUG_ON(!pages[1]); =20 - __zs_unmap_object(area, pages, off, class->size); + __zs_unmap_object(area, (struct zpdesc **)pages, off, class->size); } local_unlock(&zs_map_area.lock); =20 --=20 2.43.0 From nobody Sun Feb 8 02:21:45 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AC44E171C2 for ; Mon, 8 Jul 2024 06:28:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720420136; cv=none; b=jPgZoN/MsesVv2WPHvdYE7iJZdGb5EbJDviaru5dr2qXOIfdwiHidBAmRfQhHy+q61swSEJO1xY5Ar/NMJhccieULathHTmKIq94qUyGod2ZUNLsqzV5TG5i5SA0E8pztsLuk+UINMRGxx4g48p0oUTuVsdcFwO1rjMkZhZBMh8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720420136; c=relaxed/simple; bh=83jcf8H2ELpZcgLjgz32nYRKXID3GsZDqOi1CXzndR8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=P2AG/7Oie3ca7Pe5gtR7svd8zT7ZreuBeLw5qHxeGGB35vza5ixRMgdv7gnwRvxA6oubJTgQF7HBfWWSfzViwc9ui/I2uhpNfHudP7dL3rqt4vSFZwxjODOB49/cQe9RlwScMja9XWMvNQ8VAKHsjM4h8cLj3smwvrbEfbIQLMs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Bs2Uv9IC; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Bs2Uv9IC" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 3A694C4AF0B; Mon, 8 Jul 2024 06:28:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1720420136; bh=83jcf8H2ELpZcgLjgz32nYRKXID3GsZDqOi1CXzndR8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Bs2Uv9ICcXS8Sew+dSWNGK1eSMFRpY70r/3FWCgrZS6JN0tiH0+a5nxvKN3Yl+YtJ gE6Yz+5wSlHjGep1v2dYdKBlMunZqqBAcDd/GvG4acDD0vI6Wh6K8moqosV4r+F101 swVXn7hn2qMzhAygustSXm0WSTAGmTgHatJjzn5dtzR2Xz8f3cmGuh78nTxuRQAtSR wjFMhlS7jRFJS2gaRYSZEWKnYaraHCvpp+LwayujCyA0Vdiw5RWRlgNOGpBn98TUwj vE+qHsptRfzPeVDJXYFkB94b5EiWwl1ZPQvgc6vnBTsk1dG/FJYPxo+LpP8AkKwCIB guhs8FMg1J+Hw== From: alexs@kernel.org To: Vitaly Wool , Miaohe Lin , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, minchan@kernel.org, willy@infradead.org, senozhatsky@chromium.org, david@redhat.com, 42.hyeyoo@gmail.com, Yosry Ahmed , nphamcs@gmail.com Cc: Alex Shi Subject: [PATCH v3 04/20] mm/zsmalloc: add and use pfn/zpdesc seeking funcs Date: Mon, 8 Jul 2024 14:33:25 +0800 Message-ID: <20240708063344.1096626-5-alexs@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240708063344.1096626-1-alexs@kernel.org> References: <20240708063344.1096626-1-alexs@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Hyeonggon Yoo <42.hyeyoo@gmail.com> Add pfn_zpdesc conversion, convert obj_to_location() to take zpdesc and also convert its users to use zpdesc. Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Signed-off-by: Alex Shi (Tencent) --- mm/zpdesc.h | 9 +++++++ mm/zsmalloc.c | 75 ++++++++++++++++++++++++++------------------------- 2 files changed, 47 insertions(+), 37 deletions(-) diff --git a/mm/zpdesc.h b/mm/zpdesc.h index 3b04197cec9d..79ec40b03956 100644 --- a/mm/zpdesc.h +++ b/mm/zpdesc.h @@ -93,4 +93,13 @@ static inline void zpdesc_put(struct zpdesc *zpdesc) folio_put(zpdesc_folio(zpdesc)); } =20 +static inline unsigned long zpdesc_pfn(struct zpdesc *zpdesc) +{ + return page_to_pfn(zpdesc_page(zpdesc)); +} + +static inline struct zpdesc *pfn_zpdesc(unsigned long pfn) +{ + return page_zpdesc(pfn_to_page(pfn)); +} #endif diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 68a39c233d34..149fe2b332cb 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -773,15 +773,15 @@ static struct zpdesc *get_next_zpdesc(struct zpdesc *= zpdesc) } =20 /** - * obj_to_location - get (, ) from encoded object value + * obj_to_location - get (, ) from encoded object value * @obj: the encoded object value - * @page: page object resides in zspage + * @zpdesc: zpdesc object resides in zspage * @obj_idx: object index */ -static void obj_to_location(unsigned long obj, struct page **page, +static void obj_to_location(unsigned long obj, struct zpdesc **zpdesc, unsigned int *obj_idx) { - *page =3D pfn_to_page(obj >> OBJ_INDEX_BITS); + *zpdesc =3D pfn_zpdesc(obj >> OBJ_INDEX_BITS); *obj_idx =3D (obj & OBJ_INDEX_MASK); } =20 @@ -1208,13 +1208,13 @@ void *zs_map_object(struct zs_pool *pool, unsigned = long handle, enum zs_mapmode mm) { struct zspage *zspage; - struct page *page; + struct zpdesc *zpdesc; unsigned long obj, off; unsigned int obj_idx; =20 struct size_class *class; struct mapping_area *area; - struct page *pages[2]; + struct zpdesc *zpdescs[2]; void *ret; =20 /* @@ -1227,8 +1227,8 @@ void *zs_map_object(struct zs_pool *pool, unsigned lo= ng handle, /* It guarantees it can get zspage from handle safely */ read_lock(&pool->migrate_lock); obj =3D handle_to_obj(handle); - obj_to_location(obj, &page, &obj_idx); - zspage =3D get_zspage(page); + obj_to_location(obj, &zpdesc, &obj_idx); + zspage =3D get_zspage(zpdesc_page(zpdesc)); =20 /* * migration cannot move any zpages in this zspage. Here, class->lock @@ -1247,17 +1247,17 @@ void *zs_map_object(struct zs_pool *pool, unsigned = long handle, area->vm_mm =3D mm; if (off + class->size <=3D PAGE_SIZE) { /* this object is contained entirely within a page */ - area->vm_addr =3D kmap_atomic(page); + area->vm_addr =3D zpdesc_kmap_atomic(zpdesc); ret =3D area->vm_addr + off; goto out; } =20 /* this object spans two pages */ - pages[0] =3D page; - pages[1] =3D get_next_page(page); - BUG_ON(!pages[1]); + zpdescs[0] =3D zpdesc; + zpdescs[1] =3D get_next_zpdesc(zpdesc); + BUG_ON(!zpdescs[1]); =20 - ret =3D __zs_map_object(area, (struct zpdesc **)pages, off, class->size); + ret =3D __zs_map_object(area, zpdescs, off, class->size); out: if (likely(!ZsHugePage(zspage))) ret +=3D ZS_HANDLE_SIZE; @@ -1269,7 +1269,7 @@ EXPORT_SYMBOL_GPL(zs_map_object); void zs_unmap_object(struct zs_pool *pool, unsigned long handle) { struct zspage *zspage; - struct page *page; + struct zpdesc *zpdesc; unsigned long obj, off; unsigned int obj_idx; =20 @@ -1277,8 +1277,8 @@ void zs_unmap_object(struct zs_pool *pool, unsigned l= ong handle) struct mapping_area *area; =20 obj =3D handle_to_obj(handle); - obj_to_location(obj, &page, &obj_idx); - zspage =3D get_zspage(page); + obj_to_location(obj, &zpdesc, &obj_idx); + zspage =3D get_zspage(zpdesc_page(zpdesc)); class =3D zspage_class(pool, zspage); off =3D offset_in_page(class->size * obj_idx); =20 @@ -1286,13 +1286,13 @@ void zs_unmap_object(struct zs_pool *pool, unsigned= long handle) if (off + class->size <=3D PAGE_SIZE) kunmap_atomic(area->vm_addr); else { - struct page *pages[2]; + struct zpdesc *zpdescs[2]; =20 - pages[0] =3D page; - pages[1] =3D get_next_page(page); - BUG_ON(!pages[1]); + zpdescs[0] =3D zpdesc; + zpdescs[1] =3D get_next_zpdesc(zpdesc); + BUG_ON(!zpdescs[1]); =20 - __zs_unmap_object(area, (struct zpdesc **)pages, off, class->size); + __zs_unmap_object(area, zpdescs, off, class->size); } local_unlock(&zs_map_area.lock); =20 @@ -1434,23 +1434,24 @@ static void obj_free(int class_size, unsigned long = obj) { struct link_free *link; struct zspage *zspage; - struct page *f_page; + struct zpdesc *f_zpdesc; unsigned long f_offset; unsigned int f_objidx; void *vaddr; =20 - obj_to_location(obj, &f_page, &f_objidx); + + obj_to_location(obj, &f_zpdesc, &f_objidx); f_offset =3D offset_in_page(class_size * f_objidx); - zspage =3D get_zspage(f_page); + zspage =3D get_zspage(zpdesc_page(f_zpdesc)); =20 - vaddr =3D kmap_atomic(f_page); + vaddr =3D zpdesc_kmap_atomic(f_zpdesc); link =3D (struct link_free *)(vaddr + f_offset); =20 /* Insert this object in containing zspage's freelist */ if (likely(!ZsHugePage(zspage))) link->next =3D get_freeobj(zspage) << OBJ_TAG_BITS; else - f_page->index =3D 0; + f_zpdesc->next =3D NULL; set_freeobj(zspage, f_objidx); =20 kunmap_atomic(vaddr); @@ -1495,7 +1496,7 @@ EXPORT_SYMBOL_GPL(zs_free); static void zs_object_copy(struct size_class *class, unsigned long dst, unsigned long src) { - struct page *s_page, *d_page; + struct zpdesc *s_zpdesc, *d_zpdesc; unsigned int s_objidx, d_objidx; unsigned long s_off, d_off; void *s_addr, *d_addr; @@ -1504,8 +1505,8 @@ static void zs_object_copy(struct size_class *class, = unsigned long dst, =20 s_size =3D d_size =3D class->size; =20 - obj_to_location(src, &s_page, &s_objidx); - obj_to_location(dst, &d_page, &d_objidx); + obj_to_location(src, &s_zpdesc, &s_objidx); + obj_to_location(dst, &d_zpdesc, &d_objidx); =20 s_off =3D offset_in_page(class->size * s_objidx); d_off =3D offset_in_page(class->size * d_objidx); @@ -1516,8 +1517,8 @@ static void zs_object_copy(struct size_class *class, = unsigned long dst, if (d_off + class->size > PAGE_SIZE) d_size =3D PAGE_SIZE - d_off; =20 - s_addr =3D kmap_atomic(s_page); - d_addr =3D kmap_atomic(d_page); + s_addr =3D zpdesc_kmap_atomic(s_zpdesc); + d_addr =3D zpdesc_kmap_atomic(d_zpdesc); =20 while (1) { size =3D min(s_size, d_size); @@ -1542,17 +1543,17 @@ static void zs_object_copy(struct size_class *class= , unsigned long dst, if (s_off >=3D PAGE_SIZE) { kunmap_atomic(d_addr); kunmap_atomic(s_addr); - s_page =3D get_next_page(s_page); - s_addr =3D kmap_atomic(s_page); - d_addr =3D kmap_atomic(d_page); + s_zpdesc =3D get_next_zpdesc(s_zpdesc); + s_addr =3D zpdesc_kmap_atomic(s_zpdesc); + d_addr =3D zpdesc_kmap_atomic(d_zpdesc); s_size =3D class->size - written; s_off =3D 0; } =20 if (d_off >=3D PAGE_SIZE) { kunmap_atomic(d_addr); - d_page =3D get_next_page(d_page); - d_addr =3D kmap_atomic(d_page); + d_zpdesc =3D get_next_zpdesc(d_zpdesc); + d_addr =3D zpdesc_kmap_atomic(d_zpdesc); d_size =3D class->size - written; d_off =3D 0; } @@ -1791,7 +1792,7 @@ static int zs_page_migrate(struct page *newpage, stru= ct page *page, struct zs_pool *pool; struct size_class *class; struct zspage *zspage; - struct page *dummy; + struct zpdesc *dummy; void *s_addr, *d_addr, *addr; unsigned int offset; unsigned long handle; --=20 2.43.0 From nobody Sun Feb 8 02:21:45 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 09D891F94D for ; Mon, 8 Jul 2024 06:29:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720420140; cv=none; b=hSWEnibX4AYGLLfNWo7RCFhB7XNXm6tZTheMPFvpDSkJGmfy++EY7QFYBcEuN+u2EgcqSt81r1phRhDby1fQZ19mwiehT2rQ/M5lTiIEVHkc6wNtdDZGnld8w8q3TcCmqEYXNujRZvEdL7zuH8wbTqJeB3EseTB0rLSgq7T31PI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720420140; c=relaxed/simple; bh=Kaf63ckMzfd8JfdGDWbfHAhpNFgdzACIKJu7j9bsBGA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=SingpfeVkfsB5M3STCIPUv9DWPXDpOTzC35D6MHFFJyUEZiYXRw6209lhJgVd6iVUWxe5plcOeS8461ZWxr4tVPo2B2Rbg1M7GhRrYBVoyuzh4kpC896o76TatFHd/z7jTILOeSJ5IuXRIaMP8I1z5aI/1OFJqrJR+HH9p19Ljc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=e+9GU7+O; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="e+9GU7+O" Received: by smtp.kernel.org (Postfix) with ESMTPSA id EC9ABC116B1; Mon, 8 Jul 2024 06:28:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1720420139; bh=Kaf63ckMzfd8JfdGDWbfHAhpNFgdzACIKJu7j9bsBGA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=e+9GU7+OFriAgyhewgBuS4MpJ13g2HtbEpGw2kuHmhxZGAFHfZrMhm5wfrLZt6eSu MF2wyNfgAdjMs9E8qq5ZSCV9PMJAr4/A1OaqAx8uKJabfBUQR38cugW/jaeyMk42rK RQMEpRHjPzDEPwQrXf+OZ6N9tpRgoJRDv++LCDOZxvLEdvf+FMPtlmENm8JmLvLO6a +/RnpebZBStyMYM1IHwbCJKZpxkRFUUEpCMF/DK5JHekR4mjFG/ANV615+EmvR0SLR OXmgjPJq0yh2NRtIZKeP5dQYB1AQ/luVVGcKmwcAKJT4Lg43bXs4RbfPa+gouK3IQm ul+eMziT36sgg== From: alexs@kernel.org To: Vitaly Wool , Miaohe Lin , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, minchan@kernel.org, willy@infradead.org, senozhatsky@chromium.org, david@redhat.com, 42.hyeyoo@gmail.com, Yosry Ahmed , nphamcs@gmail.com Cc: Alex Shi Subject: [PATCH v3 05/20] mm/zsmalloc: convert obj_malloc() to use zpdesc Date: Mon, 8 Jul 2024 14:33:26 +0800 Message-ID: <20240708063344.1096626-6-alexs@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240708063344.1096626-1-alexs@kernel.org> References: <20240708063344.1096626-1-alexs@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Hyeonggon Yoo <42.hyeyoo@gmail.com> Use get_first_zpdesc/get_next_zpdesc to replace get_first_page/get_next_page. no functional change. Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Signed-off-by: Alex Shi (Tencent) --- mm/zsmalloc.c | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 149fe2b332cb..bbc165cb587d 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -1322,12 +1322,12 @@ EXPORT_SYMBOL_GPL(zs_huge_class_size); static unsigned long obj_malloc(struct zs_pool *pool, struct zspage *zspage, unsigned long handle) { - int i, nr_page, offset; + int i, nr_zpdesc, offset; unsigned long obj; struct link_free *link; struct size_class *class; =20 - struct page *m_page; + struct zpdesc *m_zpdesc; unsigned long m_offset; void *vaddr; =20 @@ -1335,14 +1335,14 @@ static unsigned long obj_malloc(struct zs_pool *poo= l, obj =3D get_freeobj(zspage); =20 offset =3D obj * class->size; - nr_page =3D offset >> PAGE_SHIFT; + nr_zpdesc =3D offset >> PAGE_SHIFT; m_offset =3D offset_in_page(offset); - m_page =3D get_first_page(zspage); + m_zpdesc =3D get_first_zpdesc(zspage); =20 - for (i =3D 0; i < nr_page; i++) - m_page =3D get_next_page(m_page); + for (i =3D 0; i < nr_zpdesc; i++) + m_zpdesc =3D get_next_zpdesc(m_zpdesc); =20 - vaddr =3D kmap_atomic(m_page); + vaddr =3D zpdesc_kmap_atomic(m_zpdesc); link =3D (struct link_free *)vaddr + m_offset / sizeof(*link); set_freeobj(zspage, link->next >> OBJ_TAG_BITS); if (likely(!ZsHugePage(zspage))) @@ -1355,7 +1355,7 @@ static unsigned long obj_malloc(struct zs_pool *pool, kunmap_atomic(vaddr); mod_zspage_inuse(zspage, 1); =20 - obj =3D location_to_obj(m_page, obj); + obj =3D location_to_obj(zpdesc_page(m_zpdesc), obj); record_obj(handle, obj); =20 return obj; --=20 2.43.0 From nobody Sun Feb 8 02:21:45 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0552F29CE8 for ; Mon, 8 Jul 2024 06:29:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720420144; cv=none; b=FuVYw9JGDr05Fwr69p/0f6KE/lVCYPkIU7aTHtiCbeWxS3wYnxXKKT8bhXNaMQLA3DBXInniKFSPoSYRR57W+Ks4nVl9IQ4alJTccmF87jstBXcUbjENtk3Jav8j9bOhyyXX/6KoFyPT4QxuO2b+uleSr4mPE97dgrUqIglY0SI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720420144; c=relaxed/simple; bh=wQxGtmbL1aK1VwnBgVGS8tWob34VVxGd8gh+B3iTQwc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=P+32ZHlmg1HyfVRTE6K/xX4hj/Pfdhds7jUrEUnp1g7VHTVrMYs0X/JQWk17T142NcVwtSWHJUti7TEwOWGBDDSUwnFvD0LKrJsLjy2RgKpD++qVxRedhh9l4uZkTzid0+dKA1EtjrFZ+pJYEhArnuHF8771PwxtvxY7oDZjvyk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=iFPjsLTe; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="iFPjsLTe" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 85057C4AF0D; Mon, 8 Jul 2024 06:29:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1720420143; bh=wQxGtmbL1aK1VwnBgVGS8tWob34VVxGd8gh+B3iTQwc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=iFPjsLTezAT14viZJrWtHow972VaopLMBb8d3TjRdKFke5FbWCPT2+C5MZvc50sws SdN4cefEBvuK4j7c5szip31wVMQetv+8wvaxKWGHgWXkF7uXCSVdH2LAOkpAY1fiBd kqwXVTS4G+22/QZM+hSywN6/rBnYYRQbgnUrAVuvKNo8mucHvrgNnMeWWothY6XpIC fmshppA9dnJBTFqzO6yl1GlmgTYZvw+T4Q1+i7uz5dvZAkJTWw5+jucyJh7WJcK7d2 vyxh/8hCvC2GuTlXHECl5wussQuJi27BpBslTQUjnLwbm7JADBR8BBQlWHr8PK/ZRS NYDv6k1PMMX6Q== From: alexs@kernel.org To: Vitaly Wool , Miaohe Lin , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, minchan@kernel.org, willy@infradead.org, senozhatsky@chromium.org, david@redhat.com, 42.hyeyoo@gmail.com, Yosry Ahmed , nphamcs@gmail.com Cc: Alex Shi Subject: [PATCH v3 06/20] mm/zsmalloc: convert create_page_chain() and its users to use zpdesc Date: Mon, 8 Jul 2024 14:33:27 +0800 Message-ID: <20240708063344.1096626-7-alexs@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240708063344.1096626-1-alexs@kernel.org> References: <20240708063344.1096626-1-alexs@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Alex Shi (Tencent) Introduce a few helper functions for conversion to convert create_page_chai= n() to use zpdesc, then use zpdesc in replace_sub_page() too. Originally-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Signed-off-by: Alex Shi (Tencent) --- mm/zpdesc.h | 6 +++ mm/zsmalloc.c | 115 +++++++++++++++++++++++++++++++++----------------- 2 files changed, 82 insertions(+), 39 deletions(-) diff --git a/mm/zpdesc.h b/mm/zpdesc.h index 79ec40b03956..2293453f5d57 100644 --- a/mm/zpdesc.h +++ b/mm/zpdesc.h @@ -102,4 +102,10 @@ static inline struct zpdesc *pfn_zpdesc(unsigned long = pfn) { return page_zpdesc(pfn_to_page(pfn)); } + +static inline void __zpdesc_set_movable(struct zpdesc *zpdesc, + const struct movable_operations *mops) +{ + __SetPageMovable(zpdesc_page(zpdesc), mops); +} #endif diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index bbc165cb587d..a8f390beeab8 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -248,6 +248,41 @@ static inline void *zpdesc_kmap_atomic(struct zpdesc *= zpdesc) return kmap_atomic(zpdesc_page(zpdesc)); } =20 +static inline void zpdesc_set_zspage(struct zpdesc *zpdesc, + struct zspage *zspage) +{ + zpdesc->zspage =3D zspage; +} + +static inline void zpdesc_set_first(struct zpdesc *zpdesc) +{ + SetPagePrivate(zpdesc_page(zpdesc)); +} + +static inline void zpdesc_inc_zone_page_state(struct zpdesc *zpdesc) +{ + inc_zone_page_state(zpdesc_page(zpdesc), NR_ZSPAGES); +} + +static inline void zpdesc_dec_zone_page_state(struct zpdesc *zpdesc) +{ + dec_zone_page_state(zpdesc_page(zpdesc), NR_ZSPAGES); +} + +static inline struct zpdesc *alloc_zpdesc(gfp_t gfp) +{ + struct page *page =3D alloc_page(gfp); + + return page_zpdesc(page); +} + +static inline void free_zpdesc(struct zpdesc *zpdesc) +{ + struct page *page =3D zpdesc_page(zpdesc); + + __free_page(page); +} + struct zspage { struct { unsigned int huge:HUGE_BITS; @@ -954,35 +989,35 @@ static void init_zspage(struct size_class *class, str= uct zspage *zspage) } =20 static void create_page_chain(struct size_class *class, struct zspage *zsp= age, - struct page *pages[]) + struct zpdesc *zpdescs[]) { int i; - struct page *page; - struct page *prev_page =3D NULL; - int nr_pages =3D class->pages_per_zspage; + struct zpdesc *zpdesc; + struct zpdesc *prev_zpdesc =3D NULL; + int nr_zpdescs =3D class->pages_per_zspage; =20 /* * Allocate individual pages and link them together as: - * 1. all pages are linked together using page->index - * 2. each sub-page point to zspage using page->private + * 1. all pages are linked together using zpdesc->next + * 2. each sub-page point to zspage using zpdesc->zspage * - * we set PG_private to identify the first page (i.e. no other sub-page + * we set PG_private to identify the first zpdesc (i.e. no other zpdesc * has this flag set). */ - for (i =3D 0; i < nr_pages; i++) { - page =3D pages[i]; - set_page_private(page, (unsigned long)zspage); - page->index =3D 0; + for (i =3D 0; i < nr_zpdescs; i++) { + zpdesc =3D zpdescs[i]; + zpdesc_set_zspage(zpdesc, zspage); + zpdesc->next =3D NULL; if (i =3D=3D 0) { - zspage->first_zpdesc =3D page_zpdesc(page); - SetPagePrivate(page); + zspage->first_zpdesc =3D zpdesc; + zpdesc_set_first(zpdesc); if (unlikely(class->objs_per_zspage =3D=3D 1 && class->pages_per_zspage =3D=3D 1)) SetZsHugePage(zspage); } else { - prev_page->index =3D (unsigned long)page; + prev_zpdesc->next =3D zpdesc; } - prev_page =3D page; + prev_zpdesc =3D zpdesc; } } =20 @@ -994,7 +1029,7 @@ static struct zspage *alloc_zspage(struct zs_pool *poo= l, gfp_t gfp) { int i; - struct page *pages[ZS_MAX_PAGES_PER_ZSPAGE]; + struct zpdesc *zpdescs[ZS_MAX_PAGES_PER_ZSPAGE]; struct zspage *zspage =3D cache_alloc_zspage(pool, gfp); =20 if (!zspage) @@ -1004,25 +1039,25 @@ static struct zspage *alloc_zspage(struct zs_pool *= pool, migrate_lock_init(zspage); =20 for (i =3D 0; i < class->pages_per_zspage; i++) { - struct page *page; + struct zpdesc *zpdesc; =20 - page =3D alloc_page(gfp); - if (!page) { + zpdesc =3D alloc_zpdesc(gfp); + if (!zpdesc) { while (--i >=3D 0) { - dec_zone_page_state(pages[i], NR_ZSPAGES); - __ClearPageZsmalloc(pages[i]); - __free_page(pages[i]); + zpdesc_dec_zone_page_state(zpdescs[i]); + __ClearPageZsmalloc(zpdesc_page(zpdescs[i])); + free_zpdesc(zpdescs[i]); } cache_free_zspage(pool, zspage); return NULL; } - __SetPageZsmalloc(page); + __SetPageZsmalloc(zpdesc_page(zpdesc)); =20 - inc_zone_page_state(page, NR_ZSPAGES); - pages[i] =3D page; + zpdesc_inc_zone_page_state(zpdesc); + zpdescs[i] =3D zpdesc; } =20 - create_page_chain(class, zspage, pages); + create_page_chain(class, zspage, zpdescs); init_zspage(class, zspage); zspage->pool =3D pool; zspage->class =3D class->index; @@ -1753,26 +1788,28 @@ static void migrate_write_unlock(struct zspage *zsp= age) static const struct movable_operations zsmalloc_mops; =20 static void replace_sub_page(struct size_class *class, struct zspage *zspa= ge, - struct page *newpage, struct page *oldpage) + struct zpdesc *newzpdesc, struct zpdesc *oldzpdesc) { - struct page *page; - struct page *pages[ZS_MAX_PAGES_PER_ZSPAGE] =3D {NULL, }; + struct zpdesc *zpdesc; + struct zpdesc *zpdescs[ZS_MAX_PAGES_PER_ZSPAGE] =3D {NULL, }; + unsigned int first_obj_offset; int idx =3D 0; =20 - page =3D get_first_page(zspage); + zpdesc =3D get_first_zpdesc(zspage); do { - if (page =3D=3D oldpage) - pages[idx] =3D newpage; + if (zpdesc =3D=3D oldzpdesc) + zpdescs[idx] =3D newzpdesc; else - pages[idx] =3D page; + zpdescs[idx] =3D zpdesc; idx++; - } while ((page =3D get_next_page(page)) !=3D NULL); + } while ((zpdesc =3D get_next_zpdesc(zpdesc)) !=3D NULL); =20 - create_page_chain(class, zspage, pages); - set_first_obj_offset(newpage, get_first_obj_offset(oldpage)); + create_page_chain(class, zspage, zpdescs); + first_obj_offset =3D get_first_obj_offset(zpdesc_page(oldzpdesc)); + set_first_obj_offset(zpdesc_page(newzpdesc), first_obj_offset); if (unlikely(ZsHugePage(zspage))) - newpage->index =3D oldpage->index; - __SetPageMovable(newpage, &zsmalloc_mops); + newzpdesc->handle =3D oldzpdesc->handle; + __zpdesc_set_movable(newzpdesc, &zsmalloc_mops); } =20 static bool zs_page_isolate(struct page *page, isolate_mode_t mode) @@ -1845,7 +1882,7 @@ static int zs_page_migrate(struct page *newpage, stru= ct page *page, } kunmap_atomic(s_addr); =20 - replace_sub_page(class, zspage, newpage, page); + replace_sub_page(class, zspage, page_zpdesc(newpage), page_zpdesc(page)); /* * Since we complete the data copy and set up new zspage structure, * it's okay to release migration_lock. --=20 2.43.0 From nobody Sun Feb 8 02:21:45 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2D749376E9 for ; Mon, 8 Jul 2024 06:29:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720420148; cv=none; b=T7fegP5Jp2tBK6RG+75P7JpoAXFEn9O0eXBGGuauukUuCePdn3bNsnAZY0mITEs7e9X31i4SMLX+b6EbsHd8LqraesvcgSLLjd6Fbme9pAVqn8EkmxX6NSx1s1oQRTArv882E6uXtdGv87PuFt6xkTo2k/322Y1liC7ECNQt6ss= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720420148; c=relaxed/simple; bh=eUVr/nevUiypHdkqs90LAQGcxdPsPYteX43+BHTjXcE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=SiEMff3RsLUsVX1fBSXuzjNCqzQCIv8AAiI3+hzFVjgIRh4S/ctk+oqJRGaroYcIF7pgeG+Y4njwoaGHdavZsOfl4agpvN5V4XWBvyd2YyaQyhOjNjCMwuwpAaaITyPwTPd2dQ78YrgGhVEq8AI3vnPAQgOdHn8qI9sA8JlZAOk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=oZUatPhA; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="oZUatPhA" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 47754C116B1; Mon, 8 Jul 2024 06:29:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1720420147; bh=eUVr/nevUiypHdkqs90LAQGcxdPsPYteX43+BHTjXcE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=oZUatPhAeDrGrCo7sj3uta4gUIoHLfsfvRqLImOTiu9Ew6pEEgEFIOq0FSp+n5nwr QUQtfQqHNsuPdOaitMA8AzGvFRLInyN24C77Tk7y0e98ECidpaZy2xPt6WcuarP7L5 TBOyX1bZOTr7g5ZtY4U7xHoKTOGv9CJ0Ii9hB6TgLH4X+XjfqtBiqYubR/UrpapQHY bvCJQvBU/k1baFZ5PsaTYfxcJGwbs7QtuI7RelgsDIjmtrGKew2vbFGQE177XxhgdU bkpQ0aVFkhdhzwcff+LWaITeqGvljAUNPo8L8o/7dFg4yBfq34RKmo2O+m8jrmccwF WyP+xYNxqjE0g== From: alexs@kernel.org To: Vitaly Wool , Miaohe Lin , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, minchan@kernel.org, willy@infradead.org, senozhatsky@chromium.org, david@redhat.com, 42.hyeyoo@gmail.com, Yosry Ahmed , nphamcs@gmail.com Cc: Alex Shi Subject: [PATCH v3 07/20] mm/zsmalloc: convert obj_allocated() and related helpers to use zpdesc Date: Mon, 8 Jul 2024 14:33:28 +0800 Message-ID: <20240708063344.1096626-8-alexs@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240708063344.1096626-1-alexs@kernel.org> References: <20240708063344.1096626-1-alexs@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Hyeonggon Yoo <42.hyeyoo@gmail.com> Convert obj_allocated(), and related helpers to take zpdesc. Also make its callers to cast (struct page *) to (struct zpdesc *) when calling them. The users will be converted gradually as there are many. Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Signed-off-by: Alex Shi (Tencent) --- mm/zsmalloc.c | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index a8f390beeab8..29b9fa5baa46 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -845,15 +845,15 @@ static unsigned long handle_to_obj(unsigned long hand= le) return *(unsigned long *)handle; } =20 -static inline bool obj_allocated(struct page *page, void *obj, +static inline bool obj_allocated(struct zpdesc *zpdesc, void *obj, unsigned long *phandle) { unsigned long handle; - struct zspage *zspage =3D get_zspage(page); + struct zspage *zspage =3D get_zspage(zpdesc_page(zpdesc)); =20 if (unlikely(ZsHugePage(zspage))) { - VM_BUG_ON_PAGE(!is_first_page(page), page); - handle =3D page->index; + VM_BUG_ON_PAGE(!is_first_zpdesc(zpdesc), zpdesc_page(zpdesc)); + handle =3D zpdesc->handle; } else handle =3D *(unsigned long *)obj; =20 @@ -1603,18 +1603,18 @@ static void zs_object_copy(struct size_class *class= , unsigned long dst, * return handle. */ static unsigned long find_alloced_obj(struct size_class *class, - struct page *page, int *obj_idx) + struct zpdesc *zpdesc, int *obj_idx) { unsigned int offset; int index =3D *obj_idx; unsigned long handle =3D 0; - void *addr =3D kmap_atomic(page); + void *addr =3D zpdesc_kmap_atomic(zpdesc); =20 - offset =3D get_first_obj_offset(page); + offset =3D get_first_obj_offset(zpdesc_page(zpdesc)); offset +=3D class->size * index; =20 while (offset < PAGE_SIZE) { - if (obj_allocated(page, addr + offset, &handle)) + if (obj_allocated(zpdesc, addr + offset, &handle)) break; =20 offset +=3D class->size; @@ -1638,7 +1638,7 @@ static void migrate_zspage(struct zs_pool *pool, stru= ct zspage *src_zspage, struct size_class *class =3D pool->size_class[src_zspage->class]; =20 while (1) { - handle =3D find_alloced_obj(class, s_page, &obj_idx); + handle =3D find_alloced_obj(class, page_zpdesc(s_page), &obj_idx); if (!handle) { s_page =3D get_next_page(s_page); if (!s_page) @@ -1871,7 +1871,7 @@ static int zs_page_migrate(struct page *newpage, stru= ct page *page, =20 for (addr =3D s_addr + offset; addr < s_addr + PAGE_SIZE; addr +=3D class->size) { - if (obj_allocated(page, addr, &handle)) { + if (obj_allocated(page_zpdesc(page), addr, &handle)) { =20 old_obj =3D handle_to_obj(handle); obj_to_location(old_obj, &dummy, &obj_idx); --=20 2.43.0 From nobody Sun Feb 8 02:21:45 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9FCBE39AEB for ; Mon, 8 Jul 2024 06:29:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720420151; cv=none; b=Jqb8CxY6oOSVHNUIye5+SBjXqHq1L3LldUrZwP3cRIrTw9o7E5ss5CiqFGKBq5+hamAcaEFrt/HZmfdgbJtD0HK+vwHeiQyoTfINQVBgQBJPVUnQDiycqWqtMYKvNELWH7haO12L/kBQi9NQI6k9tmCk/iQc9uyjN2QdkrtRAIk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720420151; c=relaxed/simple; bh=kTQuh9UlyfPp1whk32zz/TIJv5bm40Slp4CbPrYPxLg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=V+Vt1DXs2eLfk9HSgeWkuNxrh8As2kvk/+h7xhH7qJz48wwPrRvawY/UHouRRdvHbgiO6BTHCpou/sRxhyRxNr4Z9HGSkz2pYObhOBs3pNgERKuK21HzQhqRgMe+oKabIaQH7QHspXO2IupVh98zvNHv2ZYkf3B9aBVuh0ao0lQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Udx6ZKiI; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Udx6ZKiI" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 61E6FC116B1; Mon, 8 Jul 2024 06:29:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1720420151; bh=kTQuh9UlyfPp1whk32zz/TIJv5bm40Slp4CbPrYPxLg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Udx6ZKiI6UdDrWncO84TIziVamqJ5fINWm6oO/wLz3/XChs2OG1+FQ3/2CXFk5bsB +xnt0gGqyIejEqOGH8mFMn3jHKC05UXg68HadyIlWmoZJevSRe1mMUblw0apD2CvLE VVDiiNoSjd3uqsz3OlEEfcUkLSZ9kpfZB1DNcQ5zm1oZJOptjMhdj7m84Q8I/TYLUA Uyghf4PZ+BvaS3mhbU6a5VGy9FfQEJAtExYAJoU6t+X34kjcmzv8Z3mYH4eXeu+65d Qot4m1jNXsFwaryY0Q6swUwKwKZdsvF0NOycLP9VHpe5l9TAuhtbK8/5ogoE++r8jW umZhKDt6S3Gog== From: alexs@kernel.org To: Vitaly Wool , Miaohe Lin , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, minchan@kernel.org, willy@infradead.org, senozhatsky@chromium.org, david@redhat.com, 42.hyeyoo@gmail.com, Yosry Ahmed , nphamcs@gmail.com Cc: Alex Shi Subject: [PATCH v3 08/20] mm/zsmalloc: convert init_zspage() to use zpdesc Date: Mon, 8 Jul 2024 14:33:29 +0800 Message-ID: <20240708063344.1096626-9-alexs@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240708063344.1096626-1-alexs@kernel.org> References: <20240708063344.1096626-1-alexs@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Hyeonggon Yoo <42.hyeyoo@gmail.com> Replace get_first/next_page func series and kmap_atomic to new helper, no functional change. Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Signed-off-by: Alex Shi (Tencent) --- mm/zsmalloc.c | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 29b9fa5baa46..d3558f3f8bc3 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -948,16 +948,16 @@ static void init_zspage(struct size_class *class, str= uct zspage *zspage) { unsigned int freeobj =3D 1; unsigned long off =3D 0; - struct page *page =3D get_first_page(zspage); + struct zpdesc *zpdesc =3D get_first_zpdesc(zspage); =20 - while (page) { - struct page *next_page; + while (zpdesc) { + struct zpdesc *next_zpdesc; struct link_free *link; void *vaddr; =20 - set_first_obj_offset(page, off); + set_first_obj_offset(zpdesc_page(zpdesc), off); =20 - vaddr =3D kmap_atomic(page); + vaddr =3D zpdesc_kmap_atomic(zpdesc); link =3D (struct link_free *)vaddr + off / sizeof(*link); =20 while ((off +=3D class->size) < PAGE_SIZE) { @@ -970,8 +970,8 @@ static void init_zspage(struct size_class *class, struc= t zspage *zspage) * page, which must point to the first object on the next * page (if present) */ - next_page =3D get_next_page(page); - if (next_page) { + next_zpdesc =3D get_next_zpdesc(zpdesc); + if (next_zpdesc) { link->next =3D freeobj++ << OBJ_TAG_BITS; } else { /* @@ -981,7 +981,7 @@ static void init_zspage(struct size_class *class, struc= t zspage *zspage) link->next =3D -1UL << OBJ_TAG_BITS; } kunmap_atomic(vaddr); - page =3D next_page; + zpdesc =3D next_zpdesc; off %=3D PAGE_SIZE; } =20 --=20 2.43.0 From nobody Sun Feb 8 02:21:45 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A56A03BBE8 for ; Mon, 8 Jul 2024 06:29:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720420155; cv=none; b=IZ37p+GrAZ2MaC7xY4E8TbSqGg/jDi4Fpx9Mi5b+zgSK9uc3iB6REx3Y/Wd/fvZ34xndli53sEjjMNGO/bLJMQmCzO8wtGeh0nJ5mbc0F9EsVRPfOZhcmijlCIp/0LrG40oope7GGtwN0cjAu6zEbpwatavsNu0XPmbiQMYAXbM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720420155; c=relaxed/simple; bh=+/6M40wiCxzKTfpFU8y734xJ8A1hop+0lKYvujpVbRg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=a3tx3aRiEcKgB9hbPtdxiL36E4KxwUKMZKzxdEXfBAghcXXHKTxhfcYKrAVheniB68mnAaM1OSaUZFsOaw9FzFqJ5aPjmV/ziH1laV/nvpCFtZO5ZUkuAeWOwouDZnsCoCR4bRAn1gdwwKiL0En0wg5Zh20BM9uOwT7Mc0oyDgY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=bj79AIzd; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="bj79AIzd" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 23D9DC116B1; Mon, 8 Jul 2024 06:29:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1720420155; bh=+/6M40wiCxzKTfpFU8y734xJ8A1hop+0lKYvujpVbRg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=bj79AIzd5Z96XkWJssrQs+SgHiVAhDvkvP7hybFxHBlia0CX41bCz2dfAe1FEvavy ucW6BDXw013iA/lgkUiD5DgFHRea6EFDOX9+8HM2uM9TPPJNuJ1VH1g5XqEj6bfFTw 9mHe15jm4warZWmrhLIn6yMxXXdVaTxcTksSp1MlPWPMoBlnxjIEXt48H/+sYbZVF7 GCslgTjIOKhj83dyvlBFsD3uBjPddg4JuIZZAjASa4DLkpp5j746AIFjUg7QXG474a 9/l1EE3zbQ1sWJVejTgqd9VqAr9xEOKn1aYKN6KBo+pSfT2RLMFmBjoVb9jsRauqSC rznbzr55ZCVBg== From: alexs@kernel.org To: Vitaly Wool , Miaohe Lin , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, minchan@kernel.org, willy@infradead.org, senozhatsky@chromium.org, david@redhat.com, 42.hyeyoo@gmail.com, Yosry Ahmed , nphamcs@gmail.com Cc: Alex Shi Subject: [PATCH v3 09/20] mm/zsmalloc: convert obj_to_page() and zs_free() to use zpdesc Date: Mon, 8 Jul 2024 14:33:30 +0800 Message-ID: <20240708063344.1096626-10-alexs@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240708063344.1096626-1-alexs@kernel.org> References: <20240708063344.1096626-1-alexs@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Hyeonggon Yoo <42.hyeyoo@gmail.com> Rename obj_to_page() to obj_to_zpdesc() and also convert it and its user zs_free() to use zpdesc. Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Signed-off-by: Alex Shi (Tencent) --- mm/zsmalloc.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index d3558f3f8bc3..7aa4a4acaec9 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -820,9 +820,9 @@ static void obj_to_location(unsigned long obj, struct z= pdesc **zpdesc, *obj_idx =3D (obj & OBJ_INDEX_MASK); } =20 -static void obj_to_page(unsigned long obj, struct page **page) +static void obj_to_zpdesc(unsigned long obj, struct zpdesc **zpdesc) { - *page =3D pfn_to_page(obj >> OBJ_INDEX_BITS); + *zpdesc =3D pfn_zpdesc(obj >> OBJ_INDEX_BITS); } =20 /** @@ -1496,7 +1496,7 @@ static void obj_free(int class_size, unsigned long ob= j) void zs_free(struct zs_pool *pool, unsigned long handle) { struct zspage *zspage; - struct page *f_page; + struct zpdesc *f_zpdesc; unsigned long obj; struct size_class *class; int fullness; @@ -1510,8 +1510,8 @@ void zs_free(struct zs_pool *pool, unsigned long hand= le) */ read_lock(&pool->migrate_lock); obj =3D handle_to_obj(handle); - obj_to_page(obj, &f_page); - zspage =3D get_zspage(f_page); + obj_to_zpdesc(obj, &f_zpdesc); + zspage =3D get_zspage(zpdesc_page(f_zpdesc)); class =3D zspage_class(pool, zspage); spin_lock(&class->lock); read_unlock(&pool->migrate_lock); --=20 2.43.0 From nobody Sun Feb 8 02:21:45 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4198943AC0 for ; Mon, 8 Jul 2024 06:29:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720420159; cv=none; b=IY/zmcXuvk+3f9IautOexn3VM6UohzHTDXuGdTds2yYVBn8fPSFLJG2gII0NQThuDCVd/6K8tkmsSfuV2gJJhC94tG9iSSXu3W2hEYCWc6rqWV3AMH9YA0zeZk22kO3++GBECUmB+C5PeZPJZKe+cy09glzcwV+hTfqRQcxD8IA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720420159; c=relaxed/simple; bh=eLEoZgV0ypx4IcCw8JtJdFgQAqXajv6Dsx6/oMszAA4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=AUf6F1noacmUOJfssqrLyEiPgkLHSYct5M+GsSYKsGx+WK9IbSdUa6dsTHiIYPNmy109LvloD90mIScs+cXcRdfkI6s6vz7tyEX2Nh1MQHNTHuTIKjBI3daLXABqyOTCJgQyqJH5nu1JIMakTm7dOxxYHymieheP93yIjZ0OQBI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=hebeQ3k0; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="hebeQ3k0" Received: by smtp.kernel.org (Postfix) with ESMTPSA id D6E61C116B1; Mon, 8 Jul 2024 06:29:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1720420158; bh=eLEoZgV0ypx4IcCw8JtJdFgQAqXajv6Dsx6/oMszAA4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=hebeQ3k0Iltjd3ERyywf5cF9ODSAm8MNeYFZ8QxgekeMq8VG+m6vhC9W32CO4ABH+ VNQ5WsibBUHoCie0KvD+wxpe1WwWSm8Mz8PAfB/bZT2Sf2qGd6xr+Q9NIGJ/m62uwG o3gZug61V9v0A0bPozSxKy3eoUDzZ8U/L9/FYbbPgp2F+wsCcYJJ8qqBYoHl1nEXZY iI5i+1DBM8BbNOgDhV6ZehdXQXE80fsySTLO/uiZ8Sb6krWbmGzBEH6jr98MU30fHI vbfw3bUjscXkgBXIjC5Ar39fojb32Wmqggul321fu6Sr4qllQHUwE+bEWRbsjTo/jz exGZ62Z3odasg== From: alexs@kernel.org To: Vitaly Wool , Miaohe Lin , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, minchan@kernel.org, willy@infradead.org, senozhatsky@chromium.org, david@redhat.com, 42.hyeyoo@gmail.com, Yosry Ahmed , nphamcs@gmail.com Cc: Alex Shi Subject: [PATCH v3 10/20] mm/zsmalloc: add zpdesc_is_isolated/zpdesc_zone helper for zs_page_migrate Date: Mon, 8 Jul 2024 14:33:31 +0800 Message-ID: <20240708063344.1096626-11-alexs@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240708063344.1096626-1-alexs@kernel.org> References: <20240708063344.1096626-1-alexs@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Hyeonggon Yoo <42.hyeyoo@gmail.com> To convert page to zpdesc in zs_page_migrate(), we added zpdesc_is_isolated/zpdesc_zone helpers. No functional change. Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Signed-off-by: Alex Shi (Tencent) --- mm/zpdesc.h | 11 +++++++++++ mm/zsmalloc.c | 30 ++++++++++++++++-------------- 2 files changed, 27 insertions(+), 14 deletions(-) diff --git a/mm/zpdesc.h b/mm/zpdesc.h index 2293453f5d57..ad04c8337cae 100644 --- a/mm/zpdesc.h +++ b/mm/zpdesc.h @@ -108,4 +108,15 @@ static inline void __zpdesc_set_movable(struct zpdesc = *zpdesc, { __SetPageMovable(zpdesc_page(zpdesc), mops); } + +static inline bool zpdesc_is_isolated(struct zpdesc *zpdesc) +{ + return PageIsolated(zpdesc_page(zpdesc)); +} + +static inline struct zone *zpdesc_zone(struct zpdesc *zpdesc) +{ + return page_zone(zpdesc_page(zpdesc)); +} + #endif diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 7aa4a4acaec9..9bc9b14187ed 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -1830,19 +1830,21 @@ static int zs_page_migrate(struct page *newpage, st= ruct page *page, struct size_class *class; struct zspage *zspage; struct zpdesc *dummy; + struct zpdesc *newzpdesc =3D page_zpdesc(newpage); + struct zpdesc *zpdesc =3D page_zpdesc(page); void *s_addr, *d_addr, *addr; unsigned int offset; unsigned long handle; unsigned long old_obj, new_obj; unsigned int obj_idx; =20 - VM_BUG_ON_PAGE(!PageIsolated(page), page); + VM_BUG_ON_PAGE(!zpdesc_is_isolated(zpdesc), zpdesc_page(zpdesc)); =20 /* We're committed, tell the world that this is a Zsmalloc page. */ - __SetPageZsmalloc(newpage); + __SetPageZsmalloc(zpdesc_page(newzpdesc)); =20 /* The page is locked, so this pointer must remain valid */ - zspage =3D get_zspage(page); + zspage =3D get_zspage(zpdesc_page(zpdesc)); pool =3D zspage->pool; =20 /* @@ -1859,30 +1861,30 @@ static int zs_page_migrate(struct page *newpage, st= ruct page *page, /* the migrate_write_lock protects zpage access via zs_map_object */ migrate_write_lock(zspage); =20 - offset =3D get_first_obj_offset(page); - s_addr =3D kmap_atomic(page); + offset =3D get_first_obj_offset(zpdesc_page(zpdesc)); + s_addr =3D zpdesc_kmap_atomic(zpdesc); =20 /* * Here, any user cannot access all objects in the zspage so let's move. */ - d_addr =3D kmap_atomic(newpage); + d_addr =3D zpdesc_kmap_atomic(newzpdesc); copy_page(d_addr, s_addr); kunmap_atomic(d_addr); =20 for (addr =3D s_addr + offset; addr < s_addr + PAGE_SIZE; addr +=3D class->size) { - if (obj_allocated(page_zpdesc(page), addr, &handle)) { + if (obj_allocated(zpdesc, addr, &handle)) { =20 old_obj =3D handle_to_obj(handle); obj_to_location(old_obj, &dummy, &obj_idx); - new_obj =3D (unsigned long)location_to_obj(newpage, + new_obj =3D (unsigned long)location_to_obj(zpdesc_page(newzpdesc), obj_idx); record_obj(handle, new_obj); } } kunmap_atomic(s_addr); =20 - replace_sub_page(class, zspage, page_zpdesc(newpage), page_zpdesc(page)); + replace_sub_page(class, zspage, newzpdesc, zpdesc); /* * Since we complete the data copy and set up new zspage structure, * it's okay to release migration_lock. @@ -1891,14 +1893,14 @@ static int zs_page_migrate(struct page *newpage, st= ruct page *page, spin_unlock(&class->lock); migrate_write_unlock(zspage); =20 - get_page(newpage); - if (page_zone(newpage) !=3D page_zone(page)) { - dec_zone_page_state(page, NR_ZSPAGES); - inc_zone_page_state(newpage, NR_ZSPAGES); + zpdesc_get(newzpdesc); + if (zpdesc_zone(newzpdesc) !=3D zpdesc_zone(zpdesc)) { + zpdesc_dec_zone_page_state(zpdesc); + zpdesc_inc_zone_page_state(newzpdesc); } =20 reset_page(page); - put_page(page); + zpdesc_put(zpdesc); =20 return MIGRATEPAGE_SUCCESS; } --=20 2.43.0 From nobody Sun Feb 8 02:21:45 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A0433482D7 for ; Mon, 8 Jul 2024 06:29:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720420162; cv=none; b=tIAdZuQQrGRgJVuFHotIM75PlHn8+UPhnq2ijve4xTU4XqL+tm6XAsgxgJwPn030f2ymv88QRNFiyb+qUHHqGN7evdJ2g1c0Z0qtLJ9dZAr3WGNoGjWhTV+WkUlJykOIXoK73UG039TZhXXLlOV1DLvase6vlF6j0AIaj2isnaA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720420162; c=relaxed/simple; bh=78ef+c2vUld0iffCBKj3cSVjaJfSet6R8UzdeRHd7pU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=gRjR9vnjGeLxs1Qo+Gv3n/v9pPucAyujIy1Ct0mpZZcWC2SpEMG664cApRylkzX5iisyLSqaflZlYsPkFW9KpKHpZ4QhfTsTx/mn+H8zgvVp+BRU2bWs05SFRYNfZYt7O1UIKcW454/K29XU1RKR5XHhn0mU/UiyaCJvnQlLeAA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=HA0Ua3KN; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="HA0Ua3KN" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 70850C116B1; Mon, 8 Jul 2024 06:29:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1720420162; bh=78ef+c2vUld0iffCBKj3cSVjaJfSet6R8UzdeRHd7pU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=HA0Ua3KNiyLRSlvuk8iE+wvrkOCsWZzp0NhLUjf5s2/M8n++cHNi8RMTSGlC5wHEi ga/OjuScMR/XI5u8/qk7ckMACpo/8r2jtib0KRymUrBa73A5JOs5PbPHtMiENWQkNy QOu4Lx0gn6RJy2fFwwquEray1BrfIRfvhthbzt8ZR6j20/xDY9vX2VTSKs0+G2bfTX 7nl5N/ADiWZtJ0N82eBiTBBzWhTrr+C3DTl+vxqFhB5GqjpdZXcYkXFZ4x+4kdLCiZ 8YaAWhVw7ch+c7m+UhwS5zWToaQnWLCaCPvCPKJtR6CF5CUyoNhQpwoJbIQfLstmLI /YL5eKNTw0BKQ== From: alexs@kernel.org To: Vitaly Wool , Miaohe Lin , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, minchan@kernel.org, willy@infradead.org, senozhatsky@chromium.org, david@redhat.com, 42.hyeyoo@gmail.com, Yosry Ahmed , nphamcs@gmail.com Cc: Alex Shi Subject: [PATCH v3 11/20] mm/zsmalloc: rename reset_page to reset_zpdesc and use zpdesc in it Date: Mon, 8 Jul 2024 14:33:32 +0800 Message-ID: <20240708063344.1096626-12-alexs@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240708063344.1096626-1-alexs@kernel.org> References: <20240708063344.1096626-1-alexs@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Alex Shi (Tencent) zpdesc.zspage matches with page.private, zpdesc.next matches with page.index. They will be reset in reset_page() wich is called prior to free base pages of a zspage. Use zpdesc to replace page struct and rename it to reset_zpdesc(), few page helper still left since they are used too widely. Signed-off-by: Alex Shi (Tencent) --- mm/zsmalloc.c | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 9bc9b14187ed..6d1971836391 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -865,12 +865,14 @@ static inline bool obj_allocated(struct zpdesc *zpdes= c, void *obj, return true; } =20 -static void reset_page(struct page *page) +static void reset_zpdesc(struct zpdesc *zpdesc) { + struct page *page =3D zpdesc_page(zpdesc); + __ClearPageMovable(page); ClearPagePrivate(page); - set_page_private(page, 0); - page->index =3D 0; + zpdesc->zspage =3D NULL; + zpdesc->next =3D NULL; reset_first_obj_offset(page); __ClearPageZsmalloc(page); } @@ -910,7 +912,7 @@ static void __free_zspage(struct zs_pool *pool, struct = size_class *class, do { VM_BUG_ON_PAGE(!PageLocked(page), page); next =3D get_next_page(page); - reset_page(page); + reset_zpdesc(page_zpdesc(page)); unlock_page(page); dec_zone_page_state(page, NR_ZSPAGES); put_page(page); @@ -1899,7 +1901,7 @@ static int zs_page_migrate(struct page *newpage, stru= ct page *page, zpdesc_inc_zone_page_state(newzpdesc); } =20 - reset_page(page); + reset_zpdesc(zpdesc); zpdesc_put(zpdesc); =20 return MIGRATEPAGE_SUCCESS; --=20 2.43.0 From nobody Sun Feb 8 02:21:45 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6DAE254720 for ; Mon, 8 Jul 2024 06:29:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720420166; cv=none; b=ft+cFirOf6LU6bNBxdjSqWjbcq109+AeqMeL1can+D/G+7R/DIiZfXB3qXS1te48AIvirknGRSsHg5Fx3+6SKgJ31r5z3DNQv4XGPVrvVEErCis0O5FGG/gkS24Mh5kLoL1JUrhLE28oICyADNNSUoez16fuC/JC57mQS/ThfKQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720420166; c=relaxed/simple; bh=F6gyGa/Nhcaa49tqL1AcCvvFaZR6EdIliZgtM0OjhK4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=cD6e8OoD0wb/4FvQUXUFSSzfYxwMJYRh0o+lUpDzp8YSbgbgrRUXK9KIPPSfWFfn7mD6SD9L/pEuJdArSjSpx4BLG3lJI6BZmcbyKayomboeNIKHjngSwkgn0UGCG4bzKbMcXxOY9ZG3dMAGxsCZ1OtULVtVNneS7mn0gNbOgiw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=RiUIttnE; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="RiUIttnE" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 2D450C4AF0B; Mon, 8 Jul 2024 06:29:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1720420166; bh=F6gyGa/Nhcaa49tqL1AcCvvFaZR6EdIliZgtM0OjhK4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=RiUIttnE915B5POlt/rGkjRDxwMkMUiIsyhchqZid2hdw6qQfozwkJ9cRUtOfaZWY ecSVJ4QQb/fdFBE9zfzyTA2BZQ/RZsYI6z/9adNmf/+XhmJydHF+/B49U4BOPli6FZ i+H2xFep0Wh7TcTsawSQ0/5F+Km/FzU4iOKH5y+BJhwbFrL+YhRlq7VBl8TY9/DR6+ 9B9+DFLrjkjS9AcDH5+JUEVesyTatZ396Mq4zYeXPyscrToCEr2YgnPtcKHsDEslgD MVG6bEWLnGK/BxKoGIZwYSo/4qLqon08Zeykg32nXyrja7dsffK90F6QWlZSv8eCUu qhdiNIl+Xz0Ug== From: alexs@kernel.org To: Vitaly Wool , Miaohe Lin , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, minchan@kernel.org, willy@infradead.org, senozhatsky@chromium.org, david@redhat.com, 42.hyeyoo@gmail.com, Yosry Ahmed , nphamcs@gmail.com Cc: Alex Shi Subject: [PATCH v3 12/20] mm/zsmalloc: convert __free_zspage() to use zdsesc Date: Mon, 8 Jul 2024 14:33:33 +0800 Message-ID: <20240708063344.1096626-13-alexs@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240708063344.1096626-1-alexs@kernel.org> References: <20240708063344.1096626-1-alexs@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Hyeonggon Yoo <42.hyeyoo@gmail.com> Introduce zpdesc_is_locked() and convert __free_zspage() to use zpdesc. Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Signed-off-by: Alex Shi (Tencent) --- mm/zpdesc.h | 4 ++++ mm/zsmalloc.c | 20 ++++++++++---------- 2 files changed, 14 insertions(+), 10 deletions(-) diff --git a/mm/zpdesc.h b/mm/zpdesc.h index ad04c8337cae..72c8c072b4c8 100644 --- a/mm/zpdesc.h +++ b/mm/zpdesc.h @@ -119,4 +119,8 @@ static inline struct zone *zpdesc_zone(struct zpdesc *z= pdesc) return page_zone(zpdesc_page(zpdesc)); } =20 +static inline bool zpdesc_is_locked(struct zpdesc *zpdesc) +{ + return PageLocked(zpdesc_page(zpdesc)); +} #endif diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 6d1971836391..68fdea7b6e0d 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -901,23 +901,23 @@ static int trylock_zspage(struct zspage *zspage) static void __free_zspage(struct zs_pool *pool, struct size_class *class, struct zspage *zspage) { - struct page *page, *next; + struct zpdesc *zpdesc, *next; =20 assert_spin_locked(&class->lock); =20 VM_BUG_ON(get_zspage_inuse(zspage)); VM_BUG_ON(zspage->fullness !=3D ZS_INUSE_RATIO_0); =20 - next =3D page =3D get_first_page(zspage); + next =3D zpdesc =3D get_first_zpdesc(zspage); do { - VM_BUG_ON_PAGE(!PageLocked(page), page); - next =3D get_next_page(page); - reset_zpdesc(page_zpdesc(page)); - unlock_page(page); - dec_zone_page_state(page, NR_ZSPAGES); - put_page(page); - page =3D next; - } while (page !=3D NULL); + VM_BUG_ON_PAGE(!zpdesc_is_locked(zpdesc), zpdesc_page(zpdesc)); + next =3D get_next_zpdesc(zpdesc); + reset_zpdesc(zpdesc); + zpdesc_unlock(zpdesc); + zpdesc_dec_zone_page_state(zpdesc); + zpdesc_put(zpdesc); + zpdesc =3D next; + } while (zpdesc !=3D NULL); =20 cache_free_zspage(pool, zspage); =20 --=20 2.43.0 From nobody Sun Feb 8 02:21:45 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1D11A6E2AE for ; Mon, 8 Jul 2024 06:29:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720420170; cv=none; b=Hn7CcRKo1L+uwjXuKM5sRn3LEC3BjY4kV7gfZLvlQTmh2HD7Evqi1fiN5uYhotD4/h9gwFVOhGPL0Wa2vfkGJk6ZZrItg4i0ZaeqyP5no//UdIuqNtm0pPhNVD0ZR4rmhnlcplDSxGiayonNuoFgKcB9KFhJuOy4HS+FHMYPic4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720420170; c=relaxed/simple; bh=FoPFFhZjgJ7k8SPuHkOnh4ZOuMsrgriGi5An3x0tRys=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=lS9PdtJVyKm2VDScARwAGuZm0hTm3h8BKDJ3SPDCmhKF5u0W4kMBkPHEeBTSDd4WoymvJC9WdyT+YUccEa7toqdskBMOg3E4YGEjfxBWlhZO9LnzSRnoay0wHiyrli8Y1OWZ0BAUYWRZf4XKL3TNuAaverbqaQ2efAGm/i0jSx0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=SVk5iNSo; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="SVk5iNSo" Received: by smtp.kernel.org (Postfix) with ESMTPSA id E1C84C116B1; Mon, 8 Jul 2024 06:29:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1720420170; bh=FoPFFhZjgJ7k8SPuHkOnh4ZOuMsrgriGi5An3x0tRys=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=SVk5iNSoWEypU6s06xUAyZFEaiQ1Lg9SaOx3SHhS8HIM3gOa6jnzHjXuVpb7C+u+G CUS6gTMu1aJOHfYMh1aiodRaOueC1INYKRIwcbJuEHMDRzbhoYUUdn+dw/UhRWFQPb YOqyBMdjtfNuVZCvBM094GknA+ub5pPohmKxG+mvZ+Nm6SKp7IV/SH+zPDLRreMD2G sucCviNkh1ywa/cuflg9Ocj6Jy66eePgmkIiHVrIOVmO5I4t6DiBpMje8pB+6w7hz4 FOMaRLVBR7AfGWwpT4+oLOQkdenToTM2w5UgRWJw1jw7MRLjGW75nLcW44/UPvFNgM KHaM1Hk6JxHRw== From: alexs@kernel.org To: Vitaly Wool , Miaohe Lin , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, minchan@kernel.org, willy@infradead.org, senozhatsky@chromium.org, david@redhat.com, 42.hyeyoo@gmail.com, Yosry Ahmed , nphamcs@gmail.com Cc: Alex Shi Subject: [PATCH v3 13/20] mm/zsmalloc: convert location_to_obj() to take zpdesc Date: Mon, 8 Jul 2024 14:33:34 +0800 Message-ID: <20240708063344.1096626-14-alexs@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240708063344.1096626-1-alexs@kernel.org> References: <20240708063344.1096626-1-alexs@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Hyeonggon Yoo <42.hyeyoo@gmail.com> As all users of location_to_obj() now use zpdesc, convert location_to_obj() to take zpdesc. Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Signed-off-by: Alex Shi (Tencent) --- mm/zsmalloc.c | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 68fdea7b6e0d..e291c7319485 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -826,15 +826,15 @@ static void obj_to_zpdesc(unsigned long obj, struct z= pdesc **zpdesc) } =20 /** - * location_to_obj - get obj value encoded from (, ) - * @page: page object resides in zspage + * location_to_obj - get obj value encoded from (, ) + * @zpdesc: zpdesc object resides in zspage * @obj_idx: object index */ -static unsigned long location_to_obj(struct page *page, unsigned int obj_i= dx) +static unsigned long location_to_obj(struct zpdesc *zpdesc, unsigned int o= bj_idx) { unsigned long obj; =20 - obj =3D page_to_pfn(page) << OBJ_INDEX_BITS; + obj =3D zpdesc_pfn(zpdesc) << OBJ_INDEX_BITS; obj |=3D obj_idx & OBJ_INDEX_MASK; =20 return obj; @@ -1392,7 +1392,7 @@ static unsigned long obj_malloc(struct zs_pool *pool, kunmap_atomic(vaddr); mod_zspage_inuse(zspage, 1); =20 - obj =3D location_to_obj(zpdesc_page(m_zpdesc), obj); + obj =3D location_to_obj(m_zpdesc, obj); record_obj(handle, obj); =20 return obj; @@ -1879,8 +1879,7 @@ static int zs_page_migrate(struct page *newpage, stru= ct page *page, =20 old_obj =3D handle_to_obj(handle); obj_to_location(old_obj, &dummy, &obj_idx); - new_obj =3D (unsigned long)location_to_obj(zpdesc_page(newzpdesc), - obj_idx); + new_obj =3D (unsigned long)location_to_obj(newzpdesc, obj_idx); record_obj(handle, new_obj); } } --=20 2.43.0 From nobody Sun Feb 8 02:21:45 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 01FB21CA96 for ; Mon, 8 Jul 2024 06:29:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720420174; cv=none; b=QeRvfY6N2qSjipcH5d2eR9OzzLvcZyhiUo2Os72C9rELEyjLQRrVkRtrAYc0NuzO4UK2RrJp6rmD8OPZelhM3Y/x3Zv1E/Vvsk/sqZ/v39mAoRss1mr1nrxLYOS8i4WXQi9jC/5dlhOUMYMZv3V0Vparj0cytAaFpEzZTUxptb8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720420174; c=relaxed/simple; bh=cbT48olUQwlsCFwwUYnjfHlBbFy2rEQMOoWsy01+W3Y=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=rsfplf4Vh889/hUM/eAC7JbAQBfKmtMgz8X2LBNU+yNim7iZ9NVg5x+EpRhCG8l6qaUJw99Kp1jZ1CkoivqpJn6RkfXncTkUhnG7/eWHflPlnbwi1MDdJN/UZiO5UjLY9/h6CQ57xvR6tT0HIufQiWwhe+U4Qt3XvQwAnj+d+GQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=UJRGhy3n; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="UJRGhy3n" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9F7BAC4AF0B; Mon, 8 Jul 2024 06:29:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1720420173; bh=cbT48olUQwlsCFwwUYnjfHlBbFy2rEQMOoWsy01+W3Y=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=UJRGhy3np6draIrQ5dcJhaA8p3BR4ME3fuTKdWducweIJsmTBd+uSKYPdZll3y7rm BxNOBa57Zm1zxbKwTabeuj6D09i+Zx1Uqgc0/aMswDBCkx3BTVd/V86FadDBFg1jSX BwKjViQcGzHORnrteaoEWLQOTQNPKxr/BeqB/dbTV5U0VQTG/0EXsIWSJ2BmctSbcS 2viqi+gbSiBA8ugd6tbWwy5+9acn1fm262K3J2Ks5WpmB6jzbUYQUrPsTs9RMuSHtF oCsf2LjiyQPjkrP1tVBJhIa6h36V1RZwq/GgYIqENIYg8Da1WzohE7MWLj8O9AMB8z hNL8CYaHOa2kw== From: alexs@kernel.org To: Vitaly Wool , Miaohe Lin , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, minchan@kernel.org, willy@infradead.org, senozhatsky@chromium.org, david@redhat.com, 42.hyeyoo@gmail.com, Yosry Ahmed , nphamcs@gmail.com Cc: Alex Shi Subject: [PATCH v3 14/20] mm/zsmalloc: convert migrate_zspage() to use zpdesc Date: Mon, 8 Jul 2024 14:33:35 +0800 Message-ID: <20240708063344.1096626-15-alexs@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240708063344.1096626-1-alexs@kernel.org> References: <20240708063344.1096626-1-alexs@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Hyeonggon Yoo <42.hyeyoo@gmail.com> Use get_first_zpdesc/get_next_zpdesc to replace get_first/next_page. No functional change. Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Signed-off-by: Alex Shi (Tencent) --- mm/zsmalloc.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index e291c7319485..7d039b0c66db 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -1636,14 +1636,14 @@ static void migrate_zspage(struct zs_pool *pool, st= ruct zspage *src_zspage, unsigned long used_obj, free_obj; unsigned long handle; int obj_idx =3D 0; - struct page *s_page =3D get_first_page(src_zspage); + struct zpdesc *s_zpdesc =3D get_first_zpdesc(src_zspage); struct size_class *class =3D pool->size_class[src_zspage->class]; =20 while (1) { - handle =3D find_alloced_obj(class, page_zpdesc(s_page), &obj_idx); + handle =3D find_alloced_obj(class, s_zpdesc, &obj_idx); if (!handle) { - s_page =3D get_next_page(s_page); - if (!s_page) + s_zpdesc =3D get_next_zpdesc(s_zpdesc); + if (!s_zpdesc) break; obj_idx =3D 0; continue; --=20 2.43.0 From nobody Sun Feb 8 02:21:45 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B761B20B33 for ; Mon, 8 Jul 2024 06:29:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720420177; cv=none; b=MlvvIXJX3b2KGWumbbo308q/M3w8VUN3yn21E89S49mnNIpPhc9r6bs3/DRur3Cv91GBKouef7O9y9vj6fVCEvFggZbxXzvWEDTX32qV9WueBGRf/9fr7e/eCPa0Xe9Ov7b2EpFK0zWR3RJ1u0MVbdNnKFfkHDlKCDtnpUPz5to= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720420177; c=relaxed/simple; bh=7eF5F3tLj5M84TMn26AMq8BpyTBKqnCtBSnMwvbth1I=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=kstACSrpsJ3lPJFRMFUgX41t7CPvaPtR1I54EP2Xyke8SPxRQbnrqjSNVzrlPwai+tGjIIclp4beynRMmhh44JQ0DkENdKCIKyoaRb2jU26rEEj/bDFWPPoXtV02nUG3tSGZ2xJh3Dkoil1riFh7yRKLmWM+XFd2FWhq/9dT5y8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=YL80aVHo; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="YL80aVHo" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 365C6C116B1; Mon, 8 Jul 2024 06:29:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1720420177; bh=7eF5F3tLj5M84TMn26AMq8BpyTBKqnCtBSnMwvbth1I=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=YL80aVHoDjsPpQiEFbQTkgxQsZ584pywl8w3bPOSXFSqwYmZXgdEwogm6GaWDr7Jz IL40gwAKS4R5HieLcR2aixdW1ZSNWXe2OK3CXRIKSoqf6zPhWdbVbWekFTy2yUNuAM qPrnz32NbPsJ3fKvenqrjhmV08ZQio/iPNunbTc3ItLjwdwL7FM7J9vSZcOQxmzurn 6tByRNjMwTTOFDhCdCinryOYpSJDfCLxfgx27Q4A75DaxcaCPsMDUkvPp76JmkCXIH 1/sC7c74gsr+OeCq7Vh8ij43eR84+rzJcYe23EQgfrfR8uwEZSnKwA2xlTFVEXYTcO ebeTPLAH4yTSw== From: alexs@kernel.org To: Vitaly Wool , Miaohe Lin , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, minchan@kernel.org, willy@infradead.org, senozhatsky@chromium.org, david@redhat.com, 42.hyeyoo@gmail.com, Yosry Ahmed , nphamcs@gmail.com Cc: Alex Shi Subject: [PATCH v3 15/20] mm/zsmalloc: convert get_zspage() to take zpdesc Date: Mon, 8 Jul 2024 14:33:36 +0800 Message-ID: <20240708063344.1096626-16-alexs@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240708063344.1096626-1-alexs@kernel.org> References: <20240708063344.1096626-1-alexs@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Hyeonggon Yoo <42.hyeyoo@gmail.com> Now that all users except get_next_page() (which will be removed in later patch) use zpdesc, convert get_zspage() to take zpdesc instead of page. Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Signed-off-by: Alex Shi (Tencent) --- mm/zsmalloc.c | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 7d039b0c66db..458ad696b473 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -779,9 +779,9 @@ static int fix_fullness_group(struct size_class *class,= struct zspage *zspage) return newfg; } =20 -static struct zspage *get_zspage(struct page *page) +static struct zspage *get_zspage(struct zpdesc *zpdesc) { - struct zspage *zspage =3D (struct zspage *)page_private(page); + struct zspage *zspage =3D zpdesc->zspage; =20 BUG_ON(zspage->magic !=3D ZSPAGE_MAGIC); return zspage; @@ -789,7 +789,7 @@ static struct zspage *get_zspage(struct page *page) =20 static struct page *get_next_page(struct page *page) { - struct zspage *zspage =3D get_zspage(page); + struct zspage *zspage =3D get_zspage(page_zpdesc(page)); =20 if (unlikely(ZsHugePage(zspage))) return NULL; @@ -799,7 +799,7 @@ static struct page *get_next_page(struct page *page) =20 static struct zpdesc *get_next_zpdesc(struct zpdesc *zpdesc) { - struct zspage *zspage =3D get_zspage(zpdesc_page(zpdesc)); + struct zspage *zspage =3D get_zspage(zpdesc); =20 if (unlikely(ZsHugePage(zspage))) return NULL; @@ -849,7 +849,7 @@ static inline bool obj_allocated(struct zpdesc *zpdesc,= void *obj, unsigned long *phandle) { unsigned long handle; - struct zspage *zspage =3D get_zspage(zpdesc_page(zpdesc)); + struct zspage *zspage =3D get_zspage(zpdesc); =20 if (unlikely(ZsHugePage(zspage))) { VM_BUG_ON_PAGE(!is_first_zpdesc(zpdesc), zpdesc_page(zpdesc)); @@ -1265,7 +1265,7 @@ void *zs_map_object(struct zs_pool *pool, unsigned lo= ng handle, read_lock(&pool->migrate_lock); obj =3D handle_to_obj(handle); obj_to_location(obj, &zpdesc, &obj_idx); - zspage =3D get_zspage(zpdesc_page(zpdesc)); + zspage =3D get_zspage(zpdesc); =20 /* * migration cannot move any zpages in this zspage. Here, class->lock @@ -1315,7 +1315,7 @@ void zs_unmap_object(struct zs_pool *pool, unsigned l= ong handle) =20 obj =3D handle_to_obj(handle); obj_to_location(obj, &zpdesc, &obj_idx); - zspage =3D get_zspage(zpdesc_page(zpdesc)); + zspage =3D get_zspage(zpdesc); class =3D zspage_class(pool, zspage); off =3D offset_in_page(class->size * obj_idx); =20 @@ -1479,7 +1479,7 @@ static void obj_free(int class_size, unsigned long ob= j) =20 obj_to_location(obj, &f_zpdesc, &f_objidx); f_offset =3D offset_in_page(class_size * f_objidx); - zspage =3D get_zspage(zpdesc_page(f_zpdesc)); + zspage =3D get_zspage(f_zpdesc); =20 vaddr =3D zpdesc_kmap_atomic(f_zpdesc); link =3D (struct link_free *)(vaddr + f_offset); @@ -1513,7 +1513,7 @@ void zs_free(struct zs_pool *pool, unsigned long hand= le) read_lock(&pool->migrate_lock); obj =3D handle_to_obj(handle); obj_to_zpdesc(obj, &f_zpdesc); - zspage =3D get_zspage(zpdesc_page(f_zpdesc)); + zspage =3D get_zspage(f_zpdesc); class =3D zspage_class(pool, zspage); spin_lock(&class->lock); read_unlock(&pool->migrate_lock); @@ -1846,7 +1846,7 @@ static int zs_page_migrate(struct page *newpage, stru= ct page *page, __SetPageZsmalloc(zpdesc_page(newzpdesc)); =20 /* The page is locked, so this pointer must remain valid */ - zspage =3D get_zspage(zpdesc_page(zpdesc)); + zspage =3D get_zspage(zpdesc); pool =3D zspage->pool; =20 /* --=20 2.43.0 From nobody Sun Feb 8 02:21:45 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7AA1E3BBE9 for ; Mon, 8 Jul 2024 06:29:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720420181; cv=none; b=rhXxZstmKQRKgeaYT3Q5y7uAv2DJV86TlUBDVv6E+PQVFUMo7kA+c9iIo/KyJl2iruakuzTegZfdKJtXY4upfBxVntsZb9WtGVDehmfEjNqxDpr/DESz8u8Yn28cQJ89Wwc/d/f3aO69YIwleUH4ivDkMJ+VzM0hmSJyyB3U9Js= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720420181; c=relaxed/simple; bh=TaGzviLe6NGcMCDjFOvP7sDnEJpuSU1XTpwZAcepYDo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=EhUdRb9Qnhz07oDYH1eh0NU1juT38P6ff53mp0ac3yJvWFJy9H+mUy2/iuzQC1pLPbWBu2al3bCUWO839hqt6DpIe5jChne2zCSk4dscwo9dMtOOlMkA4avShHhqumhXER+jvPGqHj/xC9i1oyCZW1KPqP/BykMCYHYdsSQFpOo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=PLZIVCAc; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="PLZIVCAc" Received: by smtp.kernel.org (Postfix) with ESMTPSA id E712BC4AF0C; Mon, 8 Jul 2024 06:29:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1720420181; bh=TaGzviLe6NGcMCDjFOvP7sDnEJpuSU1XTpwZAcepYDo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=PLZIVCAcwi0LP6n9+0RZBSXSigP2dfRTuOADXKXuWbgBCN0En57eYC5RSgs/1KtfS DVIzAun5YHyUuQoBUGzwuDYGBBdNZwNvpM+rRFJ6VWSjPUG5CpnTMp5A1yDpyc7vD2 WP+uhvZep47BARC5c9jpTlVqf0y7Kz7D+Yq3L4MDQ86P1RuxQXIXTqAIQcgXDvnzwg UtAPWYZA9ek8HVwniRE5UU8dKYGfnTrePnKl8DCQDiV1Udgp7vk+n46IaoL/PkmIeF Z5YaezCR7ig33vTKcfrPcYHW37YL2UrMeTFdY3VPVBc3XwPwXe7fFLu9rM4tcZy960 NbRxZc64HmFyA== From: alexs@kernel.org To: Vitaly Wool , Miaohe Lin , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, minchan@kernel.org, willy@infradead.org, senozhatsky@chromium.org, david@redhat.com, 42.hyeyoo@gmail.com, Yosry Ahmed , nphamcs@gmail.com Cc: Alex Shi Subject: [PATCH v3 16/20] mm/zsmalloc: convert SetZsPageMovable and remove unused funcs Date: Mon, 8 Jul 2024 14:33:37 +0800 Message-ID: <20240708063344.1096626-17-alexs@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240708063344.1096626-1-alexs@kernel.org> References: <20240708063344.1096626-1-alexs@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Alex Shi (Tencent) Convert SetZsPageMovable() to use zpdesc, and then remove unused funcs:get_next_page/get_first_page/is_first_page. Originally-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Signed-off-by: Alex Shi (Tencent) --- mm/zsmalloc.c | 33 +++++---------------------------- 1 file changed, 5 insertions(+), 28 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 458ad696b473..8b713ac03902 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -468,11 +468,6 @@ static DEFINE_PER_CPU(struct mapping_area, zs_map_area= ) =3D { .lock =3D INIT_LOCAL_LOCK(lock), }; =20 -static __maybe_unused int is_first_page(struct page *page) -{ - return PagePrivate(page); -} - static int is_first_zpdesc(struct zpdesc *zpdesc) { return PagePrivate(zpdesc_page(zpdesc)); @@ -489,14 +484,6 @@ static inline void mod_zspage_inuse(struct zspage *zsp= age, int val) zspage->inuse +=3D val; } =20 -static inline struct page *get_first_page(struct zspage *zspage) -{ - struct page *first_page =3D zpdesc_page(zspage->first_zpdesc); - - VM_BUG_ON_PAGE(!is_first_page(first_page), first_page); - return first_page; -} - static struct zpdesc *get_first_zpdesc(struct zspage *zspage) { struct zpdesc *first_zpdesc =3D zspage->first_zpdesc; @@ -787,16 +774,6 @@ static struct zspage *get_zspage(struct zpdesc *zpdesc) return zspage; } =20 -static struct page *get_next_page(struct page *page) -{ - struct zspage *zspage =3D get_zspage(page_zpdesc(page)); - - if (unlikely(ZsHugePage(zspage))) - return NULL; - - return (struct page *)page->index; -} - static struct zpdesc *get_next_zpdesc(struct zpdesc *zpdesc) { struct zspage *zspage =3D get_zspage(zpdesc); @@ -1970,13 +1947,13 @@ static void init_deferred_free(struct zs_pool *pool) =20 static void SetZsPageMovable(struct zs_pool *pool, struct zspage *zspage) { - struct page *page =3D get_first_page(zspage); + struct zpdesc *zpdesc =3D get_first_zpdesc(zspage); =20 do { - WARN_ON(!trylock_page(page)); - __SetPageMovable(page, &zsmalloc_mops); - unlock_page(page); - } while ((page =3D get_next_page(page)) !=3D NULL); + WARN_ON(!zpdesc_trylock(zpdesc)); + __zpdesc_set_movable(zpdesc, &zsmalloc_mops); + zpdesc_unlock(zpdesc); + } while ((zpdesc =3D get_next_zpdesc(zpdesc)) !=3D NULL); } #else static inline void zs_flush_migration(struct zs_pool *pool) { } --=20 2.43.0 From nobody Sun Feb 8 02:21:45 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 31C3017577 for ; Mon, 8 Jul 2024 06:29:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720420185; cv=none; b=OTFcr5BWI0xuQ7Nlpbn5EbbZG3pevipT+IEB4fqNd0T8XH/9+/pn3qEXoAtJbDqNWMKI0swX/6FF0mEjR96iWMRQee+wo3Gha6HbVuNBgQq/AH7el1278ne1/r+rs23cx+ylsRXHjjtDacQeMQ9654KAdsokJbe2gJbxrhbqfjM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720420185; c=relaxed/simple; bh=Y64RHT8UUVO7Yuad89btOx9mw5qKYY1Q7Nr/pGMgLzg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=sa/2mVoAkYRMfvw6WJrTVJfr0SPQ1K3y8SpZnvsPvhyWcxsewAEFs8jeyvuaF2WpCc0Au7UWfsnTshOj0/ZN3mIYrzK0AtvVus8iWtPNgwDKYkJbNSaSKlYJLk/HExQ8zvBO1KimqGUuBDq6Yd9jaQW0Zf+0yVoJ0PeGMKtRLrU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=H3TVIOi8; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="H3TVIOi8" Received: by smtp.kernel.org (Postfix) with ESMTPSA id A7497C116B1; Mon, 8 Jul 2024 06:29:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1720420184; bh=Y64RHT8UUVO7Yuad89btOx9mw5qKYY1Q7Nr/pGMgLzg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=H3TVIOi8tn+/Jpf1JSx2+yuyzZOtImlkUm5t0LnzYklpni4kdSFiNHrqpQOXZqOL6 eKTBzQL6JVvFtsElAu7G5prq398JdgHze+6UeKvCmxHlMaC6BpNtTzo0tlkPZi5e6u DXlNgwNKMHm0UQBzrcZX6BPaN3TTk5eMyjGw7qRf9HJC8/bqtDsza7aNuxJB5E98Kh cVBOvKC38kiKD4gMr4YS7nAAFe17otobJQn5FNGbjOzVh8XnFWdWFnTy2QLvYSiqu7 +b4+8NbWTwh9n/uuXoC10fhHNQkguOvxg8q07JSihLZaF7N3nVxyDqDLJTP6G62bBx ROe9sU1M31ITQ== From: alexs@kernel.org To: Vitaly Wool , Miaohe Lin , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, minchan@kernel.org, willy@infradead.org, senozhatsky@chromium.org, david@redhat.com, 42.hyeyoo@gmail.com, Yosry Ahmed , nphamcs@gmail.com Cc: Alex Shi Subject: [PATCH v3 17/20] mm/zsmalloc: convert get/set_first_obj_offset() to take zpdesc Date: Mon, 8 Jul 2024 14:33:38 +0800 Message-ID: <20240708063344.1096626-18-alexs@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240708063344.1096626-1-alexs@kernel.org> References: <20240708063344.1096626-1-alexs@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Alex Shi (Tencent) Now that all users of get/set_first_obj_offset() are converted to use zpdesc, convert them to take zpdesc. Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Signed-off-by: Alex Shi (Tencent) --- mm/zpdesc.h | 7 ++++++- mm/zsmalloc.c | 36 ++++++++++++++++++------------------ 2 files changed, 24 insertions(+), 19 deletions(-) diff --git a/mm/zpdesc.h b/mm/zpdesc.h index 72c8c072b4c8..f64e813f4847 100644 --- a/mm/zpdesc.h +++ b/mm/zpdesc.h @@ -15,6 +15,8 @@ * @next: Next zpdesc in a zspage in zsmalloc zpool * @handle: For huge zspage in zsmalloc zpool * @zspage: Pointer to zspage in zsmalloc + * @first_obj_offset: First object offset in zsmalloc zpool + * @_refcount: Indirectly use by page migration * @memcg_data: Memory Control Group data. * * This struct overlays struct page for now. Do not modify without a good @@ -31,7 +33,8 @@ struct zpdesc { unsigned long handle; }; struct zspage *zspage; - unsigned long _zp_pad_1; + unsigned int first_obj_offset; + atomic_t _refcount; #ifdef CONFIG_MEMCG unsigned long memcg_data; #endif @@ -45,6 +48,8 @@ ZPDESC_MATCH(mapping, mops); ZPDESC_MATCH(index, next); ZPDESC_MATCH(index, handle); ZPDESC_MATCH(private, zspage); +ZPDESC_MATCH(page_type, first_obj_offset); +ZPDESC_MATCH(_refcount, _refcount); #ifdef CONFIG_MEMCG ZPDESC_MATCH(memcg_data, memcg_data); #endif diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 8b713ac03902..bb8b5f13a966 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -20,8 +20,8 @@ * zpdesc->next: links together all component pages of a zspage * For the huge page, this is always 0, so we use this field * to store handle. - * page->page_type: PG_zsmalloc, lower 16 bit locate the first object - * offset in a subpage of a zspage + * zpdesc->first_obj_offset: PG_zsmalloc, lower 16 bit locate the first + * object offset in a subpage of a zspage * * Usage of struct zpdesc(page) flags: * PG_private: identifies the first component page @@ -494,26 +494,26 @@ static struct zpdesc *get_first_zpdesc(struct zspage = *zspage) =20 #define FIRST_OBJ_PAGE_TYPE_MASK 0xffff =20 -static inline void reset_first_obj_offset(struct page *page) +static inline void reset_first_obj_offset(struct zpdesc *zpdesc) { - VM_WARN_ON_ONCE(!PageZsmalloc(page)); - page->page_type |=3D FIRST_OBJ_PAGE_TYPE_MASK; + VM_WARN_ON_ONCE(!PageZsmalloc(zpdesc_page(zpdesc))); + zpdesc->first_obj_offset |=3D FIRST_OBJ_PAGE_TYPE_MASK; } =20 -static inline unsigned int get_first_obj_offset(struct page *page) +static inline unsigned int get_first_obj_offset(struct zpdesc *zpdesc) { - VM_WARN_ON_ONCE(!PageZsmalloc(page)); - return page->page_type & FIRST_OBJ_PAGE_TYPE_MASK; + VM_WARN_ON_ONCE(!PageZsmalloc(zpdesc_page(zpdesc))); + return zpdesc->first_obj_offset & FIRST_OBJ_PAGE_TYPE_MASK; } =20 -static inline void set_first_obj_offset(struct page *page, unsigned int of= fset) +static inline void set_first_obj_offset(struct zpdesc *zpdesc, unsigned in= t offset) { /* With 16 bit available, we can support offsets into 64 KiB pages. */ BUILD_BUG_ON(PAGE_SIZE > SZ_64K); - VM_WARN_ON_ONCE(!PageZsmalloc(page)); + VM_WARN_ON_ONCE(!PageZsmalloc(zpdesc_page(zpdesc))); VM_WARN_ON_ONCE(offset & ~FIRST_OBJ_PAGE_TYPE_MASK); - page->page_type &=3D ~FIRST_OBJ_PAGE_TYPE_MASK; - page->page_type |=3D offset & FIRST_OBJ_PAGE_TYPE_MASK; + zpdesc->first_obj_offset &=3D ~FIRST_OBJ_PAGE_TYPE_MASK; + zpdesc->first_obj_offset |=3D offset & FIRST_OBJ_PAGE_TYPE_MASK; } =20 static inline unsigned int get_freeobj(struct zspage *zspage) @@ -850,7 +850,7 @@ static void reset_zpdesc(struct zpdesc *zpdesc) ClearPagePrivate(page); zpdesc->zspage =3D NULL; zpdesc->next =3D NULL; - reset_first_obj_offset(page); + reset_first_obj_offset(zpdesc); __ClearPageZsmalloc(page); } =20 @@ -934,7 +934,7 @@ static void init_zspage(struct size_class *class, struc= t zspage *zspage) struct link_free *link; void *vaddr; =20 - set_first_obj_offset(zpdesc_page(zpdesc), off); + set_first_obj_offset(zpdesc, off); =20 vaddr =3D zpdesc_kmap_atomic(zpdesc); link =3D (struct link_free *)vaddr + off / sizeof(*link); @@ -1589,7 +1589,7 @@ static unsigned long find_alloced_obj(struct size_cla= ss *class, unsigned long handle =3D 0; void *addr =3D zpdesc_kmap_atomic(zpdesc); =20 - offset =3D get_first_obj_offset(zpdesc_page(zpdesc)); + offset =3D get_first_obj_offset(zpdesc); offset +=3D class->size * index; =20 while (offset < PAGE_SIZE) { @@ -1784,8 +1784,8 @@ static void replace_sub_page(struct size_class *class= , struct zspage *zspage, } while ((zpdesc =3D get_next_zpdesc(zpdesc)) !=3D NULL); =20 create_page_chain(class, zspage, zpdescs); - first_obj_offset =3D get_first_obj_offset(zpdesc_page(oldzpdesc)); - set_first_obj_offset(zpdesc_page(newzpdesc), first_obj_offset); + first_obj_offset =3D get_first_obj_offset(oldzpdesc); + set_first_obj_offset(newzpdesc, first_obj_offset); if (unlikely(ZsHugePage(zspage))) newzpdesc->handle =3D oldzpdesc->handle; __zpdesc_set_movable(newzpdesc, &zsmalloc_mops); @@ -1840,7 +1840,7 @@ static int zs_page_migrate(struct page *newpage, stru= ct page *page, /* the migrate_write_lock protects zpage access via zs_map_object */ migrate_write_lock(zspage); =20 - offset =3D get_first_obj_offset(zpdesc_page(zpdesc)); + offset =3D get_first_obj_offset(zpdesc); s_addr =3D zpdesc_kmap_atomic(zpdesc); =20 /* --=20 2.43.0 From nobody Sun Feb 8 02:21:45 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CC7CB74077 for ; Mon, 8 Jul 2024 06:29:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720420188; cv=none; b=s5mYP4TZvJrY04s9PdpQNzGjvfWmrIA/A9bVJatVFJmj2oGWrpCUwTsS6g1vmT9ZcZwqVe8zDc4ohx8ZWpCwkkexI46uJTvo41nuAODvmPGDrw9P+ZSyLeUH9Bm8w0akjoVcdKdhz6Fufb01Go/0HvsggnijYipDwJL5wwEA9Ac= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720420188; c=relaxed/simple; bh=H24xLCJMLoMGxtoL02JioPie/ckhzBt773ZzPhU2CSg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=qtw3ThVeAkKa6Zw2A4RjZbkOphRJITCaZnKHajWQhJwLY84dMP40Z7YPONbIoKKICot7ntorNuskJ1noHfJdmIYhaSKB10YfA4KJP+ONM4y8IROeoeFmUVWF/YQxw8qQlWhYY7jBMtdGbxta+CyzytSU71SoJYw1Xbu3SbSjMH0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=jWOtxdXd; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="jWOtxdXd" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 66EF4C4AF0C; Mon, 8 Jul 2024 06:29:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1720420188; bh=H24xLCJMLoMGxtoL02JioPie/ckhzBt773ZzPhU2CSg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=jWOtxdXdW3pAl2WmrOj1bOnrw+gHOrlhH3zXwNH4eP7cA0yNru3LwOhi4tzn80Rbx cVD76ITdy6l5ySTmIFusfFSQBHDl8J2VB4jbf3A54xhBGhKvlLpl8LWFsbDN6yqiiW s2n8BudgTiO2TsxYnXK3NQY9OpoBTNB5iGwrGOdRxzmGT8ouZKbT+91GBd49d5aNq4 l09v7kThk+LyOa6yi3Rt5tTRG8o5GcSjP6xGzOshEy/RdkCcZhw0faZV7UOxRLyCzt dixnOcM1H9x7wG3WYr7joJy86clgn16ANgHuJ4rzG9xZ0sTJu/7N4idkcziq+vkXnp VaalbA+QoqWpg== From: alexs@kernel.org To: Vitaly Wool , Miaohe Lin , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, minchan@kernel.org, willy@infradead.org, senozhatsky@chromium.org, david@redhat.com, 42.hyeyoo@gmail.com, Yosry Ahmed , nphamcs@gmail.com Cc: Alex Shi Subject: [PATCH v3 18/20] mm/zsmalloc: introduce __zpdesc_clear_movable Date: Mon, 8 Jul 2024 14:33:39 +0800 Message-ID: <20240708063344.1096626-19-alexs@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240708063344.1096626-1-alexs@kernel.org> References: <20240708063344.1096626-1-alexs@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Alex Shi (Tencent) Add a helper __zpdesc_clear_movable() for __ClearPageMovable(), and use it in callers to make code clear. Signed-off-by: Alex Shi (Tencent) --- mm/zpdesc.h | 5 +++++ mm/zsmalloc.c | 2 +- 2 files changed, 6 insertions(+), 1 deletion(-) diff --git a/mm/zpdesc.h b/mm/zpdesc.h index f64e813f4847..5db4fbe2d139 100644 --- a/mm/zpdesc.h +++ b/mm/zpdesc.h @@ -114,6 +114,11 @@ static inline void __zpdesc_set_movable(struct zpdesc = *zpdesc, __SetPageMovable(zpdesc_page(zpdesc), mops); } =20 +static inline void __zpdesc_clear_movable(struct zpdesc *zpdesc) +{ + __ClearPageMovable(zpdesc_page(zpdesc)); +} + static inline bool zpdesc_is_isolated(struct zpdesc *zpdesc) { return PageIsolated(zpdesc_page(zpdesc)); diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index bb8b5f13a966..e1d3ad50538c 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -846,7 +846,7 @@ static void reset_zpdesc(struct zpdesc *zpdesc) { struct page *page =3D zpdesc_page(zpdesc); =20 - __ClearPageMovable(page); + __zpdesc_clear_movable(zpdesc); ClearPagePrivate(page); zpdesc->zspage =3D NULL; zpdesc->next =3D NULL; --=20 2.43.0 From nobody Sun Feb 8 02:21:45 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0B0E6768EE for ; Mon, 8 Jul 2024 06:29:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720420192; cv=none; b=CUb1zCls21hnc3NPyX3NZn4hCA68LqHv+Jm+mi4CbJXKygCuuisUloxCgzkGKYuJdOjC7wbiJ+NL5LlkzROiWjoRaettlAA9kfXtQAA+YNDwMyEi4GlPsUqcRtx+sJAm+RCsRPziPTbLleuddQ2BwBAhEypvKMftBPWvOo19IAw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720420192; c=relaxed/simple; bh=ld5kbfQznGqvmPXTBRxo+Gb6MyvpG3bwUJO7MMXVjOQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=brwdhlEe4aNysnJtbszy/OLBLVOzFHRPDeL0lLIz4gMp9ZEq1/YtFl+aG6/oWdrJTsJp8smGYfCYA9M31/vsVChX0jcfZ7pRwDQ1YDIwJ9azdR2dQqDwN1Yk+e/7J6s7bjnpIckYovXD8UL1AumZAzPv3L7bvlTDGwdzz+8fyNk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=UEHuelEs; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="UEHuelEs" Received: by smtp.kernel.org (Postfix) with ESMTPSA id F38EDC4AF0C; Mon, 8 Jul 2024 06:29:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1720420191; bh=ld5kbfQznGqvmPXTBRxo+Gb6MyvpG3bwUJO7MMXVjOQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=UEHuelEsMNDLfNwnCkG7vPJYg9IVTB41xkci7s51NiPFoDJgxr1CUaRji/cVlqlNA vYdN8sXnViZNe3kSp0lsazF7+z8vLY4xWfdHqBLMGbC9cmI4uqUdMGYDo3TQBRCOan bINlkUPVQMuXuhIZ+Nmq7VOu7niPAgJ43p5dnvdDeoiuXWitnqDwZtAzukSp15x9ZK cWnFn0H6I99oVMAEI4Rc5Q3DCx5/rPZKTYyg1s6wKXmbZf96IvA0lyCPzp1T9ncpt5 0eZQBV6bEE+3Jborz9xlh5OA8pkf9vfMZbQks6eVJYhboRqe0qKCucYlARPqFpn8/R Gv4XpWfGxrqZQ== From: alexs@kernel.org To: Vitaly Wool , Miaohe Lin , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, minchan@kernel.org, willy@infradead.org, senozhatsky@chromium.org, david@redhat.com, 42.hyeyoo@gmail.com, Yosry Ahmed , nphamcs@gmail.com Cc: Alex Shi Subject: [PATCH v3 19/20] mm/zsmalloc: introduce __zpdesc_clear_zsmalloc Date: Mon, 8 Jul 2024 14:33:40 +0800 Message-ID: <20240708063344.1096626-20-alexs@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240708063344.1096626-1-alexs@kernel.org> References: <20240708063344.1096626-1-alexs@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Alex Shi (Tencent) Add a helper __zpdesc_clear_zsmalloc() for __ClearPageZsmalloc(), and use it in callers to make code clear. Signed-off-by: Alex Shi (Tencent) --- mm/zpdesc.h | 5 +++++ mm/zsmalloc.c | 4 ++-- 2 files changed, 7 insertions(+), 2 deletions(-) diff --git a/mm/zpdesc.h b/mm/zpdesc.h index 5db4fbe2d139..05def4d45265 100644 --- a/mm/zpdesc.h +++ b/mm/zpdesc.h @@ -119,6 +119,11 @@ static inline void __zpdesc_clear_movable(struct zpdes= c *zpdesc) __ClearPageMovable(zpdesc_page(zpdesc)); } =20 +static inline void __zpdesc_clear_zsmalloc(struct zpdesc *zpdesc) +{ + __ClearPageZsmalloc(zpdesc_page(zpdesc)); +} + static inline bool zpdesc_is_isolated(struct zpdesc *zpdesc) { return PageIsolated(zpdesc_page(zpdesc)); diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index e1d3ad50538c..d88602fb0233 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -851,7 +851,7 @@ static void reset_zpdesc(struct zpdesc *zpdesc) zpdesc->zspage =3D NULL; zpdesc->next =3D NULL; reset_first_obj_offset(zpdesc); - __ClearPageZsmalloc(page); + __zpdesc_clear_zsmalloc(zpdesc); } =20 static int trylock_zspage(struct zspage *zspage) @@ -1024,7 +1024,7 @@ static struct zspage *alloc_zspage(struct zs_pool *po= ol, if (!zpdesc) { while (--i >=3D 0) { zpdesc_dec_zone_page_state(zpdescs[i]); - __ClearPageZsmalloc(zpdesc_page(zpdescs[i])); + __zpdesc_clear_zsmalloc(zpdescs[i]); free_zpdesc(zpdescs[i]); } cache_free_zspage(pool, zspage); --=20 2.43.0 From nobody Sun Feb 8 02:21:45 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BCB0C77F08 for ; Mon, 8 Jul 2024 06:29:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720420195; cv=none; b=Fd92JOG8BYytHzh6rsIGoqbP6oybIpHH9q6PWxroiJ2mKJ5GVeSejA0RvADca80uQ7H0wNoh/ID82LTfCMfei2ZmWuppeUoTbgoxLTRkw2ehiszLuWNcJxPqrzz5Ye0s5SFK4PHpXmjm132xD/7ilBKeGsjG+jAxBAKqLIVkqeM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720420195; c=relaxed/simple; bh=YePXlWlhu0t9cQ2FhNb4Tu9YraM33q9YNNO84M2/1AU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=abf/0bP/z7/Bp2GjuOYPRb8myNWtqWoNfG+oB5SGKE1xPbIZ86tFh972iCWSrjAvKcj9rsJzU19vcKMnOVEOcoLFT2eGKBj8yuxhM1EHe6rM/bewMmSZtcKNbQTctQhfOeCXxZW9DsFpniti4jYrtf5MlJQroHY/ULeNasAVWTs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=P9pIQxRc; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="P9pIQxRc" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 89CF3C4AF0B; Mon, 8 Jul 2024 06:29:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1720420195; bh=YePXlWlhu0t9cQ2FhNb4Tu9YraM33q9YNNO84M2/1AU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=P9pIQxRc74qXODLyeYpN9Usf+mKhKNyEtYaG/uShlHPiP4pI4bAMJ0ZxPksRq+auS Qq89KVFjK+cS/UBpysNLdcOw2E2hcw1sCMvZl3KUc+HOJbs49yZs8V9xBzD7CQpPXk xrf1GXo8qGKrN8Mfj5BxDOcq96OdiaGcpT+j9Z+Vx0xpJbuPHuGsusNl5njig/hpaW a2NyztZknNxisBfbFjaz0C56dqMEDimRCGr7nI1JjNqJV+zZNtkz4BQqyS4XDUDCt+ XUshBq06XfyGRAvEqCFVF5NOrrRoaTNdPTkLKW0edIVF8ssJm5jovq14e7nkoO7ZPB O0r04fevGt45g== From: alexs@kernel.org To: Vitaly Wool , Miaohe Lin , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, minchan@kernel.org, willy@infradead.org, senozhatsky@chromium.org, david@redhat.com, 42.hyeyoo@gmail.com, Yosry Ahmed , nphamcs@gmail.com Cc: Alex Shi Subject: [PATCH v3 20/20] mm/zsmalloc: introduce __zpdesc_set_zsmalloc() Date: Mon, 8 Jul 2024 14:33:41 +0800 Message-ID: <20240708063344.1096626-21-alexs@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240708063344.1096626-1-alexs@kernel.org> References: <20240708063344.1096626-1-alexs@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Alex Shi (Tencent) Add a helper __zpdesc_set_zsmalloc() for __SetPageZsmalloc(), and use it in callers to make code clear. Signed-off-by: Alex Shi (Tencent) --- mm/zpdesc.h | 5 +++++ mm/zsmalloc.c | 4 ++-- 2 files changed, 7 insertions(+), 2 deletions(-) diff --git a/mm/zpdesc.h b/mm/zpdesc.h index 05def4d45265..06371ce60cd1 100644 --- a/mm/zpdesc.h +++ b/mm/zpdesc.h @@ -119,6 +119,11 @@ static inline void __zpdesc_clear_movable(struct zpdes= c *zpdesc) __ClearPageMovable(zpdesc_page(zpdesc)); } =20 +static inline void __zpdesc_set_zsmalloc(struct zpdesc *zpdesc) +{ + __SetPageZsmalloc(zpdesc_page(zpdesc)); +} + static inline void __zpdesc_clear_zsmalloc(struct zpdesc *zpdesc) { __ClearPageZsmalloc(zpdesc_page(zpdesc)); diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index d88602fb0233..7f8e02df4e3e 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -1030,7 +1030,7 @@ static struct zspage *alloc_zspage(struct zs_pool *po= ol, cache_free_zspage(pool, zspage); return NULL; } - __SetPageZsmalloc(zpdesc_page(zpdesc)); + __zpdesc_set_zsmalloc(zpdesc); =20 zpdesc_inc_zone_page_state(zpdesc); zpdescs[i] =3D zpdesc; @@ -1820,7 +1820,7 @@ static int zs_page_migrate(struct page *newpage, stru= ct page *page, VM_BUG_ON_PAGE(!zpdesc_is_isolated(zpdesc), zpdesc_page(zpdesc)); =20 /* We're committed, tell the world that this is a Zsmalloc page. */ - __SetPageZsmalloc(zpdesc_page(newzpdesc)); + __zpdesc_set_zsmalloc(newzpdesc); =20 /* The page is locked, so this pointer must remain valid */ zspage =3D get_zspage(zpdesc); --=20 2.43.0