From nobody Mon Feb 9 15:06:54 2026 Received: from out-175.mta1.migadu.com (out-175.mta1.migadu.com [95.215.58.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E134D1E102E for ; Wed, 29 Jan 2025 18:06:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=95.215.58.175 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738174013; cv=none; b=Sei7Yd8qQgRtSvQjRXggRNQFnQBFFR/BNonCQGWRtW/XNoOzU/FkL+Lnz3fHgeSgbaPiNzQO9U7JEiZY3xkx7kmsCmMAgFMf8uI0dxPlmzFDl0yA+aKT1EhGmp1cxXUXKljZo4L5maIEeLRy5lnEZi9MWzvMRARSZGEaoHHsQhQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738174013; c=relaxed/simple; bh=himmZ4xjv3BI13JecayKN8qS5Mo+rMtLmqEtknkIwWs=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=AIZpM7Xuyw5oDLYC9Ar5gFagV2XglOgihVWDGGV882t9PiWnWenrYMusNbYPRIuQffbW15AqCScuIcpVkIg86smZYH1btsXI4U81PyZ6mwb+kSTNPwEhmcYi9V7Yd7X7hGjI5X20RqfSEV5blQ+Y+6dd+dWjFcAur+aYEHTTGbk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=DDP7wGSD; arc=none smtp.client-ip=95.215.58.175 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="DDP7wGSD" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1738174007; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FT4YGOt3bo9db/JwvEGY/YuMUDbNNKm3knPDe4QtXNY=; b=DDP7wGSDPsbxG/VKR86wfJzdmJHxncLLxiBFm9cPZq75611W6ZPCZ1qWqdLWFZHiKfz1p4 W0dUDzkjFQQIEVNeOdZ8CQWEQxzxfvoKT5JwB98SHvjm/x/9eHuXxO+CNYkJ3zbwAprutO 1mYG5sLZDrSkCbY6M93T8fdvtdPDsQA= From: Yosry Ahmed To: Andrew Morton Cc: Vitaly Wool , Seth Jennings , Dan Streetman , Miaohe Lin , Johannes Weiner , Nhat Pham , Chengming Zhou , Huacai Chen , WANG Xuerui , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Shakeel Butt , Vlastimil Babka , linux-mm@kvack.org, linux-kernel@vger.kernel.org, loongarch@lists.linux.dev, linux-s390@vger.kernel.org, linux-doc@vger.kernel.org Subject: [PATCH v2 1/2] mm: z3fold: remove z3fold Date: Wed, 29 Jan 2025 18:06:31 +0000 Message-ID: <20250129180633.3501650-2-yosry.ahmed@linux.dev> In-Reply-To: <20250129180633.3501650-1-yosry.ahmed@linux.dev> References: <20250129180633.3501650-1-yosry.ahmed@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Migadu-Flow: FLOW_OUT Z3fold has been marked as deprecated for 2 cycles and no one complained, as expected. As there are no known users, remove the code now. Signed-off-by: Yosry Ahmed Acked-by: Johannes Weiner Reviewed-by: Shakeel Butt Acked-by: Nhat Pham --- CREDITS | 1 + Documentation/mm/index.rst | 1 - Documentation/mm/z3fold.rst | 28 - Documentation/translations/zh_CN/mm/index.rst | 1 - .../translations/zh_CN/mm/z3fold.rst | 31 - MAINTAINERS | 7 - mm/Kconfig | 29 - mm/Makefile | 1 - mm/z3fold.c | 1447 ----------------- 9 files changed, 1 insertion(+), 1545 deletions(-) delete mode 100644 Documentation/mm/z3fold.rst delete mode 100644 Documentation/translations/zh_CN/mm/z3fold.rst delete mode 100644 mm/z3fold.c diff --git a/CREDITS b/CREDITS index 1f9f0f078b4ae..324428ad37e83 100644 --- a/CREDITS +++ b/CREDITS @@ -4313,6 +4313,7 @@ S: England N: Vitaly Wool E: vitaly.wool@konsulko.com D: Maintenance and development of zswap +D: Maintenance and development of z3fold =20 N: Chris Wright E: chrisw@sous-sol.org diff --git a/Documentation/mm/index.rst b/Documentation/mm/index.rst index 0be1c7503a010..d3ada3e45e103 100644 --- a/Documentation/mm/index.rst +++ b/Documentation/mm/index.rst @@ -62,5 +62,4 @@ documentation, or deleted if it has served its purpose. unevictable-lru vmalloced-kernel-stacks vmemmap_dedup - z3fold zsmalloc diff --git a/Documentation/mm/z3fold.rst b/Documentation/mm/z3fold.rst deleted file mode 100644 index 25b5935d06c7d..0000000000000 --- a/Documentation/mm/z3fold.rst +++ /dev/null @@ -1,28 +0,0 @@ -=3D=3D=3D=3D=3D=3D -z3fold -=3D=3D=3D=3D=3D=3D - -z3fold is a special purpose allocator for storing compressed pages. -It is designed to store up to three compressed pages per physical page. -It is a zbud derivative which allows for higher compression -ratio keeping the simplicity and determinism of its predecessor. - -The main differences between z3fold and zbud are: - -* unlike zbud, z3fold allows for up to PAGE_SIZE allocations -* z3fold can hold up to 3 compressed pages in its page -* z3fold doesn't export any API itself and is thus intended to be used - via the zpool API. - -To keep the determinism and simplicity, z3fold, just like zbud, always -stores an integral number of compressed pages per page, but it can store -up to 3 pages unlike zbud which can store at most 2. Therefore the -compression ratio goes to around 2.7x while zbud's one is around 1.7x. - -Unlike zbud (but like zsmalloc for that matter) z3fold_alloc() does not -return a dereferenceable pointer. Instead, it returns an unsigned long -handle which encodes actual location of the allocated object. - -Keeping effective compression ratio close to zsmalloc's, z3fold doesn't -depend on MMU enabled and provides more predictable reclaim behavior -which makes it a better fit for small and response-critical systems. diff --git a/Documentation/translations/zh_CN/mm/index.rst b/Documentation/= translations/zh_CN/mm/index.rst index c8726bce8f745..a71116be058fd 100644 --- a/Documentation/translations/zh_CN/mm/index.rst +++ b/Documentation/translations/zh_CN/mm/index.rst @@ -58,7 +58,6 @@ Linux=E5=86=85=E5=AD=98=E7=AE=A1=E7=90=86=E6=96=87=E6=A1= =A3 remap_file_pages split_page_table_lock vmalloced-kernel-stacks - z3fold zsmalloc =20 TODOLIST: diff --git a/Documentation/translations/zh_CN/mm/z3fold.rst b/Documentation= /translations/zh_CN/mm/z3fold.rst deleted file mode 100644 index 9569a6d882700..0000000000000 --- a/Documentation/translations/zh_CN/mm/z3fold.rst +++ /dev/null @@ -1,31 +0,0 @@ -:Original: Documentation/mm/z3fold.rst - -:=E7=BF=BB=E8=AF=91: - - =E5=8F=B8=E5=BB=B6=E8=85=BE Yanteng Si - -:=E6=A0=A1=E8=AF=91: - - -=3D=3D=3D=3D=3D=3D -z3fold -=3D=3D=3D=3D=3D=3D - -z3fold=E6=98=AF=E4=B8=80=E4=B8=AA=E4=B8=93=E9=97=A8=E7=94=A8=E4=BA=8E=E5= =AD=98=E5=82=A8=E5=8E=8B=E7=BC=A9=E9=A1=B5=E7=9A=84=E5=88=86=E9=85=8D=E5=99= =A8=E3=80=82=E5=AE=83=E8=A2=AB=E8=AE=BE=E8=AE=A1=E4=B8=BA=E6=AF=8F=E4=B8=AA= =E7=89=A9=E7=90=86=E9=A1=B5=E6=9C=80=E5=A4=9A=E5=8F=AF=E4=BB=A5=E5=AD=98=E5= =82=A8=E4=B8=89=E4=B8=AA=E5=8E=8B=E7=BC=A9=E9=A1=B5=E3=80=82 -=E5=AE=83=E6=98=AFzbud=E7=9A=84=E8=A1=8D=E7=94=9F=E7=89=A9=EF=BC=8C=E5=85= =81=E8=AE=B8=E6=9B=B4=E9=AB=98=E7=9A=84=E5=8E=8B=E7=BC=A9=E7=8E=87=EF=BC=8C= =E4=BF=9D=E6=8C=81=E5=85=B6=E5=89=8D=E8=BE=88=E7=9A=84=E7=AE=80=E5=8D=95=E6= =80=A7=E5=92=8C=E7=A1=AE=E5=AE=9A=E6=80=A7=E3=80=82 - -z3fold=E5=92=8Czbud=E7=9A=84=E4=B8=BB=E8=A6=81=E5=8C=BA=E5=88=AB=E6=98=AF: - -* =E4=B8=8Ezbud=E4=B8=8D=E5=90=8C=E7=9A=84=E6=98=AF=EF=BC=8Cz3fold=E5=85= =81=E8=AE=B8=E6=9C=80=E5=A4=A7=E7=9A=84PAGE_SIZE=E5=88=86=E9=85=8D=E3=80=82 -* z3fold=E5=9C=A8=E5=85=B6=E9=A1=B5=E9=9D=A2=E4=B8=AD=E6=9C=80=E5=A4=9A=E5= =8F=AF=E4=BB=A5=E5=AE=B9=E7=BA=B33=E4=B8=AA=E5=8E=8B=E7=BC=A9=E9=A1=B5=E9= =9D=A2 -* z3fold=E6=9C=AC=E8=BA=AB=E6=B2=A1=E6=9C=89=E8=BE=93=E5=87=BA=E4=BB=BB=E4= =BD=95API=EF=BC=8C=E5=9B=A0=E6=AD=A4=E6=89=93=E7=AE=97=E9=80=9A=E8=BF=87zpo= ol=E7=9A=84API=E6=9D=A5=E4=BD=BF=E7=94=A8 - -=E4=B8=BA=E4=BA=86=E4=BF=9D=E6=8C=81=E7=A1=AE=E5=AE=9A=E6=80=A7=E5=92=8C= =E7=AE=80=E5=8D=95=E6=80=A7=EF=BC=8Cz3fold=EF=BC=8C=E5=B0=B1=E5=83=8Fzbud= =E4=B8=80=E6=A0=B7=EF=BC=8C=E6=80=BB=E6=98=AF=E5=9C=A8=E6=AF=8F=E9=A1=B5=E5= =AD=98=E5=82=A8=E4=B8=80=E4=B8=AA=E6=95=B4=E6=95=B0=E7=9A=84=E5=8E=8B=E7=BC= =A9=E9=A1=B5=EF=BC=8C=E4=BD=86=E6=98=AF -=E5=AE=83=E6=9C=80=E5=A4=9A=E5=8F=AF=E4=BB=A5=E5=AD=98=E5=82=A83=E9=A1=B5= =EF=BC=8C=E4=B8=8D=E5=83=8Fzbud=E6=9C=80=E5=A4=9A=E5=8F=AF=E4=BB=A5=E5=AD= =98=E5=82=A82=E9=A1=B5=E3=80=82=E5=9B=A0=E6=AD=A4=E5=8E=8B=E7=BC=A9=E7=8E= =87=E8=BE=BE=E5=88=B02.7=E5=80=8D=E5=B7=A6=E5=8F=B3=EF=BC=8C=E8=80=8Czbud= =E7=9A=84=E5=8E=8B=E7=BC=A9 -=E7=8E=87=E6=98=AF1.7=E5=80=8D=E5=B7=A6=E5=8F=B3=E3=80=82 - -=E4=B8=8D=E5=83=8Fzbud=EF=BC=88=E4=BD=86=E4=B9=9F=E5=83=8Fzsmalloc=EF=BC= =89=EF=BC=8Cz3fold_alloc()=E9=82=A3=E6=A0=B7=E4=B8=8D=E8=BF=94=E5=9B=9E=E4= =B8=80=E4=B8=AA=E5=8F=AF=E9=87=8D=E5=A4=8D=E5=BC=95=E7=94=A8=E7=9A=84=E6=8C= =87=E9=92=88=E3=80=82=E7=9B=B8=E5=8F=8D=EF=BC=8C=E5=AE=83 -=E8=BF=94=E5=9B=9E=E4=B8=80=E4=B8=AA=E6=97=A0=E7=AC=A6=E5=8F=B7=E9=95=BF= =E5=8F=A5=E6=9F=84=EF=BC=8C=E5=AE=83=E7=BC=96=E7=A0=81=E4=BA=86=E8=A2=AB=E5= =88=86=E9=85=8D=E5=AF=B9=E8=B1=A1=E7=9A=84=E5=AE=9E=E9=99=85=E4=BD=8D=E7=BD= =AE=E3=80=82 - -=E4=BF=9D=E6=8C=81=E6=9C=89=E6=95=88=E7=9A=84=E5=8E=8B=E7=BC=A9=E7=8E=87= =E6=8E=A5=E8=BF=91=E4=BA=8Ezsmalloc=EF=BC=8Cz3fold=E4=B8=8D=E4=BE=9D=E8=B5= =96=E4=BA=8EMMU=E7=9A=84=E5=90=AF=E7=94=A8=EF=BC=8C=E5=B9=B6=E6=8F=90=E4=BE= =9B=E6=9B=B4=E5=8F=AF=E9=A2=84=E6=B5=8B=E7=9A=84=E5=9B=9E=E6=94=B6=E8=A1=8C -=E4=B8=BA=EF=BC=8C=E8=BF=99=E4=BD=BF=E5=BE=97=E5=AE=83=E6=9B=B4=E9=80=82= =E5=90=88=E4=BA=8E=E5=B0=8F=E5=9E=8B=E5=92=8C=E5=8F=8D=E5=BA=94=E8=BF=85=E9= =80=9F=E7=9A=84=E7=B3=BB=E7=BB=9F=E3=80=82 diff --git a/MAINTAINERS b/MAINTAINERS index feed152470f68..1583bb8eee587 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -26055,13 +26055,6 @@ S: Maintained F: Documentation/input/devices/yealink.rst F: drivers/input/misc/yealink.* =20 -Z3FOLD COMPRESSED PAGE ALLOCATOR -M: Vitaly Wool -R: Miaohe Lin -L: linux-mm@kvack.org -S: Maintained -F: mm/z3fold.c - Z8530 DRIVER FOR AX.25 M: Joerg Reuter L: linux-hams@vger.kernel.org diff --git a/mm/Kconfig b/mm/Kconfig index 1b501db064172..6fa19022c09b8 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -146,15 +146,6 @@ config ZSWAP_ZPOOL_DEFAULT_ZBUD help Use the zbud allocator as the default allocator. =20 -config ZSWAP_ZPOOL_DEFAULT_Z3FOLD_DEPRECATED - bool "z3foldi (DEPRECATED)" - select Z3FOLD_DEPRECATED - help - Use the z3fold allocator as the default allocator. - - Deprecated and scheduled for removal in a few cycles, - see CONFIG_Z3FOLD_DEPRECATED. - config ZSWAP_ZPOOL_DEFAULT_ZSMALLOC bool "zsmalloc" select ZSMALLOC @@ -166,7 +157,6 @@ config ZSWAP_ZPOOL_DEFAULT string depends on ZSWAP default "zbud" if ZSWAP_ZPOOL_DEFAULT_ZBUD - default "z3fold" if ZSWAP_ZPOOL_DEFAULT_Z3FOLD_DEPRECATED default "zsmalloc" if ZSWAP_ZPOOL_DEFAULT_ZSMALLOC default "" =20 @@ -180,25 +170,6 @@ config ZBUD deterministic reclaim properties that make it preferable to a higher density approach when reclaim will be used. =20 -config Z3FOLD_DEPRECATED - tristate "3:1 compression allocator (z3fold) (DEPRECATED)" - depends on ZSWAP - help - Deprecated and scheduled for removal in a few cycles. If you have - a good reason for using Z3FOLD over ZSMALLOC, please contact - linux-mm@kvack.org and the zswap maintainers. - - A special purpose allocator for storing compressed pages. - It is designed to store up to three compressed pages per physical - page. It is a ZBUD derivative so the simplicity and determinism are - still there. - -config Z3FOLD - tristate - default y if Z3FOLD_DEPRECATED=3Dy - default m if Z3FOLD_DEPRECATED=3Dm - depends on Z3FOLD_DEPRECATED - config ZSMALLOC tristate prompt "N:1 compression allocator (zsmalloc)" if (ZSWAP || ZRAM) diff --git a/mm/Makefile b/mm/Makefile index 850386a67b3e0..e4c03da3c0846 100644 --- a/mm/Makefile +++ b/mm/Makefile @@ -115,7 +115,6 @@ obj-$(CONFIG_MEMORY_ISOLATION) +=3D page_isolation.o obj-$(CONFIG_ZPOOL) +=3D zpool.o obj-$(CONFIG_ZBUD) +=3D zbud.o obj-$(CONFIG_ZSMALLOC) +=3D zsmalloc.o -obj-$(CONFIG_Z3FOLD) +=3D z3fold.o obj-$(CONFIG_GENERIC_EARLY_IOREMAP) +=3D early_ioremap.o obj-$(CONFIG_CMA) +=3D cma.o obj-$(CONFIG_NUMA) +=3D numa.o diff --git a/mm/z3fold.c b/mm/z3fold.c deleted file mode 100644 index 379d24b4fef99..0000000000000 --- a/mm/z3fold.c +++ /dev/null @@ -1,1447 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only -/* - * z3fold.c - * - * Author: Vitaly Wool - * Copyright (C) 2016, Sony Mobile Communications Inc. - * - * This implementation is based on zbud written by Seth Jennings. - * - * z3fold is an special purpose allocator for storing compressed pages. It - * can store up to three compressed pages per page which improves the - * compression ratio of zbud while retaining its main concepts (e. g. alwa= ys - * storing an integral number of objects per page) and simplicity. - * It still has simple and deterministic reclaim properties that make it - * preferable to a higher density approach (with no requirement on integral - * number of object per page) when reclaim is used. - * - * As in zbud, pages are divided into "chunks". The size of the chunks is - * fixed at compile time and is determined by NCHUNKS_ORDER below. - * - * z3fold doesn't export any API and is meant to be used via zpool API. - */ - -#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt - -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include - -/* - * NCHUNKS_ORDER determines the internal allocation granularity, effective= ly - * adjusting internal fragmentation. It also determines the number of - * freelists maintained in each pool. NCHUNKS_ORDER of 6 means that the - * allocation granularity will be in chunks of size PAGE_SIZE/64. Some chu= nks - * in the beginning of an allocated page are occupied by z3fold header, so - * NCHUNKS will be calculated to 63 (or 62 in case CONFIG_DEBUG_SPINLOCK= =3Dy), - * which shows the max number of free chunks in z3fold page, also there wi= ll - * be 63, or 62, respectively, freelists per pool. - */ -#define NCHUNKS_ORDER 6 - -#define CHUNK_SHIFT (PAGE_SHIFT - NCHUNKS_ORDER) -#define CHUNK_SIZE (1 << CHUNK_SHIFT) -#define ZHDR_SIZE_ALIGNED round_up(sizeof(struct z3fold_header), CHUNK_SIZ= E) -#define ZHDR_CHUNKS (ZHDR_SIZE_ALIGNED >> CHUNK_SHIFT) -#define TOTAL_CHUNKS (PAGE_SIZE >> CHUNK_SHIFT) -#define NCHUNKS (TOTAL_CHUNKS - ZHDR_CHUNKS) - -#define BUDDY_MASK (0x3) -#define BUDDY_SHIFT 2 -#define SLOTS_ALIGN (0x40) - -/***************** - * Structures -*****************/ -struct z3fold_pool; - -enum buddy { - HEADLESS =3D 0, - FIRST, - MIDDLE, - LAST, - BUDDIES_MAX =3D LAST -}; - -struct z3fold_buddy_slots { - /* - * we are using BUDDY_MASK in handle_to_buddy etc. so there should - * be enough slots to hold all possible variants - */ - unsigned long slot[BUDDY_MASK + 1]; - unsigned long pool; /* back link */ - rwlock_t lock; -}; -#define HANDLE_FLAG_MASK (0x03) - -/* - * struct z3fold_header - z3fold page metadata occupying first chunks of e= ach - * z3fold page, except for HEADLESS pages - * @buddy: links the z3fold page into the relevant list in the - * pool - * @page_lock: per-page lock - * @refcount: reference count for the z3fold page - * @work: work_struct for page layout optimization - * @slots: pointer to the structure holding buddy slots - * @pool: pointer to the containing pool - * @cpu: CPU which this page "belongs" to - * @first_chunks: the size of the first buddy in chunks, 0 if free - * @middle_chunks: the size of the middle buddy in chunks, 0 if free - * @last_chunks: the size of the last buddy in chunks, 0 if free - * @first_num: the starting number (for the first handle) - * @mapped_count: the number of objects currently mapped - */ -struct z3fold_header { - struct list_head buddy; - spinlock_t page_lock; - struct kref refcount; - struct work_struct work; - struct z3fold_buddy_slots *slots; - struct z3fold_pool *pool; - short cpu; - unsigned short first_chunks; - unsigned short middle_chunks; - unsigned short last_chunks; - unsigned short start_middle; - unsigned short first_num:2; - unsigned short mapped_count:2; - unsigned short foreign_handles:2; -}; - -/** - * struct z3fold_pool - stores metadata for each z3fold pool - * @name: pool name - * @lock: protects pool unbuddied lists - * @stale_lock: protects pool stale page list - * @unbuddied: per-cpu array of lists tracking z3fold pages that contain 2- - * buddies; the list each z3fold page is added to depends on - * the size of its free region. - * @stale: list of pages marked for freeing - * @pages_nr: number of z3fold pages in the pool. - * @c_handle: cache for z3fold_buddy_slots allocation - * @compact_wq: workqueue for page layout background optimization - * @release_wq: workqueue for safe page release - * @work: work_struct for safe page release - * - * This structure is allocated at pool creation time and maintains metadata - * pertaining to a particular z3fold pool. - */ -struct z3fold_pool { - const char *name; - spinlock_t lock; - spinlock_t stale_lock; - struct list_head __percpu *unbuddied; - struct list_head stale; - atomic64_t pages_nr; - struct kmem_cache *c_handle; - struct workqueue_struct *compact_wq; - struct workqueue_struct *release_wq; - struct work_struct work; -}; - -/* - * Internal z3fold page flags - */ -enum z3fold_page_flags { - PAGE_HEADLESS =3D 0, - MIDDLE_CHUNK_MAPPED, - NEEDS_COMPACTING, - PAGE_STALE, - PAGE_CLAIMED, /* by either reclaim or free */ - PAGE_MIGRATED, /* page is migrated and soon to be released */ -}; - -/* - * handle flags, go under HANDLE_FLAG_MASK - */ -enum z3fold_handle_flags { - HANDLES_NOFREE =3D 0, -}; - -/* - * Forward declarations - */ -static struct z3fold_header *__z3fold_alloc(struct z3fold_pool *, size_t, = bool); -static void compact_page_work(struct work_struct *w); - -/***************** - * Helpers -*****************/ - -/* Converts an allocation size in bytes to size in z3fold chunks */ -static int size_to_chunks(size_t size) -{ - return (size + CHUNK_SIZE - 1) >> CHUNK_SHIFT; -} - -#define for_each_unbuddied_list(_iter, _begin) \ - for ((_iter) =3D (_begin); (_iter) < NCHUNKS; (_iter)++) - -static inline struct z3fold_buddy_slots *alloc_slots(struct z3fold_pool *p= ool, - gfp_t gfp) -{ - struct z3fold_buddy_slots *slots =3D kmem_cache_zalloc(pool->c_handle, - gfp); - - if (slots) { - /* It will be freed separately in free_handle(). */ - kmemleak_not_leak(slots); - slots->pool =3D (unsigned long)pool; - rwlock_init(&slots->lock); - } - - return slots; -} - -static inline struct z3fold_pool *slots_to_pool(struct z3fold_buddy_slots = *s) -{ - return (struct z3fold_pool *)(s->pool & ~HANDLE_FLAG_MASK); -} - -static inline struct z3fold_buddy_slots *handle_to_slots(unsigned long han= dle) -{ - return (struct z3fold_buddy_slots *)(handle & ~(SLOTS_ALIGN - 1)); -} - -/* Lock a z3fold page */ -static inline void z3fold_page_lock(struct z3fold_header *zhdr) -{ - spin_lock(&zhdr->page_lock); -} - -/* Try to lock a z3fold page */ -static inline int z3fold_page_trylock(struct z3fold_header *zhdr) -{ - return spin_trylock(&zhdr->page_lock); -} - -/* Unlock a z3fold page */ -static inline void z3fold_page_unlock(struct z3fold_header *zhdr) -{ - spin_unlock(&zhdr->page_lock); -} - -/* return locked z3fold page if it's not headless */ -static inline struct z3fold_header *get_z3fold_header(unsigned long handle) -{ - struct z3fold_buddy_slots *slots; - struct z3fold_header *zhdr; - int locked =3D 0; - - if (!(handle & (1 << PAGE_HEADLESS))) { - slots =3D handle_to_slots(handle); - do { - unsigned long addr; - - read_lock(&slots->lock); - addr =3D *(unsigned long *)handle; - zhdr =3D (struct z3fold_header *)(addr & PAGE_MASK); - locked =3D z3fold_page_trylock(zhdr); - read_unlock(&slots->lock); - if (locked) { - struct page *page =3D virt_to_page(zhdr); - - if (!test_bit(PAGE_MIGRATED, &page->private)) - break; - z3fold_page_unlock(zhdr); - } - cpu_relax(); - } while (true); - } else { - zhdr =3D (struct z3fold_header *)(handle & PAGE_MASK); - } - - return zhdr; -} - -static inline void put_z3fold_header(struct z3fold_header *zhdr) -{ - struct page *page =3D virt_to_page(zhdr); - - if (!test_bit(PAGE_HEADLESS, &page->private)) - z3fold_page_unlock(zhdr); -} - -static inline void free_handle(unsigned long handle, struct z3fold_header = *zhdr) -{ - struct z3fold_buddy_slots *slots; - int i; - bool is_free; - - if (WARN_ON(*(unsigned long *)handle =3D=3D 0)) - return; - - slots =3D handle_to_slots(handle); - write_lock(&slots->lock); - *(unsigned long *)handle =3D 0; - - if (test_bit(HANDLES_NOFREE, &slots->pool)) { - write_unlock(&slots->lock); - return; /* simple case, nothing else to do */ - } - - if (zhdr->slots !=3D slots) - zhdr->foreign_handles--; - - is_free =3D true; - for (i =3D 0; i <=3D BUDDY_MASK; i++) { - if (slots->slot[i]) { - is_free =3D false; - break; - } - } - write_unlock(&slots->lock); - - if (is_free) { - struct z3fold_pool *pool =3D slots_to_pool(slots); - - if (zhdr->slots =3D=3D slots) - zhdr->slots =3D NULL; - kmem_cache_free(pool->c_handle, slots); - } -} - -/* Initializes the z3fold header of a newly allocated z3fold page */ -static struct z3fold_header *init_z3fold_page(struct page *page, bool head= less, - struct z3fold_pool *pool, gfp_t gfp) -{ - struct z3fold_header *zhdr =3D page_address(page); - struct z3fold_buddy_slots *slots; - - clear_bit(PAGE_HEADLESS, &page->private); - clear_bit(MIDDLE_CHUNK_MAPPED, &page->private); - clear_bit(NEEDS_COMPACTING, &page->private); - clear_bit(PAGE_STALE, &page->private); - clear_bit(PAGE_CLAIMED, &page->private); - clear_bit(PAGE_MIGRATED, &page->private); - if (headless) - return zhdr; - - slots =3D alloc_slots(pool, gfp); - if (!slots) - return NULL; - - memset(zhdr, 0, sizeof(*zhdr)); - spin_lock_init(&zhdr->page_lock); - kref_init(&zhdr->refcount); - zhdr->cpu =3D -1; - zhdr->slots =3D slots; - zhdr->pool =3D pool; - INIT_LIST_HEAD(&zhdr->buddy); - INIT_WORK(&zhdr->work, compact_page_work); - return zhdr; -} - -/* Resets the struct page fields and frees the page */ -static void free_z3fold_page(struct page *page, bool headless) -{ - if (!headless) { - lock_page(page); - __ClearPageMovable(page); - unlock_page(page); - } - __free_page(page); -} - -/* Helper function to build the index */ -static inline int __idx(struct z3fold_header *zhdr, enum buddy bud) -{ - return (bud + zhdr->first_num) & BUDDY_MASK; -} - -/* - * Encodes the handle of a particular buddy within a z3fold page. - * Zhdr->page_lock should be held as this function accesses first_num - * if bud !=3D HEADLESS. - */ -static unsigned long __encode_handle(struct z3fold_header *zhdr, - struct z3fold_buddy_slots *slots, - enum buddy bud) -{ - unsigned long h =3D (unsigned long)zhdr; - int idx =3D 0; - - /* - * For a headless page, its handle is its pointer with the extra - * PAGE_HEADLESS bit set - */ - if (bud =3D=3D HEADLESS) - return h | (1 << PAGE_HEADLESS); - - /* otherwise, return pointer to encoded handle */ - idx =3D __idx(zhdr, bud); - h +=3D idx; - if (bud =3D=3D LAST) - h |=3D (zhdr->last_chunks << BUDDY_SHIFT); - - write_lock(&slots->lock); - slots->slot[idx] =3D h; - write_unlock(&slots->lock); - return (unsigned long)&slots->slot[idx]; -} - -static unsigned long encode_handle(struct z3fold_header *zhdr, enum buddy = bud) -{ - return __encode_handle(zhdr, zhdr->slots, bud); -} - -/* only for LAST bud, returns zero otherwise */ -static unsigned short handle_to_chunks(unsigned long handle) -{ - struct z3fold_buddy_slots *slots =3D handle_to_slots(handle); - unsigned long addr; - - read_lock(&slots->lock); - addr =3D *(unsigned long *)handle; - read_unlock(&slots->lock); - return (addr & ~PAGE_MASK) >> BUDDY_SHIFT; -} - -/* - * (handle & BUDDY_MASK) < zhdr->first_num is possible in encode_handle - * but that doesn't matter. because the masking will result in the - * correct buddy number. - */ -static enum buddy handle_to_buddy(unsigned long handle) -{ - struct z3fold_header *zhdr; - struct z3fold_buddy_slots *slots =3D handle_to_slots(handle); - unsigned long addr; - - read_lock(&slots->lock); - WARN_ON(handle & (1 << PAGE_HEADLESS)); - addr =3D *(unsigned long *)handle; - read_unlock(&slots->lock); - zhdr =3D (struct z3fold_header *)(addr & PAGE_MASK); - return (addr - zhdr->first_num) & BUDDY_MASK; -} - -static inline struct z3fold_pool *zhdr_to_pool(struct z3fold_header *zhdr) -{ - return zhdr->pool; -} - -static void __release_z3fold_page(struct z3fold_header *zhdr, bool locked) -{ - struct page *page =3D virt_to_page(zhdr); - struct z3fold_pool *pool =3D zhdr_to_pool(zhdr); - - WARN_ON(!list_empty(&zhdr->buddy)); - set_bit(PAGE_STALE, &page->private); - clear_bit(NEEDS_COMPACTING, &page->private); - spin_lock(&pool->lock); - spin_unlock(&pool->lock); - - if (locked) - z3fold_page_unlock(zhdr); - - spin_lock(&pool->stale_lock); - list_add(&zhdr->buddy, &pool->stale); - queue_work(pool->release_wq, &pool->work); - spin_unlock(&pool->stale_lock); - - atomic64_dec(&pool->pages_nr); -} - -static void release_z3fold_page_locked(struct kref *ref) -{ - struct z3fold_header *zhdr =3D container_of(ref, struct z3fold_header, - refcount); - WARN_ON(z3fold_page_trylock(zhdr)); - __release_z3fold_page(zhdr, true); -} - -static void release_z3fold_page_locked_list(struct kref *ref) -{ - struct z3fold_header *zhdr =3D container_of(ref, struct z3fold_header, - refcount); - struct z3fold_pool *pool =3D zhdr_to_pool(zhdr); - - spin_lock(&pool->lock); - list_del_init(&zhdr->buddy); - spin_unlock(&pool->lock); - - WARN_ON(z3fold_page_trylock(zhdr)); - __release_z3fold_page(zhdr, true); -} - -static inline int put_z3fold_locked(struct z3fold_header *zhdr) -{ - return kref_put(&zhdr->refcount, release_z3fold_page_locked); -} - -static inline int put_z3fold_locked_list(struct z3fold_header *zhdr) -{ - return kref_put(&zhdr->refcount, release_z3fold_page_locked_list); -} - -static void free_pages_work(struct work_struct *w) -{ - struct z3fold_pool *pool =3D container_of(w, struct z3fold_pool, work); - - spin_lock(&pool->stale_lock); - while (!list_empty(&pool->stale)) { - struct z3fold_header *zhdr =3D list_first_entry(&pool->stale, - struct z3fold_header, buddy); - struct page *page =3D virt_to_page(zhdr); - - list_del(&zhdr->buddy); - if (WARN_ON(!test_bit(PAGE_STALE, &page->private))) - continue; - spin_unlock(&pool->stale_lock); - cancel_work_sync(&zhdr->work); - free_z3fold_page(page, false); - cond_resched(); - spin_lock(&pool->stale_lock); - } - spin_unlock(&pool->stale_lock); -} - -/* - * Returns the number of free chunks in a z3fold page. - * NB: can't be used with HEADLESS pages. - */ -static int num_free_chunks(struct z3fold_header *zhdr) -{ - int nfree; - /* - * If there is a middle object, pick up the bigger free space - * either before or after it. Otherwise just subtract the number - * of chunks occupied by the first and the last objects. - */ - if (zhdr->middle_chunks !=3D 0) { - int nfree_before =3D zhdr->first_chunks ? - 0 : zhdr->start_middle - ZHDR_CHUNKS; - int nfree_after =3D zhdr->last_chunks ? - 0 : TOTAL_CHUNKS - - (zhdr->start_middle + zhdr->middle_chunks); - nfree =3D max(nfree_before, nfree_after); - } else - nfree =3D NCHUNKS - zhdr->first_chunks - zhdr->last_chunks; - return nfree; -} - -/* Add to the appropriate unbuddied list */ -static inline void add_to_unbuddied(struct z3fold_pool *pool, - struct z3fold_header *zhdr) -{ - if (zhdr->first_chunks =3D=3D 0 || zhdr->last_chunks =3D=3D 0 || - zhdr->middle_chunks =3D=3D 0) { - struct list_head *unbuddied; - int freechunks =3D num_free_chunks(zhdr); - - migrate_disable(); - unbuddied =3D this_cpu_ptr(pool->unbuddied); - spin_lock(&pool->lock); - list_add(&zhdr->buddy, &unbuddied[freechunks]); - spin_unlock(&pool->lock); - zhdr->cpu =3D smp_processor_id(); - migrate_enable(); - } -} - -static inline enum buddy get_free_buddy(struct z3fold_header *zhdr, int ch= unks) -{ - enum buddy bud =3D HEADLESS; - - if (zhdr->middle_chunks) { - if (!zhdr->first_chunks && - chunks <=3D zhdr->start_middle - ZHDR_CHUNKS) - bud =3D FIRST; - else if (!zhdr->last_chunks) - bud =3D LAST; - } else { - if (!zhdr->first_chunks) - bud =3D FIRST; - else if (!zhdr->last_chunks) - bud =3D LAST; - else - bud =3D MIDDLE; - } - - return bud; -} - -static inline void *mchunk_memmove(struct z3fold_header *zhdr, - unsigned short dst_chunk) -{ - void *beg =3D zhdr; - return memmove(beg + (dst_chunk << CHUNK_SHIFT), - beg + (zhdr->start_middle << CHUNK_SHIFT), - zhdr->middle_chunks << CHUNK_SHIFT); -} - -static inline bool buddy_single(struct z3fold_header *zhdr) -{ - return !((zhdr->first_chunks && zhdr->middle_chunks) || - (zhdr->first_chunks && zhdr->last_chunks) || - (zhdr->middle_chunks && zhdr->last_chunks)); -} - -static struct z3fold_header *compact_single_buddy(struct z3fold_header *zh= dr) -{ - struct z3fold_pool *pool =3D zhdr_to_pool(zhdr); - void *p =3D zhdr; - unsigned long old_handle =3D 0; - size_t sz =3D 0; - struct z3fold_header *new_zhdr =3D NULL; - int first_idx =3D __idx(zhdr, FIRST); - int middle_idx =3D __idx(zhdr, MIDDLE); - int last_idx =3D __idx(zhdr, LAST); - unsigned short *moved_chunks =3D NULL; - - /* - * No need to protect slots here -- all the slots are "local" and - * the page lock is already taken - */ - if (zhdr->first_chunks && zhdr->slots->slot[first_idx]) { - p +=3D ZHDR_SIZE_ALIGNED; - sz =3D zhdr->first_chunks << CHUNK_SHIFT; - old_handle =3D (unsigned long)&zhdr->slots->slot[first_idx]; - moved_chunks =3D &zhdr->first_chunks; - } else if (zhdr->middle_chunks && zhdr->slots->slot[middle_idx]) { - p +=3D zhdr->start_middle << CHUNK_SHIFT; - sz =3D zhdr->middle_chunks << CHUNK_SHIFT; - old_handle =3D (unsigned long)&zhdr->slots->slot[middle_idx]; - moved_chunks =3D &zhdr->middle_chunks; - } else if (zhdr->last_chunks && zhdr->slots->slot[last_idx]) { - p +=3D PAGE_SIZE - (zhdr->last_chunks << CHUNK_SHIFT); - sz =3D zhdr->last_chunks << CHUNK_SHIFT; - old_handle =3D (unsigned long)&zhdr->slots->slot[last_idx]; - moved_chunks =3D &zhdr->last_chunks; - } - - if (sz > 0) { - enum buddy new_bud =3D HEADLESS; - short chunks =3D size_to_chunks(sz); - void *q; - - new_zhdr =3D __z3fold_alloc(pool, sz, false); - if (!new_zhdr) - return NULL; - - if (WARN_ON(new_zhdr =3D=3D zhdr)) - goto out_fail; - - new_bud =3D get_free_buddy(new_zhdr, chunks); - q =3D new_zhdr; - switch (new_bud) { - case FIRST: - new_zhdr->first_chunks =3D chunks; - q +=3D ZHDR_SIZE_ALIGNED; - break; - case MIDDLE: - new_zhdr->middle_chunks =3D chunks; - new_zhdr->start_middle =3D - new_zhdr->first_chunks + ZHDR_CHUNKS; - q +=3D new_zhdr->start_middle << CHUNK_SHIFT; - break; - case LAST: - new_zhdr->last_chunks =3D chunks; - q +=3D PAGE_SIZE - (new_zhdr->last_chunks << CHUNK_SHIFT); - break; - default: - goto out_fail; - } - new_zhdr->foreign_handles++; - memcpy(q, p, sz); - write_lock(&zhdr->slots->lock); - *(unsigned long *)old_handle =3D (unsigned long)new_zhdr + - __idx(new_zhdr, new_bud); - if (new_bud =3D=3D LAST) - *(unsigned long *)old_handle |=3D - (new_zhdr->last_chunks << BUDDY_SHIFT); - write_unlock(&zhdr->slots->lock); - add_to_unbuddied(pool, new_zhdr); - z3fold_page_unlock(new_zhdr); - - *moved_chunks =3D 0; - } - - return new_zhdr; - -out_fail: - if (new_zhdr && !put_z3fold_locked(new_zhdr)) { - add_to_unbuddied(pool, new_zhdr); - z3fold_page_unlock(new_zhdr); - } - return NULL; - -} - -#define BIG_CHUNK_GAP 3 -/* Has to be called with lock held */ -static int z3fold_compact_page(struct z3fold_header *zhdr) -{ - struct page *page =3D virt_to_page(zhdr); - - if (test_bit(MIDDLE_CHUNK_MAPPED, &page->private)) - return 0; /* can't move middle chunk, it's used */ - - if (unlikely(PageIsolated(page))) - return 0; - - if (zhdr->middle_chunks =3D=3D 0) - return 0; /* nothing to compact */ - - if (zhdr->first_chunks =3D=3D 0 && zhdr->last_chunks =3D=3D 0) { - /* move to the beginning */ - mchunk_memmove(zhdr, ZHDR_CHUNKS); - zhdr->first_chunks =3D zhdr->middle_chunks; - zhdr->middle_chunks =3D 0; - zhdr->start_middle =3D 0; - zhdr->first_num++; - return 1; - } - - /* - * moving data is expensive, so let's only do that if - * there's substantial gain (at least BIG_CHUNK_GAP chunks) - */ - if (zhdr->first_chunks !=3D 0 && zhdr->last_chunks =3D=3D 0 && - zhdr->start_middle - (zhdr->first_chunks + ZHDR_CHUNKS) >=3D - BIG_CHUNK_GAP) { - mchunk_memmove(zhdr, zhdr->first_chunks + ZHDR_CHUNKS); - zhdr->start_middle =3D zhdr->first_chunks + ZHDR_CHUNKS; - return 1; - } else if (zhdr->last_chunks !=3D 0 && zhdr->first_chunks =3D=3D 0 && - TOTAL_CHUNKS - (zhdr->last_chunks + zhdr->start_middle - + zhdr->middle_chunks) >=3D - BIG_CHUNK_GAP) { - unsigned short new_start =3D TOTAL_CHUNKS - zhdr->last_chunks - - zhdr->middle_chunks; - mchunk_memmove(zhdr, new_start); - zhdr->start_middle =3D new_start; - return 1; - } - - return 0; -} - -static void do_compact_page(struct z3fold_header *zhdr, bool locked) -{ - struct z3fold_pool *pool =3D zhdr_to_pool(zhdr); - struct page *page; - - page =3D virt_to_page(zhdr); - if (locked) - WARN_ON(z3fold_page_trylock(zhdr)); - else - z3fold_page_lock(zhdr); - if (WARN_ON(!test_and_clear_bit(NEEDS_COMPACTING, &page->private))) { - z3fold_page_unlock(zhdr); - return; - } - spin_lock(&pool->lock); - list_del_init(&zhdr->buddy); - spin_unlock(&pool->lock); - - if (put_z3fold_locked(zhdr)) - return; - - if (test_bit(PAGE_STALE, &page->private) || - test_and_set_bit(PAGE_CLAIMED, &page->private)) { - z3fold_page_unlock(zhdr); - return; - } - - if (!zhdr->foreign_handles && buddy_single(zhdr) && - zhdr->mapped_count =3D=3D 0 && compact_single_buddy(zhdr)) { - if (!put_z3fold_locked(zhdr)) { - clear_bit(PAGE_CLAIMED, &page->private); - z3fold_page_unlock(zhdr); - } - return; - } - - z3fold_compact_page(zhdr); - add_to_unbuddied(pool, zhdr); - clear_bit(PAGE_CLAIMED, &page->private); - z3fold_page_unlock(zhdr); -} - -static void compact_page_work(struct work_struct *w) -{ - struct z3fold_header *zhdr =3D container_of(w, struct z3fold_header, - work); - - do_compact_page(zhdr, false); -} - -/* returns _locked_ z3fold page header or NULL */ -static inline struct z3fold_header *__z3fold_alloc(struct z3fold_pool *poo= l, - size_t size, bool can_sleep) -{ - struct z3fold_header *zhdr =3D NULL; - struct page *page; - struct list_head *unbuddied; - int chunks =3D size_to_chunks(size), i; - -lookup: - migrate_disable(); - /* First, try to find an unbuddied z3fold page. */ - unbuddied =3D this_cpu_ptr(pool->unbuddied); - for_each_unbuddied_list(i, chunks) { - struct list_head *l =3D &unbuddied[i]; - - zhdr =3D list_first_entry_or_null(READ_ONCE(l), - struct z3fold_header, buddy); - - if (!zhdr) - continue; - - /* Re-check under lock. */ - spin_lock(&pool->lock); - if (unlikely(zhdr !=3D list_first_entry(READ_ONCE(l), - struct z3fold_header, buddy)) || - !z3fold_page_trylock(zhdr)) { - spin_unlock(&pool->lock); - zhdr =3D NULL; - migrate_enable(); - if (can_sleep) - cond_resched(); - goto lookup; - } - list_del_init(&zhdr->buddy); - zhdr->cpu =3D -1; - spin_unlock(&pool->lock); - - page =3D virt_to_page(zhdr); - if (test_bit(NEEDS_COMPACTING, &page->private) || - test_bit(PAGE_CLAIMED, &page->private)) { - z3fold_page_unlock(zhdr); - zhdr =3D NULL; - migrate_enable(); - if (can_sleep) - cond_resched(); - goto lookup; - } - - /* - * this page could not be removed from its unbuddied - * list while pool lock was held, and then we've taken - * page lock so kref_put could not be called before - * we got here, so it's safe to just call kref_get() - */ - kref_get(&zhdr->refcount); - break; - } - migrate_enable(); - - if (!zhdr) { - int cpu; - - /* look for _exact_ match on other cpus' lists */ - for_each_online_cpu(cpu) { - struct list_head *l; - - unbuddied =3D per_cpu_ptr(pool->unbuddied, cpu); - spin_lock(&pool->lock); - l =3D &unbuddied[chunks]; - - zhdr =3D list_first_entry_or_null(READ_ONCE(l), - struct z3fold_header, buddy); - - if (!zhdr || !z3fold_page_trylock(zhdr)) { - spin_unlock(&pool->lock); - zhdr =3D NULL; - continue; - } - list_del_init(&zhdr->buddy); - zhdr->cpu =3D -1; - spin_unlock(&pool->lock); - - page =3D virt_to_page(zhdr); - if (test_bit(NEEDS_COMPACTING, &page->private) || - test_bit(PAGE_CLAIMED, &page->private)) { - z3fold_page_unlock(zhdr); - zhdr =3D NULL; - if (can_sleep) - cond_resched(); - continue; - } - kref_get(&zhdr->refcount); - break; - } - } - - if (zhdr && !zhdr->slots) { - zhdr->slots =3D alloc_slots(pool, GFP_ATOMIC); - if (!zhdr->slots) - goto out_fail; - } - return zhdr; - -out_fail: - if (!put_z3fold_locked(zhdr)) { - add_to_unbuddied(pool, zhdr); - z3fold_page_unlock(zhdr); - } - return NULL; -} - -/* - * API Functions - */ - -/** - * z3fold_create_pool() - create a new z3fold pool - * @name: pool name - * @gfp: gfp flags when allocating the z3fold pool structure - * - * Return: pointer to the new z3fold pool or NULL if the metadata allocati= on - * failed. - */ -static struct z3fold_pool *z3fold_create_pool(const char *name, gfp_t gfp) -{ - struct z3fold_pool *pool =3D NULL; - int i, cpu; - - pool =3D kzalloc(sizeof(struct z3fold_pool), gfp); - if (!pool) - goto out; - pool->c_handle =3D kmem_cache_create("z3fold_handle", - sizeof(struct z3fold_buddy_slots), - SLOTS_ALIGN, 0, NULL); - if (!pool->c_handle) - goto out_c; - spin_lock_init(&pool->lock); - spin_lock_init(&pool->stale_lock); - pool->unbuddied =3D __alloc_percpu(sizeof(struct list_head) * NCHUNKS, - __alignof__(struct list_head)); - if (!pool->unbuddied) - goto out_pool; - for_each_possible_cpu(cpu) { - struct list_head *unbuddied =3D - per_cpu_ptr(pool->unbuddied, cpu); - for_each_unbuddied_list(i, 0) - INIT_LIST_HEAD(&unbuddied[i]); - } - INIT_LIST_HEAD(&pool->stale); - atomic64_set(&pool->pages_nr, 0); - pool->name =3D name; - pool->compact_wq =3D create_singlethread_workqueue(pool->name); - if (!pool->compact_wq) - goto out_unbuddied; - pool->release_wq =3D create_singlethread_workqueue(pool->name); - if (!pool->release_wq) - goto out_wq; - INIT_WORK(&pool->work, free_pages_work); - return pool; - -out_wq: - destroy_workqueue(pool->compact_wq); -out_unbuddied: - free_percpu(pool->unbuddied); -out_pool: - kmem_cache_destroy(pool->c_handle); -out_c: - kfree(pool); -out: - return NULL; -} - -/** - * z3fold_destroy_pool() - destroys an existing z3fold pool - * @pool: the z3fold pool to be destroyed - * - * The pool should be emptied before this function is called. - */ -static void z3fold_destroy_pool(struct z3fold_pool *pool) -{ - kmem_cache_destroy(pool->c_handle); - - /* - * We need to destroy pool->compact_wq before pool->release_wq, - * as any pending work on pool->compact_wq will call - * queue_work(pool->release_wq, &pool->work). - * - * There are still outstanding pages until both workqueues are drained, - * so we cannot unregister migration until then. - */ - - destroy_workqueue(pool->compact_wq); - destroy_workqueue(pool->release_wq); - free_percpu(pool->unbuddied); - kfree(pool); -} - -static const struct movable_operations z3fold_mops; - -/** - * z3fold_alloc() - allocates a region of a given size - * @pool: z3fold pool from which to allocate - * @size: size in bytes of the desired allocation - * @gfp: gfp flags used if the pool needs to grow - * @handle: handle of the new allocation - * - * This function will attempt to find a free region in the pool large enou= gh to - * satisfy the allocation request. A search of the unbuddied lists is - * performed first. If no suitable free region is found, then a new page is - * allocated and added to the pool to satisfy the request. - * - * Return: 0 if success and handle is set, otherwise -EINVAL if the size or - * gfp arguments are invalid or -ENOMEM if the pool was unable to allocate - * a new page. - */ -static int z3fold_alloc(struct z3fold_pool *pool, size_t size, gfp_t gfp, - unsigned long *handle) -{ - int chunks =3D size_to_chunks(size); - struct z3fold_header *zhdr =3D NULL; - struct page *page =3D NULL; - enum buddy bud; - bool can_sleep =3D gfpflags_allow_blocking(gfp); - - if (!size || (gfp & __GFP_HIGHMEM)) - return -EINVAL; - - if (size > PAGE_SIZE) - return -ENOSPC; - - if (size > PAGE_SIZE - ZHDR_SIZE_ALIGNED - CHUNK_SIZE) - bud =3D HEADLESS; - else { -retry: - zhdr =3D __z3fold_alloc(pool, size, can_sleep); - if (zhdr) { - bud =3D get_free_buddy(zhdr, chunks); - if (bud =3D=3D HEADLESS) { - if (!put_z3fold_locked(zhdr)) - z3fold_page_unlock(zhdr); - pr_err("No free chunks in unbuddied\n"); - WARN_ON(1); - goto retry; - } - page =3D virt_to_page(zhdr); - goto found; - } - bud =3D FIRST; - } - - page =3D alloc_page(gfp); - if (!page) - return -ENOMEM; - - zhdr =3D init_z3fold_page(page, bud =3D=3D HEADLESS, pool, gfp); - if (!zhdr) { - __free_page(page); - return -ENOMEM; - } - atomic64_inc(&pool->pages_nr); - - if (bud =3D=3D HEADLESS) { - set_bit(PAGE_HEADLESS, &page->private); - goto headless; - } - if (can_sleep) { - lock_page(page); - __SetPageMovable(page, &z3fold_mops); - unlock_page(page); - } else { - WARN_ON(!trylock_page(page)); - __SetPageMovable(page, &z3fold_mops); - unlock_page(page); - } - z3fold_page_lock(zhdr); - -found: - if (bud =3D=3D FIRST) - zhdr->first_chunks =3D chunks; - else if (bud =3D=3D LAST) - zhdr->last_chunks =3D chunks; - else { - zhdr->middle_chunks =3D chunks; - zhdr->start_middle =3D zhdr->first_chunks + ZHDR_CHUNKS; - } - add_to_unbuddied(pool, zhdr); - -headless: - spin_lock(&pool->lock); - *handle =3D encode_handle(zhdr, bud); - spin_unlock(&pool->lock); - if (bud !=3D HEADLESS) - z3fold_page_unlock(zhdr); - - return 0; -} - -/** - * z3fold_free() - frees the allocation associated with the given handle - * @pool: pool in which the allocation resided - * @handle: handle associated with the allocation returned by z3fold_alloc= () - * - * In the case that the z3fold page in which the allocation resides is und= er - * reclaim, as indicated by the PAGE_CLAIMED flag being set, this function - * only sets the first|middle|last_chunks to 0. The page is actually freed - * once all buddies are evicted (see z3fold_reclaim_page() below). - */ -static void z3fold_free(struct z3fold_pool *pool, unsigned long handle) -{ - struct z3fold_header *zhdr; - struct page *page; - enum buddy bud; - bool page_claimed; - - zhdr =3D get_z3fold_header(handle); - page =3D virt_to_page(zhdr); - page_claimed =3D test_and_set_bit(PAGE_CLAIMED, &page->private); - - if (test_bit(PAGE_HEADLESS, &page->private)) { - /* if a headless page is under reclaim, just leave. - * NB: we use test_and_set_bit for a reason: if the bit - * has not been set before, we release this page - * immediately so we don't care about its value any more. - */ - if (!page_claimed) { - put_z3fold_header(zhdr); - free_z3fold_page(page, true); - atomic64_dec(&pool->pages_nr); - } - return; - } - - /* Non-headless case */ - bud =3D handle_to_buddy(handle); - - switch (bud) { - case FIRST: - zhdr->first_chunks =3D 0; - break; - case MIDDLE: - zhdr->middle_chunks =3D 0; - break; - case LAST: - zhdr->last_chunks =3D 0; - break; - default: - pr_err("%s: unknown bud %d\n", __func__, bud); - WARN_ON(1); - put_z3fold_header(zhdr); - return; - } - - if (!page_claimed) - free_handle(handle, zhdr); - if (put_z3fold_locked_list(zhdr)) - return; - if (page_claimed) { - /* the page has not been claimed by us */ - put_z3fold_header(zhdr); - return; - } - if (test_and_set_bit(NEEDS_COMPACTING, &page->private)) { - clear_bit(PAGE_CLAIMED, &page->private); - put_z3fold_header(zhdr); - return; - } - if (zhdr->cpu < 0 || !cpu_online(zhdr->cpu)) { - zhdr->cpu =3D -1; - kref_get(&zhdr->refcount); - clear_bit(PAGE_CLAIMED, &page->private); - do_compact_page(zhdr, true); - return; - } - kref_get(&zhdr->refcount); - clear_bit(PAGE_CLAIMED, &page->private); - queue_work_on(zhdr->cpu, pool->compact_wq, &zhdr->work); - put_z3fold_header(zhdr); -} - -/** - * z3fold_map() - maps the allocation associated with the given handle - * @pool: pool in which the allocation resides - * @handle: handle associated with the allocation to be mapped - * - * Extracts the buddy number from handle and constructs the pointer to the - * correct starting chunk within the page. - * - * Returns: a pointer to the mapped allocation - */ -static void *z3fold_map(struct z3fold_pool *pool, unsigned long handle) -{ - struct z3fold_header *zhdr; - struct page *page; - void *addr; - enum buddy buddy; - - zhdr =3D get_z3fold_header(handle); - addr =3D zhdr; - page =3D virt_to_page(zhdr); - - if (test_bit(PAGE_HEADLESS, &page->private)) - goto out; - - buddy =3D handle_to_buddy(handle); - switch (buddy) { - case FIRST: - addr +=3D ZHDR_SIZE_ALIGNED; - break; - case MIDDLE: - addr +=3D zhdr->start_middle << CHUNK_SHIFT; - set_bit(MIDDLE_CHUNK_MAPPED, &page->private); - break; - case LAST: - addr +=3D PAGE_SIZE - (handle_to_chunks(handle) << CHUNK_SHIFT); - break; - default: - pr_err("unknown buddy id %d\n", buddy); - WARN_ON(1); - addr =3D NULL; - break; - } - - if (addr) - zhdr->mapped_count++; -out: - put_z3fold_header(zhdr); - return addr; -} - -/** - * z3fold_unmap() - unmaps the allocation associated with the given handle - * @pool: pool in which the allocation resides - * @handle: handle associated with the allocation to be unmapped - */ -static void z3fold_unmap(struct z3fold_pool *pool, unsigned long handle) -{ - struct z3fold_header *zhdr; - struct page *page; - enum buddy buddy; - - zhdr =3D get_z3fold_header(handle); - page =3D virt_to_page(zhdr); - - if (test_bit(PAGE_HEADLESS, &page->private)) - return; - - buddy =3D handle_to_buddy(handle); - if (buddy =3D=3D MIDDLE) - clear_bit(MIDDLE_CHUNK_MAPPED, &page->private); - zhdr->mapped_count--; - put_z3fold_header(zhdr); -} - -/** - * z3fold_get_pool_pages() - gets the z3fold pool size in pages - * @pool: pool whose size is being queried - * - * Returns: size in pages of the given pool. - */ -static u64 z3fold_get_pool_pages(struct z3fold_pool *pool) -{ - return atomic64_read(&pool->pages_nr); -} - -static bool z3fold_page_isolate(struct page *page, isolate_mode_t mode) -{ - struct z3fold_header *zhdr; - struct z3fold_pool *pool; - - VM_BUG_ON_PAGE(PageIsolated(page), page); - - if (test_bit(PAGE_HEADLESS, &page->private)) - return false; - - zhdr =3D page_address(page); - z3fold_page_lock(zhdr); - if (test_bit(NEEDS_COMPACTING, &page->private) || - test_bit(PAGE_STALE, &page->private)) - goto out; - - if (zhdr->mapped_count !=3D 0 || zhdr->foreign_handles !=3D 0) - goto out; - - if (test_and_set_bit(PAGE_CLAIMED, &page->private)) - goto out; - pool =3D zhdr_to_pool(zhdr); - spin_lock(&pool->lock); - if (!list_empty(&zhdr->buddy)) - list_del_init(&zhdr->buddy); - spin_unlock(&pool->lock); - - kref_get(&zhdr->refcount); - z3fold_page_unlock(zhdr); - return true; - -out: - z3fold_page_unlock(zhdr); - return false; -} - -static int z3fold_page_migrate(struct page *newpage, struct page *page, - enum migrate_mode mode) -{ - struct z3fold_header *zhdr, *new_zhdr; - struct z3fold_pool *pool; - - VM_BUG_ON_PAGE(!PageIsolated(page), page); - VM_BUG_ON_PAGE(!test_bit(PAGE_CLAIMED, &page->private), page); - VM_BUG_ON_PAGE(!PageLocked(newpage), newpage); - - zhdr =3D page_address(page); - pool =3D zhdr_to_pool(zhdr); - - if (!z3fold_page_trylock(zhdr)) - return -EAGAIN; - if (zhdr->mapped_count !=3D 0 || zhdr->foreign_handles !=3D 0) { - clear_bit(PAGE_CLAIMED, &page->private); - z3fold_page_unlock(zhdr); - return -EBUSY; - } - if (work_pending(&zhdr->work)) { - z3fold_page_unlock(zhdr); - return -EAGAIN; - } - new_zhdr =3D page_address(newpage); - memcpy(new_zhdr, zhdr, PAGE_SIZE); - newpage->private =3D page->private; - set_bit(PAGE_MIGRATED, &page->private); - z3fold_page_unlock(zhdr); - spin_lock_init(&new_zhdr->page_lock); - INIT_WORK(&new_zhdr->work, compact_page_work); - /* - * z3fold_page_isolate() ensures that new_zhdr->buddy is empty, - * so we only have to reinitialize it. - */ - INIT_LIST_HEAD(&new_zhdr->buddy); - __ClearPageMovable(page); - - get_page(newpage); - z3fold_page_lock(new_zhdr); - if (new_zhdr->first_chunks) - encode_handle(new_zhdr, FIRST); - if (new_zhdr->last_chunks) - encode_handle(new_zhdr, LAST); - if (new_zhdr->middle_chunks) - encode_handle(new_zhdr, MIDDLE); - set_bit(NEEDS_COMPACTING, &newpage->private); - new_zhdr->cpu =3D smp_processor_id(); - __SetPageMovable(newpage, &z3fold_mops); - z3fold_page_unlock(new_zhdr); - - queue_work_on(new_zhdr->cpu, pool->compact_wq, &new_zhdr->work); - - /* PAGE_CLAIMED and PAGE_MIGRATED are cleared now. */ - page->private =3D 0; - put_page(page); - return 0; -} - -static void z3fold_page_putback(struct page *page) -{ - struct z3fold_header *zhdr; - struct z3fold_pool *pool; - - zhdr =3D page_address(page); - pool =3D zhdr_to_pool(zhdr); - - z3fold_page_lock(zhdr); - if (!list_empty(&zhdr->buddy)) - list_del_init(&zhdr->buddy); - INIT_LIST_HEAD(&page->lru); - if (put_z3fold_locked(zhdr)) - return; - if (list_empty(&zhdr->buddy)) - add_to_unbuddied(pool, zhdr); - clear_bit(PAGE_CLAIMED, &page->private); - z3fold_page_unlock(zhdr); -} - -static const struct movable_operations z3fold_mops =3D { - .isolate_page =3D z3fold_page_isolate, - .migrate_page =3D z3fold_page_migrate, - .putback_page =3D z3fold_page_putback, -}; - -/***************** - * zpool - ****************/ - -static void *z3fold_zpool_create(const char *name, gfp_t gfp) -{ - return z3fold_create_pool(name, gfp); -} - -static void z3fold_zpool_destroy(void *pool) -{ - z3fold_destroy_pool(pool); -} - -static int z3fold_zpool_malloc(void *pool, size_t size, gfp_t gfp, - unsigned long *handle) -{ - return z3fold_alloc(pool, size, gfp, handle); -} -static void z3fold_zpool_free(void *pool, unsigned long handle) -{ - z3fold_free(pool, handle); -} - -static void *z3fold_zpool_map(void *pool, unsigned long handle, - enum zpool_mapmode mm) -{ - return z3fold_map(pool, handle); -} -static void z3fold_zpool_unmap(void *pool, unsigned long handle) -{ - z3fold_unmap(pool, handle); -} - -static u64 z3fold_zpool_total_pages(void *pool) -{ - return z3fold_get_pool_pages(pool); -} - -static struct zpool_driver z3fold_zpool_driver =3D { - .type =3D "z3fold", - .sleep_mapped =3D true, - .owner =3D THIS_MODULE, - .create =3D z3fold_zpool_create, - .destroy =3D z3fold_zpool_destroy, - .malloc =3D z3fold_zpool_malloc, - .free =3D z3fold_zpool_free, - .map =3D z3fold_zpool_map, - .unmap =3D z3fold_zpool_unmap, - .total_pages =3D z3fold_zpool_total_pages, -}; - -MODULE_ALIAS("zpool-z3fold"); - -static int __init init_z3fold(void) -{ - /* - * Make sure the z3fold header is not larger than the page size and - * there has remaining spaces for its buddy. - */ - BUILD_BUG_ON(ZHDR_SIZE_ALIGNED > PAGE_SIZE - CHUNK_SIZE); - zpool_register_driver(&z3fold_zpool_driver); - - return 0; -} - -static void __exit exit_z3fold(void) -{ - zpool_unregister_driver(&z3fold_zpool_driver); -} - -module_init(init_z3fold); -module_exit(exit_z3fold); - -MODULE_LICENSE("GPL"); -MODULE_AUTHOR("Vitaly Wool "); -MODULE_DESCRIPTION("3-Fold Allocator for Compressed Pages"); --=20 2.48.1.262.g85cc9f2d1e-goog