From nobody Mon Feb 9 02:42:15 2026 Received: from mail-pj1-f46.google.com (mail-pj1-f46.google.com [209.85.216.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 704F016E863 for ; Fri, 31 Jan 2025 09:07:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.46 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738314432; cv=none; b=EnB0RBILKtu45EarajLpFjHxUFHv+EwwgXXR2wzWi//j2Ls0EcU6uB0mOCG6JoizkTE9ygjPIrUol5NN6JZxiKEZmNLUaN79xgc2NjO0/uzukXjLvse/JbvtY9tgwnrkXceW3WwpMvyZmTklhJc60qtqBsuZn/6U4zoDbkT/Sg4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738314432; c=relaxed/simple; bh=kRrj9VavLNM5dJKp/VhjyPZ+ot3462OfKKSaaaG31CE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=LZO7J2YGlg+uSOwK2XKPtKVai6ifhWgrJk8b9f00IvUNYEueh+Kx8rkBatZo+ei2XC/NNmhlAFngq49owSW23cisTKlM4TVqmnS4/zWmLIG0Kkd2qlvQeOx6IIzVEPDKFyWWj4H8nZavi4fE90RscP1k+M+HV6DA8pvXDPfleFc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=e4gEzeVO; arc=none smtp.client-ip=209.85.216.46 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="e4gEzeVO" Received: by mail-pj1-f46.google.com with SMTP id 98e67ed59e1d1-2ee786b3277so2312367a91.1 for ; Fri, 31 Jan 2025 01:07:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1738314429; x=1738919229; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=qaFN5XJ8A1km1G4xPnULAZdvrmEAmo6pRkSTLZcPqY0=; b=e4gEzeVO3wCdOMPRt5a1TRg9h6goXxvnCabwHpJnPH9AZXJG5Z7HpQgOsnnvORPoHk ndtBtoNmySI/okvt+sTKbmdsMA0SFPi2PaAP7r1JgAGxMp05kX+nUwyO7YwU4Q++YWs2 kh60Ev40BLfWQ5w46qsNl5Kww9jAuxFoPFjl0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738314429; x=1738919229; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=qaFN5XJ8A1km1G4xPnULAZdvrmEAmo6pRkSTLZcPqY0=; b=UhudjdjbIrmsbceU+TJbUiHvvQ482dk/8RFbB78PvTHavyFsrHptD/sM9VXgWCpFza HVXZE7YugA43xJ9KcZCx+GxaCmv2aWmv6B4dA93IoV5UrUafg9nWbmYHVQeL46sWmQTR dmsW+xgDlxlbDapqdb5P/pNuevZI4thiQzjVLrgwTZscn6DvSoax0orAIJ6C6fWI+TEA 52arcHdE66/1xYMsqLaiUvLmWab/tj4xD/AZOzYpeqZow4EK5G5a5qgW6CArMDKxyuKO s5g5qqXR0Fd9cfvfCTmiMPDtl6yBux3AZJhEZgqSmiD2o7j6z6Z6Ll5LoNw5ZvL8dI4N aPVQ== X-Forwarded-Encrypted: i=1; AJvYcCX4zP/QVMFCc6/JfWFK/LCw4p//Omwp4RWZI53QYDm2curu0y5d/NHKI6el41EMLA8ooFS0Q+Qc56mO5vo=@vger.kernel.org X-Gm-Message-State: AOJu0YxjNOyRqdxYTZ937dQvq7p6n2IUObveDXY3v4bplRxnBMJP7Rbu x8BsgHWfAAgNcmBJw8m29vdzmO7EKAieX7dgrthdUESPRXeCW1k1IB7hlVPDxg== X-Gm-Gg: ASbGncuPGS+EOIi/toGWU00OTU+tdzWxSJh0690g9ksz2dDLL40Vs1w/82HLEx78pEF 0zzxioDvX5KBTwCAi7vArlSZNO+Qy/lJwkKqSRuRNpsxQ1i+lrdYNiUm8ggLjFvYz96NoeTgtsm 4q53/v69bu5P/33hiGHhmbg/xQ1RBFKLMWHkaXrKa24AVFFdFxxPvXAKHYig/YK69dReSPUeToz nNrSHiBzd52PJcOG47THEZZQ94bd7Ru/nD+mGQEnkmSQL7quMNIymRl6hGby+hm9xYpXLHjtDHC p2soxetJCnfuRqbLrw== X-Google-Smtp-Source: AGHT+IGIXN+n6f5l+y68Non+b6GFZDbdRRf5AaVlS7GSpPKO4QWiW4YKeYlyUraz6wfdYS57hfAKuA== X-Received: by 2002:a17:90a:d890:b0:2ee:e317:69ab with SMTP id 98e67ed59e1d1-2f83aa65ed2mr19463617a91.0.1738314428603; Fri, 31 Jan 2025 01:07:08 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:c752:be9d:3368:16fa]) by smtp.gmail.com with UTF8SMTPSA id 98e67ed59e1d1-2f83bf93e97sm5869177a91.31.2025.01.31.01.07.06 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 31 Jan 2025 01:07:08 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv4 01/17] zram: switch to non-atomic entry locking Date: Fri, 31 Jan 2025 18:06:00 +0900 Message-ID: <20250131090658.3386285-2-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.362.g079036d154-goog In-Reply-To: <20250131090658.3386285-1-senozhatsky@chromium.org> References: <20250131090658.3386285-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Concurrent modifications of meta table entries is now handled by per-entry spin-lock. This has a number of shortcomings. First, this imposes atomic requirements on compression backends. zram can call both zcomp_compress() and zcomp_decompress() under entry spin-lock, which implies that we can use only compression algorithms that don't schedule/sleep/wait during compression and decompression. This, for instance, makes it impossible to use some of the ASYNC compression algorithms (H/W compression, etc.) implementations. Second, this can potentially trigger watchdogs. For example, entry re-compression with secondary algorithms is performed under entry spin-lock. Given that we chain secondary compression algorithms and that some of them can be configured for best compression ratio (and worst compression speed) zram can stay under spin-lock for quite some time. Do not use per-entry spin-locks and instead convert it to an atomic_t variable which open codes reader-writer type of lock. This permits preemption from slot_lock section, also reduces the sizeof() zram entry when lockdep is enabled. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 126 ++++++++++++++++++++-------------- drivers/block/zram/zram_drv.h | 6 +- 2 files changed, 79 insertions(+), 53 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 9f5020b077c5..1c2df2341704 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -58,19 +58,50 @@ static void zram_free_page(struct zram *zram, size_t in= dex); static int zram_read_from_zspool(struct zram *zram, struct page *page, u32 index); =20 -static int zram_slot_trylock(struct zram *zram, u32 index) +static bool zram_slot_try_write_lock(struct zram *zram, u32 index) { - return spin_trylock(&zram->table[index].lock); + atomic_t *lock =3D &zram->table[index].lock; + int old =3D ZRAM_ENTRY_UNLOCKED; + + return atomic_try_cmpxchg(lock, &old, ZRAM_ENTRY_WRLOCKED); +} + +static void zram_slot_write_lock(struct zram *zram, u32 index) +{ + atomic_t *lock =3D &zram->table[index].lock; + int old =3D atomic_read(lock); + + do { + if (old !=3D ZRAM_ENTRY_UNLOCKED) { + cond_resched(); + old =3D atomic_read(lock); + continue; + } + } while (!atomic_try_cmpxchg(lock, &old, ZRAM_ENTRY_WRLOCKED)); +} + +static void zram_slot_write_unlock(struct zram *zram, u32 index) +{ + atomic_set(&zram->table[index].lock, ZRAM_ENTRY_UNLOCKED); } =20 -static void zram_slot_lock(struct zram *zram, u32 index) +static void zram_slot_read_lock(struct zram *zram, u32 index) { - spin_lock(&zram->table[index].lock); + atomic_t *lock =3D &zram->table[index].lock; + int old =3D atomic_read(lock); + + do { + if (old =3D=3D ZRAM_ENTRY_WRLOCKED) { + cond_resched(); + old =3D atomic_read(lock); + continue; + } + } while (!atomic_try_cmpxchg(lock, &old, old + 1)); } =20 -static void zram_slot_unlock(struct zram *zram, u32 index) +static void zram_slot_read_unlock(struct zram *zram, u32 index) { - spin_unlock(&zram->table[index].lock); + atomic_dec(&zram->table[index].lock); } =20 static inline bool init_done(struct zram *zram) @@ -93,7 +124,6 @@ static void zram_set_handle(struct zram *zram, u32 index= , unsigned long handle) zram->table[index].handle =3D handle; } =20 -/* flag operations require table entry bit_spin_lock() being held */ static bool zram_test_flag(struct zram *zram, u32 index, enum zram_pageflags flag) { @@ -229,9 +259,9 @@ static void release_pp_slot(struct zram *zram, struct z= ram_pp_slot *pps) { list_del_init(&pps->entry); =20 - zram_slot_lock(zram, pps->index); + zram_slot_write_lock(zram, pps->index); zram_clear_flag(zram, pps->index, ZRAM_PP_SLOT); - zram_slot_unlock(zram, pps->index); + zram_slot_write_unlock(zram, pps->index); =20 kfree(pps); } @@ -394,11 +424,11 @@ static void mark_idle(struct zram *zram, ktime_t cuto= ff) * * And ZRAM_WB slots simply cannot be ZRAM_IDLE. */ - zram_slot_lock(zram, index); + zram_slot_write_lock(zram, index); if (!zram_allocated(zram, index) || zram_test_flag(zram, index, ZRAM_WB) || zram_test_flag(zram, index, ZRAM_SAME)) { - zram_slot_unlock(zram, index); + zram_slot_write_unlock(zram, index); continue; } =20 @@ -410,7 +440,7 @@ static void mark_idle(struct zram *zram, ktime_t cutoff) zram_set_flag(zram, index, ZRAM_IDLE); else zram_clear_flag(zram, index, ZRAM_IDLE); - zram_slot_unlock(zram, index); + zram_slot_write_unlock(zram, index); } } =20 @@ -709,7 +739,7 @@ static int scan_slots_for_writeback(struct zram *zram, = u32 mode, =20 INIT_LIST_HEAD(&pps->entry); =20 - zram_slot_lock(zram, index); + zram_slot_write_lock(zram, index); if (!zram_allocated(zram, index)) goto next; =20 @@ -731,7 +761,7 @@ static int scan_slots_for_writeback(struct zram *zram, = u32 mode, place_pp_slot(zram, ctl, pps); pps =3D NULL; next: - zram_slot_unlock(zram, index); + zram_slot_write_unlock(zram, index); } =20 kfree(pps); @@ -822,7 +852,7 @@ static ssize_t writeback_store(struct device *dev, } =20 index =3D pps->index; - zram_slot_lock(zram, index); + zram_slot_read_lock(zram, index); /* * scan_slots() sets ZRAM_PP_SLOT and relases slot lock, so * slots can change in the meantime. If slots are accessed or @@ -833,7 +863,7 @@ static ssize_t writeback_store(struct device *dev, goto next; if (zram_read_from_zspool(zram, page, index)) goto next; - zram_slot_unlock(zram, index); + zram_slot_read_unlock(zram, index); =20 bio_init(&bio, zram->bdev, &bio_vec, 1, REQ_OP_WRITE | REQ_SYNC); @@ -860,7 +890,7 @@ static ssize_t writeback_store(struct device *dev, } =20 atomic64_inc(&zram->stats.bd_writes); - zram_slot_lock(zram, index); + zram_slot_write_lock(zram, index); /* * Same as above, we release slot lock during writeback so * slot can change under us: slot_free() or slot_free() and @@ -882,7 +912,7 @@ static ssize_t writeback_store(struct device *dev, zram->bd_wb_limit -=3D 1UL << (PAGE_SHIFT - 12); spin_unlock(&zram->wb_limit_lock); next: - zram_slot_unlock(zram, index); + zram_slot_write_unlock(zram, index); release_pp_slot(zram, pps); =20 cond_resched(); @@ -1001,7 +1031,7 @@ static ssize_t read_block_state(struct file *file, ch= ar __user *buf, for (index =3D *ppos; index < nr_pages; index++) { int copied; =20 - zram_slot_lock(zram, index); + zram_slot_read_lock(zram, index); if (!zram_allocated(zram, index)) goto next; =20 @@ -1019,13 +1049,13 @@ static ssize_t read_block_state(struct file *file, = char __user *buf, ZRAM_INCOMPRESSIBLE) ? 'n' : '.'); =20 if (count <=3D copied) { - zram_slot_unlock(zram, index); + zram_slot_read_unlock(zram, index); break; } written +=3D copied; count -=3D copied; next: - zram_slot_unlock(zram, index); + zram_slot_read_unlock(zram, index); *ppos +=3D 1; } =20 @@ -1473,15 +1503,11 @@ static bool zram_meta_alloc(struct zram *zram, u64 = disksize) huge_class_size =3D zs_huge_class_size(zram->mem_pool); =20 for (index =3D 0; index < num_pages; index++) - spin_lock_init(&zram->table[index].lock); + atomic_set(&zram->table[index].lock, ZRAM_ENTRY_UNLOCKED); + return true; } =20 -/* - * To protect concurrent access to the same index entry, - * caller should hold this table index entry's bit_spinlock to - * indicate this index entry is accessing. - */ static void zram_free_page(struct zram *zram, size_t index) { unsigned long handle; @@ -1602,17 +1628,17 @@ static int zram_read_page(struct zram *zram, struct= page *page, u32 index, { int ret; =20 - zram_slot_lock(zram, index); + zram_slot_read_lock(zram, index); if (!zram_test_flag(zram, index, ZRAM_WB)) { /* Slot should be locked through out the function call */ ret =3D zram_read_from_zspool(zram, page, index); - zram_slot_unlock(zram, index); + zram_slot_read_unlock(zram, index); } else { /* * The slot should be unlocked before reading from the backing * device. */ - zram_slot_unlock(zram, index); + zram_slot_read_unlock(zram, index); =20 ret =3D read_from_bdev(zram, page, zram_get_handle(zram, index), parent); @@ -1655,10 +1681,10 @@ static int zram_bvec_read(struct zram *zram, struct= bio_vec *bvec, static int write_same_filled_page(struct zram *zram, unsigned long fill, u32 index) { - zram_slot_lock(zram, index); + zram_slot_write_lock(zram, index); zram_set_flag(zram, index, ZRAM_SAME); zram_set_handle(zram, index, fill); - zram_slot_unlock(zram, index); + zram_slot_write_unlock(zram, index); =20 atomic64_inc(&zram->stats.same_pages); atomic64_inc(&zram->stats.pages_stored); @@ -1693,11 +1719,11 @@ static int write_incompressible_page(struct zram *z= ram, struct page *page, kunmap_local(src); zs_unmap_object(zram->mem_pool, handle); =20 - zram_slot_lock(zram, index); + zram_slot_write_lock(zram, index); zram_set_flag(zram, index, ZRAM_HUGE); zram_set_handle(zram, index, handle); zram_set_obj_size(zram, index, PAGE_SIZE); - zram_slot_unlock(zram, index); + zram_slot_write_unlock(zram, index); =20 atomic64_add(PAGE_SIZE, &zram->stats.compr_data_size); atomic64_inc(&zram->stats.huge_pages); @@ -1718,9 +1744,9 @@ static int zram_write_page(struct zram *zram, struct = page *page, u32 index) bool same_filled; =20 /* First, free memory allocated to this slot (if any) */ - zram_slot_lock(zram, index); + zram_slot_write_lock(zram, index); zram_free_page(zram, index); - zram_slot_unlock(zram, index); + zram_slot_write_unlock(zram, index); =20 mem =3D kmap_local_page(page); same_filled =3D page_same_filled(mem, &element); @@ -1790,10 +1816,10 @@ static int zram_write_page(struct zram *zram, struc= t page *page, u32 index) zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); zs_unmap_object(zram->mem_pool, handle); =20 - zram_slot_lock(zram, index); + zram_slot_write_lock(zram, index); zram_set_handle(zram, index, handle); zram_set_obj_size(zram, index, comp_len); - zram_slot_unlock(zram, index); + zram_slot_write_unlock(zram, index); =20 /* Update stats */ atomic64_inc(&zram->stats.pages_stored); @@ -1850,7 +1876,7 @@ static int scan_slots_for_recompress(struct zram *zra= m, u32 mode, =20 INIT_LIST_HEAD(&pps->entry); =20 - zram_slot_lock(zram, index); + zram_slot_write_lock(zram, index); if (!zram_allocated(zram, index)) goto next; =20 @@ -1871,7 +1897,7 @@ static int scan_slots_for_recompress(struct zram *zra= m, u32 mode, place_pp_slot(zram, ctl, pps); pps =3D NULL; next: - zram_slot_unlock(zram, index); + zram_slot_write_unlock(zram, index); } =20 kfree(pps); @@ -2162,7 +2188,7 @@ static ssize_t recompress_store(struct device *dev, if (!num_recomp_pages) break; =20 - zram_slot_lock(zram, pps->index); + zram_slot_write_lock(zram, pps->index); if (!zram_test_flag(zram, pps->index, ZRAM_PP_SLOT)) goto next; =20 @@ -2170,7 +2196,7 @@ static ssize_t recompress_store(struct device *dev, &num_recomp_pages, threshold, prio, prio_max); next: - zram_slot_unlock(zram, pps->index); + zram_slot_write_unlock(zram, pps->index); release_pp_slot(zram, pps); =20 if (err) { @@ -2217,9 +2243,9 @@ static void zram_bio_discard(struct zram *zram, struc= t bio *bio) } =20 while (n >=3D PAGE_SIZE) { - zram_slot_lock(zram, index); + zram_slot_write_lock(zram, index); zram_free_page(zram, index); - zram_slot_unlock(zram, index); + zram_slot_write_unlock(zram, index); atomic64_inc(&zram->stats.notify_free); index++; n -=3D PAGE_SIZE; @@ -2248,9 +2274,9 @@ static void zram_bio_read(struct zram *zram, struct b= io *bio) } flush_dcache_page(bv.bv_page); =20 - zram_slot_lock(zram, index); + zram_slot_write_lock(zram, index); zram_accessed(zram, index); - zram_slot_unlock(zram, index); + zram_slot_write_unlock(zram, index); =20 bio_advance_iter_single(bio, &iter, bv.bv_len); } while (iter.bi_size); @@ -2278,9 +2304,9 @@ static void zram_bio_write(struct zram *zram, struct = bio *bio) break; } =20 - zram_slot_lock(zram, index); + zram_slot_write_lock(zram, index); zram_accessed(zram, index); - zram_slot_unlock(zram, index); + zram_slot_write_unlock(zram, index); =20 bio_advance_iter_single(bio, &iter, bv.bv_len); } while (iter.bi_size); @@ -2321,13 +2347,13 @@ static void zram_slot_free_notify(struct block_devi= ce *bdev, zram =3D bdev->bd_disk->private_data; =20 atomic64_inc(&zram->stats.notify_free); - if (!zram_slot_trylock(zram, index)) { + if (!zram_slot_try_write_lock(zram, index)) { atomic64_inc(&zram->stats.miss_free); return; } =20 zram_free_page(zram, index); - zram_slot_unlock(zram, index); + zram_slot_write_unlock(zram, index); } =20 static void zram_comp_params_reset(struct zram *zram) diff --git a/drivers/block/zram/zram_drv.h b/drivers/block/zram/zram_drv.h index db78d7c01b9a..e20538cdf565 100644 --- a/drivers/block/zram/zram_drv.h +++ b/drivers/block/zram/zram_drv.h @@ -28,7 +28,6 @@ #define ZRAM_SECTOR_PER_LOGICAL_BLOCK \ (1 << (ZRAM_LOGICAL_BLOCK_SHIFT - SECTOR_SHIFT)) =20 - /* * ZRAM is mainly used for memory efficiency so we want to keep memory * footprint small and thus squeeze size and zram pageflags into a flags @@ -58,13 +57,14 @@ enum zram_pageflags { __NR_ZRAM_PAGEFLAGS, }; =20 -/*-- Data structures */ +#define ZRAM_ENTRY_UNLOCKED 0 +#define ZRAM_ENTRY_WRLOCKED (-1) =20 /* Allocated for each disk page */ struct zram_table_entry { unsigned long handle; unsigned int flags; - spinlock_t lock; + atomic_t lock; #ifdef CONFIG_ZRAM_TRACK_ENTRY_ACTIME ktime_t ac_time; #endif --=20 2.48.1.362.g079036d154-goog From nobody Mon Feb 9 02:42:15 2026 Received: from mail-pl1-f178.google.com (mail-pl1-f178.google.com [209.85.214.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5F83418A6DB for ; Fri, 31 Jan 2025 09:07:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738314435; cv=none; b=l+2T7FyGkxt8/eqzqNyHETYdMoFEkGUA2FX/qmH26u2amJDZIZSNw/ZCJzG8rrx8uzjdESdX4mAlAXyM5NmIG7N5b71EqRNTOvp/pQazqt6fxZKxmu6yUvW5jeUGcOYvSnzfJeMCiaArm0XC/ouJOH20l3MPS1zj78fDCRwfEYs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738314435; c=relaxed/simple; bh=jPO8HZ1Zps/Lfyb11yjVIb6bxl5LlFnczMY6NjioKvI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=haQIAT2u9pBt0iVkfa585Or1nagZshcx6w85wmY0KHNHS1OhEs5FKXmXfcceBpT5/liDuOoOUErJG1d1WJO/WtWQpVvsfr4hbpQ7oDBsdtHuN0jbwnc35eTG0Cdn1Oesbdg/njpTsz8QIMsxDg78yBea/v/A4sYrxHcmH8GXKEQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=SQVCvlpY; arc=none smtp.client-ip=209.85.214.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="SQVCvlpY" Received: by mail-pl1-f178.google.com with SMTP id d9443c01a7336-21644aca3a0so39622025ad.3 for ; Fri, 31 Jan 2025 01:07:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1738314432; x=1738919232; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ZM/BjG4bEVLcBszqat2kHDfQpVGFwc0ijbl9RU545So=; b=SQVCvlpY+DScP2gmPxS18ZKxNIcZKHNZataXPb384v2MxxlJspNaIGlu82sWmYdwGi I8YhUo3idxR30EdOTdvjZSLGQtIxB6bO7JmhLRAt1I2v+eEKi37r7ftCkEwKf9FFOeoD MiRxX4gto6GEwm2YKTGupu/RO4gW+dwLWUzP8= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738314432; x=1738919232; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ZM/BjG4bEVLcBszqat2kHDfQpVGFwc0ijbl9RU545So=; b=UGRs/Yf53BEuDPDd2oPWTaTNej8APG1PMqA8nD1ne9iTj3a2/YbV3cnZJjfv2ptz8/ U0mTye0O4orwaAj5U9ZsuUWqvxml5uINZqsps2ko9stoYZRSjil0SETzgaVSzQ76i/FW m8UJwUCHiCRhErE2uwwnOWJIfI0+9IxmuPJSUPfXq50SpJjkTdVeMGsEOIFZ2JQ3QLep KkQ1j5I9997fIw1v0RWdI+HO2y/wemyVP6nV6kfMc261qJkAcaz2TxkzmO4CVdUs6vK+ zgz+LpAiERnDqG2H4HPtEMOsIfBS3251FS4bCuS+iXTBWk9lTKxdVodjm1duupXVFSC+ xmKw== X-Forwarded-Encrypted: i=1; AJvYcCUx5ul9JVyZNtczor8TXctRpPjZ+WUf1owpid12705HDQ/3SDQOcvFMXW4P/p92e5IOlNSzF6QXiaI/aJY=@vger.kernel.org X-Gm-Message-State: AOJu0Yzu17NIw5IMT5mIfRhPX9fuMewc4577ClKZwEl0IEDt4hp1kRZc kRq3rkmSrD9yO75WUAdyLCJMgs+ZgldaAZ9jyBp2S40+1Hq+LGkMpn/br50ylg== X-Gm-Gg: ASbGncveZSRBskQb4DkTt566cmZjsTPP+rnzsJBgnQsqZFZJdym5uEWqinaWpTKZ9wM xcLfPREUx6UJHUIVAtpzKwOT05pR0iavaE+otaOoV5JnwU3EzeAw3/m5eY5su6HliQA47hRf0Fq NcJ1efGs84wlqKLMju3fLdxqfKrOlKjOH+DxexPXykzkxosMKhaB/RmBQeOxFpoa/BqERHGdYp+ PMVmVzKdaKzjidv/BiI/uOeTE4xS0RDpc2pDc+/HyEP2itE1QhA4PjUGb0PvJMWUK7CJctmW/BQ VI+mCvbTOnGM4SBkaA== X-Google-Smtp-Source: AGHT+IHUYlW/s/BROjpWDLIYuBNea13AbLtp5vL/iIvdkHnTpWdyOu8kZuE13OYSyQwa0btUHKfWmA== X-Received: by 2002:a05:6a21:9102:b0:1e1:faa:d8cf with SMTP id adf61e73a8af0-1ed7a6e1efcmr19676241637.40.1738314432483; Fri, 31 Jan 2025 01:07:12 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:c752:be9d:3368:16fa]) by smtp.gmail.com with UTF8SMTPSA id 41be03b00d2f7-acebddbb0besm2609414a12.16.2025.01.31.01.07.10 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 31 Jan 2025 01:07:12 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv4 02/17] zram: do not use per-CPU compression streams Date: Fri, 31 Jan 2025 18:06:01 +0900 Message-ID: <20250131090658.3386285-3-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.362.g079036d154-goog In-Reply-To: <20250131090658.3386285-1-senozhatsky@chromium.org> References: <20250131090658.3386285-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Similarly to per-entry spin-lock per-CPU compression streams also have a number of shortcoming. First, per-CPU stream access has to be done from a non-preemptible (atomic) section, which imposes the same atomicity requirements on compression backends as entry spin-lock do and makes it impossible to use algorithms that can schedule/wait/sleep during compression and decompression. Second, per-CPU streams noticeably increase memory usage (actually more like wastage) of secondary compression streams. The problem is that secondary compression streams are allocated per-CPU, just like the primary streams are. Yet we never use more that one secondary stream at a time, because recompression is a single threaded action. Which means that remaining num_online_cpu() - 1 streams are allocated for nothing, and this is per-priority list (we can have several secondary compression algorithms). Depending on the algorithm this may lead to a significant memory wastage, in addition each stream also carries a workmem buffer (2 physical pages). Instead of per-CPU streams, maintain a list of idle compression streams and allocate new streams on-demand (something that we used to do many years ago). So that zram read() and write() become non-atomic and ease requirements on the compression algorithm implementation. This also means that we now should have only one secondary stream per-priority list. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zcomp.c | 164 +++++++++++++++++++--------------- drivers/block/zram/zcomp.h | 17 ++-- drivers/block/zram/zram_drv.c | 29 +++--- include/linux/cpuhotplug.h | 1 - 4 files changed, 109 insertions(+), 102 deletions(-) diff --git a/drivers/block/zram/zcomp.c b/drivers/block/zram/zcomp.c index bb514403e305..982c769d5831 100644 --- a/drivers/block/zram/zcomp.c +++ b/drivers/block/zram/zcomp.c @@ -6,7 +6,7 @@ #include #include #include -#include +#include #include #include =20 @@ -43,31 +43,40 @@ static const struct zcomp_ops *backends[] =3D { NULL }; =20 -static void zcomp_strm_free(struct zcomp *comp, struct zcomp_strm *zstrm) +static void zcomp_strm_free(struct zcomp *comp, struct zcomp_strm *strm) { - comp->ops->destroy_ctx(&zstrm->ctx); - vfree(zstrm->buffer); - zstrm->buffer =3D NULL; + comp->ops->destroy_ctx(&strm->ctx); + vfree(strm->buffer); + kfree(strm); } =20 -static int zcomp_strm_init(struct zcomp *comp, struct zcomp_strm *zstrm) +static struct zcomp_strm *zcomp_strm_alloc(struct zcomp *comp) { + struct zcomp_strm *strm; int ret; =20 - ret =3D comp->ops->create_ctx(comp->params, &zstrm->ctx); - if (ret) - return ret; + strm =3D kzalloc(sizeof(*strm), GFP_KERNEL); + if (!strm) + return NULL; + + INIT_LIST_HEAD(&strm->entry); + + ret =3D comp->ops->create_ctx(comp->params, &strm->ctx); + if (ret) { + kfree(strm); + return NULL; + } =20 /* - * allocate 2 pages. 1 for compressed data, plus 1 extra for the - * case when compressed size is larger than the original one + * allocate 2 pages. 1 for compressed data, plus 1 extra in case if + * compressed data is larger than the original one. */ - zstrm->buffer =3D vzalloc(2 * PAGE_SIZE); - if (!zstrm->buffer) { - zcomp_strm_free(comp, zstrm); - return -ENOMEM; + strm->buffer =3D vzalloc(2 * PAGE_SIZE); + if (!strm->buffer) { + zcomp_strm_free(comp, strm); + return NULL; } - return 0; + return strm; } =20 static const struct zcomp_ops *lookup_backend_ops(const char *comp) @@ -109,13 +118,59 @@ ssize_t zcomp_available_show(const char *comp, char *= buf) =20 struct zcomp_strm *zcomp_stream_get(struct zcomp *comp) { - local_lock(&comp->stream->lock); - return this_cpu_ptr(comp->stream); + struct zcomp_strm *strm; + + might_sleep(); + + while (1) { + spin_lock(&comp->strm_lock); + if (!list_empty(&comp->idle_strm)) { + strm =3D list_first_entry(&comp->idle_strm, + struct zcomp_strm, + entry); + list_del(&strm->entry); + spin_unlock(&comp->strm_lock); + return strm; + } + + /* cannot allocate new stream, wait for an idle one */ + if (comp->avail_strm >=3D num_online_cpus()) { + spin_unlock(&comp->strm_lock); + wait_event(comp->strm_wait, + !list_empty(&comp->idle_strm)); + continue; + } + + /* allocate new stream */ + comp->avail_strm++; + spin_unlock(&comp->strm_lock); + + strm =3D zcomp_strm_alloc(comp); + if (strm) + break; + + spin_lock(&comp->strm_lock); + comp->avail_strm--; + spin_unlock(&comp->strm_lock); + wait_event(comp->strm_wait, !list_empty(&comp->idle_strm)); + } + + return strm; } =20 -void zcomp_stream_put(struct zcomp *comp) +void zcomp_stream_put(struct zcomp *comp, struct zcomp_strm *strm) { - local_unlock(&comp->stream->lock); + spin_lock(&comp->strm_lock); + if (comp->avail_strm <=3D num_online_cpus()) { + list_add(&strm->entry, &comp->idle_strm); + spin_unlock(&comp->strm_lock); + wake_up(&comp->strm_wait); + return; + } + + comp->avail_strm--; + spin_unlock(&comp->strm_lock); + zcomp_strm_free(comp, strm); } =20 int zcomp_compress(struct zcomp *comp, struct zcomp_strm *zstrm, @@ -148,61 +203,19 @@ int zcomp_decompress(struct zcomp *comp, struct zcomp= _strm *zstrm, return comp->ops->decompress(comp->params, &zstrm->ctx, &req); } =20 -int zcomp_cpu_up_prepare(unsigned int cpu, struct hlist_node *node) -{ - struct zcomp *comp =3D hlist_entry(node, struct zcomp, node); - struct zcomp_strm *zstrm; - int ret; - - zstrm =3D per_cpu_ptr(comp->stream, cpu); - local_lock_init(&zstrm->lock); - - ret =3D zcomp_strm_init(comp, zstrm); - if (ret) - pr_err("Can't allocate a compression stream\n"); - return ret; -} - -int zcomp_cpu_dead(unsigned int cpu, struct hlist_node *node) -{ - struct zcomp *comp =3D hlist_entry(node, struct zcomp, node); - struct zcomp_strm *zstrm; - - zstrm =3D per_cpu_ptr(comp->stream, cpu); - zcomp_strm_free(comp, zstrm); - return 0; -} - -static int zcomp_init(struct zcomp *comp, struct zcomp_params *params) -{ - int ret; - - comp->stream =3D alloc_percpu(struct zcomp_strm); - if (!comp->stream) - return -ENOMEM; - - comp->params =3D params; - ret =3D comp->ops->setup_params(comp->params); - if (ret) - goto cleanup; - - ret =3D cpuhp_state_add_instance(CPUHP_ZCOMP_PREPARE, &comp->node); - if (ret < 0) - goto cleanup; - - return 0; - -cleanup: - comp->ops->release_params(comp->params); - free_percpu(comp->stream); - return ret; -} - void zcomp_destroy(struct zcomp *comp) { - cpuhp_state_remove_instance(CPUHP_ZCOMP_PREPARE, &comp->node); + struct zcomp_strm *strm; + + while (!list_empty(&comp->idle_strm)) { + strm =3D list_first_entry(&comp->idle_strm, + struct zcomp_strm, + entry); + list_del(&strm->entry); + zcomp_strm_free(comp, strm); + } + comp->ops->release_params(comp->params); - free_percpu(comp->stream); kfree(comp); } =20 @@ -229,7 +242,12 @@ struct zcomp *zcomp_create(const char *alg, struct zco= mp_params *params) return ERR_PTR(-EINVAL); } =20 - error =3D zcomp_init(comp, params); + INIT_LIST_HEAD(&comp->idle_strm); + init_waitqueue_head(&comp->strm_wait); + spin_lock_init(&comp->strm_lock); + + comp->params =3D params; + error =3D comp->ops->setup_params(comp->params); if (error) { kfree(comp); return ERR_PTR(error); diff --git a/drivers/block/zram/zcomp.h b/drivers/block/zram/zcomp.h index ad5762813842..62330829db3f 100644 --- a/drivers/block/zram/zcomp.h +++ b/drivers/block/zram/zcomp.h @@ -3,10 +3,10 @@ #ifndef _ZCOMP_H_ #define _ZCOMP_H_ =20 -#include - #define ZCOMP_PARAM_NO_LEVEL INT_MIN =20 +#include + /* * Immutable driver (backend) parameters. The driver may attach private * data to it (e.g. driver representation of the dictionary, etc.). @@ -31,7 +31,7 @@ struct zcomp_ctx { }; =20 struct zcomp_strm { - local_lock_t lock; + struct list_head entry; /* compression buffer */ void *buffer; struct zcomp_ctx ctx; @@ -60,16 +60,15 @@ struct zcomp_ops { const char *name; }; =20 -/* dynamic per-device compression frontend */ struct zcomp { - struct zcomp_strm __percpu *stream; + struct list_head idle_strm; + spinlock_t strm_lock; + u32 avail_strm; + wait_queue_head_t strm_wait; const struct zcomp_ops *ops; struct zcomp_params *params; - struct hlist_node node; }; =20 -int zcomp_cpu_up_prepare(unsigned int cpu, struct hlist_node *node); -int zcomp_cpu_dead(unsigned int cpu, struct hlist_node *node); ssize_t zcomp_available_show(const char *comp, char *buf); bool zcomp_available_algorithm(const char *comp); =20 @@ -77,7 +76,7 @@ struct zcomp *zcomp_create(const char *alg, struct zcomp_= params *params); void zcomp_destroy(struct zcomp *comp); =20 struct zcomp_strm *zcomp_stream_get(struct zcomp *comp); -void zcomp_stream_put(struct zcomp *comp); +void zcomp_stream_put(struct zcomp *comp, struct zcomp_strm *strm); =20 int zcomp_compress(struct zcomp *comp, struct zcomp_strm *zstrm, const void *src, unsigned int *dst_len); diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 1c2df2341704..8d5974ea8ff8 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -31,7 +31,6 @@ #include #include #include -#include #include #include =20 @@ -1601,7 +1600,7 @@ static int read_compressed_page(struct zram *zram, st= ruct page *page, u32 index) ret =3D zcomp_decompress(zram->comps[prio], zstrm, src, size, dst); kunmap_local(dst); zs_unmap_object(zram->mem_pool, handle); - zcomp_stream_put(zram->comps[prio]); + zcomp_stream_put(zram->comps[prio], zstrm); =20 return ret; } @@ -1762,14 +1761,14 @@ static int zram_write_page(struct zram *zram, struc= t page *page, u32 index) kunmap_local(mem); =20 if (unlikely(ret)) { - zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); + zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP], zstrm); pr_err("Compression failed! err=3D%d\n", ret); zs_free(zram->mem_pool, handle); return ret; } =20 if (comp_len >=3D huge_class_size) { - zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); + zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP], zstrm); return write_incompressible_page(zram, page, index); } =20 @@ -1793,7 +1792,7 @@ static int zram_write_page(struct zram *zram, struct = page *page, u32 index) __GFP_HIGHMEM | __GFP_MOVABLE); if (IS_ERR_VALUE(handle)) { - zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); + zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP], zstrm); atomic64_inc(&zram->stats.writestall); handle =3D zs_malloc(zram->mem_pool, comp_len, GFP_NOIO | __GFP_HIGHMEM | @@ -1805,7 +1804,7 @@ static int zram_write_page(struct zram *zram, struct = page *page, u32 index) } =20 if (!zram_can_store_page(zram)) { - zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); + zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP], zstrm); zs_free(zram->mem_pool, handle); return -ENOMEM; } @@ -1813,7 +1812,7 @@ static int zram_write_page(struct zram *zram, struct = page *page, u32 index) dst =3D zs_map_object(zram->mem_pool, handle, ZS_MM_WO); =20 memcpy(dst, zstrm->buffer, comp_len); - zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); + zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP], zstrm); zs_unmap_object(zram->mem_pool, handle); =20 zram_slot_write_lock(zram, index); @@ -1972,7 +1971,7 @@ static int recompress_slot(struct zram *zram, u32 ind= ex, struct page *page, kunmap_local(src); =20 if (ret) { - zcomp_stream_put(zram->comps[prio]); + zcomp_stream_put(zram->comps[prio], zstrm); return ret; } =20 @@ -1982,7 +1981,7 @@ static int recompress_slot(struct zram *zram, u32 ind= ex, struct page *page, /* Continue until we make progress */ if (class_index_new >=3D class_index_old || (threshold && comp_len_new >=3D threshold)) { - zcomp_stream_put(zram->comps[prio]); + zcomp_stream_put(zram->comps[prio], zstrm); continue; } =20 @@ -2040,13 +2039,13 @@ static int recompress_slot(struct zram *zram, u32 i= ndex, struct page *page, __GFP_HIGHMEM | __GFP_MOVABLE); if (IS_ERR_VALUE(handle_new)) { - zcomp_stream_put(zram->comps[prio]); + zcomp_stream_put(zram->comps[prio], zstrm); return PTR_ERR((void *)handle_new); } =20 dst =3D zs_map_object(zram->mem_pool, handle_new, ZS_MM_WO); memcpy(dst, zstrm->buffer, comp_len_new); - zcomp_stream_put(zram->comps[prio]); + zcomp_stream_put(zram->comps[prio], zstrm); =20 zs_unmap_object(zram->mem_pool, handle_new); =20 @@ -2794,7 +2793,6 @@ static void destroy_devices(void) zram_debugfs_destroy(); idr_destroy(&zram_index_idr); unregister_blkdev(zram_major, "zram"); - cpuhp_remove_multi_state(CPUHP_ZCOMP_PREPARE); } =20 static int __init zram_init(void) @@ -2804,15 +2802,9 @@ static int __init zram_init(void) =20 BUILD_BUG_ON(__NR_ZRAM_PAGEFLAGS > sizeof(zram_te.flags) * 8); =20 - ret =3D cpuhp_setup_state_multi(CPUHP_ZCOMP_PREPARE, "block/zram:prepare", - zcomp_cpu_up_prepare, zcomp_cpu_dead); - if (ret < 0) - return ret; - ret =3D class_register(&zram_control_class); if (ret) { pr_err("Unable to register zram-control class\n"); - cpuhp_remove_multi_state(CPUHP_ZCOMP_PREPARE); return ret; } =20 @@ -2821,7 +2813,6 @@ static int __init zram_init(void) if (zram_major <=3D 0) { pr_err("Unable to get major number\n"); class_unregister(&zram_control_class); - cpuhp_remove_multi_state(CPUHP_ZCOMP_PREPARE); return -EBUSY; } =20 diff --git a/include/linux/cpuhotplug.h b/include/linux/cpuhotplug.h index 6cc5e484547c..092ace7db8ee 100644 --- a/include/linux/cpuhotplug.h +++ b/include/linux/cpuhotplug.h @@ -119,7 +119,6 @@ enum cpuhp_state { CPUHP_MM_ZS_PREPARE, CPUHP_MM_ZSWP_POOL_PREPARE, CPUHP_KVM_PPC_BOOK3S_PREPARE, - CPUHP_ZCOMP_PREPARE, CPUHP_TIMERS_PREPARE, CPUHP_TMIGR_PREPARE, CPUHP_MIPS_SOC_PREPARE, --=20 2.48.1.362.g079036d154-goog From nobody Mon Feb 9 02:42:15 2026 Received: from mail-pl1-f175.google.com (mail-pl1-f175.google.com [209.85.214.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F2CD018871F for ; Fri, 31 Jan 2025 09:07:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.175 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738314438; cv=none; b=ScyRfnab72JrpCB0a345asxf8c2GrLMaFtsTeddKAbayOWoJfscf/nosLlVUlYxgw8LYO69slCX0K238pt6Rt6aK2ax87vW9xBhL9wTXiX9nV1CMwci8piaBKacUO8ip7lWfcpCYNTlt3T+bS8vQEUfVRhNibRb2bV7zUNBqLOA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738314438; c=relaxed/simple; bh=KBisht+WpBvXm0v+20ockyV3U5DpyRL1QZ1uYJrU6jY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=MdNRNqveDSg1YSiW4ulenF9S2Q/Bm3BZKWKwV2iUfrdPbd4bx3oi3gbrMD6niekmD4eXu5ZVHliGDk1XFz6V1CRRCZ5fgDWIke1gfMMTzVeqmL1GZos/OHFye19Lw0OBg0eeH0bCkrSKfeZ3G40aJ7JG9D8zcv0cYtngOay9B8g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=PLrFX7XS; arc=none smtp.client-ip=209.85.214.175 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="PLrFX7XS" Received: by mail-pl1-f175.google.com with SMTP id d9443c01a7336-215770613dbso22348565ad.2 for ; Fri, 31 Jan 2025 01:07:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1738314436; x=1738919236; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=n/jN2+B1aQ/0Z82UnSahO/vWpa5e/0HHbY39bdZtI8w=; b=PLrFX7XSSOLvj+zYEMOgCPTlwGhahiSFyF2eE2hVRE+oaAlPCHsFdWHq6iB0MS65YI fMx2pd1mlYb605GTUctF6WSghYl5ibWfNWIm+4jdcYrqYEqVoYdyU/IhJY6q/aVVNqve LfszcNCWbab6J8hVgRgJfpQRXWGPAYGGs5gS0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738314436; x=1738919236; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=n/jN2+B1aQ/0Z82UnSahO/vWpa5e/0HHbY39bdZtI8w=; b=OTs0ggtvZWZINmIxLICz8JbiVbozN9kFDP5F84LYGxNcnnmHQSSxA0sZ6e+tj/asM0 kSAPIG321Bc5L6Q9XN/U4qZ6PZZmvLDBfF0xeYpfkRE11KGCyQzX8+k9nBMHGSjHt974 3uG0M/xDmVbiwO5vJw+gLWo9kDhSjBfQ7FVmPko/kF73/Gn6yrNaA/MSZU3Sqxcl1Fub Pl2duPVDowVAPTeqlCknlIjIFPJ0fQoU0yA7BAW2guzBb12qCTXPxwDDNK7x6nQtmWUh g9nR3wI01VChYMHkw1Wn5hlXwDdDU1H0k72aTCfaZrNMvvLOOT02BfSG9iQWmKEM6B2M iwlA== X-Forwarded-Encrypted: i=1; AJvYcCWpvsOvl0k4zBSxuDcPIfdsuA4S54ZY5+SImZLUN5SkZmE/jWrdQ5YWKdQfIDA4Mg+tpZywqjaXZd3cFUQ=@vger.kernel.org X-Gm-Message-State: AOJu0YzEQ543Bo/0HFWNurCpgUuNjsPC801IscbEFZoKVbhod52Bh0hi PTu6shyYQKaD/Mksa4W9ozAjZLDwAliS922Px8ocRwNggswHG8ubf1H7c7c0uw== X-Gm-Gg: ASbGncs26LNCZ1zG9kkCDOqiBkFWcook7kO1+xLQ/n6Imd1G946jUjdmhP160WyI3+0 bcjtBlNHXOf7i5cN5pe3j+kZH4H+m/HRTpGxh/+DX8DvNc7YfpSODzTuKrJGCnBoehj/w4V1EQj CasmNz1VuZoEWDS2H0UCLS3lsnU4WzgSBFnTA5BuFXcFh6pCRq1CujODWVctM4kJ9DW5mxnsZvq 8VC+DHeObMNnpq11/FcnSI4+RppozAdwIfbQ7S9Pk9mvJk3HE8CwsJwcWWLnEZkZAWLmFKG9yFd qHpZELfAokJPicraMg== X-Google-Smtp-Source: AGHT+IFuc7qdcZdQN6B06WAX8++wNkf8GmtQ583C/YsjPsWR7di3HOqQJ5UOroWbF4X70ISjCdoQzA== X-Received: by 2002:a05:6a00:3e13:b0:72a:a9d9:5a82 with SMTP id d2e1a72fcca58-72fd096a1bemr14710114b3a.0.1738314436186; Fri, 31 Jan 2025 01:07:16 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:c752:be9d:3368:16fa]) by smtp.gmail.com with UTF8SMTPSA id d2e1a72fcca58-72ff8ae7c9dsm575712b3a.71.2025.01.31.01.07.14 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 31 Jan 2025 01:07:15 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv4 03/17] zram: remove crypto include Date: Fri, 31 Jan 2025 18:06:02 +0900 Message-ID: <20250131090658.3386285-4-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.362.g079036d154-goog In-Reply-To: <20250131090658.3386285-1-senozhatsky@chromium.org> References: <20250131090658.3386285-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Remove a leftover crypto header include. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zcomp.c | 1 - drivers/block/zram/zram_drv.c | 4 +++- drivers/block/zram/zram_drv.h | 1 - 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/drivers/block/zram/zcomp.c b/drivers/block/zram/zcomp.c index 982c769d5831..efd5919808d9 100644 --- a/drivers/block/zram/zcomp.c +++ b/drivers/block/zram/zcomp.c @@ -7,7 +7,6 @@ #include #include #include -#include #include =20 #include "zcomp.h" diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 8d5974ea8ff8..6239fcc340b6 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -43,6 +43,8 @@ static DEFINE_MUTEX(zram_index_mutex); static int zram_major; static const char *default_compressor =3D CONFIG_ZRAM_DEF_COMP; =20 +#define ZRAM_MAX_ALGO_NAME_SZ 64 + /* Module params (documentation at end) */ static unsigned int num_devices =3D 1; /* @@ -1141,7 +1143,7 @@ static int __comp_algorithm_store(struct zram *zram, = u32 prio, const char *buf) size_t sz; =20 sz =3D strlen(buf); - if (sz >=3D CRYPTO_MAX_ALG_NAME) + if (sz >=3D ZRAM_MAX_ALGO_NAME_SZ) return -E2BIG; =20 compressor =3D kstrdup(buf, GFP_KERNEL); diff --git a/drivers/block/zram/zram_drv.h b/drivers/block/zram/zram_drv.h index e20538cdf565..3ae2988090b3 100644 --- a/drivers/block/zram/zram_drv.h +++ b/drivers/block/zram/zram_drv.h @@ -17,7 +17,6 @@ =20 #include #include -#include =20 #include "zcomp.h" =20 --=20 2.48.1.362.g079036d154-goog From nobody Mon Feb 9 02:42:15 2026 Received: from mail-pj1-f52.google.com (mail-pj1-f52.google.com [209.85.216.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 21469175D53 for ; Fri, 31 Jan 2025 09:07:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.52 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738314443; cv=none; b=lElQ34vEUX3EY1KTtF6aZvDXQJTWrHWARP5ekXkeFCt2c5Ju83xfEtQ6/KsLD+nivKKUEP6kcgeTjLtcaSw1+c2NgngH5cdIV4/fQqp6sFhGzwWayeQ13YDLjml2fMl8x1k0EqyaAOKT2ga7sAEHjweXtD9jfIDHuRp+1wbI9Mg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738314443; c=relaxed/simple; bh=ENMfLDd2Rvn+6if/WlSoG7kHUOF/ByzGPafWP6rI63s=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=XVGu1jFE/njxZxF3Zn1XvRdavc0YkhXBYkjo/O9R76lYQp5K5wPxN47ZMlZnjGOJi/g37IYAo6rixZUMNz9ov0/+W7fM5XLTDbX0+2XOv2PzrBd4qAzoUheppBqtD4pwAoythnwjG7IZZxygp0RUM/HyACNjUR42bd65iynkGQc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=cWHKgG/f; arc=none smtp.client-ip=209.85.216.52 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="cWHKgG/f" Received: by mail-pj1-f52.google.com with SMTP id 98e67ed59e1d1-2ee50ffcf14so4458504a91.0 for ; Fri, 31 Jan 2025 01:07:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1738314441; x=1738919241; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=TXbFDLYuuOhKufVYkqbgI//dvCPqRmfyGlKUIOvD+TU=; b=cWHKgG/fvFQKUCIytoAQvlCYsq0HBnArVr/BJHJXGVj2wGrv9/avq71C6tBDlbJzxm sin5dcNybHufQU7ajBcZ/rMv6bZaA0XLB92m7wCPanB8BzVIgWEQLF4Ik5/dsx+OBTuz DOYiw0mjjs4huDb3aAgdTOXxh4OYPFnAKoj8s= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738314441; x=1738919241; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=TXbFDLYuuOhKufVYkqbgI//dvCPqRmfyGlKUIOvD+TU=; b=YZouJHKRoS7UqqcdGZd4xESZuQOivoyKyaUy5uwHVUDUYWxKJXi5tre1pn2NOkZ+jk C8r2mY2SDvon9DD22+Rei3cV7SSeoRw7xd8VQqOHE6GHkUadaGs/dviRhJQw/AB1AWC7 sWQO1o0sP6EbQPt7otlM6n2lg7xDNSrBsKTGZgi6vYw/xgLtsURRCkEMkdnMZVU6U5hi e/2WjStrXsb9w9SL8YhDSjKoUWFc1KWMZXkI/H2rcNKOSufW3LJNUfb4CeL/s4pymuO5 qpMGkWSfjsCPf0jUb4DEpKzR/KBVq3IDr5x2r9M+WhrUIjW2ndUdwNHU/bOlFf9REDFf 8k0A== X-Forwarded-Encrypted: i=1; AJvYcCUbCOhvMDSiDkOknSv4fJuw1eXdj30uks3ZEYtuESjBxUK/VoDa+Mb34Zc/YwN1NKxdecXoy/KVgyOvrFg=@vger.kernel.org X-Gm-Message-State: AOJu0Yz6N0ZMEyY02SFAOHXWHha4RFINyQDr/dnu3MuzeCfDWLTr+G+E 4bYPXpzApEKEBNvLUxXhfPPw+ORVdwrTNBLTSw+LF93299r2goQunI1pLE4+SQ== X-Gm-Gg: ASbGncs5Zzv+lNMelMsx25AggLU+9lc0nQ3+hbinYijvt/1U9wYCHxo4TS2UgByg+2n 8HY7QSyrL0qss94uCsavUurAiofdS8n6TMA/LsGjdy8ZzKlIUYcozzN9l6T65HlQ/YItsKnHG20 K8fisoEb1WqIqt10jG9T2ukzl4dB3bdnC80RS9WcSMCSxpDoNXnhbCEBcGRARDe6ObznIk1YScR lQF0CcLtwSsuJoJwnqsFSlR6wI4Tra/Rr8dUdCzJV7hoyt9r9gfydG+U6OOyRj2OSLdYgVCPG3r nPjIYSPEWGKJ34IUtg== X-Google-Smtp-Source: AGHT+IEKH7u2FXrx6jWcyNcJb/PaxSPMsTyCbU1VktjzxFUwBwz+4J7kQF6itN5PPT+0HWOIKATLLw== X-Received: by 2002:a17:90b:2dc7:b0:2ee:9229:e4bd with SMTP id 98e67ed59e1d1-2f84633eaeemr10228618a91.2.1738314440902; Fri, 31 Jan 2025 01:07:20 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:c752:be9d:3368:16fa]) by smtp.gmail.com with UTF8SMTPSA id 98e67ed59e1d1-2f83bccd590sm5442929a91.12.2025.01.31.01.07.18 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 31 Jan 2025 01:07:20 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv4 04/17] zram: remove max_comp_streams device attr Date: Fri, 31 Jan 2025 18:06:03 +0900 Message-ID: <20250131090658.3386285-5-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.362.g079036d154-goog In-Reply-To: <20250131090658.3386285-1-senozhatsky@chromium.org> References: <20250131090658.3386285-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" max_comp_streams device attribute has been defunct since May 2016 when zram switched to per-CPU compression streams, remove it. Signed-off-by: Sergey Senozhatsky --- Documentation/ABI/testing/sysfs-block-zram | 8 ----- Documentation/admin-guide/blockdev/zram.rst | 36 ++++++--------------- drivers/block/zram/zram_drv.c | 23 ------------- 3 files changed, 10 insertions(+), 57 deletions(-) diff --git a/Documentation/ABI/testing/sysfs-block-zram b/Documentation/ABI= /testing/sysfs-block-zram index 1ef69e0271f9..36c57de0a10a 100644 --- a/Documentation/ABI/testing/sysfs-block-zram +++ b/Documentation/ABI/testing/sysfs-block-zram @@ -22,14 +22,6 @@ Description: device. The reset operation frees all the memory associated with this device. =20 -What: /sys/block/zram/max_comp_streams -Date: February 2014 -Contact: Sergey Senozhatsky -Description: - The max_comp_streams file is read-write and specifies the - number of backend's zcomp_strm compression streams (number of - concurrent compress operations). - What: /sys/block/zram/comp_algorithm Date: February 2014 Contact: Sergey Senozhatsky diff --git a/Documentation/admin-guide/blockdev/zram.rst b/Documentation/ad= min-guide/blockdev/zram.rst index 1576fb93f06c..9bdb30901a93 100644 --- a/Documentation/admin-guide/blockdev/zram.rst +++ b/Documentation/admin-guide/blockdev/zram.rst @@ -54,7 +54,7 @@ The list of possible return codes: If you use 'echo', the returned value is set by the 'echo' utility, and, in general case, something like:: =20 - echo 3 > /sys/block/zram0/max_comp_streams + echo foo > /sys/block/zram0/comp_algorithm if [ $? -ne 0 ]; then handle_error fi @@ -73,21 +73,7 @@ This creates 4 devices: /dev/zram{0,1,2,3} num_devices parameter is optional and tells zram how many devices should be pre-created. Default: 1. =20 -2) Set max number of compression streams -=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D - -Regardless of the value passed to this attribute, ZRAM will always -allocate multiple compression streams - one per online CPU - thus -allowing several concurrent compression operations. The number of -allocated compression streams goes down when some of the CPUs -become offline. There is no single-compression-stream mode anymore, -unless you are running a UP system or have only 1 CPU online. - -To find out how many streams are currently available:: - - cat /sys/block/zram0/max_comp_streams - -3) Select compression algorithm +2) Select compression algorithm =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D =20 Using comp_algorithm device attribute one can see available and @@ -107,7 +93,7 @@ Examples:: For the time being, the `comp_algorithm` content shows only compression algorithms that are supported by zram. =20 -4) Set compression algorithm parameters: Optional +3) Set compression algorithm parameters: Optional =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =20 Compression algorithms may support specific parameters which can be @@ -138,7 +124,7 @@ better the compression ratio, it even can take negative= s values for some algorithms), for other algorithms `level` is acceleration level (the higher the value the lower the compression ratio). =20 -5) Set Disksize +4) Set Disksize =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =20 Set disk size by writing the value to sysfs node 'disksize'. @@ -158,7 +144,7 @@ There is little point creating a zram of greater than t= wice the size of memory since we expect a 2:1 compression ratio. Note that zram uses about 0.1% of= the size of the disk when not in use so a huge zram is wasteful. =20 -6) Set memory limit: Optional +5) Set memory limit: Optional =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D =20 Set memory limit by writing the value to sysfs node 'mem_limit'. @@ -177,7 +163,7 @@ Examples:: # To disable memory limit echo 0 > /sys/block/zram0/mem_limit =20 -7) Activate +6) Activate =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =20 :: @@ -188,7 +174,7 @@ Examples:: mkfs.ext4 /dev/zram1 mount /dev/zram1 /tmp =20 -8) Add/remove zram devices +7) Add/remove zram devices =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D =20 zram provides a control interface, which enables dynamic (on-demand) device @@ -208,7 +194,7 @@ execute:: =20 echo X > /sys/class/zram-control/hot_remove =20 -9) Stats +8) Stats =3D=3D=3D=3D=3D=3D=3D=3D =20 Per-device statistics are exported as various nodes under /sys/block/zram<= id>/ @@ -228,8 +214,6 @@ mem_limit WO specifies the maximum amount of m= emory ZRAM can writeback_limit WO specifies the maximum amount of write IO zram can write out to backing device as 4KB unit writeback_limit_enable RW show and set writeback_limit feature -max_comp_streams RW the number of possible concurrent compress - operations comp_algorithm RW show and change the compression algorithm algorithm_params WO setup compression algorithm parameters compact WO trigger memory compaction @@ -310,7 +294,7 @@ a single line of text and contains the following stats = separated by whitespace: Unit: 4K bytes =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D =20 -10) Deactivate +9) Deactivate =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =20 :: @@ -318,7 +302,7 @@ a single line of text and contains the following stats = separated by whitespace: swapoff /dev/zram0 umount /dev/zram1 =20 -11) Reset +10) Reset =3D=3D=3D=3D=3D=3D=3D=3D=3D =20 Write any positive value to 'reset' sysfs node:: diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 6239fcc340b6..dd987e3942c7 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1096,27 +1096,6 @@ static void zram_debugfs_register(struct zram *zram)= {}; static void zram_debugfs_unregister(struct zram *zram) {}; #endif =20 -/* - * We switched to per-cpu streams and this attr is not needed anymore. - * However, we will keep it around for some time, because: - * a) we may revert per-cpu streams in the future - * b) it's visible to user space and we need to follow our 2 years - * retirement rule; but we already have a number of 'soon to be - * altered' attrs, so max_comp_streams need to wait for the next - * layoff cycle. - */ -static ssize_t max_comp_streams_show(struct device *dev, - struct device_attribute *attr, char *buf) -{ - return scnprintf(buf, PAGE_SIZE, "%d\n", num_online_cpus()); -} - -static ssize_t max_comp_streams_store(struct device *dev, - struct device_attribute *attr, const char *buf, size_t len) -{ - return len; -} - static void comp_algorithm_set(struct zram *zram, u32 prio, const char *al= g) { /* Do not free statically defined compression algorithms */ @@ -2533,7 +2512,6 @@ static DEVICE_ATTR_WO(reset); static DEVICE_ATTR_WO(mem_limit); static DEVICE_ATTR_WO(mem_used_max); static DEVICE_ATTR_WO(idle); -static DEVICE_ATTR_RW(max_comp_streams); static DEVICE_ATTR_RW(comp_algorithm); #ifdef CONFIG_ZRAM_WRITEBACK static DEVICE_ATTR_RW(backing_dev); @@ -2555,7 +2533,6 @@ static struct attribute *zram_disk_attrs[] =3D { &dev_attr_mem_limit.attr, &dev_attr_mem_used_max.attr, &dev_attr_idle.attr, - &dev_attr_max_comp_streams.attr, &dev_attr_comp_algorithm.attr, #ifdef CONFIG_ZRAM_WRITEBACK &dev_attr_backing_dev.attr, --=20 2.48.1.362.g079036d154-goog From nobody Mon Feb 9 02:42:15 2026 Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B22681714B2 for ; Fri, 31 Jan 2025 09:07:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738314447; cv=none; b=tgEMOT9aAAHc3FwN/CfFsvwV8HT4vEd/XxxgoSYmXjWj8TSfjGbr07KuFy/6ZkmHaykwrYmnR02EUmpbxCDmi1Jph61L6NLBV3v556qUMsZxlBUdXhivOgPe95ZuijXxk+5PXdwUotuZeOOyw0kqusIAt2dj47GyZABAd56cUOA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738314447; c=relaxed/simple; bh=wPi2H7baNCj6Tz7kl/ktVTPftjIevFwXOVvyQijZJ+8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=rn38id3AQada/4wmq8JVwIIjmPbwB+fgku9Ji5UTG4Ax3jFDXG8Vr9XiatJZGe3LQ7ObQlKcl52V/zu7yi61P9eBwZIvtwiESLN377K6tenAIHDKTbrAg0jAjGdvivpLTH3/dxkVdp6nmSv/coECp9Jtx7lmDPjgTGX/wuHLJ38= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=h/vX3dtH; arc=none smtp.client-ip=209.85.214.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="h/vX3dtH" Received: by mail-pl1-f171.google.com with SMTP id d9443c01a7336-2164b662090so33870025ad.1 for ; Fri, 31 Jan 2025 01:07:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1738314445; x=1738919245; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=MtsIOf/TBIswacsBKgt0lP8Ca9NRj+2uNLPxrfHTdS0=; b=h/vX3dtHXDwKY+3w3Ai76lY4Eh4NvDKUcg7uZbMIU9hTc5ZRr2iQMnpxk+1dcI89mm MoJEnGlqmOV7TPaUjnRxqXYdTi7kyI0O7cwGlXgnQKplIZJ4jgdo8AeYa6murIrNqoe/ XYhYVrfk0R5Trui+7KerBZqkFIw5PwiqB4Jtw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738314445; x=1738919245; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=MtsIOf/TBIswacsBKgt0lP8Ca9NRj+2uNLPxrfHTdS0=; b=a/h2Kago9G8JrRr6CCFb6kDjJi7tuUtGg8WZD89/NPWwQmJAmt3FeFRh24rNfufOBo bfbrWg+vUFUlgnBMswk5nhCnv4OqVrYzpi/eN9cCI+qIK2mXmgyZddbNDLHdvw235JWv LtLdpZ62q2SDSKVB6en2qyOu77rthsJ4HKl6IyZFpqVTZ8DcKoW+tpOn5m9LJefcO7B4 nie+5Xp+osh5TSaCoEtBRr4AVbV48OxPe3gI4SsK44qQleXR3eppvbJgLL+NDjjgTE9l YO4yIS83G1CjToXMy5pkn53W09+YzprvJRckydJj1J8PAk7yO2/0qwhqNK6yVJQUYPxn Vj9g== X-Forwarded-Encrypted: i=1; AJvYcCXJKMQS8UHGEUr6LAhHVrv63+7hYws9nMr5nFee52Jv8Srn0WSar1s04AIah/riqadp9bkT4ECuqVQCGM8=@vger.kernel.org X-Gm-Message-State: AOJu0YxxqBUbZqYJquBBK7VNsLaaOAUKWBA5gdN3+xyxd4KCQXHcj3Tm iK/Z17WS7M4IiPC9aV3hP6wM9RJkGpoX6/VYOZ+qtfl0/hOHVschJp6xpZ4xIA== X-Gm-Gg: ASbGncujIw2J85frahYDOLbkN0R4wUwYXflB/l6GwosoWNmJnnBb7mdXRsZ+EML+4PZ E8kuRD/0jSutvrsi684zCc1OrkRpwLWf9uAH5K/4TNtis/EeSwlJYtQEDPgJvcTWxg+PY1cGlZv fiZO+4XeWK0Pz6wWfiv+Kuqff61Ok2YMcHIwyV4LYJJP41ez/VV1xE1DVIc4LvGR2FGo8lVByq+ pwUlSS5NdaHhevt+DWj4cOV1w9uH9BKTrtqfvvAZDuUrbYKs2gY0mW03TJ1dRGHyNtJCtJVvsd7 UM6eBVgiy6aMo7L2Sg== X-Google-Smtp-Source: AGHT+IG1fS5h4JbjO4Mc37VY88UPXs9TdEujSTlHdi2szESpBE3nfr8tk5rye/+mNhmuHHPPmpGn9w== X-Received: by 2002:a17:902:c40a:b0:212:63c0:d9e7 with SMTP id d9443c01a7336-21dd7b61a82mr166145155ad.0.1738314444996; Fri, 31 Jan 2025 01:07:24 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:c752:be9d:3368:16fa]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-21de32eb9f1sm25752285ad.115.2025.01.31.01.07.23 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 31 Jan 2025 01:07:24 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv4 05/17] zram: remove two-staged handle allocation Date: Fri, 31 Jan 2025 18:06:04 +0900 Message-ID: <20250131090658.3386285-6-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.362.g079036d154-goog In-Reply-To: <20250131090658.3386285-1-senozhatsky@chromium.org> References: <20250131090658.3386285-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Previously zram write() was atomic which required us to pass __GFP_KSWAPD_RECLAIM to zsmalloc handle allocation on a fast path and attempt a slow path allocation (with recompression) when the fast path failed. Since it's not atomic anymore we can permit direct reclaim during allocation, and remove fast allocation path and, also, drop the recompression path (which should reduce CPU/battery usage). Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 41 +++++------------------------------ 1 file changed, 6 insertions(+), 35 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index dd987e3942c7..0404f5e35cb4 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1716,11 +1716,11 @@ static int write_incompressible_page(struct zram *z= ram, struct page *page, static int zram_write_page(struct zram *zram, struct page *page, u32 index) { int ret =3D 0; - unsigned long handle =3D -ENOMEM; - unsigned int comp_len =3D 0; + unsigned long handle; + unsigned int comp_len; void *dst, *mem; struct zcomp_strm *zstrm; - unsigned long element =3D 0; + unsigned long element; bool same_filled; =20 /* First, free memory allocated to this slot (if any) */ @@ -1734,7 +1734,6 @@ static int zram_write_page(struct zram *zram, struct = page *page, u32 index) if (same_filled) return write_same_filled_page(zram, element, index); =20 -compress_again: zstrm =3D zcomp_stream_get(zram->comps[ZRAM_PRIMARY_COMP]); mem =3D kmap_local_page(page); ret =3D zcomp_compress(zram->comps[ZRAM_PRIMARY_COMP], zstrm, @@ -1743,8 +1742,6 @@ static int zram_write_page(struct zram *zram, struct = page *page, u32 index) =20 if (unlikely(ret)) { zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP], zstrm); - pr_err("Compression failed! err=3D%d\n", ret); - zs_free(zram->mem_pool, handle); return ret; } =20 @@ -1753,36 +1750,10 @@ static int zram_write_page(struct zram *zram, struc= t page *page, u32 index) return write_incompressible_page(zram, page, index); } =20 - /* - * handle allocation has 2 paths: - * a) fast path is executed with preemption disabled (for - * per-cpu streams) and has __GFP_DIRECT_RECLAIM bit clear, - * since we can't sleep; - * b) slow path enables preemption and attempts to allocate - * the page with __GFP_DIRECT_RECLAIM bit set. we have to - * put per-cpu compression stream and, thus, to re-do - * the compression once handle is allocated. - * - * if we have a 'non-null' handle here then we are coming - * from the slow path and handle has already been allocated. - */ + handle =3D zs_malloc(zram->mem_pool, comp_len, + GFP_NOIO | __GFP_HIGHMEM | __GFP_MOVABLE); if (IS_ERR_VALUE(handle)) - handle =3D zs_malloc(zram->mem_pool, comp_len, - __GFP_KSWAPD_RECLAIM | - __GFP_NOWARN | - __GFP_HIGHMEM | - __GFP_MOVABLE); - if (IS_ERR_VALUE(handle)) { - zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP], zstrm); - atomic64_inc(&zram->stats.writestall); - handle =3D zs_malloc(zram->mem_pool, comp_len, - GFP_NOIO | __GFP_HIGHMEM | - __GFP_MOVABLE); - if (IS_ERR_VALUE(handle)) - return PTR_ERR((void *)handle); - - goto compress_again; - } + return PTR_ERR((void *)handle); =20 if (!zram_can_store_page(zram)) { zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP], zstrm); --=20 2.48.1.362.g079036d154-goog From nobody Mon Feb 9 02:42:15 2026 Received: from mail-pl1-f175.google.com (mail-pl1-f175.google.com [209.85.214.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 981191A4F22 for ; Fri, 31 Jan 2025 09:07:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.175 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738314451; cv=none; b=ZIBfzR9lXFO1Az7Vf1VBdUgNU7fkL4ajuxnB1N9INmtIkX9hsM1i3MFfRfZkswPC2EKdUzK9ReqOYmSzbN7vCDBJ75jMpApyU/gBV37j4O8x17sMuyE3hyPhm/YxcfVFO+mrWbeKo3oV/f6guPp7t9/lS8cOwPSTPZYtSBo4eIk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738314451; c=relaxed/simple; bh=PxF4UB7sFqCoWDXL7GdQZVe6pIKAQICwHDrxpS/XzVk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=E02RfpQgjSewVC0s+gf4RyhjESbunehbprl2N3mmXuxkVe68GOvqUB0hZIi2sELtSfX3f1HMZLf4nUiaSppSzqvay5VkahUSny5uikP5o/2piYUXO9Twk4zRTn2MXODfcTbWw3eegzZRyfpB5TO19wGQM+ntOL9kosNZfAO3DDM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=Paj7LqZA; arc=none smtp.client-ip=209.85.214.175 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="Paj7LqZA" Received: by mail-pl1-f175.google.com with SMTP id d9443c01a7336-21644aca3a0so39626225ad.3 for ; Fri, 31 Jan 2025 01:07:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1738314449; x=1738919249; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=loywN5MAU0/S7v3h/0kjXpkRnG0c8JJBzlP3UvQhSOM=; b=Paj7LqZA4/s8royIhodXUoUAOEd1pieLjaNCmkUnjU3kNwkVagdgWDFW10rrz80fOG c1jCgYdm7lrfzPMFcrFyZzBhyVyprquMQkTmjk7goz3OwsVY4S/BFxOq+Gm0C/Wnwdtk 6DgJLTvI6yfEFyEkGQ8OO3a395h3pgNPL9m8E= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738314449; x=1738919249; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=loywN5MAU0/S7v3h/0kjXpkRnG0c8JJBzlP3UvQhSOM=; b=jce/v9NNVm4q6QQtTnxslmCp37rIQNF7Vr2QPivjkzhwStuolA9Dov9LAsXvJze1/n wU8u7+01f9RcwPcjDAFAigKawj+5mx6HFDH+Fp7+nDR25ewybnjU2afZoG/ECgX2MdHL e+mB2H1MmwzPXE/gBjkzJZhbr/ZDC6H1WBHgNOstYs9wPVnsDXj/3PJweROydHlM4XFq uZCOcj1jO/cw9YWrt7aFA0DLgAMyVsSpCYeOifEyFIBUkdepvtMQq/Bhf/sjw20NSbiS FcmGCB2pYKuKdCfssH2rxj3+HwsJGSrs//F5N1ydEOBALGHssronKEmkmHFnnvEBEMQO skDw== X-Forwarded-Encrypted: i=1; AJvYcCV4NdfgNOiePozmoFoEvf5IS87+O26EqS52XNZ0/Hpjux+IJBworRyanCBrba/aBoMY6q1hFI+jEcMeqrk=@vger.kernel.org X-Gm-Message-State: AOJu0Yy9Ibk7J66uP68RPYlHv0QAEYMHK9ZlRdmAkfinECmn6BE830oa 4xwGtiXlNmVUjHk5rrDglD9pIkW5xw9I9rMfpTEFU4QmNSiVeAoaMSRc5w6Feg== X-Gm-Gg: ASbGncsMJZTOY7pEuEs3kXOPrsY+EUwHyVqRHi9cEYHUVrI0eAsu+eutLgVyjd0dAMf m999v5pZ1gKsz+fwZH+PWyChJVqu7t9+bA8NUj5rjOsw+DMHhRc/f43FFb+bnXRzi9ngC0Zikw6 6BO21YpHG+TcGEyeTzFOu7bzINXA26UzgRBv1isY52x6UByKp7p2kFVq7mCL/baisCSpF/1VJCV /S3djJs6fDsrwKDPF4G7xNYTpyGjhzHFV5qzmnKA2npDQrTkhVYW024iuxKT+O0GEpdXJvq0jqN yIqjAaKl7LQy6Do0UQ== X-Google-Smtp-Source: AGHT+IF+0FKD5xX6XHXYkP+petrD53hSmc7lcGie6WBqsrwN7A64ICQ6XETnp7Q7BlhjDUtZHY0kTg== X-Received: by 2002:a17:902:cf0a:b0:215:a80b:f6f9 with SMTP id d9443c01a7336-21dd7c449ffmr131180925ad.8.1738314449146; Fri, 31 Jan 2025 01:07:29 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:c752:be9d:3368:16fa]) by smtp.gmail.com with UTF8SMTPSA id 98e67ed59e1d1-2f83bcd0d9csm5896824a91.14.2025.01.31.01.07.27 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 31 Jan 2025 01:07:28 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv4 06/17] zram: permit reclaim in zstd custom allocator Date: Fri, 31 Jan 2025 18:06:05 +0900 Message-ID: <20250131090658.3386285-7-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.362.g079036d154-goog In-Reply-To: <20250131090658.3386285-1-senozhatsky@chromium.org> References: <20250131090658.3386285-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" When configured with pre-trained compression/decompression dictionary support, zstd requires custom memory allocator, which it calls internally from compression()/decompression() routines. This was a tad problematic, because that would mean allocation from atomic context (either under entry spin-lock, or per-CPU local-lock or both). Now, with non-atomic zram write(), those limitations are relaxed and we can allow direct and indirect reclaim during allocations. The tricky part is zram read() path, which is still atomic in one particular case (read_compressed_page()), due to zsmalloc handling of object mapping. However, in zram in order to read() something one has to write() it first, and write() is when zstd allocates required internal state memory, and write() path is non-atomic. Because of this write() allocation, in theory, zstd should not call its allocator from the atomic read() path. Keep the non-preemptible branch, just in case if zstd allocates memory from read(), but WARN_ON_ONCE() if it happens. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/backend_zstd.c | 11 +++-------- 1 file changed, 3 insertions(+), 8 deletions(-) diff --git a/drivers/block/zram/backend_zstd.c b/drivers/block/zram/backend= _zstd.c index 1184c0036f44..53431251ea62 100644 --- a/drivers/block/zram/backend_zstd.c +++ b/drivers/block/zram/backend_zstd.c @@ -24,19 +24,14 @@ struct zstd_params { /* * For C/D dictionaries we need to provide zstd with zstd_custom_mem, * which zstd uses internally to allocate/free memory when needed. - * - * This means that allocator.customAlloc() can be called from zcomp_compre= ss() - * under local-lock (per-CPU compression stream), in which case we must use - * GFP_ATOMIC. - * - * Another complication here is that we can be configured as a swap device. */ static void *zstd_custom_alloc(void *opaque, size_t size) { - if (!preemptible()) + /* Technically this should not happen */ + if (WARN_ON_ONCE(!preemptible())) return kvzalloc(size, GFP_ATOMIC); =20 - return kvzalloc(size, __GFP_KSWAPD_RECLAIM | __GFP_NOWARN); + return kvzalloc(size, GFP_NOIO | __GFP_NOWARN); } =20 static void zstd_custom_free(void *opaque, void *address) --=20 2.48.1.362.g079036d154-goog From nobody Mon Feb 9 02:42:15 2026 Received: from mail-pl1-f178.google.com (mail-pl1-f178.google.com [209.85.214.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 79AF41AF0BB for ; Fri, 31 Jan 2025 09:07:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738314454; cv=none; b=nG7jtXnJALcmJuFwbutf2wxA9lA0uS1YFzN4ljLyUDlBmjN2BO8lscgeP89uKbZi0jbiAYDvYIMnXz0dfBclJ1KILhLgMM7nqY9YJ90HKeqyIZktR3GreGW/daun/vt59HNU8fceJXaYWv/6jNPn8sy+NVPDPXltZfqajEld7N0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738314454; c=relaxed/simple; bh=CXhCh51ip+KADorGgBF7on/wwxCRKVCWCcm6nJuZ0l8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=NJVDZjJdYuIBxdvSq6RIl/kIgydYSucMAPO0+mmibdAxNw5kPI9hJ2/1r2wzWBykWtK3LFucTMA7Usy1cWMMvAp+JaMDxbpF4cTO+nnB6YRx0IPfzHQqHZ1jeuPUfyErAaHH9C6rkvUpWhdWRoRiIKyOOYluFWLhALA7bP00Mek= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=Ye5Apf6u; arc=none smtp.client-ip=209.85.214.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="Ye5Apf6u" Received: by mail-pl1-f178.google.com with SMTP id d9443c01a7336-21644aca3a0so39627115ad.3 for ; Fri, 31 Jan 2025 01:07:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1738314453; x=1738919253; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=yIS0Akli6K3ICDNolLXeCGhuZ0MDBrb+zo33V4R+vJw=; b=Ye5Apf6uMCUfVApfV/D6C78izOmXFDhxHH57Y44IaOwbwl3hz4ihNBUxKgX4dT0XSl f2e/ihqwZvGVf5zONIQrPCw2JJULtH77d1dX+4BTualwDkQewfYZlr5840GhNIhHT7nv T9x6Rtbh9zWI3dEwmlSDgWsToXPVyPCZlziF4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738314453; x=1738919253; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=yIS0Akli6K3ICDNolLXeCGhuZ0MDBrb+zo33V4R+vJw=; b=B+v4fch/WBvapaI7vva2nuQXM32fD/gQNv2Wdhm/xVusb0v34nwfw4iAovp3CrkrEb wQsgyhPb+Ie2cp8i7Ut7D4EyBrKQW5jGSRZ8HIeKJeXwuNo+oCfBoi3hwjmbc6GSXUY7 B6SE8jG+wJS0bXupwXLWPzH1BY1FEL5UHcQ+fd32cG0M0FvRBH5VsdCJE38AZqVCpo0r 2HD1vpYYXc4EUVbwaQZ8WWjlvFCcgIMLmthanAaqtoQXTEOmK6xvMjeBo3EmAWREOVWN XdMMbvXqpstxdYqFlebO82tGq2Xy2UItPq6lmlmFexrSTl1Fx6iWrezNCVmWWzVkEXEf qW/Q== X-Forwarded-Encrypted: i=1; AJvYcCUk0WHmY11KFJEPU/U37QDLYUY3UUwI4EE2gw8xaS/aBdxYhVkZbKCSnF/ox50M46xHW9DCd9WhnucmDXM=@vger.kernel.org X-Gm-Message-State: AOJu0YxWEk/lD7akNdTaSWzXqZ67rsKxshZbWpAoMmAizGLl0fQGm7nq 43e2TS9loWEHUYrVclI4WraOtU1K9g5tXBm1xLnFpMde1iSs9zXt8OL3sy9OUg== X-Gm-Gg: ASbGncuAit5Tn6uyMbRykKCjF/MoHQdXEMSvNW9ojnFN+z0eUD8/hx1DyZ7Xzpekhjq rn1y1yEIseUm+B6qGxB24WOfU8JpZVtl7v9+ReXOgSX2kMKLJCJBaQF1lYfEmJrPFqQshD0etLc iqsfSHgFauBnGY85w10AofC4LQmdULhPW9H0jcNWeaF2ck5kVYTz9YYiVhfRYxG5AberqmvXlV3 eT31IUV1HHcvDFeF+CPT3REXktsSeNKTTik4XbKcIwRDeJK+/fb2t70UEmWRqxjij3K8PKDm04D oEDI5xeb5w0p0cICVQ== X-Google-Smtp-Source: AGHT+IGDstI0UvROyDnrpX/fcD3zobh8n0WWgBv7PtyE0EXq8xnCiPT08ZRIDCaJlns/HLlSY9XhVw== X-Received: by 2002:a05:6a20:3d86:b0:1e1:bf32:7d3a with SMTP id adf61e73a8af0-1ed7a6c8c72mr16361840637.26.1738314452868; Fri, 31 Jan 2025 01:07:32 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:c752:be9d:3368:16fa]) by smtp.gmail.com with UTF8SMTPSA id d2e1a72fcca58-72fe6a1a75bsm2781771b3a.167.2025.01.31.01.07.31 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 31 Jan 2025 01:07:32 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv4 07/17] zram: permit reclaim in recompression handle allocation Date: Fri, 31 Jan 2025 18:06:06 +0900 Message-ID: <20250131090658.3386285-8-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.362.g079036d154-goog In-Reply-To: <20250131090658.3386285-1-senozhatsky@chromium.org> References: <20250131090658.3386285-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Recompression path can now permit direct reclaim during new zs_handle allocation, because it's not atomic anymore. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 12 +++--------- 1 file changed, 3 insertions(+), 9 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 0404f5e35cb4..33a7bfa53861 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1979,17 +1979,11 @@ static int recompress_slot(struct zram *zram, u32 i= ndex, struct page *page, return 0; =20 /* - * No direct reclaim (slow path) for handle allocation and no - * re-compression attempt (unlike in zram_write_bvec()) since - * we already have stored that object in zsmalloc. If we cannot - * alloc memory for recompressed object then we bail out and - * simply keep the old (existing) object in zsmalloc. + * If we cannot alloc memory for recompressed object then we bail out + * and simply keep the old (existing) object in zsmalloc. */ handle_new =3D zs_malloc(zram->mem_pool, comp_len_new, - __GFP_KSWAPD_RECLAIM | - __GFP_NOWARN | - __GFP_HIGHMEM | - __GFP_MOVABLE); + GFP_NOIO | __GFP_HIGHMEM | __GFP_MOVABLE); if (IS_ERR_VALUE(handle_new)) { zcomp_stream_put(zram->comps[prio], zstrm); return PTR_ERR((void *)handle_new); --=20 2.48.1.362.g079036d154-goog From nobody Mon Feb 9 02:42:15 2026 Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5CABA18CC1D for ; Fri, 31 Jan 2025 09:07:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.180 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738314458; cv=none; b=DUcgCxI++hIl+QBKnv7PRN49vOA+iOFd5EnegrA5pNq43tbdY6Zcd5m32WWcmAdkdUYOnIXJNaKQx7BJ2W2uW8huVZDr62u1a+hj9kJO7jZ0c8qw3+ehW2peeEQG2SyfxfTpdj9+xEP89pbhdsEtvgeRaS+aetl7oaT63s7AH0g= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738314458; c=relaxed/simple; bh=eYElRTtxdEwu20akPtpsJLzQq/4V058HHKoMVhHbTXs=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=qPVEAqWin8s2vprXiJrhgOYXjSri55RV4IA0SG3IWvQOEeqRXxzjXEpK3PpXdPAe5bPTB8hYZfZhukkIHJkBl+/ylkhe46w3xZ7TRVXwJy6dAecg0OS8R6HKVOhtKC/9JK6JZEUH1tGSvcRnYL3Z1krykPk0YvPCFe3ge9gr/F4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=PTQnPOlq; arc=none smtp.client-ip=209.85.214.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="PTQnPOlq" Received: by mail-pl1-f180.google.com with SMTP id d9443c01a7336-216728b1836so30036365ad.0 for ; Fri, 31 Jan 2025 01:07:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1738314456; x=1738919256; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=oU99tlqEx3VP4Fr6vsoC6pVJUm4rK42GNNjPkPWz9Uo=; b=PTQnPOlqcWCNfBZv51PKRvkCzMlxlWHY6N4F8+NMBHlbBwjGxyaoD5GClzy4XQF+1Z AzWkOrKfQ8SuSTHb+0ixKmIYZ+LzBuPnyg2tv16ae90CjPrLtK1vwb8/1bcOaisoK6o7 4RI02xKaeQupOQJuBNGMVBaVmtsLc7gGFqs6Q= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738314456; x=1738919256; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=oU99tlqEx3VP4Fr6vsoC6pVJUm4rK42GNNjPkPWz9Uo=; b=N4jdKliq/MgUxo+kaefTUZ30RS6oRCep0DQ0HYmM4sAN/ivvh9ZcXpPDVDzqUW/o7L A0zac4wYJYnoYagi2OSbMi03686GUrgc/jVnpnLJ5Ff9QMnv8py4g4RCEVjBmOGAH27h vyy33pEUF2Wrc8QZ9husoPvuvjh+5LWVjeYPWvSJmza6GdW2boxJfK17PnP51zcy8AcK 0trqTzuiGl6F57uC7m58zueKXONlJuVK3im+CmW3ohRM0t+PukVZ1vy8bgmCoKawwIOB fyaGWgDR/gYmFcVkHS3xNKvvCYH3EtB33QcNYg4E4YACn4tUIZFCpQgk5yJ5FKqP9RP6 E0Sg== X-Forwarded-Encrypted: i=1; AJvYcCXxkzxMrToFlBU21ZKCuCUNV2fm6797AaSJm8oBjh6yO1tsTCGE6zQOfjPAbGLS35MwE16w9wg3KZ1mgEg=@vger.kernel.org X-Gm-Message-State: AOJu0YyE4FFexM5h5yd8ZgyLI9cTEQtvq7JzGEez96gND8Y++Gh75Rs6 dQDQ8hVg+gt1Bmu7RYRzrf6hFFgfyn5dIqOrFCJdzr13wtl8Ejm/mDpCOvaoxQ== X-Gm-Gg: ASbGnctVoK2IofbUXdN6AKhBRS3kSzaL31SH4/wf5vbq3wgU6YAm3ySrsz50SynTYr1 KqanYSN2h1OkdY2ZOjhmbbl286i/kA0UUDMmDks+Bl9INUnclPXowRm0LVTvZn9Iz+Pvv21au1k jzkisdkwRuVEfOACqWr8cH9QPUv3e7Qu94OZxH9zDQLgZRpGtYSk7D+HwZO8nSUFSXC0ItZRFzj oRMuyUBc2tJU4zYkUt/hmYNxdBEec3BB6NxiPuU8eWzezSxLXDAOdz0RxQFsTN0WrdLDInoaGLx S064kMCc5O9X7rHNcQ== X-Google-Smtp-Source: AGHT+IGQyeOJINoq+3olFAJs8BwSF3tpMesiAuDGnBGSVFEb9/qgBFf2sNzJkKESZdssbJAVfH6Lfg== X-Received: by 2002:a05:6a21:33a8:b0:1db:915b:ab11 with SMTP id adf61e73a8af0-1ed7a640d42mr17388523637.24.1738314456497; Fri, 31 Jan 2025 01:07:36 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:c752:be9d:3368:16fa]) by smtp.gmail.com with UTF8SMTPSA id d2e1a72fcca58-72fe6a1a74esm2745081b3a.159.2025.01.31.01.07.34 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 31 Jan 2025 01:07:36 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv4 08/17] zram: remove writestall zram_stats member Date: Fri, 31 Jan 2025 18:06:07 +0900 Message-ID: <20250131090658.3386285-9-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.362.g079036d154-goog In-Reply-To: <20250131090658.3386285-1-senozhatsky@chromium.org> References: <20250131090658.3386285-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" There is no zsmalloc handle allocation slow path now and writestall is not possible any longer. Remove it from zram_stats. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 3 +-- drivers/block/zram/zram_drv.h | 1 - 2 files changed, 1 insertion(+), 3 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 33a7bfa53861..35fca4c468a7 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1430,9 +1430,8 @@ static ssize_t debug_stat_show(struct device *dev, =20 down_read(&zram->init_lock); ret =3D scnprintf(buf, PAGE_SIZE, - "version: %d\n%8llu %8llu\n", + "version: %d\n0 %8llu\n", version, - (u64)atomic64_read(&zram->stats.writestall), (u64)atomic64_read(&zram->stats.miss_free)); up_read(&zram->init_lock); =20 diff --git a/drivers/block/zram/zram_drv.h b/drivers/block/zram/zram_drv.h index 3ae2988090b3..219d405fc26e 100644 --- a/drivers/block/zram/zram_drv.h +++ b/drivers/block/zram/zram_drv.h @@ -79,7 +79,6 @@ struct zram_stats { atomic64_t huge_pages_since; /* no. of huge pages since zram set up */ atomic64_t pages_stored; /* no. of pages currently stored */ atomic_long_t max_used_pages; /* no. of maximum pages stored */ - atomic64_t writestall; /* no. of write slow paths */ atomic64_t miss_free; /* no. of missed free */ #ifdef CONFIG_ZRAM_WRITEBACK atomic64_t bd_count; /* no. of pages in backing device */ --=20 2.48.1.362.g079036d154-goog From nobody Mon Feb 9 02:42:15 2026 Received: from mail-pj1-f52.google.com (mail-pj1-f52.google.com [209.85.216.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 16C25192B66 for ; Fri, 31 Jan 2025 09:07:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.52 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738314464; cv=none; b=k6kBb4ZEOUgChTrMZa8bREcMmhiUpmlJjh77pRGGrPhuixHUXNHBcZ3tUWS1xiINnN9QK5R0HP6aJZI0pLlHQpvuLNJWhZEJjFg5Grx/CMBa+/y08W8xz0/TpF1D/Lgz9o3OZ3RaB96RoQ+NR8BPtp06e2/WBGdnrRwpty+iIxA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738314464; c=relaxed/simple; bh=7ZGwiL1tXOX3hLxvsu4O7FCgVEDZyaXHlMHsK+0BEyU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=BrRt3QF+DD3/kJq8ZqfTsGWKDh1PWUNa3ncVfenJnL8h9eBnX5sNWgRV36wlfu5W/D+6Ej2A7wG7tUM1LWwxDYzhoMv6fFhEkUxy4dvBaT9O9+7E//iwh2FmCPeVqONdm1awvtQnsBGtzCuRwtKKt2V77y1gPux3Z5AvveGC4Qo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=HljfGqT5; arc=none smtp.client-ip=209.85.216.52 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="HljfGqT5" Received: by mail-pj1-f52.google.com with SMTP id 98e67ed59e1d1-2ee397a82f6so2987389a91.2 for ; Fri, 31 Jan 2025 01:07:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1738314461; x=1738919261; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=i68glYQZ+E+noE0wy+Y9J0JFMp7tHf3uQmMwsu8iKww=; b=HljfGqT5fzaLszMgCsh75lva2gI5nxQnvLyZYY2NIP6wleqxZGrxqPfmZ4JjOBnZY0 wShkjQy0m8F8mtXu/BBkDl63pHQRe14gu3SsJd97SsJhKrvxfB+qINyMJxWFR0TWLb7w SyPyjwz+VAVFHGDcpk57UnV5m7s+j2Kl3lh6s= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738314461; x=1738919261; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=i68glYQZ+E+noE0wy+Y9J0JFMp7tHf3uQmMwsu8iKww=; b=cUPadZUiSZel16rdILt1g9vHwmwp2GSmF2xo5DIb2dHG2GPe3zlO0Fjp+NKJJiSCzq XFvDspeuhSJ9+a0bnsnW/YtBh/jICdWB0Wdhzu8wqgAd340vBh8OGUMfBm6pNCyPSCHb UIIoVvLxh7ob0RaJrTvtbIVte35ZA4LBe02sWUzO5fQPA/r4xvQzof1QKDhTnAEaAKAw ca7Q5fv6HJkEM1wrX9EVWXjYMxFdrfrB+LAcRGAMWhq4A7++IcpllRDnK71DSTY4Kyxt 7S4jWNAdLuwCZF6r8PqCUMGAlHJx9vuBNY8wkSdrTDre5ys+kjj7zYDfARP4+ONNB2Pr LIhw== X-Forwarded-Encrypted: i=1; AJvYcCVrnQN6M+Fms4X4L37hFIKvdLdb0Ir0KBzBGpOwAi2NfZ54R0qJhSoGgXsgYbHyFIV4DEse9RGS9j+ZTr0=@vger.kernel.org X-Gm-Message-State: AOJu0YxGrSgS7Jw8UBf2SCTnjI6RhRPhvF3pO/7asCEidotQ94HUd/Yl 4N6hQrHXM/5+WNgidMSmZyEBPvTvDM1k/rAasCLDFm+USq6KqLzqCB/eAzR38Q== X-Gm-Gg: ASbGnctZAttAX9npQi4NvC5MwoUpHEasigCtluWTgDFwuyo5raEBcJKKYDGQ4PsGQ2l tP3StlmVT88hBb/+WYfbZq0O7QxmxVQZOnnfrj3WILizaYNeBG4n4LJi4YEqtqIMsnroILMyqhS jn2vgpF2tPvIDgEda/sBvqjZ9kGGgQHXU7ZYp7zAFqFlQWwyX79HiPmrfceKTapB4fIf8/GOyRz PczcIfRbx5j0WqCl4VF9ChD29JmBLKs0HLPG9w4Smo4JJCdhb5AD8RWPhxCwJ3PqBR+hv4BZorC gZmW6qCu/it3HsZ4/w== X-Google-Smtp-Source: AGHT+IEm5A+dR0l39EGWgA22kpqEPsUgTOFh4GGf32kwJ92J5tCR45G0PnUGYdzf2axpCJd9jEM3rw== X-Received: by 2002:a17:90b:2702:b0:2f4:4003:f3ea with SMTP id 98e67ed59e1d1-2f83ac84706mr17607671a91.33.1738314461307; Fri, 31 Jan 2025 01:07:41 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:c752:be9d:3368:16fa]) by smtp.gmail.com with UTF8SMTPSA id 98e67ed59e1d1-2f84897a504sm3109092a91.7.2025.01.31.01.07.39 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 31 Jan 2025 01:07:40 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv4 09/17] zram: limit max recompress prio to num_active_comps Date: Fri, 31 Jan 2025 18:06:08 +0900 Message-ID: <20250131090658.3386285-10-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.362.g079036d154-goog In-Reply-To: <20250131090658.3386285-1-senozhatsky@chromium.org> References: <20250131090658.3386285-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Use the actual number of algorithms zram was configure with instead of theoretical limit of ZRAM_MAX_COMPS. Also make sure that min prio is not above max prio. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 35fca4c468a7..c500ace0d02f 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -2009,16 +2009,19 @@ static ssize_t recompress_store(struct device *dev, struct device_attribute *attr, const char *buf, size_t len) { - u32 prio =3D ZRAM_SECONDARY_COMP, prio_max =3D ZRAM_MAX_COMPS; struct zram *zram =3D dev_to_zram(dev); char *args, *param, *val, *algo =3D NULL; u64 num_recomp_pages =3D ULLONG_MAX; struct zram_pp_ctl *ctl =3D NULL; struct zram_pp_slot *pps; u32 mode =3D 0, threshold =3D 0; + u32 prio, prio_max; struct page *page; ssize_t ret; =20 + prio =3D ZRAM_SECONDARY_COMP; + prio_max =3D zram->num_active_comps; + args =3D skip_spaces(buf); while (*args) { args =3D next_arg(args, ¶m, &val); @@ -2071,7 +2074,7 @@ static ssize_t recompress_store(struct device *dev, if (prio =3D=3D ZRAM_PRIMARY_COMP) prio =3D ZRAM_SECONDARY_COMP; =20 - prio_max =3D min(prio + 1, ZRAM_MAX_COMPS); + prio_max =3D prio + 1; continue; } } @@ -2099,7 +2102,7 @@ static ssize_t recompress_store(struct device *dev, continue; =20 if (!strcmp(zram->comp_algs[prio], algo)) { - prio_max =3D min(prio + 1, ZRAM_MAX_COMPS); + prio_max =3D prio + 1; found =3D true; break; } @@ -2111,6 +2114,12 @@ static ssize_t recompress_store(struct device *dev, } } =20 + prio_max =3D min(prio_max, (u32)zram->num_active_comps); + if (prio >=3D prio_max) { + ret =3D -EINVAL; + goto release_init_lock; + } + page =3D alloc_page(GFP_KERNEL); if (!page) { ret =3D -ENOMEM; --=20 2.48.1.362.g079036d154-goog From nobody Mon Feb 9 02:42:15 2026 Received: from mail-pl1-f178.google.com (mail-pl1-f178.google.com [209.85.214.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 443281B6CEB for ; Fri, 31 Jan 2025 09:07:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738314467; cv=none; b=HQYUhLY7utQo07pZYYPHoXUANINQCWj9uKJ4Hii9MOaB12OL9XkzE6fq9fyQVYTHROzYMnIYRPdssF8z90l+EnBakByzASPlnB8mNnWQysw3YTYFpUS+W0jAk1DuyXkh6OoBpcx6QPmG+2L26UnFx4cUwCRGnQgBlPjEr7sSnks= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738314467; c=relaxed/simple; bh=eKg6mm6l3aKoooXk3pWhI6SIgefA7dd1PtfT5tLmj5E=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=bsqEoBFuOINZBSIJQa6JDssRE+QVMcHUbsxOpkReID2vEAS1MMHvl0y4UIsNh175k0nR5VRp3smn7wyy3rmuUP2zc/JmxJ+cZVzZ3OEaDfwlf3Hl90kHYvifWQt0VsykE8XjrT4hFZeZlU77TfOHkCVn69gIqnf/RmOfPuLtTm4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=CIHYp6aO; arc=none smtp.client-ip=209.85.214.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="CIHYp6aO" Received: by mail-pl1-f178.google.com with SMTP id d9443c01a7336-215770613dbso22353865ad.2 for ; Fri, 31 Jan 2025 01:07:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1738314465; x=1738919265; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=4r0pcOBNOcRsnBO2Q2AoDymvU1L0RHp2karwxkPTEA0=; b=CIHYp6aOyl03BdQiF+ba2gtq6osaQPKCF4cmMkNdLQO2tBJb1f6ykgidyaWBVBbIpb NvodFN5rzimgMglMyL2+DPTSdHGSNN9sIFEev7GeJsVciBKFcWCZnacSpETs4o+jyH0Z MQ9oDXkfTO+qHCTqRh6z0VG6ZckOI/ob++RdQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738314465; x=1738919265; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=4r0pcOBNOcRsnBO2Q2AoDymvU1L0RHp2karwxkPTEA0=; b=JYn9TFKoEiapPfL/DQWJs0EUDOB9nap8nc01AwV4eJZ9Y1VFFwmg15FaRPCUptNp13 t3AYY0Y0x09ddl04v+1/iqjxt5g9e2p7FlHdQEvy/xPAVBaCCv/3o49Tqc12L8y69BjN iXa4F5IS6umW8YbC88ZDe1KfdGZ9aiXa11NY5XaluKLhtWFcbS1FGyqbNoHpq/iefi0t 8i5Hfnzr9DevqihL8ePozEPHIm5JCU76vSmxrDHj7i2jXBRh9use80IyCHzTyoUK0YAW I0FDBJrbiCVSc9vRBEK3E+UTsYc/KCWsIHsxU2js6WcSepM+LuxJNPnKjeQ4jP/ziErT 0Y1g== X-Forwarded-Encrypted: i=1; AJvYcCWusIlWcLRmuCi3eCZ4XzpieJ+vmkwXFRy6wM8QDtO8/6Jel4BprBz/3W+8Mo2krqiCO76Qad9tGYX9fn8=@vger.kernel.org X-Gm-Message-State: AOJu0Yzcmxjiu023y8aUhu3fK524ftk20Yna5QzkntP4HKxIjh4Re/wK 4pcepBJPG6RhHKQuO/GgfFxDy/wKA+apxABwSbGt8lQYj1KeX4T+KIuRyeZFVA== X-Gm-Gg: ASbGnct9Ng4EZmQgKxdxPYAPC5iFBsvP4em0gQdPlplhaUvPUME/uTltMmksTrFz5Yh WE44yruvcUZK6kGlRcFfrYWtP5T6isBrAFOe1B2MufErMuhoHmHqAf5lGAC8WGOGkNOhCBTCKH2 hjntzuznikdn7pFOtqLfXuTR3LpcyX4cJpX7zTluP5WiB8nbnDjRr+rfBHaT982+unhcIj37rX5 CtzBM510GZAIMVicJn16dN4XrJ5y3u3ZuBMKpcdZfnl+adWuFj3ehmnEVZ0LOkRFjLieJHI1TpX n8GNrDlTnUAxjZUhkA== X-Google-Smtp-Source: AGHT+IFVF6qQXP7j68CfVjgHMYTUsYqMzGDzrbpNfx9vlhjUIvaCbiwn0ENvclvQGAhdyEYDWrJB0Q== X-Received: by 2002:a05:6a20:2d06:b0:1e6:8f30:df53 with SMTP id adf61e73a8af0-1ed7a4dcd34mr15160965637.16.1738314465661; Fri, 31 Jan 2025 01:07:45 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:c752:be9d:3368:16fa]) by smtp.gmail.com with UTF8SMTPSA id d2e1a72fcca58-72fe69cd801sm2784910b3a.123.2025.01.31.01.07.43 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 31 Jan 2025 01:07:45 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv4 10/17] zram: filter out recomp targets based on priority Date: Fri, 31 Jan 2025 18:06:09 +0900 Message-ID: <20250131090658.3386285-11-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.362.g079036d154-goog In-Reply-To: <20250131090658.3386285-1-senozhatsky@chromium.org> References: <20250131090658.3386285-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Do no select for post processing slots that are already compressed with same or higher priority compression algorithm. This should save some memory, as previously we would still put those entries into corresponding post-processing buckets and filter them out later in recompress_slot(). Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index c500ace0d02f..256439361367 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1811,7 +1811,7 @@ static int zram_bvec_write(struct zram *zram, struct = bio_vec *bvec, #define RECOMPRESS_IDLE (1 << 0) #define RECOMPRESS_HUGE (1 << 1) =20 -static int scan_slots_for_recompress(struct zram *zram, u32 mode, +static int scan_slots_for_recompress(struct zram *zram, u32 mode, u32 prio= _max, struct zram_pp_ctl *ctl) { unsigned long nr_pages =3D zram->disksize >> PAGE_SHIFT; @@ -1843,6 +1843,10 @@ static int scan_slots_for_recompress(struct zram *zr= am, u32 mode, zram_test_flag(zram, index, ZRAM_INCOMPRESSIBLE)) goto next; =20 + /* Already compressed with same of higher priority */ + if (zram_get_priority(zram, index) + 1 >=3D prio_max) + goto next; + pps->index =3D index; place_pp_slot(zram, ctl, pps); pps =3D NULL; @@ -2132,7 +2136,7 @@ static ssize_t recompress_store(struct device *dev, goto release_init_lock; } =20 - scan_slots_for_recompress(zram, mode, ctl); + scan_slots_for_recompress(zram, mode, prio_max, ctl); =20 ret =3D len; while ((pps =3D select_pp_slot(ctl))) { --=20 2.48.1.362.g079036d154-goog From nobody Mon Feb 9 02:42:15 2026 Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DACA51B87D4 for ; Fri, 31 Jan 2025 09:07:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738314472; cv=none; b=RZu83KjVhE394+6h2JqHoYLJI+fpLhfrV3W6wU7+omWJkDYFtQfybex5rpsaxn1zo3q9c1yaaYyZZiQe9s4xnFhiCLGPdAdqwo+K/b79QhIXdpG6/JJLuacZEz7IUjPUK7KrKRDptMvpQ4qeM2QpyqBYAjAowinVq2/fB33oobA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738314472; c=relaxed/simple; bh=aNMRzj06u4cEjeWmxV1/VeVb2cpxVVX3UefcDx9ncnQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=TVJimHxi8LyINErFcdTnlC+PYuRit/WD5CIFyP7B7iBp2XmcNSQ2zVuYshzArQR/qDMtgmZoEdjbpqp/puH1rG9qrRfibV12p2GUMiPm7iuLQd40RKcCaC31KH2HFihMOd2tAEU+8beAPj/5XblK1eAQP5cMrwWV0Oz32uVn5Os= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=c/TGb4kv; arc=none smtp.client-ip=209.85.214.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="c/TGb4kv" Received: by mail-pl1-f182.google.com with SMTP id d9443c01a7336-215770613dbso22354545ad.2 for ; Fri, 31 Jan 2025 01:07:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1738314470; x=1738919270; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=w3R6ZrvyVvJXmo0SkC31+WiXIWMORonA8dJRkgz1Hic=; b=c/TGb4kv9FarqgvxsX0l3Ons3yJ2nYHfAE/EipkVhajBffLdBrFkDuUN8HuwIFTWlz nRMNdqpc+MHOmV67IW23mfK++o6prbGSzlVaFICJDochX557tVArDy0k14YA7GAr+AgK o54zexk29vkH0RGr/DUb6p2wfTmwoNA5opib4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738314470; x=1738919270; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=w3R6ZrvyVvJXmo0SkC31+WiXIWMORonA8dJRkgz1Hic=; b=aZBdfBn+1ANDj5Y5U8CraGaBfARLt/RCODONkpKQpIl+BOFhTMySVffzrq438uecmN ZYqhIcKYJUk+1KFeFSSP1qCm1Cjp+/+9FC6EhiZlgG1H2dKscdBcixY92DK7Ep9qAXSg cPM7pVPnZKMBLAq1wNsN3JD0JDcpCxp/DG8gbP3o0kl9qAYKhX5cP3TtR97/1LKDAAC2 QTEbqVsT4i/h9TfiL6Q4yYP0GsjoZGD90Ffk0fTT2fbLO8qTCBeJt08r5kqsNDS5yyCA 4PrI7Z5z39OMtGWetayKOInKaRCXRdkisNuxuDhd2V0nF4CLnp6sH4vFBmoR+a27KtKo sQxw== X-Forwarded-Encrypted: i=1; AJvYcCW9jgVjNRq2FaLaHwB9vTbzbLGV9adjV2U83UIx1Ae4cPyId+sqtGq05nY71EvjCCHZzvvmH35HBi8fkRo=@vger.kernel.org X-Gm-Message-State: AOJu0YxTEJkm7yqQ+/YW0FezfoZtN/JQJEq4cq3sDhbWqU0E4/A1JLYr wGGXKJfFNJfehQYKb0swVM24WFmGRU/tBSqnzobll2zBMS5KIP7INCVYrPM5wg== X-Gm-Gg: ASbGnct5DDeEuGWlJSr0c6VcaF+wgfqRs35ySGj3Q4alsRV2Hui7l2pOxhZ3fsfclGC K4kurF/e1JPcu0Nm/QyNwWS50ax3GOa1VbvQoP5qJLGUYWousmdxyQOq8UnM+oJ+ZpUDQDHuqf3 XZSvvNDe6ZK11AzEXBV3EeoKtCXpvkGtGP9QcyeLGK3Jh+K2x23kJj/244vZv7gGNCoJ3DJX+pf ei3lDegaE322yWhxZqyeYSleOVWHCv8YrhPypfgStZhADJbEZrjixwo9fxv58KOnXnLImvMZFSO sRdrKQNZmZF92rDQuQ== X-Google-Smtp-Source: AGHT+IHLXvsFhFTC2/CpnCNKtJm8HspbzNrFJ2stc8XPxlxgUVecxgwWhc2fHSMoQmdPv+pgQ7aEHA== X-Received: by 2002:a17:903:18c:b0:215:522d:72d6 with SMTP id d9443c01a7336-21dd7dcd025mr173931335ad.38.1738314470104; Fri, 31 Jan 2025 01:07:50 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:c752:be9d:3368:16fa]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-21de3302a68sm26120825ad.180.2025.01.31.01.07.48 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 31 Jan 2025 01:07:49 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv4 11/17] zram: unlock slot during recompression Date: Fri, 31 Jan 2025 18:06:10 +0900 Message-ID: <20250131090658.3386285-12-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.362.g079036d154-goog In-Reply-To: <20250131090658.3386285-1-senozhatsky@chromium.org> References: <20250131090658.3386285-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Recompression, like writeback, makes a local copy of slot data (we need to decompress it anyway) before post-processing so we can unlock slot-entry once we have that local copy. Unlock the entry write-lock before recompression loop (secondary algorithms can be tried out one by one, in order of priority) and re-acquire it right after the loop. There is one more potentially costly operation recompress_slot() does - new zs_handle allocation, which can schedule(). Release the slot-entry write-lock before zsmalloc allocation and grab it again after the allocation. In both cases, once the slot-lock is re-acquired we examine slot's ZRAM_PP_SLOT flag to make sure that the slot has not been modified by a concurrent operation. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 80 +++++++++++++++++++---------------- 1 file changed, 44 insertions(+), 36 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 256439361367..cfbb3072ee9e 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1869,14 +1869,13 @@ static int recompress_slot(struct zram *zram, u32 i= ndex, struct page *page, u64 *num_recomp_pages, u32 threshold, u32 prio, u32 prio_max) { - struct zcomp_strm *zstrm =3D NULL; + struct zcomp_strm *zstrm; unsigned long handle_old; unsigned long handle_new; unsigned int comp_len_old; unsigned int comp_len_new; unsigned int class_index_old; unsigned int class_index_new; - u32 num_recomps =3D 0; void *src, *dst; int ret; =20 @@ -1903,6 +1902,13 @@ static int recompress_slot(struct zram *zram, u32 in= dex, struct page *page, zram_clear_flag(zram, index, ZRAM_IDLE); =20 class_index_old =3D zs_lookup_class_index(zram->mem_pool, comp_len_old); + prio =3D max(prio, zram_get_priority(zram, index) + 1); + /* Slot data copied out - unlock its bucket */ + zram_slot_write_unlock(zram, index); + /* Recompression slots scan takes care of this, but just in case */ + if (prio >=3D prio_max) + return 0; + /* * Iterate the secondary comp algorithms list (in order of priority) * and try to recompress the page. @@ -1911,24 +1917,14 @@ static int recompress_slot(struct zram *zram, u32 i= ndex, struct page *page, if (!zram->comps[prio]) continue; =20 - /* - * Skip if the object is already re-compressed with a higher - * priority algorithm (or same algorithm). - */ - if (prio <=3D zram_get_priority(zram, index)) - continue; - - num_recomps++; zstrm =3D zcomp_stream_get(zram->comps[prio]); src =3D kmap_local_page(page); ret =3D zcomp_compress(zram->comps[prio], zstrm, src, &comp_len_new); kunmap_local(src); =20 - if (ret) { - zcomp_stream_put(zram->comps[prio], zstrm); - return ret; - } + if (ret) + break; =20 class_index_new =3D zs_lookup_class_index(zram->mem_pool, comp_len_new); @@ -1937,6 +1933,7 @@ static int recompress_slot(struct zram *zram, u32 ind= ex, struct page *page, if (class_index_new >=3D class_index_old || (threshold && comp_len_new >=3D threshold)) { zcomp_stream_put(zram->comps[prio], zstrm); + zstrm =3D NULL; continue; } =20 @@ -1944,14 +1941,7 @@ static int recompress_slot(struct zram *zram, u32 in= dex, struct page *page, break; } =20 - /* - * We did not try to recompress, e.g. when we have only one - * secondary algorithm and the page is already recompressed - * using that algorithm - */ - if (!zstrm) - return 0; - + zram_slot_write_lock(zram, index); /* * Decrement the limit (if set) on pages we can recompress, even * when current recompression was unsuccessful or did not compress @@ -1961,37 +1951,55 @@ static int recompress_slot(struct zram *zram, u32 i= ndex, struct page *page, if (*num_recomp_pages) *num_recomp_pages -=3D 1; =20 - if (class_index_new >=3D class_index_old) { + /* Compression error */ + if (ret) { + zcomp_stream_put(zram->comps[prio], zstrm); + return ret; + } + + if (!zstrm) { /* * Secondary algorithms failed to re-compress the page - * in a way that would save memory, mark the object as - * incompressible so that we will not try to compress - * it again. + * in a way that would save memory. * - * We need to make sure that all secondary algorithms have - * failed, so we test if the number of recompressions matches - * the number of active secondary algorithms. + * Mark the object incompressible if the max-priority + * algorithm couldn't re-compress it. */ - if (num_recomps =3D=3D zram->num_active_comps - 1) + if (prio < zram->num_active_comps) + return 0; + if (zram_test_flag(zram, index, ZRAM_PP_SLOT)) zram_set_flag(zram, index, ZRAM_INCOMPRESSIBLE); return 0; } =20 - /* Successful recompression but above threshold */ - if (threshold && comp_len_new >=3D threshold) + /* Slot has been modified concurrently */ + if (!zram_test_flag(zram, index, ZRAM_PP_SLOT)) { + zcomp_stream_put(zram->comps[prio], zstrm); return 0; + } =20 - /* - * If we cannot alloc memory for recompressed object then we bail out - * and simply keep the old (existing) object in zsmalloc. - */ + /* zsmalloc handle allocation can schedule, unlock slot's bucket */ + zram_slot_write_unlock(zram, index); handle_new =3D zs_malloc(zram->mem_pool, comp_len_new, GFP_NOIO | __GFP_HIGHMEM | __GFP_MOVABLE); + zram_slot_write_lock(zram, index); + + /* + * If we couldn't allocate memory for recompressed object then bail + * out and simply keep the old (existing) object in mempool. + */ if (IS_ERR_VALUE(handle_new)) { zcomp_stream_put(zram->comps[prio], zstrm); return PTR_ERR((void *)handle_new); } =20 + /* Slot has been modified concurrently */ + if (!zram_test_flag(zram, index, ZRAM_PP_SLOT)) { + zcomp_stream_put(zram->comps[prio], zstrm); + zs_free(zram->mem_pool, handle_new); + return 0; + } + dst =3D zs_map_object(zram->mem_pool, handle_new, ZS_MM_WO); memcpy(dst, zstrm->buffer, comp_len_new); zcomp_stream_put(zram->comps[prio], zstrm); --=20 2.48.1.362.g079036d154-goog From nobody Mon Feb 9 02:42:15 2026 Received: from mail-pl1-f173.google.com (mail-pl1-f173.google.com [209.85.214.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D97D719E7D3 for ; Fri, 31 Jan 2025 09:07:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738314477; cv=none; b=UuGhr60fLxPLO1gQOFp0z5L1o2E78Sp1bse9F0nAKRDImT+pgfOohxjTz1z3FRJXKzSzMiWqLpxnHUeLa/9gqEM7DczqtWuOZq276rKxKDap2Z2gnCPLA0D+knRtBKFeUv+sIlKyYFk/evAo5fIZUkMOym7fOzBOw+myRvdyMQM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738314477; c=relaxed/simple; bh=XWzEpdwF5C7NhQLOAtNqbZYwOY+UZUsDyF/afAc7W3g=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=oE+3Bcm+ndj+caTdp/NK7DKZY6sxzro3ApOZUtwHsFbw87YyqOcmBDmC4qN5VaE9YXdoHqz2oHuN/STq2ypbDRztvoLBGKdLffczDp2PLn+IPieUSrBoy0p0RQ0iYG6g6vYeCYVLUFm+4l/NEa4QDE98mGtvGc6+hwE89OLNkoM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=gefoaMK5; arc=none smtp.client-ip=209.85.214.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="gefoaMK5" Received: by mail-pl1-f173.google.com with SMTP id d9443c01a7336-2166f1e589cso42500585ad.3 for ; Fri, 31 Jan 2025 01:07:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1738314475; x=1738919275; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=aqDkbsViMBfIzGLM8hcQE+HJuYFQMLk+aSWGFS+XSh0=; b=gefoaMK5706y/k7+To3RqP8c3NHpntoeoTlbbhH5uK6acUmrUU2fBBJlJZQXuQhHoR o3uUrQX84MTeFBx1n3ef/8cjmMmZ4RiLMvlcX+zbH2ktOG5nM9xaCZ1Dmu4iW9LJD1WJ YQ3ENjOoR6ttw36yuf8yRMugGn39zMP/notvM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738314475; x=1738919275; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=aqDkbsViMBfIzGLM8hcQE+HJuYFQMLk+aSWGFS+XSh0=; b=Fncp645yVHi/K+cVhF/8j80I1EIq/WiPFi33P7gmLqPkS8sAni7FcNokmk4l866MOq WSrL1sPxecIh7xtwMFYtLZHHLTJiQmqPXkQ95Zoc3BgOur8QVaupXZnmnjpQiULhXk6p GebSJzfH41pP0JBd8b8wUSl+pOP0N4/Vo1o/8Ci05nsBGqzqVbK7jxDChGfqul+t20gw nKXYTkS/JTkYAaNMxr4vZl83ZO7Wdr4WbdEE3hEUFR5Q85ZaRLzBi8nw9ZP87H0SkyH7 Ki4yCZCzfuEbgn0glJfOuqW8rzfkiT6d9+vhvxo2Ti40h+X6RN7I8CjeWjutw/nttjdF CCpA== X-Forwarded-Encrypted: i=1; AJvYcCWSDdsD1jkYzaj/P2cZXjnMcgYh3clEYJyXjQm5so0rEVkYzmYiIZ5yR7VZ8G1OWtcmkJ5d5I8Gt0EzPCg=@vger.kernel.org X-Gm-Message-State: AOJu0YyipFSgyPFvxrh6VvsAAsKF9L0y880lTPRD1pnBPXoyxAp3LyQY QW/7YL4+u2IfaOI7EbPiHvChOM28jCQOxd4puS80/gHZpR0iY2xS7O+KvBtqRg== X-Gm-Gg: ASbGncsK/nn1Z/UQAGljVOMw/+Zs0DnJxkx09RHCyL0LeJqF8Gp59LZ94FdsAaAq8Q/ y1OQdkPmoPgTGt75W/j/EeY1ZOZR1CxuY/Az+WKj5N0u7BO8J7DQSobbgK5IFCBlpfkXJ06khTO n5BMsKmbDxuk1qg7H/x+ociRazyFY0JBf1lsD85NYksvDb90y3YWCFsa+7CLxS3Kl2ppKgsEwaf BtLmKXb0rilmWc3eg0GC6WaZesIIrzs8+2f2wOCJOjqb5eni7/UcwLS5+YE3BAvEBTGdI5dn+m9 HvDD9GMYFS8wB+XQaA== X-Google-Smtp-Source: AGHT+IGX15C7/nD8ive9k+yk6hUyhUaEscDdlRfPOcsZ6APSPwhqAsXH+BxwUfuwKmae5iWyxWBwog== X-Received: by 2002:a17:902:ebc2:b0:21b:d105:26b8 with SMTP id d9443c01a7336-21eddbf3366mr33372725ad.7.1738314475027; Fri, 31 Jan 2025 01:07:55 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:c752:be9d:3368:16fa]) by smtp.gmail.com with UTF8SMTPSA id 98e67ed59e1d1-2f83bfc0ddbsm5391902a91.45.2025.01.31.01.07.52 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 31 Jan 2025 01:07:54 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky , Yosry Ahmed Subject: [PATCHv4 12/17] zsmalloc: factor out pool locking helpers Date: Fri, 31 Jan 2025 18:06:11 +0900 Message-ID: <20250131090658.3386285-13-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.362.g079036d154-goog In-Reply-To: <20250131090658.3386285-1-senozhatsky@chromium.org> References: <20250131090658.3386285-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" We currently have a mix of migrate_{read,write}_lock() helpers that lock zspages, but it's zs_pool that actually has a ->migrate_lock access to which is opene-coded. Factor out pool migrate locking into helpers, zspage migration locking API will be renamed to reduce confusion. It's worth mentioning that zsmalloc locks sync not only migration, but also compaction. Signed-off-by: Sergey Senozhatsky Cc: Yosry Ahmed --- mm/zsmalloc.c | 69 +++++++++++++++++++++++++++++++++++---------------- 1 file changed, 47 insertions(+), 22 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 817626a351f8..c129596ab960 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -18,7 +18,7 @@ /* * lock ordering: * page_lock - * pool->migrate_lock + * pool->lock * class->lock * zspage->lock */ @@ -224,10 +224,35 @@ struct zs_pool { struct work_struct free_work; #endif /* protect page/zspage migration */ - rwlock_t migrate_lock; + rwlock_t lock; atomic_t compaction_in_progress; }; =20 +static void pool_write_unlock(struct zs_pool *pool) +{ + write_unlock(&pool->lock); +} + +static void pool_write_lock(struct zs_pool *pool) +{ + write_lock(&pool->lock); +} + +static void pool_read_unlock(struct zs_pool *pool) +{ + read_unlock(&pool->lock); +} + +static void pool_read_lock(struct zs_pool *pool) +{ + read_lock(&pool->lock); +} + +static bool pool_lock_is_contended(struct zs_pool *pool) +{ + return rwlock_is_contended(&pool->lock); +} + static inline void zpdesc_set_first(struct zpdesc *zpdesc) { SetPagePrivate(zpdesc_page(zpdesc)); @@ -290,7 +315,7 @@ static bool ZsHugePage(struct zspage *zspage) return zspage->huge; } =20 -static void migrate_lock_init(struct zspage *zspage); +static void lock_init(struct zspage *zspage); static void migrate_read_lock(struct zspage *zspage); static void migrate_read_unlock(struct zspage *zspage); static void migrate_write_lock(struct zspage *zspage); @@ -992,7 +1017,7 @@ static struct zspage *alloc_zspage(struct zs_pool *poo= l, return NULL; =20 zspage->magic =3D ZSPAGE_MAGIC; - migrate_lock_init(zspage); + lock_init(zspage); =20 for (i =3D 0; i < class->pages_per_zspage; i++) { struct zpdesc *zpdesc; @@ -1206,7 +1231,7 @@ void *zs_map_object(struct zs_pool *pool, unsigned lo= ng handle, BUG_ON(in_interrupt()); =20 /* It guarantees it can get zspage from handle safely */ - read_lock(&pool->migrate_lock); + pool_read_lock(pool); obj =3D handle_to_obj(handle); obj_to_location(obj, &zpdesc, &obj_idx); zspage =3D get_zspage(zpdesc); @@ -1218,7 +1243,7 @@ void *zs_map_object(struct zs_pool *pool, unsigned lo= ng handle, * which is smaller granularity. */ migrate_read_lock(zspage); - read_unlock(&pool->migrate_lock); + pool_read_unlock(pool); =20 class =3D zspage_class(pool, zspage); off =3D offset_in_page(class->size * obj_idx); @@ -1450,16 +1475,16 @@ void zs_free(struct zs_pool *pool, unsigned long ha= ndle) return; =20 /* - * The pool->migrate_lock protects the race with zpage's migration + * The pool->lock protects the race with zpage's migration * so it's safe to get the page from handle. */ - read_lock(&pool->migrate_lock); + pool_read_lock(pool); obj =3D handle_to_obj(handle); obj_to_zpdesc(obj, &f_zpdesc); zspage =3D get_zspage(f_zpdesc); class =3D zspage_class(pool, zspage); spin_lock(&class->lock); - read_unlock(&pool->migrate_lock); + pool_read_unlock(pool); =20 class_stat_sub(class, ZS_OBJS_INUSE, 1); obj_free(class->size, obj); @@ -1703,7 +1728,7 @@ static void lock_zspage(struct zspage *zspage) } #endif /* CONFIG_COMPACTION */ =20 -static void migrate_lock_init(struct zspage *zspage) +static void lock_init(struct zspage *zspage) { rwlock_init(&zspage->lock); } @@ -1793,10 +1818,10 @@ static int zs_page_migrate(struct page *newpage, st= ruct page *page, pool =3D zspage->pool; =20 /* - * The pool migrate_lock protects the race between zpage migration + * The pool lock protects the race between zpage migration * and zs_free. */ - write_lock(&pool->migrate_lock); + pool_write_lock(pool); class =3D zspage_class(pool, zspage); =20 /* @@ -1833,7 +1858,7 @@ static int zs_page_migrate(struct page *newpage, stru= ct page *page, * Since we complete the data copy and set up new zspage structure, * it's okay to release migration_lock. */ - write_unlock(&pool->migrate_lock); + pool_write_unlock(pool); spin_unlock(&class->lock); migrate_write_unlock(zspage); =20 @@ -1956,7 +1981,7 @@ static unsigned long __zs_compact(struct zs_pool *poo= l, * protect the race between zpage migration and zs_free * as well as zpage allocation/free */ - write_lock(&pool->migrate_lock); + pool_write_lock(pool); spin_lock(&class->lock); while (zs_can_compact(class)) { int fg; @@ -1983,14 +2008,14 @@ static unsigned long __zs_compact(struct zs_pool *p= ool, src_zspage =3D NULL; =20 if (get_fullness_group(class, dst_zspage) =3D=3D ZS_INUSE_RATIO_100 - || rwlock_is_contended(&pool->migrate_lock)) { + || pool_lock_is_contended(pool)) { putback_zspage(class, dst_zspage); dst_zspage =3D NULL; =20 spin_unlock(&class->lock); - write_unlock(&pool->migrate_lock); + pool_write_unlock(pool); cond_resched(); - write_lock(&pool->migrate_lock); + pool_write_lock(pool); spin_lock(&class->lock); } } @@ -2002,7 +2027,7 @@ static unsigned long __zs_compact(struct zs_pool *poo= l, putback_zspage(class, dst_zspage); =20 spin_unlock(&class->lock); - write_unlock(&pool->migrate_lock); + pool_write_unlock(pool); =20 return pages_freed; } @@ -2014,10 +2039,10 @@ unsigned long zs_compact(struct zs_pool *pool) unsigned long pages_freed =3D 0; =20 /* - * Pool compaction is performed under pool->migrate_lock so it is basical= ly + * Pool compaction is performed under pool->lock so it is basically * single-threaded. Having more than one thread in __zs_compact() - * will increase pool->migrate_lock contention, which will impact other - * zsmalloc operations that need pool->migrate_lock. + * will increase pool->lock contention, which will impact other + * zsmalloc operations that need pool->lock. */ if (atomic_xchg(&pool->compaction_in_progress, 1)) return 0; @@ -2139,7 +2164,7 @@ struct zs_pool *zs_create_pool(const char *name) return NULL; =20 init_deferred_free(pool); - rwlock_init(&pool->migrate_lock); + rwlock_init(&pool->lock); atomic_set(&pool->compaction_in_progress, 0); =20 pool->name =3D kstrdup(name, GFP_KERNEL); --=20 2.48.1.362.g079036d154-goog From nobody Mon Feb 9 02:42:15 2026 Received: from mail-pl1-f176.google.com (mail-pl1-f176.google.com [209.85.214.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BAF2919DF99 for ; Fri, 31 Jan 2025 09:07:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738314481; cv=none; b=IbS0nejkjnKEWI+MhGVfI2axde69JSWtQDHddp2VN2IbzQV1hbVsAwiDXwkJNa1h6LsV5lmVCTkpp5K1axq9tZdfs1xbW7Fobm1YpztyllNzRh8k2uwiPovPaDx39Du30XmI7SlOqOEvHE1lQg8wp5IJrzm5oK6t4+6fhg4i9j8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738314481; c=relaxed/simple; bh=BNi+Kbwqr6U/J6I30WWLVwPz4DSS3fAEl1ytgSUOm6w=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=uuFfGBPL91dV6l+n+VwEMR37NmMuI18pbG5uwL6CRFpW4EQZABGZVgC4WYCtuKA0ibQyv6GekL+Px3E676taeB0mGrh/POyhoBzvW3FPMEbkmgPBpOXfaIhHXEdxIoN8ZmbjGQ8SxREOwRdw4cY55t/ewilV4qf28CRc+8rRYwQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=dupGKhQl; arc=none smtp.client-ip=209.85.214.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="dupGKhQl" Received: by mail-pl1-f176.google.com with SMTP id d9443c01a7336-2161eb94cceso20833015ad.2 for ; Fri, 31 Jan 2025 01:07:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1738314479; x=1738919279; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=IMJPEUQI7RZrpU8Uw6E+fMbBL3rBKdqOy7rMXGa3DVQ=; b=dupGKhQl3tnPHdKxnM3lS6zwk0kYVwpOL2P7NqKvlsEeGIiF+UjLIK8zn4jdCmxYwU 6ZNxgi8E6frZm7GiD1vx7OFbKKpBzjQpkXr7mGYU9TN9RjCZygtk61TMkELtTjks7bNj pq6WK/ZrBi6erJcVI54tkQYj4cmPJiN3LmzNw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738314479; x=1738919279; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=IMJPEUQI7RZrpU8Uw6E+fMbBL3rBKdqOy7rMXGa3DVQ=; b=BjmgTKhhLuX1pIIxSt1JAiPsd2hQOj6udM9ohes95aA+FhtjXpJA78oD2MB68P8Nbs uebSDHPe2RQsTA/aykt2q51pug7OP9OvBG9cgVSQQyPS1CaIf2Y1BiWC0GhyV9OSD8U7 r6O/+IuD/XJdpVeTfqkVL1Blfn4JBt43D5okCVNBm5w7g/K3SPXWZzssFVtR3Xf7Gpuk S6rB8/okAI4Rk4nwHJCSc5TVGQCD6WdCyFiUN9tJxyF0/q40k94Q9MEi8nxiP0dNs6Bz tj2YXRobKDya0tBxGaoWSaT/zS6TOzikpoZD2fYdsUP+YS3Tx5T+D3Fa57Q6CQCtSjPc Kxqg== X-Forwarded-Encrypted: i=1; AJvYcCUuLB8DbBOMQP2Swp+rMpkTyejG4mVPoNvR7K8n4IkNQ+ucNvyslJNH1LDaoM6+v0ddU2kjiaf+SlX/fe8=@vger.kernel.org X-Gm-Message-State: AOJu0Yz6gOgnRRXZa6b9tCqDhQ3VFhxV5vQT8nWxG0V09MAWARSPoEQK fUrsSfRf0RmTj0ES+9zArgDlRdGkeuzBoVUmNQI/BTGnLGO6HkVd0BwLs7fo/g== X-Gm-Gg: ASbGncvm5pVB77sEV0dxJh2Agz/RM9p/2dVVTk/K9QMPdE9h9gJCK/Vx1A9A9jlBObq ht4aBqTfMjeSw3AFuOmzHPmb99nCB5OAxE5Xu35gmgvC3CskkYtJGv6NgmGjYsLXjZ7VCV5GUfG wLjfCv23a91mujHdqMZitQ1W6UKtAIlb8WbXCold1yOofmX3k1+xIm/0sVUR2gJRfuIoJK82Ded mLdlTpEmu4Vem80X1oBJaz0MsjGbcJo4xLh+/az/nEw6DYz3KP5tx9dfI3EF6TV3Du22MG/YJk1 e6oINIWe7WGehv+jBQ== X-Google-Smtp-Source: AGHT+IHhqjztMQY9nyoxNY57ZCZdxtWzFwxbZ4LRhwZuQGquqZr7M2SqvkrDWTSh2oSFv8q0NO6MeA== X-Received: by 2002:a17:902:ea06:b0:215:352c:af73 with SMTP id d9443c01a7336-21dd7c57eb0mr137738715ad.18.1738314478999; Fri, 31 Jan 2025 01:07:58 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:c752:be9d:3368:16fa]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-21de31f5a60sm25837715ad.61.2025.01.31.01.07.57 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 31 Jan 2025 01:07:58 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky , Yosry Ahmed Subject: [PATCHv4 13/17] zsmalloc: factor out size-class locking helpers Date: Fri, 31 Jan 2025 18:06:12 +0900 Message-ID: <20250131090658.3386285-14-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.362.g079036d154-goog In-Reply-To: <20250131090658.3386285-1-senozhatsky@chromium.org> References: <20250131090658.3386285-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Move open-coded size-class locking to dedicated helpers. Signed-off-by: Sergey Senozhatsky Reviewed-by: Yosry Ahmed --- mm/zsmalloc.c | 47 ++++++++++++++++++++++++++++------------------- 1 file changed, 28 insertions(+), 19 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index c129596ab960..4b4c77bc08f9 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -253,6 +253,16 @@ static bool pool_lock_is_contended(struct zs_pool *poo= l) return rwlock_is_contended(&pool->lock); } =20 +static void size_class_lock(struct size_class *class) +{ + spin_lock(&class->lock); +} + +static void size_class_unlock(struct size_class *class) +{ + spin_unlock(&class->lock); +} + static inline void zpdesc_set_first(struct zpdesc *zpdesc) { SetPagePrivate(zpdesc_page(zpdesc)); @@ -613,8 +623,7 @@ static int zs_stats_size_show(struct seq_file *s, void = *v) if (class->index !=3D i) continue; =20 - spin_lock(&class->lock); - + size_class_lock(class); seq_printf(s, " %5u %5u ", i, class->size); for (fg =3D ZS_INUSE_RATIO_10; fg < NR_FULLNESS_GROUPS; fg++) { inuse_totals[fg] +=3D class_stat_read(class, fg); @@ -624,7 +633,7 @@ static int zs_stats_size_show(struct seq_file *s, void = *v) obj_allocated =3D class_stat_read(class, ZS_OBJS_ALLOCATED); obj_used =3D class_stat_read(class, ZS_OBJS_INUSE); freeable =3D zs_can_compact(class); - spin_unlock(&class->lock); + size_class_unlock(class); =20 objs_per_zspage =3D class->objs_per_zspage; pages_used =3D obj_allocated / objs_per_zspage * @@ -1399,7 +1408,7 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t = size, gfp_t gfp) class =3D pool->size_class[get_size_class_index(size)]; =20 /* class->lock effectively protects the zpage migration */ - spin_lock(&class->lock); + size_class_lock(class); zspage =3D find_get_zspage(class); if (likely(zspage)) { obj_malloc(pool, zspage, handle); @@ -1410,7 +1419,7 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t = size, gfp_t gfp) goto out; } =20 - spin_unlock(&class->lock); + size_class_unlock(class); =20 zspage =3D alloc_zspage(pool, class, gfp); if (!zspage) { @@ -1418,7 +1427,7 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t = size, gfp_t gfp) return (unsigned long)ERR_PTR(-ENOMEM); } =20 - spin_lock(&class->lock); + size_class_lock(class); obj_malloc(pool, zspage, handle); newfg =3D get_fullness_group(class, zspage); insert_zspage(class, zspage, newfg); @@ -1429,7 +1438,7 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t = size, gfp_t gfp) /* We completely set up zspage so mark them as movable */ SetZsPageMovable(pool, zspage); out: - spin_unlock(&class->lock); + size_class_unlock(class); =20 return handle; } @@ -1483,7 +1492,7 @@ void zs_free(struct zs_pool *pool, unsigned long hand= le) obj_to_zpdesc(obj, &f_zpdesc); zspage =3D get_zspage(f_zpdesc); class =3D zspage_class(pool, zspage); - spin_lock(&class->lock); + size_class_lock(class); pool_read_unlock(pool); =20 class_stat_sub(class, ZS_OBJS_INUSE, 1); @@ -1493,7 +1502,7 @@ void zs_free(struct zs_pool *pool, unsigned long hand= le) if (fullness =3D=3D ZS_INUSE_RATIO_0) free_zspage(pool, class, zspage); =20 - spin_unlock(&class->lock); + size_class_unlock(class); cache_free_handle(pool, handle); } EXPORT_SYMBOL_GPL(zs_free); @@ -1827,7 +1836,7 @@ static int zs_page_migrate(struct page *newpage, stru= ct page *page, /* * the class lock protects zpage alloc/free in the zspage. */ - spin_lock(&class->lock); + size_class_lock(class); /* the migrate_write_lock protects zpage access via zs_map_object */ migrate_write_lock(zspage); =20 @@ -1859,7 +1868,7 @@ static int zs_page_migrate(struct page *newpage, stru= ct page *page, * it's okay to release migration_lock. */ pool_write_unlock(pool); - spin_unlock(&class->lock); + size_class_unlock(class); migrate_write_unlock(zspage); =20 zpdesc_get(newzpdesc); @@ -1903,10 +1912,10 @@ static void async_free_zspage(struct work_struct *w= ork) if (class->index !=3D i) continue; =20 - spin_lock(&class->lock); + size_class_lock(class); list_splice_init(&class->fullness_list[ZS_INUSE_RATIO_0], &free_pages); - spin_unlock(&class->lock); + size_class_unlock(class); } =20 list_for_each_entry_safe(zspage, tmp, &free_pages, list) { @@ -1914,10 +1923,10 @@ static void async_free_zspage(struct work_struct *w= ork) lock_zspage(zspage); =20 class =3D zspage_class(pool, zspage); - spin_lock(&class->lock); + size_class_lock(class); class_stat_sub(class, ZS_INUSE_RATIO_0, 1); __free_zspage(pool, class, zspage); - spin_unlock(&class->lock); + size_class_unlock(class); } }; =20 @@ -1982,7 +1991,7 @@ static unsigned long __zs_compact(struct zs_pool *poo= l, * as well as zpage allocation/free */ pool_write_lock(pool); - spin_lock(&class->lock); + size_class_lock(class); while (zs_can_compact(class)) { int fg; =20 @@ -2012,11 +2021,11 @@ static unsigned long __zs_compact(struct zs_pool *p= ool, putback_zspage(class, dst_zspage); dst_zspage =3D NULL; =20 - spin_unlock(&class->lock); + size_class_unlock(class); pool_write_unlock(pool); cond_resched(); pool_write_lock(pool); - spin_lock(&class->lock); + size_class_lock(class); } } =20 @@ -2026,7 +2035,7 @@ static unsigned long __zs_compact(struct zs_pool *poo= l, if (dst_zspage) putback_zspage(class, dst_zspage); =20 - spin_unlock(&class->lock); + size_class_unlock(class); pool_write_unlock(pool); =20 return pages_freed; --=20 2.48.1.362.g079036d154-goog From nobody Mon Feb 9 02:42:15 2026 Received: from mail-pl1-f170.google.com (mail-pl1-f170.google.com [209.85.214.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3AFDD1BD50C for ; Fri, 31 Jan 2025 09:08:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738314486; cv=none; b=r41kH9dRGGUHsQ/5rY4OA66NNJ8zBaS+UaHEMGbOcBr0jiXl2n4fR3wtunx+zF4UYuGj7XEC50K/yXwt2nSSRu7f1EC+xkLrcvDvXRgS/+aXC9M0uZEGySlXzmEoh/LT6GXI6Q77FgP5uyI4bzu7khyokQ3+hXiiTq/5t6THCgs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738314486; c=relaxed/simple; bh=zwmutWgB0aePSiLsNGMtF2l6izDQjWv/zVEYjALBtvA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=XSJL3vCoBDssZhqln+S/OFABjLVzEIC/xOunOVp+lPCZAKXmIxQXW+omR1VFYqeXjKp86N5zi14OOGoNeeNU6JRoEa3a2BppyqODpTad1tPtCl2kfqIhBoXb5nAjFHTTlPvb3mW0LycS2+2ndHzaVT7hymk9HT7ELeLB2/DHUSU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=I2hgJKdl; arc=none smtp.client-ip=209.85.214.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="I2hgJKdl" Received: by mail-pl1-f170.google.com with SMTP id d9443c01a7336-21634338cfdso41851455ad.2 for ; Fri, 31 Jan 2025 01:08:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1738314483; x=1738919283; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=K14zF8RGObITiXxNWEiw4I5DkmsIF3aHWMHaZssq3hI=; b=I2hgJKdl3OPXs8+EJ1ok6ua89wJXfblooZ7a2hfIOjzl1dRAVnoLucF7xBFnLaYBAS cl8jJg9Obrnjdocs1sB5oiUkY40jpE1RO9t9DCtLGB61Dh1dcdd2afjY0A1BhubpeFaE fYyOhfGCZRprKF5UXxyENQXxYvZJpPcPr5TeA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738314483; x=1738919283; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=K14zF8RGObITiXxNWEiw4I5DkmsIF3aHWMHaZssq3hI=; b=LFyehvmsY02gpQ2iCy+fQjCd4krQTIKhBoISQ+HlftFfQFYNvAKT83AAD6+CQbuiqD DSjEpfZKXlJ2VjmuVslhVs5PEZ62oSoNHEDCr1se+t5C7xtp/LEPhpMeel+R9otnVNBS Td3AF/E/sxe9gahvPjkgVBkgbTFJ3keMswFqWbvj83EdlirhBeoi1GXgFurv8QXPaps2 7R8URLeVg2suu3c85isDuHxxU4KAMgyRJsyCQ/yOWf+yDR5oKU+jVNLFlsbNXbzzZ4PK 2vK4MFU11zXez/Dcysi6LnUYeSZpMz5VxaR7lENpM9yn+LIQlCEPK7hIY+rz46++ItMg Mt/g== X-Forwarded-Encrypted: i=1; AJvYcCXg7uO2+UUevNvGHMw9+6mKHA1rl284hTjbi1BYm08cEWTKfta8Ijmu0eWrKGi6yVVYVQ9PEppe1b/u3jM=@vger.kernel.org X-Gm-Message-State: AOJu0YwroOiuTug+zVwtmDz1+Qne3fqBIe8yFGytmMBaoM945zpcUIJj rIV38ZyGptCNeCrax3x4LhDmoSLyMVGATSzHv39elHFfC4RUbwlElYzYhhG8iA== X-Gm-Gg: ASbGncuXL7oXsXosF99hK7iTfhp5l4zHzeBogmnh+mNbW/gxfw6SI7umn6F/Ds3FzIR pNfUNlV1iFEfNlb59Kcoe06zBDXRy8xwxJQErg2ebCc+gPxsEMALLuwzzlQS1K6Rutdlhyu4ESJ 0T40nEhmBx+A/iuFI+8DVwNKkB9bWNC9nBrM6TR4HzHqr/4o6eFBxdfbE9KUGkwqyNvLKzZH0CA 6gR+5W9/RLY/+HgHm8Z09fp4Hq9lIy5xAFDByWmG7CwpiK14muzvVPA2/UFZz42/Bgp2XDdilop xpahUYJkQjXHkkRrvQ== X-Google-Smtp-Source: AGHT+IFoGKrZNGGlmGxZx2UZTouqjuhCYFoIKMcMR38ulc4zYXbCr002OD1N6TYgHMj1w9Keyos0HA== X-Received: by 2002:a17:902:ecce:b0:211:3275:3fe with SMTP id d9443c01a7336-21dd7c65555mr158196745ad.17.1738314483547; Fri, 31 Jan 2025 01:08:03 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:c752:be9d:3368:16fa]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-21de31f80c2sm26197065ad.72.2025.01.31.01.08.01 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 31 Jan 2025 01:08:03 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky , Yosry Ahmed Subject: [PATCHv4 14/17] zsmalloc: make zspage lock preemptible Date: Fri, 31 Jan 2025 18:06:13 +0900 Message-ID: <20250131090658.3386285-15-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.362.g079036d154-goog In-Reply-To: <20250131090658.3386285-1-senozhatsky@chromium.org> References: <20250131090658.3386285-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Switch over from rwlock_t to a atomic_t variable that takes negative value when the page is under migration, or positive values when the page is used by zsmalloc users (object map, etc.) Using a rwsem per-zspage is a little too memory heavy, a simple atomic_t should suffice. zspage lock is a leaf lock for zs_map_object(), where it's read-acquired. Since this lock now permits preemption extra care needs to be taken when it is write-acquired - all writers grab it in atomic context, so they cannot spin and wait for (potentially preempted) reader to unlock zspage. There are only two writers at this moment - migration and compaction. In both cases we use write-try-lock and bail out if zspage is read locked. Writers, on the other hand, never get preempted, so readers can spin waiting for the writer to unlock zspage. With this we can implement a preemptible object mapping. Signed-off-by: Sergey Senozhatsky Cc: Yosry Ahmed --- mm/zsmalloc.c | 135 +++++++++++++++++++++++++++++++------------------- 1 file changed, 83 insertions(+), 52 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 4b4c77bc08f9..f5b5fe732e50 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -292,6 +292,9 @@ static inline void free_zpdesc(struct zpdesc *zpdesc) __free_page(page); } =20 +#define ZS_PAGE_UNLOCKED 0 +#define ZS_PAGE_WRLOCKED -1 + struct zspage { struct { unsigned int huge:HUGE_BITS; @@ -304,7 +307,7 @@ struct zspage { struct zpdesc *first_zpdesc; struct list_head list; /* fullness list */ struct zs_pool *pool; - rwlock_t lock; + atomic_t lock; }; =20 struct mapping_area { @@ -314,6 +317,59 @@ struct mapping_area { enum zs_mapmode vm_mm; /* mapping mode */ }; =20 +static void zspage_lock_init(struct zspage *zspage) +{ + atomic_set(&zspage->lock, ZS_PAGE_UNLOCKED); +} + +/* + * zspage lock permits preemption on the reader-side (there can be multiple + * readers). Writers (exclusive zspage ownership), on the other hand, are + * always run in atomic context and cannot spin waiting for a (potentially + * preempted) reader to unlock zspage. This, basically, means that writers + * can only call write-try-lock and must bail out if it didn't succeed. + * + * At the same time, writers cannot reschedule under zspage write-lock, + * so readers can spin waiting for the writer to unlock zspage. + */ +static void zspage_read_lock(struct zspage *zspage) +{ + atomic_t *lock =3D &zspage->lock; + int old =3D atomic_read(lock); + + do { + if (old =3D=3D ZS_PAGE_WRLOCKED) { + cpu_relax(); + old =3D atomic_read(lock); + continue; + } + } while (!atomic_try_cmpxchg(lock, &old, old + 1)); +} + +static void zspage_read_unlock(struct zspage *zspage) +{ + atomic_dec(&zspage->lock); +} + +static bool zspage_try_write_lock(struct zspage *zspage) +{ + atomic_t *lock =3D &zspage->lock; + int old =3D ZS_PAGE_UNLOCKED; + + preempt_disable(); + if (atomic_try_cmpxchg(lock, &old, ZS_PAGE_WRLOCKED)) + return true; + + preempt_enable(); + return false; +} + +static void zspage_write_unlock(struct zspage *zspage) +{ + atomic_set(&zspage->lock, ZS_PAGE_UNLOCKED); + preempt_enable(); +} + /* huge object: pages_per_zspage =3D=3D 1 && maxobj_per_zspage =3D=3D 1 */ static void SetZsHugePage(struct zspage *zspage) { @@ -325,12 +381,6 @@ static bool ZsHugePage(struct zspage *zspage) return zspage->huge; } =20 -static void lock_init(struct zspage *zspage); -static void migrate_read_lock(struct zspage *zspage); -static void migrate_read_unlock(struct zspage *zspage); -static void migrate_write_lock(struct zspage *zspage); -static void migrate_write_unlock(struct zspage *zspage); - #ifdef CONFIG_COMPACTION static void kick_deferred_free(struct zs_pool *pool); static void init_deferred_free(struct zs_pool *pool); @@ -1026,7 +1076,7 @@ static struct zspage *alloc_zspage(struct zs_pool *po= ol, return NULL; =20 zspage->magic =3D ZSPAGE_MAGIC; - lock_init(zspage); + zspage_lock_init(zspage); =20 for (i =3D 0; i < class->pages_per_zspage; i++) { struct zpdesc *zpdesc; @@ -1251,7 +1301,7 @@ void *zs_map_object(struct zs_pool *pool, unsigned lo= ng handle, * zs_unmap_object API so delegate the locking from class to zspage * which is smaller granularity. */ - migrate_read_lock(zspage); + zspage_read_lock(zspage); pool_read_unlock(pool); =20 class =3D zspage_class(pool, zspage); @@ -1311,7 +1361,7 @@ void zs_unmap_object(struct zs_pool *pool, unsigned l= ong handle) } local_unlock(&zs_map_area.lock); =20 - migrate_read_unlock(zspage); + zspage_read_unlock(zspage); } EXPORT_SYMBOL_GPL(zs_unmap_object); =20 @@ -1705,18 +1755,18 @@ static void lock_zspage(struct zspage *zspage) /* * Pages we haven't locked yet can be migrated off the list while we're * trying to lock them, so we need to be careful and only attempt to - * lock each page under migrate_read_lock(). Otherwise, the page we lock + * lock each page under zspage_read_lock(). Otherwise, the page we lock * may no longer belong to the zspage. This means that we may wait for * the wrong page to unlock, so we must take a reference to the page - * prior to waiting for it to unlock outside migrate_read_lock(). + * prior to waiting for it to unlock outside zspage_read_lock(). */ while (1) { - migrate_read_lock(zspage); + zspage_read_lock(zspage); zpdesc =3D get_first_zpdesc(zspage); if (zpdesc_trylock(zpdesc)) break; zpdesc_get(zpdesc); - migrate_read_unlock(zspage); + zspage_read_unlock(zspage); zpdesc_wait_locked(zpdesc); zpdesc_put(zpdesc); } @@ -1727,41 +1777,16 @@ static void lock_zspage(struct zspage *zspage) curr_zpdesc =3D zpdesc; } else { zpdesc_get(zpdesc); - migrate_read_unlock(zspage); + zspage_read_unlock(zspage); zpdesc_wait_locked(zpdesc); zpdesc_put(zpdesc); - migrate_read_lock(zspage); + zspage_read_lock(zspage); } } - migrate_read_unlock(zspage); + zspage_read_unlock(zspage); } #endif /* CONFIG_COMPACTION */ =20 -static void lock_init(struct zspage *zspage) -{ - rwlock_init(&zspage->lock); -} - -static void migrate_read_lock(struct zspage *zspage) __acquires(&zspage->l= ock) -{ - read_lock(&zspage->lock); -} - -static void migrate_read_unlock(struct zspage *zspage) __releases(&zspage-= >lock) -{ - read_unlock(&zspage->lock); -} - -static void migrate_write_lock(struct zspage *zspage) -{ - write_lock(&zspage->lock); -} - -static void migrate_write_unlock(struct zspage *zspage) -{ - write_unlock(&zspage->lock); -} - #ifdef CONFIG_COMPACTION =20 static const struct movable_operations zsmalloc_mops; @@ -1803,7 +1828,7 @@ static bool zs_page_isolate(struct page *page, isolat= e_mode_t mode) } =20 static int zs_page_migrate(struct page *newpage, struct page *page, - enum migrate_mode mode) + enum migrate_mode mode) { struct zs_pool *pool; struct size_class *class; @@ -1819,15 +1844,12 @@ static int zs_page_migrate(struct page *newpage, st= ruct page *page, =20 VM_BUG_ON_PAGE(!zpdesc_is_isolated(zpdesc), zpdesc_page(zpdesc)); =20 - /* We're committed, tell the world that this is a Zsmalloc page. */ - __zpdesc_set_zsmalloc(newzpdesc); - /* The page is locked, so this pointer must remain valid */ zspage =3D get_zspage(zpdesc); pool =3D zspage->pool; =20 /* - * The pool lock protects the race between zpage migration + * The pool->lock protects the race between zpage migration * and zs_free. */ pool_write_lock(pool); @@ -1837,8 +1859,15 @@ static int zs_page_migrate(struct page *newpage, str= uct page *page, * the class lock protects zpage alloc/free in the zspage. */ size_class_lock(class); - /* the migrate_write_lock protects zpage access via zs_map_object */ - migrate_write_lock(zspage); + /* the zspage write_lock protects zpage access via zs_map_object */ + if (!zspage_try_write_lock(zspage)) { + size_class_unlock(class); + pool_write_unlock(pool); + return -EINVAL; + } + + /* We're committed, tell the world that this is a Zsmalloc page. */ + __zpdesc_set_zsmalloc(newzpdesc); =20 offset =3D get_first_obj_offset(zpdesc); s_addr =3D kmap_local_zpdesc(zpdesc); @@ -1869,7 +1898,7 @@ static int zs_page_migrate(struct page *newpage, stru= ct page *page, */ pool_write_unlock(pool); size_class_unlock(class); - migrate_write_unlock(zspage); + zspage_write_unlock(zspage); =20 zpdesc_get(newzpdesc); if (zpdesc_zone(newzpdesc) !=3D zpdesc_zone(zpdesc)) { @@ -2005,9 +2034,11 @@ static unsigned long __zs_compact(struct zs_pool *po= ol, if (!src_zspage) break; =20 - migrate_write_lock(src_zspage); + if (!zspage_try_write_lock(src_zspage)) + break; + migrate_zspage(pool, src_zspage, dst_zspage); - migrate_write_unlock(src_zspage); + zspage_write_unlock(src_zspage); =20 fg =3D putback_zspage(class, src_zspage); if (fg =3D=3D ZS_INUSE_RATIO_0) { --=20 2.48.1.362.g079036d154-goog From nobody Mon Feb 9 02:42:15 2026 Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4E88318CBFE for ; Fri, 31 Jan 2025 09:08:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738314490; cv=none; b=kFTQutDeCeINelqe0hDmRCzlANuk7IRy4ZuCw/LEh28z5ZXrjbV7P4VZHFuhGxWLCwmikfivGWoxj0Rs/u9YyF0ovZqOjAvAHT9oN3b0YbJLENBXHr/0u8Jsj/Z15Q3PAC/4a3L69RV2dq0nAnXEf//v7YzBTz7iOMlfUrFes5s= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738314490; c=relaxed/simple; bh=8ivNqsIp3GMO90oTFY6Dtx8Mq8clYFyCBE8M2NSHYLI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=IUsZxayBmxH/yZ7Rv55dEKrtMVLmu23kw6aftEBhYWT/43pPuIS2ZU0QZJq45Kz1+MxHMZiiPPMcZXc3m9iZ/t6gjXTDKe8Gz/hQ0rtxkTLBXd2HHOhcWjd1Jwzu0og1rW8awyRZPprQPMFQ+cLxTmx3f1iYD8A3qY2AI1rGBe4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=YJ2U4pX+; arc=none smtp.client-ip=209.85.214.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="YJ2U4pX+" Received: by mail-pl1-f169.google.com with SMTP id d9443c01a7336-21669fd5c7cso29292395ad.3 for ; Fri, 31 Jan 2025 01:08:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1738314488; x=1738919288; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=rSDpkbxqSvfVMXf6MtRLSP9PYlXCwlCf1VxOLTEJdKQ=; b=YJ2U4pX+Rk+VUj1IYUUeN+ODu9JfNzPCdL7hSHoOncETw/CtjqvWFo5naFgajwROtr nGXeCRPluhxayNUTG/gLQBN7A2aYx81hI6rnP2s+O1/yqXgvzpcH4vKitb2X94tIKmzI ycgaaqj6EDxy8VZjZU+Dot/eJQCa1Mfwl4TcM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738314488; x=1738919288; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=rSDpkbxqSvfVMXf6MtRLSP9PYlXCwlCf1VxOLTEJdKQ=; b=HXr4oLvLdXHQydjzem2EeSkavy1VNtfWqm2ekSygAAReRPy4WNUWz5P+bDh1GANP9/ VlQ/5da3hAB+bEY/t9LBfRBPsMjX48VxfKNrzJ/5Pee6g38xKAiGSTVYsFqEWJTOtidi qtjDDleLXBbeCX2y1WV+sTysNoIlImEP4t88oVutKVrNiQvfOV8rLJejfDvoo18hyPxw JHgvg5s2RZg9bA0y5RpGknZKF/xRzldTQ7KnURzW/seHLI6azDIM8gaIe+9wXLgXc5E0 s/XVLAKIlLYZ6+BJ1h1ISVj6jMJnTb4I6jmeKkIsnKtFHFmh86qodkscyovbqW7DJcbD PaRQ== X-Forwarded-Encrypted: i=1; AJvYcCV1spdPmBxha5jVqZz+frgGkOneER5+FMFcqLiLnDpA/RtNPChrSo0D2jwTUBhEsz7pEvz2mVJL/Et0Uuc=@vger.kernel.org X-Gm-Message-State: AOJu0YxevW0nw1FXHdbm/fHciRyPS3nqy5TKLoSYuqKsnH1TQwMCTrzR wCDIcLVl0vJmpZCi4VDbUHCIMDYngsDlE9reoRtI7lE4CvNgyzW//G/P2W84XA== X-Gm-Gg: ASbGncu9i9iUmT1opPimRn3aleSgFYSvbkS15dVBJuC+2Mhn/To6A4Xy/aL01KU1bTQ kZXcUit2SbNMmBEp7icMbetR747CPRQA2Y2KzlWq3P0r7HThcONlwhv3umYjEIXRWjth/AnQWmm +F7vAcC1EZNSHt00pG1a7ujiOv3xdVgJtOx8abqcQWU8h/nxakrUZWRvtKHbNRSrDHeMti4OZEW 41/8iTawwPUWeApprzTj2WZqX7uV862fuQzMgPcxFGRyJ9ofsRcjfZX1P8k6NpOhaM6Z+CfbQ9r C5lVspSXlk9V1c5VHQ== X-Google-Smtp-Source: AGHT+IFbB/ylqlS/xqv7OLfNnPKv/KBkJRu1pTXCHuKZ+Ljs5D2OLYSsCuLflKkc6dvQm0HS88PAuQ== X-Received: by 2002:a17:903:2f8e:b0:215:b75f:a1cb with SMTP id d9443c01a7336-21dd9fa74bcmr137340835ad.9.1738314487509; Fri, 31 Jan 2025 01:08:07 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:c752:be9d:3368:16fa]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-21de32f44a9sm26237855ad.143.2025.01.31.01.08.05 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 31 Jan 2025 01:08:07 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky , Yosry Ahmed Subject: [PATCHv4 15/17] zsmalloc: introduce new object mapping API Date: Fri, 31 Jan 2025 18:06:14 +0900 Message-ID: <20250131090658.3386285-16-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.362.g079036d154-goog In-Reply-To: <20250131090658.3386285-1-senozhatsky@chromium.org> References: <20250131090658.3386285-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Current object mapping API is a little cumbersome. First, it's inconsistent, sometimes it returns with page-faults disabled and sometimes with page-faults enabled. Second, and most importantly, it enforces atomicity restrictions on its users. zs_map_object() has to return a liner object address which is not always possible because some objects span multiple physical (non-contiguous) pages. For such objects zsmalloc uses a per-CPU buffer to which object's data is copied before a pointer to that per-CPU buffer is returned back to the caller. This leads to another, final, issue - extra memcpy(). Since the caller gets a pointer to per-CPU buffer it can memcpy() data only to that buffer, and during zs_unmap_object() zsmalloc will memcpy() from that per-CPU buffer to physical pages that object in question spans across. New API splits functions by access mode: - zs_obj_read_begin(handle, local_copy) Returns a pointer to handle memory. For objects that span two physical pages a local_copy buffer is used to store object's data before the address is returned to the caller. Otherwise the object's page is kmap_local mapped directly. - zs_obj_read_end(handle, buf) Unmaps the page if it was kmap_local mapped by zs_obj_read_begin(). - zs_obj_write(handle, buf, len) Copies len-bytes from compression buffer to handle memory (takes care of objects that span two pages). This does not need any additional (e.g. per-CPU) buffers and writes the data directly to zsmalloc pool pages. The old API will stay around until the remaining users switch to the new one. After that we'll also remove zsmalloc per-CPU buffer and CPU hotplug handling. Signed-off-by: Sergey Senozhatsky Reviewed-by: Yosry Ahmed --- include/linux/zsmalloc.h | 8 +++ mm/zsmalloc.c | 129 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 137 insertions(+) diff --git a/include/linux/zsmalloc.h b/include/linux/zsmalloc.h index a48cd0ffe57d..7d70983cf398 100644 --- a/include/linux/zsmalloc.h +++ b/include/linux/zsmalloc.h @@ -58,4 +58,12 @@ unsigned long zs_compact(struct zs_pool *pool); unsigned int zs_lookup_class_index(struct zs_pool *pool, unsigned int size= ); =20 void zs_pool_stats(struct zs_pool *pool, struct zs_pool_stats *stats); + +void *zs_obj_read_begin(struct zs_pool *pool, unsigned long handle, + void *local_copy); +void zs_obj_read_end(struct zs_pool *pool, unsigned long handle, + void *handle_mem); +void zs_obj_write(struct zs_pool *pool, unsigned long handle, + void *handle_mem, size_t mem_len); + #endif diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index f5b5fe732e50..f9d840f77b18 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -1365,6 +1365,135 @@ void zs_unmap_object(struct zs_pool *pool, unsigned= long handle) } EXPORT_SYMBOL_GPL(zs_unmap_object); =20 +void *zs_obj_read_begin(struct zs_pool *pool, unsigned long handle, + void *local_copy) +{ + struct zspage *zspage; + struct zpdesc *zpdesc; + unsigned long obj, off; + unsigned int obj_idx; + struct size_class *class; + void *addr; + + WARN_ON(in_interrupt()); + + /* Guarantee we can get zspage from handle safely */ + pool_read_lock(pool); + obj =3D handle_to_obj(handle); + obj_to_location(obj, &zpdesc, &obj_idx); + zspage =3D get_zspage(zpdesc); + + /* Make sure migration doesn't move any pages in this zspage */ + zspage_read_lock(zspage); + pool_read_unlock(pool); + + class =3D zspage_class(pool, zspage); + off =3D offset_in_page(class->size * obj_idx); + + if (off + class->size <=3D PAGE_SIZE) { + /* this object is contained entirely within a page */ + addr =3D kmap_local_zpdesc(zpdesc); + addr +=3D off; + } else { + size_t sizes[2]; + + /* this object spans two pages */ + sizes[0] =3D PAGE_SIZE - off; + sizes[1] =3D class->size - sizes[0]; + addr =3D local_copy; + + memcpy_from_page(addr, zpdesc_page(zpdesc), + off, sizes[0]); + zpdesc =3D get_next_zpdesc(zpdesc); + memcpy_from_page(addr + sizes[0], + zpdesc_page(zpdesc), + 0, sizes[1]); + } + + if (!ZsHugePage(zspage)) + addr +=3D ZS_HANDLE_SIZE; + + return addr; +} +EXPORT_SYMBOL_GPL(zs_obj_read_begin); + +void zs_obj_read_end(struct zs_pool *pool, unsigned long handle, + void *handle_mem) +{ + struct zspage *zspage; + struct zpdesc *zpdesc; + unsigned long obj, off; + unsigned int obj_idx; + struct size_class *class; + + obj =3D handle_to_obj(handle); + obj_to_location(obj, &zpdesc, &obj_idx); + zspage =3D get_zspage(zpdesc); + class =3D zspage_class(pool, zspage); + off =3D offset_in_page(class->size * obj_idx); + + if (off + class->size <=3D PAGE_SIZE) { + if (!ZsHugePage(zspage)) + off +=3D ZS_HANDLE_SIZE; + handle_mem -=3D off; + kunmap_local(handle_mem); + } + + zspage_read_unlock(zspage); +} +EXPORT_SYMBOL_GPL(zs_obj_read_end); + +void zs_obj_write(struct zs_pool *pool, unsigned long handle, + void *handle_mem, size_t mem_len) +{ + struct zspage *zspage; + struct zpdesc *zpdesc; + unsigned long obj, off; + unsigned int obj_idx; + struct size_class *class; + + WARN_ON(in_interrupt()); + + /* Guarantee we can get zspage from handle safely */ + pool_read_lock(pool); + obj =3D handle_to_obj(handle); + obj_to_location(obj, &zpdesc, &obj_idx); + zspage =3D get_zspage(zpdesc); + + /* Make sure migration doesn't move any pages in this zspage */ + zspage_read_lock(zspage); + pool_read_unlock(pool); + + class =3D zspage_class(pool, zspage); + off =3D offset_in_page(class->size * obj_idx); + + if (off + class->size <=3D PAGE_SIZE) { + /* this object is contained entirely within a page */ + void *dst =3D kmap_local_zpdesc(zpdesc); + + if (!ZsHugePage(zspage)) + off +=3D ZS_HANDLE_SIZE; + memcpy(dst + off, handle_mem, mem_len); + kunmap_local(dst); + } else { + /* this object spans two pages */ + size_t sizes[2]; + + off +=3D ZS_HANDLE_SIZE; + sizes[0] =3D PAGE_SIZE - off; + sizes[1] =3D mem_len - sizes[0]; + + memcpy_to_page(zpdesc_page(zpdesc), off, + handle_mem, sizes[0]); + zpdesc =3D get_next_zpdesc(zpdesc); + memcpy_to_page(zpdesc_page(zpdesc), 0, + handle_mem + sizes[0], sizes[1]); + } + + zspage_read_unlock(zspage); +} +EXPORT_SYMBOL_GPL(zs_obj_write); + /** * zs_huge_class_size() - Returns the size (in bytes) of the first huge * zsmalloc &size_class. --=20 2.48.1.362.g079036d154-goog From nobody Mon Feb 9 02:42:15 2026 Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0676A175D53 for ; Fri, 31 Jan 2025 09:08:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738314493; cv=none; b=uTm0EXsS4uocO3pbq+LLJPJnEwqIhLXAIi3NUsySz0wzeXgudZuy9k7tsz2+TgLGbXLDYmCQPb2Neug5UZ2bP7XaNMtmbFTN4IA1eDGK4bOmnBqGeQTkUmRJdn/Nmn+iGQTgX9n5nI7NBg6LX/13A4NU42PdQQGN2afGvWYVBE8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738314493; c=relaxed/simple; bh=VhFIMmkHatni2lHAbsPw4Uxfib58Qe3afuGaSW20Pno=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=WSd9cN8FFU0CBBUzKtXZULfP6iwEeZTSl35DL5vkE8wCWAsbTGHW1CIiTuGiB1F8TxQWOwpB7PUNezpFjNEzfQAPMqlfcPHPIdfqNV/DLfmcuvtPQ+9u+S92I04wgmZegdhQE0Rbz0I+S8qxEvVgTlxhpyG/Vnnwe8p+tVkM8Tk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=ApY1VfGf; arc=none smtp.client-ip=209.85.214.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="ApY1VfGf" Received: by mail-pl1-f169.google.com with SMTP id d9443c01a7336-2163dc5155fso30742615ad.0 for ; Fri, 31 Jan 2025 01:08:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1738314491; x=1738919291; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=YsBQiXFOiDXEVopox/DL4fBI7MjKngkv74+OC5sfCK8=; b=ApY1VfGfIdqHu+3S+JJmCdAKG9iCdpfr7CW7L2LwTxZrEoEFMyILdN3lZDgCiO3F9z XwehE8inHcpOHoiAPziUYUpH/BMygDLQCghaKkICVhtipcCilcSJ96Qd0mjajpV1muK0 t5hMIcdbIfsL2RkF+y78RYqNed+H3+Rwf3V+Q= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738314491; x=1738919291; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=YsBQiXFOiDXEVopox/DL4fBI7MjKngkv74+OC5sfCK8=; b=WspvCscDlB/6+unmaCGyJQAAQSZVIK7ds6k78fEirqV12g5BsE+IUnTTGkTBleEt1B 8PJEVgcPxQufmg83XChw9OW1rqllYK82oSlcx5rfT6u9xK7c18Z/GK1reF2a3bnPS3hL 7JoMJ8NBRNHksWqpiq4U+vAAabvqK+UShMOtdHLK8sQ7hwLcbGAO5cDV804TGiQk3OIR mTvQJT5VFQTlLnppqLfUcKYjrQwGSp47OkbXvxVuvKr4V/hWc2Akdsz4cSyrv9MAzRmO i2HSouIjUv7b2cGGxxnzp8mS29FLHwpwaCaKd+NBbbj4GkgQNBOFnsaH9JEPq/2SfK3X vKdA== X-Forwarded-Encrypted: i=1; AJvYcCWOky83UeWYe1GYHp9nOOM6ZYN1JYFB1KPblr+NS3IkhSfffflU9Fcva4zbAw2D9/aFWSF91f0RRyP21x8=@vger.kernel.org X-Gm-Message-State: AOJu0YzrQHx8g3ZA8OiBxNbNNePQd2hpeDaCVGPZNmXsngbGiZNRabj5 xOv/OZ7LQas4jco7QjEavHYae1JMQ7Zu0OKDUpqAh0vPX1odtz6Kr7BHsSXyog== X-Gm-Gg: ASbGncuvdQzOk6agk85PwqGomOuth1nw8EhP3A8m6aN4MrAj4pcGBYANwKQLp6JDD6A jxUKZFjv1VpCL+sx7gMqlin2u0hVWFMafZMKawsuhsK2n/s9MTQPnTe4CSVoeAPn2SU+bfNig00 qY3DftJDOvfdi0VAxs1WrXtFPJk52lcGOAFVNSuHQ4cxUQUUomA+Y/SSRCFi6XrWoKQeK4PO/H6 L5TdAbFewqnsOEM7TirKpQ6AoUwTzHLmRd0kriVmWl5p8Er1dV4yvFjxy9JeNHoTGBcIgPDb3Od R/FVqh9wy0w4xQfphg== X-Google-Smtp-Source: AGHT+IENlfyVLhUqijJuU7j/zhaZph0w7819UX7xL0oXUri1pMuVzz1XrUZpnmESTer0aRPTpzNpUQ== X-Received: by 2002:a17:902:dacd:b0:216:50fb:5dfc with SMTP id d9443c01a7336-21dd7c3cddemr135410395ad.9.1738314491281; Fri, 31 Jan 2025 01:08:11 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:c752:be9d:3368:16fa]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-21de33032dbsm25983505ad.191.2025.01.31.01.08.09 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 31 Jan 2025 01:08:10 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv4 16/17] zram: switch to new zsmalloc object mapping API Date: Fri, 31 Jan 2025 18:06:15 +0900 Message-ID: <20250131090658.3386285-17-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.362.g079036d154-goog In-Reply-To: <20250131090658.3386285-1-senozhatsky@chromium.org> References: <20250131090658.3386285-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Use new read/write zsmalloc object API. For cases when RO mapped object spans two physical pages (requires temp buffer) compression streams now carry around one extra physical page. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zcomp.c | 4 +++- drivers/block/zram/zcomp.h | 2 ++ drivers/block/zram/zram_drv.c | 28 ++++++++++------------------ 3 files changed, 15 insertions(+), 19 deletions(-) diff --git a/drivers/block/zram/zcomp.c b/drivers/block/zram/zcomp.c index efd5919808d9..675f2a51ad5f 100644 --- a/drivers/block/zram/zcomp.c +++ b/drivers/block/zram/zcomp.c @@ -45,6 +45,7 @@ static const struct zcomp_ops *backends[] =3D { static void zcomp_strm_free(struct zcomp *comp, struct zcomp_strm *strm) { comp->ops->destroy_ctx(&strm->ctx); + vfree(strm->local_copy); vfree(strm->buffer); kfree(strm); } @@ -66,12 +67,13 @@ static struct zcomp_strm *zcomp_strm_alloc(struct zcomp= *comp) return NULL; } =20 + strm->local_copy =3D vzalloc(PAGE_SIZE); /* * allocate 2 pages. 1 for compressed data, plus 1 extra in case if * compressed data is larger than the original one. */ strm->buffer =3D vzalloc(2 * PAGE_SIZE); - if (!strm->buffer) { + if (!strm->buffer || !strm->local_copy) { zcomp_strm_free(comp, strm); return NULL; } diff --git a/drivers/block/zram/zcomp.h b/drivers/block/zram/zcomp.h index 62330829db3f..9683d4aa822d 100644 --- a/drivers/block/zram/zcomp.h +++ b/drivers/block/zram/zcomp.h @@ -34,6 +34,8 @@ struct zcomp_strm { struct list_head entry; /* compression buffer */ void *buffer; + /* local copy of handle memory */ + void *local_copy; struct zcomp_ctx ctx; }; =20 diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index cfbb3072ee9e..f85502ae7dce 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1553,11 +1553,11 @@ static int read_incompressible_page(struct zram *zr= am, struct page *page, void *src, *dst; =20 handle =3D zram_get_handle(zram, index); - src =3D zs_map_object(zram->mem_pool, handle, ZS_MM_RO); + src =3D zs_obj_read_begin(zram->mem_pool, handle, NULL); dst =3D kmap_local_page(page); copy_page(dst, src); kunmap_local(dst); - zs_unmap_object(zram->mem_pool, handle); + zs_obj_read_end(zram->mem_pool, handle, src); =20 return 0; } @@ -1575,11 +1575,11 @@ static int read_compressed_page(struct zram *zram, = struct page *page, u32 index) prio =3D zram_get_priority(zram, index); =20 zstrm =3D zcomp_stream_get(zram->comps[prio]); - src =3D zs_map_object(zram->mem_pool, handle, ZS_MM_RO); + src =3D zs_obj_read_begin(zram->mem_pool, handle, zstrm->local_copy); dst =3D kmap_local_page(page); ret =3D zcomp_decompress(zram->comps[prio], zstrm, src, size, dst); kunmap_local(dst); - zs_unmap_object(zram->mem_pool, handle); + zs_obj_read_end(zram->mem_pool, handle, src); zcomp_stream_put(zram->comps[prio], zstrm); =20 return ret; @@ -1675,7 +1675,7 @@ static int write_incompressible_page(struct zram *zra= m, struct page *page, u32 index) { unsigned long handle; - void *src, *dst; + void *src; =20 /* * This function is called from preemptible context so we don't need @@ -1692,11 +1692,9 @@ static int write_incompressible_page(struct zram *zr= am, struct page *page, return -ENOMEM; } =20 - dst =3D zs_map_object(zram->mem_pool, handle, ZS_MM_WO); src =3D kmap_local_page(page); - memcpy(dst, src, PAGE_SIZE); + zs_obj_write(zram->mem_pool, handle, src, PAGE_SIZE); kunmap_local(src); - zs_unmap_object(zram->mem_pool, handle); =20 zram_slot_write_lock(zram, index); zram_set_flag(zram, index, ZRAM_HUGE); @@ -1717,7 +1715,7 @@ static int zram_write_page(struct zram *zram, struct = page *page, u32 index) int ret =3D 0; unsigned long handle; unsigned int comp_len; - void *dst, *mem; + void *mem; struct zcomp_strm *zstrm; unsigned long element; bool same_filled; @@ -1760,11 +1758,8 @@ static int zram_write_page(struct zram *zram, struct= page *page, u32 index) return -ENOMEM; } =20 - dst =3D zs_map_object(zram->mem_pool, handle, ZS_MM_WO); - - memcpy(dst, zstrm->buffer, comp_len); + zs_obj_write(zram->mem_pool, handle, zstrm->buffer, comp_len); zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP], zstrm); - zs_unmap_object(zram->mem_pool, handle); =20 zram_slot_write_lock(zram, index); zram_set_handle(zram, index, handle); @@ -1876,7 +1871,7 @@ static int recompress_slot(struct zram *zram, u32 ind= ex, struct page *page, unsigned int comp_len_new; unsigned int class_index_old; unsigned int class_index_new; - void *src, *dst; + void *src; int ret; =20 handle_old =3D zram_get_handle(zram, index); @@ -2000,12 +1995,9 @@ static int recompress_slot(struct zram *zram, u32 in= dex, struct page *page, return 0; } =20 - dst =3D zs_map_object(zram->mem_pool, handle_new, ZS_MM_WO); - memcpy(dst, zstrm->buffer, comp_len_new); + zs_obj_write(zram->mem_pool, handle_new, zstrm->buffer, comp_len_new); zcomp_stream_put(zram->comps[prio], zstrm); =20 - zs_unmap_object(zram->mem_pool, handle_new); - zram_free_page(zram, index); zram_set_handle(zram, index, handle_new); zram_set_obj_size(zram, index, comp_len_new); --=20 2.48.1.362.g079036d154-goog From nobody Mon Feb 9 02:42:15 2026 Received: from mail-pl1-f181.google.com (mail-pl1-f181.google.com [209.85.214.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3C1EC1A8F98 for ; Fri, 31 Jan 2025 09:08:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738314497; cv=none; b=PEIOHmsUE4rcxSSz6xfD0t/IeQaLqvHBvPRzR3Nn2rLe3pzGbGlEPFYtU2gESQ9DUMC1wxxOcIXxchKFKc6XMzIFduoblzoWf7ehlugz3T7Rp0RRque+rNEnpZSqo4+nAWmUli5hdSvq83UtvlF7H22W3OKiD6r8PSHs6sfVNRU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738314497; c=relaxed/simple; bh=pOm+fpdBAA+J1ecb0mT1Wv1tDOg3X9T9T8tfx6TqtYE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=qQrhXugeAKa8m80CQh0jbOYQCrXLA4fGIzEcwYgbN/wGZQGu5PyC9ybzHmcLcnd89VxGuOCACwa/xNtUC043kuI4qH04lcvM/VO1JwuHzJMoDYosJrAkkaCoQs5T/dTeAZ8LspJt5NyXNlAaiqF5VXrTeijOgf9xgKW1/Pi7qWk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=j4auTGAh; arc=none smtp.client-ip=209.85.214.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="j4auTGAh" Received: by mail-pl1-f181.google.com with SMTP id d9443c01a7336-2166022c5caso25914665ad.2 for ; Fri, 31 Jan 2025 01:08:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1738314495; x=1738919295; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=DQcqfpSqnss0R3EkpuRQBovJzgNoPvoL6ThFstuCphQ=; b=j4auTGAhfpit29QG5wuDpuxNC04UCN2QD/Gk/y/YkNGlR7QreOtzle5MLyBcjQOjsx uhJ75UUpNA0OGNY87JBiuRcFcBrwndujEfYEUjxlpgaF8sIvXn9bb5Z4B2SrNvMD3L9q N5VYAmlN41/HRzythvhU/kWXeeUYdP93L043Q= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738314495; x=1738919295; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=DQcqfpSqnss0R3EkpuRQBovJzgNoPvoL6ThFstuCphQ=; b=nSNEVBgXufPAHMtnZJElDP1x1Xn/yOAr6g3nB6NBfebvhOehQeKKDE5VOkoWDeBsmB DvK7XgIgLVdc1m8O+aqPHoDYIZ4GSiQ/4Rt2z5VVSdbsQYD9E6DxJIfEwgW88lpTQP1D s2npHT+VaueInAaGx7NTxAZhE9B35fjsaorq6+Sb8+EYiCnO2KfnpZ6jrSukmAmniYwH SGlcKmc7NO7cFb94WMi4cxN1BtkiO0szMDQaBcIvHwkgQrLFg8XHBZkENpmxDY+J9sqM 7yEvRKyzffQg0ZAwh7i+gvf+hf6SCClkRRfulZArgoX9BWLR6/b7L7kn6jaXcJdO7ARQ 3nKg== X-Forwarded-Encrypted: i=1; AJvYcCW1iGNtQsppzFNQmpOuGmz+c8xDOYOhZe5RB4l0KzQCbEVarphdVTMBMrY7CaDWDIIpFZhlZ+B3Xt9i4nA=@vger.kernel.org X-Gm-Message-State: AOJu0YxlL098pgoQo2t7qfevq0P60WxE+3rEw2SVJvgeac5l9roGoGFO TZOU1pL61ynsCbfVGfdGhv/ZnryBBimhBclGRka0nC7MxMZxrb3o5vyWsrqeCyT4r7B57Vr0XAY = X-Gm-Gg: ASbGnctwWrywvxNBoeTKL+P4guq6nzVo3BfpuTWKNlcAjWbZPjU+0CuZQ60gEl9WRaC KPQKVfDaUDO9D+GER0a8bNC7mSt363aGrXLa0CDGZLh+4nKm86cSSXPtOwH37qtFmjtCloXWrc8 m+Aa/KbRolDr/VxPMcaxcUMwxTpbjcTZSYPDC1V9eysHFx1H/Tb808eZXKMDmzqEXMHnh1y58dn QsBIobLgINeHKIDsSiihOofTwE9+YAmowp2lFzUXZHBlfxjJ79lB1g9sAk408mOi0zR7rIXhQ8z OwF6Z9NPjR+nBV+RuA== X-Google-Smtp-Source: AGHT+IEmt7Kd9WrX0z/MrdYcFEjWtu/6bbm1WyY2+91BuQrMC+eYVRLixUf3IzSlvbxl7TXEySCQzg== X-Received: by 2002:a17:903:2f8c:b0:215:b9a6:5cb9 with SMTP id d9443c01a7336-21dd7c3555bmr174992725ad.5.1738314495518; Fri, 31 Jan 2025 01:08:15 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:c752:be9d:3368:16fa]) by smtp.gmail.com with UTF8SMTPSA id 98e67ed59e1d1-2f8489d3982sm3396499a91.24.2025.01.31.01.08.13 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 31 Jan 2025 01:08:15 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv4 17/17] zram: add might_sleep to zcomp API Date: Fri, 31 Jan 2025 18:06:16 +0900 Message-ID: <20250131090658.3386285-18-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.362.g079036d154-goog In-Reply-To: <20250131090658.3386285-1-senozhatsky@chromium.org> References: <20250131090658.3386285-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Explicitly state that zcomp compress/decompress must be called from non-atomic context. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zcomp.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/drivers/block/zram/zcomp.c b/drivers/block/zram/zcomp.c index 675f2a51ad5f..f4235735787b 100644 --- a/drivers/block/zram/zcomp.c +++ b/drivers/block/zram/zcomp.c @@ -185,6 +185,7 @@ int zcomp_compress(struct zcomp *comp, struct zcomp_str= m *zstrm, }; int ret; =20 + might_sleep(); ret =3D comp->ops->compress(comp->params, &zstrm->ctx, &req); if (!ret) *dst_len =3D req.dst_len; @@ -201,6 +202,7 @@ int zcomp_decompress(struct zcomp *comp, struct zcomp_s= trm *zstrm, .dst_len =3D PAGE_SIZE, }; =20 + might_sleep(); return comp->ops->decompress(comp->params, &zstrm->ctx, &req); } =20 --=20 2.48.1.362.g079036d154-goog