From nobody Fri Jan 31 03:24:27 2025 Received: from mail-pj1-f54.google.com (mail-pj1-f54.google.com [209.85.216.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 142CA1D63CC for ; Thu, 30 Jan 2025 11:11:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.54 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738235480; cv=none; b=QzxqWuwheWSxbhiKqVRR6mMMmnbLxjXgamigIUN+ZhpmNYJXJ32QGSWH48Sq4iaACyu/Ym85Fdwy4dUUCIVlQMW1eyEStPX1x/fMiZ0SU5xvDR4RkGNrgjHTML7YScerVrwI9t/XN184cuqMiVH4fO5MAg9c3tYzFCKhCYkFNwo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738235480; c=relaxed/simple; bh=bi535dLoQGiTDMiC81NLzM1hPsLe/brJOggz9JySmb0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=QPp9RbBF0Xf0TKPSlHPyQ8yqxOrHZ1OmKmljl/BtjSm5KSem3OcE96xRsc9FoI0wFJ4jnwTgbqG7z02gS8jRgmUtX+ShZ/wNDqs4n5CXvizq3kYxBWMfCbYSt5+MhUH1adVhGGUbBkmYLGqK4/CYR4VQMNEttzO3UMGkVZOM5Ow= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=Ii86Ot1t; arc=none smtp.client-ip=209.85.216.54 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="Ii86Ot1t" Received: by mail-pj1-f54.google.com with SMTP id 98e67ed59e1d1-2f833af7a09so765794a91.2 for ; Thu, 30 Jan 2025 03:11:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1738235478; x=1738840278; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=vG/tk94hewr73CT+Z4xbU1wydP7qgFMVbNXS/fz+Vpw=; b=Ii86Ot1tmXH3dw1U6KSv0LLyyGSrU8w/08Dz1rkyeYRM6Z6P6M5nPHJgfarQk3P5hi nZbwSGb8kke2GTolztDNmEhFpH6F8mA6f8nBZItekJ1eJL2q8RMGQEvcngiT3PMurYqd vRp1XmJWyge5MgNvQ3OGsZR34o1lU4Tuuf6s8= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738235478; x=1738840278; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=vG/tk94hewr73CT+Z4xbU1wydP7qgFMVbNXS/fz+Vpw=; b=NZ9T2tvlCOCuOWb7Vunub0s7dmwwZr4qmxKnJ4ry0jsSjbjoXjHxB0XC+KGQ2BRc7F 0xgOzw3weX2snDvi3+K+RNPpJoBCBLuqM482TQhDA/4HLqopmr8V7X45znJyH+0nieQE 7yvqiQcSTNV3hvcuFi5NhNpwi/1c2+7Tl74K6iatPCxxQEvsNo9F8fXalB3ZutPnOmST 3nVTNYol3SkEihmgiqT94Cy0IEhXvthA3j1/VlJkEDi1RI+qpnKcCMA/BwJJJ+RcYxhy RafeQ5sHYNiaYeJexz8Fbc2AkjJWCxvKkWnDmycL0/NvMqxPKlyePprIL2GMU7hCoj3R ijHA== X-Forwarded-Encrypted: i=1; AJvYcCXvcOIhO7eaTVhgomW45Jl2WNI+ckzrG7q5TGtffgFK9F9VtEFl1YfdKTyzCFCsLFWvKSv/yaUtEjKs0jA=@vger.kernel.org X-Gm-Message-State: AOJu0YxUaak6+IFGn08eLsHYqLycPuNaz5KGdUzS2QwUKjpdcQimh+ql 4b6CZLuTPEa6WqyyhBtYVsm428TahsAHFq8akHD0W9H2VrFiAHqftstZ+iwBaToEMrXv+2JWvk4 = X-Gm-Gg: ASbGnctlVsjK8NMNko1wj1Thq+TChDsdODtV7e65F34TivDDIXarviX8z26cCQlixYL Y7gsC19ltTEl/Zm2vHl8nbCwmjLz+zmDfZo9siwIrtUaxjUFk4eWJJUXGewfpzuExBkhoR7OjW/ UTfb3u7IeqsX7g/qZZDiW2po/VJBSQUT7l6sUdHO7zJwEmpm+6u8VVLRFoTa52nYIMQkudD39CR k2n3ChmpqBW214jLTdnqIXBRzL3/MsrHFeaHpnuoviZQF1awGurDh+qRjApFBYMadxc+xvnw8hs xuimIE/UsVQYI2YP X-Google-Smtp-Source: AGHT+IEII0bxfv0lWA3uPhG032lc2h9qT5EqknbZ0568rgvm+cDL3+X00otQuH/5VTn7k1d91oV5cA== X-Received: by 2002:a17:90b:3a0c:b0:2ee:c9b6:4c42 with SMTP id 98e67ed59e1d1-2f83abff391mr10977943a91.16.1738235478355; Thu, 30 Jan 2025 03:11:18 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:d4ce:e744:f46b:4fb]) by smtp.gmail.com with UTF8SMTPSA id 98e67ed59e1d1-2f83bd0ce4dsm3638761a91.27.2025.01.30.03.11.16 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 30 Jan 2025 03:11:18 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv3 01/11] zram: switch to non-atomic entry locking Date: Thu, 30 Jan 2025 20:10:46 +0900 Message-ID: <20250130111105.2861324-2-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.262.g85cc9f2d1e-goog In-Reply-To: <20250130111105.2861324-1-senozhatsky@chromium.org> References: <20250130111105.2861324-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Concurrent modifications of meta table entries is now handled by per-entry spin-lock. This has a number of shortcomings. First, this imposes atomic requirements on compression backends. zram can call both zcomp_compress() and zcomp_decompress() under entry spin-lock, which implies that we can use only compression algorithms that don't schedule/sleep/wait during compression and decompression. This, for instance, makes it impossible to use some of the ASYNC compression algorithms (H/W compression, etc.) implementations. Second, this can potentially trigger watchdogs. For example, entry re-compression with secondary algorithms is performed under entry spin-lock. Given that we chain secondary compression algorithms and that some of them can be configured for best compression ratio (and worst compression speed) zram can stay under spin-lock for quite some time. Do not use per-entry spin-locks and instead convert it to an atomic_t variable which open codes reader-writer type of lock. This permits preemption from slot_lock section. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 148 ++++++++++++++++++++-------------- drivers/block/zram/zram_drv.h | 7 +- 2 files changed, 91 insertions(+), 64 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 9f5020b077c5..a8d22ae2a066 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -58,19 +58,50 @@ static void zram_free_page(struct zram *zram, size_t in= dex); static int zram_read_from_zspool(struct zram *zram, struct page *page, u32 index); =20 -static int zram_slot_trylock(struct zram *zram, u32 index) +static bool zram_slot_try_write_lock(struct zram *zram, u32 index) { - return spin_trylock(&zram->table[index].lock); + atomic_t *lock =3D &zram->table[index].lock; + int old =3D ZRAM_ENTRY_UNLOCKED; + + return atomic_try_cmpxchg(lock, &old, ZRAM_ENTRY_WRLOCKED); } =20 -static void zram_slot_lock(struct zram *zram, u32 index) +static void zram_slot_write_lock(struct zram *zram, u32 index) { - spin_lock(&zram->table[index].lock); + atomic_t *lock =3D &zram->table[index].lock; + int old =3D atomic_read(lock); + + do { + if (old !=3D ZRAM_ENTRY_UNLOCKED) { + cond_resched(); + old =3D atomic_read(lock); + continue; + } + } while (!atomic_try_cmpxchg(lock, &old, ZRAM_ENTRY_WRLOCKED)); +} + +static void zram_slot_write_unlock(struct zram *zram, u32 index) +{ + atomic_set(&zram->table[index].lock, ZRAM_ENTRY_UNLOCKED); +} + +static void zram_slot_read_lock(struct zram *zram, u32 index) +{ + atomic_t *lock =3D &zram->table[index].lock; + int old =3D atomic_read(lock); + + do { + if (old =3D=3D ZRAM_ENTRY_WRLOCKED) { + cond_resched(); + old =3D atomic_read(lock); + continue; + } + } while (!atomic_try_cmpxchg(lock, &old, old + 1)); } =20 -static void zram_slot_unlock(struct zram *zram, u32 index) +static void zram_slot_read_unlock(struct zram *zram, u32 index) { - spin_unlock(&zram->table[index].lock); + atomic_dec(&zram->table[index].lock); } =20 static inline bool init_done(struct zram *zram) @@ -93,7 +124,6 @@ static void zram_set_handle(struct zram *zram, u32 index= , unsigned long handle) zram->table[index].handle =3D handle; } =20 -/* flag operations require table entry bit_spin_lock() being held */ static bool zram_test_flag(struct zram *zram, u32 index, enum zram_pageflags flag) { @@ -229,9 +259,9 @@ static void release_pp_slot(struct zram *zram, struct z= ram_pp_slot *pps) { list_del_init(&pps->entry); =20 - zram_slot_lock(zram, pps->index); + zram_slot_write_lock(zram, pps->index); zram_clear_flag(zram, pps->index, ZRAM_PP_SLOT); - zram_slot_unlock(zram, pps->index); + zram_slot_write_unlock(zram, pps->index); =20 kfree(pps); } @@ -394,11 +424,11 @@ static void mark_idle(struct zram *zram, ktime_t cuto= ff) * * And ZRAM_WB slots simply cannot be ZRAM_IDLE. */ - zram_slot_lock(zram, index); + zram_slot_write_lock(zram, index); if (!zram_allocated(zram, index) || zram_test_flag(zram, index, ZRAM_WB) || zram_test_flag(zram, index, ZRAM_SAME)) { - zram_slot_unlock(zram, index); + zram_slot_write_unlock(zram, index); continue; } =20 @@ -410,7 +440,7 @@ static void mark_idle(struct zram *zram, ktime_t cutoff) zram_set_flag(zram, index, ZRAM_IDLE); else zram_clear_flag(zram, index, ZRAM_IDLE); - zram_slot_unlock(zram, index); + zram_slot_write_unlock(zram, index); } } =20 @@ -709,7 +739,7 @@ static int scan_slots_for_writeback(struct zram *zram, = u32 mode, =20 INIT_LIST_HEAD(&pps->entry); =20 - zram_slot_lock(zram, index); + zram_slot_write_lock(zram, index); if (!zram_allocated(zram, index)) goto next; =20 @@ -731,7 +761,7 @@ static int scan_slots_for_writeback(struct zram *zram, = u32 mode, place_pp_slot(zram, ctl, pps); pps =3D NULL; next: - zram_slot_unlock(zram, index); + zram_slot_write_unlock(zram, index); } =20 kfree(pps); @@ -822,7 +852,7 @@ static ssize_t writeback_store(struct device *dev, } =20 index =3D pps->index; - zram_slot_lock(zram, index); + zram_slot_read_lock(zram, index); /* * scan_slots() sets ZRAM_PP_SLOT and relases slot lock, so * slots can change in the meantime. If slots are accessed or @@ -833,7 +863,7 @@ static ssize_t writeback_store(struct device *dev, goto next; if (zram_read_from_zspool(zram, page, index)) goto next; - zram_slot_unlock(zram, index); + zram_slot_read_unlock(zram, index); =20 bio_init(&bio, zram->bdev, &bio_vec, 1, REQ_OP_WRITE | REQ_SYNC); @@ -860,7 +890,7 @@ static ssize_t writeback_store(struct device *dev, } =20 atomic64_inc(&zram->stats.bd_writes); - zram_slot_lock(zram, index); + zram_slot_write_lock(zram, index); /* * Same as above, we release slot lock during writeback so * slot can change under us: slot_free() or slot_free() and @@ -882,7 +912,7 @@ static ssize_t writeback_store(struct device *dev, zram->bd_wb_limit -=3D 1UL << (PAGE_SHIFT - 12); spin_unlock(&zram->wb_limit_lock); next: - zram_slot_unlock(zram, index); + zram_slot_write_unlock(zram, index); release_pp_slot(zram, pps); =20 cond_resched(); @@ -1001,7 +1031,7 @@ static ssize_t read_block_state(struct file *file, ch= ar __user *buf, for (index =3D *ppos; index < nr_pages; index++) { int copied; =20 - zram_slot_lock(zram, index); + zram_slot_read_lock(zram, index); if (!zram_allocated(zram, index)) goto next; =20 @@ -1019,13 +1049,13 @@ static ssize_t read_block_state(struct file *file, = char __user *buf, ZRAM_INCOMPRESSIBLE) ? 'n' : '.'); =20 if (count <=3D copied) { - zram_slot_unlock(zram, index); + zram_slot_read_unlock(zram, index); break; } written +=3D copied; count -=3D copied; next: - zram_slot_unlock(zram, index); + zram_slot_read_unlock(zram, index); *ppos +=3D 1; } =20 @@ -1455,33 +1485,31 @@ static void zram_meta_free(struct zram *zram, u64 d= isksize) =20 static bool zram_meta_alloc(struct zram *zram, u64 disksize) { - size_t num_pages, index; + size_t num_ents, index; =20 - num_pages =3D disksize >> PAGE_SHIFT; - zram->table =3D vzalloc(array_size(num_pages, sizeof(*zram->table))); + num_ents =3D disksize >> PAGE_SHIFT; + zram->table =3D vzalloc(array_size(num_ents, sizeof(*zram->table))); if (!zram->table) - return false; + goto error; =20 zram->mem_pool =3D zs_create_pool(zram->disk->disk_name); - if (!zram->mem_pool) { - vfree(zram->table); - zram->table =3D NULL; - return false; - } + if (!zram->mem_pool) + goto error; =20 if (!huge_class_size) huge_class_size =3D zs_huge_class_size(zram->mem_pool); =20 - for (index =3D 0; index < num_pages; index++) - spin_lock_init(&zram->table[index].lock); + for (index =3D 0; index < num_ents; index++) + atomic_set(&zram->table[index].lock, ZRAM_ENTRY_UNLOCKED); + return true; + +error: + vfree(zram->table); + zram->table =3D NULL; + return false; } =20 -/* - * To protect concurrent access to the same index entry, - * caller should hold this table index entry's bit_spinlock to - * indicate this index entry is accessing. - */ static void zram_free_page(struct zram *zram, size_t index) { unsigned long handle; @@ -1602,17 +1630,17 @@ static int zram_read_page(struct zram *zram, struct= page *page, u32 index, { int ret; =20 - zram_slot_lock(zram, index); + zram_slot_read_lock(zram, index); if (!zram_test_flag(zram, index, ZRAM_WB)) { /* Slot should be locked through out the function call */ ret =3D zram_read_from_zspool(zram, page, index); - zram_slot_unlock(zram, index); + zram_slot_read_unlock(zram, index); } else { /* * The slot should be unlocked before reading from the backing * device. */ - zram_slot_unlock(zram, index); + zram_slot_read_unlock(zram, index); =20 ret =3D read_from_bdev(zram, page, zram_get_handle(zram, index), parent); @@ -1655,10 +1683,10 @@ static int zram_bvec_read(struct zram *zram, struct= bio_vec *bvec, static int write_same_filled_page(struct zram *zram, unsigned long fill, u32 index) { - zram_slot_lock(zram, index); + zram_slot_write_lock(zram, index); zram_set_flag(zram, index, ZRAM_SAME); zram_set_handle(zram, index, fill); - zram_slot_unlock(zram, index); + zram_slot_write_unlock(zram, index); =20 atomic64_inc(&zram->stats.same_pages); atomic64_inc(&zram->stats.pages_stored); @@ -1693,11 +1721,11 @@ static int write_incompressible_page(struct zram *z= ram, struct page *page, kunmap_local(src); zs_unmap_object(zram->mem_pool, handle); =20 - zram_slot_lock(zram, index); + zram_slot_write_lock(zram, index); zram_set_flag(zram, index, ZRAM_HUGE); zram_set_handle(zram, index, handle); zram_set_obj_size(zram, index, PAGE_SIZE); - zram_slot_unlock(zram, index); + zram_slot_write_unlock(zram, index); =20 atomic64_add(PAGE_SIZE, &zram->stats.compr_data_size); atomic64_inc(&zram->stats.huge_pages); @@ -1718,9 +1746,9 @@ static int zram_write_page(struct zram *zram, struct = page *page, u32 index) bool same_filled; =20 /* First, free memory allocated to this slot (if any) */ - zram_slot_lock(zram, index); + zram_slot_write_lock(zram, index); zram_free_page(zram, index); - zram_slot_unlock(zram, index); + zram_slot_write_unlock(zram, index); =20 mem =3D kmap_local_page(page); same_filled =3D page_same_filled(mem, &element); @@ -1790,10 +1818,10 @@ static int zram_write_page(struct zram *zram, struc= t page *page, u32 index) zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); zs_unmap_object(zram->mem_pool, handle); =20 - zram_slot_lock(zram, index); + zram_slot_write_lock(zram, index); zram_set_handle(zram, index, handle); zram_set_obj_size(zram, index, comp_len); - zram_slot_unlock(zram, index); + zram_slot_write_unlock(zram, index); =20 /* Update stats */ atomic64_inc(&zram->stats.pages_stored); @@ -1850,7 +1878,7 @@ static int scan_slots_for_recompress(struct zram *zra= m, u32 mode, =20 INIT_LIST_HEAD(&pps->entry); =20 - zram_slot_lock(zram, index); + zram_slot_write_lock(zram, index); if (!zram_allocated(zram, index)) goto next; =20 @@ -1871,7 +1899,7 @@ static int scan_slots_for_recompress(struct zram *zra= m, u32 mode, place_pp_slot(zram, ctl, pps); pps =3D NULL; next: - zram_slot_unlock(zram, index); + zram_slot_write_unlock(zram, index); } =20 kfree(pps); @@ -2162,7 +2190,7 @@ static ssize_t recompress_store(struct device *dev, if (!num_recomp_pages) break; =20 - zram_slot_lock(zram, pps->index); + zram_slot_write_lock(zram, pps->index); if (!zram_test_flag(zram, pps->index, ZRAM_PP_SLOT)) goto next; =20 @@ -2170,7 +2198,7 @@ static ssize_t recompress_store(struct device *dev, &num_recomp_pages, threshold, prio, prio_max); next: - zram_slot_unlock(zram, pps->index); + zram_slot_write_unlock(zram, pps->index); release_pp_slot(zram, pps); =20 if (err) { @@ -2217,9 +2245,9 @@ static void zram_bio_discard(struct zram *zram, struc= t bio *bio) } =20 while (n >=3D PAGE_SIZE) { - zram_slot_lock(zram, index); + zram_slot_write_lock(zram, index); zram_free_page(zram, index); - zram_slot_unlock(zram, index); + zram_slot_write_unlock(zram, index); atomic64_inc(&zram->stats.notify_free); index++; n -=3D PAGE_SIZE; @@ -2248,9 +2276,9 @@ static void zram_bio_read(struct zram *zram, struct b= io *bio) } flush_dcache_page(bv.bv_page); =20 - zram_slot_lock(zram, index); + zram_slot_write_lock(zram, index); zram_accessed(zram, index); - zram_slot_unlock(zram, index); + zram_slot_write_unlock(zram, index); =20 bio_advance_iter_single(bio, &iter, bv.bv_len); } while (iter.bi_size); @@ -2278,9 +2306,9 @@ static void zram_bio_write(struct zram *zram, struct = bio *bio) break; } =20 - zram_slot_lock(zram, index); + zram_slot_write_lock(zram, index); zram_accessed(zram, index); - zram_slot_unlock(zram, index); + zram_slot_write_unlock(zram, index); =20 bio_advance_iter_single(bio, &iter, bv.bv_len); } while (iter.bi_size); @@ -2321,13 +2349,13 @@ static void zram_slot_free_notify(struct block_devi= ce *bdev, zram =3D bdev->bd_disk->private_data; =20 atomic64_inc(&zram->stats.notify_free); - if (!zram_slot_trylock(zram, index)) { + if (!zram_slot_try_write_lock(zram, index)) { atomic64_inc(&zram->stats.miss_free); return; } =20 zram_free_page(zram, index); - zram_slot_unlock(zram, index); + zram_slot_write_unlock(zram, index); } =20 static void zram_comp_params_reset(struct zram *zram) diff --git a/drivers/block/zram/zram_drv.h b/drivers/block/zram/zram_drv.h index db78d7c01b9a..3436ddf8ab23 100644 --- a/drivers/block/zram/zram_drv.h +++ b/drivers/block/zram/zram_drv.h @@ -15,7 +15,6 @@ #ifndef _ZRAM_DRV_H_ #define _ZRAM_DRV_H_ =20 -#include #include #include =20 @@ -28,7 +27,6 @@ #define ZRAM_SECTOR_PER_LOGICAL_BLOCK \ (1 << (ZRAM_LOGICAL_BLOCK_SHIFT - SECTOR_SHIFT)) =20 - /* * ZRAM is mainly used for memory efficiency so we want to keep memory * footprint small and thus squeeze size and zram pageflags into a flags @@ -58,13 +56,14 @@ enum zram_pageflags { __NR_ZRAM_PAGEFLAGS, }; =20 -/*-- Data structures */ +#define ZRAM_ENTRY_UNLOCKED 0 +#define ZRAM_ENTRY_WRLOCKED (-1) =20 /* Allocated for each disk page */ struct zram_table_entry { unsigned long handle; unsigned int flags; - spinlock_t lock; + atomic_t lock; #ifdef CONFIG_ZRAM_TRACK_ENTRY_ACTIME ktime_t ac_time; #endif --=20 2.48.1.262.g85cc9f2d1e-goog From nobody Fri Jan 31 03:24:27 2025 Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6758F1D88DB for ; Thu, 30 Jan 2025 11:11:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738235485; cv=none; b=qikPMsM4CtolTYqkqh8O8rrmdU5jhllymKHG/duUmxuJiArouyXI9gmyQiO7snoes63qfW/qpUiwdvsBSuG4E03DIzTnrc3nY5LhqPphoBlulh4A7zfcLXBg2cwr+Cys3V6RF7bzfnqZDUrsNXggTABtv3iXG9TvxhXKtXVhTW4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738235485; c=relaxed/simple; bh=/K1ug93PyFpiNGqzZaTzZSZpc1SzMsvbzFMySMv/590=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Ix6dZMDcr6r2k3ZH8T986d+EMkUebanvnbFt5jn9KK0nK+PMF9PrgyMlBxEzoQREkSh5FkC5aAXrTdwGAPpZ9OrAmN8mdpHU862OaJumQMFO0JocFrC8lL3ncPPpFZPYvvzPuah3Na+jbLNF+IT7PlEuDcs84ybMJUyoI6WX8Xk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=ABKgonJw; arc=none smtp.client-ip=209.85.214.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="ABKgonJw" Received: by mail-pl1-f182.google.com with SMTP id d9443c01a7336-2166360285dso9972185ad.1 for ; Thu, 30 Jan 2025 03:11:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1738235482; x=1738840282; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=UEH++QnSGY0Fi6Z8VZI2JREfdhZluHMkf5kZ0a7B1uQ=; b=ABKgonJwEHgiPu7raHlzFHGg3xXkVCC7A96yHHh+TsBCYXRN9oQz+dE2weSTmu5hH1 uZGlAlVm2g4X3IyYobhAPmqPkrFW3MO1XdnBrgdWawDYmBSJ1Tf6n/nObql90Aceo/N1 nLXe57cfIWBcUDlOsxS+9AiOuY7efXGBr5F18= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738235482; x=1738840282; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=UEH++QnSGY0Fi6Z8VZI2JREfdhZluHMkf5kZ0a7B1uQ=; b=CsCRuBgCGr9P5kxXXxa7TOfsoKl8qjrXVU5QN112jgDSlMAqTdC8gaDRtfcAyIV+55 7Rk8I2/UHpmsefYySXaunTBNA+wCpYBty5+YHfe8/SYmQArdKk/UTHg3yKSwoM/DhsrL o3P2Ud4frFqF6b1wwai6GnNbQBAVCs0eI/C6e7pAKOmwfMAVOGuboQyNpOBojLoSuyP8 I/p3Dm1EOlXzVGmT85z72buwNqHKajdyZmOjWMg6PDZr+ebzcaZ59eaDGMPU+XqNvx4U 6TSzqxk+dP8pNVDQ6PVQLJLG1/3qMXuBZU1tGunQ3hXT2H7aYShKDIr6kdDrn/IiPv19 qJ2A== X-Forwarded-Encrypted: i=1; AJvYcCXmVXws02mcSkLFHNHxPfJ2qg1mD3Sy7gjtPRGA75mPQ/EOOd+Bjp+3QBbE0x4t+vvZn8lar6R3AgD/BLY=@vger.kernel.org X-Gm-Message-State: AOJu0YwtKzGKZy8JA3/KKW9iu9vqWaldrnBRwdh/D2vsZhAVpaAU6yft JqCzh6l73Pn2xnLUqULj8k0x0zKW0vafnPbx4aJzmN62z6WFSdR1enp2n13/Vg== X-Gm-Gg: ASbGnct8aDWzasvE37fnEM43PdFrmnR5OI18Q2du7QO3hfj2a1SD2x7iS934XkYhgP9 fXKthL3n8aPvs0J6mhVzYhXRGbse6zYXQ0bV0kkNezloUvmoJxpy+Du4XM85AcxI0qiZ8aLykuE JszcwHuaJoHQu9UqGtF1rA8gWMlKHUNuvFE8OZlj67Xwn5Z6n9TuFsCtOFbx77Ykuz5qNkZkZvI Uk3nCPu2351MrldlQPCyZHgvPaOphj+l251BxGj6eQqGCQAyCCJaE0D1+Rn/cfINOscqD10NIk/ Tv2IUE2Q5KcHmFL2 X-Google-Smtp-Source: AGHT+IH+TH8HeSlHJqfOK6VsPbvyML0ro0yaIHBMXXZ5hNOFlr6JBn/DYjgqdQ5lAb/0oKdeeTUdYw== X-Received: by 2002:a05:6a21:7882:b0:1e7:6f82:3217 with SMTP id adf61e73a8af0-1ed7a5c2316mr9568435637.3.1738235482594; Thu, 30 Jan 2025 03:11:22 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:d4ce:e744:f46b:4fb]) by smtp.gmail.com with UTF8SMTPSA id d2e1a72fcca58-72fe69ba47esm1183433b3a.96.2025.01.30.03.11.20 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 30 Jan 2025 03:11:22 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv3 02/11] zram: do not use per-CPU compression streams Date: Thu, 30 Jan 2025 20:10:47 +0900 Message-ID: <20250130111105.2861324-3-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.262.g85cc9f2d1e-goog In-Reply-To: <20250130111105.2861324-1-senozhatsky@chromium.org> References: <20250130111105.2861324-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Similarly to per-entry spin-lock per-CPU compression streams also have a number of shortcoming. First, per-CPU stream access has to be done from a non-preemptible (atomic) section, which imposes the same atomicity requirements on compression backends as entry spin-lock do and makes it impossible to use algorithms that can schedule/wait/sleep during compression and decompression. Second, per-CPU streams noticeably increase memory usage (actually more like wastage) of secondary compression streams. The problem is that secondary compression streams are allocated per-CPU, just like the primary streams are. Yet we never use more that one secondary stream at a time, because recompression is a single threaded action. Which means that remaining num_online_cpu() - 1 streams are allocated for nothing, and this is per-priority list (we can have several secondary compression algorithms). Depending on the algorithm this may lead to a significant memory wastage, in addition each stream also carries a workmem buffer (2 physical pages). Instead of per-CPU streams, maintain a list of idle compression streams and allocate new streams on-demand (something that we used to do many years ago). So that zram read() and write() become non-atomic and ease requirements on the compression algorithm implementation. This also means that we now should have only one secondary stream per-priority list. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zcomp.c | 164 +++++++++++++++++++--------------- drivers/block/zram/zcomp.h | 17 ++-- drivers/block/zram/zram_drv.c | 29 +++--- include/linux/cpuhotplug.h | 1 - 4 files changed, 109 insertions(+), 102 deletions(-) diff --git a/drivers/block/zram/zcomp.c b/drivers/block/zram/zcomp.c index bb514403e305..982c769d5831 100644 --- a/drivers/block/zram/zcomp.c +++ b/drivers/block/zram/zcomp.c @@ -6,7 +6,7 @@ #include #include #include -#include +#include #include #include =20 @@ -43,31 +43,40 @@ static const struct zcomp_ops *backends[] =3D { NULL }; =20 -static void zcomp_strm_free(struct zcomp *comp, struct zcomp_strm *zstrm) +static void zcomp_strm_free(struct zcomp *comp, struct zcomp_strm *strm) { - comp->ops->destroy_ctx(&zstrm->ctx); - vfree(zstrm->buffer); - zstrm->buffer =3D NULL; + comp->ops->destroy_ctx(&strm->ctx); + vfree(strm->buffer); + kfree(strm); } =20 -static int zcomp_strm_init(struct zcomp *comp, struct zcomp_strm *zstrm) +static struct zcomp_strm *zcomp_strm_alloc(struct zcomp *comp) { + struct zcomp_strm *strm; int ret; =20 - ret =3D comp->ops->create_ctx(comp->params, &zstrm->ctx); - if (ret) - return ret; + strm =3D kzalloc(sizeof(*strm), GFP_KERNEL); + if (!strm) + return NULL; + + INIT_LIST_HEAD(&strm->entry); + + ret =3D comp->ops->create_ctx(comp->params, &strm->ctx); + if (ret) { + kfree(strm); + return NULL; + } =20 /* - * allocate 2 pages. 1 for compressed data, plus 1 extra for the - * case when compressed size is larger than the original one + * allocate 2 pages. 1 for compressed data, plus 1 extra in case if + * compressed data is larger than the original one. */ - zstrm->buffer =3D vzalloc(2 * PAGE_SIZE); - if (!zstrm->buffer) { - zcomp_strm_free(comp, zstrm); - return -ENOMEM; + strm->buffer =3D vzalloc(2 * PAGE_SIZE); + if (!strm->buffer) { + zcomp_strm_free(comp, strm); + return NULL; } - return 0; + return strm; } =20 static const struct zcomp_ops *lookup_backend_ops(const char *comp) @@ -109,13 +118,59 @@ ssize_t zcomp_available_show(const char *comp, char *= buf) =20 struct zcomp_strm *zcomp_stream_get(struct zcomp *comp) { - local_lock(&comp->stream->lock); - return this_cpu_ptr(comp->stream); + struct zcomp_strm *strm; + + might_sleep(); + + while (1) { + spin_lock(&comp->strm_lock); + if (!list_empty(&comp->idle_strm)) { + strm =3D list_first_entry(&comp->idle_strm, + struct zcomp_strm, + entry); + list_del(&strm->entry); + spin_unlock(&comp->strm_lock); + return strm; + } + + /* cannot allocate new stream, wait for an idle one */ + if (comp->avail_strm >=3D num_online_cpus()) { + spin_unlock(&comp->strm_lock); + wait_event(comp->strm_wait, + !list_empty(&comp->idle_strm)); + continue; + } + + /* allocate new stream */ + comp->avail_strm++; + spin_unlock(&comp->strm_lock); + + strm =3D zcomp_strm_alloc(comp); + if (strm) + break; + + spin_lock(&comp->strm_lock); + comp->avail_strm--; + spin_unlock(&comp->strm_lock); + wait_event(comp->strm_wait, !list_empty(&comp->idle_strm)); + } + + return strm; } =20 -void zcomp_stream_put(struct zcomp *comp) +void zcomp_stream_put(struct zcomp *comp, struct zcomp_strm *strm) { - local_unlock(&comp->stream->lock); + spin_lock(&comp->strm_lock); + if (comp->avail_strm <=3D num_online_cpus()) { + list_add(&strm->entry, &comp->idle_strm); + spin_unlock(&comp->strm_lock); + wake_up(&comp->strm_wait); + return; + } + + comp->avail_strm--; + spin_unlock(&comp->strm_lock); + zcomp_strm_free(comp, strm); } =20 int zcomp_compress(struct zcomp *comp, struct zcomp_strm *zstrm, @@ -148,61 +203,19 @@ int zcomp_decompress(struct zcomp *comp, struct zcomp= _strm *zstrm, return comp->ops->decompress(comp->params, &zstrm->ctx, &req); } =20 -int zcomp_cpu_up_prepare(unsigned int cpu, struct hlist_node *node) -{ - struct zcomp *comp =3D hlist_entry(node, struct zcomp, node); - struct zcomp_strm *zstrm; - int ret; - - zstrm =3D per_cpu_ptr(comp->stream, cpu); - local_lock_init(&zstrm->lock); - - ret =3D zcomp_strm_init(comp, zstrm); - if (ret) - pr_err("Can't allocate a compression stream\n"); - return ret; -} - -int zcomp_cpu_dead(unsigned int cpu, struct hlist_node *node) -{ - struct zcomp *comp =3D hlist_entry(node, struct zcomp, node); - struct zcomp_strm *zstrm; - - zstrm =3D per_cpu_ptr(comp->stream, cpu); - zcomp_strm_free(comp, zstrm); - return 0; -} - -static int zcomp_init(struct zcomp *comp, struct zcomp_params *params) -{ - int ret; - - comp->stream =3D alloc_percpu(struct zcomp_strm); - if (!comp->stream) - return -ENOMEM; - - comp->params =3D params; - ret =3D comp->ops->setup_params(comp->params); - if (ret) - goto cleanup; - - ret =3D cpuhp_state_add_instance(CPUHP_ZCOMP_PREPARE, &comp->node); - if (ret < 0) - goto cleanup; - - return 0; - -cleanup: - comp->ops->release_params(comp->params); - free_percpu(comp->stream); - return ret; -} - void zcomp_destroy(struct zcomp *comp) { - cpuhp_state_remove_instance(CPUHP_ZCOMP_PREPARE, &comp->node); + struct zcomp_strm *strm; + + while (!list_empty(&comp->idle_strm)) { + strm =3D list_first_entry(&comp->idle_strm, + struct zcomp_strm, + entry); + list_del(&strm->entry); + zcomp_strm_free(comp, strm); + } + comp->ops->release_params(comp->params); - free_percpu(comp->stream); kfree(comp); } =20 @@ -229,7 +242,12 @@ struct zcomp *zcomp_create(const char *alg, struct zco= mp_params *params) return ERR_PTR(-EINVAL); } =20 - error =3D zcomp_init(comp, params); + INIT_LIST_HEAD(&comp->idle_strm); + init_waitqueue_head(&comp->strm_wait); + spin_lock_init(&comp->strm_lock); + + comp->params =3D params; + error =3D comp->ops->setup_params(comp->params); if (error) { kfree(comp); return ERR_PTR(error); diff --git a/drivers/block/zram/zcomp.h b/drivers/block/zram/zcomp.h index ad5762813842..62330829db3f 100644 --- a/drivers/block/zram/zcomp.h +++ b/drivers/block/zram/zcomp.h @@ -3,10 +3,10 @@ #ifndef _ZCOMP_H_ #define _ZCOMP_H_ =20 -#include - #define ZCOMP_PARAM_NO_LEVEL INT_MIN =20 +#include + /* * Immutable driver (backend) parameters. The driver may attach private * data to it (e.g. driver representation of the dictionary, etc.). @@ -31,7 +31,7 @@ struct zcomp_ctx { }; =20 struct zcomp_strm { - local_lock_t lock; + struct list_head entry; /* compression buffer */ void *buffer; struct zcomp_ctx ctx; @@ -60,16 +60,15 @@ struct zcomp_ops { const char *name; }; =20 -/* dynamic per-device compression frontend */ struct zcomp { - struct zcomp_strm __percpu *stream; + struct list_head idle_strm; + spinlock_t strm_lock; + u32 avail_strm; + wait_queue_head_t strm_wait; const struct zcomp_ops *ops; struct zcomp_params *params; - struct hlist_node node; }; =20 -int zcomp_cpu_up_prepare(unsigned int cpu, struct hlist_node *node); -int zcomp_cpu_dead(unsigned int cpu, struct hlist_node *node); ssize_t zcomp_available_show(const char *comp, char *buf); bool zcomp_available_algorithm(const char *comp); =20 @@ -77,7 +76,7 @@ struct zcomp *zcomp_create(const char *alg, struct zcomp_= params *params); void zcomp_destroy(struct zcomp *comp); =20 struct zcomp_strm *zcomp_stream_get(struct zcomp *comp); -void zcomp_stream_put(struct zcomp *comp); +void zcomp_stream_put(struct zcomp *comp, struct zcomp_strm *strm); =20 int zcomp_compress(struct zcomp *comp, struct zcomp_strm *zstrm, const void *src, unsigned int *dst_len); diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index a8d22ae2a066..9ba3f8d97310 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -31,7 +31,6 @@ #include #include #include -#include #include #include =20 @@ -1603,7 +1602,7 @@ static int read_compressed_page(struct zram *zram, st= ruct page *page, u32 index) ret =3D zcomp_decompress(zram->comps[prio], zstrm, src, size, dst); kunmap_local(dst); zs_unmap_object(zram->mem_pool, handle); - zcomp_stream_put(zram->comps[prio]); + zcomp_stream_put(zram->comps[prio], zstrm); =20 return ret; } @@ -1764,14 +1763,14 @@ static int zram_write_page(struct zram *zram, struc= t page *page, u32 index) kunmap_local(mem); =20 if (unlikely(ret)) { - zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); + zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP], zstrm); pr_err("Compression failed! err=3D%d\n", ret); zs_free(zram->mem_pool, handle); return ret; } =20 if (comp_len >=3D huge_class_size) { - zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); + zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP], zstrm); return write_incompressible_page(zram, page, index); } =20 @@ -1795,7 +1794,7 @@ static int zram_write_page(struct zram *zram, struct = page *page, u32 index) __GFP_HIGHMEM | __GFP_MOVABLE); if (IS_ERR_VALUE(handle)) { - zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); + zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP], zstrm); atomic64_inc(&zram->stats.writestall); handle =3D zs_malloc(zram->mem_pool, comp_len, GFP_NOIO | __GFP_HIGHMEM | @@ -1807,7 +1806,7 @@ static int zram_write_page(struct zram *zram, struct = page *page, u32 index) } =20 if (!zram_can_store_page(zram)) { - zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); + zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP], zstrm); zs_free(zram->mem_pool, handle); return -ENOMEM; } @@ -1815,7 +1814,7 @@ static int zram_write_page(struct zram *zram, struct = page *page, u32 index) dst =3D zs_map_object(zram->mem_pool, handle, ZS_MM_WO); =20 memcpy(dst, zstrm->buffer, comp_len); - zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); + zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP], zstrm); zs_unmap_object(zram->mem_pool, handle); =20 zram_slot_write_lock(zram, index); @@ -1974,7 +1973,7 @@ static int recompress_slot(struct zram *zram, u32 ind= ex, struct page *page, kunmap_local(src); =20 if (ret) { - zcomp_stream_put(zram->comps[prio]); + zcomp_stream_put(zram->comps[prio], zstrm); return ret; } =20 @@ -1984,7 +1983,7 @@ static int recompress_slot(struct zram *zram, u32 ind= ex, struct page *page, /* Continue until we make progress */ if (class_index_new >=3D class_index_old || (threshold && comp_len_new >=3D threshold)) { - zcomp_stream_put(zram->comps[prio]); + zcomp_stream_put(zram->comps[prio], zstrm); continue; } =20 @@ -2042,13 +2041,13 @@ static int recompress_slot(struct zram *zram, u32 i= ndex, struct page *page, __GFP_HIGHMEM | __GFP_MOVABLE); if (IS_ERR_VALUE(handle_new)) { - zcomp_stream_put(zram->comps[prio]); + zcomp_stream_put(zram->comps[prio], zstrm); return PTR_ERR((void *)handle_new); } =20 dst =3D zs_map_object(zram->mem_pool, handle_new, ZS_MM_WO); memcpy(dst, zstrm->buffer, comp_len_new); - zcomp_stream_put(zram->comps[prio]); + zcomp_stream_put(zram->comps[prio], zstrm); =20 zs_unmap_object(zram->mem_pool, handle_new); =20 @@ -2796,7 +2795,6 @@ static void destroy_devices(void) zram_debugfs_destroy(); idr_destroy(&zram_index_idr); unregister_blkdev(zram_major, "zram"); - cpuhp_remove_multi_state(CPUHP_ZCOMP_PREPARE); } =20 static int __init zram_init(void) @@ -2806,15 +2804,9 @@ static int __init zram_init(void) =20 BUILD_BUG_ON(__NR_ZRAM_PAGEFLAGS > sizeof(zram_te.flags) * 8); =20 - ret =3D cpuhp_setup_state_multi(CPUHP_ZCOMP_PREPARE, "block/zram:prepare", - zcomp_cpu_up_prepare, zcomp_cpu_dead); - if (ret < 0) - return ret; - ret =3D class_register(&zram_control_class); if (ret) { pr_err("Unable to register zram-control class\n"); - cpuhp_remove_multi_state(CPUHP_ZCOMP_PREPARE); return ret; } =20 @@ -2823,7 +2815,6 @@ static int __init zram_init(void) if (zram_major <=3D 0) { pr_err("Unable to get major number\n"); class_unregister(&zram_control_class); - cpuhp_remove_multi_state(CPUHP_ZCOMP_PREPARE); return -EBUSY; } =20 diff --git a/include/linux/cpuhotplug.h b/include/linux/cpuhotplug.h index 6cc5e484547c..092ace7db8ee 100644 --- a/include/linux/cpuhotplug.h +++ b/include/linux/cpuhotplug.h @@ -119,7 +119,6 @@ enum cpuhp_state { CPUHP_MM_ZS_PREPARE, CPUHP_MM_ZSWP_POOL_PREPARE, CPUHP_KVM_PPC_BOOK3S_PREPARE, - CPUHP_ZCOMP_PREPARE, CPUHP_TIMERS_PREPARE, CPUHP_TMIGR_PREPARE, CPUHP_MIPS_SOC_PREPARE, --=20 2.48.1.262.g85cc9f2d1e-goog From nobody Fri Jan 31 03:24:27 2025 Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6C0CC1C1F0F for ; Thu, 30 Jan 2025 11:11:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738235489; cv=none; b=eQHnv+M636FCeKuDoXz9iSQJPDs52aQtxl8ibQAdQweRGY8lRBog/HXfcMNrce2eXkiW7+hy7l9Sb9Qn7n3XAGu2tZSnrtYRbxSODzquf1s7DkYmH5yXim1WRgFir3JmqowNIHFAHfMHtodtELKauXBcCbr+0nJ595ziM5d6s0k= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738235489; c=relaxed/simple; bh=tY358JOT6veKi7LsBzecpXyxdMh6+UMn99jF1ru1xDY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=MBxwhACuUMNZLkUtODLkL4qFnbrd+KnnmDucVS1iv4CgckAAbmUZZBFkvGMxtjTpPBKn1G9Uhwk386girfKK3rXGebwECvRvv/TGJRpM3irnbATrPa4HIRFg81b8kBG8bFpmDCz9tj0T2UhBOhu7HEKvXLZVlM9fzcAenGLyAxk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=aTr40ktQ; arc=none smtp.client-ip=209.85.214.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="aTr40ktQ" Received: by mail-pl1-f171.google.com with SMTP id d9443c01a7336-21669fd5c7cso9728925ad.3 for ; Thu, 30 Jan 2025 03:11:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1738235487; x=1738840287; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Ti4+lyxlKNsrQ+jB6ZeK4CLAAqx7hgqRlGDCYQ/kK7M=; b=aTr40ktQL4krnhSLVUO6w2Uxh63B6h4jWNTMMBtpbgKqAz5oDXQB1TbT5AvFUhjVk2 LR2VipcR/7YX+Ykhs9c/g9PxGXKem8mvQ0HxhQ03B5qjGSrzvakg9b44ixWuRG4lE2Or e2vBCsLi96c8n6qVcdkxTyzdXUsi/PShSND6o= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738235487; x=1738840287; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Ti4+lyxlKNsrQ+jB6ZeK4CLAAqx7hgqRlGDCYQ/kK7M=; b=rTFlNuyT4wHps9SVJTy0Ok05LvvKX1vvJoweA6/j0cYguQIqQb9ZcWRJHs3NH9LAto NDN1yylVC69enaaDNsSmegYQbaGYWnFByG9xsv2oUjDUaAfdicU9LXfj4E5m0In3KTvp 9cE896F/qScEBiF4O4cTVCsXXf8pnsn+piu/4Z8KtrYQnRMeUZdKxqupTK3wPY2qbJTk HhZZjlKO+fnxgCasrS5XjOMyxSMqUA+h4bu8kTyKG9skLkaEfrfmErBlEEprjpoeh3eK xTO6oNRTeI+7xiOwYO4A1g5nu9+0IBZt0iGCNg+aiIahVyW3bvdgG1XL6L9vOnzynpZ4 hntg== X-Forwarded-Encrypted: i=1; AJvYcCW5dlHId0R2jpFUqFHkvUL1pzGHu4OUlkIk2LSzwarLZwziQD3CfvO5q4kcwOD7uCJtdN8UcKFp37vS6Q4=@vger.kernel.org X-Gm-Message-State: AOJu0YxBpKSQ1oUo4D14dMbkgGD2SCBxZ9koIjsoA81fXlMK74UAAiDe gpuE57dJaSpbfDrun7kMjW15K2ImuTYHaE+tmehFNLdOLMn49BSSFIQWJU2+mQ== X-Gm-Gg: ASbGncsTbRNegXZIgGHAEfUWkx6egSVwAqPE3NWoACKt+PMtnHZGZfFJtnTe09x/I7g gLBg9m8IEzQaPCvap7L8XVMFC75ZBqjSJPl7wJ/I3L2saHBce6hIMwgdyL73UoT4vr1+uJwj3JG 0CsMlrLS88mc7NtoA1ETKgcEbYnug1hqgzczrWVZszE6uufmdkvkyr5ED6KYKVsGHK3Ia3T49x1 iMz+NIkA8nMlBRJ9ogsSm7wAPSBeHi7/1BtT4gDFdhpob8DgPIcXPVxktyducThCQ2vLmL+XRWh WHQufMVLovjFRGWe X-Google-Smtp-Source: AGHT+IHPjAb1rLSQ43OqGrbcZ/IzSuiJRWw/EpDyVdzrzHNxzjb1Y6QeLs4mCWjkG7KecIPQmqzeGg== X-Received: by 2002:a05:6a21:c94:b0:1e1:a671:7122 with SMTP id adf61e73a8af0-1ed7a5b663fmr11965895637.2.1738235487646; Thu, 30 Jan 2025 03:11:27 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:d4ce:e744:f46b:4fb]) by smtp.gmail.com with UTF8SMTPSA id d2e1a72fcca58-72fe631be3csm1169577b3a.7.2025.01.30.03.11.25 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 30 Jan 2025 03:11:27 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv3 03/11] zram: remove crypto include Date: Thu, 30 Jan 2025 20:10:48 +0900 Message-ID: <20250130111105.2861324-4-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.262.g85cc9f2d1e-goog In-Reply-To: <20250130111105.2861324-1-senozhatsky@chromium.org> References: <20250130111105.2861324-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Remove a leftover crypto header include. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zcomp.c | 1 - drivers/block/zram/zram_drv.c | 4 +++- drivers/block/zram/zram_drv.h | 1 - 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/drivers/block/zram/zcomp.c b/drivers/block/zram/zcomp.c index 982c769d5831..efd5919808d9 100644 --- a/drivers/block/zram/zcomp.c +++ b/drivers/block/zram/zcomp.c @@ -7,7 +7,6 @@ #include #include #include -#include #include =20 #include "zcomp.h" diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 9ba3f8d97310..31b653ad6d0e 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -43,6 +43,8 @@ static DEFINE_MUTEX(zram_index_mutex); static int zram_major; static const char *default_compressor =3D CONFIG_ZRAM_DEF_COMP; =20 +#define ZRAM_MAX_ALGO_NAME_SZ 64 + /* Module params (documentation at end) */ static unsigned int num_devices =3D 1; /* @@ -1141,7 +1143,7 @@ static int __comp_algorithm_store(struct zram *zram, = u32 prio, const char *buf) size_t sz; =20 sz =3D strlen(buf); - if (sz >=3D CRYPTO_MAX_ALG_NAME) + if (sz >=3D ZRAM_MAX_ALGO_NAME_SZ) return -E2BIG; =20 compressor =3D kstrdup(buf, GFP_KERNEL); diff --git a/drivers/block/zram/zram_drv.h b/drivers/block/zram/zram_drv.h index 3436ddf8ab23..045032acb22f 100644 --- a/drivers/block/zram/zram_drv.h +++ b/drivers/block/zram/zram_drv.h @@ -16,7 +16,6 @@ #define _ZRAM_DRV_H_ =20 #include -#include =20 #include "zcomp.h" =20 --=20 2.48.1.262.g85cc9f2d1e-goog From nobody Fri Jan 31 03:24:27 2025 Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6230B1C1F0F for ; Thu, 30 Jan 2025 11:11:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738235495; cv=none; b=KLq1fmez2QBYAY8P9VSfPvfDhj12gvHW047JxOlg+LFZ5v+6r+RtzJTdvOyRw1RY+CHjnm81yp1WX/jBc5O4glO9bAS78raysZhw0MEqIEKkAqLjxPeiwzJUmd9v6GQ3uwmAReK1+fJneLUaUyLxAFAXggUOqQdzJS8QP/3BizU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738235495; c=relaxed/simple; bh=d/TyFBuIGsVcz9XqaXtscP8eB4S/GZoaiYxC9PgMSIE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=oM61mrDmGbX+vZENzZaKrVVTsAqW00cosK8CE/ILP8s6XvQfL27QgR14my9fnLjJlRxbp9AVeRpAvhYkowNikT6HMRj3mzrnryVwK88xv45D+B1lWHLsyfQKpjyLYXZUZNuoqk1Y89ca8oAOteUmSvkc0h4jvWAEwphWpDmdJ2g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=fnbX7KhE; arc=none smtp.client-ip=209.85.214.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="fnbX7KhE" Received: by mail-pl1-f182.google.com with SMTP id d9443c01a7336-2166651f752so15132515ad.3 for ; Thu, 30 Jan 2025 03:11:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1738235492; x=1738840292; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Q/yW7cYsxX6u2D6RjpsktMUNNjtmh2H6QDs8XtL+Hec=; b=fnbX7KhEzlQZ+uw3gETanTxuX+LEn64BuXwI1Z3IIO4EoAb1CIZgHj7u0Vrdq6gq39 RBpMkbj5weIGqaHxG0FhAO+PCvF4fbrad1wt/2rKiK4U7x1mbZ53v8hV8S+qv49f2ggY 2248lYvyme0jhlag+RDgxFUHOfu5y62nK86oE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738235492; x=1738840292; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Q/yW7cYsxX6u2D6RjpsktMUNNjtmh2H6QDs8XtL+Hec=; b=iFmtfytOxus1A5E3k+a1U/x0M7IWqCTk6x0d/uZBbJebJFBnAYbx2GIxyXyNoGIyKg giFw3W+ZqrwpGbDSMYpu/c91NnKWYboxEsHhdb75kHFSXLNT8lNm4Fsqct3JpHPfpMFT YEPgPw4B/2Uf6Sf+LRwJLLpjJSoMYZPDrO56hbVUtarFvu6BdOBoEM+38zbx/lYZfkGB QfGtLVYTcicGQ1c2xe3yPhXIWU7UvohbQkbuSVYgnwEP/kzzBl2Gb/7NaPw1pYaUHRUT mxunqoWtj8dx9xwpjTDATkhpGRP8AVWzWQyZVtYfjvr+PQtIgtt1I8J9kMezMZHr2E1u 9zow== X-Forwarded-Encrypted: i=1; AJvYcCXSvpU58ShzLHcnW/rczE0F09CKe9WXtFOMR5Ql+/pAbtYXKLnVYekgPlTSA4hXDQgLCyKC0lyhpeGmhnM=@vger.kernel.org X-Gm-Message-State: AOJu0YwXSbl+RoYjECW/y3x3NDcTc0Bp94TcEDzH1uGOSpGCNG01ygO5 gVzIsH9I8HHfF10Ta0UEDM9Ulii6mHMpFr7FYSA/ou4+RRrDRDV6p7+Tz28wEu2a6ZWZgENVhVA = X-Gm-Gg: ASbGncuT+XtptxdIc7fLVl5lWFoX4CzcJCTKIqfDn4hE0N+GoigXYi0J74UTqvu3Jg0 IHqPc+8Ewe47Twp+iV47h8PwUGICGdJQtj08hRY1esapRxw0JZEbbIyeBPB7jV0e0OXFDm7Jd/1 UjT5eZKXrVm5GyRcnGOMdEadMgsiLdeDg5fi4JW7amOZqcpAlwnzCQRKWNCujVfOFx0XYSF2ZNq c93QWhSBfcdYtF2TFZDXKNIAM5IiS3N/0x6voFIK9Yg7ItkDkEeBaMz7cE/hVpc0fVlYeyaoMdr 6Fm/cmBpSi/YJbXt X-Google-Smtp-Source: AGHT+IHRhQXy77fDs2DUj5z16q9LiSjBM8lSRxMlB0gLWnyP7Ox6pdNXNxWqKncG1n2YwlehavUUvA== X-Received: by 2002:a17:902:d485:b0:216:4853:4c0b with SMTP id d9443c01a7336-21dd7deef0bmr99162575ad.33.1738235492597; Thu, 30 Jan 2025 03:11:32 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:d4ce:e744:f46b:4fb]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-21de3302e63sm11331925ad.172.2025.01.30.03.11.30 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 30 Jan 2025 03:11:32 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv3 04/11] zram: remove max_comp_streams device attr Date: Thu, 30 Jan 2025 20:10:49 +0900 Message-ID: <20250130111105.2861324-5-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.262.g85cc9f2d1e-goog In-Reply-To: <20250130111105.2861324-1-senozhatsky@chromium.org> References: <20250130111105.2861324-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" max_comp_streams device attribute has been defunct since May 2016 when zram switched to per-CPU compression streams, remove it. Signed-off-by: Sergey Senozhatsky --- Documentation/ABI/testing/sysfs-block-zram | 8 ----- Documentation/admin-guide/blockdev/zram.rst | 36 ++++++--------------- drivers/block/zram/zram_drv.c | 23 ------------- 3 files changed, 10 insertions(+), 57 deletions(-) diff --git a/Documentation/ABI/testing/sysfs-block-zram b/Documentation/ABI= /testing/sysfs-block-zram index 1ef69e0271f9..36c57de0a10a 100644 --- a/Documentation/ABI/testing/sysfs-block-zram +++ b/Documentation/ABI/testing/sysfs-block-zram @@ -22,14 +22,6 @@ Description: device. The reset operation frees all the memory associated with this device. =20 -What: /sys/block/zram/max_comp_streams -Date: February 2014 -Contact: Sergey Senozhatsky -Description: - The max_comp_streams file is read-write and specifies the - number of backend's zcomp_strm compression streams (number of - concurrent compress operations). - What: /sys/block/zram/comp_algorithm Date: February 2014 Contact: Sergey Senozhatsky diff --git a/Documentation/admin-guide/blockdev/zram.rst b/Documentation/ad= min-guide/blockdev/zram.rst index 1576fb93f06c..9bdb30901a93 100644 --- a/Documentation/admin-guide/blockdev/zram.rst +++ b/Documentation/admin-guide/blockdev/zram.rst @@ -54,7 +54,7 @@ The list of possible return codes: If you use 'echo', the returned value is set by the 'echo' utility, and, in general case, something like:: =20 - echo 3 > /sys/block/zram0/max_comp_streams + echo foo > /sys/block/zram0/comp_algorithm if [ $? -ne 0 ]; then handle_error fi @@ -73,21 +73,7 @@ This creates 4 devices: /dev/zram{0,1,2,3} num_devices parameter is optional and tells zram how many devices should be pre-created. Default: 1. =20 -2) Set max number of compression streams -=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D - -Regardless of the value passed to this attribute, ZRAM will always -allocate multiple compression streams - one per online CPU - thus -allowing several concurrent compression operations. The number of -allocated compression streams goes down when some of the CPUs -become offline. There is no single-compression-stream mode anymore, -unless you are running a UP system or have only 1 CPU online. - -To find out how many streams are currently available:: - - cat /sys/block/zram0/max_comp_streams - -3) Select compression algorithm +2) Select compression algorithm =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D =20 Using comp_algorithm device attribute one can see available and @@ -107,7 +93,7 @@ Examples:: For the time being, the `comp_algorithm` content shows only compression algorithms that are supported by zram. =20 -4) Set compression algorithm parameters: Optional +3) Set compression algorithm parameters: Optional =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =20 Compression algorithms may support specific parameters which can be @@ -138,7 +124,7 @@ better the compression ratio, it even can take negative= s values for some algorithms), for other algorithms `level` is acceleration level (the higher the value the lower the compression ratio). =20 -5) Set Disksize +4) Set Disksize =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =20 Set disk size by writing the value to sysfs node 'disksize'. @@ -158,7 +144,7 @@ There is little point creating a zram of greater than t= wice the size of memory since we expect a 2:1 compression ratio. Note that zram uses about 0.1% of= the size of the disk when not in use so a huge zram is wasteful. =20 -6) Set memory limit: Optional +5) Set memory limit: Optional =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D =20 Set memory limit by writing the value to sysfs node 'mem_limit'. @@ -177,7 +163,7 @@ Examples:: # To disable memory limit echo 0 > /sys/block/zram0/mem_limit =20 -7) Activate +6) Activate =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =20 :: @@ -188,7 +174,7 @@ Examples:: mkfs.ext4 /dev/zram1 mount /dev/zram1 /tmp =20 -8) Add/remove zram devices +7) Add/remove zram devices =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D =20 zram provides a control interface, which enables dynamic (on-demand) device @@ -208,7 +194,7 @@ execute:: =20 echo X > /sys/class/zram-control/hot_remove =20 -9) Stats +8) Stats =3D=3D=3D=3D=3D=3D=3D=3D =20 Per-device statistics are exported as various nodes under /sys/block/zram<= id>/ @@ -228,8 +214,6 @@ mem_limit WO specifies the maximum amount of m= emory ZRAM can writeback_limit WO specifies the maximum amount of write IO zram can write out to backing device as 4KB unit writeback_limit_enable RW show and set writeback_limit feature -max_comp_streams RW the number of possible concurrent compress - operations comp_algorithm RW show and change the compression algorithm algorithm_params WO setup compression algorithm parameters compact WO trigger memory compaction @@ -310,7 +294,7 @@ a single line of text and contains the following stats = separated by whitespace: Unit: 4K bytes =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D =20 -10) Deactivate +9) Deactivate =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =20 :: @@ -318,7 +302,7 @@ a single line of text and contains the following stats = separated by whitespace: swapoff /dev/zram0 umount /dev/zram1 =20 -11) Reset +10) Reset =3D=3D=3D=3D=3D=3D=3D=3D=3D =20 Write any positive value to 'reset' sysfs node:: diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 31b653ad6d0e..0c9b68fa8783 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1096,27 +1096,6 @@ static void zram_debugfs_register(struct zram *zram)= {}; static void zram_debugfs_unregister(struct zram *zram) {}; #endif =20 -/* - * We switched to per-cpu streams and this attr is not needed anymore. - * However, we will keep it around for some time, because: - * a) we may revert per-cpu streams in the future - * b) it's visible to user space and we need to follow our 2 years - * retirement rule; but we already have a number of 'soon to be - * altered' attrs, so max_comp_streams need to wait for the next - * layoff cycle. - */ -static ssize_t max_comp_streams_show(struct device *dev, - struct device_attribute *attr, char *buf) -{ - return scnprintf(buf, PAGE_SIZE, "%d\n", num_online_cpus()); -} - -static ssize_t max_comp_streams_store(struct device *dev, - struct device_attribute *attr, const char *buf, size_t len) -{ - return len; -} - static void comp_algorithm_set(struct zram *zram, u32 prio, const char *al= g) { /* Do not free statically defined compression algorithms */ @@ -2535,7 +2514,6 @@ static DEVICE_ATTR_WO(reset); static DEVICE_ATTR_WO(mem_limit); static DEVICE_ATTR_WO(mem_used_max); static DEVICE_ATTR_WO(idle); -static DEVICE_ATTR_RW(max_comp_streams); static DEVICE_ATTR_RW(comp_algorithm); #ifdef CONFIG_ZRAM_WRITEBACK static DEVICE_ATTR_RW(backing_dev); @@ -2557,7 +2535,6 @@ static struct attribute *zram_disk_attrs[] =3D { &dev_attr_mem_limit.attr, &dev_attr_mem_used_max.attr, &dev_attr_idle.attr, - &dev_attr_max_comp_streams.attr, &dev_attr_comp_algorithm.attr, #ifdef CONFIG_ZRAM_WRITEBACK &dev_attr_backing_dev.attr, --=20 2.48.1.262.g85cc9f2d1e-goog From nobody Fri Jan 31 03:24:27 2025 Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 596A81C1F1D for ; Thu, 30 Jan 2025 11:11:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.180 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738235499; cv=none; b=tkBCNM0atkJyaOXPC+rzp/Ps04YYWsDPf8F4jkT9dpoZK0FQx9T/WGaP9aymhyB3otsg28m1NsvJ3sAf0NuwEJDLxlos+fMAq//ohb6fyek2vwd3j9EqRtYit1a84VM01x7tkHtYFRutLuIRggLO7KZqqa7mSt33M1/Or4asUS0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738235499; c=relaxed/simple; bh=NrPXlIDNWrXWtLZ0f9UduK7J6HixgLH7eyRTETmv8ik=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=r57ZvQqY7fZUXGCAS4msYVpTv3MMkzt/GQpA5KkJyHMZ4/9IzN7WKnDptbZ2OISWL9zeD/6UeYXUeTvW3pyYQ1azZvTwKjWAQLUgIlaq/ozZ0th+IkTzxrDD68G1v4J6QpRHbqVdh21fQ65XvBZHCyGIVqTPl4AQoi2gDWnu3KU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=dWcR1wOw; arc=none smtp.client-ip=209.85.214.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="dWcR1wOw" Received: by mail-pl1-f180.google.com with SMTP id d9443c01a7336-2167141dfa1so9626135ad.1 for ; Thu, 30 Jan 2025 03:11:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1738235497; x=1738840297; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ujanKWetHL+pQynH0HIAEO2xKM/XaqicaLx2SNCooQM=; b=dWcR1wOwcJ6yFGW6/N9GgqbyoN067b4Rh5gdC/oygDviHzcFGsJ3l+81SGWOTxO8QB vk1IObGFq3roZgQvR2am5GI3YVGvhVc62Qweun2o36NnCWvKGouQMydfnPEna0KExmnA yEjECJGYKIJ3QyjQvDdDu10jSXyWF0oO/WPHw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738235497; x=1738840297; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ujanKWetHL+pQynH0HIAEO2xKM/XaqicaLx2SNCooQM=; b=mcIB1AAcZA8ldM5c8J35Rk5NHqbz5/lnp2/ARYBjYcV9GhHI/+aNtZ86fszBEvpu9L WkBlT8kKFcZdAOC8t3kMCRsphAY1HW/8naPFYIOyy25ONPzfG8oDUF/QqLNC7b2GskEB KDO8Y9jl1cS3GrZHSNg29yQM4FrM+sQRjNMrsFOQvRDrXnz14z1y5mm4qDJ9Ta82BGmW IXhB7lQZWtNKiD4cyoDHu3l4fxKutMf210NB4fUpx1jRBR6yDBj/nlXQ2hljF0DrKMO7 LhMwSgyEEd9l2Jk4GnYbOe+zqSIpeNSbF/ubwu4pFyMBM6vOkNam2TbGk8FkF5b4dUyx cXqQ== X-Forwarded-Encrypted: i=1; AJvYcCVf+S4GiS332WvPrQJYy6/6RWbd3stepIqkM7qMODycWG6bBNR+ouvy8jXKxZIaw4ENqfCpDdvTHCYw29I=@vger.kernel.org X-Gm-Message-State: AOJu0Ywbeioe6VIWiKBAE02tkRRksvcyJ3dVq9dWDp5nxQpsvkITz3Vw 7LsFJCaZQWP8hRH00/oFsEa2Rtiaoc9SIeawtxO91phi77UOm8H5+ZIuh3pyUqCr+pC4nzKxHKA = X-Gm-Gg: ASbGncvYhKDa1TX0MEXYRCz9bXFd0DQNIXvoS0VdFs654ub1xBxcdcose7Ou4siUFaK 8pHD8Bf5umECBDg+UzSGXKtCD6e2Oclqp3vPhdeyTuWUxZB+UWe/4lAfSlqiR7PysrTt0+EnUix 8i67fUygytixwwWUlYwWzjT5UiQUNrKqoJoovKWxXgnW2W3lIpTzeUf998FKV845lH605y+8Sxb aFAIlyr+mCJC/ZBxrbKRWbku3v5e4mj+b4VO//OzV78Z6IoKzvAVzowmgcw64It11jjhI/5Ld3H w+vyalyosW2QD9eQ X-Google-Smtp-Source: AGHT+IGoJPBvbRFKgcHes3aZpqhjq0bpB3vf3UMoxOpT1s+vDvy+5JFl5BGgTN400Fd1/JslRst1fw== X-Received: by 2002:a17:902:c946:b0:216:14fb:d277 with SMTP id d9443c01a7336-21de196fb0amr42131665ad.22.1738235497536; Thu, 30 Jan 2025 03:11:37 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:d4ce:e744:f46b:4fb]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-21de31f836csm11410515ad.93.2025.01.30.03.11.35 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 30 Jan 2025 03:11:37 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv3 05/11] zram: remove two-staged handle allocation Date: Thu, 30 Jan 2025 20:10:50 +0900 Message-ID: <20250130111105.2861324-6-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.262.g85cc9f2d1e-goog In-Reply-To: <20250130111105.2861324-1-senozhatsky@chromium.org> References: <20250130111105.2861324-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Previously zram write() was atomic which required us to pass __GFP_KSWAPD_RECLAIM to zsmalloc handle allocation on a fast path and attempt a slow path allocation (with recompression) when the fast path failed. Since it's not atomic anymore we can permit direct reclaim during allocation, and remove fast allocation path and, also, drop the recompression path (which should reduce CPU/battery usage). Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 41 +++++------------------------------ 1 file changed, 6 insertions(+), 35 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 0c9b68fa8783..28e5b02dc3a1 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1718,11 +1718,11 @@ static int write_incompressible_page(struct zram *z= ram, struct page *page, static int zram_write_page(struct zram *zram, struct page *page, u32 index) { int ret =3D 0; - unsigned long handle =3D -ENOMEM; - unsigned int comp_len =3D 0; + unsigned long handle; + unsigned int comp_len; void *dst, *mem; struct zcomp_strm *zstrm; - unsigned long element =3D 0; + unsigned long element; bool same_filled; =20 /* First, free memory allocated to this slot (if any) */ @@ -1736,7 +1736,6 @@ static int zram_write_page(struct zram *zram, struct = page *page, u32 index) if (same_filled) return write_same_filled_page(zram, element, index); =20 -compress_again: zstrm =3D zcomp_stream_get(zram->comps[ZRAM_PRIMARY_COMP]); mem =3D kmap_local_page(page); ret =3D zcomp_compress(zram->comps[ZRAM_PRIMARY_COMP], zstrm, @@ -1745,8 +1744,6 @@ static int zram_write_page(struct zram *zram, struct = page *page, u32 index) =20 if (unlikely(ret)) { zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP], zstrm); - pr_err("Compression failed! err=3D%d\n", ret); - zs_free(zram->mem_pool, handle); return ret; } =20 @@ -1755,36 +1752,10 @@ static int zram_write_page(struct zram *zram, struc= t page *page, u32 index) return write_incompressible_page(zram, page, index); } =20 - /* - * handle allocation has 2 paths: - * a) fast path is executed with preemption disabled (for - * per-cpu streams) and has __GFP_DIRECT_RECLAIM bit clear, - * since we can't sleep; - * b) slow path enables preemption and attempts to allocate - * the page with __GFP_DIRECT_RECLAIM bit set. we have to - * put per-cpu compression stream and, thus, to re-do - * the compression once handle is allocated. - * - * if we have a 'non-null' handle here then we are coming - * from the slow path and handle has already been allocated. - */ + handle =3D zs_malloc(zram->mem_pool, comp_len, + GFP_NOIO | __GFP_HIGHMEM | __GFP_MOVABLE); if (IS_ERR_VALUE(handle)) - handle =3D zs_malloc(zram->mem_pool, comp_len, - __GFP_KSWAPD_RECLAIM | - __GFP_NOWARN | - __GFP_HIGHMEM | - __GFP_MOVABLE); - if (IS_ERR_VALUE(handle)) { - zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP], zstrm); - atomic64_inc(&zram->stats.writestall); - handle =3D zs_malloc(zram->mem_pool, comp_len, - GFP_NOIO | __GFP_HIGHMEM | - __GFP_MOVABLE); - if (IS_ERR_VALUE(handle)) - return PTR_ERR((void *)handle); - - goto compress_again; - } + return PTR_ERR((void *)handle); =20 if (!zram_can_store_page(zram)) { zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP], zstrm); --=20 2.48.1.262.g85cc9f2d1e-goog From nobody Fri Jan 31 03:24:27 2025 Received: from mail-pj1-f54.google.com (mail-pj1-f54.google.com [209.85.216.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 17A0C1E0DDC for ; Thu, 30 Jan 2025 11:11:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.54 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738235504; cv=none; b=QSNERTs2+rHVquSLD2D8786H1BMBDaRahgYdRZYqC+jdCupKKDM/CvauIeBAOVu8zIRqDRbGUBW17ZcJv2nByxH7NWqWT1ZDGZMiixIMIWc0bglCUUAlqJzFSwIvs0Gfb41bpb+eeRmqb1lPbu/6AgULvIO17w4heYaq5FSE1JA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738235504; c=relaxed/simple; bh=Ej/13Gy59hq4pSeuXyYtdu8JOFvxFaLu/EaPO8MCcTs=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=D7dda+0v8PjdjQj+bMsli9L/niBMTWf4vFXecyQ6EVGOHIXKp6OEKMmQh+D0yTX5Cnr06tFTRMXHThuECIxV6Jvv5V8MwcNwHBhPujjIwtApG/YzbTQ8EL/kbp7JZDOwwSYR1OqT0RMnq+9tBxCHnmvoYhr1r8TU7nmzQMNWruA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=b3dvqlFu; arc=none smtp.client-ip=209.85.216.54 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="b3dvqlFu" Received: by mail-pj1-f54.google.com with SMTP id 98e67ed59e1d1-2efd81c7ca4so744391a91.2 for ; Thu, 30 Jan 2025 03:11:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1738235502; x=1738840302; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=5g614UtqxTvOkBYeQdwAP9Uo/j0Alc44lQaUMvbAkkk=; b=b3dvqlFunxfbNMT/FwIBwAptXBp9qLgF4DjTYeDLl2ckmrgxFgIHO9BMWTDF9G4DAU Emj4dc/QwanfssbgT/64F7RqGsU1BNjhLJQlGoQ8k68in8D6PUvIwGS3Mmjh0C5uYSU7 JsHjkxzeqYT8h29PSVmjd1opBEAsPfA+UaN0o= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738235502; x=1738840302; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=5g614UtqxTvOkBYeQdwAP9Uo/j0Alc44lQaUMvbAkkk=; b=gpGBy79LQB06FSR5sysuLXXCv8nFzZsl2s1XmKxORoRAI04gqw+C/iGowzi2cZSJY3 lswOQ2qlkW+FRp05w1zVhEb9G/a+eMCNvdUUVn/3QEcweTB4Kn6czuiF7UZRleDgnVdy a62sJHHY25eG4RW+p+8HRpPNUwFAfFRAjIhpf/jhRArNfxqxkEgri6rjPcIzbD6mLstz VBUxo8i+BWjm7EaMgYkMjwYUIHbfjW6smL+3MY6oHcX30qnIbker4XufyGwT/pBL/cgW TmIOBDCEVzVhxggrPaBvIVy2FGKQnue5xafe8yzG8Ag/vV9fOtR7BJ0yuA2WDWx5RAwl 8bDw== X-Forwarded-Encrypted: i=1; AJvYcCVfqljqMUccADmVrEIL5Elgq2euVVfJnJopEEQT5zSBlWU4Sb3MxEHyKEwZasADqixCACa3QCPBNCSqlho=@vger.kernel.org X-Gm-Message-State: AOJu0YxEAqcbM5C5tzhhGDF2l4vrbKw0NO2tcyHn5bbNbg3yY0D8HyU7 LXsUksUGZ2hBmcbLMvs39KqCZAtbGbUp8PnE50oW9Bz+1SBAhqihw9U/ykO8Lshpw8nGmh1puwk = X-Gm-Gg: ASbGncvqxvcunEZpczf8Qv1yC3A7+r4knZ47wMshIbGHnNDlylOy8V3PN1myKLdP8ud s1s3ZFbXF3O/+rQQ46LjQeZIyXn38g38txxzmUVkwY9oB3Kw9Gc6AQomhGhDq8al15Gj9ClcGm7 c8aLGz4nCS11kxqzkxuDJCLpJgtV66UM7rv59awsdlPymmSQU6EeGJBcXjD+ULCr6gPloel0MXT 0Qn8H+OfD8iwvmmS4qVxvx6pDRGX8v4l/MIhiRCTctS90AnwJt0hvJ4AZpdO4QHi9HS5dZRg2yw wDk0MlgIEyG1ShMH X-Google-Smtp-Source: AGHT+IHg3heFJ21F+Ok092BsmeTSf/u5V4ef9+jPuKs0cnRlT5yUCtdKYPEIddDVnItYsVqEn0xRFg== X-Received: by 2002:a17:90b:4d06:b0:2f4:49d8:e6f6 with SMTP id 98e67ed59e1d1-2f83aba9d34mr9404404a91.3.1738235502347; Thu, 30 Jan 2025 03:11:42 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:d4ce:e744:f46b:4fb]) by smtp.gmail.com with UTF8SMTPSA id 98e67ed59e1d1-2f848acbda4sm1333892a91.45.2025.01.30.03.11.40 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 30 Jan 2025 03:11:42 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv3 06/11] zram: permit reclaim in zstd custom allocator Date: Thu, 30 Jan 2025 20:10:51 +0900 Message-ID: <20250130111105.2861324-7-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.262.g85cc9f2d1e-goog In-Reply-To: <20250130111105.2861324-1-senozhatsky@chromium.org> References: <20250130111105.2861324-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" When configured with pre-trained compression/decompression dictionary support, zstd requires custom memory allocator, which it calls internally from compression()/decompression() routines. This was a tad problematic, because that would mean allocation from atomic context (either under entry spin-lock, or per-CPU local-lock or both). Now, with non-atomic zram write(), those limitations are relaxed and we can allow direct and indirect reclaim during allocations. The tricky part is zram read() path, which is still atomic in one particular case (read_compressed_page()), due to zsmalloc handling of object mapping. However, in zram in order to read() something one has to write() it first, and write() is when zstd allocates required internal state memory, and write() path is non-atomic. Because of this write() allocation, in theory, zstd should not call its allocator from the atomic read() path. Keep the non-preemptible branch, just in case if zstd allocates memory from read(), but WARN_ON_ONCE() if it happens. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/backend_zstd.c | 11 +++-------- 1 file changed, 3 insertions(+), 8 deletions(-) diff --git a/drivers/block/zram/backend_zstd.c b/drivers/block/zram/backend= _zstd.c index 1184c0036f44..53431251ea62 100644 --- a/drivers/block/zram/backend_zstd.c +++ b/drivers/block/zram/backend_zstd.c @@ -24,19 +24,14 @@ struct zstd_params { /* * For C/D dictionaries we need to provide zstd with zstd_custom_mem, * which zstd uses internally to allocate/free memory when needed. - * - * This means that allocator.customAlloc() can be called from zcomp_compre= ss() - * under local-lock (per-CPU compression stream), in which case we must use - * GFP_ATOMIC. - * - * Another complication here is that we can be configured as a swap device. */ static void *zstd_custom_alloc(void *opaque, size_t size) { - if (!preemptible()) + /* Technically this should not happen */ + if (WARN_ON_ONCE(!preemptible())) return kvzalloc(size, GFP_ATOMIC); =20 - return kvzalloc(size, __GFP_KSWAPD_RECLAIM | __GFP_NOWARN); + return kvzalloc(size, GFP_NOIO | __GFP_NOWARN); } =20 static void zstd_custom_free(void *opaque, void *address) --=20 2.48.1.262.g85cc9f2d1e-goog From nobody Fri Jan 31 03:24:27 2025 Received: from mail-pl1-f173.google.com (mail-pl1-f173.google.com [209.85.214.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D394C1E3772 for ; Thu, 30 Jan 2025 11:11:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738235509; cv=none; b=h4vEEIw5ZUe+w8UfZfDl1FrZbF5gtHEYycDQceHpAAmNC9oY9ayz5B3/dNwQAbJg9SkEVg8bHxnO4gIhLmVBNwk9G/3xt7772Nys0s0XNqS192a5TzzDK5G8gneMWnd4vkJxt2odEswGi14qJmYzfEmYMhOtStUJ3tJNnyWH6wg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738235509; c=relaxed/simple; bh=P9VdcunwmUSUgdStR+4cpgVmYLBALxuBPRS0y11jyMI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=OJTWbwRWpsEdO4zp4An3HhwUB6bAqYB+8RE9HK+S7olz/EN/d/pBsfXjl/HgS7fAq3FWv3SZ/QiAYsjsVpir2azM6iPluV+vz/wLzxfdKCxzv3UKUTuRzeSKCM+VL4HphWogOZzkRCYIyJMjdLRNTscpGAJTAjkQ5vNNEeit+uA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=TSSIe89c; arc=none smtp.client-ip=209.85.214.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="TSSIe89c" Received: by mail-pl1-f173.google.com with SMTP id d9443c01a7336-2163dc5155fso10162345ad.0 for ; Thu, 30 Jan 2025 03:11:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1738235507; x=1738840307; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=wsNyTVEtqJeOeaUMXY3SZH4okPEwKCuL4AyQgcINZOs=; b=TSSIe89c4QwixqpaTSH/2pEPJlycRQ9Ock260B+HGj83kmY377MyW7BIYXrvnCun/T Ip5Yzbo0ImzFWl/r1CFWVD+BKOSOPNrQu1viGELE9bjxHkI/Xrw/tVYA0p+zvMd5pFFY 1/YMM/9lTTpEIgiitoz3tiJavoaODnh4vFD9E= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738235507; x=1738840307; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=wsNyTVEtqJeOeaUMXY3SZH4okPEwKCuL4AyQgcINZOs=; b=oNvMp2Tv9SKxr0NiBpMVQnDG0pCe7+nkPaL2lF0IET0/Jb8B7eXK3JGMaem52xJeqT f51j6fTqbXkzlz6NVxXIcKl4gu9x8eUuRRNKKvhcgxtjOtA+rxYwjeGackC17YMQ3A3c LGufln5y0ogYEyDh0ObYgB0GtyBekp2X+o0yV7saYmlM4O5dmHIe7CIAEJR4N3rY8UZU i1DmvQv+VZy6GULf1EtBleS2jc+d3fUk5VYXQGCUQGS5GZ+y0J9GIDRsuIfTz8eya7lB agaNVDUqVTGITgF5Okez3te6xfemHQT+bCBsy9lrbL+C4jwK1Cn0fG2TymoaBHoCy4Y8 1oOQ== X-Forwarded-Encrypted: i=1; AJvYcCWUkl+raZ5NMBtG/A+B1C6r9tIbJzePy2y3nTgysVw2Wpwa1Adb2DfesVljyshBhGpqj/kFL265qJxr6mg=@vger.kernel.org X-Gm-Message-State: AOJu0YwBFIS4exm1lWJdRh3fg4PLS3XMNzfX7oyYcBiyV3jRfzr/isbz oYfhADDxr8p/EFOESWfGfEkow4F0H3V23yelwVDlt3C/4a+rqL5BGfqLHv8eVg== X-Gm-Gg: ASbGncv64oOjD6jYjd2GguwVO5hZk9TqjebfrS2uGBGIXcgoFb3DQZj4ZOhdBifdAtW Aw6CRQ6fe7pmtMe+XsT7YmxEmSsSLKMBdXLBFkk0OJZViEAcbvpzCfVLEoORbO07cIt7yu0JZA6 p3yS4Oczs+tCRHquKJWkDn4+4fAML1Ld/zOFaMCoyhtHhrdyJzUDyAhtHJ/mSXhmCLI9r+mwD57 EyQkEH6JX+pipfuClABsl8zX783Mi9p3QkUc9N1pnxG7icH+njfrH82nyMSz0Zgkj9xxLGnoTkI nFQ1OH0xr5ZF0RC9 X-Google-Smtp-Source: AGHT+IGkwzFoDqvhfsm5VPosd4bbEzVRTQxGnDnsn5b9Avd6d9LNbqhCO+f9O1miDGFRqxuRGkD6PA== X-Received: by 2002:a05:6a00:2448:b0:729:a31:892d with SMTP id d2e1a72fcca58-72fd0be5470mr11033309b3a.8.1738235507138; Thu, 30 Jan 2025 03:11:47 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:d4ce:e744:f46b:4fb]) by smtp.gmail.com with UTF8SMTPSA id d2e1a72fcca58-72fe653fe6fsm1164789b3a.80.2025.01.30.03.11.45 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 30 Jan 2025 03:11:46 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv3 07/11] zram: permit reclaim in recompression handle allocation Date: Thu, 30 Jan 2025 20:10:52 +0900 Message-ID: <20250130111105.2861324-8-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.262.g85cc9f2d1e-goog In-Reply-To: <20250130111105.2861324-1-senozhatsky@chromium.org> References: <20250130111105.2861324-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Recompression path can now permit direct reclaim during new zs_handle allocation, because it's not atomic anymore. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 12 +++--------- 1 file changed, 3 insertions(+), 9 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 28e5b02dc3a1..ac8b4f47a496 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1981,17 +1981,11 @@ static int recompress_slot(struct zram *zram, u32 i= ndex, struct page *page, return 0; =20 /* - * No direct reclaim (slow path) for handle allocation and no - * re-compression attempt (unlike in zram_write_bvec()) since - * we already have stored that object in zsmalloc. If we cannot - * alloc memory for recompressed object then we bail out and - * simply keep the old (existing) object in zsmalloc. + * If we cannot alloc memory for recompressed object then we bail out + * and simply keep the old (existing) object in zsmalloc. */ handle_new =3D zs_malloc(zram->mem_pool, comp_len_new, - __GFP_KSWAPD_RECLAIM | - __GFP_NOWARN | - __GFP_HIGHMEM | - __GFP_MOVABLE); + GFP_NOIO | __GFP_HIGHMEM | __GFP_MOVABLE); if (IS_ERR_VALUE(handle_new)) { zcomp_stream_put(zram->comps[prio], zstrm); return PTR_ERR((void *)handle_new); --=20 2.48.1.262.g85cc9f2d1e-goog From nobody Fri Jan 31 03:24:27 2025 Received: from mail-pl1-f181.google.com (mail-pl1-f181.google.com [209.85.214.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D1A0284D34 for ; Thu, 30 Jan 2025 11:11:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738235515; cv=none; b=jTsFqTsPBBNkjJIpy/uC/fjCGwJkkmBZNbD+z+eQ3QnPf3zQ4hk62uDODA8gAE6xRWxcxAZAMIxqhivBPcDnhBO6AVOltmX2jEAPfRwFJMR1UYTRyBGAnRFJwRflzXsRnkU9onBk9yYJRXcbp/RuMnOOJ+rCwdtpRdgWns7VK00= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738235515; c=relaxed/simple; bh=kRAbWKJJ5y8eaP0OOtOHFcqB8zgDUzcJUDjcrxd/eUk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=hkHK/GoeXCwae6tW8H4+ufUZ0WQbUbOmjpJqYksxTeoO0aHqxInei4OMwVrD72WnUsJfDKrCnr4LRip5DQi1+mqcGNnBPrc32AF3TCMh1i7HqUwnwTHJMF/p9MZVqGBkUF8nDx1IkdCtiRGGxmvfDtmOENE/Nx6mpG6pg1nmACI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=TN3h0kBO; arc=none smtp.client-ip=209.85.214.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="TN3h0kBO" Received: by mail-pl1-f181.google.com with SMTP id d9443c01a7336-21bc1512a63so11318085ad.1 for ; Thu, 30 Jan 2025 03:11:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1738235512; x=1738840312; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=j8q0TE8Cfn/ZZzUdNtKaagvke7zTsDMFwn2syZsA7PY=; b=TN3h0kBOcDWrdbuEEnW5Re1eTSnlLR+p1WBmyVEUsu4EJB18UOEAmPNKb7GgpdwOF5 56GCq6+67uALZOOLKXz2GPuSCE5r5lqVltF5DlTZViJrAe05BlBtmbSDriiWHrjxdKum xj03hS2QmRkgl7gKWf8bRotnosvcNEDR//6oI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738235512; x=1738840312; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=j8q0TE8Cfn/ZZzUdNtKaagvke7zTsDMFwn2syZsA7PY=; b=FPftV/kRmmr7Nj4agDQAuzqC4HHNnKmWyMOmVu1OZJH3iOVqF5OxjO6VT5FtEZuVpR tYIUzUo+0vzaPNnyO1OyBiroMJbbFFhHQuZvq9LNI8C18ZVljcYoRoOVfo4cOOTdYzdT Vt2969ZHnlhBrA6zfeGeBdP0MS7kftnDnkImFf3+MghGkfG6cG3GG3R0F1348dnQYngE loWEyRY1DZNZXqUReySHOLEwd5sM/QfNttTub4pzIQlAHh08BgHahXSQzlISgEtvrzpi EHEP9TeBgZ4Ie5pliMXjcF7xl7ZmRNqKKL2LTssScqjAeCuETFFAw2jgzD2EA4/jv2cf hXkA== X-Forwarded-Encrypted: i=1; AJvYcCW4/BvZZfAJciB4xdrmQ335OUFF+GVq2MFmyHiAsf/Aa0z7dBlAalrjKiDN/x+f88B2PGTJgvMbje8KjiI=@vger.kernel.org X-Gm-Message-State: AOJu0YyAK+4QSCFaMrP9sxXVsvwo1DOsWIabQ0XcTcmeGrmAKwBt6lfX vnnbD34Kk7Wfvb3JyGl6R09U+6IC7hm3aCSkmZ+3pGM/MO0EGS+LHArJJUeD+A== X-Gm-Gg: ASbGncsVSSh+vNq4dhRchF0cxDQdNhfIQWNzOFs0ZDAsbL7V8kt1D3F/zCgk41zUOD6 gqjCx8c6PqulhuSKs46XRLpDmu14xaz0reIhH1XLzTm+4ky3Ir7UoUV5HU0XcTsaT7tF/IhJD4R oUtbXuz8k09xOtS5wwcA82hMEXP9ADYHjo1z8YqEkT+TWq7kTek8jFYjraQqncekYz4OmN46S8U zyIEB7drp9kSSceLkNWiRUCSRpn1PfkMAWelVoFQS9aCa544psyor0wJxrGuiXjPjWLNl/vSZy3 n7b9RaQkhKrJDIFt X-Google-Smtp-Source: AGHT+IERWIfj8pq2Tiil2nncQexj/06z4KXWkO06dhH+74uQD2LlR8gXztf4GGGtlqNiKmWRpbsKLg== X-Received: by 2002:a17:903:41c3:b0:215:b190:de6 with SMTP id d9443c01a7336-21dd7c49786mr81937115ad.3.1738235512078; Thu, 30 Jan 2025 03:11:52 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:d4ce:e744:f46b:4fb]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-21de31efba5sm11500615ad.6.2025.01.30.03.11.50 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 30 Jan 2025 03:11:51 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv3 08/11] zram: remove writestall zram_stats member Date: Thu, 30 Jan 2025 20:10:53 +0900 Message-ID: <20250130111105.2861324-9-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.262.g85cc9f2d1e-goog In-Reply-To: <20250130111105.2861324-1-senozhatsky@chromium.org> References: <20250130111105.2861324-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" There is no zsmalloc handle allocation slow path now and writestall is not possible any longer. Remove it from zram_stats. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 3 +-- drivers/block/zram/zram_drv.h | 1 - 2 files changed, 1 insertion(+), 3 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index ac8b4f47a496..af4bce1e5fe1 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1430,9 +1430,8 @@ static ssize_t debug_stat_show(struct device *dev, =20 down_read(&zram->init_lock); ret =3D scnprintf(buf, PAGE_SIZE, - "version: %d\n%8llu %8llu\n", + "version: %d\n0 %8llu\n", version, - (u64)atomic64_read(&zram->stats.writestall), (u64)atomic64_read(&zram->stats.miss_free)); up_read(&zram->init_lock); =20 diff --git a/drivers/block/zram/zram_drv.h b/drivers/block/zram/zram_drv.h index 045032acb22f..b7e250d6fa02 100644 --- a/drivers/block/zram/zram_drv.h +++ b/drivers/block/zram/zram_drv.h @@ -78,7 +78,6 @@ struct zram_stats { atomic64_t huge_pages_since; /* no. of huge pages since zram set up */ atomic64_t pages_stored; /* no. of pages currently stored */ atomic_long_t max_used_pages; /* no. of maximum pages stored */ - atomic64_t writestall; /* no. of write slow paths */ atomic64_t miss_free; /* no. of missed free */ #ifdef CONFIG_ZRAM_WRITEBACK atomic64_t bd_count; /* no. of pages in backing device */ --=20 2.48.1.262.g85cc9f2d1e-goog From nobody Fri Jan 31 03:24:27 2025 Received: from mail-pj1-f45.google.com (mail-pj1-f45.google.com [209.85.216.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DF5EF1E412A for ; Thu, 30 Jan 2025 11:11:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.45 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738235519; cv=none; b=rrlBhLjcOXeIvyI3VMlbmb2hxEjHImRm3TDD0z8duoabGTPWU57oYoZIYeubf2H3sAav1mLwrX7+4uwiVDkaqK0G1m0pAaf1p1ZaWeiQtKE2I1wxDdlzEwLwC++uSNVlOC+u5uNKH0Iqcxz/c9pZguRXDu+zI9eqxdb7r1U8sww= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738235519; c=relaxed/simple; bh=mDnXt5ij7Tos2MO2avtGUyUVgMTFEnOrPvFNbYuMx3I=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=OBKOZdsULhNdPy1ZWyLkpqvmDDq5g1aTrdKDE7HqUkJwDczHUdDdWIxLu0wX5Xy9sSFw2q8Vn9N38WfDCiR4n8gI69Y0xLrPtYzTWRruXkZPPxNb9NyeYxunlmvzcZVtKwS2wMWiy+DdL+mos057DOldzd/XfwS7WXp1VCcZjcM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=R46+FyN/; arc=none smtp.client-ip=209.85.216.45 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="R46+FyN/" Received: by mail-pj1-f45.google.com with SMTP id 98e67ed59e1d1-2ee50ffcf14so2748360a91.0 for ; Thu, 30 Jan 2025 03:11:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1738235517; x=1738840317; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=lQgITLXPodNhhg8f+6QVX/YiryNImYUIb8lIo2kz1cc=; b=R46+FyN/ZNzWiPkDbXVxDeYtxDmEoAkqnVmeOf0zytlaiQ9R1yVzfZlmPuH2rpL3ce 4xCT3sZA3r5KV7SMqZ9kaoLzeJ2yZcnG/2aUz9rKtZt35dLdsPUvYvvospgdTh9sLiVo 9fklCzWF2lrQj/w5ledP060Y1fHZXPjNX7EH8= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738235517; x=1738840317; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=lQgITLXPodNhhg8f+6QVX/YiryNImYUIb8lIo2kz1cc=; b=FiiEsqw78IjC/WMnE7MaC/+IIJr620XKYTJl6d3A0FAlIgCXiMbJespRnayfqsdnWo zbWSd4yplDm3bMdrLkVHMJZwu5FhBzVEaaoEdpscl1ZDVwY2XYbH/eZ8h+5sR7qJoJ1t jZlz8UdrCSXWuYwH9S1fiaSSmeRsRzRGF8b/B1563Hx3dcUuv3ccT9+dekH2P22fW2z0 mrpe5wL4RltxEWZsQqRpmsY7h4/MgBd4VydW1UNNHf6X8U3FwdItUp0LOOCjpRxNQ+fT ERO00fYCchdHAjoNeWpH9xKeU6YR30LeHgzkM1VecIcjy2h/eUnXpezRlaZr3n2DoHey fQiw== X-Forwarded-Encrypted: i=1; AJvYcCUfSUAyxc7nElfpdrc5+X4r5+cBKzNBPE5Eh5mf6oND/Ok7+V9AC9EkBLbwFh8IxKrCthTbac7Ar0c67+g=@vger.kernel.org X-Gm-Message-State: AOJu0YzJyDwPTDKhgdTTmza3NjJv0zRWejv0UJNAaBbSgnNjBrv9LD8+ n3BfJYSHQMlX5HcQHLjGsC/OHYRNszDh8ulwsQlss+K7jWh+0b83oCrjbe6X3A== X-Gm-Gg: ASbGncvInC0QUkNbrV0tcP9/HGu4DFXLNJOwxcwV0ullIjw6At1E+bBIX9gQUYkc+qn JLfdehCs3LPgh/EWcn17s+jMC4ISzFU+sUiI8bsLdy1vwL3m8bhq/gIAwhYuYLxcqSVPs6MTO44 SM9pfw3rVbDFySyxmdiZlkOg7jkIUH9YZVNjYGQZvjOmRTJInCqerY/oHMlm57wXG6sXi9pA3uz S4VSxF7CquIeiWaBhrZyrOtRBrjODpkLkoMsbQm/fU84c43KpWwx/WA2kaOw49p+xF+UzCe/FYY z8ANzACEstj2TTrU X-Google-Smtp-Source: AGHT+IHdIqAq1Ip7rjsfMeTr4P2qYVZikb+j/nc93lXmG1/0trH7vosRiIP+YWEpphAlbCSobUlWtw== X-Received: by 2002:aa7:86d6:0:b0:729:425:fde4 with SMTP id d2e1a72fcca58-72fe2dbbf37mr4157189b3a.11.1738235517164; Thu, 30 Jan 2025 03:11:57 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:d4ce:e744:f46b:4fb]) by smtp.gmail.com with UTF8SMTPSA id d2e1a72fcca58-72fe69bba21sm1144810b3a.114.2025.01.30.03.11.55 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 30 Jan 2025 03:11:56 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv3 09/11] zram: limit max recompress prio to num_active_comps Date: Thu, 30 Jan 2025 20:10:54 +0900 Message-ID: <20250130111105.2861324-10-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.262.g85cc9f2d1e-goog In-Reply-To: <20250130111105.2861324-1-senozhatsky@chromium.org> References: <20250130111105.2861324-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Use the actual number of algorithms zram was configure with instead of theoretical limit of ZRAM_MAX_COMPS. Also make sure that min prio is not above max prio. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index af4bce1e5fe1..3e91fd1dcfc7 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -2011,16 +2011,19 @@ static ssize_t recompress_store(struct device *dev, struct device_attribute *attr, const char *buf, size_t len) { - u32 prio =3D ZRAM_SECONDARY_COMP, prio_max =3D ZRAM_MAX_COMPS; struct zram *zram =3D dev_to_zram(dev); char *args, *param, *val, *algo =3D NULL; u64 num_recomp_pages =3D ULLONG_MAX; struct zram_pp_ctl *ctl =3D NULL; struct zram_pp_slot *pps; u32 mode =3D 0, threshold =3D 0; + u32 prio, prio_max; struct page *page; ssize_t ret; =20 + prio =3D ZRAM_SECONDARY_COMP; + prio_max =3D zram->num_active_comps; + args =3D skip_spaces(buf); while (*args) { args =3D next_arg(args, ¶m, &val); @@ -2073,7 +2076,7 @@ static ssize_t recompress_store(struct device *dev, if (prio =3D=3D ZRAM_PRIMARY_COMP) prio =3D ZRAM_SECONDARY_COMP; =20 - prio_max =3D min(prio + 1, ZRAM_MAX_COMPS); + prio_max =3D prio + 1; continue; } } @@ -2087,6 +2090,12 @@ static ssize_t recompress_store(struct device *dev, goto release_init_lock; } =20 + prio_max =3D min(prio_max, (u32)zram->num_active_comps); + if (prio >=3D prio_max) { + ret =3D -EINVAL; + goto release_init_lock; + } + /* Do not permit concurrent post-processing actions. */ if (atomic_xchg(&zram->pp_in_progress, 1)) { up_read(&zram->init_lock); @@ -2101,7 +2110,7 @@ static ssize_t recompress_store(struct device *dev, continue; =20 if (!strcmp(zram->comp_algs[prio], algo)) { - prio_max =3D min(prio + 1, ZRAM_MAX_COMPS); + prio_max =3D prio + 1; found =3D true; break; } --=20 2.48.1.262.g85cc9f2d1e-goog From nobody Fri Jan 31 03:24:27 2025 Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8459F1E4937 for ; Thu, 30 Jan 2025 11:12:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.180 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738235523; cv=none; b=ZIbdPV1ZUfwcJpUUUDt1anH9I+ghWlrQsAiltOn/sdR++qA2+uzn0jVI6D1INs1UpdduKPEIX2knJYYe/x0DcGcL65muXBsc9qX1gd8UF6/lQcd0s5SSzKi/bAyztArFhjmwCl88V9hI8kE0i/15CQekCLzj9Hgq/udjjqKCFfg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738235523; c=relaxed/simple; bh=AMmTlK4+xzSvX2PK3siKLSPuSS5pdb3uTdedljhoVtE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=pWvn4wZWo9sU7+xmcro9HGIZFNtUCCYe/ECMCNP5JZJNK/k3ef79SjonFL8Gxuiqi/amMZh0nWO9el0kRteFOk7fuz+NgtL76Q4IW4yXlWHqmZk4w/fkOE2FIdoz34v32cV65jECfTtKfpTh1e5Dw9RbDSKrvTJg668xzCh98x4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=CSXFknMk; arc=none smtp.client-ip=209.85.214.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="CSXFknMk" Received: by mail-pl1-f180.google.com with SMTP id d9443c01a7336-2167141dfa1so9630675ad.1 for ; Thu, 30 Jan 2025 03:12:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1738235522; x=1738840322; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=M1+IQlt9pyJwpRT7WcoMT+zWDZRWfxL4uFL06pInV4g=; b=CSXFknMkI/HCtyJOssF7j+/RnYHL08/HSi8jDZbYQG2vFOH1zmuHiJAji6xMFi0TWf +SHhswnyR5veMKxWqaBX5gZ7/ZiR6Fu6fXT59T6cIPQ7FTP2p6/cxqI/xdvvpv0JQmxO dEooiEBD1l/+PTscPnHuWZujcTAwQvo6nl6sE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738235522; x=1738840322; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=M1+IQlt9pyJwpRT7WcoMT+zWDZRWfxL4uFL06pInV4g=; b=QuKUmOCPEWh6ZGNq7pIxY8RtD5xi8RL5aj0HQboNz+zSFAgBmeOnSkd6bQFIhfghFt pxFSgDS8eoXDxzp0m2YWUkQ0lFk2EtthOIgGKPa8GLSPMtjjzS1PztEYUn7uKuyL+tfk pp8nhx9MQ2FOp/Pi3W7bqd/R6qJEsUMOEn8UAwFfu8ozcpGgBGEnQPgm5sA6QUjYjuLq DlIsElbngdnx/jBhp/1Yxvxa3ZYznGX/vi6A7tIFViTgg+BD25w8vQvTdIJXenH7XTHL RQmQ3ZGHYVwLCtQLMKu09GbFwdCyqqp08grQGP3WYL4RdqaAbEv9yYblficuwOvo7nf7 kOWQ== X-Forwarded-Encrypted: i=1; AJvYcCU0R1sZTyV9jZamBYS2G4SDI5efmaglGvLvN4jWZd6ng25WC6Zy4QnJ83SZdAfvV7pBQSlPSWUUfkZkcJM=@vger.kernel.org X-Gm-Message-State: AOJu0YwUGfOVZWS34kZm7hEscTO2lg/lrPw6YGIbBw3izteZxq6SuI9X ZmkqyhmCGAhY793ZN8HQc5MDIrf2xgVPg7SXBwXEFswW6lz6NzxKGhPNaTmFu8B7nlCYGz5MCPY = X-Gm-Gg: ASbGncsvLEwQ1siBoPUEkd26+IwHaZculG/AIgkKZ7I8s01pZtZquBmGk0K6W294eRk VhW6Hak8NVftFepH+y4YXwdtpQJblHuChPhPHyUNg0kjVu0rllrwWgh/CQkIFYA7B8oR9Ls5rpO QFfEGih9RilS8tRJytm4kb83VIoCFXSlmrlXHZ4YRIhmYig2ax00I4FI23nN9MrarMlYq1Qp0hB kMYu5qxltxUkfxM4XZ5P5GHc66i0Ww6Pr00JZkEZi1LKn4DOPwAd9VZ+exFMmQ+phAtQffxG7QZ etzsVwuwBjQRKT6D X-Google-Smtp-Source: AGHT+IEpCpAng/2jtEa7zxIDkv9KC8ilU6auGAeJO2psWZMa0QEO4BHwp4dFoq3oCn6eJDAqgONEiw== X-Received: by 2002:a17:903:2284:b0:215:a808:61cf with SMTP id d9443c01a7336-21de19ca626mr48669045ad.25.1738235521983; Thu, 30 Jan 2025 03:12:01 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:d4ce:e744:f46b:4fb]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-21de3300880sm11526865ad.168.2025.01.30.03.12.00 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 30 Jan 2025 03:12:01 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv3 10/11] zram: filter out recomp targets based on priority Date: Thu, 30 Jan 2025 20:10:55 +0900 Message-ID: <20250130111105.2861324-11-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.262.g85cc9f2d1e-goog In-Reply-To: <20250130111105.2861324-1-senozhatsky@chromium.org> References: <20250130111105.2861324-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Do no select for post processing slots that are already compressed with same or higher priority compression algorithm. This should save some memory, as previously we would still put those entries into corresponding post-processing buckets and filter them out later in recompress_slot(). Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 3e91fd1dcfc7..500d6c8b17fc 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1813,7 +1813,7 @@ static int zram_bvec_write(struct zram *zram, struct = bio_vec *bvec, #define RECOMPRESS_IDLE (1 << 0) #define RECOMPRESS_HUGE (1 << 1) =20 -static int scan_slots_for_recompress(struct zram *zram, u32 mode, +static int scan_slots_for_recompress(struct zram *zram, u32 mode, u32 prio= _max, struct zram_pp_ctl *ctl) { unsigned long nr_pages =3D zram->disksize >> PAGE_SHIFT; @@ -1845,6 +1845,10 @@ static int scan_slots_for_recompress(struct zram *zr= am, u32 mode, zram_test_flag(zram, index, ZRAM_INCOMPRESSIBLE)) goto next; =20 + /* Already compressed with same of higher priority */ + if (zram_get_priority(zram, index) + 1 >=3D prio_max) + goto next; + pps->index =3D index; place_pp_slot(zram, ctl, pps); pps =3D NULL; @@ -2134,7 +2138,7 @@ static ssize_t recompress_store(struct device *dev, goto release_init_lock; } =20 - scan_slots_for_recompress(zram, mode, ctl); + scan_slots_for_recompress(zram, mode, prio_max, ctl); =20 ret =3D len; while ((pps =3D select_pp_slot(ctl))) { --=20 2.48.1.262.g85cc9f2d1e-goog From nobody Fri Jan 31 03:24:27 2025 Received: from mail-pl1-f181.google.com (mail-pl1-f181.google.com [209.85.214.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C370B1C1AD4 for ; Thu, 30 Jan 2025 11:12:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738235529; cv=none; b=jGliZfsXdc/FOKWOyTqilKQ+XMrnnSAl2YKcTP1UCl/+K2jx9PYsyURi+hNrRq6GokJyOv1kA9FhAeOntCn7Gt1rrVogw22D4H2l2kSisaYVcxm10f4n9zafY5szhMQq63UuYflYmTK3CWc98u+1w6tnSlE/+KSL0+ZpO8nGWNo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738235529; c=relaxed/simple; bh=iHY1ysTq1PF3D/y6XmbzczXcD651ZYxNBzFqLWai1DM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=pfbYyvz29S20Ge0/gw8P/06+9u2VX1MKRShODnh+byon9S8mlKmM1rvV4RUCA6TQbrdh61gO/xg9NqtMdxtCJT5es+ALqbFq5SvnrL31/QFG4idtuXGKgPsznq0ZqLffuEj8wv7mdcYA6i8PBlF4cL+7UHiotLNsYKG+9RtbPQg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=RhArdJKK; arc=none smtp.client-ip=209.85.214.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="RhArdJKK" Received: by mail-pl1-f181.google.com with SMTP id d9443c01a7336-2164b662090so11617795ad.1 for ; Thu, 30 Jan 2025 03:12:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1738235527; x=1738840327; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=MLxgJRPaVNwNF6Kcm4INbLog064v7nke9leBvmEg3e4=; b=RhArdJKKjZMBKAj13d7UQM5sSmoPHJtxYahRfXRgJsFBUqQMOhotVDny8Dw7t/KDMQ qEs7RW0MgBQzL4tdsmeTOPHtGc4X1TzQ+ee08Ll/MSY9vYNilkY0fH0co8I0SUPB1EFD 3+cHkvpYzl5fPSDYZ9hU9CsPTAgJxfPUN0PGo= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738235527; x=1738840327; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=MLxgJRPaVNwNF6Kcm4INbLog064v7nke9leBvmEg3e4=; b=I5kGmsVstDHvGuo6crNmEWdXHQBKfZUbzBHMDGa67KmTVs66POS0FPYSRt8sYCRDzz cUF4B8og4FQEarUTD//x7KrXCjfKJejXEYMSvAUbWi3lIMTB0Ap65/NgzS+pafgby6zE CH8gQoWGXbv/AaIviMU44UfshIb/K6yYRUlrJQ3V4fDjObsz/sWePnhAjvlKGE559Hdd 8BEa8u6HSuwJDXjYBdedz+JGQyabdAAsf7LsRaJsCT77bnPSCVEecIn86xkes/fUGo5P ve8S6Un/ZQxklUTm4mOY+Sn4cUpgDoBaMxUQ6EcBixM/4FB3CGxdjP/pFAHQB3dKILiF Bm6g== X-Forwarded-Encrypted: i=1; AJvYcCWoMNnFTmoYziW2xRRqT3VFs70xhhtuHTEnKu9rQiwGUqa9Evm7gD5z1oCMz0xCcQ4quVwY+WhZc/oxBb4=@vger.kernel.org X-Gm-Message-State: AOJu0YxZNmS2OEmKO23oJ43DAB/Rb9/RaMY8rrMZMr0ByAdQXlT5dYAD shuo9lKGwz4TG7XDFl9ZJJ25Z2bKObIon6jT0prf3KnOLb31E0/NmUr+AOGPXg== X-Gm-Gg: ASbGncvZDw6JByVtr0/uQQ+DicIE+nzMUrFR7oFCSWT1lXJYh3g3jqD5ReROCRRjc6I yMUzdNkbHzPXyaiEqDhCDVe+ihOG890GINBjO+eCflUOC37byMTZTTCts6tEwuJU08x17h7FvII IMW3H5C9HoXA6xZibomH+CPKIjkoroL8QNodRw7GSbLc6mr5KZdEU5hewaNqBf2d7Fy820LmPrb wOLSAqbPVH5xnZ1Dwb1dj0MYb5rj4YC2PbGCveMZGAJUQDsoCAi5vbhjAB6zGcgDTsmE4aBIM46 f0fxBM4cBBTd+P+b X-Google-Smtp-Source: AGHT+IEmgqIzatKzhSxji3bLavuavRKHjn6PnEoK7vJ9Qf8RpneZSWnCVW2omFdSy80mZ0nOsxnoUg== X-Received: by 2002:a05:6a20:9c8e:b0:1e6:8f39:d607 with SMTP id adf61e73a8af0-1ed7a6b17bcmr12484502637.31.1738235527034; Thu, 30 Jan 2025 03:12:07 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:d4ce:e744:f46b:4fb]) by smtp.gmail.com with UTF8SMTPSA id d2e1a72fcca58-72fe6429218sm1210151b3a.66.2025.01.30.03.12.05 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 30 Jan 2025 03:12:06 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv3 11/11] zram: unlock slot during recompression Date: Thu, 30 Jan 2025 20:10:56 +0900 Message-ID: <20250130111105.2861324-12-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.262.g85cc9f2d1e-goog In-Reply-To: <20250130111105.2861324-1-senozhatsky@chromium.org> References: <20250130111105.2861324-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Recompression, like writeback, makes a local copy of slot data (we need to decompress it anyway) before post-processing so we can unlock slot-entry once we have that local copy. Unlock the entry write-lock before recompression loop (secondary algorithms can be tried out one by one, in order of priority) and re-acquire it right after the loop. There is one more potentially costly operation recompress_slot() does - new zs_handle allocation, which can schedule(). Release the slot-entry write-lock before zsmalloc allocation and grab it again after the allocation. In both cases, once the slot-lock is re-acquired we examine slot's ZRAM_PP_SLOT flag to make sure that the slot has not been modified by a concurrent operation. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 80 +++++++++++++++++++---------------- 1 file changed, 44 insertions(+), 36 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 500d6c8b17fc..a6bc1c2dfbe6 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1871,14 +1871,13 @@ static int recompress_slot(struct zram *zram, u32 i= ndex, struct page *page, u64 *num_recomp_pages, u32 threshold, u32 prio, u32 prio_max) { - struct zcomp_strm *zstrm =3D NULL; + struct zcomp_strm *zstrm; unsigned long handle_old; unsigned long handle_new; unsigned int comp_len_old; unsigned int comp_len_new; unsigned int class_index_old; unsigned int class_index_new; - u32 num_recomps =3D 0; void *src, *dst; int ret; =20 @@ -1905,6 +1904,13 @@ static int recompress_slot(struct zram *zram, u32 in= dex, struct page *page, zram_clear_flag(zram, index, ZRAM_IDLE); =20 class_index_old =3D zs_lookup_class_index(zram->mem_pool, comp_len_old); + prio =3D max(prio, zram_get_priority(zram, index) + 1); + /* Slot data copied out - unlock its bucket */ + zram_slot_write_unlock(zram, index); + /* Recompression slots scan takes care of this, but just in case */ + if (prio >=3D prio_max) + return 0; + /* * Iterate the secondary comp algorithms list (in order of priority) * and try to recompress the page. @@ -1913,24 +1919,14 @@ static int recompress_slot(struct zram *zram, u32 i= ndex, struct page *page, if (!zram->comps[prio]) continue; =20 - /* - * Skip if the object is already re-compressed with a higher - * priority algorithm (or same algorithm). - */ - if (prio <=3D zram_get_priority(zram, index)) - continue; - - num_recomps++; zstrm =3D zcomp_stream_get(zram->comps[prio]); src =3D kmap_local_page(page); ret =3D zcomp_compress(zram->comps[prio], zstrm, src, &comp_len_new); kunmap_local(src); =20 - if (ret) { - zcomp_stream_put(zram->comps[prio], zstrm); - return ret; - } + if (ret) + break; =20 class_index_new =3D zs_lookup_class_index(zram->mem_pool, comp_len_new); @@ -1939,6 +1935,7 @@ static int recompress_slot(struct zram *zram, u32 ind= ex, struct page *page, if (class_index_new >=3D class_index_old || (threshold && comp_len_new >=3D threshold)) { zcomp_stream_put(zram->comps[prio], zstrm); + zstrm =3D NULL; continue; } =20 @@ -1946,14 +1943,7 @@ static int recompress_slot(struct zram *zram, u32 in= dex, struct page *page, break; } =20 - /* - * We did not try to recompress, e.g. when we have only one - * secondary algorithm and the page is already recompressed - * using that algorithm - */ - if (!zstrm) - return 0; - + zram_slot_write_lock(zram, index); /* * Decrement the limit (if set) on pages we can recompress, even * when current recompression was unsuccessful or did not compress @@ -1963,37 +1953,55 @@ static int recompress_slot(struct zram *zram, u32 i= ndex, struct page *page, if (*num_recomp_pages) *num_recomp_pages -=3D 1; =20 - if (class_index_new >=3D class_index_old) { + /* Compression error */ + if (ret) { + zcomp_stream_put(zram->comps[prio], zstrm); + return ret; + } + + if (!zstrm) { /* * Secondary algorithms failed to re-compress the page - * in a way that would save memory, mark the object as - * incompressible so that we will not try to compress - * it again. + * in a way that would save memory. * - * We need to make sure that all secondary algorithms have - * failed, so we test if the number of recompressions matches - * the number of active secondary algorithms. + * Mark the object incompressible if the max-priority + * algorithm couldn't re-compress it. */ - if (num_recomps =3D=3D zram->num_active_comps - 1) + if (prio < zram->num_active_comps) + return 0; + if (zram_test_flag(zram, index, ZRAM_PP_SLOT)) zram_set_flag(zram, index, ZRAM_INCOMPRESSIBLE); return 0; } =20 - /* Successful recompression but above threshold */ - if (threshold && comp_len_new >=3D threshold) + /* Slot has been modified concurrently */ + if (!zram_test_flag(zram, index, ZRAM_PP_SLOT)) { + zcomp_stream_put(zram->comps[prio], zstrm); return 0; + } =20 - /* - * If we cannot alloc memory for recompressed object then we bail out - * and simply keep the old (existing) object in zsmalloc. - */ + /* zsmalloc handle allocation can schedule, unlock slot's bucket */ + zram_slot_write_unlock(zram, index); handle_new =3D zs_malloc(zram->mem_pool, comp_len_new, GFP_NOIO | __GFP_HIGHMEM | __GFP_MOVABLE); + zram_slot_write_lock(zram, index); + + /* + * If we couldn't allocate memory for recompressed object then bail + * out and simply keep the old (existing) object in mempool. + */ if (IS_ERR_VALUE(handle_new)) { zcomp_stream_put(zram->comps[prio], zstrm); return PTR_ERR((void *)handle_new); } =20 + /* Slot has been modified concurrently */ + if (!zram_test_flag(zram, index, ZRAM_PP_SLOT)) { + zcomp_stream_put(zram->comps[prio], zstrm); + zs_free(zram->mem_pool, handle_new); + return 0; + } + dst =3D zs_map_object(zram->mem_pool, handle_new, ZS_MM_WO); memcpy(dst, zstrm->buffer, comp_len_new); zcomp_stream_put(zram->comps[prio], zstrm); --=20 2.48.1.262.g85cc9f2d1e-goog