From nobody Thu Dec 18 10:00:39 2025 Received: from mail-pl1-f174.google.com (mail-pl1-f174.google.com [209.85.214.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4DB231FBC8C for ; Wed, 12 Feb 2025 06:32:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739341930; cv=none; b=rgor+hFsr90LeGNHM/G3K/O6h6P1RhjM7k68aEW/Kp9qRZ8c/vR5wun3jxK0rEGWFOoHqwXpCOzP4d3PgOvPRTttPWIMeheIPBDg3R8DbPK1pJZPhtj/s6WSq//1/FaiNeWMT53XuWHNPtfJYeRxcg+e1eqd7jvLu7SlL9H28DM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739341930; c=relaxed/simple; bh=ftM5GNwd6HoMW8lSepoMzzNcEpey/3RV/ln/tuvRXA4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=LBlN7zUFXKdnSt7PTQn+WlPnLPoYHBtszYSRU9yyIQjJe45pkzyrQRi+H01JcnlaDzRg6D8+rIAimEdPWzyO9JA2QtNvAQg/M+EUNGSnwwoOyrMPjfNsKb6SR/eFAQkXOuhvw6Q8u9srUW+6g5qsVo/LkgMdfIwFcEPRX+sf3dg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=TjsqoVTP; arc=none smtp.client-ip=209.85.214.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="TjsqoVTP" Received: by mail-pl1-f174.google.com with SMTP id d9443c01a7336-220bfdfb3f4so6530105ad.2 for ; Tue, 11 Feb 2025 22:32:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1739341927; x=1739946727; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Nbviwh7jX+fyZgx+5zWo4yndOTBdlLf1Q5tRKthE6k8=; b=TjsqoVTPn4JxHTl9NQoIJJ9EddsPP/tnluhoC5vPcBaDWEfP9y5dBVa0lEVIh+d8Zl vuAF2PtxSjTjoKRKreB5S7VvZEjgfBFNF3lZEI2q8SOQsOJ2HKHUuPATsMtLFYA9BxmZ SdUOyZlXHmiWqditg6AaHTSjchxND7a2R4Bk0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739341927; x=1739946727; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Nbviwh7jX+fyZgx+5zWo4yndOTBdlLf1Q5tRKthE6k8=; b=KkhGt4sAw33P7LKOYmgxq7AbwmKnRkauIGtDjV276n9ejQMLZQYqj5g2EfyRjOGUls qRz9fm4TJvFnDuczsjUKzdGWk9j3goR7HnKzsRR6ee9BCXV1T5WMXSU23edy+BZq+Afd emtTLhjZTEroglqUemLqYoBbIfHcG1Jwkp9zfE574mKj+d3XrGwQ/eqHcLbskiHO9LKN I7JLjEf8pUvEwKk+Wc2wDUbEGhDZ9hBZvZ771BFz4CBSzS2m7KYBMdyioh+yfw2R4YQI 8aG3F0pNYd7YQDS1QYZWliI7dIZbi/G63z+O4tlS25fPyWJZT1hFei01CrkleMAhYe9t 0fjw== X-Forwarded-Encrypted: i=1; AJvYcCV0e9uupSDeCA1jB+wWqlyf8i3XKDCjQwuLjiu8aDoOthUXmrXAbirhdnGZW5yJLCtvfaEdP9CO0UVfr4U=@vger.kernel.org X-Gm-Message-State: AOJu0YzJJu3FLSvqJOFJ1YRoe3FfsKCmwPBUwe6cBGdkAZ+yDWuWzFlE YOcAzmsfWS8FB/ZWBoOgpDiiGQxkXdL4rFpMmlKa9w7QA3Hiqw9SKRvO5kkAxw== X-Gm-Gg: ASbGnctdT8m9K3Y7IIGfgSfUhAHnRyMNdN9452esHUeu2OA+cQttxaMkNqpuJiXZUub qBxU1L51boN2mIDwla/hLupNqHFrNu3PVqB/qaOTJhHHd36jJwdHMpzQ1wyBIpQ0l7LVIsIUbGE Xhf9NnV23U7u63Pd3PzXUzhxsCJSFRwjaeh8+LGNou4PC4enqk6Va4jI7gokrTNSa0qBWhSTF4N RDBSmlP+17IxpA/tc2Zg5e3cyyswuahNzTNMxfZ40DVAwjYG3ohWVPIDWeYs3V3kC2jz8OncF0i N6pkPm33KoQgzOze2Q== X-Google-Smtp-Source: AGHT+IFSvyC57xadlYDsJ2pjEf3GwKRx0RFUmqWoHBxBXjtPHZ1A+qKCAZJUZUiYg24snMECPbVRZA== X-Received: by 2002:a17:902:ea11:b0:216:46f4:7e3d with SMTP id d9443c01a7336-220bbad6f94mr31042255ad.15.1739341925761; Tue, 11 Feb 2025 22:32:05 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:69f5:6852:451e:8142]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-220c34f443bsm2839685ad.5.2025.02.11.22.32.03 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 11 Feb 2025 22:32:05 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Kairui Song , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v5 01/18] zram: sleepable entry locking Date: Wed, 12 Feb 2025 15:26:59 +0900 Message-ID: <20250212063153.179231-2-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.502.g6dc24dfdaf-goog In-Reply-To: <20250212063153.179231-1-senozhatsky@chromium.org> References: <20250212063153.179231-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Concurrent modifications of meta table entries is now handled by per-entry spin-lock. This has a number of shortcomings. First, this imposes atomic requirements on compression backends. zram can call both zcomp_compress() and zcomp_decompress() under entry spin-lock, which implies that we can use only compression algorithms that don't schedule/sleep/wait during compression and decompression. This, for instance, makes it impossible to use some of the ASYNC compression algorithms (H/W compression, etc.) implementations. Second, this can potentially trigger watchdogs. For example, entry re-compression with secondary algorithms is performed under entry spin-lock. Given that we chain secondary compression algorithms and that some of them can be configured for best compression ratio (and worst compression speed) zram can stay under spin-lock for quite some time. Having a per-entry mutex (or, for instance, a rw-semaphore) significantly increases sizeof() of each entry and hence the meta table. Therefore entry locking returns back to bit locking, as before, however, this time also preempt-rt friendly, because if waits-on-bit instead of spinning-on-bit. Lock owners are also now permitted to schedule, which is a first step on the path of making zram non-atomic. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 65 ++++++++++++++++++++++++++++------- drivers/block/zram/zram_drv.h | 20 +++++++---- 2 files changed, 67 insertions(+), 18 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 9f5020b077c5..3708436f1d1f 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -58,19 +58,57 @@ static void zram_free_page(struct zram *zram, size_t in= dex); static int zram_read_from_zspool(struct zram *zram, struct page *page, u32 index); =20 -static int zram_slot_trylock(struct zram *zram, u32 index) +static void zram_slot_lock_init(struct zram *zram, u32 index) { - return spin_trylock(&zram->table[index].lock); +#ifdef CONFIG_DEBUG_LOCK_ALLOC + lockdep_init_map(&zram->table[index].lockdep_map, "zram-entry->lock", + &zram->table_lockdep_key, 0); +#endif +} + +/* + * entry locking rules: + * + * 1) Lock is exclusive + * + * 2) lock() function can sleep waiting for the lock + * + * 3) Lock owner can sleep + * + * 4) Use TRY lock variant when in atomic context + * - must check return value and handle locking failers + */ +static __must_check bool zram_slot_try_lock(struct zram *zram, u32 index) +{ + unsigned long *lock =3D &zram->table[index].flags; + + if (!test_and_set_bit_lock(ZRAM_ENTRY_LOCK, lock)) { +#ifdef CONFIG_DEBUG_LOCK_ALLOC + mutex_acquire(&zram->table[index].lockdep_map, 0, 1, _RET_IP_); +#endif + return true; + } + return false; } =20 static void zram_slot_lock(struct zram *zram, u32 index) { - spin_lock(&zram->table[index].lock); + unsigned long *lock =3D &zram->table[index].flags; + +#ifdef CONFIG_DEBUG_LOCK_ALLOC + mutex_acquire(&zram->table[index].lockdep_map, 0, 0, _RET_IP_); +#endif + wait_on_bit_lock(lock, ZRAM_ENTRY_LOCK, TASK_UNINTERRUPTIBLE); } =20 static void zram_slot_unlock(struct zram *zram, u32 index) { - spin_unlock(&zram->table[index].lock); + unsigned long *lock =3D &zram->table[index].flags; + +#ifdef CONFIG_DEBUG_LOCK_ALLOC + mutex_release(&zram->table[index].lockdep_map, _RET_IP_); +#endif + clear_and_wake_up_bit(ZRAM_ENTRY_LOCK, lock); } =20 static inline bool init_done(struct zram *zram) @@ -93,7 +131,6 @@ static void zram_set_handle(struct zram *zram, u32 index= , unsigned long handle) zram->table[index].handle =3D handle; } =20 -/* flag operations require table entry bit_spin_lock() being held */ static bool zram_test_flag(struct zram *zram, u32 index, enum zram_pageflags flag) { @@ -1473,15 +1510,11 @@ static bool zram_meta_alloc(struct zram *zram, u64 = disksize) huge_class_size =3D zs_huge_class_size(zram->mem_pool); =20 for (index =3D 0; index < num_pages; index++) - spin_lock_init(&zram->table[index].lock); + zram_slot_lock_init(zram, index); + return true; } =20 -/* - * To protect concurrent access to the same index entry, - * caller should hold this table index entry's bit_spinlock to - * indicate this index entry is accessing. - */ static void zram_free_page(struct zram *zram, size_t index) { unsigned long handle; @@ -2321,7 +2354,7 @@ static void zram_slot_free_notify(struct block_device= *bdev, zram =3D bdev->bd_disk->private_data; =20 atomic64_inc(&zram->stats.notify_free); - if (!zram_slot_trylock(zram, index)) { + if (!zram_slot_try_lock(zram, index)) { atomic64_inc(&zram->stats.miss_free); return; } @@ -2625,6 +2658,10 @@ static int zram_add(void) if (ret) goto out_cleanup_disk; =20 +#ifdef CONFIG_DEBUG_LOCK_ALLOC + lockdep_register_key(&zram->table_lockdep_key); +#endif + zram_debugfs_register(zram); pr_info("Added device: %s\n", zram->disk->disk_name); return device_id; @@ -2681,6 +2718,10 @@ static int zram_remove(struct zram *zram) */ zram_reset_device(zram); =20 +#ifdef CONFIG_DEBUG_LOCK_ALLOC + lockdep_unregister_key(&zram->table_lockdep_key); +#endif + put_disk(zram->disk); kfree(zram); return 0; diff --git a/drivers/block/zram/zram_drv.h b/drivers/block/zram/zram_drv.h index db78d7c01b9a..63b933059cb6 100644 --- a/drivers/block/zram/zram_drv.h +++ b/drivers/block/zram/zram_drv.h @@ -28,7 +28,6 @@ #define ZRAM_SECTOR_PER_LOGICAL_BLOCK \ (1 << (ZRAM_LOGICAL_BLOCK_SHIFT - SECTOR_SHIFT)) =20 - /* * ZRAM is mainly used for memory efficiency so we want to keep memory * footprint small and thus squeeze size and zram pageflags into a flags @@ -46,6 +45,7 @@ /* Flags for zram pages (table[page_no].flags) */ enum zram_pageflags { ZRAM_SAME =3D ZRAM_FLAG_SHIFT, /* Page consists the same element */ + ZRAM_ENTRY_LOCK, /* entry access lock bit */ ZRAM_WB, /* page is stored on backing_device */ ZRAM_PP_SLOT, /* Selected for post-processing */ ZRAM_HUGE, /* Incompressible page */ @@ -58,13 +58,18 @@ enum zram_pageflags { __NR_ZRAM_PAGEFLAGS, }; =20 -/*-- Data structures */ - -/* Allocated for each disk page */ +/* + * Allocated for each disk page. We use bit-lock (ZRAM_ENTRY_LOCK bit + * of flags) to save memory. There can be plenty of entries and standard + * locking primitives (e.g. mutex) will significantly increase sizeof() + * of each entry and hence of the meta table. + */ struct zram_table_entry { unsigned long handle; - unsigned int flags; - spinlock_t lock; + unsigned long flags; +#ifdef CONFIG_DEBUG_LOCK_ALLOC + struct lockdep_map lockdep_map; +#endif #ifdef CONFIG_ZRAM_TRACK_ENTRY_ACTIME ktime_t ac_time; #endif @@ -137,5 +142,8 @@ struct zram { struct dentry *debugfs_dir; #endif atomic_t pp_in_progress; +#ifdef CONFIG_DEBUG_LOCK_ALLOC + struct lock_class_key table_lockdep_key; +#endif }; #endif --=20 2.48.1.502.g6dc24dfdaf-goog From nobody Thu Dec 18 10:00:39 2025 Received: from mail-pl1-f177.google.com (mail-pl1-f177.google.com [209.85.214.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 444FF1FBEAD for ; Wed, 12 Feb 2025 06:32:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739341934; cv=none; b=ph4gerq3n99Oo3YsoDlKKjwXXX/VFt/ja0QPHiCDudEdGBP5E7WVCaHwlom7zhQ3hzLa/rXeJdc1UUsWcnUlWc07k4XPCg0BPTSub20eQ3s4C6aNKwf7ezPtYSJIdBIpcjZ0zZ1ZUHT6grP2TE2zaYNLwx/yx/G+qGX7uZDoaQg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739341934; c=relaxed/simple; bh=LXWedNzaVZq0ZGYHY+ApfAhWS39CM4qqZneLEOL6z5o=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=QfJOWErEiZN8ZQ25bOoyDRAURBWUBVdcRPDoxOcI1UcgK42wt+m3CYC0mKQudSypF7F4z+YXmK1UVAp34+okHLgLz9u060JWPb7mRlL4DO0hFJIJQu19Pt2XTJinF358fO6PGQLE2q3x8K6IQGJwgDDzbaYpNV06L43RhF1dKT8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=PK/u8fMg; arc=none smtp.client-ip=209.85.214.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="PK/u8fMg" Received: by mail-pl1-f177.google.com with SMTP id d9443c01a7336-21fa56e1583so45799565ad.3 for ; Tue, 11 Feb 2025 22:32:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1739341931; x=1739946731; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=j7H+x+6xpsdKzVL/D/QPxhemiaH/onPBYnSp5xsYcj0=; b=PK/u8fMgGr0T8fiARDmlelJ3AtYEVDH9oi0FE98cUMW2DZi6nne/rkm2j1i1DE+mFL OrswG6zHm0iHXW2rpADQG7mVbAl9Z2ulwMLGBS7UpJaKBoT2HCbqhZm1RTWkbNfd8D3J BZRyaYklO4WBpQccLTRTq9jsE1QZc5Gpic7Dw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739341931; x=1739946731; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=j7H+x+6xpsdKzVL/D/QPxhemiaH/onPBYnSp5xsYcj0=; b=C3+74owzcnM++MufizMFmPVbSR+WDe42vO75p4e/qHc6fqgwnZI1Z27winYMZRyI5C itjtKaeEcAAagC4TYB9Stox4ghqMTj1GC7OuASy3WcU97Uc5GpUucBxZ7J43smgO0YQ4 wWip4GaawqXMt3L89XV0FU5WuW+V/HkSjGF+RD2qMwLVaGtTvUe+b7vpxhSXikcgZ+qP SvhUsP43Sg5ZQNwJSBq3NSe9ndPbPmAbn757PstK1stkitsQBiITjy3/Iap5ncBachFQ WF1RY3yaFULm1x+OkKsJmxRzeQxlKILf6W3+/gbu0+fBJjM/Xk30VSYIHjnwB7T+TkPu FPDA== X-Forwarded-Encrypted: i=1; AJvYcCXKUIYfs0kp2dX5uRMnKPfII4OJIKKj+NPR9+1rO7025ntyB6rnVc17tPB3+sDZON0rLouXcvjImFmKt6Y=@vger.kernel.org X-Gm-Message-State: AOJu0Yys/VdpKxmbexsWEQQCKc+q0nE4HSuaFXMHSQ2S046JZ07LOsej aMb17Clx6eCkneSVh+9f0xAIzFwMtf3pV/Q/W22ZD8QDuVxdQoSvkTRaAFusuA== X-Gm-Gg: ASbGncuyzh3eDMmmd2UmIxlrJfQ4lEoojPW6KdKEiRxYBHKey/jSBc9P98BcBsWxm59 L0bvN0lMh+yZUBDznvc4nT0rqfP80cSfTYIJfI3Nd7nW6Ym2uCQ4LCvxV+lI0J1zTJvAwO6Sfxl UTC8EBikd0S8paAe+0Bqn7n196o8tRuI4jA6CusO1y42WcxsnsluBbd3w5CtPshK/v2o3aShXxb 9fXQ9atkgYzV6GWgb0+QfN5IVHqQV4EWMSS1xlAUnSiCZHFru00REjlMtmsjcAUuzY2BCCfGoCK Z6Izv1iad6Sdl0aR+w== X-Google-Smtp-Source: AGHT+IHq2P1ROpgHywWa5M8Xpzdi4NwWEFRypribFsvZL+hUr50EoXZA3FxWT+dt8VM20Q3kk3TokQ== X-Received: by 2002:a17:903:2f86:b0:216:2259:a4bd with SMTP id d9443c01a7336-220bbca3a98mr26603225ad.52.1739341931537; Tue, 11 Feb 2025 22:32:11 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:69f5:6852:451e:8142]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-21f93c7e363sm48246805ad.94.2025.02.11.22.32.09 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 11 Feb 2025 22:32:11 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Kairui Song , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v5 02/18] zram: permit preemption with active compression stream Date: Wed, 12 Feb 2025 15:27:00 +0900 Message-ID: <20250212063153.179231-3-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.502.g6dc24dfdaf-goog In-Reply-To: <20250212063153.179231-1-senozhatsky@chromium.org> References: <20250212063153.179231-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Currently, per-CPU stream access is done from a non-preemptible (atomic) section, which imposes the same atomicity requirements on compression backends as entry spin-lock, and makes it impossible to use algorithms that can schedule/wait/sleep during compression and decompression. Switch to preemptible per-CPU model, similar to the one used in zswap. Instead of a per-CPU local lock, each stream carries a mutex which is locked throughout entire time zram uses it for compression or decompression, so that cpu-dead event waits for zram to stop using a particular per-CPU stream and release it. Suggested-by: Yosry Ahmed Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zcomp.c | 36 +++++++++++++++++++++++++---------- drivers/block/zram/zcomp.h | 6 +++--- drivers/block/zram/zram_drv.c | 20 +++++++++---------- 3 files changed, 39 insertions(+), 23 deletions(-) diff --git a/drivers/block/zram/zcomp.c b/drivers/block/zram/zcomp.c index bb514403e305..e83dd9a80a81 100644 --- a/drivers/block/zram/zcomp.c +++ b/drivers/block/zram/zcomp.c @@ -7,6 +7,7 @@ #include #include #include +#include #include #include =20 @@ -54,6 +55,7 @@ static int zcomp_strm_init(struct zcomp *comp, struct zco= mp_strm *zstrm) { int ret; =20 + mutex_init(&zstrm->lock); ret =3D comp->ops->create_ctx(comp->params, &zstrm->ctx); if (ret) return ret; @@ -109,13 +111,29 @@ ssize_t zcomp_available_show(const char *comp, char *= buf) =20 struct zcomp_strm *zcomp_stream_get(struct zcomp *comp) { - local_lock(&comp->stream->lock); - return this_cpu_ptr(comp->stream); + for (;;) { + struct zcomp_strm *zstrm =3D raw_cpu_ptr(comp->stream); + + /* + * Inspired by zswap + * + * stream is returned with ->mutex locked which prevents + * cpu_dead() from releasing this stream under us, however + * there is still a race window between raw_cpu_ptr() and + * mutex_lock(), during which we could have been migrated + * to a CPU that has already destroyed its stream. If so + * then unlock and re-try on the current CPU. + */ + mutex_lock(&zstrm->lock); + if (likely(zstrm->buffer)) + return zstrm; + mutex_unlock(&zstrm->lock); + } } =20 -void zcomp_stream_put(struct zcomp *comp) +void zcomp_stream_put(struct zcomp_strm *zstrm) { - local_unlock(&comp->stream->lock); + mutex_unlock(&zstrm->lock); } =20 int zcomp_compress(struct zcomp *comp, struct zcomp_strm *zstrm, @@ -151,12 +169,9 @@ int zcomp_decompress(struct zcomp *comp, struct zcomp_= strm *zstrm, int zcomp_cpu_up_prepare(unsigned int cpu, struct hlist_node *node) { struct zcomp *comp =3D hlist_entry(node, struct zcomp, node); - struct zcomp_strm *zstrm; + struct zcomp_strm *zstrm =3D per_cpu_ptr(comp->stream, cpu); int ret; =20 - zstrm =3D per_cpu_ptr(comp->stream, cpu); - local_lock_init(&zstrm->lock); - ret =3D zcomp_strm_init(comp, zstrm); if (ret) pr_err("Can't allocate a compression stream\n"); @@ -166,10 +181,11 @@ int zcomp_cpu_up_prepare(unsigned int cpu, struct hli= st_node *node) int zcomp_cpu_dead(unsigned int cpu, struct hlist_node *node) { struct zcomp *comp =3D hlist_entry(node, struct zcomp, node); - struct zcomp_strm *zstrm; + struct zcomp_strm *zstrm =3D per_cpu_ptr(comp->stream, cpu); =20 - zstrm =3D per_cpu_ptr(comp->stream, cpu); + mutex_lock(&zstrm->lock); zcomp_strm_free(comp, zstrm); + mutex_unlock(&zstrm->lock); return 0; } =20 diff --git a/drivers/block/zram/zcomp.h b/drivers/block/zram/zcomp.h index ad5762813842..23b8236b9090 100644 --- a/drivers/block/zram/zcomp.h +++ b/drivers/block/zram/zcomp.h @@ -3,7 +3,7 @@ #ifndef _ZCOMP_H_ #define _ZCOMP_H_ =20 -#include +#include =20 #define ZCOMP_PARAM_NO_LEVEL INT_MIN =20 @@ -31,7 +31,7 @@ struct zcomp_ctx { }; =20 struct zcomp_strm { - local_lock_t lock; + struct mutex lock; /* compression buffer */ void *buffer; struct zcomp_ctx ctx; @@ -77,7 +77,7 @@ struct zcomp *zcomp_create(const char *alg, struct zcomp_= params *params); void zcomp_destroy(struct zcomp *comp); =20 struct zcomp_strm *zcomp_stream_get(struct zcomp *comp); -void zcomp_stream_put(struct zcomp *comp); +void zcomp_stream_put(struct zcomp_strm *zstrm); =20 int zcomp_compress(struct zcomp *comp, struct zcomp_strm *zstrm, const void *src, unsigned int *dst_len); diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 3708436f1d1f..43f460a45e3e 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1608,7 +1608,7 @@ static int read_compressed_page(struct zram *zram, st= ruct page *page, u32 index) ret =3D zcomp_decompress(zram->comps[prio], zstrm, src, size, dst); kunmap_local(dst); zs_unmap_object(zram->mem_pool, handle); - zcomp_stream_put(zram->comps[prio]); + zcomp_stream_put(zstrm); =20 return ret; } @@ -1769,14 +1769,14 @@ static int zram_write_page(struct zram *zram, struc= t page *page, u32 index) kunmap_local(mem); =20 if (unlikely(ret)) { - zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); + zcomp_stream_put(zstrm); pr_err("Compression failed! err=3D%d\n", ret); zs_free(zram->mem_pool, handle); return ret; } =20 if (comp_len >=3D huge_class_size) { - zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); + zcomp_stream_put(zstrm); return write_incompressible_page(zram, page, index); } =20 @@ -1800,7 +1800,7 @@ static int zram_write_page(struct zram *zram, struct = page *page, u32 index) __GFP_HIGHMEM | __GFP_MOVABLE); if (IS_ERR_VALUE(handle)) { - zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); + zcomp_stream_put(zstrm); atomic64_inc(&zram->stats.writestall); handle =3D zs_malloc(zram->mem_pool, comp_len, GFP_NOIO | __GFP_HIGHMEM | @@ -1812,7 +1812,7 @@ static int zram_write_page(struct zram *zram, struct = page *page, u32 index) } =20 if (!zram_can_store_page(zram)) { - zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); + zcomp_stream_put(zstrm); zs_free(zram->mem_pool, handle); return -ENOMEM; } @@ -1820,7 +1820,7 @@ static int zram_write_page(struct zram *zram, struct = page *page, u32 index) dst =3D zs_map_object(zram->mem_pool, handle, ZS_MM_WO); =20 memcpy(dst, zstrm->buffer, comp_len); - zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); + zcomp_stream_put(zstrm); zs_unmap_object(zram->mem_pool, handle); =20 zram_slot_lock(zram, index); @@ -1979,7 +1979,7 @@ static int recompress_slot(struct zram *zram, u32 ind= ex, struct page *page, kunmap_local(src); =20 if (ret) { - zcomp_stream_put(zram->comps[prio]); + zcomp_stream_put(zstrm); return ret; } =20 @@ -1989,7 +1989,7 @@ static int recompress_slot(struct zram *zram, u32 ind= ex, struct page *page, /* Continue until we make progress */ if (class_index_new >=3D class_index_old || (threshold && comp_len_new >=3D threshold)) { - zcomp_stream_put(zram->comps[prio]); + zcomp_stream_put(zstrm); continue; } =20 @@ -2047,13 +2047,13 @@ static int recompress_slot(struct zram *zram, u32 i= ndex, struct page *page, __GFP_HIGHMEM | __GFP_MOVABLE); if (IS_ERR_VALUE(handle_new)) { - zcomp_stream_put(zram->comps[prio]); + zcomp_stream_put(zstrm); return PTR_ERR((void *)handle_new); } =20 dst =3D zs_map_object(zram->mem_pool, handle_new, ZS_MM_WO); memcpy(dst, zstrm->buffer, comp_len_new); - zcomp_stream_put(zram->comps[prio]); + zcomp_stream_put(zstrm); =20 zs_unmap_object(zram->mem_pool, handle_new); =20 --=20 2.48.1.502.g6dc24dfdaf-goog From nobody Thu Dec 18 10:00:39 2025 Received: from mail-pj1-f51.google.com (mail-pj1-f51.google.com [209.85.216.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D1C631FDE29 for ; Wed, 12 Feb 2025 06:32:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.51 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739341939; cv=none; b=mNXMWdcypE8x1dOJZc5bwqYUIJMJ3SJtBk3XglLB5uvuNSlXDt5+y1CBkMhhkewULD/NkrtQWSGuh2YC8WlG0uDheEbOsAvT8Pf5XAtXJ+ZkGNG/f+8ETBLH1JWAnZIedOrEO9dpXcR2APUXbR362CzY3F+smVQj11c5exVo6i4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739341939; c=relaxed/simple; bh=Z+RlvhbxEtZbFxPm2l0VEwmqX4Kk5+EzLmT2i1VTvyk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=mwmb3ud5GOXdTVd9qM/cOoSAtDXPEKoA9os74UfvJm/f6T6VKoAU/9ggjCmegEl/s/qwn0QHW4eoKoWkPG1KvAxJB9MC4TmTsoBGJk/2XYZkOZ2G5JP3SWo6j5tegEPSMmRW5HgDlPVsXunNyy7P8MRstZpeXO39+I3GPzge/8o= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=JfBRkeHv; arc=none smtp.client-ip=209.85.216.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="JfBRkeHv" Received: by mail-pj1-f51.google.com with SMTP id 98e67ed59e1d1-2f44353649aso9489449a91.0 for ; Tue, 11 Feb 2025 22:32:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1739341937; x=1739946737; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=48U7icBiaj5vHELEBDYpmRQUsi6H+QG8AACYxOChctw=; b=JfBRkeHv9t+mAGqAMgsJy12nYOvVMzEF+Y49mfanBlv3QJqiXC4AEw9VGBSspNLfpx 8x2TEEt9wOK+vBByyTJ+LUxuNULB1ta7zJta4ZbPx6JB374oXTtLjHjMlZPpoIM7tswu 2szBANudApB84b0fnNLgj3AwxuAi/b3iu2ass= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739341937; x=1739946737; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=48U7icBiaj5vHELEBDYpmRQUsi6H+QG8AACYxOChctw=; b=pb8aClmbTYhi3TSlg6vBD2qoAaWHQBXP6B6+9qD5ZqQf69lTJFCvj/gG8Iw+jzrkCw c4PT0i9mTflothwiRkr2w/xGCrOI3hYx1AmrFpxbfKkrzs637ZVz6yyjFQpqJbWMzBrF 7hNG0lQtJcAZYOYG0QfOdBpZpz7vMge1P83n2uuYGU+Km1bFbDJWVywvS9RT95T0zwjg NQGpOKhTYyvlEedGuestQIBnXs3hHm+SHdxLy39JS8QUocfcGFmmqvyrEgXq3Y4vOeFo 2SwLEBKkD8RLjdV+wyuC/R/YcnnXAuvlRAzweJic9+WDQc8QCSgwqcnVjFE2ci0b5Rpl z2Kw== X-Forwarded-Encrypted: i=1; AJvYcCXK5NW1IhJaTkLiOwfUUsEcJC/WvGcOb4GdT9EkRFRMYddKFbabtrNdxLyg1Ckk/XgrSK4RKMktB38k9rM=@vger.kernel.org X-Gm-Message-State: AOJu0Yz4CDFiyuO09yGcFxD7yBDflY8Nwg1/zzzg8OFVpoCvceTazGLP iuMqX8sN2rczXBxfnlEEvw4UMiglgyf/oFZuH4Fu7rm/D8pu8HJn9XRJDXcxSg== X-Gm-Gg: ASbGncv/JzsE3qihxxoOtKJ3j3eprlKWlK8h6I5FerI6EcYSGgLGPclXdc1Gv9f2hvO EtZGZesqh+1WohHq/mtJTwHEDwtqgkmUFBK4wWMn+5uRl05MDouePlvr2myoCXf/nv2b1Y1l0EQ fDfvCAXflSqh8PB1mlod8kELi0QKPpF4wdR6C2e2J/thxxjZgG6srx3QscVogmy+5jt+MEnZGa8 YkOU+IRdby2+d/uYRAow+p2EcooZyUpDm1K6a43FWmhZ9l76TbOA2fVGVaf/gGAasGDkTMxcBxJ 3QHJliUBEvg6dQyztg== X-Google-Smtp-Source: AGHT+IGlZ7sjxw8CKEDe37B21DbU3g+Tt4XgUZOISnGOFp7x2JKzUfH/1KMDV9dAFdU8C0jsA6InIg== X-Received: by 2002:a05:6a00:1c8e:b0:730:9659:ff4b with SMTP id d2e1a72fcca58-7322c5febedmr3482199b3a.19.1739341937213; Tue, 11 Feb 2025 22:32:17 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:69f5:6852:451e:8142]) by smtp.gmail.com with UTF8SMTPSA id d2e1a72fcca58-73048ae7c24sm10384533b3a.78.2025.02.11.22.32.15 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 11 Feb 2025 22:32:16 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Kairui Song , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v5 03/18] zram: remove crypto include Date: Wed, 12 Feb 2025 15:27:01 +0900 Message-ID: <20250212063153.179231-4-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.502.g6dc24dfdaf-goog In-Reply-To: <20250212063153.179231-1-senozhatsky@chromium.org> References: <20250212063153.179231-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Remove a leftover crypto header include. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zcomp.c | 1 - drivers/block/zram/zram_drv.c | 4 +++- drivers/block/zram/zram_drv.h | 1 - 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/drivers/block/zram/zcomp.c b/drivers/block/zram/zcomp.c index e83dd9a80a81..c393243eeb5c 100644 --- a/drivers/block/zram/zcomp.c +++ b/drivers/block/zram/zcomp.c @@ -8,7 +8,6 @@ #include #include #include -#include #include =20 #include "zcomp.h" diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 43f460a45e3e..12fb260e3355 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -44,6 +44,8 @@ static DEFINE_MUTEX(zram_index_mutex); static int zram_major; static const char *default_compressor =3D CONFIG_ZRAM_DEF_COMP; =20 +#define ZRAM_MAX_ALGO_NAME_SZ 64 + /* Module params (documentation at end) */ static unsigned int num_devices =3D 1; /* @@ -1149,7 +1151,7 @@ static int __comp_algorithm_store(struct zram *zram, = u32 prio, const char *buf) size_t sz; =20 sz =3D strlen(buf); - if (sz >=3D CRYPTO_MAX_ALG_NAME) + if (sz >=3D ZRAM_MAX_ALGO_NAME_SZ) return -E2BIG; =20 compressor =3D kstrdup(buf, GFP_KERNEL); diff --git a/drivers/block/zram/zram_drv.h b/drivers/block/zram/zram_drv.h index 63b933059cb6..97c98fa07954 100644 --- a/drivers/block/zram/zram_drv.h +++ b/drivers/block/zram/zram_drv.h @@ -17,7 +17,6 @@ =20 #include #include -#include =20 #include "zcomp.h" =20 --=20 2.48.1.502.g6dc24dfdaf-goog From nobody Thu Dec 18 10:00:39 2025 Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 638091FF1AF for ; Wed, 12 Feb 2025 06:32:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.180 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739341945; cv=none; b=WIsrSgoWCbBdgM6tWQca3oHmSR6wMjUwaLQG6Xw3iIh4X3rG4kH5TtStwde1Vu61C601OaA2muyNYUYw1swvCHcFWeov6GQUHLaDreGAFmWQzUC2sWd2vahU6NdmCzMpYCanfFvnaZzee58H4gr4QxPw7KKjxWKH96qQpzG3vdA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739341945; c=relaxed/simple; bh=XskcwS482bSYecJNApb530j9dirKULlpuwXW4GKTrhI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=I7B3cJcXoen7Ovn6wdc5HWrZIqAG4XP9u8EGQaQFycg3II+MEPRhMyuR8Trojfa/bV9V27OQIyBiZ20cu72sgZE22NsNmBh1WvDlK3qcVRkVA4OD/pgihgBPi5FkMA89UhcUZZfEThX4WDJTAKTYfk5j3BP07fw/M4ARMSUqdzg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=fqZ2zMxc; arc=none smtp.client-ip=209.85.214.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="fqZ2zMxc" Received: by mail-pl1-f180.google.com with SMTP id d9443c01a7336-21f61b01630so7532355ad.1 for ; Tue, 11 Feb 2025 22:32:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1739341942; x=1739946742; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=h2AQA5cUglzG6yhVm3aKj+975OMRekO7mg4bKTRqdCs=; b=fqZ2zMxcSxnq+hZyc3HER6NTxk+sFsjMtpjbyy7EFFiwRgh0jyMHTdPr0jaXp6sFaz XrT2/bJ9b/J/pHnSaoK45CesJgytcgViXmI6muqDPoy/AfhPM0cte6x5FJJM8e6qufGe Y762nI2WwedmFWH+fOAXXq+wVyneJx2enbPCU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739341942; x=1739946742; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=h2AQA5cUglzG6yhVm3aKj+975OMRekO7mg4bKTRqdCs=; b=lWqO31seRhoCmXL3sxv66lpwIHxbWOFQV2HfASrikgDPFCXx8UMTtkJlgHzYaboJk0 qsIvL+zBhu1rV+FvpopTIjA9XfrhMZL7QTOVCzqc4TZd4w15a+TGmvxzjfAOjGZA0ZS6 /+IVokm0+lSNsXOHrHr/1Xph77PcAFK1bJajb81qA3fyP2x18GIB1PyLR2kjNDzaZuxi b9ewOoB4drfxeaYPRc3morviE/CLH/F37ZUEGxlFeNixqVqoqmivw5QIFEhl5B63IjXA VDvkTKzGk+CiywYjJsiHXKsEhgY0Kb7dppWdcGXhrGS0p4REgPfnu/ouoLEJUucxUJ2g u/AQ== X-Forwarded-Encrypted: i=1; AJvYcCUxR4qjAsVh1HJRmKJm+//8VLDz2cn0QjX7YujsloGVrXJ3GyjmoEtsE/dTURdR54OXj2U5/ipSXy40cmc=@vger.kernel.org X-Gm-Message-State: AOJu0Yxmi9TFtf9DNQE4/jELZ3XPVMOaRV34mv+oUlWXRBMlBgD1oamu 2+n8S4WjROpGUGT2V+AFI1TKvXMBW6zDhabXdckRY08LTU1hi5WqVu+YME/ofA== X-Gm-Gg: ASbGnctP3pEVyXJBbgGRuJFCeeZm9+PDh8iO0b+AwbgyThQSGV+EtqoIl2oNHjnbWml /d7TILwBf9g8a6cXq2TWlSIe6kGV3QoZeXi2TqgufIM0ORdyEbqjcVI4Q0WJqDpHaI4LHh0lYek lnlJb2SMqli5OAZOCOpc0enm+UVhu2SGH8g9/AKCeji5hPblu7hWte3cF04THMnQXA/x8yKgkhV JWX59FHJX1e+OVUqAEXP3QNr6+uzO13ipQ2Qt2lnEX0F7SJkre1aYI9PhicZ2GKVygDi0wPfFhr VXfNugWH5YgrfJq6Lw== X-Google-Smtp-Source: AGHT+IFvCaT4j3C930RCMRawLzcITwesvr3FnYejPnLHFg8E648OUmGxFSWdbEBwV5fB9AnLznY3OA== X-Received: by 2002:a05:6a21:398f:b0:1d9:a94:feec with SMTP id adf61e73a8af0-1ee5cea8113mr3339152637.2.1739341942625; Tue, 11 Feb 2025 22:32:22 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:69f5:6852:451e:8142]) by smtp.gmail.com with UTF8SMTPSA id d2e1a72fcca58-730918f6ec7sm4475974b3a.37.2025.02.11.22.32.20 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 11 Feb 2025 22:32:22 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Kairui Song , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v5 04/18] zram: remove max_comp_streams device attr Date: Wed, 12 Feb 2025 15:27:02 +0900 Message-ID: <20250212063153.179231-5-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.502.g6dc24dfdaf-goog In-Reply-To: <20250212063153.179231-1-senozhatsky@chromium.org> References: <20250212063153.179231-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" max_comp_streams device attribute has been defunct since May 2016 when zram switched to per-CPU compression streams, remove it. Signed-off-by: Sergey Senozhatsky --- Documentation/ABI/testing/sysfs-block-zram | 8 ----- Documentation/admin-guide/blockdev/zram.rst | 36 ++++++--------------- drivers/block/zram/zram_drv.c | 23 ------------- 3 files changed, 10 insertions(+), 57 deletions(-) diff --git a/Documentation/ABI/testing/sysfs-block-zram b/Documentation/ABI= /testing/sysfs-block-zram index 1ef69e0271f9..36c57de0a10a 100644 --- a/Documentation/ABI/testing/sysfs-block-zram +++ b/Documentation/ABI/testing/sysfs-block-zram @@ -22,14 +22,6 @@ Description: device. The reset operation frees all the memory associated with this device. =20 -What: /sys/block/zram/max_comp_streams -Date: February 2014 -Contact: Sergey Senozhatsky -Description: - The max_comp_streams file is read-write and specifies the - number of backend's zcomp_strm compression streams (number of - concurrent compress operations). - What: /sys/block/zram/comp_algorithm Date: February 2014 Contact: Sergey Senozhatsky diff --git a/Documentation/admin-guide/blockdev/zram.rst b/Documentation/ad= min-guide/blockdev/zram.rst index 1576fb93f06c..9bdb30901a93 100644 --- a/Documentation/admin-guide/blockdev/zram.rst +++ b/Documentation/admin-guide/blockdev/zram.rst @@ -54,7 +54,7 @@ The list of possible return codes: If you use 'echo', the returned value is set by the 'echo' utility, and, in general case, something like:: =20 - echo 3 > /sys/block/zram0/max_comp_streams + echo foo > /sys/block/zram0/comp_algorithm if [ $? -ne 0 ]; then handle_error fi @@ -73,21 +73,7 @@ This creates 4 devices: /dev/zram{0,1,2,3} num_devices parameter is optional and tells zram how many devices should be pre-created. Default: 1. =20 -2) Set max number of compression streams -=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D - -Regardless of the value passed to this attribute, ZRAM will always -allocate multiple compression streams - one per online CPU - thus -allowing several concurrent compression operations. The number of -allocated compression streams goes down when some of the CPUs -become offline. There is no single-compression-stream mode anymore, -unless you are running a UP system or have only 1 CPU online. - -To find out how many streams are currently available:: - - cat /sys/block/zram0/max_comp_streams - -3) Select compression algorithm +2) Select compression algorithm =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D =20 Using comp_algorithm device attribute one can see available and @@ -107,7 +93,7 @@ Examples:: For the time being, the `comp_algorithm` content shows only compression algorithms that are supported by zram. =20 -4) Set compression algorithm parameters: Optional +3) Set compression algorithm parameters: Optional =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =20 Compression algorithms may support specific parameters which can be @@ -138,7 +124,7 @@ better the compression ratio, it even can take negative= s values for some algorithms), for other algorithms `level` is acceleration level (the higher the value the lower the compression ratio). =20 -5) Set Disksize +4) Set Disksize =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =20 Set disk size by writing the value to sysfs node 'disksize'. @@ -158,7 +144,7 @@ There is little point creating a zram of greater than t= wice the size of memory since we expect a 2:1 compression ratio. Note that zram uses about 0.1% of= the size of the disk when not in use so a huge zram is wasteful. =20 -6) Set memory limit: Optional +5) Set memory limit: Optional =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D =20 Set memory limit by writing the value to sysfs node 'mem_limit'. @@ -177,7 +163,7 @@ Examples:: # To disable memory limit echo 0 > /sys/block/zram0/mem_limit =20 -7) Activate +6) Activate =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =20 :: @@ -188,7 +174,7 @@ Examples:: mkfs.ext4 /dev/zram1 mount /dev/zram1 /tmp =20 -8) Add/remove zram devices +7) Add/remove zram devices =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D =20 zram provides a control interface, which enables dynamic (on-demand) device @@ -208,7 +194,7 @@ execute:: =20 echo X > /sys/class/zram-control/hot_remove =20 -9) Stats +8) Stats =3D=3D=3D=3D=3D=3D=3D=3D =20 Per-device statistics are exported as various nodes under /sys/block/zram<= id>/ @@ -228,8 +214,6 @@ mem_limit WO specifies the maximum amount of m= emory ZRAM can writeback_limit WO specifies the maximum amount of write IO zram can write out to backing device as 4KB unit writeback_limit_enable RW show and set writeback_limit feature -max_comp_streams RW the number of possible concurrent compress - operations comp_algorithm RW show and change the compression algorithm algorithm_params WO setup compression algorithm parameters compact WO trigger memory compaction @@ -310,7 +294,7 @@ a single line of text and contains the following stats = separated by whitespace: Unit: 4K bytes =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D =20 -10) Deactivate +9) Deactivate =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =20 :: @@ -318,7 +302,7 @@ a single line of text and contains the following stats = separated by whitespace: swapoff /dev/zram0 umount /dev/zram1 =20 -11) Reset +10) Reset =3D=3D=3D=3D=3D=3D=3D=3D=3D =20 Write any positive value to 'reset' sysfs node:: diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 12fb260e3355..e0e64b2610d6 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1104,27 +1104,6 @@ static void zram_debugfs_register(struct zram *zram)= {}; static void zram_debugfs_unregister(struct zram *zram) {}; #endif =20 -/* - * We switched to per-cpu streams and this attr is not needed anymore. - * However, we will keep it around for some time, because: - * a) we may revert per-cpu streams in the future - * b) it's visible to user space and we need to follow our 2 years - * retirement rule; but we already have a number of 'soon to be - * altered' attrs, so max_comp_streams need to wait for the next - * layoff cycle. - */ -static ssize_t max_comp_streams_show(struct device *dev, - struct device_attribute *attr, char *buf) -{ - return scnprintf(buf, PAGE_SIZE, "%d\n", num_online_cpus()); -} - -static ssize_t max_comp_streams_store(struct device *dev, - struct device_attribute *attr, const char *buf, size_t len) -{ - return len; -} - static void comp_algorithm_set(struct zram *zram, u32 prio, const char *al= g) { /* Do not free statically defined compression algorithms */ @@ -2541,7 +2520,6 @@ static DEVICE_ATTR_WO(reset); static DEVICE_ATTR_WO(mem_limit); static DEVICE_ATTR_WO(mem_used_max); static DEVICE_ATTR_WO(idle); -static DEVICE_ATTR_RW(max_comp_streams); static DEVICE_ATTR_RW(comp_algorithm); #ifdef CONFIG_ZRAM_WRITEBACK static DEVICE_ATTR_RW(backing_dev); @@ -2563,7 +2541,6 @@ static struct attribute *zram_disk_attrs[] =3D { &dev_attr_mem_limit.attr, &dev_attr_mem_used_max.attr, &dev_attr_idle.attr, - &dev_attr_max_comp_streams.attr, &dev_attr_comp_algorithm.attr, #ifdef CONFIG_ZRAM_WRITEBACK &dev_attr_backing_dev.attr, --=20 2.48.1.502.g6dc24dfdaf-goog From nobody Thu Dec 18 10:00:39 2025 Received: from mail-pl1-f176.google.com (mail-pl1-f176.google.com [209.85.214.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C12D51FF7C8 for ; Wed, 12 Feb 2025 06:32:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739341950; cv=none; b=CH5Q3holIk7dFcwde4IKO1bVHL6KKFzph51P9PrQ2NukUDl0fy/AEZjy0LDFgpjPAtp+Q0ZHlGZ6nD595fJFl4DSeYVY+5L5BJm2RODTJaLcow9B/apyQwmf5K70J2lnwHlV652wqwNQfTAM/3vsvAgntfFnyO0XqnRF0o1KGAs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739341950; c=relaxed/simple; bh=NU4ULCQ2BpSp4FDI53G1gJZDMlupH3u9ZyEy7/VHAdM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ikdmvhSNAQ2KURPrU+9hKuz8mrhleA5rjkuP102tnqPq11RAHlSrx64uZIaEyBBWzJ5ExbLt/JxqC+Rk8HSX50iRrqmwXhhDSBcf+eU0LBpaLaOU2DWDN9mGZmXmGZbL1PsgGmDtueev4Utrc8TWtsn7I5aMrbfP385W7t51fSc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=OSdPdUJ2; arc=none smtp.client-ip=209.85.214.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="OSdPdUJ2" Received: by mail-pl1-f176.google.com with SMTP id d9443c01a7336-21f0444b478so90877465ad.0 for ; Tue, 11 Feb 2025 22:32:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1739341948; x=1739946748; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=QEMWhAClB1d5xqcLAEaUS1zNcFLgRU1myRFnr10zi+A=; b=OSdPdUJ2CFCyjT2Vrra0/zQRJKcxYsOshDV+gpdt9jvz5WcYJ0CGpf43vVlr7LDNPt ETRZ/uAbdj/6qTubF1AZg00GpemvsQNUCwkoqmNQawvkAeofua39nLhDLJLydxAj0/Pr RCyQnJAt6WZuZ3I0MW+ichPVX/FL0bkFDWU+k= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739341948; x=1739946748; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=QEMWhAClB1d5xqcLAEaUS1zNcFLgRU1myRFnr10zi+A=; b=n5xJHCp95zd0cdr5HIIbzzbv7t2YFXrGPJmeIsjqK2HF15n58TGNPZxiyVSBE/YqL8 hbieCsSWKZ9Lk6Wfd2a3ACxyNMRuw2zI4PQmGVCYuqOXVtybuXyqCcj6U1Z+7Q1smZx9 AyEobQFOkLb0PiSH+YKZeRXpvCQLXmc6uNB5/bIuK6QnfcnI/XNNgQsJsXlhDMR9xPMY R0xxIui7TRBaf2vsKitjhOovuBH7LimQwQJgWdqHmrenIjG4qq/aRaT2v2e67D5EsPuI 5YrRO+w43t7F+9t1iU0ayh6z12etQ4xbo5i2ArYNXuXlIFSBg7SqY8Bthiel72ltPda6 rEOQ== X-Forwarded-Encrypted: i=1; AJvYcCUy/mrIElbyexcEXomPb6c40AF6rRG1UEC9VigXizF6+i3W6FjztnlvYDpgqk4VX1oXqOWw5nCAb9N7haU=@vger.kernel.org X-Gm-Message-State: AOJu0YyiCD183piDflnJhyWl3PqK9QW5iBJH47U9RgufLWoDAWcto+MM D5AE9xSV56YkZYQEZF/gdMcT18GrNcU5kRLlHb6UEgSw7sgBwhk08iiex3H/ZA== X-Gm-Gg: ASbGncvMW5MV74msfo+q3HF8abzdA/XvqkiH/b/ijRwr5Vf/0wReDj8xSYBmy19IUmG W3qRICPRWE9STja28MSsePqOf2iRIy7QlTjrMBRZ3yuYnyFqEhBW1chHhhv6ubYuJ2kdyQ/twjY byg7w83JmIFOXmTEcV47WulZbWvfOgY1FpFxbi0DlgLmZzpmrYZixSh9eXI1znihlZJsY12IzUk pPJEORiCibwoushgW6+JiNQAUNnnIFywn5j4v6JGRM/MoRbgjMBhUh5C5Hc/2GzRFuxcqu2PtFb 7eyPKXiWBG9+DFUdJg== X-Google-Smtp-Source: AGHT+IFA+EnqkupBVK3/MEPCQX2ZjyTTOUvx22Z0kt6I03ZDDZCylIMnWUqaHODE62jxmfspsUlwkQ== X-Received: by 2002:a05:6a00:804:b0:728:e2cc:bfd6 with SMTP id d2e1a72fcca58-7322c43107bmr3332267b3a.18.1739341948039; Tue, 11 Feb 2025 22:32:28 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:69f5:6852:451e:8142]) by smtp.gmail.com with UTF8SMTPSA id d2e1a72fcca58-730862e72a4sm5749487b3a.103.2025.02.11.22.32.25 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 11 Feb 2025 22:32:27 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Kairui Song , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v5 05/18] zram: remove two-staged handle allocation Date: Wed, 12 Feb 2025 15:27:03 +0900 Message-ID: <20250212063153.179231-6-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.502.g6dc24dfdaf-goog In-Reply-To: <20250212063153.179231-1-senozhatsky@chromium.org> References: <20250212063153.179231-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Previously zram write() was atomic which required us to pass __GFP_KSWAPD_RECLAIM to zsmalloc handle allocation on a fast path and attempt a slow path allocation (with recompression) when the fast path failed. Since it's not atomic anymore we can permit direct reclaim during allocation, and remove fast allocation path and, also, drop the recompression path (which should reduce CPU/battery usage). Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 38 ++++++----------------------------- 1 file changed, 6 insertions(+), 32 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index e0e64b2610d6..6384c61c03bf 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1724,11 +1724,11 @@ static int write_incompressible_page(struct zram *z= ram, struct page *page, static int zram_write_page(struct zram *zram, struct page *page, u32 index) { int ret =3D 0; - unsigned long handle =3D -ENOMEM; - unsigned int comp_len =3D 0; + unsigned long handle; + unsigned int comp_len; void *dst, *mem; struct zcomp_strm *zstrm; - unsigned long element =3D 0; + unsigned long element; bool same_filled; =20 /* First, free memory allocated to this slot (if any) */ @@ -1742,7 +1742,6 @@ static int zram_write_page(struct zram *zram, struct = page *page, u32 index) if (same_filled) return write_same_filled_page(zram, element, index); =20 -compress_again: zstrm =3D zcomp_stream_get(zram->comps[ZRAM_PRIMARY_COMP]); mem =3D kmap_local_page(page); ret =3D zcomp_compress(zram->comps[ZRAM_PRIMARY_COMP], zstrm, @@ -1752,7 +1751,6 @@ static int zram_write_page(struct zram *zram, struct = page *page, u32 index) if (unlikely(ret)) { zcomp_stream_put(zstrm); pr_err("Compression failed! err=3D%d\n", ret); - zs_free(zram->mem_pool, handle); return ret; } =20 @@ -1761,35 +1759,11 @@ static int zram_write_page(struct zram *zram, struc= t page *page, u32 index) return write_incompressible_page(zram, page, index); } =20 - /* - * handle allocation has 2 paths: - * a) fast path is executed with preemption disabled (for - * per-cpu streams) and has __GFP_DIRECT_RECLAIM bit clear, - * since we can't sleep; - * b) slow path enables preemption and attempts to allocate - * the page with __GFP_DIRECT_RECLAIM bit set. we have to - * put per-cpu compression stream and, thus, to re-do - * the compression once handle is allocated. - * - * if we have a 'non-null' handle here then we are coming - * from the slow path and handle has already been allocated. - */ - if (IS_ERR_VALUE(handle)) - handle =3D zs_malloc(zram->mem_pool, comp_len, - __GFP_KSWAPD_RECLAIM | - __GFP_NOWARN | - __GFP_HIGHMEM | - __GFP_MOVABLE); + handle =3D zs_malloc(zram->mem_pool, comp_len, + GFP_NOIO | __GFP_HIGHMEM | __GFP_MOVABLE); if (IS_ERR_VALUE(handle)) { zcomp_stream_put(zstrm); - atomic64_inc(&zram->stats.writestall); - handle =3D zs_malloc(zram->mem_pool, comp_len, - GFP_NOIO | __GFP_HIGHMEM | - __GFP_MOVABLE); - if (IS_ERR_VALUE(handle)) - return PTR_ERR((void *)handle); - - goto compress_again; + return PTR_ERR((void *)handle); } =20 if (!zram_can_store_page(zram)) { --=20 2.48.1.502.g6dc24dfdaf-goog From nobody Thu Dec 18 10:00:39 2025 Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7F4191FFC56 for ; Wed, 12 Feb 2025 06:32:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739341954; cv=none; b=L1VHRwzYN/3ThZjEzP8uycGGjmYVypafr5lQDi9yIOTZz0QaHOVs+GkdNbr9oUW11hhsIkjL0uY0BfvMfeO2Rv+kgMPe4Hsp3bEGk5fYg/7d8/EosMkBBNLfZppURs6nFFIJyulinT0myAKUNmFtN86NaGE8LPhGpd11NH43de0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739341954; c=relaxed/simple; bh=flolTbXXi8MQLaDPN1IzgsuIaefgJ6R0mz4aEpsIhfs=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=tknjJtNoypuncQ/FIjst3kyGmhQqyR21UAOo1OBOJitQkrhSknLN4sKmatixyxeonZME+ceC5acxVEDhHjv4bI0Z/zpglSBLmKQ+Y3IGBuGBRb0jakr0GsKB9FLMCgcKbLjlxu69rFEwcpfTkv2kf93UWL1XHFpE6fyGh7K7wuY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=WdmWqewW; arc=none smtp.client-ip=209.85.214.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="WdmWqewW" Received: by mail-pl1-f182.google.com with SMTP id d9443c01a7336-21f44e7eae4so108456955ad.2 for ; Tue, 11 Feb 2025 22:32:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1739341953; x=1739946753; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=MP/iUhuHVLA6wFxJBrfyM09o41KGkhO8ufnVXdUQ+KM=; b=WdmWqewWQFD8WzD3YrNhqwM1iuPxwPgB4zInTLZhqrQxplacX/sETaJRQW++hUmz3G QOyfDWDi62bojWeR9hrV6/5nOlNEoYveDGpPVzy3vyTSfVPclusVXKJMVQQl7P2cQDKi Pf3Y+oABG85TR8/F4SbjUflsA7e5ZUoiMMfB4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739341953; x=1739946753; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=MP/iUhuHVLA6wFxJBrfyM09o41KGkhO8ufnVXdUQ+KM=; b=Onz+LqWNzdszf61Q3Ux2/8qFrzbdUgtwkP+EutS6wXV67SmADzqAOXUfig2tbOmYax 71q7kXCedf5zJefu0Ytm8Yxahv59wnF7wjIS5jmf1HnsacuY+E8U5xqZEHWNH9aUuBCh cI/9KsERo0JfB5wGka4PpSBDSHqa3xBhpnEP8oXOa+PlZcD0aAmNRX/6uiL18N+r8awE FixTuEJTJSbsauTeRLHOyVswjLZ9ZjUNH2oiJ5ozo55v19Z+jn+QuSZaidJRUawXXa0j jzBsxZcGVfoUI4L5UATkJQ7cTuYx9tgmDXeTDQtBDalYmo0oh6ofSof6tCrib6kacV78 Y4Ng== X-Forwarded-Encrypted: i=1; AJvYcCVZmbC5Q1MoGwKxNMW/Txj1aetPMT1IPurtcDkwhSWs10mUdMgdiSLgVAsLAWLeWLgkXJr10GCeaFavv9U=@vger.kernel.org X-Gm-Message-State: AOJu0YzsU4N/fh4oeMBLIr2R25YNaGZM+fstPz2vrh4lIRODqjyqqlld LVcgPRiiR9tmEy9AZ0ODQmaHxJGJxttf1yVdmQ36MowyrMKvFwthjFV8nuBzdg== X-Gm-Gg: ASbGnctyZGzfHRd4myjzSX82JSMsxmB5/sqeWXYgSLmeqLfy7b6epCS1JwHFTGZ2EG2 CWjpZUOwk5W/D+q7XMZ7nOOnDCC8fsUTTnS68OcX6uBxx7nUhlEHU8VVUmS2a6D3pOjgDejM4f1 WTXoYTRK5asU5XQ/bGTLVsXzV0EWVrJdvlXeza4v94jPNz3PvB0Wj8TDe6TTSmc78UPZt3Fwa69 VePOrKC+0zbgwxaI2tReiqZ6I22Rhkdg4sbRZMEf1GhP197MKJHpHsgPPdVroDJf/7ImLcbTsLb dM+AONwbSc8jYJ3Ibw== X-Google-Smtp-Source: AGHT+IE7SyVtE81IRMZ7QuZhJlVL0CH3JofgG8lKolY4V/dbLjW0GVG8rwvz0sFKRCNuv1wj7rDucg== X-Received: by 2002:a05:6a21:6b17:b0:1cf:27bf:8e03 with SMTP id adf61e73a8af0-1ee5c840a69mr4005891637.26.1739341952903; Tue, 11 Feb 2025 22:32:32 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:69f5:6852:451e:8142]) by smtp.gmail.com with UTF8SMTPSA id d2e1a72fcca58-73048ae7f6esm10423017b3a.74.2025.02.11.22.32.30 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 11 Feb 2025 22:32:32 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Kairui Song , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v5 06/18] zram: remove writestall zram_stats member Date: Wed, 12 Feb 2025 15:27:04 +0900 Message-ID: <20250212063153.179231-7-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.502.g6dc24dfdaf-goog In-Reply-To: <20250212063153.179231-1-senozhatsky@chromium.org> References: <20250212063153.179231-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" There is no zsmalloc handle allocation slow path now and writestall is not possible any longer. Remove it from zram_stats. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 3 +-- drivers/block/zram/zram_drv.h | 1 - 2 files changed, 1 insertion(+), 3 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 6384c61c03bf..7e2694079760 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1438,9 +1438,8 @@ static ssize_t debug_stat_show(struct device *dev, =20 down_read(&zram->init_lock); ret =3D scnprintf(buf, PAGE_SIZE, - "version: %d\n%8llu %8llu\n", + "version: %d\n0 %8llu\n", version, - (u64)atomic64_read(&zram->stats.writestall), (u64)atomic64_read(&zram->stats.miss_free)); up_read(&zram->init_lock); =20 diff --git a/drivers/block/zram/zram_drv.h b/drivers/block/zram/zram_drv.h index 97c98fa07954..b9528a62521e 100644 --- a/drivers/block/zram/zram_drv.h +++ b/drivers/block/zram/zram_drv.h @@ -84,7 +84,6 @@ struct zram_stats { atomic64_t huge_pages_since; /* no. of huge pages since zram set up */ atomic64_t pages_stored; /* no. of pages currently stored */ atomic_long_t max_used_pages; /* no. of maximum pages stored */ - atomic64_t writestall; /* no. of write slow paths */ atomic64_t miss_free; /* no. of missed free */ #ifdef CONFIG_ZRAM_WRITEBACK atomic64_t bd_count; /* no. of pages in backing device */ --=20 2.48.1.502.g6dc24dfdaf-goog From nobody Thu Dec 18 10:00:39 2025 Received: from mail-pl1-f170.google.com (mail-pl1-f170.google.com [209.85.214.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 617AE1FCFCC for ; Wed, 12 Feb 2025 06:32:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739341959; cv=none; b=r+AahBcHu84wu/AxSRQtnBliQG1XVY1tAx/tQVcJvYaL0mAHiNTe3G3ysIbTjg/zFko4XEUTPAWLy5+aHVyNTs0ewrUDOeJS2u4i3SWGGWTeJwax/h6JT323AmKplDUUM+ZrCSeZ9KQHLOfFowL+aamziHo5ixOqpWGX19rziAk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739341959; c=relaxed/simple; bh=LrH528wQy0dtsPj1OFdH0HvdanwSwv/enQHWZGzdxfI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=igzFdR7HzblImxwX/E4UZkZNQZqFE3yJTy0l3GTXWRqQMlQeEiGTetMZICz+k2WLTqiryIvAoy/ONe27LYJnesucTx3zmWDvxB6p1pbRIjTCEFv7GywyyE55mHJyswRooVRqcXVM/sP8Ccq2QOrtdYG2X1pzso6K9Z5Cb4WkWE0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=WqN0C7uF; arc=none smtp.client-ip=209.85.214.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="WqN0C7uF" Received: by mail-pl1-f170.google.com with SMTP id d9443c01a7336-21f61b01630so7534955ad.1 for ; Tue, 11 Feb 2025 22:32:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1739341957; x=1739946757; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=cyCqO8rpD053xgod9EGtthWwAT9KzKKgkQ9kmE/51BA=; b=WqN0C7uF9o/2m1wgBjwNybNGYsn4mB1vFXJQxgTDFFzwB33QiH4QkZyZuWFUpu4Ei8 CaE0kZi21iJ3VLgFPrqlpqZuvjf61Tblbu0NjaCAXm0j24R09UYTXb0W+Kkvr7gxlKxE Y36sk06Mxs8doGOiB9Cwx5muT1PpiASGrQCjw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739341957; x=1739946757; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=cyCqO8rpD053xgod9EGtthWwAT9KzKKgkQ9kmE/51BA=; b=D7S5PTiSj7HjvRbiO1zTBbofw5OowsoqcpMTvIO9xXYGXDwgrLtKK6BnK9ptt6fpH+ 0RpkklR9z05NeGnah/LQftvcv2AmlNe+/VzG1KY3cDkp/cRUGxO9yDlMJgU7T/Y90KRl ZUC8IjIMACipyUz1RnEg/SlLtcVRnkPyavTiPsY0uKJhpV6W6SbPuz9/8u37pZ+aPtYh ObufRbsqMMZ1ZXxKXnqlJJ6Q/URHOf6h+B+7M/2480CC0bMYw4UjeXB/46GCQlt2a3JG Q4tn2oLOG2ZbLTq2Y21hFvKVEtagPNpdlX4mLgA4DbaIzZBYq/67R1kd3OxDA7568som TaCA== X-Forwarded-Encrypted: i=1; AJvYcCWDfkWj6HTWRjSbP3l07j/llF2EZka1NCIsYXV6nf45Nal4ZcDb9sXDvyepqXr9fGVMfJvLiWJIxoUBbjU=@vger.kernel.org X-Gm-Message-State: AOJu0Ywzu/PGSxnpYbW7F0c2P2R2530Te9JCxl7QNW6Q45fdSUtv7yq6 iVCJKGyM098T/9kAIshhUk65MMNk9cRSYXyLvNe/d2o2ZLxWIhkyYbh5Yzvsgg== X-Gm-Gg: ASbGncvh9cOFiCDRxfikA6Uk7E3o2vidV9vqHZuzpcLvviiDJ6Ty0eo9U+YQBtkfqiz TjnTMNYnjTzq053Np3tAJCvaTIBLv80Eo6yDs8UnPx6tl2/pOy2a286oT4ZnatrzAbj8ktr4rPc yKAuXN1VRsHYTWC2uPaZWsFBQlEbstMX5HTmckM4Mb4IG6rCp0SrcewjUJNJHuheSdZS07O2k8q 2ipD0KLj5a4v/dlPLlkO9adde2jFRUpEbaq3yr/gjap6rWRTQXvlNXLpCYX/hwnV0YNQky+BY7O ssjicoFWfU48czV1KA== X-Google-Smtp-Source: AGHT+IEC8Cc8dAP8Ht+pVcdn6EfMAD6X2400PsIqNRPLot1jyPj+LyGyqkVdRWRCN4K6E/Zgz3CsHA== X-Received: by 2002:a05:6a21:150d:b0:1e0:d5be:bf75 with SMTP id adf61e73a8af0-1ee5cf4b707mr3005496637.17.1739341957604; Tue, 11 Feb 2025 22:32:37 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:69f5:6852:451e:8142]) by smtp.gmail.com with UTF8SMTPSA id 41be03b00d2f7-ad5489a2ddesm5668053a12.53.2025.02.11.22.32.35 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 11 Feb 2025 22:32:37 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Kairui Song , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v5 07/18] zram: limit max recompress prio to num_active_comps Date: Wed, 12 Feb 2025 15:27:05 +0900 Message-ID: <20250212063153.179231-8-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.502.g6dc24dfdaf-goog In-Reply-To: <20250212063153.179231-1-senozhatsky@chromium.org> References: <20250212063153.179231-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Use the actual number of algorithms zram was configure with instead of theoretical limit of ZRAM_MAX_COMPS. Also make sure that min prio is not above max prio. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 7e2694079760..27148f3e5ae9 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -2026,16 +2026,19 @@ static ssize_t recompress_store(struct device *dev, struct device_attribute *attr, const char *buf, size_t len) { - u32 prio =3D ZRAM_SECONDARY_COMP, prio_max =3D ZRAM_MAX_COMPS; struct zram *zram =3D dev_to_zram(dev); char *args, *param, *val, *algo =3D NULL; u64 num_recomp_pages =3D ULLONG_MAX; struct zram_pp_ctl *ctl =3D NULL; struct zram_pp_slot *pps; u32 mode =3D 0, threshold =3D 0; + u32 prio, prio_max; struct page *page; ssize_t ret; =20 + prio =3D ZRAM_SECONDARY_COMP; + prio_max =3D zram->num_active_comps; + args =3D skip_spaces(buf); while (*args) { args =3D next_arg(args, ¶m, &val); @@ -2088,7 +2091,7 @@ static ssize_t recompress_store(struct device *dev, if (prio =3D=3D ZRAM_PRIMARY_COMP) prio =3D ZRAM_SECONDARY_COMP; =20 - prio_max =3D min(prio + 1, ZRAM_MAX_COMPS); + prio_max =3D prio + 1; continue; } } @@ -2116,7 +2119,7 @@ static ssize_t recompress_store(struct device *dev, continue; =20 if (!strcmp(zram->comp_algs[prio], algo)) { - prio_max =3D min(prio + 1, ZRAM_MAX_COMPS); + prio_max =3D prio + 1; found =3D true; break; } @@ -2128,6 +2131,12 @@ static ssize_t recompress_store(struct device *dev, } } =20 + prio_max =3D min(prio_max, (u32)zram->num_active_comps); + if (prio >=3D prio_max) { + ret =3D -EINVAL; + goto release_init_lock; + } + page =3D alloc_page(GFP_KERNEL); if (!page) { ret =3D -ENOMEM; --=20 2.48.1.502.g6dc24dfdaf-goog From nobody Thu Dec 18 10:00:39 2025 Received: from mail-pl1-f177.google.com (mail-pl1-f177.google.com [209.85.214.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0E3551FC7F0 for ; Wed, 12 Feb 2025 06:32:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739341964; cv=none; b=k7+6Wg8eYXctq9AATckC9emVlcgR98wwGPcOSfbwAbbQPkZJIPeSIvANLqUh5460UCkvZLhf4UXX9X3tPMmIga1oV7++B6jQzeIZtluMPsq3IJRdxDzpC39Oq9bRAaAP580ZR218jBYQrzEv/ZkDY7JnHAPk+3nNylv6R/ducyI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739341964; c=relaxed/simple; bh=E0MVDPJMCbp3kVaASHoB5Y9trPKKkPA/EDfpUcdE2uo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=CDzQiQB6D0rrFMUCH0EFcM7W7qmveMd08GD1dP3qHqwWUmC5xBafJ87NkW0LXgwRYTWw2n5VmQSWf7OlD84AOSphlrY6Ld/e8LSPN0fzG8taYlARNvYzu1qbHA3ZoLnOD3zxGwORoTfbM/4ppis3h2c4r3Be8AAakd+IEU9mY50= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=gQDkTi7o; arc=none smtp.client-ip=209.85.214.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="gQDkTi7o" Received: by mail-pl1-f177.google.com with SMTP id d9443c01a7336-21f6d264221so48916075ad.1 for ; Tue, 11 Feb 2025 22:32:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1739341962; x=1739946762; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=GyNDz1JqW9fZ5TiB72j3x2sTVp+ebBvZsmBm0h5y3g0=; b=gQDkTi7oERIonUriwhP/wrK5fI5v5P79BlTHGfecNJhCGbWJk8OKW5oIqAXOqZzmaB u/5IxwI2Sktv53xJHKf+A4j5+aw8WIsW+w0mxhHQonQPlP8oax/aIcbHUFdxSWcCBpC9 bfcEPwAMNvDp3gPv+HO3onNUiC16zXTNNnGKg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739341962; x=1739946762; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=GyNDz1JqW9fZ5TiB72j3x2sTVp+ebBvZsmBm0h5y3g0=; b=iMnT3nnHEUt7YS3m+6JPjEs/bZb4p7ANwy1Xbh5QVbmdiPkh0MbdxulRVLUyuFip/U TxjY3RY2fcdCX+DCS5ea+yQCEQBPZu3auw7q59ppVGTJW2YDlbQXySmPOBscHS4apitx sGOXymjHi9tks+dHSV3BbZuj0gfoVRre3uNpTE6G/HPpLMGRhaFYJ/o2pfpLqyYk5COi acWIdzAh+9ZHSKQ8ZR6J1i+C+I7iaNHlSEPoAPAx4ZGpP7s3XmFNy7yu8VpJ1X7fRlxX rcaW+o3EftvBzxrXT6VxYajn7oZ7Jbl37Q92zoKQWOfjQHAPyP3Oxc4mf+ZcAiDIGfE6 2MdQ== X-Forwarded-Encrypted: i=1; AJvYcCVD5M3zMTGXykLV0rJFdNCiObV5C5O0+S8dH+AIiNVHrpqgxxfU49Lba4g6oFkKqXtEn06jivRPisZp0GQ=@vger.kernel.org X-Gm-Message-State: AOJu0YyxA4CFuTETGyAZepSsQe4fg5QnkF3KotdRlJGf4P9qjOYUeEnb L0Gf08Vnt/gVMGEnTawIEUCZgrf0gHTYwOCcerfW9M0eqHxrb/aBvdgBN/L9MA== X-Gm-Gg: ASbGncubLTq7ZoQAwezqQfqYCSEMqM+1n8i3ybO8hNi3Qd+kn1UtLGfn7msZLTtKpVW gtUvstb6cSi/rT1Qfh9xPmBQFyl+QnrsddlN+H/Fd26RZp3z6T06rXgzO6rUqiOd7eHvYRXr0/b RQmjCpBPI71p6MGl+K43pf0xtPiP5zlV32mHOdqxCak9OmimoZee/ZSJG0UrCcUKI0gjfww8BC5 K0Bc594t0LyPJfq6mpNmLrLCZkYm3h4YSZXRGPNu5h+GaEvDCkSPy6Cz0sZMgafPxDGm/Hywgba vhmwMwkntKJZILAsjw== X-Google-Smtp-Source: AGHT+IHuvm8HCsSAsDPLpNBHM4fkgErNxht0F4ETwuQK+JzFh06XIBG0WxJwETNpSs2/VqRQClDZHQ== X-Received: by 2002:a05:6a21:600b:b0:1e1:af74:a235 with SMTP id adf61e73a8af0-1ee5c790de5mr4045324637.24.1739341962338; Tue, 11 Feb 2025 22:32:42 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:69f5:6852:451e:8142]) by smtp.gmail.com with UTF8SMTPSA id d2e1a72fcca58-7308e4881fesm4730281b3a.46.2025.02.11.22.32.40 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 11 Feb 2025 22:32:42 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Kairui Song , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v5 08/18] zram: filter out recomp targets based on priority Date: Wed, 12 Feb 2025 15:27:06 +0900 Message-ID: <20250212063153.179231-9-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.502.g6dc24dfdaf-goog In-Reply-To: <20250212063153.179231-1-senozhatsky@chromium.org> References: <20250212063153.179231-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Do no select for post processing slots that are already compressed with same or higher priority compression algorithm. This should save some memory, as previously we would still put those entries into corresponding post-processing buckets and filter them out later in recompress_slot(). Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 25 ++++++++++++++++--------- 1 file changed, 16 insertions(+), 9 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 27148f3e5ae9..31bdf5e0ff74 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1822,7 +1822,7 @@ static int zram_bvec_write(struct zram *zram, struct = bio_vec *bvec, #define RECOMPRESS_IDLE (1 << 0) #define RECOMPRESS_HUGE (1 << 1) =20 -static int scan_slots_for_recompress(struct zram *zram, u32 mode, +static int scan_slots_for_recompress(struct zram *zram, u32 mode, u32 prio= _max, struct zram_pp_ctl *ctl) { unsigned long nr_pages =3D zram->disksize >> PAGE_SHIFT; @@ -1854,6 +1854,10 @@ static int scan_slots_for_recompress(struct zram *zr= am, u32 mode, zram_test_flag(zram, index, ZRAM_INCOMPRESSIBLE)) goto next; =20 + /* Already compressed with same of higher priority */ + if (zram_get_priority(zram, index) + 1 >=3D prio_max) + goto next; + pps->index =3D index; place_pp_slot(zram, ctl, pps); pps =3D NULL; @@ -1910,6 +1914,16 @@ static int recompress_slot(struct zram *zram, u32 in= dex, struct page *page, zram_clear_flag(zram, index, ZRAM_IDLE); =20 class_index_old =3D zs_lookup_class_index(zram->mem_pool, comp_len_old); + + prio =3D max(prio, zram_get_priority(zram, index) + 1); + /* + * Recompression slots scan should not select slots that are + * already compressed with a higher priority algorithm, but + * just in case + */ + if (prio >=3D prio_max) + return 0; + /* * Iterate the secondary comp algorithms list (in order of priority) * and try to recompress the page. @@ -1918,13 +1932,6 @@ static int recompress_slot(struct zram *zram, u32 in= dex, struct page *page, if (!zram->comps[prio]) continue; =20 - /* - * Skip if the object is already re-compressed with a higher - * priority algorithm (or same algorithm). - */ - if (prio <=3D zram_get_priority(zram, index)) - continue; - num_recomps++; zstrm =3D zcomp_stream_get(zram->comps[prio]); src =3D kmap_local_page(page); @@ -2149,7 +2156,7 @@ static ssize_t recompress_store(struct device *dev, goto release_init_lock; } =20 - scan_slots_for_recompress(zram, mode, ctl); + scan_slots_for_recompress(zram, mode, prio_max, ctl); =20 ret =3D len; while ((pps =3D select_pp_slot(ctl))) { --=20 2.48.1.502.g6dc24dfdaf-goog From nobody Thu Dec 18 10:00:39 2025 Received: from mail-pj1-f49.google.com (mail-pj1-f49.google.com [209.85.216.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BBA2D1FECCC for ; Wed, 12 Feb 2025 06:32:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.49 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739341970; cv=none; b=QmVnS91lmZbYgnFzWrL2d1WCGYMz2eFm/ct0807LdOdbodMUlXrBghx8DKPU9gwP1BAqMmhIQ+x1xA3EOKNnho+UFiPLgpVDGRy44uQQpoFebEJMT3h9OVyNaMaTC01BAYAazRksHWdint634NkPco1eZ27dSB2qGpgvJTgWtRk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739341970; c=relaxed/simple; bh=Lr+Wmxc84QrcvanZk331Bl73+U3JT0S523UifFEoQ/c=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=UxNo9qjmLuuQMj3XQZ953p9LBugqgkDe3222dspwqaR7lh4DwWQksoYcKu5sdqrawfL1+KAHgcNs6FVyEPO9r9dEwp0HWXimoVfG6vV0GzEuSqRcr1uCvffbcofzeoqj1XLCR8tG+0QRcO0UgfdN6IL/kh+Hj9ZgR9nPNZEGyDU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=bqwLlobW; arc=none smtp.client-ip=209.85.216.49 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="bqwLlobW" Received: by mail-pj1-f49.google.com with SMTP id 98e67ed59e1d1-2fa51743d80so5612285a91.2 for ; Tue, 11 Feb 2025 22:32:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1739341968; x=1739946768; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=cLnpAJqzl8agqjBt4hqeQuPEXwmT47X6Rb4B201CoRE=; b=bqwLlobW914QOjbIvfm/Eq9g0Bev0DRkWWH9vBF4K1toR38jU1kPXSMRUC+5WFoIOd VwuCvnTRAr+BJWotWzHd+NHC26FYbeKE1tI+DULst2o5zfoiIm0E4YX3iai5zWla+00B +Sll0LgFcN9q9Ev4ErpEFI4u7eChactQbjEvA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739341968; x=1739946768; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=cLnpAJqzl8agqjBt4hqeQuPEXwmT47X6Rb4B201CoRE=; b=gIRVi1OA3j3xcMiweIctdr0Oq+itepEZxiYk+IOD/8rujDxPJBNBUIi9li8g8MCMZr PTmaAKL1/F4Ku7ogsiv/yI8QymrDkujoX/ixBZsfPw3ooZvtEyMgQr/Dm/ido8CqDdgO ybar628/j4cX6VOv5HUzyI7V0DwNLSbyaxECadgFUGPemd22ejFq8psdvngU0+g2uZbC Kn6D7qAudotILZ6m60F6TepvUriejWihm1mUSFxAnFcRG4rR75vxP+2xdDohHuSvHIV7 Ocij1P3hxfJNgvyYK90xUNRC1sV+dfTpmStcOgZKrGHIwk+NQO9o+qDN6nKi3Xks9Ria XKgA== X-Forwarded-Encrypted: i=1; AJvYcCWTRiLMxdFvj3Hli6/r4HZk+mSwy/J/0rK9erdkfvZpVnOQ0zBVUXrxdSz+Uhgd/4jWnV2O/+rouw5Qj1U=@vger.kernel.org X-Gm-Message-State: AOJu0YxEERYAEDzTwvKw+QuiZMq7QY/RYr1JmJ9tCbOb/80cN/HLDI2+ hYjmQL3UvwquNKsUZm59mmTlX3P0VQ1kSYDDF7cW8ZDaqK36fB2CnEXXmvoOtg== X-Gm-Gg: ASbGncvivmBSyWj7MMirTtb1nCWftLOLQb3Rp3HQQKQp0xLo2T6txaYF3UD+UoQORj1 Wp/PMV6pFROjoGwAi5qlJ8z69y2g3cNRqlx4/yyHqDhKqSkXx7w5kpn00vuZJ+ErCEvzzbwf/FQ WkLWRM25YibhbGnFdftTpD3vIVSxDdkrI8JIsvq3QU9ykGplCGOum75l4eKlZjzoI912UWfh2Yr qrQ2h/PTvYNxX0mt/Id4KiX2mkj/iASOmywj7EWh9xeMaF1t/NRemfHxMBAmMsxU0cg0coVgTio iD/aAO1KFU8TGHIhxQ== X-Google-Smtp-Source: AGHT+IHiT+N5GVxrJj2d7+UgbwxnPgo/4IOHt2XM/rWhD+WNLZnaiOqWIvo76vE7u+pn65YNGazKSQ== X-Received: by 2002:a05:6a00:1798:b0:732:24ad:8e08 with SMTP id d2e1a72fcca58-7322c376a72mr3048146b3a.1.1739341967995; Tue, 11 Feb 2025 22:32:47 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:69f5:6852:451e:8142]) by smtp.gmail.com with UTF8SMTPSA id d2e1a72fcca58-73084e182cdsm5759136b3a.99.2025.02.11.22.32.45 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 11 Feb 2025 22:32:47 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Kairui Song , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v5 09/18] zram: rework recompression loop Date: Wed, 12 Feb 2025 15:27:07 +0900 Message-ID: <20250212063153.179231-10-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.502.g6dc24dfdaf-goog In-Reply-To: <20250212063153.179231-1-senozhatsky@chromium.org> References: <20250212063153.179231-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This reworks recompression loop handling: - set a rule that stream-put NULLs the stream pointer If the loop returns with a non-NULL stream then it's a successfull recompression, otherwise the stream should always be NULL. - do not count the number of recompressions Mark object as incompressible as soon as the algorithm with the highest priority failed to compress that object. - count compression errors as resource usage Even if compression has failed, we still need to bump num_recomp_pages counter. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 53 +++++++++++++---------------------- 1 file changed, 19 insertions(+), 34 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 31bdf5e0ff74..7c4c296181a8 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1887,9 +1887,8 @@ static int recompress_slot(struct zram *zram, u32 ind= ex, struct page *page, unsigned int comp_len_new; unsigned int class_index_old; unsigned int class_index_new; - u32 num_recomps =3D 0; void *src, *dst; - int ret; + int ret =3D 0; =20 handle_old =3D zram_get_handle(zram, index); if (!handle_old) @@ -1932,7 +1931,6 @@ static int recompress_slot(struct zram *zram, u32 ind= ex, struct page *page, if (!zram->comps[prio]) continue; =20 - num_recomps++; zstrm =3D zcomp_stream_get(zram->comps[prio]); src =3D kmap_local_page(page); ret =3D zcomp_compress(zram->comps[prio], zstrm, @@ -1941,7 +1939,8 @@ static int recompress_slot(struct zram *zram, u32 ind= ex, struct page *page, =20 if (ret) { zcomp_stream_put(zstrm); - return ret; + zstrm =3D NULL; + break; } =20 class_index_new =3D zs_lookup_class_index(zram->mem_pool, @@ -1951,6 +1950,7 @@ static int recompress_slot(struct zram *zram, u32 ind= ex, struct page *page, if (class_index_new >=3D class_index_old || (threshold && comp_len_new >=3D threshold)) { zcomp_stream_put(zstrm); + zstrm =3D NULL; continue; } =20 @@ -1958,14 +1958,6 @@ static int recompress_slot(struct zram *zram, u32 in= dex, struct page *page, break; } =20 - /* - * We did not try to recompress, e.g. when we have only one - * secondary algorithm and the page is already recompressed - * using that algorithm - */ - if (!zstrm) - return 0; - /* * Decrement the limit (if set) on pages we can recompress, even * when current recompression was unsuccessful or did not compress @@ -1975,38 +1967,31 @@ static int recompress_slot(struct zram *zram, u32 i= ndex, struct page *page, if (*num_recomp_pages) *num_recomp_pages -=3D 1; =20 - if (class_index_new >=3D class_index_old) { + /* Compression error */ + if (ret) + return ret; + + if (!zstrm) { /* * Secondary algorithms failed to re-compress the page - * in a way that would save memory, mark the object as - * incompressible so that we will not try to compress - * it again. + * in a way that would save memory. * - * We need to make sure that all secondary algorithms have - * failed, so we test if the number of recompressions matches - * the number of active secondary algorithms. + * Mark the object incompressible if the max-priority + * algorithm couldn't re-compress it. */ - if (num_recomps =3D=3D zram->num_active_comps - 1) - zram_set_flag(zram, index, ZRAM_INCOMPRESSIBLE); + if (prio < zram->num_active_comps) + return 0; + zram_set_flag(zram, index, ZRAM_INCOMPRESSIBLE); return 0; } =20 - /* Successful recompression but above threshold */ - if (threshold && comp_len_new >=3D threshold) - return 0; - /* - * No direct reclaim (slow path) for handle allocation and no - * re-compression attempt (unlike in zram_write_bvec()) since - * we already have stored that object in zsmalloc. If we cannot - * alloc memory for recompressed object then we bail out and - * simply keep the old (existing) object in zsmalloc. + * We are holding per-CPU stream mutex and entry lock so better + * avoid direct reclaim. Allocation error is not fatal since + * we still have the old object in the mem_pool. */ handle_new =3D zs_malloc(zram->mem_pool, comp_len_new, - __GFP_KSWAPD_RECLAIM | - __GFP_NOWARN | - __GFP_HIGHMEM | - __GFP_MOVABLE); + GFP_NOWAIT | __GFP_HIGHMEM | __GFP_MOVABLE); if (IS_ERR_VALUE(handle_new)) { zcomp_stream_put(zstrm); return PTR_ERR((void *)handle_new); --=20 2.48.1.502.g6dc24dfdaf-goog From nobody Thu Dec 18 10:00:39 2025 Received: from mail-pl1-f172.google.com (mail-pl1-f172.google.com [209.85.214.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C024C20102D for ; Wed, 12 Feb 2025 06:32:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739341975; cv=none; b=KmLQpsouLH8+ww4GqPPyGxoF4L7oOnZl8u9uQO6VqjX9pSnT22mGUJnRu9n3CaHhrK4j2WrIhLFPjYlMDuqb5gg/au9vqe4MUsVkWBnplNA7WN1NpfSE0DmIAR6z9q6M1CZZ/N8SkPpE3Kag0RNoZMpmB9IA3UmymHUijR2JaMI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739341975; c=relaxed/simple; bh=9Vx0WYJ2miMeZYqFk4IUJzt8mlgRl+8ieJ507QhwuX4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=UZXJrqZZv88lyHoGfQ1fLSSVeT62SrfYjTuPysGzgtjsIjiePJhGY5c+3uXg6FaV2/xPhP5D6jDdzEg2a7zJi54pzoG1HE/XMqaIT+Fyp6X5ZHyx4Cv+QwlDlVPJOsUPUGwLPeA+VLq1FRWE9Vu6vnRDXcMXzdcQUV4Q+S4QO9o= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=RdFGV9Sk; arc=none smtp.client-ip=209.85.214.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="RdFGV9Sk" Received: by mail-pl1-f172.google.com with SMTP id d9443c01a7336-21f710c17baso68787605ad.1 for ; Tue, 11 Feb 2025 22:32:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1739341973; x=1739946773; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=J+hKSGhtq9SEMO5RmXUCmDriDLy9gufp9/zvmYkmXcc=; b=RdFGV9SkC678DZKlba0/QjiP2iMebkjJ1+XDk8geyJ6XDLQByNGzKs4CEblmgGfFyO XlbWI33ozsQUrvaeOUQlqifiCoSHEBq0m3T8Cqe8B9/MqrZqlYd8ilbiViip3YK8FMge Ds4c/ZWAex89iZRVYgcemOF8sAOHzSwi1pQaQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739341973; x=1739946773; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=J+hKSGhtq9SEMO5RmXUCmDriDLy9gufp9/zvmYkmXcc=; b=TODbaTIgBFpvi6A8jETOma+pWu+8yyemwBxDM+vVzFKIY0TdUq5TJ07eVY9Pk6GgCw LX3/ugYt/VXYEVdN6bxQ5AJrhRG6s9oKRHjJZIIi8NyRyB7zhUC80vsj+JhM4uGIUsCn uh158Rfrxb4So4MRlf3+az0PeBGxQ7LzV236ANcj9qes8Tz1QgU+w4WWM8tf47o6jETF UsP6aGu13BHkLA/fU3L7K95ihW6Idb6iVcUucdbp5io9QrUqtK08wFa138TYpNLx6a/M 0cNzHCoEI/rwLKu/ltx1JlomtEatYlhisWk9Ci6UbHSgA5e63pSk5WiAn3T1Vgw++uWg 7sRQ== X-Forwarded-Encrypted: i=1; AJvYcCX5WoUw5U7rTIMiSqK1EJHZMeBQ1vJImhSvVz+w89/8+jbyNfKqx4GvNBZmWYNaYDEYq+N5VRRNEgneSPk=@vger.kernel.org X-Gm-Message-State: AOJu0YzbXLHT61Euv16BcU88MK3uXbAgcbg5sU2lL8N7wECcsQ8TBkon pD5rLgU4RMHjp5gesmgsU8sBAVO0CdYlFu902Z5ysN8AQlCOUTH4Z/hINmWfgw== X-Gm-Gg: ASbGncuGgUQOjCMgl363lAGQmQy4lrhwaBjF2mkRQTlR0MZ+skD5m2duqagFoANkfGM +Co9gExgnZ6vtqOSXokX1kUfJH/oHswqssz+RGZWbg824usY4APGU6Lj1kLjO3iTH9Rox849IIR pWoXJ42SEGkWQu2vHONDvyhZ7j5vkzNMxRstnE0DOnwJHEuAZ0fGWaQOhh7vs044CopE1i1DuE8 ZvJ043mNvg9T6W8YkgCh3LqaJjSNT/WnTsJZhFoIS0XDQ0gRNbZwlDHwtyfvTBpL7aqenEu3bxF HI02e/dJfx6lQiHCnQ== X-Google-Smtp-Source: AGHT+IGDl9mV+HtGUyn4myzfZBpkMQ7To3I2eCgVPnnWhAaFxnQ0KIUaHGWBSJNUrky9tjmUsZBSrQ== X-Received: by 2002:a05:6a20:3943:b0:1ee:650b:ac22 with SMTP id adf61e73a8af0-1ee650bb853mr710537637.40.1739341972819; Tue, 11 Feb 2025 22:32:52 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:69f5:6852:451e:8142]) by smtp.gmail.com with UTF8SMTPSA id d2e1a72fcca58-73077975f05sm7037787b3a.14.2025.02.11.22.32.50 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 11 Feb 2025 22:32:52 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Kairui Song , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v5 10/18] zsmalloc: factor out pool locking helpers Date: Wed, 12 Feb 2025 15:27:08 +0900 Message-ID: <20250212063153.179231-11-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.502.g6dc24dfdaf-goog In-Reply-To: <20250212063153.179231-1-senozhatsky@chromium.org> References: <20250212063153.179231-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" We currently have a mix of migrate_{read,write}_lock() helpers that lock zspages, but it's zs_pool that actually has a ->migrate_lock access to which is opene-coded. Factor out pool migrate locking into helpers, zspage migration locking API will be renamed to reduce confusion. It's worth mentioning that zsmalloc locks sync not only migration, but also compaction. Signed-off-by: Sergey Senozhatsky --- mm/zsmalloc.c | 63 +++++++++++++++++++++++++++++++++++---------------- 1 file changed, 44 insertions(+), 19 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 6d0e47f7ae33..47c638df47c5 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -18,7 +18,7 @@ /* * lock ordering: * page_lock - * pool->migrate_lock + * pool->lock * class->lock * zspage->lock */ @@ -224,10 +224,35 @@ struct zs_pool { struct work_struct free_work; #endif /* protect page/zspage migration */ - rwlock_t migrate_lock; + rwlock_t lock; atomic_t compaction_in_progress; }; =20 +static void pool_write_unlock(struct zs_pool *pool) +{ + write_unlock(&pool->lock); +} + +static void pool_write_lock(struct zs_pool *pool) +{ + write_lock(&pool->lock); +} + +static void pool_read_unlock(struct zs_pool *pool) +{ + read_unlock(&pool->lock); +} + +static void pool_read_lock(struct zs_pool *pool) +{ + read_lock(&pool->lock); +} + +static bool pool_lock_is_contended(struct zs_pool *pool) +{ + return rwlock_is_contended(&pool->lock); +} + static inline void zpdesc_set_first(struct zpdesc *zpdesc) { SetPagePrivate(zpdesc_page(zpdesc)); @@ -1206,7 +1231,7 @@ void *zs_map_object(struct zs_pool *pool, unsigned lo= ng handle, BUG_ON(in_interrupt()); =20 /* It guarantees it can get zspage from handle safely */ - read_lock(&pool->migrate_lock); + pool_read_lock(pool); obj =3D handle_to_obj(handle); obj_to_location(obj, &zpdesc, &obj_idx); zspage =3D get_zspage(zpdesc); @@ -1218,7 +1243,7 @@ void *zs_map_object(struct zs_pool *pool, unsigned lo= ng handle, * which is smaller granularity. */ migrate_read_lock(zspage); - read_unlock(&pool->migrate_lock); + pool_read_unlock(pool); =20 class =3D zspage_class(pool, zspage); off =3D offset_in_page(class->size * obj_idx); @@ -1450,16 +1475,16 @@ void zs_free(struct zs_pool *pool, unsigned long ha= ndle) return; =20 /* - * The pool->migrate_lock protects the race with zpage's migration + * The pool->lock protects the race with zpage's migration * so it's safe to get the page from handle. */ - read_lock(&pool->migrate_lock); + pool_read_lock(pool); obj =3D handle_to_obj(handle); obj_to_zpdesc(obj, &f_zpdesc); zspage =3D get_zspage(f_zpdesc); class =3D zspage_class(pool, zspage); spin_lock(&class->lock); - read_unlock(&pool->migrate_lock); + pool_read_unlock(pool); =20 class_stat_sub(class, ZS_OBJS_INUSE, 1); obj_free(class->size, obj); @@ -1793,10 +1818,10 @@ static int zs_page_migrate(struct page *newpage, st= ruct page *page, pool =3D zspage->pool; =20 /* - * The pool migrate_lock protects the race between zpage migration + * The pool lock protects the race between zpage migration * and zs_free. */ - write_lock(&pool->migrate_lock); + pool_write_lock(pool); class =3D zspage_class(pool, zspage); =20 /* @@ -1833,7 +1858,7 @@ static int zs_page_migrate(struct page *newpage, stru= ct page *page, * Since we complete the data copy and set up new zspage structure, * it's okay to release migration_lock. */ - write_unlock(&pool->migrate_lock); + pool_write_unlock(pool); spin_unlock(&class->lock); migrate_write_unlock(zspage); =20 @@ -1956,7 +1981,7 @@ static unsigned long __zs_compact(struct zs_pool *poo= l, * protect the race between zpage migration and zs_free * as well as zpage allocation/free */ - write_lock(&pool->migrate_lock); + pool_write_lock(pool); spin_lock(&class->lock); while (zs_can_compact(class)) { int fg; @@ -1983,14 +2008,14 @@ static unsigned long __zs_compact(struct zs_pool *p= ool, src_zspage =3D NULL; =20 if (get_fullness_group(class, dst_zspage) =3D=3D ZS_INUSE_RATIO_100 - || rwlock_is_contended(&pool->migrate_lock)) { + || pool_lock_is_contended(pool)) { putback_zspage(class, dst_zspage); dst_zspage =3D NULL; =20 spin_unlock(&class->lock); - write_unlock(&pool->migrate_lock); + pool_write_unlock(pool); cond_resched(); - write_lock(&pool->migrate_lock); + pool_write_lock(pool); spin_lock(&class->lock); } } @@ -2002,7 +2027,7 @@ static unsigned long __zs_compact(struct zs_pool *poo= l, putback_zspage(class, dst_zspage); =20 spin_unlock(&class->lock); - write_unlock(&pool->migrate_lock); + pool_write_unlock(pool); =20 return pages_freed; } @@ -2014,10 +2039,10 @@ unsigned long zs_compact(struct zs_pool *pool) unsigned long pages_freed =3D 0; =20 /* - * Pool compaction is performed under pool->migrate_lock so it is basical= ly + * Pool compaction is performed under pool->lock so it is basically * single-threaded. Having more than one thread in __zs_compact() - * will increase pool->migrate_lock contention, which will impact other - * zsmalloc operations that need pool->migrate_lock. + * will increase pool->lock contention, which will impact other + * zsmalloc operations that need pool->lock. */ if (atomic_xchg(&pool->compaction_in_progress, 1)) return 0; @@ -2139,7 +2164,7 @@ struct zs_pool *zs_create_pool(const char *name) return NULL; =20 init_deferred_free(pool); - rwlock_init(&pool->migrate_lock); + rwlock_init(&pool->lock); atomic_set(&pool->compaction_in_progress, 0); =20 pool->name =3D kstrdup(name, GFP_KERNEL); --=20 2.48.1.502.g6dc24dfdaf-goog From nobody Thu Dec 18 10:00:39 2025 Received: from mail-pl1-f179.google.com (mail-pl1-f179.google.com [209.85.214.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5094F20126A for ; Wed, 12 Feb 2025 06:32:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739341979; cv=none; b=N0V4TjA8fMnidCEjRkjmgwXDeJuwkKSTX7ZyyKO/lchzNs9qsnW7mzdBgcmCRFTQ2Iz5e+OBD7KXduIEtAfdtwlQkAHsVxS0aNCHG3Md7fRklqL0jVrEXE7fX7KJXPEMo1+9sD5tITgqLis7ZB1AiOjBgl6e1gA/zoZthwchcl8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739341979; c=relaxed/simple; bh=7dbXBWyL/2c9f+MJeW6Vq0IhAO1Sls7FTtrD05JhIpw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=DOi9zLVeWALdRfFxGEjcd3rR2ZoVDy/efEbBObNV+tIySJm4+bG7rlrvZHsF/5g+fWkxXpt5Etgh7+HI2v38hI0+XPvnejl7Xz8UNrIkkHYrpEBrWgreJS5e1y51FDC9h21QfmphgFML6PcigRuKPfTzdFfqOOUOi8pzcq4gnPY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=FINMY+Cx; arc=none smtp.client-ip=209.85.214.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="FINMY+Cx" Received: by mail-pl1-f179.google.com with SMTP id d9443c01a7336-21f2339dcfdso8269055ad.1 for ; Tue, 11 Feb 2025 22:32:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1739341978; x=1739946778; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=1Z728glTbJRUijcb8TCkc0Ox5AYTm//47KQTAd0HAqQ=; b=FINMY+CxaHhC2eRYPmlBr5i9rNcCKLzZQN+oIgXweFPGVWGyGbi9OIeVRpuNAC+2DM Zj6UB9T/riyn6RBgrwURyzSvDZtEqVaPh76XpEXLvmuKAa5tc8PmJc42rJzk08BIqmri epjmNQP3ANORzINwiDyYyipRyYbBit4+JVYEM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739341978; x=1739946778; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=1Z728glTbJRUijcb8TCkc0Ox5AYTm//47KQTAd0HAqQ=; b=GKUxoEKJIQ2QusO+yIjH/IIYK232WYKc80RDonIZKTbtDhO6b+ljpuxZKftzn0XKPH 29S5d8vTovNBg9xxVst3RdusrBHEYsNUZWq68l8PCX9U+on0FrJRrWq5P9Zskkl8I7I1 pJwNTujj1j4yPIcBuM6WBn5fa14+fDHnlWw/vMfTcfMMvWXtHshYcfy2q3vk2nR6TCbn 5xeAnlIW1RccfuSMy+nPf3UI04mX0fv1oFozOCmpvr8d2cgTeIq8CMb9aFb/iue1IKa0 tJrH6P7pXE0eJA8FPYOLbKp0iTpPS7AZ5Q3IWjpiS2KY/HQI0bY5PXdauXlFGHwtDS9Z CvBA== X-Forwarded-Encrypted: i=1; AJvYcCVerjLWiKnMYrQwLBTVTrAQ+8FEVMA7B13jg+wZ70bLPiI6x8D+EfZ7pjwih5Ay5monJAXOnSHyj33JnNQ=@vger.kernel.org X-Gm-Message-State: AOJu0Yw4ZsMkoulr/HaBWxXacf0m5TNpAE3I5lIukfWJhdoEILp1iBeT 2sfxBcupTynuuXgsoJRphvGIX/IQApJi0aLgw00yNwNmIQR1FbLDdC6nwGV9Yw== X-Gm-Gg: ASbGncsqAxDrNtcNsOHQSpMZgoqw3y++QDAWyixqGbHL8A3wYWdYo7Rb2zfCHLFEQFi z+FvrMQyfg/nqY8kmFkm6lJQQmtbikjzxbq3vxJ6d2rpzlR4ileQv0d95QiNJr+sOtuZTRBEJT1 ZImhhlWPWzq0MqjisLe1OZlH7U/VmhzcOgwrd2HnN2+ro1THUR7k7uAViIAqK7fmO/tOah4tgkQ 3MRmH/YTLJjJF8BexDCXFa36Jgu0qp+jGuDQ+lg0vUH1OVGY9WfRpN2S3xhm86HRwVOALlGBFis arfvO7iXmVK91dBHlQ== X-Google-Smtp-Source: AGHT+IH9novtJAP2nahMJtf23e6d5DXnQj2BtTSpA3K6387kmaUUk7tqOB+00czVkoYIJ9iJRKWKIQ== X-Received: by 2002:a17:90b:2b45:b0:2fa:1c09:3cee with SMTP id 98e67ed59e1d1-2faa0913acemr9497572a91.9.1739341977607; Tue, 11 Feb 2025 22:32:57 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:69f5:6852:451e:8142]) by smtp.gmail.com with UTF8SMTPSA id 98e67ed59e1d1-2fbf98cffddsm658335a91.21.2025.02.11.22.32.55 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 11 Feb 2025 22:32:57 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Kairui Song , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v5 11/18] zsmalloc: factor out size-class locking helpers Date: Wed, 12 Feb 2025 15:27:09 +0900 Message-ID: <20250212063153.179231-12-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.502.g6dc24dfdaf-goog In-Reply-To: <20250212063153.179231-1-senozhatsky@chromium.org> References: <20250212063153.179231-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Move open-coded size-class locking to dedicated helpers. Signed-off-by: Sergey Senozhatsky Reviewed-by: Yosry Ahmed --- mm/zsmalloc.c | 47 ++++++++++++++++++++++++++++------------------- 1 file changed, 28 insertions(+), 19 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 47c638df47c5..c82c24b8e6a4 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -253,6 +253,16 @@ static bool pool_lock_is_contended(struct zs_pool *poo= l) return rwlock_is_contended(&pool->lock); } =20 +static void size_class_lock(struct size_class *class) +{ + spin_lock(&class->lock); +} + +static void size_class_unlock(struct size_class *class) +{ + spin_unlock(&class->lock); +} + static inline void zpdesc_set_first(struct zpdesc *zpdesc) { SetPagePrivate(zpdesc_page(zpdesc)); @@ -613,8 +623,7 @@ static int zs_stats_size_show(struct seq_file *s, void = *v) if (class->index !=3D i) continue; =20 - spin_lock(&class->lock); - + size_class_lock(class); seq_printf(s, " %5u %5u ", i, class->size); for (fg =3D ZS_INUSE_RATIO_10; fg < NR_FULLNESS_GROUPS; fg++) { inuse_totals[fg] +=3D class_stat_read(class, fg); @@ -624,7 +633,7 @@ static int zs_stats_size_show(struct seq_file *s, void = *v) obj_allocated =3D class_stat_read(class, ZS_OBJS_ALLOCATED); obj_used =3D class_stat_read(class, ZS_OBJS_INUSE); freeable =3D zs_can_compact(class); - spin_unlock(&class->lock); + size_class_unlock(class); =20 objs_per_zspage =3D class->objs_per_zspage; pages_used =3D obj_allocated / objs_per_zspage * @@ -1399,7 +1408,7 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t = size, gfp_t gfp) class =3D pool->size_class[get_size_class_index(size)]; =20 /* class->lock effectively protects the zpage migration */ - spin_lock(&class->lock); + size_class_lock(class); zspage =3D find_get_zspage(class); if (likely(zspage)) { obj_malloc(pool, zspage, handle); @@ -1410,7 +1419,7 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t = size, gfp_t gfp) goto out; } =20 - spin_unlock(&class->lock); + size_class_unlock(class); =20 zspage =3D alloc_zspage(pool, class, gfp); if (!zspage) { @@ -1418,7 +1427,7 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t = size, gfp_t gfp) return (unsigned long)ERR_PTR(-ENOMEM); } =20 - spin_lock(&class->lock); + size_class_lock(class); obj_malloc(pool, zspage, handle); newfg =3D get_fullness_group(class, zspage); insert_zspage(class, zspage, newfg); @@ -1429,7 +1438,7 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t = size, gfp_t gfp) /* We completely set up zspage so mark them as movable */ SetZsPageMovable(pool, zspage); out: - spin_unlock(&class->lock); + size_class_unlock(class); =20 return handle; } @@ -1483,7 +1492,7 @@ void zs_free(struct zs_pool *pool, unsigned long hand= le) obj_to_zpdesc(obj, &f_zpdesc); zspage =3D get_zspage(f_zpdesc); class =3D zspage_class(pool, zspage); - spin_lock(&class->lock); + size_class_lock(class); pool_read_unlock(pool); =20 class_stat_sub(class, ZS_OBJS_INUSE, 1); @@ -1493,7 +1502,7 @@ void zs_free(struct zs_pool *pool, unsigned long hand= le) if (fullness =3D=3D ZS_INUSE_RATIO_0) free_zspage(pool, class, zspage); =20 - spin_unlock(&class->lock); + size_class_unlock(class); cache_free_handle(pool, handle); } EXPORT_SYMBOL_GPL(zs_free); @@ -1827,7 +1836,7 @@ static int zs_page_migrate(struct page *newpage, stru= ct page *page, /* * the class lock protects zpage alloc/free in the zspage. */ - spin_lock(&class->lock); + size_class_lock(class); /* the migrate_write_lock protects zpage access via zs_map_object */ migrate_write_lock(zspage); =20 @@ -1859,7 +1868,7 @@ static int zs_page_migrate(struct page *newpage, stru= ct page *page, * it's okay to release migration_lock. */ pool_write_unlock(pool); - spin_unlock(&class->lock); + size_class_unlock(class); migrate_write_unlock(zspage); =20 zpdesc_get(newzpdesc); @@ -1903,10 +1912,10 @@ static void async_free_zspage(struct work_struct *w= ork) if (class->index !=3D i) continue; =20 - spin_lock(&class->lock); + size_class_lock(class); list_splice_init(&class->fullness_list[ZS_INUSE_RATIO_0], &free_pages); - spin_unlock(&class->lock); + size_class_unlock(class); } =20 list_for_each_entry_safe(zspage, tmp, &free_pages, list) { @@ -1914,10 +1923,10 @@ static void async_free_zspage(struct work_struct *w= ork) lock_zspage(zspage); =20 class =3D zspage_class(pool, zspage); - spin_lock(&class->lock); + size_class_lock(class); class_stat_sub(class, ZS_INUSE_RATIO_0, 1); __free_zspage(pool, class, zspage); - spin_unlock(&class->lock); + size_class_unlock(class); } }; =20 @@ -1982,7 +1991,7 @@ static unsigned long __zs_compact(struct zs_pool *poo= l, * as well as zpage allocation/free */ pool_write_lock(pool); - spin_lock(&class->lock); + size_class_lock(class); while (zs_can_compact(class)) { int fg; =20 @@ -2012,11 +2021,11 @@ static unsigned long __zs_compact(struct zs_pool *p= ool, putback_zspage(class, dst_zspage); dst_zspage =3D NULL; =20 - spin_unlock(&class->lock); + size_class_unlock(class); pool_write_unlock(pool); cond_resched(); pool_write_lock(pool); - spin_lock(&class->lock); + size_class_lock(class); } } =20 @@ -2026,7 +2035,7 @@ static unsigned long __zs_compact(struct zs_pool *poo= l, if (dst_zspage) putback_zspage(class, dst_zspage); =20 - spin_unlock(&class->lock); + size_class_unlock(class); pool_write_unlock(pool); =20 return pages_freed; --=20 2.48.1.502.g6dc24dfdaf-goog From nobody Thu Dec 18 10:00:39 2025 Received: from mail-pj1-f54.google.com (mail-pj1-f54.google.com [209.85.216.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 26B93202C2D for ; Wed, 12 Feb 2025 06:33:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.54 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739341985; cv=none; b=C7qxUy1f3hzvjhPd7LFKfXYnaJf4hfEs/tnegn17HZTTqaaBnHUeymo/eJLfk8dssZ0LrdyyxesZQxpSFXNVxLAVaJ/qC3Pr7LYoZ64ldFEnajC52ycTkVVKSOQ9lpuzEcUvfXcAbOJchK7VuNz+WRjGLUaHzSzm6qCaMUyDf+A= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739341985; c=relaxed/simple; bh=+ZmibeS3s1Hk7g8ASbFMW04vf9D0deCx4rN7HHvjO0Q=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ZuHtSPCP/uhrOFFt2H/8cGqwRtL/pLu5HD4KxzWZ4cIjKJ+vztrU49U8FOYpnVkog/I0C5qYIVbyCDgy0QyZE6ZDn4Ze9dXaUzswTZEoVtKuV2r+pBrDLc/3L1v3qXwoIsIq9p389TmrsYtt0soBYJ1x0emY0yCD7yt4r920SDE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=awL+YitN; arc=none smtp.client-ip=209.85.216.54 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="awL+YitN" Received: by mail-pj1-f54.google.com with SMTP id 98e67ed59e1d1-2fbffe0254fso20738a91.3 for ; Tue, 11 Feb 2025 22:33:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1739341982; x=1739946782; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=BwlPMmfdw4MbZw7eUNqN2h6GCS8i0cwiXU5uZLysFGk=; b=awL+YitNpJ23rZWcwLyUeOnn7crThAYD+3xioyHds/cXeUnJlmO4xuB0c1IDqRHyKb 2Pw3NdyOsuFP89oQ2jWRMPlflC438CddxX4ybh9uw5ZNd616V9yjis0s06uu2HPAUmf3 5lVrEzm9cKf17Tbo/NwrFJYJLPR0WWgLRcqGM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739341982; x=1739946782; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=BwlPMmfdw4MbZw7eUNqN2h6GCS8i0cwiXU5uZLysFGk=; b=QPkliBquFyO6AiJa3l2LWYO4ADRpbpn1ixYEciCg9Q0ThM+KxV30J5EeIeKYdnshE+ SFEpTofBBnm9DCeMXRF/Ji7dPyTHLNEPFRV9osuYM3QM2+z3PQ6n563Hvlpxqa1cva97 U83PAE6AWBLqRIGQ/iPPf/ynS+Q0fbeBoDnXTLZGPoyTrnZbKziFi4vi+XjEMGyPhoDv Wz4tuf96C6TsQ+VdgZGROy/cnhw8dCwg1ncCgkQw4Q96VC0n1T7qIQ/MEGPZ+8u0p+5Y O0WX4a9r0bnZtXihIwIGPf2PiO5/YhTdVDd0qT/KKXZX0Dzqr29MOlR91hrrkjI5kmXl FmMA== X-Forwarded-Encrypted: i=1; AJvYcCVtjgEBrSzxWO0uy2P1rcuvJFtMJqoa1CVxFX8bhfCgQHjDnLpz4S2IU8kwSXwRzR6OqV/gA1IM2ekNDUw=@vger.kernel.org X-Gm-Message-State: AOJu0YynyS+aic95MTSE/cWn8jdBQKijHGKmLipSndlpB0x4DmJ6JOdi we3F/enbV9O2psl+o+tgODt0U+95EWZqYq0qg/kOVupmQokArchiKfRAEgL0Mw== X-Gm-Gg: ASbGnctqEXF0BU0eLUpvuvuVuYWgvSHPhID+ih7DYU5KFKBLfIQ+zjTCQt6pT2tIjBH 1c1x2nw/LlA+z7x9OZbgrUxmI5nel+tqDnfOt3Kw5DTY0hbhzSzJENmlpPnWX7rZIlJhmi0oHcB JGMBgiGb4ei/xdmwNODMuSBpNTbdg2bq9POf6Co6jE8EKN38zbdxwrxkiqfLo+zdDScoSnJTwbB zLiivzHSWUoicjKDreapbFGKBmmwKobwK3Da4sgZlt79sh6ul/2HDkysadzrqn5m3372U+Xf5qf CoYRi/nsZtiCKGm7ZA== X-Google-Smtp-Source: AGHT+IGBqbR5rsHnduepMOHV/sDsvhxru/3OZYfP4l3JaDq83WR0FHW+hFPpJFYdg17e8CIob6LH9g== X-Received: by 2002:a05:6a00:1804:b0:732:24ad:8e15 with SMTP id d2e1a72fcca58-7322c39ac1amr3180602b3a.11.1739341982467; Tue, 11 Feb 2025 22:33:02 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:69f5:6852:451e:8142]) by smtp.gmail.com with UTF8SMTPSA id d2e1a72fcca58-7308e22e7e3sm4826244b3a.6.2025.02.11.22.33.00 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 11 Feb 2025 22:33:02 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Kairui Song , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v5 12/18] zsmalloc: make zspage lock preemptible Date: Wed, 12 Feb 2025 15:27:10 +0900 Message-ID: <20250212063153.179231-13-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.502.g6dc24dfdaf-goog In-Reply-To: <20250212063153.179231-1-senozhatsky@chromium.org> References: <20250212063153.179231-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Switch over from rwlock_t to a atomic_t variable that takes negative value when the page is under migration, or positive values when the page is used by zsmalloc users (object map, etc.) Using a rwsem per-zspage is a little too memory heavy, a simple atomic_t should suffice. zspage lock is a leaf lock for zs_map_object(), where it's read-acquired. Since this lock now permits preemption extra care needs to be taken when it is write-acquired - all writers grab it in atomic context, so they cannot spin and wait for (potentially preempted) reader to unlock zspage. There are only two writers at this moment - migration and compaction. In both cases we use write-try-lock and bail out if zspage is read locked. Writers, on the other hand, never get preempted, so readers can spin waiting for the writer to unlock zspage. With this we can implement a preemptible object mapping. Signed-off-by: Sergey Senozhatsky Cc: Yosry Ahmed --- mm/zsmalloc.c | 183 +++++++++++++++++++++++++++++++++++--------------- 1 file changed, 128 insertions(+), 55 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index c82c24b8e6a4..80261bb78cf8 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -226,6 +226,9 @@ struct zs_pool { /* protect page/zspage migration */ rwlock_t lock; atomic_t compaction_in_progress; +#ifdef CONFIG_DEBUG_LOCK_ALLOC + struct lock_class_key lockdep_key; +#endif }; =20 static void pool_write_unlock(struct zs_pool *pool) @@ -292,6 +295,9 @@ static inline void free_zpdesc(struct zpdesc *zpdesc) __free_page(page); } =20 +#define ZS_PAGE_UNLOCKED 0 +#define ZS_PAGE_WRLOCKED -1 + struct zspage { struct { unsigned int huge:HUGE_BITS; @@ -304,7 +310,11 @@ struct zspage { struct zpdesc *first_zpdesc; struct list_head list; /* fullness list */ struct zs_pool *pool; - rwlock_t lock; + atomic_t lock; + +#ifdef CONFIG_DEBUG_LOCK_ALLOC + struct lockdep_map lockdep_map; +#endif }; =20 struct mapping_area { @@ -314,6 +324,88 @@ struct mapping_area { enum zs_mapmode vm_mm; /* mapping mode */ }; =20 +static void zspage_lock_init(struct zspage *zspage) +{ +#ifdef CONFIG_DEBUG_LOCK_ALLOC + lockdep_init_map(&zspage->lockdep_map, "zsmalloc-page", + &zspage->pool->lockdep_key, 0); +#endif + + atomic_set(&zspage->lock, ZS_PAGE_UNLOCKED); +} + +/* + * zspage locking rules: + * + * 1) writer-lock is exclusive + * + * 2) writer-lock owner cannot sleep + * + * 3) writer-lock owner cannot spin waiting for the lock + * - caller (e.g. compaction and migration) must check return value and + * handle locking failures + * - there is only TRY variant of writer-lock function + * + * 4) reader-lock owners (multiple) can sleep + * + * 5) reader-lock owners can spin waiting for the lock, in any context + * - existing readers (even preempted ones) don't block new readers + * - writer-lock owners never sleep, always unlock at some point + */ +static void zspage_read_lock(struct zspage *zspage) +{ + atomic_t *lock =3D &zspage->lock; + int old =3D atomic_read_acquire(lock); + +#ifdef CONFIG_DEBUG_LOCK_ALLOC + rwsem_acquire_read(&zspage->lockdep_map, 0, 0, _RET_IP_); +#endif + + do { + if (old =3D=3D ZS_PAGE_WRLOCKED) { + cpu_relax(); + old =3D atomic_read_acquire(lock); + continue; + } + } while (!atomic_try_cmpxchg_acquire(lock, &old, old + 1)); +} + +static void zspage_read_unlock(struct zspage *zspage) +{ +#ifdef CONFIG_DEBUG_LOCK_ALLOC + rwsem_release(&zspage->lockdep_map, _RET_IP_); +#endif + atomic_dec_return_release(&zspage->lock); +} + +static __must_check bool zspage_try_write_lock(struct zspage *zspage) +{ + atomic_t *lock =3D &zspage->lock; + int old =3D ZS_PAGE_UNLOCKED; + + WARN_ON_ONCE(preemptible()); + + preempt_disable(); + if (atomic_try_cmpxchg_acquire(lock, &old, ZS_PAGE_WRLOCKED)) { +#ifdef CONFIG_DEBUG_LOCK_ALLOC + rwsem_acquire(&zspage->lockdep_map, 0, 1, _RET_IP_); +#endif + return true; + } + + preempt_enable(); + return false; +} + +static void zspage_write_unlock(struct zspage *zspage) +{ +#ifdef CONFIG_DEBUG_LOCK_ALLOC + rwsem_release(&zspage->lockdep_map, _RET_IP_); +#endif + atomic_set_release(&zspage->lock, ZS_PAGE_UNLOCKED); + preempt_enable(); +} + /* huge object: pages_per_zspage =3D=3D 1 && maxobj_per_zspage =3D=3D 1 */ static void SetZsHugePage(struct zspage *zspage) { @@ -325,12 +417,6 @@ static bool ZsHugePage(struct zspage *zspage) return zspage->huge; } =20 -static void migrate_lock_init(struct zspage *zspage); -static void migrate_read_lock(struct zspage *zspage); -static void migrate_read_unlock(struct zspage *zspage); -static void migrate_write_lock(struct zspage *zspage); -static void migrate_write_unlock(struct zspage *zspage); - #ifdef CONFIG_COMPACTION static void kick_deferred_free(struct zs_pool *pool); static void init_deferred_free(struct zs_pool *pool); @@ -1026,7 +1112,9 @@ static struct zspage *alloc_zspage(struct zs_pool *po= ol, return NULL; =20 zspage->magic =3D ZSPAGE_MAGIC; - migrate_lock_init(zspage); + zspage->pool =3D pool; + zspage->class =3D class->index; + zspage_lock_init(zspage); =20 for (i =3D 0; i < class->pages_per_zspage; i++) { struct zpdesc *zpdesc; @@ -1049,8 +1137,6 @@ static struct zspage *alloc_zspage(struct zs_pool *po= ol, =20 create_page_chain(class, zspage, zpdescs); init_zspage(class, zspage); - zspage->pool =3D pool; - zspage->class =3D class->index; =20 return zspage; } @@ -1251,7 +1337,7 @@ void *zs_map_object(struct zs_pool *pool, unsigned lo= ng handle, * zs_unmap_object API so delegate the locking from class to zspage * which is smaller granularity. */ - migrate_read_lock(zspage); + zspage_read_lock(zspage); pool_read_unlock(pool); =20 class =3D zspage_class(pool, zspage); @@ -1311,7 +1397,7 @@ void zs_unmap_object(struct zs_pool *pool, unsigned l= ong handle) } local_unlock(&zs_map_area.lock); =20 - migrate_read_unlock(zspage); + zspage_read_unlock(zspage); } EXPORT_SYMBOL_GPL(zs_unmap_object); =20 @@ -1705,18 +1791,18 @@ static void lock_zspage(struct zspage *zspage) /* * Pages we haven't locked yet can be migrated off the list while we're * trying to lock them, so we need to be careful and only attempt to - * lock each page under migrate_read_lock(). Otherwise, the page we lock + * lock each page under zspage_read_lock(). Otherwise, the page we lock * may no longer belong to the zspage. This means that we may wait for * the wrong page to unlock, so we must take a reference to the page - * prior to waiting for it to unlock outside migrate_read_lock(). + * prior to waiting for it to unlock outside zspage_read_lock(). */ while (1) { - migrate_read_lock(zspage); + zspage_read_lock(zspage); zpdesc =3D get_first_zpdesc(zspage); if (zpdesc_trylock(zpdesc)) break; zpdesc_get(zpdesc); - migrate_read_unlock(zspage); + zspage_read_unlock(zspage); zpdesc_wait_locked(zpdesc); zpdesc_put(zpdesc); } @@ -1727,41 +1813,16 @@ static void lock_zspage(struct zspage *zspage) curr_zpdesc =3D zpdesc; } else { zpdesc_get(zpdesc); - migrate_read_unlock(zspage); + zspage_read_unlock(zspage); zpdesc_wait_locked(zpdesc); zpdesc_put(zpdesc); - migrate_read_lock(zspage); + zspage_read_lock(zspage); } } - migrate_read_unlock(zspage); + zspage_read_unlock(zspage); } #endif /* CONFIG_COMPACTION */ =20 -static void migrate_lock_init(struct zspage *zspage) -{ - rwlock_init(&zspage->lock); -} - -static void migrate_read_lock(struct zspage *zspage) __acquires(&zspage->l= ock) -{ - read_lock(&zspage->lock); -} - -static void migrate_read_unlock(struct zspage *zspage) __releases(&zspage-= >lock) -{ - read_unlock(&zspage->lock); -} - -static void migrate_write_lock(struct zspage *zspage) -{ - write_lock(&zspage->lock); -} - -static void migrate_write_unlock(struct zspage *zspage) -{ - write_unlock(&zspage->lock); -} - #ifdef CONFIG_COMPACTION =20 static const struct movable_operations zsmalloc_mops; @@ -1803,7 +1864,7 @@ static bool zs_page_isolate(struct page *page, isolat= e_mode_t mode) } =20 static int zs_page_migrate(struct page *newpage, struct page *page, - enum migrate_mode mode) + enum migrate_mode mode) { struct zs_pool *pool; struct size_class *class; @@ -1819,15 +1880,12 @@ static int zs_page_migrate(struct page *newpage, st= ruct page *page, =20 VM_BUG_ON_PAGE(!zpdesc_is_isolated(zpdesc), zpdesc_page(zpdesc)); =20 - /* We're committed, tell the world that this is a Zsmalloc page. */ - __zpdesc_set_zsmalloc(newzpdesc); - /* The page is locked, so this pointer must remain valid */ zspage =3D get_zspage(zpdesc); pool =3D zspage->pool; =20 /* - * The pool lock protects the race between zpage migration + * The pool->lock protects the race between zpage migration * and zs_free. */ pool_write_lock(pool); @@ -1837,8 +1895,15 @@ static int zs_page_migrate(struct page *newpage, str= uct page *page, * the class lock protects zpage alloc/free in the zspage. */ size_class_lock(class); - /* the migrate_write_lock protects zpage access via zs_map_object */ - migrate_write_lock(zspage); + /* the zspage write_lock protects zpage access via zs_map_object */ + if (!zspage_try_write_lock(zspage)) { + size_class_unlock(class); + pool_write_unlock(pool); + return -EINVAL; + } + + /* We're committed, tell the world that this is a Zsmalloc page. */ + __zpdesc_set_zsmalloc(newzpdesc); =20 offset =3D get_first_obj_offset(zpdesc); s_addr =3D kmap_local_zpdesc(zpdesc); @@ -1869,7 +1934,7 @@ static int zs_page_migrate(struct page *newpage, stru= ct page *page, */ pool_write_unlock(pool); size_class_unlock(class); - migrate_write_unlock(zspage); + zspage_write_unlock(zspage); =20 zpdesc_get(newzpdesc); if (zpdesc_zone(newzpdesc) !=3D zpdesc_zone(zpdesc)) { @@ -2005,9 +2070,11 @@ static unsigned long __zs_compact(struct zs_pool *po= ol, if (!src_zspage) break; =20 - migrate_write_lock(src_zspage); + if (!zspage_try_write_lock(src_zspage)) + break; + migrate_zspage(pool, src_zspage, dst_zspage); - migrate_write_unlock(src_zspage); + zspage_write_unlock(src_zspage); =20 fg =3D putback_zspage(class, src_zspage); if (fg =3D=3D ZS_INUSE_RATIO_0) { @@ -2267,7 +2334,9 @@ struct zs_pool *zs_create_pool(const char *name) * trigger compaction manually. Thus, ignore return code. */ zs_register_shrinker(pool); - +#ifdef CONFIG_DEBUG_LOCK_ALLOC + lockdep_register_key(&pool->lockdep_key); +#endif return pool; =20 err: @@ -2304,6 +2373,10 @@ void zs_destroy_pool(struct zs_pool *pool) kfree(class); } =20 +#ifdef CONFIG_DEBUG_LOCK_ALLOC + lockdep_unregister_key(&pool->lockdep_key); +#endif + destroy_cache(pool); kfree(pool->name); kfree(pool); --=20 2.48.1.502.g6dc24dfdaf-goog From nobody Thu Dec 18 10:00:39 2025 Received: from mail-pj1-f53.google.com (mail-pj1-f53.google.com [209.85.216.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 120AC202C38 for ; Wed, 12 Feb 2025 06:33:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.53 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739341989; cv=none; b=aYwRrXXB4P0u7GEYzckJD2HnfplpCv4QrWNU1mYN27Q+Lq9DGdeNaRb2sHAGq4kpsMIT5ZDj0E5jpppdD59plODGs+5QmV2I3QCr2wjloZw55z0m2FGepKlkxxELNo7i1d6h7Yp4ncpOZ+ziQs7JR0eIG74c+LnfKKqh3ByZCA4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739341989; c=relaxed/simple; bh=MECIf4qWOwJd7TIEwrJXpz6WlwoNwT2TreIcypaxWhw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=TIDZUruxg+wrfTi0KH6dv4scP+Pj7EwTgFYp6tdGKWrxSVlZOnAqFC9BBCjSgSprJkaHLYyZqf+C79c6N9FROZfecsxy3xoxZyNSU4Nsx8PIXSRyr5huNQmJaLMMifFheJUqT0bX7I2DjjumVcKG9lN8HUnJ4L4ptANyDZOxKgs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=ZTA+2Lrh; arc=none smtp.client-ip=209.85.216.53 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="ZTA+2Lrh" Received: by mail-pj1-f53.google.com with SMTP id 98e67ed59e1d1-2f9b91dff71so9788183a91.2 for ; Tue, 11 Feb 2025 22:33:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1739341987; x=1739946787; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=dCEIYmpOSL5kMgSv8CIDbNXJeiLt2Oo14JvQ8JkkWQI=; b=ZTA+2LrhDixHxSDj7AMNJsRZ6De2gr00YKV1anoCApv5AFkxXDQqLkbKyclSg8n4/G KOKIVLqN54EKm63rTnqCjB96f2GTOkoHFft51FR7W86rmxl51ynSa2nrTT2toyBrpvTv Uh0HYyrEjsst155jNx0/mMwLMQapywnlh2798= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739341987; x=1739946787; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=dCEIYmpOSL5kMgSv8CIDbNXJeiLt2Oo14JvQ8JkkWQI=; b=qg1smVDaxAGy52eWaFYm5BhJvt7Hb2twaqnP4Sg0o/gr6/LGq1xOWJF3kYt38tPme+ Y4R5NnUP5aLoPlSdIcWyeFE18X+Ha5l0cG2ZARKFhOgrckg3bapw61A+uHDilm7QCoUH HFGX4J2ppAARPCDz3JnGFH5yZo4KfNvD4YguRqh5ow+Ers7yl7+AIT2ILczj/KzILzzc s4fOAYdtHvZwheMQ9Z2Tv49zO4tz3S76AlrNCW9fD6MTczVr0FsEUAaElfWIBvsC+k5w I1AQWlbQLxQC9gb2WCMkkZZMF1DPUgzQ4yWBt1ik02Aqj89EfYB2r76dB/Fi1kUZ+TY1 b0JA== X-Forwarded-Encrypted: i=1; AJvYcCVTVbbBTR8oO6FYo1RGhKH+ie8wiaT5NVgXKID31CoPpzbsA/PmCE7k5296w7ljGKwLwpeni1KsJlzVxVs=@vger.kernel.org X-Gm-Message-State: AOJu0YyuPXBd1dGWL1O+fr9q6MmdSwSX61tYMag8thMFEStI0W5GFnrw Lkw59kTsRewSO2mOkX60DBkbSUl20EnlbtRfsdlK4vTvm1OGk5tjg6SwJyN4rw== X-Gm-Gg: ASbGncv810djD0OCeOHQblqvW0rNMAM8LXGbIkToujIHf6YvX4RnWVBeyHgzpN4Cc3x wRoN18PMzW9fuo7Ks1XbaNM2LCtcOzYscUJMlq7JW+YVfEaN/hhxCR7x7dZM8k1lLHf9zR6PnZQ 7yR79ZD7qjDHScuJmy3066xXDxPVpD/N+cwFYBhVrVrZsaiXSEXGgRDSVxAyWGKO2yqhywfFqjb b2sx/gUlucgX8dN+rOhdtRwXT3KEhcv46Y3B77tiv1l9CVDpJLma8zQoVYSBm4yOYUSZhR45D+Z C1SFmPzvK3cguYaDdQ== X-Google-Smtp-Source: AGHT+IEcBz98pD8dO8lfqL3ofXePIyCx5p8Uf93CWTvB4grPFDM00/5pX1hnLpil/qbdFRArW8IpbA== X-Received: by 2002:a17:90b:568d:b0:2ea:3f34:f18f with SMTP id 98e67ed59e1d1-2fbf5c0f4b5mr2776207a91.19.1739341987103; Tue, 11 Feb 2025 22:33:07 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:69f5:6852:451e:8142]) by smtp.gmail.com with UTF8SMTPSA id 98e67ed59e1d1-2fbf973c42esm672421a91.0.2025.02.11.22.33.04 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 11 Feb 2025 22:33:06 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Kairui Song , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v5 13/18] zsmalloc: introduce new object mapping API Date: Wed, 12 Feb 2025 15:27:11 +0900 Message-ID: <20250212063153.179231-14-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.502.g6dc24dfdaf-goog In-Reply-To: <20250212063153.179231-1-senozhatsky@chromium.org> References: <20250212063153.179231-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Current object mapping API is a little cumbersome. First, it's inconsistent, sometimes it returns with page-faults disabled and sometimes with page-faults enabled. Second, and most importantly, it enforces atomicity restrictions on its users. zs_map_object() has to return a liner object address which is not always possible because some objects span multiple physical (non-contiguous) pages. For such objects zsmalloc uses a per-CPU buffer to which object's data is copied before a pointer to that per-CPU buffer is returned back to the caller. This leads to another, final, issue - extra memcpy(). Since the caller gets a pointer to per-CPU buffer it can memcpy() data only to that buffer, and during zs_unmap_object() zsmalloc will memcpy() from that per-CPU buffer to physical pages that object in question spans across. New API splits functions by access mode: - zs_obj_read_begin(handle, local_copy) Returns a pointer to handle memory. For objects that span two physical pages a local_copy buffer is used to store object's data before the address is returned to the caller. Otherwise the object's page is kmap_local mapped directly. - zs_obj_read_end(handle, buf) Unmaps the page if it was kmap_local mapped by zs_obj_read_begin(). - zs_obj_write(handle, buf, len) Copies len-bytes from compression buffer to handle memory (takes care of objects that span two pages). This does not need any additional (e.g. per-CPU) buffers and writes the data directly to zsmalloc pool pages. In terms of performance, on a synthetic and completely reproducible test that allocates fixed number of objects of fixed sizes and iterates over those objects, first mapping in RO then in RW mode: OLD API =3D=3D=3D=3D=3D=3D=3D 10 runs 369,205,778 instructions # 0.80 insn per cycle 40,467,926 branches # 113.732 M/sec 369,002,122 instructions # 0.62 insn per cycle 40,426,145 branches # 189.361 M/sec 369,036,706 instructions # 0.63 insn per cycle 40,430,860 branches # 204.105 M/sec [..] NEW API =3D=3D=3D=3D=3D=3D=3D 10 runs 265,799,293 instructions # 0.51 insn per cycle 29,834,567 branches # 170.281 M/sec 265,765,970 instructions # 0.55 insn per cycle 29,829,019 branches # 161.602 M/sec 265,764,702 instructions # 0.51 insn per cycle 29,828,015 branches # 189.677 M/sec [..] Difference at 95.0% confidence -1.03219e+08 +/- 55308.7 -27.9705% +/- 0.0149878% (Student's t, pooled s =3D 58864.4) The old API will stay around until the remaining users switch to the new one. After that we'll also remove zsmalloc per-CPU buffer and CPU hotplug handling. Signed-off-by: Sergey Senozhatsky Reviewed-by: Yosry Ahmed --- include/linux/zsmalloc.h | 8 +++ mm/zsmalloc.c | 129 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 137 insertions(+) diff --git a/include/linux/zsmalloc.h b/include/linux/zsmalloc.h index a48cd0ffe57d..7d70983cf398 100644 --- a/include/linux/zsmalloc.h +++ b/include/linux/zsmalloc.h @@ -58,4 +58,12 @@ unsigned long zs_compact(struct zs_pool *pool); unsigned int zs_lookup_class_index(struct zs_pool *pool, unsigned int size= ); =20 void zs_pool_stats(struct zs_pool *pool, struct zs_pool_stats *stats); + +void *zs_obj_read_begin(struct zs_pool *pool, unsigned long handle, + void *local_copy); +void zs_obj_read_end(struct zs_pool *pool, unsigned long handle, + void *handle_mem); +void zs_obj_write(struct zs_pool *pool, unsigned long handle, + void *handle_mem, size_t mem_len); + #endif diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 80261bb78cf8..e40268f3b655 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -1401,6 +1401,135 @@ void zs_unmap_object(struct zs_pool *pool, unsigned= long handle) } EXPORT_SYMBOL_GPL(zs_unmap_object); =20 +void *zs_obj_read_begin(struct zs_pool *pool, unsigned long handle, + void *local_copy) +{ + struct zspage *zspage; + struct zpdesc *zpdesc; + unsigned long obj, off; + unsigned int obj_idx; + struct size_class *class; + void *addr; + + WARN_ON(in_interrupt()); + + /* Guarantee we can get zspage from handle safely */ + pool_read_lock(pool); + obj =3D handle_to_obj(handle); + obj_to_location(obj, &zpdesc, &obj_idx); + zspage =3D get_zspage(zpdesc); + + /* Make sure migration doesn't move any pages in this zspage */ + zspage_read_lock(zspage); + pool_read_unlock(pool); + + class =3D zspage_class(pool, zspage); + off =3D offset_in_page(class->size * obj_idx); + + if (off + class->size <=3D PAGE_SIZE) { + /* this object is contained entirely within a page */ + addr =3D kmap_local_zpdesc(zpdesc); + addr +=3D off; + } else { + size_t sizes[2]; + + /* this object spans two pages */ + sizes[0] =3D PAGE_SIZE - off; + sizes[1] =3D class->size - sizes[0]; + addr =3D local_copy; + + memcpy_from_page(addr, zpdesc_page(zpdesc), + off, sizes[0]); + zpdesc =3D get_next_zpdesc(zpdesc); + memcpy_from_page(addr + sizes[0], + zpdesc_page(zpdesc), + 0, sizes[1]); + } + + if (!ZsHugePage(zspage)) + addr +=3D ZS_HANDLE_SIZE; + + return addr; +} +EXPORT_SYMBOL_GPL(zs_obj_read_begin); + +void zs_obj_read_end(struct zs_pool *pool, unsigned long handle, + void *handle_mem) +{ + struct zspage *zspage; + struct zpdesc *zpdesc; + unsigned long obj, off; + unsigned int obj_idx; + struct size_class *class; + + obj =3D handle_to_obj(handle); + obj_to_location(obj, &zpdesc, &obj_idx); + zspage =3D get_zspage(zpdesc); + class =3D zspage_class(pool, zspage); + off =3D offset_in_page(class->size * obj_idx); + + if (off + class->size <=3D PAGE_SIZE) { + if (!ZsHugePage(zspage)) + off +=3D ZS_HANDLE_SIZE; + handle_mem -=3D off; + kunmap_local(handle_mem); + } + + zspage_read_unlock(zspage); +} +EXPORT_SYMBOL_GPL(zs_obj_read_end); + +void zs_obj_write(struct zs_pool *pool, unsigned long handle, + void *handle_mem, size_t mem_len) +{ + struct zspage *zspage; + struct zpdesc *zpdesc; + unsigned long obj, off; + unsigned int obj_idx; + struct size_class *class; + + WARN_ON(in_interrupt()); + + /* Guarantee we can get zspage from handle safely */ + pool_read_lock(pool); + obj =3D handle_to_obj(handle); + obj_to_location(obj, &zpdesc, &obj_idx); + zspage =3D get_zspage(zpdesc); + + /* Make sure migration doesn't move any pages in this zspage */ + zspage_read_lock(zspage); + pool_read_unlock(pool); + + class =3D zspage_class(pool, zspage); + off =3D offset_in_page(class->size * obj_idx); + + if (off + class->size <=3D PAGE_SIZE) { + /* this object is contained entirely within a page */ + void *dst =3D kmap_local_zpdesc(zpdesc); + + if (!ZsHugePage(zspage)) + off +=3D ZS_HANDLE_SIZE; + memcpy(dst + off, handle_mem, mem_len); + kunmap_local(dst); + } else { + /* this object spans two pages */ + size_t sizes[2]; + + off +=3D ZS_HANDLE_SIZE; + sizes[0] =3D PAGE_SIZE - off; + sizes[1] =3D mem_len - sizes[0]; + + memcpy_to_page(zpdesc_page(zpdesc), off, + handle_mem, sizes[0]); + zpdesc =3D get_next_zpdesc(zpdesc); + memcpy_to_page(zpdesc_page(zpdesc), 0, + handle_mem + sizes[0], sizes[1]); + } + + zspage_read_unlock(zspage); +} +EXPORT_SYMBOL_GPL(zs_obj_write); + /** * zs_huge_class_size() - Returns the size (in bytes) of the first huge * zsmalloc &size_class. --=20 2.48.1.502.g6dc24dfdaf-goog From nobody Thu Dec 18 10:00:39 2025 Received: from mail-pl1-f177.google.com (mail-pl1-f177.google.com [209.85.214.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 37FCE202C5B for ; Wed, 12 Feb 2025 06:33:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739341994; cv=none; b=B4i/XKTtKIu74daiTv/n9GEtO+eIMQnCJjVf7J/6aHo7Ee1uxUROTe1dOd3IeVMLRWW4ieJzotJX8Fp+DdLOSQ2KDvehF6TGCXRRDY00vG8eGKmuQqwlSLtOr/uRibYT9NKwjqmqpXNO0wPVhEXLjMBOx3qcRhR9fQiqQ1wId6c= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739341994; c=relaxed/simple; bh=mgqECS8dn1xjq1nP602YrqXmI98zspRzEWQlMNgS6hQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ooX2RasKpp9Of0CpaWW2pLAehOHiJs+RQFamLst4bIk9k014cKFgTbdZ92u46rkNxha7tnGPgo54MxkPTH8FA6vkB5siBVYqOONyjkI83Lakau9vLusigrd/X4+ehpcSTh+3Wu7dHCscBXIEpWQC5mBjUb0wVQRcMP0w1mRpDr0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=FjGFCQu5; arc=none smtp.client-ip=209.85.214.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="FjGFCQu5" Received: by mail-pl1-f177.google.com with SMTP id d9443c01a7336-21f61b01630so7545725ad.1 for ; Tue, 11 Feb 2025 22:33:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1739341992; x=1739946792; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=G+0Z+4xYmRhftKTrRiF3jNVLaBZmv0cRLATopAZYYvQ=; b=FjGFCQu5vkf9eJ2ifR89MTduobd7oVmA+jY7RuXXdmFzfuFL63iv4C8Ghjn6hFMjod rUj4NN1ueITMRxY3u1XeN66uKhraRVsE1GzBZ7jb7Z4OEgw8Rv/IyJKJnTAuGw6Jw2H5 U3YemutVkUpL2hT7BTOcUbVScgpfqxzkiGNw4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739341992; x=1739946792; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=G+0Z+4xYmRhftKTrRiF3jNVLaBZmv0cRLATopAZYYvQ=; b=WI3v8II1HpGu1mCdAFb3vmabGBtjDAe4Sg0ZsaDhgrTno1M751Y26L89hc8KQtxd/5 hczmLpSP/FDaiBEhC3rizfqHgZF5ywcUbAWvlNlpnufYqZxI52ji1g2Hxl5P597Wb8s+ o/CHSq7z4CDM3Ff4xWSQ+Tppba2eKXveiDovowzPbB3AZpDPRGQWZduVtV2tv0egzuWk tS+bdYTEN1fEouMEE1ORf5Q0rxQBBcXM61Fj1nQ9jcAccbk3NUmKL2w6W8t4BHbx5/vG joRHS21gmFOio8MDRx8X2KKl5w3WNyGZK+bBWhfuSKNHq2/R09ToxfJqzIxSuiGzLb+S q6CQ== X-Forwarded-Encrypted: i=1; AJvYcCWlyAvANFrv62T18kv/38l4/hsiXMV7mkaJX1PdclPnTF+FFGfKcxcmWm6x2luMvL29A3e66267yQxOHYc=@vger.kernel.org X-Gm-Message-State: AOJu0Yy3BvKZ9maDJ0aVnNlsfmfJNAxBF8A8nNg098R0DJqvG8skAXBt 39SvogcCtg8f0sTKet1wf8cbe69ouqNsqOML0dCI598I8Q8Bp2gOmBFffOG9dw== X-Gm-Gg: ASbGncuvlZmm9ZDj8Isg4UfpLyiZpfOx/c8FGa+829jSEBofyYeu4mUM8g0ZXi+ZZF4 0jgetfs43OX5naF79AQ8qWuyyiaiVg20jOXNKs2mbVhULrIIF31UNI4fFmE4Z9/qw2ZKswn1j7w 0G46M1XgbmHrgn7tUEj9AQIIwszD3RVUToW+E8ZAYo6yzJXJOWy2/Gx62U2oluhS9nucAlTh63P HTaPHQknAzSc2FJKXrUdS26UhcSAu0Ud0/oBdsbsXt2ef5zDLKGa2iuv4f5US13cc+fnpGqGSFW KHnmUyF7LnZ756Hq1Q== X-Google-Smtp-Source: AGHT+IEhnNCxLoE2CNoFgq8/HUot/UmlliKB3Yg2LwQwc2F/d9hoWw6hVmqWymhRbVtPXN7CHslIlw== X-Received: by 2002:a05:6a21:9101:b0:1e0:d3e9:1f8 with SMTP id adf61e73a8af0-1ee5cedffe2mr2971435637.10.1739341992642; Tue, 11 Feb 2025 22:33:12 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:69f5:6852:451e:8142]) by smtp.gmail.com with UTF8SMTPSA id d2e1a72fcca58-73093ab315csm4424643b3a.143.2025.02.11.22.33.10 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 11 Feb 2025 22:33:12 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Kairui Song , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v5 14/18] zram: switch to new zsmalloc object mapping API Date: Wed, 12 Feb 2025 15:27:12 +0900 Message-ID: <20250212063153.179231-15-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.502.g6dc24dfdaf-goog In-Reply-To: <20250212063153.179231-1-senozhatsky@chromium.org> References: <20250212063153.179231-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Use new read/write zsmalloc object API. For cases when RO mapped object spans two physical pages (requires temp buffer) compression streams now carry around one extra physical page. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zcomp.c | 4 +++- drivers/block/zram/zcomp.h | 2 ++ drivers/block/zram/zram_drv.c | 28 ++++++++++------------------ 3 files changed, 15 insertions(+), 19 deletions(-) diff --git a/drivers/block/zram/zcomp.c b/drivers/block/zram/zcomp.c index c393243eeb5c..61a9c3ed6f7a 100644 --- a/drivers/block/zram/zcomp.c +++ b/drivers/block/zram/zcomp.c @@ -46,6 +46,7 @@ static const struct zcomp_ops *backends[] =3D { static void zcomp_strm_free(struct zcomp *comp, struct zcomp_strm *zstrm) { comp->ops->destroy_ctx(&zstrm->ctx); + vfree(zstrm->local_copy); vfree(zstrm->buffer); zstrm->buffer =3D NULL; } @@ -59,12 +60,13 @@ static int zcomp_strm_init(struct zcomp *comp, struct z= comp_strm *zstrm) if (ret) return ret; =20 + zstrm->local_copy =3D vzalloc(PAGE_SIZE); /* * allocate 2 pages. 1 for compressed data, plus 1 extra for the * case when compressed size is larger than the original one */ zstrm->buffer =3D vzalloc(2 * PAGE_SIZE); - if (!zstrm->buffer) { + if (!zstrm->buffer || !zstrm->local_copy) { zcomp_strm_free(comp, zstrm); return -ENOMEM; } diff --git a/drivers/block/zram/zcomp.h b/drivers/block/zram/zcomp.h index 23b8236b9090..25339ed1e07e 100644 --- a/drivers/block/zram/zcomp.h +++ b/drivers/block/zram/zcomp.h @@ -34,6 +34,8 @@ struct zcomp_strm { struct mutex lock; /* compression buffer */ void *buffer; + /* local copy of handle memory */ + void *local_copy; struct zcomp_ctx ctx; }; =20 diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 7c4c296181a8..c6310077c221 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1561,11 +1561,11 @@ static int read_incompressible_page(struct zram *zr= am, struct page *page, void *src, *dst; =20 handle =3D zram_get_handle(zram, index); - src =3D zs_map_object(zram->mem_pool, handle, ZS_MM_RO); + src =3D zs_obj_read_begin(zram->mem_pool, handle, NULL); dst =3D kmap_local_page(page); copy_page(dst, src); kunmap_local(dst); - zs_unmap_object(zram->mem_pool, handle); + zs_obj_read_end(zram->mem_pool, handle, src); =20 return 0; } @@ -1583,11 +1583,11 @@ static int read_compressed_page(struct zram *zram, = struct page *page, u32 index) prio =3D zram_get_priority(zram, index); =20 zstrm =3D zcomp_stream_get(zram->comps[prio]); - src =3D zs_map_object(zram->mem_pool, handle, ZS_MM_RO); + src =3D zs_obj_read_begin(zram->mem_pool, handle, zstrm->local_copy); dst =3D kmap_local_page(page); ret =3D zcomp_decompress(zram->comps[prio], zstrm, src, size, dst); kunmap_local(dst); - zs_unmap_object(zram->mem_pool, handle); + zs_obj_read_end(zram->mem_pool, handle, src); zcomp_stream_put(zstrm); =20 return ret; @@ -1683,7 +1683,7 @@ static int write_incompressible_page(struct zram *zra= m, struct page *page, u32 index) { unsigned long handle; - void *src, *dst; + void *src; =20 /* * This function is called from preemptible context so we don't need @@ -1700,11 +1700,9 @@ static int write_incompressible_page(struct zram *zr= am, struct page *page, return -ENOMEM; } =20 - dst =3D zs_map_object(zram->mem_pool, handle, ZS_MM_WO); src =3D kmap_local_page(page); - memcpy(dst, src, PAGE_SIZE); + zs_obj_write(zram->mem_pool, handle, src, PAGE_SIZE); kunmap_local(src); - zs_unmap_object(zram->mem_pool, handle); =20 zram_slot_lock(zram, index); zram_set_flag(zram, index, ZRAM_HUGE); @@ -1725,7 +1723,7 @@ static int zram_write_page(struct zram *zram, struct = page *page, u32 index) int ret =3D 0; unsigned long handle; unsigned int comp_len; - void *dst, *mem; + void *mem; struct zcomp_strm *zstrm; unsigned long element; bool same_filled; @@ -1771,11 +1769,8 @@ static int zram_write_page(struct zram *zram, struct= page *page, u32 index) return -ENOMEM; } =20 - dst =3D zs_map_object(zram->mem_pool, handle, ZS_MM_WO); - - memcpy(dst, zstrm->buffer, comp_len); + zs_obj_write(zram->mem_pool, handle, zstrm->buffer, comp_len); zcomp_stream_put(zstrm); - zs_unmap_object(zram->mem_pool, handle); =20 zram_slot_lock(zram, index); zram_set_handle(zram, index, handle); @@ -1887,7 +1882,7 @@ static int recompress_slot(struct zram *zram, u32 ind= ex, struct page *page, unsigned int comp_len_new; unsigned int class_index_old; unsigned int class_index_new; - void *src, *dst; + void *src; int ret =3D 0; =20 handle_old =3D zram_get_handle(zram, index); @@ -1997,12 +1992,9 @@ static int recompress_slot(struct zram *zram, u32 in= dex, struct page *page, return PTR_ERR((void *)handle_new); } =20 - dst =3D zs_map_object(zram->mem_pool, handle_new, ZS_MM_WO); - memcpy(dst, zstrm->buffer, comp_len_new); + zs_obj_write(zram->mem_pool, handle_new, zstrm->buffer, comp_len_new); zcomp_stream_put(zstrm); =20 - zs_unmap_object(zram->mem_pool, handle_new); - zram_free_page(zram, index); zram_set_handle(zram, index, handle_new); zram_set_obj_size(zram, index, comp_len_new); --=20 2.48.1.502.g6dc24dfdaf-goog From nobody Thu Dec 18 10:00:39 2025 Received: from mail-pl1-f173.google.com (mail-pl1-f173.google.com [209.85.214.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 47C351FF7A9 for ; Wed, 12 Feb 2025 06:33:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739341999; cv=none; b=f8mUHrWuEEi4qEXYERmY8rSmPYhg1UHg9L0rl7ndNrzBgnDvLM/t8B+2XFs6xzbRsb+qOeEHuPdZPnJnJN1ZwS46EfbprPQ77nUgzSBSTtqfTfqEqbtv6FtmYiDpKy6YUu0JAkG5tfIUGxH2aMogb+WBL+L3iAXOO7zmeQRjCO4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739341999; c=relaxed/simple; bh=9QiWxrTVWP0FfMG97AshhuEgBRDy8eLuQAYobyFch/Y=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=QLTgP6iR77ykVUNya4DC4dDoi0Dzeqam/XNPnUo7eeDI5UHAakwkfB+DWaXsG0j7lLdtZqd1zUvtjowsDiN5hg2YKNT7LiQVqTAAp9O2nzDFntHgGgcSUA5l7cmBsY7SFbFWwA6/ab8rpYdQACXeEIlE5qWx8o3zQ5Vc/hFNiaQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=oYksgKIY; arc=none smtp.client-ip=209.85.214.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="oYksgKIY" Received: by mail-pl1-f173.google.com with SMTP id d9443c01a7336-21f44e7eae4so108475635ad.2 for ; Tue, 11 Feb 2025 22:33:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1739341997; x=1739946797; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=zGWR8uwxFV58sLam3cHzynn2zEt595XLKCTA6OHBb4A=; b=oYksgKIYNCO/UBfxvt2D5HMQcEaGUpL2tKH/yoDvNCkDKW4dB55zzLDORN+2Z5Llox yY2e/1ZpeJTcn9j/xyx8p1VRDL7FtHr7tW7MCZYwmf6SThWb50nWJh49HCIHo6TMF8n7 RTyaDwMQQARZCzu5oxxgrQGhLYp0apsz+omF4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739341997; x=1739946797; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=zGWR8uwxFV58sLam3cHzynn2zEt595XLKCTA6OHBb4A=; b=F0Yb4CFUySnQx24GYQZOYdqigjuuuyeWeDcZf3CzuIZXSO3qgMtP5FYY/Uad57b4LS 3EMpACJap8/rB4J9JeGJa+9RSNCBUjA31trW0mvDEvfmGiS+OFI1+BML8CfEX7pBhCDQ B4DJA7RmA6fXY6oivFNMd6Pb01v80eRjic7ky/O5a4wFv3mzcPonRS/6IjJCyibw+Rdi R1ox+exIBeeQLBA9d+d+crqkbneADX+k7o3izIIve9DV2r1SYXkxbqNKlK3Uo3V+9Bki YvUXUICIyqJrHskNCMCjAak4JDRTrQCrvVcNP8aFntiVA67A7ctZRS9tCMrZ4yGZCl0n Sm5A== X-Forwarded-Encrypted: i=1; AJvYcCVR70KVaOikYwYsYWrftM5uuTRLJD481XmmoNSPEPqgqm3zc8jn6a6J8airNvcgBaPMSTJJJixqjajCt3w=@vger.kernel.org X-Gm-Message-State: AOJu0YxfYzFO2zYv5JPWduyWiaOuwmWzo33yVPQ+ivtRiPdVFd9GlzSu hnox2ZEm+FYm28yK/+r4TOwbqyf/mBq7WCVtgarOB3bwQzxbZIrAvLEWakqG0w== X-Gm-Gg: ASbGnctI2Z1NFVfn0js+tp3dcL0csV0V/Pmfo67UxL+y0z+ZbFsHky3sh8VwDNF86uz mLFS2t0w2EVv+1lsC/WWfACGns5uAqyurvOnGAFj6/CboST5rGiIeE9Gs4+s5A1BQTldTY8aJEV gp+D7z+qaPOKkGdqPeDKp6294+vGvHbLeHdUNbX4gWZqiUi1d+EAlT2f3gV6+8sAOlIiliR4IiO vplAGih9mtHlN67lhQXzAGkuxaoDliPtpMaRy5tg7cJX9kBtqyak+/k+k21SDsECnWcrKmB95kW TbtaBZJoY+E0aXA9gA== X-Google-Smtp-Source: AGHT+IFzbH9qNa8XB8LsSRHPJwmV7Dgu2lkCSlxYEmJLzkmFDmA/w+WWTWaak6t0Xswa1OMVGZCLYg== X-Received: by 2002:a05:6a20:728d:b0:1ee:6187:7fb3 with SMTP id adf61e73a8af0-1ee61878104mr1837250637.15.1739341997477; Tue, 11 Feb 2025 22:33:17 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:69f5:6852:451e:8142]) by smtp.gmail.com with UTF8SMTPSA id d2e1a72fcca58-73218c206a6sm2581185b3a.49.2025.02.11.22.33.15 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 11 Feb 2025 22:33:17 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Kairui Song , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v5 15/18] zram: permit reclaim in zstd custom allocator Date: Wed, 12 Feb 2025 15:27:13 +0900 Message-ID: <20250212063153.179231-16-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.502.g6dc24dfdaf-goog In-Reply-To: <20250212063153.179231-1-senozhatsky@chromium.org> References: <20250212063153.179231-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" When configured with pre-trained compression/decompression dictionary support, zstd requires custom memory allocator, which it calls internally from compression()/decompression() routines. That means allocation from atomic context (either under entry spin-lock, or per-CPU local-lock or both). Now, with non-atomic zram read()/write(), those limitations are relaxed and we can allow direct and indirect reclaim. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/backend_zstd.c | 11 +++-------- 1 file changed, 3 insertions(+), 8 deletions(-) diff --git a/drivers/block/zram/backend_zstd.c b/drivers/block/zram/backend= _zstd.c index 1184c0036f44..53431251ea62 100644 --- a/drivers/block/zram/backend_zstd.c +++ b/drivers/block/zram/backend_zstd.c @@ -24,19 +24,14 @@ struct zstd_params { /* * For C/D dictionaries we need to provide zstd with zstd_custom_mem, * which zstd uses internally to allocate/free memory when needed. - * - * This means that allocator.customAlloc() can be called from zcomp_compre= ss() - * under local-lock (per-CPU compression stream), in which case we must use - * GFP_ATOMIC. - * - * Another complication here is that we can be configured as a swap device. */ static void *zstd_custom_alloc(void *opaque, size_t size) { - if (!preemptible()) + /* Technically this should not happen */ + if (WARN_ON_ONCE(!preemptible())) return kvzalloc(size, GFP_ATOMIC); =20 - return kvzalloc(size, __GFP_KSWAPD_RECLAIM | __GFP_NOWARN); + return kvzalloc(size, GFP_NOIO | __GFP_NOWARN); } =20 static void zstd_custom_free(void *opaque, void *address) --=20 2.48.1.502.g6dc24dfdaf-goog From nobody Thu Dec 18 10:00:39 2025 Received: from mail-pl1-f174.google.com (mail-pl1-f174.google.com [209.85.214.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B9FE31FF619 for ; Wed, 12 Feb 2025 06:33:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739342004; cv=none; b=QqukyfkLrexONqCBUFYw/Cd5AhXN3BAAe410cpxk4o49e5xyPUJq/Kbd9MwpDVVy73L7wUrrmZwyde21/ef55RBgO31NMoomP9Kqlua3dKoSQza++VSsAy0EikK3GPAL62S+cUV2EzsI8LNBUdFtbdfyjdyxsJcsn51/iM45o/U= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739342004; c=relaxed/simple; bh=IRNOrDs38GxZoqcpi2Pg4Y3Su9b9UpAHfoJgnAuvVZQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=uJC2QyYIFICch0vaCUc192G8m6LKWHhBGgvfHjDxQ4LXG7mGI6zmJZaU1alwgzyx1KDI9PCny/qemAir8RWJM/wTMuAXI2h5DZyK7PA5WRXrZDyrjOtIZOaXHQJMJSq4lcXZC81lRaRXXgSxi4F0YwGC6nehk1hZqbRh0wapaiw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=HM4EOh65; arc=none smtp.client-ip=209.85.214.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="HM4EOh65" Received: by mail-pl1-f174.google.com with SMTP id d9443c01a7336-21f6d2642faso99524185ad.1 for ; Tue, 11 Feb 2025 22:33:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1739342002; x=1739946802; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=fDJRc+CzjHHBPozwYIKOoKTE7RtcGO1bK7SyyTKpM/I=; b=HM4EOh65SO/ZvQDsQcOul2wug7E+hygDh0muC+GHZAFKeVe/stD22J5UgiLmdPDlDh 2Q0YK9+fw5gQIGgnPtbg17MnD27cQDc/m/2a+XNpNBtVS7uDqe1q3xgI5U/2TRWSOnAx ErzTDmCiOPozMP/UOP1BkJ1yG33aMTgO9SJdQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739342002; x=1739946802; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=fDJRc+CzjHHBPozwYIKOoKTE7RtcGO1bK7SyyTKpM/I=; b=jVZZTGLZX5LOU5mvXTQpI8wDDyzH48by8R4oE5Kk5NeBWx4TZpITtKpQCG5qKAwAFJ qS/rz4dJMz4ko4WpauzHk2apxQJBx0ImaIOIGKvTgDU2C1SyHxuefyNBxhRhVyxaUVjk zp7DROh2MHt+MI4lO3Zxq2n3Zee0elnEkATB1yaTTomJXb/hxTlnowUOlihiPih3Vq1G LAWLpQrSFFbNK4L77R0rdCNTgiSkJiirOGty/bu2mUMj6z1nkn3A99+AcBwaWRckI2IO TeraOX9CZjMc2RzlkLU/MkZwMciKtHLy2Aqi8rE//ItKr9zf+jKXfvJB+xR/5KIrOYkZ 7mOA== X-Forwarded-Encrypted: i=1; AJvYcCXxQcZfFgr8JbYFrj14+VQPzhbOJPBIPElaL9n++Osw/GNxaaFDuh/iHbZbMu4Fr4GqP9yUWw11BbloHSo=@vger.kernel.org X-Gm-Message-State: AOJu0YxZGwjQt/Yjsk0NVhT88SawMC0SfrdkEvqsraM5yDzLXxf2jSGi ciy3OdmA4CJEcugXHlN/ovM6MKKMLV0rt93GPqtaMQfJYAGexpzG2vhMQuYjoA== X-Gm-Gg: ASbGncub7VrU3PFpCwYYi0s1IXRFZ0e8msnmARyfcSJTwCnRaYU88bUfcROI3UkRmQ5 UfsbXD4TCP2RfNhhZa72trahLiTRkFxS5UHSw1v8Ah0e5uaICaRf1GY1Fvc7/8cfc+fD05AD+Ha nnp0KjjshjB21qZfQoFEI/8g/NijJXsj57zrPHHB0JWm0Z6pIiSQy9VCfr99WojS6TXN4hxotsP NUouTeK8kcjifuBFVQ5YqjZnB+7tEQsa/u8xtduTtbL0zfGvDKfFw22GyS9Os/IrE1VVOEHpenP GIFAG9ztSaqV9gYEFw== X-Google-Smtp-Source: AGHT+IGOuYujF/oHMf514Cu6FZkIg94rGPYgxxAA4Ed/XcTGYHy4Z7A3N+oerldYL8V8e8DNCN1OUw== X-Received: by 2002:a17:902:da87:b0:216:3297:92a4 with SMTP id d9443c01a7336-220bbc73ab5mr34950945ad.46.1739342002127; Tue, 11 Feb 2025 22:33:22 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:69f5:6852:451e:8142]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-21f3687bde1sm106393605ad.195.2025.02.11.22.33.20 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 11 Feb 2025 22:33:21 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Kairui Song , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v5 16/18] zram: do not leak page on recompress_store error path Date: Wed, 12 Feb 2025 15:27:14 +0900 Message-ID: <20250212063153.179231-17-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.502.g6dc24dfdaf-goog In-Reply-To: <20250212063153.179231-1-senozhatsky@chromium.org> References: <20250212063153.179231-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Ensure the page used for local object data is freed on error out path. Fixes: 3f909a60cec1 ("zram: rework recompress target selection strategy") Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index c6310077c221..f4644c29f74e 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -2017,7 +2017,7 @@ static ssize_t recompress_store(struct device *dev, struct zram_pp_slot *pps; u32 mode =3D 0, threshold =3D 0; u32 prio, prio_max; - struct page *page; + struct page *page =3D NULL; ssize_t ret; =20 prio =3D ZRAM_SECONDARY_COMP; @@ -2161,9 +2161,9 @@ static ssize_t recompress_store(struct device *dev, cond_resched(); } =20 - __free_page(page); - release_init_lock: + if (page) + __free_page(page); release_pp_ctl(zram, ctl); atomic_set(&zram->pp_in_progress, 0); up_read(&zram->init_lock); --=20 2.48.1.502.g6dc24dfdaf-goog From nobody Thu Dec 18 10:00:39 2025 Received: from mail-pj1-f52.google.com (mail-pj1-f52.google.com [209.85.216.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CB5801FCF4F for ; Wed, 12 Feb 2025 06:33:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.52 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739342009; cv=none; b=l96VZ6KWNATcOMa5h9htTJ7W6scefAPwJPpskK497kP6QlSYlRBCO5AixvdTEwJiH2VVQb6cvt1wFUrJTa2jxhzaa86bbLuR6vSl0S9jwmc6u19Be72Fi7PBx3lPvsDQ2GbBOur6x4intwJTgHVPLB50326u96ZetHL6//hQVck= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739342009; c=relaxed/simple; bh=Axm24glkz+ptvP6aGlVEW9BVP4weLfzYaUzPtrth5jM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=AqNyhMD9DL5c6smJ2NvR6QIa9Wz/yS43cbMMDV3EbyYDDMHYsi9hKhrKGq2rEQmzeROWsR5igDfcVYFEW9y37+r1ifkEYlBj+k+ShD4t9jguTva2lAT+0Qfm+cslEKyylonhCXRi0+1+wWgaQUBWRhGg5tBNSLBAGgsU0tbp8Ms= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=PUevrnX6; arc=none smtp.client-ip=209.85.216.52 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="PUevrnX6" Received: by mail-pj1-f52.google.com with SMTP id 98e67ed59e1d1-2f44353649aso9490939a91.0 for ; Tue, 11 Feb 2025 22:33:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1739342007; x=1739946807; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=a8HQ6KKZTY10gG1FVyTeRkLW7p1KkTSFkDVXwYemgoI=; b=PUevrnX6/eVrPtzTPOcEdWP6ZrbDNroSBhI6Q6IE+Lmpul95SMtpgPkJnL2Mg3tfwh TvY1EKtH64Fwcrddn+IwNLW2LV9CWK6Z4T6PVmgp5qJSzNI7EFrzN3ChZUMRHBqpWsKX t4yCYGvnfGov0PwsxzZy6PvyQJNRljc1+sits= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739342007; x=1739946807; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=a8HQ6KKZTY10gG1FVyTeRkLW7p1KkTSFkDVXwYemgoI=; b=dTFZzmo3ynstd4X0QA70WVhS09SNgb2aDA06siVrECkgDZxkjzptWYF0FSqDbjM8yE /3Mb9YSCOOd05WZVtDnN5wAicZgb/0s/HAUjJmTPrC1DrVWxnt+jLfefuIKUuyThOLSd X2kiq+WP62AC8mcKycVWlRU0C7rsGUAeY+kTLraTAn1q/8izRoHfplWCia6rsVyKFCew lpj4bxMCeVHsf9tzysXBRpuY8zD1jrC/Z86h9UdpzkB8LGFsATnwdzW9apqhGVLcEY/4 RbiDDHLrPCmWxBjxR9K0mfXu9ZloqpCV7aSht4kDMYFbw7hHqpA3ALmmiYFOcOsRgITw rjqA== X-Forwarded-Encrypted: i=1; AJvYcCVks2OeamlZ3TBMbbpfToud4B2kyN+VotbC0pAHTE4GfoQF6wU7+xxMGoqxc81QtArm4WCm8d7HspTEnfk=@vger.kernel.org X-Gm-Message-State: AOJu0Yy6uzTS409+u8O3LciiiVy0E4tIOdMc9w3vHgfVOxgt8DjxfKvc gQ1N82JyO9+Lwh0af91015HCqjzWlKa7EYSOYSEZfrgZE2NVr66CkRbFdEPIoQ== X-Gm-Gg: ASbGnctxaDmQo7lwFzf/dxxM9U9CFrnpCQ/3Td9Lz9FRqL35TvzvQVvvtIsvkdpyiM/ FpXn0t3rMeR0JdnOU0p3blmqnZsFLtFmLn5YN56NMqS98Km0y2srSudhFSIDia2b0eSiyz1xHQx HID/VmBoUERyPoAueGUcirk15hD7sJhyfjqUmxh2Q02g3TI8IXl1SCRCNx+4kpWiKHL8Mehs6me 8x6nBkXu9VbKOXET9SYb/PfNu/mf7QhrtyRQ82Rog0n2xdGbbokKEWbrK9exQDzf8cxc9Fmnj01 XMA/T1adVA3Mwk5TVQ== X-Google-Smtp-Source: AGHT+IEQRw5PJmq7wTBgXFBNQOsAEHuqWdKMzAKbQj8udw0upyNAtfwplObq30KxfZI8V98xY/jwVA== X-Received: by 2002:a05:6a00:22d3:b0:730:8e97:bd76 with SMTP id d2e1a72fcca58-7322c591b87mr2706438b3a.9.1739342007145; Tue, 11 Feb 2025 22:33:27 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:69f5:6852:451e:8142]) by smtp.gmail.com with UTF8SMTPSA id d2e1a72fcca58-730851fa25bsm5556237b3a.180.2025.02.11.22.33.25 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 11 Feb 2025 22:33:26 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Kairui Song , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v5 17/18] zram: do not leak page on writeback_store error path Date: Wed, 12 Feb 2025 15:27:15 +0900 Message-ID: <20250212063153.179231-18-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.502.g6dc24dfdaf-goog In-Reply-To: <20250212063153.179231-1-senozhatsky@chromium.org> References: <20250212063153.179231-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Ensure the page used for local object data is freed on error out path. Fixes: 330edc2bc059 (zram: rework writeback target selection strategy) Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index f4644c29f74e..10239aea5ce0 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -787,7 +787,7 @@ static ssize_t writeback_store(struct device *dev, unsigned long index =3D 0; struct bio bio; struct bio_vec bio_vec; - struct page *page; + struct page *page =3D NULL; ssize_t ret =3D len; int mode, err; unsigned long blk_idx =3D 0; @@ -929,8 +929,10 @@ static ssize_t writeback_store(struct device *dev, =20 if (blk_idx) free_block_bdev(zram, blk_idx); - __free_page(page); + release_init_lock: + if (page) + __free_page(page); release_pp_ctl(zram, ctl); atomic_set(&zram->pp_in_progress, 0); up_read(&zram->init_lock); --=20 2.48.1.502.g6dc24dfdaf-goog From nobody Thu Dec 18 10:00:39 2025 Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 80AE81FBC8C for ; Wed, 12 Feb 2025 06:33:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739342017; cv=none; b=iBE2wmwuq6lgVDHL660i0gNxeugN9Cs7vPWEGw2XzD/3R49EGmvqeXq2eDrFimgsZf11gb6E02565srC9lkCazaFTRLs025v04oz6nJ11AqCW7G7HBHa+ZpgDA3eih3H9ffcc0fR+FwUk7jWYahNWJih+Kpjmnu7yHfv9gOYH10= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739342017; c=relaxed/simple; bh=Ju/AXldQ4LXY+lfPObz58y+sBiIJtoWIokAHacy3FF8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=QT7yXYB2PB3yH6SLhly79yZY0DO6eWmua/OVWYSoX8iR6QsJBpLNbcTJSH+vM1ZXUG0bLuMfBc84kbuimuAjT6MMRrbhXtuKOGLH21acjFQESxkW8bKbpSGzIonOkwGQRHQs2NxCmY+dCbXa9U85gZrCmoWtf33ByxjcX0WS5o8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=bMMR/Bpn; arc=none smtp.client-ip=209.85.214.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="bMMR/Bpn" Received: by mail-pl1-f171.google.com with SMTP id d9443c01a7336-21f50895565so73732025ad.2 for ; Tue, 11 Feb 2025 22:33:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1739342013; x=1739946813; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=tBUgKAKzM6MxWLflFY2zWKudIzejP2AyvCz3ga5cMyM=; b=bMMR/BpnRmSel7rAAQNWTk4hXuR1wX4iTxJmtogsRU3FcPZn0Enc+VPExmK1kaMRNn 7WpkkkcDTch/1bQ6Ta9J7mSJP/SbC9YmXIeBBRL4xTURGxLYQoa1FLiG5/9yUDxPj1y1 VZlhcMfgS1UAX9M5gOBMusJYdSBI+UcBk1glI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739342013; x=1739946813; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=tBUgKAKzM6MxWLflFY2zWKudIzejP2AyvCz3ga5cMyM=; b=hrCXRTUjQyrHh7EHYrHdk+vplgvrFWehkh4iYzyzFLwwocK1T/Xu8iWpoYJPKOTuk2 87Q9q9MfakQbHzn2CmwwwD2poj1ExWr9Pqw5YWqyzN+GlKbmSdRBvke6EjNv9VUVGz64 srTvUb8KI9l8j5G2veUX6rc8X4TnAfWU2jVy2urnPW5P8pznrzmvet0X0Tce0cD0RWvQ eAS00P2Fc/1/aSDY2L/8L05WSdY3JIPjZueFLZl+Fnpc+zAoydwPdbT00sh2CMa0f+jO 4qFeyPvlaVzmZJl/xS9Yb1vboFi1WeLy+43OVwqPgeuyzZpartApfaHy5/7Gmn5rc+nr AEew== X-Forwarded-Encrypted: i=1; AJvYcCX+XktmhKBrWxZqE6u5hcNuOf+qias1aUf0teWTOzhEYQSv94oXvVn3xK/TDKY14wXqJNtdChgls0bkFfs=@vger.kernel.org X-Gm-Message-State: AOJu0YwAhcPJ0fHbYJ2s/8NJY+Ibdt8I4MNvw4IEC7xIS4U4uSId12cT tYf2M515YJUAye8YY7qAcIkIwwv6wQFM77WUU5hYWAi9fTg1t5J5ZmStI064og== X-Gm-Gg: ASbGnctoU92pVpdfYZFNttc2/qokkbTWqJvyXuAlTA5K0M4rjDyr3pfSDu1m3PqH7l3 SgLPyPyLcNdDkDC7/3I54fiJAZWm9Ti3IORoqJwMuBs7uHdgo8GiPaYXU0VzS8QFc4K1Vw4Vi/O c/N14OWOF8Vx3pbBeVhF3MxRsxG4zqNY55YDluMAe/uLvRmdF524SN7Dpkol2so6atVt5ulTrQH 4r9hdFtAb5uI7has+JKHO8BGlxSjBJRs86KL2diXfdk1Jj7O2xzPk0Ecuh/81gdvCDk6M0MRQAB oD0Z+X9Sdkyr2apeqw== X-Google-Smtp-Source: AGHT+IFlFIHAdO/k5TBM9JhT/XTZlmlIvZZXfAUmXRCed2AuW1r8OudMsiWeuA3jTbNKnlIXBGxP7A== X-Received: by 2002:a05:6a21:3996:b0:1e1:afa9:d39b with SMTP id adf61e73a8af0-1ee5e530b58mr3670243637.7.1739342012925; Tue, 11 Feb 2025 22:33:32 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:69f5:6852:451e:8142]) by smtp.gmail.com with UTF8SMTPSA id d2e1a72fcca58-73083999abasm5932358b3a.10.2025.02.11.22.33.30 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 11 Feb 2025 22:33:32 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Kairui Song , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v5 18/18] zram: add might_sleep to zcomp API Date: Wed, 12 Feb 2025 15:27:16 +0900 Message-ID: <20250212063153.179231-19-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.502.g6dc24dfdaf-goog In-Reply-To: <20250212063153.179231-1-senozhatsky@chromium.org> References: <20250212063153.179231-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Explicitly state that zcomp compress/decompress must be called from non-atomic context. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zcomp.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/drivers/block/zram/zcomp.c b/drivers/block/zram/zcomp.c index 61a9c3ed6f7a..217a77e09dc7 100644 --- a/drivers/block/zram/zcomp.c +++ b/drivers/block/zram/zcomp.c @@ -148,6 +148,7 @@ int zcomp_compress(struct zcomp *comp, struct zcomp_str= m *zstrm, }; int ret; =20 + might_sleep(); ret =3D comp->ops->compress(comp->params, &zstrm->ctx, &req); if (!ret) *dst_len =3D req.dst_len; @@ -164,6 +165,7 @@ int zcomp_decompress(struct zcomp *comp, struct zcomp_s= trm *zstrm, .dst_len =3D PAGE_SIZE, }; =20 + might_sleep(); return comp->ops->decompress(comp->params, &zstrm->ctx, &req); } =20 --=20 2.48.1.502.g6dc24dfdaf-goog