From nobody Fri Dec 19 18:31:38 2025 Received: from mail-pj1-f54.google.com (mail-pj1-f54.google.com [209.85.216.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1B299188704 for ; Fri, 14 Feb 2025 04:52:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.54 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739508747; cv=none; b=uJhnaMTaBQ1pNRYYMXoYaPGTDf9jE4tDDKI6wNxBMCT/yRiJ8b8oHLyaYYgpCulUnI0f4P+862sndbghypBrwujo/nuh9kua/zRLWTo9CcQFPCKUMouEqgYdnE7N7OEHj1nfZAUtwlXQ9aHOD5vrJejgfPE8+LbVsWPgRT90Mt0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739508747; c=relaxed/simple; bh=Blyd3YPW6njqDUqhcU13sU79nqRZ3wtUsF7vCotFKpw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Vole9joegakjdZIfTYsRKzq0csD8VWC3UNJ6dkCjxPeDL2xQ/LrJC1YjOM8HsMMq/A1iiTf/4nU2cy8OjfHyA7WyuP4ddMSjyV0dyQaVx3l3A0t5m+jMQJDxk6caCfybCpMLjMNQAg4lilO8lx1PPHul2GFv7EfLvuwqOOxiuZg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=OMYBkowV; arc=none smtp.client-ip=209.85.216.54 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="OMYBkowV" Received: by mail-pj1-f54.google.com with SMTP id 98e67ed59e1d1-2fa1a3c88c5so2431422a91.3 for ; Thu, 13 Feb 2025 20:52:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1739508745; x=1740113545; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=8Ip299Sr+pOGhTseTfOOpc4VtCVbO+2qU2TEqvTRYlA=; b=OMYBkowV2nv1mZuyOh3ftnd8ohDjP7mAAp3Zg8yZw+SYVhqurETqOOAuBj0eMpNW3G 4LkuXOcnLyMCh4sNaDvsvKC0woLUp8hsqHMemUiDM2yjtvrL02bfgfXCvj+1ccFQwyR1 C7LJjQ24dEreiZ6xxsOMchOOZ7B6fSczz3ahY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739508745; x=1740113545; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8Ip299Sr+pOGhTseTfOOpc4VtCVbO+2qU2TEqvTRYlA=; b=dlkzhqZkw/NNaHlqNhbXl2uw6o7o8IPP+zMLrNP0oJNHDQuwVgZCmqSKb7Bsc3hZL0 GfKoKkCsdbOsLA3sM6miiczKD18bGOPIYdE8QLxvxCjQv6YJ7CrqHoi6DSgkT1ZDar02 uyu08mwAtFkEZvL5ViMFtsfVBNdmn7LKRdsZSv2a84yO9rcpHRl29+2gK1cLLfHJ2qZU v1ElmUKFBtIOL3bDjfWqJB8UfhLlBXe9ts39YhDDMVkInddfIJnbdhYeaHEw5L+4YGz7 MjW4aEj4PUBMEH4FXq5utw9Yj4MUq9BoxOVNr3sjdoiwP1z97b9E233NAt1dyOkzOs3B NzDw== X-Forwarded-Encrypted: i=1; AJvYcCVOaCacqKUrEz2zpKfzzjdYCSzlGn42YFkMQ6azyB6uRVYkSppV0i/FW4mqlnjSFiVQe5EGXPDWHyncXLg=@vger.kernel.org X-Gm-Message-State: AOJu0YzwyPKbzBkB6i5l49UOcmkKKzk5H4wzlzIndYbrIlQgMFdVmYob xC5q3pYuKihGzrpz5jVDcBYEaXgl761vCNCV1ehcWutGwDjsxJXHX4pne6jG8g== X-Gm-Gg: ASbGncsqYYrlwtTlLzEbWxP7E3jEtDp/lp/qIMcG6g+dokpEGKKOLy7lLElY4HzKTG4 qfwxQxUBdtNyoTVK9Nm35RweznlOCHCxn6EbvrFQ8A1LoOg8GYiG2v0udKbAwumVjeQrzWlCE7M 8JBNnHL6IWPZcXTqeDWKKbdsgNXyC4kD/EeuLqIaRKbFFMH3cwkCLO6g/1GIO/crN/2XSQMC936 3PGhHNCsOoLoQgD5YIez74WukP9qRr2duaEcX6u+VAml+BjdvU/8pSBTNNmS6A4ewDEw1AhMwmp eY1jmqMkP6uUcUxMZA== X-Google-Smtp-Source: AGHT+IElhSM58kSia8Ygok9EIOPYpgVDSk1Jw5b34RsIuUpVrp592z+ExnE3ByHbPmlZ4fx+OfAh+g== X-Received: by 2002:a05:6a00:3989:b0:729:a31:892d with SMTP id d2e1a72fcca58-7322c591baemr18875311b3a.8.1739508745187; Thu, 13 Feb 2025 20:52:25 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:942d:9291:22aa:8126]) by smtp.gmail.com with UTF8SMTPSA id d2e1a72fcca58-7324f64a39asm588836b3a.69.2025.02.13.20.52.22 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 13 Feb 2025 20:52:24 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Hillf Danton , Kairui Song , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v6 01/17] zram: sleepable entry locking Date: Fri, 14 Feb 2025 13:50:13 +0900 Message-ID: <20250214045208.1388854-2-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog In-Reply-To: <20250214045208.1388854-1-senozhatsky@chromium.org> References: <20250214045208.1388854-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Concurrent modifications of meta table entries is now handled by per-entry spin-lock. This has a number of shortcomings. First, this imposes atomic requirements on compression backends. zram can call both zcomp_compress() and zcomp_decompress() under entry spin-lock, which implies that we can use only compression algorithms that don't schedule/sleep/wait during compression and decompression. This, for instance, makes it impossible to use some of the ASYNC compression algorithms (H/W compression, etc.) implementations. Second, this can potentially trigger watchdogs. For example, entry re-compression with secondary algorithms is performed under entry spin-lock. Given that we chain secondary compression algorithms and that some of them can be configured for best compression ratio (and worst compression speed) zram can stay under spin-lock for quite some time. Having a per-entry mutex (or, for instance, a rw-semaphore) significantly increases sizeof() of each entry and hence the meta table. Therefore entry locking returns back to bit locking, as before, however, this time also preempt-rt friendly, because if waits-on-bit instead of spinning-on-bit. Lock owners are also now permitted to schedule, which is a first step on the path of making zram non-atomic. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 105 ++++++++++++++++++++++++++++++---- drivers/block/zram/zram_drv.h | 20 +++++-- 2 files changed, 108 insertions(+), 17 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 9f5020b077c5..65e16117f2db 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -58,19 +58,99 @@ static void zram_free_page(struct zram *zram, size_t in= dex); static int zram_read_from_zspool(struct zram *zram, struct page *page, u32 index); =20 -static int zram_slot_trylock(struct zram *zram, u32 index) +static void zram_slot_lock_init(struct zram *zram, u32 index) { - return spin_trylock(&zram->table[index].lock); +#ifdef CONFIG_DEBUG_LOCK_ALLOC + lockdep_init_map(&zram->table[index].dep_map, + "zram->table[index].lock", + &zram->lock_class, 0); +#endif +} + +#ifdef CONFIG_DEBUG_LOCK_ALLOC +static inline bool __slot_trylock(struct zram *zram, u32 index) +{ + struct lockdep_map *dep_map =3D &zram->table[index].dep_map; + unsigned long *lock =3D &zram->table[index].flags; + + if (!test_and_set_bit_lock(ZRAM_ENTRY_LOCK, lock)) { + mutex_acquire(dep_map, 0, 1, _RET_IP_); + lock_acquired(dep_map, _RET_IP_); + return true; + } + + lock_contended(dep_map, _RET_IP_); + return false; +} + +static inline void __slot_lock(struct zram *zram, u32 index) +{ + struct lockdep_map *dep_map =3D &zram->table[index].dep_map; + unsigned long *lock =3D &zram->table[index].flags; + + mutex_acquire(dep_map, 0, 0, _RET_IP_); + wait_on_bit_lock(lock, ZRAM_ENTRY_LOCK, TASK_UNINTERRUPTIBLE); + lock_acquired(dep_map, _RET_IP_); +} + +static inline void __slot_unlock(struct zram *zram, u32 index) +{ + struct lockdep_map *dep_map =3D &zram->table[index].dep_map; + unsigned long *lock =3D &zram->table[index].flags; + + mutex_release(dep_map, _RET_IP_); + clear_and_wake_up_bit(ZRAM_ENTRY_LOCK, lock); +} +#else +static inline bool __slot_trylock(struct zram *zram, u32 index) +{ + unsigned long *lock =3D &zram->table[index].flags; + + if (!test_and_set_bit_lock(ZRAM_ENTRY_LOCK, lock)) + return true; + return false; +} + +static inline void __slot_lock(struct zram *zram, u32 index) +{ + unsigned long *lock =3D &zram->table[index].flags; + + wait_on_bit_lock(lock, ZRAM_ENTRY_LOCK, TASK_UNINTERRUPTIBLE); +} + +static inline void __slot_unlock(struct zram *zram, u32 index) +{ + unsigned long *lock =3D &zram->table[index].flags; + + clear_and_wake_up_bit(ZRAM_ENTRY_LOCK, lock); +} +#endif /* CONFIG_DEBUG_LOCK_ALLOC */ + +/* + * entry locking rules: + * + * 1) Lock is exclusive + * + * 2) lock() function can sleep waiting for the lock + * + * 3) Lock owner can sleep + * + * 4) Use TRY lock variant when in atomic context + * - must check return value and handle locking failers + */ +static __must_check bool zram_slot_trylock(struct zram *zram, u32 index) +{ + return __slot_trylock(zram, index); } =20 static void zram_slot_lock(struct zram *zram, u32 index) { - spin_lock(&zram->table[index].lock); + return __slot_lock(zram, index); } =20 static void zram_slot_unlock(struct zram *zram, u32 index) { - spin_unlock(&zram->table[index].lock); + return __slot_unlock(zram, index); } =20 static inline bool init_done(struct zram *zram) @@ -93,7 +173,6 @@ static void zram_set_handle(struct zram *zram, u32 index= , unsigned long handle) zram->table[index].handle =3D handle; } =20 -/* flag operations require table entry bit_spin_lock() being held */ static bool zram_test_flag(struct zram *zram, u32 index, enum zram_pageflags flag) { @@ -1473,15 +1552,11 @@ static bool zram_meta_alloc(struct zram *zram, u64 = disksize) huge_class_size =3D zs_huge_class_size(zram->mem_pool); =20 for (index =3D 0; index < num_pages; index++) - spin_lock_init(&zram->table[index].lock); + zram_slot_lock_init(zram, index); + return true; } =20 -/* - * To protect concurrent access to the same index entry, - * caller should hold this table index entry's bit_spinlock to - * indicate this index entry is accessing. - */ static void zram_free_page(struct zram *zram, size_t index) { unsigned long handle; @@ -2625,6 +2700,10 @@ static int zram_add(void) if (ret) goto out_cleanup_disk; =20 +#ifdef CONFIG_DEBUG_LOCK_ALLOC + lockdep_register_key(&zram->lock_class); +#endif + zram_debugfs_register(zram); pr_info("Added device: %s\n", zram->disk->disk_name); return device_id; @@ -2681,6 +2760,10 @@ static int zram_remove(struct zram *zram) */ zram_reset_device(zram); =20 +#ifdef CONFIG_DEBUG_LOCK_ALLOC + lockdep_unregister_key(&zram->lock_class); +#endif + put_disk(zram->disk); kfree(zram); return 0; diff --git a/drivers/block/zram/zram_drv.h b/drivers/block/zram/zram_drv.h index db78d7c01b9a..794c9234e627 100644 --- a/drivers/block/zram/zram_drv.h +++ b/drivers/block/zram/zram_drv.h @@ -28,7 +28,6 @@ #define ZRAM_SECTOR_PER_LOGICAL_BLOCK \ (1 << (ZRAM_LOGICAL_BLOCK_SHIFT - SECTOR_SHIFT)) =20 - /* * ZRAM is mainly used for memory efficiency so we want to keep memory * footprint small and thus squeeze size and zram pageflags into a flags @@ -46,6 +45,7 @@ /* Flags for zram pages (table[page_no].flags) */ enum zram_pageflags { ZRAM_SAME =3D ZRAM_FLAG_SHIFT, /* Page consists the same element */ + ZRAM_ENTRY_LOCK, /* entry access lock bit */ ZRAM_WB, /* page is stored on backing_device */ ZRAM_PP_SLOT, /* Selected for post-processing */ ZRAM_HUGE, /* Incompressible page */ @@ -58,13 +58,18 @@ enum zram_pageflags { __NR_ZRAM_PAGEFLAGS, }; =20 -/*-- Data structures */ - -/* Allocated for each disk page */ +/* + * Allocated for each disk page. We use bit-lock (ZRAM_ENTRY_LOCK bit + * of flags) to save memory. There can be plenty of entries and standard + * locking primitives (e.g. mutex) will significantly increase sizeof() + * of each entry and hence of the meta table. + */ struct zram_table_entry { unsigned long handle; - unsigned int flags; - spinlock_t lock; + unsigned long flags; +#ifdef CONFIG_DEBUG_LOCK_ALLOC + struct lockdep_map dep_map; +#endif #ifdef CONFIG_ZRAM_TRACK_ENTRY_ACTIME ktime_t ac_time; #endif @@ -137,5 +142,8 @@ struct zram { struct dentry *debugfs_dir; #endif atomic_t pp_in_progress; +#ifdef CONFIG_DEBUG_LOCK_ALLOC + struct lock_class_key lock_class; +#endif }; #endif --=20 2.48.1.601.g30ceb7b040-goog From nobody Fri Dec 19 18:31:38 2025 Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 30883189F5C for ; Fri, 14 Feb 2025 04:52:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.180 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739508752; cv=none; b=EFMbVPnWbnjim/jxnU37a0Oz9+xq7cgpMQkjuGL748TZDID7fZ84kiRbqE1z4Nz8nTz6OIn5NPg2GXyl7qWZYEPzlTjVl6wzNel9/2WJrs2M1ZsQAh/kgyBbWT33iw8z/JW1KKkRQ80vccoX1IY8qy/8T6H/e6Lk4HfO5I8rWYs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739508752; c=relaxed/simple; bh=cTSQlUOAophX88GzGIXzoLy3Kd/dLmUyKJ8dAbZqZ1w=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ECQXy0gIzoL+LbWHDbGOdOKH9wE+v3aBIVC78uqlfrfwv5F/+FpClEYyM5G2ZcfbEB+EfidiZLXYVHrRXrh8J0GTtqxGNpUyx0iEr5PX1tMk/OqaY9lW3Smm6PJB4r0IDtO5c1bSq/ROOSpMx4R0uJm2LhsCUebLIBSxdFt/7HU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=LGhHhtXM; arc=none smtp.client-ip=209.85.214.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="LGhHhtXM" Received: by mail-pl1-f180.google.com with SMTP id d9443c01a7336-21f48ebaadfso33032225ad.2 for ; Thu, 13 Feb 2025 20:52:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1739508750; x=1740113550; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=F9dA9jZ1k0KVleOPZIXSXpGbAAlfZaWcl2QUySckvQY=; b=LGhHhtXMivP4d30hhHPNx0u25y7iJjDr0GP/twjBk49s0h6jqnYh+6IpkIYZSRu8Li ioReIZtM1TDFfWxWFK1XeSBVF71FeavQ2DkeE6oclOO31ny6fE/paIds2q8PNflmoL31 dbLjsJsE6ur/K3tR820c5d/KvkPOnh5jC7Zm4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739508750; x=1740113550; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=F9dA9jZ1k0KVleOPZIXSXpGbAAlfZaWcl2QUySckvQY=; b=wIzo5vgXdsoy52yzDkoEnYhlfMDhqWhfvnb8ffNXPsQOAElSKWGzp7SCqr37nJ+PS1 5nyk83a38hGg+ROcODQyUvGtMAIA315EljALRmjuTezmBBnhkeOCjfqXTySboQ6mYlty SZwh+6XQGaqTwIlbAnocMBbiEQltz8/nOrLF//b2ngGVn3KPHia5CM2WjvkwujpMTekp 1/c97xHuF9bIN3nIkIHUmnPKAmpe2OAqazULyA1d9ID/JkpFnuG09yTZKFfhnryvDSGl uUlTva+qJRi3M9QkxwZYDdwYtzxgvg6n+dGMu3AlyDt4Sd5PMv7fT9DgLa8l69KKuh6O oomw== X-Forwarded-Encrypted: i=1; AJvYcCWt1NmsDQsKl90U1CHCro7ec/Lwy+IaYZYhmcH0Sz2Fo02zGaELeaPVjtjg+IZFs6zU6u+wFA5A6Wmb/EI=@vger.kernel.org X-Gm-Message-State: AOJu0YxzrKDqUPEoRN7+hbAGMCKvqUAs0GGMs3GkGgVCROUmLUv4F9HE icOTM+b/cp8Msp9JAjCMMUz6ClkAcSOn9bxf2j+ZDfl5fHVjxUniBWEf6UJnlQ== X-Gm-Gg: ASbGnctn3QAf0u7K2gB8YWeA1mIr3DQiJZnkHykYWarhySahvjJpsY2FvsvW4MXg6C8 m1upunRBrYCXIglIFFu4lMRNZdjIhUAfy7JJAD/kxGum/e5BWWisFkgyrvuzoMiD6tcfAeK7GUy cC2wZXAp+ngYjz8MvG7gCdlOL4Qv/ObgSBwkhLFVf+JXvEFtlZ6FMWOH4GMyQ4TxYNTx+gfywf7 X6F0Ffe7k/Clqc8mE/QiFkXe55O1s4FPnDDL7E5vU/qgSqgeB8WEwV0yL2w6uRaHDUB/gY31MlV B4xg5lWs+SLu8a2QcA== X-Google-Smtp-Source: AGHT+IHeCccqw7BKz3Nps1nE3WUkuQd67prY/a7knQl57N4aYVrwMvzOwtITAs71quYBmeF0jfumUw== X-Received: by 2002:a05:6a21:898a:b0:1ee:5d05:a197 with SMTP id adf61e73a8af0-1ee5d05a2b1mr13268542637.35.1739508750392; Thu, 13 Feb 2025 20:52:30 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:942d:9291:22aa:8126]) by smtp.gmail.com with UTF8SMTPSA id d2e1a72fcca58-73242761c13sm2252705b3a.136.2025.02.13.20.52.27 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 13 Feb 2025 20:52:30 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Hillf Danton , Kairui Song , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v6 02/17] zram: permit preemption with active compression stream Date: Fri, 14 Feb 2025 13:50:14 +0900 Message-ID: <20250214045208.1388854-3-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog In-Reply-To: <20250214045208.1388854-1-senozhatsky@chromium.org> References: <20250214045208.1388854-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Currently, per-CPU stream access is done from a non-preemptible (atomic) section, which imposes the same atomicity requirements on compression backends as entry spin-lock, and makes it impossible to use algorithms that can schedule/wait/sleep during compression and decompression. Switch to preemptible per-CPU model, similar to the one used in zswap. Instead of a per-CPU local lock, each stream carries a mutex which is locked throughout entire time zram uses it for compression or decompression, so that cpu-dead event waits for zram to stop using a particular per-CPU stream and release it. Suggested-by: Yosry Ahmed Signed-off-by: Sergey Senozhatsky Reviewed-by: Yosry Ahmed --- drivers/block/zram/zcomp.c | 41 +++++++++++++++++++++++++---------- drivers/block/zram/zcomp.h | 6 ++--- drivers/block/zram/zram_drv.c | 20 ++++++++--------- 3 files changed, 42 insertions(+), 25 deletions(-) diff --git a/drivers/block/zram/zcomp.c b/drivers/block/zram/zcomp.c index bb514403e305..53e4c37441be 100644 --- a/drivers/block/zram/zcomp.c +++ b/drivers/block/zram/zcomp.c @@ -6,7 +6,7 @@ #include #include #include -#include +#include #include #include =20 @@ -109,13 +109,29 @@ ssize_t zcomp_available_show(const char *comp, char *= buf) =20 struct zcomp_strm *zcomp_stream_get(struct zcomp *comp) { - local_lock(&comp->stream->lock); - return this_cpu_ptr(comp->stream); + for (;;) { + struct zcomp_strm *zstrm =3D raw_cpu_ptr(comp->stream); + + /* + * Inspired by zswap + * + * stream is returned with ->mutex locked which prevents + * cpu_dead() from releasing this stream under us, however + * there is still a race window between raw_cpu_ptr() and + * mutex_lock(), during which we could have been migrated + * from a CPU that has already destroyed its stream. If + * so then unlock and re-try on the current CPU. + */ + mutex_lock(&zstrm->lock); + if (likely(zstrm->buffer)) + return zstrm; + mutex_unlock(&zstrm->lock); + } } =20 -void zcomp_stream_put(struct zcomp *comp) +void zcomp_stream_put(struct zcomp_strm *zstrm) { - local_unlock(&comp->stream->lock); + mutex_unlock(&zstrm->lock); } =20 int zcomp_compress(struct zcomp *comp, struct zcomp_strm *zstrm, @@ -151,12 +167,9 @@ int zcomp_decompress(struct zcomp *comp, struct zcomp_= strm *zstrm, int zcomp_cpu_up_prepare(unsigned int cpu, struct hlist_node *node) { struct zcomp *comp =3D hlist_entry(node, struct zcomp, node); - struct zcomp_strm *zstrm; + struct zcomp_strm *zstrm =3D per_cpu_ptr(comp->stream, cpu); int ret; =20 - zstrm =3D per_cpu_ptr(comp->stream, cpu); - local_lock_init(&zstrm->lock); - ret =3D zcomp_strm_init(comp, zstrm); if (ret) pr_err("Can't allocate a compression stream\n"); @@ -166,16 +179,17 @@ int zcomp_cpu_up_prepare(unsigned int cpu, struct hli= st_node *node) int zcomp_cpu_dead(unsigned int cpu, struct hlist_node *node) { struct zcomp *comp =3D hlist_entry(node, struct zcomp, node); - struct zcomp_strm *zstrm; + struct zcomp_strm *zstrm =3D per_cpu_ptr(comp->stream, cpu); =20 - zstrm =3D per_cpu_ptr(comp->stream, cpu); + mutex_lock(&zstrm->lock); zcomp_strm_free(comp, zstrm); + mutex_unlock(&zstrm->lock); return 0; } =20 static int zcomp_init(struct zcomp *comp, struct zcomp_params *params) { - int ret; + int ret, cpu; =20 comp->stream =3D alloc_percpu(struct zcomp_strm); if (!comp->stream) @@ -186,6 +200,9 @@ static int zcomp_init(struct zcomp *comp, struct zcomp_= params *params) if (ret) goto cleanup; =20 + for_each_possible_cpu(cpu) + mutex_init(&per_cpu_ptr(comp->stream, cpu)->lock); + ret =3D cpuhp_state_add_instance(CPUHP_ZCOMP_PREPARE, &comp->node); if (ret < 0) goto cleanup; diff --git a/drivers/block/zram/zcomp.h b/drivers/block/zram/zcomp.h index ad5762813842..23b8236b9090 100644 --- a/drivers/block/zram/zcomp.h +++ b/drivers/block/zram/zcomp.h @@ -3,7 +3,7 @@ #ifndef _ZCOMP_H_ #define _ZCOMP_H_ =20 -#include +#include =20 #define ZCOMP_PARAM_NO_LEVEL INT_MIN =20 @@ -31,7 +31,7 @@ struct zcomp_ctx { }; =20 struct zcomp_strm { - local_lock_t lock; + struct mutex lock; /* compression buffer */ void *buffer; struct zcomp_ctx ctx; @@ -77,7 +77,7 @@ struct zcomp *zcomp_create(const char *alg, struct zcomp_= params *params); void zcomp_destroy(struct zcomp *comp); =20 struct zcomp_strm *zcomp_stream_get(struct zcomp *comp); -void zcomp_stream_put(struct zcomp *comp); +void zcomp_stream_put(struct zcomp_strm *zstrm); =20 int zcomp_compress(struct zcomp *comp, struct zcomp_strm *zstrm, const void *src, unsigned int *dst_len); diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 65e16117f2db..ca439f3b1b9a 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1650,7 +1650,7 @@ static int read_compressed_page(struct zram *zram, st= ruct page *page, u32 index) ret =3D zcomp_decompress(zram->comps[prio], zstrm, src, size, dst); kunmap_local(dst); zs_unmap_object(zram->mem_pool, handle); - zcomp_stream_put(zram->comps[prio]); + zcomp_stream_put(zstrm); =20 return ret; } @@ -1811,14 +1811,14 @@ static int zram_write_page(struct zram *zram, struc= t page *page, u32 index) kunmap_local(mem); =20 if (unlikely(ret)) { - zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); + zcomp_stream_put(zstrm); pr_err("Compression failed! err=3D%d\n", ret); zs_free(zram->mem_pool, handle); return ret; } =20 if (comp_len >=3D huge_class_size) { - zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); + zcomp_stream_put(zstrm); return write_incompressible_page(zram, page, index); } =20 @@ -1842,7 +1842,7 @@ static int zram_write_page(struct zram *zram, struct = page *page, u32 index) __GFP_HIGHMEM | __GFP_MOVABLE); if (IS_ERR_VALUE(handle)) { - zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); + zcomp_stream_put(zstrm); atomic64_inc(&zram->stats.writestall); handle =3D zs_malloc(zram->mem_pool, comp_len, GFP_NOIO | __GFP_HIGHMEM | @@ -1854,7 +1854,7 @@ static int zram_write_page(struct zram *zram, struct = page *page, u32 index) } =20 if (!zram_can_store_page(zram)) { - zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); + zcomp_stream_put(zstrm); zs_free(zram->mem_pool, handle); return -ENOMEM; } @@ -1862,7 +1862,7 @@ static int zram_write_page(struct zram *zram, struct = page *page, u32 index) dst =3D zs_map_object(zram->mem_pool, handle, ZS_MM_WO); =20 memcpy(dst, zstrm->buffer, comp_len); - zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); + zcomp_stream_put(zstrm); zs_unmap_object(zram->mem_pool, handle); =20 zram_slot_lock(zram, index); @@ -2021,7 +2021,7 @@ static int recompress_slot(struct zram *zram, u32 ind= ex, struct page *page, kunmap_local(src); =20 if (ret) { - zcomp_stream_put(zram->comps[prio]); + zcomp_stream_put(zstrm); return ret; } =20 @@ -2031,7 +2031,7 @@ static int recompress_slot(struct zram *zram, u32 ind= ex, struct page *page, /* Continue until we make progress */ if (class_index_new >=3D class_index_old || (threshold && comp_len_new >=3D threshold)) { - zcomp_stream_put(zram->comps[prio]); + zcomp_stream_put(zstrm); continue; } =20 @@ -2089,13 +2089,13 @@ static int recompress_slot(struct zram *zram, u32 i= ndex, struct page *page, __GFP_HIGHMEM | __GFP_MOVABLE); if (IS_ERR_VALUE(handle_new)) { - zcomp_stream_put(zram->comps[prio]); + zcomp_stream_put(zstrm); return PTR_ERR((void *)handle_new); } =20 dst =3D zs_map_object(zram->mem_pool, handle_new, ZS_MM_WO); memcpy(dst, zstrm->buffer, comp_len_new); - zcomp_stream_put(zram->comps[prio]); + zcomp_stream_put(zstrm); =20 zs_unmap_object(zram->mem_pool, handle_new); =20 --=20 2.48.1.601.g30ceb7b040-goog From nobody Fri Dec 19 18:31:38 2025 Received: from mail-pj1-f51.google.com (mail-pj1-f51.google.com [209.85.216.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6EC3E18A6D7 for ; Fri, 14 Feb 2025 04:52:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.51 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739508757; cv=none; b=E6oVfrWhKYDhlEZQRqcnYhKgUO5OlmUwRP7zLgAZpYx3vSN8tjvoENl6bHUJ2R74GQdYn0uB5P+RC8cZGP9kEnQOoIPT6kJzzwod8Wk3CjtKQ1KhbumpjvCpeYCGkgqnusiUUvXEpqCVjDXfPnqgeoW/DcEPD+/4b0RMtKHQbvE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739508757; c=relaxed/simple; bh=o4G52ayGrgbwDqZT8PM/VL3I8xDpG9z4TscAzs3WuNY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ScaSLOKxaTDaRbO3gVAxmLz4YEak17r8CxjkNdO7jGGJYf9fRoOG8kGLqSJLMuUFhBQ+9Uq/mD3V/sgEKlXuqYAz31tHVtKNJwbkPzmX1oHwqIZZK4wGC3B3qnDRq3OVrwX5TLz1pXEaihCA+oqSFKxuKrNsFHknFK+8i1NirpA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=CVtMpFOD; arc=none smtp.client-ip=209.85.216.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="CVtMpFOD" Received: by mail-pj1-f51.google.com with SMTP id 98e67ed59e1d1-2fc11834404so2071862a91.0 for ; Thu, 13 Feb 2025 20:52:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1739508756; x=1740113556; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=HFfEwtj8s2kkg4bqWgjDWMl6rQxMO17ZCkJqcn1sbGY=; b=CVtMpFODSATdq7CXM2Kg+BrLUwe/g7G0HH3e+OKl01RsjAd5+0V7TG1GmcPMOGnqkY 82S2t2Dxh8F0ykG9rkCz+3xCM7cykGsNv78xkYosBxnUFSRHuhnLIzE0FNcvenO7pFPg 3QtfTu1AJDyjBbIDW4E29aOUMNwVRQXVwlh7I= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739508756; x=1740113556; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HFfEwtj8s2kkg4bqWgjDWMl6rQxMO17ZCkJqcn1sbGY=; b=LI/zqZ870KFmk/hZX8V8xVoOSVs/qglyTORv4tAoMYghVUyOxEsFZGW0o4emFohRDh +BeTQEd4D0JzXGZems6z/fQPOgq6WoeqEBnAfBAj1inzB1aTDHMRxFvBVGpfqeCYGlJZ jjoX2L2eJ2NO1MBZme39sJAPG6xS9YmyYUe5MlzxVjpBfxG+GODPDN/TSBzGimvRQ4/d 91/u4xxEAi3a27J6V6EYgcrEuyZ0CMAI0GQcDrEKhHp055De1OGRZcDBPWHL/XzV0Mfi NzJhXsoceknbyeqgYurZpd+U+XLuF//seIf1uAP8XkTlTYHKCXCI535QJBzX7nKvM0Me T0ig== X-Forwarded-Encrypted: i=1; AJvYcCVMepOGxOb1CNKe2l5QnPBq5KyK1Ce5ak2oNGDGamZ1X4RJcoUrhm13r2nxSGS0h1lQvX/b1zyMf0uW2Q8=@vger.kernel.org X-Gm-Message-State: AOJu0Ywarmx9ff5uYtS/wRYudd520XNqJEniodqlDPQln8exDWzFWJDu JFLaruirhV7A9JOOjA8ztD+QYpGWdxgT2NAjS+s4EkSJm6R4OhNGsUL7zTuDzw== X-Gm-Gg: ASbGncuFwtwoPqAUyuT9A1Wl7WsYqOR58AzeTE8pHwjJWbGeC5kV6pw8OfWTBkHy8q2 422db7jdW0gU/kwCYdJN052UtBuy9jkblGDOA+maOTfemD4AGzX7IWQhlSlwUbC2ygGo0pqIbE6 /oIBHljqYj/gbhDJ14otYBh84BUYdAILsY22qJitjh+GDbpvJpT5kocHLt0/UW8Tx1bgL3PnzW0 64MP7UU57rvNKgcX+M68IIoCIGj0r+FtzImu4FYqSCuo7E9T4sAH4s/QVJF5zRQ3Sn4J5rlV4Zy bUgUmA3azQz4mvAdrw== X-Google-Smtp-Source: AGHT+IHXO0KDJFtD3EzFKW9dTfi8p4jF/4dEQXWLK1VPzLY1TsXZB+Jthq5QTwRY8WV4GLgW0+AEXQ== X-Received: by 2002:a05:6a00:b95:b0:725:df1a:288 with SMTP id d2e1a72fcca58-7322c41b9damr17804053b3a.24.1739508755619; Thu, 13 Feb 2025 20:52:35 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:942d:9291:22aa:8126]) by smtp.gmail.com with UTF8SMTPSA id d2e1a72fcca58-73242761676sm2272540b3a.142.2025.02.13.20.52.33 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 13 Feb 2025 20:52:35 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Hillf Danton , Kairui Song , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v6 03/17] zram: remove unused crypto include Date: Fri, 14 Feb 2025 13:50:15 +0900 Message-ID: <20250214045208.1388854-4-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog In-Reply-To: <20250214045208.1388854-1-senozhatsky@chromium.org> References: <20250214045208.1388854-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" We stopped using crypto API (for the time being), so remove its include and replace CRYPTO_MAX_ALG_NAME with a local define. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zcomp.c | 1 - drivers/block/zram/zram_drv.c | 4 +++- drivers/block/zram/zram_drv.h | 1 - 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/drivers/block/zram/zcomp.c b/drivers/block/zram/zcomp.c index 53e4c37441be..cfdde2e0748a 100644 --- a/drivers/block/zram/zcomp.c +++ b/drivers/block/zram/zcomp.c @@ -7,7 +7,6 @@ #include #include #include -#include #include =20 #include "zcomp.h" diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index ca439f3b1b9a..0c8821b2b2d9 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -44,6 +44,8 @@ static DEFINE_MUTEX(zram_index_mutex); static int zram_major; static const char *default_compressor =3D CONFIG_ZRAM_DEF_COMP; =20 +#define ZRAM_MAX_ALGO_NAME_SZ 128 + /* Module params (documentation at end) */ static unsigned int num_devices =3D 1; /* @@ -1191,7 +1193,7 @@ static int __comp_algorithm_store(struct zram *zram, = u32 prio, const char *buf) size_t sz; =20 sz =3D strlen(buf); - if (sz >=3D CRYPTO_MAX_ALG_NAME) + if (sz >=3D ZRAM_MAX_ALGO_NAME_SZ) return -E2BIG; =20 compressor =3D kstrdup(buf, GFP_KERNEL); diff --git a/drivers/block/zram/zram_drv.h b/drivers/block/zram/zram_drv.h index 794c9234e627..2c380ea9a816 100644 --- a/drivers/block/zram/zram_drv.h +++ b/drivers/block/zram/zram_drv.h @@ -17,7 +17,6 @@ =20 #include #include -#include =20 #include "zcomp.h" =20 --=20 2.48.1.601.g30ceb7b040-goog From nobody Fri Dec 19 18:31:38 2025 Received: from mail-pl1-f173.google.com (mail-pl1-f173.google.com [209.85.214.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 55280186284 for ; Fri, 14 Feb 2025 04:52:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739508764; cv=none; b=PENSL68pG8nd+gdPKuj2EM8+sv1vGNVDKC8qr+s5XhJTjthGtdUH6SX5MnqZA3pg3cuZrF09J6gSNOakVSzfAJC2xwzxrmBPpNh4hux1dISpRorDWPHo9QC5wOsDyImjnWssZjmb6/OCFn19/MYgmnYECTtL2kdzIKCYeT3Idmk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739508764; c=relaxed/simple; bh=5psVqs0h4asKQIjI/bLFrSEEINcrkLgrQXk/4LDvZDc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=L9iAaFt57oq5UOqiSwlW9674J/g2c7KRpY2Dx30unbBly4niGV4+BslzZUYEbtSM0BFtbSM5GYvW9RcccCxPi32ksEDctC08bn/WHQ0fxvnQ9fFpu8EejhnjuoID3MKtKQtrRT9HMep/j8Bgy9hl32jWbO71OKphl5P8xKv7yWc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=Qpdm3v2m; arc=none smtp.client-ip=209.85.214.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="Qpdm3v2m" Received: by mail-pl1-f173.google.com with SMTP id d9443c01a7336-21f44e7eae4so30059325ad.2 for ; Thu, 13 Feb 2025 20:52:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1739508762; x=1740113562; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=N0yPBZnADO0X5Vq2D3cbKq9jy8e5O0FW3A6DJipxArU=; b=Qpdm3v2mTss7BDmfe4/9YGiWI9ZWySIjPRTLCbpxdB/dewEkrt/JlPEp7ANjIXuhZH 9nXdOB0n+Byj6Bzn1D8AbbAmqSVLr18NCmODJURjcj5aHw3NZyaeVKM6z2wx9aHYkn++ M0pb7QOVX3ATcpVC4yz92EPEGsANC9fM8qdYU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739508762; x=1740113562; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=N0yPBZnADO0X5Vq2D3cbKq9jy8e5O0FW3A6DJipxArU=; b=WLyAmFYZ2EI04LAHX6EWLfcaGg7nNnP7/Zn8DvkBBUy3BfLke5mseRw7rcAvclCnPJ PePMyBTCc4xyXkEBqr1BeSeGZmJuVfGzYWc6au/XiHCYTyeW1s31sO9GC9miTLNUyhyl 1z6ODFzmLM8ZfkNnGgI+pQ4qwNjoPFj+VjzJRD6GX6iYvGcUsgcULZmJ1JxI2kP4LGnz WQSbvnLbKfLAM+pyb1NjKpFdya7HqXlzcXIOjIGNyr74OrnHFNEpS/Bl6UUBvlh4H51q 2GMew43hiIklaxbNnnRCcerofjA53d9Cx5ZAajxhRMK86XNC/ktnIeSd28KdsHGygv/2 kz+w== X-Forwarded-Encrypted: i=1; AJvYcCUZTzMxUIeKSB84fLX91XM8jM/WjXGyj4bVG1rdmLb0Hjt6DRus7ZW5WyyYmo/vzrq5yUgLdO2Jeo24VUQ=@vger.kernel.org X-Gm-Message-State: AOJu0YyeRufCJTEGCb2fUybuR3USkz9XL5VWWOJ/R0msUr7uHnZQAeO+ LP3s5jAw5aPBhty2Mip1UI8bZg2H4kEr5wwxZsk4XqbCvwupESUEixaUUESaIQ== X-Gm-Gg: ASbGncsyovnOjcI2K4+cqj2eXB8RV26RPIEHU0sO/QYgx8IxENwVzStbIFsXR2aTS5T 77Vm2oUgWwPemhOzltRl7fyAW5awfY5G21nv+0cWJiTvSFXKE9gNVpv4dJETFosHYMWgKANL0Fp rni0Sz6UMgfSwaWSt7KLhbejQTSxmnFcFUNNV1FcVSFXQ0jBsH8EWNcChPVJZa7uGz6vcaJHMRA x907u2pP+x9ml/oS8FPZIKum2zY58umT21fN+ncB+aG52oelrLIlHN1P1o0CL7OENzfU+6cupo/ Zp9lPp70vwb1DogrOQ== X-Google-Smtp-Source: AGHT+IE0+HWw32m7Iz+CmsIdF7+EuP5uBVLXDu0uGMgAdMpaEB0vwPlOHOUHVWWjUpawD6FafZNm6g== X-Received: by 2002:a17:903:1ca:b0:220:d257:cdbd with SMTP id d9443c01a7336-220d257ce68mr96793445ad.48.1739508761612; Thu, 13 Feb 2025 20:52:41 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:942d:9291:22aa:8126]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-220d545d474sm20667535ad.116.2025.02.13.20.52.39 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 13 Feb 2025 20:52:41 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Hillf Danton , Kairui Song , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v6 04/17] zram: remove max_comp_streams device attr Date: Fri, 14 Feb 2025 13:50:16 +0900 Message-ID: <20250214045208.1388854-5-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog In-Reply-To: <20250214045208.1388854-1-senozhatsky@chromium.org> References: <20250214045208.1388854-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" max_comp_streams device attribute has been defunct since May 2016 when zram switched to per-CPU compression streams, remove it. Signed-off-by: Sergey Senozhatsky --- Documentation/ABI/testing/sysfs-block-zram | 8 ----- Documentation/admin-guide/blockdev/zram.rst | 36 ++++++--------------- drivers/block/zram/zram_drv.c | 23 ------------- 3 files changed, 10 insertions(+), 57 deletions(-) diff --git a/Documentation/ABI/testing/sysfs-block-zram b/Documentation/ABI= /testing/sysfs-block-zram index 1ef69e0271f9..36c57de0a10a 100644 --- a/Documentation/ABI/testing/sysfs-block-zram +++ b/Documentation/ABI/testing/sysfs-block-zram @@ -22,14 +22,6 @@ Description: device. The reset operation frees all the memory associated with this device. =20 -What: /sys/block/zram/max_comp_streams -Date: February 2014 -Contact: Sergey Senozhatsky -Description: - The max_comp_streams file is read-write and specifies the - number of backend's zcomp_strm compression streams (number of - concurrent compress operations). - What: /sys/block/zram/comp_algorithm Date: February 2014 Contact: Sergey Senozhatsky diff --git a/Documentation/admin-guide/blockdev/zram.rst b/Documentation/ad= min-guide/blockdev/zram.rst index 1576fb93f06c..9bdb30901a93 100644 --- a/Documentation/admin-guide/blockdev/zram.rst +++ b/Documentation/admin-guide/blockdev/zram.rst @@ -54,7 +54,7 @@ The list of possible return codes: If you use 'echo', the returned value is set by the 'echo' utility, and, in general case, something like:: =20 - echo 3 > /sys/block/zram0/max_comp_streams + echo foo > /sys/block/zram0/comp_algorithm if [ $? -ne 0 ]; then handle_error fi @@ -73,21 +73,7 @@ This creates 4 devices: /dev/zram{0,1,2,3} num_devices parameter is optional and tells zram how many devices should be pre-created. Default: 1. =20 -2) Set max number of compression streams -=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D - -Regardless of the value passed to this attribute, ZRAM will always -allocate multiple compression streams - one per online CPU - thus -allowing several concurrent compression operations. The number of -allocated compression streams goes down when some of the CPUs -become offline. There is no single-compression-stream mode anymore, -unless you are running a UP system or have only 1 CPU online. - -To find out how many streams are currently available:: - - cat /sys/block/zram0/max_comp_streams - -3) Select compression algorithm +2) Select compression algorithm =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D =20 Using comp_algorithm device attribute one can see available and @@ -107,7 +93,7 @@ Examples:: For the time being, the `comp_algorithm` content shows only compression algorithms that are supported by zram. =20 -4) Set compression algorithm parameters: Optional +3) Set compression algorithm parameters: Optional =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =20 Compression algorithms may support specific parameters which can be @@ -138,7 +124,7 @@ better the compression ratio, it even can take negative= s values for some algorithms), for other algorithms `level` is acceleration level (the higher the value the lower the compression ratio). =20 -5) Set Disksize +4) Set Disksize =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =20 Set disk size by writing the value to sysfs node 'disksize'. @@ -158,7 +144,7 @@ There is little point creating a zram of greater than t= wice the size of memory since we expect a 2:1 compression ratio. Note that zram uses about 0.1% of= the size of the disk when not in use so a huge zram is wasteful. =20 -6) Set memory limit: Optional +5) Set memory limit: Optional =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D =20 Set memory limit by writing the value to sysfs node 'mem_limit'. @@ -177,7 +163,7 @@ Examples:: # To disable memory limit echo 0 > /sys/block/zram0/mem_limit =20 -7) Activate +6) Activate =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =20 :: @@ -188,7 +174,7 @@ Examples:: mkfs.ext4 /dev/zram1 mount /dev/zram1 /tmp =20 -8) Add/remove zram devices +7) Add/remove zram devices =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D =20 zram provides a control interface, which enables dynamic (on-demand) device @@ -208,7 +194,7 @@ execute:: =20 echo X > /sys/class/zram-control/hot_remove =20 -9) Stats +8) Stats =3D=3D=3D=3D=3D=3D=3D=3D =20 Per-device statistics are exported as various nodes under /sys/block/zram<= id>/ @@ -228,8 +214,6 @@ mem_limit WO specifies the maximum amount of m= emory ZRAM can writeback_limit WO specifies the maximum amount of write IO zram can write out to backing device as 4KB unit writeback_limit_enable RW show and set writeback_limit feature -max_comp_streams RW the number of possible concurrent compress - operations comp_algorithm RW show and change the compression algorithm algorithm_params WO setup compression algorithm parameters compact WO trigger memory compaction @@ -310,7 +294,7 @@ a single line of text and contains the following stats = separated by whitespace: Unit: 4K bytes =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D =20 -10) Deactivate +9) Deactivate =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =20 :: @@ -318,7 +302,7 @@ a single line of text and contains the following stats = separated by whitespace: swapoff /dev/zram0 umount /dev/zram1 =20 -11) Reset +10) Reset =3D=3D=3D=3D=3D=3D=3D=3D=3D =20 Write any positive value to 'reset' sysfs node:: diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 0c8821b2b2d9..cc4afa01b281 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1146,27 +1146,6 @@ static void zram_debugfs_register(struct zram *zram)= {}; static void zram_debugfs_unregister(struct zram *zram) {}; #endif =20 -/* - * We switched to per-cpu streams and this attr is not needed anymore. - * However, we will keep it around for some time, because: - * a) we may revert per-cpu streams in the future - * b) it's visible to user space and we need to follow our 2 years - * retirement rule; but we already have a number of 'soon to be - * altered' attrs, so max_comp_streams need to wait for the next - * layoff cycle. - */ -static ssize_t max_comp_streams_show(struct device *dev, - struct device_attribute *attr, char *buf) -{ - return scnprintf(buf, PAGE_SIZE, "%d\n", num_online_cpus()); -} - -static ssize_t max_comp_streams_store(struct device *dev, - struct device_attribute *attr, const char *buf, size_t len) -{ - return len; -} - static void comp_algorithm_set(struct zram *zram, u32 prio, const char *al= g) { /* Do not free statically defined compression algorithms */ @@ -2583,7 +2562,6 @@ static DEVICE_ATTR_WO(reset); static DEVICE_ATTR_WO(mem_limit); static DEVICE_ATTR_WO(mem_used_max); static DEVICE_ATTR_WO(idle); -static DEVICE_ATTR_RW(max_comp_streams); static DEVICE_ATTR_RW(comp_algorithm); #ifdef CONFIG_ZRAM_WRITEBACK static DEVICE_ATTR_RW(backing_dev); @@ -2605,7 +2583,6 @@ static struct attribute *zram_disk_attrs[] =3D { &dev_attr_mem_limit.attr, &dev_attr_mem_used_max.attr, &dev_attr_idle.attr, - &dev_attr_max_comp_streams.attr, &dev_attr_comp_algorithm.attr, #ifdef CONFIG_ZRAM_WRITEBACK &dev_attr_backing_dev.attr, --=20 2.48.1.601.g30ceb7b040-goog From nobody Fri Dec 19 18:31:38 2025 Received: from mail-pl1-f181.google.com (mail-pl1-f181.google.com [209.85.214.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6164A18FDDA for ; Fri, 14 Feb 2025 04:52:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739508769; cv=none; b=VmbzF3O3r4mNhf88zIDvXuBWKomgoaIaX9JUrFjSnYF+qAiheAL9cWP2N2YaB0R3xaBWu8XHsBAeY1RRUJSATg6zh68GX4kj1OciJZy3xTGVCs2cOpnLJ9dJgivgVAKMeR7rrKPTzA/6Mj13o9Gc+FmrEL1uPEInA04ndn7SJmo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739508769; c=relaxed/simple; bh=JlfGwux0kZAH8W/lbMsM+5fFpDTKJRZIPd5k4wFpEQk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=CIrE2bFGWMLk/Ug89BKb0RnkIX0ohITKThYahnDQI3AiOgBe31jowwWSfI+E2a0bLd+Lj0tva8zEeadPW3jd+vZS05Rv6uJSQ3sdfa2Ujdvqv1BBwZxR5oAFYpIlNgo8WaTVrwSvCyt46GDKsLLvGaNV4gu3NMDKyy0MeynCHrc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=JVjwezGq; arc=none smtp.client-ip=209.85.214.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="JVjwezGq" Received: by mail-pl1-f181.google.com with SMTP id d9443c01a7336-21f62cc4088so29707985ad.3 for ; Thu, 13 Feb 2025 20:52:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1739508767; x=1740113567; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=3bmpTeEVGC+rQQykpv7em5nfwUamOzB3Vnh8lKCRtbQ=; b=JVjwezGq+bzLtlIOIbiH6lZST02bfBjxDipjkk5GgMRBRLuE+klDrkexQw4HCxyq4K 0wdKns5bKB8A0htOi6wWfQvZMSekcnFzkhjm4YdxXGQ9hCCdFrzzQ02wys2repRjVgfU mW5LG8kG50oSevg/LRH0EtPFCwh4RiSTmeG3M= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739508767; x=1740113567; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=3bmpTeEVGC+rQQykpv7em5nfwUamOzB3Vnh8lKCRtbQ=; b=OCGMOZT0n+0PLg1fN0nBUtQF3xXGZYIUY23Wu/mEdoL6FEDkMXjp+r7VLYGDlFui2D JLe0AjLlFoIVob6IubNTckiKMlSnIOSdZFWDslBCK2DuSocbQJ27IL1T6XDGFCLVwERU BklOWQ2XbnFatjlZEdmij9dsfseNpRB48c4Et9vDhHjg+cMkuyuv1Vc9OItxrS/3UvI4 q6WoUjheSy13rADTzg1ELtnHR2lxqC8Mt5C73nrTF8NpNUim2ukucjYRkN9TpxvvjfY0 yOlE16tsktkpXPE4wBCC2bO1ExD4yJh9jEHD0OiGw2OFvFfNlz6CiToT0M6Vb4UWEwcc YTRw== X-Forwarded-Encrypted: i=1; AJvYcCWLTJJZ4kX1T1CtRZPu6LiAFf3SQ3dUjEbjwkKuZemS96JSO+hY90XqYllrxj2EQDYryYUkqnReHNW4u1U=@vger.kernel.org X-Gm-Message-State: AOJu0YysxzB5kcZpfarjH/6wuUavo3Gn9Qvm93Wyj5gLn7Mi14YcNLw5 KMSR+s3LYNy4jpwA/bHNxE02RVdTxMgwFSLIN8IbFq25qmFJN4VeYOKWi5LKOXLdI1oc9zXiY0c = X-Gm-Gg: ASbGncuMI0Py/0sE8vcrWhUN5m8yLAq99oTpF6THYHp6rL4tGwbSlYEcvjiw4oEvJ0B oFsd4g+xpxI5Ud71gqI0u0MChGww4NCpjJy+CiWaXzx6ejop82oRdAj7i/mtKEag+MdLzksvkDH +SyDxUshd8PRLODL1NnVa+fSR1rlVOTfR1bf781retkRmtFEo+joA1wC9uVqDyZhm1Rw0lthupC 8EQo2NLLfv0VA/w1UXdDlSPz4wX4/E5y4DeKOUoHhCw6DYUI4JqulELqci0MqcTPJl7A26fCU/v G0q119BjadNR5GqOiw== X-Google-Smtp-Source: AGHT+IGEF7Dn0tVy7rM1XTaKFMcH+A1xI++T8yz9kJ3lVrfR5w/GZH5hDQABoV3SVr2TB7FEUspOGQ== X-Received: by 2002:a17:903:228b:b0:21f:6546:9af0 with SMTP id d9443c01a7336-220d216b686mr106732985ad.44.1739508766756; Thu, 13 Feb 2025 20:52:46 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:942d:9291:22aa:8126]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-220d558fe3asm20666155ad.234.2025.02.13.20.52.44 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 13 Feb 2025 20:52:46 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Hillf Danton , Kairui Song , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v6 05/17] zram: remove two-staged handle allocation Date: Fri, 14 Feb 2025 13:50:17 +0900 Message-ID: <20250214045208.1388854-6-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog In-Reply-To: <20250214045208.1388854-1-senozhatsky@chromium.org> References: <20250214045208.1388854-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Previously zram write() was atomic which required us to pass __GFP_KSWAPD_RECLAIM to zsmalloc handle allocation on a fast path and attempt a slow path allocation (with recompression) when the fast path failed. Since it's not atomic anymore we can permit direct reclaim during allocation, and remove fast allocation path and, also, drop the recompression path (which should reduce CPU/battery usage). Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 38 ++++++----------------------------- 1 file changed, 6 insertions(+), 32 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index cc4afa01b281..b6bb52c49990 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1766,11 +1766,11 @@ static int write_incompressible_page(struct zram *z= ram, struct page *page, static int zram_write_page(struct zram *zram, struct page *page, u32 index) { int ret =3D 0; - unsigned long handle =3D -ENOMEM; - unsigned int comp_len =3D 0; + unsigned long handle; + unsigned int comp_len; void *dst, *mem; struct zcomp_strm *zstrm; - unsigned long element =3D 0; + unsigned long element; bool same_filled; =20 /* First, free memory allocated to this slot (if any) */ @@ -1784,7 +1784,6 @@ static int zram_write_page(struct zram *zram, struct = page *page, u32 index) if (same_filled) return write_same_filled_page(zram, element, index); =20 -compress_again: zstrm =3D zcomp_stream_get(zram->comps[ZRAM_PRIMARY_COMP]); mem =3D kmap_local_page(page); ret =3D zcomp_compress(zram->comps[ZRAM_PRIMARY_COMP], zstrm, @@ -1794,7 +1793,6 @@ static int zram_write_page(struct zram *zram, struct = page *page, u32 index) if (unlikely(ret)) { zcomp_stream_put(zstrm); pr_err("Compression failed! err=3D%d\n", ret); - zs_free(zram->mem_pool, handle); return ret; } =20 @@ -1803,35 +1801,11 @@ static int zram_write_page(struct zram *zram, struc= t page *page, u32 index) return write_incompressible_page(zram, page, index); } =20 - /* - * handle allocation has 2 paths: - * a) fast path is executed with preemption disabled (for - * per-cpu streams) and has __GFP_DIRECT_RECLAIM bit clear, - * since we can't sleep; - * b) slow path enables preemption and attempts to allocate - * the page with __GFP_DIRECT_RECLAIM bit set. we have to - * put per-cpu compression stream and, thus, to re-do - * the compression once handle is allocated. - * - * if we have a 'non-null' handle here then we are coming - * from the slow path and handle has already been allocated. - */ - if (IS_ERR_VALUE(handle)) - handle =3D zs_malloc(zram->mem_pool, comp_len, - __GFP_KSWAPD_RECLAIM | - __GFP_NOWARN | - __GFP_HIGHMEM | - __GFP_MOVABLE); + handle =3D zs_malloc(zram->mem_pool, comp_len, + GFP_NOIO | __GFP_HIGHMEM | __GFP_MOVABLE); if (IS_ERR_VALUE(handle)) { zcomp_stream_put(zstrm); - atomic64_inc(&zram->stats.writestall); - handle =3D zs_malloc(zram->mem_pool, comp_len, - GFP_NOIO | __GFP_HIGHMEM | - __GFP_MOVABLE); - if (IS_ERR_VALUE(handle)) - return PTR_ERR((void *)handle); - - goto compress_again; + return PTR_ERR((void *)handle); } =20 if (!zram_can_store_page(zram)) { --=20 2.48.1.601.g30ceb7b040-goog From nobody Fri Dec 19 18:31:38 2025 Received: from mail-pl1-f176.google.com (mail-pl1-f176.google.com [209.85.214.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DFF23192598 for ; Fri, 14 Feb 2025 04:52:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739508775; cv=none; b=X8W+ohAYUo6FQqwOYaFewvMRvOn3WZQgFdq/3Jwj5sE6ugXRghN4cvhJcZgk3sotus7jhGn1GO9FImp0B+/Af8/Il698pgJr/uve03ss2Bi8RPVPg6GNHEJ4benj6ge/eG4vMVmQqED5FRrbWaJ2iS18E/d48R08Fkrb+WnAnhE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739508775; c=relaxed/simple; bh=RiX9bjZjw/Rrq5yHUz/wGgkrHcum1pzGpRmA11AODKY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Q8eY+TvhjE7PULLhRAAqCv++XIJP1fpHDs5+bnSxIMeIOnjWpAbyqyrp9/4U16Twaa4l74mK+qSO5gzuTHED9Fi6pjH6GSDZxwV32rMuViNAf8Yy0xhy9jce1aRs4/1vEpEWYiIiJDn7NMMPDhmkw6CIoJZHpIkbukUd6+pRrRI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=Cr2jDV97; arc=none smtp.client-ip=209.85.214.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="Cr2jDV97" Received: by mail-pl1-f176.google.com with SMTP id d9443c01a7336-21f2339dcfdso25104755ad.1 for ; Thu, 13 Feb 2025 20:52:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1739508772; x=1740113572; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=6lbJ5wxYoLG970llEmrArbpgT7YZVPssHOdZszPf+hc=; b=Cr2jDV97E07HATgutJzxeObQ21LhAkEYMYzkUsqfjhT4eUD8H0YBwc1eJp/1DZeH1X Qf+axQjZD+du2/7N0yqgos7yxHXD7zJ8iDcBV6+qJeUbqskWGqDKgAD/XYmNRAc26KVJ OcmG56AdCKfpdQWzbu3O+iIKt9NHvXqQyIL+Q= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739508772; x=1740113572; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=6lbJ5wxYoLG970llEmrArbpgT7YZVPssHOdZszPf+hc=; b=L+tUEFhTdAF3DFx/gezl/9fv+ial7Brz9K+rSdws5EtRliFX3Qvs2UW2HKftpSF5my 8XUCWrAg/DR7KfYoMJh+cXKfiKBCPqNHDcvWFAcCF7eevaxupwfpJhoI0//Rk1CuscDA 9ySGJD9IwYqbHLzniH0MARGMLAh4z6GGuNdQzUd5QNevcIqoY7MLEhBF5RAZAsYbEJIl 8OF1gIiSuiwo/XtYpFO+A7I0UUGcmew/DHEJh5d5K4fjRcr7Xroz8uGpAxidrz233+SZ s5PxB+bFbG175iJVqaeB0dT69LbiIvML5CdNEi9nIqIuPeqP0WDUGC0xSNvr0plXZW1O Xzpg== X-Forwarded-Encrypted: i=1; AJvYcCWhf7DuePmFJdGkKJz4pmcVDA2o/jD3mTGeJK60jzs6Fiv9ZtrUON1P0bnZBs5m2ZjbpjEBUVMkDpvfh68=@vger.kernel.org X-Gm-Message-State: AOJu0YznSef7g5rLm5Age2lm7jtJlBtc6lwmcIKqOw8a75RUXcthgUBH xEJQVM1py/9ysqOsTtpOxoPgruB4pth8INEt1kPNFQZ81kWBg9sXTxGXDLTG0Q== X-Gm-Gg: ASbGncuC3717BLAlqasC47WUSwJcDSK3WHmiT44BAx2+Meho9ksIrGFeqbLfh5D6RZr iU02zqDXaQBCbUXLS5JXa1HI8M9QOEImPCAC1gAqfMPp4cMvN5kZppCSvo4WQ6WXxQ2r3NyYzim IRIcZ92v/Nk4Pler14mxOQZW2AFl6IyDmYaC25bdy3c1xrWKwHf95lmQfu6LRbLtcZAhL/bOtlz HIpupNa8YW4B+7aD3OxdenFZYobcMtNDgLVt8fidyBF//pWK/m/G6ek6QICX6wNS1RZGQ+j4k7g ve38A2HGzSfhG3LDOg== X-Google-Smtp-Source: AGHT+IF8aIJx2X7OJz6+UwgJWzttQh3HN4r8qEKkoAECXCtprMdGPtLnD2KHE/zkfmDa28LIY15LzQ== X-Received: by 2002:a17:902:ef4c:b0:215:8847:4377 with SMTP id d9443c01a7336-220d37151e7mr82187875ad.15.1739508772047; Thu, 13 Feb 2025 20:52:52 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:942d:9291:22aa:8126]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-220d5348f34sm21017955ad.10.2025.02.13.20.52.49 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 13 Feb 2025 20:52:51 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Hillf Danton , Kairui Song , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v6 06/17] zram: remove writestall zram_stats member Date: Fri, 14 Feb 2025 13:50:18 +0900 Message-ID: <20250214045208.1388854-7-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog In-Reply-To: <20250214045208.1388854-1-senozhatsky@chromium.org> References: <20250214045208.1388854-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" There is no zsmalloc handle allocation slow path now and writestall is not possible any longer. Remove it from zram_stats. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 3 +-- drivers/block/zram/zram_drv.h | 1 - 2 files changed, 1 insertion(+), 3 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index b6bb52c49990..c28e1b97bf92 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1480,9 +1480,8 @@ static ssize_t debug_stat_show(struct device *dev, =20 down_read(&zram->init_lock); ret =3D scnprintf(buf, PAGE_SIZE, - "version: %d\n%8llu %8llu\n", + "version: %d\n0 %8llu\n", version, - (u64)atomic64_read(&zram->stats.writestall), (u64)atomic64_read(&zram->stats.miss_free)); up_read(&zram->init_lock); =20 diff --git a/drivers/block/zram/zram_drv.h b/drivers/block/zram/zram_drv.h index 2c380ea9a816..59c75154524f 100644 --- a/drivers/block/zram/zram_drv.h +++ b/drivers/block/zram/zram_drv.h @@ -84,7 +84,6 @@ struct zram_stats { atomic64_t huge_pages_since; /* no. of huge pages since zram set up */ atomic64_t pages_stored; /* no. of pages currently stored */ atomic_long_t max_used_pages; /* no. of maximum pages stored */ - atomic64_t writestall; /* no. of write slow paths */ atomic64_t miss_free; /* no. of missed free */ #ifdef CONFIG_ZRAM_WRITEBACK atomic64_t bd_count; /* no. of pages in backing device */ --=20 2.48.1.601.g30ceb7b040-goog From nobody Fri Dec 19 18:31:38 2025 Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D6B29188704 for ; Fri, 14 Feb 2025 04:52:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739508779; cv=none; b=iBjM1cLG3ssdUL1m2lnflHC5lihGi5XFssRFNZtLrfmLWOvBEYrac6nD/aMRr7T2ZRyZA87AP5larC5vyCOdN+u0hMf/c07071upU/tBYjELBZRhl/mj4xWDZ/ACI2avRqT4n21q8II8mT2KXxlDsQHfIkqD+kkYPlYXvxdNp/g= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739508779; c=relaxed/simple; bh=o6yg0Rm1JkZpxLVpaNCkKzt7lYA5VMorFqotjBdLMOk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=K6coBTOnxPofD7yudzsdzpv3uIhV/xDaW7m9MEgSVESOFHUom96BdcaDCeogsKrnGr6zMl/R3eIAOAbsii/l1hFM+n2aE9fyoxVcShSALNhYU6puKfahl3iiMg89Njz/ixTssrKkCbyIGra6ixDtPEAF7eB/OD6HP5nj5JJDkdI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=balfzeqK; arc=none smtp.client-ip=209.85.214.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="balfzeqK" Received: by mail-pl1-f171.google.com with SMTP id d9443c01a7336-21f7f1e1194so41043555ad.2 for ; Thu, 13 Feb 2025 20:52:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1739508777; x=1740113577; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=gJYj+/a+/IqgLXusoRYSgVepNrwhE22l9CBy17XcMXs=; b=balfzeqK/iy8QgMIeJHdbvjJEfrMMPBrmacnt24lAuIrTNIJ5gc3920EwptazJ3q/M sHkMeSmgof2y62da/odHp/wr2cjXfwdU7eG+oeicE+yS8IVXrMStfTry/V5CDUuVh/WU iHic7JpedPWbw/mU8iFlx4OtPkxSj6dehuiXM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739508777; x=1740113577; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=gJYj+/a+/IqgLXusoRYSgVepNrwhE22l9CBy17XcMXs=; b=Evkk7eFTobXRAZVuWMnp+xIicpwfN7o11By+z9ShOHjhoo69RAxbkYNyPcZem0ZkM/ pC6esMwM16WDE82oItA/Kz6EM5oUM+ycVagY2ZmXyOPtRz24E+93J9zr6ntuQbfuDLTe lR5jQ9aHvuXsg84tTcTVcKgEbkLzE/05gnIutypjaWP3bAF0sMVT+MSu/38veer5JP+V ipEdPXorYpEY6GLL18NC2lMTsVEOPB4tG1aiMsImK5M5I7jQc/QAyZf4e1u3fFygQR0g q91Ng9qZdNt0Pt11MO1V3dhbY8e7NWQdNrohaZyTMi03OOYfbUBviqoI2PbL5yLKdKRE Km2A== X-Forwarded-Encrypted: i=1; AJvYcCULGj9QOhxLDoeobUvMeatYUS2+NQtXzfBXmKR0/+0qBpTWwgs0RTutXWlh2QBgs5OSMF+WxqnrqjqN4ZQ=@vger.kernel.org X-Gm-Message-State: AOJu0Ywlv9ftW1xxsstHg2rfaCyrLMLWPP8G0RJuf1l9ryCKU6nxrODG BudeDRQVL9HFNqyRd8ETdMAGUawa//2A/l8aI7afIUk7wld4zQgekJ51YCH0Sw== X-Gm-Gg: ASbGncsx4u1/+kwMtOsL6MBua9Rgczf+LXBzciZIx/3qzcYGF0AHwmEqXK9i8Ft0qvz vQfRiA7/eEV2HRh/rnJjXOQrLEeBY5JWB841ztE95h/mCnRov8Ma59yOj+7lZT33FGKQBeS5KI7 SDx3mtu+JKY62M8FdKIKDDbw6kF3zEi0TWJAOztSkMEfMcWBFoJx3KzTKiptN1tJVT+XZ0YhZnW Xm7UeZB7Oh9lufx5O2wVel0KTtMt7J3U1eMQdNuozIb9N6U/+1rV9Ca6teof7dgNrhzVGYZf1X+ eBEqvxKgNCpyiFS2jw== X-Google-Smtp-Source: AGHT+IFt4PulwuqoCTMVnGkv5xAvCqNUHqsd70C5Y0GLWGm62Ls+ETbNspIM52jswK714tvw53XBug== X-Received: by 2002:a17:902:f68b:b0:21f:1096:7d5 with SMTP id d9443c01a7336-220bbb209camr155241455ad.17.1739508777155; Thu, 13 Feb 2025 20:52:57 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:942d:9291:22aa:8126]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-220d558591asm20952015ad.224.2025.02.13.20.52.54 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 13 Feb 2025 20:52:56 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Hillf Danton , Kairui Song , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v6 07/17] zram: limit max recompress prio to num_active_comps Date: Fri, 14 Feb 2025 13:50:19 +0900 Message-ID: <20250214045208.1388854-8-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog In-Reply-To: <20250214045208.1388854-1-senozhatsky@chromium.org> References: <20250214045208.1388854-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Use the actual number of algorithms zram was configure with instead of theoretical limit of ZRAM_MAX_COMPS. Also make sure that min prio is not above max prio. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index c28e1b97bf92..c11ed2dc23aa 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -2068,16 +2068,19 @@ static ssize_t recompress_store(struct device *dev, struct device_attribute *attr, const char *buf, size_t len) { - u32 prio =3D ZRAM_SECONDARY_COMP, prio_max =3D ZRAM_MAX_COMPS; struct zram *zram =3D dev_to_zram(dev); char *args, *param, *val, *algo =3D NULL; u64 num_recomp_pages =3D ULLONG_MAX; struct zram_pp_ctl *ctl =3D NULL; struct zram_pp_slot *pps; u32 mode =3D 0, threshold =3D 0; + u32 prio, prio_max; struct page *page; ssize_t ret; =20 + prio =3D ZRAM_SECONDARY_COMP; + prio_max =3D zram->num_active_comps; + args =3D skip_spaces(buf); while (*args) { args =3D next_arg(args, ¶m, &val); @@ -2130,7 +2133,7 @@ static ssize_t recompress_store(struct device *dev, if (prio =3D=3D ZRAM_PRIMARY_COMP) prio =3D ZRAM_SECONDARY_COMP; =20 - prio_max =3D min(prio + 1, ZRAM_MAX_COMPS); + prio_max =3D prio + 1; continue; } } @@ -2158,7 +2161,7 @@ static ssize_t recompress_store(struct device *dev, continue; =20 if (!strcmp(zram->comp_algs[prio], algo)) { - prio_max =3D min(prio + 1, ZRAM_MAX_COMPS); + prio_max =3D prio + 1; found =3D true; break; } @@ -2170,6 +2173,12 @@ static ssize_t recompress_store(struct device *dev, } } =20 + prio_max =3D min(prio_max, (u32)zram->num_active_comps); + if (prio >=3D prio_max) { + ret =3D -EINVAL; + goto release_init_lock; + } + page =3D alloc_page(GFP_KERNEL); if (!page) { ret =3D -ENOMEM; --=20 2.48.1.601.g30ceb7b040-goog From nobody Fri Dec 19 18:31:38 2025 Received: from mail-pl1-f172.google.com (mail-pl1-f172.google.com [209.85.214.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DE144188704 for ; Fri, 14 Feb 2025 04:53:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739508784; cv=none; b=T1nxGRfikoNLKD1CAYLqwLxPS5WHT8RmjwDeE2E2uUVuI9A0J5vFb8IuKsf6+mgXXoeA1Ftp+fFzdf0VSz2uC4XM2Esqe8VQc6JB8DVb9VphdX1VeEHu8tJz67TuLdQsRgFWrC99q03//W78WtJeJeblGFM05ih47OxsvrUYzS4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739508784; c=relaxed/simple; bh=apKzz45NLBZ9ql4LjJ2iuBWBMPcIN0crE8B9KnkiBDY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=aGa2/7ia+UhaqcrclAtnTmWzGDyKL/zY4+7idcd+6ooLdZu+cYmOGvDoguR2z9EjIeo1EKfBgauk/kMtRlntvwTrEbsI7xYbKaNBqDZGw82NmSlfVtnsKo452LDbqHDy/IKoH0nZnqyvC8/LSmdhUWxr8OK6Q2aIaJ1fdFdxciY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=g9au7z8L; arc=none smtp.client-ip=209.85.214.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="g9au7z8L" Received: by mail-pl1-f172.google.com with SMTP id d9443c01a7336-220e6028214so18853545ad.0 for ; Thu, 13 Feb 2025 20:53:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1739508782; x=1740113582; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=u8qIWvBqgXd04+Bt3tanZdpv0pd6FYzP6TsmdRGZtIY=; b=g9au7z8LcDi2GDTzvW/TEAVJ+GkvJFloBERZfmuxf+3n6vLWZ9fMpN6ZM/haehTaE4 Lp8fjkkRIiYe274IVrs66Jg9ujChDU7F/OiVAZAhqJw+w+YLbwWjwyEYF/qjZtaoSoA5 T3tzXqcZedt/g/UuNVkAcS17W2Q9bwXf6/Zjg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739508782; x=1740113582; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=u8qIWvBqgXd04+Bt3tanZdpv0pd6FYzP6TsmdRGZtIY=; b=ehNo3wLNycgwcW5E7rEzcOoX2Lhl1wbBORw/elJlP4R6WLMzZiLSHV4EH1eh9eDYDd y7JQnXYfIMselR5k7E6Ai1Pz/22NQdttxwyrztefm4ShM9Wz3Xsnb7SjKfdDXN7JM6vb W+v1r/zeN/40mU/g3uS9taV5LWvfzQr0XtC6DZrbqDdHlbZYHx8Ov6Kpg9tmW1nqgTR8 6eJ1J4n6Av3/4L65K2tELQfB+r+x1jpVh0vX6/dEY8Yl8wmHLxYnw5pyz7ZkXuxZAKBp lNn8ZbSJqpw/FusvTtbsnSQ1IYNChbZMdpUcqW4lTabaOyqc7hoXUC+ykSYG1wTtIvnB TGWQ== X-Forwarded-Encrypted: i=1; AJvYcCWT+WBHe74RMtuH+l9TAje27lrtn6Opvn82iROaGcUpTXHN04ya6CuAjoEZYv4NU+P7agbaetbzzoJ1qec=@vger.kernel.org X-Gm-Message-State: AOJu0Yx+WZ2PI4MmHMfeegV3iY/ezTmozAwclKNYyioGcGz8rJM7ZPsC PiYeRPBqDoZ6pzsLAa1Z6Hqayt6XMEsD9VUz2LWCWEDi/CV87oxsbg+C1tjtlw== X-Gm-Gg: ASbGncvMOUMJOroYi5XtvePob4FqbA/CIdhNq88qEMleUkeXkrGjho/G2NbnmZG+8GY ueTx8J/KjNp5DYzO8lviHuNdoD24PGejWyTu2pcRNs5yPl1yiyN4/I5UNAwc43YUJIvm7RgH8YS wOgBiZRTTHlB4UhrhlXZcFquxoRFqAwzs+5C4IWJRJqSXUvm4lQUhrDFwhZjZiqNqMa6eE8UcGU zKSZsAMQ9RC3QsshQA1EE168DALOlp8n99FBqdfQy6UyCxZohBFv2cS3v/V0mqUrWCoNKgs3FVk qzNPftlJqadKySr6fg== X-Google-Smtp-Source: AGHT+IHUcQM2DJmewLKGuE+CMu76WyyV+1GS8gtwww+ICNVj3SW61vpHXjXYA/HCtx5vMpfKv1a5mQ== X-Received: by 2002:a17:902:f707:b0:216:725c:a137 with SMTP id d9443c01a7336-220bdf56c72mr172323375ad.28.1739508782199; Thu, 13 Feb 2025 20:53:02 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:942d:9291:22aa:8126]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-220d5349203sm21043715ad.29.2025.02.13.20.52.59 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 13 Feb 2025 20:53:01 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Hillf Danton , Kairui Song , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v6 08/17] zram: filter out recomp targets based on priority Date: Fri, 14 Feb 2025 13:50:20 +0900 Message-ID: <20250214045208.1388854-9-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog In-Reply-To: <20250214045208.1388854-1-senozhatsky@chromium.org> References: <20250214045208.1388854-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Do no select for post processing slots that are already compressed with same or higher priority compression algorithm. This should save some memory, as previously we would still put those entries into corresponding post-processing buckets and filter them out later in recompress_slot(). Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 25 ++++++++++++++++--------- 1 file changed, 16 insertions(+), 9 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index c11ed2dc23aa..5f8e963bd513 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1864,7 +1864,7 @@ static int zram_bvec_write(struct zram *zram, struct = bio_vec *bvec, #define RECOMPRESS_IDLE (1 << 0) #define RECOMPRESS_HUGE (1 << 1) =20 -static int scan_slots_for_recompress(struct zram *zram, u32 mode, +static int scan_slots_for_recompress(struct zram *zram, u32 mode, u32 prio= _max, struct zram_pp_ctl *ctl) { unsigned long nr_pages =3D zram->disksize >> PAGE_SHIFT; @@ -1896,6 +1896,10 @@ static int scan_slots_for_recompress(struct zram *zr= am, u32 mode, zram_test_flag(zram, index, ZRAM_INCOMPRESSIBLE)) goto next; =20 + /* Already compressed with same of higher priority */ + if (zram_get_priority(zram, index) + 1 >=3D prio_max) + goto next; + pps->index =3D index; place_pp_slot(zram, ctl, pps); pps =3D NULL; @@ -1952,6 +1956,16 @@ static int recompress_slot(struct zram *zram, u32 in= dex, struct page *page, zram_clear_flag(zram, index, ZRAM_IDLE); =20 class_index_old =3D zs_lookup_class_index(zram->mem_pool, comp_len_old); + + prio =3D max(prio, zram_get_priority(zram, index) + 1); + /* + * Recompression slots scan should not select slots that are + * already compressed with a higher priority algorithm, but + * just in case + */ + if (prio >=3D prio_max) + return 0; + /* * Iterate the secondary comp algorithms list (in order of priority) * and try to recompress the page. @@ -1960,13 +1974,6 @@ static int recompress_slot(struct zram *zram, u32 in= dex, struct page *page, if (!zram->comps[prio]) continue; =20 - /* - * Skip if the object is already re-compressed with a higher - * priority algorithm (or same algorithm). - */ - if (prio <=3D zram_get_priority(zram, index)) - continue; - num_recomps++; zstrm =3D zcomp_stream_get(zram->comps[prio]); src =3D kmap_local_page(page); @@ -2191,7 +2198,7 @@ static ssize_t recompress_store(struct device *dev, goto release_init_lock; } =20 - scan_slots_for_recompress(zram, mode, ctl); + scan_slots_for_recompress(zram, mode, prio_max, ctl); =20 ret =3D len; while ((pps =3D select_pp_slot(ctl))) { --=20 2.48.1.601.g30ceb7b040-goog From nobody Fri Dec 19 18:31:38 2025 Received: from mail-pl1-f175.google.com (mail-pl1-f175.google.com [209.85.214.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5C0DA17B401 for ; Fri, 14 Feb 2025 04:53:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.175 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739508791; cv=none; b=l6K63yShkIXqroXPtqiTbdHCt4bdTF5agpBL3rMQU4ljQIIH3fZy+1VVUHSfOoQ1r6pO1nwHN0agqEKZbm2TfC2iRBixNA0aMswRVaxqYnLK7H2LqY/IEQAbEols1Xk0WSQ7Pf6FfHJZOIdmfL6FH9u1+w+LBOPZ5PZImdu+eLk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739508791; c=relaxed/simple; bh=tsClRihFgcoC34SqlnjNg7NJPY67B75g233Q8v+7jwU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=cBt2vpbsk+bss3dbOWJUZQ6luLpEfxz4hFUXExoCrVHg+hw6x63BQsgkFfDS5IK6DQnbd35yRLqGXhJb40rIuSTQej9ke61lBBtw+rsJxqyfRa8x788SHxflvYJbYaH6uTjxgb1aj7UHXjLi41SBIFPY2g2t7RIOT6uVsKdFJM4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=Vd3FEwaw; arc=none smtp.client-ip=209.85.214.175 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="Vd3FEwaw" Received: by mail-pl1-f175.google.com with SMTP id d9443c01a7336-220bff984a0so28940265ad.3 for ; Thu, 13 Feb 2025 20:53:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1739508788; x=1740113588; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=l/xbb64LFlIEuFMcJ/r7e+VdeaEJuKPUYJ75LZ96nQ8=; b=Vd3FEwawGLB2Yn2cgGsE2acKNu9IL3VEbu51BUN8uuM9foH/+Nst9n6DrUYE6i+hUa uS7t6XzcKuKHFyfPJ2g2PbXSbg+7iuglvOfSHurYYoPKSsn59spe8chFCTiuZvMYcjBe pqGoTTTGWr4uQkUb7EzH3ulNkFiKNmE0uXCvY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739508788; x=1740113588; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=l/xbb64LFlIEuFMcJ/r7e+VdeaEJuKPUYJ75LZ96nQ8=; b=jyWQ1l4zTEJINhbA5Uxk8iL4ZFsOvovYG3G67+PBPMeCUu1sbRI0vgXo1XUlvy/2X4 bCSdw9LT5H9secwUq+v8VI6/vtfjWRCre7kpeGIcLBiFAybM8BnG0UtlYZyZmpL7wkV1 NSu5PTjKc5BwtWrBwYFQHIENr3HOlAAQz16f3O+ahdW3uqjTPsGY6wD/OhcBEgopRuJ3 nSOTnyili695U22qGlfNBQZwn/hKq01PDTuQ2iLMiFBn7k4UYu+jrgCfhOhfYguxzJie jrB9WKlTQGLoivyWXRCj6Yaky8EpW7byuuNNVY/dUU/YkY2yt6X6fi4MZqjVWteK+I51 y68A== X-Forwarded-Encrypted: i=1; AJvYcCUrkdIROs4LJFnR8ChZ+tQVE1VOHCzcntqzT+w564rAYvFdIZ6rYldEpDNQKNWbLOlm7ezI5SE8rAjzE1c=@vger.kernel.org X-Gm-Message-State: AOJu0YwkF84zCex/k2dFUjp2ll2CDSE3xgLEqSBY86rZTuR9EXdnj7c5 36IhXZXSHvoPRHEnYsojSxiV0mBkX7pN7bCmkd9gnLHFk+6eXOz1Mlm+Ggv7jA== X-Gm-Gg: ASbGnctNqaZ/m5/xU2DopaLkC/mLa3RH+NcXDpL3Oiei7uDUEHRH0VElBfb24G1tKph X0j0XDaticg8qM74eh6uG8SFIiNRNvtCGAErrjo7Hn/7gv15fGxhNcs1iFtZL44HF2n2K1dP4JJ Q0OUHJi2BTge6AD4vkwCcj4IdDrKRA4VxVXlErJ0O1cA8I1y7mxBXs4vaMN9BZFafC9y+IX1r99 uN4FZ3gTk1qLhM6AlEBur1Xc9L62famcVeYYPZkI59nZ4lEjjKoYUqMzXKoVDtj2/l66nEgeGtW iO5oXHUUVMXapepEQw== X-Google-Smtp-Source: AGHT+IH47sqL4VstyP3CQTqYF0OisImdj6UBrruloe47Ytl0rlHucdtsjUQ25uJQt/qj5N2jpQbI3w== X-Received: by 2002:a17:902:dad2:b0:21f:baa:80c1 with SMTP id d9443c01a7336-220d2364ac7mr109012865ad.53.1739508788597; Thu, 13 Feb 2025 20:53:08 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:942d:9291:22aa:8126]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-220d556e15esm21001265ad.190.2025.02.13.20.53.06 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 13 Feb 2025 20:53:08 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Hillf Danton , Kairui Song , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v6 09/17] zram: rework recompression loop Date: Fri, 14 Feb 2025 13:50:21 +0900 Message-ID: <20250214045208.1388854-10-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog In-Reply-To: <20250214045208.1388854-1-senozhatsky@chromium.org> References: <20250214045208.1388854-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This reworks recompression loop handling: - set a rule that stream-put NULLs the stream pointer If the loop returns with a non-NULL stream then it's a successfull recompression, otherwise the stream should always be NULL. - do not count the number of recompressions Mark object as incompressible as soon as the algorithm with the highest priority failed to compress that object. - count compression errors as resource usage Even if compression has failed, we still need to bump num_recomp_pages counter. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 53 +++++++++++++---------------------- 1 file changed, 19 insertions(+), 34 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 5f8e963bd513..2e46ea542ecd 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1929,9 +1929,8 @@ static int recompress_slot(struct zram *zram, u32 ind= ex, struct page *page, unsigned int comp_len_new; unsigned int class_index_old; unsigned int class_index_new; - u32 num_recomps =3D 0; void *src, *dst; - int ret; + int ret =3D 0; =20 handle_old =3D zram_get_handle(zram, index); if (!handle_old) @@ -1974,7 +1973,6 @@ static int recompress_slot(struct zram *zram, u32 ind= ex, struct page *page, if (!zram->comps[prio]) continue; =20 - num_recomps++; zstrm =3D zcomp_stream_get(zram->comps[prio]); src =3D kmap_local_page(page); ret =3D zcomp_compress(zram->comps[prio], zstrm, @@ -1983,7 +1981,8 @@ static int recompress_slot(struct zram *zram, u32 ind= ex, struct page *page, =20 if (ret) { zcomp_stream_put(zstrm); - return ret; + zstrm =3D NULL; + break; } =20 class_index_new =3D zs_lookup_class_index(zram->mem_pool, @@ -1993,6 +1992,7 @@ static int recompress_slot(struct zram *zram, u32 ind= ex, struct page *page, if (class_index_new >=3D class_index_old || (threshold && comp_len_new >=3D threshold)) { zcomp_stream_put(zstrm); + zstrm =3D NULL; continue; } =20 @@ -2000,14 +2000,6 @@ static int recompress_slot(struct zram *zram, u32 in= dex, struct page *page, break; } =20 - /* - * We did not try to recompress, e.g. when we have only one - * secondary algorithm and the page is already recompressed - * using that algorithm - */ - if (!zstrm) - return 0; - /* * Decrement the limit (if set) on pages we can recompress, even * when current recompression was unsuccessful or did not compress @@ -2017,38 +2009,31 @@ static int recompress_slot(struct zram *zram, u32 i= ndex, struct page *page, if (*num_recomp_pages) *num_recomp_pages -=3D 1; =20 - if (class_index_new >=3D class_index_old) { + /* Compression error */ + if (ret) + return ret; + + if (!zstrm) { /* * Secondary algorithms failed to re-compress the page - * in a way that would save memory, mark the object as - * incompressible so that we will not try to compress - * it again. + * in a way that would save memory. * - * We need to make sure that all secondary algorithms have - * failed, so we test if the number of recompressions matches - * the number of active secondary algorithms. + * Mark the object incompressible if the max-priority + * algorithm couldn't re-compress it. */ - if (num_recomps =3D=3D zram->num_active_comps - 1) - zram_set_flag(zram, index, ZRAM_INCOMPRESSIBLE); + if (prio < zram->num_active_comps) + return 0; + zram_set_flag(zram, index, ZRAM_INCOMPRESSIBLE); return 0; } =20 - /* Successful recompression but above threshold */ - if (threshold && comp_len_new >=3D threshold) - return 0; - /* - * No direct reclaim (slow path) for handle allocation and no - * re-compression attempt (unlike in zram_write_bvec()) since - * we already have stored that object in zsmalloc. If we cannot - * alloc memory for recompressed object then we bail out and - * simply keep the old (existing) object in zsmalloc. + * We are holding per-CPU stream mutex and entry lock so better + * avoid direct reclaim. Allocation error is not fatal since + * we still have the old object in the mem_pool. */ handle_new =3D zs_malloc(zram->mem_pool, comp_len_new, - __GFP_KSWAPD_RECLAIM | - __GFP_NOWARN | - __GFP_HIGHMEM | - __GFP_MOVABLE); + GFP_NOWAIT | __GFP_HIGHMEM | __GFP_MOVABLE); if (IS_ERR_VALUE(handle_new)) { zcomp_stream_put(zstrm); return PTR_ERR((void *)handle_new); --=20 2.48.1.601.g30ceb7b040-goog From nobody Fri Dec 19 18:31:38 2025 Received: from mail-pj1-f52.google.com (mail-pj1-f52.google.com [209.85.216.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9D9A118D65C for ; Fri, 14 Feb 2025 04:53:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.52 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739508799; cv=none; b=mxxxs4rkVSekFhAibAgSk8UWgdh5Nc3yjlV4bVxgcUS2yDa+ojuQPqKCZnELIzzP85FcN2c4pMQfbUFLWK3E0LLPHD7Y2PD2e/MDSxDOu2LzwzaqIUyQU1AK/ttI047/ttD+wxYkk4eT+u+210fQ/u4mBk7dT8zF5qw4+PX3xKc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739508799; c=relaxed/simple; bh=DY5fwEJgbhMX0yrNqukgQuyP+HTesDPA2ZycKuhFn/M=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=tMVqEk+izU6PhGU7YeV8hMO3lUsMLvgXTbAAh4S4KADTAUX628W3HvPDecJUXY5GNHWiYJ5BHvzr0FKx4E9Mc3UCWrYkznR5Zxn3nuRxznqClozzDaktAosel6Cj0iv0eIKjPzBWvBMZQQMETY96WCIssUQ5lXATEY1VfaruwfE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=N2MMCudz; arc=none smtp.client-ip=209.85.216.52 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="N2MMCudz" Received: by mail-pj1-f52.google.com with SMTP id 98e67ed59e1d1-2fa1a3c88c5so2432013a91.3 for ; Thu, 13 Feb 2025 20:53:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1739508797; x=1740113597; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=nEsYVf2FanYsVWFazbZFyY/NjoKwvYYj/znay1eygdw=; b=N2MMCudzhQrddy9pC/m+isxXcJvPQ3Glmzzq4pdLE0SaIlL2VUHV7/F+jk/yKRNL/1 I15grkwPTCPEgRBoGpKaGW4MiGoIAimk9KroG2ghuwzjqvbmNJbH83BgKaT85CobgCZx BYcHB66HLT0P8WOysMely6fUbZp5dl5k+b1dI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739508797; x=1740113597; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=nEsYVf2FanYsVWFazbZFyY/NjoKwvYYj/znay1eygdw=; b=l3vNkWBPySCLXYSMS0KIrv5xbryqyAIsDUoLQXxIgXYAnFqQMbg7kPlIMSef77XnC1 8hREF5YGpw4MP+OvQ1b8olc+vtMRkvno5po1fzvQa6FYm+5i8StFkPxmihgRb8TIOqkC IQtigjCWPIeH07ZRTccn69MUKloV1XySQipkGQ3BZmEr1FD/l4S5qjY7S+7qrYoe/laB 1XlcU2nhFTNt9ybf4OtYaZ1Z0VEMfLNR0j17hi5u36huRcPMVKEHqHn3Tz4HOQiZ+fe8 1JWWQhdGqpQ+tL/o5HGPda/BZduMzQysETCR5Tz+KlLAY8OxNb7iN6NOjN+iU8Jau0Oe x19g== X-Forwarded-Encrypted: i=1; AJvYcCXWvOLvxJnfIcrla7G/TeEvfjZ/rlYYC31n/Tr0YwA3Wh/oXcY62mPWx6BKT6w1Qmmmv/gpjB9R/GpFbfI=@vger.kernel.org X-Gm-Message-State: AOJu0YyOCUhadC4wqzGicA282tF1FyTSJ++RsH06giE++fSb2hIQsFXG eQgzqCufAUbf7TlXBMLObSO5mEZ2u5j+1MixZUdvnc/ElAdNci6nbj2B0T7CyQ== X-Gm-Gg: ASbGncvP9SMeKpd2VoHycruu8wW2aVPflY+F4L5PvKvMqeknQzLbBPLtFnTuamixhaf CqT7MmFP3EMUPbWbBu6SKM2ImmWD7+qamZETXxuBMUvFfDHl7GH+ijRBDSL1hNGJkP4PICppN64 CVI0hWEtbCEZD8sT7E8vbjY96pRiQ5ABzNFfWQwzX+AhNMj9MS78o+zeU7TcXZfP3Mo9YMa2Yg+ bK6fimo7zdTHSGQzj92aFDpOtJwjPq7AAERMfhQ/jQyGq9Rx69WyhZjQ6myshmaWW5MkwksYn0O goyGPk5dh17cMnCrzQ== X-Google-Smtp-Source: AGHT+IGwpYmFxf0CUvgBN28Mu9LR6po8v4fVBJ8EuK4NGdlYOspY9jkFKQ6vUxoKOF+lj1g+wrgKpQ== X-Received: by 2002:a05:6a00:8c8:b0:730:7970:1f8f with SMTP id d2e1a72fcca58-7322c5a9058mr14486712b3a.12.1739508796987; Thu, 13 Feb 2025 20:53:16 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:942d:9291:22aa:8126]) by smtp.gmail.com with UTF8SMTPSA id d2e1a72fcca58-73242764cd1sm2272338b3a.149.2025.02.13.20.53.14 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 13 Feb 2025 20:53:16 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Hillf Danton , Kairui Song , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v6 10/17] zsmalloc: rename pool lock Date: Fri, 14 Feb 2025 13:50:22 +0900 Message-ID: <20250214045208.1388854-11-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog In-Reply-To: <20250214045208.1388854-1-senozhatsky@chromium.org> References: <20250214045208.1388854-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The old name comes from the times when the pool did not have compaction (defragmentation). Rename it to ->lock because these days it synchronizes not only migration. Signed-off-by: Sergey Senozhatsky Reviewed-by: Yosry Ahmed --- mm/zsmalloc.c | 38 +++++++++++++++++++------------------- 1 file changed, 19 insertions(+), 19 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 6d0e47f7ae33..2e338cde0d21 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -18,7 +18,7 @@ /* * lock ordering: * page_lock - * pool->migrate_lock + * pool->lock * class->lock * zspage->lock */ @@ -223,8 +223,8 @@ struct zs_pool { #ifdef CONFIG_COMPACTION struct work_struct free_work; #endif - /* protect page/zspage migration */ - rwlock_t migrate_lock; + /* protect zspage migration/compaction */ + rwlock_t lock; atomic_t compaction_in_progress; }; =20 @@ -1206,7 +1206,7 @@ void *zs_map_object(struct zs_pool *pool, unsigned lo= ng handle, BUG_ON(in_interrupt()); =20 /* It guarantees it can get zspage from handle safely */ - read_lock(&pool->migrate_lock); + read_lock(&pool->lock); obj =3D handle_to_obj(handle); obj_to_location(obj, &zpdesc, &obj_idx); zspage =3D get_zspage(zpdesc); @@ -1218,7 +1218,7 @@ void *zs_map_object(struct zs_pool *pool, unsigned lo= ng handle, * which is smaller granularity. */ migrate_read_lock(zspage); - read_unlock(&pool->migrate_lock); + read_unlock(&pool->lock); =20 class =3D zspage_class(pool, zspage); off =3D offset_in_page(class->size * obj_idx); @@ -1450,16 +1450,16 @@ void zs_free(struct zs_pool *pool, unsigned long ha= ndle) return; =20 /* - * The pool->migrate_lock protects the race with zpage's migration + * The pool->lock protects the race with zpage's migration * so it's safe to get the page from handle. */ - read_lock(&pool->migrate_lock); + read_lock(&pool->lock); obj =3D handle_to_obj(handle); obj_to_zpdesc(obj, &f_zpdesc); zspage =3D get_zspage(f_zpdesc); class =3D zspage_class(pool, zspage); spin_lock(&class->lock); - read_unlock(&pool->migrate_lock); + read_unlock(&pool->lock); =20 class_stat_sub(class, ZS_OBJS_INUSE, 1); obj_free(class->size, obj); @@ -1796,7 +1796,7 @@ static int zs_page_migrate(struct page *newpage, stru= ct page *page, * The pool migrate_lock protects the race between zpage migration * and zs_free. */ - write_lock(&pool->migrate_lock); + write_lock(&pool->lock); class =3D zspage_class(pool, zspage); =20 /* @@ -1833,7 +1833,7 @@ static int zs_page_migrate(struct page *newpage, stru= ct page *page, * Since we complete the data copy and set up new zspage structure, * it's okay to release migration_lock. */ - write_unlock(&pool->migrate_lock); + write_unlock(&pool->lock); spin_unlock(&class->lock); migrate_write_unlock(zspage); =20 @@ -1956,7 +1956,7 @@ static unsigned long __zs_compact(struct zs_pool *poo= l, * protect the race between zpage migration and zs_free * as well as zpage allocation/free */ - write_lock(&pool->migrate_lock); + write_lock(&pool->lock); spin_lock(&class->lock); while (zs_can_compact(class)) { int fg; @@ -1983,14 +1983,14 @@ static unsigned long __zs_compact(struct zs_pool *p= ool, src_zspage =3D NULL; =20 if (get_fullness_group(class, dst_zspage) =3D=3D ZS_INUSE_RATIO_100 - || rwlock_is_contended(&pool->migrate_lock)) { + || rwlock_is_contended(&pool->lock)) { putback_zspage(class, dst_zspage); dst_zspage =3D NULL; =20 spin_unlock(&class->lock); - write_unlock(&pool->migrate_lock); + write_unlock(&pool->lock); cond_resched(); - write_lock(&pool->migrate_lock); + write_lock(&pool->lock); spin_lock(&class->lock); } } @@ -2002,7 +2002,7 @@ static unsigned long __zs_compact(struct zs_pool *poo= l, putback_zspage(class, dst_zspage); =20 spin_unlock(&class->lock); - write_unlock(&pool->migrate_lock); + write_unlock(&pool->lock); =20 return pages_freed; } @@ -2014,10 +2014,10 @@ unsigned long zs_compact(struct zs_pool *pool) unsigned long pages_freed =3D 0; =20 /* - * Pool compaction is performed under pool->migrate_lock so it is basical= ly + * Pool compaction is performed under pool->lock so it is basically * single-threaded. Having more than one thread in __zs_compact() - * will increase pool->migrate_lock contention, which will impact other - * zsmalloc operations that need pool->migrate_lock. + * will increase pool->lock contention, which will impact other + * zsmalloc operations that need pool->lock. */ if (atomic_xchg(&pool->compaction_in_progress, 1)) return 0; @@ -2139,7 +2139,7 @@ struct zs_pool *zs_create_pool(const char *name) return NULL; =20 init_deferred_free(pool); - rwlock_init(&pool->migrate_lock); + rwlock_init(&pool->lock); atomic_set(&pool->compaction_in_progress, 0); =20 pool->name =3D kstrdup(name, GFP_KERNEL); --=20 2.48.1.601.g30ceb7b040-goog From nobody Fri Dec 19 18:31:38 2025 Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4CDD6185B48 for ; Fri, 14 Feb 2025 04:53:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.180 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739508810; cv=none; b=oMoq85HFWUUNtn03gqS2ag+ZDD83Zuq+fwg09nfyMID4ZsAz5AYpNqfDijoGW3uBCF35uDHYqgDsuUHyDjdIU6M7FqxQRDoM303vsE+xysA5DnK1fnwySAvyonyCbgo4NOeGWRKPgBLJYurvevuyovEsD25OET4EoT7ewz1eBqs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739508810; c=relaxed/simple; bh=jmAx9VzBvJEcK7LEsaUKoFFJmtepfH49Z010DExZrxo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=YfcI7eDMaKKpI0Cob4gizi72RYYKMBZZFbfsRyjA0IbCrnD7OCnLDPY3aep1qZKdeJ/uDTY3+gjOMXYlBMhOn5RTVsC6NhaosTf9PuejZWnN2PcodfDwwGh0WOrbAsMfszZV+j8a97f2JNKGzsSjf6uV5IYque+D+DBXfr5iILU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=ZYpWOKf2; arc=none smtp.client-ip=209.85.214.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="ZYpWOKf2" Received: by mail-pl1-f180.google.com with SMTP id d9443c01a7336-220e83d65e5so15993545ad.1 for ; Thu, 13 Feb 2025 20:53:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1739508807; x=1740113607; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=qnJprTwBbnPgr4JVTYJPq6LS7kls9Uwm5mA/DEYKF7Y=; b=ZYpWOKf2wIM9KTHARFXMzP3NPE0KiS4fmHm7jh//ot/FZDWKTERs+53YCKXutFiXQL aOSD5anUPRpMmmhoWnRkJhzipUZD5q+5pmryJASnkjLwBH0RJR5C4ytQFn3gGA2TVGT1 8flELVpKecCHkWjuWpuZ1A0t8sR9oJvbCmtRg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739508807; x=1740113607; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=qnJprTwBbnPgr4JVTYJPq6LS7kls9Uwm5mA/DEYKF7Y=; b=PCin6uzyEYVA12UcCqHSjnRl1jQE5y9DF0ZlKWUPo8o0YrKn/Frx5tZ3ihgw8L26Yl qFmVDfWAQxm8ui6kENAQ/iNXwH+unIOZF2vYx8ylarW0nTftmxWPRU8lvLeWRrLzMEpM CwsY8GwzFO9PBeY412oEbqtWZpl7He5ZmT4QqAbCrtmSD1XkmJMuDYpZxeQZKdGo0PWj GTwbiVhG42dGyJOcdUiis4FippX9DyHvPfJDe7+ksLiSx3i5/L9ZqE/k+AT7itQNXdAl +6qmlIxLVMVu026OQdmXX/eJ1/qMzXOGEOYq6BzuK6k3Lw/AF1A2X/ZGiZTDQEzUeMAQ wW1w== X-Forwarded-Encrypted: i=1; AJvYcCWEUkZx/37FZn+DeznSojNYeJrn3j/0kgqww3Td0pye1n9RbzIR08dlk85hJZa7T0XAVzbLHl2DMPTRCAQ=@vger.kernel.org X-Gm-Message-State: AOJu0YxNIXSzMir+1RwOjY53uhYLcn3z3orzxRmHTOrHAdGbp/BJLJZl NR25OCmLKU6A2oYvzuyuJqaDnGAPiQa5NPjiViMqFlNhxwFUeehvR5kCMXGhsQ== X-Gm-Gg: ASbGncvdPGi9gpkz3zpI3OCL+D/JHKJ4SVhOSxRnBIkp+uzpbWUKmqh5ZhCUqm8eeZJ nVpd+J7bYGGX7ljVYOvViSoSqwaSMUp0YMFlyNSnW1hdv+/I/RSPldY7yej2TXlmjRbUH5wF1yF 1VRAhAmzB10NnzuVV/3nip4z35dOehIhKa5b1DYBXj114VEfr97ZehW3UN/XptXshOIjG6Y+A32 FNrOnrmgx53y5lE5Skzxk7ekMsXAZ2iMAx9v1rlCd3cHDcJ0LSxqiw92iCQYuxU9OBOiUC1Hyy5 QZsk9TDZoVPXhlyssA== X-Google-Smtp-Source: AGHT+IFFD5AmsVRlFVnjhldrCVgoR4+3BSYYM094ysV/C/O/38mcNtXqChtRYmpdp4CXK407opgp9w== X-Received: by 2002:a17:903:32c8:b0:220:ea90:192a with SMTP id d9443c01a7336-220ea901a5fmr37761435ad.5.1739508807544; Thu, 13 Feb 2025 20:53:27 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:942d:9291:22aa:8126]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-220d53492d0sm20892945ad.35.2025.02.13.20.53.25 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 13 Feb 2025 20:53:27 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Hillf Danton , Kairui Song , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v6 11/17] zsmalloc: make zspage lock preemptible Date: Fri, 14 Feb 2025 13:50:23 +0900 Message-ID: <20250214045208.1388854-12-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog In-Reply-To: <20250214045208.1388854-1-senozhatsky@chromium.org> References: <20250214045208.1388854-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In order to implement preemptible object mapping we need a zspage lock that satisfies several preconditions: - it should be reader-write type of a lock - it should be possible to hold it from any context, but also being preemptible if the context allows it - we never sleep while acquiring but can sleep while holding in read mode An rwsemaphore doesn't suffice, due to atomicity requirements, rwlock doesn't satisfy due to reader-preemptability requirement. It's also worth to mention, that per-zspage rwsem is a little too memory heavy (we can easily have double digits megabytes used only on rwsemaphores). Switch over from rwlock_t to a atomic_t-based implementation of a reader-writer semaphore that satisfies all of the preconditions. The spin-lock based zspage_lock is suggested by Hillf Danton. Suggested-by: Hillf Danton Signed-off-by: Sergey Senozhatsky --- mm/zsmalloc.c | 246 +++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 192 insertions(+), 54 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 2e338cde0d21..bc679a3e1718 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -226,6 +226,9 @@ struct zs_pool { /* protect zspage migration/compaction */ rwlock_t lock; atomic_t compaction_in_progress; +#ifdef CONFIG_DEBUG_LOCK_ALLOC + struct lock_class_key lock_class; +#endif }; =20 static inline void zpdesc_set_first(struct zpdesc *zpdesc) @@ -257,6 +260,18 @@ static inline void free_zpdesc(struct zpdesc *zpdesc) __free_page(page); } =20 +#define ZS_PAGE_UNLOCKED 0 +#define ZS_PAGE_WRLOCKED -1 + +struct zspage_lock { + spinlock_t lock; + int cnt; + +#ifdef CONFIG_DEBUG_LOCK_ALLOC + struct lockdep_map dep_map; +#endif +}; + struct zspage { struct { unsigned int huge:HUGE_BITS; @@ -269,7 +284,7 @@ struct zspage { struct zpdesc *first_zpdesc; struct list_head list; /* fullness list */ struct zs_pool *pool; - rwlock_t lock; + struct zspage_lock zsl; }; =20 struct mapping_area { @@ -279,6 +294,148 @@ struct mapping_area { enum zs_mapmode vm_mm; /* mapping mode */ }; =20 +static void zspage_lock_init(struct zspage *zspage) +{ +#ifdef CONFIG_DEBUG_LOCK_ALLOC + lockdep_init_map(&zspage->zsl.dep_map, "zspage->lock", + &zspage->pool->lock_class, 0); +#endif + + spin_lock_init(&zspage->zsl.lock); + zspage->zsl.cnt =3D ZS_PAGE_UNLOCKED; +} + +#ifdef CONFIG_DEBUG_LOCK_ALLOC +static inline void __read_lock(struct zspage *zspage) +{ + struct zspage_lock *zsl =3D &zspage->zsl; + + rwsem_acquire_read(&zsl->dep_map, 0, 0, _RET_IP_); + + spin_lock(&zsl->lock); + zsl->cnt++; + spin_unlock(&zsl->lock); + + lock_acquired(&zsl->dep_map, _RET_IP_); +} + +static inline void __read_unlock(struct zspage *zspage) +{ + struct zspage_lock *zsl =3D &zspage->zsl; + + rwsem_release(&zsl->dep_map, _RET_IP_); + + spin_lock(&zsl->lock); + zsl->cnt--; + spin_unlock(&zsl->lock); +} + +static inline bool __write_trylock(struct zspage *zspage) +{ + struct zspage_lock *zsl =3D &zspage->zsl; + + spin_lock(&zsl->lock); + if (zsl->cnt =3D=3D ZS_PAGE_UNLOCKED) { + zsl->cnt =3D ZS_PAGE_WRLOCKED; + rwsem_acquire(&zsl->dep_map, 0, 1, _RET_IP_); + lock_acquired(&zsl->dep_map, _RET_IP_); + return true; + } + + lock_contended(&zsl->dep_map, _RET_IP_); + spin_unlock(&zsl->lock); + return false; +} + +static inline void __write_unlock(struct zspage *zspage) +{ + struct zspage_lock *zsl =3D &zspage->zsl; + + rwsem_release(&zsl->dep_map, _RET_IP_); + + zsl->cnt =3D ZS_PAGE_UNLOCKED; + spin_unlock(&zsl->lock); +} +#else +static inline void __read_lock(struct zspage *zspage) +{ + struct zspage_lock *zsl =3D &zspage->zsl; + + spin_lock(&zsl->lock); + zsl->cnt++; + spin_unlock(&zsl->lock); +} + +static inline void __read_unlock(struct zspage *zspage) +{ + struct zspage_lock *zsl =3D &zspage->zsl; + + spin_lock(&zsl->lock); + zsl->cnt--; + spin_unlock(&zsl->lock); +} + +static inline bool __write_trylock(struct zspage *zspage) +{ + struct zspage_lock *zsl =3D &zspage->zsl; + + spin_lock(&zsl->lock); + if (zsl->cnt =3D=3D ZS_PAGE_UNLOCKED) { + zsl->cnt =3D ZS_PAGE_WRLOCKED; + return true; + } + + spin_unlock(&zsl->lock); + return false; +} + +static inline void __write_unlock(struct zspage *zspage) +{ + struct zspage_lock *zsl =3D &zspage->zsl; + + zsl->cnt =3D ZS_PAGE_UNLOCKED; + spin_unlock(&zsl->lock); +} +#endif /* CONFIG_DEBUG_LOCK_ALLOC */ + +/* + * The zspage lock can be held from atomic contexts, but it needs to remain + * preemptible when held for reading because it remains held outside of th= ose + * atomic contexts, otherwise we unnecessarily lose preemptibility. + * + * To achieve this, the following rules are enforced on readers and writer= s: + * + * - Writers are blocked by both writers and readers, while readers are on= ly + * blocked by writers (i.e. normal rwlock semantics). + * + * - Writers are always atomic (to allow readers to spin waiting for them). + * + * - Writers always use trylock (as the lock may be held be sleeping reade= rs). + * + * - Readers may spin on the lock (as they can only wait for atomic writer= s). + * + * - Readers may sleep while holding the lock (as writes only use trylock). + */ +static void zspage_read_lock(struct zspage *zspage) +{ + return __read_lock(zspage); +} + +static void zspage_read_unlock(struct zspage *zspage) +{ + return __read_unlock(zspage); +} + +static __must_check bool zspage_write_trylock(struct zspage *zspage) +{ + return __write_trylock(zspage); +} + +static void zspage_write_unlock(struct zspage *zspage) +{ + return __write_unlock(zspage); +} + /* huge object: pages_per_zspage =3D=3D 1 && maxobj_per_zspage =3D=3D 1 */ static void SetZsHugePage(struct zspage *zspage) { @@ -290,12 +447,6 @@ static bool ZsHugePage(struct zspage *zspage) return zspage->huge; } =20 -static void migrate_lock_init(struct zspage *zspage); -static void migrate_read_lock(struct zspage *zspage); -static void migrate_read_unlock(struct zspage *zspage); -static void migrate_write_lock(struct zspage *zspage); -static void migrate_write_unlock(struct zspage *zspage); - #ifdef CONFIG_COMPACTION static void kick_deferred_free(struct zs_pool *pool); static void init_deferred_free(struct zs_pool *pool); @@ -992,7 +1143,9 @@ static struct zspage *alloc_zspage(struct zs_pool *poo= l, return NULL; =20 zspage->magic =3D ZSPAGE_MAGIC; - migrate_lock_init(zspage); + zspage->pool =3D pool; + zspage->class =3D class->index; + zspage_lock_init(zspage); =20 for (i =3D 0; i < class->pages_per_zspage; i++) { struct zpdesc *zpdesc; @@ -1015,8 +1168,6 @@ static struct zspage *alloc_zspage(struct zs_pool *po= ol, =20 create_page_chain(class, zspage, zpdescs); init_zspage(class, zspage); - zspage->pool =3D pool; - zspage->class =3D class->index; =20 return zspage; } @@ -1217,7 +1368,7 @@ void *zs_map_object(struct zs_pool *pool, unsigned lo= ng handle, * zs_unmap_object API so delegate the locking from class to zspage * which is smaller granularity. */ - migrate_read_lock(zspage); + zspage_read_lock(zspage); read_unlock(&pool->lock); =20 class =3D zspage_class(pool, zspage); @@ -1277,7 +1428,7 @@ void zs_unmap_object(struct zs_pool *pool, unsigned l= ong handle) } local_unlock(&zs_map_area.lock); =20 - migrate_read_unlock(zspage); + zspage_read_unlock(zspage); } EXPORT_SYMBOL_GPL(zs_unmap_object); =20 @@ -1671,18 +1822,18 @@ static void lock_zspage(struct zspage *zspage) /* * Pages we haven't locked yet can be migrated off the list while we're * trying to lock them, so we need to be careful and only attempt to - * lock each page under migrate_read_lock(). Otherwise, the page we lock + * lock each page under zspage_read_lock(). Otherwise, the page we lock * may no longer belong to the zspage. This means that we may wait for * the wrong page to unlock, so we must take a reference to the page - * prior to waiting for it to unlock outside migrate_read_lock(). + * prior to waiting for it to unlock outside zspage_read_lock(). */ while (1) { - migrate_read_lock(zspage); + zspage_read_lock(zspage); zpdesc =3D get_first_zpdesc(zspage); if (zpdesc_trylock(zpdesc)) break; zpdesc_get(zpdesc); - migrate_read_unlock(zspage); + zspage_read_unlock(zspage); zpdesc_wait_locked(zpdesc); zpdesc_put(zpdesc); } @@ -1693,41 +1844,16 @@ static void lock_zspage(struct zspage *zspage) curr_zpdesc =3D zpdesc; } else { zpdesc_get(zpdesc); - migrate_read_unlock(zspage); + zspage_read_unlock(zspage); zpdesc_wait_locked(zpdesc); zpdesc_put(zpdesc); - migrate_read_lock(zspage); + zspage_read_lock(zspage); } } - migrate_read_unlock(zspage); + zspage_read_unlock(zspage); } #endif /* CONFIG_COMPACTION */ =20 -static void migrate_lock_init(struct zspage *zspage) -{ - rwlock_init(&zspage->lock); -} - -static void migrate_read_lock(struct zspage *zspage) __acquires(&zspage->l= ock) -{ - read_lock(&zspage->lock); -} - -static void migrate_read_unlock(struct zspage *zspage) __releases(&zspage-= >lock) -{ - read_unlock(&zspage->lock); -} - -static void migrate_write_lock(struct zspage *zspage) -{ - write_lock(&zspage->lock); -} - -static void migrate_write_unlock(struct zspage *zspage) -{ - write_unlock(&zspage->lock); -} - #ifdef CONFIG_COMPACTION =20 static const struct movable_operations zsmalloc_mops; @@ -1769,7 +1895,7 @@ static bool zs_page_isolate(struct page *page, isolat= e_mode_t mode) } =20 static int zs_page_migrate(struct page *newpage, struct page *page, - enum migrate_mode mode) + enum migrate_mode mode) { struct zs_pool *pool; struct size_class *class; @@ -1785,9 +1911,6 @@ static int zs_page_migrate(struct page *newpage, stru= ct page *page, =20 VM_BUG_ON_PAGE(!zpdesc_is_isolated(zpdesc), zpdesc_page(zpdesc)); =20 - /* We're committed, tell the world that this is a Zsmalloc page. */ - __zpdesc_set_zsmalloc(newzpdesc); - /* The page is locked, so this pointer must remain valid */ zspage =3D get_zspage(zpdesc); pool =3D zspage->pool; @@ -1803,8 +1926,15 @@ static int zs_page_migrate(struct page *newpage, str= uct page *page, * the class lock protects zpage alloc/free in the zspage. */ spin_lock(&class->lock); - /* the migrate_write_lock protects zpage access via zs_map_object */ - migrate_write_lock(zspage); + /* the zspage write_lock protects zpage access via zs_map_object */ + if (!zspage_write_trylock(zspage)) { + spin_unlock(&class->lock); + write_unlock(&pool->lock); + return -EINVAL; + } + + /* We're committed, tell the world that this is a Zsmalloc page. */ + __zpdesc_set_zsmalloc(newzpdesc); =20 offset =3D get_first_obj_offset(zpdesc); s_addr =3D kmap_local_zpdesc(zpdesc); @@ -1835,7 +1965,7 @@ static int zs_page_migrate(struct page *newpage, stru= ct page *page, */ write_unlock(&pool->lock); spin_unlock(&class->lock); - migrate_write_unlock(zspage); + zspage_write_unlock(zspage); =20 zpdesc_get(newzpdesc); if (zpdesc_zone(newzpdesc) !=3D zpdesc_zone(zpdesc)) { @@ -1971,9 +2101,11 @@ static unsigned long __zs_compact(struct zs_pool *po= ol, if (!src_zspage) break; =20 - migrate_write_lock(src_zspage); + if (!zspage_write_trylock(src_zspage)) + break; + migrate_zspage(pool, src_zspage, dst_zspage); - migrate_write_unlock(src_zspage); + zspage_write_unlock(src_zspage); =20 fg =3D putback_zspage(class, src_zspage); if (fg =3D=3D ZS_INUSE_RATIO_0) { @@ -2233,7 +2365,9 @@ struct zs_pool *zs_create_pool(const char *name) * trigger compaction manually. Thus, ignore return code. */ zs_register_shrinker(pool); - +#ifdef CONFIG_DEBUG_LOCK_ALLOC + lockdep_register_key(&pool->lock_class); +#endif return pool; =20 err: @@ -2270,6 +2404,10 @@ void zs_destroy_pool(struct zs_pool *pool) kfree(class); } =20 +#ifdef CONFIG_DEBUG_LOCK_ALLOC + lockdep_unregister_key(&pool->lock_class); +#endif + destroy_cache(pool); kfree(pool->name); kfree(pool); --=20 2.48.1.601.g30ceb7b040-goog From nobody Fri Dec 19 18:31:38 2025 Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4034C1891AA for ; Fri, 14 Feb 2025 04:53:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.180 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739508815; cv=none; b=YqlojN1BZgXmBbc1otxfx26scaGyAZ7zVr9EIBedrzeCnf9H3o3o+d5fXhUhtRCI2WksRTu6C4nYwo2ae1+SSExha1YsjN1X2TymLQeJEJJngYkMSwzqUpfRegZxmxG2PRuYDvyCSHasZ12QcThSAbHBhhscjhBDplYB0vWll9A= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739508815; c=relaxed/simple; bh=JVdpqsVkVNBsSrcDuQ+DKnxw+zDr/4gILM29OeA2KLU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=mhQ6LepMv4TIYD0CaKbMtktkMbq3D/cU9xMLlBpcAFbfZ8JRWlmBzFrtn1i6d+VOcURTXrVLTCa23Sc8Ea4Hf1kLqZEA1YimKkW/B04bEsxjXeXlFmONozu6c9zAmh4XCr6rEwO56eWLLd69u74Vi3EDbjSH7PuIU8YzJdfyFHA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=BRlo6Lsx; arc=none smtp.client-ip=209.85.214.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="BRlo6Lsx" Received: by mail-pl1-f180.google.com with SMTP id d9443c01a7336-21f2339dcfdso25109815ad.1 for ; Thu, 13 Feb 2025 20:53:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1739508813; x=1740113613; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=QjapF9ytftJrXMbeHEPYkCcc21hA6s0G0LeFdpvgJ8k=; b=BRlo6Lsx+CjifXz9kUcaF2clII+wXp+NoileBa9pijXCqllf0x0tfwlHu5e3zTxhj7 U4Q2U5GbsBzS/WwuRfpyuikwEDoV1mKVHKtwEnwVxbFzZnIBF3MIrzPhTjBYbhv7Ectm 8EMErG+331hNH47gqhRNYFWlAKZ/eE/zDWhF0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739508813; x=1740113613; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=QjapF9ytftJrXMbeHEPYkCcc21hA6s0G0LeFdpvgJ8k=; b=CmxWz4nQrL+fdFDlzmFbRI/J2GyAdgVp9UAckkJ6jvjKkY8TAWXieJqIIeXH9bzKV/ 4W/hqoyVhpkTvzUFG0P3+jC5WzS4q6ITwByHyvVl1C/YOkn3KDiCYbq7DG1vuuxd4de5 wCLhyQ/CmvE15GG6IUKzgGte0oDr4Aiyvb6hM10cmSnfqgkHlDkxOkp5UnzHwhcm8sYZ MEBg/HFfuGnDqzh2KwSr1BSNfnYELESd5ShhxeGKLDY8BUwrLE1LyzpUgBNPSotbbOuO ee6NVdT21l3oMGj12KVqAh/WD78obM9RcWWqF86gnDAUYihIaXjy2Usd+rZW5sfkiZHH DY7w== X-Forwarded-Encrypted: i=1; AJvYcCWZTvaEGGis5O5wJ+aDpu8a13BY5qXtJR3lCHW65H5asBnhX9RmX9qaCH9mBdE04yFWRbg0+QahCZFV1k4=@vger.kernel.org X-Gm-Message-State: AOJu0YzAbGzYde35Y+fxnQi3NnQPX78G8Fb216cWEcZ3ZuVScVSEvm1S YcjR6MzTcldeKOz1HLzwpsSYx/d0Fqv4/5fTzrlFLEG0fiLJLqwZkjS4blPk5Q== X-Gm-Gg: ASbGnct3VyBvzZrdWIsZz1Fp/a9iYz7ez8Qfvw7bK1kIV1+fdb3LR8/p0ox+4ODRcvQ guvFyfRmAtNrdtV7/XsjscAvTf9Lc/uBwy9Aupbs1i2zX4GMrYQhtYjp+KqxFOUP9G6+eAhMNm/ wZLzh4q36KJZw1KGKsG+qw4iyec68JEhEAyrzarNyXYMclQrReDPLXHX+EHmzE7Ayeq1JBokzb6 TEBw5tWRlX8SKDm9tCnU3GpPkB6CnEocH/ShLBeE61mw92OU259lwMFpxmP44lnE67+L8PH/DSo HMso76hzTXa5Ee81nA== X-Google-Smtp-Source: AGHT+IFqSB8efCw3bZ9x/H3wNB/h7WT1Sozgx+u9wqyq8bV1nirLfbUOM2pE4SmRpm8pCa8TR9oGIg== X-Received: by 2002:a17:902:ef05:b0:220:e91f:4408 with SMTP id d9443c01a7336-220e91f487dmr29258795ad.22.1739508812669; Thu, 13 Feb 2025 20:53:32 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:942d:9291:22aa:8126]) by smtp.gmail.com with UTF8SMTPSA id 98e67ed59e1d1-2fbf98b4c45sm4198281a91.2.2025.02.13.20.53.30 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 13 Feb 2025 20:53:32 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Hillf Danton , Kairui Song , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v6 12/17] zsmalloc: introduce new object mapping API Date: Fri, 14 Feb 2025 13:50:24 +0900 Message-ID: <20250214045208.1388854-13-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog In-Reply-To: <20250214045208.1388854-1-senozhatsky@chromium.org> References: <20250214045208.1388854-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Current object mapping API is a little cumbersome. First, it's inconsistent, sometimes it returns with page-faults disabled and sometimes with page-faults enabled. Second, and most importantly, it enforces atomicity restrictions on its users. zs_map_object() has to return a liner object address which is not always possible because some objects span multiple physical (non-contiguous) pages. For such objects zsmalloc uses a per-CPU buffer to which object's data is copied before a pointer to that per-CPU buffer is returned back to the caller. This leads to another, final, issue - extra memcpy(). Since the caller gets a pointer to per-CPU buffer it can memcpy() data only to that buffer, and during zs_unmap_object() zsmalloc will memcpy() from that per-CPU buffer to physical pages that object in question spans across. New API splits functions by access mode: - zs_obj_read_begin(handle, local_copy) Returns a pointer to handle memory. For objects that span two physical pages a local_copy buffer is used to store object's data before the address is returned to the caller. Otherwise the object's page is kmap_local mapped directly. - zs_obj_read_end(handle, buf) Unmaps the page if it was kmap_local mapped by zs_obj_read_begin(). - zs_obj_write(handle, buf, len) Copies len-bytes from compression buffer to handle memory (takes care of objects that span two pages). This does not need any additional (e.g. per-CPU) buffers and writes the data directly to zsmalloc pool pages. In terms of performance, on a synthetic and completely reproducible test that allocates fixed number of objects of fixed sizes and iterates over those objects, first mapping in RO then in RW mode: OLD API =3D=3D=3D=3D=3D=3D=3D 3 first results out of 10 369,205,778 instructions # 0.80 insn per cycle 40,467,926 branches # 113.732 M/sec 369,002,122 instructions # 0.62 insn per cycle 40,426,145 branches # 189.361 M/sec 369,036,706 instructions # 0.63 insn per cycle 40,430,860 branches # 204.105 M/sec [..] NEW API =3D=3D=3D=3D=3D=3D=3D 3 first results out of 10 265,799,293 instructions # 0.51 insn per cycle 29,834,567 branches # 170.281 M/sec 265,765,970 instructions # 0.55 insn per cycle 29,829,019 branches # 161.602 M/sec 265,764,702 instructions # 0.51 insn per cycle 29,828,015 branches # 189.677 M/sec [..] T-test on all 10 runs =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Difference at 95.0% confidence -1.03219e+08 +/- 55308.7 -27.9705% +/- 0.0149878% (Student's t, pooled s =3D 58864.4) The old API will stay around until the remaining users switch to the new one. After that we'll also remove zsmalloc per-CPU buffer and CPU hotplug handling. The split of map(RO) and map(WO) into read_{begin/end}/write is suggested by Yosry Ahmed. Suggested-by: Yosry Ahmed Signed-off-by: Sergey Senozhatsky --- include/linux/zsmalloc.h | 8 +++ mm/zsmalloc.c | 129 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 137 insertions(+) diff --git a/include/linux/zsmalloc.h b/include/linux/zsmalloc.h index a48cd0ffe57d..7d70983cf398 100644 --- a/include/linux/zsmalloc.h +++ b/include/linux/zsmalloc.h @@ -58,4 +58,12 @@ unsigned long zs_compact(struct zs_pool *pool); unsigned int zs_lookup_class_index(struct zs_pool *pool, unsigned int size= ); =20 void zs_pool_stats(struct zs_pool *pool, struct zs_pool_stats *stats); + +void *zs_obj_read_begin(struct zs_pool *pool, unsigned long handle, + void *local_copy); +void zs_obj_read_end(struct zs_pool *pool, unsigned long handle, + void *handle_mem); +void zs_obj_write(struct zs_pool *pool, unsigned long handle, + void *handle_mem, size_t mem_len); + #endif diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index bc679a3e1718..51d8080f187a 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -1432,6 +1432,135 @@ void zs_unmap_object(struct zs_pool *pool, unsigned= long handle) } EXPORT_SYMBOL_GPL(zs_unmap_object); =20 +void *zs_obj_read_begin(struct zs_pool *pool, unsigned long handle, + void *local_copy) +{ + struct zspage *zspage; + struct zpdesc *zpdesc; + unsigned long obj, off; + unsigned int obj_idx; + struct size_class *class; + void *addr; + + WARN_ON(in_interrupt()); + + /* Guarantee we can get zspage from handle safely */ + read_lock(&pool->lock); + obj =3D handle_to_obj(handle); + obj_to_location(obj, &zpdesc, &obj_idx); + zspage =3D get_zspage(zpdesc); + + /* Make sure migration doesn't move any pages in this zspage */ + zspage_read_lock(zspage); + read_unlock(&pool->lock); + + class =3D zspage_class(pool, zspage); + off =3D offset_in_page(class->size * obj_idx); + + if (off + class->size <=3D PAGE_SIZE) { + /* this object is contained entirely within a page */ + addr =3D kmap_local_zpdesc(zpdesc); + addr +=3D off; + } else { + size_t sizes[2]; + + /* this object spans two pages */ + sizes[0] =3D PAGE_SIZE - off; + sizes[1] =3D class->size - sizes[0]; + addr =3D local_copy; + + memcpy_from_page(addr, zpdesc_page(zpdesc), + off, sizes[0]); + zpdesc =3D get_next_zpdesc(zpdesc); + memcpy_from_page(addr + sizes[0], + zpdesc_page(zpdesc), + 0, sizes[1]); + } + + if (!ZsHugePage(zspage)) + addr +=3D ZS_HANDLE_SIZE; + + return addr; +} +EXPORT_SYMBOL_GPL(zs_obj_read_begin); + +void zs_obj_read_end(struct zs_pool *pool, unsigned long handle, + void *handle_mem) +{ + struct zspage *zspage; + struct zpdesc *zpdesc; + unsigned long obj, off; + unsigned int obj_idx; + struct size_class *class; + + obj =3D handle_to_obj(handle); + obj_to_location(obj, &zpdesc, &obj_idx); + zspage =3D get_zspage(zpdesc); + class =3D zspage_class(pool, zspage); + off =3D offset_in_page(class->size * obj_idx); + + if (off + class->size <=3D PAGE_SIZE) { + if (!ZsHugePage(zspage)) + off +=3D ZS_HANDLE_SIZE; + handle_mem -=3D off; + kunmap_local(handle_mem); + } + + zspage_read_unlock(zspage); +} +EXPORT_SYMBOL_GPL(zs_obj_read_end); + +void zs_obj_write(struct zs_pool *pool, unsigned long handle, + void *handle_mem, size_t mem_len) +{ + struct zspage *zspage; + struct zpdesc *zpdesc; + unsigned long obj, off; + unsigned int obj_idx; + struct size_class *class; + + WARN_ON(in_interrupt()); + + /* Guarantee we can get zspage from handle safely */ + read_lock(&pool->lock); + obj =3D handle_to_obj(handle); + obj_to_location(obj, &zpdesc, &obj_idx); + zspage =3D get_zspage(zpdesc); + + /* Make sure migration doesn't move any pages in this zspage */ + zspage_read_lock(zspage); + read_unlock(&pool->lock); + + class =3D zspage_class(pool, zspage); + off =3D offset_in_page(class->size * obj_idx); + + if (off + class->size <=3D PAGE_SIZE) { + /* this object is contained entirely within a page */ + void *dst =3D kmap_local_zpdesc(zpdesc); + + if (!ZsHugePage(zspage)) + off +=3D ZS_HANDLE_SIZE; + memcpy(dst + off, handle_mem, mem_len); + kunmap_local(dst); + } else { + /* this object spans two pages */ + size_t sizes[2]; + + off +=3D ZS_HANDLE_SIZE; + sizes[0] =3D PAGE_SIZE - off; + sizes[1] =3D mem_len - sizes[0]; + + memcpy_to_page(zpdesc_page(zpdesc), off, + handle_mem, sizes[0]); + zpdesc =3D get_next_zpdesc(zpdesc); + memcpy_to_page(zpdesc_page(zpdesc), 0, + handle_mem + sizes[0], sizes[1]); + } + + zspage_read_unlock(zspage); +} +EXPORT_SYMBOL_GPL(zs_obj_write); + /** * zs_huge_class_size() - Returns the size (in bytes) of the first huge * zsmalloc &size_class. --=20 2.48.1.601.g30ceb7b040-goog From nobody Fri Dec 19 18:31:38 2025 Received: from mail-pl1-f177.google.com (mail-pl1-f177.google.com [209.85.214.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8D55B18FDA5 for ; Fri, 14 Feb 2025 04:53:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739508821; cv=none; b=sftnM6kpi9uy3rx+pRp/mDV3PuR1+nb9RN1YZ5QfJMjOzqSXOEYrKq1w03kJAnRjB5rW3XVepmCsplV4JgSdrhn2dg07LABJ0/aGo0/mqu0JnaLh8qJwFxmHri1En55BQICCHx25Z2YODTSSwPDxFDA5J2Tz1mpp6Un9jF4npec= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739508821; c=relaxed/simple; bh=COsjrWCrhKbYAUy1yAf/i0nha4gWII3XHI6ohEWeZ4s=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=P/MyV527DacUNYLABQyPTZPDv+kplZ/9ORsKr9g7lha8pQkUUwsL41rdtZPpsGVXD9SnezEchgWl+8xFRABB9iyRJ/OFkKveXkMPDZQJUxVTbCt8/nttyvErlHD51j2Vks9WzrqkPFs81vuJXU8LCflh5tVl1yQnL9lRasXGDTI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=iapLYMpT; arc=none smtp.client-ip=209.85.214.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="iapLYMpT" Received: by mail-pl1-f177.google.com with SMTP id d9443c01a7336-21f2339dcfdso25110725ad.1 for ; Thu, 13 Feb 2025 20:53:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1739508819; x=1740113619; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=g8MSYx1r6vOiRHwxjnCDNpKvIAk9Qp1UAjAoAUDj4z4=; b=iapLYMpTrM2FlHXqcT8qiHAFx20mP/GrXKuXcadlP31bx9hXGCZpWhrd1AabJsqlcR K8UA6zterGxnW3/pIL8vOWcLpgzWeV6nwz4kwdLavVx8KIUS5ZSUApqQi0YXzn49v5LH 8WxZdxL94YtKXDwC/0Z69PlFATmif6fD2ls9c= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739508819; x=1740113619; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=g8MSYx1r6vOiRHwxjnCDNpKvIAk9Qp1UAjAoAUDj4z4=; b=Lp77TOlFargiWXXQCbDknlNpSPqSXy1oUJAyYc7lpVFCDlHTr5O3vQTYcj+VU4KpPq LYKE7e7Ro4fPFaFzBMDrteWB9gAxWKiZnMOs8JgNGiDCnW+IWISHFtotfeSYq08QtW1l Gx1nH0Bcb3vnO8xwUba+cEOZArK+4IPjTBcOkWiSrbThMo+sOgJbRp66DB/PCFe3sQR3 vEqPd3b+JNPGvGvG3tfG54Vu2a3arqpjC8WwI3HhAvCj2hvDHZXUOJpqWysTzVXZLXK7 VfdB5A6ut/YPcQ+0JsVfkRdGuHoArKGfi2iHVCTDhXq+hpOFkube+8PqUm5/4GtLMHGI Ue9A== X-Forwarded-Encrypted: i=1; AJvYcCX12JwOudXQPKfVCS9MfAuyqhnlKK50yydg9MgteW3ZMV6IFvA5LCcCgzHoVD/07XUTti3/Vvi0N6MuBbU=@vger.kernel.org X-Gm-Message-State: AOJu0Yxb0pf/jYDmRYeszsbcuWcxJftPeYcIhMagSuyamcnxh3id7OCZ Ddg2BQLLeH3F5roe5AMpwT9xPyhP7lhYCglHFHb4A3uckgSkPRFZhsynf0Ztmw== X-Gm-Gg: ASbGncuVVHDHwN7NP4KsBPJ48/TGoH7xFiQy5Fs2L/2JKEp8kl+7kt5tckPs3HGtCkL eCWS1Db1GClpX0UYDVhKTpOB7+1aZ8lF2VFKqLpc2skMcA8d2IqABiQWNReFXe3MQC0Aow8O+bj qNQSWEZH0H7qsExPJ/ZxX8SdvXGAITVOHWmU1eWh7qzeRNtAVRM4+jnIIJWAAYU+Http9rq7T/F xZg7kF27ZkOlx4KwIGIghdYodOOmOLuB1GPe+86OL/dmsauXQdXHyIZn2D44p4NRW5cUgLShMQt 23ZruWe8Om2q6JVp5w== X-Google-Smtp-Source: AGHT+IFj0J84blWWRorX7QC5QTsPm4yxqb/UskRAxNtWaOKmY3i/R9l8RGftlk8+B1lrXJZdO59P7Q== X-Received: by 2002:a17:903:18a:b0:220:e98e:4f1b with SMTP id d9443c01a7336-220e98e52b9mr44389295ad.0.1739508818756; Thu, 13 Feb 2025 20:53:38 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:942d:9291:22aa:8126]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-220d53492d0sm20895595ad.35.2025.02.13.20.53.36 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 13 Feb 2025 20:53:38 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Hillf Danton , Kairui Song , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v6 13/17] zram: switch to new zsmalloc object mapping API Date: Fri, 14 Feb 2025 13:50:25 +0900 Message-ID: <20250214045208.1388854-14-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog In-Reply-To: <20250214045208.1388854-1-senozhatsky@chromium.org> References: <20250214045208.1388854-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Use new read/write zsmalloc object API. For cases when RO mapped object spans two physical pages (requires temp buffer) compression streams now carry around one extra physical page. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zcomp.c | 4 +++- drivers/block/zram/zcomp.h | 2 ++ drivers/block/zram/zram_drv.c | 28 ++++++++++------------------ 3 files changed, 15 insertions(+), 19 deletions(-) diff --git a/drivers/block/zram/zcomp.c b/drivers/block/zram/zcomp.c index cfdde2e0748a..a1d627054bb1 100644 --- a/drivers/block/zram/zcomp.c +++ b/drivers/block/zram/zcomp.c @@ -45,6 +45,7 @@ static const struct zcomp_ops *backends[] =3D { static void zcomp_strm_free(struct zcomp *comp, struct zcomp_strm *zstrm) { comp->ops->destroy_ctx(&zstrm->ctx); + vfree(zstrm->local_copy); vfree(zstrm->buffer); zstrm->buffer =3D NULL; } @@ -57,12 +58,13 @@ static int zcomp_strm_init(struct zcomp *comp, struct z= comp_strm *zstrm) if (ret) return ret; =20 + zstrm->local_copy =3D vzalloc(PAGE_SIZE); /* * allocate 2 pages. 1 for compressed data, plus 1 extra for the * case when compressed size is larger than the original one */ zstrm->buffer =3D vzalloc(2 * PAGE_SIZE); - if (!zstrm->buffer) { + if (!zstrm->buffer || !zstrm->local_copy) { zcomp_strm_free(comp, zstrm); return -ENOMEM; } diff --git a/drivers/block/zram/zcomp.h b/drivers/block/zram/zcomp.h index 23b8236b9090..25339ed1e07e 100644 --- a/drivers/block/zram/zcomp.h +++ b/drivers/block/zram/zcomp.h @@ -34,6 +34,8 @@ struct zcomp_strm { struct mutex lock; /* compression buffer */ void *buffer; + /* local copy of handle memory */ + void *local_copy; struct zcomp_ctx ctx; }; =20 diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 2e46ea542ecd..87ce304a60aa 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1603,11 +1603,11 @@ static int read_incompressible_page(struct zram *zr= am, struct page *page, void *src, *dst; =20 handle =3D zram_get_handle(zram, index); - src =3D zs_map_object(zram->mem_pool, handle, ZS_MM_RO); + src =3D zs_obj_read_begin(zram->mem_pool, handle, NULL); dst =3D kmap_local_page(page); copy_page(dst, src); kunmap_local(dst); - zs_unmap_object(zram->mem_pool, handle); + zs_obj_read_end(zram->mem_pool, handle, src); =20 return 0; } @@ -1625,11 +1625,11 @@ static int read_compressed_page(struct zram *zram, = struct page *page, u32 index) prio =3D zram_get_priority(zram, index); =20 zstrm =3D zcomp_stream_get(zram->comps[prio]); - src =3D zs_map_object(zram->mem_pool, handle, ZS_MM_RO); + src =3D zs_obj_read_begin(zram->mem_pool, handle, zstrm->local_copy); dst =3D kmap_local_page(page); ret =3D zcomp_decompress(zram->comps[prio], zstrm, src, size, dst); kunmap_local(dst); - zs_unmap_object(zram->mem_pool, handle); + zs_obj_read_end(zram->mem_pool, handle, src); zcomp_stream_put(zstrm); =20 return ret; @@ -1725,7 +1725,7 @@ static int write_incompressible_page(struct zram *zra= m, struct page *page, u32 index) { unsigned long handle; - void *src, *dst; + void *src; =20 /* * This function is called from preemptible context so we don't need @@ -1742,11 +1742,9 @@ static int write_incompressible_page(struct zram *zr= am, struct page *page, return -ENOMEM; } =20 - dst =3D zs_map_object(zram->mem_pool, handle, ZS_MM_WO); src =3D kmap_local_page(page); - memcpy(dst, src, PAGE_SIZE); + zs_obj_write(zram->mem_pool, handle, src, PAGE_SIZE); kunmap_local(src); - zs_unmap_object(zram->mem_pool, handle); =20 zram_slot_lock(zram, index); zram_set_flag(zram, index, ZRAM_HUGE); @@ -1767,7 +1765,7 @@ static int zram_write_page(struct zram *zram, struct = page *page, u32 index) int ret =3D 0; unsigned long handle; unsigned int comp_len; - void *dst, *mem; + void *mem; struct zcomp_strm *zstrm; unsigned long element; bool same_filled; @@ -1813,11 +1811,8 @@ static int zram_write_page(struct zram *zram, struct= page *page, u32 index) return -ENOMEM; } =20 - dst =3D zs_map_object(zram->mem_pool, handle, ZS_MM_WO); - - memcpy(dst, zstrm->buffer, comp_len); + zs_obj_write(zram->mem_pool, handle, zstrm->buffer, comp_len); zcomp_stream_put(zstrm); - zs_unmap_object(zram->mem_pool, handle); =20 zram_slot_lock(zram, index); zram_set_handle(zram, index, handle); @@ -1929,7 +1924,7 @@ static int recompress_slot(struct zram *zram, u32 ind= ex, struct page *page, unsigned int comp_len_new; unsigned int class_index_old; unsigned int class_index_new; - void *src, *dst; + void *src; int ret =3D 0; =20 handle_old =3D zram_get_handle(zram, index); @@ -2039,12 +2034,9 @@ static int recompress_slot(struct zram *zram, u32 in= dex, struct page *page, return PTR_ERR((void *)handle_new); } =20 - dst =3D zs_map_object(zram->mem_pool, handle_new, ZS_MM_WO); - memcpy(dst, zstrm->buffer, comp_len_new); + zs_obj_write(zram->mem_pool, handle_new, zstrm->buffer, comp_len_new); zcomp_stream_put(zstrm); =20 - zs_unmap_object(zram->mem_pool, handle_new); - zram_free_page(zram, index); zram_set_handle(zram, index, handle_new); zram_set_obj_size(zram, index, comp_len_new); --=20 2.48.1.601.g30ceb7b040-goog From nobody Fri Dec 19 18:31:38 2025 Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2C6D418FC9F for ; Fri, 14 Feb 2025 04:53:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739508827; cv=none; b=QyjbxsiJRD/37SUXBvoNMEXiU0ynWJnjzjyuPyBY8frC/kX0vCPMty4rF5nbD76ll4fBXRIW7sW2TQvSTxI+VUkOgBKY1fWBEx0sk0vcujtLowgOZyfaWbYfbNFniGUnXrkQAXWZhevvWiJ/pDyAxs6cd5a876HpyzCtgjE0iPU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739508827; c=relaxed/simple; bh=3bl4FZa1Jj9/SryddhJVpujsF+YKBcOIbQtdPgtbXnI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=TX2t7of7DsEcT/1E/imAWGJCVS+q/T4FbViIDOJHR6WR4Whnte1UbbR2nvcLo3WcEgIGvqyKoXPj99qZplnrG0TSxDzOjkChwQgzREy9LqhkEFFLuQlbpqvTsBVq96ZQ8q4X0GGgM1/M/NJCl0hZWPAnkXb6VJgWZ4vDspyc2wU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=UdYFM98s; arc=none smtp.client-ip=209.85.214.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="UdYFM98s" Received: by mail-pl1-f182.google.com with SMTP id d9443c01a7336-220c8cf98bbso32562385ad.1 for ; Thu, 13 Feb 2025 20:53:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1739508825; x=1740113625; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=cvgHEnC+lRfOrRLv2RIVOYlOBk4EIx2tIzOwTFt2Qag=; b=UdYFM98se8kl4KobKhRtHWD97BzEtVkFXufCI80+8os4v3p4trqh7cETGGCIy5uC5F XvagkRB51cr47Rg1R6rZoePCDAkHP+KSadNDafZJ1sI09JHYCPalQmNIod3ja7t1k9Z3 ROXAKIw37K84VDOuPuGhMdODafpvr/pGvB3Ew= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739508825; x=1740113625; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=cvgHEnC+lRfOrRLv2RIVOYlOBk4EIx2tIzOwTFt2Qag=; b=o1Vk4xZc2v0790I7EcsWxHs6DrIsfcbslzD0GXusgbsRuN45MFISDfoeHgfRjX9PWw 5N8c6fTZnH40NgbiGpL7DLLJGE0vql44JfjsI+TZmdSLeG7potOdynmTC/7176xMVZFn INmDdFf/Wox1jzolbPTqu3zrbYJQDpL1GFi33WNRzSmJPvxFbB+LnQ82Y31gJ6GcIX41 Rl2/uG+Hon8PwKU7Ve0jYkovtoF/eCCE3j2mwhXnaxrMwl/ddjllnYrD4KhwyonCBoPg gf5E/vrlKlWLptkfCCMew9fih62DcDsVoVMp/jCaMbt2RZdkwAWI2vMsBTaASNtVgIl/ j4tQ== X-Forwarded-Encrypted: i=1; AJvYcCX8OwVWNfcHdbEHs8riGwCGudpW0aqxyBxP8AylUZf0piTrwZ5h3w4pz0q818zrwEClpikHONQzJYNJJ4Q=@vger.kernel.org X-Gm-Message-State: AOJu0YzZdGXW2IRnApcnD9EmtUTeJAkDuJ2qBrNkH/L+A/TCI0KNRazm BR+R03hHkCwpDcFTJ766cs2eg/ypRg83WFyX+hQGSX7afcky9oBYBifg/nuDeA== X-Gm-Gg: ASbGncsv7d/5QPXShj7d+okr7wwki6Hr8HuFPNwzQVZceSZGjAPR1WNV2KdmO3iwZLO l77p3rBOdbBMM2vH1g7/rcIWMIqYhlLAaSX0qn53QhWq5VZt+Xque1W9Ks+fd5ml7YKt9Bjh5BV v7SlLPk6ztoL9gzVsbAFLAfNnF4p9YmlD8mEyNg6J26Nq1MFuHjg0D+IyUNrfvffb+zgbRAWL7c pvSlNIdsjruN0SLv2WOxT7yz8mUFF8ANUpNv3fTbgkNy0Ka557bJYIfLLgqtZqv0xkQYE84pWH9 TbzIHii3RrmJvqIn7g== X-Google-Smtp-Source: AGHT+IFtBGqgz6iEWt7LMNhyXXy7Ao2/LISUPvugqZVTStAhfwtsv9UYgCngz+ynDyppB1uJogIN/g== X-Received: by 2002:a05:6a21:6d85:b0:1ee:85b2:ef73 with SMTP id adf61e73a8af0-1ee85b2efcemr458480637.34.1739508825453; Thu, 13 Feb 2025 20:53:45 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:942d:9291:22aa:8126]) by smtp.gmail.com with UTF8SMTPSA id 41be03b00d2f7-adb5a92c6d8sm2017586a12.65.2025.02.13.20.53.43 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 13 Feb 2025 20:53:45 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Hillf Danton , Kairui Song , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v6 14/17] zram: permit reclaim in zstd custom allocator Date: Fri, 14 Feb 2025 13:50:26 +0900 Message-ID: <20250214045208.1388854-15-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog In-Reply-To: <20250214045208.1388854-1-senozhatsky@chromium.org> References: <20250214045208.1388854-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" When configured with pre-trained compression/decompression dictionary support, zstd requires custom memory allocator, which it calls internally from compression()/decompression() routines. That means allocation from atomic context (either under entry spin-lock, or per-CPU local-lock or both). Now, with non-atomic zram read()/write(), those limitations are relaxed and we can allow direct and indirect reclaim. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/backend_zstd.c | 11 +++-------- 1 file changed, 3 insertions(+), 8 deletions(-) diff --git a/drivers/block/zram/backend_zstd.c b/drivers/block/zram/backend= _zstd.c index 1184c0036f44..53431251ea62 100644 --- a/drivers/block/zram/backend_zstd.c +++ b/drivers/block/zram/backend_zstd.c @@ -24,19 +24,14 @@ struct zstd_params { /* * For C/D dictionaries we need to provide zstd with zstd_custom_mem, * which zstd uses internally to allocate/free memory when needed. - * - * This means that allocator.customAlloc() can be called from zcomp_compre= ss() - * under local-lock (per-CPU compression stream), in which case we must use - * GFP_ATOMIC. - * - * Another complication here is that we can be configured as a swap device. */ static void *zstd_custom_alloc(void *opaque, size_t size) { - if (!preemptible()) + /* Technically this should not happen */ + if (WARN_ON_ONCE(!preemptible())) return kvzalloc(size, GFP_ATOMIC); =20 - return kvzalloc(size, __GFP_KSWAPD_RECLAIM | __GFP_NOWARN); + return kvzalloc(size, GFP_NOIO | __GFP_NOWARN); } =20 static void zstd_custom_free(void *opaque, void *address) --=20 2.48.1.601.g30ceb7b040-goog From nobody Fri Dec 19 18:31:38 2025 Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8E1D11922EE for ; Fri, 14 Feb 2025 04:53:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739508834; cv=none; b=QJst0rN3QX9Gd62al3OTfxYyDxGdUX0aLCeM+Km0Oy2Ozw8sntRhcN9CwjNLwzWJJHl79IDc/f4wY+RDJpLvau10n5+ODw/j1vIfFiesjEsA/3CI7xRyaLP8Myouts8a2eVTqmjRh4oibO5xF+5ljRvdxTXQHdTcusgRMCaTfUY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739508834; c=relaxed/simple; bh=Q00m6RhnC2QFthHDOznvkFOKfPvZy0Gqt9YMxLMX4xQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=mD27tFbsdI3LXg1VMWCcOFqwvwSscQrTM+TVblEOwnK0FGM8iWwWFSxC5W/r+NFNcIZL1yPB/ZLqF5vGSOrZFXlVbqdMNLNEiPcw1tNA67+1zRYTfDNwH8sFHkWmQm2QnNHdWw5puKY2emrA3MWOpURzBZ4oBJDTUGUlquSI2kA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=HrnUaK/2; arc=none smtp.client-ip=209.85.214.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="HrnUaK/2" Received: by mail-pl1-f169.google.com with SMTP id d9443c01a7336-21f49bd087cso24743665ad.0 for ; Thu, 13 Feb 2025 20:53:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1739508832; x=1740113632; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=dOTlfVzF3e2Wm9f7wmP5BEorbNSbvme8XOlj2WM+zhc=; b=HrnUaK/2POlzpked4ms4Y91XsPUU4V6VlbAjaCfUP6J/91AythWYng8gWEnnHuxO58 Aqt8S9Ev7PlYxUvKtyHjTuInDLDPKYRSFPERjORYfGIup/uJR1vsGrowTY8ywKbt8sUv WdAj29pPn/gg6V86So9x6dRC2Law4ufR0XCmc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739508832; x=1740113632; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=dOTlfVzF3e2Wm9f7wmP5BEorbNSbvme8XOlj2WM+zhc=; b=iZGfOlxkNCVHXtheyMo1QYhnWPjVivwhMfewbItGutA0MVKA/vVBAAcKQWM4oMSn2H OzVTtROq3gZ2yay2vQ+FrgDRjri7tFhc/4A/DUagK2G/cgJ0WQhuwi+R4mn13rwXsStR Wa/OD1NFjuH/E0Vu1vbgpikpi314Q9P38J1z0ELOJygtOqV5BiYVDPGXodMRv3Ne1xH2 XvLRHpLnrGGFncNoQOiLGpA0T/yTTbATkZJle9GP2Z/raAVduQdkuUz6uM5GwD318W8d 5ujPLZfP13MrgHwLx1Dy/Fp5F3xVnsh9iknns4NGSlB8VoNVmLIrXox9AouK/EvGOfOH W/9Q== X-Forwarded-Encrypted: i=1; AJvYcCUATbzuhoLZ36eGGXo3QT71GGgRxgTViiUB6N9X5wWBqyXOnwYjbZXwIjT+nMBQsSfWqWZxEEQZ03FSEEc=@vger.kernel.org X-Gm-Message-State: AOJu0YxvR/gdWHc+gXgC30rQqyfSrgwBZleQZe+B70VW9E1wWe/5L9bx D5wUdIi2nparwNZC1Qv05HVNfEFuRNuS1Eh+Iwm86gSbzaL4QtxjpALvnqYi1g== X-Gm-Gg: ASbGncvrAFDOwmcGVTjEMMHwossF95jxmroHADYXj1OTF0NHGrGaMRKu1p882Co+kUQ rQxRC7/IXHnRGWdOhihIteYb+AxDcUXBkLFmDP5TnqrhbzY7mNWLr0xR4HKOxWZqVgxKAR0A2Wg cd6ZoJTCFNEc5658NKPVo+jIJUMhUs57mzjKPTaPGfQPrU/40MUBcrobTRHibEZi1+kpVdHxtH9 CQPk3lcB7Np7OqgQQtPxrj/6F9HVBtCjIj0ThRiu8NpdzijHINLoBcZF8XP+2he+sROti9xHXfg 2HzVCSTrIZuEJLs4CQ== X-Google-Smtp-Source: AGHT+IHKsmcyZIg17GYlLEbKgjy5fMkUQ7BmQ1HUJMxSkfACov/TMDteDP157pPV1tX1h7GlE/qMmQ== X-Received: by 2002:a17:902:d48b:b0:215:9bc2:42ec with SMTP id d9443c01a7336-220d216ce9fmr80849875ad.47.1739508832004; Thu, 13 Feb 2025 20:53:52 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:942d:9291:22aa:8126]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-220d5585fbfsm20932385ad.226.2025.02.13.20.53.49 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 13 Feb 2025 20:53:51 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Hillf Danton , Kairui Song , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v6 15/17] zram: do not leak page on recompress_store error path Date: Fri, 14 Feb 2025 13:50:27 +0900 Message-ID: <20250214045208.1388854-16-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog In-Reply-To: <20250214045208.1388854-1-senozhatsky@chromium.org> References: <20250214045208.1388854-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Ensure the page used for local object data is freed on error out path. Fixes: 3f909a60cec1 ("zram: rework recompress target selection strategy") Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 87ce304a60aa..b4661de6bc29 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -2059,7 +2059,7 @@ static ssize_t recompress_store(struct device *dev, struct zram_pp_slot *pps; u32 mode =3D 0, threshold =3D 0; u32 prio, prio_max; - struct page *page; + struct page *page =3D NULL; ssize_t ret; =20 prio =3D ZRAM_SECONDARY_COMP; @@ -2203,9 +2203,9 @@ static ssize_t recompress_store(struct device *dev, cond_resched(); } =20 - __free_page(page); - release_init_lock: + if (page) + __free_page(page); release_pp_ctl(zram, ctl); atomic_set(&zram->pp_in_progress, 0); up_read(&zram->init_lock); --=20 2.48.1.601.g30ceb7b040-goog From nobody Fri Dec 19 18:31:38 2025 Received: from mail-pj1-f41.google.com (mail-pj1-f41.google.com [209.85.216.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 81CAB1922DE for ; Fri, 14 Feb 2025 04:53:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.41 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739508840; cv=none; b=mgxLHM/gN7xSB0ZypI/EkNP9kDjIApLqtS5TxtzlHPt7AWcgqI714tdlmLoMV7M88sDydRxfVoiHaxVGvo54Fth8/qHdw/K4Y1QUQYDsVQOKWnZFk0RKplny5lGbNIjFvoA4c25do1YKTmXhaZBXy23uWh7YCnMPESW9P/OVbRI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739508840; c=relaxed/simple; bh=k/laeVohRtQCUN13mPbpDtF2OZR5zuNMWh4dlmbx9Ls=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=LbXy7ELShD+hu2cEPzxhCC3tE+cMl873nbAY0dXkSJbbkgYgU2IpgKtvv3kr4dHr3sm/2GLKClCaEtus3TWR8EEy9sGgpTqLD3CBmUK78YTK7xqI4ugcCo5sf4+KvaIgr/DcrFFqIEshZQK+1KvdBsU/24tihRrMt+N4MXhD2wk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=jBoWgbLW; arc=none smtp.client-ip=209.85.216.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="jBoWgbLW" Received: by mail-pj1-f41.google.com with SMTP id 98e67ed59e1d1-2f42992f608so2688034a91.0 for ; Thu, 13 Feb 2025 20:53:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1739508838; x=1740113638; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=c3vcsQRfcy+wCE7pS6Fz7pnfmcjhxh9cIXH2LP2gfks=; b=jBoWgbLWCVh1PeTlXM3acW87h4kKHGaQ8AzT/UiXWlW//Oi7Lv1LqFs9NQ29mqAbpW Cm+/mRtZISSWZhfILXq41oC0wbFkZWKnvcibdNH0CBIZDNlD1/uOCTRrZXwgP1OvQmgF 7OgetVGBMMvg8Xjvh7xs60QiIDJwQu3v1oKMA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739508838; x=1740113638; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=c3vcsQRfcy+wCE7pS6Fz7pnfmcjhxh9cIXH2LP2gfks=; b=sr7IkzVKvGKfXgAMctpOcolwNEm988x2tBg4LkTjKF4D8hgqELM/YDIIk1ZFt1N7TT yA+WEi/yvJMKUat9DLqNDags3SThhaiJRLQRn2JFD15cYjRayI/oPivIxXUaIq0jKmHm KcqczmE0W0o7gIvgEcQ3m+F8mBco7R16cKPKCpBemfv4XBfmSfw8Yt4fe91YPalJ1yB5 uPeOn48WeXYMH8Y6vv5LkNVppCuPSjoKfpaEydTSmaQNyGJSdjlEzdqtMLs1LPoGVBD1 hc6dCx7tFTMkExN2c8s0vgXo8et8xojsVMPrj5lRONJmxhY9Jv0rsp72Qu/ReRqbIM5t Jmfg== X-Forwarded-Encrypted: i=1; AJvYcCXajCZ5DmMQzE+DvmdfYVj+P8jl4+XNTr3mNBgEf3bzvD6zcoemfwjFny72udw5heRc5rsJ+I78Rh24bMQ=@vger.kernel.org X-Gm-Message-State: AOJu0YybndzT9GVcX+NZkueB9r5K8YP6ufBR6gaijgadGxoDS6YSLozP eetPNUewXPQ+deGedDqGlYk9izy90zoAJL3H+vnxHp3Q/gvWnqlwjNAxTWIQaunixG95DxRUrdg = X-Gm-Gg: ASbGncvuVycXzZS7/gjetHx9Fu8C2JV+I+HMohC8khA74eDeAQOKQulpT+XzxR/yTKI 91Mah5xEjSaRz7enJMOSB8cExpQ8sktUFB3utaWGfrtHSzfodasjAnXoRm10Ug70oFZoKqNiHdR xkXoWYxBd9G8eIFe4zEo/V0StiGs+VUmQV2wuF/DB9p52en8l7APQxQqJTmIkvBkRTPpJ9+fllB i5K5fL9AFNn2YhBc4b2+9co+SPk1vMaPdgEx+8wtsuZA3+2xyBFyxvCqkfssxZgL9LjPU4+7nxN P2CCr6pXJJGqyjx6eg== X-Google-Smtp-Source: AGHT+IHAcm0g7zYqkHRuT4x03coHX+AA6wb7u2m15AhzZy57LDAWmkRXZjtrdEJDwTr85W6SV3VPqg== X-Received: by 2002:a17:90b:3c48:b0:2fa:1a23:c01d with SMTP id 98e67ed59e1d1-2fc0ea881c9mr7990978a91.21.1739508837865; Thu, 13 Feb 2025 20:53:57 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:942d:9291:22aa:8126]) by smtp.gmail.com with UTF8SMTPSA id 98e67ed59e1d1-2fc13ba6ac7sm2214490a91.45.2025.02.13.20.53.55 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 13 Feb 2025 20:53:57 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Hillf Danton , Kairui Song , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v6 16/17] zram: do not leak page on writeback_store error path Date: Fri, 14 Feb 2025 13:50:28 +0900 Message-ID: <20250214045208.1388854-17-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog In-Reply-To: <20250214045208.1388854-1-senozhatsky@chromium.org> References: <20250214045208.1388854-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Ensure the page used for local object data is freed on error out path. Fixes: 330edc2bc059 (zram: rework writeback target selection strategy) Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index b4661de6bc29..6adaad56f4ce 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -829,7 +829,7 @@ static ssize_t writeback_store(struct device *dev, unsigned long index =3D 0; struct bio bio; struct bio_vec bio_vec; - struct page *page; + struct page *page =3D NULL; ssize_t ret =3D len; int mode, err; unsigned long blk_idx =3D 0; @@ -971,8 +971,10 @@ static ssize_t writeback_store(struct device *dev, =20 if (blk_idx) free_block_bdev(zram, blk_idx); - __free_page(page); + release_init_lock: + if (page) + __free_page(page); release_pp_ctl(zram, ctl); atomic_set(&zram->pp_in_progress, 0); up_read(&zram->init_lock); --=20 2.48.1.601.g30ceb7b040-goog From nobody Fri Dec 19 18:31:38 2025 Received: from mail-pl1-f178.google.com (mail-pl1-f178.google.com [209.85.214.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 886C618A6B8 for ; Fri, 14 Feb 2025 04:54:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739508845; cv=none; b=joo+Ov4wmo0r3VFJA4A29T4U2z5U5xG8l0lNiMHZvBZ/45EpfXtL+uK+7H0cEyoESNmzlxB51i8NKVwG+Jezy/3A/INtfR80sCKotCV3d/KEVHAe7jLOJkpWp3wc9S/CcZDLqlAYx1W+Y/rJd6VF3wUNucLA417mrLOo8s3JEOU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739508845; c=relaxed/simple; bh=/fYuZtpqwtnnGkN/fr/wmgCPg2VCEfWFjvPSN5XmwhA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=MV12a9dPWedjXAtYWP8n1Hm+b3wRmwrGn8UnOrpsNF6kel2HEcvm9d3TGvJfk/G67qI58027ZiLECj3qcQNJV50dYuTQf3busXVoGivY6D8h6TOv7J4LWDGMdUIFtJQfaSb8D5b2C320UsviVn3WqlWCQ+WKrdEDiJyUg9Cyzhc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=oXp5AZNn; arc=none smtp.client-ip=209.85.214.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="oXp5AZNn" Received: by mail-pl1-f178.google.com with SMTP id d9443c01a7336-220e989edb6so16982035ad.1 for ; Thu, 13 Feb 2025 20:54:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1739508843; x=1740113643; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=sFE2Vkh4W4kK9IrN6erQ+4Og/P/7Lb06X2GD4vSsp54=; b=oXp5AZNn6pp3J6fgWR20BpnTVFg21+zKe5IqVuw2yVPeLSnFv8glkI52C9zTGzPk6e aaiMtdLo2x7mpuob/UCS9qLPeXZwVfB84BGS9S6hwS8bvPP0Ztnh2Z8JxZ3ZOO2LZc3p +Agj4DmGfiZ5Xk3b8irV1AMMC/7b1JFI/3W9g= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739508843; x=1740113643; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=sFE2Vkh4W4kK9IrN6erQ+4Og/P/7Lb06X2GD4vSsp54=; b=vnofGsdGsA4GvdyMNzlE2/OnyGKjQcenQ37M5sXMyCul5YNNkJgAa0qO51aC5zCUYi KCtyoGGeV8UbccIBU6G4gs9v9mAuCve0G5/XsgXqWLKw0z0ad9PXxbggCaC2RZ2ykK6p gfLl30NIRdhXjrIjuCKcr9KuCeBHk1642MC5EHLS1xc9gDdSm1wC/eukim2qfLfx0dIT dckCYedWZYurhm2xuS7JRI/LBJwQaiSbK5bZb5ef+1wIeQbcQPxfQ0ms/SE51eiEeJMp 66LLfP3spJT0qDb9Vkt833qyu8PM6T+mtocLVM6NBOdyG7W3T8tA5eX7URfhacBk2gLr 1t2A== X-Forwarded-Encrypted: i=1; AJvYcCURCuWdrUvGfCUip3r/Fvn7P+w01cKmZGhXTrk41FzusRRR6yRotWWmkWNKOr45lR85WpU5tlKB3Xx2ZHo=@vger.kernel.org X-Gm-Message-State: AOJu0YwQsAyOWdZY7OjgCg9zXt0iKEjOyzCsCJUs9gi9GAJFgoWIOaAh BFUp/gIkEiMt+GeeioBtrmYH7O8582lwFMQziiPw5EahibNDE5Zk/k5pWMpqGg== X-Gm-Gg: ASbGncswyb9nV6WCWx53p+ubX3IOeWOIG3Wm7QSUsv0n03vbxhTi3VTKU7TLTrMHeZf Rx5zXyuNx1dxpZkJakipA3ahuCNAFFuIcFk3kHcMnBR8n9Ob5YspMmqIRsHJqD27E59wuBbQ0sS yjKbvqlehEw0GeKVyN3mvUJZJPD0zkuEngjLl7oGYqVKD8LUJ0LxcedEZTXV9+6xjQjBPyn26CI jms68irZswq8GOiDQ96MIp0SzhROylsvXCmyHmnTIGF0oivUn9AAxEVZuRvl5sX7Bf3HIxMSMNA Z/J4fftBZQLeHIDKEA== X-Google-Smtp-Source: AGHT+IEdA+mk+O5VEtAV37EuQ9q5WsoauseiUVXxS1yqw/ceGVJW5mSRJN4cdhO3tV9GqvV3fWIxjg== X-Received: by 2002:a17:902:f550:b0:21e:ff3a:7593 with SMTP id d9443c01a7336-220d1ec17dbmr77579475ad.6.1739508842864; Thu, 13 Feb 2025 20:54:02 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:942d:9291:22aa:8126]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-220d5349394sm20869245ad.49.2025.02.13.20.54.00 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 13 Feb 2025 20:54:02 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Hillf Danton , Kairui Song , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v6 17/17] zram: add might_sleep to zcomp API Date: Fri, 14 Feb 2025 13:50:29 +0900 Message-ID: <20250214045208.1388854-18-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog In-Reply-To: <20250214045208.1388854-1-senozhatsky@chromium.org> References: <20250214045208.1388854-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Explicitly state that zcomp compress/decompress must be called from non-atomic context. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zcomp.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/drivers/block/zram/zcomp.c b/drivers/block/zram/zcomp.c index a1d627054bb1..d26a58c67e95 100644 --- a/drivers/block/zram/zcomp.c +++ b/drivers/block/zram/zcomp.c @@ -146,6 +146,7 @@ int zcomp_compress(struct zcomp *comp, struct zcomp_str= m *zstrm, }; int ret; =20 + might_sleep(); ret =3D comp->ops->compress(comp->params, &zstrm->ctx, &req); if (!ret) *dst_len =3D req.dst_len; @@ -162,6 +163,7 @@ int zcomp_decompress(struct zcomp *comp, struct zcomp_s= trm *zstrm, .dst_len =3D PAGE_SIZE, }; =20 + might_sleep(); return comp->ops->decompress(comp->params, &zstrm->ctx, &req); } =20 --=20 2.48.1.601.g30ceb7b040-goog