From nobody Tue Dec 16 19:24:32 2025 Received: from mail-pj1-f44.google.com (mail-pj1-f44.google.com [209.85.216.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B1B8217CA1B for ; Mon, 3 Mar 2025 02:24:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.44 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740968683; cv=none; b=Jh7MyIrLDr0INay0xHqKP50yV8idmhWRNz00NuRX9BriO5Fl5rybz0Ts188sZ19gqquOUNnPMQzSSyh5kZl6K2GePdyg3xvW5ANyJSbVnH6UW2pnJkJ+IY/B/ypxVAFccYlYaI98bHtEezcwMs9EgeYIYSzA0ch+MYrpRWTs6ws= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740968683; c=relaxed/simple; bh=Ueba+oa0oa0PlNzLN9Uzd8MRvSnqENM622OyfHdWL+o=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=rc0aAWALOqxUV8SRzZHOn8Kiux6p+wsxz0cuznukNkKS9BtURNODTdiXb6Dt91X5lpNWDSKuflYb7FWTvcqOiDAzcvebU/U19d1vbDW2YjxRObLMMOQvkVq2yjWaoh1RyA3S/4GNL7rblLoy72q/OD0DMILOQFYQMXTWjSxlw7I= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=fcdQkdjA; arc=none smtp.client-ip=209.85.216.44 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="fcdQkdjA" Received: by mail-pj1-f44.google.com with SMTP id 98e67ed59e1d1-2f9b9c0088fso6591084a91.0 for ; Sun, 02 Mar 2025 18:24:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1740968681; x=1741573481; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=24epfTRD5B0/Gn0qvdHVkfmT+fNP4MZPLg9AakHcZPk=; b=fcdQkdjArD+e0N5ajKPC2TgXyj2kIraBpBQpLvB9yzX+eYZwvHH863qGOu3eTO0/5F 6zOa9pcUdPcompt4bmCo0QlqC2D8iTdD9yTi8WGEHsb/2I6RASB9EumE2aUPELfa3T2c ADgR6OFpm7Ys4MzY1QWzPICrQUxezV5Ct22Lg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740968681; x=1741573481; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=24epfTRD5B0/Gn0qvdHVkfmT+fNP4MZPLg9AakHcZPk=; b=BxWHpTu3ZLBhUGlxa/PO/R/e2sKtMlQOypxchNvJVMpD9wl38gYcT2IR5NsTJL8IX3 5yLoI9hvnAu2hueLRL9xOPc30zMorEZ4Ax24HSru0WAx5KGr8fKHI3TKqnD9kICn2CYV B2vmVUT4l0RINduZIj8CZLVA26xzpXKpfOjm28uC2+o5aqtpfW9WIHyxwmKBp37FxNbX B1PfhY9+o3pi3QyTNVJiXG/Jriycs5vyE/ZW6rEJ1TnyPFQJgliDQFSGUFx35eOpe4I/ c1czYVuMkb5f/Rm9Rf3CAzqajSwI7pGqP/Axo9zbYzaZrFrjkNXO6KIw6ndAf7pI5Faq TGzg== X-Forwarded-Encrypted: i=1; AJvYcCVNrKxGz/RcDExgmbpNkExSSv/UpP8Op+cL4QkS27k7CKS6xelA9+etJEB5dEWXvWyds+8fl7vcPTxxNcc=@vger.kernel.org X-Gm-Message-State: AOJu0YzzmcCZjFWbAjhR1dYijyACkOXaQ7yT5001pNczlCFKzS05uLpd RsIWk0uAR0JLDxejT8nUp2Ig6Ko6iPjyKjPOYTa9ch7BzS7RPRSzGx4nVJdZYQ== X-Gm-Gg: ASbGncuA6TNXwRk5qsYIHWbaHwPvZMbRWgu16IxLzCIofq3ErC2S4sf/tuu8kEe8Thc V6NWJ6I52tZ/Veqgg4ZbE+O18T3OKSi79nTpwkUU5NHZueTWEwTSa7wpKBsuCz4M/0yxDQ4dxxp qARUn4hUS2jv9b0+3w2X/16dVcOzeODDMRGXfw2uxK300il7PhOQp0/WssBDBqbPxEAez+urZiV JgCReOzybKKFPZYpM1obAAwREYXh5dNkrMusnDnaGS1qcwR8hDx6G3kHaZIHpAt7iX8nk2Z05jv T9xRgMYJZrPyJYSYKfD5GWwR/GymChDGEuZf9EHuFO0/ovc= X-Google-Smtp-Source: AGHT+IEzPpFTdNA4qdxJB0dxJj4fKFXJpJ2zIhvE5aIYLQNZyq296Wz9RmH9FDhwIK3/6h7XuPIjIQ== X-Received: by 2002:a17:90b:3884:b0:2fa:2252:f436 with SMTP id 98e67ed59e1d1-2feba5ced0cmr17673056a91.3.1740968680936; Sun, 02 Mar 2025 18:24:40 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:1513:4f61:a4d3:b418]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-223501d2610sm66715445ad.14.2025.03.02.18.24.38 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sun, 02 Mar 2025 18:24:40 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Hillf Danton , Kairui Song , Sebastian Andrzej Siewior , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v10 01/19] zram: sleepable entry locking Date: Mon, 3 Mar 2025 11:03:10 +0900 Message-ID: <20250303022425.285971-2-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.711.g2feabab25a-goog In-Reply-To: <20250303022425.285971-1-senozhatsky@chromium.org> References: <20250303022425.285971-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Concurrent modifications of meta table entries is now handled by per-entry spin-lock. This has a number of shortcomings. First, this imposes atomic requirements on compression backends. zram can call both zcomp_compress() and zcomp_decompress() under entry spin-lock, which implies that we can use only compression algorithms that don't schedule/sleep/wait during compression and decompression. This, for instance, makes it impossible to use some of the ASYNC compression algorithms (H/W compression, etc.) implementations. Second, this can potentially trigger watchdogs. For example, entry re-compression with secondary algorithms is performed under entry spin-lock. Given that we chain secondary compression algorithms and that some of them can be configured for best compression ratio (and worst compression speed) zram can stay under spin-lock for quite some time. Having a per-entry mutex (or, for instance, a rw-semaphore) significantly increases sizeof() of each entry and hence the meta table. Therefore entry locking returns back to bit locking, as before, however, this time also preempt-rt friendly, because if waits-on-bit instead of spinning-on-bit. Lock owners are also now permitted to schedule, which is a first step on the path of making zram non-atomic. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 54 ++++++++++++++++++++++++++++------- drivers/block/zram/zram_drv.h | 15 ++++++---- 2 files changed, 52 insertions(+), 17 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 9f5020b077c5..70599d41b828 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -58,19 +58,56 @@ static void zram_free_page(struct zram *zram, size_t in= dex); static int zram_read_from_zspool(struct zram *zram, struct page *page, u32 index); =20 -static int zram_slot_trylock(struct zram *zram, u32 index) +#define slot_dep_map(zram, index) (&(zram)->table[(index)].dep_map) + +static void zram_slot_lock_init(struct zram *zram, u32 index) +{ + static struct lock_class_key __key; + + lockdep_init_map(slot_dep_map(zram, index), "zram->table[index].lock", + &__key, 0); +} + +/* + * entry locking rules: + * + * 1) Lock is exclusive + * + * 2) lock() function can sleep waiting for the lock + * + * 3) Lock owner can sleep + * + * 4) Use TRY lock variant when in atomic context + * - must check return value and handle locking failers + */ +static __must_check bool zram_slot_trylock(struct zram *zram, u32 index) { - return spin_trylock(&zram->table[index].lock); + unsigned long *lock =3D &zram->table[index].flags; + + if (!test_and_set_bit_lock(ZRAM_ENTRY_LOCK, lock)) { + mutex_acquire(slot_dep_map(zram, index), 0, 1, _RET_IP_); + lock_acquired(slot_dep_map(zram, index), _RET_IP_); + return true; + } + + return false; } =20 static void zram_slot_lock(struct zram *zram, u32 index) { - spin_lock(&zram->table[index].lock); + unsigned long *lock =3D &zram->table[index].flags; + + mutex_acquire(slot_dep_map(zram, index), 0, 0, _RET_IP_); + wait_on_bit_lock(lock, ZRAM_ENTRY_LOCK, TASK_UNINTERRUPTIBLE); + lock_acquired(slot_dep_map(zram, index), _RET_IP_); } =20 static void zram_slot_unlock(struct zram *zram, u32 index) { - spin_unlock(&zram->table[index].lock); + unsigned long *lock =3D &zram->table[index].flags; + + mutex_release(slot_dep_map(zram, index), _RET_IP_); + clear_and_wake_up_bit(ZRAM_ENTRY_LOCK, lock); } =20 static inline bool init_done(struct zram *zram) @@ -93,7 +130,6 @@ static void zram_set_handle(struct zram *zram, u32 index= , unsigned long handle) zram->table[index].handle =3D handle; } =20 -/* flag operations require table entry bit_spin_lock() being held */ static bool zram_test_flag(struct zram *zram, u32 index, enum zram_pageflags flag) { @@ -1473,15 +1509,11 @@ static bool zram_meta_alloc(struct zram *zram, u64 = disksize) huge_class_size =3D zs_huge_class_size(zram->mem_pool); =20 for (index =3D 0; index < num_pages; index++) - spin_lock_init(&zram->table[index].lock); + zram_slot_lock_init(zram, index); + return true; } =20 -/* - * To protect concurrent access to the same index entry, - * caller should hold this table index entry's bit_spinlock to - * indicate this index entry is accessing. - */ static void zram_free_page(struct zram *zram, size_t index) { unsigned long handle; diff --git a/drivers/block/zram/zram_drv.h b/drivers/block/zram/zram_drv.h index db78d7c01b9a..c804f78a7fa8 100644 --- a/drivers/block/zram/zram_drv.h +++ b/drivers/block/zram/zram_drv.h @@ -28,7 +28,6 @@ #define ZRAM_SECTOR_PER_LOGICAL_BLOCK \ (1 << (ZRAM_LOGICAL_BLOCK_SHIFT - SECTOR_SHIFT)) =20 - /* * ZRAM is mainly used for memory efficiency so we want to keep memory * footprint small and thus squeeze size and zram pageflags into a flags @@ -46,6 +45,7 @@ /* Flags for zram pages (table[page_no].flags) */ enum zram_pageflags { ZRAM_SAME =3D ZRAM_FLAG_SHIFT, /* Page consists the same element */ + ZRAM_ENTRY_LOCK, /* entry access lock bit */ ZRAM_WB, /* page is stored on backing_device */ ZRAM_PP_SLOT, /* Selected for post-processing */ ZRAM_HUGE, /* Incompressible page */ @@ -58,16 +58,19 @@ enum zram_pageflags { __NR_ZRAM_PAGEFLAGS, }; =20 -/*-- Data structures */ - -/* Allocated for each disk page */ +/* + * Allocated for each disk page. We use bit-lock (ZRAM_ENTRY_LOCK bit + * of flags) to save memory. There can be plenty of entries and standard + * locking primitives (e.g. mutex) will significantly increase sizeof() + * of each entry and hence of the meta table. + */ struct zram_table_entry { unsigned long handle; - unsigned int flags; - spinlock_t lock; + unsigned long flags; #ifdef CONFIG_ZRAM_TRACK_ENTRY_ACTIME ktime_t ac_time; #endif + struct lockdep_map dep_map; }; =20 struct zram_stats { --=20 2.48.1.711.g2feabab25a-goog From nobody Tue Dec 16 19:24:32 2025 Received: from mail-pl1-f175.google.com (mail-pl1-f175.google.com [209.85.214.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 49F3918A6C4 for ; Mon, 3 Mar 2025 02:24:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.175 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740968687; cv=none; b=g6GXZVo3HkUAQ6uxoi3X65qyeFb3g9YiIV234cc5TKCNIi5tnLSky3iyH9iCBi396I3IXrYxaDRpXebgwiX1XGY2sH72j18AtV+qjkgbJk8s6g6AjUTEbZz0JosOScnBAdhWIVf5VFWu4o+Yn38nbTsEXhBuRpdVurzmaYB4USM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740968687; c=relaxed/simple; bh=vsGOSwouO8U4URKs20bupIG27NGRE2VX5d3kClMGF88=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=tDyd5qPBbgK7vyIEy+lGtIpulI0DjUweg4SUq4fC3A1sapsoHN8+1Gu4wn8BvYzhRvDPG4vB+Z39vMYpADoroEbpkjdnMGbahT7IsC3rjzUQJQW8KrO7JY45orwgqtO/L1D3ThvipyonmUxMZihzraPyKALb0g63rkvrrR7Mvl8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=Ln8MM9VO; arc=none smtp.client-ip=209.85.214.175 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="Ln8MM9VO" Received: by mail-pl1-f175.google.com with SMTP id d9443c01a7336-2235189adaeso58833455ad.0 for ; Sun, 02 Mar 2025 18:24:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1740968685; x=1741573485; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=jDqZrH9W9+qGmX1OXU2Vyl3xsabTv9L/+SLAveE4fcw=; b=Ln8MM9VO0B43F3BMxnTUKkBTRBu8UXscN14HqzUPFNPUfMCmLpL8K08LKO7cvt84mZ glDoTK3S1BoCjIRU9mRaCyOwMggbsfnNK4K8oYHCDzD6sYWG6mUOciqMP2GTtyb98wHF 0uNP654qRi8tiJzMlfr1Nc0OChArLuCvyhSPQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740968685; x=1741573485; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=jDqZrH9W9+qGmX1OXU2Vyl3xsabTv9L/+SLAveE4fcw=; b=A1mnIqxFRfsMHLUmQdOl92xzU6cOnxW3B20b2m9XUos/QrCRHIT3XJhkwoTAYT/FY/ 5OqbuSR0pX1Sj4XOG14mi6ih0KFxamW1tj7cGCdQ+Yn5aSXJ53A6RmqnSwiL1GuPHwY9 Cv2rFKz1hq6UGznLjrUxsEwfhYorLuJzTWmrndvuq0h1agPkXRZFVldyuF+6yNsWpyas WcT+B5LA6BZXSQeDD0RTvU06+T5cZOk7TSMDTOrDIkHXosGK4gMFPvLnSPtNll2BVR2E EQF6Tj3RNlJU7sh24tKP6V8oCG/cgjB1g4Srf1/NJhXNLY8LH/XOhzdJZgn9urFK4VGU CLwA== X-Forwarded-Encrypted: i=1; AJvYcCXrrC+NJZHN8kxpCVqhgm3LsJDPJRd5GlRUToicrUWvY6BaY9JtrVBEc1LE3oDXv7WkrbhIm1vgMjny9Nw=@vger.kernel.org X-Gm-Message-State: AOJu0YzPpXlrtC+yXgYXSqf6u3uiUUwH3NpRterpsS5JwKaMo/jtYV9R YLTDa86jhs8tYv/JJgHHMEjubRrgmSoPpbI6bbeLJMjQ372rGpdKKxx0A8IutA== X-Gm-Gg: ASbGnctzd4sXZA9YES4hGhhFiCBEY5zUzgpfeVC/WEDrpjvbFl+73IJVYeov1ewgJyQ OWfyKT82EDUwkgh7kYDrffnpeiVhJZTS+fLuslU9LcHiG9eSIouCJV6EhV+0g+ynWdaXiRVJ7nN n9RSe+BpPDdehc3nazi67BDuFoXu9X+32qM0tsJ3xcLJ+Dc0ZRl3+X1xWq6JuFyMZw49ocCkqc9 UEkIW36gmtwqvKmpg8WoapV1n7EcI8Qt9qZw83/GGi2P9H0jqvcwrPSONxlD7yhXKjvAKB9IqV3 jKFTmFMUKHt61h98oe3UI/ryi4RJTMJHLTE0FJUPNxFTA/k= X-Google-Smtp-Source: AGHT+IGEtduyJTi51HPkBHttjdqcJfHiW6RA1OErSt85BYtPP72gIAiy5XzU4m9bzS3f4VCOBOOEpw== X-Received: by 2002:a17:903:2986:b0:21f:6546:9adc with SMTP id d9443c01a7336-2234a38cd53mr253511975ad.13.1740968685435; Sun, 02 Mar 2025 18:24:45 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:1513:4f61:a4d3:b418]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-223501d5247sm66569735ad.26.2025.03.02.18.24.43 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sun, 02 Mar 2025 18:24:45 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Hillf Danton , Kairui Song , Sebastian Andrzej Siewior , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v10 02/19] zram: permit preemption with active compression stream Date: Mon, 3 Mar 2025 11:03:11 +0900 Message-ID: <20250303022425.285971-3-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.711.g2feabab25a-goog In-Reply-To: <20250303022425.285971-1-senozhatsky@chromium.org> References: <20250303022425.285971-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Currently, per-CPU stream access is done from a non-preemptible (atomic) section, which imposes the same atomicity requirements on compression backends as entry spin-lock, and makes it impossible to use algorithms that can schedule/wait/sleep during compression and decompression. Switch to preemptible per-CPU model, similar to the one used in zswap. Instead of a per-CPU local lock, each stream carries a mutex which is locked throughout entire time zram uses it for compression or decompression, so that cpu-dead event waits for zram to stop using a particular per-CPU stream and release it. Suggested-by: Yosry Ahmed Signed-off-by: Sergey Senozhatsky Reviewed-by: Yosry Ahmed --- drivers/block/zram/zcomp.c | 41 +++++++++++++++++++++++++---------- drivers/block/zram/zcomp.h | 6 ++--- drivers/block/zram/zram_drv.c | 20 ++++++++--------- 3 files changed, 42 insertions(+), 25 deletions(-) diff --git a/drivers/block/zram/zcomp.c b/drivers/block/zram/zcomp.c index bb514403e305..53e4c37441be 100644 --- a/drivers/block/zram/zcomp.c +++ b/drivers/block/zram/zcomp.c @@ -6,7 +6,7 @@ #include #include #include -#include +#include #include #include =20 @@ -109,13 +109,29 @@ ssize_t zcomp_available_show(const char *comp, char *= buf) =20 struct zcomp_strm *zcomp_stream_get(struct zcomp *comp) { - local_lock(&comp->stream->lock); - return this_cpu_ptr(comp->stream); + for (;;) { + struct zcomp_strm *zstrm =3D raw_cpu_ptr(comp->stream); + + /* + * Inspired by zswap + * + * stream is returned with ->mutex locked which prevents + * cpu_dead() from releasing this stream under us, however + * there is still a race window between raw_cpu_ptr() and + * mutex_lock(), during which we could have been migrated + * from a CPU that has already destroyed its stream. If + * so then unlock and re-try on the current CPU. + */ + mutex_lock(&zstrm->lock); + if (likely(zstrm->buffer)) + return zstrm; + mutex_unlock(&zstrm->lock); + } } =20 -void zcomp_stream_put(struct zcomp *comp) +void zcomp_stream_put(struct zcomp_strm *zstrm) { - local_unlock(&comp->stream->lock); + mutex_unlock(&zstrm->lock); } =20 int zcomp_compress(struct zcomp *comp, struct zcomp_strm *zstrm, @@ -151,12 +167,9 @@ int zcomp_decompress(struct zcomp *comp, struct zcomp_= strm *zstrm, int zcomp_cpu_up_prepare(unsigned int cpu, struct hlist_node *node) { struct zcomp *comp =3D hlist_entry(node, struct zcomp, node); - struct zcomp_strm *zstrm; + struct zcomp_strm *zstrm =3D per_cpu_ptr(comp->stream, cpu); int ret; =20 - zstrm =3D per_cpu_ptr(comp->stream, cpu); - local_lock_init(&zstrm->lock); - ret =3D zcomp_strm_init(comp, zstrm); if (ret) pr_err("Can't allocate a compression stream\n"); @@ -166,16 +179,17 @@ int zcomp_cpu_up_prepare(unsigned int cpu, struct hli= st_node *node) int zcomp_cpu_dead(unsigned int cpu, struct hlist_node *node) { struct zcomp *comp =3D hlist_entry(node, struct zcomp, node); - struct zcomp_strm *zstrm; + struct zcomp_strm *zstrm =3D per_cpu_ptr(comp->stream, cpu); =20 - zstrm =3D per_cpu_ptr(comp->stream, cpu); + mutex_lock(&zstrm->lock); zcomp_strm_free(comp, zstrm); + mutex_unlock(&zstrm->lock); return 0; } =20 static int zcomp_init(struct zcomp *comp, struct zcomp_params *params) { - int ret; + int ret, cpu; =20 comp->stream =3D alloc_percpu(struct zcomp_strm); if (!comp->stream) @@ -186,6 +200,9 @@ static int zcomp_init(struct zcomp *comp, struct zcomp_= params *params) if (ret) goto cleanup; =20 + for_each_possible_cpu(cpu) + mutex_init(&per_cpu_ptr(comp->stream, cpu)->lock); + ret =3D cpuhp_state_add_instance(CPUHP_ZCOMP_PREPARE, &comp->node); if (ret < 0) goto cleanup; diff --git a/drivers/block/zram/zcomp.h b/drivers/block/zram/zcomp.h index ad5762813842..23b8236b9090 100644 --- a/drivers/block/zram/zcomp.h +++ b/drivers/block/zram/zcomp.h @@ -3,7 +3,7 @@ #ifndef _ZCOMP_H_ #define _ZCOMP_H_ =20 -#include +#include =20 #define ZCOMP_PARAM_NO_LEVEL INT_MIN =20 @@ -31,7 +31,7 @@ struct zcomp_ctx { }; =20 struct zcomp_strm { - local_lock_t lock; + struct mutex lock; /* compression buffer */ void *buffer; struct zcomp_ctx ctx; @@ -77,7 +77,7 @@ struct zcomp *zcomp_create(const char *alg, struct zcomp_= params *params); void zcomp_destroy(struct zcomp *comp); =20 struct zcomp_strm *zcomp_stream_get(struct zcomp *comp); -void zcomp_stream_put(struct zcomp *comp); +void zcomp_stream_put(struct zcomp_strm *zstrm); =20 int zcomp_compress(struct zcomp *comp, struct zcomp_strm *zstrm, const void *src, unsigned int *dst_len); diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 70599d41b828..dd669d48ae6f 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1607,7 +1607,7 @@ static int read_compressed_page(struct zram *zram, st= ruct page *page, u32 index) ret =3D zcomp_decompress(zram->comps[prio], zstrm, src, size, dst); kunmap_local(dst); zs_unmap_object(zram->mem_pool, handle); - zcomp_stream_put(zram->comps[prio]); + zcomp_stream_put(zstrm); =20 return ret; } @@ -1768,14 +1768,14 @@ static int zram_write_page(struct zram *zram, struc= t page *page, u32 index) kunmap_local(mem); =20 if (unlikely(ret)) { - zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); + zcomp_stream_put(zstrm); pr_err("Compression failed! err=3D%d\n", ret); zs_free(zram->mem_pool, handle); return ret; } =20 if (comp_len >=3D huge_class_size) { - zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); + zcomp_stream_put(zstrm); return write_incompressible_page(zram, page, index); } =20 @@ -1799,7 +1799,7 @@ static int zram_write_page(struct zram *zram, struct = page *page, u32 index) __GFP_HIGHMEM | __GFP_MOVABLE); if (IS_ERR_VALUE(handle)) { - zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); + zcomp_stream_put(zstrm); atomic64_inc(&zram->stats.writestall); handle =3D zs_malloc(zram->mem_pool, comp_len, GFP_NOIO | __GFP_HIGHMEM | @@ -1811,7 +1811,7 @@ static int zram_write_page(struct zram *zram, struct = page *page, u32 index) } =20 if (!zram_can_store_page(zram)) { - zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); + zcomp_stream_put(zstrm); zs_free(zram->mem_pool, handle); return -ENOMEM; } @@ -1819,7 +1819,7 @@ static int zram_write_page(struct zram *zram, struct = page *page, u32 index) dst =3D zs_map_object(zram->mem_pool, handle, ZS_MM_WO); =20 memcpy(dst, zstrm->buffer, comp_len); - zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); + zcomp_stream_put(zstrm); zs_unmap_object(zram->mem_pool, handle); =20 zram_slot_lock(zram, index); @@ -1978,7 +1978,7 @@ static int recompress_slot(struct zram *zram, u32 ind= ex, struct page *page, kunmap_local(src); =20 if (ret) { - zcomp_stream_put(zram->comps[prio]); + zcomp_stream_put(zstrm); return ret; } =20 @@ -1988,7 +1988,7 @@ static int recompress_slot(struct zram *zram, u32 ind= ex, struct page *page, /* Continue until we make progress */ if (class_index_new >=3D class_index_old || (threshold && comp_len_new >=3D threshold)) { - zcomp_stream_put(zram->comps[prio]); + zcomp_stream_put(zstrm); continue; } =20 @@ -2046,13 +2046,13 @@ static int recompress_slot(struct zram *zram, u32 i= ndex, struct page *page, __GFP_HIGHMEM | __GFP_MOVABLE); if (IS_ERR_VALUE(handle_new)) { - zcomp_stream_put(zram->comps[prio]); + zcomp_stream_put(zstrm); return PTR_ERR((void *)handle_new); } =20 dst =3D zs_map_object(zram->mem_pool, handle_new, ZS_MM_WO); memcpy(dst, zstrm->buffer, comp_len_new); - zcomp_stream_put(zram->comps[prio]); + zcomp_stream_put(zstrm); =20 zs_unmap_object(zram->mem_pool, handle_new); =20 --=20 2.48.1.711.g2feabab25a-goog From nobody Tue Dec 16 19:24:32 2025 Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 91A97198E76 for ; Mon, 3 Mar 2025 02:24:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740968692; cv=none; b=PECrNFc8yXanOb4OVe00E3GcSL5XpegBdamBb8m2YgmKBWQgKcqtf2fkfhGgf+Yh7Dt1RkKr8lmgpjRNpGPILfmMFNxdakMIrYdsf0FOKqPLF1fstsHItkGlsiR8AAoBZM/1wrh6r+HsKb32BUBipsbpyH8eWPdYDV4jTmEcMxg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740968692; c=relaxed/simple; bh=6cNSwFN+g9dAhAE6jUN8x7pSCR8FfvK/1Zq7Fjm30rQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=f40UZvPq+CsyTm9NoRWy3ty0RoY+VMdXs8fp4rsg/ZL7GdmQZHQ9VRgbRI8bH0W83y+dv1L5WHvzXNc3uFsBKa40m3b+Cghpzkm4vdJ7U03QSd4hKjHMpxlG9oPm53usmiqRsybZ28FQ2WPncoaJY+GMcpjsvzum5TZ9GKK112k= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=RB6R/qRL; arc=none smtp.client-ip=209.85.214.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="RB6R/qRL" Received: by mail-pl1-f171.google.com with SMTP id d9443c01a7336-2234e5347e2so76906755ad.1 for ; Sun, 02 Mar 2025 18:24:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1740968690; x=1741573490; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=pZsh5cPaFoEDPyEJH9aQiY1L0urDRF/CUPRrOejpG5k=; b=RB6R/qRL5oT6d4GQQvuMi13rGEfhO0u7EebOnwVxoDPxHZiVetiDPY2Rn3P7sK3ydQ uGfjWPFB6JIJOAoBkInoj2BogxvunzfEt/hBtzbARJarpLbSDoiQoM23uLOAJ49dmenT dX+p1qj+uqITJ5A7kQ0hOL1RsDZHg/WqG9+cM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740968690; x=1741573490; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=pZsh5cPaFoEDPyEJH9aQiY1L0urDRF/CUPRrOejpG5k=; b=l4oNt9Tn5NznV1UOUGgwIKWJ5efC3mOI1zJ10eQPCm9EL3xSMk4jBEDjDU0JbR2glW ck+f7AsQZ65Z4Pyw415F4xVEgOW+7LSK1A+QbHMOYRTbaPOUnDOoMHkf416qhWycgGNu XXOIysi80Bg6VDeyt8ZfqYQaelJIdh32vXpQf7ewjOXdWFMRgeEMIg8eiyS40spCCBhy qAG23uaVd108qO5n9xqFd3ikGP/2UiNLy3MQU+222FBTlfRmeGrHaZrUtM9sJWZG9QmN MNesXKE+L7c+5g34J0pYUQ0lwPUso1UbEfxK8D6C7DboDTHCEWXp5YVJKmcIMH6XhxJS wEQA== X-Forwarded-Encrypted: i=1; AJvYcCW1yxTt6b00tcvgwE6xW6QnWAWboH8CcqUonMrYrCn+/FNKqEE+JutJmYIuGuhpKImNmyb7MvJ4kK4zhZU=@vger.kernel.org X-Gm-Message-State: AOJu0Yx5v7wELS6kFZEt2g7Vawq4+EiCtEQYL4ywKH+UoWexOAy/bznJ O9GB0cUYwCW82bChtO2ws9a5LlYewaGliDM/OzuVIG7CiMO6D1pzWL2kqLs/gw== X-Gm-Gg: ASbGncvLou2M2UjSsu66jaZwel6W6zTRbkyox8uuckeg3vdscyjmqR5mwPi06zRAZtm fmFo4ADMw1x46ezAZTTw0tA2NH/1lELvICJou3EVUYLuGLcWn5+/uEgdJ3yZUhV2bfzRi1QfNHw nGQ/TiMzE5wfchscUTLv1NoxQd581CDDL1exuxbGzucAdw535JS6p5d8sLYFu+2Sy7Q0JEgOLOI rD/H/qAoCvAj4t0YQ+Zz9DbLbBrXDCyb5mgbwktigAbVByI7DZdrRijYWY2NMCh3NOfX2V9A32F Lw+iDmHLBeucoq0sUd8Jz8wbdIMc0vPYGVaLbr9T+NjMh0k= X-Google-Smtp-Source: AGHT+IFRsspYWzVXCyDoL5ZFIh1SjDZmc0euiZ+Gem/pRrrBGftzVfrbZvTQsBlZb+pBOhlTra4AuQ== X-Received: by 2002:a05:6a20:7289:b0:1ee:6ec3:e82e with SMTP id adf61e73a8af0-1f2f4e3cc66mr16110519637.29.1740968689829; Sun, 02 Mar 2025 18:24:49 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:1513:4f61:a4d3:b418]) by smtp.gmail.com with UTF8SMTPSA id 41be03b00d2f7-af221346042sm3142765a12.28.2025.03.02.18.24.47 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sun, 02 Mar 2025 18:24:49 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Hillf Danton , Kairui Song , Sebastian Andrzej Siewior , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v10 03/19] zram: remove unused crypto include Date: Mon, 3 Mar 2025 11:03:12 +0900 Message-ID: <20250303022425.285971-4-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.711.g2feabab25a-goog In-Reply-To: <20250303022425.285971-1-senozhatsky@chromium.org> References: <20250303022425.285971-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" We stopped using crypto API (for the time being), so remove its include and replace CRYPTO_MAX_ALG_NAME with a local define. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zcomp.c | 1 - drivers/block/zram/zram_drv.c | 4 +++- drivers/block/zram/zram_drv.h | 1 - 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/drivers/block/zram/zcomp.c b/drivers/block/zram/zcomp.c index 53e4c37441be..cfdde2e0748a 100644 --- a/drivers/block/zram/zcomp.c +++ b/drivers/block/zram/zcomp.c @@ -7,7 +7,6 @@ #include #include #include -#include #include =20 #include "zcomp.h" diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index dd669d48ae6f..248dab7cc7f4 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -44,6 +44,8 @@ static DEFINE_MUTEX(zram_index_mutex); static int zram_major; static const char *default_compressor =3D CONFIG_ZRAM_DEF_COMP; =20 +#define ZRAM_MAX_ALGO_NAME_SZ 128 + /* Module params (documentation at end) */ static unsigned int num_devices =3D 1; /* @@ -1148,7 +1150,7 @@ static int __comp_algorithm_store(struct zram *zram, = u32 prio, const char *buf) size_t sz; =20 sz =3D strlen(buf); - if (sz >=3D CRYPTO_MAX_ALG_NAME) + if (sz >=3D ZRAM_MAX_ALGO_NAME_SZ) return -E2BIG; =20 compressor =3D kstrdup(buf, GFP_KERNEL); diff --git a/drivers/block/zram/zram_drv.h b/drivers/block/zram/zram_drv.h index c804f78a7fa8..7c11f9dab335 100644 --- a/drivers/block/zram/zram_drv.h +++ b/drivers/block/zram/zram_drv.h @@ -17,7 +17,6 @@ =20 #include #include -#include =20 #include "zcomp.h" =20 --=20 2.48.1.711.g2feabab25a-goog From nobody Tue Dec 16 19:24:32 2025 Received: from mail-pl1-f173.google.com (mail-pl1-f173.google.com [209.85.214.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 361DE1C84BF for ; Mon, 3 Mar 2025 02:24:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740968696; cv=none; b=VnElUjRkggRs7gS1OTzh1cFpeTRn3eNxg0ahfNyKmOrP8n5PKtqH6OUm7CEX3eqqts8rriIDZVjznA43vv4SYjTuGsRadLHdwjqiDpNdJCajeJ8ifB8ntWG7GfErHIQOCwHnGmw13FQjbaiBtPtJie4YMiKOTURRI2agUxQkjFI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740968696; c=relaxed/simple; bh=/pWWX6OOpsdvhmJgrRvgon6ifyWCV3iEUzc87eJfQsA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=h8Qau58zdTKhCehRdC9w1ZKfQSIS4W+hxss+nwWfnS1+k2GS2vo4nb7g4urh/vCZYGpm697iDaK1+bnYnK01zx92LET0zO9lDBe1kdjHslaovM0rxRZEiZUH8f4bCwkhn+SgZYdJgE8t6VPScM9BQgZ3cb5z9MBXBVaCFHBzDaI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=ZqEemLxz; arc=none smtp.client-ip=209.85.214.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="ZqEemLxz" Received: by mail-pl1-f173.google.com with SMTP id d9443c01a7336-2239f8646f6so14277205ad.2 for ; Sun, 02 Mar 2025 18:24:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1740968694; x=1741573494; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=bWS/5MPcWfCH0A9JfQHKT/0nLElRz817vLkO7NmScms=; b=ZqEemLxzj6A7Dj/OlyL3C7uxpMVOXWTXikKrysZwssgvBb1Lpbo3qscAPLRPsZ7iyt XqqCP73Er8VF9H1gya12V9HsenhMTL5tgcIGA9zV8XRnxoQIYAVq8f90BwiSILEeKXnO 6FF++Udnp5leFazhUu/kxQivvGvUIuQvB2FS0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740968694; x=1741573494; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=bWS/5MPcWfCH0A9JfQHKT/0nLElRz817vLkO7NmScms=; b=nXoZM5x6sDsFQjvaHk5S1HTcEHA3JmDrzi14sVR9AB6NOtuoAt1Y5mai1WyupSFm9E RlxzFoemU29mx3jZSPZQW5SVKfXCxXPI/AtYw42jA4TvH3tldS4demlxNcUy4eEBharc vFjxKDsqfwZ1xN4PEiOK1ebFkryLrPrDV/L5LWWCpf+H2BtgYy10MEIcaSZemTDGOSis Os5qXxKfPzcAomNv2wgYiBBwX3/w4SXmiTnxxWX7axR2+yS9EpMino1Y/bJN7fQb5M6I VeoIMSDyvjSYi/GiEkOBQkQvb4VNdxJam0VQPpL+hlRMvkc+6z5oCeKmhPNltvPpHZmW kB9Q== X-Forwarded-Encrypted: i=1; AJvYcCU6MJ3D3+jRRiqd6iywxksoyjHXlAiF/+7GeW168UoWMDMZwjwQh0xc/qDgtlkrpA+l0RqYISvFBfCAVb8=@vger.kernel.org X-Gm-Message-State: AOJu0YxOrLOZPJzD+LqXOlmkCrt1Wr2GHgveovhbPaJsAl2X7weHxeDY IOqcy1JShAbWBUy5H1PX+QrubcVsAaV7qOKFz24uXXBpRIlht2GuKeSdAfvTdw== X-Gm-Gg: ASbGnctwN6X9KgBhRTAdRQaERTYt72EYnFqdx+jYY5g98At5d1Ml5hF1+rtOFA4eg1q GwefiwYFS3RisUGNUOLdg6JfWxITchH1SKGZhcX5j8uPlxaY7/Jo4CVLBgpvTiQfJAGeuT7EEa4 SxFOy47hZaomE/o9HprMTorRKuPKJNDqjCOoBeFrPkuwESYEtTIkmqHk0KQ1KaVoYuUyd1mU4Ty B3erjdBH8LFwrQgNvZoTDceuAyMc5accg3YA5ZLOFDPTIWsvkoHfb61jzhQHucswp/kqlbrijVk NJwnDH6i8iOfsrENMUPLgPot7iT1meyN//m2gLglnlv3M5I= X-Google-Smtp-Source: AGHT+IHSDPg6+GlHfcOXSQ76GatOFpUQPAH1lRliAnvxfV2K3YTVpkBHpg9ehwbe5zk/2PLiRfIsOg== X-Received: by 2002:a17:902:e807:b0:223:6657:5003 with SMTP id d9443c01a7336-223691f8f1cmr201725005ad.32.1740968694360; Sun, 02 Mar 2025 18:24:54 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:1513:4f61:a4d3:b418]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-223504c5cc6sm67129725ad.112.2025.03.02.18.24.52 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sun, 02 Mar 2025 18:24:54 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Hillf Danton , Kairui Song , Sebastian Andrzej Siewior , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v10 04/19] zram: remove max_comp_streams device attr Date: Mon, 3 Mar 2025 11:03:13 +0900 Message-ID: <20250303022425.285971-5-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.711.g2feabab25a-goog In-Reply-To: <20250303022425.285971-1-senozhatsky@chromium.org> References: <20250303022425.285971-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" max_comp_streams device attribute has been defunct since May 2016 when zram switched to per-CPU compression streams, remove it. Signed-off-by: Sergey Senozhatsky --- Documentation/ABI/testing/sysfs-block-zram | 8 ----- Documentation/admin-guide/blockdev/zram.rst | 36 ++++++--------------- drivers/block/zram/zram_drv.c | 23 ------------- 3 files changed, 10 insertions(+), 57 deletions(-) diff --git a/Documentation/ABI/testing/sysfs-block-zram b/Documentation/ABI= /testing/sysfs-block-zram index 1ef69e0271f9..36c57de0a10a 100644 --- a/Documentation/ABI/testing/sysfs-block-zram +++ b/Documentation/ABI/testing/sysfs-block-zram @@ -22,14 +22,6 @@ Description: device. The reset operation frees all the memory associated with this device. =20 -What: /sys/block/zram/max_comp_streams -Date: February 2014 -Contact: Sergey Senozhatsky -Description: - The max_comp_streams file is read-write and specifies the - number of backend's zcomp_strm compression streams (number of - concurrent compress operations). - What: /sys/block/zram/comp_algorithm Date: February 2014 Contact: Sergey Senozhatsky diff --git a/Documentation/admin-guide/blockdev/zram.rst b/Documentation/ad= min-guide/blockdev/zram.rst index 714a5171bfc0..7ad4c86f8258 100644 --- a/Documentation/admin-guide/blockdev/zram.rst +++ b/Documentation/admin-guide/blockdev/zram.rst @@ -54,7 +54,7 @@ The list of possible return codes: If you use 'echo', the returned value is set by the 'echo' utility, and, in general case, something like:: =20 - echo 3 > /sys/block/zram0/max_comp_streams + echo foo > /sys/block/zram0/comp_algorithm if [ $? -ne 0 ]; then handle_error fi @@ -73,21 +73,7 @@ This creates 4 devices: /dev/zram{0,1,2,3} num_devices parameter is optional and tells zram how many devices should be pre-created. Default: 1. =20 -2) Set max number of compression streams -=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D - -Regardless of the value passed to this attribute, ZRAM will always -allocate multiple compression streams - one per online CPU - thus -allowing several concurrent compression operations. The number of -allocated compression streams goes down when some of the CPUs -become offline. There is no single-compression-stream mode anymore, -unless you are running a UP system or have only 1 CPU online. - -To find out how many streams are currently available:: - - cat /sys/block/zram0/max_comp_streams - -3) Select compression algorithm +2) Select compression algorithm =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D =20 Using comp_algorithm device attribute one can see available and @@ -107,7 +93,7 @@ Examples:: For the time being, the `comp_algorithm` content shows only compression algorithms that are supported by zram. =20 -4) Set compression algorithm parameters: Optional +3) Set compression algorithm parameters: Optional =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =20 Compression algorithms may support specific parameters which can be @@ -138,7 +124,7 @@ better the compression ratio, it even can take negative= s values for some algorithms), for other algorithms `level` is acceleration level (the higher the value the lower the compression ratio). =20 -5) Set Disksize +4) Set Disksize =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =20 Set disk size by writing the value to sysfs node 'disksize'. @@ -158,7 +144,7 @@ There is little point creating a zram of greater than t= wice the size of memory since we expect a 2:1 compression ratio. Note that zram uses about 0.1% of= the size of the disk when not in use so a huge zram is wasteful. =20 -6) Set memory limit: Optional +5) Set memory limit: Optional =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D =20 Set memory limit by writing the value to sysfs node 'mem_limit'. @@ -177,7 +163,7 @@ Examples:: # To disable memory limit echo 0 > /sys/block/zram0/mem_limit =20 -7) Activate +6) Activate =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =20 :: @@ -188,7 +174,7 @@ Examples:: mkfs.ext4 /dev/zram1 mount /dev/zram1 /tmp =20 -8) Add/remove zram devices +7) Add/remove zram devices =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D =20 zram provides a control interface, which enables dynamic (on-demand) device @@ -208,7 +194,7 @@ execute:: =20 echo X > /sys/class/zram-control/hot_remove =20 -9) Stats +8) Stats =3D=3D=3D=3D=3D=3D=3D=3D =20 Per-device statistics are exported as various nodes under /sys/block/zram<= id>/ @@ -228,8 +214,6 @@ mem_limit WO specifies the maximum amount of m= emory ZRAM can writeback_limit WO specifies the maximum amount of write IO zram can write out to backing device as 4KB unit writeback_limit_enable RW show and set writeback_limit feature -max_comp_streams RW the number of possible concurrent compress - operations comp_algorithm RW show and change the compression algorithm algorithm_params WO setup compression algorithm parameters compact WO trigger memory compaction @@ -310,7 +294,7 @@ a single line of text and contains the following stats = separated by whitespace: Unit: 4K bytes =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D =20 -10) Deactivate +9) Deactivate =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =20 :: @@ -318,7 +302,7 @@ a single line of text and contains the following stats = separated by whitespace: swapoff /dev/zram0 umount /dev/zram1 =20 -11) Reset +10) Reset =3D=3D=3D=3D=3D=3D=3D=3D=3D =20 Write any positive value to 'reset' sysfs node:: diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 248dab7cc7f4..93cedc60ac16 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1103,27 +1103,6 @@ static void zram_debugfs_register(struct zram *zram)= {}; static void zram_debugfs_unregister(struct zram *zram) {}; #endif =20 -/* - * We switched to per-cpu streams and this attr is not needed anymore. - * However, we will keep it around for some time, because: - * a) we may revert per-cpu streams in the future - * b) it's visible to user space and we need to follow our 2 years - * retirement rule; but we already have a number of 'soon to be - * altered' attrs, so max_comp_streams need to wait for the next - * layoff cycle. - */ -static ssize_t max_comp_streams_show(struct device *dev, - struct device_attribute *attr, char *buf) -{ - return scnprintf(buf, PAGE_SIZE, "%d\n", num_online_cpus()); -} - -static ssize_t max_comp_streams_store(struct device *dev, - struct device_attribute *attr, const char *buf, size_t len) -{ - return len; -} - static void comp_algorithm_set(struct zram *zram, u32 prio, const char *al= g) { /* Do not free statically defined compression algorithms */ @@ -2540,7 +2519,6 @@ static DEVICE_ATTR_WO(reset); static DEVICE_ATTR_WO(mem_limit); static DEVICE_ATTR_WO(mem_used_max); static DEVICE_ATTR_WO(idle); -static DEVICE_ATTR_RW(max_comp_streams); static DEVICE_ATTR_RW(comp_algorithm); #ifdef CONFIG_ZRAM_WRITEBACK static DEVICE_ATTR_RW(backing_dev); @@ -2562,7 +2540,6 @@ static struct attribute *zram_disk_attrs[] =3D { &dev_attr_mem_limit.attr, &dev_attr_mem_used_max.attr, &dev_attr_idle.attr, - &dev_attr_max_comp_streams.attr, &dev_attr_comp_algorithm.attr, #ifdef CONFIG_ZRAM_WRITEBACK &dev_attr_backing_dev.attr, --=20 2.48.1.711.g2feabab25a-goog From nobody Tue Dec 16 19:24:32 2025 Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 811EE1CDA3F for ; Mon, 3 Mar 2025 02:24:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740968701; cv=none; b=KSzLqCBEXo3hZcc0GxCgiMQ83uHGADLq6eXPraAFXbMncyWoWSdaTuxFbWgNfJUmuesOfrWf7KShf1I/3htxcUS/9uXK6vOfF3bDkQL+/wKc4jvou3nOEBnKdeixbJMbyqvw8L3RrmEGjJJzFYGJwZa7Me2CvpSYmZn65ZR5dsQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740968701; c=relaxed/simple; bh=OOj45A09pa0t1zyHCgNjU5kknK7w1UAoEzwsLl3drpw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=S67MFtBBSf+Sm3K/wU99Km6WX5LHCZr1iMS+5881t+0k3ztkvKtRprT0puH/KcAqsoxGHWE6bBUMjJFN4ss2W8o1wW8JcO+K+oJcZe6S1Z+dH37I3BfIfhhJI7uVWcITW5rCdYsTGzB19D91hWFRuwzIrHxk1iCiV9A+Zq1bwro= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=QFeHJA69; arc=none smtp.client-ip=209.85.214.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="QFeHJA69" Received: by mail-pl1-f182.google.com with SMTP id d9443c01a7336-223959039f4so17316175ad.3 for ; Sun, 02 Mar 2025 18:24:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1740968699; x=1741573499; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=8qnIjXigJRpaB/eOXemhT39qobrTOIzhvUPCZvOaudo=; b=QFeHJA69EktCeOAb0jhzUEZq0IgEfASNeFglpnHMHzOtpDNomCUX9hJxXIkyuMASPs ypL4+geDLa8G/cT2XRHyCcDoK/+Z/w0Awb5Z52D7Sp6IXTF48xDBJeX/hG9hFyULdJTz 4LQXOUfSvhvJMMvt2039bWTRdJ9RtcOpmOoPU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740968699; x=1741573499; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8qnIjXigJRpaB/eOXemhT39qobrTOIzhvUPCZvOaudo=; b=YYthZx+wa968nV91JKiOJuB4RdFYJJnuWSnvgFUkKz/qcp0Emh3kMhLuFyB7vraEYB oRXPc6PWexoRQ3uwwXTMt6o2sN6Qj+uMgxszd0EqhDl0WGGrPqxPvZOprC9T9yQ6IoP6 bvo8e4MhKmIosiXGZF7xvrWvrPVltmJDqAbElft0tz0aW9yRtxgAoDp2yBMjzlsHyfoy IQHDIhMZzsdNWkCYwxYGVL6Lkq98IixSHcxbOCf7fufJrH+NZ+EWTfbEZ4D6inn9ak8w mIe0cAiN76VyvNqLOd3qexqGIXKe915Ow9phnicQ+FHS2EYYhw4xqTYGSPIP3jc4r4wB WNYQ== X-Forwarded-Encrypted: i=1; AJvYcCVC86yZ1qC2VfRtrzTxHuOTt51CW9nJk3M1PR6aMY84SOt/YsjVcJ9KDXVZDC8Gu6ata9FpcqHrxt2BTkI=@vger.kernel.org X-Gm-Message-State: AOJu0Yy1TQMz+W2rzNr5cNk/6ZuKhMTCiaBAPbmBavLo9pASQfP7z3CE Axhnq/WKLHAdbJDg43vz2AUaBKYTMrI7d4OtO703ggwfmt+HNGldJgaiM8qwiA== X-Gm-Gg: ASbGnctN1m7UlE2QT1diZgNBR6/CH1CdhjBhTfu7cJn6TBuLuB4ZnfdAx3gmXI+Ya5t jazebt7KO9/4WCbfl+HOG10Ei1kGpmmAP0Iwfa15VNS9eehkHp+U0fMzHITgcBnMuGcEr4z/2/W h5fyIKio2RPUzYOlRMFcpnHWiU64XbZhK5QFYuMzyzuWHnGFyhBpQNq+gDLyRDyHataB/kPa7rW cKj7mhx36cAXttUBE/Bsbhr8EpXtQac3bzzv/3bRSwe+Zdvwzx8DFDn4PvUagFs4DLJqnGLCl+q EtxSv4d9EmsU+28sAXxPSmQqNEI9g7ZqCQLtth7IxbLbh6k= X-Google-Smtp-Source: AGHT+IH2tyXRUFNLNEQKOHaFYFPbUYX5TokW3hDEBo4R90Yp1othTxhspagFYn7ka7w60KBXBXHJeA== X-Received: by 2002:a05:6a00:1795:b0:736:3cc4:76db with SMTP id d2e1a72fcca58-7363cc4886amr7491350b3a.24.1740968698689; Sun, 02 Mar 2025 18:24:58 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:1513:4f61:a4d3:b418]) by smtp.gmail.com with UTF8SMTPSA id d2e1a72fcca58-73632e76e1dsm3701620b3a.89.2025.03.02.18.24.56 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sun, 02 Mar 2025 18:24:58 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Hillf Danton , Kairui Song , Sebastian Andrzej Siewior , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v10 05/19] zram: remove second stage of handle allocation Date: Mon, 3 Mar 2025 11:03:14 +0900 Message-ID: <20250303022425.285971-6-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.711.g2feabab25a-goog In-Reply-To: <20250303022425.285971-1-senozhatsky@chromium.org> References: <20250303022425.285971-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Previously zram write() was atomic which required us to pass __GFP_KSWAPD_RECLAIM to zsmalloc handle allocation on a fast path and attempt a slow path allocation (with recompression) if the fast path failed. Since we are not in atomic context anymore we can permit direct reclaim during handle allocation, and hence can have a single allocation path. There is no slow path anymore so we don't unlock per-CPU stream (and don't lose compressed data) which means that there is no need to do recompression now (which should reduce CPU and battery usage). Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 39 +++++++---------------------------- 1 file changed, 7 insertions(+), 32 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 93cedc60ac16..f043f35b17a4 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1723,11 +1723,11 @@ static int write_incompressible_page(struct zram *z= ram, struct page *page, static int zram_write_page(struct zram *zram, struct page *page, u32 index) { int ret =3D 0; - unsigned long handle =3D -ENOMEM; - unsigned int comp_len =3D 0; + unsigned long handle; + unsigned int comp_len; void *dst, *mem; struct zcomp_strm *zstrm; - unsigned long element =3D 0; + unsigned long element; bool same_filled; =20 /* First, free memory allocated to this slot (if any) */ @@ -1741,7 +1741,6 @@ static int zram_write_page(struct zram *zram, struct = page *page, u32 index) if (same_filled) return write_same_filled_page(zram, element, index); =20 -compress_again: zstrm =3D zcomp_stream_get(zram->comps[ZRAM_PRIMARY_COMP]); mem =3D kmap_local_page(page); ret =3D zcomp_compress(zram->comps[ZRAM_PRIMARY_COMP], zstrm, @@ -1751,7 +1750,6 @@ static int zram_write_page(struct zram *zram, struct = page *page, u32 index) if (unlikely(ret)) { zcomp_stream_put(zstrm); pr_err("Compression failed! err=3D%d\n", ret); - zs_free(zram->mem_pool, handle); return ret; } =20 @@ -1760,35 +1758,12 @@ static int zram_write_page(struct zram *zram, struc= t page *page, u32 index) return write_incompressible_page(zram, page, index); } =20 - /* - * handle allocation has 2 paths: - * a) fast path is executed with preemption disabled (for - * per-cpu streams) and has __GFP_DIRECT_RECLAIM bit clear, - * since we can't sleep; - * b) slow path enables preemption and attempts to allocate - * the page with __GFP_DIRECT_RECLAIM bit set. we have to - * put per-cpu compression stream and, thus, to re-do - * the compression once handle is allocated. - * - * if we have a 'non-null' handle here then we are coming - * from the slow path and handle has already been allocated. - */ - if (IS_ERR_VALUE(handle)) - handle =3D zs_malloc(zram->mem_pool, comp_len, - __GFP_KSWAPD_RECLAIM | - __GFP_NOWARN | - __GFP_HIGHMEM | - __GFP_MOVABLE); + handle =3D zs_malloc(zram->mem_pool, comp_len, + GFP_NOIO | __GFP_NOWARN | + __GFP_HIGHMEM | __GFP_MOVABLE); if (IS_ERR_VALUE(handle)) { zcomp_stream_put(zstrm); - atomic64_inc(&zram->stats.writestall); - handle =3D zs_malloc(zram->mem_pool, comp_len, - GFP_NOIO | __GFP_HIGHMEM | - __GFP_MOVABLE); - if (IS_ERR_VALUE(handle)) - return PTR_ERR((void *)handle); - - goto compress_again; + return PTR_ERR((void *)handle); } =20 if (!zram_can_store_page(zram)) { --=20 2.48.1.711.g2feabab25a-goog From nobody Tue Dec 16 19:24:32 2025 Received: from mail-pj1-f52.google.com (mail-pj1-f52.google.com [209.85.216.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4B43B1D54E2 for ; Mon, 3 Mar 2025 02:25:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.52 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740968705; cv=none; b=Tw46G/JBOJhayA2eVl+rnXl2jkhlYXd6clYMlyiY1vDokYECGtq8YMmEoYhBuAjOWDUYDKzLZNz7p2KGwCEeEIf5pmsanW9GfozIcKJhqUFufEcnT7TqKmMByhl931t/GmIY3yUylqyYy5ToFerDzW344YRdf49Z34yIGTwXS50= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740968705; c=relaxed/simple; bh=QqMqZBwRwdekx8gHdGlvH68RTT3BuwngwE/XH0oXg0w=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ljBU6mnPL9rCjqUASdI8yg/frNCrqDUvMozkT7bVp+jYadG22kIcZy6rOevYb4etfYJLZvb2Bhm0A3hXWZXCH0A79plOumW6hyk/C09knh3lSQmYrk62hXSreYDmN47iWvW78rXQDPrO0mq3XTSnGI7HaYJSVHopNXFBjB2IMDI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=Egd64HHT; arc=none smtp.client-ip=209.85.216.52 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="Egd64HHT" Received: by mail-pj1-f52.google.com with SMTP id 98e67ed59e1d1-2fe82414cf7so7678500a91.0 for ; Sun, 02 Mar 2025 18:25:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1740968703; x=1741573503; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=DytrayV9Mjuo6syNe/AR1TRl5JXW87zNZiaeyGLF4cM=; b=Egd64HHTq8M1FnwREeLOcxtVUlz0ogONDR1hYS9C3dVsewHbu6hLncvRbTaWETEp9U pMvcwF+KaWaZKhrUOiAcikCpBaAleaknwBy04vYDnr7hqYdYX86W7Ox85gyA9hU8NVWV l/X6Qf+rEQGHUqN6kEZccv8AdkwPrqOoCLorM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740968703; x=1741573503; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=DytrayV9Mjuo6syNe/AR1TRl5JXW87zNZiaeyGLF4cM=; b=OcJoX6+u+INnJ/MxqpodH7fwSCtliCgE0Q+6BGqgj5JeRIoqTJPw+Xd3V1fuEyvCju ORDU/tuD0NLQYhCJ+vvsPaNPZG8iawi7rFrrt+oj+z4TVHvtpSzc5uXhFSCKNu0LiIwA IIJFjQu18LjPBzt9QXs5Ql/nWcscjFC9VIIfANoMIaL+THXnJ5PNCIH412IMtNwoWp88 86H2mpNQlnPeZRuK2X4av9u5qbUWs10ZSkleK7WxJOwDPYNKSWGtU4lvc7pMIQF0mNWP TJB10p9YZiZFcuhl6wVhpLYNDyyx3uEWP/i3UXKW6a9nAVqxqjFjLCE1MPTnkAJnDyEC GwWg== X-Forwarded-Encrypted: i=1; AJvYcCVpWoocjQUolp/MBBr/95XBGnMI1B76VZWQO9T2Zoq8RHf2lqMJExIl59zATGO2qY2XEw4sW2abxEzy6MQ=@vger.kernel.org X-Gm-Message-State: AOJu0YxObSKdIOO/e8oJ5WeoSK5XU5KEZm8quoJWWRW9iffrYC5P8S92 8cnQbn6ZgMihqIjHFszL90j1toIwqybuPCvg3h85Lig48Lpf1STVNpORsJHHXQ== X-Gm-Gg: ASbGnct+lhb2k0QyyRkrzOR5voEN45wvCa00k8XYothdA7hPQIK1PmsJUoGERyCJbE1 m3OXMeQjpnDsRkeQNaAraDB3rFLyOtVmv5xCLpZtZrMP4k42Yp1gmy5QbJZE0hkCcOGMR6tmF7s tEX3opJgzPUlUf6FxUWjVMENq8S4NcnAmNA8CspLHfuU9FHo5pYeG9VBHzF1jeTYa1VFbqZ4Fx/ S6voRdqZH2VSaYNe9PxCpqplKlWn8BuYbh7lsanIZiq5LDc2e+z24jTphQTq3hEdsb4Gab3a6Zt aVdKZZuybeqhftY7jM4qAMEP5pGPv4v54Zm1+/JHZLZftGk= X-Google-Smtp-Source: AGHT+IHl5Crctco8Hu+E8EG5MtD0b7+xg8Bu5mtdXcu6UNEW0JPimU6uMCyi2CRbQ02nGaq5KYpYtQ== X-Received: by 2002:a17:90b:3ec4:b0:2fa:b84:b31f with SMTP id 98e67ed59e1d1-2febabedb6amr16911551a91.25.1740968703617; Sun, 02 Mar 2025 18:25:03 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:1513:4f61:a4d3:b418]) by smtp.gmail.com with UTF8SMTPSA id 98e67ed59e1d1-2fea676c4dcsm8609448a91.17.2025.03.02.18.25.01 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sun, 02 Mar 2025 18:25:03 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Hillf Danton , Kairui Song , Sebastian Andrzej Siewior , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v10 06/19] zram: add GFP_NOWARN to incompressible zsmalloc handle allocation Date: Mon, 3 Mar 2025 11:03:15 +0900 Message-ID: <20250303022425.285971-7-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.711.g2feabab25a-goog In-Reply-To: <20250303022425.285971-1-senozhatsky@chromium.org> References: <20250303022425.285971-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" We normally use __GFP_NOWARN for zsmalloc handle allocations, add it to write_incompressible_page() allocation too. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index f043f35b17a4..249a936b6aac 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1691,7 +1691,8 @@ static int write_incompressible_page(struct zram *zra= m, struct page *page, * like we do for compressible pages. */ handle =3D zs_malloc(zram->mem_pool, PAGE_SIZE, - GFP_NOIO | __GFP_HIGHMEM | __GFP_MOVABLE); + GFP_NOIO | __GFP_NOWARN | + __GFP_HIGHMEM | __GFP_MOVABLE); if (IS_ERR_VALUE(handle)) return PTR_ERR((void *)handle); =20 --=20 2.48.1.711.g2feabab25a-goog From nobody Tue Dec 16 19:24:32 2025 Received: from mail-pj1-f44.google.com (mail-pj1-f44.google.com [209.85.216.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 12F401D54E2 for ; Mon, 3 Mar 2025 02:25:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.44 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740968710; cv=none; b=GQuFFnEfbcv2DnoX3adKs4sGEw5Geji5FsDV5/l/Wru4ufmzaVdI5MuYbyujTOIIZWfp/59B57aIWwso23DZgo9xUECrvT3705oagQKTo5kwvAoAnYNyitd7QNuCuaTHiPhNvd5qCYmeNU6c7c8EfNWLv5kxbfCZ09qZOU+HvrY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740968710; c=relaxed/simple; bh=CoKXRzXgooycgC4/aLBIRCF3+UwIHim9p6IKRBVdXSI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=LVxA1FVHE3V7ilgnbChwQBR1yulNx85N02YhXWR1gtP3i1FnYDN0G/zSpMzGJ9kjn5dp/G2baXdr4i0SJYv+FpKeJfRPT0lXNgQbiR64Y/z3WV4BfApxVhL/3TPDRoJ76Dmhkkm13Cezh3egTmdnxxJuhJqbwjziBfyjEhVGkdU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=R0knPVUK; arc=none smtp.client-ip=209.85.216.44 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="R0knPVUK" Received: by mail-pj1-f44.google.com with SMTP id 98e67ed59e1d1-2ff187f027fso250333a91.1 for ; Sun, 02 Mar 2025 18:25:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1740968708; x=1741573508; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Q9mhj8QPNzHEXvTH0OVXBitiCwr2gmDik1RNTqYshT0=; b=R0knPVUKRGYgYMElTm8D9PYsBZyTqUjxbImOdJREqYmTIkK5FvHVzxIlhhPrqthIGU +ZoMwILptYuMKbCL2BFFm0R/J35CyeVhwxzJCLb56Agm7fIFxG0v/WubiS5+ncBLnF8S iIx8wblR57C/u5HfnVh5ycFgwmj5dJ8Z/LpaQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740968708; x=1741573508; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Q9mhj8QPNzHEXvTH0OVXBitiCwr2gmDik1RNTqYshT0=; b=ftGUngcN2nlW1XjPqnvqEXNPjCpNVGVOPFRGO1VUg0Q4Smy2m0d6TeZdDcbPlL9lD7 cTfuVJO2fH2DzP7/Lxncn+s4K/tif2Jd0LIQzdkNW6jCiTsSeWcnLHh7Uq3fefG0K43C eFWNvImHvIj+SJtfkDfbn+jy0DZKU0p63GTKBRIs8HelT8Lq3blilh9K2VM8TF7nDu1p H9tvSPanZZm8wHudkYSzWUD5e2Wt5bE4hSZTnmKfj501MSlf4KAz1whpvRRvcWbgpnBp bFBR23s1zfcEmL22y2ub11ZTilaoreypQSFTjHJ29pTSJxFeAY6cIG9VkL1H5p8M4Vur LHDg== X-Forwarded-Encrypted: i=1; AJvYcCURHWHZ26w2ceC4vT7QK45HdGRxtos4/1xd06j3gVGIiogcDq2LokOWzxkT/t/62kcKowzGwt2znwcRJlY=@vger.kernel.org X-Gm-Message-State: AOJu0YwxaOmABSWK+EyfWrOzZnGiKZWNUku+zGfHcJPntrnftSj5cZFO xx8r8AcBDrFwtMLu9ECtwYCU+mrfSSuvi27Mj6ZuIbmalw6RDPRnD72lD407hA== X-Gm-Gg: ASbGncsyCWK/RDQiTrENOdm/wiPU48gQGCpepL6Poh+fOIbqi+5VlKsXMI0nGJefbFG dFXtafi3yL4+zNF5xxFZOdHvlRhLrSIMA4fftuIsF22ZvQfz2wXhltA5ExFlD3blOR7qIhwVuvR ziZdKdqEgqpijR0+xsFbgP2P/lkinl7VRxKxxqU2PpOCmUBBNEIHQi1i6QsW69S6e+WO4eYIjXu 3pjHSD/AoZsRrpoCQZCkDDaaAqIIScgJYTaheg1g9Ry9QsZNUlCJgkkVBdxs5wMizaIcDmzaeJW QL8L30JKyzWcmKJ8fXPR8FCLnk0hvWq21Yjmp7qpInHF4UA= X-Google-Smtp-Source: AGHT+IF66VEKY5wPrt/KNtaj7+PAWT785N+ktrJ+PZh+m4XyOgJ988yBEbAqjba5GwAAOb7NWMraHg== X-Received: by 2002:a17:90b:2513:b0:2f9:d0cd:3403 with SMTP id 98e67ed59e1d1-2feba95b413mr17873240a91.16.1740968708303; Sun, 02 Mar 2025 18:25:08 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:1513:4f61:a4d3:b418]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-223501fae03sm66894545ad.84.2025.03.02.18.25.05 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sun, 02 Mar 2025 18:25:07 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Hillf Danton , Kairui Song , Sebastian Andrzej Siewior , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v10 07/19] zram: remove writestall zram_stats member Date: Mon, 3 Mar 2025 11:03:16 +0900 Message-ID: <20250303022425.285971-8-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.711.g2feabab25a-goog In-Reply-To: <20250303022425.285971-1-senozhatsky@chromium.org> References: <20250303022425.285971-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" There is no zsmalloc handle allocation slow path now and writestall is not possible any longer. Remove it from zram_stats. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 3 +-- drivers/block/zram/zram_drv.h | 1 - 2 files changed, 1 insertion(+), 3 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 249a936b6aac..fc9321af3ef4 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1437,9 +1437,8 @@ static ssize_t debug_stat_show(struct device *dev, =20 down_read(&zram->init_lock); ret =3D scnprintf(buf, PAGE_SIZE, - "version: %d\n%8llu %8llu\n", + "version: %d\n0 %8llu\n", version, - (u64)atomic64_read(&zram->stats.writestall), (u64)atomic64_read(&zram->stats.miss_free)); up_read(&zram->init_lock); =20 diff --git a/drivers/block/zram/zram_drv.h b/drivers/block/zram/zram_drv.h index 7c11f9dab335..6cee93f9c0d0 100644 --- a/drivers/block/zram/zram_drv.h +++ b/drivers/block/zram/zram_drv.h @@ -82,7 +82,6 @@ struct zram_stats { atomic64_t huge_pages_since; /* no. of huge pages since zram set up */ atomic64_t pages_stored; /* no. of pages currently stored */ atomic_long_t max_used_pages; /* no. of maximum pages stored */ - atomic64_t writestall; /* no. of write slow paths */ atomic64_t miss_free; /* no. of missed free */ #ifdef CONFIG_ZRAM_WRITEBACK atomic64_t bd_count; /* no. of pages in backing device */ --=20 2.48.1.711.g2feabab25a-goog From nobody Tue Dec 16 19:24:32 2025 Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E4B1B19CD07 for ; Mon, 3 Mar 2025 02:25:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740968715; cv=none; b=l7S8UYUnz0f1RAi40k6Nkhv/QeBI92ZP1CrPW/COqnEyFxPHYSon/UvrPad8fNcEP+PzDDaYEGMfzgUD4uJBIWBz2tIaOkI95B60xkTxtPS01dhvavqHS4SlZTEBAiS8YaH8lSd2hexNJsY/Ig0gH2+DGQwr8SNMsY09GjwRYtc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740968715; c=relaxed/simple; bh=09Qa7iY/cz/uCJcBw8zuCyFMVwvRxKAO3reUrH0m5QY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=fP1ZqEVSENULJNLJieU9mYfynZzKKnVKW1t3GTY5qLZIbG/GFcPlYZS2WX6EZ0iDUcxSdE5wvRxeg6c/tkeP4RTy/Ds9SPN24IC4e4UNRAaWrXHPKLFDNJ2tveQsvzJSOIrGAPDzoumIW5we0M95/aSC+iTZlADjlkEoec/euwE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=OkGqLc+y; arc=none smtp.client-ip=209.85.214.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="OkGqLc+y" Received: by mail-pl1-f169.google.com with SMTP id d9443c01a7336-219f8263ae0so70240735ad.0 for ; Sun, 02 Mar 2025 18:25:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1740968713; x=1741573513; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=eo+gLdw4ALzCaj1nm0+nyzWVeAdZV8iG/qfb0LErfSY=; b=OkGqLc+yiPHADt6hm56o5YtZvjdbBg1w5rAiBhJezzpAnMc+9cOZysd4qrFOqN/7bA KD5O9yhNRg8m1BDhs1wJbI34pUos/dDacSQ5ptjHR0dOD7VvMbOQmg6qkf46hdB5EeIK Mj7zzJvx4bkX62fyjPZ9it+QLfpz2QJa4N+rs= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740968713; x=1741573513; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=eo+gLdw4ALzCaj1nm0+nyzWVeAdZV8iG/qfb0LErfSY=; b=KGIX5MLP+ZojdB4IaToWTBJCulvJ0riXZSjtjvn+t8i491YwO6aM4c2pfPa/jLuL/U d94tS464IdcSkMUKrF6s55n/YFaW2IcKbeD+nAJGIlIYBVMAsfDJXVuqM1rYYRVHrIpo ZSFglX35MIWOf8SKPxoDSvfDN87vAwGdzLbaMa/KqqZcAjoyRM06vsVBBHLBmMPMSeLb Qoj/GcjMhfhhUKNwsRSs2Gf201bc1fl17BFTmgPr9FM18jayrpDeWWEIwi+tC0y+4TuJ h37vEyT9bPav7LTUPcWVTij/S7MlbtHsFM0aSjZOZ9fsJwjCtEK3h9fAF2uSe5s0qPu7 o12Q== X-Forwarded-Encrypted: i=1; AJvYcCXOKN4woNznw3h50ZJi8YX+CVviYrfrdAVWP8as1/jd14CXSQW0rBgU6jh4OwlvsWWW4OpZF53Cg0hLlQ8=@vger.kernel.org X-Gm-Message-State: AOJu0Yx1N3JR53Dss0KZlvNULnc2QIehoGkyQfgM85NL0q3vv4hXi4Qx yMuqHmFVYeErwl/z5gosrxZoEa0+qkm+H1u7KpCtxEH6AWRxomMDoGtzVa8QDg== X-Gm-Gg: ASbGnct+/4Jk/li37X09+obSrUn98nfV5EG0ie4j13W7PD38/zPp+9q2Sufbg8wQ9pP IFeoYUCDE6nNuxCatyxfo8/Sl4MQlbey1fFalxN6aQkerNrS/J73ztxwpWtsq7g0Ne3eXXlW987 VfL+riqIczAwMPaGffTZEjcpjbOyLL9ZCXlYONnq9QYqumNmzLcX2it3fe2lMHs4YSi5hc44clL OkkK86tVftA+X7ppQwJQgpTeYE5m24Y5+sYr3Gj5O1RVwe2d8yNvOSFL2mMTGWcDYAHbLslEAc8 QhtCCiDI5Cail4jEp9p0dUcGfpGl3utKe3UMih32gWUAUEA= X-Google-Smtp-Source: AGHT+IFnUy2t+MQbv/ON6Qzd6lspOJ7em303a/vGQ6h5MRjg7inxZUPbfNYd5XTa2AnmAT9Ovl5HQw== X-Received: by 2002:a17:903:1983:b0:223:5ada:891e with SMTP id d9443c01a7336-22368f6a1d9mr130419345ad.7.1740968713338; Sun, 02 Mar 2025 18:25:13 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:1513:4f61:a4d3:b418]) by smtp.gmail.com with UTF8SMTPSA id d2e1a72fcca58-73638e4ab93sm3434904b3a.103.2025.03.02.18.25.11 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sun, 02 Mar 2025 18:25:13 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Hillf Danton , Kairui Song , Sebastian Andrzej Siewior , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v10 08/19] zram: limit max recompress prio to num_active_comps Date: Mon, 3 Mar 2025 11:03:17 +0900 Message-ID: <20250303022425.285971-9-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.711.g2feabab25a-goog In-Reply-To: <20250303022425.285971-1-senozhatsky@chromium.org> References: <20250303022425.285971-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Use the actual number of algorithms zram was configure with instead of theoretical limit of ZRAM_MAX_COMPS. Also make sure that min prio is not above max prio. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index fc9321af3ef4..776c31606eec 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -2027,16 +2027,19 @@ static ssize_t recompress_store(struct device *dev, struct device_attribute *attr, const char *buf, size_t len) { - u32 prio =3D ZRAM_SECONDARY_COMP, prio_max =3D ZRAM_MAX_COMPS; struct zram *zram =3D dev_to_zram(dev); char *args, *param, *val, *algo =3D NULL; u64 num_recomp_pages =3D ULLONG_MAX; struct zram_pp_ctl *ctl =3D NULL; struct zram_pp_slot *pps; u32 mode =3D 0, threshold =3D 0; + u32 prio, prio_max; struct page *page; ssize_t ret; =20 + prio =3D ZRAM_SECONDARY_COMP; + prio_max =3D zram->num_active_comps; + args =3D skip_spaces(buf); while (*args) { args =3D next_arg(args, ¶m, &val); @@ -2089,7 +2092,7 @@ static ssize_t recompress_store(struct device *dev, if (prio =3D=3D ZRAM_PRIMARY_COMP) prio =3D ZRAM_SECONDARY_COMP; =20 - prio_max =3D min(prio + 1, ZRAM_MAX_COMPS); + prio_max =3D prio + 1; continue; } } @@ -2117,7 +2120,7 @@ static ssize_t recompress_store(struct device *dev, continue; =20 if (!strcmp(zram->comp_algs[prio], algo)) { - prio_max =3D min(prio + 1, ZRAM_MAX_COMPS); + prio_max =3D prio + 1; found =3D true; break; } @@ -2129,6 +2132,12 @@ static ssize_t recompress_store(struct device *dev, } } =20 + prio_max =3D min(prio_max, (u32)zram->num_active_comps); + if (prio >=3D prio_max) { + ret =3D -EINVAL; + goto release_init_lock; + } + page =3D alloc_page(GFP_KERNEL); if (!page) { ret =3D -ENOMEM; --=20 2.48.1.711.g2feabab25a-goog From nobody Tue Dec 16 19:24:32 2025 Received: from mail-pl1-f178.google.com (mail-pl1-f178.google.com [209.85.214.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5850519CD07 for ; Mon, 3 Mar 2025 02:25:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740968720; cv=none; b=NEc6SREPHxUrDqlbDjXB4XGVpe0KFW2rlVbp4c43RndWcnvziUoH5fLvJ/39lu7ZhzkewOO4ly7+zwRGOmq1dubQQqNgTtf/90V8ANDfz3IVzH7P274tjHdN5IWJnIs0lp8b3XC/RrcGCowoDnl4DsQDkvuOJyScc+5TesCBA2A= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740968720; c=relaxed/simple; bh=eBUSVGGrh1XSgDzgF0DRhjlLomM6rB+SwQkemEOuOno=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=dzwca3OgBRQ/GpQH+Zyn+/UnVWFNPzeYAd0Wn2fEunHXMEnUlyhcxyAzJaHgimiYNsdQithnZ9atCku7nDWwLwsF3FowKFvVYbjMW+m9Qdz38uNdH/Q+uc7zL+4JHDsiusQ84i/XHDnZlke81eYkQT7hPOV3TKBwr0uBFjWOHJo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=DAJrhxB9; arc=none smtp.client-ip=209.85.214.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="DAJrhxB9" Received: by mail-pl1-f178.google.com with SMTP id d9443c01a7336-22355618fd9so63391275ad.3 for ; Sun, 02 Mar 2025 18:25:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1740968718; x=1741573518; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=NTrj6AIBGg3fen5fqcQqiq+dr13fIgG4uK8Ep+QAN48=; b=DAJrhxB9lgMm8nG6qS92LV88CKydbKKooIMnBQtlgLDgV2JFforERRHJDk7a7q/M/j RTBeRx0CNGmCNlRILMv4MdVUBlHKWPLXnDGVl5c/0xCOzdrvm/7CmqL/T0jc0X1OX062 ee4CokoRXTPqiH6KGFRnWK1LMYOJp5m7RERQw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740968718; x=1741573518; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=NTrj6AIBGg3fen5fqcQqiq+dr13fIgG4uK8Ep+QAN48=; b=SmakPFp3S2dVK4lWfYv2Zty4tI7DUCn8nvCBO3VUvHMiQBmaaB0a/luvjKAdOjNFCn kh1k6++ct3Zm2++32pR8wyWEuq6Rk47yYs1KxORQeuNPLeiGPGdMv0VP/leVxzX1Fw7w rOGDwDjuD4sOpklp3ZFV5oz5C2TuvG33vfP6+EK1c2DKtZcMLrO4AR/9cjuIF7FYvdf5 +MkA9VRSvo7K+ZsehKSA72ZUtJtG8OvA/OcT70m61/nLViJ90SesF4ICQLDkOGYhyjC0 jj4XB3irGVRFYCA4AKqmpUsKCNrUFl+wFr+HCFu9rhGa4Y7VrYUd/M5lgKxStOs5TPwj UJEQ== X-Forwarded-Encrypted: i=1; AJvYcCWGEk7QFQtboGmT/4ebs/4+nFohOQoqlkgwzfO/BHZJ7JL7e9dgkboToCkz4xDzl6LwVJU4PhBwv34q4d4=@vger.kernel.org X-Gm-Message-State: AOJu0Yyw5tvpgfOd2O+Mj4mVB7Ejpi3Dz1uNjUBmzPYfUkkhwVX8+c6n X1WFeaS6kr9ciogbsE30SOJb14bxm5ZbHE8ZYA948ygHo0SaTOzroEYJa8qAyQ== X-Gm-Gg: ASbGnctwqRGibp8VIPBRXJrJnGlQz0IG+1G/ZaJQPyQf37S2fKE0vCMFEdtwWlGAnyy 4b0JTNd2sgSZ8J7x2CKh/fw0gBBRgdSMgin9RRRuSnKkNBJoBUig/LiG9YhBdm7CyUp1V3NaQjQ u6cvvHAIosavqzirf9zkhl9XLCfCkcdUuADWwSS8TMzkbsaqYHegmnEb3DO5nrg8is4i5w4wrNJ L05w/9XcuyB8UQNh2u0VzgMze4yszNoyfvCqqSqajbHPeAoWWaHQrNk8tr5WVHtOEoyuDMDhOTy uuYNn9g3HAANIDXl9lJbxjaEI+4r7bQVctr7KI+EfkHcIzA= X-Google-Smtp-Source: AGHT+IH8tkcTWEOl7teGtIH3prNMr9e6TjPXzSkEQyQdyulF5O353ekND1QrhJUZq9VyBdS7SHU/9A== X-Received: by 2002:a05:6a00:4f96:b0:736:51ab:7ae1 with SMTP id d2e1a72fcca58-73651ab7b97mr3452819b3a.16.1740968718515; Sun, 02 Mar 2025 18:25:18 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:1513:4f61:a4d3:b418]) by smtp.gmail.com with UTF8SMTPSA id d2e1a72fcca58-7363f62117asm2509047b3a.57.2025.03.02.18.25.16 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sun, 02 Mar 2025 18:25:18 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Hillf Danton , Kairui Song , Sebastian Andrzej Siewior , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v10 09/19] zram: filter out recomp targets based on priority Date: Mon, 3 Mar 2025 11:03:18 +0900 Message-ID: <20250303022425.285971-10-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.711.g2feabab25a-goog In-Reply-To: <20250303022425.285971-1-senozhatsky@chromium.org> References: <20250303022425.285971-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Do no select for post processing slots that are already compressed with same or higher priority compression algorithm. This should save some memory, as previously we would still put those entries into corresponding post-processing buckets and filter them out later in recompress_slot(). Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 25 ++++++++++++++++--------- 1 file changed, 16 insertions(+), 9 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 776c31606eec..6dee885bef9b 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1823,7 +1823,7 @@ static int zram_bvec_write(struct zram *zram, struct = bio_vec *bvec, #define RECOMPRESS_IDLE (1 << 0) #define RECOMPRESS_HUGE (1 << 1) =20 -static int scan_slots_for_recompress(struct zram *zram, u32 mode, +static int scan_slots_for_recompress(struct zram *zram, u32 mode, u32 prio= _max, struct zram_pp_ctl *ctl) { unsigned long nr_pages =3D zram->disksize >> PAGE_SHIFT; @@ -1855,6 +1855,10 @@ static int scan_slots_for_recompress(struct zram *zr= am, u32 mode, zram_test_flag(zram, index, ZRAM_INCOMPRESSIBLE)) goto next; =20 + /* Already compressed with same of higher priority */ + if (zram_get_priority(zram, index) + 1 >=3D prio_max) + goto next; + pps->index =3D index; place_pp_slot(zram, ctl, pps); pps =3D NULL; @@ -1911,6 +1915,16 @@ static int recompress_slot(struct zram *zram, u32 in= dex, struct page *page, zram_clear_flag(zram, index, ZRAM_IDLE); =20 class_index_old =3D zs_lookup_class_index(zram->mem_pool, comp_len_old); + + prio =3D max(prio, zram_get_priority(zram, index) + 1); + /* + * Recompression slots scan should not select slots that are + * already compressed with a higher priority algorithm, but + * just in case + */ + if (prio >=3D prio_max) + return 0; + /* * Iterate the secondary comp algorithms list (in order of priority) * and try to recompress the page. @@ -1919,13 +1933,6 @@ static int recompress_slot(struct zram *zram, u32 in= dex, struct page *page, if (!zram->comps[prio]) continue; =20 - /* - * Skip if the object is already re-compressed with a higher - * priority algorithm (or same algorithm). - */ - if (prio <=3D zram_get_priority(zram, index)) - continue; - num_recomps++; zstrm =3D zcomp_stream_get(zram->comps[prio]); src =3D kmap_local_page(page); @@ -2150,7 +2157,7 @@ static ssize_t recompress_store(struct device *dev, goto release_init_lock; } =20 - scan_slots_for_recompress(zram, mode, ctl); + scan_slots_for_recompress(zram, mode, prio_max, ctl); =20 ret =3D len; while ((pps =3D select_pp_slot(ctl))) { --=20 2.48.1.711.g2feabab25a-goog From nobody Tue Dec 16 19:24:32 2025 Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A55671993B9 for ; Mon, 3 Mar 2025 02:25:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740968725; cv=none; b=p2IN38JDFzqc3rDUH7rVUQteUh/0KrFVD/SpmlkOpJn9z9rKXbnrU+LHu/4T5njE0jKeG9f4RAkht1K7O/sAYdff8jAEUD7bIJotZu6jIg52KdJDFu8xE34j5RokGEaMmTQxYU0FkaiOk2MeLI2PAf5jlDhLuFHHOsAyprCWF3Q= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740968725; c=relaxed/simple; bh=2ZpbQwE8lXjU9ojwoVL4iVfganER1zQltghI5w7rM7A=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=MHz7QWEWamdaeMM/Ux33o3INp5KG3mkbd/piBVr0+oDq9HgWDE0CA2SqpOiEr5E02CJaZK6LYM4DnLlTR9fjsUvjGWDLrjId2tZ4R7hCTwFNXBXV345DA3JN7gnzqhQEQ0wKwOFAtWZ8Ew4UKYJXgvnPPJ+08rphzJJIjRNllTo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=NI7s5naV; arc=none smtp.client-ip=209.85.214.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="NI7s5naV" Received: by mail-pl1-f169.google.com with SMTP id d9443c01a7336-22356471820so59491915ad.0 for ; Sun, 02 Mar 2025 18:25:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1740968723; x=1741573523; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=HsfvfF6qJrjX8Yvxt2fgmxWzD/A8epAgFC3OUlbDf8w=; b=NI7s5naVbZK2QE4oUmiZKHjO/xm8txM+v4tJB+kVy/smMeQEKGsZBzkLxhNmDjPqhJ zuzWh/rxUp3efDOuKB33lb2ljAMM5KYdCm6dk/hQSSBU9RnnvJ4QGW3TGYQY9XVbCidM GBhetfLOxoHW5yZOAEFhneUTZJGPzS1D8u+0I= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740968723; x=1741573523; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HsfvfF6qJrjX8Yvxt2fgmxWzD/A8epAgFC3OUlbDf8w=; b=Sndv2nDBBlHl9nSnGzvuvnY5PwkXoGznS3P+jRFnwW/IbMVpIq/EJA63KVOEakfD6z ky6yr39y9Wubhg/UOwR/ugr/kMrjzAQ6lUVEd3szCyTLQoLsEPz4kaZleeJ/u8Ek3e68 gqmY3mBjSKSOOc+Tl3laqBi43dloHsYbH1Y5V1Rssj1WrMuo7dmYzEBZ1qUUcD+6PqGw TzQ1qjzhu58wooRt5M/s+6OWKp3yU4icxf7dOn0lpe5TZI9QoCc1RUKYMXfZy0FMOSnV PiwwQA6dfT7dN5mSY2HMvFP4lmt0fa4AhQdJX8IKVfiA8I8CUPx+jF+puWRarDmbhACI MfJQ== X-Forwarded-Encrypted: i=1; AJvYcCV87UWl3NhnqebukseGtdOsH79M/xakzq90RISV7oqw3vLsMEVWzUb37iOR8mBa8F4ihdrzEhcPok1DrBw=@vger.kernel.org X-Gm-Message-State: AOJu0YwwNUDH1kyGA2R9L/oGrPLaYj7/KPnzCkekGC0BjeCI4g8HOKDz VFNPQ+FjfgwKukatL6i0uQZArPXzBSDL9Jwm09wtC0EMQllnn4jvvmasjjU09w== X-Gm-Gg: ASbGncvEYnrscL6INoO0mSDGoJ8sA058ihXZ6erCX9bBiwNzRPUPHQmGNAU0CW5x+Uv 0k4u4sHSzLdteWgN7fXa5Cj78AsEWsAoJaqQJ1gLloEthN2QYjnlgD3vU65e9in198Q9EQqJPXw r9CqqLey+PftOrDPBBFx+tuQbpttFl+YSjvPWyeIN4jhU0RnsskxmiXWJ7Zl4qMD/aBNT6uTUDq Skr+LdJXg5WuXDkfTXIkzwTHM8lAm6xrKSyZpTzKk/W5MPFYFIl036l68AbulDde/lil1wrUgQb MiCiXSoh9/L8iK3Qibpa4Zm5U0qiqZRMtPbW/HzfKC6V4eU= X-Google-Smtp-Source: AGHT+IESsySExdYFB+0EQ5FWR9mq2/h7S2JmgVfs+L+05mw3XKfkerymt3bltrTq9TH/mZJI9SEzSA== X-Received: by 2002:a05:6a00:a11:b0:730:927c:d451 with SMTP id d2e1a72fcca58-734ac42ce9fmr14388946b3a.20.1740968722924; Sun, 02 Mar 2025 18:25:22 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:1513:4f61:a4d3:b418]) by smtp.gmail.com with UTF8SMTPSA id d2e1a72fcca58-7364ba1371asm1572928b3a.5.2025.03.02.18.25.20 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sun, 02 Mar 2025 18:25:22 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Hillf Danton , Kairui Song , Sebastian Andrzej Siewior , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v10 10/19] zram: rework recompression loop Date: Mon, 3 Mar 2025 11:03:19 +0900 Message-ID: <20250303022425.285971-11-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.711.g2feabab25a-goog In-Reply-To: <20250303022425.285971-1-senozhatsky@chromium.org> References: <20250303022425.285971-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This reworks recompression loop handling: - set a rule that stream-put NULLs the stream pointer If the loop returns with a non-NULL stream then it's a successful recompression, otherwise the stream should always be NULL. - do not count the number of recompressions Mark object as incompressible as soon as the algorithm with the highest priority failed to compress that object. - count compression errors as resource usage Even if compression has failed, we still need to bump num_recomp_pages counter. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 54 +++++++++++++---------------------- 1 file changed, 20 insertions(+), 34 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 6dee885bef9b..bb88b63d193b 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1888,9 +1888,8 @@ static int recompress_slot(struct zram *zram, u32 ind= ex, struct page *page, unsigned int comp_len_new; unsigned int class_index_old; unsigned int class_index_new; - u32 num_recomps =3D 0; void *src, *dst; - int ret; + int ret =3D 0; =20 handle_old =3D zram_get_handle(zram, index); if (!handle_old) @@ -1933,7 +1932,6 @@ static int recompress_slot(struct zram *zram, u32 ind= ex, struct page *page, if (!zram->comps[prio]) continue; =20 - num_recomps++; zstrm =3D zcomp_stream_get(zram->comps[prio]); src =3D kmap_local_page(page); ret =3D zcomp_compress(zram->comps[prio], zstrm, @@ -1942,7 +1940,8 @@ static int recompress_slot(struct zram *zram, u32 ind= ex, struct page *page, =20 if (ret) { zcomp_stream_put(zstrm); - return ret; + zstrm =3D NULL; + break; } =20 class_index_new =3D zs_lookup_class_index(zram->mem_pool, @@ -1952,6 +1951,7 @@ static int recompress_slot(struct zram *zram, u32 ind= ex, struct page *page, if (class_index_new >=3D class_index_old || (threshold && comp_len_new >=3D threshold)) { zcomp_stream_put(zstrm); + zstrm =3D NULL; continue; } =20 @@ -1959,14 +1959,6 @@ static int recompress_slot(struct zram *zram, u32 in= dex, struct page *page, break; } =20 - /* - * We did not try to recompress, e.g. when we have only one - * secondary algorithm and the page is already recompressed - * using that algorithm - */ - if (!zstrm) - return 0; - /* * Decrement the limit (if set) on pages we can recompress, even * when current recompression was unsuccessful or did not compress @@ -1976,38 +1968,32 @@ static int recompress_slot(struct zram *zram, u32 i= ndex, struct page *page, if (*num_recomp_pages) *num_recomp_pages -=3D 1; =20 - if (class_index_new >=3D class_index_old) { + /* Compression error */ + if (ret) + return ret; + + if (!zstrm) { /* * Secondary algorithms failed to re-compress the page - * in a way that would save memory, mark the object as - * incompressible so that we will not try to compress - * it again. + * in a way that would save memory. * - * We need to make sure that all secondary algorithms have - * failed, so we test if the number of recompressions matches - * the number of active secondary algorithms. + * Mark the object incompressible if the max-priority + * algorithm couldn't re-compress it. */ - if (num_recomps =3D=3D zram->num_active_comps - 1) - zram_set_flag(zram, index, ZRAM_INCOMPRESSIBLE); + if (prio < zram->num_active_comps) + return 0; + zram_set_flag(zram, index, ZRAM_INCOMPRESSIBLE); return 0; } =20 - /* Successful recompression but above threshold */ - if (threshold && comp_len_new >=3D threshold) - return 0; - /* - * No direct reclaim (slow path) for handle allocation and no - * re-compression attempt (unlike in zram_write_bvec()) since - * we already have stored that object in zsmalloc. If we cannot - * alloc memory for recompressed object then we bail out and - * simply keep the old (existing) object in zsmalloc. + * We are holding per-CPU stream mutex and entry lock so better + * avoid direct reclaim. Allocation error is not fatal since + * we still have the old object in the mem_pool. */ handle_new =3D zs_malloc(zram->mem_pool, comp_len_new, - __GFP_KSWAPD_RECLAIM | - __GFP_NOWARN | - __GFP_HIGHMEM | - __GFP_MOVABLE); + GFP_NOIO | __GFP_NOWARN | + __GFP_HIGHMEM | __GFP_MOVABLE); if (IS_ERR_VALUE(handle_new)) { zcomp_stream_put(zstrm); return PTR_ERR((void *)handle_new); --=20 2.48.1.711.g2feabab25a-goog From nobody Tue Dec 16 19:24:32 2025 Received: from mail-pl1-f170.google.com (mail-pl1-f170.google.com [209.85.214.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 97B041DE882 for ; Mon, 3 Mar 2025 02:25:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740968731; cv=none; b=P6KlUgWT8zPtp8d8sdW7bn9BQAqmSeP402MXeGogRIW8LLoD7AbXe7A3yPwOoDcUYG5jXTnXr2xs5qyOPj2sPojB6R6GywVd8tSN3U2lckTyuq4xBSuhQtklNF8CACo+DAv7LYK4r+Lm4cw7eI95xnDynyJTPGwsZA1LBQnzZCY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740968731; c=relaxed/simple; bh=NDKu3Rf24m5hNyNA1vH544UmtsqcvVbbfEpVOZRgTaU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=WoEcZpSyWOQVpm2BIOaM3ss8L/+LWj4lJxeDC7Q3PuEc+yJtkd7GS1EA7Du3kOmpet2VP4iC8/We88P+rdaXXOwBlTLHqBFHRJZYRgvAenLL+sq0rZW9QjhSRryZmyKxn7g9+CVKrxDSgM7TUjbaThja2pAav7smnp9bzxTgEGs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=K+KMLpaJ; arc=none smtp.client-ip=209.85.214.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="K+KMLpaJ" Received: by mail-pl1-f170.google.com with SMTP id d9443c01a7336-2234bec7192so48546795ad.2 for ; Sun, 02 Mar 2025 18:25:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1740968728; x=1741573528; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=/kGJl06ulFbB/829XkPkygYoBTQLBZ6+y+IJrZY3g6o=; b=K+KMLpaJEd10fiKp166BDRo18oqHxVbJb6lGxWA3EbvKlVj3lCH1ULiaECgONtYRas wNqJKIbsIGB0EC5GGqsV2V1wzpUCmm9G7fbIZ4uJTxJRMawffzI9P1diyDKWauCM7FsZ oh3xgv8UjzRoIJ9i4KkpdGNPThEe2T5WcAOWg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740968728; x=1741573528; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=/kGJl06ulFbB/829XkPkygYoBTQLBZ6+y+IJrZY3g6o=; b=mWxkRQ62AVodflaDP2/WSA2NDnqnXlQS4U9SEkbrPfeW+iDFUILe12wn2P4JX6dVt3 3ZcecIe1gBtq1N5BqCBSf9PIFHhP62Q9cxtzd03vTInViqykcjwNLHcfWjdZ20eIH1C2 9OX5YeJEanto/Tg+Ww+XLNC8csXTA6y8RipXwkrxBSjr0XmaKG3ddiR3oeGInbKWSeQ4 olFbghTX3LFgAcFu6cqga0saMgKHlyH17jl+1DJjOA447VTKVMchOdcoMFtGSwIo/WTo Z/tpr8ftTFFBp20HMV3++RAEYNCKVYAy1+qTgAvqEk2nCgJwdLtElLueAvN21WSXLDzc uPBg== X-Forwarded-Encrypted: i=1; AJvYcCWOK29MQmw7dKXmZ9oY5ooCZX89VY5Tk8o94grRGgvyBUgjZKmswr4Z5Td3W8wHy/1Ux29gjzSKf2jaH40=@vger.kernel.org X-Gm-Message-State: AOJu0YwvTOBGJ738ZZ9dp0Gj7RbsePfp5/d1krJmeosijExkMeaDHecH xYv25u7sLbcz8HDXmq67DO8jXvKCQJx4hkhQbTP4Gd/vck1Q1+f/vXxcz5xubw== X-Gm-Gg: ASbGncvRdCfVFQ1+tQdEQiatB4rteeFgJgKaLqIvsQCnemduj+5GcHi45EmFUWVFhXh Agrrx/nIII9R3Oe7KGK5kAhSqLQMLNVUQixP/Ey8aQU16RRZydhlTfjOjLXRVMXfvX3PTu50ydi S9E5XV5ZhFU4lg9dEU85ZYdfr8uE0JXOuXKlaq0Sx6RtXI1DtDikR17uTcC/6H96KP7m4yAOpTX +fkwIZM6ypOO6Vx7mZ/IHJ/6B6DZzuKtEQUnj6GXJIX4rz8lPpou4/ptxMBHrDW20JRVvHE2Pxt otyK0RrlT853VbdM9MLCAAwqdmemEHg+XZ6m05oa/dQrVFk= X-Google-Smtp-Source: AGHT+IEugHIYcHEDUOXp58GXQB8o5prMjl8/tvIa3ET7IFiNu4ViKFsWk4lTQ15ZE884fzJ8YzuupA== X-Received: by 2002:a17:903:198e:b0:220:d272:534d with SMTP id d9443c01a7336-22368f92a4dmr206219935ad.22.1740968727919; Sun, 02 Mar 2025 18:25:27 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:1513:4f61:a4d3:b418]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-223504dc489sm66538975ad.159.2025.03.02.18.25.25 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sun, 02 Mar 2025 18:25:27 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Hillf Danton , Kairui Song , Sebastian Andrzej Siewior , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v10 11/19] zram: move post-processing target allocation Date: Mon, 3 Mar 2025 11:03:20 +0900 Message-ID: <20250303022425.285971-12-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.711.g2feabab25a-goog In-Reply-To: <20250303022425.285971-1-senozhatsky@chromium.org> References: <20250303022425.285971-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Allocate post-processing target in place_pp_slot(). This simplifies scan_slots_for_writeback() and scan_slots_for_recompress() loops because we don't need to track pps pointer state anymore. Previously we have to explicitly NULL the point if it has been added to a post-processing bucket or re-use previously allocated pointer otherwise and make sure we don't leak the memory in the end. We are also fine doing GFP_NOIO allocation, as post-processing can be called under memory pressure so we better pick as many slots as we can as soon as we can and start post-processing them, possibly saving the memory. Allocation failure there is not fatal, we will post-process whatever we put into the buckets on previous iterations. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 50 +++++++++++++++-------------------- 1 file changed, 22 insertions(+), 28 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index bb88b63d193b..f6e887f94b71 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -295,15 +295,24 @@ static void release_pp_ctl(struct zram *zram, struct = zram_pp_ctl *ctl) kfree(ctl); } =20 -static void place_pp_slot(struct zram *zram, struct zram_pp_ctl *ctl, - struct zram_pp_slot *pps) +static bool place_pp_slot(struct zram *zram, struct zram_pp_ctl *ctl, + u32 index) { - u32 idx; + struct zram_pp_slot *pps; + u32 bid; + + pps =3D kmalloc(sizeof(*pps), GFP_NOIO | __GFP_NOWARN); + if (!pps) + return false; =20 - idx =3D zram_get_obj_size(zram, pps->index) / PP_BUCKET_SIZE_RANGE; - list_add(&pps->entry, &ctl->pp_buckets[idx]); + INIT_LIST_HEAD(&pps->entry); + pps->index =3D index; + + bid =3D zram_get_obj_size(zram, pps->index) / PP_BUCKET_SIZE_RANGE; + list_add(&pps->entry, &ctl->pp_buckets[bid]); =20 zram_set_flag(zram, pps->index, ZRAM_PP_SLOT); + return true; } =20 static struct zram_pp_slot *select_pp_slot(struct zram_pp_ctl *ctl) @@ -737,15 +746,8 @@ static int scan_slots_for_writeback(struct zram *zram,= u32 mode, unsigned long index, struct zram_pp_ctl *ctl) { - struct zram_pp_slot *pps =3D NULL; - for (; nr_pages !=3D 0; index++, nr_pages--) { - if (!pps) - pps =3D kmalloc(sizeof(*pps), GFP_KERNEL); - if (!pps) - return -ENOMEM; - - INIT_LIST_HEAD(&pps->entry); + bool ok =3D true; =20 zram_slot_lock(zram, index); if (!zram_allocated(zram, index)) @@ -765,14 +767,13 @@ static int scan_slots_for_writeback(struct zram *zram= , u32 mode, !zram_test_flag(zram, index, ZRAM_INCOMPRESSIBLE)) goto next; =20 - pps->index =3D index; - place_pp_slot(zram, ctl, pps); - pps =3D NULL; + ok =3D place_pp_slot(zram, ctl, index); next: zram_slot_unlock(zram, index); + if (!ok) + break; } =20 - kfree(pps); return 0; } =20 @@ -1827,16 +1828,10 @@ static int scan_slots_for_recompress(struct zram *z= ram, u32 mode, u32 prio_max, struct zram_pp_ctl *ctl) { unsigned long nr_pages =3D zram->disksize >> PAGE_SHIFT; - struct zram_pp_slot *pps =3D NULL; unsigned long index; =20 for (index =3D 0; index < nr_pages; index++) { - if (!pps) - pps =3D kmalloc(sizeof(*pps), GFP_KERNEL); - if (!pps) - return -ENOMEM; - - INIT_LIST_HEAD(&pps->entry); + bool ok =3D true; =20 zram_slot_lock(zram, index); if (!zram_allocated(zram, index)) @@ -1859,14 +1854,13 @@ static int scan_slots_for_recompress(struct zram *z= ram, u32 mode, u32 prio_max, if (zram_get_priority(zram, index) + 1 >=3D prio_max) goto next; =20 - pps->index =3D index; - place_pp_slot(zram, ctl, pps); - pps =3D NULL; + ok =3D place_pp_slot(zram, ctl, index); next: zram_slot_unlock(zram, index); + if (!ok) + break; } =20 - kfree(pps); return 0; } =20 --=20 2.48.1.711.g2feabab25a-goog From nobody Tue Dec 16 19:24:32 2025 Received: from mail-pl1-f178.google.com (mail-pl1-f178.google.com [209.85.214.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2AB2D185E4A for ; Mon, 3 Mar 2025 02:25:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740968734; cv=none; b=D15rhqCzw9u6TEd5Nf2999g8IEnNEJxifKxRq+PvtOl4AOMxlF5XAYiDk9nm488PMA85MFQRKquPZv+RqR2lx9JEKvheAVNtH+sGPSoEgwf/FxdbgFo1G4ZjY+Y8vptq6JWI96A6zL/nKnl+xERdLVxBTxMpjhD14uRU8ZaxXuI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740968734; c=relaxed/simple; bh=rb8a56w971m7nD0c4uVo5Hdp0pePYf++4G7cHJu/a1Y=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=qZp+FDWpnNUlcd12asZxg7qLeX5TZWrGWEXf0SYRvf5P6esthhV7wy2jrMZnWGwRob8bL777a/XMp9tViCkrUdb9TZ69T6v3Y3bO13yElMVt2nsSjwoui0YRietFsVwAZBA84CcB7zWjkYbJYnC0H/aKzDPj5OtgKusZGnf1Og0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=YNB3VeBC; arc=none smtp.client-ip=209.85.214.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="YNB3VeBC" Received: by mail-pl1-f178.google.com with SMTP id d9443c01a7336-2234e4b079cso69136865ad.1 for ; Sun, 02 Mar 2025 18:25:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1740968732; x=1741573532; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=CYVJH5MaLfbxaKGZoDOncyDyqc70bzD7eQWV2O8vsj8=; b=YNB3VeBCk+y4uxx+HPEyBKh5GlTi3hmsXycUj/yFMw8+yO1LGFOSi1p3TBvdZFdiJm gqE/hk/JopkOqsPMdUQIgOB9wKuVHcUeylMEdqqEazb2u7pGwZ448rGF/cyZHBiiyVzD 5YihWvk6+q9YjDELmnet9G1zuCWGrCvlE9f2c= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740968732; x=1741573532; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=CYVJH5MaLfbxaKGZoDOncyDyqc70bzD7eQWV2O8vsj8=; b=JAA1x8o6CRqN8osL+kfur1AgduM8kxUAP06WfGCHfOAqs4N6/c2mLu6FJkcFdEYtYV OPsuJgup1a5XBiEqWRSlE5RaLpndxgbuTO1yG3s4CVzunjn/GjmhJ8B7ZCCU/a2pBnQF 8fkLlCqzWD/cqbA47Xf5BkGHIx2iRcTDTyXlmtp3Yhfw7U7og2zxioyEZwYOZnp80INX ICaAYjVTWokeCkDq09D2eprLe3SePPCJ2Mmn3O1sgC3EPazytfpjcepJvqoC/p1QW5Mg 44KWoZmIujnHujkZ3BKzaStc8cjfocRTkXiVnclGts3GfcSltOuupMF16B4gaCQchSKi qYhw== X-Forwarded-Encrypted: i=1; AJvYcCWkkAi8rp5kUBofD1PApWn0Dmls0sMp3MMjTEDYjbV0zKJsN81AKRIfG24hLKcVR7V9p/Kc24fTG6npKEM=@vger.kernel.org X-Gm-Message-State: AOJu0YzOyBvrBFa7YV2UWh8aBSfDGCHIY7X/VE2ni+a34ZZ82xkRRrSy eJaz4251egkAdDRy67zeE7742SPyDIH/M6bShDKStn+EITGGHVhgfJWArcwJuA== X-Gm-Gg: ASbGnctmga3H8prGOvwf+I+pNMK6Hq6idDgGHBJMNXgAc4k/tl6G6yxZSkJFgmDTvxV WTQ2tn7/5Urj5UP52fAkZnJhtd3vuNxQaEBLhcYKe7hX5JAfJgbasEb/iQBzD5NyEGRqSPMYcQk Kb5HrOMJZAKJcJbphttGcQ5ZzlSEE64G9KDPprjTv76gN/7e1HXIZ75jvPabGopHrkF6cQbSu8m ycuj0Fl4V2Y7IqmRl11OeOo/qH2rn8bahun+vT4uSgql/zVRIjPxAEog0seqL/4LNg5k7K+F4bW f7vPTSv39aF5rl7rPh/DfqaadL8AA3By5rm2BKGgxLcLk5s= X-Google-Smtp-Source: AGHT+IEgYxZUnegN30z6tSMQEsHJ1L6y7oPydeFduqm2ElX6isqJ4TV2BparuhraX3uF1fCfP0eGTw== X-Received: by 2002:a17:902:d2d0:b0:21f:6cb2:e949 with SMTP id d9443c01a7336-22369244fb0mr142389775ad.52.1740968732436; Sun, 02 Mar 2025 18:25:32 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:1513:4f61:a4d3:b418]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-223501fb004sm66940215ad.64.2025.03.02.18.25.30 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sun, 02 Mar 2025 18:25:32 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Hillf Danton , Kairui Song , Sebastian Andrzej Siewior , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v10 12/19] zsmalloc: rename pool lock Date: Mon, 3 Mar 2025 11:03:21 +0900 Message-ID: <20250303022425.285971-13-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.711.g2feabab25a-goog In-Reply-To: <20250303022425.285971-1-senozhatsky@chromium.org> References: <20250303022425.285971-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The old name comes from the times when the pool did not have compaction (defragmentation). Rename it to ->lock because these days it synchronizes not only migration. Reviewed-by: Yosry Ahmed Signed-off-by: Sergey Senozhatsky --- mm/zsmalloc.c | 38 +++++++++++++++++++------------------- 1 file changed, 19 insertions(+), 19 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 817626a351f8..1424ee73cbb5 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -18,7 +18,7 @@ /* * lock ordering: * page_lock - * pool->migrate_lock + * pool->lock * class->lock * zspage->lock */ @@ -223,8 +223,8 @@ struct zs_pool { #ifdef CONFIG_COMPACTION struct work_struct free_work; #endif - /* protect page/zspage migration */ - rwlock_t migrate_lock; + /* protect zspage migration/compaction */ + rwlock_t lock; atomic_t compaction_in_progress; }; =20 @@ -1206,7 +1206,7 @@ void *zs_map_object(struct zs_pool *pool, unsigned lo= ng handle, BUG_ON(in_interrupt()); =20 /* It guarantees it can get zspage from handle safely */ - read_lock(&pool->migrate_lock); + read_lock(&pool->lock); obj =3D handle_to_obj(handle); obj_to_location(obj, &zpdesc, &obj_idx); zspage =3D get_zspage(zpdesc); @@ -1218,7 +1218,7 @@ void *zs_map_object(struct zs_pool *pool, unsigned lo= ng handle, * which is smaller granularity. */ migrate_read_lock(zspage); - read_unlock(&pool->migrate_lock); + read_unlock(&pool->lock); =20 class =3D zspage_class(pool, zspage); off =3D offset_in_page(class->size * obj_idx); @@ -1450,16 +1450,16 @@ void zs_free(struct zs_pool *pool, unsigned long ha= ndle) return; =20 /* - * The pool->migrate_lock protects the race with zpage's migration + * The pool->lock protects the race with zpage's migration * so it's safe to get the page from handle. */ - read_lock(&pool->migrate_lock); + read_lock(&pool->lock); obj =3D handle_to_obj(handle); obj_to_zpdesc(obj, &f_zpdesc); zspage =3D get_zspage(f_zpdesc); class =3D zspage_class(pool, zspage); spin_lock(&class->lock); - read_unlock(&pool->migrate_lock); + read_unlock(&pool->lock); =20 class_stat_sub(class, ZS_OBJS_INUSE, 1); obj_free(class->size, obj); @@ -1796,7 +1796,7 @@ static int zs_page_migrate(struct page *newpage, stru= ct page *page, * The pool migrate_lock protects the race between zpage migration * and zs_free. */ - write_lock(&pool->migrate_lock); + write_lock(&pool->lock); class =3D zspage_class(pool, zspage); =20 /* @@ -1833,7 +1833,7 @@ static int zs_page_migrate(struct page *newpage, stru= ct page *page, * Since we complete the data copy and set up new zspage structure, * it's okay to release migration_lock. */ - write_unlock(&pool->migrate_lock); + write_unlock(&pool->lock); spin_unlock(&class->lock); migrate_write_unlock(zspage); =20 @@ -1956,7 +1956,7 @@ static unsigned long __zs_compact(struct zs_pool *poo= l, * protect the race between zpage migration and zs_free * as well as zpage allocation/free */ - write_lock(&pool->migrate_lock); + write_lock(&pool->lock); spin_lock(&class->lock); while (zs_can_compact(class)) { int fg; @@ -1983,14 +1983,14 @@ static unsigned long __zs_compact(struct zs_pool *p= ool, src_zspage =3D NULL; =20 if (get_fullness_group(class, dst_zspage) =3D=3D ZS_INUSE_RATIO_100 - || rwlock_is_contended(&pool->migrate_lock)) { + || rwlock_is_contended(&pool->lock)) { putback_zspage(class, dst_zspage); dst_zspage =3D NULL; =20 spin_unlock(&class->lock); - write_unlock(&pool->migrate_lock); + write_unlock(&pool->lock); cond_resched(); - write_lock(&pool->migrate_lock); + write_lock(&pool->lock); spin_lock(&class->lock); } } @@ -2002,7 +2002,7 @@ static unsigned long __zs_compact(struct zs_pool *poo= l, putback_zspage(class, dst_zspage); =20 spin_unlock(&class->lock); - write_unlock(&pool->migrate_lock); + write_unlock(&pool->lock); =20 return pages_freed; } @@ -2014,10 +2014,10 @@ unsigned long zs_compact(struct zs_pool *pool) unsigned long pages_freed =3D 0; =20 /* - * Pool compaction is performed under pool->migrate_lock so it is basical= ly + * Pool compaction is performed under pool->lock so it is basically * single-threaded. Having more than one thread in __zs_compact() - * will increase pool->migrate_lock contention, which will impact other - * zsmalloc operations that need pool->migrate_lock. + * will increase pool->lock contention, which will impact other + * zsmalloc operations that need pool->lock. */ if (atomic_xchg(&pool->compaction_in_progress, 1)) return 0; @@ -2139,7 +2139,7 @@ struct zs_pool *zs_create_pool(const char *name) return NULL; =20 init_deferred_free(pool); - rwlock_init(&pool->migrate_lock); + rwlock_init(&pool->lock); atomic_set(&pool->compaction_in_progress, 0); =20 pool->name =3D kstrdup(name, GFP_KERNEL); --=20 2.48.1.711.g2feabab25a-goog From nobody Tue Dec 16 19:24:32 2025 Received: from mail-pl1-f176.google.com (mail-pl1-f176.google.com [209.85.214.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5780C188CDB for ; Mon, 3 Mar 2025 02:25:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740968740; cv=none; b=nB2L19wmmr1aFe3HViIiZxfHnLbaoM7bWhYalGVAk5D6UK8jdZX4bPsgjq+zhZ3MU/Waedaxn6+sXgOgbUbSy86nltlLmyFRuoXr67dKXZljXVLh6BizrPZVHwsieINRH6qInqqSvnomcdgFg7AOjH/RvyaY6FbfRrV412E6xNM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740968740; c=relaxed/simple; bh=5e1Mw52dK5rWahT2ui1CD/SDhXblwaXUyFSdhkgPlkM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=jRRoG7jIQETymF0yRmQIouFJjL06MJ4024FIAmMXJz6PnrOv1VfhFg+BHkKtt14SSOIssHhu4pG+a9YoHiNgLl7j3A10DbqBQPLLSO4gJWhHclslerxqLTT5VLiJo0AUFEJ27ytgAdK+osoNK83xmMnNSZPlAFZSCxQIP7ynomI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=hjH/w9Ob; arc=none smtp.client-ip=209.85.214.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="hjH/w9Ob" Received: by mail-pl1-f176.google.com with SMTP id d9443c01a7336-22337bc9ac3so72886855ad.1 for ; Sun, 02 Mar 2025 18:25:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1740968738; x=1741573538; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Ua9rnaxl59hYuly6c376LLOYIZDnLCeuJzWW228A97g=; b=hjH/w9Obudy67Qyw2NUgMapFo9xoB48ZgwujFADcGoDeR/K5qhhYli0mOIvCTrHU97 DoGAiEs3XsiY6w4rO3UWGmOh5jftz300Iiu5ofiBITO7lTbZmFR1Sq/B/V2j8A1rFXGo t/BOeKnfxMGpmVVJ/yvOASKOgBMRXB5Nq15XI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740968738; x=1741573538; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Ua9rnaxl59hYuly6c376LLOYIZDnLCeuJzWW228A97g=; b=kqcepy4SfA/4zmAWEd194IVBUKo+rLUwjtYfGv0BBYx9YrAh/FuQTPHKqSESmeEBSZ RS/MLL3Wurv4N6W5+y8UHWacKqU1q27qCwOyGoAdOuXRmHohLhIkQUmNqO7P8ah7RNt6 JpD8GJlKSV6sEHJbgFEH7REAZZML9OgZYjTguUmMYZDMU2QNGbiPZfSzgoZ6vTttE99P U6g/qRAh2UL3xdams8dais71gjU2+AlbcR4rJm1fK2lTlrDqqWYA+9GC/0SNwLZ1HQtW 9vAWbHsL/h8++HDyLd139+ZatNQDyrNCMMNLfM0SybthxfAdvuE8DWt7tuqCUmjlOgx+ qH1g== X-Forwarded-Encrypted: i=1; AJvYcCUKss2rSLNveXxE0GtsbnGVlDbO7/4YcAkAP/7wGGmRAvpOnxoVugwQYfajC1JsIPz2FzW2fBwvJWMD4Wk=@vger.kernel.org X-Gm-Message-State: AOJu0YyPyHFwvSaTexfT5T9NhtQ7i4qGZ1/EKcIJ4AXGaYouR/ynTHof LXzrS9T678osu+zoYUlPiSWmEISRUfbfU8yIZUY8iGhLiliuQ3da39yQQnU4Pw== X-Gm-Gg: ASbGncuSKNBYwg0t3Fp1WpJW9ZpbhWYqHLp/Jeq+XnbJDb8pNnKwXhttjFaZuFcSkV9 I1KIKyr3aWKu3HOnr5Z7SMakvU5KWuF1q6EL91KZ5nf5xKAS8yTyT+uhnHski7T2nFrv3oGnYpG egkKSY8ahsOXZ4ycsQNKv3Ao+m/7lPwzlNsQKZHyBRarHoftNQ/5rWvOU50s+4qe7qp9hoV1ku5 +TiA2qeMrVw6nejL/sssuhlgF6HNTwj8DYR81c6oxuJmOouMG3e34iFLxMRoP40GusOLm66/v+K kbqe+C93g8uPq5hqGH3Z9JDg3q9X/R6xSG+FZ43wDvVfkCI= X-Google-Smtp-Source: AGHT+IHtZ5c68bsAtTNZLqPzKISnIUv2uwTEuuqfc15VikOP6YUn0971YwA/MQ1+Ofo0Wu55CvXwCA== X-Received: by 2002:a05:6a00:a91:b0:730:99cb:7c32 with SMTP id d2e1a72fcca58-734ac350410mr17588314b3a.7.1740968737528; Sun, 02 Mar 2025 18:25:37 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:1513:4f61:a4d3:b418]) by smtp.gmail.com with UTF8SMTPSA id d2e1a72fcca58-7356173aedfsm5377318b3a.95.2025.03.02.18.25.35 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sun, 02 Mar 2025 18:25:37 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Hillf Danton , Kairui Song , Sebastian Andrzej Siewior , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v10 13/19] zsmalloc: sleepable zspage reader-lock Date: Mon, 3 Mar 2025 11:03:22 +0900 Message-ID: <20250303022425.285971-14-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.711.g2feabab25a-goog In-Reply-To: <20250303022425.285971-1-senozhatsky@chromium.org> References: <20250303022425.285971-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In order to implement preemptible object mapping we need a zspage lock that satisfies several preconditions: - it should be reader-write type of a lock - it should be possible to hold it from any context, but also being preemptible if the context allows it - we never sleep while acquiring but can sleep while holding in read mode An rwsemaphore doesn't suffice, due to atomicity requirements, rwlock doesn't satisfy due to reader-preemptability requirement. It's also worth to mention, that per-zspage rwsem is a little too memory heavy (we can easily have double digits megabytes used only on rwsemaphores). Switch over from rwlock_t to a atomic_t-based implementation of a reader-writer semaphore that satisfies all of the preconditions. The spin-lock based zspage_lock is suggested by Hillf Danton. Suggested-by: Hillf Danton Signed-off-by: Sergey Senozhatsky --- mm/zsmalloc.c | 166 ++++++++++++++++++++++++++++++++++---------------- 1 file changed, 114 insertions(+), 52 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 1424ee73cbb5..afbd72363731 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -257,6 +257,15 @@ static inline void free_zpdesc(struct zpdesc *zpdesc) __free_page(page); } =20 +#define ZS_PAGE_UNLOCKED 0 +#define ZS_PAGE_WRLOCKED -1 + +struct zspage_lock { + spinlock_t lock; + int cnt; + struct lockdep_map dep_map; +}; + struct zspage { struct { unsigned int huge:HUGE_BITS; @@ -269,7 +278,7 @@ struct zspage { struct zpdesc *first_zpdesc; struct list_head list; /* fullness list */ struct zs_pool *pool; - rwlock_t lock; + struct zspage_lock zsl; }; =20 struct mapping_area { @@ -279,6 +288,84 @@ struct mapping_area { enum zs_mapmode vm_mm; /* mapping mode */ }; =20 +static void zspage_lock_init(struct zspage *zspage) +{ + static struct lock_class_key __key; + struct zspage_lock *zsl =3D &zspage->zsl; + + lockdep_init_map(&zsl->dep_map, "zspage->lock", &__key, 0); + spin_lock_init(&zsl->lock); + zsl->cnt =3D ZS_PAGE_UNLOCKED; +} + +/* + * The zspage lock can be held from atomic contexts, but it needs to remain + * preemptible when held for reading because it remains held outside of th= ose + * atomic contexts, otherwise we unnecessarily lose preemptibility. + * + * To achieve this, the following rules are enforced on readers and writer= s: + * + * - Writers are blocked by both writers and readers, while readers are on= ly + * blocked by writers (i.e. normal rwlock semantics). + * + * - Writers are always atomic (to allow readers to spin waiting for them). + * + * - Writers always use trylock (as the lock may be held be sleeping reade= rs). + * + * - Readers may spin on the lock (as they can only wait for atomic writer= s). + * + * - Readers may sleep while holding the lock (as writes only use trylock). + */ +static void zspage_read_lock(struct zspage *zspage) +{ + struct zspage_lock *zsl =3D &zspage->zsl; + + rwsem_acquire_read(&zsl->dep_map, 0, 0, _RET_IP_); + + spin_lock(&zsl->lock); + zsl->cnt++; + spin_unlock(&zsl->lock); + + lock_acquired(&zsl->dep_map, _RET_IP_); +} + +static void zspage_read_unlock(struct zspage *zspage) +{ + struct zspage_lock *zsl =3D &zspage->zsl; + + rwsem_release(&zsl->dep_map, _RET_IP_); + + spin_lock(&zsl->lock); + zsl->cnt--; + spin_unlock(&zsl->lock); +} + +static __must_check bool zspage_write_trylock(struct zspage *zspage) +{ + struct zspage_lock *zsl =3D &zspage->zsl; + + spin_lock(&zsl->lock); + if (zsl->cnt =3D=3D ZS_PAGE_UNLOCKED) { + zsl->cnt =3D ZS_PAGE_WRLOCKED; + rwsem_acquire(&zsl->dep_map, 0, 1, _RET_IP_); + lock_acquired(&zsl->dep_map, _RET_IP_); + return true; + } + + spin_unlock(&zsl->lock); + return false; +} + +static void zspage_write_unlock(struct zspage *zspage) +{ + struct zspage_lock *zsl =3D &zspage->zsl; + + rwsem_release(&zsl->dep_map, _RET_IP_); + + zsl->cnt =3D ZS_PAGE_UNLOCKED; + spin_unlock(&zsl->lock); +} + /* huge object: pages_per_zspage =3D=3D 1 && maxobj_per_zspage =3D=3D 1 */ static void SetZsHugePage(struct zspage *zspage) { @@ -290,12 +377,6 @@ static bool ZsHugePage(struct zspage *zspage) return zspage->huge; } =20 -static void migrate_lock_init(struct zspage *zspage); -static void migrate_read_lock(struct zspage *zspage); -static void migrate_read_unlock(struct zspage *zspage); -static void migrate_write_lock(struct zspage *zspage); -static void migrate_write_unlock(struct zspage *zspage); - #ifdef CONFIG_COMPACTION static void kick_deferred_free(struct zs_pool *pool); static void init_deferred_free(struct zs_pool *pool); @@ -992,7 +1073,9 @@ static struct zspage *alloc_zspage(struct zs_pool *poo= l, return NULL; =20 zspage->magic =3D ZSPAGE_MAGIC; - migrate_lock_init(zspage); + zspage->pool =3D pool; + zspage->class =3D class->index; + zspage_lock_init(zspage); =20 for (i =3D 0; i < class->pages_per_zspage; i++) { struct zpdesc *zpdesc; @@ -1015,8 +1098,6 @@ static struct zspage *alloc_zspage(struct zs_pool *po= ol, =20 create_page_chain(class, zspage, zpdescs); init_zspage(class, zspage); - zspage->pool =3D pool; - zspage->class =3D class->index; =20 return zspage; } @@ -1217,7 +1298,7 @@ void *zs_map_object(struct zs_pool *pool, unsigned lo= ng handle, * zs_unmap_object API so delegate the locking from class to zspage * which is smaller granularity. */ - migrate_read_lock(zspage); + zspage_read_lock(zspage); read_unlock(&pool->lock); =20 class =3D zspage_class(pool, zspage); @@ -1277,7 +1358,7 @@ void zs_unmap_object(struct zs_pool *pool, unsigned l= ong handle) } local_unlock(&zs_map_area.lock); =20 - migrate_read_unlock(zspage); + zspage_read_unlock(zspage); } EXPORT_SYMBOL_GPL(zs_unmap_object); =20 @@ -1671,18 +1752,18 @@ static void lock_zspage(struct zspage *zspage) /* * Pages we haven't locked yet can be migrated off the list while we're * trying to lock them, so we need to be careful and only attempt to - * lock each page under migrate_read_lock(). Otherwise, the page we lock + * lock each page under zspage_read_lock(). Otherwise, the page we lock * may no longer belong to the zspage. This means that we may wait for * the wrong page to unlock, so we must take a reference to the page - * prior to waiting for it to unlock outside migrate_read_lock(). + * prior to waiting for it to unlock outside zspage_read_lock(). */ while (1) { - migrate_read_lock(zspage); + zspage_read_lock(zspage); zpdesc =3D get_first_zpdesc(zspage); if (zpdesc_trylock(zpdesc)) break; zpdesc_get(zpdesc); - migrate_read_unlock(zspage); + zspage_read_unlock(zspage); zpdesc_wait_locked(zpdesc); zpdesc_put(zpdesc); } @@ -1693,41 +1774,16 @@ static void lock_zspage(struct zspage *zspage) curr_zpdesc =3D zpdesc; } else { zpdesc_get(zpdesc); - migrate_read_unlock(zspage); + zspage_read_unlock(zspage); zpdesc_wait_locked(zpdesc); zpdesc_put(zpdesc); - migrate_read_lock(zspage); + zspage_read_lock(zspage); } } - migrate_read_unlock(zspage); + zspage_read_unlock(zspage); } #endif /* CONFIG_COMPACTION */ =20 -static void migrate_lock_init(struct zspage *zspage) -{ - rwlock_init(&zspage->lock); -} - -static void migrate_read_lock(struct zspage *zspage) __acquires(&zspage->l= ock) -{ - read_lock(&zspage->lock); -} - -static void migrate_read_unlock(struct zspage *zspage) __releases(&zspage-= >lock) -{ - read_unlock(&zspage->lock); -} - -static void migrate_write_lock(struct zspage *zspage) -{ - write_lock(&zspage->lock); -} - -static void migrate_write_unlock(struct zspage *zspage) -{ - write_unlock(&zspage->lock); -} - #ifdef CONFIG_COMPACTION =20 static const struct movable_operations zsmalloc_mops; @@ -1785,9 +1841,6 @@ static int zs_page_migrate(struct page *newpage, stru= ct page *page, =20 VM_BUG_ON_PAGE(!zpdesc_is_isolated(zpdesc), zpdesc_page(zpdesc)); =20 - /* We're committed, tell the world that this is a Zsmalloc page. */ - __zpdesc_set_zsmalloc(newzpdesc); - /* The page is locked, so this pointer must remain valid */ zspage =3D get_zspage(zpdesc); pool =3D zspage->pool; @@ -1803,8 +1856,15 @@ static int zs_page_migrate(struct page *newpage, str= uct page *page, * the class lock protects zpage alloc/free in the zspage. */ spin_lock(&class->lock); - /* the migrate_write_lock protects zpage access via zs_map_object */ - migrate_write_lock(zspage); + /* the zspage write_lock protects zpage access via zs_map_object */ + if (!zspage_write_trylock(zspage)) { + spin_unlock(&class->lock); + write_unlock(&pool->lock); + return -EINVAL; + } + + /* We're committed, tell the world that this is a Zsmalloc page. */ + __zpdesc_set_zsmalloc(newzpdesc); =20 offset =3D get_first_obj_offset(zpdesc); s_addr =3D kmap_local_zpdesc(zpdesc); @@ -1835,7 +1895,7 @@ static int zs_page_migrate(struct page *newpage, stru= ct page *page, */ write_unlock(&pool->lock); spin_unlock(&class->lock); - migrate_write_unlock(zspage); + zspage_write_unlock(zspage); =20 zpdesc_get(newzpdesc); if (zpdesc_zone(newzpdesc) !=3D zpdesc_zone(zpdesc)) { @@ -1971,9 +2031,11 @@ static unsigned long __zs_compact(struct zs_pool *po= ol, if (!src_zspage) break; =20 - migrate_write_lock(src_zspage); + if (!zspage_write_trylock(src_zspage)) + break; + migrate_zspage(pool, src_zspage, dst_zspage); - migrate_write_unlock(src_zspage); + zspage_write_unlock(src_zspage); =20 fg =3D putback_zspage(class, src_zspage); if (fg =3D=3D ZS_INUSE_RATIO_0) { --=20 2.48.1.711.g2feabab25a-goog From nobody Tue Dec 16 19:24:32 2025 Received: from mail-pl1-f175.google.com (mail-pl1-f175.google.com [209.85.214.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7E01D1E51F5 for ; Mon, 3 Mar 2025 02:25:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.175 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740968745; cv=none; b=mRTk6hTCnL5OsOcGmrOozuUhAR32DWdELiiZKdWpbPrBlst49nz4Z7ZznoLDKoGlI0M4bReswrO6wKDArXkndJemtMa80YMnXiYSeyv0EbF3aCyBKkrIZBlTmRVzLJnGP7FogOW5D2l+9BSbFyoa1OEUkKYske2IYu8aEH5jhiU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740968745; c=relaxed/simple; bh=3iFieOUYn4iqBiRwGHIpWNqgK/MF7aWW1jnWLFdMd18=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=G0yKbqTetdU/TtibGzyI2QDTMFfuhZAF8QOF0wKu5G+/alHUYNneTfR0TEQ/0m+UKKBfzuxATfR8mUg55L3h0oVtKNgOmyP1aU+lmxi8ecvuvhBCCxfhwLDi32i0ZZxE553cf3xqiysEVqRua+wNDXvi5aXZngs/fryMoMla6ug= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=Q9hyLQMB; arc=none smtp.client-ip=209.85.214.175 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="Q9hyLQMB" Received: by mail-pl1-f175.google.com with SMTP id d9443c01a7336-223a3c035c9so9000995ad.1 for ; Sun, 02 Mar 2025 18:25:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1740968743; x=1741573543; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=LmmGIMp09Xa9+ajjYrlkAeQFbd0l0inskGKz/hBM/Wo=; b=Q9hyLQMBz2vaE4DoH0SR7wV8R1Gvuqc9ctvxkhjuQ7vT1v5TI23ZFUcpOky7dKmO1y RBWeBBBG9vNAUmmyiCcFc3G4F2liQbeuH0rkXI3+EqPU69/E6vSmHv/zwD7ioc5xhO4j GS7AnxPr4dEdqtF94y6wZzf0Wm9U2IfLsplUw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740968743; x=1741573543; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=LmmGIMp09Xa9+ajjYrlkAeQFbd0l0inskGKz/hBM/Wo=; b=RKzg3X5OlMa6RfPp/I4jbKN01jT/Fhn83YznkhlixIDKD21PMgTW+WCOUtE73NMm2L LBNvzNCZmR0S20V6QL3aJdm79VdS9MAsc3C3sTZRIlejRYnFMvGT7qYSCF+eVJKN10MT PI71oB8amirVEZE0738IH8Qz4oebVmFhlMGembct07lLg03aK118w4/WJydkU5bT32+x qi6MNcEVNBPHHj9Dyf2seiQvRp589I0QpBQ68Vf1rr3oh5NMUoEp/M6UAYbs3UOSrysY 35oQrHpugie4ebfPOq5JZxSVNVFb3+cGKqEDqZzk5twJuWHTbbTWFffmYWb+alwcDvCT eG6Q== X-Forwarded-Encrypted: i=1; AJvYcCXVxnRe+BOl1PnMBoCr8pl2y7bQmoegxQ03hYJ9Dv1Ak7MipulZbt3fWB8mF1VKxHOZq57XDLaB4BUa1eE=@vger.kernel.org X-Gm-Message-State: AOJu0YyQkiYh0UzSeY9z6h7WXBtKFSIwu80PrRNSvHaqgC5iHEMNGe70 HKoa0hu4P+LHT1JbG17SFzA9We8pGolKSDTgEEjJZqT9pjNtqKwF6wwPM2KwHQ== X-Gm-Gg: ASbGnctZnwzyoBoYE8DujTXsHnxSJgIWzj7Erbr2DAYbNEJvhkMvs8mYP+ghvBFXhlU ZNHS6TBEoDtP9jGnuG7sI1n4VvZcFQTPzyc9sr2uR3G5wpJSrBVnRWtW39a+kc50ejyfq+k9y1r GWrpla/qBx23SJcYw3+yGKSmz7UsexuFHfD58oy3UrdK+rPkLCDlhY3y2zraPi7E94tNrTa4Pyg Rchgv+Uzyj6Y2Vpk/ZdAciA7uZniRSQlAuMC0w6K3PAr/4ZsfaA7jFQsup5cW82L273quOxk7SD LdznxTvuTgJjxPHowukYvj0k0bMN1IFc0Aics31RheoVvdY= X-Google-Smtp-Source: AGHT+IE+JByFLuBtDUrc5bcjQDL8kXzJ75Xakggo0iV8YwzjqWoQMMXMd+ywhB7PLbJgopN3ZGVCPQ== X-Received: by 2002:a17:902:e84e:b0:21f:6f33:f96 with SMTP id d9443c01a7336-2234a188d50mr267570275ad.6.1740968742708; Sun, 02 Mar 2025 18:25:42 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:1513:4f61:a4d3:b418]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-22350533a10sm67008045ad.247.2025.03.02.18.25.40 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sun, 02 Mar 2025 18:25:42 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Hillf Danton , Kairui Song , Sebastian Andrzej Siewior , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v10 14/19] zsmalloc: introduce new object mapping API Date: Mon, 3 Mar 2025 11:03:23 +0900 Message-ID: <20250303022425.285971-15-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.711.g2feabab25a-goog In-Reply-To: <20250303022425.285971-1-senozhatsky@chromium.org> References: <20250303022425.285971-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Current object mapping API is a little cumbersome. First, it's inconsistent, sometimes it returns with page-faults disabled and sometimes with page-faults enabled. Second, and most importantly, it enforces atomicity restrictions on its users. zs_map_object() has to return a liner object address which is not always possible because some objects span multiple physical (non-contiguous) pages. For such objects zsmalloc uses a per-CPU buffer to which object's data is copied before a pointer to that per-CPU buffer is returned back to the caller. This leads to another, final, issue - extra memcpy(). Since the caller gets a pointer to per-CPU buffer it can memcpy() data only to that buffer, and during zs_unmap_object() zsmalloc will memcpy() from that per-CPU buffer to physical pages that object in question spans across. New API splits functions by access mode: - zs_obj_read_begin(handle, local_copy) Returns a pointer to handle memory. For objects that span two physical pages a local_copy buffer is used to store object's data before the address is returned to the caller. Otherwise the object's page is kmap_local mapped directly. - zs_obj_read_end(handle, buf) Unmaps the page if it was kmap_local mapped by zs_obj_read_begin(). - zs_obj_write(handle, buf, len) Copies len-bytes from compression buffer to handle memory (takes care of objects that span two pages). This does not need any additional (e.g. per-CPU) buffers and writes the data directly to zsmalloc pool pages. In terms of performance, on a synthetic and completely reproducible test that allocates fixed number of objects of fixed sizes and iterates over those objects, first mapping in RO then in RW mode: OLD API =3D=3D=3D=3D=3D=3D=3D 3 first results out of 10 369,205,778 instructions # 0.80 insn per cycle 40,467,926 branches # 113.732 M/sec 369,002,122 instructions # 0.62 insn per cycle 40,426,145 branches # 189.361 M/sec 369,036,706 instructions # 0.63 insn per cycle 40,430,860 branches # 204.105 M/sec [..] NEW API =3D=3D=3D=3D=3D=3D=3D 3 first results out of 10 265,799,293 instructions # 0.51 insn per cycle 29,834,567 branches # 170.281 M/sec 265,765,970 instructions # 0.55 insn per cycle 29,829,019 branches # 161.602 M/sec 265,764,702 instructions # 0.51 insn per cycle 29,828,015 branches # 189.677 M/sec [..] T-test on all 10 runs =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Difference at 95.0% confidence -1.03219e+08 +/- 55308.7 -27.9705% +/- 0.0149878% (Student's t, pooled s =3D 58864.4) The old API will stay around until the remaining users switch to the new one. After that we'll also remove zsmalloc per-CPU buffer and CPU hotplug handling. The split of map(RO) and map(WO) into read_{begin/end}/write is suggested by Yosry Ahmed. Suggested-by: Yosry Ahmed Signed-off-by: Sergey Senozhatsky Reviewed-by: Yosry Ahmed --- include/linux/zsmalloc.h | 8 +++ mm/zsmalloc.c | 125 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 133 insertions(+) diff --git a/include/linux/zsmalloc.h b/include/linux/zsmalloc.h index a48cd0ffe57d..7d70983cf398 100644 --- a/include/linux/zsmalloc.h +++ b/include/linux/zsmalloc.h @@ -58,4 +58,12 @@ unsigned long zs_compact(struct zs_pool *pool); unsigned int zs_lookup_class_index(struct zs_pool *pool, unsigned int size= ); =20 void zs_pool_stats(struct zs_pool *pool, struct zs_pool_stats *stats); + +void *zs_obj_read_begin(struct zs_pool *pool, unsigned long handle, + void *local_copy); +void zs_obj_read_end(struct zs_pool *pool, unsigned long handle, + void *handle_mem); +void zs_obj_write(struct zs_pool *pool, unsigned long handle, + void *handle_mem, size_t mem_len); + #endif diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index afbd72363731..7566070729ee 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -1362,6 +1362,131 @@ void zs_unmap_object(struct zs_pool *pool, unsigned= long handle) } EXPORT_SYMBOL_GPL(zs_unmap_object); =20 +void *zs_obj_read_begin(struct zs_pool *pool, unsigned long handle, + void *local_copy) +{ + struct zspage *zspage; + struct zpdesc *zpdesc; + unsigned long obj, off; + unsigned int obj_idx; + struct size_class *class; + void *addr; + + /* Guarantee we can get zspage from handle safely */ + read_lock(&pool->lock); + obj =3D handle_to_obj(handle); + obj_to_location(obj, &zpdesc, &obj_idx); + zspage =3D get_zspage(zpdesc); + + /* Make sure migration doesn't move any pages in this zspage */ + zspage_read_lock(zspage); + read_unlock(&pool->lock); + + class =3D zspage_class(pool, zspage); + off =3D offset_in_page(class->size * obj_idx); + + if (off + class->size <=3D PAGE_SIZE) { + /* this object is contained entirely within a page */ + addr =3D kmap_local_zpdesc(zpdesc); + addr +=3D off; + } else { + size_t sizes[2]; + + /* this object spans two pages */ + sizes[0] =3D PAGE_SIZE - off; + sizes[1] =3D class->size - sizes[0]; + addr =3D local_copy; + + memcpy_from_page(addr, zpdesc_page(zpdesc), + off, sizes[0]); + zpdesc =3D get_next_zpdesc(zpdesc); + memcpy_from_page(addr + sizes[0], + zpdesc_page(zpdesc), + 0, sizes[1]); + } + + if (!ZsHugePage(zspage)) + addr +=3D ZS_HANDLE_SIZE; + + return addr; +} +EXPORT_SYMBOL_GPL(zs_obj_read_begin); + +void zs_obj_read_end(struct zs_pool *pool, unsigned long handle, + void *handle_mem) +{ + struct zspage *zspage; + struct zpdesc *zpdesc; + unsigned long obj, off; + unsigned int obj_idx; + struct size_class *class; + + obj =3D handle_to_obj(handle); + obj_to_location(obj, &zpdesc, &obj_idx); + zspage =3D get_zspage(zpdesc); + class =3D zspage_class(pool, zspage); + off =3D offset_in_page(class->size * obj_idx); + + if (off + class->size <=3D PAGE_SIZE) { + if (!ZsHugePage(zspage)) + off +=3D ZS_HANDLE_SIZE; + handle_mem -=3D off; + kunmap_local(handle_mem); + } + + zspage_read_unlock(zspage); +} +EXPORT_SYMBOL_GPL(zs_obj_read_end); + +void zs_obj_write(struct zs_pool *pool, unsigned long handle, + void *handle_mem, size_t mem_len) +{ + struct zspage *zspage; + struct zpdesc *zpdesc; + unsigned long obj, off; + unsigned int obj_idx; + struct size_class *class; + + /* Guarantee we can get zspage from handle safely */ + read_lock(&pool->lock); + obj =3D handle_to_obj(handle); + obj_to_location(obj, &zpdesc, &obj_idx); + zspage =3D get_zspage(zpdesc); + + /* Make sure migration doesn't move any pages in this zspage */ + zspage_read_lock(zspage); + read_unlock(&pool->lock); + + class =3D zspage_class(pool, zspage); + off =3D offset_in_page(class->size * obj_idx); + + if (off + class->size <=3D PAGE_SIZE) { + /* this object is contained entirely within a page */ + void *dst =3D kmap_local_zpdesc(zpdesc); + + if (!ZsHugePage(zspage)) + off +=3D ZS_HANDLE_SIZE; + memcpy(dst + off, handle_mem, mem_len); + kunmap_local(dst); + } else { + /* this object spans two pages */ + size_t sizes[2]; + + off +=3D ZS_HANDLE_SIZE; + sizes[0] =3D PAGE_SIZE - off; + sizes[1] =3D mem_len - sizes[0]; + + memcpy_to_page(zpdesc_page(zpdesc), off, + handle_mem, sizes[0]); + zpdesc =3D get_next_zpdesc(zpdesc); + memcpy_to_page(zpdesc_page(zpdesc), 0, + handle_mem + sizes[0], sizes[1]); + } + + zspage_read_unlock(zspage); +} +EXPORT_SYMBOL_GPL(zs_obj_write); + /** * zs_huge_class_size() - Returns the size (in bytes) of the first huge * zsmalloc &size_class. --=20 2.48.1.711.g2feabab25a-goog From nobody Tue Dec 16 19:24:32 2025 Received: from mail-pj1-f48.google.com (mail-pj1-f48.google.com [209.85.216.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 028AF1E5B7A for ; Mon, 3 Mar 2025 02:25:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.48 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740968750; cv=none; b=ONgOYrNIloy1oZ4c3DvER7BzyFGEZwv9Rs8GLlBqwaRoBax8xCnWjaD3XjLfojevJ2EliFNKRrxPMnpRcATzW0K53tXYwvcC2ZwOnjhj2tssZG5418S2ZQcsrc2gUwJj92PlLofm/EH5ieMRPwa0O8n7rArYEs0MywliX3+QQCM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740968750; c=relaxed/simple; bh=wcSr27N3MI4dvCoc9p6MifR/HaNjuWH8mKzrev84nnk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=D2j8gPiT7IWPx7pNpQUFHVNA2jysndrN5/T/7uJrgqBJnBDbugoKBhkTZT++jRc1h6HT/jOGPuvKsPy1V/isHZ1TfTKzD+ti/HKQsD+oSPFbfso8Bgr/LllNHcI6pu5QD+NsWb2N2fRS3TiExs7Y4HN/jjBradU5ncdXzwxqcLw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=lRfVeWDz; arc=none smtp.client-ip=209.85.216.48 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="lRfVeWDz" Received: by mail-pj1-f48.google.com with SMTP id 98e67ed59e1d1-2fe96dd93b4so7159852a91.0 for ; Sun, 02 Mar 2025 18:25:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1740968748; x=1741573548; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=T67dKHYswiGrddgSZiZdCiBHKq1q/ZI+uUwWC6eNCTs=; b=lRfVeWDzWDD4EbRAf7thShz7Gr0KkoRIZ3+aiuFPzVGqi2pJ6y0cceU1LGxzDgJH5o Bgp5cvwA4lMK9+7K0YdybnfYSIMQlvv6DZIZBdh8ryZc69snpCw9g4oHGYhZMuxLarsS DY27LDSZBGHEycd6yWGyEseLXfNp9YegQtRoc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740968748; x=1741573548; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=T67dKHYswiGrddgSZiZdCiBHKq1q/ZI+uUwWC6eNCTs=; b=Uccg71/o24H8FippjYbFus5Y8G6OxgQtvzTJbsJ3MxoeUJddjP3Kzp2duOEKHAz41x Ucu1qfcEm7Ig1/pY71SsrWrurIlUlhsMoHaFcV3pad3pO6E3YqdopIzYO3z3Iu68MTbl XC7H4H3JRTVmxd2fKaX3yAlBE6T8tkdx9gXla7NdvXx/1dMaABFedjFQWXL2MzhILh4y qLnwv6n8ClRgqCW3pR0CgXos+9H8oJaB1ujmyARJc0nIsxFxBJK0WQYN7fuJTVVUwZxu 9hbBxPCHmMYXM4WYYN2kCgGgHi/drP9EfEQUYhtED7LZ3B8IWY5sgv5n8A3Xov1tyV5R HqbQ== X-Forwarded-Encrypted: i=1; AJvYcCUBQqsRIME2UApEJ8l8JNpuY1Pmj33Xfyzu0y1HI9Eu6D4rT3ymDDhPILPbCfob8+MIY2omKXDPCfa3PaY=@vger.kernel.org X-Gm-Message-State: AOJu0YyWbFvhWcdbKgE+SROi6X9StVQ4KNDxC8M2YQ3OvH7eJ88b3xn1 i455KKCu2u+gNIdkKNh3myGVzvMlN3d+vhs3hgd7zPQgVb5hXJsIJ0SC1SgN1w== X-Gm-Gg: ASbGnctmY5m+Gf3xPxjpdgX3QWs7WTJNLlmbnU9VKUo7Dc9oqcXX5swBKj+1Snc13Q8 8JhTy0TM4LpjQtwG8DCxLg49PgZfRV1GViZ8jAjSdDgg33jAyWrd4RmQrH/hOGZvs5iSjnW/KjX rl/0ACZcV//hiLIcXsRW8EZ+MGVYySrZvYlZm/3atiWGiffOWBBP1LZHI5SJ5TwjIUxQSre/RmM +tOT6+12z3urIDd1gGjaLRx8tFchVn82L59EJTqhkbpcpd7SBPPuaYLb0AWjRF6jARc/O1t/uPY f0eUGNyFkLoYWC/JNrIIm3pS24tbGMYTcpg0d80rA6Ka9G8= X-Google-Smtp-Source: AGHT+IH55gKbbFLH/4p1SdtWLi5VFnXEFbXHma9n71ec/lgw7BNRJwBiDWwI7fYS8F4jPBgKdgrzew== X-Received: by 2002:a05:6a21:6e03:b0:1f0:e2e6:f41e with SMTP id adf61e73a8af0-1f2f3cee5b8mr17138562637.8.1740968748420; Sun, 02 Mar 2025 18:25:48 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:1513:4f61:a4d3:b418]) by smtp.gmail.com with UTF8SMTPSA id d2e1a72fcca58-7364002f247sm2421551b3a.90.2025.03.02.18.25.46 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sun, 02 Mar 2025 18:25:48 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Hillf Danton , Kairui Song , Sebastian Andrzej Siewior , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v10 15/19] zram: switch to new zsmalloc object mapping API Date: Mon, 3 Mar 2025 11:03:24 +0900 Message-ID: <20250303022425.285971-16-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.711.g2feabab25a-goog In-Reply-To: <20250303022425.285971-1-senozhatsky@chromium.org> References: <20250303022425.285971-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Use new read/write zsmalloc object API. For cases when RO mapped object spans two physical pages (requires temp buffer) compression streams now carry around one extra physical page. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zcomp.c | 4 +++- drivers/block/zram/zcomp.h | 2 ++ drivers/block/zram/zram_drv.c | 28 ++++++++++------------------ 3 files changed, 15 insertions(+), 19 deletions(-) diff --git a/drivers/block/zram/zcomp.c b/drivers/block/zram/zcomp.c index cfdde2e0748a..a1d627054bb1 100644 --- a/drivers/block/zram/zcomp.c +++ b/drivers/block/zram/zcomp.c @@ -45,6 +45,7 @@ static const struct zcomp_ops *backends[] =3D { static void zcomp_strm_free(struct zcomp *comp, struct zcomp_strm *zstrm) { comp->ops->destroy_ctx(&zstrm->ctx); + vfree(zstrm->local_copy); vfree(zstrm->buffer); zstrm->buffer =3D NULL; } @@ -57,12 +58,13 @@ static int zcomp_strm_init(struct zcomp *comp, struct z= comp_strm *zstrm) if (ret) return ret; =20 + zstrm->local_copy =3D vzalloc(PAGE_SIZE); /* * allocate 2 pages. 1 for compressed data, plus 1 extra for the * case when compressed size is larger than the original one */ zstrm->buffer =3D vzalloc(2 * PAGE_SIZE); - if (!zstrm->buffer) { + if (!zstrm->buffer || !zstrm->local_copy) { zcomp_strm_free(comp, zstrm); return -ENOMEM; } diff --git a/drivers/block/zram/zcomp.h b/drivers/block/zram/zcomp.h index 23b8236b9090..25339ed1e07e 100644 --- a/drivers/block/zram/zcomp.h +++ b/drivers/block/zram/zcomp.h @@ -34,6 +34,8 @@ struct zcomp_strm { struct mutex lock; /* compression buffer */ void *buffer; + /* local copy of handle memory */ + void *local_copy; struct zcomp_ctx ctx; }; =20 diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index f6e887f94b71..62aef12417a4 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1561,11 +1561,11 @@ static int read_incompressible_page(struct zram *zr= am, struct page *page, void *src, *dst; =20 handle =3D zram_get_handle(zram, index); - src =3D zs_map_object(zram->mem_pool, handle, ZS_MM_RO); + src =3D zs_obj_read_begin(zram->mem_pool, handle, NULL); dst =3D kmap_local_page(page); copy_page(dst, src); kunmap_local(dst); - zs_unmap_object(zram->mem_pool, handle); + zs_obj_read_end(zram->mem_pool, handle, src); =20 return 0; } @@ -1583,11 +1583,11 @@ static int read_compressed_page(struct zram *zram, = struct page *page, u32 index) prio =3D zram_get_priority(zram, index); =20 zstrm =3D zcomp_stream_get(zram->comps[prio]); - src =3D zs_map_object(zram->mem_pool, handle, ZS_MM_RO); + src =3D zs_obj_read_begin(zram->mem_pool, handle, zstrm->local_copy); dst =3D kmap_local_page(page); ret =3D zcomp_decompress(zram->comps[prio], zstrm, src, size, dst); kunmap_local(dst); - zs_unmap_object(zram->mem_pool, handle); + zs_obj_read_end(zram->mem_pool, handle, src); zcomp_stream_put(zstrm); =20 return ret; @@ -1683,7 +1683,7 @@ static int write_incompressible_page(struct zram *zra= m, struct page *page, u32 index) { unsigned long handle; - void *src, *dst; + void *src; =20 /* * This function is called from preemptible context so we don't need @@ -1701,11 +1701,9 @@ static int write_incompressible_page(struct zram *zr= am, struct page *page, return -ENOMEM; } =20 - dst =3D zs_map_object(zram->mem_pool, handle, ZS_MM_WO); src =3D kmap_local_page(page); - memcpy(dst, src, PAGE_SIZE); + zs_obj_write(zram->mem_pool, handle, src, PAGE_SIZE); kunmap_local(src); - zs_unmap_object(zram->mem_pool, handle); =20 zram_slot_lock(zram, index); zram_set_flag(zram, index, ZRAM_HUGE); @@ -1726,7 +1724,7 @@ static int zram_write_page(struct zram *zram, struct = page *page, u32 index) int ret =3D 0; unsigned long handle; unsigned int comp_len; - void *dst, *mem; + void *mem; struct zcomp_strm *zstrm; unsigned long element; bool same_filled; @@ -1773,11 +1771,8 @@ static int zram_write_page(struct zram *zram, struct= page *page, u32 index) return -ENOMEM; } =20 - dst =3D zs_map_object(zram->mem_pool, handle, ZS_MM_WO); - - memcpy(dst, zstrm->buffer, comp_len); + zs_obj_write(zram->mem_pool, handle, zstrm->buffer, comp_len); zcomp_stream_put(zstrm); - zs_unmap_object(zram->mem_pool, handle); =20 zram_slot_lock(zram, index); zram_set_handle(zram, index, handle); @@ -1882,7 +1877,7 @@ static int recompress_slot(struct zram *zram, u32 ind= ex, struct page *page, unsigned int comp_len_new; unsigned int class_index_old; unsigned int class_index_new; - void *src, *dst; + void *src; int ret =3D 0; =20 handle_old =3D zram_get_handle(zram, index); @@ -1993,12 +1988,9 @@ static int recompress_slot(struct zram *zram, u32 in= dex, struct page *page, return PTR_ERR((void *)handle_new); } =20 - dst =3D zs_map_object(zram->mem_pool, handle_new, ZS_MM_WO); - memcpy(dst, zstrm->buffer, comp_len_new); + zs_obj_write(zram->mem_pool, handle_new, zstrm->buffer, comp_len_new); zcomp_stream_put(zstrm); =20 - zs_unmap_object(zram->mem_pool, handle_new); - zram_free_page(zram, index); zram_set_handle(zram, index, handle_new); zram_set_obj_size(zram, index, comp_len_new); --=20 2.48.1.711.g2feabab25a-goog From nobody Tue Dec 16 19:24:32 2025 Received: from mail-pl1-f176.google.com (mail-pl1-f176.google.com [209.85.214.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0A59D1E98F4 for ; Mon, 3 Mar 2025 02:25:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740968755; cv=none; b=NB+uM3sfzTuGSXv2SoAUS4JLqGjsUucC7dKVvwHwGPgoTekqgpJ9V82c6uCdXYCXGC/jc/0P1QA1WATOKA89c2t+5m/jgo1uVj9QaPS7+b86TT5+QBG1jHQEAs1WjRoeCTyGS+H5iUpZ8kNw8HSrhynS4f68XZwlBY3XiyzyKfQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740968755; c=relaxed/simple; bh=nq698JtCOHpeNKIC4Ok0mLG99cT8esicaS6FMZLlIWM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=iSHwZMtN0yGaNkdo0e5hdYvLkSkF5oK/xTZNbdccdqErsNnFZ17gkXnyhkl9B9Sy0MdvdECmUStFC8p550bmLPLcjrS+/Rm1w2vKWEgvrreJdqjjbhZLDZY0/bauF9n5Pvv/5a7FVrHIWnJCPXnfkG4PsNpKBzL3/oterGIc1KQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=gkiot92r; arc=none smtp.client-ip=209.85.214.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="gkiot92r" Received: by mail-pl1-f176.google.com with SMTP id d9443c01a7336-22356471820so59496305ad.0 for ; Sun, 02 Mar 2025 18:25:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1740968753; x=1741573553; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=m9V04z24we0PKbaMB+rapzgCCE9wJRDyCG7EP1iiRhA=; b=gkiot92rZmadQ/IxCsYqFxW1Rk2PE6u87nH3VXclSUVJ2UZ+tBSjjKFeUYhUjt27VT eliT4WfFSw8n0f+MzgDrJ34C6v+6b6a6MINK5vRF7igjWtWUqlUn4ZfCIfrVMjlZCk6c 1WaJ8MDlqsIv3gWlO4GfELT9LqeL4VXDqjZMc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740968753; x=1741573553; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=m9V04z24we0PKbaMB+rapzgCCE9wJRDyCG7EP1iiRhA=; b=sBmZ9l8AYwKTwZt6l9hni75G2vr4pyar1PA+5S5qB0CbeNc1kojQRzI5qh8WX2d3ds NvmmZRsspFhT7iUCAh+1Gu30di7gHftSy9zJfo+T+XVMGwk3gVL75fDJ4v4OF5wYVsAy dkC3LlxDxHjoxdEa1ICNpRJh4ixbfD2lfIuYV56Ve5Iv1o4CjPZj8v7RyDX61IRtYjyb A6jVXmwT1OCgcU1eQOZJ5m57ku7CwpOkIz6Z+Yaf9ISiGkgWIW3l3Gn4KjpN0TNIS9Jv BaTnbt0QWB/05RzSrrnMNpysaVvnfshMVKGt01LoiPV9VmpAvImEYpFw/rqOQTwoZViD Rn4g== X-Forwarded-Encrypted: i=1; AJvYcCXnPDGtHYMMthkZnrOVGElnYwGxR1+cpL+DS9XuIMmrQqj0eL5ImdwDdfqql3uvYkd9vcfHA+A/I+fr/9g=@vger.kernel.org X-Gm-Message-State: AOJu0YxzPO1qADETv9V5360KzZU8BQ7yJ9lKJXxHn/i/wQHlPxGV8yMX rRw+Gv+upGKqSseqGUkhkVTiKLI6QKlk0O2MuFI6xmKD4y06AtmerCOgTQWYqg== X-Gm-Gg: ASbGncvRn2X73rYv2anTgfQB7ODly4C297U9yqrzOmDuy0sL1yJSDzR9187u5BnX505 pV1/1XtpGb9AwvQO+MFRWn93LCSoG2kKduVWkYz4pDxUwk9+uapEV95Z5aTysQuH3wrx6+jsG79 3/z/oqlAguqCM8yZTFX+254D8Kj4Q5IYRHD8jjCXxsELkqzYquRek2BgnZp6rdK7oVJr0/qg80u LKZy0jnbCseBEiLkGMSwcYR7RlIv2ez3y0Hdk+U8DTuNsoN0sqm7nZyqJ8XoHVmxQSNdQChjdAu co3CXu2adcDkt5DXT1fYXJpjYDujyF/2gZkD+DeZYfkTmCw= X-Google-Smtp-Source: AGHT+IFRGkg4HAVHTSk7P/5WiDLiMFVD4ws7CbIAd4dpZ2NPws7biEyrBHhEpztlue9ApQq6yVEBhg== X-Received: by 2002:a17:903:32c8:b0:223:245a:11e2 with SMTP id d9443c01a7336-22368f71ed9mr148206025ad.6.1740968753298; Sun, 02 Mar 2025 18:25:53 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:1513:4f61:a4d3:b418]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-223504c7f12sm66879285ad.122.2025.03.02.18.25.51 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sun, 02 Mar 2025 18:25:52 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Hillf Danton , Kairui Song , Sebastian Andrzej Siewior , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v10 16/19] zram: permit reclaim in zstd custom allocator Date: Mon, 3 Mar 2025 11:03:25 +0900 Message-ID: <20250303022425.285971-17-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.711.g2feabab25a-goog In-Reply-To: <20250303022425.285971-1-senozhatsky@chromium.org> References: <20250303022425.285971-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" When configured with pre-trained compression/decompression dictionary support, zstd requires custom memory allocator, which it calls internally from compression()/decompression() routines. That means allocation from atomic context (either under entry spin-lock, or per-CPU local-lock or both). Now, with non-atomic zram read()/write(), those limitations are relaxed and we can allow direct and indirect reclaim. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/backend_zstd.c | 11 +---------- 1 file changed, 1 insertion(+), 10 deletions(-) diff --git a/drivers/block/zram/backend_zstd.c b/drivers/block/zram/backend= _zstd.c index 1184c0036f44..22c8067536f3 100644 --- a/drivers/block/zram/backend_zstd.c +++ b/drivers/block/zram/backend_zstd.c @@ -24,19 +24,10 @@ struct zstd_params { /* * For C/D dictionaries we need to provide zstd with zstd_custom_mem, * which zstd uses internally to allocate/free memory when needed. - * - * This means that allocator.customAlloc() can be called from zcomp_compre= ss() - * under local-lock (per-CPU compression stream), in which case we must use - * GFP_ATOMIC. - * - * Another complication here is that we can be configured as a swap device. */ static void *zstd_custom_alloc(void *opaque, size_t size) { - if (!preemptible()) - return kvzalloc(size, GFP_ATOMIC); - - return kvzalloc(size, __GFP_KSWAPD_RECLAIM | __GFP_NOWARN); + return kvzalloc(size, GFP_NOIO | __GFP_NOWARN); } =20 static void zstd_custom_free(void *opaque, void *address) --=20 2.48.1.711.g2feabab25a-goog From nobody Tue Dec 16 19:24:32 2025 Received: from mail-pj1-f47.google.com (mail-pj1-f47.google.com [209.85.216.47]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2B2B21EB18D for ; Mon, 3 Mar 2025 02:25:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.47 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740968759; cv=none; b=CxyIDvrlUwIRxqdYC1OT1rrVM8aBxSs1DmTk7c9VvkCn+YhHu2wsKnVXLuhyuiLHACnhBuKc+Ask/AuwrvFzKTmfln/iBP1NtqWEY8jcHKS2uxMri1WITXTgxA3YS48ZFDkFCM+Sk8whOkpd/S6HluFqSymfjoJAzUSuLpHgtZ8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740968759; c=relaxed/simple; bh=ke1dPd6qfbXoNkqwU7aSzkDQ+8P3AgWfJkB0cvYqPAo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=o0pzX2TFqEYI4kuHuGkOTeSeNNDnoJ85+gB6kPifnOBs94+o0d6D+cc/b2iMU0qk0vfv2eK8+lsBzWiG+TqNLG/Y2aNAZrCu67N63R41mt/yERaENP4R/3tzQlRBh8D7iwx8rSVQWgQReD8OPCPz7RCwDmjrQDRy7v1SzoerBCc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=N95+E9mm; arc=none smtp.client-ip=209.85.216.47 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="N95+E9mm" Received: by mail-pj1-f47.google.com with SMTP id 98e67ed59e1d1-2fe848040b1so8152854a91.3 for ; Sun, 02 Mar 2025 18:25:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1740968757; x=1741573557; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=iyo5ByX4ms7OSRp4VomEFa7attH8xkBy9eJ/3JfBMpA=; b=N95+E9mmnrze8GxtynAIUhw2YF+SQxOjvsBWwMYOHeBp8YEpadQF1QpQ1pKyosEUOX HkYb2SFVYXBF25zWJlx6eFAyn0XM/bdWUfcde5StU/1KqwUmN0ln0VoLXwG7WjTFMoRY 3LAl96+Ga9QrTStC+6PAVPAgSlnj21feCYS1w= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740968757; x=1741573557; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=iyo5ByX4ms7OSRp4VomEFa7attH8xkBy9eJ/3JfBMpA=; b=LLZtfd8G0P5b0QPnzYAphptnaA5UjqNQnT1TK0HVNttRjrRYDWzFvLbmbUpPAssR1t MJXYlozecFwdD428y4V+p9IocrVuC2Ety+ZsbuwcqrsoXLcGIuaUSCu364tkRa7fY5r/ TRMbRWQkfwd7JO1dLFL86yNbhAE3Vt3Frv5E2L2AXx5z9eoz4VDcqRjPiGTg3fKpAx+X mYW0lnxZRVtKnHVgiVxj91RcK68j0MgCgFISd9wky/625lo67hDOwM0O65BN9pTe+d0k 5zgsrIjJhebeSZy8XEuj1382rIdGs9raX8337rrMW/W5gS2GJt+6+JNuMCrm7jE0haYl 4W/w== X-Forwarded-Encrypted: i=1; AJvYcCXmyZ6s111ex4uLjD3M5/Zf09VoC/YAKZvCOiKuJ7onEAjvc/WNIuDhDPMmodx5Lq8vFJxYui3JvwK89PY=@vger.kernel.org X-Gm-Message-State: AOJu0YynuxHdjKEvoZyKw6p0qXZKVUo3kjkmSGoRVEgIDdzEc4gYOAcd eqW/I7LFY403W0cC/lzpg0WVOxkkQ2Eo3bjRMUV67NH0H9YeMBzytCmws3n8gg== X-Gm-Gg: ASbGncsj37txfRgZ6YM2+Me1ZJP01pMKNtUXLMeTAW2f/hczngmb/+Hc15QbRUEuUJO Nn3KbBX7aK/efC8j20u/64TNtV/jHfMTNbe4zlcYfe0MP/hRP63LqaJgP3AhW41kw/Gtw6uLOZd UB+Kp3T6c1LGYD0PtrWuTLshNeTO1mfEX0Z7cGWakiMyL75HRkV4wTJCv8tB6vTUixm/M6PZ1ho mINlCoqcC3wVu/bVcVqpaLi09zF1K+o9aBk+SS0kOFcM0dVb/SrLNe5f59zNcJQ4FRWR8/ZaVN7 tOrX/rUXwZEZjbnVn4PxBFOI0twHjj/ZadMLgSCNWEC71pk= X-Google-Smtp-Source: AGHT+IFJotMZFWyExSutVlC701wQGCarIM/cdTnnq47i9KiZ5lx5Nlm+Rg/03s7HRjl5+WkOajCTxg== X-Received: by 2002:a17:90b:288e:b0:2fc:c262:ef4b with SMTP id 98e67ed59e1d1-2febab787f0mr20378028a91.18.1740968757557; Sun, 02 Mar 2025 18:25:57 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:1513:4f61:a4d3:b418]) by smtp.gmail.com with UTF8SMTPSA id 98e67ed59e1d1-2fea698f3b2sm7671186a91.41.2025.03.02.18.25.55 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sun, 02 Mar 2025 18:25:57 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Hillf Danton , Kairui Song , Sebastian Andrzej Siewior , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v10 17/19] zram: do not leak page on recompress_store error path Date: Mon, 3 Mar 2025 11:03:26 +0900 Message-ID: <20250303022425.285971-18-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.711.g2feabab25a-goog In-Reply-To: <20250303022425.285971-1-senozhatsky@chromium.org> References: <20250303022425.285971-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Ensure the page used for local object data is freed on error out path. Fixes: 3f909a60cec1 ("zram: rework recompress target selection strategy") Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 62aef12417a4..e50a5a216974 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -2013,7 +2013,7 @@ static ssize_t recompress_store(struct device *dev, struct zram_pp_slot *pps; u32 mode =3D 0, threshold =3D 0; u32 prio, prio_max; - struct page *page; + struct page *page =3D NULL; ssize_t ret; =20 prio =3D ZRAM_SECONDARY_COMP; @@ -2157,9 +2157,9 @@ static ssize_t recompress_store(struct device *dev, cond_resched(); } =20 - __free_page(page); - release_init_lock: + if (page) + __free_page(page); release_pp_ctl(zram, ctl); atomic_set(&zram->pp_in_progress, 0); up_read(&zram->init_lock); --=20 2.48.1.711.g2feabab25a-goog From nobody Tue Dec 16 19:24:32 2025 Received: from mail-pj1-f50.google.com (mail-pj1-f50.google.com [209.85.216.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BA2D61D86DC for ; Mon, 3 Mar 2025 02:26:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.50 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740968764; cv=none; b=aKyOfqi4RaopuUoc0lFJOq1fcz7VBowzWxxuykt3NbW6Q604DOGr9/rT4cMnaAM4s7mLqP9dHy6Tb/6KcWKfQseLVCqQVUrYq5t5d5o0TcZf/Mjln16jXAL3eO80496iBjqxE8KNZ5GdSYz6T8HZERXyKB5pv2/e2+N5K1Bj3T4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740968764; c=relaxed/simple; bh=vtzctO98UUy+6GJzvnuXMNw49YXzhxByPxbJICgMnlU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=G2d0mNmFVy4+FdbDixR9rmHu6xYhEAYV6fbHgtBq3G9JWp8K0/P/z+GioFfB3f9TSCm9ioBhUFEIBO5La4+DkZM18PB0qa0SLTPhyYrEG4MFASizwabZkaiKMJ/nV65+Fqi5OoXfSAQ1T+30WiQWgyzcSQV4l6uBp1ogx4vaEcw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=glQoVSoj; arc=none smtp.client-ip=209.85.216.50 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="glQoVSoj" Received: by mail-pj1-f50.google.com with SMTP id 98e67ed59e1d1-2f9b91dff71so6176085a91.2 for ; Sun, 02 Mar 2025 18:26:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1740968762; x=1741573562; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=7MsZXmzA7gZttOoABIScoWzRHdPrz6HqbSztfy5OTfM=; b=glQoVSoj04RnBu/WPWJxXCL1SMno2vpIQFCA9ZSzwwHfF8g2TN4xJ4p2zdeMXCUbui Sl1RGxMT8fAr8mNmYdKX3V6Ua62glHZ+DJ8bzYYPhl6QqkBIhM0us/cBC86/dWbD3Zba n0qYduV4ziyyhlGwRAihyo9prHnAsz7974z1I= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740968762; x=1741573562; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7MsZXmzA7gZttOoABIScoWzRHdPrz6HqbSztfy5OTfM=; b=FnOBavK+uJpcb6fvouObnCk30pcD29gCY/NQHxeJzIEFq8Im8OdMhakde7YSvRpFVA WZFkIt9dElAKho+QYyeliiaiCb3PBQVcvwYe9blTU1t17eQ0x7CAmtZM81D5Kz9ZQCfu isy4yvrUTAOuzu9WL6f9hX2/zntBoNPxRIbFF2MrQNcPIooqX8gaf4UFsw+gm0/TAVgm OR/NiusOyI6T2CeMebrZ/SIJLy/Gir35qkVZwIX27ldTeh08IBiLYCuJvsuZHKh62ZXT sN112tIa3buaGX8hBD8oE+SWExhxeVno9i5ovCH1t0hKTgHjuAFgq17b0TNVbspSEin6 GJzw== X-Forwarded-Encrypted: i=1; AJvYcCWrdRyExRhMKvm4QK6vy1uuSCGG0JNqj1e4TNiE/SeO6EiHSCPHKU1HlNFgCOiFDjs7A4pTtlIZ7q21HOo=@vger.kernel.org X-Gm-Message-State: AOJu0Yz7ua2cYwENRH4/yXqxLrpB4t5BMbPQwXJtoW27JnKKH5QEWa74 bV9MQGKctO95+PWA7LzCQHLSoMcKJ7k6hMO5zVaLbIzWdjYXJT6y/9cDlTJb1A== X-Gm-Gg: ASbGncsgedag7N8hoCawm7wql1MpCOs2O5GAbg09romv1KUN6GLbLpYtE92n0l2wcvN so9dtLZbTyrBlUiK0NFxB3cZO+i4MAb8xyKhDpoLuEpMxJdYa3yaa/HXhYVA4yyYrAy+ChQmHdm bR06bOxuoHcFCwu/BiqsqSHZKuQBWFmm0zbhuhvrCvXWxvp819Wgp4EtgdPqAueCOeYOka2+mfm CnkSY6EPrVQFMXrs73c5qgImFphutdclotBDmlok6fqwUGoLQFJnIPyoofeZikxTd8pquZ7x2eO pRYOXvOEXig/o6g9cLxkP3zeWvJUjkS+E8y8JkqHTv5ACKM= X-Google-Smtp-Source: AGHT+IEwLUQ8+oZJN0pj6P2S75Tk9SYQBRSYoeVDDP2x+EgMUD0KKR9qXN1nP4rIBG836Z6i3jCjHQ== X-Received: by 2002:a17:90b:1ccd:b0:2ee:c918:cd60 with SMTP id 98e67ed59e1d1-2febab78da2mr17797763a91.20.1740968761993; Sun, 02 Mar 2025 18:26:01 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:1513:4f61:a4d3:b418]) by smtp.gmail.com with UTF8SMTPSA id 98e67ed59e1d1-2fea699090dsm7711363a91.48.2025.03.02.18.25.59 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sun, 02 Mar 2025 18:26:01 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Hillf Danton , Kairui Song , Sebastian Andrzej Siewior , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v10 18/19] zram: do not leak page on writeback_store error path Date: Mon, 3 Mar 2025 11:03:27 +0900 Message-ID: <20250303022425.285971-19-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.711.g2feabab25a-goog In-Reply-To: <20250303022425.285971-1-senozhatsky@chromium.org> References: <20250303022425.285971-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Ensure the page used for local object data is freed on error out path. Fixes: 330edc2bc059 (zram: rework writeback target selection strategy) Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index e50a5a216974..fda7d8624889 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -787,7 +787,7 @@ static ssize_t writeback_store(struct device *dev, unsigned long index =3D 0; struct bio bio; struct bio_vec bio_vec; - struct page *page; + struct page *page =3D NULL; ssize_t ret =3D len; int mode, err; unsigned long blk_idx =3D 0; @@ -929,8 +929,10 @@ static ssize_t writeback_store(struct device *dev, =20 if (blk_idx) free_block_bdev(zram, blk_idx); - __free_page(page); + release_init_lock: + if (page) + __free_page(page); release_pp_ctl(zram, ctl); atomic_set(&zram->pp_in_progress, 0); up_read(&zram->init_lock); --=20 2.48.1.711.g2feabab25a-goog From nobody Tue Dec 16 19:24:32 2025 Received: from mail-pl1-f179.google.com (mail-pl1-f179.google.com [209.85.214.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 26923159596 for ; Mon, 3 Mar 2025 02:26:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740968769; cv=none; b=Ufig/PQ9M9SLDJe+PT4ZB1z8FsIWiR9bjlGF2JNredW5G9I9g6JJmGt9odUhOOZLV7VDMt7gluAzuNHEYH2XmobLT4L6Ht7G2burf+kRpP3Bxe2Ssq+4h2i2xLvAzZuVKa84jUEtDGROYfEanAtPIoAhYQdu5mNU9q21kQ6r9AI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740968769; c=relaxed/simple; bh=aFGLpXQ2zJikLhp0V/7S2kKJQjxoxLph0MK/8nTuEpk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Qnb8tERxm6bTxgUt1buXtnwVR+9PtzJxSGXvU/rhp6KOShuvKUf0iR1Cdt1ggaEWzgh+sYFTDMbIcY9IyMyK8HaxlbipmDY9Xx1dx6AVZ6N5i0ny8ioEjtyD0DoMazDnvDeeGT8VpgFGiV2xTMAVq4Ja8zygZKF5oqosqKTneZw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=kapbHxaS; arc=none smtp.client-ip=209.85.214.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="kapbHxaS" Received: by mail-pl1-f179.google.com with SMTP id d9443c01a7336-223480ea43aso97214455ad.1 for ; Sun, 02 Mar 2025 18:26:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1740968767; x=1741573567; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=uqfsMunD7m+H43cqKfQWJpYb2LZblnvSJ7I+sWg0jpM=; b=kapbHxaST4JZ8fxG7saZcDPoMIAKzHHSAypTh4gYUbUSp45836zQ1UnA2J6hnnhJCF oqA+opmk0b1emeTOg4KZpXz1Yok94Jj2mwfP1/tBU2YgW9n+XZQa32DAlecYcDsPddSJ yslT6kSWnYSFREUmUZu5qNGcurFSUzC65irPU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740968767; x=1741573567; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=uqfsMunD7m+H43cqKfQWJpYb2LZblnvSJ7I+sWg0jpM=; b=sHIA89N3ZSbPP9R3wO/zSc2ilHKRKmRM2KoLIhcg1nwvBr/5WYQ0/oFJVciSsJWNSf aXz5L/O2TvNQJMEpwTyG8b7NDXLF1H40jtasICKbv9d1dctHzDdCs6TyaJ5dH4lnobEN bIztyOtCLjOKMxjiJ0vrALCZomrQ+8XNVPS67Ym80q+UGTKx3sM6dJMQDnDXjNWrjgQi uda2jJ+MM9dZbxzN91B8S8lvAQC4YeaaLZoEHRlHcUacLSY4/8Cmy4fpnja8wX6lnjt/ Ldt2mY3d/rYKvF0xnhDbh2bExhD/z74Bb7+v55nlcQxFNqQ61dvlne6PFZGk9kq43C4T MD9A== X-Forwarded-Encrypted: i=1; AJvYcCUN0g9aMhqfbnZz/P25Gr5X1S2kP4640On/AMZIjMckUqaG9gj4Cyp5puDSRcMEg8q92QOr6UEkamjTYGc=@vger.kernel.org X-Gm-Message-State: AOJu0YxUwrPzW+mjCaeEdoHPt4oZYE4s7MeD7r7TfOdnYwqJE8BiRl9E xr89NGJAt2AI27WumfjrySuotZQ+YOcm2UqEeylHJaK2U9tIe/2Rq4JV7faH3w== X-Gm-Gg: ASbGncu3feeZLqCTfDnbVnL/ZKksiq0zmab6KsTyer44OH2hiJhaKID9BcZ4lfa2WG0 ca+uEqkFZVzaH/6J/te/55hRP4wX1G3AYQ+Oni2kP3gVFBlkvcGmQ/8XufekxHfwWRg8MRxL8AQ o3elVhkTr8zLmS4w5BIV6dfUFkTBKLrCRjJYnJjpCvpVXdYguRgXMES0PfvZHUH81z0mUY0tUY8 srBg/lKYfaplUUBU/vEGzAjaTc8Jj+lY4abaWMdfpREfpX27GHdyiOJ3zFZ8lj4Y+xiVoyGEpz1 EfkQoVv18D0jEZvdEYW9ZEex+wvcVnRb1VsWH8ivRTs1l1M= X-Google-Smtp-Source: AGHT+IFee2geL4KKk2MpUs+7Jc0KaKLEfwBOxn21cLTIdP8Q7RgxcXKrybsQAiLxlVLTASPWxF9odg== X-Received: by 2002:a05:6a00:b86:b0:730:8a0a:9ef9 with SMTP id d2e1a72fcca58-734ac3fffa7mr16371017b3a.22.1740968767388; Sun, 02 Mar 2025 18:26:07 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:1513:4f61:a4d3:b418]) by smtp.gmail.com with UTF8SMTPSA id d2e1a72fcca58-736556c8dccsm787620b3a.100.2025.03.02.18.26.05 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sun, 02 Mar 2025 18:26:07 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Hillf Danton , Kairui Song , Sebastian Andrzej Siewior , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v10 19/19] zram: add might_sleep to zcomp API Date: Mon, 3 Mar 2025 11:03:28 +0900 Message-ID: <20250303022425.285971-20-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.711.g2feabab25a-goog In-Reply-To: <20250303022425.285971-1-senozhatsky@chromium.org> References: <20250303022425.285971-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Explicitly state that zcomp compress/decompress must be called from non-atomic context. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zcomp.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/drivers/block/zram/zcomp.c b/drivers/block/zram/zcomp.c index a1d627054bb1..d26a58c67e95 100644 --- a/drivers/block/zram/zcomp.c +++ b/drivers/block/zram/zcomp.c @@ -146,6 +146,7 @@ int zcomp_compress(struct zcomp *comp, struct zcomp_str= m *zstrm, }; int ret; =20 + might_sleep(); ret =3D comp->ops->compress(comp->params, &zstrm->ctx, &req); if (!ret) *dst_len =3D req.dst_len; @@ -162,6 +163,7 @@ int zcomp_decompress(struct zcomp *comp, struct zcomp_s= trm *zstrm, .dst_len =3D PAGE_SIZE, }; =20 + might_sleep(); return comp->ops->decompress(comp->params, &zstrm->ctx, &req); } =20 --=20 2.48.1.711.g2feabab25a-goog