From nobody Wed Feb 11 08:33:20 2026 Received: from mail-pl1-f175.google.com (mail-pl1-f175.google.com [209.85.214.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 44D906A33F for ; Wed, 29 Jan 2025 06:49:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.175 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738133350; cv=none; b=DM75l8F9iiI3AX4Cbzi2jMHP/UutfMgEknsr6nD0rVSJDvQ4hkaEF5mlAcXhTFPQPLBq/4OXUIWJDita8SBUXcZRNUEu5XhSZiXRn7Ehy7t+9dfdMwS7OpVoD1EhoK87ih3AZrDMyVk2XZgr+IkVt3HyleW/j438PlnW+yt7VP0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738133350; c=relaxed/simple; bh=ZScH+/u827xTncdAcWR5+7bSGH80h8CH3edBKvkZXcE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=lSeDyJhTcB7zhvtiIvNLsh7E/rRykhGGPRnCndhiFcwPotar6evb766ZrmheWX4cUhzvnvhhpyFV1s//+T8NYw4tMjg4EhLuVz1rYbzQz34AvUkizOT9bSG094wTBrHI7234n6ZeT7zlb6sfUrAqd2JdDeQbSH1Q+oW0S99zsyc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=nXAFhGLM; arc=none smtp.client-ip=209.85.214.175 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="nXAFhGLM" Received: by mail-pl1-f175.google.com with SMTP id d9443c01a7336-21654fdd5daso109598035ad.1 for ; Tue, 28 Jan 2025 22:49:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1738133348; x=1738738148; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=KC3ZyRGR1rlydzPUPnDKs73KJRkoEFLjCJSbJ+Sf7vQ=; b=nXAFhGLMxIZv8hKitaEwZ0A8h8EYUkBH+Tih+KfkvkXaMx9v6qjMvs3dUHUUMDOU+j BY3L5kexYc8duekz/M0IHjrwaormWFvjG7OVhItGrUo63feMx+S3N9ewH9LUGmAJBFJz qQrc3GEzcU1NxVH0pMzG8aWoTbP7T2K7gxk9s= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738133348; x=1738738148; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=KC3ZyRGR1rlydzPUPnDKs73KJRkoEFLjCJSbJ+Sf7vQ=; b=dznbdgHO4EkgCW0l1RV7Umk0G1fnR/AcrfQOTz/rUZy+GaUx8InPr/dDHimFKo3Tpo 2mmWFqWAXM3FQshzUDO6LpcXpTTvPqItIDfQlTVgJMwdpfDjIPUN3fHFI26C7+W+Kg9w WIvEy/CEtTb53Zh/D7kzBr2j89loToFZq7UmnqzWXfqjnXyV+PjGaSogsnB72OYY3gsa w0CfSg3lxyRC78bo7EJlK2pfP2OWP7W3gVoajiq/ojABKwodvgFK1Fu1zKj/aXDcooXO L/G4o3x8+entugiEYNFA7lS7nqiecP+IqHTUEMzOfeiyyFf5Ut8gqLkKF7SJ5oNNhoPX vn5w== X-Forwarded-Encrypted: i=1; AJvYcCUyzKeoVSPrEAA2deLZ8CEDTY4OqtDOV+188hUIBvcHDEOv4hWxEG4tyTVwmPo6hPy0NY2ybsIGMgsBzb0=@vger.kernel.org X-Gm-Message-State: AOJu0Yz2mlRzeiQLJU/xYlv7PVrL0neTCiXtOqvbCATuJQx7KMFi/p5s JrPXt6qJVXoR7DZAyOA0rktExTsw3Em8to45LmQG74ib5uwiqvMbRZg4t2LsKQ== X-Gm-Gg: ASbGncuVauJhjuRBJrVU3TZ+qUzK8q6q3o0v7+1gQzz5hZyNVEl76pG43KylsYzTmD1 VNVWruRa5Ht+euaePOaADWzrcwwjk6Lr3DhyQXtTOYinw3rD7XC3MO0B1HcdziZpyut/62CteVF cmsTo7G/3w0e+WiBMbQJDobvQ3HgfW87v8SFLwtkaJ2Z8TW2AqpzOa30gEQX5agUXWaTFY9bfM/ ZrXYDvjRZ7VQPcGBWd5r1XmvThLgMFAX8JSynpCDFpTupZl9OkX3LtSRyO6MZCeNOFa/2zKweQz /Z4dRor7zLdN0wxcWw== X-Google-Smtp-Source: AGHT+IELG1ixvgXjpQkcQZp7u89eXn+v+EqpAJ6lPiX4gzZ3VprPTBr1WTOe4RzEZR60eYHdIh3XEQ== X-Received: by 2002:a17:902:d492:b0:216:6f1a:1c81 with SMTP id d9443c01a7336-21dd7c35597mr28059235ad.2.1738133348518; Tue, 28 Jan 2025 22:49:08 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:b323:d70b:a1b8:1683]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-21da3d9c9f2sm92158755ad.42.2025.01.28.22.49.06 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 28 Jan 2025 22:49:08 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton , Minchan Kim , Johannes Weiner , Yosry Ahmed , Nhat Pham Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv1 1/6] zsmalloc: factor out pool locking helpers Date: Wed, 29 Jan 2025 15:43:47 +0900 Message-ID: <20250129064853.2210753-2-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.262.g85cc9f2d1e-goog In-Reply-To: <20250129064853.2210753-1-senozhatsky@chromium.org> References: <20250129064853.2210753-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" We currently have a mix of migrate_{read,write}_lock() helpers that lock zspages, but it's zs_pool that actually has a ->migrate_lock access to which is opene-coded. Factor out pool migrate locking into helpers, zspage migration locking API will be renamed to reduce confusion. Signed-off-by: Sergey Senozhatsky --- mm/zsmalloc.c | 56 +++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 41 insertions(+), 15 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 817626a351f8..2f8a2b139919 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -204,7 +204,8 @@ struct link_free { }; =20 struct zs_pool { - const char *name; + /* protect page/zspage migration */ + rwlock_t migrate_lock; =20 struct size_class *size_class[ZS_SIZE_CLASSES]; struct kmem_cache *handle_cachep; @@ -213,6 +214,7 @@ struct zs_pool { atomic_long_t pages_allocated; =20 struct zs_pool_stats stats; + atomic_t compaction_in_progress; =20 /* Compact classes */ struct shrinker *shrinker; @@ -223,11 +225,35 @@ struct zs_pool { #ifdef CONFIG_COMPACTION struct work_struct free_work; #endif - /* protect page/zspage migration */ - rwlock_t migrate_lock; - atomic_t compaction_in_progress; + + const char *name; }; =20 +static void pool_write_unlock(struct zs_pool *pool) +{ + write_unlock(&pool->migrate_lock); +} + +static void pool_write_lock(struct zs_pool *pool) +{ + write_lock(&pool->migrate_lock); +} + +static void pool_read_unlock(struct zs_pool *pool) +{ + read_unlock(&pool->migrate_lock); +} + +static void pool_read_lock(struct zs_pool *pool) +{ + read_lock(&pool->migrate_lock); +} + +static bool pool_lock_is_contended(struct zs_pool *pool) +{ + return rwlock_is_contended(&pool->migrate_lock); +} + static inline void zpdesc_set_first(struct zpdesc *zpdesc) { SetPagePrivate(zpdesc_page(zpdesc)); @@ -1206,7 +1232,7 @@ void *zs_map_object(struct zs_pool *pool, unsigned lo= ng handle, BUG_ON(in_interrupt()); =20 /* It guarantees it can get zspage from handle safely */ - read_lock(&pool->migrate_lock); + pool_read_lock(pool); obj =3D handle_to_obj(handle); obj_to_location(obj, &zpdesc, &obj_idx); zspage =3D get_zspage(zpdesc); @@ -1218,7 +1244,7 @@ void *zs_map_object(struct zs_pool *pool, unsigned lo= ng handle, * which is smaller granularity. */ migrate_read_lock(zspage); - read_unlock(&pool->migrate_lock); + pool_read_unlock(pool); =20 class =3D zspage_class(pool, zspage); off =3D offset_in_page(class->size * obj_idx); @@ -1453,13 +1479,13 @@ void zs_free(struct zs_pool *pool, unsigned long ha= ndle) * The pool->migrate_lock protects the race with zpage's migration * so it's safe to get the page from handle. */ - read_lock(&pool->migrate_lock); + pool_read_lock(pool); obj =3D handle_to_obj(handle); obj_to_zpdesc(obj, &f_zpdesc); zspage =3D get_zspage(f_zpdesc); class =3D zspage_class(pool, zspage); spin_lock(&class->lock); - read_unlock(&pool->migrate_lock); + pool_read_unlock(pool); =20 class_stat_sub(class, ZS_OBJS_INUSE, 1); obj_free(class->size, obj); @@ -1796,7 +1822,7 @@ static int zs_page_migrate(struct page *newpage, stru= ct page *page, * The pool migrate_lock protects the race between zpage migration * and zs_free. */ - write_lock(&pool->migrate_lock); + pool_write_lock(pool); class =3D zspage_class(pool, zspage); =20 /* @@ -1833,7 +1859,7 @@ static int zs_page_migrate(struct page *newpage, stru= ct page *page, * Since we complete the data copy and set up new zspage structure, * it's okay to release migration_lock. */ - write_unlock(&pool->migrate_lock); + pool_write_unlock(pool); spin_unlock(&class->lock); migrate_write_unlock(zspage); =20 @@ -1956,7 +1982,7 @@ static unsigned long __zs_compact(struct zs_pool *poo= l, * protect the race between zpage migration and zs_free * as well as zpage allocation/free */ - write_lock(&pool->migrate_lock); + pool_write_lock(pool); spin_lock(&class->lock); while (zs_can_compact(class)) { int fg; @@ -1983,14 +2009,14 @@ static unsigned long __zs_compact(struct zs_pool *p= ool, src_zspage =3D NULL; =20 if (get_fullness_group(class, dst_zspage) =3D=3D ZS_INUSE_RATIO_100 - || rwlock_is_contended(&pool->migrate_lock)) { + || pool_lock_is_contended(pool)) { putback_zspage(class, dst_zspage); dst_zspage =3D NULL; =20 spin_unlock(&class->lock); - write_unlock(&pool->migrate_lock); + pool_write_unlock(pool); cond_resched(); - write_lock(&pool->migrate_lock); + pool_write_lock(pool); spin_lock(&class->lock); } } @@ -2002,7 +2028,7 @@ static unsigned long __zs_compact(struct zs_pool *poo= l, putback_zspage(class, dst_zspage); =20 spin_unlock(&class->lock); - write_unlock(&pool->migrate_lock); + pool_write_unlock(pool); =20 return pages_freed; } --=20 2.48.1.262.g85cc9f2d1e-goog From nobody Wed Feb 11 08:33:20 2026 Received: from mail-pl1-f177.google.com (mail-pl1-f177.google.com [209.85.214.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E2A7F6A33F for ; Wed, 29 Jan 2025 06:49:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738133356; cv=none; b=koyQfccG0q5qE2/CnRho40Y8U13yICqsFQmnuJZMpNDmTu/MLzKBPkI51J2SSK54JRVUHmzYuH7BVh53JuPx60a3Po9YY3lJc1qZJ0PjdopJeA5mCd1aSVwF9SLXwivP5/weOVzWQwvA3dGD7yT/2AbRwtF8PrwdjM2UZWaIlrw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738133356; c=relaxed/simple; bh=jT3RS7060nG9mjcjdxoe/GfXqx37HRR/2sZkVogWWqY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=DSPhB1vgLSo/VC5GtyWlrheT+ZIrymoU6UA3e9Cbgpzx+WBgWlSZxfCgcfEVCltbTwMvIPxvRXhFa8L43Oc6NnJTORp3MwLpL5c8ZK6ZZLPl5zqK8kLjYbAtu31+4iQikp2qf73PFTJExKnuS6ZEWhi0X9hQcgZqhhD2DBOGmg0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=gRcbw7dT; arc=none smtp.client-ip=209.85.214.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="gRcbw7dT" Received: by mail-pl1-f177.google.com with SMTP id d9443c01a7336-21c2f1b610dso151660355ad.0 for ; Tue, 28 Jan 2025 22:49:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1738133354; x=1738738154; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=U5xlt93WdnDLIhYO6MmK7r5usvow5aZkNdIQfTNWjy0=; b=gRcbw7dTQhcGCUlwgO2iLF1Z3h70lx5D2iHTHjodt/40kRuASmLH3wIVaWweDPgmRt Jx384Rw0YrSWB7rUaK+iNiXp2m9KTZNW99G5yxoWIoBxn25YSpv+WV44/P+brawZYl6T Sn6ZXU5++BUsUsmjiSz6y6z7rV+lLORBq33+g= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738133354; x=1738738154; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=U5xlt93WdnDLIhYO6MmK7r5usvow5aZkNdIQfTNWjy0=; b=XrIsxv+sselymG6t3vziFvvVzaoqzLb2qXPmNhC8K857+VmuSFPY0sIHG5QNAIn4oc zi0b+THiPzSYWYymPSYnDxhYDOJpDeMD3jryS09P17uT42NDfaMlxJTeC2UdYFj5pYDe cy+i0KFth/xltlcfkM7KT4JvwbwRIfp6K+UnpdZWyubhAB+hAjl5SvmhEMTU3uVm/b/b 6nT9SoLnUMZV1PJuGmwt4DVuNqZDLro7givMIJEFkVo+FSwRYmS10DDIIWBlEoJs1T48 kz0EfdSLxUBWZ0M1UGCHeAy5QL6nIjDtYmFM+bw6S6iB31ymsLfraC5X6wcoOT426l1q uzbg== X-Forwarded-Encrypted: i=1; AJvYcCVneJ0/mcA2caGxb9RsXcQtsr/Mf1Q40SBhiZxTr19lN7W/gTBWa8t0/mtn5rwqcZTi4smFDNeRfmHbkWc=@vger.kernel.org X-Gm-Message-State: AOJu0YzbxbBkuDgpNWt8tl2a6nkHLbFfV9UfmH3v6EPIqvbBFH83CUpx taRX76B/1kmd4MK0WdIGX/yONzQhBasx5mH9iygO4gVow6FPZGTHbZPc1kzK0A== X-Gm-Gg: ASbGncsBJ2lIdRf41nYBDn9rg4hSOWTN0tmfzKIqUM0zl+8UffZ76PkNcHdyu851bFt heNWQ58nSY5/lDHFGmyw4GxqXYDeP7ZiYO7/04XxIKrI8v6Yh9K4ePRRaSfAWmLloJTULAEtYI9 iOPvraRcgSncTZJ1uDWKNBoMNhfrmjiHh1jGGjEfIkcHMxkaxGc3VQAfHXLnAxZELWsOnO8VCxB pk8nwjft7lgfNVVcn85/ie9ZJwueX/fhf5Hq45Uf1UAPAMdZvm0FJTlNt0XxJt3O9dZ3IhsuoeT yR97IvyD2HOy0poZPA== X-Google-Smtp-Source: AGHT+IFzqhCwAv4hpOOWLJMonKQwPRQTh3PdlNYEJTZLHBEywcIgCKkqWY9wtUBWb0Oa2eHoYdi0kA== X-Received: by 2002:a17:902:da85:b0:215:6489:cfbf with SMTP id d9443c01a7336-21dd7c44a4dmr33973515ad.11.1738133354160; Tue, 28 Jan 2025 22:49:14 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:b323:d70b:a1b8:1683]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-21da414d87fsm92309165ad.172.2025.01.28.22.49.11 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 28 Jan 2025 22:49:13 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton , Minchan Kim , Johannes Weiner , Yosry Ahmed , Nhat Pham Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv1 2/6] zsmalloc: factor out size-class locking helpers Date: Wed, 29 Jan 2025 15:43:48 +0900 Message-ID: <20250129064853.2210753-3-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.262.g85cc9f2d1e-goog In-Reply-To: <20250129064853.2210753-1-senozhatsky@chromium.org> References: <20250129064853.2210753-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Move open-coded size-class locking to dedicated helpers. Signed-off-by: Sergey Senozhatsky Reviewed-by: Yosry Ahmed --- mm/zsmalloc.c | 47 ++++++++++++++++++++++++++++------------------- 1 file changed, 28 insertions(+), 19 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 2f8a2b139919..0f575307675d 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -254,6 +254,16 @@ static bool pool_lock_is_contended(struct zs_pool *poo= l) return rwlock_is_contended(&pool->migrate_lock); } =20 +static void size_class_lock(struct size_class *class) +{ + spin_lock(&class->lock); +} + +static void size_class_unlock(struct size_class *class) +{ + spin_unlock(&class->lock); +} + static inline void zpdesc_set_first(struct zpdesc *zpdesc) { SetPagePrivate(zpdesc_page(zpdesc)); @@ -614,8 +624,7 @@ static int zs_stats_size_show(struct seq_file *s, void = *v) if (class->index !=3D i) continue; =20 - spin_lock(&class->lock); - + size_class_lock(class); seq_printf(s, " %5u %5u ", i, class->size); for (fg =3D ZS_INUSE_RATIO_10; fg < NR_FULLNESS_GROUPS; fg++) { inuse_totals[fg] +=3D class_stat_read(class, fg); @@ -625,7 +634,7 @@ static int zs_stats_size_show(struct seq_file *s, void = *v) obj_allocated =3D class_stat_read(class, ZS_OBJS_ALLOCATED); obj_used =3D class_stat_read(class, ZS_OBJS_INUSE); freeable =3D zs_can_compact(class); - spin_unlock(&class->lock); + size_class_unlock(class); =20 objs_per_zspage =3D class->objs_per_zspage; pages_used =3D obj_allocated / objs_per_zspage * @@ -1400,7 +1409,7 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t = size, gfp_t gfp) class =3D pool->size_class[get_size_class_index(size)]; =20 /* class->lock effectively protects the zpage migration */ - spin_lock(&class->lock); + size_class_lock(class); zspage =3D find_get_zspage(class); if (likely(zspage)) { obj_malloc(pool, zspage, handle); @@ -1411,7 +1420,7 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t = size, gfp_t gfp) goto out; } =20 - spin_unlock(&class->lock); + size_class_unlock(class); =20 zspage =3D alloc_zspage(pool, class, gfp); if (!zspage) { @@ -1419,7 +1428,7 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t = size, gfp_t gfp) return (unsigned long)ERR_PTR(-ENOMEM); } =20 - spin_lock(&class->lock); + size_class_lock(class); obj_malloc(pool, zspage, handle); newfg =3D get_fullness_group(class, zspage); insert_zspage(class, zspage, newfg); @@ -1430,7 +1439,7 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t = size, gfp_t gfp) /* We completely set up zspage so mark them as movable */ SetZsPageMovable(pool, zspage); out: - spin_unlock(&class->lock); + size_class_unlock(class); =20 return handle; } @@ -1484,7 +1493,7 @@ void zs_free(struct zs_pool *pool, unsigned long hand= le) obj_to_zpdesc(obj, &f_zpdesc); zspage =3D get_zspage(f_zpdesc); class =3D zspage_class(pool, zspage); - spin_lock(&class->lock); + size_class_lock(class); pool_read_unlock(pool); =20 class_stat_sub(class, ZS_OBJS_INUSE, 1); @@ -1494,7 +1503,7 @@ void zs_free(struct zs_pool *pool, unsigned long hand= le) if (fullness =3D=3D ZS_INUSE_RATIO_0) free_zspage(pool, class, zspage); =20 - spin_unlock(&class->lock); + size_class_unlock(class); cache_free_handle(pool, handle); } EXPORT_SYMBOL_GPL(zs_free); @@ -1828,7 +1837,7 @@ static int zs_page_migrate(struct page *newpage, stru= ct page *page, /* * the class lock protects zpage alloc/free in the zspage. */ - spin_lock(&class->lock); + size_class_lock(class); /* the migrate_write_lock protects zpage access via zs_map_object */ migrate_write_lock(zspage); =20 @@ -1860,7 +1869,7 @@ static int zs_page_migrate(struct page *newpage, stru= ct page *page, * it's okay to release migration_lock. */ pool_write_unlock(pool); - spin_unlock(&class->lock); + size_class_unlock(class); migrate_write_unlock(zspage); =20 zpdesc_get(newzpdesc); @@ -1904,10 +1913,10 @@ static void async_free_zspage(struct work_struct *w= ork) if (class->index !=3D i) continue; =20 - spin_lock(&class->lock); + size_class_lock(class); list_splice_init(&class->fullness_list[ZS_INUSE_RATIO_0], &free_pages); - spin_unlock(&class->lock); + size_class_unlock(class); } =20 list_for_each_entry_safe(zspage, tmp, &free_pages, list) { @@ -1915,10 +1924,10 @@ static void async_free_zspage(struct work_struct *w= ork) lock_zspage(zspage); =20 class =3D zspage_class(pool, zspage); - spin_lock(&class->lock); + size_class_lock(class); class_stat_sub(class, ZS_INUSE_RATIO_0, 1); __free_zspage(pool, class, zspage); - spin_unlock(&class->lock); + size_class_unlock(class); } }; =20 @@ -1983,7 +1992,7 @@ static unsigned long __zs_compact(struct zs_pool *poo= l, * as well as zpage allocation/free */ pool_write_lock(pool); - spin_lock(&class->lock); + size_class_lock(class); while (zs_can_compact(class)) { int fg; =20 @@ -2013,11 +2022,11 @@ static unsigned long __zs_compact(struct zs_pool *p= ool, putback_zspage(class, dst_zspage); dst_zspage =3D NULL; =20 - spin_unlock(&class->lock); + size_class_unlock(class); pool_write_unlock(pool); cond_resched(); pool_write_lock(pool); - spin_lock(&class->lock); + size_class_lock(class); } } =20 @@ -2027,7 +2036,7 @@ static unsigned long __zs_compact(struct zs_pool *poo= l, if (dst_zspage) putback_zspage(class, dst_zspage); =20 - spin_unlock(&class->lock); + size_class_unlock(class); pool_write_unlock(pool); =20 return pages_freed; --=20 2.48.1.262.g85cc9f2d1e-goog From nobody Wed Feb 11 08:33:20 2026 Received: from mail-pl1-f174.google.com (mail-pl1-f174.google.com [209.85.214.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D4A386A33F for ; Wed, 29 Jan 2025 06:49:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738133362; cv=none; b=qS71FylFHOLNW2GZb6VIgjBMFUMKzlcMgi/lRVMaBu3XIVMQN30/cBP5GIZBfnUxD+OIeuAYSb7YunCcmO6QQepXo3/8wf1Frt9APoHcwpgWTb8jvYmNjH3GzeiEjmt8zJJTFjXaS5aLlk5VlbLD/ZE3lkt4bhkeQtxeSIuOZfU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738133362; c=relaxed/simple; bh=fzYDvFpn7l278GtMlsOrD1WMOqzuixn8g1JCbBgaGvw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=F+93mWxG+BxRtmBhc2tgj/xB8+qnBYpHfuiv3oza7AMU7wvC79m2yKJSBG9UXV/jA7R73s0gl2FRcW8O7tmUVraia4Kw5Ybo9zSsQ3FKDoeCqrl0t8hCUSV034BMirtlcxTQBzKKuqdatNul4glgoNWYXjurJLz5T+AoNI8qYt0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=RN6DBQ4D; arc=none smtp.client-ip=209.85.214.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="RN6DBQ4D" Received: by mail-pl1-f174.google.com with SMTP id d9443c01a7336-21654fdd5daso109599485ad.1 for ; Tue, 28 Jan 2025 22:49:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1738133360; x=1738738160; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=QnAPrjmsHkTmFvv1w6err0gFxFxc4b8ZHvjvrGPNAh8=; b=RN6DBQ4DZ26GCej5kjSkWT6Ka5M0p2+Tls1Hf7kmAaTtKK8rciC5DbN01hGFAEDNYg tgszl15YI8YtEXsqunOAEQWf7W2AmAMPVE5IY01lcw31pBYfQfr5baO0k7NMvMtsvMIA hnARl0pfXF7PRlU59goqB74liEvqDYAt2N/KI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738133360; x=1738738160; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=QnAPrjmsHkTmFvv1w6err0gFxFxc4b8ZHvjvrGPNAh8=; b=c//Qml42RdDT7QUxTJSLmYuBI1BbzTdgOmAENJLmJdH4S2nJizk0jq2PX1d18zZ+fJ TyUPUeK+S8tjFDtGgHQO745eu3DL9kFJf0MuyQQ+LxTuGukc9qmFJMlFpr96wqVd1u5v Cpuc/0Ev1KfBCIDvAOiOEYQvxweLYK0RwVmaOdlna6NXqZhXlfeBFV+Im/VhwVOEL4Jp G3V+swUO90GwH8946nvJ6D/ezEW3pBpxsPC8fmbp5+4SGqaC9fuogPzuCqEtvx8A1cjj nlksS+fSh9CQKJ9lEUUk2YWXuHIBL6Yk3zYR1AnLXOMwKHMXEi7zeoRdPrwasIkS3tnh Vs4w== X-Forwarded-Encrypted: i=1; AJvYcCVipenO4G86XOn0BmCqLHLkhiRJVty1fMnrXZaGDtcwFNZAPj7q45CTKoF+4Xbeed0UyxrLt5PMd/E157A=@vger.kernel.org X-Gm-Message-State: AOJu0YxI8vDEAu1lf7WkCrWeXItgMoV9HOgY+FFF8HiPeFcGOU/WGlXw 4in9bfhTtHwPR3UWw2OgB/OCo2gHYxl5Kt00CCkr9PPONArmfcKZq87GJ3wO1w== X-Gm-Gg: ASbGncteGRSuObLnaa+TEdqR2rc9rvQMDa3SThPBKn6IX9G2uTd0etpTUCvw+ccrdUo gduN+TO0iU7lRn3DcgUaYZDzMFX7/sVX7h5soqcNuPt756RcFMtByM5pkBqu1eo6wLwGtJuwyQF kwGVRurXwGmkiSpDNfzfFUPHG1xEt8FN6Wr/gjk+OBagst+/b+Mx63kENQIsNnDiKDWuqSAhD2c GOYPXG6HqXWKezshDBnbu6GXf00Nlm4enIhhw7v7wOxvl3zsZNIpkeS2Bkc6dCyA0g4TykmYRJc zb0Pzh/mDcCxDOOudQ== X-Google-Smtp-Source: AGHT+IHjUFNFjYC3vAlkkDSXkUk59QE2BpQabcLqZFb+C3g33UA0CBbzBu0rrjhlkGsVrTPsZsZh4Q== X-Received: by 2002:a17:902:db03:b0:215:b33b:e26d with SMTP id d9443c01a7336-21dd7c5141fmr27092835ad.21.1738133359961; Tue, 28 Jan 2025 22:49:19 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:b323:d70b:a1b8:1683]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-21da424f61fsm91599795ad.237.2025.01.28.22.49.17 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 28 Jan 2025 22:49:19 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton , Minchan Kim , Johannes Weiner , Yosry Ahmed , Nhat Pham Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv1 3/6] zsmalloc: make zspage lock preemptible Date: Wed, 29 Jan 2025 15:43:49 +0900 Message-ID: <20250129064853.2210753-4-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.262.g85cc9f2d1e-goog In-Reply-To: <20250129064853.2210753-1-senozhatsky@chromium.org> References: <20250129064853.2210753-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Switch over from rwlock_t to a atomic_t variable that takes negative value when the page is under migration, or positive values when the page is used by zsmalloc users (object map, etc.) Using a rwsem per-zspage is a little too memory heavy, a simple atomic_t should suffice. zspage lock is a leaf lock for zs_map_object(), where it's read-acquired. Since this lock now permits preemption extra care needs to be taken when it is write-acquired - all writers grab it in atomic context, so they cannot spin and wait for (potentially preempted) reader to unlock zspage. There are only two writers at this moment - migration and compaction. In both cases we use write-try-lock and bail out if zspage is read locked. Writers, on the other hand, never get preempted, so readers can spin waiting for the writer to unlock zspage. With this we can implement a preemptible object mapping. Signed-off-by: Sergey Senozhatsky --- mm/zsmalloc.c | 140 +++++++++++++++++++++++++++++++------------------- 1 file changed, 88 insertions(+), 52 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 0f575307675d..8f4011713bc8 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -293,6 +293,9 @@ static inline void free_zpdesc(struct zpdesc *zpdesc) __free_page(page); } =20 +#define ZS_PAGE_UNLOCKED 0 +#define ZS_PAGE_WRLOCKED -1 + struct zspage { struct { unsigned int huge:HUGE_BITS; @@ -305,7 +308,7 @@ struct zspage { struct zpdesc *first_zpdesc; struct list_head list; /* fullness list */ struct zs_pool *pool; - rwlock_t lock; + atomic_t lock; }; =20 struct mapping_area { @@ -315,6 +318,64 @@ struct mapping_area { enum zs_mapmode vm_mm; /* mapping mode */ }; =20 +static void zspage_lock_init(struct zspage *zspage) +{ + atomic_set(&zspage->lock, ZS_PAGE_UNLOCKED); +} + +/* + * zspage lock permits preemption on the reader-side (there can be multiple + * readers). Writers (exclusive zspage ownership), on the other hand, are + * always run in atomic context and cannot spin waiting for a (potentially + * preempted) reader to unlock zspage. This, basically, means that writers + * can only call write-try-lock and must bail out if it didn't succeed. + * + * At the same time, writers cannot reschedule under zspage write-lock, + * so readers can spin waiting for the writer to unlock zspage. + */ +static void zspage_read_lock(struct zspage *zspage) +{ + atomic_t *lock =3D &zspage->lock; + int old; + + while (1) { + old =3D atomic_read(lock); + if (old =3D=3D ZS_PAGE_WRLOCKED) { + cpu_relax(); + continue; + } + + if (atomic_try_cmpxchg(lock, &old, old + 1)) + return; + + cpu_relax(); + } +} + +static void zspage_read_unlock(struct zspage *zspage) +{ + atomic_dec(&zspage->lock); +} + +static int zspage_try_write_lock(struct zspage *zspage) +{ + atomic_t *lock =3D &zspage->lock; + int old =3D ZS_PAGE_UNLOCKED; + + preempt_disable(); + if (atomic_try_cmpxchg(lock, &old, ZS_PAGE_WRLOCKED)) + return 1; + + preempt_enable(); + return 0; +} + +static void zspage_write_unlock(struct zspage *zspage) +{ + atomic_set(&zspage->lock, ZS_PAGE_UNLOCKED); + preempt_enable(); +} + /* huge object: pages_per_zspage =3D=3D 1 && maxobj_per_zspage =3D=3D 1 */ static void SetZsHugePage(struct zspage *zspage) { @@ -326,12 +387,6 @@ static bool ZsHugePage(struct zspage *zspage) return zspage->huge; } =20 -static void migrate_lock_init(struct zspage *zspage); -static void migrate_read_lock(struct zspage *zspage); -static void migrate_read_unlock(struct zspage *zspage); -static void migrate_write_lock(struct zspage *zspage); -static void migrate_write_unlock(struct zspage *zspage); - #ifdef CONFIG_COMPACTION static void kick_deferred_free(struct zs_pool *pool); static void init_deferred_free(struct zs_pool *pool); @@ -1027,7 +1082,7 @@ static struct zspage *alloc_zspage(struct zs_pool *po= ol, return NULL; =20 zspage->magic =3D ZSPAGE_MAGIC; - migrate_lock_init(zspage); + zspage_lock_init(zspage); =20 for (i =3D 0; i < class->pages_per_zspage; i++) { struct zpdesc *zpdesc; @@ -1252,7 +1307,7 @@ void *zs_map_object(struct zs_pool *pool, unsigned lo= ng handle, * zs_unmap_object API so delegate the locking from class to zspage * which is smaller granularity. */ - migrate_read_lock(zspage); + zspage_read_lock(zspage); pool_read_unlock(pool); =20 class =3D zspage_class(pool, zspage); @@ -1312,7 +1367,7 @@ void zs_unmap_object(struct zs_pool *pool, unsigned l= ong handle) } local_unlock(&zs_map_area.lock); =20 - migrate_read_unlock(zspage); + zspage_read_unlock(zspage); } EXPORT_SYMBOL_GPL(zs_unmap_object); =20 @@ -1706,18 +1761,18 @@ static void lock_zspage(struct zspage *zspage) /* * Pages we haven't locked yet can be migrated off the list while we're * trying to lock them, so we need to be careful and only attempt to - * lock each page under migrate_read_lock(). Otherwise, the page we lock + * lock each page under zspage_read_lock(). Otherwise, the page we lock * may no longer belong to the zspage. This means that we may wait for * the wrong page to unlock, so we must take a reference to the page - * prior to waiting for it to unlock outside migrate_read_lock(). + * prior to waiting for it to unlock outside zspage_read_lock(). */ while (1) { - migrate_read_lock(zspage); + zspage_read_lock(zspage); zpdesc =3D get_first_zpdesc(zspage); if (zpdesc_trylock(zpdesc)) break; zpdesc_get(zpdesc); - migrate_read_unlock(zspage); + zspage_read_unlock(zspage); zpdesc_wait_locked(zpdesc); zpdesc_put(zpdesc); } @@ -1728,41 +1783,16 @@ static void lock_zspage(struct zspage *zspage) curr_zpdesc =3D zpdesc; } else { zpdesc_get(zpdesc); - migrate_read_unlock(zspage); + zspage_read_unlock(zspage); zpdesc_wait_locked(zpdesc); zpdesc_put(zpdesc); - migrate_read_lock(zspage); + zspage_read_lock(zspage); } } - migrate_read_unlock(zspage); + zspage_read_unlock(zspage); } #endif /* CONFIG_COMPACTION */ =20 -static void migrate_lock_init(struct zspage *zspage) -{ - rwlock_init(&zspage->lock); -} - -static void migrate_read_lock(struct zspage *zspage) __acquires(&zspage->l= ock) -{ - read_lock(&zspage->lock); -} - -static void migrate_read_unlock(struct zspage *zspage) __releases(&zspage-= >lock) -{ - read_unlock(&zspage->lock); -} - -static void migrate_write_lock(struct zspage *zspage) -{ - write_lock(&zspage->lock); -} - -static void migrate_write_unlock(struct zspage *zspage) -{ - write_unlock(&zspage->lock); -} - #ifdef CONFIG_COMPACTION =20 static const struct movable_operations zsmalloc_mops; @@ -1804,7 +1834,7 @@ static bool zs_page_isolate(struct page *page, isolat= e_mode_t mode) } =20 static int zs_page_migrate(struct page *newpage, struct page *page, - enum migrate_mode mode) + enum migrate_mode mode) { struct zs_pool *pool; struct size_class *class; @@ -1820,15 +1850,12 @@ static int zs_page_migrate(struct page *newpage, st= ruct page *page, =20 VM_BUG_ON_PAGE(!zpdesc_is_isolated(zpdesc), zpdesc_page(zpdesc)); =20 - /* We're committed, tell the world that this is a Zsmalloc page. */ - __zpdesc_set_zsmalloc(newzpdesc); - /* The page is locked, so this pointer must remain valid */ zspage =3D get_zspage(zpdesc); pool =3D zspage->pool; =20 /* - * The pool migrate_lock protects the race between zpage migration + * The pool->migrate_lock protects the race between zpage migration * and zs_free. */ pool_write_lock(pool); @@ -1838,8 +1865,15 @@ static int zs_page_migrate(struct page *newpage, str= uct page *page, * the class lock protects zpage alloc/free in the zspage. */ size_class_lock(class); - /* the migrate_write_lock protects zpage access via zs_map_object */ - migrate_write_lock(zspage); + /* the zspage write_lock protects zpage access via zs_map_object */ + if (!zspage_try_write_lock(zspage)) { + size_class_unlock(class); + pool_write_unlock(pool); + return -EINVAL; + } + + /* We're committed, tell the world that this is a Zsmalloc page. */ + __zpdesc_set_zsmalloc(newzpdesc); =20 offset =3D get_first_obj_offset(zpdesc); s_addr =3D kmap_local_zpdesc(zpdesc); @@ -1870,7 +1904,7 @@ static int zs_page_migrate(struct page *newpage, stru= ct page *page, */ pool_write_unlock(pool); size_class_unlock(class); - migrate_write_unlock(zspage); + zspage_write_unlock(zspage); =20 zpdesc_get(newzpdesc); if (zpdesc_zone(newzpdesc) !=3D zpdesc_zone(zpdesc)) { @@ -2006,9 +2040,11 @@ static unsigned long __zs_compact(struct zs_pool *po= ol, if (!src_zspage) break; =20 - migrate_write_lock(src_zspage); + if (!zspage_try_write_lock(src_zspage)) + break; + migrate_zspage(pool, src_zspage, dst_zspage); - migrate_write_unlock(src_zspage); + zspage_write_unlock(src_zspage); =20 fg =3D putback_zspage(class, src_zspage); if (fg =3D=3D ZS_INUSE_RATIO_0) { --=20 2.48.1.262.g85cc9f2d1e-goog From nobody Wed Feb 11 08:33:20 2026 Received: from mail-pl1-f177.google.com (mail-pl1-f177.google.com [209.85.214.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 51BD2195980 for ; Wed, 29 Jan 2025 06:49:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738133367; cv=none; b=hpZ2cxIS/PTdV2agyb0+Wb9/wFb3Q+I3VvmbzUEe5AJODPaAfozBxYwv+8tP4FYtJvTX6unwQmuLIaMqznbmt383b2QSV6v93CvU+bj5EhFZ65RDoGqO+/05dgDX5ah5UxJW4odPRAcZq2cISYV2b2HI4N3MIcA1XdKyHCIziv8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738133367; c=relaxed/simple; bh=mBQQQes7a2bNzMDM9rrUHwBBnEgQN7rYfC6EIx4LXF8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=OJQUpKqNY0iwO/2ya+beCvFDxLlD5Ll6q9MGnibqMrrOi4rvZ3Pxn7PylgHBOa0w6TwB//q4qmFvNvBcVugPZ+3vpMHveVUpcRYS7ZtRs3enh9BkbA8SspBuBzraMWNbGOrN9pcP+PMuJpGxGjKzFMFUHTZt9CvNawrBzJbMX5I= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=amPwcfj1; arc=none smtp.client-ip=209.85.214.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="amPwcfj1" Received: by mail-pl1-f177.google.com with SMTP id d9443c01a7336-21680814d42so106156305ad.2 for ; Tue, 28 Jan 2025 22:49:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1738133365; x=1738738165; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=KXGVSyFQ7wxPOOSznd8eqGTmOOS9W52Yl2SySbGkS2E=; b=amPwcfj1p6J+TWVZU4j3s+3/kK9Jz1FjwcKNy7wgFqUypF4EaFO4RZD27LMkGN50bu hUUuUWJyUVwkcjmrrz8mQWGc2ITFBRmP3EJFz5e++Ea9NhNyghl+3iiFeujmNXXfE+Jg +lTaubiqYY9G+tw+9U2vdXItVgnCohVE4NAzA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738133365; x=1738738165; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=KXGVSyFQ7wxPOOSznd8eqGTmOOS9W52Yl2SySbGkS2E=; b=Ro5haKkTGEitIMsoNE/63FJyXK3juZ3ZU26t7IpEXIXdLeDYQ5JCAMCHQOyI6Dm/zl Z3HlGOdUwGaWORdabVQ+qPOnQFBEaIcUAOzSQcu/AkFM9gOcJUvV6Z7wJtFtiSJHj6OV WlQDQN0lrMERU1k5hzvyWqVROLPr24YLF6P/0L2vsjG+YvgWmcpywa2HymOf86mSazE5 iVvZUER48BN/07gMlv1uBrWWRQjfD+DWeaHTAeifw4xngo1aZSLIeD5NJVws4wqkGBzE utU45N7alrZWEGHjhDrxDia4a7B2bljw6pOd58SCMyMnRuFHc44SvnSFSG/vS5UM+Hjb qKnA== X-Forwarded-Encrypted: i=1; AJvYcCV9xOogUpzzcI45wABXIdhYO7gZRXNo9WT+AjF9UVhUzmspazFaTmgD7QRiLq/omiFgGgKe+FHfk8XDXec=@vger.kernel.org X-Gm-Message-State: AOJu0Yy2Mps07gpjsUgQuzFlnhGwrSZzApt7i+qyrJ0gS8I/t+nH0Xct RC0xSI1CMH/k4G6gR/dWEtZzgFqZu2zO1ricVcw1cH2bwEL5KUJZ2hWKvA6d4w== X-Gm-Gg: ASbGncsaaYSsL9YLdNaJLBufuxCLnBflE9OS83HrPhhNYdBkB4RFg0c03qZfNvh9ANA 1JWKZEkwRiNmBcKk7hcXFeuqOuH+qRUUZze+OHYaBcyh58qbUl8wTQDxWwZeGEVKgjPCpMjYywU 58Hgb5/ObfPtOiyw1kc7wNUceKV+Sp4G3P6Ct1W/O6pGaCJTN2zlzrLDzYb2wuVfgT8XY6rPms3 aUGS28HUSmV45KMxAy83pYVkPYFxwwhc5NGMBd7LCtViElpnyGduHuaNl1lP26XOatEkqbCo8Gd sqLuoJxFjUoUX13BEA== X-Google-Smtp-Source: AGHT+IHRM9r7XrdtSHnutLuiUDtUgDRAtF8VGeBOts49IOHmepNWeYTTAUBt2jiNw5hFypkaNy8rEg== X-Received: by 2002:a17:903:41cc:b0:215:bb50:6a05 with SMTP id d9443c01a7336-21dd7c499d2mr21877945ad.9.1738133365611; Tue, 28 Jan 2025 22:49:25 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:b323:d70b:a1b8:1683]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-21da3d9d9fbsm92959935ad.9.2025.01.28.22.49.23 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 28 Jan 2025 22:49:25 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton , Minchan Kim , Johannes Weiner , Yosry Ahmed , Nhat Pham Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv1 4/6] zsmalloc: introduce new object mapping API Date: Wed, 29 Jan 2025 15:43:50 +0900 Message-ID: <20250129064853.2210753-5-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.262.g85cc9f2d1e-goog In-Reply-To: <20250129064853.2210753-1-senozhatsky@chromium.org> References: <20250129064853.2210753-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Current object mapping API is a little cumbersome. First, it's inconsistent, sometimes it returns with page-faults disabled and sometimes with page-faults enabled. Second, and most importantly, it enforces atomicity restrictions on its users. zs_map_object() has to return a liner object address which is not always possible because some objects span multiple physical (non-contiguous) pages. For such objects zsmalloc uses a per-CPU buffer to which object's data is copied before a pointer to that per-CPU buffer is returned back to the caller. This leads to another, final, issue - extra memcpy(). Since the caller gets a pointer to per-CPU buffer it can memcpy() data only to that buffer, and during zs_unmap_object() zsmalloc will memcpy() from that per-CPU buffer to physical pages that object in question spans across. New API splits functions by access mode: - zs_obj_read_begin(handle, local_copy) Returns a pointer to handle memory. For objects that span two physical pages a local_copy buffer is used to store object's data before the address is returned to the caller. Otherwise the object's page is kmap_local mapped directly. - zs_obj_read_end(handle, buf) Unmaps the page if it was kmap_local mapped by zs_obj_read_begin(). - zs_obj_write(handle, buf, len) Copies len-bytes from compression buffer to handle memory (takes care of objects that span two pages). This does not need any additional (e.g. per-CPU) buffers and writes the data directly to zsmalloc pool pages. The old API will stay around until the remaining users switch to the new one. After that we'll also remove zsmalloc per-CPU buffer and CPU hotplug handling. Signed-off-by: Sergey Senozhatsky Reviewed-by: Yosry Ahmed --- include/linux/zsmalloc.h | 8 +++ mm/zsmalloc.c | 129 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 137 insertions(+) diff --git a/include/linux/zsmalloc.h b/include/linux/zsmalloc.h index a48cd0ffe57d..625adae8e547 100644 --- a/include/linux/zsmalloc.h +++ b/include/linux/zsmalloc.h @@ -58,4 +58,12 @@ unsigned long zs_compact(struct zs_pool *pool); unsigned int zs_lookup_class_index(struct zs_pool *pool, unsigned int size= ); =20 void zs_pool_stats(struct zs_pool *pool, struct zs_pool_stats *stats); + +void zs_obj_read_end(struct zs_pool *pool, unsigned long handle, + void *handle_mem); +void *zs_obj_read_begin(struct zs_pool *pool, unsigned long handle, + void *local_copy); +void zs_obj_write(struct zs_pool *pool, unsigned long handle, + void *handle_mem, size_t mem_len); + #endif diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 8f4011713bc8..0e21bc57470b 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -1371,6 +1371,135 @@ void zs_unmap_object(struct zs_pool *pool, unsigned= long handle) } EXPORT_SYMBOL_GPL(zs_unmap_object); =20 +void zs_obj_write(struct zs_pool *pool, unsigned long handle, + void *handle_mem, size_t mem_len) +{ + struct zspage *zspage; + struct zpdesc *zpdesc; + unsigned long obj, off; + unsigned int obj_idx; + struct size_class *class; + + WARN_ON(in_interrupt()); + + /* Guarantee we can get zspage from handle safely */ + pool_read_lock(pool); + obj =3D handle_to_obj(handle); + obj_to_location(obj, &zpdesc, &obj_idx); + zspage =3D get_zspage(zpdesc); + + /* Make sure migration doesn't move any pages in this zspage */ + zspage_read_lock(zspage); + pool_read_unlock(pool); + + class =3D zspage_class(pool, zspage); + off =3D offset_in_page(class->size * obj_idx); + + if (off + class->size <=3D PAGE_SIZE) { + /* this object is contained entirely within a page */ + void *dst =3D kmap_local_zpdesc(zpdesc); + + if (!ZsHugePage(zspage)) + off +=3D ZS_HANDLE_SIZE; + memcpy(dst + off, handle_mem, mem_len); + kunmap_local(dst); + } else { + size_t sizes[2]; + + /* this object spans two pages */ + off +=3D ZS_HANDLE_SIZE; + sizes[0] =3D PAGE_SIZE - off; + sizes[1] =3D mem_len - sizes[0]; + + memcpy_to_page(zpdesc_page(zpdesc), off, + handle_mem, sizes[0]); + zpdesc =3D get_next_zpdesc(zpdesc); + memcpy_to_page(zpdesc_page(zpdesc), 0, + handle_mem + sizes[0], sizes[1]); + } + + zspage_read_unlock(zspage); +} +EXPORT_SYMBOL_GPL(zs_obj_write); + +void zs_obj_read_end(struct zs_pool *pool, unsigned long handle, + void *handle_mem) +{ + struct zspage *zspage; + struct zpdesc *zpdesc; + unsigned long obj, off; + unsigned int obj_idx; + struct size_class *class; + + obj =3D handle_to_obj(handle); + obj_to_location(obj, &zpdesc, &obj_idx); + zspage =3D get_zspage(zpdesc); + class =3D zspage_class(pool, zspage); + off =3D offset_in_page(class->size * obj_idx); + + if (off + class->size <=3D PAGE_SIZE) { + if (!ZsHugePage(zspage)) + off +=3D ZS_HANDLE_SIZE; + handle_mem -=3D off; + kunmap_local(handle_mem); + } + + zspage_read_unlock(zspage); +} +EXPORT_SYMBOL_GPL(zs_obj_read_end); + +void *zs_obj_read_begin(struct zs_pool *pool, unsigned long handle, + void *local_copy) +{ + struct zspage *zspage; + struct zpdesc *zpdesc; + unsigned long obj, off; + unsigned int obj_idx; + struct size_class *class; + void *addr; + + WARN_ON(in_interrupt()); + + /* Guarantee we can get zspage from handle safely */ + pool_read_lock(pool); + obj =3D handle_to_obj(handle); + obj_to_location(obj, &zpdesc, &obj_idx); + zspage =3D get_zspage(zpdesc); + + /* Make sure migration doesn't move any pages in this zspage */ + zspage_read_lock(zspage); + pool_read_unlock(pool); + + class =3D zspage_class(pool, zspage); + off =3D offset_in_page(class->size * obj_idx); + + if (off + class->size <=3D PAGE_SIZE) { + /* this object is contained entirely within a page */ + addr =3D kmap_local_zpdesc(zpdesc); + addr +=3D off; + } else { + size_t sizes[2]; + + /* this object spans two pages */ + sizes[0] =3D PAGE_SIZE - off; + sizes[1] =3D class->size - sizes[0]; + addr =3D local_copy; + + memcpy_from_page(addr, zpdesc_page(zpdesc), + off, sizes[0]); + zpdesc =3D get_next_zpdesc(zpdesc); + memcpy_from_page(addr + sizes[0], + zpdesc_page(zpdesc), + 0, sizes[1]); + } + + if (!ZsHugePage(zspage)) + addr +=3D ZS_HANDLE_SIZE; + + return addr; +} +EXPORT_SYMBOL_GPL(zs_obj_read_begin); + /** * zs_huge_class_size() - Returns the size (in bytes) of the first huge * zsmalloc &size_class. --=20 2.48.1.262.g85cc9f2d1e-goog From nobody Wed Feb 11 08:33:20 2026 Received: from mail-pl1-f176.google.com (mail-pl1-f176.google.com [209.85.214.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 489051990AB for ; Wed, 29 Jan 2025 06:49:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738133373; cv=none; b=NAOClE+3nYGmG7PYIlO8RThVfYjpat7BgA0bkORVrygLhDhLXrBYmATHiNDVRQ38lcyGHuAbq8X5Ww2h65ASwgfbc+LJo/xCf3/iMK986Na6D9POczrkrTmqxtcVNnsB1cN3/XTz3+TxcKdAVI4zDywkBFCJ8/l580sOvkdbJ98= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738133373; c=relaxed/simple; bh=b8wby6Mnz0jOlZOj4F5r8wWAxtOJ93uxJ6l+SPSOOYI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=qikDaXQK6at6wlTYAo7wVVpKC3uANL9i1WIeiIOleyaZHU2CGPBRFe/W84mm6AG/26/WZ2qKWv2CepTZ3xgxSUgY2Ti3TU9RcMofYC0g8SGm/sxaS84lIBGs+tUhd4+1rB6IzwV7pCIAVT/9J2E954vd+LuRlkKy0e8WAgZqEYU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=j8UWXpz5; arc=none smtp.client-ip=209.85.214.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="j8UWXpz5" Received: by mail-pl1-f176.google.com with SMTP id d9443c01a7336-21634338cfdso48330805ad.2 for ; Tue, 28 Jan 2025 22:49:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1738133371; x=1738738171; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=jm+b3R/xkgu1u5aIyHjEy9+3hXzV4kVBwpygpgrob3Q=; b=j8UWXpz54tv05J41Pj9jDH+oCoWAjjvUrXTeHlJzpMwrtcmb5VMms7PaNEa3BPy78F kLF59kxOwlDj/Jv/yvGrYBbPFuTazDmPvANXR04/DZnibkEEigdRszDy1kU6yk0rlUp0 qF2fUHEfsoA12vcsaXRtxRIZYd5j24LOdauXE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738133371; x=1738738171; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=jm+b3R/xkgu1u5aIyHjEy9+3hXzV4kVBwpygpgrob3Q=; b=g3SOzqpeEOdQkxlkGRjI+ZW/ypnu+g3m1HE1dIakn4+iahXhfkOQ4vKRjLoJ88gd/C EzU/k8Ka2c3c0SNgwFbTd7IrYMhmveFbi7XgnAJRHw/5GWOGB+q/vBQ8d8nxoN45xJeU rHLx7yeP4oRl7d/3Pms3VSnsDa1BhS7xxIo4nQoJ6rtlWwb0ZhqQdrgvuHbaG7f91TEl h3uiwdJy+AM3uSQPsRR1T87F9xkTlVK45CbFFe1g8lebY/Prl0JVNmfJ9J9Qr+1tqcNF KMw7HroVW3CB6IR8+DiUU91NB4yfw6hBGTBA2UP+E8BhTC7VA69fztZiAiYmthjNW2TR VwxQ== X-Forwarded-Encrypted: i=1; AJvYcCXQH8cIOSmhH84/y6jssn3wIDoRcoteKo+/NfhI3BiBDWHNHkPJ3Jeon5JQJhaH8QTk7DKpC53GkR8LWeM=@vger.kernel.org X-Gm-Message-State: AOJu0Ywum1OicpuOVGDfnj/QF1FRsUIJwoBYreuqd5o3ViIPTIh4RjaF 2OXLkWeEh0JVOIdXbjQtlvdwcVmnb0OeflY2latAnqZoCUZzL2RWs+NF3mXzBw== X-Gm-Gg: ASbGncuSqq/4VHaudgx/PqdRkq8tApqtyHgYAYf3HALK/RKR/DbTh55pl60QpOAsPwu or+pfD3aX30PdMyS5fww1sbWfyeJLfOuw8mA4HUcTKiB/Sht/LW5LaqUeEcOzvDnPsQqYe1+9Hg 9NC0V+01Koh9H+6wPenoZXsHcAw5YR5fEZ37pdPR3o+vrIK3VttI2BSrT1qXk+FOfYOgCbzO+h2 4L5/igQTARUi9Srv+XBQUzBS1y1U9VJD2ydVzwu4vbbKswkfleZQyStldziKYxcdY9vTWEf3Wwy ivxsZWrUUUnhvhT1eQ== X-Google-Smtp-Source: AGHT+IEK5agvR4qJHKICT0NvwP5kGDqjZ0/0rvwL4OJXcfuzB8y1QaZ9wi33U86Y9zDYeiv8HE16ww== X-Received: by 2002:a05:6a00:2406:b0:72d:9cbc:730d with SMTP id d2e1a72fcca58-72fd0c2246cmr2931620b3a.11.1738133371409; Tue, 28 Jan 2025 22:49:31 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:b323:d70b:a1b8:1683]) by smtp.gmail.com with UTF8SMTPSA id d2e1a72fcca58-72f8a7608b1sm10312341b3a.116.2025.01.28.22.49.29 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 28 Jan 2025 22:49:31 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton , Minchan Kim , Johannes Weiner , Yosry Ahmed , Nhat Pham Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv1 5/6] zram: switch to new zsmalloc object mapping API Date: Wed, 29 Jan 2025 15:43:51 +0900 Message-ID: <20250129064853.2210753-6-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.262.g85cc9f2d1e-goog In-Reply-To: <20250129064853.2210753-1-senozhatsky@chromium.org> References: <20250129064853.2210753-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Use new read/write zsmalloc object API. For cases when RO mapped object spans two physical pages (requires temp buffer) compression streams now carry around one extra physical page. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zcomp.c | 4 +++- drivers/block/zram/zcomp.h | 2 ++ drivers/block/zram/zram_drv.c | 28 ++++++++++------------------ 3 files changed, 15 insertions(+), 19 deletions(-) diff --git a/drivers/block/zram/zcomp.c b/drivers/block/zram/zcomp.c index efd5919808d9..675f2a51ad5f 100644 --- a/drivers/block/zram/zcomp.c +++ b/drivers/block/zram/zcomp.c @@ -45,6 +45,7 @@ static const struct zcomp_ops *backends[] =3D { static void zcomp_strm_free(struct zcomp *comp, struct zcomp_strm *strm) { comp->ops->destroy_ctx(&strm->ctx); + vfree(strm->local_copy); vfree(strm->buffer); kfree(strm); } @@ -66,12 +67,13 @@ static struct zcomp_strm *zcomp_strm_alloc(struct zcomp= *comp) return NULL; } =20 + strm->local_copy =3D vzalloc(PAGE_SIZE); /* * allocate 2 pages. 1 for compressed data, plus 1 extra in case if * compressed data is larger than the original one. */ strm->buffer =3D vzalloc(2 * PAGE_SIZE); - if (!strm->buffer) { + if (!strm->buffer || !strm->local_copy) { zcomp_strm_free(comp, strm); return NULL; } diff --git a/drivers/block/zram/zcomp.h b/drivers/block/zram/zcomp.h index 62330829db3f..9683d4aa822d 100644 --- a/drivers/block/zram/zcomp.h +++ b/drivers/block/zram/zcomp.h @@ -34,6 +34,8 @@ struct zcomp_strm { struct list_head entry; /* compression buffer */ void *buffer; + /* local copy of handle memory */ + void *local_copy; struct zcomp_ctx ctx; }; =20 diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index ad3e8885b0d2..d73e8374e9cc 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1562,11 +1562,11 @@ static int read_incompressible_page(struct zram *zr= am, struct page *page, void *src, *dst; =20 handle =3D zram_get_handle(zram, index); - src =3D zs_map_object(zram->mem_pool, handle, ZS_MM_RO); + src =3D zs_obj_read_begin(zram->mem_pool, handle, NULL); dst =3D kmap_local_page(page); copy_page(dst, src); kunmap_local(dst); - zs_unmap_object(zram->mem_pool, handle); + zs_obj_read_end(zram->mem_pool, handle, src); =20 return 0; } @@ -1584,11 +1584,11 @@ static int read_compressed_page(struct zram *zram, = struct page *page, u32 index) prio =3D zram_get_priority(zram, index); =20 zstrm =3D zcomp_stream_get(zram->comps[prio]); - src =3D zs_map_object(zram->mem_pool, handle, ZS_MM_RO); + src =3D zs_obj_read_begin(zram->mem_pool, handle, zstrm->local_copy); dst =3D kmap_local_page(page); ret =3D zcomp_decompress(zram->comps[prio], zstrm, src, size, dst); kunmap_local(dst); - zs_unmap_object(zram->mem_pool, handle); + zs_obj_read_end(zram->mem_pool, handle, src); zcomp_stream_put(zram->comps[prio], zstrm); =20 return ret; @@ -1684,7 +1684,7 @@ static int write_incompressible_page(struct zram *zra= m, struct page *page, u32 index) { unsigned long handle; - void *src, *dst; + void *src; =20 /* * This function is called from preemptible context so we don't need @@ -1701,11 +1701,9 @@ static int write_incompressible_page(struct zram *zr= am, struct page *page, return -ENOMEM; } =20 - dst =3D zs_map_object(zram->mem_pool, handle, ZS_MM_WO); src =3D kmap_local_page(page); - memcpy(dst, src, PAGE_SIZE); + zs_obj_write(zram->mem_pool, handle, src, PAGE_SIZE); kunmap_local(src); - zs_unmap_object(zram->mem_pool, handle); =20 zram_slot_write_lock(zram, index); zram_set_flag(zram, index, ZRAM_HUGE); @@ -1726,7 +1724,7 @@ static int zram_write_page(struct zram *zram, struct = page *page, u32 index) int ret =3D 0; unsigned long handle; unsigned int comp_len; - void *dst, *mem; + void *mem; struct zcomp_strm *zstrm; unsigned long element; bool same_filled; @@ -1769,11 +1767,8 @@ static int zram_write_page(struct zram *zram, struct= page *page, u32 index) return -ENOMEM; } =20 - dst =3D zs_map_object(zram->mem_pool, handle, ZS_MM_WO); - - memcpy(dst, zstrm->buffer, comp_len); + zs_obj_write(zram->mem_pool, handle, zstrm->buffer, comp_len); zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP], zstrm); - zs_unmap_object(zram->mem_pool, handle); =20 zram_slot_write_lock(zram, index); zram_set_handle(zram, index, handle); @@ -1882,7 +1877,7 @@ static int recompress_slot(struct zram *zram, u32 ind= ex, struct page *page, unsigned int class_index_old; unsigned int class_index_new; u32 num_recomps =3D 0; - void *src, *dst; + void *src; int ret; =20 handle_old =3D zram_get_handle(zram, index); @@ -2020,12 +2015,9 @@ static int recompress_slot(struct zram *zram, u32 in= dex, struct page *page, return 0; } =20 - dst =3D zs_map_object(zram->mem_pool, handle_new, ZS_MM_WO); - memcpy(dst, zstrm->buffer, comp_len_new); + zs_obj_write(zram->mem_pool, handle_new, zstrm->buffer, comp_len_new); zcomp_stream_put(zram->comps[prio], zstrm); =20 - zs_unmap_object(zram->mem_pool, handle_new); - zram_free_page(zram, index); zram_set_handle(zram, index, handle_new); zram_set_obj_size(zram, index, comp_len_new); --=20 2.48.1.262.g85cc9f2d1e-goog From nobody Wed Feb 11 08:33:20 2026 Received: from mail-pl1-f177.google.com (mail-pl1-f177.google.com [209.85.214.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 152FE192B9D for ; Wed, 29 Jan 2025 06:49:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738133379; cv=none; b=U7mq2xem0jnCQuZWoJ1vhCoOjFy9309SkUX2aUbWZFzBME9Qq3MnRAzUVSh1FM/c0SgITbffnYUkrsVaR5f8O/4C4b5XXAN+YS1r6GLNfvYRkfjsrIRyw2NFprFm08XHZCGqYmqZ5O5hZacO9f0qaFS8xOnIV6i+rCPe/xkqoyk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738133379; c=relaxed/simple; bh=kQOe19FMRhzWZQFPSQJ2h1HjTwoC6pXZ4Qv7RQgvqrg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=SyPeWgQrYIe0rOyP/LSWUl8PxQW6xcnCOaDuU/G0U9p719bJSvgoBs72tolmfCr77r2IiZlTkj1fgGlAmN3uio9Z6NGXwoeDfrTalVztOk8Piv1HTu4M/heYKw7TDmf2FYkdkzdo0m7o0Ef11TcolPRbiXbLO7Gya+rN6FTC7TI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=Axq6eQhq; arc=none smtp.client-ip=209.85.214.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="Axq6eQhq" Received: by mail-pl1-f177.google.com with SMTP id d9443c01a7336-21649a7bcdcso111511385ad.1 for ; Tue, 28 Jan 2025 22:49:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1738133377; x=1738738177; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=LjUByDm3ba8JrZJ2xgEVKiM3B6T3dmGtNNYJyOcrU3Y=; b=Axq6eQhqfPofTgDjwTBZSvteBOvxI8yeEklaGk1uEBK3DamVd7SFkxLPeCJfBMgvcV IGSnOUcd1Oq+/nJb6+30v8Doaq3O5vp1ZZ9QNMrYwqs2TvKCR1sdsJTXt88k/MvRJS2X q+mK+BZPk+Gy3wURaHCS3mrm/9d3/VzaGAF7o= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738133377; x=1738738177; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=LjUByDm3ba8JrZJ2xgEVKiM3B6T3dmGtNNYJyOcrU3Y=; b=Rtwxt64+gIiPKj1oWh97dUF86AVpxVRn9OIX8x7lhhq2qrmov51FMn2Qn/pZLnfoNW jEUOVv1Vensd234gWr/fdVTj4eKT+GME2Uzi5WmvkS9e5AtyZPLZsChX3W/dPId37ggJ Pf7eAKK5spznKsZD5+LXUjJRssCRlruZu3NTuNo1xJ57Q23gi7DGKtWgQJlJWp0m3Alx Eo7xpE/fd28V9xBLBtJ62c2srNl9IlQNxk7+K7+ioFm4kCrODNmiD+NuWv3i5c7e1crr L+B4kIRlb3AZ4vKH3s6j8kmIHHVpbKxZ/IlG2j7xY+oldNzpy6WEfphEGy98exJ+x3Gr dHxA== X-Forwarded-Encrypted: i=1; AJvYcCW3VOItCVZhshkyEtxxOjw7J5D0Q637lxdbPYPfeU18ar6u3yGi0cs7OLiVndNqqbY/oPXH07B1bcQ7q+E=@vger.kernel.org X-Gm-Message-State: AOJu0YxvEaI4JoJkx6ChmLF8W14fPN/olu4Ajky16trBLgKHvOVDRbst CCYrwZl7Vk+s5Cl0Os9a27ccHh58QwAhM4msy2Uy8ovGg/AOXBOtBqjhaTYZxg== X-Gm-Gg: ASbGncsvpH9nasJhmAQOjaB0wpXTfEHZszjXbL+zjPafhmJBmIExipUuPhWKqj3tlxf +5j7FaZzlO+Jn0/ffgFOC2VMMa+jgCrvc8dxQ3ud9PE4QF9sA1RBSg1t1lYsdWpzuVtTIXtwh55 ZJJMuE4GeWz8XJCjA0fTkdfQcLwpWFZ2byWux7QvW1mWoQTwJ9prt2s4j16GcRk1bH85XJ80PKp O4IybDG7/3HZkUGXMh2RehWB0EY4Yu3ygt8yCbm6LwXFnOfDI8CAK28IHcb0hSBAH0pNDdJFG0t yn2ovIvFrHgA5Kvi7Q== X-Google-Smtp-Source: AGHT+IHpYdhdLXmXOk5qBuPck2oehFerwvlvK33kiZAY6yd9iS5YnP5pPEjTkuad2sOYZEHR81MPJw== X-Received: by 2002:a05:6300:668b:b0:1e1:a0b6:9861 with SMTP id adf61e73a8af0-1ed7a5f6fc0mr3451155637.12.1738133377322; Tue, 28 Jan 2025 22:49:37 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:b323:d70b:a1b8:1683]) by smtp.gmail.com with UTF8SMTPSA id 41be03b00d2f7-ac495d58478sm9397597a12.54.2025.01.28.22.49.34 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 28 Jan 2025 22:49:37 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton , Minchan Kim , Johannes Weiner , Yosry Ahmed , Nhat Pham Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv1 6/6] zram: add might_sleep to zcomp API Date: Wed, 29 Jan 2025 15:43:52 +0900 Message-ID: <20250129064853.2210753-7-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.262.g85cc9f2d1e-goog In-Reply-To: <20250129064853.2210753-1-senozhatsky@chromium.org> References: <20250129064853.2210753-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Explicitly state that zcomp compress/decompress must be called from non-atomic context. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zcomp.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/drivers/block/zram/zcomp.c b/drivers/block/zram/zcomp.c index 675f2a51ad5f..f4235735787b 100644 --- a/drivers/block/zram/zcomp.c +++ b/drivers/block/zram/zcomp.c @@ -185,6 +185,7 @@ int zcomp_compress(struct zcomp *comp, struct zcomp_str= m *zstrm, }; int ret; =20 + might_sleep(); ret =3D comp->ops->compress(comp->params, &zstrm->ctx, &req); if (!ret) *dst_len =3D req.dst_len; @@ -201,6 +202,7 @@ int zcomp_decompress(struct zcomp *comp, struct zcomp_s= trm *zstrm, .dst_len =3D PAGE_SIZE, }; =20 + might_sleep(); return comp->ops->decompress(comp->params, &zstrm->ctx, &req); } =20 --=20 2.48.1.262.g85cc9f2d1e-goog