From nobody Wed Feb 11 08:33:20 2026 Received: from mail-pl1-f172.google.com (mail-pl1-f172.google.com [209.85.214.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8CCCD149C50 for ; Thu, 30 Jan 2025 04:45:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738212312; cv=none; b=LncvHpo3uBMprc9/I7ddLZSB0YN5NO0zlIQ3STrD001obJ5WYNYX3qTk5qeNEbc9QkR1GUtdWumPlIWPTIXw3la5BC74MgGm6+2Pqc6W6XqauU5oF5WU4mdgrXmkgTCKTcbQI9ZoqeXROj0Lpvqdn/wi+HTM+vTsRhcvGB/YCoY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738212312; c=relaxed/simple; bh=BziQv4iV5STepAfuHgjzd6hfIYwqdyff/nRvfV2Mdgw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=giUQTR96XgeFJX4Slwe/Bozzzd4YiV+d9RNnwQODbcvKW/uALD3b1RG9w9TdpjyBwQHiyuczx6sfng47HM6KltQ8HdDLfHAeZTVnYhpbeDSykVaD3iQtEPTWacANmsFvFrJbkcgxiZjqWjiwlWRfSG0SDjDPSFe5ZJJ/2jrWCHg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=RH0T0V/B; arc=none smtp.client-ip=209.85.214.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="RH0T0V/B" Received: by mail-pl1-f172.google.com with SMTP id d9443c01a7336-2164b1f05caso5695945ad.3 for ; Wed, 29 Jan 2025 20:45:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1738212309; x=1738817109; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=MkZn1kC+dMtAGDYo9TtmpyB1pyvIkC5NDATu7QnxsYw=; b=RH0T0V/BKaIB59ZUpFze0UZpkYBCFsQFpmmBJY82Y5lKVKE9O9GN1LC8SeszqJSmzv 5PfE4ZJ+4/Ij/UoxHUnEEu1XXLAI2H4z++EqeEQuHZCXN3psLruhclla7yv+xzNT5w2h jXa8g5F/WuFYs7ojlQAXeknHakORlzGe1UR6w= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738212309; x=1738817109; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=MkZn1kC+dMtAGDYo9TtmpyB1pyvIkC5NDATu7QnxsYw=; b=PEfH9G09IMLVcsgCCnoA1WvZN8PSuYxgfvjK9sYiQjTsJICTfM69M/Z6RcEakDgjQJ LvpiZpszoi8ECo3tudpwQDKiyUvX3X+uPjSP9eLxHS80RYg1Sv3YftDPG2CJvOfVNXji GzUJnNDmor9uUl8CR2svo+Gksen3m6QPypb2pkdqQ+sMTvJJpunbGJpZcq+J87oTYVb3 qT3sBVjHXlyAUnL1KJDuXt6an9aqIvQ/SLmB9xWC55h6tpQk/wghJPj/5WUxcDdiqatf 5/+cvliTU9qcEYY1aG8upUjQXtkGCIeLTrA7QSThP1lz5O46W1scoSIaYe7iyPe42cQl dwZg== X-Forwarded-Encrypted: i=1; AJvYcCVD/5N8pbL66P5eQCLezpOGNALympBByhDIRMX8Z451FV+421TZpFp3bw/9V6RVcc4hsr+EYN5tSYhkYlY=@vger.kernel.org X-Gm-Message-State: AOJu0YxCGJeqhBtbIs1qmoYRtM3EMbspetlhyZeC+RIzguWhvAQibz84 j1nkbxFeDlmNl70CKObEG+31kzg37/ZZkPWvtNxrheQSykIp2uBoSXVK5Ny4Ww== X-Gm-Gg: ASbGnctWdnyJ2rWFwGTXRFVYqxR7tY8IANFn7tQ3wuYkvKZF5BaVFQ9FeyabcNavqQT F3iAnrtNPWPpbAcVgSj+gG5nvqOhB46fSx1lbtUCUk8QDZctcO3Xos+7xNGn/sOWnyLcd/nUHXt ZVx1CZ29JyrVhi0zOXENtuZoXGPFW5/ZJTi9O3M1q+6BX6pKCjlBqTw9KQajl8XSLqamXkC723v 9/7aIQNwUBUhYughJx36p+Xv74fJy6b6wrWOLhV6yggo2HI0BaN6895NAoLdPF4GGI/3oDBu6BV tNaGIn4GwOX41jv9sA== X-Google-Smtp-Source: AGHT+IGZ3YC1dM+qQG3bGdWrkOZcidEoUlnu3m+rti9TiEBtmbAsSMQiEuLAx2BRY18SnHSZlarSww== X-Received: by 2002:a05:6a20:1584:b0:1e3:cf57:5f5f with SMTP id adf61e73a8af0-1ed7a6b290dmr9388630637.27.1738212308930; Wed, 29 Jan 2025 20:45:08 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:d794:9e7a:5186:857e]) by smtp.gmail.com with UTF8SMTPSA id 41be03b00d2f7-acebe8568fesm402564a12.40.2025.01.29.20.45.06 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 29 Jan 2025 20:45:08 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton , Yosry Ahmed Cc: Minchan Kim , Johannes Weiner , Nhat Pham , Uros Bizjak , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv2 1/7] zsmalloc: factor out pool locking helpers Date: Thu, 30 Jan 2025 13:42:44 +0900 Message-ID: <20250130044455.2642465-2-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.262.g85cc9f2d1e-goog In-Reply-To: <20250130044455.2642465-1-senozhatsky@chromium.org> References: <20250130044455.2642465-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" We currently have a mix of migrate_{read,write}_lock() helpers that lock zspages, but it's zs_pool that actually has a ->migrate_lock access to which is opene-coded. Factor out pool migrate locking into helpers, zspage migration locking API will be renamed to reduce confusion. Signed-off-by: Sergey Senozhatsky --- mm/zsmalloc.c | 47 ++++++++++++++++++++++++++++++++++++----------- 1 file changed, 36 insertions(+), 11 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 6d0e47f7ae33..8972b4b56cd3 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -228,6 +228,31 @@ struct zs_pool { atomic_t compaction_in_progress; }; =20 +static void pool_write_unlock(struct zs_pool *pool) +{ + write_unlock(&pool->migrate_lock); +} + +static void pool_write_lock(struct zs_pool *pool) +{ + write_lock(&pool->migrate_lock); +} + +static void pool_read_unlock(struct zs_pool *pool) +{ + read_unlock(&pool->migrate_lock); +} + +static void pool_read_lock(struct zs_pool *pool) +{ + read_lock(&pool->migrate_lock); +} + +static bool pool_lock_is_contended(struct zs_pool *pool) +{ + return rwlock_is_contended(&pool->migrate_lock); +} + static inline void zpdesc_set_first(struct zpdesc *zpdesc) { SetPagePrivate(zpdesc_page(zpdesc)); @@ -1206,7 +1231,7 @@ void *zs_map_object(struct zs_pool *pool, unsigned lo= ng handle, BUG_ON(in_interrupt()); =20 /* It guarantees it can get zspage from handle safely */ - read_lock(&pool->migrate_lock); + pool_read_lock(pool); obj =3D handle_to_obj(handle); obj_to_location(obj, &zpdesc, &obj_idx); zspage =3D get_zspage(zpdesc); @@ -1218,7 +1243,7 @@ void *zs_map_object(struct zs_pool *pool, unsigned lo= ng handle, * which is smaller granularity. */ migrate_read_lock(zspage); - read_unlock(&pool->migrate_lock); + pool_read_unlock(pool); =20 class =3D zspage_class(pool, zspage); off =3D offset_in_page(class->size * obj_idx); @@ -1453,13 +1478,13 @@ void zs_free(struct zs_pool *pool, unsigned long ha= ndle) * The pool->migrate_lock protects the race with zpage's migration * so it's safe to get the page from handle. */ - read_lock(&pool->migrate_lock); + pool_read_lock(pool); obj =3D handle_to_obj(handle); obj_to_zpdesc(obj, &f_zpdesc); zspage =3D get_zspage(f_zpdesc); class =3D zspage_class(pool, zspage); spin_lock(&class->lock); - read_unlock(&pool->migrate_lock); + pool_read_unlock(pool); =20 class_stat_sub(class, ZS_OBJS_INUSE, 1); obj_free(class->size, obj); @@ -1796,7 +1821,7 @@ static int zs_page_migrate(struct page *newpage, stru= ct page *page, * The pool migrate_lock protects the race between zpage migration * and zs_free. */ - write_lock(&pool->migrate_lock); + pool_write_lock(pool); class =3D zspage_class(pool, zspage); =20 /* @@ -1833,7 +1858,7 @@ static int zs_page_migrate(struct page *newpage, stru= ct page *page, * Since we complete the data copy and set up new zspage structure, * it's okay to release migration_lock. */ - write_unlock(&pool->migrate_lock); + pool_write_unlock(pool); spin_unlock(&class->lock); migrate_write_unlock(zspage); =20 @@ -1956,7 +1981,7 @@ static unsigned long __zs_compact(struct zs_pool *poo= l, * protect the race between zpage migration and zs_free * as well as zpage allocation/free */ - write_lock(&pool->migrate_lock); + pool_write_lock(pool); spin_lock(&class->lock); while (zs_can_compact(class)) { int fg; @@ -1983,14 +2008,14 @@ static unsigned long __zs_compact(struct zs_pool *p= ool, src_zspage =3D NULL; =20 if (get_fullness_group(class, dst_zspage) =3D=3D ZS_INUSE_RATIO_100 - || rwlock_is_contended(&pool->migrate_lock)) { + || pool_lock_is_contended(pool)) { putback_zspage(class, dst_zspage); dst_zspage =3D NULL; =20 spin_unlock(&class->lock); - write_unlock(&pool->migrate_lock); + pool_write_unlock(pool); cond_resched(); - write_lock(&pool->migrate_lock); + pool_write_lock(pool); spin_lock(&class->lock); } } @@ -2002,7 +2027,7 @@ static unsigned long __zs_compact(struct zs_pool *poo= l, putback_zspage(class, dst_zspage); =20 spin_unlock(&class->lock); - write_unlock(&pool->migrate_lock); + pool_write_unlock(pool); =20 return pages_freed; } --=20 2.48.1.262.g85cc9f2d1e-goog From nobody Wed Feb 11 08:33:20 2026 Received: from mail-pj1-f41.google.com (mail-pj1-f41.google.com [209.85.216.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6945B14F102 for ; Thu, 30 Jan 2025 04:45:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.41 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738212317; cv=none; b=BnESuGAeq0OSGDQzfyjo7jvei0kHHcJpjP846rgr+3ZUXmW/HVtFvvSFYKHhrS1nYmOQelzEuay/XwdNkhA/T89KsGLiUWihQFtI6bR7Mx4BPbD4Yv4q1Z0S+WGcCqKqaQ4rvehZ/OBxUzg0tNG8aPLB6JOBEHR7zmSD5jUK5YY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738212317; c=relaxed/simple; bh=mf8hb9oMGAcg6xZZocFsqBCJKRD7U4e7NciQTHCeCJk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=sS+YqSJR+o/3V6nPzxmTBCNKIwr8LP6aKZFWRR0aW0rvOScHU1vHCZQRrFnSm93N/kguMM2b7Ys2cxMea21ovvtmfNmNbC1OKS+ehY1Oa1TpnFUIZIDr3EQbmTTQt5e9NaICqa9Gx0fCMXZfhvDbmsysXflPkRZZoy6SvVAfXxs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=Tc2klEBG; arc=none smtp.client-ip=209.85.216.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="Tc2klEBG" Received: by mail-pj1-f41.google.com with SMTP id 98e67ed59e1d1-2ef8c012913so427934a91.3 for ; Wed, 29 Jan 2025 20:45:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1738212315; x=1738817115; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=qoanfGA+iKM2lkr04fEBeNmdyi5bnUfmaHtYhcyT7Yw=; b=Tc2klEBGL5ZPnOVYJaERNOol/ecdMnAh4PAIcnlDdkfNhm/1gALnZG75b2WEBLD3z+ T8zzAbOTg1LrbTPfzaqzSdT5F2ebixwXM+bwsjjkbu1TWlcBm0VdnFz+owMOqxMzM2kX clBRVRI9bfwWMbXOWY4CjYj9Op+Jj2M2XdKW4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738212315; x=1738817115; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=qoanfGA+iKM2lkr04fEBeNmdyi5bnUfmaHtYhcyT7Yw=; b=quq+EZ7lBFn/uCvIFdx4IPOdDQAwVfRw4wQQIIQGuyxMKURx0elmo2BrZziJNX61H2 FFAnJwpT3fMlLcaHmzdBd+cgJhojKHLfVwZAePsJoTgAZHoo6jS4jujDxThBmB0TZpUH O66JwyOcAKd08+tUVGfEdOyrQ5dwvLYKLOMDaVijNmr0B+vmEQNC0NJBOH/Gc3Pk04Ba boL6UCVNwwbd3yAlO3FtCjuwcdNTVQNtiTNhJmomILZJYUdK1Z7ryZqJ+gACT6gbWELJ O89tt8cdc6KPEJeAsSN2s8Pa9o8jMNF9x+b3V7EkKAk9XRmVxc1e/gtZczlpI9L/pcZ0 NWFg== X-Forwarded-Encrypted: i=1; AJvYcCVIfm8sXaqDvZvdYDrnfWML6/PhRmSOn9k5jJeweaEhWuU8HLNmMnxIFssR9dYS81ZhR4E24y4BAMsGLaU=@vger.kernel.org X-Gm-Message-State: AOJu0YyL7M0FMyq0ygZ1igXb6jtnNbT5vuwZ+p90UQE/BuGYqgz6S+3P U/LST9W9iiBkjbuFxldGtG7KBN4G45UK3+Uur3mtwlKXKr4CwZ7nltr6yYbvKw== X-Gm-Gg: ASbGncskxsE5m8hZj2xeNFc1sllnsvSeAgb123U+jp1uv/gUvq9OOC0V9e8AZe4gqfn bZHku1v7qi2uEKgVIO6Xl6eQ9A3jFyJgThtdqhKUb+/YDjU5/SArH2loJrBnXcypDt+BRv0npgX HSlm1A4AotLdkrRhIrkSLqSNdpF1OKeDCeuwWw/MnHIhPJuSaYBIJstQPVK5ouXIFgh8sHPoQ19 FK0Bq3ZtL+k1mtqgwZauXf+UlZzN8Um4nvsbbVeeX5XYjaXhLbM/EHqMCVX5RSZqsakeRYScnOF sV3zwMckF/hhNiEulQ== X-Google-Smtp-Source: AGHT+IF2UbQJQ89YEqryPhFxoeZX28FVlFBfWDCACr57M7BnO6sGVckgAr34ZiwemLJQw2ugQkpK1Q== X-Received: by 2002:a17:90b:5190:b0:2ee:ad18:b30d with SMTP id 98e67ed59e1d1-2f83abbd886mr7881534a91.6.1738212314683; Wed, 29 Jan 2025 20:45:14 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:d794:9e7a:5186:857e]) by smtp.gmail.com with UTF8SMTPSA id 98e67ed59e1d1-2f83bc98ae9sm3012541a91.10.2025.01.29.20.45.12 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 29 Jan 2025 20:45:14 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton , Yosry Ahmed Cc: Minchan Kim , Johannes Weiner , Nhat Pham , Uros Bizjak , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv2 2/7] zsmalloc: re-shuffle zs_pool members Date: Thu, 30 Jan 2025 13:42:45 +0900 Message-ID: <20250130044455.2642465-3-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.262.g85cc9f2d1e-goog In-Reply-To: <20250130044455.2642465-1-senozhatsky@chromium.org> References: <20250130044455.2642465-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" There is no need in having ->name at offset 0, it's never user and is being followed but a very large RO (after init) array of size classes, so each time we need to lock pool->migrate_lock (which happens relatively often) we need to jump over that RO region. Swap ->migrate_lock and ->name. Move ->pages_allocated is modified relatively often, move it up right after RO size_classes arrays. We modify ->compaction_in_progress and ->stats only from compaction (defragmentation), which can run only on one CPU at a time and, depending on the case, may or may not be common. Move both to down. Signed-off-by: Sergey Senozhatsky --- mm/zsmalloc.c | 20 +++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 8972b4b56cd3..2280ea17796b 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -204,18 +204,15 @@ struct link_free { }; =20 struct zs_pool { - const char *name; + /* protect page/zspage migration */ + rwlock_t migrate_lock; =20 struct size_class *size_class[ZS_SIZE_CLASSES]; - struct kmem_cache *handle_cachep; - struct kmem_cache *zspage_cachep; =20 atomic_long_t pages_allocated; =20 - struct zs_pool_stats stats; - - /* Compact classes */ - struct shrinker *shrinker; + struct kmem_cache *handle_cachep; + struct kmem_cache *zspage_cachep; =20 #ifdef CONFIG_ZSMALLOC_STAT struct dentry *stat_dentry; @@ -223,9 +220,14 @@ struct zs_pool { #ifdef CONFIG_COMPACTION struct work_struct free_work; #endif - /* protect page/zspage migration */ - rwlock_t migrate_lock; + const char *name; + + /* Compact classes */ + struct shrinker *shrinker; + /* Permit only once compaction at a time */ atomic_t compaction_in_progress; + /* compaction stats */ + struct zs_pool_stats stats; }; =20 static void pool_write_unlock(struct zs_pool *pool) --=20 2.48.1.262.g85cc9f2d1e-goog From nobody Wed Feb 11 08:33:20 2026 Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3B8E81547C9 for ; Thu, 30 Jan 2025 04:45:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.180 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738212322; cv=none; b=nf4RUQFEVIIuMzfrVf/d/tJvtVQiO1BkfEIU4Pq/MX72SWzhLk+xRl55m5pva8L5/odjrEvSqbh7xeMlA4fEsUR2mHBBdjtNJ5k8mZxJSPPKSlEPwootaihwST9l8Rs1OKmaMYY4jBZUDwEKwzlTB4lF/YUxlsTCK0fRyzm0eJ8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738212322; c=relaxed/simple; bh=0k23mwnYe/7vfQzpi+Blz+DQHj0FF3rrHJ6fxvyUseM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=t/srlwQnCYBa2gQl+Mtgvp418VTyXrQt3ZDQCecN5IcQLg6if+0TkTlXEXAbg1opo0UwBkRQKu3j3/5dXWQc7/pYX1+20AGFjkKhWg8MWAqJgov62vEz7oyzpUia8XNOECdkjqZhYqxDCC2OFmp9Lo3MrhgvUO2b2QefvvedCKM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=CQD14uPe; arc=none smtp.client-ip=209.85.214.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="CQD14uPe" Received: by mail-pl1-f180.google.com with SMTP id d9443c01a7336-2163dc5155fso5594875ad.0 for ; Wed, 29 Jan 2025 20:45:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1738212320; x=1738817120; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=9DZ2RsvFcvw71ALze2h8Qgc3kShnLK348TMZbsDChLU=; b=CQD14uPequKFGvZu5vA6LwYnH4tX2gxsXD2t9uxa8BRtbNY0H7S9+C9fTGbL5k3gv8 ENy2VKo/Q/XJpvmklM1VAzuTMQumeqAEsGoQ5+NLy/+TsROXZHbbri3lTvV77h0FxDwN xyCdVXAMcnA+fLG437DCU1BOchHB1OsCBWB14= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738212320; x=1738817120; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=9DZ2RsvFcvw71ALze2h8Qgc3kShnLK348TMZbsDChLU=; b=o6IkrMv8sFbLkc3hrDXVvDWz2l0Gof74lnN7BbU0+qHRKlECtT3d+lByldCEs+UNPX 0pdg6JSgegglA1w4xWqEMm79LXittw17044yGoZpzZDrK50pcB0TA8fNKFiY4+WCqosy esBts4nGeJgKbdu9T5OW5nH8sOcu1SNNI5tM6fiROxzZmA+hB8Dtl31DUfbdAgGAufPI Ogr1Ir79li8DP6taS0GyF0yC3D785X64D9U782/Irvw3iSUrO+15+5bCby2D8euZOMn0 tZOLMLJCuJLVehus8FJw0jf9qsmrA6zHGctk71epvgUQUPQA/y+Zg8dVgC5CrimTuT1K SC3w== X-Forwarded-Encrypted: i=1; AJvYcCVCMwIYxjP5VIO3aoXv7+dEAOe8o5Kmq9VnQQd49JcyGO6nDRGSb59JfeZPQlaJPiWr69hcad1F/7YBsAk=@vger.kernel.org X-Gm-Message-State: AOJu0Yw1dQNAU5VOcYEiV28BSJzELGUYfmv9W1Bs6KrvzvjFUuHHIr21 Tm8IZ1YNtjlzs2Ov4sDNxNWVBYXBLDPBx07tzF9vp8fx5PK2RuaStCxADImVJw== X-Gm-Gg: ASbGncskY1DhIIESZEAcAnUtVRbIJiPGJ1wS3gKsbP7VsbllxmOPXrvMa3yogE/HOes kaoXDEtDTu2s3Or/PqrvNkhejlti5NhKy709DDtqiqrwSFjYLrst4WYQTPx4l/2hdWTPV390H/A h8Wo7E55I4novf0N/ubUiniSNU1xezmWYNu1OIo9mYhtv/4yKrgl/vfy7DygMfjvpwFG3oKr7vZ 5RT/NYUk5JuYr651FwiS5TzsX3H/KcbwlZGhZwGODRulaRU2oT/K0gEu/KattNAKYIz0tHpMNcW Em56ZADKj6haRcc6YQ== X-Google-Smtp-Source: AGHT+IFN7QhZkttButj+44a8KQus5JiMx2Oq0RGEUpSgZ97hnkDVL1qKxutrj1FjlwXhcYZwTuD5rg== X-Received: by 2002:a05:6a21:9004:b0:1e1:9f24:2e4c with SMTP id adf61e73a8af0-1ed7a5f0665mr8510856637.16.1738212320451; Wed, 29 Jan 2025 20:45:20 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:d794:9e7a:5186:857e]) by smtp.gmail.com with UTF8SMTPSA id 41be03b00d2f7-acebe385085sm321668a12.25.2025.01.29.20.45.17 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 29 Jan 2025 20:45:20 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton , Yosry Ahmed Cc: Minchan Kim , Johannes Weiner , Nhat Pham , Uros Bizjak , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv2 3/7] zsmalloc: factor out size-class locking helpers Date: Thu, 30 Jan 2025 13:42:46 +0900 Message-ID: <20250130044455.2642465-4-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.262.g85cc9f2d1e-goog In-Reply-To: <20250130044455.2642465-1-senozhatsky@chromium.org> References: <20250130044455.2642465-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Move open-coded size-class locking to dedicated helpers. Signed-off-by: Sergey Senozhatsky Reviewed-by: Yosry Ahmed --- mm/zsmalloc.c | 47 ++++++++++++++++++++++++++++------------------- 1 file changed, 28 insertions(+), 19 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 2280ea17796b..9053777035af 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -255,6 +255,16 @@ static bool pool_lock_is_contended(struct zs_pool *poo= l) return rwlock_is_contended(&pool->migrate_lock); } =20 +static void size_class_lock(struct size_class *class) +{ + spin_lock(&class->lock); +} + +static void size_class_unlock(struct size_class *class) +{ + spin_unlock(&class->lock); +} + static inline void zpdesc_set_first(struct zpdesc *zpdesc) { SetPagePrivate(zpdesc_page(zpdesc)); @@ -615,8 +625,7 @@ static int zs_stats_size_show(struct seq_file *s, void = *v) if (class->index !=3D i) continue; =20 - spin_lock(&class->lock); - + size_class_lock(class); seq_printf(s, " %5u %5u ", i, class->size); for (fg =3D ZS_INUSE_RATIO_10; fg < NR_FULLNESS_GROUPS; fg++) { inuse_totals[fg] +=3D class_stat_read(class, fg); @@ -626,7 +635,7 @@ static int zs_stats_size_show(struct seq_file *s, void = *v) obj_allocated =3D class_stat_read(class, ZS_OBJS_ALLOCATED); obj_used =3D class_stat_read(class, ZS_OBJS_INUSE); freeable =3D zs_can_compact(class); - spin_unlock(&class->lock); + size_class_unlock(class); =20 objs_per_zspage =3D class->objs_per_zspage; pages_used =3D obj_allocated / objs_per_zspage * @@ -1401,7 +1410,7 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t = size, gfp_t gfp) class =3D pool->size_class[get_size_class_index(size)]; =20 /* class->lock effectively protects the zpage migration */ - spin_lock(&class->lock); + size_class_lock(class); zspage =3D find_get_zspage(class); if (likely(zspage)) { obj_malloc(pool, zspage, handle); @@ -1412,7 +1421,7 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t = size, gfp_t gfp) goto out; } =20 - spin_unlock(&class->lock); + size_class_unlock(class); =20 zspage =3D alloc_zspage(pool, class, gfp); if (!zspage) { @@ -1420,7 +1429,7 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t = size, gfp_t gfp) return (unsigned long)ERR_PTR(-ENOMEM); } =20 - spin_lock(&class->lock); + size_class_lock(class); obj_malloc(pool, zspage, handle); newfg =3D get_fullness_group(class, zspage); insert_zspage(class, zspage, newfg); @@ -1431,7 +1440,7 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t = size, gfp_t gfp) /* We completely set up zspage so mark them as movable */ SetZsPageMovable(pool, zspage); out: - spin_unlock(&class->lock); + size_class_unlock(class); =20 return handle; } @@ -1485,7 +1494,7 @@ void zs_free(struct zs_pool *pool, unsigned long hand= le) obj_to_zpdesc(obj, &f_zpdesc); zspage =3D get_zspage(f_zpdesc); class =3D zspage_class(pool, zspage); - spin_lock(&class->lock); + size_class_lock(class); pool_read_unlock(pool); =20 class_stat_sub(class, ZS_OBJS_INUSE, 1); @@ -1495,7 +1504,7 @@ void zs_free(struct zs_pool *pool, unsigned long hand= le) if (fullness =3D=3D ZS_INUSE_RATIO_0) free_zspage(pool, class, zspage); =20 - spin_unlock(&class->lock); + size_class_unlock(class); cache_free_handle(pool, handle); } EXPORT_SYMBOL_GPL(zs_free); @@ -1829,7 +1838,7 @@ static int zs_page_migrate(struct page *newpage, stru= ct page *page, /* * the class lock protects zpage alloc/free in the zspage. */ - spin_lock(&class->lock); + size_class_lock(class); /* the migrate_write_lock protects zpage access via zs_map_object */ migrate_write_lock(zspage); =20 @@ -1861,7 +1870,7 @@ static int zs_page_migrate(struct page *newpage, stru= ct page *page, * it's okay to release migration_lock. */ pool_write_unlock(pool); - spin_unlock(&class->lock); + size_class_unlock(class); migrate_write_unlock(zspage); =20 zpdesc_get(newzpdesc); @@ -1905,10 +1914,10 @@ static void async_free_zspage(struct work_struct *w= ork) if (class->index !=3D i) continue; =20 - spin_lock(&class->lock); + size_class_lock(class); list_splice_init(&class->fullness_list[ZS_INUSE_RATIO_0], &free_pages); - spin_unlock(&class->lock); + size_class_unlock(class); } =20 list_for_each_entry_safe(zspage, tmp, &free_pages, list) { @@ -1916,10 +1925,10 @@ static void async_free_zspage(struct work_struct *w= ork) lock_zspage(zspage); =20 class =3D zspage_class(pool, zspage); - spin_lock(&class->lock); + size_class_lock(class); class_stat_sub(class, ZS_INUSE_RATIO_0, 1); __free_zspage(pool, class, zspage); - spin_unlock(&class->lock); + size_class_unlock(class); } }; =20 @@ -1984,7 +1993,7 @@ static unsigned long __zs_compact(struct zs_pool *poo= l, * as well as zpage allocation/free */ pool_write_lock(pool); - spin_lock(&class->lock); + size_class_lock(class); while (zs_can_compact(class)) { int fg; =20 @@ -2014,11 +2023,11 @@ static unsigned long __zs_compact(struct zs_pool *p= ool, putback_zspage(class, dst_zspage); dst_zspage =3D NULL; =20 - spin_unlock(&class->lock); + size_class_unlock(class); pool_write_unlock(pool); cond_resched(); pool_write_lock(pool); - spin_lock(&class->lock); + size_class_lock(class); } } =20 @@ -2028,7 +2037,7 @@ static unsigned long __zs_compact(struct zs_pool *poo= l, if (dst_zspage) putback_zspage(class, dst_zspage); =20 - spin_unlock(&class->lock); + size_class_unlock(class); pool_write_unlock(pool); =20 return pages_freed; --=20 2.48.1.262.g85cc9f2d1e-goog From nobody Wed Feb 11 08:33:20 2026 Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 986FD15575C for ; Thu, 30 Jan 2025 04:45:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738212328; cv=none; b=opRfPtpHfpMFQbvtpHkho3JL9zZyhQbMHS2k3H9hIjadwKT9CkUhbjMZJmHDgc/e8NSzay91MVEAnUiAtcBZfQXgC1VB8mJ9WOyQxjsC08XqKDueiZasMYCRwJj72+C/KBjFCzIM0yIVEm6FWA3HXHxWtFIrYpE0vUUG3dYwPCo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738212328; c=relaxed/simple; bh=bBN3uqcIxfZxysLVVud1Gski6UXDpdWptNKobXsiPrE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=kfls80vnTN/R72tS3FbeMz5ULG64nJelqKbqDmkTvnzpnhSpW4kVlpIGKIN/yZtimpeniqmPABVmPQvGjT/OJsdTsUvOYdNCp5582qyWRxTB+32nIqYnJt29OH9Km0DktlaIKNQVfc1cP87SQt1WTXgOEfv8451xHqXHiSrwB+o= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=alEtrYLm; arc=none smtp.client-ip=209.85.214.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="alEtrYLm" Received: by mail-pl1-f171.google.com with SMTP id d9443c01a7336-21669fd5c7cso5197155ad.3 for ; Wed, 29 Jan 2025 20:45:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1738212326; x=1738817126; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=7b48F13IaZHEp77kXliQ7Ulgv7dkxFEkUKlwPQJd7i8=; b=alEtrYLmGuw6rQvg5UFMA3UrWHjzt+c3Mf0emqngLbbebjqxfA1Mc+/bw6REQH3vuY +pYjBkolIJodeNkBa/OzDcuN/8M7/zj4X6pJCfXRHKENkNsK+oWuJJBzkvKPQNWR26Dp u8DTOYgOifbgA/Cx58eKPt2fdOwhypeta9rmA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738212326; x=1738817126; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7b48F13IaZHEp77kXliQ7Ulgv7dkxFEkUKlwPQJd7i8=; b=MlDRmhSkDlJ7tuH7Sxr+aqxx0IjBiuPWsXLz8B9SV6xD+DwjsBNuEd8aiH7RZWFlB5 YpYV2zS/WCI0eBC5F4y7kfjm2fM66ewcewUQhTnHikksfOyg5ktVxrkWeO1BVaeh/dRz 7RWII5UvKZLpRgyVib4XOMDbv8vmDCJa9cffvx2Jt8y+GxjGQLEtfJlLF9JbjoXpunwY 4pBuPyhbOXm5C9y96JnnhG46VP3qG9qwnZdJ92YA7i9b0ixi56C7NvyLlZdUJ3zpCgO3 EginXFkDSGYspqTF+zaCV9uDec74h2OBhT9e+B0FRvYgZYk0GIE/nypwC6P38TUDPo0V bIAA== X-Forwarded-Encrypted: i=1; AJvYcCV7SwXvPJWN18gB91PqA0tbgq/934oli8HtnfB4sw2Ca7S411ojMsNRxOfa4t0pTi8pVkPsmi1NZLm9x1U=@vger.kernel.org X-Gm-Message-State: AOJu0YzMEv6EQ3HqMiasWyYLnRwEZ6fJ3MvQyicDXyxYlFrL/En+vW5W RiH2Ig/uw13Q7Kd7gG9+RnfZajq4F/+IfqqH2w45TpNF/KfZQGqSVloKI6m8/g== X-Gm-Gg: ASbGncsIyL/XmaLGRT+O6+n3FjZoxwLE80ZZT3ooX60t2RY6uA/ZKdy0hV7gXSU/kdx YhdBZNElRnkZk6E6/xtxN9Mq7FIGFUtLuBxBgZ8rBMxFwCygHFsL+tE6395FMk44SVs6cvXTo3J B7tZlu8BmpmG66DBQuodUadMZcYuNn/FPOYk6KLOu3cIlqmhYMnbZn8JpOQwhknGTrHVl31hFlK S38sMymEaPUVVDY+kwFQHtBtgR+/Xpm1qJMAS0pkbO2KIMUHclU+8I810WFA4wEaBS2ht+0TM5R fzyP0x7WesCY0b8EsQ== X-Google-Smtp-Source: AGHT+IGash5oYuGdcs755/3JeUad/VkRFkG5UkbWIaV1ZyGTPZK/ra3hdD962IUxBHUrw6ORscpOlA== X-Received: by 2002:a17:903:594:b0:216:36ff:ba33 with SMTP id d9443c01a7336-21dd7c67e06mr67812015ad.26.1738212325806; Wed, 29 Jan 2025 20:45:25 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:d794:9e7a:5186:857e]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-21de31ee137sm4551505ad.23.2025.01.29.20.45.23 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 29 Jan 2025 20:45:25 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton , Yosry Ahmed Cc: Minchan Kim , Johannes Weiner , Nhat Pham , Uros Bizjak , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv2 4/7] zsmalloc: make zspage lock preemptible Date: Thu, 30 Jan 2025 13:42:47 +0900 Message-ID: <20250130044455.2642465-5-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.262.g85cc9f2d1e-goog In-Reply-To: <20250130044455.2642465-1-senozhatsky@chromium.org> References: <20250130044455.2642465-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Switch over from rwlock_t to a atomic_t variable that takes negative value when the page is under migration, or positive values when the page is used by zsmalloc users (object map, etc.) Using a rwsem per-zspage is a little too memory heavy, a simple atomic_t should suffice. zspage lock is a leaf lock for zs_map_object(), where it's read-acquired. Since this lock now permits preemption extra care needs to be taken when it is write-acquired - all writers grab it in atomic context, so they cannot spin and wait for (potentially preempted) reader to unlock zspage. There are only two writers at this moment - migration and compaction. In both cases we use write-try-lock and bail out if zspage is read locked. Writers, on the other hand, never get preempted, so readers can spin waiting for the writer to unlock zspage. With this we can implement a preemptible object mapping. Signed-off-by: Sergey Senozhatsky --- mm/zsmalloc.c | 135 +++++++++++++++++++++++++++++++------------------- 1 file changed, 83 insertions(+), 52 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 9053777035af..d8cc8e2598cc 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -294,6 +294,9 @@ static inline void free_zpdesc(struct zpdesc *zpdesc) __free_page(page); } =20 +#define ZS_PAGE_UNLOCKED 0 +#define ZS_PAGE_WRLOCKED -1 + struct zspage { struct { unsigned int huge:HUGE_BITS; @@ -306,7 +309,7 @@ struct zspage { struct zpdesc *first_zpdesc; struct list_head list; /* fullness list */ struct zs_pool *pool; - rwlock_t lock; + atomic_t lock; }; =20 struct mapping_area { @@ -316,6 +319,59 @@ struct mapping_area { enum zs_mapmode vm_mm; /* mapping mode */ }; =20 +static void zspage_lock_init(struct zspage *zspage) +{ + atomic_set(&zspage->lock, ZS_PAGE_UNLOCKED); +} + +/* + * zspage lock permits preemption on the reader-side (there can be multiple + * readers). Writers (exclusive zspage ownership), on the other hand, are + * always run in atomic context and cannot spin waiting for a (potentially + * preempted) reader to unlock zspage. This, basically, means that writers + * can only call write-try-lock and must bail out if it didn't succeed. + * + * At the same time, writers cannot reschedule under zspage write-lock, + * so readers can spin waiting for the writer to unlock zspage. + */ +static void zspage_read_lock(struct zspage *zspage) +{ + atomic_t *lock =3D &zspage->lock; + int old =3D atomic_read(lock); + + do { + if (old =3D=3D ZS_PAGE_WRLOCKED) { + cpu_relax(); + old =3D atomic_read(lock); + continue; + } + } while (!atomic_try_cmpxchg(lock, &old, old + 1)); +} + +static void zspage_read_unlock(struct zspage *zspage) +{ + atomic_dec(&zspage->lock); +} + +static bool zspage_try_write_lock(struct zspage *zspage) +{ + atomic_t *lock =3D &zspage->lock; + int old =3D ZS_PAGE_UNLOCKED; + + preempt_disable(); + if (atomic_try_cmpxchg(lock, &old, ZS_PAGE_WRLOCKED)) + return true; + + preempt_enable(); + return false; +} + +static void zspage_write_unlock(struct zspage *zspage) +{ + atomic_set(&zspage->lock, ZS_PAGE_UNLOCKED); + preempt_enable(); +} + /* huge object: pages_per_zspage =3D=3D 1 && maxobj_per_zspage =3D=3D 1 */ static void SetZsHugePage(struct zspage *zspage) { @@ -327,12 +383,6 @@ static bool ZsHugePage(struct zspage *zspage) return zspage->huge; } =20 -static void migrate_lock_init(struct zspage *zspage); -static void migrate_read_lock(struct zspage *zspage); -static void migrate_read_unlock(struct zspage *zspage); -static void migrate_write_lock(struct zspage *zspage); -static void migrate_write_unlock(struct zspage *zspage); - #ifdef CONFIG_COMPACTION static void kick_deferred_free(struct zs_pool *pool); static void init_deferred_free(struct zs_pool *pool); @@ -1028,7 +1078,7 @@ static struct zspage *alloc_zspage(struct zs_pool *po= ol, return NULL; =20 zspage->magic =3D ZSPAGE_MAGIC; - migrate_lock_init(zspage); + zspage_lock_init(zspage); =20 for (i =3D 0; i < class->pages_per_zspage; i++) { struct zpdesc *zpdesc; @@ -1253,7 +1303,7 @@ void *zs_map_object(struct zs_pool *pool, unsigned lo= ng handle, * zs_unmap_object API so delegate the locking from class to zspage * which is smaller granularity. */ - migrate_read_lock(zspage); + zspage_read_lock(zspage); pool_read_unlock(pool); =20 class =3D zspage_class(pool, zspage); @@ -1313,7 +1363,7 @@ void zs_unmap_object(struct zs_pool *pool, unsigned l= ong handle) } local_unlock(&zs_map_area.lock); =20 - migrate_read_unlock(zspage); + zspage_read_unlock(zspage); } EXPORT_SYMBOL_GPL(zs_unmap_object); =20 @@ -1707,18 +1757,18 @@ static void lock_zspage(struct zspage *zspage) /* * Pages we haven't locked yet can be migrated off the list while we're * trying to lock them, so we need to be careful and only attempt to - * lock each page under migrate_read_lock(). Otherwise, the page we lock + * lock each page under zspage_read_lock(). Otherwise, the page we lock * may no longer belong to the zspage. This means that we may wait for * the wrong page to unlock, so we must take a reference to the page - * prior to waiting for it to unlock outside migrate_read_lock(). + * prior to waiting for it to unlock outside zspage_read_lock(). */ while (1) { - migrate_read_lock(zspage); + zspage_read_lock(zspage); zpdesc =3D get_first_zpdesc(zspage); if (zpdesc_trylock(zpdesc)) break; zpdesc_get(zpdesc); - migrate_read_unlock(zspage); + zspage_read_unlock(zspage); zpdesc_wait_locked(zpdesc); zpdesc_put(zpdesc); } @@ -1729,41 +1779,16 @@ static void lock_zspage(struct zspage *zspage) curr_zpdesc =3D zpdesc; } else { zpdesc_get(zpdesc); - migrate_read_unlock(zspage); + zspage_read_unlock(zspage); zpdesc_wait_locked(zpdesc); zpdesc_put(zpdesc); - migrate_read_lock(zspage); + zspage_read_lock(zspage); } } - migrate_read_unlock(zspage); + zspage_read_unlock(zspage); } #endif /* CONFIG_COMPACTION */ =20 -static void migrate_lock_init(struct zspage *zspage) -{ - rwlock_init(&zspage->lock); -} - -static void migrate_read_lock(struct zspage *zspage) __acquires(&zspage->l= ock) -{ - read_lock(&zspage->lock); -} - -static void migrate_read_unlock(struct zspage *zspage) __releases(&zspage-= >lock) -{ - read_unlock(&zspage->lock); -} - -static void migrate_write_lock(struct zspage *zspage) -{ - write_lock(&zspage->lock); -} - -static void migrate_write_unlock(struct zspage *zspage) -{ - write_unlock(&zspage->lock); -} - #ifdef CONFIG_COMPACTION =20 static const struct movable_operations zsmalloc_mops; @@ -1805,7 +1830,7 @@ static bool zs_page_isolate(struct page *page, isolat= e_mode_t mode) } =20 static int zs_page_migrate(struct page *newpage, struct page *page, - enum migrate_mode mode) + enum migrate_mode mode) { struct zs_pool *pool; struct size_class *class; @@ -1821,15 +1846,12 @@ static int zs_page_migrate(struct page *newpage, st= ruct page *page, =20 VM_BUG_ON_PAGE(!zpdesc_is_isolated(zpdesc), zpdesc_page(zpdesc)); =20 - /* We're committed, tell the world that this is a Zsmalloc page. */ - __zpdesc_set_zsmalloc(newzpdesc); - /* The page is locked, so this pointer must remain valid */ zspage =3D get_zspage(zpdesc); pool =3D zspage->pool; =20 /* - * The pool migrate_lock protects the race between zpage migration + * The pool->migrate_lock protects the race between zpage migration * and zs_free. */ pool_write_lock(pool); @@ -1839,8 +1861,15 @@ static int zs_page_migrate(struct page *newpage, str= uct page *page, * the class lock protects zpage alloc/free in the zspage. */ size_class_lock(class); - /* the migrate_write_lock protects zpage access via zs_map_object */ - migrate_write_lock(zspage); + /* the zspage write_lock protects zpage access via zs_map_object */ + if (!zspage_try_write_lock(zspage)) { + size_class_unlock(class); + pool_write_unlock(pool); + return -EINVAL; + } + + /* We're committed, tell the world that this is a Zsmalloc page. */ + __zpdesc_set_zsmalloc(newzpdesc); =20 offset =3D get_first_obj_offset(zpdesc); s_addr =3D kmap_local_zpdesc(zpdesc); @@ -1871,7 +1900,7 @@ static int zs_page_migrate(struct page *newpage, stru= ct page *page, */ pool_write_unlock(pool); size_class_unlock(class); - migrate_write_unlock(zspage); + zspage_write_unlock(zspage); =20 zpdesc_get(newzpdesc); if (zpdesc_zone(newzpdesc) !=3D zpdesc_zone(zpdesc)) { @@ -2007,9 +2036,11 @@ static unsigned long __zs_compact(struct zs_pool *po= ol, if (!src_zspage) break; =20 - migrate_write_lock(src_zspage); + if (!zspage_try_write_lock(src_zspage)) + break; + migrate_zspage(pool, src_zspage, dst_zspage); - migrate_write_unlock(src_zspage); + zspage_write_unlock(src_zspage); =20 fg =3D putback_zspage(class, src_zspage); if (fg =3D=3D ZS_INUSE_RATIO_0) { --=20 2.48.1.262.g85cc9f2d1e-goog From nobody Wed Feb 11 08:33:20 2026 Received: from mail-pl1-f178.google.com (mail-pl1-f178.google.com [209.85.214.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 95B8815381A for ; Thu, 30 Jan 2025 04:45:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738212334; cv=none; b=Pa/b0vaa2dS3dAuJ1KMNsyCpvyxni44zgDOaL5eYncsZdFTKzg5ODBDqYserkDhXSa4wEHYWPWSrgvEweOL+925NKcCWvXQlQvh638BY543Tx96qjz/Aqdy6X8fkZbizvfT2N9I0BuWp8vgubJ+22QsbWAN65KxXYlM5P8fmlVo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738212334; c=relaxed/simple; bh=xCuw2jrFXoMSpQY3luYSq9RLiwWs/XA7cmJVoraP5Qs=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=tpZhQtFcbR1hfOHVKrdqaineNrDTsGcj5Z42Pog/k2ta+uCltkddozvQdCQLC/bWgicfALDJ5gKSStUpITdPBpQsL6ymAtEf2D+M5mwlpmEO+qXRJUz6Qu4/1c/h42AD01VJNW3bJzKq44ZeCnouFXtVdujgxUS+lpd7njIIrGM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=Xe77Uv/J; arc=none smtp.client-ip=209.85.214.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="Xe77Uv/J" Received: by mail-pl1-f178.google.com with SMTP id d9443c01a7336-2167141dfa1so5072675ad.1 for ; Wed, 29 Jan 2025 20:45:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1738212332; x=1738817132; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=tPaya4S3Kfqs3oMtYONdXDrgcp/vC/ArYdL0b17yO2E=; b=Xe77Uv/JDxjX+Vv0LtsY3DaLJSIphbZpW7bt017TqFlqutL3fmcEnMvBN/L487uqa8 2UqSfJ6450JVHeKwvgA++2UTGtTTY5la0yjGexfZPO59Nw2wqlMQSQUdw+DSLigiMN5p ryTdQuHgNuyPbelfPuJDQ3rkZiEZtpD08Xq0Y= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738212332; x=1738817132; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=tPaya4S3Kfqs3oMtYONdXDrgcp/vC/ArYdL0b17yO2E=; b=JGj1Thkk5WmKbBCcAhEdfSUS5ftKmoBBerxmA3XHNf5LooDjYCxhVIRhHYlxc9roWx tPTClelejICxT9yQq6hkg8/KHIIUM8silNiszV0VpMpBwCeQZ8M8nZfPme7RyHfWQs80 1DiqrbQVUYHzc76fXMtmx4LnhTveogTK/Lf/EXExHTci7NQDDtc4YB7gSerUWH1nG1JC d1vlT3fUw/RFUI7jnJvu4taz45diK31kZSp6BKXGBog9qbcBeZL01HgvKyBAQe8LXpW5 J14mwBuOy3FTkRlgYcE5pE8w1pSLtdF4kRkKHi/c8OHYb6ozPe1XTJYMbUiV19CSZhA/ pshQ== X-Forwarded-Encrypted: i=1; AJvYcCWWqxTfFd0VcD5FcaHgXQfw7oRmq6vcGRGj2uwLAOXsJXk58DT9U6Tx4DV4L7bQbhg2T2avrr9T30dxOAA=@vger.kernel.org X-Gm-Message-State: AOJu0Yzc9kADUqzWBPa47LeMBPTAxgFjUBMN0dzx9jAnjr9156XjLO/i lW5VZy9JrDKmIP5Aed8rabUxo/GIfy7LVh/DB1hCKpxfSXM2S1bzB1OWvvGuug== X-Gm-Gg: ASbGncvb+AmnAxIIFtFm+DA6esCa9qs+0G14T1hAsri+rLrs9q9ncthQhBHp0pJhwam QlGhzia3p1qft+ndxV7F+/IhaoF0hg1VEdHzcYJk+G8wf1ukdS/7uTXZagK875gict7uWlwThz6 kgoMvpiA4ONVpczttnfWZpKbwnt3zCfnie7+waXU5RbpCnIRsrWkOBVuMGq0pRmblho+k06hjLg R/q9GsQbvzacrs4wWQ/zHkhB5K0LUKiXi/kuXnaN8hKuy2C7B7L2LZ8iN/u28iWA3C9S3U5EAyd zd6h6jwMtL7uOe/DUQ== X-Google-Smtp-Source: AGHT+IFnBawetG6OISjgDXcbQ2wf8X+tTg83rOrIMUBmE/TOfvysIil5FAzSXvjYS33jjv+nXChEoA== X-Received: by 2002:a17:903:41c3:b0:216:1079:82bb with SMTP id d9443c01a7336-21de195cd1emr27201265ad.19.1738212331708; Wed, 29 Jan 2025 20:45:31 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:d794:9e7a:5186:857e]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-21de31ee557sm4533535ad.1.2025.01.29.20.45.29 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 29 Jan 2025 20:45:31 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton , Yosry Ahmed Cc: Minchan Kim , Johannes Weiner , Nhat Pham , Uros Bizjak , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv2 5/7] zsmalloc: introduce new object mapping API Date: Thu, 30 Jan 2025 13:42:48 +0900 Message-ID: <20250130044455.2642465-6-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.262.g85cc9f2d1e-goog In-Reply-To: <20250130044455.2642465-1-senozhatsky@chromium.org> References: <20250130044455.2642465-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Current object mapping API is a little cumbersome. First, it's inconsistent, sometimes it returns with page-faults disabled and sometimes with page-faults enabled. Second, and most importantly, it enforces atomicity restrictions on its users. zs_map_object() has to return a liner object address which is not always possible because some objects span multiple physical (non-contiguous) pages. For such objects zsmalloc uses a per-CPU buffer to which object's data is copied before a pointer to that per-CPU buffer is returned back to the caller. This leads to another, final, issue - extra memcpy(). Since the caller gets a pointer to per-CPU buffer it can memcpy() data only to that buffer, and during zs_unmap_object() zsmalloc will memcpy() from that per-CPU buffer to physical pages that object in question spans across. New API splits functions by access mode: - zs_obj_read_begin(handle, local_copy) Returns a pointer to handle memory. For objects that span two physical pages a local_copy buffer is used to store object's data before the address is returned to the caller. Otherwise the object's page is kmap_local mapped directly. - zs_obj_read_end(handle, buf) Unmaps the page if it was kmap_local mapped by zs_obj_read_begin(). - zs_obj_write(handle, buf, len) Copies len-bytes from compression buffer to handle memory (takes care of objects that span two pages). This does not need any additional (e.g. per-CPU) buffers and writes the data directly to zsmalloc pool pages. The old API will stay around until the remaining users switch to the new one. After that we'll also remove zsmalloc per-CPU buffer and CPU hotplug handling. Signed-off-by: Sergey Senozhatsky Reviewed-by: Yosry Ahmed --- include/linux/zsmalloc.h | 8 +++ mm/zsmalloc.c | 130 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 138 insertions(+) diff --git a/include/linux/zsmalloc.h b/include/linux/zsmalloc.h index a48cd0ffe57d..7d70983cf398 100644 --- a/include/linux/zsmalloc.h +++ b/include/linux/zsmalloc.h @@ -58,4 +58,12 @@ unsigned long zs_compact(struct zs_pool *pool); unsigned int zs_lookup_class_index(struct zs_pool *pool, unsigned int size= ); =20 void zs_pool_stats(struct zs_pool *pool, struct zs_pool_stats *stats); + +void *zs_obj_read_begin(struct zs_pool *pool, unsigned long handle, + void *local_copy); +void zs_obj_read_end(struct zs_pool *pool, unsigned long handle, + void *handle_mem); +void zs_obj_write(struct zs_pool *pool, unsigned long handle, + void *handle_mem, size_t mem_len); + #endif diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index d8cc8e2598cc..67c934ed4be9 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -1367,6 +1367,136 @@ void zs_unmap_object(struct zs_pool *pool, unsigned= long handle) } EXPORT_SYMBOL_GPL(zs_unmap_object); =20 +void *zs_obj_read_begin(struct zs_pool *pool, unsigned long handle, + void *local_copy) +{ + struct zspage *zspage; + struct zpdesc *zpdesc; + unsigned long obj, off; + unsigned int obj_idx; + struct size_class *class; + void *addr; + + WARN_ON(in_interrupt()); + + /* Guarantee we can get zspage from handle safely */ + pool_read_lock(pool); + obj =3D handle_to_obj(handle); + obj_to_location(obj, &zpdesc, &obj_idx); + zspage =3D get_zspage(zpdesc); + + /* Make sure migration doesn't move any pages in this zspage */ + zspage_read_lock(zspage); + pool_read_unlock(pool); + + class =3D zspage_class(pool, zspage); + off =3D offset_in_page(class->size * obj_idx); + + if (off + class->size <=3D PAGE_SIZE) { + /* this object is contained entirely within a page */ + addr =3D kmap_local_zpdesc(zpdesc); + addr +=3D off; + } else { + size_t sizes[2]; + + /* this object spans two pages */ + sizes[0] =3D PAGE_SIZE - off; + sizes[1] =3D class->size - sizes[0]; + addr =3D local_copy; + + memcpy_from_page(addr, zpdesc_page(zpdesc), + off, sizes[0]); + zpdesc =3D get_next_zpdesc(zpdesc); + memcpy_from_page(addr + sizes[0], + zpdesc_page(zpdesc), + 0, sizes[1]); + } + + if (!ZsHugePage(zspage)) + addr +=3D ZS_HANDLE_SIZE; + + return addr; +} +EXPORT_SYMBOL_GPL(zs_obj_read_begin); + +void zs_obj_read_end(struct zs_pool *pool, unsigned long handle, + void *handle_mem) +{ + struct zspage *zspage; + struct zpdesc *zpdesc; + unsigned long obj, off; + unsigned int obj_idx; + struct size_class *class; + + obj =3D handle_to_obj(handle); + obj_to_location(obj, &zpdesc, &obj_idx); + zspage =3D get_zspage(zpdesc); + class =3D zspage_class(pool, zspage); + off =3D offset_in_page(class->size * obj_idx); + + if (off + class->size <=3D PAGE_SIZE) { + if (!ZsHugePage(zspage)) + off +=3D ZS_HANDLE_SIZE; + handle_mem -=3D off; + kunmap_local(handle_mem); + } + + zspage_read_unlock(zspage); +} +EXPORT_SYMBOL_GPL(zs_obj_read_end); + +void zs_obj_write(struct zs_pool *pool, unsigned long handle, + void *handle_mem, size_t mem_len) +{ + struct zspage *zspage; + struct zpdesc *zpdesc; + unsigned long obj, off; + unsigned int obj_idx; + struct size_class *class; + + WARN_ON(in_interrupt()); + + /* Guarantee we can get zspage from handle safely */ + pool_read_lock(pool); + obj =3D handle_to_obj(handle); + obj_to_location(obj, &zpdesc, &obj_idx); + zspage =3D get_zspage(zpdesc); + + /* Make sure migration doesn't move any pages in this zspage */ + zspage_read_lock(zspage); + pool_read_unlock(pool); + + class =3D zspage_class(pool, zspage); + off =3D offset_in_page(class->size * obj_idx); + + if (off + class->size <=3D PAGE_SIZE) { + /* this object is contained entirely within a page */ + void *dst =3D kmap_local_zpdesc(zpdesc); + + if (!ZsHugePage(zspage)) + off +=3D ZS_HANDLE_SIZE; + memcpy(dst + off, handle_mem, mem_len); + kunmap_local(dst); + } else { + /* this object spans two pages */ + size_t sizes[2]; + + off +=3D ZS_HANDLE_SIZE; + + sizes[0] =3D PAGE_SIZE - off; + sizes[1] =3D mem_len - sizes[0]; + + memcpy_to_page(zpdesc_page(zpdesc), off, + handle_mem, sizes[0]); + zpdesc =3D get_next_zpdesc(zpdesc); + memcpy_to_page(zpdesc_page(zpdesc), 0, + handle_mem + sizes[0], sizes[1]); + } + + zspage_read_unlock(zspage); +} +EXPORT_SYMBOL_GPL(zs_obj_write); + /** * zs_huge_class_size() - Returns the size (in bytes) of the first huge * zsmalloc &size_class. --=20 2.48.1.262.g85cc9f2d1e-goog From nobody Wed Feb 11 08:33:20 2026 Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6BC5038DD3 for ; Thu, 30 Jan 2025 04:45:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738212339; cv=none; b=PWRmaiy1f/GTcE9rPsvvGiuQcqxHlfnvuMpRdKVjjtQepg6u5jWeWzWegmvbESSMaEURsP/f8AF7naree37OmDmUz1lkhO71sS1RTO+92nkVNKwfho/YzrCw8D5kONoHew8yvxmbI72U6n5nk4z3CvW2zM+xLWBT7HbE/0Lbtpk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738212339; c=relaxed/simple; bh=b8wby6Mnz0jOlZOj4F5r8wWAxtOJ93uxJ6l+SPSOOYI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=JrRJ4nU3yWxBQBRP3YIPb3hu3kGuJxr99jqkhHephgj+WFWUQ9XuI4cSUZDanDNHJ9+E4U3A5/nXX8APMCNIhA2NWNRN6GMTYmZPS0a0ahGHAB3lpnn161xqGT9PBy53i+d2dACAmGi9/WHnIjfm/x9U/p2Szg+wnDTDEHdm/eE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=f3qDZNPk; arc=none smtp.client-ip=209.85.214.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="f3qDZNPk" Received: by mail-pl1-f171.google.com with SMTP id d9443c01a7336-2166022c5caso4765505ad.2 for ; Wed, 29 Jan 2025 20:45:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1738212337; x=1738817137; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=jm+b3R/xkgu1u5aIyHjEy9+3hXzV4kVBwpygpgrob3Q=; b=f3qDZNPki8xjA716pcRiJ+I9oDDhnVCBz4QBoWxelR/xzVfzu3sTTMjgdOdiwLe/iX uWzLQWT5WBHgxE1506JUdukbKcUCI1G4FWhhsxqB9AOdyNEp9cIoL2Yv+ox6Ti/vsvIG wTYLASIimQ7EF/ym9ZaemzSoDG3dg+fWEOZ3s= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738212337; x=1738817137; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=jm+b3R/xkgu1u5aIyHjEy9+3hXzV4kVBwpygpgrob3Q=; b=lHEDM784/7/yuvrTDILAX1stwovr53eNn5GTb89bJHHO58o5pLOyllS1+ud60epA7N 4TXj+NHC56l5rIzzqI1fDm88JUcMN89aJ/RLHuPLnwR92PtALxfUQOu7EKKPTqx4Iwe+ Rdu5xavAXhND+8VWPXmNb8wuv6eIowxhtKxLwrHIy79Pxe6LC9VO2I5wMpWddEGVNWHw SxLJwyqNAYt9/BzmLkFkaNgOUkEuN5NfAbLRopU4o3zL6skDUuUXaLcDoB4FLIo3GZSj 6nWSsQfUInPWMTT4CCLLrB2uxfcidPRVYZgmYWl3SfOB7VXqD1I64Zl3VoJTQc8fxk0x /q7Q== X-Forwarded-Encrypted: i=1; AJvYcCU/UJ8lH5mg4akgp800G54j1RncRwe1mLEpNxB7iScSmzjbJkdEeS2LoaQ2jhbFvbSY17W5J5ejRzOIiOQ=@vger.kernel.org X-Gm-Message-State: AOJu0YwX/9jBI+jHUA92A3gdmfmMtc1Mr+3lQvCvsoxycL7/dmmUfvsY iRs4eO9gDoxD7gNQaH/wntp4M3mev8JbuTThPlAj5puEz31eVTCf0adEAXF/nQ== X-Gm-Gg: ASbGncsbPf2jmxY0KP2NZLIeCeKUCAuSW1CGZt+CLyGZIle3ZvDztOmQM/p5t2YP527 JcAIPtC3v86UPhUqToTcqp++zn3Z6ZmlCpuitYB4YlWedEQiFBCeG3H5IAtDU5aSyNppBEbs8e3 DwxdBAVf6/6aga7M3MEvEFj6gFjB742Ujr1Rshd0YO+hzdndy5MzJr/G+7nCMtYPe/C1lPwNvnJ BY4SY1NEHl39PLUFQLhjVF9pu4lbnGXqAO5Sow3D036oszXDc26ZR37Wm3PWw3Zp7DJquU7Zd4T abZ383QK/GlLvqm7WA== X-Google-Smtp-Source: AGHT+IHUyr8/2Fq8AovilKRJ28NBYk+8bvRw/3TGJlSrvnfXxKWiWthlwKdxoZwtf3TrFQyikvGQUQ== X-Received: by 2002:a17:903:594:b0:216:36ff:ba33 with SMTP id d9443c01a7336-21dd7c67e06mr67817495ad.26.1738212336948; Wed, 29 Jan 2025 20:45:36 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:d794:9e7a:5186:857e]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-21de33000e0sm4566895ad.167.2025.01.29.20.45.34 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 29 Jan 2025 20:45:36 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton , Yosry Ahmed Cc: Minchan Kim , Johannes Weiner , Nhat Pham , Uros Bizjak , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv2 6/7] zram: switch to new zsmalloc object mapping API Date: Thu, 30 Jan 2025 13:42:49 +0900 Message-ID: <20250130044455.2642465-7-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.262.g85cc9f2d1e-goog In-Reply-To: <20250130044455.2642465-1-senozhatsky@chromium.org> References: <20250130044455.2642465-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Use new read/write zsmalloc object API. For cases when RO mapped object spans two physical pages (requires temp buffer) compression streams now carry around one extra physical page. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zcomp.c | 4 +++- drivers/block/zram/zcomp.h | 2 ++ drivers/block/zram/zram_drv.c | 28 ++++++++++------------------ 3 files changed, 15 insertions(+), 19 deletions(-) diff --git a/drivers/block/zram/zcomp.c b/drivers/block/zram/zcomp.c index efd5919808d9..675f2a51ad5f 100644 --- a/drivers/block/zram/zcomp.c +++ b/drivers/block/zram/zcomp.c @@ -45,6 +45,7 @@ static const struct zcomp_ops *backends[] =3D { static void zcomp_strm_free(struct zcomp *comp, struct zcomp_strm *strm) { comp->ops->destroy_ctx(&strm->ctx); + vfree(strm->local_copy); vfree(strm->buffer); kfree(strm); } @@ -66,12 +67,13 @@ static struct zcomp_strm *zcomp_strm_alloc(struct zcomp= *comp) return NULL; } =20 + strm->local_copy =3D vzalloc(PAGE_SIZE); /* * allocate 2 pages. 1 for compressed data, plus 1 extra in case if * compressed data is larger than the original one. */ strm->buffer =3D vzalloc(2 * PAGE_SIZE); - if (!strm->buffer) { + if (!strm->buffer || !strm->local_copy) { zcomp_strm_free(comp, strm); return NULL; } diff --git a/drivers/block/zram/zcomp.h b/drivers/block/zram/zcomp.h index 62330829db3f..9683d4aa822d 100644 --- a/drivers/block/zram/zcomp.h +++ b/drivers/block/zram/zcomp.h @@ -34,6 +34,8 @@ struct zcomp_strm { struct list_head entry; /* compression buffer */ void *buffer; + /* local copy of handle memory */ + void *local_copy; struct zcomp_ctx ctx; }; =20 diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index ad3e8885b0d2..d73e8374e9cc 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1562,11 +1562,11 @@ static int read_incompressible_page(struct zram *zr= am, struct page *page, void *src, *dst; =20 handle =3D zram_get_handle(zram, index); - src =3D zs_map_object(zram->mem_pool, handle, ZS_MM_RO); + src =3D zs_obj_read_begin(zram->mem_pool, handle, NULL); dst =3D kmap_local_page(page); copy_page(dst, src); kunmap_local(dst); - zs_unmap_object(zram->mem_pool, handle); + zs_obj_read_end(zram->mem_pool, handle, src); =20 return 0; } @@ -1584,11 +1584,11 @@ static int read_compressed_page(struct zram *zram, = struct page *page, u32 index) prio =3D zram_get_priority(zram, index); =20 zstrm =3D zcomp_stream_get(zram->comps[prio]); - src =3D zs_map_object(zram->mem_pool, handle, ZS_MM_RO); + src =3D zs_obj_read_begin(zram->mem_pool, handle, zstrm->local_copy); dst =3D kmap_local_page(page); ret =3D zcomp_decompress(zram->comps[prio], zstrm, src, size, dst); kunmap_local(dst); - zs_unmap_object(zram->mem_pool, handle); + zs_obj_read_end(zram->mem_pool, handle, src); zcomp_stream_put(zram->comps[prio], zstrm); =20 return ret; @@ -1684,7 +1684,7 @@ static int write_incompressible_page(struct zram *zra= m, struct page *page, u32 index) { unsigned long handle; - void *src, *dst; + void *src; =20 /* * This function is called from preemptible context so we don't need @@ -1701,11 +1701,9 @@ static int write_incompressible_page(struct zram *zr= am, struct page *page, return -ENOMEM; } =20 - dst =3D zs_map_object(zram->mem_pool, handle, ZS_MM_WO); src =3D kmap_local_page(page); - memcpy(dst, src, PAGE_SIZE); + zs_obj_write(zram->mem_pool, handle, src, PAGE_SIZE); kunmap_local(src); - zs_unmap_object(zram->mem_pool, handle); =20 zram_slot_write_lock(zram, index); zram_set_flag(zram, index, ZRAM_HUGE); @@ -1726,7 +1724,7 @@ static int zram_write_page(struct zram *zram, struct = page *page, u32 index) int ret =3D 0; unsigned long handle; unsigned int comp_len; - void *dst, *mem; + void *mem; struct zcomp_strm *zstrm; unsigned long element; bool same_filled; @@ -1769,11 +1767,8 @@ static int zram_write_page(struct zram *zram, struct= page *page, u32 index) return -ENOMEM; } =20 - dst =3D zs_map_object(zram->mem_pool, handle, ZS_MM_WO); - - memcpy(dst, zstrm->buffer, comp_len); + zs_obj_write(zram->mem_pool, handle, zstrm->buffer, comp_len); zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP], zstrm); - zs_unmap_object(zram->mem_pool, handle); =20 zram_slot_write_lock(zram, index); zram_set_handle(zram, index, handle); @@ -1882,7 +1877,7 @@ static int recompress_slot(struct zram *zram, u32 ind= ex, struct page *page, unsigned int class_index_old; unsigned int class_index_new; u32 num_recomps =3D 0; - void *src, *dst; + void *src; int ret; =20 handle_old =3D zram_get_handle(zram, index); @@ -2020,12 +2015,9 @@ static int recompress_slot(struct zram *zram, u32 in= dex, struct page *page, return 0; } =20 - dst =3D zs_map_object(zram->mem_pool, handle_new, ZS_MM_WO); - memcpy(dst, zstrm->buffer, comp_len_new); + zs_obj_write(zram->mem_pool, handle_new, zstrm->buffer, comp_len_new); zcomp_stream_put(zram->comps[prio], zstrm); =20 - zs_unmap_object(zram->mem_pool, handle_new); - zram_free_page(zram, index); zram_set_handle(zram, index, handle_new); zram_set_obj_size(zram, index, comp_len_new); --=20 2.48.1.262.g85cc9f2d1e-goog From nobody Wed Feb 11 08:33:20 2026 Received: from mail-pj1-f49.google.com (mail-pj1-f49.google.com [209.85.216.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3F2F614293 for ; Thu, 30 Jan 2025 04:45:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.49 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738212345; cv=none; b=OUshBSDF8f7aIv57iteGjmiCv9kn//WwtcSDq5vAgqGFl6w4zCPz2Hs3azrQVKp4V0ljZ0MQkM9jBHarM4jU4hNSl4KPYTE9It+Hp0jT7e/v4bKoplzYQvw0wgXJNaQ1i/jHZf5UAU5blNV22qRyKVAGHPhhOhfC/ed88uPUo1s= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738212345; c=relaxed/simple; bh=kQOe19FMRhzWZQFPSQJ2h1HjTwoC6pXZ4Qv7RQgvqrg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=OFQ/dNtxEvy7CrEYWQbrSCkcm7eMZ5lKFgN5RYQGRHZ2uEkqaFDsDJETVvulb08NqT4TSnCnwKp9ZDWgk3ALrvapBQcu62VTlRxEe5ld0veI0SXStHoEz3CTBu9GdrtJdcT2O/TqA0YaaXSp1W58Iw+H7cv19+YD9vPmtzXS57I= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=F+eTlotq; arc=none smtp.client-ip=209.85.216.49 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="F+eTlotq" Received: by mail-pj1-f49.google.com with SMTP id 98e67ed59e1d1-2ee8aa26415so576718a91.1 for ; Wed, 29 Jan 2025 20:45:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1738212342; x=1738817142; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=LjUByDm3ba8JrZJ2xgEVKiM3B6T3dmGtNNYJyOcrU3Y=; b=F+eTlotquju/yGzTNB3Z3pmZsr3B/m8cjhRR9eMEhn4SHaN8LrN0xpBlHOd/3Wfzms eQH3KIUbqluO/chKooONiXu3FPoK25N0FHYaXw7S6x8hx3QcZSGBTfkv8XDjjiLoimEJ IJ9CmwH33yAwDIbtLTqcjdCNIjNDYoih37q+k= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738212342; x=1738817142; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=LjUByDm3ba8JrZJ2xgEVKiM3B6T3dmGtNNYJyOcrU3Y=; b=aftu54CCAtTBiaz3rPHzwMoP/AIWurcyUvsteCvC0QgueDIUUqW4xvKLGDBUNe7APM cy8u8lU9VBtPqAryPLRdKBdGHbj+/1z/MkjVUFO1h6603mnUw02nlSiE3qlwb8jZoVtw ojwF2opgRppT8UcwFVPMie4m+W7CwGbObj77Eyw2ddDCcukJKJCtSa0rmO6kb7+9NFSc KlGogLdFsTsGXZF+fEoGlZ9VKgDguJWBpu/VNvtQ6hoIIWIMvcqL6qmneHlfrM1lAudC 9t/7ekcHW+8PQjcMXa4gioT/Rgh3KBvgBQovZptomNPcRc14LY5mcPZdOf+lSNlUzItj TWLQ== X-Forwarded-Encrypted: i=1; AJvYcCV42sfZC8y8VkO6845wIVXBQxxkAWxlqxDzjPzPszwuHvWRg4nzCxxGH8cv7X8gyAPchhE1rplHKQcf2ZI=@vger.kernel.org X-Gm-Message-State: AOJu0Yyt3Jk5Ii4GG/AkW8vKrpyh+/eSxlCCUI2GMssSJYwhGX/pcYMa CsaJFJzFURX46djZI0YUrCgaQttaEE++qvBu+cXftHIjqCAav5eY2m/P/hN4Fg== X-Gm-Gg: ASbGncvhTRZY2YXK43q4jG+gZoLhpDz+LN8zOyo1F8j7wBvTUuH4Vej8p0/gdGiln/W gegeKRC6B9RytNNO9N5sGP6JQfhq9upD0DsCmPSXxPZNrybMCegUX7gtqDvRTFbFONqc/NOAIGC 9K7odilpxE3xkCol1x4zq440vJNHLtQUb/PYKljBIVbHB/ZY3Ak+WDVJHX66hYzKML5agFj1GZS yXxPvmMzm0nC6CacohazqxCAfyE4BSZ26KNhLPe3ElaYWbnooyN/iHHnPYkuI2TAJeBZv8KJtly vmSkt9pl8+PZj7/gEQ== X-Google-Smtp-Source: AGHT+IGkGP9NGStcdcyDxCKvmSx/umN39Bj6YAsUZh4NBQvKyyiQWE/6vXXCCD2x2gkOQQ39AR/qSQ== X-Received: by 2002:a17:90b:288a:b0:2ee:d35c:3996 with SMTP id 98e67ed59e1d1-2f83ac8371amr8132916a91.31.1738212342552; Wed, 29 Jan 2025 20:45:42 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:d794:9e7a:5186:857e]) by smtp.gmail.com with UTF8SMTPSA id 98e67ed59e1d1-2f83a194a0csm2205072a91.1.2025.01.29.20.45.40 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 29 Jan 2025 20:45:42 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton , Yosry Ahmed Cc: Minchan Kim , Johannes Weiner , Nhat Pham , Uros Bizjak , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv2 7/7] zram: add might_sleep to zcomp API Date: Thu, 30 Jan 2025 13:42:50 +0900 Message-ID: <20250130044455.2642465-8-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.262.g85cc9f2d1e-goog In-Reply-To: <20250130044455.2642465-1-senozhatsky@chromium.org> References: <20250130044455.2642465-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Explicitly state that zcomp compress/decompress must be called from non-atomic context. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zcomp.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/drivers/block/zram/zcomp.c b/drivers/block/zram/zcomp.c index 675f2a51ad5f..f4235735787b 100644 --- a/drivers/block/zram/zcomp.c +++ b/drivers/block/zram/zcomp.c @@ -185,6 +185,7 @@ int zcomp_compress(struct zcomp *comp, struct zcomp_str= m *zstrm, }; int ret; =20 + might_sleep(); ret =3D comp->ops->compress(comp->params, &zstrm->ctx, &req); if (!ret) *dst_len =3D req.dst_len; @@ -201,6 +202,7 @@ int zcomp_decompress(struct zcomp *comp, struct zcomp_s= trm *zstrm, .dst_len =3D PAGE_SIZE, }; =20 + might_sleep(); return comp->ops->decompress(comp->params, &zstrm->ctx, &req); } =20 --=20 2.48.1.262.g85cc9f2d1e-goog