From nobody Fri Jan 31 00:18:34 2025 Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DC4C31FCCFB for ; Mon, 27 Jan 2025 08:03:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737965009; cv=none; b=neyGGpROW4hiQFPlWOTQjgHOVV1gnzWAalLtVZU+nkklvtj18Aj2hvWs2YUyJbkJHSQyh++jCY6HD6SfxvfSlFdPuJ2qR9t1xKTXteRpgt7kteJ678svtNswkpdOld7MJnhk5cfy2zic5iAh15Ysfa8EGW1iXkODo3clGpd88r0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737965009; c=relaxed/simple; bh=S+oej9yKPiXP1ZIDAZIqZ7EnK0E9uzxVjD1SpZpU2aY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=kRdpobZL6yFGYM5nJrqvdzie2cH5xviv6yJIlx2lVP4DggLLnVFW2R+26VH3SllMEjADkeOk3p6jX48gKffGoqMGq5ieh5U+A8xAfpJd3hUst0VaI8e6+r2GzWbWeSvINGzIyYD56PgrySfv1GGNydRwRlULvB/LVZLE0I6ZgN4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=U7UPrs8K; arc=none smtp.client-ip=209.85.214.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="U7UPrs8K" Received: by mail-pl1-f182.google.com with SMTP id d9443c01a7336-2166f1e589cso105743375ad.3 for ; Mon, 27 Jan 2025 00:03:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1737965007; x=1738569807; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=kn1+fRD/5z0gRUNjM1y0pmoc8sr5OaAxZlYjVZdB2hU=; b=U7UPrs8KkFC6wn5lYKtnwd8CJ0oHYCZU7h4Paf2d0qQFCH1l36ORxdjY82wCm3FV8L sZ5RlovGy3UXybkGuegCvju0kD2Zy6Hipq3agkeF5kWeUwPsLSvdo/OSWYTmRg2Z0MdO bPIn/IO7txq1UUF2xhNv5MHZlAwYdkIWG5AyI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1737965007; x=1738569807; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kn1+fRD/5z0gRUNjM1y0pmoc8sr5OaAxZlYjVZdB2hU=; b=jezuditOtU6LpF4empzvtgabFIfIEMr8Gscd3mtudKgINaE7jZCERNfVvkpiQ7joqZ MLxq6xijUwIugkdbkRXm4D0X2K0YN9jJ9r0LLx/JCVqZPRGvAPM0zDB2kL7T310r/Zxp RiOOuB2JbvM3HtPRUy9T4JnEcfCneIl3x6tmn9sabUtjO4yQlap4Byozz/pfu+k+hg+1 4n9SDFpZnO41yMJN1IeTw8jvlApZz3rMrGd4+Wk3Sg0Jiu9zMyEKPRZbuO/MRkhnyXgK MBpIbppjQ+TzUvzNdawND9YCp0WreSHXW56aFf/B1aKvuk6bGSuOjHrMw16WnOxM7sqJ B3Rw== X-Forwarded-Encrypted: i=1; AJvYcCVTHYrYp2lzN2aN6YcQK/bx9THyraKQGWNC7dKDyDnRZIN060jCW+lR6U4Zp9/OvCp2DwTebM6ERT2u/5o=@vger.kernel.org X-Gm-Message-State: AOJu0YzaQLu2eR6j5EWdqBNwOIG0P7YLwFqRsQ+n0/ELB5H10LvsAnQO c9oGqrl79YRJxWEgL8fUbY1487BKtK+q4kp1pRplZsbgfY1wsNaQNjG32tGCtQ== X-Gm-Gg: ASbGncuGvzVqH6YoZpN3zXQflEpxk5YAyP5j+DkuxlNstjhol2gBcVMVPYcMJAVoqi2 C+ZVEerF38+F+IoaeRlzlkPJgxAyfbqWbEXhemI4l3cuGQ6rrt5tNqON5w7x/1aECP0GoGRnKzc hChdyMUPOrsHa7fAxsQcq/BF20/MV63LMlJQk0z1ZLpRoqEt4iV3iSxjxhSDWucted0LSVM5PIO jXP8sK9gCvPixYx3OzI6abmUPwtiN0y0Gs2SdXmPjg+1GaV502p5RvDnwPegpt/iP0ehqoKyIW0 8Lsaf+E= X-Google-Smtp-Source: AGHT+IE0KtyysZw7cacIrE1leqh4Co/IJ/3WMyHKfL3/R554bfYapuUKAhoc+fS50PnMBkNIdu63xg== X-Received: by 2002:a17:902:ecc6:b0:217:803a:e47a with SMTP id d9443c01a7336-21c3553b21cmr580819105ad.4.1737965007128; Mon, 27 Jan 2025 00:03:27 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:566d:6152:c049:8d3a]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-21da3ea3081sm56832255ad.62.2025.01.27.00.03.24 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 27 Jan 2025 00:03:26 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton , Minchan Kim , Johannes Weiner , Yosry Ahmed , Nhat Pham Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [RFC PATCH 4/6] zsmalloc: make class lock sleepable Date: Mon, 27 Jan 2025 16:59:29 +0900 Message-ID: <20250127080254.1302026-5-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.262.g85cc9f2d1e-goog In-Reply-To: <20250127080254.1302026-1-senozhatsky@chromium.org> References: <20250127080254.1302026-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Switch over from spin-lock to mutex, also introduce simple helpers to lock/unlock size class. This is needed to make zsmalloc preemptible in the future. Signed-off-by: Sergey Senozhatsky --- mm/zsmalloc.c | 54 ++++++++++++++++++++++++++++----------------------- 1 file changed, 30 insertions(+), 24 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 751871ec533f..a5c1f9852072 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -168,7 +168,7 @@ static struct dentry *zs_stat_root; static size_t huge_class_size; =20 struct size_class { - spinlock_t lock; + struct mutex lock; struct list_head fullness_list[NR_FULLNESS_GROUPS]; /* * Size of objects stored in this class. Must be multiple @@ -252,6 +252,16 @@ static bool zspool_lock_is_contended(struct zs_pool *p= ool) return rwsem_is_contended(&pool->migrate_lock); } =20 +static void size_class_lock(struct size_class *class) +{ + mutex_lock(&class->lock); +} + +static void size_class_unlock(struct size_class *class) +{ + mutex_unlock(&class->lock); +} + static inline void zpdesc_set_first(struct zpdesc *zpdesc) { SetPagePrivate(zpdesc_page(zpdesc)); @@ -657,8 +667,7 @@ static int zs_stats_size_show(struct seq_file *s, void = *v) if (class->index !=3D i) continue; =20 - spin_lock(&class->lock); - + size_class_lock(class); seq_printf(s, " %5u %5u ", i, class->size); for (fg =3D ZS_INUSE_RATIO_10; fg < NR_FULLNESS_GROUPS; fg++) { inuse_totals[fg] +=3D class_stat_read(class, fg); @@ -668,7 +677,7 @@ static int zs_stats_size_show(struct seq_file *s, void = *v) obj_allocated =3D class_stat_read(class, ZS_OBJS_ALLOCATED); obj_used =3D class_stat_read(class, ZS_OBJS_INUSE); freeable =3D zs_can_compact(class); - spin_unlock(&class->lock); + size_class_unlock(class); =20 objs_per_zspage =3D class->objs_per_zspage; pages_used =3D obj_allocated / objs_per_zspage * @@ -926,8 +935,6 @@ static void __free_zspage(struct zs_pool *pool, struct = size_class *class, { struct zpdesc *zpdesc, *next; =20 - assert_spin_locked(&class->lock); - VM_BUG_ON(get_zspage_inuse(zspage)); VM_BUG_ON(zspage->fullness !=3D ZS_INUSE_RATIO_0); =20 @@ -1443,7 +1450,7 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t = size, gfp_t gfp) class =3D pool->size_class[get_size_class_index(size)]; =20 /* class->lock effectively protects the zpage migration */ - spin_lock(&class->lock); + size_class_lock(class); zspage =3D find_get_zspage(class); if (likely(zspage)) { obj_malloc(pool, zspage, handle); @@ -1453,8 +1460,7 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t = size, gfp_t gfp) =20 goto out; } - - spin_unlock(&class->lock); + size_class_unlock(class); =20 zspage =3D alloc_zspage(pool, class, gfp); if (!zspage) { @@ -1462,7 +1468,7 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t = size, gfp_t gfp) return (unsigned long)ERR_PTR(-ENOMEM); } =20 - spin_lock(&class->lock); + size_class_lock(class); obj_malloc(pool, zspage, handle); newfg =3D get_fullness_group(class, zspage); insert_zspage(class, zspage, newfg); @@ -1473,7 +1479,7 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t = size, gfp_t gfp) /* We completely set up zspage so mark them as movable */ SetZsPageMovable(pool, zspage); out: - spin_unlock(&class->lock); + size_class_unlock(class); =20 return handle; } @@ -1527,7 +1533,7 @@ void zs_free(struct zs_pool *pool, unsigned long hand= le) obj_to_zpdesc(obj, &f_zpdesc); zspage =3D get_zspage(f_zpdesc); class =3D zspage_class(pool, zspage); - spin_lock(&class->lock); + size_class_lock(class); pool_read_unlock(pool); =20 class_stat_sub(class, ZS_OBJS_INUSE, 1); @@ -1537,7 +1543,7 @@ void zs_free(struct zs_pool *pool, unsigned long hand= le) if (fullness =3D=3D ZS_INUSE_RATIO_0) free_zspage(pool, class, zspage); =20 - spin_unlock(&class->lock); + size_class_unlock(class); cache_free_handle(pool, handle); } EXPORT_SYMBOL_GPL(zs_free); @@ -1846,7 +1852,7 @@ static int zs_page_migrate(struct page *newpage, stru= ct page *page, /* * the class lock protects zpage alloc/free in the zspage. */ - spin_lock(&class->lock); + size_class_lock(class); /* the zspage_write_lock protects zpage access via zs_map_object */ zspage_write_lock(zspage); =20 @@ -1878,7 +1884,7 @@ static int zs_page_migrate(struct page *newpage, stru= ct page *page, * it's okay to release migration_lock. */ pool_write_unlock(pool); - spin_unlock(&class->lock); + size_class_unlock(class); zspage_write_unlock(zspage); =20 zpdesc_get(newzpdesc); @@ -1922,10 +1928,10 @@ static void async_free_zspage(struct work_struct *w= ork) if (class->index !=3D i) continue; =20 - spin_lock(&class->lock); + size_class_lock(class); list_splice_init(&class->fullness_list[ZS_INUSE_RATIO_0], &free_pages); - spin_unlock(&class->lock); + size_class_unlock(class); } =20 list_for_each_entry_safe(zspage, tmp, &free_pages, list) { @@ -1933,10 +1939,10 @@ static void async_free_zspage(struct work_struct *w= ork) lock_zspage(zspage); =20 class =3D zspage_class(pool, zspage); - spin_lock(&class->lock); + size_class_lock(class); class_stat_sub(class, ZS_INUSE_RATIO_0, 1); __free_zspage(pool, class, zspage); - spin_unlock(&class->lock); + size_class_unlock(class); } }; =20 @@ -2001,7 +2007,7 @@ static unsigned long __zs_compact(struct zs_pool *poo= l, * as well as zpage allocation/free */ pool_write_lock(pool); - spin_lock(&class->lock); + size_class_lock(class); while (zs_can_compact(class)) { int fg; =20 @@ -2031,11 +2037,11 @@ static unsigned long __zs_compact(struct zs_pool *p= ool, putback_zspage(class, dst_zspage); dst_zspage =3D NULL; =20 - spin_unlock(&class->lock); + size_class_unlock(class); pool_write_unlock(pool); cond_resched(); pool_write_lock(pool); - spin_lock(&class->lock); + size_class_lock(class); } } =20 @@ -2045,7 +2051,7 @@ static unsigned long __zs_compact(struct zs_pool *poo= l, if (dst_zspage) putback_zspage(class, dst_zspage); =20 - spin_unlock(&class->lock); + size_class_unlock(class); pool_write_unlock(pool); =20 return pages_freed; @@ -2255,7 +2261,7 @@ struct zs_pool *zs_create_pool(const char *name) class->index =3D i; class->pages_per_zspage =3D pages_per_zspage; class->objs_per_zspage =3D objs_per_zspage; - spin_lock_init(&class->lock); + mutex_init(&class->lock); pool->size_class[i] =3D class; =20 fullness =3D ZS_INUSE_RATIO_0; --=20 2.48.1.262.g85cc9f2d1e-goog