From nobody Sun Feb 8 01:33:05 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 031CBC83005 for ; Fri, 9 Jun 2023 08:18:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240429AbjFIISS (ORCPT ); Fri, 9 Jun 2023 04:18:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51724 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240224AbjFIIRl (ORCPT ); Fri, 9 Jun 2023 04:17:41 -0400 Received: from out-63.mta0.migadu.com (out-63.mta0.migadu.com [91.218.175.63]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E122730D1 for ; Fri, 9 Jun 2023 01:17:08 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1686298623; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=50XF484pkGZZlvaX6wjD3S7ullpcs6Au3jWyE87o9lo=; b=VJW/sTjlrsD06VeMs5MWZHOINSVeIuvUVV3XeMcdlaZgU6yHH1KDKeoDlQ0uNJ8ow7jI5i C2f1mQy0VShusDu7ScqyZuVT65khmY5y8dLoApSOfxCjNpnx2aeqmPB13WMTVIUTOMDiTn dvzcgxnTuwP9thnsK4Ff5TbiUeHu3ak= From: Qi Zheng To: akpm@linux-foundation.org Cc: david@fromorbit.com, tkhai@ya.ru, roman.gushchin@linux.dev, vbabka@suse.cz, muchun.song@linux.dev, yujie.liu@intel.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH 1/7] Revert "mm: shrinkers: convert shrinker_rwsem to mutex" Date: Fri, 9 Jun 2023 08:15:12 +0000 Message-Id: <20230609081518.3039120-2-qi.zheng@linux.dev> In-Reply-To: <20230609081518.3039120-1-qi.zheng@linux.dev> References: <20230609081518.3039120-1-qi.zheng@linux.dev> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Migadu-Flow: FLOW_OUT Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Qi Zheng This reverts commit cf2e309ebca7bb0916771839f9b580b06c778530. Kernel test robot reports -88.8% regression in stress-ng.ramfs.ops_per_sec test case [1], which is caused by commit f95bdb700bc6 ("mm: vmscan: make global slab shrink lockless"). The root cause is that SRCU has to be careful to not frequently check for SRCU read-side critical section exits. Therefor= e, even if no one is currently in the SRCU read-side critical section, synchronize_srcu() cannot return quickly. That's why unregister_shrinker() has become slower. After discussion, we will try to use the refcount+RCU method [2] proposed by Dave Chinner to continue to re-implement the lockless slab shrink. So revert the shrinker_mutex back to shrinker_rwsem first. [1]. https://lore.kernel.org/lkml/202305230837.db2c233f-yujie.liu@intel.com/ [2]. https://lore.kernel.org/lkml/ZIJhou1d55d4H1s0@dread.disaster.area/ Reported-by: kernel test robot Closes: https://lore.kernel.org/oe-lkp/202305230837.db2c233f-yujie.liu@inte= l.com Signed-off-by: Qi Zheng --- drivers/md/dm-cache-metadata.c | 2 +- drivers/md/dm-thin-metadata.c | 2 +- fs/super.c | 2 +- mm/shrinker_debug.c | 14 +++++++------- mm/vmscan.c | 34 +++++++++++++++++----------------- 5 files changed, 27 insertions(+), 27 deletions(-) diff --git a/drivers/md/dm-cache-metadata.c b/drivers/md/dm-cache-metadata.c index 9e0c69958587..acffed750e3e 100644 --- a/drivers/md/dm-cache-metadata.c +++ b/drivers/md/dm-cache-metadata.c @@ -1828,7 +1828,7 @@ int dm_cache_metadata_abort(struct dm_cache_metadata = *cmd) * Replacement block manager (new_bm) is created and old_bm destroyed out= side of * cmd root_lock to avoid ABBA deadlock that would result (due to life-cy= cle of * shrinker associated with the block manager's bufio client vs cmd root_= lock). - * - must take shrinker_mutex without holding cmd->root_lock + * - must take shrinker_rwsem without holding cmd->root_lock */ new_bm =3D dm_block_manager_create(cmd->bdev, DM_CACHE_METADATA_BLOCK_SIZ= E << SECTOR_SHIFT, CACHE_MAX_CONCURRENT_LOCKS); diff --git a/drivers/md/dm-thin-metadata.c b/drivers/md/dm-thin-metadata.c index 9f5cb52c5763..fd464fb024c3 100644 --- a/drivers/md/dm-thin-metadata.c +++ b/drivers/md/dm-thin-metadata.c @@ -1887,7 +1887,7 @@ int dm_pool_abort_metadata(struct dm_pool_metadata *p= md) * Replacement block manager (new_bm) is created and old_bm destroyed out= side of * pmd root_lock to avoid ABBA deadlock that would result (due to life-cy= cle of * shrinker associated with the block manager's bufio client vs pmd root_= lock). - * - must take shrinker_mutex without holding pmd->root_lock + * - must take shrinker_rwsem without holding pmd->root_lock */ new_bm =3D dm_block_manager_create(pmd->bdev, THIN_METADATA_BLOCK_SIZE <<= SECTOR_SHIFT, THIN_MAX_CONCURRENT_LOCKS); diff --git a/fs/super.c b/fs/super.c index 34afe411cf2b..04bc62ab7dfe 100644 --- a/fs/super.c +++ b/fs/super.c @@ -54,7 +54,7 @@ static char *sb_writers_name[SB_FREEZE_LEVELS] =3D { * One thing we have to be careful of with a per-sb shrinker is that we do= n't * drop the last active reference to the superblock from within the shrink= er. * If that happens we could trigger unregistering the shrinker from within= the - * shrinker path and that leads to deadlock on the shrinker_mutex. Hence we + * shrinker path and that leads to deadlock on the shrinker_rwsem. Hence we * take a passive reference to the superblock to avoid this from occurring. */ static unsigned long super_cache_scan(struct shrinker *shrink, diff --git a/mm/shrinker_debug.c b/mm/shrinker_debug.c index fe10436d9911..2be15b8a6d0b 100644 --- a/mm/shrinker_debug.c +++ b/mm/shrinker_debug.c @@ -8,7 +8,7 @@ #include =20 /* defined in vmscan.c */ -extern struct mutex shrinker_mutex; +extern struct rw_semaphore shrinker_rwsem; extern struct list_head shrinker_list; extern struct srcu_struct shrinker_srcu; =20 @@ -168,7 +168,7 @@ int shrinker_debugfs_add(struct shrinker *shrinker) char buf[128]; int id; =20 - lockdep_assert_held(&shrinker_mutex); + lockdep_assert_held(&shrinker_rwsem); =20 /* debugfs isn't initialized yet, add debugfs entries later. */ if (!shrinker_debugfs_root) @@ -211,7 +211,7 @@ int shrinker_debugfs_rename(struct shrinker *shrinker, = const char *fmt, ...) if (!new) return -ENOMEM; =20 - mutex_lock(&shrinker_mutex); + down_write(&shrinker_rwsem); =20 old =3D shrinker->name; shrinker->name =3D new; @@ -229,7 +229,7 @@ int shrinker_debugfs_rename(struct shrinker *shrinker, = const char *fmt, ...) shrinker->debugfs_entry =3D entry; } =20 - mutex_unlock(&shrinker_mutex); + up_write(&shrinker_rwsem); =20 kfree_const(old); =20 @@ -242,7 +242,7 @@ struct dentry *shrinker_debugfs_detach(struct shrinker = *shrinker, { struct dentry *entry =3D shrinker->debugfs_entry; =20 - lockdep_assert_held(&shrinker_mutex); + lockdep_assert_held(&shrinker_rwsem); =20 kfree_const(shrinker->name); shrinker->name =3D NULL; @@ -271,14 +271,14 @@ static int __init shrinker_debugfs_init(void) shrinker_debugfs_root =3D dentry; =20 /* Create debugfs entries for shrinkers registered at boot */ - mutex_lock(&shrinker_mutex); + down_write(&shrinker_rwsem); list_for_each_entry(shrinker, &shrinker_list, list) if (!shrinker->debugfs_entry) { ret =3D shrinker_debugfs_add(shrinker); if (ret) break; } - mutex_unlock(&shrinker_mutex); + up_write(&shrinker_rwsem); =20 return ret; } diff --git a/mm/vmscan.c b/mm/vmscan.c index 6d0cd2840cf0..4730dba253c8 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -35,7 +35,7 @@ #include #include #include -#include +#include #include #include #include @@ -190,7 +190,7 @@ struct scan_control { int vm_swappiness =3D 60; =20 LIST_HEAD(shrinker_list); -DEFINE_MUTEX(shrinker_mutex); +DECLARE_RWSEM(shrinker_rwsem); DEFINE_SRCU(shrinker_srcu); static atomic_t shrinker_srcu_generation =3D ATOMIC_INIT(0); =20 @@ -213,7 +213,7 @@ static struct shrinker_info *shrinker_info_protected(st= ruct mem_cgroup *memcg, { return srcu_dereference_check(memcg->nodeinfo[nid]->shrinker_info, &shrinker_srcu, - lockdep_is_held(&shrinker_mutex)); + lockdep_is_held(&shrinker_rwsem)); } =20 static struct shrinker_info *shrinker_info_srcu(struct mem_cgroup *memcg, @@ -292,7 +292,7 @@ int alloc_shrinker_info(struct mem_cgroup *memcg) int nid, size, ret =3D 0; int map_size, defer_size =3D 0; =20 - mutex_lock(&shrinker_mutex); + down_write(&shrinker_rwsem); map_size =3D shrinker_map_size(shrinker_nr_max); defer_size =3D shrinker_defer_size(shrinker_nr_max); size =3D map_size + defer_size; @@ -308,7 +308,7 @@ int alloc_shrinker_info(struct mem_cgroup *memcg) info->map_nr_max =3D shrinker_nr_max; rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_info, info); } - mutex_unlock(&shrinker_mutex); + up_write(&shrinker_rwsem); =20 return ret; } @@ -324,7 +324,7 @@ static int expand_shrinker_info(int new_id) if (!root_mem_cgroup) goto out; =20 - lockdep_assert_held(&shrinker_mutex); + lockdep_assert_held(&shrinker_rwsem); =20 map_size =3D shrinker_map_size(new_nr_max); defer_size =3D shrinker_defer_size(new_nr_max); @@ -374,7 +374,7 @@ static int prealloc_memcg_shrinker(struct shrinker *shr= inker) if (mem_cgroup_disabled()) return -ENOSYS; =20 - mutex_lock(&shrinker_mutex); + down_write(&shrinker_rwsem); id =3D idr_alloc(&shrinker_idr, shrinker, 0, 0, GFP_KERNEL); if (id < 0) goto unlock; @@ -388,7 +388,7 @@ static int prealloc_memcg_shrinker(struct shrinker *shr= inker) shrinker->id =3D id; ret =3D 0; unlock: - mutex_unlock(&shrinker_mutex); + up_write(&shrinker_rwsem); return ret; } =20 @@ -398,7 +398,7 @@ static void unregister_memcg_shrinker(struct shrinker *= shrinker) =20 BUG_ON(id < 0); =20 - lockdep_assert_held(&shrinker_mutex); + lockdep_assert_held(&shrinker_rwsem); =20 idr_remove(&shrinker_idr, id); } @@ -433,7 +433,7 @@ void reparent_shrinker_deferred(struct mem_cgroup *memc= g) parent =3D root_mem_cgroup; =20 /* Prevent from concurrent shrinker_info expand */ - mutex_lock(&shrinker_mutex); + down_write(&shrinker_rwsem); for_each_node(nid) { child_info =3D shrinker_info_protected(memcg, nid); parent_info =3D shrinker_info_protected(parent, nid); @@ -442,7 +442,7 @@ void reparent_shrinker_deferred(struct mem_cgroup *memc= g) atomic_long_add(nr, &parent_info->nr_deferred[i]); } } - mutex_unlock(&shrinker_mutex); + up_write(&shrinker_rwsem); } =20 static bool cgroup_reclaim(struct scan_control *sc) @@ -743,9 +743,9 @@ void free_prealloced_shrinker(struct shrinker *shrinker) shrinker->name =3D NULL; #endif if (shrinker->flags & SHRINKER_MEMCG_AWARE) { - mutex_lock(&shrinker_mutex); + down_write(&shrinker_rwsem); unregister_memcg_shrinker(shrinker); - mutex_unlock(&shrinker_mutex); + up_write(&shrinker_rwsem); return; } =20 @@ -755,11 +755,11 @@ void free_prealloced_shrinker(struct shrinker *shrink= er) =20 void register_shrinker_prepared(struct shrinker *shrinker) { - mutex_lock(&shrinker_mutex); + down_write(&shrinker_rwsem); list_add_tail_rcu(&shrinker->list, &shrinker_list); shrinker->flags |=3D SHRINKER_REGISTERED; shrinker_debugfs_add(shrinker); - mutex_unlock(&shrinker_mutex); + up_write(&shrinker_rwsem); } =20 static int __register_shrinker(struct shrinker *shrinker) @@ -810,13 +810,13 @@ void unregister_shrinker(struct shrinker *shrinker) if (!(shrinker->flags & SHRINKER_REGISTERED)) return; =20 - mutex_lock(&shrinker_mutex); + down_write(&shrinker_rwsem); list_del_rcu(&shrinker->list); shrinker->flags &=3D ~SHRINKER_REGISTERED; if (shrinker->flags & SHRINKER_MEMCG_AWARE) unregister_memcg_shrinker(shrinker); debugfs_entry =3D shrinker_debugfs_detach(shrinker, &debugfs_id); - mutex_unlock(&shrinker_mutex); + up_write(&shrinker_rwsem); =20 atomic_inc(&shrinker_srcu_generation); synchronize_srcu(&shrinker_srcu); --=20 2.30.2 From nobody Sun Feb 8 01:33:05 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 56EFEC7EE25 for ; Fri, 9 Jun 2023 08:18:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240446AbjFIISY (ORCPT ); Fri, 9 Jun 2023 04:18:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51730 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240271AbjFIIRs (ORCPT ); Fri, 9 Jun 2023 04:17:48 -0400 Received: from out-13.mta0.migadu.com (out-13.mta0.migadu.com [91.218.175.13]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5A6FA30E3 for ; Fri, 9 Jun 2023 01:17:14 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1686298627; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=h+9C3yhXhe+HsjvGtX6Jxz3ecYmnsu4DemmWgRFXfj8=; b=wuAhbqfYpfLg4xQIqqPuVD2Rfwpmfb6tYYJSrXiOZnhHIakZm73KLHKdJ6ypEmmwJn/sYE EYRhoFMKrGVqbr6Pte8aRyp7U9WXdtvlr2I/JW1Y/9T0TEaerWHAcw46nLvdes6f6TOMB6 jHd4kdRFfCPCSHABsGW0lzVmG/1Xkxw= From: Qi Zheng To: akpm@linux-foundation.org Cc: david@fromorbit.com, tkhai@ya.ru, roman.gushchin@linux.dev, vbabka@suse.cz, muchun.song@linux.dev, yujie.liu@intel.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH 2/7] Revert "mm: vmscan: remove shrinker_rwsem from synchronize_shrinkers()" Date: Fri, 9 Jun 2023 08:15:13 +0000 Message-Id: <20230609081518.3039120-3-qi.zheng@linux.dev> In-Reply-To: <20230609081518.3039120-1-qi.zheng@linux.dev> References: <20230609081518.3039120-1-qi.zheng@linux.dev> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Migadu-Flow: FLOW_OUT Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Qi Zheng This reverts commit 1643db98d9b314e0a592d152603094fbf7ab906e. Kernel test robot reports -88.8% regression in stress-ng.ramfs.ops_per_sec test case [1], which is caused by commit f95bdb700bc6 ("mm: vmscan: make global slab shrink lockless"). The root cause is that SRCU has to be careful to not frequently check for SRCU read-side critical section exits. Therefor= e, even if no one is currently in the SRCU read-side critical section, synchronize_srcu() cannot return quickly. That's why unregister_shrinker() has become slower. We will try to use the refcount+RCU method [2] proposed by Dave Chinner to continue to re-implement the lockless slab shrink. So we still need shrinker_rwsem in synchronize_shrinkers() after reverting the shrinker_srcu related changes. [1]. https://lore.kernel.org/lkml/202305230837.db2c233f-yujie.liu@intel.com/ [2]. https://lore.kernel.org/lkml/ZIJhou1d55d4H1s0@dread.disaster.area/ Reported-by: kernel test robot Closes: https://lore.kernel.org/oe-lkp/202305230837.db2c233f-yujie.liu@inte= l.com Signed-off-by: Qi Zheng --- mm/vmscan.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 4730dba253c8..0ba0e1180f3f 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -831,11 +831,15 @@ EXPORT_SYMBOL(unregister_shrinker); /** * synchronize_shrinkers - Wait for all running shrinkers to complete. * - * This is useful to guarantee that all shrinker invocations have seen an - * update, before freeing memory. + * This is equivalent to calling unregister_shrink() and register_shrinker= (), + * but atomically and with less overhead. This is useful to guarantee that= all + * shrinker invocations have seen an update, before freeing memory, simila= r to + * rcu. */ void synchronize_shrinkers(void) { + down_write(&shrinker_rwsem); + up_write(&shrinker_rwsem); atomic_inc(&shrinker_srcu_generation); synchronize_srcu(&shrinker_srcu); } --=20 2.30.2 From nobody Sun Feb 8 01:33:05 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 491AAC7EE37 for ; Fri, 9 Jun 2023 08:18:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240325AbjFIISa (ORCPT ); Fri, 9 Jun 2023 04:18:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51856 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240298AbjFIIRu (ORCPT ); Fri, 9 Jun 2023 04:17:50 -0400 Received: from out-12.mta0.migadu.com (out-12.mta0.migadu.com [91.218.175.12]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D5B273A95 for ; Fri, 9 Jun 2023 01:17:19 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1686298633; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=z1jd6DS57tmtOuv1f+tPHUMa7LOeeEWjdKx6Y4gNtNw=; b=jXFgIXzLdDJojoKAItNEv34otXyCWjyar4fawUm43LXwyCFBcR8kY9Gpt4WbtUzDcH3jkd /70kJs6ZiU9T/QhpzSJOCDGSO1yj+OA/JWTW7Qjwmfr5MuKM15sXtrYty/nAw5lTGHgi5v rbaXvSWMUUp8lu38yF2ylUnJkp1peeY= From: Qi Zheng To: akpm@linux-foundation.org Cc: david@fromorbit.com, tkhai@ya.ru, roman.gushchin@linux.dev, vbabka@suse.cz, muchun.song@linux.dev, yujie.liu@intel.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH 3/7] Revert "mm: vmscan: hold write lock to reparent shrinker nr_deferred" Date: Fri, 9 Jun 2023 08:15:14 +0000 Message-Id: <20230609081518.3039120-4-qi.zheng@linux.dev> In-Reply-To: <20230609081518.3039120-1-qi.zheng@linux.dev> References: <20230609081518.3039120-1-qi.zheng@linux.dev> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Migadu-Flow: FLOW_OUT Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Qi Zheng This reverts commit b3cabea3c9153fd42fe5cb851ac58b51ea2b32b8. Kernel test robot reports -88.8% regression in stress-ng.ramfs.ops_per_sec test case [1], which is caused by commit f95bdb700bc6 ("mm: vmscan: make global slab shrink lockless"). The root cause is that SRCU has to be careful to not frequently check for SRCU read-side critical section exits. Therefor= e, even if no one is currently in the SRCU read-side critical section, synchronize_srcu() cannot return quickly. That's why unregister_shrinker() has become slower. We will try to use the refcount+RCU method [2] proposed by Dave Chinner to continue to re-implement the lockless slab shrink. Because there will be other readers after reverting the shrinker_srcu related changes, so it is better to restore to hold read lock to reparent shrinker nr_deferred. [1]. https://lore.kernel.org/lkml/202305230837.db2c233f-yujie.liu@intel.com/ [2]. https://lore.kernel.org/lkml/ZIJhou1d55d4H1s0@dread.disaster.area/ Reported-by: kernel test robot Closes: https://lore.kernel.org/oe-lkp/202305230837.db2c233f-yujie.liu@inte= l.com Signed-off-by: Qi Zheng --- mm/vmscan.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 0ba0e1180f3f..d1d309fc3212 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -433,7 +433,7 @@ void reparent_shrinker_deferred(struct mem_cgroup *memc= g) parent =3D root_mem_cgroup; =20 /* Prevent from concurrent shrinker_info expand */ - down_write(&shrinker_rwsem); + down_read(&shrinker_rwsem); for_each_node(nid) { child_info =3D shrinker_info_protected(memcg, nid); parent_info =3D shrinker_info_protected(parent, nid); @@ -442,7 +442,7 @@ void reparent_shrinker_deferred(struct mem_cgroup *memc= g) atomic_long_add(nr, &parent_info->nr_deferred[i]); } } - up_write(&shrinker_rwsem); + up_read(&shrinker_rwsem); } =20 static bool cgroup_reclaim(struct scan_control *sc) --=20 2.30.2 From nobody Sun Feb 8 01:33:05 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 37E9BC7EE25 for ; Fri, 9 Jun 2023 08:18:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240514AbjFIISf (ORCPT ); Fri, 9 Jun 2023 04:18:35 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51716 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240324AbjFIIRv (ORCPT ); Fri, 9 Jun 2023 04:17:51 -0400 Received: from out-21.mta0.migadu.com (out-21.mta0.migadu.com [91.218.175.21]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A94463AA4 for ; Fri, 9 Jun 2023 01:17:23 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1686298638; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=fW5UuGuy06FRX8Px/N9YG/QTNF0coGlqdNSj4AuBaaM=; b=tddMOX9hrmBcUy1TaN6PYmjZr8N6htWv1AKIwBbCiMwwdMPB0GeJMYgG8Z8I93aOe/HBvd 5BXFnyv6bfT5hursOmsbSE208IL5wiQKL+bwZlti5+T99GzbUZ4tm7iuHx6iPsFe8QHCGt jtNfOoIRqSFtiW109WCZ6dTaNr7m36M= From: Qi Zheng To: akpm@linux-foundation.org Cc: david@fromorbit.com, tkhai@ya.ru, roman.gushchin@linux.dev, vbabka@suse.cz, muchun.song@linux.dev, yujie.liu@intel.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH 4/7] Revert "mm: shrinkers: make count and scan in shrinker debugfs lockless" Date: Fri, 9 Jun 2023 08:15:15 +0000 Message-Id: <20230609081518.3039120-5-qi.zheng@linux.dev> In-Reply-To: <20230609081518.3039120-1-qi.zheng@linux.dev> References: <20230609081518.3039120-1-qi.zheng@linux.dev> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Migadu-Flow: FLOW_OUT Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Qi Zheng This reverts commit 20cd1892fcc3efc10a7ac327cc3790494bec46b5. Kernel test robot reports -88.8% regression in stress-ng.ramfs.ops_per_sec test case [1], which is caused by commit f95bdb700bc6 ("mm: vmscan: make global slab shrink lockless"). The root cause is that SRCU has to be careful to not frequently check for SRCU read-side critical section exits. Therefor= e, even if no one is currently in the SRCU read-side critical section, synchronize_srcu() cannot return quickly. That's why unregister_shrinker() has become slower. We will try to use the refcount+RCU method [2] proposed by Dave Chinner to continue to re-implement the lockless slab shrink. So revert the shrinker_srcu related changes first. [1]. https://lore.kernel.org/lkml/202305230837.db2c233f-yujie.liu@intel.com/ [2]. https://lore.kernel.org/lkml/ZIJhou1d55d4H1s0@dread.disaster.area/ Reported-by: kernel test robot Closes: https://lore.kernel.org/oe-lkp/202305230837.db2c233f-yujie.liu@inte= l.com Signed-off-by: Qi Zheng --- mm/shrinker_debug.c | 25 +++++++++++++++++-------- 1 file changed, 17 insertions(+), 8 deletions(-) diff --git a/mm/shrinker_debug.c b/mm/shrinker_debug.c index 2be15b8a6d0b..3ab53fad8876 100644 --- a/mm/shrinker_debug.c +++ b/mm/shrinker_debug.c @@ -5,12 +5,10 @@ #include #include #include -#include =20 /* defined in vmscan.c */ extern struct rw_semaphore shrinker_rwsem; extern struct list_head shrinker_list; -extern struct srcu_struct shrinker_srcu; =20 static DEFINE_IDA(shrinker_debugfs_ida); static struct dentry *shrinker_debugfs_root; @@ -51,13 +49,18 @@ static int shrinker_debugfs_count_show(struct seq_file = *m, void *v) struct mem_cgroup *memcg; unsigned long total; bool memcg_aware; - int ret =3D 0, nid, srcu_idx; + int ret, nid; =20 count_per_node =3D kcalloc(nr_node_ids, sizeof(unsigned long), GFP_KERNEL= ); if (!count_per_node) return -ENOMEM; =20 - srcu_idx =3D srcu_read_lock(&shrinker_srcu); + ret =3D down_read_killable(&shrinker_rwsem); + if (ret) { + kfree(count_per_node); + return ret; + } + rcu_read_lock(); =20 memcg_aware =3D shrinker->flags & SHRINKER_MEMCG_AWARE; =20 @@ -88,7 +91,8 @@ static int shrinker_debugfs_count_show(struct seq_file *m= , void *v) } } while ((memcg =3D mem_cgroup_iter(NULL, memcg, NULL)) !=3D NULL); =20 - srcu_read_unlock(&shrinker_srcu, srcu_idx); + rcu_read_unlock(); + up_read(&shrinker_rwsem); =20 kfree(count_per_node); return ret; @@ -111,8 +115,9 @@ static ssize_t shrinker_debugfs_scan_write(struct file = *file, .gfp_mask =3D GFP_KERNEL, }; struct mem_cgroup *memcg =3D NULL; - int nid, srcu_idx; + int nid; char kbuf[72]; + ssize_t ret; =20 read_len =3D size < (sizeof(kbuf) - 1) ? size : (sizeof(kbuf) - 1); if (copy_from_user(kbuf, buf, read_len)) @@ -141,7 +146,11 @@ static ssize_t shrinker_debugfs_scan_write(struct file= *file, return -EINVAL; } =20 - srcu_idx =3D srcu_read_lock(&shrinker_srcu); + ret =3D down_read_killable(&shrinker_rwsem); + if (ret) { + mem_cgroup_put(memcg); + return ret; + } =20 sc.nid =3D nid; sc.memcg =3D memcg; @@ -150,7 +159,7 @@ static ssize_t shrinker_debugfs_scan_write(struct file = *file, =20 shrinker->scan_objects(shrinker, &sc); =20 - srcu_read_unlock(&shrinker_srcu, srcu_idx); + up_read(&shrinker_rwsem); mem_cgroup_put(memcg); =20 return size; --=20 2.30.2 From nobody Sun Feb 8 01:33:05 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 767FAC7EE25 for ; Fri, 9 Jun 2023 08:18:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240357AbjFIISs (ORCPT ); Fri, 9 Jun 2023 04:18:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52232 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240375AbjFIISH (ORCPT ); Fri, 9 Jun 2023 04:18:07 -0400 Received: from out-59.mta0.migadu.com (out-59.mta0.migadu.com [IPv6:2001:41d0:1004:224b::3b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 565C81988 for ; Fri, 9 Jun 2023 01:17:27 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1686298645; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ptGmpfMk/kHVim/COA1N68hD8MGHSDZD7U0gMdC3pxY=; b=dUWb27UUXt+8QDbQIc9tustr3Ex0BabLxTJee5D+fXjeHZBQ4qcFjuXfmS9deSWg9pnPCR 9dTR+kP4JvZRsxWTBrikjSS0f2Fm0CUyHJiy5VG+bhRcgClpEyEjDlHhnKlTs1O4PWyCUu 6wpdq+dene7opH5wdk+agSCOHXRmYcg= From: Qi Zheng To: akpm@linux-foundation.org Cc: david@fromorbit.com, tkhai@ya.ru, roman.gushchin@linux.dev, vbabka@suse.cz, muchun.song@linux.dev, yujie.liu@intel.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH 5/7] Revert "mm: vmscan: add shrinker_srcu_generation" Date: Fri, 9 Jun 2023 08:15:16 +0000 Message-Id: <20230609081518.3039120-6-qi.zheng@linux.dev> In-Reply-To: <20230609081518.3039120-1-qi.zheng@linux.dev> References: <20230609081518.3039120-1-qi.zheng@linux.dev> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Migadu-Flow: FLOW_OUT Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Qi Zheng This reverts commit 475733dda5aedba9e086379aafe6b5ffd53e8f5e. Kernel test robot reports -88.8% regression in stress-ng.ramfs.ops_per_sec test case [1], which is caused by commit f95bdb700bc6 ("mm: vmscan: make global slab shrink lockless"). The root cause is that SRCU has to be careful to not frequently check for SRCU read-side critical section exits. Therefor= e, even if no one is currently in the SRCU read-side critical section, synchronize_srcu() cannot return quickly. That's why unregister_shrinker() has become slower. We will try to use the refcount+RCU method [2] proposed by Dave Chinner to continue to re-implement the lockless slab shrink. So revert the shrinker_srcu related changes first. [1]. https://lore.kernel.org/lkml/202305230837.db2c233f-yujie.liu@intel.com/ [2]. https://lore.kernel.org/lkml/ZIJhou1d55d4H1s0@dread.disaster.area/ Reported-by: kernel test robot Closes: https://lore.kernel.org/oe-lkp/202305230837.db2c233f-yujie.liu@inte= l.com Signed-off-by: Qi Zheng --- mm/vmscan.c | 24 ++++-------------------- 1 file changed, 4 insertions(+), 20 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index d1d309fc3212..50775b73d0c7 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -192,7 +192,6 @@ int vm_swappiness =3D 60; LIST_HEAD(shrinker_list); DECLARE_RWSEM(shrinker_rwsem); DEFINE_SRCU(shrinker_srcu); -static atomic_t shrinker_srcu_generation =3D ATOMIC_INIT(0); =20 #ifdef CONFIG_MEMCG static int shrinker_nr_max; @@ -818,7 +817,6 @@ void unregister_shrinker(struct shrinker *shrinker) debugfs_entry =3D shrinker_debugfs_detach(shrinker, &debugfs_id); up_write(&shrinker_rwsem); =20 - atomic_inc(&shrinker_srcu_generation); synchronize_srcu(&shrinker_srcu); =20 shrinker_debugfs_remove(debugfs_entry, debugfs_id); @@ -840,7 +838,6 @@ void synchronize_shrinkers(void) { down_write(&shrinker_rwsem); up_write(&shrinker_rwsem); - atomic_inc(&shrinker_srcu_generation); synchronize_srcu(&shrinker_srcu); } EXPORT_SYMBOL(synchronize_shrinkers); @@ -950,20 +947,18 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask= , int nid, { struct shrinker_info *info; unsigned long ret, freed =3D 0; - int srcu_idx, generation; - int i =3D 0; + int srcu_idx; + int i; =20 if (!mem_cgroup_online(memcg)) return 0; =20 -again: srcu_idx =3D srcu_read_lock(&shrinker_srcu); info =3D shrinker_info_srcu(memcg, nid); if (unlikely(!info)) goto unlock; =20 - generation =3D atomic_read(&shrinker_srcu_generation); - for_each_set_bit_from(i, info->map, info->map_nr_max) { + for_each_set_bit(i, info->map, info->map_nr_max) { struct shrink_control sc =3D { .gfp_mask =3D gfp_mask, .nid =3D nid, @@ -1009,11 +1004,6 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mas= k, int nid, set_shrinker_bit(memcg, nid, i); } freed +=3D ret; - if (atomic_read(&shrinker_srcu_generation) !=3D generation) { - srcu_read_unlock(&shrinker_srcu, srcu_idx); - i++; - goto again; - } } unlock: srcu_read_unlock(&shrinker_srcu, srcu_idx); @@ -1053,7 +1043,7 @@ static unsigned long shrink_slab(gfp_t gfp_mask, int = nid, { unsigned long ret, freed =3D 0; struct shrinker *shrinker; - int srcu_idx, generation; + int srcu_idx; =20 /* * The root memcg might be allocated even though memcg is disabled @@ -1067,7 +1057,6 @@ static unsigned long shrink_slab(gfp_t gfp_mask, int = nid, =20 srcu_idx =3D srcu_read_lock(&shrinker_srcu); =20 - generation =3D atomic_read(&shrinker_srcu_generation); list_for_each_entry_srcu(shrinker, &shrinker_list, list, srcu_read_lock_held(&shrinker_srcu)) { struct shrink_control sc =3D { @@ -1080,11 +1069,6 @@ static unsigned long shrink_slab(gfp_t gfp_mask, int= nid, if (ret =3D=3D SHRINK_EMPTY) ret =3D 0; freed +=3D ret; - - if (atomic_read(&shrinker_srcu_generation) !=3D generation) { - freed =3D freed ? : 1; - break; - } } =20 srcu_read_unlock(&shrinker_srcu, srcu_idx); --=20 2.30.2 From nobody Sun Feb 8 01:33:05 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EF676C7EE37 for ; Fri, 9 Jun 2023 08:19:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239128AbjFIITK (ORCPT ); Fri, 9 Jun 2023 04:19:10 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52010 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240417AbjFIISQ (ORCPT ); Fri, 9 Jun 2023 04:18:16 -0400 Received: from out-6.mta0.migadu.com (out-6.mta0.migadu.com [91.218.175.6]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C211C2715; Fri, 9 Jun 2023 01:17:33 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1686298649; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=B/E+4G7xduNcnh08om90Yauw78sQGFjRZ4mRQx21A2Q=; b=Hp4UWFqQJil8Kf9Kbk7PpaCcvuRjBeSALwQccnwRZWKqDBGAxb1HaJdVYPnL+2SqJ79gWL Dvz4OkZmXEAisnQB6TAIkcV1qOdjUk0ksJQqHC9jb/4iwoI78cHfc6hLpTA+6ksOQfKmVx x7esjKuYGnPxyQBoj7U7f+XOMRKIYW8= From: Qi Zheng To: akpm@linux-foundation.org Cc: david@fromorbit.com, tkhai@ya.ru, roman.gushchin@linux.dev, vbabka@suse.cz, muchun.song@linux.dev, yujie.liu@intel.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH 6/7] Revert "mm: vmscan: make memcg slab shrink lockless" Date: Fri, 9 Jun 2023 08:15:17 +0000 Message-Id: <20230609081518.3039120-7-qi.zheng@linux.dev> In-Reply-To: <20230609081518.3039120-1-qi.zheng@linux.dev> References: <20230609081518.3039120-1-qi.zheng@linux.dev> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Migadu-Flow: FLOW_OUT Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Qi Zheng This reverts commit caa05325c9126c77ebf114edce51536a0d0a9a08. Kernel test robot reports -88.8% regression in stress-ng.ramfs.ops_per_sec test case [1], which is caused by commit f95bdb700bc6 ("mm: vmscan: make global slab shrink lockless"). The root cause is that SRCU has to be careful to not frequently check for SRCU read-side critical section exits. Therefor= e, even if no one is currently in the SRCU read-side critical section, synchronize_srcu() cannot return quickly. That's why unregister_shrinker() has become slower. After discussion, we will try to use the refcount+RCU method [2] proposed by Dave Chinner to continue to re-implement the lockless slab shrink. So revert the shrinker_srcu related changes first. [1]. https://lore.kernel.org/lkml/202305230837.db2c233f-yujie.liu@intel.com/ [2]. https://lore.kernel.org/lkml/ZIJhou1d55d4H1s0@dread.disaster.area/ Reported-by: kernel test robot Closes: https://lore.kernel.org/oe-lkp/202305230837.db2c233f-yujie.liu@inte= l.com Signed-off-by: Qi Zheng --- mm/vmscan.c | 45 +++++++++++++++++++-------------------------- 1 file changed, 19 insertions(+), 26 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 50775b73d0c7..a008d7f2d0fc 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -210,21 +210,8 @@ static inline int shrinker_defer_size(int nr_items) static struct shrinker_info *shrinker_info_protected(struct mem_cgroup *me= mcg, int nid) { - return srcu_dereference_check(memcg->nodeinfo[nid]->shrinker_info, - &shrinker_srcu, - lockdep_is_held(&shrinker_rwsem)); -} - -static struct shrinker_info *shrinker_info_srcu(struct mem_cgroup *memcg, - int nid) -{ - return srcu_dereference(memcg->nodeinfo[nid]->shrinker_info, - &shrinker_srcu); -} - -static void free_shrinker_info_rcu(struct rcu_head *head) -{ - kvfree(container_of(head, struct shrinker_info, rcu)); + return rcu_dereference_protected(memcg->nodeinfo[nid]->shrinker_info, + lockdep_is_held(&shrinker_rwsem)); } =20 static int expand_one_shrinker_info(struct mem_cgroup *memcg, @@ -265,7 +252,7 @@ static int expand_one_shrinker_info(struct mem_cgroup *= memcg, defer_size - old_defer_size); =20 rcu_assign_pointer(pn->shrinker_info, new); - call_srcu(&shrinker_srcu, &old->rcu, free_shrinker_info_rcu); + kvfree_rcu(old, rcu); } =20 return 0; @@ -351,16 +338,15 @@ void set_shrinker_bit(struct mem_cgroup *memcg, int n= id, int shrinker_id) { if (shrinker_id >=3D 0 && memcg && !mem_cgroup_is_root(memcg)) { struct shrinker_info *info; - int srcu_idx; =20 - srcu_idx =3D srcu_read_lock(&shrinker_srcu); - info =3D shrinker_info_srcu(memcg, nid); + rcu_read_lock(); + info =3D rcu_dereference(memcg->nodeinfo[nid]->shrinker_info); if (!WARN_ON_ONCE(shrinker_id >=3D info->map_nr_max)) { /* Pairs with smp mb in shrink_slab() */ smp_mb__before_atomic(); set_bit(shrinker_id, info->map); } - srcu_read_unlock(&shrinker_srcu, srcu_idx); + rcu_read_unlock(); } } =20 @@ -374,6 +360,7 @@ static int prealloc_memcg_shrinker(struct shrinker *shr= inker) return -ENOSYS; =20 down_write(&shrinker_rwsem); + /* This may call shrinker, so it must use down_read_trylock() */ id =3D idr_alloc(&shrinker_idr, shrinker, 0, 0, GFP_KERNEL); if (id < 0) goto unlock; @@ -407,7 +394,7 @@ static long xchg_nr_deferred_memcg(int nid, struct shri= nker *shrinker, { struct shrinker_info *info; =20 - info =3D shrinker_info_srcu(memcg, nid); + info =3D shrinker_info_protected(memcg, nid); return atomic_long_xchg(&info->nr_deferred[shrinker->id], 0); } =20 @@ -416,7 +403,7 @@ static long add_nr_deferred_memcg(long nr, int nid, str= uct shrinker *shrinker, { struct shrinker_info *info; =20 - info =3D shrinker_info_srcu(memcg, nid); + info =3D shrinker_info_protected(memcg, nid); return atomic_long_add_return(nr, &info->nr_deferred[shrinker->id]); } =20 @@ -947,14 +934,15 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask= , int nid, { struct shrinker_info *info; unsigned long ret, freed =3D 0; - int srcu_idx; int i; =20 if (!mem_cgroup_online(memcg)) return 0; =20 - srcu_idx =3D srcu_read_lock(&shrinker_srcu); - info =3D shrinker_info_srcu(memcg, nid); + if (!down_read_trylock(&shrinker_rwsem)) + return 0; + + info =3D shrinker_info_protected(memcg, nid); if (unlikely(!info)) goto unlock; =20 @@ -1004,9 +992,14 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask= , int nid, set_shrinker_bit(memcg, nid, i); } freed +=3D ret; + + if (rwsem_is_contended(&shrinker_rwsem)) { + freed =3D freed ? : 1; + break; + } } unlock: - srcu_read_unlock(&shrinker_srcu, srcu_idx); + up_read(&shrinker_rwsem); return freed; } #else /* CONFIG_MEMCG */ --=20 2.30.2 From nobody Sun Feb 8 01:33:05 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A2D24C7EE2E for ; Fri, 9 Jun 2023 08:19:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240479AbjFIITO (ORCPT ); Fri, 9 Jun 2023 04:19:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52476 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240271AbjFIISY (ORCPT ); Fri, 9 Jun 2023 04:18:24 -0400 Received: from out-34.mta0.migadu.com (out-34.mta0.migadu.com [IPv6:2001:41d0:1004:224b::22]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A0AB12697 for ; Fri, 9 Jun 2023 01:17:37 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1686298655; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=EMn+msnIJXvC+06grZ542lhB8pLG9zNhZSiyiHhRbaA=; b=QnAzENdGYWeYO57Spgjhk1adZxJY7gan2t4wYWPeZIaItdmJZDM8xOICqPyq/QELcJD9OY hG30tKtjBO4CWu/+8SDMmqFoQybaFEZQceY8dbZCLKigPc6SX9awgQJwPEusZAlihLEw5W Hw8FqnopH5nBR0yG+Weoe/3z4ob2VOY= From: Qi Zheng To: akpm@linux-foundation.org Cc: david@fromorbit.com, tkhai@ya.ru, roman.gushchin@linux.dev, vbabka@suse.cz, muchun.song@linux.dev, yujie.liu@intel.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH 7/7] Revert "mm: vmscan: make global slab shrink lockless" Date: Fri, 9 Jun 2023 08:15:18 +0000 Message-Id: <20230609081518.3039120-8-qi.zheng@linux.dev> In-Reply-To: <20230609081518.3039120-1-qi.zheng@linux.dev> References: <20230609081518.3039120-1-qi.zheng@linux.dev> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Migadu-Flow: FLOW_OUT Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Qi Zheng This reverts commit f95bdb700bc6bb74e1199b1f5f90c613e152cfa7. Kernel test robot reports -88.8% regression in stress-ng.ramfs.ops_per_sec test case [1], which is caused by commit f95bdb700bc6 ("mm: vmscan: make global slab shrink lockless"). The root cause is that SRCU has to be careful to not frequently check for SRCU read-side critical section exits. Therefor= e, even if no one is currently in the SRCU read-side critical section, synchronize_srcu() cannot return quickly. That's why unregister_shrinker() has become slower. After discussion, we will try to use the refcount+RCU method [2] proposed by Dave Chinner to continue to re-implement the lockless slab shrink. So revert the shrinker_srcu related changes first. [1]. https://lore.kernel.org/lkml/202305230837.db2c233f-yujie.liu@intel.com/ [2]. https://lore.kernel.org/lkml/ZIJhou1d55d4H1s0@dread.disaster.area/ Reported-by: kernel test robot Closes: https://lore.kernel.org/oe-lkp/202305230837.db2c233f-yujie.liu@inte= l.com Signed-off-by: Qi Zheng --- mm/vmscan.c | 28 ++++++++++++++++------------ 1 file changed, 16 insertions(+), 12 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index a008d7f2d0fc..5bf98d0a22c9 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -57,7 +57,6 @@ #include #include #include -#include =20 #include #include @@ -191,7 +190,6 @@ int vm_swappiness =3D 60; =20 LIST_HEAD(shrinker_list); DECLARE_RWSEM(shrinker_rwsem); -DEFINE_SRCU(shrinker_srcu); =20 #ifdef CONFIG_MEMCG static int shrinker_nr_max; @@ -742,7 +740,7 @@ void free_prealloced_shrinker(struct shrinker *shrinker) void register_shrinker_prepared(struct shrinker *shrinker) { down_write(&shrinker_rwsem); - list_add_tail_rcu(&shrinker->list, &shrinker_list); + list_add_tail(&shrinker->list, &shrinker_list); shrinker->flags |=3D SHRINKER_REGISTERED; shrinker_debugfs_add(shrinker); up_write(&shrinker_rwsem); @@ -797,15 +795,13 @@ void unregister_shrinker(struct shrinker *shrinker) return; =20 down_write(&shrinker_rwsem); - list_del_rcu(&shrinker->list); + list_del(&shrinker->list); shrinker->flags &=3D ~SHRINKER_REGISTERED; if (shrinker->flags & SHRINKER_MEMCG_AWARE) unregister_memcg_shrinker(shrinker); debugfs_entry =3D shrinker_debugfs_detach(shrinker, &debugfs_id); up_write(&shrinker_rwsem); =20 - synchronize_srcu(&shrinker_srcu); - shrinker_debugfs_remove(debugfs_entry, debugfs_id); =20 kfree(shrinker->nr_deferred); @@ -825,7 +821,6 @@ void synchronize_shrinkers(void) { down_write(&shrinker_rwsem); up_write(&shrinker_rwsem); - synchronize_srcu(&shrinker_srcu); } EXPORT_SYMBOL(synchronize_shrinkers); =20 @@ -1036,7 +1031,6 @@ static unsigned long shrink_slab(gfp_t gfp_mask, int = nid, { unsigned long ret, freed =3D 0; struct shrinker *shrinker; - int srcu_idx; =20 /* * The root memcg might be allocated even though memcg is disabled @@ -1048,10 +1042,10 @@ static unsigned long shrink_slab(gfp_t gfp_mask, in= t nid, if (!mem_cgroup_disabled() && !mem_cgroup_is_root(memcg)) return shrink_slab_memcg(gfp_mask, nid, memcg, priority); =20 - srcu_idx =3D srcu_read_lock(&shrinker_srcu); + if (!down_read_trylock(&shrinker_rwsem)) + goto out; =20 - list_for_each_entry_srcu(shrinker, &shrinker_list, list, - srcu_read_lock_held(&shrinker_srcu)) { + list_for_each_entry(shrinker, &shrinker_list, list) { struct shrink_control sc =3D { .gfp_mask =3D gfp_mask, .nid =3D nid, @@ -1062,9 +1056,19 @@ static unsigned long shrink_slab(gfp_t gfp_mask, int= nid, if (ret =3D=3D SHRINK_EMPTY) ret =3D 0; freed +=3D ret; + /* + * Bail out if someone want to register a new shrinker to + * prevent the registration from being stalled for long periods + * by parallel ongoing shrinking. + */ + if (rwsem_is_contended(&shrinker_rwsem)) { + freed =3D freed ? : 1; + break; + } } =20 - srcu_read_unlock(&shrinker_srcu, srcu_idx); + up_read(&shrinker_rwsem); +out: cond_resched(); return freed; } --=20 2.30.2