From nobody Sat Feb 7 17:48:32 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 75BE2C7EE23 for ; Mon, 5 Jun 2023 19:03:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232845AbjFETDR (ORCPT ); Mon, 5 Jun 2023 15:03:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59414 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230466AbjFETDP (ORCPT ); Mon, 5 Jun 2023 15:03:15 -0400 Received: from forward102b.mail.yandex.net (forward102b.mail.yandex.net [178.154.239.149]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6C2A110B for ; Mon, 5 Jun 2023 12:03:13 -0700 (PDT) Received: from mail-nwsmtp-smtp-production-main-25.sas.yp-c.yandex.net (mail-nwsmtp-smtp-production-main-25.sas.yp-c.yandex.net [IPv6:2a02:6b8:c08:2e14:0:640:2cd1:0]) by forward102b.mail.yandex.net (Yandex) with ESMTP id 71C1B60042; Mon, 5 Jun 2023 22:03:08 +0300 (MSK) Received: by mail-nwsmtp-smtp-production-main-25.sas.yp-c.yandex.net (smtp/Yandex) with ESMTPSA id 23YUOG7DgKo0-B7EanhI6; Mon, 05 Jun 2023 22:03:07 +0300 X-Yandex-Fwd: 1 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ya.ru; s=mail; t=1685991787; bh=W1Bi1QeA3RV6fDYh5nFGYXyCEkJP+gzRRvK3oGgjt0Y=; h=In-Reply-To:Message-Id:References:Date:Subject:To:From; b=fv3xvnujLxOzwiwx7oy8Fmj7DhRGkL1HQjX6HTZuI0ksIExohWUw/ppxsFP26wrQG ABBRGtJf4o4VNLVlTLGPG1t6h7tqg5icrp4ezWXm+YNnpCz010eUgXfzsA+M8cGqAe PTxMdU9REozFF2iPapv6z6sRp0YkcM1+kv/mCFMA= Authentication-Results: mail-nwsmtp-smtp-production-main-25.sas.yp-c.yandex.net; dkim=pass header.i=@ya.ru From: Kirill Tkhai To: akpm@linux-foundation.org, tkhai@ya.ru, roman.gushchin@linux.dev, vbabka@suse.cz, viro@zeniv.linux.org.uk, brauner@kernel.org, djwong@kernel.org, hughd@google.com, paulmck@kernel.org, muchun.song@linux.dev, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-xfs@vger.kernel.org, linux-kernel@vger.kernel.org, zhengqi.arch@bytedance.com, david@fromorbit.com Subject: [PATCH v2 1/3] mm: vmscan: move shrinker_debugfs_remove() before synchronize_srcu() Date: Mon, 5 Jun 2023 22:03:02 +0300 Message-Id: <168599178203.70911.18350742045278218790.stgit@pro.pro> X-Mailer: git-send-email 2.40.1 In-Reply-To: <168599103578.70911.9402374667983518835.stgit@pro.pro> References: <168599103578.70911.9402374667983518835.stgit@pro.pro> User-Agent: StGit/0.19 MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Qi Zheng The debugfs_remove_recursive() will wait for debugfs_file_put() to return, so there is no need to put it after synchronize_srcu() to wait for the rcu read-side critical section to exit. Just move it before synchronize_srcu(), which is also convenient to put the heavy synchronize_srcu() in the delayed work later. Signed-off-by: Qi Zheng Acked-by: Roman Gushchin --- mm/vmscan.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index eeca83e28c9b..a773e97e152e 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -818,11 +818,11 @@ void unregister_shrinker(struct shrinker *shrinker) debugfs_entry =3D shrinker_debugfs_detach(shrinker, &debugfs_id); mutex_unlock(&shrinker_mutex); =20 + shrinker_debugfs_remove(debugfs_entry, debugfs_id); + atomic_inc(&shrinker_srcu_generation); synchronize_srcu(&shrinker_srcu); =20 - shrinker_debugfs_remove(debugfs_entry, debugfs_id); - kfree(shrinker->nr_deferred); shrinker->nr_deferred =3D NULL; } From nobody Sat Feb 7 17:48:32 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id AF72CC7EE23 for ; Mon, 5 Jun 2023 19:03:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233540AbjFETDn (ORCPT ); Mon, 5 Jun 2023 15:03:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59452 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233417AbjFETD1 (ORCPT ); Mon, 5 Jun 2023 15:03:27 -0400 Received: from forward102b.mail.yandex.net (forward102b.mail.yandex.net [178.154.239.149]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CF56E10B for ; Mon, 5 Jun 2023 12:03:23 -0700 (PDT) Received: from mail-nwsmtp-smtp-production-main-73.iva.yp-c.yandex.net (mail-nwsmtp-smtp-production-main-73.iva.yp-c.yandex.net [IPv6:2a02:6b8:c0c:160b:0:640:acd0:0]) by forward102b.mail.yandex.net (Yandex) with ESMTP id 37B946003D; Mon, 5 Jun 2023 22:03:20 +0300 (MSK) Received: by mail-nwsmtp-smtp-production-main-73.iva.yp-c.yandex.net (smtp/Yandex) with ESMTPSA id D3YHdS6DcOs0-TqtIfVrH; Mon, 05 Jun 2023 22:03:19 +0300 X-Yandex-Fwd: 1 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ya.ru; s=mail; t=1685991799; bh=+IzLiAr4hGoU+Pviuk7ygHtq+6fMQv9f3smgHpwNKuw=; h=In-Reply-To:Message-Id:References:Date:Subject:To:From; b=OGrv/TbDGlW78nIq/v3X13MdFHBKPT4rbxik+/dLXWSsNg3JAt1QziJRND95B7QWd t2F1dPFG+s24W3tEx7lNJP1KDrTIoNVWl9dB7AZmq2ZA4ZCtBlOaz2CDlUmD7vcZYK NpKTRl0e23MTd0mT9DvzVIrgrnTx7Xf/nVp+2aAg= Authentication-Results: mail-nwsmtp-smtp-production-main-73.iva.yp-c.yandex.net; dkim=pass header.i=@ya.ru From: Kirill Tkhai To: akpm@linux-foundation.org, tkhai@ya.ru, roman.gushchin@linux.dev, vbabka@suse.cz, viro@zeniv.linux.org.uk, brauner@kernel.org, djwong@kernel.org, hughd@google.com, paulmck@kernel.org, muchun.song@linux.dev, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-xfs@vger.kernel.org, linux-kernel@vger.kernel.org, zhengqi.arch@bytedance.com, david@fromorbit.com Subject: [PATCH v2 2/3] mm: Split unregister_shrinker() in fast and slow part Date: Mon, 5 Jun 2023 22:03:13 +0300 Message-Id: <168599179360.70911.4102140966715923751.stgit@pro.pro> X-Mailer: git-send-email 2.40.1 In-Reply-To: <168599103578.70911.9402374667983518835.stgit@pro.pro> References: <168599103578.70911.9402374667983518835.stgit@pro.pro> User-Agent: StGit/0.19 MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org This splits unregister_shrinker() in two parts, and this allows to make the unregistration faster by moving the slow part in delayed asynchronous work. Note, that the guarantees remain the same: no do_shrink_slab() calls are possible after the first part. This will be used in next patch. Signed-off-by: Kirill Tkhai --- include/linux/shrinker.h | 4 ++++ mm/vmscan.c | 35 +++++++++++++++++++++++++++++------ 2 files changed, 33 insertions(+), 6 deletions(-) diff --git a/include/linux/shrinker.h b/include/linux/shrinker.h index 224293b2dd06..1cc572fa6070 100644 --- a/include/linux/shrinker.h +++ b/include/linux/shrinker.h @@ -4,6 +4,7 @@ =20 #include #include +#include =20 /* * This struct is used to pass information from page reclaim to the shrink= ers. @@ -83,6 +84,7 @@ struct shrinker { #endif /* objs pending delete, per node */ atomic_long_t *nr_deferred; + struct rw_semaphore rwsem; }; #define DEFAULT_SEEKS 2 /* A good number if you don't know better. */ =20 @@ -102,6 +104,8 @@ extern void register_shrinker_prepared(struct shrinker = *shrinker); extern int __printf(2, 3) register_shrinker(struct shrinker *shrinker, const char *fmt, ...); extern void unregister_shrinker(struct shrinker *shrinker); +extern void unregister_shrinker_delayed_initiate(struct shrinker *shrinker= ); +extern void unregister_shrinker_delayed_finalize(struct shrinker *shrinker= ); extern void free_prealloced_shrinker(struct shrinker *shrinker); extern void synchronize_shrinkers(void); =20 diff --git a/mm/vmscan.c b/mm/vmscan.c index a773e97e152e..f24fd58dcc2a 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -706,6 +706,7 @@ static int __prealloc_shrinker(struct shrinker *shrinke= r) if (!shrinker->nr_deferred) return -ENOMEM; =20 + init_rwsem(&shrinker->rwsem); return 0; } =20 @@ -757,7 +758,9 @@ void register_shrinker_prepared(struct shrinker *shrink= er) { mutex_lock(&shrinker_mutex); list_add_tail_rcu(&shrinker->list, &shrinker_list); + down_write(&shrinker->rwsem); shrinker->flags |=3D SHRINKER_REGISTERED; + up_write(&shrinker->rwsem); shrinker_debugfs_add(shrinker); mutex_unlock(&shrinker_mutex); } @@ -802,7 +805,7 @@ EXPORT_SYMBOL(register_shrinker); /* * Remove one */ -void unregister_shrinker(struct shrinker *shrinker) +void unregister_shrinker_delayed_initiate(struct shrinker *shrinker) { struct dentry *debugfs_entry; int debugfs_id; @@ -812,20 +815,33 @@ void unregister_shrinker(struct shrinker *shrinker) =20 mutex_lock(&shrinker_mutex); list_del_rcu(&shrinker->list); + down_write(&shrinker->rwsem); shrinker->flags &=3D ~SHRINKER_REGISTERED; + up_write(&shrinker->rwsem); if (shrinker->flags & SHRINKER_MEMCG_AWARE) unregister_memcg_shrinker(shrinker); debugfs_entry =3D shrinker_debugfs_detach(shrinker, &debugfs_id); mutex_unlock(&shrinker_mutex); =20 shrinker_debugfs_remove(debugfs_entry, debugfs_id); +} +EXPORT_SYMBOL(unregister_shrinker_delayed_initiate); =20 +void unregister_shrinker_delayed_finalize(struct shrinker *shrinker) +{ atomic_inc(&shrinker_srcu_generation); synchronize_srcu(&shrinker_srcu); =20 kfree(shrinker->nr_deferred); shrinker->nr_deferred =3D NULL; } +EXPORT_SYMBOL(unregister_shrinker_delayed_finalize); + +void unregister_shrinker(struct shrinker *shrinker) +{ + unregister_shrinker_delayed_initiate(shrinker); + unregister_shrinker_delayed_finalize(shrinker); +} EXPORT_SYMBOL(unregister_shrinker); =20 /** @@ -856,9 +872,15 @@ static unsigned long do_shrink_slab(struct shrink_cont= rol *shrinkctl, : SHRINK_BATCH; long scanned =3D 0, next_deferred; =20 + if (!down_read_trylock(&shrinker->rwsem)) + return 0; + if (!(shrinker->flags & SHRINKER_REGISTERED)) + goto unlock; freeable =3D shrinker->count_objects(shrinker, shrinkctl); - if (freeable =3D=3D 0 || freeable =3D=3D SHRINK_EMPTY) - return freeable; + if (freeable =3D=3D 0 || freeable =3D=3D SHRINK_EMPTY) { + freed =3D freeable; + goto unlock; + } =20 /* * copy the current shrinker scan count into a local variable @@ -937,6 +959,8 @@ static unsigned long do_shrink_slab(struct shrink_contr= ol *shrinkctl, new_nr =3D add_nr_deferred(next_deferred, shrinker, shrinkctl); =20 trace_mm_shrink_slab_end(shrinker, shrinkctl->nid, freed, nr, new_nr, tot= al_scan); +unlock: + up_read(&shrinker->rwsem); return freed; } =20 @@ -968,9 +992,8 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, = int nid, struct shrinker *shrinker; =20 shrinker =3D idr_find(&shrinker_idr, i); - if (unlikely(!shrinker || !(shrinker->flags & SHRINKER_REGISTERED))) { - if (!shrinker) - clear_bit(i, info->map); + if (unlikely(!shrinker)) { + clear_bit(i, info->map); continue; } From nobody Sat Feb 7 17:48:32 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 582C2C77B73 for ; Mon, 5 Jun 2023 19:03:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234087AbjFETDv (ORCPT ); Mon, 5 Jun 2023 15:03:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59766 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234301AbjFETDh (ORCPT ); Mon, 5 Jun 2023 15:03:37 -0400 Received: from forward102b.mail.yandex.net (forward102b.mail.yandex.net [178.154.239.149]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1435110B for ; Mon, 5 Jun 2023 12:03:34 -0700 (PDT) Received: from mail-nwsmtp-smtp-production-main-33.iva.yp-c.yandex.net (mail-nwsmtp-smtp-production-main-33.iva.yp-c.yandex.net [IPv6:2a02:6b8:c0c:7c8d:0:640:daf4:0]) by forward102b.mail.yandex.net (Yandex) with ESMTP id 7088560037; Mon, 5 Jun 2023 22:03:31 +0300 (MSK) Received: by mail-nwsmtp-smtp-production-main-33.iva.yp-c.yandex.net (smtp/Yandex) with ESMTPSA id P3Y5PY6DbSw0-D03FsbaO; Mon, 05 Jun 2023 22:03:30 +0300 X-Yandex-Fwd: 1 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ya.ru; s=mail; t=1685991810; bh=MEigQzDY74Nvn1JaDAKnEZHd5i4CDwKJ3Ad6AHXfzeY=; h=In-Reply-To:Message-Id:References:Date:Subject:To:From; b=nku73zf1DQChaO49JXDE6MZ6i2mQGvLbi+bBUqJu+M4OnrmerRoH+8GLIIlD2BtYk Lf0Cl3rp5rYLS6m/8g/rhZOY/fpBJI5utQRJhIvdaUPH+qQyix5BtbHC8GP3T/LS+r K4Gj7k0LH642cC+bqZ6rQWO6ULwPjnkJd/QHfnAw= Authentication-Results: mail-nwsmtp-smtp-production-main-33.iva.yp-c.yandex.net; dkim=pass header.i=@ya.ru From: Kirill Tkhai To: akpm@linux-foundation.org, tkhai@ya.ru, roman.gushchin@linux.dev, vbabka@suse.cz, viro@zeniv.linux.org.uk, brauner@kernel.org, djwong@kernel.org, hughd@google.com, paulmck@kernel.org, muchun.song@linux.dev, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-xfs@vger.kernel.org, linux-kernel@vger.kernel.org, zhengqi.arch@bytedance.com, david@fromorbit.com Subject: [PATCH v2 3/3] fs: Use delayed shrinker unregistration Date: Mon, 5 Jun 2023 22:03:25 +0300 Message-Id: <168599180526.70911.14606767590861123431.stgit@pro.pro> X-Mailer: git-send-email 2.40.1 In-Reply-To: <168599103578.70911.9402374667983518835.stgit@pro.pro> References: <168599103578.70911.9402374667983518835.stgit@pro.pro> User-Agent: StGit/0.19 MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Kernel test robot reports -88.8% regression in stress-ng.ramfs.ops_per_sec test case caused by commit: f95bdb700bc6 ("mm: vmscan: make global slab shrink lockless"). Qi Zheng investigated that the reason is in long SRCU's synchronize_srcu() occuring in unregister_shrinker(). This patch fixes the problem by using new unregistration interfaces, which split unregister_shrinker() in two parts. First part actually only notifies shrinker subsystem about the fact of unregistration and it prevents future shrinker methods calls. The second part completes the unregistration and it insures, that struct shrinker is not used during shrinker chain iteration anymore, so shrinker memory may be freed. Since the long second part is called from delayed work asynchronously, it hides synchronize_srcu() delay from a user. Signed-off-by: Kirill Tkhai --- fs/super.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/fs/super.c b/fs/super.c index 8d8d68799b34..f3e4f205ec79 100644 --- a/fs/super.c +++ b/fs/super.c @@ -159,6 +159,7 @@ static void destroy_super_work(struct work_struct *work) destroy_work); int i; =20 + unregister_shrinker_delayed_finalize(&s->s_shrink); for (i =3D 0; i < SB_FREEZE_LEVELS; i++) percpu_free_rwsem(&s->s_writers.rw_sem[i]); kfree(s); @@ -327,7 +328,7 @@ void deactivate_locked_super(struct super_block *s) { struct file_system_type *fs =3D s->s_type; if (atomic_dec_and_test(&s->s_active)) { - unregister_shrinker(&s->s_shrink); + unregister_shrinker_delayed_initiate(&s->s_shrink); fs->kill_sb(s); =20 /*