From nobody Tue Sep 9 22:19:38 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 00CD7C61DA4 for ; Thu, 23 Feb 2023 13:28:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234332AbjBWN21 (ORCPT ); Thu, 23 Feb 2023 08:28:27 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57750 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234084AbjBWN2U (ORCPT ); Thu, 23 Feb 2023 08:28:20 -0500 Received: from mail-pl1-x632.google.com (mail-pl1-x632.google.com [IPv6:2607:f8b0:4864:20::632]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CB9C516AFD for ; Thu, 23 Feb 2023 05:27:54 -0800 (PST) Received: by mail-pl1-x632.google.com with SMTP id bh1so12833262plb.11 for ; Thu, 23 Feb 2023 05:27:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=HrlDcamq7Bi2YsIBW+HX78wqxYPHwSepVRJN6rWNlHI=; b=ZOzVR5Mq59QyAziI48QmC8waUhSJJ+JNAgYhrkofGCSeV4WavcSYIdZxOhkoYD32Uy lN22vIocIhlb1bTeM1MN3wOb2DPCrzoU9gEV0RzEOao1nVLaad65GzZ9eoRcdfu0yRIw 1AkgXLFFTpYJ6QCOy8HEgbyCf1dvNFXPCDLEPMYBpU2SBjVJ7fChrOZSBnJtmdulnM/u HHQn+mpyQ5p/8kzqT6hdqMUVjI0tcFTrz6VAVhYe2PsK4L3DmoGMWmnMb9trM8hOJWbn 4brJwLGR5uq6FPraW/MLHJfslc46DMOsDOrC999BVws6VMn/wx8kKhcEVvTP+hlgyV13 Yc0w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HrlDcamq7Bi2YsIBW+HX78wqxYPHwSepVRJN6rWNlHI=; b=QOM0+y1mNSYhpyVxPKNm7oIchsuYiv+OEGzeFHJr6frBsUelAziqpBuG32c6aD9AKz nF1aV6b0zM/3VoQshrOYhyj+O5gfdpQc9v5LeczeSxO5Kk79nPPaya4CKxEclzIt+kya mACxL6PUeElo1uFMZ71kB2GYBxTXpbHnJHDpTQ2Gy5dCwe87Aqm2crqx6RcnC2NghEIK f0AevcxwGmp01lhcyGEsCvmCDBOTddQNVpZCDb5GAL548XN5iQ2awvHr2pI45CapD0bR saQalVEyIHJh+6AJ1NSPGhjznhlX79mgC+h96Tt71CUvW4s/9VOn9MRt/VZvWqit2SKp bavw== X-Gm-Message-State: AO0yUKV2QX8p40BX+ARlpmsguJd0RM/v+PUbuJMeT5pEKKEPh/JazWa8 9cmf6cneE9xgLJ1S92ZvUtZHGw== X-Google-Smtp-Source: AK7set/Q5vUZSF1XZejx+2VH547lt1MLTL0Wio2yOw08/SNhuOs+jhsXUnSmQScf+sU82FAyj/kmPA== X-Received: by 2002:a05:6a20:6914:b0:c7:5700:30bb with SMTP id q20-20020a056a20691400b000c7570030bbmr6050862pzj.4.1677158874314; Thu, 23 Feb 2023 05:27:54 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.245]) by smtp.gmail.com with ESMTPSA id g18-20020aa78752000000b005a9bf65b591sm3848591pfo.135.2023.02.23.05.27.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Feb 2023 05:27:54 -0800 (PST) From: Qi Zheng To: akpm@linux-foundation.org, tkhai@ya.ru, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com Cc: sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v2 1/7] mm: vmscan: add a map_nr_max field to shrinker_info Date: Thu, 23 Feb 2023 21:27:19 +0800 Message-Id: <20230223132725.11685-2-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230223132725.11685-1-zhengqi.arch@bytedance.com> References: <20230223132725.11685-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" To prepare for the subsequent lockless memcg slab shrink, add a map_nr_max field to struct shrinker_info to records its own real shrinker_nr_max. No functional changes. Signed-off-by: Qi Zheng Suggested-by: Kirill Tkhai --- include/linux/memcontrol.h | 1 + mm/vmscan.c | 29 ++++++++++++++++++----------- 2 files changed, 19 insertions(+), 11 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index b6eda2ab205d..aa69ea98e2d8 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -97,6 +97,7 @@ struct shrinker_info { struct rcu_head rcu; atomic_long_t *nr_deferred; unsigned long *map; + int map_nr_max; }; =20 struct lruvec_stats_percpu { diff --git a/mm/vmscan.c b/mm/vmscan.c index 9c1c5e8b24b8..9f895ca6216c 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -224,9 +224,16 @@ static struct shrinker_info *shrinker_info_protected(s= truct mem_cgroup *memcg, lockdep_is_held(&shrinker_rwsem)); } =20 +static inline bool need_expand(int new_nr_max, int old_nr_max) +{ + return round_up(new_nr_max, BITS_PER_LONG) > + round_up(old_nr_max, BITS_PER_LONG); +} + static int expand_one_shrinker_info(struct mem_cgroup *memcg, int map_size, int defer_size, - int old_map_size, int old_defer_size) + int old_map_size, int old_defer_size, + int new_nr_max) { struct shrinker_info *new, *old; struct mem_cgroup_per_node *pn; @@ -240,12 +247,16 @@ static int expand_one_shrinker_info(struct mem_cgroup= *memcg, if (!old) return 0; =20 + if (!need_expand(new_nr_max, old->map_nr_max)) + return 0; + new =3D kvmalloc_node(sizeof(*new) + size, GFP_KERNEL, nid); if (!new) return -ENOMEM; =20 new->nr_deferred =3D (atomic_long_t *)(new + 1); new->map =3D (void *)new->nr_deferred + defer_size; + new->map_nr_max =3D new_nr_max; =20 /* map: set all old bits, clear all new bits */ memset(new->map, (int)0xff, old_map_size); @@ -295,6 +306,7 @@ int alloc_shrinker_info(struct mem_cgroup *memcg) } info->nr_deferred =3D (atomic_long_t *)(info + 1); info->map =3D (void *)info->nr_deferred + defer_size; + info->map_nr_max =3D shrinker_nr_max; rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_info, info); } up_write(&shrinker_rwsem); @@ -302,12 +314,6 @@ int alloc_shrinker_info(struct mem_cgroup *memcg) return ret; } =20 -static inline bool need_expand(int nr_max) -{ - return round_up(nr_max, BITS_PER_LONG) > - round_up(shrinker_nr_max, BITS_PER_LONG); -} - static int expand_shrinker_info(int new_id) { int ret =3D 0; @@ -316,7 +322,7 @@ static int expand_shrinker_info(int new_id) int old_map_size, old_defer_size =3D 0; struct mem_cgroup *memcg; =20 - if (!need_expand(new_nr_max)) + if (!need_expand(new_nr_max, shrinker_nr_max)) goto out; =20 if (!root_mem_cgroup) @@ -332,7 +338,8 @@ static int expand_shrinker_info(int new_id) memcg =3D mem_cgroup_iter(NULL, NULL, NULL); do { ret =3D expand_one_shrinker_info(memcg, map_size, defer_size, - old_map_size, old_defer_size); + old_map_size, old_defer_size, + new_nr_max); if (ret) { mem_cgroup_iter_break(NULL, memcg); goto out; @@ -432,7 +439,7 @@ void reparent_shrinker_deferred(struct mem_cgroup *memc= g) for_each_node(nid) { child_info =3D shrinker_info_protected(memcg, nid); parent_info =3D shrinker_info_protected(parent, nid); - for (i =3D 0; i < shrinker_nr_max; i++) { + for (i =3D 0; i < child_info->map_nr_max; i++) { nr =3D atomic_long_read(&child_info->nr_deferred[i]); atomic_long_add(nr, &parent_info->nr_deferred[i]); } @@ -899,7 +906,7 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, = int nid, if (unlikely(!info)) goto unlock; =20 - for_each_set_bit(i, info->map, shrinker_nr_max) { + for_each_set_bit(i, info->map, info->map_nr_max) { struct shrink_control sc =3D { .gfp_mask =3D gfp_mask, .nid =3D nid, --=20 2.20.1 From nobody Tue Sep 9 22:19:38 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C58F3C64ED6 for ; Thu, 23 Feb 2023 13:28:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233795AbjBWN2F (ORCPT ); Thu, 23 Feb 2023 08:28:05 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57800 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229461AbjBWN2D (ORCPT ); Thu, 23 Feb 2023 08:28:03 -0500 Received: from mail-pl1-x62e.google.com (mail-pl1-x62e.google.com [IPv6:2607:f8b0:4864:20::62e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CCEAE28863 for ; Thu, 23 Feb 2023 05:28:01 -0800 (PST) Received: by mail-pl1-x62e.google.com with SMTP id q11so13162713plx.5 for ; Thu, 23 Feb 2023 05:28:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=dmBjkO7XqRgvyMtVzOfBB5ntqP3V7YsqXGlIu6MjJGc=; b=HuKGyT5g9HX+trnuB9PFEZiZQnLZwCXwzriYOLJXDNbiFiIZPICp1ljy4qSOgSab2I NMzsv7Vmj7E/5NMfLc1BWIKOnCxL+BzoZpgKcnJYEltjWlGXo0fOwx9C6Pu3NsN9u46U Kv3wFXahyCB63qXHH0daVyjbjFbiPIcvdexuFVwLoVsLDYQ5kqjx3EO/D+d1pxqe6zU/ RO8l29x25cF9n7jVV5kbIhqnom267s/kJXkRQI4UJegeb5b/0C073tGKMlhkXpxOhbeS ztqiE80MiB/pXo60Z+CGiUXjkhVqipIt+okmhgJTC36MafGn2SoK2Bm4TUXoufO3abiD loIA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=dmBjkO7XqRgvyMtVzOfBB5ntqP3V7YsqXGlIu6MjJGc=; b=rk2TBmhHxJp5KdzLl6xgOmcKylIT/gjG+WIj71AA3K7iOjYV6G5z88ZMkoGO7addzn eZSGQCG2kVqfE/3NNvdQt3k7WI0AJNLV4oQPFr1+WxojBBh651rpa6numZUUOTv4Eau2 UOCrO6SAYF3I4b+vMpjNF3Kb6wxUsnHRBc8TV3obdCVOBIBW2b+fPBPSW1mEoGHVuslI 7zyXfyss8494kzL6wvRfdG+UzEO/fw+NdIKyNc2pOCflgazN5AxL0w4soPaC3NLuZXbc YVZGcEjWXJ2TrZQ7yNYEdzZ8y7zTLFFUH7PpCigo7GAxhUUFIEk9P/N1+3HzFYmWjHim /KNQ== X-Gm-Message-State: AO0yUKXcx4IYDeURbWfYPfLTQJi/zuKGi9Z7EbFQUQr8QIvEGR9WJDCy mGce8ElgEcbL/8CCObIUa3hiNg== X-Google-Smtp-Source: AK7set/Rzr6xcsH9ySEoci9FlTvKdtpASQ4i1a9Z/tDDR2gcVSm1DDUdIYED6NggeEbWXrwoEk7f6g== X-Received: by 2002:a05:6a20:7f9c:b0:cc:4118:65c4 with SMTP id d28-20020a056a207f9c00b000cc411865c4mr889206pzj.5.1677158881203; Thu, 23 Feb 2023 05:28:01 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.245]) by smtp.gmail.com with ESMTPSA id g18-20020aa78752000000b005a9bf65b591sm3848591pfo.135.2023.02.23.05.27.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Feb 2023 05:28:00 -0800 (PST) From: Qi Zheng To: akpm@linux-foundation.org, tkhai@ya.ru, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com Cc: sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v2 2/7] mm: vmscan: make global slab shrink lockless Date: Thu, 23 Feb 2023 21:27:20 +0800 Message-Id: <20230223132725.11685-3-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230223132725.11685-1-zhengqi.arch@bytedance.com> References: <20230223132725.11685-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" The shrinker_rwsem is a global lock in shrinkers subsystem, it is easy to cause blocking in the following cases: a. the write lock of shrinker_rwsem was held for too long. For example, there are many memcgs in the system, which causes some paths to hold locks and traverse it for too long. (e.g. expand_shrinker_info()) b. the read lock of shrinker_rwsem was held for too long, and a writer came at this time. Then this writer will be forced to wait and block all subsequent readers. For example: - be scheduled when the read lock of shrinker_rwsem is held in do_shrink_slab() - some shrinker are blocked for too long. Like the case mentioned in the patchset[1]. Therefore, many times in history ([2],[3],[4],[5]), some people wanted to replace shrinker_rwsem reader with SRCU, but they all gave up because SRCU was not unconditionally enabled. But now, since commit 1cd0bd06093c ("rcu: Remove CONFIG_SRCU"), the SRCU is unconditionally enabled. So it's time to use SRCU to protect readers who previously held shrinker_rwsem. [1]. https://lore.kernel.org/lkml/20191129214541.3110-1-ptikhomirov@virtuoz= zo.com/ [2]. https://lore.kernel.org/all/1437080113.3596.2.camel@stgolabs.net/ [3]. https://lore.kernel.org/lkml/1510609063-3327-1-git-send-email-penguin-= kernel@I-love.SAKURA.ne.jp/ [4]. https://lore.kernel.org/lkml/153365347929.19074.12509495712735843805.s= tgit@localhost.localdomain/ [5]. https://lore.kernel.org/lkml/20210927074823.5825-1-sultan@kerneltoast.= com/ Signed-off-by: Qi Zheng --- mm/vmscan.c | 27 +++++++++++---------------- 1 file changed, 11 insertions(+), 16 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 9f895ca6216c..02987a6f95d1 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -202,6 +202,7 @@ static void set_task_reclaim_state(struct task_struct *= task, =20 LIST_HEAD(shrinker_list); DECLARE_RWSEM(shrinker_rwsem); +DEFINE_SRCU(shrinker_srcu); =20 #ifdef CONFIG_MEMCG static int shrinker_nr_max; @@ -706,7 +707,7 @@ void free_prealloced_shrinker(struct shrinker *shrinker) void register_shrinker_prepared(struct shrinker *shrinker) { down_write(&shrinker_rwsem); - list_add_tail(&shrinker->list, &shrinker_list); + list_add_tail_rcu(&shrinker->list, &shrinker_list); shrinker->flags |=3D SHRINKER_REGISTERED; shrinker_debugfs_add(shrinker); up_write(&shrinker_rwsem); @@ -760,13 +761,15 @@ void unregister_shrinker(struct shrinker *shrinker) return; =20 down_write(&shrinker_rwsem); - list_del(&shrinker->list); + list_del_rcu(&shrinker->list); shrinker->flags &=3D ~SHRINKER_REGISTERED; if (shrinker->flags & SHRINKER_MEMCG_AWARE) unregister_memcg_shrinker(shrinker); debugfs_entry =3D shrinker_debugfs_remove(shrinker); up_write(&shrinker_rwsem); =20 + synchronize_srcu(&shrinker_srcu); + debugfs_remove_recursive(debugfs_entry); =20 kfree(shrinker->nr_deferred); @@ -786,6 +789,7 @@ void synchronize_shrinkers(void) { down_write(&shrinker_rwsem); up_write(&shrinker_rwsem); + synchronize_srcu(&shrinker_srcu); } EXPORT_SYMBOL(synchronize_shrinkers); =20 @@ -996,6 +1000,7 @@ static unsigned long shrink_slab(gfp_t gfp_mask, int n= id, { unsigned long ret, freed =3D 0; struct shrinker *shrinker; + int srcu_idx; =20 /* * The root memcg might be allocated even though memcg is disabled @@ -1007,10 +1012,10 @@ static unsigned long shrink_slab(gfp_t gfp_mask, in= t nid, if (!mem_cgroup_disabled() && !mem_cgroup_is_root(memcg)) return shrink_slab_memcg(gfp_mask, nid, memcg, priority); =20 - if (!down_read_trylock(&shrinker_rwsem)) - goto out; + srcu_idx =3D srcu_read_lock(&shrinker_srcu); =20 - list_for_each_entry(shrinker, &shrinker_list, list) { + list_for_each_entry_srcu(shrinker, &shrinker_list, list, + srcu_read_lock_held(&shrinker_srcu)) { struct shrink_control sc =3D { .gfp_mask =3D gfp_mask, .nid =3D nid, @@ -1021,19 +1026,9 @@ static unsigned long shrink_slab(gfp_t gfp_mask, int= nid, if (ret =3D=3D SHRINK_EMPTY) ret =3D 0; freed +=3D ret; - /* - * Bail out if someone want to register a new shrinker to - * prevent the registration from being stalled for long periods - * by parallel ongoing shrinking. - */ - if (rwsem_is_contended(&shrinker_rwsem)) { - freed =3D freed ? : 1; - break; - } } =20 - up_read(&shrinker_rwsem); -out: + srcu_read_unlock(&shrinker_srcu, srcu_idx); cond_resched(); return freed; } --=20 2.20.1 From nobody Tue Sep 9 22:19:38 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 13540C636D6 for ; Thu, 23 Feb 2023 13:28:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234585AbjBWN2p (ORCPT ); Thu, 23 Feb 2023 08:28:45 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57952 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234364AbjBWN2e (ORCPT ); Thu, 23 Feb 2023 08:28:34 -0500 Received: from mail-pg1-x52c.google.com (mail-pg1-x52c.google.com [IPv6:2607:f8b0:4864:20::52c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E4D573608B for ; Thu, 23 Feb 2023 05:28:07 -0800 (PST) Received: by mail-pg1-x52c.google.com with SMTP id 130so3254099pgg.3 for ; Thu, 23 Feb 2023 05:28:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1677158887; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Syv0drMPggMMkttRMMkloSH5R6RYZ1YZAM5MesH77qc=; b=hQ0T37qts3rJeuZtmDQARv10wVciADPxTYE7nHXMnr9/N5MAW0787FBKg4eYhKDGIR D+o6nrvNrTCJZgauHW1A4MDrBHjxksREY4lTnxwUa84Fsm5l7e+fbi03X46BgEoXdVQ+ dOAn3WrLOyf4bs99Rq8TgWRjbgf9v7Q1uHlPOpcxV2y0uxirjEy5+OIZMhdR26f/63fU P2Pmq2VOLIEelMQtA+O9p5Iee99sRG2fY+Tl1/DItQIUTAgSBqbuxd7az9ZKW/udBx0b cWC6J1VhJL/7sSkSUyQ5U+ZKGWW3NEEgGBmU1h341ArROsLUeQb5G50uiU2GeEJsA+K+ hEqg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1677158887; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Syv0drMPggMMkttRMMkloSH5R6RYZ1YZAM5MesH77qc=; b=RPTQQkViz2DQEbVzqRJXrhQBuEiON3WGq0wRmIufYw/xT4cjly1dHRjzMjh4dRZtUU e3VUaJVgWWstK0riczGgQ0OR4TVrrKa3f4vBNLqX3VmFuWq7SxzgUHTKzmRxcN9nOty7 1xYyGgiDlkBaI4saW4xwmysY6EOAY5KDvaCzOs9bt6xKamcP/BlLPk+abEl6yk4yn9wD SDPs8nckfIvfDYIrfUpmHhLQwPjvvjuOgguLzt8TeFQ/i8kgIvpPu+asVhIiRaRk73Xi jeAJppbFiexdZUz1937mf6M92CfOOYazGaFmZCpMVvm1/SnuuHKzaOqdUAZ+BlTahgR9 i29g== X-Gm-Message-State: AO0yUKVLaYFi5oZ6+RVJlm60D37NAvtFxRHil88EX6deiw/bQ78XqHyq UDFHQOcKbh2e79ILhYxc63/uhw== X-Google-Smtp-Source: AK7set/3KhLevmt8IbsCAmbb8GxI559C7KBiMEGLPkGOAY2YkyfrjWjj15z30QZAy/t94RwKaPBlcQ== X-Received: by 2002:aa7:90cf:0:b0:5a9:c2d5:136c with SMTP id k15-20020aa790cf000000b005a9c2d5136cmr12399865pfk.3.1677158887363; Thu, 23 Feb 2023 05:28:07 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.245]) by smtp.gmail.com with ESMTPSA id g18-20020aa78752000000b005a9bf65b591sm3848591pfo.135.2023.02.23.05.28.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Feb 2023 05:28:07 -0800 (PST) From: Qi Zheng To: akpm@linux-foundation.org, tkhai@ya.ru, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com Cc: sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v2 3/7] mm: vmscan: make memcg slab shrink lockless Date: Thu, 23 Feb 2023 21:27:21 +0800 Message-Id: <20230223132725.11685-4-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230223132725.11685-1-zhengqi.arch@bytedance.com> References: <20230223132725.11685-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Like global slab shrink, since commit 1cd0bd06093c ("rcu: Remove CONFIG_SRCU"), it's time to use SRCU to protect readers who previously held shrinker_rwsem. We can test with the following script: ``` DIR=3D"/root/shrinker/memcg/mnt" do_create() { mkdir /sys/fs/cgroup/memory/test echo 200M > /sys/fs/cgroup/memory/test/memory.limit_in_bytes for i in `seq 0 $1`; do mkdir /sys/fs/cgroup/memory/test/$i; echo $$ > /sys/fs/cgroup/memory/test/$i/cgroup.procs; mkdir -p $DIR/$i; done } do_mount() { for i in `seq $1 $2`; do mount -t tmpfs $i $DIR/$i; done } do_touch() { for i in `seq $1 $2`; do echo $$ > /sys/fs/cgroup/memory/test/$i/cgroup.procs; dd if=3D/dev/zero of=3D$DIR/$i/file$i bs=3D1M count=3D1 & done } do_create 2000 do_mount 0 2000 do_touch 0 1000 ``` Before applying: 46.60% [kernel] [k] down_read_trylock 18.70% [kernel] [k] up_read 15.44% [kernel] [k] shrink_slab 4.37% [kernel] [k] _find_next_bit 2.75% [kernel] [k] xa_load 2.07% [kernel] [k] idr_find 1.73% [kernel] [k] do_shrink_slab 1.42% [kernel] [k] shrink_lruvec 0.74% [kernel] [k] shrink_node 0.60% [kernel] [k] list_lru_count_one After applying: 19.53% [kernel] [k] _find_next_bit 14.63% [kernel] [k] do_shrink_slab 14.58% [kernel] [k] shrink_slab 11.83% [kernel] [k] shrink_lruvec 9.33% [kernel] [k] __blk_flush_plug 6.67% [kernel] [k] mem_cgroup_iter 3.73% [kernel] [k] list_lru_count_one 2.43% [kernel] [k] shrink_node 1.96% [kernel] [k] super_cache_count 1.78% [kernel] [k] __rcu_read_unlock 1.38% [kernel] [k] __srcu_read_lock 1.30% [kernel] [k] xas_descend We can see that the readers is no longer blocked. Signed-off-by: Qi Zheng --- mm/vmscan.c | 46 +++++++++++++++++++++++++++------------------- 1 file changed, 27 insertions(+), 19 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 02987a6f95d1..25a4a660e45f 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -57,6 +57,7 @@ #include #include #include +#include =20 #include #include @@ -221,8 +222,21 @@ static inline int shrinker_defer_size(int nr_items) static struct shrinker_info *shrinker_info_protected(struct mem_cgroup *me= mcg, int nid) { - return rcu_dereference_protected(memcg->nodeinfo[nid]->shrinker_info, - lockdep_is_held(&shrinker_rwsem)); + return srcu_dereference_check(memcg->nodeinfo[nid]->shrinker_info, + &shrinker_srcu, + lockdep_is_held(&shrinker_rwsem)); +} + +static struct shrinker_info *shrinker_info_srcu(struct mem_cgroup *memcg, + int nid) +{ + return srcu_dereference(memcg->nodeinfo[nid]->shrinker_info, + &shrinker_srcu); +} + +static void free_shrinker_info_rcu(struct rcu_head *head) +{ + kvfree(container_of(head, struct shrinker_info, rcu)); } =20 static inline bool need_expand(int new_nr_max, int old_nr_max) @@ -268,7 +282,7 @@ static int expand_one_shrinker_info(struct mem_cgroup *= memcg, defer_size - old_defer_size); =20 rcu_assign_pointer(pn->shrinker_info, new); - kvfree_rcu(old, rcu); + call_srcu(&shrinker_srcu, &old->rcu, free_shrinker_info_rcu); } =20 return 0; @@ -357,13 +371,14 @@ void set_shrinker_bit(struct mem_cgroup *memcg, int n= id, int shrinker_id) { if (shrinker_id >=3D 0 && memcg && !mem_cgroup_is_root(memcg)) { struct shrinker_info *info; + int srcu_idx; =20 - rcu_read_lock(); - info =3D rcu_dereference(memcg->nodeinfo[nid]->shrinker_info); + srcu_idx =3D srcu_read_lock(&shrinker_srcu); + info =3D shrinker_info_srcu(memcg, nid); /* Pairs with smp mb in shrink_slab() */ smp_mb__before_atomic(); set_bit(shrinker_id, info->map); - rcu_read_unlock(); + srcu_read_unlock(&shrinker_srcu, srcu_idx); } } =20 @@ -377,7 +392,6 @@ static int prealloc_memcg_shrinker(struct shrinker *shr= inker) return -ENOSYS; =20 down_write(&shrinker_rwsem); - /* This may call shrinker, so it must use down_read_trylock() */ id =3D idr_alloc(&shrinker_idr, shrinker, 0, 0, GFP_KERNEL); if (id < 0) goto unlock; @@ -411,7 +425,7 @@ static long xchg_nr_deferred_memcg(int nid, struct shri= nker *shrinker, { struct shrinker_info *info; =20 - info =3D shrinker_info_protected(memcg, nid); + info =3D shrinker_info_srcu(memcg, nid); return atomic_long_xchg(&info->nr_deferred[shrinker->id], 0); } =20 @@ -420,7 +434,7 @@ static long add_nr_deferred_memcg(long nr, int nid, str= uct shrinker *shrinker, { struct shrinker_info *info; =20 - info =3D shrinker_info_protected(memcg, nid); + info =3D shrinker_info_srcu(memcg, nid); return atomic_long_add_return(nr, &info->nr_deferred[shrinker->id]); } =20 @@ -898,15 +912,14 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask= , int nid, { struct shrinker_info *info; unsigned long ret, freed =3D 0; + int srcu_idx; int i; =20 if (!mem_cgroup_online(memcg)) return 0; =20 - if (!down_read_trylock(&shrinker_rwsem)) - return 0; - - info =3D shrinker_info_protected(memcg, nid); + srcu_idx =3D srcu_read_lock(&shrinker_srcu); + info =3D shrinker_info_srcu(memcg, nid); if (unlikely(!info)) goto unlock; =20 @@ -956,14 +969,9 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask,= int nid, set_shrinker_bit(memcg, nid, i); } freed +=3D ret; - - if (rwsem_is_contended(&shrinker_rwsem)) { - freed =3D freed ? : 1; - break; - } } unlock: - up_read(&shrinker_rwsem); + srcu_read_unlock(&shrinker_srcu, srcu_idx); return freed; } #else /* CONFIG_MEMCG */ --=20 2.20.1 From nobody Tue Sep 9 22:19:38 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 59B03C61DA4 for ; Thu, 23 Feb 2023 13:28:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233956AbjBWN2T (ORCPT ); Thu, 23 Feb 2023 08:28:19 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58028 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229461AbjBWN2Q (ORCPT ); Thu, 23 Feb 2023 08:28:16 -0500 Received: from mail-pg1-x532.google.com (mail-pg1-x532.google.com [IPv6:2607:f8b0:4864:20::532]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 48DC232E41 for ; Thu, 23 Feb 2023 05:28:14 -0800 (PST) Received: by mail-pg1-x532.google.com with SMTP id q23so396099pgt.7 for ; Thu, 23 Feb 2023 05:28:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=4ra+jS9qPdJDcLL3tCNVjDgmNt8q5h7NH0ipc1XhyKg=; b=DpDn8Y+EU7riH4+ntcDG6C1rbsBVGtUkMmoAHDS+pC4P++8EynweyAV0JYqlyYGOtq X1Q+BLwoJwcnB84jpi52eVKNQ4Z7H9xuUGTHfERfoIzzzU6ACyTdYigUXyW6IU9Ious4 Dcv7MyI3TgRfz/xnXu5LeFECNtdpthD72ekZGWDFwliGf58bj+lRnE/qeFjqbyzPR/dU QS9YrLbbOfovo0TIt/QhGWwwvNLxWcvBspvW0zUIZAUavOJoYjXfSG6Wy0VfFlInlS7B HNvoy7xqBZa0nU3FIEoV776Q0WiC55EpZXUOtrYeZXGMXYVopQbU7AkcvEji/pZV0uXq Rv+A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=4ra+jS9qPdJDcLL3tCNVjDgmNt8q5h7NH0ipc1XhyKg=; b=EOH+8mc5ZoMOuS8Z+f0TutJe8L+3xekWgzJOYt0yR5dGenWZwvKDz+mE+XZN52jX0q 08y/pICnSO6aA5NF2w9u625xkHA0JZluKqhuMmbIHMov+TR3Qn7v97a9eknVCbnv1gfl r/xZRMs9+HL4PZSXkO316sbRSLyUumLiIGQw29/+/u2rafzt0g1cnFhMKxxUxbvKXvCd envoLjrPyd0bu1NOkZ/HmgzeUlJRWVWiz7LV2uo25SehWS8NxRwPirdzSvVdh8ls1Nbo o2wItosdzTIcLx9iM3dhe+ai2rKk0VB+MRNxC5pzjIErYLBUvv/EdYAkHkla76BPRxFW WBqQ== X-Gm-Message-State: AO0yUKWtsGKTO1K1eSX0InzCnCwV7O51wr9hWMzlswUVa6MafxC1Trz+ VhcUfUg874b3mkN6BEXnXBjAPg== X-Google-Smtp-Source: AK7set8Z/82YLh8rwyoUePgrhBmvgU8h/2aIZeBSX/0142PpIqvK6RYABRkBwNOTr58e5iHS5l9UsQ== X-Received: by 2002:a05:6a00:299b:b0:5a8:ae97:25f2 with SMTP id cj27-20020a056a00299b00b005a8ae9725f2mr12832291pfb.0.1677158893792; Thu, 23 Feb 2023 05:28:13 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.245]) by smtp.gmail.com with ESMTPSA id g18-20020aa78752000000b005a9bf65b591sm3848591pfo.135.2023.02.23.05.28.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Feb 2023 05:28:13 -0800 (PST) From: Qi Zheng To: akpm@linux-foundation.org, tkhai@ya.ru, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com Cc: sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v2 4/7] mm: shrinkers: make count and scan in shrinker debugfs lockless Date: Thu, 23 Feb 2023 21:27:22 +0800 Message-Id: <20230223132725.11685-5-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230223132725.11685-1-zhengqi.arch@bytedance.com> References: <20230223132725.11685-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Like global and memcg slab shrink, also use SRCU to make count and scan operations in memory shrinker debugfs lockless. Signed-off-by: Qi Zheng --- mm/shrinker_debug.c | 24 +++++++----------------- 1 file changed, 7 insertions(+), 17 deletions(-) diff --git a/mm/shrinker_debug.c b/mm/shrinker_debug.c index 39c3491e28a3..6aa7a7ec69da 100644 --- a/mm/shrinker_debug.c +++ b/mm/shrinker_debug.c @@ -9,6 +9,7 @@ /* defined in vmscan.c */ extern struct rw_semaphore shrinker_rwsem; extern struct list_head shrinker_list; +extern struct srcu_struct shrinker_srcu; =20 static DEFINE_IDA(shrinker_debugfs_ida); static struct dentry *shrinker_debugfs_root; @@ -49,18 +50,13 @@ static int shrinker_debugfs_count_show(struct seq_file = *m, void *v) struct mem_cgroup *memcg; unsigned long total; bool memcg_aware; - int ret, nid; + int ret =3D 0, nid, srcu_idx; =20 count_per_node =3D kcalloc(nr_node_ids, sizeof(unsigned long), GFP_KERNEL= ); if (!count_per_node) return -ENOMEM; =20 - ret =3D down_read_killable(&shrinker_rwsem); - if (ret) { - kfree(count_per_node); - return ret; - } - rcu_read_lock(); + srcu_idx =3D srcu_read_lock(&shrinker_srcu); =20 memcg_aware =3D shrinker->flags & SHRINKER_MEMCG_AWARE; =20 @@ -91,8 +87,7 @@ static int shrinker_debugfs_count_show(struct seq_file *m= , void *v) } } while ((memcg =3D mem_cgroup_iter(NULL, memcg, NULL)) !=3D NULL); =20 - rcu_read_unlock(); - up_read(&shrinker_rwsem); + srcu_read_unlock(&shrinker_srcu, srcu_idx); =20 kfree(count_per_node); return ret; @@ -115,9 +110,8 @@ static ssize_t shrinker_debugfs_scan_write(struct file = *file, .gfp_mask =3D GFP_KERNEL, }; struct mem_cgroup *memcg =3D NULL; - int nid; + int nid, srcu_idx; char kbuf[72]; - ssize_t ret; =20 read_len =3D size < (sizeof(kbuf) - 1) ? size : (sizeof(kbuf) - 1); if (copy_from_user(kbuf, buf, read_len)) @@ -146,11 +140,7 @@ static ssize_t shrinker_debugfs_scan_write(struct file= *file, return -EINVAL; } =20 - ret =3D down_read_killable(&shrinker_rwsem); - if (ret) { - mem_cgroup_put(memcg); - return ret; - } + srcu_idx =3D srcu_read_lock(&shrinker_srcu); =20 sc.nid =3D nid; sc.memcg =3D memcg; @@ -159,7 +149,7 @@ static ssize_t shrinker_debugfs_scan_write(struct file = *file, =20 shrinker->scan_objects(shrinker, &sc); =20 - up_read(&shrinker_rwsem); + srcu_read_unlock(&shrinker_srcu, srcu_idx); mem_cgroup_put(memcg); =20 return size; --=20 2.20.1 From nobody Tue Sep 9 22:19:38 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2136DC64ED6 for ; Thu, 23 Feb 2023 13:28:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234261AbjBWN2a (ORCPT ); Thu, 23 Feb 2023 08:28:30 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58468 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234367AbjBWN2Y (ORCPT ); Thu, 23 Feb 2023 08:28:24 -0500 Received: from mail-pg1-x535.google.com (mail-pg1-x535.google.com [IPv6:2607:f8b0:4864:20::535]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F246456786 for ; Thu, 23 Feb 2023 05:28:20 -0800 (PST) Received: by mail-pg1-x535.google.com with SMTP id z10so5621816pgr.8 for ; Thu, 23 Feb 2023 05:28:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1677158900; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=B8oNy9C+Y/zmSEnFHZueuiP36FW/sue7PAxuyUaJKNA=; b=P9pp+ZyMjawowl2CXNZ8AfWA4FDgA5DbRRV8Xgup0TXSnw3gGKc4YOluAUAPyTi+8N Eowu4hN9SdqhC2D6NScYL5anKi0WPpJZ/kDZVTjDQu+/GSTSXQemisE6KgdRU/vueJCH m5Z2d5bsrBDjNoX+11dfufhV13nj8waePZO9KexhDHGlwIWF2Eic+82YX2EcOzsUP/aF sdp2wkR1m8GXzkKkL1ndWZsnYALrWwQQa3JIsN3tgm5LwRWjlG9XLi3n5hjONfp4f/gx actRJ9hq1VYMZLcfQYjdow3np6b8hCQxlhtnE9rDWirXh8AxOKVOLv1vhYi0fp0ratLL LzNA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1677158900; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=B8oNy9C+Y/zmSEnFHZueuiP36FW/sue7PAxuyUaJKNA=; b=5fyNi3WQxSYT8wAa44gvho+9UWGId/lurvmJ5HLQcO+5h+0jTQ/LXBZaJGQ5ssmw9P wpOGSR5lJla6nMRKW0ktDCQhm8QkLUm4rNDU9zee8f0rAVOwAqj0Bva3Ngrp7hkHd6aM mcYPtWYnUj/xgnSU3QZFthxJ7ZxtmPdDBF1hWqDpBlkU262UzAOBTQjTEUrrcbXFRlnY uDF78BJewJiuG5XqbDqFUiAOCvpE1mAWwh6eKwlVFpGDp+eBJV84lO7rwqFS0t+jA4M+ WlYEu5zvHoM2cJdGBwBdvlzwq+9SK2exSURwjMTwBxmziIK74Sbn/+NMnjCdyvRvlUDO totQ== X-Gm-Message-State: AO0yUKXTs+gPEeZDfcgqj7hhDMGHoktlXNCvdwNEYbvovkdqmCib7vHB PUDQ4bagfSo+/0LjIvebcQevsA== X-Google-Smtp-Source: AK7set/qxxKJpUWiKsq4lL0Upj2vUZx+VL5loQcaP10700XUcw0lQhPoWvn8hF2CG9W/WlEr4c7i8A== X-Received: by 2002:a05:6a00:2e83:b0:5d9:bfc9:a4f with SMTP id fd3-20020a056a002e8300b005d9bfc90a4fmr3754231pfb.3.1677158900390; Thu, 23 Feb 2023 05:28:20 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.245]) by smtp.gmail.com with ESMTPSA id g18-20020aa78752000000b005a9bf65b591sm3848591pfo.135.2023.02.23.05.28.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Feb 2023 05:28:20 -0800 (PST) From: Qi Zheng To: akpm@linux-foundation.org, tkhai@ya.ru, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com Cc: sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v2 5/7] mm: vmscan: hold write lock to reparent shrinker nr_deferred Date: Thu, 23 Feb 2023 21:27:23 +0800 Message-Id: <20230223132725.11685-6-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230223132725.11685-1-zhengqi.arch@bytedance.com> References: <20230223132725.11685-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" For now, reparent_shrinker_deferred() is the only holder of read lock of shrinker_rwsem. And it already holds the global cgroup_mutex, so it will not be called in parallel. Therefore, in order to convert shrinker_rwsem to shrinker_mutex later, here we change to hold the write lock of shrinker_rwsem to reparent. Signed-off-by: Qi Zheng --- mm/vmscan.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 25a4a660e45f..89602e97583a 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -450,7 +450,7 @@ void reparent_shrinker_deferred(struct mem_cgroup *memc= g) parent =3D root_mem_cgroup; =20 /* Prevent from concurrent shrinker_info expand */ - down_read(&shrinker_rwsem); + down_write(&shrinker_rwsem); for_each_node(nid) { child_info =3D shrinker_info_protected(memcg, nid); parent_info =3D shrinker_info_protected(parent, nid); @@ -459,7 +459,7 @@ void reparent_shrinker_deferred(struct mem_cgroup *memc= g) atomic_long_add(nr, &parent_info->nr_deferred[i]); } } - up_read(&shrinker_rwsem); + up_write(&shrinker_rwsem); } =20 static bool cgroup_reclaim(struct scan_control *sc) --=20 2.20.1 From nobody Tue Sep 9 22:19:38 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9F1CFC61DA4 for ; Thu, 23 Feb 2023 13:28:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234160AbjBWN2s (ORCPT ); Thu, 23 Feb 2023 08:28:48 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58520 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234578AbjBWN2f (ORCPT ); Thu, 23 Feb 2023 08:28:35 -0500 Received: from mail-pj1-x102a.google.com (mail-pj1-x102a.google.com [IPv6:2607:f8b0:4864:20::102a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 395B1366BC for ; Thu, 23 Feb 2023 05:28:27 -0800 (PST) Received: by mail-pj1-x102a.google.com with SMTP id il18-20020a17090b165200b0023127b2d602so12255919pjb.2 for ; Thu, 23 Feb 2023 05:28:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1677158906; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=CCg67xK9T4mK7fnQXbRxQLg+KO6dWMg4osRlqGhwwto=; b=lETgDORHGvM9YstoHGbA1w0c/zosszK5Tl/tNTL5K5GecaMVH3OBn+bE4nEakg6egq Pvrj1imBQ3mvMDd4ZkkT+KTl+KI7731rrPWGJr93xhck6jDmBdEQXOnXXEO1QfhSK6RX ygXjpnptKdeVKz4W3Pi6mq9d0qV+R8FOe534PaajduVj/IHLnCHezzp/AdV++OCktNuP 8w/SRgZpNscp1wPYqJzwvIGsa1g6aKBO/N4rrdP+sT3ClXNrbTOpm5Hjwya0rBJVadLP 3Nf3Tv+Z4D5QSw8ejD088MsxJyTzHJO0ORVGeG8DQ79R3RuVcqlVVKIQp+9BWC5EuICj qEMw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1677158906; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=CCg67xK9T4mK7fnQXbRxQLg+KO6dWMg4osRlqGhwwto=; b=8QACjcmf20IJ75f1GJnqvKISNknRGJ3v/jsiL7efsiNAOPErq1/BJyF9Oa7nnJJkPz QMeGXTJv7T0iNc+qYWHUUyujOmJdymp7dYIFHSA4JIrOJxRZlStPLppZfhax7uAjx5xl OzsxgwYHVRQwF2FumfIlyyhBA5iJuB8mJupK3kshFJQxQM2IoMLQPH8ygEmwCEAxjCdB c+k7WO2DEwnHY6L27jOwPIM7G8nkFD18Qf4qbRHJe/mcqFg3o+UAONJgOd48I8iNtoEz CyY4dhiu1nPnk7QdmFSdQ/wk+8e1YaMyGdUZ+CSXWK9oNGb8Xtxcr6V9W9yA7KHOBzQp tiog== X-Gm-Message-State: AO0yUKXtQnur+ggpe2FiPEtSGuN2/lB1bIyIYy6ST7LzMpHP/Kky0EhL cyr1SKcddY0pITc+7wC6o1yEDw== X-Google-Smtp-Source: AK7set/QZbVb36Sv4JLFW6OnERnzffyx6wAbjX3Cubg0yhkLOIgaCU3CSjc6M3m8hEj3yZzy5iBl6A== X-Received: by 2002:a05:6a21:33a4:b0:cc:35f5:1a84 with SMTP id yy36-20020a056a2133a400b000cc35f51a84mr1073433pzb.5.1677158906681; Thu, 23 Feb 2023 05:28:26 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.245]) by smtp.gmail.com with ESMTPSA id g18-20020aa78752000000b005a9bf65b591sm3848591pfo.135.2023.02.23.05.28.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Feb 2023 05:28:26 -0800 (PST) From: Qi Zheng To: akpm@linux-foundation.org, tkhai@ya.ru, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com Cc: sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v2 6/7] mm: vmscan: remove shrinker_rwsem from synchronize_shrinkers() Date: Thu, 23 Feb 2023 21:27:24 +0800 Message-Id: <20230223132725.11685-7-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230223132725.11685-1-zhengqi.arch@bytedance.com> References: <20230223132725.11685-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Now there are no readers of shrinker_rwsem, so synchronize_shrinkers() does not need to hold the writer of shrinker_rwsem to wait for all running shinkers to complete, synchronize_srcu() is enough. Signed-off-by: Qi Zheng --- mm/vmscan.c | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 89602e97583a..d1a95d60d127 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -794,15 +794,11 @@ EXPORT_SYMBOL(unregister_shrinker); /** * synchronize_shrinkers - Wait for all running shrinkers to complete. * - * This is equivalent to calling unregister_shrink() and register_shrinker= (), - * but atomically and with less overhead. This is useful to guarantee that= all - * shrinker invocations have seen an update, before freeing memory, simila= r to - * rcu. + * This is useful to guarantee that all shrinker invocations have seen an + * update, before freeing memory. */ void synchronize_shrinkers(void) { - down_write(&shrinker_rwsem); - up_write(&shrinker_rwsem); synchronize_srcu(&shrinker_srcu); } EXPORT_SYMBOL(synchronize_shrinkers); --=20 2.20.1 From nobody Tue Sep 9 22:19:38 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7B1DAC636D6 for ; Thu, 23 Feb 2023 13:29:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234595AbjBWN3E (ORCPT ); Thu, 23 Feb 2023 08:29:04 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59610 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234610AbjBWN2z (ORCPT ); Thu, 23 Feb 2023 08:28:55 -0500 Received: from mail-pg1-x52e.google.com (mail-pg1-x52e.google.com [IPv6:2607:f8b0:4864:20::52e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A39E6580E2 for ; Thu, 23 Feb 2023 05:28:33 -0800 (PST) Received: by mail-pg1-x52e.google.com with SMTP id s17so5734044pgv.4 for ; Thu, 23 Feb 2023 05:28:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1677158912; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=RPT0nfWhMLdXxcydHyGYHy0yeE/SVJFKwHgDhpkqgDM=; b=F520qmCvl0UEhHNkuOAEZmp1N3sJcaWVf0fYb3nWYbFg7xaH1VreJ8NIxoJDOHmFWr fELDUoigMokRIqsdrxc8qGRde4TK1nfn/8m7o3tkhJa2mENEjDgadGrxnY5rqaQzkamU /RQ3vIZgVEe9OraPz+1/MMz1EuxiVvI+ZzojzqD2odCLdWNcrBtEzqVENWF6dvVswnZp a5ziG4nnVMEgF32yQD6np3wXBRQz+Q5XT41L4iM1wqrQtqSrf4Zc9extvwH12AOU6F09 0I920tJGW3EBzguj8MSWNnfH8JyQvo7rAULuvJmoQCsifU2+4i+3CRIKl8d3zaj+Lf9T fE0Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1677158912; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=RPT0nfWhMLdXxcydHyGYHy0yeE/SVJFKwHgDhpkqgDM=; b=OSwzX/hIORNmafzfslC01cVAXvUQXQ55+vTzvyBeHgTC/w9acJm6c/47iKxnSEJSA9 g7VKlU+6FtiXhvAm/eAnqxz76v9/Xz/VLf9SRAeoergjDC3RMRbORH/6V3aDc6m8RMlq Unmhe1Y5Gi0t6uo5uiz+YRN94KSJZZ7+Lay/Tn9XR+XI/OlK3LoKO5OlAN9chFOaM9Mo 1enoVFKyale2oSqLwr793j4RDVyHn+Zhf9O4wvayt1HvL41EHQIdce/PBrqG4uYK8ti0 28zFVXWlQj38k5JDFQkkSyxFYz3mQYi9jew8hnFDk7EQrr9WkJBqWl6D9rjxxzCuEl9Q z1rg== X-Gm-Message-State: AO0yUKXM5Zr0OsfzOV6clZ1Wl+eedJMtA4hLuN+GYwH2jKGM8vSzvoOk 4n2qD58KXNa8GmhJ/0u6wyo77A== X-Google-Smtp-Source: AK7set/SbEDbsO6C9L1y9CGpBIcDIvy9sr0sucapoT0kr6uR9zXluUZnIOxXK5HS7OjmFknadGlwnA== X-Received: by 2002:a05:6a00:4006:b0:5d9:f3a6:ef8e with SMTP id by6-20020a056a00400600b005d9f3a6ef8emr3764664pfb.2.1677158912664; Thu, 23 Feb 2023 05:28:32 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.245]) by smtp.gmail.com with ESMTPSA id g18-20020aa78752000000b005a9bf65b591sm3848591pfo.135.2023.02.23.05.28.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Feb 2023 05:28:32 -0800 (PST) From: Qi Zheng To: akpm@linux-foundation.org, tkhai@ya.ru, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com Cc: sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v2 7/7] mm: shrinkers: convert shrinker_rwsem to mutex Date: Thu, 23 Feb 2023 21:27:25 +0800 Message-Id: <20230223132725.11685-8-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230223132725.11685-1-zhengqi.arch@bytedance.com> References: <20230223132725.11685-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Now there are no readers of shrinker_rwsem, so we can simply replace it with mutex lock. Signed-off-by: Qi Zheng --- drivers/md/dm-cache-metadata.c | 2 +- drivers/md/dm-thin-metadata.c | 2 +- fs/super.c | 2 +- mm/shrinker_debug.c | 14 +++++++------- mm/vmscan.c | 34 +++++++++++++++++----------------- 5 files changed, 27 insertions(+), 27 deletions(-) diff --git a/drivers/md/dm-cache-metadata.c b/drivers/md/dm-cache-metadata.c index acffed750e3e..9e0c69958587 100644 --- a/drivers/md/dm-cache-metadata.c +++ b/drivers/md/dm-cache-metadata.c @@ -1828,7 +1828,7 @@ int dm_cache_metadata_abort(struct dm_cache_metadata = *cmd) * Replacement block manager (new_bm) is created and old_bm destroyed out= side of * cmd root_lock to avoid ABBA deadlock that would result (due to life-cy= cle of * shrinker associated with the block manager's bufio client vs cmd root_= lock). - * - must take shrinker_rwsem without holding cmd->root_lock + * - must take shrinker_mutex without holding cmd->root_lock */ new_bm =3D dm_block_manager_create(cmd->bdev, DM_CACHE_METADATA_BLOCK_SIZ= E << SECTOR_SHIFT, CACHE_MAX_CONCURRENT_LOCKS); diff --git a/drivers/md/dm-thin-metadata.c b/drivers/md/dm-thin-metadata.c index fd464fb024c3..9f5cb52c5763 100644 --- a/drivers/md/dm-thin-metadata.c +++ b/drivers/md/dm-thin-metadata.c @@ -1887,7 +1887,7 @@ int dm_pool_abort_metadata(struct dm_pool_metadata *p= md) * Replacement block manager (new_bm) is created and old_bm destroyed out= side of * pmd root_lock to avoid ABBA deadlock that would result (due to life-cy= cle of * shrinker associated with the block manager's bufio client vs pmd root_= lock). - * - must take shrinker_rwsem without holding pmd->root_lock + * - must take shrinker_mutex without holding pmd->root_lock */ new_bm =3D dm_block_manager_create(pmd->bdev, THIN_METADATA_BLOCK_SIZE <<= SECTOR_SHIFT, THIN_MAX_CONCURRENT_LOCKS); diff --git a/fs/super.c b/fs/super.c index 84332d5cb817..91a4037b1d95 100644 --- a/fs/super.c +++ b/fs/super.c @@ -54,7 +54,7 @@ static char *sb_writers_name[SB_FREEZE_LEVELS] =3D { * One thing we have to be careful of with a per-sb shrinker is that we do= n't * drop the last active reference to the superblock from within the shrink= er. * If that happens we could trigger unregistering the shrinker from within= the - * shrinker path and that leads to deadlock on the shrinker_rwsem. Hence we + * shrinker path and that leads to deadlock on the shrinker_mutex. Hence we * take a passive reference to the superblock to avoid this from occurring. */ static unsigned long super_cache_scan(struct shrinker *shrink, diff --git a/mm/shrinker_debug.c b/mm/shrinker_debug.c index 6aa7a7ec69da..b0f6aff372df 100644 --- a/mm/shrinker_debug.c +++ b/mm/shrinker_debug.c @@ -7,7 +7,7 @@ #include =20 /* defined in vmscan.c */ -extern struct rw_semaphore shrinker_rwsem; +extern struct mutex shrinker_mutex; extern struct list_head shrinker_list; extern struct srcu_struct shrinker_srcu; =20 @@ -167,7 +167,7 @@ int shrinker_debugfs_add(struct shrinker *shrinker) char buf[128]; int id; =20 - lockdep_assert_held(&shrinker_rwsem); + lockdep_assert_held(&shrinker_mutex); =20 /* debugfs isn't initialized yet, add debugfs entries later. */ if (!shrinker_debugfs_root) @@ -210,7 +210,7 @@ int shrinker_debugfs_rename(struct shrinker *shrinker, = const char *fmt, ...) if (!new) return -ENOMEM; =20 - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); =20 old =3D shrinker->name; shrinker->name =3D new; @@ -228,7 +228,7 @@ int shrinker_debugfs_rename(struct shrinker *shrinker, = const char *fmt, ...) shrinker->debugfs_entry =3D entry; } =20 - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); =20 kfree_const(old); =20 @@ -240,7 +240,7 @@ struct dentry *shrinker_debugfs_remove(struct shrinker = *shrinker) { struct dentry *entry =3D shrinker->debugfs_entry; =20 - lockdep_assert_held(&shrinker_rwsem); + lockdep_assert_held(&shrinker_mutex); =20 kfree_const(shrinker->name); shrinker->name =3D NULL; @@ -265,14 +265,14 @@ static int __init shrinker_debugfs_init(void) shrinker_debugfs_root =3D dentry; =20 /* Create debugfs entries for shrinkers registered at boot */ - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); list_for_each_entry(shrinker, &shrinker_list, list) if (!shrinker->debugfs_entry) { ret =3D shrinker_debugfs_add(shrinker); if (ret) break; } - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); =20 return ret; } diff --git a/mm/vmscan.c b/mm/vmscan.c index d1a95d60d127..27ef9946ae8a 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -35,7 +35,7 @@ #include #include #include -#include +#include #include #include #include @@ -202,7 +202,7 @@ static void set_task_reclaim_state(struct task_struct *= task, } =20 LIST_HEAD(shrinker_list); -DECLARE_RWSEM(shrinker_rwsem); +DEFINE_MUTEX(shrinker_mutex); DEFINE_SRCU(shrinker_srcu); =20 #ifdef CONFIG_MEMCG @@ -224,7 +224,7 @@ static struct shrinker_info *shrinker_info_protected(st= ruct mem_cgroup *memcg, { return srcu_dereference_check(memcg->nodeinfo[nid]->shrinker_info, &shrinker_srcu, - lockdep_is_held(&shrinker_rwsem)); + lockdep_is_held(&shrinker_mutex)); } =20 static struct shrinker_info *shrinker_info_srcu(struct mem_cgroup *memcg, @@ -308,7 +308,7 @@ int alloc_shrinker_info(struct mem_cgroup *memcg) int nid, size, ret =3D 0; int map_size, defer_size =3D 0; =20 - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); map_size =3D shrinker_map_size(shrinker_nr_max); defer_size =3D shrinker_defer_size(shrinker_nr_max); size =3D map_size + defer_size; @@ -324,7 +324,7 @@ int alloc_shrinker_info(struct mem_cgroup *memcg) info->map_nr_max =3D shrinker_nr_max; rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_info, info); } - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); =20 return ret; } @@ -343,7 +343,7 @@ static int expand_shrinker_info(int new_id) if (!root_mem_cgroup) goto out; =20 - lockdep_assert_held(&shrinker_rwsem); + lockdep_assert_held(&shrinker_mutex); =20 map_size =3D shrinker_map_size(new_nr_max); defer_size =3D shrinker_defer_size(new_nr_max); @@ -391,7 +391,7 @@ static int prealloc_memcg_shrinker(struct shrinker *shr= inker) if (mem_cgroup_disabled()) return -ENOSYS; =20 - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); id =3D idr_alloc(&shrinker_idr, shrinker, 0, 0, GFP_KERNEL); if (id < 0) goto unlock; @@ -405,7 +405,7 @@ static int prealloc_memcg_shrinker(struct shrinker *shr= inker) shrinker->id =3D id; ret =3D 0; unlock: - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); return ret; } =20 @@ -415,7 +415,7 @@ static void unregister_memcg_shrinker(struct shrinker *= shrinker) =20 BUG_ON(id < 0); =20 - lockdep_assert_held(&shrinker_rwsem); + lockdep_assert_held(&shrinker_mutex); =20 idr_remove(&shrinker_idr, id); } @@ -450,7 +450,7 @@ void reparent_shrinker_deferred(struct mem_cgroup *memc= g) parent =3D root_mem_cgroup; =20 /* Prevent from concurrent shrinker_info expand */ - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); for_each_node(nid) { child_info =3D shrinker_info_protected(memcg, nid); parent_info =3D shrinker_info_protected(parent, nid); @@ -459,7 +459,7 @@ void reparent_shrinker_deferred(struct mem_cgroup *memc= g) atomic_long_add(nr, &parent_info->nr_deferred[i]); } } - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); } =20 static bool cgroup_reclaim(struct scan_control *sc) @@ -708,9 +708,9 @@ void free_prealloced_shrinker(struct shrinker *shrinker) shrinker->name =3D NULL; #endif if (shrinker->flags & SHRINKER_MEMCG_AWARE) { - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); unregister_memcg_shrinker(shrinker); - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); return; } =20 @@ -720,11 +720,11 @@ void free_prealloced_shrinker(struct shrinker *shrink= er) =20 void register_shrinker_prepared(struct shrinker *shrinker) { - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); list_add_tail_rcu(&shrinker->list, &shrinker_list); shrinker->flags |=3D SHRINKER_REGISTERED; shrinker_debugfs_add(shrinker); - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); } =20 static int __register_shrinker(struct shrinker *shrinker) @@ -774,13 +774,13 @@ void unregister_shrinker(struct shrinker *shrinker) if (!(shrinker->flags & SHRINKER_REGISTERED)) return; =20 - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); list_del_rcu(&shrinker->list); shrinker->flags &=3D ~SHRINKER_REGISTERED; if (shrinker->flags & SHRINKER_MEMCG_AWARE) unregister_memcg_shrinker(shrinker); debugfs_entry =3D shrinker_debugfs_remove(shrinker); - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); =20 synchronize_srcu(&shrinker_srcu); =20 --=20 2.20.1