From nobody Mon Sep 8 18:52:44 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6202BC7EE31 for ; Sun, 26 Feb 2023 14:53:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230404AbjBZOxy (ORCPT ); Sun, 26 Feb 2023 09:53:54 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55102 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230330AbjBZOxQ (ORCPT ); Sun, 26 Feb 2023 09:53:16 -0500 Received: from mail-pj1-x1035.google.com (mail-pj1-x1035.google.com [IPv6:2607:f8b0:4864:20::1035]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4650A46BF for ; Sun, 26 Feb 2023 06:49:48 -0800 (PST) Received: by mail-pj1-x1035.google.com with SMTP id qa18-20020a17090b4fd200b0023750b675f5so7526296pjb.3 for ; Sun, 26 Feb 2023 06:49:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=jZz7Wrj6RpcNJkd0HlCRkRoarWFliGItBtXoMhl1VyQ=; b=OEQgF8XFhS1YnbfOQOJh1IFogO5l8aVUQrCwR9nyy7oN1K9Ji93rp0LItnd8h4XE6q oXvlgHNvYZY7Sru38EHw5XpbSsjVDCpErYLQOtN774iKL22Mm3meSyNn/DoDpseqe7VK M148czmxLrasp9EktCEGPZBaRaJCIL/h+oxW/Len8VuNVSZLJgVQPTgQZl2H0oXCxaKT th31uEB6vb+9QTQp2lgTemj0Wo5Eo3476YMBJNBoEXl1eXjK1lRElSTkvEo/pHe8O8zz kKJciJ2YT4YMKnGfyaACr4ey76nUPX2Xt475p0Ss+/Ot3sZ5uU3wsAqqCutxvYDzlxUi SM6Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=jZz7Wrj6RpcNJkd0HlCRkRoarWFliGItBtXoMhl1VyQ=; b=LAVMf7rtYuChFwk9LjoYprScrpXUZ2a8xb9PjRk7p9mL4nWq3TlQv0olhss1Km1CJE wSQT3A25hcZGg9YBRvc4PK8g4qZgb6VZph1euCVuQLVslQtuNFgm0sy2fkHniY4TNDpj 2pDMfIhI4Cjpl/L+AiqKWT1J/n00s7a008R4+5bydOF4TTN8U2ree6Y0OfQnCZErtY5v Ey6jqf6kctQMU3F4J+Dss4ueq/ewQAgTAAX8dYae6fYyEjB6++HlrCZMl+tsZUfu7kXZ gdJxtgeMMJXAXr6EnD2/eH2J/bDUINcZEgdb+Ulo7qdmWxsnwHTTZY/6zGm4PG6UTjSa xVDg== X-Gm-Message-State: AO0yUKVmocvjQkl7hhdIB6ay0VrLEH0BM6xwpgaJyKE+shBu6sKIGUjn KFVJ5JJSEItLqMk1jDwZleeVDA== X-Google-Smtp-Source: AK7set+gayem3mXhkOe0NYaFyVrW/tJ1qNYdSQjiJZ8SAHMh4YRVx23yElXFyFWK9z6fB2W3nd2NCA== X-Received: by 2002:a17:902:d4c8:b0:19a:7217:32af with SMTP id o8-20020a170902d4c800b0019a721732afmr23248669plg.5.1677422896084; Sun, 26 Feb 2023 06:48:16 -0800 (PST) Received: from localhost.localdomain ([139.177.225.248]) by smtp.gmail.com with ESMTPSA id y20-20020a170902ed5400b0019c2cf12d15sm2755589plb.116.2023.02.26.06.48.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 26 Feb 2023 06:48:15 -0800 (PST) From: Qi Zheng To: akpm@linux-foundation.org, tkhai@ya.ru, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com Cc: sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v3 1/8] mm: vmscan: add a map_nr_max field to shrinker_info Date: Sun, 26 Feb 2023 22:46:48 +0800 Message-Id: <20230226144655.79778-2-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230226144655.79778-1-zhengqi.arch@bytedance.com> References: <20230226144655.79778-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" To prepare for the subsequent lockless memcg slab shrink, add a map_nr_max field to struct shrinker_info to records its own real shrinker_nr_max. Suggested-by: Kirill Tkhai Signed-off-by: Qi Zheng --- include/linux/memcontrol.h | 1 + mm/vmscan.c | 41 ++++++++++++++++++++++---------------- 2 files changed, 25 insertions(+), 17 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index b6eda2ab205d..aa69ea98e2d8 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -97,6 +97,7 @@ struct shrinker_info { struct rcu_head rcu; atomic_long_t *nr_deferred; unsigned long *map; + int map_nr_max; }; =20 struct lruvec_stats_percpu { diff --git a/mm/vmscan.c b/mm/vmscan.c index 9c1c5e8b24b8..546c07ccb3bc 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -224,9 +224,16 @@ static struct shrinker_info *shrinker_info_protected(s= truct mem_cgroup *memcg, lockdep_is_held(&shrinker_rwsem)); } =20 +static inline bool need_expand(int new_nr_max, int old_nr_max) +{ + return round_up(new_nr_max, BITS_PER_LONG) > + round_up(old_nr_max, BITS_PER_LONG); +} + static int expand_one_shrinker_info(struct mem_cgroup *memcg, int map_size, int defer_size, - int old_map_size, int old_defer_size) + int old_map_size, int old_defer_size, + int new_nr_max) { struct shrinker_info *new, *old; struct mem_cgroup_per_node *pn; @@ -240,12 +247,17 @@ static int expand_one_shrinker_info(struct mem_cgroup= *memcg, if (!old) return 0; =20 + /* Already expanded this shrinker_info */ + if (!need_expand(new_nr_max, old->map_nr_max)) + return 0; + new =3D kvmalloc_node(sizeof(*new) + size, GFP_KERNEL, nid); if (!new) return -ENOMEM; =20 new->nr_deferred =3D (atomic_long_t *)(new + 1); new->map =3D (void *)new->nr_deferred + defer_size; + new->map_nr_max =3D new_nr_max; =20 /* map: set all old bits, clear all new bits */ memset(new->map, (int)0xff, old_map_size); @@ -295,6 +307,7 @@ int alloc_shrinker_info(struct mem_cgroup *memcg) } info->nr_deferred =3D (atomic_long_t *)(info + 1); info->map =3D (void *)info->nr_deferred + defer_size; + info->map_nr_max =3D shrinker_nr_max; rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_info, info); } up_write(&shrinker_rwsem); @@ -302,23 +315,14 @@ int alloc_shrinker_info(struct mem_cgroup *memcg) return ret; } =20 -static inline bool need_expand(int nr_max) -{ - return round_up(nr_max, BITS_PER_LONG) > - round_up(shrinker_nr_max, BITS_PER_LONG); -} - static int expand_shrinker_info(int new_id) { int ret =3D 0; - int new_nr_max =3D new_id + 1; + int new_nr_max =3D round_up(new_id + 1, BITS_PER_LONG); int map_size, defer_size =3D 0; int old_map_size, old_defer_size =3D 0; struct mem_cgroup *memcg; =20 - if (!need_expand(new_nr_max)) - goto out; - if (!root_mem_cgroup) goto out; =20 @@ -332,7 +336,8 @@ static int expand_shrinker_info(int new_id) memcg =3D mem_cgroup_iter(NULL, NULL, NULL); do { ret =3D expand_one_shrinker_info(memcg, map_size, defer_size, - old_map_size, old_defer_size); + old_map_size, old_defer_size, + new_nr_max); if (ret) { mem_cgroup_iter_break(NULL, memcg); goto out; @@ -352,9 +357,11 @@ void set_shrinker_bit(struct mem_cgroup *memcg, int ni= d, int shrinker_id) =20 rcu_read_lock(); info =3D rcu_dereference(memcg->nodeinfo[nid]->shrinker_info); - /* Pairs with smp mb in shrink_slab() */ - smp_mb__before_atomic(); - set_bit(shrinker_id, info->map); + if (!WARN_ON_ONCE(shrinker_id >=3D info->map_nr_max)) { + /* Pairs with smp mb in shrink_slab() */ + smp_mb__before_atomic(); + set_bit(shrinker_id, info->map); + } rcu_read_unlock(); } } @@ -432,7 +439,7 @@ void reparent_shrinker_deferred(struct mem_cgroup *memc= g) for_each_node(nid) { child_info =3D shrinker_info_protected(memcg, nid); parent_info =3D shrinker_info_protected(parent, nid); - for (i =3D 0; i < shrinker_nr_max; i++) { + for (i =3D 0; i < child_info->map_nr_max; i++) { nr =3D atomic_long_read(&child_info->nr_deferred[i]); atomic_long_add(nr, &parent_info->nr_deferred[i]); } @@ -899,7 +906,7 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, = int nid, if (unlikely(!info)) goto unlock; =20 - for_each_set_bit(i, info->map, shrinker_nr_max) { + for_each_set_bit(i, info->map, info->map_nr_max) { struct shrink_control sc =3D { .gfp_mask =3D gfp_mask, .nid =3D nid, --=20 2.20.1 From nobody Mon Sep 8 18:52:44 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 11841C7EE23 for ; Sun, 26 Feb 2023 14:53:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230392AbjBZOxw (ORCPT ); Sun, 26 Feb 2023 09:53:52 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44776 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230208AbjBZOxL (ORCPT ); Sun, 26 Feb 2023 09:53:11 -0500 Received: from mail-pl1-x632.google.com (mail-pl1-x632.google.com [IPv6:2607:f8b0:4864:20::632]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BE4E91A4A4 for ; Sun, 26 Feb 2023 06:49:30 -0800 (PST) Received: by mail-pl1-x632.google.com with SMTP id i3so4209103plg.6 for ; Sun, 26 Feb 2023 06:49:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1677422902; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=xJVL9LTM3YR7R2xJV6pOC5vrFajMNWUdaUGoIRklN4g=; b=ALVQXXOfu8Q3gVhZC/7gcT/As/x0PQNyX2nqdY6UDPAFzmCXukW/onK5TzeE9pcwvk vXJzQkiBP+sXbTYXsw+HpB7JRRbrwmsjkOEq5DQ1S1Eac1N0S/0/DsgoqOjYjHQYnPSR P4R9Pfo0jggo+dcJDr8XJLDD9doWZVqghSIkea6uqcKjvoOktQF/G7VDIKPmpqIX0uaq CUp5L8HF8li82mResfb29HBRedCwzzg7BH3UMRf7q7wMBQ/7+ybQtKL36gQDJZu0gwHP FCqtYPLddEJNMwfVTwEbd49hBEqsrqinMleX5XcFcj1s7BFSCNs7Wfig3bKPX0H4mz4V BKmQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1677422902; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xJVL9LTM3YR7R2xJV6pOC5vrFajMNWUdaUGoIRklN4g=; b=kx69l754ul2Sqs4Xs1g0SDtaTzNs2jNSxZqK9n+D2Zq0rrLCiVbdckVUIiuaRFhYTN c8QFmWshy3iFhhmzlXyLSlk/1fDoNOiHxW1+vHbc0iCpx2Dw+x4OXTXVp0ukGQ7Ne5k+ UvS+k/AgAxDmhemEXQe8TrIZPkq5GXPe+2yQjS8UGeTOdJpV6Of0Tkv7eEIa/JVJeowc D/IDUbzatyVvj9Fe4I2YZ61IAo+6U8bnjnhFRtb7DREZowZN6A90cQ+HM+c9ovAbGL3G Cqxihy/TiyA0BK74Oj8KRH+jLbd2U0qomOvEVmdo+4hoFb32WNKseBr2FRxZqMd/znAt Vy5A== X-Gm-Message-State: AO0yUKW0avX26tNNgSBfu+/wImHGDSfTj6u7rPky0oodb/epom0Yc/kc gxCwgvkCoE4L9SQdenslcYDjew== X-Google-Smtp-Source: AK7set+bbch+q47tvYRNxM7CkTvsssKUNlySDXHnmC0iMbwnLjHUXiyS9TSOg6OMzWfUw8IIF2O97w== X-Received: by 2002:a17:902:e84b:b0:19a:7439:3e98 with SMTP id t11-20020a170902e84b00b0019a74393e98mr23301977plg.4.1677422901796; Sun, 26 Feb 2023 06:48:21 -0800 (PST) Received: from localhost.localdomain ([139.177.225.248]) by smtp.gmail.com with ESMTPSA id y20-20020a170902ed5400b0019c2cf12d15sm2755589plb.116.2023.02.26.06.48.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 26 Feb 2023 06:48:21 -0800 (PST) From: Qi Zheng To: akpm@linux-foundation.org, tkhai@ya.ru, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com Cc: sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v3 2/8] mm: vmscan: make global slab shrink lockless Date: Sun, 26 Feb 2023 22:46:49 +0800 Message-Id: <20230226144655.79778-3-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230226144655.79778-1-zhengqi.arch@bytedance.com> References: <20230226144655.79778-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" The shrinker_rwsem is a global lock in shrinkers subsystem, it is easy to cause blocking in the following cases: a. the write lock of shrinker_rwsem was held for too long. For example, there are many memcgs in the system, which causes some paths to hold locks and traverse it for too long. (e.g. expand_shrinker_info()) b. the read lock of shrinker_rwsem was held for too long, and a writer came at this time. Then this writer will be forced to wait and block all subsequent readers. For example: - be scheduled when the read lock of shrinker_rwsem is held in do_shrink_slab() - some shrinker are blocked for too long. Like the case mentioned in the patchset[1]. Therefore, many times in history ([2],[3],[4],[5]), some people wanted to replace shrinker_rwsem reader with SRCU, but they all gave up because SRCU was not unconditionally enabled. But now, since commit 1cd0bd06093c ("rcu: Remove CONFIG_SRCU"), the SRCU is unconditionally enabled. So it's time to use SRCU to protect readers who previously held shrinker_rwsem. [1]. https://lore.kernel.org/lkml/20191129214541.3110-1-ptikhomirov@virtuoz= zo.com/ [2]. https://lore.kernel.org/all/1437080113.3596.2.camel@stgolabs.net/ [3]. https://lore.kernel.org/lkml/1510609063-3327-1-git-send-email-penguin-= kernel@I-love.SAKURA.ne.jp/ [4]. https://lore.kernel.org/lkml/153365347929.19074.12509495712735843805.s= tgit@localhost.localdomain/ [5]. https://lore.kernel.org/lkml/20210927074823.5825-1-sultan@kerneltoast.= com/ Signed-off-by: Qi Zheng --- mm/vmscan.c | 27 +++++++++++---------------- 1 file changed, 11 insertions(+), 16 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 546c07ccb3bc..2a21a84d3db1 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -202,6 +202,7 @@ static void set_task_reclaim_state(struct task_struct *= task, =20 LIST_HEAD(shrinker_list); DECLARE_RWSEM(shrinker_rwsem); +DEFINE_SRCU(shrinker_srcu); =20 #ifdef CONFIG_MEMCG static int shrinker_nr_max; @@ -706,7 +707,7 @@ void free_prealloced_shrinker(struct shrinker *shrinker) void register_shrinker_prepared(struct shrinker *shrinker) { down_write(&shrinker_rwsem); - list_add_tail(&shrinker->list, &shrinker_list); + list_add_tail_rcu(&shrinker->list, &shrinker_list); shrinker->flags |=3D SHRINKER_REGISTERED; shrinker_debugfs_add(shrinker); up_write(&shrinker_rwsem); @@ -760,13 +761,15 @@ void unregister_shrinker(struct shrinker *shrinker) return; =20 down_write(&shrinker_rwsem); - list_del(&shrinker->list); + list_del_rcu(&shrinker->list); shrinker->flags &=3D ~SHRINKER_REGISTERED; if (shrinker->flags & SHRINKER_MEMCG_AWARE) unregister_memcg_shrinker(shrinker); debugfs_entry =3D shrinker_debugfs_remove(shrinker); up_write(&shrinker_rwsem); =20 + synchronize_srcu(&shrinker_srcu); + debugfs_remove_recursive(debugfs_entry); =20 kfree(shrinker->nr_deferred); @@ -786,6 +789,7 @@ void synchronize_shrinkers(void) { down_write(&shrinker_rwsem); up_write(&shrinker_rwsem); + synchronize_srcu(&shrinker_srcu); } EXPORT_SYMBOL(synchronize_shrinkers); =20 @@ -996,6 +1000,7 @@ static unsigned long shrink_slab(gfp_t gfp_mask, int n= id, { unsigned long ret, freed =3D 0; struct shrinker *shrinker; + int srcu_idx; =20 /* * The root memcg might be allocated even though memcg is disabled @@ -1007,10 +1012,10 @@ static unsigned long shrink_slab(gfp_t gfp_mask, in= t nid, if (!mem_cgroup_disabled() && !mem_cgroup_is_root(memcg)) return shrink_slab_memcg(gfp_mask, nid, memcg, priority); =20 - if (!down_read_trylock(&shrinker_rwsem)) - goto out; + srcu_idx =3D srcu_read_lock(&shrinker_srcu); =20 - list_for_each_entry(shrinker, &shrinker_list, list) { + list_for_each_entry_srcu(shrinker, &shrinker_list, list, + srcu_read_lock_held(&shrinker_srcu)) { struct shrink_control sc =3D { .gfp_mask =3D gfp_mask, .nid =3D nid, @@ -1021,19 +1026,9 @@ static unsigned long shrink_slab(gfp_t gfp_mask, int= nid, if (ret =3D=3D SHRINK_EMPTY) ret =3D 0; freed +=3D ret; - /* - * Bail out if someone want to register a new shrinker to - * prevent the registration from being stalled for long periods - * by parallel ongoing shrinking. - */ - if (rwsem_is_contended(&shrinker_rwsem)) { - freed =3D freed ? : 1; - break; - } } =20 - up_read(&shrinker_rwsem); -out: + srcu_read_unlock(&shrinker_srcu, srcu_idx); cond_resched(); return freed; } --=20 2.20.1 From nobody Mon Sep 8 18:52:44 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CF0CBC64ED6 for ; Sun, 26 Feb 2023 14:54:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230435AbjBZOyM (ORCPT ); Sun, 26 Feb 2023 09:54:12 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54748 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230317AbjBZOxb (ORCPT ); Sun, 26 Feb 2023 09:53:31 -0500 Received: from mail-pl1-x62f.google.com (mail-pl1-x62f.google.com [IPv6:2607:f8b0:4864:20::62f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A688814213 for ; Sun, 26 Feb 2023 06:49:43 -0800 (PST) Received: by mail-pl1-x62f.google.com with SMTP id v11so680713plz.8 for ; Sun, 26 Feb 2023 06:49:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=gvRso6AuWsz0X9JCAZwWH9GIT0TN5B8gocPlJ++Lup8=; b=LzQ+kcdgxQjvYV6/aBTUiIdXty1SduBzej96ViyOvdlzB3Fsg4Lcdgbz3j4hmKO+8+ TIcLX9NYqDrIQ5/ezPI3RaFfYwQ1ZEDeoEpqbn5P8z2NLz+savfhhRBcSPVq7BwrsDXr R1O+OCmiRS291b9X4XNwI5b5W0qVttXgY3Oqixpog+eVUFyly+Rm9YGT9YU8Ro/3F5Pw d/nQ47A2O7bUl+GBn4gAU0PulKhyQ7Bu/CfMAW5zYIkO2kY76GczKehRR3kCEzOW262q ug1kCR6iRSBx0q+cNBVn+mB+nddqScXB+yCOQ55hN5exNnBS/N7LGAnn4tefuskRqhyI gJZw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=gvRso6AuWsz0X9JCAZwWH9GIT0TN5B8gocPlJ++Lup8=; b=HhtZSwgclEP/XebHoIteUQPhgLBJgRKHRySrYebUH6CQB1wHmMpU7u46yPLU2dcGuf KLwv/hd7sYlm+YVzzJS54bQX//MyxAXOakbSmiTJ7rEKiY/tBZOJXkKl2VTGGTpvdNXi jjq17L/aboFZnEXqxAJRyLymi5FruidXvdO8+9BvwVeOVdWHVpTZ4Hb3OgJUgZY5syT+ tnSeT9zR1+7/XSmTW9n04lyN9VPiK4lYs/ty38O9yodaWm2XiVBlzxxGnAvfJgOwKFh6 ksC3IDkjYKwkyUoQwudnfnXZDZ5/W8htryt5zCHtK/9HMEWs0cLO3FVYMPWnXo5t3QgS fryg== X-Gm-Message-State: AO0yUKUTQHkrXvEwvkAVUeG9zm0gphN9148oDbrsynVxVZ8QolxlU+oH cJC5q9xyg6M2o11Z0vGk91ZiiA== X-Google-Smtp-Source: AK7set8spgroQQIMXk8gdX51DNtELr9O76gOJSC+OLLzgfCw5s0Vzk98NSKahnR1+3e2KLwZ3nhiFQ== X-Received: by 2002:a17:903:230f:b0:19a:7060:948 with SMTP id d15-20020a170903230f00b0019a70600948mr21481855plh.1.1677422907679; Sun, 26 Feb 2023 06:48:27 -0800 (PST) Received: from localhost.localdomain ([139.177.225.248]) by smtp.gmail.com with ESMTPSA id y20-20020a170902ed5400b0019c2cf12d15sm2755589plb.116.2023.02.26.06.48.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 26 Feb 2023 06:48:27 -0800 (PST) From: Qi Zheng To: akpm@linux-foundation.org, tkhai@ya.ru, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com Cc: sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v3 3/8] mm: vmscan: make memcg slab shrink lockless Date: Sun, 26 Feb 2023 22:46:50 +0800 Message-Id: <20230226144655.79778-4-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230226144655.79778-1-zhengqi.arch@bytedance.com> References: <20230226144655.79778-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Like global slab shrink, since commit 1cd0bd06093c ("rcu: Remove CONFIG_SRCU"), it's time to use SRCU to protect readers who previously held shrinker_rwsem. We can test with the following script: ``` DIR=3D"/root/shrinker/memcg/mnt" do_create() { mkdir /sys/fs/cgroup/memory/test echo 200M > /sys/fs/cgroup/memory/test/memory.limit_in_bytes for i in `seq 0 $1`; do mkdir /sys/fs/cgroup/memory/test/$i; echo $$ > /sys/fs/cgroup/memory/test/$i/cgroup.procs; mkdir -p $DIR/$i; done } do_mount() { for i in `seq $1 $2`; do mount -t tmpfs $i $DIR/$i; done } do_touch() { for i in `seq $1 $2`; do echo $$ > /sys/fs/cgroup/memory/test/$i/cgroup.procs; dd if=3D/dev/zero of=3D$DIR/$i/file$i bs=3D1M count=3D1 & done } do_create 2000 do_mount 0 2000 do_touch 0 1000 ``` Before applying: 46.60% [kernel] [k] down_read_trylock 18.70% [kernel] [k] up_read 15.44% [kernel] [k] shrink_slab 4.37% [kernel] [k] _find_next_bit 2.75% [kernel] [k] xa_load 2.07% [kernel] [k] idr_find 1.73% [kernel] [k] do_shrink_slab 1.42% [kernel] [k] shrink_lruvec 0.74% [kernel] [k] shrink_node 0.60% [kernel] [k] list_lru_count_one After applying: 19.53% [kernel] [k] _find_next_bit 14.63% [kernel] [k] do_shrink_slab 14.58% [kernel] [k] shrink_slab 11.83% [kernel] [k] shrink_lruvec 9.33% [kernel] [k] __blk_flush_plug 6.67% [kernel] [k] mem_cgroup_iter 3.73% [kernel] [k] list_lru_count_one 2.43% [kernel] [k] shrink_node 1.96% [kernel] [k] super_cache_count 1.78% [kernel] [k] __rcu_read_unlock 1.38% [kernel] [k] __srcu_read_lock 1.30% [kernel] [k] xas_descend We can see that the readers is no longer blocked. Signed-off-by: Qi Zheng --- mm/vmscan.c | 46 +++++++++++++++++++++++++++------------------- 1 file changed, 27 insertions(+), 19 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 2a21a84d3db1..490764f8e085 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -57,6 +57,7 @@ #include #include #include +#include =20 #include #include @@ -221,8 +222,21 @@ static inline int shrinker_defer_size(int nr_items) static struct shrinker_info *shrinker_info_protected(struct mem_cgroup *me= mcg, int nid) { - return rcu_dereference_protected(memcg->nodeinfo[nid]->shrinker_info, - lockdep_is_held(&shrinker_rwsem)); + return srcu_dereference_check(memcg->nodeinfo[nid]->shrinker_info, + &shrinker_srcu, + lockdep_is_held(&shrinker_rwsem)); +} + +static struct shrinker_info *shrinker_info_srcu(struct mem_cgroup *memcg, + int nid) +{ + return srcu_dereference(memcg->nodeinfo[nid]->shrinker_info, + &shrinker_srcu); +} + +static void free_shrinker_info_rcu(struct rcu_head *head) +{ + kvfree(container_of(head, struct shrinker_info, rcu)); } =20 static inline bool need_expand(int new_nr_max, int old_nr_max) @@ -269,7 +283,7 @@ static int expand_one_shrinker_info(struct mem_cgroup *= memcg, defer_size - old_defer_size); =20 rcu_assign_pointer(pn->shrinker_info, new); - kvfree_rcu(old, rcu); + call_srcu(&shrinker_srcu, &old->rcu, free_shrinker_info_rcu); } =20 return 0; @@ -355,15 +369,16 @@ void set_shrinker_bit(struct mem_cgroup *memcg, int n= id, int shrinker_id) { if (shrinker_id >=3D 0 && memcg && !mem_cgroup_is_root(memcg)) { struct shrinker_info *info; + int srcu_idx; =20 - rcu_read_lock(); - info =3D rcu_dereference(memcg->nodeinfo[nid]->shrinker_info); + srcu_idx =3D srcu_read_lock(&shrinker_srcu); + info =3D shrinker_info_srcu(memcg, nid); if (!WARN_ON_ONCE(shrinker_id >=3D info->map_nr_max)) { /* Pairs with smp mb in shrink_slab() */ smp_mb__before_atomic(); set_bit(shrinker_id, info->map); } - rcu_read_unlock(); + srcu_read_unlock(&shrinker_srcu, srcu_idx); } } =20 @@ -377,7 +392,6 @@ static int prealloc_memcg_shrinker(struct shrinker *shr= inker) return -ENOSYS; =20 down_write(&shrinker_rwsem); - /* This may call shrinker, so it must use down_read_trylock() */ id =3D idr_alloc(&shrinker_idr, shrinker, 0, 0, GFP_KERNEL); if (id < 0) goto unlock; @@ -411,7 +425,7 @@ static long xchg_nr_deferred_memcg(int nid, struct shri= nker *shrinker, { struct shrinker_info *info; =20 - info =3D shrinker_info_protected(memcg, nid); + info =3D shrinker_info_srcu(memcg, nid); return atomic_long_xchg(&info->nr_deferred[shrinker->id], 0); } =20 @@ -420,7 +434,7 @@ static long add_nr_deferred_memcg(long nr, int nid, str= uct shrinker *shrinker, { struct shrinker_info *info; =20 - info =3D shrinker_info_protected(memcg, nid); + info =3D shrinker_info_srcu(memcg, nid); return atomic_long_add_return(nr, &info->nr_deferred[shrinker->id]); } =20 @@ -898,15 +912,14 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask= , int nid, { struct shrinker_info *info; unsigned long ret, freed =3D 0; + int srcu_idx; int i; =20 if (!mem_cgroup_online(memcg)) return 0; =20 - if (!down_read_trylock(&shrinker_rwsem)) - return 0; - - info =3D shrinker_info_protected(memcg, nid); + srcu_idx =3D srcu_read_lock(&shrinker_srcu); + info =3D shrinker_info_srcu(memcg, nid); if (unlikely(!info)) goto unlock; =20 @@ -956,14 +969,9 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask,= int nid, set_shrinker_bit(memcg, nid, i); } freed +=3D ret; - - if (rwsem_is_contended(&shrinker_rwsem)) { - freed =3D freed ? : 1; - break; - } } unlock: - up_read(&shrinker_rwsem); + srcu_read_unlock(&shrinker_srcu, srcu_idx); return freed; } #else /* CONFIG_MEMCG */ --=20 2.20.1 From nobody Mon Sep 8 18:52:44 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 015BAC64ED6 for ; Sun, 26 Feb 2023 14:54:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230472AbjBZOy1 (ORCPT ); Sun, 26 Feb 2023 09:54:27 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55210 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230222AbjBZOxo (ORCPT ); Sun, 26 Feb 2023 09:53:44 -0500 Received: from mail-pj1-x1036.google.com (mail-pj1-x1036.google.com [IPv6:2607:f8b0:4864:20::1036]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C2C235278 for ; Sun, 26 Feb 2023 06:49:51 -0800 (PST) Received: by mail-pj1-x1036.google.com with SMTP id y2so3594626pjg.3 for ; Sun, 26 Feb 2023 06:49:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=zCpviF+GWpW2q6LnTDjAWSvloYoZWztn7l/3EVg4xkQ=; b=CsO5aMnVYOQmnyjcQDWffm4ibQqMe8oamD5qA31APfwGf7gzHtJWrrI9olR/qCLH3m +Dg9XEOXL03hNqlNAICq5ya5pc+4BAUaxlKS89rulXGKJbAvILD5lY4IpjVkqVKRPLvh 40gqi3qq8AOZ1mGSmaZCdTm8hikTcJ9NRlhGcNGYHgwL8+XrC0lLnzZo9UmwdvnTP54e eUYL2ktptCc5CxNqWcNGg2KrPSG53OlDew6ZxZjtTpBl8B2IbscuG/h4GalyWHIQrMzw 3OBbkZBqaEHnKb1ujMBfyfX5Hxby7rTMu2I2Ehi1U62/G1skp5O5yL+5jSowBJMa0FAX IEvA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=zCpviF+GWpW2q6LnTDjAWSvloYoZWztn7l/3EVg4xkQ=; b=mrCCfqqu93S3EXxAvrWxSDkhl6NF85GhSsMMj9vhz+0FhMc37WrvMNUVfYN2NZ8+vw o1/K6GG+x+4199CnRuukfCOwvwyMMBkbgGUMfEQzfkUEJCwysq2MyyBufBJxmt509uOM BLQhY/IXOEjqktk10+YAdelrH+xj6ddtCpBqj7sbvMDfQgg3XmeBE5qSgFEmYm+sX1pi fILHmI3gArL9FIGTTDAW7VdnCdIxX6Oj2et0bNaJD9Q0ZKryFfn2tiQgPIeK2GUbjqy/ 5XwK+pr4aIDzFBpzrQE4gFCG/fV6o2y6PAUN0D+XzBOBLqkvCuWEjIJP9A48p2Q0Roag lotA== X-Gm-Message-State: AO0yUKXOPTaAiWl6OIJ5uZW2pqpvG4T2KIsjNgWS/KWiU54DvVvBIeOf vFBsGAdqbc7u62Ny89pfXSlSsg== X-Google-Smtp-Source: AK7set+c04U6J0wwMClaGVyTvpOvZFG7I93CD+bD7JQJKcXtMBNQuebr82PnrpOoKfjQ0YfQoyG7og== X-Received: by 2002:a17:902:ab0c:b0:19a:b1ac:45d4 with SMTP id ik12-20020a170902ab0c00b0019ab1ac45d4mr23569774plb.3.1677422913584; Sun, 26 Feb 2023 06:48:33 -0800 (PST) Received: from localhost.localdomain ([139.177.225.248]) by smtp.gmail.com with ESMTPSA id y20-20020a170902ed5400b0019c2cf12d15sm2755589plb.116.2023.02.26.06.48.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 26 Feb 2023 06:48:33 -0800 (PST) From: Qi Zheng To: akpm@linux-foundation.org, tkhai@ya.ru, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com Cc: sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v3 4/8] mm: vmscan: add shrinker_srcu_generation Date: Sun, 26 Feb 2023 22:46:51 +0800 Message-Id: <20230226144655.79778-5-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230226144655.79778-1-zhengqi.arch@bytedance.com> References: <20230226144655.79778-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Kirill Tkhai After we make slab shrink lockless with SRCU, the longest sleep unregister_shrinker() will be a sleep waiting for all do_shrink_slab() calls. To aviod long unbreakable action in the unregister_shrinker(), add shrinker_srcu_generation to restore a check similar to the rwsem_is_contendent() check that we had before. And for memcg slab shrink, we unlock SRCU and continue iterations from the next shrinker id. Signed-off-by: Kirill Tkhai Signed-off-by: Qi Zheng --- mm/vmscan.c | 24 ++++++++++++++++++++---- 1 file changed, 20 insertions(+), 4 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 490764f8e085..99e852c0ab9e 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -204,6 +204,7 @@ static void set_task_reclaim_state(struct task_struct *= task, LIST_HEAD(shrinker_list); DECLARE_RWSEM(shrinker_rwsem); DEFINE_SRCU(shrinker_srcu); +static atomic_t shrinker_srcu_generation =3D ATOMIC_INIT(0); =20 #ifdef CONFIG_MEMCG static int shrinker_nr_max; @@ -782,6 +783,7 @@ void unregister_shrinker(struct shrinker *shrinker) debugfs_entry =3D shrinker_debugfs_remove(shrinker); up_write(&shrinker_rwsem); =20 + atomic_inc(&shrinker_srcu_generation); synchronize_srcu(&shrinker_srcu); =20 debugfs_remove_recursive(debugfs_entry); @@ -803,6 +805,7 @@ void synchronize_shrinkers(void) { down_write(&shrinker_rwsem); up_write(&shrinker_rwsem); + atomic_inc(&shrinker_srcu_generation); synchronize_srcu(&shrinker_srcu); } EXPORT_SYMBOL(synchronize_shrinkers); @@ -912,18 +915,20 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask= , int nid, { struct shrinker_info *info; unsigned long ret, freed =3D 0; - int srcu_idx; - int i; + int srcu_idx, generation; + int i =3D 0; =20 if (!mem_cgroup_online(memcg)) return 0; =20 +again: srcu_idx =3D srcu_read_lock(&shrinker_srcu); info =3D shrinker_info_srcu(memcg, nid); if (unlikely(!info)) goto unlock; =20 - for_each_set_bit(i, info->map, info->map_nr_max) { + generation =3D atomic_read(&shrinker_srcu_generation); + for_each_set_bit_from(i, info->map, info->map_nr_max) { struct shrink_control sc =3D { .gfp_mask =3D gfp_mask, .nid =3D nid, @@ -969,6 +974,11 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask,= int nid, set_shrinker_bit(memcg, nid, i); } freed +=3D ret; + if (atomic_read(&shrinker_srcu_generation) !=3D generation) { + srcu_read_unlock(&shrinker_srcu, srcu_idx); + i++; + goto again; + } } unlock: srcu_read_unlock(&shrinker_srcu, srcu_idx); @@ -1008,7 +1018,7 @@ static unsigned long shrink_slab(gfp_t gfp_mask, int = nid, { unsigned long ret, freed =3D 0; struct shrinker *shrinker; - int srcu_idx; + int srcu_idx, generation; =20 /* * The root memcg might be allocated even though memcg is disabled @@ -1022,6 +1032,7 @@ static unsigned long shrink_slab(gfp_t gfp_mask, int = nid, =20 srcu_idx =3D srcu_read_lock(&shrinker_srcu); =20 + generation =3D atomic_read(&shrinker_srcu_generation); list_for_each_entry_srcu(shrinker, &shrinker_list, list, srcu_read_lock_held(&shrinker_srcu)) { struct shrink_control sc =3D { @@ -1034,6 +1045,11 @@ static unsigned long shrink_slab(gfp_t gfp_mask, int= nid, if (ret =3D=3D SHRINK_EMPTY) ret =3D 0; freed +=3D ret; + + if (atomic_read(&shrinker_srcu_generation) !=3D generation) { + freed =3D freed ? : 1; + break; + } } =20 srcu_read_unlock(&shrinker_srcu, srcu_idx); --=20 2.20.1 From nobody Mon Sep 8 18:52:44 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B4096C7EE23 for ; Sun, 26 Feb 2023 14:58:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231151AbjBZO62 (ORCPT ); Sun, 26 Feb 2023 09:58:28 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55644 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231555AbjBZOzp (ORCPT ); Sun, 26 Feb 2023 09:55:45 -0500 Received: from mail-pl1-f173.google.com (mail-pl1-f173.google.com [209.85.214.173]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 38AE114223 for ; Sun, 26 Feb 2023 06:51:05 -0800 (PST) Received: by mail-pl1-f173.google.com with SMTP id p20so2987759plw.13 for ; Sun, 26 Feb 2023 06:51:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=4ra+jS9qPdJDcLL3tCNVjDgmNt8q5h7NH0ipc1XhyKg=; b=PLf5B69Jh3pz5GXkfJ0kzyAzNE9QmfDh24Eym4+RIsIY+kh7/HFsEQ2eahiZrlAtR+ 4MxgpRyO+JR9bXbQmyfDQ7Xzvmlu8kCxHtggdFCMFTGHU/zrqq+iyrlQZ1x2SIvb3hxE GprjsegftQOiKVGJiPih2MUwmk/ZVPeSzgftTgUUilU4HagO6HlGsAjToabMn30Ye2jJ D1sW5UHlLVaMC0PQVaPxtdHsFbPDqUB7VsgFSalXcyA5wqm6EJXO+viNO6MMi5fGoOK4 u4SJ8zlHcmMYd0X0NrZ/p1mkOI0zz70UPWZaATCxnS4NbaHp2WXdx+JDlArqTdLRnVOn lPsA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=4ra+jS9qPdJDcLL3tCNVjDgmNt8q5h7NH0ipc1XhyKg=; b=FJApLAFXdKlFvXPpOomb1WAu7un9LMHeCfjsahFcsjzLGYM2Ou/Hca9QDkKQPkPD+7 SaqANBo09j1HGyU1BQeWlrvMuE65kFcklZLJD6KF0uw6t6Vy7iEy77uooypxyIZIb9OT vAkXoCd+h41iwolBToRK8kAuWLCF5HeBg2i/YKNNjGpThyRUT3Sq8kKrnzF8JP0lcvDC yMQ0nho4hCn7oe7tT2W38RFycCJRYKy7lRHzNxZheev9Ep+eZj+LxBipDUTqzmWXeHjp ZRAOGg22LI4lq4axzzK79bNA31Ei9/EcWPGb+eJYAK4tfz8KZH1HUspK9p8BwTkXYKeH fs7A== X-Gm-Message-State: AO0yUKWMb6pWVbtvSHKMaHWNSNLPKq7SUBw0CJ/ctNk9TRsBXJO+REFo zfi1wG7tkA3JPZoBALHPBUfN/g== X-Google-Smtp-Source: AK7set+f72fXg+fOOH45vAOIpM4RdqUXZxaFOCsUrACUIANqZsAcaZRsNmxNPLzIzBDfISXFBkJnQw== X-Received: by 2002:a17:903:2291:b0:19a:8202:2dad with SMTP id b17-20020a170903229100b0019a82022dadmr23250501plh.2.1677422919354; Sun, 26 Feb 2023 06:48:39 -0800 (PST) Received: from localhost.localdomain ([139.177.225.248]) by smtp.gmail.com with ESMTPSA id y20-20020a170902ed5400b0019c2cf12d15sm2755589plb.116.2023.02.26.06.48.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 26 Feb 2023 06:48:39 -0800 (PST) From: Qi Zheng To: akpm@linux-foundation.org, tkhai@ya.ru, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com Cc: sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v3 5/8] mm: shrinkers: make count and scan in shrinker debugfs lockless Date: Sun, 26 Feb 2023 22:46:52 +0800 Message-Id: <20230226144655.79778-6-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230226144655.79778-1-zhengqi.arch@bytedance.com> References: <20230226144655.79778-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Like global and memcg slab shrink, also use SRCU to make count and scan operations in memory shrinker debugfs lockless. Signed-off-by: Qi Zheng --- mm/shrinker_debug.c | 24 +++++++----------------- 1 file changed, 7 insertions(+), 17 deletions(-) diff --git a/mm/shrinker_debug.c b/mm/shrinker_debug.c index 39c3491e28a3..6aa7a7ec69da 100644 --- a/mm/shrinker_debug.c +++ b/mm/shrinker_debug.c @@ -9,6 +9,7 @@ /* defined in vmscan.c */ extern struct rw_semaphore shrinker_rwsem; extern struct list_head shrinker_list; +extern struct srcu_struct shrinker_srcu; =20 static DEFINE_IDA(shrinker_debugfs_ida); static struct dentry *shrinker_debugfs_root; @@ -49,18 +50,13 @@ static int shrinker_debugfs_count_show(struct seq_file = *m, void *v) struct mem_cgroup *memcg; unsigned long total; bool memcg_aware; - int ret, nid; + int ret =3D 0, nid, srcu_idx; =20 count_per_node =3D kcalloc(nr_node_ids, sizeof(unsigned long), GFP_KERNEL= ); if (!count_per_node) return -ENOMEM; =20 - ret =3D down_read_killable(&shrinker_rwsem); - if (ret) { - kfree(count_per_node); - return ret; - } - rcu_read_lock(); + srcu_idx =3D srcu_read_lock(&shrinker_srcu); =20 memcg_aware =3D shrinker->flags & SHRINKER_MEMCG_AWARE; =20 @@ -91,8 +87,7 @@ static int shrinker_debugfs_count_show(struct seq_file *m= , void *v) } } while ((memcg =3D mem_cgroup_iter(NULL, memcg, NULL)) !=3D NULL); =20 - rcu_read_unlock(); - up_read(&shrinker_rwsem); + srcu_read_unlock(&shrinker_srcu, srcu_idx); =20 kfree(count_per_node); return ret; @@ -115,9 +110,8 @@ static ssize_t shrinker_debugfs_scan_write(struct file = *file, .gfp_mask =3D GFP_KERNEL, }; struct mem_cgroup *memcg =3D NULL; - int nid; + int nid, srcu_idx; char kbuf[72]; - ssize_t ret; =20 read_len =3D size < (sizeof(kbuf) - 1) ? size : (sizeof(kbuf) - 1); if (copy_from_user(kbuf, buf, read_len)) @@ -146,11 +140,7 @@ static ssize_t shrinker_debugfs_scan_write(struct file= *file, return -EINVAL; } =20 - ret =3D down_read_killable(&shrinker_rwsem); - if (ret) { - mem_cgroup_put(memcg); - return ret; - } + srcu_idx =3D srcu_read_lock(&shrinker_srcu); =20 sc.nid =3D nid; sc.memcg =3D memcg; @@ -159,7 +149,7 @@ static ssize_t shrinker_debugfs_scan_write(struct file = *file, =20 shrinker->scan_objects(shrinker, &sc); =20 - up_read(&shrinker_rwsem); + srcu_read_unlock(&shrinker_srcu, srcu_idx); mem_cgroup_put(memcg); =20 return size; --=20 2.20.1 From nobody Mon Sep 8 18:52:44 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2E239C7EE31 for ; Sun, 26 Feb 2023 14:56:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230511AbjBZO4r (ORCPT ); Sun, 26 Feb 2023 09:56:47 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54712 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230151AbjBZOyE (ORCPT ); Sun, 26 Feb 2023 09:54:04 -0500 Received: from mail-pl1-x62f.google.com (mail-pl1-x62f.google.com [IPv6:2607:f8b0:4864:20::62f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 292599763 for ; Sun, 26 Feb 2023 06:50:04 -0800 (PST) Received: by mail-pl1-x62f.google.com with SMTP id n6so2822311plf.5 for ; Sun, 26 Feb 2023 06:50:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1677422925; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=c6cuoJEo92PScDqgnKjc0MgPFLiMUUPUvBDJj7SgxHo=; b=lIKiq8ErtDRmfz+i/ano83moR805w7pQEVFBs5DWz8iogOWtMgCKlWMtTjjG4CUYGf qZUEa/ayLAYcSmEePRMBNnD4UMS+HxFoUzmlroe0ZVHuqZ3z2bv2HGbh7Yq0wKCvHk4k QE8yyj88IWGiNyUTVZhSPdvEoYldgr4C6pMLd3y42cOtg65TmytTMhFeDnjLzWUSuqD4 9aNiX3gjRzbrv4MYxQIZGaPfntfaShxVFiNdV3/73U+gNlaqjtHmoW98ZiAC2kGLB21U 7G4hnPw+a24oAWIJ7kfT7KLIQYmuet0PscCXlnmSZhQ2rIyHh9ik/cyvUbqhLd+kEnv8 Uwjg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1677422925; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=c6cuoJEo92PScDqgnKjc0MgPFLiMUUPUvBDJj7SgxHo=; b=NnSkB4euJawKue+gYwn7w1aVgBcrOOIx/1LFZJAunt6QHlVTAT0NtzZk0MFa7TeSS1 3rpDbrWfSSpMLjz9LrxAoST4Yi7jUqBdYKJFuRIc0evIMxX1yfcxSun60ChsPNlH+2bH HYWyfCgbHTPZDaoO4HEK3n2sh8vX4usCIUyoB2YeJG/anMyAHo85h1ITLR+Q9tLrGyBv Ta79JOyTic+v61QNC3lpILuELVYQYQmEJ2xuSpJvJHSTD96O2BTs8KI11sTGxT9kJbxz mvUFAl+GefHyWwDWNUu1k+5yzrYqbnNkApy4c8+IRMWZhtz+Civ+d1QH6GiKG8qUufpt psNw== X-Gm-Message-State: AO0yUKV3yP3Sxn1qxSxtx/cm6t09wfVUG3NfhZTUypGer4ylaePsl1KX Q95oTmmGkKbVeLt9FQhH+m7XwQ== X-Google-Smtp-Source: AK7set+XvuciT55+1IMf9hAHsturuZDzsZyl61PmuY/3HoeWXaqHrVSGXMTK34S3WOCoi78lnQD8RA== X-Received: by 2002:a17:902:bb0f:b0:197:8e8e:f15 with SMTP id im15-20020a170902bb0f00b001978e8e0f15mr23385229plb.6.1677422925086; Sun, 26 Feb 2023 06:48:45 -0800 (PST) Received: from localhost.localdomain ([139.177.225.248]) by smtp.gmail.com with ESMTPSA id y20-20020a170902ed5400b0019c2cf12d15sm2755589plb.116.2023.02.26.06.48.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 26 Feb 2023 06:48:44 -0800 (PST) From: Qi Zheng To: akpm@linux-foundation.org, tkhai@ya.ru, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com Cc: sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v3 6/8] mm: vmscan: hold write lock to reparent shrinker nr_deferred Date: Sun, 26 Feb 2023 22:46:53 +0800 Message-Id: <20230226144655.79778-7-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230226144655.79778-1-zhengqi.arch@bytedance.com> References: <20230226144655.79778-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" For now, reparent_shrinker_deferred() is the only holder of read lock of shrinker_rwsem. And it already holds the global cgroup_mutex, so it will not be called in parallel. Therefore, in order to convert shrinker_rwsem to shrinker_mutex later, here we change to hold the write lock of shrinker_rwsem to reparent. Signed-off-by: Qi Zheng --- mm/vmscan.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 99e852c0ab9e..16ff64813175 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -451,7 +451,7 @@ void reparent_shrinker_deferred(struct mem_cgroup *memc= g) parent =3D root_mem_cgroup; =20 /* Prevent from concurrent shrinker_info expand */ - down_read(&shrinker_rwsem); + down_write(&shrinker_rwsem); for_each_node(nid) { child_info =3D shrinker_info_protected(memcg, nid); parent_info =3D shrinker_info_protected(parent, nid); @@ -460,7 +460,7 @@ void reparent_shrinker_deferred(struct mem_cgroup *memc= g) atomic_long_add(nr, &parent_info->nr_deferred[i]); } } - up_read(&shrinker_rwsem); + up_write(&shrinker_rwsem); } =20 static bool cgroup_reclaim(struct scan_control *sc) --=20 2.20.1 From nobody Mon Sep 8 18:52:44 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8E1DAC64ED6 for ; Sun, 26 Feb 2023 14:54:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230480AbjBZOye (ORCPT ); Sun, 26 Feb 2023 09:54:34 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56440 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230323AbjBZOxt (ORCPT ); Sun, 26 Feb 2023 09:53:49 -0500 Received: from mail-pl1-x62a.google.com (mail-pl1-x62a.google.com [IPv6:2607:f8b0:4864:20::62a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E58AA19BA for ; Sun, 26 Feb 2023 06:50:05 -0800 (PST) Received: by mail-pl1-x62a.google.com with SMTP id i10so4190450plr.9 for ; Sun, 26 Feb 2023 06:50:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=rPbK2mA8SftO30bweBpSZUhXCxYptD2QUCquMlUkzZQ=; b=RLNAX82ej6WiOp+gJxxrYUevqQBzdDGYpvRp2WltPYHzsFagaq47ozL06y+Hy6OgmK JK9iGaHN7dJvbFey7hQXB7aig8jRb8Fzh2ihtgS4KPi3JJgVeer/QP403MbnYAJfXhak bYN0OJJ/krh591ZGhC6ZmW4Bo9d/0ELhp3VjgtJ7eyFa0hScQMuwE81nRfrktkjFE6Os OjbYs65uBBhgwi4YlDp3YuXDzBBPVqJDlkDh/2welq1ly/HLAfshGinj1f1s9zkkezz/ 2xhknARP+taXDeqDm4xAjtZ8aw0C7dGDxIicSpRhb2gQC9so4gEZ84pp+H7E0riZtKLc Tpsg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=rPbK2mA8SftO30bweBpSZUhXCxYptD2QUCquMlUkzZQ=; b=lTqDzm1efkpTNt4k/tKU05+Of29BHCQCgBFmQ5Wdvz7aV0BfbDeD/EPr2i5xRmx9V/ Siqc42IS5V1pdzJuae3TQ0MaDeJDZatYBCRrWIo0aEtBZcuL7XZtgwyvgMKhiiPiWVxc IGljWhfcBHQNsJ638ojfT8d0qEbdx+134jJD0fFPsXXlX2gTLA/iOWnVVNrJi2jhJEfg xMKB4OIrWNfx70QcCbj2CuX3DbE7tgyzeE9A/DFBTcLX8yuEM0XIny0EX6v41jL2LT4a 9ZKzD2SvIRdO+2KfxsT4Xe04L8fywOhggSkzM/aUYxxShLEhUzaQqWWx3qQZzJGtoSU2 LeRg== X-Gm-Message-State: AO0yUKUtq9TLwP/GkJDiNfkS87JRAzn9T49lmsXQLdDGRDzLDq0K/5Zf CUvbSW7UAJB7mBU64B9UXMHY+Q== X-Google-Smtp-Source: AK7set8jSrtu5jvJSft27Jtxn5UB2T4NTHUh1NNGGVH4kZw+rpTeROnHO4aYE0DylXNd7RBZ0PVGQw== X-Received: by 2002:a17:902:d4c8:b0:19a:7217:32af with SMTP id o8-20020a170902d4c800b0019a721732afmr23249925plg.5.1677422930708; Sun, 26 Feb 2023 06:48:50 -0800 (PST) Received: from localhost.localdomain ([139.177.225.248]) by smtp.gmail.com with ESMTPSA id y20-20020a170902ed5400b0019c2cf12d15sm2755589plb.116.2023.02.26.06.48.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 26 Feb 2023 06:48:50 -0800 (PST) From: Qi Zheng To: akpm@linux-foundation.org, tkhai@ya.ru, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com Cc: sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v3 7/8] mm: vmscan: remove shrinker_rwsem from synchronize_shrinkers() Date: Sun, 26 Feb 2023 22:46:54 +0800 Message-Id: <20230226144655.79778-8-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230226144655.79778-1-zhengqi.arch@bytedance.com> References: <20230226144655.79778-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Now there are no readers of shrinker_rwsem, so synchronize_shrinkers() does not need to hold the writer of shrinker_rwsem to wait for all running shinkers to complete, synchronize_srcu() is enough. Signed-off-by: Qi Zheng --- mm/vmscan.c | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 16ff64813175..2d71fd565c78 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -796,15 +796,11 @@ EXPORT_SYMBOL(unregister_shrinker); /** * synchronize_shrinkers - Wait for all running shrinkers to complete. * - * This is equivalent to calling unregister_shrink() and register_shrinker= (), - * but atomically and with less overhead. This is useful to guarantee that= all - * shrinker invocations have seen an update, before freeing memory, simila= r to - * rcu. + * This is useful to guarantee that all shrinker invocations have seen an + * update, before freeing memory. */ void synchronize_shrinkers(void) { - down_write(&shrinker_rwsem); - up_write(&shrinker_rwsem); atomic_inc(&shrinker_srcu_generation); synchronize_srcu(&shrinker_srcu); } --=20 2.20.1 From nobody Mon Sep 8 18:52:44 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8DA35C6FA8E for ; Sun, 26 Feb 2023 14:57:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231205AbjBZO5M (ORCPT ); Sun, 26 Feb 2023 09:57:12 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55222 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230321AbjBZOyT (ORCPT ); Sun, 26 Feb 2023 09:54:19 -0500 Received: from mail-pj1-x1036.google.com (mail-pj1-x1036.google.com [IPv6:2607:f8b0:4864:20::1036]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D61CF1ADFD for ; Sun, 26 Feb 2023 06:50:14 -0800 (PST) Received: by mail-pj1-x1036.google.com with SMTP id k21-20020a17090aaa1500b002376652e160so3811503pjq.0 for ; Sun, 26 Feb 2023 06:50:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=f5xm65H9BLTSDcpU/nYsAyAok8lJrBUsJgeNL2FDrMI=; b=DU9ImCfzPDt6r5qsdIJgFlMqPyv1XNzrFdxX1MYevySp7rV/vBy4oHtHlXM0ehu+/q e7Sjxf8qsqyQGdUSKu80ZuyH8gv9rpDyqCA+6sYlagLU/RVK4FI8pl7xZiXV+Ojb41zx ezPpwq3uJvNV5TVTXIF7WKeiEmB772qpgZU/rJEebve9b47zOunZTUMe/f+gk7PISSjc Ehcf7cS8nj1sKVkD/JveUqUJzsUCxQfCPh/rzpzR2amvQmZsujUa/Epz4Eonu2OG9utb VbC/USw9zVaaeY/TFpc3MuKTAHI4ssQxadvh9v+878R2uidsPviikdk/mxseIFk0IHSC +N3A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=f5xm65H9BLTSDcpU/nYsAyAok8lJrBUsJgeNL2FDrMI=; b=ymEbfvJJflIhHcxxQE6BoqxhGeoqCLhJXYgVuxlV97/IVd1UihoKpx/CWKHzOIzRCr JvpzA2OjjJVrVtxK9YU+bpd/nic4OYuP/0GzSUKDxfaIJWnkecyzsf39L+TjifeNfatH LjPAJhpvO1eQYKTyZ+tAc0O0gshce6cSOcwdNO2/dQlqhsL8qRCuSUJiJ7RVq+X2zxHG Mfue5gjC4q7eS6EdhUHZIpRw4dyJdc3Ipy8w92qqVqYBKC6f/sqAnFbZONUmg8ukf5XO uOUbBLruTtMl01XueY4e+dYrxWwoRhf/OX6zBX4cr/X54DXXnNNo+whM3kz0pCCcinMo XoBg== X-Gm-Message-State: AO0yUKVy+ht/6CI68ZBL7QtyC1hYcKXWhRbxLDk3PHmHEMi2jRQtyjXh EETxISgFJfYxYZw1sMYhRZYPdg== X-Google-Smtp-Source: AK7set9OLUlQVvloHLhW8LBiRt1OW+15leJOMZQHQ9/zIbOmm9Z4KelXtK4HEXm+zbHSQlGGRoJ3MA== X-Received: by 2002:a17:902:d4c8:b0:19a:7217:32af with SMTP id o8-20020a170902d4c800b0019a721732afmr23250125plg.5.1677422936561; Sun, 26 Feb 2023 06:48:56 -0800 (PST) Received: from localhost.localdomain ([139.177.225.248]) by smtp.gmail.com with ESMTPSA id y20-20020a170902ed5400b0019c2cf12d15sm2755589plb.116.2023.02.26.06.48.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 26 Feb 2023 06:48:56 -0800 (PST) From: Qi Zheng To: akpm@linux-foundation.org, tkhai@ya.ru, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com Cc: sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v3 8/8] mm: shrinkers: convert shrinker_rwsem to mutex Date: Sun, 26 Feb 2023 22:46:55 +0800 Message-Id: <20230226144655.79778-9-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230226144655.79778-1-zhengqi.arch@bytedance.com> References: <20230226144655.79778-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Now there are no readers of shrinker_rwsem, so we can simply replace it with mutex lock. Signed-off-by: Qi Zheng --- drivers/md/dm-cache-metadata.c | 2 +- drivers/md/dm-thin-metadata.c | 2 +- fs/super.c | 2 +- mm/shrinker_debug.c | 14 +++++++------- mm/vmscan.c | 34 +++++++++++++++++----------------- 5 files changed, 27 insertions(+), 27 deletions(-) diff --git a/drivers/md/dm-cache-metadata.c b/drivers/md/dm-cache-metadata.c index acffed750e3e..9e0c69958587 100644 --- a/drivers/md/dm-cache-metadata.c +++ b/drivers/md/dm-cache-metadata.c @@ -1828,7 +1828,7 @@ int dm_cache_metadata_abort(struct dm_cache_metadata = *cmd) * Replacement block manager (new_bm) is created and old_bm destroyed out= side of * cmd root_lock to avoid ABBA deadlock that would result (due to life-cy= cle of * shrinker associated with the block manager's bufio client vs cmd root_= lock). - * - must take shrinker_rwsem without holding cmd->root_lock + * - must take shrinker_mutex without holding cmd->root_lock */ new_bm =3D dm_block_manager_create(cmd->bdev, DM_CACHE_METADATA_BLOCK_SIZ= E << SECTOR_SHIFT, CACHE_MAX_CONCURRENT_LOCKS); diff --git a/drivers/md/dm-thin-metadata.c b/drivers/md/dm-thin-metadata.c index fd464fb024c3..9f5cb52c5763 100644 --- a/drivers/md/dm-thin-metadata.c +++ b/drivers/md/dm-thin-metadata.c @@ -1887,7 +1887,7 @@ int dm_pool_abort_metadata(struct dm_pool_metadata *p= md) * Replacement block manager (new_bm) is created and old_bm destroyed out= side of * pmd root_lock to avoid ABBA deadlock that would result (due to life-cy= cle of * shrinker associated with the block manager's bufio client vs pmd root_= lock). - * - must take shrinker_rwsem without holding pmd->root_lock + * - must take shrinker_mutex without holding pmd->root_lock */ new_bm =3D dm_block_manager_create(pmd->bdev, THIN_METADATA_BLOCK_SIZE <<= SECTOR_SHIFT, THIN_MAX_CONCURRENT_LOCKS); diff --git a/fs/super.c b/fs/super.c index 84332d5cb817..91a4037b1d95 100644 --- a/fs/super.c +++ b/fs/super.c @@ -54,7 +54,7 @@ static char *sb_writers_name[SB_FREEZE_LEVELS] =3D { * One thing we have to be careful of with a per-sb shrinker is that we do= n't * drop the last active reference to the superblock from within the shrink= er. * If that happens we could trigger unregistering the shrinker from within= the - * shrinker path and that leads to deadlock on the shrinker_rwsem. Hence we + * shrinker path and that leads to deadlock on the shrinker_mutex. Hence we * take a passive reference to the superblock to avoid this from occurring. */ static unsigned long super_cache_scan(struct shrinker *shrink, diff --git a/mm/shrinker_debug.c b/mm/shrinker_debug.c index 6aa7a7ec69da..b0f6aff372df 100644 --- a/mm/shrinker_debug.c +++ b/mm/shrinker_debug.c @@ -7,7 +7,7 @@ #include =20 /* defined in vmscan.c */ -extern struct rw_semaphore shrinker_rwsem; +extern struct mutex shrinker_mutex; extern struct list_head shrinker_list; extern struct srcu_struct shrinker_srcu; =20 @@ -167,7 +167,7 @@ int shrinker_debugfs_add(struct shrinker *shrinker) char buf[128]; int id; =20 - lockdep_assert_held(&shrinker_rwsem); + lockdep_assert_held(&shrinker_mutex); =20 /* debugfs isn't initialized yet, add debugfs entries later. */ if (!shrinker_debugfs_root) @@ -210,7 +210,7 @@ int shrinker_debugfs_rename(struct shrinker *shrinker, = const char *fmt, ...) if (!new) return -ENOMEM; =20 - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); =20 old =3D shrinker->name; shrinker->name =3D new; @@ -228,7 +228,7 @@ int shrinker_debugfs_rename(struct shrinker *shrinker, = const char *fmt, ...) shrinker->debugfs_entry =3D entry; } =20 - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); =20 kfree_const(old); =20 @@ -240,7 +240,7 @@ struct dentry *shrinker_debugfs_remove(struct shrinker = *shrinker) { struct dentry *entry =3D shrinker->debugfs_entry; =20 - lockdep_assert_held(&shrinker_rwsem); + lockdep_assert_held(&shrinker_mutex); =20 kfree_const(shrinker->name); shrinker->name =3D NULL; @@ -265,14 +265,14 @@ static int __init shrinker_debugfs_init(void) shrinker_debugfs_root =3D dentry; =20 /* Create debugfs entries for shrinkers registered at boot */ - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); list_for_each_entry(shrinker, &shrinker_list, list) if (!shrinker->debugfs_entry) { ret =3D shrinker_debugfs_add(shrinker); if (ret) break; } - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); =20 return ret; } diff --git a/mm/vmscan.c b/mm/vmscan.c index 2d71fd565c78..6c5d21ba0c9a 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -35,7 +35,7 @@ #include #include #include -#include +#include #include #include #include @@ -202,7 +202,7 @@ static void set_task_reclaim_state(struct task_struct *= task, } =20 LIST_HEAD(shrinker_list); -DECLARE_RWSEM(shrinker_rwsem); +DEFINE_MUTEX(shrinker_mutex); DEFINE_SRCU(shrinker_srcu); static atomic_t shrinker_srcu_generation =3D ATOMIC_INIT(0); =20 @@ -225,7 +225,7 @@ static struct shrinker_info *shrinker_info_protected(st= ruct mem_cgroup *memcg, { return srcu_dereference_check(memcg->nodeinfo[nid]->shrinker_info, &shrinker_srcu, - lockdep_is_held(&shrinker_rwsem)); + lockdep_is_held(&shrinker_mutex)); } =20 static struct shrinker_info *shrinker_info_srcu(struct mem_cgroup *memcg, @@ -310,7 +310,7 @@ int alloc_shrinker_info(struct mem_cgroup *memcg) int nid, size, ret =3D 0; int map_size, defer_size =3D 0; =20 - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); map_size =3D shrinker_map_size(shrinker_nr_max); defer_size =3D shrinker_defer_size(shrinker_nr_max); size =3D map_size + defer_size; @@ -326,7 +326,7 @@ int alloc_shrinker_info(struct mem_cgroup *memcg) info->map_nr_max =3D shrinker_nr_max; rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_info, info); } - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); =20 return ret; } @@ -342,7 +342,7 @@ static int expand_shrinker_info(int new_id) if (!root_mem_cgroup) goto out; =20 - lockdep_assert_held(&shrinker_rwsem); + lockdep_assert_held(&shrinker_mutex); =20 map_size =3D shrinker_map_size(new_nr_max); defer_size =3D shrinker_defer_size(new_nr_max); @@ -392,7 +392,7 @@ static int prealloc_memcg_shrinker(struct shrinker *shr= inker) if (mem_cgroup_disabled()) return -ENOSYS; =20 - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); id =3D idr_alloc(&shrinker_idr, shrinker, 0, 0, GFP_KERNEL); if (id < 0) goto unlock; @@ -406,7 +406,7 @@ static int prealloc_memcg_shrinker(struct shrinker *shr= inker) shrinker->id =3D id; ret =3D 0; unlock: - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); return ret; } =20 @@ -416,7 +416,7 @@ static void unregister_memcg_shrinker(struct shrinker *= shrinker) =20 BUG_ON(id < 0); =20 - lockdep_assert_held(&shrinker_rwsem); + lockdep_assert_held(&shrinker_mutex); =20 idr_remove(&shrinker_idr, id); } @@ -451,7 +451,7 @@ void reparent_shrinker_deferred(struct mem_cgroup *memc= g) parent =3D root_mem_cgroup; =20 /* Prevent from concurrent shrinker_info expand */ - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); for_each_node(nid) { child_info =3D shrinker_info_protected(memcg, nid); parent_info =3D shrinker_info_protected(parent, nid); @@ -460,7 +460,7 @@ void reparent_shrinker_deferred(struct mem_cgroup *memc= g) atomic_long_add(nr, &parent_info->nr_deferred[i]); } } - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); } =20 static bool cgroup_reclaim(struct scan_control *sc) @@ -709,9 +709,9 @@ void free_prealloced_shrinker(struct shrinker *shrinker) shrinker->name =3D NULL; #endif if (shrinker->flags & SHRINKER_MEMCG_AWARE) { - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); unregister_memcg_shrinker(shrinker); - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); return; } =20 @@ -721,11 +721,11 @@ void free_prealloced_shrinker(struct shrinker *shrink= er) =20 void register_shrinker_prepared(struct shrinker *shrinker) { - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); list_add_tail_rcu(&shrinker->list, &shrinker_list); shrinker->flags |=3D SHRINKER_REGISTERED; shrinker_debugfs_add(shrinker); - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); } =20 static int __register_shrinker(struct shrinker *shrinker) @@ -775,13 +775,13 @@ void unregister_shrinker(struct shrinker *shrinker) if (!(shrinker->flags & SHRINKER_REGISTERED)) return; =20 - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); list_del_rcu(&shrinker->list); shrinker->flags &=3D ~SHRINKER_REGISTERED; if (shrinker->flags & SHRINKER_MEMCG_AWARE) unregister_memcg_shrinker(shrinker); debugfs_entry =3D shrinker_debugfs_remove(shrinker); - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); =20 atomic_inc(&shrinker_srcu_generation); synchronize_srcu(&shrinker_srcu); --=20 2.20.1