From nobody Fri Sep 25 10:38:24 2026 Received: from mailgw1.hygon.cn (unknown [101.204.27.37]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 5EE1137C933; Mon, 14 Sep 2026 06:15:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=101.204.27.37 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789366553; cv=none; b=HJds6dTSqDMYeteZmcQtiFl5Lon/PWxALFfjJkwsPY+RK766v/ehRHBDOjeK5+uBwVlG5/Duso532j4/b+kD/J6v/nsH93ir1RSvKYTkG70R8LauTeq1YrtEtJ15XK8yDgP56yR2rO7kmKXu1GpEFLZDm0OKcbeWuFUEWE+OArE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789366553; c=relaxed/simple; bh=P7rXGNebSpfRTNesgQtTHCpMRKRDDGiHlz8N5zawpgw=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=ZVhM++PR4eZXMmC966bzSqG/80CFYNjmQ+tS6EZs8DNDroz1gdvjhiBivdPhztuzysbCwdtmSUG3nQQjIAIIMpT6BThEJIPuxDmNBGaRGKTOzcpH4wwCmkbebfpadFK78SzMhMYsKxQ56yJiyoy2075hF/wvwGNRI1eooaLkZyk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=hygon.cn; spf=pass smtp.mailfrom=hygon.cn; arc=none smtp.client-ip=101.204.27.37 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=hygon.cn Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=hygon.cn Received: from maildlp2.hygon.cn (unknown [127.0.0.1]) by mailgw1.hygon.cn (Postfix) with ESMTP id 4hjvyr6rcmz2CZMW; Mon, 14 Sep 2026 14:15:32 +0800 (CST) Received: from maildlp2.hygon.cn (unknown [172.23.18.61]) by mailgw1.hygon.cn (Postfix) with ESMTP id 4hjvyp5s4Fz2CZMW; Mon, 14 Sep 2026 14:15:30 +0800 (CST) Received: from cncheex04.Hygon.cn (unknown [172.23.18.114]) by maildlp2.hygon.cn (Postfix) with ESMTPS id D27B0310BECC; Mon, 14 Sep 2026 14:11:16 +0800 (CST) Received: from hsj-2U-Workstation.hygon.cn (172.19.24.226) by cncheex04.Hygon.cn (172.23.18.114) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.36; Mon, 14 Sep 2026 14:15:29 +0800 From: Huang Shijie To: , CC: , , , , , , , , , , , , , , , , , , , , , , , , Huang Shijie Subject: [RFC PATCH 1/3] fs/drop_caches: do not scan procfs in drop_pagecache_sb() Date: Mon, 14 Sep 2026 14:14:47 +0800 Message-ID: <20260914061449.4024632-2-huangsj@hygon.cn> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914061449.4024632-1-huangsj@hygon.cn> References: <20260914061449.4024632-1-huangsj@hygon.cn> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: cncheex06.Hygon.cn (172.23.18.116) To cncheex04.Hygon.cn (172.23.18.114) Content-Type: text/plain; charset="utf-8" There is no page cache for the procfs, so do not scan the inodes in procfs. Signed-off-by: Huang Shijie --- fs/drop_caches.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/fs/drop_caches.c b/fs/drop_caches.c index 49f56a598ecb..1959090089af 100644 --- a/fs/drop_caches.c +++ b/fs/drop_caches.c @@ -5,6 +5,7 @@ =20 #include #include +#include #include #include #include @@ -20,6 +21,9 @@ static void drop_pagecache_sb(struct super_block *sb, voi= d *unused) { struct inode *inode, *toput_inode =3D NULL; =20 + if (sb->s_magic =3D=3D PROC_SUPER_MAGIC) + return; + spin_lock(&sb->s_inode_list_lock); list_for_each_entry(inode, &sb->s_inodes, i_sb_list) { spin_lock(&inode->i_lock); --=20 2.53.0 From nobody Fri Sep 25 10:38:24 2026 Received: from mailgw1.hygon.cn (unknown [101.204.27.37]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 6C4873AC0EF; Mon, 14 Sep 2026 06:15:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=101.204.27.37 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789366555; cv=none; b=BKXQ6mpxj9OeoGba5lUcibMcV5+UMzXjKqjBHRKM8iKfSNVlQill95rYqjnuIk1Wf4lZq5Cy/Q3V2KxVDCEqg0e98APXY5Dm0O5UyilD3Q6pSd91cwvdwxgmvEji5n1cHQBTUkg9WNyR7P5Z5IcATXXrwrBx0dg3EgBqEQJLYyU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789366555; c=relaxed/simple; bh=7aBG096Seog8JCOWQO3Ag7h9GZlL4pvzcDvzlFs60Rc=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=F7fnyVmaYva/8l7PPlZYY17lTCCSeC2MxQ9EbcWFeciYEwacE13suTHMg0+OGiMyMrQdoWJqbClFerlMKgJSTwlggAea+lMd7rKuJ1jCthyKkRvpDunuHblsWRPPIJB/LsnV2s3QDEkr2TbVFVBAA/VuBZQ/OC/5aCf5ElHIX3U= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=hygon.cn; spf=pass smtp.mailfrom=hygon.cn; arc=none smtp.client-ip=101.204.27.37 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=hygon.cn Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=hygon.cn Received: from maildlp1.hygon.cn (unknown [127.0.0.1]) by mailgw1.hygon.cn (Postfix) with ESMTP id 4hjvyv6rkkz2CZqx; Mon, 14 Sep 2026 14:15:35 +0800 (CST) Received: from maildlp1.hygon.cn (unknown [172.23.18.60]) by mailgw1.hygon.cn (Postfix) with ESMTP id 4hjvys0SLkz2CZMr; Mon, 14 Sep 2026 14:15:33 +0800 (CST) Received: from cncheex04.Hygon.cn (unknown [172.23.18.114]) by maildlp1.hygon.cn (Postfix) with ESMTPS id 697E34B59; Mon, 14 Sep 2026 14:15:30 +0800 (CST) Received: from hsj-2U-Workstation.hygon.cn (172.19.24.226) by cncheex04.Hygon.cn (172.23.18.114) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.36; Mon, 14 Sep 2026 14:15:31 +0800 From: Huang Shijie To: , CC: , , , , , , , , , , , , , , , , , , , , , , , , Huang Shijie Subject: [RFC PATCH 2/3] fs/super: introduce a helper sb_inodes_empty() Date: Mon, 14 Sep 2026 14:14:48 +0800 Message-ID: <20260914061449.4024632-3-huangsj@hygon.cn> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914061449.4024632-1-huangsj@hygon.cn> References: <20260914061449.4024632-1-huangsj@hygon.cn> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: cncheex06.Hygon.cn (172.23.18.116) To cncheex04.Hygon.cn (172.23.18.114) Content-Type: text/plain; charset="utf-8" Introduce a helper sb_inodes_empty() which is used to detect if the inodes list is empty. Signed-off-by: Huang Shijie --- fs/super.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/fs/super.c b/fs/super.c index 05e443173038..4b1314545522 100644 --- a/fs/super.c +++ b/fs/super.c @@ -724,6 +724,11 @@ void retire_super(struct super_block *sb) } EXPORT_SYMBOL(retire_super); =20 +static bool sb_inodes_empty(struct super_block *sb) +{ + return list_empty(&sb->s_inodes); +} + /** * generic_shutdown_super - common helper for ->kill_sb() * @sb: superblock to kill @@ -774,7 +779,7 @@ void generic_shutdown_super(struct super_block *sb) */ fscrypt_destroy_keyring(sb); =20 - if (CHECK_DATA_CORRUPTION(!list_empty(&sb->s_inodes), NULL, + if (CHECK_DATA_CORRUPTION(!sb_inodes_empty(sb), NULL, "VFS: Busy inodes after unmount of %s (%s)", sb->s_id, sb->s_type->name)) { /* --=20 2.53.0 From nobody Fri Sep 25 10:38:24 2026 Received: from mailgw1.hygon.cn (unknown [101.204.27.37]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 6C51F3B777C; Mon, 14 Sep 2026 06:15:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=101.204.27.37 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789366555; cv=none; b=ddwZVD7wq2gWrTWPeSjrPZa3RtwrKxbgBX61UUQ/4DLPoIL7Zy/Pf5VagQkySYrPO4ZieRGTyzdnB09Ow0pdm3IwpKprxFIqdnahQW41XT3IG417Zf8uWr/0WYo/av+lywmlTQm1QX2vyQcajJc2CW3OVaQj4dfo2V8OlYbSaVM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789366555; c=relaxed/simple; bh=C63JGwt3r2MzGq97jEKhLYlX+zaKnfqtHMCYCSMOg64=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=i1WHxBCkhG2q1R8twv2v100Pxj12ZuW4CPzAeY7IwqWLqVSw5uaxbakY1KbhLlYj2AFqMB7XTL6Ly4ElkxGVlX+L5bdtm1ZEWd1dnNG3lthc7tsCqqumyFc1QUeTIb4fS5jUK4458ovynOAHd8hQVEO/4oAatKYiKuMbgMj069M= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=hygon.cn; spf=pass smtp.mailfrom=hygon.cn; arc=none smtp.client-ip=101.204.27.37 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=hygon.cn Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=hygon.cn Received: from maildlp2.hygon.cn (unknown [127.0.0.1]) by mailgw1.hygon.cn (Postfix) with ESMTP id 4hjvyv6rgLz2CZLy; Mon, 14 Sep 2026 14:15:35 +0800 (CST) Received: from maildlp2.hygon.cn (unknown [172.23.18.61]) by mailgw1.hygon.cn (Postfix) with ESMTP id 4hjvyv6W5Tz2CZLy; Mon, 14 Sep 2026 14:15:35 +0800 (CST) Received: from cncheex04.Hygon.cn (unknown [172.23.18.114]) by maildlp2.hygon.cn (Postfix) with ESMTPS id E9145310BECC; Mon, 14 Sep 2026 14:11:21 +0800 (CST) Received: from hsj-2U-Workstation.hygon.cn (172.19.24.226) by cncheex04.Hygon.cn (172.23.18.114) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.36; Mon, 14 Sep 2026 14:15:34 +0800 From: Huang Shijie To: , CC: , , , , , , , , , , , , , , , , , , , , , , , , Huang Shijie Subject: [RFC PATCH 3/3] fs/proc: split the inode list Date: Mon, 14 Sep 2026 14:14:49 +0800 Message-ID: <20260914061449.4024632-4-huangsj@hygon.cn> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260914061449.4024632-1-huangsj@hygon.cn> References: <20260914061449.4024632-1-huangsj@hygon.cn> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: cncheex06.Hygon.cn (172.23.18.116) To cncheex04.Hygon.cn (172.23.18.114) Content-Type: text/plain; charset="utf-8" The global s_inode_list_lock is heavily contended in procfs on a 384-CPU, 12-NUMA-node Hygon machine running Hadoop TestDFSIO: #hadoop jar xxxx.jar TestDFSIO -read -nrFiles 1000 -size 100MB The hadoop will create lot of threads during the test. The perf shows it consuming ~90% of the lock hotspot in procfs. The lock is hit from both directions: -- inode creation (~49%) : getdents64 -> proc_readfd_common -> new_inode -> inode_sb_list_add() -- inode eviction (~41%): process exit -> release_task -> proc_invalidate_siblings_dcache -> evict -> inode_sb_list_del() This patch spreads the inode list across per-shard locks for procfs: --- Add "shards" pointer shares a union with s_inodes. Guard all shard accesses with an explicit s_inode_list_sharded field in super_block. --- Add inode_list_add()/inode_list_del() callbacks to super_operations; procfs implements them to round-robin inodes onto nr_shards =3D DIV_ROUND_UP(num_possible_cpus(), 32) shards allocated at mount time, each protected by its own spinlock. --- evict_inodes(), generic_shutdown_super() iterate the shards when the super_block inode list is sharded. struct inode_shard is cacheline-aligned to avoid false sharing between shard locks on different NUMA nodes. This reduces the s_inode_list_lock hotspot from ~90% to ~1% in above TestDFSIO. And we can improve the hadoop exec performance over 50%. Signed-off-by: Huang Shijie --- fs/inode.c | 80 ++++++++++++++++++++++------------ fs/proc/inode.c | 53 ++++++++++++++++++++++ fs/proc/internal.h | 2 + fs/proc/root.c | 6 +++ fs/super.c | 40 +++++++++++++---- include/linux/fs/super_types.h | 22 +++++++++- include/linux/proc_fs.h | 1 + 7 files changed, 167 insertions(+), 37 deletions(-) diff --git a/fs/inode.c b/fs/inode.c index ba7da39be4a3..687c2e299258 100644 --- a/fs/inode.c +++ b/fs/inode.c @@ -635,6 +635,10 @@ void inode_sb_list_add(struct inode *inode) { struct super_block *sb =3D inode->i_sb; =20 + if (sb->s_inode_list_sharded && sb->s_op->inode_list_add) { + sb->s_op->inode_list_add(sb, inode); + return; + } spin_lock(&sb->s_inode_list_lock); list_add(&inode->i_sb_list, &sb->s_inodes); spin_unlock(&sb->s_inode_list_lock); @@ -646,6 +650,10 @@ static inline void inode_sb_list_del(struct inode *ino= de) struct super_block *sb =3D inode->i_sb; =20 if (!list_empty(&inode->i_sb_list)) { + if (sb->s_inode_list_sharded && sb->s_op->inode_list_del) { + sb->s_op->inode_list_del(sb, inode); + return; + } spin_lock(&sb->s_inode_list_lock); list_del_init(&inode->i_sb_list); spin_unlock(&sb->s_inode_list_lock); @@ -879,41 +887,57 @@ void evict_inodes(struct super_block *sb) { struct inode *inode; LIST_HEAD(dispose); + struct list_head *head; + spinlock_t *lock; + unsigned int nr, i; + const bool sharded =3D sb->s_inode_list_sharded; =20 -again: - spin_lock(&sb->s_inode_list_lock); - list_for_each_entry(inode, &sb->s_inodes, i_sb_list) { - if (icount_read_once(inode)) - continue; + nr =3D sharded ? sb->nr_shards : 1; =20 - spin_lock(&inode->i_lock); - if (icount_read(inode)) { - spin_unlock(&inode->i_lock); - continue; - } - if (inode_state_read(inode) & (I_NEW | I_FREEING | I_WILL_FREE)) { - spin_unlock(&inode->i_lock); - continue; + for (i =3D 0; i < nr; i++) { + if (sharded) { + head =3D &sb->shards[i].list; + lock =3D &sb->shards[i].lock; + } else { + head =3D &sb->s_inodes; + lock =3D &sb->s_inode_list_lock; } =20 - inode_state_set(inode, I_FREEING); - inode_lru_list_del(inode); - spin_unlock(&inode->i_lock); - list_add(&inode->i_lru, &dispose); +again: + spin_lock(lock); + list_for_each_entry(inode, head, i_sb_list) { + if (icount_read_once(inode)) + continue; =20 - /* - * We can have a ton of inodes to evict at unmount time given - * enough memory, check to see if we need to go to sleep for a - * bit so we don't livelock. - */ - if (need_resched()) { - spin_unlock(&sb->s_inode_list_lock); - cond_resched(); - dispose_list(&dispose); - goto again; + spin_lock(&inode->i_lock); + if (icount_read(inode)) { + spin_unlock(&inode->i_lock); + continue; + } + if (inode_state_read(inode) & (I_NEW | I_FREEING | I_WILL_FREE)) { + spin_unlock(&inode->i_lock); + continue; + } + + inode_state_set(inode, I_FREEING); + inode_lru_list_del(inode); + spin_unlock(&inode->i_lock); + list_add(&inode->i_lru, &dispose); + + /* + * We can have a ton of inodes to evict at unmount time given + * enough memory, check to see if we need to go to sleep for a + * bit so we don't livelock. + */ + if (need_resched()) { + spin_unlock(lock); + cond_resched(); + dispose_list(&dispose); + goto again; + } } + spin_unlock(lock); } - spin_unlock(&sb->s_inode_list_lock); =20 dispose_list(&dispose); } diff --git a/fs/proc/inode.c b/fs/proc/inode.c index b7634f975d98..d412018b73d9 100644 --- a/fs/proc/inode.c +++ b/fs/proc/inode.c @@ -65,6 +65,7 @@ static struct inode *proc_alloc_inode(struct super_block = *sb) ei->sysctl_entry =3D NULL; INIT_HLIST_NODE(&ei->sibling_inodes); ei->ns_ops =3D NULL; + ei->shard_idx =3D 0; return &ei->vfs_inode; } =20 @@ -80,6 +81,56 @@ static void proc_free_inode(struct inode *inode) kmem_cache_free(proc_inode_cachep, PROC_I(inode)); } =20 +static void proc_inode_list_add(struct super_block *sb, struct inode *inod= e) +{ + struct proc_fs_info *fs_info =3D proc_sb_info(sb); + struct proc_inode *ei =3D PROC_I(inode); + struct inode_shard *shard; + unsigned int idx, seq; + + seq =3D atomic_fetch_inc(&fs_info->shard_seq); + idx =3D seq % sb->nr_shards; + ei->shard_idx =3D idx; + shard =3D &sb->shards[idx]; + + spin_lock(&shard->lock); + list_add(&inode->i_sb_list, &shard->list); + spin_unlock(&shard->lock); +} + +static void proc_inode_list_del(struct super_block *sb, struct inode *inod= e) +{ + struct proc_inode *ei =3D PROC_I(inode); + struct inode_shard *shard =3D &sb->shards[ei->shard_idx]; + + spin_lock(&shard->lock); + list_del_init(&inode->i_sb_list); + spin_unlock(&shard->lock); +} + +#define PROC_LIST_ALIGN 32 +int proc_init_inode_shards(struct super_block *sb) +{ + struct inode_shard *shards; + unsigned int nr_shards; + int i; + + nr_shards =3D DIV_ROUND_UP(num_possible_cpus(), PROC_LIST_ALIGN); + shards =3D kcalloc(nr_shards, sizeof(*shards), GFP_KERNEL); + if (!shards) + return -ENOMEM; + + for (i =3D 0; i < nr_shards; i++) { + INIT_LIST_HEAD(&shards[i].list); + spin_lock_init(&shards[i].lock); + } + + sb->shards =3D shards; + sb->nr_shards =3D nr_shards; + sb->s_inode_list_sharded =3D true; + return 0; +} + static void init_once(void *foo) { struct proc_inode *ei =3D (struct proc_inode *) foo; @@ -191,6 +242,8 @@ const struct super_operations proc_sops =3D { .evict_inode =3D proc_evict_inode, .statfs =3D simple_statfs, .show_options =3D proc_show_options, + .inode_list_add =3D proc_inode_list_add, + .inode_list_del =3D proc_inode_list_del, }; =20 enum {BIAS =3D -1U<<31}; diff --git a/fs/proc/internal.h b/fs/proc/internal.h index 04bd6c9e65a7..edbe357263c2 100644 --- a/fs/proc/internal.h +++ b/fs/proc/internal.h @@ -127,6 +127,7 @@ struct proc_inode { struct hlist_node sibling_inodes; const struct proc_ns_operations *ns_ops; struct inode vfs_inode; + unsigned int shard_idx; } __randomize_layout; =20 /* @@ -317,6 +318,7 @@ void proc_init_kmemcache(void); void proc_invalidate_siblings_dcache(struct hlist_head *inodes, spinlock_t= *lock); void set_proc_pid_nlink(void); extern struct inode *proc_get_inode(struct super_block *, struct proc_dir_= entry *); +extern int proc_init_inode_shards(struct super_block *); extern void proc_entry_rundown(struct proc_dir_entry *); =20 /* diff --git a/fs/proc/root.c b/fs/proc/root.c index 99adddfeb4a4..4bd916b88cdf 100644 --- a/fs/proc/root.c +++ b/fs/proc/root.c @@ -275,6 +275,10 @@ static int proc_fill_super(struct super_block *s, stru= ct fs_context *fc) s->s_time_gran =3D 1; s->s_fs_info =3D fs_info; =20 + ret =3D proc_init_inode_shards(s); + if (ret) + return ret; + if (fs_info->pidonly =3D=3D PROC_PIDONLY_ON) s->s_iflags |=3D SB_I_RESTRICTED_VARIANT; =20 @@ -359,6 +363,8 @@ static void proc_kill_sb(struct super_block *sb) struct proc_fs_info *fs_info =3D proc_sb_info(sb); =20 kill_anon_super(sb); + if (sb->s_inode_list_sharded) + kfree(sb->shards); if (fs_info) { put_pid_ns(fs_info->pid_ns); put_cred(fs_info->mounter_cred); diff --git a/fs/super.c b/fs/super.c index 4b1314545522..38a452e78ca2 100644 --- a/fs/super.c +++ b/fs/super.c @@ -726,7 +726,16 @@ EXPORT_SYMBOL(retire_super); =20 static bool sb_inodes_empty(struct super_block *sb) { - return list_empty(&sb->s_inodes); + unsigned int i; + + if (!sb->s_inode_list_sharded) + return list_empty(&sb->s_inodes); + + for (i =3D 0; i < sb->nr_shards; i++) + if (!list_empty(&sb->shards[i].list)) + return false; + + return true; } =20 /** @@ -788,14 +797,29 @@ void generic_shutdown_super(struct super_block *sb) * iput_final() or such crashes cleanly. */ struct inode *inode; - - spin_lock(&sb->s_inode_list_lock); - list_for_each_entry(inode, &sb->s_inodes, i_sb_list) { - inode->i_op =3D VFS_PTR_POISON; - inode->i_sb =3D VFS_PTR_POISON; - inode->i_mapping =3D VFS_PTR_POISON; + struct list_head *head; + spinlock_t *lock; + unsigned int nr, i; + const bool sharded =3D sb->s_inode_list_sharded; + + nr =3D sharded ? sb->nr_shards : 1; + for (i =3D 0; i < nr; i++) { + if (sharded) { + head =3D &sb->shards[i].list; + lock =3D &sb->shards[i].lock; + } else { + head =3D &sb->s_inodes; + lock =3D &sb->s_inode_list_lock; + } + + spin_lock(lock); + list_for_each_entry(inode, head, i_sb_list) { + inode->i_op =3D VFS_PTR_POISON; + inode->i_sb =3D VFS_PTR_POISON; + inode->i_mapping =3D VFS_PTR_POISON; + } + spin_unlock(lock); } - spin_unlock(&sb->s_inode_list_lock); } } /* diff --git a/include/linux/fs/super_types.h b/include/linux/fs/super_types.h index ecd96aeb1cee..6d6aae2d1fbc 100644 --- a/include/linux/fs/super_types.h +++ b/include/linux/fs/super_types.h @@ -130,8 +130,20 @@ struct super_operations { =20 /* Report a filesystem error */ void (*report_error)(const struct fserror_event *event); + + void (*inode_list_add)(struct super_block *sb, struct inode *inode); + void (*inode_list_del)(struct super_block *sb, struct inode *inode); }; =20 +/* + * Sharded inode list used by procfs to spread s_inode_list_lock contention + * across per-shard locks. + */ +struct inode_shard { + struct list_head list; + spinlock_t lock; +} ____cacheline_aligned_in_smp; + struct super_block { struct list_head s_list; /* Keep this first */ dev_t s_dev; /* search index; _not_ kdev_t */ @@ -269,9 +281,17 @@ struct super_block { */ int s_stack_depth; =20 + bool s_inode_list_sharded; + /* s_inode_list_lock protects s_inodes */ spinlock_t s_inode_list_lock ____cacheline_aligned_in_smp; - struct list_head s_inodes; /* all inodes */ + union { + struct list_head s_inodes; /* all inodes */ + struct { + struct inode_shard *shards; /* used by procfs */ + unsigned int nr_shards; + }; + }; =20 spinlock_t s_inode_wblist_lock; struct list_head s_inodes_wb; /* writeback inodes */ diff --git a/include/linux/proc_fs.h b/include/linux/proc_fs.h index 47d7deaeed8f..37eed34f3a93 100644 --- a/include/linux/proc_fs.h +++ b/include/linux/proc_fs.h @@ -70,6 +70,7 @@ struct proc_fs_info { const struct cred *mounter_cred; enum proc_hidepid hide_pid; enum proc_pidonly pidonly; + atomic_t shard_seq; struct rcu_head rcu; }; =20 --=20 2.53.0