From nobody Tue Oct 7 05:23:48 2025 Received: from szxga03-in.huawei.com (szxga03-in.huawei.com [45.249.212.189]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 921E41FBCA7; Mon, 14 Jul 2025 13:18:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=45.249.212.189 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752499116; cv=none; b=EIEtSB8iegSzb0w0RlALNge51B+8akRhd/TQAtbd7PFBB09lFGXNZ6yudnYBFy0ajoVViodsgmcZat8AhywlH+EoGM+XKLHFWK2XtShUEFkFlgIs27oohH3Qn2IlP3P6454j3F14in5OrF4hUGvAcXLNPFBxNI8IY/jzAKOXw10= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752499116; c=relaxed/simple; bh=26iNogcD/IZq1J6MQAH9InkD+zCLDu3ta/LmyX2eBag=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=OZ6ahOUsktTYnnwR7OxQuUCu09L80iQcX/KUkCop54zZzHFJ8RaEwWWdZOmIJF9PrdHgK8235wKhHpYneokxG3Dd7rSkNncl/BOAntIp6y7XPW0PspdY/SoeJQlsAE30CrZfhr75ZyQyOPPZaBcVsZu259Ckqg5zA+uBFsfsDco= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com; spf=pass smtp.mailfrom=huawei.com; arc=none smtp.client-ip=45.249.212.189 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=huawei.com Received: from mail.maildlp.com (unknown [172.19.163.252]) by szxga03-in.huawei.com (SkyGuard) with ESMTP id 4bgjTC0BQHzHrVG; Mon, 14 Jul 2025 21:14:23 +0800 (CST) Received: from dggpemf500013.china.huawei.com (unknown [7.185.36.188]) by mail.maildlp.com (Postfix) with ESMTPS id 6D158180B43; Mon, 14 Jul 2025 21:18:29 +0800 (CST) Received: from huawei.com (10.175.112.188) by dggpemf500013.china.huawei.com (7.185.36.188) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.11; Mon, 14 Jul 2025 21:18:28 +0800 From: Baokun Li To: CC: , , , , , , , , , Subject: [PATCH v3 01/17] ext4: add ext4_try_lock_group() to skip busy groups Date: Mon, 14 Jul 2025 21:03:11 +0800 Message-ID: <20250714130327.1830534-2-libaokun1@huawei.com> X-Mailer: git-send-email 2.46.1 In-Reply-To: <20250714130327.1830534-1-libaokun1@huawei.com> References: <20250714130327.1830534-1-libaokun1@huawei.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: kwepems500001.china.huawei.com (7.221.188.70) To dggpemf500013.china.huawei.com (7.185.36.188) Content-Type: text/plain; charset="utf-8" When ext4 allocates blocks, we used to just go through the block groups one by one to find a good one. But when there are tons of block groups (like hundreds of thousands or even millions) and not many have free space (meaning they're mostly full), it takes a really long time to check them all, and performance gets bad. So, we added the "mb_optimize_scan" mount option (which is on by default now). It keeps track of some group lists, so when we need a free block, we can just grab a likely group from the right list. This saves time and makes block allocation much faster. But when multiple processes or containers are doing similar things, like constantly allocating 8k blocks, they all try to use the same block group in the same list. Even just two processes doing this can cut the IOPS in half. For example, one container might do 300,000 IOPS, but if you run two at the same time, the total is only 150,000. Since we can already look at block groups in a non-linear way, the first and last groups in the same list are basically the same for finding a block right now. Therefore, add an ext4_try_lock_group() helper function to skip the current group when it is locked by another process, thereby avoiding contention with other processes. This helps ext4 make better use of having multiple block groups. Also, to make sure we don't skip all the groups that have free space when allocating blocks, we won't try to skip busy groups anymore when ac_criteria is CR_ANY_FREE. Performance test data follows: Test: Running will-it-scale/fallocate2 on CPU-bound containers. Observation: Average fallocate operations per container per second. |CPU: Kunpeng 920 | P80 | |Memory: 512GB |-------------------------| |960GB SSD (0.5GB/s)| base | patched | |-------------------|-------|-----------------| |mb_optimize_scan=3D0 | 2667 | 4821 (+80.7%) | |mb_optimize_scan=3D1 | 2643 | 4784 (+81.0%) | |CPU: AMD 9654 * 2 | P96 | |Memory: 1536GB |-------------------------| |960GB SSD (1GB/s) | base | patched | |-------------------|-------|-----------------| |mb_optimize_scan=3D0 | 3450 | 15371 (+345%) | |mb_optimize_scan=3D1 | 3209 | 6101 (+90.0%) | Signed-off-by: Baokun Li Reviewed-by: Jan Kara Reviewed-by: Zhang Yi --- fs/ext4/ext4.h | 23 ++++++++++++++--------- fs/ext4/mballoc.c | 19 ++++++++++++++++--- 2 files changed, 30 insertions(+), 12 deletions(-) diff --git a/fs/ext4/ext4.h b/fs/ext4/ext4.h index 18373de980f2..9df74123e7e6 100644 --- a/fs/ext4/ext4.h +++ b/fs/ext4/ext4.h @@ -3541,23 +3541,28 @@ static inline int ext4_fs_is_busy(struct ext4_sb_in= fo *sbi) return (atomic_read(&sbi->s_lock_busy) > EXT4_CONTENTION_THRESHOLD); } =20 +static inline bool ext4_try_lock_group(struct super_block *sb, ext4_group_= t group) +{ + if (!spin_trylock(ext4_group_lock_ptr(sb, group))) + return false; + /* + * We're able to grab the lock right away, so drop the lock + * contention counter. + */ + atomic_add_unless(&EXT4_SB(sb)->s_lock_busy, -1, 0); + return true; +} + static inline void ext4_lock_group(struct super_block *sb, ext4_group_t gr= oup) { - spinlock_t *lock =3D ext4_group_lock_ptr(sb, group); - if (spin_trylock(lock)) - /* - * We're able to grab the lock right away, so drop the - * lock contention counter. - */ - atomic_add_unless(&EXT4_SB(sb)->s_lock_busy, -1, 0); - else { + if (!ext4_try_lock_group(sb, group)) { /* * The lock is busy, so bump the contention counter, * and then wait on the spin lock. */ atomic_add_unless(&EXT4_SB(sb)->s_lock_busy, 1, EXT4_MAX_CONTENTION); - spin_lock(lock); + spin_lock(ext4_group_lock_ptr(sb, group)); } } =20 diff --git a/fs/ext4/mballoc.c b/fs/ext4/mballoc.c index 1e98c5be4e0a..336d65c4f6a2 100644 --- a/fs/ext4/mballoc.c +++ b/fs/ext4/mballoc.c @@ -896,7 +896,8 @@ static void ext4_mb_choose_next_group_p2_aligned(struct= ext4_allocation_context bb_largest_free_order_node) { if (sbi->s_mb_stats) atomic64_inc(&sbi->s_bal_cX_groups_considered[CR_POWER2_ALIGNED]); - if (likely(ext4_mb_good_group(ac, iter->bb_group, CR_POWER2_ALIGNED))) { + if (!spin_is_locked(ext4_group_lock_ptr(ac->ac_sb, iter->bb_group)) && + likely(ext4_mb_good_group(ac, iter->bb_group, CR_POWER2_ALIGNED))) { *group =3D iter->bb_group; ac->ac_flags |=3D EXT4_MB_CR_POWER2_ALIGNED_OPTIMIZED; read_unlock(&sbi->s_mb_largest_free_orders_locks[i]); @@ -932,7 +933,8 @@ ext4_mb_find_good_group_avg_frag_lists(struct ext4_allo= cation_context *ac, int o list_for_each_entry(iter, frag_list, bb_avg_fragment_size_node) { if (sbi->s_mb_stats) atomic64_inc(&sbi->s_bal_cX_groups_considered[cr]); - if (likely(ext4_mb_good_group(ac, iter->bb_group, cr))) { + if (!spin_is_locked(ext4_group_lock_ptr(ac->ac_sb, iter->bb_group)) && + likely(ext4_mb_good_group(ac, iter->bb_group, cr))) { grp =3D iter; break; } @@ -2899,6 +2901,11 @@ ext4_mb_regular_allocator(struct ext4_allocation_con= text *ac) nr, &prefetch_ios); } =20 + /* prevent unnecessary buddy loading. */ + if (cr < CR_ANY_FREE && + spin_is_locked(ext4_group_lock_ptr(sb, group))) + continue; + /* This now checks without needing the buddy page */ ret =3D ext4_mb_good_group_nolock(ac, group, cr); if (ret <=3D 0) { @@ -2911,7 +2918,13 @@ ext4_mb_regular_allocator(struct ext4_allocation_con= text *ac) if (err) goto out; =20 - ext4_lock_group(sb, group); + /* skip busy group */ + if (cr >=3D CR_ANY_FREE) { + ext4_lock_group(sb, group); + } else if (!ext4_try_lock_group(sb, group)) { + ext4_mb_unload_buddy(&e4b); + continue; + } =20 /* * We need to check again after locking the --=20 2.46.1 From nobody Tue Oct 7 05:23:48 2025 Received: from szxga01-in.huawei.com (szxga01-in.huawei.com [45.249.212.187]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D89E6248F5C; Mon, 14 Jul 2025 13:18:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=45.249.212.187 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752499116; cv=none; b=kY93+OkIu8+sdBYGFOexyzF9INGpmaHdeIJWJ3FY2V2H5MgP21HCQF08hX6NsWigxb9FZ8SyPWnh4e6ObqxZFt6apHw9EKHVcQ8BdRrX4w6DUB30U6ov1YTiOu2IhLIHZ2d+6oucX97vVFMe79G+HGQ2DxzlB7wZ6akuV42O5ew= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752499116; c=relaxed/simple; bh=s4gNMvY9cZOwDGjkUhM/jI1sppzTPno8BNjqtEuLb5w=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=AAx2CKul2LSMeWgZzSaH0DtqXDaPttndbqCXBFV0pg/eHsucrFEcnNFGk7dZy2Ms1LDxF/qWGSNZL6uZHV8oJzop1LSLAq8now3uhaeFCQm8D3tLIouDTzWXuhVOG0CvPZk9gMi9Rg8BVbcAqQOjpNxZMsW5mBC4biFgJs72VIE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com; spf=pass smtp.mailfrom=huawei.com; arc=none smtp.client-ip=45.249.212.187 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=huawei.com Received: from mail.maildlp.com (unknown [172.19.163.174]) by szxga01-in.huawei.com (SkyGuard) with ESMTP id 4bgjVm6SVXz13Mnb; Mon, 14 Jul 2025 21:15:44 +0800 (CST) Received: from dggpemf500013.china.huawei.com (unknown [7.185.36.188]) by mail.maildlp.com (Postfix) with ESMTPS id 69D6014011A; Mon, 14 Jul 2025 21:18:30 +0800 (CST) Received: from huawei.com (10.175.112.188) by dggpemf500013.china.huawei.com (7.185.36.188) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.11; Mon, 14 Jul 2025 21:18:29 +0800 From: Baokun Li To: CC: , , , , , , , , , Subject: [PATCH v3 02/17] ext4: separate stream goal hits from s_bal_goals for better tracking Date: Mon, 14 Jul 2025 21:03:12 +0800 Message-ID: <20250714130327.1830534-3-libaokun1@huawei.com> X-Mailer: git-send-email 2.46.1 In-Reply-To: <20250714130327.1830534-1-libaokun1@huawei.com> References: <20250714130327.1830534-1-libaokun1@huawei.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: kwepems500001.china.huawei.com (7.221.188.70) To dggpemf500013.china.huawei.com (7.185.36.188) Content-Type: text/plain; charset="utf-8" In ext4_mb_regular_allocator(), after the call to ext4_mb_find_by_goal() fails to achieve the inode goal, allocation continues with the stream allocation global goal. Currently, hits for both are combined in sbi->s_bal_goals, hindering accurate optimization. This commit separates global goal hits into sbi->s_bal_stream_goals. Since stream allocation doesn't use ac->ac_g_ex.fe_start, set fe_start to -1. This prevents stream allocations from being counted in s_bal_goals. Also clear EXT4_MB_HINT_TRY_GOAL to avoid calling ext4_mb_find_by_goal again. After adding `stream_goal_hits`, `/proc/fs/ext4/sdx/mb_stats` will show: mballoc: reqs: 840347 success: 750992 groups_scanned: 1230506 cr_p2_aligned_stats: hits: 21531 groups_considered: 411664 extents_scanned: 21531 useless_loops: 0 bad_suggestions: 6 cr_goal_fast_stats: hits: 111222 groups_considered: 1806728 extents_scanned: 467908 useless_loops: 0 bad_suggestions: 13 cr_best_avail_stats: hits: 36267 groups_considered: 1817631 extents_scanned: 156143 useless_loops: 0 bad_suggestions: 204 cr_goal_slow_stats: hits: 106396 groups_considered: 5671710 extents_scanned: 22540056 useless_loops: 123747 cr_any_free_stats: hits: 138071 groups_considered: 724692 extents_scanned: 23615593 useless_loops: 585 extents_scanned: 46804261 goal_hits: 1307 stream_goal_hits: 236317 len_goal_hits: 155549 2^n_hits: 21531 breaks: 225096 lost: 35062 buddies_generated: 40/40 buddies_time_used: 48004 preallocated: 5962467 discarded: 4847560 Signed-off-by: Baokun Li Reviewed-by: Zhang Yi --- fs/ext4/ext4.h | 1 + fs/ext4/mballoc.c | 11 +++++++++-- 2 files changed, 10 insertions(+), 2 deletions(-) diff --git a/fs/ext4/ext4.h b/fs/ext4/ext4.h index 9df74123e7e6..8750ace12935 100644 --- a/fs/ext4/ext4.h +++ b/fs/ext4/ext4.h @@ -1646,6 +1646,7 @@ struct ext4_sb_info { atomic_t s_bal_cX_ex_scanned[EXT4_MB_NUM_CRS]; /* total extents scanned */ atomic_t s_bal_groups_scanned; /* number of groups scanned */ atomic_t s_bal_goals; /* goal hits */ + atomic_t s_bal_stream_goals; /* stream allocation global goal hits */ atomic_t s_bal_len_goals; /* len goal hits */ atomic_t s_bal_breaks; /* too long searches */ atomic_t s_bal_2orders; /* 2^order hits */ diff --git a/fs/ext4/mballoc.c b/fs/ext4/mballoc.c index 336d65c4f6a2..f56ac477c464 100644 --- a/fs/ext4/mballoc.c +++ b/fs/ext4/mballoc.c @@ -2849,8 +2849,9 @@ ext4_mb_regular_allocator(struct ext4_allocation_cont= ext *ac) /* TBD: may be hot point */ spin_lock(&sbi->s_md_lock); ac->ac_g_ex.fe_group =3D sbi->s_mb_last_group; - ac->ac_g_ex.fe_start =3D sbi->s_mb_last_start; spin_unlock(&sbi->s_md_lock); + ac->ac_g_ex.fe_start =3D -1; + ac->ac_flags &=3D ~EXT4_MB_HINT_TRY_GOAL; } =20 /* @@ -3000,8 +3001,12 @@ ext4_mb_regular_allocator(struct ext4_allocation_con= text *ac) } } =20 - if (sbi->s_mb_stats && ac->ac_status =3D=3D AC_STATUS_FOUND) + if (sbi->s_mb_stats && ac->ac_status =3D=3D AC_STATUS_FOUND) { atomic64_inc(&sbi->s_bal_cX_hits[ac->ac_criteria]); + if (ac->ac_flags & EXT4_MB_STREAM_ALLOC && + ac->ac_b_ex.fe_group =3D=3D ac->ac_g_ex.fe_group) + atomic_inc(&sbi->s_bal_stream_goals); + } out: if (!err && ac->ac_status !=3D AC_STATUS_FOUND && first_err) err =3D first_err; @@ -3194,6 +3199,8 @@ int ext4_seq_mb_stats_show(struct seq_file *seq, void= *offset) seq_printf(seq, "\textents_scanned: %u\n", atomic_read(&sbi->s_bal_ex_scanned)); seq_printf(seq, "\t\tgoal_hits: %u\n", atomic_read(&sbi->s_bal_goals)); + seq_printf(seq, "\t\tstream_goal_hits: %u\n", + atomic_read(&sbi->s_bal_stream_goals)); seq_printf(seq, "\t\tlen_goal_hits: %u\n", atomic_read(&sbi->s_bal_len_goals)); seq_printf(seq, "\t\t2^n_hits: %u\n", atomic_read(&sbi->s_bal_2orders)); --=20 2.46.1 From nobody Tue Oct 7 05:23:48 2025 Received: from szxga04-in.huawei.com (szxga04-in.huawei.com [45.249.212.190]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 346EB24A06E; Mon, 14 Jul 2025 13:18:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=45.249.212.190 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752499117; cv=none; b=ehs2hure6pNeWA1HHhcePMWFStwtmtrpF0HJpxu4T6dVqZ1VqBfkOxMs287Km2Ea9ZbxmDRKZ9e+VnkzTZudCm8sgm4v48xS3iKYBY5SkEXuNqEAtwQ49yH29Lwy/GSHgi0otG9SKB3qWJDrUbDRi0LInCfW7Fn9BPBoONPPUAA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752499117; c=relaxed/simple; bh=qT8ji9P4W0ZmRATvMr/WmpW4YqF/L+hpzvAaZZrkZ+0=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=S4gDMbM7e4oAmEf0+UU4fIRRHPJGWBuKKjCfsRToLpuBHmg4uzpwuFciCxv38nQUELPT9BMRdxdjIi+d9j+9sMNP0E9Fct49ZI6wfxHbX165Hr4FDJ0EyyjKS15ZmryPrWUhR8XhcqiGHYWmMBf/jNISKiIJr62lRQjcqCTnAGM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com; spf=pass smtp.mailfrom=huawei.com; arc=none smtp.client-ip=45.249.212.190 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=huawei.com Received: from mail.maildlp.com (unknown [172.19.163.17]) by szxga04-in.huawei.com (SkyGuard) with ESMTP id 4bgjWd3mP1z2TT0R; Mon, 14 Jul 2025 21:16:29 +0800 (CST) Received: from dggpemf500013.china.huawei.com (unknown [7.185.36.188]) by mail.maildlp.com (Postfix) with ESMTPS id 7340E1A0188; Mon, 14 Jul 2025 21:18:31 +0800 (CST) Received: from huawei.com (10.175.112.188) by dggpemf500013.china.huawei.com (7.185.36.188) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.11; Mon, 14 Jul 2025 21:18:30 +0800 From: Baokun Li To: CC: , , , , , , , , , Subject: [PATCH v3 03/17] ext4: remove unnecessary s_mb_last_start Date: Mon, 14 Jul 2025 21:03:13 +0800 Message-ID: <20250714130327.1830534-4-libaokun1@huawei.com> X-Mailer: git-send-email 2.46.1 In-Reply-To: <20250714130327.1830534-1-libaokun1@huawei.com> References: <20250714130327.1830534-1-libaokun1@huawei.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: kwepems500001.china.huawei.com (7.221.188.70) To dggpemf500013.china.huawei.com (7.185.36.188) Content-Type: text/plain; charset="utf-8" Since stream allocation does not use ac->ac_f_ex.fe_start, it is set to -1 by default, so the no longer needed sbi->s_mb_last_start is removed. Signed-off-by: Baokun Li Reviewed-by: Jan Kara Reviewed-by: Ojaswin Mujoo Reviewed-by: Zhang Yi --- fs/ext4/ext4.h | 1 - fs/ext4/mballoc.c | 1 - 2 files changed, 2 deletions(-) diff --git a/fs/ext4/ext4.h b/fs/ext4/ext4.h index 8750ace12935..b83095541c98 100644 --- a/fs/ext4/ext4.h +++ b/fs/ext4/ext4.h @@ -1631,7 +1631,6 @@ struct ext4_sb_info { unsigned int s_max_dir_size_kb; /* where last allocation was done - for stream allocation */ unsigned long s_mb_last_group; - unsigned long s_mb_last_start; unsigned int s_mb_prefetch; unsigned int s_mb_prefetch_limit; unsigned int s_mb_best_avail_max_trim_order; diff --git a/fs/ext4/mballoc.c b/fs/ext4/mballoc.c index f56ac477c464..e3a5103e1620 100644 --- a/fs/ext4/mballoc.c +++ b/fs/ext4/mballoc.c @@ -2171,7 +2171,6 @@ static void ext4_mb_use_best_found(struct ext4_alloca= tion_context *ac, if (ac->ac_flags & EXT4_MB_STREAM_ALLOC) { spin_lock(&sbi->s_md_lock); sbi->s_mb_last_group =3D ac->ac_f_ex.fe_group; - sbi->s_mb_last_start =3D ac->ac_f_ex.fe_start; spin_unlock(&sbi->s_md_lock); } /* --=20 2.46.1 From nobody Tue Oct 7 05:23:48 2025 Received: from szxga01-in.huawei.com (szxga01-in.huawei.com [45.249.212.187]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6BFEF24BBEE; Mon, 14 Jul 2025 13:18:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=45.249.212.187 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752499117; cv=none; b=ZVuqvVUoHSzMOb/t6tbdBcyZHNUkHzHc9muMxTMVrUM4eOe3j4+5hcrbk33P3pyo8GowyhIF3aZ23Ho2IaN0OQ6/rRcEbB8c/9TVvny+sU6Ms0I+E9NQH79t+rsLCIvZ2uWKFUyy/GFBtyZq5XfmlaUxVidA6HXPUgcBAZGUFbg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752499117; c=relaxed/simple; bh=Gjex9zbY6rDfJ6Nyy5o5qQ8vIt7nCR0bSjIsrJBgUy0=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=QofZBr4Utio5xGTbfMtzejgmzLuy44D87sAb7vYkEZA3nTpvDk4TwlYTBnIH07bJHq3Sh81W1LQeVSInaYYoZE0iMRrD5gDa1Zhafa8XZJMlnvOIhi94MErX9Z1waXTGIDExp8lZkgcBT2+jz+wt8k2xk8ydElFx0wHvmeBsQfw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com; spf=pass smtp.mailfrom=huawei.com; arc=none smtp.client-ip=45.249.212.187 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=huawei.com Received: from mail.maildlp.com (unknown [172.19.163.48]) by szxga01-in.huawei.com (SkyGuard) with ESMTP id 4bgjSV3lGYz14M0B; Mon, 14 Jul 2025 21:13:46 +0800 (CST) Received: from dggpemf500013.china.huawei.com (unknown [7.185.36.188]) by mail.maildlp.com (Postfix) with ESMTPS id 78B9218006C; Mon, 14 Jul 2025 21:18:32 +0800 (CST) Received: from huawei.com (10.175.112.188) by dggpemf500013.china.huawei.com (7.185.36.188) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.11; Mon, 14 Jul 2025 21:18:31 +0800 From: Baokun Li To: CC: , , , , , , , , , Subject: [PATCH v3 04/17] ext4: remove unnecessary s_md_lock on update s_mb_last_group Date: Mon, 14 Jul 2025 21:03:14 +0800 Message-ID: <20250714130327.1830534-5-libaokun1@huawei.com> X-Mailer: git-send-email 2.46.1 In-Reply-To: <20250714130327.1830534-1-libaokun1@huawei.com> References: <20250714130327.1830534-1-libaokun1@huawei.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: kwepems500001.china.huawei.com (7.221.188.70) To dggpemf500013.china.huawei.com (7.185.36.188) Content-Type: text/plain; charset="utf-8" After we optimized the block group lock, we found another lock contention issue when running will-it-scale/fallocate2 with multiple processes. The fallocate's block allocation and the truncate's block release were fighting over the s_md_lock. The problem is, this lock protects totally different things in those two processes: the list of freed data blocks (s_freed_data_list) when releasing, and where to start looking for new blocks (mb_last_group) when allocating. Now we only need to track s_mb_last_group and no longer need to track s_mb_last_start, so we don't need the s_md_lock lock to ensure that the two are consistent. Since s_mb_last_group is merely a hint and doesn't require strong synchronization, READ_ONCE/WRITE_ONCE is sufficient. Besides, the s_mb_last_group data type only requires ext4_group_t (i.e., unsigned int), rendering unsigned long superfluous. Performance test data follows: Test: Running will-it-scale/fallocate2 on CPU-bound containers. Observation: Average fallocate operations per container per second. |CPU: Kunpeng 920 | P80 | P1 | |Memory: 512GB |------------------------|-------------------------| |960GB SSD (0.5GB/s)| base | patched | base | patched | |-------------------|-------|----------------|--------|----------------| |mb_optimize_scan=3D0 | 4821 | 9636 (+99.8%) | 314065 | 337597 (+7.4%) | |mb_optimize_scan=3D1 | 4784 | 4834 (+1.04%) | 316344 | 341440 (+7.9%) | |CPU: AMD 9654 * 2 | P96 | P1 | |Memory: 1536GB |------------------------|-------------------------| |960GB SSD (1GB/s) | base | patched | base | patched | |-------------------|-------|----------------|--------|----------------| |mb_optimize_scan=3D0 | 15371 | 22341 (+45.3%) | 205851 | 219707 (+6.7%) | |mb_optimize_scan=3D1 | 6101 | 9177 (+50.4%) | 207373 | 215732 (+4.0%) | Suggested-by: Jan Kara Signed-off-by: Baokun Li Reviewed-by: Ojaswin Mujoo Reviewed-by: Zhang Yi --- fs/ext4/ext4.h | 2 +- fs/ext4/mballoc.c | 12 +++--------- 2 files changed, 4 insertions(+), 10 deletions(-) diff --git a/fs/ext4/ext4.h b/fs/ext4/ext4.h index b83095541c98..7f5c070de0fb 100644 --- a/fs/ext4/ext4.h +++ b/fs/ext4/ext4.h @@ -1630,7 +1630,7 @@ struct ext4_sb_info { unsigned int s_mb_group_prealloc; unsigned int s_max_dir_size_kb; /* where last allocation was done - for stream allocation */ - unsigned long s_mb_last_group; + ext4_group_t s_mb_last_group; unsigned int s_mb_prefetch; unsigned int s_mb_prefetch_limit; unsigned int s_mb_best_avail_max_trim_order; diff --git a/fs/ext4/mballoc.c b/fs/ext4/mballoc.c index e3a5103e1620..025b759ca643 100644 --- a/fs/ext4/mballoc.c +++ b/fs/ext4/mballoc.c @@ -2168,11 +2168,8 @@ static void ext4_mb_use_best_found(struct ext4_alloc= ation_context *ac, ac->ac_buddy_folio =3D e4b->bd_buddy_folio; folio_get(ac->ac_buddy_folio); /* store last allocated for subsequent stream allocation */ - if (ac->ac_flags & EXT4_MB_STREAM_ALLOC) { - spin_lock(&sbi->s_md_lock); - sbi->s_mb_last_group =3D ac->ac_f_ex.fe_group; - spin_unlock(&sbi->s_md_lock); - } + if (ac->ac_flags & EXT4_MB_STREAM_ALLOC) + WRITE_ONCE(sbi->s_mb_last_group, ac->ac_f_ex.fe_group); /* * As we've just preallocated more space than * user requested originally, we store allocated @@ -2845,10 +2842,7 @@ ext4_mb_regular_allocator(struct ext4_allocation_con= text *ac) =20 /* if stream allocation is enabled, use global goal */ if (ac->ac_flags & EXT4_MB_STREAM_ALLOC) { - /* TBD: may be hot point */ - spin_lock(&sbi->s_md_lock); - ac->ac_g_ex.fe_group =3D sbi->s_mb_last_group; - spin_unlock(&sbi->s_md_lock); + ac->ac_g_ex.fe_group =3D READ_ONCE(sbi->s_mb_last_group); ac->ac_g_ex.fe_start =3D -1; ac->ac_flags &=3D ~EXT4_MB_HINT_TRY_GOAL; } --=20 2.46.1 From nobody Tue Oct 7 05:23:48 2025 Received: from szxga02-in.huawei.com (szxga02-in.huawei.com [45.249.212.188]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 47EC0253950; Mon, 14 Jul 2025 13:18:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=45.249.212.188 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752499120; cv=none; b=oFmP4NHC2q1wm0aDjG6Z6dksLEKFMqD5rmofavWaWxM3rjeH7au2ByaeF9aHJmzRXdZker5paGtfk+y0koVYHKKicvLxCGmwp6cCfzWn35oy0ptZvQf1GpjYs9KbUL9ydRBau+fMP60hOQHsLLhackbvSm8kUyPox02B0ojgxTA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752499120; c=relaxed/simple; bh=XULyWieM9zg0kqxJskYWGW2vaJqp9xDya4lJBbjdcx0=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=rS+nQ7t1noJzJmOUDlLqXoL/FcP8XHEk4Ksfb+OS0PTzD9/PF2vvEPnjM+1wLvxIi7ThyYB8aRDG2v83IG7FB+2tofflwGKDi7t9a/ExEUdM9DaGDJ3SZFema6uLHdczdHreqEi/vseD5akISF4Op+WGv/jCfhKb9WtI125WKDE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com; spf=pass smtp.mailfrom=huawei.com; arc=none smtp.client-ip=45.249.212.188 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=huawei.com Received: from mail.maildlp.com (unknown [172.19.162.254]) by szxga02-in.huawei.com (SkyGuard) with ESMTP id 4bgjSs2vS7zXdyt; Mon, 14 Jul 2025 21:14:05 +0800 (CST) Received: from dggpemf500013.china.huawei.com (unknown [7.185.36.188]) by mail.maildlp.com (Postfix) with ESMTPS id 7F75E180450; Mon, 14 Jul 2025 21:18:33 +0800 (CST) Received: from huawei.com (10.175.112.188) by dggpemf500013.china.huawei.com (7.185.36.188) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.11; Mon, 14 Jul 2025 21:18:32 +0800 From: Baokun Li To: CC: , , , , , , , , , Subject: [PATCH v3 05/17] ext4: utilize multiple global goals to reduce contention Date: Mon, 14 Jul 2025 21:03:15 +0800 Message-ID: <20250714130327.1830534-6-libaokun1@huawei.com> X-Mailer: git-send-email 2.46.1 In-Reply-To: <20250714130327.1830534-1-libaokun1@huawei.com> References: <20250714130327.1830534-1-libaokun1@huawei.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: kwepems500001.china.huawei.com (7.221.188.70) To dggpemf500013.china.huawei.com (7.185.36.188) When allocating data blocks, if the first try (goal allocation) fails and stream allocation is on, it tries a global goal starting from the last group we used (s_mb_last_group). This helps cluster large files together to reduce free space fragmentation, and the data block contiguity also accelerates write-back to disk. However, when multiple processes allocate blocks,=C2=A0having just one glob= al goal means they all fight over the same group. This drastically lowers the chances of extents merging and leads to much worse file fragmentation. To mitigate this multi-process contention, we now employ multiple global goals, with the number of goals being the minimum between the number of possible CPUs and one-quarter of the filesystem's total block group count. To ensure a consistent goal for each inode, we select the corresponding goal by taking the inode number modulo the total number of goals. Performance test data follows: Test: Running will-it-scale/fallocate2 on CPU-bound containers. Observation: Average fallocate operations per container per second. |CPU: Kunpeng 920 | P80 | P1 | |Memory: 512GB |------------------------|-------------------------| |960GB SSD (0.5GB/s)| base | patched | base | patched | |-------------------|-------|----------------|--------|----------------| |mb_optimize_scan=3D0 | 9636 | 19628 (+103%) | 337597 | 320885 (-4.9%) | |mb_optimize_scan=3D1 | 4834 | 7129 (+47.4%) | 341440 | 321275 (-5.9%) | |CPU: AMD 9654 * 2 | P96 | P1 | |Memory: 1536GB |------------------------|-------------------------| |960GB SSD (1GB/s) | base | patched | base | patched | |-------------------|-------|----------------|--------|----------------| |mb_optimize_scan=3D0 | 22341 | 53760 (+140%) | 219707 | 213145 (-2.9%) | |mb_optimize_scan=3D1 | 9177 | 12716 (+38.5%) | 215732 | 215262 (+0.2%) | Suggested-by: Jan Kara Signed-off-by: Baokun Li Reviewed-by: Zhang Yi --- fs/ext4/ext4.h | 6 ++++-- fs/ext4/mballoc.c | 27 +++++++++++++++++++++++---- 2 files changed, 27 insertions(+), 6 deletions(-) diff --git a/fs/ext4/ext4.h b/fs/ext4/ext4.h index 7f5c070de0fb..ad97c693d56a 100644 --- a/fs/ext4/ext4.h +++ b/fs/ext4/ext4.h @@ -1629,14 +1629,16 @@ struct ext4_sb_info { unsigned int s_mb_order2_reqs; unsigned int s_mb_group_prealloc; unsigned int s_max_dir_size_kb; - /* where last allocation was done - for stream allocation */ - ext4_group_t s_mb_last_group; unsigned int s_mb_prefetch; unsigned int s_mb_prefetch_limit; unsigned int s_mb_best_avail_max_trim_order; unsigned int s_sb_update_sec; unsigned int s_sb_update_kb; =20 + /* where last allocation was done - for stream allocation */ + ext4_group_t *s_mb_last_groups; + unsigned int s_mb_nr_global_goals; + /* stats for buddy allocator */ atomic_t s_bal_reqs; /* number of reqs with len > 1 */ atomic_t s_bal_success; /* we found long enough chunks */ diff --git a/fs/ext4/mballoc.c b/fs/ext4/mballoc.c index 025b759ca643..b6aa24b48543 100644 --- a/fs/ext4/mballoc.c +++ b/fs/ext4/mballoc.c @@ -2168,8 +2168,12 @@ static void ext4_mb_use_best_found(struct ext4_alloc= ation_context *ac, ac->ac_buddy_folio =3D e4b->bd_buddy_folio; folio_get(ac->ac_buddy_folio); /* store last allocated for subsequent stream allocation */ - if (ac->ac_flags & EXT4_MB_STREAM_ALLOC) - WRITE_ONCE(sbi->s_mb_last_group, ac->ac_f_ex.fe_group); + if (ac->ac_flags & EXT4_MB_STREAM_ALLOC) { + int hash =3D ac->ac_inode->i_ino % sbi->s_mb_nr_global_goals; + + WRITE_ONCE(sbi->s_mb_last_groups[hash], ac->ac_f_ex.fe_group); + } + /* * As we've just preallocated more space than * user requested originally, we store allocated @@ -2842,7 +2846,9 @@ ext4_mb_regular_allocator(struct ext4_allocation_cont= ext *ac) =20 /* if stream allocation is enabled, use global goal */ if (ac->ac_flags & EXT4_MB_STREAM_ALLOC) { - ac->ac_g_ex.fe_group =3D READ_ONCE(sbi->s_mb_last_group); + int hash =3D ac->ac_inode->i_ino % sbi->s_mb_nr_global_goals; + + ac->ac_g_ex.fe_group =3D READ_ONCE(sbi->s_mb_last_groups[hash]); ac->ac_g_ex.fe_start =3D -1; ac->ac_flags &=3D ~EXT4_MB_HINT_TRY_GOAL; } @@ -3722,10 +3728,19 @@ int ext4_mb_init(struct super_block *sb) sbi->s_mb_group_prealloc, EXT4_NUM_B2C(sbi, sbi->s_stripe)); } =20 + sbi->s_mb_nr_global_goals =3D umin(num_possible_cpus(), + DIV_ROUND_UP(sbi->s_groups_count, 4)); + sbi->s_mb_last_groups =3D kcalloc(sbi->s_mb_nr_global_goals, + sizeof(ext4_group_t), GFP_KERNEL); + if (sbi->s_mb_last_groups =3D=3D NULL) { + ret =3D -ENOMEM; + goto out; + } + sbi->s_locality_groups =3D alloc_percpu(struct ext4_locality_group); if (sbi->s_locality_groups =3D=3D NULL) { ret =3D -ENOMEM; - goto out; + goto out_free_last_groups; } for_each_possible_cpu(i) { struct ext4_locality_group *lg; @@ -3750,6 +3765,9 @@ int ext4_mb_init(struct super_block *sb) out_free_locality_groups: free_percpu(sbi->s_locality_groups); sbi->s_locality_groups =3D NULL; +out_free_last_groups: + kfree(sbi->s_mb_last_groups); + sbi->s_mb_last_groups =3D NULL; out: kfree(sbi->s_mb_avg_fragment_size); kfree(sbi->s_mb_avg_fragment_size_locks); @@ -3854,6 +3872,7 @@ void ext4_mb_release(struct super_block *sb) } =20 free_percpu(sbi->s_locality_groups); + kfree(sbi->s_mb_last_groups); } =20 static inline int ext4_issue_discard(struct super_block *sb, --=20 2.46.1 From nobody Tue Oct 7 05:23:48 2025 Received: from szxga04-in.huawei.com (szxga04-in.huawei.com [45.249.212.190]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 49F6A25392D; Mon, 14 Jul 2025 13:18:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=45.249.212.190 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752499121; cv=none; b=T8szNkjUxeJ3iqB9MCwDoaoHRzLhZbyyqDdHq0qmH3smN84v1bxAYzpA/YwR1G65/xjcGkQrcagkxUBbUta6xQ1ohyUc9UJ8lfEcq8SjU2J6Oma8rISkW6ij2jtcC9RW2x0BdDBJ+PegEfR0SiNZIF9tjdJpAr+8mEBThZmQrjI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752499121; c=relaxed/simple; bh=lljnkuKc04lOY0n4mHEyd6ao08zitQ2BV9p916AKCQs=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=cWgkmAF6M4Hpl7+RyRGOfIxj03sTYus7cqsTDlQQITKYfuT92OQosvUS/zXv3nnW9tffs2GgnjRnf3xQtjYOAWxq9t5vUcot2AzSVVy9PHaE4bvpp1ES5e1aOp09cpQ/q2xSQFDEfWKWzh4RatWdEEaf+QNNd87G4wqoNOZwN9M= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com; spf=pass smtp.mailfrom=huawei.com; arc=none smtp.client-ip=45.249.212.190 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=huawei.com Received: from mail.maildlp.com (unknown [172.19.88.163]) by szxga04-in.huawei.com (SkyGuard) with ESMTP id 4bgjWh40vwz2TT31; Mon, 14 Jul 2025 21:16:32 +0800 (CST) Received: from dggpemf500013.china.huawei.com (unknown [7.185.36.188]) by mail.maildlp.com (Postfix) with ESMTPS id 7BD4E1800B2; Mon, 14 Jul 2025 21:18:34 +0800 (CST) Received: from huawei.com (10.175.112.188) by dggpemf500013.china.huawei.com (7.185.36.188) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.11; Mon, 14 Jul 2025 21:18:33 +0800 From: Baokun Li To: CC: , , , , , , , , , Subject: [PATCH v3 06/17] ext4: get rid of some obsolete EXT4_MB_HINT flags Date: Mon, 14 Jul 2025 21:03:16 +0800 Message-ID: <20250714130327.1830534-7-libaokun1@huawei.com> X-Mailer: git-send-email 2.46.1 In-Reply-To: <20250714130327.1830534-1-libaokun1@huawei.com> References: <20250714130327.1830534-1-libaokun1@huawei.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: kwepems500001.china.huawei.com (7.221.188.70) To dggpemf500013.china.huawei.com (7.185.36.188) Content-Type: text/plain; charset="utf-8" Since nobody has used these EXT4_MB_HINT flags for ages, let's remove them. Signed-off-by: Baokun Li Reviewed-by: Ojaswin Mujoo Reviewed-by: Jan Kara Reviewed-by: Zhang Yi --- fs/ext4/ext4.h | 6 ------ include/trace/events/ext4.h | 3 --- 2 files changed, 9 deletions(-) diff --git a/fs/ext4/ext4.h b/fs/ext4/ext4.h index ad97c693d56a..4ebc665cf871 100644 --- a/fs/ext4/ext4.h +++ b/fs/ext4/ext4.h @@ -185,14 +185,8 @@ enum criteria { =20 /* prefer goal again. length */ #define EXT4_MB_HINT_MERGE 0x0001 -/* blocks already reserved */ -#define EXT4_MB_HINT_RESERVED 0x0002 -/* metadata is being allocated */ -#define EXT4_MB_HINT_METADATA 0x0004 /* first blocks in the file */ #define EXT4_MB_HINT_FIRST 0x0008 -/* search for the best chunk */ -#define EXT4_MB_HINT_BEST 0x0010 /* data is being allocated */ #define EXT4_MB_HINT_DATA 0x0020 /* don't preallocate (for tails) */ diff --git a/include/trace/events/ext4.h b/include/trace/events/ext4.h index 156908641e68..33b204165cc0 100644 --- a/include/trace/events/ext4.h +++ b/include/trace/events/ext4.h @@ -23,10 +23,7 @@ struct partial_cluster; =20 #define show_mballoc_flags(flags) __print_flags(flags, "|", \ { EXT4_MB_HINT_MERGE, "HINT_MERGE" }, \ - { EXT4_MB_HINT_RESERVED, "HINT_RESV" }, \ - { EXT4_MB_HINT_METADATA, "HINT_MDATA" }, \ { EXT4_MB_HINT_FIRST, "HINT_FIRST" }, \ - { EXT4_MB_HINT_BEST, "HINT_BEST" }, \ { EXT4_MB_HINT_DATA, "HINT_DATA" }, \ { EXT4_MB_HINT_NOPREALLOC, "HINT_NOPREALLOC" }, \ { EXT4_MB_HINT_GROUP_ALLOC, "HINT_GRP_ALLOC" }, \ --=20 2.46.1 From nobody Tue Oct 7 05:23:48 2025 Received: from szxga01-in.huawei.com (szxga01-in.huawei.com [45.249.212.187]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A9D422561AE; Mon, 14 Jul 2025 13:18:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=45.249.212.187 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752499120; cv=none; b=PCjNG32tlHswTFS0I3o5zXZJZjdqJ7Ql3ToVpinSQFe4B+HE5VjJ9RwViEF6ng7RknksIerg1ALbUFsNb3IandW+I3x/OEEkBNym/oovlLo7fNtFwowpK+ElJUDFLXBJ6EHznakaZUUjJjTm/6Enrce0sY5jYkI9GgwvoW8va14= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752499120; c=relaxed/simple; bh=cvcDGHDegoFsLa6WORXBMr6boH8vdKGFBRvDlhOWpzI=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=i139XNSQXTu0EQiNSISK2ivQg9mIH7cr7uDkLZppLyrA4Uz4KXW/+FOnvzyb+RBMVcOMsU1JROXy/4B4U67Wwfc36A86SCzZKi0FKf2D8foeZnVdHSWW6FJDCKEDbOwKCIHPvDHPyyeO5ELcVrMAuHrx5o9CoZXTPIWYxAzDxmo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com; spf=pass smtp.mailfrom=huawei.com; arc=none smtp.client-ip=45.249.212.187 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=huawei.com Received: from mail.maildlp.com (unknown [172.19.163.252]) by szxga01-in.huawei.com (SkyGuard) with ESMTP id 4bgjVs73Gtz13Mkm; Mon, 14 Jul 2025 21:15:49 +0800 (CST) Received: from dggpemf500013.china.huawei.com (unknown [7.185.36.188]) by mail.maildlp.com (Postfix) with ESMTPS id 7DDF6180B43; Mon, 14 Jul 2025 21:18:35 +0800 (CST) Received: from huawei.com (10.175.112.188) by dggpemf500013.china.huawei.com (7.185.36.188) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.11; Mon, 14 Jul 2025 21:18:34 +0800 From: Baokun Li To: CC: , , , , , , , , , Subject: [PATCH v3 07/17] ext4: fix typo in CR_GOAL_LEN_SLOW comment Date: Mon, 14 Jul 2025 21:03:17 +0800 Message-ID: <20250714130327.1830534-8-libaokun1@huawei.com> X-Mailer: git-send-email 2.46.1 In-Reply-To: <20250714130327.1830534-1-libaokun1@huawei.com> References: <20250714130327.1830534-1-libaokun1@huawei.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: kwepems500001.china.huawei.com (7.221.188.70) To dggpemf500013.china.huawei.com (7.185.36.188) Content-Type: text/plain; charset="utf-8" Remove the superfluous "find_". Signed-off-by: Baokun Li Reviewed-by: Ojaswin Mujoo Reviewed-by: Jan Kara Reviewed-by: Zhang Yi --- fs/ext4/ext4.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fs/ext4/ext4.h b/fs/ext4/ext4.h index 4ebc665cf871..0379f2974252 100644 --- a/fs/ext4/ext4.h +++ b/fs/ext4/ext4.h @@ -157,7 +157,7 @@ enum criteria { =20 /* * Reads each block group sequentially, performing disk IO if - * necessary, to find find_suitable block group. Tries to + * necessary, to find suitable block group. Tries to * allocate goal length but might trim the request if nothing * is found after enough tries. */ --=20 2.46.1 From nobody Tue Oct 7 05:23:48 2025 Received: from szxga01-in.huawei.com (szxga01-in.huawei.com [45.249.212.187]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A9CAA2561AA; Mon, 14 Jul 2025 13:18:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=45.249.212.187 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752499121; cv=none; b=Lwd0Wj2wZlDiqVG0UvuL+iAUMXBYrxlqybtUkjBwBnHnrI3WIp33B2owrLeAamazzHB7qe5V799eIQWJKYY7unojBHTXBLc2Gc+m9PwJ6zK36dhYHNM9u4h6fFsoRqoy3PLQE6vnkIACZP5TDau960mGX4vLdMHbcK83xR/nAHw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752499121; c=relaxed/simple; bh=yISilfdq1aynH/B0nTo1wGYPwP+08qT1V257vVEa1HI=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=CUi668vzODzWldZJscVj9FZJEuIMK+EZjxNGWc25QQTsSlfiDgdR2QFG0uaJTlibKEgWC3uR5HnSm7qSe0MIWNDfy0MEib3uu7nBTvIv6BfnUUM6rhl961mYF7/G2duZ0HVleAVWJLCK3xaApMFNxP8ZjzrvTthtkChoXYtLr8A= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com; spf=pass smtp.mailfrom=huawei.com; arc=none smtp.client-ip=45.249.212.187 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=huawei.com Received: from mail.maildlp.com (unknown [172.19.162.254]) by szxga01-in.huawei.com (SkyGuard) with ESMTP id 4bgjSZ4KPmz14M0P; Mon, 14 Jul 2025 21:13:50 +0800 (CST) Received: from dggpemf500013.china.huawei.com (unknown [7.185.36.188]) by mail.maildlp.com (Postfix) with ESMTPS id 8C09D180450; Mon, 14 Jul 2025 21:18:36 +0800 (CST) Received: from huawei.com (10.175.112.188) by dggpemf500013.china.huawei.com (7.185.36.188) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.11; Mon, 14 Jul 2025 21:18:35 +0800 From: Baokun Li To: CC: , , , , , , , , , Subject: [PATCH v3 08/17] ext4: convert sbi->s_mb_free_pending to atomic_t Date: Mon, 14 Jul 2025 21:03:18 +0800 Message-ID: <20250714130327.1830534-9-libaokun1@huawei.com> X-Mailer: git-send-email 2.46.1 In-Reply-To: <20250714130327.1830534-1-libaokun1@huawei.com> References: <20250714130327.1830534-1-libaokun1@huawei.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: kwepems500001.china.huawei.com (7.221.188.70) To dggpemf500013.china.huawei.com (7.185.36.188) Content-Type: text/plain; charset="utf-8" Previously, s_md_lock was used to protect s_mb_free_pending during modifications, while smp_mb() ensured fresh reads, so s_md_lock just guarantees the atomicity of s_mb_free_pending. Thus we optimized it by converting s_mb_free_pending into an atomic variable, thereby eliminating s_md_lock and minimizing lock contention. This also prepares for future lockless merging of free extents. Following this modification, s_md_lock is exclusively responsible for managing insertions and deletions within s_freed_data_list, along with operations involving list_splice. Performance test data follows: Test: Running will-it-scale/fallocate2 on CPU-bound containers. Observation: Average fallocate operations per container per second. |CPU: Kunpeng 920 | P80 | P1 | |Memory: 512GB |------------------------|-------------------------| |960GB SSD (0.5GB/s)| base | patched | base | patched | |-------------------|-------|----------------|--------|----------------| |mb_optimize_scan=3D0 | 19628 | 20043 (+2.1%) | 320885 | 314331 (-2.0%) | |mb_optimize_scan=3D1 | 7129 | 7290 (+2.2%) | 321275 | 324226 (+0.9%) | |CPU: AMD 9654 * 2 | P96 | P1 | |Memory: 1536GB |------------------------|-------------------------| |960GB SSD (1GB/s) | base | patched | base | patched | |-------------------|-------|----------------|--------|----------------| |mb_optimize_scan=3D0 | 53760 | 54999 (+2.3%) | 213145 | 214380 (+0.5%) | |mb_optimize_scan=3D1 | 12716 | 13497 (+6.1%) | 215262 | 216276 (+0.4%) | Signed-off-by: Baokun Li Reviewed-by: Jan Kara Reviewed-by: Zhang Yi --- fs/ext4/balloc.c | 2 +- fs/ext4/ext4.h | 2 +- fs/ext4/mballoc.c | 9 +++------ 3 files changed, 5 insertions(+), 8 deletions(-) diff --git a/fs/ext4/balloc.c b/fs/ext4/balloc.c index c48fd36b2d74..c9329ed5c094 100644 --- a/fs/ext4/balloc.c +++ b/fs/ext4/balloc.c @@ -703,7 +703,7 @@ int ext4_should_retry_alloc(struct super_block *sb, int= *retries) * possible we just missed a transaction commit that did so */ smp_mb(); - if (sbi->s_mb_free_pending =3D=3D 0) { + if (atomic_read(&sbi->s_mb_free_pending) =3D=3D 0) { if (test_opt(sb, DISCARD)) { atomic_inc(&sbi->s_retry_alloc_pending); flush_work(&sbi->s_discard_work); diff --git a/fs/ext4/ext4.h b/fs/ext4/ext4.h index 0379f2974252..52a72af6ec34 100644 --- a/fs/ext4/ext4.h +++ b/fs/ext4/ext4.h @@ -1602,7 +1602,7 @@ struct ext4_sb_info { unsigned short *s_mb_offsets; unsigned int *s_mb_maxs; unsigned int s_group_info_size; - unsigned int s_mb_free_pending; + atomic_t s_mb_free_pending; struct list_head s_freed_data_list[2]; /* List of blocks to be freed after commit completed */ struct list_head s_discard_list; diff --git a/fs/ext4/mballoc.c b/fs/ext4/mballoc.c index b6aa24b48543..ba3cdacbc9f9 100644 --- a/fs/ext4/mballoc.c +++ b/fs/ext4/mballoc.c @@ -3687,7 +3687,7 @@ int ext4_mb_init(struct super_block *sb) } =20 spin_lock_init(&sbi->s_md_lock); - sbi->s_mb_free_pending =3D 0; + atomic_set(&sbi->s_mb_free_pending, 0); INIT_LIST_HEAD(&sbi->s_freed_data_list[0]); INIT_LIST_HEAD(&sbi->s_freed_data_list[1]); INIT_LIST_HEAD(&sbi->s_discard_list); @@ -3903,10 +3903,7 @@ static void ext4_free_data_in_buddy(struct super_blo= ck *sb, /* we expect to find existing buddy because it's pinned */ BUG_ON(err !=3D 0); =20 - spin_lock(&EXT4_SB(sb)->s_md_lock); - EXT4_SB(sb)->s_mb_free_pending -=3D entry->efd_count; - spin_unlock(&EXT4_SB(sb)->s_md_lock); - + atomic_sub(entry->efd_count, &EXT4_SB(sb)->s_mb_free_pending); db =3D e4b.bd_info; /* there are blocks to put in buddy to make them really free */ count +=3D entry->efd_count; @@ -6401,7 +6398,7 @@ ext4_mb_free_metadata(handle_t *handle, struct ext4_b= uddy *e4b, =20 spin_lock(&sbi->s_md_lock); list_add_tail(&new_entry->efd_list, &sbi->s_freed_data_list[new_entry->ef= d_tid & 1]); - sbi->s_mb_free_pending +=3D clusters; + atomic_add(clusters, &sbi->s_mb_free_pending); spin_unlock(&sbi->s_md_lock); } =20 --=20 2.46.1 From nobody Tue Oct 7 05:23:49 2025 Received: from szxga05-in.huawei.com (szxga05-in.huawei.com [45.249.212.191]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7193A2571D7; Mon, 14 Jul 2025 13:18:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=45.249.212.191 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752499122; cv=none; b=IQND6MEO38PFChAZRgvTgk5jseUKanu7iWlSLUwYjSV1Nxs+5CMtKAA0QuIXZL4oh8YLE0TcGoF8/Dx2s+C3xolnPnWdRTjMNy/VX0ViGmKA6UBFiHOlMPtI9An9NLHY1Ep38L5FKJAgn2NDd38Ip5M2DxudurV7NHNfWijqhpk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752499122; c=relaxed/simple; bh=Zyb/YD7Y8CYqsp1QQt7pJhd5nqZD3H4+r0rysYiRtNI=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=qVx+fdGafOiYtFNQo115vgPk4KiG987Y0hYSWRs/XyQddP4JUO33YE8v0t5XXCS3EZRQf56BE1N00smGBUKv3VcoDGD9eSE81Ye44s2Mp4YxFB0d06i93PTSwtXLOxbmCqbC9cwf5CQgk+M4Ae7ory1Wtlhpdm/AYwXQIelwVCU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com; spf=pass smtp.mailfrom=huawei.com; arc=none smtp.client-ip=45.249.212.191 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=huawei.com Received: from mail.maildlp.com (unknown [172.19.162.112]) by szxga05-in.huawei.com (SkyGuard) with ESMTP id 4bgjW50Y3Pz29dns; Mon, 14 Jul 2025 21:16:01 +0800 (CST) Received: from dggpemf500013.china.huawei.com (unknown [7.185.36.188]) by mail.maildlp.com (Postfix) with ESMTPS id 93C3B140230; Mon, 14 Jul 2025 21:18:37 +0800 (CST) Received: from huawei.com (10.175.112.188) by dggpemf500013.china.huawei.com (7.185.36.188) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.11; Mon, 14 Jul 2025 21:18:36 +0800 From: Baokun Li To: CC: , , , , , , , , , Subject: [PATCH v3 09/17] ext4: merge freed extent with existing extents before insertion Date: Mon, 14 Jul 2025 21:03:19 +0800 Message-ID: <20250714130327.1830534-10-libaokun1@huawei.com> X-Mailer: git-send-email 2.46.1 In-Reply-To: <20250714130327.1830534-1-libaokun1@huawei.com> References: <20250714130327.1830534-1-libaokun1@huawei.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: kwepems500001.china.huawei.com (7.221.188.70) To dggpemf500013.china.huawei.com (7.185.36.188) Content-Type: text/plain; charset="utf-8" Attempt to merge ext4_free_data with already inserted free extents prior to adding new ones. This strategy drastically cuts down the number of times locks are held. For example, if prev, new, and next extents are all mergeable, the existing code (before this patch) requires acquiring the s_md_lock three times: prev merge into new and free prev // hold lock next merge into new and free next // hold lock insert new // hold lock After the patch, it only needs to be acquired once: new merge into next and free new // no lock next merge into prev and free next // hold lock Performance test data follows: Test: Running will-it-scale/fallocate2 on CPU-bound containers. Observation: Average fallocate operations per container per second. |CPU: Kunpeng 920 | P80 | P1 | |Memory: 512GB |------------------------|-------------------------| |960GB SSD (0.5GB/s)| base | patched | base | patched | |-------------------|-------|----------------|--------|----------------| |mb_optimize_scan=3D0 | 20043 | 20097 (+0.2%) | 314331 | 316141 (+0.5%) | |mb_optimize_scan=3D1 | 7290 | 13318 (+87.4%) | 324226 | 325273 (+0.3%) | |CPU: AMD 9654 * 2 | P96 | P1 | |Memory: 1536GB |------------------------|-------------------------| |960GB SSD (1GB/s) | base | patched | base | patched | |-------------------|-------|----------------|--------|----------------| |mb_optimize_scan=3D0 | 54999 | 53603 (-2.5%) | 214380 | 214243 (-0.06%)| |mb_optimize_scan=3D1 | 13497 | 20887 (+54.6%) | 216276 | 213632 (-1.2%) | Signed-off-by: Baokun Li Reviewed-by: Jan Kara Reviewed-by: Zhang Yi --- fs/ext4/mballoc.c | 113 +++++++++++++++++++++++++++++++--------------- 1 file changed, 76 insertions(+), 37 deletions(-) diff --git a/fs/ext4/mballoc.c b/fs/ext4/mballoc.c index ba3cdacbc9f9..6d98f2a5afc4 100644 --- a/fs/ext4/mballoc.c +++ b/fs/ext4/mballoc.c @@ -6307,28 +6307,63 @@ ext4_fsblk_t ext4_mb_new_blocks(handle_t *handle, * are contiguous, AND the extents were freed by the same transaction, * AND the blocks are associated with the same group. */ -static void ext4_try_merge_freed_extent(struct ext4_sb_info *sbi, - struct ext4_free_data *entry, - struct ext4_free_data *new_entry, - struct rb_root *entry_rb_root) +static inline bool +ext4_freed_extents_can_be_merged(struct ext4_free_data *entry1, + struct ext4_free_data *entry2) { - if ((entry->efd_tid !=3D new_entry->efd_tid) || - (entry->efd_group !=3D new_entry->efd_group)) - return; - if (entry->efd_start_cluster + entry->efd_count =3D=3D - new_entry->efd_start_cluster) { - new_entry->efd_start_cluster =3D entry->efd_start_cluster; - new_entry->efd_count +=3D entry->efd_count; - } else if (new_entry->efd_start_cluster + new_entry->efd_count =3D=3D - entry->efd_start_cluster) { - new_entry->efd_count +=3D entry->efd_count; - } else - return; + if (entry1->efd_tid !=3D entry2->efd_tid) + return false; + if (entry1->efd_start_cluster + entry1->efd_count !=3D + entry2->efd_start_cluster) + return false; + if (WARN_ON_ONCE(entry1->efd_group !=3D entry2->efd_group)) + return false; + return true; +} + +static inline void +ext4_merge_freed_extents(struct ext4_sb_info *sbi, struct rb_root *root, + struct ext4_free_data *entry1, + struct ext4_free_data *entry2) +{ + entry1->efd_count +=3D entry2->efd_count; spin_lock(&sbi->s_md_lock); - list_del(&entry->efd_list); + list_del(&entry2->efd_list); spin_unlock(&sbi->s_md_lock); - rb_erase(&entry->efd_node, entry_rb_root); - kmem_cache_free(ext4_free_data_cachep, entry); + rb_erase(&entry2->efd_node, root); + kmem_cache_free(ext4_free_data_cachep, entry2); +} + +static inline void +ext4_try_merge_freed_extent_prev(struct ext4_sb_info *sbi, struct rb_root = *root, + struct ext4_free_data *entry) +{ + struct ext4_free_data *prev; + struct rb_node *node; + + node =3D rb_prev(&entry->efd_node); + if (!node) + return; + + prev =3D rb_entry(node, struct ext4_free_data, efd_node); + if (ext4_freed_extents_can_be_merged(prev, entry)) + ext4_merge_freed_extents(sbi, root, prev, entry); +} + +static inline void +ext4_try_merge_freed_extent_next(struct ext4_sb_info *sbi, struct rb_root = *root, + struct ext4_free_data *entry) +{ + struct ext4_free_data *next; + struct rb_node *node; + + node =3D rb_next(&entry->efd_node); + if (!node) + return; + + next =3D rb_entry(node, struct ext4_free_data, efd_node); + if (ext4_freed_extents_can_be_merged(entry, next)) + ext4_merge_freed_extents(sbi, root, entry, next); } =20 static noinline_for_stack void @@ -6338,11 +6373,12 @@ ext4_mb_free_metadata(handle_t *handle, struct ext4= _buddy *e4b, ext4_group_t group =3D e4b->bd_group; ext4_grpblk_t cluster; ext4_grpblk_t clusters =3D new_entry->efd_count; - struct ext4_free_data *entry; + struct ext4_free_data *entry =3D NULL; struct ext4_group_info *db =3D e4b->bd_info; struct super_block *sb =3D e4b->bd_sb; struct ext4_sb_info *sbi =3D EXT4_SB(sb); - struct rb_node **n =3D &db->bb_free_root.rb_node, *node; + struct rb_root *root =3D &db->bb_free_root; + struct rb_node **n =3D &root->rb_node; struct rb_node *parent =3D NULL, *new_node; =20 BUG_ON(!ext4_handle_valid(handle)); @@ -6378,27 +6414,30 @@ ext4_mb_free_metadata(handle_t *handle, struct ext4= _buddy *e4b, } } =20 - rb_link_node(new_node, parent, n); - rb_insert_color(new_node, &db->bb_free_root); - - /* Now try to see the extent can be merged to left and right */ - node =3D rb_prev(new_node); - if (node) { - entry =3D rb_entry(node, struct ext4_free_data, efd_node); - ext4_try_merge_freed_extent(sbi, entry, new_entry, - &(db->bb_free_root)); + atomic_add(clusters, &sbi->s_mb_free_pending); + if (!entry) + goto insert; + + /* Now try to see the extent can be merged to prev and next */ + if (ext4_freed_extents_can_be_merged(new_entry, entry)) { + entry->efd_start_cluster =3D cluster; + entry->efd_count +=3D new_entry->efd_count; + kmem_cache_free(ext4_free_data_cachep, new_entry); + ext4_try_merge_freed_extent_prev(sbi, root, entry); + return; } - - node =3D rb_next(new_node); - if (node) { - entry =3D rb_entry(node, struct ext4_free_data, efd_node); - ext4_try_merge_freed_extent(sbi, entry, new_entry, - &(db->bb_free_root)); + if (ext4_freed_extents_can_be_merged(entry, new_entry)) { + entry->efd_count +=3D new_entry->efd_count; + kmem_cache_free(ext4_free_data_cachep, new_entry); + ext4_try_merge_freed_extent_next(sbi, root, entry); + return; } +insert: + rb_link_node(new_node, parent, n); + rb_insert_color(new_node, root); =20 spin_lock(&sbi->s_md_lock); list_add_tail(&new_entry->efd_list, &sbi->s_freed_data_list[new_entry->ef= d_tid & 1]); - atomic_add(clusters, &sbi->s_mb_free_pending); spin_unlock(&sbi->s_md_lock); } =20 --=20 2.46.1 From nobody Tue Oct 7 05:23:49 2025 Received: from szxga04-in.huawei.com (szxga04-in.huawei.com [45.249.212.190]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 03D1A256C87; Mon, 14 Jul 2025 13:18:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=45.249.212.190 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752499123; cv=none; b=RQnaCcml3WqPi1qR/cRfxXhguLNDhFO9K3BwMK7ibp2pRpvF7pafwm7ogUj1G9Humnuz9qGlltk8sLlCcmsnnolN2ssteYbgJU98bYuyZb/pVA04gh7AQxdf29Ex69aal1uXG+zxULlo+ga45Wz8tAhMYogiJh4zGVimF1fKRUI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752499123; c=relaxed/simple; bh=IUW4gaV1KDTb5jP0+YAOyulP1v3r3Bqb+J9SVwIt7oc=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=NfuDGlBBIKDpXvTf9YF7U7QdwKrlZuQrOrW+dsyKZ0YRXmn3DyB43SQzccJrKfTuifwp7MFVLkbhXXa/kxrok9wQ5PgdWdbuPXQ5h3CETjyVl30kRAfQk4EK8azmjKniV4zK8VyS/qS9ytHem2zk+/lvJU15PaaTF1Jopef8tmQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com; spf=pass smtp.mailfrom=huawei.com; arc=none smtp.client-ip=45.249.212.190 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=huawei.com Received: from mail.maildlp.com (unknown [172.19.163.44]) by szxga04-in.huawei.com (SkyGuard) with ESMTP id 4bgjWm52Cgz2TT0R; Mon, 14 Jul 2025 21:16:36 +0800 (CST) Received: from dggpemf500013.china.huawei.com (unknown [7.185.36.188]) by mail.maildlp.com (Postfix) with ESMTPS id 9DC78140135; Mon, 14 Jul 2025 21:18:38 +0800 (CST) Received: from huawei.com (10.175.112.188) by dggpemf500013.china.huawei.com (7.185.36.188) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.11; Mon, 14 Jul 2025 21:18:37 +0800 From: Baokun Li To: CC: , , , , , , , , , , Subject: [PATCH v3 10/17] ext4: fix zombie groups in average fragment size lists Date: Mon, 14 Jul 2025 21:03:20 +0800 Message-ID: <20250714130327.1830534-11-libaokun1@huawei.com> X-Mailer: git-send-email 2.46.1 In-Reply-To: <20250714130327.1830534-1-libaokun1@huawei.com> References: <20250714130327.1830534-1-libaokun1@huawei.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: kwepems500001.china.huawei.com (7.221.188.70) To dggpemf500013.china.huawei.com (7.185.36.188) Content-Type: text/plain; charset="utf-8" Groups with no free blocks shouldn't be in any average fragment size list. However, when all blocks in a group are allocated(i.e., bb_fragments or bb_free is 0), we currently skip updating the average fragment size, which means the group isn't removed from its previous s_mb_avg_fragment_size[old] list. This created "zombie" groups that were always skipped during traversal as they couldn't satisfy any block allocation requests, negatively impacting traversal efficiency. Therefore, when a group becomes completely full, bb_avg_fragment_size_order is now set to -1. If the old order was not -1, a removal operation is performed; if the new order is not -1, an insertion is performed. Fixes: 196e402adf2e ("ext4: improve cr 0 / cr 1 group scanning") CC: stable@vger.kernel.org Signed-off-by: Baokun Li Reviewed-by: Jan Kara Reviewed-by: Zhang Yi --- fs/ext4/mballoc.c | 36 ++++++++++++++++++------------------ 1 file changed, 18 insertions(+), 18 deletions(-) diff --git a/fs/ext4/mballoc.c b/fs/ext4/mballoc.c index 6d98f2a5afc4..72b20fc52bbf 100644 --- a/fs/ext4/mballoc.c +++ b/fs/ext4/mballoc.c @@ -841,30 +841,30 @@ static void mb_update_avg_fragment_size(struct super_block *sb, struct ext4_group_info= *grp) { struct ext4_sb_info *sbi =3D EXT4_SB(sb); - int new_order; + int new, old; =20 - if (!test_opt2(sb, MB_OPTIMIZE_SCAN) || grp->bb_fragments =3D=3D 0) + if (!test_opt2(sb, MB_OPTIMIZE_SCAN)) return; =20 - new_order =3D mb_avg_fragment_size_order(sb, - grp->bb_free / grp->bb_fragments); - if (new_order =3D=3D grp->bb_avg_fragment_size_order) + old =3D grp->bb_avg_fragment_size_order; + new =3D grp->bb_fragments =3D=3D 0 ? -1 : + mb_avg_fragment_size_order(sb, grp->bb_free / grp->bb_fragments); + if (new =3D=3D old) return; =20 - if (grp->bb_avg_fragment_size_order !=3D -1) { - write_lock(&sbi->s_mb_avg_fragment_size_locks[ - grp->bb_avg_fragment_size_order]); + if (old >=3D 0) { + write_lock(&sbi->s_mb_avg_fragment_size_locks[old]); list_del(&grp->bb_avg_fragment_size_node); - write_unlock(&sbi->s_mb_avg_fragment_size_locks[ - grp->bb_avg_fragment_size_order]); - } - grp->bb_avg_fragment_size_order =3D new_order; - write_lock(&sbi->s_mb_avg_fragment_size_locks[ - grp->bb_avg_fragment_size_order]); - list_add_tail(&grp->bb_avg_fragment_size_node, - &sbi->s_mb_avg_fragment_size[grp->bb_avg_fragment_size_order]); - write_unlock(&sbi->s_mb_avg_fragment_size_locks[ - grp->bb_avg_fragment_size_order]); + write_unlock(&sbi->s_mb_avg_fragment_size_locks[old]); + } + + grp->bb_avg_fragment_size_order =3D new; + if (new >=3D 0) { + write_lock(&sbi->s_mb_avg_fragment_size_locks[new]); + list_add_tail(&grp->bb_avg_fragment_size_node, + &sbi->s_mb_avg_fragment_size[new]); + write_unlock(&sbi->s_mb_avg_fragment_size_locks[new]); + } } =20 /* --=20 2.46.1 From nobody Tue Oct 7 05:23:49 2025 Received: from szxga03-in.huawei.com (szxga03-in.huawei.com [45.249.212.189]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5FB10259C94; Mon, 14 Jul 2025 13:18:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=45.249.212.189 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752499124; cv=none; b=OKnnzhWc/hV6BdatYYBIEZYORlWhjz1kyjaubrleLHMxRFQ7a335sMWKdJUk9+9KCjRh8PQBqjKoKxv+pCEF5QpItnGpYk9Uh7nafE04zhidAUHEPSV7sujB/V2zJgsq4Ns0GjgcPgYzVlFA25xSi3XnuznS/DkTAc55qb2RQ2I= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752499124; c=relaxed/simple; bh=o+D8a/79NxtbMn5ghc10dsgUJUbi2CKg8I/NGszDWXU=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=JGEoViarbDA39jSrn+3vz0fuWniqpZ0dv8QSzlo1yk2h5CJbjfZcDjXBb+SL9Vevki9PCdXTqPLImTYVlowI0kiR+HsL5Y7TPh+EotoBR3NyWsM4ZGQGhIIO8lKCt+zdMKNcqtjWbFzOAD1HbKAYYgNxWus5lvOQUQXMDLXNyBw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com; spf=pass smtp.mailfrom=huawei.com; arc=none smtp.client-ip=45.249.212.189 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=huawei.com Received: from mail.maildlp.com (unknown [172.19.88.194]) by szxga03-in.huawei.com (SkyGuard) with ESMTP id 4bgjTP21YvzHrVM; Mon, 14 Jul 2025 21:14:33 +0800 (CST) Received: from dggpemf500013.china.huawei.com (unknown [7.185.36.188]) by mail.maildlp.com (Postfix) with ESMTPS id AC1B11402C7; Mon, 14 Jul 2025 21:18:39 +0800 (CST) Received: from huawei.com (10.175.112.188) by dggpemf500013.china.huawei.com (7.185.36.188) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.11; Mon, 14 Jul 2025 21:18:38 +0800 From: Baokun Li To: CC: , , , , , , , , , , Subject: [PATCH v3 11/17] ext4: fix largest free orders lists corruption on mb_optimize_scan switch Date: Mon, 14 Jul 2025 21:03:21 +0800 Message-ID: <20250714130327.1830534-12-libaokun1@huawei.com> X-Mailer: git-send-email 2.46.1 In-Reply-To: <20250714130327.1830534-1-libaokun1@huawei.com> References: <20250714130327.1830534-1-libaokun1@huawei.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: kwepems500001.china.huawei.com (7.221.188.70) To dggpemf500013.china.huawei.com (7.185.36.188) Content-Type: text/plain; charset="utf-8" The grp->bb_largest_free_order is updated regardless of whether mb_optimize_scan is enabled. This can lead to inconsistencies between grp->bb_largest_free_order and the actual s_mb_largest_free_orders list index when mb_optimize_scan is repeatedly enabled and disabled via remount. For example, if mb_optimize_scan is initially enabled, largest free order is 3, and the group is in s_mb_largest_free_orders[3]. Then, mb_optimize_scan is disabled via remount, block allocations occur, updating largest free order to 2. Finally, mb_optimize_scan is re-enabled via remount, more block allocations update largest free order to 1. At this point, the group would be removed from s_mb_largest_free_orders[3] under the protection of s_mb_largest_free_orders_locks[2]. This lock mismatch can lead to list corruption. To fix this, whenever grp->bb_largest_free_order changes, we now always attempt to remove the group from its old order list. However, we only insert the group into the new order list if `mb_optimize_scan` is enabled. This approach helps prevent lock inconsistencies and ensures the data in the order lists remains reliable. Fixes: 196e402adf2e ("ext4: improve cr 0 / cr 1 group scanning") CC: stable@vger.kernel.org Suggested-by: Jan Kara Signed-off-by: Baokun Li Reviewed-by: Zhang Yi --- fs/ext4/mballoc.c | 33 ++++++++++++++------------------- 1 file changed, 14 insertions(+), 19 deletions(-) diff --git a/fs/ext4/mballoc.c b/fs/ext4/mballoc.c index 72b20fc52bbf..fada0d1b3fdb 100644 --- a/fs/ext4/mballoc.c +++ b/fs/ext4/mballoc.c @@ -1152,33 +1152,28 @@ static void mb_set_largest_free_order(struct super_block *sb, struct ext4_group_info *= grp) { struct ext4_sb_info *sbi =3D EXT4_SB(sb); - int i; + int new, old =3D grp->bb_largest_free_order; =20 - for (i =3D MB_NUM_ORDERS(sb) - 1; i >=3D 0; i--) - if (grp->bb_counters[i] > 0) + for (new =3D MB_NUM_ORDERS(sb) - 1; new >=3D 0; new--) + if (grp->bb_counters[new] > 0) break; + /* No need to move between order lists? */ - if (!test_opt2(sb, MB_OPTIMIZE_SCAN) || - i =3D=3D grp->bb_largest_free_order) { - grp->bb_largest_free_order =3D i; + if (new =3D=3D old) return; - } =20 - if (grp->bb_largest_free_order >=3D 0) { - write_lock(&sbi->s_mb_largest_free_orders_locks[ - grp->bb_largest_free_order]); + if (old >=3D 0 && !list_empty(&grp->bb_largest_free_order_node)) { + write_lock(&sbi->s_mb_largest_free_orders_locks[old]); list_del_init(&grp->bb_largest_free_order_node); - write_unlock(&sbi->s_mb_largest_free_orders_locks[ - grp->bb_largest_free_order]); + write_unlock(&sbi->s_mb_largest_free_orders_locks[old]); } - grp->bb_largest_free_order =3D i; - if (grp->bb_largest_free_order >=3D 0 && grp->bb_free) { - write_lock(&sbi->s_mb_largest_free_orders_locks[ - grp->bb_largest_free_order]); + + grp->bb_largest_free_order =3D new; + if (test_opt2(sb, MB_OPTIMIZE_SCAN) && new >=3D 0 && grp->bb_free) { + write_lock(&sbi->s_mb_largest_free_orders_locks[new]); list_add_tail(&grp->bb_largest_free_order_node, - &sbi->s_mb_largest_free_orders[grp->bb_largest_free_order]); - write_unlock(&sbi->s_mb_largest_free_orders_locks[ - grp->bb_largest_free_order]); + &sbi->s_mb_largest_free_orders[new]); + write_unlock(&sbi->s_mb_largest_free_orders_locks[new]); } } =20 --=20 2.46.1 From nobody Tue Oct 7 05:23:49 2025 Received: from szxga01-in.huawei.com (szxga01-in.huawei.com [45.249.212.187]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 04C1125C708; Mon, 14 Jul 2025 13:18:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=45.249.212.187 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752499128; cv=none; b=M2UJX6KILGhYraSMC6lKnbBAUg7y5t+4hPuwRM3FPq24pIR97HASkxsI/XPJME5XIym0ZcQkBHXuYKjbN9CCRmkXopiZnDQQfmYzsF5r6KOiwYvlDdDc7/gfbuMsr6/cqx1Pnz9DwluRZWg8k9cFhDBdJL1vBbc7R7hGV42Ovvk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752499128; c=relaxed/simple; bh=dFK9KXWwf7d+4+fDMAhnq78iN8DJwhZ4NLC/oTTDzYE=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=AeQ5MQ38hdfSNrrtHdGjAHEw5HFysSYKMCGSGpMstxGxgomGi2VLFl9E8oMbJf97o3SUpRWdgQDSb5dQ5jRJY2OybS2UzGll7IP42OzQsUxeadn9W2W5B7gMdNTx5r3tQaBssGyCSS52SysBML8iROn9G5ZJ0/NFB4HDY6bTZkg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com; spf=pass smtp.mailfrom=huawei.com; arc=none smtp.client-ip=45.249.212.187 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=huawei.com Received: from mail.maildlp.com (unknown [172.19.163.48]) by szxga01-in.huawei.com (SkyGuard) with ESMTP id 4bgjSf5RxGz14M0X; Mon, 14 Jul 2025 21:13:54 +0800 (CST) Received: from dggpemf500013.china.huawei.com (unknown [7.185.36.188]) by mail.maildlp.com (Postfix) with ESMTPS id B3DC218006C; Mon, 14 Jul 2025 21:18:40 +0800 (CST) Received: from huawei.com (10.175.112.188) by dggpemf500013.china.huawei.com (7.185.36.188) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.11; Mon, 14 Jul 2025 21:18:39 +0800 From: Baokun Li To: CC: , , , , , , , , , Subject: [PATCH v3 12/17] ext4: factor out __ext4_mb_scan_group() Date: Mon, 14 Jul 2025 21:03:22 +0800 Message-ID: <20250714130327.1830534-13-libaokun1@huawei.com> X-Mailer: git-send-email 2.46.1 In-Reply-To: <20250714130327.1830534-1-libaokun1@huawei.com> References: <20250714130327.1830534-1-libaokun1@huawei.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: kwepems500001.china.huawei.com (7.221.188.70) To dggpemf500013.china.huawei.com (7.185.36.188) Content-Type: text/plain; charset="utf-8" Extract __ext4_mb_scan_group() to make the code clearer and to prepare for the later conversion of 'choose group' to 'scan groups'. No functional changes. Signed-off-by: Baokun Li Reviewed-by: Zhang Yi --- fs/ext4/mballoc.c | 45 +++++++++++++++++++++++++++------------------ fs/ext4/mballoc.h | 2 ++ 2 files changed, 29 insertions(+), 18 deletions(-) diff --git a/fs/ext4/mballoc.c b/fs/ext4/mballoc.c index fada0d1b3fdb..650eb6366eb0 100644 --- a/fs/ext4/mballoc.c +++ b/fs/ext4/mballoc.c @@ -2568,6 +2568,30 @@ void ext4_mb_scan_aligned(struct ext4_allocation_con= text *ac, } } =20 +static void __ext4_mb_scan_group(struct ext4_allocation_context *ac) +{ + bool is_stripe_aligned; + struct ext4_sb_info *sbi; + enum criteria cr =3D ac->ac_criteria; + + ac->ac_groups_scanned++; + if (cr =3D=3D CR_POWER2_ALIGNED) + return ext4_mb_simple_scan_group(ac, ac->ac_e4b); + + sbi =3D EXT4_SB(ac->ac_sb); + is_stripe_aligned =3D false; + if ((sbi->s_stripe >=3D sbi->s_cluster_ratio) && + !(ac->ac_g_ex.fe_len % EXT4_NUM_B2C(sbi, sbi->s_stripe))) + is_stripe_aligned =3D true; + + if ((cr =3D=3D CR_GOAL_LEN_FAST || cr =3D=3D CR_BEST_AVAIL_LEN) && + is_stripe_aligned) + ext4_mb_scan_aligned(ac, ac->ac_e4b); + + if (ac->ac_status =3D=3D AC_STATUS_CONTINUE) + ext4_mb_complex_scan_group(ac, ac->ac_e4b); +} + /* * This is also called BEFORE we load the buddy bitmap. * Returns either 1 or 0 indicating that the group is either suitable @@ -2855,6 +2879,8 @@ ext4_mb_regular_allocator(struct ext4_allocation_cont= ext *ac) */ if (ac->ac_2order) cr =3D CR_POWER2_ALIGNED; + + ac->ac_e4b =3D &e4b; repeat: for (; cr < EXT4_MB_NUM_CRS && ac->ac_status =3D=3D AC_STATUS_CONTINUE; c= r++) { ac->ac_criteria =3D cr; @@ -2932,24 +2958,7 @@ ext4_mb_regular_allocator(struct ext4_allocation_con= text *ac) continue; } =20 - ac->ac_groups_scanned++; - if (cr =3D=3D CR_POWER2_ALIGNED) - ext4_mb_simple_scan_group(ac, &e4b); - else { - bool is_stripe_aligned =3D - (sbi->s_stripe >=3D - sbi->s_cluster_ratio) && - !(ac->ac_g_ex.fe_len % - EXT4_NUM_B2C(sbi, sbi->s_stripe)); - - if ((cr =3D=3D CR_GOAL_LEN_FAST || - cr =3D=3D CR_BEST_AVAIL_LEN) && - is_stripe_aligned) - ext4_mb_scan_aligned(ac, &e4b); - - if (ac->ac_status =3D=3D AC_STATUS_CONTINUE) - ext4_mb_complex_scan_group(ac, &e4b); - } + __ext4_mb_scan_group(ac); =20 ext4_unlock_group(sb, group); ext4_mb_unload_buddy(&e4b); diff --git a/fs/ext4/mballoc.h b/fs/ext4/mballoc.h index f8280de3e882..7a60b0103e64 100644 --- a/fs/ext4/mballoc.h +++ b/fs/ext4/mballoc.h @@ -204,6 +204,8 @@ struct ext4_allocation_context { __u8 ac_2order; /* if request is to allocate 2^N blocks and * N > 0, the field stores N, otherwise 0 */ __u8 ac_op; /* operation, for history only */ + + struct ext4_buddy *ac_e4b; struct folio *ac_bitmap_folio; struct folio *ac_buddy_folio; struct ext4_prealloc_space *ac_pa; --=20 2.46.1 From nobody Tue Oct 7 05:23:49 2025 Received: from szxga01-in.huawei.com (szxga01-in.huawei.com [45.249.212.187]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3BBCA25B2E1; Mon, 14 Jul 2025 13:18:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=45.249.212.187 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752499126; cv=none; b=VHS4HaYMeMliO8jTgMvuHQTCeaHaQuyWXbBr41jjUN1zRkZJI++WzoLbqgob32qDGTyPwv5pVtmbQ2cUl5C0nmK6UT2UIqjQypnuCkfLFNziaHjONT6p91ppUCvvJ/s+gTYcI2xosfLD5Qbqfq53L46lOp7M8Xs/Wfbsk+tgyMk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752499126; c=relaxed/simple; bh=GPRJ5GoM3SwV4dyvAcG5wABuAhG1VZrIaYIKoiDktIQ=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=trs6vSpQW6k6TcAFFZH/QO1nFd27oZmiJ8InzVr/4jFtG3v5U0YMvsm2WrhVaQB09Wcpff02neDJ2nJyYpX+W678U1iW3oEt1tyzTsRc5G6ESA98hSXJvWnmmvP6KfKrH8cUGgQ7kBXg6bPOf7PnrF0ME1zoOC3gpVE14QOqEas= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com; spf=pass smtp.mailfrom=huawei.com; arc=none smtp.client-ip=45.249.212.187 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=huawei.com Received: from mail.maildlp.com (unknown [172.19.88.105]) by szxga01-in.huawei.com (SkyGuard) with ESMTP id 4bgjSg5X89z14M0b; Mon, 14 Jul 2025 21:13:55 +0800 (CST) Received: from dggpemf500013.china.huawei.com (unknown [7.185.36.188]) by mail.maildlp.com (Postfix) with ESMTPS id B75091401F2; Mon, 14 Jul 2025 21:18:41 +0800 (CST) Received: from huawei.com (10.175.112.188) by dggpemf500013.china.huawei.com (7.185.36.188) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.11; Mon, 14 Jul 2025 21:18:40 +0800 From: Baokun Li To: CC: , , , , , , , , , Subject: [PATCH v3 13/17] ext4: factor out ext4_mb_might_prefetch() Date: Mon, 14 Jul 2025 21:03:23 +0800 Message-ID: <20250714130327.1830534-14-libaokun1@huawei.com> X-Mailer: git-send-email 2.46.1 In-Reply-To: <20250714130327.1830534-1-libaokun1@huawei.com> References: <20250714130327.1830534-1-libaokun1@huawei.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: kwepems500001.china.huawei.com (7.221.188.70) To dggpemf500013.china.huawei.com (7.185.36.188) Content-Type: text/plain; charset="utf-8" Extract ext4_mb_might_prefetch() to make the code clearer and to prepare for the later conversion of 'choose group' to 'scan groups'. No functional changes. Signed-off-by: Baokun Li Reviewed-by: Zhang Yi --- fs/ext4/mballoc.c | 62 +++++++++++++++++++++++++++++------------------ fs/ext4/mballoc.h | 4 +++ 2 files changed, 42 insertions(+), 24 deletions(-) diff --git a/fs/ext4/mballoc.c b/fs/ext4/mballoc.c index 650eb6366eb0..52ec59f58c36 100644 --- a/fs/ext4/mballoc.c +++ b/fs/ext4/mballoc.c @@ -2781,6 +2781,37 @@ ext4_group_t ext4_mb_prefetch(struct super_block *sb= , ext4_group_t group, return group; } =20 +/* + * Batch reads of the block allocation bitmaps to get + * multiple READs in flight; limit prefetching at inexpensive + * CR, otherwise mballoc can spend a lot of time loading + * imperfect groups + */ +static void ext4_mb_might_prefetch(struct ext4_allocation_context *ac, + ext4_group_t group) +{ + struct ext4_sb_info *sbi; + + if (ac->ac_prefetch_grp !=3D group) + return; + + sbi =3D EXT4_SB(ac->ac_sb); + if (ext4_mb_cr_expensive(ac->ac_criteria) || + ac->ac_prefetch_ios < sbi->s_mb_prefetch_limit) { + unsigned int nr =3D sbi->s_mb_prefetch; + + if (ext4_has_feature_flex_bg(ac->ac_sb)) { + nr =3D 1 << sbi->s_log_groups_per_flex; + nr -=3D group & (nr - 1); + nr =3D umin(nr, sbi->s_mb_prefetch); + } + + ac->ac_prefetch_nr =3D nr; + ac->ac_prefetch_grp =3D ext4_mb_prefetch(ac->ac_sb, group, nr, + &ac->ac_prefetch_ios); + } +} + /* * Prefetching reads the block bitmap into the buffer cache; but we * need to make sure that the buddy bitmap in the page cache has been @@ -2817,10 +2848,9 @@ void ext4_mb_prefetch_fini(struct super_block *sb, e= xt4_group_t group, static noinline_for_stack int ext4_mb_regular_allocator(struct ext4_allocation_context *ac) { - ext4_group_t prefetch_grp =3D 0, ngroups, group, i; + ext4_group_t ngroups, group, i; enum criteria new_cr, cr =3D CR_GOAL_LEN_FAST; int err =3D 0, first_err =3D 0; - unsigned int nr =3D 0, prefetch_ios =3D 0; struct ext4_sb_info *sbi; struct super_block *sb; struct ext4_buddy e4b; @@ -2881,6 +2911,7 @@ ext4_mb_regular_allocator(struct ext4_allocation_cont= ext *ac) cr =3D CR_POWER2_ALIGNED; =20 ac->ac_e4b =3D &e4b; + ac->ac_prefetch_ios =3D 0; repeat: for (; cr < EXT4_MB_NUM_CRS && ac->ac_status =3D=3D AC_STATUS_CONTINUE; c= r++) { ac->ac_criteria =3D cr; @@ -2890,8 +2921,8 @@ ext4_mb_regular_allocator(struct ext4_allocation_cont= ext *ac) */ group =3D ac->ac_g_ex.fe_group; ac->ac_groups_linear_remaining =3D sbi->s_mb_max_linear_groups; - prefetch_grp =3D group; - nr =3D 0; + ac->ac_prefetch_grp =3D group; + ac->ac_prefetch_nr =3D 0; =20 for (i =3D 0, new_cr =3D cr; i < ngroups; i++, ext4_mb_choose_next_group(ac, &new_cr, &group, ngroups)) { @@ -2903,24 +2934,7 @@ ext4_mb_regular_allocator(struct ext4_allocation_con= text *ac) goto repeat; } =20 - /* - * Batch reads of the block allocation bitmaps - * to get multiple READs in flight; limit - * prefetching at inexpensive CR, otherwise mballoc - * can spend a lot of time loading imperfect groups - */ - if ((prefetch_grp =3D=3D group) && - (ext4_mb_cr_expensive(cr) || - prefetch_ios < sbi->s_mb_prefetch_limit)) { - nr =3D sbi->s_mb_prefetch; - if (ext4_has_feature_flex_bg(sb)) { - nr =3D 1 << sbi->s_log_groups_per_flex; - nr -=3D group & (nr - 1); - nr =3D min(nr, sbi->s_mb_prefetch); - } - prefetch_grp =3D ext4_mb_prefetch(sb, group, - nr, &prefetch_ios); - } + ext4_mb_might_prefetch(ac, group); =20 /* prevent unnecessary buddy loading. */ if (cr < CR_ANY_FREE && @@ -3018,8 +3032,8 @@ ext4_mb_regular_allocator(struct ext4_allocation_cont= ext *ac) ac->ac_b_ex.fe_len, ac->ac_o_ex.fe_len, ac->ac_status, ac->ac_flags, cr, err); =20 - if (nr) - ext4_mb_prefetch_fini(sb, prefetch_grp, nr); + if (ac->ac_prefetch_nr) + ext4_mb_prefetch_fini(sb, ac->ac_prefetch_grp, ac->ac_prefetch_nr); =20 return err; } diff --git a/fs/ext4/mballoc.h b/fs/ext4/mballoc.h index 7a60b0103e64..9f66b1d5db67 100644 --- a/fs/ext4/mballoc.h +++ b/fs/ext4/mballoc.h @@ -192,6 +192,10 @@ struct ext4_allocation_context { */ ext4_grpblk_t ac_orig_goal_len; =20 + ext4_group_t ac_prefetch_grp; + unsigned int ac_prefetch_ios; + unsigned int ac_prefetch_nr; + __u32 ac_flags; /* allocation hints */ __u32 ac_groups_linear_remaining; __u16 ac_groups_scanned; --=20 2.46.1 From nobody Tue Oct 7 05:23:49 2025 Received: from szxga04-in.huawei.com (szxga04-in.huawei.com [45.249.212.190]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 05A9625BF00; Mon, 14 Jul 2025 13:18:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=45.249.212.190 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752499126; cv=none; b=mzwtLNfwi13B/XnZfcoV+qWFO/o0sgGpU1Pq/X5HT1AycNaac7YEx2Qm4eaTPl4V5g6YqFXLH2wGZqcvzrv4BiRMCz06mDxDchucoAfQpoE2H7/PpGNL0HbRHFy7sRn1H+IOrwMd7jEklV/FqZGiyN4rajDxkJTonfeq8t5CNOc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752499126; c=relaxed/simple; bh=zqadfgTzPISl72Ux5+SNigRkg+60u4a0j4JrJnr+Nbc=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=JtiUWfhulm3Eo1CCq8nRHKxNr9ewDGjzBRAKWvi+LHwPie3kWhDdCr1rdwmPOUbd9JBoT+WEjjONboX8mRG8zB42XwQ3HcGJ/q4X5ggjPOvJb26lM7n2BLU3ImG+56wjNVU2WTVc+aA9lW4cxC9jnumMyqoZcNTFbXXDmgTHDYI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com; spf=pass smtp.mailfrom=huawei.com; arc=none smtp.client-ip=45.249.212.190 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=huawei.com Received: from mail.maildlp.com (unknown [172.19.163.44]) by szxga04-in.huawei.com (SkyGuard) with ESMTP id 4bgjWr64Gdz2TT3C; Mon, 14 Jul 2025 21:16:40 +0800 (CST) Received: from dggpemf500013.china.huawei.com (unknown [7.185.36.188]) by mail.maildlp.com (Postfix) with ESMTPS id C232B140135; Mon, 14 Jul 2025 21:18:42 +0800 (CST) Received: from huawei.com (10.175.112.188) by dggpemf500013.china.huawei.com (7.185.36.188) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.11; Mon, 14 Jul 2025 21:18:41 +0800 From: Baokun Li To: CC: , , , , , , , , , Subject: [PATCH v3 14/17] ext4: factor out ext4_mb_scan_group() Date: Mon, 14 Jul 2025 21:03:24 +0800 Message-ID: <20250714130327.1830534-15-libaokun1@huawei.com> X-Mailer: git-send-email 2.46.1 In-Reply-To: <20250714130327.1830534-1-libaokun1@huawei.com> References: <20250714130327.1830534-1-libaokun1@huawei.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: kwepems500001.china.huawei.com (7.221.188.70) To dggpemf500013.china.huawei.com (7.185.36.188) Content-Type: text/plain; charset="utf-8" Extract ext4_mb_scan_group() to make the code clearer and to prepare for the later conversion of 'choose group' to 'scan groups'. No functional changes. Signed-off-by: Baokun Li Reviewed-by: Zhang Yi --- fs/ext4/mballoc.c | 93 +++++++++++++++++++++++++---------------------- fs/ext4/mballoc.h | 2 + 2 files changed, 51 insertions(+), 44 deletions(-) diff --git a/fs/ext4/mballoc.c b/fs/ext4/mballoc.c index 52ec59f58c36..0c3cbc7e2e85 100644 --- a/fs/ext4/mballoc.c +++ b/fs/ext4/mballoc.c @@ -2845,12 +2845,56 @@ void ext4_mb_prefetch_fini(struct super_block *sb, = ext4_group_t group, } } =20 +static int ext4_mb_scan_group(struct ext4_allocation_context *ac, + ext4_group_t group) +{ + int ret; + struct super_block *sb =3D ac->ac_sb; + enum criteria cr =3D ac->ac_criteria; + + ext4_mb_might_prefetch(ac, group); + + /* prevent unnecessary buddy loading. */ + if (cr < CR_ANY_FREE && spin_is_locked(ext4_group_lock_ptr(sb, group))) + return 0; + + /* This now checks without needing the buddy page */ + ret =3D ext4_mb_good_group_nolock(ac, group, cr); + if (ret <=3D 0) { + if (!ac->ac_first_err) + ac->ac_first_err =3D ret; + return 0; + } + + ret =3D ext4_mb_load_buddy(sb, group, ac->ac_e4b); + if (ret) + return ret; + + /* skip busy group */ + if (cr >=3D CR_ANY_FREE) + ext4_lock_group(sb, group); + else if (!ext4_try_lock_group(sb, group)) + goto out_unload; + + /* We need to check again after locking the block group. */ + if (unlikely(!ext4_mb_good_group(ac, group, cr))) + goto out_unlock; + + __ext4_mb_scan_group(ac); + +out_unlock: + ext4_unlock_group(sb, group); +out_unload: + ext4_mb_unload_buddy(ac->ac_e4b); + return ret; +} + static noinline_for_stack int ext4_mb_regular_allocator(struct ext4_allocation_context *ac) { ext4_group_t ngroups, group, i; enum criteria new_cr, cr =3D CR_GOAL_LEN_FAST; - int err =3D 0, first_err =3D 0; + int err =3D 0; struct ext4_sb_info *sbi; struct super_block *sb; struct ext4_buddy e4b; @@ -2912,6 +2956,7 @@ ext4_mb_regular_allocator(struct ext4_allocation_cont= ext *ac) =20 ac->ac_e4b =3D &e4b; ac->ac_prefetch_ios =3D 0; + ac->ac_first_err =3D 0; repeat: for (; cr < EXT4_MB_NUM_CRS && ac->ac_status =3D=3D AC_STATUS_CONTINUE; c= r++) { ac->ac_criteria =3D cr; @@ -2926,7 +2971,6 @@ ext4_mb_regular_allocator(struct ext4_allocation_cont= ext *ac) =20 for (i =3D 0, new_cr =3D cr; i < ngroups; i++, ext4_mb_choose_next_group(ac, &new_cr, &group, ngroups)) { - int ret =3D 0; =20 cond_resched(); if (new_cr !=3D cr) { @@ -2934,49 +2978,10 @@ ext4_mb_regular_allocator(struct ext4_allocation_co= ntext *ac) goto repeat; } =20 - ext4_mb_might_prefetch(ac, group); - - /* prevent unnecessary buddy loading. */ - if (cr < CR_ANY_FREE && - spin_is_locked(ext4_group_lock_ptr(sb, group))) - continue; - - /* This now checks without needing the buddy page */ - ret =3D ext4_mb_good_group_nolock(ac, group, cr); - if (ret <=3D 0) { - if (!first_err) - first_err =3D ret; - continue; - } - - err =3D ext4_mb_load_buddy(sb, group, &e4b); + err =3D ext4_mb_scan_group(ac, group); if (err) goto out; =20 - /* skip busy group */ - if (cr >=3D CR_ANY_FREE) { - ext4_lock_group(sb, group); - } else if (!ext4_try_lock_group(sb, group)) { - ext4_mb_unload_buddy(&e4b); - continue; - } - - /* - * We need to check again after locking the - * block group - */ - ret =3D ext4_mb_good_group(ac, group, cr); - if (ret =3D=3D 0) { - ext4_unlock_group(sb, group); - ext4_mb_unload_buddy(&e4b); - continue; - } - - __ext4_mb_scan_group(ac); - - ext4_unlock_group(sb, group); - ext4_mb_unload_buddy(&e4b); - if (ac->ac_status !=3D AC_STATUS_CONTINUE) break; } @@ -3025,8 +3030,8 @@ ext4_mb_regular_allocator(struct ext4_allocation_cont= ext *ac) atomic_inc(&sbi->s_bal_stream_goals); } out: - if (!err && ac->ac_status !=3D AC_STATUS_FOUND && first_err) - err =3D first_err; + if (!err && ac->ac_status !=3D AC_STATUS_FOUND && ac->ac_first_err) + err =3D ac->ac_first_err; =20 mb_debug(sb, "Best len %d, origin len %d, ac_status %u, ac_flags 0x%x, cr= %d ret %d\n", ac->ac_b_ex.fe_len, ac->ac_o_ex.fe_len, ac->ac_status, diff --git a/fs/ext4/mballoc.h b/fs/ext4/mballoc.h index 9f66b1d5db67..83886fc9521b 100644 --- a/fs/ext4/mballoc.h +++ b/fs/ext4/mballoc.h @@ -196,6 +196,8 @@ struct ext4_allocation_context { unsigned int ac_prefetch_ios; unsigned int ac_prefetch_nr; =20 + int ac_first_err; + __u32 ac_flags; /* allocation hints */ __u32 ac_groups_linear_remaining; __u16 ac_groups_scanned; --=20 2.46.1 From nobody Tue Oct 7 05:23:49 2025 Received: from szxga02-in.huawei.com (szxga02-in.huawei.com [45.249.212.188]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0074225B311; Mon, 14 Jul 2025 13:18:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=45.249.212.188 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752499129; cv=none; b=Wj2w5S8jppFhSLFCitinFA8KPI7buYAtq6CYIcGZVUFwhyhRuNDvR6f0aVhdR3dUW0vXSo/OLpob28H5ZrU/YnQliLsfyhT+9oC0XDDqR4rTs9sFkJ/eMMmY/pWLPBloBKn3Bo/tOMqfTirmrmKW7sKQV3wEe4ngEVldFVUHqLA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752499129; c=relaxed/simple; bh=LrjfXJ/dqd74/iQS83AvO7Shl8OfYWxeFZgDz4/+onM=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=JkcQ0MzpouQ/qUoqZvXa0g47Z+qVL2Qi3Hy/SPMLD5eZOd9tnFEpKOpDXCC136S0G1gn6K04pnCw+NHIvOjV8amACyQqUy+2pOxm0rYCpiP2zwL/atDTP6hLz4utrvx/zsRYSpSui3WqLWG0g91om3EmUAbzLQq+ItLnTTmAB6Y= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com; spf=pass smtp.mailfrom=huawei.com; arc=none smtp.client-ip=45.249.212.188 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=huawei.com Received: from mail.maildlp.com (unknown [172.19.163.48]) by szxga02-in.huawei.com (SkyGuard) with ESMTP id 4bgjXy1jD2ztSml; Mon, 14 Jul 2025 21:17:38 +0800 (CST) Received: from dggpemf500013.china.huawei.com (unknown [7.185.36.188]) by mail.maildlp.com (Postfix) with ESMTPS id D0A0718006C; Mon, 14 Jul 2025 21:18:43 +0800 (CST) Received: from huawei.com (10.175.112.188) by dggpemf500013.china.huawei.com (7.185.36.188) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.11; Mon, 14 Jul 2025 21:18:42 +0800 From: Baokun Li To: CC: , , , , , , , , , Subject: [PATCH v3 15/17] ext4: convert free groups order lists to xarrays Date: Mon, 14 Jul 2025 21:03:25 +0800 Message-ID: <20250714130327.1830534-16-libaokun1@huawei.com> X-Mailer: git-send-email 2.46.1 In-Reply-To: <20250714130327.1830534-1-libaokun1@huawei.com> References: <20250714130327.1830534-1-libaokun1@huawei.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: kwepems500001.china.huawei.com (7.221.188.70) To dggpemf500013.china.huawei.com (7.185.36.188) Content-Type: text/plain; charset="utf-8" While traversing the list, holding a spin_lock prevents load_buddy, making direct use of ext4_try_lock_group impossible. This can lead to a bouncing scenario where spin_is_locked(grp_A) succeeds, but ext4_try_lock_group() fails, forcing the list traversal to repeatedly restart from grp_A. In contrast, linear traversal directly uses ext4_try_lock_group(), avoiding this bouncing. Therefore, we need a lockless, ordered traversal to achieve linear-like efficiency. Therefore, this commit converts both average fragment size lists and largest free order lists into ordered xarrays. In an xarray, the index represents the block group number and the value holds the block group information; a non-empty value indicates the block group's presence. While insertion and deletion complexity remain O(1), lookup complexity changes from O(1) to O(nlogn), which may slightly reduce single-threaded performance. Additionally, xarray insertions might fail, potentially due to memory allocation issues. However, since we have linear traversal as a fallback, this isn't a major problem. Therefore, we've only added a warning message for insertion failures here. A helper function ext4_mb_find_good_group_xarray() is added to find good groups in the specified xarray starting at the specified position start, and when it reaches ngroups-1, it wraps around to 0 and then to start-1. This ensures an ordered traversal within the xarray. Performance test results are as follows: Single-process operations on an empty disk show negligible impact, while multi-process workloads demonstrate a noticeable performance gain. |CPU: Kunpeng 920 | P80 | P1 | |Memory: 512GB |------------------------|-------------------------| |960GB SSD (0.5GB/s)| base | patched | base | patched | |-------------------|-------|----------------|--------|----------------| |mb_optimize_scan=3D0 | 20097 | 19555 (-2.6%) | 316141 | 315636 (-0.2%) | |mb_optimize_scan=3D1 | 13318 | 15496 (+16.3%) | 325273 | 323569 (-0.5%) | |CPU: AMD 9654 * 2 | P96 | P1 | |Memory: 1536GB |------------------------|-------------------------| |960GB SSD (1GB/s) | base | patched | base | patched | |-------------------|-------|----------------|--------|----------------| |mb_optimize_scan=3D0 | 53603 | 53192 (-0.7%) | 214243 | 212678 (-0.7%) | |mb_optimize_scan=3D1 | 20887 | 37636 (+80.1%) | 213632 | 214189 (+0.2%) | Signed-off-by: Baokun Li Reviewed-by: Zhang Yi Tested-by: Guenter Roeck --- fs/ext4/ext4.h | 8 +- fs/ext4/mballoc.c | 254 +++++++++++++++++++++++++--------------------- 2 files changed, 140 insertions(+), 122 deletions(-) diff --git a/fs/ext4/ext4.h b/fs/ext4/ext4.h index 52a72af6ec34..ea412fdb0b76 100644 --- a/fs/ext4/ext4.h +++ b/fs/ext4/ext4.h @@ -1608,10 +1608,8 @@ struct ext4_sb_info { struct list_head s_discard_list; struct work_struct s_discard_work; atomic_t s_retry_alloc_pending; - struct list_head *s_mb_avg_fragment_size; - rwlock_t *s_mb_avg_fragment_size_locks; - struct list_head *s_mb_largest_free_orders; - rwlock_t *s_mb_largest_free_orders_locks; + struct xarray *s_mb_avg_fragment_size; + struct xarray *s_mb_largest_free_orders; =20 /* tunables */ unsigned long s_stripe; @@ -3485,8 +3483,6 @@ struct ext4_group_info { void *bb_bitmap; #endif struct rw_semaphore alloc_sem; - struct list_head bb_avg_fragment_size_node; - struct list_head bb_largest_free_order_node; ext4_grpblk_t bb_counters[]; /* Nr of free power-of-two-block * regions, index is order. * bb_counters[3] =3D 5 means diff --git a/fs/ext4/mballoc.c b/fs/ext4/mballoc.c index 0c3cbc7e2e85..a9eb997b8c9b 100644 --- a/fs/ext4/mballoc.c +++ b/fs/ext4/mballoc.c @@ -132,25 +132,30 @@ * If "mb_optimize_scan" mount option is set, we maintain in memory group = info * structures in two data structures: * - * 1) Array of largest free order lists (sbi->s_mb_largest_free_orders) + * 1) Array of largest free order xarrays (sbi->s_mb_largest_free_orders) * - * Locking: sbi->s_mb_largest_free_orders_locks(array of rw locks) + * Locking: Writers use xa_lock, readers use rcu_read_lock. * - * This is an array of lists where the index in the array represents the + * This is an array of xarrays where the index in the array represents = the * largest free order in the buddy bitmap of the participating group in= fos of - * that list. So, there are exactly MB_NUM_ORDERS(sb) (which means total - * number of buddy bitmap orders possible) number of lists. Group-infos= are - * placed in appropriate lists. + * that xarray. So, there are exactly MB_NUM_ORDERS(sb) (which means to= tal + * number of buddy bitmap orders possible) number of xarrays. Group-inf= os are + * placed in appropriate xarrays. * - * 2) Average fragment size lists (sbi->s_mb_avg_fragment_size) + * 2) Average fragment size xarrays (sbi->s_mb_avg_fragment_size) * - * Locking: sbi->s_mb_avg_fragment_size_locks(array of rw locks) + * Locking: Writers use xa_lock, readers use rcu_read_lock. * - * This is an array of lists where in the i-th list there are groups wi= th + * This is an array of xarrays where in the i-th xarray there are group= s with * average fragment size >=3D 2^i and < 2^(i+1). The average fragment s= ize * is computed as ext4_group_info->bb_free / ext4_group_info->bb_fragme= nts. - * Note that we don't bother with a special list for completely empty g= roups - * so we only have MB_NUM_ORDERS(sb) lists. + * Note that we don't bother with a special xarray for completely empty + * groups so we only have MB_NUM_ORDERS(sb) xarrays. Group-infos are pl= aced + * in appropriate xarrays. + * + * In xarray, the index is the block group number, the value is the block = group + * information, and a non-empty value indicates the block group is present= in + * the current xarray. * * When "mb_optimize_scan" mount option is set, mballoc consults the above= data * structures to decide the order in which groups are to be traversed for @@ -852,21 +857,75 @@ mb_update_avg_fragment_size(struct super_block *sb, s= truct ext4_group_info *grp) if (new =3D=3D old) return; =20 - if (old >=3D 0) { - write_lock(&sbi->s_mb_avg_fragment_size_locks[old]); - list_del(&grp->bb_avg_fragment_size_node); - write_unlock(&sbi->s_mb_avg_fragment_size_locks[old]); - } + if (old >=3D 0) + xa_erase(&sbi->s_mb_avg_fragment_size[old], grp->bb_group); =20 grp->bb_avg_fragment_size_order =3D new; if (new >=3D 0) { - write_lock(&sbi->s_mb_avg_fragment_size_locks[new]); - list_add_tail(&grp->bb_avg_fragment_size_node, - &sbi->s_mb_avg_fragment_size[new]); - write_unlock(&sbi->s_mb_avg_fragment_size_locks[new]); + /* + * Cannot use __GFP_NOFAIL because we hold the group lock. + * Although allocation for insertion may fails, it's not fatal + * as we have linear traversal to fall back on. + */ + int err =3D xa_insert(&sbi->s_mb_avg_fragment_size[new], + grp->bb_group, grp, GFP_ATOMIC); + if (err) + mb_debug(sb, "insert group: %u to s_mb_avg_fragment_size[%d] failed, er= r %d", + grp->bb_group, new, err); } } =20 +static struct ext4_group_info * +ext4_mb_find_good_group_xarray(struct ext4_allocation_context *ac, + struct xarray *xa, ext4_group_t start) +{ + struct super_block *sb =3D ac->ac_sb; + struct ext4_sb_info *sbi =3D EXT4_SB(sb); + enum criteria cr =3D ac->ac_criteria; + ext4_group_t ngroups =3D ext4_get_groups_count(sb); + unsigned long group =3D start; + ext4_group_t end =3D ngroups; + struct ext4_group_info *grp; + + if (WARN_ON_ONCE(start >=3D end)) + return NULL; + +wrap_around: + xa_for_each_range(xa, group, grp, start, end - 1) { + if (sbi->s_mb_stats) + atomic64_inc(&sbi->s_bal_cX_groups_considered[cr]); + + if (!spin_is_locked(ext4_group_lock_ptr(sb, group)) && + likely(ext4_mb_good_group(ac, group, cr))) + return grp; + + cond_resched(); + } + + if (start) { + end =3D start; + start =3D 0; + goto wrap_around; + } + + return NULL; +} + +/* + * Find a suitable group of given order from the largest free orders xarra= y. + */ +static struct ext4_group_info * +ext4_mb_find_good_group_largest_free_order(struct ext4_allocation_context = *ac, + int order, ext4_group_t start) +{ + struct xarray *xa =3D &EXT4_SB(ac->ac_sb)->s_mb_largest_free_orders[order= ]; + + if (xa_empty(xa)) + return NULL; + + return ext4_mb_find_good_group_xarray(ac, xa, start); +} + /* * Choose next group by traversing largest_free_order lists. Updates *new_= cr if * cr level needs an update. @@ -875,7 +934,7 @@ static void ext4_mb_choose_next_group_p2_aligned(struct= ext4_allocation_context enum criteria *new_cr, ext4_group_t *group) { struct ext4_sb_info *sbi =3D EXT4_SB(ac->ac_sb); - struct ext4_group_info *iter; + struct ext4_group_info *grp; int i; =20 if (ac->ac_status =3D=3D AC_STATUS_FOUND) @@ -885,26 +944,12 @@ static void ext4_mb_choose_next_group_p2_aligned(stru= ct ext4_allocation_context atomic_inc(&sbi->s_bal_p2_aligned_bad_suggestions); =20 for (i =3D ac->ac_2order; i < MB_NUM_ORDERS(ac->ac_sb); i++) { - if (list_empty(&sbi->s_mb_largest_free_orders[i])) - continue; - read_lock(&sbi->s_mb_largest_free_orders_locks[i]); - if (list_empty(&sbi->s_mb_largest_free_orders[i])) { - read_unlock(&sbi->s_mb_largest_free_orders_locks[i]); - continue; - } - list_for_each_entry(iter, &sbi->s_mb_largest_free_orders[i], - bb_largest_free_order_node) { - if (sbi->s_mb_stats) - atomic64_inc(&sbi->s_bal_cX_groups_considered[CR_POWER2_ALIGNED]); - if (!spin_is_locked(ext4_group_lock_ptr(ac->ac_sb, iter->bb_group)) && - likely(ext4_mb_good_group(ac, iter->bb_group, CR_POWER2_ALIGNED))) { - *group =3D iter->bb_group; - ac->ac_flags |=3D EXT4_MB_CR_POWER2_ALIGNED_OPTIMIZED; - read_unlock(&sbi->s_mb_largest_free_orders_locks[i]); - return; - } + grp =3D ext4_mb_find_good_group_largest_free_order(ac, i, *group); + if (grp) { + *group =3D grp->bb_group; + ac->ac_flags |=3D EXT4_MB_CR_POWER2_ALIGNED_OPTIMIZED; + return; } - read_unlock(&sbi->s_mb_largest_free_orders_locks[i]); } =20 /* Increment cr and search again if no group is found */ @@ -912,35 +957,18 @@ static void ext4_mb_choose_next_group_p2_aligned(stru= ct ext4_allocation_context } =20 /* - * Find a suitable group of given order from the average fragments list. + * Find a suitable group of given order from the average fragments xarray. */ static struct ext4_group_info * -ext4_mb_find_good_group_avg_frag_lists(struct ext4_allocation_context *ac,= int order) +ext4_mb_find_good_group_avg_frag_xarray(struct ext4_allocation_context *ac, + int order, ext4_group_t start) { - struct ext4_sb_info *sbi =3D EXT4_SB(ac->ac_sb); - struct list_head *frag_list =3D &sbi->s_mb_avg_fragment_size[order]; - rwlock_t *frag_list_lock =3D &sbi->s_mb_avg_fragment_size_locks[order]; - struct ext4_group_info *grp =3D NULL, *iter; - enum criteria cr =3D ac->ac_criteria; + struct xarray *xa =3D &EXT4_SB(ac->ac_sb)->s_mb_avg_fragment_size[order]; =20 - if (list_empty(frag_list)) + if (xa_empty(xa)) return NULL; - read_lock(frag_list_lock); - if (list_empty(frag_list)) { - read_unlock(frag_list_lock); - return NULL; - } - list_for_each_entry(iter, frag_list, bb_avg_fragment_size_node) { - if (sbi->s_mb_stats) - atomic64_inc(&sbi->s_bal_cX_groups_considered[cr]); - if (!spin_is_locked(ext4_group_lock_ptr(ac->ac_sb, iter->bb_group)) && - likely(ext4_mb_good_group(ac, iter->bb_group, cr))) { - grp =3D iter; - break; - } - } - read_unlock(frag_list_lock); - return grp; + + return ext4_mb_find_good_group_xarray(ac, xa, start); } =20 /* @@ -961,7 +989,7 @@ static void ext4_mb_choose_next_group_goal_fast(struct = ext4_allocation_context * =20 for (i =3D mb_avg_fragment_size_order(ac->ac_sb, ac->ac_g_ex.fe_len); i < MB_NUM_ORDERS(ac->ac_sb); i++) { - grp =3D ext4_mb_find_good_group_avg_frag_lists(ac, i); + grp =3D ext4_mb_find_good_group_avg_frag_xarray(ac, i, *group); if (grp) { *group =3D grp->bb_group; ac->ac_flags |=3D EXT4_MB_CR_GOAL_LEN_FAST_OPTIMIZED; @@ -1057,7 +1085,8 @@ static void ext4_mb_choose_next_group_best_avail(stru= ct ext4_allocation_context frag_order =3D mb_avg_fragment_size_order(ac->ac_sb, ac->ac_g_ex.fe_len); =20 - grp =3D ext4_mb_find_good_group_avg_frag_lists(ac, frag_order); + grp =3D ext4_mb_find_good_group_avg_frag_xarray(ac, frag_order, + *group); if (grp) { *group =3D grp->bb_group; ac->ac_flags |=3D EXT4_MB_CR_BEST_AVAIL_LEN_OPTIMIZED; @@ -1162,18 +1191,25 @@ mb_set_largest_free_order(struct super_block *sb, s= truct ext4_group_info *grp) if (new =3D=3D old) return; =20 - if (old >=3D 0 && !list_empty(&grp->bb_largest_free_order_node)) { - write_lock(&sbi->s_mb_largest_free_orders_locks[old]); - list_del_init(&grp->bb_largest_free_order_node); - write_unlock(&sbi->s_mb_largest_free_orders_locks[old]); + if (old >=3D 0) { + struct xarray *xa =3D &sbi->s_mb_largest_free_orders[old]; + + if (!xa_empty(xa) && xa_load(xa, grp->bb_group)) + xa_erase(xa, grp->bb_group); } =20 grp->bb_largest_free_order =3D new; if (test_opt2(sb, MB_OPTIMIZE_SCAN) && new >=3D 0 && grp->bb_free) { - write_lock(&sbi->s_mb_largest_free_orders_locks[new]); - list_add_tail(&grp->bb_largest_free_order_node, - &sbi->s_mb_largest_free_orders[new]); - write_unlock(&sbi->s_mb_largest_free_orders_locks[new]); + /* + * Cannot use __GFP_NOFAIL because we hold the group lock. + * Although allocation for insertion may fails, it's not fatal + * as we have linear traversal to fall back on. + */ + int err =3D xa_insert(&sbi->s_mb_largest_free_orders[new], + grp->bb_group, grp, GFP_ATOMIC); + if (err) + mb_debug(sb, "insert group: %u to s_mb_largest_free_orders[%d] failed, = err %d", + grp->bb_group, new, err); } } =20 @@ -3269,6 +3305,7 @@ static int ext4_mb_seq_structs_summary_show(struct se= q_file *seq, void *v) unsigned long position =3D ((unsigned long) v); struct ext4_group_info *grp; unsigned int count; + unsigned long idx; =20 position--; if (position >=3D MB_NUM_ORDERS(sb)) { @@ -3277,11 +3314,8 @@ static int ext4_mb_seq_structs_summary_show(struct s= eq_file *seq, void *v) seq_puts(seq, "avg_fragment_size_lists:\n"); =20 count =3D 0; - read_lock(&sbi->s_mb_avg_fragment_size_locks[position]); - list_for_each_entry(grp, &sbi->s_mb_avg_fragment_size[position], - bb_avg_fragment_size_node) + xa_for_each(&sbi->s_mb_avg_fragment_size[position], idx, grp) count++; - read_unlock(&sbi->s_mb_avg_fragment_size_locks[position]); seq_printf(seq, "\tlist_order_%u_groups: %u\n", (unsigned int)position, count); return 0; @@ -3293,11 +3327,8 @@ static int ext4_mb_seq_structs_summary_show(struct s= eq_file *seq, void *v) seq_puts(seq, "max_free_order_lists:\n"); } count =3D 0; - read_lock(&sbi->s_mb_largest_free_orders_locks[position]); - list_for_each_entry(grp, &sbi->s_mb_largest_free_orders[position], - bb_largest_free_order_node) + xa_for_each(&sbi->s_mb_largest_free_orders[position], idx, grp) count++; - read_unlock(&sbi->s_mb_largest_free_orders_locks[position]); seq_printf(seq, "\tlist_order_%u_groups: %u\n", (unsigned int)position, count); =20 @@ -3417,8 +3448,6 @@ int ext4_mb_add_groupinfo(struct super_block *sb, ext= 4_group_t group, INIT_LIST_HEAD(&meta_group_info[i]->bb_prealloc_list); init_rwsem(&meta_group_info[i]->alloc_sem); meta_group_info[i]->bb_free_root =3D RB_ROOT; - INIT_LIST_HEAD(&meta_group_info[i]->bb_largest_free_order_node); - INIT_LIST_HEAD(&meta_group_info[i]->bb_avg_fragment_size_node); meta_group_info[i]->bb_largest_free_order =3D -1; /* uninit */ meta_group_info[i]->bb_avg_fragment_size_order =3D -1; /* uninit */ meta_group_info[i]->bb_group =3D group; @@ -3628,6 +3657,20 @@ static void ext4_discard_work(struct work_struct *wo= rk) ext4_mb_unload_buddy(&e4b); } =20 +static inline void ext4_mb_avg_fragment_size_destory(struct ext4_sb_info *= sbi) +{ + for (int i =3D 0; i < MB_NUM_ORDERS(sbi->s_sb); i++) + xa_destroy(&sbi->s_mb_avg_fragment_size[i]); + kfree(sbi->s_mb_avg_fragment_size); +} + +static inline void ext4_mb_largest_free_orders_destory(struct ext4_sb_info= *sbi) +{ + for (int i =3D 0; i < MB_NUM_ORDERS(sbi->s_sb); i++) + xa_destroy(&sbi->s_mb_largest_free_orders[i]); + kfree(sbi->s_mb_largest_free_orders); +} + int ext4_mb_init(struct super_block *sb) { struct ext4_sb_info *sbi =3D EXT4_SB(sb); @@ -3673,41 +3716,24 @@ int ext4_mb_init(struct super_block *sb) } while (i < MB_NUM_ORDERS(sb)); =20 sbi->s_mb_avg_fragment_size =3D - kmalloc_array(MB_NUM_ORDERS(sb), sizeof(struct list_head), + kmalloc_array(MB_NUM_ORDERS(sb), sizeof(struct xarray), GFP_KERNEL); if (!sbi->s_mb_avg_fragment_size) { ret =3D -ENOMEM; goto out; } - sbi->s_mb_avg_fragment_size_locks =3D - kmalloc_array(MB_NUM_ORDERS(sb), sizeof(rwlock_t), - GFP_KERNEL); - if (!sbi->s_mb_avg_fragment_size_locks) { - ret =3D -ENOMEM; - goto out; - } - for (i =3D 0; i < MB_NUM_ORDERS(sb); i++) { - INIT_LIST_HEAD(&sbi->s_mb_avg_fragment_size[i]); - rwlock_init(&sbi->s_mb_avg_fragment_size_locks[i]); - } + for (i =3D 0; i < MB_NUM_ORDERS(sb); i++) + xa_init(&sbi->s_mb_avg_fragment_size[i]); + sbi->s_mb_largest_free_orders =3D - kmalloc_array(MB_NUM_ORDERS(sb), sizeof(struct list_head), + kmalloc_array(MB_NUM_ORDERS(sb), sizeof(struct xarray), GFP_KERNEL); if (!sbi->s_mb_largest_free_orders) { ret =3D -ENOMEM; goto out; } - sbi->s_mb_largest_free_orders_locks =3D - kmalloc_array(MB_NUM_ORDERS(sb), sizeof(rwlock_t), - GFP_KERNEL); - if (!sbi->s_mb_largest_free_orders_locks) { - ret =3D -ENOMEM; - goto out; - } - for (i =3D 0; i < MB_NUM_ORDERS(sb); i++) { - INIT_LIST_HEAD(&sbi->s_mb_largest_free_orders[i]); - rwlock_init(&sbi->s_mb_largest_free_orders_locks[i]); - } + for (i =3D 0; i < MB_NUM_ORDERS(sb); i++) + xa_init(&sbi->s_mb_largest_free_orders[i]); =20 spin_lock_init(&sbi->s_md_lock); atomic_set(&sbi->s_mb_free_pending, 0); @@ -3792,10 +3818,8 @@ int ext4_mb_init(struct super_block *sb) kfree(sbi->s_mb_last_groups); sbi->s_mb_last_groups =3D NULL; out: - kfree(sbi->s_mb_avg_fragment_size); - kfree(sbi->s_mb_avg_fragment_size_locks); - kfree(sbi->s_mb_largest_free_orders); - kfree(sbi->s_mb_largest_free_orders_locks); + ext4_mb_avg_fragment_size_destory(sbi); + ext4_mb_largest_free_orders_destory(sbi); kfree(sbi->s_mb_offsets); sbi->s_mb_offsets =3D NULL; kfree(sbi->s_mb_maxs); @@ -3862,10 +3886,8 @@ void ext4_mb_release(struct super_block *sb) kvfree(group_info); rcu_read_unlock(); } - kfree(sbi->s_mb_avg_fragment_size); - kfree(sbi->s_mb_avg_fragment_size_locks); - kfree(sbi->s_mb_largest_free_orders); - kfree(sbi->s_mb_largest_free_orders_locks); + ext4_mb_avg_fragment_size_destory(sbi); + ext4_mb_largest_free_orders_destory(sbi); kfree(sbi->s_mb_offsets); kfree(sbi->s_mb_maxs); iput(sbi->s_buddy_cache); --=20 2.46.1 From nobody Tue Oct 7 05:23:49 2025 Received: from szxga02-in.huawei.com (szxga02-in.huawei.com [45.249.212.188]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 007A925C838; Mon, 14 Jul 2025 13:18:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=45.249.212.188 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752499130; cv=none; b=R5+MDsoWV/rpMSnfKYKaz74vBHfSsLFNg+iMkIv4R9RfII3uD0seLNADMbVI8wMWUqpfiyF8PGZoqVq39TeENQgf4jlE0nGwcqTbt8XGHf9YHxK4XnqsIeBBOPDFD1AMWwMgTzrEsK3htQ2Bw8pquMJjHEBAC4LfJ14yImnrNjA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752499130; c=relaxed/simple; bh=TBdP69I3tQAgATC2WnfYyL1KXYt6iZ5jfHMuPG3EBvY=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=apPWEs4JdPlyTxKwuclCcD+1DX7efjMoG/CSkVR1hYCdpW/ckEBp9mGBxzUwjd+KLROOOpgt5or6xRs06tzKZJGCtUrtNnWxlPVC1obvhoAWmdQ8FCPOWa9WCQgYn4FiHdItnFNnOxvlQiCcppIoL/I7aPl4laIzkS4mDbv16JQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com; spf=pass smtp.mailfrom=huawei.com; arc=none smtp.client-ip=45.249.212.188 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=huawei.com Received: from mail.maildlp.com (unknown [172.19.162.254]) by szxga02-in.huawei.com (SkyGuard) with ESMTP id 4bgjT45HY5zXdyr; Mon, 14 Jul 2025 21:14:16 +0800 (CST) Received: from dggpemf500013.china.huawei.com (unknown [7.185.36.188]) by mail.maildlp.com (Postfix) with ESMTPS id D1991180450; Mon, 14 Jul 2025 21:18:44 +0800 (CST) Received: from huawei.com (10.175.112.188) by dggpemf500013.china.huawei.com (7.185.36.188) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.11; Mon, 14 Jul 2025 21:18:43 +0800 From: Baokun Li To: CC: , , , , , , , , , Subject: [PATCH v3 16/17] ext4: refactor choose group to scan group Date: Mon, 14 Jul 2025 21:03:26 +0800 Message-ID: <20250714130327.1830534-17-libaokun1@huawei.com> X-Mailer: git-send-email 2.46.1 In-Reply-To: <20250714130327.1830534-1-libaokun1@huawei.com> References: <20250714130327.1830534-1-libaokun1@huawei.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: kwepems500001.china.huawei.com (7.221.188.70) To dggpemf500013.china.huawei.com (7.185.36.188) Content-Type: text/plain; charset="utf-8" This commit converts the `choose group` logic to `scan group` using previously prepared helper functions. This allows us to leverage xarrays for ordered non-linear traversal, thereby mitigating the "bouncing" issue inherent in the `choose group` mechanism. This also decouples linear and non-linear traversals, leading to cleaner and more readable code. Key changes: * ext4_mb_choose_next_group() is refactored to ext4_mb_scan_groups(). * Replaced ext4_mb_good_group() with ext4_mb_scan_group() in non-linear traversals, and related functions now return error codes instead of group info. * Added ext4_mb_scan_groups_linear() for performing linear scans starting from a specific group for a set number of times. * Linear scans now execute up to sbi->s_mb_max_linear_groups times, so ac_groups_linear_remaining is removed as it's no longer used. * ac->ac_criteria is now used directly instead of passing cr around. Also, ac->ac_criteria is incremented directly after groups scan fails for the corresponding criteria. * Since we're now directly scanning groups instead of finding a good group then scanning, the following variables and flags are no longer needed, s_bal_cX_groups_considered is sufficient. s_bal_p2_aligned_bad_suggestions s_bal_goal_fast_bad_suggestions s_bal_best_avail_bad_suggestions EXT4_MB_CR_POWER2_ALIGNED_OPTIMIZED EXT4_MB_CR_GOAL_LEN_FAST_OPTIMIZED EXT4_MB_CR_BEST_AVAIL_LEN_OPTIMIZED Signed-off-by: Baokun Li Reviewed-by: Zhang Yi --- fs/ext4/ext4.h | 12 -- fs/ext4/mballoc.c | 292 +++++++++++++++++++++------------------------- fs/ext4/mballoc.h | 1 - 3 files changed, 131 insertions(+), 174 deletions(-) diff --git a/fs/ext4/ext4.h b/fs/ext4/ext4.h index ea412fdb0b76..6afd3447bfca 100644 --- a/fs/ext4/ext4.h +++ b/fs/ext4/ext4.h @@ -207,15 +207,6 @@ enum criteria { #define EXT4_MB_USE_RESERVED 0x2000 /* Do strict check for free blocks while retrying block allocation */ #define EXT4_MB_STRICT_CHECK 0x4000 -/* Large fragment size list lookup succeeded at least once for - * CR_POWER2_ALIGNED */ -#define EXT4_MB_CR_POWER2_ALIGNED_OPTIMIZED 0x8000 -/* Avg fragment size rb tree lookup succeeded at least once for - * CR_GOAL_LEN_FAST */ -#define EXT4_MB_CR_GOAL_LEN_FAST_OPTIMIZED 0x00010000 -/* Avg fragment size rb tree lookup succeeded at least once for - * CR_BEST_AVAIL_LEN */ -#define EXT4_MB_CR_BEST_AVAIL_LEN_OPTIMIZED 0x00020000 =20 struct ext4_allocation_request { /* target inode for block we're allocating */ @@ -1643,9 +1634,6 @@ struct ext4_sb_info { atomic_t s_bal_len_goals; /* len goal hits */ atomic_t s_bal_breaks; /* too long searches */ atomic_t s_bal_2orders; /* 2^order hits */ - atomic_t s_bal_p2_aligned_bad_suggestions; - atomic_t s_bal_goal_fast_bad_suggestions; - atomic_t s_bal_best_avail_bad_suggestions; atomic64_t s_bal_cX_groups_considered[EXT4_MB_NUM_CRS]; atomic64_t s_bal_cX_hits[EXT4_MB_NUM_CRS]; atomic64_t s_bal_cX_failed[EXT4_MB_NUM_CRS]; /* cX loop didn't find bloc= ks */ diff --git a/fs/ext4/mballoc.c b/fs/ext4/mballoc.c index a9eb997b8c9b..79b2c6b37fbd 100644 --- a/fs/ext4/mballoc.c +++ b/fs/ext4/mballoc.c @@ -425,8 +425,8 @@ static void ext4_mb_generate_from_pa(struct super_block= *sb, void *bitmap, ext4_group_t group); static void ext4_mb_new_preallocation(struct ext4_allocation_context *ac); =20 -static bool ext4_mb_good_group(struct ext4_allocation_context *ac, - ext4_group_t group, enum criteria cr); +static int ext4_mb_scan_group(struct ext4_allocation_context *ac, + ext4_group_t group); =20 static int ext4_try_to_trim_range(struct super_block *sb, struct ext4_buddy *e4b, ext4_grpblk_t start, @@ -875,9 +875,8 @@ mb_update_avg_fragment_size(struct super_block *sb, str= uct ext4_group_info *grp) } } =20 -static struct ext4_group_info * -ext4_mb_find_good_group_xarray(struct ext4_allocation_context *ac, - struct xarray *xa, ext4_group_t start) +static int ext4_mb_scan_groups_xarray(struct ext4_allocation_context *ac, + struct xarray *xa, ext4_group_t start) { struct super_block *sb =3D ac->ac_sb; struct ext4_sb_info *sbi =3D EXT4_SB(sb); @@ -888,16 +887,18 @@ ext4_mb_find_good_group_xarray(struct ext4_allocation= _context *ac, struct ext4_group_info *grp; =20 if (WARN_ON_ONCE(start >=3D end)) - return NULL; + return 0; =20 wrap_around: xa_for_each_range(xa, group, grp, start, end - 1) { + int err; + if (sbi->s_mb_stats) atomic64_inc(&sbi->s_bal_cX_groups_considered[cr]); =20 - if (!spin_is_locked(ext4_group_lock_ptr(sb, group)) && - likely(ext4_mb_good_group(ac, group, cr))) - return grp; + err =3D ext4_mb_scan_group(ac, grp->bb_group); + if (err || ac->ac_status !=3D AC_STATUS_CONTINUE) + return err; =20 cond_resched(); } @@ -908,95 +909,82 @@ ext4_mb_find_good_group_xarray(struct ext4_allocation= _context *ac, goto wrap_around; } =20 - return NULL; + return 0; } =20 /* * Find a suitable group of given order from the largest free orders xarra= y. */ -static struct ext4_group_info * -ext4_mb_find_good_group_largest_free_order(struct ext4_allocation_context = *ac, - int order, ext4_group_t start) +static int +ext4_mb_scan_groups_largest_free_order(struct ext4_allocation_context *ac, + int order, ext4_group_t start) { struct xarray *xa =3D &EXT4_SB(ac->ac_sb)->s_mb_largest_free_orders[order= ]; =20 if (xa_empty(xa)) - return NULL; + return 0; =20 - return ext4_mb_find_good_group_xarray(ac, xa, start); + return ext4_mb_scan_groups_xarray(ac, xa, start); } =20 /* * Choose next group by traversing largest_free_order lists. Updates *new_= cr if * cr level needs an update. */ -static void ext4_mb_choose_next_group_p2_aligned(struct ext4_allocation_co= ntext *ac, - enum criteria *new_cr, ext4_group_t *group) +static int ext4_mb_scan_groups_p2_aligned(struct ext4_allocation_context *= ac, + ext4_group_t group) { struct ext4_sb_info *sbi =3D EXT4_SB(ac->ac_sb); - struct ext4_group_info *grp; int i; - - if (ac->ac_status =3D=3D AC_STATUS_FOUND) - return; - - if (unlikely(sbi->s_mb_stats && ac->ac_flags & EXT4_MB_CR_POWER2_ALIGNED_= OPTIMIZED)) - atomic_inc(&sbi->s_bal_p2_aligned_bad_suggestions); + int ret =3D 0; =20 for (i =3D ac->ac_2order; i < MB_NUM_ORDERS(ac->ac_sb); i++) { - grp =3D ext4_mb_find_good_group_largest_free_order(ac, i, *group); - if (grp) { - *group =3D grp->bb_group; - ac->ac_flags |=3D EXT4_MB_CR_POWER2_ALIGNED_OPTIMIZED; - return; - } + ret =3D ext4_mb_scan_groups_largest_free_order(ac, i, group); + if (ret || ac->ac_status !=3D AC_STATUS_CONTINUE) + return ret; } =20 + if (sbi->s_mb_stats) + atomic64_inc(&sbi->s_bal_cX_failed[ac->ac_criteria]); + /* Increment cr and search again if no group is found */ - *new_cr =3D CR_GOAL_LEN_FAST; + ac->ac_criteria =3D CR_GOAL_LEN_FAST; + return ret; } =20 /* * Find a suitable group of given order from the average fragments xarray. */ -static struct ext4_group_info * -ext4_mb_find_good_group_avg_frag_xarray(struct ext4_allocation_context *ac, - int order, ext4_group_t start) +static int ext4_mb_scan_groups_avg_frag_order(struct ext4_allocation_conte= xt *ac, + int order, ext4_group_t start) { struct xarray *xa =3D &EXT4_SB(ac->ac_sb)->s_mb_avg_fragment_size[order]; =20 if (xa_empty(xa)) - return NULL; + return 0; =20 - return ext4_mb_find_good_group_xarray(ac, xa, start); + return ext4_mb_scan_groups_xarray(ac, xa, start); } =20 /* * Choose next group by traversing average fragment size list of suitable * order. Updates *new_cr if cr level needs an update. */ -static void ext4_mb_choose_next_group_goal_fast(struct ext4_allocation_con= text *ac, - enum criteria *new_cr, ext4_group_t *group) +static int ext4_mb_scan_groups_goal_fast(struct ext4_allocation_context *a= c, + ext4_group_t group) { struct ext4_sb_info *sbi =3D EXT4_SB(ac->ac_sb); - struct ext4_group_info *grp =3D NULL; - int i; + int i, ret =3D 0; =20 - if (unlikely(ac->ac_flags & EXT4_MB_CR_GOAL_LEN_FAST_OPTIMIZED)) { - if (sbi->s_mb_stats) - atomic_inc(&sbi->s_bal_goal_fast_bad_suggestions); - } - - for (i =3D mb_avg_fragment_size_order(ac->ac_sb, ac->ac_g_ex.fe_len); - i < MB_NUM_ORDERS(ac->ac_sb); i++) { - grp =3D ext4_mb_find_good_group_avg_frag_xarray(ac, i, *group); - if (grp) { - *group =3D grp->bb_group; - ac->ac_flags |=3D EXT4_MB_CR_GOAL_LEN_FAST_OPTIMIZED; - return; - } + i =3D mb_avg_fragment_size_order(ac->ac_sb, ac->ac_g_ex.fe_len); + for (; i < MB_NUM_ORDERS(ac->ac_sb); i++) { + ret =3D ext4_mb_scan_groups_avg_frag_order(ac, i, group); + if (ret || ac->ac_status !=3D AC_STATUS_CONTINUE) + return ret; } =20 + if (sbi->s_mb_stats) + atomic64_inc(&sbi->s_bal_cX_failed[ac->ac_criteria]); /* * CR_BEST_AVAIL_LEN works based on the concept that we have * a larger normalized goal len request which can be trimmed to @@ -1006,9 +994,11 @@ static void ext4_mb_choose_next_group_goal_fast(struc= t ext4_allocation_context * * See function ext4_mb_normalize_request() (EXT4_MB_HINT_DATA). */ if (ac->ac_flags & EXT4_MB_HINT_DATA) - *new_cr =3D CR_BEST_AVAIL_LEN; + ac->ac_criteria =3D CR_BEST_AVAIL_LEN; else - *new_cr =3D CR_GOAL_LEN_SLOW; + ac->ac_criteria =3D CR_GOAL_LEN_SLOW; + + return ret; } =20 /* @@ -1020,19 +1010,14 @@ static void ext4_mb_choose_next_group_goal_fast(str= uct ext4_allocation_context * * preallocations. However, we make sure that we don't trim the request too * much and fall to CR_GOAL_LEN_SLOW in that case. */ -static void ext4_mb_choose_next_group_best_avail(struct ext4_allocation_co= ntext *ac, - enum criteria *new_cr, ext4_group_t *group) +static int ext4_mb_scan_groups_best_avail(struct ext4_allocation_context *= ac, + ext4_group_t group) { + int ret =3D 0; struct ext4_sb_info *sbi =3D EXT4_SB(ac->ac_sb); - struct ext4_group_info *grp =3D NULL; int i, order, min_order; unsigned long num_stripe_clusters =3D 0; =20 - if (unlikely(ac->ac_flags & EXT4_MB_CR_BEST_AVAIL_LEN_OPTIMIZED)) { - if (sbi->s_mb_stats) - atomic_inc(&sbi->s_bal_best_avail_bad_suggestions); - } - /* * mb_avg_fragment_size_order() returns order in a way that makes * retrieving back the length using (1 << order) inaccurate. Hence, use @@ -1085,18 +1070,18 @@ static void ext4_mb_choose_next_group_best_avail(st= ruct ext4_allocation_context frag_order =3D mb_avg_fragment_size_order(ac->ac_sb, ac->ac_g_ex.fe_len); =20 - grp =3D ext4_mb_find_good_group_avg_frag_xarray(ac, frag_order, - *group); - if (grp) { - *group =3D grp->bb_group; - ac->ac_flags |=3D EXT4_MB_CR_BEST_AVAIL_LEN_OPTIMIZED; - return; - } + ret =3D ext4_mb_scan_groups_avg_frag_order(ac, frag_order, group); + if (ret || ac->ac_status !=3D AC_STATUS_CONTINUE) + return ret; } =20 /* Reset goal length to original goal length before falling into CR_GOAL_= LEN_SLOW */ ac->ac_g_ex.fe_len =3D ac->ac_orig_goal_len; - *new_cr =3D CR_GOAL_LEN_SLOW; + if (sbi->s_mb_stats) + atomic64_inc(&sbi->s_bal_cX_failed[ac->ac_criteria]); + ac->ac_criteria =3D CR_GOAL_LEN_SLOW; + + return ret; } =20 static inline int should_optimize_scan(struct ext4_allocation_context *ac) @@ -1111,59 +1096,82 @@ static inline int should_optimize_scan(struct ext4_= allocation_context *ac) } =20 /* - * Return next linear group for allocation. + * next linear group for allocation. */ -static ext4_group_t -next_linear_group(ext4_group_t group, ext4_group_t ngroups) +static void next_linear_group(ext4_group_t *group, ext4_group_t ngroups) { /* * Artificially restricted ngroups for non-extent * files makes group > ngroups possible on first loop. */ - return group + 1 >=3D ngroups ? 0 : group + 1; + *group =3D *group + 1 >=3D ngroups ? 0 : *group + 1; } =20 -/* - * ext4_mb_choose_next_group: choose next group for allocation. - * - * @ac Allocation Context - * @new_cr This is an output parameter. If the there is no good group - * available at current CR level, this field is updated to indi= cate - * the new cr level that should be used. - * @group This is an input / output parameter. As an input it indicate= s the - * next group that the allocator intends to use for allocation.= As - * output, this field indicates the next group that should be u= sed as - * determined by the optimization functions. - * @ngroups Total number of groups - */ -static void ext4_mb_choose_next_group(struct ext4_allocation_context *ac, - enum criteria *new_cr, ext4_group_t *group, ext4_group_t ngroups) +static int ext4_mb_scan_groups_linear(struct ext4_allocation_context *ac, + ext4_group_t ngroups, ext4_group_t *start, ext4_group_t count) { - *new_cr =3D ac->ac_criteria; + int ret, i; + enum criteria cr =3D ac->ac_criteria; + struct super_block *sb =3D ac->ac_sb; + struct ext4_sb_info *sbi =3D EXT4_SB(sb); + ext4_group_t group =3D *start; =20 - if (!should_optimize_scan(ac)) { - *group =3D next_linear_group(*group, ngroups); - return; + for (i =3D 0; i < count; i++, next_linear_group(&group, ngroups)) { + ret =3D ext4_mb_scan_group(ac, group); + if (ret || ac->ac_status !=3D AC_STATUS_CONTINUE) + return ret; + cond_resched(); } =20 + *start =3D group; + if (count =3D=3D ngroups) + ac->ac_criteria++; + + /* Processed all groups and haven't found blocks */ + if (sbi->s_mb_stats && i =3D=3D ngroups) + atomic64_inc(&sbi->s_bal_cX_failed[cr]); + + return 0; +} + +static int ext4_mb_scan_groups(struct ext4_allocation_context *ac) +{ + int ret =3D 0; + ext4_group_t start; + struct ext4_sb_info *sbi =3D EXT4_SB(ac->ac_sb); + ext4_group_t ngroups =3D ext4_get_groups_count(ac->ac_sb); + + /* non-extent files are limited to low blocks/groups */ + if (!(ext4_test_inode_flag(ac->ac_inode, EXT4_INODE_EXTENTS))) + ngroups =3D sbi->s_blockfile_groups; + + /* searching for the right group start from the goal value specified */ + start =3D ac->ac_g_ex.fe_group; + ac->ac_prefetch_grp =3D start; + ac->ac_prefetch_nr =3D 0; + + if (!should_optimize_scan(ac)) + return ext4_mb_scan_groups_linear(ac, ngroups, &start, ngroups); + /* * Optimized scanning can return non adjacent groups which can cause * seek overhead for rotational disks. So try few linear groups before * trying optimized scan. */ - if (ac->ac_groups_linear_remaining) { - *group =3D next_linear_group(*group, ngroups); - ac->ac_groups_linear_remaining--; - return; - } + if (sbi->s_mb_max_linear_groups) + ret =3D ext4_mb_scan_groups_linear(ac, ngroups, &start, + sbi->s_mb_max_linear_groups); + if (ret || ac->ac_status !=3D AC_STATUS_CONTINUE) + return ret; =20 - if (*new_cr =3D=3D CR_POWER2_ALIGNED) { - ext4_mb_choose_next_group_p2_aligned(ac, new_cr, group); - } else if (*new_cr =3D=3D CR_GOAL_LEN_FAST) { - ext4_mb_choose_next_group_goal_fast(ac, new_cr, group); - } else if (*new_cr =3D=3D CR_BEST_AVAIL_LEN) { - ext4_mb_choose_next_group_best_avail(ac, new_cr, group); - } else { + switch (ac->ac_criteria) { + case CR_POWER2_ALIGNED: + return ext4_mb_scan_groups_p2_aligned(ac, start); + case CR_GOAL_LEN_FAST: + return ext4_mb_scan_groups_goal_fast(ac, start); + case CR_BEST_AVAIL_LEN: + return ext4_mb_scan_groups_best_avail(ac, start); + default: /* * TODO: For CR_GOAL_LEN_SLOW, we can arrange groups in an * rb tree sorted by bb_free. But until that happens, we should @@ -1171,6 +1179,8 @@ static void ext4_mb_choose_next_group(struct ext4_all= ocation_context *ac, */ WARN_ON(1); } + + return 0; } =20 /* @@ -2928,20 +2938,11 @@ static int ext4_mb_scan_group(struct ext4_allocatio= n_context *ac, static noinline_for_stack int ext4_mb_regular_allocator(struct ext4_allocation_context *ac) { - ext4_group_t ngroups, group, i; - enum criteria new_cr, cr =3D CR_GOAL_LEN_FAST; + ext4_group_t i; int err =3D 0; - struct ext4_sb_info *sbi; - struct super_block *sb; + struct super_block *sb =3D ac->ac_sb; + struct ext4_sb_info *sbi =3D EXT4_SB(sb); struct ext4_buddy e4b; - int lost; - - sb =3D ac->ac_sb; - sbi =3D EXT4_SB(sb); - ngroups =3D ext4_get_groups_count(sb); - /* non-extent files are limited to low blocks/groups */ - if (!(ext4_test_inode_flag(ac->ac_inode, EXT4_INODE_EXTENTS))) - ngroups =3D sbi->s_blockfile_groups; =20 BUG_ON(ac->ac_status =3D=3D AC_STATUS_FOUND); =20 @@ -2987,48 +2988,21 @@ ext4_mb_regular_allocator(struct ext4_allocation_co= ntext *ac) * start with CR_GOAL_LEN_FAST, unless it is power of 2 * aligned, in which case let's do that faster approach first. */ + ac->ac_criteria =3D CR_GOAL_LEN_FAST; if (ac->ac_2order) - cr =3D CR_POWER2_ALIGNED; + ac->ac_criteria =3D CR_POWER2_ALIGNED; =20 ac->ac_e4b =3D &e4b; ac->ac_prefetch_ios =3D 0; ac->ac_first_err =3D 0; repeat: - for (; cr < EXT4_MB_NUM_CRS && ac->ac_status =3D=3D AC_STATUS_CONTINUE; c= r++) { - ac->ac_criteria =3D cr; - /* - * searching for the right group start - * from the goal value specified - */ - group =3D ac->ac_g_ex.fe_group; - ac->ac_groups_linear_remaining =3D sbi->s_mb_max_linear_groups; - ac->ac_prefetch_grp =3D group; - ac->ac_prefetch_nr =3D 0; - - for (i =3D 0, new_cr =3D cr; i < ngroups; i++, - ext4_mb_choose_next_group(ac, &new_cr, &group, ngroups)) { - - cond_resched(); - if (new_cr !=3D cr) { - cr =3D new_cr; - goto repeat; - } - - err =3D ext4_mb_scan_group(ac, group); - if (err) - goto out; - - if (ac->ac_status !=3D AC_STATUS_CONTINUE) - break; - } - /* Processed all groups and haven't found blocks */ - if (sbi->s_mb_stats && i =3D=3D ngroups) - atomic64_inc(&sbi->s_bal_cX_failed[cr]); + while (ac->ac_criteria < EXT4_MB_NUM_CRS) { + err =3D ext4_mb_scan_groups(ac); + if (err) + goto out; =20 - if (i =3D=3D ngroups && ac->ac_criteria =3D=3D CR_BEST_AVAIL_LEN) - /* Reset goal length to original goal length before - * falling into CR_GOAL_LEN_SLOW */ - ac->ac_g_ex.fe_len =3D ac->ac_orig_goal_len; + if (ac->ac_status !=3D AC_STATUS_CONTINUE) + break; } =20 if (ac->ac_b_ex.fe_len > 0 && ac->ac_status !=3D AC_STATUS_FOUND && @@ -3039,6 +3013,8 @@ ext4_mb_regular_allocator(struct ext4_allocation_cont= ext *ac) */ ext4_mb_try_best_found(ac, &e4b); if (ac->ac_status !=3D AC_STATUS_FOUND) { + int lost; + /* * Someone more lucky has already allocated it. * The only thing we can do is just take first @@ -3054,7 +3030,7 @@ ext4_mb_regular_allocator(struct ext4_allocation_cont= ext *ac) ac->ac_b_ex.fe_len =3D 0; ac->ac_status =3D AC_STATUS_CONTINUE; ac->ac_flags |=3D EXT4_MB_HINT_FIRST; - cr =3D CR_ANY_FREE; + ac->ac_criteria =3D CR_ANY_FREE; goto repeat; } } @@ -3071,7 +3047,7 @@ ext4_mb_regular_allocator(struct ext4_allocation_cont= ext *ac) =20 mb_debug(sb, "Best len %d, origin len %d, ac_status %u, ac_flags 0x%x, cr= %d ret %d\n", ac->ac_b_ex.fe_len, ac->ac_o_ex.fe_len, ac->ac_status, - ac->ac_flags, cr, err); + ac->ac_flags, ac->ac_criteria, err); =20 if (ac->ac_prefetch_nr) ext4_mb_prefetch_fini(sb, ac->ac_prefetch_grp, ac->ac_prefetch_nr); @@ -3197,8 +3173,6 @@ int ext4_seq_mb_stats_show(struct seq_file *seq, void= *offset) atomic_read(&sbi->s_bal_cX_ex_scanned[CR_POWER2_ALIGNED])); seq_printf(seq, "\t\tuseless_loops: %llu\n", atomic64_read(&sbi->s_bal_cX_failed[CR_POWER2_ALIGNED])); - seq_printf(seq, "\t\tbad_suggestions: %u\n", - atomic_read(&sbi->s_bal_p2_aligned_bad_suggestions)); =20 /* CR_GOAL_LEN_FAST stats */ seq_puts(seq, "\tcr_goal_fast_stats:\n"); @@ -3211,8 +3185,6 @@ int ext4_seq_mb_stats_show(struct seq_file *seq, void= *offset) atomic_read(&sbi->s_bal_cX_ex_scanned[CR_GOAL_LEN_FAST])); seq_printf(seq, "\t\tuseless_loops: %llu\n", atomic64_read(&sbi->s_bal_cX_failed[CR_GOAL_LEN_FAST])); - seq_printf(seq, "\t\tbad_suggestions: %u\n", - atomic_read(&sbi->s_bal_goal_fast_bad_suggestions)); =20 /* CR_BEST_AVAIL_LEN stats */ seq_puts(seq, "\tcr_best_avail_stats:\n"); @@ -3226,8 +3198,6 @@ int ext4_seq_mb_stats_show(struct seq_file *seq, void= *offset) atomic_read(&sbi->s_bal_cX_ex_scanned[CR_BEST_AVAIL_LEN])); seq_printf(seq, "\t\tuseless_loops: %llu\n", atomic64_read(&sbi->s_bal_cX_failed[CR_BEST_AVAIL_LEN])); - seq_printf(seq, "\t\tbad_suggestions: %u\n", - atomic_read(&sbi->s_bal_best_avail_bad_suggestions)); =20 /* CR_GOAL_LEN_SLOW stats */ seq_puts(seq, "\tcr_goal_slow_stats:\n"); diff --git a/fs/ext4/mballoc.h b/fs/ext4/mballoc.h index 83886fc9521b..15a049f05d04 100644 --- a/fs/ext4/mballoc.h +++ b/fs/ext4/mballoc.h @@ -199,7 +199,6 @@ struct ext4_allocation_context { int ac_first_err; =20 __u32 ac_flags; /* allocation hints */ - __u32 ac_groups_linear_remaining; __u16 ac_groups_scanned; __u16 ac_found; __u16 ac_cX_found[EXT4_MB_NUM_CRS]; --=20 2.46.1 From nobody Tue Oct 7 05:23:49 2025 Received: from szxga04-in.huawei.com (szxga04-in.huawei.com [45.249.212.190]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 683DE25DB07; Mon, 14 Jul 2025 13:18:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=45.249.212.190 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752499130; cv=none; b=lmRKbz80SEgFalR3DppHImbqWp89hAZrYY3raMLUTGopO9i3kGzn1VxvRn6s4wcH3fOoGkLtjAthv1fzL1Jc/YB25oyRM5Gh561Itu4SjYR3OmFpH0n9W09MGWg9B6SsZzptB76qAwV9EJBw2XjM9xk35QQc1ArvCxRraLnEcZ0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752499130; c=relaxed/simple; bh=eTCTy6pujNn4ijrPbF3C+LkW3BUQLjeqwj7hkS2cXw4=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=b9bPueYf/OQXhkehsxVYuBd/Y88Gw3wOdT1xZ9bmPMllRBVWv6LpAzjqTvmPQfwsQzwt4nTPm/qK5IX6RBMbPTAc7jfd+u8x9T1bv/YaBWfcKHnlspEVAVvztAOLVsNQR00yT97yLaOrZz76erwVvoQXrDpFqY2LPf52JTCuluA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com; spf=pass smtp.mailfrom=huawei.com; arc=none smtp.client-ip=45.249.212.190 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=huawei.com Received: from mail.maildlp.com (unknown [172.19.88.214]) by szxga04-in.huawei.com (SkyGuard) with ESMTP id 4bgjTW4w2Gz2Cdk8; Mon, 14 Jul 2025 21:14:39 +0800 (CST) Received: from dggpemf500013.china.huawei.com (unknown [7.185.36.188]) by mail.maildlp.com (Postfix) with ESMTPS id D31F71A016C; Mon, 14 Jul 2025 21:18:45 +0800 (CST) Received: from huawei.com (10.175.112.188) by dggpemf500013.china.huawei.com (7.185.36.188) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.11; Mon, 14 Jul 2025 21:18:44 +0800 From: Baokun Li To: CC: , , , , , , , , , Subject: [PATCH v3 17/17] ext4: implement linear-like traversal across order xarrays Date: Mon, 14 Jul 2025 21:03:27 +0800 Message-ID: <20250714130327.1830534-18-libaokun1@huawei.com> X-Mailer: git-send-email 2.46.1 In-Reply-To: <20250714130327.1830534-1-libaokun1@huawei.com> References: <20250714130327.1830534-1-libaokun1@huawei.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: kwepems500001.china.huawei.com (7.221.188.70) To dggpemf500013.china.huawei.com (7.185.36.188) Content-Type: text/plain; charset="utf-8" Although we now perform ordered traversal within an xarray, this is currently limited to a single xarray. However, we have multiple such xarrays, which prevents us from guaranteeing a linear-like traversal where all groups on the right are visited before all groups on the left. For example, suppose we have 128 block groups, with a target group of 64, a target length corresponding to an order of 1, and available free groups of 16 (order 1) and group 65 (order 8): For linear traversal, when no suitable free block is found in group 64, it will search in the next block group until group 127, then start searching from 0 up to block group 63. It ensures continuous forward traversal, which is consistent with the unidirectional rotation behavior of HDD platters. Additionally, the block group lock contention during freeing block is unavoidable. The goal increasing from 0 to 64 indicates that previously scanned groups (which had no suitable free space and are likely to free blocks later) and skipped groups (which are currently in use) have newly freed some used blocks. If we allocate blocks in these groups, the probability of competing with other processes increases. For non-linear traversal, we first traverse all groups in order_1. If only group 16 has free space in this list, we first traverse [63, 128), then traverse [0, 64) to find the available group 16, and then allocate blocks in group 16. Therefore, it cannot guarantee continuous traversal in one direction, thus increasing the probability of contention. So refactor ext4_mb_scan_groups_xarray() to ext4_mb_scan_groups_xa_range() to only traverse a fixed range of groups, and move the logic for handling wrap around to the caller. The caller first iterates through all xarrays in the range [start, ngroups) and then through the range [0, start). This approach simulates a linear scan, which reduces contention between freeing blocks and allocating blocks. Assume we have the following groups, where "|" denotes the xarray traversal start position: order_1_groups: AB | CD order_2_groups: EF | GH Traversal order: Before: C > D > A > B > G > H > E > F After: C > D > G > H > A > B > E > F Performance test data follows: |CPU: Kunpeng 920 | P80 | P1 | |Memory: 512GB |------------------------|-------------------------| |960GB SSD (0.5GB/s)| base | patched | base | patched | |-------------------|-------|----------------|--------|----------------| |mb_optimize_scan=3D0 | 19555 | 20049 (+2.5%) | 315636 | 316724 (-0.3%) | |mb_optimize_scan=3D1 | 15496 | 19342 (+24.8%) | 323569 | 328324 (+1.4%) | |CPU: AMD 9654 * 2 | P96 | P1 | |Memory: 1536GB |------------------------|-------------------------| |960GB SSD (1GB/s) | base | patched | base | patched | |-------------------|-------|----------------|--------|----------------| |mb_optimize_scan=3D0 | 53192 | 52125 (-2.0%) | 212678 | 215136 (+1.1%) | |mb_optimize_scan=3D1 | 37636 | 50331 (+33.7%) | 214189 | 209431 (-2.2%) | Signed-off-by: Baokun Li Reviewed-by: Zhang Yi --- fs/ext4/mballoc.c | 68 ++++++++++++++++++++++++++++++++--------------- 1 file changed, 47 insertions(+), 21 deletions(-) diff --git a/fs/ext4/mballoc.c b/fs/ext4/mballoc.c index 79b2c6b37fbd..742124c8213b 100644 --- a/fs/ext4/mballoc.c +++ b/fs/ext4/mballoc.c @@ -875,21 +875,20 @@ mb_update_avg_fragment_size(struct super_block *sb, s= truct ext4_group_info *grp) } } =20 -static int ext4_mb_scan_groups_xarray(struct ext4_allocation_context *ac, - struct xarray *xa, ext4_group_t start) +static int ext4_mb_scan_groups_xa_range(struct ext4_allocation_context *ac, + struct xarray *xa, + ext4_group_t start, ext4_group_t end) { struct super_block *sb =3D ac->ac_sb; struct ext4_sb_info *sbi =3D EXT4_SB(sb); enum criteria cr =3D ac->ac_criteria; ext4_group_t ngroups =3D ext4_get_groups_count(sb); unsigned long group =3D start; - ext4_group_t end =3D ngroups; struct ext4_group_info *grp; =20 - if (WARN_ON_ONCE(start >=3D end)) + if (WARN_ON_ONCE(end > ngroups || start >=3D end)) return 0; =20 -wrap_around: xa_for_each_range(xa, group, grp, start, end - 1) { int err; =20 @@ -903,28 +902,23 @@ static int ext4_mb_scan_groups_xarray(struct ext4_all= ocation_context *ac, cond_resched(); } =20 - if (start) { - end =3D start; - start =3D 0; - goto wrap_around; - } - return 0; } =20 /* * Find a suitable group of given order from the largest free orders xarra= y. */ -static int -ext4_mb_scan_groups_largest_free_order(struct ext4_allocation_context *ac, - int order, ext4_group_t start) +static inline int +ext4_mb_scan_groups_largest_free_order_range(struct ext4_allocation_contex= t *ac, + int order, ext4_group_t start, + ext4_group_t end) { struct xarray *xa =3D &EXT4_SB(ac->ac_sb)->s_mb_largest_free_orders[order= ]; =20 if (xa_empty(xa)) return 0; =20 - return ext4_mb_scan_groups_xarray(ac, xa, start); + return ext4_mb_scan_groups_xa_range(ac, xa, start, end); } =20 /* @@ -937,12 +931,22 @@ static int ext4_mb_scan_groups_p2_aligned(struct ext4= _allocation_context *ac, struct ext4_sb_info *sbi =3D EXT4_SB(ac->ac_sb); int i; int ret =3D 0; + ext4_group_t start, end; =20 + start =3D group; + end =3D ext4_get_groups_count(ac->ac_sb); +wrap_around: for (i =3D ac->ac_2order; i < MB_NUM_ORDERS(ac->ac_sb); i++) { - ret =3D ext4_mb_scan_groups_largest_free_order(ac, i, group); + ret =3D ext4_mb_scan_groups_largest_free_order_range(ac, i, + start, end); if (ret || ac->ac_status !=3D AC_STATUS_CONTINUE) return ret; } + if (start) { + end =3D start; + start =3D 0; + goto wrap_around; + } =20 if (sbi->s_mb_stats) atomic64_inc(&sbi->s_bal_cX_failed[ac->ac_criteria]); @@ -955,15 +959,17 @@ static int ext4_mb_scan_groups_p2_aligned(struct ext4= _allocation_context *ac, /* * Find a suitable group of given order from the average fragments xarray. */ -static int ext4_mb_scan_groups_avg_frag_order(struct ext4_allocation_conte= xt *ac, - int order, ext4_group_t start) +static int +ext4_mb_scan_groups_avg_frag_order_range(struct ext4_allocation_context *a= c, + int order, ext4_group_t start, + ext4_group_t end) { struct xarray *xa =3D &EXT4_SB(ac->ac_sb)->s_mb_avg_fragment_size[order]; =20 if (xa_empty(xa)) return 0; =20 - return ext4_mb_scan_groups_xarray(ac, xa, start); + return ext4_mb_scan_groups_xa_range(ac, xa, start, end); } =20 /* @@ -975,13 +981,23 @@ static int ext4_mb_scan_groups_goal_fast(struct ext4_= allocation_context *ac, { struct ext4_sb_info *sbi =3D EXT4_SB(ac->ac_sb); int i, ret =3D 0; + ext4_group_t start, end; =20 + start =3D group; + end =3D ext4_get_groups_count(ac->ac_sb); +wrap_around: i =3D mb_avg_fragment_size_order(ac->ac_sb, ac->ac_g_ex.fe_len); for (; i < MB_NUM_ORDERS(ac->ac_sb); i++) { - ret =3D ext4_mb_scan_groups_avg_frag_order(ac, i, group); + ret =3D ext4_mb_scan_groups_avg_frag_order_range(ac, i, + start, end); if (ret || ac->ac_status !=3D AC_STATUS_CONTINUE) return ret; } + if (start) { + end =3D start; + start =3D 0; + goto wrap_around; + } =20 if (sbi->s_mb_stats) atomic64_inc(&sbi->s_bal_cX_failed[ac->ac_criteria]); @@ -1017,6 +1033,7 @@ static int ext4_mb_scan_groups_best_avail(struct ext4= _allocation_context *ac, struct ext4_sb_info *sbi =3D EXT4_SB(ac->ac_sb); int i, order, min_order; unsigned long num_stripe_clusters =3D 0; + ext4_group_t start, end; =20 /* * mb_avg_fragment_size_order() returns order in a way that makes @@ -1048,6 +1065,9 @@ static int ext4_mb_scan_groups_best_avail(struct ext4= _allocation_context *ac, if (1 << min_order < ac->ac_o_ex.fe_len) min_order =3D fls(ac->ac_o_ex.fe_len); =20 + start =3D group; + end =3D ext4_get_groups_count(ac->ac_sb); +wrap_around: for (i =3D order; i >=3D min_order; i--) { int frag_order; /* @@ -1070,10 +1090,16 @@ static int ext4_mb_scan_groups_best_avail(struct ex= t4_allocation_context *ac, frag_order =3D mb_avg_fragment_size_order(ac->ac_sb, ac->ac_g_ex.fe_len); =20 - ret =3D ext4_mb_scan_groups_avg_frag_order(ac, frag_order, group); + ret =3D ext4_mb_scan_groups_avg_frag_order_range(ac, frag_order, + start, end); if (ret || ac->ac_status !=3D AC_STATUS_CONTINUE) return ret; } + if (start) { + end =3D start; + start =3D 0; + goto wrap_around; + } =20 /* Reset goal length to original goal length before falling into CR_GOAL_= LEN_SLOW */ ac->ac_g_ex.fe_len =3D ac->ac_orig_goal_len; --=20 2.46.1