From: Zhang Yi <yi.zhang@huawei.com>

Changes since v3:
 - Fix the end_pos assignment in patch 01.
 - Rename mpage_submit_buffers() to mpage_submit_partial_folio(), and
   fix a left shift out-of-bounds problem in patch 03.
 - Fix the spelling errors in patch 04.
 - Add a comment for NULL 'handle' test in
   ext4_journal_ensure_extent_credits().
 - Add patch 11 to limit the maximum order of the folio to 2048 fs
   blocks, prevent the overestimation of reserve journal credits during
   folios write-back.
Changes since v2:
 - Convert the processing of folios writeback in bytes instead of pages.
 - Refactor ext4_page_mkwrite() and ensure journal credits in
   ext4_block_write_begin() instead of in _ext4_get_block().
 - Enhance tracepoints in ext4_do_writepages().
 - Replace the outdated ext4_da_writepages_trans_blocks() and
   ext4_writepage_trans_blocks() with the new helper used to reserve
   credits for a single extent.
Changes since v1:
 - Make the write-back process supports writing a partial folio if it
   exits the mapping loop prematurely due to insufficient sapce or
   journal credits, it also fix the potential stale data and
   inconsistency issues.
 - Fix the same issue regarding the allocation of blocks in
   ext4_write_begin() and ext4_page_mkwrite() when delalloc is not
   enabled.

v3: https://lore.kernel.org/linux-ext4/20250701130635.4079595-1-yi.zhang@huaweicloud.com/
v2: https://lore.kernel.org/linux-ext4/20250611111625.1668035-1-yi.zhang@huaweicloud.com/
v1: https://lore.kernel.org/linux-ext4/20250530062858.458039-1-yi.zhang@huaweicloud.com/

Original Description

This series addresses the issue that Jan pointed out regarding large
folios support for ext4[1]. The problem is that the credits calculation
may insufficient in ext4_meta_trans_blocks() when allocating blocks
during write back a sufficiently large and discontinuous folio, it
doesn't involve the credits for updating bitmap and group descriptor
block. However, if we fix this issue, it may lead to significant
overestimation on the some filesystems with a lot of block groups.

The solution involves first ensure that the current journal transaction
has enough credits when we mapping an extent during allocating blocks.
Then if the credits reach the upper limit, exit the current mapping
loop, submit the partial folio and restart a new transaction. Finally,
fix the wrong credits calculation in ext4_meta_trans_blocks(). Please
see the following patches for details.

[1] https://lore.kernel.org/linux-ext4/ht54j6bvjmiqt62xmcveqlo7bmrunqs4ji7wikfteftdjijzek@7tz5gpejaoen/

Thanks,
Yi.

Zhang Yi (11):
  ext4: process folios writeback in bytes
  ext4: move the calculation of wbc->nr_to_write to mpage_folio_done()
  ext4: fix stale data if it bail out of the extents mapping loop
  ext4: refactor the block allocation process of ext4_page_mkwrite()
  ext4: restart handle if credits are insufficient during allocating
    blocks
  ext4: enhance tracepoints during the folios writeback
  ext4: correct the reserved credits for extent conversion
  ext4: reserved credits for one extent during the folio writeback
  ext4: replace ext4_writepage_trans_blocks()
  ext4: fix insufficient credits calculation in ext4_meta_trans_blocks()
  ext4: limit the maximum folio order

 fs/ext4/ext4.h              |   4 +-
 fs/ext4/extents.c           |   6 +-
 fs/ext4/ialloc.c            |   3 +-
 fs/ext4/inline.c            |   6 +-
 fs/ext4/inode.c             | 349 +++++++++++++++++++++++-------------
 fs/ext4/move_extent.c       |   3 +-
 fs/ext4/xattr.c             |   2 +-
 include/trace/events/ext4.h |  47 ++++-
 8 files changed, 272 insertions(+), 148 deletions(-)

-- 
2.46.1

On Mon, 07 Jul 2025 22:08:03 +0800, Zhang Yi wrote:
> Changes since v3:
>  - Fix the end_pos assignment in patch 01.
>  - Rename mpage_submit_buffers() to mpage_submit_partial_folio(), and
>    fix a left shift out-of-bounds problem in patch 03.
>  - Fix the spelling errors in patch 04.
>  - Add a comment for NULL 'handle' test in
>    ext4_journal_ensure_extent_credits().
>  - Add patch 11 to limit the maximum order of the folio to 2048 fs
>    blocks, prevent the overestimation of reserve journal credits during
>    folios write-back.
> Changes since v2:
>  - Convert the processing of folios writeback in bytes instead of pages.
>  - Refactor ext4_page_mkwrite() and ensure journal credits in
>    ext4_block_write_begin() instead of in _ext4_get_block().
>  - Enhance tracepoints in ext4_do_writepages().
>  - Replace the outdated ext4_da_writepages_trans_blocks() and
>    ext4_writepage_trans_blocks() with the new helper used to reserve
>    credits for a single extent.
> Changes since v1:
>  - Make the write-back process supports writing a partial folio if it
>    exits the mapping loop prematurely due to insufficient sapce or
>    journal credits, it also fix the potential stale data and
>    inconsistency issues.
>  - Fix the same issue regarding the allocation of blocks in
>    ext4_write_begin() and ext4_page_mkwrite() when delalloc is not
>    enabled.
> 
> [...]

Applied, thanks!

[01/11] ext4: process folios writeback in bytes
        commit: 1bfe6354e0975fe89c3d25e81b6546d205556a4b
[02/11] ext4: move the calculation of wbc->nr_to_write to mpage_folio_done()
        commit: f922c8c2461b022a2efd9914484901fb358a5b2a
[03/11] ext4: fix stale data if it bail out of the extents mapping loop
        commit: ded2d726a3041fce8afd88005cbfe15cd4737702
[04/11] ext4: refactor the block allocation process of ext4_page_mkwrite()
        commit: 2bddafea3d0d85ee9ac3cf5ba9a4b2f2d2f50257
[05/11] ext4: restart handle if credits are insufficient during allocating blocks
        commit: e2c4c49dee64ca2f42ad2958cbe1805de96b6732
[06/11] ext4: enhance tracepoints during the folios writeback
        commit: 6b132759b0fe78e518abafb62190c294100db6d6
[07/11] ext4: correct the reserved credits for extent conversion
        commit: 95ad8ee45cdbc321c135a2db895d48b374ef0f87
[08/11] ext4: reserved credits for one extent during the folio writeback
        commit: bbbf150f3f85619569ac19dc6458cca7c492e715
[09/11] ext4: replace ext4_writepage_trans_blocks()
        commit: 57661f28756c59510e31543520b5b8f5e591f384
[10/11] ext4: fix insufficient credits calculation in ext4_meta_trans_blocks()
        commit: 5137d6c8906b55b3c7b5d1aa5a549753ec8520f5
[11/11] ext4: limit the maximum folio order
        commit: e14bef2a00b5e9b9a100210ec0676aac384904be

Best regards,
-- 
Theodore Ts'o <tytso@mit.edu>

On Mon, 7 Jul 2025 at 19:53, Zhang Yi <yi.zhang@huaweicloud.com> wrote:
>
> From: Zhang Yi <yi.zhang@huawei.com>
>
> Changes since v3:
>  - Fix the end_pos assignment in patch 01.
>  - Rename mpage_submit_buffers() to mpage_submit_partial_folio(), and
>    fix a left shift out-of-bounds problem in patch 03.
>  - Fix the spelling errors in patch 04.
>  - Add a comment for NULL 'handle' test in
>    ext4_journal_ensure_extent_credits().
>  - Add patch 11 to limit the maximum order of the folio to 2048 fs
>    blocks, prevent the overestimation of reserve journal credits during
>    folios write-back.
> Changes since v2:
>  - Convert the processing of folios writeback in bytes instead of pages.
>  - Refactor ext4_page_mkwrite() and ensure journal credits in
>    ext4_block_write_begin() instead of in _ext4_get_block().
>  - Enhance tracepoints in ext4_do_writepages().
>  - Replace the outdated ext4_da_writepages_trans_blocks() and
>    ext4_writepage_trans_blocks() with the new helper used to reserve
>    credits for a single extent.
> Changes since v1:
>  - Make the write-back process supports writing a partial folio if it
>    exits the mapping loop prematurely due to insufficient sapce or
>    journal credits, it also fix the potential stale data and
>    inconsistency issues.
>  - Fix the same issue regarding the allocation of blocks in
>    ext4_write_begin() and ext4_page_mkwrite() when delalloc is not
>    enabled.
>
> v3: https://lore.kernel.org/linux-ext4/20250701130635.4079595-1-yi.zhang@huaweicloud.com/
> v2: https://lore.kernel.org/linux-ext4/20250611111625.1668035-1-yi.zhang@huaweicloud.com/
> v1: https://lore.kernel.org/linux-ext4/20250530062858.458039-1-yi.zhang@huaweicloud.com/
>
> Original Description
>
> This series addresses the issue that Jan pointed out regarding large
> folios support for ext4[1]. The problem is that the credits calculation
> may insufficient in ext4_meta_trans_blocks() when allocating blocks
> during write back a sufficiently large and discontinuous folio, it
> doesn't involve the credits for updating bitmap and group descriptor
> block. However, if we fix this issue, it may lead to significant
> overestimation on the some filesystems with a lot of block groups.
>
> The solution involves first ensure that the current journal transaction
> has enough credits when we mapping an extent during allocating blocks.
> Then if the credits reach the upper limit, exit the current mapping
> loop, submit the partial folio and restart a new transaction. Finally,
> fix the wrong credits calculation in ext4_meta_trans_blocks(). Please
> see the following patches for details.

I have applied this patch set on top of the Linux next tree and performed
testing. The previously reported regressions [a] are no longer observed.
Thank you for providing the fix.

Tested-by: Linux Kernel Functional Testing <lkft@linaro.org>

Reference link:
[a] https://lore.kernel.org/all/CA+G9fYsyYQ3ZL4xaSg1-Tt5Evto7Zd+hgNWZEa9cQLbahA1+xg@mail.gmail.com/

>
> [1] https://lore.kernel.org/linux-ext4/ht54j6bvjmiqt62xmcveqlo7bmrunqs4ji7wikfteftdjijzek@7tz5gpejaoen/
>
> Thanks,
> Yi.
>
> Zhang Yi (11):
>   ext4: process folios writeback in bytes
>   ext4: move the calculation of wbc->nr_to_write to mpage_folio_done()
>   ext4: fix stale data if it bail out of the extents mapping loop
>   ext4: refactor the block allocation process of ext4_page_mkwrite()
>   ext4: restart handle if credits are insufficient during allocating
>     blocks
>   ext4: enhance tracepoints during the folios writeback
>   ext4: correct the reserved credits for extent conversion
>   ext4: reserved credits for one extent during the folio writeback
>   ext4: replace ext4_writepage_trans_blocks()
>   ext4: fix insufficient credits calculation in ext4_meta_trans_blocks()
>   ext4: limit the maximum folio order
>
>  fs/ext4/ext4.h              |   4 +-
>  fs/ext4/extents.c           |   6 +-
>  fs/ext4/ialloc.c            |   3 +-
>  fs/ext4/inline.c            |   6 +-
>  fs/ext4/inode.c             | 349 +++++++++++++++++++++++-------------
>  fs/ext4/move_extent.c       |   3 +-
>  fs/ext4/xattr.c             |   2 +-
>  include/trace/events/ext4.h |  47 ++++-
>  8 files changed, 272 insertions(+), 148 deletions(-)
>
> --
> 2.46.1

--
Linaro LKFT
https://lkft.linaro.org