We hit recurring RCU stalls / soft lockups on a busy ext4 filesystem
that serves as the CacheFiles backing store for a Ceph (fscache)
client.  The stall is always the journal commit thread spinning on
journal->j_list_lock:

  rcu: INFO: rcu_sched self-detected stall on CPU
  rcu:   106-....: (2099 ticks this GP) ... (t=2100 jiffies)
  CPU: 106 Comm: jbd2/nvme1n1p1-
  RIP: 0010:queued_spin_lock_slowpath+0x20a/0x240
  Call Trace:
   jbd2_journal_write_metadata_buffer+0x1c0/0x310
   jbd2_journal_commit_transaction+0x5e2/0x16e0
   kjournald2+0xa1/0x220
   kthread+0xe4/0x1d0

kjournald2 has already passed the "wait for outstanding handles"
barrier and is in the metadata write-out loop; it is simply unable to
acquire j_list_lock for >21s.  The lock holder is the jbd2 checkpoint
shrinker.

Under memory pressure, the shrinker (jbd2_journal_shrink_scan ->
jbd2_journal_shrink_checkpoint_list -> journal_shrink_one_cp_list)
walks a transaction's checkpoint list under j_list_lock.  On this
workload, the lists are large and dominated by busy buffers (dirty /
under writeback / attached to the running transaction), and
journal_shrink_one_cp_list() can therefore hold j_list_lock for a time
proportional to the whole list length, for two reasons:

1. The JBD2_SHRINK_BUSY_SKIP path uses "continue", which also skips
   the need_resched() check, so the scan does not yield even when a
   reschedule is pending.

2. The scan budget (nr_to_scan) is decremented only for buffers that
   are actually freed, so busy buffers do not consume it and the loop
   is not bounded by the shrinker's batch at all.

Both were introduced by commit b98dba273a0e ("jbd2: remove
journal_clean_one_cp_list()"), which folded
journal_clean_one_cp_list() into journal_shrink_one_cp_list() and
dropped the per-examined-buffer budget that the shrinker previously
had.

The result is a self-reinforcing collapse: shrinker instances across
many CPUs hold/contend j_list_lock, kjournald2 cannot commit, the
journal fills, CacheFiles lookups block in ext4, fscache cookies get
stuck in LOOKING_UP, and netfs I/O times out.

Max Kellermann (2):
  jbd2: check need_resched() when skipping busy checkpoint buffers
  jbd2: bound shrinker scans by examined checkpoint buffers

 fs/jbd2/checkpoint.c | 28 +++++++++++++++-------------
 1 file changed, 15 insertions(+), 13 deletions(-)

-- 
2.47.3

On Mon, 13 Jul 2026 12:22:27 +0200, Max Kellermann wrote:
> We hit recurring RCU stalls / soft lockups on a busy ext4 filesystem
> that serves as the CacheFiles backing store for a Ceph (fscache)
> client.  The stall is always the journal commit thread spinning on
> journal->j_list_lock:
> 
>   rcu: INFO: rcu_sched self-detected stall on CPU
>   rcu:   106-....: (2099 ticks this GP) ... (t=2100 jiffies)
>   CPU: 106 Comm: jbd2/nvme1n1p1-
>   RIP: 0010:queued_spin_lock_slowpath+0x20a/0x240
>   Call Trace:
>    jbd2_journal_write_metadata_buffer+0x1c0/0x310
>    jbd2_journal_commit_transaction+0x5e2/0x16e0
>    kjournald2+0xa1/0x220
>    kthread+0xe4/0x1d0
> 
> [...]

Applied, thanks!

[1/2] jbd2: check need_resched() when skipping busy checkpoint buffers
      commit: f213e12ff5c9590b1034ae8da0e6d09665c772d0
[2/2] jbd2: bound shrinker scans by examined checkpoint buffers
      commit: 15cb16496446b94e67f7abcb049b8e2c75cd3d02

Best regards,
-- 
Theodore Ts'o <tytso@mit.edu>

On 7/13/2026 6:22 PM, Max Kellermann wrote:
> We hit recurring RCU stalls / soft lockups on a busy ext4 filesystem
> that serves as the CacheFiles backing store for a Ceph (fscache)
> client.  The stall is always the journal commit thread spinning on
> journal->j_list_lock:
> 
>   rcu: INFO: rcu_sched self-detected stall on CPU
>   rcu:   106-....: (2099 ticks this GP) ... (t=2100 jiffies)
>   CPU: 106 Comm: jbd2/nvme1n1p1-
>   RIP: 0010:queued_spin_lock_slowpath+0x20a/0x240
>   Call Trace:
>    jbd2_journal_write_metadata_buffer+0x1c0/0x310
>    jbd2_journal_commit_transaction+0x5e2/0x16e0
>    kjournald2+0xa1/0x220
>    kthread+0xe4/0x1d0

Hi, Max!

Thanks for the report!

I'm a bit curious about which kernel version and CONFIG you
encountered this issue on. IIRC, after commits [1] and [2], the
spin_needbreak() in jbd2_journal_shrink_checkpoint_list() should
always be effective (unless the hardware does not support it, i.e.,
ARCH_NO_PREEMPT). Moreover, the number of buffer_heads in the
checkpoint list of a single transaction should not be large enough
to trigger such soft lockups. So I suspect this issue should be
unlikely to occur on the latest kernel releases.

[1] 7dadeaa6e851 ("sched: Further restrict the preemption modes")
[2] 7c70cb94d29c ("sched: Add Lazy preemption model")

Thanks,
Yi.

> 
> kjournald2 has already passed the "wait for outstanding handles"
> barrier and is in the metadata write-out loop; it is simply unable to
> acquire j_list_lock for >21s.  The lock holder is the jbd2 checkpoint
> shrinker.
> 
> Under memory pressure, the shrinker (jbd2_journal_shrink_scan ->
> jbd2_journal_shrink_checkpoint_list -> journal_shrink_one_cp_list)
> walks a transaction's checkpoint list under j_list_lock.  On this
> workload, the lists are large and dominated by busy buffers (dirty /
> under writeback / attached to the running transaction), and
> journal_shrink_one_cp_list() can therefore hold j_list_lock for a time
> proportional to the whole list length, for two reasons:
> 
> 1. The JBD2_SHRINK_BUSY_SKIP path uses "continue", which also skips
>    the need_resched() check, so the scan does not yield even when a
>    reschedule is pending.
> 
> 2. The scan budget (nr_to_scan) is decremented only for buffers that
>    are actually freed, so busy buffers do not consume it and the loop
>    is not bounded by the shrinker's batch at all.
> 
> Both were introduced by commit b98dba273a0e ("jbd2: remove
> journal_clean_one_cp_list()"), which folded
> journal_clean_one_cp_list() into journal_shrink_one_cp_list() and
> dropped the per-examined-buffer budget that the shrinker previously
> had.
> 
> The result is a self-reinforcing collapse: shrinker instances across
> many CPUs hold/contend j_list_lock, kjournald2 cannot commit, the
> journal fills, CacheFiles lookups block in ext4, fscache cookies get
> stuck in LOOKING_UP, and netfs I/O times out.
> 
> Max Kellermann (2):
>   jbd2: check need_resched() when skipping busy checkpoint buffers
>   jbd2: bound shrinker scans by examined checkpoint buffers
> 
>  fs/jbd2/checkpoint.c | 28 +++++++++++++++-------------
>  1 file changed, 15 insertions(+), 13 deletions(-)
>

On Mon, Jul 13, 2026 at 2:53 PM Zhang Yi <yi.zhang@huaweicloud.com> wrote:
> I'm a bit curious about which kernel version and CONFIG you
> encountered this issue on. IIRC, after commits [1] and [2], the
> spin_needbreak() in jbd2_journal_shrink_checkpoint_list() should
> always be effective (unless the hardware does not support it, i.e.,
> ARCH_NO_PREEMPT).

We run 6.18.38, and preemption is disabled. Therefore,
spin_needbreak() is a no-op.
(We used to always run the latest stable, but got burned by way too
many critical Ceph/netfs regressions and settled on the LTS.)

But even with preemption enabled, this would likely be a problem,
though a much smaller one.

> Moreover, the number of buffer_heads in the
> checkpoint list of a single transaction should not be large enough
> to trigger such soft lockups. So I suspect this issue should be
> unlikely to occur on the latest kernel releases.

Unfortunately, I don't have any numbers for you, because this problem
occurs only every other week, but we have been haunted by this for a
long time.

The setup is a 7 TB ext4 partition dedicated to fscache/Ceph. It's
always full, and culling is running quite often.

For culling, we don't use cachefilesd but our own reimplementation
(https://github.com/CM4all/cash) because cachefilesd scales
quadratically and never makes any progress while burning CPU cycles
and I/O forever; ours is massively parallel thanks to io_uring. That
puts a lot of pressure on the ext4 filesystem, and a cgroup
memory.high setting means it's always under memory pressure (or else
the kernel will accumulate many gigabytes of RAM usage very quickly
during the filesystem scan that is necessary for culling). Which means
the shrinker is running all the time. Plus the usual kernel-initiated
I/O on that partition.
Every time this problem occurred, fscache culling was active.

-- 
Max Kellermann
Principal Architect
Hosting Technology

cm4all | Im Mediapark 6a | 50670 Köln | Germany
General information about the company can be found here:
https://www.cm4all.com/impressum
A member of the IONOS Group

On 7/13/2026 11:03 PM, Max Kellermann wrote:
> On Mon, Jul 13, 2026 at 2:53 PM Zhang Yi <yi.zhang@huaweicloud.com> wrote:
>> I'm a bit curious about which kernel version and CONFIG you
>> encountered this issue on. IIRC, after commits [1] and [2], the
>> spin_needbreak() in jbd2_journal_shrink_checkpoint_list() should
>> always be effective (unless the hardware does not support it, i.e.,
>> ARCH_NO_PREEMPT).
> 
> We run 6.18.38, and preemption is disabled. Therefore,
> spin_needbreak() is a no-op.
> (We used to always run the latest stable, but got burned by way too
> many critical Ceph/netfs regressions and settled on the LTS.)
> 
> But even with preemption enabled, this would likely be a problem,
> though a much smaller one.
> 
>> Moreover, the number of buffer_heads in the
>> checkpoint list of a single transaction should not be large enough
>> to trigger such soft lockups. So I suspect this issue should be
>> unlikely to occur on the latest kernel releases.
> 
> Unfortunately, I don't have any numbers for you, because this problem
> occurs only every other week, but we have been haunted by this for a
> long time.
> 
> The setup is a 7 TB ext4 partition dedicated to fscache/Ceph. It's
> always full, and culling is running quite often.
> 
> For culling, we don't use cachefilesd but our own reimplementation
> (https://github.com/CM4all/cash) because cachefilesd scales
> quadratically and never makes any progress while burning CPU cycles
> and I/O forever; ours is massively parallel thanks to io_uring. That
> puts a lot of pressure on the ext4 filesystem, and a cgroup
> memory.high setting means it's always under memory pressure (or else
> the kernel will accumulate many gigabytes of RAM usage very quickly
> during the filesystem scan that is necessary for culling). Which means
> the shrinker is running all the time. Plus the usual kernel-initiated
> I/O on that partition.
> Every time this problem occurred, fscache culling was active.
> 

Thank you for the details!

We recently encountered the same issue on older kernels (up to v6.6)
during extreme memory pressure stress testing. Interestingly, we were
unable to reproduce it on the latest v7.1 kernel.

After investigation, we identified that commit 7dadeaa6e851 ("sched:
Further restrict the preemption modes") effectively prevents disabling
preemption entirely, which makes spin_needbreak() always work in
practice. So I was previously curious whether you discovered this issue
in the latest kernel.

However, let's come back to the point. I agree this is a real issue
that deserves a proper fix, independent of the preemption model changes.

Thanks,
Yi.