[PATCH 0/5] ceph: reduce mdsc->mutex contention in the cephfs kclient

Xiubo Li via B4 Relay posted 5 patches 1 week, 5 days ago
There is a newer version of this series
fs/ceph/debugfs.c    |   6 +-
fs/ceph/mds_client.c | 207 +++++++++++++++++++++++++++------------------------
fs/ceph/mds_client.h |   8 +-
3 files changed, 117 insertions(+), 104 deletions(-)
[PATCH 0/5] ceph: reduce mdsc->mutex contention in the cephfs kclient
Posted by Xiubo Li via B4 Relay 1 week, 5 days ago
This series reduces mdsc->mutex hold times from hundreds of
microseconds to tens of microseconds on the hot request-submit and
reply-handling paths.

The approach is incremental:
  1. Convert oldest_tid to atomic64_t so that __prepare_send_request()
     and __send_request() no longer need the mutex.
  2. Replace the request_tree rbtree with an xarray for O(1) lookups
     and internally-locked iteration.
  3. Add a dedicated wait_list_lock spinlock so wait-list operations
     no longer depend on the global mutex.
  4. Move mdsc->mutex acquisition inside __do_request(), then release
     it during the send phase (message construction and path walking),
     leaving only the brief setup/teardown under the lock.
  5. Narrow the mutex scope in replay_unsafe_requests() similarly.

Tested with concurrent readdir + stat on a 5000-file directory
(32 threads).  bpftrace measurements show:

                 before         after
  __do_request   354-2327 us    10-68 us    (34x)
  handle_reply   51-416 us     10-32 us    (13x)
  submit_request 89-211 us     10-41 us    (5x)

No functional changes intended.

Signed-off-by: Xiubo Li <xiubo.li@clyso.com>
---
Xiubo Li (5):
      ceph: convert oldest_tid to atomic64_t
      ceph: replace the request_tree rbtree with an xarray keyed by r_tid.
      ceph: add wait_list_lock for wait-list serialization
      ceph: move mdsc->mutex into __do_request()
      ceph: narrow mdsc->mutex scope in replay_unsafe_requests

 fs/ceph/debugfs.c    |   6 +-
 fs/ceph/mds_client.c | 207 +++++++++++++++++++++++++++------------------------
 fs/ceph/mds_client.h |   8 +-
 3 files changed, 117 insertions(+), 104 deletions(-)
---
base-commit: 7e1f9e2cd2d0e780c394a4402c40e125109fec72
change-id: 20260713-ceph-mdsc-mutex-optimization-7e74ab6bbc8b

Best regards,
--  
Xiubo Li <xiubo.li@clyso.com>