From nobody Sat Sep 26 23:31:07 2026 Received: from mail-wr1-f41.google.com (mail-wr1-f41.google.com [209.85.221.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B809C395D8E for ; Fri, 28 Aug 2026 17:45:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.41 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787939119; cv=none; b=eynftaySWBD3VU5ngDTxOjGlf1t05sOkPgPlzePffjIeZ5DZFPNflt/z+UdFiGhP6IYplwu0ojjOLkdMdoV4NPQSUTjboG3c3wzFaQ6z/P+gbJIU/sxcLrSV72tdD2h1U6NR/bfpCvhW8JibRvUw46yLhgG7M0lUNJKxj2fdqjE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787939119; c=relaxed/simple; bh=htZYsknmaKglxEdP11yJuMBy0iYQlKpuXbCq7oMS5fg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=KnEuobAXg43Wf8SOHUakp1hjRM6KThhTTVD5NS8pU1yn7Ra5UKUpfw1QjerLk+MJYC/AboVrordPLXYSGvdqGJivwptYfh49XDWqpBJckQzD3biveCG/3DKyRSz9SWPbcVy4IGO9ZcZyO+xOdBI9ePnGM+1e8RiaJHASc6gIjng= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=ionos.com; spf=pass smtp.mailfrom=ionos.com; dkim=pass (2048-bit key) header.d=ionos.com header.i=@ionos.com header.b=e06FYy4/; arc=none smtp.client-ip=209.85.221.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=ionos.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=ionos.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ionos.com header.i=@ionos.com header.b="e06FYy4/" Received: by mail-wr1-f41.google.com with SMTP id ffacd0b85a97d-482db627cd8so836636f8f.1 for ; Fri, 28 Aug 2026 10:45:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ionos.com; s=google; t=1787939113; x=1788543913; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=ldhPBd6gTP49jgwzRdZBbN0z+sgukl9cHOVoHnp8Uvo=; b=e06FYy4/Zwtrh3EYhn7yN77Fh5F/8wpSLc/5/0XIrUBOWiRQmmh3D9PaXMyHvpwWHo ILKODPj6DSsYFolMMn3rK06fAVQgXdxX1lQ+Uiu586GmdBQaOEf94IvqHNeWafBdHFPG Vo6s8TXEV4gUIHS1QLl4Ejx2fwU/ixu46hKsPLsmF3LFhMa39XmdMHfH9LxFsmA1hkut y9GOUVjDE2uQI5XgcYm+nd1Q3Mgy+gbHgFgHSeBDBQEovn122rDTuP2GjkUlywRF2znx XrqsNYws1WrUbaJtSj3tUE/bX7bAVUxBWiqvI+PZx0s+t1xC5rUVSrZ++jSDJrmPUc9C xDAg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787939113; x=1788543913; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=ldhPBd6gTP49jgwzRdZBbN0z+sgukl9cHOVoHnp8Uvo=; b=kX3geaxBNCQoWrEmP83jBiCHGYtmtML1AGlcn0NiZnBb9xqFVbRk/SFr5oVnCB5qv3 MhMzM/7lORgvWqiwHM2Hu/ySgMdet8vLMcS4RVAChvIHIw9srD+KHuyfF8QCCv+YUg9/ hrH7SU5jdxiI4AjeILpMf1OWbo7C2OdkbIs4+JEKb9dcu5R7WQTZTY3MpId9yaULBFC+ dLpLPWbWqCGMhlNOGOrN9CLLG2gKHmN+fhfgt30cuGL57v1rdQEzBaVMIt4CcS0Vqsfl okCAtOiIka2XZEaga2aLHaWjeQi7EB+u+g96YKakhdnOMXXKDHjHOIFhXvvxciLjj6Ib P1Rg== X-Forwarded-Encrypted: i=1; AHgh+Rqe7E2p9MQb4kpK/FWeSI64G6YkdT4V7xp/ryx6X12LZ+QyII/U/7WmUuRFYymnTCXu/MipyPjdKLRgnUs=@vger.kernel.org X-Gm-Message-State: AFuF++lcRaz+EL926auZMq7h6pe2ojM8M4abqLCMUHDdift77Ywh/YMf exfcwdN73eMqJjiQVSmXkV+OXLezRfnxfc3/0HuzXJt5kPmN5912NO77JqSRjovdF5w= X-Gm-Gg: AR+sD11SzmEzakki/R1rC6MukUouLr6mPJQVb6Af+sXHSE3aGRrYfR34NawbCfuZhRn H5v95CJ2ra78EgMM/VITe2QlGImm9lwAme98YPhp/zDEbX5b4q6EZOIyXugU9YIlqmXAOLUHj1D Qut4SCPyuZWTe6o0+A8Zcg7JjwiPh9XgUSDO0YtroHYTwEWU/9UC99X96g8poW07N/p6mCIKxND 0H9R6sJqe71XMYYGecLImfL9WrQjVzV9PED/LKltOszESRhvKLAp/7NNu+X3BBfX6SkT7PIHcH7 koQO6ND17XScrn6MF2ogBTWQNP1cOz3LiXg6gZReud4gFkcjNYfWhNe5hM4QIi9sIulzQVlgmpo 9N5MrGFwWb60T8BtYC7634HxEwnY+vAwutO1vLoKmPE8CFPeWJSJY8WipmGxqNVjE0FDPBrw1ct /JzHCp4nixBJgJbqZ/uE89ifkGXEtvqO3bNbqIPF8nzhHnqGcffYoBu4YJ1wTxGWa7s3rHRL8zJ mdXD1NoVgXinUHnkLIlohhUP4ncnR703Cr1SuP96c+aN8TqfnyvnyeE5wOVAH2sWBsbzIdOUMs= X-Received: by 2002:a05:6000:18a6:b0:482:ea08:8c97 with SMTP id ffacd0b85a97d-482f79cb828mr11667503f8f.2.1787939113387; Fri, 28 Aug 2026 10:45:13 -0700 (PDT) Received: from raven.intern.cm-ag (p200300dc6f02b200023064fffe740809.dip0.t-ipconnect.de. [2003:dc:6f02:b200:230:64ff:fe74:809]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-482fbb2794dsm5480678f8f.25.2026.08.28.10.45.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 Aug 2026 10:45:13 -0700 (PDT) From: Max Kellermann To: idryomov@gmail.com, amarkuze@redhat.com, xiubo.li@clyso.com, ceph-devel@vger.kernel.org, linux-kernel@vger.kernel.org Cc: Max Kellermann Subject: [PATCH v3 1/3] ceph/mds_client: pin sessions while checking a new MDS map Date: Fri, 28 Aug 2026 19:45:02 +0200 Message-ID: <20260828174504.1247038-2-max.kellermann@ionos.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20260828174504.1247038-1-max.kellermann@ionos.com> References: <20260828174504.1247038-1-max.kellermann@ionos.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" check_new_map() drops `mdsc->mutex` while it locks a session, prepares a reconnect, or cleans up caps. A concurrent teardown can unregister and release a session, invalidating check_new_map()'s local variable `s`. Fix this by taking a temporary session reference using __ceph_lookup_mds_session() instead of accessing the `sessions[]` array directly. Signed-off-by: Max Kellermann --- fs/ceph/mds_client.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/fs/ceph/mds_client.c b/fs/ceph/mds_client.c index 3c692ad02c85..160f23e2edd3 100644 --- a/fs/ceph/mds_client.c +++ b/fs/ceph/mds_client.c @@ -5798,9 +5798,9 @@ static void check_new_map(struct ceph_mds_client *mds= c, } =20 for (i =3D 0; i < oldmap->possible_max_rank && i < mdsc->max_sessions; i+= +) { - if (!mdsc->sessions[i]) + s =3D __ceph_lookup_mds_session(mdsc, i); + if (!s) continue; - s =3D mdsc->sessions[i]; oldstate =3D ceph_mdsmap_get_state(oldmap, i); newstate =3D ceph_mdsmap_get_state(newmap, i); =20 @@ -5813,7 +5813,6 @@ static void check_new_map(struct ceph_mds_client *mds= c, =20 if (i >=3D newmap->possible_max_rank) { /* force close session for stopped mds */ - ceph_get_mds_session(s); __unregister_session(mdsc, s); __wake_requests(mdsc, &s->s_waiting); mutex_unlock(&mdsc->mutex); @@ -5841,6 +5840,7 @@ static void check_new_map(struct ceph_mds_client *mds= c, mutex_unlock(&s->s_mutex); s->s_state =3D CEPH_MDS_SESSION_RESTARTING; } else if (oldstate =3D=3D newstate) { + ceph_put_mds_session(s); continue; /* nothing new with this mds */ } =20 @@ -5878,6 +5878,7 @@ static void check_new_map(struct ceph_mds_client *mds= c, mutex_unlock(&s->s_mutex); wake_up_session_caps(s, RECONNECT); } + ceph_put_mds_session(s); } =20 /* --=20 2.47.3 From nobody Sat Sep 26 23:31:07 2026 Received: from mail-wr1-f46.google.com (mail-wr1-f46.google.com [209.85.221.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1C6F43B2FDB for ; Fri, 28 Aug 2026 17:45:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.46 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787939121; cv=none; b=vA0PF8j1XgG5jZV4ux8DTSRPHt4xLZaeJtWFN5NnHw1sanzEjxrnYmfVD0hm2M1LTiURiWRPHSJjK5qffcRUXMoC8QbOKcs4RbodkVlimtYb/ZM0e4daGNxDj48ruT+1YpBsi+PmkPkV2xglolntBX6h5kRReO5Wo2pdfKbHDlU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787939121; c=relaxed/simple; bh=r4WMTTppi1E36eQj/o9V1iKpwfdz5dGoX1URXLna3jo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=I1+wQa9QC6Ep33y6zlL+LGAHXdwN5lu7vjxUYuQFLS0RdZ6m0CtPPaPmuzgEWVzPvOvFYTQuZRBfKWrT7lbGiCyi7dlzEw7b59DXVPVVdPQJPEgw63XumWxgfFTCpnTltjqmEn4O5DcaKvQ4rvRNPhCLSDOi1YB8H1KmlnGpVvE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=ionos.com; spf=pass smtp.mailfrom=ionos.com; dkim=pass (2048-bit key) header.d=ionos.com header.i=@ionos.com header.b=HLHclhpG; arc=none smtp.client-ip=209.85.221.46 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=ionos.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=ionos.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ionos.com header.i=@ionos.com header.b="HLHclhpG" Received: by mail-wr1-f46.google.com with SMTP id ffacd0b85a97d-47f71156e1aso468513f8f.3 for ; Fri, 28 Aug 2026 10:45:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ionos.com; s=google; t=1787939114; x=1788543914; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=wJu5xRqwJGsup27bnOejvK4Mm0h/OhLvcqFDdKFUl8c=; b=HLHclhpGQb6MA8D+ACV/vm4X5zjhVbllZtuUlsEevii8k3ttgV8qqfCEqUf8vwEzkc G4EAYNhmkmU9ft8jPkro1KjLjVjWna7NzRcfIY7mh6cFl9qZK09whdmMdEm+DSOmTqA0 ALQP0mAKRs+JjsDjDQvBey48ybJ8TJYEroboItnh0K1WHioQEuSk5zW75ZQ1Bn6FMIE7 6Oi0ruekR2zmrXPjqh+Snv6wmbIXcbACKxJ4/5OsxloIZwp3zgaAt0HE36s3AgVEIpC3 WekfQML1ozryISl2oMSgvNkBQ1/iOlDWChtxdArYszUPPCG14dtTa4DJUqiXc6uRyZOT RPqQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787939114; x=1788543914; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=wJu5xRqwJGsup27bnOejvK4Mm0h/OhLvcqFDdKFUl8c=; b=OXJ0Zc3m09H5a3LvGA7DuEaulGEZhxprG3u/Fi6D4BnjP1AraMX7vF6CHE/Y4ER/2t tWUTpa5I7JVRu3f9Ga/lAFV03DJmcrohXEjRITb7UrJI9LXv5r6IAhAq03PZuzAiUULQ C/0BlaS0pf3s0yTPfuGBT6JfPRXcuHhlD7p3MsjUG5ZE9yGajfOOx1ThOOpRc9xaMT7t 9B75leto765hi+FXa8XtEL9AFLlSL+4ZAI40Kotng/nvABcGwuHEu95Tk+4nkpsuc8o/ OQDdgS2EygqK6Kc/kqA0hk0RimxxX9UYd0EmkEhFnfPCnC7tWSAQJCRA/dRu+qShjlWz B7TQ== X-Forwarded-Encrypted: i=1; AHgh+RqF0j9UKvft/YJT/uZrvJuFnu7CIO6BdMg2PBMALjSAeqiIva3Nmjb7vag3SCUMle6g364gG+AnP9vRt3A=@vger.kernel.org X-Gm-Message-State: AFuF++luS7GKYrioJA+CyowiQ6cPODLp9hTegITj8hqzoNBNw4HmC0yS wHXNe4uncESD0rXzNvyum5GljJnibz6/IF1++Fgy6cMKhJYRDXdUk6FswKiJdfpFMRS2XWeuE8Z shvhjE4s= X-Gm-Gg: AR+sD11Nte7WxHOjcSbEp9P8+Ata5zpuZb8HSkyxjoYJlcaZ43IQsexHJQ4BAhG18V5 lEViEX5Hf8XW7vuIrIYJIYyWKytUd9+ljUambPQK3ZY3oxbehrpaFSBdBQqANskMWjWxczNT8OJ Y/2PvTfWiF5Cc348v+ssDakv2Kd2Pa2TCd+0OPM2hzhwh2K3ugJJTR2l7QBGFgZYvVNcM2ngIdm 2S8FcNOEkjrtmIeNlPIuoi+DJwsDyNiqmYvQadGkx2odrd+0nujdxPDGYDwdkGEOqSl0AYDScWS jpmZkuBSoGa0PI9abxbma6DlCHm9+j6kLJrFDFfYB6D+Mjm4Q3goDdOVJZamnMGsuWm9AmFe2T3 HifaacojlzaECbXsk1vzXEW9sjTMpWs0e7X8kinWYCjGLoZcNBtn8K8pa+UeSt3UiR7533uMLh0 6kjKtCIHIdaw6nTg+WZI8xQesQkOuJ76+nzxCjI3MZSVGTYexGiNXGTKg7hkMDJalo3BBNNXUsi fZGnpmQTE3NYMvE2NaUuuz7MlSmFMYF+xUDmRZE53Ow2hV2QB+a1dlKAzckjtAV X-Received: by 2002:a05:6000:41d1:b0:482:f4f8:7a00 with SMTP id ffacd0b85a97d-482f79cf9fbmr15323631f8f.16.1787939114063; Fri, 28 Aug 2026 10:45:14 -0700 (PDT) Received: from raven.intern.cm-ag (p200300dc6f02b200023064fffe740809.dip0.t-ipconnect.de. [2003:dc:6f02:b200:230:64ff:fe74:809]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-482fbb2794dsm5480678f8f.25.2026.08.28.10.45.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 Aug 2026 10:45:13 -0700 (PDT) From: Max Kellermann To: idryomov@gmail.com, amarkuze@redhat.com, xiubo.li@clyso.com, ceph-devel@vger.kernel.org, linux-kernel@vger.kernel.org Cc: Max Kellermann Subject: [PATCH v3 2/3] ceph/mds_client: stop checking a stale MDS map after dropping mutex Date: Fri, 28 Aug 2026 19:45:03 +0200 Message-ID: <20260828174504.1247038-3-max.kellermann@ionos.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20260828174504.1247038-1-max.kellermann@ionos.com> References: <20260828174504.1247038-1-max.kellermann@ionos.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" check_new_map() drops `mdsc->mutex` in several slow paths. Another map handler can install a newer map and destroy the map being checked before the original invocation re-locks the mutex. Use the `mdsmap->m_epoch` field to identify stale maps. Signed-off-by: Max Kellermann --- fs/ceph/mds_client.c | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/fs/ceph/mds_client.c b/fs/ceph/mds_client.c index 160f23e2edd3..03809328e4aa 100644 --- a/fs/ceph/mds_client.c +++ b/fs/ceph/mds_client.c @@ -5782,6 +5782,7 @@ static void check_new_map(struct ceph_mds_client *mds= c, struct ceph_mdsmap *newmap, struct ceph_mdsmap *oldmap) { + u32 map_epoch =3D newmap->m_epoch; int i, j, err; int oldstate, newstate; struct ceph_mds_session *s; @@ -5825,6 +5826,8 @@ static void check_new_map(struct ceph_mds_client *mds= c, ceph_put_mds_session(s); =20 mutex_lock(&mdsc->mutex); + if (mdsc->mdsmap->m_epoch !=3D map_epoch) + return; kick_requests(mdsc, i); continue; } @@ -5836,6 +5839,11 @@ static void check_new_map(struct ceph_mds_client *md= sc, mutex_unlock(&mdsc->mutex); mutex_lock(&s->s_mutex); mutex_lock(&mdsc->mutex); + if (mdsc->mdsmap->m_epoch !=3D map_epoch) { + mutex_unlock(&s->s_mutex); + ceph_put_mds_session(s); + return; + } ceph_con_close(&s->s_con); mutex_unlock(&s->s_mutex); s->s_state =3D CEPH_MDS_SESSION_RESTARTING; @@ -5859,6 +5867,10 @@ static void check_new_map(struct ceph_mds_client *md= sc, "mds%d reconnect failed: %d\n", i, rc); mutex_lock(&mdsc->mutex); + if (mdsc->mdsmap->m_epoch !=3D map_epoch) { + ceph_put_mds_session(s); + return; + } } =20 /* @@ -5874,6 +5886,11 @@ static void check_new_map(struct ceph_mds_client *md= sc, mutex_unlock(&mdsc->mutex); mutex_lock(&s->s_mutex); mutex_lock(&mdsc->mutex); + if (mdsc->mdsmap->m_epoch !=3D map_epoch) { + mutex_unlock(&s->s_mutex); + ceph_put_mds_session(s); + return; + } ceph_kick_flushing_caps(mdsc, s); mutex_unlock(&s->s_mutex); wake_up_session_caps(s, RECONNECT); @@ -5929,6 +5946,8 @@ static void check_new_map(struct ceph_mds_client *mds= c, i, err); ceph_put_mds_session(s); mutex_lock(&mdsc->mutex); + if (mdsc->mdsmap->m_epoch !=3D map_epoch) + return; } =20 for (i =3D 0; i < newmap->possible_max_rank && i < mdsc->max_sessions; i+= +) { --=20 2.47.3 From nobody Sat Sep 26 23:31:07 2026 Received: from mail-wr1-f43.google.com (mail-wr1-f43.google.com [209.85.221.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0DB873B0586 for ; Fri, 28 Aug 2026 17:45:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.43 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787939123; cv=none; b=p14DTy6mDhqDqAeEMx1DGVHIvzoxR3RI7TFg7KJJZCfQBI23U1EcsTKQ55AASpMGG5VDzITJgfY2Ip1YBU6Uba7Jl8pppt9zQzAJA9mvGV5aFoWLqVe8dH8HGATmR1KpPiW1KGj4JzPybGiV3kW5xtk9rxrRs7/BPVIm8Q/LA1o= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787939123; c=relaxed/simple; bh=h+BpGgpvdKB55f0B5VB6T0U7Da2OYH7jV3SSwp/hjCY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=pTZv37EKJbGDOpLLlS3c390kianZI8p95A3N+fykt7tpSUyBU8/IZL9vS993E1p38Z7kfI0nPndNTg2lfTSvdVQWr2mtyzbaQPiX71Q8Q/70ovsyACsPQQtU+AFMyApolw/sQSPfTixvj2zKj4uK6DOveVy7ed4lPEDb8r5mFoc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=ionos.com; spf=pass smtp.mailfrom=ionos.com; dkim=pass (2048-bit key) header.d=ionos.com header.i=@ionos.com header.b=Hk6MPKNR; arc=none smtp.client-ip=209.85.221.43 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=ionos.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=ionos.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ionos.com header.i=@ionos.com header.b="Hk6MPKNR" Received: by mail-wr1-f43.google.com with SMTP id ffacd0b85a97d-47ddf7b09e5so1127435f8f.1 for ; Fri, 28 Aug 2026 10:45:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ionos.com; s=google; t=1787939115; x=1788543915; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=rJMkwM6Vbb2kq+duXqwTI1+B7IDTJdsELVF1bs1+FLc=; b=Hk6MPKNRZ5i/98yla2ivxIIpQF8MJyjXWpOLrz3nvMxD9LJ+mbckaIvAAFy+1bq5me AdtspL2mUYr+CX+sg969wiNnN8IbZvc+Ln0Du/UDaLWYy9ovFFjXysyEnSSYGBOJ0cdw eVH3Lxm0ia89FJt2SDXtaV1/gtcZL0cA+hUDOODjwwGen9vwzwMIWV6KPP8viqotFxgl kN3TVOTC2ZTjEJ5PXXiaNtnmj7qc2ao1Mtc0MC5FfK2bW1dxHsOcpPQfH+QilyA4KDvK XtOZS2EUT+hTTeR2etRayOwhHTR8gmGBuEOnp7M8xqSMy62YaeX8QmtTeFWD/GSoJmtT uSyg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787939115; x=1788543915; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=rJMkwM6Vbb2kq+duXqwTI1+B7IDTJdsELVF1bs1+FLc=; b=ULEXNnfgMKZZpsOBLv/JTQGGd1VEOwnZJuxG2flxdTEUoOjbtSiG1GdTCHSVIu+zZ5 AfrRgfEuvsfZzUU+wrmy6wvyAr/lGkYdZnPKt79Y5kiLbhlhlOdDYC3hr5Uspk/0qYXU kCrpo711G1KezC3rygBOvthNRcjERbxXlKwiMCrtdjqpEG1u1zY+1bjJt4D+CweYZqhV Tq7R6QYLfyL+qKWFwAlvAU5mHExt2tinFFUSbXqU7oZ8HVNZjPVpvjzeAXeWViQ1v/Jk t9SCjB8gUXxTYeZ1tT5Su1tvII+x2zjibE7IkwVPHgDgfrmjSpVET8rKX5UMpNOPqdmj u8vg== X-Forwarded-Encrypted: i=1; AHgh+Rrl77lcTQIkdm4Vdq6V1CI0z2Zhu5KRWV+3DzSFZzkHi86WEejgDqdKY+AqP9CSKe5JDzs4kaMYN7lGIwU=@vger.kernel.org X-Gm-Message-State: AFuF++lXvYk4PFw5zEVVeITFXBqDpCof041gz+wV1UBuJWKArcyL/HnI 181A38mImqAmsF859ElPt18oWRICH/yNGZXG88GX08wgCGJTSmHb/oIiT9BLmNYl+Dg= X-Gm-Gg: AR+sD10FGmsH7rd0ZpA1oIFRN0AKfqsxaf3bBtPaVHhEVT2sAniTPmrs6+9NbVA6IZc SiX3dLm/gxQPyvpNa5ccYYFddPbdbmKSPfOzVMouLaweCCLvAY4SGg2rb4CXj3D3HCr9jAVMuya t39C9NZz4NZn4b3NzQr31UspXMl9Z/0Q8N+RU0dBk2vWedngZ2IkwVu7g368lPyZx+MmVO2hoOn mTvWjR8Tyovpwf+T/JXhCp+Vfn7b2h7Pf5vfR2ELC9HdNo7T2IhA7UqTV+p/+sK2IIM6L1MNd9f gCM7FljLdNFo1uY7EJIuMdBIBhMriGIvESskptjLC0V2qK4jLciFszEBH16MnA8rDNjH14mrt4h fdk45AwymFLbMqPHf6wtBK28DDAk2R3JM6fNfFR8jjMnve0UbLG0NaMvuczWFePlfgo3X8Vo5bw EecyTfFI1kdRcq8Jp3hW2xzryK5+tIfOGZLZ5vjUa26pcVldrbIO7RTCitXPux/ln178jrYLMBZ bmLPdv3jn1WI7foe8bO5YubxLHijrOmwJG9CHG5JAxnl7OYnDLCpzyilyeUhtio X-Received: by 2002:a05:6000:18a1:b0:482:bf78:300a with SMTP id ffacd0b85a97d-482f79c78abmr15473232f8f.17.1787939114762; Fri, 28 Aug 2026 10:45:14 -0700 (PDT) Received: from raven.intern.cm-ag (p200300dc6f02b200023064fffe740809.dip0.t-ipconnect.de. [2003:dc:6f02:b200:230:64ff:fe74:809]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-482fbb2794dsm5480678f8f.25.2026.08.28.10.45.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 Aug 2026 10:45:14 -0700 (PDT) From: Max Kellermann To: idryomov@gmail.com, amarkuze@redhat.com, xiubo.li@clyso.com, ceph-devel@vger.kernel.org, linux-kernel@vger.kernel.org Cc: Max Kellermann Subject: [PATCH v3 3/3] ceph: don't unregister an MDS session before removing its caps Date: Fri, 28 Aug 2026 19:45:04 +0200 Message-ID: <20260828174504.1247038-4-max.kellermann@ionos.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20260828174504.1247038-1-max.kellermann@ionos.com> References: <20260828174504.1247038-1-max.kellermann@ionos.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" handle_session() removed the session from mdsc->sessions[] at the very top of `CEPH_SESSION_CLOSE` handling, before taking `s_mutex`. Between session unregistration and remove_session_caps(), the MDS rank has no registered session while the old session still owns all caps it was granted. Any concurrent filesystem operation may walk into that and the next __do_request() call registers a new session for this rank. Once it is open and the MDS issues caps, ceph_fill_inode() calls ceph_add_cap(), which looks caps up by rank, not by session identity, finding old caps linked to the old session. The list_move_tail() call then moves the cap object to the new session, which is already a bad thing to do. Since it doesn't decrement `old_session->s_nr_caps`, this will quickly run into a BUG() instead of crashing: kernel BUG at fs/ceph/mds_client.c:1959! Internal error: Oops - BUG: 00000000f2000800 [#1] SMP [...] Workqueue: ceph-msgr ceph_con_workfn pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=3D--) pc : remove_session_caps+0x2bc/0x2d8 lr : remove_session_caps+0x74/0x2d8 [...] Call trace: remove_session_caps+0x2bc/0x2d8 (P) mds_dispatch+0xf48/0x1b60 ceph_con_process_message+0x74/0xa0 ceph_con_v1_try_read+0x3a0/0x1510 ceph_con_workfn+0x260/0x460 process_one_work+0x168/0x3b8 worker_thread+0x1bc/0x3a0 kthread+0x118/0x1e0 ret_from_fork+0x10/0x20 That's BUG_ON(session->s_nr_caps > 0). I was able to reproduce this reliably by delaying the close and starting I/O during the delay. This patch keeps the session registered with `CEPH_MDS_SESSION_CLOSED`. New requests will be put on the `s_waiting` list where they will be resumed on the new session. Signed-off-by: Max Kellermann --- v1->v2: skip CLOSED sessions in check_new_map() v2->v3: split session pinning and stale-map guards into two preparatory patches; rework CLOSED-session and export-target teardown handling Signed-off-by: Max Kellermann --- fs/ceph/caps.c | 3 + fs/ceph/mds_client.c | 138 ++++++++++++++++++++++++++++++++++++++----- 2 files changed, 127 insertions(+), 14 deletions(-) diff --git a/fs/ceph/caps.c b/fs/ceph/caps.c index d7283fb54cec..b21d0a9ac324 100644 --- a/fs/ceph/caps.c +++ b/fs/ceph/caps.c @@ -4195,6 +4195,9 @@ static void handle_cap_export(struct inode *inode, st= ruct ceph_mds_caps *ex, } new_cap =3D ceph_get_cap(mdsc, NULL); } else { + if (tsession =3D=3D ERR_PTR(-EAGAIN)) + return; + WARN_ON(1); tsession =3D NULL; target =3D -1; diff --git a/fs/ceph/mds_client.c b/fs/ceph/mds_client.c index 03809328e4aa..ec8ee95cad6e 100644 --- a/fs/ceph/mds_client.c +++ b/fs/ceph/mds_client.c @@ -1733,6 +1733,30 @@ static int __open_session(struct ceph_mds_client *md= sc, return 0; } =20 +/* Is this rank still occupied by a session being torn down? */ +static bool __mds_rank_closing(struct ceph_mds_client *mdsc, int mds) +{ + return mds < mdsc->max_sessions && mdsc->sessions[mds] && + mdsc->sessions[mds]->s_state =3D=3D CEPH_MDS_SESSION_CLOSED; +} + +/* + * Wait until the rank is no longer occupied by a CLOSED session. + * + * The caller must hold mdsc->mutex. The mutex is dropped while sleeping = and + * held again on return. Another session may occupy the rank by then. The + * wait also ends when shutdown starts, so the caller must check + * mdsc->stopping before proceeding. + */ +static void wait_for_mds_rank_not_closing(struct ceph_mds_client *mdsc, + int mds) +{ + wait_event_cmd(mdsc->session_close_wq, + !__mds_rank_closing(mdsc, mds) || mdsc->stopping, + mutex_unlock(&mdsc->mutex), + mutex_lock(&mdsc->mutex)); +} + /* * open sessions for any export targets for the given mds * @@ -1750,6 +1774,16 @@ __open_export_target_session(struct ceph_mds_client = *mdsc, int target) if (IS_ERR(session)) return session; } + if (session->s_state =3D=3D CEPH_MDS_SESSION_CLOSED) { + /* + * handle_session() is currently closing this session; + * it stays registered until its caps are gone. Do + * not return it to our caller because we don't want + * it to attach new caps to it. + */ + ceph_put_mds_session(session); + return ERR_PTR(-EAGAIN); + } if (session->s_state =3D=3D CEPH_MDS_SESSION_NEW || session->s_state =3D=3D CEPH_MDS_SESSION_CLOSING) { ret =3D __open_session(mdsc, session); @@ -1769,7 +1803,19 @@ ceph_mdsc_open_export_target_session(struct ceph_mds= _client *mdsc, int target) doutc(cl, "to mds%d\n", target); =20 mutex_lock(&mdsc->mutex); - session =3D __open_export_target_session(mdsc, target); + for (;;) { + session =3D __open_export_target_session(mdsc, target); + if (session !=3D ERR_PTR(-EAGAIN) || mdsc->stopping) + break; + + /* + * Keep the exported cap on its old session until the + * target rank is vacant. Dropping it here can discard a + * dirty auth cap; handle_session() wakes us after removing + * the old target session's caps and unregistering it. + */ + wait_for_mds_rank_not_closing(mdsc, target); + } mutex_unlock(&mdsc->mutex); =20 return session; @@ -4492,8 +4538,20 @@ static void handle_session(struct ceph_mds_session *= session, ceph_metric_bind_session(mdsc, session); } if (op =3D=3D CEPH_SESSION_CLOSE) { + /* + * Pin the session for the rest of this function. The + * __unregister_session() call is deferred until after + * remove_session_caps() below, or else other + * processes may find caps still assigned to this + * session while working with a new session object. + */ ceph_get_mds_session(session); - __unregister_session(mdsc, session); + + if (session->s_state =3D=3D CEPH_MDS_SESSION_RECONNECTING) + pr_info_client(cl, "mds%d reconnect denied\n", + session->s_mds); + + session->s_state =3D CEPH_MDS_SESSION_CLOSED; } /* FIXME: this ttl calculation is generous */ session->s_ttl =3D jiffies + HZ*mdsc->mdsmap->m_session_autoclose; @@ -4551,12 +4609,24 @@ static void handle_session(struct ceph_mds_session = *session, break; =20 case CEPH_SESSION_CLOSE: - if (session->s_state =3D=3D CEPH_MDS_SESSION_RECONNECTING) - pr_info_client(cl, "mds%d reconnect denied\n", - session->s_mds); - session->s_state =3D CEPH_MDS_SESSION_CLOSED; cleanup_session_requests(mdsc, session); remove_session_caps(session); + + /* + * Now that all caps are removed, it is safe release + * the MDS rank and allow other processes to create a + * new session object. + * + * A concurrent ceph_mdsc_close_sessions() or + * check_new_map() may have unregistered the session + * already, so check __verify_registered_session() + * first. + */ + mutex_lock(&mdsc->mutex); + if (!__verify_registered_session(mdsc, session)) + __unregister_session(mdsc, session); + mutex_unlock(&mdsc->mutex); + wake =3D 2; /* for good measure */ wake_up_all(&mdsc->session_close_wq); break; @@ -5802,6 +5872,10 @@ static void check_new_map(struct ceph_mds_client *md= sc, s =3D __ceph_lookup_mds_session(mdsc, i); if (!s) continue; + if (s->s_state =3D=3D CEPH_MDS_SESSION_CLOSED) { + ceph_put_mds_session(s); + continue; + } oldstate =3D ceph_mdsmap_get_state(oldmap, i); newstate =3D ceph_mdsmap_get_state(newmap, i); =20 @@ -5814,18 +5888,24 @@ static void check_new_map(struct ceph_mds_client *m= dsc, =20 if (i >=3D newmap->possible_max_rank) { /* force close session for stopped mds */ - __unregister_session(mdsc, s); - __wake_requests(mdsc, &s->s_waiting); + s->s_state =3D CEPH_MDS_SESSION_CLOSED; mutex_unlock(&mdsc->mutex); =20 mutex_lock(&s->s_mutex); cleanup_session_requests(mdsc, s); remove_session_caps(s); + + mutex_lock(&mdsc->mutex); + if (!__verify_registered_session(mdsc, s)) + __unregister_session(mdsc, s); + mutex_unlock(&mdsc->mutex); mutex_unlock(&s->s_mutex); =20 - ceph_put_mds_session(s); + wake_up_all(&mdsc->session_close_wq); =20 mutex_lock(&mdsc->mutex); + __wake_requests(mdsc, &s->s_waiting); + ceph_put_mds_session(s); if (mdsc->mdsmap->m_epoch !=3D map_epoch) return; kick_requests(mdsc, i); @@ -5844,6 +5924,16 @@ static void check_new_map(struct ceph_mds_client *md= sc, ceph_put_mds_session(s); return; } + if (s->s_state =3D=3D CEPH_MDS_SESSION_CLOSED) { + /* + * handle_session() set state=3DCLOSED + * in the mutex gap above and is tearing it + * down + */ + mutex_unlock(&s->s_mutex); + ceph_put_mds_session(s); + continue; + } ceph_con_close(&s->s_con); mutex_unlock(&s->s_mutex); s->s_state =3D CEPH_MDS_SESSION_RESTARTING; @@ -5902,6 +5992,9 @@ static void check_new_map(struct ceph_mds_client *mds= c, * Only open and reconnect sessions that don't exist yet. */ for (i =3D 0; i < newmap->possible_max_rank; i++) { + if (mdsc->stopping) + return; + /* * In case the import MDS is crashed just after * the EImportStart journal is flushed, so when @@ -5927,6 +6020,19 @@ static void check_new_map(struct ceph_mds_client *md= sc, * reconnection request in up:reconnect state. */ s =3D __ceph_lookup_mds_session(mdsc, i); + if (s && s->s_state =3D=3D CEPH_MDS_SESSION_CLOSED) { + /* + * Wait for handle_session() to remove the caps and + * unregister this session, so the reconnect below + * uses a fresh session on the now vacant rank + */ + ceph_put_mds_session(s); + wait_for_mds_rank_not_closing(mdsc, i); + if (mdsc->stopping || + mdsc->mdsmap->m_epoch !=3D map_epoch) + return; + s =3D NULL; + } if (likely(!s)) { s =3D __open_export_target_session(mdsc, i); if (IS_ERR(s)) { @@ -7087,9 +7193,7 @@ static void mds_peer_reset(struct ceph_connection *co= n) * Snapshot session state with READ_ONCE, then revalidate under * mdsc->mutex before acting. The subsequent mdsc->mutex * section rechecks s_state to catch concurrent transitions, so - * the lockless snapshot here is safe. s->s_mutex is taken - * separately for cleanup after unregistration, which avoids - * introducing a new s->s_mutex + mdsc->mutex nesting. + * the lockless snapshot here is safe. */ session_state =3D READ_ONCE(s->s_state); =20 @@ -7114,18 +7218,24 @@ static void mds_peer_reset(struct ceph_connection *= con) =20 ceph_get_mds_session(s); s->s_state =3D CEPH_MDS_SESSION_CLOSED; - __unregister_session(mdsc, s); - __wake_requests(mdsc, &s->s_waiting); mutex_unlock(&mdsc->mutex); =20 mutex_lock(&s->s_mutex); cleanup_session_requests(mdsc, s); remove_session_caps(s); + + /* Keep the rank occupied until all old-session caps are gone. */ + mutex_lock(&mdsc->mutex); + if (!__verify_registered_session(mdsc, s)) + __unregister_session(mdsc, s); + mutex_unlock(&mdsc->mutex); + mutex_unlock(&s->s_mutex); =20 wake_up_all(&mdsc->session_close_wq); =20 mutex_lock(&mdsc->mutex); + __wake_requests(mdsc, &s->s_waiting); kick_requests(mdsc, s->s_mds); mutex_unlock(&mdsc->mutex); =20 --=20 2.47.3