From nobody Tue Dec 16 23:57:28 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 59F51219A93 for ; Thu, 9 Jan 2025 13:31:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736429468; cv=none; b=LHodEk0QuA8hHrhMhN7xpL0qNpFbBZPLCPuj7SJKKm5hMTsDvDu7adG5wiHVKZESA5NtB0L0ifzHJDCIR8bJHIf1m2CeT5yW3ubNldh7D2itH4booPThfDyQWxWn9+tVZohPzCczml7yBx6x93R4esqcFApdY0DIPhaXCZuydMA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736429468; c=relaxed/simple; bh=3QGicS42gxFxR+GV2Bop0Uf8DmbC5XfA8QuhBkB8Huw=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=n4D5yRESHhZb3qpCRceAUjFecNbZvmcTksi89a0uRl0d7oH8uvOBI36RkSRSG25pWKCKbLhdDdNXZbnkDWxy71ZvQv2rqCUKHvmYBUnfn3lS0wuqaemznzvMCv/2xo41azZ1w+biAsHKv2WfoPImpno72Wt3pnUNM0Xn3lbVrsk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Ti40nWHn; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Ti40nWHn" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9C63BC4CEE2; Thu, 9 Jan 2025 13:31:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1736429468; bh=3QGicS42gxFxR+GV2Bop0Uf8DmbC5XfA8QuhBkB8Huw=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=Ti40nWHnCy/ZXojSWdvg1iO3hRS5GpEC6vp8nynZkVEXi52dLW4EqzjW4Iu+er5h5 VHaUg3j5altZz6OsHIOd84ZJMedlm5AfMl8saD19CI5JOIHt3DGZj7r0df0Z1FG3LF xNRu2Hfax79Wl6uCZlR7kP6jWCys+o0v4QZj5S/PlCCh0QOgTPDVKel6HgJOovlL1T HGCGRBcudEHn2d7OGA08Eo6c7YCyjuNvKT+V1e/J/bxn+H6xEYTlQ6kGcaGHyQ8+MU wE+ZJMhuNbBopQHO718cNn/1SZ88qoCuMOY4apM4qIg2p79ntyhEMxjEdFxI8uhSGQ XJjdE/DagdbJw== From: Daniel Wagner Date: Thu, 09 Jan 2025 14:30:49 +0100 Subject: [PATCH v4 3/3] nvme-fc: do not ignore connectivity loss during connecting Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20250109-nvme-fc-handle-com-lost-v4-3-fe5cae17b492@kernel.org> References: <20250109-nvme-fc-handle-com-lost-v4-0-fe5cae17b492@kernel.org> In-Reply-To: <20250109-nvme-fc-handle-com-lost-v4-0-fe5cae17b492@kernel.org> To: James Smart , Keith Busch , Christoph Hellwig , Sagi Grimberg , Hannes Reinecke , Paul Ely Cc: linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org, Daniel Wagner X-Mailer: b4 0.14.2 When a connectivity loss occurs while nvme_fc_create_assocation is being executed, it's possible that the ctrl ends up stuck in the LIVE state: 1) nvme nvme10: NVME-FC{10}: create association : ... 2) nvme nvme10: NVME-FC{10}: controller connectivity lost. Awaiting Reconnect nvme nvme10: queue_size 128 > ctrl maxcmd 32, reducing to maxcmd 3) nvme nvme10: Could not set queue count (880) nvme nvme10: Failed to configure AEN (cfg 900) 4) nvme nvme10: NVME-FC{10}: controller connect complete 5) nvme nvme10: failed nvme_keep_alive_end_io error=3D4 A connection attempt starts 1) and the ctrl is in state CONNECTING. Shortly after the LLDD driver detects a connection lost event and calls nvme_fc_ctrl_connectivity_loss 2). Because we are still in CONNECTING state, this event is ignored. nvme_fc_create_association continues to run in parallel and tries to communicate with the controller and these commands will fail. Though these errors are filtered out, e.g in 3) setting the I/O queues numbers fails which leads to an early exit in nvme_fc_create_io_queues. Because the number of IO queues is 0 at this point, there is nothing left in nvme_fc_create_association which could detected the connection drop. Thus the ctrl enters LIVE state 4). Eventually the keep alive handler times out 5) but because nothing is being done, the ctrl stays in LIVE state. There is already the ASSOC_FAILED flag to track connectivity loss event but this bit is set too late in the recovery code path. Move this into the connectivity loss event handler and synchronize it with the state change. This ensures that the ASSOC_FAILED flag is seen by nvme_fc_create_io_queues and it does not enter the LIVE state after a connectivity loss event. If the connectivity loss event happens after we entered the LIVE state the normal error recovery path is executed. Signed-off-by: Daniel Wagner Reviewed-by: Hannes Reinecke Reviewed-by: Sagi Grimberg --- drivers/nvme/host/fc.c | 23 ++++++++++++++++++----- 1 file changed, 18 insertions(+), 5 deletions(-) diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c index 7409da42b9ee580cdd6fe78c0f93e78c4ad08675..55884d3df6f291cfddb4742e135= b54a72f1cfa05 100644 --- a/drivers/nvme/host/fc.c +++ b/drivers/nvme/host/fc.c @@ -781,11 +781,19 @@ nvme_fc_abort_lsops(struct nvme_fc_rport *rport) static void nvme_fc_ctrl_connectivity_loss(struct nvme_fc_ctrl *ctrl) { + enum nvme_ctrl_state state; + unsigned long flags; + dev_info(ctrl->ctrl.device, "NVME-FC{%d}: controller connectivity lost. Awaiting " "Reconnect", ctrl->cnum); =20 - switch (nvme_ctrl_state(&ctrl->ctrl)) { + spin_lock_irqsave(&ctrl->lock, flags); + set_bit(ASSOC_FAILED, &ctrl->flags); + state =3D nvme_ctrl_state(&ctrl->ctrl); + spin_unlock_irqrestore(&ctrl->lock, flags); + + switch (state) { case NVME_CTRL_NEW: case NVME_CTRL_LIVE: /* @@ -2542,7 +2550,6 @@ nvme_fc_error_recovery(struct nvme_fc_ctrl *ctrl, cha= r *errmsg) */ if (ctrl->ctrl.state =3D=3D NVME_CTRL_CONNECTING) { __nvme_fc_abort_outstanding_ios(ctrl, true); - set_bit(ASSOC_FAILED, &ctrl->flags); dev_warn(ctrl->ctrl.device, "NVME-FC{%d}: transport error during (re)connect\n", ctrl->cnum); @@ -3167,12 +3174,18 @@ nvme_fc_create_association(struct nvme_fc_ctrl *ctr= l) else ret =3D nvme_fc_recreate_io_queues(ctrl); } - if (!ret && test_bit(ASSOC_FAILED, &ctrl->flags)) - ret =3D -EIO; if (ret) goto out_term_aen_ops; =20 - changed =3D nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_LIVE); + spin_lock_irqsave(&ctrl->lock, flags); + if (!test_bit(ASSOC_FAILED, &ctrl->flags)) + changed =3D nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_LIVE); + else + ret =3D -EIO; + spin_unlock_irqrestore(&ctrl->lock, flags); + + if (ret) + goto out_term_aen_ops; =20 ctrl->ctrl.nr_reconnects =3D 0; =20 --=20 2.47.1