From nobody Thu Apr 2 23:54:50 2026 Received: from mail-dl1-f53.google.com (mail-dl1-f53.google.com [74.125.82.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 17BA024A07C for ; Sat, 14 Feb 2026 04:28:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.82.53 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771043319; cv=none; b=sAmBtmxUf1bjyj6NrYXZnb4uEVO/AuC3724XkYIyBo/OL1Or6XqxO+sC5PqQ1sPhcbg3KgDVqjYcdHIL0ENgkXvLb2ycpqDza1D0d4BfKuMxkUwoqKCp6VLIfWtOjpMJ/WdMDa92rEoeFAyydQBwuUQfUOikv9U05+aNBL9CbvQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771043319; c=relaxed/simple; bh=2Lob27LXacmU41JJLUPDIryQknZBfYBRwlb0ekDMVq8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=W/dzfl3E7haFKUnSp6TMNV+M8VRsEwzW/sUVFTl6FS2PTa9qZTW1anzF6+tPvAEXh030EWif+eu5ezWntYORYLfW10OE4/yvk7PxtpgIkp8ARJ23NP0erKbb6WfERJe2jAGfSZUk+Iv9dSjWFOqiquzaAaZTFkPZhmyIASPAB8k= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=purestorage.com; spf=fail smtp.mailfrom=purestorage.com; dkim=pass (2048-bit key) header.d=purestorage.com header.i=@purestorage.com header.b=gZr/fsa5; arc=none smtp.client-ip=74.125.82.53 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=purestorage.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=purestorage.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=purestorage.com header.i=@purestorage.com header.b="gZr/fsa5" Received: by mail-dl1-f53.google.com with SMTP id a92af1059eb24-1270be4d125so3698307c88.1 for ; Fri, 13 Feb 2026 20:28:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=purestorage.com; s=google2022; t=1771043311; x=1771648111; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Imu19h/MshhAGpnKdw7Is91vPSU7w8nWppIzdQ0Ksj0=; b=gZr/fsa5PsYMuQLK3TBlKF6aP8RrFlhFOfQwMIOCiijS9CzPHdOvk5mJhi1JQPpSmm +3ZHZtbjdOqZ9eaPL6KOA3faXMskIoaiRGECZku5hhq6ME8Pu9HMNxytfVg/LHk10tIM 7FGxC1GM3sMusO+c3qvtfWbhnfXLvBjIKzPY8pj/2+4gRADp9v2/t1OHfJqHjgf5Y6sQ aMleGN9RH0uzvKsS+aTdmyd+Ad2pGXK75sObTJEbZ4rF8bWAjO0OeufDL7SrOrlc8Bs2 wnB/cPYXN98XUy6aGHWtdnTtbltzyGPsD7NampM64cEMBU+xjd8PCcKAcXpCKgHL1hlF HpRg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1771043311; x=1771648111; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=Imu19h/MshhAGpnKdw7Is91vPSU7w8nWppIzdQ0Ksj0=; b=WNCX0P4B5f6C/R6ckrUrLePGElt5VssTtBHzD5dRJMZlyx25Ge0WQ7GTIO0OSMZ4h0 iMSJTlFJYpIzJPWGHIfNrqw5ctVbUp+UNtInM3JccAdaCWGrpGCMreh5LLGLXkhmjT/i KU1IySOHpRxgOWB//zSNfSy7QocqHGoZPCeWWfwxe21l+rDRzmTAMfDPceCNSzbHrpfr nhX0ESyQe+mhDtS5oNnZxV/kl7rESJKWUApsuHmuEkYV1VRuYowf871Rat4Hogk6FPx6 IW/CUycf/AYPQ3MMorCL2EcI3doAdrtyKiKKCrSr/oYIk6w7Vgy/rlQSTNmyrCe7yDdJ Eucw== X-Forwarded-Encrypted: i=1; AJvYcCVgwzj/p+X0MxJQX3nqn4pXHPLeV2BdZFmMCUzE1aVj5Xq6qd4XiHDJx7nMyzdHn2fTqaWMUXf6z5UIh/U=@vger.kernel.org X-Gm-Message-State: AOJu0YzX5EVEHCRSPoXbzyC91vwN9npqP7GvX21GxPYHSj1qyBGJXIEl HnnG+RVmAz5lS5SXuhrl6F7JdOcFFdlu8OAn5zS7Tv6TVJbzEEcn5s38tpp9KVUJf+w= X-Gm-Gg: AZuq6aJIyucL6R07HF2ZXmKglZC3A78xhRWT3TaxHlteJmAoqYJHGur7IG5K/4rTO3T kRLgjM7CA99r/mGz6YnPmM2RuB9pevPF+M5AvsqmaDA2qxRf/zQdBzFDALRY4S2/VFzWR68s1lb Ehv3rlb4L8WnInIKgDIbyFTGU+hilOSiGlvVmOXVPfWfuxYjagDAnrVCyzTDctr6mm/18+acocr QSLks9Rfk6P2zZtaCiOOAuKb5DHRc/hfYuEKW9xwUoWWLeyBNgxnenBNorLUt36rD5PRimLD5pE EyVmHto/AR0+5YKKsaJF28AFKp1Y73WFsEiZMJEupFq82IEqSvIIdmvf5v/exu/MxKU+WO87op7 OoJW2jopZT4/jhdjGHXZ9q5clldxMF7kkYzZ352+bnQGNqvfPJW5S5YZz3UcLTxvS2SYkvngd13 uK03o96w946SjIPqL767wjulgYktVfbC01Ogvah9dLOytg+g== X-Received: by 2002:a05:7022:6897:b0:127:337e:3301 with SMTP id a92af1059eb24-12741b6fd42mr661529c88.12.1771043310981; Fri, 13 Feb 2026 20:28:30 -0800 (PST) Received: from apollo.purestorage.com ([208.88.152.253]) by smtp.googlemail.com with ESMTPSA id a92af1059eb24-12742cbc900sm1021042c88.14.2026.02.13.20.28.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 13 Feb 2026 20:28:30 -0800 (PST) From: Mohamed Khalfella To: Justin Tee , Naresh Gottumukkala , Paul Ely , Chaitanya Kulkarni , Christoph Hellwig , Jens Axboe , Keith Busch , Sagi Grimberg , James Smart , Hannes Reinecke Cc: Aaron Dailey , Randy Jennings , Dhaval Giani , linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org, Mohamed Khalfella Subject: [PATCH v3 12/21] nvme-fc: Decouple error recovery from controller reset Date: Fri, 13 Feb 2026 20:25:13 -0800 Message-ID: <20260214042753.4073668-13-mkhalfella@purestorage.com> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20260214042753.4073668-1-mkhalfella@purestorage.com> References: <20260214042753.4073668-1-mkhalfella@purestorage.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" nvme_fc_error_recovery() called from nvme_fc_timeout() while controller in CONNECTING state results in deadlock reported in link below. Update nvme_fc_timeout() to schedule error recovery to avoid the deadlock. Previous to this change if controller was LIVE error recovery resets the controller and this does not match nvme-tcp and nvme-rdma. Decouple error recovery from controller reset to match other fabric transports. Link: https://lore.kernel.org/all/20250529214928.2112990-1-mkhalfella@pures= torage.com/ Signed-off-by: Mohamed Khalfella --- drivers/nvme/host/fc.c | 120 +++++++++++++++++++++++------------------ 1 file changed, 67 insertions(+), 53 deletions(-) diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c index 6948de3f438a..e6ffaa19aba4 100644 --- a/drivers/nvme/host/fc.c +++ b/drivers/nvme/host/fc.c @@ -227,6 +227,10 @@ static DEFINE_IDA(nvme_fc_ctrl_cnt); static struct device *fc_udev_device; =20 static void nvme_fc_complete_rq(struct request *rq); +static void nvme_fc_start_ioerr_recovery(struct nvme_fc_ctrl *ctrl, + char *errmsg); +static void __nvme_fc_abort_outstanding_ios(struct nvme_fc_ctrl *ctrl, + bool start_queues); =20 /* *********************** FC-NVME Port Management ***********************= * */ =20 @@ -788,7 +792,7 @@ nvme_fc_ctrl_connectivity_loss(struct nvme_fc_ctrl *ctr= l) "Reconnect", ctrl->cnum); =20 set_bit(ASSOC_FAILED, &ctrl->flags); - nvme_reset_ctrl(&ctrl->ctrl); + nvme_fc_start_ioerr_recovery(ctrl, "Connectivity Loss"); } =20 /** @@ -985,7 +989,7 @@ fc_dma_unmap_sg(struct device *dev, struct scatterlist = *sg, int nents, static void nvme_fc_ctrl_put(struct nvme_fc_ctrl *); static int nvme_fc_ctrl_get(struct nvme_fc_ctrl *); =20 -static void nvme_fc_error_recovery(struct nvme_fc_ctrl *ctrl, char *errmsg= ); +static void nvme_fc_error_recovery(struct nvme_fc_ctrl *ctrl); =20 static void __nvme_fc_finish_ls_req(struct nvmefc_ls_req_op *lsop) @@ -1567,9 +1571,8 @@ nvme_fc_ls_disconnect_assoc(struct nvmefc_ls_rcv_op *= lsop) * for the association have been ABTS'd by * nvme_fc_delete_association(). */ - - /* fail the association */ - nvme_fc_error_recovery(ctrl, "Disconnect Association LS received"); + nvme_fc_start_ioerr_recovery(ctrl, + "Disconnect Association LS received"); =20 /* release the reference taken by nvme_fc_match_disconn_ls() */ nvme_fc_ctrl_put(ctrl); @@ -1871,7 +1874,22 @@ nvme_fc_ctrl_ioerr_work(struct work_struct *work) struct nvme_fc_ctrl *ctrl =3D container_of(work, struct nvme_fc_ctrl, ioerr_work); =20 - nvme_fc_error_recovery(ctrl, "transport detected io error"); + /* + * if an error (io timeout, etc) while (re)connecting, the remote + * port requested terminating of the association (disconnect_ls) + * or an error (timeout or abort) occurred on an io while creating + * the controller. Abort any ios on the association and let the + * create_association error path resolve things. + */ + if (nvme_ctrl_state(&ctrl->ctrl) =3D=3D NVME_CTRL_CONNECTING) { + __nvme_fc_abort_outstanding_ios(ctrl, true); + dev_warn(ctrl->ctrl.device, + "NVME-FC{%d}: transport error during (re)connect\n", + ctrl->cnum); + return; + } + + nvme_fc_error_recovery(ctrl); } =20 /* @@ -1892,6 +1910,25 @@ char *nvme_fc_io_getuuid(struct nvmefc_fcp_req *req) } EXPORT_SYMBOL_GPL(nvme_fc_io_getuuid); =20 +static void nvme_fc_start_ioerr_recovery(struct nvme_fc_ctrl *ctrl, + char *errmsg) +{ + enum nvme_ctrl_state state; + + if (nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_RESETTING)) { + dev_warn(ctrl->ctrl.device, "NVME-FC{%d}: starting error recovery %s\n", + ctrl->cnum, errmsg); + queue_work(nvme_reset_wq, &ctrl->ioerr_work); + return; + } + + state =3D nvme_ctrl_state(&ctrl->ctrl); + if (state =3D=3D NVME_CTRL_CONNECTING || state =3D=3D NVME_CTRL_DELETING = || + state =3D=3D NVME_CTRL_DELETING_NOIO) { + queue_work(nvme_reset_wq, &ctrl->ioerr_work); + } +} + static void nvme_fc_fcpio_done(struct nvmefc_fcp_req *req) { @@ -2049,9 +2086,8 @@ nvme_fc_fcpio_done(struct nvmefc_fcp_req *req) nvme_fc_complete_rq(rq); =20 check_error: - if (terminate_assoc && - nvme_ctrl_state(&ctrl->ctrl) !=3D NVME_CTRL_RESETTING) - queue_work(nvme_reset_wq, &ctrl->ioerr_work); + if (terminate_assoc) + nvme_fc_start_ioerr_recovery(ctrl, "io error"); } =20 static int @@ -2495,39 +2531,6 @@ __nvme_fc_abort_outstanding_ios(struct nvme_fc_ctrl = *ctrl, bool start_queues) nvme_unquiesce_admin_queue(&ctrl->ctrl); } =20 -static void -nvme_fc_error_recovery(struct nvme_fc_ctrl *ctrl, char *errmsg) -{ - enum nvme_ctrl_state state =3D nvme_ctrl_state(&ctrl->ctrl); - - /* - * if an error (io timeout, etc) while (re)connecting, the remote - * port requested terminating of the association (disconnect_ls) - * or an error (timeout or abort) occurred on an io while creating - * the controller. Abort any ios on the association and let the - * create_association error path resolve things. - */ - if (state =3D=3D NVME_CTRL_CONNECTING) { - __nvme_fc_abort_outstanding_ios(ctrl, true); - dev_warn(ctrl->ctrl.device, - "NVME-FC{%d}: transport error during (re)connect\n", - ctrl->cnum); - return; - } - - /* Otherwise, only proceed if in LIVE state - e.g. on first error */ - if (state !=3D NVME_CTRL_LIVE) - return; - - dev_warn(ctrl->ctrl.device, - "NVME-FC{%d}: transport association event: %s\n", - ctrl->cnum, errmsg); - dev_warn(ctrl->ctrl.device, - "NVME-FC{%d}: resetting controller\n", ctrl->cnum); - - nvme_reset_ctrl(&ctrl->ctrl); -} - static enum blk_eh_timer_return nvme_fc_timeout(struct request *rq) { struct nvme_fc_fcp_op *op =3D blk_mq_rq_to_pdu(rq); @@ -2536,24 +2539,14 @@ static enum blk_eh_timer_return nvme_fc_timeout(str= uct request *rq) struct nvme_fc_cmd_iu *cmdiu =3D &op->cmd_iu; struct nvme_command *sqe =3D &cmdiu->sqe; =20 - /* - * Attempt to abort the offending command. Command completion - * will detect the aborted io and will fail the connection. - */ dev_info(ctrl->ctrl.device, "NVME-FC{%d.%d}: io timeout: opcode %d fctype %d (%s) w10/11: " "x%08x/x%08x\n", ctrl->cnum, qnum, sqe->common.opcode, sqe->fabrics.fctype, nvme_fabrics_opcode_str(qnum, sqe), sqe->common.cdw10, sqe->common.cdw11); - if (__nvme_fc_abort_op(ctrl, op)) - nvme_fc_error_recovery(ctrl, "io timeout abort failed"); =20 - /* - * the io abort has been initiated. Have the reset timer - * restarted and the abort completion will complete the io - * shortly. Avoids a synchronous wait while the abort finishes. - */ + nvme_fc_start_ioerr_recovery(ctrl, "io timeout"); return BLK_EH_RESET_TIMER; } =20 @@ -3352,6 +3345,27 @@ nvme_fc_reset_ctrl_work(struct work_struct *work) } } =20 +static void +nvme_fc_error_recovery(struct nvme_fc_ctrl *ctrl) +{ + nvme_stop_keep_alive(&ctrl->ctrl); + nvme_stop_ctrl(&ctrl->ctrl); + flush_work(&ctrl->ctrl.async_event_work); + + /* will block while waiting for io to terminate */ + nvme_fc_delete_association(ctrl); + + /* Do not reconnect if controller is being deleted */ + if (!nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_CONNECTING)) + return; + + if (ctrl->rport->remoteport.port_state =3D=3D FC_OBJSTATE_ONLINE) { + queue_delayed_work(nvme_wq, &ctrl->connect_work, 0); + return; + } + + nvme_fc_reconnect_or_delete(ctrl, -ENOTCONN); +} =20 static const struct nvme_ctrl_ops nvme_fc_ctrl_ops =3D { .name =3D "fc", --=20 2.52.0