From nobody Mon Feb 9 07:54:49 2026 Received: from mail-dl1-f48.google.com (mail-dl1-f48.google.com [74.125.82.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4EA8F38A9C6 for ; Fri, 30 Jan 2026 22:36:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.82.48 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1769812599; cv=none; b=P+BFGAOZM18NMAlkKx8iDRETin8dCqHXMRHXeBPUK9N+zz+T5k4EQasGMZIpaojfWY48ohBe4RmVtl5JjYNANbxpVJn0lLKMkdSxXFHsH90TW1kZzMNl8ZlGVpfB5K/5MazqOyE6c0n4SUYJW88H8ESXK0Tb8CXAcLWLQukq5Jo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1769812599; c=relaxed/simple; bh=+DCK/lb5J8eX53TqCpHbTMAXo4LWmuWp48ouPQr88/s=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=G/z/uuDIxDIL74lmuMvIWEok75rnI5US3oyivRPLoDnCXHvD1Md+haWbOiA96uok68TwV99mPptfC9H+0/fQ1HSKu2+4ykKtQ5SsDEXbalOBPhxATh9a909Dgq3DH1NJpwmZRClwGHgXYO25F2o7kKoaao7CClgMvrwZqr9W5go= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=purestorage.com; spf=fail smtp.mailfrom=purestorage.com; dkim=pass (2048-bit key) header.d=purestorage.com header.i=@purestorage.com header.b=ZDIzzuKk; arc=none smtp.client-ip=74.125.82.48 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=purestorage.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=purestorage.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=purestorage.com header.i=@purestorage.com header.b="ZDIzzuKk" Received: by mail-dl1-f48.google.com with SMTP id a92af1059eb24-1233bb90317so3047257c88.1 for ; Fri, 30 Jan 2026 14:36:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=purestorage.com; s=google2022; t=1769812593; x=1770417393; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=xDu7rcI44pdZ7ozlYZ8/MRynW2sKdwev0diNQdJpydQ=; b=ZDIzzuKkNWcJp4De+Og6fPsp4dxfDCAtcBdGHqN7CVv/2+YC8Zal24Pud+0YPpZvcD vxC+WCiI93lwFnnTwy7nkFHrSeOrdMTtb0yMvTMPNvig+9haIH8w0p4EPOcBc3ObhHeb PPJmaSupZCGH5gW4FBcsoUoB8i7mWRrnQDHvp7S3nIWwYS2hQAMe2/1t3MBzwj/fy6Vy WfKvO16F9PMC7bNGJDZJzxzAhyV+mQLbclJAwbLCdtup4f4v8KxA8H6JPZWoyoCgGf6X bJomCOmSWpCEeAr23dXTT9my0xJOqElM5MAK8Khhx9ln+i0PVZ4KKWEYzHZnmQBSb3WW HWKg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1769812593; x=1770417393; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=xDu7rcI44pdZ7ozlYZ8/MRynW2sKdwev0diNQdJpydQ=; b=NOs2JwUJAtc/ExC7MTjrY/IFu+xvyZqkCX4oepbIdcSJ8i7DrF9SF+FGn8qIJehWYj zFWLifwxRsvQEBtPQQQOj0WHPhU+D+uJ+CNti2hUkxEkUbs74h0xgNxwNObEjzd7rGlo OGtFQEIkQL2GL93oniFlZgczyBOrTIfh6MKUuMJqul1ARTxWTPRGvJA+aQA1Mvz4t/st BwE59uZNkSDevjN7gu42FBLLyRkZUuMrSJjLajeTHqox38gGoJQi8wC7rChT7UeEXnm+ 67ChG2FsyWN+4TJ/k2DlmXW36In4+IlADyd8fQTl7xse7inab15G/O8+2FCMBHcqadAF tKTw== X-Forwarded-Encrypted: i=1; AJvYcCW2kKKfEqqMcJWEsXMsUwZ8FRC7YAABRWPTdhqWJy8F8PX2vAByygZ7EiS3m+oVWDuJQJb4zsuueOtK8gQ=@vger.kernel.org X-Gm-Message-State: AOJu0YwsdzegHQYiqRBTY+gWNm4CY+8t0bb9AOoRks2m/rh2BqusFuA+ yZa2zsCYxdVZu2dijKD0z+ogVIICYQgW31oIBV9ZZVxR+SaFraGoQho+cUt1f1ib9p8= X-Gm-Gg: AZuq6aI0WhTwt2vvcjNSYiNdTq68cn5Pnl41CJoXqpTEil0bfRhNIwIcjmD68Co0vNe 9nR5q4aXhp9czroJ6nicV5uHw9D2ma1mt6CvjFou6YjInl4BFa+iXq3uuqI6OBKQryOgGNi52IW j9w3hZKKR+jMU2ojOvZOl7sH212Wi8ttzxcA3KUPu6ul+dlVv9TQUSug4runrhcks/x7y92KbSV Qvo76JfkpdQ7N4io5KokmHcAxi97u6SU6G4eu54LGwbqPGiRq0mWPktwjd6sJhz/TCBoZtSa7/6 E6TZsrLiEz6gyR8II1BO3f8v2Ohd7yGzAMYVk3Lm3AgWLfn8rayAFEy1XTsL/NY5FZVO3852ihF hGOjiewuUqNWZxl3VwnMNZSew87/TUcwK19YlsDyrHZZSjx10pEFgl9ZRZl5dL6eCbH7HsQdC6h bvvTdIV03H18TrgKpQs8fOsKT3uTV6mXoheg== X-Received: by 2002:a05:7022:4589:b0:11e:4fc:9b33 with SMTP id a92af1059eb24-124b100d8d5mr4190399c88.1.1769812593015; Fri, 30 Jan 2026 14:36:33 -0800 (PST) Received: from apollo.purestorage.com ([208.88.152.253]) by smtp.googlemail.com with ESMTPSA id a92af1059eb24-124a9d6b906sm13161717c88.4.2026.01.30.14.36.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 30 Jan 2026 14:36:32 -0800 (PST) From: Mohamed Khalfella To: Justin Tee , Naresh Gottumukkala , Paul Ely , Chaitanya Kulkarni , Christoph Hellwig , Jens Axboe , Keith Busch , Sagi Grimberg Cc: Aaron Dailey , Randy Jennings , Dhaval Giani , Hannes Reinecke , linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org, Mohamed Khalfella Subject: [PATCH v2 10/14] nvme-tcp: Use CCR to recover controller that hits an error Date: Fri, 30 Jan 2026 14:34:14 -0800 Message-ID: <20260130223531.2478849-11-mkhalfella@purestorage.com> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20260130223531.2478849-1-mkhalfella@purestorage.com> References: <20260130223531.2478849-1-mkhalfella@purestorage.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" An alive nvme controller that hits an error now will move to FENCING state instead of RESETTING state. ctrl->fencing_work attempts CCR to terminate inflight IOs. If CCR succeeds, switch to FENCED -> RESETTING and continue error recovery as usual. If CCR fails, the behavior depends on whether the subsystem supports CQT or not. If CQT is not supported then reset the controller immediately as if CCR succeeded in order to maintain the current behavior. If CQT is supported switch to time-based recovery. Schedule ctrl->fenced_work resets the controller when time based recovery finishes. Either ctrl->err_work or ctrl->reset_work can run after a controller is fenced. Flush fencing work when either work run. Signed-off-by: Mohamed Khalfella --- drivers/nvme/host/tcp.c | 62 ++++++++++++++++++++++++++++++++++++++++- 1 file changed, 61 insertions(+), 1 deletion(-) diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c index 69cb04406b47..af8d3b36a4bb 100644 --- a/drivers/nvme/host/tcp.c +++ b/drivers/nvme/host/tcp.c @@ -193,6 +193,8 @@ struct nvme_tcp_ctrl { struct sockaddr_storage src_addr; struct nvme_ctrl ctrl; =20 + struct work_struct fencing_work; + struct delayed_work fenced_work; struct work_struct err_work; struct delayed_work connect_work; struct nvme_tcp_request async_req; @@ -611,6 +613,12 @@ static void nvme_tcp_init_recv_ctx(struct nvme_tcp_que= ue *queue) =20 static void nvme_tcp_error_recovery(struct nvme_ctrl *ctrl) { + if (nvme_change_ctrl_state(ctrl, NVME_CTRL_FENCING)) { + dev_warn(ctrl->device, "starting controller fencing\n"); + queue_work(nvme_wq, &to_tcp_ctrl(ctrl)->fencing_work); + return; + } + if (!nvme_change_ctrl_state(ctrl, NVME_CTRL_RESETTING)) return; =20 @@ -2470,12 +2478,59 @@ static void nvme_tcp_reconnect_ctrl_work(struct wor= k_struct *work) nvme_tcp_reconnect_or_remove(ctrl, ret); } =20 +static void nvme_tcp_fenced_work(struct work_struct *work) +{ + struct nvme_tcp_ctrl *tcp_ctrl =3D container_of(to_delayed_work(work), + struct nvme_tcp_ctrl, fenced_work); + struct nvme_ctrl *ctrl =3D &tcp_ctrl->ctrl; + + nvme_change_ctrl_state(ctrl, NVME_CTRL_FENCED); + if (nvme_change_ctrl_state(ctrl, NVME_CTRL_RESETTING)) + queue_work(nvme_reset_wq, &tcp_ctrl->err_work); +} + +static void nvme_tcp_fencing_work(struct work_struct *work) +{ + struct nvme_tcp_ctrl *tcp_ctrl =3D container_of(work, + struct nvme_tcp_ctrl, fencing_work); + struct nvme_ctrl *ctrl =3D &tcp_ctrl->ctrl; + unsigned long rem; + + rem =3D nvme_fence_ctrl(ctrl); + if (!rem) + goto done; + + if (!ctrl->cqt) { + dev_info(ctrl->device, + "CCR failed, CQT not supported, skip time-based recovery\n"); + goto done; + } + + dev_info(ctrl->device, + "CCR failed, switch to time-based recovery, timeout =3D %ums\n", + jiffies_to_msecs(rem)); + queue_delayed_work(nvme_wq, &tcp_ctrl->fenced_work, rem); + return; + +done: + nvme_change_ctrl_state(ctrl, NVME_CTRL_FENCED); + if (nvme_change_ctrl_state(ctrl, NVME_CTRL_RESETTING)) + queue_work(nvme_reset_wq, &tcp_ctrl->err_work); +} + +static void nvme_tcp_flush_fencing_work(struct nvme_ctrl *ctrl) +{ + flush_work(&to_tcp_ctrl(ctrl)->fencing_work); + flush_delayed_work(&to_tcp_ctrl(ctrl)->fenced_work); +} + static void nvme_tcp_error_recovery_work(struct work_struct *work) { struct nvme_tcp_ctrl *tcp_ctrl =3D container_of(work, struct nvme_tcp_ctrl, err_work); struct nvme_ctrl *ctrl =3D &tcp_ctrl->ctrl; =20 + nvme_tcp_flush_fencing_work(ctrl); if (nvme_tcp_key_revoke_needed(ctrl)) nvme_auth_revoke_tls_key(ctrl); nvme_stop_keep_alive(ctrl); @@ -2518,6 +2573,7 @@ static void nvme_reset_ctrl_work(struct work_struct *= work) container_of(work, struct nvme_ctrl, reset_work); int ret; =20 + nvme_tcp_flush_fencing_work(ctrl); if (nvme_tcp_key_revoke_needed(ctrl)) nvme_auth_revoke_tls_key(ctrl); nvme_stop_ctrl(ctrl); @@ -2643,13 +2699,15 @@ static enum blk_eh_timer_return nvme_tcp_timeout(st= ruct request *rq) struct nvme_tcp_cmd_pdu *pdu =3D nvme_tcp_req_cmd_pdu(req); struct nvme_command *cmd =3D &pdu->cmd; int qid =3D nvme_tcp_queue_id(req->queue); + enum nvme_ctrl_state state; =20 dev_warn(ctrl->device, "I/O tag %d (%04x) type %d opcode %#x (%s) QID %d timeout\n", rq->tag, nvme_cid(rq), pdu->hdr.type, cmd->common.opcode, nvme_fabrics_opcode_str(qid, cmd), qid); =20 - if (nvme_ctrl_state(ctrl) !=3D NVME_CTRL_LIVE) { + state =3D nvme_ctrl_state(ctrl); + if (state !=3D NVME_CTRL_LIVE && state !=3D NVME_CTRL_FENCING) { /* * If we are resetting, connecting or deleting we should * complete immediately because we may block controller @@ -2904,6 +2962,8 @@ static struct nvme_tcp_ctrl *nvme_tcp_alloc_ctrl(stru= ct device *dev, =20 INIT_DELAYED_WORK(&ctrl->connect_work, nvme_tcp_reconnect_ctrl_work); + INIT_DELAYED_WORK(&ctrl->fenced_work, nvme_tcp_fenced_work); + INIT_WORK(&ctrl->fencing_work, nvme_tcp_fencing_work); INIT_WORK(&ctrl->err_work, nvme_tcp_error_recovery_work); INIT_WORK(&ctrl->ctrl.reset_work, nvme_reset_ctrl_work); =20 --=20 2.52.0