From nobody Tue Dec 2 00:05:28 2025 Received: from mail-pf1-f174.google.com (mail-pf1-f174.google.com [209.85.210.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 53A1F3112AB for ; Wed, 26 Nov 2025 02:13:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764123216; cv=none; b=iIRstIdeULBSKQeQvkIcvhsTSRH6p0DNiRkzgCAw3jWO6EWB5f/2n2pJDYu9ITk02z4RWTSVvwBac67KauCW1E8PTAdcgQ7KGUQ6BBdh3tiMVjwk4n/vCTkgADnrBYgPe6PrEWOYE8xVWkCLluE0X4oQrgr8+FEi5AcFU/NXvg0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764123216; c=relaxed/simple; bh=SiJziCso6wAVNLljyCcG+GZhg19EWsYooPO7XAzD1Sg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=FnjyUyaBWRqDOR+PIy1uK67ZPA2E6oA2TydCDa/bIzVxCQio5diHVDaCrFxO9Py//t3fBDZc2+TTZdZVSTntVQxKyI4p1lfErFXIeXM6BPhkH0n8xUWvzezUeezdcZH8yXv8z7ODXGKAcNfpCz12iTS4vlSd5ud23G5f8tWwvcY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=purestorage.com; spf=fail smtp.mailfrom=purestorage.com; dkim=pass (2048-bit key) header.d=purestorage.com header.i=@purestorage.com header.b=bVlwmVrY; arc=none smtp.client-ip=209.85.210.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=purestorage.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=purestorage.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=purestorage.com header.i=@purestorage.com header.b="bVlwmVrY" Received: by mail-pf1-f174.google.com with SMTP id d2e1a72fcca58-7bc248dc16aso4625427b3a.0 for ; Tue, 25 Nov 2025 18:13:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=purestorage.com; s=google2022; t=1764123213; x=1764728013; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=o+tOUSVfYAveGgYFyI0DvGkCr7R3J34UcR6zCL8mu2o=; b=bVlwmVrY9qAOpe5pnOg0EgLm4uMnMGeV6q+8sHw8vOO0YEHn+f7R0gvvUc1VbkIDTB 8OZkwK3U+aBL7DRMFKZuELLF0G439pg/lISopiJchyAfqq82nNaATsyp8vUqD2JgBYeK /YaG9v2S4E/X+lwgg5Oa0chaXiloawqC1BXKbR9QDyMi73iKPPHvcIYYLAmXgLKIBH1F 5Ddii3zhiMiYIm1K+4iW4f2/Nmj7GIWxDPZ/xvcWAFHPlIQcZfjHF5jfihkWj5ZCmplu W7GDNhiG2bbH51Un4pGt8g3ETmzQogE66DYDYkbaWrSmDPQRDy1IMSPXORI+xEKY3f7u 7+aQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1764123213; x=1764728013; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=o+tOUSVfYAveGgYFyI0DvGkCr7R3J34UcR6zCL8mu2o=; b=u0er/rw2cO4ExEgtGm7Ba62/ZtihMEZObaliMXxfTFsiTRVPxSQypvQwUmC0TxmrIY pkHRL+0ZHUVpgpK49SoQoTtiVeiMd8Q+H0FOyW25/izJknU4YHPJnYvs5KfWsarEz2g+ 0cLbGGGvGVqdCcoUQk12//P5+LylYVt5kqcI1pu1fHnB9rTQgR2UcqlJZBiFq4gaXvH0 cyka8M0Q8RKrkXhgtt62SRM18Ob/i2JGlecJ7uwB6g0RFZY3jioFVkfXqtYF53zss26t rR3eFodbT1ZfRed018JxW+UlXGXY6BWX0piVSGZCK26tkSbf7jnUKBohUit9CNzIy4Kj jvkg== X-Forwarded-Encrypted: i=1; AJvYcCWwqIh0VvFTIKfr8ZAzfaPg01WoOUOOqCr3C9IqiGyMMZ99RDcO3ief0FZFX3w/KU33NOh6s2KZGzlw00M=@vger.kernel.org X-Gm-Message-State: AOJu0YwqufCwL48yUiCPGTN8LqwUkHF0M/S3ZEIGIQl8ZGw0BPmJmY/E BezO0JqfszCY8F43klsgXvQqEEuA8KiIaQ9t2/nFH6wOOBXfNwalqNsBAu/WMdnyKns= X-Gm-Gg: ASbGncvCwWh6FoemypbvASHZAxV+6moPL/bXkTMYYjdrbhTT4ghhzC312BI0Ib2mZhA e+eePa0G13K9s7vK+R/Tc8KCGbNKVPkjJdUOACWQ2bhpYbcmPeY6jnjaH1D1gefXCdDyov7tbnm gbHWRgQHcbYCJssvBNhrF2dnXhyMrOwqDwQBtD/EeDe52aOexlNWlRf7VlFsTktB4rPOf8qS8Ly 6PnALXmngQ1ynMzYOCENd+BOa0A7gPOwubdpJE2c1mIt19H9yIFbQWC5l4SALLIVwaiiZChGakO kttp+N2B8wE52d9TQXf1Oy/hi/dtgM43lZAUETKRd1WfltkXGJcBvgvUFgSXrL0JrXBJ5kk4E6H yCPeGU6E6eWjbgTqQIujRluJE4etMkstuc3KVh5wU85uRy/Foo6OVQOhZv1FfWxVYL62MaILPiO V2P7kSxVzMHz4GOu/EhARyk8AwgOqObkf0qB0rzk80B5eH X-Google-Smtp-Source: AGHT+IEX52DbN1W3fLNeFgULMbeGoHOJm/Hbe+XFe8rU25L89rCrrZxsAphbup4ofHcY7P6zLwy4Ww== X-Received: by 2002:a05:7022:ff48:b0:119:e56b:989b with SMTP id a92af1059eb24-11cb3ecc810mr3258084c88.2.1764123213235; Tue, 25 Nov 2025 18:13:33 -0800 (PST) Received: from apollo.purestorage.com ([208.88.152.253]) by smtp.googlemail.com with ESMTPSA id a92af1059eb24-11cc631c236sm17922979c88.7.2025.11.25.18.13.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 25 Nov 2025 18:13:32 -0800 (PST) From: Mohamed Khalfella To: Chaitanya Kulkarni , Christoph Hellwig , Jens Axboe , Keith Busch , Sagi Grimberg Cc: Aaron Dailey , Randy Jennings , John Meneghini , Hannes Reinecke , linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org, Mohamed Khalfella Subject: [RFC PATCH 13/14] nvme-fc: Use CCR to recover controller that hits an error Date: Tue, 25 Nov 2025 18:12:00 -0800 Message-ID: <20251126021250.2583630-14-mkhalfella@purestorage.com> X-Mailer: git-send-email 2.51.2 In-Reply-To: <20251126021250.2583630-1-mkhalfella@purestorage.com> References: <20251126021250.2583630-1-mkhalfella@purestorage.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" An alive nvme controller that hits an error will now move to RECOVERING state instead of RESETTING state. In RECOVERING state, ctrl->err_work will attempt to use cross-controller recovery to terminate inflight IOs on the controller. If CCR succeeds, then switch to RESETTING state and continue error recovery as usuall by tearing down the controller, and attempting reconnect to target. If CCR fails, the behavior of recovery depends on whether CQT is supported or not. If CQT is supported, switch to time-based recovery by holding inflight IOs until it is safe for them to be retried. If CQT is not supported proceed to retry requests immediately, as the code currently does. Currently, inflight IOs can get completed during time-based recovery. This will be addressed in the next patch. Signed-off-by: Mohamed Khalfella --- drivers/nvme/host/fc.c | 52 ++++++++++++++++++++++++++++++++++++------ 1 file changed, 45 insertions(+), 7 deletions(-) diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c index 8b6a7c80015c..0e4d271bb4b6 100644 --- a/drivers/nvme/host/fc.c +++ b/drivers/nvme/host/fc.c @@ -166,7 +166,7 @@ struct nvme_fc_ctrl { struct blk_mq_tag_set admin_tag_set; struct blk_mq_tag_set tag_set; =20 - struct work_struct ioerr_work; + struct delayed_work ioerr_work; struct delayed_work connect_work; =20 struct kref ref; @@ -1862,11 +1862,48 @@ __nvme_fc_fcpop_chk_teardowns(struct nvme_fc_ctrl *= ctrl, } } =20 +static int nvme_fc_recover_ctrl(struct nvme_ctrl *ctrl) +{ + unsigned long rem; + + if (test_and_clear_bit(NVME_CTRL_RECOVERED, &ctrl->flags)) { + dev_info(ctrl->device, "completed time-based recovery\n"); + goto done; + } + + rem =3D nvme_recover_ctrl(ctrl); + if (!rem) + goto done; + + if (!ctrl->cqt) { + dev_info(ctrl->device, + "CCR failed, CQT not supported, skip time-based recovery\n"); + goto done; + } + + dev_info(ctrl->device, + "CCR failed, switch to time-based recovery, timeout =3D %ums\n", + jiffies_to_msecs(rem)); + + set_bit(NVME_CTRL_RECOVERED, &ctrl->flags); + queue_delayed_work(nvme_reset_wq, &to_fc_ctrl(ctrl)->ioerr_work, rem); + return -EAGAIN; + +done: + nvme_end_ctrl_recovery(ctrl); + return 0; +} + static void nvme_fc_ctrl_ioerr_work(struct work_struct *work) { - struct nvme_fc_ctrl *ctrl =3D - container_of(work, struct nvme_fc_ctrl, ioerr_work); + struct nvme_fc_ctrl *ctrl =3D container_of(to_delayed_work(work), + struct nvme_fc_ctrl, ioerr_work); + + if (nvme_ctrl_state(&ctrl->ctrl) =3D=3D NVME_CTRL_RECOVERING) { + if (nvme_fc_recover_ctrl(&ctrl->ctrl)) + return; + } =20 nvme_fc_error_recovery(ctrl); } @@ -1892,7 +1929,8 @@ EXPORT_SYMBOL_GPL(nvme_fc_io_getuuid); static void nvme_fc_start_ioerr_recovery(struct nvme_fc_ctrl *ctrl, char *errmsg) { - if (!nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_RESETTING)) + if (!nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_RECOVERING) && + !nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_RESETTING)) return; =20 dev_warn(ctrl->ctrl.device, "NVME-FC{%d}: starting error recovery %s\n", @@ -3227,7 +3265,7 @@ nvme_fc_delete_ctrl(struct nvme_ctrl *nctrl) { struct nvme_fc_ctrl *ctrl =3D to_fc_ctrl(nctrl); =20 - cancel_work_sync(&ctrl->ioerr_work); + cancel_delayed_work_sync(&ctrl->ioerr_work); cancel_delayed_work_sync(&ctrl->connect_work); /* * kill the association on the link side. this will block @@ -3465,7 +3503,7 @@ nvme_fc_alloc_ctrl(struct device *dev, struct nvmf_ct= rl_options *opts, =20 INIT_WORK(&ctrl->ctrl.reset_work, nvme_fc_reset_ctrl_work); INIT_DELAYED_WORK(&ctrl->connect_work, nvme_fc_connect_ctrl_work); - INIT_WORK(&ctrl->ioerr_work, nvme_fc_ctrl_ioerr_work); + INIT_DELAYED_WORK(&ctrl->ioerr_work, nvme_fc_ctrl_ioerr_work); spin_lock_init(&ctrl->lock); =20 /* io queue count */ @@ -3563,7 +3601,7 @@ nvme_fc_init_ctrl(struct device *dev, struct nvmf_ctr= l_options *opts, =20 fail_ctrl: nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_DELETING); - cancel_work_sync(&ctrl->ioerr_work); + cancel_delayed_work_sync(&ctrl->ioerr_work); cancel_work_sync(&ctrl->ctrl.reset_work); cancel_delayed_work_sync(&ctrl->connect_work); =20 --=20 2.51.2