From nobody Tue Dec 16 21:28:45 2025 Received: from mail-pl1-f173.google.com (mail-pl1-f173.google.com [209.85.214.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5948323AB98 for ; Tue, 2 Dec 2025 01:35:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764639332; cv=none; b=i/J/yBakZYReruGGGzxLl8WZTnD0Ljzlrtbj2BMvHUO4ai1Cvf/lmEjKkTGobrHy1rDf12h8l5rMnSV/QtQpSmeQsnQ6OpJmAXX0315nYG4OstSJ18E3QHQexZH7vgP8Pop1li14865BGviOflSRn5Tt7vD+jo6c+Iluisnc1MM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764639332; c=relaxed/simple; bh=RUtCX8CP3W/yuDQ/ZW/ZJRJEJWXpd/A/PFm0/jER/wQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=jLG46k+e9SuhKzX5NovrJuEx1M7hmz+ZVHikaSrHmdjidIAv5tvdIggHxAP/g/1WEMrgY2shg9UAs6x+V8L2EgqxuKsjZpl5C0YbMbDFJXqjECYuMPOrsA98DEuvdkF+3lu74ptjO8u/izkl/RVBHVvuneh6ItEv9ghJPBwiPuc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=NqR8s5jE; arc=none smtp.client-ip=209.85.214.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="NqR8s5jE" Received: by mail-pl1-f173.google.com with SMTP id d9443c01a7336-2957850c63bso36207675ad.0 for ; Mon, 01 Dec 2025 17:35:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1764639329; x=1765244129; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=alrcnF8hffQyqVii+35fGG8FiEZV1WdxKAf95IBHMhY=; b=NqR8s5jEztxYwuc0E7kXEy/Vfefn7i/ZsNniBNvBNGX9Cg3cKzL4Ot0B+IPNkFrLPO rxQjVGOdb3xVlE0PpSBu45Yjr34pvbUiaE5bRRmsdZpoadefanYEEBMBhr5FrELOhWbG omwO4NzJpWxdcujzNa9HaBiqLlq+ztJAV+CJdG7LP/JfisTjC+Bjn1E24AmNjjwOAElS 3E3L1USIAm+sQuDxIhhJVgwqti72jxJnXd0dwUReqrQ6MRpa9TVPkuu3JL+ZPc9eFhgT pN+tJ1O5ev8RIPr6IMhmhF/01ImVTNSMw9XRoCPajRfJkArnaF53zHN6L44FfhWQWU/8 bMIA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1764639329; x=1765244129; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=alrcnF8hffQyqVii+35fGG8FiEZV1WdxKAf95IBHMhY=; b=mcqJwhbFz0SYZ+wAPhStFIFiQrD3TC+HTrL9RdDZRgbNXv2g+D4W9sZUs5lOQwpXU7 TwQfvN0M80imcm0trYJmVs2ZrCPqz5hAjVpK5BfignAODkHZDhqeinoBMpmFUQPZLpQF USZ7hmOvEu+XbytMCdxs4FdUq8uthsD6KyKrQ03X76SJNpfdbOumc9Ayoxp1sWr/DEV+ OsaLnmeQ4ZoPOCUxIv9ymxdfIQ/wPOXzUlPXtcfwa6jb0wF3rybrkk0lACEy3KHCFEEm AfZfwmsnULoMMO7dMn8P722LjoCD+ZjkgDtbVM2+fGPHmza9GBja2ibFoSZfLn8bpQwa In2w== X-Forwarded-Encrypted: i=1; AJvYcCUX4j7zThwopi/0gkjg6JewGdbS/NpIWlW4DPLHdHHjhaeAi2+rOPyLmF/9sZC+efiiwbJWhoKEWf6dg3k=@vger.kernel.org X-Gm-Message-State: AOJu0YyO6prLt1/o8P7A5mbNea//SYZoEcAbPYRkHyp4uzkD/uptkXEK KivIOdGfTXHggKNg4w4j0XVRid1MxOCgqgcrfaH78MDbWL5Ad3fjZZvk X-Gm-Gg: ASbGncsgOmRWZzJVPYe/8jbhtw7pWEbSLWi4O+vVDsUaM7g+RhgmWx5y9N/AgIYcnwk M3stLibMuZ3LN5O4KVqsa8sw1Yip6lF7kDKcDjEIcY8amL9QsB4npmu/prBpTOiRtS3ANEqLM+W mYT8viaBAa0HKzICZyjCprbkzAt/YMtVAHUJLuJbKOH1mq/gtgcOftMMBDcZENmYf0F9tqQlmaU ar2p0Cl5OakajbbFrIv1PqU3D8NtEvSf24GzB7ZKj44bxxI7LEyxxna3rja7VKRcw0CidOEciuv IFiOQGxkLwfZtGKDFp4hrIoUcmDUPS+x29sXiqgCm3rmOjEEuurAFgTsbE6P0O4iTp9ick2Rcsg irSPDmPCO14PIWxTjtRgy6DaOZIuf1xki82zwIm+axijH6/ce3+5Sca1bbBh/tf8kriuEa8uVpr DnY7fFD1DiTrc1ycUq61LyMJ+uNHFYz2LzQLwxEsAetvA2h3xqOdoXGuu1Yte7ezLnv7fSkcjCQ ZlqXfsLWfjOI3Gy/p4= X-Google-Smtp-Source: AGHT+IHdYVOCmmOlbphB1W7UrAZ5O3XT23PE61UI8Mwhm5s9BH8EvESHLmAwNWu7QZRIDDb2agOulw== X-Received: by 2002:a17:902:f786:b0:298:88:c006 with SMTP id d9443c01a7336-29d5a48b3afmr8809815ad.4.1764639329398; Mon, 01 Dec 2025 17:35:29 -0800 (PST) Received: from toolbx.alistair23.me (2403-580b-97e8-0-82ce-f179-8a79-69f4.ip6.aussiebb.net. [2403:580b:97e8:0:82ce:f179:8a79:69f4]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-29bceb54563sm132378575ad.89.2025.12.01.17.35.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 01 Dec 2025 17:35:28 -0800 (PST) From: alistair23@gmail.com X-Google-Original-From: alistair.francis@wdc.com To: chuck.lever@oracle.com, hare@kernel.org, kernel-tls-handshake@lists.linux.dev, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-nvme@lists.infradead.org, linux-nfs@vger.kernel.org Cc: kbusch@kernel.org, axboe@kernel.dk, hch@lst.de, sagi@grimberg.me, kch@nvidia.com, hare@suse.de, alistair23@gmail.com, Alistair Francis Subject: [PATCH v6 4/5] nvme-tcp: Support KeyUpdate Date: Tue, 2 Dec 2025 11:34:28 +1000 Message-ID: <20251202013429.1199659-5-alistair.francis@wdc.com> X-Mailer: git-send-email 2.51.1 In-Reply-To: <20251202013429.1199659-1-alistair.francis@wdc.com> References: <20251202013429.1199659-1-alistair.francis@wdc.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Alistair Francis If the nvme_tcp_try_send() or nvme_tcp_try_recv() functions return EKEYEXPIRED then the underlying TLS keys need to be updated. This occurs on an KeyUpdate event as described in RFC8446 https://datatracker.ietf.org/doc/html/rfc8446#section-4.6.3. If the NVMe Target (TLS server) initiates a KeyUpdate this patch will allow the NVMe layer to process the KeyUpdate request and forward the request to userspace. Userspace must then update the key to keep the connection alive. This patch allows us to handle the NVMe target sending a KeyUpdate request without aborting the connection. At this time we don't support initiating a KeyUpdate. Based-on: 2cbe1350-0bf5-4487-be33-1d317cb73acf@suse.de Signed-off-by: Alistair Francis Reviewed-by: Hannes Reinecke Reviewed-by: Christoph Hellwig --- v6: - Don't use `struct nvme_tcp_hdr` to determine TLS_HANDSHAKE_KEYUPDATE, instead look at the cmsg fields. - Don't flush async_event_work v5: - Cleanup code flow - Check for MSG_CTRUNC in the msg_flags return from recvmsg and use that to determine if it's a control message v4: - Remove all support for initiating KeyUpdate - Don't call cancel_work() when updating keys v3: - Don't cancel existing handshake requests v2: - Don't change the state - Use a helper function for KeyUpdates - Continue sending in nvme_tcp_send_all() after a KeyUpdate - Remove command message using recvmsg drivers/nvme/host/tcp.c | 93 +++++++++++++++++++++++++++++++++-------- 1 file changed, 75 insertions(+), 18 deletions(-) diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c index 4797a4532b0d..c0966929f97b 100644 --- a/drivers/nvme/host/tcp.c +++ b/drivers/nvme/host/tcp.c @@ -172,6 +172,7 @@ struct nvme_tcp_queue { bool tls_enabled; u32 rcv_crc; u32 snd_crc; + key_serial_t handshake_session_id; __le32 exp_ddgst; __le32 recv_ddgst; struct completion tls_complete; @@ -858,7 +859,10 @@ static void nvme_tcp_handle_c2h_term(struct nvme_tcp_q= ueue *queue, static int nvme_tcp_recvmsg_pdu(struct nvme_tcp_queue *queue) { char *pdu =3D queue->pdu; + char cbuf[CMSG_LEN(sizeof(char))] =3D {}; struct msghdr msg =3D { + .msg_control =3D cbuf, + .msg_controllen =3D sizeof(cbuf), .msg_flags =3D MSG_DONTWAIT, }; struct kvec iov =3D { @@ -873,6 +877,12 @@ static int nvme_tcp_recvmsg_pdu(struct nvme_tcp_queue = *queue) if (ret <=3D 0) return ret; =20 + if (cbuf[CMSG_LEN(sizeof(char)) - 1] =3D=3D TLS_RECORD_TYPE_HANDSHAKE && + pdu[queue->pdu_offset] =3D=3D TLS_HANDSHAKE_KEYUPDATE) { + dev_err(queue->ctrl->ctrl.device, "KeyUpdate message\n"); + return 1; + } + queue->pdu_remaining -=3D ret; queue->pdu_offset +=3D ret; if (queue->pdu_remaining) @@ -944,14 +954,14 @@ static int nvme_tcp_recvmsg_data(struct nvme_tcp_queu= e *queue) struct request *rq =3D nvme_cid_to_rq(nvme_tcp_tagset(queue), pdu->command_id); struct nvme_tcp_request *req =3D blk_mq_rq_to_pdu(rq); + char cbuf[CMSG_LEN(sizeof(char))] =3D {}; + struct msghdr msg; + int ret; =20 if (nvme_tcp_recv_state(queue) !=3D NVME_TCP_RECV_DATA) return 0; =20 while (queue->data_remaining) { - struct msghdr msg; - int ret; - if (!iov_iter_count(&req->iter)) { req->curr_bio =3D req->curr_bio->bi_next; =20 @@ -975,11 +985,8 @@ static int nvme_tcp_recvmsg_data(struct nvme_tcp_queue= *queue) msg.msg_flags =3D MSG_DONTWAIT; =20 ret =3D sock_recvmsg(queue->sock, &msg, msg.msg_flags); - if (ret < 0) { - dev_err(queue->ctrl->ctrl.device, - "queue %d failed to receive request %#x data", - nvme_tcp_queue_id(queue), rq->tag); - return ret; + if (unlikely(ret < 0)) { + goto msg_control; } if (queue->data_digest) nvme_tcp_ddgst_calc(req, &queue->rcv_crc, ret); @@ -1002,6 +1009,29 @@ static int nvme_tcp_recvmsg_data(struct nvme_tcp_que= ue *queue) } } =20 + return 0; + +msg_control: + /* + * If MSG_CTRUNC is set, it's a control message, + * so let's read the control message. + */ + if (msg.msg_flags & MSG_CTRUNC) { + memset(&msg, 0, sizeof(msg)); + msg.msg_flags =3D MSG_DONTWAIT; + msg.msg_control =3D cbuf; + msg.msg_controllen =3D sizeof(cbuf); + + ret =3D sock_recvmsg(queue->sock, &msg, msg.msg_flags); + } + + if (ret < 0) { + dev_err(queue->ctrl->ctrl.device, + "queue %d failed to receive request %#x data, %d", + nvme_tcp_queue_id(queue), rq->tag, ret); + return ret; + } + return 0; } =20 @@ -1384,15 +1414,37 @@ static int nvme_tcp_try_recvmsg(struct nvme_tcp_que= ue *queue) } } while (result >=3D 0); =20 - if (result < 0 && result !=3D -EAGAIN) { - dev_err(queue->ctrl->ctrl.device, - "receive failed: %d\n", result); - queue->rd_enabled =3D false; - nvme_tcp_error_recovery(&queue->ctrl->ctrl); - } else if (result =3D=3D -EAGAIN) - result =3D 0; + if (result < 0) { + if (result !=3D -EKEYEXPIRED && result !=3D -EAGAIN) { + dev_err(queue->ctrl->ctrl.device, + "receive failed: %d\n", result); + queue->rd_enabled =3D false; + nvme_tcp_error_recovery(&queue->ctrl->ctrl); + } + return result; + } =20 - return result < 0 ? result : (queue->nr_cqe =3D nr_cqe); + queue->nr_cqe =3D nr_cqe; + return nr_cqe; +} + +static void update_tls_keys(struct nvme_tcp_queue *queue) +{ + int qid =3D nvme_tcp_queue_id(queue); + int ret; + + dev_dbg(queue->ctrl->ctrl.device, + "updating key for queue %d\n", qid); + + ret =3D nvme_tcp_start_tls(&(queue->ctrl->ctrl), + queue, queue->ctrl->ctrl.tls_pskid, + HANDSHAKE_KEY_UPDATE_TYPE_RECEIVED); + + if (ret < 0) { + dev_err(queue->ctrl->ctrl.device, + "failed to update the keys %d\n", ret); + nvme_tcp_fail_request(queue->request); + } } =20 static void nvme_tcp_io_work(struct work_struct *w) @@ -1417,8 +1469,11 @@ static void nvme_tcp_io_work(struct work_struct *w) result =3D nvme_tcp_try_recvmsg(queue); if (result > 0) pending =3D true; - else if (unlikely(result < 0)) - return; + else if (unlikely(result < 0)) { + if (result =3D=3D -EKEYEXPIRED) + update_tls_keys(queue); + break; + } =20 /* did we get some space after spending time in recv? */ if (nvme_tcp_queue_has_pending(queue) && @@ -1726,6 +1781,7 @@ static void nvme_tcp_tls_done(void *data, int status,= key_serial_t pskid, ctrl->ctrl.tls_pskid =3D key_serial(tls_key); key_put(tls_key); queue->tls_err =3D 0; + queue->handshake_session_id =3D handshake_session_id; } =20 out_complete: @@ -1755,6 +1811,7 @@ static int nvme_tcp_start_tls(struct nvme_ctrl *nctrl, keyring =3D key_serial(nctrl->opts->keyring); args.ta_keyring =3D keyring; args.ta_timeout_ms =3D tls_handshake_timeout * 1000; + args.ta_handshake_session_id =3D queue->handshake_session_id; queue->tls_err =3D -EOPNOTSUPP; init_completion(&queue->tls_complete); if (keyupdate =3D=3D HANDSHAKE_KEY_UPDATE_TYPE_UNSPEC) --=20 2.51.1