From nobody Sat Feb 7 15:22:27 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D9CB3C77B73 for ; Wed, 31 May 2023 12:46:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235929AbjEaMqa (ORCPT ); Wed, 31 May 2023 08:46:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43148 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235901AbjEaMq2 (ORCPT ); Wed, 31 May 2023 08:46:28 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 02DCE124 for ; Wed, 31 May 2023 05:45:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1685537142; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=VUrQef8DmIGGqxD5N5RNkU1BHyTMou9oFGzjwHHlMuE=; b=H5yRwNeVIjOnlS5NbLxWGa6hsALP2DgsXpmWB5byNmH/bj/eNw71uz3bZdOwWFtI/nvJhU 5rWTUaAGkSL09UJIWE0YSaiDFR5GScNUu1OziMoAHBDYKBhVEbvvAVgAwEk2q4no7YnUE/ IN+QKUgcOkPNSI68si7CnfvicpooxVk= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-479-MLbnjYjsMtCMKcbnjqfptw-1; Wed, 31 May 2023 08:45:38 -0400 X-MC-Unique: MLbnjYjsMtCMKcbnjqfptw-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id DE0311C0150B; Wed, 31 May 2023 12:45:37 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.182]) by smtp.corp.redhat.com (Postfix) with ESMTP id 1CF7F20296C8; Wed, 31 May 2023 12:45:34 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , Chuck Lever , Boris Pismenny , John Fastabend , Jakub Kicinski , "David S. Miller" , Eric Dumazet , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Jens Axboe , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Christoph Hellwig , Linus Torvalds , Al Viro , Jan Kara , Jeff Layton , David Hildenbrand , Christian Brauner , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org Subject: [PATCH net-next v2 1/6] splice, net: Fix MSG_MORE signalling in splice_direct_to_actor() Date: Wed, 31 May 2023 13:45:23 +0100 Message-ID: <20230531124528.699123-2-dhowells@redhat.com> In-Reply-To: <20230531124528.699123-1-dhowells@redhat.com> References: <20230531124528.699123-1-dhowells@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" splice_direct_to_actor() doesn't manage SPLICE_F_MORE correctly - and, as a result, incorrectly signals MSG_MORE when splicing to a socket. The problem happens when a short splice occurs because we got a short read due to hitting the EOF on a file. Because the length read (read_len) is less than the remaining size to be spliced (len), SPLICE_F_MORE is set. This causes MSG_MORE to be set by pipe_to_sendpage(), indicating to the network protocol that more data is to be expected. With the changes I want to make to switch from using sendpage to using sendmsg(MSG_SPLICE_PAGES), MSG_MORE needs to work properly. This was observed with the multi_chunk_sendfile tests in the tls kselftest program. Some of those tests would hang and time out when the last chunk of file was less than the sendfile request size. This has been observed before[1] and worked around in AF_TLS[2]. Fix this by checking to see if the source file is seekable if we get a short read and, if it is, checking to see if we hit the file size. This should also work for block devices. This won't help procfiles and suchlike as they're zero length files that can be read from[3]. To handle that, should splice make a zero-length call with SPLICE_F_MORE cleared (assuming it wasn't set by userspace via splice()) if it gets a zero-length read? Signed-off-by: David Howells cc: Jakub Kicinski cc: Jens Axboe cc: Christoph Hellwig cc: Linus Torvalds cc: Al Viro cc: Matthew Wilcox cc: Jan Kara cc: Jeff Layton cc: David Hildenbrand cc: Christian Brauner cc: Chuck Lever cc: Boris Pismenny cc: John Fastabend cc: Eric Dumazet cc: "David S. Miller" cc: Paolo Abeni cc: linux-fsdevel@vger.kernel.org cc: linux-block@vger.kernel.org cc: linux-mm@kvack.org cc: netdev@vger.kernel.org Link: https://lore.kernel.org/netdev/1591392508-14592-1-git-send-email-pooj= a.trivedi@stackpath.com/ [1] Link: https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/c= ommit/?id=3Dd452d48b9f8b1a7f8152d33ef52cfd7fe1735b0a [2] Link: https://lore.kernel.org/r/CAHk-=3DwjDq5_wLWrapzFiJ3ZNn6aGFWeMJpAj5q+4= z-Ok8DD9dA@mail.gmail.com/ [3] --- fs/splice.c | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-) diff --git a/fs/splice.c b/fs/splice.c index 3e06611d19ae..a7cf216c02a7 100644 --- a/fs/splice.c +++ b/fs/splice.c @@ -982,10 +982,21 @@ ssize_t splice_direct_to_actor(struct file *in, struc= t splice_desc *sd, * If this is the last data and SPLICE_F_MORE was not set * initially, clears it. */ - if (read_len < len) - sd->flags |=3D SPLICE_F_MORE; - else if (!more) + if (read_len < len) { + struct inode *ii =3D in->f_mapping->host; + + if (ii->i_fop->llseek !=3D noop_llseek && + pos >=3D i_size_read(ii)) { + if (!more) + sd->flags &=3D ~SPLICE_F_MORE; + } else { + sd->flags |=3D SPLICE_F_MORE; + } + + } else if (!more) { sd->flags &=3D ~SPLICE_F_MORE; + } + /* * NOTE: nonblocking mode only applies to the input. We * must not do the output in nonblocking mode as then we From nobody Sat Feb 7 15:22:27 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B8BBFC77B7A for ; Wed, 31 May 2023 12:46:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235950AbjEaMqh (ORCPT ); Wed, 31 May 2023 08:46:37 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43160 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235909AbjEaMqa (ORCPT ); Wed, 31 May 2023 08:46:30 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AE7B7128 for ; Wed, 31 May 2023 05:45:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1685537144; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=704Gul1es5vKB8f3fNlqLP0xfed1dEhsuWb+SLm3klg=; b=WEbs/vgqtdc7UsAEh8O4M0ghNHBj1PiB9QnqdNiMATXe6KuZ8WDckAVxnrIYpEeGtRZpVz Dgg/KbMZeVf2+T/0e7BHrIPzac6BBLQbKaYG70TI6YRnEFVgKwlbEQiY+ZZqfCwLCr/t0y 9hwdKHHc0uU/gAIZpQIyaU3KoQxhhLU= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-625-u-3ocbAEPfKvWCDqcrmGEg-1; Wed, 31 May 2023 08:45:41 -0400 X-MC-Unique: u-3ocbAEPfKvWCDqcrmGEg-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id A721F3C0C897; Wed, 31 May 2023 12:45:40 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.182]) by smtp.corp.redhat.com (Postfix) with ESMTP id B74EE112132E; Wed, 31 May 2023 12:45:38 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , Chuck Lever , Boris Pismenny , John Fastabend , Jakub Kicinski , "David S. Miller" , Eric Dumazet , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Jens Axboe , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH net-next v2 2/6] net: Block MSG_SENDPAGE_* from being passed to sendmsg() by userspace Date: Wed, 31 May 2023 13:45:24 +0100 Message-ID: <20230531124528.699123-3-dhowells@redhat.com> In-Reply-To: <20230531124528.699123-1-dhowells@redhat.com> References: <20230531124528.699123-1-dhowells@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" It is necessary to allow MSG_SENDPAGE_* to be passed into ->sendmsg() to allow sendmsg(MSG_SPLICE_PAGES) to replace ->sendpage(). Unblocking them in the network protocol, however, allows these flags to be passed in by userspace too[1]. Fix this by marking MSG_SENDPAGE_NOPOLICY, MSG_SENDPAGE_NOTLAST and MSG_SENDPAGE_DECRYPTED as internal flags, which causes sendmsg() to object if they are passed to sendmsg() by userspace. Network protocol ->sendmsg() implementations can then allow them through. Note that it should be possible to remove MSG_SENDPAGE_NOTLAST once sendpage is removed as a whole slew of pages will be passed in in one go by splice through sendmsg, with MSG_MORE being set if it has more data waiting in the pipe. Signed-off-by: David Howells cc: Jakub Kicinski cc: Chuck Lever cc: Boris Pismenny cc: John Fastabend cc: Eric Dumazet cc: "David S. Miller" cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org Link: https://lore.kernel.org/r/20230526181338.03a99016@kernel.org/ [1] --- include/linux/socket.h | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/include/linux/socket.h b/include/linux/socket.h index bd1cc3238851..3fd3436bc09f 100644 --- a/include/linux/socket.h +++ b/include/linux/socket.h @@ -339,7 +339,9 @@ struct ucred { #endif =20 /* Flags to be cleared on entry by sendmsg and sendmmsg syscalls */ -#define MSG_INTERNAL_SENDMSG_FLAGS (MSG_SPLICE_PAGES) +#define MSG_INTERNAL_SENDMSG_FLAGS \ + (MSG_SPLICE_PAGES | MSG_SENDPAGE_NOPOLICY | MSG_SENDPAGE_NOTLAST | \ + MSG_SENDPAGE_DECRYPTED) =20 /* Setsockoptions(2) level. Thanks to BSD these must match IPPROTO_xxx */ #define SOL_IP 0 From nobody Sat Feb 7 15:22:27 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 98598C77B73 for ; Wed, 31 May 2023 12:47:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235957AbjEaMrN (ORCPT ); Wed, 31 May 2023 08:47:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43308 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235902AbjEaMrI (ORCPT ); Wed, 31 May 2023 08:47:08 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8BA9311F for ; Wed, 31 May 2023 05:45:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1685537148; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=xFpJiBMaV6haKLFUiEEfF2pWt3wFFqyfTnw9cLbfIuI=; b=ZE7fg6BIXSpMhLnW1clDpdZMGuwFlPPwrT4GfLeLtte4aZ40GZam6apjFQMk3M7vdDCbRm jokXXHV8C7jkofiVAF5OjRWlU2lDfl7umRAJ/aJ9On2aZLMxxM7Q3/OJbW5bArFCq4XdiR uqZQsMD3dYPzpLirFrtrRmS4DoOe+h8= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-116-ZBHr42RPMCeBYoDi-5JaaA-1; Wed, 31 May 2023 08:45:44 -0400 X-MC-Unique: ZBHr42RPMCeBYoDi-5JaaA-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id C5504811E7F; Wed, 31 May 2023 12:45:43 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.182]) by smtp.corp.redhat.com (Postfix) with ESMTP id 99826112132C; Wed, 31 May 2023 12:45:41 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , Chuck Lever , Boris Pismenny , John Fastabend , Jakub Kicinski , "David S. Miller" , Eric Dumazet , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Jens Axboe , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH net-next v2 3/6] tls/sw: Support MSG_SPLICE_PAGES Date: Wed, 31 May 2023 13:45:25 +0100 Message-ID: <20230531124528.699123-4-dhowells@redhat.com> In-Reply-To: <20230531124528.699123-1-dhowells@redhat.com> References: <20230531124528.699123-1-dhowells@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Make TLS's sendmsg() support MSG_SPLICE_PAGES. This causes pages to be spliced from the source iterator if possible. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: Chuck Lever cc: Boris Pismenny cc: John Fastabend cc: Jakub Kicinski cc: Eric Dumazet cc: "David S. Miller" cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- Notes: ver #2) - "rls_" should be "tls_". net/tls/tls_sw.c | 46 +++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 45 insertions(+), 1 deletion(-) diff --git a/net/tls/tls_sw.c b/net/tls/tls_sw.c index 6e6a7c37d685..60d807b63f75 100644 --- a/net/tls/tls_sw.c +++ b/net/tls/tls_sw.c @@ -929,6 +929,38 @@ static int tls_sw_push_pending_record(struct sock *sk,= int flags) &copied, flags); } =20 +static int tls_sw_sendmsg_splice(struct sock *sk, struct msghdr *msg, + struct sk_msg *msg_pl, size_t try_to_copy, + ssize_t *copied) +{ + struct page *page =3D NULL, **pages =3D &page; + + do { + ssize_t part; + size_t off; + bool put =3D false; + + part =3D iov_iter_extract_pages(&msg->msg_iter, &pages, + try_to_copy, 1, 0, &off); + if (part <=3D 0) + return part ?: -EIO; + + if (WARN_ON_ONCE(!sendpage_ok(page))) { + iov_iter_revert(&msg->msg_iter, part); + return -EIO; + } + + sk_msg_page_add(msg_pl, page, part, off); + sk_mem_charge(sk, part); + if (put) + put_page(page); + *copied +=3D part; + try_to_copy -=3D part; + } while (try_to_copy && !sk_msg_full(msg_pl)); + + return 0; +} + int tls_sw_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) { long timeo =3D sock_sndtimeo(sk, msg->msg_flags & MSG_DONTWAIT); @@ -1018,6 +1050,17 @@ int tls_sw_sendmsg(struct sock *sk, struct msghdr *m= sg, size_t size) full_record =3D true; } =20 + if (try_to_copy && (msg->msg_flags & MSG_SPLICE_PAGES)) { + ret =3D tls_sw_sendmsg_splice(sk, msg, msg_pl, + try_to_copy, &copied); + if (ret < 0) + goto send_end; + tls_ctx->pending_open_record_frags =3D true; + if (full_record || eor || sk_msg_full(msg_pl)) + goto copied; + continue; + } + if (!is_kvec && (full_record || eor) && !async_capable) { u32 first =3D msg_pl->sg.end; =20 @@ -1080,8 +1123,9 @@ int tls_sw_sendmsg(struct sock *sk, struct msghdr *ms= g, size_t size) /* Open records defined only if successfully copied, otherwise * we would trim the sg but not reset the open record frags. */ - tls_ctx->pending_open_record_frags =3D true; copied +=3D try_to_copy; +copied: + tls_ctx->pending_open_record_frags =3D true; if (full_record || eor) { ret =3D bpf_exec_tx_verdict(msg_pl, sk, full_record, record_type, &copied, From nobody Sat Feb 7 15:22:27 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 31CF7C77B7A for ; Wed, 31 May 2023 12:47:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235968AbjEaMrQ (ORCPT ); Wed, 31 May 2023 08:47:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43306 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233059AbjEaMrK (ORCPT ); Wed, 31 May 2023 08:47:10 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 87707129 for ; Wed, 31 May 2023 05:45:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1685537153; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=W8aEdMrYnuprZg7eZCUzLy+kKHthZDT31VQbMyJsAn8=; b=ShjY5812qP/ExB4WDJqrXmXVKXQ/Gk0bs0OguC20oTpz4LcvF7IDDvHpzJ5RQe2o6EMDZQ rd7KuxwTMiG/w9ouCAjaNB2VETPyNuJnqYHUeJxQLPfBGhTpHLn1MIRPfPJ2P5jrVvZkDB X7wuH6NdWEpvT4yOZ5vvTRPy61ki28Q= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-370-LbK9yftePiayyMeYHhVc5g-1; Wed, 31 May 2023 08:45:48 -0400 X-MC-Unique: LbK9yftePiayyMeYHhVc5g-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id D0DF53825BA9; Wed, 31 May 2023 12:45:47 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.182]) by smtp.corp.redhat.com (Postfix) with ESMTP id F3555140E963; Wed, 31 May 2023 12:45:44 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , Chuck Lever , Boris Pismenny , John Fastabend , Jakub Kicinski , "David S. Miller" , Eric Dumazet , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Jens Axboe , linux-mm@kvack.org, linux-kernel@vger.kernel.org, bpf@vger.kernel.org Subject: [PATCH net-next v2 4/6] tls/sw: Convert tls_sw_sendpage() to use MSG_SPLICE_PAGES Date: Wed, 31 May 2023 13:45:26 +0100 Message-ID: <20230531124528.699123-5-dhowells@redhat.com> In-Reply-To: <20230531124528.699123-1-dhowells@redhat.com> References: <20230531124528.699123-1-dhowells@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.1 on 10.11.54.7 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Convert tls_sw_sendpage() and tls_sw_sendpage_locked() to use sendmsg() with MSG_SPLICE_PAGES rather than directly splicing in the pages itself. [!] Note that tls_sw_sendpage_locked() appears to have the wrong locking upstream. I think the caller will only hold the socket lock, but it should hold tls_ctx->tx_lock too. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: Chuck Lever cc: Boris Pismenny cc: John Fastabend cc: Jakub Kicinski cc: Eric Dumazet cc: "David S. Miller" cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org cc: bpf@vger.kernel.org --- net/tls/tls_sw.c | 165 +++++++++-------------------------------------- 1 file changed, 31 insertions(+), 134 deletions(-) diff --git a/net/tls/tls_sw.c b/net/tls/tls_sw.c index 60d807b63f75..f63e4405cf34 100644 --- a/net/tls/tls_sw.c +++ b/net/tls/tls_sw.c @@ -961,7 +961,8 @@ static int tls_sw_sendmsg_splice(struct sock *sk, struc= t msghdr *msg, return 0; } =20 -int tls_sw_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) +static int tls_sw_sendmsg_locked(struct sock *sk, struct msghdr *msg, + size_t size) { long timeo =3D sock_sndtimeo(sk, msg->msg_flags & MSG_DONTWAIT); struct tls_context *tls_ctx =3D tls_get_ctx(sk); @@ -984,15 +985,6 @@ int tls_sw_sendmsg(struct sock *sk, struct msghdr *msg= , size_t size) int ret =3D 0; int pending; =20 - if (msg->msg_flags & ~(MSG_MORE | MSG_DONTWAIT | MSG_NOSIGNAL | - MSG_CMSG_COMPAT)) - return -EOPNOTSUPP; - - ret =3D mutex_lock_interruptible(&tls_ctx->tx_lock); - if (ret) - return ret; - lock_sock(sk); - if (unlikely(msg->msg_controllen)) { ret =3D tls_process_cmsg(sk, msg, &record_type); if (ret) { @@ -1193,157 +1185,62 @@ int tls_sw_sendmsg(struct sock *sk, struct msghdr = *msg, size_t size) =20 send_end: ret =3D sk_stream_error(sk, msg->msg_flags, ret); - - release_sock(sk); - mutex_unlock(&tls_ctx->tx_lock); return copied > 0 ? copied : ret; } =20 -static int tls_sw_do_sendpage(struct sock *sk, struct page *page, - int offset, size_t size, int flags) +int tls_sw_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) { - long timeo =3D sock_sndtimeo(sk, flags & MSG_DONTWAIT); struct tls_context *tls_ctx =3D tls_get_ctx(sk); - struct tls_sw_context_tx *ctx =3D tls_sw_ctx_tx(tls_ctx); - struct tls_prot_info *prot =3D &tls_ctx->prot_info; - unsigned char record_type =3D TLS_RECORD_TYPE_DATA; - struct sk_msg *msg_pl; - struct tls_rec *rec; - int num_async =3D 0; - ssize_t copied =3D 0; - bool full_record; - int record_room; - int ret =3D 0; - bool eor; - - eor =3D !(flags & MSG_SENDPAGE_NOTLAST); - sk_clear_bit(SOCKWQ_ASYNC_NOSPACE, sk); - - /* Call the sk_stream functions to manage the sndbuf mem. */ - while (size > 0) { - size_t copy, required_size; - - if (sk->sk_err) { - ret =3D -sk->sk_err; - goto sendpage_end; - } - - if (ctx->open_rec) - rec =3D ctx->open_rec; - else - rec =3D ctx->open_rec =3D tls_get_rec(sk); - if (!rec) { - ret =3D -ENOMEM; - goto sendpage_end; - } - - msg_pl =3D &rec->msg_plaintext; - - full_record =3D false; - record_room =3D TLS_MAX_PAYLOAD_SIZE - msg_pl->sg.size; - copy =3D size; - if (copy >=3D record_room) { - copy =3D record_room; - full_record =3D true; - } - - required_size =3D msg_pl->sg.size + copy + prot->overhead_size; - - if (!sk_stream_memory_free(sk)) - goto wait_for_sndbuf; -alloc_payload: - ret =3D tls_alloc_encrypted_msg(sk, required_size); - if (ret) { - if (ret !=3D -ENOSPC) - goto wait_for_memory; - - /* Adjust copy according to the amount that was - * actually allocated. The difference is due - * to max sg elements limit - */ - copy -=3D required_size - msg_pl->sg.size; - full_record =3D true; - } - - sk_msg_page_add(msg_pl, page, copy, offset); - sk_mem_charge(sk, copy); - - offset +=3D copy; - size -=3D copy; - copied +=3D copy; - - tls_ctx->pending_open_record_frags =3D true; - if (full_record || eor || sk_msg_full(msg_pl)) { - ret =3D bpf_exec_tx_verdict(msg_pl, sk, full_record, - record_type, &copied, flags); - if (ret) { - if (ret =3D=3D -EINPROGRESS) - num_async++; - else if (ret =3D=3D -ENOMEM) - goto wait_for_memory; - else if (ret !=3D -EAGAIN) { - if (ret =3D=3D -ENOSPC) - ret =3D 0; - goto sendpage_end; - } - } - } - continue; -wait_for_sndbuf: - set_bit(SOCK_NOSPACE, &sk->sk_socket->flags); -wait_for_memory: - ret =3D sk_stream_wait_memory(sk, &timeo); - if (ret) { - if (ctx->open_rec) - tls_trim_both_msgs(sk, msg_pl->sg.size); - goto sendpage_end; - } + int ret; =20 - if (ctx->open_rec) - goto alloc_payload; - } + if (msg->msg_flags & ~(MSG_MORE | MSG_DONTWAIT | MSG_NOSIGNAL | + MSG_CMSG_COMPAT | MSG_SPLICE_PAGES | + MSG_SENDPAGE_NOTLAST | MSG_SENDPAGE_NOPOLICY)) + return -EOPNOTSUPP; =20 - if (num_async) { - /* Transmit if any encryptions have completed */ - if (test_and_clear_bit(BIT_TX_SCHEDULED, &ctx->tx_bitmask)) { - cancel_delayed_work(&ctx->tx_work.work); - tls_tx_records(sk, flags); - } - } -sendpage_end: - ret =3D sk_stream_error(sk, flags, ret); - return copied > 0 ? copied : ret; + ret =3D mutex_lock_interruptible(&tls_ctx->tx_lock); + if (ret) + return ret; + lock_sock(sk); + ret =3D tls_sw_sendmsg_locked(sk, msg, size); + release_sock(sk); + mutex_unlock(&tls_ctx->tx_lock); + return ret; } =20 int tls_sw_sendpage_locked(struct sock *sk, struct page *page, int offset, size_t size, int flags) { + struct bio_vec bvec; + struct msghdr msg =3D { .msg_flags =3D flags | MSG_SPLICE_PAGES, }; + if (flags & ~(MSG_MORE | MSG_DONTWAIT | MSG_NOSIGNAL | MSG_SENDPAGE_NOTLAST | MSG_SENDPAGE_NOPOLICY | MSG_NO_SHARED_FRAGS)) return -EOPNOTSUPP; + if (flags & MSG_SENDPAGE_NOTLAST) + msg.msg_flags |=3D MSG_MORE; =20 - return tls_sw_do_sendpage(sk, page, offset, size, flags); + bvec_set_page(&bvec, page, size, offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); + return tls_sw_sendmsg_locked(sk, &msg, size); } =20 int tls_sw_sendpage(struct sock *sk, struct page *page, int offset, size_t size, int flags) { - struct tls_context *tls_ctx =3D tls_get_ctx(sk); - int ret; + struct bio_vec bvec; + struct msghdr msg =3D { .msg_flags =3D flags | MSG_SPLICE_PAGES, }; =20 if (flags & ~(MSG_MORE | MSG_DONTWAIT | MSG_NOSIGNAL | MSG_SENDPAGE_NOTLAST | MSG_SENDPAGE_NOPOLICY)) return -EOPNOTSUPP; + if (flags & MSG_SENDPAGE_NOTLAST) + msg.msg_flags |=3D MSG_MORE; =20 - ret =3D mutex_lock_interruptible(&tls_ctx->tx_lock); - if (ret) - return ret; - lock_sock(sk); - ret =3D tls_sw_do_sendpage(sk, page, offset, size, flags); - release_sock(sk); - mutex_unlock(&tls_ctx->tx_lock); - return ret; + bvec_set_page(&bvec, page, size, offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); + return tls_sw_sendmsg(sk, &msg, size); } =20 static int From nobody Sat Feb 7 15:22:27 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 56E45C77B73 for ; Wed, 31 May 2023 12:47:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235892AbjEaMrK (ORCPT ); Wed, 31 May 2023 08:47:10 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43310 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233059AbjEaMrI (ORCPT ); Wed, 31 May 2023 08:47:08 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 10EF812E for ; Wed, 31 May 2023 05:45:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1685537154; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=uuYRuRqPrqYrWBy2myOfoGGw2Qio81ibrygQkKyKemI=; b=G09xyAbtaLd0YusgR+cF8uHNuSVjQQSuw1TsOzPj7hQdntCUi0JadTv4rYgXGARJSuDioR Fr5gGLzY1KcR0F9XnZMXtw9/gsFVvV8/zECc+g1yiEKURxPLYzvKPOR/yQCO+5YZp3S0Sw gbuUMpzuiIX8dPNmfV9Fuomq/deQrh0= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-54-R6gsZvE5P7aGqYDd_4hZXw-1; Wed, 31 May 2023 08:45:51 -0400 X-MC-Unique: R6gsZvE5P7aGqYDd_4hZXw-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 650F3811E7F; Wed, 31 May 2023 12:45:50 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.182]) by smtp.corp.redhat.com (Postfix) with ESMTP id 994162166B25; Wed, 31 May 2023 12:45:48 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , Chuck Lever , Boris Pismenny , John Fastabend , Jakub Kicinski , "David S. Miller" , Eric Dumazet , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Jens Axboe , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH net-next v2 5/6] tls/device: Support MSG_SPLICE_PAGES Date: Wed, 31 May 2023 13:45:27 +0100 Message-ID: <20230531124528.699123-6-dhowells@redhat.com> In-Reply-To: <20230531124528.699123-1-dhowells@redhat.com> References: <20230531124528.699123-1-dhowells@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.1 on 10.11.54.6 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Make TLS's device sendmsg() support MSG_SPLICE_PAGES. This causes pages to be spliced from the source iterator if possible. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: Chuck Lever cc: Boris Pismenny cc: John Fastabend cc: Jakub Kicinski cc: Eric Dumazet cc: "David S. Miller" cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- net/tls/tls_device.c | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/net/tls/tls_device.c b/net/tls/tls_device.c index a959572a816f..221fb30cba51 100644 --- a/net/tls/tls_device.c +++ b/net/tls/tls_device.c @@ -508,6 +508,29 @@ static int tls_push_data(struct sock *sk, tls_append_frag(record, &zc_pfrag, copy); =20 iter_offset.offset +=3D copy; + } else if (copy && (flags & MSG_SPLICE_PAGES)) { + struct page_frag zc_pfrag; + struct page **pages =3D &zc_pfrag.page; + size_t off; + + rc =3D iov_iter_extract_pages(iter_offset.msg_iter, + &pages, copy, 1, 0, &off); + if (rc <=3D 0) { + if (rc =3D=3D 0) + rc =3D -EIO; + goto handle_error; + } + copy =3D rc; + + if (WARN_ON_ONCE(!sendpage_ok(zc_pfrag.page))) { + iov_iter_revert(iter_offset.msg_iter, copy); + rc =3D -EIO; + goto handle_error; + } + + zc_pfrag.offset =3D off; + zc_pfrag.size =3D copy; + tls_append_frag(record, &zc_pfrag, copy); } else if (copy) { copy =3D min_t(size_t, copy, pfrag->size - pfrag->offset); =20 @@ -571,6 +594,9 @@ int tls_device_sendmsg(struct sock *sk, struct msghdr *= msg, size_t size) union tls_iter_offset iter; int rc; =20 + if (!tls_ctx->zerocopy_sendfile) + msg->msg_flags &=3D ~MSG_SPLICE_PAGES; + mutex_lock(&tls_ctx->tx_lock); lock_sock(sk); From nobody Sat Feb 7 15:22:27 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 63A9BC77B7A for ; Wed, 31 May 2023 12:47:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235964AbjEaMrZ (ORCPT ); Wed, 31 May 2023 08:47:25 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43320 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235952AbjEaMrS (ORCPT ); Wed, 31 May 2023 08:47:18 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 80415134 for ; Wed, 31 May 2023 05:45:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1685537155; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=jiNOYIAVcjsuvSMtZGlGjCJlFKnRRxqCgJAQxTqAqH0=; b=P2rFeRnDBXTO8rq+tGSr2ui7YwqxORfzS2p2woTWB5rktf2HKwYkRZjUW3MjzBUCla925K i61pGochNNWVPHAYMDQufTkufD9AhENE6pBySiKrRZuOXhzUMNnU967Hm1gBO+XAxt0GoR Ho5F3V9Tz8boDrzjhHxZdb9nJq/bf4o= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-198-cKdTb43SN-mmwECm3okGRg-1; Wed, 31 May 2023 08:45:54 -0400 X-MC-Unique: cKdTb43SN-mmwECm3okGRg-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 3AE7B8032E4; Wed, 31 May 2023 12:45:53 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.182]) by smtp.corp.redhat.com (Postfix) with ESMTP id 2F12040C6EC4; Wed, 31 May 2023 12:45:51 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , Chuck Lever , Boris Pismenny , John Fastabend , Jakub Kicinski , "David S. Miller" , Eric Dumazet , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Jens Axboe , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH net-next v2 6/6] tls/device: Convert tls_device_sendpage() to use MSG_SPLICE_PAGES Date: Wed, 31 May 2023 13:45:28 +0100 Message-ID: <20230531124528.699123-7-dhowells@redhat.com> In-Reply-To: <20230531124528.699123-1-dhowells@redhat.com> References: <20230531124528.699123-1-dhowells@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.1 on 10.11.54.2 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Convert tls_device_sendpage() to use sendmsg() with MSG_SPLICE_PAGES rather than directly splicing in the pages itself. With that, the tls_iter_offset union is no longer necessary and can be replaced with an iov_iter pointer and the zc_page argument to tls_push_data() can also be removed. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: Chuck Lever cc: Boris Pismenny cc: John Fastabend cc: Jakub Kicinski cc: Eric Dumazet cc: "David S. Miller" cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- net/tls/tls_device.c | 84 +++++++++++--------------------------------- 1 file changed, 20 insertions(+), 64 deletions(-) diff --git a/net/tls/tls_device.c b/net/tls/tls_device.c index 221fb30cba51..684171363dd9 100644 --- a/net/tls/tls_device.c +++ b/net/tls/tls_device.c @@ -422,16 +422,10 @@ static int tls_device_copy_data(void *addr, size_t by= tes, struct iov_iter *i) return 0; } =20 -union tls_iter_offset { - struct iov_iter *msg_iter; - int offset; -}; - static int tls_push_data(struct sock *sk, - union tls_iter_offset iter_offset, + struct iov_iter *iter, size_t size, int flags, - unsigned char record_type, - struct page *zc_page) + unsigned char record_type) { struct tls_context *tls_ctx =3D tls_get_ctx(sk); struct tls_prot_info *prot =3D &tls_ctx->prot_info; @@ -499,22 +493,13 @@ static int tls_push_data(struct sock *sk, record =3D ctx->open_record; =20 copy =3D min_t(size_t, size, max_open_record_len - record->len); - if (copy && zc_page) { - struct page_frag zc_pfrag; - - zc_pfrag.page =3D zc_page; - zc_pfrag.offset =3D iter_offset.offset; - zc_pfrag.size =3D copy; - tls_append_frag(record, &zc_pfrag, copy); - - iter_offset.offset +=3D copy; - } else if (copy && (flags & MSG_SPLICE_PAGES)) { + if (copy && (flags & MSG_SPLICE_PAGES)) { struct page_frag zc_pfrag; struct page **pages =3D &zc_pfrag.page; size_t off; =20 - rc =3D iov_iter_extract_pages(iter_offset.msg_iter, - &pages, copy, 1, 0, &off); + rc =3D iov_iter_extract_pages(iter, &pages, + copy, 1, 0, &off); if (rc <=3D 0) { if (rc =3D=3D 0) rc =3D -EIO; @@ -523,7 +508,7 @@ static int tls_push_data(struct sock *sk, copy =3D rc; =20 if (WARN_ON_ONCE(!sendpage_ok(zc_pfrag.page))) { - iov_iter_revert(iter_offset.msg_iter, copy); + iov_iter_revert(iter, copy); rc =3D -EIO; goto handle_error; } @@ -536,7 +521,7 @@ static int tls_push_data(struct sock *sk, =20 rc =3D tls_device_copy_data(page_address(pfrag->page) + pfrag->offset, copy, - iter_offset.msg_iter); + iter); if (rc) goto handle_error; tls_append_frag(record, pfrag, copy); @@ -591,7 +576,6 @@ int tls_device_sendmsg(struct sock *sk, struct msghdr *= msg, size_t size) { unsigned char record_type =3D TLS_RECORD_TYPE_DATA; struct tls_context *tls_ctx =3D tls_get_ctx(sk); - union tls_iter_offset iter; int rc; =20 if (!tls_ctx->zerocopy_sendfile) @@ -606,8 +590,8 @@ int tls_device_sendmsg(struct sock *sk, struct msghdr *= msg, size_t size) goto out; } =20 - iter.msg_iter =3D &msg->msg_iter; - rc =3D tls_push_data(sk, iter, size, msg->msg_flags, record_type, NULL); + rc =3D tls_push_data(sk, &msg->msg_iter, size, msg->msg_flags, + record_type); =20 out: release_sock(sk); @@ -618,44 +602,18 @@ int tls_device_sendmsg(struct sock *sk, struct msghdr= *msg, size_t size) int tls_device_sendpage(struct sock *sk, struct page *page, int offset, size_t size, int flags) { - struct tls_context *tls_ctx =3D tls_get_ctx(sk); - union tls_iter_offset iter_offset; - struct iov_iter msg_iter; - char *kaddr; - struct kvec iov; - int rc; + struct bio_vec bvec; + struct msghdr msg =3D { .msg_flags =3D flags | MSG_SPLICE_PAGES, }; =20 if (flags & MSG_SENDPAGE_NOTLAST) - flags |=3D MSG_MORE; - - mutex_lock(&tls_ctx->tx_lock); - lock_sock(sk); + msg.msg_flags |=3D MSG_MORE; =20 - if (flags & MSG_OOB) { - rc =3D -EOPNOTSUPP; - goto out; - } - - if (tls_ctx->zerocopy_sendfile) { - iter_offset.offset =3D offset; - rc =3D tls_push_data(sk, iter_offset, size, - flags, TLS_RECORD_TYPE_DATA, page); - goto out; - } - - kaddr =3D kmap(page); - iov.iov_base =3D kaddr + offset; - iov.iov_len =3D size; - iov_iter_kvec(&msg_iter, ITER_SOURCE, &iov, 1, size); - iter_offset.msg_iter =3D &msg_iter; - rc =3D tls_push_data(sk, iter_offset, size, flags, TLS_RECORD_TYPE_DATA, - NULL); - kunmap(page); + if (flags & MSG_OOB) + return -EOPNOTSUPP; =20 -out: - release_sock(sk); - mutex_unlock(&tls_ctx->tx_lock); - return rc; + bvec_set_page(&bvec, page, size, offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); + return tls_device_sendmsg(sk, &msg, size); } =20 struct tls_record_info *tls_get_record(struct tls_offload_context_tx *cont= ext, @@ -720,12 +678,10 @@ EXPORT_SYMBOL(tls_get_record); =20 static int tls_device_push_pending_record(struct sock *sk, int flags) { - union tls_iter_offset iter; - struct iov_iter msg_iter; + struct iov_iter iter; =20 - iov_iter_kvec(&msg_iter, ITER_SOURCE, NULL, 0, 0); - iter.msg_iter =3D &msg_iter; - return tls_push_data(sk, iter, 0, flags, TLS_RECORD_TYPE_DATA, NULL); + iov_iter_kvec(&iter, ITER_SOURCE, NULL, 0, 0); + return tls_push_data(sk, &iter, 0, flags, TLS_RECORD_TYPE_DATA); } =20 void tls_device_write_space(struct sock *sk, struct tls_context *ctx)