From nobody Sat Apr 11 07:03:17 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8393AC64EC4 for ; Fri, 3 Mar 2023 12:16:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230491AbjCCMQu (ORCPT ); Fri, 3 Mar 2023 07:16:50 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35326 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229881AbjCCMQs (ORCPT ); Fri, 3 Mar 2023 07:16:48 -0500 X-Greylist: delayed 562 seconds by postgrey-1.37 at lindbergh.monkeyblade.net; Fri, 03 Mar 2023 04:16:45 PST Received: from esmtp-1.proxad.net (esmtp-1.proxad.net [213.36.6.1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CBCD62A98D; Fri, 3 Mar 2023 04:16:45 -0800 (PST) Received: from localhost (localhost [127.0.0.1]) by esmtp-1.proxad.net (Postfix) with ESMTP id 7051881BB1; Fri, 3 Mar 2023 13:07:22 +0100 (CET) X-Virus-Scanned: Debian amavisd-new at proxad.net Received: from esmtp-1.proxad.net ([127.0.0.1]) by localhost (esmtp-b23-1.proxad.net [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id kvvE6ffmOTT2; Fri, 3 Mar 2023 13:07:21 +0100 (CET) Received: from zstore-5.mgt.proxad.net (unknown [172.18.94.8]) by esmtp-1.proxad.net (Postfix) with ESMTP id 9A57181A18; Fri, 3 Mar 2023 13:07:15 +0100 (CET) Date: Fri, 3 Mar 2023 13:07:15 +0100 (CET) From: Adrien Moulin To: Boris Pismenny , John Fastabend , Jakub Kicinski Cc: netdev@vger.kernel.org, linux-kernel@vger.kernel.org, Tariq Toukan Message-ID: <61481278.42813558.1677845235112.JavaMail.zimbra@corp.free.fr> Subject: TLS zerocopy sendfile offset causes data corruption MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Mailer: Zimbra 8.8.12_GA_3803 (ZimbraWebClient - SAF16.2 (Mac)/8.8.12_GA_3794) Thread-Index: lTDPthBuN+Zxmy4UPuAb4ZDFQmGN9Q== Thread-Topic: TLS zerocopy sendfile offset causes data corruption Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Hi, When doing a sendfile call on a TLS_TX_ZEROCOPY_RO-enabled socket with an o= ffset that is neither zero nor 4k-aligned, and with a "count" bigger than a= single TLS record, part of the data received will be corrupted. I am seeing this on 5.19 and 6.2.1 (x86_64) with a ConnectX-6 Dx NIC, with = TLS NIC offload including sendfile otherwise working perfectly when not usi= ng TLS_TX_ZEROCOPY_RO. I have a simple reproducer program available here https://gist.github.com/e= lyosh/922e6c15f8d4d7102c8ac9508b0cdc3b Doing sendfile of a 32K file with a 8 bytes offset, first without zerocopy : # ./ktls_test -i testfile -p 443 -c cert.pem -k key.pem -o 8 Serving file testfile, will send 32760 bytes (8 - 32768) with SHA1 sum 83fc= 1e3900cf900025311f2c27378a357f9f4d2c sendfile(5, 3, 8, 32760) =3D 32760 % wget -S -q -O test_copy https://xxxxxx/; shasum test_copy HTTP/1.1 200 OK Content-Type: application/octet-stream Content-Length: 32760 X-Source-SHA1: 83fc1e3900cf900025311f2c27378a357f9f4d2c 83fc1e3900cf900025311f2c27378a357f9f4d2c test_copy Same with TLS_TX_ZEROCOPY_RO enabled, received data will be corrupted : # ./ktls_test -i testfile -p 443 -c cert.pem -k key.pem -o 8 -z Serving file testfile, will send 32760 bytes (8 - 32768) with SHA1 sum 83fc= 1e3900cf900025311f2c27378a357f9f4d2c TLS_TX_ZEROCOPY_RO enabled sendfile(5, 3, 8, 32760) =3D 32760 % wget -S -q -O test_zerocopy https://xxxxxx/; shasum test_zerocopy HTTP/1.1 200 OK Content-Type: application/octet-stream Content-Length: 32760 X-Source-SHA1: 83fc1e3900cf900025311f2c27378a357f9f4d2c 03374f669f98d5f56837660a3817ce1d2a2819f8 test_zerocopy % diff -U 1 -d <(xxd test_copy) <(xxd test_zerocopy) = =20 --- /dev/fd/11 2023-03-03 10:13:26 +++ /dev/fd/12 2023-03-03 10:13:26 @@ -1087,3 +1087,3 @@ 000043e0: 1010 1010 1010 1010 1010 1010 1010 1010 ................ -000043f0: 1010 1010 1010 1010 1111 1111 1111 1111 ................ +000043f0: 1010 1010 1010 1010 1010 1010 1010 1010 ................ 00004400: 1111 1111 1111 1111 1111 1111 1111 1111 ................ @@ -1151,3 +1151,3 @@ 000047e0: 1111 1111 1111 1111 1111 1111 1111 1111 ................ -000047f0: 1111 1111 1111 1111 1212 1212 1212 1212 ................ +000047f0: 1111 1111 1111 1111 1111 1111 1111 1111 ................ 00004800: 1212 1212 1212 1212 1212 1212 1212 1212 ................ @@ -1215,3 +1215,3 @@ 00004be0: 1212 1212 1212 1212 1212 1212 1212 1212 ................ -00004bf0: 1212 1212 1212 1212 1313 1313 1313 1313 ................ +00004bf0: 1212 1212 1212 1212 1212 1212 1212 1212 ................ 00004c00: 1313 1313 1313 1313 1313 1313 1313 1313 ................ For context, I noticed this issue trying to serve cached files with nginx. = For static files this works fine (sendfile offset is zero at first, then 16= k-aligned), but cached files are stored with a ~500 bytes header that is sk= ipped in the sendfile call, triggering this issue. Best regards --=20 Adrien Moulin