From nobody Fri May 8 04:33:26 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BCF3DC433EF for ; Wed, 11 May 2022 03:55:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239334AbiEKDzl (ORCPT ); Tue, 10 May 2022 23:55:41 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35644 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238500AbiEKDzb (ORCPT ); Tue, 10 May 2022 23:55:31 -0400 Received: from mail-pf1-x435.google.com (mail-pf1-x435.google.com [IPv6:2607:f8b0:4864:20::435]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BEFDE21013A for ; Tue, 10 May 2022 20:55:28 -0700 (PDT) Received: by mail-pf1-x435.google.com with SMTP id d25so874616pfo.10 for ; Tue, 10 May 2022 20:55:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fastly.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=2p2fPjfsIsJ6E9/ORkfpw1BwG9ijb5KY/aOnRzBglX8=; b=pUfw03v2yYPpGXgiiO1AI1VBqjOOMzwAHj2d2Yks2pr5NwxeXTDwKx+aDG4hnBLUPJ DyRLE8I/8b4bz9D8mjyxi6HdnDigwv22Gs6PYIxPKG7cAtPUMJKXw++F4iyGPgCS7oB/ ySiOS8IQFGWr+Cax/kEOnhxLlG7w4r3dNHAB0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=2p2fPjfsIsJ6E9/ORkfpw1BwG9ijb5KY/aOnRzBglX8=; b=drbUV7REZxZdUpDZGWfAKcXjI+IYPu7c9+D+RcX4uNLZk5qqZYbRfOsVBwi8qXcpLq zFlTCXB9/S1lGWJzRjN6uU955h8uTjeiZNkr4ImeQ7aUJR/ney/zntLxPzY87ZnVYPCG FBExtTFQVon/kJw2y8tsxEmtGxFObsrQADf4m/S0tTRhqm/xubZuxF/k+sZd0oQeF2ZY my4yB/J67r89lwQ5INuF1rBBB6WCS+UTDY6kW7aLGi4I2onKDv4U2o88UtqgHWiN+hnH qgRWNPvr2VsUhHe+GfwnttIH5dtBNLdU375TujZxNoQAGXMlBKA3LEBNDBH+SiOC4C/I qMpw== X-Gm-Message-State: AOAM533NK9GvQLXGl8x14vu2fWpadatGVnucItAIzGLsxeAuC+7D8o2J kCoDG9HLc4hiClwcthgoK0yLtQ== X-Google-Smtp-Source: ABdhPJzc/72l8fO9n7RvaaUC+1kyZHe2Tt9jHZADFhzTnhiYXdy4MIHTlfNLC8Izcw43sJcjBnCDXA== X-Received: by 2002:a63:e16:0:b0:3c6:12b1:ce15 with SMTP id d22-20020a630e16000000b003c612b1ce15mr19238632pgl.37.1652241328326; Tue, 10 May 2022 20:55:28 -0700 (PDT) Received: from localhost.localdomain (c-73-223-190-181.hsd1.ca.comcast.net. [73.223.190.181]) by smtp.gmail.com with ESMTPSA id d7-20020a170903230700b0015e8d4eb1f7sm442789plh.65.2022.05.10.20.55.26 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 10 May 2022 20:55:27 -0700 (PDT) From: Joe Damato To: netdev@vger.kernel.org, davem@davemloft.net, kuba@kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Cc: Joe Damato Subject: [RFC,net-next,x86 1/6] arch, x86, uaccess: Add nontemporal copy functions Date: Tue, 10 May 2022 20:54:22 -0700 Message-Id: <1652241268-46732-2-git-send-email-jdamato@fastly.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1652241268-46732-1-git-send-email-jdamato@fastly.com> References: <1652241268-46732-1-git-send-email-jdamato@fastly.com> Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Add a generic non-temporal wrapper to uaccess which can be overridden by arches that support non-temporal copies. An implementation is added for x86 which wraps an existing non-temporal copy in the kernel. Signed-off-by: Joe Damato --- arch/x86/include/asm/uaccess_64.h | 6 ++++++ include/linux/uaccess.h | 6 ++++++ 2 files changed, 12 insertions(+) diff --git a/arch/x86/include/asm/uaccess_64.h b/arch/x86/include/asm/uacce= ss_64.h index 45697e0..ed41dba 100644 --- a/arch/x86/include/asm/uaccess_64.h +++ b/arch/x86/include/asm/uaccess_64.h @@ -65,6 +65,12 @@ extern long __copy_user_flushcache(void *dst, const void= __user *src, unsigned s extern void memcpy_page_flushcache(char *to, struct page *page, size_t off= set, size_t len); =20 +static inline unsigned long +__copy_from_user_nocache(void *dst, const void __user *src, unsigned long = size) +{ + return (unsigned long)__copy_user_nocache(dst, src, (unsigned int) size, = 0); +} + static inline int __copy_from_user_inatomic_nocache(void *dst, const void __user *src, unsigned size) diff --git a/include/linux/uaccess.h b/include/linux/uaccess.h index 5461794..d1f57a1 100644 --- a/include/linux/uaccess.h +++ b/include/linux/uaccess.h @@ -234,6 +234,12 @@ static inline bool pagefault_disabled(void) #ifndef ARCH_HAS_NOCACHE_UACCESS =20 static inline __must_check unsigned long +__copy_from_user_nocache(void *to, const void __user *from, unsigned long = n) +{ + return __copy_from_user(to, from, n); +} + +static inline __must_check unsigned long __copy_from_user_inatomic_nocache(void *to, const void __user *from, unsigned long n) { --=20 2.7.4 From nobody Fri May 8 04:33:26 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1A77EC433F5 for ; Wed, 11 May 2022 03:55:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236779AbiEKDzq (ORCPT ); Tue, 10 May 2022 23:55:46 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35700 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238686AbiEKDzc (ORCPT ); Tue, 10 May 2022 23:55:32 -0400 Received: from mail-pl1-x630.google.com (mail-pl1-x630.google.com [IPv6:2607:f8b0:4864:20::630]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 103A72108BC for ; Tue, 10 May 2022 20:55:31 -0700 (PDT) Received: by mail-pl1-x630.google.com with SMTP id x18so671575plg.6 for ; Tue, 10 May 2022 20:55:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fastly.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=OGmy4QLI/8fGmA3FerkRtku5TKK3UOkAtDIIlXf/W7s=; b=Lvz2NFU0wxJq7i37Ze4DZNA+uM0xodpQ2/GD0sncaN3Rc26o53fkHgOXTmfUTOGYMe PQ8YhJRQE2joOk55gZTNRlr0yhTfNgbEswFespAwO54TVWi3uXnQh34OkhOecNRke75l vkqlDGGRVgUjE7Hj88oeaMjrx58TpDNV9ffOA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=OGmy4QLI/8fGmA3FerkRtku5TKK3UOkAtDIIlXf/W7s=; b=4kSNnlZGqG+7/nhTCm5kQ4sIW26Ke8wMkPdJMUgoLNHbLYXApWMOfmPLggVTQYOIFt fI4sHTbV6DC/HV2WmXJJluJh1T1uOISJBWRC1m3PJ+YK1O1vNDKl8G25Agy45vKFrbvQ DW7wP2K8smhu5GHiAFTkowSHzM9wXCcx9Lw6EQYCyXNvaJNpB8K6crYmVMV8pgPvg91O KQIpqA07rhFrifID6H3Ny1IjzyHTEFTjyhPzfbbiNoEtkip9+P/bnDEmicgs3iJMTkpR ZcaV5TodIYSnKKpwE9Ucdhvh0aEGKJgbwxu2GxngZ0BAap1HiIEKKhrTdMP0/zjQsBKM v9Ug== X-Gm-Message-State: AOAM531JZ3VYzgYYIQ9d/QWrAUh3GXTFuJoP7hvw7nITHDWrs0bDPvdj 17mPzQ5G/P+L0GoL+2cpS+sD2s7/c58upQ== X-Google-Smtp-Source: ABdhPJzw3DCldBbT9UirZQI+OHb9LG4rGmq8XkuDK/k9AREZfDPPdxjaEuFH1i4LKnAoxp/vAOJzyw== X-Received: by 2002:a17:90b:14ce:b0:1dc:eff5:52b6 with SMTP id jz14-20020a17090b14ce00b001dceff552b6mr3169500pjb.148.1652241330638; Tue, 10 May 2022 20:55:30 -0700 (PDT) Received: from localhost.localdomain (c-73-223-190-181.hsd1.ca.comcast.net. [73.223.190.181]) by smtp.gmail.com with ESMTPSA id d7-20020a170903230700b0015e8d4eb1f7sm442789plh.65.2022.05.10.20.55.28 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 10 May 2022 20:55:30 -0700 (PDT) From: Joe Damato To: netdev@vger.kernel.org, davem@davemloft.net, kuba@kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, Alexander Viro Cc: Joe Damato Subject: [RFC,net-next 2/6] iov_iter: Allow custom copyin function Date: Tue, 10 May 2022 20:54:23 -0700 Message-Id: <1652241268-46732-3-git-send-email-jdamato@fastly.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1652241268-46732-1-git-send-email-jdamato@fastly.com> References: <1652241268-46732-1-git-send-email-jdamato@fastly.com> Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" When calling copy_page_from_iter_iovec, allow callers to specify the copy function they'd like to use. The only caller is updated to pass raw_copy_from_user. Signed-off-by: Joe Damato --- lib/iov_iter.c | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/lib/iov_iter.c b/lib/iov_iter.c index 6dd5330..ef22ec1 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -253,7 +253,9 @@ static size_t copy_page_to_iter_iovec(struct page *page= , size_t offset, size_t b } =20 static size_t copy_page_from_iter_iovec(struct page *page, size_t offset, = size_t bytes, - struct iov_iter *i) + struct iov_iter *i, + unsigned long (*_copyin)(void *to, const void __user *from, + unsigned long n)) { size_t skip, copy, left, wanted; const struct iovec *iov; @@ -278,7 +280,7 @@ static size_t copy_page_from_iter_iovec(struct page *pa= ge, size_t offset, size_t to =3D kaddr + offset; =20 /* first chunk, usually the only one */ - left =3D copyin(to, buf, copy); + left =3D _copyin(to, buf, copy); copy -=3D left; skip +=3D copy; to +=3D copy; @@ -288,7 +290,7 @@ static size_t copy_page_from_iter_iovec(struct page *pa= ge, size_t offset, size_t iov++; buf =3D iov->iov_base; copy =3D min(bytes, iov->iov_len); - left =3D copyin(to, buf, copy); + left =3D _copyin(to, buf, copy); copy -=3D left; skip =3D copy; to +=3D copy; @@ -307,7 +309,7 @@ static size_t copy_page_from_iter_iovec(struct page *pa= ge, size_t offset, size_t =20 kaddr =3D kmap(page); to =3D kaddr + offset; - left =3D copyin(to, buf, copy); + left =3D _copyin(to, buf, copy); copy -=3D left; skip +=3D copy; to +=3D copy; @@ -316,7 +318,7 @@ static size_t copy_page_from_iter_iovec(struct page *pa= ge, size_t offset, size_t iov++; buf =3D iov->iov_base; copy =3D min(bytes, iov->iov_len); - left =3D copyin(to, buf, copy); + left =3D _copyin(to, buf, copy); copy -=3D left; skip =3D copy; to +=3D copy; @@ -899,7 +901,7 @@ size_t copy_page_from_iter(struct page *page, size_t of= fset, size_t bytes, if (unlikely(!page_copy_sane(page, offset, bytes))) return 0; if (likely(iter_is_iovec(i))) - return copy_page_from_iter_iovec(page, offset, bytes, i); + return copy_page_from_iter_iovec(page, offset, bytes, i, raw_copy_from_u= ser); if (iov_iter_is_bvec(i) || iov_iter_is_kvec(i) || iov_iter_is_xarray(i)) { void *kaddr =3D kmap_local_page(page); size_t wanted =3D _copy_from_iter(kaddr + offset, bytes, i); --=20 2.7.4 From nobody Fri May 8 04:33:26 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 665EAC433EF for ; Wed, 11 May 2022 03:56:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241896AbiEKDzx (ORCPT ); Tue, 10 May 2022 23:55:53 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35774 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239579AbiEKDzd (ORCPT ); Tue, 10 May 2022 23:55:33 -0400 Received: from mail-pg1-x52a.google.com (mail-pg1-x52a.google.com [IPv6:2607:f8b0:4864:20::52a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F34C3213324 for ; Tue, 10 May 2022 20:55:32 -0700 (PDT) Received: by mail-pg1-x52a.google.com with SMTP id 202so687400pgc.9 for ; Tue, 10 May 2022 20:55:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fastly.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=tkAZPhAkhghXKAcrYQ097N+yUjSoD43IH6MO8WVhkW0=; b=XTIXKAsQLSW9h7YC+Hl0jhrCRlhJoL7ZN43rl4HojAKB6QTZDyKFGdsQ3+/4q11Gf9 1yrehn7dms/+YB3orW/WWboVn+fzhaRuXuXvjIXG6v7vC15AgKmlBlD8gilh49cnbXYJ dkZW2WjQK5eGZGAmm374sgwDgQoibYPvNPMlg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=tkAZPhAkhghXKAcrYQ097N+yUjSoD43IH6MO8WVhkW0=; b=t1C/oga0umC6+5Hxn5nWw72jhpEGtS4H38+rS4JdDFD2oJO8sPaLDrdSKErAMfcjK2 4EWcyS3XbLtjJnwdR64sT6Kz7UE/Z+/rwJdZTBaQNqmbeVenA5d9dURtdFNM1pBZ2bOG f/b3laMYVADcbTSE+A/MlC+0RlPlBsN8atSa0/nhRasC7lMWh4N8KfCymX9Tid22WgIL 1kHA4RW4LDOYR2urqYEmxj94AgNhyClQrXW8uO6zvhTig2e2u18WEPxo3wYS9x+HpED+ QR77/ThJi/Zjzftvi9tpx7bs+7YmhLG2Wm/Dgl3BnfRD/Z5Zdcht1pDoTwJlnBgyHmwF HYsw== X-Gm-Message-State: AOAM531D25sDMSEsqySnQTAwFr3qANMt05LKKl2wJLDYAnpU8HOkUKi/ P3fwQVfIJ2w4amtpjSX45s5ZtA== X-Google-Smtp-Source: ABdhPJxYgsPGtWhs7POp5A5xnYfBle+QWXILF7HGaYr8jorK2248KV18yTDhWf6EaGfBzvQ3RnPdig== X-Received: by 2002:a05:6a00:10cc:b0:505:ada6:e03e with SMTP id d12-20020a056a0010cc00b00505ada6e03emr23348965pfu.45.1652241332539; Tue, 10 May 2022 20:55:32 -0700 (PDT) Received: from localhost.localdomain (c-73-223-190-181.hsd1.ca.comcast.net. [73.223.190.181]) by smtp.gmail.com with ESMTPSA id d7-20020a170903230700b0015e8d4eb1f7sm442789plh.65.2022.05.10.20.55.31 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 10 May 2022 20:55:32 -0700 (PDT) From: Joe Damato To: netdev@vger.kernel.org, davem@davemloft.net, kuba@kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, Alexander Viro Cc: Joe Damato Subject: [RFC,net-next 3/6] iov_iter: Add a nocache copy iov iterator Date: Tue, 10 May 2022 20:54:24 -0700 Message-Id: <1652241268-46732-4-git-send-email-jdamato@fastly.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1652241268-46732-1-git-send-email-jdamato@fastly.com> References: <1652241268-46732-1-git-send-email-jdamato@fastly.com> Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Add copy_page_from_iter_nocache, which wraps copy_page_from_iter_iovec and passes in a custom copyin function: __copy_from_user_nocache. This allows callers of copy_page_from_iter_nocache to copy data without disturbing the CPU cache. Signed-off-by: Joe Damato --- include/linux/uio.h | 2 ++ lib/iov_iter.c | 20 ++++++++++++++++++++ 2 files changed, 22 insertions(+) diff --git a/include/linux/uio.h b/include/linux/uio.h index 739285f..58c7946 100644 --- a/include/linux/uio.h +++ b/include/linux/uio.h @@ -142,6 +142,8 @@ size_t copy_page_to_iter(struct page *page, size_t offs= et, size_t bytes, struct iov_iter *i); size_t copy_page_from_iter(struct page *page, size_t offset, size_t bytes, struct iov_iter *i); +size_t copy_page_from_iter_nocache(struct page *page, size_t offset, size_= t bytes, + struct iov_iter *i); =20 size_t _copy_to_iter(const void *addr, size_t bytes, struct iov_iter *i); size_t _copy_from_iter(void *addr, size_t bytes, struct iov_iter *i); diff --git a/lib/iov_iter.c b/lib/iov_iter.c index ef22ec1..985bf58 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -895,6 +895,26 @@ size_t copy_page_to_iter(struct page *page, size_t off= set, size_t bytes, } EXPORT_SYMBOL(copy_page_to_iter); =20 +size_t copy_page_from_iter_nocache(struct page *page, size_t offset, size_t + bytes, struct iov_iter *i) +{ + if (unlikely(!page_copy_sane(page, offset, bytes))) + return 0; + if (unlikely(iov_iter_is_pipe(i) || iov_iter_is_discard(i))) { + WARN_ON(1); + return 0; + } + if (iov_iter_is_bvec(i) || iov_iter_is_kvec(i) || iov_iter_is_xarray(i)) { + void *kaddr =3D kmap_atomic(page); + size_t wanted =3D _copy_from_iter_nocache(kaddr + offset, bytes, i); + + kunmap_atomic(kaddr); + return wanted; + } else + return copy_page_from_iter_iovec(page, offset, bytes, i, + __copy_from_user_nocache); +} + size_t copy_page_from_iter(struct page *page, size_t offset, size_t bytes, struct iov_iter *i) { --=20 2.7.4 From nobody Fri May 8 04:33:26 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 69350C433EF for ; Wed, 11 May 2022 03:58:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242014AbiEKD4B (ORCPT ); Tue, 10 May 2022 23:56:01 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36190 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240223AbiEKDzi (ORCPT ); Tue, 10 May 2022 23:55:38 -0400 Received: from mail-pj1-x1032.google.com (mail-pj1-x1032.google.com [IPv6:2607:f8b0:4864:20::1032]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3ADF2217195 for ; Tue, 10 May 2022 20:55:34 -0700 (PDT) Received: by mail-pj1-x1032.google.com with SMTP id x88so1065385pjj.1 for ; Tue, 10 May 2022 20:55:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fastly.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=aZBrMhde8FpEVWkUnj31KTacbZRR8gMfK1YB4YXQ0jc=; b=h83XAA98G4q5ePsZrIsLvPfnCfnbTjI1gSU8AzJSpPDJntT477r7QuDZCMsCDJKWG9 TqetClC0MszlVLsO1ojgFUTsLAAGc1OgU2jb+FQKxhxP0S81l5KGJ6eWfk7b45R21mEG Eq2ntBM7BWiZhqRxdb4HIcuGcH1vbYaRo55F4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=aZBrMhde8FpEVWkUnj31KTacbZRR8gMfK1YB4YXQ0jc=; b=tsLirsClRY9/axLL15nXel1Fw8xvF2nx9X4kE7xF5yEpsi2gU4fJN+ZIxfQisHqXpW sqMCUsc3mOiC/Db8NLVj4d3RJ/Fwsq/rx3aHnF8XH8nfOuZY1Kdw0Rcku1VIPJILlzd0 SlQHN62hEA1gXkq9xDl91zHYlFmSy9Ywdchik+eq0TYMNXh8+4xHKi4ZhtFxs/lugbfS 66pL8Uy5YLkPn67QiMBSqGgvREVBha6w1H28yPqCZFkDGCJJWIpic9zgiAlDhib+DZpK eDJ742hlKG4O7l7i3QujsiLCIbSvPdM10JgsfaPnNMNeWo35ONaUtf9FRBZqVGAqz2aE KcFA== X-Gm-Message-State: AOAM530mBT8otpIIrStLgZVoT1WA5m17Aigpm6An8/GUn22eHE3RrNNf p+0TqGO3WJT+KOT+2SCSzZbBLKo4E+SJQg== X-Google-Smtp-Source: ABdhPJxhMetGgdMphFhnMhts3MbZSr4+18d+mQckIRBqUPMXsNdPRazRpLHEHvEAaWw9V+TQ0BG9Vg== X-Received: by 2002:a17:902:ecca:b0:15e:8971:4540 with SMTP id a10-20020a170902ecca00b0015e89714540mr23357838plh.43.1652241334487; Tue, 10 May 2022 20:55:34 -0700 (PDT) Received: from localhost.localdomain (c-73-223-190-181.hsd1.ca.comcast.net. [73.223.190.181]) by smtp.gmail.com with ESMTPSA id d7-20020a170903230700b0015e8d4eb1f7sm442789plh.65.2022.05.10.20.55.33 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 10 May 2022 20:55:33 -0700 (PDT) From: Joe Damato To: netdev@vger.kernel.org, davem@davemloft.net, kuba@kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, Eric Dumazet , Paolo Abeni Cc: Joe Damato Subject: [RFC,net-next 4/6] net: Add a struct for managing copy functions Date: Tue, 10 May 2022 20:54:25 -0700 Message-Id: <1652241268-46732-5-git-send-email-jdamato@fastly.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1652241268-46732-1-git-send-email-jdamato@fastly.com> References: <1652241268-46732-1-git-send-email-jdamato@fastly.com> Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Add struct skb_copier which encapsulates two functions for copying data, and provide a default copier, skb_copier. Separate skb_copy_datagram_from_iter into a a helper function, do_skb_copy_datagram, which takes a struct skb_copier. Signed-off-by: Joe Damato --- net/core/datagram.c | 49 ++++++++++++++++++++++++++++++++----------------- 1 file changed, 32 insertions(+), 17 deletions(-) diff --git a/net/core/datagram.c b/net/core/datagram.c index 50f4fae..a87c41b 100644 --- a/net/core/datagram.c +++ b/net/core/datagram.c @@ -532,18 +532,19 @@ int skb_copy_datagram_iter(const struct sk_buff *skb,= int offset, } EXPORT_SYMBOL(skb_copy_datagram_iter); =20 -/** - * skb_copy_datagram_from_iter - Copy a datagram from an iov_iter. - * @skb: buffer to copy - * @offset: offset in the buffer to start copying to - * @from: the copy source - * @len: amount of data to copy to buffer from iovec - * - * Returns 0 or -EFAULT. - */ -int skb_copy_datagram_from_iter(struct sk_buff *skb, int offset, - struct iov_iter *from, - int len) +struct skb_copier { + size_t (*copy_from_iter)(void *addr, size_t bytes, struct iov_iter *i); + size_t (*copy_page_from_iter)(struct page *page, size_t offset, size_t by= tes, + struct iov_iter *i); +}; + +struct skb_copier skb_copier =3D { + .copy_from_iter =3D copy_from_iter, + .copy_page_from_iter =3D copy_page_from_iter +}; + +static int do_skb_copy_datagram(struct sk_buff *skb, int offset, + struct iov_iter *from, int len, struct skb_copier copier) { int start =3D skb_headlen(skb); int i, copy =3D start - offset; @@ -553,7 +554,7 @@ int skb_copy_datagram_from_iter(struct sk_buff *skb, in= t offset, if (copy > 0) { if (copy > len) copy =3D len; - if (copy_from_iter(skb->data + offset, copy, from) !=3D copy) + if (copier.copy_from_iter(skb->data + offset, copy, from) !=3D copy) goto fault; if ((len -=3D copy) =3D=3D 0) return 0; @@ -573,7 +574,7 @@ int skb_copy_datagram_from_iter(struct sk_buff *skb, in= t offset, =20 if (copy > len) copy =3D len; - copied =3D copy_page_from_iter(skb_frag_page(frag), + copied =3D copier.copy_page_from_iter(skb_frag_page(frag), skb_frag_off(frag) + offset - start, copy, from); if (copied !=3D copy) @@ -595,9 +596,7 @@ int skb_copy_datagram_from_iter(struct sk_buff *skb, in= t offset, if ((copy =3D end - offset) > 0) { if (copy > len) copy =3D len; - if (skb_copy_datagram_from_iter(frag_iter, - offset - start, - from, copy)) + if (do_skb_copy_datagram(frag_iter, offset - start, from, copy, copier)) goto fault; if ((len -=3D copy) =3D=3D 0) return 0; @@ -611,6 +610,22 @@ int skb_copy_datagram_from_iter(struct sk_buff *skb, i= nt offset, fault: return -EFAULT; } + +/** + * skb_copy_datagram_from_iter - Copy a datagram from an iov_iter. + * @skb: buffer to copy + * @offset: offset in the buffer to start copying to + * @from: the copy source + * @len: amount of data to copy to buffer from iovec + * + * Returns 0 or -EFAULT. + */ +int skb_copy_datagram_from_iter(struct sk_buff *skb, int offset, + struct iov_iter *from, + int len) +{ + return do_skb_copy_datagram(skb, offset, from, len, skb_copier); +} EXPORT_SYMBOL(skb_copy_datagram_from_iter); =20 int __zerocopy_sg_from_iter(struct sock *sk, struct sk_buff *skb, --=20 2.7.4 From nobody Fri May 8 04:33:26 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7F05CC433FE for ; Wed, 11 May 2022 03:58:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242117AbiEKD4I (ORCPT ); Tue, 10 May 2022 23:56:08 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36282 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241209AbiEKDzj (ORCPT ); Tue, 10 May 2022 23:55:39 -0400 Received: from mail-pj1-x102d.google.com (mail-pj1-x102d.google.com [IPv6:2607:f8b0:4864:20::102d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 280D4218FF8 for ; Tue, 10 May 2022 20:55:37 -0700 (PDT) Received: by mail-pj1-x102d.google.com with SMTP id l11-20020a17090a49cb00b001d923a9ca99so938986pjm.1 for ; Tue, 10 May 2022 20:55:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fastly.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=HVRP118ZOlUZuoG/hDkho2/EllajCBtZWJSv42QBoCc=; b=GEO29e6cFLU3C6rQeRvcHHx1B7Gl6areArEPEnkD1l5qphc906h2sIwiQ9NEQT3nxy hnqzbFtx/vuGJLQlvdprTdtvWNTrHhwoY3C4BKHov0yQYeFQb7vB7W02Zx/MJLmHLzIm 0waUvGXOJQ5G3GNz0yafi51anAAeRtIfzso7A= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=HVRP118ZOlUZuoG/hDkho2/EllajCBtZWJSv42QBoCc=; b=ZEH+Rk1YOo6wMwQCVQFVDK7NBdPEiaNtcRBzF7Pa2T7MzRAeGk343qJ3KsQ1ia1INO wDNhXlHKJkEF1A/bf1wzJzIiT7KbDsHk+ecORCH9AsFdv6qUwrZ8IJj+gFa313hHV1gD 7RXa8C9Q685O1N0zG48YFFBzVD45V9c0BauOODN39sNuDJzme0Vx5jPi+LhMgvRpDAi0 2dpvnib/5xLan1dC0C8BPifiSdycZGtN5lvTbLgDXl2A2ScjRGUx/BzpLKKncXOfVKk9 a46omV6vanW6xyYbIuQer1bterR09jhxTGWTG+MSa1M2Bg5xc42JO7RlH1wWX20Kzuc9 j74g== X-Gm-Message-State: AOAM530bxCmzwtewQ0UUiOxAQUpSPYRnvz5SkAcHAsqo6HEnNMroUq3f z2VqOP+ayuKinOmi0vD7eGJDaQ== X-Google-Smtp-Source: ABdhPJx2cBOX8TKr/h/WTeqUC4FZm457izcVFqtGtuZ/tZr6Sm6aO2hN978l1aOE3sJBvl184EerOw== X-Received: by 2002:a17:902:8644:b0:15a:3b4a:538a with SMTP id y4-20020a170902864400b0015a3b4a538amr23511594plt.146.1652241336651; Tue, 10 May 2022 20:55:36 -0700 (PDT) Received: from localhost.localdomain (c-73-223-190-181.hsd1.ca.comcast.net. [73.223.190.181]) by smtp.gmail.com with ESMTPSA id d7-20020a170903230700b0015e8d4eb1f7sm442789plh.65.2022.05.10.20.55.35 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 10 May 2022 20:55:35 -0700 (PDT) From: Joe Damato To: netdev@vger.kernel.org, davem@davemloft.net, kuba@kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, Eric Dumazet , Paolo Abeni Cc: Joe Damato Subject: [RFC,net-next 5/6] net: Add a way to copy skbs without affect cache Date: Tue, 10 May 2022 20:54:26 -0700 Message-Id: <1652241268-46732-6-git-send-email-jdamato@fastly.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1652241268-46732-1-git-send-email-jdamato@fastly.com> References: <1652241268-46732-1-git-send-email-jdamato@fastly.com> Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Add an skb_copier, skb_nocache_copier, which contains function pointers to nontemporal copy routines. Using skb_nocache_copier and do_skb_copy_datagram implement skb_copy_datagram_from_iter_nocache. This function is intended to be used by callers which would like to copy data into SKBs using nontemporal instructions to avoid the CPU cache. Signed-off-by: Joe Damato --- include/linux/skbuff.h | 2 ++ net/core/datagram.c | 12 ++++++++++++ 2 files changed, 14 insertions(+) diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index 97de40b..32c0cba 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -3918,6 +3918,8 @@ int skb_copy_and_hash_datagram_iter(const struct sk_b= uff *skb, int offset, struct ahash_request *hash); int skb_copy_datagram_from_iter(struct sk_buff *skb, int offset, struct iov_iter *from, int len); +int skb_copy_datagram_from_iter_nocache(struct sk_buff *skb, int offset, + struct iov_iter *from, int len); int zerocopy_sg_from_iter(struct sk_buff *skb, struct iov_iter *frm); void skb_free_datagram(struct sock *sk, struct sk_buff *skb); void __skb_free_datagram_locked(struct sock *sk, struct sk_buff *skb, int = len); diff --git a/net/core/datagram.c b/net/core/datagram.c index a87c41b..da8557b 100644 --- a/net/core/datagram.c +++ b/net/core/datagram.c @@ -543,6 +543,11 @@ struct skb_copier skb_copier =3D { .copy_page_from_iter =3D copy_page_from_iter }; =20 +struct skb_copier skb_nocache_copier =3D { + .copy_from_iter =3D copy_from_iter_nocache, + .copy_page_from_iter =3D copy_page_from_iter_nocache +}; + static int do_skb_copy_datagram(struct sk_buff *skb, int offset, struct iov_iter *from, int len, struct skb_copier copier) { @@ -611,6 +616,13 @@ static int do_skb_copy_datagram(struct sk_buff *skb, i= nt offset, return -EFAULT; } =20 +int skb_copy_datagram_from_iter_nocache(struct sk_buff *skb, int offset, + struct iov_iter *from, int len) +{ + return do_skb_copy_datagram(skb, offset, from, len, skb_nocache_copier); +} +EXPORT_SYMBOL(skb_copy_datagram_from_iter_nocache); + /** * skb_copy_datagram_from_iter - Copy a datagram from an iov_iter. * @skb: buffer to copy --=20 2.7.4 From nobody Fri May 8 04:33:26 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0EA25C433F5 for ; Wed, 11 May 2022 03:58:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241961AbiEKDz7 (ORCPT ); Tue, 10 May 2022 23:55:59 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36626 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241882AbiEKDzp (ORCPT ); Tue, 10 May 2022 23:55:45 -0400 Received: from mail-pl1-x62e.google.com (mail-pl1-x62e.google.com [IPv6:2607:f8b0:4864:20::62e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 126CD4CD55 for ; Tue, 10 May 2022 20:55:39 -0700 (PDT) Received: by mail-pl1-x62e.google.com with SMTP id q4so653297plr.11 for ; Tue, 10 May 2022 20:55:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fastly.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=8Oj5fYgOyP9r6j4Oo2YFgfJ89J2j62xWeof0Jo9AY3w=; b=v9w+8OSo8NSVmDmBQbc8obXr4Wi1nhD7MIDl5mmowMNt72/kNMxqluqzW8jS3Kp7UB /7VISt6MIJKtVD9o0A7m6LzNO5Lqp9ow+x5B9kBGB7t7PtEZGsvXfcXksfmbNr4z51g2 LTzKHBx8TvH8VwSw5qQxFSvqLW2aYGDGcyTCY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=8Oj5fYgOyP9r6j4Oo2YFgfJ89J2j62xWeof0Jo9AY3w=; b=gKBNlz3W4ErVwNaV1LBEXKfjk7dCw+kncT24w8YgNAZMfksrnNqLEjZ6DORKeShwBx igHWdzB+GyTeyq3If4B9nthhjFGz8ia39GQvuynsCyFmOB9gp3L+uiMaSzYYUSlW8oNe H1Rqca0uRaQ2eWCKttcrrSujNy+gCXXgCUGHNDbTJLWfZAU5v82Y62LRfoDRJqhd4nFo 3vDgYJDJzsdOpgMYNeYUIMsu0MHBFmFAFhZViA+r7rxNqjYZnIvVCg532GpOiVWa2DxX nU5IgyQI5uhCqMmqqLNTJ/BP2zVE4v1ft5xeW4ZFm4wyw5SiwP8MbrrJNbakXW81IorF 1rSA== X-Gm-Message-State: AOAM531dBxzYBmWuyzNcoVP/2nYXsVki2W4LEXf29DifWLr2XuKxWIpJ VCYA9InzH8te8LaODbwvlJAJLA== X-Google-Smtp-Source: ABdhPJzDBQpUZk7W8Mq0NWHrteV9nx9UH/BVPj5fyK4gC7QCMNmi9ReuRK03KcV0ElJJJiZ8qW3FTA== X-Received: by 2002:a17:902:8644:b0:153:9f01:2090 with SMTP id y4-20020a170902864400b001539f012090mr22917397plt.101.1652241338601; Tue, 10 May 2022 20:55:38 -0700 (PDT) Received: from localhost.localdomain (c-73-223-190-181.hsd1.ca.comcast.net. [73.223.190.181]) by smtp.gmail.com with ESMTPSA id d7-20020a170903230700b0015e8d4eb1f7sm442789plh.65.2022.05.10.20.55.37 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 10 May 2022 20:55:38 -0700 (PDT) From: Joe Damato To: netdev@vger.kernel.org, davem@davemloft.net, kuba@kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, Eric Dumazet , Paolo Abeni Cc: Joe Damato Subject: [RFC,net-next 6/6] net: unix: Add MSG_NTCOPY Date: Tue, 10 May 2022 20:54:27 -0700 Message-Id: <1652241268-46732-7-git-send-email-jdamato@fastly.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1652241268-46732-1-git-send-email-jdamato@fastly.com> References: <1652241268-46732-1-git-send-email-jdamato@fastly.com> Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Add a new sendmsg flag, MSG_NTCOPY, which user programs can use to signal to the kernel that data copied into the kernel during sendmsg should be done so using nontemporal copies, if it is supported by the architecture. Signed-off-by: Joe Damato --- include/linux/socket.h | 1 + net/unix/af_unix.c | 13 +++++++++++-- 2 files changed, 12 insertions(+), 2 deletions(-) diff --git a/include/linux/socket.h b/include/linux/socket.h index 12085c9..c9b10aa 100644 --- a/include/linux/socket.h +++ b/include/linux/socket.h @@ -318,6 +318,7 @@ struct ucred { * plain text and require encryption */ =20 +#define MSG_NTCOPY 0x2000000 /* Use a non-temporal copy */ #define MSG_ZEROCOPY 0x4000000 /* Use user data in kernel path */ #define MSG_FASTOPEN 0x20000000 /* Send data in TCP SYN */ #define MSG_CMSG_CLOEXEC 0x40000000 /* Set close_on_exec for file diff --git a/net/unix/af_unix.c b/net/unix/af_unix.c index e1dd9e9..ccbd643 100644 --- a/net/unix/af_unix.c +++ b/net/unix/af_unix.c @@ -1907,7 +1907,11 @@ static int unix_dgram_sendmsg(struct socket *sock, s= truct msghdr *msg, skb_put(skb, len - data_len); skb->data_len =3D data_len; skb->len =3D len; - err =3D skb_copy_datagram_from_iter(skb, 0, &msg->msg_iter, len); + if (msg->msg_flags & MSG_NTCOPY) + err =3D skb_copy_datagram_from_iter_nocache(skb, 0, &msg->msg_iter, len); + else + err =3D skb_copy_datagram_from_iter(skb, 0, &msg->msg_iter, len); + if (err) goto out_free; =20 @@ -2167,7 +2171,12 @@ static int unix_stream_sendmsg(struct socket *sock, = struct msghdr *msg, skb_put(skb, size - data_len); skb->data_len =3D data_len; skb->len =3D size; - err =3D skb_copy_datagram_from_iter(skb, 0, &msg->msg_iter, size); + + if (msg->msg_flags & MSG_NTCOPY) + err =3D skb_copy_datagram_from_iter_nocache(skb, 0, &msg->msg_iter, siz= e); + else + err =3D skb_copy_datagram_from_iter(skb, 0, &msg->msg_iter, size); + if (err) { kfree_skb(skb); goto out_err; --=20 2.7.4