From nobody Fri Dec 19 14:21:05 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 579E1288C0A; Mon, 19 May 2025 17:51:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1747677117; cv=none; b=NHXzz3kqQDPb+If0oCVxRv07Ao75H+ngwgGABVYeAXKRGUGQkTmZykfaVFm58KFi9ElUTt1MbAro8ehNAfaWy3HeyRrVK5NanfZN7miltQHDfTJcaVYGT5Qf4hzFoOn6sBSA/NGK0qvBawyIyB+2nMO8Pr5lHu7XRkgcvA0kd0w= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1747677117; c=relaxed/simple; bh=X8dbjSHr2U6lQaQcWmgJZbE8GedZCYfvJMCrGbZTikY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=t71t3KSC57XHPFYnLkavQlpgy3VejZk+9Vg5pUXWTtRbWDL+HYItdskhfNEC+AoBvIIu43LvMNfB8aQuXUBkLFUqiImp/gKHX5O84uKZtG9XfUG4HdpvTBJ3pq3xs5gLAGQhwNrhanszxv1d5Il0s7c5auZEkOS+XpXCOHpA8mI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=mlY9EiFo; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="mlY9EiFo" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 91366C4CEF1; Mon, 19 May 2025 17:51:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1747677116; bh=X8dbjSHr2U6lQaQcWmgJZbE8GedZCYfvJMCrGbZTikY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=mlY9EiFojWR3LwToOtmFKpKqswni2sODtLWJbQOJSCZyOK3D+ckVrbbhOgVWs2kBC jVpl3rCL7kbqWbN28BQ2XXqsRU7IqUu4cAsoE2sOZfX5RL6xi/vudcPhnJa6MNKo9r U1VvGoA6aSxgLm5UbwrFmZBSjHsroSYw01xEjP76EVYxZIRBLF6BzDT1HzmskXtGUq GMCVwoYXrWKjOFGvTjVQUKtIVaAwLDLeM98s1sd7Cm+3lrVAajh7hQSnZo2SWEcyQa igBEz3Dvjktpl3HwTfoRnmTv+fZ/uDuSFGVIzEkK3+KNNL2F6DrrKLX5qCsxBL9YDU Xk/2zo2X+V+vw== From: Eric Biggers To: netdev@vger.kernel.org Cc: linux-nvme@lists.infradead.org, linux-sctp@vger.kernel.org, linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org, Daniel Borkmann , Marcelo Ricardo Leitner , Sagi Grimberg , Ard Biesheuvel Subject: [PATCH v2 01/10] net: introduce CONFIG_NET_CRC32C Date: Mon, 19 May 2025 10:50:03 -0700 Message-ID: <20250519175012.36581-2-ebiggers@kernel.org> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250519175012.36581-1-ebiggers@kernel.org> References: <20250519175012.36581-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Eric Biggers Add a hidden kconfig symbol NET_CRC32C that will group together the functions that calculate CRC32C checksums of packets, so that these don't have to be built into NET-enabled kernels that don't need them. Make skb_crc32c_csum_help() (which is called only when IP_SCTP is enabled) conditional on this symbol, and make IP_SCTP select it. Signed-off-by: Eric Biggers Reviewed-by: Hannes Reinecke --- net/Kconfig | 4 ++++ net/core/dev.c | 2 ++ net/sctp/Kconfig | 1 + 3 files changed, 7 insertions(+) diff --git a/net/Kconfig b/net/Kconfig index 202cc595e5e6f..5b71a52987d33 100644 --- a/net/Kconfig +++ b/net/Kconfig @@ -73,10 +73,14 @@ config NET_DEVMEM depends on PAGE_POOL =20 config NET_SHAPER bool =20 +config NET_CRC32C + bool + select CRC32 + menu "Networking options" =20 source "net/packet/Kconfig" source "net/unix/Kconfig" source "net/tls/Kconfig" diff --git a/net/core/dev.c b/net/core/dev.c index fccf2167b2352..d93bee5eb5d8c 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -3594,10 +3594,11 @@ int skb_checksum_help(struct sk_buff *skb) out: return ret; } EXPORT_SYMBOL(skb_checksum_help); =20 +#ifdef CONFIG_NET_CRC32C int skb_crc32c_csum_help(struct sk_buff *skb) { __le32 crc32c_csum; int ret =3D 0, offset, start; =20 @@ -3633,10 +3634,11 @@ int skb_crc32c_csum_help(struct sk_buff *skb) skb_reset_csum_not_inet(skb); out: return ret; } EXPORT_SYMBOL(skb_crc32c_csum_help); +#endif /* CONFIG_NET_CRC32C */ =20 __be16 skb_network_protocol(struct sk_buff *skb, int *depth) { __be16 type =3D skb->protocol; =20 diff --git a/net/sctp/Kconfig b/net/sctp/Kconfig index d18a72df3654e..3669ba3518563 100644 --- a/net/sctp/Kconfig +++ b/net/sctp/Kconfig @@ -9,10 +9,11 @@ menuconfig IP_SCTP depends on IPV6 || IPV6=3Dn select CRC32 select CRYPTO select CRYPTO_HMAC select CRYPTO_SHA1 + select NET_CRC32C select NET_UDP_TUNNEL help Stream Control Transmission Protocol =20 From RFC 2960 . base-commit: a8ae8a0e848e3506c95e45e7cb6e640502495f1a --=20 2.49.0 From nobody Fri Dec 19 14:21:05 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B5D1C288C8A; Mon, 19 May 2025 17:51:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1747677117; cv=none; b=Z2pMDMNX/DncoOI9XarX2+KiE+1B2vImjv1jJjlQ2No66b8sVYoYnUBg5PwpNkvcDqy4fSlNTq6A6Wu+MgQFX5E3CpAyX5U0DEq9p1zlGH8JOkyU8pal9HXBrIVY52f1qYBWkBNBKkIbHrUj+dWAQJ9aeTB6/wQhQknTL3gpsno= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1747677117; c=relaxed/simple; bh=HUzbwNVRvrtqASUz0MiDlh5yFYtIHGu8zQ8EoJJzhm4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=b4+7KPHuoU3YNSrlqt17vjHvCnYu3UMRrOwT/515q8kMNJ1C1hwt17mZBC5TGNclQbxOgSfaERpOjiyx8Ib25fuxLC/I4WasKPkXjykCPD5achBi+s6IFpSRekiA96zrYviPNlwIsz0oiZa0HsKqB90wyG9n2cEqoCrdVRlKrsg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=ETkvhMyz; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="ETkvhMyz" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 004ADC4CEE4; Mon, 19 May 2025 17:51:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1747677117; bh=HUzbwNVRvrtqASUz0MiDlh5yFYtIHGu8zQ8EoJJzhm4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ETkvhMyzboMhacPdNyumj17YtYzcsaWXVZG9idr9C+sX7gyBNK74FR5xT2D5k8UJA dc76liUjUB3qpNRyOzUzQozlSMqR2brpk8OP5yfRgyF9BVEln8Mj1CFjoCxaEjXoxw tcHFu6VZFi/i9e3vJMc0mY5ugdTvkBLSu4y5MabuCF2Ec5ykXaZNz+d+PhGveAkf2P /fDIfGf9HoVYBua+CouMubyvN1DPLzuEdEZfrH6x2C+Dei/IthncDLd0Ex2ewkDhLG 0oUWMvMhM1j/ICcVFRuVZFfHjCg9SuldGnLWbw65+OLnkiZSemBsyy3h8HdvNoSdb7 9b4LQcCl/arPA== From: Eric Biggers To: netdev@vger.kernel.org Cc: linux-nvme@lists.infradead.org, linux-sctp@vger.kernel.org, linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org, Daniel Borkmann , Marcelo Ricardo Leitner , Sagi Grimberg , Ard Biesheuvel Subject: [PATCH v2 02/10] net: add skb_crc32c() Date: Mon, 19 May 2025 10:50:04 -0700 Message-ID: <20250519175012.36581-3-ebiggers@kernel.org> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250519175012.36581-1-ebiggers@kernel.org> References: <20250519175012.36581-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Eric Biggers Add skb_crc32c(), which calculates the CRC32C of a sk_buff. It will replace __skb_checksum(), which unnecessarily supports arbitrary checksums. Compared to __skb_checksum(), skb_crc32c(): - Uses the correct type for CRC32C values (u32, not __wsum). - Does not require the caller to provide a skb_checksum_ops struct. - Is faster because it does not use indirect calls and does not use the very slow crc32c_combine(). According to commit 2817a336d4d5 ("net: skb_checksum: allow custom update/combine for walking skb") which added __skb_checksum(), the original motivation for the abstraction layer was to avoid code duplication for CRC32C and other checksums in the future. However: - No additional checksums showed up after CRC32C. __skb_checksum() is only used with the "regular" net checksum and CRC32C. - Indirect calls are expensive. Commit 2544af0344ba ("net: avoid indirect calls in L4 checksum calculation") worked around this using the INDIRECT_CALL_1 macro. But that only avoided the indirect call for the net checksum, and at the cost of an extra branch. - The checksums use different types (__wsum and u32), causing casts to be needed. - It made the checksums of fragments be combined (rather than chained) for both checksums, despite this being highly counterproductive for CRC32C due to how slow crc32c_combine() is. This can clearly be seen in commit 4c2f24549644 ("sctp: linearize early if it's not GSO") which tried to work around this performance bug. With a dedicated function for each checksum, we can instead just use the proper strategy for each checksum. As shown by the following tables, the new function skb_crc32c() is faster than __skb_checksum(), with the improvement varying greatly from 5% to 2500% depending on the case. The largest improvements come from fragmented packets, mainly due to eliminating the inefficient crc32c_combine(). But linear packets are improved too, especially shorter ones, mainly due to eliminating indirect calls. These benchmarks were done on AMD Zen 5. On that CPU, Linux uses IBRS instead of retpoline; an even greater improvement might be seen with retpoline: Linear sk_buffs Length in bytes __skb_checksum cycles skb_crc32c cycles =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D 64 43 18 256 94 77 1420 204 161 16384 1735 1642 Nonlinear sk_buffs (even split between head and one fragment) Length in bytes __skb_checksum cycles skb_crc32c cycles =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D 64 579 22 256 829 77 1420 1506 194 16384 4365 1682 Signed-off-by: Eric Biggers Reviewed-by: Hannes Reinecke --- include/linux/skbuff.h | 1 + net/core/skbuff.c | 73 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 74 insertions(+) diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index c7397b17bb08e..7ccc6356acaca 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -4201,10 +4201,11 @@ extern const struct skb_checksum_ops *crc32c_csum_s= tub __read_mostly; =20 __wsum __skb_checksum(const struct sk_buff *skb, int offset, int len, __wsum csum, const struct skb_checksum_ops *ops); __wsum skb_checksum(const struct sk_buff *skb, int offset, int len, __wsum csum); +u32 skb_crc32c(const struct sk_buff *skb, int offset, int len, u32 crc); =20 static inline void * __must_check __skb_header_pointer(const struct sk_buff *skb, int offset, int len, const void *data, int hlen, void *buffer) { diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 4159107f1666c..94b977db47f9d 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -62,10 +62,11 @@ #include #include #include #include #include +#include =20 #include #include #include #include @@ -3631,10 +3632,82 @@ __wsum skb_copy_and_csum_bits(const struct sk_buff = *skb, int offset, BUG_ON(len); return csum; } EXPORT_SYMBOL(skb_copy_and_csum_bits); =20 +#ifdef CONFIG_NET_CRC32C +u32 skb_crc32c(const struct sk_buff *skb, int offset, int len, u32 crc) +{ + int start =3D skb_headlen(skb); + int i, copy =3D start - offset; + struct sk_buff *frag_iter; + + if (copy > 0) { + copy =3D min(copy, len); + crc =3D crc32c(crc, skb->data + offset, copy); + len -=3D copy; + if (len =3D=3D 0) + return crc; + offset +=3D copy; + } + + if (WARN_ON_ONCE(!skb_frags_readable(skb))) + return 0; + + for (i =3D 0; i < skb_shinfo(skb)->nr_frags; i++) { + int end; + skb_frag_t *frag =3D &skb_shinfo(skb)->frags[i]; + + WARN_ON(start > offset + len); + + end =3D start + skb_frag_size(frag); + copy =3D end - offset; + if (copy > 0) { + u32 p_off, p_len, copied; + struct page *p; + u8 *vaddr; + + copy =3D min(copy, len); + skb_frag_foreach_page(frag, + skb_frag_off(frag) + offset - start, + copy, p, p_off, p_len, copied) { + vaddr =3D kmap_atomic(p); + crc =3D crc32c(crc, vaddr + p_off, p_len); + kunmap_atomic(vaddr); + } + len -=3D copy; + if (len =3D=3D 0) + return crc; + offset +=3D copy; + } + start =3D end; + } + + skb_walk_frags(skb, frag_iter) { + int end; + + WARN_ON(start > offset + len); + + end =3D start + frag_iter->len; + copy =3D end - offset; + if (copy > 0) { + copy =3D min(copy, len); + crc =3D skb_crc32c(frag_iter, offset - start, copy, crc); + len -=3D copy; + if (len =3D=3D 0) + return crc; + offset +=3D copy; + } + start =3D end; + } + BUG_ON(len); + + return crc; +} +EXPORT_SYMBOL(skb_crc32c); +#endif /* CONFIG_NET_CRC32C */ + __sum16 __skb_checksum_complete_head(struct sk_buff *skb, int len) { __sum16 sum; =20 sum =3D csum_fold(skb_checksum(skb, 0, len, skb->csum)); --=20 2.49.0 From nobody Fri Dec 19 14:21:05 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EBE55288CA9; Mon, 19 May 2025 17:51:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1747677118; cv=none; b=jmP/ptO+0Te+J8pyaJYsTQvVKKZ1bcYpyoEQXpGmdZzktmckoDbl0J/Y7npo4hxPnfGKzBf3caR9jHnwre+QxJkdV9MueOGCTCVJ4mBy+btUOgugze16kzeeCg+cs5hQb/OhgRjS1bPeGbxza5jBB48qKmWAaK8fbgfPM1e6Q9w= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1747677118; c=relaxed/simple; bh=CZeHbIh3l9QIcf9yTA+bjElzQVy8LoKCuirQSGJR4qw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=pep5Fn5zWh70qY71d8nmRkLaIHOQgJOlQKX97JuZnkYOsdfVFIdx9WNesqdfjU2aLdvLA3WpFb0+z/vxK7V6YIJ6l19e/YFi09FoDsd6KPNmRojCk8S/ztrMtMdlF9eS2oQ5umixYAb+lqVf65dQv/Fmh5DAsaQdY2WEdNR3jG0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=g+NY6IHX; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="g+NY6IHX" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 63DBFC4CEED; Mon, 19 May 2025 17:51:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1747677117; bh=CZeHbIh3l9QIcf9yTA+bjElzQVy8LoKCuirQSGJR4qw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=g+NY6IHX2iV5SsY3RZYOow0Nztm2bCj7yjW+42uNiNSFXbjFwzEOzIa1yHobNoIhb Jv+foKP87FEXKxWolLkrodJKpVSjv4B9H6I+GRogdRK2Z4nuyzhOckT+8nJ5ZDO6u6 bgTAfn5MWjHJOkFPnE22alDEqU8YZ7gFez4x/tPKtOifW0lsdtxFP/1nwDGl75C34T DU6mk33vfWRZVLao7AKYFosr0dpXFaWcwaOcY/RaNbU9mKu+Wae54MLODBQcPKkMR9 oX6rrbKA+6GmCjpGtO2UdFB+En5qzUkjpnNSabxBiQDMB3ByfHC5K7WlUMoOQKfHsJ uP2ZUx4JCFUoA== From: Eric Biggers To: netdev@vger.kernel.org Cc: linux-nvme@lists.infradead.org, linux-sctp@vger.kernel.org, linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org, Daniel Borkmann , Marcelo Ricardo Leitner , Sagi Grimberg , Ard Biesheuvel Subject: [PATCH v2 03/10] net: use skb_crc32c() in skb_crc32c_csum_help() Date: Mon, 19 May 2025 10:50:05 -0700 Message-ID: <20250519175012.36581-4-ebiggers@kernel.org> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250519175012.36581-1-ebiggers@kernel.org> References: <20250519175012.36581-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Eric Biggers Instead of calling __skb_checksum() with a skb_checksum_ops struct that does CRC32C, just call the new function skb_crc32c(). This is faster and simpler. Signed-off-by: Eric Biggers Reviewed-by: Hannes Reinecke --- net/core/dev.c | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/net/core/dev.c b/net/core/dev.c index d93bee5eb5d8c..430b3d3240d8f 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -3597,11 +3597,11 @@ int skb_checksum_help(struct sk_buff *skb) EXPORT_SYMBOL(skb_checksum_help); =20 #ifdef CONFIG_NET_CRC32C int skb_crc32c_csum_help(struct sk_buff *skb) { - __le32 crc32c_csum; + u32 crc; int ret =3D 0, offset, start; =20 if (skb->ip_summed !=3D CHECKSUM_PARTIAL) goto out; =20 @@ -3625,14 +3625,12 @@ int skb_crc32c_csum_help(struct sk_buff *skb) =20 ret =3D skb_ensure_writable(skb, offset + sizeof(__le32)); if (ret) goto out; =20 - crc32c_csum =3D cpu_to_le32(~__skb_checksum(skb, start, - skb->len - start, ~(__u32)0, - crc32c_csum_stub)); - *(__le32 *)(skb->data + offset) =3D crc32c_csum; + crc =3D ~skb_crc32c(skb, start, skb->len - start, ~0); + *(__le32 *)(skb->data + offset) =3D cpu_to_le32(crc); skb_reset_csum_not_inet(skb); out: return ret; } EXPORT_SYMBOL(skb_crc32c_csum_help); --=20 2.49.0 From nobody Fri Dec 19 14:21:05 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AC11C28937F; Mon, 19 May 2025 17:51:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1747677118; cv=none; b=AlRVVx8epQ6Nkz/SG0mIfyVSKNqlcWZAygAdvX0SvS74DK42it9mTHCv5msKZtnBpsBWN5PgMi/iiCU4EfzH+04j/tz7+C5f5BeBKEl4FEDarcBozWObZ3kEwEDoTq4g84TGtA2ZxieC8V1yY8noDbYF2GsSPrDJ2T7C7IQhyEI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1747677118; c=relaxed/simple; bh=qXgV9satb45eNRR3L104eZrX/IdR9NTQp49GiqE+3gE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=AA8BA3bCgbo/17a5+XGDobNMfDwbdl/DXYQJ0gsO7iQ48gLvEVMl6eAjU67U+gsA5P+ZtlsoRCu9py55ZGsAQU4L0DOM/5iuECOvmYhtDDc242tNsRTNkWH2KMA2BogGIEl69zI7BX2qYBtIsNWwEhUj32A3/jTajGO1ZzFUIPY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=CpqOLcbd; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="CpqOLcbd" Received: by smtp.kernel.org (Postfix) with ESMTPSA id C762EC4CEF4; Mon, 19 May 2025 17:51:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1747677118; bh=qXgV9satb45eNRR3L104eZrX/IdR9NTQp49GiqE+3gE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=CpqOLcbddjxKZISEVtDFk+/IF3H693CmEJVHkGUHSaHVKbdhuQx70BGwFU7IJhx6B W9wBUwc+orbMN2pXbfpEvvH1D9aMRr+yqdI1KR9/l3/Niak2jcHpiC4sIZpASg0tre +jl753yjFJx4dVmw+VsnvnsiE2lSDoeFw3P5kSSRIgzBPOSQMSZ6u6Ulh/b8xCDVwJ K8zXhQlSxcM+ycmCftLdAnVhnhQGJzjQY+90r6Iz1oIJqYBhb8SffJGFKCwGY2Bq3h NBUKYcVpi4rS2s2i+jW/wj5iz878bGlGCNufREw1AFZb7ALnlGC5fjgFOjvb526ZeC RsZdP9qnwDJBw== From: Eric Biggers To: netdev@vger.kernel.org Cc: linux-nvme@lists.infradead.org, linux-sctp@vger.kernel.org, linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org, Daniel Borkmann , Marcelo Ricardo Leitner , Sagi Grimberg , Ard Biesheuvel , Bernard Metzler Subject: [PATCH v2 04/10] RDMA/siw: use skb_crc32c() instead of __skb_checksum() Date: Mon, 19 May 2025 10:50:06 -0700 Message-ID: <20250519175012.36581-5-ebiggers@kernel.org> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250519175012.36581-1-ebiggers@kernel.org> References: <20250519175012.36581-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Eric Biggers Instead of calling __skb_checksum() with a skb_checksum_ops struct that does CRC32C, just call the new function skb_crc32c(). This is faster and simpler. Reviewed-by: Bernard Metzler Signed-off-by: Eric Biggers Reviewed-by: Hannes Reinecke --- drivers/infiniband/sw/siw/Kconfig | 1 + drivers/infiniband/sw/siw/siw.h | 22 +--------------------- 2 files changed, 2 insertions(+), 21 deletions(-) diff --git a/drivers/infiniband/sw/siw/Kconfig b/drivers/infiniband/sw/siw/= Kconfig index ae4a953e2a039..186f182b80e79 100644 --- a/drivers/infiniband/sw/siw/Kconfig +++ b/drivers/infiniband/sw/siw/Kconfig @@ -1,10 +1,11 @@ config RDMA_SIW tristate "Software RDMA over TCP/IP (iWARP) driver" depends on INET && INFINIBAND depends on INFINIBAND_VIRT_DMA select CRC32 + select NET_CRC32C help This driver implements the iWARP RDMA transport over the Linux TCP/IP network stack. It enables a system with a standard Ethernet adapter to interoperate with a iWARP adapter or with another system running the SIW driver. diff --git a/drivers/infiniband/sw/siw/siw.h b/drivers/infiniband/sw/siw/si= w.h index 385067e07faf1..d9e5a2e4c471a 100644 --- a/drivers/infiniband/sw/siw/siw.h +++ b/drivers/infiniband/sw/siw/siw.h @@ -691,33 +691,13 @@ static inline void siw_crc_oneshot(const void *data, = size_t len, u8 out[4]) siw_crc_init(&crc); siw_crc_update(&crc, data, len); return siw_crc_final(&crc, out); } =20 -static inline __wsum siw_csum_update(const void *buff, int len, __wsum sum) -{ - return (__force __wsum)crc32c((__force __u32)sum, buff, len); -} - -static inline __wsum siw_csum_combine(__wsum csum, __wsum csum2, int offse= t, - int len) -{ - return (__force __wsum)crc32c_combine((__force __u32)csum, - (__force __u32)csum2, len); -} - static inline void siw_crc_skb(struct siw_rx_stream *srx, unsigned int len) { - const struct skb_checksum_ops siw_cs_ops =3D { - .update =3D siw_csum_update, - .combine =3D siw_csum_combine, - }; - __wsum crc =3D (__force __wsum)srx->mpa_crc; - - crc =3D __skb_checksum(srx->skb, srx->skb_offset, len, crc, - &siw_cs_ops); - srx->mpa_crc =3D (__force u32)crc; + srx->mpa_crc =3D skb_crc32c(srx->skb, srx->skb_offset, len, srx->mpa_crc); } =20 #define siw_dbg(ibdev, fmt, ...) = \ ibdev_dbg(ibdev, "%s: " fmt, __func__, ##__VA_ARGS__) =20 --=20 2.49.0 From nobody Fri Dec 19 14:21:05 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F101B28981B; Mon, 19 May 2025 17:51:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1747677119; cv=none; b=jPaLrEhiHfnEQZokU3T24xatAy0fgF/5rjqbY06g4i1Ecym5bT3Ks95b4t6KjLjRzMfKsA2DwdqgHI9qm+jOpMMtw56s6b/yH0TkL87VZAMGdssgDs8UfVxli2XAwfOeurdchd+19a22giwkDUsUm444jqAbTz1EZVxb9aFzZao= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1747677119; c=relaxed/simple; bh=QEzyKZNR9hWHlDwB+VK93Xjz4V0tHxsun14G4o7oHVg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=N3nkBu0DR2cGJOoNTR4YFyM91gyzvgPYVva1sLBbj9bVX98SIw0tgaDsrKy6ECpfKuuMPpf1Y2vAo2XGtI54gnYAhI1e6Wt+XNEx4novkqFynuEwxxMe+NDfc1/0+F2tz23Gqe3dDRcz/AHHRyLEChDjh6iDlF9+hXAuBFWOPIM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=cxsxIwb4; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="cxsxIwb4" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 3D035C4CEF5; Mon, 19 May 2025 17:51:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1747677118; bh=QEzyKZNR9hWHlDwB+VK93Xjz4V0tHxsun14G4o7oHVg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=cxsxIwb4LAiva1ctYTyPPgd7NXR/uYyeTelWi32wxcysl7WfQhp5EizqLbpHLCUUN xciri2xmBTGIy2ZB41A+Dt0jJAV0GDeKgudaztvEmhfoni7vBQZjdawtU87+4nRegy L8tYKVgC4MLoP1ad5ft/YlCJ5xNqtcYwmj+PhMfHTm7jYTwfW0qfVLL/8HGrsOfAfj xjlCeoDtGe919W5IFcvTotuafAgszX1zBtf2yzlRHeHhqskjdUkvQQWvhFbx1QPSfB 3xVK+t3a0uy2pJTiBMSFClUR6kKtOD7Jl1Hy6czJBZ787QSdCB2I9bTK/ziCESuaOY oJfXzxczVvYBg== From: Eric Biggers To: netdev@vger.kernel.org Cc: linux-nvme@lists.infradead.org, linux-sctp@vger.kernel.org, linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org, Daniel Borkmann , Marcelo Ricardo Leitner , Sagi Grimberg , Ard Biesheuvel Subject: [PATCH v2 05/10] sctp: use skb_crc32c() instead of __skb_checksum() Date: Mon, 19 May 2025 10:50:07 -0700 Message-ID: <20250519175012.36581-6-ebiggers@kernel.org> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250519175012.36581-1-ebiggers@kernel.org> References: <20250519175012.36581-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Eric Biggers Make sctp_compute_cksum() just use the new function skb_crc32c(), instead of calling __skb_checksum() with a skb_checksum_ops struct that does CRC32C. This is faster and simpler. Signed-off-by: Eric Biggers Reviewed-by: Hannes Reinecke --- include/net/sctp/checksum.h | 29 +++-------------------------- net/netfilter/Kconfig | 4 ++-- net/netfilter/ipvs/Kconfig | 2 +- net/openvswitch/Kconfig | 2 +- net/sched/Kconfig | 2 +- net/sctp/Kconfig | 1 - net/sctp/offload.c | 1 - 7 files changed, 8 insertions(+), 33 deletions(-) diff --git a/include/net/sctp/checksum.h b/include/net/sctp/checksum.h index 291465c258102..654d37ec04029 100644 --- a/include/net/sctp/checksum.h +++ b/include/net/sctp/checksum.h @@ -13,51 +13,28 @@ * * Written or modified by: * Dinakaran Joseph * Jon Grimm * Sridhar Samudrala - * - * Rewritten to use libcrc32c by: * Vlad Yasevich */ =20 #ifndef __sctp_checksum_h__ #define __sctp_checksum_h__ =20 #include #include -#include -#include - -static inline __wsum sctp_csum_update(const void *buff, int len, __wsum su= m) -{ - return (__force __wsum)crc32c((__force __u32)sum, buff, len); -} - -static inline __wsum sctp_csum_combine(__wsum csum, __wsum csum2, - int offset, int len) -{ - return (__force __wsum)crc32c_combine((__force __u32)csum, - (__force __u32)csum2, len); -} - -static const struct skb_checksum_ops sctp_csum_ops =3D { - .update =3D sctp_csum_update, - .combine =3D sctp_csum_combine, -}; =20 static inline __le32 sctp_compute_cksum(const struct sk_buff *skb, unsigned int offset) { struct sctphdr *sh =3D (struct sctphdr *)(skb->data + offset); __le32 old =3D sh->checksum; - __wsum new; + u32 new; =20 sh->checksum =3D 0; - new =3D ~__skb_checksum(skb, offset, skb->len - offset, ~(__wsum)0, - &sctp_csum_ops); + new =3D ~skb_crc32c(skb, offset, skb->len - offset, ~0); sh->checksum =3D old; - - return cpu_to_le32((__force __u32)new); + return cpu_to_le32(new); } =20 #endif /* __sctp_checksum_h__ */ diff --git a/net/netfilter/Kconfig b/net/netfilter/Kconfig index 3b2183fc7e563..2560416218d07 100644 --- a/net/netfilter/Kconfig +++ b/net/netfilter/Kconfig @@ -210,11 +210,11 @@ config NF_CT_PROTO_GRE =20 config NF_CT_PROTO_SCTP bool 'SCTP protocol connection tracking support' depends on NETFILTER_ADVANCED default y - select CRC32 + select NET_CRC32C help With this option enabled, the layer 3 independent connection tracking code will be able to do state tracking on SCTP connections. =20 If unsure, say Y. @@ -473,11 +473,11 @@ config NETFILTER_SYNPROXY =20 endif # NF_CONNTRACK =20 config NF_TABLES select NETFILTER_NETLINK - select CRC32 + select NET_CRC32C tristate "Netfilter nf_tables support" help nftables is the new packet classification framework that intends to replace the existing {ip,ip6,arp,eb}_tables infrastructure. It provides a pseudo-state machine with an extensible instruction-set diff --git a/net/netfilter/ipvs/Kconfig b/net/netfilter/ipvs/Kconfig index 8c5b1fe12d078..c203252e856d8 100644 --- a/net/netfilter/ipvs/Kconfig +++ b/net/netfilter/ipvs/Kconfig @@ -103,11 +103,11 @@ config IP_VS_PROTO_AH This option enables support for load balancing AH (Authentication Header) transport protocol. Say Y if unsure. =20 config IP_VS_PROTO_SCTP bool "SCTP load balancing support" - select CRC32 + select NET_CRC32C help This option enables support for load balancing SCTP transport protocol. Say Y if unsure. =20 comment "IPVS scheduler" diff --git a/net/openvswitch/Kconfig b/net/openvswitch/Kconfig index 5481bd561eb41..e6aaee92dba48 100644 --- a/net/openvswitch/Kconfig +++ b/net/openvswitch/Kconfig @@ -9,12 +9,12 @@ config OPENVSWITCH depends on !NF_CONNTRACK || \ (NF_CONNTRACK && ((!NF_DEFRAG_IPV6 || NF_DEFRAG_IPV6) && \ (!NF_NAT || NF_NAT) && \ (!NETFILTER_CONNCOUNT || NETFILTER_CONNCOUNT))) depends on PSAMPLE || !PSAMPLE - select CRC32 select MPLS + select NET_CRC32C select NET_MPLS_GSO select DST_CACHE select NET_NSH select NF_CONNTRACK_OVS if NF_CONNTRACK select NF_NAT_OVS if NF_NAT diff --git a/net/sched/Kconfig b/net/sched/Kconfig index 9f0b3f943fca8..ad914d2b2e221 100644 --- a/net/sched/Kconfig +++ b/net/sched/Kconfig @@ -794,11 +794,11 @@ config NET_ACT_SKBEDIT module will be called act_skbedit. =20 config NET_ACT_CSUM tristate "Checksum Updating" depends on NET_CLS_ACT && INET - select CRC32 + select NET_CRC32C help Say Y here to update some common checksum after some direct packet alterations. =20 To compile this code as a module, choose M here: the diff --git a/net/sctp/Kconfig b/net/sctp/Kconfig index 3669ba3518563..24d5a35ce894a 100644 --- a/net/sctp/Kconfig +++ b/net/sctp/Kconfig @@ -5,11 +5,10 @@ =20 menuconfig IP_SCTP tristate "The SCTP Protocol" depends on INET depends on IPV6 || IPV6=3Dn - select CRC32 select CRYPTO select CRYPTO_HMAC select CRYPTO_SHA1 select NET_CRC32C select NET_UDP_TUNNEL diff --git a/net/sctp/offload.c b/net/sctp/offload.c index 502095173d885..e6f863c031b4c 100644 --- a/net/sctp/offload.c +++ b/net/sctp/offload.c @@ -109,11 +109,10 @@ int __init sctp_offload_init(void) =20 ret =3D inet6_add_offload(&sctp6_offload, IPPROTO_SCTP); if (ret) goto ipv4; =20 - crc32c_csum_stub =3D &sctp_csum_ops; return ret; =20 ipv4: inet_del_offload(&sctp_offload, IPPROTO_SCTP); out: --=20 2.49.0 From nobody Fri Dec 19 14:21:05 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0DF8A289826; Mon, 19 May 2025 17:51:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1747677119; cv=none; b=fR1LQ3uGBiY1fJmMVBqywbVk2UssKEbYug0UpCSIj9eDmdyQCdaU/VyMNkqHzaAfcUNCFX5d9fe2aAy7xdje6rYY3k69fmyeJyyQr64Sy/LqeSkdK5ZplMl+1jtpuGBuMYPxY0DrpZOTjRQs73t4r+W7FLVHM1//vFSnqcqWz5I= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1747677119; c=relaxed/simple; bh=9W6A/eAI/6T9jjyybm1nmFEiGSJ8YHRariNqFak7k5Y=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=rFvJPgSq9m0BejEEATnnhvU9L1u/uR3UdXrRW7DZNbcn3fSQ2Td/S4ROO3918z9CiolD7gz6cQeSYOgSNzW6SFRmdHRodleWDnYl7+b7Xd1v9kGM/dDmJVwofsmqeMBE/OLUIMNHMDo9UVMjNOLwy20/rttGLkNneTSbUVn31Bo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=fqDgCFHe; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="fqDgCFHe" Received: by smtp.kernel.org (Postfix) with ESMTPSA id A12ECC4CEEB; Mon, 19 May 2025 17:51:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1747677118; bh=9W6A/eAI/6T9jjyybm1nmFEiGSJ8YHRariNqFak7k5Y=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=fqDgCFHe8B1mn5iXFFjbvnu0VTeibtAMAGGfbkUHmpP7Q7Z1c3PP00hxMn5pp4WSg 3IMqnhfcPeQsfYb/NqBJgwYBrnNc9lz3LSFHVUzQvQ1lHjgqJC9X/ByN9wYtC4sjnx SCuMMmkVH36210UXmI7/b9ZmiudEil9YC9ev0e+Rb7ZuZkNkd1F1YGYVROgxSnrlNw 3/sO4HKDQtLo28wVhHc4quW//i/ZhOToGftyUMmtdZ+KnaX310Qv5Lrw9C2mUimJbj h89YDlxguBIjMBjbWRQmFNXj8nqd9cFWKCna2NSieZe1VW9b1ZiAAa8i3+1qQmQZsk lPivyl4aEPijQ== From: Eric Biggers To: netdev@vger.kernel.org Cc: linux-nvme@lists.infradead.org, linux-sctp@vger.kernel.org, linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org, Daniel Borkmann , Marcelo Ricardo Leitner , Sagi Grimberg , Ard Biesheuvel Subject: [PATCH v2 06/10] net: fold __skb_checksum() into skb_checksum() Date: Mon, 19 May 2025 10:50:08 -0700 Message-ID: <20250519175012.36581-7-ebiggers@kernel.org> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250519175012.36581-1-ebiggers@kernel.org> References: <20250519175012.36581-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Eric Biggers Now that the only remaining caller of __skb_checksum() is skb_checksum(), fold __skb_checksum() into skb_checksum(). This makes struct skb_checksum_ops unnecessary, so remove that too and simply do the "regular" net checksum. It also makes the wrapper functions csum_partial_ext() and csum_block_add_ext() unnecessary, so remove those too and just use the underlying functions. Signed-off-by: Eric Biggers Reviewed-by: Hannes Reinecke --- include/linux/skbuff.h | 9 ------- include/net/checksum.h | 12 --------- net/core/skbuff.c | 59 +++++------------------------------------- 3 files changed, 7 insertions(+), 73 deletions(-) diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index 7ccc6356acaca..018c072305133 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -4190,19 +4190,10 @@ static inline int memcpy_from_msg(void *data, struc= t msghdr *msg, int len) static inline int memcpy_to_msg(struct msghdr *msg, void *data, int len) { return copy_to_iter(data, len, &msg->msg_iter) =3D=3D len ? 0 : -EFAULT; } =20 -struct skb_checksum_ops { - __wsum (*update)(const void *mem, int len, __wsum wsum); - __wsum (*combine)(__wsum csum, __wsum csum2, int offset, int len); -}; - -extern const struct skb_checksum_ops *crc32c_csum_stub __read_mostly; - -__wsum __skb_checksum(const struct sk_buff *skb, int offset, int len, - __wsum csum, const struct skb_checksum_ops *ops); __wsum skb_checksum(const struct sk_buff *skb, int offset, int len, __wsum csum); u32 skb_crc32c(const struct sk_buff *skb, int offset, int len, u32 crc); =20 static inline void * __must_check diff --git a/include/net/checksum.h b/include/net/checksum.h index 243f972267b8d..e57986b173f8e 100644 --- a/include/net/checksum.h +++ b/include/net/checksum.h @@ -96,16 +96,10 @@ static __always_inline __wsum csum_block_add(__wsum csum, __wsum csum2, int offset) { return csum_add(csum, csum_shift(csum2, offset)); } =20 -static __always_inline __wsum -csum_block_add_ext(__wsum csum, __wsum csum2, int offset, int len) -{ - return csum_block_add(csum, csum2, offset); -} - static __always_inline __wsum csum_block_sub(__wsum csum, __wsum csum2, int offset) { return csum_block_add(csum, ~csum2, offset); } @@ -113,16 +107,10 @@ csum_block_sub(__wsum csum, __wsum csum2, int offset) static __always_inline __wsum csum_unfold(__sum16 n) { return (__force __wsum)n; } =20 -static __always_inline -__wsum csum_partial_ext(const void *buff, int len, __wsum sum) -{ - return csum_partial(buff, len, sum); -} - #define CSUM_MANGLED_0 ((__force __sum16)0xffff) =20 static __always_inline void csum_replace_by_diff(__sum16 *sum, __wsum diff) { *sum =3D csum_fold(csum_add(diff, ~csum_unfold(*sum))); diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 94b977db47f9d..85fc82f72d268 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -3443,24 +3443,22 @@ int skb_store_bits(struct sk_buff *skb, int offset,= const void *from, int len) return -EFAULT; } EXPORT_SYMBOL(skb_store_bits); =20 /* Checksum skb data. */ -__wsum __skb_checksum(const struct sk_buff *skb, int offset, int len, - __wsum csum, const struct skb_checksum_ops *ops) +__wsum skb_checksum(const struct sk_buff *skb, int offset, int len, __wsum= csum) { int start =3D skb_headlen(skb); int i, copy =3D start - offset; struct sk_buff *frag_iter; int pos =3D 0; =20 /* Checksum header. */ if (copy > 0) { if (copy > len) copy =3D len; - csum =3D INDIRECT_CALL_1(ops->update, csum_partial_ext, - skb->data + offset, copy, csum); + csum =3D csum_partial(skb->data + offset, copy, csum); if ((len -=3D copy) =3D=3D 0) return csum; offset +=3D copy; pos =3D copy; } @@ -3486,17 +3484,13 @@ __wsum __skb_checksum(const struct sk_buff *skb, in= t offset, int len, =20 skb_frag_foreach_page(frag, skb_frag_off(frag) + offset - start, copy, p, p_off, p_len, copied) { vaddr =3D kmap_atomic(p); - csum2 =3D INDIRECT_CALL_1(ops->update, - csum_partial_ext, - vaddr + p_off, p_len, 0); + csum2 =3D csum_partial(vaddr + p_off, p_len, 0); kunmap_atomic(vaddr); - csum =3D INDIRECT_CALL_1(ops->combine, - csum_block_add_ext, csum, - csum2, pos, p_len); + csum =3D csum_block_add(csum, csum2, pos); pos +=3D p_len; } =20 if (!(len -=3D copy)) return csum; @@ -3513,14 +3507,13 @@ __wsum __skb_checksum(const struct sk_buff *skb, in= t offset, int len, end =3D start + frag_iter->len; if ((copy =3D end - offset) > 0) { __wsum csum2; if (copy > len) copy =3D len; - csum2 =3D __skb_checksum(frag_iter, offset - start, - copy, 0, ops); - csum =3D INDIRECT_CALL_1(ops->combine, csum_block_add_ext, - csum, csum2, pos, copy); + csum2 =3D skb_checksum(frag_iter, offset - start, copy, + 0); + csum =3D csum_block_add(csum, csum2, pos); if ((len -=3D copy) =3D=3D 0) return csum; offset +=3D copy; pos +=3D copy; } @@ -3528,22 +3521,10 @@ __wsum __skb_checksum(const struct sk_buff *skb, in= t offset, int len, } BUG_ON(len); =20 return csum; } -EXPORT_SYMBOL(__skb_checksum); - -__wsum skb_checksum(const struct sk_buff *skb, int offset, - int len, __wsum csum) -{ - const struct skb_checksum_ops ops =3D { - .update =3D csum_partial_ext, - .combine =3D csum_block_add_ext, - }; - - return __skb_checksum(skb, offset, len, csum, &ops); -} EXPORT_SYMBOL(skb_checksum); =20 /* Both of above in one bottle. */ =20 __wsum skb_copy_and_csum_bits(const struct sk_buff *skb, int offset, @@ -3763,36 +3744,10 @@ __sum16 __skb_checksum_complete(struct sk_buff *skb) =20 return sum; } EXPORT_SYMBOL(__skb_checksum_complete); =20 -static __wsum warn_crc32c_csum_update(const void *buff, int len, __wsum su= m) -{ - net_warn_ratelimited( - "%s: attempt to compute crc32c without libcrc32c.ko\n", - __func__); - return 0; -} - -static __wsum warn_crc32c_csum_combine(__wsum csum, __wsum csum2, - int offset, int len) -{ - net_warn_ratelimited( - "%s: attempt to compute crc32c without libcrc32c.ko\n", - __func__); - return 0; -} - -static const struct skb_checksum_ops default_crc32c_ops =3D { - .update =3D warn_crc32c_csum_update, - .combine =3D warn_crc32c_csum_combine, -}; - -const struct skb_checksum_ops *crc32c_csum_stub __read_mostly =3D - &default_crc32c_ops; -EXPORT_SYMBOL(crc32c_csum_stub); - /** * skb_zerocopy_headlen - Calculate headroom needed for skb_zerocopy() * @from: source buffer * * Calculates the amount of linear headroom needed in the 'to' skb passed --=20 2.49.0 From nobody Fri Dec 19 14:21:05 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 74110289E29; Mon, 19 May 2025 17:51:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1747677119; cv=none; b=D0ThWG52DktPbIR/wno7HwdPWcCk2+qf8YpqXgxA1wf5Zz4bWot3qHnMV6zpwnF8anMxpoe1kQaCzHyBHHwcsSZbX5A40jQPMK6reBHzY0bI9S41QxL4Y/gOODAXOhNkJfbk5fh5INizafCEjieONRX0THyTpliUE7fKJmNlMWo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1747677119; c=relaxed/simple; bh=Oo53xf+vPT0XtWshqsuhuWmUJl5Op5w9Tt26Wx4wMsw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=SIZkj2tBbth8SWRAgZ/GEyvJZ8fbyxdD8a4y5ALeK9MffmQLlVuH83EKYLcGj9kUvZsEM3PyuatDmFXvTBUljzHnjR6GQl/1+rrt3lYDa9pXtQrCHxNAS/JGBYbfdEQOxaC1Rb7OIpcJ/OghCDPWy02RZNoHwyFnc8mTaT+1TWA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=S0fI/+ym; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="S0fI/+ym" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1503EC4CEED; Mon, 19 May 2025 17:51:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1747677119; bh=Oo53xf+vPT0XtWshqsuhuWmUJl5Op5w9Tt26Wx4wMsw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=S0fI/+ymiRyCNifQz3w6CL5IrooEnfvl2XmRIp3GzU7Uz8tECZaFU1C7/yjE2x3Rx mASFqRafI7kBpTNDdValHZy34XNWOpD0q3sotIZDeRlQDTNXCCeV4eqW8z0W6q0z+F SDZwVRBuBKNvMa+bbsxIGADFCF44JFhMEncbJGaHI0xKPFWTyKh1x89MqZNqMipAcp Qpe3UCxUO1Y1C2PnUYb1CjDycIIx8k1f80NKyxJEfIcwyv/7qEZYJsR4DndwJVxDd7 bP+Kl7+W4R3CAIxaQB0LZpz7AMzodl6esgFeBDGYtcrhKTdggH3HQN2XJAHhyUJ9YA yCj6Pu/KIIdYQ== From: Eric Biggers To: netdev@vger.kernel.org Cc: linux-nvme@lists.infradead.org, linux-sctp@vger.kernel.org, linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org, Daniel Borkmann , Marcelo Ricardo Leitner , Sagi Grimberg , Ard Biesheuvel Subject: [PATCH v2 07/10] lib/crc32: remove unused support for CRC32C combination Date: Mon, 19 May 2025 10:50:09 -0700 Message-ID: <20250519175012.36581-8-ebiggers@kernel.org> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250519175012.36581-1-ebiggers@kernel.org> References: <20250519175012.36581-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Eric Biggers crc32c_combine() and crc32c_shift() are no longer used (except by the KUnit test that tests them), and their current implementation is very slow. Remove them. Signed-off-by: Eric Biggers Reviewed-by: Hannes Reinecke --- include/linux/crc32.h | 23 ----------------------- lib/crc32.c | 6 ------ lib/tests/crc_kunit.c | 6 ------ 3 files changed, 35 deletions(-) diff --git a/include/linux/crc32.h b/include/linux/crc32.h index 69c2e8bb37829..7f7d0be8a0acc 100644 --- a/include/linux/crc32.h +++ b/include/linux/crc32.h @@ -74,33 +74,10 @@ u32 crc32_le_shift(u32 crc, size_t len); static inline u32 crc32_le_combine(u32 crc1, u32 crc2, size_t len2) { return crc32_le_shift(crc1, len2) ^ crc2; } =20 -u32 crc32c_shift(u32 crc, size_t len); - -/** - * crc32c_combine - Combine two crc32c check values into one. For two sequ= ences - * of bytes, seq1 and seq2 with lengths len1 and len2, crc32c() - * check values were calculated for each, crc1 and crc2. - * - * @crc1: crc32c of the first block - * @crc2: crc32c of the second block - * @len2: length of the second block - * - * Return: The crc32c() check value of seq1 and seq2 concatenated, requiri= ng - * only crc1, crc2, and len2. Note: If seq_full denotes the concatenated - * memory area of seq1 with seq2, and crc_full the crc32c() value of - * seq_full, then crc_full =3D=3D crc32c_combine(crc1, crc2, len2) when - * crc_full was seeded with the same initializer as crc1, and crc2 seed - * was 0. See also crc_combine_test(). - */ -static inline u32 crc32c_combine(u32 crc1, u32 crc2, size_t len2) -{ - return crc32c_shift(crc1, len2) ^ crc2; -} - #define crc32(seed, data, length) crc32_le(seed, (unsigned char const *)(= data), length) =20 /* * Helpers for hash table generation of ethernet nics: * diff --git a/lib/crc32.c b/lib/crc32.c index fddd424ff2245..ade48bbb00834 100644 --- a/lib/crc32.c +++ b/lib/crc32.c @@ -117,16 +117,10 @@ u32 crc32_le_shift(u32 crc, size_t len) { return crc32_generic_shift(crc, len, CRC32_POLY_LE); } EXPORT_SYMBOL(crc32_le_shift); =20 -u32 crc32c_shift(u32 crc, size_t len) -{ - return crc32_generic_shift(crc, len, CRC32C_POLY_LE); -} -EXPORT_SYMBOL(crc32c_shift); - u32 crc32_be_base(u32 crc, const u8 *p, size_t len) { while (len--) crc =3D (crc << 8) ^ crc32table_be[(crc >> 24) ^ *p++]; return crc; diff --git a/lib/tests/crc_kunit.c b/lib/tests/crc_kunit.c index 585c48b65cefd..064c2d5815579 100644 --- a/lib/tests/crc_kunit.c +++ b/lib/tests/crc_kunit.c @@ -389,21 +389,15 @@ static void crc32_be_benchmark(struct kunit *test) static u64 crc32c_wrapper(u64 crc, const u8 *p, size_t len) { return crc32c(crc, p, len); } =20 -static u64 crc32c_combine_wrapper(u64 crc1, u64 crc2, size_t len2) -{ - return crc32c_combine(crc1, crc2, len2); -} - static const struct crc_variant crc_variant_crc32c =3D { .bits =3D 32, .le =3D true, .poly =3D 0x82f63b78, .func =3D crc32c_wrapper, - .combine_func =3D crc32c_combine_wrapper, }; =20 static void crc32c_test(struct kunit *test) { crc_test(test, &crc_variant_crc32c); --=20 2.49.0 From nobody Fri Dec 19 14:21:05 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 488CF28A404; Mon, 19 May 2025 17:51:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1747677120; cv=none; b=Rs3ad0ZcM8g75UazYj8VAOdK88InXgVgK61s2O1Fw4MHZ8WVTFRm4HZMZ/iCWm/EO+vHQ4HlCgpvymJpfOdXb6qwHduMRPuDrsH3yeTxCJplTCgssymvEarA9Urn6LFdqmk2jP3tLu/sSFgIuuMtACU7vfGQTKSUkNtPvuqUqDA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1747677120; c=relaxed/simple; bh=kKAbK/ihktd10bayLsP//PWEOGB9vYQ3zKxqoKSXAE8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=UZEww7fP0IA7vGdjOFJyxIPnBNh9yU7BY2+V0Fe6lgbnj67BbsT7wTFopiCrFh/YFsnuGtjzCwdAk47AxZsrdOGpkNXMdgo4fOaBR7JxI5Om5OER0HjSTkVTmYtagRopXd/pICJMKY7LP50LpuRmjY0QADCdB3IEsN6L0wAyyjM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Oq+0yzHg; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Oq+0yzHg" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 78D79C4CEF4; Mon, 19 May 2025 17:51:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1747677119; bh=kKAbK/ihktd10bayLsP//PWEOGB9vYQ3zKxqoKSXAE8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Oq+0yzHgg/BE5fx8mWtvGsUZsDNEl+TglApCHMhKrDLYo7bdS+pUHN2mZTrEmoPHR IM5pPe9XgnOziiQyvPwuQmseGRtl4SiOwhx9anLsXUMc8AL78m1bT13Mmc0zn/wx5o 523Ppp6ZmG6Gr3yiwa3AARwmMKgGpu7K/ufv0sduClRfWk7uM4mP81MuC/e7DLyrg1 eTPJmldloKsrqZbFspNXjm32F9V3C72jFsShlXfO3eQ4X7CpJoxLVybdT1lP5hDhSE yug0J+IvMku+I7yPD/9hd2gS6l0LEACk5LD0XXVtvVf9oWTOSAssUAfQTi7NQneTDb kFTNIJa5EehDg== From: Eric Biggers To: netdev@vger.kernel.org Cc: linux-nvme@lists.infradead.org, linux-sctp@vger.kernel.org, linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org, Daniel Borkmann , Marcelo Ricardo Leitner , Sagi Grimberg , Ard Biesheuvel Subject: [PATCH v2 08/10] net: add skb_copy_and_crc32c_datagram_iter() Date: Mon, 19 May 2025 10:50:10 -0700 Message-ID: <20250519175012.36581-9-ebiggers@kernel.org> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250519175012.36581-1-ebiggers@kernel.org> References: <20250519175012.36581-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Eric Biggers Since skb_copy_and_hash_datagram_iter() is used only with CRC32C, the crypto_ahash abstraction provides no value. Add skb_copy_and_crc32c_datagram_iter() which just calls crc32c() directly. This is faster and simpler. It also doesn't have the weird dependency issue where skb_copy_and_hash_datagram_iter() depends on CONFIG_CRYPTO_HASH=3Dy without that being expressed explicitly in the kconfig (presumably because it was too heavyweight for NET to select). The new function is conditional on the hidden boolean symbol NET_CRC32C, which selects CRC32. So it gets compiled only when something that actually needs CRC32C packet checksums is enabled, it has no implicit dependency, and it doesn't depend on the heavyweight crypto layer. Signed-off-by: Eric Biggers Reviewed-by: Hannes Reinecke --- include/linux/skbuff.h | 2 ++ net/core/datagram.c | 33 +++++++++++++++++++++++++++++++++ 2 files changed, 35 insertions(+) diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index 018c072305133..510adf63c2113 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -4135,10 +4135,12 @@ static inline int skb_copy_datagram_msg(const struc= t sk_buff *from, int offset, int skb_copy_and_csum_datagram_msg(struct sk_buff *skb, int hlen, struct msghdr *msg); int skb_copy_and_hash_datagram_iter(const struct sk_buff *skb, int offset, struct iov_iter *to, int len, struct ahash_request *hash); +int skb_copy_and_crc32c_datagram_iter(const struct sk_buff *skb, int offse= t, + struct iov_iter *to, int len, u32 *crcp); int skb_copy_datagram_from_iter(struct sk_buff *skb, int offset, struct iov_iter *from, int len); int zerocopy_sg_from_iter(struct sk_buff *skb, struct iov_iter *frm); void skb_free_datagram(struct sock *sk, struct sk_buff *skb); int skb_kill_datagram(struct sock *sk, struct sk_buff *skb, unsigned int f= lags); diff --git a/net/core/datagram.c b/net/core/datagram.c index 9ef5442536f59..fa87abb666324 100644 --- a/net/core/datagram.c +++ b/net/core/datagram.c @@ -50,10 +50,11 @@ #include #include #include #include #include +#include =20 #include #include =20 #include @@ -517,10 +518,42 @@ int skb_copy_and_hash_datagram_iter(const struct sk_b= uff *skb, int offset, return __skb_datagram_iter(skb, offset, to, len, true, hash_and_copy_to_iter, hash); } EXPORT_SYMBOL(skb_copy_and_hash_datagram_iter); =20 +#ifdef CONFIG_NET_CRC32C +static size_t crc32c_and_copy_to_iter(const void *addr, size_t bytes, + void *_crcp, struct iov_iter *i) +{ + u32 *crcp =3D _crcp; + size_t copied; + + copied =3D copy_to_iter(addr, bytes, i); + *crcp =3D crc32c(*crcp, addr, copied); + return copied; +} + +/** + * skb_copy_and_crc32c_datagram_iter - Copy datagram to an iovec iterator + * and update a CRC32C value. + * @skb: buffer to copy + * @offset: offset in the buffer to start copying from + * @to: iovec iterator to copy to + * @len: amount of data to copy from buffer to iovec + * @crcp: pointer to CRC32C value to update + * + * Return: 0 on success, -EFAULT if there was a fault during copy. + */ +int skb_copy_and_crc32c_datagram_iter(const struct sk_buff *skb, int offse= t, + struct iov_iter *to, int len, u32 *crcp) +{ + return __skb_datagram_iter(skb, offset, to, len, true, + crc32c_and_copy_to_iter, crcp); +} +EXPORT_SYMBOL(skb_copy_and_crc32c_datagram_iter); +#endif /* CONFIG_NET_CRC32C */ + static size_t simple_copy_to_iter(const void *addr, size_t bytes, void *data __always_unused, struct iov_iter *i) { return copy_to_iter(addr, bytes, i); } --=20 2.49.0 From nobody Fri Dec 19 14:21:05 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AEDAB28A715; Mon, 19 May 2025 17:52:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1747677120; cv=none; b=VRqzSNTa6K6na7L8fY0qjAue1QBllY6BQoZVNb1D2QrSSQ+tyJW0xvB04kscRT4tMmDdBQL+T/3tqE3Z3kJFA1kiycZdqhS6t4VDs7eYQ4CDTAKt9ZTkhDUBGrtpBO89cJW2Or7simUPGM4ew31ZbKH7jWxHXfw8uXuGTv4ED80= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1747677120; c=relaxed/simple; bh=wJP8T2TD425wEzkz5/xhkjJMoKHuDj15HpiXYQizJiI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=GXu03Akij/NjCjPUqvDCg/8owjvh3fsThTPUNFrce2BmAHXcG+QPxkXCOSGf8Mw0OBqMgekGbUm/gaq19SN6h4mB2MjdAIipWbXCc5mJPSNJSIrGKprFCcYpUbjc45lofuFT6aexpQWSZifiSbb6Hhg1Cc0PDLN6ag3MnSW2tgw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=ldxK9dw+; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="ldxK9dw+" Received: by smtp.kernel.org (Postfix) with ESMTPSA id DD6F8C4CEE4; Mon, 19 May 2025 17:51:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1747677120; bh=wJP8T2TD425wEzkz5/xhkjJMoKHuDj15HpiXYQizJiI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ldxK9dw+qB9yi477JSw2w62fHlbk+gbZMw5KHZGuCRbcKH7ZHuarO18o7Z5CQVX+i DXnnJIPe2FutEnBkG3o08O3jtiue2nXCme3gL048jTSo9xCg9h56VJ9NUXOX5Cngv/ tJn83pA2xTK0JXBFODyuldig4BYNp17Z4H6eJpMWPOlebzI1XPotGsZaRLmmBhShyr 1ke5xpA0iwQLzDw2Pzy9O1S/5zgQr1b8jmT9XHNHh5hiuW+MOfKrvpq7WbEbWBbvMq TKAdtL+F04l2NfH6gTq+IOVruGhxu2a49bWNBeIgeJXYhjP2sIe5O4jpLIcZ7HZ5sx tNTESKaHzeIEQ== From: Eric Biggers To: netdev@vger.kernel.org Cc: linux-nvme@lists.infradead.org, linux-sctp@vger.kernel.org, linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org, Daniel Borkmann , Marcelo Ricardo Leitner , Sagi Grimberg , Ard Biesheuvel , Christoph Hellwig Subject: [PATCH v2 09/10] nvme-tcp: use crc32c() and skb_copy_and_crc32c_datagram_iter() Date: Mon, 19 May 2025 10:50:11 -0700 Message-ID: <20250519175012.36581-10-ebiggers@kernel.org> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250519175012.36581-1-ebiggers@kernel.org> References: <20250519175012.36581-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Eric Biggers Now that the crc32c() library function directly takes advantage of architecture-specific optimizations and there also now exists a function skb_copy_and_crc32c_datagram_iter(), it is unnecessary to go through the crypto_ahash API. Just use those functions. This is much simpler, and it also improves performance due to eliminating the crypto API overhead. Reviewed-by: Christoph Hellwig Reviewed-by: Sagi Grimberg Signed-off-by: Eric Biggers Reviewed-by: Hannes Reinecke --- drivers/nvme/host/Kconfig | 4 +- drivers/nvme/host/tcp.c | 124 ++++++++++++-------------------------- 2 files changed, 42 insertions(+), 86 deletions(-) diff --git a/drivers/nvme/host/Kconfig b/drivers/nvme/host/Kconfig index 4d64b6935bb91..7dca58f0a237b 100644 --- a/drivers/nvme/host/Kconfig +++ b/drivers/nvme/host/Kconfig @@ -82,13 +82,13 @@ config NVME_FC =20 config NVME_TCP tristate "NVM Express over Fabrics TCP host driver" depends on INET depends on BLOCK + select CRC32 + select NET_CRC32C select NVME_FABRICS - select CRYPTO - select CRYPTO_CRC32C help This provides support for the NVMe over Fabrics protocol using the TCP transport. This allows you to use remote block devices exported using the NVMe protocol set. =20 diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c index aba365f97cf6b..8ae6cc2280caa 100644 --- a/drivers/nvme/host/tcp.c +++ b/drivers/nvme/host/tcp.c @@ -6,19 +6,19 @@ #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt #include #include #include #include +#include #include #include #include #include #include #include #include #include -#include #include #include =20 #include "nvme.h" #include "fabrics.h" @@ -166,12 +166,12 @@ struct nvme_tcp_queue { bool rd_enabled; =20 bool hdr_digest; bool data_digest; bool tls_enabled; - struct ahash_request *rcv_hash; - struct ahash_request *snd_hash; + u32 rcv_crc; + u32 snd_crc; __le32 exp_ddgst; __le32 recv_ddgst; struct completion tls_complete; int tls_err; struct page_frag_cache pf_cache; @@ -454,36 +454,42 @@ nvme_tcp_fetch_request(struct nvme_tcp_queue *queue) =20 list_del(&req->entry); return req; } =20 -static inline void nvme_tcp_ddgst_final(struct ahash_request *hash, - __le32 *dgst) +#define NVME_TCP_CRC_SEED (~0) + +static inline void nvme_tcp_ddgst_update(u32 *crcp, + struct page *page, size_t off, size_t len) { - ahash_request_set_crypt(hash, NULL, (u8 *)dgst, 0); - crypto_ahash_final(hash); + page +=3D off / PAGE_SIZE; + off %=3D PAGE_SIZE; + while (len) { + const void *vaddr =3D kmap_local_page(page); + size_t n =3D min(len, (size_t)PAGE_SIZE - off); + + *crcp =3D crc32c(*crcp, vaddr + off, n); + kunmap_local(vaddr); + page++; + off =3D 0; + len -=3D n; + } } =20 -static inline void nvme_tcp_ddgst_update(struct ahash_request *hash, - struct page *page, off_t off, size_t len) +static inline __le32 nvme_tcp_ddgst_final(u32 crc) { - struct scatterlist sg; - - sg_init_table(&sg, 1); - sg_set_page(&sg, page, len, off); - ahash_request_set_crypt(hash, &sg, NULL, len); - crypto_ahash_update(hash); + return cpu_to_le32(~crc); } =20 -static inline void nvme_tcp_hdgst(struct ahash_request *hash, - void *pdu, size_t len) +static inline __le32 nvme_tcp_hdgst(const void *pdu, size_t len) { - struct scatterlist sg; + return cpu_to_le32(~crc32c(NVME_TCP_CRC_SEED, pdu, len)); +} =20 - sg_init_one(&sg, pdu, len); - ahash_request_set_crypt(hash, &sg, pdu + len, len); - crypto_ahash_digest(hash); +static inline void nvme_tcp_set_hdgst(void *pdu, size_t len) +{ + *(__le32 *)(pdu + len) =3D nvme_tcp_hdgst(pdu, len); } =20 static int nvme_tcp_verify_hdgst(struct nvme_tcp_queue *queue, void *pdu, size_t pdu_len) { @@ -497,12 +503,11 @@ static int nvme_tcp_verify_hdgst(struct nvme_tcp_queu= e *queue, nvme_tcp_queue_id(queue)); return -EPROTO; } =20 recv_digest =3D *(__le32 *)(pdu + hdr->hlen); - nvme_tcp_hdgst(queue->rcv_hash, pdu, pdu_len); - exp_digest =3D *(__le32 *)(pdu + hdr->hlen); + exp_digest =3D nvme_tcp_hdgst(pdu, pdu_len); if (recv_digest !=3D exp_digest) { dev_err(queue->ctrl->ctrl.device, "header digest error: recv %#x expected %#x\n", le32_to_cpu(recv_digest), le32_to_cpu(exp_digest)); return -EIO; @@ -524,11 +529,11 @@ static int nvme_tcp_check_ddgst(struct nvme_tcp_queue= *queue, void *pdu) dev_err(queue->ctrl->ctrl.device, "queue %d: data digest flag is cleared\n", nvme_tcp_queue_id(queue)); return -EPROTO; } - crypto_ahash_init(queue->rcv_hash); + queue->rcv_crc =3D NVME_TCP_CRC_SEED; =20 return 0; } =20 static void nvme_tcp_exit_request(struct blk_mq_tag_set *set, @@ -924,12 +929,12 @@ static int nvme_tcp_recv_data(struct nvme_tcp_queue *= queue, struct sk_buff *skb, /* we can read only from what is left in this bio */ recv_len =3D min_t(size_t, recv_len, iov_iter_count(&req->iter)); =20 if (queue->data_digest) - ret =3D skb_copy_and_hash_datagram_iter(skb, *offset, - &req->iter, recv_len, queue->rcv_hash); + ret =3D skb_copy_and_crc32c_datagram_iter(skb, *offset, + &req->iter, recv_len, &queue->rcv_crc); else ret =3D skb_copy_datagram_iter(skb, *offset, &req->iter, recv_len); if (ret) { dev_err(queue->ctrl->ctrl.device, @@ -943,11 +948,11 @@ static int nvme_tcp_recv_data(struct nvme_tcp_queue *= queue, struct sk_buff *skb, queue->data_remaining -=3D recv_len; } =20 if (!queue->data_remaining) { if (queue->data_digest) { - nvme_tcp_ddgst_final(queue->rcv_hash, &queue->exp_ddgst); + queue->exp_ddgst =3D nvme_tcp_ddgst_final(queue->rcv_crc); queue->ddgst_remaining =3D NVME_TCP_DIGEST_LENGTH; } else { if (pdu->hdr.flags & NVME_TCP_F_DATA_SUCCESS) { nvme_tcp_end_request(rq, le16_to_cpu(req->status)); @@ -1145,11 +1150,11 @@ static int nvme_tcp_try_send_data(struct nvme_tcp_r= equest *req) ret =3D sock_sendmsg(queue->sock, &msg); if (ret <=3D 0) return ret; =20 if (queue->data_digest) - nvme_tcp_ddgst_update(queue->snd_hash, page, + nvme_tcp_ddgst_update(&queue->snd_crc, page, offset, ret); =20 /* * update the request iterator except for the last payload send * in the request where we don't want to modify it as we may @@ -1159,12 +1164,12 @@ static int nvme_tcp_try_send_data(struct nvme_tcp_r= equest *req) nvme_tcp_advance_req(req, ret); =20 /* fully successful last send in current PDU */ if (last && ret =3D=3D len) { if (queue->data_digest) { - nvme_tcp_ddgst_final(queue->snd_hash, - &req->ddgst); + req->ddgst =3D + nvme_tcp_ddgst_final(queue->snd_crc); req->state =3D NVME_TCP_SEND_DDGST; req->offset =3D 0; } else { if (h2cdata_left) nvme_tcp_setup_h2c_data_pdu(req); @@ -1192,11 +1197,11 @@ static int nvme_tcp_try_send_cmd_pdu(struct nvme_tc= p_request *req) msg.msg_flags |=3D MSG_MORE; else msg.msg_flags |=3D MSG_EOR; =20 if (queue->hdr_digest && !req->offset) - nvme_tcp_hdgst(queue->snd_hash, pdu, sizeof(*pdu)); + nvme_tcp_set_hdgst(pdu, sizeof(*pdu)); =20 bvec_set_virt(&bvec, (void *)pdu + req->offset, len); iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, len); ret =3D sock_sendmsg(queue->sock, &msg); if (unlikely(ret <=3D 0)) @@ -1205,11 +1210,11 @@ static int nvme_tcp_try_send_cmd_pdu(struct nvme_tc= p_request *req) len -=3D ret; if (!len) { if (inline_data) { req->state =3D NVME_TCP_SEND_DATA; if (queue->data_digest) - crypto_ahash_init(queue->snd_hash); + queue->snd_crc =3D NVME_TCP_CRC_SEED; } else { nvme_tcp_done_send_req(queue); } return 1; } @@ -1227,11 +1232,11 @@ static int nvme_tcp_try_send_data_pdu(struct nvme_t= cp_request *req) u8 hdgst =3D nvme_tcp_hdgst_len(queue); int len =3D sizeof(*pdu) - req->offset + hdgst; int ret; =20 if (queue->hdr_digest && !req->offset) - nvme_tcp_hdgst(queue->snd_hash, pdu, sizeof(*pdu)); + nvme_tcp_set_hdgst(pdu, sizeof(*pdu)); =20 if (!req->h2cdata_left) msg.msg_flags |=3D MSG_SPLICE_PAGES; =20 bvec_set_virt(&bvec, (void *)pdu + req->offset, len); @@ -1242,11 +1247,11 @@ static int nvme_tcp_try_send_data_pdu(struct nvme_t= cp_request *req) =20 len -=3D ret; if (!len) { req->state =3D NVME_TCP_SEND_DATA; if (queue->data_digest) - crypto_ahash_init(queue->snd_hash); + queue->snd_crc =3D NVME_TCP_CRC_SEED; return 1; } req->offset +=3D ret; =20 return -EAGAIN; @@ -1382,45 +1387,10 @@ static void nvme_tcp_io_work(struct work_struct *w) } while (!time_after(jiffies, deadline)); /* quota is exhausted */ =20 queue_work_on(queue->io_cpu, nvme_tcp_wq, &queue->io_work); } =20 -static void nvme_tcp_free_crypto(struct nvme_tcp_queue *queue) -{ - struct crypto_ahash *tfm =3D crypto_ahash_reqtfm(queue->rcv_hash); - - ahash_request_free(queue->rcv_hash); - ahash_request_free(queue->snd_hash); - crypto_free_ahash(tfm); -} - -static int nvme_tcp_alloc_crypto(struct nvme_tcp_queue *queue) -{ - struct crypto_ahash *tfm; - - tfm =3D crypto_alloc_ahash("crc32c", 0, CRYPTO_ALG_ASYNC); - if (IS_ERR(tfm)) - return PTR_ERR(tfm); - - queue->snd_hash =3D ahash_request_alloc(tfm, GFP_KERNEL); - if (!queue->snd_hash) - goto free_tfm; - ahash_request_set_callback(queue->snd_hash, 0, NULL, NULL); - - queue->rcv_hash =3D ahash_request_alloc(tfm, GFP_KERNEL); - if (!queue->rcv_hash) - goto free_snd_hash; - ahash_request_set_callback(queue->rcv_hash, 0, NULL, NULL); - - return 0; -free_snd_hash: - ahash_request_free(queue->snd_hash); -free_tfm: - crypto_free_ahash(tfm); - return -ENOMEM; -} - static void nvme_tcp_free_async_req(struct nvme_tcp_ctrl *ctrl) { struct nvme_tcp_request *async =3D &ctrl->async_req; =20 page_frag_free(async->pdu); @@ -1449,13 +1419,10 @@ static void nvme_tcp_free_queue(struct nvme_ctrl *n= ctrl, int qid) unsigned int noreclaim_flag; =20 if (!test_and_clear_bit(NVME_TCP_Q_ALLOCATED, &queue->flags)) return; =20 - if (queue->hdr_digest || queue->data_digest) - nvme_tcp_free_crypto(queue); - page_frag_cache_drain(&queue->pf_cache); =20 noreclaim_flag =3D memalloc_noreclaim_save(); /* ->sock will be released by fput() */ fput(queue->sock->file); @@ -1865,25 +1832,17 @@ static int nvme_tcp_alloc_queue(struct nvme_ctrl *n= ctrl, int qid, } } =20 queue->hdr_digest =3D nctrl->opts->hdr_digest; queue->data_digest =3D nctrl->opts->data_digest; - if (queue->hdr_digest || queue->data_digest) { - ret =3D nvme_tcp_alloc_crypto(queue); - if (ret) { - dev_err(nctrl->device, - "failed to allocate queue %d crypto\n", qid); - goto err_sock; - } - } =20 rcv_pdu_size =3D sizeof(struct nvme_tcp_rsp_pdu) + nvme_tcp_hdgst_len(queue); queue->pdu =3D kmalloc(rcv_pdu_size, GFP_KERNEL); if (!queue->pdu) { ret =3D -ENOMEM; - goto err_crypto; + goto err_sock; } =20 dev_dbg(nctrl->device, "connecting queue %d\n", nvme_tcp_queue_id(queue)); =20 @@ -1912,13 +1871,10 @@ static int nvme_tcp_alloc_queue(struct nvme_ctrl *n= ctrl, int qid, =20 err_init_connect: kernel_sock_shutdown(queue->sock, SHUT_RDWR); err_rcv_pdu: kfree(queue->pdu); -err_crypto: - if (queue->hdr_digest || queue->data_digest) - nvme_tcp_free_crypto(queue); err_sock: /* ->sock will be released by fput() */ fput(queue->sock->file); queue->sock =3D NULL; err_destroy_mutex: --=20 2.49.0 From nobody Fri Dec 19 14:21:05 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B2D7628A719; Mon, 19 May 2025 17:52:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1747677120; cv=none; b=Gn3COtg7/ImEVRwqqEemI7jkV4tgk300/BF/TngsQqHT8zZCoFweVUTO5CS0ouLxXn92Lvu6NTPngHRf0ivbtqvporiF7NjzL5rXNGhocy+QQUMl6LOkvuJFkOWnWuGRG2zvcioAUxWs+wfkA7hvon+jtG0TRRrxvhRdtKNtg6A= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1747677120; c=relaxed/simple; bh=mIsfiVI45v2VPTWBozlSjRtw4VqrI/GcVcTcOwb4ZRY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=dev2BLGtVx1kP7FXZZDS+bsr18pP2pw5AukOizD9jk5qfoJoQIvUMbWmay25ANl9oyKKiWPpn5aqyzWFtZdRzZ3I0/JL4EwYHXqwMFPVuY9ZsN36cxWNBv5jAC+f7ahbmQ/1P4BIGgR5Qwxriz1E2JXHW0Fgdo1FRo0IWpaZOag= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=QR2+iVWK; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="QR2+iVWK" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 56F53C4CEF1; Mon, 19 May 2025 17:52:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1747677120; bh=mIsfiVI45v2VPTWBozlSjRtw4VqrI/GcVcTcOwb4ZRY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=QR2+iVWKJqeHqPio6M358dZ6cQUWbOX/PBRAO2eoWIddik+He5js4DXnq6aqPbeYC 6x+To/VR2K2WcczMELmWTmH2uhqmvKf9Ge4oKVWuJrZhPiZgZ4xAl34/1FENmTSEIe P5c8E3xZSsSx+Et92DossoX9+FwAlX7/23r2kUxcCQ9eiNGcjZH6nAgYvrI9UZxBmB qKSfwx/i2h7c5g7THDj7SPzNsg2zNqCqQVmW9CB5Ay+YoFCZbrvUXZ021BFfmkpBSD V+155XQBuNbNnmaTsxGZ08hIfgzf+MmGtyY1z1ND0UtLvq/iYbjAf8ccvf2cPmNuRw 3/bMDyqVcqqiw== From: Eric Biggers To: netdev@vger.kernel.org Cc: linux-nvme@lists.infradead.org, linux-sctp@vger.kernel.org, linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org, Daniel Borkmann , Marcelo Ricardo Leitner , Sagi Grimberg , Ard Biesheuvel Subject: [PATCH v2 10/10] net: remove skb_copy_and_hash_datagram_iter() Date: Mon, 19 May 2025 10:50:12 -0700 Message-ID: <20250519175012.36581-11-ebiggers@kernel.org> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250519175012.36581-1-ebiggers@kernel.org> References: <20250519175012.36581-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Eric Biggers Now that skb_copy_and_hash_datagram_iter() is no longer used, remove it. Signed-off-by: Eric Biggers Reviewed-by: Hannes Reinecke --- include/linux/skbuff.h | 4 ---- net/core/datagram.c | 37 ------------------------------------- 2 files changed, 41 deletions(-) diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index 510adf63c2113..5520524c93bff 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -272,11 +272,10 @@ /* return minimum truesize of one skb containing X bytes of data */ #define SKB_TRUESIZE(X) ((X) + \ SKB_DATA_ALIGN(sizeof(struct sk_buff)) + \ SKB_DATA_ALIGN(sizeof(struct skb_shared_info))) =20 -struct ahash_request; struct net_device; struct scatterlist; struct pipe_inode_info; struct iov_iter; struct napi_struct; @@ -4132,13 +4131,10 @@ static inline int skb_copy_datagram_msg(const struc= t sk_buff *from, int offset, { return skb_copy_datagram_iter(from, offset, &msg->msg_iter, size); } int skb_copy_and_csum_datagram_msg(struct sk_buff *skb, int hlen, struct msghdr *msg); -int skb_copy_and_hash_datagram_iter(const struct sk_buff *skb, int offset, - struct iov_iter *to, int len, - struct ahash_request *hash); int skb_copy_and_crc32c_datagram_iter(const struct sk_buff *skb, int offse= t, struct iov_iter *to, int len, u32 *crcp); int skb_copy_datagram_from_iter(struct sk_buff *skb, int offset, struct iov_iter *from, int len); int zerocopy_sg_from_iter(struct sk_buff *skb, struct iov_iter *frm); diff --git a/net/core/datagram.c b/net/core/datagram.c index fa87abb666324..b352a10093041 100644 --- a/net/core/datagram.c +++ b/net/core/datagram.c @@ -60,11 +60,10 @@ #include #include #include #include #include -#include =20 #include "devmem.h" =20 /* * Is a socket 'connection oriented' ? @@ -482,46 +481,10 @@ static int __skb_datagram_iter(const struct sk_buff *= skb, int offset, goto fault; =20 return 0; } =20 -static size_t hash_and_copy_to_iter(const void *addr, size_t bytes, void *= hashp, - struct iov_iter *i) -{ -#ifdef CONFIG_CRYPTO_HASH - struct ahash_request *hash =3D hashp; - struct scatterlist sg; - size_t copied; - - copied =3D copy_to_iter(addr, bytes, i); - sg_init_one(&sg, addr, copied); - ahash_request_set_crypt(hash, &sg, NULL, copied); - crypto_ahash_update(hash); - return copied; -#else - return 0; -#endif -} - -/** - * skb_copy_and_hash_datagram_iter - Copy datagram to an iovec iterator - * and update a hash. - * @skb: buffer to copy - * @offset: offset in the buffer to start copying from - * @to: iovec iterator to copy to - * @len: amount of data to copy from buffer to iovec - * @hash: hash request to update - */ -int skb_copy_and_hash_datagram_iter(const struct sk_buff *skb, int offset, - struct iov_iter *to, int len, - struct ahash_request *hash) -{ - return __skb_datagram_iter(skb, offset, to, len, true, - hash_and_copy_to_iter, hash); -} -EXPORT_SYMBOL(skb_copy_and_hash_datagram_iter); - #ifdef CONFIG_NET_CRC32C static size_t crc32c_and_copy_to_iter(const void *addr, size_t bytes, void *_crcp, struct iov_iter *i) { u32 *crcp =3D _crcp; --=20 2.49.0