Loading a large (~2.1G) files with kexec crashes the host with when
running:

  # kexec --load kernel --initrd initrd_with_2G_or_more

  UBSAN: signed-integer-overflow in ./include/crypto/sha256_base.h:64:19
  34152083 * 64 cannot be represented in type 'int'
  ...
  BUG: unable to handle page fault for address: ff9fffff83b624c0
  sha256_update (lib/crypto/sha256.c:137)
  crypto_sha256_update (crypto/sha256_generic.c:40)
  kexec_calculate_store_digests (kernel/kexec_file.c:769)
  __se_sys_kexec_file_load (kernel/kexec_file.c:397 kernel/kexec_file.c:332)
  ...

(Line numbers based on commit da274362a7bd9 ("Linux 6.12.49")

This is not happening upstream (v6.16+), given that `block` type was
upgraded from "int" to "size_t" in commit 74a43a2cf5e8 ("crypto:
lib/sha256 - Move partial block handling out")

Upgrade the block type similar to the commit above, avoiding hitting the
overflow.

This patch is only suitable for the stable tree, and before 6.16, which
got commit 74a43a2cf5e8 ("crypto: lib/sha256 - Move partial block
handling out")

Signed-off-by: Breno Leitao <leitao@debian.org>
Fixes: 11b8d5ef9138 ("crypto: sha256 - implement base layer for SHA-256") # not after v6.16
Reported-by: Michael van der Westhuizen <rmikey@meta.com>
Reported-by: Tobias Fleig <tfleig@meta.com>
---
 include/crypto/sha256_base.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/include/crypto/sha256_base.h b/include/crypto/sha256_base.h
index e0418818d63c8..fa63af10102b2 100644
--- a/include/crypto/sha256_base.h
+++ b/include/crypto/sha256_base.h
@@ -44,7 +44,7 @@ static inline int lib_sha256_base_do_update(struct sha256_state *sctx,
 	sctx->count += len;
 
 	if (unlikely((partial + len) >= SHA256_BLOCK_SIZE)) {
-		int blocks;
+		size_t blocks;
 
 		if (partial) {
 			int p = SHA256_BLOCK_SIZE - partial;

---
base-commit: da274362a7bd9ab3a6e46d15945029145ebce672
change-id: 20251001-stable_crash-f2151baf043b

Best regards,
--  
Breno Leitao <leitao@debian.org>

On Wed, Oct 01, 2025 at 09:07:07AM -0700, Breno Leitao wrote:
> Loading a large (~2.1G) files with kexec crashes the host with when
> running:
> 
>   # kexec --load kernel --initrd initrd_with_2G_or_more
> 
>   UBSAN: signed-integer-overflow in ./include/crypto/sha256_base.h:64:19
>   34152083 * 64 cannot be represented in type 'int'
>   ...
>   BUG: unable to handle page fault for address: ff9fffff83b624c0
>   sha256_update (lib/crypto/sha256.c:137)
>   crypto_sha256_update (crypto/sha256_generic.c:40)
>   kexec_calculate_store_digests (kernel/kexec_file.c:769)
>   __se_sys_kexec_file_load (kernel/kexec_file.c:397 kernel/kexec_file.c:332)
>   ...
> 
> (Line numbers based on commit da274362a7bd9 ("Linux 6.12.49")
> 
> This is not happening upstream (v6.16+), given that `block` type was
> upgraded from "int" to "size_t" in commit 74a43a2cf5e8 ("crypto:
> lib/sha256 - Move partial block handling out")
> 
> Upgrade the block type similar to the commit above, avoiding hitting the
> overflow.
> 
> This patch is only suitable for the stable tree, and before 6.16, which
> got commit 74a43a2cf5e8 ("crypto: lib/sha256 - Move partial block
> handling out")
> 
> Signed-off-by: Breno Leitao <leitao@debian.org>
> Fixes: 11b8d5ef9138 ("crypto: sha256 - implement base layer for SHA-256") # not after v6.16
> Reported-by: Michael van der Westhuizen <rmikey@meta.com>
> Reported-by: Tobias Fleig <tfleig@meta.com>
> ---
>  include/crypto/sha256_base.h | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/include/crypto/sha256_base.h b/include/crypto/sha256_base.h
> index e0418818d63c8..fa63af10102b2 100644
> --- a/include/crypto/sha256_base.h
> +++ b/include/crypto/sha256_base.h
> @@ -44,7 +44,7 @@ static inline int lib_sha256_base_do_update(struct sha256_state *sctx,
>  	sctx->count += len;
>  
>  	if (unlikely((partial + len) >= SHA256_BLOCK_SIZE)) {
> -		int blocks;
> +		size_t blocks;
>  
>  		if (partial) {
>  			int p = SHA256_BLOCK_SIZE - partial;
> 
> ---

This looks fine, but technically 'unsigned int' would be more
appropriate here, given the context.  If we look at the whole function
in 6.12, we can see that it took an 'unsigned int' length:

 static inline int lib_sha256_base_do_update(struct sha256_state *sctx,
 					    const u8 *data,
 					    unsigned int len,
 					    sha256_block_fn *block_fn)
 {
 	unsigned int partial = sctx->count % SHA256_BLOCK_SIZE;
 
 	sctx->count += len;
 
 	if (unlikely((partial + len) >= SHA256_BLOCK_SIZE)) {
-		int blocks;
+		size_t blocks;
 
 		if (partial) {
 			int p = SHA256_BLOCK_SIZE - partial;
 
 			memcpy(sctx->buf + partial, data, p);
 			data += p;
 			len -= p;
 
 			block_fn(sctx, sctx->buf, 1);
 		}
 
 		blocks = len / SHA256_BLOCK_SIZE;
 		len %= SHA256_BLOCK_SIZE;
 
 		if (blocks) {
 			block_fn(sctx, data, blocks);
 			data += blocks * SHA256_BLOCK_SIZE;
 		}
 		partial = 0;
 	}
 	if (len)
 		memcpy(sctx->buf + partial, data, len);
 
 	return 0;
 }

This also suggests that files with lengths greater than UINT_MAX are
still broken.  Is that okay?

Anyway, I'm glad that I fixed all these functions to use size_t lengths
in newer kernels...

- Eric

Hello Eric,

On Wed, Oct 01, 2025 at 09:23:05AM -0700, Eric Biggers wrote:

> This looks fine, but technically 'unsigned int' would be more
> appropriate here, given the context.  If we look at the whole function
> in 6.12, we can see that it took an 'unsigned int' length:

Ack. Do you want me to send a v2 with `unsigned int` instead?

> This also suggests that files with lengths greater than UINT_MAX are
> still broken.  Is that okay?

I've tested it but kexec fails to load it, so, it seems we are safe
here:

	# kexec --load kernel --initrd foo 
	kexec_file_load failed: File too large

> Anyway, I'm glad that I fixed all these functions to use size_t lengths
> in newer kernels...

Thanks for that!
--breno

On Wed, Oct 01, 2025 at 09:45:07AM -0700, Breno Leitao wrote:
> Hello Eric,
> 
> On Wed, Oct 01, 2025 at 09:23:05AM -0700, Eric Biggers wrote:
> 
> > This looks fine, but technically 'unsigned int' would be more
> > appropriate here, given the context.  If we look at the whole function
> > in 6.12, we can see that it took an 'unsigned int' length:
> 
> Ack. Do you want me to send a v2 with `unsigned int` instead?
> 

Sure.  Could you also make it clear which kernel version(s) you are
expecting the patch to be applied to?  Is it everything 5.4 through
6.15?  It looks like this bug actually got exposed by f4da7afe07523f
("kexec_file: increase maximum file size to 4G") in 6.0.  But
backporting to older versions should be fine too, if it applies to them.

- Eric