From nobody Sun Feb 8 17:41:56 2026 Received: from mailscanner05.zoner.fi (mailscanner05.zoner.fi [5.44.246.14]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 726EC85620 for ; Wed, 20 Mar 2024 18:48:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=5.44.246.14 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710960532; cv=none; b=kMeq2qeekBlF7mSD2B81RGg14OAjWrRE5spMoquinY19gLjhXrAjmnVxp2qoyJhZAX64A0tzC/A8uL5cWoANfRi8m7BQX/2scV/saPVSGX/w+vuJ+9BI5s8BdQLF2NYQ4bSMvWzW2KmNkR7kTqZxHpvQxgoz5BhGqX/Yi6ZL+gM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710960532; c=relaxed/simple; bh=/SCw4biXcQwxbS8rB0xZT7OMJzC9utywPDK96DZ5Xao=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=SN0HllpqZ3Uil7pSXaN0U4oAhKpJ0pc0wiI4+fX0S7VMssRp/w4KU6dhHKKSVCN5JXr5OA5YajZhnWjlwS7UjNRaOgbYXOhigBlo8qiYD1JI0/lMdipfMHRSDC2hDTuI82igatBDKg+1cp4eBKK1HspiigvHwu4z0d3iiUduE7k= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=tukaani.org; spf=pass smtp.mailfrom=tukaani.org; arc=none smtp.client-ip=5.44.246.14 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=tukaani.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=tukaani.org Received: from www25.zoner.fi (www25.zoner.fi [84.34.147.45]) by mailscanner05.zoner.fi (Postfix) with ESMTPS id F269E21235; Wed, 20 Mar 2024 20:39:25 +0200 (EET) Received: from mail.zoner.fi ([84.34.147.244]) by www25.zoner.fi with esmtp (Exim 4.96.1-7-g79877b70e) (envelope-from ) id 1rn0qT-0001dW-2Y; Wed, 20 Mar 2024 20:39:25 +0200 From: Lasse Collin To: Andrew Morton Cc: Lasse Collin , Jia Tan , linux-kernel@vger.kernel.org Subject: [PATCH 01/11] MAINTAINERS: Add XZ Embedded maintainers Date: Wed, 20 Mar 2024 20:38:34 +0200 Message-ID: <20240320183846.19475-2-lasse.collin@tukaani.org> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240320183846.19475-1-lasse.collin@tukaani.org> References: <20240320183846.19475-1-lasse.collin@tukaani.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" I have been the maintainer of the upstream project since I submitted the code to Linux in 2010 but I forgot to add myself to MAINTAINERS. Nowadays Jia Tan is the other maintainer. Reviewed-by: Jia Tan Signed-off-by: Lasse Collin --- MAINTAINERS | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/MAINTAINERS b/MAINTAINERS index 43b39956694a..35f4ee968626 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -24371,6 +24371,20 @@ S: Maintained F: drivers/spi/spi-xtensa-xtfpga.c F: sound/soc/xtensa/xtfpga-i2s.c =20 +XZ EMBEDDED +M: Lasse Collin +M: Jia Tan +S: Maintained +W: https://xz.tukaani.org/xz-embedded/ +B: https://github.com/tukaani-project/xz-embedded/issues +C: irc://irc.libera.chat/tukaani +F: Documentation/staging/xz.rst +F: include/linux/decompress/unxz.h +F: include/linux/xz.h +F: lib/decompress_unxz.c +F: lib/xz/ +F: scripts/xz_wrap.sh + YAM DRIVER FOR AX.25 M: Jean-Paul Roubelat L: linux-hams@vger.kernel.org --=20 2.44.0 From nobody Sun Feb 8 17:41:56 2026 Received: from mailscanner01.zoner.fi (mailscanner01.zoner.fi [84.34.166.10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4FAD285266; Wed, 20 Mar 2024 18:47:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=84.34.166.10 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710960469; cv=none; b=Fg8A0GZQ6KQEuZKYzWxYLgDgeoDM4/B7BDnlFfiEn9pQz1LfjWGzV7y+Zw3pJqtgJWubJGAV84Wc//i03B1N6BfiLJ6ORvCukdXj2fDfOHlZHhlyuQS1NLR5gn7SS+9U3CA/7lBJN9j1hS1rlQyd8vNmaTOig4RStDzkUgaC7wI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710960469; c=relaxed/simple; bh=VmUzXlBLA59J9Pdr+pMwk3dhjJxt13xaiba0K2cr7oc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=P4NovFwnjQPMDLHmTot47fclAcXWCeLFLDY3NOjcxxYNjr2jHD/7Xhe2oEKElCN6rXs7XEgILUgdG3WSYe+NEwBzub8ol4zPOjJ8RF+rqXnb+huPrUpVQsuOK7zNqGD48PAy5z0FUbDqOiWHZocivIADyOK9gDN+U5bzCVbC29w= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=tukaani.org; spf=pass smtp.mailfrom=tukaani.org; arc=none smtp.client-ip=84.34.166.10 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=tukaani.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=tukaani.org Received: from www25.zoner.fi (www25.zoner.fi [84.34.147.45]) by mailscanner01.zoner.fi (Postfix) with ESMTPS id 2141D4247B; Wed, 20 Mar 2024 20:39:26 +0200 (EET) Received: from mail.zoner.fi ([84.34.147.244]) by www25.zoner.fi with esmtp (Exim 4.96.1-7-g79877b70e) (envelope-from ) id 1rn0qT-0001dW-2r; Wed, 20 Mar 2024 20:39:25 +0200 From: Lasse Collin To: Andrew Morton Cc: Lasse Collin , Jia Tan , linux-kernel@vger.kernel.org, linux-spdx@vger.kernel.org Subject: [PATCH 02/11] LICENSES: Add 0BSD license text Date: Wed, 20 Mar 2024 20:38:35 +0200 Message-ID: <20240320183846.19475-3-lasse.collin@tukaani.org> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240320183846.19475-1-lasse.collin@tukaani.org> References: <20240320183846.19475-1-lasse.collin@tukaani.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The license text was copied from: https://spdx.org/licenses/0BSD.html Reviewed-by: Jia Tan Signed-off-by: Lasse Collin --- Notes: 0BSD is the ISC license without the requirements to preserve copyright and license notices. LICENSES/deprecated/0BSD | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) create mode 100644 LICENSES/deprecated/0BSD diff --git a/LICENSES/deprecated/0BSD b/LICENSES/deprecated/0BSD new file mode 100644 index 000000000000..e4b95b749966 --- /dev/null +++ b/LICENSES/deprecated/0BSD @@ -0,0 +1,23 @@ +Valid-License-Identifier: 0BSD +SPDX-URL: https://spdx.org/licenses/0BSD.html +Usage-Guide: + To use the BSD Zero Clause License put the following SPDX tag/value + pair into a comment according to the placement guidelines in the + licensing rules documentation: + SPDX-License-Identifier: 0BSD +License-Text: + +BSD Zero Clause License + +Copyright (c) + +Permission to use, copy, modify, and/or distribute this software for any +purpose with or without fee is hereby granted. + +THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES +WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF +MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY +SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES +WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTI= ON +OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN +CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. --=20 2.44.0 From nobody Sun Feb 8 17:41:56 2026 Received: from mailscanner01.zoner.fi (mailscanner01.zoner.fi [84.34.166.10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 69B9185297; Wed, 20 Mar 2024 18:47:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=84.34.166.10 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710960468; cv=none; b=BVwFY6ISKgwpJEBIPVbbdAxuf9CecGshCoxZ9ppn//6YfbbSy5i1fArYZEaionpCKqW0MyAqcuauKSq4xbLeIhqqWNmfsugntvZDqpzTe0j/C9ll0JyCRXiTvMJK3AJHo8hukXEldw62ol6ZOL39+GtH32xKqgLTDwQR0DNFnWY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710960468; c=relaxed/simple; bh=k/qt7fuT3VtOc5fsjebM1i2URNOFtuXnd0jQK+GfIy0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=NulNXVON2bbbiJN+hISqcngBvVEv3zZiAaA2/yF6TJ3/bGUbWSniafif0f7yymwKtWB3QVy/d3+qcaBUYydrCuTG3v/vh+45d6ixfqqeSx338OF9VrU3aFp4m1Ivp/rBnuoZBf5LN+WDTP1LSM6On/IaZWNG3Sgv9rLlBlQ4hZ8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=tukaani.org; spf=pass smtp.mailfrom=tukaani.org; arc=none smtp.client-ip=84.34.166.10 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=tukaani.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=tukaani.org Received: from www25.zoner.fi (www25.zoner.fi [84.34.147.45]) by mailscanner01.zoner.fi (Postfix) with ESMTPS id 4724C42243; Wed, 20 Mar 2024 20:39:26 +0200 (EET) Received: from mail.zoner.fi ([84.34.147.244]) by www25.zoner.fi with esmtp (Exim 4.96.1-7-g79877b70e) (envelope-from ) id 1rn0qU-0001dW-06; Wed, 20 Mar 2024 20:39:26 +0200 From: Lasse Collin To: Andrew Morton Cc: Lasse Collin , Jia Tan , linux-kernel@vger.kernel.org, linux-spdx@vger.kernel.org Subject: [PATCH 03/11] xz: Switch from public domain to BSD Zero Clause License (0BSD) Date: Wed, 20 Mar 2024 20:38:36 +0200 Message-ID: <20240320183846.19475-4-lasse.collin@tukaani.org> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240320183846.19475-1-lasse.collin@tukaani.org> References: <20240320183846.19475-1-lasse.collin@tukaani.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Remove the public domain notices and add SPDX license identifiers. Change MODULE_LICENSE from "GPL" to "Dual BSD/GPL" because 0BSD should count as a BSD license variant here. The switch to 0BSD was done in the upstream XZ Embedded project because public domain has (real or perceived) legal issues in some jurisdictions. Reviewed-by: Jia Tan Signed-off-by: Lasse Collin --- include/linux/decompress/unxz.h | 5 ++--- include/linux/xz.h | 5 ++--- lib/decompress_unxz.c | 5 ++--- lib/xz/xz_crc32.c | 5 ++--- lib/xz/xz_dec_bcj.c | 5 ++--- lib/xz/xz_dec_lzma2.c | 5 ++--- lib/xz/xz_dec_stream.c | 5 ++--- lib/xz/xz_dec_syms.c | 12 +++--------- lib/xz/xz_dec_test.c | 12 +++--------- lib/xz/xz_lzma2.h | 5 ++--- lib/xz/xz_private.h | 5 ++--- lib/xz/xz_stream.h | 5 ++--- scripts/xz_wrap.sh | 5 +---- 13 files changed, 27 insertions(+), 52 deletions(-) diff --git a/include/linux/decompress/unxz.h b/include/linux/decompress/unx= z.h index f764e2a7201e..3dd2658a9dab 100644 --- a/include/linux/decompress/unxz.h +++ b/include/linux/decompress/unxz.h @@ -1,10 +1,9 @@ +/* SPDX-License-Identifier: 0BSD */ + /* * Wrapper for decompressing XZ-compressed kernel, initramfs, and initrd * * Author: Lasse Collin - * - * This file has been put into the public domain. - * You can do whatever you want with this file. */ =20 #ifndef DECOMPRESS_UNXZ_H diff --git a/include/linux/xz.h b/include/linux/xz.h index 7285ca5d56e9..5728d57aecc0 100644 --- a/include/linux/xz.h +++ b/include/linux/xz.h @@ -1,11 +1,10 @@ +/* SPDX-License-Identifier: 0BSD */ + /* * XZ decompressor * * Authors: Lasse Collin * Igor Pavlov - * - * This file has been put into the public domain. - * You can do whatever you want with this file. */ =20 #ifndef XZ_H diff --git a/lib/decompress_unxz.c b/lib/decompress_unxz.c index 842894158944..34bb7efc0412 100644 --- a/lib/decompress_unxz.c +++ b/lib/decompress_unxz.c @@ -1,10 +1,9 @@ +// SPDX-License-Identifier: 0BSD + /* * Wrapper for decompressing XZ-compressed kernel, initramfs, and initrd * * Author: Lasse Collin - * - * This file has been put into the public domain. - * You can do whatever you want with this file. */ =20 /* diff --git a/lib/xz/xz_crc32.c b/lib/xz/xz_crc32.c index 88a2c35e1b59..30b8a27110b1 100644 --- a/lib/xz/xz_crc32.c +++ b/lib/xz/xz_crc32.c @@ -1,11 +1,10 @@ +// SPDX-License-Identifier: 0BSD + /* * CRC32 using the polynomial from IEEE-802.3 * * Authors: Lasse Collin * Igor Pavlov - * - * This file has been put into the public domain. - * You can do whatever you want with this file. */ =20 /* diff --git a/lib/xz/xz_dec_bcj.c b/lib/xz/xz_dec_bcj.c index ef449e97d1a1..ab9237ed6db8 100644 --- a/lib/xz/xz_dec_bcj.c +++ b/lib/xz/xz_dec_bcj.c @@ -1,11 +1,10 @@ +// SPDX-License-Identifier: 0BSD + /* * Branch/Call/Jump (BCJ) filter decoders * * Authors: Lasse Collin * Igor Pavlov - * - * This file has been put into the public domain. - * You can do whatever you want with this file. */ =20 #include "xz_private.h" diff --git a/lib/xz/xz_dec_lzma2.c b/lib/xz/xz_dec_lzma2.c index 27ce34520e78..613939f5dd6c 100644 --- a/lib/xz/xz_dec_lzma2.c +++ b/lib/xz/xz_dec_lzma2.c @@ -1,11 +1,10 @@ +// SPDX-License-Identifier: 0BSD + /* * LZMA2 decoder * * Authors: Lasse Collin * Igor Pavlov - * - * This file has been put into the public domain. - * You can do whatever you want with this file. */ =20 #include "xz_private.h" diff --git a/lib/xz/xz_dec_stream.c b/lib/xz/xz_dec_stream.c index 683570b93a8c..0058406ccd17 100644 --- a/lib/xz/xz_dec_stream.c +++ b/lib/xz/xz_dec_stream.c @@ -1,10 +1,9 @@ +// SPDX-License-Identifier: 0BSD + /* * .xz Stream decoder * * Author: Lasse Collin - * - * This file has been put into the public domain. - * You can do whatever you want with this file. */ =20 #include "xz_private.h" diff --git a/lib/xz/xz_dec_syms.c b/lib/xz/xz_dec_syms.c index 61098c67a413..495d2cc2e6e8 100644 --- a/lib/xz/xz_dec_syms.c +++ b/lib/xz/xz_dec_syms.c @@ -1,10 +1,9 @@ +// SPDX-License-Identifier: 0BSD + /* * XZ decoder module information * * Author: Lasse Collin - * - * This file has been put into the public domain. - * You can do whatever you want with this file. */ =20 #include @@ -25,9 +24,4 @@ EXPORT_SYMBOL(xz_dec_microlzma_end); MODULE_DESCRIPTION("XZ decompressor"); MODULE_VERSION("1.1"); MODULE_AUTHOR("Lasse Collin and Igor Pavlov"); - -/* - * This code is in the public domain, but in Linux it's simplest to just - * say it's GPL and consider the authors as the copyright holders. - */ -MODULE_LICENSE("GPL"); +MODULE_LICENSE("Dual BSD/GPL"); diff --git a/lib/xz/xz_dec_test.c b/lib/xz/xz_dec_test.c index da28a19d6c98..53d3600f2ddb 100644 --- a/lib/xz/xz_dec_test.c +++ b/lib/xz/xz_dec_test.c @@ -1,10 +1,9 @@ +// SPDX-License-Identifier: 0BSD + /* * XZ decoder tester * * Author: Lasse Collin - * - * This file has been put into the public domain. - * You can do whatever you want with this file. */ =20 #include @@ -212,9 +211,4 @@ module_exit(xz_dec_test_exit); MODULE_DESCRIPTION("XZ decompressor tester"); MODULE_VERSION("1.0"); MODULE_AUTHOR("Lasse Collin "); - -/* - * This code is in the public domain, but in Linux it's simplest to just - * say it's GPL and consider the authors as the copyright holders. - */ -MODULE_LICENSE("GPL"); +MODULE_LICENSE("Dual BSD/GPL"); diff --git a/lib/xz/xz_lzma2.h b/lib/xz/xz_lzma2.h index 92d852d4f87a..d2632b7dfb9c 100644 --- a/lib/xz/xz_lzma2.h +++ b/lib/xz/xz_lzma2.h @@ -1,11 +1,10 @@ +/* SPDX-License-Identifier: 0BSD */ + /* * LZMA2 definitions * * Authors: Lasse Collin * Igor Pavlov - * - * This file has been put into the public domain. - * You can do whatever you want with this file. */ =20 #ifndef XZ_LZMA2_H diff --git a/lib/xz/xz_private.h b/lib/xz/xz_private.h index bf1e94ec7873..2412a5d54801 100644 --- a/lib/xz/xz_private.h +++ b/lib/xz/xz_private.h @@ -1,10 +1,9 @@ +/* SPDX-License-Identifier: 0BSD */ + /* * Private includes and definitions * * Author: Lasse Collin - * - * This file has been put into the public domain. - * You can do whatever you want with this file. */ =20 #ifndef XZ_PRIVATE_H diff --git a/lib/xz/xz_stream.h b/lib/xz/xz_stream.h index 430bb3a0d195..55f9f6f94b78 100644 --- a/lib/xz/xz_stream.h +++ b/lib/xz/xz_stream.h @@ -1,10 +1,9 @@ +/* SPDX-License-Identifier: 0BSD */ + /* * Definitions for handling the .xz file format * * Author: Lasse Collin - * - * This file has been put into the public domain. - * You can do whatever you want with this file. */ =20 #ifndef XZ_STREAM_H diff --git a/scripts/xz_wrap.sh b/scripts/xz_wrap.sh index d06baf626abe..bb760b721b2c 100755 --- a/scripts/xz_wrap.sh +++ b/scripts/xz_wrap.sh @@ -1,13 +1,10 @@ #!/bin/sh +# SPDX-License-Identifier: 0BSD # # This is a wrapper for xz to compress the kernel image using appropriate # compression options depending on the architecture. # # Author: Lasse Collin -# -# This file has been put into the public domain. -# You can do whatever you want with this file. -# =20 BCJ=3D LZMA2OPTS=3D --=20 2.44.0 From nobody Sun Feb 8 17:41:56 2026 Received: from mailscanner07.zoner.fi (mailscanner07.zoner.fi [5.44.246.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C0BC785931 for ; Wed, 20 Mar 2024 18:48:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=5.44.246.16 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710960541; cv=none; b=vDm+f65XwtVhfCWSrhq5D+vX1OKLIjaj9fyVXLNAcGSWBXv/vdhunvy5SyBpR//wvbisHi/FjX/dTdb2l6/f61LwePlfYe/DNXkdlXdSk7lRbzIoCPEZBSZj1fggBxHFPLep7aeJP5JMlcFWep7I29EXwsnbncoEatKFOy3UvWI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710960541; c=relaxed/simple; bh=2QBczl5jsg6EeCkH1elB74pqXKrCgP+CdZwrAvNomqc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=dcJOHjjE29r1XqHXnjLNtSo2UtyASM6UOjxQwEFXOk2L1EJs19FCTm5Ky2nbyd+UrNANpa60Y9bbLZaljSwbF514q9ICAalf8xovfAqG8LYgFlYxp+rf/vs9elJIpsffJCPWyoZaBL8Kf1L5k8HOdH1JOBW8eme9eegCQF3BsRY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=tukaani.org; spf=pass smtp.mailfrom=tukaani.org; arc=none smtp.client-ip=5.44.246.16 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=tukaani.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=tukaani.org Received: from www25.zoner.fi (www25.zoner.fi [84.34.147.45]) by mailscanner07.zoner.fi (Postfix) with ESMTPS id 62D3721257; Wed, 20 Mar 2024 20:39:26 +0200 (EET) Received: from mail.zoner.fi ([84.34.147.244]) by www25.zoner.fi with esmtp (Exim 4.96.1-7-g79877b70e) (envelope-from ) id 1rn0qU-0001dW-0c; Wed, 20 Mar 2024 20:39:26 +0200 From: Lasse Collin To: Andrew Morton Cc: Lasse Collin , Jia Tan , Rui Li , linux-kernel@vger.kernel.org Subject: [PATCH 04/11] xz: Documentation/staging/xz.rst: Revise thoroughly Date: Wed, 20 Mar 2024 20:38:37 +0200 Message-ID: <20240320183846.19475-5-lasse.collin@tukaani.org> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240320183846.19475-1-lasse.collin@tukaani.org> References: <20240320183846.19475-1-lasse.collin@tukaani.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add SPDX license identifier. Omit xz_dec_test info. That isn't relevant to developers of non-XZ code. Revise the docs about xzkern and add xzkern_with_size. The latter was added to scripts/Makefile.lib in the commit 7ce7e984ab2b218d ("kbuild: rename cmd_{bzip2,lzma,lzo,lz4,xzkern,zstd22}"). Omit contact info as MAINTAINERS has it. Omit other info that is outdated or not relevant in the kernel context. Co-developed-by: Jia Tan Signed-off-by: Jia Tan Signed-off-by: Lasse Collin --- Documentation/staging/xz.rst | 130 +++++++++++++---------------------- 1 file changed, 46 insertions(+), 84 deletions(-) diff --git a/Documentation/staging/xz.rst b/Documentation/staging/xz.rst index b2f5ff12a161..301b1eeff411 100644 --- a/Documentation/staging/xz.rst +++ b/Documentation/staging/xz.rst @@ -1,3 +1,5 @@ +.. SPDX-License-Identifier: 0BSD + =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D XZ data compression in Linux =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D @@ -6,21 +8,17 @@ Introduction =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =20 XZ is a general purpose data compression format with high compression -ratio and relatively fast decompression. The primary compression -algorithm (filter) is LZMA2. Additional filters can be used to improve -compression ratio even further. E.g. Branch/Call/Jump (BCJ) filters -improve compression ratio of executable data. - -The XZ decompressor in Linux is called XZ Embedded. It supports -the LZMA2 filter and optionally also BCJ filters. CRC32 is supported -for integrity checking. The home page of XZ Embedded is at -, where you can find the -latest version and also information about using the code outside -the Linux kernel. +ratio. The XZ decompressor in Linux is called XZ Embedded. It supports +the LZMA2 filter and optionally also Branch/Call/Jump (BCJ) filters +for executable code. CRC32 is supported for integrity checking. The +home page of XZ Embedded is at . +There you can find the latest version which includes a few optional +extra features that aren't required in the Linux kernel and information +about using the code outside the Linux kernel. =20 For userspace, XZ Utils provide a zlib-like compression library and a gzip-like command line tool. XZ Utils can be downloaded from -. +. =20 XZ related components in the kernel =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D @@ -29,29 +27,24 @@ The xz_dec module provides XZ decompressor with single-= call (buffer to buffer) and multi-call (stateful) APIs. The usage of the xz_dec module is documented in include/linux/xz.h. =20 -The xz_dec_test module is for testing xz_dec. xz_dec_test is not -useful unless you are hacking the XZ decompressor. xz_dec_test -allocates a char device major dynamically to which one can write -.xz files from userspace. The decompressed output is thrown away. -Keep an eye on dmesg to see diagnostics printed by xz_dec_test. -See the xz_dec_test source code for the details. - For decompressing the kernel image, initramfs, and initrd, there is a wrapper function in lib/decompress_unxz.c. Its API is the same as in other decompress_*.c files, which is defined in include/linux/decompress/generic.h. =20 -scripts/xz_wrap.sh is a wrapper for the xz command line tool found -from XZ Utils. The wrapper sets compression options to values suitable -for compressing the kernel image. +For kernel makefiles, three commands are provided for use with +$(call if_changed). They require the xz tool from XZ Utils. + + - $(call if_changed,xzkern) is for compressing the kernel image. + It runs the script scripts/xz_wrap.sh which uses arch-optimized + options and a big LZMA2 dictionary. =20 -For kernel makefiles, two commands are provided for use with -$(call if_needed). The kernel image should be compressed with -$(call if_needed,xzkern) which will use a BCJ filter and a big LZMA2 -dictionary. It will also append a four-byte trailer containing the -uncompressed size of the file, which is needed by the boot code. -Other things should be compressed with $(call if_needed,xzmisc) -which will use no BCJ filter and 1 MiB LZMA2 dictionary. + - $(call if_changed,xzkern_with_size) is like xzkern above but this + also appends a four-byte trailer containing the uncompressed size + of the file, which is needed by the boot code on some archs. + + - Other things can be compressed with $(call if_needed,xzmisc) + which will use no BCJ filter and 1 MiB LZMA2 dictionary. =20 Notes on compression options =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D @@ -61,7 +54,7 @@ CRC32, make sure that you don't use some other integrity = check type when encoding files that are supposed to be decoded by the kernel. With liblzma, you need to use either LZMA_CHECK_NONE or LZMA_CHECK_CRC32 when encoding. With the xz command line tool, use --check=3Dnone or ---check=3Dcrc32. +--check=3Dcrc32 to override the default --check=3Dcrc64. =20 Using CRC32 is strongly recommended unless there is some other layer which will verify the integrity of the uncompressed data anyway. @@ -71,57 +64,26 @@ by the decoder; you can only change the integrity check= type (or disable it) for the actual uncompressed data. =20 In userspace, LZMA2 is typically used with dictionary sizes of several -megabytes. The decoder needs to have the dictionary in RAM, thus big -dictionaries cannot be used for files that are intended to be decoded -by the kernel. 1 MiB is probably the maximum reasonable dictionary -size for in-kernel use (maybe more is OK for initramfs). The presets -in XZ Utils may not be optimal when creating files for the kernel, -so don't hesitate to use custom settings. Example:: - - xz --check=3Dcrc32 --lzma2=3Ddict=3D512KiB inputfile - -An exception to above dictionary size limitation is when the decoder -is used in single-call mode. Decompressing the kernel itself is an -example of this situation. In single-call mode, the memory usage -doesn't depend on the dictionary size, and it is perfectly fine to -use a big dictionary: for maximum compression, the dictionary should -be at least as big as the uncompressed data itself. - -Future plans -=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D - -Creating a limited XZ encoder may be considered if people think it is -useful. LZMA2 is slower to compress than e.g. Deflate or LZO even at -the fastest settings, so it isn't clear if LZMA2 encoder is wanted -into the kernel. - -Support for limited random-access reading is planned for the -decompression code. I don't know if it could have any use in the -kernel, but I know that it would be useful in some embedded projects -outside the Linux kernel. - -Conformance to the .xz file format specification -=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D - -There are a couple of corner cases where things have been simplified -at expense of detecting errors as early as possible. These should not -matter in practice all, since they don't cause security issues. But -it is good to know this if testing the code e.g. with the test files -from XZ Utils. - -Reporting bugs -=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D - -Before reporting a bug, please check that it's not fixed already -at upstream. See to get the -latest code. - -Report bugs to or visit #tukaani on -Freenode and talk to Larhzu. I don't actively read LKML or other -kernel-related mailing lists, so if there's something I should know, -you should email to me personally or use IRC. - -Don't bother Igor Pavlov with questions about the XZ implementation -in the kernel or about XZ Utils. While these two implementations -include essential code that is directly based on Igor Pavlov's code, -these implementations aren't maintained nor supported by him. +megabytes. The decoder needs to have the dictionary in RAM: + + - In multi-call mode the dictionary is allocated as part of the + decoder state. The reasonable maximum dictionary size for in-kernel + use will depend on the target hardware: a few megabytes should be + fine for desktop systems while 64 KiB to 1 MiB might be more + appropriate on some embedded systems. + + - In single-call mode the output buffer is used as the dictionary + buffer. That is, the size of the dictionary doesn't affect the + decompressor memory usage at all. Only the base data structures + are allocated which take a little less than 30 KiB of memory. + For the best compression, the dictionary should be at least + as big as the uncompressed data. A notable example of single-call + mode is decompressing the kernel itself (except on PowerPC). + +The compression presets in XZ Utils may not be optimal when creating +files for the kernel, so don't hesitate to use custom settings to, +for example, set the dictionary size. Also, xz may produce a smaller +file in single-threaded mode so setting that explicitly is recommended. +Example:: + + xz --threads=3D1 --check=3Dcrc32 --lzma2=3Ddict=3D512KiB inputfile --=20 2.44.0 From nobody Sun Feb 8 17:41:56 2026 Received: from mailscanner01.zoner.fi (mailscanner01.zoner.fi [84.34.166.10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 49E4E6A8A6 for ; Wed, 20 Mar 2024 18:47:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=84.34.166.10 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710960468; cv=none; b=pC/w6xI4u9zOmUh6L9bba2rRXPUdKVaOiLyHyRl4nTh2DmwFD9SEdoUFxBE0KFHLTmpSqhsx4OP4d9ZZF09M9QD9yhkrzx9s9p7HCVh51MbRrKD5CG35yj2bJ4xhiTK/X4oN/hdDG8wzCNMLmSuitjISRUCxTu6vUJfgi+qppcI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710960468; c=relaxed/simple; bh=3mhzjDEKYr5v5j74va0Zmjj1DohqL1K2cOtcr+XhpeQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=u8hEkEXOcyNVlj2tQVm9vQo0dy7lnrK0b3UDzNRQRe+erInWiqwS/w5Q3iyX3FZ3GJSFWTcxgjxvMo2qKqwTxQ7OD737YhMnhxq/0uC/GLEfZS3BrAMdgucYWcjf2noIhOIfnQeSmjy0b8Yq1AuYcG0HbEtL32xjevJUjHq/Vac= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=tukaani.org; spf=pass smtp.mailfrom=tukaani.org; arc=none smtp.client-ip=84.34.166.10 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=tukaani.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=tukaani.org Received: from www25.zoner.fi (www25.zoner.fi [84.34.147.45]) by mailscanner01.zoner.fi (Postfix) with ESMTPS id 7BD364251A; Wed, 20 Mar 2024 20:39:26 +0200 (EET) Received: from mail.zoner.fi ([84.34.147.244]) by www25.zoner.fi with esmtp (Exim 4.96.1-7-g79877b70e) (envelope-from ) id 1rn0qU-0001dW-0x; Wed, 20 Mar 2024 20:39:26 +0200 From: Lasse Collin To: Andrew Morton Cc: Lasse Collin , Jia Tan , linux-kernel@vger.kernel.org Subject: [PATCH 05/11] xz: Fix comments and coding style Date: Wed, 20 Mar 2024 20:38:38 +0200 Message-ID: <20240320183846.19475-6-lasse.collin@tukaani.org> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240320183846.19475-1-lasse.collin@tukaani.org> References: <20240320183846.19475-1-lasse.collin@tukaani.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" - Update the URL of the .xz file format specification. - Fix comments that were no longer in sync with the code below them. - Fix language errors. - Fix coding style. Reviewed-by: Jia Tan Signed-off-by: Lasse Collin --- lib/decompress_unxz.c | 20 ++++++++++---------- lib/xz/Kconfig | 3 ++- lib/xz/xz_stream.h | 2 +- scripts/Makefile.lib | 13 ++++++++----- 4 files changed, 21 insertions(+), 17 deletions(-) diff --git a/lib/decompress_unxz.c b/lib/decompress_unxz.c index 34bb7efc0412..46aa3be13fc5 100644 --- a/lib/decompress_unxz.c +++ b/lib/decompress_unxz.c @@ -102,7 +102,7 @@ #ifdef STATIC # define XZ_PREBOOT #else -#include +# include #endif #ifdef __KERNEL__ # include @@ -219,7 +219,7 @@ void *memmove(void *dest, const void *src, size_t size) #endif =20 /* - * Since we need memmove anyway, would use it as memcpy too. + * Since we need memmove anyway, we could use it as memcpy too. * Commented out for now to avoid breaking things. */ /* @@ -389,17 +389,17 @@ STATIC int INIT unxz(unsigned char *in, long in_size, } =20 /* - * This macro is used by architecture-specific files to decompress + * This function is used by architecture-specific files to decompress * the kernel image. */ #ifdef XZ_PREBOOT -STATIC int INIT __decompress(unsigned char *buf, long len, - long (*fill)(void*, unsigned long), - long (*flush)(void*, unsigned long), - unsigned char *out_buf, long olen, - long *pos, - void (*error)(char *x)) +STATIC int INIT __decompress(unsigned char *in, long in_size, + long (*fill)(void *dest, unsigned long size), + long (*flush)(void *src, unsigned long size), + unsigned char *out, long out_size, + long *in_used, + void (*error)(char *x)) { - return unxz(buf, len, fill, flush, out_buf, pos, error); + return unxz(in, in_size, fill, flush, out, in_used, error); } #endif diff --git a/lib/xz/Kconfig b/lib/xz/Kconfig index aef086a6bf2f..6b80453d8f54 100644 --- a/lib/xz/Kconfig +++ b/lib/xz/Kconfig @@ -5,7 +5,8 @@ config XZ_DEC help LZMA2 compression algorithm and BCJ filters are supported using the .xz file format as the container. For integrity checking, - CRC32 is supported. See Documentation/staging/xz.rst for more informati= on. + CRC32 is supported. See Documentation/staging/xz.rst for more + information. =20 if XZ_DEC =20 diff --git a/lib/xz/xz_stream.h b/lib/xz/xz_stream.h index 55f9f6f94b78..242500bd025d 100644 --- a/lib/xz/xz_stream.h +++ b/lib/xz/xz_stream.h @@ -18,7 +18,7 @@ =20 /* * See the .xz file format specification at - * https://tukaani.org/xz/xz-file-format.txt + * https://xz.tukaani.org/format/xz-file-format.txt * to understand the container format. */ =20 diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib index 1bd59b8db05f..986c79c5af89 100644 --- a/scripts/Makefile.lib +++ b/scripts/Makefile.lib @@ -500,14 +500,17 @@ quiet_cmd_uimage =3D UIMAGE $@ =20 # XZ # ------------------------------------------------------------------------= --- -# Use xzkern to compress the kernel image and xzmisc to compress other thi= ngs. +# Use xzkern or xzkern_with_size to compress the kernel image and xzmisc to +# compress other things. # # xzkern uses a big LZMA2 dictionary since it doesn't increase memory usage # of the kernel decompressor. A BCJ filter is used if it is available for -# the target architecture. xzkern also appends uncompressed size of the da= ta -# using size_append. The .xz format has the size information available at -# the end of the file too, but it's in more complex format and it's good to -# avoid changing the part of the boot code that reads the uncompressed siz= e. +# the target architecture. +# +# xzkern_with_size also appends uncompressed size of the data using +# size_append. The .xz format has the size information available at the end +# of the file too, but it's in more complex format and it's good to avoid +# changing the part of the boot code that reads the uncompressed size. # Note that the bytes added by size_append will make the xz tool think that # the file is corrupt. This is expected. # --=20 2.44.0 From nobody Sun Feb 8 17:41:56 2026 Received: from mailscanner11.zoner.fi (mailscanner11.zoner.fi [5.44.246.20]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3726185623 for ; Wed, 20 Mar 2024 18:49:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=5.44.246.20 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710960565; cv=none; b=g6Ppy4rqDsEuTKc6zteajNdOwe/Xv4B/rPxbW5xmuT+Y3OI5KelHj8rzKCMrGWnOaMZS+s8UrZrW+TQKtkrJ5sAMUJHN9pe3rhQy+UNeDY5dLiP4IhhjGfN3BcHdZAHTb6IcC1gRJlFUsBv5yO3VZbB9vgtgXeL764ZGIqoPIKU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710960565; c=relaxed/simple; bh=j0isJCYION1kfv7Iyxp5a8hbbdmD6XkxP4eup1q1PPw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=MELP9MvPXEDmF2azPSQDySp7qKZr8MBPNE9WuP9n71RXw9L4jgZhKDnnflFHbN2PewSgW7aPQBgZ1DLj2h8iT41dagdhUSk23MogMLKFL4O3bAZZ50XA+Y0Fb064c4rFb1xx83lTBLJHdeqazuXhSVju0UCyMtLehYQA9/D2i3E= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=tukaani.org; spf=pass smtp.mailfrom=tukaani.org; arc=none smtp.client-ip=5.44.246.20 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=tukaani.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=tukaani.org Received: from www25.zoner.fi (www25.zoner.fi [84.34.147.45]) by mailscanner11.zoner.fi (Postfix) with ESMTPS id 8C1E320ED0; Wed, 20 Mar 2024 20:39:26 +0200 (EET) Received: from mail.zoner.fi ([84.34.147.244]) by www25.zoner.fi with esmtp (Exim 4.96.1-7-g79877b70e) (envelope-from ) id 1rn0qU-0001dW-1H; Wed, 20 Mar 2024 20:39:26 +0200 From: Lasse Collin To: Andrew Morton Cc: Lasse Collin , Jia Tan , linux-kernel@vger.kernel.org Subject: [PATCH 06/11] xz: Cleanup CRC32 edits from 2018 Date: Wed, 20 Mar 2024 20:38:39 +0200 Message-ID: <20240320183846.19475-7-lasse.collin@tukaani.org> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240320183846.19475-1-lasse.collin@tukaani.org> References: <20240320183846.19475-1-lasse.collin@tukaani.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The commit faa16bc404d72a5a ("lib: Use existing define with polynomial") in 2018 added a dependency on to avoid duplicating the same constant in multiple files. Two months later it was found to be a bad idea and the commit 242cdad873a75652 ("lib/xz: Put CRC32_POLY_LE in xz_private.h") added the definition of CRC32_POLY_LE macro into xz_private.h to avoid including . xz_private.h is a wrong place for it too. Revert back to the upstream version which has the poly in xz_crc32_init() in xz_crc32.c. Reviewed-by: Jia Tan Signed-off-by: Lasse Collin --- lib/xz/xz_crc32.c | 2 +- lib/xz/xz_private.h | 4 ---- 2 files changed, 1 insertion(+), 5 deletions(-) diff --git a/lib/xz/xz_crc32.c b/lib/xz/xz_crc32.c index 30b8a27110b1..effdf34ec48d 100644 --- a/lib/xz/xz_crc32.c +++ b/lib/xz/xz_crc32.c @@ -28,7 +28,7 @@ STATIC_RW_DATA uint32_t xz_crc32_table[256]; =20 XZ_EXTERN void xz_crc32_init(void) { - const uint32_t poly =3D CRC32_POLY_LE; + const uint32_t poly =3D 0xEDB88320; =20 uint32_t i; uint32_t j; diff --git a/lib/xz/xz_private.h b/lib/xz/xz_private.h index 2412a5d54801..811add814ae4 100644 --- a/lib/xz/xz_private.h +++ b/lib/xz/xz_private.h @@ -104,10 +104,6 @@ # endif #endif =20 -#ifndef CRC32_POLY_LE -#define CRC32_POLY_LE 0xedb88320 -#endif - /* * Allocate memory for LZMA2 decoder. xz_dec_lzma2_reset() must be used * before calling xz_dec_lzma2_run(). --=20 2.44.0 From nobody Sun Feb 8 17:41:56 2026 Received: from mailscanner01.zoner.fi (mailscanner01.zoner.fi [84.34.166.10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4FB0785282 for ; Wed, 20 Mar 2024 18:47:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=84.34.166.10 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710960470; cv=none; b=eQ+UR/b+pEx396LgvvfrCPH8VBLrxpkOSan5UOhyfYKLgXSJwYlUHNfWdOERU0SEayJ7sGknd6Mn/7NYhXRptbwczYpfU+tfCcmucqHek4e10+kzMSo/tc8hyUKETBXLATC+nAxG1e8TPcBtr6YJsbpmmwoXiU9m3Kbx/16iRWU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710960470; c=relaxed/simple; bh=wEjY393WFSzEeWiwjkl/GFZDH84R1QcgM4sHjnLfaQY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=caZm1KAOF35LbhftrX+mXNtGpNMAjz6IPw5HpqiYSNMuWKp+LtdWC6/WyngW4XY+TAutoM8zzsg+O350J+/lzlCEtypule4EJbeVZ/Gqetb3IJs2oT/YCF+c5UlJOPqn+uFN7CBIrBunQsC5Mb1CNWDKjimkmaqAqlnOsTmXTA4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=tukaani.org; spf=pass smtp.mailfrom=tukaani.org; arc=none smtp.client-ip=84.34.166.10 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=tukaani.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=tukaani.org Received: from www25.zoner.fi (www25.zoner.fi [84.34.147.45]) by mailscanner01.zoner.fi (Postfix) with ESMTPS id B7686402D4; Wed, 20 Mar 2024 20:39:26 +0200 (EET) Received: from mail.zoner.fi ([84.34.147.244]) by www25.zoner.fi with esmtp (Exim 4.96.1-7-g79877b70e) (envelope-from ) id 1rn0qU-0001dW-1Z; Wed, 20 Mar 2024 20:39:26 +0200 From: Lasse Collin To: Andrew Morton Cc: Lasse Collin , Jia Tan , linux-kernel@vger.kernel.org Subject: [PATCH 07/11] xz: Optimize for-loop conditions in the BCJ decoders Date: Wed, 20 Mar 2024 20:38:40 +0200 Message-ID: <20240320183846.19475-8-lasse.collin@tukaani.org> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240320183846.19475-1-lasse.collin@tukaani.org> References: <20240320183846.19475-1-lasse.collin@tukaani.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Compilers cannot optimize the addition "i + 4" away since theoretically it could overflow. Reviewed-by: Jia Tan Signed-off-by: Lasse Collin --- lib/xz/xz_dec_bcj.c | 23 ++++++++++++++++++----- 1 file changed, 18 insertions(+), 5 deletions(-) diff --git a/lib/xz/xz_dec_bcj.c b/lib/xz/xz_dec_bcj.c index ab9237ed6db8..e0b4bf4999c0 100644 --- a/lib/xz/xz_dec_bcj.c +++ b/lib/xz/xz_dec_bcj.c @@ -161,7 +161,9 @@ static size_t bcj_powerpc(struct xz_dec_bcj *s, uint8_t= *buf, size_t size) size_t i; uint32_t instr; =20 - for (i =3D 0; i + 4 <=3D size; i +=3D 4) { + size &=3D ~(size_t)3; + + for (i =3D 0; i < size; i +=3D 4) { instr =3D get_unaligned_be32(buf + i); if ((instr & 0xFC000003) =3D=3D 0x48000001) { instr &=3D 0x03FFFFFC; @@ -218,7 +220,9 @@ static size_t bcj_ia64(struct xz_dec_bcj *s, uint8_t *b= uf, size_t size) /* Instruction normalized with bit_res for easier manipulation */ uint64_t norm; =20 - for (i =3D 0; i + 16 <=3D size; i +=3D 16) { + size &=3D ~(size_t)15; + + for (i =3D 0; i < size; i +=3D 16) { mask =3D branch_table[buf[i] & 0x1F]; for (slot =3D 0, bit_pos =3D 5; slot < 3; ++slot, bit_pos +=3D 41) { if (((mask >> slot) & 1) =3D=3D 0) @@ -266,7 +270,9 @@ static size_t bcj_arm(struct xz_dec_bcj *s, uint8_t *bu= f, size_t size) size_t i; uint32_t addr; =20 - for (i =3D 0; i + 4 <=3D size; i +=3D 4) { + size &=3D ~(size_t)3; + + for (i =3D 0; i < size; i +=3D 4) { if (buf[i + 3] =3D=3D 0xEB) { addr =3D (uint32_t)buf[i] | ((uint32_t)buf[i + 1] << 8) | ((uint32_t)buf[i + 2] << 16); @@ -289,7 +295,12 @@ static size_t bcj_armthumb(struct xz_dec_bcj *s, uint8= _t *buf, size_t size) size_t i; uint32_t addr; =20 - for (i =3D 0; i + 4 <=3D size; i +=3D 2) { + if (size < 4) + return 0; + + size -=3D 4; + + for (i =3D 0; i <=3D size; i +=3D 2) { if ((buf[i + 1] & 0xF8) =3D=3D 0xF0 && (buf[i + 3] & 0xF8) =3D=3D 0xF8) { addr =3D (((uint32_t)buf[i + 1] & 0x07) << 19) @@ -317,7 +328,9 @@ static size_t bcj_sparc(struct xz_dec_bcj *s, uint8_t *= buf, size_t size) size_t i; uint32_t instr; =20 - for (i =3D 0; i + 4 <=3D size; i +=3D 4) { + size &=3D ~(size_t)3; + + for (i =3D 0; i < size; i +=3D 4) { instr =3D get_unaligned_be32(buf + i); if ((instr >> 22) =3D=3D 0x100 || (instr >> 22) =3D=3D 0x1FF) { instr <<=3D 2; --=20 2.44.0 From nobody Sun Feb 8 17:41:56 2026 Received: from mailscanner09.zoner.fi (mailscanner09.zoner.fi [5.44.246.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6DCD485623 for ; Wed, 20 Mar 2024 18:49:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=5.44.246.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710960554; cv=none; b=SgDwIC4JT2jqbnIV5Eu/BykbzQHfqds9HnknT2oTZ8bkRsG/GvbWSXVfeXhkvrYrHKBn2LSDKbeHy2W1/uR5L4RW/ZWIKC9lrGjQ55fPpwATGhmYA5cXo9ASbHdwEnNBpSw8T6Mb9cjNrxTMkOuilgAPXOBFX1b3bORBoApyvNo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710960554; c=relaxed/simple; bh=ByoVqAuRdD1oD3TBj0N5QxLkM+fd/TEn0WNYkz4c3B0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=K3/lONLuUwx12vz/L/Om0skqGmAf9ZnmLwWZOMw9bUaw1v0SKcMM42GFoEsmAwMhBifIqhN220fp9eE4ozELF5/RZgkHYPpop/qVEAKsXTJNIuJWLk6xUxlrx20fmSmSpBMeVjTvFPklifyOe57uDQ8SZQ4sbT1dUsFYlK9QywU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=tukaani.org; spf=pass smtp.mailfrom=tukaani.org; arc=none smtp.client-ip=5.44.246.18 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=tukaani.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=tukaani.org Received: from www25.zoner.fi (www25.zoner.fi [84.34.147.45]) by mailscanner09.zoner.fi (Postfix) with ESMTPS id CC5652108E; Wed, 20 Mar 2024 20:39:26 +0200 (EET) Received: from mail.zoner.fi ([84.34.147.244]) by www25.zoner.fi with esmtp (Exim 4.96.1-7-g79877b70e) (envelope-from ) id 1rn0qU-0001dW-23; Wed, 20 Mar 2024 20:39:26 +0200 From: Lasse Collin To: Andrew Morton Cc: Lasse Collin , Jia Tan , linux-kernel@vger.kernel.org Subject: [PATCH 08/11] xz: Add ARM64 BCJ filter Date: Wed, 20 Mar 2024 20:38:41 +0200 Message-ID: <20240320183846.19475-9-lasse.collin@tukaani.org> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240320183846.19475-1-lasse.collin@tukaani.org> References: <20240320183846.19475-1-lasse.collin@tukaani.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Also omit a duplicated check for XZ_DEC_ARM in xz_private.h. This filter can be used by Squashfs without modifications to the Squashfs kernel code (only needs support in userspace Squashfs-tools). Reviewed-by: Jia Tan Signed-off-by: Lasse Collin --- Notes: Compared to the first patch I submitted on 2023-11-08 (see ), this has a minor tweak to make the for-loop condition faster. =20 Squashfs-tools Git repository already has support for creating file systems that use the ARM64 filter. lib/xz/Kconfig | 5 +++++ lib/xz/xz_dec_bcj.c | 52 ++++++++++++++++++++++++++++++++++++++++++++- lib/xz/xz_private.h | 7 ++++-- 3 files changed, 61 insertions(+), 3 deletions(-) diff --git a/lib/xz/Kconfig b/lib/xz/Kconfig index 6b80453d8f54..1166627a87dc 100644 --- a/lib/xz/Kconfig +++ b/lib/xz/Kconfig @@ -30,6 +30,11 @@ config XZ_DEC_ARMTHUMB default y select XZ_DEC_BCJ =20 +config XZ_DEC_ARM64 + bool "ARM64 BCJ filter decoder" if EXPERT + default y + select XZ_DEC_BCJ + config XZ_DEC_SPARC bool "SPARC BCJ filter decoder" if EXPERT default y diff --git a/lib/xz/xz_dec_bcj.c b/lib/xz/xz_dec_bcj.c index e0b4bf4999c0..941198a8a55b 100644 --- a/lib/xz/xz_dec_bcj.c +++ b/lib/xz/xz_dec_bcj.c @@ -23,7 +23,8 @@ struct xz_dec_bcj { BCJ_IA64 =3D 6, /* Big or little endian */ BCJ_ARM =3D 7, /* Little endian only */ BCJ_ARMTHUMB =3D 8, /* Little endian only */ - BCJ_SPARC =3D 9 /* Big or little endian */ + BCJ_SPARC =3D 9, /* Big or little endian */ + BCJ_ARM64 =3D 10 /* AArch64 */ } type; =20 /* @@ -346,6 +347,47 @@ static size_t bcj_sparc(struct xz_dec_bcj *s, uint8_t = *buf, size_t size) } #endif =20 +#ifdef XZ_DEC_ARM64 +static size_t bcj_arm64(struct xz_dec_bcj *s, uint8_t *buf, size_t size) +{ + size_t i; + uint32_t instr; + uint32_t addr; + + size &=3D ~(size_t)3; + + for (i =3D 0; i < size; i +=3D 4) { + instr =3D get_unaligned_le32(buf + i); + + if ((instr >> 26) =3D=3D 0x25) { + /* BL instruction */ + addr =3D instr - ((s->pos + (uint32_t)i) >> 2); + instr =3D 0x94000000 | (addr & 0x03FFFFFF); + put_unaligned_le32(instr, buf + i); + + } else if ((instr & 0x9F000000) =3D=3D 0x90000000) { + /* ADRP instruction */ + addr =3D ((instr >> 29) & 3) | ((instr >> 3) & 0x1FFFFC); + + /* Only convert values in the range +/-512 MiB. */ + if ((addr + 0x020000) & 0x1C0000) + continue; + + addr -=3D (s->pos + (uint32_t)i) >> 12; + + instr &=3D 0x9000001F; + instr |=3D (addr & 3) << 29; + instr |=3D (addr & 0x03FFFC) << 3; + instr |=3D (0U - (addr & 0x020000)) & 0xE00000; + + put_unaligned_le32(instr, buf + i); + } + } + + return i; +} +#endif + /* * Apply the selected BCJ filter. Update *pos and s->pos to match the amou= nt * of data that got filtered. @@ -392,6 +434,11 @@ static void bcj_apply(struct xz_dec_bcj *s, case BCJ_SPARC: filtered =3D bcj_sparc(s, buf, size); break; +#endif +#ifdef XZ_DEC_ARM64 + case BCJ_ARM64: + filtered =3D bcj_arm64(s, buf, size); + break; #endif default: /* Never reached but silence compiler warnings. */ @@ -565,6 +612,9 @@ XZ_EXTERN enum xz_ret xz_dec_bcj_reset(struct xz_dec_bc= j *s, uint8_t id) #endif #ifdef XZ_DEC_SPARC case BCJ_SPARC: +#endif +#ifdef XZ_DEC_ARM64 + case BCJ_ARM64: #endif break; =20 diff --git a/lib/xz/xz_private.h b/lib/xz/xz_private.h index 811add814ae4..307e0de8c260 100644 --- a/lib/xz/xz_private.h +++ b/lib/xz/xz_private.h @@ -36,6 +36,9 @@ # ifdef CONFIG_XZ_DEC_SPARC # define XZ_DEC_SPARC # endif +# ifdef CONFIG_XZ_DEC_ARM64 +# define XZ_DEC_ARM64 +# endif # ifdef CONFIG_XZ_DEC_MICROLZMA # define XZ_DEC_MICROLZMA # endif @@ -97,9 +100,9 @@ */ #ifndef XZ_DEC_BCJ # if defined(XZ_DEC_X86) || defined(XZ_DEC_POWERPC) \ - || defined(XZ_DEC_IA64) || defined(XZ_DEC_ARM) \ + || defined(XZ_DEC_IA64) \ || defined(XZ_DEC_ARM) || defined(XZ_DEC_ARMTHUMB) \ - || defined(XZ_DEC_SPARC) + || defined(XZ_DEC_SPARC) || defined(XZ_DEC_ARM64) # define XZ_DEC_BCJ # endif #endif --=20 2.44.0 From nobody Sun Feb 8 17:41:56 2026 Received: from mailscanner01.zoner.fi (mailscanner01.zoner.fi [84.34.166.10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C9A3C6A8A6 for ; Wed, 20 Mar 2024 18:39:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=84.34.166.10 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710959979; cv=none; b=lpPQDWlbB0XW4OwEZBa3OzSMDbKatceJJYQOK7JX/W+S36Jx1G2ezUgnDs32flLl+LD+pk8v2bLmKLCxDSpG3LMsZ3bvboEtbuz1t2zLd9hV9/bbpsEY1z52oiWXH1GZHyhOWbdzLkqXW4J9znLLR1yWvIxxVqBFkDy21VIslDI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710959979; c=relaxed/simple; bh=aVTXMepemamYIgEqhcZaCqWZdKxJee31zHLwpcDfTPM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Onkh/z4DLnXcxtrZnsYnVWCw5NocFF+gWdmLOtU1MNbgZbknOEEh3oPk1tzCXCm2Yzd57GZihoeKizWCcQFjW4jQq3R5c8iW2is9RZtKLoRA/P/KCfbbI2oSGMKZnvDmJTlZI462KCnUiR+MBHKZRSO0BPZb5IW9FPjWMh7XLLo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=tukaani.org; spf=pass smtp.mailfrom=tukaani.org; arc=none smtp.client-ip=84.34.166.10 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=tukaani.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=tukaani.org Received: from www25.zoner.fi (www25.zoner.fi [84.34.147.45]) by mailscanner01.zoner.fi (Postfix) with ESMTPS id E55284254A; Wed, 20 Mar 2024 20:39:26 +0200 (EET) Received: from mail.zoner.fi ([84.34.147.244]) by www25.zoner.fi with esmtp (Exim 4.96.1-7-g79877b70e) (envelope-from ) id 1rn0qU-0001dW-2N; Wed, 20 Mar 2024 20:39:26 +0200 From: Lasse Collin To: Andrew Morton Cc: Lasse Collin , Jia Tan , linux-kernel@vger.kernel.org Subject: [PATCH 09/11] xz: Add RISC-V BCJ filter Date: Wed, 20 Mar 2024 20:38:42 +0200 Message-ID: <20240320183846.19475-10-lasse.collin@tukaani.org> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240320183846.19475-1-lasse.collin@tukaani.org> References: <20240320183846.19475-1-lasse.collin@tukaani.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This filter can be used by Squashfs without modifications to the Squashfs kernel code (only needs support in userspace Squashfs-tools). Co-developed-by: Jia Tan Signed-off-by: Jia Tan Signed-off-by: Lasse Collin --- lib/xz/Kconfig | 5 +++ lib/xz/xz_dec_bcj.c | 105 ++++++++++++++++++++++++++++++++++++++++++- lib/xz/xz_dec_syms.c | 4 +- lib/xz/xz_private.h | 6 ++- 4 files changed, 116 insertions(+), 4 deletions(-) diff --git a/lib/xz/Kconfig b/lib/xz/Kconfig index 1166627a87dc..20aa459bfb3e 100644 --- a/lib/xz/Kconfig +++ b/lib/xz/Kconfig @@ -40,6 +40,11 @@ config XZ_DEC_SPARC default y select XZ_DEC_BCJ =20 +config XZ_DEC_RISCV + bool "RISC-V BCJ filter decoder" if EXPERT + default y + select XZ_DEC_BCJ + config XZ_DEC_MICROLZMA bool "MicroLZMA decoder" default n diff --git a/lib/xz/xz_dec_bcj.c b/lib/xz/xz_dec_bcj.c index 941198a8a55b..c04767d357bd 100644 --- a/lib/xz/xz_dec_bcj.c +++ b/lib/xz/xz_dec_bcj.c @@ -5,6 +5,7 @@ * * Authors: Lasse Collin * Igor Pavlov + * Jia Tan */ =20 #include "xz_private.h" @@ -24,7 +25,8 @@ struct xz_dec_bcj { BCJ_ARM =3D 7, /* Little endian only */ BCJ_ARMTHUMB =3D 8, /* Little endian only */ BCJ_SPARC =3D 9, /* Big or little endian */ - BCJ_ARM64 =3D 10 /* AArch64 */ + BCJ_ARM64 =3D 10, /* AArch64 */ + BCJ_RISCV =3D 11 /* RV32GQC_Zfh, RV64GQC_Zfh */ } type; =20 /* @@ -388,6 +390,99 @@ static size_t bcj_arm64(struct xz_dec_bcj *s, uint8_t = *buf, size_t size) } #endif =20 +#ifdef XZ_DEC_RISCV +static size_t bcj_riscv(struct xz_dec_bcj *s, uint8_t *buf, size_t size) +{ + size_t i; + uint32_t b1; + uint32_t b2; + uint32_t b3; + uint32_t instr; + uint32_t instr2; + uint32_t instr2_rs1; + uint32_t addr; + + if (size < 8) + return 0; + + size -=3D 8; + + for (i =3D 0; i <=3D size; i +=3D 2) { + instr =3D buf[i]; + + if (instr =3D=3D 0xEF) { + /* JAL */ + b1 =3D buf[i + 1]; + if ((b1 & 0x0D) !=3D 0) + continue; + + b2 =3D buf[i + 2]; + b3 =3D buf[i + 3]; + + addr =3D ((b1 & 0xF0) << 13) | (b2 << 9) | (b3 << 1); + addr -=3D s->pos + (uint32_t)i; + + buf[i + 1] =3D (uint8_t)((b1 & 0x0F) + | ((addr >> 8) & 0xF0)); + + buf[i + 2] =3D (uint8_t)(((addr >> 16) & 0x0F) + | ((addr >> 7) & 0x10) + | ((addr << 4) & 0xE0)); + + buf[i + 3] =3D (uint8_t)(((addr >> 4) & 0x7F) + | ((addr >> 13) & 0x80)); + + i +=3D 4 - 2; + + } else if ((instr & 0x7F) =3D=3D 0x17) { + /* AUIPC */ + instr |=3D (uint32_t)buf[i + 1] << 8; + instr |=3D (uint32_t)buf[i + 2] << 16; + instr |=3D (uint32_t)buf[i + 3] << 24; + + if (instr & 0xE80) { + /* AUIPC's rd doesn't equal x0 or x2. */ + instr2 =3D get_unaligned_le32(buf + i + 4); + + if (((instr << 8) ^ (instr2 - 3)) & 0xF8003) { + i +=3D 6 - 2; + continue; + } + + addr =3D (instr & 0xFFFFF000) + (instr2 >> 20); + + instr =3D 0x17 | (2 << 7) | (instr2 << 12); + instr2 =3D addr; + } else { + /* AUIPC's rd equals x0 or x2. */ + instr2_rs1 =3D instr >> 27; + + if ((uint32_t)((instr - 0x3117) << 18) + >=3D (instr2_rs1 & 0x1D)) { + i +=3D 4 - 2; + continue; + } + + addr =3D get_unaligned_be32(buf + i + 4); + addr -=3D s->pos + (uint32_t)i; + + instr2 =3D (instr >> 12) | (addr << 20); + + instr =3D 0x17 | (instr2_rs1 << 7) + | ((addr + 0x800) & 0xFFFFF000); + } + + put_unaligned_le32(instr, buf + i); + put_unaligned_le32(instr2, buf + i + 4); + + i +=3D 8 - 2; + } + } + + return i; +} +#endif + /* * Apply the selected BCJ filter. Update *pos and s->pos to match the amou= nt * of data that got filtered. @@ -439,6 +534,11 @@ static void bcj_apply(struct xz_dec_bcj *s, case BCJ_ARM64: filtered =3D bcj_arm64(s, buf, size); break; +#endif +#ifdef XZ_DEC_RISCV + case BCJ_RISCV: + filtered =3D bcj_riscv(s, buf, size); + break; #endif default: /* Never reached but silence compiler warnings. */ @@ -615,6 +715,9 @@ XZ_EXTERN enum xz_ret xz_dec_bcj_reset(struct xz_dec_bc= j *s, uint8_t id) #endif #ifdef XZ_DEC_ARM64 case BCJ_ARM64: +#endif +#ifdef XZ_DEC_RISCV + case BCJ_RISCV: #endif break; =20 diff --git a/lib/xz/xz_dec_syms.c b/lib/xz/xz_dec_syms.c index 495d2cc2e6e8..8f4925af9737 100644 --- a/lib/xz/xz_dec_syms.c +++ b/lib/xz/xz_dec_syms.c @@ -22,6 +22,6 @@ EXPORT_SYMBOL(xz_dec_microlzma_end); #endif =20 MODULE_DESCRIPTION("XZ decompressor"); -MODULE_VERSION("1.1"); -MODULE_AUTHOR("Lasse Collin and Igor Pavlov"); +MODULE_VERSION("1.2"); +MODULE_AUTHOR("Lasse Collin, Igor Pavlov, Jia Tan"); MODULE_LICENSE("Dual BSD/GPL"); diff --git a/lib/xz/xz_private.h b/lib/xz/xz_private.h index 307e0de8c260..a8b1cbe8d21d 100644 --- a/lib/xz/xz_private.h +++ b/lib/xz/xz_private.h @@ -39,6 +39,9 @@ # ifdef CONFIG_XZ_DEC_ARM64 # define XZ_DEC_ARM64 # endif +# ifdef CONFIG_XZ_DEC_RISCV +# define XZ_DEC_RISCV +# endif # ifdef CONFIG_XZ_DEC_MICROLZMA # define XZ_DEC_MICROLZMA # endif @@ -102,7 +105,8 @@ # if defined(XZ_DEC_X86) || defined(XZ_DEC_POWERPC) \ || defined(XZ_DEC_IA64) \ || defined(XZ_DEC_ARM) || defined(XZ_DEC_ARMTHUMB) \ - || defined(XZ_DEC_SPARC) || defined(XZ_DEC_ARM64) + || defined(XZ_DEC_SPARC) || defined(XZ_DEC_ARM64) \ + || defined(XZ_DEC_RISCV) # define XZ_DEC_BCJ # endif #endif --=20 2.44.0 From nobody Sun Feb 8 17:41:56 2026 Received: from mailscanner05.zoner.fi (mailscanner05.zoner.fi [5.44.246.14]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8073B8562D for ; Wed, 20 Mar 2024 19:17:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=5.44.246.14 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710962267; cv=none; b=ZHFrDkKEhiIMqh6OXzMn3idIqOdUdb5nmKMIOD5H2lgU5Eb8x3zHA3h7UaN6Hl8rMerul4Mllhzuccz8sB/t80BKdSIvI0Z/45R+6Pcjsw27DoTs+ppAo3moPcwfYG7fuuuAoC3tBhuUvNh+m9+afTSxdkkd+GMGOcK+cXl1gUo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710962267; c=relaxed/simple; bh=1STaSSKFT9aWSutIcaXf5t9tigM61VZiBi4b8crf1L0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ZmN9DiQpAin3kMEyS0OMvSXEaf+hGjkdFf910gbk0snFPzm/2gVtu8dqDn12wZem5PRY89UgKlo750rNSoh72mzGbMbcU3dRoEoZHoWJlO6FRPAtUEFO4d1KpaAIWIZ6DR21kRUcLI0SHE81QY8VOt06gH+9Cn2NMi/Bghjfcb4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=tukaani.org; spf=pass smtp.mailfrom=tukaani.org; arc=none smtp.client-ip=5.44.246.14 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=tukaani.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=tukaani.org Received: from www25.zoner.fi (www25.zoner.fi [84.34.147.45]) by mailscanner05.zoner.fi (Postfix) with ESMTPS id 4D5E3211A4; Wed, 20 Mar 2024 21:17:42 +0200 (EET) Received: from mail.zoner.fi ([84.34.147.244]) by www25.zoner.fi with esmtp (Exim 4.96.1-7-g79877b70e) (envelope-from ) id 1rn0qU-0001dW-2g; Wed, 20 Mar 2024 20:39:26 +0200 From: Lasse Collin To: Andrew Morton Cc: Lasse Collin , Jia Tan , linux-kernel@vger.kernel.org Subject: [PATCH 10/11] xz: Use 128 MiB dictionary and force single-threaded mode Date: Wed, 20 Mar 2024 20:38:43 +0200 Message-ID: <20240320183846.19475-11-lasse.collin@tukaani.org> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240320183846.19475-1-lasse.collin@tukaani.org> References: <20240320183846.19475-1-lasse.collin@tukaani.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This only affects kernel image compression, not any other xz usage. Desktop kernels on x86-64 are already around 60 MiB. Using a dictionary larger than 32 MiB should have no downsides nowadays as anyone building the kernel should have plenty of RAM. 128 MiB dictionary needs 1346 MiB of RAM with xz versions 5.0.x - 5.6.x in single-threaded mode. On archs that use xz_wrap.sh, kernel decompression is done in single-call mode so a larger dictionary doesn't affect boot-time memory requirements. xz >=3D 5.6.0 uses multithreaded mode by default which compresses slightly worse than single-threaded mode. Kernel compression rarely used more than one thread anyway because with 32 MiB dictionary size the default block size was 96 MiB in multithreaded mode. So only a single thread was used anyway unless the kernel was over 96 MiB. Comparison to CONFIG_KERNEL_LZMA: It uses "lzma -9" which mapped to 32 MiB dictionary in LZMA Utils 4.32.7 (the final release in 2008). Nowadays the lzma tool on most systems is from XZ Utils where -9 maps to 64 MiB dictionary. So using a 32 MiB dictionary with CONFIG_KERNEL_XZ may have compressed big kernels slightly worse than the old LZMA option. Comparison to CONFIG_KERNEL_ZSTD: zstd uses 128 MiB dictionary. Reviewed-by: Jia Tan Signed-off-by: Lasse Collin --- scripts/xz_wrap.sh | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/scripts/xz_wrap.sh b/scripts/xz_wrap.sh index bb760b721b2c..c8c36441ab70 100755 --- a/scripts/xz_wrap.sh +++ b/scripts/xz_wrap.sh @@ -16,4 +16,15 @@ case $SRCARCH in sparc) BCJ=3D--sparc ;; esac =20 -exec $XZ --check=3Dcrc32 $BCJ --lzma2=3D$LZMA2OPTS,dict=3D32MiB +# Use single-threaded mode because it compresses a little better +# (and uses less RAM) than multithreaded mode. +# +# For the best compression, the dictionary size shouldn't be +# smaller than the uncompressed kernel. 128 MiB dictionary +# needs less than 1400 MiB of RAM in single-threaded mode. +# +# On the archs that use this script to compress the kernel, +# decompression in the preboot code is done in single-call mode. +# Thus the dictionary size doesn't affect the memory requirements +# of the preboot decompressor at all. +exec $XZ --check=3Dcrc32 --threads=3D1 $BCJ --lzma2=3D$LZMA2OPTS,dict=3D12= 8MiB --=20 2.44.0 From nobody Sun Feb 8 17:41:56 2026 Received: from mailscanner01.zoner.fi (mailscanner01.zoner.fi [84.34.166.10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 11A8385297 for ; Wed, 20 Mar 2024 19:17:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=84.34.166.10 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710962266; cv=none; b=T89IkXLUAr1Sc6YpaNqFqDGa/DPM1zsQYpGjYTaZAAGs4BgxB/ToV+GuDTCU3RNS011KyQGwfzUdbI00gaSXIDxPbCSK5qUoY9B7wdpYPL0wFJAPFswh+/+UDVBaZRJt1H0eoBgMmCKj0VP1uOBz09GDPoJ+6ndj9qgWFbES4Ao= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710962266; c=relaxed/simple; bh=BduBwpnC+lnh0fNzh+quQgYOi+N1VvuKwdT8DqQQw6I=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=FRA8J70IYqyLLkGuMQvjYNXcdYrGcpCr/pF7WUcwlyXsBTB1AmRB3rneOBceF9F2rCKdSP7NIw3WicaW8v7U4ewcqI/+8pz5xD4oO3hwiFtAM8MhO2Ni6TnCbQa9nD+EyKyPCyUz8Sen1rcPfpEX9NIrLuF2uybdc/4ouc4sZyw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=tukaani.org; spf=pass smtp.mailfrom=tukaani.org; arc=none smtp.client-ip=84.34.166.10 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=tukaani.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=tukaani.org Received: from www25.zoner.fi (www25.zoner.fi [84.34.147.45]) by mailscanner01.zoner.fi (Postfix) with ESMTPS id 2E1EA4220B; Wed, 20 Mar 2024 21:17:42 +0200 (EET) Received: from mail.zoner.fi ([84.34.147.244]) by www25.zoner.fi with esmtp (Exim 4.96.1-7-g79877b70e) (envelope-from ) id 1rn0qU-0001dW-2z; Wed, 20 Mar 2024 20:39:26 +0200 From: Lasse Collin To: Andrew Morton Cc: Lasse Collin , Jia Tan , Jubin Zhong , Jules Maselbas , linux-kernel@vger.kernel.org Subject: [PATCH 11/11] xz: Adjust arch-specific options for better kernel compression Date: Wed, 20 Mar 2024 20:38:44 +0200 Message-ID: <20240320183846.19475-12-lasse.collin@tukaani.org> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240320183846.19475-1-lasse.collin@tukaani.org> References: <20240320183846.19475-1-lasse.collin@tukaani.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Use LZMA2 options that match the arch-specific alignment of instructions. This change reduces compressed kernel size 0-2 % depending on the arch. On 1-byte-aligned x86 it makes no difference and on 4-byte-aligned archs it helps the most. Use the ARM-Thumb filter for ARM-Thumb2 kernels. This reduces compressed kernel size about 5 %.[1] Previously such kernels were compressed using the ARM filter which didn't do anything useful with ARM-Thumb2 code. Add BCJ filter support for ARM64 and RISC-V. On ARM64 the compressed kernel size is reduced about 5 % and on RISC-V by 7-8 % compared to unfiltered XZ or plain LZMA. However: - arch/arm64/boot/Makefile and arch/riscv/boot/Makefile don't include the build rule (two lines) for XZ support even though they support six other compressors. It would be trivial to add the rule but boot loaders would need XZ support too. - A new enough version of the xz tool is required: 5.4.0 for ARM64 and 5.6.0 for RISC-V. With an old xz version a message is printed to standard error and the kernel is compressed without the filter. Update lib/decompress_unxz.c to match the changes to xz_wrap.sh. Update the CONFIG_KERNEL_XZ help text in init/Kconfig: - Add the RISC-V and ARM64 filters. - Clarify that the PowerPC filter is for big endian only. - Omit IA-64. Link: https://lore.kernel.org/lkml/1637379771-39449-1-git-send-email-zhongj= ubin@huawei.com/ [1] Reviewed-by: Jia Tan Signed-off-by: Lasse Collin --- init/Kconfig | 5 +- lib/decompress_unxz.c | 14 ++++- scripts/xz_wrap.sh | 141 ++++++++++++++++++++++++++++++++++++++++-- 3 files changed, 151 insertions(+), 9 deletions(-) diff --git a/init/Kconfig b/init/Kconfig index f3ea5dea9c85..785e15aa5395 100644 --- a/init/Kconfig +++ b/init/Kconfig @@ -308,8 +308,9 @@ config KERNEL_XZ BCJ filters which can improve compression ratio of executable code. The size of the kernel is about 30% smaller with XZ in comparison to gzip. On architectures for which there is a BCJ - filter (i386, x86_64, ARM, IA-64, PowerPC, and SPARC), XZ - will create a few percent smaller kernel than plain LZMA. + filter (i386, x86_64, ARM, ARM64, RISC-V, big endian PowerPC, + and SPARC), XZ will create a few percent smaller kernel than + plain LZMA. =20 The speed is about the same as with LZMA: The decompression speed of XZ is better than that of bzip2 but worse than gzip diff --git a/lib/decompress_unxz.c b/lib/decompress_unxz.c index 46aa3be13fc5..cae00395d7a6 100644 --- a/lib/decompress_unxz.c +++ b/lib/decompress_unxz.c @@ -126,11 +126,21 @@ #ifdef CONFIG_X86 # define XZ_DEC_X86 #endif -#ifdef CONFIG_PPC +#if defined(CONFIG_PPC) && defined(CONFIG_CPU_BIG_ENDIAN) # define XZ_DEC_POWERPC #endif #ifdef CONFIG_ARM -# define XZ_DEC_ARM +# ifdef CONFIG_THUMB2_KERNEL +# define XZ_DEC_ARMTHUMB +# else +# define XZ_DEC_ARM +# endif +#endif +#ifdef CONFIG_ARM64 +# define XZ_DEC_ARM64 +#endif +#ifdef CONFIG_RISCV +# define XZ_DEC_RISCV #endif #ifdef CONFIG_SPARC # define XZ_DEC_SPARC diff --git a/scripts/xz_wrap.sh b/scripts/xz_wrap.sh index c8c36441ab70..5bdf0c35cc85 100755 --- a/scripts/xz_wrap.sh +++ b/scripts/xz_wrap.sh @@ -6,14 +6,145 @@ # # Author: Lasse Collin =20 +# This has specialized settings for the following archs. However, +# XZ-compressed kernel isn't currently supported on every listed arch. +# +# Arch Align Notes +# arm 2/4 ARM and ARM-Thumb2 +# arm64 4 +# csky 2 +# loongarch 4 +# mips 2/4 MicroMIPS is 2-byte aligned +# parisc 4 +# powerpc 4 Uses its own wrapper for compressors instead of th= is. +# riscv 2/4 +# s390 2 +# sh 2 +# sparc 4 +# x86 1 + +# A few archs use 2-byte or 4-byte aligned instructions depending on +# the kernel config. This function is used to check if the relevant +# config option is set to "y". +is_enabled() +{ + grep -q "^$1=3Dy$" include/config/auto.conf +} + +# Set XZ_VERSION (and LIBLZMA_VERSION). This is needed to disable features +# that aren't available in old XZ Utils versions. +eval "$($XZ --robot --version)" || exit + +# Assume that no BCJ filter is available. BCJ=3D -LZMA2OPTS=3D =20 +# Set the instruction alignment to 1, 2, or 4 bytes. +# +# Set the BCJ filter if one is available. +# It must match the #ifdef usage in lib/decompress_unxz.c. case $SRCARCH in - x86) BCJ=3D--x86 ;; - powerpc) BCJ=3D--powerpc ;; - arm) BCJ=3D--arm ;; - sparc) BCJ=3D--sparc ;; + arm) + if is_enabled CONFIG_THUMB2_KERNEL; then + ALIGN=3D2 + BCJ=3D--armthumb + else + ALIGN=3D4 + BCJ=3D--arm + fi + ;; + + arm64) + ALIGN=3D4 + + # ARM64 filter was added in XZ Utils 5.4.0. + if [ "$XZ_VERSION" -ge 50040002 ]; then + BCJ=3D--arm64 + else + echo "$0: Upgrading to xz >=3D 5.4.0" \ + "would enable the ARM64 filter" \ + "for better compression" >&2 + fi + ;; + + csky) + ALIGN=3D2 + ;; + + loongarch) + ALIGN=3D4 + ;; + + mips) + if is_enabled CONFIG_CPU_MICROMIPS; then + ALIGN=3D2 + else + ALIGN=3D4 + fi + ;; + + parisc) + ALIGN=3D4 + ;; + + powerpc) + ALIGN=3D4 + + # The filter is only for big endian instruction encoding. + if is_enabled CONFIG_CPU_BIG_ENDIAN; then + BCJ=3D--powerpc + fi + ;; + + riscv) + if is_enabled CONFIG_RISCV_ISA_C; then + ALIGN=3D2 + else + ALIGN=3D4 + fi + + # RISC-V filter was added in XZ Utils 5.6.0. + if [ "$XZ_VERSION" -ge 50060002 ]; then + BCJ=3D--riscv + else + echo "$0: Upgrading to xz >=3D 5.6.0" \ + "would enable the RISC-V filter" \ + "for better compression" >&2 + fi + ;; + + s390) + ALIGN=3D2 + ;; + + sh) + ALIGN=3D2 + ;; + + sparc) + ALIGN=3D4 + BCJ=3D--sparc + ;; + + x86) + ALIGN=3D1 + BCJ=3D--x86 + ;; + + *) + echo "$0: Arch-specific tuning is missing for '$SRCARCH'" >&2 + + # Guess 2-byte-aligned instructions. Guessing too low + # should hurt less than guessing too high. + ALIGN=3D2 + ;; +esac + +# Select the LZMA2 options matching the instruction alignment. +case $ALIGN in + 1) LZMA2OPTS=3D ;; + 2) LZMA2OPTS=3Dlp=3D1 ;; + 4) LZMA2OPTS=3Dlp=3D2,lc=3D2 ;; + *) echo "$0: ALIGN wrong or missing" >&2; exit 1 ;; esac =20 # Use single-threaded mode because it compresses a little better --=20 2.44.0