[v2] kbuild: switch from lz4c to lz4 for compression

[PATCH v2] kbuild: switch from lz4c to lz4 for compression

Posted by Parth Pancholi 1 week, 1 day ago

From: Parth Pancholi <parth.pancholi@toradex.com>

Replace lz4c with lz4 for kernel image compression.
Although lz4 and lz4c are functionally similar, lz4c has been deprecated
upstream since 2018. Since as early as Ubuntu 16.04 and Fedora 25, lz4
and lz4c have been packaged together, making it safe to update the
requirement from lz4c to lz4.

Consequently, some distributions and build systems, such as OpenEmbedded,
have fully transitioned to using lz4. OpenEmbedded core adopted this
change in commit fe167e082cbd ("bitbake.conf: require lz4 instead of
lz4c"), causing compatibility issues when building the mainline kernel
in the latest OpenEmbedded environment, as seen in the errors below.

This change also updates the LZ4 compression commands to make it backward
compatible by replacing stdin and stdout with the '-' option, due to some
unclear reason, the stdout keyword does not work for lz4 and '-' works for
both. In addition, this modifies the legacy '-c1' with '-9' which is also
compatible with both. This fixes the mainline kernel build failures with
the latest master OpenEmbedded builds associated with the mentioned
compatibility issues.

LZ4     arch/arm/boot/compressed/piggy_data
/bin/sh: 1: lz4c: not found
...
...
ERROR: oe_runmake failed

Cc: stable@vger.kernel.org
Link: https://github.com/lz4/lz4/pull/553
Suggested-by: Francesco Dolcini <francesco.dolcini@toradex.com>
Signed-off-by: Parth Pancholi <parth.pancholi@toradex.com>
---
v2: correct the compression command line to make it compatible with lz4
v1: https://lore.kernel.org/all/20241112150006.265900-1-parth105105@gmail.com/
---
 Makefile             | 2 +-
 scripts/Makefile.lib | 4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/Makefile b/Makefile
index 79192a3024bf..7630f763f5b2 100644
--- a/Makefile
+++ b/Makefile
@@ -508,7 +508,7 @@ KGZIP		= gzip
 KBZIP2		= bzip2
 KLZOP		= lzop
 LZMA		= lzma
-LZ4		= lz4c
+LZ4		= lz4
 XZ		= xz
 ZSTD		= zstd
 
diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib
index 01a9f567d5af..fe5e132fcea8 100644
--- a/scripts/Makefile.lib
+++ b/scripts/Makefile.lib
@@ -371,10 +371,10 @@ quiet_cmd_lzo_with_size = LZO     $@
       cmd_lzo_with_size = { cat $(real-prereqs) | $(KLZOP) -9; $(size_append); } > $@
 
 quiet_cmd_lz4 = LZ4     $@
-      cmd_lz4 = cat $(real-prereqs) | $(LZ4) -l -c1 stdin stdout > $@
+      cmd_lz4 = cat $(real-prereqs) | $(LZ4) -l -9 - - > $@
 
 quiet_cmd_lz4_with_size = LZ4     $@
-      cmd_lz4_with_size = { cat $(real-prereqs) | $(LZ4) -l -c1 stdin stdout; \
+      cmd_lz4_with_size = { cat $(real-prereqs) | $(LZ4) -l -9 - -; \
                   $(size_append); } > $@
 
 # U-Boot mkimage
-- 
2.34.1

Re: [PATCH v2] kbuild: switch from lz4c to lz4 for compression

Posted by Greg KH 1 week, 1 day ago

On Thu, Nov 14, 2024 at 03:56:44PM +0100, Parth Pancholi wrote:
> From: Parth Pancholi <parth.pancholi@toradex.com>
> 
> Replace lz4c with lz4 for kernel image compression.
> Although lz4 and lz4c are functionally similar, lz4c has been deprecated
> upstream since 2018. Since as early as Ubuntu 16.04 and Fedora 25, lz4
> and lz4c have been packaged together, making it safe to update the
> requirement from lz4c to lz4.
> 
> Consequently, some distributions and build systems, such as OpenEmbedded,
> have fully transitioned to using lz4. OpenEmbedded core adopted this
> change in commit fe167e082cbd ("bitbake.conf: require lz4 instead of
> lz4c"), causing compatibility issues when building the mainline kernel
> in the latest OpenEmbedded environment, as seen in the errors below.
> 
> This change also updates the LZ4 compression commands to make it backward
> compatible by replacing stdin and stdout with the '-' option, due to some
> unclear reason, the stdout keyword does not work for lz4 and '-' works for
> both. In addition, this modifies the legacy '-c1' with '-9' which is also
> compatible with both. This fixes the mainline kernel build failures with
> the latest master OpenEmbedded builds associated with the mentioned
> compatibility issues.
> 
> LZ4     arch/arm/boot/compressed/piggy_data
> /bin/sh: 1: lz4c: not found
> ...
> ...
> ERROR: oe_runmake failed
> 
> Cc: stable@vger.kernel.org

What bug does this resolve that it needs to be backported to stable
kernels?

thanks,

greg k-h

Re: [PATCH v2] kbuild: switch from lz4c to lz4 for compression

Posted by Francesco Dolcini 1 week ago

Hello Greg,

On Thu, Nov 14, 2024 at 05:02:01PM +0100, Greg KH wrote:
> On Thu, Nov 14, 2024 at 03:56:44PM +0100, Parth Pancholi wrote:
> > From: Parth Pancholi <parth.pancholi@toradex.com>
> > 
> > Replace lz4c with lz4 for kernel image compression.
> > Although lz4 and lz4c are functionally similar, lz4c has been deprecated
> > upstream since 2018. Since as early as Ubuntu 16.04 and Fedora 25, lz4
> > and lz4c have been packaged together, making it safe to update the
> > requirement from lz4c to lz4.
> > 
> > Consequently, some distributions and build systems, such as OpenEmbedded,
> > have fully transitioned to using lz4. OpenEmbedded core adopted this
> > change in commit fe167e082cbd ("bitbake.conf: require lz4 instead of
> > lz4c"), causing compatibility issues when building the mainline kernel
> > in the latest OpenEmbedded environment, as seen in the errors below.
> > 
> > This change also updates the LZ4 compression commands to make it backward
> > compatible by replacing stdin and stdout with the '-' option, due to some
> > unclear reason, the stdout keyword does not work for lz4 and '-' works for
> > both. In addition, this modifies the legacy '-c1' with '-9' which is also
> > compatible with both. This fixes the mainline kernel build failures with
> > the latest master OpenEmbedded builds associated with the mentioned
> > compatibility issues.
> > 
> > LZ4     arch/arm/boot/compressed/piggy_data
> > /bin/sh: 1: lz4c: not found
> > ...
> > ...
> > ERROR: oe_runmake failed
> > 
> > Cc: stable@vger.kernel.org
> 
> What bug does this resolve that it needs to be backported to stable
> kernels?

This is not solving any existing actual bug, and therefore there is no
fixes tag.

The issue here is that the kernel build system is using lz4c, that is
deprecated since 2018, and now distributions are actively moving away from it. 

openSUSE Tumbleweed and OE already removed it, so you would not be able
to compile a stable kernel on such distribution when using lz4 unless we
backport such a patch.

Everything should be properly documented in the commit message already.

My understanding is that something like that would be a reason for
backporting to stable, if my understanding is not correct we'll remove
the cc:stable and send a v3.

Francesco

Re: [PATCH v2] kbuild: switch from lz4c to lz4 for compression

Posted by Greg KH 1 week ago

On Fri, Nov 15, 2024 at 09:39:40AM +0100, Francesco Dolcini wrote:
> Hello Greg,
> 
> On Thu, Nov 14, 2024 at 05:02:01PM +0100, Greg KH wrote:
> > On Thu, Nov 14, 2024 at 03:56:44PM +0100, Parth Pancholi wrote:
> > > From: Parth Pancholi <parth.pancholi@toradex.com>
> > > 
> > > Replace lz4c with lz4 for kernel image compression.
> > > Although lz4 and lz4c are functionally similar, lz4c has been deprecated
> > > upstream since 2018. Since as early as Ubuntu 16.04 and Fedora 25, lz4
> > > and lz4c have been packaged together, making it safe to update the
> > > requirement from lz4c to lz4.
> > > 
> > > Consequently, some distributions and build systems, such as OpenEmbedded,
> > > have fully transitioned to using lz4. OpenEmbedded core adopted this
> > > change in commit fe167e082cbd ("bitbake.conf: require lz4 instead of
> > > lz4c"), causing compatibility issues when building the mainline kernel
> > > in the latest OpenEmbedded environment, as seen in the errors below.
> > > 
> > > This change also updates the LZ4 compression commands to make it backward
> > > compatible by replacing stdin and stdout with the '-' option, due to some
> > > unclear reason, the stdout keyword does not work for lz4 and '-' works for
> > > both. In addition, this modifies the legacy '-c1' with '-9' which is also
> > > compatible with both. This fixes the mainline kernel build failures with
> > > the latest master OpenEmbedded builds associated with the mentioned
> > > compatibility issues.
> > > 
> > > LZ4     arch/arm/boot/compressed/piggy_data
> > > /bin/sh: 1: lz4c: not found
> > > ...
> > > ...
> > > ERROR: oe_runmake failed
> > > 
> > > Cc: stable@vger.kernel.org
> > 
> > What bug does this resolve that it needs to be backported to stable
> > kernels?
> 
> This is not solving any existing actual bug, and therefore there is no
> fixes tag.
> 
> The issue here is that the kernel build system is using lz4c, that is
> deprecated since 2018, and now distributions are actively moving away from it. 
> 
> openSUSE Tumbleweed and OE already removed it, so you would not be able
> to compile a stable kernel on such distribution when using lz4 unless we
> backport such a patch.
> 
> Everything should be properly documented in the commit message already.
> 
> My understanding is that something like that would be a reason for
> backporting to stable, if my understanding is not correct we'll remove
> the cc:stable and send a v3.

Please read:
    https://www.kernel.org/doc/html/latest/process/stable-kernel-rules.html
for what meets stable kernel requirements.  I don't think that this
patch is that.

thanks,

greg k-h

Re: [PATCH v2] kbuild: switch from lz4c to lz4 for compression

Posted by Francesco Dolcini 1 week ago

On Fri, Nov 15, 2024 at 10:22:13AM +0100, Greg KH wrote:
> On Fri, Nov 15, 2024 at 09:39:40AM +0100, Francesco Dolcini wrote:
> > On Thu, Nov 14, 2024 at 05:02:01PM +0100, Greg KH wrote:
> > > On Thu, Nov 14, 2024 at 03:56:44PM +0100, Parth Pancholi wrote:
> > > > From: Parth Pancholi <parth.pancholi@toradex.com>
> > > > 
> > > > Replace lz4c with lz4 for kernel image compression.
> > > > Although lz4 and lz4c are functionally similar, lz4c has been deprecated
> > > > upstream since 2018. Since as early as Ubuntu 16.04 and Fedora 25, lz4
> > > > and lz4c have been packaged together, making it safe to update the
> > > > requirement from lz4c to lz4.
> > > > 
> > > > Consequently, some distributions and build systems, such as OpenEmbedded,
> > > > have fully transitioned to using lz4. OpenEmbedded core adopted this
> > > > change in commit fe167e082cbd ("bitbake.conf: require lz4 instead of
> > > > lz4c"), causing compatibility issues when building the mainline kernel
> > > > in the latest OpenEmbedded environment, as seen in the errors below.
> > > > 
> > > > This change also updates the LZ4 compression commands to make it backward
> > > > compatible by replacing stdin and stdout with the '-' option, due to some
> > > > unclear reason, the stdout keyword does not work for lz4 and '-' works for
> > > > both. In addition, this modifies the legacy '-c1' with '-9' which is also
> > > > compatible with both. This fixes the mainline kernel build failures with
> > > > the latest master OpenEmbedded builds associated with the mentioned
> > > > compatibility issues.
> > > > 
> > > > LZ4     arch/arm/boot/compressed/piggy_data
> > > > /bin/sh: 1: lz4c: not found
> > > > ...
> > > > ...
> > > > ERROR: oe_runmake failed
> > > > 
> > > > Cc: stable@vger.kernel.org
> > > 
> > > What bug does this resolve that it needs to be backported to stable
> > > kernels?
> > 
> > This is not solving any existing actual bug, and therefore there is no
> > fixes tag.
> > 
> > The issue here is that the kernel build system is using lz4c, that is
> > deprecated since 2018, and now distributions are actively moving away from it. 
> > 
> > openSUSE Tumbleweed and OE already removed it, so you would not be able
> > to compile a stable kernel on such distribution when using lz4 unless we
> > backport such a patch.
> > 
> > Everything should be properly documented in the commit message already.
> > 
> > My understanding is that something like that would be a reason for
> > backporting to stable, if my understanding is not correct we'll remove
> > the cc:stable and send a v3.
> 
> Please read:
>     https://www.kernel.org/doc/html/latest/process/stable-kernel-rules.html
> for what meets stable kernel requirements.  I don't think that this
> patch is that.

Greg, ack.

Masahiro, can you please let me know if we should send a v3 with the stable
tag removed or you can remove it yourself when applying?

Francesco

Re: [PATCH v2] kbuild: switch from lz4c to lz4 for compression

Posted by Masahiro Yamada 6 days, 13 hours ago

On Fri, Nov 15, 2024 at 6:45 PM Francesco Dolcini <francesco@dolcini.it> wrote:
>
> On Fri, Nov 15, 2024 at 10:22:13AM +0100, Greg KH wrote:
> > On Fri, Nov 15, 2024 at 09:39:40AM +0100, Francesco Dolcini wrote:
> > > On Thu, Nov 14, 2024 at 05:02:01PM +0100, Greg KH wrote:
> > > > On Thu, Nov 14, 2024 at 03:56:44PM +0100, Parth Pancholi wrote:
> > > > > From: Parth Pancholi <parth.pancholi@toradex.com>
> > > > >
> > > > > Replace lz4c with lz4 for kernel image compression.
> > > > > Although lz4 and lz4c are functionally similar, lz4c has been deprecated
> > > > > upstream since 2018. Since as early as Ubuntu 16.04 and Fedora 25, lz4
> > > > > and lz4c have been packaged together, making it safe to update the
> > > > > requirement from lz4c to lz4.
> > > > >
> > > > > Consequently, some distributions and build systems, such as OpenEmbedded,
> > > > > have fully transitioned to using lz4. OpenEmbedded core adopted this
> > > > > change in commit fe167e082cbd ("bitbake.conf: require lz4 instead of
> > > > > lz4c"), causing compatibility issues when building the mainline kernel
> > > > > in the latest OpenEmbedded environment, as seen in the errors below.
> > > > >
> > > > > This change also updates the LZ4 compression commands to make it backward
> > > > > compatible by replacing stdin and stdout with the '-' option, due to some
> > > > > unclear reason, the stdout keyword does not work for lz4 and '-' works for
> > > > > both. In addition, this modifies the legacy '-c1' with '-9' which is also
> > > > > compatible with both. This fixes the mainline kernel build failures with
> > > > > the latest master OpenEmbedded builds associated with the mentioned
> > > > > compatibility issues.
> > > > >
> > > > > LZ4     arch/arm/boot/compressed/piggy_data
> > > > > /bin/sh: 1: lz4c: not found
> > > > > ...
> > > > > ...
> > > > > ERROR: oe_runmake failed
> > > > >
> > > > > Cc: stable@vger.kernel.org
> > > >
> > > > What bug does this resolve that it needs to be backported to stable
> > > > kernels?
> > >
> > > This is not solving any existing actual bug, and therefore there is no
> > > fixes tag.
> > >
> > > The issue here is that the kernel build system is using lz4c, that is
> > > deprecated since 2018, and now distributions are actively moving away from it.
> > >
> > > openSUSE Tumbleweed and OE already removed it, so you would not be able
> > > to compile a stable kernel on such distribution when using lz4 unless we
> > > backport such a patch.
> > >
> > > Everything should be properly documented in the commit message already.
> > >
> > > My understanding is that something like that would be a reason for
> > > backporting to stable, if my understanding is not correct we'll remove
> > > the cc:stable and send a v3.
> >
> > Please read:
> >     https://www.kernel.org/doc/html/latest/process/stable-kernel-rules.html
> > for what meets stable kernel requirements.  I don't think that this
> > patch is that.
>
> Greg, ack.
>
> Masahiro, can you please let me know if we should send a v3 with the stable
> tag removed or you can remove it yourself when applying?
>

I applied this with the stable tag removed.
Thanks.

(I guess someone may want to backport this eventually,
as such distros cannot build stable kernels with ld4 compression.)



-- 
Best Regards
Masahiro Yamada