[PATCH v6 7/8] scripts/make_fit: Support a few more parallel compressors

Simon Glass posted 8 patches 2 months, 3 weeks ago
There is a newer version of this series
[PATCH v6 7/8] scripts/make_fit: Support a few more parallel compressors
Posted by Simon Glass 2 months, 3 weeks ago
Add support for pbzip2 and plzip which can compress in parallel. This
speeds up the ramdisk compression.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

(no changes since v1)

 scripts/make_fit.py | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)

diff --git a/scripts/make_fit.py b/scripts/make_fit.py
index 9dfef11fc4b3..64038f17a51e 100755
--- a/scripts/make_fit.py
+++ b/scripts/make_fit.py
@@ -57,10 +57,10 @@ import libfdt
 CompTool = collections.namedtuple('CompTool', 'ext,tools')
 
 COMP_TOOLS = {
-    'bzip2': CompTool('.bz2', 'bzip2'),
+    'bzip2': CompTool('.bz2', 'pbzip2,bzip2'),
     'gzip': CompTool('.gz', 'pigz,gzip'),
     'lz4': CompTool('.lz4', 'lz4'),
-    'lzma': CompTool('.lzma', 'lzma'),
+    'lzma': CompTool('.lzma', 'plzip,lzma'),
     'lzo': CompTool('.lzo', 'lzop'),
     'zstd': CompTool('.zstd', 'zstd'),
 }
@@ -220,7 +220,12 @@ def compress_data(inf, compress):
             done = False
             for tool in comp.tools.split(','):
                 try:
-                    subprocess.call([tool, '-c'], stdin=inf, stdout=outf)
+                    # Add parallel flags for tools that support them
+                    cmd = [tool]
+                    if tool in ('zstd', 'xz'):
+                        cmd.extend(['-T0'])  # Use all available cores
+                    cmd.append('-c')
+                    subprocess.call(cmd, stdin=inf, stdout=outf)
                     done = True
                     break
                 except FileNotFoundError:
-- 
2.43.0
Re: [PATCH v6 7/8] scripts/make_fit: Support a few more parallel compressors
Posted by Chen-Yu Tsai 2 months, 1 week ago
On Thu, Nov 20, 2025 at 2:14 AM Simon Glass <sjg@chromium.org> wrote:
>
> Add support for pbzip2 and plzip which can compress in parallel. This
> speeds up the ramdisk compression.
>
> Signed-off-by: Simon Glass <sjg@chromium.org>
> ---
>
> (no changes since v1)
>
>  scripts/make_fit.py | 11 ++++++++---
>  1 file changed, 8 insertions(+), 3 deletions(-)
>
> diff --git a/scripts/make_fit.py b/scripts/make_fit.py
> index 9dfef11fc4b3..64038f17a51e 100755
> --- a/scripts/make_fit.py
> +++ b/scripts/make_fit.py
> @@ -57,10 +57,10 @@ import libfdt
>  CompTool = collections.namedtuple('CompTool', 'ext,tools')
>
>  COMP_TOOLS = {
> -    'bzip2': CompTool('.bz2', 'bzip2'),
> +    'bzip2': CompTool('.bz2', 'pbzip2,bzip2'),
>      'gzip': CompTool('.gz', 'pigz,gzip'),
>      'lz4': CompTool('.lz4', 'lz4'),
> -    'lzma': CompTool('.lzma', 'lzma'),
> +    'lzma': CompTool('.lzma', 'plzip,lzma'),
>      'lzo': CompTool('.lzo', 'lzop'),
>      'zstd': CompTool('.zstd', 'zstd'),
>  }
> @@ -220,7 +220,12 @@ def compress_data(inf, compress):
>              done = False
>              for tool in comp.tools.split(','):
>                  try:
> -                    subprocess.call([tool, '-c'], stdin=inf, stdout=outf)
> +                    # Add parallel flags for tools that support them
> +                    cmd = [tool]
> +                    if tool in ('zstd', 'xz'):

There's no 'xz' tool in the list though.

And this hunk isn't mentioned or explained in the commit message.

> +                        cmd.extend(['-T0'])  # Use all available cores

Zero is already the default for 'xz' in 5.8.1.


ChenYu

> +                    cmd.append('-c')
> +                    subprocess.call(cmd, stdin=inf, stdout=outf)
>                      done = True
>                      break
>                  except FileNotFoundError:
> --
> 2.43.0
>
Re: [PATCH v6 7/8] scripts/make_fit: Support a few more parallel compressors
Posted by Simon Glass 2 months ago
Hi Chen-Yu,

On Tue, 2 Dec 2025 at 03:18, Chen-Yu Tsai <wenst@chromium.org> wrote:
>
> On Thu, Nov 20, 2025 at 2:14 AM Simon Glass <sjg@chromium.org> wrote:
> >
> > Add support for pbzip2 and plzip which can compress in parallel. This
> > speeds up the ramdisk compression.
> >
> > Signed-off-by: Simon Glass <sjg@chromium.org>
> > ---
> >
> > (no changes since v1)
> >
> >  scripts/make_fit.py | 11 ++++++++---
> >  1 file changed, 8 insertions(+), 3 deletions(-)
> >
> > diff --git a/scripts/make_fit.py b/scripts/make_fit.py
> > index 9dfef11fc4b3..64038f17a51e 100755
> > --- a/scripts/make_fit.py
> > +++ b/scripts/make_fit.py
> > @@ -57,10 +57,10 @@ import libfdt
> >  CompTool = collections.namedtuple('CompTool', 'ext,tools')
> >
> >  COMP_TOOLS = {
> > -    'bzip2': CompTool('.bz2', 'bzip2'),
> > +    'bzip2': CompTool('.bz2', 'pbzip2,bzip2'),
> >      'gzip': CompTool('.gz', 'pigz,gzip'),
> >      'lz4': CompTool('.lz4', 'lz4'),
> > -    'lzma': CompTool('.lzma', 'lzma'),
> > +    'lzma': CompTool('.lzma', 'plzip,lzma'),
> >      'lzo': CompTool('.lzo', 'lzop'),
> >      'zstd': CompTool('.zstd', 'zstd'),
> >  }
> > @@ -220,7 +220,12 @@ def compress_data(inf, compress):
> >              done = False
> >              for tool in comp.tools.split(','):
> >                  try:
> > -                    subprocess.call([tool, '-c'], stdin=inf, stdout=outf)
> > +                    # Add parallel flags for tools that support them
> > +                    cmd = [tool]
> > +                    if tool in ('zstd', 'xz'):
>
> There's no 'xz' tool in the list though.
>
> And this hunk isn't mentioned or explained in the commit message.

OK I will update it.

>
> > +                        cmd.extend(['-T0'])  # Use all available cores
>
> Zero is already the default for 'xz' in 5.8.1.

My system is fairly up-to-date (Noble) and I have:

xz -V
xz (XZ Utils) 5.4.5
liblzma 5.4.5

Does the new version reject -T0 ? If not, then I think it is best to
provide it in this script.

>
>
> ChenYu
>
> > +                    cmd.append('-c')
> > +                    subprocess.call(cmd, stdin=inf, stdout=outf)
> >                      done = True
> >                      break
> >                  except FileNotFoundError:
> > --
> > 2.43.0
> >

Regards,
Simon