From nobody Sun Feb 8 13:45:51 2026 Received: from mail-ot1-f46.google.com (mail-ot1-f46.google.com [209.85.210.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id ADDBA2EACEE for ; Tue, 6 Jan 2026 16:28:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.46 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1767716911; cv=none; b=Trvgfd3pJkSC99pzIz/io0eekVm79+UJ5tfqmWxqBX6Uw8uTZ1lWkxhuUrgF1ai9CyL7BQzqkBS42toUXn+1Jl6GwpU7UcVy4Ha3cmU7oju1Y8t+KZj0wvkFqbjWQo4FkO8FToYYF+B3mP/bAFm/6qzkl5vA+4jmvhHiIGC+6Mo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1767716911; c=relaxed/simple; bh=6WiN2eHlXWLKbTPNMWm+xNazMd3hRxima/bQfzAVX98=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=XPgs6m8Sp1aFovoTMbPsPWvacO/CMspXZ89f3oj9WV79uOAVoQvtBbZ6sGINMj2isWIxgkaDgDP26mWaJhDs0OTzPjp0OmDNK0EsoiBJCQjRnKbwrVP2fPNf8Fr9W/oKcajPJuTK9MvGA7Q2UoXbCQmFN56EOI7s87pOLAUbIvw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=Kh2hRudu; arc=none smtp.client-ip=209.85.210.46 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="Kh2hRudu" Received: by mail-ot1-f46.google.com with SMTP id 46e09a7af769-7c6da42fbd4so685678a34.1 for ; Tue, 06 Jan 2026 08:28:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1767716908; x=1768321708; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=8l+PMSx1dMo0V8MdT06UoUXh18bncmRY0mnlTCdtsng=; b=Kh2hRuduSY84yycG3gd3sGFEm4Ibo9kqIgp+x9NKV1TK/t+ihSIcYwpsFtrF4hNIhV 5IV06XWn9VK+wQl8pYQIx+IA24Y6rO6OTi+frw8tsn4GWMfFCQF0cBggET9nzxmGijtF qD2QJJ9M/Cc6PcCfbhPrYV1PUMChhFuMkB1u8= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1767716908; x=1768321708; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=8l+PMSx1dMo0V8MdT06UoUXh18bncmRY0mnlTCdtsng=; b=ohNcLe0+yddwHeoVnMFxfx/16lno/eoLG+oH9EBg0Fp3eZuARBJnZSIb6tDjh9aj/c GhDgeV4crdSemqp6mMhspyQpUmlkt6viBNNqlHWNrq31cjbgNUKVeiwqxkf9fvgJld0u iTelpyEEQtYfbneUniGoqIF7wFOBNSYkq28+P8tqSf6e9FsqKXfIxXQbUfyNxMtEBeR5 5Te+6tYHk/G/28f9wUzzFJt07Q/AHYFBmFCNC9jML2Qf5Yfs1ZNcyVF1tQLhTl87DwnM gHBcOawOHZuchqXrO4jTQLIWMOX6K2iFZeCI5LZ61Jd5EzjqnOfNCCJ6O4M4VXcJ3lhG MZfw== X-Forwarded-Encrypted: i=1; AJvYcCWo5nIdV5vmbC53L8zylYs5FMINKtQTBwi1NfzPHjLKNdl6ucZ2ZLZcmXr+yl1wKXeNm0U8DsV0O/++tHw=@vger.kernel.org X-Gm-Message-State: AOJu0YzcjFSVYg23v09qeMnou8Lpgogr6AQM09fuDlMFp2/gTN4z08Os LmRi+MCmJnMw2FrskmhCFc03MvhoJa1+WvasQqulFM9vK1Sa1wCeQUV0AuxwXUfkkQ== X-Gm-Gg: AY/fxX6Wl6ekF6r3vRkP0cYfxCEhGMfCJIgFtwBTETkYr9v0YayXPLmIbKdZvtxWh7J fe1O40lVFU32fkjqly++iIya+P1nPr2NvF0s9z9/3b3VJjmPPr50Gp7mJ/Ortc85dOfKYHgtoDQ gxfSIMeRz7+XOmSOn7X37gjGpUF1raEHLFgrBvCP1iThflM0S0fCXcVlhk9BhAXgFWDnvw+mISK a91Y9KuQv7ghCRwAIcYUTCqCA2zvTsx4Z3LnhnlNBuNOvLTKwD/1lDsO6QyQ6QR3sgu0H7toz61 AgHVqrc8x/Y+qLxxkvRxM9T6vseco+N8//WjZlwoi5NSNhz9tHAAG+GbyfO67Wo+kJbexuivqE2 fGyU3d0oFuLqxQna8hlONRIgYBMMTszMNwFgJ1cOcR8sOUa23NHtCRyNSBcUiLp2TvTJqwJbaCs DDGUx/ X-Google-Smtp-Source: AGHT+IG1N3IU6CP3R1t+h3+E8Tm3EouijsvGbZCZNnDD2IeLB1xOLNvsC4ZitJZZxIKks46Ly/KPWQ== X-Received: by 2002:a05:6830:4489:b0:7c5:2f8f:2527 with SMTP id 46e09a7af769-7ce4654f12fmr1812171a34.0.1767716908552; Tue, 06 Jan 2026 08:28:28 -0800 (PST) Received: from chromium.org ([73.34.74.121]) by smtp.gmail.com with ESMTPSA id 46e09a7af769-7ce478af8b2sm1630951a34.15.2026.01.06.08.28.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Jan 2026 08:28:23 -0800 (PST) From: Simon Glass To: linux-arm-kernel@lists.infradead.org Cc: =?UTF-8?q?Thomas=20Wei=C3=9Fschuh?= , =?UTF-8?q?J=20=2E=20Neusch=C3=A4fer?= , Masahiro Yamada , Ahmad Fatoum , Nicolas Schier , Tom Rini , Chen-Yu Tsai , Simon Glass , linux-kernel@vger.kernel.org Subject: [PATCH v9 6/6] scripts/make_fit: Compress dtbs in parallel Date: Tue, 6 Jan 2026 09:27:36 -0700 Message-ID: <20260106162738.2605574-7-sjg@chromium.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260106162738.2605574-1-sjg@chromium.org> References: <20260106162738.2605574-1-sjg@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" When there are 1500 device tree files it takes quite a while to compress them. Do it in parallel. Signed-off-by: Simon Glass --- (no changes since v8) Changes in v8: - Drop erroneous line 'kbuild: Allow adding modules into the FIT ...' scripts/make_fit.py | 56 ++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 50 insertions(+), 6 deletions(-) diff --git a/scripts/make_fit.py b/scripts/make_fit.py index defb87f223d7..e923cc8b05b7 100755 --- a/scripts/make_fit.py +++ b/scripts/make_fit.py @@ -37,6 +37,7 @@ as U-Boot, Linuxboot, Tianocore, etc. =20 import argparse import collections +import multiprocessing import os import subprocess import sys @@ -225,15 +226,31 @@ def compress_data(inf, compress): return comp_data =20 =20 -def output_dtb(fsw, seq, fname, arch, compress): +def compress_dtb(fname, compress): + """Compress a single DTB file + + Args: + fname (str): Filename containing the DTB + compress (str): Compression algorithm, e.g. 'gzip' + + Returns: + tuple: (str: fname, bytes: compressed_data) + """ + with open(fname, 'rb') as inf: + compressed =3D compress_data(inf, compress) + return fname, compressed + + +def output_dtb(fsw, seq, fname, arch, compress, data=3DNone): """Write out a single devicetree to the FIT =20 Args: fsw (libfdt.FdtSw): Object to use for writing seq (int): Sequence number (1 for first) fname (str): Filename containing the DTB - arch: FIT architecture, e.g. 'arm64' + arch (str): FIT architecture, e.g. 'arm64' compress (str): Compressed algorithm, e.g. 'gzip' + data (bytes): Pre-compressed data (optional) """ with fsw.add_node(f'fdt-{seq}'): fsw.property_string('description', os.path.basename(fname)) @@ -241,9 +258,10 @@ def output_dtb(fsw, seq, fname, arch, compress): fsw.property_string('arch', arch) fsw.property_string('compression', compress) =20 - with open(fname, 'rb') as inf: - compressed =3D compress_data(inf, compress) - fsw.property('data', compressed) + if data is None: + with open(fname, 'rb') as inf: + data =3D compress_data(inf, compress) + fsw.property('data', data) =20 =20 def process_dtb(fname, args): @@ -300,6 +318,11 @@ def _process_dtbs(args, fsw, entries, fdts): """ seq =3D 0 size =3D 0 + + # First figure out the unique DTB files that need compression + todo =3D [] + file_info =3D [] # List of (fname, model, compat, files) tuples + for fname in args.dtbs: # Ignore non-DTB (*.dtb) files if os.path.splitext(fname)[1] !=3D '.dtb': @@ -311,11 +334,32 @@ def _process_dtbs(args, fsw, entries, fdts): sys.stderr.write(f'Error processing {fname}:\n') raise e =20 + file_info.append((fname, model, compat, files)) + for fn in files: + if fn not in fdts and fn not in todo: + todo.append(fn) + + # Compress all DTBs in parallel + cache =3D {} + if todo and args.compress !=3D 'none': + if args.verbose: + print(f'Compressing {len(todo)} DTBs...') + + with multiprocessing.Pool() as pool: + compress_args =3D [(fn, args.compress) for fn in todo] + # unpacks each tuple, calls compress_dtb(fn, compress) in para= llel + results =3D pool.starmap(compress_dtb, compress_args) + + cache =3D dict(results) + + # Now write all DTBs to the FIT using pre-compressed data + for fname, model, compat, files in file_info: for fn in files: if fn not in fdts: seq +=3D 1 size +=3D os.path.getsize(fn) - output_dtb(fsw, seq, fn, args.arch, args.compress) + output_dtb(fsw, seq, fn, args.arch, args.compress, + cache.get(fn)) fdts[fn] =3D seq =20 files_seq =3D [fdts[fn] for fn in files] --=20 2.43.0