From nobody Tue Dec 2 02:19:52 2025 Received: from mail-io1-f41.google.com (mail-io1-f41.google.com [209.85.166.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C76EB30C601 for ; Wed, 19 Nov 2025 18:14:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.41 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763576051; cv=none; b=FktDv+QRV4pScO5EZsxT4dYfCNKlqCQ9uDX2lLcm2I+uDnwFkZrHceux8CtKrLGyffy0Dmy0maBrKZ+JeqPp07Mrey0WvUGD5pURoFXUwS0FoMUQ1pnBMVTPSefOKoYw/USJzvoIgfn+T6/bLjpkvV53/8yAxhPBJCgHGOuLsQg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763576051; c=relaxed/simple; bh=WxmuUnG9mgIBxKipIANrDQZ0nIjxvThuVKA/zPa2iQ4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=akpK6OoLe7d7Km/B++IKByObEKZ23qbuio//dt7EeSUFlQYIXNUGg73BvlTU6WLh8k74BqqF5Npt5ZaRKB8w92Xhkjvy5F0WAX6V4nVVIPkRvci2HCqrfbBGrO9r+1qmY/ZpR803lnBPs9Wz6IXD9k4g74xO/Dwsx/94WYnsYuw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=lD2aUj95; arc=none smtp.client-ip=209.85.166.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="lD2aUj95" Received: by mail-io1-f41.google.com with SMTP id ca18e2360f4ac-948f8fa9451so137070839f.3 for ; Wed, 19 Nov 2025 10:14:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1763576043; x=1764180843; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=VOwDYlRH5DhNzOzjWWbwzUOQiOeiolgsqjgpBqGQ14g=; b=lD2aUj95/Ik+uiY44n0boB1hFIZHU2AO77fpKDbuLUqXQX79MhuuTsptZyuC8u2AoB 18K1laa8DbARtdFqjAUHkJiAQJ2aE0CkcawNWl/5WVq4qURgPNN05wtxLCES4I+AljJA pSRv70ovy4r8HaXMzSj2zNng9O5zrAi2tn3x0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1763576043; x=1764180843; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=VOwDYlRH5DhNzOzjWWbwzUOQiOeiolgsqjgpBqGQ14g=; b=OtQ8oHUfA+iRwmmFEv8RRoaD1i4Tse9hxW+nxX0qFuy3NGm1bIcEpfd8rKUUbKksb4 4PBHOt0zdO+K99X6+hGlwqa05Asz8BRKR0Q7czY9pQTd36BtcvHZt4meCAQSlwcJi9o1 3O5/P0x8ImmV2+6PnG9ogJpO9CFyjewmATBpSFDH7STiF5jOqdjIuqWskkA4P5m967sB z8PH1TA1zNXx4ZJL/OpgphN1FatlhFs+5YiultACMbtcHJHM8npDlodn0Ee3dNTfji1U gy+/pJLIsVGf21tyZGonBBp6ek0J9lnArENGyuOg7crKGrOlTtCD5iKgKO235IqhN4jT NZXQ== X-Forwarded-Encrypted: i=1; AJvYcCVrUDddXTEGjssKHjVD8+bx0DkYjLI3177bq1RhEXA3qMM9qDf0lc1LpTV+g+erW98Q+J5FczC8SnnhKVc=@vger.kernel.org X-Gm-Message-State: AOJu0Yyp1RSx2QU3JdFbzDSRaIAfuoO37Dn0/eGYzQ/UTsg2heCL20kd YLa/edtWpvTOy6bqbVcPDERMlwbPFJmQDVG5QywPy6EdIMw4uNG2tkJmIfqTPsncTg== X-Gm-Gg: ASbGncuxJigAqHPo9BYvgOWrhbbA8eeb0Ohy4mFRqo+awNO2spk8N+qyltcF0c9P5ZN SJCF1dV8MwArG19kXiEUB1SdTS22ZymNVzJugPonryk9oLQ/fqhhwVpGXZQcX1IrgkXJnhIYqp+ 0b54mc1As7jZgstNarjdnTAfOH+Lg7vUeGvJ7V6StIcCRUXTjSx/oDVr7eIspefIWyDQYsDE4Ra CROxnIAnM3nAOTCs7QEitq5ZbsDewVwykFX46W0kSA/uqpgY+97hOrgtqY8PK2WeqsUfIRQRBeb 3nixy5OY2mSNp44kn5fh1K1tgnV71HhwQFExEpev1E2fI3XgSpYs9q2ri3M3LPXncA7YYvd9KBl SzAMyxWszyJQo+6E5d3khmCA5dX7GVvartOqESSNzOWDJ43np8/Dwuqu+IGNUsPwtYBqD2rgGZR Mo5X8TazxrWEiZovw= X-Google-Smtp-Source: AGHT+IH6yIX6WHieDPQzbEo8QY+iE45HF4Ibs2aq1PWlH9l6naUWz/+RgOAJxj/Z0iBtnKJ4hNu+AQ== X-Received: by 2002:a05:6602:640e:b0:948:707f:1421 with SMTP id ca18e2360f4ac-9492b6b66bfmr387613439f.15.1763576043003; Wed, 19 Nov 2025 10:14:03 -0800 (PST) Received: from chromium.org ([73.34.74.121]) by smtp.gmail.com with ESMTPSA id ca18e2360f4ac-949385ae254sm4838639f.1.2025.11.19.10.14.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Nov 2025 10:14:01 -0800 (PST) From: Simon Glass To: linux-arm-kernel@lists.infradead.org Cc: =?UTF-8?q?Thomas=20Wei=C3=9Fschuh?= , Masahiro Yamada , Tom Rini , Ahmad Fatoum , =?UTF-8?q?J=20=2E=20Neusch=C3=A4fer?= , Nicolas Schier , Chen-Yu Tsai , Simon Glass , Nicolas Schier , linux-kernel@vger.kernel.org Subject: [PATCH v6 1/8] scripts/make_fit: Speed up operation Date: Wed, 19 Nov 2025 11:13:22 -0700 Message-ID: <20251119181333.991099-2-sjg@chromium.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20251119181333.991099-1-sjg@chromium.org> References: <20251119181333.991099-1-sjg@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The kernel is likely at least 16MB so we may as well use that as a step size when reallocating space for the FIT in memory. Pack the FIT at the end, so there is no wasted space. This reduces the time to pack by an order of magnitude, or so. Signed-off-by: Simon Glass Reviewed-by: Nicolas Schier Reviewed-by: Ahmad Fatoum --- (no changes since v3) Changes in v3: - Move the ramdisk chunk into the correct patch scripts/make_fit.py | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/scripts/make_fit.py b/scripts/make_fit.py index 1683e5ec6e67..0f5e7c4b8aed 100755 --- a/scripts/make_fit.py +++ b/scripts/make_fit.py @@ -98,7 +98,7 @@ def setup_fit(fsw, name): fsw (libfdt.FdtSw): Object to use for writing name (str): Name of kernel image """ - fsw.INC_SIZE =3D 65536 + fsw.INC_SIZE =3D 16 << 20 fsw.finish_reservemap() fsw.begin_node('') fsw.property_string('description', f'{name} with devicetree set') @@ -299,7 +299,9 @@ def build_fit(args): finish_fit(fsw, entries) =20 # Include the kernel itself in the returned file count - return fsw.as_fdt().as_bytearray(), seq + 1, size + fdt =3D fsw.as_fdt() + fdt.pack() + return fdt.as_bytearray(), seq + 1, size =20 =20 def run_make_fit(): --=20 2.43.0 From nobody Tue Dec 2 02:19:52 2025 Received: from mail-io1-f49.google.com (mail-io1-f49.google.com [209.85.166.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A168030DD22 for ; Wed, 19 Nov 2025 18:14:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.49 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763576064; cv=none; b=KdGAzTBRlEkCihEgKeSkrc3Nmow+OrnqBu4SkAET6qYjaWGYrr/WoMmfVslnQzAzL5DD7rEgjX2bw9wHEGwwXTOBgggsbYBn3+Cb6t2I8EwJYbNZ0j1QeLroTBLYWVwz6zoLSmVwQmqu/zzFvaWR8JzufT5/sRHLlpfu9GsqnOY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763576064; c=relaxed/simple; bh=bIhJcMaNr2LuLFf95ViTv9PAdwNxqqzPJpRV+yf7Mgk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=i+HQpCpw5HVq+Y7J7myC0ZdeSyvgCRDCPWbsgAhivZh5iHNtJ1dFSOroBf4HdTFgFnufvodIAez+DX2OsOl42un7153P9tK3B90JiTZx/u6ZOKzSPSbtCA0NO81v7d4xcdEmNKrf2OO6oP2qdvlGhplwSUcmZPNDihRDup4yg70= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=NP74OFrj; arc=none smtp.client-ip=209.85.166.49 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="NP74OFrj" Received: by mail-io1-f49.google.com with SMTP id ca18e2360f4ac-94880a46f3fso223670039f.2 for ; Wed, 19 Nov 2025 10:14:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1763576045; x=1764180845; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=vGHNU0Y9qZ5AJfcI+Hz5PCDOOAK0uiMHY3/rFUZcMgQ=; b=NP74OFrjpknweEr1EM+zl1BAWDoKJiP3AtXNpnEUgbZOXs6BNJPE4By+KqzU9N1mXq fnQ/eFSlViVrAR0MtvJg5B3STdiywzS0YHKFNdRsMUS0G2xfslg/K7rigdiIKLULZsfo 2WO70hi0tIgkfSbPiQnWUHun7cbdHCHLVZtQw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1763576045; x=1764180845; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=vGHNU0Y9qZ5AJfcI+Hz5PCDOOAK0uiMHY3/rFUZcMgQ=; b=uDL3xRVXnGPmCH6Rv+NjtDhxlZ8kjkH/hmU6HPras5KZrGKCiKcQG4WVPCPIhbLpD/ fC8XWfgk4Po1i6m2knK9dKUaYcHTs+HGXDMHazzOJp6QJxaiDPMQywY3YUHWWMxwWhhc m/Iu335Y3QuuuLbbn3XA8UK+9Ji4e2f/TprQLkhMFFX1+o/BPdqpO73bhZKnB/RQyYxA DDB+N7qg/cu9eUpBGHrjGMDfaYNbLp/l7j185UIMeaQOkebg1gZtLIp/lhr1ekc5Dl1P rHN8Pn4ftr5BeNlS/IQCvkqgCqcPpzXGApgWcArocXyjI9WlXPtlGwq+rwjsrVqp7uXW f+tw== X-Forwarded-Encrypted: i=1; AJvYcCV7PPtXdUvwvRW6aJyPXeg277Qds/Gg7EdJ+8y3IQ1qD4lAcbjs09e0i5ngibQTWPI20BJWKX2f2smXIMw=@vger.kernel.org X-Gm-Message-State: AOJu0Yx5HkH98hD44YzkNQt1fFxjiDaRv0dolGckS5r6Gopsa8/q6rVo nEN6wMVAxvVo0D7WzBarQo5I/mRkMf9EdKE8hcnm5PDJnzwtLULnWPEUlpiERkDCUw== X-Gm-Gg: ASbGncsoWheccz83BpF9/2pvQti9aQA+adY2gB3zfFmb7IlNY01Zz3qoiveQtVjyKcB irMdd1ypEsPRwhzYbfy3k+UMlCceGnVlRnNjpveWitnDw9rwzMoGEetCZR9oPM2SScuL6QtMDnH bBu0d0+9bZO8R9JG4sDXDE5XAMbg4FUQ8JVxodw85hoO+ugcAjhzLxXKdt3yfER24DdmgKVf4b+ EhGFkpwYaqp9f4gTnVbo4TWHAHjtNi8HDr4I5U2hkbvJYTmYbC7nZfu0eygytY4jA3WNgGqPxQX b0S9hqzfvW30hjhVRxr2e/XgJ5/+myn0L86lyZnyF+6kG5v1tHZuoQCvMSX4uz1H1+ALqn0ezX6 7TcHRIaW9YM48lf5uUhgJ6Yubjv0tY625OU1tunIz4La0NDioqc0wxSvUXmzU0OK3HyiLt8V2MX 1HksCov2hWxpZ/xu8= X-Google-Smtp-Source: AGHT+IFizsIKzcR3jkJP4zRDwTEP7H8DQprwvCzIj1bXPI84SnGAVfnAaADLyypxEJkbDennAkqXxw== X-Received: by 2002:a6b:4e19:0:b0:949:3623:dc25 with SMTP id ca18e2360f4ac-9493623efc5mr107582639f.13.1763576045401; Wed, 19 Nov 2025 10:14:05 -0800 (PST) Received: from chromium.org ([73.34.74.121]) by smtp.gmail.com with ESMTPSA id ca18e2360f4ac-949385ae254sm4838639f.1.2025.11.19.10.14.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Nov 2025 10:14:04 -0800 (PST) From: Simon Glass To: linux-arm-kernel@lists.infradead.org Cc: =?UTF-8?q?Thomas=20Wei=C3=9Fschuh?= , Masahiro Yamada , Tom Rini , Ahmad Fatoum , =?UTF-8?q?J=20=2E=20Neusch=C3=A4fer?= , Nicolas Schier , Chen-Yu Tsai , Simon Glass , Nicolas Schier , linux-kernel@vger.kernel.org Subject: [PATCH v6 2/8] scripts/make_fit: Support an initial ramdisk Date: Wed, 19 Nov 2025 11:13:23 -0700 Message-ID: <20251119181333.991099-3-sjg@chromium.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20251119181333.991099-1-sjg@chromium.org> References: <20251119181333.991099-1-sjg@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" FIT (Flat Image Tree) allows a ramdisk to be included in each configuration. Add support for this to the script. This feature is not available via 'make image.fit' since the ramdisk likely needs to be built separately anyway, e.g. using modules from the kernel build. A later patch in this series provides support for doing that. Note that the uncompressed size is not correct when a ramdisk is used, since it is too expensive to decompress the ramdisk. Signed-off-by: Simon Glass Reviewed-by: Nicolas Schier Reviewed-by: Ahmad Fatoum --- (no changes since v5) Changes in v5: - Fix 'use' typo Changes in v4: - Update the commit message Changes in v3: - Add a comment at the top of the file about the -r option - Count the ramdisk in the total files - Update the commit message Changes in v2: - Don't compress the ramdisk as it is already compressed scripts/make_fit.py | 52 ++++++++++++++++++++++++++++++++++++++------- 1 file changed, 44 insertions(+), 8 deletions(-) diff --git a/scripts/make_fit.py b/scripts/make_fit.py index 0f5e7c4b8aed..984371f505bc 100755 --- a/scripts/make_fit.py +++ b/scripts/make_fit.py @@ -10,10 +10,14 @@ Usage: make_fit.py -A arm64 -n 'Linux-6.6' -O linux -o arch/arm64/boot/image.fit -k /tmp/kern/arch/arm64/boot/image.itk - @arch/arm64/boot/dts/dtbs-list -E -c gzip + -r /boot/initrd.img-6.14.0-27-generic @arch/arm64/boot/dts/dtbs-li= st + -E -c gzip =20 -Creates a FIT containing the supplied kernel and a set of devicetree files, -either specified individually or listed in a file (with an '@' prefix). +Creates a FIT containing the supplied kernel, an optional ramdisk, and a s= et of +devicetree files, either specified individually or listed in a file (with = an +'@' prefix). + +Use -r to specify an existing ramdisk/initrd file. =20 Use -E to generate an external FIT (where the data is placed after the FIT data structure). This allows parsing of the data without loading @@ -29,8 +33,6 @@ looks at the .cmd files produced by the kernel build. =20 The resulting FIT can be booted by bootloaders which support FIT, such as U-Boot, Linuxboot, Tianocore, etc. - -Note that this tool does not yet support adding a ramdisk / initrd. """ =20 import argparse @@ -81,6 +83,8 @@ def parse_args(): help=3D'Specifies the operating system') parser.add_argument('-k', '--kernel', type=3Dstr, required=3DTrue, help=3D'Specifies the (uncompressed) kernel input file (.itk)') + parser.add_argument('-r', '--ramdisk', type=3Dstr, + help=3D'Specifies the ramdisk/initrd input file') parser.add_argument('-v', '--verbose', action=3D'store_true', help=3D'Enable verbose output') parser.add_argument('dtbs', type=3Dstr, nargs=3D'*', @@ -133,7 +137,28 @@ def write_kernel(fsw, data, args): fsw.property_u32('entry', 0) =20 =20 -def finish_fit(fsw, entries): +def write_ramdisk(fsw, data, args): + """Write out the ramdisk image + + Writes a ramdisk node along with the required properties + + Args: + fsw (libfdt.FdtSw): Object to use for writing + data (bytes): Data to write (possibly compressed) + args (Namespace): Contains necessary strings: + arch: FIT architecture, e.g. 'arm64' + fit_os: Operating Systems, e.g. 'linux' + """ + with fsw.add_node('ramdisk'): + fsw.property_string('description', 'Ramdisk') + fsw.property_string('type', 'ramdisk') + fsw.property_string('arch', args.arch) + fsw.property_string('os', args.os) + fsw.property('data', data) + fsw.property_u32('load', 0) + + +def finish_fit(fsw, entries, has_ramdisk=3DFalse): """Finish the FIT ready for use =20 Writes the /configurations node and subnodes @@ -143,6 +168,7 @@ def finish_fit(fsw, entries): entries (list of tuple): List of configurations: str: Description of model str: Compatible stringlist + has_ramdisk (bool): True if a ramdisk is included in the FIT """ fsw.end_node() seq =3D 0 @@ -154,6 +180,8 @@ def finish_fit(fsw, entries): fsw.property_string('description', model) fsw.property('fdt', bytes(''.join(f'fdt-{x}\x00' for x in = files), "ascii")) fsw.property_string('kernel', 'kernel') + if has_ramdisk: + fsw.property_string('ramdisk', 'ramdisk') fsw.end_node() =20 =20 @@ -274,6 +302,14 @@ def build_fit(args): size +=3D os.path.getsize(args.kernel) write_kernel(fsw, comp_data, args) =20 + # Handle the ramdisk if provided. Compression is not supported as it is + # already compressed. + if args.ramdisk: + with open(args.ramdisk, 'rb') as inf: + data =3D inf.read() + size +=3D len(data) + write_ramdisk(fsw, data, args) + for fname in args.dtbs: # Ignore non-DTB (*.dtb) files if os.path.splitext(fname)[1] !=3D '.dtb': @@ -296,12 +332,12 @@ def build_fit(args): =20 entries.append([model, compat, files_seq]) =20 - finish_fit(fsw, entries) + finish_fit(fsw, entries, bool(args.ramdisk)) =20 # Include the kernel itself in the returned file count fdt =3D fsw.as_fdt() fdt.pack() - return fdt.as_bytearray(), seq + 1, size + return fdt.as_bytearray(), seq + 1 + bool(args.ramdisk), size =20 =20 def run_make_fit(): --=20 2.43.0 From nobody Tue Dec 2 02:19:52 2025 Received: from mail-io1-f44.google.com (mail-io1-f44.google.com [209.85.166.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3C61D327C15 for ; Wed, 19 Nov 2025 18:14:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.44 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763576066; cv=none; b=F8o8k0CKGXGrrY6CAuN5wnVc3cMrWQ/4XfB2VnFRu5A84jzDs0fEy9yqjf7qCGv0vKLWB5q5dxBP4seRs1IauNVo7D2czaDBly4OPpsUUfmudPOsrYnnvtpnZy8QvngRF/MYnW+mSBhtJxc0wikcwVRkoU4tj10IKsvTl4Dvd/Q= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763576066; c=relaxed/simple; bh=BGaCAKlcyk5xO8jgGliNm42nPEYthddki81jtkWbq08=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=jcbjDeGa9hDnb32apyN+IctECY23cs9ZSlQwHC0zzNMtLM6PWEPnQ0cXKirXtFy2qUrta2D4EggX7JRfMTMRplDULHGAuLtU3ImDZlukUOtoTm8dF4J5IGmADUcj0JLrSQDK1Jdg6YFaU7PIoq02leSuu181syNGCpmbkd6EiQQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=fOBqTwZD; arc=none smtp.client-ip=209.85.166.44 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="fOBqTwZD" Received: by mail-io1-f44.google.com with SMTP id ca18e2360f4ac-9491ba846b2so83648539f.2 for ; Wed, 19 Nov 2025 10:14:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1763576048; x=1764180848; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=t+WQ+WlElZR6b8foj+jTnRrBNTpepaWdsCyG6VTxCeY=; b=fOBqTwZDqBnDz/Vbs/qtBfpdCYGink4pV72SMpx0M0NrnziE32AVlonwfqA53WnnNn /Vbdwq5BfB10iXa1ur1dgE9gCinwVrEsQnvp+fHKIhND00nJhjGQnfge6iuAEZkqGCMb wR9KJ40YpR4a5eqJDuyQtNN07c2d3QzPetQrQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1763576048; x=1764180848; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=t+WQ+WlElZR6b8foj+jTnRrBNTpepaWdsCyG6VTxCeY=; b=U+V40nrKwsWhTYy3XGozXFb6u5BPURYc/yzaaZhYL/pcxvMj5okKxs7rWBnRCUXgaY YpRxHAoFULxTlWd4K+fUI2CZJXWQRg8CX2Ly6Y75Yxa1qupw6Nl6s1qEtermqyykvtnO zdizWnHTgeNqne2Ogob+0/1pVX6237Q//PBNlY3HEtkFYMySdJ4n8gKL2LbOPLhQGUyD p07YLygOtrWDEhE2RoRHOxci2SjXBZXags0cfrWOKEAcTpaHOowxet6mzp14e9+bKz58 2fMqCMxl8WscUSmIAs7+rCA0cJMWihlEeDL0txdUDscQ8fVf8YymfJyCAlgg3oeSzmxp 6rog== X-Forwarded-Encrypted: i=1; AJvYcCWZP8HomWuyPLh7j7gJ+up8oPO181CGQtgUpV9h+XR3PZnutNFPDK/dn5dRUFVKF8QPPjjA4nTeIS1rFu0=@vger.kernel.org X-Gm-Message-State: AOJu0Yz2d1IYuKDHAsmxH0VOD03MvHDSpwYiBpo/zTnpwvWQd1H0szLh aBmRXUSDuDP2J+g2pvfM5vk6ntzoiCUYyjxtjyvN7EArW8iudZAZLWHdG1lIFniKig== X-Gm-Gg: ASbGnctEjw58vkIsHQ6DQFaoodQ1ijL9HEcEYieWFhvCtlLbERPhzz+aP2b2ym9RA8S V+eoATG0w0cegLvMmhN32d7/y8hWfB1PXXL5Mx2tn4OFFDmS4QkqvPShbHZE6SEYIo73VIzkrH0 vrefvDn9r81HJ8ytMaeP9BThk/rjUr2BvznX5M2//NZ6UeXiO89z4GGWZwD2iS9w7uMa+gwY2ff HwzEsVujZ3sr0vCSOr1AEF00tJWusq7a4Fw4h2SHkMNa4icLobpFFZZkShxZZcO5j5GYDs7wPPM kqbExbxPWNVB/n2MaSFzo88oe6u3qGUZ5C6XOyz9ETUFr9b/CBnHf7JqR8no+B9bF7/O93cHU2c dz2SuddoRNY0eUdu+NEXNuefqDWVkBY7kxs6Jip6k23Cz6bpxvgfZLaAhjROfEGBMVc4yUy31vc b5o58b52GxWTL9hes= X-Google-Smtp-Source: AGHT+IEWVC1WUvwuvE7gE75ZizxaJkg7b27WdqYa4Fj1cP88bi/r3Jqe9CCCTezK2EVjIde2401KXw== X-Received: by 2002:a05:6602:4913:b0:938:9f22:ed34 with SMTP id ca18e2360f4ac-948e0d8eedamr1676190439f.16.1763576048598; Wed, 19 Nov 2025 10:14:08 -0800 (PST) Received: from chromium.org ([73.34.74.121]) by smtp.gmail.com with ESMTPSA id ca18e2360f4ac-949385ae254sm4838639f.1.2025.11.19.10.14.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Nov 2025 10:14:07 -0800 (PST) From: Simon Glass To: linux-arm-kernel@lists.infradead.org Cc: =?UTF-8?q?Thomas=20Wei=C3=9Fschuh?= , Masahiro Yamada , Tom Rini , Ahmad Fatoum , =?UTF-8?q?J=20=2E=20Neusch=C3=A4fer?= , Nicolas Schier , Chen-Yu Tsai , Simon Glass , Nicolas Schier , linux-kernel@vger.kernel.org Subject: [PATCH v6 3/8] scripts/make_fit: Move dtb processing into a function Date: Wed, 19 Nov 2025 11:13:24 -0700 Message-ID: <20251119181333.991099-4-sjg@chromium.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20251119181333.991099-1-sjg@chromium.org> References: <20251119181333.991099-1-sjg@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Since build_fit() is getting quite long, move the dtb processing into a separate function. Signed-off-by: Simon Glass Reviewed-by: Nicolas Schier Reviewed-by: Ahmad Fatoum --- (no changes since v1) scripts/make_fit.py | 67 +++++++++++++++++++++++++++++---------------- 1 file changed, 44 insertions(+), 23 deletions(-) diff --git a/scripts/make_fit.py b/scripts/make_fit.py index 984371f505bc..1a74a9dcd85e 100755 --- a/scripts/make_fit.py +++ b/scripts/make_fit.py @@ -277,6 +277,47 @@ def process_dtb(fname, args): =20 return (model, compat, files) =20 + +def _process_dtbs(args, fsw, entries, fdts): + """Process all DTB files and add them to the FIT + + Args: + args: Program arguments + fsw: FIT writer object + entries: List to append entries to + fdts: Dictionary of processed DTBs + + Returns: + tuple: + Number of files processed + Total size of files processed + """ + seq =3D 0 + size =3D 0 + for fname in args.dtbs: + # Ignore non-DTB (*.dtb) files + if os.path.splitext(fname)[1] !=3D '.dtb': + continue + + try: + (model, compat, files) =3D process_dtb(fname, args) + except Exception as e: + sys.stderr.write(f'Error processing {fname}:\n') + raise e + + for fn in files: + if fn not in fdts: + seq +=3D 1 + size +=3D os.path.getsize(fn) + output_dtb(fsw, seq, fn, args.arch, args.compress) + fdts[fn] =3D seq + + files_seq =3D [fdts[fn] for fn in files] + entries.append([model, compat, files_seq]) + + return seq, size + + def build_fit(args): """Build the FIT from the provided files and arguments =20 @@ -289,7 +330,6 @@ def build_fit(args): int: Number of configurations generated size: Total uncompressed size of data """ - seq =3D 0 size =3D 0 fsw =3D libfdt.FdtSw() setup_fit(fsw, args.name) @@ -310,34 +350,15 @@ def build_fit(args): size +=3D len(data) write_ramdisk(fsw, data, args) =20 - for fname in args.dtbs: - # Ignore non-DTB (*.dtb) files - if os.path.splitext(fname)[1] !=3D '.dtb': - continue - - try: - (model, compat, files) =3D process_dtb(fname, args) - except Exception as e: - sys.stderr.write(f"Error processing {fname}:\n") - raise e - - for fn in files: - if fn not in fdts: - seq +=3D 1 - size +=3D os.path.getsize(fn) - output_dtb(fsw, seq, fn, args.arch, args.compress) - fdts[fn] =3D seq - - files_seq =3D [fdts[fn] for fn in files] - - entries.append([model, compat, files_seq]) + count, fdt_size =3D _process_dtbs(args, fsw, entries, fdts) + size +=3D fdt_size =20 finish_fit(fsw, entries, bool(args.ramdisk)) =20 # Include the kernel itself in the returned file count fdt =3D fsw.as_fdt() fdt.pack() - return fdt.as_bytearray(), seq + 1 + bool(args.ramdisk), size + return fdt.as_bytearray(), count + 1 + bool(args.ramdisk), size =20 =20 def run_make_fit(): --=20 2.43.0 From nobody Tue Dec 2 02:19:52 2025 Received: from mail-io1-f50.google.com (mail-io1-f50.google.com [209.85.166.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E1569307AFB for ; Wed, 19 Nov 2025 18:14:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.50 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763576066; cv=none; b=OrZc7PhgauyzOeEN0bqS/6Q87uyoYkJ7fFnLwsrSKBo+th4HfqhDUTVJXBamKMJ+CZMbQQZyiUQKWLZVjyO0Yb3/NzauolNR+YhQCvecncTZsRXjooivxcFew8+TkhiMnMVy45E+Cji4LAn7R1h8lKLRkaS3xmVEW05p6ucb8+A= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763576066; c=relaxed/simple; bh=EeemFWJ3VEPTTKqxw+38LYskKYP/EN/4v6MeWplNysA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=dDBPei5Fazu+zeQaEubxyqGXQg9pw2aeeHqY8nXxQxxHLiNF4F4VPACBBNEy6BnaKeZcv9UqGh0H+QO5ByEv0HHUrh+hvjK1Ss0HEnqv7hxklxq+qpM6pOidR4GhwdxH5Ltn8m4ecPcUvu5nGdFHfmxbVruLxZDNgnQcucxs7cM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=NdmVk7k+; arc=none smtp.client-ip=209.85.166.50 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="NdmVk7k+" Received: by mail-io1-f50.google.com with SMTP id ca18e2360f4ac-9486adc1aa9so43139f.1 for ; Wed, 19 Nov 2025 10:14:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1763576051; x=1764180851; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=zF1Pzd7Y25nbnANuZLJhVbfb7fwymTpO79JMBdCEa0k=; b=NdmVk7k+HYBFwU7tpUjd6CFFJGvSWBItbc6mGXoROAtkGwIsAwP5BRMyoOA5tXPIgl aLv4yHZukW7zYEboJyhtOHfp6ddDhNou6x2nqRsv/QH23vbQhJwR+cuKkjKzBPF6P/fZ eXMpsvLp1kFlEuQy/+nlyZeZfhCCWDZe956OE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1763576051; x=1764180851; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=zF1Pzd7Y25nbnANuZLJhVbfb7fwymTpO79JMBdCEa0k=; b=ggsH7VwdSE82e5rYFaZDDy3fiZudQbl4Dayya/PSDlM4SB4HYDV9fZxerTjUDF1kzl RYQhh98KmZsqeq1SBC/Yyi20Jp6x9tpVi+K60VNUu5ZN4RLDTGviUQFyzw2oGPgZUO6O sGtyVzmng1XpSZNJw4Kjz0VKeCB5hu+UffH0pttJqQ+EtuRqGPvbc/mAT4M3BO5GX/5r tUC6qV45e9G7jNdLl8uUYANvqyGTj4h1X0q/Ea0EBnXq63fj0JBgzYgsSYWrOH5XwntR +I1Ief3FpKBMGjs5eCnxuYTKSqujoDYAUUaY+DsjaNVjHuZ/o1Jy+1NChBLh2FItZj1H lbUQ== X-Forwarded-Encrypted: i=1; AJvYcCUfU49bx8VrzFnYK/D1RxflgN5h1kl30w9D7dq7BeVVQFx3A63MQQf9rNJNG+TgeyWieFXylokh5cgS9x0=@vger.kernel.org X-Gm-Message-State: AOJu0YxO5gk+UElPQKWZ7ma9LnIWX7/6j82/VnA1f1nq4MCGgxjQcLTm e0J55rf9QcHDprYbEQMI+HDNVokiuZYCgdIobJ7XT4VTyZidxkxEyeBSTgVRJMANKw== X-Gm-Gg: ASbGncu2IKhXG2mAtaodkRDtlYNgzOSiUr9BpBAOA4+hMXX1KBIh+lqhQpOP9nfeDSj 9LAlaoexOReilFG1tmprqNaywtsYHebOfgcKn1QJd7A2XRNOi4JHaLVPYE7ibkGAaWOMoc1J3qR JzO8ANo4Z6LRGwfrz1PllINAl/Sw0QYhOL8TPG+Eks88nf+BJv2RyvfWZTsGGk+8WKI08y9c28e jZMrYZsKjBefx3QLINMpjcHjWVfpFnjMvSibiq6750Mo7ci40gvf5DFSJhvF6Up4q5sjTUkd56b 6fW4kUcaoqhFkvTLb9/+po0VaWUb/M6W/NM2hdvtk7CJC4/uckQwbQ8buOZmM3LVHtNU8Q4xNPI gaHb9ww8/XhqU+hY+hZHhYp1MjjxIV7YQzYbqYIRIE4nznZ8b0HUf6XcnnO/iaTgIY5+YlqedTP NxkPr0 X-Google-Smtp-Source: AGHT+IGXok4pX6SwDb8ES6pk1m2C6nMJbpP1DT6LiLX3sEVL1CGlpcIazdW8D38rS+pPrrxt+7nSNQ== X-Received: by 2002:a05:6602:168a:b0:949:92c:8049 with SMTP id ca18e2360f4ac-949092c8360mr1301383839f.12.1763576050905; Wed, 19 Nov 2025 10:14:10 -0800 (PST) Received: from chromium.org ([73.34.74.121]) by smtp.gmail.com with ESMTPSA id ca18e2360f4ac-949385ae254sm4838639f.1.2025.11.19.10.14.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Nov 2025 10:14:09 -0800 (PST) From: Simon Glass To: linux-arm-kernel@lists.infradead.org Cc: =?UTF-8?q?Thomas=20Wei=C3=9Fschuh?= , Masahiro Yamada , Tom Rini , Ahmad Fatoum , =?UTF-8?q?J=20=2E=20Neusch=C3=A4fer?= , Nicolas Schier , Chen-Yu Tsai , Simon Glass , linux-kernel@vger.kernel.org Subject: [PATCH v6 4/8] scripts/make_fit: Provide a way to add built modules Date: Wed, 19 Nov 2025 11:13:25 -0700 Message-ID: <20251119181333.991099-5-sjg@chromium.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20251119181333.991099-1-sjg@chromium.org> References: <20251119181333.991099-1-sjg@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Provide arguments to support building a ramdisk from a directory tree of modules. This is a convenient way to try out a kernel with its modules. This makes use usr/gen_initramfs.sh where it can be found. Failing that it uses the cpio tool rather than attempting to use a python module or our own code. The list of modules is provided in a file. Signed-off-by: Simon Glass Suggested-by: Ahmad Fatoum --- Note: The approach of depending on modules_install caused build problems as mentioned in the cover letter. I dropped the review tag from Nicolas Schier since there are quite a few changes in this version. Changes in v6: - Using the modules.order file instead of 'find' - Use gen_initramfs.sh where available Changes in v4: - Provide the list of modules from the Makefile - Reduce verbosity (don't print every module filename) Changes in v3: - Add a way to add built modules into the FIT scripts/make_fit.py | 127 +++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 119 insertions(+), 8 deletions(-) diff --git a/scripts/make_fit.py b/scripts/make_fit.py index 1a74a9dcd85e..9dfef11fc4b3 100755 --- a/scripts/make_fit.py +++ b/scripts/make_fit.py @@ -13,11 +13,17 @@ Usage: -r /boot/initrd.img-6.14.0-27-generic @arch/arm64/boot/dts/dtbs-li= st -E -c gzip =20 + # Build with modules ramdisk instead of external ramdisk: + make_fit.py -A arm64 -n 'Linux-6.17' -O linux + -o arch/arm64/boot/image.fit -k /tmp/kern/arch/arm64/boot/image.itk + -m module1.ko module2.ko module3.ko @arch/arm64/boot/dts/dtbs-list + Creates a FIT containing the supplied kernel, an optional ramdisk, and a s= et of devicetree files, either specified individually or listed in a file (with = an '@' prefix). =20 Use -r to specify an existing ramdisk/initrd file. +Use -m to build a ramdisk from specified kernel module files. =20 Use -E to generate an external FIT (where the data is placed after the FIT data structure). This allows parsing of the data without loading @@ -38,6 +44,7 @@ as U-Boot, Linuxboot, Tianocore, etc. import argparse import collections import os +import shutil import subprocess import sys import tempfile @@ -83,8 +90,14 @@ def parse_args(): help=3D'Specifies the operating system') parser.add_argument('-k', '--kernel', type=3Dstr, required=3DTrue, help=3D'Specifies the (uncompressed) kernel input file (.itk)') - parser.add_argument('-r', '--ramdisk', type=3Dstr, + + # Create mutually exclusive group for ramdisk options + rd_group =3D parser.add_mutually_exclusive_group() + rd_group.add_argument('-r', '--ramdisk', type=3Dstr, help=3D'Specifies the ramdisk/initrd input file') + rd_group.add_argument('-m', '--modules', type=3Dstr, nargs=3D'+', + help=3D'List of module filenames to include in ramdisk') + parser.add_argument('-v', '--verbose', action=3D'store_true', help=3D'Enable verbose output') parser.add_argument('dtbs', type=3Dstr, nargs=3D'*', @@ -240,6 +253,89 @@ def output_dtb(fsw, seq, fname, arch, compress): fsw.property('data', compressed) =20 =20 +def find_gen_initramfs(): + """Find the gen_initramfs.sh script + + Looks for usr/gen_initramfs.sh relative to the current directory, + or in the source directory if running from a build directory. + + Returns: + str: Path to gen_initramfs.sh if found, None otherwise + """ + path =3D 'usr/gen_initramfs.sh' + if not os.path.exists(path): + srctree =3D os.environ.get('srctree') + if not srctree: + return None + path =3D os.path.join(srctree, path) + if not os.path.exists(path): + return None + + return path + + +def build_ramdisk(args, tmpdir): + """Build a cpio ramdisk containing kernel modules + + Similar to mkinitramfs, this creates a compressed cpio-archive contain= ing + the kernel modules for the current kernel version. + + Args: + args (Namespace): Program arguments + tmpdir (str): Temporary directory to use for modules installation + + Returns: + tuple: + bytes: Compressed cpio data containing modules + int: total uncompressed size + """ + suppress =3D None if args.verbose else subprocess.DEVNULL + + if args.verbose: + print(f'Copying {len(args.modules)} modules to ramdisk') + + # Create output-directory structure + outdir =3D os.path.join(tmpdir, 'initramfs') + modules_dir =3D os.path.join(outdir, 'usr', 'lib', 'modules') + os.makedirs(modules_dir, exist_ok=3DTrue) + + # Copy in the specified modules, renaming .o to .ko + for module in args.modules: + name, ext =3D os.path.splitext(module) + dest =3D name + '.ko' if ext =3D=3D '.o' else module + + # Preserve directory structure under modules_dir + dest_path =3D os.path.join(modules_dir, dest) + os.makedirs(os.path.dirname(dest_path), exist_ok=3DTrue) + shutil.copy2(module, dest_path) + + if args.verbose: + print(f'Creating cpio archive from {outdir}') + + with tempfile.NamedTemporaryFile() as cpio_file: + # Use gen_initramfs.sh to create the cpio archive if possible + gen_initramfs =3D find_gen_initramfs() + if gen_initramfs: + subprocess.run([gen_initramfs, '-o', cpio_file.name, outdir], + check=3DTrue, stdout=3Dsuppress, stderr=3Dsuppr= ess) + else: + # Change to initramfs directory and create cpio archive + with subprocess.Popen(['find', '.', '-print0'], cwd=3Doutdir, + stdout=3Dsubprocess.PIPE) as find: + with subprocess.Popen(['cpio', '-o', '-0', '-H', 'newc'], + stdin=3Dfind.stdout, stdout=3Dcpio_f= ile, + stderr=3Dsuppress, cwd=3Doutdir) as = cpio: + find.stdout.close() + cpio.wait() + find.wait() + + cpio_file.seek(0) # Reset to beginning for reading + if args.verbose: + print('Reading ramdisk...' if args.compress =3D=3D 'none' else + f'Compressing ramdisk with {args.compress}...') + return compress_data(cpio_file, args.compress), cpio_file.tell() + + def process_dtb(fname, args): """Process an input DTB, decomposing it if requested and is possible =20 @@ -318,11 +414,12 @@ def _process_dtbs(args, fsw, entries, fdts): return seq, size =20 =20 -def build_fit(args): +def build_fit(args, tmpdir): """Build the FIT from the provided files and arguments =20 Args: args (Namespace): Program arguments + tmpdir (str): Temporary directory for any temporary files =20 Returns: tuple: @@ -344,20 +441,29 @@ def build_fit(args): =20 # Handle the ramdisk if provided. Compression is not supported as it is # already compressed. + ramdisk_data =3D None if args.ramdisk: with open(args.ramdisk, 'rb') as inf: - data =3D inf.read() - size +=3D len(data) - write_ramdisk(fsw, data, args) + ramdisk_data =3D inf.read() + size +=3D len(ramdisk_data) + elif args.modules: + if args.verbose: + print('Building modules ramdisk...') + ramdisk_data, uncomp_size =3D build_ramdisk(args, tmpdir) + size +=3D uncomp_size + + if ramdisk_data: + write_ramdisk(fsw, ramdisk_data, args) =20 count, fdt_size =3D _process_dtbs(args, fsw, entries, fdts) size +=3D fdt_size =20 - finish_fit(fsw, entries, bool(args.ramdisk)) + finish_fit(fsw, entries, has_ramdisk=3Dbool(ramdisk_data)) =20 - # Include the kernel itself in the returned file count fdt =3D fsw.as_fdt() fdt.pack() + + # Count FDT files, kernel, plus ramdisk if present return fdt.as_bytearray(), count + 1 + bool(args.ramdisk), size =20 =20 @@ -365,7 +471,12 @@ def run_make_fit(): """Run the tool's main logic""" args =3D parse_args() =20 - out_data, count, size =3D build_fit(args) + tmpdir =3D tempfile.mkdtemp(prefix=3D'make_fit_') + try: + out_data, count, size =3D build_fit(args, tmpdir) + finally: + shutil.rmtree(tmpdir) + with open(args.output, 'wb') as outf: outf.write(out_data) =20 --=20 2.43.0 From nobody Tue Dec 2 02:19:52 2025 Received: from mail-il1-f174.google.com (mail-il1-f174.google.com [209.85.166.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9E72D364E89 for ; Wed, 19 Nov 2025 18:14:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763576072; cv=none; b=r4V6MNW76O8TFvmE8+Tuv+MR+qymLsomNmPL4g07GZtax3+zSvWIov+ZRrPgI3r7K3yvFLvL5wAPRAnMeKRqQvhQep+6UyoSLgBwddIRi2PqbDglU+IpZnCly3EJgjW7o5cedHCMMXXR4RfH/DSW6W4B4+LWujzoAWreKfOj4RE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763576072; c=relaxed/simple; bh=fOaYndhzO7GXOWo11e/wLosmESF6vtdq2dn5BPwRTXU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=cs9UcXKzPxMQpdr3Ug7/9XCxtyXFKonomHQaJ0PsVyPcLKLCp4CWnRC2T/+gLXplCHmSJJrIyCo62syjpO7+AB5ZuG3tqCI9BPWlQAApxZzuTiHJma0wGxG1lyhGVfomAyNAokfoMkkUedyz2HEWXKuXESYOOGLUOIRRUB8UxvA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=jWBuzbhK; arc=none smtp.client-ip=209.85.166.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="jWBuzbhK" Received: by mail-il1-f174.google.com with SMTP id e9e14a558f8ab-4349044be0eso341875ab.1 for ; Wed, 19 Nov 2025 10:14:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1763576055; x=1764180855; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=RPirOi0w0Cb/s6FnCbMEuNgCnH0qPfOw0gyvUWbzUDU=; b=jWBuzbhKm7GmCR4B66v1R+1kHtaBSiBzlrhRU4GLgcpLANPgD/EM4Td1RUcdi9wae8 SJioEUcQO6Sf41HoOUwf/ZkZOwuR7VXC0DoIBNFepXtNw0BEEeGGGJGeTdmi5FlgBGoH 52CZHvipqomWdrTxgOkCcWmRCx/ahmCblq+x0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1763576055; x=1764180855; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=RPirOi0w0Cb/s6FnCbMEuNgCnH0qPfOw0gyvUWbzUDU=; b=EJojHVSjsm3QKmbtlmEKEVuVkrQUc7sYJZte939maSM5qZwczh4QDtlzX7aqvE8yWq 9EvrVJHeTD/8gn+hUY0qwxoHxl0MPrBC1X2CTZwonx1caThaYATlJk1OCi9lfVc9fGYN B+axiay0Y8/+oEm2dg9mcGoD5DOLHBEBxe/SDnqPbQrfxbj2YgR04ZcDJpjSKv+gMLlz v4M5aSzH1LWyRCfZY/nWZCqHSm0o1hLuwVSboIO9iuwQX6s8Y9B9HDLijvNcT81M+hgY EYFYu5iCI9vPDANCFQTww9pj/ZB+lWtvRMlLlH6YaYxYkiKALXrGvdjsB29yn5p42Zxf 8O7w== X-Forwarded-Encrypted: i=1; AJvYcCXWJIXNUanQL5pOym4N6vIcBMdwkCFGAnrC5dV2qXjM96Q5TH+6eZCYy07BF6j5GPDD6qHZ+bcu3tselZk=@vger.kernel.org X-Gm-Message-State: AOJu0Yyrw87VBeyFE4ErB0SAeq4BTwdHSVnhCworohKQnOum35gOEhls k9nsbmLss/gAmpgyy7MTMP1LXlS+HwCouiTw3YP8Nn11RgpOHyKYWx3ns580wzDPAg== X-Gm-Gg: ASbGnctTaTAiz+6wKngYDoMk5sSR+4bRbZVW6FKxtLVovyg+5SN0ELwRo/VDk4Y9NB6 G6wFuhXL+Z6Pd8acoSbDvL1RDj77CJ1CWArDTVl36sq4PPNFZokCI+/zCAt/1iGkQsJM8iYxYio bFKrIE4yuIZBS1JRSBIYZax+fRelpEshR5yDrTDDYLqf1zeU2narNU7neYoeohXZbM4DW3eBfdF 0twkdwQZudu1VSKCYjs0fp6OpJPVZoiQLYGTuDAibtaa5RUoxQleXtT7wQYWDLdGOLhTEmD0hVn /ygvmjsv9lYEcZ9LaOk/gr3ub8JDjjXGpDFijxCrxnsrHLt4M27437wa/hpgpip/AETwQaS1WuF 5cRpKn3WijcJomSXwypl/ee/UvwZ8nrZSTi/hWTVDsgISbglqHCFuysWcemiM35FC6oyoGPYb0f tbX2TG X-Google-Smtp-Source: AGHT+IEqawcvOIPqsXpALDErL/1Go+TjtEXbiX8/+Jw/9VSxcMjIN9Ju3r68ajG6bVKTrOJBVHHikA== X-Received: by 2002:a05:6e02:3398:b0:433:7f13:3a8c with SMTP id e9e14a558f8ab-435a902d081mr4341845ab.16.1763576054629; Wed, 19 Nov 2025 10:14:14 -0800 (PST) Received: from chromium.org ([73.34.74.121]) by smtp.gmail.com with ESMTPSA id ca18e2360f4ac-949385ae254sm4838639f.1.2025.11.19.10.14.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Nov 2025 10:14:12 -0800 (PST) From: Simon Glass To: linux-arm-kernel@lists.infradead.org Cc: =?UTF-8?q?Thomas=20Wei=C3=9Fschuh?= , Masahiro Yamada , Tom Rini , Ahmad Fatoum , =?UTF-8?q?J=20=2E=20Neusch=C3=A4fer?= , Nicolas Schier , Chen-Yu Tsai , Simon Glass , Bill Wendling , Justin Stitt , Miguel Ojeda , Nathan Chancellor , Nick Desaulniers , Nicolas Schier , Tamir Duberstein , linux-kbuild@vger.kernel.org, linux-kernel@vger.kernel.org, llvm@lists.linux.dev Subject: [PATCH v6 5/8] kbuild: Split out module targets into a variable Date: Wed, 19 Nov 2025 11:13:26 -0700 Message-ID: <20251119181333.991099-6-sjg@chromium.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20251119181333.991099-1-sjg@chromium.org> References: <20251119181333.991099-1-sjg@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add a modules-targets variable to list the targets which cause modules to be built, since we want to add a conditional target. Signed-off-by: Simon Glass Reviewed-by: Ahmad Fatoum Reviewed-by: Nicolas Schier --- (no changes since v5) Changes in v5: - Add a new patch to split out module targets into a variable Makefile | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/Makefile b/Makefile index 638bc09a546a..8cd46222fc48 100644 --- a/Makefile +++ b/Makefile @@ -772,7 +772,12 @@ endif # in addition to whatever we do anyway. # Just "make" or "make all" shall build modules as well =20 -ifneq ($(filter all modules nsdeps compile_commands.json clang-%,$(MAKECMD= GOALS)),) +modules-targets :=3D all +modules-targets +=3D modules +modules-targets +=3D nsdeps +modules-targets +=3D compile_commands.json +modules-targets +=3D clang-% +ifneq ($(filter $(modules-targets),$(MAKECMDGOALS)),) KBUILD_MODULES :=3D y endif =20 --=20 2.43.0 From nobody Tue Dec 2 02:19:52 2025 Received: from mail-io1-f42.google.com (mail-io1-f42.google.com [209.85.166.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A5E9E3054F9 for ; Wed, 19 Nov 2025 18:14:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.42 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763576069; cv=none; b=oAdePk4UdTCiekeQOBXDR5F3grAeR3jgZSo7bg7Jqc8hvShlDnf9dK5Ke0EX3NJd4CJxSkmCqoUKRpVf5krpKbam5PSv5BeKXQwuJ5yyyyVynOYL5ROqz/RlfgzJPeZRAgAaXDiK1T2L/qHHYYAgX50HwUr0OpTX18zFKSMIuFA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763576069; c=relaxed/simple; bh=N9znOiAe5PJwsEoCpATqeH68yT6+oiXSUDjwEep8rqE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=oTcp0IzXU1iKwSwXvGj/nsHN4XcVSa8PHMu8BkxqKxGVJoUCPkaUth5oaaG9DBbfgmqxj9ffE9FCPud5IFc5LRvzZsJdLxxxO7z16+DRJ9NjPvptYB6wHOf4Rna0bVvGAggqhUaL8xsIR4csmmvUvBgSD7UtWVHj9Hhhp28cbSw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=HU9Idn5S; arc=none smtp.client-ip=209.85.166.42 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="HU9Idn5S" Received: by mail-io1-f42.google.com with SMTP id ca18e2360f4ac-948f8fa9451so137080539f.3 for ; Wed, 19 Nov 2025 10:14:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1763576057; x=1764180857; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=4cU75xuJ/5xIhcW50wA4hHS4Fk7gDNqRb9uRUlMbq3I=; b=HU9Idn5Sc0tWMN4qxsg6yLFHv93vAMQ/I8sgSVGYpPdM6jF0yzLOzARkhO0n9KIHKB 374OVxf5P7PZObVGHap9Db1pmONd6uOjn1SvmwzBQLIMX+cpAsktg/snLcKySHLupyDt mvtCnGqdjYGGfw2zV9QwTX2Gtg7Zw/TY/uTZM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1763576057; x=1764180857; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=4cU75xuJ/5xIhcW50wA4hHS4Fk7gDNqRb9uRUlMbq3I=; b=vX2B9LUZA5NYbzZJ5vpPZ8p4n4mnsQ7yMAyklySj3ne60d0teEiA62POXtFr11zN5g cLNavVie/vYiLlrCWUul+oHnIBN63Yj2aXRI6/CfGpj6T51t2qbcpB7FyQwPPc4XznRj lR8+WsNvrn6t+lcyMl3c87TbnuIhzKC1w/3K89E4jY7aFzEafXX0K6M3yt9RKrf4rqCH GENijrLWWzPmyQ/t9PhhPt3OL4bfrBHuhrCkUxMHLGPj1pNLnUPY75btXge7muiZlJqg /Xx9MTd6Mu92qBM3Pcc3KNLbEERAiuzJMcRixLriCow+ry2Pe7Eah5+/LL5YYkfvRuvO qGgQ== X-Forwarded-Encrypted: i=1; AJvYcCXnV/acMYf0rEPrlZzyYQ72ZiFT57mFwxcIpAHDM9eG1BVT7OIkQ3xNvph6lwLj6ap74UH9K0Yu6EIwzuw=@vger.kernel.org X-Gm-Message-State: AOJu0Yz1o2jXHOFHTfZu3RV28xBsgIBI3mnzvXh4ro9ZV6+kQLQpSNnu tP8xsdvkcup+tHNIq4qlcuhFCB7mZpI5PBEWQKvkPBOtTcRYfS0pRpRvHf2T9TpwBw== X-Gm-Gg: ASbGncv5v5bkRC/7gXsU1NHHqZyGKu3EbOi3Ks0+dE5yye43rqH5St0SYDSG5Lfd0HW bhUF2DA6DQv278e4LeeiApMJMOg3Es5wbRbc2hPshK13tzr9tGfMbVdsAmFOKrQel67XiOvbvPW uQqGvivCjv+LLiY23LcXuNwBdcXb6HS8VD/Wp9T17wFrvoY/NpFfpvavZv/xVYqOBmuys9SGY0f 4yqeG1NX5iCzGdj7IRtn32lo1Tf2I5O6BrMnq0V4gKp3d46LvjGpGktBvzV1lDtIGRJkmGk6AVt QdV0gb0V0loXoNzMQhP2cHR0k4gaaktzyLY91crz/pVfy1BbGPHALehKEbopW21BOKEfDl6hSkp wXhh1vJfZbyNfYbSq0RJw4Lwqjy0xHGBhxqSY8wa3HtWfVIHnwdhMyLCOGC9BIGkrFy5QJfym2j qAMMuE X-Google-Smtp-Source: AGHT+IEIAcWoFGwcNgccxAP828+yd7/kMEFFY+sH8+j/iQjjBsmYniUh67wbp4Ar9H6A3QqKNZyn4Q== X-Received: by 2002:a05:6602:1607:b0:948:e11f:5b4c with SMTP id ca18e2360f4ac-9492b6ed063mr334288439f.19.1763576057415; Wed, 19 Nov 2025 10:14:17 -0800 (PST) Received: from chromium.org ([73.34.74.121]) by smtp.gmail.com with ESMTPSA id ca18e2360f4ac-949385ae254sm4838639f.1.2025.11.19.10.14.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Nov 2025 10:14:16 -0800 (PST) From: Simon Glass To: linux-arm-kernel@lists.infradead.org Cc: =?UTF-8?q?Thomas=20Wei=C3=9Fschuh?= , Masahiro Yamada , Tom Rini , Ahmad Fatoum , =?UTF-8?q?J=20=2E=20Neusch=C3=A4fer?= , Nicolas Schier , Chen-Yu Tsai , Simon Glass , "Reviewed-by : Nicolas Schier" , Nathan Chancellor , Ard Biesheuvel , Catalin Marinas , Josh Poimboeuf , Kees Cook , Miguel Ojeda , Nicolas Schier , Rong Xu , Tamir Duberstein , Will Deacon , linux-kbuild@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v6 6/8] kbuild: Allow adding modules into the FIT ramdisk Date: Wed, 19 Nov 2025 11:13:27 -0700 Message-ID: <20251119181333.991099-7-sjg@chromium.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20251119181333.991099-1-sjg@chromium.org> References: <20251119181333.991099-1-sjg@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Support 'make image.fit FIT_MODULES=3D1' to put all the modules into a ramdisk image within the FIT. Add image.fit as a target which requires modules, so that modules will built automatically when using FIT_MODULES is not empty. Signed-off-by: Simon Glass Suggested-by: Ahmad Fatoum Suggested-by: Reviewed-by: Nicolas Schier Acked-by: Nathan Chancellor Reviewed-by: Nicolas Schier --- Changes in v6: - Mention that FIT_MODULES just needs to be non-empty - Make use of modules.order instead of using 'find' Changes in v5: - Build modules automatically if needed (fix from Nicolas Schier) Changes in v4: - Rename the Makefile variable from 'EXTRA' to 'MAKE_FIT_FLAGS' - Use an empty FIT_MODULES to disable the feature, instead of '0' - Make use of the 'modules' dependency to ensure modules are built - Pass the list of modules to the script Makefile | 1 + arch/arm64/Makefile | 1 + scripts/Makefile.lib | 6 +++++- 3 files changed, 7 insertions(+), 1 deletion(-) diff --git a/Makefile b/Makefile index 8cd46222fc48..4eccaef95826 100644 --- a/Makefile +++ b/Makefile @@ -773,6 +773,7 @@ endif # Just "make" or "make all" shall build modules as well =20 modules-targets :=3D all +modules-targets +=3D $(if $(FIT_MODULES),image.fit) modules-targets +=3D modules modules-targets +=3D nsdeps modules-targets +=3D compile_commands.json diff --git a/arch/arm64/Makefile b/arch/arm64/Makefile index 73a10f65ce8b..7036f251ab40 100644 --- a/arch/arm64/Makefile +++ b/arch/arm64/Makefile @@ -174,6 +174,7 @@ endif all: $(notdir $(KBUILD_IMAGE)) =20 image.fit: dtbs +image.fit: $(if $(FIT_MODULES),modules) =20 vmlinuz.efi image.fit: Image $(BOOT_TARGETS): vmlinux diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib index 1d581ba5df66..28e0cc0865b1 100644 --- a/scripts/Makefile.lib +++ b/scripts/Makefile.lib @@ -398,11 +398,15 @@ MAKE_FIT :=3D $(srctree)/scripts/make_fit.py # Use this to override the compression algorithm FIT_COMPRESSION ?=3D gzip =20 +# Set this to non-empty to include an initrd with all the kernel modules +FIT_MODULES ?=3D + quiet_cmd_fit =3D FIT $@ cmd_fit =3D $(MAKE_FIT) -o $@ --arch $(UIMAGE_ARCH) --os linux \ - --name '$(UIMAGE_NAME)' \ + --name '$(UIMAGE_NAME)' $(MAKE_FIT_FLAGS) \ $(if $(findstring 1,$(KBUILD_VERBOSE)),-v) \ $(if $(FIT_DECOMPOSE_DTBS),--decompose-dtbs) \ + $(if $(FIT_MODULES),--modules @$(objtree)/modules.order) \ --compress $(FIT_COMPRESSION) -k $< @$(word 2,$^) =20 # XZ --=20 2.43.0 From nobody Tue Dec 2 02:19:52 2025 Received: from mail-io1-f54.google.com (mail-io1-f54.google.com [209.85.166.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A914C310784 for ; Wed, 19 Nov 2025 18:14:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.54 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763576068; cv=none; b=S8Yzigx37rjnWypM+kWbRzbWjjaSG/PMvkb3fmBI/JkTAHwm8b4gVyJt5OjVu8G/OM6kLjasV/ekM6sXLUK4+vltUM4zl7Hfx0oj4Sr+JM38AbE2nOsEAQUkC51cOTsRjLyZ/igBNvewIPKRCfm8Nx+h0Vo7x+8iC5ccvGbI62U= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763576068; c=relaxed/simple; bh=qjmpXKLSqaLLlWEExSmghH/nPTqlb57sqeHubFiGdmY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=TthbG0ICYu5hJ9g3X2A2MnC1il4j/fv8vanyCl6O3TyaJEu8M3QEj58DBJp3y6XrbrE+XZeKC3kdSxw2geh307DmYBeEUghEF3YHYS7eClv175fSCM8E7b5UTX+hw0xZqusvRWGZlIUmm4QVz2QkucyAuXWqi+2Jh0874bJ28u4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=eGdo3hR0; arc=none smtp.client-ip=209.85.166.54 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="eGdo3hR0" Received: by mail-io1-f54.google.com with SMTP id ca18e2360f4ac-9491604d00fso118166939f.2 for ; Wed, 19 Nov 2025 10:14:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1763576060; x=1764180860; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=/Kb7vK4VelnueHMlrM8ApAVfsFwcfEWfXn4uz9Oo23Y=; b=eGdo3hR0Zs7Id2hq0evYkn34NEqt1/UcabOXK/N76oX3SbAqhRbJg2KgTvEK3lBcIk ujK0wD8yAFCCDsPUeVarF2MK7wiNpWu30da1jfFtxgUzDArHpPc2Utj3UKuvlWaplz5Z V9J6Dz4HxrM3Dr3WbJxIkhgYC7x7H+oRrBrL0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1763576060; x=1764180860; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=/Kb7vK4VelnueHMlrM8ApAVfsFwcfEWfXn4uz9Oo23Y=; b=Wl6N15eCLs2CuaMuqwuOTh2bendqi3sw9TuR3VwlUtkFbw87X+1bAF7HYavlQ2bFuJ zIprv0fKGNoSdNgXL7uwGxylZ+qrgAjNSegn3T7I3eXYNuvLlD+F6yZRf43JVi3Ydpxp 7uNDc4xsyFKxcIXW/01m5EnKF+/iqiDpcVv6MFBLZvKdRGee2LY4Tx2k4F8gx6mlj0fi oFCcJ70GfLxIn1KdkelHoDNcM6zfFJ6O4yWpi6pgR2w7F5hlVTVSc/w8vWreaQnImSy1 eg5KghxPXi3uABz3PHH1xxBhgRbuCyoixP3WwvCpBTC1blZP1tkGtdHquwlATpDPoK+d UNTQ== X-Forwarded-Encrypted: i=1; AJvYcCWjPXjk9WVv6EKDE9ELQZSjkrYL6dg4Q9SiUxAygiBAMgskvkLis0z8GTiLVYdig+NdrS7tEVthnoIWOPg=@vger.kernel.org X-Gm-Message-State: AOJu0Ywnt/WjpkLcJmO/sXuiBDGu1ZxAEymlk54c+fYiqlI3Od4gOGND VJgqcHZyLz5OPKAlLcuoXnO7SKVSOy8wVwCFjIDfS2BBcWDKjKw4I8xriM+pyylGFw== X-Gm-Gg: ASbGncv/MEUMkUQsAK2Iq57hvWGaadQLAE73DqwaE6/HbM4Dtgv6lHFkHCmxvdslg08 o24cJAEUHg2S/zIoFE4I8WLnVe6FmRUgrRN9h3QhXTOWy9A4xx4MR7lOdgoECHfREcdOrxelv0o 3JE4NrAMHpfZ+6ZQZkJ1XAkFiWwTUkdUn+QT776a/2hzitvsFXHjiSc5O12XBdjxdyqgKyamvnx qhWFiFnYo32rjE2/MHu6HYsaNoc2cAVfe2Ot+IXxRmSNkbKEGjZ7pU+N4SuTNx4lUFdEMdUFDJH 4PNv2uZ6VwZe5+0PxjAfKkYTX4fPyF3ph0tAW0jg0p/ua/0j8+fm9zCtf04b9hRC3ogRmB2YNcB 0BmDLv5AD0i6xghUioW4JKlEbMwgK0OtJhIkq3wWkrO0qsNDwHTE5jkldqBEg253QzrzlDNXTJw Oo4n3n X-Google-Smtp-Source: AGHT+IEW7AT1anDDeb5eUBJlU592g0AeT8YdBL/KDYsyZnupxjh5lyG6UUoFw9lH3XXKvCmP33BRyg== X-Received: by 2002:a05:6602:2b0b:b0:949:1123:13b1 with SMTP id ca18e2360f4ac-94911231590mr1034792539f.19.1763576060105; Wed, 19 Nov 2025 10:14:20 -0800 (PST) Received: from chromium.org ([73.34.74.121]) by smtp.gmail.com with ESMTPSA id ca18e2360f4ac-949385ae254sm4838639f.1.2025.11.19.10.14.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Nov 2025 10:14:18 -0800 (PST) From: Simon Glass To: linux-arm-kernel@lists.infradead.org Cc: =?UTF-8?q?Thomas=20Wei=C3=9Fschuh?= , Masahiro Yamada , Tom Rini , Ahmad Fatoum , =?UTF-8?q?J=20=2E=20Neusch=C3=A4fer?= , Nicolas Schier , Chen-Yu Tsai , Simon Glass , David Sterba , Nick Terrell , linux-kernel@vger.kernel.org Subject: [PATCH v6 7/8] scripts/make_fit: Support a few more parallel compressors Date: Wed, 19 Nov 2025 11:13:28 -0700 Message-ID: <20251119181333.991099-8-sjg@chromium.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20251119181333.991099-1-sjg@chromium.org> References: <20251119181333.991099-1-sjg@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add support for pbzip2 and plzip which can compress in parallel. This speeds up the ramdisk compression. Signed-off-by: Simon Glass --- (no changes since v1) scripts/make_fit.py | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/scripts/make_fit.py b/scripts/make_fit.py index 9dfef11fc4b3..64038f17a51e 100755 --- a/scripts/make_fit.py +++ b/scripts/make_fit.py @@ -57,10 +57,10 @@ import libfdt CompTool =3D collections.namedtuple('CompTool', 'ext,tools') =20 COMP_TOOLS =3D { - 'bzip2': CompTool('.bz2', 'bzip2'), + 'bzip2': CompTool('.bz2', 'pbzip2,bzip2'), 'gzip': CompTool('.gz', 'pigz,gzip'), 'lz4': CompTool('.lz4', 'lz4'), - 'lzma': CompTool('.lzma', 'lzma'), + 'lzma': CompTool('.lzma', 'plzip,lzma'), 'lzo': CompTool('.lzo', 'lzop'), 'zstd': CompTool('.zstd', 'zstd'), } @@ -220,7 +220,12 @@ def compress_data(inf, compress): done =3D False for tool in comp.tools.split(','): try: - subprocess.call([tool, '-c'], stdin=3Dinf, stdout=3Dou= tf) + # Add parallel flags for tools that support them + cmd =3D [tool] + if tool in ('zstd', 'xz'): + cmd.extend(['-T0']) # Use all available cores + cmd.append('-c') + subprocess.call(cmd, stdin=3Dinf, stdout=3Doutf) done =3D True break except FileNotFoundError: --=20 2.43.0 From nobody Tue Dec 2 02:19:52 2025 Received: from mail-io1-f49.google.com (mail-io1-f49.google.com [209.85.166.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 03A1831062C for ; Wed, 19 Nov 2025 18:14:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.49 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763576089; cv=none; b=TDUEsV0sbaDvloqE/PYIcRUL4SLvCX7RpnLDppkUy53tcPoQ5xkQv3AHEQT0I52IpHCPfxgOpqCxk4kfwaq1FPcLwZzbQ0dtw1H9kghR4ScwEsIEqzfSA/2l+rjzwZy5uwC5zMzvlRWo9MCkdiqo083LO/N0FOy7hKkT+u+VgAs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763576089; c=relaxed/simple; bh=a8gdpWOOLE5LiK7plSUbvBIWdhXzlJGSmDcnpnDBAF0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=ih7Xvb1e7e10ubul6QIjXOyEE3xeSgAijpKyWm+QU/WThItiFumVJtYuLtAOiObG7ohA9KdtO3SC4OvZzkC4/S+ceK3mVbusa6ch4iQiL32YSAkxB2kHYicqkIg+U2ZF6bGk6PO7ApD6VyAuMmbTw8rY7hMwJCNrurX+jQPEshI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=K59a6naT; arc=none smtp.client-ip=209.85.166.49 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="K59a6naT" Received: by mail-io1-f49.google.com with SMTP id ca18e2360f4ac-948f9b5987dso556439f.0 for ; Wed, 19 Nov 2025 10:14:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1763576063; x=1764180863; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=dHy+9knFfat2mLLa5PVf4uFPqoXtGTRpTYZkU+z6PsU=; b=K59a6naTiKD0Y7zFpdqf0UN9+H8FJjIYT6nASIlY2e/zILICwd5AHKivwHVtR1geT+ 9KcGmRNrKfj/z8z1bZ3iyiOZTFQcVOvFkbXrl+wm2RCUCggsb2T+k1rJ98UnMI4MCb/v A6AxF+qHb6pRpa7KWiX9a2oDSa/Sd4EhywCwE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1763576063; x=1764180863; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=dHy+9knFfat2mLLa5PVf4uFPqoXtGTRpTYZkU+z6PsU=; b=NuYSXcVxygs0YD5qOIVtd+h4yfND7y2oBhfRu+0rD6+OvLcv3g2z13ORwxWatHOmLW Z6cKQGNDRGC79CRfpLDMyrajaayupK5lLSzIa1Ls5gUqCtTfSfdiS68x+LXcJocIezYz Szoz8oHpP+ApiUbhzwVA8QNXFXhrj9jUmgqZwRYIsRfj+/Re2kSnnVOi7x1RtGLcDXXE KSWEt1LDUL5XP5UrCBpKa40l31TkhKCBqDkNogPauHkufgpMW/dvSW7KEO9aD7TE36p6 VtfmYNIdaPxGFApfaNpZW68339jv0bkA55bUpLs1vxT9SYckGjJwAiAVX87iv98RuA+w VNqw== X-Forwarded-Encrypted: i=1; AJvYcCVUaCwhOKqry+x8/9yIzwKwq30iOJOIdlzgs8q6Gxj0fFKFBrM2XM/rau0KAJ/LE72AuNxECYyfbcWScPI=@vger.kernel.org X-Gm-Message-State: AOJu0Yz3zBtP0R/BwnDNqV6d0WFuRLXZOQnMsvkuaYKYI9haXxpJMG2Q XCowjVCqPs28SMd0pUgzYKbXGAMJd+pPkZQ9M1na0oolmsAFRAGOsbps2cepDJvtFA== X-Gm-Gg: ASbGncuOadHZmk6ZO7eyI5XDQ8mHVPrkiNlg96m/WBbjYkSHFLZisBpfPMNXhMG4A0f Tr7rdHchrUtLahb5R+1Xv2FZIybbZLY6fyTzmfHLJfUeTq3twbgxfSXuPdfi3RQTVfVwjj8qggM 0fBAXZ9MvKrXLIjqTdL9ztghSmEf1ScvQl3SxSVYRGCK0zxAzVCUIjNYbhVUIqQ6EERqHOIKaFW L4FBQWwPjbbyiv9MGTJ+rnliO5vJ80Mj5tRRBdSJmnWW/o6m4k3fYZaRF/51WZiifNZtqz3MUau K+5vm8OwBBQDeLY1/ue3AaHNY9XEP7F+WxEO0Zd9VZORdW0OStRoxcu/1CwzP0/HoBQGef/YL+E Sln3+04pegj+W4XTbUQiP2qLubHDG2E1IktBlLFMK5O4IjpmwPFyWWNoTFMt8qSKt+pkNrLUzm2 v76X3A X-Google-Smtp-Source: AGHT+IGaRIx4ZLtKiudNzXMDULOFDlM2zIQYOJnUxhKiqXGAH6TvITMlqEdCQfP7NTjCb2mlaYsLtg== X-Received: by 2002:a05:6602:2b84:b0:948:a3ca:15a3 with SMTP id ca18e2360f4ac-948e0deae37mr2850382839f.18.1763576063169; Wed, 19 Nov 2025 10:14:23 -0800 (PST) Received: from chromium.org ([73.34.74.121]) by smtp.gmail.com with ESMTPSA id ca18e2360f4ac-949385ae254sm4838639f.1.2025.11.19.10.14.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Nov 2025 10:14:21 -0800 (PST) From: Simon Glass To: linux-arm-kernel@lists.infradead.org Cc: =?UTF-8?q?Thomas=20Wei=C3=9Fschuh?= , Masahiro Yamada , Tom Rini , Ahmad Fatoum , =?UTF-8?q?J=20=2E=20Neusch=C3=A4fer?= , Nicolas Schier , Chen-Yu Tsai , Simon Glass , linux-kernel@vger.kernel.org Subject: [PATCH v6 8/8] scripts/make_fit: Compress dtbs in parallel Date: Wed, 19 Nov 2025 11:13:29 -0700 Message-ID: <20251119181333.991099-9-sjg@chromium.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20251119181333.991099-1-sjg@chromium.org> References: <20251119181333.991099-1-sjg@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" When there are 1500 device tree files it takes quite a while to compress them. Do it in parallel. Signed-off-by: Simon Glass --- (no changes since v1) scripts/make_fit.py | 56 ++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 50 insertions(+), 6 deletions(-) diff --git a/scripts/make_fit.py b/scripts/make_fit.py index 64038f17a51e..0a58b5b378b0 100755 --- a/scripts/make_fit.py +++ b/scripts/make_fit.py @@ -43,6 +43,7 @@ as U-Boot, Linuxboot, Tianocore, etc. =20 import argparse import collections +import multiprocessing import os import shutil import subprocess @@ -237,15 +238,31 @@ def compress_data(inf, compress): return comp_data =20 =20 -def output_dtb(fsw, seq, fname, arch, compress): +def compress_dtb(fname, compress): + """Compress a single DTB file + + Args: + fname (str): Filename containing the DTB + compress (str): Compression algorithm, e.g. 'gzip' + + Returns: + tuple: (str: fname, bytes: compressed_data) + """ + with open(fname, 'rb') as inf: + compressed =3D compress_data(inf, compress) + return fname, compressed + + +def output_dtb(fsw, seq, fname, arch, compress, data=3DNone): """Write out a single devicetree to the FIT =20 Args: fsw (libfdt.FdtSw): Object to use for writing seq (int): Sequence number (1 for first) fname (str): Filename containing the DTB - arch: FIT architecture, e.g. 'arm64' + arch (str): FIT architecture, e.g. 'arm64' compress (str): Compressed algorithm, e.g. 'gzip' + data (bytes): Pre-compressed data (optional) """ with fsw.add_node(f'fdt-{seq}'): fsw.property_string('description', os.path.basename(fname)) @@ -253,9 +270,10 @@ def output_dtb(fsw, seq, fname, arch, compress): fsw.property_string('arch', arch) fsw.property_string('compression', compress) =20 - with open(fname, 'rb') as inf: - compressed =3D compress_data(inf, compress) - fsw.property('data', compressed) + if data is None: + with open(fname, 'rb') as inf: + data =3D compress_data(inf, compress) + fsw.property('data', data) =20 =20 def find_gen_initramfs(): @@ -395,6 +413,11 @@ def _process_dtbs(args, fsw, entries, fdts): """ seq =3D 0 size =3D 0 + + # First figure out the unique DTB files that need compression + todo =3D [] + file_info =3D [] # List of (fname, model, compat, files) tuples + for fname in args.dtbs: # Ignore non-DTB (*.dtb) files if os.path.splitext(fname)[1] !=3D '.dtb': @@ -406,11 +429,32 @@ def _process_dtbs(args, fsw, entries, fdts): sys.stderr.write(f'Error processing {fname}:\n') raise e =20 + file_info.append((fname, model, compat, files)) + for fn in files: + if fn not in fdts and fn not in todo: + todo.append(fn) + + # Compress all DTBs in parallel + cache =3D {} + if todo and args.compress !=3D 'none': + if args.verbose: + print(f'Compressing {len(todo)} DTBs...') + + with multiprocessing.Pool() as pool: + compress_args =3D [(fn, args.compress) for fn in todo] + # unpacks each tuple, calls compress_dtb(fn, compress) in para= llel + results =3D pool.starmap(compress_dtb, compress_args) + + cache =3D dict(results) + + # Now write all DTBs to the FIT using pre-compressed data + for fname, model, compat, files in file_info: for fn in files: if fn not in fdts: seq +=3D 1 size +=3D os.path.getsize(fn) - output_dtb(fsw, seq, fn, args.arch, args.compress) + output_dtb(fsw, seq, fn, args.arch, args.compress, + cache.get(fn)) fdts[fn] =3D seq =20 files_seq =3D [fdts[fn] for fn in files] --=20 2.43.0