From nobody Fri Oct 23 03:30:57 2026 Received: from mail-dl1-f44.google.com (mail-dl1-f44.google.com [74.125.82.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1958D56E051 for ; Thu, 15 Oct 2026 18:23:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.82.44 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1792088589; cv=none; b=Y7NmsExr4s6ivKxBO/NHrnmHXeR/mMh3v/czJbKTbp3W1Uu5WPhFuzj4E/Bejrh8Gml/tDnzoL0ex6VgCvDi+LVs/kDgnsOS+7Szcy4o/OOsN89wI8hiEGs366X99rqj3LD2vuXzOhz6h6m+4ROMmH+9docgX1WHkwvtpaYCuFk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1792088589; c=relaxed/simple; bh=141w4atlYroacLxgjGyINyQC13zsX0zf7Nm5/ty9B/Q=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=nkgWnytyAoaTGtVIDe6DeTmRtw97nPNEBKZED8nCpTOdgumm/Kk0D3AB0lPE8Aiz5C/4jA4kXNoOaO1BCE+PBrcWJmLZ1f0OIN+z545nNw16oEXWl7rjqUzCF0++8vosLg2603gGtHvhXH62rWZnooMbQ86l4oBMII1fe8hP+ho= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=BNanEG8K; arc=none smtp.client-ip=74.125.82.44 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="BNanEG8K" Received: by mail-dl1-f44.google.com with SMTP id a92af1059eb24-17bc533fda3so2600236c88.1 for ; Thu, 15 Oct 2026 11:23:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1792088584; x=1792693384; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=DG4aA9TEdH04EQSq+KQevC5W4t18y00GnPull2P8sMI=; b=BNanEG8KIjUIiGEUJJpgAK2k2a5L58o1DTXmAgYCdV7+MuNTpiMbYyGX1B6H1pIVMs W4QP4t2guX5bFp2PKC+eszz+0nPXE/DIIqdZkWmbxfwlUOdp22id3TMeucsz8+ASHDpc vUen6CPlJ96MeL0j5uOCiWFt3ztwLgQOXCnOg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1792088584; x=1792693384; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=DG4aA9TEdH04EQSq+KQevC5W4t18y00GnPull2P8sMI=; b=tYAx6KxR/4Yamov5JBmr6kZ61RBJ+m/BFf1ScR4kQDWT9JE//425IA2YAy4I+M4Z8a 3BvGBJXb8DO99Hu+daBmgWLQTmueKZpGJ4KviA/In/3tgcO6sNpdWeK62LXTVBEREhnV DEmo0m/gnxgwO1hjr0HfXSh5A2NS59GiziA22ybjOLSG7C2E0ApRQMVq+v4O530V1BRF HSIK4OVC9PKHJtnC7u+S6+Fcct8WnfBFM4MN/bXAumiCanDG5G8MUl2J03HygKDrDrrK LGgfvYFEW3AlTjm2p1uy2gd7tM2vmALpDl6US6y38T3rA4QaMickekU9lV812/qo3P3C XC9g== X-Forwarded-Encrypted: i=1; AKwUvBwdgRjGa2XbfiseKa7BovBU9YLIhcHYH0XCkq+yXqpcxtjALYXYHT0z+II7/4sTW/4tj6vqRRX4oAQpLYw=@vger.kernel.org X-Gm-Message-State: AFq9FYIfyLp0c3m0Ip2C792AmK1i3ckPGb9TqVqe6KIZ+RgwatK3Ox3h YwRZub8TryzkejiOCw/yOHLZ76P3Oaw3UunbRj+WZSy1DkogEyRpn67wRD6NRD+xbA== X-Gm-Gg: AYBFou2ggMTiTZQG2imA+02lgnB2tBz43rgfFEqXQM4cp4nmXy3w5CpTc7DwMF7wYmW JlSSOtpXYNQc3SKJzywv61l/f05UQUsXJOkfojVeHDbrACHGKq8wBCUDjdS1b9OslEhoyjSGK8C HoQgDlNegaw1fV5RwpKcBy5ls3c0v4RRVD6DyIhPf1/W3gauw6fdDC/n/N8v384ZoZqa/4ujUxT Qedmw5LVIy4CI1X95TI2emh4o4U2pV+e0iCY0CEDsE+KQ7jhCOPf+HTcmNVbvg1ljn6kZcT8S2j Jsi3Jzm31iszv2CI9R+Q98u3GAWsdsv8QikXhY/woI8pQxP5QwJCw/4vwtAJeoEsKoTjxBfO9mm 64n55OarMDnssSMxUfqqWyvJwRnp/jjvwmgnvJvCY1Yh254zgVfUe/wpHwRbfK4FmTL3E5n6Vxi DRU/zZS6I2iwc5DXiXmyKcHuPZFicWq039j0CThDEul5aEUBMCR3aO034Prud0ipgpEHZeeDHsP wHYshLnBAmbqiGAuxEHCsDE0bTUfyHIfavqUYcU4UBcW5yJO7f15WG/fQXW X-Received: by 2002:a05:7022:b0c8:b0:147:a685:912d with SMTP id a92af1059eb24-16a60811e7amr22989727c88.29.1792088584004; Thu, 15 Oct 2026 11:23:04 -0700 (PDT) Received: from dianders.sjc.corp.google.com ([2a00:79e0:2e7c:8:9ea0:3a29:323e:5f11]) by smtp.gmail.com with ESMTPSA id a92af1059eb24-182a8a4a56bsm727204c88.20.2026.10.15.11.23.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 15 Oct 2026 11:23:03 -0700 (PDT) From: Douglas Anderson To: Devicetree-compiler@vger.kernel.org, Rob Herring , Krzysztof Kozlowski , Conor Dooley , Konrad Dybcio Cc: Chen-Yu Tsai , linux-kernel@vger.kernel.org, Devicetree@vger.kernel.org, Brian Norris , Tudor Ambarus , =?UTF-8?q?Andr=C3=A9=20Draszik?= , Julius Werner , Saravana Kannan , Peter Griffin , Douglas Anderson Subject: [PATCH 1/4] tests: Fix Python test environment setup in meson.build Date: Thu, 15 Oct 2026 11:21:34 -0700 Message-ID: <20261015182221.2931742-2-dianders@chromium.org> X-Mailer: git-send-email 2.56.0.176.g3575e65682-goog In-Reply-To: <20261015182221.2931742-1-dianders@chromium.org> References: <20261015182221.2931742-1-dianders@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Commit ecb21febfdd3 ("meson: port python bindings to build natively via meson and meson-python") moved the generated `libfdt.py` and `_libfdt` extension module into `/pylibfdt`, leaving `/pylibfdt` with only the source files. `tests/meson.build` was still setting `PYTHONPATH` to `meson.project_source_root() / 'pylibfdt'`, which only went unnoticed because `tests/pylibfdt_tests.py` manually inserts `../pylibfdt` into `sys.path`. It also checked `if not py.found()` instead of `if not pylibfdt_enabled` when setting `NO_PYTHON`. Update `tests/meson.build` to check `pylibfdt_enabled` and set `PYTHONPATH` to `meson.project_build_root() / 'pylibfdt'` so Python tools invoked by `run_tests.sh` can import `libfdt` directly. Assisted-by: Gemini:gemini-pro-prerelease Signed-off-by: Douglas Anderson --- tests/meson.build | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/meson.build b/tests/meson.build index c6ac2d7f8ec4..c25500606059 100644 --- a/tests/meson.build +++ b/tests/meson.build @@ -114,12 +114,12 @@ endforeach run_tests =3D find_program('run_tests.sh') =20 env =3D [] -if not py.found() +if not pylibfdt_enabled env +=3D 'NO_PYTHON=3D1' else env +=3D [ 'PYTHON=3D' + py.full_path(), - 'PYTHONPATH=3D' + meson.project_source_root() / 'pylibfdt', + 'PYTHONPATH=3D' + meson.project_build_root() / 'pylibfdt', 'NO_PYTHON=3D0', ] endif --=20 2.56.0.176.g3575e65682-goog From nobody Fri Oct 23 03:30:57 2026 Received: from mail-dl1-f53.google.com (mail-dl1-f53.google.com [74.125.82.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0A61C559C87 for ; Thu, 15 Oct 2026 18:23:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.82.53 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1792088599; cv=none; b=lFiBw3C074QkPymmUJzXiG2h5oxTODuvDZW8VUsVzuDHwuGCrlfwaLuwKWPQVQUdUcDO0P9jUe0S61tIaozDSZ1xBjwdkpQpKu58M/eYG54rjmnut2EzMZGTb2/lc4NEjWi/j8PMk4Dn5d2nqJna5sQvgKL/YNRP23/2QbNnCHw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1792088599; c=relaxed/simple; bh=t2fqYTeQ3tLkWFZBMtKQ0809KY8q9nI0g489z2XRh0o=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ap+1jIR+aLPNx9TCSLqBkrsPzCJh/27b6PBqvqAWc/my7fHTw0rQG9xL9p51wy/V2O5WYOQvLI4T+3tcldytg7A5I4khcZ8FrZeg3ToPsUby5PHqHy/LYk2gt4Hy+mLztqDCbecuHuyZ7SvqmLt1Ijt8pOGA8feWyU/cJ00/2eg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=Neaq+YC8; arc=none smtp.client-ip=74.125.82.53 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="Neaq+YC8" Received: by mail-dl1-f53.google.com with SMTP id a92af1059eb24-16079d54c17so786659c88.0 for ; Thu, 15 Oct 2026 11:23:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1792088589; x=1792693389; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=Zh1jMr4tvV8ouhv3YnVFweWqZZaDob4xX1pfVsBf53M=; b=Neaq+YC8qYKgCLDTw0ucnKt7uMa7x40cX9XwNJ/rss2u6+aUyAvHlDy1yj7QOQmFVC NJDW3FPRssLy795wxGyKceKzzkHk9U819ePXY5QUAs3gtIZYU42lnv7XnLLQoDT+GJhm 33g4NnPfP7s3HPP8KkaVDpOGGcCr/9FO4ZcI8= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1792088589; x=1792693389; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=Zh1jMr4tvV8ouhv3YnVFweWqZZaDob4xX1pfVsBf53M=; b=0Te3WFzC0t+Ktl39sNLRzHzvBeCNYON0wVdUmeE/MY948NHSHQ/E0FSlCD4Ht4VFKJ oYLkn7dbJA3M+wHWZSfNt2EXPqqI50nf5sE89CJliz8qYECZa3cbs93kBYQjVWxDD43S f08NIXhZBSOfVTEUbL+wHrO03hYsy2bg/eX1Qn9GKZS4rpvhBDLYTKX+bDCvtEz1dx4w aPDC11C4Bo3g+IPxWkUNZHuZHHfiNN9c0jtFZESUnDLB18AZe2zbejNQV+T5EP/M13gl DNhVrsf4+/+vcORsvH1aaCx+FInDRmzyGrqZq8AtJHHruoswQy69En303D0lSjcUYt9n Iu5g== X-Forwarded-Encrypted: i=1; AKwUvBwcZBwDwWqOzXV/MoG2oeHGvZgJTqzdXrXpkad+m8KsZm+3D47idDrZNXA4mf9d+XlHbYCN/NiRHpbeUgs=@vger.kernel.org X-Gm-Message-State: AFq9FYIdSGx1QXCa4nuR4R1uK4qmoe4SpkkxsO/P5zpxPJnFFuQwhKR+ wXkXIYif2YvCNKb7OwNlZR6exdhuFdfoOugT0zFOKWnmB/3+JyIO3xf1PbhvNXM25g== X-Gm-Gg: AYBFou0nfROoY191TTcUJzj3emrnyPOvm/Ik4LHz/envxi42kPRrKB7ZUFpzpc/vhzY xYpj8cYJ2ZcLVtNgWKg0QFBbHdW+tX/9ncUVjJYvhasobgqDDtY/O1og66vW+UBr6a3eDATwxvQ FZnVpDik1qeM5Mj7/nBjIF6rPU7TemlxJfi03ZE3dYYl0isvwZ595jGMidEo3hLkqn0ReEI+u2m SoqPSzSLK7MDWSHkc/CbqIviv9YDvb47n1Fr7uDW6CLYpGvpq8vP1awAEWkmN1ELFOqoaidH9Dj KZo/x3xF/6K9nWJAOyZgPVe5FlePG/QMSqwly8seVBf6VpMGel1OyE4D1pKZ9mNL/7hx61IDHJW kFnxW1XbbDfQx13uw4j79yMfE6RUmw4WS8U6pAdXYKIhZVwdZiJ+l/VcU23esWbdeXrfa/t9PTx iL2kvHzrcfJz7Hw25ednMDmT4I0QviV3SlwdYRo91oV1f/MJ3FLFUIUk5kGvky04g7W8s4rHzNo SyqYKEJGzjXTlrqq+P+10iWWKMG2xyLMFknOPVDntsJHV+/sqe5JO3Mvfg= X-Received: by 2002:a05:701b:2903:b0:156:1d8:7026 with SMTP id a92af1059eb24-16a60de3224mr24282386c88.46.1792088588293; Thu, 15 Oct 2026 11:23:08 -0700 (PDT) Received: from dianders.sjc.corp.google.com ([2a00:79e0:2e7c:8:9ea0:3a29:323e:5f11]) by smtp.gmail.com with ESMTPSA id a92af1059eb24-182a8a4a56bsm727204c88.20.2026.10.15.11.23.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 15 Oct 2026 11:23:05 -0700 (PDT) From: Douglas Anderson To: Devicetree-compiler@vger.kernel.org, Rob Herring , Krzysztof Kozlowski , Conor Dooley , Konrad Dybcio Cc: Chen-Yu Tsai , linux-kernel@vger.kernel.org, Devicetree@vger.kernel.org, Brian Norris , Tudor Ambarus , =?UTF-8?q?Andr=C3=A9=20Draszik?= , Julius Werner , Saravana Kannan , Peter Griffin , Douglas Anderson Subject: [PATCH 2/4] fdtfactor: Add tool to split DTBs into a base DTB and overlays Date: Thu, 15 Oct 2026 11:21:35 -0700 Message-ID: <20261015182221.2931742-3-dianders@chromium.org> X-Mailer: git-send-email 2.56.0.176.g3575e65682-goog In-Reply-To: <20261015182221.2931742-1-dianders@chromium.org> References: <20261015182221.2931742-1-dianders@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" When shipping a single firmware or kernel image that supports many closely related board variants or SKUs, bundling a full compiled .dtb for every variant results in substantial duplication. Add `fdtfactor`, a Python tool built on `pylibfdt` that takes a set of compiled device tree blobs (.dtb) compiled with `-@ -L` (`/__symbols__` and `/__local_fixups__`) and splits them into a single common `base.dtb` and per-board `.dtbo` overlays: - Common nodes present across all input DTBs are placed in `base.dtb`, with each common property assigned the value shared by at least two boards that maximizes byte savings across overlays. - Using `/__local_fixups__`, phandle references inside property values are normalized to target node paths so common properties referencing common nodes match even when numeric phandles differ across input DTBs. - Any common node that carries a phandle in any input DTB receives its phandle in `base.dtb` so overlays only emit `phandle` properties on newly added subnodes, matching the requirements of `libfdt`, Linux's `drivers/of/overlay.c`, and coreboot's `dt_apply_overlay()`. On a set of 47 Qualcomm SC7180 Trogdor Chromebook board DTBs, `fdtfactor -S` reduces total uncompressed size from 7.69 MB to 634 KB (91.8% savings). Add 3-board test cases covering default mode and `--strip-local-fixups` to the `fdtoverlay` test suite. Assisted-by: Gemini:gemini-pro-prerelease Signed-off-by: Douglas Anderson --- Makefile | 2 +- README.md | 58 ++ fdtfactor | 1218 ++++++++++++++++++++++++++++++++++++ meson.build | 1 + tests/Makefile.tests | 2 +- tests/fdtfactor_board1.dts | 69 ++ tests/fdtfactor_board2.dts | 73 +++ tests/fdtfactor_board3.dts | 96 +++ tests/run_tests.sh | 61 +- tests/testutils.sh | 1 + 10 files changed, 1578 insertions(+), 3 deletions(-) create mode 100755 fdtfactor create mode 100644 tests/fdtfactor_board1.dts create mode 100644 tests/fdtfactor_board2.dts create mode 100644 tests/fdtfactor_board3.dts diff --git a/Makefile b/Makefile index 9110d17796b0..4df5bbda9ce2 100644 --- a/Makefile +++ b/Makefile @@ -165,7 +165,7 @@ BIN +=3D fdtget BIN +=3D fdtput BIN +=3D fdtoverlay =20 -SCRIPTS =3D dtdiff +SCRIPTS =3D dtdiff fdtfactor =20 all: $(BIN) libfdt =20 diff --git a/README.md b/README.md index 8f1a5d2a4d67..e2ff2374b01d 100644 --- a/README.md +++ b/README.md @@ -91,6 +91,64 @@ $ make NO_PYTHON=3D1 More work remains to support all of libfdt, including access to numeric values. =20 +## fdtfactor + +`fdtfactor` is a Python tool built on `pylibfdt` that takes a set of +compiled device tree blobs (`.dtb`) and splits them into a single common +`base.dtb` and per-board `.dtbo` overlays so that applying a board's +`.dtbo` to `base.dtb` recreates the original board device tree while +reducing total storage size. + +Input `.dtb` files must be compiled with `dtc -@ -L` so that both +`/__symbols__` (used to resolve overlay `/__fixups__` against `base.dtb`) +and `/__local_fixups__` (used to identify phandle references inside +property values) are present. + +Basic usage: + +``` +$ fdtfactor -b base.dtb -o out_dir board1.dtb board2.dtb board3.dtb +``` + +Options: + +* `-b`, `--base-dtb`: Output path for the common base `.dtb` (placed in + `--outdir` if specified without a directory component). +* `-o`, `--outdir`: Output directory for generated `.dtbo` files (and for + `--base-dtb` when given without a directory component). +* `-S`, `--strip-local-fixups`: Omit `/__local_fixups__` from the generated + `base.dtb` and final reconstructed device trees. When `-L` is passed to + `dtc` solely so `fdtfactor` can locate phandle cells at build time, the + original DTB's `/__local_fixups__` subtree is not needed at runtime by + either the overlay loader (`fdt_overlay_apply()`) or the OS kernel; + passing `-S` strips it to reduce the size of both `base.dtb` and the + `.dtbo` overlays. Omit `-S` if downstream tooling or verification + expects applying a board's `.dtbo` to `base.dtb` to reproduce the + original `.dtb`'s `/__local_fixups__` subtree verbatim. +* `--verify`: Verify that applying each generated `.dtbo` to `base.dtb` + with `libfdt` recreates the original `.dtb`. +* `-v`, `--verbose`: Print size savings summary. + +### TODO + +* **Explore omitting `/__fixups__` and `/__symbols__` from overlays by + hardcoding base DT phandles**: Normally a standalone `.dtbo` stores + references to base tree nodes as `0xffffffff` with string entries in + `/__fixups__` that the overlay loader resolves at runtime against + `base.dtb`'s `/__symbols__`. Because `fdtfactor` generates `base.dtb` + and all `.dtbo` overlays together, it already knows the exact numeric + phandle values assigned to every node in `base.dtb`. Neither `libfdt` nor + coreboot modifies property cells that are not listed in `/__fixups__` or + `/__local_fixups__`. Writing the base phandle integers directly into + overlay properties would allow omitting `/__fixups__` from the `.dtbo` + files (and omitting `/__symbols__` when runtime symbol lookup is not + needed), further reducing overlay size and speeding up boot-time overlay + application. +* **Multi-level overlays**: Support hierarchical overlay layers (for + example, SoC base -> board family intermediate overlay -> SKU leaf + overlay) when multiple boards share large sub-trees not present across + all boards. + ## Mailing lists =20 * The [devicetree-compiler](mailto:devicetree-compiler@vger.kernel.org) diff --git a/fdtfactor b/fdtfactor new file mode 100755 index 000000000000..377f499bea50 --- /dev/null +++ b/fdtfactor @@ -0,0 +1,1218 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0-or-later +"""Split compiled device tree blobs (.dtb) into a base .dtb and .dtbo over= lays. + +Given a set of compiled device tree blobs, fdtfactor extracts all common n= odes +and properties into a single base .dtb file and writes the per-board +differences into .dtbo overlay files. Applying an overlay to the base .dtb +using fdtoverlay or libfdt's fdt_overlay_apply() recreates the correspondi= ng +original device tree. +""" + +import argparse +from collections import Counter +import os +import struct +import sys + +import libfdt + + +def split_dtb_filename(path): + """Split a DTB path into its directory and filename stem. + + Example: + '/dir/board.dtb' -> ('/dir', 'board') + + Args: + path: File path to a .dtb or .dtbo file. + + Returns: + Tuple of (directory, stem) where `stem` has any trailing `.dtb` or + `.dtbo` suffix removed. + """ + directory, filename =3D os.path.split(path) + stem, sep, ext =3D filename.rpartition('.') + if not stem or not sep or ext not in ('dtb', 'dtbo'): + stem =3D filename + return directory, stem + + +class NodeTree: + """In-memory tree of nodes and properties for sequential FDT creation.= """ + + def __init__(self): + self.props =3D {} + self.children =3D {} + + def get_node(self, path): + """Return the descendant NodeTree at `path`, creating missing node= s. + + Looks up a child or deeper descendant `NodeTree` by following the + slash-separated node names in `path` starting from `self` (leading= and + trailing `'/'` characters are ignored, so `'/soc/i2c@4000'` and + `'soc/i2c@4000'` behave identically). If any node along `path` doe= s not + exist yet in `self.children`, a new empty `NodeTree` is created fo= r it + automatically (similar to `mkdir -p`). Passing `''` or `'/'` retur= ns + `self`. + + Example: + root =3D NodeTree() + # The following automatically creates root.children['soc'] and + # root.children['soc'].children['i2c@4000'], and returns the + # 'i2c@4000' NodeTree. + i2c =3D root.get_node('/soc/i2c@4000') + i2c.props['status'] =3D b'okay\\x00' + + Args: + path: Slash-separated node path starting from `self` (leading = `'/'` + is optional and ignored, e.g. `'/soc/i2c@4000'` or + `'soc/i2c@4000'`). + + Returns: + The `NodeTree` instance at `path`. + """ + if path =3D=3D '/' or not path: + return self + curr =3D self + for part in path.strip('/').split('/'): + if part not in curr.children: + curr.children[part] =3D NodeTree() + curr =3D curr.children[part] + return curr + + def _write_to_sw(self, sw, name=3D''): + """Recursively write this NodeTree and its children into an FdtSw. + + Args: + sw: A `libfdt.FdtSw` sequential-write instance. + name: Node name for this subtree (empty string for the root no= de). + """ + sw.begin_node(name) + for prop_name, val in self.props.items(): + sw.property(prop_name, bytes(val)) + for child_name, child_node in self.children.items(): + child_node._write_to_sw(sw, child_name) + sw.end_node() + + def to_fdt_blob(self, mem_rsv=3D(), boot_cpuid_phys=3D0, size_hint=3D6= 5536): + """Serialize this root NodeTree into a packed FDT byte string. + + Args: + mem_rsv: Iterable of (address, size) tuples for the FDT memory + reservation block. + boot_cpuid_phys: Physical boot CPU ID for the FDT header. + size_hint: Initial buffer size hint in bytes for `libfdt.FdtSw= `. + + Returns: + Serialized FDT blob as `bytes`. + """ + sw =3D libfdt.FdtSw(size_hint) + for addr, size in mem_rsv: + sw.add_reservemap_entry(addr, size) + sw.finish_reservemap() + self._write_to_sw(sw, '') + # Unlike other FdtSw methods, as_fdt() does not call check_space() + # when fdt_finish() appends the 4-byte FDT_END token. If fewer than + # 4 bytes remain in the buffer, resize once and retry. + try: + fdt =3D sw.as_fdt() + except libfdt.FdtException as exc: + if not sw.check_space(exc.err): + raise + fdt =3D sw.as_fdt() + libfdt.fdt_set_boot_cpuid_phys(fdt._fdt, boot_cpuid_phys) + fdt.pack() + return bytes(fdt.as_bytearray()) + + +def _open_fdt_ro(raw_bytes, filename): + """Validate and wrap an uncompressed FDT blob in `libfdt.FdtRo`. + + Args: + raw_bytes: Uncompressed FDT binary contents as `bytes`. + filename: Display name or file path for error reporting. + + Returns: + A validated `libfdt.FdtRo` instance. + + Raises: + ValueError: If `raw_bytes` fails FDT header or full-structure chec= ks. + """ + try: + fdt =3D libfdt.FdtRo(raw_bytes) + libfdt.check_err(libfdt.fdt_check_full(fdt._fdt, len(raw_bytes))) + except libfdt.FdtException as exc: + raise ValueError(f'{filename}: Invalid FDT blob ({exc})') from exc + return fdt + + +def _is_dtbo_overlay(fdt_ro): + """Return True if `fdt_ro` is a `.dtbo` overlay rather than a base DTB. + + Args: + fdt_ro: A `libfdt.FdtRo` instance wrapping an uncompressed FDT blo= b. + + Returns: + True if the FDT has `/__fixups__` or a top-level fragment node + containing an `__overlay__` subnode. + """ + if fdt_ro.subnode_offset(0, '__fixups__', quiet=3Dlibfdt.QUIET_NOTFOUN= D) >=3D 0: + return True + soff =3D fdt_ro.first_subnode(0, quiet=3Dlibfdt.QUIET_NOTFOUND) + while soff >=3D 0: + if fdt_ro.subnode_offset( + soff, '__overlay__', quiet=3Dlibfdt.QUIET_NOTFOUND + ) >=3D 0: + return True + soff =3D fdt_ro.next_subnode(soff, quiet=3Dlibfdt.QUIET_NOTFOUND) + return False + + +class ParsedDtb: + """Parsed representation of an input .dtb blob.""" + + def __init__(self, filename, raw_bytes): + """Parse an uncompressed FDT blob into nodes, symbols, and fixups. + + Args: + filename: Display name or file path for error reporting. + raw_bytes: Uncompressed FDT binary contents as `bytes`. + """ + self.filename =3D filename + self.raw_bytes =3D raw_bytes + fdt =3D _open_fdt_ro(raw_bytes, filename) + if _is_dtbo_overlay(fdt): + raise ValueError( + f'{filename}: Input is already a DTB overlay (.dtbo)' + ) + + # Extract FDT header metadata (boot_cpuid_phys and /memreserve/ ta= ble) + # stored outside the DT node/property tree; see _validate_dtb_head= ers() + # for why these require special handling. + self.boot_cpuid_phys =3D fdt.boot_cpuid_phys() + self.mem_rsv =3D [ + tuple(fdt.get_mem_rsv(i)) for i in range(fdt.num_mem_rsv()) + ] + + self.has_symbols =3D False + self.symbols =3D {} + self.has_local_fixups =3D False + self.local_fixups =3D {} + + self.nodes =3D {} + self.phandles =3D {} + self.has_phandle_prop =3D {} + self.has_linux_phandle_prop =3D {} + self.phandle_to_path =3D {} + self.sym_props =3D {} + + self._parse_tree(fdt) + + @classmethod + def from_file(cls, path): + """Read and parse a .dtb file from disk. + + Args: + path: Path to the input .dtb file. + + Returns: + A new `ParsedDtb` instance. + """ + with open(path, 'rb') as f: + raw_bytes =3D f.read() + return cls(path, raw_bytes) + + def _parse_tree(self, fdt): + """Walk the FDT and extract nodes, properties, symbols, and fixups. + + Args: + fdt: A `libfdt.FdtRo` instance wrapping the input blob. + """ + self._walk_regular(fdt, 0, '/') + + sym_off =3D fdt.subnode_offset(0, '__symbols__', + quiet=3Dlibfdt.QUIET_NOTFOUND) + if sym_off >=3D 0: + self.has_symbols =3D True + poff =3D fdt.first_property_offset(sym_off, + quiet=3Dlibfdt.QUIET_NOTFOUND) + while poff >=3D 0: + prop =3D fdt.get_property_by_offset(poff) + if prop.name not in ('phandle', 'linux,phandle'): + target_path =3D prop.as_str() if prop else '' + if target_path not in self.nodes: + raise ValueError( + f'{self.filename}: symbol {prop.name!r} refere= nces ' + f'missing node {target_path!r}' + ) + self.symbols[prop.name] =3D target_path + poff =3D fdt.next_property_offset(poff, + quiet=3Dlibfdt.QUIET_NOTFO= UND) + + lf_off =3D fdt.subnode_offset(0, '__local_fixups__', + quiet=3Dlibfdt.QUIET_NOTFOUND) + if lf_off >=3D 0: + self.has_local_fixups =3D True + self._walk_local_fixups(fdt, lf_off, '/') + + def _walk_regular(self, fdt, nodeoffset, path): + """Recursively collect regular nodes and their properties. + + Args: + fdt: A `libfdt.FdtRo` instance. + nodeoffset: Offset of the current node in `fdt`. + path: Full device tree path of the current node (e.g. '/soc'). + """ + props =3D {} + ph_val =3D None + has_ph =3D False + has_linux_ph =3D False + + poff =3D fdt.first_property_offset(nodeoffset, + quiet=3Dlibfdt.QUIET_NOTFOUND) + while poff >=3D 0: + prop =3D fdt.get_property_by_offset(poff) + raw =3D bytes(prop) + if prop.name in ('phandle', 'linux,phandle'): + if len(raw) !=3D 4: + raise ValueError( + f'{self.filename}: {path}: {prop.name} length ' + f'{len(raw)} !=3D 4' + ) + val =3D struct.unpack('>I', raw)[0] + if ph_val is not None and ph_val !=3D val: + raise ValueError( + f'{self.filename}: {path}: mismatched phandle ' + f'({ph_val:#x} vs {val:#x})' + ) + ph_val =3D val + if prop.name =3D=3D 'phandle': + has_ph =3D True + else: + has_linux_ph =3D True + else: + props[prop.name] =3D raw + poff =3D fdt.next_property_offset(poff, quiet=3Dlibfdt.QUIET_N= OTFOUND) + + self.nodes[path] =3D props + self.phandles[path] =3D ph_val + self.has_phandle_prop[path] =3D has_ph + self.has_linux_phandle_prop[path] =3D has_linux_ph + if ph_val is not None: + if ph_val in (0, 0xffffffff): + raise ValueError( + f'{self.filename}: invalid phandle {ph_val:#x} on {pat= h}' + ) + if ph_val in self.phandle_to_path: + raise ValueError( + f'{self.filename}: duplicate phandle {ph_val:#x} on ' + f'{self.phandle_to_path[ph_val]} and {path}' + ) + self.phandle_to_path[ph_val] =3D path + + soff =3D fdt.first_subnode(nodeoffset, quiet=3Dlibfdt.QUIET_NOTFOU= ND) + while soff >=3D 0: + name =3D fdt.get_name(soff) + if path =3D=3D '/' and name in ('__symbols__', '__local_fixups= __'): + soff =3D fdt.next_subnode(soff, quiet=3Dlibfdt.QUIET_NOTFO= UND) + continue + child_path =3D '/' + name if path =3D=3D '/' else f'{path}/{na= me}' + self._walk_regular(fdt, soff, child_path) + soff =3D fdt.next_subnode(soff, quiet=3Dlibfdt.QUIET_NOTFOUND) + + def _walk_local_fixups(self, fdt, nodeoffset, rel_path): + """Recursively collect phandle byte offsets from `/__local_fixups_= _`. + + Args: + fdt: A `libfdt.FdtRo` instance. + nodeoffset: Offset of the current node inside `/__local_fixups= __`. + rel_path: Corresponding target node path in the device tree. + + Raises: + ValueError: If a fixup property length is not a multiple of 4. + """ + poff =3D fdt.first_property_offset(nodeoffset, + quiet=3Dlibfdt.QUIET_NOTFOUND) + while poff >=3D 0: + prop =3D fdt.get_property_by_offset(poff) + raw =3D bytes(prop) + if len(raw) % 4 !=3D 0: + raise ValueError( + f'{self.filename}: malformed __local_fixups__ property= ' + f'{rel_path}:{prop.name} (length {len(raw)})' + ) + count =3D len(raw) // 4 + offsets =3D struct.unpack(f'>{count}I', raw) + self.local_fixups[(rel_path, prop.name)] =3D offsets + poff =3D fdt.next_property_offset(poff, quiet=3Dlibfdt.QUIET_N= OTFOUND) + + soff =3D fdt.first_subnode(nodeoffset, quiet=3Dlibfdt.QUIET_NOTFOU= ND) + while soff >=3D 0: + name =3D fdt.get_name(soff) + child_path =3D '/' + name if rel_path =3D=3D '/' else f'{rel_p= ath}/{name}' + self._walk_local_fixups(fdt, soff, child_path) + soff =3D fdt.next_subnode(soff, quiet=3Dlibfdt.QUIET_NOTFOUND) + + def build_symbolic_props(self): + """Build normalized property values for comparison across DTBs. + + Populates `self.sym_props[path][prop_name]` with a tuple: + `(masked_bytes, ((offset, target_node_path), ...))` + where phandle cells listed in `self.local_fixups` are zeroed in + `masked_bytes` and recorded symbolically by their target node path. + + Raises: + ValueError: If a fixup references a missing property, its offs= et is + out of range, or it references an unknown phandle value. + """ + for path, prop_name in self.local_fixups: + if path not in self.nodes or prop_name not in self.nodes[path]: + raise ValueError( + f'{self.filename}: local fixup references missing prop= erty ' + f'{path}:{prop_name}' + ) + for path, props in self.nodes.items(): + node_sym_props =3D {} + for prop_name, raw in props.items(): + offsets =3D self.local_fixups.get((path, prop_name), ()) + if not offsets: + node_sym_props[prop_name] =3D (raw, ()) + continue + masked =3D bytearray(raw) + targets =3D [] + for off in offsets: + if off + 4 > len(raw): + raise ValueError( + f'{self.filename}: fixup offset {off} out of r= ange ' + f'on {path}:{prop_name} (len {len(raw)})' + ) + ph =3D struct.unpack_from('>I', raw, off)[0] + target_path =3D self.phandle_to_path.get(ph) + if target_path is None: + raise ValueError( + f'{self.filename}: unknown phandle {ph:#x} at ' + f'{path}:{prop_name}+{off}' + ) + masked[off:off + 4] =3D b'\x00\x00\x00\x00' + targets.append((off, target_path)) + node_sym_props[prop_name] =3D (bytes(masked), tuple(target= s)) + self.sym_props[path] =3D node_sym_props + + +def select_best_candidate(candidates, byte_len_fn): + """Choose the value that maximizes space savings across overlays. + + Only candidates shared by at least two input DTBs (count > 1) are elig= ible + for inclusion in the base DTB; if every candidate value is unique to a + single DTB, placing one in the base DTB would yield zero net size savi= ngs + (adding one copy to the base DTB to save one overlay copy). + + Args: + candidates: Iterable of candidate values (in input DTB order). + byte_len_fn: Function returning the byte cost of a candidate value= in + an overlay. + + Returns: + The chosen candidate value, or None if no value appears more than = once. + """ + counts =3D Counter(candidates) + best_val =3D None + best_key =3D None + for val, count in counts.items(): + if count <=3D 1: + continue + length =3D byte_len_fn(val) + # Maximize total bytes saved across overlays: + # (count - 1) * (length + 12) + key =3D ((count - 1) * (length + 12), count, length) + if best_key is None or key > best_key: + best_key =3D key + best_val =3D val + return best_val + + +def _validate_dtb_headers(dtbs): + """Verify that FDT header fields and /memreserve/ entries match across= DTBs. + + `boot_cpuid_phys` and `/memreserve/` require special handling because: + * Unlike regular DT nodes (including `/reserved-memory`), they are sto= red + in the FDT binary header and memory reservation block rather than the + DT structure block. + * `fdt_overlay_apply()` only merges the `.dtbo` structure block into t= he + base DTB and ignores the `.dtbo`'s header and `/memreserve/` block. + * Therefore, every reconstructed board DTB inherits `boot_cpuid_phys` = and + `/memreserve/` directly from the base DTB, so all DTBs sharing a base + DTB must have identical values. + + Args: + dtbs: Sequence of `ParsedDtb` objects to validate. + + Raises: + ValueError: If `boot_cpuid_phys` or `mem_rsv` differs between any = two + input DTBs. + """ + for d in dtbs[1:]: + if d.boot_cpuid_phys !=3D dtbs[0].boot_cpuid_phys: + raise ValueError( + f'boot_cpuid_phys mismatch between {dtbs[0].filename} ' + f'({dtbs[0].boot_cpuid_phys}) and {d.filename} ' + f'({d.boot_cpuid_phys})' + ) + if d.mem_rsv !=3D dtbs[0].mem_rsv: + raise ValueError( + f'/memreserve/ mismatch between {dtbs[0].filename} ' + f'({dtbs[0].mem_rsv}) and {d.filename} ({d.mem_rsv})' + ) + + +def _find_common_nodes_and_phandles(dtbs): + """Identify common node paths and assign sequential base DTB phandles. + + Any common node that carries a phandle in any input DTB is assigned a + phandle in the base DTB so overlays never attempt to add a `phandle` + property to an already-existing base DTB node (which Linux and + coreboot/depthcharge disallow). + + Args: + dtbs: Sequence of `ParsedDtb` objects. + + Returns: + Tuple of `(base_dtb_nodes, common_node_set, base_dtb_phandles)` wh= ere: + * `base_dtb_nodes`: Ordered list of node paths common to all `dtbs= `. + * `common_node_set`: Set of node paths in `base_dtb_nodes`. + * `base_dtb_phandles`: Dict mapping node path to assigned integer + phandle in the base DTB. + """ + common_node_set =3D set(dtbs[0].nodes.keys()) + for d in dtbs[1:]: + common_node_set &=3D set(d.nodes.keys()) + base_dtb_nodes =3D [p for p in dtbs[0].nodes.keys() if p in common_nod= e_set] + + base_dtb_phandles =3D {} + next_ph =3D 1 + for path in base_dtb_nodes: + if any(d.phandles[path] is not None for d in dtbs): + base_dtb_phandles[path] =3D next_ph + next_ph +=3D 1 + + return base_dtb_nodes, common_node_set, base_dtb_phandles + + +def _find_common_symbols(dtbs, common_node_set, base_dtb_phandles): + """Find symbols common to all input DTBs that point to the same base n= ode. + + Only symbols that exist in all DTBs and resolve to the same common base + DTB node path are placed in the base DTB's `/__symbols__`, ensuring + overlays only add new symbols and never need to overwrite existing ones + (which coreboot/depthcharge merges with `upd=3D0`). + + Args: + dtbs: Sequence of `ParsedDtb` objects. + common_node_set: Set of node paths present in all input DTBs. + base_dtb_phandles: Dict mapping base DTB node paths to their assig= ned + integer phandles. + + Returns: + Tuple of `(base_dtb_symbols, path_to_base_dtb_symbol)` where: + * `base_dtb_symbols`: Dict mapping symbol label to target node pat= h. + * `path_to_base_dtb_symbol`: Dict mapping target node path to its + primary symbol label in `base_dtb_symbols`. + """ + base_dtb_symbols =3D {} + path_to_base_dtb_symbol =3D {} + + common_sym_names =3D set(dtbs[0].symbols.keys()) + for d in dtbs[1:]: + common_sym_names &=3D set(d.symbols.keys()) + for label in dtbs[0].symbols.keys(): + if label not in common_sym_names: + continue + target_paths =3D {d.symbols[label] for d in dtbs} + if len(target_paths) =3D=3D 1: + chosen_path =3D dtbs[0].symbols[label] + if chosen_path in common_node_set: + base_dtb_symbols[label] =3D chosen_path + if (chosen_path in base_dtb_phandles and + chosen_path not in path_to_base_dtb_symbol): + path_to_base_dtb_symbol[chosen_path] =3D label + + return base_dtb_symbols, path_to_base_dtb_symbol + + +def _select_base_dtb_properties(dtbs, base_dtb_nodes, base_dtb_phandles, + preserve_local_fixups): + """Select the common property values to place on each node in the base= DTB. + + Args: + dtbs: Sequence of `ParsedDtb` objects with `sym_props` populated. + base_dtb_nodes: Ordered list of node paths in the base DTB. + base_dtb_phandles: Dict mapping base DTB node paths to their assig= ned + integer phandles. + preserve_local_fixups: Whether `/__local_fixups__` from the input = DTBs + must be preserved in the merged output. + + Returns: + Tuple of `(base_dtb_props, base_dtb_fixup_offsets)` where: + * `base_dtb_props`: Dict mapping `path -> {prop_name: sym_val}` for + properties placed in the base DTB. + * `base_dtb_fixup_offsets`: Dict mapping `(path, prop_name)` to a = tuple + of byte offsets for the base DTB's `/__local_fixups__`. + """ + base_dtb_props =3D {} + base_dtb_fixup_offsets =3D {} + + for path in base_dtb_nodes: + common_prop_names =3D set(dtbs[0].nodes[path].keys()) + for d in dtbs[1:]: + common_prop_names &=3D set(d.nodes[path].keys()) + + node_base_props =3D {} + for prop_name in dtbs[0].nodes[path].keys(): + if prop_name not in common_prop_names: + continue + + all_have_prop_fixups =3D all( + bool(d.sym_props[path][prop_name][1]) for d in dtbs + ) + eligible =3D [] + for d in dtbs: + sym_val =3D d.sym_props[path][prop_name] + _, targets =3D sym_val + if targets: + # Overlays cannot delete a property from /__local_fixu= ps__, + # so a phandle-bearing value can only go into base.dtb= when + # every board also has phandle fixups on this property. + if preserve_local_fixups and not all_have_prop_fixups: + continue + if not all( + t_path in base_dtb_phandles for _, t_path in targe= ts + ): + continue + eligible.append(sym_val) + + fixup_entry_cost =3D len(path) + len(prop_name) + 28 + chosen_val =3D select_best_candidate( + eligible, + lambda sv, fc=3Dfixup_entry_cost: ( + len(sv[0]) + len(sv[1]) * fc + ), + ) + if chosen_val is not None: + node_base_props[prop_name] =3D chosen_val + if preserve_local_fixups and chosen_val[1]: + offs =3D tuple(off for off, _ in chosen_val[1]) + base_dtb_fixup_offsets[(path, prop_name)] =3D offs + + base_dtb_props[path] =3D node_base_props + + return base_dtb_props, base_dtb_fixup_offsets + + +def _build_base_dtb(dtbs, base_dtb_nodes, base_dtb_props, base_dtb_phandle= s, + base_dtb_symbols, base_dtb_fixup_offsets): + """Construct and serialize the common base DTB blob. + + Args: + dtbs: Sequence of `ParsedDtb` objects. + base_dtb_nodes: Ordered list of node paths in the base DTB. + base_dtb_props: Dict mapping `path -> {prop_name: sym_val}`. + base_dtb_phandles: Dict mapping base DTB node paths to their assig= ned + integer phandles. + base_dtb_symbols: Dict mapping symbol label to target node path fo= r the + base DTB's `/__symbols__`. + base_dtb_fixup_offsets: Dict mapping `(path, prop_name)` to phandle + byte offsets for the base DTB's `/__local_fixups__`. + + Returns: + Serialized base DTB blob as `bytes`. + """ + base_dtb_root =3D NodeTree() + for path in base_dtb_nodes: + node =3D base_dtb_root.get_node(path) + for prop_name, (masked_bytes, targets) in base_dtb_props[path].ite= ms(): + if not targets: + node.props[prop_name] =3D masked_bytes + else: + val =3D bytearray(masked_bytes) + for off, t_path in targets: + struct.pack_into('>I', val, off, base_dtb_phandles[t_p= ath]) + node.props[prop_name] =3D bytes(val) + + if path in base_dtb_phandles: + ph_bytes =3D struct.pack('>I', base_dtb_phandles[path]) + if any(d.has_phandle_prop[path] for d in dtbs): + node.props['phandle'] =3D ph_bytes + if any(d.has_linux_phandle_prop[path] for d in dtbs): + node.props['linux,phandle'] =3D ph_bytes + + if base_dtb_symbols: + sym_node =3D base_dtb_root.get_node('/__symbols__') + for label, target_path in base_dtb_symbols.items(): + sym_node.props[label] =3D target_path.encode('utf-8') + b'\x00' + + if base_dtb_fixup_offsets: + lf_root =3D base_dtb_root.get_node('/__local_fixups__') + for (path, prop_name), offs in base_dtb_fixup_offsets.items(): + lf_node =3D lf_root.get_node(path) + lf_node.props[prop_name] =3D struct.pack(f'>{len(offs)}I', *of= fs) + + return base_dtb_root.to_fdt_blob( + mem_rsv=3Ddtbs[0].mem_rsv, + boot_cpuid_phys=3Ddtbs[0].boot_cpuid_phys, + size_hint=3Dmax(len(dtbs[0].raw_bytes), 4096), + ) + + +def _assign_local_phandles(dtb, base_dtb_phandles): + """Assign sequential overlay-local phandles to board-specific nodes. + + Args: + dtb: The board's `ParsedDtb`. + base_dtb_phandles: Dict mapping base DTB node paths to their phand= les. + + Returns: + Dict mapping board-specific node path to its overlay-local integer + phandle (`1, 2, ...`). + """ + local_phandles =3D {} + next_local_ph =3D 1 + for path in dtb.nodes.keys(): + if dtb.phandles[path] is not None and path not in base_dtb_phandle= s: + local_phandles[path] =3D next_local_ph + next_local_ph +=3D 1 + return local_phandles + + +def _encode_dtbo_property(path, prop_name, sym_val, base_dtb_phandles, + path_to_base_dtb_symbol, local_phandles, ext_fix= ups, + ov_local_fixups): + """Encode a property value for `/fragment@0/__overlay__` and record fi= xups. + + Args: + path: Target node path in the device tree (e.g. '/soc/i2c@4000'). + prop_name: Property name. + sym_val: Normalized `(masked_bytes, targets)` tuple for the proper= ty. + base_dtb_phandles: Dict mapping base DTB node paths to their phand= les. + path_to_base_dtb_symbol: Dict mapping base DTB node paths to symbol + names in the base DTB's `/__symbols__`. + local_phandles: Dict mapping board-only node paths to overlay-local + phandles. + ext_fixups: Dict of `label -> [fixup_str, ...]` updated in-place f= or + references to base DTB symbols. + ov_local_fixups: Dict of `(path, prop_name) -> [offset, ...]` upda= ted + in-place for references to overlay-local phandles. + + Returns: + Encoded property value as `bytes`. + """ + masked_bytes, targets =3D sym_val + if not targets: + return masked_bytes + + val =3D bytearray(masked_bytes) + for off, t_path in targets: + if t_path in base_dtb_phandles: + sym_label =3D path_to_base_dtb_symbol.get(t_path) + if sym_label is not None: + struct.pack_into('>I', val, off, 0xffffffff) + ov_path =3D ( + '/fragment@0/__overlay__' + if path =3D=3D '/' + else f'/fragment@0/__overlay__{path}' + ) + entry =3D f'{ov_path}:{prop_name}:{off}' + ext_fixups.setdefault(sym_label, []).append(entry) + else: + # Target is a common base DTB node without a symbol shared= by + # all input DTBs (e.g. an anonymous &{/path} reference or a + # board-specific label); use its assigned base DTB phandle. + struct.pack_into('>I', val, off, base_dtb_phandles[t_path]) + else: + struct.pack_into('>I', val, off, local_phandles[t_path]) + ov_local_fixups.setdefault((path, prop_name), []).append(off) + return bytes(val) + + +def _populate_dtbo_overlay_nodes(ov_root, dtb, common_node_set, base_dtb_p= rops, + base_dtb_phandles, path_to_base_dtb_symbo= l, + local_phandles): + """Populate `/fragment@0/__overlay__` with board-specific nodes and pr= ops. + + Args: + ov_root: The `NodeTree` for `/fragment@0/__overlay__`. + dtb: The board's `ParsedDtb`. + common_node_set: Set of node paths present in the base DTB. + base_dtb_props: Dict mapping `path -> {prop_name: sym_val}` in the + base DTB. + base_dtb_phandles: Dict mapping base DTB node paths to their phand= les. + path_to_base_dtb_symbol: Dict mapping base DTB node paths to symbol + names in the base DTB's `/__symbols__`. + local_phandles: Dict mapping board-only node paths to overlay-local + phandles. + + Returns: + Tuple of `(ext_fixups, ov_local_fixups)` collected across all over= lay + properties. + """ + ext_fixups =3D {} + ov_local_fixups =3D {} + + for path, props in dtb.nodes.items(): + in_base_dtb =3D path in common_node_set + base_node_props =3D base_dtb_props.get(path, {}) + + diff_props =3D [ + (prop_name, dtb.sym_props[path][prop_name]) + for prop_name in props.keys() + if (not in_base_dtb or + base_node_props.get(prop_name) !=3D + dtb.sym_props[path][prop_name]) + ] + + if not in_base_dtb or diff_props: + ov_node =3D ov_root.get_node(path) + for prop_name, sym_val in diff_props: + ov_node.props[prop_name] =3D _encode_dtbo_property( + path, prop_name, sym_val, base_dtb_phandles, + path_to_base_dtb_symbol, local_phandles, ext_fixups, + ov_local_fixups, + ) + + if path in local_phandles: + ph_bytes =3D struct.pack('>I', local_phandles[path]) + if dtb.has_phandle_prop[path]: + ov_node.props['phandle'] =3D ph_bytes + if dtb.has_linux_phandle_prop[path]: + ov_node.props['linux,phandle'] =3D ph_bytes + + return ext_fixups, ov_local_fixups + + +def _populate_dtbo_metadata(dtbo_root, ov_root, dtb, base_dtb_symbols, + base_dtb_fixup_offsets, ext_fixups, + ov_local_fixups, preserve_local_fixups): + """Populate `/__symbols__`, `/__fixups__`, and `/__local_fixups__` on = DTBO. + + Args: + dtbo_root: Root `NodeTree` of the `.dtbo` overlay. + ov_root: The `NodeTree` for `/fragment@0/__overlay__`. + dtb: The board's `ParsedDtb`. + base_dtb_symbols: Dict of symbols already present in the base DTB. + base_dtb_fixup_offsets: Dict of `/__local_fixups__` offsets already + present in the base DTB. + ext_fixups: Dict of `label -> [fixup_str, ...]` referencing base D= TB + symbols. + ov_local_fixups: Dict of `(path, prop_name) -> [offset, ...]` + referencing overlay-local phandles. + preserve_local_fixups: Whether to emit differing input + `/__local_fixups__` entries into + `/fragment@0/__overlay__/__local_fixups__`. + """ + # If preserving input /__local_fixups__ in the merged tree, place any + # differing fixup offset lists under + # /fragment@0/__overlay__/__local_fixups__ + if preserve_local_fixups: + for (path, prop_name), offs in dtb.local_fixups.items(): + if base_dtb_fixup_offsets.get((path, prop_name)) !=3D offs: + merged_lf_root =3D ov_root.get_node('/__local_fixups__') + merged_lf_node =3D merged_lf_root.get_node(path) + merged_lf_node.props[prop_name] =3D struct.pack( + f'>{len(offs)}I', *offs + ) + + # Emit overlay root /__symbols__ for new or modified symbols + diff_syms =3D [ + (label, t_path) + for label, t_path in dtb.symbols.items() + if base_dtb_symbols.get(label) !=3D t_path + ] + if diff_syms: + sym_node =3D dtbo_root.get_node('/__symbols__') + for label, t_path in diff_syms: + # Ensure target node exists in __overlay__ so dtc -I dtb -O dts + # on the standalone .dtbo does not warn about a missing node. + ov_root.get_node(t_path) + ov_target =3D ( + '/fragment@0/__overlay__' + if t_path =3D=3D '/' + else f'/fragment@0/__overlay__{t_path}' + ) + sym_node.props[label] =3D ov_target.encode('utf-8') + b'\x00' + + # Emit overlay root /__fixups__ for references to base DTB symbols + if ext_fixups: + fixups_node =3D dtbo_root.get_node('/__fixups__') + for label, entries in ext_fixups.items(): + fixups_node.props[label] =3D ( + '\x00'.join(entries).encode('utf-8') + b'\x00' + ) + + # Emit overlay root /__local_fixups__ for overlay-local phandles + if ov_local_fixups: + ov_lf_root =3D dtbo_root.get_node( + '/__local_fixups__/fragment@0/__overlay__' + ) + for (path, prop_name), offs in ov_local_fixups.items(): + lf_node =3D ov_lf_root.get_node(path) + lf_node.props[prop_name] =3D struct.pack(f'>{len(offs)}I', *of= fs) + + +def _build_board_dtbo(dtb, common_node_set, base_dtb_props, base_dtb_phand= les, + base_dtb_symbols, path_to_base_dtb_symbol, + base_dtb_fixup_offsets, preserve_local_fixups): + """Construct and serialize the `.dtbo` overlay blob for a single board. + + Args: + dtb: The board's `ParsedDtb`. + common_node_set: Set of node paths present in the base DTB. + base_dtb_props: Dict mapping `path -> {prop_name: sym_val}` in the + base DTB. + base_dtb_phandles: Dict mapping base DTB node paths to their phand= les. + base_dtb_symbols: Dict of symbols present in the base DTB. + path_to_base_dtb_symbol: Dict mapping base DTB node paths to symbol + names in the base DTB's `/__symbols__`. + base_dtb_fixup_offsets: Dict of `/__local_fixups__` offsets presen= t in + the base DTB. + preserve_local_fixups: Whether to preserve `/__local_fixups__` in = the + merged output. + + Returns: + Serialized `.dtbo` overlay blob as `bytes`. + """ + dtbo_root =3D NodeTree() + frag_node =3D dtbo_root.get_node('/fragment@0') + frag_node.props['target-path'] =3D b'/\x00' + ov_root =3D frag_node.get_node('__overlay__') + + local_phandles =3D _assign_local_phandles(dtb, base_dtb_phandles) + ext_fixups, ov_local_fixups =3D _populate_dtbo_overlay_nodes( + ov_root, dtb, common_node_set, base_dtb_props, base_dtb_phandles, + path_to_base_dtb_symbol, local_phandles, + ) + _populate_dtbo_metadata( + dtbo_root, ov_root, dtb, base_dtb_symbols, base_dtb_fixup_offsets, + ext_fixups, ov_local_fixups, preserve_local_fixups, + ) + + return dtbo_root.to_fdt_blob(size_hint=3D16384) + + +def create_base_dtb_and_overlays(dtbs, strip_local_fixups=3DFalse): + """Compute the base DTB and per-board DTBO blobs from parsed DTBs. + + Args: + dtbs: List of `ParsedDtb` objects (all compiled with `-@ -L`). + strip_local_fixups: If True, omit `/__local_fixups__` from the bas= e DTB + and from the merged result. + + Returns: + Tuple of `(base_dtb_blob, dtbo_blobs)` where `base_dtb_blob` is `b= ytes` + and `dtbo_blobs` is a list of `bytes` corresponding to `dtbs`. + """ + missing_meta =3D [ + d.filename for d in dtbs if not (d.has_symbols and d.has_local_fix= ups) + ] + if missing_meta: + raise ValueError( + f'input DTB(s) lack /__symbols__ or /__local_fixups__: ' + f'{", ".join(missing_meta)} (compile input DTBs with dtc -@ -L= )' + ) + + _validate_dtb_headers(dtbs) + + for d in dtbs: + d.build_symbolic_props() + + base_dtb_nodes, common_node_set, base_dtb_phandles =3D ( + _find_common_nodes_and_phandles(dtbs) + ) + base_dtb_symbols, path_to_base_dtb_symbol =3D _find_common_symbols( + dtbs, common_node_set, base_dtb_phandles + ) + preserve_local_fixups =3D not strip_local_fixups + base_dtb_props, base_dtb_fixup_offsets =3D _select_base_dtb_properties( + dtbs, base_dtb_nodes, base_dtb_phandles, preserve_local_fixups + ) + + base_dtb_blob =3D _build_base_dtb( + dtbs, base_dtb_nodes, base_dtb_props, base_dtb_phandles, + base_dtb_symbols, base_dtb_fixup_offsets, + ) + dtbo_blobs =3D [ + _build_board_dtbo( + d, common_node_set, base_dtb_props, base_dtb_phandles, + base_dtb_symbols, path_to_base_dtb_symbol, base_dtb_fixup_offs= ets, + preserve_local_fixups, + ) + for d in dtbs + ] + return base_dtb_blob, dtbo_blobs + + +def verify_recreated_dtb(orig_dtb, base_dtb_blob, dtbo_blob, + strip_local_fixups): + """Apply `dtbo_blob` to `base_dtb_blob` and verify it matches `orig_dt= b`. + + Args: + orig_dtb: The original `ParsedDtb` instance for the board. + base_dtb_blob: Uncompressed base DTB binary as `bytes`. + dtbo_blob: Uncompressed board `.dtbo` overlay binary as `bytes`. + strip_local_fixups: Whether `/__local_fixups__` was stripped. + + Raises: + RuntimeError: If `fdt_overlay_apply()` fails or any node, property, + phandle, symbol, or header field in the merged DTB differs from + `orig_dtb`. + """ + merged_fdt =3D libfdt.Fdt(base_dtb_blob) + merged_fdt.resize(len(base_dtb_blob) + len(dtbo_blob) * 2 + 65536) + fdto =3D bytearray(dtbo_blob) + err =3D libfdt.fdt_overlay_apply(merged_fdt._fdt, fdto) + if err !=3D 0: + raise RuntimeError( + f'{orig_dtb.filename}: fdt_overlay_apply failed with error {er= r} ' + f'({libfdt.strerror(err)})' + ) + merged_fdt.pack() + recreated =3D ParsedDtb( + f'{orig_dtb.filename} (recreated)', bytes(merged_fdt.as_bytearray(= )) + ) + if strip_local_fixups: + if recreated.has_local_fixups: + raise RuntimeError( + f'{orig_dtb.filename}: /__local_fixups__ present despite ' + f'--strip-local-fixups' + ) + recreated.local_fixups =3D orig_dtb.local_fixups + else: + if orig_dtb.local_fixups !=3D recreated.local_fixups: + raise RuntimeError( + f'{orig_dtb.filename}: /__local_fixups__ mismatch' + ) + + recreated.build_symbolic_props() + + if set(orig_dtb.nodes.keys()) !=3D set(recreated.nodes.keys()): + missing =3D set(orig_dtb.nodes.keys()) - set(recreated.nodes.keys(= )) + extra =3D set(recreated.nodes.keys()) - set(orig_dtb.nodes.keys()) + raise RuntimeError( + f'{orig_dtb.filename}: node mismatch (missing=3D{missing}, ' + f'extra=3D{extra})' + ) + + for path in orig_dtb.nodes.keys(): + if orig_dtb.sym_props[path] !=3D recreated.sym_props[path]: + for p in (set(orig_dtb.sym_props[path]) | + set(recreated.sym_props[path])): + if (orig_dtb.sym_props[path].get(p) !=3D + recreated.sym_props[path].get(p)): + raise RuntimeError( + f'{orig_dtb.filename}: property mismatch at {path}= :{p}' + ) + # Common base nodes may gain a phandle if another board referenced= them. + if (orig_dtb.has_phandle_prop[path] and + not recreated.has_phandle_prop[path]): + raise RuntimeError( + f'{orig_dtb.filename}: missing phandle at {path}' + ) + if (orig_dtb.has_linux_phandle_prop[path] and + not recreated.has_linux_phandle_prop[path]): + raise RuntimeError( + f'{orig_dtb.filename}: missing linux,phandle at {path}' + ) + + if orig_dtb.symbols !=3D recreated.symbols: + raise RuntimeError(f'{orig_dtb.filename}: /__symbols__ mismatch') + if orig_dtb.boot_cpuid_phys !=3D recreated.boot_cpuid_phys: + raise RuntimeError( + f'{orig_dtb.filename}: boot_cpuid_phys mismatch ' + f'({orig_dtb.boot_cpuid_phys} !=3D {recreated.boot_cpuid_phys}= )' + ) + if orig_dtb.mem_rsv !=3D recreated.mem_rsv: + raise RuntimeError( + f'{orig_dtb.filename}: /memreserve/ mismatch ' + f'({orig_dtb.mem_rsv} !=3D {recreated.mem_rsv})' + ) + + +def get_dtbo_output_path(dtb_path, outdir): + """Determine the output `.dtbo` file path for an input `.dtb` path. + + Args: + dtb_path: Path to the input `.dtb` file. + outdir: Output directory if `--outdir` was specified, else None. + + Returns: + Output `.dtbo` file path string. + """ + dtb_dir, stem =3D split_dtb_filename(dtb_path) + dtbo_name =3D f'{stem}.dtbo' + target_dir =3D outdir if outdir is not None else dtb_dir + return os.path.join(target_dir, dtbo_name) + + +def parse_args(): + """Parse command-line arguments. + + Returns: + Parsed `argparse.Namespace` object. + """ + parser =3D argparse.ArgumentParser( + description=3D( + 'Split a set of compiled device tree blobs (.dtb) into a singl= e ' + 'common base .dtb and per-board overlay blobs (.dtbo).' + ) + ) + parser.add_argument( + '-b', '--base-dtb', + required=3DTrue, + metavar=3D'BASE_DTB', + help=3D( + 'Output common base .dtb file path (placed in --outdir if ' + 'specified without a directory component)' + ), + ) + parser.add_argument( + '-o', '--outdir', + default=3DNone, + metavar=3D'DIR', + help=3D( + 'Output directory for generated .dtbo files and bare --base-dt= b ' + 'filenames (default: alongside each input .dtb)' + ), + ) + parser.add_argument( + '-S', '--strip-local-fixups', + action=3D'store_true', + help=3D( + 'Omit /__local_fixups__ from the generated base .dtb and ' + 'reconstructed device trees' + ), + ) + parser.add_argument( + '--verify', + action=3D'store_true', + help=3D( + 'Verify that applying each generated .dtbo to the base .dtb ' + 'recreates the original .dtb' + ), + ) + parser.add_argument( + '-v', '--verbose', + action=3D'store_true', + help=3D'Print size statistics summary', + ) + parser.add_argument( + 'dtbs', + nargs=3D'+', + metavar=3D'DTB', + help=3D'Input .dtb files (at least two required)', + ) + args =3D parser.parse_args() + if len(args.dtbs) < 2: + parser.error('at least two input .dtb files are required') + return args + + +def factor_dtb_files(args): + """Factor a list of standalone .dtb files into a base .dtb and .dtbo f= iles. + + Args: + args: Parsed `argparse.Namespace` object. + """ + input_dtb_paths =3D args.dtbs + + dtbs =3D [ParsedDtb.from_file(p) for p in input_dtb_paths] + + base_dtb_path =3D ( + os.path.join(args.outdir, args.base_dtb) + if args.outdir is not None and not os.path.dirname(args.base_dtb) + else args.base_dtb + ) + + input_abs_paths =3D {os.path.abspath(p) for p in input_dtb_paths} + base_abs_path =3D os.path.abspath(base_dtb_path) + if base_abs_path in input_abs_paths: + raise ValueError( + f'base DTB output path {base_dtb_path!r} collides with ' + f'an input DTB file' + ) + + dtbo_paths =3D [] + seen_paths =3D {base_abs_path} + for dtb_path in input_dtb_paths: + out_path =3D get_dtbo_output_path(dtb_path, args.outdir) + abs_out =3D os.path.abspath(out_path) + if abs_out in input_abs_paths: + raise ValueError( + f'output path {out_path!r} collides with an input DTB file' + ) + if abs_out in seen_paths: + raise ValueError( + f'output path collision on {out_path!r}; ensure input ' + f'files have distinct names' + ) + seen_paths.add(abs_out) + dtbo_paths.append(out_path) + + base_dtb_blob, dtbo_blobs =3D create_base_dtb_and_overlays( + dtbs, + strip_local_fixups=3Dargs.strip_local_fixups, + ) + + if args.verify: + for d, dtbo_blob in zip(dtbs, dtbo_blobs): + verify_recreated_dtb( + d, base_dtb_blob, dtbo_blob, args.strip_local_fixups + ) + + base_dtb_dir =3D os.path.dirname(base_dtb_path) + if base_dtb_dir: + os.makedirs(base_dtb_dir, exist_ok=3DTrue) + with open(base_dtb_path, 'wb') as f: + f.write(base_dtb_blob) + + if args.outdir: + os.makedirs(args.outdir, exist_ok=3DTrue) + for out_path, dtbo_blob in zip(dtbo_paths, dtbo_blobs): + out_dir =3D os.path.dirname(out_path) + if out_dir: + os.makedirs(out_dir, exist_ok=3DTrue) + with open(out_path, 'wb') as f: + f.write(dtbo_blob) + + if args.verbose: + orig_uncomp =3D sum(len(d.raw_bytes) for d in dtbs) + base_dtb_uncomp =3D len(base_dtb_blob) + dtbo_uncomp =3D sum(len(b) for b in dtbo_blobs) + new_uncomp =3D base_dtb_uncomp + dtbo_uncomp + saved_uncomp =3D orig_uncomp - new_uncomp + pct_uncomp =3D ( + (saved_uncomp / orig_uncomp * 100.0) if orig_uncomp else 0.0 + ) + + print( + f'Uncompressed DTBs: {len(dtbs)} files ({orig_uncomp:,} bytes)= -> ' + f'base {base_dtb_uncomp:,} B + {len(dtbo_blobs)} overlays ' + f'{dtbo_uncomp:,} B =3D {new_uncomp:,} bytes ' + f'(saved {saved_uncomp:,} B / {pct_uncomp:.1f}%)' + ) + + +def main(): + """Run the fdtfactor command-line tool. + + Returns: + Process exit code (0 on success). + """ + args =3D parse_args() + try: + factor_dtb_files(args) + except (OSError, ValueError, RuntimeError, libfdt.FdtException) as exc: + sys.exit(f'Error: {exc}') + return 0 + + +if __name__ =3D=3D '__main__': + sys.exit(main()) diff --git a/meson.build b/meson.build index 520cd35f9075..73b4687a2852 100644 --- a/meson.build +++ b/meson.build @@ -114,6 +114,7 @@ if get_option('tools') and not wheel_only =20 install_data( 'dtdiff', + 'fdtfactor', install_dir: get_option('bindir'), install_mode: 'rwxr-xr-x', ) diff --git a/tests/Makefile.tests b/tests/Makefile.tests index af5d309cf8a9..9f8a2896d597 100644 --- a/tests/Makefile.tests +++ b/tests/Makefile.tests @@ -53,7 +53,7 @@ TESTS_DEPFILES =3D $(TESTS:%=3D%.d) \ $(addprefix $(TESTS_PREFIX),testutils.d) =20 TESTS_CLEANFILES_L =3D $(STD_CLEANFILES) \ - *.dtb *.test.dts *.test.dt.yaml *.dtsv1 tmp.* *.bak \ + *.dtb *.test.dtbo *.test.dts *.test.dt.yaml *.dtsv1 tmp.* *.bak \ treegen TESTS_CLEANFILES =3D $(TESTS) $(TESTS_CLEANFILES_L:%=3D$(TESTS_PREFIX)%) TESTS_CLEANDIRS_L =3D fs diff --git a/tests/fdtfactor_board1.dts b/tests/fdtfactor_board1.dts new file mode 100644 index 000000000000..cab6b3a4a3cd --- /dev/null +++ b/tests/fdtfactor_board1.dts @@ -0,0 +1,69 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/dts-v1/; + +/ { + model =3D "Test Board 1"; + compatible =3D "vendor,board1", "vendor,common"; + #address-cells =3D <1>; + #size-cells =3D <1>; + + intc: interrupt-controller@1000 { + compatible =3D "vendor,intc"; + reg =3D <0x1000 0x100>; + interrupt-controller; + #interrupt-cells =3D <2>; + }; + + clk: clock-controller@2000 { + compatible =3D "vendor,clk"; + reg =3D <0x2000 0x100>; + #clock-cells =3D <1>; + }; + + soc { + compatible =3D "simple-bus"; + #address-cells =3D <1>; + #size-cells =3D <1>; + ranges; + interrupt-parent =3D <&intc>; + + uart0: serial@3000 { + compatible =3D "vendor,uart"; + reg =3D <0x3000 0x100>; + interrupts =3D <1 0>; + clocks =3D <&clk 1>; + status =3D "okay"; + }; + + i2c0: i2c@4000 { + compatible =3D "vendor,i2c"; + reg =3D <0x4000 0x100>; + interrupts =3D <2 0>; + clocks =3D <&clk 2>; + status =3D "okay"; + + pmic: pmic@40 { + compatible =3D "vendor,pmic"; + reg =3D <0x40>; + + reg_vdd: vdd { + regulator-min-microvolt =3D <1800000>; + regulator-max-microvolt =3D <1800000>; + }; + }; + }; + + spi0: spi@5000 { + compatible =3D "vendor,spi"; + reg =3D <0x5000 0x100>; + clocks =3D <&clk 3>; + vdd-supply =3D <®_vdd>; + cs-supply =3D <&board1_reg>; + }; + }; + + board1_reg: regulator-board1 { + compatible =3D "regulator-fixed"; + regulator-name =3D "board1-cs"; + }; +}; diff --git a/tests/fdtfactor_board2.dts b/tests/fdtfactor_board2.dts new file mode 100644 index 000000000000..8d3322e5ce74 --- /dev/null +++ b/tests/fdtfactor_board2.dts @@ -0,0 +1,73 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/dts-v1/; + +/ { + model =3D "Test Board 2"; + compatible =3D "vendor,board2", "vendor,common"; + #address-cells =3D <1>; + #size-cells =3D <1>; + + /* + * Extra phandle-bearing node before intc/clk shifts all subsequent + * phandle numbers in board2.dtb relative to board1.dtb. + */ + board2_early_reg: regulator-early { + compatible =3D "regulator-fixed"; + regulator-name =3D "board2-early"; + }; + + intc: interrupt-controller@1000 { + compatible =3D "vendor,intc"; + reg =3D <0x1000 0x100>; + interrupt-controller; + #interrupt-cells =3D <2>; + }; + + clk: clock-controller@2000 { + compatible =3D "vendor,clk"; + reg =3D <0x2000 0x100>; + #clock-cells =3D <1>; + }; + + soc { + compatible =3D "simple-bus"; + #address-cells =3D <1>; + #size-cells =3D <1>; + ranges; + interrupt-parent =3D <&intc>; + + uart0: serial@3000 { + compatible =3D "vendor,uart"; + reg =3D <0x3000 0x100>; + interrupts =3D <1 0>; + clocks =3D <&clk 1>; + status =3D "disabled"; + }; + + i2c0: i2c@4000 { + compatible =3D "vendor,i2c"; + reg =3D <0x4000 0x100>; + interrupts =3D <2 0>; + clocks =3D <&clk 2>; + status =3D "okay"; + + pmic: pmic@40 { + compatible =3D "vendor,pmic"; + reg =3D <0x40>; + + reg_vdd: vdd { + regulator-min-microvolt =3D <1800000>; + regulator-max-microvolt =3D <3300000>; + }; + }; + }; + + mmc0: mmc@6000 { + compatible =3D "vendor,mmc"; + reg =3D <0x6000 0x100>; + clocks =3D <&clk 4>; + vmmc-supply =3D <®_vdd>; + vqmmc-supply =3D <&board2_early_reg>; + }; + }; +}; diff --git a/tests/fdtfactor_board3.dts b/tests/fdtfactor_board3.dts new file mode 100644 index 000000000000..1a0fb1641241 --- /dev/null +++ b/tests/fdtfactor_board3.dts @@ -0,0 +1,96 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/dts-v1/; + +/ { + model =3D "Test Board 3"; + compatible =3D "vendor,board3", "vendor,common"; + #address-cells =3D <1>; + #size-cells =3D <1>; + + /* + * Two extra phandle-bearing nodes before intc/clk shift all subsequent + * phandle numbers in board3.dtb relative to both board1.dtb and + * board2.dtb. + */ + board3_early_reg1: regulator-early1 { + compatible =3D "regulator-fixed"; + regulator-name =3D "board3-early1"; + }; + + board3_early_reg2: regulator-early2 { + compatible =3D "regulator-fixed"; + regulator-name =3D "board3-early2"; + vin-supply =3D <&board3_early_reg1>; + }; + + intc: interrupt-controller@1000 { + compatible =3D "vendor,intc"; + reg =3D <0x1000 0x100>; + interrupt-controller; + #interrupt-cells =3D <2>; + }; + + clk: clock-controller@2000 { + compatible =3D "vendor,clk"; + reg =3D <0x2000 0x100>; + #clock-cells =3D <1>; + }; + + soc { + compatible =3D "simple-bus"; + #address-cells =3D <1>; + #size-cells =3D <1>; + ranges; + interrupt-parent =3D <&intc>; + + uart0: serial@3000 { + compatible =3D "vendor,uart"; + reg =3D <0x3000 0x100>; + interrupts =3D <1 0>; + clocks =3D <&clk 1>; + status =3D "okay"; + }; + + i2c0: i2c@4000 { + compatible =3D "vendor,i2c"; + reg =3D <0x4000 0x100>; + interrupts =3D <2 0>; + clocks =3D <&clk 2>; + status =3D "okay"; + + pmic: pmic@40 { + compatible =3D "vendor,pmic"; + reg =3D <0x40>; + + reg_vdd: vdd { + regulator-min-microvolt =3D <1800000>; + regulator-max-microvolt =3D <3300000>; + }; + }; + }; + + spi0: spi@5000 { + compatible =3D "vendor,spi"; + reg =3D <0x5000 0x100>; + clocks =3D <&clk 3>; + vdd-supply =3D <®_vdd>; + cs-supply =3D <&board3_early_reg2>; + }; + + mmc0: mmc@6000 { + compatible =3D "vendor,mmc"; + reg =3D <0x6000 0x100>; + clocks =3D <&clk 4>; + vmmc-supply =3D <®_vdd>; + vqmmc-supply =3D <&board3_early_reg1>; + }; + + usb0: usb@7000 { + compatible =3D "vendor,usb"; + reg =3D <0x7000 0x100>; + interrupts =3D <5 0>; + clocks =3D <&clk 5>; + vbus-supply =3D <&board3_early_reg2>; + }; + }; +}; diff --git a/tests/run_tests.sh b/tests/run_tests.sh index 842b543059bb..92fa417d3457 100755 --- a/tests/run_tests.sh +++ b/tests/run_tests.sh @@ -239,6 +239,21 @@ run_fdtoverlay_test() { base_run_test sh "$SRCDIR/fdtoverlay-runtest.sh" "$expect" "$@" } =20 +# Compare an original .dtb and a recreated .dtb by decompiling and sorting= both +# with dtc -I dtb -O dts -qq -f -s (like dtdiff) and normalizing the numer= ic +# phandle properties assigned to target nodes. Because both blobs contain +# /__symbols__ and /__local_fixups__, dtc restores all node labels and con= verts +# all phandle references in properties back into symbolic <&label> referen= ces. +check_dtb_recreated () { + orig=3D"$1" + recreated=3D"$2" + $DTC -I dtb -O dts -qq -f -s -o "$orig.raw.test.dts" "$orig" || return= 1 + $DTC -I dtb -O dts -qq -f -s -o "$recreated.raw.test.dts" "$recreated"= || return 1 + sed 's/phandle =3D <0x[0-9a-f]*>;/phandle =3D <0xffffffff>;/' "$orig.r= aw.test.dts" > "$orig.cmp.test.dts" || return 1 + sed 's/phandle =3D <0x[0-9a-f]*>;/phandle =3D <0xffffffff>;/' "$recrea= ted.raw.test.dts" > "$recreated.cmp.test.dts" || return 1 + diff -u "$orig.cmp.test.dts" "$recreated.cmp.test.dts" +} + BAD_FIXUP_TREES=3D"bad_index \ empty \ empty_index \ @@ -1089,6 +1104,50 @@ fdtoverlay_tests() { =20 shorten_echo "check phandle to noda d wasn't overwritten: " run_wrap_test test "$bd" =3D "$pd" + + if ! $no_python; then + run_dtc_test -@ -L -I dts -O dtb -o fdtfactor_board1.test.dtb "$SRCDIR/fd= tfactor_board1.dts" + run_dtc_test -@ -L -I dts -O dtb -o fdtfactor_board2.test.dtb "$SRCDIR/fd= tfactor_board2.dts" + run_dtc_test -@ -L -I dts -O dtb -o fdtfactor_board3.test.dtb "$SRCDIR/fd= tfactor_board3.dts" + + # Symbolic phandle deduplication mode (3 boards) + run_wrap_test $PYTHON "$FDTFACTOR" -v --verify \ + -b fdtfactor_base.test.dtb \ + fdtfactor_board1.test.dtb fdtfactor_board2.test.dtb fdtfactor_board3.= test.dtb + run_wrap_test $FDTOVERLAY -i fdtfactor_base.test.dtb \ + -o fdtfactor_board1.recreated.test.dtb fdtfactor_board1.test.dtbo + run_wrap_test $FDTOVERLAY -i fdtfactor_base.test.dtb \ + -o fdtfactor_board2.recreated.test.dtb fdtfactor_board2.test.dtbo + run_wrap_test $FDTOVERLAY -i fdtfactor_base.test.dtb \ + -o fdtfactor_board3.recreated.test.dtb fdtfactor_board3.test.dtbo + run_wrap_test check_dtb_recreated fdtfactor_board1.test.dtb fdtfactor_boa= rd1.recreated.test.dtb + run_wrap_test check_dtb_recreated fdtfactor_board2.test.dtb fdtfactor_boa= rd2.recreated.test.dtb + run_wrap_test check_dtb_recreated fdtfactor_board3.test.dtb fdtfactor_boa= rd3.recreated.test.dtb + # Majority value ("okay" in boards 1 & 3) should be in base.dtb + run_fdtget_test "okay" fdtfactor_base.test.dtb /soc/serial@3000 status -ts + + # Strip local fixups mode (3 boards) + run_wrap_test $PYTHON "$FDTFACTOR" --verify -S \ + --base-dtb fdtfactor_base_stripped.test.dtb -o . \ + fdtfactor_board1.test.dtb fdtfactor_board2.test.dtb fdtfactor_board3.= test.dtb + run_wrap_test $FDTOVERLAY -i fdtfactor_base_stripped.test.dtb \ + -o fdtfactor_board1.stripped.test.dtb fdtfactor_board1.test.dtbo + run_wrap_test $FDTOVERLAY -i fdtfactor_base_stripped.test.dtb \ + -o fdtfactor_board2.stripped.test.dtb fdtfactor_board2.test.dtbo + run_wrap_test $FDTOVERLAY -i fdtfactor_base_stripped.test.dtb \ + -o fdtfactor_board3.stripped.test.dtb fdtfactor_board3.test.dtbo + run_test check_path fdtfactor_board1.stripped.test.dtb not-exists "/__loc= al_fixups__" + # Without /__local_fixups__, dtc -I dtb -O dts cannot turn phandle cells + # inside properties back into <&label> references, so compare stripped + # against recreated (after removing /__local_fixups__), which shares + # fdtfactor's phandle numbering and was already verified against orig. + run_wrap_test $DTPUT -r fdtfactor_board1.recreated.test.dtb /__local_fixu= ps__ + run_wrap_test $DTPUT -r fdtfactor_board2.recreated.test.dtb /__local_fixu= ps__ + run_wrap_test $DTPUT -r fdtfactor_board3.recreated.test.dtb /__local_fixu= ps__ + run_wrap_test check_dtb_recreated fdtfactor_board1.recreated.test.dtb fdt= factor_board1.stripped.test.dtb + run_wrap_test check_dtb_recreated fdtfactor_board2.recreated.test.dtb fdt= factor_board2.stripped.test.dtb + run_wrap_test check_dtb_recreated fdtfactor_board3.recreated.test.dtb fdt= factor_board3.stripped.test.dtb + fi } =20 pylibfdt_tests () { @@ -1139,7 +1198,7 @@ if [ -z "$TESTSETS" ]; then fi =20 # Make sure we don't have stale blobs lying around -rm -f *.test.dtb *.test.dts +rm -f *.test.dtb *.test.dtbo *.test.dts =20 for set in $TESTSETS; do case $set in diff --git a/tests/testutils.sh b/tests/testutils.sh index 6b2f0d14e3e5..7f35ba8e4158 100644 --- a/tests/testutils.sh +++ b/tests/testutils.sh @@ -27,6 +27,7 @@ DTGET=3D${TEST_BINDIR}/fdtget DTPUT=3D${TEST_BINDIR}/fdtput FDTDUMP=3D${TEST_BINDIR}/fdtdump FDTOVERLAY=3D${TEST_BINDIR}/fdtoverlay +FDTFACTOR=3D${SRCDIR}/../fdtfactor =20 verbose_run () { if [ -z "$QUIET_TEST" ]; then --=20 2.56.0.176.g3575e65682-goog From nobody Fri Oct 23 03:30:57 2026 Received: from mail-dy1-f175.google.com (mail-dy1-f175.google.com [74.125.82.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C043457F73B for ; Thu, 15 Oct 2026 18:23:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.82.175 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1792088603; cv=none; b=CrifhE0t/PNwn4+vKgIaY3XTXvkAo/wQfjH6TK7lpfbUevaqYm3A1VsvCf6kGufBEuTGR7nKPdsZ5uXlzQq8cL16FQfyfgnmoXEvT/hThfKIqirWMJwIiFJGP3nnG03mHS0XvMytcrnZf+jvGIkDMuPxp8Mfmbusxdgxmcrncio= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1792088603; c=relaxed/simple; bh=tioGJIMeyj2PWzM6Uv7IGFe5Z2skPmofv2szHNtMkvk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=bD0cLIpTnhnvaWgTHRkvspAm/JiAn5p85uMrrVa3m7YmsUYONGlBIqN/bPsDf5GBHJGZs5fpv7Z4cvf8a5F9Y7MtPHQDso1aLLJdKBozs3IMHoBh0+kO7Yu98m0ethQsf4CUelntOHj0sfd+BGwQ1EMo1UjHIxkxQqkWptgF6zo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=FyJTf6Nk; arc=none smtp.client-ip=74.125.82.175 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="FyJTf6Nk" Received: by mail-dy1-f175.google.com with SMTP id 5a478bee46e88-35829219da8so320926eec.0 for ; Thu, 15 Oct 2026 11:23:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1792088591; x=1792693391; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=WQ9fGwFGxn8dcozazfByEfm6bIeBTesM/0e15adWCYw=; b=FyJTf6NkbS32YVKK9emYHQyFLysA8c4pbaIaBhU6oJ1yJO1n2RxlqF58kko1zf0eFl 4uhO6+zyb61a7D52nGHcJin11PMWnAKgDoDuEHtpmTcn+6RXeDN+h6w29J9gLpIzWNlb JNFLP5wk5QvklAKPs+KtwGLaCEKfA+fcrDKPs= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1792088591; x=1792693391; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=WQ9fGwFGxn8dcozazfByEfm6bIeBTesM/0e15adWCYw=; b=IzVQ5/0YrU6HHNE83yQCQtzJh1gGIXnbshh/FC0XF9G5cNQNCn2jSumpSXzHWespy8 AIoldm8NsEjRiVoCCZmpEfSW0a7cbAw819TAlBHZzZuzGLoGSEDuOMx23JV9+WZBjN+P Ltus7C/pHN1ErdJTAzfElWpNp6jKghONOFcXiz356Ho+L/v/X0BhCxktB/SUb/0qJju9 rfoVRSSTB7J5ARmn7v/etswbPIB2BqjnJ3A5Bt7r62ykG3xOgEi1xiZ3D2qbHOCqCTUp UZ1FdDRDW3otrYtCL5oTKGwkFFEz0bokG5QP5OOkQnSkM45UYE6H52sS1j8aXk+UlMKN 0sNw== X-Forwarded-Encrypted: i=1; AKwUvBwmGsfDONOcm++a5A7tsS/njU6h8NOgLWQPEJuAXfIvN9TPRPul5UIhLAvO8Le4yDSypyH0fCPN+RLbO54=@vger.kernel.org X-Gm-Message-State: AFq9FYLUKIdPrrdOqQYjAW3eLLvNcTpyAZyMmq2ffd43oVDH7xIbLeoI UqyXY4/CWNMyjzsV7clmgeo64Y8IJrTfvPxCVVSWZNTK6z2w/o4OzzK1kUAMGQGxng== X-Gm-Gg: AYBFou3n86MVqADXRKII021HRbOJ0TmB6d6wJ6oxGfWPCUOLK8jNp65CdMoYYJDgctc JgG5HCpT9mCtJUjb8FKXIzsccTngQNgiFC/Gpfg99Ieh86HakvBvFq7fNUe6FKkG5cRTxVUvepw w9Q5+YygiW/G01AU0jwPvCNQ1bBbKFUStDkbDdmnaEvIw1IeMrFF/bYVHrOVrmpzSETPy3KYFrj FA7NOT8orG1fgy855HlaioBPCv1MSGY98SLC+WZTj859PdVoDPPKt9w/HLTObpeGN9PihrNtQ+N r53Q7HLnCDgR2T0aq5S4GFi5gTEMaS+4OBl3XxxRbZv79cdedz2lqqdHVZv74KnJMAdCGzpfm1O pJbfEeeEQ51LeaP07z/eD9SMn8jrYjTJ7J+n33CRZKoW3POiTd6X8ojTMHP/QY3GzVAwuKNmfwi 2FPPgLuTupDvFr1wTa3rWM+O4fS3WrpPFrmzaiKsmWugqx5DEXO//rr+JDlPGkiDp+v/Qx+os3h PBE/bP/YdLIqDoollY7GzbwVqxoOoCqzPipLyq7hT7KR9A+7cXm6kYDqE4= X-Received: by 2002:a05:701b:4504:10b0:182:c472:b123 with SMTP id a92af1059eb24-182c472d135mr173954c88.1.1792088591080; Thu, 15 Oct 2026 11:23:11 -0700 (PDT) Received: from dianders.sjc.corp.google.com ([2a00:79e0:2e7c:8:9ea0:3a29:323e:5f11]) by smtp.gmail.com with ESMTPSA id a92af1059eb24-182a8a4a56bsm727204c88.20.2026.10.15.11.23.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 15 Oct 2026 11:23:09 -0700 (PDT) From: Douglas Anderson To: Devicetree-compiler@vger.kernel.org, Rob Herring , Krzysztof Kozlowski , Conor Dooley , Konrad Dybcio Cc: Chen-Yu Tsai , linux-kernel@vger.kernel.org, Devicetree@vger.kernel.org, Brian Norris , Tudor Ambarus , =?UTF-8?q?Andr=C3=A9=20Draszik?= , Julius Werner , Saravana Kannan , Peter Griffin , Douglas Anderson Subject: [PATCH 3/4] fdtfactor: Support compressed DTB inputs and outputs Date: Thu, 15 Oct 2026 11:21:36 -0700 Message-ID: <20261015182221.2931742-4-dianders@chromium.org> X-Mailer: git-send-email 2.56.0.176.g3575e65682-goog In-Reply-To: <20261015182221.2931742-1-dianders@chromium.org> References: <20261015182221.2931742-1-dianders@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Firmware and FIT image build pipelines frequently store device tree blobs compressed with LZ4, LZMA/XZ, or gzip (e.g., `.dtb.lz4`). Extend `fdtfactor` to detect compressed input blobs by magic bytes or file extension, transparently decompress them before factoring, and compress the resulting `.dtbo` and `base.dtb` outputs when appropriate: - Support `.lz4` (via the `lz4` CLI), `.lzma`/`.xz` (via Python's `lzma` module), and `.gz` (via Python's `gzip` module). - Preserve the input compression format on generated `.dtbo` files and compress `base.dtb` when its output path carries a compression suffix (e.g., `-b base.dtb.lz4`). - Report compressed size savings alongside uncompressed savings when `-v`/`--verbose` is specified. On the 47 Qualcomm SC7180 Trogdor Chromebook `.dtb.lz4` files, `fdtfactor -S` reduces total LZ4-compressed size from 1.71 MB to 213 KB (87.5% savings). Assisted-by: Gemini:gemini-pro-prerelease Signed-off-by: Douglas Anderson --- README.md | 16 +-- fdtfactor | 267 ++++++++++++++++++++++++++++++++++++++----- tests/Makefile.tests | 5 +- tests/run_tests.sh | 34 +++++- 4 files changed, 284 insertions(+), 38 deletions(-) diff --git a/README.md b/README.md index e2ff2374b01d..6ace31bfbfdb 100644 --- a/README.md +++ b/README.md @@ -94,10 +94,10 @@ values. ## fdtfactor =20 `fdtfactor` is a Python tool built on `pylibfdt` that takes a set of -compiled device tree blobs (`.dtb`) and splits them into a single common -`base.dtb` and per-board `.dtbo` overlays so that applying a board's -`.dtbo` to `base.dtb` recreates the original board device tree while -reducing total storage size. +compiled device tree blobs (`.dtb`, or compressed `.dtb.lz4`, `.dtb.xz`, +`.dtb.lzma`, `.dtb.gz`) and splits them into a single common `base.dtb` and +per-board `.dtbo` overlays so that applying a board's `.dtbo` to `base.dtb` +recreates the original board device tree while reducing total storage size. =20 Input `.dtb` files must be compiled with `dtc -@ -L` so that both `/__symbols__` (used to resolve overlay `/__fixups__` against `base.dtb`) @@ -112,8 +112,10 @@ $ fdtfactor -b base.dtb -o out_dir board1.dtb board2.d= tb board3.dtb =20 Options: =20 -* `-b`, `--base-dtb`: Output path for the common base `.dtb` (placed in - `--outdir` if specified without a directory component). +* `-b`, `--base-dtb`: Output path for the common base `.dtb` (compressed if + given a `.lz4`, `.xz`, `.lzma`, or `.gz` suffix, whereas each `.dtbo` + preserves its input `.dtb`'s compression; placed in `--outdir` if specif= ied + without a directory component). * `-o`, `--outdir`: Output directory for generated `.dtbo` files (and for `--base-dtb` when given without a directory component). * `-S`, `--strip-local-fixups`: Omit `/__local_fixups__` from the generated @@ -127,7 +129,7 @@ Options: original `.dtb`'s `/__local_fixups__` subtree verbatim. * `--verify`: Verify that applying each generated `.dtbo` to `base.dtb` with `libfdt` recreates the original `.dtb`. -* `-v`, `--verbose`: Print size savings summary. +* `-v`, `--verbose`: Print uncompressed and compressed size savings. =20 ### TODO =20 diff --git a/fdtfactor b/fdtfactor index 377f499bea50..9f06801f973c 100755 --- a/fdtfactor +++ b/fdtfactor @@ -11,31 +11,195 @@ original device tree. =20 import argparse from collections import Counter +import gzip +import lzma import os import struct +import subprocess import sys =20 import libfdt =20 =20 +FDT_MAGIC =3D b'\xd0\x0d\xfe\xed' +LZ4_MAGIC =3D b'\x04\x22\x4d\x18' +LZ4_LEGACY_MAGIC =3D b'\x02\x21\x4c\x18' +GZIP_MAGIC =3D b'\x1f\x8b' +XZ_MAGIC =3D b'\xfd7zXZ\x00' +# Matches raw LZMA_ALONE headers with default lc=3D3, lp=3D0, pb=3D2 (0x5d= ) and +# power-of-two dictionary sizes >=3D 64 KiB; .lzma extension handles any +# non-default LZMA headers. +LZMA_ALONE_MAGIC =3D b'\x5d\x00\x00' + +COMP_TO_EXT =3D { + 'none': '', + 'lz4': '.lz4', + 'lzma': '.lzma', + 'xz': '.xz', + 'gzip': '.gz', +} + +EXT_TO_COMP =3D { + '.lz4': 'lz4', + '.lzma': 'lzma', + '.xz': 'xz', + '.gz': 'gzip', +} + + +def detect_compression(raw_bytes, filename=3D''): + """Detect compression algorithm from magic bytes or filename extension. + + Args: + raw_bytes: Input file or blob contents as `bytes`. + filename: Optional file path or name for extension fallback. + + Returns: + Compression algorithm name (`'none'`, `'lz4'`, `'lzma'`, `'xz'`, or + `'gzip'`). + """ + if raw_bytes.startswith(FDT_MAGIC): + return 'none' + if (raw_bytes.startswith(LZ4_MAGIC) or + raw_bytes.startswith(LZ4_LEGACY_MAGIC)): + return 'lz4' + if raw_bytes.startswith(GZIP_MAGIC): + return 'gzip' + if raw_bytes.startswith(XZ_MAGIC): + return 'xz' + if raw_bytes.startswith(LZMA_ALONE_MAGIC): + return 'lzma' + _, ext =3D os.path.splitext(filename) + if ext in EXT_TO_COMP: + return EXT_TO_COMP[ext] + return 'none' + + +def decompress_blob(raw_bytes, filename=3D'', comp=3DNone): + """Decompress a DTB blob if compressed. + + Args: + raw_bytes: Input file or blob contents as `bytes`. + filename: Optional file path or name for error reporting and exten= sion + detection. + comp: Optional explicit compression algorithm name; auto-detected = if + None or empty. + + Returns: + Tuple of `(uncompressed_bytes, compression_type)`. + + Raises: + RuntimeError: If `lz4` is missing or fails to decompress `raw_byte= s`. + ValueError: If the compression type is unsupported. + """ + if not comp: + comp =3D detect_compression(raw_bytes, filename) + if comp =3D=3D 'none': + return raw_bytes, 'none' + if comp =3D=3D 'lz4': + try: + res =3D subprocess.run( + ['lz4', '-d', '-q', '-c'], + input=3Draw_bytes, + capture_output=3DTrue, + check=3DFalse, + ) + except FileNotFoundError as exc: + raise RuntimeError( + 'lz4 command not found; install lz4 to process .lz4 files' + ) from exc + if res.returncode !=3D 0: + raise RuntimeError( + f'Failed to decompress {filename} with lz4: ' + f'{res.stderr.decode("utf-8", errors=3D"replace").strip()}' + ) + return res.stdout, 'lz4' + if comp =3D=3D 'gzip': + try: + return gzip.decompress(raw_bytes), 'gzip' + except Exception as exc: + raise RuntimeError( + f'Failed to decompress {filename} with gzip: {exc}' + ) from exc + if comp in ('lzma', 'xz'): + try: + return lzma.decompress(raw_bytes), comp + except Exception as exc: + raise RuntimeError( + f'Failed to decompress {filename} with {comp}: {exc}' + ) from exc + prefix =3D f'{filename}: ' if filename else '' + raise ValueError(f'{prefix}Unsupported compression type: {comp}') + + +def compress_blob(raw_bytes, comp): + """Compress an FDT blob using the specified compression algorithm. + + Args: + raw_bytes: Uncompressed FDT blob as `bytes`. + comp: Compression algorithm name (`'none'`, `'lz4'`, `'lzma'`, `'x= z'`, + or `'gzip'`). + + Returns: + Compressed (or uncompressed if `comp` is `'none'`) blob as `bytes`. + + Raises: + RuntimeError: If `lz4` is missing or fails to compress `raw_bytes`. + ValueError: If `comp` is not a supported compression type. + """ + if comp in ('none', '', None): + return raw_bytes + if comp =3D=3D 'lz4': + try: + res =3D subprocess.run( + ['lz4', '-12', '-z', '-q', '-c'], + input=3Draw_bytes, + capture_output=3DTrue, + check=3DFalse, + ) + except FileNotFoundError as exc: + raise RuntimeError( + 'lz4 command not found; install lz4 to process .lz4 files' + ) from exc + if res.returncode !=3D 0: + raise RuntimeError( + f'Failed to compress blob with lz4: ' + f'{res.stderr.decode("utf-8", errors=3D"replace").strip()}' + ) + return res.stdout + if comp =3D=3D 'gzip': + return gzip.compress(raw_bytes, mtime=3D0) + if comp =3D=3D 'lzma': + return lzma.compress(raw_bytes, format=3Dlzma.FORMAT_ALONE) + if comp =3D=3D 'xz': + return lzma.compress(raw_bytes, format=3Dlzma.FORMAT_XZ) + raise ValueError(f'Unsupported compression type: {comp}') + + def split_dtb_filename(path): - """Split a DTB path into its directory and filename stem. + """Split a DTB path into its directory, stem, and compression type. =20 - Example: - '/dir/board.dtb' -> ('/dir', 'board') + Examples: + '/dir/board.dtb' -> ('/dir', 'board', 'none') + '/dir/board.dtb.lz4' -> ('/dir', 'board', 'lz4') =20 Args: - path: File path to a .dtb or .dtbo file. + path: File path to a `.dtb` or `.dtbo` (optionally compressed) fil= e. =20 Returns: - Tuple of (directory, stem) where `stem` has any trailing `.dtb` or - `.dtbo` suffix removed. + Tuple of `(directory, stem, compression_type)` where `stem` has any + trailing compression and `.dtb`/`.dtbo` suffixes removed. """ directory, filename =3D os.path.split(path) + comp =3D 'none' + prefix, sep, ext =3D filename.rpartition('.') + if prefix and sep and f'.{ext}' in EXT_TO_COMP: + comp =3D EXT_TO_COMP[f'.{ext}'] + filename =3D prefix stem, sep, ext =3D filename.rpartition('.') if not stem or not sep or ext not in ('dtb', 'dtbo'): stem =3D filename - return directory, stem + return directory, stem, comp =20 =20 class NodeTree: @@ -172,15 +336,24 @@ def _is_dtbo_overlay(fdt_ro): class ParsedDtb: """Parsed representation of an input .dtb blob.""" =20 - def __init__(self, filename, raw_bytes): + def __init__(self, filename, raw_bytes, compressed_size=3DNone, + compression=3D'none'): """Parse an uncompressed FDT blob into nodes, symbols, and fixups. =20 Args: filename: Display name or file path for error reporting. raw_bytes: Uncompressed FDT binary contents as `bytes`. + compressed_size: Original on-disk size in bytes before + decompression (defaults to `len(raw_bytes)`). + compression: Compression algorithm of the input file (`'none'`, + `'lz4'`, `'lzma'`, `'xz'`, or `'gzip'`). """ self.filename =3D filename self.raw_bytes =3D raw_bytes + self.compressed_size =3D ( + compressed_size if compressed_size is not None else len(raw_by= tes) + ) + self.compression =3D compression fdt =3D _open_fdt_ro(raw_bytes, filename) if _is_dtbo_overlay(fdt): raise ValueError( @@ -211,17 +384,19 @@ class ParsedDtb: =20 @classmethod def from_file(cls, path): - """Read and parse a .dtb file from disk. + """Read and parse a .dtb (or compressed .dtb.lz4/.lzma/.xz/.gz) fi= le. =20 Args: - path: Path to the input .dtb file. + path: Path to the input .dtb (or compressed .dtb) file. =20 Returns: A new `ParsedDtb` instance. """ with open(path, 'rb') as f: - raw_bytes =3D f.read() - return cls(path, raw_bytes) + file_bytes =3D f.read() + raw_bytes, comp =3D decompress_blob(file_bytes, path) + return cls(path, raw_bytes, compressed_size=3Dlen(file_bytes), + compression=3Dcomp) =20 def _parse_tree(self, fdt): """Walk the FDT and extract nodes, properties, symbols, and fixups. @@ -1036,18 +1211,20 @@ def verify_recreated_dtb(orig_dtb, base_dtb_blob, d= tbo_blob, ) =20 =20 -def get_dtbo_output_path(dtb_path, outdir): - """Determine the output `.dtbo` file path for an input `.dtb` path. +def get_dtbo_output_path(dtb_path, outdir, comp=3D'none'): + """Determine the output `.dtbo[.comp]` file path for an input `.dtb` p= ath. =20 Args: - dtb_path: Path to the input `.dtb` file. + dtb_path: Path to the input `.dtb` (or compressed `.dtb`) file. outdir: Output directory if `--outdir` was specified, else None. + comp: Compression algorithm name (`'none'`, `'lz4'`, `'lzma'`, `'x= z'`, + or `'gzip'`). =20 Returns: - Output `.dtbo` file path string. + Output `.dtbo[.comp]` file path string. """ - dtb_dir, stem =3D split_dtb_filename(dtb_path) - dtbo_name =3D f'{stem}.dtbo' + dtb_dir, stem, _ =3D split_dtb_filename(dtb_path) + dtbo_name =3D f'{stem}.dtbo{COMP_TO_EXT[comp]}' target_dir =3D outdir if outdir is not None else dtb_dir return os.path.join(target_dir, dtbo_name) =20 @@ -1060,8 +1237,9 @@ def parse_args(): """ parser =3D argparse.ArgumentParser( description=3D( - 'Split a set of compiled device tree blobs (.dtb) into a singl= e ' - 'common base .dtb and per-board overlay blobs (.dtbo).' + 'Split a set of compiled device tree blobs (.dtb, or compresse= d ' + '.dtb.lz4/.xz/.lzma/.gz) into a single common base .dtb and ' + 'per-board overlay blobs (.dtbo).' ) ) parser.add_argument( @@ -1069,8 +1247,8 @@ def parse_args(): required=3DTrue, metavar=3D'BASE_DTB', help=3D( - 'Output common base .dtb file path (placed in --outdir if ' - 'specified without a directory component)' + 'Output common base .dtb (or .dtb.lz4/.xz/.lzma/.gz) file path= ' + '(placed in --outdir if specified without a directory componen= t)' ), ) parser.add_argument( @@ -1101,13 +1279,16 @@ def parse_args(): parser.add_argument( '-v', '--verbose', action=3D'store_true', - help=3D'Print size statistics summary', + help=3D'Print size statistics and compression summary', ) parser.add_argument( 'dtbs', nargs=3D'+', metavar=3D'DTB', - help=3D'Input .dtb files (at least two required)', + help=3D( + 'Input .dtb (or compressed .dtb.lz4/.xz/.lzma/.gz) files (at ' + 'least two required)' + ), ) args =3D parser.parse_args() if len(args.dtbs) < 2: @@ -1130,6 +1311,7 @@ def factor_dtb_files(args): if args.outdir is not None and not os.path.dirname(args.base_dtb) else args.base_dtb ) + _, _, base_dtb_comp =3D split_dtb_filename(base_dtb_path) =20 input_abs_paths =3D {os.path.abspath(p) for p in input_dtb_paths} base_abs_path =3D os.path.abspath(base_dtb_path) @@ -1141,8 +1323,8 @@ def factor_dtb_files(args): =20 dtbo_paths =3D [] seen_paths =3D {base_abs_path} - for dtb_path in input_dtb_paths: - out_path =3D get_dtbo_output_path(dtb_path, args.outdir) + for dtb_path, d in zip(input_dtb_paths, dtbs): + out_path =3D get_dtbo_output_path(dtb_path, args.outdir, comp=3Dd.= compression) abs_out =3D os.path.abspath(out_path) if abs_out in input_abs_paths: raise ValueError( @@ -1167,20 +1349,24 @@ def factor_dtb_files(args): d, base_dtb_blob, dtbo_blob, args.strip_local_fixups ) =20 + base_dtb_file_bytes =3D compress_blob(base_dtb_blob, base_dtb_comp) base_dtb_dir =3D os.path.dirname(base_dtb_path) if base_dtb_dir: os.makedirs(base_dtb_dir, exist_ok=3DTrue) with open(base_dtb_path, 'wb') as f: - f.write(base_dtb_blob) + f.write(base_dtb_file_bytes) =20 if args.outdir: os.makedirs(args.outdir, exist_ok=3DTrue) - for out_path, dtbo_blob in zip(dtbo_paths, dtbo_blobs): + dtbo_file_sizes =3D [] + for out_path, dtbo_blob, d in zip(dtbo_paths, dtbo_blobs, dtbs): out_dir =3D os.path.dirname(out_path) if out_dir: os.makedirs(out_dir, exist_ok=3DTrue) + out_bytes =3D compress_blob(dtbo_blob, d.compression) + dtbo_file_sizes.append(len(out_bytes)) with open(out_path, 'wb') as f: - f.write(dtbo_blob) + f.write(out_bytes) =20 if args.verbose: orig_uncomp =3D sum(len(d.raw_bytes) for d in dtbs) @@ -1198,6 +1384,29 @@ def factor_dtb_files(args): f'{dtbo_uncomp:,} B =3D {new_uncomp:,} bytes ' f'(saved {saved_uncomp:,} B / {pct_uncomp:.1f}%)' ) + all_comps =3D {base_dtb_comp, *(d.compression for d in dtbs)} + comp_names =3D sorted(c for c in all_comps if c !=3D 'none') + if comp_names: + comp_label =3D '/'.join(sorted(all_comps)) + hdr =3D ( + f'Compressed ({comp_label}):' + if len(all_comps) =3D=3D 1 + else f'On-disk ({comp_label}): ' + ) + orig_comp =3D sum(d.compressed_size for d in dtbs) + base_dtb_csize =3D len(base_dtb_file_bytes) + dtbo_csize =3D sum(dtbo_file_sizes) + new_comp =3D base_dtb_csize + dtbo_csize + saved_comp =3D orig_comp - new_comp + pct_comp =3D (saved_comp / orig_comp * 100.0) if orig_comp els= e 0.0 + print( + f'{hdr} {len(dtbs)} files ' + f'({orig_comp:,} bytes) -> base {base_dtb_csize:,} B + ' + f'{len(dtbo_blobs)} overlays {dtbo_csize:,} B ' + f'(avg {dtbo_csize // len(dtbo_blobs):,} B) =3D ' + f'{new_comp:,} bytes ' + f'(saved {saved_comp:,} B / {pct_comp:.1f}%)' + ) =20 =20 def main(): diff --git a/tests/Makefile.tests b/tests/Makefile.tests index 9f8a2896d597..d0bdc4afefb2 100644 --- a/tests/Makefile.tests +++ b/tests/Makefile.tests @@ -53,7 +53,10 @@ TESTS_DEPFILES =3D $(TESTS:%=3D%.d) \ $(addprefix $(TESTS_PREFIX),testutils.d) =20 TESTS_CLEANFILES_L =3D $(STD_CLEANFILES) \ - *.dtb *.test.dtbo *.test.dts *.test.dt.yaml *.dtsv1 tmp.* *.bak \ + *.dtb *.test.dtbo *.test.dts *.test.dt.yaml *.dtsv1 \ + *.test.dtb.gz *.test.dtbo.gz *.test.dtb.xz *.test.dtbo.xz \ + *.test.dtb.lzma *.test.dtbo.lzma *.test.dtb.lz4 *.test.dtbo.lz4 \ + tmp.* *.bak \ treegen TESTS_CLEANFILES =3D $(TESTS) $(TESTS_CLEANFILES_L:%=3D$(TESTS_PREFIX)%) TESTS_CLEANDIRS_L =3D fs diff --git a/tests/run_tests.sh b/tests/run_tests.sh index 92fa417d3457..92edec8b724c 100755 --- a/tests/run_tests.sh +++ b/tests/run_tests.sh @@ -1147,6 +1147,36 @@ fdtoverlay_tests() { run_wrap_test check_dtb_recreated fdtfactor_board1.recreated.test.dtb fdt= factor_board1.stripped.test.dtb run_wrap_test check_dtb_recreated fdtfactor_board2.recreated.test.dtb fdt= factor_board2.stripped.test.dtb run_wrap_test check_dtb_recreated fdtfactor_board3.recreated.test.dtb fdt= factor_board3.stripped.test.dtb + + # Compressed (.dtb.gz / .dtb.xz / .dtb.lzma / .dtb.lz4) modes (3 boards) + for comp_spec in "gz:gzip -9 -k -f:gzip -d -f" \ + "xz:xz -k -f:xz -d -f" \ + "lzma:xz --format=3Dlzma -k -f:xz -d -f" \ + "lz4:lz4 -12 -q -f -m:lz4 -d -q -f --rm"; do + ext=3D"${comp_spec%%:*}" + rest=3D"${comp_spec#*:}" + comp_cmd=3D"${rest%%:*}" + decomp_cmd=3D"${rest#*:}" + if ! command -v "${comp_cmd%% *}" >/dev/null 2>&1; then + continue + fi + run_wrap_test $comp_cmd \ + fdtfactor_board1.test.dtb fdtfactor_board2.test.dtb fdtfactor_board3.tes= t.dtb + run_wrap_test $PYTHON "$FDTFACTOR" -v --verify \ + -b "fdtfactor_base.test.dtb.$ext" \ + "fdtfactor_board1.test.dtb.$ext" \ + "fdtfactor_board2.test.dtb.$ext" \ + "fdtfactor_board3.test.dtb.$ext" + cp "fdtfactor_base.test.dtb.$ext" "fdtfactor_base.${ext}_dec.test.dtb= .$ext" + cp "fdtfactor_board1.test.dtbo.$ext" "fdtfactor_board1.${ext}_dec.tes= t.dtbo.$ext" + run_wrap_test $decomp_cmd "fdtfactor_base.${ext}_dec.test.dtb.$ext" + run_wrap_test $decomp_cmd "fdtfactor_board1.${ext}_dec.test.dtbo.$ext" + run_wrap_test $FDTOVERLAY -i "fdtfactor_base.${ext}_dec.test.dtb" \ + -o "fdtfactor_board1.${ext}_recreated.test.dtb" \ + "fdtfactor_board1.${ext}_dec.test.dtbo" + run_wrap_test check_dtb_recreated \ + fdtfactor_board1.test.dtb "fdtfactor_board1.${ext}_recreated.test.dtb" + done fi } =20 @@ -1198,7 +1228,9 @@ if [ -z "$TESTSETS" ]; then fi =20 # Make sure we don't have stale blobs lying around -rm -f *.test.dtb *.test.dtbo *.test.dts +rm -f *.test.dtb *.test.dtbo *.test.dts *.test.dtb.gz *.test.dtbo.gz \ + *.test.dtb.xz *.test.dtbo.xz *.test.dtb.lzma *.test.dtbo.lzma \ + *.test.dtb.lz4 *.test.dtbo.lz4 =20 for set in $TESTSETS; do case $set in --=20 2.56.0.176.g3575e65682-goog From nobody Fri Oct 23 03:30:57 2026 Received: from mail-dl1-f45.google.com (mail-dl1-f45.google.com [74.125.82.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8EB2057F736 for ; Thu, 15 Oct 2026 18:23:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.82.45 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1792088602; cv=none; b=QWKgSp14+NUSS/HP46fU9fYOgUuqsDfNkbinX5vvcXfs7sYDpklJmIz2uZ7J1eugezgT/RKCsd+ymTOLeIMSB+rTPXzxf+/5MGEgswogpXzvpzuEZS8Ag/AZrwjo2S2Y9yTXWZoDy+WI8Sxhu3+/Lx4YCO/G/jD721urh1bfxKg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1792088602; c=relaxed/simple; bh=wHOu/4NhG/vx0QMW09TB3htvNNcuw0KISlmwjVzTnEI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=RlBITZSE/cf/zcn1QXrQ03cWdWr/xoP2VQ88ZpVgxcD13kRt1Z7DcN1RuxTEoNNTNLFu0C6stRMTzGTCKFNSBz1WIHXf8ETkHp4iySTakwzDTKveC46j0eX4bUkWL5NxBvz6j4vzyt957yYH1GeaB8yk/fbM3ldaTq34M9POVF8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=HaH1emtd; arc=none smtp.client-ip=74.125.82.45 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="HaH1emtd" Received: by mail-dl1-f45.google.com with SMTP id a92af1059eb24-1419d3416ceso4315549c88.0 for ; Thu, 15 Oct 2026 11:23:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1792088594; x=1792693394; darn=vger.kernel.org; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to:content-type; bh=f6V2cNjmTjZYPHFLnaysmAiB2NW1C0sPrfcyehbr8Gg=; b=HaH1emtdIF3TdEXfeV5bpNsEJVnCjzMwx3Jidg4JiBAR2XdUW2VtI8kBP9qSXAP0vF gbyddqCZn4bFcAaTlOsKYvaC3tYtq2itT8BqFTf3vDAxNHJJPcaGM4xV0CGBRXlDiVwY xZ41qK5usSLEEMkbG5ybYxUDGYnn3Zf256r80= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1792088594; x=1792693394; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=f6V2cNjmTjZYPHFLnaysmAiB2NW1C0sPrfcyehbr8Gg=; b=PpDikb2ACN06zy4/Weq0qidPELwycHOJZamSk0Ax7NRONb5RjrCf+eGgc7UpYy00w3 f/OYASmnMD9gRkT6vhgK3Cv5tlJcFKxC6CdTcNV9dyoST5sIIl7pVON0BZGby9ds4s/V O1EKq9qsVB3c6x0tpIVV4RAOu313TNXy9EPNogwz0m54vDJ7bRu/ic7raaAqTfcVfx5V yWzu35m5rk6rD+2GiPOwEPHEFP8SbrWfHtXB7ZDM1Ofhtvw1fPSIz9Usts4wDGJr2V8t v6zClrzE4HBjgGqMGVsw3IWoNaTNxh49p5owmKykVzNvncYS0JEbC6j4YpKNJySsFY6d 3B3Q== X-Forwarded-Encrypted: i=1; AKwUvBxcugGejvwCYdLbhITINbtxfTAS0lHw5LlKVFyu+7sIjlK004tdouh2CZ1U4fLHdjoA+4sI2RGSCHIMwvY=@vger.kernel.org X-Gm-Message-State: AFq9FYLVkoPIv/xU1QqU6ARwmO1qLYtMMFjC766B2n6FzxbFWI+bcU20 YkSiKQJwBuLR92VuTkwZR/JX5LzVG1IvpU65VKbT9oOF9eo8YdzJ5eEkIzCF54Nysw== X-Gm-Gg: AYBFou3B7t+ktP5SzzfV/v5ZbQkuYLWFPQaabkmq1FHdpZ7pMBCQkY1vcSL+X/PYnF0 IY0jqWH9eAaVab6xZ6kKFAWLL/joVp/lfvQxcYd6zoeLEVF+qXGcb+aSGW/FsJRQeZZh8umF9Ol SV9/xF0gnHn4r8CsGBPRFVrkpG7pZAPaBjpLjxyAAC9XKMaataR0m2JEtlNSzcyUlU8aDJFH+zy CxmbDKTFWbZdj6DPmyHs4AxHsuZNMQR/3dXR/4I68svfhShSObhzOIsAzsHBR06CEybyjyjy014 /uukqclT8wS8PgDPd3AbcGMfGVw702iP9OyI5Qbi9AR2VEIhSWPLZuR3c8YzXEntvRwhKK/buq8 t2y299quChqpYdiVCZB3j19xOMAhaYklXfGsx+28Z6WsZSc6wa8L0+C/PwstwW6yM2Hv/bXEmwt AqCtiMc8hhQWamu3tpS0M7KPxlz1j44qeTA6H+PH0ZqwQpJKGXrXpaFnKy5zoPQ1sAq6XPyp62t IPMkJ0DQPBnHH28vuyuLkuBHkFeOUd1RLuzqMobHaTeIoAKMGe9Cbfypn5h X-Received: by 2002:a05:701b:21c5:b0:149:7f69:a3b3 with SMTP id a92af1059eb24-16a586a30a1mr25486611c88.2.1792088593971; Thu, 15 Oct 2026 11:23:13 -0700 (PDT) Received: from dianders.sjc.corp.google.com ([2a00:79e0:2e7c:8:9ea0:3a29:323e:5f11]) by smtp.gmail.com with ESMTPSA id a92af1059eb24-182a8a4a56bsm727204c88.20.2026.10.15.11.23.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 15 Oct 2026 11:23:12 -0700 (PDT) From: Douglas Anderson To: Devicetree-compiler@vger.kernel.org, Rob Herring , Krzysztof Kozlowski , Conor Dooley , Konrad Dybcio Cc: Chen-Yu Tsai , linux-kernel@vger.kernel.org, Devicetree@vger.kernel.org, Brian Norris , Tudor Ambarus , =?UTF-8?q?Andr=C3=A9=20Draszik?= , Julius Werner , Saravana Kannan , Peter Griffin , Douglas Anderson Subject: [PATCH 4/4] fdtfactor: Add compiled FIT image factoring support Date: Thu, 15 Oct 2026 11:21:37 -0700 Message-ID: <20261015182221.2931742-5-dianders@chromium.org> X-Mailer: git-send-email 2.56.0.176.g3575e65682-goog In-Reply-To: <20261015182221.2931742-1-dianders@chromium.org> References: <20261015182221.2931742-1-dianders@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Bootloaders such as depthcharge on Chromebooks and U-Boot use Flattened Image Tree (FIT) images to bundle a kernel image with multiple board FDT blobs (`fdt-1`, `fdt-2`, ...) and per-board configuration nodes. Add `-f`/`--fit`, `-F`/`--fit-out`, and `--single-base-dtb` options to `fdtfactor` to factor a compiled FIT image directly: - Parse non-overlay `type =3D "flat_dt"` nodes under `/images`, decompressing their `data` blobs according to their `compression` property (`none`, `lz4`, `lzma`, `gzip`). - Partition the `flat_dt` blobs into compatible factor groups (by compression, architecture, FDT header fields, and node-path overlap) so a single FIT image containing multiple board/SoC families produces a separate `fdt-base-1`, `fdt-base-2`, etc. for each family. - Support `--single-base-dtb` to force all `flat_dt` blobs into a single base DTB regardless of node-path overlap, failing only if their FDT headers, architecture, or compression differ. - Insert each `fdt-base-N` node into `/images`, replace each factored `fdt-*` node's `data` with its compressed `.dtbo` overlay, recompute any `hash*` subnode digests, and prepend `fdt-base-N` to each matching configuration's `fdt` string list while preserving all other configuration properties (`description`, `compatible`, etc., populating `compatible` from the board DTB's root node when omitted) and any pre-existing overlay entries. Add uncompressed, LZ4-compressed multi-family, and `--single-base-dtb` FIT image test cases to the `fdtoverlay` test suite. Assisted-by: Gemini:gemini-pro-prerelease Signed-off-by: Douglas Anderson --- README.md | 15 + fdtfactor | 726 +++++++++++++++++++++++++++++++++++-- tests/Makefile.tests | 2 +- tests/fdtfactor_other1.dts | 69 ++++ tests/fdtfactor_other2.dts | 69 ++++ tests/fdtfactor_test.its | 109 ++++++ tests/run_tests.sh | 69 +++- 7 files changed, 1033 insertions(+), 26 deletions(-) create mode 100644 tests/fdtfactor_other1.dts create mode 100644 tests/fdtfactor_other2.dts create mode 100644 tests/fdtfactor_test.its diff --git a/README.md b/README.md index 6ace31bfbfdb..30fc28fa7545 100644 --- a/README.md +++ b/README.md @@ -110,6 +110,12 @@ Basic usage: $ fdtfactor -b base.dtb -o out_dir board1.dtb board2.dtb board3.dtb ``` =20 +Compiled FIT image usage (targeting depthcharge / coreboot / U-Boot): + +``` +$ fdtfactor -S -f image.fit -F image.factored.fit +``` + Options: =20 * `-b`, `--base-dtb`: Output path for the common base `.dtb` (compressed if @@ -118,6 +124,11 @@ Options: without a directory component). * `-o`, `--outdir`: Output directory for generated `.dtbo` files (and for `--base-dtb` when given without a directory component). +* `-f`, `--fit`: Input compiled FIT image (`.fit` / `.itb`) to factor in + place (or write to `-F`), creating one `fdt-base-N` per compatible + board/SoC family. +* `-F`, `--fit-out`: Output path for the factored FIT image (default: + overwrite `--fit`). * `-S`, `--strip-local-fixups`: Omit `/__local_fixups__` from the generated `base.dtb` and final reconstructed device trees. When `-L` is passed to `dtc` solely so `fdtfactor` can locate phandle cells at build time, the @@ -127,6 +138,10 @@ Options: `.dtbo` overlays. Omit `-S` if downstream tooling or verification expects applying a board's `.dtbo` to `base.dtb` to reproduce the original `.dtb`'s `/__local_fixups__` subtree verbatim. +* `--single-base-dtb`: In FIT mode (`-f`/`--fit`), force all `flat_dt` + images to be factored into a single `fdt-base-1` base `.dtb` regardless = of + node-path overlap, failing only if their FDT headers (`boot_cpuid_phys`, + `/memreserve/`), architecture, or compression differ. * `--verify`: Verify that applying each generated `.dtbo` to `base.dtb` with `libfdt` recreates the original `.dtb`. * `-v`, `--verbose`: Print uncompressed and compressed size savings. diff --git a/fdtfactor b/fdtfactor index 9f06801f973c..8a0c996581e5 100755 --- a/fdtfactor +++ b/fdtfactor @@ -2,21 +2,24 @@ # SPDX-License-Identifier: GPL-2.0-or-later """Split compiled device tree blobs (.dtb) into a base .dtb and .dtbo over= lays. =20 -Given a set of compiled device tree blobs, fdtfactor extracts all common n= odes -and properties into a single base .dtb file and writes the per-board -differences into .dtbo overlay files. Applying an overlay to the base .dtb -using fdtoverlay or libfdt's fdt_overlay_apply() recreates the correspondi= ng -original device tree. +Given a set of compiled device tree blobs (or a compiled FIT image), +fdtfactor extracts all common nodes and properties into a base .dtb and wr= ites +the per-board differences into .dtbo overlays. Applying an overlay to the = base +.dtb using fdtoverlay, libfdt's fdt_overlay_apply(), or depthcharge's FIT +loader recreates the corresponding original device tree. """ =20 import argparse from collections import Counter +import copy import gzip +import hashlib import lzma import os import struct import subprocess import sys +import zlib =20 import libfdt =20 @@ -27,8 +30,8 @@ LZ4_LEGACY_MAGIC =3D b'\x02\x21\x4c\x18' GZIP_MAGIC =3D b'\x1f\x8b' XZ_MAGIC =3D b'\xfd7zXZ\x00' # Matches raw LZMA_ALONE headers with default lc=3D3, lp=3D0, pb=3D2 (0x5d= ) and -# power-of-two dictionary sizes >=3D 64 KiB; .lzma extension handles any -# non-default LZMA headers. +# power-of-two dictionary sizes >=3D 64 KiB; .lzma extension or FIT compre= ssion +# property handles any non-default LZMA headers. LZMA_ALONE_MAGIC =3D b'\x5d\x00\x00' =20 COMP_TO_EXT =3D { @@ -209,6 +212,51 @@ class NodeTree: self.props =3D {} self.children =3D {} =20 + @classmethod + def from_fdt(cls, fdt, nodeoffset=3D0): + """Recursively copy a `libfdt.FdtRo` subtree into a `NodeTree`. + + Args: + fdt: A `libfdt.FdtRo` instance. + nodeoffset: Node offset in `fdt` to copy (defaults to root `0`= ). + + Returns: + A `NodeTree` containing all properties and subnodes of + `nodeoffset`. + """ + node =3D cls() + poff =3D fdt.first_property_offset(nodeoffset, quiet=3Dlibfdt.QUIE= T_NOTFOUND) + while poff >=3D 0: + prop =3D fdt.get_property_by_offset(poff) + node.props[prop.name] =3D bytes(prop) + poff =3D fdt.next_property_offset(poff, quiet=3Dlibfdt.QUIET_N= OTFOUND) + + soff =3D fdt.first_subnode(nodeoffset, quiet=3Dlibfdt.QUIET_NOTFOU= ND) + while soff >=3D 0: + name =3D fdt.get_name(soff) + node.children[name] =3D cls.from_fdt(fdt, soff) + soff =3D fdt.next_subnode(soff, quiet=3Dlibfdt.QUIET_NOTFOUND) + return node + + def copy(self): + """Return a deep copy of this `NodeTree`.""" + return copy.deepcopy(self) + + def update_fit_hashes(self): + """Recompute `value` properties on any FIT `hash*` subnodes of `se= lf`.""" + data_bytes =3D self.props.get('data') + if data_bytes is None: + return + for sub_name, sub_node in self.children.items(): + if not sub_name.startswith('hash'): + continue + algo =3D _prop_as_str(sub_node.props.get('algo')) + if not algo: + raise ValueError( + f'FIT hash subnode {sub_name!r} lacks an algo property' + ) + sub_node.props['value'] =3D _compute_fit_hash(algo, data_bytes) + def get_node(self, path): """Return the descendant NodeTree at `path`, creating missing node= s. =20 @@ -337,7 +385,7 @@ class ParsedDtb: """Parsed representation of an input .dtb blob.""" =20 def __init__(self, filename, raw_bytes, compressed_size=3DNone, - compression=3D'none'): + compression=3D'none', fdt=3DNone): """Parse an uncompressed FDT blob into nodes, symbols, and fixups. =20 Args: @@ -347,6 +395,7 @@ class ParsedDtb: decompression (defaults to `len(raw_bytes)`). compression: Compression algorithm of the input file (`'none'`, `'lz4'`, `'lzma'`, `'xz'`, or `'gzip'`). + fdt: Optional pre-validated `libfdt.FdtRo` instance for `raw_b= ytes`. """ self.filename =3D filename self.raw_bytes =3D raw_bytes @@ -354,11 +403,12 @@ class ParsedDtb: compressed_size if compressed_size is not None else len(raw_by= tes) ) self.compression =3D compression - fdt =3D _open_fdt_ro(raw_bytes, filename) - if _is_dtbo_overlay(fdt): - raise ValueError( - f'{filename}: Input is already a DTB overlay (.dtbo)' - ) + if fdt is None: + fdt =3D _open_fdt_ro(raw_bytes, filename) + if _is_dtbo_overlay(fdt): + raise ValueError( + f'{filename}: Input is already a DTB overlay (.dtbo)' + ) =20 # Extract FDT header metadata (boot_cpuid_phys and /memreserve/ ta= ble) # stored outside the DT node/property tree; see _validate_dtb_head= ers() @@ -1211,6 +1261,588 @@ def verify_recreated_dtb(orig_dtb, base_dtb_blob, d= tbo_blob, ) =20 =20 +# ------------------------------------------------------------------------= --- +# Compiled FIT image factoring (targeting depthcharge / coreboot / U-Boot) +# ------------------------------------------------------------------------= --- + +def _prop_as_str(raw_bytes): + """Decode a NUL-terminated FDT string property. + + Args: + raw_bytes: Raw property bytes (or None). + + Returns: + Decoded string with trailing NULs stripped, or `''` if empty/None. + """ + if not raw_bytes: + return '' + return raw_bytes.rstrip(b'\x00').decode('utf-8', errors=3D'replace') + + +def _prop_as_str_list(raw_bytes): + """Decode a NUL-separated FDT stringlist property. + + Args: + raw_bytes: Raw property bytes (or None). + + Returns: + List of decoded non-empty strings. + """ + if not raw_bytes: + return [] + parts =3D raw_bytes.rstrip(b'\x00').split(b'\x00') + return [p.decode('utf-8', errors=3D'replace') for p in parts if p] + + +def _compute_fit_hash(algo_name, data_bytes): + """Compute a binary hash digest for a FIT image `hash*` subnode. + + Args: + algo_name: Hash algorithm name (e.g. `'sha1'`, `'sha256'`, `'crc32= '`). + data_bytes: Image `data` property bytes to hash. + + Returns: + Binary digest as `bytes`. + """ + if algo_name =3D=3D 'crc32': + return struct.pack('>I', zlib.crc32(data_bytes) & 0xffffffff) + h =3D hashlib.new(algo_name) + h.update(data_bytes) + return h.digest() + + +def _is_dtb_filename_desc(desc): + """Return True if `desc` looks like a `.dtb` filename.""" + if not desc or ' ' in desc: + return False + _, ext =3D os.path.splitext(desc) + if ext in EXT_TO_COMP: + desc =3D desc[:-len(ext)] + return desc.endswith('.dtb') and len(os.path.basename(desc)) > 4 + + +def _derive_base_dtb_description(entries, base_dtb_node_name, comp): + """Derive a `description` string for a factored base FDT image node. + + Extracts the common prefix across `.dtb` filename descriptions in `ent= ries` + and trims any partially matched trailing token before appending + `'-base.dtb[.comp]'` (falling back to `'.dtb[.comp= ]'` + if no common prefix remains): + + * Diverging at a letter (truncate back to the last `-`, `_`, or `.`): + - `['sc7180-trogdor-lazor', 'sc7180-trogdor-pompom']` + -> `'sc7180-trogdor-base.dtb'` + - `['sc7180-lazor', 'sc7180-limozeen']` -> `'sc7180-base.dtb'` + - `['alpha', 'apricot']` -> `'fdt-base-1.dtb'` (no prior separator) + * Diverging at a digit inside a token (strip trailing digits from pref= ix): + - `['fdtfactor_board1', 'fdtfactor_board2']` -> `'fdtfactor_board-ba= se.dtb'` + - `['sc7180-trogdor', 'sc7280-herobrine']` -> `'sc-base.dtb'` + * Diverging at or right after a separator (keep full tokens before + separator): + - `['sc7180-1', 'sc7180-2']` -> `'sc7180-base.dtb'` + - `['sc7180-trogdor-r1', 'sc7180-trogdor-r1-lte']` + -> `'sc7180-trogdor-r1-base.dtb'` + + Args: + entries: List of FIT `flat_dt` entry dicts in the factor group. + base_dtb_node_name: Node name for the base FDT (e.g. `'fdt-base-1'= `). + comp: Compression algorithm name (`'none'`, `'lz4'`, etc.). + + Returns: + Description string (e.g. `'sc7180-trogdor-base.dtb.lz4'`). + """ + stems =3D [] + for e in entries: + desc =3D _prop_as_str(e['node'].props.get('description')) + if _is_dtb_filename_desc(desc): + _, stem, _ =3D split_dtb_filename(desc) + if stem: + stems.append(stem) + if len(stems) >=3D 2: + prefix =3D os.path.commonprefix(stems) + if any( + len(s) > len(prefix) and s[len(prefix)].isalpha() + for s in stems + ): + sep_idx =3D max(prefix.rfind(c) for c in ('-', '_', '.')) + prefix =3D prefix[:sep_idx] if sep_idx >=3D 0 else '' + elif ( + any( + len(s) > len(prefix) and s[len(prefix)].isdigit() + for s in stems + ) + and not prefix.endswith(('-', '_', '.')) + ): + prefix =3D prefix.rstrip('0123456789') + prefix =3D prefix.rstrip('-_.') + if prefix: + return f'{prefix}-base.dtb{COMP_TO_EXT[comp]}' + return f'{base_dtb_node_name}.dtb{COMP_TO_EXT[comp]}' + + +def group_fit_dtbs(fdt_entries): + """Partition FIT `flat_dt` entries into compatible factor groups. + + DTBs are grouped by `(compression, arch, load, entry, boot_cpuid_phys, + mem_rsv)` and node-path overlap (>=3D 50%) so that a single FIT image + containing multiple board/SoC families (e.g. `sc7180-trogdor` and + `sc7280-herobrine`) produces one base DTB per family. + + Args: + fdt_entries: List of FIT `flat_dt` entry dicts. + + Returns: + List of entry lists, one per compatible factor group. + """ + groups =3D [] + for entry in fdt_entries: + dtb =3D entry['dtb'] + header_key =3D ( + entry['compression'], + entry['arch'], + entry['node'].props.get('load'), + entry['node'].props.get('entry'), + dtb.boot_cpuid_phys, + tuple(dtb.mem_rsv), + ) + entry_nodes =3D set(dtb.nodes.keys()) + + placed =3D False + for grp in groups: + if grp['key'] !=3D header_key: + continue + ref_nodes =3D grp['ref_nodes'] + union_len =3D len(entry_nodes | ref_nodes) + overlap =3D ( + len(entry_nodes & ref_nodes) / union_len if union_len else= 1.0 + ) + if overlap >=3D 0.5: + grp['entries'].append(entry) + placed =3D True + break + + if not placed: + groups.append({ + 'key': header_key, + 'ref_nodes': entry_nodes, + 'entries': [entry], + }) + + return [grp['entries'] for grp in groups] + + +def _extract_fit_fdt_entries(fit_in_path, images_node): + """Extract and parse non-overlay `flat_dt` entries from `/images`. + + Any `/images` nodes that are already `.dtbo` overlays are skipped and = left + intact in `/images`. + + Args: + fit_in_path: Input FIT file path for error messages. + images_node: The `NodeTree` for `/images`. + + Returns: + List of parsed `flat_dt` entry dicts. + """ + if any(name.startswith('fdt-base-') for name in images_node.children): + raise ValueError( + f'{fit_in_path}: FIT image already contains fdt-base-* node(s)' + ) + + fdt_entries =3D [] + for img_name, img_node in images_node.children.items(): + # Check all /images/* nodes (including kernel/ramdisk) because + # re-serializing fit_root would drop external data appended past + # the FDT totalsize. + if ('data-offset' in img_node.props or + 'data-position' in img_node.props): + raise ValueError( + f'{fit_in_path}: /images/{img_name} uses external data ' + f'(data-offset/data-position), which is not supported' + ) + if _prop_as_str(img_node.props.get('type')) !=3D 'flat_dt': + continue + raw_data =3D img_node.props.get('data') + if raw_data is None: + raise ValueError( + f'{fit_in_path}: /images/{img_name} lacks a data property' + ) + node_label =3D f'{fit_in_path}:/images/{img_name}' + comp_prop =3D _prop_as_str(img_node.props.get('compression')) or '= none' + dtb_bytes, comp =3D decompress_blob( + raw_data, filename=3Dnode_label, comp=3Dcomp_prop + ) + fdt_ro =3D _open_fdt_ro(dtb_bytes, node_label) + if _is_dtbo_overlay(fdt_ro): + continue + + arch_str =3D _prop_as_str(img_node.props.get('arch')) + parsed_dtb =3D ParsedDtb( + node_label, + dtb_bytes, + compressed_size=3Dlen(raw_data), + compression=3Dcomp, + fdt=3Dfdt_ro, + ) + fdt_entries.append({ + 'name': img_name, + 'node': img_node, + 'dtb': parsed_dtb, + 'compression': comp, + 'arch': arch_str, + }) + + if len(fdt_entries) < 2: + raise ValueError( + 'FIT image must contain at least two non-overlay flat_dt ' + 'nodes to factor' + ) + + return fdt_entries + + +def _partition_fit_fdt_entries(fdt_entries, single_base_dtb): + """Partition `fdt_entries` into factorable groups and validate constra= ints. + + Args: + fdt_entries: List of non-overlay FIT `flat_dt` entry dicts. + single_base_dtb: Whether `--single-base-dtb` was requested. + + Returns: + List of entry lists to factor (each with `len >=3D 2`). + """ + if single_base_dtb: + ref_entry =3D fdt_entries[0] + for entry in fdt_entries[1:]: + if entry['compression'] !=3D ref_entry['compression']: + raise ValueError( + f'--single-base-dtb requires all FIT flat_dt nodes to = use ' + f'the same compression: {ref_entry["name"]} has ' + f'{ref_entry["compression"]!r} vs {entry["name"]} ' + f'{entry["compression"]!r}' + ) + if entry['arch'] !=3D ref_entry['arch']: + raise ValueError( + f'--single-base-dtb requires all FIT flat_dt nodes to = use ' + f'the same arch: {ref_entry["name"]} has ' + f'{ref_entry["arch"]!r} vs {entry["name"]} ' + f'{entry["arch"]!r}' + ) + for prop in ('load', 'entry'): + if (entry['node'].props.get(prop) !=3D + ref_entry['node'].props.get(prop)): + raise ValueError( + f'--single-base-dtb requires all FIT flat_dt nodes= to ' + f'use the same {prop} property: {ref_entry["name"]= } vs ' + f'{entry["name"]}' + ) + _validate_dtb_headers([e['dtb'] for e in fdt_entries]) + return [fdt_entries] + + groups =3D group_fit_dtbs(fdt_entries) + factorable_groups =3D [g for g in groups if len(g) >=3D 2] + if not factorable_groups: + raise ValueError( + 'FIT image must contain at least two compatible ' + 'flat_dt nodes to factor' + ) + for g in groups: + if len(g) =3D=3D 1: + print( + f'Note: FIT flat_dt node {g[0]["name"]} has no ' + f'compatible peers and will be left unfactored', + file=3Dsys.stderr, + ) + return factorable_groups + + +def _factor_fit_group(entries, base_dtb_node_name, strip_local_fixups, ver= ify): + """Factor one compatible group of FIT `flat_dt` entries. + + Updates each `entry['node']` in place with its compressed `.dtbo` over= lay + and returns the newly created `base_dtb_node` along with group size + statistics. + + Args: + entries: List of FIT `flat_dt` entry dicts in the group. + base_dtb_node_name: Name for the group's base DTB node (e.g. + `'fdt-base-1'`). + strip_local_fixups: Whether to strip `/__local_fixups__`. + verify: Whether to verify each reconstructed DTB. + + Returns: + Tuple of `(base_dtb_node, group_stat)`. + """ + group_dtbs =3D [e['dtb'] for e in entries] + comp =3D entries[0]['compression'] + + base_dtb_blob, dtbo_blobs =3D create_base_dtb_and_overlays( + group_dtbs, + strip_local_fixups=3Dstrip_local_fixups, + ) + + if verify: + for d, dtbo_blob in zip(group_dtbs, dtbo_blobs): + verify_recreated_dtb( + d, base_dtb_blob, dtbo_blob, strip_local_fixups + ) + + base_dtb_comp_bytes =3D compress_blob(base_dtb_blob, comp) + base_dtb_node =3D entries[0]['node'].copy() + for prop in ('phandle', 'linux,phandle', 'compatible'): + base_dtb_node.props.pop(prop, None) + base_dtb_node.children =3D { + k: v + for k, v in base_dtb_node.children.items() + if k.startswith('hash') + } + for sub_node in base_dtb_node.children.values(): + sub_node.props.pop('phandle', None) + sub_node.props.pop('linux,phandle', None) + if 'description' in base_dtb_node.props: + base_dtb_desc =3D _derive_base_dtb_description( + entries, base_dtb_node_name, comp + ) + base_dtb_node.props['description'] =3D ( + base_dtb_desc.encode('utf-8') + b'\x00' + ) + base_dtb_node.props['data'] =3D base_dtb_comp_bytes + if 'compression' in base_dtb_node.props or comp !=3D 'none': + base_dtb_node.props['compression'] =3D comp.encode('utf-8') + b'\x= 00' + base_dtb_node.update_fit_hashes() + + dtbo_comp_sizes =3D [] + for entry, dtbo_blob in zip(entries, dtbo_blobs): + img_node =3D entry['node'] + dtbo_comp_bytes =3D compress_blob(dtbo_blob, comp) + dtbo_comp_sizes.append(len(dtbo_comp_bytes)) + + if 'description' in img_node.props: + old_desc =3D _prop_as_str(img_node.props['description']) + if _is_dtb_filename_desc(old_desc): + desc_dir, desc_stem, _ =3D split_dtb_filename(old_desc) + new_desc_name =3D f'{desc_stem}.dtbo{COMP_TO_EXT[comp]}' + new_desc =3D ( + os.path.join(desc_dir, new_desc_name) + if desc_dir + else new_desc_name + ) + img_node.props['description'] =3D ( + new_desc.encode('utf-8') + b'\x00' + ) + + for prop in ('load', 'entry'): + img_node.props.pop(prop, None) + for sub_name in list(img_node.children): + if sub_name.startswith('signature'): + print( + f'Warning: Stripping invalidated signature subnode ' + f'/images/{entry["name"]}/{sub_name}', + file=3Dsys.stderr, + ) + del img_node.children[sub_name] + + img_node.props['data'] =3D dtbo_comp_bytes + img_node.update_fit_hashes() + + group_stat =3D { + 'base_dtb_name': base_dtb_node_name, + 'comp': comp, + 'dtbs': group_dtbs, + 'base_dtb_blob': base_dtb_blob, + 'dtbo_blobs': dtbo_blobs, + 'base_dtb_comp_size': len(base_dtb_comp_bytes), + 'dtbo_comp_sizes': dtbo_comp_sizes, + } + return base_dtb_node, group_stat + + +def _update_fit_images_and_configs(images_node, configs_node, + insert_before_first, fdt_to_base_dtb): + """Insert `fdt-base-*` nodes into `/images` and update `/configuration= s`. + + Args: + images_node: The `NodeTree` for `/images`. + configs_node: The `NodeTree` for `/configurations`. + insert_before_first: Dict mapping the first `flat_dt` node name of= each + group to `(base_dtb_node_name, base_dtb_node)`. + fdt_to_base_dtb: Dict mapping each factored `flat_dt` node name to + `(base_dtb_node_name, orig_dtb)`. + """ + new_images_children =3D {} + for img_name, img_node in images_node.children.items(): + if img_name in insert_before_first: + b_name, b_node =3D insert_before_first[img_name] + new_images_children[b_name] =3D b_node + new_images_children[img_name] =3D img_node + images_node.children =3D new_images_children + + for conf_name, conf_node in configs_node.children.items(): + fdt_list =3D _prop_as_str_list(conf_node.props.get('fdt')) + if not fdt_list: + continue + primary_fdt =3D fdt_list[0] + for extra_fdt in fdt_list[1:]: + if extra_fdt in fdt_to_base_dtb: + raise ValueError( + f'/configurations/{conf_name} lists factored flat_dt ' + f'{extra_fdt!r} after the primary FDT' + ) + if primary_fdt in fdt_to_base_dtb: + base_dtb_name, orig_dtb =3D fdt_to_base_dtb[primary_fdt] + new_fdt_list =3D [base_dtb_name] + fdt_list + conf_node.props['fdt'] =3D ( + '\x00'.join(new_fdt_list).encode('utf-8') + b'\x00' + ) + # If the configuration omitted 'compatible' (relying on the + # bootloader reading root 'compatible' from fdt_list[0]), copy= the + # board's root 'compatible' property onto the configuration no= de + # now that fdt_list[0] is the shared base DTB. + if 'compatible' not in conf_node.props: + root_compat =3D orig_dtb.nodes.get('/', {}).get('compatibl= e') + if root_compat is not None: + conf_node.props['compatible'] =3D root_compat + for sub_name in list(conf_node.children): + if sub_name.startswith(('signature', 'hash')): + print( + f'Warning: Stripping invalidated {sub_name} subnod= e ' + f'/configurations/{conf_name}/{sub_name}', + file=3Dsys.stderr, + ) + del conf_node.children[sub_name] + + +def _print_fit_stats(group_stats, orig_fit_size, new_fit_size): + """Print per-group and total FIT image size savings. + + Args: + group_stats: List of group statistics dicts from `_factor_fit_grou= p()`. + orig_fit_size: Original FIT image size in bytes. + new_fit_size: Factored FIT image size in bytes. + """ + for st in group_stats: + g_dtbs =3D st['dtbs'] + orig_uncomp =3D sum(len(d.raw_bytes) for d in g_dtbs) + base_dtb_uncomp =3D len(st['base_dtb_blob']) + dtbo_uncomp =3D sum(len(b) for b in st['dtbo_blobs']) + new_uncomp =3D base_dtb_uncomp + dtbo_uncomp + saved_uncomp =3D orig_uncomp - new_uncomp + pct_uncomp =3D ( + (saved_uncomp / orig_uncomp * 100.0) if orig_uncomp else 0.0 + ) + print( + f'[{st["base_dtb_name"]}] Uncompressed DTBs: {len(g_dtbs)} ' + f'files ({orig_uncomp:,} B) -> base {base_dtb_uncomp:,} B + ' + f'{len(st["dtbo_blobs"])} overlays {dtbo_uncomp:,} B =3D ' + f'{new_uncomp:,} B (saved {saved_uncomp:,} B / ' + f'{pct_uncomp:.1f}%)' + ) + if st['comp'] !=3D 'none': + orig_comp =3D sum(d.compressed_size for d in g_dtbs) + base_dtb_csize =3D st['base_dtb_comp_size'] + dtbo_csize =3D sum(st['dtbo_comp_sizes']) + new_comp =3D base_dtb_csize + dtbo_csize + saved_comp =3D orig_comp - new_comp + pct_comp =3D ( + (saved_comp / orig_comp * 100.0) if orig_comp else 0.0 + ) + print( + f'[{st["base_dtb_name"]}] Compressed ({st["comp"]}): ' + f'{len(g_dtbs)} files ({orig_comp:,} B) -> base ' + f'{base_dtb_csize:,} B + {len(st["dtbo_blobs"])} overlays ' + f'{dtbo_csize:,} B =3D {new_comp:,} B ' + f'(saved {saved_comp:,} B / {pct_comp:.1f}%)' + ) + fit_saved =3D orig_fit_size - new_fit_size + fit_pct =3D (fit_saved / orig_fit_size * 100.0) if orig_fit_size else = 0.0 + print( + f'FIT image total: {orig_fit_size:,} bytes -> {new_fit_size:,} byt= es ' + f'(saved {fit_saved:,} B / {fit_pct:.1f}%)' + ) + + +def factor_fit_image(fit_in_path, fit_out_path, strip_local_fixups=3DFalse, + single_base_dtb=3DFalse, verify=3DFalse, verbose=3DFa= lse): + """Factor all `flat_dt` images in a compiled FIT image into base + ove= rlays. + + Args: + fit_in_path: Input `.fit` / `.itb` file path. + fit_out_path: Output `.fit` / `.itb` file path. + strip_local_fixups: Whether to strip `/__local_fixups__` from fact= ored + DTBs. + single_base_dtb: If True, force all `flat_dt` nodes in the FIT ima= ge to + be factored into a single `fdt-base-1` base DTB, failing only = if + their FDT headers, architecture, or compression differ. + verify: Whether to verify each board's reconstructed DTB using + `fdt_overlay_apply()`. + verbose: Whether to print size savings statistics. + + Raises: + ValueError: If the FIT image lacks `/images` or `/configurations`, + a `flat_dt` node lacks a `data` property, or `single_base_dtb`= is + requested and the `flat_dt` nodes have incompatible headers, + architecture, or compression. + """ + with open(fit_in_path, 'rb') as f: + orig_fit_bytes =3D f.read() + + orig_fit_fdt =3D _open_fdt_ro(orig_fit_bytes, fit_in_path) + fit_root =3D NodeTree.from_fdt(orig_fit_fdt) + + images_node =3D fit_root.children.get('images') + configs_node =3D fit_root.children.get('configurations') + if images_node is None or configs_node is None: + raise ValueError( + f'{fit_in_path}: FIT image must contain /images and ' + f'/configurations nodes' + ) + + fdt_entries =3D _extract_fit_fdt_entries(fit_in_path, images_node) + factorable_groups =3D _partition_fit_fdt_entries( + fdt_entries, single_base_dtb + ) + + fdt_to_base_dtb =3D {} + insert_before_first =3D {} + group_stats =3D [] + + for idx, entries in enumerate(factorable_groups, 1): + base_dtb_node_name =3D f'fdt-base-{idx}' + + base_dtb_node, group_stat =3D _factor_fit_group( + entries, base_dtb_node_name, strip_local_fixups, verify + ) + insert_before_first[entries[0]['name']] =3D ( + base_dtb_node_name, base_dtb_node + ) + for entry in entries: + fdt_to_base_dtb[entry['name']] =3D (base_dtb_node_name, entry[= 'dtb']) + group_stats.append(group_stat) + + _update_fit_images_and_configs( + images_node, configs_node, insert_before_first, fdt_to_base_dtb + ) + + fit_mem_rsv =3D [ + tuple(orig_fit_fdt.get_mem_rsv(i)) + for i in range(orig_fit_fdt.num_mem_rsv()) + ] + new_fit_bytes =3D fit_root.to_fdt_blob( + mem_rsv=3Dfit_mem_rsv, + boot_cpuid_phys=3Dorig_fit_fdt.boot_cpuid_phys(), + size_hint=3Dmax(len(orig_fit_bytes) + 65536, 65536), + ) + + out_dir =3D os.path.dirname(fit_out_path) + if out_dir: + os.makedirs(out_dir, exist_ok=3DTrue) + with open(fit_out_path, 'wb') as f: + f.write(new_fit_bytes) + + if verbose: + _print_fit_stats(group_stats, len(orig_fit_bytes), len(new_fit_byt= es)) + + def get_dtbo_output_path(dtb_path, outdir, comp=3D'none'): """Determine the output `.dtbo[.comp]` file path for an input `.dtb` p= ath. =20 @@ -1238,13 +1870,13 @@ def parse_args(): parser =3D argparse.ArgumentParser( description=3D( 'Split a set of compiled device tree blobs (.dtb, or compresse= d ' - '.dtb.lz4/.xz/.lzma/.gz) into a single common base .dtb and ' - 'per-board overlay blobs (.dtbo).' + '.dtb.lz4/.xz/.lzma/.gz) or a compiled FIT image into common ' + 'base .dtb(s) and per-board overlay blobs (.dtbo).' ) ) parser.add_argument( '-b', '--base-dtb', - required=3DTrue, + default=3DNone, metavar=3D'BASE_DTB', help=3D( 'Output common base .dtb (or .dtb.lz4/.xz/.lzma/.gz) file path= ' @@ -1260,6 +1892,23 @@ def parse_args(): 'filenames (default: alongside each input .dtb)' ), ) + parser.add_argument( + '-f', '--fit', + default=3DNone, + metavar=3D'FIT_IMAGE', + help=3D( + 'Input compiled FIT image (.fit / .itb) to factor (updated ' + 'in-place unless -F/--fit-out is specified)' + ), + ) + parser.add_argument( + '-F', '--fit-out', + default=3DNone, + metavar=3D'FIT_OUT', + help=3D( + 'Output path for the factored FIT image (default: overwrite --= fit)' + ), + ) parser.add_argument( '-S', '--strip-local-fixups', action=3D'store_true', @@ -1268,6 +1917,15 @@ def parse_args(): 'reconstructed device trees' ), ) + parser.add_argument( + '--single-base-dtb', + action=3D'store_true', + help=3D( + 'Force all flat_dt nodes in a FIT image (-f/--fit) to be ' + 'factored into a single fdt-base-1 base DTB, failing only if ' + 'headers, arch, or compression differ' + ), + ) parser.add_argument( '--verify', action=3D'store_true', @@ -1283,16 +1941,29 @@ def parse_args(): ) parser.add_argument( 'dtbs', - nargs=3D'+', + nargs=3D'*', metavar=3D'DTB', help=3D( - 'Input .dtb (or compressed .dtb.lz4/.xz/.lzma/.gz) files (at ' - 'least two required)' + 'Input .dtb (or compressed .dtb.lz4/.xz/.lzma/.gz) files (when= ' + '--fit is not used)' ), ) args =3D parser.parse_args() - if len(args.dtbs) < 2: - parser.error('at least two input .dtb files are required') + if args.fit_out and not args.fit: + parser.error('-F/--fit-out requires -f/--fit') + if args.single_base_dtb and not args.fit: + parser.error('--single-base-dtb requires -f/--fit') + if args.fit: + if args.dtbs or args.base_dtb or args.outdir: + parser.error( + '-f/--fit cannot be combined with -b/--base-dtb, ' + '-o/--outdir, or positional DTB arguments' + ) + else: + if not args.base_dtb: + parser.error('-b/--base-dtb is required when -f/--fit is not u= sed') + if len(args.dtbs) < 2: + parser.error('at least two input .dtb files are required') return args =20 =20 @@ -1417,7 +2088,18 @@ def main(): """ args =3D parse_args() try: - factor_dtb_files(args) + if args.fit: + fit_out =3D args.fit_out if args.fit_out else args.fit + factor_fit_image( + args.fit, + fit_out, + strip_local_fixups=3Dargs.strip_local_fixups, + single_base_dtb=3Dargs.single_base_dtb, + verify=3Dargs.verify, + verbose=3Dargs.verbose, + ) + else: + factor_dtb_files(args) except (OSError, ValueError, RuntimeError, libfdt.FdtException) as exc: sys.exit(f'Error: {exc}') return 0 diff --git a/tests/Makefile.tests b/tests/Makefile.tests index d0bdc4afefb2..0ca2d70457f9 100644 --- a/tests/Makefile.tests +++ b/tests/Makefile.tests @@ -53,7 +53,7 @@ TESTS_DEPFILES =3D $(TESTS:%=3D%.d) \ $(addprefix $(TESTS_PREFIX),testutils.d) =20 TESTS_CLEANFILES_L =3D $(STD_CLEANFILES) \ - *.dtb *.test.dtbo *.test.dts *.test.dt.yaml *.dtsv1 \ + *.dtb *.test.dtbo *.test.dts *.test.its *.test.fit *.test.dt.yaml *.dtsv1= \ *.test.dtb.gz *.test.dtbo.gz *.test.dtb.xz *.test.dtbo.xz \ *.test.dtb.lzma *.test.dtbo.lzma *.test.dtb.lz4 *.test.dtbo.lz4 \ tmp.* *.bak \ diff --git a/tests/fdtfactor_other1.dts b/tests/fdtfactor_other1.dts new file mode 100644 index 000000000000..26fca3816d5a --- /dev/null +++ b/tests/fdtfactor_other1.dts @@ -0,0 +1,69 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/dts-v1/; + +/ { + model =3D "Other Board 1"; + compatible =3D "vendor,other1", "vendor,other"; + #address-cells =3D <1>; + #size-cells =3D <1>; + + intc: interrupt-controller@8000 { + compatible =3D "vendor,other-intc"; + reg =3D <0x8000 0x100>; + interrupt-controller; + #interrupt-cells =3D <2>; + }; + + clk: clock-controller@9000 { + compatible =3D "vendor,other-clk"; + reg =3D <0x9000 0x100>; + #clock-cells =3D <1>; + }; + + soc@10000000 { + compatible =3D "simple-bus"; + #address-cells =3D <1>; + #size-cells =3D <1>; + ranges; + interrupt-parent =3D <&intc>; + + uart0: serial@a000 { + compatible =3D "vendor,other-uart"; + reg =3D <0xa000 0x100>; + interrupts =3D <1 0>; + clocks =3D <&clk 1>; + status =3D "okay"; + }; + + i2c0: i2c@b000 { + compatible =3D "vendor,other-i2c"; + reg =3D <0xb000 0x100>; + interrupts =3D <2 0>; + clocks =3D <&clk 2>; + status =3D "okay"; + + pmic: pmic@50 { + compatible =3D "vendor,other-pmic"; + reg =3D <0x50>; + + reg_buck1: buck1 { + regulator-min-microvolt =3D <1800000>; + regulator-max-microvolt =3D <1800000>; + }; + }; + }; + + spi0: spi@c000 { + compatible =3D "vendor,other-spi"; + reg =3D <0xc000 0x100>; + clocks =3D <&clk 3>; + vdd-supply =3D <®_buck1>; + cs-supply =3D <&other1_reg>; + }; + }; + + other1_reg: regulator-other1 { + compatible =3D "regulator-fixed"; + regulator-name =3D "other1-cs"; + }; +}; diff --git a/tests/fdtfactor_other2.dts b/tests/fdtfactor_other2.dts new file mode 100644 index 000000000000..50cbc73759d2 --- /dev/null +++ b/tests/fdtfactor_other2.dts @@ -0,0 +1,69 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/dts-v1/; + +/ { + model =3D "Other Board 2"; + compatible =3D "vendor,other2", "vendor,other"; + #address-cells =3D <1>; + #size-cells =3D <1>; + + other2_early_reg: regulator-other-early { + compatible =3D "regulator-fixed"; + regulator-name =3D "other2-early"; + }; + + intc: interrupt-controller@8000 { + compatible =3D "vendor,other-intc"; + reg =3D <0x8000 0x100>; + interrupt-controller; + #interrupt-cells =3D <2>; + }; + + clk: clock-controller@9000 { + compatible =3D "vendor,other-clk"; + reg =3D <0x9000 0x100>; + #clock-cells =3D <1>; + }; + + soc@10000000 { + compatible =3D "simple-bus"; + #address-cells =3D <1>; + #size-cells =3D <1>; + ranges; + interrupt-parent =3D <&intc>; + + uart0: serial@a000 { + compatible =3D "vendor,other-uart"; + reg =3D <0xa000 0x100>; + interrupts =3D <1 0>; + clocks =3D <&clk 1>; + status =3D "disabled"; + }; + + i2c0: i2c@b000 { + compatible =3D "vendor,other-i2c"; + reg =3D <0xb000 0x100>; + interrupts =3D <2 0>; + clocks =3D <&clk 2>; + status =3D "okay"; + + pmic: pmic@50 { + compatible =3D "vendor,other-pmic"; + reg =3D <0x50>; + + reg_buck1: buck1 { + regulator-min-microvolt =3D <1800000>; + regulator-max-microvolt =3D <3300000>; + }; + }; + }; + + mmc0: mmc@d000 { + compatible =3D "vendor,other-mmc"; + reg =3D <0xd000 0x100>; + clocks =3D <&clk 4>; + vmmc-supply =3D <®_buck1>; + vqmmc-supply =3D <&other2_early_reg>; + }; + }; +}; diff --git a/tests/fdtfactor_test.its b/tests/fdtfactor_test.its new file mode 100644 index 000000000000..bf3b9ef8c439 --- /dev/null +++ b/tests/fdtfactor_test.its @@ -0,0 +1,109 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/dts-v1/; + +/ { + description =3D "Test FIT image for fdtfactor"; + #address-cells =3D <1>; + + images { + kernel-1 { + description =3D "Unavailable"; + data =3D /incbin/("fdtfactor_board1.test.dtb"); + type =3D "kernel_noload"; + arch =3D "arm64"; + os =3D "linux"; + compression =3D "none"; + load =3D <0>; + entry =3D <0>; + }; + fdt-1 { + description =3D "fdtfactor_board1.test.dtb"; + data =3D /incbin/("fdtfactor_board1.test.dtb"); + type =3D "flat_dt"; + arch =3D "arm64"; + compression =3D "none"; + hash-1 { + algo =3D "sha1"; + }; + }; + fdt-2 { + description =3D "fdtfactor_board2.test.dtb"; + data =3D /incbin/("fdtfactor_board2.test.dtb"); + type =3D "flat_dt"; + arch =3D "arm64"; + compression =3D "none"; + hash-1 { + algo =3D "sha1"; + }; + }; + fdt-3 { + description =3D "fdtfactor_board3.test.dtb"; + data =3D /incbin/("fdtfactor_board3.test.dtb"); + type =3D "flat_dt"; + arch =3D "arm64"; + compression =3D "none"; + hash-1 { + algo =3D "sha1"; + }; + }; + fdt-4 { + description =3D "fdtfactor_other1.test.dtb"; + data =3D /incbin/("fdtfactor_other1.test.dtb"); + type =3D "flat_dt"; + arch =3D "arm64"; + compression =3D "none"; + hash-1 { + algo =3D "sha1"; + }; + }; + fdt-5 { + description =3D "fdtfactor_other2.test.dtb"; + data =3D /incbin/("fdtfactor_other2.test.dtb"); + type =3D "flat_dt"; + arch =3D "arm64"; + compression =3D "none"; + hash-1 { + algo =3D "sha1"; + }; + }; + fdt-ovl-1 { + description =3D "overlay_overlay_phandle.test.dtb"; + data =3D /incbin/("overlay_overlay_phandle.test.dtb"); + type =3D "flat_dt"; + arch =3D "arm64"; + compression =3D "none"; + }; + }; + configurations { + default =3D "conf-1"; + conf-1 { + description =3D "board-meta-1"; + kernel =3D "kernel-1"; + fdt =3D "fdt-1"; + compatible =3D "vendor,board1", "vendor,common"; + }; + conf-2 { + description =3D "board-meta-2"; + kernel =3D "kernel-1"; + fdt =3D "fdt-2"; + }; + conf-3 { + description =3D "board-meta-3"; + kernel =3D "kernel-1"; + fdt =3D "fdt-3", "fdt-ovl-1"; + compatible =3D "vendor,board3", "vendor,common"; + }; + conf-4 { + description =3D "board-meta-4"; + kernel =3D "kernel-1"; + fdt =3D "fdt-4"; + compatible =3D "vendor,other1", "vendor,other"; + }; + conf-5 { + description =3D "board-meta-5"; + kernel =3D "kernel-1"; + fdt =3D "fdt-5"; + compatible =3D "vendor,other2", "vendor,other"; + }; + }; +}; diff --git a/tests/run_tests.sh b/tests/run_tests.sh index 92edec8b724c..70a2f5ef6d97 100755 --- a/tests/run_tests.sh +++ b/tests/run_tests.sh @@ -1177,6 +1177,69 @@ fdtoverlay_tests() { run_wrap_test check_dtb_recreated \ fdtfactor_board1.test.dtb "fdtfactor_board1.${ext}_recreated.test.dtb" done + + # Compiled FIT image mode (2 families with <50% node-path overlap: 3 boar= ds + 2 boards) + run_dtc_test -@ -L -I dts -O dtb -o fdtfactor_other1.test.dtb "$SRCDIR/fd= tfactor_other1.dts" + run_dtc_test -@ -L -I dts -O dtb -o fdtfactor_other2.test.dtb "$SRCDIR/fd= tfactor_other2.dts" + + run_dtc_test -i . -I dts -O dtb -o fdtfactor_in.test.fit "$SRCDIR/fdtfact= or_test.its" + run_wrap_test $PYTHON "$FDTFACTOR" -v --verify \ + -f fdtfactor_in.test.fit -F fdtfactor.test.fit + run_fdtget_test "fdtfactor_board-base.dtb" fdtfactor.test.fit /images/fdt= -base-1 description -ts + run_fdtget_test "fdtfactor_other-base.dtb" fdtfactor.test.fit /images/fdt= -base-2 description -ts + run_fdtget_test "fdtfactor_board1.test.dtbo" fdtfactor.test.fit /images/f= dt-1 description -ts + run_fdtget_test "fdtfactor_board2.test.dtbo" fdtfactor.test.fit /images/f= dt-2 description -ts + run_fdtget_test "fdtfactor_board3.test.dtbo" fdtfactor.test.fit /images/f= dt-3 description -ts + run_fdtget_test "fdtfactor_other1.test.dtbo" fdtfactor.test.fit /images/f= dt-4 description -ts + run_fdtget_test "fdtfactor_other2.test.dtbo" fdtfactor.test.fit /images/f= dt-5 description -ts + run_wrap_test test -n "$($DTGET fdtfactor.test.fit /images/fdt-base-1/has= h-1 value -tx)" + run_wrap_test test -n "$($DTGET fdtfactor.test.fit /images/fdt-1/hash-1 v= alue -tx)" + run_fdtget_test "fdt-base-1 fdt-1" fdtfactor.test.fit /configurations/con= f-1 fdt -ts + run_fdtget_test "fdt-base-1 fdt-2" fdtfactor.test.fit /configurations/con= f-2 fdt -ts + run_fdtget_test "fdt-base-1 fdt-3 fdt-ovl-1" fdtfactor.test.fit /configur= ations/conf-3 fdt -ts + run_fdtget_test "fdt-base-2 fdt-4" fdtfactor.test.fit /configurations/con= f-4 fdt -ts + run_fdtget_test "fdt-base-2 fdt-5" fdtfactor.test.fit /configurations/con= f-5 fdt -ts + run_fdtget_test "overlay_overlay_phandle.test.dtb" fdtfactor.test.fit /im= ages/fdt-ovl-1 description -ts + run_fdtget_test "board-meta-1" fdtfactor.test.fit /configurations/conf-1 = description -ts + run_fdtget_test "vendor,board1 vendor,common" fdtfactor.test.fit /configu= rations/conf-1 compatible -ts + run_fdtget_test "vendor,board2 vendor,common" fdtfactor.test.fit /configu= rations/conf-2 compatible -ts + run_fdtget_test "vendor,other2 vendor,other" fdtfactor.test.fit /configur= ations/conf-5 compatible -ts + + # --single-base-dtb combines all boards (even with <50% overlap) into 1 b= ase DTB + run_wrap_test $PYTHON "$FDTFACTOR" --verify -S --single-base-dtb \ + -f fdtfactor_in.test.fit -F fdtfactor_single.test.fit + run_fdtget_test "kernel-1\nfdt-base-1\nfdt-1\nfdt-2\nfdt-3\nfdt-4\nfdt-5\= nfdt-ovl-1" \ + fdtfactor_single.test.fit -l /images + run_fdtget_test "fdtfactor-base.dtb" fdtfactor_single.test.fit /images/fd= t-base-1 description -ts + run_fdtget_test "fdt-base-1 fdt-5" fdtfactor_single.test.fit /configurati= ons/conf-5 fdt -ts + + # --single-base-dtb fails when flat_dt nodes have mismatched arch/headers + cp fdtfactor_in.test.fit fdtfactor_mismatch.test.fit + run_wrap_test $DTPUT -ts fdtfactor_mismatch.test.fit /images/fdt-5 arch a= rm + run_wrap_error_test $PYTHON "$FDTFACTOR" --verify --single-base-dtb \ + -f fdtfactor_mismatch.test.fit -F fdtfactor_single_fail.test.fit + + # Compressed (.dtb.lz4) FIT image mode (fdtfactor_board{1,2,3}.test.dtb.l= z4 + # were already generated by the compressed test loop above) + if command -v lz4 >/dev/null 2>&1; then + run_wrap_test lz4 -12 -q -f -m \ + fdtfactor_other1.test.dtb fdtfactor_other2.test.dtb overlay_overlay_phan= dle.test.dtb + sed -e 's/\.test\.dtb/.test.dtb.lz4/g' \ + -e 's/compression =3D "none"/compression =3D "lz4"/g' \ + -e 's/description =3D "fdtfactor_other1/description =3D "soc99-alpha/' \ + -e 's/description =3D "fdtfactor_other2/description =3D "soc99-beta/' \ + "$SRCDIR/fdtfactor_test.its" > fdtfactor_lz4.test.its + run_dtc_test -I dts -O dtb -o fdtfactor_lz4.test.fit fdtfactor_lz4.te= st.its + run_wrap_test $PYTHON "$FDTFACTOR" --verify \ + -f fdtfactor_lz4.test.fit + run_fdtget_test "lz4" fdtfactor_lz4.test.fit /images/fdt-base-1 compr= ession -ts + run_fdtget_test "lz4" fdtfactor_lz4.test.fit /images/fdt-base-2 compr= ession -ts + run_fdtget_test "fdtfactor_board-base.dtb.lz4" fdtfactor_lz4.test.fit= /images/fdt-base-1 description -ts + run_fdtget_test "soc99-base.dtb.lz4" fdtfactor_lz4.test.fit /images/f= dt-base-2 description -ts + run_fdtget_test "fdtfactor_board1.test.dtbo.lz4" fdtfactor_lz4.test.f= it /images/fdt-1 description -ts + run_fdtget_test "fdt-base-1 fdt-3 fdt-ovl-1" fdtfactor_lz4.test.fit /= configurations/conf-3 fdt -ts + run_fdtget_test "fdt-base-2 fdt-5" fdtfactor_lz4.test.fit /configurat= ions/conf-5 fdt -ts + fi fi } =20 @@ -1228,9 +1291,9 @@ if [ -z "$TESTSETS" ]; then fi =20 # Make sure we don't have stale blobs lying around -rm -f *.test.dtb *.test.dtbo *.test.dts *.test.dtb.gz *.test.dtbo.gz \ - *.test.dtb.xz *.test.dtbo.xz *.test.dtb.lzma *.test.dtbo.lzma \ - *.test.dtb.lz4 *.test.dtbo.lz4 +rm -f *.test.dtb *.test.dtbo *.test.dts *.test.its *.test.fit \ + *.test.dtb.gz *.test.dtbo.gz *.test.dtb.xz *.test.dtbo.xz \ + *.test.dtb.lzma *.test.dtbo.lzma *.test.dtb.lz4 *.test.dtbo.lz4 =20 for set in $TESTSETS; do case $set in --=20 2.56.0.176.g3575e65682-goog