arch/arm64/boot/dts/freescale/fsl-lx2160a.dtsi | 43 ++++++++++++++++++++++---- 1 file changed, 37 insertions(+), 6 deletions(-)
LX2160 SoC pci-e controller supports 64-bit memory regions up to 16GB,
32-bit regions up to 3GB and 16-bit regions up to 64k.
For each pci-e controller:
- extend the existing 32-bit regions to 3GB size
- add 16-bit region
- add 64-bit region
See [1] amd [2] for boot messages showing ranges before and after.
The 64-bit area flags are very particular:
- IORESOURCE_AUTO: ensures of_bus_pci_get_flags sets prefetch flag
(avoids "Memory resource size exceeds max for 32 bits" error during
boot, generated by pci_parse_request_of_pci_ranges)
- IORESOURCE_SYSRAM_DRIVER_MANAGED: ensures IORESOURCE_MEM flag is not
cleared, as required bydevm_of_pci_get_host_bridge_resources to print
correct resource type during boot:
MEM 0xa700000000..0xa7ffffffff -> 0xa700000000 (with this flag)
err 0xa700000000..0xa7ffffffff -> 0xa700000000 (without)
- IORESOURCE_MEM_64: pci address space is 64-bit
- IORESOURCE_PREFETCH: is prefetchable
- IORESOURCE_MEM: is memory (set implicitly when omitted in dts)
IORESOURCE_BUSY is dropped since it has no effect when specified in dts.
The 16GB 64-bit area is split into 4 pieces because the layerscape pcie
driver fails to program atu for larger ranges [3].
The range for 16-bit io window was defined by Jon Nettleton, and
includes flag IORESOURCE_EXT_TYPE_BITS to support multiport io cards.
Similar memory allocation with similar flags was tested with UEFI and ACPI
on pcie3 and pcie5.
This specific set of ranges was tested with nxp bsp versions lsdk-21.08,
ls-5.15.71-2.2.0, ls-6.6.52-2.2.0 - and Debian 13 (v6.12.41),
using u-boot:
- pcie5 with a Radeon Pro WX2100 with Gnome Desktop
- pcie3 with an ADATA NVME
This fixes allocation of large, and 64-bit BARs as requested by many pci
cards - especially graphics processors or AI accelerators, e.g.:
[ 2.941187] pci 0000:01:00.0: BAR 0: no space for [mem size 0x200000000 64bit pref]
[ 2.948834] pci 0000:01:00.0: BAR 0: failed to assign [mem size 0x200000000 64bit pref]
[1] example of new allocations (pcie5):
[ 1.716942] layerscape-pcie 3800000.pcie: host bridge /soc/pcie@3800000 ranges:
[ 1.724261] layerscape-pcie 3800000.pcie: MEM 0xa700000000..0xa7ffffffff -> 0xa700000000
[ 1.732795] layerscape-pcie 3800000.pcie: MEM 0xa600000000..0xa6ffffffff -> 0xa600000000
[ 1.741325] layerscape-pcie 3800000.pcie: MEM 0xa500000000..0xa5ffffffff -> 0xa500000000
[ 1.749861] layerscape-pcie 3800000.pcie: MEM 0xa400000000..0xa4ffffffff -> 0xa400000000
[ 1.758389] layerscape-pcie 3800000.pcie: MEM 0xa040000000..0xa0ffffffff -> 0x0040000000
[ 1.766915] layerscape-pcie 3800000.pcie: IO 0xa010000000..0xa01000ffff -> 0x0000000000
[ 1.776141] layerscape-pcie 3800000.pcie: iATU: unroll F, 256 ob, 24 ib, align 4K, limit 4G
[ 1.880382] layerscape-pcie 3800000.pcie: PCIe Gen.3 x8 link up
[ 1.886349] layerscape-pcie 3800000.pcie: PCI host bridge to bus 0001:00
[ 1.893046] pci_bus 0001:00: root bus resource [bus 00-ff]
[ 1.898525] pci_bus 0001:00: root bus resource [mem 0xa700000000-0xa7ffffffff pref]
[ 1.906174] pci_bus 0001:00: root bus resource [mem 0xa600000000-0xa6ffffffff pref]
[ 1.913822] pci_bus 0001:00: root bus resource [mem 0xa500000000-0xa5ffffffff pref]
[ 1.921471] pci_bus 0001:00: root bus resource [mem 0xa400000000-0xa4ffffffff pref]
[ 1.929120] pci_bus 0001:00: root bus resource [mem 0xa040000000-0xa0ffffffff] (bus address [0x40000000-0xffffffff])
[ 1.939633] pci_bus 0001:00: root bus resource [io 0x0000-0xffff]
[ 1.945824] pci 0001:00:00.0: [1957:8d80] type 01 class 0x060400 PCIe Root Port
[ 1.953146] pci 0001:00:00.0: PCI bridge to [bus 01-ff]
[ 1.958369] pci 0001:00:00.0: bridge window [io 0x1000-0x1fff]
[ 1.964456] pci 0001:00:00.0: bridge window [mem 0xa040000000-0xa0502fffff]
[2] example of previous allocations (pcie5):
[ 1.716744] layerscape-pcie 3800000.pcie: host bridge /soc/pcie@3800000 ranges:
[ 1.724060] layerscape-pcie 3800000.pcie: MEM 0xa040000000..0xa07fffffff -> 0x0040000000
[ 1.733277] layerscape-pcie 3800000.pcie: iATU: unroll F, 256 ob, 24 ib, align 4K, limit 4G
[ 1.836220] layerscape-pcie 3800000.pcie: PCIe Gen.3 x8 link up
[ 1.842186] layerscape-pcie 3800000.pcie: PCI host bridge to bus 0001:00
[ 1.848883] pci_bus 0001:00: root bus resource [bus 00-ff]
[ 1.854363] pci_bus 0001:00: root bus resource [mem 0xa040000000-0xa07fffffff] (bus address [0x40000000-0x7fffffff])
[ 1.864892] pci 0001:00:00.0: [1957:8d80] type 01 class 0x060400 PCIe Root Port
[ 1.872216] pci 0001:00:00.0: PCI bridge to [bus 01-ff]
[ 1.877438] pci 0001:00:00.0: bridge window [io 0x1000-0x1fff]
[ 1.883526] pci 0001:00:00.0: bridge window [mem 0xa040000000-0xa0502fffff]
[3] error programming atu beyond 4GB:
[ 1.716762] layerscape-pcie 3800000.pcie: host bridge /soc/pcie@3800000 ranges:
[ 1.724080] layerscape-pcie 3800000.pcie: MEM 0xa400000000..0xa7ffffffff -> 0xa400000000
[ 1.732615] layerscape-pcie 3800000.pcie: MEM 0xa040000000..0xa0ffffffff -> 0x0040000000
[ 1.741142] layerscape-pcie 3800000.pcie: IO 0xa010000000..0xa01000ffff -> 0x0000000000
[ 1.750379] layerscape-pcie 3800000.pcie: iATU: unroll F, 256 ob, 24 ib, align 4K, limit 4G
[ 1.759089] layerscape-pcie 3800000.pcie: Failed to set MEM range [mem 0xa400000000-0xa7ffffffff flags 0x2200]
[ 1.769089] layerscape-pcie 3800000.pcie: probe with driver layerscape-pcie failed with error -22
Signed-off-by: Josua Mayer <josua@solid-run.com>
---
Changes in v3:
- dropped rfc label
- adjusted flags
- split 16GB area into 4x4GB sections.
- enhance commit description with details explanation
- Link to v2: https://lore.kernel.org/r/20240429-lx2160-pci-v2-1-1b94576d6263@solid-run.com
Changes in v2:
- adjusted flags to fix several errors during probe and bar allocation
- explicitly tested with 2 pci cards on Debian (Linux 6.1)
- still rfc because a limitation in designware pci driver
- Link to v1: https://lore.kernel.org/r/20240321-lx2160-pci-v1-1-3673708f7eb6@solid-run.com
---
arch/arm64/boot/dts/freescale/fsl-lx2160a.dtsi | 43 ++++++++++++++++++++++----
1 file changed, 37 insertions(+), 6 deletions(-)
diff --git a/arch/arm64/boot/dts/freescale/fsl-lx2160a.dtsi b/arch/arm64/boot/dts/freescale/fsl-lx2160a.dtsi
index c9541403bcd8239a48d4ef79c7c4f9e3b607b556..1a399ca8572b7ca2707856d5a041e4e024248e7e 100644
--- a/arch/arm64/boot/dts/freescale/fsl-lx2160a.dtsi
+++ b/arch/arm64/boot/dts/freescale/fsl-lx2160a.dtsi
@@ -1,3 +1,4 @@
+
// SPDX-License-Identifier: (GPL-2.0 OR MIT)
//
// Device Tree Include file for Layerscape-LX2160A family SoC.
@@ -1181,7 +1182,12 @@ pcie1: pcie@3400000 {
apio-wins = <8>;
ppio-wins = <8>;
bus-range = <0x0 0xff>;
- ranges = <0x82000000 0x0 0x40000000 0x80 0x40000000 0x0 0x40000000>; /* non-prefetchable memory */
+ ranges = <0x42102200 0x87 0x00000000 0x87 0x00000000 0x01 0x00000000>, /* 64-Bit - prefetchable - 4GB chunk */
+ <0x42102200 0x86 0x00000000 0x86 0x00000000 0x01 0x00000000>, /* 64-Bit - prefetchable - 4GB chunk */
+ <0x42102200 0x85 0x00000000 0x85 0x00000000 0x01 0x00000000>, /* 64-Bit - prefetchable - 4GB chunk */
+ <0x42102200 0x84 0x00000000 0x84 0x00000000 0x01 0x00000000>, /* 64-Bit - prefetchable - 4GB chunk */
+ <0x02000200 0x00 0x40000000 0x80 0x40000000 0x00 0xc0000000>, /* 32-Bit - non-prefetchable */
+ <0x01200100 0x00 0x00000000 0x80 0x10000000 0x00 0x00010000>; /* 16-Bit IO Window */
msi-parent = <&its 0>;
#interrupt-cells = <1>;
interrupt-map-mask = <0 0 0 7>;
@@ -1209,7 +1215,12 @@ pcie2: pcie@3500000 {
apio-wins = <8>;
ppio-wins = <8>;
bus-range = <0x0 0xff>;
- ranges = <0x82000000 0x0 0x40000000 0x88 0x40000000 0x0 0x40000000>; /* non-prefetchable memory */
+ ranges = <0x42102200 0x8f 0x00000000 0x8f 0x00000000 0x01 0x00000000>, /* 64-Bit - prefetchable - 4GB chunk */
+ <0x42102200 0x8e 0x00000000 0x8e 0x00000000 0x01 0x00000000>, /* 64-Bit - prefetchable - 4GB chunk */
+ <0x42102200 0x8d 0x00000000 0x8d 0x00000000 0x01 0x00000000>, /* 64-Bit - prefetchable - 4GB chunk */
+ <0x42102200 0x8c 0x00000000 0x8c 0x00000000 0x01 0x00000000>, /* 64-Bit - prefetchable - 4GB chunk */
+ <0x02000200 0x00 0x40000000 0x88 0x40000000 0x00 0xc0000000>, /* 32-Bit - non-prefetchable */
+ <0x01200100 0x00 0x00000000 0x88 0x10000000 0x00 0x00010000>; /* 16-Bit IO Window */
msi-parent = <&its 0>;
#interrupt-cells = <1>;
interrupt-map-mask = <0 0 0 7>;
@@ -1237,7 +1248,12 @@ pcie3: pcie@3600000 {
apio-wins = <256>;
ppio-wins = <24>;
bus-range = <0x0 0xff>;
- ranges = <0x82000000 0x0 0x40000000 0x90 0x40000000 0x0 0x40000000>; /* non-prefetchable memory */
+ ranges = <0x42102200 0x97 0x00000000 0x97 0x00000000 0x01 0x00000000>, /* 64-Bit - prefetchable - 4GB chunk */
+ <0x42102200 0x96 0x00000000 0x96 0x00000000 0x01 0x00000000>, /* 64-Bit - prefetchable - 4GB chunk */
+ <0x42102200 0x95 0x00000000 0x95 0x00000000 0x01 0x00000000>, /* 64-Bit - prefetchable - 4GB chunk */
+ <0x42102200 0x94 0x00000000 0x94 0x00000000 0x01 0x00000000>, /* 64-Bit - prefetchable - 4GB chunk */
+ <0x02000200 0x00 0x40000000 0x90 0x40000000 0x00 0xc0000000>, /* 32-Bit - non-prefetchable */
+ <0x01200100 0x00 0x00000000 0x90 0x10000000 0x00 0x00010000>; /* 16-Bit IO Window */
msi-parent = <&its 0>;
#interrupt-cells = <1>;
interrupt-map-mask = <0 0 0 7>;
@@ -1265,7 +1281,12 @@ pcie4: pcie@3700000 {
apio-wins = <8>;
ppio-wins = <8>;
bus-range = <0x0 0xff>;
- ranges = <0x82000000 0x0 0x40000000 0x98 0x40000000 0x0 0x40000000>; /* non-prefetchable memory */
+ ranges = <0x42102200 0x9f 0x00000000 0x9f 0x00000000 0x01 0x00000000>, /* 64-Bit - prefetchable - 4GB chunk */
+ <0x42102200 0x9e 0x00000000 0x9e 0x00000000 0x01 0x00000000>, /* 64-Bit - prefetchable - 4GB chunk */
+ <0x42102200 0x9d 0x00000000 0x9d 0x00000000 0x01 0x00000000>, /* 64-Bit - prefetchable - 4GB chunk */
+ <0x42102200 0x9c 0x00000000 0x9c 0x00000000 0x01 0x00000000>, /* 64-Bit - prefetchable - 4GB chunk */
+ <0x02000200 0x00 0x40000000 0x98 0x40000000 0x00 0xc0000000>, /* 32-Bit - non-prefetchable */
+ <0x01200100 0x00 0x00000000 0x98 0x10000000 0x00 0x00010000>; /* 16-Bit IO Window */
msi-parent = <&its 0>;
#interrupt-cells = <1>;
interrupt-map-mask = <0 0 0 7>;
@@ -1293,7 +1314,12 @@ pcie5: pcie@3800000 {
apio-wins = <256>;
ppio-wins = <24>;
bus-range = <0x0 0xff>;
- ranges = <0x82000000 0x0 0x40000000 0xa0 0x40000000 0x0 0x40000000>; /* non-prefetchable memory */
+ ranges = <0x42102200 0xa7 0x00000000 0xa7 0x00000000 0x01 0x00000000>, /* 64-Bit - prefetchable - 4GB chunk */
+ <0x42102200 0xa6 0x00000000 0xa6 0x00000000 0x01 0x00000000>, /* 64-Bit - prefetchable - 4GB chunk */
+ <0x42102200 0xa5 0x00000000 0xa5 0x00000000 0x01 0x00000000>, /* 64-Bit - prefetchable - 4GB chunk */
+ <0x42102200 0xa4 0x00000000 0xa4 0x00000000 0x01 0x00000000>, /* 64-Bit - prefetchable - 4GB chunk */
+ <0x02000200 0x00 0x40000000 0xa0 0x40000000 0x00 0xc0000000>, /* 32-Bit - non-prefetchable */
+ <0x01200100 0x00 0x00000000 0xa0 0x10000000 0x00 0x00010000>; /* 16-Bit IO Window */
msi-parent = <&its 0>;
#interrupt-cells = <1>;
interrupt-map-mask = <0 0 0 7>;
@@ -1321,7 +1347,12 @@ pcie6: pcie@3900000 {
apio-wins = <8>;
ppio-wins = <8>;
bus-range = <0x0 0xff>;
- ranges = <0x82000000 0x0 0x40000000 0xa8 0x40000000 0x0 0x40000000>; /* non-prefetchable memory */
+ ranges = <0x42102200 0xaf 0x00000000 0xaf 0x00000000 0x01 0x00000000>, /* 64-Bit - prefetchable - 4GB chunk */
+ <0x42102200 0xae 0x00000000 0xae 0x00000000 0x01 0x00000000>, /* 64-Bit - prefetchable - 4GB chunk */
+ <0x42102200 0xad 0x00000000 0xad 0x00000000 0x01 0x00000000>, /* 64-Bit - prefetchable - 4GB chunk */
+ <0x42102200 0xac 0x00000000 0xac 0x00000000 0x01 0x00000000>, /* 64-Bit - prefetchable - 4GB chunk */
+ <0x02000200 0x00 0x40000000 0xa8 0x40000000 0x00 0xc0000000>, /* 32-Bit - non-prefetchable */
+ <0x01200100 0x00 0x00000000 0xa8 0x10000000 0x00 0x00010000>; /* 16-Bit IO Window */
msi-parent = <&its 0>;
#interrupt-cells = <1>;
interrupt-map-mask = <0 0 0 7>;
---
base-commit: 8f5ae30d69d7543eee0d70083daf4de8fe15d585
change-id: 20240118-lx2160-pci-4bdb196e58f3
Best regards,
--
Josua Mayer <josua@solid-run.com>© 2016 - 2025 Red Hat, Inc.