Resetting resource is problematic as it prevent attempting to allocate
the resource later, unless something in between restores the resource.
Similarly, if fail_head does not contain all resources that were reset,
those resource cannot be restored later.
The entire reset/restore cycle adds complexity and leaving resources
into reseted state causes issues to other code such as for checks done
in pci_enable_resources(). Take a small step towards not resetting
resources by delaying reset until the end of resource assignment and
build failure list (fail_head) in sync with the reset to avoid leaving
behind resources that cannot be restored (for the case where the caller
provides fail_head in the first place to allow restore somewhere in the
callchain, as is not all callers pass non-NULL fail_head).
The Expansion ROM check is temporarily left in place while building the
failure list until the upcoming change which reworks optional resource
handling.
Ideally, whole resource reset could be removed but doing that in a big
step would make the impact non-tractable due to complexity of all
related code.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
---
drivers/pci/setup-bus.c | 35 +++++++++++++++++++++++++++++------
1 file changed, 29 insertions(+), 6 deletions(-)
diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c
index fec7d68fb971..b61f24a5cfa5 100644
--- a/drivers/pci/setup-bus.c
+++ b/drivers/pci/setup-bus.c
@@ -252,9 +252,14 @@ static void reassign_resources_sorted(struct list_head *realloc_head,
res = add_res->res;
dev = add_res->dev;
+ idx = pci_resource_num(dev, res);
- /* Skip resource that has been reset */
- if (!res->flags)
+ /*
+ * Skip resource that failed the earlier assignment and is
+ * not optional as it would just fail again.
+ */
+ if (!res->parent && resource_size(res) &&
+ !pci_resource_is_optional(dev, idx))
goto out;
/* Skip this resource if not found in head list */
@@ -267,7 +272,6 @@ static void reassign_resources_sorted(struct list_head *realloc_head,
if (!found_match) /* Just skip */
continue;
- idx = pci_resource_num(dev, res);
res_name = pci_resource_name(dev, idx);
add_size = add_res->add_size;
align = add_res->min_align;
@@ -277,7 +281,6 @@ static void reassign_resources_sorted(struct list_head *realloc_head,
pci_dbg(dev,
"%s %pR: ignoring failure in optional allocation\n",
res_name, res);
- reset_resource(res);
}
} else {
res->flags |= add_res->flags &
@@ -332,7 +335,6 @@ static void assign_requested_resources_sorted(struct list_head *head,
0 /* don't care */,
0 /* don't care */);
}
- reset_resource(res);
}
}
}
@@ -518,13 +520,34 @@ static void __assign_resources_sorted(struct list_head *head,
requested_and_reassign:
/* Satisfy the must-have resource requests */
- assign_requested_resources_sorted(head, fail_head);
+ assign_requested_resources_sorted(head, NULL);
/* Try to satisfy any additional optional resource requests */
if (!list_empty(realloc_head))
reassign_resources_sorted(realloc_head, head);
out:
+ /* Reset any failed resource, cannot use fail_head as it can be NULL. */
+ list_for_each_entry(dev_res, head, list) {
+ res = dev_res->res;
+ dev = dev_res->dev;
+
+ if (res->parent)
+ continue;
+
+ /*
+ * If the failed resource is a ROM BAR and it will
+ * be enabled later, don't add it to the list.
+ */
+ if (fail_head && !pci_resource_is_disabled_rom(res, idx)) {
+ add_to_list(fail_head, dev, res,
+ 0 /* don't care */,
+ 0 /* don't care */);
+ }
+
+ reset_resource(res);
+ }
+
free_list(head);
}
--
2.39.5
Ilpo Järvinen <ilpo.jarvinen@linux.intel.com> writes:
> Resetting resource is problematic as it prevent attempting to allocate
> the resource later, unless something in between restores the resource.
> Similarly, if fail_head does not contain all resources that were reset,
> those resource cannot be restored later.
>
> The entire reset/restore cycle adds complexity and leaving resources
> into reseted state causes issues to other code such as for checks done
> in pci_enable_resources(). Take a small step towards not resetting
> resources by delaying reset until the end of resource assignment and
> build failure list (fail_head) in sync with the reset to avoid leaving
> behind resources that cannot be restored (for the case where the caller
> provides fail_head in the first place to allow restore somewhere in the
> callchain, as is not all callers pass non-NULL fail_head).
>
> The Expansion ROM check is temporarily left in place while building the
> failure list until the upcoming change which reworks optional resource
> handling.
>
> Ideally, whole resource reset could be removed but doing that in a big
> step would make the impact non-tractable due to complexity of all
> related code.
>
> Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
Hi Ilpo, I'm seeing a crash on arm64 at boot that I bisected to this
change. I don't think it's the same as the other issues reported. I've
confirmed the crash is still there after your follow up patches. The
crash itself is below[1].
It looks like the problem begins when:
amdgpu_device_resize_fb_bar()
pci_resize_resource()
pci_reassign_bridge_resources()
__pci_bus_size_bridges()
adds pci_hotplug_io_size to `realloc_head`. The io resource allocation
has failed earlier because the root port doesn't have an io window[2].
Then with this patch, pci_reassign_bridge_resources()'s call to
__pci_bridge_assign_resources() now returns the io added space for
hotplug in the `failed` list where the old code dropped it and did not.
That sends pci_reassign_bridge_resources() into the `cleanup:` path,
where I think the cleanup code doesn't properly release the resources
that were assigned by __pci_bridge_assign_resources() and there's a
conflict reported in pci_claim_resource() where a restored resource is
found as conflicting with itself:
> pcieport 000d:00:01.0: bridge window [mem 0x340000000000-0x340017ffffff 64bit pref]: can't claim; address conflict with PCI Bus 000d:01 [mem 0x340000000000-0x340017ffffff 64bit pref]
Setting `pci=hpiosize=0` avoids this crash, as does this change:
diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c
index 16d5d390599a..59ece11702da 100644
--- a/drivers/pci/setup-bus.c
+++ b/drivers/pci/setup-bus.c
@@ -2442,7 +2442,7 @@ int pci_reassign_bridge_resources(struct pci_dev *bridge, unsigned long type)
LIST_HEAD(saved);
LIST_HEAD(added);
LIST_HEAD(failed);
- unsigned int i;
+ unsigned int i, relevant_fails;
int ret;
down_read(&pci_bus_sem);
@@ -2490,7 +2490,16 @@ int pci_reassign_bridge_resources(struct pci_dev *bridge, unsigned long type)
__pci_bridge_assign_resources(bridge, &added, &failed);
BUG_ON(!list_empty(&added));
- if (!list_empty(&failed)) {
+ relevant_fails = 0;
+ list_for_each_entry(dev_res, &failed, list) {
+ restore_dev_resource(dev_res);
+ if (((dev_res->res->flags ^ type) & PCI_RES_TYPE_MASK) == 0)
+ relevant_fails++;
+ }
+ free_list(&failed);
+
+ /* Cleanup if we had failures in resources of interest */
+ if (relevant_fails != 0) {
ret = -ENOSPC;
goto cleanup;
}
@@ -2509,11 +2518,6 @@ int pci_reassign_bridge_resources(struct pci_dev *bridge, unsigned long type)
return 0;
cleanup:
- /* Restore size and flags */
- list_for_each_entry(dev_res, &failed, list)
- restore_dev_resource(dev_res);
- free_list(&failed);
-
/* Revert to the old configuration */
list_for_each_entry(dev_res, &saved, list) {
struct resource *res = dev_res->res;
I don't know this code well enough to know if that changes is completely
bonkers or what. Maybe a change that gets zero as pci_hotplug_io_size
for root ports that don't have an io window would be better? Any other
ideas, or other information about the crash that I could provide?
Thanks,
Scott
[1]: Crash:
> SError Interrupt on CPU0, code 0x00000000be000411 -- SError
> CPU: 0 UID: 0 PID: 9 Comm: kworker/0:0 Not tainted 6.15.2+ #16 PREEMPT(undef)
> Hardware name: Adlink Ampere Altra Developer Platform/Ampere Altra Developer Platform, BIOS TianoCore 2.09.100.00 (SYS: 2.10.20221028) 04/30/2
> Workqueue: events work_for_cpu_fn
> pstate: 204000c9 (nzCv daIF +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
> pc : _raw_spin_lock_irqsave+0x34/0xb0
> lr : __wake_up+0x30/0x80
> sp : ffff800080003e20
> x29: ffff800080003e20 x28: ffff07ff808b0000 x27: 0000000000050260
> x26: 0000000000000001 x25: ffffcc5decad6bd8 x24: ffffcc5decc8b5b8
> x23: ffff080138a10000 x22: ffff080138a00000 x21: 0000000000000003
> x20: ffff080138a14dc8 x19: 0000000000000000 x18: 006808018e775f03
> x17: ffff3bc1330d2000 x16: ffffcc5e3a520ed8 x15: 8daad055c6e77021
> x14: f231631cf9328575 x13: 0e51168d06a6cf91 x12: f5db8c23b764520c
> x11: 0000000000000040 x10: 0000000000000000 x9 : ffffcc5e3a520f08
> x8 : 0000000000002113 x7 : 0000000000000000 x6 : 0000000000000000
> x5 : 0000000000000000 x4 : ffff800080003e08 x3 : 00000000000000c0
> x2 : 0000000000000001 x1 : 0000000000000000 x0 : ffff080138a14dc8
> Kernel panic - not syncing: Asynchronous SError Interrupt
> CPU: 0 UID: 0 PID: 9 Comm: kworker/0:0 Not tainted 6.15.2+ #16 PREEMPT(undef)
> Hardware name: Adlink Ampere Altra Developer Platform/Ampere Altra Developer Platform, BIOS TianoCore 2.09.100.00 (SYS: 2.10.20221028) 04/30/2
> Workqueue: events work_for_cpu_fn
> Call trace:
> show_stack+0x30/0x98 (C)
> dump_stack_lvl+0x7c/0xa0
> dump_stack+0x18/0x2c
> panic+0x184/0x3c0
> nmi_panic+0x90/0xa0
> arm64_serror_panic+0x6c/0x90
> do_serror+0x58/0x60
> el1h_64_error_handler+0x38/0x60
> el1h_64_error+0x84/0x88
> _raw_spin_lock_irqsave+0x34/0xb0 (P)
> amdgpu_ih_process+0xf0/0x150 [amdgpu]
> amdgpu_irq_handler+0x34/0xa0 [amdgpu]
> __handle_irq_event_percpu+0x60/0x248
> handle_irq_event+0x4c/0xc0
> handle_fasteoi_irq+0xa0/0x1c8
> handle_irq_desc+0x3c/0x68
> generic_handle_domain_irq+0x24/0x40
> __gic_handle_irq_from_irqson.isra.0+0x15c/0x260
> gic_handle_irq+0x28/0x80
> call_on_irq_stack+0x24/0x30
> do_interrupt_handler+0x88/0xa0
> el1_interrupt+0x44/0xd0
> el1h_64_irq_handler+0x18/0x28
> el1h_64_irq+0x84/0x88
> amdgpu_device_rreg.part.0+0x4c/0x190 [amdgpu] (P)
> amdgpu_device_rreg+0x24/0x40 [amdgpu]
> psp_wait_for+0x88/0xd8 [amdgpu]
> psp_v11_0_bootloader_load_component+0x164/0x1b0 [amdgpu]
> psp_v11_0_bootloader_load_kdb+0x20/0x40 [amdgpu]
> psp_hw_start+0x5c/0x580 [amdgpu]
> psp_load_fw+0x9c/0x280 [amdgpu]
> psp_hw_init+0x44/0xa0 [amdgpu]
> amdgpu_device_fw_loading+0xf8/0x358 [amdgpu]
> amdgpu_device_ip_init+0x684/0x990 [amdgpu]
> amdgpu_device_init+0xba4/0x1038 [amdgpu]
> amdgpu_driver_load_kms+0x20/0xb8 [amdgpu]
> amdgpu_pci_probe+0x1f8/0x7f8 [amdgpu]
> local_pci_probe+0x44/0xb0
> work_for_cpu_fn+0x24/0x40
> process_one_work+0x17c/0x410
> worker_thread+0x254/0x388
> kthread+0x10c/0x128
> ret_from_fork+0x10/0x20
> Kernel Offset: 0x4c5dba380000 from 0xffff800080000000
> PHYS_OFFSET: 0x80000000
> CPU features: 0x0800,000042e0,01202650,8241720b
> Memory Limit: none
> ---[ end Kernel panic - not syncing: Asynchronous SError Interrupt ]---
[2]: boot snippet
> ACPI: PCI Root Bridge [PCI1] (domain 000d [bus 00-ff])
> acpi PNP0A08:01: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI EDR HPX-Type3]
> acpi PNP0A08:01: _OSC: platform does not support [PCIeHotplug PME LTR DPC]
> acpi PNP0A08:01: _OSC: OS now controls [PCIeCapability]
> acpi PNP0A08:01: FADT indicates ASPM is unsupported, using BIOS configuration
> acpi PNP0A08:01: MCFG quirk: ECAM at [mem 0x37fff0000000-0x37ffffffffff] for [bus 00-ff] with pci_32b_read_ops
> acpi PNP0A08:01: ECAM area [mem 0x37fff0000000-0x37ffffffffff] reserved by PNP0C02:00
> acpi PNP0A08:01: ECAM at [mem 0x37fff0000000-0x37ffffffffff] for [bus 00-ff]
> PCI host bridge to bus 000d:00
> pci_bus 000d:00: root bus resource [mem 0x50000000-0x5fffffff window]
> pci_bus 000d:00: root bus resource [mem 0x340000000000-0x37ffdfffffff window]
> pci_bus 000d:00: root bus resource [bus 00-ff]
> pci 000d:00:00.0: [1def:e100] type 00 class 0x060000 conventional PCI endpoint
> pci 000d:00:01.0: [1def:e101] type 01 class 0x060400 PCIe Root Port
> pci 000d:00:01.0: PCI bridge to [bus 01-03]
> pci 000d:00:01.0: bridge window [io 0xe000-0xefff]
> pci 000d:00:01.0: bridge window [mem 0x50000000-0x502fffff]
> pci 000d:00:01.0: bridge window [mem 0x340000000000-0x3400101fffff 64bit pref]
> pci 000d:00:01.0: supports D1 D2
> pci 000d:00:01.0: PME# supported from D0 D1 D3hot
> ...
> pci 000d:00:01.0: bridge window [mem 0x340000000000-0x340017ffffff 64bit pref]: assigned
> pci 000d:00:01.0: bridge window [mem 0x50000000-0x502fffff]: assigned
> pci 000d:00:01.0: bridge window [io size 0x1000]: can't assign; no space
> pci 000d:00:01.0: bridge window [io size 0x1000]: failed to assign
> pci 000d:00:01.0: bridge window [io size 0x1000]: can't assign; no space
> pci 000d:00:01.0: bridge window [io size 0x1000]: failed to assign
On Wed, 18 Jun 2025, D Scott Phillips wrote:
> Ilpo Järvinen <ilpo.jarvinen@linux.intel.com> writes:
>
> > Resetting resource is problematic as it prevent attempting to allocate
> > the resource later, unless something in between restores the resource.
> > Similarly, if fail_head does not contain all resources that were reset,
> > those resource cannot be restored later.
> >
> > The entire reset/restore cycle adds complexity and leaving resources
> > into reseted state causes issues to other code such as for checks done
> > in pci_enable_resources(). Take a small step towards not resetting
> > resources by delaying reset until the end of resource assignment and
> > build failure list (fail_head) in sync with the reset to avoid leaving
> > behind resources that cannot be restored (for the case where the caller
> > provides fail_head in the first place to allow restore somewhere in the
> > callchain, as is not all callers pass non-NULL fail_head).
> >
> > The Expansion ROM check is temporarily left in place while building the
> > failure list until the upcoming change which reworks optional resource
> > handling.
> >
> > Ideally, whole resource reset could be removed but doing that in a big
> > step would make the impact non-tractable due to complexity of all
> > related code.
> >
> > Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
>
> Hi Ilpo, I'm seeing a crash on arm64 at boot that I bisected to this
> change. I don't think it's the same as the other issues reported. I've
> confirmed the crash is still there after your follow up patches. The
> crash itself is below[1].
>
> It looks like the problem begins when:
>
> amdgpu_device_resize_fb_bar()
> pci_resize_resource()
> pci_reassign_bridge_resources()
> __pci_bus_size_bridges()
>
> adds pci_hotplug_io_size to `realloc_head`. The io resource allocation
> has failed earlier because the root port doesn't have an io window[2].
>
> Then with this patch, pci_reassign_bridge_resources()'s call to
> __pci_bridge_assign_resources() now returns the io added space for
> hotplug in the `failed` list where the old code dropped it and did not.
>
> That sends pci_reassign_bridge_resources() into the `cleanup:` path,
> where I think the cleanup code doesn't properly release the resources
> that were assigned by __pci_bridge_assign_resources() and there's a
> conflict reported in pci_claim_resource() where a restored resource is
> found as conflicting with itself:
>
> > pcieport 000d:00:01.0: bridge window [mem 0x340000000000-0x340017ffffff 64bit pref]: can't claim; address conflict with PCI Bus 000d:01 [mem 0x340000000000-0x340017ffffff 64bit pref]
>
> Setting `pci=hpiosize=0` avoids this crash, as does this change:
>
> diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c
> index 16d5d390599a..59ece11702da 100644
> --- a/drivers/pci/setup-bus.c
> +++ b/drivers/pci/setup-bus.c
> @@ -2442,7 +2442,7 @@ int pci_reassign_bridge_resources(struct pci_dev *bridge, unsigned long type)
> LIST_HEAD(saved);
> LIST_HEAD(added);
> LIST_HEAD(failed);
> - unsigned int i;
> + unsigned int i, relevant_fails;
> int ret;
>
> down_read(&pci_bus_sem);
> @@ -2490,7 +2490,16 @@ int pci_reassign_bridge_resources(struct pci_dev *bridge, unsigned long type)
> __pci_bridge_assign_resources(bridge, &added, &failed);
> BUG_ON(!list_empty(&added));
>
> - if (!list_empty(&failed)) {
> + relevant_fails = 0;
> + list_for_each_entry(dev_res, &failed, list) {
> + restore_dev_resource(dev_res);
> + if (((dev_res->res->flags ^ type) & PCI_RES_TYPE_MASK) == 0)
> + relevant_fails++;
> + }
> + free_list(&failed);
> +
> + /* Cleanup if we had failures in resources of interest */
> + if (relevant_fails != 0) {
> ret = -ENOSPC;
> goto cleanup;
> }
> @@ -2509,11 +2518,6 @@ int pci_reassign_bridge_resources(struct pci_dev *bridge, unsigned long type)
> return 0;
>
> cleanup:
> - /* Restore size and flags */
> - list_for_each_entry(dev_res, &failed, list)
> - restore_dev_resource(dev_res);
> - free_list(&failed);
> -
> /* Revert to the old configuration */
> list_for_each_entry(dev_res, &saved, list) {
> struct resource *res = dev_res->res;
>
> I don't know this code well enough to know if that changes is completely
> bonkers or what.
Hi again,
Thanks for all the details what you think went wrong, it was really
useful. I think you have it towards the right direction but a more
targetted seems enough to address this (this needs to be confirmed, please
test the patch below).
The most correct solution would be to make all the resource fitting code
to focus on the resources that match the type filter. However, that looks
way too scary change at the moment to implement, and especially, let it
end up into stable (to fix this issue). So it looks this somewhat band-aid
solution similar to your attempt might be better as a fix for now.
In medium term, I'd want to avoid using type as a filter and base all
such decisions on matching the bridge window resource the dev resource
belongs to. I've some work towards that direction already which removes
lots of complexity in which bridge window is going to be selected as
there will be a single place to make always the same decision. That change
is also going to simplify the internal interfaces between functions very
noticably (but the change require more testing before I've enough
confidence to submit it). That work doesn't cover this resize side yet but
it should be extended there as well.
So please test this somewhat band-aid patch:
From 971686ed85e341e7234f8fe8b666140187f63ad1 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Ilpo=20J=C3=A4rvinen?= <ilpo.jarvinen@linux.intel.com>
Date: Wed, 25 Jun 2025 20:30:43 +0300
Subject: [PATCH 1/1] PCI: Fix failure detectiong during resource resize
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Since the commit 96336ec70264 ("PCI: Perform reset_resource() and build
fail list in sync") the failed list is always built and returned to let
the caller decide if what to do with the failures. The caller may want
to retry resource fitting and assignment and before that can happen,
the resources should be restored to their original state (a reset
effectively clears the struct resource), which requires returning them
on the failed list so that the original state remains stored in the
associated struct pci_dev_resource.
Resource resizing is different from the ordinary resource fitting and
assignment in that it only considers part of the resources. This means
failures for other resource types are not relevant at all and should be
ignored. As resize doesn't unassign such unrelated resources, those
resource ending up into the failed list implies assignment of that
resource must have failed before resize too. The check in
pci_reassign_bridge_resources() to decide if the whole assignment is
successful, however, is based on list emptiness which may cause false
negatives when the failed list resources with unrelated type.
If the failed list is not empty, call pci_required_resource_failed()
and extend it to be able to filter on specific resource types too (if
provided).
Calling pci_required_resource_failed() at this point is slightly
problematic because the resource itself is reset when the failed list
is constructed in __assign_resources_sorted(). As a result,
pci_resource_is_optional() does not have access to the original
resource flags. This could be worked around by restoring and
re-reseting the resource around the call to pci_resource_is_optional(),
however, it shouldn't cause issue as resource resizing is meant for
64-bit prefetchable resources according to Christian König (see the
Link which unfortunately doesn't point directly to Christian's reply
because lore didn't store that email at all).
Link: https://lore.kernel.org/all/c5d1b5d8-8669-5572-75a7-0b480f581ac1@linux.intel.com/
Reported-by: D Scott Phillips <scott@os.amperecomputing.com>
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
Cc: Christian König <christian.koenig@amd.com>
---
drivers/pci/setup-bus.c | 26 ++++++++++++++++++--------
1 file changed, 18 insertions(+), 8 deletions(-)
diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c
index 07c3d021a47e..8284bbdc44b4 100644
--- a/drivers/pci/setup-bus.c
+++ b/drivers/pci/setup-bus.c
@@ -28,6 +28,10 @@
#include <linux/acpi.h>
#include "pci.h"
+#define PCI_RES_TYPE_MASK \
+ (IORESOURCE_IO | IORESOURCE_MEM | IORESOURCE_PREFETCH |\
+ IORESOURCE_MEM_64)
+
unsigned int pci_flags;
EXPORT_SYMBOL_GPL(pci_flags);
@@ -384,13 +388,19 @@ static bool pci_need_to_release(unsigned long mask, struct resource *res)
}
/* Return: @true if assignment of a required resource failed. */
-static bool pci_required_resource_failed(struct list_head *fail_head)
+static bool pci_required_resource_failed(struct list_head *fail_head,
+ unsigned long type)
{
struct pci_dev_resource *fail_res;
+ type &= ~PCI_RES_TYPE_MASK;
+
list_for_each_entry(fail_res, fail_head, list) {
int idx = pci_resource_num(fail_res->dev, fail_res->res);
+ if (type && (fail_res->flags & PCI_RES_TYPE_MASK) != type)
+ continue;
+
if (!pci_resource_is_optional(fail_res->dev, idx))
return true;
}
@@ -504,7 +514,7 @@ static void __assign_resources_sorted(struct list_head *head,
}
/* Without realloc_head and only optional fails, nothing more to do. */
- if (!pci_required_resource_failed(&local_fail_head) &&
+ if (!pci_required_resource_failed(&local_fail_head, 0) &&
list_empty(realloc_head)) {
list_for_each_entry(save_res, &save_head, list) {
struct resource *res = save_res->res;
@@ -1704,10 +1714,6 @@ static void __pci_bridge_assign_resources(const struct pci_dev *bridge,
}
}
-#define PCI_RES_TYPE_MASK \
- (IORESOURCE_IO | IORESOURCE_MEM | IORESOURCE_PREFETCH |\
- IORESOURCE_MEM_64)
-
static void pci_bridge_release_resources(struct pci_bus *bus,
unsigned long type)
{
@@ -2445,8 +2451,12 @@ int pci_reassign_bridge_resources(struct pci_dev *bridge, unsigned long type)
free_list(&added);
if (!list_empty(&failed)) {
- ret = -ENOSPC;
- goto cleanup;
+ if (pci_required_resource_failed(&failed, type)) {
+ ret = -ENOSPC;
+ goto cleanup;
+ }
+ /* Only resources with unrelated types failed (again) */
+ free_list(&failed);
}
list_for_each_entry(dev_res, &saved, list) {
base-commit: 19272b37aa4f83ca52bdf9c16d5d81bdd1354494
--
2.39.5
Ilpo Järvinen <ilpo.jarvinen@linux.intel.com> writes:
> On Wed, 18 Jun 2025, D Scott Phillips wrote:
>
>> Ilpo Järvinen <ilpo.jarvinen@linux.intel.com> writes:
>>
>> > Resetting resource is problematic as it prevent attempting to allocate
>> > the resource later, unless something in between restores the resource.
>> > Similarly, if fail_head does not contain all resources that were reset,
>> > those resource cannot be restored later.
>> >
>> > The entire reset/restore cycle adds complexity and leaving resources
>> > into reseted state causes issues to other code such as for checks done
>> > in pci_enable_resources(). Take a small step towards not resetting
>> > resources by delaying reset until the end of resource assignment and
>> > build failure list (fail_head) in sync with the reset to avoid leaving
>> > behind resources that cannot be restored (for the case where the caller
>> > provides fail_head in the first place to allow restore somewhere in the
>> > callchain, as is not all callers pass non-NULL fail_head).
>> >
>> > The Expansion ROM check is temporarily left in place while building the
>> > failure list until the upcoming change which reworks optional resource
>> > handling.
>> >
>> > Ideally, whole resource reset could be removed but doing that in a big
>> > step would make the impact non-tractable due to complexity of all
>> > related code.
>> >
>> > Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
>>
>> Hi Ilpo, I'm seeing a crash on arm64 at boot that I bisected to this
>> change. I don't think it's the same as the other issues reported. I've
>> confirmed the crash is still there after your follow up patches. The
>> crash itself is below[1].
>>
>> It looks like the problem begins when:
>>
>> amdgpu_device_resize_fb_bar()
>> pci_resize_resource()
>> pci_reassign_bridge_resources()
>> __pci_bus_size_bridges()
>>
>> adds pci_hotplug_io_size to `realloc_head`. The io resource allocation
>> has failed earlier because the root port doesn't have an io window[2].
>>
>> Then with this patch, pci_reassign_bridge_resources()'s call to
>> __pci_bridge_assign_resources() now returns the io added space for
>> hotplug in the `failed` list where the old code dropped it and did not.
>>
>> That sends pci_reassign_bridge_resources() into the `cleanup:` path,
>> where I think the cleanup code doesn't properly release the resources
>> that were assigned by __pci_bridge_assign_resources() and there's a
>> conflict reported in pci_claim_resource() where a restored resource is
>> found as conflicting with itself:
>>
>> > pcieport 000d:00:01.0: bridge window [mem 0x340000000000-0x340017ffffff 64bit pref]: can't claim; address conflict with PCI Bus 000d:01 [mem 0x340000000000-0x340017ffffff 64bit pref]
>>
>> Setting `pci=hpiosize=0` avoids this crash, as does this change:
>>
>> diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c
>> index 16d5d390599a..59ece11702da 100644
>> --- a/drivers/pci/setup-bus.c
>> +++ b/drivers/pci/setup-bus.c
>> @@ -2442,7 +2442,7 @@ int pci_reassign_bridge_resources(struct pci_dev *bridge, unsigned long type)
>> LIST_HEAD(saved);
>> LIST_HEAD(added);
>> LIST_HEAD(failed);
>> - unsigned int i;
>> + unsigned int i, relevant_fails;
>> int ret;
>>
>> down_read(&pci_bus_sem);
>> @@ -2490,7 +2490,16 @@ int pci_reassign_bridge_resources(struct pci_dev *bridge, unsigned long type)
>> __pci_bridge_assign_resources(bridge, &added, &failed);
>> BUG_ON(!list_empty(&added));
>>
>> - if (!list_empty(&failed)) {
>> + relevant_fails = 0;
>> + list_for_each_entry(dev_res, &failed, list) {
>> + restore_dev_resource(dev_res);
>> + if (((dev_res->res->flags ^ type) & PCI_RES_TYPE_MASK) == 0)
>> + relevant_fails++;
>> + }
>> + free_list(&failed);
>> +
>> + /* Cleanup if we had failures in resources of interest */
>> + if (relevant_fails != 0) {
>> ret = -ENOSPC;
>> goto cleanup;
>> }
>> @@ -2509,11 +2518,6 @@ int pci_reassign_bridge_resources(struct pci_dev *bridge, unsigned long type)
>> return 0;
>>
>> cleanup:
>> - /* Restore size and flags */
>> - list_for_each_entry(dev_res, &failed, list)
>> - restore_dev_resource(dev_res);
>> - free_list(&failed);
>> -
>> /* Revert to the old configuration */
>> list_for_each_entry(dev_res, &saved, list) {
>> struct resource *res = dev_res->res;
>>
>> I don't know this code well enough to know if that changes is completely
>> bonkers or what.
>
> Hi again,
>
> Thanks for all the details what you think went wrong, it was really
> useful. I think you have it towards the right direction but a more
> targetted seems enough to address this (this needs to be confirmed, please
> test the patch below).
>
> The most correct solution would be to make all the resource fitting code
> to focus on the resources that match the type filter. However, that looks
> way too scary change at the moment to implement, and especially, let it
> end up into stable (to fix this issue). So it looks this somewhat band-aid
> solution similar to your attempt might be better as a fix for now.
>
> In medium term, I'd want to avoid using type as a filter and base all
> such decisions on matching the bridge window resource the dev resource
> belongs to. I've some work towards that direction already which removes
> lots of complexity in which bridge window is going to be selected as
> there will be a single place to make always the same decision. That change
> is also going to simplify the internal interfaces between functions very
> noticably (but the change require more testing before I've enough
> confidence to submit it). That work doesn't cover this resize side yet but
> it should be extended there as well.
>
> So please test this somewhat band-aid patch:
>
> From 971686ed85e341e7234f8fe8b666140187f63ad1 Mon Sep 17 00:00:00 2001
> From: =?UTF-8?q?Ilpo=20J=C3=A4rvinen?= <ilpo.jarvinen@linux.intel.com>
> Date: Wed, 25 Jun 2025 20:30:43 +0300
> Subject: [PATCH 1/1] PCI: Fix failure detectiong during resource resize
detection
> MIME-Version: 1.0
> Content-Type: text/plain; charset=UTF-8
> Content-Transfer-Encoding: 8bit
>
> Since the commit 96336ec70264 ("PCI: Perform reset_resource() and build
> fail list in sync") the failed list is always built and returned to let
> the caller decide if what to do with the failures. The caller may want
> to retry resource fitting and assignment and before that can happen,
> the resources should be restored to their original state (a reset
> effectively clears the struct resource), which requires returning them
> on the failed list so that the original state remains stored in the
> associated struct pci_dev_resource.
>
> Resource resizing is different from the ordinary resource fitting and
> assignment in that it only considers part of the resources. This means
> failures for other resource types are not relevant at all and should be
> ignored. As resize doesn't unassign such unrelated resources, those
> resource ending up into the failed list implies assignment of that
> resource must have failed before resize too. The check in
> pci_reassign_bridge_resources() to decide if the whole assignment is
> successful, however, is based on list emptiness which may cause false
> negatives when the failed list resources with unrelated type.
>
> If the failed list is not empty, call pci_required_resource_failed()
> and extend it to be able to filter on specific resource types too (if
> provided).
>
> Calling pci_required_resource_failed() at this point is slightly
> problematic because the resource itself is reset when the failed list
> is constructed in __assign_resources_sorted(). As a result,
> pci_resource_is_optional() does not have access to the original
> resource flags. This could be worked around by restoring and
> re-reseting the resource around the call to pci_resource_is_optional(),
> however, it shouldn't cause issue as resource resizing is meant for
> 64-bit prefetchable resources according to Christian König (see the
> Link which unfortunately doesn't point directly to Christian's reply
> because lore didn't store that email at all).
>
> Link: https://lore.kernel.org/all/c5d1b5d8-8669-5572-75a7-0b480f581ac1@linux.intel.com/
> Reported-by: D Scott Phillips <scott@os.amperecomputing.com>
> Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
> Cc: Christian König <christian.koenig@amd.com>
> ---
> drivers/pci/setup-bus.c | 26 ++++++++++++++++++--------
> 1 file changed, 18 insertions(+), 8 deletions(-)
>
> diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c
> index 07c3d021a47e..8284bbdc44b4 100644
> --- a/drivers/pci/setup-bus.c
> +++ b/drivers/pci/setup-bus.c
> @@ -28,6 +28,10 @@
> #include <linux/acpi.h>
> #include "pci.h"
>
> +#define PCI_RES_TYPE_MASK \
> + (IORESOURCE_IO | IORESOURCE_MEM | IORESOURCE_PREFETCH |\
> + IORESOURCE_MEM_64)
> +
> unsigned int pci_flags;
> EXPORT_SYMBOL_GPL(pci_flags);
>
> @@ -384,13 +388,19 @@ static bool pci_need_to_release(unsigned long mask, struct resource *res)
> }
>
> /* Return: @true if assignment of a required resource failed. */
> -static bool pci_required_resource_failed(struct list_head *fail_head)
> +static bool pci_required_resource_failed(struct list_head *fail_head,
> + unsigned long type)
> {
> struct pci_dev_resource *fail_res;
>
> + type &= ~PCI_RES_TYPE_MASK;
Is this meant to be `type &= PCI_RES_TYPE_MASK`? If not, then I think
the new `if` check below is effectively just `if (type)`.
FWIW, the patch in the current state is fixing the problem that I've
been hitting.
> +
> list_for_each_entry(fail_res, fail_head, list) {
> int idx = pci_resource_num(fail_res->dev, fail_res->res);
>
> + if (type && (fail_res->flags & PCI_RES_TYPE_MASK) != type)
> + continue;
> +
> if (!pci_resource_is_optional(fail_res->dev, idx))
> return true;
> }
> @@ -504,7 +514,7 @@ static void __assign_resources_sorted(struct list_head *head,
> }
>
> /* Without realloc_head and only optional fails, nothing more to do. */
> - if (!pci_required_resource_failed(&local_fail_head) &&
> + if (!pci_required_resource_failed(&local_fail_head, 0) &&
> list_empty(realloc_head)) {
> list_for_each_entry(save_res, &save_head, list) {
> struct resource *res = save_res->res;
> @@ -1704,10 +1714,6 @@ static void __pci_bridge_assign_resources(const struct pci_dev *bridge,
> }
> }
>
> -#define PCI_RES_TYPE_MASK \
> - (IORESOURCE_IO | IORESOURCE_MEM | IORESOURCE_PREFETCH |\
> - IORESOURCE_MEM_64)
> -
> static void pci_bridge_release_resources(struct pci_bus *bus,
> unsigned long type)
> {
> @@ -2445,8 +2451,12 @@ int pci_reassign_bridge_resources(struct pci_dev *bridge, unsigned long type)
> free_list(&added);
>
> if (!list_empty(&failed)) {
> - ret = -ENOSPC;
> - goto cleanup;
> + if (pci_required_resource_failed(&failed, type)) {
> + ret = -ENOSPC;
> + goto cleanup;
> + }
> + /* Only resources with unrelated types failed (again) */
> + free_list(&failed);
> }
>
> list_for_each_entry(dev_res, &saved, list) {
>
> base-commit: 19272b37aa4f83ca52bdf9c16d5d81bdd1354494
> --
> 2.39.5
On Wed, 25 Jun 2025, D Scott Phillips wrote:
> Ilpo Järvinen <ilpo.jarvinen@linux.intel.com> writes:
>
> > On Wed, 18 Jun 2025, D Scott Phillips wrote:
> >
> >> Ilpo Järvinen <ilpo.jarvinen@linux.intel.com> writes:
> >>
> >> > Resetting resource is problematic as it prevent attempting to allocate
> >> > the resource later, unless something in between restores the resource.
> >> > Similarly, if fail_head does not contain all resources that were reset,
> >> > those resource cannot be restored later.
> >> >
> >> > The entire reset/restore cycle adds complexity and leaving resources
> >> > into reseted state causes issues to other code such as for checks done
> >> > in pci_enable_resources(). Take a small step towards not resetting
> >> > resources by delaying reset until the end of resource assignment and
> >> > build failure list (fail_head) in sync with the reset to avoid leaving
> >> > behind resources that cannot be restored (for the case where the caller
> >> > provides fail_head in the first place to allow restore somewhere in the
> >> > callchain, as is not all callers pass non-NULL fail_head).
> >> >
> >> > The Expansion ROM check is temporarily left in place while building the
> >> > failure list until the upcoming change which reworks optional resource
> >> > handling.
> >> >
> >> > Ideally, whole resource reset could be removed but doing that in a big
> >> > step would make the impact non-tractable due to complexity of all
> >> > related code.
> >> >
> >> > Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
> >>
> >> Hi Ilpo, I'm seeing a crash on arm64 at boot that I bisected to this
> >> change. I don't think it's the same as the other issues reported. I've
> >> confirmed the crash is still there after your follow up patches. The
> >> crash itself is below[1].
> >>
> >> It looks like the problem begins when:
> >>
> >> amdgpu_device_resize_fb_bar()
> >> pci_resize_resource()
> >> pci_reassign_bridge_resources()
> >> __pci_bus_size_bridges()
> >>
> >> adds pci_hotplug_io_size to `realloc_head`. The io resource allocation
> >> has failed earlier because the root port doesn't have an io window[2].
> >>
> >> Then with this patch, pci_reassign_bridge_resources()'s call to
> >> __pci_bridge_assign_resources() now returns the io added space for
> >> hotplug in the `failed` list where the old code dropped it and did not.
> >>
> >> That sends pci_reassign_bridge_resources() into the `cleanup:` path,
> >> where I think the cleanup code doesn't properly release the resources
> >> that were assigned by __pci_bridge_assign_resources() and there's a
> >> conflict reported in pci_claim_resource() where a restored resource is
> >> found as conflicting with itself:
> >>
> >> > pcieport 000d:00:01.0: bridge window [mem 0x340000000000-0x340017ffffff 64bit pref]: can't claim; address conflict with PCI Bus 000d:01 [mem 0x340000000000-0x340017ffffff 64bit pref]
> >>
> >> Setting `pci=hpiosize=0` avoids this crash, as does this change:
> >>
> >> diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c
> >> index 16d5d390599a..59ece11702da 100644
> >> --- a/drivers/pci/setup-bus.c
> >> +++ b/drivers/pci/setup-bus.c
> >> @@ -2442,7 +2442,7 @@ int pci_reassign_bridge_resources(struct pci_dev *bridge, unsigned long type)
> >> LIST_HEAD(saved);
> >> LIST_HEAD(added);
> >> LIST_HEAD(failed);
> >> - unsigned int i;
> >> + unsigned int i, relevant_fails;
> >> int ret;
> >>
> >> down_read(&pci_bus_sem);
> >> @@ -2490,7 +2490,16 @@ int pci_reassign_bridge_resources(struct pci_dev *bridge, unsigned long type)
> >> __pci_bridge_assign_resources(bridge, &added, &failed);
> >> BUG_ON(!list_empty(&added));
> >>
> >> - if (!list_empty(&failed)) {
> >> + relevant_fails = 0;
> >> + list_for_each_entry(dev_res, &failed, list) {
> >> + restore_dev_resource(dev_res);
> >> + if (((dev_res->res->flags ^ type) & PCI_RES_TYPE_MASK) == 0)
> >> + relevant_fails++;
> >> + }
> >> + free_list(&failed);
> >> +
> >> + /* Cleanup if we had failures in resources of interest */
> >> + if (relevant_fails != 0) {
> >> ret = -ENOSPC;
> >> goto cleanup;
> >> }
> >> @@ -2509,11 +2518,6 @@ int pci_reassign_bridge_resources(struct pci_dev *bridge, unsigned long type)
> >> return 0;
> >>
> >> cleanup:
> >> - /* Restore size and flags */
> >> - list_for_each_entry(dev_res, &failed, list)
> >> - restore_dev_resource(dev_res);
> >> - free_list(&failed);
> >> -
> >> /* Revert to the old configuration */
> >> list_for_each_entry(dev_res, &saved, list) {
> >> struct resource *res = dev_res->res;
> >>
> >> I don't know this code well enough to know if that changes is completely
> >> bonkers or what.
> >
> > Hi again,
> >
> > Thanks for all the details what you think went wrong, it was really
> > useful. I think you have it towards the right direction but a more
> > targetted seems enough to address this (this needs to be confirmed, please
> > test the patch below).
> >
> > The most correct solution would be to make all the resource fitting code
> > to focus on the resources that match the type filter. However, that looks
> > way too scary change at the moment to implement, and especially, let it
> > end up into stable (to fix this issue). So it looks this somewhat band-aid
> > solution similar to your attempt might be better as a fix for now.
> >
> > In medium term, I'd want to avoid using type as a filter and base all
> > such decisions on matching the bridge window resource the dev resource
> > belongs to. I've some work towards that direction already which removes
> > lots of complexity in which bridge window is going to be selected as
> > there will be a single place to make always the same decision. That change
> > is also going to simplify the internal interfaces between functions very
> > noticably (but the change require more testing before I've enough
> > confidence to submit it). That work doesn't cover this resize side yet but
> > it should be extended there as well.
> >
> > So please test this somewhat band-aid patch:
> >
> > From 971686ed85e341e7234f8fe8b666140187f63ad1 Mon Sep 17 00:00:00 2001
> > From: =?UTF-8?q?Ilpo=20J=C3=A4rvinen?= <ilpo.jarvinen@linux.intel.com>
> > Date: Wed, 25 Jun 2025 20:30:43 +0300
> > Subject: [PATCH 1/1] PCI: Fix failure detectiong during resource resize
>
> detection
>
> > MIME-Version: 1.0
> > Content-Type: text/plain; charset=UTF-8
> > Content-Transfer-Encoding: 8bit
> >
> > Since the commit 96336ec70264 ("PCI: Perform reset_resource() and build
> > fail list in sync") the failed list is always built and returned to let
> > the caller decide if what to do with the failures. The caller may want
> > to retry resource fitting and assignment and before that can happen,
> > the resources should be restored to their original state (a reset
> > effectively clears the struct resource), which requires returning them
> > on the failed list so that the original state remains stored in the
> > associated struct pci_dev_resource.
> >
> > Resource resizing is different from the ordinary resource fitting and
> > assignment in that it only considers part of the resources. This means
> > failures for other resource types are not relevant at all and should be
> > ignored. As resize doesn't unassign such unrelated resources, those
> > resource ending up into the failed list implies assignment of that
> > resource must have failed before resize too. The check in
> > pci_reassign_bridge_resources() to decide if the whole assignment is
> > successful, however, is based on list emptiness which may cause false
> > negatives when the failed list resources with unrelated type.
> >
> > If the failed list is not empty, call pci_required_resource_failed()
> > and extend it to be able to filter on specific resource types too (if
> > provided).
> >
> > Calling pci_required_resource_failed() at this point is slightly
> > problematic because the resource itself is reset when the failed list
> > is constructed in __assign_resources_sorted(). As a result,
> > pci_resource_is_optional() does not have access to the original
> > resource flags. This could be worked around by restoring and
> > re-reseting the resource around the call to pci_resource_is_optional(),
> > however, it shouldn't cause issue as resource resizing is meant for
> > 64-bit prefetchable resources according to Christian König (see the
> > Link which unfortunately doesn't point directly to Christian's reply
> > because lore didn't store that email at all).
> >
> > Link: https://lore.kernel.org/all/c5d1b5d8-8669-5572-75a7-0b480f581ac1@linux.intel.com/
> > Reported-by: D Scott Phillips <scott@os.amperecomputing.com>
> > Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
> > Cc: Christian König <christian.koenig@amd.com>
> > ---
> > drivers/pci/setup-bus.c | 26 ++++++++++++++++++--------
> > 1 file changed, 18 insertions(+), 8 deletions(-)
> >
> > diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c
> > index 07c3d021a47e..8284bbdc44b4 100644
> > --- a/drivers/pci/setup-bus.c
> > +++ b/drivers/pci/setup-bus.c
> > @@ -28,6 +28,10 @@
> > #include <linux/acpi.h>
> > #include "pci.h"
> >
> > +#define PCI_RES_TYPE_MASK \
> > + (IORESOURCE_IO | IORESOURCE_MEM | IORESOURCE_PREFETCH |\
> > + IORESOURCE_MEM_64)
> > +
> > unsigned int pci_flags;
> > EXPORT_SYMBOL_GPL(pci_flags);
> >
> > @@ -384,13 +388,19 @@ static bool pci_need_to_release(unsigned long mask, struct resource *res)
> > }
> >
> > /* Return: @true if assignment of a required resource failed. */
> > -static bool pci_required_resource_failed(struct list_head *fail_head)
> > +static bool pci_required_resource_failed(struct list_head *fail_head,
> > + unsigned long type)
> > {
> > struct pci_dev_resource *fail_res;
> >
> > + type &= ~PCI_RES_TYPE_MASK;
>
> Is this meant to be `type &= PCI_RES_TYPE_MASK`? If not, then I think
> the new `if` check below is effectively just `if (type)`.
Yes, it should have been without that ~. Can you test the change with
that changed? I'm sorry about the extra trouble.
> FWIW, the patch in the current state is fixing the problem that I've
> been hitting.
>
> > +
> > list_for_each_entry(fail_res, fail_head, list) {
> > int idx = pci_resource_num(fail_res->dev, fail_res->res);
> >
> > + if (type && (fail_res->flags & PCI_RES_TYPE_MASK) != type)
> > + continue;
> > +
> > if (!pci_resource_is_optional(fail_res->dev, idx))
> > return true;
> > }
> > @@ -504,7 +514,7 @@ static void __assign_resources_sorted(struct list_head *head,
> > }
> >
> > /* Without realloc_head and only optional fails, nothing more to do. */
> > - if (!pci_required_resource_failed(&local_fail_head) &&
> > + if (!pci_required_resource_failed(&local_fail_head, 0) &&
> > list_empty(realloc_head)) {
> > list_for_each_entry(save_res, &save_head, list) {
> > struct resource *res = save_res->res;
> > @@ -1704,10 +1714,6 @@ static void __pci_bridge_assign_resources(const struct pci_dev *bridge,
> > }
> > }
> >
> > -#define PCI_RES_TYPE_MASK \
> > - (IORESOURCE_IO | IORESOURCE_MEM | IORESOURCE_PREFETCH |\
> > - IORESOURCE_MEM_64)
> > -
> > static void pci_bridge_release_resources(struct pci_bus *bus,
> > unsigned long type)
> > {
> > @@ -2445,8 +2451,12 @@ int pci_reassign_bridge_resources(struct pci_dev *bridge, unsigned long type)
> > free_list(&added);
> >
> > if (!list_empty(&failed)) {
> > - ret = -ENOSPC;
> > - goto cleanup;
> > + if (pci_required_resource_failed(&failed, type)) {
> > + ret = -ENOSPC;
> > + goto cleanup;
> > + }
> > + /* Only resources with unrelated types failed (again) */
> > + free_list(&failed);
> > }
> >
> > list_for_each_entry(dev_res, &saved, list) {
> >
> > base-commit: 19272b37aa4f83ca52bdf9c16d5d81bdd1354494
> > --
> > 2.39.5
>
--
i.
Ilpo Järvinen <ilpo.jarvinen@linux.intel.com> writes:
> On Wed, 25 Jun 2025, D Scott Phillips wrote:
>
>> Ilpo Järvinen <ilpo.jarvinen@linux.intel.com> writes:
>>
>> > On Wed, 18 Jun 2025, D Scott Phillips wrote:
>> >
>> >> Ilpo Järvinen <ilpo.jarvinen@linux.intel.com> writes:
>> >>
>> >> > Resetting resource is problematic as it prevent attempting to allocate
>> >> > the resource later, unless something in between restores the resource.
>> >> > Similarly, if fail_head does not contain all resources that were reset,
>> >> > those resource cannot be restored later.
>> >> >
>> >> > The entire reset/restore cycle adds complexity and leaving resources
>> >> > into reseted state causes issues to other code such as for checks done
>> >> > in pci_enable_resources(). Take a small step towards not resetting
>> >> > resources by delaying reset until the end of resource assignment and
>> >> > build failure list (fail_head) in sync with the reset to avoid leaving
>> >> > behind resources that cannot be restored (for the case where the caller
>> >> > provides fail_head in the first place to allow restore somewhere in the
>> >> > callchain, as is not all callers pass non-NULL fail_head).
>> >> >
>> >> > The Expansion ROM check is temporarily left in place while building the
>> >> > failure list until the upcoming change which reworks optional resource
>> >> > handling.
>> >> >
>> >> > Ideally, whole resource reset could be removed but doing that in a big
>> >> > step would make the impact non-tractable due to complexity of all
>> >> > related code.
>> >> >
>> >> > Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
>> >>
>> >> Hi Ilpo, I'm seeing a crash on arm64 at boot that I bisected to this
>> >> change. I don't think it's the same as the other issues reported. I've
>> >> confirmed the crash is still there after your follow up patches. The
>> >> crash itself is below[1].
>> >>
>> >> It looks like the problem begins when:
>> >>
>> >> amdgpu_device_resize_fb_bar()
>> >> pci_resize_resource()
>> >> pci_reassign_bridge_resources()
>> >> __pci_bus_size_bridges()
>> >>
>> >> adds pci_hotplug_io_size to `realloc_head`. The io resource allocation
>> >> has failed earlier because the root port doesn't have an io window[2].
>> >>
>> >> Then with this patch, pci_reassign_bridge_resources()'s call to
>> >> __pci_bridge_assign_resources() now returns the io added space for
>> >> hotplug in the `failed` list where the old code dropped it and did not.
>> >>
>> >> That sends pci_reassign_bridge_resources() into the `cleanup:` path,
>> >> where I think the cleanup code doesn't properly release the resources
>> >> that were assigned by __pci_bridge_assign_resources() and there's a
>> >> conflict reported in pci_claim_resource() where a restored resource is
>> >> found as conflicting with itself:
>> >>
>> >> > pcieport 000d:00:01.0: bridge window [mem 0x340000000000-0x340017ffffff 64bit pref]: can't claim; address conflict with PCI Bus 000d:01 [mem 0x340000000000-0x340017ffffff 64bit pref]
>> >>
>> >> Setting `pci=hpiosize=0` avoids this crash, as does this change:
>> >>
>> >> diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c
>> >> index 16d5d390599a..59ece11702da 100644
>> >> --- a/drivers/pci/setup-bus.c
>> >> +++ b/drivers/pci/setup-bus.c
>> >> @@ -2442,7 +2442,7 @@ int pci_reassign_bridge_resources(struct pci_dev *bridge, unsigned long type)
>> >> LIST_HEAD(saved);
>> >> LIST_HEAD(added);
>> >> LIST_HEAD(failed);
>> >> - unsigned int i;
>> >> + unsigned int i, relevant_fails;
>> >> int ret;
>> >>
>> >> down_read(&pci_bus_sem);
>> >> @@ -2490,7 +2490,16 @@ int pci_reassign_bridge_resources(struct pci_dev *bridge, unsigned long type)
>> >> __pci_bridge_assign_resources(bridge, &added, &failed);
>> >> BUG_ON(!list_empty(&added));
>> >>
>> >> - if (!list_empty(&failed)) {
>> >> + relevant_fails = 0;
>> >> + list_for_each_entry(dev_res, &failed, list) {
>> >> + restore_dev_resource(dev_res);
>> >> + if (((dev_res->res->flags ^ type) & PCI_RES_TYPE_MASK) == 0)
>> >> + relevant_fails++;
>> >> + }
>> >> + free_list(&failed);
>> >> +
>> >> + /* Cleanup if we had failures in resources of interest */
>> >> + if (relevant_fails != 0) {
>> >> ret = -ENOSPC;
>> >> goto cleanup;
>> >> }
>> >> @@ -2509,11 +2518,6 @@ int pci_reassign_bridge_resources(struct pci_dev *bridge, unsigned long type)
>> >> return 0;
>> >>
>> >> cleanup:
>> >> - /* Restore size and flags */
>> >> - list_for_each_entry(dev_res, &failed, list)
>> >> - restore_dev_resource(dev_res);
>> >> - free_list(&failed);
>> >> -
>> >> /* Revert to the old configuration */
>> >> list_for_each_entry(dev_res, &saved, list) {
>> >> struct resource *res = dev_res->res;
>> >>
>> >> I don't know this code well enough to know if that changes is completely
>> >> bonkers or what.
>> >
>> > Hi again,
>> >
>> > Thanks for all the details what you think went wrong, it was really
>> > useful. I think you have it towards the right direction but a more
>> > targetted seems enough to address this (this needs to be confirmed, please
>> > test the patch below).
>> >
>> > The most correct solution would be to make all the resource fitting code
>> > to focus on the resources that match the type filter. However, that looks
>> > way too scary change at the moment to implement, and especially, let it
>> > end up into stable (to fix this issue). So it looks this somewhat band-aid
>> > solution similar to your attempt might be better as a fix for now.
>> >
>> > In medium term, I'd want to avoid using type as a filter and base all
>> > such decisions on matching the bridge window resource the dev resource
>> > belongs to. I've some work towards that direction already which removes
>> > lots of complexity in which bridge window is going to be selected as
>> > there will be a single place to make always the same decision. That change
>> > is also going to simplify the internal interfaces between functions very
>> > noticably (but the change require more testing before I've enough
>> > confidence to submit it). That work doesn't cover this resize side yet but
>> > it should be extended there as well.
>> >
>> > So please test this somewhat band-aid patch:
>> >
>> > From 971686ed85e341e7234f8fe8b666140187f63ad1 Mon Sep 17 00:00:00 2001
>> > From: =?UTF-8?q?Ilpo=20J=C3=A4rvinen?= <ilpo.jarvinen@linux.intel.com>
>> > Date: Wed, 25 Jun 2025 20:30:43 +0300
>> > Subject: [PATCH 1/1] PCI: Fix failure detectiong during resource resize
>>
>> detection
>>
>> > MIME-Version: 1.0
>> > Content-Type: text/plain; charset=UTF-8
>> > Content-Transfer-Encoding: 8bit
>> >
>> > Since the commit 96336ec70264 ("PCI: Perform reset_resource() and build
>> > fail list in sync") the failed list is always built and returned to let
>> > the caller decide if what to do with the failures. The caller may want
>> > to retry resource fitting and assignment and before that can happen,
>> > the resources should be restored to their original state (a reset
>> > effectively clears the struct resource), which requires returning them
>> > on the failed list so that the original state remains stored in the
>> > associated struct pci_dev_resource.
>> >
>> > Resource resizing is different from the ordinary resource fitting and
>> > assignment in that it only considers part of the resources. This means
>> > failures for other resource types are not relevant at all and should be
>> > ignored. As resize doesn't unassign such unrelated resources, those
>> > resource ending up into the failed list implies assignment of that
>> > resource must have failed before resize too. The check in
>> > pci_reassign_bridge_resources() to decide if the whole assignment is
>> > successful, however, is based on list emptiness which may cause false
>> > negatives when the failed list resources with unrelated type.
>> >
>> > If the failed list is not empty, call pci_required_resource_failed()
>> > and extend it to be able to filter on specific resource types too (if
>> > provided).
>> >
>> > Calling pci_required_resource_failed() at this point is slightly
>> > problematic because the resource itself is reset when the failed list
>> > is constructed in __assign_resources_sorted(). As a result,
>> > pci_resource_is_optional() does not have access to the original
>> > resource flags. This could be worked around by restoring and
>> > re-reseting the resource around the call to pci_resource_is_optional(),
>> > however, it shouldn't cause issue as resource resizing is meant for
>> > 64-bit prefetchable resources according to Christian König (see the
>> > Link which unfortunately doesn't point directly to Christian's reply
>> > because lore didn't store that email at all).
>> >
>> > Link: https://lore.kernel.org/all/c5d1b5d8-8669-5572-75a7-0b480f581ac1@linux.intel.com/
>> > Reported-by: D Scott Phillips <scott@os.amperecomputing.com>
>> > Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
>> > Cc: Christian König <christian.koenig@amd.com>
>> > ---
>> > drivers/pci/setup-bus.c | 26 ++++++++++++++++++--------
>> > 1 file changed, 18 insertions(+), 8 deletions(-)
>> >
>> > diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c
>> > index 07c3d021a47e..8284bbdc44b4 100644
>> > --- a/drivers/pci/setup-bus.c
>> > +++ b/drivers/pci/setup-bus.c
>> > @@ -28,6 +28,10 @@
>> > #include <linux/acpi.h>
>> > #include "pci.h"
>> >
>> > +#define PCI_RES_TYPE_MASK \
>> > + (IORESOURCE_IO | IORESOURCE_MEM | IORESOURCE_PREFETCH |\
>> > + IORESOURCE_MEM_64)
>> > +
>> > unsigned int pci_flags;
>> > EXPORT_SYMBOL_GPL(pci_flags);
>> >
>> > @@ -384,13 +388,19 @@ static bool pci_need_to_release(unsigned long mask, struct resource *res)
>> > }
>> >
>> > /* Return: @true if assignment of a required resource failed. */
>> > -static bool pci_required_resource_failed(struct list_head *fail_head)
>> > +static bool pci_required_resource_failed(struct list_head *fail_head,
>> > + unsigned long type)
>> > {
>> > struct pci_dev_resource *fail_res;
>> >
>> > + type &= ~PCI_RES_TYPE_MASK;
>>
>> Is this meant to be `type &= PCI_RES_TYPE_MASK`? If not, then I think
>> the new `if` check below is effectively just `if (type)`.
>
> Yes, it should have been without that ~. Can you test the change with
> that changed? I'm sorry about the extra trouble.
Hi Ilpo, no trouble at all, and thanks for your effort in fixing this
case. With that change to ~, the patch keeps working for my case.
Tested-by: D Scott Phillips <scott@os.amperecomputing.com>
Reviewed-by: D Scott Phillips <scott@os.amperecomputing.com>
On Wed, 18 Jun 2025, D Scott Phillips wrote:
> Ilpo Järvinen <ilpo.jarvinen@linux.intel.com> writes:
>
> > Resetting resource is problematic as it prevent attempting to allocate
> > the resource later, unless something in between restores the resource.
> > Similarly, if fail_head does not contain all resources that were reset,
> > those resource cannot be restored later.
> >
> > The entire reset/restore cycle adds complexity and leaving resources
> > into reseted state causes issues to other code such as for checks done
> > in pci_enable_resources(). Take a small step towards not resetting
> > resources by delaying reset until the end of resource assignment and
> > build failure list (fail_head) in sync with the reset to avoid leaving
> > behind resources that cannot be restored (for the case where the caller
> > provides fail_head in the first place to allow restore somewhere in the
> > callchain, as is not all callers pass non-NULL fail_head).
> >
> > The Expansion ROM check is temporarily left in place while building the
> > failure list until the upcoming change which reworks optional resource
> > handling.
> >
> > Ideally, whole resource reset could be removed but doing that in a big
> > step would make the impact non-tractable due to complexity of all
> > related code.
> >
> > Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
>
> Hi Ilpo, I'm seeing a crash on arm64 at boot that I bisected to this
> change. I don't think it's the same as the other issues reported. I've
> confirmed the crash is still there after your follow up patches. The
> crash itself is below[1].
Thanks for the report. I'm sorry about the delay, I was away a few days
(and hoping all the gremlins in that resource change would already have
been finally solved but it seems my hopes were premature).
> It looks like the problem begins when:
>
> amdgpu_device_resize_fb_bar()
> pci_resize_resource()
> pci_reassign_bridge_resources()
> __pci_bus_size_bridges()
>
> adds pci_hotplug_io_size to `realloc_head`. The io resource allocation
> has failed earlier because the root port doesn't have an io window[2].
>
> Then with this patch, pci_reassign_bridge_resources()'s call to
> __pci_bridge_assign_resources() now returns the io added space for
> hotplug in the `failed` list where the old code dropped it and did not.
>
> That sends pci_reassign_bridge_resources() into the `cleanup:` path,
> where I think the cleanup code doesn't properly release the resources
> that were assigned by __pci_bridge_assign_resources() and there's a
> conflict reported in pci_claim_resource() where a restored resource is
> found as conflicting with itself:
>
> > pcieport 000d:00:01.0: bridge window [mem 0x340000000000-0x340017ffffff 64bit pref]: can't claim; address conflict with PCI Bus 000d:01 [mem 0x340000000000-0x340017ffffff 64bit pref]
>
> Setting `pci=hpiosize=0` avoids this crash, as does this change:
>
> diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c
> index 16d5d390599a..59ece11702da 100644
> --- a/drivers/pci/setup-bus.c
> +++ b/drivers/pci/setup-bus.c
> @@ -2442,7 +2442,7 @@ int pci_reassign_bridge_resources(struct pci_dev *bridge, unsigned long type)
> LIST_HEAD(saved);
> LIST_HEAD(added);
> LIST_HEAD(failed);
> - unsigned int i;
> + unsigned int i, relevant_fails;
> int ret;
>
> down_read(&pci_bus_sem);
> @@ -2490,7 +2490,16 @@ int pci_reassign_bridge_resources(struct pci_dev *bridge, unsigned long type)
> __pci_bridge_assign_resources(bridge, &added, &failed);
> BUG_ON(!list_empty(&added));
>
> - if (!list_empty(&failed)) {
> + relevant_fails = 0;
> + list_for_each_entry(dev_res, &failed, list) {
> + restore_dev_resource(dev_res);
> + if (((dev_res->res->flags ^ type) & PCI_RES_TYPE_MASK) == 0)
> + relevant_fails++;
> + }
> + free_list(&failed);
> +
> + /* Cleanup if we had failures in resources of interest */
> + if (relevant_fails != 0) {
> ret = -ENOSPC;
> goto cleanup;
> }
> @@ -2509,11 +2518,6 @@ int pci_reassign_bridge_resources(struct pci_dev *bridge, unsigned long type)
> return 0;
>
> cleanup:
> - /* Restore size and flags */
> - list_for_each_entry(dev_res, &failed, list)
> - restore_dev_resource(dev_res);
> - free_list(&failed);
> -
> /* Revert to the old configuration */
> list_for_each_entry(dev_res, &saved, list) {
> struct resource *res = dev_res->res;
>
> I don't know this code well enough to know if that changes is completely
> bonkers or what.
It will take me a bit of time to assimilate all the information too, this
code has astronomical complexity and a failure due to resize is yet
another angle to it. I'll get back to you once I've figured more out.
> Maybe a change that gets zero as pci_hotplug_io_size
> for root ports that don't have an io window would be better?
Similar to this, I personally don't like having a non-zero IO window for
the cases where there is no io ports required for any devices. On dual
socket systems here, it always leads to exhausting the io port space and
the later probed devices that would actually have io port resources will
not get them. While those failures are not critical/important in my case,
it seems pointless to allocate IO window just for that sake of potential
hotplug, hotplug could just get it when there's a true need for the IO
window (and nothing bogusly reserved all io port space).
So IMO, a patch to change the default hotplug IO window to zero would
make sense which effectively matches what you provided on the cmdline. But
this is said before fully understanding what goes wrong on the resize
path, it may need corrections too. But regardless of that, I think zeroing
HP IO window is worthwhile on its own right to avoid exhausting the IO
port space for nothing.
> Any other
> ideas, or other information about the crash that I could provide?
I suppose the actual crash itself is because the driver does make some
invalid assumption about the resources without checking the resources
first (I've not had time to locate and check the driver code yet).
> [1]: Crash:
> > SError Interrupt on CPU0, code 0x00000000be000411 -- SError
> > CPU: 0 UID: 0 PID: 9 Comm: kworker/0:0 Not tainted 6.15.2+ #16 PREEMPT(undef)
> > Hardware name: Adlink Ampere Altra Developer Platform/Ampere Altra Developer Platform, BIOS TianoCore 2.09.100.00 (SYS: 2.10.20221028) 04/30/2
> > Workqueue: events work_for_cpu_fn
> > pstate: 204000c9 (nzCv daIF +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
> > pc : _raw_spin_lock_irqsave+0x34/0xb0
> > lr : __wake_up+0x30/0x80
> > sp : ffff800080003e20
> > x29: ffff800080003e20 x28: ffff07ff808b0000 x27: 0000000000050260
> > x26: 0000000000000001 x25: ffffcc5decad6bd8 x24: ffffcc5decc8b5b8
> > x23: ffff080138a10000 x22: ffff080138a00000 x21: 0000000000000003
> > x20: ffff080138a14dc8 x19: 0000000000000000 x18: 006808018e775f03
> > x17: ffff3bc1330d2000 x16: ffffcc5e3a520ed8 x15: 8daad055c6e77021
> > x14: f231631cf9328575 x13: 0e51168d06a6cf91 x12: f5db8c23b764520c
> > x11: 0000000000000040 x10: 0000000000000000 x9 : ffffcc5e3a520f08
> > x8 : 0000000000002113 x7 : 0000000000000000 x6 : 0000000000000000
> > x5 : 0000000000000000 x4 : ffff800080003e08 x3 : 00000000000000c0
> > x2 : 0000000000000001 x1 : 0000000000000000 x0 : ffff080138a14dc8
> > Kernel panic - not syncing: Asynchronous SError Interrupt
> > CPU: 0 UID: 0 PID: 9 Comm: kworker/0:0 Not tainted 6.15.2+ #16 PREEMPT(undef)
> > Hardware name: Adlink Ampere Altra Developer Platform/Ampere Altra Developer Platform, BIOS TianoCore 2.09.100.00 (SYS: 2.10.20221028) 04/30/2
> > Workqueue: events work_for_cpu_fn
> > Call trace:
> > show_stack+0x30/0x98 (C)
> > dump_stack_lvl+0x7c/0xa0
> > dump_stack+0x18/0x2c
> > panic+0x184/0x3c0
> > nmi_panic+0x90/0xa0
> > arm64_serror_panic+0x6c/0x90
> > do_serror+0x58/0x60
> > el1h_64_error_handler+0x38/0x60
> > el1h_64_error+0x84/0x88
> > _raw_spin_lock_irqsave+0x34/0xb0 (P)
> > amdgpu_ih_process+0xf0/0x150 [amdgpu]
> > amdgpu_irq_handler+0x34/0xa0 [amdgpu]
> > __handle_irq_event_percpu+0x60/0x248
> > handle_irq_event+0x4c/0xc0
> > handle_fasteoi_irq+0xa0/0x1c8
> > handle_irq_desc+0x3c/0x68
> > generic_handle_domain_irq+0x24/0x40
> > __gic_handle_irq_from_irqson.isra.0+0x15c/0x260
> > gic_handle_irq+0x28/0x80
> > call_on_irq_stack+0x24/0x30
> > do_interrupt_handler+0x88/0xa0
> > el1_interrupt+0x44/0xd0
> > el1h_64_irq_handler+0x18/0x28
> > el1h_64_irq+0x84/0x88
> > amdgpu_device_rreg.part.0+0x4c/0x190 [amdgpu] (P)
> > amdgpu_device_rreg+0x24/0x40 [amdgpu]
> > psp_wait_for+0x88/0xd8 [amdgpu]
> > psp_v11_0_bootloader_load_component+0x164/0x1b0 [amdgpu]
> > psp_v11_0_bootloader_load_kdb+0x20/0x40 [amdgpu]
> > psp_hw_start+0x5c/0x580 [amdgpu]
> > psp_load_fw+0x9c/0x280 [amdgpu]
> > psp_hw_init+0x44/0xa0 [amdgpu]
> > amdgpu_device_fw_loading+0xf8/0x358 [amdgpu]
> > amdgpu_device_ip_init+0x684/0x990 [amdgpu]
> > amdgpu_device_init+0xba4/0x1038 [amdgpu]
> > amdgpu_driver_load_kms+0x20/0xb8 [amdgpu]
> > amdgpu_pci_probe+0x1f8/0x7f8 [amdgpu]
> > local_pci_probe+0x44/0xb0
> > work_for_cpu_fn+0x24/0x40
> > process_one_work+0x17c/0x410
> > worker_thread+0x254/0x388
> > kthread+0x10c/0x128
> > ret_from_fork+0x10/0x20
> > Kernel Offset: 0x4c5dba380000 from 0xffff800080000000
> > PHYS_OFFSET: 0x80000000
> > CPU features: 0x0800,000042e0,01202650,8241720b
> > Memory Limit: none
> > ---[ end Kernel panic - not syncing: Asynchronous SError Interrupt ]---
>
> [2]: boot snippet
> > ACPI: PCI Root Bridge [PCI1] (domain 000d [bus 00-ff])
> > acpi PNP0A08:01: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI EDR HPX-Type3]
> > acpi PNP0A08:01: _OSC: platform does not support [PCIeHotplug PME LTR DPC]
> > acpi PNP0A08:01: _OSC: OS now controls [PCIeCapability]
> > acpi PNP0A08:01: FADT indicates ASPM is unsupported, using BIOS configuration
> > acpi PNP0A08:01: MCFG quirk: ECAM at [mem 0x37fff0000000-0x37ffffffffff] for [bus 00-ff] with pci_32b_read_ops
> > acpi PNP0A08:01: ECAM area [mem 0x37fff0000000-0x37ffffffffff] reserved by PNP0C02:00
> > acpi PNP0A08:01: ECAM at [mem 0x37fff0000000-0x37ffffffffff] for [bus 00-ff]
> > PCI host bridge to bus 000d:00
> > pci_bus 000d:00: root bus resource [mem 0x50000000-0x5fffffff window]
> > pci_bus 000d:00: root bus resource [mem 0x340000000000-0x37ffdfffffff window]
> > pci_bus 000d:00: root bus resource [bus 00-ff]
> > pci 000d:00:00.0: [1def:e100] type 00 class 0x060000 conventional PCI endpoint
> > pci 000d:00:01.0: [1def:e101] type 01 class 0x060400 PCIe Root Port
> > pci 000d:00:01.0: PCI bridge to [bus 01-03]
> > pci 000d:00:01.0: bridge window [io 0xe000-0xefff]
> > pci 000d:00:01.0: bridge window [mem 0x50000000-0x502fffff]
> > pci 000d:00:01.0: bridge window [mem 0x340000000000-0x3400101fffff 64bit pref]
> > pci 000d:00:01.0: supports D1 D2
> > pci 000d:00:01.0: PME# supported from D0 D1 D3hot
> > ...
> > pci 000d:00:01.0: bridge window [mem 0x340000000000-0x340017ffffff 64bit pref]: assigned
> > pci 000d:00:01.0: bridge window [mem 0x50000000-0x502fffff]: assigned
> > pci 000d:00:01.0: bridge window [io size 0x1000]: can't assign; no space
> > pci 000d:00:01.0: bridge window [io size 0x1000]: failed to assign
> > pci 000d:00:01.0: bridge window [io size 0x1000]: can't assign; no space
> > pci 000d:00:01.0: bridge window [io size 0x1000]: failed to assign
>
--
i.
Hi! On 12/16/24 5:56 PM, Ilpo Järvinen wrote: > Resetting resource is problematic as it prevent attempting to allocate > the resource later, unless something in between restores the resource. > Similarly, if fail_head does not contain all resources that were reset, > those resource cannot be restored later. > > The entire reset/restore cycle adds complexity and leaving resources > into reseted state causes issues to other code such as for checks done > in pci_enable_resources(). Take a small step towards not resetting > resources by delaying reset until the end of resource assignment and > build failure list (fail_head) in sync with the reset to avoid leaving > behind resources that cannot be restored (for the case where the caller > provides fail_head in the first place to allow restore somewhere in the > callchain, as is not all callers pass non-NULL fail_head). > > The Expansion ROM check is temporarily left in place while building the > failure list until the upcoming change which reworks optional resource > handling. > > Ideally, whole resource reset could be removed but doing that in a big > step would make the impact non-tractable due to complexity of all > related code. > > Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com> I'm hitting the BUG_ON(!list_empty(&add_list)); in pci_assign_unassigned_bus_resources() [1] with 6.15-rc5 and the the pixel6 downstream pcie driver. I saw the thread where "a34d74877c66 PCI: Restore assigned resources fully after release" fixes things for some other cases, but it's not the case here. Reverting the following patches fixes the problem: a34d74877c66 PCI: Restore assigned resources fully after release 2499f5348431 PCI: Rework optional resource handling 96336ec70264 PCI: Perform reset_resource() and build fail list in sync In the working case the add_list list is empty throughout the entire body of pci_assign_unassigned_bus_resources(). In the failing case __pci_bus_size_bridges() leaves the add_list not empty and __pci_bus_assign_resources() does not consume the list, thus the BUG_ON. The failing case contains an extra print that's not shown when reverting the blamed commits: [ 13.951185][ T1101] pcieport 0000:00:00.0: bridge window [mem 0x00100000-0x001fffff] to [bus 01-ff] add_size 100000 add_align 100000 I've added some prints trying to describe the code path, see https://paste.ofcode.org/Aeu2YBpLztc49ZDw3uUJmd# Failing case: [ 13.944231][ T1101] pci 0000:01:00.0: [144d:a5a5] type 00 class 0x000000 PCIe Endpoint [ 13.944412][ T1101] pci 0000:01:00.0: BAR 0 [mem 0x00000000-0x000fffff 64bit] [ 13.944532][ T1101] pci 0000:01:00.0: ROM [mem 0x00000000-0x0000ffff pref] [ 13.944649][ T1101] pci 0000:01:00.0: enabling Extended Tags [ 13.944844][ T1101] pci 0000:01:00.0: PME# supported from D0 D3hot D3cold [ 13.945015][ T1101] pci 0000:01:00.0: 15.752 Gb/s available PCIe bandwidth, limited by 8.0 GT/s PCIe x2 link at 0000:00:00.0 (capable of 31.506 Gb/s with 16.0 GT/s PCIe x2 link) [ 13.950616][ T1101] __pci_bus_size_bridges: before pbus_size_mem. list empty? 1 [ 13.950784][ T1101] pbus_size_mem: 2. list empty? 1 [ 13.950886][ T1101] pbus_size_mem: 1 list empty? 0 [ 13.950982][ T1101] pbus_size_mem: 3. list empty? 0 [ 13.951082][ T1101] pbus_size_mem: 4. list empty? 0 [ 13.951185][ T1101] pcieport 0000:00:00.0: bridge window [mem 0x00100000-0x001fffff] to [bus 01-ff] add_size 100000 add_align 100000 [ 13.951448][ T1101] __pci_bus_size_bridges: after pbus_size_mem. list empty? 0 [ 13.951643][ T1101] pci_assign_unassigned_bus_resources: before __pci_bus_assign_resources -> list empty? 0 [ 13.951924][ T1101] pcieport 0000:00:00.0: bridge window [mem 0x40000000-0x401fffff]: assigned [ 13.952248][ T1101] pci_assign_unassigned_bus_resources: after __pci_bus_assign_resources -> list empty? 0 [ 13.952634][ T1101] ------------[ cut here ]------------ [ 13.952818][ T1101] kernel BUG at drivers/pci/setup-bus.c:2514! [ 13.953045][ T1101] Internal error: Oops - BUG: 00000000f2000800 [#1] SMP ... [ 13.976086][ T1101] Call trace: [ 13.976206][ T1101] pci_assign_unassigned_bus_resources+0x110/0x114 (P) [ 13.976462][ T1101] pci_rescan_bus+0x28/0x48 [ 13.976628][ T1101] exynos_pcie_rc_poweron Working case: [ 13.786961][ T1120] pci 0000:01:00.0: [144d:a5a5] type 00 class 0x000000 PCIe Endpoint [ 13.787136][ T1120] pci 0000:01:00.0: BAR 0 [mem 0x00000000-0x000fffff 64bit] [ 13.787280][ T1120] pci 0000:01:00.0: ROM [mem 0x00000000-0x0000ffff pref] [ 13.787541][ T1120] pci 0000:01:00.0: enabling Extended Tags [ 13.787808][ T1120] pci 0000:01:00.0: PME# supported from D0 D3hot D3cold [ 13.787988][ T1120] pci 0000:01:00.0: 15.752 Gb/s available PCIe bandwidth, limited by 8.0 GT/s PCIe x2 link at 0000:00:00.0 (capable of 31.506 Gb/s with 16.0 GT/s PCIe x2 link) [ 13.795279][ T1120] __pci_bus_size_bridges: before pbus_size_mem. list empty? 1 [ 13.795408][ T1120] pbus_size_mem: 2. list empty? 1 [ 13.795495][ T1120] pbus_size_mem: 2. list empty? 1 [ 13.795577][ T1120] __pci_bus_size_bridges: after pbus_size_mem. list empty? 1 [ 13.795692][ T1120] pci_assign_unassigned_bus_resources: before __pci_bus_assign_resources -> list empty? 1 [ 13.795849][ T1120] pcieport 0000:00:00.0: bridge window [mem 0x40000000-0x401fffff]: assigned [ 13.796072][ T1120] pci_assign_unassigned_bus_resources: after __pci_bus_assign_resources -> list empty? 1 [ 13.796662][ T1120] cpif: s5100_poweron_pcie: DBG: MSI sfr not set up, yet(s5100_pdev is NULL) [ 13.796666][ T1120] cpif: register_pcie: s51xx_pcie_init start Any hints are welcomed. Thanks, ta [1] https://web.git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/pci/setup-bus.c?h=v6.15-rc5#n2500
On Tue, 6 May 2025, Tudor Ambarus wrote: > Hi! > > On 12/16/24 5:56 PM, Ilpo Järvinen wrote: > > Resetting resource is problematic as it prevent attempting to allocate > > the resource later, unless something in between restores the resource. > > Similarly, if fail_head does not contain all resources that were reset, > > those resource cannot be restored later. > > > > The entire reset/restore cycle adds complexity and leaving resources > > into reseted state causes issues to other code such as for checks done > > in pci_enable_resources(). Take a small step towards not resetting > > resources by delaying reset until the end of resource assignment and > > build failure list (fail_head) in sync with the reset to avoid leaving > > behind resources that cannot be restored (for the case where the caller > > provides fail_head in the first place to allow restore somewhere in the > > callchain, as is not all callers pass non-NULL fail_head). > > > > The Expansion ROM check is temporarily left in place while building the > > failure list until the upcoming change which reworks optional resource > > handling. > > > > Ideally, whole resource reset could be removed but doing that in a big > > step would make the impact non-tractable due to complexity of all > > related code. > > > > Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com> > > I'm hitting the BUG_ON(!list_empty(&add_list)); in > pci_assign_unassigned_bus_resources() [1] with 6.15-rc5 and the the > pixel6 downstream pcie driver. > > I saw the thread where "a34d74877c66 PCI: Restore assigned resources > fully after release" fixes things for some other cases, but it's not the > case here. > > Reverting the following patches fixes the problem: > a34d74877c66 PCI: Restore assigned resources fully after release > 2499f5348431 PCI: Rework optional resource handling > 96336ec70264 PCI: Perform reset_resource() and build fail list in sync So it's confirmed that you needed to revert also this last commit 96336ec70264, not just the rework change? > In the working case the add_list list is empty throughout the entire > body of pci_assign_unassigned_bus_resources(). > > In the failing case __pci_bus_size_bridges() leaves the add_list not > empty and __pci_bus_assign_resources() does not consume the list, thus > the BUG_ON. The failing case contains an extra print that's not shown > when reverting the blamed commits: > [ 13.951185][ T1101] pcieport 0000:00:00.0: bridge window [mem > 0x00100000-0x001fffff] to [bus 01-ff] add_size 100000 add_align 100000 > > I've added some prints trying to describe the code path, see > https://paste.ofcode.org/Aeu2YBpLztc49ZDw3uUJmd# > > Failing case: > [ 13.944231][ T1101] pci 0000:01:00.0: [144d:a5a5] type 00 class > 0x000000 PCIe Endpoint > [ 13.944412][ T1101] pci 0000:01:00.0: BAR 0 [mem > 0x00000000-0x000fffff 64bit] > [ 13.944532][ T1101] pci 0000:01:00.0: ROM [mem 0x00000000-0x0000ffff > pref] > [ 13.944649][ T1101] pci 0000:01:00.0: enabling Extended Tags > [ 13.944844][ T1101] pci 0000:01:00.0: PME# supported from D0 D3hot D3cold > [ 13.945015][ T1101] pci 0000:01:00.0: 15.752 Gb/s available PCIe > bandwidth, limited by 8.0 GT/s PCIe x2 link at 0000:00:00.0 (capable of > 31.506 Gb/s with 16.0 GT/s PCIe x2 link) > [ 13.950616][ T1101] __pci_bus_size_bridges: before pbus_size_mem. > list empty? 1 > [ 13.950784][ T1101] pbus_size_mem: 2. list empty? 1 > [ 13.950886][ T1101] pbus_size_mem: 1 list empty? 0 > [ 13.950982][ T1101] pbus_size_mem: 3. list empty? 0 > [ 13.951082][ T1101] pbus_size_mem: 4. list empty? 0 > [ 13.951185][ T1101] pcieport 0000:00:00.0: bridge window [mem > 0x00100000-0x001fffff] to [bus 01-ff] add_size 100000 add_align 100000 > [ 13.951448][ T1101] __pci_bus_size_bridges: after pbus_size_mem. list > empty? 0 > [ 13.951643][ T1101] pci_assign_unassigned_bus_resources: before > __pci_bus_assign_resources -> list empty? 0 > [ 13.951924][ T1101] pcieport 0000:00:00.0: bridge window [mem > 0x40000000-0x401fffff]: assigned > [ 13.952248][ T1101] pci_assign_unassigned_bus_resources: after > __pci_bus_assign_resources -> list empty? 0 > [ 13.952634][ T1101] ------------[ cut here ]------------ > [ 13.952818][ T1101] kernel BUG at drivers/pci/setup-bus.c:2514! > [ 13.953045][ T1101] Internal error: Oops - BUG: 00000000f2000800 [#1] > SMP > ... > [ 13.976086][ T1101] Call trace: > [ 13.976206][ T1101] pci_assign_unassigned_bus_resources+0x110/0x114 (P) > [ 13.976462][ T1101] pci_rescan_bus+0x28/0x48 > [ 13.976628][ T1101] exynos_pcie_rc_poweron > > Working case: > [ 13.786961][ T1120] pci 0000:01:00.0: [144d:a5a5] type 00 class > 0x000000 PCIe Endpoint > [ 13.787136][ T1120] pci 0000:01:00.0: BAR 0 [mem > 0x00000000-0x000fffff 64bit] > [ 13.787280][ T1120] pci 0000:01:00.0: ROM [mem 0x00000000-0x0000ffff > pref] > [ 13.787541][ T1120] pci 0000:01:00.0: enabling Extended Tags > [ 13.787808][ T1120] pci 0000:01:00.0: PME# supported from D0 D3hot D3cold > [ 13.787988][ T1120] pci 0000:01:00.0: 15.752 Gb/s available PCIe > bandwidth, limited by 8.0 GT/s PCIe x2 link at 0000:00:00.0 (capable of > 31.506 Gb/s with 16.0 GT/s PCIe x2 link) > [ 13.795279][ T1120] __pci_bus_size_bridges: before pbus_size_mem. > list empty? 1 > [ 13.795408][ T1120] pbus_size_mem: 2. list empty? 1 > [ 13.795495][ T1120] pbus_size_mem: 2. list empty? 1 > [ 13.795577][ T1120] __pci_bus_size_bridges: after pbus_size_mem. list > empty? 1 > [ 13.795692][ T1120] pci_assign_unassigned_bus_resources: before > __pci_bus_assign_resources -> list empty? 1 > [ 13.795849][ T1120] pcieport 0000:00:00.0: bridge window [mem > 0x40000000-0x401fffff]: assigned > [ 13.796072][ T1120] pci_assign_unassigned_bus_resources: after > __pci_bus_assign_resources -> list empty? 1 > [ 13.796662][ T1120] cpif: s5100_poweron_pcie: DBG: MSI sfr not set > up, yet(s5100_pdev is NULL) > [ 13.796666][ T1120] cpif: register_pcie: s51xx_pcie_init start > > > Any hints are welcomed. Thanks, > ta Hi and thanks for the report. The interesting part occurs inside reassign_resources_sorted() where most items are eliminated from realloc_head by the list_del(). My guess is that somehow, the change in 96336ec70264 from !res->flags to the more complicated check somehow causes this. If the new check doesn't match and subsequently, no match is found from the head list, the loop will do continue and not remove the entry from realloc_head. But it's hard to confirm without knowing what that resources realloc_head contains. Perhaps if you print the resources that are processed around that part of the code in reassign_resources_sorted(), comparing the log from the reverted code with the non-working case might help to understand what is different there and why. To understand better what is in the head list, it would be also useful to know from which device the resources were added into the head list in pdev_sort_resources(). In any case, that BUG_ON() seems a bit drastic action for what might be just a single resource allocation failure so it should be downgraded to: if (WARN_ON(!list_empty(&add_list)) free_list(&add_list); ... or WARN_ON_ONCE(). -- i.
On 5/6/25 4:53 PM, Ilpo Järvinen wrote: > On Tue, 6 May 2025, Tudor Ambarus wrote: > >> Hi! >> >> On 12/16/24 5:56 PM, Ilpo Järvinen wrote: >>> Resetting resource is problematic as it prevent attempting to allocate >>> the resource later, unless something in between restores the resource. >>> Similarly, if fail_head does not contain all resources that were reset, >>> those resource cannot be restored later. >>> >>> The entire reset/restore cycle adds complexity and leaving resources >>> into reseted state causes issues to other code such as for checks done >>> in pci_enable_resources(). Take a small step towards not resetting >>> resources by delaying reset until the end of resource assignment and >>> build failure list (fail_head) in sync with the reset to avoid leaving >>> behind resources that cannot be restored (for the case where the caller >>> provides fail_head in the first place to allow restore somewhere in the >>> callchain, as is not all callers pass non-NULL fail_head). >>> >>> The Expansion ROM check is temporarily left in place while building the >>> failure list until the upcoming change which reworks optional resource >>> handling. >>> >>> Ideally, whole resource reset could be removed but doing that in a big >>> step would make the impact non-tractable due to complexity of all >>> related code. >>> >>> Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com> >> >> I'm hitting the BUG_ON(!list_empty(&add_list)); in >> pci_assign_unassigned_bus_resources() [1] with 6.15-rc5 and the the >> pixel6 downstream pcie driver. >> >> I saw the thread where "a34d74877c66 PCI: Restore assigned resources >> fully after release" fixes things for some other cases, but it's not the >> case here. >> >> Reverting the following patches fixes the problem: >> a34d74877c66 PCI: Restore assigned resources fully after release >> 2499f5348431 PCI: Rework optional resource handling >> 96336ec70264 PCI: Perform reset_resource() and build fail list in sync > > So it's confirmed that you needed to revert also this last commit > 96336ec70264, not just the rework change? I needed to revert 96336ec70264 as well otherwise the build fails. > >> In the working case the add_list list is empty throughout the entire >> body of pci_assign_unassigned_bus_resources(). >> >> In the failing case __pci_bus_size_bridges() leaves the add_list not >> empty and __pci_bus_assign_resources() does not consume the list, thus >> the BUG_ON. The failing case contains an extra print that's not shown >> when reverting the blamed commits: >> [ 13.951185][ T1101] pcieport 0000:00:00.0: bridge window [mem >> 0x00100000-0x001fffff] to [bus 01-ff] add_size 100000 add_align 100000 >> >> I've added some prints trying to describe the code path, see >> https://paste.ofcode.org/Aeu2YBpLztc49ZDw3uUJmd# >> >> Failing case: >> [ 13.944231][ T1101] pci 0000:01:00.0: [144d:a5a5] type 00 class >> 0x000000 PCIe Endpoint >> [ 13.944412][ T1101] pci 0000:01:00.0: BAR 0 [mem >> 0x00000000-0x000fffff 64bit] >> [ 13.944532][ T1101] pci 0000:01:00.0: ROM [mem 0x00000000-0x0000ffff >> pref] >> [ 13.944649][ T1101] pci 0000:01:00.0: enabling Extended Tags >> [ 13.944844][ T1101] pci 0000:01:00.0: PME# supported from D0 D3hot D3cold >> [ 13.945015][ T1101] pci 0000:01:00.0: 15.752 Gb/s available PCIe >> bandwidth, limited by 8.0 GT/s PCIe x2 link at 0000:00:00.0 (capable of >> 31.506 Gb/s with 16.0 GT/s PCIe x2 link) >> [ 13.950616][ T1101] __pci_bus_size_bridges: before pbus_size_mem. >> list empty? 1 >> [ 13.950784][ T1101] pbus_size_mem: 2. list empty? 1 >> [ 13.950886][ T1101] pbus_size_mem: 1 list empty? 0 >> [ 13.950982][ T1101] pbus_size_mem: 3. list empty? 0 >> [ 13.951082][ T1101] pbus_size_mem: 4. list empty? 0 >> [ 13.951185][ T1101] pcieport 0000:00:00.0: bridge window [mem >> 0x00100000-0x001fffff] to [bus 01-ff] add_size 100000 add_align 100000 >> [ 13.951448][ T1101] __pci_bus_size_bridges: after pbus_size_mem. list >> empty? 0 >> [ 13.951643][ T1101] pci_assign_unassigned_bus_resources: before >> __pci_bus_assign_resources -> list empty? 0 >> [ 13.951924][ T1101] pcieport 0000:00:00.0: bridge window [mem >> 0x40000000-0x401fffff]: assigned >> [ 13.952248][ T1101] pci_assign_unassigned_bus_resources: after >> __pci_bus_assign_resources -> list empty? 0 >> [ 13.952634][ T1101] ------------[ cut here ]------------ >> [ 13.952818][ T1101] kernel BUG at drivers/pci/setup-bus.c:2514! >> [ 13.953045][ T1101] Internal error: Oops - BUG: 00000000f2000800 [#1] >> SMP >> ... >> [ 13.976086][ T1101] Call trace: >> [ 13.976206][ T1101] pci_assign_unassigned_bus_resources+0x110/0x114 (P) >> [ 13.976462][ T1101] pci_rescan_bus+0x28/0x48 >> [ 13.976628][ T1101] exynos_pcie_rc_poweron >> >> Working case: >> [ 13.786961][ T1120] pci 0000:01:00.0: [144d:a5a5] type 00 class >> 0x000000 PCIe Endpoint >> [ 13.787136][ T1120] pci 0000:01:00.0: BAR 0 [mem >> 0x00000000-0x000fffff 64bit] >> [ 13.787280][ T1120] pci 0000:01:00.0: ROM [mem 0x00000000-0x0000ffff >> pref] >> [ 13.787541][ T1120] pci 0000:01:00.0: enabling Extended Tags >> [ 13.787808][ T1120] pci 0000:01:00.0: PME# supported from D0 D3hot D3cold >> [ 13.787988][ T1120] pci 0000:01:00.0: 15.752 Gb/s available PCIe >> bandwidth, limited by 8.0 GT/s PCIe x2 link at 0000:00:00.0 (capable of >> 31.506 Gb/s with 16.0 GT/s PCIe x2 link) >> [ 13.795279][ T1120] __pci_bus_size_bridges: before pbus_size_mem. >> list empty? 1 >> [ 13.795408][ T1120] pbus_size_mem: 2. list empty? 1 >> [ 13.795495][ T1120] pbus_size_mem: 2. list empty? 1 >> [ 13.795577][ T1120] __pci_bus_size_bridges: after pbus_size_mem. list >> empty? 1 >> [ 13.795692][ T1120] pci_assign_unassigned_bus_resources: before >> __pci_bus_assign_resources -> list empty? 1 >> [ 13.795849][ T1120] pcieport 0000:00:00.0: bridge window [mem >> 0x40000000-0x401fffff]: assigned >> [ 13.796072][ T1120] pci_assign_unassigned_bus_resources: after >> __pci_bus_assign_resources -> list empty? 1 >> [ 13.796662][ T1120] cpif: s5100_poweron_pcie: DBG: MSI sfr not set >> up, yet(s5100_pdev is NULL) >> [ 13.796666][ T1120] cpif: register_pcie: s51xx_pcie_init start >> >> >> Any hints are welcomed. Thanks, >> ta > > Hi and thanks for the report. Hi! Thanks for the help. I've been out of office for the last 2 weeks, sorry for the delayed reply. > > The interesting part occurs inside reassign_resources_sorted() where most > items are eliminated from realloc_head by the list_del(). > > My guess is that somehow, the change in 96336ec70264 from !res->flags > to the more complicated check somehow causes this. If the new check > doesn't match and subsequently, no match is found from the head list, the > loop will do continue and not remove the entry from realloc_head. I added a print right there and it seems it's something else. See below. > > But it's hard to confirm without knowing what that resources realloc_head > contains. Perhaps if you print the resources that are processed around > that part of the code in reassign_resources_sorted(), comparing the log > from the reverted code with the non-working case might help to understand > what is different there and why. To understand better what is in the head > list, it would be also useful to know from which device the resources were > added into the head list in pdev_sort_resources(). > I added the suggested prints (https://paste.ofcode.org/DgmZGGgS6D36nWEzmfCqMm) on top of v6.15 with the downstream PCIe pixel driver and I obtain the following. Note that all added prints contain "tudor" for differentiation. [ 15.211179][ T1107] pci 0001:01:00.0: [144d:a5a5] type 00 class 0x000000 PCIe Endpoint [ 15.212248][ T1107] pci 0001:01:00.0: BAR 0 [mem 0x00000000-0x000fffff 64bit] [ 15.212775][ T1107] pci 0001:01:00.0: ROM [mem 0x00000000-0x0000ffff pref] [ 15.213195][ T1107] pci 0001:01:00.0: enabling Extended Tags [ 15.213720][ T1107] pci 0001:01:00.0: PME# supported from D0 D3hot D3cold [ 15.214035][ T1107] pci 0001:01:00.0: 15.752 Gb/s available PCIe bandwidth, limited by 8.0 GT/s PCIe x2 link at 0001:00:00.0 (capable of 31.506 Gb/s with 16.0 GT/s PCIe x2 link) [ 15.222286][ T1107] pci 0001:01:00.0: tudor: 1: pbus_size_mem: BAR 0 [mem 0x00000000-0x000fffff 64bit] list empty? 1 [ 15.222813][ T1107] pci 0001:01:00.0: tudor: 1: pbus_size_mem: ROM [mem 0x00000000-0x0000ffff pref] list empty? 1 [ 15.224429][ T1107] pci 0001:01:00.0: tudor: 2: pbus_size_mem: ROM [mem 0x00000000-0x0000ffff pref] list empty? 0 [ 15.224750][ T1107] pcieport 0001:00:00.0: bridge window [mem 0x00100000-0x001fffff] to [bus 01-ff] add_size 100000 add_align 100000 [ 15.225393][ T1107] tudor : pci_assign_unassigned_bus_resources: before __pci_bus_assign_resources -> list empty? 0 [ 15.225594][ T1107] pcieport 0001:00:00.0: tudor: pdev_sort_resources: bridge window [mem 0x00100000-0x001fffff] resource added in head list [ 15.226078][ T1107] pcieport 0001:00:00.0: bridge window [mem 0x40000000-0x401fffff]: assigned [ 15.226419][ T1107] tudor : pci_assign_unassigned_bus_resources: after __pci_bus_assign_resources -> list empty? 0 [ 15.226442][ T1107] ------------[ cut here ]------------ [ 15.227587][ T1107] kernel BUG at drivers/pci/setup-bus.c:2522! [ 15.227813][ T1107] Internal error: Oops - BUG: 00000000f2000800 [#1] SMP ... [ 15.251570][ T1107] Call trace: [ 15.251690][ T1107] pci_assign_unassigned_bus_resources+0x110/0x114 (P) [ 15.251945][ T1107] pci_rescan_bus+0x28/0x48 I obtain the following output when using the same prints adapted (https://paste.ofcode.org/37w7RnKkPaCxyNhi5yhZPbZ) and with the blamed commits reverted: a34d74877c66 PCI: Restore assigned resources fully after release 2499f5348431 PCI: Rework optional resource handling 96336ec70264 PCI: Perform reset_resource() and build fail list in sync [ 15.200456][ T1102] pci 0000:01:00.0: [144d:a5a5] type 00 class 0x000000 PCIe Endpoint [ 15.200632][ T1102] pci 0000:01:00.0: BAR 0 [mem 0x00000000-0x000fffff 64bit] [ 15.200755][ T1102] pci 0000:01:00.0: ROM [mem 0x00000000-0x0000ffff pref] [ 15.200876][ T1102] pci 0000:01:00.0: enabling Extended Tags [ 15.201075][ T1102] pci 0000:01:00.0: PME# supported from D0 D3hot D3cold [ 15.201254][ T1102] pci 0000:01:00.0: 15.752 Gb/s available PCIe bandwidth, limited by 8.0 GT/s PCIe x2 link at 0000:00:00.0 (capable of 31.506 Gb/s with 16.0 GT/s PCIe x2 link) [ 15.206555][ T1102] pci 0000:01:00.0: tudor: 1: pbus_size_mem: BAR 0 [mem 0x00000000-0x000fffff 64bit] list empty? 1 [ 15.206737][ T1102] pci 0000:01:00.0: tudor: 1: pbus_size_mem: ROM [mem 0x00000000-0x0000ffff pref] list empty? 1 [ 15.206901][ T1102] tudor : pci_assign_unassigned_bus_resources: before __pci_bus_assign_resources -> list empty? 1 [ 15.207072][ T1102] pcieport 0000:00:00.0: tudor: pdev_sort_resources: bridge window [mem 0x00100000-0x002fffff] resource added in head list [ 15.207396][ T1102] pcieport 0000:00:00.0: bridge window [mem 0x40000000-0x401fffff]: assigned [ 15.208165][ T1102] tudor : pci_assign_unassigned_bus_resources: after __pci_bus_assign_resources -> list empty? 1 [ 15.208783][ T1102] cpif: s5100_poweron_pcie: DBG: MSI sfr not set up, yet(s5100_pdev is NULL) [ 15.208786][ T1102] cpif: register_pcie: s51xx_pcie_init start > In any case, that BUG_ON() seems a bit drastic action for what might be > just a single resource allocation failure so it should be downgraded to: > > if (WARN_ON(!list_empty(&add_list)) > free_list(&add_list); > > ... or WARN_ON_ONCE(). I saw your patch doing this, the phone now boots, but obviously I still see the WARN, so maybe there's still something to be fixed. Thanks! ta
On Wed, 28 May 2025, Tudor Ambarus wrote:
> On 5/6/25 4:53 PM, Ilpo Järvinen wrote:
> > On Tue, 6 May 2025, Tudor Ambarus wrote:
> >> On 12/16/24 5:56 PM, Ilpo Järvinen wrote:
> >>> Resetting resource is problematic as it prevent attempting to allocate
> >>> the resource later, unless something in between restores the resource.
> >>> Similarly, if fail_head does not contain all resources that were reset,
> >>> those resource cannot be restored later.
> >>>
> >>> The entire reset/restore cycle adds complexity and leaving resources
> >>> into reseted state causes issues to other code such as for checks done
> >>> in pci_enable_resources(). Take a small step towards not resetting
> >>> resources by delaying reset until the end of resource assignment and
> >>> build failure list (fail_head) in sync with the reset to avoid leaving
> >>> behind resources that cannot be restored (for the case where the caller
> >>> provides fail_head in the first place to allow restore somewhere in the
> >>> callchain, as is not all callers pass non-NULL fail_head).
> >>>
> >>> The Expansion ROM check is temporarily left in place while building the
> >>> failure list until the upcoming change which reworks optional resource
> >>> handling.
> >>>
> >>> Ideally, whole resource reset could be removed but doing that in a big
> >>> step would make the impact non-tractable due to complexity of all
> >>> related code.
> >>>
> >>> Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
> >>
> >> I'm hitting the BUG_ON(!list_empty(&add_list)); in
> >> pci_assign_unassigned_bus_resources() [1] with 6.15-rc5 and the the
> >> pixel6 downstream pcie driver.
> >>
> >> I saw the thread where "a34d74877c66 PCI: Restore assigned resources
> >> fully after release" fixes things for some other cases, but it's not the
> >> case here.
> >>
> >> Reverting the following patches fixes the problem:
> >> a34d74877c66 PCI: Restore assigned resources fully after release
> >> 2499f5348431 PCI: Rework optional resource handling
> >> 96336ec70264 PCI: Perform reset_resource() and build fail list in sync
> >
> > So it's confirmed that you needed to revert also this last commit
> > 96336ec70264, not just the rework change?
>
> I needed to revert 96336ec70264 as well otherwise the build fails.
Hi again,
That's news to me... I seem to have botched the resource assignment rework
series at some point when I reordered patches and dropped that helper as a
result. And it seems the intermediate build fail wasn't caught by LKP :-(.
(Pretty annoying as I intentionally separated these two to make them
bisectable but not it isn't without amends.)
The missing helper is basically this:
static bool pci_resource_is_disabled_rom(const struct pci_dev *dev, int resno)
{
const struct resource *res = pci_resource_n(dev, resno);
return resno == PCI_ROM_RESOURCE && !(res->flags & IORESOURCE_ROM_ENABLE)
}
(I didn't build test that.)
Because of this, the actual culprit could be in 2499f5348431, not it
96336ec70264 (which would make more sense as it does significant rework
on the assignment algorithm).
> >> In the working case the add_list list is empty throughout the entire
> >> body of pci_assign_unassigned_bus_resources().
> >>
> >> In the failing case __pci_bus_size_bridges() leaves the add_list not
> >> empty and __pci_bus_assign_resources() does not consume the list, thus
> >> the BUG_ON. The failing case contains an extra print that's not shown
> >> when reverting the blamed commits:
> >> [ 13.951185][ T1101] pcieport 0000:00:00.0: bridge window [mem
> >> 0x00100000-0x001fffff] to [bus 01-ff] add_size 100000 add_align 100000
> >>
> >> I've added some prints trying to describe the code path, see
> >> https://paste.ofcode.org/Aeu2YBpLztc49ZDw3uUJmd#
> >>
> >> Failing case:
> >> [ 13.944231][ T1101] pci 0000:01:00.0: [144d:a5a5] type 00 class
> >> 0x000000 PCIe Endpoint
> >> [ 13.944412][ T1101] pci 0000:01:00.0: BAR 0 [mem
> >> 0x00000000-0x000fffff 64bit]
> >> [ 13.944532][ T1101] pci 0000:01:00.0: ROM [mem 0x00000000-0x0000ffff
> >> pref]
> >> [ 13.944649][ T1101] pci 0000:01:00.0: enabling Extended Tags
> >> [ 13.944844][ T1101] pci 0000:01:00.0: PME# supported from D0 D3hot D3cold
> >> [ 13.945015][ T1101] pci 0000:01:00.0: 15.752 Gb/s available PCIe
> >> bandwidth, limited by 8.0 GT/s PCIe x2 link at 0000:00:00.0 (capable of
> >> 31.506 Gb/s with 16.0 GT/s PCIe x2 link)
> >> [ 13.950616][ T1101] __pci_bus_size_bridges: before pbus_size_mem.
> >> list empty? 1
> >> [ 13.950784][ T1101] pbus_size_mem: 2. list empty? 1
> >> [ 13.950886][ T1101] pbus_size_mem: 1 list empty? 0
> >> [ 13.950982][ T1101] pbus_size_mem: 3. list empty? 0
> >> [ 13.951082][ T1101] pbus_size_mem: 4. list empty? 0
> >> [ 13.951185][ T1101] pcieport 0000:00:00.0: bridge window [mem
> >> 0x00100000-0x001fffff] to [bus 01-ff] add_size 100000 add_align 100000
> >> [ 13.951448][ T1101] __pci_bus_size_bridges: after pbus_size_mem. list
> >> empty? 0
> >> [ 13.951643][ T1101] pci_assign_unassigned_bus_resources: before
> >> __pci_bus_assign_resources -> list empty? 0
> >> [ 13.951924][ T1101] pcieport 0000:00:00.0: bridge window [mem
> >> 0x40000000-0x401fffff]: assigned
> >> [ 13.952248][ T1101] pci_assign_unassigned_bus_resources: after
> >> __pci_bus_assign_resources -> list empty? 0
> >> [ 13.952634][ T1101] ------------[ cut here ]------------
> >> [ 13.952818][ T1101] kernel BUG at drivers/pci/setup-bus.c:2514!
> >> [ 13.953045][ T1101] Internal error: Oops - BUG: 00000000f2000800 [#1]
> >> SMP
> >> ...
> >> [ 13.976086][ T1101] Call trace:
> >> [ 13.976206][ T1101] pci_assign_unassigned_bus_resources+0x110/0x114 (P)
> >> [ 13.976462][ T1101] pci_rescan_bus+0x28/0x48
> >> [ 13.976628][ T1101] exynos_pcie_rc_poweron
> >>
> >> Working case:
> >> [ 13.786961][ T1120] pci 0000:01:00.0: [144d:a5a5] type 00 class
> >> 0x000000 PCIe Endpoint
> >> [ 13.787136][ T1120] pci 0000:01:00.0: BAR 0 [mem
> >> 0x00000000-0x000fffff 64bit]
> >> [ 13.787280][ T1120] pci 0000:01:00.0: ROM [mem 0x00000000-0x0000ffff
> >> pref]
> >> [ 13.787541][ T1120] pci 0000:01:00.0: enabling Extended Tags
> >> [ 13.787808][ T1120] pci 0000:01:00.0: PME# supported from D0 D3hot D3cold
> >> [ 13.787988][ T1120] pci 0000:01:00.0: 15.752 Gb/s available PCIe
> >> bandwidth, limited by 8.0 GT/s PCIe x2 link at 0000:00:00.0 (capable of
> >> 31.506 Gb/s with 16.0 GT/s PCIe x2 link)
> >> [ 13.795279][ T1120] __pci_bus_size_bridges: before pbus_size_mem.
> >> list empty? 1
> >> [ 13.795408][ T1120] pbus_size_mem: 2. list empty? 1
> >> [ 13.795495][ T1120] pbus_size_mem: 2. list empty? 1
> >> [ 13.795577][ T1120] __pci_bus_size_bridges: after pbus_size_mem. list
> >> empty? 1
> >> [ 13.795692][ T1120] pci_assign_unassigned_bus_resources: before
> >> __pci_bus_assign_resources -> list empty? 1
> >> [ 13.795849][ T1120] pcieport 0000:00:00.0: bridge window [mem
> >> 0x40000000-0x401fffff]: assigned
> >> [ 13.796072][ T1120] pci_assign_unassigned_bus_resources: after
> >> __pci_bus_assign_resources -> list empty? 1
> >> [ 13.796662][ T1120] cpif: s5100_poweron_pcie: DBG: MSI sfr not set
> >> up, yet(s5100_pdev is NULL)
> >> [ 13.796666][ T1120] cpif: register_pcie: s51xx_pcie_init start
> >>
> >>
> >> Any hints are welcomed. Thanks,
> >> ta
> >
> > Hi and thanks for the report.
>
> Hi! Thanks for the help. I've been out of office for the last 2 weeks,
> sorry for the delayed reply.
Np.
> > The interesting part occurs inside reassign_resources_sorted() where most
> > items are eliminated from realloc_head by the list_del().
> >
> > My guess is that somehow, the change in 96336ec70264 from !res->flags
> > to the more complicated check somehow causes this. If the new check
> > doesn't match and subsequently, no match is found from the head list, the
> > loop will do continue and not remove the entry from realloc_head.
>
> I added a print right there and it seems it's something else. See below.
> >
> > But it's hard to confirm without knowing what that resources realloc_head
> > contains. Perhaps if you print the resources that are processed around
> > that part of the code in reassign_resources_sorted(), comparing the log
> > from the reverted code with the non-working case might help to understand
> > what is different there and why. To understand better what is in the head
> > list, it would be also useful to know from which device the resources were
> > added into the head list in pdev_sort_resources().
> >
>
> I added the suggested prints
> (https://paste.ofcode.org/DgmZGGgS6D36nWEzmfCqMm) on top of v6.15 with
> the downstream PCIe pixel driver and I obtain the following. Note that
> all added prints contain "tudor" for differentiation.
>
> [ 15.211179][ T1107] pci 0001:01:00.0: [144d:a5a5] type 00 class
> 0x000000 PCIe Endpoint
> [ 15.212248][ T1107] pci 0001:01:00.0: BAR 0 [mem
> 0x00000000-0x000fffff 64bit]
> [ 15.212775][ T1107] pci 0001:01:00.0: ROM [mem 0x00000000-0x0000ffff
> pref]
> [ 15.213195][ T1107] pci 0001:01:00.0: enabling Extended Tags
> [ 15.213720][ T1107] pci 0001:01:00.0: PME# supported from D0 D3hot
> D3cold
> [ 15.214035][ T1107] pci 0001:01:00.0: 15.752 Gb/s available PCIe
> bandwidth, limited by 8.0 GT/s PCIe x2 link at 0001:00:00.0 (capable of
> 31.506 Gb/s with 16.0 GT/s PCIe x2 link)
> [ 15.222286][ T1107] pci 0001:01:00.0: tudor: 1: pbus_size_mem: BAR 0
> [mem 0x00000000-0x000fffff 64bit] list empty? 1
> [ 15.222813][ T1107] pci 0001:01:00.0: tudor: 1: pbus_size_mem: ROM
> [mem 0x00000000-0x0000ffff pref] list empty? 1
> [ 15.224429][ T1107] pci 0001:01:00.0: tudor: 2: pbus_size_mem: ROM
> [mem 0x00000000-0x0000ffff pref] list empty? 0
> [ 15.224750][ T1107] pcieport 0001:00:00.0: bridge window [mem
> 0x00100000-0x001fffff] to [bus 01-ff] add_size 100000 add_align 100000
>
> [ 15.225393][ T1107] tudor : pci_assign_unassigned_bus_resources:
> before __pci_bus_assign_resources -> list empty? 0
> [ 15.225594][ T1107] pcieport 0001:00:00.0: tudor:
> pdev_sort_resources: bridge window [mem 0x00100000-0x001fffff] resource
> added in head list
> [ 15.226078][ T1107] pcieport 0001:00:00.0: bridge window [mem
> 0x40000000-0x401fffff]: assigned
So here it ends up assigning the resource here I think.
That print isn't one of yours in reassign_resources_sorted() so the
assignment must have been made in assign_requested_resources_sorted(). But
then nothing is printed out from reassign_resources_sorted() so I suspect
__assign_resources_sorted() has short-circuited.
We know that realloc_head is not empty, so that leaves the goto out from
if (list_empty(&local_fail_head)), which kind of makes sense, all
entries on the head list were assigned. But the code there tries to remove
all head list resources from realloc_head so why it doesn't get removed is
still a mystery. assign_requested_resources_sorted() doesn't seem to
remove anything from the head list so that resource should still be on the
head list AFAICT so it should call that remove_from_list(realloc_head,
dev_res->res) for it.
So can you see if that theory holds water and it short-circuits without
removing the entry from realloc_head?
> [ 15.226419][ T1107] tudor : pci_assign_unassigned_bus_resources:
> after __pci_bus_assign_resources -> list empty? 0
> [ 15.226442][ T1107] ------------[ cut here ]------------
> [ 15.227587][ T1107] kernel BUG at drivers/pci/setup-bus.c:2522!
> [ 15.227813][ T1107] Internal error: Oops - BUG: 00000000f2000800 [#1]
> SMP
> ...
> [ 15.251570][ T1107] Call trace:
> [ 15.251690][ T1107] pci_assign_unassigned_bus_resources+0x110/0x114 (P)
> [ 15.251945][ T1107] pci_rescan_bus+0x28/0x48
>
> I obtain the following output when using the same prints adapted
> (https://paste.ofcode.org/37w7RnKkPaCxyNhi5yhZPbZ) and with the blamed
> commits reverted:
> a34d74877c66 PCI: Restore assigned resources fully after release
> 2499f5348431 PCI: Rework optional resource handling
> 96336ec70264 PCI: Perform reset_resource() and build fail list in sync
>
> [ 15.200456][ T1102] pci 0000:01:00.0: [144d:a5a5] type 00 class
> 0x000000 PCIe Endpoint
> [ 15.200632][ T1102] pci 0000:01:00.0: BAR 0 [mem
> 0x00000000-0x000fffff 64bit]
> [ 15.200755][ T1102] pci 0000:01:00.0: ROM [mem 0x00000000-0x0000ffff
> pref]
> [ 15.200876][ T1102] pci 0000:01:00.0: enabling Extended Tags
> [ 15.201075][ T1102] pci 0000:01:00.0: PME# supported from D0 D3hot D3cold
> [ 15.201254][ T1102] pci 0000:01:00.0: 15.752 Gb/s available PCIe
> bandwidth, limited by 8.0 GT/s PCIe x2 link at 0000:00:00.0 (capable of
> 31.506 Gb/s with 16.0 GT/s PCIe x2 link)
> [ 15.206555][ T1102] pci 0000:01:00.0: tudor: 1: pbus_size_mem: BAR 0
> [mem 0x00000000-0x000fffff 64bit] list empty? 1
> [ 15.206737][ T1102] pci 0000:01:00.0: tudor: 1: pbus_size_mem: ROM
> [mem 0x00000000-0x0000ffff pref] list empty? 1
> [ 15.206901][ T1102] tudor : pci_assign_unassigned_bus_resources:
> before __pci_bus_assign_resources -> list empty? 1
> [ 15.207072][ T1102] pcieport 0000:00:00.0: tudor:
> pdev_sort_resources: bridge window [mem 0x00100000-0x002fffff] resource
> added in head list
> [ 15.207396][ T1102] pcieport 0000:00:00.0: bridge window [mem
> 0x40000000-0x401fffff]: assigned
> [ 15.208165][ T1102] tudor : pci_assign_unassigned_bus_resources:
> after __pci_bus_assign_resources -> list empty? 1
> [ 15.208783][ T1102] cpif: s5100_poweron_pcie: DBG: MSI sfr not set
> up, yet(s5100_pdev is NULL)
> [ 15.208786][ T1102] cpif: register_pcie: s51xx_pcie_init start
>
> > In any case, that BUG_ON() seems a bit drastic action for what might be
> > just a single resource allocation failure so it should be downgraded to:
> >
> > if (WARN_ON(!list_empty(&add_list))
> > free_list(&add_list);
> >
> > ... or WARN_ON_ONCE().
>
> I saw your patch doing this, the phone now boots, but obviously I still
> see the WARN, so maybe there's still something to be fixed.
Yes, I don't expect BUG_ON() -> WARN "fix" anything, it just downgrades
the severity so that the system can still try to boot, which can often
succeed as this tends to be non-critical failure in many cases so it's
useful change to have regardless despite the splat.
Now that it boots, can you please check if /proc/iomem is the same both in
the non-working and working config. If that resource got assigned
successfully, it might well be there is no actual differences in the
assigned resources (which again doesn't mean there wouldn't be a bug in
the logic as discussed above).
--
i.
On 5/30/25 7:38 AM, Ilpo Järvinen wrote:
cut
>>>> Reverting the following patches fixes the problem:
>>>> a34d74877c66 PCI: Restore assigned resources fully after release
>>>> 2499f5348431 PCI: Rework optional resource handling
>>>> 96336ec70264 PCI: Perform reset_resource() and build fail list in sync
>>>
>>> So it's confirmed that you needed to revert also this last commit
>>> 96336ec70264, not just the rework change?
>>
>> I needed to revert 96336ec70264 as well otherwise the build fails.
>
> Hi again,
Hi!
cut
>
> The missing helper is basically this:
cut
I used the following:
+static bool pci_resource_is_disabled_rom(const struct resource *res,
int resno)
+{
+ return resno == PCI_ROM_RESOURCE && !(res->flags &
IORESOURCE_ROM_ENABLE);
+}
>
> Because of this, the actual culprit could be in 2499f5348431, not it
> 96336ec70264 (which would make more sense as it does significant rework
> on the assignment algorithm).
I confirm with the above that the problem is in 2499f5348431 indeed.
cut
>> I added the suggested prints
>> (https://paste.ofcode.org/DgmZGGgS6D36nWEzmfCqMm) on top of v6.15 with
>> the downstream PCIe pixel driver and I obtain the following. Note that
>> all added prints contain "tudor" for differentiation.
>>
>> [ 15.211179][ T1107] pci 0001:01:00.0: [144d:a5a5] type 00 class
>> 0x000000 PCIe Endpoint
>> [ 15.212248][ T1107] pci 0001:01:00.0: BAR 0 [mem
>> 0x00000000-0x000fffff 64bit]
>> [ 15.212775][ T1107] pci 0001:01:00.0: ROM [mem 0x00000000-0x0000ffff
>> pref]
>> [ 15.213195][ T1107] pci 0001:01:00.0: enabling Extended Tags
>> [ 15.213720][ T1107] pci 0001:01:00.0: PME# supported from D0 D3hot
>> D3cold
>> [ 15.214035][ T1107] pci 0001:01:00.0: 15.752 Gb/s available PCIe
>> bandwidth, limited by 8.0 GT/s PCIe x2 link at 0001:00:00.0 (capable of
>> 31.506 Gb/s with 16.0 GT/s PCIe x2 link)
>> [ 15.222286][ T1107] pci 0001:01:00.0: tudor: 1: pbus_size_mem: BAR 0
>> [mem 0x00000000-0x000fffff 64bit] list empty? 1
>> [ 15.222813][ T1107] pci 0001:01:00.0: tudor: 1: pbus_size_mem: ROM
>> [mem 0x00000000-0x0000ffff pref] list empty? 1
>> [ 15.224429][ T1107] pci 0001:01:00.0: tudor: 2: pbus_size_mem: ROM
>> [mem 0x00000000-0x0000ffff pref] list empty? 0
>> [ 15.224750][ T1107] pcieport 0001:00:00.0: bridge window [mem
>> 0x00100000-0x001fffff] to [bus 01-ff] add_size 100000 add_align 100000
>>
>> [ 15.225393][ T1107] tudor : pci_assign_unassigned_bus_resources:
>> before __pci_bus_assign_resources -> list empty? 0
>> [ 15.225594][ T1107] pcieport 0001:00:00.0: tudor:
>> pdev_sort_resources: bridge window [mem 0x00100000-0x001fffff] resource
>> added in head list
>> [ 15.226078][ T1107] pcieport 0001:00:00.0: bridge window [mem
>> 0x40000000-0x401fffff]: assigned
>
> So here it ends up assigning the resource here I think.
>
>
> That print isn't one of yours in reassign_resources_sorted() so the
> assignment must have been made in assign_requested_resources_sorted(). But
> then nothing is printed out from reassign_resources_sorted() so I suspect
> __assign_resources_sorted() has short-circuited.
>
> We know that realloc_head is not empty, so that leaves the goto out from
> if (list_empty(&local_fail_head)), which kind of makes sense, all
> entries on the head list were assigned. But the code there tries to remove
> all head list resources from realloc_head so why it doesn't get removed is
> still a mystery. assign_requested_resources_sorted() doesn't seem to
> remove anything from the head list so that resource should still be on the
> head list AFAICT so it should call that remove_from_list(realloc_head,
> dev_res->res) for it.
>
> So can you see if that theory holds water and it short-circuits without
> removing the entry from realloc_head?
>
cut. I saw your other reply. Will check a bit both and respond there
directly.
>>
>>> In any case, that BUG_ON() seems a bit drastic action for what might be
>>> just a single resource allocation failure so it should be downgraded to:
>>>
>>> if (WARN_ON(!list_empty(&add_list))
>>> free_list(&add_list);
>>>
>>> ... or WARN_ON_ONCE().
>>
>> I saw your patch doing this, the phone now boots, but obviously I still
>> see the WARN, so maybe there's still something to be fixed.
>
cut
> Now that it boots, can you please check if /proc/iomem is the same both in
> the non-working and working config. If that resource got assigned
> successfully, it might well be there is no actual differences in the
> assigned resources (which again doesn't mean there wouldn't be a bug in
> the logic as discussed above).
I confirm /proc/iomem is identical when comparing the no revert and the
WARN_ON_ONCE() case, and when reverting the blamed commit case.
On Fri, 30 May 2025, Ilpo Järvinen wrote:
> On Wed, 28 May 2025, Tudor Ambarus wrote:
> > On 5/6/25 4:53 PM, Ilpo Järvinen wrote:
> > > On Tue, 6 May 2025, Tudor Ambarus wrote:
> > >> On 12/16/24 5:56 PM, Ilpo Järvinen wrote:
> > >>> Resetting resource is problematic as it prevent attempting to allocate
> > >>> the resource later, unless something in between restores the resource.
> > >>> Similarly, if fail_head does not contain all resources that were reset,
> > >>> those resource cannot be restored later.
> > >>>
> > >>> The entire reset/restore cycle adds complexity and leaving resources
> > >>> into reseted state causes issues to other code such as for checks done
> > >>> in pci_enable_resources(). Take a small step towards not resetting
> > >>> resources by delaying reset until the end of resource assignment and
> > >>> build failure list (fail_head) in sync with the reset to avoid leaving
> > >>> behind resources that cannot be restored (for the case where the caller
> > >>> provides fail_head in the first place to allow restore somewhere in the
> > >>> callchain, as is not all callers pass non-NULL fail_head).
> > >>>
> > >>> The Expansion ROM check is temporarily left in place while building the
> > >>> failure list until the upcoming change which reworks optional resource
> > >>> handling.
> > >>>
> > >>> Ideally, whole resource reset could be removed but doing that in a big
> > >>> step would make the impact non-tractable due to complexity of all
> > >>> related code.
> > >>>
> > >>> Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
> > >>
> > >> I'm hitting the BUG_ON(!list_empty(&add_list)); in
> > >> pci_assign_unassigned_bus_resources() [1] with 6.15-rc5 and the the
> > >> pixel6 downstream pcie driver.
> > >>
> > >> I saw the thread where "a34d74877c66 PCI: Restore assigned resources
> > >> fully after release" fixes things for some other cases, but it's not the
> > >> case here.
> > >>
> > >> Reverting the following patches fixes the problem:
> > >> a34d74877c66 PCI: Restore assigned resources fully after release
> > >> 2499f5348431 PCI: Rework optional resource handling
> > >> 96336ec70264 PCI: Perform reset_resource() and build fail list in sync
> > >
> > > So it's confirmed that you needed to revert also this last commit
> > > 96336ec70264, not just the rework change?
> >
> > I needed to revert 96336ec70264 as well otherwise the build fails.
>
> Hi again,
>
> That's news to me... I seem to have botched the resource assignment rework
> series at some point when I reordered patches and dropped that helper as a
> result. And it seems the intermediate build fail wasn't caught by LKP :-(.
> (Pretty annoying as I intentionally separated these two to make them
> bisectable but not it isn't without amends.)
>
> The missing helper is basically this:
>
> static bool pci_resource_is_disabled_rom(const struct pci_dev *dev, int resno)
> {
> const struct resource *res = pci_resource_n(dev, resno);
>
> return resno == PCI_ROM_RESOURCE && !(res->flags & IORESOURCE_ROM_ENABLE)
> }
>
> (I didn't build test that.)
>
> Because of this, the actual culprit could be in 2499f5348431, not it
> 96336ec70264 (which would make more sense as it does significant rework
> on the assignment algorithm).
>
> > >> In the working case the add_list list is empty throughout the entire
> > >> body of pci_assign_unassigned_bus_resources().
> > >>
> > >> In the failing case __pci_bus_size_bridges() leaves the add_list not
> > >> empty and __pci_bus_assign_resources() does not consume the list, thus
> > >> the BUG_ON. The failing case contains an extra print that's not shown
> > >> when reverting the blamed commits:
> > >> [ 13.951185][ T1101] pcieport 0000:00:00.0: bridge window [mem
> > >> 0x00100000-0x001fffff] to [bus 01-ff] add_size 100000 add_align 100000
> > >>
> > >> I've added some prints trying to describe the code path, see
> > >> https://paste.ofcode.org/Aeu2YBpLztc49ZDw3uUJmd#
> > >>
> > >> Failing case:
> > >> [ 13.944231][ T1101] pci 0000:01:00.0: [144d:a5a5] type 00 class
> > >> 0x000000 PCIe Endpoint
> > >> [ 13.944412][ T1101] pci 0000:01:00.0: BAR 0 [mem
> > >> 0x00000000-0x000fffff 64bit]
> > >> [ 13.944532][ T1101] pci 0000:01:00.0: ROM [mem 0x00000000-0x0000ffff
> > >> pref]
> > >> [ 13.944649][ T1101] pci 0000:01:00.0: enabling Extended Tags
> > >> [ 13.944844][ T1101] pci 0000:01:00.0: PME# supported from D0 D3hot D3cold
> > >> [ 13.945015][ T1101] pci 0000:01:00.0: 15.752 Gb/s available PCIe
> > >> bandwidth, limited by 8.0 GT/s PCIe x2 link at 0000:00:00.0 (capable of
> > >> 31.506 Gb/s with 16.0 GT/s PCIe x2 link)
> > >> [ 13.950616][ T1101] __pci_bus_size_bridges: before pbus_size_mem.
> > >> list empty? 1
> > >> [ 13.950784][ T1101] pbus_size_mem: 2. list empty? 1
> > >> [ 13.950886][ T1101] pbus_size_mem: 1 list empty? 0
> > >> [ 13.950982][ T1101] pbus_size_mem: 3. list empty? 0
> > >> [ 13.951082][ T1101] pbus_size_mem: 4. list empty? 0
> > >> [ 13.951185][ T1101] pcieport 0000:00:00.0: bridge window [mem
> > >> 0x00100000-0x001fffff] to [bus 01-ff] add_size 100000 add_align 100000
> > >> [ 13.951448][ T1101] __pci_bus_size_bridges: after pbus_size_mem. list
> > >> empty? 0
> > >> [ 13.951643][ T1101] pci_assign_unassigned_bus_resources: before
> > >> __pci_bus_assign_resources -> list empty? 0
> > >> [ 13.951924][ T1101] pcieport 0000:00:00.0: bridge window [mem
> > >> 0x40000000-0x401fffff]: assigned
> > >> [ 13.952248][ T1101] pci_assign_unassigned_bus_resources: after
> > >> __pci_bus_assign_resources -> list empty? 0
> > >> [ 13.952634][ T1101] ------------[ cut here ]------------
> > >> [ 13.952818][ T1101] kernel BUG at drivers/pci/setup-bus.c:2514!
> > >> [ 13.953045][ T1101] Internal error: Oops - BUG: 00000000f2000800 [#1]
> > >> SMP
> > >> ...
> > >> [ 13.976086][ T1101] Call trace:
> > >> [ 13.976206][ T1101] pci_assign_unassigned_bus_resources+0x110/0x114 (P)
> > >> [ 13.976462][ T1101] pci_rescan_bus+0x28/0x48
> > >> [ 13.976628][ T1101] exynos_pcie_rc_poweron
> > >>
> > >> Working case:
> > >> [ 13.786961][ T1120] pci 0000:01:00.0: [144d:a5a5] type 00 class
> > >> 0x000000 PCIe Endpoint
> > >> [ 13.787136][ T1120] pci 0000:01:00.0: BAR 0 [mem
> > >> 0x00000000-0x000fffff 64bit]
> > >> [ 13.787280][ T1120] pci 0000:01:00.0: ROM [mem 0x00000000-0x0000ffff
> > >> pref]
> > >> [ 13.787541][ T1120] pci 0000:01:00.0: enabling Extended Tags
> > >> [ 13.787808][ T1120] pci 0000:01:00.0: PME# supported from D0 D3hot D3cold
> > >> [ 13.787988][ T1120] pci 0000:01:00.0: 15.752 Gb/s available PCIe
> > >> bandwidth, limited by 8.0 GT/s PCIe x2 link at 0000:00:00.0 (capable of
> > >> 31.506 Gb/s with 16.0 GT/s PCIe x2 link)
> > >> [ 13.795279][ T1120] __pci_bus_size_bridges: before pbus_size_mem.
> > >> list empty? 1
> > >> [ 13.795408][ T1120] pbus_size_mem: 2. list empty? 1
> > >> [ 13.795495][ T1120] pbus_size_mem: 2. list empty? 1
> > >> [ 13.795577][ T1120] __pci_bus_size_bridges: after pbus_size_mem. list
> > >> empty? 1
> > >> [ 13.795692][ T1120] pci_assign_unassigned_bus_resources: before
> > >> __pci_bus_assign_resources -> list empty? 1
> > >> [ 13.795849][ T1120] pcieport 0000:00:00.0: bridge window [mem
> > >> 0x40000000-0x401fffff]: assigned
> > >> [ 13.796072][ T1120] pci_assign_unassigned_bus_resources: after
> > >> __pci_bus_assign_resources -> list empty? 1
> > >> [ 13.796662][ T1120] cpif: s5100_poweron_pcie: DBG: MSI sfr not set
> > >> up, yet(s5100_pdev is NULL)
> > >> [ 13.796666][ T1120] cpif: register_pcie: s51xx_pcie_init start
> > >>
> > >>
> > >> Any hints are welcomed. Thanks,
> > >> ta
> > >
> > > Hi and thanks for the report.
> >
> > Hi! Thanks for the help. I've been out of office for the last 2 weeks,
> > sorry for the delayed reply.
>
> Np.
>
> > > The interesting part occurs inside reassign_resources_sorted() where most
> > > items are eliminated from realloc_head by the list_del().
> > >
> > > My guess is that somehow, the change in 96336ec70264 from !res->flags
> > > to the more complicated check somehow causes this. If the new check
> > > doesn't match and subsequently, no match is found from the head list, the
> > > loop will do continue and not remove the entry from realloc_head.
> >
> > I added a print right there and it seems it's something else. See below.
> > >
> > > But it's hard to confirm without knowing what that resources realloc_head
> > > contains. Perhaps if you print the resources that are processed around
> > > that part of the code in reassign_resources_sorted(), comparing the log
> > > from the reverted code with the non-working case might help to understand
> > > what is different there and why. To understand better what is in the head
> > > list, it would be also useful to know from which device the resources were
> > > added into the head list in pdev_sort_resources().
> > >
> >
> > I added the suggested prints
> > (https://paste.ofcode.org/DgmZGGgS6D36nWEzmfCqMm) on top of v6.15 with
> > the downstream PCIe pixel driver and I obtain the following. Note that
> > all added prints contain "tudor" for differentiation.
> >
> > [ 15.211179][ T1107] pci 0001:01:00.0: [144d:a5a5] type 00 class
> > 0x000000 PCIe Endpoint
> > [ 15.212248][ T1107] pci 0001:01:00.0: BAR 0 [mem
> > 0x00000000-0x000fffff 64bit]
> > [ 15.212775][ T1107] pci 0001:01:00.0: ROM [mem 0x00000000-0x0000ffff
> > pref]
> > [ 15.213195][ T1107] pci 0001:01:00.0: enabling Extended Tags
> > [ 15.213720][ T1107] pci 0001:01:00.0: PME# supported from D0 D3hot
> > D3cold
> > [ 15.214035][ T1107] pci 0001:01:00.0: 15.752 Gb/s available PCIe
> > bandwidth, limited by 8.0 GT/s PCIe x2 link at 0001:00:00.0 (capable of
> > 31.506 Gb/s with 16.0 GT/s PCIe x2 link)
> > [ 15.222286][ T1107] pci 0001:01:00.0: tudor: 1: pbus_size_mem: BAR 0
> > [mem 0x00000000-0x000fffff 64bit] list empty? 1
> > [ 15.222813][ T1107] pci 0001:01:00.0: tudor: 1: pbus_size_mem: ROM
> > [mem 0x00000000-0x0000ffff pref] list empty? 1
> > [ 15.224429][ T1107] pci 0001:01:00.0: tudor: 2: pbus_size_mem: ROM
> > [mem 0x00000000-0x0000ffff pref] list empty? 0
> > [ 15.224750][ T1107] pcieport 0001:00:00.0: bridge window [mem
> > 0x00100000-0x001fffff] to [bus 01-ff] add_size 100000 add_align 100000
> >
> > [ 15.225393][ T1107] tudor : pci_assign_unassigned_bus_resources:
> > before __pci_bus_assign_resources -> list empty? 0
> > [ 15.225594][ T1107] pcieport 0001:00:00.0: tudor:
> > pdev_sort_resources: bridge window [mem 0x00100000-0x001fffff] resource
> > added in head list
> > [ 15.226078][ T1107] pcieport 0001:00:00.0: bridge window [mem
> > 0x40000000-0x401fffff]: assigned
>
> So here it ends up assigning the resource here I think.
>
>
> That print isn't one of yours in reassign_resources_sorted() so the
> assignment must have been made in assign_requested_resources_sorted(). But
> then nothing is printed out from reassign_resources_sorted() so I suspect
> __assign_resources_sorted() has short-circuited.
>
> We know that realloc_head is not empty, so that leaves the goto out from
> if (list_empty(&local_fail_head)), which kind of makes sense, all
> entries on the head list were assigned. But the code there tries to remove
> all head list resources from realloc_head so why it doesn't get removed is
> still a mystery. assign_requested_resources_sorted() doesn't seem to
> remove anything from the head list so that resource should still be on the
> head list AFAICT so it should call that remove_from_list(realloc_head,
> dev_res->res) for it.
>
> So can you see if that theory holds water and it short-circuits without
> removing the entry from realloc_head?
I think I figured out more about the reason. It's not related to that
bridge window resource.
pbus_size_mem() will add also that ROM resource into realloc_head
as it is considered (intentionally) optional after the optional change
(as per "tudor: 2:" line). And that resource is never assigned because
pdev_sort_resources() didn't pick it up into the head list. The next
question is why the ROM resource isn't in the head list.
While it is not necessarily related to issue, I think the bridge sizing
functions too should consider pdev_resources_assignable() so that it
won't ever add resources from such devices onto the realloc_head. This is
yet another small inconsistency within all this fitting/assignment logic.
pbus_size_mem() seems to consider IORESOURCE_PCI_FIXED so that cannot
explain it as the ROM resource wouldn't be on the realloc_head list in
that case.
Just wanted to let you know early even if I don't fully understand
everything so you can hopefully avoid unnecessary debugging.
> > [ 15.226419][ T1107] tudor : pci_assign_unassigned_bus_resources:
> > after __pci_bus_assign_resources -> list empty? 0
> > [ 15.226442][ T1107] ------------[ cut here ]------------
> > [ 15.227587][ T1107] kernel BUG at drivers/pci/setup-bus.c:2522!
> > [ 15.227813][ T1107] Internal error: Oops - BUG: 00000000f2000800 [#1]
> > SMP
> > ...
> > [ 15.251570][ T1107] Call trace:
> > [ 15.251690][ T1107] pci_assign_unassigned_bus_resources+0x110/0x114 (P)
> > [ 15.251945][ T1107] pci_rescan_bus+0x28/0x48
> >
> > I obtain the following output when using the same prints adapted
> > (https://paste.ofcode.org/37w7RnKkPaCxyNhi5yhZPbZ) and with the blamed
> > commits reverted:
> > a34d74877c66 PCI: Restore assigned resources fully after release
> > 2499f5348431 PCI: Rework optional resource handling
> > 96336ec70264 PCI: Perform reset_resource() and build fail list in sync
> >
> > [ 15.200456][ T1102] pci 0000:01:00.0: [144d:a5a5] type 00 class
> > 0x000000 PCIe Endpoint
> > [ 15.200632][ T1102] pci 0000:01:00.0: BAR 0 [mem
> > 0x00000000-0x000fffff 64bit]
> > [ 15.200755][ T1102] pci 0000:01:00.0: ROM [mem 0x00000000-0x0000ffff
> > pref]
> > [ 15.200876][ T1102] pci 0000:01:00.0: enabling Extended Tags
> > [ 15.201075][ T1102] pci 0000:01:00.0: PME# supported from D0 D3hot D3cold
> > [ 15.201254][ T1102] pci 0000:01:00.0: 15.752 Gb/s available PCIe
> > bandwidth, limited by 8.0 GT/s PCIe x2 link at 0000:00:00.0 (capable of
> > 31.506 Gb/s with 16.0 GT/s PCIe x2 link)
> > [ 15.206555][ T1102] pci 0000:01:00.0: tudor: 1: pbus_size_mem: BAR 0
> > [mem 0x00000000-0x000fffff 64bit] list empty? 1
> > [ 15.206737][ T1102] pci 0000:01:00.0: tudor: 1: pbus_size_mem: ROM
> > [mem 0x00000000-0x0000ffff pref] list empty? 1
> > [ 15.206901][ T1102] tudor : pci_assign_unassigned_bus_resources:
> > before __pci_bus_assign_resources -> list empty? 1
> > [ 15.207072][ T1102] pcieport 0000:00:00.0: tudor:
> > pdev_sort_resources: bridge window [mem 0x00100000-0x002fffff] resource
> > added in head list
> > [ 15.207396][ T1102] pcieport 0000:00:00.0: bridge window [mem
> > 0x40000000-0x401fffff]: assigned
> > [ 15.208165][ T1102] tudor : pci_assign_unassigned_bus_resources:
> > after __pci_bus_assign_resources -> list empty? 1
> > [ 15.208783][ T1102] cpif: s5100_poweron_pcie: DBG: MSI sfr not set
> > up, yet(s5100_pdev is NULL)
> > [ 15.208786][ T1102] cpif: register_pcie: s51xx_pcie_init start
> >
> > > In any case, that BUG_ON() seems a bit drastic action for what might be
> > > just a single resource allocation failure so it should be downgraded to:
> > >
> > > if (WARN_ON(!list_empty(&add_list))
> > > free_list(&add_list);
> > >
> > > ... or WARN_ON_ONCE().
> >
> > I saw your patch doing this, the phone now boots, but obviously I still
> > see the WARN, so maybe there's still something to be fixed.
>
> Yes, I don't expect BUG_ON() -> WARN "fix" anything, it just downgrades
> the severity so that the system can still try to boot, which can often
> succeed as this tends to be non-critical failure in many cases so it's
> useful change to have regardless despite the splat.
>
> Now that it boots, can you please check if /proc/iomem is the same both in
> the non-working and working config. If that resource got assigned
> successfully, it might well be there is no actual differences in the
> assigned resources (which again doesn't mean there wouldn't be a bug in
> the logic as discussed above).
>
>
>
--
i.
On 5/30/25 3:48 PM, Ilpo Järvinen wrote: >>> I added the suggested prints >>> (https://paste.ofcode.org/DgmZGGgS6D36nWEzmfCqMm) on top of v6.15 with >>> the downstream PCIe pixel driver and I obtain the following. Note that >>> all added prints contain "tudor" for differentiation. >>> >>> [ 15.211179][ T1107] pci 0001:01:00.0: [144d:a5a5] type 00 class >>> 0x000000 PCIe Endpoint >>> [ 15.212248][ T1107] pci 0001:01:00.0: BAR 0 [mem >>> 0x00000000-0x000fffff 64bit] >>> [ 15.212775][ T1107] pci 0001:01:00.0: ROM [mem 0x00000000-0x0000ffff >>> pref] >>> [ 15.213195][ T1107] pci 0001:01:00.0: enabling Extended Tags >>> [ 15.213720][ T1107] pci 0001:01:00.0: PME# supported from D0 D3hot >>> D3cold >>> [ 15.214035][ T1107] pci 0001:01:00.0: 15.752 Gb/s available PCIe >>> bandwidth, limited by 8.0 GT/s PCIe x2 link at 0001:00:00.0 (capable of >>> 31.506 Gb/s with 16.0 GT/s PCIe x2 link) >>> [ 15.222286][ T1107] pci 0001:01:00.0: tudor: 1: pbus_size_mem: BAR 0 >>> [mem 0x00000000-0x000fffff 64bit] list empty? 1 >>> [ 15.222813][ T1107] pci 0001:01:00.0: tudor: 1: pbus_size_mem: ROM >>> [mem 0x00000000-0x0000ffff pref] list empty? 1 >>> [ 15.224429][ T1107] pci 0001:01:00.0: tudor: 2: pbus_size_mem: ROM >>> [mem 0x00000000-0x0000ffff pref] list empty? 0 >>> [ 15.224750][ T1107] pcieport 0001:00:00.0: bridge window [mem >>> 0x00100000-0x001fffff] to [bus 01-ff] add_size 100000 add_align 100000 >>> >>> [ 15.225393][ T1107] tudor : pci_assign_unassigned_bus_resources: >>> before __pci_bus_assign_resources -> list empty? 0 >>> [ 15.225594][ T1107] pcieport 0001:00:00.0: tudor: >>> pdev_sort_resources: bridge window [mem 0x00100000-0x001fffff] resource >>> added in head list >>> [ 15.226078][ T1107] pcieport 0001:00:00.0: bridge window [mem >>> 0x40000000-0x401fffff]: assigned >> So here it ends up assigning the resource here I think. >> >> >> That print isn't one of yours in reassign_resources_sorted() so the >> assignment must have been made in assign_requested_resources_sorted(). But >> then nothing is printed out from reassign_resources_sorted() so I suspect >> __assign_resources_sorted() has short-circuited. >> >> We know that realloc_head is not empty, so that leaves the goto out from >> if (list_empty(&local_fail_head)), which kind of makes sense, all >> entries on the head list were assigned. But the code there tries to remove >> all head list resources from realloc_head so why it doesn't get removed is >> still a mystery. assign_requested_resources_sorted() doesn't seem to >> remove anything from the head list so that resource should still be on the >> head list AFAICT so it should call that remove_from_list(realloc_head, >> dev_res->res) for it. >> >> So can you see if that theory holds water and it short-circuits without >> removing the entry from realloc_head? > I think I figured out more about the reason. It's not related to that > bridge window resource. > > pbus_size_mem() will add also that ROM resource into realloc_head > as it is considered (intentionally) optional after the optional change > (as per "tudor: 2:" line). And that resource is never assigned because right, the ROM resource is added into realloc_head here: https://web.git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/drivers/pci/setup-bus.c#n1202 Then in the failing case, and extra resource is added: [ 15.224750][ T1107] pcieport 0001:00:00.0: bridge window [mem 0x00100000-0x001fffff] to [bus 01-ff] add_size 100000 add_align 100000 The above extra print happens just in the failing case. Here's where the extra resource is added: https://web.git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/drivers/pci/setup-bus.c#n1285 It seems that in the failing case 2 resources are added into realloc_head at the pbus_size_mem() time, whereas with the patch reverted - none. Also, in the failing case a smaller resource is added into the list: pdev_sort_resources: bridge window [mem 0x00100000-0x001fffff] compared to the working case: pdev_sort_resources: bridge window [mem 0x00100000-0x002fffff] Can this make a difference? > pdev_sort_resources() didn't pick it up into the head list. The next > question is why the ROM resource isn't in the head list. > It seems the ROM resource is skipped at: https://web.git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/drivers/pci/setup-bus.c#n175 tudor: pdev_sort_resources: ROM [??? 0x00000000 flags 0x0] resource skipped due to !(r->flags) || r->parent > > While it is not necessarily related to issue, I think the bridge sizing > functions too should consider pdev_resources_assignable() so that it > won't ever add resources from such devices onto the realloc_head. This is > yet another small inconsistency within all this fitting/assignment logic. > > pbus_size_mem() seems to consider IORESOURCE_PCI_FIXED so that cannot > explain it as the ROM resource wouldn't be on the realloc_head list in > that case. > > > Just wanted to let you know early even if I don't fully understand > everything so you can hopefully avoid unnecessary debugging. Thanks! Would adding some prints in pbus_size_mem() to describe the code paths in the working and non-working case help? Cheers, ta
On Mon, 2 Jun 2025, Tudor Ambarus wrote: > On 5/30/25 3:48 PM, Ilpo Järvinen wrote: > >>> I added the suggested prints > >>> (https://paste.ofcode.org/DgmZGGgS6D36nWEzmfCqMm) on top of v6.15 with > >>> the downstream PCIe pixel driver and I obtain the following. Note that > >>> all added prints contain "tudor" for differentiation. > >>> > >>> [ 15.211179][ T1107] pci 0001:01:00.0: [144d:a5a5] type 00 class > >>> 0x000000 PCIe Endpoint > >>> [ 15.212248][ T1107] pci 0001:01:00.0: BAR 0 [mem > >>> 0x00000000-0x000fffff 64bit] > >>> [ 15.212775][ T1107] pci 0001:01:00.0: ROM [mem 0x00000000-0x0000ffff > >>> pref] > >>> [ 15.213195][ T1107] pci 0001:01:00.0: enabling Extended Tags > >>> [ 15.213720][ T1107] pci 0001:01:00.0: PME# supported from D0 D3hot > >>> D3cold > >>> [ 15.214035][ T1107] pci 0001:01:00.0: 15.752 Gb/s available PCIe > >>> bandwidth, limited by 8.0 GT/s PCIe x2 link at 0001:00:00.0 (capable of > >>> 31.506 Gb/s with 16.0 GT/s PCIe x2 link) > >>> [ 15.222286][ T1107] pci 0001:01:00.0: tudor: 1: pbus_size_mem: BAR 0 > >>> [mem 0x00000000-0x000fffff 64bit] list empty? 1 > >>> [ 15.222813][ T1107] pci 0001:01:00.0: tudor: 1: pbus_size_mem: ROM > >>> [mem 0x00000000-0x0000ffff pref] list empty? 1 > >>> [ 15.224429][ T1107] pci 0001:01:00.0: tudor: 2: pbus_size_mem: ROM > >>> [mem 0x00000000-0x0000ffff pref] list empty? 0 > >>> [ 15.224750][ T1107] pcieport 0001:00:00.0: bridge window [mem > >>> 0x00100000-0x001fffff] to [bus 01-ff] add_size 100000 add_align 100000 > >>> > >>> [ 15.225393][ T1107] tudor : pci_assign_unassigned_bus_resources: > >>> before __pci_bus_assign_resources -> list empty? 0 > >>> [ 15.225594][ T1107] pcieport 0001:00:00.0: tudor: > >>> pdev_sort_resources: bridge window [mem 0x00100000-0x001fffff] resource > >>> added in head list > >>> [ 15.226078][ T1107] pcieport 0001:00:00.0: bridge window [mem > >>> 0x40000000-0x401fffff]: assigned > >> So here it ends up assigning the resource here I think. > >> > >> > >> That print isn't one of yours in reassign_resources_sorted() so the > >> assignment must have been made in assign_requested_resources_sorted(). But > >> then nothing is printed out from reassign_resources_sorted() so I suspect > >> __assign_resources_sorted() has short-circuited. > >> > >> We know that realloc_head is not empty, so that leaves the goto out from > >> if (list_empty(&local_fail_head)), which kind of makes sense, all > >> entries on the head list were assigned. But the code there tries to remove > >> all head list resources from realloc_head so why it doesn't get removed is > >> still a mystery. assign_requested_resources_sorted() doesn't seem to > >> remove anything from the head list so that resource should still be on the > >> head list AFAICT so it should call that remove_from_list(realloc_head, > >> dev_res->res) for it. > >> > >> So can you see if that theory holds water and it short-circuits without > >> removing the entry from realloc_head? > > I think I figured out more about the reason. It's not related to that > > bridge window resource. > > > > pbus_size_mem() will add also that ROM resource into realloc_head > > as it is considered (intentionally) optional after the optional change > > (as per "tudor: 2:" line). And that resource is never assigned because > > right, the ROM resource is added into realloc_head here: > https://web.git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/drivers/pci/setup-bus.c#n1202 > > Then in the failing case, and extra resource is added: > [ 15.224750][ T1107] pcieport 0001:00:00.0: bridge window [mem > 0x00100000-0x001fffff] to [bus 01-ff] add_size 100000 add_align 100000 > The above extra print happens just in the failing case. Here's where the > extra resource is added: > https://web.git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/drivers/pci/setup-bus.c#n1285 > > It seems that in the failing case 2 resources are added into > realloc_head at the pbus_size_mem() time, whereas with the patch > reverted - none. > > Also, in the failing case a smaller resource is added into the list: > pdev_sort_resources: bridge window [mem 0x00100000-0x001fffff] > compared to the working case: > pdev_sort_resources: bridge window [mem 0x00100000-0x002fffff] > > Can this make a difference? I don't think the bridge window resource different matters, that bridge resource got assigned just fine so it is a red herring. The code just calculates optional sizes in different place after the rework than before it. The successful assignment itself is for the same size so there's nothing wrong there AFAICT. > > pdev_sort_resources() didn't pick it up into the head list. The next > > question is why the ROM resource isn't in the head list. > > > > It seems the ROM resource is skipped at: > https://web.git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/drivers/pci/setup-bus.c#n175 > > tudor: pdev_sort_resources: ROM [??? 0x00000000 flags 0x0] resource > skipped due to !(r->flags) || r->parent I don't see the device in this print, hope it is for the same device. In any case, I don't understand what reset resource's flags in between pbus_size_mem() and pdev_sort_resources(), or alternative, why type checking in pbus_size_mem() matches if flags == 0 at that point. Those two functions should work on the same resources, if one skips something, the other should too. Disparity between them can cause issues, but despite reading the code multiple times, I couldn't figure out how that disparity occurs (except for the !pdev_resources_assignable() case). > > While it is not necessarily related to issue, I think the bridge sizing > > functions too should consider pdev_resources_assignable() so that it > > won't ever add resources from such devices onto the realloc_head. This is > > yet another small inconsistency within all this fitting/assignment logic. > > > > pbus_size_mem() seems to consider IORESOURCE_PCI_FIXED so that cannot > > explain it as the ROM resource wouldn't be on the realloc_head list in > > that case. > > > > > > Just wanted to let you know early even if I don't fully understand > > everything so you can hopefully avoid unnecessary debugging. > > Thanks! Would adding some prints in pbus_size_mem() to describe the code > paths in the working and non-working case help? It is of interest to know why the same resource is treated differently. So what were the resource flags, type* args when it's processed by pbus_size_mem()? If resource's flags are zero at that point but it matches one of the types, that would be a bug. -- i.
On 6/2/25 4:08 PM, Ilpo Järvinen wrote: >>> I think I figured out more about the reason. It's not related to that >>> bridge window resource. >>> >>> pbus_size_mem() will add also that ROM resource into realloc_head >>> as it is considered (intentionally) optional after the optional change >>> (as per "tudor: 2:" line). And that resource is never assigned because cut >>> pdev_sort_resources() didn't pick it up into the head list. The next >>> question is why the ROM resource isn't in the head list. >>> >> It seems the ROM resource is skipped at: >> https://web.git.kernel.org/pub/scm/linux/kernel/git/next/linux- >> next.git/tree/drivers/pci/setup-bus.c#n175 >> >> tudor: pdev_sort_resources: ROM [??? 0x00000000 flags 0x0] resource >> skipped due to !(r->flags) || r->parent > I don't see the device in this print, hope it is for the same device. > > In any case, I don't understand what reset resource's flags in between > pbus_size_mem() and pdev_sort_resources(), or alternative, why type > checking in pbus_size_mem() matches if flags == 0 at that point. > > Those two functions should work on the same resources, if one skips > something, the other should too. Disparity between them can cause issues, > but despite reading the code multiple times, I couldn't figure out how > that disparity occurs (except for the !pdev_resources_assignable() case). cut > It is of interest to know why the same resource is treated differently. > So what were the resource flags, type* args when it's processed by > pbus_size_mem()? If resource's flags are zero at that point but it matches This is the full output: https://termbin.com/mn1x for the following prints: https://termbin.com/q57h It seems ROM resource is of type 2 at pbus_size_mem() time. > one of the types, that would be a bug. I'll give another try tomorrow. Thanks, ta
On Mon, 2 Jun 2025, Tudor Ambarus wrote:
> On 6/2/25 4:08 PM, Ilpo Järvinen wrote:
> >>> I think I figured out more about the reason. It's not related to that
> >>> bridge window resource.
> >>>
> >>> pbus_size_mem() will add also that ROM resource into realloc_head
> >>> as it is considered (intentionally) optional after the optional change
> >>> (as per "tudor: 2:" line). And that resource is never assigned because
>
> cut
>
> >>> pdev_sort_resources() didn't pick it up into the head list. The next
> >>> question is why the ROM resource isn't in the head list.
> >>>
> >> It seems the ROM resource is skipped at:
> >> https://web.git.kernel.org/pub/scm/linux/kernel/git/next/linux-
> >> next.git/tree/drivers/pci/setup-bus.c#n175
> >>
> >> tudor: pdev_sort_resources: ROM [??? 0x00000000 flags 0x0] resource
> >> skipped due to !(r->flags) || r->parent
> > I don't see the device in this print, hope it is for the same device.
> >
> > In any case, I don't understand what reset resource's flags in between
> > pbus_size_mem() and pdev_sort_resources(), or alternative, why type
> > checking in pbus_size_mem() matches if flags == 0 at that point.
> >
> > Those two functions should work on the same resources, if one skips
> > something, the other should too. Disparity between them can cause issues,
> > but despite reading the code multiple times, I couldn't figure out how
> > that disparity occurs (except for the !pdev_resources_assignable() case).
>
> cut
>
> > It is of interest to know why the same resource is treated differently.
> > So what were the resource flags, type* args when it's processed by
> > pbus_size_mem()? If resource's flags are zero at that point but it matches
>
> This is the full output: https://termbin.com/mn1x
> for the following prints: https://termbin.com/q57h
>
> It seems ROM resource is of type 2 at pbus_size_mem() time.
>
> > one of the types, that would be a bug.
>
> I'll give another try tomorrow. Thanks,
Those are not the same device, so not the same resource either:
[ 16.262745][ T1113] pci 0001:01:00.0: tudor: 2: pbus_size_mem: ROM [mem 0x00000000-0x0000ffff pref] list empty? 0
[ 16.267736][ T1113] pcieport 0001:00:00.0: tudor: pdev_sort_resources: ROM [??? 0x00000000 flags 0x0] resource skipped due to !(r->flags) || r->parent
0001:01:00.0
vs
0001:00:00.0
And the resources for 0001:01:00.0 were never processed by
__pci_bus_assign_resources(). But __pci_bus_assign_resources() should
recurse to subordinate busses.
And, it seems this boils down to the inconsistency I noticed earlier:
[ 16.253464][ T1113] pci 0001:01:00.0: [144d:a5a5] type 00 class 0x000000 PCIe Endpoint
include/linux/pci_ids.h:#define PCI_CLASS_NOT_DEFINED 0x0000
pdev_resources_assignable() checks if class == PCI_CLASS_NOT_DEFINED and
pdev_sort_resources() bails out without processing those resources.
So please test if this patch solves your problem:
From 00e440f505a79568cf5203dce265488cb3f66941 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Ilpo=20J=C3=A4rvinen?= <ilpo.jarvinen@linux.intel.com>
Date: Tue, 3 Jun 2025 11:07:38 +0300
Subject: [PATCH 1/1] PCI: Fix pdev_resources_assignable() disparity
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
pdev_sort_resources() uses pdev_resources_assignable() helper to decide
if device's resources cannot be assigned. pbus_size_mem(), on the other
hand, does not do the same check. This could lead into a situation
where a resource ends up on realloc_head list but is not on the head
list, which is turn prevents emptying the resource from the
realloc_head list in __assign_resources_sorted().
A non-empty realloc_head is unacceptable because it triggers an
internal sanity check as show in this log with a device that has class
0 (PCI_CLASS_NOT_DEFINED):
pci 0001:01:00.0: [144d:a5a5] type 00 class 0x000000 PCIe Endpoint
pci 0001:01:00.0: BAR 0 [mem 0x00000000-0x000fffff 64bit]
pci 0001:01:00.0: ROM [mem 0x00000000-0x0000ffff pref]
pci 0001:01:00.0: enabling Extended Tags
pci 0001:01:00.0: PME# supported from D0 D3hot D3cold
pci 0001:01:00.0: 15.752 Gb/s available PCIe bandwidth, limited by 8.0 GT/s PCIe x2 link at 0001:00:00.0 (capable of 31.506 Gb/s with 16.0 GT/s PCIe x2 link)
pcieport 0001:00:00.0: bridge window [mem 0x00100000-0x001fffff] to [bus 01-ff] add_size 100000 add_align 100000
pcieport 0001:00:00.0: bridge window [mem 0x40000000-0x401fffff]: assigned
------------[ cut here ]------------
kernel BUG at drivers/pci/setup-bus.c:2532!
Internal error: Oops - BUG: 00000000f2000800 [#1] SMP
...
Call trace:
pci_assign_unassigned_bus_resources+0x110/0x114 (P)
pci_rescan_bus+0x28/0x48
Use pdev_resources_assignable() also within pbus_size_mem() to skip
processing of non-assignable resources which removes the disparity in
between what resources pdev_sort_resources() and pbus_size_mem()
consider. As non-assignable resources are no longer processed, they are
not added to the realloc_head list, thus the sanity check no longer
triggers.
This disparity problem is very old but only now became apparent after
the commit 2499f5348431 ("PCI: Rework optional resource handling") that
made the ROM resources optional when calculating bridge window sizes
which required adding the resource to the realloc_head list.
Previously, bridge windows were just sized larger than necessary.
Fixes: 2499f5348431 ("PCI: Rework optional resource handling")
Cc: stable@vger.kernel.org
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
---
drivers/pci/setup-bus.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c
index 54d6f4fa3ce1..da084251df43 100644
--- a/drivers/pci/setup-bus.c
+++ b/drivers/pci/setup-bus.c
@@ -1187,6 +1187,7 @@ static int pbus_size_mem(struct pci_bus *bus, unsigned long mask,
resource_size_t r_size;
if (r->parent || (r->flags & IORESOURCE_PCI_FIXED) ||
+ !pdev_resources_assignable(dev) ||
((r->flags & mask) != type &&
(r->flags & mask) != type2 &&
(r->flags & mask) != type3))
base-commit: 0af2f6be1b4281385b618cb86ad946eded089ac8
--
2.39.5
On 6/3/25 9:13 AM, Ilpo Järvinen wrote: > So please test if this patch solves your problem: It fails in a different way, the bridge window resource never gets assigned with the proposed patch. With the patch applied: https://termbin.com/h3w0 With the blamed commit reverted: https://termbin.com/3rh6
On Tue, 3 Jun 2025, Tudor Ambarus wrote: > On 6/3/25 9:13 AM, Ilpo Järvinen wrote: > > So please test if this patch solves your problem: > > It fails in a different way, the bridge window resource never gets > assigned with the proposed patch. Is that a failure? I was expecting that to occur. It didn't assign any resources into that bridge window. If there's nothing to be assigned into the bridge window, the bridge window itself is not created, that is the expected behavior (working as designed). So you're comparing to the bridge window that was made too large due to the disparity (and left unused, AFAICT). It would be possible to put the condition inside the block which adds the resource to the realloc_head, I initially put it there but then decided to remove the disparity completely because why keep it if no resource is going to be placed into the bridge window. What's that class 0 device anyway? Why it has class 0? > With the patch applied: https://termbin.com/h3w0 > With the blamed commit reverted: https://termbin.com/3rh6 -- i.
On 6/3/25 3:13 PM, Ilpo Järvinen wrote: > On Tue, 3 Jun 2025, Tudor Ambarus wrote: >> On 6/3/25 9:13 AM, Ilpo Järvinen wrote: >>> So please test if this patch solves your problem: >> >> It fails in a different way, the bridge window resource never gets >> assigned with the proposed patch. > > Is that a failure? I was expecting that to occur. It didn't assign > any resources into that bridge window. It leads to a watchdog interrupt on my pixel6. Last print I see on my console is related to the modem booting status. My wild guess is that that modem accesses something from the unassigned bridge window. In the working case I see the bridge window printed: [ 15.457310][ T1083] pcieport 0000:00:00.0: [s51xx_pcie_probe] BAR 14: tmp rsc : [mem 0x40000000-0x401fffff] [ 15.457683][ T1083] cpif: s51xx_pcie_probe: Set Doorbell register address. In the failing case I see: [ 15.623270][ T1113] pcieport 0000:00:00.0: [s51xx_pcie_probe] BAR 14: tmp rsc : [??? 0x00000000 flags 0x0] [ 15.623638][ T1113] cpif: s51xx_pcie_probe: Set Doorbell register address. > > If there's nothing to be assigned into the bridge window, the bridge > window itself is not created, that is the expected behavior (working as > designed). So you're comparing to the bridge window that was made too > large due to the disparity (and left unused, AFAICT). > > It would be possible to put the condition inside the block which adds > the resource to the realloc_head, I initially put it there but then > decided to remove the disparity completely because why keep it if no > resource is going to be placed into the bridge window. > Thanks for the educative answers. > What's that class 0 device anyway? Why it has class 0? > I don't know yet, it's the first time I'm dealing with a PCI driver. Any idea where is the class typically assigned? >> With the patch applied: https://termbin.com/h3w0 >> With the blamed commit reverted: https://termbin.com/3rh6 >
On Tue, 3 Jun 2025, Tudor Ambarus wrote:
> On 6/3/25 3:13 PM, Ilpo Järvinen wrote:
> > On Tue, 3 Jun 2025, Tudor Ambarus wrote:
> >> On 6/3/25 9:13 AM, Ilpo Järvinen wrote:
> >>> So please test if this patch solves your problem:
> >>
> >> It fails in a different way, the bridge window resource never gets
> >> assigned with the proposed patch.
> >
> > Is that a failure? I was expecting that to occur. It didn't assign
> > any resources into that bridge window.
>
> It leads to a watchdog interrupt on my pixel6. Last print I see on my
> console is related to the modem booting status. My wild guess is that
> that modem accesses something from the unassigned bridge window.
The bridge window is not for the bridge device itself. It's as the name
says, a window into where subordinate busses can assign their resources.
The bridge knows it must forward that window address range to the
subordinate bus.
> In the working case I see the bridge window printed:
> [ 15.457310][ T1083] pcieport 0000:00:00.0: [s51xx_pcie_probe] BAR 14:
> tmp rsc : [mem 0x40000000-0x401fffff]
>
> [ 15.457683][ T1083] cpif: s51xx_pcie_probe: Set Doorbell register
> address.
>
> In the failing case I see:
> [ 15.623270][ T1113] pcieport 0000:00:00.0: [s51xx_pcie_probe] BAR 14:
> tmp rsc : [??? 0x00000000 flags 0x0]
>
> [ 15.623638][ T1113] cpif: s51xx_pcie_probe: Set Doorbell register
> address.
Oh, is it this one?
https://github.com/oberdfr/google-modules_radio_samsung_s5300/blob/11a10f955a267a45a1997f65671d7054adf1a33a/s51xx_pcie.c#L366
There are number of crazy things going on there... Probe shouldn't be
messing resources like that. If it wants to change resources, a quirk
would be more appropriate place I guess but I'm very unsure what that
even tries to achieve with all that craziness ("Disable BAR resources" by
assigning them :-/).
But yes, it seems to take the bridge window's address and assumes
something is there (which isn't there as we know). So this driver code is
plain wrong.
Perhaps it would want to use the address of some endpoint device resource
instead of the bridge window address (e.g., that device with class 0?).
> > If there's nothing to be assigned into the bridge window, the bridge
> > window itself is not created, that is the expected behavior (working as
> > designed). So you're comparing to the bridge window that was made too
> > large due to the disparity (and left unused, AFAICT).
> >
> > It would be possible to put the condition inside the block which adds
> > the resource to the realloc_head, I initially put it there but then
> > decided to remove the disparity completely because why keep it if no
> > resource is going to be placed into the bridge window.
> >
> Thanks for the educative answers.
>
> > What's that class 0 device anyway? Why it has class 0?
> >
> I don't know yet, it's the first time I'm dealing with a PCI driver. Any
> idea where is the class typically assigned?
https://pcisig.com/sites/default/files/files/PCI_Code-ID_r_1_12__v9_Jan_2020.pdf
Perhaps try a quirk which changes the class of the device underneath the
bridge to something else than 0, it should make the resource fitting and
allocation to assign its resources.
But honestly, that s51xx_pcie_probe() has more than one thing wrong.
> >> With the patch applied: https://termbin.com/h3w0
> >> With the blamed commit reverted: https://termbin.com/3rh6
> >
>
--
i.
On Tue, 3 Jun 2025, Ilpo Järvinen wrote:
> On Tue, 3 Jun 2025, Tudor Ambarus wrote:
> > On 6/3/25 3:13 PM, Ilpo Järvinen wrote:
> > > On Tue, 3 Jun 2025, Tudor Ambarus wrote:
> > >> On 6/3/25 9:13 AM, Ilpo Järvinen wrote:
> > >>> So please test if this patch solves your problem:
> > >>
> > >> It fails in a different way, the bridge window resource never gets
> > >> assigned with the proposed patch.
> > >
> > > Is that a failure? I was expecting that to occur. It didn't assign
> > > any resources into that bridge window.
> >
> > It leads to a watchdog interrupt on my pixel6. Last print I see on my
> > console is related to the modem booting status. My wild guess is that
> > that modem accesses something from the unassigned bridge window.
>
> The bridge window is not for the bridge device itself. It's as the name
> says, a window into where subordinate busses can assign their resources.
> The bridge knows it must forward that window address range to the
> subordinate bus.
>
> > In the working case I see the bridge window printed:
> > [ 15.457310][ T1083] pcieport 0000:00:00.0: [s51xx_pcie_probe] BAR 14:
> > tmp rsc : [mem 0x40000000-0x401fffff]
> >
> > [ 15.457683][ T1083] cpif: s51xx_pcie_probe: Set Doorbell register
> > address.
> >
> > In the failing case I see:
> > [ 15.623270][ T1113] pcieport 0000:00:00.0: [s51xx_pcie_probe] BAR 14:
> > tmp rsc : [??? 0x00000000 flags 0x0]
> >
> > [ 15.623638][ T1113] cpif: s51xx_pcie_probe: Set Doorbell register
> > address.
>
> Oh, is it this one?
>
> https://github.com/oberdfr/google-modules_radio_samsung_s5300/blob/11a10f955a267a45a1997f65671d7054adf1a33a/s51xx_pcie.c#L366
>
> There are number of crazy things going on there... Probe shouldn't be
> messing resources like that. If it wants to change resources, a quirk
> would be more appropriate place I guess but I'm very unsure what that
> even tries to achieve with all that craziness ("Disable BAR resources" by
> assigning them :-/).
Or maybe DT, I'm not very familiar with DT things.
--
i.
> But yes, it seems to take the bridge window's address and assumes
> something is there (which isn't there as we know). So this driver code is
> plain wrong.
>
> Perhaps it would want to use the address of some endpoint device resource
> instead of the bridge window address (e.g., that device with class 0?).
>
> > > If there's nothing to be assigned into the bridge window, the bridge
> > > window itself is not created, that is the expected behavior (working as
> > > designed). So you're comparing to the bridge window that was made too
> > > large due to the disparity (and left unused, AFAICT).
> > >
> > > It would be possible to put the condition inside the block which adds
> > > the resource to the realloc_head, I initially put it there but then
> > > decided to remove the disparity completely because why keep it if no
> > > resource is going to be placed into the bridge window.
> > >
> > Thanks for the educative answers.
> >
> > > What's that class 0 device anyway? Why it has class 0?
> > >
> > I don't know yet, it's the first time I'm dealing with a PCI driver. Any
> > idea where is the class typically assigned?
>
> https://pcisig.com/sites/default/files/files/PCI_Code-ID_r_1_12__v9_Jan_2020.pdf
>
> Perhaps try a quirk which changes the class of the device underneath the
> bridge to something else than 0, it should make the resource fitting and
> allocation to assign its resources.
>
> But honestly, that s51xx_pcie_probe() has more than one thing wrong.
>
> > >> With the patch applied: https://termbin.com/h3w0
> > >> With the blamed commit reverted: https://termbin.com/3rh6
> > >
> >
>
>
On 6/3/25 11:36 AM, Tudor Ambarus wrote: > > > On 6/3/25 9:13 AM, Ilpo Järvinen wrote: >> So please test if this patch solves your problem: > > It fails in a different way, the bridge window resource never gets > assigned with the proposed patch. > > With the patch applied: https://termbin.com/h3w0 above is no revert and with the proposed fix. It also contains the prints https://termbin.com/g4zn It seems the prints in pbus_size_mem are not longer hit, likely because of the new condition added: ``!pdev_resources_assignable(dev) ||``, pci_dev_for_each_resource() finishes without doing anything. > With the blamed commit reverted: https://termbin.com/3rh6
On 6/3/25 11:48 AM, Tudor Ambarus wrote: > > > On 6/3/25 11:36 AM, Tudor Ambarus wrote: >> >> >> On 6/3/25 9:13 AM, Ilpo Järvinen wrote: >>> So please test if this patch solves your problem: >> >> It fails in a different way, the bridge window resource never gets >> assigned with the proposed patch. >> >> With the patch applied: https://termbin.com/h3w0 > > above is no revert and with the proposed fix. It also contains the > prints https://termbin.com/g4zn > > It seems the prints in pbus_size_mem are not longer hit, likely because > of the new condition added: ``!pdev_resources_assignable(dev) ||``, > pci_dev_for_each_resource() finishes without doing anything. > >> With the blamed commit reverted: https://termbin.com/3rh6 > I think I found the inconsistency. __pci_bus_assign_resources() pbus_assign_resources_sorted() pdev_sort_resources(dev, &head); But pdev_sort_resources() is called with a newly LIST_HEAD(head), not with realloc_head, thus the resources never get sorted. pdev_sort_resources() exits early at ``if (!pdev_resources_assignable(dev))``
On Tue, 3 Jun 2025, Tudor Ambarus wrote: > On 6/3/25 11:48 AM, Tudor Ambarus wrote: > > On 6/3/25 11:36 AM, Tudor Ambarus wrote: > >> On 6/3/25 9:13 AM, Ilpo Järvinen wrote: > >>> So please test if this patch solves your problem: > >> > >> It fails in a different way, the bridge window resource never gets > >> assigned with the proposed patch. > >> > >> With the patch applied: https://termbin.com/h3w0 > > > > above is no revert and with the proposed fix. It also contains the > > prints https://termbin.com/g4zn > > > > It seems the prints in pbus_size_mem are not longer hit, likely because > > of the new condition added: ``!pdev_resources_assignable(dev) ||``, > > pci_dev_for_each_resource() finishes without doing anything. > > > >> With the blamed commit reverted: https://termbin.com/3rh6 > > > > I think I found the inconsistency. > > __pci_bus_assign_resources() > pbus_assign_resources_sorted() > pdev_sort_resources(dev, &head); > > But pdev_sort_resources() is called with a newly LIST_HEAD(head), not > with realloc_head, thus the resources never get sorted. pdev_sort_resources() is not supposed to add resources into realloc_head but just collects all the relevant resources in the descending order by size. There are two main lists here. The head list contains all relevant resources we're going to process and realloc_head keeps track which of them are optional (or optional in part, that is, some resources have the base size and the optional size). __assign_resources_sorted() will apply the optional sizes from realloc_head and re-sorts the head list while changing the sizes. If not all resources can be assigned, rollback happens and base sizes are assigned first, and then reassign_resources_sorted() handles the realloc_head ones afterwards for as many resources as possible. > pdev_sort_resources() exits early at > ``if (!pdev_resources_assignable(dev))`` Yes it does, for 0001:01:00.0. -- i.
On Tue, 3 Jun 2025, Ilpo Järvinen wrote: > On Tue, 3 Jun 2025, Tudor Ambarus wrote: > > On 6/3/25 11:48 AM, Tudor Ambarus wrote: > > > On 6/3/25 11:36 AM, Tudor Ambarus wrote: > > >> On 6/3/25 9:13 AM, Ilpo Järvinen wrote: > > >>> So please test if this patch solves your problem: > > >> > > >> It fails in a different way, the bridge window resource never gets > > >> assigned with the proposed patch. > > >> > > >> With the patch applied: https://termbin.com/h3w0 > > > > > > above is no revert and with the proposed fix. It also contains the > > > prints https://termbin.com/g4zn > > > > > > It seems the prints in pbus_size_mem are not longer hit, likely because > > > of the new condition added: ``!pdev_resources_assignable(dev) ||``, > > > pci_dev_for_each_resource() finishes without doing anything. > > > > > >> With the blamed commit reverted: https://termbin.com/3rh6 > > > > > > > I think I found the inconsistency. > > > > __pci_bus_assign_resources() > > pbus_assign_resources_sorted() > > pdev_sort_resources(dev, &head); > > > > But pdev_sort_resources() is called with a newly LIST_HEAD(head), not > > with realloc_head, thus the resources never get sorted. > > pdev_sort_resources() is not supposed to add resources into realloc_head > but just collects all the relevant resources in the descending order by > size. Small correction, they're ordered by alignment, not by size. For other than iov resources and bridge window, alignment order effectively the same as size order. > There are two main lists here. The head list contains all relevant > resources we're going to process and realloc_head keeps track which of > them are optional (or optional in part, that is, some resources have the > base size and the optional size). > > __assign_resources_sorted() will apply the optional sizes from > realloc_head and re-sorts the head list while changing the sizes. > If not all resources can be assigned, rollback happens and base sizes are > assigned first, and then reassign_resources_sorted() handles the > realloc_head ones afterwards for as many resources as possible. > > > pdev_sort_resources() exits early at > > ``if (!pdev_resources_assignable(dev))`` > > Yes it does, for 0001:01:00.0. > > -- i.
On 5/28/25 12:22 PM, Tudor Ambarus wrote: > > On 5/6/25 4:53 PM, Ilpo Järvinen wrote: >> On Tue, 6 May 2025, Tudor Ambarus wrote: >> >>> Hi! >>> >>> On 12/16/24 5:56 PM, Ilpo Järvinen wrote: >>>> Resetting resource is problematic as it prevent attempting to allocate >>>> the resource later, unless something in between restores the resource. >>>> Similarly, if fail_head does not contain all resources that were reset, >>>> those resource cannot be restored later. >>>> >>>> The entire reset/restore cycle adds complexity and leaving resources >>>> into reseted state causes issues to other code such as for checks done >>>> in pci_enable_resources(). Take a small step towards not resetting >>>> resources by delaying reset until the end of resource assignment and >>>> build failure list (fail_head) in sync with the reset to avoid leaving >>>> behind resources that cannot be restored (for the case where the caller >>>> provides fail_head in the first place to allow restore somewhere in the >>>> callchain, as is not all callers pass non-NULL fail_head). >>>> >>>> The Expansion ROM check is temporarily left in place while building the >>>> failure list until the upcoming change which reworks optional resource >>>> handling. >>>> >>>> Ideally, whole resource reset could be removed but doing that in a big >>>> step would make the impact non-tractable due to complexity of all >>>> related code. >>>> >>>> Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com> >>> I'm hitting the BUG_ON(!list_empty(&add_list)); in >>> pci_assign_unassigned_bus_resources() [1] with 6.15-rc5 and the the >>> pixel6 downstream pcie driver. >>> >>> I saw the thread where "a34d74877c66 PCI: Restore assigned resources >>> fully after release" fixes things for some other cases, but it's not the >>> case here. >>> >>> Reverting the following patches fixes the problem: >>> a34d74877c66 PCI: Restore assigned resources fully after release >>> 2499f5348431 PCI: Rework optional resource handling >>> 96336ec70264 PCI: Perform reset_resource() and build fail list in sync >> So it's confirmed that you needed to revert also this last commit >> 96336ec70264, not just the rework change? > I needed to revert 96336ec70264 as well otherwise the build fails. >>> In the working case the add_list list is empty throughout the entire >>> body of pci_assign_unassigned_bus_resources(). >>> >>> In the failing case __pci_bus_size_bridges() leaves the add_list not >>> empty and __pci_bus_assign_resources() does not consume the list, thus >>> the BUG_ON. The failing case contains an extra print that's not shown >>> when reverting the blamed commits: >>> [ 13.951185][ T1101] pcieport 0000:00:00.0: bridge window [mem >>> 0x00100000-0x001fffff] to [bus 01-ff] add_size 100000 add_align 100000 >>> >>> I've added some prints trying to describe the code path, see >>> https://paste.ofcode.org/Aeu2YBpLztc49ZDw3uUJmd# >>> >>> Failing case: >>> [ 13.944231][ T1101] pci 0000:01:00.0: [144d:a5a5] type 00 class >>> 0x000000 PCIe Endpoint >>> [ 13.944412][ T1101] pci 0000:01:00.0: BAR 0 [mem >>> 0x00000000-0x000fffff 64bit] >>> [ 13.944532][ T1101] pci 0000:01:00.0: ROM [mem 0x00000000-0x0000ffff >>> pref] >>> [ 13.944649][ T1101] pci 0000:01:00.0: enabling Extended Tags >>> [ 13.944844][ T1101] pci 0000:01:00.0: PME# supported from D0 D3hot D3cold >>> [ 13.945015][ T1101] pci 0000:01:00.0: 15.752 Gb/s available PCIe >>> bandwidth, limited by 8.0 GT/s PCIe x2 link at 0000:00:00.0 (capable of >>> 31.506 Gb/s with 16.0 GT/s PCIe x2 link) >>> [ 13.950616][ T1101] __pci_bus_size_bridges: before pbus_size_mem. >>> list empty? 1 >>> [ 13.950784][ T1101] pbus_size_mem: 2. list empty? 1 >>> [ 13.950886][ T1101] pbus_size_mem: 1 list empty? 0 >>> [ 13.950982][ T1101] pbus_size_mem: 3. list empty? 0 >>> [ 13.951082][ T1101] pbus_size_mem: 4. list empty? 0 >>> [ 13.951185][ T1101] pcieport 0000:00:00.0: bridge window [mem >>> 0x00100000-0x001fffff] to [bus 01-ff] add_size 100000 add_align 100000 >>> [ 13.951448][ T1101] __pci_bus_size_bridges: after pbus_size_mem. list >>> empty? 0 >>> [ 13.951643][ T1101] pci_assign_unassigned_bus_resources: before >>> __pci_bus_assign_resources -> list empty? 0 >>> [ 13.951924][ T1101] pcieport 0000:00:00.0: bridge window [mem >>> 0x40000000-0x401fffff]: assigned >>> [ 13.952248][ T1101] pci_assign_unassigned_bus_resources: after >>> __pci_bus_assign_resources -> list empty? 0 >>> [ 13.952634][ T1101] ------------[ cut here ]------------ >>> [ 13.952818][ T1101] kernel BUG at drivers/pci/setup-bus.c:2514! >>> [ 13.953045][ T1101] Internal error: Oops - BUG: 00000000f2000800 [#1] >>> SMP >>> ... >>> [ 13.976086][ T1101] Call trace: >>> [ 13.976206][ T1101] pci_assign_unassigned_bus_resources+0x110/0x114 (P) >>> [ 13.976462][ T1101] pci_rescan_bus+0x28/0x48 >>> [ 13.976628][ T1101] exynos_pcie_rc_poweron >>> >>> Working case: >>> [ 13.786961][ T1120] pci 0000:01:00.0: [144d:a5a5] type 00 class >>> 0x000000 PCIe Endpoint >>> [ 13.787136][ T1120] pci 0000:01:00.0: BAR 0 [mem >>> 0x00000000-0x000fffff 64bit] >>> [ 13.787280][ T1120] pci 0000:01:00.0: ROM [mem 0x00000000-0x0000ffff >>> pref] >>> [ 13.787541][ T1120] pci 0000:01:00.0: enabling Extended Tags >>> [ 13.787808][ T1120] pci 0000:01:00.0: PME# supported from D0 D3hot D3cold >>> [ 13.787988][ T1120] pci 0000:01:00.0: 15.752 Gb/s available PCIe >>> bandwidth, limited by 8.0 GT/s PCIe x2 link at 0000:00:00.0 (capable of >>> 31.506 Gb/s with 16.0 GT/s PCIe x2 link) >>> [ 13.795279][ T1120] __pci_bus_size_bridges: before pbus_size_mem. >>> list empty? 1 >>> [ 13.795408][ T1120] pbus_size_mem: 2. list empty? 1 >>> [ 13.795495][ T1120] pbus_size_mem: 2. list empty? 1 >>> [ 13.795577][ T1120] __pci_bus_size_bridges: after pbus_size_mem. list >>> empty? 1 >>> [ 13.795692][ T1120] pci_assign_unassigned_bus_resources: before >>> __pci_bus_assign_resources -> list empty? 1 >>> [ 13.795849][ T1120] pcieport 0000:00:00.0: bridge window [mem >>> 0x40000000-0x401fffff]: assigned >>> [ 13.796072][ T1120] pci_assign_unassigned_bus_resources: after >>> __pci_bus_assign_resources -> list empty? 1 >>> [ 13.796662][ T1120] cpif: s5100_poweron_pcie: DBG: MSI sfr not set >>> up, yet(s5100_pdev is NULL) >>> [ 13.796666][ T1120] cpif: register_pcie: s51xx_pcie_init start >>> >>> >>> Any hints are welcomed. Thanks, >>> ta >> Hi and thanks for the report. > Hi! Thanks for the help. I've been out of office for the last 2 weeks, > sorry for the delayed reply. > >> The interesting part occurs inside reassign_resources_sorted() where most >> items are eliminated from realloc_head by the list_del(). >> >> My guess is that somehow, the change in 96336ec70264 from !res->flags >> to the more complicated check somehow causes this. If the new check >> doesn't match and subsequently, no match is found from the head list, the >> loop will do continue and not remove the entry from realloc_head. > I added a print right there and it seems it's something else. See below. >> But it's hard to confirm without knowing what that resources realloc_head >> contains. Perhaps if you print the resources that are processed around >> that part of the code in reassign_resources_sorted(), comparing the log >> from the reverted code with the non-working case might help to understand >> what is different there and why. To understand better what is in the head >> list, it would be also useful to know from which device the resources were >> added into the head list in pdev_sort_resources(). >> > I added the suggested prints > (https://paste.ofcode.org/DgmZGGgS6D36nWEzmfCqMm) on top of v6.15 with > the downstream PCIe pixel driver and I obtain the following. Note that > all added prints contain "tudor" for differentiation. > > [ 15.211179][ T1107] pci 0001:01:00.0: [144d:a5a5] type 00 class > 0x000000 PCIe Endpoint > [ 15.212248][ T1107] pci 0001:01:00.0: BAR 0 [mem > 0x00000000-0x000fffff 64bit] > [ 15.212775][ T1107] pci 0001:01:00.0: ROM [mem 0x00000000-0x0000ffff > pref] > [ 15.213195][ T1107] pci 0001:01:00.0: enabling Extended Tags > [ 15.213720][ T1107] pci 0001:01:00.0: PME# supported from D0 D3hot > D3cold > [ 15.214035][ T1107] pci 0001:01:00.0: 15.752 Gb/s available PCIe > bandwidth, limited by 8.0 GT/s PCIe x2 link at 0001:00:00.0 (capable of > 31.506 Gb/s with 16.0 GT/s PCIe x2 link) > [ 15.222286][ T1107] pci 0001:01:00.0: tudor: 1: pbus_size_mem: BAR 0 > [mem 0x00000000-0x000fffff 64bit] list empty? 1 > [ 15.222813][ T1107] pci 0001:01:00.0: tudor: 1: pbus_size_mem: ROM > [mem 0x00000000-0x0000ffff pref] list empty? 1 > [ 15.224429][ T1107] pci 0001:01:00.0: tudor: 2: pbus_size_mem: ROM > [mem 0x00000000-0x0000ffff pref] list empty? 0 > [ 15.224750][ T1107] pcieport 0001:00:00.0: bridge window [mem > 0x00100000-0x001fffff] to [bus 01-ff] add_size 100000 add_align 100000 > > [ 15.225393][ T1107] tudor : pci_assign_unassigned_bus_resources: > before __pci_bus_assign_resources -> list empty? 0 > [ 15.225594][ T1107] pcieport 0001:00:00.0: tudor: > pdev_sort_resources: bridge window [mem 0x00100000-0x001fffff] resource > added in head list > [ 15.226078][ T1107] pcieport 0001:00:00.0: bridge window [mem > 0x40000000-0x401fffff]: assigned > [ 15.226419][ T1107] tudor : pci_assign_unassigned_bus_resources: > after __pci_bus_assign_resources -> list empty? 0 > [ 15.226442][ T1107] ------------[ cut here ]------------ > [ 15.227587][ T1107] kernel BUG at drivers/pci/setup-bus.c:2522! > [ 15.227813][ T1107] Internal error: Oops - BUG: 00000000f2000800 [#1] > SMP > ... > [ 15.251570][ T1107] Call trace: > [ 15.251690][ T1107] pci_assign_unassigned_bus_resources+0x110/0x114 (P) > [ 15.251945][ T1107] pci_rescan_bus+0x28/0x48 > > I obtain the following output when using the same prints adapted > (https://paste.ofcode.org/37w7RnKkPaCxyNhi5yhZPbZ) and with the blamed > commits reverted: > a34d74877c66 PCI: Restore assigned resources fully after release > 2499f5348431 PCI: Rework optional resource handling > 96336ec70264 PCI: Perform reset_resource() and build fail list in sync > > [ 15.200456][ T1102] pci 0000:01:00.0: [144d:a5a5] type 00 class > 0x000000 PCIe Endpoint > [ 15.200632][ T1102] pci 0000:01:00.0: BAR 0 [mem > 0x00000000-0x000fffff 64bit] > [ 15.200755][ T1102] pci 0000:01:00.0: ROM [mem 0x00000000-0x0000ffff > pref] > [ 15.200876][ T1102] pci 0000:01:00.0: enabling Extended Tags > [ 15.201075][ T1102] pci 0000:01:00.0: PME# supported from D0 D3hot D3cold > [ 15.201254][ T1102] pci 0000:01:00.0: 15.752 Gb/s available PCIe > bandwidth, limited by 8.0 GT/s PCIe x2 link at 0000:00:00.0 (capable of > 31.506 Gb/s with 16.0 GT/s PCIe x2 link) > [ 15.206555][ T1102] pci 0000:01:00.0: tudor: 1: pbus_size_mem: BAR 0 > [mem 0x00000000-0x000fffff 64bit] list empty? 1 > [ 15.206737][ T1102] pci 0000:01:00.0: tudor: 1: pbus_size_mem: ROM > [mem 0x00000000-0x0000ffff pref] list empty? 1 > [ 15.206901][ T1102] tudor : pci_assign_unassigned_bus_resources: > before __pci_bus_assign_resources -> list empty? 1 > [ 15.207072][ T1102] pcieport 0000:00:00.0: tudor: > pdev_sort_resources: bridge window [mem 0x00100000-0x002fffff] resource > added in head list > [ 15.207396][ T1102] pcieport 0000:00:00.0: bridge window [mem > 0x40000000-0x401fffff]: assigned > [ 15.208165][ T1102] tudor : pci_assign_unassigned_bus_resources: > after __pci_bus_assign_resources -> list empty? 1 > [ 15.208783][ T1102] cpif: s5100_poweron_pcie: DBG: MSI sfr not set > up, yet(s5100_pdev is NULL) > [ 15.208786][ T1102] cpif: register_pcie: s51xx_pcie_init start I see my email client split the lines for the prints making the output very hard to read. Added the output here too: https://paste.ofcode.org/AEfjASQW8Z2jbMak5VkmpJ
On 5/28/25 12:39 PM, Tudor Ambarus wrote:
>
>
> On 5/28/25 12:22 PM, Tudor Ambarus wrote:
>>
>> On 5/6/25 4:53 PM, Ilpo Järvinen wrote:
>>> On Tue, 6 May 2025, Tudor Ambarus wrote:
>>>
>>>> Hi!
>>>>
>>>> On 12/16/24 5:56 PM, Ilpo Järvinen wrote:
>>>>> Resetting resource is problematic as it prevent attempting to allocate
>>>>> the resource later, unless something in between restores the resource.
>>>>> Similarly, if fail_head does not contain all resources that were reset,
>>>>> those resource cannot be restored later.
>>>>>
>>>>> The entire reset/restore cycle adds complexity and leaving resources
>>>>> into reseted state causes issues to other code such as for checks done
>>>>> in pci_enable_resources(). Take a small step towards not resetting
>>>>> resources by delaying reset until the end of resource assignment and
>>>>> build failure list (fail_head) in sync with the reset to avoid leaving
>>>>> behind resources that cannot be restored (for the case where the caller
>>>>> provides fail_head in the first place to allow restore somewhere in the
>>>>> callchain, as is not all callers pass non-NULL fail_head).
>>>>>
>>>>> The Expansion ROM check is temporarily left in place while building the
>>>>> failure list until the upcoming change which reworks optional resource
>>>>> handling.
>>>>>
>>>>> Ideally, whole resource reset could be removed but doing that in a big
>>>>> step would make the impact non-tractable due to complexity of all
>>>>> related code.
>>>>>
>>>>> Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
>>>> I'm hitting the BUG_ON(!list_empty(&add_list)); in
>>>> pci_assign_unassigned_bus_resources() [1] with 6.15-rc5 and the the
>>>> pixel6 downstream pcie driver.
>>>>
>>>> I saw the thread where "a34d74877c66 PCI: Restore assigned resources
>>>> fully after release" fixes things for some other cases, but it's not the
>>>> case here.
>>>>
>>>> Reverting the following patches fixes the problem:
>>>> a34d74877c66 PCI: Restore assigned resources fully after release
>>>> 2499f5348431 PCI: Rework optional resource handling
>>>> 96336ec70264 PCI: Perform reset_resource() and build fail list in sync
>>> So it's confirmed that you needed to revert also this last commit
>>> 96336ec70264, not just the rework change?
>> I needed to revert 96336ec70264 as well otherwise the build fails.
>>>> In the working case the add_list list is empty throughout the entire
>>>> body of pci_assign_unassigned_bus_resources().
>>>>
>>>> In the failing case __pci_bus_size_bridges() leaves the add_list not
>>>> empty and __pci_bus_assign_resources() does not consume the list, thus
>>>> the BUG_ON. The failing case contains an extra print that's not shown
>>>> when reverting the blamed commits:
>>>> [ 13.951185][ T1101] pcieport 0000:00:00.0: bridge window [mem
>>>> 0x00100000-0x001fffff] to [bus 01-ff] add_size 100000 add_align 100000
>>>>
>>>> I've added some prints trying to describe the code path, see
>>>> https://paste.ofcode.org/Aeu2YBpLztc49ZDw3uUJmd#
>>>>
>>>> Failing case:
>>>> [ 13.944231][ T1101] pci 0000:01:00.0: [144d:a5a5] type 00 class
>>>> 0x000000 PCIe Endpoint
>>>> [ 13.944412][ T1101] pci 0000:01:00.0: BAR 0 [mem
>>>> 0x00000000-0x000fffff 64bit]
>>>> [ 13.944532][ T1101] pci 0000:01:00.0: ROM [mem 0x00000000-0x0000ffff
>>>> pref]
>>>> [ 13.944649][ T1101] pci 0000:01:00.0: enabling Extended Tags
>>>> [ 13.944844][ T1101] pci 0000:01:00.0: PME# supported from D0 D3hot D3cold
>>>> [ 13.945015][ T1101] pci 0000:01:00.0: 15.752 Gb/s available PCIe
>>>> bandwidth, limited by 8.0 GT/s PCIe x2 link at 0000:00:00.0 (capable of
>>>> 31.506 Gb/s with 16.0 GT/s PCIe x2 link)
>>>> [ 13.950616][ T1101] __pci_bus_size_bridges: before pbus_size_mem.
>>>> list empty? 1
>>>> [ 13.950784][ T1101] pbus_size_mem: 2. list empty? 1
>>>> [ 13.950886][ T1101] pbus_size_mem: 1 list empty? 0
>>>> [ 13.950982][ T1101] pbus_size_mem: 3. list empty? 0
>>>> [ 13.951082][ T1101] pbus_size_mem: 4. list empty? 0
>>>> [ 13.951185][ T1101] pcieport 0000:00:00.0: bridge window [mem
>>>> 0x00100000-0x001fffff] to [bus 01-ff] add_size 100000 add_align 100000
>>>> [ 13.951448][ T1101] __pci_bus_size_bridges: after pbus_size_mem. list
>>>> empty? 0
>>>> [ 13.951643][ T1101] pci_assign_unassigned_bus_resources: before
>>>> __pci_bus_assign_resources -> list empty? 0
>>>> [ 13.951924][ T1101] pcieport 0000:00:00.0: bridge window [mem
>>>> 0x40000000-0x401fffff]: assigned
>>>> [ 13.952248][ T1101] pci_assign_unassigned_bus_resources: after
>>>> __pci_bus_assign_resources -> list empty? 0
>>>> [ 13.952634][ T1101] ------------[ cut here ]------------
>>>> [ 13.952818][ T1101] kernel BUG at drivers/pci/setup-bus.c:2514!
>>>> [ 13.953045][ T1101] Internal error: Oops - BUG: 00000000f2000800 [#1]
>>>> SMP
>>>> ...
>>>> [ 13.976086][ T1101] Call trace:
>>>> [ 13.976206][ T1101] pci_assign_unassigned_bus_resources+0x110/0x114 (P)
>>>> [ 13.976462][ T1101] pci_rescan_bus+0x28/0x48
>>>> [ 13.976628][ T1101] exynos_pcie_rc_poweron
>>>>
>>>> Working case:
>>>> [ 13.786961][ T1120] pci 0000:01:00.0: [144d:a5a5] type 00 class
>>>> 0x000000 PCIe Endpoint
>>>> [ 13.787136][ T1120] pci 0000:01:00.0: BAR 0 [mem
>>>> 0x00000000-0x000fffff 64bit]
>>>> [ 13.787280][ T1120] pci 0000:01:00.0: ROM [mem 0x00000000-0x0000ffff
>>>> pref]
>>>> [ 13.787541][ T1120] pci 0000:01:00.0: enabling Extended Tags
>>>> [ 13.787808][ T1120] pci 0000:01:00.0: PME# supported from D0 D3hot D3cold
>>>> [ 13.787988][ T1120] pci 0000:01:00.0: 15.752 Gb/s available PCIe
>>>> bandwidth, limited by 8.0 GT/s PCIe x2 link at 0000:00:00.0 (capable of
>>>> 31.506 Gb/s with 16.0 GT/s PCIe x2 link)
>>>> [ 13.795279][ T1120] __pci_bus_size_bridges: before pbus_size_mem.
>>>> list empty? 1
>>>> [ 13.795408][ T1120] pbus_size_mem: 2. list empty? 1
>>>> [ 13.795495][ T1120] pbus_size_mem: 2. list empty? 1
>>>> [ 13.795577][ T1120] __pci_bus_size_bridges: after pbus_size_mem. list
>>>> empty? 1
>>>> [ 13.795692][ T1120] pci_assign_unassigned_bus_resources: before
>>>> __pci_bus_assign_resources -> list empty? 1
>>>> [ 13.795849][ T1120] pcieport 0000:00:00.0: bridge window [mem
>>>> 0x40000000-0x401fffff]: assigned
>>>> [ 13.796072][ T1120] pci_assign_unassigned_bus_resources: after
>>>> __pci_bus_assign_resources -> list empty? 1
>>>> [ 13.796662][ T1120] cpif: s5100_poweron_pcie: DBG: MSI sfr not set
>>>> up, yet(s5100_pdev is NULL)
>>>> [ 13.796666][ T1120] cpif: register_pcie: s51xx_pcie_init start
>>>>
>>>>
>>>> Any hints are welcomed. Thanks,
>>>> ta
>>> Hi and thanks for the report.
>> Hi! Thanks for the help. I've been out of office for the last 2 weeks,
>> sorry for the delayed reply.
>>
>>> The interesting part occurs inside reassign_resources_sorted() where most
>>> items are eliminated from realloc_head by the list_del().
>>>
>>> My guess is that somehow, the change in 96336ec70264 from !res->flags
>>> to the more complicated check somehow causes this. If the new check
>>> doesn't match and subsequently, no match is found from the head list, the
>>> loop will do continue and not remove the entry from realloc_head.
>> I added a print right there and it seems it's something else. See below.
>>> But it's hard to confirm without knowing what that resources realloc_head
>>> contains. Perhaps if you print the resources that are processed around
>>> that part of the code in reassign_resources_sorted(), comparing the log
>>> from the reverted code with the non-working case might help to understand
>>> what is different there and why. To understand better what is in the head
>>> list, it would be also useful to know from which device the resources were
>>> added into the head list in pdev_sort_resources().
>>>
>> I added the suggested prints
>> (https://paste.ofcode.org/DgmZGGgS6D36nWEzmfCqMm) on top of v6.15 with
>> the downstream PCIe pixel driver and I obtain the following. Note that
>> all added prints contain "tudor" for differentiation.
>>
>> [ 15.211179][ T1107] pci 0001:01:00.0: [144d:a5a5] type 00 class
>> 0x000000 PCIe Endpoint
>> [ 15.212248][ T1107] pci 0001:01:00.0: BAR 0 [mem
>> 0x00000000-0x000fffff 64bit]
>> [ 15.212775][ T1107] pci 0001:01:00.0: ROM [mem 0x00000000-0x0000ffff
>> pref]
>> [ 15.213195][ T1107] pci 0001:01:00.0: enabling Extended Tags
>> [ 15.213720][ T1107] pci 0001:01:00.0: PME# supported from D0 D3hot
>> D3cold
>> [ 15.214035][ T1107] pci 0001:01:00.0: 15.752 Gb/s available PCIe
>> bandwidth, limited by 8.0 GT/s PCIe x2 link at 0001:00:00.0 (capable of
>> 31.506 Gb/s with 16.0 GT/s PCIe x2 link)
>> [ 15.222286][ T1107] pci 0001:01:00.0: tudor: 1: pbus_size_mem: BAR 0
>> [mem 0x00000000-0x000fffff 64bit] list empty? 1
>> [ 15.222813][ T1107] pci 0001:01:00.0: tudor: 1: pbus_size_mem: ROM
>> [mem 0x00000000-0x0000ffff pref] list empty? 1
>> [ 15.224429][ T1107] pci 0001:01:00.0: tudor: 2: pbus_size_mem: ROM
>> [mem 0x00000000-0x0000ffff pref] list empty? 0
>> [ 15.224750][ T1107] pcieport 0001:00:00.0: bridge window [mem
>> 0x00100000-0x001fffff] to [bus 01-ff] add_size 100000 add_align 100000
>>
>> [ 15.225393][ T1107] tudor : pci_assign_unassigned_bus_resources:
>> before __pci_bus_assign_resources -> list empty? 0
>> [ 15.225594][ T1107] pcieport 0001:00:00.0: tudor:
>> pdev_sort_resources: bridge window [mem 0x00100000-0x001fffff] resource
>> added in head list
>> [ 15.226078][ T1107] pcieport 0001:00:00.0: bridge window [mem
>> 0x40000000-0x401fffff]: assigned
>> [ 15.226419][ T1107] tudor : pci_assign_unassigned_bus_resources:
>> after __pci_bus_assign_resources -> list empty? 0
>> [ 15.226442][ T1107] ------------[ cut here ]------------
>> [ 15.227587][ T1107] kernel BUG at drivers/pci/setup-bus.c:2522!
>> [ 15.227813][ T1107] Internal error: Oops - BUG: 00000000f2000800 [#1]
>> SMP
>> ...
>> [ 15.251570][ T1107] Call trace:
>> [ 15.251690][ T1107] pci_assign_unassigned_bus_resources+0x110/0x114 (P)
>> [ 15.251945][ T1107] pci_rescan_bus+0x28/0x48
>>
>> I obtain the following output when using the same prints adapted
>> (https://paste.ofcode.org/37w7RnKkPaCxyNhi5yhZPbZ) and with the blamed
>> commits reverted:
>> a34d74877c66 PCI: Restore assigned resources fully after release
>> 2499f5348431 PCI: Rework optional resource handling
>> 96336ec70264 PCI: Perform reset_resource() and build fail list in sync
>>
>> [ 15.200456][ T1102] pci 0000:01:00.0: [144d:a5a5] type 00 class
>> 0x000000 PCIe Endpoint
>> [ 15.200632][ T1102] pci 0000:01:00.0: BAR 0 [mem
>> 0x00000000-0x000fffff 64bit]
>> [ 15.200755][ T1102] pci 0000:01:00.0: ROM [mem 0x00000000-0x0000ffff
>> pref]
>> [ 15.200876][ T1102] pci 0000:01:00.0: enabling Extended Tags
>> [ 15.201075][ T1102] pci 0000:01:00.0: PME# supported from D0 D3hot D3cold
>> [ 15.201254][ T1102] pci 0000:01:00.0: 15.752 Gb/s available PCIe
>> bandwidth, limited by 8.0 GT/s PCIe x2 link at 0000:00:00.0 (capable of
>> 31.506 Gb/s with 16.0 GT/s PCIe x2 link)
>> [ 15.206555][ T1102] pci 0000:01:00.0: tudor: 1: pbus_size_mem: BAR 0
>> [mem 0x00000000-0x000fffff 64bit] list empty? 1
>> [ 15.206737][ T1102] pci 0000:01:00.0: tudor: 1: pbus_size_mem: ROM
>> [mem 0x00000000-0x0000ffff pref] list empty? 1
>> [ 15.206901][ T1102] tudor : pci_assign_unassigned_bus_resources:
>> before __pci_bus_assign_resources -> list empty? 1
>> [ 15.207072][ T1102] pcieport 0000:00:00.0: tudor:
>> pdev_sort_resources: bridge window [mem 0x00100000-0x002fffff] resource
>> added in head list
>> [ 15.207396][ T1102] pcieport 0000:00:00.0: bridge window [mem
>> 0x40000000-0x401fffff]: assigned
>> [ 15.208165][ T1102] tudor : pci_assign_unassigned_bus_resources:
>> after __pci_bus_assign_resources -> list empty? 1
>> [ 15.208783][ T1102] cpif: s5100_poweron_pcie: DBG: MSI sfr not set
>> up, yet(s5100_pdev is NULL)
>> [ 15.208786][ T1102] cpif: register_pcie: s51xx_pcie_init start
>
> I see my email client split the lines for the prints making the output
> very hard to read. Added the output here too:
> https://paste.ofcode.org/AEfjASQW8Z2jbMak5VkmpJ
With the following change things get back to how they were before
2499f5348431:
diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c
index 5247370010aa..1589dd8afa69 100644
--- a/drivers/pci/setup-bus.c
+++ b/drivers/pci/setup-bus.c
@@ -1214,9 +1214,10 @@ static int pbus_size_mem(struct pci_bus *bus,
unsigned long mask,
__func__, r_name, r, list_empty(realloc_head));
/* Put SRIOV requested res to the optional list */
- if (realloc_head && pci_resource_is_optional(dev, i)) {
+ if (realloc_head && pci_resource_is_iov(i)) {
add_align = max(pci_resource_alignment(dev, r), add_align);
- add_to_list(realloc_head, dev, r, 0, 0 /* Don't care */);
+ resource_set_size(r, 0);
+ add_to_list(realloc_head, dev, r, r_size, 0 /* Don't care */);
children_add_size += r_size;
pci_info(dev, "tudor: 2: %s: %s %pR list empty? %d\n",
__func__, r_name, r, list_empty(realloc_head));
--
2.49.0.1238.gf8c92423fb-goog
On Wed, 28 May 2025, Tudor Ambarus wrote:
> On 5/28/25 12:39 PM, Tudor Ambarus wrote:
> > On 5/28/25 12:22 PM, Tudor Ambarus wrote:
> >> On 5/6/25 4:53 PM, Ilpo Järvinen wrote:
> >>> On Tue, 6 May 2025, Tudor Ambarus wrote:
> >>>
> >>>> Hi!
> >>>>
> >>>> On 12/16/24 5:56 PM, Ilpo Järvinen wrote:
> >>>>> Resetting resource is problematic as it prevent attempting to allocate
> >>>>> the resource later, unless something in between restores the resource.
> >>>>> Similarly, if fail_head does not contain all resources that were reset,
> >>>>> those resource cannot be restored later.
> >>>>>
> >>>>> The entire reset/restore cycle adds complexity and leaving resources
> >>>>> into reseted state causes issues to other code such as for checks done
> >>>>> in pci_enable_resources(). Take a small step towards not resetting
> >>>>> resources by delaying reset until the end of resource assignment and
> >>>>> build failure list (fail_head) in sync with the reset to avoid leaving
> >>>>> behind resources that cannot be restored (for the case where the caller
> >>>>> provides fail_head in the first place to allow restore somewhere in the
> >>>>> callchain, as is not all callers pass non-NULL fail_head).
> >>>>>
> >>>>> The Expansion ROM check is temporarily left in place while building the
> >>>>> failure list until the upcoming change which reworks optional resource
> >>>>> handling.
> >>>>>
> >>>>> Ideally, whole resource reset could be removed but doing that in a big
> >>>>> step would make the impact non-tractable due to complexity of all
> >>>>> related code.
> >>>>>
> >>>>> Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
> >>>> I'm hitting the BUG_ON(!list_empty(&add_list)); in
> >>>> pci_assign_unassigned_bus_resources() [1] with 6.15-rc5 and the the
> >>>> pixel6 downstream pcie driver.
> >>>>
> >>>> I saw the thread where "a34d74877c66 PCI: Restore assigned resources
> >>>> fully after release" fixes things for some other cases, but it's not the
> >>>> case here.
> >>>>
> >>>> Reverting the following patches fixes the problem:
> >>>> a34d74877c66 PCI: Restore assigned resources fully after release
> >>>> 2499f5348431 PCI: Rework optional resource handling
> >>>> 96336ec70264 PCI: Perform reset_resource() and build fail list in sync
> >>> So it's confirmed that you needed to revert also this last commit
> >>> 96336ec70264, not just the rework change?
> >> I needed to revert 96336ec70264 as well otherwise the build fails.
> >>>> In the working case the add_list list is empty throughout the entire
> >>>> body of pci_assign_unassigned_bus_resources().
> >>>>
> >>>> In the failing case __pci_bus_size_bridges() leaves the add_list not
> >>>> empty and __pci_bus_assign_resources() does not consume the list, thus
> >>>> the BUG_ON. The failing case contains an extra print that's not shown
> >>>> when reverting the blamed commits:
> >>>> [ 13.951185][ T1101] pcieport 0000:00:00.0: bridge window [mem
> >>>> 0x00100000-0x001fffff] to [bus 01-ff] add_size 100000 add_align 100000
> >>>>
> >>>> I've added some prints trying to describe the code path, see
> >>>> https://paste.ofcode.org/Aeu2YBpLztc49ZDw3uUJmd#
> >>>>
> >>>> Failing case:
> >>>> [ 13.944231][ T1101] pci 0000:01:00.0: [144d:a5a5] type 00 class
> >>>> 0x000000 PCIe Endpoint
> >>>> [ 13.944412][ T1101] pci 0000:01:00.0: BAR 0 [mem
> >>>> 0x00000000-0x000fffff 64bit]
> >>>> [ 13.944532][ T1101] pci 0000:01:00.0: ROM [mem 0x00000000-0x0000ffff
> >>>> pref]
> >>>> [ 13.944649][ T1101] pci 0000:01:00.0: enabling Extended Tags
> >>>> [ 13.944844][ T1101] pci 0000:01:00.0: PME# supported from D0 D3hot D3cold
> >>>> [ 13.945015][ T1101] pci 0000:01:00.0: 15.752 Gb/s available PCIe
> >>>> bandwidth, limited by 8.0 GT/s PCIe x2 link at 0000:00:00.0 (capable of
> >>>> 31.506 Gb/s with 16.0 GT/s PCIe x2 link)
> >>>> [ 13.950616][ T1101] __pci_bus_size_bridges: before pbus_size_mem.
> >>>> list empty? 1
> >>>> [ 13.950784][ T1101] pbus_size_mem: 2. list empty? 1
> >>>> [ 13.950886][ T1101] pbus_size_mem: 1 list empty? 0
> >>>> [ 13.950982][ T1101] pbus_size_mem: 3. list empty? 0
> >>>> [ 13.951082][ T1101] pbus_size_mem: 4. list empty? 0
> >>>> [ 13.951185][ T1101] pcieport 0000:00:00.0: bridge window [mem
> >>>> 0x00100000-0x001fffff] to [bus 01-ff] add_size 100000 add_align 100000
> >>>> [ 13.951448][ T1101] __pci_bus_size_bridges: after pbus_size_mem. list
> >>>> empty? 0
> >>>> [ 13.951643][ T1101] pci_assign_unassigned_bus_resources: before
> >>>> __pci_bus_assign_resources -> list empty? 0
> >>>> [ 13.951924][ T1101] pcieport 0000:00:00.0: bridge window [mem
> >>>> 0x40000000-0x401fffff]: assigned
> >>>> [ 13.952248][ T1101] pci_assign_unassigned_bus_resources: after
> >>>> __pci_bus_assign_resources -> list empty? 0
> >>>> [ 13.952634][ T1101] ------------[ cut here ]------------
> >>>> [ 13.952818][ T1101] kernel BUG at drivers/pci/setup-bus.c:2514!
> >>>> [ 13.953045][ T1101] Internal error: Oops - BUG: 00000000f2000800 [#1]
> >>>> SMP
> >>>> ...
> >>>> [ 13.976086][ T1101] Call trace:
> >>>> [ 13.976206][ T1101] pci_assign_unassigned_bus_resources+0x110/0x114 (P)
> >>>> [ 13.976462][ T1101] pci_rescan_bus+0x28/0x48
> >>>> [ 13.976628][ T1101] exynos_pcie_rc_poweron
> >>>>
> >>>> Working case:
> >>>> [ 13.786961][ T1120] pci 0000:01:00.0: [144d:a5a5] type 00 class
> >>>> 0x000000 PCIe Endpoint
> >>>> [ 13.787136][ T1120] pci 0000:01:00.0: BAR 0 [mem
> >>>> 0x00000000-0x000fffff 64bit]
> >>>> [ 13.787280][ T1120] pci 0000:01:00.0: ROM [mem 0x00000000-0x0000ffff
> >>>> pref]
> >>>> [ 13.787541][ T1120] pci 0000:01:00.0: enabling Extended Tags
> >>>> [ 13.787808][ T1120] pci 0000:01:00.0: PME# supported from D0 D3hot D3cold
> >>>> [ 13.787988][ T1120] pci 0000:01:00.0: 15.752 Gb/s available PCIe
> >>>> bandwidth, limited by 8.0 GT/s PCIe x2 link at 0000:00:00.0 (capable of
> >>>> 31.506 Gb/s with 16.0 GT/s PCIe x2 link)
> >>>> [ 13.795279][ T1120] __pci_bus_size_bridges: before pbus_size_mem.
> >>>> list empty? 1
> >>>> [ 13.795408][ T1120] pbus_size_mem: 2. list empty? 1
> >>>> [ 13.795495][ T1120] pbus_size_mem: 2. list empty? 1
> >>>> [ 13.795577][ T1120] __pci_bus_size_bridges: after pbus_size_mem. list
> >>>> empty? 1
> >>>> [ 13.795692][ T1120] pci_assign_unassigned_bus_resources: before
> >>>> __pci_bus_assign_resources -> list empty? 1
> >>>> [ 13.795849][ T1120] pcieport 0000:00:00.0: bridge window [mem
> >>>> 0x40000000-0x401fffff]: assigned
> >>>> [ 13.796072][ T1120] pci_assign_unassigned_bus_resources: after
> >>>> __pci_bus_assign_resources -> list empty? 1
> >>>> [ 13.796662][ T1120] cpif: s5100_poweron_pcie: DBG: MSI sfr not set
> >>>> up, yet(s5100_pdev is NULL)
> >>>> [ 13.796666][ T1120] cpif: register_pcie: s51xx_pcie_init start
> >>>>
> >>>>
> >>>> Any hints are welcomed. Thanks,
> >>>> ta
> >>> Hi and thanks for the report.
> >> Hi! Thanks for the help. I've been out of office for the last 2 weeks,
> >> sorry for the delayed reply.
> >>
> >>> The interesting part occurs inside reassign_resources_sorted() where most
> >>> items are eliminated from realloc_head by the list_del().
> >>>
> >>> My guess is that somehow, the change in 96336ec70264 from !res->flags
> >>> to the more complicated check somehow causes this. If the new check
> >>> doesn't match and subsequently, no match is found from the head list, the
> >>> loop will do continue and not remove the entry from realloc_head.
> >> I added a print right there and it seems it's something else. See below.
> >>> But it's hard to confirm without knowing what that resources realloc_head
> >>> contains. Perhaps if you print the resources that are processed around
> >>> that part of the code in reassign_resources_sorted(), comparing the log
> >>> from the reverted code with the non-working case might help to understand
> >>> what is different there and why. To understand better what is in the head
> >>> list, it would be also useful to know from which device the resources were
> >>> added into the head list in pdev_sort_resources().
> >>>
> >> I added the suggested prints
> >> (https://paste.ofcode.org/DgmZGGgS6D36nWEzmfCqMm) on top of v6.15 with
> >> the downstream PCIe pixel driver and I obtain the following. Note that
> >> all added prints contain "tudor" for differentiation.
> >>
> >> [ 15.211179][ T1107] pci 0001:01:00.0: [144d:a5a5] type 00 class
> >> 0x000000 PCIe Endpoint
> >> [ 15.212248][ T1107] pci 0001:01:00.0: BAR 0 [mem
> >> 0x00000000-0x000fffff 64bit]
> >> [ 15.212775][ T1107] pci 0001:01:00.0: ROM [mem 0x00000000-0x0000ffff
> >> pref]
> >> [ 15.213195][ T1107] pci 0001:01:00.0: enabling Extended Tags
> >> [ 15.213720][ T1107] pci 0001:01:00.0: PME# supported from D0 D3hot
> >> D3cold
> >> [ 15.214035][ T1107] pci 0001:01:00.0: 15.752 Gb/s available PCIe
> >> bandwidth, limited by 8.0 GT/s PCIe x2 link at 0001:00:00.0 (capable of
> >> 31.506 Gb/s with 16.0 GT/s PCIe x2 link)
> >> [ 15.222286][ T1107] pci 0001:01:00.0: tudor: 1: pbus_size_mem: BAR 0
> >> [mem 0x00000000-0x000fffff 64bit] list empty? 1
> >> [ 15.222813][ T1107] pci 0001:01:00.0: tudor: 1: pbus_size_mem: ROM
> >> [mem 0x00000000-0x0000ffff pref] list empty? 1
> >> [ 15.224429][ T1107] pci 0001:01:00.0: tudor: 2: pbus_size_mem: ROM
> >> [mem 0x00000000-0x0000ffff pref] list empty? 0
> >> [ 15.224750][ T1107] pcieport 0001:00:00.0: bridge window [mem
> >> 0x00100000-0x001fffff] to [bus 01-ff] add_size 100000 add_align 100000
> >>
> >> [ 15.225393][ T1107] tudor : pci_assign_unassigned_bus_resources:
> >> before __pci_bus_assign_resources -> list empty? 0
> >> [ 15.225594][ T1107] pcieport 0001:00:00.0: tudor:
> >> pdev_sort_resources: bridge window [mem 0x00100000-0x001fffff] resource
> >> added in head list
> >> [ 15.226078][ T1107] pcieport 0001:00:00.0: bridge window [mem
> >> 0x40000000-0x401fffff]: assigned
> >> [ 15.226419][ T1107] tudor : pci_assign_unassigned_bus_resources:
> >> after __pci_bus_assign_resources -> list empty? 0
> >> [ 15.226442][ T1107] ------------[ cut here ]------------
> >> [ 15.227587][ T1107] kernel BUG at drivers/pci/setup-bus.c:2522!
> >> [ 15.227813][ T1107] Internal error: Oops - BUG: 00000000f2000800 [#1]
> >> SMP
> >> ...
> >> [ 15.251570][ T1107] Call trace:
> >> [ 15.251690][ T1107] pci_assign_unassigned_bus_resources+0x110/0x114 (P)
> >> [ 15.251945][ T1107] pci_rescan_bus+0x28/0x48
> >>
> >> I obtain the following output when using the same prints adapted
> >> (https://paste.ofcode.org/37w7RnKkPaCxyNhi5yhZPbZ) and with the blamed
> >> commits reverted:
> >> a34d74877c66 PCI: Restore assigned resources fully after release
> >> 2499f5348431 PCI: Rework optional resource handling
> >> 96336ec70264 PCI: Perform reset_resource() and build fail list in sync
> >>
> >> [ 15.200456][ T1102] pci 0000:01:00.0: [144d:a5a5] type 00 class
> >> 0x000000 PCIe Endpoint
> >> [ 15.200632][ T1102] pci 0000:01:00.0: BAR 0 [mem
> >> 0x00000000-0x000fffff 64bit]
> >> [ 15.200755][ T1102] pci 0000:01:00.0: ROM [mem 0x00000000-0x0000ffff
> >> pref]
> >> [ 15.200876][ T1102] pci 0000:01:00.0: enabling Extended Tags
> >> [ 15.201075][ T1102] pci 0000:01:00.0: PME# supported from D0 D3hot D3cold
> >> [ 15.201254][ T1102] pci 0000:01:00.0: 15.752 Gb/s available PCIe
> >> bandwidth, limited by 8.0 GT/s PCIe x2 link at 0000:00:00.0 (capable of
> >> 31.506 Gb/s with 16.0 GT/s PCIe x2 link)
> >> [ 15.206555][ T1102] pci 0000:01:00.0: tudor: 1: pbus_size_mem: BAR 0
> >> [mem 0x00000000-0x000fffff 64bit] list empty? 1
> >> [ 15.206737][ T1102] pci 0000:01:00.0: tudor: 1: pbus_size_mem: ROM
> >> [mem 0x00000000-0x0000ffff pref] list empty? 1
> >> [ 15.206901][ T1102] tudor : pci_assign_unassigned_bus_resources:
> >> before __pci_bus_assign_resources -> list empty? 1
> >> [ 15.207072][ T1102] pcieport 0000:00:00.0: tudor:
> >> pdev_sort_resources: bridge window [mem 0x00100000-0x002fffff] resource
> >> added in head list
> >> [ 15.207396][ T1102] pcieport 0000:00:00.0: bridge window [mem
> >> 0x40000000-0x401fffff]: assigned
> >> [ 15.208165][ T1102] tudor : pci_assign_unassigned_bus_resources:
> >> after __pci_bus_assign_resources -> list empty? 1
> >> [ 15.208783][ T1102] cpif: s5100_poweron_pcie: DBG: MSI sfr not set
> >> up, yet(s5100_pdev is NULL)
> >> [ 15.208786][ T1102] cpif: register_pcie: s51xx_pcie_init start
> >
> > I see my email client split the lines for the prints making the output
> > very hard to read. Added the output here too:
> > https://paste.ofcode.org/AEfjASQW8Z2jbMak5VkmpJ
>
> With the following change things get back to how they were before
> 2499f5348431:
>
> diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c
> index 5247370010aa..1589dd8afa69 100644
> --- a/drivers/pci/setup-bus.c
> +++ b/drivers/pci/setup-bus.c
> @@ -1214,9 +1214,10 @@ static int pbus_size_mem(struct pci_bus *bus,
> unsigned long mask,
> __func__, r_name, r, list_empty(realloc_head));
>
> /* Put SRIOV requested res to the optional list */
> - if (realloc_head && pci_resource_is_optional(dev, i)) {
> + if (realloc_head && pci_resource_is_iov(i)) {
> add_align = max(pci_resource_alignment(dev, r), add_align);
> - add_to_list(realloc_head, dev, r, 0, 0 /* Don't care */);
> + resource_set_size(r, 0);
> + add_to_list(realloc_head, dev, r, r_size, 0 /* Don't care */);
> children_add_size += r_size;
> pci_info(dev, "tudor: 2: %s: %s %pR list empty? %d\n",
> __func__, r_name, r, list_empty(realloc_head));
Okay, thanks for the additional data point, but I don't think the problem
is at this site. That removal of that resource_set_size(r, 0) is crucial
piece in my changes (and as is, it even doesn't extend far enough and we
can end up lose resources in some situations when their sizes are reset
like that).
This entire resource fitting and assignment code as whole is very complex
spanning many functions. The steps are roughly these:
1) calculate sizing in advance (which the above code relates to)
2) collect all resources we want to assign in a PCI bus subtree
3) opportunistically assign with the optional sizes
4) if everything was assigned, skip steps 5-7.
5) release some/all from step 2 assignments
6) assign only required resources
7) reassign optional parts (enlargen to include the optional sizes) what
we can
8) As a final sanity check, realloc_head is checked to have been fully
consumed by the earlier steps (~ no unchecked work todo remains)
In some cases, kernel iterates the entire process a few times (but
that mostly occurs when pci=realloc is given on the kernel cmdline).
Now, as discussed in my other replay, it looks at some point the algorithm
assigned the bridge window resource successfully, but it still remained in
the realloc_head for some reason if I parsed all the information from your
debug patch right. The resource should have been removed from realloc_head
before __assign_resources_sorted() exits if it got assigned, if it
wasn't, that's the bug we're looking for. (Alternatively, it could have
been that the assignment was never tried for the particular resource,
but it now looks that might not be the case here).
--
i.
Hi, On Mon, Dec 16, 2024 at 07:56:31PM +0200, Ilpo Järvinen wrote: > Resetting resource is problematic as it prevent attempting to allocate > the resource later, unless something in between restores the resource. > Similarly, if fail_head does not contain all resources that were reset, > those resource cannot be restored later. > > The entire reset/restore cycle adds complexity and leaving resources > into reseted state causes issues to other code such as for checks done > in pci_enable_resources(). Take a small step towards not resetting > resources by delaying reset until the end of resource assignment and > build failure list (fail_head) in sync with the reset to avoid leaving > behind resources that cannot be restored (for the case where the caller > provides fail_head in the first place to allow restore somewhere in the > callchain, as is not all callers pass non-NULL fail_head). > > The Expansion ROM check is temporarily left in place while building the > failure list until the upcoming change which reworks optional resource > handling. > > Ideally, whole resource reset could be removed but doing that in a big > step would make the impact non-tractable due to complexity of all > related code. > > Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com> With this patch in the mainline kernel, all mips:boston qemu emulations fail when running a 64-bit little endian configuration (64r6el_defconfig). The problem is that the PCI based IDE/ATA controller is not initialized. There are a number of pci error messages. pci_bus 0002:01: extended config space not accessible pci 0002:01:00.0: [8086:2922] type 00 class 0x010601 conventional PCI endpoint pci 0002:01:00.0: BAR 4 [io 0x0000-0x001f] pci 0002:01:00.0: BAR 5 [mem 0x00000000-0x00000fff] pci 0002:00:00.0: PCI bridge to [bus 01-ff] pci_bus 0002:01: busn_res: [bus 01-ff] end is updated to 01 pci 0002:00:00.0: bridge window [mem 0x16000000-0x160fffff]: assigned pci 0002:00:00.0: bridge window [mem size 0x00100000 64bit pref]: can't assign; no space pci 0002:00:00.0: bridge window [mem size 0x00100000 64bit pref]: failed to assign pci 0002:00:00.0: bridge window [io size 0x1000]: can't assign; no space pci 0002:00:00.0: bridge window [io size 0x1000]: failed to assign pci 0002:00:00.0: bridge window [mem size 0x00100000]: can't assign; bogus alignment pci 0002:00:00.0: bridge window [mem 0x16000000-0x160fffff 64bit pref]: assigned pci 0002:00:00.0: bridge window [io size 0x1000]: can't assign; no space pci 0002:00:00.0: bridge window [io size 0x1000]: failed to assign pci 0002:01:00.0: BAR 5 [mem size 0x00001000]: can't assign; no space pci 0002:01:00.0: BAR 5 [mem size 0x00001000]: failed to assign pci 0002:01:00.0: BAR 4 [io size 0x0020]: can't assign; no space pci 0002:01:00.0: BAR 4 [io size 0x0020]: failed to assign pci 0002:01:00.0: BAR 5 [mem size 0x00001000]: can't assign; no space pci 0002:01:00.0: BAR 5 [mem size 0x00001000]: failed to assign pci 0002:01:00.0: BAR 4 [io size 0x0020]: can't assign; no space pci 0002:01:00.0: BAR 4 [io size 0x0020]: failed to assign pci 0002:00:00.0: PCI bridge to [bus 01] pci 0002:00:00.0: bridge window [mem 0x16000000-0x160fffff 64bit pref] pci_bus 0002:00: Some PCI device resources are unassigned, try booting with pci=realloc pci_bus 0002:00: resource 4 [mem 0x16000000-0x160fffff] pci_bus 0002:01: resource 2 [mem 0x16000000-0x160fffff 64bit pref] ... pci 0002:00:00.0: enabling device (0000 -> 0002) ahci 0002:01:00.0: probe with driver ahci failed with error -12 Bisect points to this patch. Reverting it together with "PCI: Rework optional resource handling" fixes the problem. For comparison, after reverting the offending patches, the log messages are as follows. pci_bus 0002:00: root bus resource [bus 00-ff] pci_bus 0002:00: root bus resource [mem 0x16000000-0x160fffff] pci 0002:00:00.0: [10ee:7021] type 01 class 0x060400 PCIe Root Complex Integrated Endpoint pci 0002:00:00.0: PCI bridge to [bus 00] pci 0002:00:00.0: bridge window [io 0x0000-0x0fff] pci 0002:00:00.0: bridge window [mem 0x00000000-0x000fffff] pci 0002:00:00.0: bridge window [mem 0x00000000-0x000fffff 64bit pref] pci 0002:00:00.0: enabling Extended Tags pci 0002:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring pci_bus 0002:01: extended config space not accessible pci 0002:01:00.0: [8086:2922] type 00 class 0x010601 conventional PCI endpoint pci 0002:01:00.0: BAR 4 [io 0x0000-0x001f] pci 0002:01:00.0: BAR 5 [mem 0x00000000-0x00000fff] pci 0002:00:00.0: PCI bridge to [bus 01-ff] pci_bus 0002:01: busn_res: [bus 01-ff] end is updated to 01 pci 0002:00:00.0: bridge window [mem 0x16000000-0x160fffff]: assigned pci 0002:00:00.0: bridge window [mem size 0x00100000 64bit pref]: can't assign; no space pci 0002:00:00.0: bridge window [mem size 0x00100000 64bit pref]: failed to assign pci 0002:00:00.0: bridge window [io size 0x1000]: can't assign; no space pci 0002:00:00.0: bridge window [io size 0x1000]: failed to assign pci 0002:01:00.0: BAR 5 [mem 0x16000000-0x16000fff]: assigned pci 0002:01:00.0: BAR 4 [io size 0x0020]: can't assign; no space pci 0002:01:00.0: BAR 4 [io size 0x0020]: failed to assign pci 0002:00:00.0: PCI bridge to [bus 01] pci 0002:00:00.0: bridge window [mem 0x16000000-0x160fffff] pci_bus 0002:00: Some PCI device resources are unassigned, try booting with pci=realloc pci_bus 0002:00: resource 4 [mem 0x16000000-0x160fffff] pci_bus 0002:01: resource 1 [mem 0x16000000-0x160fffff] ... pci 0002:00:00.0: enabling device (0000 -> 0002) ahci 0002:01:00.0: enabling device (0000 -> 0002) ahci 0002:01:00.0: AHCI vers 0001.0000, 32 command slots, 1.5 Gbps, SATA mode ahci 0002:01:00.0: 6/6 ports implemented (port mask 0x3f) ahci 0002:01:00.0: flags: 64bit ncq only Bisect log is attached for reference. Guenter --- # bad: [609706855d90bcab6080ba2cd030b9af322a1f0c] Merge tag 'trace-latency-v6.15-3' of git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace # good: [acb4f33713b9f6cadb6143f211714c343465411c] Merge tag 'm68knommu-for-v6.15' of git://git.kernel.org/pub/scm/linux/kernel/git/gerg/m68knommu git bisect start 'HEAD' 'acb4f33713b9' # good: [cf05922d63e2ae6a9b1b52ff5236a44c3b29f78c] Merge tag 'drm-intel-gt-next-2025-03-12' of https://gitlab.freedesktop.org/drm/i915/kernel into drm-next git bisect good cf05922d63e2ae6a9b1b52ff5236a44c3b29f78c # bad: [93d52288679e29aaa44a6f12d5a02e8a90e742c5] Merge tag 'backlight-next-6.15' of git://git.kernel.org/pub/scm/linux/kernel/git/lee/backlight git bisect bad 93d52288679e29aaa44a6f12d5a02e8a90e742c5 # bad: [e5e0e6bebef3a21081fd1057c40468d4cff1a60d] Merge tag 'v6.15-p1' of git://git.kernel.org/pub/scm/linux/kernel/git/herbert/crypto-2.6 git bisect bad e5e0e6bebef3a21081fd1057c40468d4cff1a60d # bad: [dea140198b846f7432d78566b7b0b83979c72c2b] Merge branch 'pci/misc' git bisect bad dea140198b846f7432d78566b7b0b83979c72c2b # bad: [a113afb84ae63ec4c893bc3204945ef6f3bb89f7] Merge branch 'pci/endpoint' git bisect bad a113afb84ae63ec4c893bc3204945ef6f3bb89f7 # good: [a7a8e7996c1c114b50df5599229b1e7be38be3db] Merge branch 'pci/reset' git bisect good a7a8e7996c1c114b50df5599229b1e7be38be3db # bad: [7d4bcc0f2631e4ee10b5bcfff24a423d1c3c02a3] PCI: Move resource reassignment func declarations into pci/pci.h git bisect bad 7d4bcc0f2631e4ee10b5bcfff24a423d1c3c02a3 # good: [acba174d2e754346c07578ad2220258706a203e2] PCI: Use while loop and break instead of gotos git bisect good acba174d2e754346c07578ad2220258706a203e2 # good: [8884b5637b794ae541e8d6fb72102b1d8dba2b8d] PCI: Add debug print when releasing resources before retry git bisect good 8884b5637b794ae541e8d6fb72102b1d8dba2b8d # bad: [5af473941b56189423a7d16c05efabaf77299847] PCI: Increase Resizable BAR support from 512 GB to 128 TB git bisect bad 5af473941b56189423a7d16c05efabaf77299847 # bad: [96336ec702643aec2addb3b1cdb81d687fe362f0] PCI: Perform reset_resource() and build fail list in sync git bisect bad 96336ec702643aec2addb3b1cdb81d687fe362f0 # good: [e89df6d2beae847e931d84a190b192dfac41eba7] PCI: Use res->parent to check if resource is assigned git bisect good e89df6d2beae847e931d84a190b192dfac41eba7 # first bad commit: [96336ec702643aec2addb3b1cdb81d687fe362f0] PCI: Perform reset_resource() and build fail list in sync
On Mon, 31 Mar 2025, Guenter Roeck wrote: > On Mon, Dec 16, 2024 at 07:56:31PM +0200, Ilpo Järvinen wrote: > > Resetting resource is problematic as it prevent attempting to allocate > > the resource later, unless something in between restores the resource. > > Similarly, if fail_head does not contain all resources that were reset, > > those resource cannot be restored later. > > > > The entire reset/restore cycle adds complexity and leaving resources > > into reseted state causes issues to other code such as for checks done > > in pci_enable_resources(). Take a small step towards not resetting > > resources by delaying reset until the end of resource assignment and > > build failure list (fail_head) in sync with the reset to avoid leaving > > behind resources that cannot be restored (for the case where the caller > > provides fail_head in the first place to allow restore somewhere in the > > callchain, as is not all callers pass non-NULL fail_head). > > > > The Expansion ROM check is temporarily left in place while building the > > failure list until the upcoming change which reworks optional resource > > handling. > > > > Ideally, whole resource reset could be removed but doing that in a big > > step would make the impact non-tractable due to complexity of all > > related code. > > > > Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com> > > With this patch in the mainline kernel, all mips:boston qemu emulations > fail when running a 64-bit little endian configuration (64r6el_defconfig). > > The problem is that the PCI based IDE/ATA controller is not initialized. > There are a number of pci error messages. > > pci_bus 0002:01: extended config space not accessible > pci 0002:01:00.0: [8086:2922] type 00 class 0x010601 conventional PCI endpoint > pci 0002:01:00.0: BAR 4 [io 0x0000-0x001f] > pci 0002:01:00.0: BAR 5 [mem 0x00000000-0x00000fff] > pci 0002:00:00.0: PCI bridge to [bus 01-ff] > pci_bus 0002:01: busn_res: [bus 01-ff] end is updated to 01 > pci 0002:00:00.0: bridge window [mem 0x16000000-0x160fffff]: assigned > pci 0002:00:00.0: bridge window [mem size 0x00100000 64bit pref]: can't assign; no space > pci 0002:00:00.0: bridge window [mem size 0x00100000 64bit pref]: failed to assign > pci 0002:00:00.0: bridge window [io size 0x1000]: can't assign; no space > pci 0002:00:00.0: bridge window [io size 0x1000]: failed to assign > pci 0002:00:00.0: bridge window [mem size 0x00100000]: can't assign; bogus alignment > pci 0002:00:00.0: bridge window [mem 0x16000000-0x160fffff 64bit pref]: assigned > pci 0002:00:00.0: bridge window [io size 0x1000]: can't assign; no space > pci 0002:00:00.0: bridge window [io size 0x1000]: failed to assign > pci 0002:01:00.0: BAR 5 [mem size 0x00001000]: can't assign; no space > pci 0002:01:00.0: BAR 5 [mem size 0x00001000]: failed to assign > pci 0002:01:00.0: BAR 4 [io size 0x0020]: can't assign; no space > pci 0002:01:00.0: BAR 4 [io size 0x0020]: failed to assign > pci 0002:01:00.0: BAR 5 [mem size 0x00001000]: can't assign; no space > pci 0002:01:00.0: BAR 5 [mem size 0x00001000]: failed to assign > pci 0002:01:00.0: BAR 4 [io size 0x0020]: can't assign; no space > pci 0002:01:00.0: BAR 4 [io size 0x0020]: failed to assign > pci 0002:00:00.0: PCI bridge to [bus 01] > pci 0002:00:00.0: bridge window [mem 0x16000000-0x160fffff 64bit pref] > pci_bus 0002:00: Some PCI device resources are unassigned, try booting with pci=realloc > pci_bus 0002:00: resource 4 [mem 0x16000000-0x160fffff] > pci_bus 0002:01: resource 2 [mem 0x16000000-0x160fffff 64bit pref] > ... > pci 0002:00:00.0: enabling device (0000 -> 0002) > ahci 0002:01:00.0: probe with driver ahci failed with error -12 > > Bisect points to this patch. Reverting it together with "PCI: Rework > optional resource handling" fixes the problem. For comparison, after > reverting the offending patches, the log messages are as follows. > > pci_bus 0002:00: root bus resource [bus 00-ff] > pci_bus 0002:00: root bus resource [mem 0x16000000-0x160fffff] > pci 0002:00:00.0: [10ee:7021] type 01 class 0x060400 PCIe Root Complex Integrated Endpoint > pci 0002:00:00.0: PCI bridge to [bus 00] > pci 0002:00:00.0: bridge window [io 0x0000-0x0fff] > pci 0002:00:00.0: bridge window [mem 0x00000000-0x000fffff] > pci 0002:00:00.0: bridge window [mem 0x00000000-0x000fffff 64bit pref] > pci 0002:00:00.0: enabling Extended Tags > pci 0002:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring > pci_bus 0002:01: extended config space not accessible > pci 0002:01:00.0: [8086:2922] type 00 class 0x010601 conventional PCI endpoint > pci 0002:01:00.0: BAR 4 [io 0x0000-0x001f] > pci 0002:01:00.0: BAR 5 [mem 0x00000000-0x00000fff] > pci 0002:00:00.0: PCI bridge to [bus 01-ff] > pci_bus 0002:01: busn_res: [bus 01-ff] end is updated to 01 > pci 0002:00:00.0: bridge window [mem 0x16000000-0x160fffff]: assigned > pci 0002:00:00.0: bridge window [mem size 0x00100000 64bit pref]: can't assign; no space > pci 0002:00:00.0: bridge window [mem size 0x00100000 64bit pref]: failed to assign > pci 0002:00:00.0: bridge window [io size 0x1000]: can't assign; no space > pci 0002:00:00.0: bridge window [io size 0x1000]: failed to assign > pci 0002:01:00.0: BAR 5 [mem 0x16000000-0x16000fff]: assigned > pci 0002:01:00.0: BAR 4 [io size 0x0020]: can't assign; no space > pci 0002:01:00.0: BAR 4 [io size 0x0020]: failed to assign > pci 0002:00:00.0: PCI bridge to [bus 01] > pci 0002:00:00.0: bridge window [mem 0x16000000-0x160fffff] > pci_bus 0002:00: Some PCI device resources are unassigned, try booting with pci=realloc > pci_bus 0002:00: resource 4 [mem 0x16000000-0x160fffff] > pci_bus 0002:01: resource 1 [mem 0x16000000-0x160fffff] > ... > pci 0002:00:00.0: enabling device (0000 -> 0002) > ahci 0002:01:00.0: enabling device (0000 -> 0002) > ahci 0002:01:00.0: AHCI vers 0001.0000, 32 command slots, 1.5 Gbps, SATA mode > ahci 0002:01:00.0: 6/6 ports implemented (port mask 0x3f) > ahci 0002:01:00.0: flags: 64bit ncq only Hi, Thanks for reporting. Please add this to the command line to get the resource releasing between the steps to show: dyndbg="file drivers/pci/setup-bus.c +p" Also, the log snippet just shows it fails but it is impossible to know from it why the resource assigments do not fit so could you please provide a complete dmesg logs. Also providing the contents of /proc/iomem from the working case would save me quite a bit of decoding the iomem layout from the dmesgs. > Bisect log is attached for reference. > > Guenter > --- > # bad: [609706855d90bcab6080ba2cd030b9af322a1f0c] Merge tag 'trace-latency-v6.15-3' of git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace > # good: [acb4f33713b9f6cadb6143f211714c343465411c] Merge tag 'm68knommu-for-v6.15' of git://git.kernel.org/pub/scm/linux/kernel/git/gerg/m68knommu > git bisect start 'HEAD' 'acb4f33713b9' > # good: [cf05922d63e2ae6a9b1b52ff5236a44c3b29f78c] Merge tag 'drm-intel-gt-next-2025-03-12' of https://gitlab.freedesktop.org/drm/i915/kernel into drm-next > git bisect good cf05922d63e2ae6a9b1b52ff5236a44c3b29f78c > # bad: [93d52288679e29aaa44a6f12d5a02e8a90e742c5] Merge tag 'backlight-next-6.15' of git://git.kernel.org/pub/scm/linux/kernel/git/lee/backlight > git bisect bad 93d52288679e29aaa44a6f12d5a02e8a90e742c5 > # bad: [e5e0e6bebef3a21081fd1057c40468d4cff1a60d] Merge tag 'v6.15-p1' of git://git.kernel.org/pub/scm/linux/kernel/git/herbert/crypto-2.6 > git bisect bad e5e0e6bebef3a21081fd1057c40468d4cff1a60d > # bad: [dea140198b846f7432d78566b7b0b83979c72c2b] Merge branch 'pci/misc' > git bisect bad dea140198b846f7432d78566b7b0b83979c72c2b > # bad: [a113afb84ae63ec4c893bc3204945ef6f3bb89f7] Merge branch 'pci/endpoint' > git bisect bad a113afb84ae63ec4c893bc3204945ef6f3bb89f7 > # good: [a7a8e7996c1c114b50df5599229b1e7be38be3db] Merge branch 'pci/reset' > git bisect good a7a8e7996c1c114b50df5599229b1e7be38be3db > # bad: [7d4bcc0f2631e4ee10b5bcfff24a423d1c3c02a3] PCI: Move resource reassignment func declarations into pci/pci.h > git bisect bad 7d4bcc0f2631e4ee10b5bcfff24a423d1c3c02a3 > # good: [acba174d2e754346c07578ad2220258706a203e2] PCI: Use while loop and break instead of gotos > git bisect good acba174d2e754346c07578ad2220258706a203e2 > # good: [8884b5637b794ae541e8d6fb72102b1d8dba2b8d] PCI: Add debug print when releasing resources before retry > git bisect good 8884b5637b794ae541e8d6fb72102b1d8dba2b8d > # bad: [5af473941b56189423a7d16c05efabaf77299847] PCI: Increase Resizable BAR support from 512 GB to 128 TB > git bisect bad 5af473941b56189423a7d16c05efabaf77299847 > # bad: [96336ec702643aec2addb3b1cdb81d687fe362f0] PCI: Perform reset_resource() and build fail list in sync > git bisect bad 96336ec702643aec2addb3b1cdb81d687fe362f0 > # good: [e89df6d2beae847e931d84a190b192dfac41eba7] PCI: Use res->parent to check if resource is assigned > git bisect good e89df6d2beae847e931d84a190b192dfac41eba7 > # first bad commit: [96336ec702643aec2addb3b1cdb81d687fe362f0] PCI: Perform reset_resource() and build fail list in sync > -- i.
On 4/1/25 03:18, Ilpo Järvinen wrote: > On Mon, 31 Mar 2025, Guenter Roeck wrote: >> On Mon, Dec 16, 2024 at 07:56:31PM +0200, Ilpo Järvinen wrote: >>> Resetting resource is problematic as it prevent attempting to allocate >>> the resource later, unless something in between restores the resource. >>> Similarly, if fail_head does not contain all resources that were reset, >>> those resource cannot be restored later. >>> >>> The entire reset/restore cycle adds complexity and leaving resources >>> into reseted state causes issues to other code such as for checks done >>> in pci_enable_resources(). Take a small step towards not resetting >>> resources by delaying reset until the end of resource assignment and >>> build failure list (fail_head) in sync with the reset to avoid leaving >>> behind resources that cannot be restored (for the case where the caller >>> provides fail_head in the first place to allow restore somewhere in the >>> callchain, as is not all callers pass non-NULL fail_head). >>> >>> The Expansion ROM check is temporarily left in place while building the >>> failure list until the upcoming change which reworks optional resource >>> handling. >>> >>> Ideally, whole resource reset could be removed but doing that in a big >>> step would make the impact non-tractable due to complexity of all >>> related code. >>> >>> Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com> >> >> With this patch in the mainline kernel, all mips:boston qemu emulations >> fail when running a 64-bit little endian configuration (64r6el_defconfig). >> >> The problem is that the PCI based IDE/ATA controller is not initialized. >> There are a number of pci error messages. >> >> pci_bus 0002:01: extended config space not accessible >> pci 0002:01:00.0: [8086:2922] type 00 class 0x010601 conventional PCI endpoint >> pci 0002:01:00.0: BAR 4 [io 0x0000-0x001f] >> pci 0002:01:00.0: BAR 5 [mem 0x00000000-0x00000fff] >> pci 0002:00:00.0: PCI bridge to [bus 01-ff] >> pci_bus 0002:01: busn_res: [bus 01-ff] end is updated to 01 >> pci 0002:00:00.0: bridge window [mem 0x16000000-0x160fffff]: assigned >> pci 0002:00:00.0: bridge window [mem size 0x00100000 64bit pref]: can't assign; no space >> pci 0002:00:00.0: bridge window [mem size 0x00100000 64bit pref]: failed to assign >> pci 0002:00:00.0: bridge window [io size 0x1000]: can't assign; no space >> pci 0002:00:00.0: bridge window [io size 0x1000]: failed to assign >> pci 0002:00:00.0: bridge window [mem size 0x00100000]: can't assign; bogus alignment >> pci 0002:00:00.0: bridge window [mem 0x16000000-0x160fffff 64bit pref]: assigned >> pci 0002:00:00.0: bridge window [io size 0x1000]: can't assign; no space >> pci 0002:00:00.0: bridge window [io size 0x1000]: failed to assign >> pci 0002:01:00.0: BAR 5 [mem size 0x00001000]: can't assign; no space >> pci 0002:01:00.0: BAR 5 [mem size 0x00001000]: failed to assign >> pci 0002:01:00.0: BAR 4 [io size 0x0020]: can't assign; no space >> pci 0002:01:00.0: BAR 4 [io size 0x0020]: failed to assign >> pci 0002:01:00.0: BAR 5 [mem size 0x00001000]: can't assign; no space >> pci 0002:01:00.0: BAR 5 [mem size 0x00001000]: failed to assign >> pci 0002:01:00.0: BAR 4 [io size 0x0020]: can't assign; no space >> pci 0002:01:00.0: BAR 4 [io size 0x0020]: failed to assign >> pci 0002:00:00.0: PCI bridge to [bus 01] >> pci 0002:00:00.0: bridge window [mem 0x16000000-0x160fffff 64bit pref] >> pci_bus 0002:00: Some PCI device resources are unassigned, try booting with pci=realloc >> pci_bus 0002:00: resource 4 [mem 0x16000000-0x160fffff] >> pci_bus 0002:01: resource 2 [mem 0x16000000-0x160fffff 64bit pref] >> ... >> pci 0002:00:00.0: enabling device (0000 -> 0002) >> ahci 0002:01:00.0: probe with driver ahci failed with error -12 >> >> Bisect points to this patch. Reverting it together with "PCI: Rework >> optional resource handling" fixes the problem. For comparison, after >> reverting the offending patches, the log messages are as follows. >> >> pci_bus 0002:00: root bus resource [bus 00-ff] >> pci_bus 0002:00: root bus resource [mem 0x16000000-0x160fffff] >> pci 0002:00:00.0: [10ee:7021] type 01 class 0x060400 PCIe Root Complex Integrated Endpoint >> pci 0002:00:00.0: PCI bridge to [bus 00] >> pci 0002:00:00.0: bridge window [io 0x0000-0x0fff] >> pci 0002:00:00.0: bridge window [mem 0x00000000-0x000fffff] >> pci 0002:00:00.0: bridge window [mem 0x00000000-0x000fffff 64bit pref] >> pci 0002:00:00.0: enabling Extended Tags >> pci 0002:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring >> pci_bus 0002:01: extended config space not accessible >> pci 0002:01:00.0: [8086:2922] type 00 class 0x010601 conventional PCI endpoint >> pci 0002:01:00.0: BAR 4 [io 0x0000-0x001f] >> pci 0002:01:00.0: BAR 5 [mem 0x00000000-0x00000fff] >> pci 0002:00:00.0: PCI bridge to [bus 01-ff] >> pci_bus 0002:01: busn_res: [bus 01-ff] end is updated to 01 >> pci 0002:00:00.0: bridge window [mem 0x16000000-0x160fffff]: assigned >> pci 0002:00:00.0: bridge window [mem size 0x00100000 64bit pref]: can't assign; no space >> pci 0002:00:00.0: bridge window [mem size 0x00100000 64bit pref]: failed to assign >> pci 0002:00:00.0: bridge window [io size 0x1000]: can't assign; no space >> pci 0002:00:00.0: bridge window [io size 0x1000]: failed to assign >> pci 0002:01:00.0: BAR 5 [mem 0x16000000-0x16000fff]: assigned >> pci 0002:01:00.0: BAR 4 [io size 0x0020]: can't assign; no space >> pci 0002:01:00.0: BAR 4 [io size 0x0020]: failed to assign >> pci 0002:00:00.0: PCI bridge to [bus 01] >> pci 0002:00:00.0: bridge window [mem 0x16000000-0x160fffff] >> pci_bus 0002:00: Some PCI device resources are unassigned, try booting with pci=realloc >> pci_bus 0002:00: resource 4 [mem 0x16000000-0x160fffff] >> pci_bus 0002:01: resource 1 [mem 0x16000000-0x160fffff] >> ... >> pci 0002:00:00.0: enabling device (0000 -> 0002) >> ahci 0002:01:00.0: enabling device (0000 -> 0002) >> ahci 0002:01:00.0: AHCI vers 0001.0000, 32 command slots, 1.5 Gbps, SATA mode >> ahci 0002:01:00.0: 6/6 ports implemented (port mask 0x3f) >> ahci 0002:01:00.0: flags: 64bit ncq only > > Hi, > > Thanks for reporting. Please add this to the command line to get the > resource releasing between the steps to show: > > dyndbg="file drivers/pci/setup-bus.c +p" > > Also, the log snippet just shows it fails but it is impossible to know > from it why the resource assigments do not fit so could you please provide > a complete dmesg logs. Also providing the contents of /proc/iomem from the > working case would save me quite a bit of decoding the iomem layout from > the dmesgs. > You should find everything you need at http://server.roeck-us.net/qemu/mipsel64/. Please let me know if you need anything else. Thanks, Guenter
On Tue, 1 Apr 2025, Ilpo Järvinen wrote:
> On Mon, 31 Mar 2025, Guenter Roeck wrote:
> > On Mon, Dec 16, 2024 at 07:56:31PM +0200, Ilpo Järvinen wrote:
> > > Resetting resource is problematic as it prevent attempting to allocate
> > > the resource later, unless something in between restores the resource.
> > > Similarly, if fail_head does not contain all resources that were reset,
> > > those resource cannot be restored later.
> > >
> > > The entire reset/restore cycle adds complexity and leaving resources
> > > into reseted state causes issues to other code such as for checks done
> > > in pci_enable_resources(). Take a small step towards not resetting
> > > resources by delaying reset until the end of resource assignment and
> > > build failure list (fail_head) in sync with the reset to avoid leaving
> > > behind resources that cannot be restored (for the case where the caller
> > > provides fail_head in the first place to allow restore somewhere in the
> > > callchain, as is not all callers pass non-NULL fail_head).
> > >
> > > The Expansion ROM check is temporarily left in place while building the
> > > failure list until the upcoming change which reworks optional resource
> > > handling.
> > >
> > > Ideally, whole resource reset could be removed but doing that in a big
> > > step would make the impact non-tractable due to complexity of all
> > > related code.
> > >
> > > Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
> >
> > With this patch in the mainline kernel, all mips:boston qemu emulations
> > fail when running a 64-bit little endian configuration (64r6el_defconfig).
> >
> > The problem is that the PCI based IDE/ATA controller is not initialized.
> > There are a number of pci error messages.
> >
> > pci_bus 0002:01: extended config space not accessible
> > pci 0002:01:00.0: [8086:2922] type 00 class 0x010601 conventional PCI endpoint
> > pci 0002:01:00.0: BAR 4 [io 0x0000-0x001f]
> > pci 0002:01:00.0: BAR 5 [mem 0x00000000-0x00000fff]
> > pci 0002:00:00.0: PCI bridge to [bus 01-ff]
> > pci_bus 0002:01: busn_res: [bus 01-ff] end is updated to 01
> > pci 0002:00:00.0: bridge window [mem 0x16000000-0x160fffff]: assigned
> > pci 0002:00:00.0: bridge window [mem size 0x00100000 64bit pref]: can't assign; no space
> > pci 0002:00:00.0: bridge window [mem size 0x00100000 64bit pref]: failed to assign
> > pci 0002:00:00.0: bridge window [io size 0x1000]: can't assign; no space
> > pci 0002:00:00.0: bridge window [io size 0x1000]: failed to assign
> > pci 0002:00:00.0: bridge window [mem size 0x00100000]: can't assign; bogus alignment
> > pci 0002:00:00.0: bridge window [mem 0x16000000-0x160fffff 64bit pref]: assigned
> > pci 0002:00:00.0: bridge window [io size 0x1000]: can't assign; no space
> > pci 0002:00:00.0: bridge window [io size 0x1000]: failed to assign
> > pci 0002:01:00.0: BAR 5 [mem size 0x00001000]: can't assign; no space
> > pci 0002:01:00.0: BAR 5 [mem size 0x00001000]: failed to assign
> > pci 0002:01:00.0: BAR 4 [io size 0x0020]: can't assign; no space
> > pci 0002:01:00.0: BAR 4 [io size 0x0020]: failed to assign
> > pci 0002:01:00.0: BAR 5 [mem size 0x00001000]: can't assign; no space
> > pci 0002:01:00.0: BAR 5 [mem size 0x00001000]: failed to assign
> > pci 0002:01:00.0: BAR 4 [io size 0x0020]: can't assign; no space
> > pci 0002:01:00.0: BAR 4 [io size 0x0020]: failed to assign
> > pci 0002:00:00.0: PCI bridge to [bus 01]
> > pci 0002:00:00.0: bridge window [mem 0x16000000-0x160fffff 64bit pref]
> > pci_bus 0002:00: Some PCI device resources are unassigned, try booting with pci=realloc
> > pci_bus 0002:00: resource 4 [mem 0x16000000-0x160fffff]
> > pci_bus 0002:01: resource 2 [mem 0x16000000-0x160fffff 64bit pref]
> > ...
> > pci 0002:00:00.0: enabling device (0000 -> 0002)
> > ahci 0002:01:00.0: probe with driver ahci failed with error -12
> >
> > Bisect points to this patch. Reverting it together with "PCI: Rework
> > optional resource handling" fixes the problem. For comparison, after
> > reverting the offending patches, the log messages are as follows.
> >
> > pci_bus 0002:00: root bus resource [bus 00-ff]
> > pci_bus 0002:00: root bus resource [mem 0x16000000-0x160fffff]
> > pci 0002:00:00.0: [10ee:7021] type 01 class 0x060400 PCIe Root Complex Integrated Endpoint
> > pci 0002:00:00.0: PCI bridge to [bus 00]
> > pci 0002:00:00.0: bridge window [io 0x0000-0x0fff]
> > pci 0002:00:00.0: bridge window [mem 0x00000000-0x000fffff]
> > pci 0002:00:00.0: bridge window [mem 0x00000000-0x000fffff 64bit pref]
> > pci 0002:00:00.0: enabling Extended Tags
> > pci 0002:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
> > pci_bus 0002:01: extended config space not accessible
> > pci 0002:01:00.0: [8086:2922] type 00 class 0x010601 conventional PCI endpoint
> > pci 0002:01:00.0: BAR 4 [io 0x0000-0x001f]
> > pci 0002:01:00.0: BAR 5 [mem 0x00000000-0x00000fff]
> > pci 0002:00:00.0: PCI bridge to [bus 01-ff]
> > pci_bus 0002:01: busn_res: [bus 01-ff] end is updated to 01
> > pci 0002:00:00.0: bridge window [mem 0x16000000-0x160fffff]: assigned
> > pci 0002:00:00.0: bridge window [mem size 0x00100000 64bit pref]: can't assign; no space
> > pci 0002:00:00.0: bridge window [mem size 0x00100000 64bit pref]: failed to assign
> > pci 0002:00:00.0: bridge window [io size 0x1000]: can't assign; no space
> > pci 0002:00:00.0: bridge window [io size 0x1000]: failed to assign
> > pci 0002:01:00.0: BAR 5 [mem 0x16000000-0x16000fff]: assigned
> > pci 0002:01:00.0: BAR 4 [io size 0x0020]: can't assign; no space
> > pci 0002:01:00.0: BAR 4 [io size 0x0020]: failed to assign
> > pci 0002:00:00.0: PCI bridge to [bus 01]
> > pci 0002:00:00.0: bridge window [mem 0x16000000-0x160fffff]
> > pci_bus 0002:00: Some PCI device resources are unassigned, try booting with pci=realloc
> > pci_bus 0002:00: resource 4 [mem 0x16000000-0x160fffff]
> > pci_bus 0002:01: resource 1 [mem 0x16000000-0x160fffff]
> > ...
> > pci 0002:00:00.0: enabling device (0000 -> 0002)
> > ahci 0002:01:00.0: enabling device (0000 -> 0002)
> > ahci 0002:01:00.0: AHCI vers 0001.0000, 32 command slots, 1.5 Gbps, SATA mode
> > ahci 0002:01:00.0: 6/6 ports implemented (port mask 0x3f)
> > ahci 0002:01:00.0: flags: 64bit ncq only
>
> Hi,
>
> Thanks for reporting. Please add this to the command line to get the
> resource releasing between the steps to show:
>
> dyndbg="file drivers/pci/setup-bus.c +p"
>
> Also, the log snippet just shows it fails but it is impossible to know
> from it why the resource assigments do not fit so could you please provide
> a complete dmesg logs. Also providing the contents of /proc/iomem from the
> working case would save me quite a bit of decoding the iomem layout from
> the dmesgs.
Hi again,
If you could kindly include this patch into the test with pci_dbg()
enabled so the resource reset/restore is better tracked.
From e7cb98904ab2ee235bbc13b3e981332e944dd476 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Ilpo=20J=C3=A4rvinen?= <ilpo.jarvinen@linux.intel.com>
Date: Tue, 1 Apr 2025 15:05:13 +0300
Subject: [PATCH 1/1] PCI: Log reset and restore of resources
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
PCI resource fitting and assignment is complicated to track because it
performs many actions without any logging. One of these is resource
reset (zeroing the resource) and the restore during the multi-passed
resource fitting algorithm.
Resource reset does not play well with the other PCI code if the code
later wants to reattempt assignment of that resource, knowing that a
resource was left into the reset state without a pairing restore is
useful for understanding issues that show up as resource assignment
failures.
Add pci_dbg() to both reset and restore to be better able to track
what's going on within the resource fitting algorithm.
The resource fitting and assignment tracking should eventually be
convert to use trace events to cover all changes made into the
resources fully but that requires significant effort. In the meantime,
logging resource reset and restore with pci_dbg() seems low-hanging
fruit.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
---
drivers/pci/setup-bus.c | 16 ++++++++++++++--
1 file changed, 14 insertions(+), 2 deletions(-)
diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c
index 54d6f4fa3ce1..e67af19cb876 100644
--- a/drivers/pci/setup-bus.c
+++ b/drivers/pci/setup-bus.c
@@ -130,10 +130,15 @@ static resource_size_t get_res_add_align(struct list_head *head,
static void restore_dev_resource(struct pci_dev_resource *dev_res)
{
struct resource *res = dev_res->res;
+ struct pci_dev *dev = dev_res->dev;
+ int idx = pci_resource_num(dev, res);
+ const char *res_name = pci_resource_name(dev, idx);
res->start = dev_res->start;
res->end = dev_res->end;
res->flags = dev_res->flags;
+
+ pci_dbg(dev_res->dev, "%s %pR: resource restored\n", res_name, res);
}
static bool pdev_resources_assignable(struct pci_dev *dev)
@@ -218,8 +223,15 @@ bool pci_resource_is_optional(const struct pci_dev *dev, int resno)
return false;
}
-static inline void reset_resource(struct resource *res)
+static void reset_resource(struct pci_dev_resource *dev_res)
{
+ struct resource *res = dev_res->res;
+ struct pci_dev *dev = dev_res->dev;
+ int idx = pci_resource_num(dev, res);
+ const char *res_name = pci_resource_name(dev, idx);
+
+ pci_dbg(dev_res->dev, "%s %pR: resetting resource\n", res_name, res);
+
res->start = 0;
res->end = 0;
res->flags = 0;
@@ -573,7 +585,7 @@ static void __assign_resources_sorted(struct list_head *head,
0 /* don't care */);
}
- reset_resource(res);
+ reset_resource(dev_res);
}
free_list(head);
base-commit: 7d06015d936c861160803e020f68f413b5c3cd9d
--
2.39.5
On 4/1/25 05:07, Ilpo Järvinen wrote: > On Tue, 1 Apr 2025, Ilpo Järvinen wrote: > >> On Mon, 31 Mar 2025, Guenter Roeck wrote: >>> On Mon, Dec 16, 2024 at 07:56:31PM +0200, Ilpo Järvinen wrote: >>>> Resetting resource is problematic as it prevent attempting to allocate >>>> the resource later, unless something in between restores the resource. >>>> Similarly, if fail_head does not contain all resources that were reset, >>>> those resource cannot be restored later. >>>> >>>> The entire reset/restore cycle adds complexity and leaving resources >>>> into reseted state causes issues to other code such as for checks done >>>> in pci_enable_resources(). Take a small step towards not resetting >>>> resources by delaying reset until the end of resource assignment and >>>> build failure list (fail_head) in sync with the reset to avoid leaving >>>> behind resources that cannot be restored (for the case where the caller >>>> provides fail_head in the first place to allow restore somewhere in the >>>> callchain, as is not all callers pass non-NULL fail_head). >>>> >>>> The Expansion ROM check is temporarily left in place while building the >>>> failure list until the upcoming change which reworks optional resource >>>> handling. >>>> >>>> Ideally, whole resource reset could be removed but doing that in a big >>>> step would make the impact non-tractable due to complexity of all >>>> related code. >>>> >>>> Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com> >>> >>> With this patch in the mainline kernel, all mips:boston qemu emulations >>> fail when running a 64-bit little endian configuration (64r6el_defconfig). >>> >>> The problem is that the PCI based IDE/ATA controller is not initialized. >>> There are a number of pci error messages. >>> >>> pci_bus 0002:01: extended config space not accessible >>> pci 0002:01:00.0: [8086:2922] type 00 class 0x010601 conventional PCI endpoint >>> pci 0002:01:00.0: BAR 4 [io 0x0000-0x001f] >>> pci 0002:01:00.0: BAR 5 [mem 0x00000000-0x00000fff] >>> pci 0002:00:00.0: PCI bridge to [bus 01-ff] >>> pci_bus 0002:01: busn_res: [bus 01-ff] end is updated to 01 >>> pci 0002:00:00.0: bridge window [mem 0x16000000-0x160fffff]: assigned >>> pci 0002:00:00.0: bridge window [mem size 0x00100000 64bit pref]: can't assign; no space >>> pci 0002:00:00.0: bridge window [mem size 0x00100000 64bit pref]: failed to assign >>> pci 0002:00:00.0: bridge window [io size 0x1000]: can't assign; no space >>> pci 0002:00:00.0: bridge window [io size 0x1000]: failed to assign >>> pci 0002:00:00.0: bridge window [mem size 0x00100000]: can't assign; bogus alignment >>> pci 0002:00:00.0: bridge window [mem 0x16000000-0x160fffff 64bit pref]: assigned >>> pci 0002:00:00.0: bridge window [io size 0x1000]: can't assign; no space >>> pci 0002:00:00.0: bridge window [io size 0x1000]: failed to assign >>> pci 0002:01:00.0: BAR 5 [mem size 0x00001000]: can't assign; no space >>> pci 0002:01:00.0: BAR 5 [mem size 0x00001000]: failed to assign >>> pci 0002:01:00.0: BAR 4 [io size 0x0020]: can't assign; no space >>> pci 0002:01:00.0: BAR 4 [io size 0x0020]: failed to assign >>> pci 0002:01:00.0: BAR 5 [mem size 0x00001000]: can't assign; no space >>> pci 0002:01:00.0: BAR 5 [mem size 0x00001000]: failed to assign >>> pci 0002:01:00.0: BAR 4 [io size 0x0020]: can't assign; no space >>> pci 0002:01:00.0: BAR 4 [io size 0x0020]: failed to assign >>> pci 0002:00:00.0: PCI bridge to [bus 01] >>> pci 0002:00:00.0: bridge window [mem 0x16000000-0x160fffff 64bit pref] >>> pci_bus 0002:00: Some PCI device resources are unassigned, try booting with pci=realloc >>> pci_bus 0002:00: resource 4 [mem 0x16000000-0x160fffff] >>> pci_bus 0002:01: resource 2 [mem 0x16000000-0x160fffff 64bit pref] >>> ... >>> pci 0002:00:00.0: enabling device (0000 -> 0002) >>> ahci 0002:01:00.0: probe with driver ahci failed with error -12 >>> >>> Bisect points to this patch. Reverting it together with "PCI: Rework >>> optional resource handling" fixes the problem. For comparison, after >>> reverting the offending patches, the log messages are as follows. >>> >>> pci_bus 0002:00: root bus resource [bus 00-ff] >>> pci_bus 0002:00: root bus resource [mem 0x16000000-0x160fffff] >>> pci 0002:00:00.0: [10ee:7021] type 01 class 0x060400 PCIe Root Complex Integrated Endpoint >>> pci 0002:00:00.0: PCI bridge to [bus 00] >>> pci 0002:00:00.0: bridge window [io 0x0000-0x0fff] >>> pci 0002:00:00.0: bridge window [mem 0x00000000-0x000fffff] >>> pci 0002:00:00.0: bridge window [mem 0x00000000-0x000fffff 64bit pref] >>> pci 0002:00:00.0: enabling Extended Tags >>> pci 0002:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring >>> pci_bus 0002:01: extended config space not accessible >>> pci 0002:01:00.0: [8086:2922] type 00 class 0x010601 conventional PCI endpoint >>> pci 0002:01:00.0: BAR 4 [io 0x0000-0x001f] >>> pci 0002:01:00.0: BAR 5 [mem 0x00000000-0x00000fff] >>> pci 0002:00:00.0: PCI bridge to [bus 01-ff] >>> pci_bus 0002:01: busn_res: [bus 01-ff] end is updated to 01 >>> pci 0002:00:00.0: bridge window [mem 0x16000000-0x160fffff]: assigned >>> pci 0002:00:00.0: bridge window [mem size 0x00100000 64bit pref]: can't assign; no space >>> pci 0002:00:00.0: bridge window [mem size 0x00100000 64bit pref]: failed to assign >>> pci 0002:00:00.0: bridge window [io size 0x1000]: can't assign; no space >>> pci 0002:00:00.0: bridge window [io size 0x1000]: failed to assign >>> pci 0002:01:00.0: BAR 5 [mem 0x16000000-0x16000fff]: assigned >>> pci 0002:01:00.0: BAR 4 [io size 0x0020]: can't assign; no space >>> pci 0002:01:00.0: BAR 4 [io size 0x0020]: failed to assign >>> pci 0002:00:00.0: PCI bridge to [bus 01] >>> pci 0002:00:00.0: bridge window [mem 0x16000000-0x160fffff] >>> pci_bus 0002:00: Some PCI device resources are unassigned, try booting with pci=realloc >>> pci_bus 0002:00: resource 4 [mem 0x16000000-0x160fffff] >>> pci_bus 0002:01: resource 1 [mem 0x16000000-0x160fffff] >>> ... >>> pci 0002:00:00.0: enabling device (0000 -> 0002) >>> ahci 0002:01:00.0: enabling device (0000 -> 0002) >>> ahci 0002:01:00.0: AHCI vers 0001.0000, 32 command slots, 1.5 Gbps, SATA mode >>> ahci 0002:01:00.0: 6/6 ports implemented (port mask 0x3f) >>> ahci 0002:01:00.0: flags: 64bit ncq only >> >> Hi, >> >> Thanks for reporting. Please add this to the command line to get the >> resource releasing between the steps to show: >> >> dyndbg="file drivers/pci/setup-bus.c +p" >> >> Also, the log snippet just shows it fails but it is impossible to know >> from it why the resource assigments do not fit so could you please provide >> a complete dmesg logs. Also providing the contents of /proc/iomem from the >> working case would save me quite a bit of decoding the iomem layout from >> the dmesgs. > > Hi again, > > If you could kindly include this patch into the test with pci_dbg() > enabled so the resource reset/restore is better tracked. > Same link as before (http://server.roeck-us.net/qemu/mipsel64/). The log with the patch applied is in bad-extra.log. Hope this helps, Guenter
On Tue, 1 Apr 2025, Guenter Roeck wrote: > On 4/1/25 05:07, Ilpo Järvinen wrote: > > On Tue, 1 Apr 2025, Ilpo Järvinen wrote: > > > > > On Mon, 31 Mar 2025, Guenter Roeck wrote: > > > > On Mon, Dec 16, 2024 at 07:56:31PM +0200, Ilpo Järvinen wrote: > > > > > Resetting resource is problematic as it prevent attempting to allocate > > > > > the resource later, unless something in between restores the resource. > > > > > Similarly, if fail_head does not contain all resources that were > > > > > reset, > > > > > those resource cannot be restored later. > > > > > > > > > > The entire reset/restore cycle adds complexity and leaving resources > > > > > into reseted state causes issues to other code such as for checks done > > > > > in pci_enable_resources(). Take a small step towards not resetting > > > > > resources by delaying reset until the end of resource assignment and > > > > > build failure list (fail_head) in sync with the reset to avoid leaving > > > > > behind resources that cannot be restored (for the case where the > > > > > caller > > > > > provides fail_head in the first place to allow restore somewhere in > > > > > the > > > > > callchain, as is not all callers pass non-NULL fail_head). > > > > > > > > > > The Expansion ROM check is temporarily left in place while building > > > > > the > > > > > failure list until the upcoming change which reworks optional resource > > > > > handling. > > > > > > > > > > Ideally, whole resource reset could be removed but doing that in a big > > > > > step would make the impact non-tractable due to complexity of all > > > > > related code. > > > > > > > > > > Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com> > > > > > > > > With this patch in the mainline kernel, all mips:boston qemu emulations > > > > fail when running a 64-bit little endian configuration > > > > (64r6el_defconfig). > > > > > > > > The problem is that the PCI based IDE/ATA controller is not initialized. > > > > There are a number of pci error messages. > > > > > > > > pci_bus 0002:01: extended config space not accessible > > > > pci 0002:01:00.0: [8086:2922] type 00 class 0x010601 conventional PCI > > > > endpoint > > > > pci 0002:01:00.0: BAR 4 [io 0x0000-0x001f] > > > > pci 0002:01:00.0: BAR 5 [mem 0x00000000-0x00000fff] > > > > pci 0002:00:00.0: PCI bridge to [bus 01-ff] > > > > pci_bus 0002:01: busn_res: [bus 01-ff] end is updated to 01 > > > > pci 0002:00:00.0: bridge window [mem 0x16000000-0x160fffff]: assigned > > > > pci 0002:00:00.0: bridge window [mem size 0x00100000 64bit pref]: can't > > > > assign; no space > > > > pci 0002:00:00.0: bridge window [mem size 0x00100000 64bit pref]: failed > > > > to assign > > > > pci 0002:00:00.0: bridge window [io size 0x1000]: can't assign; no > > > > space > > > > pci 0002:00:00.0: bridge window [io size 0x1000]: failed to assign > > > > pci 0002:00:00.0: bridge window [mem size 0x00100000]: can't assign; > > > > bogus alignment > > > > pci 0002:00:00.0: bridge window [mem 0x16000000-0x160fffff 64bit pref]: > > > > assigned > > > > pci 0002:00:00.0: bridge window [io size 0x1000]: can't assign; no > > > > space > > > > pci 0002:00:00.0: bridge window [io size 0x1000]: failed to assign > > > > pci 0002:01:00.0: BAR 5 [mem size 0x00001000]: can't assign; no space > > > > pci 0002:01:00.0: BAR 5 [mem size 0x00001000]: failed to assign > > > > pci 0002:01:00.0: BAR 4 [io size 0x0020]: can't assign; no space > > > > pci 0002:01:00.0: BAR 4 [io size 0x0020]: failed to assign > > > > pci 0002:01:00.0: BAR 5 [mem size 0x00001000]: can't assign; no space > > > > pci 0002:01:00.0: BAR 5 [mem size 0x00001000]: failed to assign > > > > pci 0002:01:00.0: BAR 4 [io size 0x0020]: can't assign; no space > > > > pci 0002:01:00.0: BAR 4 [io size 0x0020]: failed to assign > > > > pci 0002:00:00.0: PCI bridge to [bus 01] > > > > pci 0002:00:00.0: bridge window [mem 0x16000000-0x160fffff 64bit pref] > > > > pci_bus 0002:00: Some PCI device resources are unassigned, try booting > > > > with pci=realloc > > > > pci_bus 0002:00: resource 4 [mem 0x16000000-0x160fffff] > > > > pci_bus 0002:01: resource 2 [mem 0x16000000-0x160fffff 64bit pref] > > > > ... > > > > pci 0002:00:00.0: enabling device (0000 -> 0002) > > > > ahci 0002:01:00.0: probe with driver ahci failed with error -12 > > > > > > > > Bisect points to this patch. Reverting it together with "PCI: Rework > > > > optional resource handling" fixes the problem. For comparison, after > > > > reverting the offending patches, the log messages are as follows. > > > > > > > > pci_bus 0002:00: root bus resource [bus 00-ff] > > > > pci_bus 0002:00: root bus resource [mem 0x16000000-0x160fffff] > > > > pci 0002:00:00.0: [10ee:7021] type 01 class 0x060400 PCIe Root Complex > > > > Integrated Endpoint > > > > pci 0002:00:00.0: PCI bridge to [bus 00] > > > > pci 0002:00:00.0: bridge window [io 0x0000-0x0fff] > > > > pci 0002:00:00.0: bridge window [mem 0x00000000-0x000fffff] > > > > pci 0002:00:00.0: bridge window [mem 0x00000000-0x000fffff 64bit pref] > > > > pci 0002:00:00.0: enabling Extended Tags > > > > pci 0002:00:00.0: bridge configuration invalid ([bus 00-00]), > > > > reconfiguring > > > > pci_bus 0002:01: extended config space not accessible > > > > pci 0002:01:00.0: [8086:2922] type 00 class 0x010601 conventional PCI > > > > endpoint > > > > pci 0002:01:00.0: BAR 4 [io 0x0000-0x001f] > > > > pci 0002:01:00.0: BAR 5 [mem 0x00000000-0x00000fff] > > > > pci 0002:00:00.0: PCI bridge to [bus 01-ff] > > > > pci_bus 0002:01: busn_res: [bus 01-ff] end is updated to 01 > > > > pci 0002:00:00.0: bridge window [mem 0x16000000-0x160fffff]: assigned > > > > pci 0002:00:00.0: bridge window [mem size 0x00100000 64bit pref]: can't > > > > assign; no space > > > > pci 0002:00:00.0: bridge window [mem size 0x00100000 64bit pref]: failed > > > > to assign > > > > pci 0002:00:00.0: bridge window [io size 0x1000]: can't assign; no > > > > space > > > > pci 0002:00:00.0: bridge window [io size 0x1000]: failed to assign > > > > pci 0002:01:00.0: BAR 5 [mem 0x16000000-0x16000fff]: assigned > > > > pci 0002:01:00.0: BAR 4 [io size 0x0020]: can't assign; no space > > > > pci 0002:01:00.0: BAR 4 [io size 0x0020]: failed to assign > > > > pci 0002:00:00.0: PCI bridge to [bus 01] > > > > pci 0002:00:00.0: bridge window [mem 0x16000000-0x160fffff] > > > > pci_bus 0002:00: Some PCI device resources are unassigned, try booting > > > > with pci=realloc > > > > pci_bus 0002:00: resource 4 [mem 0x16000000-0x160fffff] > > > > pci_bus 0002:01: resource 1 [mem 0x16000000-0x160fffff] > > > > ... > > > > pci 0002:00:00.0: enabling device (0000 -> 0002) > > > > ahci 0002:01:00.0: enabling device (0000 -> 0002) > > > > ahci 0002:01:00.0: AHCI vers 0001.0000, 32 command slots, 1.5 Gbps, SATA > > > > mode > > > > ahci 0002:01:00.0: 6/6 ports implemented (port mask 0x3f) > > > > ahci 0002:01:00.0: flags: 64bit ncq only > > > > > > Hi, > > > > > > Thanks for reporting. Please add this to the command line to get the > > > resource releasing between the steps to show: > > > > > > dyndbg="file drivers/pci/setup-bus.c +p" > > > > > > Also, the log snippet just shows it fails but it is impossible to know > > > from it why the resource assigments do not fit so could you please provide > > > a complete dmesg logs. Also providing the contents of /proc/iomem from the > > > working case would save me quite a bit of decoding the iomem layout from > > > the dmesgs. > > > > Hi again, > > > > If you could kindly include this patch into the test with pci_dbg() > > enabled so the resource reset/restore is better tracked. > > > > Same link as before (http://server.roeck-us.net/qemu/mipsel64/). > The log with the patch applied is in bad-extra.log. That log wasn't taken from a bad case but it doesn't matter anymore as I could test this with qemu myself, thanks for providing enough to make it easy to reproduce & test it locally :-). The problem is caused by assign+release cycle being destructive on start aligned resources because successful assigment overwrites the start field. I'll send a patch to fix the problem once I've given it a bit more thought as this resource fitting code is somewhat complex. -- i.
Hello Ilpo,
On Tue, Apr 01, 2025 at 08:38:48PM +0300, Ilpo Järvinen wrote:
> That log wasn't taken from a bad case but it doesn't matter anymore as I
> could test this with qemu myself, thanks for providing enough to make it
> easy to reproduce & test it locally :-).
>
> The problem is caused by assign+release cycle being destructive on start
> aligned resources because successful assigment overwrites the start field.
> I'll send a patch to fix the problem once I've given it a bit more thought
> as this resource fitting code is somewhat complex.
BTW, same thing here on a different SoC:
https://lore.kernel.org/lkml/hrcsm2bo4ysqj2ggejndlou32cdc7yiknnm5nrlcoz4d64wall@7te4dfqsoe3y/T/#u
There are kernel logs there, too, although I don't have dynamic debug enabled
in the kernel.
Interestingly, bisect pointed me initially to a different commit. Reverting
it helped, but just on one board (QuartzPro64).
And this is iomem:
0010f000-0010f0ff : 10f000.sram sram@10f000
00200000-e2bbffff : System RAM
02010000-0474ffff : Kernel code
04750000-0498ffff : reserved
04990000-0508ffff : Kernel data
daa00000-e29fffff : reserved
e2bc0000-ecbbffff : reserved
e2bc0000-ecbbffff : reserved
ecbc0000-efffffff : System RAM
ecbc7000-ecbdffff : reserved
f0000000-f00fffff : a40000000.pcie config
f0200000-f0ffffff : pcie@fe150000
f0200000-f020ffff : 0000:00:00.0
f0300000-f03fffff : PCI Bus 0000:01
f0300000-f0303fff : 0000:01:00.0
f0300000-f0303fff : nvme
f0304000-f03040ff : 0000:01:00.0
f0304000-f03040ff : nvme
f2000000-f20fffff : a40800000.pcie config
f2200000-f2ffffff : pcie@fe170000
f2200000-f27fffff : PCI Bus 0002:21
f2200000-f220ffff : 0002:21:00.0
f2400000-f27fffff : 0002:21:00.0
f2800000-f280ffff : 0002:20:00.0
f3000000-f30fffff : a40c00000.pcie config
f3200000-f3ffffff : pcie@fe180000
f3200000-f320ffff : 0003:30:00.0
f3300000-f33fffff : PCI Bus 0003:31
f3300000-f3303fff : 0003:31:00.0
f3304000-f3304fff : 0003:31:00.0
f3304000-f3304fff : r8169
fb000000-fb1fffff : fb000000.gpu gpu@fb000000
fc00c100-fc3fffff : fc000000.usb usb@fc000000
fc400000-fc407fff : usb@fc400000
fc400000-fc407fff : xhci-hcd.10.auto usb@fc400000
fc40c100-fc7fffff : fc400000.usb usb@fc400000
fc800000-fc83ffff : fc800000.usb usb@fc800000
fc840000-fc87ffff : fc840000.usb usb@fc840000
fc880000-fc8bffff : fc880000.usb usb@fc880000
fc8c0000-fc8fffff : fc8c0000.usb usb@fc8c0000
fc900000-fc900dff : fc900000.iommu
fc910000-fc910dff : fc900000.iommu
fd600000-fd6fffff : fd600000.sram sram@fd600000
fd8a0000-fd8a00ff : fd8a0000.gpio gpio@fd8a0000
fdb50000-fdb507ff : fdb50000.video-codec video-codec@fdb50000
fdb50800-fdb5083f : fdb50800.iommu iommu@fdb50800
fdb80000-fdb8017f : fdb80000.rga rga@fdb80000
fdba0000-fdba07ff : fdba0000.video-codec video-codec@fdba0000
fdba0800-fdba083f : fdba0800.iommu iommu@fdba0800
fdba4800-fdba483f : fdba4800.iommu iommu@fdba4800
fdba8800-fdba883f : fdba8800.iommu iommu@fdba8800
fdbac800-fdbac83f : fdbac800.iommu iommu@fdbac800
fdc70000-fdc707ff : fdc70000.video-codec video-codec@fdc70000
fdd90000-fdd941ff : fdd90000.vop vop
fdd95000-fdd95fff : fdd90000.vop gamma-lut
fdd97e00-fdd97eff : fdd97e00.iommu iommu@fdd97e00
fdd97f00-fdd97fff : fdd97e00.iommu iommu@fdd97e00
fddf0000-fddf0fff : fddf0000.i2s i2s@fddf0000
fddf4000-fddf4fff : fddf4000.i2s i2s@fddf4000
fde80000-fde9ffff : fde80000.hdmi hdmi@fde80000
fdea0000-fdebffff : fdea0000.hdmi hdmi@fdea0000
fdee0000-fdee5fff : fdee0000.hdmi_receiver hdmi_receiver@fdee0000
fe060000-fe06ffff : fe060000.dfi dfi@fe060000
fe150000-fe15ffff : a40000000.pcie apb
fe170000-fe17ffff : a40800000.pcie apb
fe180000-fe18ffff : a40c00000.pcie apb
fe1b0000-fe1bffff : fe1b0000.ethernet ethernet@fe1b0000
fe210000-fe210fff : fe210000.sata sata@fe210000
fe2c0000-fe2c3fff : fe2c0000.mmc mmc@fe2c0000
fe2e0000-fe2effff : fe2e0000.mmc mmc@fe2e0000
fe470000-fe470fff : fe470000.i2s i2s@fe470000
fe600000-fe60ffff : GICD
fe680000-fe77ffff : GICR
fea10000-fea13fff : dma-controller@fea10000
fea10000-fea13fff : fea10000.dma-controller dma-controller@fea10000
fea30000-fea33fff : dma-controller@fea30000
fea30000-fea33fff : fea30000.dma-controller dma-controller@fea30000
feaa0000-feaa0fff : feaa0000.i2c i2c@feaa0000
feaf0000-feaf00ff : feaf0000.watchdog watchdog@feaf0000
feb20000-feb20fff : feb20000.spi spi@feb20000
feb50000-feb500ff : serial
fec00000-fec003ff : fec00000.tsadc tsadc@fec00000
fec10000-fec1ffff : fec10000.adc adc@fec10000
fec20000-fec200ff : fec20000.gpio gpio@fec20000
fec30000-fec300ff : fec30000.gpio gpio@fec30000
fec40000-fec400ff : fec40000.gpio gpio@fec40000
fec50000-fec500ff : fec50000.gpio gpio@fec50000
fec90000-fec90fff : fec90000.i2c i2c@fec90000
fed10000-fed13fff : dma-controller@fed10000
fed10000-fed13fff : fed10000.dma-controller dma-controller@fed10000
fed60000-fed61fff : fed60000.phy phy@fed60000
fed70000-fed71fff : fed70000.phy phy@fed70000
fed80000-fed8ffff : fed80000.phy phy@fed80000
fed90000-fed9ffff : fed90000.phy phy@fed90000
fee00000-fee000ff : fee00000.phy phy@fee00000
fee10000-fee100ff : fee10000.phy phy@fee10000
fee20000-fee200ff : fee20000.phy phy@fee20000
fee80000-fee9ffff : fee80000.phy phy@fee80000
ff001000-ff0effff : ff001000.sram sram@ff001000
100000000-3fbffffff : System RAM
3ec000000-3fbffffff : reserved
3fc500000-3ffefffff : System RAM
4f0000000-4ffffffff : System RAM
4fc611000-4fc6d0fff : reserved
4fc6d1000-4fded1fff : reserved
4fded2000-4fdf91fff : reserved
4fdf93000-4fdf96fff : reserved
4fdf97000-4fdfabfff : reserved
4fdfac000-4fe051fff : reserved
4fe052000-4ffffffff : reserved
900000000-93fffffff : pcie@fe150000
900000000-93fffffff : 0000:00:00.0
980000000-9bfffffff : pcie@fe170000
9c0000000-9ffffffff : pcie@fe180000
a40000000-a403fffff : a40000000.pcie dbi
a40800000-a40bfffff : a40800000.pcie dbi
a40c00000-a40ffffff : a40c00000.pcie dbi
Thank you,
o.
> --
> i.
On Fri, 11 Apr 2025, Ondřej Jirman wrote: > Hello Ilpo, > > On Tue, Apr 01, 2025 at 08:38:48PM +0300, Ilpo Järvinen wrote: > > That log wasn't taken from a bad case but it doesn't matter anymore as I > > could test this with qemu myself, thanks for providing enough to make it > > easy to reproduce & test it locally :-). > > > > The problem is caused by assign+release cycle being destructive on start > > aligned resources because successful assigment overwrites the start field. > > I'll send a patch to fix the problem once I've given it a bit more thought > > as this resource fitting code is somewhat complex. > > BTW, same thing here on a different SoC: > > https://lore.kernel.org/lkml/hrcsm2bo4ysqj2ggejndlou32cdc7yiknnm5nrlcoz4d64wall@7te4dfqsoe3y/T/#u > > There are kernel logs there, too, although I don't have dynamic debug enabled > in the kernel. > > Interestingly, bisect pointed me initially to a different commit. Reverting > it helped, but just on one board (QuartzPro64). Hi, Since you didn't mention it, I guess you haven't tried the fix: https://patchwork.kernel.org/project/linux-pci/patch/20250403093137.1481-1-ilpo.jarvinen@linux.intel.com/ -- i. > And this is iomem: > > 0010f000-0010f0ff : 10f000.sram sram@10f000 > 00200000-e2bbffff : System RAM > 02010000-0474ffff : Kernel code > 04750000-0498ffff : reserved > 04990000-0508ffff : Kernel data > daa00000-e29fffff : reserved > e2bc0000-ecbbffff : reserved > e2bc0000-ecbbffff : reserved > ecbc0000-efffffff : System RAM > ecbc7000-ecbdffff : reserved > f0000000-f00fffff : a40000000.pcie config > f0200000-f0ffffff : pcie@fe150000 > f0200000-f020ffff : 0000:00:00.0 > f0300000-f03fffff : PCI Bus 0000:01 > f0300000-f0303fff : 0000:01:00.0 > f0300000-f0303fff : nvme > f0304000-f03040ff : 0000:01:00.0 > f0304000-f03040ff : nvme > f2000000-f20fffff : a40800000.pcie config > f2200000-f2ffffff : pcie@fe170000 > f2200000-f27fffff : PCI Bus 0002:21 > f2200000-f220ffff : 0002:21:00.0 > f2400000-f27fffff : 0002:21:00.0 > f2800000-f280ffff : 0002:20:00.0 > f3000000-f30fffff : a40c00000.pcie config > f3200000-f3ffffff : pcie@fe180000 > f3200000-f320ffff : 0003:30:00.0 > f3300000-f33fffff : PCI Bus 0003:31 > f3300000-f3303fff : 0003:31:00.0 > f3304000-f3304fff : 0003:31:00.0 > f3304000-f3304fff : r8169 > fb000000-fb1fffff : fb000000.gpu gpu@fb000000 > fc00c100-fc3fffff : fc000000.usb usb@fc000000 > fc400000-fc407fff : usb@fc400000 > fc400000-fc407fff : xhci-hcd.10.auto usb@fc400000 > fc40c100-fc7fffff : fc400000.usb usb@fc400000 > fc800000-fc83ffff : fc800000.usb usb@fc800000 > fc840000-fc87ffff : fc840000.usb usb@fc840000 > fc880000-fc8bffff : fc880000.usb usb@fc880000 > fc8c0000-fc8fffff : fc8c0000.usb usb@fc8c0000 > fc900000-fc900dff : fc900000.iommu > fc910000-fc910dff : fc900000.iommu > fd600000-fd6fffff : fd600000.sram sram@fd600000 > fd8a0000-fd8a00ff : fd8a0000.gpio gpio@fd8a0000 > fdb50000-fdb507ff : fdb50000.video-codec video-codec@fdb50000 > fdb50800-fdb5083f : fdb50800.iommu iommu@fdb50800 > fdb80000-fdb8017f : fdb80000.rga rga@fdb80000 > fdba0000-fdba07ff : fdba0000.video-codec video-codec@fdba0000 > fdba0800-fdba083f : fdba0800.iommu iommu@fdba0800 > fdba4800-fdba483f : fdba4800.iommu iommu@fdba4800 > fdba8800-fdba883f : fdba8800.iommu iommu@fdba8800 > fdbac800-fdbac83f : fdbac800.iommu iommu@fdbac800 > fdc70000-fdc707ff : fdc70000.video-codec video-codec@fdc70000 > fdd90000-fdd941ff : fdd90000.vop vop > fdd95000-fdd95fff : fdd90000.vop gamma-lut > fdd97e00-fdd97eff : fdd97e00.iommu iommu@fdd97e00 > fdd97f00-fdd97fff : fdd97e00.iommu iommu@fdd97e00 > fddf0000-fddf0fff : fddf0000.i2s i2s@fddf0000 > fddf4000-fddf4fff : fddf4000.i2s i2s@fddf4000 > fde80000-fde9ffff : fde80000.hdmi hdmi@fde80000 > fdea0000-fdebffff : fdea0000.hdmi hdmi@fdea0000 > fdee0000-fdee5fff : fdee0000.hdmi_receiver hdmi_receiver@fdee0000 > fe060000-fe06ffff : fe060000.dfi dfi@fe060000 > fe150000-fe15ffff : a40000000.pcie apb > fe170000-fe17ffff : a40800000.pcie apb > fe180000-fe18ffff : a40c00000.pcie apb > fe1b0000-fe1bffff : fe1b0000.ethernet ethernet@fe1b0000 > fe210000-fe210fff : fe210000.sata sata@fe210000 > fe2c0000-fe2c3fff : fe2c0000.mmc mmc@fe2c0000 > fe2e0000-fe2effff : fe2e0000.mmc mmc@fe2e0000 > fe470000-fe470fff : fe470000.i2s i2s@fe470000 > fe600000-fe60ffff : GICD > fe680000-fe77ffff : GICR > fea10000-fea13fff : dma-controller@fea10000 > fea10000-fea13fff : fea10000.dma-controller dma-controller@fea10000 > fea30000-fea33fff : dma-controller@fea30000 > fea30000-fea33fff : fea30000.dma-controller dma-controller@fea30000 > feaa0000-feaa0fff : feaa0000.i2c i2c@feaa0000 > feaf0000-feaf00ff : feaf0000.watchdog watchdog@feaf0000 > feb20000-feb20fff : feb20000.spi spi@feb20000 > feb50000-feb500ff : serial > fec00000-fec003ff : fec00000.tsadc tsadc@fec00000 > fec10000-fec1ffff : fec10000.adc adc@fec10000 > fec20000-fec200ff : fec20000.gpio gpio@fec20000 > fec30000-fec300ff : fec30000.gpio gpio@fec30000 > fec40000-fec400ff : fec40000.gpio gpio@fec40000 > fec50000-fec500ff : fec50000.gpio gpio@fec50000 > fec90000-fec90fff : fec90000.i2c i2c@fec90000 > fed10000-fed13fff : dma-controller@fed10000 > fed10000-fed13fff : fed10000.dma-controller dma-controller@fed10000 > fed60000-fed61fff : fed60000.phy phy@fed60000 > fed70000-fed71fff : fed70000.phy phy@fed70000 > fed80000-fed8ffff : fed80000.phy phy@fed80000 > fed90000-fed9ffff : fed90000.phy phy@fed90000 > fee00000-fee000ff : fee00000.phy phy@fee00000 > fee10000-fee100ff : fee10000.phy phy@fee10000 > fee20000-fee200ff : fee20000.phy phy@fee20000 > fee80000-fee9ffff : fee80000.phy phy@fee80000 > ff001000-ff0effff : ff001000.sram sram@ff001000 > 100000000-3fbffffff : System RAM > 3ec000000-3fbffffff : reserved > 3fc500000-3ffefffff : System RAM > 4f0000000-4ffffffff : System RAM > 4fc611000-4fc6d0fff : reserved > 4fc6d1000-4fded1fff : reserved > 4fded2000-4fdf91fff : reserved > 4fdf93000-4fdf96fff : reserved > 4fdf97000-4fdfabfff : reserved > 4fdfac000-4fe051fff : reserved > 4fe052000-4ffffffff : reserved > 900000000-93fffffff : pcie@fe150000 > 900000000-93fffffff : 0000:00:00.0 > 980000000-9bfffffff : pcie@fe170000 > 9c0000000-9ffffffff : pcie@fe180000 > a40000000-a403fffff : a40000000.pcie dbi > a40800000-a40bfffff : a40800000.pcie dbi > a40c00000-a40ffffff : a40c00000.pcie dbi > > Thank you, > o. > > > -- > > i. >
Hello Ilpo, On Mon, Apr 14, 2025 at 12:52:15PM +0300, Ilpo Järvinen wrote: > On Fri, 11 Apr 2025, Ondřej Jirman wrote: > > > Hello Ilpo, > > > > On Tue, Apr 01, 2025 at 08:38:48PM +0300, Ilpo Järvinen wrote: > > > That log wasn't taken from a bad case but it doesn't matter anymore as I > > > could test this with qemu myself, thanks for providing enough to make it > > > easy to reproduce & test it locally :-). > > > > > > The problem is caused by assign+release cycle being destructive on start > > > aligned resources because successful assigment overwrites the start field. > > > I'll send a patch to fix the problem once I've given it a bit more thought > > > as this resource fitting code is somewhat complex. > > > > BTW, same thing here on a different SoC: > > > > https://lore.kernel.org/lkml/hrcsm2bo4ysqj2ggejndlou32cdc7yiknnm5nrlcoz4d64wall@7te4dfqsoe3y/T/#u > > > > There are kernel logs there, too, although I don't have dynamic debug enabled > > in the kernel. > > > > Interestingly, bisect pointed me initially to a different commit. Reverting > > it helped, but just on one board (QuartzPro64). > > Hi, > > Since you didn't mention it, I guess you haven't tried the fix: > > https://patchwork.kernel.org/project/linux-pci/patch/20250403093137.1481-1-ilpo.jarvinen@linux.intel.com/ This patch works. Thank you. One difference compared to 6.14 that I'm noticing in the kernel log is that "save config" sequences now are printed twice for unpopulated port. Not sure if it's related to your patches. Previously it was printed just once. Kind regards, o. rockchip-dw-pcie a40800000.pcie: PCI host bridge to bus 0002:20 pci_bus 0002:20: root bus resource [bus 20-2f] pci_bus 0002:20: root bus resource [io 0x300000-0x3fffff] (bus address [0xf2100000-0xf21fffff]) pci_bus 0002:20: root bus resource [mem 0xf2200000-0xf2ffffff] pci_bus 0002:20: root bus resource [mem 0x980000000-0x9bfffffff] (bus address [0x40000000-0x7fffffff]) pci_bus 0002:20: scanning bus pci 0002:20:00.0: [1d87:3588] type 01 class 0x060400 PCIe Root Port pci 0002:20:00.0: ROM [mem 0x00000000-0x0000ffff pref] pci 0002:20:00.0: PCI bridge to [bus 01-ff] pci 0002:20:00.0: bridge window [io 0x0000-0x0fff] pci 0002:20:00.0: bridge window [mem 0x00000000-0x000fffff] pci 0002:20:00.0: bridge window [mem 0x00000000-0x000fffff 64bit pref] pci 0002:20:00.0: supports D1 D2 pci 0002:20:00.0: PME# supported from D0 D1 D3hot pci 0002:20:00.0: PME# disabled pci 0002:20:00.0: Adding to iommu group 9 pci 0002:20:00.0: vgaarb: pci_notify pci_bus 0002:20: fixups for bus pci 0002:20:00.0: scanning [bus 01-ff] behind bridge, pass 0 pci 0002:20:00.0: Primary bus is hard wired to 0 pci 0002:20:00.0: bridge configuration invalid ([bus 01-ff]), reconfiguring pci 0002:20:00.0: scanning [bus 00-00] behind bridge, pass 1 pci_bus 0002:20: bus scan returning with max=21 pci 0002:20:00.0: ROM [mem 0xf2200000-0xf220ffff pref]: assigned pci 0002:20:00.0: PCI bridge to [bus 21] pci_bus 0002:20: resource 4 [io 0x300000-0x3fffff] pci_bus 0002:20: resource 5 [mem 0xf2200000-0xf2ffffff] pci_bus 0002:20: resource 6 [mem 0x980000000-0x9bfffffff] pcieport 0002:20:00.0: vgaarb: pci_notify pcieport 0002:20:00.0: assign IRQ: got 148 pcieport 0002:20:00.0: PME: Signaling with IRQ 157 pcieport 0002:20:00.0: AER: enabled with IRQ 158 pcieport 0002:20:00.0: bwctrl: enabled with IRQ 157 pcieport 0002:20:00.0: save config 0x00: 0x35881d87 pcieport 0002:20:00.0: save config 0x04: 0x00100507 pcieport 0002:20:00.0: save config 0x08: 0x06040001 pcieport 0002:20:00.0: save config 0x0c: 0x00010000 pcieport 0002:20:00.0: save config 0x10: 0x00000000 pcieport 0002:20:00.0: save config 0x14: 0x00000000 pcieport 0002:20:00.0: save config 0x18: 0x00212120 pcieport 0002:20:00.0: save config 0x1c: 0x000000f0 pcieport 0002:20:00.0: save config 0x20: 0x0000fff0 pcieport 0002:20:00.0: save config 0x24: 0x0001fff1 pcieport 0002:20:00.0: save config 0x28: 0x00000000 pcieport 0002:20:00.0: save config 0x2c: 0x00000000 pcieport 0002:20:00.0: save config 0x30: 0x00000000 pcieport 0002:20:00.0: save config 0x34: 0x00000040 pcieport 0002:20:00.0: save config 0x38: 0x00000000 pcieport 0002:20:00.0: save config 0x3c: 0x00020194 pcieport 0002:20:00.0: vgaarb: pci_notify pcieport 0002:20:00.0: save config 0x00: 0x35881d87 pcieport 0002:20:00.0: save config 0x04: 0x00100507 pcieport 0002:20:00.0: save config 0x08: 0x06040001 pcieport 0002:20:00.0: save config 0x0c: 0x00010000 pcieport 0002:20:00.0: save config 0x10: 0x00000000 pcieport 0002:20:00.0: save config 0x14: 0x00000000 pcieport 0002:20:00.0: save config 0x18: 0x00212120 pcieport 0002:20:00.0: save config 0x1c: 0x000000f0 pcieport 0002:20:00.0: save config 0x20: 0x0000fff0 pcieport 0002:20:00.0: save config 0x24: 0x0001fff1 pcieport 0002:20:00.0: save config 0x28: 0x00000000 pcieport 0002:20:00.0: save config 0x2c: 0x00000000 pcieport 0002:20:00.0: save config 0x30: 0x00000000 pcieport 0002:20:00.0: save config 0x34: 0x00000040 pcieport 0002:20:00.0: save config 0x38: 0x00000000 pcieport 0002:20:00.0: save config 0x3c: 0x00020194 pcieport 0002:20:00.0: PME# enabled > -- > i. > > > And this is iomem: > > > > 0010f000-0010f0ff : 10f000.sram sram@10f000 > > 00200000-e2bbffff : System RAM > > 02010000-0474ffff : Kernel code > > 04750000-0498ffff : reserved > > 04990000-0508ffff : Kernel data > > daa00000-e29fffff : reserved > > e2bc0000-ecbbffff : reserved > > e2bc0000-ecbbffff : reserved > > ecbc0000-efffffff : System RAM > > ecbc7000-ecbdffff : reserved > > f0000000-f00fffff : a40000000.pcie config > > f0200000-f0ffffff : pcie@fe150000 > > f0200000-f020ffff : 0000:00:00.0 > > f0300000-f03fffff : PCI Bus 0000:01 > > f0300000-f0303fff : 0000:01:00.0 > > f0300000-f0303fff : nvme > > f0304000-f03040ff : 0000:01:00.0 > > f0304000-f03040ff : nvme > > f2000000-f20fffff : a40800000.pcie config > > f2200000-f2ffffff : pcie@fe170000 > > f2200000-f27fffff : PCI Bus 0002:21 > > f2200000-f220ffff : 0002:21:00.0 > > f2400000-f27fffff : 0002:21:00.0 > > f2800000-f280ffff : 0002:20:00.0 > > f3000000-f30fffff : a40c00000.pcie config > > f3200000-f3ffffff : pcie@fe180000 > > f3200000-f320ffff : 0003:30:00.0 > > f3300000-f33fffff : PCI Bus 0003:31 > > f3300000-f3303fff : 0003:31:00.0 > > f3304000-f3304fff : 0003:31:00.0 > > f3304000-f3304fff : r8169 > > fb000000-fb1fffff : fb000000.gpu gpu@fb000000 > > fc00c100-fc3fffff : fc000000.usb usb@fc000000 > > fc400000-fc407fff : usb@fc400000 > > fc400000-fc407fff : xhci-hcd.10.auto usb@fc400000 > > fc40c100-fc7fffff : fc400000.usb usb@fc400000 > > fc800000-fc83ffff : fc800000.usb usb@fc800000 > > fc840000-fc87ffff : fc840000.usb usb@fc840000 > > fc880000-fc8bffff : fc880000.usb usb@fc880000 > > fc8c0000-fc8fffff : fc8c0000.usb usb@fc8c0000 > > fc900000-fc900dff : fc900000.iommu > > fc910000-fc910dff : fc900000.iommu > > fd600000-fd6fffff : fd600000.sram sram@fd600000 > > fd8a0000-fd8a00ff : fd8a0000.gpio gpio@fd8a0000 > > fdb50000-fdb507ff : fdb50000.video-codec video-codec@fdb50000 > > fdb50800-fdb5083f : fdb50800.iommu iommu@fdb50800 > > fdb80000-fdb8017f : fdb80000.rga rga@fdb80000 > > fdba0000-fdba07ff : fdba0000.video-codec video-codec@fdba0000 > > fdba0800-fdba083f : fdba0800.iommu iommu@fdba0800 > > fdba4800-fdba483f : fdba4800.iommu iommu@fdba4800 > > fdba8800-fdba883f : fdba8800.iommu iommu@fdba8800 > > fdbac800-fdbac83f : fdbac800.iommu iommu@fdbac800 > > fdc70000-fdc707ff : fdc70000.video-codec video-codec@fdc70000 > > fdd90000-fdd941ff : fdd90000.vop vop > > fdd95000-fdd95fff : fdd90000.vop gamma-lut > > fdd97e00-fdd97eff : fdd97e00.iommu iommu@fdd97e00 > > fdd97f00-fdd97fff : fdd97e00.iommu iommu@fdd97e00 > > fddf0000-fddf0fff : fddf0000.i2s i2s@fddf0000 > > fddf4000-fddf4fff : fddf4000.i2s i2s@fddf4000 > > fde80000-fde9ffff : fde80000.hdmi hdmi@fde80000 > > fdea0000-fdebffff : fdea0000.hdmi hdmi@fdea0000 > > fdee0000-fdee5fff : fdee0000.hdmi_receiver hdmi_receiver@fdee0000 > > fe060000-fe06ffff : fe060000.dfi dfi@fe060000 > > fe150000-fe15ffff : a40000000.pcie apb > > fe170000-fe17ffff : a40800000.pcie apb > > fe180000-fe18ffff : a40c00000.pcie apb > > fe1b0000-fe1bffff : fe1b0000.ethernet ethernet@fe1b0000 > > fe210000-fe210fff : fe210000.sata sata@fe210000 > > fe2c0000-fe2c3fff : fe2c0000.mmc mmc@fe2c0000 > > fe2e0000-fe2effff : fe2e0000.mmc mmc@fe2e0000 > > fe470000-fe470fff : fe470000.i2s i2s@fe470000 > > fe600000-fe60ffff : GICD > > fe680000-fe77ffff : GICR > > fea10000-fea13fff : dma-controller@fea10000 > > fea10000-fea13fff : fea10000.dma-controller dma-controller@fea10000 > > fea30000-fea33fff : dma-controller@fea30000 > > fea30000-fea33fff : fea30000.dma-controller dma-controller@fea30000 > > feaa0000-feaa0fff : feaa0000.i2c i2c@feaa0000 > > feaf0000-feaf00ff : feaf0000.watchdog watchdog@feaf0000 > > feb20000-feb20fff : feb20000.spi spi@feb20000 > > feb50000-feb500ff : serial > > fec00000-fec003ff : fec00000.tsadc tsadc@fec00000 > > fec10000-fec1ffff : fec10000.adc adc@fec10000 > > fec20000-fec200ff : fec20000.gpio gpio@fec20000 > > fec30000-fec300ff : fec30000.gpio gpio@fec30000 > > fec40000-fec400ff : fec40000.gpio gpio@fec40000 > > fec50000-fec500ff : fec50000.gpio gpio@fec50000 > > fec90000-fec90fff : fec90000.i2c i2c@fec90000 > > fed10000-fed13fff : dma-controller@fed10000 > > fed10000-fed13fff : fed10000.dma-controller dma-controller@fed10000 > > fed60000-fed61fff : fed60000.phy phy@fed60000 > > fed70000-fed71fff : fed70000.phy phy@fed70000 > > fed80000-fed8ffff : fed80000.phy phy@fed80000 > > fed90000-fed9ffff : fed90000.phy phy@fed90000 > > fee00000-fee000ff : fee00000.phy phy@fee00000 > > fee10000-fee100ff : fee10000.phy phy@fee10000 > > fee20000-fee200ff : fee20000.phy phy@fee20000 > > fee80000-fee9ffff : fee80000.phy phy@fee80000 > > ff001000-ff0effff : ff001000.sram sram@ff001000 > > 100000000-3fbffffff : System RAM > > 3ec000000-3fbffffff : reserved > > 3fc500000-3ffefffff : System RAM > > 4f0000000-4ffffffff : System RAM > > 4fc611000-4fc6d0fff : reserved > > 4fc6d1000-4fded1fff : reserved > > 4fded2000-4fdf91fff : reserved > > 4fdf93000-4fdf96fff : reserved > > 4fdf97000-4fdfabfff : reserved > > 4fdfac000-4fe051fff : reserved > > 4fe052000-4ffffffff : reserved > > 900000000-93fffffff : pcie@fe150000 > > 900000000-93fffffff : 0000:00:00.0 > > 980000000-9bfffffff : pcie@fe170000 > > 9c0000000-9ffffffff : pcie@fe180000 > > a40000000-a403fffff : a40000000.pcie dbi > > a40800000-a40bfffff : a40800000.pcie dbi > > a40c00000-a40ffffff : a40c00000.pcie dbi > > > > Thank you, > > o. > > > > > -- > > > i. > >
On Mon, 14 Apr 2025, Ondřej Jirman wrote: > On Mon, Apr 14, 2025 at 12:52:15PM +0300, Ilpo Järvinen wrote: > > On Fri, 11 Apr 2025, Ondřej Jirman wrote: > > > > > Hello Ilpo, > > > > > > On Tue, Apr 01, 2025 at 08:38:48PM +0300, Ilpo Järvinen wrote: > > > > That log wasn't taken from a bad case but it doesn't matter anymore as I > > > > could test this with qemu myself, thanks for providing enough to make it > > > > easy to reproduce & test it locally :-). > > > > > > > > The problem is caused by assign+release cycle being destructive on start > > > > aligned resources because successful assigment overwrites the start field. > > > > I'll send a patch to fix the problem once I've given it a bit more thought > > > > as this resource fitting code is somewhat complex. > > > > > > BTW, same thing here on a different SoC: > > > > > > https://lore.kernel.org/lkml/hrcsm2bo4ysqj2ggejndlou32cdc7yiknnm5nrlcoz4d64wall@7te4dfqsoe3y/T/#u > > > > > > There are kernel logs there, too, although I don't have dynamic debug enabled > > > in the kernel. > > > > > > Interestingly, bisect pointed me initially to a different commit. Reverting > > > it helped, but just on one board (QuartzPro64). > > > > Hi, > > > > Since you didn't mention it, I guess you haven't tried the fix: > > > > https://patchwork.kernel.org/project/linux-pci/patch/20250403093137.1481-1-ilpo.jarvinen@linux.intel.com/ > > This patch works. Thank you. Thanks for testing. If you feel confident enough, please send your Tested-by tag to its thread (not this one). > One difference compared to 6.14 that I'm noticing > in the kernel log is that "save config" sequences now are printed twice for > unpopulated port. Not sure if it's related to your patches. Previously it was > printed just once. I don't think it is related. The resource fitting/assignment changes should not impact PCI save/restore AFAIK (except by preventing device from working in the first place if necessary resources do not get assigned). PCI state saving has always seemed like that in most logs I've seen with dyndbg enabled, that is, the state is seemingly saved multiple times, often right after the previous save too. So TBH, I'm not convinced it's even something new. If you have backtraces to show those places that invoke pci_save_state() I can take a look but I don't expect much to come out of that. -- i. > Kind regards, > o. > > rockchip-dw-pcie a40800000.pcie: PCI host bridge to bus 0002:20 > pci_bus 0002:20: root bus resource [bus 20-2f] > pci_bus 0002:20: root bus resource [io 0x300000-0x3fffff] (bus address [0xf2100000-0xf21fffff]) > pci_bus 0002:20: root bus resource [mem 0xf2200000-0xf2ffffff] > pci_bus 0002:20: root bus resource [mem 0x980000000-0x9bfffffff] (bus address [0x40000000-0x7fffffff]) > pci_bus 0002:20: scanning bus > pci 0002:20:00.0: [1d87:3588] type 01 class 0x060400 PCIe Root Port > pci 0002:20:00.0: ROM [mem 0x00000000-0x0000ffff pref] > pci 0002:20:00.0: PCI bridge to [bus 01-ff] > pci 0002:20:00.0: bridge window [io 0x0000-0x0fff] > pci 0002:20:00.0: bridge window [mem 0x00000000-0x000fffff] > pci 0002:20:00.0: bridge window [mem 0x00000000-0x000fffff 64bit pref] > pci 0002:20:00.0: supports D1 D2 > pci 0002:20:00.0: PME# supported from D0 D1 D3hot > pci 0002:20:00.0: PME# disabled > pci 0002:20:00.0: Adding to iommu group 9 > pci 0002:20:00.0: vgaarb: pci_notify > pci_bus 0002:20: fixups for bus > pci 0002:20:00.0: scanning [bus 01-ff] behind bridge, pass 0 > pci 0002:20:00.0: Primary bus is hard wired to 0 > pci 0002:20:00.0: bridge configuration invalid ([bus 01-ff]), reconfiguring > pci 0002:20:00.0: scanning [bus 00-00] behind bridge, pass 1 > pci_bus 0002:20: bus scan returning with max=21 > pci 0002:20:00.0: ROM [mem 0xf2200000-0xf220ffff pref]: assigned > pci 0002:20:00.0: PCI bridge to [bus 21] > pci_bus 0002:20: resource 4 [io 0x300000-0x3fffff] > pci_bus 0002:20: resource 5 [mem 0xf2200000-0xf2ffffff] > pci_bus 0002:20: resource 6 [mem 0x980000000-0x9bfffffff] > pcieport 0002:20:00.0: vgaarb: pci_notify > pcieport 0002:20:00.0: assign IRQ: got 148 > pcieport 0002:20:00.0: PME: Signaling with IRQ 157 > pcieport 0002:20:00.0: AER: enabled with IRQ 158 > pcieport 0002:20:00.0: bwctrl: enabled with IRQ 157 > pcieport 0002:20:00.0: save config 0x00: 0x35881d87 > pcieport 0002:20:00.0: save config 0x04: 0x00100507 > pcieport 0002:20:00.0: save config 0x08: 0x06040001 > pcieport 0002:20:00.0: save config 0x0c: 0x00010000 > pcieport 0002:20:00.0: save config 0x10: 0x00000000 > pcieport 0002:20:00.0: save config 0x14: 0x00000000 > pcieport 0002:20:00.0: save config 0x18: 0x00212120 > pcieport 0002:20:00.0: save config 0x1c: 0x000000f0 > pcieport 0002:20:00.0: save config 0x20: 0x0000fff0 > pcieport 0002:20:00.0: save config 0x24: 0x0001fff1 > pcieport 0002:20:00.0: save config 0x28: 0x00000000 > pcieport 0002:20:00.0: save config 0x2c: 0x00000000 > pcieport 0002:20:00.0: save config 0x30: 0x00000000 > pcieport 0002:20:00.0: save config 0x34: 0x00000040 > pcieport 0002:20:00.0: save config 0x38: 0x00000000 > pcieport 0002:20:00.0: save config 0x3c: 0x00020194 > pcieport 0002:20:00.0: vgaarb: pci_notify > pcieport 0002:20:00.0: save config 0x00: 0x35881d87 > pcieport 0002:20:00.0: save config 0x04: 0x00100507 > pcieport 0002:20:00.0: save config 0x08: 0x06040001 > pcieport 0002:20:00.0: save config 0x0c: 0x00010000 > pcieport 0002:20:00.0: save config 0x10: 0x00000000 > pcieport 0002:20:00.0: save config 0x14: 0x00000000 > pcieport 0002:20:00.0: save config 0x18: 0x00212120 > pcieport 0002:20:00.0: save config 0x1c: 0x000000f0 > pcieport 0002:20:00.0: save config 0x20: 0x0000fff0 > pcieport 0002:20:00.0: save config 0x24: 0x0001fff1 > pcieport 0002:20:00.0: save config 0x28: 0x00000000 > pcieport 0002:20:00.0: save config 0x2c: 0x00000000 > pcieport 0002:20:00.0: save config 0x30: 0x00000000 > pcieport 0002:20:00.0: save config 0x34: 0x00000040 > pcieport 0002:20:00.0: save config 0x38: 0x00000000 > pcieport 0002:20:00.0: save config 0x3c: 0x00020194 > pcieport 0002:20:00.0: PME# enabled > > > > -- > > i. > > > > > And this is iomem: > > > > > > 0010f000-0010f0ff : 10f000.sram sram@10f000 > > > 00200000-e2bbffff : System RAM > > > 02010000-0474ffff : Kernel code > > > 04750000-0498ffff : reserved > > > 04990000-0508ffff : Kernel data > > > daa00000-e29fffff : reserved > > > e2bc0000-ecbbffff : reserved > > > e2bc0000-ecbbffff : reserved > > > ecbc0000-efffffff : System RAM > > > ecbc7000-ecbdffff : reserved > > > f0000000-f00fffff : a40000000.pcie config > > > f0200000-f0ffffff : pcie@fe150000 > > > f0200000-f020ffff : 0000:00:00.0 > > > f0300000-f03fffff : PCI Bus 0000:01 > > > f0300000-f0303fff : 0000:01:00.0 > > > f0300000-f0303fff : nvme > > > f0304000-f03040ff : 0000:01:00.0 > > > f0304000-f03040ff : nvme > > > f2000000-f20fffff : a40800000.pcie config > > > f2200000-f2ffffff : pcie@fe170000 > > > f2200000-f27fffff : PCI Bus 0002:21 > > > f2200000-f220ffff : 0002:21:00.0 > > > f2400000-f27fffff : 0002:21:00.0 > > > f2800000-f280ffff : 0002:20:00.0 > > > f3000000-f30fffff : a40c00000.pcie config > > > f3200000-f3ffffff : pcie@fe180000 > > > f3200000-f320ffff : 0003:30:00.0 > > > f3300000-f33fffff : PCI Bus 0003:31 > > > f3300000-f3303fff : 0003:31:00.0 > > > f3304000-f3304fff : 0003:31:00.0 > > > f3304000-f3304fff : r8169 > > > fb000000-fb1fffff : fb000000.gpu gpu@fb000000 > > > fc00c100-fc3fffff : fc000000.usb usb@fc000000 > > > fc400000-fc407fff : usb@fc400000 > > > fc400000-fc407fff : xhci-hcd.10.auto usb@fc400000 > > > fc40c100-fc7fffff : fc400000.usb usb@fc400000 > > > fc800000-fc83ffff : fc800000.usb usb@fc800000 > > > fc840000-fc87ffff : fc840000.usb usb@fc840000 > > > fc880000-fc8bffff : fc880000.usb usb@fc880000 > > > fc8c0000-fc8fffff : fc8c0000.usb usb@fc8c0000 > > > fc900000-fc900dff : fc900000.iommu > > > fc910000-fc910dff : fc900000.iommu > > > fd600000-fd6fffff : fd600000.sram sram@fd600000 > > > fd8a0000-fd8a00ff : fd8a0000.gpio gpio@fd8a0000 > > > fdb50000-fdb507ff : fdb50000.video-codec video-codec@fdb50000 > > > fdb50800-fdb5083f : fdb50800.iommu iommu@fdb50800 > > > fdb80000-fdb8017f : fdb80000.rga rga@fdb80000 > > > fdba0000-fdba07ff : fdba0000.video-codec video-codec@fdba0000 > > > fdba0800-fdba083f : fdba0800.iommu iommu@fdba0800 > > > fdba4800-fdba483f : fdba4800.iommu iommu@fdba4800 > > > fdba8800-fdba883f : fdba8800.iommu iommu@fdba8800 > > > fdbac800-fdbac83f : fdbac800.iommu iommu@fdbac800 > > > fdc70000-fdc707ff : fdc70000.video-codec video-codec@fdc70000 > > > fdd90000-fdd941ff : fdd90000.vop vop > > > fdd95000-fdd95fff : fdd90000.vop gamma-lut > > > fdd97e00-fdd97eff : fdd97e00.iommu iommu@fdd97e00 > > > fdd97f00-fdd97fff : fdd97e00.iommu iommu@fdd97e00 > > > fddf0000-fddf0fff : fddf0000.i2s i2s@fddf0000 > > > fddf4000-fddf4fff : fddf4000.i2s i2s@fddf4000 > > > fde80000-fde9ffff : fde80000.hdmi hdmi@fde80000 > > > fdea0000-fdebffff : fdea0000.hdmi hdmi@fdea0000 > > > fdee0000-fdee5fff : fdee0000.hdmi_receiver hdmi_receiver@fdee0000 > > > fe060000-fe06ffff : fe060000.dfi dfi@fe060000 > > > fe150000-fe15ffff : a40000000.pcie apb > > > fe170000-fe17ffff : a40800000.pcie apb > > > fe180000-fe18ffff : a40c00000.pcie apb > > > fe1b0000-fe1bffff : fe1b0000.ethernet ethernet@fe1b0000 > > > fe210000-fe210fff : fe210000.sata sata@fe210000 > > > fe2c0000-fe2c3fff : fe2c0000.mmc mmc@fe2c0000 > > > fe2e0000-fe2effff : fe2e0000.mmc mmc@fe2e0000 > > > fe470000-fe470fff : fe470000.i2s i2s@fe470000 > > > fe600000-fe60ffff : GICD > > > fe680000-fe77ffff : GICR > > > fea10000-fea13fff : dma-controller@fea10000 > > > fea10000-fea13fff : fea10000.dma-controller dma-controller@fea10000 > > > fea30000-fea33fff : dma-controller@fea30000 > > > fea30000-fea33fff : fea30000.dma-controller dma-controller@fea30000 > > > feaa0000-feaa0fff : feaa0000.i2c i2c@feaa0000 > > > feaf0000-feaf00ff : feaf0000.watchdog watchdog@feaf0000 > > > feb20000-feb20fff : feb20000.spi spi@feb20000 > > > feb50000-feb500ff : serial > > > fec00000-fec003ff : fec00000.tsadc tsadc@fec00000 > > > fec10000-fec1ffff : fec10000.adc adc@fec10000 > > > fec20000-fec200ff : fec20000.gpio gpio@fec20000 > > > fec30000-fec300ff : fec30000.gpio gpio@fec30000 > > > fec40000-fec400ff : fec40000.gpio gpio@fec40000 > > > fec50000-fec500ff : fec50000.gpio gpio@fec50000 > > > fec90000-fec90fff : fec90000.i2c i2c@fec90000 > > > fed10000-fed13fff : dma-controller@fed10000 > > > fed10000-fed13fff : fed10000.dma-controller dma-controller@fed10000 > > > fed60000-fed61fff : fed60000.phy phy@fed60000 > > > fed70000-fed71fff : fed70000.phy phy@fed70000 > > > fed80000-fed8ffff : fed80000.phy phy@fed80000 > > > fed90000-fed9ffff : fed90000.phy phy@fed90000 > > > fee00000-fee000ff : fee00000.phy phy@fee00000 > > > fee10000-fee100ff : fee10000.phy phy@fee10000 > > > fee20000-fee200ff : fee20000.phy phy@fee20000 > > > fee80000-fee9ffff : fee80000.phy phy@fee80000 > > > ff001000-ff0effff : ff001000.sram sram@ff001000 > > > 100000000-3fbffffff : System RAM > > > 3ec000000-3fbffffff : reserved > > > 3fc500000-3ffefffff : System RAM > > > 4f0000000-4ffffffff : System RAM > > > 4fc611000-4fc6d0fff : reserved > > > 4fc6d1000-4fded1fff : reserved > > > 4fded2000-4fdf91fff : reserved > > > 4fdf93000-4fdf96fff : reserved > > > 4fdf97000-4fdfabfff : reserved > > > 4fdfac000-4fe051fff : reserved > > > 4fe052000-4ffffffff : reserved > > > 900000000-93fffffff : pcie@fe150000 > > > 900000000-93fffffff : 0000:00:00.0 > > > 980000000-9bfffffff : pcie@fe170000 > > > 9c0000000-9ffffffff : pcie@fe180000 > > > a40000000-a403fffff : a40000000.pcie dbi > > > a40800000-a40bfffff : a40800000.pcie dbi > > > a40c00000-a40ffffff : a40c00000.pcie dbi > > > > > > Thank you, > > > o. > > > > > > > -- > > > > i. > > > >
On Mon, Apr 14, 2025 at 04:15:01PM +0300, Ilpo Järvinen wrote: > On Mon, 14 Apr 2025, Ondřej Jirman wrote: > > On Mon, Apr 14, 2025 at 12:52:15PM +0300, Ilpo Järvinen wrote: > > > On Fri, 11 Apr 2025, Ondřej Jirman wrote: > > > > > > > Hello Ilpo, > > > > > > > > On Tue, Apr 01, 2025 at 08:38:48PM +0300, Ilpo Järvinen wrote: > > > > > That log wasn't taken from a bad case but it doesn't matter anymore as I > > > > > could test this with qemu myself, thanks for providing enough to make it > > > > > easy to reproduce & test it locally :-). > > > > > > > > > > The problem is caused by assign+release cycle being destructive on start > > > > > aligned resources because successful assigment overwrites the start field. > > > > > I'll send a patch to fix the problem once I've given it a bit more thought > > > > > as this resource fitting code is somewhat complex. > > > > > > > > BTW, same thing here on a different SoC: > > > > > > > > https://lore.kernel.org/lkml/hrcsm2bo4ysqj2ggejndlou32cdc7yiknnm5nrlcoz4d64wall@7te4dfqsoe3y/T/#u > > > > > > > > There are kernel logs there, too, although I don't have dynamic debug enabled > > > > in the kernel. > > > > > > > > Interestingly, bisect pointed me initially to a different commit. Reverting > > > > it helped, but just on one board (QuartzPro64). > > > > > > Hi, > > > > > > Since you didn't mention it, I guess you haven't tried the fix: > > > > > > https://patchwork.kernel.org/project/linux-pci/patch/20250403093137.1481-1-ilpo.jarvinen@linux.intel.com/ > > > > This patch works. Thank you. > > Thanks for testing. If you feel confident enough, please send your > Tested-by tag to its thread (not this one). > > > One difference compared to 6.14 that I'm noticing > > in the kernel log is that "save config" sequences now are printed twice for > > unpopulated port. Not sure if it's related to your patches. Previously it was > > printed just once. > > I don't think it is related. The resource fitting/assignment changes > should not impact PCI save/restore AFAIK (except by preventing device from > working in the first place if necessary resources do not get assigned). > > PCI state saving has always seemed like that in most logs I've seen with > dyndbg enabled, that is, the state is seemingly saved multiple times, > often right after the previous save too. So TBH, I'm not convinced it's > even something new. If you have backtraces to show those places that > invoke pci_save_state() I can take a look but I don't expect much to > come out of that. You're right. It's unrelated. The traces: [ 1.878732] show_stack+0x28/0x80 (C) [ 1.878751] dump_stack_lvl+0x58/0x74 [ 1.878762] dump_stack+0x14/0x1c [ 1.878772] pci_save_state+0x34/0x1e8 [ 1.878783] pcie_portdrv_probe+0x330/0x6a8 [ 1.878794] local_pci_probe+0x3c/0xa0 [ 1.878804] pci_device_probe+0xac/0x194 [ 1.878813] really_probe+0xbc/0x388 [ 1.878825] __driver_probe_device+0x78/0x144 [ 1.878837] driver_probe_device+0x38/0x110 [ 1.878850] __device_attach_driver+0xb0/0x150 [ 1.878862] bus_for_each_drv+0x6c/0xb0 [ 1.878873] __device_attach+0x98/0x1a0 [ 1.878886] device_attach+0x10/0x20 [ 1.878897] pci_bus_add_device+0x84/0x12c [ 1.878911] pci_bus_add_devices+0x40/0x90 [ 1.878924] pci_host_probe+0x88/0xe4 [ 1.878933] dw_pcie_host_init+0x258/0x714 [ 1.878945] rockchip_pcie_probe+0x2f0/0x3f8 [ 1.878956] platform_probe+0x64/0xcc [ 1.878965] really_probe+0xbc/0x388 [ 1.878977] __driver_probe_device+0x78/0x144 [ 1.878989] driver_probe_device+0x38/0x110 [ 1.879001] __device_attach_driver+0xb0/0x150 [ 1.879014] bus_for_each_drv+0x6c/0xb0 [ 1.879025] __device_attach+0x98/0x1a0 [ 1.879037] device_initial_probe+0x10/0x20 [ 1.879049] bus_probe_device+0xa8/0xb8 [ 1.879061] deferred_probe_work_func+0xa4/0xf0 [ 1.879072] process_one_work+0x13c/0x2bc [ 1.879084] worker_thread+0x284/0x460 [ 1.879094] kthread+0xe4/0x1a0 [ 1.879107] ret_from_fork+0x10/0x20 [ 1.879121] pcieport 0002:20:00.0: save config 0x00: 0x35881d87 [ 1.879158] pcieport 0002:20:00.0: save config 0x04: 0x00100507 [ 1.879168] pcieport 0002:20:00.0: save config 0x08: 0x06040001 [ 1.879177] pcieport 0002:20:00.0: save config 0x0c: 0x00010000 [ 1.879221] pcieport 0002:20:00.0: save config 0x10: 0x00000000 [ 1.879232] pcieport 0002:20:00.0: save config 0x14: 0x00000000 [ 1.879242] pcieport 0002:20:00.0: save config 0x18: 0x00212120 [ 1.879251] pcieport 0002:20:00.0: save config 0x1c: 0x000000f0 [ 1.879260] pcieport 0002:20:00.0: save config 0x20: 0x0000fff0 [ 1.879270] pcieport 0002:20:00.0: save config 0x24: 0x0001fff1 [ 1.879279] pcieport 0002:20:00.0: save config 0x28: 0x00000000 [ 1.879289] pcieport 0002:20:00.0: save config 0x2c: 0x00000000 [ 1.879298] pcieport 0002:20:00.0: save config 0x30: 0x00000000 [ 1.879307] pcieport 0002:20:00.0: save config 0x34: 0x00000040 [ 1.879316] pcieport 0002:20:00.0: save config 0x38: 0x00000000 [ 1.879325] pcieport 0002:20:00.0: save config 0x3c: 0x00020194 [ 1.879376] pcieport 0002:20:00.0: vgaarb: pci_notify second time it's from: [ 2.004478] Workqueue: pm pm_runtime_work [ 2.004488] Call trace: [ 2.004491] show_stack+0x28/0x80 (C) [ 2.004500] dump_stack_lvl+0x58/0x74 [ 2.004506] dump_stack+0x14/0x1c [ 2.004511] pci_save_state+0x34/0x1e8 [ 2.004516] pci_pm_runtime_suspend+0xa8/0x1e0 [ 2.004521] __rpm_callback+0x3c/0x220 [ 2.004527] rpm_callback+0x6c/0x80 [ 2.004532] rpm_suspend+0xec/0x674 [ 2.004537] pm_runtime_work+0x104/0x114 [ 2.004542] process_one_work+0x13c/0x2bc [ 2.004548] worker_thread+0x284/0x460 [ 2.004552] kthread+0xe4/0x1a0 [ 2.004559] ret_from_fork+0x10/0x20 [ 2.004567] pcieport 0002:20:00.0: save config 0x00: 0x35881d87 [ 2.004578] pcieport 0002:20:00.0: save config 0x04: 0x00100507 [ 2.004583] pcieport 0002:20:00.0: save config 0x08: 0x06040001 [ 2.004587] pcieport 0002:20:00.0: save config 0x0c: 0x00010000 [ 2.004591] pcieport 0002:20:00.0: save config 0x10: 0x00000000 [ 2.004596] pcieport 0002:20:00.0: save config 0x14: 0x00000000 [ 2.004600] pcieport 0002:20:00.0: save config 0x18: 0x00212120 [ 2.004604] pcieport 0002:20:00.0: save config 0x1c: 0x000000f0 [ 2.004608] pcieport 0002:20:00.0: save config 0x20: 0x0000fff0 [ 2.004613] pcieport 0002:20:00.0: save config 0x24: 0x0001fff1 [ 2.004617] pcieport 0002:20:00.0: save config 0x28: 0x00000000 [ 2.004621] pcieport 0002:20:00.0: save config 0x2c: 0x00000000 [ 2.004625] pcieport 0002:20:00.0: save config 0x30: 0x00000000 [ 2.004629] pcieport 0002:20:00.0: save config 0x34: 0x00000040 [ 2.004633] pcieport 0002:20:00.0: save config 0x38: 0x00000000 [ 2.004638] pcieport 0002:20:00.0: save config 0x3c: 0x00020194 [ 2.004662] pcieport 0002:20:00.0: PME# enabled on 6.15-rc: cat /sys/class/pci_bus/0002:20/device/0002:20:00.0/power/runtime_status OK suspended on 6.14: cat /sys/class/pci_bus/0002:20/device/0002:20:00.0/power/runtime_status OK active So some other unrelated change in 6.15-rc just enabled RPM to suspend the unused port. Thnak you, o. > -- > i. > > > Kind regards, > > o. > > > > rockchip-dw-pcie a40800000.pcie: PCI host bridge to bus 0002:20 > > pci_bus 0002:20: root bus resource [bus 20-2f] > > pci_bus 0002:20: root bus resource [io 0x300000-0x3fffff] (bus address [0xf2100000-0xf21fffff]) > > pci_bus 0002:20: root bus resource [mem 0xf2200000-0xf2ffffff] > > pci_bus 0002:20: root bus resource [mem 0x980000000-0x9bfffffff] (bus address [0x40000000-0x7fffffff]) > > pci_bus 0002:20: scanning bus > > pci 0002:20:00.0: [1d87:3588] type 01 class 0x060400 PCIe Root Port > > pci 0002:20:00.0: ROM [mem 0x00000000-0x0000ffff pref] > > pci 0002:20:00.0: PCI bridge to [bus 01-ff] > > pci 0002:20:00.0: bridge window [io 0x0000-0x0fff] > > pci 0002:20:00.0: bridge window [mem 0x00000000-0x000fffff] > > pci 0002:20:00.0: bridge window [mem 0x00000000-0x000fffff 64bit pref] > > pci 0002:20:00.0: supports D1 D2 > > pci 0002:20:00.0: PME# supported from D0 D1 D3hot > > pci 0002:20:00.0: PME# disabled > > pci 0002:20:00.0: Adding to iommu group 9 > > pci 0002:20:00.0: vgaarb: pci_notify > > pci_bus 0002:20: fixups for bus > > pci 0002:20:00.0: scanning [bus 01-ff] behind bridge, pass 0 > > pci 0002:20:00.0: Primary bus is hard wired to 0 > > pci 0002:20:00.0: bridge configuration invalid ([bus 01-ff]), reconfiguring > > pci 0002:20:00.0: scanning [bus 00-00] behind bridge, pass 1 > > pci_bus 0002:20: bus scan returning with max=21 > > pci 0002:20:00.0: ROM [mem 0xf2200000-0xf220ffff pref]: assigned > > pci 0002:20:00.0: PCI bridge to [bus 21] > > pci_bus 0002:20: resource 4 [io 0x300000-0x3fffff] > > pci_bus 0002:20: resource 5 [mem 0xf2200000-0xf2ffffff] > > pci_bus 0002:20: resource 6 [mem 0x980000000-0x9bfffffff] > > pcieport 0002:20:00.0: vgaarb: pci_notify > > pcieport 0002:20:00.0: assign IRQ: got 148 > > pcieport 0002:20:00.0: PME: Signaling with IRQ 157 > > pcieport 0002:20:00.0: AER: enabled with IRQ 158 > > pcieport 0002:20:00.0: bwctrl: enabled with IRQ 157 > > pcieport 0002:20:00.0: save config 0x00: 0x35881d87 > > pcieport 0002:20:00.0: save config 0x04: 0x00100507 > > pcieport 0002:20:00.0: save config 0x08: 0x06040001 > > pcieport 0002:20:00.0: save config 0x0c: 0x00010000 > > pcieport 0002:20:00.0: save config 0x10: 0x00000000 > > pcieport 0002:20:00.0: save config 0x14: 0x00000000 > > pcieport 0002:20:00.0: save config 0x18: 0x00212120 > > pcieport 0002:20:00.0: save config 0x1c: 0x000000f0 > > pcieport 0002:20:00.0: save config 0x20: 0x0000fff0 > > pcieport 0002:20:00.0: save config 0x24: 0x0001fff1 > > pcieport 0002:20:00.0: save config 0x28: 0x00000000 > > pcieport 0002:20:00.0: save config 0x2c: 0x00000000 > > pcieport 0002:20:00.0: save config 0x30: 0x00000000 > > pcieport 0002:20:00.0: save config 0x34: 0x00000040 > > pcieport 0002:20:00.0: save config 0x38: 0x00000000 > > pcieport 0002:20:00.0: save config 0x3c: 0x00020194 > > pcieport 0002:20:00.0: vgaarb: pci_notify > > pcieport 0002:20:00.0: save config 0x00: 0x35881d87 > > pcieport 0002:20:00.0: save config 0x04: 0x00100507 > > pcieport 0002:20:00.0: save config 0x08: 0x06040001 > > pcieport 0002:20:00.0: save config 0x0c: 0x00010000 > > pcieport 0002:20:00.0: save config 0x10: 0x00000000 > > pcieport 0002:20:00.0: save config 0x14: 0x00000000 > > pcieport 0002:20:00.0: save config 0x18: 0x00212120 > > pcieport 0002:20:00.0: save config 0x1c: 0x000000f0 > > pcieport 0002:20:00.0: save config 0x20: 0x0000fff0 > > pcieport 0002:20:00.0: save config 0x24: 0x0001fff1 > > pcieport 0002:20:00.0: save config 0x28: 0x00000000 > > pcieport 0002:20:00.0: save config 0x2c: 0x00000000 > > pcieport 0002:20:00.0: save config 0x30: 0x00000000 > > pcieport 0002:20:00.0: save config 0x34: 0x00000040 > > pcieport 0002:20:00.0: save config 0x38: 0x00000000 > > pcieport 0002:20:00.0: save config 0x3c: 0x00020194 > > pcieport 0002:20:00.0: PME# enabled > > > > > > > -- > > > i. > > > > > > > And this is iomem: > > > > > > > > 0010f000-0010f0ff : 10f000.sram sram@10f000 > > > > 00200000-e2bbffff : System RAM > > > > 02010000-0474ffff : Kernel code > > > > 04750000-0498ffff : reserved > > > > 04990000-0508ffff : Kernel data > > > > daa00000-e29fffff : reserved > > > > e2bc0000-ecbbffff : reserved > > > > e2bc0000-ecbbffff : reserved > > > > ecbc0000-efffffff : System RAM > > > > ecbc7000-ecbdffff : reserved > > > > f0000000-f00fffff : a40000000.pcie config > > > > f0200000-f0ffffff : pcie@fe150000 > > > > f0200000-f020ffff : 0000:00:00.0 > > > > f0300000-f03fffff : PCI Bus 0000:01 > > > > f0300000-f0303fff : 0000:01:00.0 > > > > f0300000-f0303fff : nvme > > > > f0304000-f03040ff : 0000:01:00.0 > > > > f0304000-f03040ff : nvme > > > > f2000000-f20fffff : a40800000.pcie config > > > > f2200000-f2ffffff : pcie@fe170000 > > > > f2200000-f27fffff : PCI Bus 0002:21 > > > > f2200000-f220ffff : 0002:21:00.0 > > > > f2400000-f27fffff : 0002:21:00.0 > > > > f2800000-f280ffff : 0002:20:00.0 > > > > f3000000-f30fffff : a40c00000.pcie config > > > > f3200000-f3ffffff : pcie@fe180000 > > > > f3200000-f320ffff : 0003:30:00.0 > > > > f3300000-f33fffff : PCI Bus 0003:31 > > > > f3300000-f3303fff : 0003:31:00.0 > > > > f3304000-f3304fff : 0003:31:00.0 > > > > f3304000-f3304fff : r8169 > > > > fb000000-fb1fffff : fb000000.gpu gpu@fb000000 > > > > fc00c100-fc3fffff : fc000000.usb usb@fc000000 > > > > fc400000-fc407fff : usb@fc400000 > > > > fc400000-fc407fff : xhci-hcd.10.auto usb@fc400000 > > > > fc40c100-fc7fffff : fc400000.usb usb@fc400000 > > > > fc800000-fc83ffff : fc800000.usb usb@fc800000 > > > > fc840000-fc87ffff : fc840000.usb usb@fc840000 > > > > fc880000-fc8bffff : fc880000.usb usb@fc880000 > > > > fc8c0000-fc8fffff : fc8c0000.usb usb@fc8c0000 > > > > fc900000-fc900dff : fc900000.iommu > > > > fc910000-fc910dff : fc900000.iommu > > > > fd600000-fd6fffff : fd600000.sram sram@fd600000 > > > > fd8a0000-fd8a00ff : fd8a0000.gpio gpio@fd8a0000 > > > > fdb50000-fdb507ff : fdb50000.video-codec video-codec@fdb50000 > > > > fdb50800-fdb5083f : fdb50800.iommu iommu@fdb50800 > > > > fdb80000-fdb8017f : fdb80000.rga rga@fdb80000 > > > > fdba0000-fdba07ff : fdba0000.video-codec video-codec@fdba0000 > > > > fdba0800-fdba083f : fdba0800.iommu iommu@fdba0800 > > > > fdba4800-fdba483f : fdba4800.iommu iommu@fdba4800 > > > > fdba8800-fdba883f : fdba8800.iommu iommu@fdba8800 > > > > fdbac800-fdbac83f : fdbac800.iommu iommu@fdbac800 > > > > fdc70000-fdc707ff : fdc70000.video-codec video-codec@fdc70000 > > > > fdd90000-fdd941ff : fdd90000.vop vop > > > > fdd95000-fdd95fff : fdd90000.vop gamma-lut > > > > fdd97e00-fdd97eff : fdd97e00.iommu iommu@fdd97e00 > > > > fdd97f00-fdd97fff : fdd97e00.iommu iommu@fdd97e00 > > > > fddf0000-fddf0fff : fddf0000.i2s i2s@fddf0000 > > > > fddf4000-fddf4fff : fddf4000.i2s i2s@fddf4000 > > > > fde80000-fde9ffff : fde80000.hdmi hdmi@fde80000 > > > > fdea0000-fdebffff : fdea0000.hdmi hdmi@fdea0000 > > > > fdee0000-fdee5fff : fdee0000.hdmi_receiver hdmi_receiver@fdee0000 > > > > fe060000-fe06ffff : fe060000.dfi dfi@fe060000 > > > > fe150000-fe15ffff : a40000000.pcie apb > > > > fe170000-fe17ffff : a40800000.pcie apb > > > > fe180000-fe18ffff : a40c00000.pcie apb > > > > fe1b0000-fe1bffff : fe1b0000.ethernet ethernet@fe1b0000 > > > > fe210000-fe210fff : fe210000.sata sata@fe210000 > > > > fe2c0000-fe2c3fff : fe2c0000.mmc mmc@fe2c0000 > > > > fe2e0000-fe2effff : fe2e0000.mmc mmc@fe2e0000 > > > > fe470000-fe470fff : fe470000.i2s i2s@fe470000 > > > > fe600000-fe60ffff : GICD > > > > fe680000-fe77ffff : GICR > > > > fea10000-fea13fff : dma-controller@fea10000 > > > > fea10000-fea13fff : fea10000.dma-controller dma-controller@fea10000 > > > > fea30000-fea33fff : dma-controller@fea30000 > > > > fea30000-fea33fff : fea30000.dma-controller dma-controller@fea30000 > > > > feaa0000-feaa0fff : feaa0000.i2c i2c@feaa0000 > > > > feaf0000-feaf00ff : feaf0000.watchdog watchdog@feaf0000 > > > > feb20000-feb20fff : feb20000.spi spi@feb20000 > > > > feb50000-feb500ff : serial > > > > fec00000-fec003ff : fec00000.tsadc tsadc@fec00000 > > > > fec10000-fec1ffff : fec10000.adc adc@fec10000 > > > > fec20000-fec200ff : fec20000.gpio gpio@fec20000 > > > > fec30000-fec300ff : fec30000.gpio gpio@fec30000 > > > > fec40000-fec400ff : fec40000.gpio gpio@fec40000 > > > > fec50000-fec500ff : fec50000.gpio gpio@fec50000 > > > > fec90000-fec90fff : fec90000.i2c i2c@fec90000 > > > > fed10000-fed13fff : dma-controller@fed10000 > > > > fed10000-fed13fff : fed10000.dma-controller dma-controller@fed10000 > > > > fed60000-fed61fff : fed60000.phy phy@fed60000 > > > > fed70000-fed71fff : fed70000.phy phy@fed70000 > > > > fed80000-fed8ffff : fed80000.phy phy@fed80000 > > > > fed90000-fed9ffff : fed90000.phy phy@fed90000 > > > > fee00000-fee000ff : fee00000.phy phy@fee00000 > > > > fee10000-fee100ff : fee10000.phy phy@fee10000 > > > > fee20000-fee200ff : fee20000.phy phy@fee20000 > > > > fee80000-fee9ffff : fee80000.phy phy@fee80000 > > > > ff001000-ff0effff : ff001000.sram sram@ff001000 > > > > 100000000-3fbffffff : System RAM > > > > 3ec000000-3fbffffff : reserved > > > > 3fc500000-3ffefffff : System RAM > > > > 4f0000000-4ffffffff : System RAM > > > > 4fc611000-4fc6d0fff : reserved > > > > 4fc6d1000-4fded1fff : reserved > > > > 4fded2000-4fdf91fff : reserved > > > > 4fdf93000-4fdf96fff : reserved > > > > 4fdf97000-4fdfabfff : reserved > > > > 4fdfac000-4fe051fff : reserved > > > > 4fe052000-4ffffffff : reserved > > > > 900000000-93fffffff : pcie@fe150000 > > > > 900000000-93fffffff : 0000:00:00.0 > > > > 980000000-9bfffffff : pcie@fe170000 > > > > 9c0000000-9ffffffff : pcie@fe180000 > > > > a40000000-a403fffff : a40000000.pcie dbi > > > > a40800000-a40bfffff : a40800000.pcie dbi > > > > a40c00000-a40ffffff : a40c00000.pcie dbi > > > > > > > > Thank you, > > > > o. > > > > > > > > > -- > > > > > i. > > > > > >
On Mon, 14 Apr 2025, Ondřej Jirman wrote: > On Mon, Apr 14, 2025 at 04:15:01PM +0300, Ilpo Järvinen wrote: > > On Mon, 14 Apr 2025, Ondřej Jirman wrote: > > > On Mon, Apr 14, 2025 at 12:52:15PM +0300, Ilpo Järvinen wrote: > > > > On Fri, 11 Apr 2025, Ondřej Jirman wrote: > > > > > > > > > Hello Ilpo, > > > > > > > > > > On Tue, Apr 01, 2025 at 08:38:48PM +0300, Ilpo Järvinen wrote: > > > > > > That log wasn't taken from a bad case but it doesn't matter anymore as I > > > > > > could test this with qemu myself, thanks for providing enough to make it > > > > > > easy to reproduce & test it locally :-). > > > > > > > > > > > > The problem is caused by assign+release cycle being destructive on start > > > > > > aligned resources because successful assigment overwrites the start field. > > > > > > I'll send a patch to fix the problem once I've given it a bit more thought > > > > > > as this resource fitting code is somewhat complex. > > > > > > > > > > BTW, same thing here on a different SoC: > > > > > > > > > > https://lore.kernel.org/lkml/hrcsm2bo4ysqj2ggejndlou32cdc7yiknnm5nrlcoz4d64wall@7te4dfqsoe3y/T/#u > > > > > > > > > > There are kernel logs there, too, although I don't have dynamic debug enabled > > > > > in the kernel. > > > > > > > > > > Interestingly, bisect pointed me initially to a different commit. Reverting > > > > > it helped, but just on one board (QuartzPro64). > > > > > > > > Hi, > > > > > > > > Since you didn't mention it, I guess you haven't tried the fix: > > > > > > > > https://patchwork.kernel.org/project/linux-pci/patch/20250403093137.1481-1-ilpo.jarvinen@linux.intel.com/ > > > > > > This patch works. Thank you. > > > > Thanks for testing. If you feel confident enough, please send your > > Tested-by tag to its thread (not this one). > > > > > One difference compared to 6.14 that I'm noticing > > > in the kernel log is that "save config" sequences now are printed twice for > > > unpopulated port. Not sure if it's related to your patches. Previously it was > > > printed just once. > > > > I don't think it is related. The resource fitting/assignment changes > > should not impact PCI save/restore AFAIK (except by preventing device from > > working in the first place if necessary resources do not get assigned). > > > > PCI state saving has always seemed like that in most logs I've seen with > > dyndbg enabled, that is, the state is seemingly saved multiple times, > > often right after the previous save too. So TBH, I'm not convinced it's > > even something new. If you have backtraces to show those places that > > invoke pci_save_state() I can take a look but I don't expect much to > > come out of that. > > You're right. It's unrelated. The traces: > > [ 1.878732] show_stack+0x28/0x80 (C) > [ 1.878751] dump_stack_lvl+0x58/0x74 > [ 1.878762] dump_stack+0x14/0x1c > [ 1.878772] pci_save_state+0x34/0x1e8 > [ 1.878783] pcie_portdrv_probe+0x330/0x6a8 > [ 1.878794] local_pci_probe+0x3c/0xa0 > [ 1.878804] pci_device_probe+0xac/0x194 > [ 1.878813] really_probe+0xbc/0x388 > [ 1.878825] __driver_probe_device+0x78/0x144 > [ 1.878837] driver_probe_device+0x38/0x110 > [ 1.878850] __device_attach_driver+0xb0/0x150 > [ 1.878862] bus_for_each_drv+0x6c/0xb0 > [ 1.878873] __device_attach+0x98/0x1a0 > [ 1.878886] device_attach+0x10/0x20 > [ 1.878897] pci_bus_add_device+0x84/0x12c > [ 1.878911] pci_bus_add_devices+0x40/0x90 > [ 1.878924] pci_host_probe+0x88/0xe4 > [ 1.878933] dw_pcie_host_init+0x258/0x714 > [ 1.878945] rockchip_pcie_probe+0x2f0/0x3f8 > [ 1.878956] platform_probe+0x64/0xcc > [ 1.878965] really_probe+0xbc/0x388 > [ 1.878977] __driver_probe_device+0x78/0x144 > [ 1.878989] driver_probe_device+0x38/0x110 > [ 1.879001] __device_attach_driver+0xb0/0x150 > [ 1.879014] bus_for_each_drv+0x6c/0xb0 > [ 1.879025] __device_attach+0x98/0x1a0 > [ 1.879037] device_initial_probe+0x10/0x20 > [ 1.879049] bus_probe_device+0xa8/0xb8 > [ 1.879061] deferred_probe_work_func+0xa4/0xf0 > [ 1.879072] process_one_work+0x13c/0x2bc > [ 1.879084] worker_thread+0x284/0x460 > [ 1.879094] kthread+0xe4/0x1a0 > [ 1.879107] ret_from_fork+0x10/0x20 > [ 1.879121] pcieport 0002:20:00.0: save config 0x00: 0x35881d87 > [ 1.879158] pcieport 0002:20:00.0: save config 0x04: 0x00100507 > [ 1.879168] pcieport 0002:20:00.0: save config 0x08: 0x06040001 > [ 1.879177] pcieport 0002:20:00.0: save config 0x0c: 0x00010000 > [ 1.879221] pcieport 0002:20:00.0: save config 0x10: 0x00000000 > [ 1.879232] pcieport 0002:20:00.0: save config 0x14: 0x00000000 > [ 1.879242] pcieport 0002:20:00.0: save config 0x18: 0x00212120 > [ 1.879251] pcieport 0002:20:00.0: save config 0x1c: 0x000000f0 > [ 1.879260] pcieport 0002:20:00.0: save config 0x20: 0x0000fff0 > [ 1.879270] pcieport 0002:20:00.0: save config 0x24: 0x0001fff1 > [ 1.879279] pcieport 0002:20:00.0: save config 0x28: 0x00000000 > [ 1.879289] pcieport 0002:20:00.0: save config 0x2c: 0x00000000 > [ 1.879298] pcieport 0002:20:00.0: save config 0x30: 0x00000000 > [ 1.879307] pcieport 0002:20:00.0: save config 0x34: 0x00000040 > [ 1.879316] pcieport 0002:20:00.0: save config 0x38: 0x00000000 > [ 1.879325] pcieport 0002:20:00.0: save config 0x3c: 0x00020194 > [ 1.879376] pcieport 0002:20:00.0: vgaarb: pci_notify > > second time it's from: > > [ 2.004478] Workqueue: pm pm_runtime_work > [ 2.004488] Call trace: > [ 2.004491] show_stack+0x28/0x80 (C) > [ 2.004500] dump_stack_lvl+0x58/0x74 > [ 2.004506] dump_stack+0x14/0x1c > [ 2.004511] pci_save_state+0x34/0x1e8 > [ 2.004516] pci_pm_runtime_suspend+0xa8/0x1e0 > [ 2.004521] __rpm_callback+0x3c/0x220 > [ 2.004527] rpm_callback+0x6c/0x80 > [ 2.004532] rpm_suspend+0xec/0x674 > [ 2.004537] pm_runtime_work+0x104/0x114 > [ 2.004542] process_one_work+0x13c/0x2bc > [ 2.004548] worker_thread+0x284/0x460 > [ 2.004552] kthread+0xe4/0x1a0 > [ 2.004559] ret_from_fork+0x10/0x20 > [ 2.004567] pcieport 0002:20:00.0: save config 0x00: 0x35881d87 > [ 2.004578] pcieport 0002:20:00.0: save config 0x04: 0x00100507 > [ 2.004583] pcieport 0002:20:00.0: save config 0x08: 0x06040001 > [ 2.004587] pcieport 0002:20:00.0: save config 0x0c: 0x00010000 > [ 2.004591] pcieport 0002:20:00.0: save config 0x10: 0x00000000 > [ 2.004596] pcieport 0002:20:00.0: save config 0x14: 0x00000000 > [ 2.004600] pcieport 0002:20:00.0: save config 0x18: 0x00212120 > [ 2.004604] pcieport 0002:20:00.0: save config 0x1c: 0x000000f0 > [ 2.004608] pcieport 0002:20:00.0: save config 0x20: 0x0000fff0 > [ 2.004613] pcieport 0002:20:00.0: save config 0x24: 0x0001fff1 > [ 2.004617] pcieport 0002:20:00.0: save config 0x28: 0x00000000 > [ 2.004621] pcieport 0002:20:00.0: save config 0x2c: 0x00000000 > [ 2.004625] pcieport 0002:20:00.0: save config 0x30: 0x00000000 > [ 2.004629] pcieport 0002:20:00.0: save config 0x34: 0x00000040 > [ 2.004633] pcieport 0002:20:00.0: save config 0x38: 0x00000000 > [ 2.004638] pcieport 0002:20:00.0: save config 0x3c: 0x00020194 > [ 2.004662] pcieport 0002:20:00.0: PME# enabled > > on 6.15-rc: > > cat /sys/class/pci_bus/0002:20/device/0002:20:00.0/power/runtime_status OK > suspended > > on 6.14: > > cat /sys/class/pci_bus/0002:20/device/0002:20:00.0/power/runtime_status OK > active > > So some other unrelated change in 6.15-rc just enabled RPM to suspend the unused > port. Ah right, makes sense, now that you mention I recall D3 became allowed for PCI bridge on non-x86 archs but I just couldn't make the connection in my mind. -- i.
© 2016 - 2026 Red Hat, Inc.