There is no reason to use the PCIe root port widget for MSIs for
pcie5 while both pcie4 and pcie6a are enjoying the ITS.

This is specially useful when booting the kernel at EL2, as KVM
can then configure the ITS to have MSIs directly injected in guests
(since this machine has a GICv4.1 implementation).

Tested on a x1e001de devkit.

Signed-off-by: Marc Zyngier <maz@kernel.org>
Cc: Sibi Sankar <quic_sibis@quicinc.com>
Cc: Konrad Dybcio <konradybcio@kernel.org>
Cc: Abel Vesa <abel.vesa@linaro.org>
Cc: Johan Hovold <johan+linaro@kernel.org>
---
 arch/arm64/boot/dts/qcom/x1e80100.dtsi | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/arch/arm64/boot/dts/qcom/x1e80100.dtsi b/arch/arm64/boot/dts/qcom/x1e80100.dtsi
index 3441d167a5cc..48f0ebd66863 100644
--- a/arch/arm64/boot/dts/qcom/x1e80100.dtsi
+++ b/arch/arm64/boot/dts/qcom/x1e80100.dtsi
@@ -3281,6 +3281,8 @@ pcie5: pci@1c00000 {
 			linux,pci-domain = <5>;
 			num-lanes = <2>;
 
+			msi-map = <0x0 &gic_its 0xd0000 0x10000>;
+
 			interrupts = <GIC_SPI 94 IRQ_TYPE_LEVEL_HIGH>,
 				     <GIC_SPI 95 IRQ_TYPE_LEVEL_HIGH>,
 				     <GIC_SPI 96 IRQ_TYPE_LEVEL_HIGH>,
-- 
2.39.5

On Thu, Oct 24, 2024 at 05:18:14PM +0100, Marc Zyngier wrote:
> There is no reason to use the PCIe root port widget for MSIs for
> pcie5 while both pcie4 and pcie6a are enjoying the ITS.
> 
> This is specially useful when booting the kernel at EL2, as KVM
> can then configure the ITS to have MSIs directly injected in guests
> (since this machine has a GICv4.1 implementation).
> 
> Tested on a x1e001de devkit.
> 
> Signed-off-by: Marc Zyngier <maz@kernel.org>
> Cc: Sibi Sankar <quic_sibis@quicinc.com>
> Cc: Konrad Dybcio <konradybcio@kernel.org>
> Cc: Abel Vesa <abel.vesa@linaro.org>
> Cc: Johan Hovold <johan+linaro@kernel.org>
> ---
>  arch/arm64/boot/dts/qcom/x1e80100.dtsi | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/arch/arm64/boot/dts/qcom/x1e80100.dtsi b/arch/arm64/boot/dts/qcom/x1e80100.dtsi
> index 3441d167a5cc..48f0ebd66863 100644
> --- a/arch/arm64/boot/dts/qcom/x1e80100.dtsi
> +++ b/arch/arm64/boot/dts/qcom/x1e80100.dtsi
> @@ -3281,6 +3281,8 @@ pcie5: pci@1c00000 {
>  			linux,pci-domain = <5>;
>  			num-lanes = <2>;
>  
> +			msi-map = <0x0 &gic_its 0xd0000 0x10000>;

As I just mentioned in another thread, and in the commit message of
9c4cd0aef259 ("arm64: dts: qcom: x1e80100: enable GICv3 ITS for PCIe")
this was done on purpose as

	PCIe5 (and PCIe3) can currently only be used with the internal
	MSI controller due to a platform (firmware) limitation

Did you try this when booting in EL1? If so we may need to enable this
per board.

Johan

On Thu, 24 Oct 2024 17:25:25 +0100,
Johan Hovold <johan@kernel.org> wrote:
> 
> On Thu, Oct 24, 2024 at 05:18:14PM +0100, Marc Zyngier wrote:
> > There is no reason to use the PCIe root port widget for MSIs for
> > pcie5 while both pcie4 and pcie6a are enjoying the ITS.
> > 
> > This is specially useful when booting the kernel at EL2, as KVM
> > can then configure the ITS to have MSIs directly injected in guests
> > (since this machine has a GICv4.1 implementation).
> > 
> > Tested on a x1e001de devkit.
> > 
> > Signed-off-by: Marc Zyngier <maz@kernel.org>
> > Cc: Sibi Sankar <quic_sibis@quicinc.com>
> > Cc: Konrad Dybcio <konradybcio@kernel.org>
> > Cc: Abel Vesa <abel.vesa@linaro.org>
> > Cc: Johan Hovold <johan+linaro@kernel.org>
> > ---
> >  arch/arm64/boot/dts/qcom/x1e80100.dtsi | 2 ++
> >  1 file changed, 2 insertions(+)
> > 
> > diff --git a/arch/arm64/boot/dts/qcom/x1e80100.dtsi b/arch/arm64/boot/dts/qcom/x1e80100.dtsi
> > index 3441d167a5cc..48f0ebd66863 100644
> > --- a/arch/arm64/boot/dts/qcom/x1e80100.dtsi
> > +++ b/arch/arm64/boot/dts/qcom/x1e80100.dtsi
> > @@ -3281,6 +3281,8 @@ pcie5: pci@1c00000 {
> >  			linux,pci-domain = <5>;
> >  			num-lanes = <2>;
> >  
> > +			msi-map = <0x0 &gic_its 0xd0000 0x10000>;
> 
> As I just mentioned in another thread, and in the commit message of
> 9c4cd0aef259 ("arm64: dts: qcom: x1e80100: enable GICv3 ITS for PCIe")
> this was done on purpose as
> 
> 	PCIe5 (and PCIe3) can currently only be used with the internal
> 	MSI controller due to a platform (firmware) limitation
> 
> Did you try this when booting in EL1? If so we may need to enable this
> per board.

Nah, you are absolutely correct: when booted at EL1, the ITS driver
reports that the ITS queue is no longer making forward progress as
soon as we are trying to map something in that range:

[    5.068749] ITS queue timeout (9984 9921)
[    5.072871] ITS cmd its_build_mapd_cmd failed

I suspect it trips over itself trying to interpret the command, and
that the other PCIe ports work by pure luck (maybe thanks to having
a even number?).

Comparing the logs, it is obvious that the hypervisor is not showing
us the actual HW topology: the ITS supports 64kB pages, which we use
when booted at EL2, while we only see 4kB support at EL1.

And the boot really is hilarious:

[    0.000000] ITS@0x0000000017040000: Devices Table too large, reduce ids 32->19

19 bits is the maximum the kernel can allocate with a 4kB page size.
I would like to see the face of a HW person if they had to design a
system with 32bit worth of DeviceID...

[    0.000000] ITS@0x0000000017040000: Devices too large, reduce ITS pages 1024->256

and 256 pages is the maximum we can describe to the ITS...

Obviously, this emulation was never really tested, since Windows
replaces it at boot time. Oh well.

I'll stash this patch as part of my "make EL2 great again" branch! ;-)

Thanks,

	M.

-- 
Without deviation from the norm, progress is not possible.