1. Patchew
  2. Xen
  3. View series

AW: AW: AW: AW: AW: Xen data from meta-virtualization layer

Leo Krueger posted 1 patch 3 years, 5 months ago
Failed in applying to current master (apply log)
AW: AW: AW: AW: AW: Xen data from meta-virtualization layer
Posted by Leo Krueger 3 years, 5 months ago 
Hi Julien,

finally I could try out what you suggested, please find my answers inline.

> -----Ursprüngliche Nachricht-----
> Von: Julien Grall <julien@xen.org>
> Gesendet: Mittwoch, 18. November 2020 13:24
> An: Stefano Stabellini <stefano.stabellini@xilinx.com>; Leo Krueger
> <leo.krueger@zal.aero>
> Cc: Peng Fan <peng.fan@nxp.com>; brucea@xilinx.com; Cornelia Bruelhart
> <cornelia.bruelhart@zal.aero>; oleksandr_andrushchenko@epam.com; xen-
> devel@lists.xenproject.org; Bertrand.Marquis@arm.com
> Betreff: Re: AW: AW: AW: AW: Xen data from meta-virtualization layer
> 
> Hi,
> 
> On 17/11/2020 23:53, Stefano Stabellini wrote:
> > Adding Bertrand, Oleksandr, Julien, and others -- they have a more
> > recent experience with GICv3 ITS than me and might be able to help.
> > I am attaching the device tree Leo sent a few days ago for reference.
> >
> >
> > Typically when you can set the ethernet link up and no packets are
> > exchanged it is because of a missing interrupt. In this case a missing
> > MSI.
> >
> > Bertrand, I believe you tried the GIC ITS driver with PCI devices
> > recently. It is expected to work correctly with MSIs in Dom0, right?
> 
> OSSTest has some hardware (e.g. Thunder-X) where ITS is required to boot
> Dom0. I haven't seen any failure on recent Xen. We are testing 4.11 and
> onwards on Thunder-X.
> 
> However, it may be possible that some more work is necessary for other
> hardware (e.g. workaround, missing code...). See more below.
> 
> >
> >
> >
> > On Tue, 17 Nov 2020, Leo Krueger wrote:
> >> Hi,
> >>
> >> I enabled CONFIG_HAS_ITS (what a stupid mistake by me to not set it
> >> before...) but then had to add the following node to my device tree
> >>
> >> 	gic_lpi_base: syscon@0x80000000 {
> >> 		compatible = "gic-lpi-base";
> 
> I couldn't find this compatible defined/used in Linux 5.10-rc4. @Leo, could
> you clarify which flavor/version of Linux you are using?

It is Linux 4.19 from Yocto (Warror release). XEN 4.13.2.
While searching around the Internet for any solution, I came across [0] which contained the gic-lpi-base node.
So I just tried adding it (quite desperate I know) and voila, it at least brought me one step further (XEN exposing the ITS)...

> 
> >> 		reg = <0x0 0x80000000 0x0 0x100000>;
> >> 		max-gic-redistributors = <2>;
> >> 	};
> >>
> >> to somehow change something in regard to the ITS and MSI/MSI-X
> >>
> >> (XEN) GICv3 initialization:
> >> (XEN)       gic_dist_addr=0x00000006000000
> >> (XEN)       gic_maintenance_irq=25
> >> (XEN)       gic_rdist_stride=0
> >> (XEN)       gic_rdist_regions=1
> >> (XEN)       redistributor regions:
> >> (XEN)         - region 0: 0x00000006040000 - 0x00000006080000
> >> (XEN) GICv3: using at most 57344 LPIs on the host.
> >> (XEN) GICv3: 288 lines, (IID 0001143b).
> >> (XEN) GICv3: Found ITS @0x6020000
> >> (XEN) using non-cacheable ITS command queue
> >> (XEN) GICv3: CPU0: Found redistributor in region 0 @000000004001c000
> >>
> >> [    0.000000] GICv3: Distributor has no Range Selector support
> >> [    0.000000] GICv3: no VLPI support, no direct LPI support
> >> [    0.000000] ITS [mem 0x06020000-0x0603ffff]
> >> [    0.000000] ITS@0x0000000006020000: allocated 65536 Devices
> @dc880000 (flat, esz 8, psz 64K, shr 1)
> >> [    0.000000] ITS@0x0000000006020000: allocated 32768 Interrupt
> Collections @dc820000 (flat, esz 2, psz 64K, shr 1)
> >> [    0.000000] GIC: using LPI property table @0x00000000dc830000
> >> [    0.000000] GICv3: CPU0: found redistributor 0 region
> 0:0x0000000006040000
> >> [    0.000000] CPU0: using LPI pending table @0x00000000dc840000
> >> ...
> >> [    0.040080] Platform MSI: gic-its domain created
> >> [    0.040136] PCI/MSI: /interrupt-controller/gic-its domain created
> >> [    0.040181] fsl-mc MSI: /interrupt-controller/gic-its domain created
> >>
> >>
> >> Still I am ending up with the " Failed to add - passthrough or MSI/MSI-X
> might fail!" log messages for some PCI devices, but at least the on-board
> ethernet ports (fsl_enetc ) are initialized.
> >> I can set the link up and a link is successfully established.
> 
> This message is normal. Xen on Arm is not yet aware of PCI devices and
> therefore the hypercalls to add PCI devices will return -EOPNOTSUPP.
> 
> However, this is not an issue because the virtual ITS implementation will
> allow dom0 to configure any devices.
> 
> >>
> >> But (!) I cannot receive or transmit anything (no error message...) and
> there seem to be no interrupts:
> >>
> >> 29:          0   ITS-MSI   1 Edge      gbe0-rxtx0
> >>   32:          0   ITS-MSI 8192 Edge      ptp_qoriq
> >>
> >> (from /proc/interrupts).
> >>
> >> Any idea on this one? I keep digging and checking whether my device tree
> needs some additional fixes.
> 
> Can you apply patch [1] and provide the logs? This will dump more
> information about the command received by the vITS and the one send to
> the host ITS.

For XEN 4.13.2 I had to adapt your patch slightly [1], see below (yes I know, quite ugly in parts).
Find attached the boot log and an output of "xl dmesg" which is truncated due to the large amount of messages.

When enabling the network interface (gbe0), the following output is visible:

root@kontron-sal28:~# ip link set up dev gbe0
(XEN) vgic-v3-its.c:902:d0v0 vITS  cmd 0x0c: 000000170000000c 0000000000000001 0000000000000000 0000000000000000
(XEN) vgic-v3-its.c:902:d0v0 vITS  cmd 0x05: 0000000000000005 0000000000000000 0000000000000000 0000000000000000
[   34.034598] Atheros 8031 ethernet 0000:00:00.3:05: attached PHY driver [Atheros 8031 ethernet] (mii_bus:phy_addr=0000:00:00.3:05, irq=POLL)
[   34.041111] 8021q: adding VLAN 0 to HW filter on device gbe0
[   34.041209] IPv6: ADDRCONF(NETDEV_UP): gbe0: link is not ready
root@kontron-sal28:~# [   35.041951] fsl_enetc 0000:00:00.0 gbe0: Link is Down
[   38.114426] fsl_enetc 0000:00:00.0 gbe0: Link is Up - 1Gbps/Full - flow control off
[   38.114508] IPv6: ADDRCONF(NETDEV_CHANGE): gbe0: link becomes ready

Does that tell you anything?

> 
> Note that Xen will need to be build with CONFIG_DEBUG=y in order to get
> some of the messages.
> 
> [...]
> 
> >>>> [    0.000000] GICv3: Distributor has no Range Selector support
> >>>>
> >>>> [    0.000000] GICv3: no VLPI support, no direct LPI support
> >>>>
> >>>> [    0.000000] GICv3: CPU0: found redistributor 0 region
> >>>> 0:0x0000000006040000
> >>>
> >>> "no VLPI support" is very suspicious, it looks like Dom0 doesn't
> >>> find any ITS support.
> VLPI is a feature that was introduced in GICv4 to directly inject LPI in the
> guest. So this is normal to see this message when running on Xen because
> we are going to only expose a GICv3 to a domain until at least we support
> nested virt.
> 
> However, you were right about that Xen didn't expose the ITS because the
> following lines were missing:
> 
> [    0.000000] ITS@0x0000000006020000: allocated 65536 Devices @dc880000
> (flat, esz 8, psz 64K, shr 1)
> 
> Cheers,

Best regards,
Leo

> 
> [1]
> diff --git a/xen/arch/arm/gic-v3-its.c b/xen/arch/arm/gic-v3-its.c index
> 9558bad96ac3..8a0a02308e74 100644
> --- a/xen/arch/arm/gic-v3-its.c
> +++ b/xen/arch/arm/gic-v3-its.c
> @@ -87,6 +87,10 @@ static int its_send_command(struct host_its *hw_its,
> const void *its_cmd)
>       /* No ITS commands from an interrupt handler (at the moment). */
>       ASSERT(!in_irq());
> 
> +    printk(XENLOG_WARNING, "pITS  cmd 0x%02lx: %016lx %016lx %016lx
> %016lx\n",
> +           its_cmd_get_command(command),
> +           command[0], command[1], command[2], command[3]);
> +
>       spin_lock(&hw_its->cmd_lock);
> 
>       do {
> diff --git a/xen/arch/arm/gic-v3-lpi.c b/xen/arch/arm/gic-v3-lpi.c index
> 869bc97fa1aa..e7c5bcd8d423 100644
> --- a/xen/arch/arm/gic-v3-lpi.c
> +++ b/xen/arch/arm/gic-v3-lpi.c
> @@ -183,7 +183,10 @@ void gicv3_do_LPI(unsigned int lpi)
>       /* Find out if a guest mapped something to this physical LPI. */
>       hlpip = gic_get_host_lpi(lpi);
>       if ( !hlpip )
> +    {
> +        printk("%s: Received LPI %u but it is not mapped?\n", __func__,
> lpi);
>           goto out;
> +    }
> 
>       hlpi.data = read_u64_atomic(&hlpip->data);
> 
> @@ -222,6 +225,9 @@ void gicv3_lpi_update_host_entry(uint32_t host_lpi,
> int domain_id,
>   {
>       union host_lpi *hlpip, hlpi;
> 
> +    printk("%s: host_lpi %u domain %d virq_lpi %u\n",
> +           __func__, host_lpi, domain_id, virq_lpi);
> +
>       ASSERT(host_lpi >= LPI_OFFSET);
> 
>       host_lpi -= LPI_OFFSET;
> diff --git a/xen/arch/arm/vgic-v3-its.c b/xen/arch/arm/vgic-v3-its.c index
> 58d939b85f92..89ef137b3e6b 100644
> --- a/xen/arch/arm/vgic-v3-its.c
> +++ b/xen/arch/arm/vgic-v3-its.c
> @@ -897,7 +897,7 @@ out_unlock:
> 
>   static void dump_its_command(uint64_t *command)
>   {
> -    gdprintk(XENLOG_WARNING, "  cmd 0x%02lx: %016lx %016lx %016lx
> %016lx\n",
> +    gdprintk(XENLOG_WARNING, "vITS  cmd 0x%02lx: %016lx %016lx %016lx
> %016lx\n",
>                its_cmd_get_command(command),
>                command[0], command[1], command[2], command[3]);
>   }
> @@ -926,6 +926,8 @@ static int vgic_its_handle_cmds(struct domain *d,
> struct virt_its *its)
>           if ( ret )
>               return ret;
> 
> +        dump_its_command(command);
> +
>           switch ( its_cmd_get_command(command) )
>           {
>           case GITS_CMD_CLEAR:
> 
> 
> --
> Julien Grall

[0] https://www.mail-archive.com/u-boot@lists.denx.de/msg379708.html
[1] 
diff --git a/xen/arch/arm/gic-v3-its.c b/xen/arch/arm/gic-v3-its.c
index 9558bad96a..d175ba52b0 100644
--- a/xen/arch/arm/gic-v3-its.c
+++ b/xen/arch/arm/gic-v3-its.c
@@ -87,6 +87,10 @@ static int its_send_command(struct host_its *hw_its, const void *its_cmd)
     /* No ITS commands from an interrupt handler (at the moment). */
     ASSERT(!in_irq());

+    printk(XENLOG_WARNING "pITS  cmd 0x%02lx: %016lx %016lx %016lx %016lx\n",
+        (((uint64_t *) its_cmd)[0] >> 0) & GENMASK(8 - 1, 0),
+        ((uint64_t *) its_cmd)[0], ((uint64_t *) its_cmd)[1], ((uint64_t *) its_cmd)[2], ((uint64_t *) its_cmd)[3]);
+
     spin_lock(&hw_its->cmd_lock);

     do {
diff --git a/xen/arch/arm/gic-v3-lpi.c b/xen/arch/arm/gic-v3-lpi.c
index 78b9521b21..2c3b0fc9e5 100644
--- a/xen/arch/arm/gic-v3-lpi.c
+++ b/xen/arch/arm/gic-v3-lpi.c
@@ -181,8 +181,10 @@ void gicv3_do_LPI(unsigned int lpi)

     /* Find out if a guest mapped something to this physical LPI. */
     hlpip = gic_get_host_lpi(lpi);
-    if ( !hlpip )
+    if ( !hlpip ) {
+        printk("%s: Received LPI %u but it is not mapped?\n", __func__, lpi);
         goto out;
+    }

     hlpi.data = read_u64_atomic(&hlpip->data);

@@ -221,6 +223,9 @@ void gicv3_lpi_update_host_entry(uint32_t host_lpi, int domain_id,
 {
     union host_lpi *hlpip, hlpi;

+    printk("%s: host_lpi %u domain %d virt_lpi %u\n",
+        __func__, host_lpi, domain_id, virt_lpi);
+
     ASSERT(host_lpi >= LPI_OFFSET);

     host_lpi -= LPI_OFFSET;
diff --git a/xen/arch/arm/vgic-v3-its.c b/xen/arch/arm/vgic-v3-its.c
index 6e153c698d..dd5081ef80 100644
--- a/xen/arch/arm/vgic-v3-its.c
+++ b/xen/arch/arm/vgic-v3-its.c
@@ -897,7 +897,7 @@ out_unlock:

 static void dump_its_command(uint64_t *command)
 {
-    gdprintk(XENLOG_WARNING, "  cmd 0x%02lx: %016lx %016lx %016lx %016lx\n",
+    gdprintk(XENLOG_WARNING, "vITS  cmd 0x%02lx: %016lx %016lx %016lx %016lx\n",
              its_cmd_get_command(command),
              command[0], command[1], command[2], command[3]);
 }
@@ -926,6 +926,8 @@ static int vgic_its_handle_cmds(struct domain *d, struct virt_its *its)
         if ( ret )
             return ret;

+        dump_its_command(command);
+
         switch ( its_cmd_get_command(command) )
         {
         case GITS_CMD_CLEAR:
 Xen 4.13.2
(XEN) Xen version 4.13.2 (@) (aarch64-poky-linux-gcc (GCC) 8.3.0) debug=y  Sun Nov 22 21:56:06 UTC 2020
(XEN) Latest ChangeSet: Fri Oct 30 12:24:39 2020 +0100 git:0060ac29bc-dirty
(XEN) build-id: 1f6c38235b06eb99a3e73b3846f910a4df9dc028
(XEN) Processor: 410fd083: "ARM Limited", variant: 0x0, part 0xd08, rev 0x3
(XEN) 64-bit Execution:
(XEN)   Processor Features: 0000000001002222 0000000000000000
(XEN)     Exception Levels: EL3:64+32 EL2:64+32 EL1:64+32 EL0:64+32
(XEN)     Extensions: FloatingPoint AdvancedSIMD GICv3-SysReg
(XEN)   Debug Features: 0000000010305106 0000000000000000
(XEN)   Auxiliary Features: 0000000000000000 0000000000000000
(XEN)   Memory Model Features: 0000000000001124 0000000000000000
(XEN)   ISA Features:  0000000000010000 0000000000000000
(XEN) 32-bit Execution:
(XEN)   Processor Features: 00000131:10011011
(XEN)     Instruction Sets: AArch32 A32 Thumb Thumb-2 Jazelle
(XEN)     Extensions: GenericTimer Security
(XEN)   Debug Features: 03010066
(XEN)   Auxiliary Features: 00000000
(XEN)   Memory Model Features: 10201105 40000000 01260000 02102211
(XEN)  ISA Features: 02101110 13112111 21232042 01112131 00011142 00010001
(XEN) Using SMC Calling Convention v1.0
(XEN) Using PSCI v0.2
(XEN) SMP: Allowing 2 CPUs
(XEN) enabled workaround for: ARM erratum 1319537
(XEN) Generic Timer IRQ: phys=30 hyp=26 virt=27 Freq: 25000 KHz
(XEN) GICv3 initialization:
(XEN)       gic_dist_addr=0x00000006000000
(XEN)       gic_maintenance_irq=25
(XEN)       gic_rdist_stride=0
(XEN)       gic_rdist_regions=1
(XEN)       redistributor regions:
(XEN)         - region 0: 0x00000006040000 - 0x00000006080000
(XEN) GICv3: using at most 57344 LPIs on the host.
(XEN) GICv3: 288 lines, (IID 0001143b).
(XEN) GICv3: Found ITS @0x6020000
(XEN) using non-cacheable ITS command queue
(XEN) GICv3: CPU0: Found redistributor in region 0 @000000004001c000
(XEN) pITS  cmd 0x09: 0000000000000009 0000000000000000 8000000000000000 0000000000000000
(XEN) pITS  cmd 0x05: 0000000000000005 0000000000000000 0000000000000000 0000000000000000
(XEN) XSM Framework v1.0.0 initialized
(XEN) Initialising XSM SILO mode
(XEN) Using scheduler: null Scheduler (null)
(XEN) Initializing null scheduler
(XEN) WARNING: This is experimental software in development.
(XEN) Use at your own risk.
(XEN) Allocated console ring of 16 KiB.
(XEN) CPU0: Guest atomics will try 13 times before pausing the domain
(XEN) Bringing up CPU1
(XEN) GICv3: CPU1: Found redistributor in region 0 @000000004003c000
(XEN) pITS  cmd 0x09: 0000000000000009 0000000000000000 8000000000010001 0000000000000000
(XEN) pITS  cmd 0x05: 0000000000000005 0000000000000000 0000000000010000 0000000000000000
(XEN) CPU1: Guest atomics will try 12 times before pausing the domain
(XEN) CPU 1 booted.
(XEN) Brought up 2 CPUs
(XEN) I/O virtualisation disabled
(XEN) P2M: 44-bit IPA with 44-bit PA and 8-bit VMID
(XEN) P2M: 4 levels with order-0 root, VTCR 0x80043594
(XEN) alternatives: Patching with alt table 00000000002d40e8 -> 00000000002d48bc
(XEN) *** LOADING DOMAIN 0 ***
(XEN) Loading d0 kernel from boot module @ 0000000081200000
(XEN) Allocating 1:1 mappings totalling 512MB for dom0:
(XEN) BANK[0] 0x000000c0000000-0x000000e0000000 (512MB)
(XEN) Grant table range: 0x00000081000000-0x00000081040000
(XEN) Allocating PPI 16 for event channel interrupt
(XEN) Loading zImage from 0000000081200000 to 00000000c0080000-00000000c1597008
(XEN) Loading d0 DTB to 0x00000000c8000000-0x00000000c8006cb6
(XEN) Initial low memory virq threshold set at 0x4000 pages.
(XEN) Std. Loglevel: All
(XEN) Guest Loglevel: All
(XEN) ***************************************************
(XEN) No support for ARM_SMCCC_ARCH_WORKAROUND_1.
(XEN) Please update your firmware.
(XEN) ***************************************************
(XEN) No support for ARM_SMCCC_ARCH_WORKAROUND_1.
(XEN) Please update your firmware.
(XEN) ***************************************************
(XEN) 3... 2... 1...
(XEN) *** Serial input to DOM0 (type 'CTRL-a' three times to switch input)
(XEN) sched_null.c:344: 0 <-- d0v0
(XEN) Freed 344kB init memory.
(XEN) d0v0: vGICD: unhandled word write 0x000000ffffffff to ICACTIVER4
(XEN) d0v0: vGICD: unhandled word write 0x000000ffffffff to ICACTIVER8
(XEN) d0v0: vGICD: unhandled word write 0x000000ffffffff to ICACTIVER12
(XEN) d0v0: vGICD: unhandled word write 0x000000ffffffff to ICACTIVER16
(XEN) d0v0: vGICD: unhandled word write 0x000000ffffffff to ICACTIVER20
(XEN) d0v0: vGICD: unhandled word write 0x000000ffffffff to ICACTIVER24
(XEN) d0v0: vGICD: unhandled word write 0x000000ffffffff to ICACTIVER28
(XEN) d0v0: vGICD: unhandled word write 0x000000ffffffff to ICACTIVER32
(XEN) d0v0: vGICR: SGI: unhandled word write 0x000000ffffffff to ICACTIVER0
(XEN) vgic-v3-its.c:902:d0v0 vITS  cmd 0x09: 0000000000000009 0000000000000000 8000000000000000 0000000000000000
(XEN) vgic-v3-its.c:902:d0v0 vITS  cmd 0x05: 0000000000000005 0000000000000000 0000000000000000 0000000000000000
(XEN) vgic-v3-its.c:902:d0v0 vITS  cmd 0x0d: 000000000000000d 0000000000000000 0000000000000000 0000000000000000
[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd083]
[    0.000000] Linux version 4.19.68+g4f0ecdbd4f33 (oe-user@oe-host) (gcc version 8.3.0 (GCC)) #1 SMP PREEMPT Thu Oct 29 22:05:56 UTC 2020
[    0.000000] Machine model: Kontron SMARC-sAL28
[    0.000000] Xen 4.13 support found
[    0.000000] cma: Reserved 32 MiB at 0x00000000de000000
[    0.000000] NUMA: No NUMA configuration found
[    0.000000] NUMA: Faking a node at [mem 0x0000000000000000-0x00000000dfffffff]
[    0.000000] NUMA: NODE_DATA [mem 0xddfd5b40-0xddfd72ff]
[    0.000000] Zone ranges:
[    0.000000]   DMA32    [mem 0x00000000c0000000-0x00000000dfffffff]
[    0.000000]   Normal   empty
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x00000000c0000000-0x00000000dfffffff]
[    0.000000] Initmem setup node 0 [mem 0x00000000c0000000-0x00000000dfffffff]
[    0.000000] psci: probing for conduit method from DT.
[    0.000000] psci: PSCIv1.1 detected in firmware.
[    0.000000] psci: Using standard PSCI v0.2 function IDs
[    0.000000] psci: Trusted OS migration not required
[    0.000000] psci: SMC Calling Convention v1.1
[    0.000000] random: get_random_bytes called from start_kernel+0x94/0x3ec with crng_init=0
[    0.000000] percpu: Embedded 25 pages/cpu s62296 r8192 d31912 u102400
[    0.000000] Detected PIPT I-cache on CPU0
[    0.000000] CPU features: enabling workaround for EL2 vector hardening
[    0.000000] CPU features: enabling workaround for Speculative Store Bypass Disable
[    0.000000] CPU features: detected: Kernel page table isolation (KPTI)
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 129024
[    0.000000] Policy zone: DMA32
[    0.000000] Kernel command line: root=/dev/mmcblk1p2 console=hvc0 earlycon=xen earlyprintk=xen clk_ignore_unused rootwait
[    0.000000] Memory: 452116K/524288K available (12862K kernel code, 1628K rwdata, 5708K rodata, 1344K init, 899K bss, 39404K reserved, 32768K cma-reserved)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[    0.000000] rcu: Preemptible hierarchical RCU implementation.
[    0.000000] rcu:     RCU restricting CPUs from NR_CPUS=64 to nr_cpu_ids=1.
[    0.000000]  Tasks RCU enabled.
[    0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1
[    0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
[    0.000000] GICv3: Distributor has no Range Selector support
[    0.000000] GICv3: no VLPI support, no direct LPI support
[    0.000000] ITS [mem 0x06020000-0x0603ffff]
[    0.000000] ITS@0x0000000006020000: allocated 65536 Devices @dc880000 (flat, esz 8, psz 64K, shr 1)
[    0.000000] ITS@0x0000000006020000: allocated 32768 Interrupt Collections @dc820000 (flat, esz 2, psz 64K, shr 1)
[    0.000000] GIC: using LPI property table @0x00000000dc830000
[    0.000000] GICv3: CPU0: found redistributor 0 region 0:0x0000000006040000
[    0.000000] CPU0: using LPI pending table @0x00000000dc840000
[    0.000000] arch_timer: cp15 timer(s) running at 25.00MHz (virt).
[    0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0x5c40939b5, max_idle_ns: 440795202646 ns
[    0.000003] sched_clock: 56 bits at 25MHz, resolution 40ns, wraps every 4398046511100ns
[    0.000230] Console: colour dummy device 80x25
[    0.000812] console [hvc0] enabled
[    0.000857] Calibrating delay loop (skipped), value calculated using timer frequency.. 50.00 BogoMIPS (lpj=100000)
[    0.000882] pid_max: default: 32768 minimum: 301
[    0.000927] Security Framework initialized
[    0.001178] Dentry cache hash table entries: 65536 (order: 7, 524288 bytes)
[    0.001293] Inode-cache hash table entries: 32768 (order: 6, 262144 bytes)
[    0.001323] Mount-cache hash table entries: 1024 (order: 1, 8192 bytes)
[    0.001342] Mountpoint-cache hash table entries: 1024 (order: 1, 8192 bytes)
[    0.024034] ASID allocator initialised with 32768 entries
[    0.024549] xen:grant_table: Grant tables using version 1 layout
[    0.024575] Grant table initialized
[    0.024606] xen:events: Using FIFO-based ABI
[    0.024646] Xen: initializing cpu0
[    0.032041] rcu: Hierarchical SRCU implementation.
[    0.040081] Platform MSI: gic-its domain created
[    0.040140] PCI/MSI: /interrupt-controller/gic-its domain created
[    0.040187] fsl-mc MSI: /interrupt-controller/gic-its domain created
[    0.048072] smp: Bringing up secondary CPUs ...
[    0.048086] smp: Brought up 1 node, 1 CPU
[    0.048097] SMP: Total of 1 processors activated.
[    0.048113] CPU features: detected: GIC system register CPU interface
[    0.048129] CPU features: detected: 32-bit EL0 Support
[    0.051116] CPU: All CPU(s) started at EL1
[    0.051133] alternatives: patching kernel code
[    0.051728] devtmpfs: initialized
[    0.056337] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    0.056374] futex hash table entries: 256 (order: 2, 16384 bytes)
[    0.056912] pinctrl core: initialized pinctrl subsystem
[    0.057704] NET: Registered protocol family 16
[    0.057908] audit: initializing netlink subsys (disabled)
[    0.058469] audit: type=2000 audit(0.056:1): state=initialized audit_enabled=0 res=1
[    0.058925] vdso: 2 pages (1 code @ (____ptrval____), 1 data @ (____ptrval____))
[    0.058947] hw-breakpoint: found 6 breakpoint and 4 watchpoint registers.
[    0.062295] DMA: preallocated 256 KiB pool for atomic allocations
[    0.062394] xen:swiotlb_xen: Warning: only able to allocate 4 MB for software IO TLB
[    0.063142] software IO TLB: mapped [mem 0xdbc00000-0xdc000000] (4MB)
[    0.064269] Serial: AMBA PL011 UART driver
[    0.104947] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
[    0.108728] cryptd: max_cpu_qlen set to 1000
[    0.117877] xen:balloon: Initialising balloon driver
[    0.119767] vgaarb: loaded
[    0.120078] SCSI subsystem initialized
[    0.121057] usbcore: registered new interface driver usbfs
[    0.121130] usbcore: registered new interface driver hub
[    0.121204] usbcore: registered new device driver usb
[    0.121736] imx-i2c 2000000.i2c: scl-gpios not found
[    0.122155] of_dma_request_slave_channel: dma-names property of node '/soc/i2c@2000000' missing or empty
[    0.122179] i2c i2c-0: IMX I2C adapter registered
[    0.122328] imx-i2c 2030000.i2c: scl-gpios not found
[    0.122452] of_dma_request_slave_channel: dma-names property of node '/soc/i2c@2030000' missing or empty
[    0.122475] i2c i2c-1: IMX I2C adapter registered
[    0.122616] imx-i2c 2040000.i2c: scl-gpios not found
[    0.122802] of_dma_request_slave_channel: dma-names property of node '/soc/i2c@2040000' missing or empty
[    0.122823] i2c i2c-2: IMX I2C adapter registered
[    0.123273] pps_core: LinuxPPS API ver. 1 registered
[    0.123288] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[    0.123346] PTP clock support registered
[    0.123546] EDAC MC: Ver: 3.0.0
[    0.126576] No BMan portals available!
[    0.126954] QMan: Allocated lookup table at (____ptrval____), entry count 65537
[    0.127075] No QMan portals available!
[    0.127223] No USDPAA memory, no 'fsl,usdpaa-mem' in device-tree
[    0.128688] Advanced Linux Sound Architecture Driver Initialized.
[    0.129396] clocksource: Switched to clocksource arch_sys_counter
[    0.129484] VFS: Disk quotas dquot_6.6.0
[    0.129525] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    0.138006] NET: Registered protocol family 2
[    0.138292] tcp_listen_portaddr_hash hash table entries: 256 (order: 0, 4096 bytes)
[    0.138406] TCP established hash table entries: 4096 (order: 3, 32768 bytes)
[    0.138440] TCP bind hash table entries: 4096 (order: 4, 65536 bytes)
[    0.138498] TCP: Hash tables configured (established 4096 bind 4096)
[    0.138567] UDP hash table entries: 256 (order: 1, 8192 bytes)
[    0.138589] UDP-Lite hash table entries: 256 (order: 1, 8192 bytes)
[    0.138654] NET: Registered protocol family 1
[    0.148017] RPC: Registered named UNIX socket transport module.
[    0.148040] RPC: Registered udp transport module.
[    0.148052] RPC: Registered tcp transport module.
[    0.148064] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    0.148406] kvm [1]: HYP mode not available
[    0.149251] Initialise system trusted keyrings
[    0.149762] workingset: timestamp_bits=44 max_order=17 bucket_order=0
[    0.152943] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    0.156914] NFS: Registering the id_resolver key type
[    0.156948] Key type id_resolver registered
[    0.156959] Key type id_legacy registered
[    0.156973] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[    0.157009] jffs2: version 2.2. (NAND) © 2001-2006 Red Hat, Inc.
[    0.157486] 9p: Installing v9fs 9p2000 file system support
[    0.161737] Key type asymmetric registered
[    0.161758] Asymmetric key parser 'x509' registered
[    0.161808] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 244)
[    0.161826] io scheduler noop registered
[    0.161837] io scheduler deadline registered
[    0.161931] io scheduler cfq registered (default)
[    0.161944] io scheduler mq-deadline registered
[    0.161957] io scheduler kyber registered
[    0.164994] pci-host-generic 1f0000000.pcie: host bridge /soc/pcie@1f0000000 ranges:
[    0.165029] pci-host-generic 1f0000000.pcie:   MEM 0x1f8000000..0x1f815ffff -> 0x00000000
[    0.165054] pci-host-generic 1f0000000.pcie:   MEM 0x1f8160000..0x1f81cffff -> 0x00000000
[    0.165077] pci-host-generic 1f0000000.pcie:   MEM 0x1f81d0000..0x1f81effff -> 0x00000000
[    0.165100] pci-host-generic 1f0000000.pcie:   MEM 0x1f81f0000..0x1f820ffff -> 0x00000000
[    0.165123] pci-host-generic 1f0000000.pcie:   MEM 0x1f8210000..0x1f822ffff -> 0x00000000
[    0.165146] pci-host-generic 1f0000000.pcie:   MEM 0x1f8230000..0x1f824ffff -> 0x00000000
[    0.165167] pci-host-generic 1f0000000.pcie:   MEM 0x1fc000000..0x1fc3fffff -> 0x00000000
[    0.165218] pci-host-generic 1f0000000.pcie: ECAM at [mem 0x1f0000000-0x1f00fffff] for [bus 00]
[    0.165305] pci-host-generic 1f0000000.pcie: PCI host bridge to bus 0000:00
[    0.165322] pci_bus 0000:00: root bus resource [bus 00]
[    0.165337] pci_bus 0000:00: root bus resource [mem 0x1f8000000-0x1f815ffff] (bus address [0x00000000-0x0015ffff])
[    0.165368] pci_bus 0000:00: root bus resource [mem 0x1f8160000-0x1f81cffff pref] (bus address [0x00000000-0x0006ffff])
[    0.168237] pci_bus 0000:00: root bus resource [mem 0x1f81d0000-0x1f81effff] (bus address [0x00000000-0x0001ffff])
[    0.168270] pci_bus 0000:00: root bus resource [mem 0x1f81f0000-0x1f820ffff pref] (bus address [0x00000000-0x0001ffff])
[    0.168293] pci_bus 0000:00: root bus resource [mem 0x1f8210000-0x1f822ffff] (bus address [0x00000000-0x0001ffff])
[    0.168316] pci_bus 0000:00: root bus resource [mem 0x1f8230000-0x1f824ffff pref] (bus address [0x00000000-0x0001ffff])
[    0.168339] pci_bus 0000:00: root bus resource [mem 0x1fc000000-0x1fc3fffff] (bus address [0x00000000-0x003fffff])
[    0.168463] pci 0000:00:00.0: VF(n) BAR0 space: [mem 0x1f81d0000-0x1f81effff 64bit] (contains BAR0 for 2 VFs)
[    0.168486] pci 0000:00:00.0: VF(n) BAR2 space: [mem 0x1f81f0000-0x1f820ffff 64bit pref] (contains BAR2 for 2 VFs)
(XEN) physdev.c:16:d0v0 PHYSDEVOP cmd=25: not implemented
(XEN) physdev.c:16:d0v0 PHYSDEVOP cmd=15: not implemented
[    0.168639] pci 0000:00:00.0: Failed to add - passthrough or MSI/MSI-X might fail!
[    0.168805] pci 0000:00:00.1: VF(n) BAR0 space: [mem 0x1f8210000-0x1f822ffff 64bit] (contains BAR0 for 2 VFs)
[    0.168829] pci 0000:00:00.1: VF(n) BAR2 space: [mem 0x1f8230000-0x1f824ffff 64bit pref] (contains BAR2 for 2 VFs)
(XEN) physdev.c:16:d0v0 PHYSDEVOP cmd=15: not implemented
[    0.168931] pci 0000:00:00.1: Failed to add - passthrough or MSI/MSI-X might fail!
(XEN) physdev.c:16:d0v0 PHYSDEVOP cmd=15: not implemented
[    0.169138] pci 0000:00:00.2: Failed to add - passthrough or MSI/MSI-X might fail!
(XEN) physdev.c:16:d0v0 PHYSDEVOP cmd=15: not implemented
[    0.169332] pci 0000:00:00.3: Failed to add - passthrough or MSI/MSI-X might fail!
(XEN) physdev.c:16:d0v0 PHYSDEVOP cmd=15: not implemented
[    0.169613] pci 0000:00:00.4: Failed to add - passthrough or MSI/MSI-X might fail!
(XEN) physdev.c:16:d0v0 PHYSDEVOP cmd=15: not implemented
[    0.172781] pci 0000:00:00.5: Failed to add - passthrough or MSI/MSI-X might fail!
(XEN) physdev.c:16:d0v0 PHYSDEVOP cmd=15: not implemented
[    0.172995] pci 0000:00:00.6: Failed to add - passthrough or MSI/MSI-X might fail!
[    0.173964] OF: /soc/pcie@1f0000000: Invalid msi-map translation - no match for rid 0xf8 on           (null)
(XEN) physdev.c:16:d0v0 PHYSDEVOP cmd=15: not implemented
[    0.174044] pci 0000:00:1f.0: Failed to add - passthrough or MSI/MSI-X might fail!
[    0.174538] layerscape-pcie 3400000.pcie: host bridge /soc/pcie@3400000 ranges:
[    0.174570] layerscape-pcie 3400000.pcie:    IO 0x8000010000..0x800001ffff -> 0x00000000
[    0.174594] layerscape-pcie 3400000.pcie:   MEM 0x8040000000..0x807fffffff -> 0x40000000
[    0.174837] layerscape-pcie 3400000.pcie: PCI host bridge to bus 0001:00
[    0.174856] pci_bus 0001:00: root bus resource [bus 00-ff]
[    0.174870] pci_bus 0001:00: root bus resource [io  0x0000-0xffff]
[    0.174887] pci_bus 0001:00: root bus resource [mem 0x8040000000-0x807fffffff] (bus address [0x40000000-0x7fffffff])
[    0.175070] pci 0001:00:00.0: Failed to add - passthrough or MSI/MSI-X might fail!
[    0.176593] pci 0001:00:00.0: BAR 6: assigned [mem 0x8040000000-0x80400007ff pref]
[    0.176613] pci 0001:00:00.0: PCI bridge to [bus 01]
[    0.177125] pcieport 0001:00:00.0: Signaling PME with IRQ 16
[    0.177281] pcieport 0001:00:00.0: AER enabled with IRQ 17
[    0.177874] layerscape-pcie 3500000.pcie: host bridge /soc/pcie@3500000 ranges:
[    0.177908] layerscape-pcie 3500000.pcie:    IO 0x8800010000..0x880001ffff -> 0x00000000
[    0.177939] layerscape-pcie 3500000.pcie:   MEM 0x8840000000..0x887fffffff -> 0x40000000
[    0.178174] layerscape-pcie 3500000.pcie: PCI host bridge to bus 0002:00
[    0.178193] pci_bus 0002:00: root bus resource [bus 00-ff]
[    0.178208] pci_bus 0002:00: root bus resource [io  0x10000-0x1ffff] (bus address [0x0000-0xffff])
[    0.178229] pci_bus 0002:00: root bus resource [mem 0x8840000000-0x887fffffff] (bus address [0x40000000-0x7fffffff])
[    0.178416] pci 0002:00:00.0: Failed to add - passthrough or MSI/MSI-X might fail!
[    0.179940] pci 0002:00:00.0: BAR 6: assigned [mem 0x8840000000-0x88400007ff pref]
[    0.179961] pci 0002:00:00.0: PCI bridge to [bus 01]
[    0.180274] pcieport 0002:00:00.0: Signaling PME with IRQ 18
[    0.180458] pcieport 0002:00:00.0: AER enabled with IRQ 19
[    0.186395] Freescale LS2 console driver
[    0.186531] fsl-ls2-console: device fsl_mc_console registered
[    0.186618] fsl-ls2-console: device fsl_aiop_console registered
[    0.189325] xen:xen_evtchn: Event-channel device installed
[    0.196223] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[    0.198204] SuperH (H)SCI(F) driver initialized
[    0.198499] msm_serial: driver initialized
[    0.198726] 2270000.serial: ttyLP0 at MMIO 0x2270000 (irq = 10, base_baud = 12500000) is a FSL_LPUART
[    0.203664] cacheinfo: Unable to detect cache hierarchy for CPU 0
[    0.209143] loop: module loaded
[    0.209237] Invalid max_queues (4), will use default max: 1.
[    0.211203] at24 0-0050: 4096 byte 24c32 EEPROM, writable, 32 bytes/write
[    0.212609] at24 2-0050: 4096 byte 24c32 EEPROM, writable, 32 bytes/write
[    0.216406] sl28cpld 0-004a: registered IRQ 27
[    0.216426] sl28cpld 0-004a: successfully probed. CPLD version 18.
[    0.239712] m25p80 spi0.0: w25q32jw (4096 Kbytes)
[    0.240316] 10 fixed-partitions partitions found on MTD device 20c0000.spi
[    0.240335] Creating 10 MTD partitions on "20c0000.spi":
[    0.240351] 0x000000000000-0x000000010000 : "rcw"
[    0.240834] 0x000000010000-0x000000100000 : "failsafe bootloader"
[    0.241282] 0x000000100000-0x000000140000 : "failsafe DP firmware"
[    0.241792] 0x000000140000-0x0000001e0000 : "failsafe trusted firmware"
[    0.242252] 0x0000001e0000-0x000000200000 : "reserved"
[    0.242708] 0x000000200000-0x000000210000 : "configuration store"
[    0.243159] 0x000000210000-0x000000300000 : "bootloader"
[    0.243613] 0x000000300000-0x000000340000 : "DP firmware"
[    0.244069] 0x000000340000-0x0000003e0000 : "trusted firmware"
[    0.244511] 0x0000003e0000-0x000000400000 : "bootloader environment"
[    0.250047] libphy: Fixed MDIO Bus: probed
[    0.252046] tun: Universal TUN/TAP device driver, 1.6
[    0.256135] thunder_xcv, ver 1.0
[    0.256220] thunder_bgx, ver 1.0
[    0.256286] nicpf, ver 1.0
[    0.256506] Freescale FM module, FMD API version 21.1.0
[    0.256626] Freescale FM Ports module
[    0.256637] fsl_mac: fsl_mac: FSL FMan MAC API based driver
[    0.256737] fsl_dpa: FSL DPAA Ethernet driver
[    0.256825] fsl_advanced: FSL DPAA Advanced drivers:
[    0.256838] fsl_proxy: FSL DPAA Proxy initialization driver
[    0.256922] fsl_oh: FSL FMan Offline Parsing port driver
[    0.361406] fsl_enetc 0000:00:00.0: enabling device (0400 -> 0402)
[    0.361469] fsl_enetc 0000:00:00.0: no MAC address specified for SI1, using 42:14:fd:42:55:1d
[    0.361491] fsl_enetc 0000:00:00.0: no MAC address specified for SI2, using 1e:22:15:9a:c9:e9
(XEN) vgic-v3-its.c:902:d0v0 vITS  cmd 0x08: 0000001700000008 0000000000000000 80000000db471200 0000000000000000
(XEN) pITS  cmd 0x08: 0000001700000008 0000000000000004 80000020f9de4600 0000000000000000
(XEN) pITS  cmd 0x0a: 000000170000000a 0000200000000000 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0c: 000000170000000c 0000000000000000 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0a: 000000170000000a 0000200100000001 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0c: 000000170000000c 0000000000000001 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0a: 000000170000000a 0000200200000002 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0c: 000000170000000c 0000000000000002 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0a: 000000170000000a 0000200300000003 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0c: 000000170000000c 0000000000000003 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0a: 000000170000000a 0000200400000004 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0c: 000000170000000c 0000000000000004 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0a: 000000170000000a 0000200500000005 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0c: 000000170000000c 0000000000000005 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0a: 000000170000000a 0000200600000006 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0c: 000000170000000c 0000000000000006 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0a: 000000170000000a 0000200700000007 0000000000000000 00000000000000[    1.656243] fsl_enetc 0000:00:00.1 gbe1: renamed from eth1
[    1.721814] fsl_enetc 0000:00:00.0 gbe0: renamed from eth0
[    3.011356] urandom_read: 3 callbacks suppressed
[    3.011360] random: udevd: uninitialized urandom read (16 bytes read)
[    3.011490] random: udevd: uninitialized urandom read (16 bytes read)
[    3.011542] random: udevd: uninitialized urandom read (16 bytes read)
[    3.012782] udevd[1812]: specified group 'kvm' unknown
[    4.010698] FAT-fs (mmcblk1p1): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.
[    6.043586] oak_pci: loading out-of-tree module taints kernel.
[    6.049496] Marvell PCIe Switch Driver - (oak) version 0.0.0
[    6.049523] Copyright (c) Marvell - 2018
[    6.073648] EXT4-fs (mmcblk1p2): re-mounted. Opts: (null)
[    6.516987] random: dd: uninitialized urandom read (512 bytes read)
ALSA: Restoring mixer settings...
/usr/sbin/alsactl: load_state:1735: No soundcards found...
INIT: Entering runlevel: 5
Configuring network interfaces... done.
Starting system message bus: dbus.
Mounting cgroups...Done
Starting Dropbear SSH server: [    6.764603] NET: Registered protocol family 10
[    6.769738] Segment Routing with IPv6
dropbear.
Starting rpcbind daemon...done.
Starting bluetooth: bluetoothd.
Starting dockerd:       [    6.922993] Bluetooth: Core ver 2.22
[    6.923057] NET: Registered protocol family 31
[    6.923070] Bluetooth: HCI device and connection manager initialized
[    6.923090] Bluetooth: HCI socket layer initialized
[    6.923113] Bluetooth: L2CAP socket layer initialized
[    6.923137] Bluetooth: SCO socket layer initialized


Starting ntpd: done
Starting syslogd/klogd: done
Starting internet superserver: xinetd.
 * Starting Avahi mDNS/DNS-SD Daemon: avahi-daemon                       [ ok ]
Starting Telephony daemon
Starting /usr/sbin/xenstored...
Setting domain 0 name, domid and JSON config...
Done setting up Dom0
Starting xenconsoled...
Starting QEMU as disk backend for dom0
/etc/rc5.d/S80xencommons: line 73: /usr/bin/qemu-system-i386: No such file or directory
Starting domain watchdog daemon: xenwatchdogd startup

[done]

Poky (Yocto Project Reference Distro) 2.7.1 kontron-sal28 hvc0

) pITS  cmd 0x0c: 000000170000000c 0000000000000013 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0a: 000000170000000a 0000201400000014 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0c: 000000170000000c 0000000000000014 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0a: 000000170000000a 0000201500000015 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0c: 000000170000000c 0000000000000015 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0a: 000000170000000a 0000201600000016 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0c: 000000170000000c 0000000000000016 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0a: 000000170000000a 0000201700000017 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0c: 000000170000000c 0000000000000017 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0a: 000000170000000a 0000201800000018 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0c: 000000170000000c 0000000000000018 0000000000000000 0000000000000000
000000000
(XEN) pITS  cmd 0x0c: 000000170000000c 0000000000000019 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0a: 000000170000000a 0000201a0000001a 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0c: 000000170000000c 000000000000001a 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0a: 000000170000000a 0000201b0000001b 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0c: 000000170000000c 000000000000001b 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0a: 000000170000000a 0000201c0000001c 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0c: 000000170000000c 000000000000001c 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0a: 000000170000000a 0000201d0000001d 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0c: 000000170000000c 000000000000001d 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0a: 000000170000000a 0000201e0000001e 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0c: 000000170000000c 000000000000001e 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0a: 000000170000000a 0000201f0000001f 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0c: 000000170000000c 000000000000001f 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x05: 0000000000000005 0000000000000000 0000000000000000 0000000000000000
(XEN) vgic-v3-its.c:902:d0v0 vITS  cmd 0x0a: 000000170000000a 0000200000000000 0000000000000000 0000000000000000
(XEN) gicv3_lpi_update_host_entry: host_lpi 8192 domain 0 virt_lpi 8192
(XEN) vgic-v3-its.c:902:d0v0 vITS  cmd 0x05: 0000000000000005 0000000000000000 0000000000000000 0000000000000000
(XEN) vgic-v3-its.c:902:d0v0 vITS  cmd 0x0a: 000000170000000a 0000200100000001 0000000000000000 0000000000000000
(XEN) gicv3_lpi_update_host_entry: host_lpi 8193 domain 0 virt_lpi 8193
(XEN) vgic-v3-its.c:902:d0v0 vITS  cmd 0x05: 0000000000000005 0000000000000000 0000000000000000 0000000000000000
(XEN) vgic-v3-its.c:902:d0v0 vITS  cmd 0x08: 0000001800000008 0000000000000000 80000000db471a00 0000000000000000
(XEN) pITS  cmd 0x08: 0000001800000008 0000000000000004 80000020f9de2700 0000000000000000
(XEN) pITS  cmd 0x0a: 000000180000000a 0000202000000000 0000000000000000 0000000000000000
000000000
(XEN) pITS  cmd 0x0a: 000000180000000a 0000202100000001 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0c: 000000180000000c 0000000000000001 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0a: 000000180000000a 0000202200000002 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0c: 000000180000000c 0000000000000002 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0a: 000000180000000a 0000202300000003 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0c: 000000180000000c 0000000000000003 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0a: 000000180000000a 0000202400000004 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0c: 000000180000000c 0000000000000004 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0a: 000000180000000a 0000202500000005 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0c: 000000180000000c 0000000000000005 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0a: 000000180000000a 0000202600000006 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0c: 000000180000000c 0000000000000006 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0a: 000000180000000a 0000202700000007 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0c: 000000180000000c 0000000000000007 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0a: 000000180000000a 0000202800000008 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0c: 000000180000000c 0000000000000008 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0a: 000000180000000a 0000202900000009 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0c: 000000180000000c 0000000000000009 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0a: 000000180000000a 0000202a0000000a 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0c: 000000180000000c 000000000000000a 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0a: 000000180000000a 0000202b0000000b 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0c: 000000180000000c 000000000000000b 0000000000000000 0000000000000000
000000000
(XEN) pITS  cmd 0x0c: 000000180000000c 000000000000000c 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0a: 000000180000000a 0000202d0000000d 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0c: 000000180000000c 000000000000000d 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0a: 000000180000000a 0000202e0000000e 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0c: 000000180000000c 000000000000000e 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0a: 000000180000000a 0000202f0000000f 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0c: 000000180000000c 000000000000000f 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0a: 000000180000000a 0000203000000010 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0c: 000000180000000c 0000000000000010 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0a: 000000180000000a 0000203100000011 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0c: 000000180000000c 0000000000000011 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0a: 000000180000000a 0000203200000012 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0c: 000000180000000c 0000000000000012 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0a: 000000180000000a 0000203300000013 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0c: 000000180000000c 0000000000000013 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0a: 000000180000000a 0000203400000014 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0c: 000000180000000c 0000000000000014 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0a: 000000180000000a 0000203500000015 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0c: 000000180000000c 0000000000000015 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0a: 000000180000000a 0000203600000016 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0c: 000000180000000c 0000000000000016 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0a: 000000180000000a 0000203700000017 0000000000000000 0000000000000000
000000000
(XEN) pITS  cmd 0x0a: 000000180000000a 0000203800000018 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0c: 000000180000000c 0000000000000018 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0a: 000000180000000a 0000203900000019 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0c: 000000180000000c 0000000000000019 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0a: 000000180000000a 0000203a0000001a 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0c: 000000180000000c 000000000000001a 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0a: 000000180000000a 0000203b0000001b 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0c: 000000180000000c 000000000000001b 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0a: 000000180000000a 0000203c0000001c 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0c: 000000180000000c 000000000000001c 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0a: 000000180000000a 0000203d0000001d 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0c: 000000180000000c 000000000000001d 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0a: 000000180000000a 0000203e0000001e 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0c: 000000180000000c 000000000000001e 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0a: 000000180000000a 0000203f0000001f 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0c: 000000180000000c 000000000000001f 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x05: 0000000000000005 0000000000000000 0000000000000000 0000000000000000
(XEN) vgic-v3-its.c:902:d0v0 vITS  cmd 0x0a: 000000180000000a 0000200200000000 0000000000000000 0000000000000000
(XEN) gicv3_lpi_update_host_entry: host_lpi 8224 domain 0 virt_lpi 8194
(XEN) vgic-v3-its.c:902:d0v0 vITS  cmd 0x05: 0000000000000005 0000000000000000 0000000000000000 0000000000000000
(XEN) vgic-v3-its.c:902:d0v0 vITS  cmd 0x0a: 000000180000000a 0000200300000001 0000000000000000 0000000000000000
(XEN) gicv3_lpi_update_host_entry: host_lpi 8225 domain 0 virt_lpi 8195
(XEN) vgic-v3-its.c:902:d0v0 vITS  cmd 0x05: 0000000000000005 0000000000000000 0000000000000000 0000000000000000
0000000dbb69200 0000000000000000
(XEN) pITS  cmd 0x08: 0000001b00000008 0000000000000004 80000020f9de1100 0000000000000000
(XEN) pITS  cmd 0x0a: 0000001b0000000a 0000204000000000 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0c: 0000001b0000000c 0000000000000000 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0a: 0000001b0000000a 0000204100000001 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0c: 0000001b0000000c 0000000000000001 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0a: 0000001b0000000a 0000204200000002 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0c: 0000001b0000000c 0000000000000002 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0a: 0000001b0000000a 0000204300000003 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0c: 0000001b0000000c 0000000000000003 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0a: 0000001b0000000a 0000204400000004 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0c: 0000001b0000000c 0000000000000004 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0a: 0000001b0000000a 0000204500000005 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0c: 0000001b0000000c 0000000000000005 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0a: 0000001b0000000a 0000204600000006 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0c: 0000001b0000000c 0000000000000006 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0a: 0000001b0000000a 0000204700000007 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0c: 0000001b0000000c 0000000000000007 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0a: 0000001b0000000a 0000204800000008 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0c: 0000001b0000000c 0000000000000008 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0a: 0000001b0000000a 0000204900000009 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0c: 0000001b0000000c 0000000000000009 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0a: 0000001b0000000a 0000204a0000000a 0000000000000000 0000000000000000
000000000
(XEN) pITS  cmd 0x0a: 0000001b0000000a 0000204b0000000b 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0c: 0000001b0000000c 000000000000000b 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0a: 0000001b0000000a 0000204c0000000c 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0c: 0000001b0000000c 000000000000000c 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0a: 0000001b0000000a 0000204d0000000d 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0c: 0000001b0000000c 000000000000000d 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0a: 0000001b0000000a 0000204e0000000e 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0c: 0000001b0000000c 000000000000000e 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0a: 0000001b0000000a 0000204f0000000f 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0c: 0000001b0000000c 000000000000000f 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0a: 0000001b0000000a 0000205000000010 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0c: 0000001b0000000c 0000000000000010 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0a: 0000001b0000000a 0000205100000011 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0c: 0000001b0000000c 0000000000000011 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0a: 0000001b0000000a 0000205200000012 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0c: 0000001b0000000c 0000000000000012 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0a: 0000001b0000000a 0000205300000013 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0c: 0000001b0000000c 0000000000000013 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0a: 0000001b0000000a 0000205400000014 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0c: 0000001b0000000c 0000000000000014 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0a: 0000001b0000000a 0000205500000015 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0c: 0000001b0000000c 0000000000000015 0000000000000000 0000000000000000
000000000
(XEN) pITS  cmd 0x0c: 0000001b0000000c 0000000000000016 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0a: 0000001b0000000a 0000205700000017 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0c: 0000001b0000000c 0000000000000017 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0a: 0000001b0000000a 0000205800000018 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0c: 0000001b0000000c 0000000000000018 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0a: 0000001b0000000a 0000205900000019 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0c: 0000001b0000000c 0000000000000019 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0a: 0000001b0000000a 0000205a0000001a 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0c: 0000001b0000000c 000000000000001a 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0a: 0000001b0000000a 0000205b0000001b 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0c: 0000001b0000000c 000000000000001b 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0a: 0000001b0000000a 0000205c0000001c 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0c: 0000001b0000000c 000000000000001c 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0a: 0000001b0000000a 0000205d0000001d 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0c: 0000001b0000000c 000000000000001d 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0a: 0000001b0000000a 0000205e0000001e 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0c: 0000001b0000000c 000000000000001e 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0a: 0000001b0000000a 0000205f0000001f 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x0c: 0000001b0000000c 000000000000001f 0000000000000000 0000000000000000
(XEN) pITS  cmd 0x05: 0000000000000005 0000000000000000 0000000000000000 0000000000000000
(XEN) vgic-v3-its.c:902:d0v0 vITS  cmd 0x0a: 0000001b0000000a 0000200400000000 0000000000000000 0000000000000000
(XEN) gicv3_lpi_update_host_entry: host_lpi 8256 domain 0 virt_lpi 8196
(XEN) vgic-v3-its.c:902:d0v0 vITS  cmd 0x05: 0000000000000005 0000000000000000 0000000000000000 0000000000000000
(XEN) vgic-v3-its.c:902:d0v0 vITS  cmd 0x0c: 0000001b0000000c 0000000000000000 0000000000000000 0000000000000000
(XEN) vgic-v3-its.c:902:d0v0 vITS  cmd 0x05: 0000000000000005 0000000000000000 0000000000000000 0000000000000000
(XEN) vgic-v3-its.c:902:d0v0 vITS  cmd 0x0c: 000000170000000c 0000000000000001 0000000000000000 0000000000000000
(XEN) vgic-v3-its.c:902:d0v0 vITS  cmd 0x05: 0000000000000005 0000000000000000 0000000000000000 0000000000000000
Re: AW: AW: AW: AW: AW: Xen data from meta-virtualization layer
Posted by Julien Grall 3 years, 5 months ago 

On 22/11/2020 22:55, Leo Krueger wrote:
> Hi Julien,

Hi Leo,

> 
> finally I could try out what you suggested, please find my answers inline.

Thank you for sending the logs!

> 
>> -----Ursprüngliche Nachricht-----
>> Von: Julien Grall <julien@xen.org>
>> Gesendet: Mittwoch, 18. November 2020 13:24
>> An: Stefano Stabellini <stefano.stabellini@xilinx.com>; Leo Krueger
>> <leo.krueger@zal.aero>
>> Cc: Peng Fan <peng.fan@nxp.com>; brucea@xilinx.com; Cornelia Bruelhart
>> <cornelia.bruelhart@zal.aero>; oleksandr_andrushchenko@epam.com; xen-
>> devel@lists.xenproject.org; Bertrand.Marquis@arm.com
>> Betreff: Re: AW: AW: AW: AW: Xen data from meta-virtualization layer
>>
>> Hi,
>>
>> On 17/11/2020 23:53, Stefano Stabellini wrote:
>>> Adding Bertrand, Oleksandr, Julien, and others -- they have a more
>>> recent experience with GICv3 ITS than me and might be able to help.
>>> I am attaching the device tree Leo sent a few days ago for reference.
>>>
>>>
>>> Typically when you can set the ethernet link up and no packets are
>>> exchanged it is because of a missing interrupt. In this case a missing
>>> MSI.
>>>
>>> Bertrand, I believe you tried the GIC ITS driver with PCI devices
>>> recently. It is expected to work correctly with MSIs in Dom0, right?
>>
>> OSSTest has some hardware (e.g. Thunder-X) where ITS is required to boot
>> Dom0. I haven't seen any failure on recent Xen. We are testing 4.11 and
>> onwards on Thunder-X.
>>
>> However, it may be possible that some more work is necessary for other
>> hardware (e.g. workaround, missing code...). See more below.
>>
>>>
>>>
>>>
>>> On Tue, 17 Nov 2020, Leo Krueger wrote:
>>>> Hi,
>>>>
>>>> I enabled CONFIG_HAS_ITS (what a stupid mistake by me to not set it
>>>> before...) but then had to add the following node to my device tree
>>>>
>>>> 	gic_lpi_base: syscon@0x80000000 {
>>>> 		compatible = "gic-lpi-base";
>>
>> I couldn't find this compatible defined/used in Linux 5.10-rc4. @Leo, could
>> you clarify which flavor/version of Linux you are using?
> 
> It is Linux 4.19 from Yocto (Warror release). XEN 4.13.2.

Do you have a link to the Linux tree? Is there any additional patches on 
top of vanilla?

> While searching around the Internet for any solution, I came across [0] which contained the gic-lpi-base node.
> So I just tried adding it (quite desperate I know) and voila, it at least brought me one step further (XEN exposing the ITS)...

I am slightly confused to how this would help. Xen and, AFAICT, Linux 
don't understand gic-lpi-base. Do you have modification in your Linux to 
use it?

Looking at the DT changes in [0], it looks like the node is not a child 
of gic@. So I think Xen will map the region to Dom0.

There are two things that I can notice:
   1) This region is RAM, but I can't find any reserve node. Is there 
any specific code in Linux to reserve it?
   2) The implementation in U-boot seems to suggest that the firmware 
will configure the LPIs and then enable it. If that's the case, then Xen 
needs to re-use the table in the DT rather than allocating a new one. 
However, I would have expected an error message in the log:

    "GICv3: CPUx: Cannot initialize LPIs"

At least Xen should not expose gic-lpi-base to the kernel, but I will 
wait on more details about the Linux kernel used before commenting more.

I would also be interested to know more details about the failure when 
gic-lpi-base is not added in your DT. In particular, I am interested to 
understand why Xen would not expose the ITS as we don't parse that node.

[...]

> For XEN 4.13.2 I had to adapt your patch slightly [1], see below (yes I know, quite ugly in parts).

No worries, debug patches are not meant to be nice to read ;).

> Find attached the boot log and an output of "xl dmesg" which is truncated due to the large amount of messages.
> 
> When enabling the network interface (gbe0), the following output is visible:
> 
> root@kontron-sal28:~# ip link set up dev gbe0
> (XEN) vgic-v3-its.c:902:d0v0 vITS  cmd 0x0c: 000000170000000c 0000000000000001 0000000000000000 0000000000000000
> (XEN) vgic-v3-its.c:902:d0v0 vITS  cmd 0x05: 0000000000000005 0000000000000000 0000000000000000 0000000000000000

0xc is INV and 0x5 is SYNC. Most likely the driver unmask the interrupt 
by writing in the property table (access are not trapped to Xen) and 
then requested to invalidate the cache state.

> [   34.034598] Atheros 8031 ethernet 0000:00:00.3:05: attached PHY driver [Atheros 8031 ethernet] (mii_bus:phy_addr=0000:00:00.3:05, irq=POLL)
> [   34.041111] 8021q: adding VLAN 0 to HW filter on device gbe0
> [   34.041209] IPv6: ADDRCONF(NETDEV_UP): gbe0: link is not ready
> root@kontron-sal28:~# [   35.041951] fsl_enetc 0000:00:00.0 gbe0: Link is Down
> [   38.114426] fsl_enetc 0000:00:00.0 gbe0: Link is Up - 1Gbps/Full - flow control off
> [   38.114508] IPv6: ADDRCONF(NETDEV_CHANGE): gbe0: link becomes ready
> 
> Does that tell you anything?

It is at least a good sign because it means Linux is able to 
initialize/talk to the vITS.

I would lean towards one (or multiple) issue with pITS and/or the 
device-tree exposed to Linux. I am not entirely what exactly... I think 
having more details about the Linux setup would be helpful.

I will reply on Rahul's e-mail separately.

Cheers,

-- 
Julien Grall
AW: AW: AW: AW: AW: AW: Xen data from meta-virtualization layer
Posted by Leo Krueger 3 years, 5 months ago 
Hi,

> -----Ursprüngliche Nachricht-----
> Von: Julien Grall <julien@xen.org>
> Gesendet: Montag, 23. November 2020 19:27
> An: Leo Krueger <leo.krueger@zal.aero>; Stefano Stabellini
> <stefano.stabellini@xilinx.com>
> Cc: Peng Fan <peng.fan@nxp.com>; brucea@xilinx.com; Cornelia Bruelhart
> <cornelia.bruelhart@zal.aero>; oleksandr_andrushchenko@epam.com; xen-
> devel@lists.xenproject.org; Bertrand.Marquis@arm.com
> Betreff: Re: AW: AW: AW: AW: AW: Xen data from meta-virtualization layer
> 
> 
> 
> On 22/11/2020 22:55, Leo Krueger wrote:
> > Hi Julien,
> 
> Hi Leo,
> 
> >
> > finally I could try out what you suggested, please find my answers inline.
> 
> Thank you for sending the logs!
> 
> >
> >> -----Ursprüngliche Nachricht-----
> >> Von: Julien Grall <julien@xen.org>
> >> Gesendet: Mittwoch, 18. November 2020 13:24
> >> An: Stefano Stabellini <stefano.stabellini@xilinx.com>; Leo Krueger
> >> <leo.krueger@zal.aero>
> >> Cc: Peng Fan <peng.fan@nxp.com>; brucea@xilinx.com; Cornelia
> >> Bruelhart <cornelia.bruelhart@zal.aero>;
> >> oleksandr_andrushchenko@epam.com; xen- devel@lists.xenproject.org;
> >> Bertrand.Marquis@arm.com
> >> Betreff: Re: AW: AW: AW: AW: Xen data from meta-virtualization layer
> >>
> >> Hi,
> >>
> >> On 17/11/2020 23:53, Stefano Stabellini wrote:
> >>> Adding Bertrand, Oleksandr, Julien, and others -- they have a more
> >>> recent experience with GICv3 ITS than me and might be able to help.
> >>> I am attaching the device tree Leo sent a few days ago for reference.
> >>>
> >>>
> >>> Typically when you can set the ethernet link up and no packets are
> >>> exchanged it is because of a missing interrupt. In this case a
> >>> missing MSI.
> >>>
> >>> Bertrand, I believe you tried the GIC ITS driver with PCI devices
> >>> recently. It is expected to work correctly with MSIs in Dom0, right?
> >>
> >> OSSTest has some hardware (e.g. Thunder-X) where ITS is required to
> >> boot Dom0. I haven't seen any failure on recent Xen. We are testing
> >> 4.11 and onwards on Thunder-X.
> >>
> >> However, it may be possible that some more work is necessary for
> >> other hardware (e.g. workaround, missing code...). See more below.
> >>
> >>>
> >>>
> >>>
> >>> On Tue, 17 Nov 2020, Leo Krueger wrote:
> >>>> Hi,
> >>>>
> >>>> I enabled CONFIG_HAS_ITS (what a stupid mistake by me to not set it
> >>>> before...) but then had to add the following node to my device tree
> >>>>
> >>>> 	gic_lpi_base: syscon@0x80000000 {
> >>>> 		compatible = "gic-lpi-base";
> >>
> >> I couldn't find this compatible defined/used in Linux 5.10-rc4. @Leo,
> >> could you clarify which flavor/version of Linux you are using?
> >
> > It is Linux 4.19 from Yocto (Warror release). XEN 4.13.2.
> 
> Do you have a link to the Linux tree? Is there any additional patches on top of
> vanilla?

Linux tree is found here: https://github.com/kontron/linux-smarc-sal28/commits/master-LSDK-19.09
(up to the latest commit in that branch)

> 
> > While searching around the Internet for any solution, I came across [0]
> which contained the gic-lpi-base node.
> > So I just tried adding it (quite desperate I know) and voila, it at least
> brought me one step further (XEN exposing the ITS)...
> 
> I am slightly confused to how this would help. Xen and, AFAICT, Linux don't
> understand gic-lpi-base. Do you have modification in your Linux to use it?

I have no idea, to be honest. Maybe it is about the memory being reserved due to that node or something like that?

> 
> Looking at the DT changes in [0], it looks like the node is not a child of gic@.
> So I think Xen will map the region to Dom0.
> 
> There are two things that I can notice:
>    1) This region is RAM, but I can't find any reserve node. Is there any specific
> code in Linux to reserve it?
>    2) The implementation in U-boot seems to suggest that the firmware will
> configure the LPIs and then enable it. If that's the case, then Xen needs to
> re-use the table in the DT rather than allocating a new one.
> However, I would have expected an error message in the log:
> 
>     "GICv3: CPUx: Cannot initialize LPIs"
> 
> At least Xen should not expose gic-lpi-base to the kernel, but I will wait on
> more details about the Linux kernel used before commenting more.
> 
> I would also be interested to know more details about the failure when gic-
> lpi-base is not added in your DT. In particular, I am interested to understand
> why Xen would not expose the ITS as we don't parse that node.

How can I supply you with more information in regard to that? Without that node, ITS was not exposed at all.

> 
> [...]
> 
> > For XEN 4.13.2 I had to adapt your patch slightly [1], see below (yes I know,
> quite ugly in parts).
> 
> No worries, debug patches are not meant to be nice to read ;).
> 
> > Find attached the boot log and an output of "xl dmesg" which is truncated
> due to the large amount of messages.
> >
> > When enabling the network interface (gbe0), the following output is
> visible:
> >
> > root@kontron-sal28:~# ip link set up dev gbe0
> > (XEN) vgic-v3-its.c:902:d0v0 vITS  cmd 0x0c: 000000170000000c
> > 0000000000000001 0000000000000000 0000000000000000
> > (XEN) vgic-v3-its.c:902:d0v0 vITS  cmd 0x05: 0000000000000005
> > 0000000000000000 0000000000000000 0000000000000000
> 
> 0xc is INV and 0x5 is SYNC. Most likely the driver unmask the interrupt by
> writing in the property table (access are not trapped to Xen) and then
> requested to invalidate the cache state.
> 
> > [   34.034598] Atheros 8031 ethernet 0000:00:00.3:05: attached PHY driver
> [Atheros 8031 ethernet] (mii_bus:phy_addr=0000:00:00.3:05, irq=POLL)
> > [   34.041111] 8021q: adding VLAN 0 to HW filter on device gbe0
> > [   34.041209] IPv6: ADDRCONF(NETDEV_UP): gbe0: link is not ready
> > root@kontron-sal28:~# [   35.041951] fsl_enetc 0000:00:00.0 gbe0: Link is
> Down
> > [   38.114426] fsl_enetc 0000:00:00.0 gbe0: Link is Up - 1Gbps/Full - flow
> control off
> > [   38.114508] IPv6: ADDRCONF(NETDEV_CHANGE): gbe0: link becomes
> ready
> >
> > Does that tell you anything?
> 
> It is at least a good sign because it means Linux is able to initialize/talk to the
> vITS.
> 
> I would lean towards one (or multiple) issue with pITS and/or the device-tree
> exposed to Linux. I am not entirely what exactly... I think having more details
> about the Linux setup would be helpful.

Ok let me know what you need from my side. I was considering the following things to try out next:

- a more recent u-boot version as this might fix problems with the msi-map (at least that is what I think, I am not an expert here)
- a different device tree (a more recent one, ...)
- ...

> 
> I will reply on Rahul's e-mail separately.
> 
> Cheers,

Best wishes!

> 
> --
> Julien Grall
Re: AW: AW: AW: AW: AW: AW: Xen data from meta-virtualization layer
Posted by Stefano Stabellini 3 years, 5 months ago 
+ Zhiqiang Hou

On Tue, 24 Nov 2020, Leo Krueger wrote:
> > >>> On Tue, 17 Nov 2020, Leo Krueger wrote:
> > >>>> Hi,
> > >>>>
> > >>>> I enabled CONFIG_HAS_ITS (what a stupid mistake by me to not set it
> > >>>> before...) but then had to add the following node to my device tree
> > >>>>
> > >>>> 	gic_lpi_base: syscon@0x80000000 {
> > >>>> 		compatible = "gic-lpi-base";
> > >>
> > >> I couldn't find this compatible defined/used in Linux 5.10-rc4. @Leo,
> > >> could you clarify which flavor/version of Linux you are using?
> > >
> > > It is Linux 4.19 from Yocto (Warror release). XEN 4.13.2.
> > 
> > Do you have a link to the Linux tree? Is there any additional patches on top of
> > vanilla?
> 
> Linux tree is found here: https://github.com/kontron/linux-smarc-sal28/commits/master-LSDK-19.09
> (up to the latest commit in that branch)

[...]

> > Looking at the DT changes in [0], it looks like the node is not a child of gic@.
> > So I think Xen will map the region to Dom0.
> > 
> > There are two things that I can notice:
> >    1) This region is RAM, but I can't find any reserve node. Is there any specific
> > code in Linux to reserve it?
> >    2) The implementation in U-boot seems to suggest that the firmware will
> > configure the LPIs and then enable it. If that's the case, then Xen needs to
> > re-use the table in the DT rather than allocating a new one.

That Linux tree has no mentions of gic-lpi-base. That means that
gic-lpi-base is only used in u-boot, not in Linux. In particular the
most relevant commit is af288cb291da3abef6be0875527729296f7de7a0. 

In regards to the reserved-memory regions, maybe we are not seeing them
because Leo posted the host device tree, not the one passed at runtime
from u-boot to Linux?

If so, Leo, could you please boot Linux on native (no Xen) and get the
device tree from there at runtime using dtc -I fs -O dts
/proc/device-tree ?


However, the name of the reserved-memory region created by u-boot seems
to be "lpi_rd_table". I cannot find any mentions of lpi_rd_table in the
Linux kernel tree either.

Zhiqiang, Leo is trying to boot Xen on sAL28. Linux booting on Xen
throws errors in regards to GIC/ITS initialization. On other hardware
Xen can use and virtualize GICv3 and ITS just fine. Could you please
explain what is different about sAL28 and how Xen/Linux is expected to
use the lpi_rd_table reserved-memory region?
Re: Xen data from meta-virtualization layer
Posted by Michael Walle 2 years, 2 months ago 
Hi all,

> + Zhiqiang Hou
> 
> On Tue, 24 Nov 2020, Leo Krueger wrote:
> > > >>> On Tue, 17 Nov 2020, Leo Krueger wrote:
> > > >>>> Hi,
> > > >>>>
> > > >>>> I enabled CONFIG_HAS_ITS (what a stupid mistake by me to not set it
> > > >>>> before...) but then had to add the following node to my device tree
> > > >>>>
> > > >>>> 	gic_lpi_base: syscon@0x80000000 {
> > > >>>> 		compatible = "gic-lpi-base";
> > > >>
> > > >> I couldn't find this compatible defined/used in Linux 5.10-rc4. @Leo,
> > > >> could you clarify which flavor/version of Linux you are using?
> > > >
> > > > It is Linux 4.19 from Yocto (Warror release). XEN 4.13.2.
> > > 
> > > Do you have a link to the Linux tree? Is there any additional patches on top of
> > > vanilla?
> > 
> > Linux tree is found here: https://github.com/kontron/linux-smarc-sal28/commits/master-LSDK-19.09
> > (up to the latest commit in that branch)

FWIW, I'm the developer of the support for this board both in the
mentioned branch as well as upstream.

If you don't need graphics you shouldn't really be using the branch
above, but the latest vanilla kernel release.

> [...]
> 
> > > Looking at the DT changes in [0], it looks like the node is not a child of gic@.
> > > So I think Xen will map the region to Dom0.
> > > 
> > > There are two things that I can notice:
> > >    1) This region is RAM, but I can't find any reserve node. Is there any specific
> > > code in Linux to reserve it?
> > >    2) The implementation in U-boot seems to suggest that the firmware will
> > > configure the LPIs and then enable it. If that's the case, then Xen needs to
> > > re-use the table in the DT rather than allocating a new one.
> 
> That Linux tree has no mentions of gic-lpi-base. That means that
> gic-lpi-base is only used in u-boot, not in Linux. In particular the
> most relevant commit is af288cb291da3abef6be0875527729296f7de7a0. 

That node was horrible hack from NXP for u-boot and was removed in
a84cea06bb8fff69810a890ac0e4b47ea5726512.

> In regards to the reserved-memory regions, maybe we are not seeing them
> because Leo posted the host device tree, not the one passed at runtime
> from u-boot to Linux?
> 
> If so, Leo, could you please boot Linux on native (no Xen) and get the
> device tree from there at runtime using dtc -I fs -O dts
> /proc/device-tree ?
> 
> 
> However, the name of the reserved-memory region created by u-boot seems
> to be "lpi_rd_table". I cannot find any mentions of lpi_rd_table in the
> Linux kernel tree either.
> 
> Zhiqiang, Leo is trying to boot Xen on sAL28. Linux booting on Xen
> throws errors in regards to GIC/ITS initialization. On other hardware
> Xen can use and virtualize GICv3 and ITS just fine. Could you please
> explain what is different about sAL28 and how Xen/Linux is expected to
> use the lpi_rd_table reserved-memory region?

I actually stumbled across this thread after trying out Xen myself. I'm
using lastest vanilla u-boot (with pending PSCI patches), vanilla kernel
and vanilla Xen.

So far I've discovered, that xen complains that it cannot route IRQ64 to
dom0. That is because on the LS1028A there is a dual UART (two 16550
with one shared interrupt) and xen takes the first UART and then tries
to map the interrupt of the second UART to linux. For now, I don't know
how this is solved correctly. As a quick hack, I removed the second uart
node from the device tree.

But what is more severe is that the iommu isn't set up correctly. I'm
getting the following faults:

(XEN) smmu: /soc/iommu@5000000: Unexpected global fault, this could be serious
(XEN) smmu: /soc/iommu@5000000: 	GFSR 0x80000002, GFSYNR0 0x00000000, GFSYNR1 0x0000042a, GFSYNR2 0x00000000

If I decode it correctly, the streamid should be 0x2a which would be one
of the PCI devices on the internal root complex. Probably the network
card.

This is the first developer experience with Xen, so please bear with me
:) It seems that Xen doesn't add the master to the IOMMU. To me it seems
that only devices with a 'iommus' dt property are added. But in case of
PCI devices the parent only has a iommu-map property.

And it makes me wonder why Leo has an almost working setup. Maybe I'm
missing some patches though.

-michael
Re: Xen data from meta-virtualization layer
Posted by Stefano Stabellini 2 years, 2 months ago 
On Fri, 4 Feb 2022, Michael Walle wrote:
> > In regards to the reserved-memory regions, maybe we are not seeing them
> > because Leo posted the host device tree, not the one passed at runtime
> > from u-boot to Linux?
> > 
> > If so, Leo, could you please boot Linux on native (no Xen) and get the
> > device tree from there at runtime using dtc -I fs -O dts
> > /proc/device-tree ?
> > 
> > 
> > However, the name of the reserved-memory region created by u-boot seems
> > to be "lpi_rd_table". I cannot find any mentions of lpi_rd_table in the
> > Linux kernel tree either.
> > 
> > Zhiqiang, Leo is trying to boot Xen on sAL28. Linux booting on Xen
> > throws errors in regards to GIC/ITS initialization. On other hardware
> > Xen can use and virtualize GICv3 and ITS just fine. Could you please
> > explain what is different about sAL28 and how Xen/Linux is expected to
> > use the lpi_rd_table reserved-memory region?
> 
> I actually stumbled across this thread after trying out Xen myself. I'm
> using lastest vanilla u-boot (with pending PSCI patches), vanilla kernel
> and vanilla Xen.
> 
> So far I've discovered, that xen complains that it cannot route IRQ64 to
> dom0. That is because on the LS1028A there is a dual UART (two 16550
> with one shared interrupt) and xen takes the first UART and then tries
> to map the interrupt of the second UART to linux. For now, I don't know
> how this is solved correctly. As a quick hack, I removed the second uart
> node from the device tree.

This is an interesting problem. Removing the second UART is a good
workaround for now as there is no obvious solution I think.


> But what is more severe is that the iommu isn't set up correctly. I'm
> getting the following faults:
> 
> (XEN) smmu: /soc/iommu@5000000: Unexpected global fault, this could be serious
> (XEN) smmu: /soc/iommu@5000000: 	GFSR 0x80000002, GFSYNR0 0x00000000, GFSYNR1 0x0000042a, GFSYNR2 0x00000000
> 
> If I decode it correctly, the streamid should be 0x2a which would be one
> of the PCI devices on the internal root complex. Probably the network
> card.

Yes there is DMA transaction with an "unknown" StreamID. I think the
StreamID is 0x42a. It means that there is a DMA master on the board with
StreamID 0x42a that is either:

- not described in device tree
- described in device tree with a different StreamID
- the right StreamID is described device tree, but it is not picked up
  by Xen


> This is the first developer experience with Xen, so please bear with me
> :) It seems that Xen doesn't add the master to the IOMMU. To me it seems
> that only devices with a 'iommus' dt property are added. But in case of
> PCI devices the parent only has a iommu-map property.
> 
> And it makes me wonder why Leo has an almost working setup. Maybe I'm
> missing some patches though.

Xen 4.16 is able to parse StreamID in the "iommus" property and also
"mmu-masters" property. But It is not able to parse the "iommu-map"
property yet. So if 0x42a is described in device tree using "iommu-map"
then the error makes sense.

A simple solution is to replace iommu-map with iommus in device tree.
It is possible that someone in CC to this email might already have a
patch to introduce parsing of iommu-map in Xen.
Re: Xen data from meta-virtualization layer
Posted by Michael Walle 2 years, 2 months ago 
Hi Stefano,

Am 2022-02-04 22:11, schrieb Stefano Stabellini:
> On Fri, 4 Feb 2022, Michael Walle wrote:
>> > In regards to the reserved-memory regions, maybe we are not seeing them
>> > because Leo posted the host device tree, not the one passed at runtime
>> > from u-boot to Linux?
>> >
>> > If so, Leo, could you please boot Linux on native (no Xen) and get the
>> > device tree from there at runtime using dtc -I fs -O dts
>> > /proc/device-tree ?
>> >
>> >
>> > However, the name of the reserved-memory region created by u-boot seems
>> > to be "lpi_rd_table". I cannot find any mentions of lpi_rd_table in the
>> > Linux kernel tree either.
>> >
>> > Zhiqiang, Leo is trying to boot Xen on sAL28. Linux booting on Xen
>> > throws errors in regards to GIC/ITS initialization. On other hardware
>> > Xen can use and virtualize GICv3 and ITS just fine. Could you please
>> > explain what is different about sAL28 and how Xen/Linux is expected to
>> > use the lpi_rd_table reserved-memory region?
>> 
>> I actually stumbled across this thread after trying out Xen myself. 
>> I'm
>> using lastest vanilla u-boot (with pending PSCI patches), vanilla 
>> kernel
>> and vanilla Xen.
>> 
>> So far I've discovered, that xen complains that it cannot route IRQ64 
>> to
>> dom0. That is because on the LS1028A there is a dual UART (two 16550
>> with one shared interrupt) and xen takes the first UART and then tries
>> to map the interrupt of the second UART to linux. For now, I don't 
>> know
>> how this is solved correctly. As a quick hack, I removed the second 
>> uart
>> node from the device tree.
> 
> This is an interesting problem. Removing the second UART is a good
> workaround for now as there is no obvious solution I think.

But not a very user friendly one, though. I guess the first UART
is disabled/removed by Xen? I haven't looked at how it is handled.

Can't we search for other uarts with the same interrupt and disable
these, too? Maybe conditionally by the SoC compatible?

>> But what is more severe is that the iommu isn't set up correctly. I'm
>> getting the following faults:
>> 
>> (XEN) smmu: /soc/iommu@5000000: Unexpected global fault, this could be 
>> serious
>> (XEN) smmu: /soc/iommu@5000000: 	GFSR 0x80000002, GFSYNR0 0x00000000, 
>> GFSYNR1 0x0000042a, GFSYNR2 0x00000000
>> 
>> If I decode it correctly, the streamid should be 0x2a which would be 
>> one
>> of the PCI devices on the internal root complex. Probably the network
>> card.
> 
> Yes there is DMA transaction with an "unknown" StreamID. I think the
> StreamID is 0x42a. It means that there is a DMA master on the board 
> with
> StreamID 0x42a that is either:
> 
> - not described in device tree
> - described in device tree with a different StreamID
> - the right StreamID is described device tree, but it is not picked up
>   by Xen

See below.

>> This is the first developer experience with Xen, so please bear with 
>> me
>> :) It seems that Xen doesn't add the master to the IOMMU. To me it 
>> seems
>> that only devices with a 'iommus' dt property are added. But in case 
>> of
>> PCI devices the parent only has a iommu-map property.
>> 
>> And it makes me wonder why Leo has an almost working setup. Maybe I'm
>> missing some patches though.
> 
> Xen 4.16 is able to parse StreamID in the "iommus" property and also
> "mmu-masters" property. But It is not able to parse the "iommu-map"
> property yet. So if 0x42a is described in device tree using "iommu-map"
> then the error makes sense.
> 
> A simple solution is to replace iommu-map with iommus in device tree.

I'm not sure this is so easy, because they are dynamically assigned
by the bootloader. Sure for now I could do that I guess, but iommu=0
works as well ;)

I now got Xen and Linux booting and see the same problems with the
GIC ITS, that is that the enetc interrupts aren't delivered to the
dom0 linux. I've also applied the patch in this thread and I'm
seeing the same as Leo. Full boot log is here [1].

I noticed the following.
[    0.168544] pci 0000:00:00.0: Failed to add - passthrough or 
MSI/MSI-X might fail!

Not sure if it should work nonetheless.

> It is possible that someone in CC to this email might already have a
> patch to introduce parsing of iommu-map in Xen.

I guess they've used the old mmu-masters property.

Btw. I don't know if it matters, but the SMARC-sAL28 normally doesn't
use TF-A and runs without it. Nonetheless, I've booted the board with
the bl31 from NXP and it doesn't help either. There is still a
difference between the NXP bootflow which uses bl1/bl2/bl31/u-boot
and this board which uses u-boot-spl/u-boot or u-boot-spl/bl31/u-boot.

I just found GIC setup code in the bl31. I'll also have a look there.

-michael

[1] https://pastebin.com/raw/XMjE3BvG
Re: Xen data from meta-virtualization layer
Posted by Julien Grall 2 years, 2 months ago 
Hi Michael,

On Fri, 4 Feb 2022 at 22:42, Michael Walle <michael@walle.cc> wrote:
> Am 2022-02-04 22:11, schrieb Stefano Stabellini:
> > On Fri, 4 Feb 2022, Michael Walle wrote:
> >> > In regards to the reserved-memory regions, maybe we are not seeing them
> >> > because Leo posted the host device tree, not the one passed at runtime
> >> > from u-boot to Linux?
> >> >
> >> > If so, Leo, could you please boot Linux on native (no Xen) and get the
> >> > device tree from there at runtime using dtc -I fs -O dts
> >> > /proc/device-tree ?
> >> >
> >> >
> >> > However, the name of the reserved-memory region created by u-boot seems
> >> > to be "lpi_rd_table". I cannot find any mentions of lpi_rd_table in the
> >> > Linux kernel tree either.
> >> >
> >> > Zhiqiang, Leo is trying to boot Xen on sAL28. Linux booting on Xen
> >> > throws errors in regards to GIC/ITS initialization. On other hardware
> >> > Xen can use and virtualize GICv3 and ITS just fine. Could you please
> >> > explain what is different about sAL28 and how Xen/Linux is expected to
> >> > use the lpi_rd_table reserved-memory region?
> >>
> >> I actually stumbled across this thread after trying out Xen myself.
> >> I'm
> >> using lastest vanilla u-boot (with pending PSCI patches), vanilla
> >> kernel
> >> and vanilla Xen.
> >>
> >> So far I've discovered, that xen complains that it cannot route IRQ64
> >> to
> >> dom0. That is because on the LS1028A there is a dual UART (two 16550
> >> with one shared interrupt) and xen takes the first UART and then tries
> >> to map the interrupt of the second UART to linux. For now, I don't
> >> know
> >> how this is solved correctly. As a quick hack, I removed the second
> >> uart
> >> node from the device tree.
> >
> > This is an interesting problem. Removing the second UART is a good
> > workaround for now as there is no obvious solution I think.
>
> But not a very user friendly one, though. I guess the first UART
> is disabled/removed by Xen? I haven't looked at how it is handled.
>
> Can't we search for other uarts with the same interrupt and disable
> these, too? Maybe conditionally by the SoC compatible?

The problem sounds quite similar to the one we had on sunxi. Although
the UART was on the same page rather than sharing interrupts.

Xen has per-platform hook to prevent a device been assigned
to dom0. For an example, you could look at:

https://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=xen/arch/arm/platforms/sunxi.c

> >> This is the first developer experience with Xen, so please bear with
> >> me
> >> :) It seems that Xen doesn't add the master to the IOMMU. To me it
> >> seems
> >> that only devices with a 'iommus' dt property are added. But in case
> >> of
> >> PCI devices the parent only has a iommu-map property.
> >>
> >> And it makes me wonder why Leo has an almost working setup. Maybe I'm
> >> missing some patches though.
> >
> > Xen 4.16 is able to parse StreamID in the "iommus" property and also
> > "mmu-masters" property. But It is not able to parse the "iommu-map"
> > property yet. So if 0x42a is described in device tree using "iommu-map"
> > then the error makes sense.
> >
> > A simple solution is to replace iommu-map with iommus in device tree.
>
> I'm not sure this is so easy, because they are dynamically assigned
> by the bootloader. Sure for now I could do that I guess, but iommu=0
> works as well ;)
>
> I now got Xen and Linux booting and see the same problems with the
> GIC ITS, that is that the enetc interrupts aren't delivered to the
> dom0 linux. I've also applied the patch in this thread and I'm
> seeing the same as Leo. Full boot log is here [1].
>
> I noticed the following.
> [    0.168544] pci 0000:00:00.0: Failed to add - passthrough or
> MSI/MSI-X might fail!

This message is harmless. This is printed because Xen on Arm
doesn't hypercall the hypercall to add a PCI device. On Arm,
we don't need it yet (it might be necessary for PCI passthrough) and
MSI/MSI-X are handled/registered the same way as on bare-metal
(for your case through the ITS)

>
> Not sure if it should work nonetheless.

I looked through the log and couldn't spot anything obvious. However,
skimming through Linux, I noticed there are some changes in the
ITS for freescale (now NXP) such as:

drivers/irqchip/irq-gic-v3-its-fsl-mc-msi.c

Is it something that might be used on the board you are using?

If the answer is yes, then my suggestion would be to look
how this is meant to interact with the ITS. It might be possible
that we are missing some pieces in Xen to properly support it.

>
> > It is possible that someone in CC to this email might already have a
> > patch to introduce parsing of iommu-map in Xen.

Pass. I don't have any and couldn't find any submission on the ML.


>
> I guess they've used the old mmu-masters property.
>
> Btw. I don't know if it matters, but the SMARC-sAL28 normally doesn't
> use TF-A and runs without it. Nonetheless, I've booted the board with
> the bl31 from NXP and it doesn't help either. There is still a
> difference between the NXP bootflow which uses bl1/bl2/bl31/u-boot
> and this board which uses u-boot-spl/u-boot or u-boot-spl/bl31/u-boot.
>
> I just found GIC setup code in the bl31. I'll also have a look there.
>
> -michael
>
> [1] https://pastebin.com/raw/XMjE3BvG
Re: Xen data from meta-virtualization layer
Posted by Michael Walle 2 years, 2 months ago 
Hi Julien,

Am 2022-02-05 00:29, schrieb Julien Grall:
[..]
>> But not a very user friendly one, though. I guess the first UART
>> is disabled/removed by Xen? I haven't looked at how it is handled.
>> 
>> Can't we search for other uarts with the same interrupt and disable
>> these, too? Maybe conditionally by the SoC compatible?
> 
> The problem sounds quite similar to the one we had on sunxi. Although
> the UART was on the same page rather than sharing interrupts.
> 
> Xen has per-platform hook to prevent a device been assigned
> to dom0. For an example, you could look at:
> 
> https://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=xen/arch/arm/platforms/sunxi.c

Nice. At least, this is working now ;) I'll send a patch in
the next few days.

I guess it's safe to assume that we can always remove both UARTs
on the LS1028A (probably on most layerscapes). The most common
use case is to use the first UART for Xen. You could run Xen
without console (?), then you'd miss the possibility to map
the DUART. Or there could be a new driver for the LPUART on the
LS1028A. In this case, the DUART wouldn't be used by Xen either.

But I think we should start simple and just remove the DUART
altogether via that hook.

-michael
Re: Xen data from meta-virtualization layer
Posted by Julien Grall 2 years, 2 months ago 

On 04/02/2022 23:29, Julien Grall wrote:
> This message is harmless. This is printed because Xen on Arm
> doesn't hypercall the hypercall to add a PCI device. On Arm,

I meant "doesn't need the hypercall...".

> we don't need it yet (it might be necessary for PCI passthrough) and
> MSI/MSI-X are handled/registered the same way as on bare-metal
> (for your case through the ITS)
> 
>>
>> Not sure if it should work nonetheless.
> 
> I looked through the log and couldn't spot anything obvious. However,
> skimming through Linux, I noticed there are some changes in the
> ITS for freescale (now NXP) such as:
> 
> drivers/irqchip/irq-gic-v3-its-fsl-mc-msi.c
> 
> Is it something that might be used on the board you are using?
> 
> If the answer is yes, then my suggestion would be to look
> how this is meant to interact with the ITS. It might be possible
> that we are missing some pieces in Xen to properly support it.

Another tree you might want to look at is 
https://source.codeaurora.org/external/imx/imx-xen. It contains a bunch 
of patches for NXP SoC that was never upstreamed.

> 
>>
>>> It is possible that someone in CC to this email might already have a
>>> patch to introduce parsing of iommu-map in Xen.
> 
> Pass. I don't have any and couldn't find any submission on the ML.
> 
> 
>>
>> I guess they've used the old mmu-masters property.
>>
>> Btw. I don't know if it matters, but the SMARC-sAL28 normally doesn't
>> use TF-A and runs without it. Nonetheless, I've booted the board with
>> the bl31 from NXP and it doesn't help either. There is still a
>> difference between the NXP bootflow which uses bl1/bl2/bl31/u-boot
>> and this board which uses u-boot-spl/u-boot or u-boot-spl/bl31/u-boot.
>>
>> I just found GIC setup code in the bl31. I'll also have a look there.
>>
>> -michael
>>
>> [1] https://pastebin.com/raw/XMjE3BvG

Cheers,

-- 
Julien Grall
Re: Xen data from meta-virtualization layer
Posted by Rahul Singh 3 years, 5 months ago 
Hello ,

> On 22 Nov 2020, at 10:55 pm, Leo Krueger <leo.krueger@zal.aero> wrote:
> 
> Hi Julien,
> 
> finally I could try out what you suggested, please find my answers inline.
> 
>> -----Ursprüngliche Nachricht-----
>> Von: Julien Grall <julien@xen.org>
>> Gesendet: Mittwoch, 18. November 2020 13:24
>> An: Stefano Stabellini <stefano.stabellini@xilinx.com>; Leo Krueger
>> <leo.krueger@zal.aero>
>> Cc: Peng Fan <peng.fan@nxp.com>; brucea@xilinx.com; Cornelia Bruelhart
>> <cornelia.bruelhart@zal.aero>; oleksandr_andrushchenko@epam.com; xen-
>> devel@lists.xenproject.org; Bertrand.Marquis@arm.com
>> Betreff: Re: AW: AW: AW: AW: Xen data from meta-virtualization layer
>> 
>> Hi,
>> 
>> On 17/11/2020 23:53, Stefano Stabellini wrote:
>>> Adding Bertrand, Oleksandr, Julien, and others -- they have a more
>>> recent experience with GICv3 ITS than me and might be able to help.
>>> I am attaching the device tree Leo sent a few days ago for reference.
>>> 
>>> 
>>> Typically when you can set the ethernet link up and no packets are
>>> exchanged it is because of a missing interrupt. In this case a missing
>>> MSI.
>>> 
>>> Bertrand, I believe you tried the GIC ITS driver with PCI devices
>>> recently. It is expected to work correctly with MSIs in Dom0, right?
>> 
>> OSSTest has some hardware (e.g. Thunder-X) where ITS is required to boot
>> Dom0. I haven't seen any failure on recent Xen. We are testing 4.11 and
>> onwards on Thunder-X.
>> 
>> However, it may be possible that some more work is necessary for other
>> hardware (e.g. workaround, missing code...). See more below.
>> 
>>> 
>>> 
>>> 
>>> On Tue, 17 Nov 2020, Leo Krueger wrote:
>>>> Hi,
>>>> 
>>>> I enabled CONFIG_HAS_ITS (what a stupid mistake by me to not set it
>>>> before...) but then had to add the following node to my device tree
>>>> 
>>>> 	gic_lpi_base: syscon@0x80000000 {
>>>> 		compatible = "gic-lpi-base";
>> 
>> I couldn't find this compatible defined/used in Linux 5.10-rc4. @Leo, could
>> you clarify which flavor/version of Linux you are using?
> 
> It is Linux 4.19 from Yocto (Warror release). XEN 4.13.2.
> While searching around the Internet for any solution, I came across [0] which contained the gic-lpi-base node.
> So I just tried adding it (quite desperate I know) and voila, it at least brought me one step further (XEN exposing the ITS)...
> 
>> 
>>>> 		reg = <0x0 0x80000000 0x0 0x100000>;
>>>> 		max-gic-redistributors = <2>;
>>>> 	};
>>>> 
>>>> to somehow change something in regard to the ITS and MSI/MSI-X
>>>> 
>>>> (XEN) GICv3 initialization:
>>>> (XEN)       gic_dist_addr=0x00000006000000
>>>> (XEN)       gic_maintenance_irq=25
>>>> (XEN)       gic_rdist_stride=0
>>>> (XEN)       gic_rdist_regions=1
>>>> (XEN)       redistributor regions:
>>>> (XEN)         - region 0: 0x00000006040000 - 0x00000006080000
>>>> (XEN) GICv3: using at most 57344 LPIs on the host.
>>>> (XEN) GICv3: 288 lines, (IID 0001143b).
>>>> (XEN) GICv3: Found ITS @0x6020000
>>>> (XEN) using non-cacheable ITS command queue
>>>> (XEN) GICv3: CPU0: Found redistributor in region 0 @000000004001c000
>>>> 
>>>> [    0.000000] GICv3: Distributor has no Range Selector support
>>>> [    0.000000] GICv3: no VLPI support, no direct LPI support
>>>> [    0.000000] ITS [mem 0x06020000-0x0603ffff]
>>>> [    0.000000] ITS@0x0000000006020000: allocated 65536 Devices
>> @dc880000 (flat, esz 8, psz 64K, shr 1)
>>>> [    0.000000] ITS@0x0000000006020000: allocated 32768 Interrupt
>> Collections @dc820000 (flat, esz 2, psz 64K, shr 1)
>>>> [    0.000000] GIC: using LPI property table @0x00000000dc830000
>>>> [    0.000000] GICv3: CPU0: found redistributor 0 region
>> 0:0x0000000006040000
>>>> [    0.000000] CPU0: using LPI pending table @0x00000000dc840000
>>>> ...
>>>> [    0.040080] Platform MSI: gic-its domain created
>>>> [    0.040136] PCI/MSI: /interrupt-controller/gic-its domain created
>>>> [    0.040181] fsl-mc MSI: /interrupt-controller/gic-its domain created
>>>> 
>>>> 
>>>> Still I am ending up with the " Failed to add - passthrough or MSI/MSI-X
>> might fail!" log messages for some PCI devices, but at least the on-board
>> ethernet ports (fsl_enetc ) are initialized.
>>>> I can set the link up and a link is successfully established.
>> 
>> This message is normal. Xen on Arm is not yet aware of PCI devices and
>> therefore the hypercalls to add PCI devices will return -EOPNOTSUPP.
>> 
>> However, this is not an issue because the virtual ITS implementation will
>> allow dom0 to configure any devices.
>> 
>>>> 
>>>> But (!) I cannot receive or transmit anything (no error message...) and
>> there seem to be no interrupts:
>>>> 
>>>> 29:          0   ITS-MSI   1 Edge      gbe0-rxtx0
>>>>  32:          0   ITS-MSI 8192 Edge      ptp_qoriq
>>>> 
>>>> (from /proc/interrupts).
>>>> 
>>>> Any idea on this one? I keep digging and checking whether my device tree
>> needs some additional fixes.
>> 
>> Can you apply patch [1] and provide the logs? This will dump more
>> information about the command received by the vITS and the one send to
>> the host ITS.
> 
> For XEN 4.13.2 I had to adapt your patch slightly [1], see below (yes I know, quite ugly in parts).
> Find attached the boot log and an output of "xl dmesg" which is truncated due to the large amount of messages.
> 
> When enabling the network interface (gbe0), the following output is visible:
> 
> root@kontron-sal28:~# ip link set up dev gbe0
> (XEN) vgic-v3-its.c:902:d0v0 vITS  cmd 0x0c: 000000170000000c 0000000000000001 0000000000000000 0000000000000000
> (XEN) vgic-v3-its.c:902:d0v0 vITS  cmd 0x05: 0000000000000005 0000000000000000 0000000000000000 0000000000000000
> [   34.034598] Atheros 8031 ethernet 0000:00:00.3:05: attached PHY driver [Atheros 8031 ethernet] (mii_bus:phy_addr=0000:00:00.3:05, irq=POLL)
> [   34.041111] 8021q: adding VLAN 0 to HW filter on device gbe0
> [   34.041209] IPv6: ADDRCONF(NETDEV_UP): gbe0: link is not ready
> root@kontron-sal28:~# [   35.041951] fsl_enetc 0000:00:00.0 gbe0: Link is Down
> [   38.114426] fsl_enetc 0000:00:00.0 gbe0: Link is Up - 1Gbps/Full - flow control off
> [   38.114508] IPv6: ADDRCONF(NETDEV_CHANGE): gbe0: link becomes ready
> 
> Does that tell you anything?
> 

I just checked the logs shared, what I found out that there’s is an error while booting to configure the MSI for the PCI device because of that there will be case that Device Id generate out-of-band is not mapped correctly to ITS device table created while initialising the MSI for the device. 
I might be wrong let someone else also comments on this. 

 
[    0.173964] OF: /soc/pcie@1f0000000: Invalid msi-map translation - no match for rid 0xf8 on           (null)
 
Regards,
Rahul

>> 
>> Note that Xen will need to be build with CONFIG_DEBUG=y in order to get
>> some of the messages.
>> 
>> [...]
>> 
>>>>>> [    0.000000] GICv3: Distributor has no Range Selector support
>>>>>> 
>>>>>> [    0.000000] GICv3: no VLPI support, no direct LPI support
>>>>>> 
>>>>>> [    0.000000] GICv3: CPU0: found redistributor 0 region
>>>>>> 0:0x0000000006040000
>>>>> 
>>>>> "no VLPI support" is very suspicious, it looks like Dom0 doesn't
>>>>> find any ITS support.
>> VLPI is a feature that was introduced in GICv4 to directly inject LPI in the
>> guest. So this is normal to see this message when running on Xen because
>> we are going to only expose a GICv3 to a domain until at least we support
>> nested virt.
>> 
>> However, you were right about that Xen didn't expose the ITS because the
>> following lines were missing:
>> 
>> [    0.000000] ITS@0x0000000006020000: allocated 65536 Devices @dc880000
>> (flat, esz 8, psz 64K, shr 1)
>> 
>> Cheers,
> 
> Best regards,
> Leo
> 
>> 
>> [1]
>> diff --git a/xen/arch/arm/gic-v3-its.c b/xen/arch/arm/gic-v3-its.c index
>> 9558bad96ac3..8a0a02308e74 100644
>> --- a/xen/arch/arm/gic-v3-its.c
>> +++ b/xen/arch/arm/gic-v3-its.c
>> @@ -87,6 +87,10 @@ static int its_send_command(struct host_its *hw_its,
>> const void *its_cmd)
>>      /* No ITS commands from an interrupt handler (at the moment). */
>>      ASSERT(!in_irq());
>> 
>> +    printk(XENLOG_WARNING, "pITS  cmd 0x%02lx: %016lx %016lx %016lx
>> %016lx\n",
>> +           its_cmd_get_command(command),
>> +           command[0], command[1], command[2], command[3]);
>> +
>>      spin_lock(&hw_its->cmd_lock);
>> 
>>      do {
>> diff --git a/xen/arch/arm/gic-v3-lpi.c b/xen/arch/arm/gic-v3-lpi.c index
>> 869bc97fa1aa..e7c5bcd8d423 100644
>> --- a/xen/arch/arm/gic-v3-lpi.c
>> +++ b/xen/arch/arm/gic-v3-lpi.c
>> @@ -183,7 +183,10 @@ void gicv3_do_LPI(unsigned int lpi)
>>      /* Find out if a guest mapped something to this physical LPI. */
>>      hlpip = gic_get_host_lpi(lpi);
>>      if ( !hlpip )
>> +    {
>> +        printk("%s: Received LPI %u but it is not mapped?\n", __func__,
>> lpi);
>>          goto out;
>> +    }
>> 
>>      hlpi.data = read_u64_atomic(&hlpip->data);
>> 
>> @@ -222,6 +225,9 @@ void gicv3_lpi_update_host_entry(uint32_t host_lpi,
>> int domain_id,
>>  {
>>      union host_lpi *hlpip, hlpi;
>> 
>> +    printk("%s: host_lpi %u domain %d virq_lpi %u\n",
>> +           __func__, host_lpi, domain_id, virq_lpi);
>> +
>>      ASSERT(host_lpi >= LPI_OFFSET);
>> 
>>      host_lpi -= LPI_OFFSET;
>> diff --git a/xen/arch/arm/vgic-v3-its.c b/xen/arch/arm/vgic-v3-its.c index
>> 58d939b85f92..89ef137b3e6b 100644
>> --- a/xen/arch/arm/vgic-v3-its.c
>> +++ b/xen/arch/arm/vgic-v3-its.c
>> @@ -897,7 +897,7 @@ out_unlock:
>> 
>>  static void dump_its_command(uint64_t *command)
>>  {
>> -    gdprintk(XENLOG_WARNING, "  cmd 0x%02lx: %016lx %016lx %016lx
>> %016lx\n",
>> +    gdprintk(XENLOG_WARNING, "vITS  cmd 0x%02lx: %016lx %016lx %016lx
>> %016lx\n",
>>               its_cmd_get_command(command),
>>               command[0], command[1], command[2], command[3]);
>>  }
>> @@ -926,6 +926,8 @@ static int vgic_its_handle_cmds(struct domain *d,
>> struct virt_its *its)
>>          if ( ret )
>>              return ret;
>> 
>> +        dump_its_command(command);
>> +
>>          switch ( its_cmd_get_command(command) )
>>          {
>>          case GITS_CMD_CLEAR:
>> 
>> 
>> --
>> Julien Grall
> 
> [0] https://www.mail-archive.com/u-boot@lists.denx.de/msg379708.html
> [1] 
> diff --git a/xen/arch/arm/gic-v3-its.c b/xen/arch/arm/gic-v3-its.c
> index 9558bad96a..d175ba52b0 100644
> --- a/xen/arch/arm/gic-v3-its.c
> +++ b/xen/arch/arm/gic-v3-its.c
> @@ -87,6 +87,10 @@ static int its_send_command(struct host_its *hw_its, const void *its_cmd)
>     /* No ITS commands from an interrupt handler (at the moment). */
>     ASSERT(!in_irq());
> 
> +    printk(XENLOG_WARNING "pITS  cmd 0x%02lx: %016lx %016lx %016lx %016lx\n",
> +        (((uint64_t *) its_cmd)[0] >> 0) & GENMASK(8 - 1, 0),
> +        ((uint64_t *) its_cmd)[0], ((uint64_t *) its_cmd)[1], ((uint64_t *) its_cmd)[2], ((uint64_t *) its_cmd)[3]);
> +
>     spin_lock(&hw_its->cmd_lock);
> 
>     do {
> diff --git a/xen/arch/arm/gic-v3-lpi.c b/xen/arch/arm/gic-v3-lpi.c
> index 78b9521b21..2c3b0fc9e5 100644
> --- a/xen/arch/arm/gic-v3-lpi.c
> +++ b/xen/arch/arm/gic-v3-lpi.c
> @@ -181,8 +181,10 @@ void gicv3_do_LPI(unsigned int lpi)
> 
>     /* Find out if a guest mapped something to this physical LPI. */
>     hlpip = gic_get_host_lpi(lpi);
> -    if ( !hlpip )
> +    if ( !hlpip ) {
> +        printk("%s: Received LPI %u but it is not mapped?\n", __func__, lpi);
>         goto out;
> +    }
> 
>     hlpi.data = read_u64_atomic(&hlpip->data);
> 
> @@ -221,6 +223,9 @@ void gicv3_lpi_update_host_entry(uint32_t host_lpi, int domain_id,
> {
>     union host_lpi *hlpip, hlpi;
> 
> +    printk("%s: host_lpi %u domain %d virt_lpi %u\n",
> +        __func__, host_lpi, domain_id, virt_lpi);
> +
>     ASSERT(host_lpi >= LPI_OFFSET);
> 
>     host_lpi -= LPI_OFFSET;
> diff --git a/xen/arch/arm/vgic-v3-its.c b/xen/arch/arm/vgic-v3-its.c
> index 6e153c698d..dd5081ef80 100644
> --- a/xen/arch/arm/vgic-v3-its.c
> +++ b/xen/arch/arm/vgic-v3-its.c
> @@ -897,7 +897,7 @@ out_unlock:
> 
> static void dump_its_command(uint64_t *command)
> {
> -    gdprintk(XENLOG_WARNING, "  cmd 0x%02lx: %016lx %016lx %016lx %016lx\n",
> +    gdprintk(XENLOG_WARNING, "vITS  cmd 0x%02lx: %016lx %016lx %016lx %016lx\n",
>              its_cmd_get_command(command),
>              command[0], command[1], command[2], command[3]);
> }
> @@ -926,6 +926,8 @@ static int vgic_its_handle_cmds(struct domain *d, struct virt_its *its)
>         if ( ret )
>             return ret;
> 
> +        dump_its_command(command);
> +
>         switch ( its_cmd_get_command(command) )
>         {
>         case GITS_CMD_CLEAR:
> <boot-xendebug.log><xen-debug-output.txt>
Re: Xen data from meta-virtualization layer
Posted by Julien Grall 3 years, 5 months ago 

On 23/11/2020 11:41, Rahul Singh wrote:
> Hello ,

Hi Rahul,

>> On 22 Nov 2020, at 10:55 pm, Leo Krueger <leo.krueger@zal.aero> wrote:
>> root@kontron-sal28:~# ip link set up dev gbe0
>> (XEN) vgic-v3-its.c:902:d0v0 vITS  cmd 0x0c: 000000170000000c 0000000000000001 0000000000000000 0000000000000000
>> (XEN) vgic-v3-its.c:902:d0v0 vITS  cmd 0x05: 0000000000000005 0000000000000000 0000000000000000 0000000000000000
>> [   34.034598] Atheros 8031 ethernet 0000:00:00.3:05: attached PHY driver [Atheros 8031 ethernet] (mii_bus:phy_addr=0000:00:00.3:05, irq=POLL)
>> [   34.041111] 8021q: adding VLAN 0 to HW filter on device gbe0
>> [   34.041209] IPv6: ADDRCONF(NETDEV_UP): gbe0: link is not ready
>> root@kontron-sal28:~# [   35.041951] fsl_enetc 0000:00:00.0 gbe0: Link is Down
>> [   38.114426] fsl_enetc 0000:00:00.0 gbe0: Link is Up - 1Gbps/Full - flow control off
>> [   38.114508] IPv6: ADDRCONF(NETDEV_CHANGE): gbe0: link becomes ready
>>
>> Does that tell you anything?
>>
> 
> I just checked the logs shared, what I found out that there’s is an error while booting to configure the MSI for the PCI device because of that there will be case that Device Id generate out-of-band is not mapped correctly to ITS device table created while initialising the MSI for the device.
> I might be wrong let someone else also comments on this.

I think there might be multiple issues. You spotted one below :).

> [    0.173964] OF: /soc/pcie@1f0000000: Invalid msi-map translation - no match for rid 0xf8 on           (null)

Leo, just to confirm, this error message is not spotted when booting 
Linux on baremetal?

Cheers,

-- 
Julien Grall
AW: Xen data from meta-virtualization layer
Posted by Leo Krueger 3 years, 5 months ago 
Hi,

Thanks for your effort!

> -----Ursprüngliche Nachricht-----
> Von: Julien Grall <julien@xen.org>
> Gesendet: Montag, 23. November 2020 19:42
> An: Rahul Singh <Rahul.Singh@arm.com>; Leo Krueger
> <leo.krueger@zal.aero>
> Cc: Stefano Stabellini <stefano.stabellini@xilinx.com>; Peng Fan
> <peng.fan@nxp.com>; brucea@xilinx.com; Cornelia Bruelhart
> <cornelia.bruelhart@zal.aero>; oleksandr_andrushchenko@epam.com; xen-
> devel@lists.xenproject.org; Bertrand Marquis
> <Bertrand.Marquis@arm.com>
> Betreff: Re: Xen data from meta-virtualization layer
> 
> 
> 
> On 23/11/2020 11:41, Rahul Singh wrote:
> > Hello ,
> 
> Hi Rahul,
> 
> >> On 22 Nov 2020, at 10:55 pm, Leo Krueger <leo.krueger@zal.aero> wrote:
> >> root@kontron-sal28:~# ip link set up dev gbe0
> >> (XEN) vgic-v3-its.c:902:d0v0 vITS  cmd 0x0c: 000000170000000c
> >> 0000000000000001 0000000000000000 0000000000000000
> >> (XEN) vgic-v3-its.c:902:d0v0 vITS  cmd 0x05: 0000000000000005
> 0000000000000000 0000000000000000 0000000000000000
> >> [   34.034598] Atheros 8031 ethernet 0000:00:00.3:05: attached PHY driver
> [Atheros 8031 ethernet] (mii_bus:phy_addr=0000:00:00.3:05, irq=POLL)
> >> [   34.041111] 8021q: adding VLAN 0 to HW filter on device gbe0
> >> [   34.041209] IPv6: ADDRCONF(NETDEV_UP): gbe0: link is not ready
> >> root@kontron-sal28:~# [   35.041951] fsl_enetc 0000:00:00.0 gbe0: Link is
> Down
> >> [   38.114426] fsl_enetc 0000:00:00.0 gbe0: Link is Up - 1Gbps/Full - flow
> control off
> >> [   38.114508] IPv6: ADDRCONF(NETDEV_CHANGE): gbe0: link becomes
> ready
> >>
> >> Does that tell you anything?
> >>
> >
> > I just checked the logs shared, what I found out that there’s is an error
> while booting to configure the MSI for the PCI device because of that there
> will be case that Device Id generate out-of-band is not mapped correctly to
> ITS device table created while initialising the MSI for the device.
> > I might be wrong let someone else also comments on this.
> 
> I think there might be multiple issues. You spotted one below :).
> 
> > [    0.173964] OF: /soc/pcie@1f0000000: Invalid msi-map translation - no
> match for rid 0xf8 on           (null)
> 
> Leo, just to confirm, this error message is not spotted when booting Linux on
> baremetal?

In fact it is:

[    0.160077] OF: /soc/pcie@1f0000000: Invalid msi-map translation - no match for rid 0xf8 on           (null)

But everything works as expected here:

110:      34288          0   ITS-MSI   1 Edge      gbe0-rxtx0
111:          0       6196   ITS-MSI   2 Edge      gbe0-rxtx1

> 
> Cheers,

Best wishes

> 
> --
> Julien Grall

Patchew 2.1-devel-0870f84

© 2016 - 2024 Red Hat, Inc.