The T-head TH1520 SoC supports a hardware mailbox that enables two cores
within the SoC to communicate and coordinate [1]. One example of such
coordination would be cooperation with the T-Head E902 core, which is
responsible for power, clock, and resource management. For example, in
the specific case of the BXM-4-64 GPU, it needs to be powered on by the
E902 core, and the kernel running on the E910 needs to 'ask' the
firmware running on the E902 core to enable power to the GPU island.
Given recent advancements in work on the upstream GPU driver [2], there
is an emerging need to get this code in the mainline kernel.

Link: https://openbeagle.org/beaglev-ahead/beaglev-ahead/-/blob/main/docs/TH1520%20System%20User%20Manual.pdf [1]
Link: https://gitlab.freedesktop.org/imagination/linux-firmware/-/issues/1 [2]

Thanks, Krzysztof and Rob, for your review! Since this series is gaining
some interest, I've dropped the RFC prefix with the v3 update.

v4:
- fixed warning of unused variable
- added Reviewed-by from Krzysztof
- fixed minor cosmetic issues in dt-binding

v3:
- added a comment about mixing devm_ and non-devm resources in the context
  of shared interrupts and explained why it's safe to do so in this
  particular case
- changed the order of resource freeing in the .shutdown callback
- used a wrapper function for register mapping
- since the only conceivable use case for this mailbox driver is
  communication with cores not managed by the kernel, I’ve hard-coded
  this by removing the thead,icu-cpu-id property and adjusted the mailbox
  driver code accordingly.
- added a more detailed description for mbox-cells.
- made some cosmetic changes.
- retested by applying the patch with non-yet-upstreamed patches,
  confirming that the drm/imagination driver can read the registers
  correctly.

v2:
 - fixed thead,th1520-mbox.yaml binding file by dropping redundant
   descriptions, renaming reg-names, removing unnecessary clocks,
   providing constraints and defining ICU's
 - fixed the mailbox driver code to work well with updated binding-file,
   removed clocks support, as it's not necessary for mailbox to work
 - adjusted the device tree node instance of mbox_910t so it will work
   with updated bindings file

Michal Wilczynski (3):
  mailbox: Introduce support for T-head TH1520 Mailbox driver
  dt-bindings: mailbox: Add thead,th1520-mailbox bindings
  riscv: dts: thead: Add mailbox node

 .../bindings/mailbox/thead,th1520-mbox.yaml   |  80 +++
 MAINTAINERS                                   |   2 +
 arch/riscv/boot/dts/thead/th1520.dtsi         |  12 +
 drivers/mailbox/Kconfig                       |  10 +
 drivers/mailbox/Makefile                      |   2 +
 drivers/mailbox/mailbox-th1520.c              | 572 ++++++++++++++++++
 6 files changed, 678 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/mailbox/thead,th1520-mbox.yaml
 create mode 100644 drivers/mailbox/mailbox-th1520.c

-- 
2.34.1

On Mon, Oct 14, 2024 at 02:33:11PM +0200, Michal Wilczynski wrote:
> The T-head TH1520 SoC supports a hardware mailbox that enables two cores
> within the SoC to communicate and coordinate [1]. One example of such
> coordination would be cooperation with the T-Head E902 core, which is
> responsible for power, clock, and resource management. For example, in
> the specific case of the BXM-4-64 GPU, it needs to be powered on by the
> E902 core, and the kernel running on the E910 needs to 'ask' the
> firmware running on the E902 core to enable power to the GPU island.
> Given recent advancements in work on the upstream GPU driver [2], there
> is an emerging need to get this code in the mainline kernel.
> 
> Link: https://openbeagle.org/beaglev-ahead/beaglev-ahead/-/blob/main/docs/TH1520%20System%20User%20Manual.pdf [1]
> Link: https://gitlab.freedesktop.org/imagination/linux-firmware/-/issues/1 [2]
> 
> Thanks, Krzysztof and Rob, for your review! Since this series is gaining
> some interest, I've dropped the RFC prefix with the v3 update.

I've applied this series and booted okay. I see the driver loaded:

 /sys/devices/platform/soc/ffffc38000.mailbox/driver points to
 /sys/bus/platform/drivers/th1520-mbox

How do you test that the communication with the E902 is working
correctly?

Thanks,
Drew

On 10/14/24 20:31, Drew Fustini wrote:
> On Mon, Oct 14, 2024 at 02:33:11PM +0200, Michal Wilczynski wrote:
>> The T-head TH1520 SoC supports a hardware mailbox that enables two cores
>> within the SoC to communicate and coordinate [1]. One example of such
>> coordination would be cooperation with the T-Head E902 core, which is
>> responsible for power, clock, and resource management. For example, in
>> the specific case of the BXM-4-64 GPU, it needs to be powered on by the
>> E902 core, and the kernel running on the E910 needs to 'ask' the
>> firmware running on the E902 core to enable power to the GPU island.
>> Given recent advancements in work on the upstream GPU driver [2], there
>> is an emerging need to get this code in the mainline kernel.
>>
>> Link: https://protect2.fireeye.com/v1/url?k=2021d256-7fbdfb7c-20205919-000babe598f7-ca654d1a9bc866ac&q=1&e=11e97355-e6e9-4aac-a996-cc475156b3c8&u=https%3A%2F%2Fopenbeagle.org%2Fbeaglev-ahead%2Fbeaglev-ahead%2F-%2Fblob%2Fmain%2Fdocs%2FTH1520%2520System%2520User%2520Manual.pdf [1]
>> Link: https://gitlab.freedesktop.org/imagination/linux-firmware/-/issues/1 [2]
>>
>> Thanks, Krzysztof and Rob, for your review! Since this series is gaining
>> some interest, I've dropped the RFC prefix with the v3 update.
> 
> I've applied this series and booted okay. I see the driver loaded:
> 
>  /sys/devices/platform/soc/ffffc38000.mailbox/driver points to
>  /sys/bus/platform/drivers/th1520-mbox
> 
> How do you test that the communication with the E902 is working
> correctly?

Thank you for your interest. To test this, I've prepared a diff that
includes the missing drivers utilizing the mailbox and enabled the GPU
node in the device tree to use the drm/imagination driver.

I've observed that when the power was turned off through the E902 core
using the mailbox, the drm/imagination driver would hang in
pvr_load_gpu_id() while attempting to read its BVNC from the register.
However, when the GPU was turned on via the mailbox, the BVNC could be
read correctly. Still, the firmware fails to boot due to some missing
programming in the drm/imagination driver, which is currently being
worked on. I've briefly explained this in the first commit of this
series.

If you'd like to try this yourself, I'd be happy to push these setups to
a GitHub repository and provide you with a link, so you can see the
setup in action.

Michał

> 
> Thanks,
> Drew
> 

On Wed, Oct 16, 2024 at 12:03:05AM +0200, Michal Wilczynski wrote:
> 
> 
> On 10/14/24 20:31, Drew Fustini wrote:
> > On Mon, Oct 14, 2024 at 02:33:11PM +0200, Michal Wilczynski wrote:
> >> The T-head TH1520 SoC supports a hardware mailbox that enables two cores
> >> within the SoC to communicate and coordinate [1]. One example of such
> >> coordination would be cooperation with the T-Head E902 core, which is
> >> responsible for power, clock, and resource management. For example, in
> >> the specific case of the BXM-4-64 GPU, it needs to be powered on by the
> >> E902 core, and the kernel running on the E910 needs to 'ask' the
> >> firmware running on the E902 core to enable power to the GPU island.
> >> Given recent advancements in work on the upstream GPU driver [2], there
> >> is an emerging need to get this code in the mainline kernel.
> >>
> >> Link: https://protect2.fireeye.com/v1/url?k=2021d256-7fbdfb7c-20205919-000babe598f7-ca654d1a9bc866ac&q=1&e=11e97355-e6e9-4aac-a996-cc475156b3c8&u=https%3A%2F%2Fopenbeagle.org%2Fbeaglev-ahead%2Fbeaglev-ahead%2F-%2Fblob%2Fmain%2Fdocs%2FTH1520%2520System%2520User%2520Manual.pdf [1]
> >> Link: https://gitlab.freedesktop.org/imagination/linux-firmware/-/issues/1 [2]
> >>
> >> Thanks, Krzysztof and Rob, for your review! Since this series is gaining
> >> some interest, I've dropped the RFC prefix with the v3 update.
> > 
> > I've applied this series and booted okay. I see the driver loaded:
> > 
> >  /sys/devices/platform/soc/ffffc38000.mailbox/driver points to
> >  /sys/bus/platform/drivers/th1520-mbox
> > 
> > How do you test that the communication with the E902 is working
> > correctly?
> 
> Thank you for your interest. To test this, I've prepared a diff that
> includes the missing drivers utilizing the mailbox and enabled the GPU
> node in the device tree to use the drm/imagination driver.
> 
> I've observed that when the power was turned off through the E902 core
> using the mailbox, the drm/imagination driver would hang in
> pvr_load_gpu_id() while attempting to read its BVNC from the register.
> However, when the GPU was turned on via the mailbox, the BVNC could be
> read correctly. Still, the firmware fails to boot due to some missing
> programming in the drm/imagination driver, which is currently being
> worked on. I've briefly explained this in the first commit of this
> series.
> 
> If you'd like to try this yourself, I'd be happy to push these setups to
> a GitHub repository and provide you with a link, so you can see the
> setup in action.

I think that would be helpful for myself and others to be able to see
the interaction.

Thanks,
Drew

On 10/16/24 20:08, Drew Fustini wrote:
> On Wed, Oct 16, 2024 at 12:03:05AM +0200, Michal Wilczynski wrote:
>>
>>
>> On 10/14/24 20:31, Drew Fustini wrote:
>>> On Mon, Oct 14, 2024 at 02:33:11PM +0200, Michal Wilczynski wrote:
>>>> The T-head TH1520 SoC supports a hardware mailbox that enables two cores
>>>> within the SoC to communicate and coordinate [1]. One example of such
>>>> coordination would be cooperation with the T-Head E902 core, which is
>>>> responsible for power, clock, and resource management. For example, in
>>>> the specific case of the BXM-4-64 GPU, it needs to be powered on by the
>>>> E902 core, and the kernel running on the E910 needs to 'ask' the
>>>> firmware running on the E902 core to enable power to the GPU island.
>>>> Given recent advancements in work on the upstream GPU driver [2], there
>>>> is an emerging need to get this code in the mainline kernel.
>>>>
>>>> Link: https://protect2.fireeye.com/v1/url?k=2021d256-7fbdfb7c-20205919-000babe598f7-ca654d1a9bc866ac&q=1&e=11e97355-e6e9-4aac-a996-cc475156b3c8&u=https%3A%2F%2Fopenbeagle.org%2Fbeaglev-ahead%2Fbeaglev-ahead%2F-%2Fblob%2Fmain%2Fdocs%2FTH1520%2520System%2520User%2520Manual.pdf [1]
>>>> Link: https://gitlab.freedesktop.org/imagination/linux-firmware/-/issues/1 [2]
>>>>
>>>> Thanks, Krzysztof and Rob, for your review! Since this series is gaining
>>>> some interest, I've dropped the RFC prefix with the v3 update.
>>>
>>> I've applied this series and booted okay. I see the driver loaded:
>>>
>>>  /sys/devices/platform/soc/ffffc38000.mailbox/driver points to
>>>  /sys/bus/platform/drivers/th1520-mbox
>>>
>>> How do you test that the communication with the E902 is working
>>> correctly?
>>
>> Thank you for your interest. To test this, I've prepared a diff that
>> includes the missing drivers utilizing the mailbox and enabled the GPU
>> node in the device tree to use the drm/imagination driver.
>>
>> I've observed that when the power was turned off through the E902 core
>> using the mailbox, the drm/imagination driver would hang in
>> pvr_load_gpu_id() while attempting to read its BVNC from the register.
>> However, when the GPU was turned on via the mailbox, the BVNC could be
>> read correctly. Still, the firmware fails to boot due to some missing
>> programming in the drm/imagination driver, which is currently being
>> worked on. I've briefly explained this in the first commit of this
>> series.
>>
>> If you'd like to try this yourself, I'd be happy to push these setups to
>> a GitHub repository and provide you with a link, so you can see the
>> setup in action.
> 
> I think that would be helpful for myself and others to be able to see
> the interaction.

I’ve cleaned up the code and it’s ready to share. I’ll include this
description and the GitHub links in the cover letter for the next
revision.

I’ve created two branches. Both contain the same code, including this
patch series and some extra commits with mailbox consumers. The only
difference is that one branch has an additional commit that switches ON
calls to OFF calls (and vice versa) to observe the behavior when the GPU
is turned off via the mailbox.

To reproduce, simply clone the repository, add the following extra options
in the config:

CONFIG_DRM_POWERVR=y
CONFIG_LIGHT_AON=y
CONFIG_LIGHT_AON_PD=y

Then build and deploy on the target.

In case [1], the output should be:
[    2.478394] light_aon_probe: virtual_log_mem=0x000000005faf564a, phy base=0x33600000, size:2097152
[    2.488589] succeed to create power domain debugfs direntry
[    2.494987] powervr ffef400000.gpu: Before reading BVNC
At this point, the system will hang because the driver is trying to read
memory-mapped registers while the GPU isn’t powered on.

In case [2], the GPU powers on correctly, and the BVNC can be read
without issues. If the specific firmware file isn’t available, the
output would look like this:
root@revyos-lpi4a:~# dmesg | grep gpu
[    2.408207] powervr ffef400000.gpu: Before reading BVNC
[    2.413533] powervr ffef400000.gpu: After reading BVNC
[    2.418930] powervr ffef400000.gpu: Direct firmware load for powervr/rogue_36.52.104.182_v1.fw failed with error -2
[    2.429568] powervr ffef400000.gpu: [drm] *ERROR* failed to load firmware powervr/rogue_36.52.104.182_v1.fw (err=-2)
[    2.440403] powervr ffef400000.gpu: probe with driver powervr failed with error -2

Here are the links:
[1] - https://github.com/mwilczy/linux/tree/22_october_demonstrate_mailbox_not_working
[2] - https://github.com/mwilczy/linux/tree/22_october_demonstrate_mailbox_working

Note: U-Boot must load the AON firmware at startup to the address mapped
for AON for this to work properly.

Thanks,
Michał

> 
> Thanks,
> Drew
>