[PATCH 0/2] J2 Turtle Board fixes

Artur Rojek posted 2 patches 10 months ago
arch/sh/kernel/vmlinux.lds.S    | 15 ++++++++++++++-
drivers/clocksource/jcore-pit.c | 15 ++++++++++++++-
drivers/irqchip/irq-jcore-aic.c |  2 +-
3 files changed, 29 insertions(+), 3 deletions(-)
[PATCH 0/2] J2 Turtle Board fixes
Posted by Artur Rojek 10 months ago
Hi all,

this series fixes boot issues and allows J2 Turtle Board to boot
upstream Linux again.

Patch [1/2] enforces 8-byte alignment for the dtb offset.

Patch [2/2] resolves a problem with PIT interrupts failing to register.

Even with the above fixes, Turtle Board is prone to occasional freezes
related to clock source transition from periodic to hrtimers. I however
decided to send those two patches ahead and debug the third issue at a
later time. 

Cheers,
Artur

Artur Rojek (2):
  sh: align .bss section padding to 8-byte boundary
  irqchip: clocksource: fix jcore-pit irq request

 arch/sh/kernel/vmlinux.lds.S    | 15 ++++++++++++++-
 drivers/clocksource/jcore-pit.c | 15 ++++++++++++++-
 drivers/irqchip/irq-jcore-aic.c |  2 +-
 3 files changed, 29 insertions(+), 3 deletions(-)

-- 
2.48.1
Re: [PATCH 0/2] J2 Turtle Board fixes
Posted by John Paul Adrian Glaubitz 9 months, 3 weeks ago
Hi Artur,

On Sun, 2025-02-16 at 18:55 +0100, Artur Rojek wrote:
> this series fixes boot issues and allows J2 Turtle Board to boot
> upstream Linux again.
> 
> Patch [1/2] enforces 8-byte alignment for the dtb offset.
> 
> Patch [2/2] resolves a problem with PIT interrupts failing to register.

I can confirm that this series makes my J2 Turtle Board boot again!

> Even with the above fixes, Turtle Board is prone to occasional freezes
> related to clock source transition from periodic to hrtimers. I however
> decided to send those two patches ahead and debug the third issue at a
> later time. 

Yep, it just got stuck for me right after these messages at my first boot attempt:

clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
futex hash table entries: 512 (order: 1, 8192 bytes, linear)
NET: Registered PF_NETLINK/PF_ROUTE protocol family
clocksource: Switched to clocksource jcore_pit_cs

It boots past these messages on second attempt, although it's now stuck trying to start
/init. However, it's still echoing <RETURN> strokes, so it might be an issue with Toybox.

Adrian

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer
`. `'   Physicist
  `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913
Re: [PATCH 0/2] J2 Turtle Board fixes
Posted by Rob Landley 9 months, 3 weeks ago
On 2/27/25 01:52, John Paul Adrian Glaubitz wrote:
> Hi Artur,
> 
> On Sun, 2025-02-16 at 18:55 +0100, Artur Rojek wrote:
>> this series fixes boot issues and allows J2 Turtle Board to boot
>> upstream Linux again.
>>
>> Patch [1/2] enforces 8-byte alignment for the dtb offset.
>>
>> Patch [2/2] resolves a problem with PIT interrupts failing to register.
> 
> I can confirm that this series makes my J2 Turtle Board boot again!
> 
>> Even with the above fixes, Turtle Board is prone to occasional freezes
>> related to clock source transition from periodic to hrtimers. I however
>> decided to send those two patches ahead and debug the third issue at a
>> later time.
> 
> Yep, it just got stuck for me right after these messages at my first boot attempt:
> 
> clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
> futex hash table entries: 512 (order: 1, 8192 bytes, linear)
> NET: Registered PF_NETLINK/PF_ROUTE protocol family
> clocksource: Switched to clocksource jcore_pit_cs
> 
> It boots past these messages on second attempt, although it's now stuck trying to start
> /init. However, it's still echoing <RETURN> strokes, so it might be an issue with Toybox.

Which was fixed a year ago, which is why I told you to use the new 
toolchain with a current musl-libc:

http://lists.landley.net/pipermail/toybox-landley.net/2024-February/030040.html

Unless you're hitting the OTHER issue I fixed last year...

https://github.com/landley/toybox/commit/0b2d5c2bb3f1

Rob
Re: [PATCH 0/2] J2 Turtle Board fixes
Posted by John Paul Adrian Glaubitz 9 months, 3 weeks ago
Hi Rob,

On Thu, 2025-02-27 at 21:03 -0600, Rob Landley wrote:
> On 2/27/25 01:52, John Paul Adrian Glaubitz wrote:
> > Hi Artur,
> > 
> > On Sun, 2025-02-16 at 18:55 +0100, Artur Rojek wrote:
> > > this series fixes boot issues and allows J2 Turtle Board to boot
> > > upstream Linux again.
> > > 
> > > Patch [1/2] enforces 8-byte alignment for the dtb offset.
> > > 
> > > Patch [2/2] resolves a problem with PIT interrupts failing to register.
> > 
> > I can confirm that this series makes my J2 Turtle Board boot again!
> > 
> > > Even with the above fixes, Turtle Board is prone to occasional freezes
> > > related to clock source transition from periodic to hrtimers. I however
> > > decided to send those two patches ahead and debug the third issue at a
> > > later time.
> > 
> > Yep, it just got stuck for me right after these messages at my first boot attempt:
> > 
> > clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
> > futex hash table entries: 512 (order: 1, 8192 bytes, linear)
> > NET: Registered PF_NETLINK/PF_ROUTE protocol family
> > clocksource: Switched to clocksource jcore_pit_cs
> > 
> > It boots past these messages on second attempt, although it's now stuck trying to start
> > /init. However, it's still echoing <RETURN> strokes, so it might be an issue with Toybox.
> 
> Which was fixed a year ago, which is why I told you to use the new 
> toolchain with a current musl-libc:
> 
> http://lists.landley.net/pipermail/toybox-landley.net/2024-February/030040.html
> 
> Unless you're hitting the OTHER issue I fixed last year...
> 
> https://github.com/landley/toybox/commit/0b2d5c2bb3f1

I just downloaded the latest toolchain from:

https://landley.net/bin/toolchains/latest/sh2eb-linux-muslfdpic-cross.tar.xz

and the issue still persists.

Am I missing anything?

Thanks,
Adrian

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer
`. `'   Physicist
  `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913
Re: [PATCH 0/2] J2 Turtle Board fixes
Posted by Rob Landley 9 months, 3 weeks ago
On 2/28/25 02:34, John Paul Adrian Glaubitz wrote:
> Hi Rob,
> 
> On Thu, 2025-02-27 at 21:03 -0600, Rob Landley wrote:
>> On 2/27/25 01:52, John Paul Adrian Glaubitz wrote:
>>> Hi Artur,
>>>
>>> On Sun, 2025-02-16 at 18:55 +0100, Artur Rojek wrote:
>>>> this series fixes boot issues and allows J2 Turtle Board to boot
>>>> upstream Linux again.
>>>>
>>>> Patch [1/2] enforces 8-byte alignment for the dtb offset.
>>>>
>>>> Patch [2/2] resolves a problem with PIT interrupts failing to register.
>>>
>>> I can confirm that this series makes my J2 Turtle Board boot again!
>>>
>>>> Even with the above fixes, Turtle Board is prone to occasional freezes
>>>> related to clock source transition from periodic to hrtimers. I however
>>>> decided to send those two patches ahead and debug the third issue at a
>>>> later time.
>>>
>>> Yep, it just got stuck for me right after these messages at my first boot attempt:
>>>
>>> clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
>>> futex hash table entries: 512 (order: 1, 8192 bytes, linear)
>>> NET: Registered PF_NETLINK/PF_ROUTE protocol family
>>> clocksource: Switched to clocksource jcore_pit_cs
>>>
>>> It boots past these messages on second attempt, although it's now stuck trying to start
>>> /init. However, it's still echoing <RETURN> strokes, so it might be an issue with Toybox.
>>
>> Which was fixed a year ago, which is why I told you to use the new
>> toolchain with a current musl-libc:
>>
>> http://lists.landley.net/pipermail/toybox-landley.net/2024-February/030040.html
>>
>> Unless you're hitting the OTHER issue I fixed last year...
>>
>> https://github.com/landley/toybox/commit/0b2d5c2bb3f1
> 
> I just downloaded the latest toolchain from:
> 
> https://landley.net/bin/toolchains/latest/sh2eb-linux-muslfdpic-cross.tar.xz
> 
> and the issue still persists.
> 
> Am I missing anything?

The march 2024 rebuild was in response to that Feb 2024 bugfix, so it 
_should_ have the fix? (I'm waiting for another musl release to rebuild 
them again...)

I just downloaded the toolchain currently at that URL and built mkroot 
and it worked for me:

Run /init as init process
sntp: time.google.com:123: Try again
Type exit when done.
$ cat /proc/version
Linux version 6.14.0-rc3 (landley@driftwood) (sh2eb-linux-muslfdpic-cc 
(GCC) 11.2.0, GNU ld (GNU Binutils) 2.33.1) #1 SMP Fri Feb 28 15:47:36 
CST 2025

And the failure _without_ the fix was deterministic rather than 
intermittent, so...

Keep in mind the init script has a 3 second timeout trying to call sntp 
to set the clock, which will fail if the ethernet isn't connected (or no 
driver, or no internet...)

Rob

P.S. Speaking of intermittent, I hit that hang after "clocksource: 
Switched to clocksource jcore_pit_cs" on one attempt just now. I should 
sit down with the engineers next time I'm in japan and try to root cause 
it. The scheduler fires reliably, so it's _probably_ not a hardware 
issue? We've had Linux uptime of over a year, not just idle but running 
an energy monitoring app, so it's pretty stable in our systems...
Re: [PATCH 0/2] J2 Turtle Board fixes
Posted by John Paul Adrian Glaubitz 8 months, 2 weeks ago
Hi Rob,

On Fri, 2025-02-28 at 16:19 -0600, Rob Landley wrote:
> > > Which was fixed a year ago, which is why I told you to use the new
> > > toolchain with a current musl-libc:
> > > 
> > > http://lists.landley.net/pipermail/toybox-landley.net/2024-February/030040.html
> > > 
> > > Unless you're hitting the OTHER issue I fixed last year...
> > > 
> > > https://github.com/landley/toybox/commit/0b2d5c2bb3f1
> > 
> > I just downloaded the latest toolchain from:
> > 
> > https://landley.net/bin/toolchains/latest/sh2eb-linux-muslfdpic-cross.tar.xz
> > 
> > and the issue still persists.
> > 
> > Am I missing anything?
> 
> The march 2024 rebuild was in response to that Feb 2024 bugfix, so it 
> _should_ have the fix? (I'm waiting for another musl release to rebuild 
> them again...)
> 
> I just downloaded the toolchain currently at that URL and built mkroot 
> and it worked for me:
> 
> Run /init as init process
> sntp: time.google.com:123: Try again
> Type exit when done.
> $ cat /proc/version
> Linux version 6.14.0-rc3 (landley@driftwood) (sh2eb-linux-muslfdpic-cc 
> (GCC) 11.2.0, GNU ld (GNU Binutils) 2.33.1) #1 SMP Fri Feb 28 15:47:36 
> CST 2025
> 
> And the failure _without_ the fix was deterministic rather than 
> intermittent, so...
> 
> Keep in mind the init script has a 3 second timeout trying to call sntp 
> to set the clock, which will fail if the ethernet isn't connected (or no 
> driver, or no internet...)

I just gave it another try and it still hangs for me at:

	Run /init as init process

with the latest toolchain, toybox and kernel (v6.15-rc-1).

FWIW, I did not connect an ethernet cable.

Adrian

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer
`. `'   Physicist
  `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913
Re: [PATCH 0/2] J2 Turtle Board fixes
Posted by Rob Landley 8 months, 1 week ago
On 4/8/25 10:23, John Paul Adrian Glaubitz wrote:
> Hi Rob,
> 
> On Fri, 2025-02-28 at 16:19 -0600, Rob Landley wrote:
>>>> Which was fixed a year ago, which is why I told you to use the new
>>>> toolchain with a current musl-libc:
>>>>
>>>> http://lists.landley.net/pipermail/toybox-landley.net/2024-February/030040.html
>>>>
>>>> Unless you're hitting the OTHER issue I fixed last year...
>>>>
>>>> https://github.com/landley/toybox/commit/0b2d5c2bb3f1
>>>
>>> I just downloaded the latest toolchain from:
>>>
>>> https://landley.net/bin/toolchains/latest/sh2eb-linux-muslfdpic-cross.tar.xz
>>>
>>> and the issue still persists.
>>>
>>> Am I missing anything?
>>
>> The march 2024 rebuild was in response to that Feb 2024 bugfix, so it
>> _should_ have the fix? (I'm waiting for another musl release to rebuild
>> them again...)
>>
>> I just downloaded the toolchain currently at that URL and built mkroot
>> and it worked for me:
>>
>> Run /init as init process
>> sntp: time.google.com:123: Try again
>> Type exit when done.
>> $ cat /proc/version
>> Linux version 6.14.0-rc3 (landley@driftwood) (sh2eb-linux-muslfdpic-cc
>> (GCC) 11.2.0, GNU ld (GNU Binutils) 2.33.1) #1 SMP Fri Feb 28 15:47:36
>> CST 2025
>>
>> And the failure _without_ the fix was deterministic rather than
>> intermittent, so...
>>
>> Keep in mind the init script has a 3 second timeout trying to call sntp
>> to set the clock, which will fail if the ethernet isn't connected (or no
>> driver, or no internet...)
> 
> I just gave it another try and it still hangs for me at:
> 
> 	Run /init as init process
> 
> with the latest toolchain, toybox and kernel (v6.15-rc-1).

FYI I reproduced this but haven't tracked it down yet.

Rob
Re: [PATCH 0/2] J2 Turtle Board fixes
Posted by John Paul Adrian Glaubitz 6 months, 1 week ago
Hi Rob,

On Fri, 2025-04-11 at 06:25 -0500, Rob Landley wrote:
> > I just gave it another try and it still hangs for me at:
> > 
> > 	Run /init as init process
> > 
> > with the latest toolchain, toybox and kernel (v6.15-rc-1).
> 
> FYI I reproduced this but haven't tracked it down yet.

I have updated to the latest git revision now with the result that it
fails executing /init now. Can you confirm this error?

Adrian

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer
`. `'   Physicist
  `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913
Re: [PATCH 0/2] J2 Turtle Board fixes
Posted by Rob Landley 6 months, 1 week ago
On 6/9/25 05:01, John Paul Adrian Glaubitz wrote:
> Hi Rob,
> 
> On Fri, 2025-04-11 at 06:25 -0500, Rob Landley wrote:
>>> I just gave it another try and it still hangs for me at:
>>>
>>> 	Run /init as init process
>>>
>>> with the latest toolchain, toybox and kernel (v6.15-rc-1).
>>
>> FYI I reproduced this but haven't tracked it down yet.
> 
> I have updated to the latest git revision now with the result that it
> fails executing /init now. Can you confirm this error?

6.15 works for me? How did I reproduce it last time... Ah right, I 
always build turtle with my patch stack applied:

https://landley.net/bin/mkroot/latest/linux-patches/

Sorry, fell off my todo list. I'll take another look. Thanks for the poke.

Rob
Re: [PATCH 0/2] J2 Turtle Board fixes
Posted by John Paul Adrian Glaubitz 9 months, 3 weeks ago
Hi,

On Fri, 2025-02-28 at 16:19 -0600, Rob Landley wrote:
> The march 2024 rebuild was in response to that Feb 2024 bugfix, so it 
> _should_ have the fix? (I'm waiting for another musl release to rebuild 
> them again...)
> 
> I just downloaded the toolchain currently at that URL and built mkroot 
> and it worked for me:
> 
> Run /init as init process
> sntp: time.google.com:123: Try again
> Type exit when done.
> $ cat /proc/version
> Linux version 6.14.0-rc3 (landley@driftwood) (sh2eb-linux-muslfdpic-cc 
> (GCC) 11.2.0, GNU ld (GNU Binutils) 2.33.1) #1 SMP Fri Feb 28 15:47:36 
> CST 2025

Is that on Toybox git HEAD?

> And the failure _without_ the fix was deterministic rather than 
> intermittent, so...
> 
> Keep in mind the init script has a 3 second timeout trying to call sntp 
> to set the clock, which will fail if the ethernet isn't connected (or no 
> driver, or no internet...)

I'll try again this weekend. Also, I will review and pick up the fix.

> P.S. Speaking of intermittent, I hit that hang after "clocksource: 
> Switched to clocksource jcore_pit_cs" on one attempt just now. I should 
> sit down with the engineers next time I'm in japan and try to root cause 
> it. The scheduler fires reliably, so it's _probably_ not a hardware 
> issue? We've had Linux uptime of over a year, not just idle but running 
> an energy monitoring app, so it's pretty stable in our systems...

I thought it was a software issue?

Adrian

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer
`. `'   Physicist
  `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913
Re: [PATCH 0/2] J2 Turtle Board fixes
Posted by Rob Landley 9 months, 3 weeks ago
On 2/28/25 16:34, John Paul Adrian Glaubitz wrote:
> Hi,
> 
> On Fri, 2025-02-28 at 16:19 -0600, Rob Landley wrote:
>> The march 2024 rebuild was in response to that Feb 2024 bugfix, so it
>> _should_ have the fix? (I'm waiting for another musl release to rebuild
>> them again...)
>>
>> I just downloaded the toolchain currently at that URL and built mkroot
>> and it worked for me:
>>
>> Run /init as init process
>> sntp: time.google.com:123: Try again
>> Type exit when done.
>> $ cat /proc/version
>> Linux version 6.14.0-rc3 (landley@driftwood) (sh2eb-linux-muslfdpic-cc
>> (GCC) 11.2.0, GNU ld (GNU Binutils) 2.33.1) #1 SMP Fri Feb 28 15:47:36
>> CST 2025
> 
> Is that on Toybox git HEAD?

Yes. Commit b4fd71d18f84.

>> And the failure _without_ the fix was deterministic rather than
>> intermittent, so...
>>
>> Keep in mind the init script has a 3 second timeout trying to call sntp
>> to set the clock, which will fail if the ethernet isn't connected (or no
>> driver, or no internet...)
> 
> I'll try again this weekend. Also, I will review and pick up the fix.

Ok. (I'm available Saturday if you need to poke me, but traveling sunday.)

>> P.S. Speaking of intermittent, I hit that hang after "clocksource:
>> Switched to clocksource jcore_pit_cs" on one attempt just now. I should
>> sit down with the engineers next time I'm in japan and try to root cause
>> it. The scheduler fires reliably, so it's _probably_ not a hardware
>> issue? We've had Linux uptime of over a year, not just idle but running
>> an energy monitoring app, so it's pretty stable in our systems...
> 
> I thought it was a software issue?

I agree. That's why I said it's probably not a hardware issue. (The 
config has NO_HZ_IDLE so if the PIT didn't fire reliably when 
reprogrammed the scheduler would flake, and it's not, so...)

> Adrian

Rob