Currently sched_clock shows the relative time to the boot starting of
kernel, while there could be long firmware start time before it and
after the hardware reset.

On modern server platforms, there could be several software running in
parallel. Like for arm64, it could have  SCP (System Control Processor)
firmware running on SCP processor, and ATF (Arm Trusted Firmware) and
Linux OS on the main processor.

Debugging some nasty issues on these platform may need to cross-check
the logs from these firmwares and Linux kernel for specific events,
where a unified reference timeline is critical. All these software can
read the hardware timer, which is also the base of sched_clock for
Linux kernel. Using the absolute counter since hardware timer reset
makes it possible for all kinds of software to have a same time base.

Add 'abs_sched_clock' parameter to provide an option for using absolute
counter, and users should make sure their sched_clock (hardware timer)
is capable of supporting absolute counter before enabling the option.

Locally, it did help on chasing some RAS issues which needed cooperation
between kernel, SCP firmware and ATF, by mapping the actions from each
players into one timeline based on the timestamps in their logs.

Signed-off-by: Feng Tang <feng.tang@linux.alibaba.com>
---
 kernel/time/sched_clock.c | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)

diff --git a/kernel/time/sched_clock.c b/kernel/time/sched_clock.c
index f3aaef695b8c..321c580d6799 100644
--- a/kernel/time/sched_clock.c
+++ b/kernel/time/sched_clock.c
@@ -49,6 +49,10 @@ static int irqtime = -1;
 
 core_param(irqtime, irqtime, int, 0400);
 
+/* Whether to use the absolute counter since the clock hardware reset */
+static bool abs_sched_clock;
+core_param(abs_sched_clock, abs_sched_clock, bool, 0400);
+
 static u64 notrace jiffy_sched_clock_read(void)
 {
 	/*
@@ -200,10 +204,14 @@ void sched_clock_register(u64 (*read)(void), int bits, unsigned long rate)
 
 	rd = cd.read_data[0];
 
-	/* Update epoch for new counter and update 'epoch_ns' from old counter*/
+	/* Update epoch for new counter and update 'epoch_ns' */
 	new_epoch = read();
-	cyc = cd.actual_read_sched_clock();
-	ns = rd.epoch_ns + cyc_to_ns((cyc - rd.epoch_cyc) & rd.sched_clock_mask, rd.mult, rd.shift);
+	if (abs_sched_clock) {
+		ns = cyc_to_ns(new_epoch & new_mask, new_mult, new_shift);
+	} else {
+		cyc = cd.actual_read_sched_clock();
+		ns = rd.epoch_ns + cyc_to_ns((cyc - rd.epoch_cyc) & rd.sched_clock_mask, rd.mult, rd.shift);
+	}
 	cd.actual_read_sched_clock = read;
 
 	rd.read_sched_clock	= read;
-- 
2.39.5 (Apple Git-154)

On Wed, Sep 02 2026 at 16:21, Feng Tang wrote:
> Locally, it did help on chasing some RAS issues which needed cooperation
> between kernel, SCP firmware and ATF, by mapping the actions from each
> players into one timeline based on the timestamps in their logs.

I told you before that we don't care about your bug chasing war stories
at all. Educate your firmware people and stop pestering us with your
firmware debug hacks.

On Sat, Sep 05, 2026 at 10:39:26PM +0200, Thomas Gleixner wrote:
> On Wed, Sep 02 2026 at 16:21, Feng Tang wrote:
> > Locally, it did help on chasing some RAS issues which needed cooperation
> > between kernel, SCP firmware and ATF, by mapping the actions from each
> > players into one timeline based on the timestamps in their logs.
> 
> I told you before that we don't care about your bug chasing war stories
> at all. Educate your firmware people and stop pestering us with your
> firmware debug hacks.

Hi Thomas,

Yes, this patch is hacky.

From the reply of Marc in this thread:

"
I really have to ask: why isn't this just a one-off sampling of the
counter, kept in some user accessible location (debugfs or something
else), and ultimately post-processed to align your logs? People have
been doing this... forever, and that has been "good enough" so far.
"

IIUC, getting an absolute timestamp for logs is still practical in some
cases. How about adding the offset-since-reset info into the message
like Marc suggested:

---
diff --git a/kernel/time/sched_clock.c b/kernel/time/sched_clock.c
index f3aaef695b8c..d5a54276b49d 100644
--- a/kernel/time/sched_clock.c
+++ b/kernel/time/sched_clock.c
@@ -176,7 +176,7 @@ static enum hrtimer_restart sched_clock_poll(struct hrtimer *hrt)
 
 void sched_clock_register(u64 (*read)(void), int bits, unsigned long rate)
 {
-	u64 res, wrap, new_mask, new_epoch, cyc, ns;
+	u64 res, wrap, new_mask, new_epoch, cyc, ns, reset_ns;
 	u32 new_mult, new_shift;
 	unsigned long r, flags;
 	char r_unit;
@@ -235,8 +235,11 @@ void sched_clock_register(u64 (*read)(void), int bits, unsigned long rate)
 	/* Calculate the ns resolution of this counter */
 	res = cyc_to_ns(1ULL, new_mult, new_shift);
 
-	pr_info("sched_clock: %u bits at %lu%cHz, resolution %lluns, wraps every %lluns\n",
-		bits, r, r_unit, res, wrap);
+	/* Calculate the time since last counter resetting to 0 */
+	reset_ns = mul_u64_u64_div_u64(new_epoch, NSEC_PER_SEC, rate);
+
+	pr_info("sched_clock: %u bits at %lu%cHz, resolution %lluns, wraps every %lluns, has run %lluns since counter reset\n",
+		bits, r, r_unit, res, wrap, reset_ns);
 
 	/* Enable IRQ time accounting if we have a fast enough sched_clock() */
 	if (irqtime > 0 || (irqtime == -1 && rate >= 1000000))

Thanks,
Feng

On Mon, 07 Sep 2026 10:05:40 +0100,
Feng Tang <feng.tang@linux.alibaba.com> wrote:
> 
> On Sat, Sep 05, 2026 at 10:39:26PM +0200, Thomas Gleixner wrote:
> > On Wed, Sep 02 2026 at 16:21, Feng Tang wrote:
> > > Locally, it did help on chasing some RAS issues which needed cooperation
> > > between kernel, SCP firmware and ATF, by mapping the actions from each
> > > players into one timeline based on the timestamps in their logs.
> > 
> > I told you before that we don't care about your bug chasing war stories
> > at all. Educate your firmware people and stop pestering us with your
> > firmware debug hacks.
> 
> Hi Thomas,
> 
> Yes, this patch is hacky.
> 
> From the reply of Marc in this thread:
> 
> "
> I really have to ask: why isn't this just a one-off sampling of the
> counter, kept in some user accessible location (debugfs or something
> else), and ultimately post-processed to align your logs? People have
> been doing this... forever, and that has been "good enough" so far.
> "
> 
> IIUC, getting an absolute timestamp for logs is still practical in some
> cases. How about adding the offset-since-reset info into the message
> like Marc suggested:
> 
> ---
> diff --git a/kernel/time/sched_clock.c b/kernel/time/sched_clock.c
> index f3aaef695b8c..d5a54276b49d 100644
> --- a/kernel/time/sched_clock.c
> +++ b/kernel/time/sched_clock.c
> @@ -176,7 +176,7 @@ static enum hrtimer_restart sched_clock_poll(struct hrtimer *hrt)
>  
>  void sched_clock_register(u64 (*read)(void), int bits, unsigned long rate)
>  {
> -	u64 res, wrap, new_mask, new_epoch, cyc, ns;
> +	u64 res, wrap, new_mask, new_epoch, cyc, ns, reset_ns;
>  	u32 new_mult, new_shift;
>  	unsigned long r, flags;
>  	char r_unit;
> @@ -235,8 +235,11 @@ void sched_clock_register(u64 (*read)(void), int bits, unsigned long rate)
>  	/* Calculate the ns resolution of this counter */
>  	res = cyc_to_ns(1ULL, new_mult, new_shift);
>  
> -	pr_info("sched_clock: %u bits at %lu%cHz, resolution %lluns, wraps every %lluns\n",
> -		bits, r, r_unit, res, wrap);
> +	/* Calculate the time since last counter resetting to 0 */
> +	reset_ns = mul_u64_u64_div_u64(new_epoch, NSEC_PER_SEC, rate);
> +
> +	pr_info("sched_clock: %u bits at %lu%cHz, resolution %lluns, wraps every %lluns, has run %lluns since counter reset\n",
> +		bits, r, r_unit, res, wrap, reset_ns);
>  
>  	/* Enable IRQ time accounting if we have a fast enough sched_clock() */
>  	if (irqtime > 0 || (irqtime == -1 && rate >= 1000000))
> 

Which is what people have done locally since the beginning of times.
And the only time this is actually useful is when bringing up new
HW/FW that is broken.

So why the need to put that in an upstream kernel, instead of being as
part of your debug toolbox? I don't think the kernel shouldn't be the
recipient of this sort of stuff.

	M.

-- 
Without deviation from the norm, progress is not possible.

On Mon, Sep 07, 2026 at 10:37:03AM +0100, Marc Zyngier wrote:
[...]
> > -	pr_info("sched_clock: %u bits at %lu%cHz, resolution %lluns, wraps every %lluns\n",
> > -		bits, r, r_unit, res, wrap);
> > +	/* Calculate the time since last counter resetting to 0 */
> > +	reset_ns = mul_u64_u64_div_u64(new_epoch, NSEC_PER_SEC, rate);
> > +
> > +	pr_info("sched_clock: %u bits at %lu%cHz, resolution %lluns, wraps every %lluns, has run %lluns since counter reset\n",
> > +		bits, r, r_unit, res, wrap, reset_ns);
> >  
> >  	/* Enable IRQ time accounting if we have a fast enough sched_clock() */
> >  	if (irqtime > 0 || (irqtime == -1 && rate >= 1000000))
> > 
> 
> Which is what people have done locally since the beginning of times.
> And the only time this is actually useful is when bringing up new
> HW/FW that is broken.
> 
> So why the need to put that in an upstream kernel, instead of being as
> part of your debug toolbox? I don't think the kernel shouldn't be the
> recipient of this sort of stuff.

I see your point now, thanks!

- Feng

> 
> 	M.
> 
> -- 
> Without deviation from the norm, progress is not possible.

On Wed, 02 Sep 2026 09:21:23 +0100,
Feng Tang <feng.tang@linux.alibaba.com> wrote:
> 
> Currently sched_clock shows the relative time to the boot starting of
> kernel, while there could be long firmware start time before it and
> after the hardware reset.
> 
> On modern server platforms, there could be several software running in
> parallel. Like for arm64, it could have  SCP (System Control Processor)
> firmware running on SCP processor, and ATF (Arm Trusted Firmware) and
> Linux OS on the main processor.
> 
> Debugging some nasty issues on these platform may need to cross-check
> the logs from these firmwares and Linux kernel for specific events,
> where a unified reference timeline is critical. All these software can
> read the hardware timer, which is also the base of sched_clock for
> Linux kernel. Using the absolute counter since hardware timer reset
> makes it possible for all kinds of software to have a same time base.
> 
> Add 'abs_sched_clock' parameter to provide an option for using absolute
> counter, and users should make sure their sched_clock (hardware timer)
> is capable of supporting absolute counter before enabling the option.
> 
> Locally, it did help on chasing some RAS issues which needed cooperation
> between kernel, SCP firmware and ATF, by mapping the actions from each
> players into one timeline based on the timestamps in their logs.

I really have to ask: why isn't this just a one-off sampling of the
counter, kept in some user accessible location (debugfs or something
else), and ultimately post-processed to align your logs? People have
been doing this... forever, and that has been "good enough" so far.

The other thing is that your "absolute" clock isn't absolute at
all. This doesn't consider SW running at EL2 that could happily offset
thing by an arbitrary value.

I appreciate this is not what your case, but I'm somewhat reluctant to
burden the kernel with something that is, by definition, unreliable.

Thanks,

	M.

-- 
Jazz isn't dead. It just smells funny.

On Wed, Sep 02, 2026 at 04:32:25PM +0100, Marc Zyngier wrote:
> On Wed, 02 Sep 2026 09:21:23 +0100,
> Feng Tang <feng.tang@linux.alibaba.com> wrote:
> > 
> > Currently sched_clock shows the relative time to the boot starting of
> > kernel, while there could be long firmware start time before it and
> > after the hardware reset.
> > 
> > On modern server platforms, there could be several software running in
> > parallel. Like for arm64, it could have  SCP (System Control Processor)
> > firmware running on SCP processor, and ATF (Arm Trusted Firmware) and
> > Linux OS on the main processor.
> > 
> > Debugging some nasty issues on these platform may need to cross-check
> > the logs from these firmwares and Linux kernel for specific events,
> > where a unified reference timeline is critical. All these software can
> > read the hardware timer, which is also the base of sched_clock for
> > Linux kernel. Using the absolute counter since hardware timer reset
> > makes it possible for all kinds of software to have a same time base.
> > 
> > Add 'abs_sched_clock' parameter to provide an option for using absolute
> > counter, and users should make sure their sched_clock (hardware timer)
> > is capable of supporting absolute counter before enabling the option.
> > 
> > Locally, it did help on chasing some RAS issues which needed cooperation
> > between kernel, SCP firmware and ATF, by mapping the actions from each
> > players into one timeline based on the timestamps in their logs.

Hi Marc,

Thanks for the great inputs!

I forgot to emphasized that this is only a debug option, mostly for bug
chasing. 
 
> I really have to ask: why isn't this just a one-off sampling of the
> counter, kept in some user accessible location (debugfs or something
> else), and ultimately post-processed to align your logs? People have
> been doing this... forever, and that has been "good enough" so far.

This option is for chasing nasty bugs (involving SCP/ATF), which could
be panic during boot, so debugfs may not work. But yes, we can achieve
this by printing this out in dmesg.

I agree the post-processing logs works, while this option could be more
convenient, for kernel developers, and firmware developers too :)

> The other thing is that your "absolute" clock isn't absolute at
> all. This doesn't consider SW running at EL2 that could happily offset
> thing by an arbitrary value.

Good point! It may not work for guest environment. And the main purpose
is to debug baremetal issues, which need to cross-check logs from
SCP, ATF and kernel.
 
> I appreciate this is not what your case, but I'm somewhat reluctant to
> burden the kernel with something that is, by definition, unreliable.

You are right! There are all kinds of hardware timers out there, many
of which have various issues, and 'unreliable' for this usage.

The needed features I can think of for a HW timer to use absolute
counter are:
* always running, won't stop on entering cpuidle or system suspend
* won't change frequency on cpufreq change
* the wrap period is big enough

I thought about using CLOCK_SOURCE_SUSPEND_NONSTOP to do a capability
check, but it's not available in sched_clock.c.

So I mentioned in commit log that users enabling this option should
make sure the HW counter is capable.

btw, fwiw, Sashiko also gave some good comments, like I shouldn't use
cyc_to_ns() for calculating the epoch_ns, which should be solved by
using mul_u64_u64_div_u64()

Thanks,
Feng

> 
> Thanks,
> 
> 	M.
> 
> -- 
> Jazz isn't dead. It just smells funny.

On Wed, Sep 02, 2026 at 04:32:25PM +0800, Marc Zyngier wrote:
> On Wed, 02 Sep 2026 09:21:23 +0100,
> Feng Tang <feng.tang@linux.alibaba.com> wrote:
> >
> > Currently sched_clock shows the relative time to the boot starting of
> > kernel, while there could be long firmware start time before it and
> > after the hardware reset.
> >
> > On modern server platforms, there could be several software running in
> > parallel. Like for arm64, it could have  SCP (System Control Processor)
> > firmware running on SCP processor, and ATF (Arm Trusted Firmware) and
> > Linux OS on the main processor.
> >
> > Debugging some nasty issues on these platform may need to cross-check
> > the logs from these firmwares and Linux kernel for specific events,
> > where a unified reference timeline is critical. All these software can
> > read the hardware timer, which is also the base of sched_clock for
> > Linux kernel. Using the absolute counter since hardware timer reset
> > makes it possible for all kinds of software to have a same time base.
> >
> > Add 'abs_sched_clock' parameter to provide an option for using absolute
> > counter, and users should make sure their sched_clock (hardware timer)
> > is capable of supporting absolute counter before enabling the option.
> >
> > Locally, it did help on chasing some RAS issues which needed cooperation
> > between kernel, SCP firmware and ATF, by mapping the actions from each
> > players into one timeline based on the timestamps in their logs.
>
> I really have to ask: why isn't this just a one-off sampling of the
> counter, kept in some user accessible location (debugfs or something
> else), and ultimately post-processed to align your logs? People have
> been doing this... forever, and that has been "good enough" so far.
>

Hi Marc,

> The other thing is that your "absolute" clock isn't absolute at
> all. This doesn't consider SW running at EL2 that could happily offset
> thing by an arbitrary value.

Do you mean the VM case that VM's vcounter can be changed by hypervisor
in EL2, Thus it's not that absolute in such scenario ?

>
> I appreciate this is not what your case, but I'm somewhat reluctant to
> burden the kernel with something that is, by definition, unreliable.
>
> Thanks,
>
> 	M.
>
> --
> Jazz isn't dead. It just smells funny.

On Thu, 03 Sep 2026 07:51:03 +0100,
Yao Yuan <yaoyuan@linux.alibaba.com> wrote:
> 
> On Wed, Sep 02, 2026 at 04:32:25PM +0800, Marc Zyngier wrote:
> > On Wed, 02 Sep 2026 09:21:23 +0100,
> > Feng Tang <feng.tang@linux.alibaba.com> wrote:
> > >
> > > Currently sched_clock shows the relative time to the boot starting of
> > > kernel, while there could be long firmware start time before it and
> > > after the hardware reset.
> > >
> > > On modern server platforms, there could be several software running in
> > > parallel. Like for arm64, it could have  SCP (System Control Processor)
> > > firmware running on SCP processor, and ATF (Arm Trusted Firmware) and
> > > Linux OS on the main processor.
> > >
> > > Debugging some nasty issues on these platform may need to cross-check
> > > the logs from these firmwares and Linux kernel for specific events,
> > > where a unified reference timeline is critical. All these software can
> > > read the hardware timer, which is also the base of sched_clock for
> > > Linux kernel. Using the absolute counter since hardware timer reset
> > > makes it possible for all kinds of software to have a same time base.
> > >
> > > Add 'abs_sched_clock' parameter to provide an option for using absolute
> > > counter, and users should make sure their sched_clock (hardware timer)
> > > is capable of supporting absolute counter before enabling the option.
> > >
> > > Locally, it did help on chasing some RAS issues which needed cooperation
> > > between kernel, SCP firmware and ATF, by mapping the actions from each
> > > players into one timeline based on the timestamps in their logs.
> >
> > I really have to ask: why isn't this just a one-off sampling of the
> > counter, kept in some user accessible location (debugfs or something
> > else), and ultimately post-processed to align your logs? People have
> > been doing this... forever, and that has been "good enough" so far.
> >
> 
> Hi Marc,
> 
> > The other thing is that your "absolute" clock isn't absolute at
> > all. This doesn't consider SW running at EL2 that could happily offset
> > thing by an arbitrary value.
> 
> Do you mean the VM case that VM's vcounter can be changed by hypervisor
> in EL2, Thus it's not that absolute in such scenario ?

That's indeed one of the possibilities. EL2 controls both virtual and
physical offsets, and therefore provides the kernel with a different
view of time.

This doesn't even have to be a VM. There is a lot of non-hypervisor SW
out there that just hogs EL2 for more or less nefarious purposes (such
as "protecting" the kernel), and offsetting the counter values is one
of thing they could do to hide what they are doing.

The other thing is that this change seems to break the sched_clock()
handover, since the new clock doesn't start where the old one ends.
This doesn't affect arm64, which can only have one true source of
time, but other archs would probably suffer from this.

	M.

-- 
Jazz isn't dead. It just smells funny.

On Thu, Sep 03, 2026 at 08:38:57AM +0800, Marc Zyngier wrote:
> On Thu, 03 Sep 2026 07:51:03 +0100,
> Yao Yuan <yaoyuan@linux.alibaba.com> wrote:
> >
> > On Wed, Sep 02, 2026 at 04:32:25PM +0800, Marc Zyngier wrote:
> > > On Wed, 02 Sep 2026 09:21:23 +0100,
> > > Feng Tang <feng.tang@linux.alibaba.com> wrote:
> > > >
> > > > Currently sched_clock shows the relative time to the boot starting of
> > > > kernel, while there could be long firmware start time before it and
> > > > after the hardware reset.
> > > >
> > > > On modern server platforms, there could be several software running in
> > > > parallel. Like for arm64, it could have  SCP (System Control Processor)
> > > > firmware running on SCP processor, and ATF (Arm Trusted Firmware) and
> > > > Linux OS on the main processor.
> > > >
> > > > Debugging some nasty issues on these platform may need to cross-check
> > > > the logs from these firmwares and Linux kernel for specific events,
> > > > where a unified reference timeline is critical. All these software can
> > > > read the hardware timer, which is also the base of sched_clock for
> > > > Linux kernel. Using the absolute counter since hardware timer reset
> > > > makes it possible for all kinds of software to have a same time base.
> > > >
> > > > Add 'abs_sched_clock' parameter to provide an option for using absolute
> > > > counter, and users should make sure their sched_clock (hardware timer)
> > > > is capable of supporting absolute counter before enabling the option.
> > > >
> > > > Locally, it did help on chasing some RAS issues which needed cooperation
> > > > between kernel, SCP firmware and ATF, by mapping the actions from each
> > > > players into one timeline based on the timestamps in their logs.
> > >
> > > I really have to ask: why isn't this just a one-off sampling of the
> > > counter, kept in some user accessible location (debugfs or something
> > > else), and ultimately post-processed to align your logs? People have
> > > been doing this... forever, and that has been "good enough" so far.
> > >
> >
> > Hi Marc,
> >
> > > The other thing is that your "absolute" clock isn't absolute at
> > > all. This doesn't consider SW running at EL2 that could happily offset
> > > thing by an arbitrary value.
> >
> > Do you mean the VM case that VM's vcounter can be changed by hypervisor
> > in EL2, Thus it's not that absolute in such scenario ?
>
> That's indeed one of the possibilities. EL2 controls both virtual and
> physical offsets, and therefore provides the kernel with a different
> view of time.
>
> This doesn't even have to be a VM. There is a lot of non-hypervisor SW
> out there that just hogs EL2 for more or less nefarious purposes (such
> as "protecting" the kernel), and offsetting the counter values is one
> of thing they could do to hide what they are doing.

Thanks for the details !
I see, this yet like some hypervisor hide its handling time
from guest to me :-)

>
> The other thing is that this change seems to break the sched_clock()
> handover, since the new clock doesn't start where the old one ends.
> This doesn't affect arm64, which can only have one true source of
> time, but other archs would probably suffer from this.

Yes, we can discuss this more in Feng's reply, thanks!

>
> 	M.
>
> --
> Jazz isn't dead. It just smells funny.

On Thu, Sep 03, 2026 at 08:38:57AM +0100, Marc Zyngier wrote:
[...]
> > Hi Marc,
> > 
> > > The other thing is that your "absolute" clock isn't absolute at
> > > all. This doesn't consider SW running at EL2 that could happily offset
> > > thing by an arbitrary value.
> > 
> > Do you mean the VM case that VM's vcounter can be changed by hypervisor
> > in EL2, Thus it's not that absolute in such scenario ?
> 
> That's indeed one of the possibilities. EL2 controls both virtual and
> physical offsets, and therefore provides the kernel with a different
> view of time.
> 
> This doesn't even have to be a VM. There is a lot of non-hypervisor SW
> out there that just hogs EL2 for more or less nefarious purposes (such
> as "protecting" the kernel), and offsetting the counter values is one
> of thing they could do to hide what they are doing.

Thanks for the info.

Manipulating time counter(sched_clock) runtimely doesn't sound like a
good thing. There used to be similar things happened for TSC on x86
platforms, which Thomas has mentioned several times :)

> 
> The other thing is that this change seems to break the sched_clock()
> handover, since the new clock doesn't start where the old one ends.
> This doesn't affect arm64, which can only have one true source of
> time, but other archs would probably suffer from this.

Yes, I has similar concern. And for sched_clock, it usually should be
done by a very low read cost, high precison, and better per-cpu timer.
I only have some knowledge about x86 and arm64, and don't know whether
there is other architecture that really switches sched_clock .

Thanks,
Feng