ktime_mono_to_any() only fetches the offset inside the loop. This is a
single word on 64-bit hosts, and seqcount_read_begin() implies a full
SMP barrier.

When BITS_PER_LONG == 64, use READ_ONCE to fetch the offset instead of
doing a seqcount loop. This means that we also need to use WRITE_ONCE to
update the offsets in tk_set_wall_to_mono() and tk_update_sleep_time().

Cc: Vadim Fedorenko <vadim.fedorenko@linux.dev>
Signed-off-by: Jeff Layton <jlayton@kernel.org>
---
This one puts back the READ_ONCE and adds WRITE_ONCEs (as Thomas
suggested).
---
Changes in v3:
- add back the READ_ONCE, and use WRITE_ONCE to update the offsets
- Link to v2: https://lore.kernel.org/r/20240910-mgtime-v2-1-e96826ac56f0@kernel.org

Changes in v2:
- drop the READ_ONCE
- clean up changelog
- Link to v1: https://lore.kernel.org/r/20240910-mgtime-v1-1-35fb64bd0af5@kernel.org
---
 kernel/time/timekeeping.c | 18 +++++++++++++++---
 1 file changed, 15 insertions(+), 3 deletions(-)

diff --git a/kernel/time/timekeeping.c b/kernel/time/timekeeping.c
index 99381e04a871..ec999a39d9cb 100644
--- a/kernel/time/timekeeping.c
+++ b/kernel/time/timekeeping.c
@@ -161,13 +161,15 @@ static void tk_set_wall_to_mono(struct timekeeper *tk, struct timespec64 wtm)
 	WARN_ON_ONCE(tk->offs_real != timespec64_to_ktime(tmp));
 	tk->wall_to_monotonic = wtm;
 	set_normalized_timespec64(&tmp, -wtm.tv_sec, -wtm.tv_nsec);
-	tk->offs_real = timespec64_to_ktime(tmp);
-	tk->offs_tai = ktime_add(tk->offs_real, ktime_set(tk->tai_offset, 0));
+	/* paired with READ_ONCE in ktime_mono_to_any */
+	WRITE_ONCE(tk->offs_real, timespec64_to_ktime(tmp));
+	WRITE_ONCE(tk->offs_tai, ktime_add(tk->offs_real, ktime_set(tk->tai_offset, 0)));
 }
 
 static inline void tk_update_sleep_time(struct timekeeper *tk, ktime_t delta)
 {
-	tk->offs_boot = ktime_add(tk->offs_boot, delta);
+	/* paired with READ_ONCE in ktime_mono_to_any */
+	WRITE_ONCE(tk->offs_boot, ktime_add(tk->offs_boot, delta));
 	/*
 	 * Timespec representation for VDSO update to avoid 64bit division
 	 * on every update.
@@ -954,6 +956,15 @@ EXPORT_SYMBOL_GPL(ktime_get_coarse_with_floor_and_offset);
  * @tmono:	time to convert.
  * @offs:	which offset to use
  */
+#if BITS_PER_LONG == 64
+ktime_t ktime_mono_to_any(ktime_t tmono, enum tk_offsets offs)
+{
+	ktime_t *offset = offsets[offs];
+
+	/* paired with WRITE_ONCEs in tk_set_wall_to_mono and tk_update_sleep_time */
+	return ktime_add(tmono, READ_ONCE(*offset));
+}
+#else /* BITS_PER_LONG == 64 */
 ktime_t ktime_mono_to_any(ktime_t tmono, enum tk_offsets offs)
 {
 	ktime_t *offset = offsets[offs];
@@ -967,6 +978,7 @@ ktime_t ktime_mono_to_any(ktime_t tmono, enum tk_offsets offs)
 
 	return tconv;
 }
+#endif /* BITS_PER_LONG == 64 */
 EXPORT_SYMBOL_GPL(ktime_mono_to_any);
 
 /**

---
base-commit: a833754dba0fcc8984e8e86042a8877be70187d9
change-id: 20240910-mgtime-731eace7cca5

Best regards,
-- 
Jeff Layton <jlayton@kernel.org>

The following commit has been merged into the timers/core branch of tip:

Commit-ID:     8c111f1b967687f47bb0cfbedf2863b62c23223c
Gitweb:        https://git.kernel.org/tip/8c111f1b967687f47bb0cfbedf2863b62c23223c
Author:        Jeff Layton <jlayton@kernel.org>
AuthorDate:    Tue, 10 Sep 2024 13:43:34 -04:00
Committer:     Thomas Gleixner <tglx@linutronix.de>
CommitterDate: Wed, 02 Oct 2024 18:06:03 +02:00

timekeeping: Don't use seqcount loop in ktime_mono_to_any() on 64-bit systems

ktime_mono_to_any() only fetches the offset inside the loop. This is a
single word on 64-bit CPUs, and seqcount_read_begin() implies a full SMP
barrier.

Use READ_ONCE() to fetch the offset instead of doing a seqcount loop on
64-bit and add the matching WRITE_ONCE()'s to update the offsets in
tk_set_wall_to_mono() and tk_update_sleep_time().

[ tglx: Get rid of the #ifdeffery ]

Signed-off-by: Jeff Layton <jlayton@kernel.org>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Link: https://lore.kernel.org/all/20240910-mgtime-v3-1-84406ed53fad@kernel.org
---
 kernel/time/timekeeping.c | 16 +++++++++++++---
 1 file changed, 13 insertions(+), 3 deletions(-)

diff --git a/kernel/time/timekeeping.c b/kernel/time/timekeeping.c
index 47e44b9..a57f2ee 100644
--- a/kernel/time/timekeeping.c
+++ b/kernel/time/timekeeping.c
@@ -161,13 +161,15 @@ static void tk_set_wall_to_mono(struct timekeeper *tk, struct timespec64 wtm)
 	WARN_ON_ONCE(tk->offs_real != timespec64_to_ktime(tmp));
 	tk->wall_to_monotonic = wtm;
 	set_normalized_timespec64(&tmp, -wtm.tv_sec, -wtm.tv_nsec);
-	tk->offs_real = timespec64_to_ktime(tmp);
-	tk->offs_tai = ktime_add(tk->offs_real, ktime_set(tk->tai_offset, 0));
+	/* Paired with READ_ONCE() in ktime_mono_to_any() */
+	WRITE_ONCE(tk->offs_real, timespec64_to_ktime(tmp));
+	WRITE_ONCE(tk->offs_tai, ktime_add(tk->offs_real, ktime_set(tk->tai_offset, 0)));
 }
 
 static inline void tk_update_sleep_time(struct timekeeper *tk, ktime_t delta)
 {
-	tk->offs_boot = ktime_add(tk->offs_boot, delta);
+	/* Paired with READ_ONCE() in ktime_mono_to_any() */
+	WRITE_ONCE(tk->offs_boot, ktime_add(tk->offs_boot, delta));
 	/*
 	 * Timespec representation for VDSO update to avoid 64bit division
 	 * on every update.
@@ -930,6 +932,14 @@ ktime_t ktime_mono_to_any(ktime_t tmono, enum tk_offsets offs)
 	unsigned int seq;
 	ktime_t tconv;
 
+	if (IS_ENABLED(CONFIG_64BIT)) {
+		/*
+		 * Paired with WRITE_ONCE()s in tk_set_wall_to_mono() and
+		 * tk_update_sleep_time().
+		 */
+		return ktime_add(tmono, READ_ONCE(*offset));
+	}
+
 	do {
 		seq = read_seqcount_begin(&tk_core.seq);
 		tconv = ktime_add(tmono, *offset);