From nobody Sat Feb 7 11:31:10 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6F54E15ECC4; Mon, 25 Mar 2024 06:40:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.9 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711348847; cv=none; b=KI1I8WUf4WKZPSdZffNrBsbaw/yp+MY0uOjlknd4DJgI2W5jFO4Q8uTSfpDO8wYzXOoHE1PjmkaTi4A3w0jLjiSJFVu0m5oZjvdpPtN2Hg0YOqlwWiFzCiE+oX7WJir6qss02KsFmuPFbx7aeWq6rvtrWx9yKf6SlHQuABvgaV8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711348847; c=relaxed/simple; bh=+W4K6lkGmf5ZfckuHCgiOpf7pq6FEiJVqi6VEtd9uvE=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=rktLBeQbNy69mp6URDqkRJ4TvicxePJM219GZuiDtsqP+ecA2yAdSYncteaOPPugErakm6AbA+BlN7jEWtn8VuPelrl+EzUgmtW+rhtZSQzUr3KLTh21MKGXg8WCUushXJWsYSWDcKHqtqBMxYFmqUkwW3iUp6krJdXsJxdOzmo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=KGKbAahb; arc=none smtp.client-ip=192.198.163.9 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="KGKbAahb" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1711348845; x=1742884845; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=+W4K6lkGmf5ZfckuHCgiOpf7pq6FEiJVqi6VEtd9uvE=; b=KGKbAahbFLATVQf1u4Ezo8PQ0p3VdtTfjfNCYeb4sLw8h7QV1lAlmVB7 jW+46FLMtggeW6LioUmNXmKySMLavtCGyZrS3hW5BTL1WW23VlAXnxhY4 wqn4zgZgzUMGJ2aB8HaeYvEJLsofMgm1en7Q7akDd+bqR3kCF5YU39qXx kK+AfWCI8b9h620b7C0eSsM+kXRKtheOz4J3OwUrNzlHYv3k0CCTUDbYt COvw1cnzIrNNDgw5yf0Yileb/Zfv8IFV0c/xPDXcyrOrTp4J5dmTaiGKu OJWikf4GtCsgR1fBRbbuLxs2B+A/IAJ6z6fxXxEUdFwWmX1yV9zRFlFUH A==; X-IronPort-AV: E=McAfee;i="6600,9927,11023"; a="17065072" X-IronPort-AV: E=Sophos;i="6.07,152,1708416000"; d="scan'208";a="17065072" Received: from fmviesa002.fm.intel.com ([10.60.135.142]) by fmvoesa103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Mar 2024 23:40:45 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,152,1708416000"; d="scan'208";a="38629565" Received: from ahunter6-mobl1.ger.corp.intel.com (HELO ahunter-VirtualBox.home\044ger.corp.intel.com) ([10.251.211.155]) by fmviesa002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Mar 2024 23:40:38 -0700 From: Adrian Hunter To: Thomas Gleixner Cc: Michael Ellerman , Nicholas Piggin , Christophe Leroy , "Aneesh Kumar K.V" , "Naveen N. Rao" , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Andy Lutomirski , Vincenzo Frascino , John Stultz , Stephen Boyd , Peter Zijlstra , Randy Dunlap , Bjorn Helgaas , Arnd Bergmann , Anna-Maria Behnsen , linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, linux-s390@vger.kernel.org Subject: [PATCH V2 01/19] vdso: Consolidate vdso_calc_delta() Date: Mon, 25 Mar 2024 08:40:05 +0200 Message-Id: <20240325064023.2997-2-adrian.hunter@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240325064023.2997-1-adrian.hunter@intel.com> References: <20240325064023.2997-1-adrian.hunter@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Organization: Intel Finland Oy, Registered Address: PL 281, 00181 Helsinki, Business Identity Code: 0357606 - 4, Domiciled in Helsinki Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Consolidate vdso_calc_delta(), in preparation for further simplification. Suggested-by: Thomas Gleixner Signed-off-by: Adrian Hunter --- Changes in V2: Keep powerpc comment about mask Move ifdef out of function arch/powerpc/include/asm/vdso/gettimeofday.h | 26 +++++++++----------- arch/s390/include/asm/vdso/gettimeofday.h | 7 ++---- lib/vdso/gettimeofday.c | 9 ++++++- 3 files changed, 21 insertions(+), 21 deletions(-) diff --git a/arch/powerpc/include/asm/vdso/gettimeofday.h b/arch/powerpc/in= clude/asm/vdso/gettimeofday.h index f0a4cf01e85c..ac21a2a0c2f9 100644 --- a/arch/powerpc/include/asm/vdso/gettimeofday.h +++ b/arch/powerpc/include/asm/vdso/gettimeofday.h @@ -14,6 +14,17 @@ =20 #define VDSO_HAS_TIME 1 =20 +/* + * powerpc specific delta calculation. + * + * This variant removes the masking of the subtraction because the + * clocksource mask of all VDSO capable clocksources on powerpc is U64_MAX + * which would result in a pointless operation. The compiler cannot + * optimize it away as the mask comes from the vdso data and is not compile + * time constant. + */ +#define VDSO_DELTA_NOMASK 1 + static __always_inline int do_syscall_2(const unsigned long _r0, const uns= igned long _r3, const unsigned long _r4) { @@ -105,21 +116,6 @@ static inline bool vdso_clocksource_ok(const struct vd= so_data *vd) } #define vdso_clocksource_ok vdso_clocksource_ok =20 -/* - * powerpc specific delta calculation. - * - * This variant removes the masking of the subtraction because the - * clocksource mask of all VDSO capable clocksources on powerpc is U64_MAX - * which would result in a pointless operation. The compiler cannot - * optimize it away as the mask comes from the vdso data and is not compile - * time constant. - */ -static __always_inline u64 vdso_calc_delta(u64 cycles, u64 last, u64 mask,= u32 mult) -{ - return (cycles - last) * mult; -} -#define vdso_calc_delta vdso_calc_delta - #ifndef __powerpc64__ static __always_inline u64 vdso_shift_ns(u64 ns, unsigned long shift) { diff --git a/arch/s390/include/asm/vdso/gettimeofday.h b/arch/s390/include/= asm/vdso/gettimeofday.h index db84942eb78f..7937765ccfa5 100644 --- a/arch/s390/include/asm/vdso/gettimeofday.h +++ b/arch/s390/include/asm/vdso/gettimeofday.h @@ -6,16 +6,13 @@ =20 #define VDSO_HAS_CLOCK_GETRES 1 =20 +#define VDSO_DELTA_NOMASK 1 + #include #include #include #include =20 -#define vdso_calc_delta __arch_vdso_calc_delta -static __always_inline u64 __arch_vdso_calc_delta(u64 cycles, u64 last, u6= 4 mask, u32 mult) -{ - return (cycles - last) * mult; -} =20 static __always_inline const struct vdso_data *__arch_get_vdso_data(void) { diff --git a/lib/vdso/gettimeofday.c b/lib/vdso/gettimeofday.c index ce2f69552003..faccf12f7c03 100644 --- a/lib/vdso/gettimeofday.c +++ b/lib/vdso/gettimeofday.c @@ -6,6 +6,13 @@ #include =20 #ifndef vdso_calc_delta + +#ifdef VDSO_DELTA_NOMASK +# define VDSO_DELTA_MASK(mask) U64_MAX +#else +# define VDSO_DELTA_MASK(mask) (mask) +#endif + /* * Default implementation which works for all sane clocksources. That * obviously excludes x86/TSC. @@ -13,7 +20,7 @@ static __always_inline u64 vdso_calc_delta(u64 cycles, u64 last, u64 mask, u32 mult) { - return ((cycles - last) & mask) * mult; + return ((cycles - last) & VDSO_DELTA_MASK(mask)) * mult; } #endif =20 --=20 2.34.1 From nobody Sat Feb 7 11:31:10 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5A66B15ECD6; Mon, 25 Mar 2024 06:40:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.9 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711348854; cv=none; b=FxTHgS6VRnZ8L2R/oqwF7rXu0B5k/NTm9DatsirF3sxFnTYoo4YOzcwcDK11N3lodlR6OXNbLn/oQlzFkF4X8ZqENo8Z67d+zbc0QJd2tCECDxqgiAtO6487DG2++v4IJ9/NTMDcGUaKaQXqLxWBjjydfQcugLq59VweTymfm0I= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711348854; c=relaxed/simple; bh=yWOijviXmOaJVq1Hs5fWW3GVEDdadkoX8Z1Ik5RcbIA=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=dbot4E4l2bQ8xhYucSwoccVwv1h803CJte/dA8AXXaNTemlb/CdAmR83vJdQa9JUov+QfKHk6ccGp/vs8nEr/QVMUpyhVVre0YxnuaGUrxyT21iPxu3mhMv3b4PJLDJHNBoMQU+dzj/HnTQFAhWZABNAChV2dCR/m+EeZzFnsBg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=nGShMZX+; arc=none smtp.client-ip=192.198.163.9 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="nGShMZX+" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1711348852; x=1742884852; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=yWOijviXmOaJVq1Hs5fWW3GVEDdadkoX8Z1Ik5RcbIA=; b=nGShMZX+RrYjy9j/8tcpH1ty7snay4MpVZ00fdz6ste2FvMQWFi7D+a9 0SEBR8sJR+TPHYPx3J2ssJ8MU/n8Y1czkIM3It4PMDa0iRsA9YIO+A4EZ pwEOVLSL6MfBniTL9oLHZ09J2rZbrZ2Aq+B803TlXnN+jEbQuCEg3XRiu LIl+xn1AzogewDDD6A/r3EFC3nlKDyuqAqxlGpVd5wm2nok/bWp+t8UHg IAcxk62frQrBQRFH9EPf/WzUuAt8UO6K6P1alJ2qT+YsxiVEIM5VPIZ1R /g9JqKIQt5+8TIIX2NwNlbmYbT+S8s4G3v0wlrUayrSVPKelNEIW8gM+f w==; X-IronPort-AV: E=McAfee;i="6600,9927,11023"; a="17065100" X-IronPort-AV: E=Sophos;i="6.07,152,1708416000"; d="scan'208";a="17065100" Received: from fmviesa002.fm.intel.com ([10.60.135.142]) by fmvoesa103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Mar 2024 23:40:51 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,152,1708416000"; d="scan'208";a="38629571" Received: from ahunter6-mobl1.ger.corp.intel.com (HELO ahunter-VirtualBox.home\044ger.corp.intel.com) ([10.251.211.155]) by fmviesa002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Mar 2024 23:40:45 -0700 From: Adrian Hunter To: Thomas Gleixner Cc: Michael Ellerman , Nicholas Piggin , Christophe Leroy , "Aneesh Kumar K.V" , "Naveen N. Rao" , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Andy Lutomirski , Vincenzo Frascino , John Stultz , Stephen Boyd , Peter Zijlstra , Randy Dunlap , Bjorn Helgaas , Arnd Bergmann , Anna-Maria Behnsen , linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, linux-s390@vger.kernel.org Subject: [PATCH V2 02/19] vdso: Consolidate nanoseconds calculation Date: Mon, 25 Mar 2024 08:40:06 +0200 Message-Id: <20240325064023.2997-3-adrian.hunter@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240325064023.2997-1-adrian.hunter@intel.com> References: <20240325064023.2997-1-adrian.hunter@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Organization: Intel Finland Oy, Registered Address: PL 281, 00181 Helsinki, Business Identity Code: 0357606 - 4, Domiciled in Helsinki Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Consolidate nanoseconds calculation to simplify and reduce code duplication. Suggested-by: Thomas Gleixner Signed-off-by: Adrian Hunter --- Changes in V2: Adjusted due to changes in "vdso: Consolidate vdso_calc_delta()" arch/x86/include/asm/vdso/gettimeofday.h | 17 +++++---- lib/vdso/gettimeofday.c | 45 +++++++++++------------- 2 files changed, 28 insertions(+), 34 deletions(-) diff --git a/arch/x86/include/asm/vdso/gettimeofday.h b/arch/x86/include/as= m/vdso/gettimeofday.h index 8e048ca980df..5727dedd3549 100644 --- a/arch/x86/include/asm/vdso/gettimeofday.h +++ b/arch/x86/include/asm/vdso/gettimeofday.h @@ -300,7 +300,7 @@ static inline bool arch_vdso_cycles_ok(u64 cycles) #define vdso_cycles_ok arch_vdso_cycles_ok =20 /* - * x86 specific delta calculation. + * x86 specific calculation of nanoseconds for the current cycle count * * The regular implementation assumes that clocksource reads are globally * monotonic. The TSC can be slightly off across sockets which can cause @@ -308,8 +308,8 @@ static inline bool arch_vdso_cycles_ok(u64 cycles) * jump. * * Therefore it needs to be verified that @cycles are greater than - * @last. If not then use @last, which is the base time of the current - * conversion period. + * @vd->cycles_last. If not then use @vd->cycles_last, which is the base + * time of the current conversion period. * * This variant also uses a custom mask because while the clocksource mask= of * all the VDSO capable clocksources on x86 is U64_MAX, the above code uses @@ -317,25 +317,24 @@ static inline bool arch_vdso_cycles_ok(u64 cycles) * declares everything with the MSB/Sign-bit set as invalid. Therefore the * effective mask is S64_MAX. */ -static __always_inline -u64 vdso_calc_delta(u64 cycles, u64 last, u64 mask, u32 mult) +static __always_inline u64 vdso_calc_ns(const struct vdso_data *vd, u64 cy= cles, u64 base) { /* * Due to the MSB/Sign-bit being used as invalid marker (see * arch_vdso_cycles_valid() above), the effective mask is S64_MAX. */ - u64 delta =3D (cycles - last) & S64_MAX; + u64 delta =3D (cycles - vd->cycle_last) & S64_MAX; =20 /* * Due to the above mentioned TSC wobbles, filter out negative motion. * Per the above masking, the effective sign bit is now bit 62. */ if (unlikely(delta & (1ULL << 62))) - return 0; + return base >> vd->shift; =20 - return delta * mult; + return ((delta * vd->mult) + base) >> vd->shift; } -#define vdso_calc_delta vdso_calc_delta +#define vdso_calc_ns vdso_calc_ns =20 #endif /* !__ASSEMBLY__ */ =20 diff --git a/lib/vdso/gettimeofday.c b/lib/vdso/gettimeofday.c index faccf12f7c03..9fa90e0794c9 100644 --- a/lib/vdso/gettimeofday.c +++ b/lib/vdso/gettimeofday.c @@ -5,23 +5,12 @@ #include #include =20 -#ifndef vdso_calc_delta +#ifndef vdso_calc_ns =20 #ifdef VDSO_DELTA_NOMASK -# define VDSO_DELTA_MASK(mask) U64_MAX +# define VDSO_DELTA_MASK(vd) U64_MAX #else -# define VDSO_DELTA_MASK(mask) (mask) -#endif - -/* - * Default implementation which works for all sane clocksources. That - * obviously excludes x86/TSC. - */ -static __always_inline -u64 vdso_calc_delta(u64 cycles, u64 last, u64 mask, u32 mult) -{ - return ((cycles - last) & VDSO_DELTA_MASK(mask)) * mult; -} +# define VDSO_DELTA_MASK(vd) (vd->mask) #endif =20 #ifndef vdso_shift_ns @@ -31,6 +20,18 @@ static __always_inline u64 vdso_shift_ns(u64 ns, u32 shi= ft) } #endif =20 +/* + * Default implementation which works for all sane clocksources. That + * obviously excludes x86/TSC. + */ +static __always_inline u64 vdso_calc_ns(const struct vdso_data *vd, u64 cy= cles, u64 base) +{ + u64 delta =3D (cycles - vd->cycle_last) & VDSO_DELTA_MASK(vd); + + return vdso_shift_ns((delta * vd->mult) + base, vd->shift); +} +#endif /* vdso_calc_ns */ + #ifndef __arch_vdso_hres_capable static inline bool __arch_vdso_hres_capable(void) { @@ -56,10 +57,10 @@ static inline bool vdso_cycles_ok(u64 cycles) static __always_inline int do_hres_timens(const struct vdso_data *vdns, cl= ockid_t clk, struct __kernel_timespec *ts) { - const struct vdso_data *vd; const struct timens_offset *offs =3D &vdns->offset[clk]; const struct vdso_timestamp *vdso_ts; - u64 cycles, last, ns; + const struct vdso_data *vd; + u64 cycles, ns; u32 seq; s64 sec; =20 @@ -80,10 +81,7 @@ static __always_inline int do_hres_timens(const struct v= dso_data *vdns, clockid_ cycles =3D __arch_get_hw_counter(vd->clock_mode, vd); if (unlikely(!vdso_cycles_ok(cycles))) return -1; - ns =3D vdso_ts->nsec; - last =3D vd->cycle_last; - ns +=3D vdso_calc_delta(cycles, last, vd->mask, vd->mult); - ns =3D vdso_shift_ns(ns, vd->shift); + ns =3D vdso_calc_ns(vd, cycles, vdso_ts->nsec); sec =3D vdso_ts->sec; } while (unlikely(vdso_read_retry(vd, seq))); =20 @@ -118,7 +116,7 @@ static __always_inline int do_hres(const struct vdso_da= ta *vd, clockid_t clk, struct __kernel_timespec *ts) { const struct vdso_timestamp *vdso_ts =3D &vd->basetime[clk]; - u64 cycles, last, sec, ns; + u64 cycles, sec, ns; u32 seq; =20 /* Allows to compile the high resolution parts out */ @@ -151,10 +149,7 @@ static __always_inline int do_hres(const struct vdso_d= ata *vd, clockid_t clk, cycles =3D __arch_get_hw_counter(vd->clock_mode, vd); if (unlikely(!vdso_cycles_ok(cycles))) return -1; - ns =3D vdso_ts->nsec; - last =3D vd->cycle_last; - ns +=3D vdso_calc_delta(cycles, last, vd->mask, vd->mult); - ns =3D vdso_shift_ns(ns, vd->shift); + ns =3D vdso_calc_ns(vd, cycles, vdso_ts->nsec); sec =3D vdso_ts->sec; } while (unlikely(vdso_read_retry(vd, seq))); =20 --=20 2.34.1 From nobody Sat Feb 7 11:31:10 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C7E5415ECCC; Mon, 25 Mar 2024 06:40:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.9 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711348860; cv=none; b=PFQ1l2KdkmU39PbFjzsEAlelshSWacjh6Cce4TXBYRbli2OgrSbvIKngkmgZxQ2Qlcpwe2uMt7SUvKgWuKogzsy0cVQ1TaDw8WEdOJV4Q5qsrsnomB6bYTezIY30bcOQSfWg/viIl4C8c57vIKoyDyUv4XOKFvPKmjAQqTgXPq0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711348860; c=relaxed/simple; bh=bS1S2X2CJjfZI9Wdu/gKC/PtojbjfoojrJoco+YT8K0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=FONK/Qc9G2oRRUFRsK6T4xFLA0OJjRbzn6Tvha7tgxjWP+GqVXo70yhQxf7VFPsoShkolB+SEmDT4tGOntWGsv7bA0aYZOg3A9HeadTglGO/bTCCw30Ia535SNBV2WXYzH/DvSzxMv0kaPI0i1ssQ4kQNuFhOPONP/5lOyBO0ok= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=iCvUAqM+; arc=none smtp.client-ip=192.198.163.9 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="iCvUAqM+" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1711348858; x=1742884858; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=bS1S2X2CJjfZI9Wdu/gKC/PtojbjfoojrJoco+YT8K0=; b=iCvUAqM+P763FFkqvP9fbv69XTONy7MqqfFgwaGSej2T5t05KuRiZri/ HGqhHtZ+8zYRtnnMf/OEQk7iHFCZWOhPSAlt0yyhH/u5ydovpU+lym2Ct BDZ0LWQXMWylXc2l3AAIUXhqhT1zSzACUpRzgxgNGKBnLDtAHGpIv38CH ZVlHp0T2Oj2N9tmaCWN/eYT9lsayUYIBkf6Zz9y0Lsq17KMp4OK2Y/LIi C+UNSAKpaDaXMTe3VizOlPvo5OTBF006bn9sd0eD8edmcoG9LOl7hi/Hz jzgYhRapenVRCnafneDk38xovJOaJ75Z/P7CtKy7ZejDv4CWQgHemnTMA w==; X-IronPort-AV: E=McAfee;i="6600,9927,11023"; a="17065130" X-IronPort-AV: E=Sophos;i="6.07,152,1708416000"; d="scan'208";a="17065130" Received: from fmviesa002.fm.intel.com ([10.60.135.142]) by fmvoesa103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Mar 2024 23:40:58 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,152,1708416000"; d="scan'208";a="38629577" Received: from ahunter6-mobl1.ger.corp.intel.com (HELO ahunter-VirtualBox.home\044ger.corp.intel.com) ([10.251.211.155]) by fmviesa002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Mar 2024 23:40:52 -0700 From: Adrian Hunter To: Thomas Gleixner Cc: Michael Ellerman , Nicholas Piggin , Christophe Leroy , "Aneesh Kumar K.V" , "Naveen N. Rao" , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Andy Lutomirski , Vincenzo Frascino , John Stultz , Stephen Boyd , Peter Zijlstra , Randy Dunlap , Bjorn Helgaas , Arnd Bergmann , Anna-Maria Behnsen , linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, linux-s390@vger.kernel.org Subject: [PATCH V2 03/19] vdso: Add CONFIG_GENERIC_VDSO_OVERFLOW_PROTECT Date: Mon, 25 Mar 2024 08:40:07 +0200 Message-Id: <20240325064023.2997-4-adrian.hunter@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240325064023.2997-1-adrian.hunter@intel.com> References: <20240325064023.2997-1-adrian.hunter@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Organization: Intel Finland Oy, Registered Address: PL 281, 00181 Helsinki, Business Identity Code: 0357606 - 4, Domiciled in Helsinki Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add CONFIG_GENERIC_VDSO_OVERFLOW_PROTECT in preparation to add multiplication overflow protection to the VDSO time getter functions. Suggested-by: Thomas Gleixner Signed-off-by: Adrian Hunter --- lib/vdso/Kconfig | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/lib/vdso/Kconfig b/lib/vdso/Kconfig index d883ac299508..c46c2300517c 100644 --- a/lib/vdso/Kconfig +++ b/lib/vdso/Kconfig @@ -30,4 +30,11 @@ config GENERIC_VDSO_TIME_NS Selected by architectures which support time namespaces in the VDSO =20 +config GENERIC_VDSO_OVERFLOW_PROTECT + bool + help + Select to add multiplication overflow protection to the VDSO + time getter functions for the price of an extra conditional + in the hotpath. + endif --=20 2.34.1 From nobody Sat Feb 7 11:31:10 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0F1E015EFBD; Mon, 25 Mar 2024 06:41:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.9 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711348868; cv=none; b=ZcnbHVy4DJblQuGWmTgSXH5hvjwCiAgX3/AgMAicyFypy7/GTZfuCEY1sbJHCPxVXas1dhpL/iLt54kjeXwAkbUyhcz7UwLBi6K8GPV/+Ko4U605fFjYoqyTC4gevbbBpNOc+sJKPXr9IEEXe+Fs6SGLIvjJ5vMzLD3uElCxrpw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711348868; c=relaxed/simple; bh=xuamuB97LVfxyJSvE4i73CrmF8XOSgafgM3yHCD1nqo=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=WJwgwf42UfYVb9GtzM4Au7b4uD34u/MV4ezOQSM+DdwAfLzX1LFUfqDlQWE88Bv+g8KYEGmYoq8cLfaHoKFkCPURqB4vqt5RB5KvRDNwaTB5VbB8U9wLiBbmQyAVj6zGxtQZlX71ViJh9K0Ec9yE1VJJhteSVU6+1AaohpRO60w= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=VBfAmxSw; arc=none smtp.client-ip=192.198.163.9 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="VBfAmxSw" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1711348866; x=1742884866; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=xuamuB97LVfxyJSvE4i73CrmF8XOSgafgM3yHCD1nqo=; b=VBfAmxSwBgdrrwCzbrTm2FzySbU9TNxzt/YH/zf8WHDRYCaan3Y0029B Trie5g67Xa3Ds590b5fkpTqaglCV9Muk4kwo09p0PhggO531VEU0W2yzf GyXWOZprCjad0P1zIsE9p6RZicuAgxe/yxr6+iSFDXuOUpyyZ3Wdl+1cL KcXpE6ZXMiP8qRw3rGnB7x+lqYtn56IYj+oIztCvPTN1ruUz3dU6mNHJS cFGsXjkh7SdzwLX137/ifFuD1ySbbIHMKk1tHYSP+f6yV2bIOeERlGN3n LiU1gekwYKIB8Y1cv8Ucq0G+gCxHNOixMsmzNLSH1OBa9QDcCfg6RgeoR g==; X-IronPort-AV: E=McAfee;i="6600,9927,11023"; a="17065143" X-IronPort-AV: E=Sophos;i="6.07,152,1708416000"; d="scan'208";a="17065143" Received: from fmviesa002.fm.intel.com ([10.60.135.142]) by fmvoesa103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Mar 2024 23:41:04 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,152,1708416000"; d="scan'208";a="38629583" Received: from ahunter6-mobl1.ger.corp.intel.com (HELO ahunter-VirtualBox.home\044ger.corp.intel.com) ([10.251.211.155]) by fmviesa002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Mar 2024 23:40:58 -0700 From: Adrian Hunter To: Thomas Gleixner Cc: Michael Ellerman , Nicholas Piggin , Christophe Leroy , "Aneesh Kumar K.V" , "Naveen N. Rao" , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Andy Lutomirski , Vincenzo Frascino , John Stultz , Stephen Boyd , Peter Zijlstra , Randy Dunlap , Bjorn Helgaas , Arnd Bergmann , Anna-Maria Behnsen , linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, linux-s390@vger.kernel.org Subject: [PATCH V2 04/19] math64: Tidy mul_u64_u32_shr() Date: Mon, 25 Mar 2024 08:40:08 +0200 Message-Id: <20240325064023.2997-5-adrian.hunter@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240325064023.2997-1-adrian.hunter@intel.com> References: <20240325064023.2997-1-adrian.hunter@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Organization: Intel Finland Oy, Registered Address: PL 281, 00181 Helsinki, Business Identity Code: 0357606 - 4, Domiciled in Helsinki Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Put together declaration and initialization of local variables. Suggested-by: Thomas Gleixner Signed-off-by: Adrian Hunter --- include/linux/math64.h | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/include/linux/math64.h b/include/linux/math64.h index bf74478926d4..fd13622b2056 100644 --- a/include/linux/math64.h +++ b/include/linux/math64.h @@ -179,16 +179,12 @@ static __always_inline u64 mul_u64_u64_shr(u64 a, u64= mul, unsigned int shift) #ifndef mul_u64_u32_shr static __always_inline u64 mul_u64_u32_shr(u64 a, u32 mul, unsigned int sh= ift) { - u32 ah, al; + u32 ah =3D a >> 32, al =3D a; u64 ret; =20 - al =3D a; - ah =3D a >> 32; - ret =3D mul_u32_u32(al, mul) >> shift; if (ah) ret +=3D mul_u32_u32(ah, mul) << (32 - shift); - return ret; } #endif /* mul_u64_u32_shr */ --=20 2.34.1 From nobody Sat Feb 7 11:31:10 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A9FEA15EFD0; Mon, 25 Mar 2024 06:41:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.9 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711348873; cv=none; b=RJ4hrVZVlLiCBdz5aOvho0SkmClz7LGrjsGolAgEvB7zRjglGvxgSUpYY9IpcOjNXG/PROBKj7CitAoA5s86PShLVPx0AmVIwKQxEb3eyU4vcDMzq6PeFBOopDvgFRIOX33+8rSR6uD7mHSEiD+6eMfXL58no9nK5e5aso7Gw9g= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711348873; c=relaxed/simple; bh=LAWkJDgSOoQXttIOaNX5OjXrKNZqw1gNI6g5JMXltWY=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=hNj+Rb68NzrAlV9rTNqn237ofr1FioGZfDNvaOmREWsf3+eIWx/w5SWe2cNNsB25rsG5eFX6e9O+m2J7nkG0z2o1efLIhs2GOZsW+HXVa2ngJ45snPULloDO7bMxa1rJvFnMc4zjj3MVs/yg+GF7XIDTi0Y93smNJ5wWquQ857k= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=ItsxdqfW; arc=none smtp.client-ip=192.198.163.9 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="ItsxdqfW" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1711348871; x=1742884871; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=LAWkJDgSOoQXttIOaNX5OjXrKNZqw1gNI6g5JMXltWY=; b=ItsxdqfWSLQVXE13YnVyQihi5iSU35UJX3V5+hvuaOTDf7WMkEf7kVtu xvnqn0q7FXOrPZ0gwg0YV/XqbpC5MKHgczrZU/+2XXJJHCVJhlhd1/ply Q184ewDN7lWa+aljtgSKiMVlw/EH8QuzOoGeZXVhWP9R9mrhf2CxeQVvb RpDPaZ0gwdmPBF3MrSaCPQlpSsVEKycJvFoW0oNIIVZ7RRGrSo6rCnD21 F/ECVF9O1mY9Jxzet8wqeXKWsz4QQiG9jRF4G4jMokW0Jxj2Oqwi8bszN Ei4r6Be6K+9l6L01JyeomU8Llq6c0uY28ETabUAiWHOGbIhetARlaUYmJ Q==; X-IronPort-AV: E=McAfee;i="6600,9927,11023"; a="17065158" X-IronPort-AV: E=Sophos;i="6.07,152,1708416000"; d="scan'208";a="17065158" Received: from fmviesa002.fm.intel.com ([10.60.135.142]) by fmvoesa103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Mar 2024 23:41:11 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,152,1708416000"; d="scan'208";a="38629590" Received: from ahunter6-mobl1.ger.corp.intel.com (HELO ahunter-VirtualBox.home\044ger.corp.intel.com) ([10.251.211.155]) by fmviesa002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Mar 2024 23:41:05 -0700 From: Adrian Hunter To: Thomas Gleixner Cc: Michael Ellerman , Nicholas Piggin , Christophe Leroy , "Aneesh Kumar K.V" , "Naveen N. Rao" , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Andy Lutomirski , Vincenzo Frascino , John Stultz , Stephen Boyd , Peter Zijlstra , Randy Dunlap , Bjorn Helgaas , Arnd Bergmann , Anna-Maria Behnsen , linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, linux-s390@vger.kernel.org Subject: [PATCH V2 05/19] vdso: math64: Provide mul_u64_u32_add_u64_shr() Date: Mon, 25 Mar 2024 08:40:09 +0200 Message-Id: <20240325064023.2997-6-adrian.hunter@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240325064023.2997-1-adrian.hunter@intel.com> References: <20240325064023.2997-1-adrian.hunter@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Organization: Intel Finland Oy, Registered Address: PL 281, 00181 Helsinki, Business Identity Code: 0357606 - 4, Domiciled in Helsinki Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Provide mul_u64_u32_add_u64_shr() which is a calculation that will be used by timekeeping and VDSO. Place #include after #include to allow architecture-specific overrides, at least for the kernel. Signed-off-by: Adrian Hunter --- include/linux/math64.h | 2 +- include/vdso/math64.h | 38 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 39 insertions(+), 1 deletion(-) diff --git a/include/linux/math64.h b/include/linux/math64.h index fd13622b2056..d34def7f9a8c 100644 --- a/include/linux/math64.h +++ b/include/linux/math64.h @@ -4,8 +4,8 @@ =20 #include #include -#include #include +#include =20 #if BITS_PER_LONG =3D=3D 64 =20 diff --git a/include/vdso/math64.h b/include/vdso/math64.h index 7da703ee5561..22ae212f8b28 100644 --- a/include/vdso/math64.h +++ b/include/vdso/math64.h @@ -21,4 +21,42 @@ __iter_div_u64_rem(u64 dividend, u32 divisor, u64 *remai= nder) return ret; } =20 +#if defined(CONFIG_ARCH_SUPPORTS_INT128) && defined(__SIZEOF_INT128__) + +#ifndef mul_u64_u32_add_u64_shr +static __always_inline u64 mul_u64_u32_add_u64_shr(u64 a, u32 mul, u64 b, = unsigned int shift) +{ + return (u64)((((unsigned __int128)a * mul) + b) >> shift); +} +#endif /* mul_u64_u32_add_u64_shr */ + +#else + +#ifndef mul_u64_u32_add_u64_shr +#ifndef mul_u32_u32 +static inline u64 mul_u32_u32(u32 a, u32 b) +{ + return (u64)a * b; +} +#define mul_u32_u32 mul_u32_u32 +#endif +static __always_inline u64 mul_u64_u32_add_u64_shr(u64 a, u32 mul, u64 b, = unsigned int shift) +{ + u32 ah =3D a >> 32, al =3D a; + bool ovf; + u64 ret; + + ovf =3D __builtin_add_overflow(mul_u32_u32(al, mul), b, &ret); + ret >>=3D shift; + if (ovf && shift) + ret +=3D 1ULL << (64 - shift); + if (ah) + ret +=3D mul_u32_u32(ah, mul) << (32 - shift); + + return ret; +} +#endif /* mul_u64_u32_add_u64_shr */ + +#endif + #endif /* __VDSO_MATH64_H */ --=20 2.34.1 From nobody Sat Feb 7 11:31:10 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 40E1815EFDE; Mon, 25 Mar 2024 06:41:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.9 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711348879; cv=none; b=BkeO8AxC+ap8l7zMes2aj+Es4rLB6ydfAADSvfh34cowfrNeThDwXnjCmDSnJCaqBxNus/ZYmSkCTR8dfTyhjUeO9E2IKY1mQfNAF2rjBvg2dxTJc4nZP3GJBQNVdHSCZM3b9ZNJs/Y3AHKZa4gbjWnAnxPukSLvUaBCHwL727c= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711348879; c=relaxed/simple; bh=Exf/ex05wg/ZdHrQtagsjpQZFoomRp7dCTSrSntMNmc=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=HNxqlXLfCbZDGXBL2O+exk8c1nf1LfRptNoboCsNYST/kydEjhwoxFmb3XkSxg52b4/0UfvgSAwDEnj46kjO/XkSP6I2DxEOA52cUCOcF26hNnbxdFlkdYDghec+I6z2tMnv+9z3Q+EUbvYa+7QPVODbzlORAxJQjE5Ea9Yttu8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=axHArCxd; arc=none smtp.client-ip=192.198.163.9 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="axHArCxd" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1711348878; x=1742884878; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Exf/ex05wg/ZdHrQtagsjpQZFoomRp7dCTSrSntMNmc=; b=axHArCxdlLJWcdpUdFpdUKDb4vGFMy8xAeYle9WYj4wO9bYvG0qfdVB+ e519CX6ze37Otw54tKE+4S5TEfFzN1kkyHo5LuP30R3udjLM+5En+xL05 JX2dw4TKgWIbI2ZBqsuM9eGRWj0h1Qlsr71U6qh29hUykxzWK+CwNb777 a2kJX2K4lHYWJgK9IHChPfcU287b7Ne3lNeHhYfz/LcdeAGexjyG9jbS9 WR1rC31mVvhcTVuc7p2PGV/TJ8t8calE2Pl9bUuvI2nj91QyYDKaLyYxR lgAQcSl4FRyahAaRgXzLlA4RbxPCVk/pany5AANWQGquvMYwMf34SDG8g w==; X-IronPort-AV: E=McAfee;i="6600,9927,11023"; a="17065180" X-IronPort-AV: E=Sophos;i="6.07,152,1708416000"; d="scan'208";a="17065180" Received: from fmviesa002.fm.intel.com ([10.60.135.142]) by fmvoesa103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Mar 2024 23:41:17 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,152,1708416000"; d="scan'208";a="38629593" Received: from ahunter6-mobl1.ger.corp.intel.com (HELO ahunter-VirtualBox.home\044ger.corp.intel.com) ([10.251.211.155]) by fmviesa002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Mar 2024 23:41:11 -0700 From: Adrian Hunter To: Thomas Gleixner Cc: Michael Ellerman , Nicholas Piggin , Christophe Leroy , "Aneesh Kumar K.V" , "Naveen N. Rao" , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Andy Lutomirski , Vincenzo Frascino , John Stultz , Stephen Boyd , Peter Zijlstra , Randy Dunlap , Bjorn Helgaas , Arnd Bergmann , Anna-Maria Behnsen , linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, linux-s390@vger.kernel.org Subject: [PATCH V2 06/19] vdso: Add vdso_data::max_cycles Date: Mon, 25 Mar 2024 08:40:10 +0200 Message-Id: <20240325064023.2997-7-adrian.hunter@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240325064023.2997-1-adrian.hunter@intel.com> References: <20240325064023.2997-1-adrian.hunter@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Organization: Intel Finland Oy, Registered Address: PL 281, 00181 Helsinki, Business Identity Code: 0357606 - 4, Domiciled in Helsinki Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add vdso_data::max_cycles in preparation to use it to detect potential multiplication overflow. Suggested-by: Thomas Gleixner Signed-off-by: Adrian Hunter --- include/vdso/datapage.h | 4 ++++ kernel/time/vsyscall.c | 6 ++++++ 2 files changed, 10 insertions(+) diff --git a/include/vdso/datapage.h b/include/vdso/datapage.h index 5d5c0b8efff2..6c3d67d6b758 100644 --- a/include/vdso/datapage.h +++ b/include/vdso/datapage.h @@ -67,6 +67,7 @@ struct vdso_timestamp { * @seq: timebase sequence counter * @clock_mode: clock mode * @cycle_last: timebase at clocksource init + * @max_cycles: maximum cycles which won't overflow 64bit multiplication * @mask: clocksource mask * @mult: clocksource multiplier * @shift: clocksource shift @@ -98,6 +99,9 @@ struct vdso_data { =20 s32 clock_mode; u64 cycle_last; +#ifdef CONFIG_GENERIC_VDSO_OVERFLOW_PROTECT + u64 max_cycles; +#endif u64 mask; u32 mult; u32 shift; diff --git a/kernel/time/vsyscall.c b/kernel/time/vsyscall.c index f0d5062d9cbc..9193d6133e5d 100644 --- a/kernel/time/vsyscall.c +++ b/kernel/time/vsyscall.c @@ -22,10 +22,16 @@ static inline void update_vdso_data(struct vdso_data *v= data, u64 nsec, sec; =20 vdata[CS_HRES_COARSE].cycle_last =3D tk->tkr_mono.cycle_last; +#ifdef CONFIG_GENERIC_VDSO_OVERFLOW_PROTECT + vdata[CS_HRES_COARSE].max_cycles =3D tk->tkr_mono.clock->max_cycles; +#endif vdata[CS_HRES_COARSE].mask =3D tk->tkr_mono.mask; vdata[CS_HRES_COARSE].mult =3D tk->tkr_mono.mult; vdata[CS_HRES_COARSE].shift =3D tk->tkr_mono.shift; vdata[CS_RAW].cycle_last =3D tk->tkr_raw.cycle_last; +#ifdef CONFIG_GENERIC_VDSO_OVERFLOW_PROTECT + vdata[CS_RAW].max_cycles =3D tk->tkr_raw.clock->max_cycles; +#endif vdata[CS_RAW].mask =3D tk->tkr_raw.mask; vdata[CS_RAW].mult =3D tk->tkr_raw.mult; vdata[CS_RAW].shift =3D tk->tkr_raw.shift; --=20 2.34.1 From nobody Sat Feb 7 11:31:10 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 63F4815F3E2; Mon, 25 Mar 2024 06:41:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.9 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711348888; cv=none; b=p0yL1ZHw9sxfAeOSn/hjVJi2IXh5Z9iBOpaqS1j4uL9uECJzpi4Om2/hgHZ/zww/FsvvkUBsR2NobdAfZ26kPcGqY4wUEu56iuY5xMDr5NdDH6KnxaeKWrX0sfI7HIbov7wzwH2dtv0QZo9ChzuOyTCco5bWxnweGgYoesJx/eI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711348888; c=relaxed/simple; bh=Zjh6IUndlImHk90vhCoelXOI1d7rMagiowI5tzl77NQ=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=nI59wntMQwHShCXljF/y9H0PtnTBJo93RYSVIWo3D0vOuGK3lw6CYgLX5ZzfS56dT668gVPb50GxzjmwRtQBPUb6PGbhvBLLKfLi38DvPjj12J0GI1M5jiEg5bNcOnkBP83RbrvxZEl4UOfxQoR7UYY614AR3auXtOcBhyn8Zbs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=d2HRW4RJ; arc=none smtp.client-ip=192.198.163.9 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="d2HRW4RJ" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1711348886; x=1742884886; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Zjh6IUndlImHk90vhCoelXOI1d7rMagiowI5tzl77NQ=; b=d2HRW4RJ+Whp0NALxuEas08Z+GpG7vgYUH0VRQzlWPQq9HbPC0k4bN6A 0kJcy6Jrvhn6cwVCPLaxIXxitFfkdcTSGbuevtu5QcKHIf9vCYufhnOSg +WJfdRqxOvbZM+VQIH1qoPxQzAQYztrotZ6aNvMminNzuemJMlT4ooJRK 1SnsQYtfqCzqgK6lKnpi2xVmwcLwthVJCLa/Qrf7YHkcvb5WKKDjJoeKx de9gHSzCdP2tSAs+GTPWMhdfVH35Jx5hJfjPFatMx42I8rWTJ7XqfqRzY tbpRag46J17Gym4ftdr1IpxzraYdehNh664IVNnFiMBwzsHUVgiAMQVi5 g==; X-IronPort-AV: E=McAfee;i="6600,9927,11023"; a="17065197" X-IronPort-AV: E=Sophos;i="6.07,152,1708416000"; d="scan'208";a="17065197" Received: from fmviesa002.fm.intel.com ([10.60.135.142]) by fmvoesa103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Mar 2024 23:41:24 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,152,1708416000"; d="scan'208";a="38629597" Received: from ahunter6-mobl1.ger.corp.intel.com (HELO ahunter-VirtualBox.home\044ger.corp.intel.com) ([10.251.211.155]) by fmviesa002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Mar 2024 23:41:18 -0700 From: Adrian Hunter To: Thomas Gleixner Cc: Michael Ellerman , Nicholas Piggin , Christophe Leroy , "Aneesh Kumar K.V" , "Naveen N. Rao" , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Andy Lutomirski , Vincenzo Frascino , John Stultz , Stephen Boyd , Peter Zijlstra , Randy Dunlap , Bjorn Helgaas , Arnd Bergmann , Anna-Maria Behnsen , linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, linux-s390@vger.kernel.org Subject: [PATCH V2 07/19] vdso: Make delta calculation overflow safe Date: Mon, 25 Mar 2024 08:40:11 +0200 Message-Id: <20240325064023.2997-8-adrian.hunter@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240325064023.2997-1-adrian.hunter@intel.com> References: <20240325064023.2997-1-adrian.hunter@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Organization: Intel Finland Oy, Registered Address: PL 281, 00181 Helsinki, Business Identity Code: 0357606 - 4, Domiciled in Helsinki Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Kernel timekeeping is designed to keep the change in cycles (since the last timer interrupt) below max_cycles, which prevents multiplication overflow when converting cycles to nanoseconds. However, if timer interrupts stop, the calculation will eventually overflow. Add protection against that, enabled by config option CONFIG_GENERIC_VDSO_OVERFLOW_PROTECT. Check against max_cycles, falling back to a slower higher precision calculation. Suggested-by: Thomas Gleixner Signed-off-by: Adrian Hunter --- lib/vdso/gettimeofday.c | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/lib/vdso/gettimeofday.c b/lib/vdso/gettimeofday.c index 9fa90e0794c9..9c3a8d2440c9 100644 --- a/lib/vdso/gettimeofday.c +++ b/lib/vdso/gettimeofday.c @@ -13,6 +13,18 @@ # define VDSO_DELTA_MASK(vd) (vd->mask) #endif =20 +#ifdef CONFIG_GENERIC_VDSO_OVERFLOW_PROTECT +static __always_inline bool vdso_delta_ok(const struct vdso_data *vd, u64 = delta) +{ + return delta < vd->max_cycles; +} +#else +static __always_inline bool vdso_delta_ok(const struct vdso_data *vd, u64 = delta) +{ + return true; +} +#endif + #ifndef vdso_shift_ns static __always_inline u64 vdso_shift_ns(u64 ns, u32 shift) { @@ -28,7 +40,10 @@ static __always_inline u64 vdso_calc_ns(const struct vds= o_data *vd, u64 cycles, { u64 delta =3D (cycles - vd->cycle_last) & VDSO_DELTA_MASK(vd); =20 - return vdso_shift_ns((delta * vd->mult) + base, vd->shift); + if (likely(vdso_delta_ok(vd, delta))) + return vdso_shift_ns((delta * vd->mult) + base, vd->shift); + + return mul_u64_u32_add_u64_shr(delta, vd->mult, base, vd->shift); } #endif /* vdso_calc_ns */ =20 --=20 2.34.1 From nobody Sat Feb 7 11:31:10 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9768F15F3F5; Mon, 25 Mar 2024 06:41:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.9 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711348893; cv=none; b=Bd9vEP7DOT7qDCuoduFrcE0PMOWwr3JCW4x9wC0AT99wC7SbzWyQSTIiXcNM6T4PJG+FFzy8GQplsFQpyiD4oGTGjjVyNZmM/H3cbIYSjgBNy1x9/D08td039bjN+rQg3n32fSENaBF0Uyhfqtn/v4+AyXappcIp8T1BMWWBaUQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711348893; c=relaxed/simple; bh=tRyxc+NjWa10JwJq5Oo62d3MoJ/4i/AwqfqN12/XrkQ=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=hySxjraNHZdVf7LXVJ4vAERUQmMC1R2XsCa7gyRR8SXbMvDsXM3fdvN+lyyXfY4HPBMcFi2Im1g4OwYYkQUUk9ng79KygkRgvXyyM34m5ZUfuk8xThm8mh0tprCKbEnKCbMyzXQdIwhrgdRod5djMlXUiYiNubdFN/oq28LFMf0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=ZcnCP8Hd; arc=none smtp.client-ip=192.198.163.9 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="ZcnCP8Hd" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1711348891; x=1742884891; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=tRyxc+NjWa10JwJq5Oo62d3MoJ/4i/AwqfqN12/XrkQ=; b=ZcnCP8Hdj6Yunb/uBCTaem/QbJJ8IgqxvJCAsf5tHn3HTDQj+VY9Tie0 OyyKUvdpkZP3ijhc3fUQ1T81sdsUTRHQamG+mCS4H9fNoJI12lHr1P+lA ADVREu1dGSZahXZbw4xs9SrmNscn7bY5BHw0wcZDzHMxbfFFGjJGTR27D bLgah26ik1UPYqhWr4M+XZBiKC1353O2jIQso2ZrxoRVZLgno2OjKn7E9 RY21KbPVDVYwjCMyHd4CgiohCAgya8x2pq8y64oYlkHpPMiQ542j+IjJ/ YBlMJcSQiVyrxLZm2/116ixLRMA+d2abSlihRmDKN607KZSWuWd8v7Jpi w==; X-IronPort-AV: E=McAfee;i="6600,9927,11023"; a="17065225" X-IronPort-AV: E=Sophos;i="6.07,152,1708416000"; d="scan'208";a="17065225" Received: from fmviesa002.fm.intel.com ([10.60.135.142]) by fmvoesa103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Mar 2024 23:41:31 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,152,1708416000"; d="scan'208";a="38629602" Received: from ahunter6-mobl1.ger.corp.intel.com (HELO ahunter-VirtualBox.home\044ger.corp.intel.com) ([10.251.211.155]) by fmviesa002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Mar 2024 23:41:24 -0700 From: Adrian Hunter To: Thomas Gleixner Cc: Michael Ellerman , Nicholas Piggin , Christophe Leroy , "Aneesh Kumar K.V" , "Naveen N. Rao" , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Andy Lutomirski , Vincenzo Frascino , John Stultz , Stephen Boyd , Peter Zijlstra , Randy Dunlap , Bjorn Helgaas , Arnd Bergmann , Anna-Maria Behnsen , linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, linux-s390@vger.kernel.org Subject: [PATCH V2 08/19] x86/vdso: Make delta calculation overflow safe Date: Mon, 25 Mar 2024 08:40:12 +0200 Message-Id: <20240325064023.2997-9-adrian.hunter@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240325064023.2997-1-adrian.hunter@intel.com> References: <20240325064023.2997-1-adrian.hunter@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Organization: Intel Finland Oy, Registered Address: PL 281, 00181 Helsinki, Business Identity Code: 0357606 - 4, Domiciled in Helsinki Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Kernel timekeeping is designed to keep the change in cycles (since the last timer interrupt) below max_cycles, which prevents multiplication overflow when converting cycles to nanoseconds. However, if timer interrupts stop, the calculation will eventually overflow. Add protection against that. Select GENERIC_VDSO_OVERFLOW_PROTECT so that max_cycles is made available in the VDSO data page. Check against max_cycles, falling back to a slower higher precision calculation. Take advantage of the opportunity to move masking and negative motion check into the slow path. The result is a calculation that has similar performance as before. Newer machines showed performance benefit, whereas older Skylake-based hardware such as Intel Kaby Lake was seen <1% worse. Suggested-by: Thomas Gleixner Signed-off-by: Adrian Hunter --- arch/x86/Kconfig | 1 + arch/x86/include/asm/vdso/gettimeofday.h | 29 +++++++++++++++++------- 2 files changed, 22 insertions(+), 8 deletions(-) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 03483b23a009..3a70ebb558e7 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -168,6 +168,7 @@ config X86 select GENERIC_TIME_VSYSCALL select GENERIC_GETTIMEOFDAY select GENERIC_VDSO_TIME_NS + select GENERIC_VDSO_OVERFLOW_PROTECT select GUP_GET_PXX_LOW_HIGH if X86_PAE select HARDIRQS_SW_RESEND select HARDLOCKUP_CHECK_TIMESTAMP if X86_64 diff --git a/arch/x86/include/asm/vdso/gettimeofday.h b/arch/x86/include/as= m/vdso/gettimeofday.h index 5727dedd3549..0ef36190abe6 100644 --- a/arch/x86/include/asm/vdso/gettimeofday.h +++ b/arch/x86/include/asm/vdso/gettimeofday.h @@ -319,18 +319,31 @@ static inline bool arch_vdso_cycles_ok(u64 cycles) */ static __always_inline u64 vdso_calc_ns(const struct vdso_data *vd, u64 cy= cles, u64 base) { + u64 delta =3D cycles - vd->cycle_last; + /* + * Negative motion and deltas which can cause multiplication + * overflow require special treatment. This check covers both as + * negative motion is guaranteed to be greater than @vd::max_cycles + * due to unsigned comparison. + * * Due to the MSB/Sign-bit being used as invalid marker (see - * arch_vdso_cycles_valid() above), the effective mask is S64_MAX. + * arch_vdso_cycles_valid() above), the effective mask is S64_MAX, + * but that case is also unlikely and will also take the unlikely path + * here. */ - u64 delta =3D (cycles - vd->cycle_last) & S64_MAX; + if (unlikely(delta > vd->max_cycles)) { + /* + * Due to the above mentioned TSC wobbles, filter out + * negative motion. Per the above masking, the effective + * sign bit is now bit 62. + */ + if (delta & (1ULL << 62)) + return base >> vd->shift; =20 - /* - * Due to the above mentioned TSC wobbles, filter out negative motion. - * Per the above masking, the effective sign bit is now bit 62. - */ - if (unlikely(delta & (1ULL << 62))) - return base >> vd->shift; + /* Handle multiplication overflow gracefully */ + return mul_u64_u32_add_u64_shr(delta & S64_MAX, vd->mult, base, vd->shif= t); + } =20 return ((delta * vd->mult) + base) >> vd->shift; } --=20 2.34.1 From nobody Sat Feb 7 11:31:10 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4E1A815F417; Mon, 25 Mar 2024 06:41:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.9 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711348901; cv=none; b=mohslj15zagYyKq+a7CtKUPGPm4L/CZCu8JwrGxx7OH3PJvxECPZh1d5zTNC26o9FqGzoBiczBnDjGbdCAR1jPBxxm8As5G7h74zYH+pJTPlATnCb4ClSt/5/tpSFHMDwNUNaCKyA0SI2sCCGUaNGWyJflta6qY3cCmc87i01Qw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711348901; c=relaxed/simple; bh=zboSVI3HgVZXbpDBNnsTotqgGm69H9hHwUY1hZ67rA8=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Xl9tQ5d4Pn+3ewuxKsayb4b5emAOWhNgvKydTwhC5KbikgEWM6rL8Huu+Is6sMgBuoPZybBP+/pHw9uW8vsqWjofrS3VSXIMRewHWVAdmqwMvWH1YcIEcLSK1JMHHvUxLiZUzP0ZcYIDlKb2Zy8YtPaDuTdJFRmgA52RXQZJ+oM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=a2v8Ujy0; arc=none smtp.client-ip=192.198.163.9 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="a2v8Ujy0" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1711348899; x=1742884899; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=zboSVI3HgVZXbpDBNnsTotqgGm69H9hHwUY1hZ67rA8=; b=a2v8Ujy0sQ+IDOD9pVB2CMc/Jh0i+4V3YJqsJGXnddv7kTrDgxiRVM+l CnIaUQnNIoYKWi1z11gbB/+zdprGVb7j4PkZnywCdOIitt/tUz8WgbDC5 sh/v+jHy+IyUbD7kYeQgedV2zHCP2uT557QzVDImuQ1YM6lwbHy/i6Rxc Y0cLTVn6Mq1lFxtlY4uDd3Kxc09+tmzCAmFUk4k7CkpimyNdmLJP4Qxk/ vQCHSBDvSLV4mbOGPjtCAWbZ0aEnyj0/56oBmw5cIWCYINWUNDTRNvARo CYAYDQSstyiiHSysQuvOdGcX3JBGjPJysdyU6El041idJ0T6Txmj5D0lN Q==; X-IronPort-AV: E=McAfee;i="6600,9927,11023"; a="17065255" X-IronPort-AV: E=Sophos;i="6.07,152,1708416000"; d="scan'208";a="17065255" Received: from fmviesa002.fm.intel.com ([10.60.135.142]) by fmvoesa103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Mar 2024 23:41:37 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,152,1708416000"; d="scan'208";a="38629609" Received: from ahunter6-mobl1.ger.corp.intel.com (HELO ahunter-VirtualBox.home\044ger.corp.intel.com) ([10.251.211.155]) by fmviesa002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Mar 2024 23:41:31 -0700 From: Adrian Hunter To: Thomas Gleixner Cc: Michael Ellerman , Nicholas Piggin , Christophe Leroy , "Aneesh Kumar K.V" , "Naveen N. Rao" , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Andy Lutomirski , Vincenzo Frascino , John Stultz , Stephen Boyd , Peter Zijlstra , Randy Dunlap , Bjorn Helgaas , Arnd Bergmann , Anna-Maria Behnsen , linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, linux-s390@vger.kernel.org Subject: [PATCH V2 09/19] timekeeping: Move timekeeping helper functions Date: Mon, 25 Mar 2024 08:40:13 +0200 Message-Id: <20240325064023.2997-10-adrian.hunter@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240325064023.2997-1-adrian.hunter@intel.com> References: <20240325064023.2997-1-adrian.hunter@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Organization: Intel Finland Oy, Registered Address: PL 281, 00181 Helsinki, Business Identity Code: 0357606 - 4, Domiciled in Helsinki Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Move timekeeping helper functions to prepare for their reuse. Suggested-by: Thomas Gleixner Signed-off-by: Adrian Hunter --- kernel/time/timekeeping.c | 34 +++++++++++++++++----------------- 1 file changed, 17 insertions(+), 17 deletions(-) diff --git a/kernel/time/timekeeping.c b/kernel/time/timekeeping.c index b58dffc58a8f..3375f0a6400f 100644 --- a/kernel/time/timekeeping.c +++ b/kernel/time/timekeeping.c @@ -381,6 +381,23 @@ static inline u64 timekeeping_delta_to_ns(const struct= tk_read_base *tkr, u64 de return nsec; } =20 +static inline u64 timekeeping_cycles_to_ns(const struct tk_read_base *tkr,= u64 cycles) +{ + u64 delta; + + /* calculate the delta since the last update_wall_time */ + delta =3D clocksource_delta(cycles, tkr->cycle_last, tkr->mask); + return timekeeping_delta_to_ns(tkr, delta); +} + +static __always_inline u64 fast_tk_get_delta_ns(struct tk_read_base *tkr) +{ + u64 delta, cycles =3D tk_clock_read(tkr); + + delta =3D clocksource_delta(cycles, tkr->cycle_last, tkr->mask); + return timekeeping_delta_to_ns(tkr, delta); +} + static inline u64 timekeeping_get_ns(const struct tk_read_base *tkr) { u64 delta; @@ -389,15 +406,6 @@ static inline u64 timekeeping_get_ns(const struct tk_r= ead_base *tkr) return timekeeping_delta_to_ns(tkr, delta); } =20 -static inline u64 timekeeping_cycles_to_ns(const struct tk_read_base *tkr,= u64 cycles) -{ - u64 delta; - - /* calculate the delta since the last update_wall_time */ - delta =3D clocksource_delta(cycles, tkr->cycle_last, tkr->mask); - return timekeeping_delta_to_ns(tkr, delta); -} - /** * update_fast_timekeeper - Update the fast and NMI safe monotonic timekee= per. * @tkr: Timekeeping readout base from which we take the update @@ -431,14 +439,6 @@ static void update_fast_timekeeper(const struct tk_rea= d_base *tkr, memcpy(base + 1, base, sizeof(*base)); } =20 -static __always_inline u64 fast_tk_get_delta_ns(struct tk_read_base *tkr) -{ - u64 delta, cycles =3D tk_clock_read(tkr); - - delta =3D clocksource_delta(cycles, tkr->cycle_last, tkr->mask); - return timekeeping_delta_to_ns(tkr, delta); -} - static __always_inline u64 __ktime_get_fast_ns(struct tk_fast *tkf) { struct tk_read_base *tkr; --=20 2.34.1 From nobody Sat Feb 7 11:31:10 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D0E8E15FA6B; Mon, 25 Mar 2024 06:41:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.9 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711348906; cv=none; b=VEOBm23YoE9aQp9R7hA9RYfckUd7JVdW8L5Dg11pZh0F62dgPTJAl4K5w6Q86UtkWazqd70NqbNlH7SrkimU8Srd76htn+vU9HWLjl18PWxrwT8myvQjXUSdvB7SJe0/uh+ttRWak5iKSejMBjXXEG+/lROD2oTEIirrHZe3jBs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711348906; c=relaxed/simple; bh=qTxeGZya5tNrF+OulKJd9BaTzgUsvcEDLjRYcSx3zxw=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=XwrAFMfeotmgKTHTwhQ+TGPckM5M8uMnUQdLRn/M53Snjdaa3ROqUmIfl6/ML0f+ajCRyVzpzoqknujpzBTiT0EQi2Vq4TQcEULEXIWkYO8SquOZ3raGA0EFlkHjAgSSg8HMzUOk1XTKJDrAEj0K/C1M8XIc8NZJLdaNMFUHGKo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=KyfdVzKA; arc=none smtp.client-ip=192.198.163.9 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="KyfdVzKA" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1711348904; x=1742884904; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=qTxeGZya5tNrF+OulKJd9BaTzgUsvcEDLjRYcSx3zxw=; b=KyfdVzKA2s5LNA6XOx/QJjp5EVFFa5tHytycZznjyGLYetPwC1yYsH5z iazY84CZnioFJ/Cfj9VEnaLJ5YJu97gqPxJovMOLsZyNbHnFOrE3vaUMj P0kpCcE8epnToPWEqCLVDFZK5RkrLxe7U4GEdyql/AtO1e0rVK7t39HsT uSOt895YxqM0mou8zC+9yusH8MjJw9VhPI7EEfL03LlaP+R+NKVX4UPqe LN0dEFEJwOeDD6ccscGrkefuF4eMmtaKuaBCNdyWNdgF+WoOUWDUPA4zU 75clAGEnfQ5M7uuxNhKr71Lchz50VuxeXRwRvFRJ0hP6yAXNhE3Ga4rEu w==; X-IronPort-AV: E=McAfee;i="6600,9927,11023"; a="17065285" X-IronPort-AV: E=Sophos;i="6.07,152,1708416000"; d="scan'208";a="17065285" Received: from fmviesa002.fm.intel.com ([10.60.135.142]) by fmvoesa103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Mar 2024 23:41:44 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,152,1708416000"; d="scan'208";a="38629624" Received: from ahunter6-mobl1.ger.corp.intel.com (HELO ahunter-VirtualBox.home\044ger.corp.intel.com) ([10.251.211.155]) by fmviesa002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Mar 2024 23:41:38 -0700 From: Adrian Hunter To: Thomas Gleixner Cc: Michael Ellerman , Nicholas Piggin , Christophe Leroy , "Aneesh Kumar K.V" , "Naveen N. Rao" , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Andy Lutomirski , Vincenzo Frascino , John Stultz , Stephen Boyd , Peter Zijlstra , Randy Dunlap , Bjorn Helgaas , Arnd Bergmann , Anna-Maria Behnsen , linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, linux-s390@vger.kernel.org Subject: [PATCH V2 10/19] timekeeping: Rename fast_tk_get_delta_ns() to __timekeeping_get_ns() Date: Mon, 25 Mar 2024 08:40:14 +0200 Message-Id: <20240325064023.2997-11-adrian.hunter@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240325064023.2997-1-adrian.hunter@intel.com> References: <20240325064023.2997-1-adrian.hunter@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Organization: Intel Finland Oy, Registered Address: PL 281, 00181 Helsinki, Business Identity Code: 0357606 - 4, Domiciled in Helsinki Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Rename fast_tk_get_delta_ns() to __timekeeping_get_ns() to prepare for its reuse as a general timekeeping helper function. Suggested-by: Thomas Gleixner Signed-off-by: Adrian Hunter --- kernel/time/timekeeping.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/kernel/time/timekeeping.c b/kernel/time/timekeeping.c index 3375f0a6400f..63061332a75c 100644 --- a/kernel/time/timekeeping.c +++ b/kernel/time/timekeeping.c @@ -390,7 +390,7 @@ static inline u64 timekeeping_cycles_to_ns(const struct= tk_read_base *tkr, u64 c return timekeeping_delta_to_ns(tkr, delta); } =20 -static __always_inline u64 fast_tk_get_delta_ns(struct tk_read_base *tkr) +static __always_inline u64 __timekeeping_get_ns(const struct tk_read_base = *tkr) { u64 delta, cycles =3D tk_clock_read(tkr); =20 @@ -449,7 +449,7 @@ static __always_inline u64 __ktime_get_fast_ns(struct t= k_fast *tkf) seq =3D raw_read_seqcount_latch(&tkf->seq); tkr =3D tkf->base + (seq & 0x01); now =3D ktime_to_ns(tkr->base); - now +=3D fast_tk_get_delta_ns(tkr); + now +=3D __timekeeping_get_ns(tkr); } while (raw_read_seqcount_latch_retry(&tkf->seq, seq)); =20 return now; @@ -565,7 +565,7 @@ static __always_inline u64 __ktime_get_real_fast(struct= tk_fast *tkf, u64 *mono) tkr =3D tkf->base + (seq & 0x01); basem =3D ktime_to_ns(tkr->base); baser =3D ktime_to_ns(tkr->base_real); - delta =3D fast_tk_get_delta_ns(tkr); + delta =3D __timekeeping_get_ns(tkr); } while (raw_read_seqcount_latch_retry(&tkf->seq, seq)); =20 if (mono) --=20 2.34.1 From nobody Sat Feb 7 11:31:10 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5CFED15FA84; Mon, 25 Mar 2024 06:41:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.9 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711348913; cv=none; b=rHfSodhmCvGI5R8dMRuUXN3o6X6ftDMHgvy3N7WnXUvT6rcc4zboBm8qvFcaZyAVrDC6eeEN6EVZf1aeOEvD8IndHNUp3bHqCzJaAaow/JSjNXVzt0OZ+SPR1roiVPioKaTEiVvfT4vrAdnn5IVS/EKwXXH31Tbb6sIZ/7LyEsk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711348913; c=relaxed/simple; bh=EgUgzUoucqqDqOU2t7gKumIS3qm9Toh3D3HqJIoc9lY=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=HzcSeeZRLydl2iH7T23mYB4QzM4x63aWkpjb5lxL55zquB0eAjiWpSVQP2vzc9R8wFoJIqM/K4Y3VWsZgPNnEPHvMGvAWPcLQ+tWfLuXcWQJM72foJr+pw4E1+Iefq/uuknLj6PQ++Qi2ZLhVn0Vki/wnmVIxQNoyTzK6ICTIpc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=ZedpXBTo; arc=none smtp.client-ip=192.198.163.9 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="ZedpXBTo" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1711348912; x=1742884912; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=EgUgzUoucqqDqOU2t7gKumIS3qm9Toh3D3HqJIoc9lY=; b=ZedpXBToNbQxuBmTYxz2khw+RLO14UlpptIQnSBqowq6QkIUPcafPRG5 6VmtzfMV4yW0bnXVF/cu+OmPw5IumhCjkKSAl7gbc0OotQJz3pIhFs6KE Wmz12j3g6oNv4x11IK7Hk1uw+6yES9k5h+Nv5uVo+YM8EDvjgHTLvAn/3 czaS+/XZRnH8XqQGri8We/DAVuC5QGaTwg7+/5yfQg5X1Aq8+5+KVrJSW Use67OcpXIo1+a+s4pGE4+UvzOT9E+jmtBbWkvPj+ecxytriEjjjQh2De zB2fZGlPLveIizGQG1X7vRIYPdhtmQkPporrhNvkXQJkNTNB3Oeug4L2k g==; X-IronPort-AV: E=McAfee;i="6600,9927,11023"; a="17065310" X-IronPort-AV: E=Sophos;i="6.07,152,1708416000"; d="scan'208";a="17065310" Received: from fmviesa002.fm.intel.com ([10.60.135.142]) by fmvoesa103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Mar 2024 23:41:50 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,152,1708416000"; d="scan'208";a="38629637" Received: from ahunter6-mobl1.ger.corp.intel.com (HELO ahunter-VirtualBox.home\044ger.corp.intel.com) ([10.251.211.155]) by fmviesa002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Mar 2024 23:41:44 -0700 From: Adrian Hunter To: Thomas Gleixner Cc: Michael Ellerman , Nicholas Piggin , Christophe Leroy , "Aneesh Kumar K.V" , "Naveen N. Rao" , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Andy Lutomirski , Vincenzo Frascino , John Stultz , Stephen Boyd , Peter Zijlstra , Randy Dunlap , Bjorn Helgaas , Arnd Bergmann , Anna-Maria Behnsen , linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, linux-s390@vger.kernel.org Subject: [PATCH V2 11/19] timekeeping: Tidy timekeeping_cycles_to_ns() slightly Date: Mon, 25 Mar 2024 08:40:15 +0200 Message-Id: <20240325064023.2997-12-adrian.hunter@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240325064023.2997-1-adrian.hunter@intel.com> References: <20240325064023.2997-1-adrian.hunter@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Organization: Intel Finland Oy, Registered Address: PL 281, 00181 Helsinki, Business Identity Code: 0357606 - 4, Domiciled in Helsinki Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Put together declaration and initialization of the local variable 'delta'. Suggested-by: Thomas Gleixner Signed-off-by: Adrian Hunter --- kernel/time/timekeeping.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/kernel/time/timekeeping.c b/kernel/time/timekeeping.c index 63061332a75c..c698219b152d 100644 --- a/kernel/time/timekeeping.c +++ b/kernel/time/timekeeping.c @@ -383,10 +383,9 @@ static inline u64 timekeeping_delta_to_ns(const struct= tk_read_base *tkr, u64 de =20 static inline u64 timekeeping_cycles_to_ns(const struct tk_read_base *tkr,= u64 cycles) { - u64 delta; + /* Calculate the delta since the last update_wall_time() */ + u64 delta =3D clocksource_delta(cycles, tkr->cycle_last, tkr->mask); =20 - /* calculate the delta since the last update_wall_time */ - delta =3D clocksource_delta(cycles, tkr->cycle_last, tkr->mask); return timekeeping_delta_to_ns(tkr, delta); } =20 --=20 2.34.1 From nobody Sat Feb 7 11:31:10 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E659515FCED; Mon, 25 Mar 2024 06:41:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.9 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711348919; cv=none; b=PUeJ1IEJ5KsVyAjBojIuXGYzOxxp1gx4ovqSaSdEjrAO6If5GRCSWg3+2NkpRfSC64npQfFTyaAMvnFj9xcDHF3EobjfVKRCg5ccSw5nBClpTOJpUnC+6hXv/Xs/7lxeheJDHPNXAC8R2NJmx78HI6uAzRAu7p6z/jU2xyLVsuI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711348919; c=relaxed/simple; bh=ul0hfPOmfS7i90s07MvVRoTCajL4zQDIAp9Wq9nGTHo=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=WsrXdA9TkyWaeCsctnev+XxOXq7C/lpx1U23JTmDnIyVyW8snol5GiLo+fzxJZBOIjLXyySnmSCTOm2ynICcbg66g8Jyyzko2PsEgEyEYT+gACk4XQj+0q5oEnxEvYlOGv5gquwdI4TXhqN2q+0IFss3ubRP1DzmoWYRUcm2WOU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=alPU62rn; arc=none smtp.client-ip=192.198.163.9 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="alPU62rn" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1711348918; x=1742884918; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=ul0hfPOmfS7i90s07MvVRoTCajL4zQDIAp9Wq9nGTHo=; b=alPU62rn9qVu1yt3kG5Uiop0cXCzhPb0lBkiE7x2bAMlTGr0BQKfiVVr qVpmo2CUQilDIzajd4EN+ZEFld7xbl15W3/xYL9zzCfPJAjxVb6l4BYfz yTDuPH3g6l7pDHti7RSkJ9zzL50m8ad8CxaCnMOTVTjThRQgi4yI4HCMQ cRidxz6QGGXSp2wqua/q26pLJN5wzc7sAA8ytybpnF8huLv38aDYhwn8n Zp8cDuS6Fv4jOxvq9sFjTBwv7EfFeG9tnQqEq9DghedEGCm4I1nyeZ9eX MlOg/nYtNbCfMihg4ZhY7WKWjkDiv9S40IXfErYyUJZaXMr8Y6G9I6SgX A==; X-IronPort-AV: E=McAfee;i="6600,9927,11023"; a="17065335" X-IronPort-AV: E=Sophos;i="6.07,152,1708416000"; d="scan'208";a="17065335" Received: from fmviesa002.fm.intel.com ([10.60.135.142]) by fmvoesa103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Mar 2024 23:41:57 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,152,1708416000"; d="scan'208";a="38629647" Received: from ahunter6-mobl1.ger.corp.intel.com (HELO ahunter-VirtualBox.home\044ger.corp.intel.com) ([10.251.211.155]) by fmviesa002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Mar 2024 23:41:51 -0700 From: Adrian Hunter To: Thomas Gleixner Cc: Michael Ellerman , Nicholas Piggin , Christophe Leroy , "Aneesh Kumar K.V" , "Naveen N. Rao" , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Andy Lutomirski , Vincenzo Frascino , John Stultz , Stephen Boyd , Peter Zijlstra , Randy Dunlap , Bjorn Helgaas , Arnd Bergmann , Anna-Maria Behnsen , linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, linux-s390@vger.kernel.org Subject: [PATCH V2 12/19] timekeeping: Reuse timekeeping_cycles_to_ns() Date: Mon, 25 Mar 2024 08:40:16 +0200 Message-Id: <20240325064023.2997-13-adrian.hunter@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240325064023.2997-1-adrian.hunter@intel.com> References: <20240325064023.2997-1-adrian.hunter@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Organization: Intel Finland Oy, Registered Address: PL 281, 00181 Helsinki, Business Identity Code: 0357606 - 4, Domiciled in Helsinki Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Simplify __timekeeping_get_ns() by reusing timekeeping_cycles_to_ns(). Suggested-by: Thomas Gleixner Signed-off-by: Adrian Hunter --- kernel/time/timekeeping.c | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/kernel/time/timekeeping.c b/kernel/time/timekeeping.c index c698219b152d..f81d675291e0 100644 --- a/kernel/time/timekeeping.c +++ b/kernel/time/timekeeping.c @@ -391,10 +391,7 @@ static inline u64 timekeeping_cycles_to_ns(const struc= t tk_read_base *tkr, u64 c =20 static __always_inline u64 __timekeeping_get_ns(const struct tk_read_base = *tkr) { - u64 delta, cycles =3D tk_clock_read(tkr); - - delta =3D clocksource_delta(cycles, tkr->cycle_last, tkr->mask); - return timekeeping_delta_to_ns(tkr, delta); + return timekeeping_cycles_to_ns(tkr, tk_clock_read(tkr)); } =20 static inline u64 timekeeping_get_ns(const struct tk_read_base *tkr) --=20 2.34.1 From nobody Sat Feb 7 11:31:10 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 039391607A9; Mon, 25 Mar 2024 06:42:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.17 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711348936; cv=none; b=CpvxMsFVcak5N43K6gx2TmQR4bH6gI/iIvBn+1G95YxuHuYohDbJ2B/qQFU1dWUALwDrGpirdSyZSk3cS03bLZQmKZdYS0pBPqb//drjkg/EW/cGmJCuu3x7y1X61GxKj+RrDGju5OFhZI+yfzzYFYuvF/r3ntsd79Njp1zEu2c= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711348936; c=relaxed/simple; bh=16GQ0vc+mvjtkGYNmUSKlWdfc5Q8uWzvZ2NpVFo2g5s=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=p08Nnba5CYMaatFqWY9C/1uT9n0CaBGu1cK5AOuM/TZI/LqP/9eE6+zEJBaX3Hj6vRcMDCHB/zXLUUV9N4BQy16z0orK2IC9jh8qMgWsTkRbClMGFgdIjGRgnFFPg+IRUOAhKGUFOuIi7b2FyEhLHWcUP93cSTsU+qLpgRLe1qk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=kacXCTsG; arc=none smtp.client-ip=192.198.163.17 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="kacXCTsG" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1711348935; x=1742884935; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=16GQ0vc+mvjtkGYNmUSKlWdfc5Q8uWzvZ2NpVFo2g5s=; b=kacXCTsGv4OpHRPuTJrcg6HgzlpoQXPEGWjLDeYr8GcjKPFx1yYrrZGp n/cSY714LqlZGpKjqYkRp+MIMZNv0vPBqeoSi/L2K8Pc1qYpn1lWbBqeb sB43kHwjXHxWLITOHq5h5jykb0Ccd+UrWY2hizkMFj1ifyGHemoEF21bX etXVgkQJiybwSp6UyvjhQpdokIwWnl/TlGTqD48DD70+vO6NblqwZF9ne GlBR7wmfmLMSMSTiSWV/T/XFqpAgrDno+UVHGUlvFpbgWUP8iJVxfj+Bt VPtq/cm2ee/NcqBhAnVJmAEEnFHFCeOSPEAi7yvRexcfOxwX74jWBFHgY A==; X-IronPort-AV: E=McAfee;i="6600,9927,11023"; a="6191502" X-IronPort-AV: E=Sophos;i="6.07,152,1708416000"; d="scan'208";a="6191502" Received: from fmviesa002.fm.intel.com ([10.60.135.142]) by fmvoesa111.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Mar 2024 23:42:09 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,152,1708416000"; d="scan'208";a="38629669" Received: from ahunter6-mobl1.ger.corp.intel.com (HELO ahunter-VirtualBox.home\044ger.corp.intel.com) ([10.251.211.155]) by fmviesa002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Mar 2024 23:41:58 -0700 From: Adrian Hunter To: Thomas Gleixner Cc: Michael Ellerman , Nicholas Piggin , Christophe Leroy , "Aneesh Kumar K.V" , "Naveen N. Rao" , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Andy Lutomirski , Vincenzo Frascino , John Stultz , Stephen Boyd , Peter Zijlstra , Randy Dunlap , Bjorn Helgaas , Arnd Bergmann , Anna-Maria Behnsen , linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, linux-s390@vger.kernel.org Subject: [PATCH V2 13/19] timekeeping: Refactor timekeeping helpers Date: Mon, 25 Mar 2024 08:40:17 +0200 Message-Id: <20240325064023.2997-14-adrian.hunter@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240325064023.2997-1-adrian.hunter@intel.com> References: <20240325064023.2997-1-adrian.hunter@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Organization: Intel Finland Oy, Registered Address: PL 281, 00181 Helsinki, Business Identity Code: 0357606 - 4, Domiciled in Helsinki Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Simplify use of timekeeping sanity checking, in preparation for consolidating timekeeping helpers. This works towards eliminating timekeeping_delta_to_ns() in favour of timekeeping_cycles_to_ns(). Suggested-by: Thomas Gleixner Signed-off-by: Adrian Hunter --- kernel/time/timekeeping.c | 20 ++++++-------------- 1 file changed, 6 insertions(+), 14 deletions(-) diff --git a/kernel/time/timekeeping.c b/kernel/time/timekeeping.c index f81d675291e0..618328cd4bc4 100644 --- a/kernel/time/timekeeping.c +++ b/kernel/time/timekeeping.c @@ -237,7 +237,7 @@ static void timekeeping_check_update(struct timekeeper = *tk, u64 offset) } } =20 -static inline u64 timekeeping_get_delta(const struct tk_read_base *tkr) +static inline u64 timekeeping_debug_get_delta(const struct tk_read_base *t= kr) { struct timekeeper *tk =3D &tk_core.timekeeper; u64 now, last, mask, max, delta; @@ -281,17 +281,9 @@ static inline u64 timekeeping_get_delta(const struct t= k_read_base *tkr) static inline void timekeeping_check_update(struct timekeeper *tk, u64 off= set) { } -static inline u64 timekeeping_get_delta(const struct tk_read_base *tkr) +static inline u64 timekeeping_debug_get_delta(const struct tk_read_base *t= kr) { - u64 cycle_now, delta; - - /* read clocksource */ - cycle_now =3D tk_clock_read(tkr); - - /* calculate the delta since the last update_wall_time */ - delta =3D clocksource_delta(cycle_now, tkr->cycle_last, tkr->mask); - - return delta; + BUG(); } #endif =20 @@ -396,10 +388,10 @@ static __always_inline u64 __timekeeping_get_ns(const= struct tk_read_base *tkr) =20 static inline u64 timekeeping_get_ns(const struct tk_read_base *tkr) { - u64 delta; + if (IS_ENABLED(CONFIG_DEBUG_TIMEKEEPING)) + return timekeeping_delta_to_ns(tkr, timekeeping_debug_get_delta(tkr)); =20 - delta =3D timekeeping_get_delta(tkr); - return timekeeping_delta_to_ns(tkr, delta); + return __timekeeping_get_ns(tkr); } =20 /** --=20 2.34.1 From nobody Sat Feb 7 11:31:10 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 688C61607A8; Mon, 25 Mar 2024 06:42:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.17 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711348938; cv=none; b=NJmunuLwrk5asK+cfFDPjPeB02ItgKdzOmTuVHKmLmItfL/Ym01nPaYDBnmg8dwKR7P5/D+wDqTQIChnZbGqyavkjZOFZVkR5W0wRrzdFC1M2Gi20EnsKin4kKwTIBk9HK/WFr9Efk0MMIoIZ25nDq4tg+M5mj3T6BpJapP/sTo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711348938; c=relaxed/simple; bh=I/EwrCmdvQfW3F1Dq8qT6GRyUT8dQLcc34j3rdWdn4M=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=ely7px65MhMkmqMkyWutyHuxxVSor1JnwQZfTuC7sA1Wk4II8ojecNjDUVgueMeK2c8wihSDiE+B78GvuSP7PckrCacP52XR55oGronj6AFX9aH/ry5D+P6941kSh0C+Rrer+AqIQcbAX0KFB8QGAlA9sMSVedyDrT839FRY/8g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=mE13p/MM; arc=none smtp.client-ip=192.198.163.17 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="mE13p/MM" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1711348937; x=1742884937; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=I/EwrCmdvQfW3F1Dq8qT6GRyUT8dQLcc34j3rdWdn4M=; b=mE13p/MM7yPeotba7fz8QdXsEs13TycvCiaJm1eGYiWn4HHsxjFBKmHA WkBRHvkwKWNa8+pyRs+D7A5TXd5B4KGWxjB8Ga9VlulKKZtVXWbHBKLjv 72udsjtT4KjWc+oTN1/1R8ddhaU4c0qp/Tf0mtx7P40KrsnE1cal4Q8DK Fox9War5vxx+KLSraVPnEyD+siFKVKllV41QyhPoUd3LqCuKOiLMp5kbf EyxdufrYG2auz1rzcot4R9p8g9KWZPs+AlQJAtjwdOx8EQxGUixVHwfal gE3yjoNPKE2b2K2SRQriiDxc3n4+QObhb3JFFwqfaYpijrsRMFrN8N2Um w==; X-IronPort-AV: E=McAfee;i="6600,9927,11023"; a="6191517" X-IronPort-AV: E=Sophos;i="6.07,152,1708416000"; d="scan'208";a="6191517" Received: from fmviesa002.fm.intel.com ([10.60.135.142]) by fmvoesa111.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Mar 2024 23:42:11 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,152,1708416000"; d="scan'208";a="38629681" Received: from ahunter6-mobl1.ger.corp.intel.com (HELO ahunter-VirtualBox.home\044ger.corp.intel.com) ([10.251.211.155]) by fmviesa002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Mar 2024 23:42:04 -0700 From: Adrian Hunter To: Thomas Gleixner Cc: Michael Ellerman , Nicholas Piggin , Christophe Leroy , "Aneesh Kumar K.V" , "Naveen N. Rao" , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Andy Lutomirski , Vincenzo Frascino , John Stultz , Stephen Boyd , Peter Zijlstra , Randy Dunlap , Bjorn Helgaas , Arnd Bergmann , Anna-Maria Behnsen , linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, linux-s390@vger.kernel.org Subject: [PATCH V2 14/19] timekeeping: Consolidate timekeeping helpers Date: Mon, 25 Mar 2024 08:40:18 +0200 Message-Id: <20240325064023.2997-15-adrian.hunter@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240325064023.2997-1-adrian.hunter@intel.com> References: <20240325064023.2997-1-adrian.hunter@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Organization: Intel Finland Oy, Registered Address: PL 281, 00181 Helsinki, Business Identity Code: 0357606 - 4, Domiciled in Helsinki Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Consolidate timekeeping helpers, making use of timekeeping_cycles_to_ns() in preference to directly using timekeeping_delta_to_ns(). Suggested-by: Thomas Gleixner Signed-off-by: Adrian Hunter --- kernel/time/timekeeping.c | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/kernel/time/timekeeping.c b/kernel/time/timekeeping.c index 618328cd4bc4..1bbfe1ff8d24 100644 --- a/kernel/time/timekeeping.c +++ b/kernel/time/timekeeping.c @@ -237,7 +237,9 @@ static void timekeeping_check_update(struct timekeeper = *tk, u64 offset) } } =20 -static inline u64 timekeeping_debug_get_delta(const struct tk_read_base *t= kr) +static inline u64 timekeeping_cycles_to_ns(const struct tk_read_base *tkr,= u64 cycles); + +static inline u64 timekeeping_debug_get_ns(const struct tk_read_base *tkr) { struct timekeeper *tk =3D &tk_core.timekeeper; u64 now, last, mask, max, delta; @@ -266,22 +268,22 @@ static inline u64 timekeeping_debug_get_delta(const s= truct tk_read_base *tkr) */ if (unlikely((~delta & mask) < (mask >> 3))) { tk->underflow_seen =3D 1; - delta =3D 0; + now =3D last; } =20 /* Cap delta value to the max_cycles values to avoid mult overflows */ if (unlikely(delta > max)) { tk->overflow_seen =3D 1; - delta =3D tkr->clock->max_cycles; + now =3D last + max; } =20 - return delta; + return timekeeping_cycles_to_ns(tkr, now); } #else static inline void timekeeping_check_update(struct timekeeper *tk, u64 off= set) { } -static inline u64 timekeeping_debug_get_delta(const struct tk_read_base *t= kr) +static inline u64 timekeeping_debug_get_ns(const struct tk_read_base *tkr) { BUG(); } @@ -389,7 +391,7 @@ static __always_inline u64 __timekeeping_get_ns(const s= truct tk_read_base *tkr) static inline u64 timekeeping_get_ns(const struct tk_read_base *tkr) { if (IS_ENABLED(CONFIG_DEBUG_TIMEKEEPING)) - return timekeeping_delta_to_ns(tkr, timekeeping_debug_get_delta(tkr)); + return timekeeping_debug_get_ns(tkr); =20 return __timekeeping_get_ns(tkr); } --=20 2.34.1 From nobody Sat Feb 7 11:31:10 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1DD8415FD0F; Mon, 25 Mar 2024 06:42:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.17 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711348940; cv=none; b=hWIj8TH68KIcimqqb24eij+fXz7nq3eZ+/cJbg3J86bBXaixDasF/PdmXcgLY+LajlDzbpQqlm2yt8Yl8WBIkumJsrRW7zF7ws1BGFklfrgHttIOFnwKmmnRCbtWfEDfoHfl3x70xZdNK5603NUa12oSHdNTh0VnjBL91SEEBtg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711348940; c=relaxed/simple; bh=XagHsKZy0o0b/SfCKSGfthtDX1F7t80ikEGTTX3TaPM=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=eEpEZiOEnr6Quqnldh66MGafCyPmlGhtFW8b7foHZSBCt1GPxM96jBUJ7AeJPGwTzDU0wdXl2ghS29nZWiXV6vJDgi0Rck4fqgcI4DITGYLwDX0pj9ioTlN0Z9bjXNPYILi2P4uR4p0sdM61wSqaKrrZPjTa7dCQPWvAnDmgfpM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=ckqX5eEn; arc=none smtp.client-ip=192.198.163.17 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="ckqX5eEn" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1711348939; x=1742884939; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=XagHsKZy0o0b/SfCKSGfthtDX1F7t80ikEGTTX3TaPM=; b=ckqX5eEniHY+A6sK/F41fC5asgkog6yJ0hGbXmo1RNw3f8lqLii86HlO 6UviiCgfDFKMhGw7rZ9Ni0DMvVT7BzDGhw0TfjXFp14tS3BAcgYUFRaZ1 xPWF0tUMSgZLmkq9DX6PP3jzqoCk+aA5QMLsHfaQE0kaLEEFcVxsefziE vMXgNDZ889wHbsdj5eR+8mH6ER7G3Cy5Fkkrka6E0DnYlaTePjkQMnOVT PyXYJj3jF22TmwrVdLLdG3G7/a5IRp7T29MfVTeGy6TboaGE5Krs3zM7I Si1L1YWBuwQMurEJSde9ghqFhu17noba7FII2sWx5DJlNwEPL+SBJeQXR A==; X-IronPort-AV: E=McAfee;i="6600,9927,11023"; a="6191550" X-IronPort-AV: E=Sophos;i="6.07,152,1708416000"; d="scan'208";a="6191550" Received: from fmviesa002.fm.intel.com ([10.60.135.142]) by fmvoesa111.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Mar 2024 23:42:17 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,152,1708416000"; d="scan'208";a="38629691" Received: from ahunter6-mobl1.ger.corp.intel.com (HELO ahunter-VirtualBox.home\044ger.corp.intel.com) ([10.251.211.155]) by fmviesa002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Mar 2024 23:42:11 -0700 From: Adrian Hunter To: Thomas Gleixner Cc: Michael Ellerman , Nicholas Piggin , Christophe Leroy , "Aneesh Kumar K.V" , "Naveen N. Rao" , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Andy Lutomirski , Vincenzo Frascino , John Stultz , Stephen Boyd , Peter Zijlstra , Randy Dunlap , Bjorn Helgaas , Arnd Bergmann , Anna-Maria Behnsen , linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, linux-s390@vger.kernel.org Subject: [PATCH V2 15/19] timekeeping: Fold in timekeeping_delta_to_ns() Date: Mon, 25 Mar 2024 08:40:19 +0200 Message-Id: <20240325064023.2997-16-adrian.hunter@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240325064023.2997-1-adrian.hunter@intel.com> References: <20240325064023.2997-1-adrian.hunter@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Organization: Intel Finland Oy, Registered Address: PL 281, 00181 Helsinki, Business Identity Code: 0357606 - 4, Domiciled in Helsinki Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" timekeeping_delta_to_ns() is now called only from timekeeping_cycles_to_ns(), and it is not useful otherwise. Simplify by folding it into timekeeping_cycles_to_ns(). Suggested-by: Thomas Gleixner Signed-off-by: Adrian Hunter --- kernel/time/timekeeping.c | 13 +------------ 1 file changed, 1 insertion(+), 12 deletions(-) diff --git a/kernel/time/timekeeping.c b/kernel/time/timekeeping.c index 1bbfe1ff8d24..749387f22f0f 100644 --- a/kernel/time/timekeeping.c +++ b/kernel/time/timekeeping.c @@ -364,23 +364,12 @@ static void tk_setup_internals(struct timekeeper *tk,= struct clocksource *clock) } =20 /* Timekeeper helper functions. */ - -static inline u64 timekeeping_delta_to_ns(const struct tk_read_base *tkr, = u64 delta) -{ - u64 nsec; - - nsec =3D delta * tkr->mult + tkr->xtime_nsec; - nsec >>=3D tkr->shift; - - return nsec; -} - static inline u64 timekeeping_cycles_to_ns(const struct tk_read_base *tkr,= u64 cycles) { /* Calculate the delta since the last update_wall_time() */ u64 delta =3D clocksource_delta(cycles, tkr->cycle_last, tkr->mask); =20 - return timekeeping_delta_to_ns(tkr, delta); + return ((delta * tkr->mult) + tkr->xtime_nsec) >> tkr->shift; } =20 static __always_inline u64 __timekeeping_get_ns(const struct tk_read_base = *tkr) --=20 2.34.1 From nobody Sat Feb 7 11:31:10 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 27B5A1607AB; Mon, 25 Mar 2024 06:42:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.17 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711348946; cv=none; b=H5hyoFRMqCV8HGi8nJZMSvVuUGjUvqxpfsAp7u4NZaPkHftGQVztT88aiOHeDuSNMrcIFnJl7ajt2wmvHD+VmvM2Wa1uLO0atb58X2QMcROr6ZF6c5CVAJvnVDn++UqWwSkUVxqk8fiMa2n2al3Z/bTdEatPcx7bCeuM+hZyRN4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711348946; c=relaxed/simple; bh=DWUH05pMFGBZtgh8QTM1fOtl5JuwFtwudYwKfS+CudI=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=bOfeaW7s/FePoBegQgL62/f/7RCGgt1pQHspHOinp+kQ5DGF/oJjaUd4Ne0n9nWnwGyFELWvutuCoxaDEVbqCOU2+TSffLKxlzEjPaFdl2vcWFp3Aq/ArvLuxmco5tg3cruYEPFmMNidklECBhmZ/mSNoQ+3TdxVM4jNJLJ7sTo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=GOEeOJN8; arc=none smtp.client-ip=192.198.163.17 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="GOEeOJN8" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1711348945; x=1742884945; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=DWUH05pMFGBZtgh8QTM1fOtl5JuwFtwudYwKfS+CudI=; b=GOEeOJN8Se0rzdnqooiTSNgPIv3jrJHcOdQeU8VbVlHlyPn+ZvUHGw8H JqLPNNlNd0BUDVH/HTsjmqoi6cO3rVX6b3d3ehaCuCClAKyd7868HquKS IDQqnjkn96TK7D1lKq9NhCtMx6Jaes90ayzl5LmeFkoXeN0DeT0HMX7iK QC1K+zuxHFDil83Bc1bobYe4m8zbUyp1EyFoLmf4vgZHV4KIKdQJfdw5B 0ceqEBJkw0wHh2KQD229rgQ6uekzLB729eyrSx/qaQWGwEMSevEPTNkhI d+mD9GKK0kj1swlvABC/6Hge9JJ4BtnsrgYdYGF89Ph4F/YTU0Wc3pLzV Q==; X-IronPort-AV: E=McAfee;i="6600,9927,11023"; a="6191558" X-IronPort-AV: E=Sophos;i="6.07,152,1708416000"; d="scan'208";a="6191558" Received: from fmviesa002.fm.intel.com ([10.60.135.142]) by fmvoesa111.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Mar 2024 23:42:24 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,152,1708416000"; d="scan'208";a="38629701" Received: from ahunter6-mobl1.ger.corp.intel.com (HELO ahunter-VirtualBox.home\044ger.corp.intel.com) ([10.251.211.155]) by fmviesa002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Mar 2024 23:42:18 -0700 From: Adrian Hunter To: Thomas Gleixner Cc: Michael Ellerman , Nicholas Piggin , Christophe Leroy , "Aneesh Kumar K.V" , "Naveen N. Rao" , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Andy Lutomirski , Vincenzo Frascino , John Stultz , Stephen Boyd , Peter Zijlstra , Randy Dunlap , Bjorn Helgaas , Arnd Bergmann , Anna-Maria Behnsen , linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, linux-s390@vger.kernel.org Subject: [PATCH V2 16/19] timekeeping: Prepare timekeeping_cycles_to_ns() for overflow safety Date: Mon, 25 Mar 2024 08:40:20 +0200 Message-Id: <20240325064023.2997-17-adrian.hunter@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240325064023.2997-1-adrian.hunter@intel.com> References: <20240325064023.2997-1-adrian.hunter@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Organization: Intel Finland Oy, Registered Address: PL 281, 00181 Helsinki, Business Identity Code: 0357606 - 4, Domiciled in Helsinki Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Open code clocksource_delta() in timekeeping_cycles_to_ns() so that overflow safety can be added efficiently. Suggested-by: Thomas Gleixner Signed-off-by: Adrian Hunter --- kernel/time/timekeeping.c | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/kernel/time/timekeeping.c b/kernel/time/timekeeping.c index 749387f22f0f..d17484082e2c 100644 --- a/kernel/time/timekeeping.c +++ b/kernel/time/timekeeping.c @@ -367,7 +367,17 @@ static void tk_setup_internals(struct timekeeper *tk, = struct clocksource *clock) static inline u64 timekeeping_cycles_to_ns(const struct tk_read_base *tkr,= u64 cycles) { /* Calculate the delta since the last update_wall_time() */ - u64 delta =3D clocksource_delta(cycles, tkr->cycle_last, tkr->mask); + u64 mask =3D tkr->mask, delta =3D (cycles - tkr->cycle_last) & mask; + + if (IS_ENABLED(CONFIG_CLOCKSOURCE_VALIDATE_LAST_CYCLE)) { + /* + * Handle clocksource inconsistency between CPUs to prevent + * time from going backwards by checking for the MSB of the + * mask being set in the delta. + */ + if (unlikely(delta & ~(mask >> 1))) + return tkr->xtime_nsec >> tkr->shift; + } =20 return ((delta * tkr->mult) + tkr->xtime_nsec) >> tkr->shift; } --=20 2.34.1 From nobody Sat Feb 7 11:31:10 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E99E316088F; Mon, 25 Mar 2024 06:42:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.17 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711348955; cv=none; b=i4wDC2Jx0OO3S2jz4JiK1/Dy0evGhNBLVe+qkfDzFk6+IH8S122GKA5BhmDW/gn1t7gT/mmS4F0y7e881n/9Wq82NxcLAUeKV0v7dcCSukD4Mtw126ljOmD98Md5Q1DUTkwDkAdXQ2QDCTiRjuiGHWJ0H5M3lBSwnmz898jkdUo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711348955; c=relaxed/simple; bh=xuu117cG1lowhe30UpLIG5BgAby+hnwaDFUtdrI4YDU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=DO7efubDDtCmzv8vv0d9QVQsTc1qBFUtSTjP2u673TWfIkf23EFAKLVlAl7eJyhSw1OUMsRPmrCK29THKtE+iVITZHBPQQ2Sjwjy/zbqPJZijuiaxRnCqO0q1SjxNSYHOlLT9CchuZtDukUKnmdSPjt7sUsErz5GdQSC0gOw8GI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=fiUweDl0; arc=none smtp.client-ip=192.198.163.17 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="fiUweDl0" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1711348954; x=1742884954; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=xuu117cG1lowhe30UpLIG5BgAby+hnwaDFUtdrI4YDU=; b=fiUweDl0XiWKu+0t4CCWMTTtgAkXMgZU6DdRsN3cLJsNHyTzagWnBDLF AfS4B3AsskVYSr8T0xZcyPmm9ScQuUkX1tGa4KqHFY/82z2dD3cb4UVWS vg9ctU+f4x651PySsdVzNe0ZKComNZNg1KnmW/KxquUCjK35VMYRjIcuc 3CnDV5C0//jww8kU0V0tz06F05QtJennssMva8F9IGWz+uhqRGYEEdl1T hx9uggZcnSg0ArEydwc+L1J6k9QT7MkgfStcEm9yv7NS9qxeiNuapn0z1 35zjIIJDI7dBFNPIObb5JtVUCPez15sGNLIXe8fCnkxhzlekMKteFp/fG Q==; X-IronPort-AV: E=McAfee;i="6600,9927,11023"; a="6191576" X-IronPort-AV: E=Sophos;i="6.07,152,1708416000"; d="scan'208";a="6191576" Received: from fmviesa002.fm.intel.com ([10.60.135.142]) by fmvoesa111.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Mar 2024 23:42:31 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,152,1708416000"; d="scan'208";a="38629707" Received: from ahunter6-mobl1.ger.corp.intel.com (HELO ahunter-VirtualBox.home\044ger.corp.intel.com) ([10.251.211.155]) by fmviesa002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Mar 2024 23:42:25 -0700 From: Adrian Hunter To: Thomas Gleixner Cc: Michael Ellerman , Nicholas Piggin , Christophe Leroy , "Aneesh Kumar K.V" , "Naveen N. Rao" , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Andy Lutomirski , Vincenzo Frascino , John Stultz , Stephen Boyd , Peter Zijlstra , Randy Dunlap , Bjorn Helgaas , Arnd Bergmann , Anna-Maria Behnsen , linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, linux-s390@vger.kernel.org Subject: [PATCH V2 17/19] timekeeping: Make delta calculation overflow safe Date: Mon, 25 Mar 2024 08:40:21 +0200 Message-Id: <20240325064023.2997-18-adrian.hunter@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240325064023.2997-1-adrian.hunter@intel.com> References: <20240325064023.2997-1-adrian.hunter@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Organization: Intel Finland Oy, Registered Address: PL 281, 00181 Helsinki, Business Identity Code: 0357606 - 4, Domiciled in Helsinki Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Kernel timekeeping is designed to keep the change in cycles (since the last timer interrupt) below max_cycles, which prevents multiplication overflow when converting cycles to nanoseconds. However, if timer interrupts stop, the calculation will eventually overflow. Add protection against that. In timekeeping_cycles_to_ns() calculation, check against max_cycles, falling back to a slower higher precision calculation. In timekeeping_forward_now(), process delta in chunks of at most max_cycles. Suggested-by: Thomas Gleixner Signed-off-by: Adrian Hunter --- kernel/time/timekeeping.c | 40 ++++++++++++++++++++++++++++----------- 1 file changed, 29 insertions(+), 11 deletions(-) diff --git a/kernel/time/timekeeping.c b/kernel/time/timekeeping.c index d17484082e2c..111dfdbd488f 100644 --- a/kernel/time/timekeeping.c +++ b/kernel/time/timekeeping.c @@ -364,19 +364,32 @@ static void tk_setup_internals(struct timekeeper *tk,= struct clocksource *clock) } =20 /* Timekeeper helper functions. */ +static noinline u64 delta_to_ns_safe(const struct tk_read_base *tkr, u64 d= elta) +{ + return mul_u64_u32_add_u64_shr(delta, tkr->mult, tkr->xtime_nsec, tkr->sh= ift); +} + static inline u64 timekeeping_cycles_to_ns(const struct tk_read_base *tkr,= u64 cycles) { /* Calculate the delta since the last update_wall_time() */ u64 mask =3D tkr->mask, delta =3D (cycles - tkr->cycle_last) & mask; =20 - if (IS_ENABLED(CONFIG_CLOCKSOURCE_VALIDATE_LAST_CYCLE)) { - /* - * Handle clocksource inconsistency between CPUs to prevent - * time from going backwards by checking for the MSB of the - * mask being set in the delta. - */ - if (unlikely(delta & ~(mask >> 1))) - return tkr->xtime_nsec >> tkr->shift; + /* + * This detects the case where the delta overflows the multiplication + * with tkr->mult. + */ + if (unlikely(delta > tkr->clock->max_cycles)) { + if (IS_ENABLED(CONFIG_CLOCKSOURCE_VALIDATE_LAST_CYCLE)) { + /* + * Handle clocksource inconsistency between CPUs to prevent + * time from going backwards by checking for the MSB of the + * mask being set in the delta. + */ + if (unlikely(delta & ~(mask >> 1))) + return tkr->xtime_nsec >> tkr->shift; + } + + return delta_to_ns_safe(tkr, delta); } =20 return ((delta * tkr->mult) + tkr->xtime_nsec) >> tkr->shift; @@ -789,10 +802,15 @@ static void timekeeping_forward_now(struct timekeeper= *tk) tk->tkr_mono.cycle_last =3D cycle_now; tk->tkr_raw.cycle_last =3D cycle_now; =20 - tk->tkr_mono.xtime_nsec +=3D delta * tk->tkr_mono.mult; - tk->tkr_raw.xtime_nsec +=3D delta * tk->tkr_raw.mult; + while (delta > 0) { + u64 max =3D tk->tkr_mono.clock->max_cycles; + u64 incr =3D delta < max ? delta : max; =20 - tk_normalize_xtime(tk); + tk->tkr_mono.xtime_nsec +=3D incr * tk->tkr_mono.mult; + tk->tkr_raw.xtime_nsec +=3D incr * tk->tkr_raw.mult; + tk_normalize_xtime(tk); + delta -=3D incr; + } } =20 /** --=20 2.34.1 From nobody Sat Feb 7 11:31:10 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9BBCF16131F; Mon, 25 Mar 2024 06:42:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.17 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711348961; cv=none; b=GZplR6lTWEycOPWKtaSOaVZmc/kmWjXvd+hlCXqWa8d5zPBdCj6c390QdNKjCanNODDiTO5KeVw/PfBM8oGyOw/bQ8tt3iA7ir7jl4BxOpHGdwehXo0shI26sboSZDKvx4ZNbL4JMWN8O2SjdeY0WRndo6KaLfoGlO21QS+Ip5k= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711348961; c=relaxed/simple; bh=qOOcsyBxpUsFxkAH1hi2UNpYCz2148dQEbBRMVKVw/4=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=rAvk1vyUzTjzbcQHFHntucgHdsemm2MLjTib9UItFITf6c1x5AAnZLF6YsClzbG9JFusen9REPTJyG+sc5+VJB0QBRGQYR7zJJ0lR8YD1/a+u/xT9ENhwL1xjhWhLeWqgDrt4XeUybCNIGVXl57ehWIlgPcHx4U37VQRX9OxHts= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=Ccf6e5TX; arc=none smtp.client-ip=192.198.163.17 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="Ccf6e5TX" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1711348959; x=1742884959; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=qOOcsyBxpUsFxkAH1hi2UNpYCz2148dQEbBRMVKVw/4=; b=Ccf6e5TX60YVh3RAGKD3v7K8r7FMP6j0jRM+vq7+0d4MdzRYVxan1guz wjBg9bugYYqDetiF6mtAuSfZolfOphFky2rbAwxGjZcQTO2MGxfJjxllV tZ8AD+2EtZZX1TnIuC55tGvCkFaLRxlgjIxes2bLe2VqUtLaAAQS5JtHY HJFFtvI7h/4A6LoqnF5uXX8zVnbbe0C2Dn6nMtkDsrdR/BT+icAlOM54e FfLjaKi1AJiGWWoXVBtIjS2tKUY9xSu9mfPuur7Y/+eHnYxS8Y+vkJ+12 8MnMGxkUoqxC48em02KDlKZW+Ub7CDlJrp/ElQOchg/RCOwjSKBlT2Yv9 g==; X-IronPort-AV: E=McAfee;i="6600,9927,11023"; a="6191594" X-IronPort-AV: E=Sophos;i="6.07,152,1708416000"; d="scan'208";a="6191594" Received: from fmviesa002.fm.intel.com ([10.60.135.142]) by fmvoesa111.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Mar 2024 23:42:38 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,152,1708416000"; d="scan'208";a="38629710" Received: from ahunter6-mobl1.ger.corp.intel.com (HELO ahunter-VirtualBox.home\044ger.corp.intel.com) ([10.251.211.155]) by fmviesa002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Mar 2024 23:42:31 -0700 From: Adrian Hunter To: Thomas Gleixner Cc: Michael Ellerman , Nicholas Piggin , Christophe Leroy , "Aneesh Kumar K.V" , "Naveen N. Rao" , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Andy Lutomirski , Vincenzo Frascino , John Stultz , Stephen Boyd , Peter Zijlstra , Randy Dunlap , Bjorn Helgaas , Arnd Bergmann , Anna-Maria Behnsen , linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, linux-s390@vger.kernel.org Subject: [PATCH V2 18/19] timekeeping: Let timekeeping_cycles_to_ns() handle both under and overflow Date: Mon, 25 Mar 2024 08:40:22 +0200 Message-Id: <20240325064023.2997-19-adrian.hunter@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240325064023.2997-1-adrian.hunter@intel.com> References: <20240325064023.2997-1-adrian.hunter@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Organization: Intel Finland Oy, Registered Address: PL 281, 00181 Helsinki, Business Identity Code: 0357606 - 4, Domiciled in Helsinki Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" For the case !CONFIG_CLOCKSOURCE_VALIDATE_LAST_CYCLE, forego overflow protection in the range (mask << 1) < delta <=3D mask, and interpret it always as an inconsistency between CPU clock values. That allows slightly neater code, and it is on a slow path so has no effect on performance. Suggested-by: Thomas Gleixner Signed-off-by: Adrian Hunter --- kernel/time/timekeeping.c | 31 +++++++++++++------------------ 1 file changed, 13 insertions(+), 18 deletions(-) diff --git a/kernel/time/timekeeping.c b/kernel/time/timekeeping.c index 111dfdbd488f..4e18db1819f8 100644 --- a/kernel/time/timekeeping.c +++ b/kernel/time/timekeeping.c @@ -266,17 +266,14 @@ static inline u64 timekeeping_debug_get_ns(const stru= ct tk_read_base *tkr) * Try to catch underflows by checking if we are seeing small * mask-relative negative values. */ - if (unlikely((~delta & mask) < (mask >> 3))) { + if (unlikely((~delta & mask) < (mask >> 3))) tk->underflow_seen =3D 1; - now =3D last; - } =20 - /* Cap delta value to the max_cycles values to avoid mult overflows */ - if (unlikely(delta > max)) { + /* Check for multiplication overflows */ + if (unlikely(delta > max)) tk->overflow_seen =3D 1; - now =3D last + max; - } =20 + /* timekeeping_cycles_to_ns() handles both under and overflow */ return timekeeping_cycles_to_ns(tkr, now); } #else @@ -375,19 +372,17 @@ static inline u64 timekeeping_cycles_to_ns(const stru= ct tk_read_base *tkr, u64 c u64 mask =3D tkr->mask, delta =3D (cycles - tkr->cycle_last) & mask; =20 /* - * This detects the case where the delta overflows the multiplication - * with tkr->mult. + * This detects both negative motion and the case where the delta + * overflows the multiplication with tkr->mult. */ if (unlikely(delta > tkr->clock->max_cycles)) { - if (IS_ENABLED(CONFIG_CLOCKSOURCE_VALIDATE_LAST_CYCLE)) { - /* - * Handle clocksource inconsistency between CPUs to prevent - * time from going backwards by checking for the MSB of the - * mask being set in the delta. - */ - if (unlikely(delta & ~(mask >> 1))) - return tkr->xtime_nsec >> tkr->shift; - } + /* + * Handle clocksource inconsistency between CPUs to prevent + * time from going backwards by checking for the MSB of the + * mask being set in the delta. + */ + if (delta & ~(mask >> 1)) + return tkr->xtime_nsec >> tkr->shift; =20 return delta_to_ns_safe(tkr, delta); } --=20 2.34.1 From nobody Sat Feb 7 11:31:10 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 022AA16132B; Mon, 25 Mar 2024 06:42:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.17 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711348966; cv=none; b=gK2JwSk6GyYwVJ9S4Pb92Mox71mS1NyhR9m0kblJ4eDLhIYm6kVYDJWtJdhve7qcvUaqtyuu7ofEXgPVWEuiXXYPbSmJ6mg0q9pSiosuznYe8UPQZo8ij+MPAY4iGsnYtJ3cLScmn9tWEYfbwTEPvpzQUMwMF8XtYmelO7waH58= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711348966; c=relaxed/simple; bh=jWLLlF4KdyRZzGfn4U2lSdQ4Wei7Cj+yXLsz+I6EvZY=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=LE9FZcWd3irT9IPTb6zwdxVOrwrBSrf9nEwHBckP7kQimDWD9bVhpT5Ph+vvMDUeZsbOxOSoKU00Bu7Ds0FKlKWK1mhu10nZ16Z1gNfkOaqTreVELOE+ScKRSIrQSOAGwP6ru/hjFEuEqcHY63BV7mHkNBl2EujTjzooME7ZAbg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=Iusz8THA; arc=none smtp.client-ip=192.198.163.17 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="Iusz8THA" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1711348965; x=1742884965; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=jWLLlF4KdyRZzGfn4U2lSdQ4Wei7Cj+yXLsz+I6EvZY=; b=Iusz8THA13pqubRw6kvkL5p5vNxVzykvnS/33uQZKIHDT+sb1IA3LO5W OqgjDeLnZvl/26QtHvzEliO4MDb+09t6wQm8fEFboF5kAPLoJT0mNBkPA Y3drThHz2EovOYR9Dx4xU9d1XlODfXO44aVF+ya3xqPSrh/Pv+3FyjmPN PykU0G74QD/wy7uaamV/+vSpJP1sRHWVPke0Oz6Lv7ckGvAzsf3E44uED jd1pJwNHD+9Ih+5c2RkTvIiTFcLY+tPyO2ilQCRUHNbzbhc+hFiV5JMcc Tpr3x3Jng+qxMTIpEaFugbo5lml35Xbt2Z1bH66mkH40gzU/xps9bBVT8 w==; X-IronPort-AV: E=McAfee;i="6600,9927,11023"; a="6191609" X-IronPort-AV: E=Sophos;i="6.07,152,1708416000"; d="scan'208";a="6191609" Received: from fmviesa002.fm.intel.com ([10.60.135.142]) by fmvoesa111.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Mar 2024 23:42:44 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,152,1708416000"; d="scan'208";a="38629719" Received: from ahunter6-mobl1.ger.corp.intel.com (HELO ahunter-VirtualBox.home\044ger.corp.intel.com) ([10.251.211.155]) by fmviesa002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Mar 2024 23:42:38 -0700 From: Adrian Hunter To: Thomas Gleixner Cc: Michael Ellerman , Nicholas Piggin , Christophe Leroy , "Aneesh Kumar K.V" , "Naveen N. Rao" , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Andy Lutomirski , Vincenzo Frascino , John Stultz , Stephen Boyd , Peter Zijlstra , Randy Dunlap , Bjorn Helgaas , Arnd Bergmann , Anna-Maria Behnsen , linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, linux-s390@vger.kernel.org Subject: [PATCH V2 19/19] clocksource: Make watchdog and suspend-timing multiplication overflow safe Date: Mon, 25 Mar 2024 08:40:23 +0200 Message-Id: <20240325064023.2997-20-adrian.hunter@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240325064023.2997-1-adrian.hunter@intel.com> References: <20240325064023.2997-1-adrian.hunter@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Organization: Intel Finland Oy, Registered Address: PL 281, 00181 Helsinki, Business Identity Code: 0357606 - 4, Domiciled in Helsinki Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Kernel timekeeping is designed to keep the change in cycles (since the last timer interrupt) below max_cycles, which prevents multiplication overflow when converting cycles to nanoseconds. However, if timer interrupts stop, the clocksource_cyc2ns() calculation will eventually overflow. Add protection against that. Simplify by folding together clocksource_delta() and clocksource_cyc2ns() into cycles_to_nsec_safe(). Check against max_cycles, falling back to a slower higher precision calculation. Suggested-by: Thomas Gleixner Signed-off-by: Adrian Hunter --- kernel/time/clocksource.c | 42 +++++++++++++++++++-------------------- 1 file changed, 20 insertions(+), 22 deletions(-) diff --git a/kernel/time/clocksource.c b/kernel/time/clocksource.c index e5b260aa0e02..4d50d53ac719 100644 --- a/kernel/time/clocksource.c +++ b/kernel/time/clocksource.c @@ -20,6 +20,16 @@ #include "tick-internal.h" #include "timekeeping_internal.h" =20 +static noinline u64 cycles_to_nsec_safe(struct clocksource *cs, u64 start,= u64 end) +{ + u64 delta =3D clocksource_delta(end, start, cs->mask); + + if (likely(delta < cs->max_cycles)) + return clocksource_cyc2ns(delta, cs->mult, cs->shift); + + return mul_u64_u32_shr(delta, cs->mult, cs->shift); +} + /** * clocks_calc_mult_shift - calculate mult/shift factors for scaled math o= f clocks * @mult: pointer to mult variable @@ -222,8 +232,8 @@ enum wd_read_status { static enum wd_read_status cs_watchdog_read(struct clocksource *cs, u64 *c= snow, u64 *wdnow) { unsigned int nretries, max_retries; - u64 wd_end, wd_end2, wd_delta; int64_t wd_delay, wd_seq_delay; + u64 wd_end, wd_end2; =20 max_retries =3D clocksource_get_max_watchdog_retry(); for (nretries =3D 0; nretries <=3D max_retries; nretries++) { @@ -234,9 +244,7 @@ static enum wd_read_status cs_watchdog_read(struct cloc= ksource *cs, u64 *csnow, wd_end2 =3D watchdog->read(watchdog); local_irq_enable(); =20 - wd_delta =3D clocksource_delta(wd_end, *wdnow, watchdog->mask); - wd_delay =3D clocksource_cyc2ns(wd_delta, watchdog->mult, - watchdog->shift); + wd_delay =3D cycles_to_nsec_safe(watchdog, *wdnow, wd_end); if (wd_delay <=3D WATCHDOG_MAX_SKEW) { if (nretries > 1 || nretries >=3D max_retries) { pr_warn("timekeeping watchdog on CPU%d: %s retried %d times before suc= cess\n", @@ -254,8 +262,7 @@ static enum wd_read_status cs_watchdog_read(struct cloc= ksource *cs, u64 *csnow, * report system busy, reinit the watchdog and skip the current * watchdog test. */ - wd_delta =3D clocksource_delta(wd_end2, wd_end, watchdog->mask); - wd_seq_delay =3D clocksource_cyc2ns(wd_delta, watchdog->mult, watchdog->= shift); + wd_seq_delay =3D cycles_to_nsec_safe(watchdog, wd_end, wd_end2); if (wd_seq_delay > WATCHDOG_MAX_SKEW/2) goto skip_test; } @@ -366,8 +373,7 @@ void clocksource_verify_percpu(struct clocksource *cs) delta =3D (csnow_end - csnow_mid) & cs->mask; if (delta < 0) cpumask_set_cpu(cpu, &cpus_ahead); - delta =3D clocksource_delta(csnow_end, csnow_begin, cs->mask); - cs_nsec =3D clocksource_cyc2ns(delta, cs->mult, cs->shift); + cs_nsec =3D cycles_to_nsec_safe(cs, csnow_begin, csnow_end); if (cs_nsec > cs_nsec_max) cs_nsec_max =3D cs_nsec; if (cs_nsec < cs_nsec_min) @@ -398,8 +404,8 @@ static inline void clocksource_reset_watchdog(void) =20 static void clocksource_watchdog(struct timer_list *unused) { - u64 csnow, wdnow, cslast, wdlast, delta; int64_t wd_nsec, cs_nsec, interval; + u64 csnow, wdnow, cslast, wdlast; int next_cpu, reset_pending; struct clocksource *cs; enum wd_read_status read_ret; @@ -456,12 +462,8 @@ static void clocksource_watchdog(struct timer_list *un= used) continue; } =20 - delta =3D clocksource_delta(wdnow, cs->wd_last, watchdog->mask); - wd_nsec =3D clocksource_cyc2ns(delta, watchdog->mult, - watchdog->shift); - - delta =3D clocksource_delta(csnow, cs->cs_last, cs->mask); - cs_nsec =3D clocksource_cyc2ns(delta, cs->mult, cs->shift); + wd_nsec =3D cycles_to_nsec_safe(watchdog, cs->wd_last, wdnow); + cs_nsec =3D cycles_to_nsec_safe(cs, cs->cs_last, csnow); wdlast =3D cs->wd_last; /* save these in case we print them */ cslast =3D cs->cs_last; cs->cs_last =3D csnow; @@ -832,7 +834,7 @@ void clocksource_start_suspend_timing(struct clocksourc= e *cs, u64 start_cycles) */ u64 clocksource_stop_suspend_timing(struct clocksource *cs, u64 cycle_now) { - u64 now, delta, nsec =3D 0; + u64 now, nsec =3D 0; =20 if (!suspend_clocksource) return 0; @@ -847,12 +849,8 @@ u64 clocksource_stop_suspend_timing(struct clocksource= *cs, u64 cycle_now) else now =3D suspend_clocksource->read(suspend_clocksource); =20 - if (now > suspend_start) { - delta =3D clocksource_delta(now, suspend_start, - suspend_clocksource->mask); - nsec =3D mul_u64_u32_shr(delta, suspend_clocksource->mult, - suspend_clocksource->shift); - } + if (now > suspend_start) + nsec =3D cycles_to_nsec_safe(suspend_clocksource, suspend_start, now); =20 /* * Disable the suspend timer to save power if current clocksource is --=20 2.34.1