From nobody Mon Feb 9 15:59:12 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 37D881F944; Fri, 8 Mar 2024 13:15:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.9 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709903742; cv=none; b=b5T211ePj12IP6rSz3DQuaf5iKjM3yS7tyJ74DHgTC6S6ERGUr1IWdfx38pCbxehqCRVLrA6fG18nmvK8Sz4QBP/JXVRVRk+mA3fPk53z5JEy9p62wLgusb/rFthxpwZq0mxLEXUyZ3lfcnIvU+OvH9AVgB5La5OorHBlXS3lNs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709903742; c=relaxed/simple; bh=Iuin2TLNITc+qEDBj9pIL7Ct7NIGQ3DNopSaa0alFbU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=s0IWNwfVwzuhd7OUvt6s49H3tiIi4gXEj1QzwyaG5s4/v00dbkM+4vnGnOiLbK/Xm47/v7XQlTW90UPK7iuXCorMKEdnIqNi7W7yNRahi18KTKMT4bjdoob9zZXj/wQ+YE86RcWwRRz/aTzerJ2BoICJByjhcInEPsc+caffNfg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=jzutiNMX; arc=none smtp.client-ip=192.198.163.9 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="jzutiNMX" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1709903740; x=1741439740; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Iuin2TLNITc+qEDBj9pIL7Ct7NIGQ3DNopSaa0alFbU=; b=jzutiNMX99aq5E9FQ4oXdAGS/TeSRWAVR0gYCQ5XQXbl1XE5hCMSI9gy ZbI6TyyK5pTFsQGGoRF6xBXUlkSEC1l8Wx5WN9yoyIVCcMYohg5O6kjVm RZlHI0a423r5vlbUhyUmEgGM4fM4DTiK7Rdjc34I3KLim33gf9IoAZQl7 vyU+t11P/2Kx2xhS3bFoNL6WWGIvOpXT7mWb7kcsVB5GFQ67MnmP5FWvt 0YdXuWG+Pxch0JdoDnXUb9UfcBcj+uLaVVpYr/ShkM7eYTv6KHhWOg3HH 5hvIkg1rMIiP2bXq9ps7hWWDQNuFFL/xpVp46b06+86dqqpphjFbTSZ6c w==; X-IronPort-AV: E=McAfee;i="6600,9927,11006"; a="15342392" X-IronPort-AV: E=Sophos;i="6.07,109,1708416000"; d="scan'208";a="15342392" Received: from orviesa005.jf.intel.com ([10.64.159.145]) by fmvoesa103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Mar 2024 05:15:35 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,109,1708416000"; d="scan'208";a="15161352" Received: from ahunter6-mobl1.ger.corp.intel.com (HELO ahunter-VirtualBox.home\044ger.corp.intel.com) ([10.249.46.63]) by orviesa005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Mar 2024 05:15:27 -0800 From: Adrian Hunter To: Thomas Gleixner Cc: Michael Ellerman , Nicholas Piggin , Christophe Leroy , "Aneesh Kumar K.V" , "Naveen N. Rao" , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Andy Lutomirski , Vincenzo Frascino , John Stultz , Stephen Boyd , Peter Zijlstra , Randy Dunlap , Bjorn Helgaas , Arnd Bergmann , Anna-Maria Behnsen , linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, linux-s390@vger.kernel.org Subject: [PATCH 01/19] vdso: Consolidate vdso_calc_delta() Date: Fri, 8 Mar 2024 15:14:54 +0200 Message-Id: <20240308131512.44324-2-adrian.hunter@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240308131512.44324-1-adrian.hunter@intel.com> References: <20240308131512.44324-1-adrian.hunter@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Organization: Intel Finland Oy, Registered Address: PL 281, 00181 Helsinki, Business Identity Code: 0357606 - 4, Domiciled in Helsinki Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Consolidate vdso_calc_delta(), in preparation for further simplification. Suggested-by: Thomas Gleixner Signed-off-by: Adrian Hunter --- arch/powerpc/include/asm/vdso/gettimeofday.h | 17 ++--------------- arch/s390/include/asm/vdso/gettimeofday.h | 7 ++----- lib/vdso/gettimeofday.c | 4 ++++ 3 files changed, 8 insertions(+), 20 deletions(-) diff --git a/arch/powerpc/include/asm/vdso/gettimeofday.h b/arch/powerpc/in= clude/asm/vdso/gettimeofday.h index f0a4cf01e85c..f4da8e18cdf3 100644 --- a/arch/powerpc/include/asm/vdso/gettimeofday.h +++ b/arch/powerpc/include/asm/vdso/gettimeofday.h @@ -14,6 +14,8 @@ =20 #define VDSO_HAS_TIME 1 =20 +#define VDSO_DELTA_NOMASK 1 + static __always_inline int do_syscall_2(const unsigned long _r0, const uns= igned long _r3, const unsigned long _r4) { @@ -105,21 +107,6 @@ static inline bool vdso_clocksource_ok(const struct vd= so_data *vd) } #define vdso_clocksource_ok vdso_clocksource_ok =20 -/* - * powerpc specific delta calculation. - * - * This variant removes the masking of the subtraction because the - * clocksource mask of all VDSO capable clocksources on powerpc is U64_MAX - * which would result in a pointless operation. The compiler cannot - * optimize it away as the mask comes from the vdso data and is not compile - * time constant. - */ -static __always_inline u64 vdso_calc_delta(u64 cycles, u64 last, u64 mask,= u32 mult) -{ - return (cycles - last) * mult; -} -#define vdso_calc_delta vdso_calc_delta - #ifndef __powerpc64__ static __always_inline u64 vdso_shift_ns(u64 ns, unsigned long shift) { diff --git a/arch/s390/include/asm/vdso/gettimeofday.h b/arch/s390/include/= asm/vdso/gettimeofday.h index db84942eb78f..7937765ccfa5 100644 --- a/arch/s390/include/asm/vdso/gettimeofday.h +++ b/arch/s390/include/asm/vdso/gettimeofday.h @@ -6,16 +6,13 @@ =20 #define VDSO_HAS_CLOCK_GETRES 1 =20 +#define VDSO_DELTA_NOMASK 1 + #include #include #include #include =20 -#define vdso_calc_delta __arch_vdso_calc_delta -static __always_inline u64 __arch_vdso_calc_delta(u64 cycles, u64 last, u6= 4 mask, u32 mult) -{ - return (cycles - last) * mult; -} =20 static __always_inline const struct vdso_data *__arch_get_vdso_data(void) { diff --git a/lib/vdso/gettimeofday.c b/lib/vdso/gettimeofday.c index ce2f69552003..042b95e8164d 100644 --- a/lib/vdso/gettimeofday.c +++ b/lib/vdso/gettimeofday.c @@ -13,7 +13,11 @@ static __always_inline u64 vdso_calc_delta(u64 cycles, u64 last, u64 mask, u32 mult) { +#ifdef VDSO_DELTA_NOMASK + return (cycles - last) * mult; +#else return ((cycles - last) & mask) * mult; +#endif } #endif =20 --=20 2.34.1 From nobody Mon Feb 9 15:59:12 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 535A33612C; Fri, 8 Mar 2024 13:15:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.9 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709903744; cv=none; b=V57tXz+p4Qi5+XtH0yP2soaZ0MrJE3/wU+LWuym2gNdnLMVkZPWDMLRpgyunW7rFRrBvTns6mfbFf9tmfP0nzEhmZoU62szs6q6QqfhenxPdhzBIsfJtyUzDj0gMCqJsjbdXjmFwvPqe4mbxasfH+hVjjoXf2FjkqUXU7mKYEHQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709903744; c=relaxed/simple; bh=BicpP74oj5pt4/R+FPIitTPzD9PABJ1vfjH9IXWSi8g=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=P/noxvMQ7mSgaHd8lI6DoQT6U4tPPSwnR+rm8j3xb2fV1IjhEPsne1f2dAr3SzxviB5BUFQGB+Idg11bi6LW1B6ko29uQCvq6z053Mubiivxpn5Ms8SCaLAS5LvBc/ydYUlJDpu8rZNN+DwFXQWtYzn2xW6A09EOLV2fcwdqQ94= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=NB7Jqa6v; arc=none smtp.client-ip=192.198.163.9 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="NB7Jqa6v" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1709903742; x=1741439742; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=BicpP74oj5pt4/R+FPIitTPzD9PABJ1vfjH9IXWSi8g=; b=NB7Jqa6vewlLZE6RprqYTi60C2g2QnEDjHyn+1qsDOISGj2nT7lD8Flr i7CEQ8NNRxcpmMKAGPVS7lxED+0AWLKZQVU1zehSnkSpLCYL5yQge051/ 3sRIcxqyluZapj467NBa7ckRXXPb4uUkaTeIc0yhGroSTG6wHzn82NJvn K0wlQVpUX6IRmo/tETJ8LF/tDLwtbIEJaL/5iXoMDb7mCg+1TTUhCjzH9 DC53T+2RXPi54OyI3IGfPkvmgUsYs5qb8p3axKRvqLr+zokvhFcZuQexZ /ouAD1cFJ/8HG2dAiEfSV5hzrMKu/sPkMLQf3AygOzZTROmROkTyVvVqk w==; X-IronPort-AV: E=McAfee;i="6600,9927,11006"; a="15342418" X-IronPort-AV: E=Sophos;i="6.07,109,1708416000"; d="scan'208";a="15342418" Received: from orviesa005.jf.intel.com ([10.64.159.145]) by fmvoesa103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Mar 2024 05:15:42 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,109,1708416000"; d="scan'208";a="15161362" Received: from ahunter6-mobl1.ger.corp.intel.com (HELO ahunter-VirtualBox.home\044ger.corp.intel.com) ([10.249.46.63]) by orviesa005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Mar 2024 05:15:34 -0800 From: Adrian Hunter To: Thomas Gleixner Cc: Michael Ellerman , Nicholas Piggin , Christophe Leroy , "Aneesh Kumar K.V" , "Naveen N. Rao" , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Andy Lutomirski , Vincenzo Frascino , John Stultz , Stephen Boyd , Peter Zijlstra , Randy Dunlap , Bjorn Helgaas , Arnd Bergmann , Anna-Maria Behnsen , linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, linux-s390@vger.kernel.org Subject: [PATCH 02/19] vdso: Consolidate nanoseconds calculation Date: Fri, 8 Mar 2024 15:14:55 +0200 Message-Id: <20240308131512.44324-3-adrian.hunter@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240308131512.44324-1-adrian.hunter@intel.com> References: <20240308131512.44324-1-adrian.hunter@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Organization: Intel Finland Oy, Registered Address: PL 281, 00181 Helsinki, Business Identity Code: 0357606 - 4, Domiciled in Helsinki Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Consolidate nanoseconds calculation to simplify and reduce code duplication. Suggested-by: Thomas Gleixner Signed-off-by: Adrian Hunter --- arch/x86/include/asm/vdso/gettimeofday.h | 17 +++++---- lib/vdso/gettimeofday.c | 44 +++++++++++------------- 2 files changed, 29 insertions(+), 32 deletions(-) diff --git a/arch/x86/include/asm/vdso/gettimeofday.h b/arch/x86/include/as= m/vdso/gettimeofday.h index 8e048ca980df..5727dedd3549 100644 --- a/arch/x86/include/asm/vdso/gettimeofday.h +++ b/arch/x86/include/asm/vdso/gettimeofday.h @@ -300,7 +300,7 @@ static inline bool arch_vdso_cycles_ok(u64 cycles) #define vdso_cycles_ok arch_vdso_cycles_ok =20 /* - * x86 specific delta calculation. + * x86 specific calculation of nanoseconds for the current cycle count * * The regular implementation assumes that clocksource reads are globally * monotonic. The TSC can be slightly off across sockets which can cause @@ -308,8 +308,8 @@ static inline bool arch_vdso_cycles_ok(u64 cycles) * jump. * * Therefore it needs to be verified that @cycles are greater than - * @last. If not then use @last, which is the base time of the current - * conversion period. + * @vd->cycles_last. If not then use @vd->cycles_last, which is the base + * time of the current conversion period. * * This variant also uses a custom mask because while the clocksource mask= of * all the VDSO capable clocksources on x86 is U64_MAX, the above code uses @@ -317,25 +317,24 @@ static inline bool arch_vdso_cycles_ok(u64 cycles) * declares everything with the MSB/Sign-bit set as invalid. Therefore the * effective mask is S64_MAX. */ -static __always_inline -u64 vdso_calc_delta(u64 cycles, u64 last, u64 mask, u32 mult) +static __always_inline u64 vdso_calc_ns(const struct vdso_data *vd, u64 cy= cles, u64 base) { /* * Due to the MSB/Sign-bit being used as invalid marker (see * arch_vdso_cycles_valid() above), the effective mask is S64_MAX. */ - u64 delta =3D (cycles - last) & S64_MAX; + u64 delta =3D (cycles - vd->cycle_last) & S64_MAX; =20 /* * Due to the above mentioned TSC wobbles, filter out negative motion. * Per the above masking, the effective sign bit is now bit 62. */ if (unlikely(delta & (1ULL << 62))) - return 0; + return base >> vd->shift; =20 - return delta * mult; + return ((delta * vd->mult) + base) >> vd->shift; } -#define vdso_calc_delta vdso_calc_delta +#define vdso_calc_ns vdso_calc_ns =20 #endif /* !__ASSEMBLY__ */ =20 diff --git a/lib/vdso/gettimeofday.c b/lib/vdso/gettimeofday.c index 042b95e8164d..9fa90e0794c9 100644 --- a/lib/vdso/gettimeofday.c +++ b/lib/vdso/gettimeofday.c @@ -5,20 +5,12 @@ #include #include =20 -#ifndef vdso_calc_delta -/* - * Default implementation which works for all sane clocksources. That - * obviously excludes x86/TSC. - */ -static __always_inline -u64 vdso_calc_delta(u64 cycles, u64 last, u64 mask, u32 mult) -{ +#ifndef vdso_calc_ns + #ifdef VDSO_DELTA_NOMASK - return (cycles - last) * mult; +# define VDSO_DELTA_MASK(vd) U64_MAX #else - return ((cycles - last) & mask) * mult; -#endif -} +# define VDSO_DELTA_MASK(vd) (vd->mask) #endif =20 #ifndef vdso_shift_ns @@ -28,6 +20,18 @@ static __always_inline u64 vdso_shift_ns(u64 ns, u32 shi= ft) } #endif =20 +/* + * Default implementation which works for all sane clocksources. That + * obviously excludes x86/TSC. + */ +static __always_inline u64 vdso_calc_ns(const struct vdso_data *vd, u64 cy= cles, u64 base) +{ + u64 delta =3D (cycles - vd->cycle_last) & VDSO_DELTA_MASK(vd); + + return vdso_shift_ns((delta * vd->mult) + base, vd->shift); +} +#endif /* vdso_calc_ns */ + #ifndef __arch_vdso_hres_capable static inline bool __arch_vdso_hres_capable(void) { @@ -53,10 +57,10 @@ static inline bool vdso_cycles_ok(u64 cycles) static __always_inline int do_hres_timens(const struct vdso_data *vdns, cl= ockid_t clk, struct __kernel_timespec *ts) { - const struct vdso_data *vd; const struct timens_offset *offs =3D &vdns->offset[clk]; const struct vdso_timestamp *vdso_ts; - u64 cycles, last, ns; + const struct vdso_data *vd; + u64 cycles, ns; u32 seq; s64 sec; =20 @@ -77,10 +81,7 @@ static __always_inline int do_hres_timens(const struct v= dso_data *vdns, clockid_ cycles =3D __arch_get_hw_counter(vd->clock_mode, vd); if (unlikely(!vdso_cycles_ok(cycles))) return -1; - ns =3D vdso_ts->nsec; - last =3D vd->cycle_last; - ns +=3D vdso_calc_delta(cycles, last, vd->mask, vd->mult); - ns =3D vdso_shift_ns(ns, vd->shift); + ns =3D vdso_calc_ns(vd, cycles, vdso_ts->nsec); sec =3D vdso_ts->sec; } while (unlikely(vdso_read_retry(vd, seq))); =20 @@ -115,7 +116,7 @@ static __always_inline int do_hres(const struct vdso_da= ta *vd, clockid_t clk, struct __kernel_timespec *ts) { const struct vdso_timestamp *vdso_ts =3D &vd->basetime[clk]; - u64 cycles, last, sec, ns; + u64 cycles, sec, ns; u32 seq; =20 /* Allows to compile the high resolution parts out */ @@ -148,10 +149,7 @@ static __always_inline int do_hres(const struct vdso_d= ata *vd, clockid_t clk, cycles =3D __arch_get_hw_counter(vd->clock_mode, vd); if (unlikely(!vdso_cycles_ok(cycles))) return -1; - ns =3D vdso_ts->nsec; - last =3D vd->cycle_last; - ns +=3D vdso_calc_delta(cycles, last, vd->mask, vd->mult); - ns =3D vdso_shift_ns(ns, vd->shift); + ns =3D vdso_calc_ns(vd, cycles, vdso_ts->nsec); sec =3D vdso_ts->sec; } while (unlikely(vdso_read_retry(vd, seq))); =20 --=20 2.34.1 From nobody Mon Feb 9 15:59:12 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BD3633D3B9; Fri, 8 Mar 2024 13:15:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.9 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709903751; cv=none; b=Y6Wek6dxxuPvFaLHnRQuJKNTK7zzZApOlrBTY7dIRF1FPWMWd1+TKEz6Nu2f64hNwPebLUDUyYiz9lO03moNGjEY1tR/ghyR3CWPL1ihdBQK0ggoPUhUhZBP55YB9x0gQB5Adn8M+MUyS0/11Tf915cbCzwSQoB6AyOUfu2TgHc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709903751; c=relaxed/simple; bh=bS1S2X2CJjfZI9Wdu/gKC/PtojbjfoojrJoco+YT8K0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=dPL3qCRDJppDeKEJS1+waz5c6hVfcYq0xgdts0p6jHlkACaF3Vx+pmwr0rtuKD9r6yxqpaKI+HXdSmgPk8vToRsbCxJi8SotKhzC3ekzGRltk5kAdZkERQG/B+eTYNKOVbZdf/onyGwpk5LoOxnLGo9EdxQ8RKg3h04mfmB48qo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=AR+K6M+P; arc=none smtp.client-ip=192.198.163.9 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="AR+K6M+P" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1709903750; x=1741439750; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=bS1S2X2CJjfZI9Wdu/gKC/PtojbjfoojrJoco+YT8K0=; b=AR+K6M+PiMSFInoWU8aHUXT/cUkPQYyi+gOWGcMZr0z1yQJnBqxCGy06 /ABuvKs3SF3shO4GnDTPrtYYoJHAICe0cOlE9mxKVLO2uv9pjVJc08agL vU78ykaHhn70ZsPMzv+Dzv5EK3UO0dEi7jqvGKTYy+4n/OI1giL08PJhU kuXCa2f9kz5RO2Cuvv81rHGRXI9sOII2LchJjZXbJgj1OOasjAbI7kOYg IuZ/EaPy64+jxruVnKGteV2gIXiSNswxaz1PuX4USE6MVl9SMj54fWRTh IRWic35vuj/jezoma+UlZWO6nWO9BHY9i2SEFUCBy2d7Fl1UL/JqPEUFe w==; X-IronPort-AV: E=McAfee;i="6600,9927,11006"; a="15342442" X-IronPort-AV: E=Sophos;i="6.07,109,1708416000"; d="scan'208";a="15342442" Received: from orviesa005.jf.intel.com ([10.64.159.145]) by fmvoesa103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Mar 2024 05:15:48 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,109,1708416000"; d="scan'208";a="15161375" Received: from ahunter6-mobl1.ger.corp.intel.com (HELO ahunter-VirtualBox.home\044ger.corp.intel.com) ([10.249.46.63]) by orviesa005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Mar 2024 05:15:41 -0800 From: Adrian Hunter To: Thomas Gleixner Cc: Michael Ellerman , Nicholas Piggin , Christophe Leroy , "Aneesh Kumar K.V" , "Naveen N. Rao" , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Andy Lutomirski , Vincenzo Frascino , John Stultz , Stephen Boyd , Peter Zijlstra , Randy Dunlap , Bjorn Helgaas , Arnd Bergmann , Anna-Maria Behnsen , linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, linux-s390@vger.kernel.org Subject: [PATCH 03/19] vdso: Add CONFIG_GENERIC_VDSO_OVERFLOW_PROTECT Date: Fri, 8 Mar 2024 15:14:56 +0200 Message-Id: <20240308131512.44324-4-adrian.hunter@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240308131512.44324-1-adrian.hunter@intel.com> References: <20240308131512.44324-1-adrian.hunter@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Organization: Intel Finland Oy, Registered Address: PL 281, 00181 Helsinki, Business Identity Code: 0357606 - 4, Domiciled in Helsinki Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add CONFIG_GENERIC_VDSO_OVERFLOW_PROTECT in preparation to add multiplication overflow protection to the VDSO time getter functions. Suggested-by: Thomas Gleixner Signed-off-by: Adrian Hunter --- lib/vdso/Kconfig | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/lib/vdso/Kconfig b/lib/vdso/Kconfig index d883ac299508..c46c2300517c 100644 --- a/lib/vdso/Kconfig +++ b/lib/vdso/Kconfig @@ -30,4 +30,11 @@ config GENERIC_VDSO_TIME_NS Selected by architectures which support time namespaces in the VDSO =20 +config GENERIC_VDSO_OVERFLOW_PROTECT + bool + help + Select to add multiplication overflow protection to the VDSO + time getter functions for the price of an extra conditional + in the hotpath. + endif --=20 2.34.1 From nobody Mon Feb 9 15:59:12 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D80E23FB9E; Fri, 8 Mar 2024 13:15:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.9 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709903757; cv=none; b=iz0iDZOQbc/qiijW2BuJPwW3lBDrxmhtaUfyHycSDREtg/X9NWQVc5/rxzoMISpX3LKQO1LFMbaGvP4vqU7SSACqwK1QNIVCF817KgjoJU5EE7lldQwm9bnQ+f78Ag8cwn3zVbjALkfBmv65tC2VYrbdSCsHiUUZWpe5Rll53+Q= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709903757; c=relaxed/simple; bh=xuamuB97LVfxyJSvE4i73CrmF8XOSgafgM3yHCD1nqo=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=QZ98/BnSm73aHVCqgA+pAawwtFNgkZFIxcOrr0KIqVh4ZSwlx1AJtIhGfGCsQm9VcenVLg5gozZ98NLK5v64wF0HAPzJnT0o7BonmkdD6RULZYkG/odvlR9hv5YdhUxDQQsextvm6VLh+PEP/a7uO6dubnrZdlhRT7nVYRb9Ch4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=F1ED09X2; arc=none smtp.client-ip=192.198.163.9 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="F1ED09X2" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1709903756; x=1741439756; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=xuamuB97LVfxyJSvE4i73CrmF8XOSgafgM3yHCD1nqo=; b=F1ED09X29kFWjM5C5PwVaAA7y8jaeSuUbl6VD6JSNH7gRjM2r3kgk8hK Ef8CXuNV0kZiKQb+fUknVTpSiyl3tQRWc2dEUWAQGhpg/pcGQwegyVHuy kpjs7I20WWQVL2TvLQRjYzksSB0o4oxGraVK2/Ds2onCB5tFw5S/rAB9t 6PNEIgeolPoq3SFgXmqh1mh1EcfAUy5L0baZnPlwRUMOFlnvtV60sRJJ7 lh+EiMouTXlxXjgg8Vgdb7UDyspGLVobKmgokHOpFGgIT+PeDGGMlNjvl L1OLANYMTGVY052OotV53pd/5WbVlfjXDYWEvKN38UPlSsd4Wu6Z6ehwV Q==; X-IronPort-AV: E=McAfee;i="6600,9927,11006"; a="15342483" X-IronPort-AV: E=Sophos;i="6.07,109,1708416000"; d="scan'208";a="15342483" Received: from orviesa005.jf.intel.com ([10.64.159.145]) by fmvoesa103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Mar 2024 05:15:55 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,109,1708416000"; d="scan'208";a="15161379" Received: from ahunter6-mobl1.ger.corp.intel.com (HELO ahunter-VirtualBox.home\044ger.corp.intel.com) ([10.249.46.63]) by orviesa005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Mar 2024 05:15:48 -0800 From: Adrian Hunter To: Thomas Gleixner Cc: Michael Ellerman , Nicholas Piggin , Christophe Leroy , "Aneesh Kumar K.V" , "Naveen N. Rao" , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Andy Lutomirski , Vincenzo Frascino , John Stultz , Stephen Boyd , Peter Zijlstra , Randy Dunlap , Bjorn Helgaas , Arnd Bergmann , Anna-Maria Behnsen , linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, linux-s390@vger.kernel.org Subject: [PATCH 04/19] math64: Tidy mul_u64_u32_shr() Date: Fri, 8 Mar 2024 15:14:57 +0200 Message-Id: <20240308131512.44324-5-adrian.hunter@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240308131512.44324-1-adrian.hunter@intel.com> References: <20240308131512.44324-1-adrian.hunter@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Organization: Intel Finland Oy, Registered Address: PL 281, 00181 Helsinki, Business Identity Code: 0357606 - 4, Domiciled in Helsinki Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Put together declaration and initialization of local variables. Suggested-by: Thomas Gleixner Signed-off-by: Adrian Hunter --- include/linux/math64.h | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/include/linux/math64.h b/include/linux/math64.h index bf74478926d4..fd13622b2056 100644 --- a/include/linux/math64.h +++ b/include/linux/math64.h @@ -179,16 +179,12 @@ static __always_inline u64 mul_u64_u64_shr(u64 a, u64= mul, unsigned int shift) #ifndef mul_u64_u32_shr static __always_inline u64 mul_u64_u32_shr(u64 a, u32 mul, unsigned int sh= ift) { - u32 ah, al; + u32 ah =3D a >> 32, al =3D a; u64 ret; =20 - al =3D a; - ah =3D a >> 32; - ret =3D mul_u32_u32(al, mul) >> shift; if (ah) ret +=3D mul_u32_u32(ah, mul) << (32 - shift); - return ret; } #endif /* mul_u64_u32_shr */ --=20 2.34.1 From nobody Mon Feb 9 15:59:12 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7A4033FBA3; Fri, 8 Mar 2024 13:16:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.9 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709903764; cv=none; b=WfWSIZ5dpi8OKYGkJgzSPiF63WHf1LXwGoCL3NOezfYyJ/XqGshlO9Pym7uEdYvSNLoCy1I4j+0R7drD6aTME7igm7T9Y/QM0+wAS55THd/+tnr+x92tfveRxTkRwirStAGUjeJmdbDblhccLD3PihbOhbL2BQ4JCy9hAa/ikKI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709903764; c=relaxed/simple; bh=LAWkJDgSOoQXttIOaNX5OjXrKNZqw1gNI6g5JMXltWY=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=CmHZqrJBBNKdBY3oKfVqvq9dqSurSlE4dXvez/7FRZ7P44PbGpvKjTb7OCWaW31rzJzA/xJxBfAScdEwOSJIHUkdXgJOPy9oOoRFuYKJsW0n0Y/dzWEXQs6xse5ZNiFVWJ2aoaX4XJZvn0kBQdjTNewiWmBIsbQKlxhm49uNkOs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=dKkQwqht; arc=none smtp.client-ip=192.198.163.9 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="dKkQwqht" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1709903763; x=1741439763; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=LAWkJDgSOoQXttIOaNX5OjXrKNZqw1gNI6g5JMXltWY=; b=dKkQwqhtq7hi+RivoAFGfQlxov+s7gDXrTtI3l1nemA7D7hPSZ7OoAug 9lcVHg2rHWicohP6AbvPe5yaKsYiXJ8D64XJsaHn7Vn2CLY+DPtZsGGee F3umKTMckNOIyU0x2F18q3d2QL8ZUFbI+jIpxTg2TWZRXRrwR6b09YkT5 VixNKbJb3qgmGpxD0xPlvaLR6GtsT3eFOIvDyEbC+yNsTcvhZaJPBin79 7PfiK0WUOCOlVYcyQW0+WmRSb4gQ1R9wYmHVS1zhZb4DCajF+eMk3Iec3 PHgNieXTm5Z37U6gk3cR0AnF2uEFi6t2DnFBs0xUYN3bm0g+JlI8cStPH w==; X-IronPort-AV: E=McAfee;i="6600,9927,11006"; a="15342493" X-IronPort-AV: E=Sophos;i="6.07,109,1708416000"; d="scan'208";a="15342493" Received: from orviesa005.jf.intel.com ([10.64.159.145]) by fmvoesa103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Mar 2024 05:16:02 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,109,1708416000"; d="scan'208";a="15161388" Received: from ahunter6-mobl1.ger.corp.intel.com (HELO ahunter-VirtualBox.home\044ger.corp.intel.com) ([10.249.46.63]) by orviesa005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Mar 2024 05:15:55 -0800 From: Adrian Hunter To: Thomas Gleixner Cc: Michael Ellerman , Nicholas Piggin , Christophe Leroy , "Aneesh Kumar K.V" , "Naveen N. Rao" , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Andy Lutomirski , Vincenzo Frascino , John Stultz , Stephen Boyd , Peter Zijlstra , Randy Dunlap , Bjorn Helgaas , Arnd Bergmann , Anna-Maria Behnsen , linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, linux-s390@vger.kernel.org Subject: [PATCH 05/19] vdso: math64: Provide mul_u64_u32_add_u64_shr() Date: Fri, 8 Mar 2024 15:14:58 +0200 Message-Id: <20240308131512.44324-6-adrian.hunter@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240308131512.44324-1-adrian.hunter@intel.com> References: <20240308131512.44324-1-adrian.hunter@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Organization: Intel Finland Oy, Registered Address: PL 281, 00181 Helsinki, Business Identity Code: 0357606 - 4, Domiciled in Helsinki Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Provide mul_u64_u32_add_u64_shr() which is a calculation that will be used by timekeeping and VDSO. Place #include after #include to allow architecture-specific overrides, at least for the kernel. Signed-off-by: Adrian Hunter --- include/linux/math64.h | 2 +- include/vdso/math64.h | 38 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 39 insertions(+), 1 deletion(-) diff --git a/include/linux/math64.h b/include/linux/math64.h index fd13622b2056..d34def7f9a8c 100644 --- a/include/linux/math64.h +++ b/include/linux/math64.h @@ -4,8 +4,8 @@ =20 #include #include -#include #include +#include =20 #if BITS_PER_LONG =3D=3D 64 =20 diff --git a/include/vdso/math64.h b/include/vdso/math64.h index 7da703ee5561..22ae212f8b28 100644 --- a/include/vdso/math64.h +++ b/include/vdso/math64.h @@ -21,4 +21,42 @@ __iter_div_u64_rem(u64 dividend, u32 divisor, u64 *remai= nder) return ret; } =20 +#if defined(CONFIG_ARCH_SUPPORTS_INT128) && defined(__SIZEOF_INT128__) + +#ifndef mul_u64_u32_add_u64_shr +static __always_inline u64 mul_u64_u32_add_u64_shr(u64 a, u32 mul, u64 b, = unsigned int shift) +{ + return (u64)((((unsigned __int128)a * mul) + b) >> shift); +} +#endif /* mul_u64_u32_add_u64_shr */ + +#else + +#ifndef mul_u64_u32_add_u64_shr +#ifndef mul_u32_u32 +static inline u64 mul_u32_u32(u32 a, u32 b) +{ + return (u64)a * b; +} +#define mul_u32_u32 mul_u32_u32 +#endif +static __always_inline u64 mul_u64_u32_add_u64_shr(u64 a, u32 mul, u64 b, = unsigned int shift) +{ + u32 ah =3D a >> 32, al =3D a; + bool ovf; + u64 ret; + + ovf =3D __builtin_add_overflow(mul_u32_u32(al, mul), b, &ret); + ret >>=3D shift; + if (ovf && shift) + ret +=3D 1ULL << (64 - shift); + if (ah) + ret +=3D mul_u32_u32(ah, mul) << (32 - shift); + + return ret; +} +#endif /* mul_u64_u32_add_u64_shr */ + +#endif + #endif /* __VDSO_MATH64_H */ --=20 2.34.1 From nobody Mon Feb 9 15:59:12 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 02B8E3BBD4; Fri, 8 Mar 2024 13:16:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.9 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709903772; cv=none; b=pircRuc4KJSBe8hMvYe0S9fBBngSHEAKqalL0IBBIB+7TQCVbkkBEciIMHkXhMjvW4AbvhSZws51xZdGUQnDujL3KyOF1K9f+CnbCZPlcquZGntCVkTP2QKcvZHvQ4Nq/4RddWQ8cxXGH+BdxUoo5Bqsyxp/92aMZjLRzL1zXjQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709903772; c=relaxed/simple; bh=Exf/ex05wg/ZdHrQtagsjpQZFoomRp7dCTSrSntMNmc=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=KMKFLY1MVV0vGxdOpyuE5KYnEChCxfhJFUrXoWuyDPBDQFG4gd6597xzl1CgnOKH2Kbcf4gUuwBTi8ZGu8CIL8OoY7qwZozP0atKfFa++oD69u15pcpvClQJQqV/qLluUCzgC7pVA9VUSWfYpcYFQS1mwcMjryNMOFkYUjy8Fk4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=OYhgJIC1; arc=none smtp.client-ip=192.198.163.9 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="OYhgJIC1" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1709903771; x=1741439771; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Exf/ex05wg/ZdHrQtagsjpQZFoomRp7dCTSrSntMNmc=; b=OYhgJIC1etTv9J1TYIuQSauCnvt9vh4Bl+wecQFvcVd8ndT6qVN4KYGn z8nD2VSe9/hRizhPvHvbg77VgTjwiipgWCfR8p5dc0ycfkIdSseqmM2YU 0hWnZ2oAbAgSI0d8mQ7az8HmuJJ3fcB8CdatWWICZ9WDU/PBaaJqOHjit wGD/26H10F4odAGnwc8SEr/AvdnsOpdwhwAaUTiN1YJs9XjOVaSCqVKup sfabu/kNQyK1kIgU2AlC1FSUoMQVcmaGrk/iuJ6E3zIDaA3EM88ww7zjR pzYQlAxxl4Fk8RQVSMFalTaSoucyi/+z0qEbASlRTChrkbshu/I2run83 w==; X-IronPort-AV: E=McAfee;i="6600,9927,11006"; a="15342539" X-IronPort-AV: E=Sophos;i="6.07,109,1708416000"; d="scan'208";a="15342539" Received: from orviesa005.jf.intel.com ([10.64.159.145]) by fmvoesa103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Mar 2024 05:16:10 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,109,1708416000"; d="scan'208";a="15161409" Received: from ahunter6-mobl1.ger.corp.intel.com (HELO ahunter-VirtualBox.home\044ger.corp.intel.com) ([10.249.46.63]) by orviesa005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Mar 2024 05:16:02 -0800 From: Adrian Hunter To: Thomas Gleixner Cc: Michael Ellerman , Nicholas Piggin , Christophe Leroy , "Aneesh Kumar K.V" , "Naveen N. Rao" , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Andy Lutomirski , Vincenzo Frascino , John Stultz , Stephen Boyd , Peter Zijlstra , Randy Dunlap , Bjorn Helgaas , Arnd Bergmann , Anna-Maria Behnsen , linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, linux-s390@vger.kernel.org Subject: [PATCH 06/19] vdso: Add vdso_data::max_cycles Date: Fri, 8 Mar 2024 15:14:59 +0200 Message-Id: <20240308131512.44324-7-adrian.hunter@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240308131512.44324-1-adrian.hunter@intel.com> References: <20240308131512.44324-1-adrian.hunter@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Organization: Intel Finland Oy, Registered Address: PL 281, 00181 Helsinki, Business Identity Code: 0357606 - 4, Domiciled in Helsinki Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add vdso_data::max_cycles in preparation to use it to detect potential multiplication overflow. Suggested-by: Thomas Gleixner Signed-off-by: Adrian Hunter --- include/vdso/datapage.h | 4 ++++ kernel/time/vsyscall.c | 6 ++++++ 2 files changed, 10 insertions(+) diff --git a/include/vdso/datapage.h b/include/vdso/datapage.h index 5d5c0b8efff2..6c3d67d6b758 100644 --- a/include/vdso/datapage.h +++ b/include/vdso/datapage.h @@ -67,6 +67,7 @@ struct vdso_timestamp { * @seq: timebase sequence counter * @clock_mode: clock mode * @cycle_last: timebase at clocksource init + * @max_cycles: maximum cycles which won't overflow 64bit multiplication * @mask: clocksource mask * @mult: clocksource multiplier * @shift: clocksource shift @@ -98,6 +99,9 @@ struct vdso_data { =20 s32 clock_mode; u64 cycle_last; +#ifdef CONFIG_GENERIC_VDSO_OVERFLOW_PROTECT + u64 max_cycles; +#endif u64 mask; u32 mult; u32 shift; diff --git a/kernel/time/vsyscall.c b/kernel/time/vsyscall.c index f0d5062d9cbc..9193d6133e5d 100644 --- a/kernel/time/vsyscall.c +++ b/kernel/time/vsyscall.c @@ -22,10 +22,16 @@ static inline void update_vdso_data(struct vdso_data *v= data, u64 nsec, sec; =20 vdata[CS_HRES_COARSE].cycle_last =3D tk->tkr_mono.cycle_last; +#ifdef CONFIG_GENERIC_VDSO_OVERFLOW_PROTECT + vdata[CS_HRES_COARSE].max_cycles =3D tk->tkr_mono.clock->max_cycles; +#endif vdata[CS_HRES_COARSE].mask =3D tk->tkr_mono.mask; vdata[CS_HRES_COARSE].mult =3D tk->tkr_mono.mult; vdata[CS_HRES_COARSE].shift =3D tk->tkr_mono.shift; vdata[CS_RAW].cycle_last =3D tk->tkr_raw.cycle_last; +#ifdef CONFIG_GENERIC_VDSO_OVERFLOW_PROTECT + vdata[CS_RAW].max_cycles =3D tk->tkr_raw.clock->max_cycles; +#endif vdata[CS_RAW].mask =3D tk->tkr_raw.mask; vdata[CS_RAW].mult =3D tk->tkr_raw.mult; vdata[CS_RAW].shift =3D tk->tkr_raw.shift; --=20 2.34.1 From nobody Mon Feb 9 15:59:12 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 797FC3BBE0; Fri, 8 Mar 2024 13:16:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.9 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709903778; cv=none; b=VUSEHVgXS4Np+paUL73vwoFt8/EDbAOp2BvqOPvxRis3ZPf/8aWUR7xj4avQ3KykPTAFA8ODIHvie72tAu1Vok9DdpAU2ljWuOZC0In25yTy3Hv6DmVCbnMTRzTLcY/Rj5zzsstuCpQcorgJYBCS38U/+SRrrXX1y4shGUKGPqg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709903778; c=relaxed/simple; bh=Zjh6IUndlImHk90vhCoelXOI1d7rMagiowI5tzl77NQ=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=eUaoqblTdJwU92vTP02KtVICQflYuaraO4IcXoJszNAmogrve51GNccufOE+rK9xUZelpCZDNE/37qbu0GZJPDXYi++9dtxmb1F+J/6Q9z25Mn3xXJ73IW+SgFsb+Cwo0fzXZSSjpYbdor/DSxTCjIUPesb2vux5QdY4raJPUa4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=diYBaxQV; arc=none smtp.client-ip=192.198.163.9 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="diYBaxQV" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1709903777; x=1741439777; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Zjh6IUndlImHk90vhCoelXOI1d7rMagiowI5tzl77NQ=; b=diYBaxQV8nYBFa2FFhkrINNiKgQELbj4e+fe0P+H0Cs9EG0kLwXn6hEJ X7h/vL0ufYtVdvrfxAteRW3u8LIzHAiCLyrboyRD5kz7OsU/8xycBimXs HPovD1cht86BtfK3WtiRZDx7HlW7Q9YMF5IG7CQ95pH77lFxfxmKpXzJe KL+3EnrywT9BjEZ8JMNHlw2xugwdQ4GVz8hFKFx5KTts1k67UpxZQ5Tv4 CuZU2vXiCFmNbSJyzOxJdGcBumA/yxbNWCmFsb/xFE3w4wO51rYHCnEGd JR8r7prvCZME+vCPH7z5UmD4y01mOS1H1COKC8UrwS9v05kcH5RDPZyP2 w==; X-IronPort-AV: E=McAfee;i="6600,9927,11006"; a="15342569" X-IronPort-AV: E=Sophos;i="6.07,109,1708416000"; d="scan'208";a="15342569" Received: from orviesa005.jf.intel.com ([10.64.159.145]) by fmvoesa103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Mar 2024 05:16:16 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,109,1708416000"; d="scan'208";a="15161445" Received: from ahunter6-mobl1.ger.corp.intel.com (HELO ahunter-VirtualBox.home\044ger.corp.intel.com) ([10.249.46.63]) by orviesa005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Mar 2024 05:16:09 -0800 From: Adrian Hunter To: Thomas Gleixner Cc: Michael Ellerman , Nicholas Piggin , Christophe Leroy , "Aneesh Kumar K.V" , "Naveen N. Rao" , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Andy Lutomirski , Vincenzo Frascino , John Stultz , Stephen Boyd , Peter Zijlstra , Randy Dunlap , Bjorn Helgaas , Arnd Bergmann , Anna-Maria Behnsen , linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, linux-s390@vger.kernel.org Subject: [PATCH 07/19] vdso: Make delta calculation overflow safe Date: Fri, 8 Mar 2024 15:15:00 +0200 Message-Id: <20240308131512.44324-8-adrian.hunter@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240308131512.44324-1-adrian.hunter@intel.com> References: <20240308131512.44324-1-adrian.hunter@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Organization: Intel Finland Oy, Registered Address: PL 281, 00181 Helsinki, Business Identity Code: 0357606 - 4, Domiciled in Helsinki Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Kernel timekeeping is designed to keep the change in cycles (since the last timer interrupt) below max_cycles, which prevents multiplication overflow when converting cycles to nanoseconds. However, if timer interrupts stop, the calculation will eventually overflow. Add protection against that, enabled by config option CONFIG_GENERIC_VDSO_OVERFLOW_PROTECT. Check against max_cycles, falling back to a slower higher precision calculation. Suggested-by: Thomas Gleixner Signed-off-by: Adrian Hunter --- lib/vdso/gettimeofday.c | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/lib/vdso/gettimeofday.c b/lib/vdso/gettimeofday.c index 9fa90e0794c9..9c3a8d2440c9 100644 --- a/lib/vdso/gettimeofday.c +++ b/lib/vdso/gettimeofday.c @@ -13,6 +13,18 @@ # define VDSO_DELTA_MASK(vd) (vd->mask) #endif =20 +#ifdef CONFIG_GENERIC_VDSO_OVERFLOW_PROTECT +static __always_inline bool vdso_delta_ok(const struct vdso_data *vd, u64 = delta) +{ + return delta < vd->max_cycles; +} +#else +static __always_inline bool vdso_delta_ok(const struct vdso_data *vd, u64 = delta) +{ + return true; +} +#endif + #ifndef vdso_shift_ns static __always_inline u64 vdso_shift_ns(u64 ns, u32 shift) { @@ -28,7 +40,10 @@ static __always_inline u64 vdso_calc_ns(const struct vds= o_data *vd, u64 cycles, { u64 delta =3D (cycles - vd->cycle_last) & VDSO_DELTA_MASK(vd); =20 - return vdso_shift_ns((delta * vd->mult) + base, vd->shift); + if (likely(vdso_delta_ok(vd, delta))) + return vdso_shift_ns((delta * vd->mult) + base, vd->shift); + + return mul_u64_u32_add_u64_shr(delta, vd->mult, base, vd->shift); } #endif /* vdso_calc_ns */ =20 --=20 2.34.1 From nobody Mon Feb 9 15:59:12 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0F4E454BD6; Fri, 8 Mar 2024 13:16:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.9 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709903784; cv=none; b=vCgpXmwUyIhIl4VXEwH5OulO6I0gtt3aMFU6WNc0IK+VMD+LOaenSk+iuDryGB07Yiot3ATWkfbf164CMunMDtVp+nmvgiqxqiqmnEkjsFkbEbbTq4xkNFSsNvTcMy4wt8tyByJcNmmnzLROfzvw8aBOPeksooqeUzcOspR334U= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709903784; c=relaxed/simple; bh=Jf0Ify4PqkXxXfo1lgCW0RH1p8hV+xM9o246fKxlK4k=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=krrM9rZ7FFHGeEjfWRtpUD15GeNfk7RdZ0DB1YXFuFGsMnrrNB3t/E2YLtue88F2lBp2WngU2wQqWiU9kcau2ZP2glpGm5rPkllxocsjdmODXfrRUwXjiIJ6rxDezcxB6ZYslt9zSQnYZkXMunwh7VujOy5C3HE+wQ9zg1UXcZQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=i5gJnMO8; arc=none smtp.client-ip=192.198.163.9 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="i5gJnMO8" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1709903783; x=1741439783; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Jf0Ify4PqkXxXfo1lgCW0RH1p8hV+xM9o246fKxlK4k=; b=i5gJnMO81AWqRB22K7efOPL82V8tiTFR/t9QNdR5oe/NDcGUyj83UYLO jZ2tPm1o7+bE7DHCPRlT2fLcC8NzUZYMKmTH02Ss25M11ijx6QlkkliEW +yEHy7yb0j3+u44xv/jiloqdTybwzBSdQWsvcSqx+GjytsvWLHyOeiRKZ Z0u/Xr5DZJcsOIIeYe3IJnibgNeWGmbGZoQksEHAMsWz2HXjpp3vvgkUS cf1u0DGQsQxNBedEuCYxC+S4kyuB91Vhk9BBdAebCZn6JfcXsmcZzHCmt S+BDu6Dbn5t9Z1IIgoFkxiFk04IDOelphA7fltuqV7yJ/+/TGmQ+DDrXW Q==; X-IronPort-AV: E=McAfee;i="6600,9927,11006"; a="15342591" X-IronPort-AV: E=Sophos;i="6.07,109,1708416000"; d="scan'208";a="15342591" Received: from orviesa005.jf.intel.com ([10.64.159.145]) by fmvoesa103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Mar 2024 05:16:23 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,109,1708416000"; d="scan'208";a="15161463" Received: from ahunter6-mobl1.ger.corp.intel.com (HELO ahunter-VirtualBox.home\044ger.corp.intel.com) ([10.249.46.63]) by orviesa005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Mar 2024 05:16:16 -0800 From: Adrian Hunter To: Thomas Gleixner Cc: Michael Ellerman , Nicholas Piggin , Christophe Leroy , "Aneesh Kumar K.V" , "Naveen N. Rao" , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Andy Lutomirski , Vincenzo Frascino , John Stultz , Stephen Boyd , Peter Zijlstra , Randy Dunlap , Bjorn Helgaas , Arnd Bergmann , Anna-Maria Behnsen , linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, linux-s390@vger.kernel.org Subject: [PATCH 08/19] x86/vdso: Make delta calculation overflow safe Date: Fri, 8 Mar 2024 15:15:01 +0200 Message-Id: <20240308131512.44324-9-adrian.hunter@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240308131512.44324-1-adrian.hunter@intel.com> References: <20240308131512.44324-1-adrian.hunter@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Organization: Intel Finland Oy, Registered Address: PL 281, 00181 Helsinki, Business Identity Code: 0357606 - 4, Domiciled in Helsinki Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Kernel timekeeping is designed to keep the change in cycles (since the last timer interrupt) below max_cycles, which prevents multiplication overflow when converting cycles to nanoseconds. However, if timer interrupts stop, the calculation will eventually overflow. Add protection against that. Select GENERIC_VDSO_OVERFLOW_PROTECT so that max_cycles is made available in the VDSO data page. Check against max_cycles, falling back to a slower higher precision calculation. Take advantage of the opportunity to move masking and negative motion check into the slow path. The result is a calculation that has similar performance as before. Newer machines showed performance benefit, whereas older Skylake-based hardware such as Intel Kaby Lake was seen <1% worse. Suggested-by: Thomas Gleixner Signed-off-by: Adrian Hunter --- arch/x86/Kconfig | 1 + arch/x86/include/asm/vdso/gettimeofday.h | 29 +++++++++++++++++------- 2 files changed, 22 insertions(+), 8 deletions(-) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 720b96388191..200f80a36593 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -168,6 +168,7 @@ config X86 select GENERIC_TIME_VSYSCALL select GENERIC_GETTIMEOFDAY select GENERIC_VDSO_TIME_NS + select GENERIC_VDSO_OVERFLOW_PROTECT select GUP_GET_PXX_LOW_HIGH if X86_PAE select HARDIRQS_SW_RESEND select HARDLOCKUP_CHECK_TIMESTAMP if X86_64 diff --git a/arch/x86/include/asm/vdso/gettimeofday.h b/arch/x86/include/as= m/vdso/gettimeofday.h index 5727dedd3549..0ef36190abe6 100644 --- a/arch/x86/include/asm/vdso/gettimeofday.h +++ b/arch/x86/include/asm/vdso/gettimeofday.h @@ -319,18 +319,31 @@ static inline bool arch_vdso_cycles_ok(u64 cycles) */ static __always_inline u64 vdso_calc_ns(const struct vdso_data *vd, u64 cy= cles, u64 base) { + u64 delta =3D cycles - vd->cycle_last; + /* + * Negative motion and deltas which can cause multiplication + * overflow require special treatment. This check covers both as + * negative motion is guaranteed to be greater than @vd::max_cycles + * due to unsigned comparison. + * * Due to the MSB/Sign-bit being used as invalid marker (see - * arch_vdso_cycles_valid() above), the effective mask is S64_MAX. + * arch_vdso_cycles_valid() above), the effective mask is S64_MAX, + * but that case is also unlikely and will also take the unlikely path + * here. */ - u64 delta =3D (cycles - vd->cycle_last) & S64_MAX; + if (unlikely(delta > vd->max_cycles)) { + /* + * Due to the above mentioned TSC wobbles, filter out + * negative motion. Per the above masking, the effective + * sign bit is now bit 62. + */ + if (delta & (1ULL << 62)) + return base >> vd->shift; =20 - /* - * Due to the above mentioned TSC wobbles, filter out negative motion. - * Per the above masking, the effective sign bit is now bit 62. - */ - if (unlikely(delta & (1ULL << 62))) - return base >> vd->shift; + /* Handle multiplication overflow gracefully */ + return mul_u64_u32_add_u64_shr(delta & S64_MAX, vd->mult, base, vd->shif= t); + } =20 return ((delta * vd->mult) + base) >> vd->shift; } --=20 2.34.1 From nobody Mon Feb 9 15:59:12 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id ECEDC5676F; Fri, 8 Mar 2024 13:16:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.9 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709903792; cv=none; b=O6na9JZaFLVvSmu/+VyF0aoHkPQUr/gDD7F/gf6Ue6TGHoXaI/Ih5pO2VpKRMfkOEtMJqQ0gwyrcYVU5iT3V/xUP7ve/A6OGeJ6W0LP6ux7GjT+SZM6h4axHeZrsJjwc2gsy4XvvVOkEydC/+v0b2/Y0QCDWjgY0tFcQGlsxaHg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709903792; c=relaxed/simple; bh=zboSVI3HgVZXbpDBNnsTotqgGm69H9hHwUY1hZ67rA8=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=M855xKmvO3Mfx+0dyRrH2yT9VcUHMELrNIjFtWGZFMV7BvW2FTKAXF6YzOEmnPa6eSv0v8uwkBgUXKpVEOVslgzZwYrrlX8yOHULlXC3fi/IzGSUUBIGrhOWAR2/WBNfPbVk+rX9uxXhbDRBp7YY2h5bNjjsHJB0uwTYlLMJ3T0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=AJeLwDrF; arc=none smtp.client-ip=192.198.163.9 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="AJeLwDrF" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1709903791; x=1741439791; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=zboSVI3HgVZXbpDBNnsTotqgGm69H9hHwUY1hZ67rA8=; b=AJeLwDrFChumvLrBcVDDcwJ2IZDo/B+x7XSl9UYaAL++dgpQCa9IAqqT zFn9SMdcvy+vIHoGotWQlsBzpxvEvBtJcotG0CghSQ6aaJue7kqlKh6eV HRykUwuCB10l1H1RwN2aqdIEyiW/DMMyJBZsOo7YBVTslmfdpOycUTuug jZc9zgWMo5QTnuFnkONSAd75Xe7qCZJvgG8a3N9gdjBUMmn/JA/a1TlYN ZNlcRyOfGO6ac7iPng3bBSIxWCWgRVDakYchBloQ0rqQtktJiy6dyu6B/ HXFgLYm4z/1+tkPrwgizuWT4f9CQ/OVVZ8YHtd26t9ehUaji4F878gNCF w==; X-IronPort-AV: E=McAfee;i="6600,9927,11006"; a="15342636" X-IronPort-AV: E=Sophos;i="6.07,109,1708416000"; d="scan'208";a="15342636" Received: from orviesa005.jf.intel.com ([10.64.159.145]) by fmvoesa103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Mar 2024 05:16:30 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,109,1708416000"; d="scan'208";a="15161489" Received: from ahunter6-mobl1.ger.corp.intel.com (HELO ahunter-VirtualBox.home\044ger.corp.intel.com) ([10.249.46.63]) by orviesa005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Mar 2024 05:16:23 -0800 From: Adrian Hunter To: Thomas Gleixner Cc: Michael Ellerman , Nicholas Piggin , Christophe Leroy , "Aneesh Kumar K.V" , "Naveen N. Rao" , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Andy Lutomirski , Vincenzo Frascino , John Stultz , Stephen Boyd , Peter Zijlstra , Randy Dunlap , Bjorn Helgaas , Arnd Bergmann , Anna-Maria Behnsen , linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, linux-s390@vger.kernel.org Subject: [PATCH 09/19] timekeeping: Move timekeeping helper functions Date: Fri, 8 Mar 2024 15:15:02 +0200 Message-Id: <20240308131512.44324-10-adrian.hunter@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240308131512.44324-1-adrian.hunter@intel.com> References: <20240308131512.44324-1-adrian.hunter@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Organization: Intel Finland Oy, Registered Address: PL 281, 00181 Helsinki, Business Identity Code: 0357606 - 4, Domiciled in Helsinki Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Move timekeeping helper functions to prepare for their reuse. Suggested-by: Thomas Gleixner Signed-off-by: Adrian Hunter --- kernel/time/timekeeping.c | 34 +++++++++++++++++----------------- 1 file changed, 17 insertions(+), 17 deletions(-) diff --git a/kernel/time/timekeeping.c b/kernel/time/timekeeping.c index b58dffc58a8f..3375f0a6400f 100644 --- a/kernel/time/timekeeping.c +++ b/kernel/time/timekeeping.c @@ -381,6 +381,23 @@ static inline u64 timekeeping_delta_to_ns(const struct= tk_read_base *tkr, u64 de return nsec; } =20 +static inline u64 timekeeping_cycles_to_ns(const struct tk_read_base *tkr,= u64 cycles) +{ + u64 delta; + + /* calculate the delta since the last update_wall_time */ + delta =3D clocksource_delta(cycles, tkr->cycle_last, tkr->mask); + return timekeeping_delta_to_ns(tkr, delta); +} + +static __always_inline u64 fast_tk_get_delta_ns(struct tk_read_base *tkr) +{ + u64 delta, cycles =3D tk_clock_read(tkr); + + delta =3D clocksource_delta(cycles, tkr->cycle_last, tkr->mask); + return timekeeping_delta_to_ns(tkr, delta); +} + static inline u64 timekeeping_get_ns(const struct tk_read_base *tkr) { u64 delta; @@ -389,15 +406,6 @@ static inline u64 timekeeping_get_ns(const struct tk_r= ead_base *tkr) return timekeeping_delta_to_ns(tkr, delta); } =20 -static inline u64 timekeeping_cycles_to_ns(const struct tk_read_base *tkr,= u64 cycles) -{ - u64 delta; - - /* calculate the delta since the last update_wall_time */ - delta =3D clocksource_delta(cycles, tkr->cycle_last, tkr->mask); - return timekeeping_delta_to_ns(tkr, delta); -} - /** * update_fast_timekeeper - Update the fast and NMI safe monotonic timekee= per. * @tkr: Timekeeping readout base from which we take the update @@ -431,14 +439,6 @@ static void update_fast_timekeeper(const struct tk_rea= d_base *tkr, memcpy(base + 1, base, sizeof(*base)); } =20 -static __always_inline u64 fast_tk_get_delta_ns(struct tk_read_base *tkr) -{ - u64 delta, cycles =3D tk_clock_read(tkr); - - delta =3D clocksource_delta(cycles, tkr->cycle_last, tkr->mask); - return timekeeping_delta_to_ns(tkr, delta); -} - static __always_inline u64 __ktime_get_fast_ns(struct tk_fast *tkf) { struct tk_read_base *tkr; --=20 2.34.1 From nobody Mon Feb 9 15:59:12 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D041056742; Fri, 8 Mar 2024 13:16:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.9 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709903798; cv=none; b=QPPVn7VdmRzRIClmZR2St9WmrLWw2IoQ9VRXHcQsiVeRFlTrVwpdYKdf/m2pmf7aN/BzAdkWCSflHmmq6jWdXpMnVeg5tTQlcck1OFTn6i9iOhRbLSLbK2WCHOlVx+8QcF+xYqIiete9Y7lLT1/wlaN/a0FRlOfd2fl3L6eYUGI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709903798; c=relaxed/simple; bh=qTxeGZya5tNrF+OulKJd9BaTzgUsvcEDLjRYcSx3zxw=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=ns6Jp92EZkLSSDgLvaDCKkaLmsvu8FL7wiJV6LoxX/k5u19JeiwZQ5Xnh3KnsQvxEcT/gFfWLfN0HWlcHv9A8eQkWTu+/PZADJkhObJ3f7aaiv2nemTXw2HuGZ0AUN/6iDvk/yB+PVlaIwAVu1mlvHYk2nQf+jxCQ5aGCqiiRlg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=I/UdE8Pt; arc=none smtp.client-ip=192.198.163.9 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="I/UdE8Pt" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1709903797; x=1741439797; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=qTxeGZya5tNrF+OulKJd9BaTzgUsvcEDLjRYcSx3zxw=; b=I/UdE8PtyN1Qda+66NMUirJBtgIS6ZB+a4cl2T29iMFHT6OzgmU5Xq+4 QbC8IeJtD3JkodRxsFK4mxVGiiIzRtnFr6hPkpE0nW8yx1//T+mDcHFmg hIq3bs68I0w3FnPZ6rcNLIYfJmznxchhghPN+/G0sXwFtcmRkiwrUhrAA 6NtkuWMxzRzQ29aI283X1IlybQzSW38NAVcNjiXUtspf/K4cigEK5ubXc JuRAuitphot+Hgm/3G+vtM7lDcfelYDGhdi8SZMhFbfckPDu34WdlRdaM TN1vVa+F8AAmfZbYT92U09suzKr2RG6cBFuFhLpAong/YQTpH/LhUiVlz A==; X-IronPort-AV: E=McAfee;i="6600,9927,11006"; a="15342667" X-IronPort-AV: E=Sophos;i="6.07,109,1708416000"; d="scan'208";a="15342667" Received: from orviesa005.jf.intel.com ([10.64.159.145]) by fmvoesa103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Mar 2024 05:16:36 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,109,1708416000"; d="scan'208";a="15161544" Received: from ahunter6-mobl1.ger.corp.intel.com (HELO ahunter-VirtualBox.home\044ger.corp.intel.com) ([10.249.46.63]) by orviesa005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Mar 2024 05:16:29 -0800 From: Adrian Hunter To: Thomas Gleixner Cc: Michael Ellerman , Nicholas Piggin , Christophe Leroy , "Aneesh Kumar K.V" , "Naveen N. Rao" , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Andy Lutomirski , Vincenzo Frascino , John Stultz , Stephen Boyd , Peter Zijlstra , Randy Dunlap , Bjorn Helgaas , Arnd Bergmann , Anna-Maria Behnsen , linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, linux-s390@vger.kernel.org Subject: [PATCH 10/19] timekeeping: Rename fast_tk_get_delta_ns() to __timekeeping_get_ns() Date: Fri, 8 Mar 2024 15:15:03 +0200 Message-Id: <20240308131512.44324-11-adrian.hunter@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240308131512.44324-1-adrian.hunter@intel.com> References: <20240308131512.44324-1-adrian.hunter@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Organization: Intel Finland Oy, Registered Address: PL 281, 00181 Helsinki, Business Identity Code: 0357606 - 4, Domiciled in Helsinki Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Rename fast_tk_get_delta_ns() to __timekeeping_get_ns() to prepare for its reuse as a general timekeeping helper function. Suggested-by: Thomas Gleixner Signed-off-by: Adrian Hunter --- kernel/time/timekeeping.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/kernel/time/timekeeping.c b/kernel/time/timekeeping.c index 3375f0a6400f..63061332a75c 100644 --- a/kernel/time/timekeeping.c +++ b/kernel/time/timekeeping.c @@ -390,7 +390,7 @@ static inline u64 timekeeping_cycles_to_ns(const struct= tk_read_base *tkr, u64 c return timekeeping_delta_to_ns(tkr, delta); } =20 -static __always_inline u64 fast_tk_get_delta_ns(struct tk_read_base *tkr) +static __always_inline u64 __timekeeping_get_ns(const struct tk_read_base = *tkr) { u64 delta, cycles =3D tk_clock_read(tkr); =20 @@ -449,7 +449,7 @@ static __always_inline u64 __ktime_get_fast_ns(struct t= k_fast *tkf) seq =3D raw_read_seqcount_latch(&tkf->seq); tkr =3D tkf->base + (seq & 0x01); now =3D ktime_to_ns(tkr->base); - now +=3D fast_tk_get_delta_ns(tkr); + now +=3D __timekeeping_get_ns(tkr); } while (raw_read_seqcount_latch_retry(&tkf->seq, seq)); =20 return now; @@ -565,7 +565,7 @@ static __always_inline u64 __ktime_get_real_fast(struct= tk_fast *tkf, u64 *mono) tkr =3D tkf->base + (seq & 0x01); basem =3D ktime_to_ns(tkr->base); baser =3D ktime_to_ns(tkr->base_real); - delta =3D fast_tk_get_delta_ns(tkr); + delta =3D __timekeeping_get_ns(tkr); } while (raw_read_seqcount_latch_retry(&tkf->seq, seq)); =20 if (mono) --=20 2.34.1 From nobody Mon Feb 9 15:59:12 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2C4333FBB3; Fri, 8 Mar 2024 13:16:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.9 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709903805; cv=none; b=Gcp+LVraJ4PK30I6xMM8CZ46zUS2lvZsRvP2YnVMxwqf/bDSNPvN9eUEa7vKT1imlzNyuu7SfAtQ3/+dbEZKCKgIUO+2RcojCoZG9gF4C5uuJPmzHzk1gzGUuRZQq2WOCoCllmPzLXAhlZ828AUd+5pIzS7S/CQ+zSI4N34Tbpc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709903805; c=relaxed/simple; bh=EgUgzUoucqqDqOU2t7gKumIS3qm9Toh3D3HqJIoc9lY=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=QvYDegdH8S4vVQUnlzS206a3PnoARFUb9dEM564ogZtRFq2Qlz+Y5jMKp74Xc9mQjQglcW3EJjsob9EfuqwW1fovRCYPJiQBveBZ8a7/3xZ/jGDONlZkXVWAt/99EpnwDyVXnmu4rbJJhdA4micRWCQo21je+lJNMKUaAVYuiNY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=n0+LTGFA; arc=none smtp.client-ip=192.198.163.9 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="n0+LTGFA" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1709903804; x=1741439804; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=EgUgzUoucqqDqOU2t7gKumIS3qm9Toh3D3HqJIoc9lY=; b=n0+LTGFA7aFM3U8UwzwvDUpQ5sTjVH+/guIJD9bbgzGPG6ILukLswQrO OvY0PkgSXTpNRDgvmM89PUhK1gRXCSi1pIUcrw4+x80ri74LD3WblXhWk JSqcQDuMpBjFrCuVDhiy9sAFkyZyZ4B5YfeT9onYmdBFSulnlQ6pZBNjJ i4gYEdIieyFYium16cgVbmFEEGapVXGxwahsS2H6ALvkwmowMnofLWQiS HL0UD43TPH/14KJ6a5btKTFissWklcLgfcHzahtjs9p3VelIqmXuJAPWB NWi1axYxwnIKmOc7vQkKH0EBHVURrYWa0/JNntMZO/ccTv2cQOGe50wQH g==; X-IronPort-AV: E=McAfee;i="6600,9927,11006"; a="15342715" X-IronPort-AV: E=Sophos;i="6.07,109,1708416000"; d="scan'208";a="15342715" Received: from orviesa005.jf.intel.com ([10.64.159.145]) by fmvoesa103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Mar 2024 05:16:43 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,109,1708416000"; d="scan'208";a="15161559" Received: from ahunter6-mobl1.ger.corp.intel.com (HELO ahunter-VirtualBox.home\044ger.corp.intel.com) ([10.249.46.63]) by orviesa005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Mar 2024 05:16:36 -0800 From: Adrian Hunter To: Thomas Gleixner Cc: Michael Ellerman , Nicholas Piggin , Christophe Leroy , "Aneesh Kumar K.V" , "Naveen N. Rao" , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Andy Lutomirski , Vincenzo Frascino , John Stultz , Stephen Boyd , Peter Zijlstra , Randy Dunlap , Bjorn Helgaas , Arnd Bergmann , Anna-Maria Behnsen , linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, linux-s390@vger.kernel.org Subject: [PATCH 11/19] timekeeping: Tidy timekeeping_cycles_to_ns() slightly Date: Fri, 8 Mar 2024 15:15:04 +0200 Message-Id: <20240308131512.44324-12-adrian.hunter@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240308131512.44324-1-adrian.hunter@intel.com> References: <20240308131512.44324-1-adrian.hunter@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Organization: Intel Finland Oy, Registered Address: PL 281, 00181 Helsinki, Business Identity Code: 0357606 - 4, Domiciled in Helsinki Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Put together declaration and initialization of the local variable 'delta'. Suggested-by: Thomas Gleixner Signed-off-by: Adrian Hunter --- kernel/time/timekeeping.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/kernel/time/timekeeping.c b/kernel/time/timekeeping.c index 63061332a75c..c698219b152d 100644 --- a/kernel/time/timekeeping.c +++ b/kernel/time/timekeeping.c @@ -383,10 +383,9 @@ static inline u64 timekeeping_delta_to_ns(const struct= tk_read_base *tkr, u64 de =20 static inline u64 timekeeping_cycles_to_ns(const struct tk_read_base *tkr,= u64 cycles) { - u64 delta; + /* Calculate the delta since the last update_wall_time() */ + u64 delta =3D clocksource_delta(cycles, tkr->cycle_last, tkr->mask); =20 - /* calculate the delta since the last update_wall_time */ - delta =3D clocksource_delta(cycles, tkr->cycle_last, tkr->mask); return timekeeping_delta_to_ns(tkr, delta); } =20 --=20 2.34.1 From nobody Mon Feb 9 15:59:12 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AD24A38387; Fri, 8 Mar 2024 13:16:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.9 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709903812; cv=none; b=sK6o7mmH6i2rkCydt+hlf5DVJi4H3x10EMGfTCSHH/tdR9Pq6UwSlQYOxNrtcDQueBiqqeFmdjg3oEKE04dC9DxJZPrSPEg3NQPVSD1wd526OfsXlWp3qsCeLCR04o1hixNZUcWmcgp+sjHk7xj0byktTju7Rd61218VKBEBSuY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709903812; c=relaxed/simple; bh=ul0hfPOmfS7i90s07MvVRoTCajL4zQDIAp9Wq9nGTHo=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=k+Jneco8SpbXarMh1HENw/Me6kxPY8jAxv/2gFQPLdR3xsEPKWKXGQFt4aQEhqDA5TBgxfQyBSHccsJPF3R7JcE76pQQmiVh/Lh89crzr+QU47+WnUlCyrDOicuT+br90nuRcKF9S9ghv0s0DSYv6q0zxLzFP+10cMoNNPnSU2Q= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=NfUiGBOy; arc=none smtp.client-ip=192.198.163.9 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="NfUiGBOy" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1709903811; x=1741439811; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=ul0hfPOmfS7i90s07MvVRoTCajL4zQDIAp9Wq9nGTHo=; b=NfUiGBOyiZV4+jXBf/e9uiH+/EySwh1FZmreDK9GkWWEq4rgqQVd8MCL NwBcsFZXkX7TLM2INcgSYKcEk76Ie4Yx8B83OvWHwu2VNW0C5Ha637r4A H95pnJilJ+B93WCk4WSwH9sg0LollqX4pyasR0tRtzjR09IDeYcssiQPh 3uOJhB4QPMsShEsJYPDcPg6M0bS4xGHy8aB0mSYSddIsV5JoMNBmpy58P ZFrLJwOzsTJq2TlvWMJ4v2ubrt70y+tC4YoK+L1rx4cpqPBtE/TN7SQzg BgQmjLlHujWuznMnoToNY52kSDC9477uepSZt9Lpm8bnUHi7iGBuWYI5A Q==; X-IronPort-AV: E=McAfee;i="6600,9927,11006"; a="15342750" X-IronPort-AV: E=Sophos;i="6.07,109,1708416000"; d="scan'208";a="15342750" Received: from orviesa005.jf.intel.com ([10.64.159.145]) by fmvoesa103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Mar 2024 05:16:50 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,109,1708416000"; d="scan'208";a="15161564" Received: from ahunter6-mobl1.ger.corp.intel.com (HELO ahunter-VirtualBox.home\044ger.corp.intel.com) ([10.249.46.63]) by orviesa005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Mar 2024 05:16:43 -0800 From: Adrian Hunter To: Thomas Gleixner Cc: Michael Ellerman , Nicholas Piggin , Christophe Leroy , "Aneesh Kumar K.V" , "Naveen N. Rao" , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Andy Lutomirski , Vincenzo Frascino , John Stultz , Stephen Boyd , Peter Zijlstra , Randy Dunlap , Bjorn Helgaas , Arnd Bergmann , Anna-Maria Behnsen , linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, linux-s390@vger.kernel.org Subject: [PATCH 12/19] timekeeping: Reuse timekeeping_cycles_to_ns() Date: Fri, 8 Mar 2024 15:15:05 +0200 Message-Id: <20240308131512.44324-13-adrian.hunter@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240308131512.44324-1-adrian.hunter@intel.com> References: <20240308131512.44324-1-adrian.hunter@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Organization: Intel Finland Oy, Registered Address: PL 281, 00181 Helsinki, Business Identity Code: 0357606 - 4, Domiciled in Helsinki Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Simplify __timekeeping_get_ns() by reusing timekeeping_cycles_to_ns(). Suggested-by: Thomas Gleixner Signed-off-by: Adrian Hunter --- kernel/time/timekeeping.c | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/kernel/time/timekeeping.c b/kernel/time/timekeeping.c index c698219b152d..f81d675291e0 100644 --- a/kernel/time/timekeeping.c +++ b/kernel/time/timekeeping.c @@ -391,10 +391,7 @@ static inline u64 timekeeping_cycles_to_ns(const struc= t tk_read_base *tkr, u64 c =20 static __always_inline u64 __timekeeping_get_ns(const struct tk_read_base = *tkr) { - u64 delta, cycles =3D tk_clock_read(tkr); - - delta =3D clocksource_delta(cycles, tkr->cycle_last, tkr->mask); - return timekeeping_delta_to_ns(tkr, delta); + return timekeeping_cycles_to_ns(tkr, tk_clock_read(tkr)); } =20 static inline u64 timekeeping_get_ns(const struct tk_read_base *tkr) --=20 2.34.1 From nobody Mon Feb 9 15:59:12 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4690758201; Fri, 8 Mar 2024 13:16:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.9 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709903819; cv=none; b=t9AKKrhlDzBytx+nC+StyMj9H3Vg1rYwmZu/lFmrQmxqs4Sx/2um/hnSvK5WknfBFMsoOgD3m3YxEH89ruWISx9Ttdg1MgaYFCd1EdI16oB+qOuKpuoM/x6pdAnR2qrI5R0qI3z/VMteDIPBI3M4eWoMBntaS+/WzWF0+0F5bDc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709903819; c=relaxed/simple; bh=16GQ0vc+mvjtkGYNmUSKlWdfc5Q8uWzvZ2NpVFo2g5s=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=kYBTLups2TKgphcwpAtkD2AUxEh8dbmpx3/43JZBZZdzi/KMmdkf7ZMDgJIp91SUsfHP38nRIx8wYVbzoRTARbSE9Yz0De31pB9oh7dGq38ntDfZtAoaQvTjNMHjxTiDNTTEvbUJ2HDIWrS98u7951hs3A6pskydYJzintJQQHw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=PEyPtNHG; arc=none smtp.client-ip=192.198.163.9 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="PEyPtNHG" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1709903818; x=1741439818; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=16GQ0vc+mvjtkGYNmUSKlWdfc5Q8uWzvZ2NpVFo2g5s=; b=PEyPtNHGAHMaQeIokG1H8Frt0SS5NHvLyECmGUCSWQRpJtfi03UH/f9f wUk8AUcMRXklr1dYUvnSRJQ9VsMIKJz3pbBEMTYtzcukAVlI2LUXHyl8q lyQqaFw4HLHfBkYGczB0dHR1Eh2HmFLOY35MFKfyhf2zzylLHjKeLEKW5 M9LtHJmVTjegJb4dPJqKHaFiTSYXKjt73q9hn1R39O5wIjvV4k3igPPph WS53jrQ2Py79G7Nzxq5f87lbCKzMMvMnouG6phvhNu5EaLxxttiE93sil nAwgZSB7B6R/sw5Waj2lD15fyQwLd3tIUQpH3NENQwwWDjdCY6kwRz994 Q==; X-IronPort-AV: E=McAfee;i="6600,9927,11006"; a="15342797" X-IronPort-AV: E=Sophos;i="6.07,109,1708416000"; d="scan'208";a="15342797" Received: from orviesa005.jf.intel.com ([10.64.159.145]) by fmvoesa103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Mar 2024 05:16:58 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,109,1708416000"; d="scan'208";a="15161573" Received: from ahunter6-mobl1.ger.corp.intel.com (HELO ahunter-VirtualBox.home\044ger.corp.intel.com) ([10.249.46.63]) by orviesa005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Mar 2024 05:16:50 -0800 From: Adrian Hunter To: Thomas Gleixner Cc: Michael Ellerman , Nicholas Piggin , Christophe Leroy , "Aneesh Kumar K.V" , "Naveen N. Rao" , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Andy Lutomirski , Vincenzo Frascino , John Stultz , Stephen Boyd , Peter Zijlstra , Randy Dunlap , Bjorn Helgaas , Arnd Bergmann , Anna-Maria Behnsen , linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, linux-s390@vger.kernel.org Subject: [PATCH 13/19] timekeeping: Refactor timekeeping helpers Date: Fri, 8 Mar 2024 15:15:06 +0200 Message-Id: <20240308131512.44324-14-adrian.hunter@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240308131512.44324-1-adrian.hunter@intel.com> References: <20240308131512.44324-1-adrian.hunter@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Organization: Intel Finland Oy, Registered Address: PL 281, 00181 Helsinki, Business Identity Code: 0357606 - 4, Domiciled in Helsinki Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Simplify use of timekeeping sanity checking, in preparation for consolidating timekeeping helpers. This works towards eliminating timekeeping_delta_to_ns() in favour of timekeeping_cycles_to_ns(). Suggested-by: Thomas Gleixner Signed-off-by: Adrian Hunter --- kernel/time/timekeeping.c | 20 ++++++-------------- 1 file changed, 6 insertions(+), 14 deletions(-) diff --git a/kernel/time/timekeeping.c b/kernel/time/timekeeping.c index f81d675291e0..618328cd4bc4 100644 --- a/kernel/time/timekeeping.c +++ b/kernel/time/timekeeping.c @@ -237,7 +237,7 @@ static void timekeeping_check_update(struct timekeeper = *tk, u64 offset) } } =20 -static inline u64 timekeeping_get_delta(const struct tk_read_base *tkr) +static inline u64 timekeeping_debug_get_delta(const struct tk_read_base *t= kr) { struct timekeeper *tk =3D &tk_core.timekeeper; u64 now, last, mask, max, delta; @@ -281,17 +281,9 @@ static inline u64 timekeeping_get_delta(const struct t= k_read_base *tkr) static inline void timekeeping_check_update(struct timekeeper *tk, u64 off= set) { } -static inline u64 timekeeping_get_delta(const struct tk_read_base *tkr) +static inline u64 timekeeping_debug_get_delta(const struct tk_read_base *t= kr) { - u64 cycle_now, delta; - - /* read clocksource */ - cycle_now =3D tk_clock_read(tkr); - - /* calculate the delta since the last update_wall_time */ - delta =3D clocksource_delta(cycle_now, tkr->cycle_last, tkr->mask); - - return delta; + BUG(); } #endif =20 @@ -396,10 +388,10 @@ static __always_inline u64 __timekeeping_get_ns(const= struct tk_read_base *tkr) =20 static inline u64 timekeeping_get_ns(const struct tk_read_base *tkr) { - u64 delta; + if (IS_ENABLED(CONFIG_DEBUG_TIMEKEEPING)) + return timekeeping_delta_to_ns(tkr, timekeeping_debug_get_delta(tkr)); =20 - delta =3D timekeeping_get_delta(tkr); - return timekeeping_delta_to_ns(tkr, delta); + return __timekeeping_get_ns(tkr); } =20 /** --=20 2.34.1 From nobody Mon Feb 9 15:59:12 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6A9F058AC5; Fri, 8 Mar 2024 13:17:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.9 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709903826; cv=none; b=Wq429A5gVGa6TXS2rj5Hj8llqnz3CK0F1+41S60M/pM4ezDlx4L3soWUZsg0O77t7YsDcdVnmoKkMNbb61nrOjfFxfCAJNWCqj8jnT5gbp050GYKtKuWFaVQar3BNOfbKHE5/d4K/k976fVFDYAqQxuBhH8jBD06IGPq218pi1A= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709903826; c=relaxed/simple; bh=I/EwrCmdvQfW3F1Dq8qT6GRyUT8dQLcc34j3rdWdn4M=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=nxcyQagdkq7cC/F5MFQxwpOm0apWKVAlpDdmGjpeszgXs6sSGK0HzeNj7OP9l15eMwK1eSgmM08WmMK8sIWygNq6sBxy+IG0xd1Y/H33HfDV0tG5Nk8814yyQ/oL1TCx2qW/Jcwbm3Ed7Vg2wvMNynGaqBNiE6TqQ1DshBu9Jd4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=TMKTYmab; arc=none smtp.client-ip=192.198.163.9 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="TMKTYmab" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1709903825; x=1741439825; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=I/EwrCmdvQfW3F1Dq8qT6GRyUT8dQLcc34j3rdWdn4M=; b=TMKTYmabi6SzTweRmMcK8UlmPkGn56vK9xHxKc3eO22gWVkT295tj64D Q8W7Pm9t37VncRetjZSJx9lRIrrv90B7dp2SqqBftx/7nOR67RI4QNLAF A3Fbz5ueTP133HJwoIoMfs3tR9sNkslcWspPr4+OZMh2ZgwiyizO1BsSV 25NnOuMqFWi4zoUaVIpYLdEAwKEu3NmB6H7IcIGiKBgjCCirp9YnK5E4q RCf6nzpZ4+f+bwBvACy6MSJsl1IMR0XY5maoaGQK/hHUl8NKnFgfIotIt v6phDSIyBR9ZYtcez1VgDAD2fr08FxVDmWSR3qDZOrwtCV1UcGUw2KXg1 Q==; X-IronPort-AV: E=McAfee;i="6600,9927,11006"; a="15342824" X-IronPort-AV: E=Sophos;i="6.07,109,1708416000"; d="scan'208";a="15342824" Received: from orviesa005.jf.intel.com ([10.64.159.145]) by fmvoesa103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Mar 2024 05:17:04 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,109,1708416000"; d="scan'208";a="15161581" Received: from ahunter6-mobl1.ger.corp.intel.com (HELO ahunter-VirtualBox.home\044ger.corp.intel.com) ([10.249.46.63]) by orviesa005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Mar 2024 05:16:57 -0800 From: Adrian Hunter To: Thomas Gleixner Cc: Michael Ellerman , Nicholas Piggin , Christophe Leroy , "Aneesh Kumar K.V" , "Naveen N. Rao" , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Andy Lutomirski , Vincenzo Frascino , John Stultz , Stephen Boyd , Peter Zijlstra , Randy Dunlap , Bjorn Helgaas , Arnd Bergmann , Anna-Maria Behnsen , linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, linux-s390@vger.kernel.org Subject: [PATCH 14/19] timekeeping: Consolidate timekeeping helpers Date: Fri, 8 Mar 2024 15:15:07 +0200 Message-Id: <20240308131512.44324-15-adrian.hunter@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240308131512.44324-1-adrian.hunter@intel.com> References: <20240308131512.44324-1-adrian.hunter@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Organization: Intel Finland Oy, Registered Address: PL 281, 00181 Helsinki, Business Identity Code: 0357606 - 4, Domiciled in Helsinki Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Consolidate timekeeping helpers, making use of timekeeping_cycles_to_ns() in preference to directly using timekeeping_delta_to_ns(). Suggested-by: Thomas Gleixner Signed-off-by: Adrian Hunter --- kernel/time/timekeeping.c | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/kernel/time/timekeeping.c b/kernel/time/timekeeping.c index 618328cd4bc4..1bbfe1ff8d24 100644 --- a/kernel/time/timekeeping.c +++ b/kernel/time/timekeeping.c @@ -237,7 +237,9 @@ static void timekeeping_check_update(struct timekeeper = *tk, u64 offset) } } =20 -static inline u64 timekeeping_debug_get_delta(const struct tk_read_base *t= kr) +static inline u64 timekeeping_cycles_to_ns(const struct tk_read_base *tkr,= u64 cycles); + +static inline u64 timekeeping_debug_get_ns(const struct tk_read_base *tkr) { struct timekeeper *tk =3D &tk_core.timekeeper; u64 now, last, mask, max, delta; @@ -266,22 +268,22 @@ static inline u64 timekeeping_debug_get_delta(const s= truct tk_read_base *tkr) */ if (unlikely((~delta & mask) < (mask >> 3))) { tk->underflow_seen =3D 1; - delta =3D 0; + now =3D last; } =20 /* Cap delta value to the max_cycles values to avoid mult overflows */ if (unlikely(delta > max)) { tk->overflow_seen =3D 1; - delta =3D tkr->clock->max_cycles; + now =3D last + max; } =20 - return delta; + return timekeeping_cycles_to_ns(tkr, now); } #else static inline void timekeeping_check_update(struct timekeeper *tk, u64 off= set) { } -static inline u64 timekeeping_debug_get_delta(const struct tk_read_base *t= kr) +static inline u64 timekeeping_debug_get_ns(const struct tk_read_base *tkr) { BUG(); } @@ -389,7 +391,7 @@ static __always_inline u64 __timekeeping_get_ns(const s= truct tk_read_base *tkr) static inline u64 timekeeping_get_ns(const struct tk_read_base *tkr) { if (IS_ENABLED(CONFIG_DEBUG_TIMEKEEPING)) - return timekeeping_delta_to_ns(tkr, timekeeping_debug_get_delta(tkr)); + return timekeeping_debug_get_ns(tkr); =20 return __timekeeping_get_ns(tkr); } --=20 2.34.1 From nobody Mon Feb 9 15:59:12 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E42E240856; Fri, 8 Mar 2024 13:17:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.9 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709903833; cv=none; b=fe9C1uawRmX85BlFeKCwJy2VINgR+TYfof1fd//IZVLjRZIu1JK3/rmAxR4PZPHVg/6qoDbqqRq5QsBzhxm/jyejO7FMvtLXo5CeAJSjn6Z/+lSwLz8vbPb2YFZdxVBjsnuCRzyjo6A0iLtZKwrgbUzAnFmZHnlJKkfUPiulPVk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709903833; c=relaxed/simple; bh=XagHsKZy0o0b/SfCKSGfthtDX1F7t80ikEGTTX3TaPM=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=oW45qaC/QhfoCM/AVbzjHzNA8ySRngyyH2HZu+Kyqy3chCPe6CpGidAsoYgAR2xbfeH2/+9gXpIYFuqfal53STnFLBwRk7ITUIiNbAJwqA3pHFVJpMhVmx3ZW0zVGS3Ua601WFjOsUMTANnPcsYHYLfultkptuQKH6XnQ6b3k4g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=bHR9MvAj; arc=none smtp.client-ip=192.198.163.9 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="bHR9MvAj" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1709903832; x=1741439832; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=XagHsKZy0o0b/SfCKSGfthtDX1F7t80ikEGTTX3TaPM=; b=bHR9MvAjaqnQxkyg2Gsn3BDEvOde6jql6Vk/hG2/5BNMgzCSIqz08zrs O9UbyRuoU0BW7aeBQ26CZtENnWrOjO5E+kvaDlWFhDbm7XvvzN6+NqGqF /dHDhnjfXPvhvh33suQkh47xtDrG/a4YcB9uhGDz0m+qzujQQBUS+/46U yh8mSdnV0dcKIbNMiDcOyvn/5Co0u46GPCukl9deg01rvbKlvJDInK8o9 Et/bfPjjvmZPe/Kff7xezDBOvMRLhv9BkqcZqXUikb9blbjNWTaPHiV9r aMsIEQYLR+c0q2aXLPFf0LkouZK4dw+JHxMMn3WTuz5dBGJv2GiLcOzbb g==; X-IronPort-AV: E=McAfee;i="6600,9927,11006"; a="15342855" X-IronPort-AV: E=Sophos;i="6.07,109,1708416000"; d="scan'208";a="15342855" Received: from orviesa005.jf.intel.com ([10.64.159.145]) by fmvoesa103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Mar 2024 05:17:11 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,109,1708416000"; d="scan'208";a="15161586" Received: from ahunter6-mobl1.ger.corp.intel.com (HELO ahunter-VirtualBox.home\044ger.corp.intel.com) ([10.249.46.63]) by orviesa005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Mar 2024 05:17:04 -0800 From: Adrian Hunter To: Thomas Gleixner Cc: Michael Ellerman , Nicholas Piggin , Christophe Leroy , "Aneesh Kumar K.V" , "Naveen N. Rao" , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Andy Lutomirski , Vincenzo Frascino , John Stultz , Stephen Boyd , Peter Zijlstra , Randy Dunlap , Bjorn Helgaas , Arnd Bergmann , Anna-Maria Behnsen , linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, linux-s390@vger.kernel.org Subject: [PATCH 15/19] timekeeping: Fold in timekeeping_delta_to_ns() Date: Fri, 8 Mar 2024 15:15:08 +0200 Message-Id: <20240308131512.44324-16-adrian.hunter@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240308131512.44324-1-adrian.hunter@intel.com> References: <20240308131512.44324-1-adrian.hunter@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Organization: Intel Finland Oy, Registered Address: PL 281, 00181 Helsinki, Business Identity Code: 0357606 - 4, Domiciled in Helsinki Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" timekeeping_delta_to_ns() is now called only from timekeeping_cycles_to_ns(), and it is not useful otherwise. Simplify by folding it into timekeeping_cycles_to_ns(). Suggested-by: Thomas Gleixner Signed-off-by: Adrian Hunter --- kernel/time/timekeeping.c | 13 +------------ 1 file changed, 1 insertion(+), 12 deletions(-) diff --git a/kernel/time/timekeeping.c b/kernel/time/timekeeping.c index 1bbfe1ff8d24..749387f22f0f 100644 --- a/kernel/time/timekeeping.c +++ b/kernel/time/timekeeping.c @@ -364,23 +364,12 @@ static void tk_setup_internals(struct timekeeper *tk,= struct clocksource *clock) } =20 /* Timekeeper helper functions. */ - -static inline u64 timekeeping_delta_to_ns(const struct tk_read_base *tkr, = u64 delta) -{ - u64 nsec; - - nsec =3D delta * tkr->mult + tkr->xtime_nsec; - nsec >>=3D tkr->shift; - - return nsec; -} - static inline u64 timekeeping_cycles_to_ns(const struct tk_read_base *tkr,= u64 cycles) { /* Calculate the delta since the last update_wall_time() */ u64 delta =3D clocksource_delta(cycles, tkr->cycle_last, tkr->mask); =20 - return timekeeping_delta_to_ns(tkr, delta); + return ((delta * tkr->mult) + tkr->xtime_nsec) >> tkr->shift; } =20 static __always_inline u64 __timekeeping_get_ns(const struct tk_read_base = *tkr) --=20 2.34.1 From nobody Mon Feb 9 15:59:12 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5438240856; Fri, 8 Mar 2024 13:17:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.9 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709903839; cv=none; b=cmd19LWbEqoZD9BnVZtvMTGm9SnU/CvtWtsVUHYhtWlMIA3N/PbjLCBqb0wQfx/vZbqB04kl1rWyPbbuo+plYingMcF4buMD0YJ4/jaoSe1rJtOAsjdbu7IZfly2MyVJciwVSRw8MuKoR9o2h+b+uAnKqUJOKv0f3CvLOTRXDpQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709903839; c=relaxed/simple; bh=DWUH05pMFGBZtgh8QTM1fOtl5JuwFtwudYwKfS+CudI=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=nnjjgtykDRn9D6djKyLPfVdQNK1c3LyDnP3pfdRtWS474gd9iQQdjsMLqiGNDo/RUUBO6Alvg5MJT59J+XrLmLI6Lld8RqH5m2cUITo58aZynOowU74KIVtnUDWfwIyqGeBTHWTjmlBvCxql0kPSAFWzHRvW988o3bPVQn5i+wI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=Rlh/Hufe; arc=none smtp.client-ip=192.198.163.9 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="Rlh/Hufe" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1709903838; x=1741439838; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=DWUH05pMFGBZtgh8QTM1fOtl5JuwFtwudYwKfS+CudI=; b=Rlh/HufegFvRLv5oAj4aJNd+EEh/Xfl36NqoNDiDO5Wj873wZhJz8X8h 0Ev7/NhVhiqqN/6GVKQZmsXKuKpM/Jvbm1TbCpAgzAZ+IUcuwaNNXlHkk fV2fzClrqTRuB0ZjpIvf06EPmzc16NaaNFa3uH33vfeq/zk+MMyzX8gr2 lcFHHowllXGZyns5qnPBN/RygdzAQr5LZssIwVfYq2/MVDPp1/3/LsW4r eYeGPZ9Ki3f+JwXi8voObU4J7abNwPnIAOxOoNq/FRQ4yBfPeiQJ9zgC4 YN/+CfUPqHSNs3ZHWlO1eL95KhVhu9oMQaccx/7jJ9JdQ6MpuSUo4kJuk w==; X-IronPort-AV: E=McAfee;i="6600,9927,11006"; a="15342880" X-IronPort-AV: E=Sophos;i="6.07,109,1708416000"; d="scan'208";a="15342880" Received: from orviesa005.jf.intel.com ([10.64.159.145]) by fmvoesa103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Mar 2024 05:17:18 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,109,1708416000"; d="scan'208";a="15161597" Received: from ahunter6-mobl1.ger.corp.intel.com (HELO ahunter-VirtualBox.home\044ger.corp.intel.com) ([10.249.46.63]) by orviesa005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Mar 2024 05:17:11 -0800 From: Adrian Hunter To: Thomas Gleixner Cc: Michael Ellerman , Nicholas Piggin , Christophe Leroy , "Aneesh Kumar K.V" , "Naveen N. Rao" , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Andy Lutomirski , Vincenzo Frascino , John Stultz , Stephen Boyd , Peter Zijlstra , Randy Dunlap , Bjorn Helgaas , Arnd Bergmann , Anna-Maria Behnsen , linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, linux-s390@vger.kernel.org Subject: [PATCH 16/19] timekeeping: Prepare timekeeping_cycles_to_ns() for overflow safety Date: Fri, 8 Mar 2024 15:15:09 +0200 Message-Id: <20240308131512.44324-17-adrian.hunter@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240308131512.44324-1-adrian.hunter@intel.com> References: <20240308131512.44324-1-adrian.hunter@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Organization: Intel Finland Oy, Registered Address: PL 281, 00181 Helsinki, Business Identity Code: 0357606 - 4, Domiciled in Helsinki Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Open code clocksource_delta() in timekeeping_cycles_to_ns() so that overflow safety can be added efficiently. Suggested-by: Thomas Gleixner Signed-off-by: Adrian Hunter --- kernel/time/timekeeping.c | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/kernel/time/timekeeping.c b/kernel/time/timekeeping.c index 749387f22f0f..d17484082e2c 100644 --- a/kernel/time/timekeeping.c +++ b/kernel/time/timekeeping.c @@ -367,7 +367,17 @@ static void tk_setup_internals(struct timekeeper *tk, = struct clocksource *clock) static inline u64 timekeeping_cycles_to_ns(const struct tk_read_base *tkr,= u64 cycles) { /* Calculate the delta since the last update_wall_time() */ - u64 delta =3D clocksource_delta(cycles, tkr->cycle_last, tkr->mask); + u64 mask =3D tkr->mask, delta =3D (cycles - tkr->cycle_last) & mask; + + if (IS_ENABLED(CONFIG_CLOCKSOURCE_VALIDATE_LAST_CYCLE)) { + /* + * Handle clocksource inconsistency between CPUs to prevent + * time from going backwards by checking for the MSB of the + * mask being set in the delta. + */ + if (unlikely(delta & ~(mask >> 1))) + return tkr->xtime_nsec >> tkr->shift; + } =20 return ((delta * tkr->mult) + tkr->xtime_nsec) >> tkr->shift; } --=20 2.34.1 From nobody Mon Feb 9 15:59:12 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F0D7B4C627; Fri, 8 Mar 2024 13:17:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.9 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709903847; cv=none; b=Jyfmjs7AgxV35ADznSUfwQXkacefJQt64DfW8U2ZRBfF0ylojV8nfgGtelatXQGZ+q2pEkGZ0hK6tRt/i16E3DDbjfV9gWFGn09uAlBa2QSoqP4iPhLXfICgP8tkKowLQUS05T1MuxcNeWqwTF+aiyTpK1el63LRYsIQwsGVHeU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709903847; c=relaxed/simple; bh=xuu117cG1lowhe30UpLIG5BgAby+hnwaDFUtdrI4YDU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=rBWAYzrQyqGacIb1WzKq4KCV3ehn2rpp6seI17jiGBSRep4qAtFG7Cmhv1YP9NWV3Ea4pbu4BsIqbweGZylByAN6AMlFHa05IW3YKPkX5Dsm0i+i+w9QhQMDZsKcJcsc7dhDXuViYVPFXhqMbYAd6PGeYG795I/6ewl2ichUduE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=kVV262wt; arc=none smtp.client-ip=192.198.163.9 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="kVV262wt" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1709903846; x=1741439846; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=xuu117cG1lowhe30UpLIG5BgAby+hnwaDFUtdrI4YDU=; b=kVV262wt53lXrVuccER/6ZhgkEZ5lolyETvnrVOFdl4Tas2CnAzZh6oq QIUKPlLh7+qJCrSpJOVbqRh/RjOR9SHcOaWg5kFGP8BcCGnrJxw+rZ5nQ mPrnV/CpshfgTLFPUC99B8APdyz0Jg7wkWy6ZyoqPXRynreUi0YVwuG3D LTwhsCiSXghJ1VCklqSGTRC+QSQ+S6tJgRbcBIXlUlveW4kn6fT0iXUbB V5ClsKWlJYSvdawleTN/CmA85swSRQJBhBHUd59ZO6f8xxC9D/sc1yJZr q/7Ush0ZhZUeti5dIQcA1saZo8zuR4AUHXIc3NERCYKI7TAIAONyvZI1W g==; X-IronPort-AV: E=McAfee;i="6600,9927,11006"; a="15342906" X-IronPort-AV: E=Sophos;i="6.07,109,1708416000"; d="scan'208";a="15342906" Received: from orviesa005.jf.intel.com ([10.64.159.145]) by fmvoesa103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Mar 2024 05:17:25 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,109,1708416000"; d="scan'208";a="15161603" Received: from ahunter6-mobl1.ger.corp.intel.com (HELO ahunter-VirtualBox.home\044ger.corp.intel.com) ([10.249.46.63]) by orviesa005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Mar 2024 05:17:18 -0800 From: Adrian Hunter To: Thomas Gleixner Cc: Michael Ellerman , Nicholas Piggin , Christophe Leroy , "Aneesh Kumar K.V" , "Naveen N. Rao" , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Andy Lutomirski , Vincenzo Frascino , John Stultz , Stephen Boyd , Peter Zijlstra , Randy Dunlap , Bjorn Helgaas , Arnd Bergmann , Anna-Maria Behnsen , linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, linux-s390@vger.kernel.org Subject: [PATCH 17/19] timekeeping: Make delta calculation overflow safe Date: Fri, 8 Mar 2024 15:15:10 +0200 Message-Id: <20240308131512.44324-18-adrian.hunter@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240308131512.44324-1-adrian.hunter@intel.com> References: <20240308131512.44324-1-adrian.hunter@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Organization: Intel Finland Oy, Registered Address: PL 281, 00181 Helsinki, Business Identity Code: 0357606 - 4, Domiciled in Helsinki Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Kernel timekeeping is designed to keep the change in cycles (since the last timer interrupt) below max_cycles, which prevents multiplication overflow when converting cycles to nanoseconds. However, if timer interrupts stop, the calculation will eventually overflow. Add protection against that. In timekeeping_cycles_to_ns() calculation, check against max_cycles, falling back to a slower higher precision calculation. In timekeeping_forward_now(), process delta in chunks of at most max_cycles. Suggested-by: Thomas Gleixner Signed-off-by: Adrian Hunter --- kernel/time/timekeeping.c | 40 ++++++++++++++++++++++++++++----------- 1 file changed, 29 insertions(+), 11 deletions(-) diff --git a/kernel/time/timekeeping.c b/kernel/time/timekeeping.c index d17484082e2c..111dfdbd488f 100644 --- a/kernel/time/timekeeping.c +++ b/kernel/time/timekeeping.c @@ -364,19 +364,32 @@ static void tk_setup_internals(struct timekeeper *tk,= struct clocksource *clock) } =20 /* Timekeeper helper functions. */ +static noinline u64 delta_to_ns_safe(const struct tk_read_base *tkr, u64 d= elta) +{ + return mul_u64_u32_add_u64_shr(delta, tkr->mult, tkr->xtime_nsec, tkr->sh= ift); +} + static inline u64 timekeeping_cycles_to_ns(const struct tk_read_base *tkr,= u64 cycles) { /* Calculate the delta since the last update_wall_time() */ u64 mask =3D tkr->mask, delta =3D (cycles - tkr->cycle_last) & mask; =20 - if (IS_ENABLED(CONFIG_CLOCKSOURCE_VALIDATE_LAST_CYCLE)) { - /* - * Handle clocksource inconsistency between CPUs to prevent - * time from going backwards by checking for the MSB of the - * mask being set in the delta. - */ - if (unlikely(delta & ~(mask >> 1))) - return tkr->xtime_nsec >> tkr->shift; + /* + * This detects the case where the delta overflows the multiplication + * with tkr->mult. + */ + if (unlikely(delta > tkr->clock->max_cycles)) { + if (IS_ENABLED(CONFIG_CLOCKSOURCE_VALIDATE_LAST_CYCLE)) { + /* + * Handle clocksource inconsistency between CPUs to prevent + * time from going backwards by checking for the MSB of the + * mask being set in the delta. + */ + if (unlikely(delta & ~(mask >> 1))) + return tkr->xtime_nsec >> tkr->shift; + } + + return delta_to_ns_safe(tkr, delta); } =20 return ((delta * tkr->mult) + tkr->xtime_nsec) >> tkr->shift; @@ -789,10 +802,15 @@ static void timekeeping_forward_now(struct timekeeper= *tk) tk->tkr_mono.cycle_last =3D cycle_now; tk->tkr_raw.cycle_last =3D cycle_now; =20 - tk->tkr_mono.xtime_nsec +=3D delta * tk->tkr_mono.mult; - tk->tkr_raw.xtime_nsec +=3D delta * tk->tkr_raw.mult; + while (delta > 0) { + u64 max =3D tk->tkr_mono.clock->max_cycles; + u64 incr =3D delta < max ? delta : max; =20 - tk_normalize_xtime(tk); + tk->tkr_mono.xtime_nsec +=3D incr * tk->tkr_mono.mult; + tk->tkr_raw.xtime_nsec +=3D incr * tk->tkr_raw.mult; + tk_normalize_xtime(tk); + delta -=3D incr; + } } =20 /** --=20 2.34.1 From nobody Mon Feb 9 15:59:12 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 485205427E; Fri, 8 Mar 2024 13:17:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.9 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709903853; cv=none; b=pv4uzQWoJ4ojrGPaEIpzz+4k3L2LaFTHA0t/Yvvjp8lvJvFm0zXNMDog5Uvl7trpMNEmUGyJ12UHbUl+MaeDce4CN6sxLK+7doRgAnnLGEVX7xynvv7m+g15l0vZNKY10PnYHCjX+9kmbHwpgXP5gK+95DT+K0eb9kyn0kXAXhY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709903853; c=relaxed/simple; bh=qOOcsyBxpUsFxkAH1hi2UNpYCz2148dQEbBRMVKVw/4=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=njVPUfjSqqgOT1ucmk1KKv8I9NewZdeHbnVMt84qE+FwWS0fUa89umMN+snQl5VoJ6uA4l1ZmPeDICER7lQZhvV+BVmFMf+QUl2rkBX4aFiUX6q3sBiCen3tNkXEyTO2MJMxZPlaQagQjlcs7aiMfeyoNG0qN5zl7YskdWtB6CI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=JMIScr6W; arc=none smtp.client-ip=192.198.163.9 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="JMIScr6W" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1709903852; x=1741439852; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=qOOcsyBxpUsFxkAH1hi2UNpYCz2148dQEbBRMVKVw/4=; b=JMIScr6WEWbd9C6QngGhKDh3rKzNdpKf+t3DmDH3TXi73/csFH6nu19i g/nvvPpyVaVzcye5UjqVqgN6a/K1YQWgDFi2lUc5AqzrnH5tq5pUemB9+ Y8AHaLmmcdXV2N5voWCQYNv0ky1TQGnXcf3zEXISVVNz3qdOh6AnWxqqv dW4SIt3qMVxEC09lkk798rOb8KBPQYKHndtT8LGAxvu23qs6mqnRe1XS4 dF4OdRkLDw3GKylkCICinzE6PyNF8fYy8jhbXF06UYj4lHqsqM98VrdKM vjHRT31INNUqm2d/NvADqIbr7zWk8mNx8qJZ4axtDnqmLkK0WPXPSx6u5 g==; X-IronPort-AV: E=McAfee;i="6600,9927,11006"; a="15342936" X-IronPort-AV: E=Sophos;i="6.07,109,1708416000"; d="scan'208";a="15342936" Received: from orviesa005.jf.intel.com ([10.64.159.145]) by fmvoesa103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Mar 2024 05:17:32 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,109,1708416000"; d="scan'208";a="15161611" Received: from ahunter6-mobl1.ger.corp.intel.com (HELO ahunter-VirtualBox.home\044ger.corp.intel.com) ([10.249.46.63]) by orviesa005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Mar 2024 05:17:25 -0800 From: Adrian Hunter To: Thomas Gleixner Cc: Michael Ellerman , Nicholas Piggin , Christophe Leroy , "Aneesh Kumar K.V" , "Naveen N. Rao" , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Andy Lutomirski , Vincenzo Frascino , John Stultz , Stephen Boyd , Peter Zijlstra , Randy Dunlap , Bjorn Helgaas , Arnd Bergmann , Anna-Maria Behnsen , linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, linux-s390@vger.kernel.org Subject: [PATCH 18/19] timekeeping: Let timekeeping_cycles_to_ns() handle both under and overflow Date: Fri, 8 Mar 2024 15:15:11 +0200 Message-Id: <20240308131512.44324-19-adrian.hunter@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240308131512.44324-1-adrian.hunter@intel.com> References: <20240308131512.44324-1-adrian.hunter@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Organization: Intel Finland Oy, Registered Address: PL 281, 00181 Helsinki, Business Identity Code: 0357606 - 4, Domiciled in Helsinki Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" For the case !CONFIG_CLOCKSOURCE_VALIDATE_LAST_CYCLE, forego overflow protection in the range (mask << 1) < delta <=3D mask, and interpret it always as an inconsistency between CPU clock values. That allows slightly neater code, and it is on a slow path so has no effect on performance. Suggested-by: Thomas Gleixner Signed-off-by: Adrian Hunter --- kernel/time/timekeeping.c | 31 +++++++++++++------------------ 1 file changed, 13 insertions(+), 18 deletions(-) diff --git a/kernel/time/timekeeping.c b/kernel/time/timekeeping.c index 111dfdbd488f..4e18db1819f8 100644 --- a/kernel/time/timekeeping.c +++ b/kernel/time/timekeeping.c @@ -266,17 +266,14 @@ static inline u64 timekeeping_debug_get_ns(const stru= ct tk_read_base *tkr) * Try to catch underflows by checking if we are seeing small * mask-relative negative values. */ - if (unlikely((~delta & mask) < (mask >> 3))) { + if (unlikely((~delta & mask) < (mask >> 3))) tk->underflow_seen =3D 1; - now =3D last; - } =20 - /* Cap delta value to the max_cycles values to avoid mult overflows */ - if (unlikely(delta > max)) { + /* Check for multiplication overflows */ + if (unlikely(delta > max)) tk->overflow_seen =3D 1; - now =3D last + max; - } =20 + /* timekeeping_cycles_to_ns() handles both under and overflow */ return timekeeping_cycles_to_ns(tkr, now); } #else @@ -375,19 +372,17 @@ static inline u64 timekeeping_cycles_to_ns(const stru= ct tk_read_base *tkr, u64 c u64 mask =3D tkr->mask, delta =3D (cycles - tkr->cycle_last) & mask; =20 /* - * This detects the case where the delta overflows the multiplication - * with tkr->mult. + * This detects both negative motion and the case where the delta + * overflows the multiplication with tkr->mult. */ if (unlikely(delta > tkr->clock->max_cycles)) { - if (IS_ENABLED(CONFIG_CLOCKSOURCE_VALIDATE_LAST_CYCLE)) { - /* - * Handle clocksource inconsistency between CPUs to prevent - * time from going backwards by checking for the MSB of the - * mask being set in the delta. - */ - if (unlikely(delta & ~(mask >> 1))) - return tkr->xtime_nsec >> tkr->shift; - } + /* + * Handle clocksource inconsistency between CPUs to prevent + * time from going backwards by checking for the MSB of the + * mask being set in the delta. + */ + if (delta & ~(mask >> 1)) + return tkr->xtime_nsec >> tkr->shift; =20 return delta_to_ns_safe(tkr, delta); } --=20 2.34.1 From nobody Mon Feb 9 15:59:12 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E8F751EF0D; Fri, 8 Mar 2024 13:17:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.9 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709903860; cv=none; b=kHCCTUvIhKCaV7nEYIojaoaKARayVHCAu/ho5B1VKbUtmHDr5rfuYuBqn7Rk5M2mdD3KjtGRcLbCQEu2Y6kl5nIjRvcLmVGncLhIFbF06y+ut0HmO783vH3aMHVyB4tWEMxScAFmyf24fvM2M7KFki1kAK8I1/gjiQwogIybH+k= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709903860; c=relaxed/simple; bh=jWLLlF4KdyRZzGfn4U2lSdQ4Wei7Cj+yXLsz+I6EvZY=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=FS8hmDd4ACQTk0hUagyPWnOUe56UnWP527U6iBqYQ0PoB9J8yonel2eYADJ2tdw+OOYpFlMqdYLwhlGkHE1hZDc204qMtOjrSFr7B4pDNa8vUEwBGXpPYxKyRRRr66OWCTIIczMZ60yCsJG0Xs6Ax3yB678AiD1IDwA6KOulDlI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=f33PGGSg; arc=none smtp.client-ip=192.198.163.9 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="f33PGGSg" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1709903859; x=1741439859; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=jWLLlF4KdyRZzGfn4U2lSdQ4Wei7Cj+yXLsz+I6EvZY=; b=f33PGGSgb/HnKtokrdoH1eMQqRHhCRKtSs1SsqkFp7FcngBHuJzO6y+u UiPNKhv0ivszANvYaZXsZhhFBFtkhX7GYjd4eXKeSbid4dzNvCCBLzz3/ 6z3kaYVWmMOpDb0kD0T77DAXhDfFs8paheKeIIBdtwkT+aswt6KL/LCtk cBcwitGEIe0SS3f9KYNBQF7zNHt4xrNRCe8ZNkxzDLA6ecLqwnMnPddrz tnZ80KwULdSn2nioK2TsM9sFUlSs0CFgeK7f0Zw29MZrs5HTR5QNWh0aX WXhzkKzWcagG3Xrvh+Gpm7inS+/TynqxSnWbZFgAAMgfDB6OSNc3erhJ3 g==; X-IronPort-AV: E=McAfee;i="6600,9927,11006"; a="15342951" X-IronPort-AV: E=Sophos;i="6.07,109,1708416000"; d="scan'208";a="15342951" Received: from orviesa005.jf.intel.com ([10.64.159.145]) by fmvoesa103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Mar 2024 05:17:38 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,109,1708416000"; d="scan'208";a="15161627" Received: from ahunter6-mobl1.ger.corp.intel.com (HELO ahunter-VirtualBox.home\044ger.corp.intel.com) ([10.249.46.63]) by orviesa005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Mar 2024 05:17:32 -0800 From: Adrian Hunter To: Thomas Gleixner Cc: Michael Ellerman , Nicholas Piggin , Christophe Leroy , "Aneesh Kumar K.V" , "Naveen N. Rao" , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Andy Lutomirski , Vincenzo Frascino , John Stultz , Stephen Boyd , Peter Zijlstra , Randy Dunlap , Bjorn Helgaas , Arnd Bergmann , Anna-Maria Behnsen , linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, linux-s390@vger.kernel.org Subject: [PATCH 19/19] clocksource: Make watchdog and suspend-timing multiplication overflow safe Date: Fri, 8 Mar 2024 15:15:12 +0200 Message-Id: <20240308131512.44324-20-adrian.hunter@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240308131512.44324-1-adrian.hunter@intel.com> References: <20240308131512.44324-1-adrian.hunter@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Organization: Intel Finland Oy, Registered Address: PL 281, 00181 Helsinki, Business Identity Code: 0357606 - 4, Domiciled in Helsinki Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Kernel timekeeping is designed to keep the change in cycles (since the last timer interrupt) below max_cycles, which prevents multiplication overflow when converting cycles to nanoseconds. However, if timer interrupts stop, the clocksource_cyc2ns() calculation will eventually overflow. Add protection against that. Simplify by folding together clocksource_delta() and clocksource_cyc2ns() into cycles_to_nsec_safe(). Check against max_cycles, falling back to a slower higher precision calculation. Suggested-by: Thomas Gleixner Signed-off-by: Adrian Hunter --- kernel/time/clocksource.c | 42 +++++++++++++++++++-------------------- 1 file changed, 20 insertions(+), 22 deletions(-) diff --git a/kernel/time/clocksource.c b/kernel/time/clocksource.c index e5b260aa0e02..4d50d53ac719 100644 --- a/kernel/time/clocksource.c +++ b/kernel/time/clocksource.c @@ -20,6 +20,16 @@ #include "tick-internal.h" #include "timekeeping_internal.h" =20 +static noinline u64 cycles_to_nsec_safe(struct clocksource *cs, u64 start,= u64 end) +{ + u64 delta =3D clocksource_delta(end, start, cs->mask); + + if (likely(delta < cs->max_cycles)) + return clocksource_cyc2ns(delta, cs->mult, cs->shift); + + return mul_u64_u32_shr(delta, cs->mult, cs->shift); +} + /** * clocks_calc_mult_shift - calculate mult/shift factors for scaled math o= f clocks * @mult: pointer to mult variable @@ -222,8 +232,8 @@ enum wd_read_status { static enum wd_read_status cs_watchdog_read(struct clocksource *cs, u64 *c= snow, u64 *wdnow) { unsigned int nretries, max_retries; - u64 wd_end, wd_end2, wd_delta; int64_t wd_delay, wd_seq_delay; + u64 wd_end, wd_end2; =20 max_retries =3D clocksource_get_max_watchdog_retry(); for (nretries =3D 0; nretries <=3D max_retries; nretries++) { @@ -234,9 +244,7 @@ static enum wd_read_status cs_watchdog_read(struct cloc= ksource *cs, u64 *csnow, wd_end2 =3D watchdog->read(watchdog); local_irq_enable(); =20 - wd_delta =3D clocksource_delta(wd_end, *wdnow, watchdog->mask); - wd_delay =3D clocksource_cyc2ns(wd_delta, watchdog->mult, - watchdog->shift); + wd_delay =3D cycles_to_nsec_safe(watchdog, *wdnow, wd_end); if (wd_delay <=3D WATCHDOG_MAX_SKEW) { if (nretries > 1 || nretries >=3D max_retries) { pr_warn("timekeeping watchdog on CPU%d: %s retried %d times before suc= cess\n", @@ -254,8 +262,7 @@ static enum wd_read_status cs_watchdog_read(struct cloc= ksource *cs, u64 *csnow, * report system busy, reinit the watchdog and skip the current * watchdog test. */ - wd_delta =3D clocksource_delta(wd_end2, wd_end, watchdog->mask); - wd_seq_delay =3D clocksource_cyc2ns(wd_delta, watchdog->mult, watchdog->= shift); + wd_seq_delay =3D cycles_to_nsec_safe(watchdog, wd_end, wd_end2); if (wd_seq_delay > WATCHDOG_MAX_SKEW/2) goto skip_test; } @@ -366,8 +373,7 @@ void clocksource_verify_percpu(struct clocksource *cs) delta =3D (csnow_end - csnow_mid) & cs->mask; if (delta < 0) cpumask_set_cpu(cpu, &cpus_ahead); - delta =3D clocksource_delta(csnow_end, csnow_begin, cs->mask); - cs_nsec =3D clocksource_cyc2ns(delta, cs->mult, cs->shift); + cs_nsec =3D cycles_to_nsec_safe(cs, csnow_begin, csnow_end); if (cs_nsec > cs_nsec_max) cs_nsec_max =3D cs_nsec; if (cs_nsec < cs_nsec_min) @@ -398,8 +404,8 @@ static inline void clocksource_reset_watchdog(void) =20 static void clocksource_watchdog(struct timer_list *unused) { - u64 csnow, wdnow, cslast, wdlast, delta; int64_t wd_nsec, cs_nsec, interval; + u64 csnow, wdnow, cslast, wdlast; int next_cpu, reset_pending; struct clocksource *cs; enum wd_read_status read_ret; @@ -456,12 +462,8 @@ static void clocksource_watchdog(struct timer_list *un= used) continue; } =20 - delta =3D clocksource_delta(wdnow, cs->wd_last, watchdog->mask); - wd_nsec =3D clocksource_cyc2ns(delta, watchdog->mult, - watchdog->shift); - - delta =3D clocksource_delta(csnow, cs->cs_last, cs->mask); - cs_nsec =3D clocksource_cyc2ns(delta, cs->mult, cs->shift); + wd_nsec =3D cycles_to_nsec_safe(watchdog, cs->wd_last, wdnow); + cs_nsec =3D cycles_to_nsec_safe(cs, cs->cs_last, csnow); wdlast =3D cs->wd_last; /* save these in case we print them */ cslast =3D cs->cs_last; cs->cs_last =3D csnow; @@ -832,7 +834,7 @@ void clocksource_start_suspend_timing(struct clocksourc= e *cs, u64 start_cycles) */ u64 clocksource_stop_suspend_timing(struct clocksource *cs, u64 cycle_now) { - u64 now, delta, nsec =3D 0; + u64 now, nsec =3D 0; =20 if (!suspend_clocksource) return 0; @@ -847,12 +849,8 @@ u64 clocksource_stop_suspend_timing(struct clocksource= *cs, u64 cycle_now) else now =3D suspend_clocksource->read(suspend_clocksource); =20 - if (now > suspend_start) { - delta =3D clocksource_delta(now, suspend_start, - suspend_clocksource->mask); - nsec =3D mul_u64_u32_shr(delta, suspend_clocksource->mult, - suspend_clocksource->shift); - } + if (now > suspend_start) + nsec =3D cycles_to_nsec_safe(suspend_clocksource, suspend_start, now); =20 /* * Disable the suspend timer to save power if current clocksource is --=20 2.34.1