From nobody Mon Dec 1 22:37:29 2025 Received: from mail-qt1-f202.google.com (mail-qt1-f202.google.com [209.85.160.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 889101391 for ; Sat, 29 Nov 2025 09:57:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764410267; cv=none; b=PkyfUTrJjBNyoXvxVEE7x+C1jhzWR51d1KximVuaWl0l61AZI9fD0/6NO3mpAGybBsf7qcyyg3zps/Zs0w9+5OsOHMgZBPeTT44dMgzkEtyBHJOi2RQ8yjo9SMUrZ8Wav0t+aP5eQAnWpMrf5e9yYQsQYKGxOiLhXfn4ncBfehA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764410267; c=relaxed/simple; bh=kjC2CRoj1ICKpdLHPJNJwc/suA2ZEqoPeuQugHkxNh8=; h=Date:Mime-Version:Message-ID:Subject:From:To:Cc:Content-Type; b=SoOslbCh7snILi00suKkTxB0yzs2OixfNe4YsJ2tEUH0R77syMxmfB+op8QnFmLKHGoKpxeSk3904VmE0qjezoYcR7ka3my7NplVpEDqgD5xsrVjaOWYDZRwTdEuF3eObKyXTVIZRE48g8P81A9xBOplPEW4uI0Qcf32N+02W94= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=n1rlC5L1; arc=none smtp.client-ip=209.85.160.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="n1rlC5L1" Received: by mail-qt1-f202.google.com with SMTP id d75a77b69052e-4ee3296e984so42573651cf.0 for ; Sat, 29 Nov 2025 01:57:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1764410264; x=1765015064; darn=vger.kernel.org; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=RaHWpzLeVxoNOwQpfQ6Ps3eCZ1QEUD7c077XzEg2PUA=; b=n1rlC5L1Yuo5AgXpxUcRROP9hTO6oVTyIkZ+rfMJ/LnSarblaqqZxDbklhx5JUU+F1 R4mgjsQdKtaw4oOdcqL+iLWwXdDMxeeNjRxvMpAXZyZ/UeWoNuGbJ2EO4jhfi7ooJ4pb +IvGqVsNDjVs/DLkriuMTGVh0QbqVszO71hrspw47FBFjHpC2AIlsGf8/Ae3FBnCXx3Q BihRYed6HwAAkthUiypT/MSTVIt0Q9Zb1V+4Udfcv65niAWsO+imho6kHHM0RFXFrCN9 NAUGKtw3VdSRNiaDDLwhYcW5zZsHK632jvGZALXgNlx0T96FcoB+lhRL34NMq4/9HHyx /ipg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1764410264; x=1765015064; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=RaHWpzLeVxoNOwQpfQ6Ps3eCZ1QEUD7c077XzEg2PUA=; b=eUKNmVzJgEMJFhnwJereBiA4pytYDTWgUKl+m2scz95R/jLa5Yq16UxL5cgxtMayn4 8mQJlULrzxh06D0M7/1C6CKHbML7+EDOZw2t5JAEs77pn7Cd2es0k13p6OWxrdWDlZpj ngSGYLl62sQ5/QmYwqEgC2gEfTum079S8ghj0aamvHNV0jIyFoDoM6t8RBbT9NAwxROo SbVPJYNiXPa6R35MlE/ZA6SUQFZYEfewzck3jBuIgqcw2795BQgMcbZEKYVgaKEX9MZc yTUh6hHnYYrbqHsczjon53AyY98JP23p6aEEGPzRwxw/AhG7lg02MoQ3BC8HmvXeZI52 v1UA== X-Gm-Message-State: AOJu0YxgyS0BT1FmIUaBQJ6s+pYuL+ps7J3vYDticTL7akq5i+zaTGF7 IBgn60GkD9h8418ECcC917t5gVwGX2SVrhP06/slb7OapO9LUs2FKg1JjQefJ1Mji8E21sXSO5r tMlbVU90RjGFvtA== X-Google-Smtp-Source: AGHT+IE9e52KmeeUSI7PHmKs+tbnInUYzJa7BV62x1xd1VKYwrIlfZmBDNqCs536+iI8/0ZLLbiasYsjsJ41Ug== X-Received: from qtql8.prod.google.com ([2002:ac8:4a88:0:b0:4ed:3cc:ba24]) (user=edumazet job=prod-delivery.src-stubby-dispatcher) by 2002:ac8:7d0e:0:b0:4ed:70fd:1453 with SMTP id d75a77b69052e-4ee58935f18mr428211591cf.60.1764410264593; Sat, 29 Nov 2025 01:57:44 -0800 (PST) Date: Sat, 29 Nov 2025 09:57:40 +0000 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 X-Mailer: git-send-email 2.52.0.107.ga0afd4fd5b-goog Message-ID: <20251129095740.3338476-1-edumazet@google.com> Subject: [PATCH] time/timecounter: inline timecounter_cyc2time() From: Eric Dumazet To: John Stultz , Thomas Gleixner , Stephen Boyd , Jakub Kicinski , Paolo Abeni Cc: linux-kernel , netdev@vger.kernel.org, Eric Dumazet , Eric Dumazet , Kevin Yang , Willem de Bruijn , Neal Cardwell , Yuchung Cheng Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" New network transport protocols want NIC drivers to get hwtstamps of all incoming packets, and possibly all outgoing packets. Swift congestion control is used by good old TCP transport and is our primary need for timecounter_cyc2time(). This will be upstreamed soon. This means timecounter_cyc2time() can be called more than 100 million times per second on a busy server. Inlining timecounter_cyc2time() brings a 12 % improvement on a UDP receive stress test on a 100Gbit NIC. Note that FDO, LTO, PGO are unable to magically help for this case, presumably because NIC drivers are almost exclusively shipped as modules. Add an unlikely() around the cc_cyc2ns_backwards() case, even if FDO (when used) is able to take care of this optimization. Signed-off-by: Eric Dumazet Link: https://research.google/pubs/swift-delay-is-simple-and-effective-for-= congestion-control-in-the-datacenter/ Cc: Kevin Yang Cc: Willem de Bruijn Cc: Neal Cardwell Cc: Yuchung Cheng Reviewed-by: Kevin Yang Reviewed-by: Willem de Bruijn --- include/linux/timecounter.h | 33 +++++++++++++++++++++++++++++++-- kernel/time/timecounter.c | 35 ----------------------------------- 2 files changed, 31 insertions(+), 37 deletions(-) diff --git a/include/linux/timecounter.h b/include/linux/timecounter.h index dce03a5cafb7cfe61bd83304ef49f7721735da64..de21312ebed0f41946304f95f05= cc625ea7f2a68 100644 --- a/include/linux/timecounter.h +++ b/include/linux/timecounter.h @@ -115,6 +115,16 @@ extern void timecounter_init(struct timecounter *tc, */ extern u64 timecounter_read(struct timecounter *tc); =20 +/* + * This is like cyclecounter_cyc2ns(), but it is used for computing a + * time previous to the time stored in the cycle counter. + */ +static inline u64 cc_cyc2ns_backwards(const struct cyclecounter *cc, + u64 cycles, u64 frac) +{ + return ((cycles * cc->mult) - frac) >> cc->shift; +} + /** * timecounter_cyc2time - convert a cycle counter to same * time base as values returned by @@ -131,7 +141,26 @@ extern u64 timecounter_read(struct timecounter *tc); * * Returns: cycle counter converted to nanoseconds since the initial time = stamp */ -extern u64 timecounter_cyc2time(const struct timecounter *tc, - u64 cycle_tstamp); +static inline u64 timecounter_cyc2time(const struct timecounter *tc, + u64 cycle_tstamp) +{ + const struct cyclecounter *cc =3D tc->cc; + u64 delta =3D (cycle_tstamp - tc->cycle_last) & cc->mask; + u64 nsec =3D tc->nsec, frac =3D tc->frac; + + /* + * Instead of always treating cycle_tstamp as more recent + * than tc->cycle_last, detect when it is too far in the + * future and treat it as old time stamp instead. + */ + if (unlikely(delta > cc->mask / 2)) { + delta =3D (tc->cycle_last - cycle_tstamp) & cc->mask; + nsec -=3D cc_cyc2ns_backwards(cc, delta, frac); + } else { + nsec +=3D cyclecounter_cyc2ns(cc, delta, tc->mask, &frac); + } + + return nsec; +} =20 #endif diff --git a/kernel/time/timecounter.c b/kernel/time/timecounter.c index 3d2a354cfe1c1b205aa960a748a76f8dd94335e2..2e64dbb6302d71d8b092d92dda0= b71c99cdc053d 100644 --- a/kernel/time/timecounter.c +++ b/kernel/time/timecounter.c @@ -62,38 +62,3 @@ u64 timecounter_read(struct timecounter *tc) } EXPORT_SYMBOL_GPL(timecounter_read); =20 -/* - * This is like cyclecounter_cyc2ns(), but it is used for computing a - * time previous to the time stored in the cycle counter. - */ -static u64 cc_cyc2ns_backwards(const struct cyclecounter *cc, - u64 cycles, u64 mask, u64 frac) -{ - u64 ns =3D (u64) cycles; - - ns =3D ((ns * cc->mult) - frac) >> cc->shift; - - return ns; -} - -u64 timecounter_cyc2time(const struct timecounter *tc, - u64 cycle_tstamp) -{ - u64 delta =3D (cycle_tstamp - tc->cycle_last) & tc->cc->mask; - u64 nsec =3D tc->nsec, frac =3D tc->frac; - - /* - * Instead of always treating cycle_tstamp as more recent - * than tc->cycle_last, detect when it is too far in the - * future and treat it as old time stamp instead. - */ - if (delta > tc->cc->mask / 2) { - delta =3D (tc->cycle_last - cycle_tstamp) & tc->cc->mask; - nsec -=3D cc_cyc2ns_backwards(tc->cc, delta, tc->mask, frac); - } else { - nsec +=3D cyclecounter_cyc2ns(tc->cc, delta, tc->mask, &frac); - } - - return nsec; -} -EXPORT_SYMBOL_GPL(timecounter_cyc2time); --=20 2.52.0.107.ga0afd4fd5b-goog