From nobody Sun Apr 5 18:00:33 2026 Received: from mail-pl1-f201.google.com (mail-pl1-f201.google.com [209.85.214.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8AA3A385514 for ; Mon, 23 Mar 2026 23:48:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1774309719; cv=none; b=NLwUsCBC/PE4s5ZkS65QAFFKhGVXXf+BKrpQ4+hLDqapDYomgn6rtt+r3KY5ZaSzVbdcATQmStJItIZqROq/II49vmaCMu2P9zP1jfyzJjVzlo6u2lxAQ/9RT5mfDJ9WJO0LRGcaWDv/uiXj/jxpeHDo3X8CdPkGM32LtROJ7As= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1774309719; c=relaxed/simple; bh=ipi/hmuzw0I0dIR12Hqh883My688JUPKmeeIE1N+G0c=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=Rik3eVQ265L0HxMzLkCeKHbdoI00FY+n+sNT7SSumhS34Uif9UkTIg6s06WswIMPiADcuEGk5HwNw9yNzn7PEea7qsw7zHq7GrB5DBH4H6EJJG4ptp/WhoG332AhOhZriCk2unhVMKgKMfdUDROwJlm5Oqs6pzO55FTG7LkEeDg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--hramamurthy.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=r4kxlH6d; arc=none smtp.client-ip=209.85.214.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--hramamurthy.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="r4kxlH6d" Received: by mail-pl1-f201.google.com with SMTP id d9443c01a7336-2b051befbb8so41583645ad.2 for ; Mon, 23 Mar 2026 16:48:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1774309717; x=1774914517; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Jw/WcolBOl5E4xg9/GOqX1azVpZAcATB1CeYqZdKYX0=; b=r4kxlH6dQ8gNcXqiXUYBvhU2MzB0cmwNJcJpokv7gZI7EuPl8FUR/W8Z32aNNUiSd5 fSN8ioWCVYNZjqU2K0ahT2hsyfjf/LfoT9zU4z/HbIPmqOkPNhZurfdXOtJqr2/TeY80 uLbZoYAyMLH5nTD/3LSxZ7v4MxLz/M2BC2+q3+U2r9YrzpP5y/Eym3RVrx5DdcHRBnT6 VCf2RAygh1kzvtuUjpfkYTYevzhInCjEsgiDXYNOCF717jLKATANV0Ux8VWvrZ79DIUY cHljJmEMKCFqHH/7Rf7s88U2EY2c8YZuOTaQcIujfgVPn7YBR/vx/z9sIMxJokJxEsav 0gWQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1774309717; x=1774914517; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Jw/WcolBOl5E4xg9/GOqX1azVpZAcATB1CeYqZdKYX0=; b=UDW2ARmAGyj5Nej+327gNNV6xNz+/ZMwCvOeggeS4RQMI85tO08QyMb6MtMpxzxs7F yZ1FLhfDiWqw+OeAL+tNCH+eddv0LbLPWx8upCNozusVQcNiQ2S1oZjUsUMdNKQjne/R NFv59lQJTKPbwGHE1l/csqnTwRwB9a6nji8Lj/6NbTO1qP6ccwTWkhM3JoLMsqTeyI9e WQegaXeLTp1gSUCojgMkZe9WWlHh0VCt5jVWvMT0ZGLSXhUk8W8q/y694bszix6CSCTs rT2LjCMk/12EUg9PzblSv4r8Zx9urJlWz4hT7VRzyUl7eolA0vxJihaIodXBaeEKoe5y 9UzQ== X-Forwarded-Encrypted: i=1; AJvYcCWblyQQ9uyWIb5fEdCmJtI1uOOVIm+maFMpPjXprb6TCmajkiExT9uSrQIlereTkvcsChkAIF40xKfPk24=@vger.kernel.org X-Gm-Message-State: AOJu0YwTLcvcndUTtEPNEOSLFdiLNt6jPW2I4K8/mdOb6B/pnBgyih0y /iqBXva9i+M+U1j1t4DNCopeXyJT1c6RrYYTYbcJP8J6ZqxjFt75NzYDY4M4LIZoEgSXxBcXs5u eZg/wTTQcTaJp23BfaeE+0LNbNQ== X-Received: from plgc17.prod.google.com ([2002:a17:902:d491:b0:2ae:cd48:3a6f]) (user=hramamurthy job=prod-delivery.src-stubby-dispatcher) by 2002:a17:902:dac7:b0:2b0:7224:a4f3 with SMTP id d9443c01a7336-2b0827eefc7mr135145895ad.48.1774309716791; Mon, 23 Mar 2026 16:48:36 -0700 (PDT) Date: Mon, 23 Mar 2026 23:48:29 +0000 In-Reply-To: <20260323234829.3185051-1-hramamurthy@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260323234829.3185051-1-hramamurthy@google.com> X-Mailer: git-send-email 2.53.0.983.g0bb29b3bc5-goog Message-ID: <20260323234829.3185051-4-hramamurthy@google.com> Subject: [PATCH net-next 3/3] gve: implement PTP gettimex64 From: Harshitha Ramamurthy To: netdev@vger.kernel.org Cc: joshwash@google.com, hramamurthy@google.com, andrew+netdev@lunn.ch, davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, richardcochran@gmail.com, willemb@google.com, nktgrg@google.com, jfraker@google.com, ziweixiao@google.com, maolson@google.com, thostet@google.com, jordanrhee@google.com, jefrogers@google.com, alok.a.tiwari@oracle.com, linux-kernel@vger.kernel.org, Kevin Yang , Naman Gulati Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Jordan Rhee Enable chrony and phc2sys to synchronize system clock to NIC clock. The system cycle counters are sampled by the device to minimize the uncertainty window. If the system times are sampled in the host, the delta between pre and post readings is 100us or more due to AQ command latency. The system times returned by the device have a delta of ~1us, which enables significantly more accurate clock synchronization. Reviewed-by: Willem de Bruijn Reviewed-by: Kevin Yang Reviewed-by: Naman Gulati Signed-off-by: Jordan Rhee Signed-off-by: Harshitha Ramamurthy --- drivers/net/ethernet/google/gve/gve_adminq.h | 4 +- drivers/net/ethernet/google/gve/gve_ptp.c | 189 ++++++++++++++++++- 2 files changed, 184 insertions(+), 9 deletions(-) diff --git a/drivers/net/ethernet/google/gve/gve_adminq.h b/drivers/net/eth= ernet/google/gve/gve_adminq.h index 22a74b6aa17e..e6dcf6da9091 100644 --- a/drivers/net/ethernet/google/gve/gve_adminq.h +++ b/drivers/net/ethernet/google/gve/gve_adminq.h @@ -411,8 +411,8 @@ static_assert(sizeof(struct gve_adminq_report_nic_ts) = =3D=3D 16); =20 struct gve_nic_ts_report { __be64 nic_timestamp; /* NIC clock in nanoseconds */ - __be64 reserved1; - __be64 reserved2; + __be64 pre_cycles; /* System cycle counter before NIC clock read */ + __be64 post_cycles; /* System cycle counter after NIC clock read */ __be64 reserved3; __be64 reserved4; }; diff --git a/drivers/net/ethernet/google/gve/gve_ptp.c b/drivers/net/ethern= et/google/gve/gve_ptp.c index 140b8fbce4f4..889f0120e1dd 100644 --- a/drivers/net/ethernet/google/gve/gve_ptp.c +++ b/drivers/net/ethernet/google/gve/gve_ptp.c @@ -10,28 +10,203 @@ /* Interval to schedule a nic timestamp calibration, 250ms. */ #define GVE_NIC_TS_SYNC_INTERVAL_MS 250 =20 +/* + * Stores cycle counter samples in get_cycles() units from a + * sandwiched NIC clock read + */ +struct gve_sysclock_sample { + /* Cycle counter from NIC before clock read */ + u64 nic_pre_cycles; + /* Cycle counter from NIC after clock read */ + u64 nic_post_cycles; + /* Cycle counter from host before issuing AQ command */ + cycles_t host_pre_cycles; + /* Cycle counter from host after AQ command returns */ + cycles_t host_post_cycles; +}; + +/* + * Read NIC clock by issuing the AQ command. The command is subject to + * rate limiting and may need to be retried. Requires nic_ts_read_lock + * to be held. + */ +static int gve_adminq_read_timestamp(struct gve_priv *priv, + cycles_t *pre_cycles, + cycles_t *post_cycles) +{ + unsigned long delay_us =3D 1000; + int retry_count =3D 0; + int err; + + lockdep_assert_held(&priv->nic_ts_read_lock); + + do { + *pre_cycles =3D get_cycles(); + err =3D gve_adminq_report_nic_ts(priv, priv->nic_ts_report_bus); + + /* Ensure cycle counter is sampled after AdminQ cmd returns */ + rmb(); + *post_cycles =3D get_cycles(); + if (likely(err !=3D -EAGAIN)) + return err; + + fsleep(delay_us); + + /* Exponential backoff */ + delay_us *=3D 2; + retry_count++; + } while (retry_count < 5); + + return -ETIMEDOUT; +} + /* Read the nic timestamp from hardware via the admin queue. */ -static int gve_clock_nic_ts_read(struct gve_priv *priv, u64 *nic_raw) +static int gve_clock_nic_ts_read(struct gve_priv *priv, u64 *nic_raw, + struct gve_sysclock_sample *sysclock) { + cycles_t host_pre_cycles, host_post_cycles; + struct gve_nic_ts_report *ts_report; int err; =20 mutex_lock(&priv->nic_ts_read_lock); - err =3D gve_adminq_report_nic_ts(priv, priv->nic_ts_report_bus); - if (err) + err =3D gve_adminq_read_timestamp(priv, &host_pre_cycles, + &host_post_cycles); + if (err) { + dev_err_ratelimited(&priv->pdev->dev, + "AdminQ timestamp read failed: %d\n", err); goto out; + } =20 - *nic_raw =3D be64_to_cpu(priv->nic_ts_report->nic_timestamp); + ts_report =3D priv->nic_ts_report; + *nic_raw =3D be64_to_cpu(ts_report->nic_timestamp); + + if (sysclock) { + sysclock->nic_pre_cycles =3D be64_to_cpu(ts_report->pre_cycles); + sysclock->nic_post_cycles =3D be64_to_cpu(ts_report->post_cycles); + sysclock->host_pre_cycles =3D host_pre_cycles; + sysclock->host_post_cycles =3D host_post_cycles; + } =20 out: mutex_unlock(&priv->nic_ts_read_lock); return err; } =20 +struct gve_cycles_to_clock_callback_ctx { + u64 cycles; +}; + +static int gve_cycles_to_clock_fn(ktime_t *device_time, + struct system_counterval_t *system_counterval, + void *ctx) +{ + struct gve_cycles_to_clock_callback_ctx *context =3D ctx; + + *device_time =3D 0; + + system_counterval->cycles =3D context->cycles; + system_counterval->use_nsecs =3D false; + + if (IS_ENABLED(CONFIG_X86)) + system_counterval->cs_id =3D CSID_X86_TSC; + else if (IS_ENABLED(CONFIG_ARM64)) + system_counterval->cs_id =3D CSID_ARM_ARCH_COUNTER; + else + return -EOPNOTSUPP; + + return 0; +} + +/* + * Convert a raw cycle count (e.g. from get_cycles()) to the system clock + * type specified by clockid. The system_time_snapshot must be taken before + * the cycle counter is sampled. + */ +static int gve_cycles_to_timespec64(struct gve_priv *priv, clockid_t clock= id, + struct system_time_snapshot *snap, + u64 cycles, struct timespec64 *ts) +{ + struct gve_cycles_to_clock_callback_ctx ctx =3D {0}; + struct system_device_crosststamp xtstamp; + int err; + + ctx.cycles =3D cycles; + err =3D get_device_system_crosststamp(gve_cycles_to_clock_fn, &ctx, snap, + &xtstamp); + if (err) { + dev_err_ratelimited(&priv->pdev->dev, + "get_device_system_crosststamp() failed to convert %lld cycles to = system time: %d\n", + cycles, + err); + return err; + } + + switch (clockid) { + case CLOCK_REALTIME: + *ts =3D ktime_to_timespec64(xtstamp.sys_realtime); + break; + case CLOCK_MONOTONIC_RAW: + *ts =3D ktime_to_timespec64(xtstamp.sys_monoraw); + break; + default: + dev_err_ratelimited(&priv->pdev->dev, + "Cycle count conversion to clockid %d not supported\n", + clockid); + return -EOPNOTSUPP; + } + + return 0; +} + static int gve_ptp_gettimex64(struct ptp_clock_info *info, struct timespec64 *ts, struct ptp_system_timestamp *sts) { - return -EOPNOTSUPP; + struct gve_ptp *ptp =3D container_of(info, struct gve_ptp, info); + struct gve_sysclock_sample sysclock =3D {0}; + struct gve_priv *priv =3D ptp->priv; + struct system_time_snapshot snap; + u64 nic_ts; + int err; + + /* Take system clock snapshot before sampling cycle counters */ + if (sts) + ktime_get_snapshot(&snap); + + err =3D gve_clock_nic_ts_read(priv, &nic_ts, &sysclock); + if (err) + return err; + + if (sts) { + /* Reject samples with out of order system clock values */ + if (!(sysclock.host_pre_cycles <=3D sysclock.nic_pre_cycles && + sysclock.nic_pre_cycles <=3D sysclock.nic_post_cycles && + sysclock.nic_post_cycles <=3D sysclock.host_post_cycles)) { + dev_err_ratelimited(&priv->pdev->dev, + "AdminQ system clock cycle counts out of order. Expecting %llu <= =3D %llu <=3D %llu <=3D %llu\n", + sysclock.host_pre_cycles, + sysclock.nic_pre_cycles, + sysclock.nic_post_cycles, + sysclock.host_post_cycles); + return -EBADMSG; + } + + err =3D gve_cycles_to_timespec64(priv, sts->clockid, &snap, + sysclock.nic_pre_cycles, + &sts->pre_ts); + if (err) + return err; + + err =3D gve_cycles_to_timespec64(priv, sts->clockid, &snap, + sysclock.nic_post_cycles, + &sts->post_ts); + if (err) + return err; + } + + *ts =3D ns_to_timespec64(nic_ts); + + return 0; } =20 static int gve_ptp_settime64(struct ptp_clock_info *info, @@ -50,7 +225,7 @@ static long gve_ptp_do_aux_work(struct ptp_clock_info *i= nfo) if (gve_get_reset_in_progress(priv) || !gve_get_admin_queue_ok(priv)) goto out; =20 - err =3D gve_clock_nic_ts_read(priv, &nic_raw); + err =3D gve_clock_nic_ts_read(priv, &nic_raw, NULL); if (err) { dev_err_ratelimited(&priv->pdev->dev, "%s read err %d\n", __func__, err); @@ -132,7 +307,7 @@ int gve_init_clock(struct gve_priv *priv) goto release_ptp; } mutex_init(&priv->nic_ts_read_lock); - err =3D gve_clock_nic_ts_read(priv, &nic_raw); + err =3D gve_clock_nic_ts_read(priv, &nic_raw, NULL); if (err) { dev_err(&priv->pdev->dev, "failed to read NIC clock %d\n", err); goto release_nic_ts_report; --=20 2.53.0.851.ga537e3e6e9-goog