From nobody Tue Nov 26 07:22:16 2024 Received: from mail-pl1-f172.google.com (mail-pl1-f172.google.com [209.85.214.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 90B2D2E401 for ; Mon, 21 Oct 2024 01:53:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729475633; cv=none; b=bZ0wDmDedK3tf0JYodXXDcHHkFxP6ftl2tS0XdIasgHvSFc16CoQ7lfGFuEcoZWutolfxhDqXLoLAt0EzAK+adux5gjc/RB/fFPUEQv8Ql7uHft63fG2eTNs1jB66UjSih5EloyeYoxPozL0sdndxYPRwVMNdnJc1UQefbUO950= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729475633; c=relaxed/simple; bh=VE4qm8jApOLVerU00KCWXCwt/67d+CL4QjGzcJExsOA=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=QrzM/re6gp7Kbj1vucKSMpzIbXkVJFQbOR1V3a+fagjTuFYqEa3lpedQFMdQ1klXHDHILpI5yyX/KyARiQ5JTLXG3G/+YtL3NSUIOrReKOtZhcZBAHGALDKps/u8NqTsQCfDStBqfHfEM3d1OlsnoPEfAlBXqdpiLBo7wvJoQNo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=fastly.com; spf=pass smtp.mailfrom=fastly.com; dkim=pass (1024-bit key) header.d=fastly.com header.i=@fastly.com header.b=BbCVy6X3; arc=none smtp.client-ip=209.85.214.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=fastly.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=fastly.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=fastly.com header.i=@fastly.com header.b="BbCVy6X3" Received: by mail-pl1-f172.google.com with SMTP id d9443c01a7336-20cbca51687so40027135ad.1 for ; Sun, 20 Oct 2024 18:53:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fastly.com; s=google; t=1729475631; x=1730080431; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=AfpWC01kTavxQZPrDblAx1MDT+scLEsoUAk88HI2mLY=; b=BbCVy6X3EKt5jkfPv5QHE8O0pIVG9Uq7APlMayjzsAbeaLZnZ6+8U/3Nzmgy6NHyW8 /LUNkv4036NJYFQEyG8y734hZMhWbU8Vr1yndJFzFad7TShSBuJMQxh/qd3TRtAWLwaf xDrgF0eJefo8cgZZ0/Kfv5/Hb/OLUN8SCagDo= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729475631; x=1730080431; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=AfpWC01kTavxQZPrDblAx1MDT+scLEsoUAk88HI2mLY=; b=gxyhJ8Zmzj124CWUlfjzhS4vm7jPQVLHTuYKZ4So4CxLp+S6cRvQUXBOrtDvSTo5UR I1bamacYUrM19zxrYDrlpEC8jcN1dePGCnFu5V32Z5IhUjPSXJ9g6hJu5TPYRmjfhzDZ FdWCRREmQrhR6YA+m9I0T/o8z6XRgjJT4ubBjJR/eZgBJS5aRAU9zjjEnIvaDOuw1ghb u4Za/G/ojUelftDaau8Xjf+kOqf1jqjVjn31j9fTmsFePn2yJdai4f2TBctn01MUoVBF CvjAa3+O5FlWuHXQzu7vWG6vQGln1Z3QnZ5QnlFeB0JdbPpT7IgdOZR09l5kDRSWBGnr KgrQ== X-Forwarded-Encrypted: i=1; AJvYcCVM41LE1N/GbeKDFTcPfNMGf7BgDOEkg8dV+BZQPBBkV/puLvUgHg8thbfiZcWIwLUvNX5etm1XuAeuYrc=@vger.kernel.org X-Gm-Message-State: AOJu0YxhXBsxd60Jh+17G/ZAJ2KMGYJAbm9qRBD+Tsr/qZc9U556jAzV MEv/j5IDnyQsla24aPYcNyrcetZJKSL9cKEIgt6+b8gQj59nI2rd9s3zGOUKQGU= X-Google-Smtp-Source: AGHT+IGjTJnjyjVlBYRXVayLwuiOfn+qQDvfmpxxzMcQOZPyIDPOAhsGL2RHj1rLhlA3CwruLaZYlQ== X-Received: by 2002:a17:902:cec1:b0:1fb:57e7:5bb4 with SMTP id d9443c01a7336-20e5a9061d1mr121980015ad.37.1729475630765; Sun, 20 Oct 2024 18:53:50 -0700 (PDT) Received: from localhost.localdomain ([2620:11a:c019:0:65e:3115:2f58:c5fd]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-20e7eee650bsm15859985ad.34.2024.10.20.18.53.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 20 Oct 2024 18:53:50 -0700 (PDT) From: Joe Damato To: netdev@vger.kernel.org Cc: namangulati@google.com, edumazet@google.com, amritha.nambiar@intel.com, sridhar.samudrala@intel.com, sdf@fomichev.me, peter@typeblog.net, m2shafiei@uwaterloo.ca, bjorn@rivosinc.com, hch@infradead.org, willy@infradead.org, willemdebruijn.kernel@gmail.com, skhawaja@google.com, kuba@kernel.org, Martin Karsten , Joe Damato , "David S. Miller" , Paolo Abeni , Donald Hunter , Jesper Dangaard Brouer , Mina Almasry , Xuan Zhuo , Jiri Pirko , Sebastian Andrzej Siewior , Lorenzo Bianconi , Alexander Lobakin , Johannes Berg , Daniel Jurgens , linux-kernel@vger.kernel.org (open list) Subject: [PATCH net-next v2 1/6] net: Add napi_struct parameter irq_suspend_timeout Date: Mon, 21 Oct 2024 01:52:56 +0000 Message-Id: <20241021015311.95468-2-jdamato@fastly.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20241021015311.95468-1-jdamato@fastly.com> References: <20241021015311.95468-1-jdamato@fastly.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Martin Karsten Add a per-NAPI IRQ suspension parameter, which can be get/set with netdev-genl. This patch doesn't change any behavior but prepares the code for other changes in the following commits which use irq_suspend_timeout as a timeout for IRQ suspension. Signed-off-by: Martin Karsten Co-developed-by: Joe Damato Signed-off-by: Joe Damato Tested-by: Joe Damato Tested-by: Martin Karsten Acked-by: Stanislav Fomichev --- v1 -> v2: - Rewrote this patch to make irq_suspend_timeout per-NAPI via netdev-genl. rfc -> v1: - Removed napi.rst documentation from this patch; added to patch 6. Documentation/netlink/specs/netdev.yaml | 7 +++++++ include/linux/netdevice.h | 2 ++ include/uapi/linux/netdev.h | 1 + net/core/dev.c | 2 ++ net/core/dev.h | 25 +++++++++++++++++++++++++ net/core/netdev-genl-gen.c | 5 +++-- net/core/netdev-genl.c | 12 ++++++++++++ tools/include/uapi/linux/netdev.h | 1 + 8 files changed, 53 insertions(+), 2 deletions(-) diff --git a/Documentation/netlink/specs/netdev.yaml b/Documentation/netlin= k/specs/netdev.yaml index f9cb97d6106c..cbb544bd6c84 100644 --- a/Documentation/netlink/specs/netdev.yaml +++ b/Documentation/netlink/specs/netdev.yaml @@ -263,6 +263,11 @@ attribute-sets: the end of a NAPI cycle. This may add receive latency in exch= ange for reducing the number of frames processed by the network st= ack. type: uint + - + name: irq-suspend-timeout + doc: The timeout, in nanoseconds, of how long to suspend irq + processing, if event polling finds events + type: uint - name: queue attributes: @@ -653,6 +658,7 @@ operations: - pid - defer-hard-irqs - gro-flush-timeout + - irq-suspend-timeout dump: request: attributes: @@ -704,6 +710,7 @@ operations: - id - defer-hard-irqs - gro-flush-timeout + - irq-suspend-timeout =20 kernel-family: headers: [ "linux/list.h"] diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h index 8feaca12655e..9c3be8455707 100644 --- a/include/linux/netdevice.h +++ b/include/linux/netdevice.h @@ -347,6 +347,7 @@ struct gro_list { */ struct napi_config { u64 gro_flush_timeout; + u64 irq_suspend_timeout; u32 defer_hard_irqs; unsigned int napi_id; }; @@ -383,6 +384,7 @@ struct napi_struct { struct hrtimer timer; struct task_struct *thread; unsigned long gro_flush_timeout; + unsigned long irq_suspend_timeout; u32 defer_hard_irqs; /* control-path-only fields follow */ struct list_head dev_list; diff --git a/include/uapi/linux/netdev.h b/include/uapi/linux/netdev.h index e3ebb49f60d2..e4be227d3ad6 100644 --- a/include/uapi/linux/netdev.h +++ b/include/uapi/linux/netdev.h @@ -124,6 +124,7 @@ enum { NETDEV_A_NAPI_PID, NETDEV_A_NAPI_DEFER_HARD_IRQS, NETDEV_A_NAPI_GRO_FLUSH_TIMEOUT, + NETDEV_A_NAPI_IRQ_SUSPEND_TIMEOUT, =20 __NETDEV_A_NAPI_MAX, NETDEV_A_NAPI_MAX =3D (__NETDEV_A_NAPI_MAX - 1) diff --git a/net/core/dev.c b/net/core/dev.c index c682173a7642..68d0b16c2f2c 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -6662,6 +6662,7 @@ static void napi_restore_config(struct napi_struct *n) { n->defer_hard_irqs =3D n->config->defer_hard_irqs; n->gro_flush_timeout =3D n->config->gro_flush_timeout; + n->irq_suspend_timeout =3D n->config->irq_suspend_timeout; /* a NAPI ID might be stored in the config, if so use it. if not, use * napi_hash_add to generate one for us. It will be saved to the config * in napi_disable. @@ -6676,6 +6677,7 @@ static void napi_save_config(struct napi_struct *n) { n->config->defer_hard_irqs =3D n->defer_hard_irqs; n->config->gro_flush_timeout =3D n->gro_flush_timeout; + n->config->irq_suspend_timeout =3D n->irq_suspend_timeout; n->config->napi_id =3D n->napi_id; napi_hash_del(n); } diff --git a/net/core/dev.h b/net/core/dev.h index 7881bced70a9..d043dee25a68 100644 --- a/net/core/dev.h +++ b/net/core/dev.h @@ -236,6 +236,31 @@ static inline void netdev_set_gro_flush_timeout(struct= net_device *netdev, netdev->napi_config[i].gro_flush_timeout =3D timeout; } =20 +/** + * napi_get_irq_suspend_timeout - get the irq_suspend_timeout + * @n: napi struct to get the irq_suspend_timeout from + * + * Return: the per-NAPI value of the irq_suspend_timeout field. + */ +static inline unsigned long +napi_get_irq_suspend_timeout(const struct napi_struct *n) +{ + return READ_ONCE(n->irq_suspend_timeout); +} + +/** + * napi_set_irq_suspend_timeout - set the irq_suspend_timeout for a napi + * @n: napi struct to set the irq_suspend_timeout + * @timeout: timeout value to set + * + * napi_set_irq_suspend_timeout sets the per-NAPI irq_suspend_timeout + */ +static inline void napi_set_irq_suspend_timeout(struct napi_struct *n, + unsigned long timeout) +{ + WRITE_ONCE(n->irq_suspend_timeout, timeout); +} + int rps_cpumask_housekeeping(struct cpumask *mask); =20 #if defined(CONFIG_DEBUG_NET) && defined(CONFIG_BPF_SYSCALL) diff --git a/net/core/netdev-genl-gen.c b/net/core/netdev-genl-gen.c index 21de7e10be16..a89cbd8d87c3 100644 --- a/net/core/netdev-genl-gen.c +++ b/net/core/netdev-genl-gen.c @@ -92,10 +92,11 @@ static const struct nla_policy netdev_bind_rx_nl_policy= [NETDEV_A_DMABUF_FD + 1] }; =20 /* NETDEV_CMD_NAPI_SET - do */ -static const struct nla_policy netdev_napi_set_nl_policy[NETDEV_A_NAPI_GRO= _FLUSH_TIMEOUT + 1] =3D { +static const struct nla_policy netdev_napi_set_nl_policy[NETDEV_A_NAPI_IRQ= _SUSPEND_TIMEOUT + 1] =3D { [NETDEV_A_NAPI_ID] =3D { .type =3D NLA_U32, }, [NETDEV_A_NAPI_DEFER_HARD_IRQS] =3D NLA_POLICY_FULL_RANGE(NLA_U32, &netde= v_a_napi_defer_hard_irqs_range), [NETDEV_A_NAPI_GRO_FLUSH_TIMEOUT] =3D { .type =3D NLA_UINT, }, + [NETDEV_A_NAPI_IRQ_SUSPEND_TIMEOUT] =3D { .type =3D NLA_UINT, }, }; =20 /* Ops table for netdev */ @@ -186,7 +187,7 @@ static const struct genl_split_ops netdev_nl_ops[] =3D { .cmd =3D NETDEV_CMD_NAPI_SET, .doit =3D netdev_nl_napi_set_doit, .policy =3D netdev_napi_set_nl_policy, - .maxattr =3D NETDEV_A_NAPI_GRO_FLUSH_TIMEOUT, + .maxattr =3D NETDEV_A_NAPI_IRQ_SUSPEND_TIMEOUT, .flags =3D GENL_ADMIN_PERM | GENL_CMD_CAP_DO, }, }; diff --git a/net/core/netdev-genl.c b/net/core/netdev-genl.c index b49c3b4e5fbe..765ce7c9d73b 100644 --- a/net/core/netdev-genl.c +++ b/net/core/netdev-genl.c @@ -161,6 +161,7 @@ static int netdev_nl_napi_fill_one(struct sk_buff *rsp, struct napi_struct *napi, const struct genl_info *info) { + unsigned long irq_suspend_timeout; unsigned long gro_flush_timeout; u32 napi_defer_hard_irqs; void *hdr; @@ -196,6 +197,11 @@ netdev_nl_napi_fill_one(struct sk_buff *rsp, struct na= pi_struct *napi, napi_defer_hard_irqs)) goto nla_put_failure; =20 + irq_suspend_timeout =3D napi_get_irq_suspend_timeout(napi); + if (nla_put_uint(rsp, NETDEV_A_NAPI_IRQ_SUSPEND_TIMEOUT, + irq_suspend_timeout)) + goto nla_put_failure; + gro_flush_timeout =3D napi_get_gro_flush_timeout(napi); if (nla_put_uint(rsp, NETDEV_A_NAPI_GRO_FLUSH_TIMEOUT, gro_flush_timeout)) @@ -306,6 +312,7 @@ int netdev_nl_napi_get_dumpit(struct sk_buff *skb, stru= ct netlink_callback *cb) static int netdev_nl_napi_set_config(struct napi_struct *napi, struct genl_info *info) { + u64 irq_suspend_timeout =3D 0; u64 gro_flush_timeout =3D 0; u32 defer =3D 0; =20 @@ -314,6 +321,11 @@ netdev_nl_napi_set_config(struct napi_struct *napi, st= ruct genl_info *info) napi_set_defer_hard_irqs(napi, defer); } =20 + if (info->attrs[NETDEV_A_NAPI_IRQ_SUSPEND_TIMEOUT]) { + irq_suspend_timeout =3D nla_get_uint(info->attrs[NETDEV_A_NAPI_IRQ_SUSPE= ND_TIMEOUT]); + napi_set_irq_suspend_timeout(napi, irq_suspend_timeout); + } + if (info->attrs[NETDEV_A_NAPI_GRO_FLUSH_TIMEOUT]) { gro_flush_timeout =3D nla_get_uint(info->attrs[NETDEV_A_NAPI_GRO_FLUSH_T= IMEOUT]); napi_set_gro_flush_timeout(napi, gro_flush_timeout); diff --git a/tools/include/uapi/linux/netdev.h b/tools/include/uapi/linux/n= etdev.h index e3ebb49f60d2..e4be227d3ad6 100644 --- a/tools/include/uapi/linux/netdev.h +++ b/tools/include/uapi/linux/netdev.h @@ -124,6 +124,7 @@ enum { NETDEV_A_NAPI_PID, NETDEV_A_NAPI_DEFER_HARD_IRQS, NETDEV_A_NAPI_GRO_FLUSH_TIMEOUT, + NETDEV_A_NAPI_IRQ_SUSPEND_TIMEOUT, =20 __NETDEV_A_NAPI_MAX, NETDEV_A_NAPI_MAX =3D (__NETDEV_A_NAPI_MAX - 1) --=20 2.25.1 From nobody Tue Nov 26 07:22:16 2024 Received: from mail-pl1-f172.google.com (mail-pl1-f172.google.com [209.85.214.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A42C93EA9A for ; Mon, 21 Oct 2024 01:53:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729475636; cv=none; b=qALCHyVm34v2M3ulNeJngAsA3ZqWXyjFLENXikJa5h2jMKgz8gVU1jh9K1K1rrmlrz0ezFJ+vgc0C/kDwzXpBP+HyLBzkVM9tAgYEywxxXwxpmAZOqAOY2yhwvltpI4FBb7ywjgsWVoIPK5eEq05FLbd0U7VkYYEM0p1cQXELUw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729475636; c=relaxed/simple; bh=j/xq+NkIfYdhz94qa/CP8OLzI/XZjv3RfYV1I/panlE=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=H/FCQhr2di83Plywnh6mD6wd6QrnvDjoSP9HcJ4JYNc5YGWQiuKVMVeZVDvjIhF0QduCTVWz0dKfh+YGU/eFxL2dDSUpB806+AWbpKe4ELeVsBW3a9gYVMznPxbB6XNlbjapTBcVT7iNgwmMKGAKknmdhbvIVAJnVQc7aSDiPDs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=fastly.com; spf=pass smtp.mailfrom=fastly.com; dkim=pass (1024-bit key) header.d=fastly.com header.i=@fastly.com header.b=lZ7Rwt1I; arc=none smtp.client-ip=209.85.214.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=fastly.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=fastly.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=fastly.com header.i=@fastly.com header.b="lZ7Rwt1I" Received: by mail-pl1-f172.google.com with SMTP id d9443c01a7336-20c77459558so33158265ad.0 for ; Sun, 20 Oct 2024 18:53:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fastly.com; s=google; t=1729475634; x=1730080434; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=UTXGBPwWb2UjgB4YRsP1U7o6ISppA/AhjO8WcV0ADKY=; b=lZ7Rwt1Ijr1N+kRYA6bsUYdYehqVJbTHzS1U9g6IC33+L5oDDi03q4w3XR/AUg4Mmz M6xTrSLhCRhkykHAavaZYBHYM3b/eUehrtup1kMswrPAf8LbP+ySDuYu/MxDXPltwqEb fLrEFUaEy8IjLdMcdemTtnxVxouL70tvGLgo4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729475634; x=1730080434; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=UTXGBPwWb2UjgB4YRsP1U7o6ISppA/AhjO8WcV0ADKY=; b=JBTGW9B0hMVpKEs0UrmtPm4V/TSmaxAg3gNOG6AJ6S1fvTcwgC9icRbm0yD4ueFrbe yBVCMPIdyeAjzlrTFK8e4Nykr+HJbc3aAKxEW8UsETHbmOn7EY4hx4OMhj0M6AT2M+Qz StEzJP+1RBRFu+3uSqtpmJqQzsW7cc9J/7XvMi7bohF1DxFKJglyzTVpL2wO/H6ytVZq GMZjRfJJkq3GfZQVs+YSfumBem+zOjXBUegHOub/LNBRK45Q7mGmKFusI1k/BU+cZNDn m8M+63jgdV1M3UpFgngFGlQ4BH12yPzvR9plb4kdiP6+BqrKLo2FDeXUSpK5+aqWAsjA hDPQ== X-Forwarded-Encrypted: i=1; AJvYcCWHVHpRmuO0GsVffTdR5rZYxpBM4Al3ZvlSm3moZUuFiFwj8bkTzwajpZEAovfGoSKlPvDa9b/ytztpjUU=@vger.kernel.org X-Gm-Message-State: AOJu0Yz6yeUGEiYk9cKyizoSi8UB+Bpi9zQcf5kj7idyX/wxUq6mDv3C xppHkkb6y1D75qd+mc0j0Raub08m4Nhphu0q9poQewFrkq3N3BHMPEyxL8Tshk0= X-Google-Smtp-Source: AGHT+IHwIA2NfupKP85DHpqMkbeTwP0qaJp0Gh7HUU0pAgnV6BG8mMO/V6AL9FsCCHPaR3ALMNsp4g== X-Received: by 2002:a17:902:da8f:b0:20c:c18f:c3a0 with SMTP id d9443c01a7336-20e5a728241mr140048925ad.7.1729475634077; Sun, 20 Oct 2024 18:53:54 -0700 (PDT) Received: from localhost.localdomain ([2620:11a:c019:0:65e:3115:2f58:c5fd]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-20e7eee650bsm15859985ad.34.2024.10.20.18.53.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 20 Oct 2024 18:53:53 -0700 (PDT) From: Joe Damato To: netdev@vger.kernel.org Cc: namangulati@google.com, edumazet@google.com, amritha.nambiar@intel.com, sridhar.samudrala@intel.com, sdf@fomichev.me, peter@typeblog.net, m2shafiei@uwaterloo.ca, bjorn@rivosinc.com, hch@infradead.org, willy@infradead.org, willemdebruijn.kernel@gmail.com, skhawaja@google.com, kuba@kernel.org, Martin Karsten , Joe Damato , "David S. Miller" , Paolo Abeni , Jiri Pirko , Sebastian Andrzej Siewior , Lorenzo Bianconi , Alexander Lobakin , linux-kernel@vger.kernel.org (open list) Subject: [PATCH net-next v2 2/6] net: Suspend softirq when prefer_busy_poll is set Date: Mon, 21 Oct 2024 01:52:57 +0000 Message-Id: <20241021015311.95468-3-jdamato@fastly.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20241021015311.95468-1-jdamato@fastly.com> References: <20241021015311.95468-1-jdamato@fastly.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Martin Karsten When NAPI_F_PREFER_BUSY_POLL is set during busy_poll_stop and the irq_suspend_timeout sysfs is nonzero, this timeout is used to defer softirq scheduling, potentially longer than gro_flush_timeout. This can be used to effectively suspend softirq processing during the time it takes for an application to process data and return to the next busy-poll. The call to napi->poll in busy_poll_stop might lead to an invocation of napi_complete_done, but the prefer-busy flag is still set at that time, so the same logic is used to defer softirq scheduling for irq_suspend_timeout. Signed-off-by: Martin Karsten Co-developed-by: Joe Damato Signed-off-by: Joe Damato Tested-by: Joe Damato Tested-by: Martin Karsten Acked-by: Stanislav Fomichev --- net/core/dev.c | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/net/core/dev.c b/net/core/dev.c index 68d0b16c2f2c..a388303b26b9 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -6235,7 +6235,12 @@ bool napi_complete_done(struct napi_struct *n, int w= ork_done) timeout =3D napi_get_gro_flush_timeout(n); n->defer_hard_irqs_count =3D napi_get_defer_hard_irqs(n); } - if (n->defer_hard_irqs_count > 0) { + if (napi_prefer_busy_poll(n)) { + timeout =3D napi_get_irq_suspend_timeout(n); + if (timeout) + ret =3D false; + } + if (ret && n->defer_hard_irqs_count > 0) { n->defer_hard_irqs_count--; timeout =3D napi_get_gro_flush_timeout(n); if (timeout) @@ -6371,9 +6376,13 @@ static void busy_poll_stop(struct napi_struct *napi,= void *have_poll_lock, bpf_net_ctx =3D bpf_net_ctx_set(&__bpf_net_ctx); =20 if (flags & NAPI_F_PREFER_BUSY_POLL) { - napi->defer_hard_irqs_count =3D napi_get_defer_hard_irqs(napi); - timeout =3D napi_get_gro_flush_timeout(napi); - if (napi->defer_hard_irqs_count && timeout) { + timeout =3D napi_get_irq_suspend_timeout(napi); + if (!timeout) { + napi->defer_hard_irqs_count =3D napi_get_defer_hard_irqs(napi); + if (napi->defer_hard_irqs_count) + timeout =3D napi_get_gro_flush_timeout(napi); + } + if (timeout) { hrtimer_start(&napi->timer, ns_to_ktime(timeout), HRTIMER_MODE_REL_PINN= ED); skip_schedule =3D true; } --=20 2.25.1 From nobody Tue Nov 26 07:22:16 2024 Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4616B54BD4 for ; Mon, 21 Oct 2024 01:53:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729475640; cv=none; b=u0LVFGe1v/YdauxxMDeSK32NNQbGdNCVyLTJtBUa07eUygLK/BZ18ExpsmhTM8PRcGK5RJEfD0dnwYb4Y6iAoewE3n5EPtMQaNq6+CdUT75nivAQQr2ux+1+BxsqbUAIu3MFTPayaZ6IkJUqRCX9oAL+vfW9spU13T8judu1PpA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729475640; c=relaxed/simple; bh=fNcofMdMGqSdkpD7cvRXzDTPFTdNqqwK2UlNXiWtTNU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=VDjksuNngLUi4BfjRLuj0mjlDnvEVWWQdmtJm4K/8xfP5cn14KEDiRmAeMkwHtmPDUuzQ3SM00/4d4nLCBse7eGvcvgDyxkpRDxSahwpvo1hzkY8y3i70TXxm3VRAATqXYXeIJHQlYDwAJV+MthcjL/MlQZ/C86B8rB7c1exPwQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=fastly.com; spf=pass smtp.mailfrom=fastly.com; dkim=pass (1024-bit key) header.d=fastly.com header.i=@fastly.com header.b=JvfTU+Ys; arc=none smtp.client-ip=209.85.214.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=fastly.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=fastly.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=fastly.com header.i=@fastly.com header.b="JvfTU+Ys" Received: by mail-pl1-f171.google.com with SMTP id d9443c01a7336-207115e3056so30921315ad.2 for ; Sun, 20 Oct 2024 18:53:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fastly.com; s=google; t=1729475637; x=1730080437; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=FePUDPVukA9o3l+UniZnQUedPYlrvLH3uHGO+zGSqIk=; b=JvfTU+YsliO/DYqIBVIOHGzeWWoFl/dssMDMKmIAz+j/9ez7j7kukXKPhYQb6Z2tWg BNnqC5nyhhegkGbg3lZTJ+lh9MxL53Ekyn4GQ4CDuirErr6Bqql9Hg62ALnzPCob1/kr SLPWOum86rtBZH1FRsoF+Wzsqklsn8OaGSCYs= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729475637; x=1730080437; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=FePUDPVukA9o3l+UniZnQUedPYlrvLH3uHGO+zGSqIk=; b=BWvo3kTGtrU3WB9GvmbRICFMJhuZdOpF9s8qx2OXNJyCi1lOW/I6ZQdD1evhZgWWJE b2xkHvmeYqAqoYKjH53LEivjKPPMln+IhqtYe4fn8cXkhwa48AqIcx+Tfg3JaRTh+ScH BIhiOlZeCqcpBj2N3PsQNEH9K9bUTesvIwnhK5nXE2dfjh+AtRKCnK7gBr9yINhkVRCE Xn48R2nrwg4siCqU7fVrlJuqztU0ynTrjEb+Fs0N64swCCUUqg/ph5tyym8nXL3mFIZq LWVKtMqXm/NZKRITM9wKgo18mVRJ2TrTbRUoUn3Nm7lrFX5spxMAJ8TYo2Y7dxMgrP0X yQTg== X-Forwarded-Encrypted: i=1; AJvYcCUqTRGOYeuUf9TAt99m79wL5Z/3g/kOr6XLfA5OO+zB859PNc1VoMIlscAi6JiB2AcPdaJjn5e81XSK7nQ=@vger.kernel.org X-Gm-Message-State: AOJu0YxTACxi6BihejWuNIODMK5m0sxWnoHXpE1ATPkXlLf4NNHqpZhP b8JESR52zxkSsN2356/3/ZwzFTNBFv4pwtXjnxSTeeKV3Cm1ahqhEUiNYYuDfls= X-Google-Smtp-Source: AGHT+IEhfBRZa4e9pXBSkhlDjOdj0kT4hP3ReRxh8NImFNH/yZPYPV6MnIfG85+rxL+mQSbyDRjkwg== X-Received: by 2002:a17:902:e74d:b0:20c:bb1c:d216 with SMTP id d9443c01a7336-20e5a773ebbmr154528285ad.21.1729475637501; Sun, 20 Oct 2024 18:53:57 -0700 (PDT) Received: from localhost.localdomain ([2620:11a:c019:0:65e:3115:2f58:c5fd]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-20e7eee650bsm15859985ad.34.2024.10.20.18.53.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 20 Oct 2024 18:53:57 -0700 (PDT) From: Joe Damato To: netdev@vger.kernel.org Cc: namangulati@google.com, edumazet@google.com, amritha.nambiar@intel.com, sridhar.samudrala@intel.com, sdf@fomichev.me, peter@typeblog.net, m2shafiei@uwaterloo.ca, bjorn@rivosinc.com, hch@infradead.org, willy@infradead.org, willemdebruijn.kernel@gmail.com, skhawaja@google.com, kuba@kernel.org, Martin Karsten , Joe Damato , "David S. Miller" , Paolo Abeni , Jiri Pirko , Sebastian Andrzej Siewior , Lorenzo Bianconi , Alexander Lobakin , linux-kernel@vger.kernel.org (open list) Subject: [PATCH net-next v2 3/6] net: Add control functions for irq suspension Date: Mon, 21 Oct 2024 01:52:58 +0000 Message-Id: <20241021015311.95468-4-jdamato@fastly.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20241021015311.95468-1-jdamato@fastly.com> References: <20241021015311.95468-1-jdamato@fastly.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Martin Karsten The napi_suspend_irqs routine bootstraps irq suspension by elongating the defer timeout to irq_suspend_timeout. The napi_resume_irqs routine effectively cancels irq suspension by forcing the napi to be scheduled immediately. Signed-off-by: Martin Karsten Co-developed-by: Joe Damato Signed-off-by: Joe Damato Tested-by: Joe Damato Tested-by: Martin Karsten Acked-by: Stanislav Fomichev --- v1 -> v2: - Added a comment to napi_resume_irqs. =20 include/net/busy_poll.h | 3 +++ net/core/dev.c | 39 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 42 insertions(+) diff --git a/include/net/busy_poll.h b/include/net/busy_poll.h index f03040baaefd..c858270141bc 100644 --- a/include/net/busy_poll.h +++ b/include/net/busy_poll.h @@ -52,6 +52,9 @@ void napi_busy_loop_rcu(unsigned int napi_id, bool (*loop_end)(void *, unsigned long), void *loop_end_arg, bool prefer_busy_poll, u16 budget); =20 +void napi_suspend_irqs(unsigned int napi_id); +void napi_resume_irqs(unsigned int napi_id); + #else /* CONFIG_NET_RX_BUSY_POLL */ static inline unsigned long net_busy_loop_on(void) { diff --git a/net/core/dev.c b/net/core/dev.c index a388303b26b9..143b3e690169 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -6512,6 +6512,45 @@ void napi_busy_loop(unsigned int napi_id, } EXPORT_SYMBOL(napi_busy_loop); =20 +void napi_suspend_irqs(unsigned int napi_id) +{ + struct napi_struct *napi; + + rcu_read_lock(); + napi =3D napi_by_id(napi_id); + if (napi) { + unsigned long timeout =3D napi_get_irq_suspend_timeout(napi); + + if (timeout) + hrtimer_start(&napi->timer, ns_to_ktime(timeout), + HRTIMER_MODE_REL_PINNED); + } + rcu_read_unlock(); +} +EXPORT_SYMBOL(napi_suspend_irqs); + +void napi_resume_irqs(unsigned int napi_id) +{ + struct napi_struct *napi; + + rcu_read_lock(); + napi =3D napi_by_id(napi_id); + if (napi) { + /* If irq_suspend_timeout is set to 0 between the call to + * napi_suspend_irqs and now, the original value still + * determines the safety timeout as intended and napi_watchdog + * will resume irq processing. + */ + if (napi_get_irq_suspend_timeout(napi)) { + local_bh_disable(); + napi_schedule(napi); + local_bh_enable(); + } + } + rcu_read_unlock(); +} +EXPORT_SYMBOL(napi_resume_irqs); + #endif /* CONFIG_NET_RX_BUSY_POLL */ =20 static void __napi_hash_add_with_id(struct napi_struct *napi, --=20 2.25.1 From nobody Tue Nov 26 07:22:16 2024 Received: from mail-pl1-f172.google.com (mail-pl1-f172.google.com [209.85.214.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 31E9379DC7 for ; Mon, 21 Oct 2024 01:54:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729475642; cv=none; b=A7Ubjpbz/wqJgyx/4EKzOwFnc9mHMJ4pYXgjEA4sc+poyMaOzJ1KUwE9pzp29ZaoQM48tQ52ht+rRDN9blW4w3V3hCV/DqZ0kBtRY76QQ4rj3rUAUfK1ph6LY/0YGGODk04AmQoZuLxju/Z+wEMHCII155cIkGZSvwfwKaZ1s8k= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729475642; c=relaxed/simple; bh=0e84nxtTo0ZoFDbU9IHdh0nF4DdHDes+nR9YAcbdfkw=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=E3pp+8WQ1Q3qku5Cn0waTy8DeOKVwZneYP5pKNILGrU4jQDqCoMN5BDb+oxTYwBw5jXl8IDPBrz5elOR1KtODqVonnV89WMLM3dDmKD6AB6aWlZFXr0p9vRf9/E8iaTADxwzcGYmucKxzZzA3zSgzp8Pb5nObjAB5dGFkFKqWIQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=fastly.com; spf=pass smtp.mailfrom=fastly.com; dkim=pass (1024-bit key) header.d=fastly.com header.i=@fastly.com header.b=lhiwI1As; arc=none smtp.client-ip=209.85.214.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=fastly.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=fastly.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=fastly.com header.i=@fastly.com header.b="lhiwI1As" Received: by mail-pl1-f172.google.com with SMTP id d9443c01a7336-20c693b68f5so41301525ad.1 for ; Sun, 20 Oct 2024 18:54:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fastly.com; s=google; t=1729475640; x=1730080440; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=iFrcV5k7ZDwjmKi04AgmgViZJXfV02gap8kxsUW7sc8=; b=lhiwI1Asx/53ccFpny/+Vd1zgltvJ9LGoMvnvxf9v83LiC7Ag6E+lygWgeGZYz3Fgu muEz7Dee79iUJH2WnT8fXQ2T+J3aOUCIT/L7gPeVgLg3YrswXh41gRxYlqPxOGF8n/oW 8goIPQl9fwlGnI7ne3jEsflH0kLuvUpyYoOR8= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729475640; x=1730080440; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=iFrcV5k7ZDwjmKi04AgmgViZJXfV02gap8kxsUW7sc8=; b=lgchm4rVTX0letEBR5LlC+aJlofJGnkoCkjPoc2DnIlDK2DQKcMZool80qjreWjnXY x4ZKOcdST+5cDy7xSFfozOHziKXSwW1tSnYKrdWKdlt6YElvApOxSFr+eAHRXYmPgFT+ adCEGBzmRzqWzWFOqFt1dtAU6XBj/2vCdqZw2ibJpZy6g1ixsVHTnFESPVikzUyIImzV WgF3CGetapKScCEfkjWOV/0vxesHhrWmMDOduOsQUEtnjEX+WrnY13LP2Q9AhFgA5qbV hqHxgixYVzPcnEhvskXq28d7kDcolg6LmN2MQzifotcVuR0Ps0Z7apS68ySx4fRxLmMA BmSg== X-Forwarded-Encrypted: i=1; AJvYcCUCuphZjWLfAnAuiw4tkDtSfNRYgaLYfr8EFY1vGMufLhCCoHWUqHHOsIq9LywIEkJJjEnpqmxGGEGXtQE=@vger.kernel.org X-Gm-Message-State: AOJu0YwQtiYh94gD5g0mFZkm0isYTR40S8t9HodSUAJD6hBUVO4etVoI tmWTn3FtA14tsQZZs5D9PRaZEtaiJ6IWsImi8f5vhu8/KAw98wXkSzL4Hkt8tB8= X-Google-Smtp-Source: AGHT+IHHA/oGS3p+tLHWaHAvFQTIjHcw3aME6fzgwVhWuS7b1NgLYLejs28l5AZoGl8AQkfb80AvWQ== X-Received: by 2002:a17:903:2308:b0:20b:6d71:4140 with SMTP id d9443c01a7336-20e5a8f5fabmr135068245ad.44.1729475640402; Sun, 20 Oct 2024 18:54:00 -0700 (PDT) Received: from localhost.localdomain ([2620:11a:c019:0:65e:3115:2f58:c5fd]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-20e7eee650bsm15859985ad.34.2024.10.20.18.53.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 20 Oct 2024 18:54:00 -0700 (PDT) From: Joe Damato To: netdev@vger.kernel.org Cc: namangulati@google.com, edumazet@google.com, amritha.nambiar@intel.com, sridhar.samudrala@intel.com, sdf@fomichev.me, peter@typeblog.net, m2shafiei@uwaterloo.ca, bjorn@rivosinc.com, hch@infradead.org, willy@infradead.org, willemdebruijn.kernel@gmail.com, skhawaja@google.com, kuba@kernel.org, Martin Karsten , Joe Damato , Alexander Viro , Christian Brauner , Jan Kara , linux-fsdevel@vger.kernel.org (open list:FILESYSTEMS (VFS and infrastructure)), linux-kernel@vger.kernel.org (open list) Subject: [PATCH net-next v2 4/6] eventpoll: Trigger napi_busy_loop, if prefer_busy_poll is set Date: Mon, 21 Oct 2024 01:52:59 +0000 Message-Id: <20241021015311.95468-5-jdamato@fastly.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20241021015311.95468-1-jdamato@fastly.com> References: <20241021015311.95468-1-jdamato@fastly.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Martin Karsten Setting prefer_busy_poll now leads to an effectively nonblocking iteration though napi_busy_loop, even when busy_poll_usecs is 0. Signed-off-by: Martin Karsten Co-developed-by: Joe Damato Signed-off-by: Joe Damato Tested-by: Joe Damato Tested-by: Martin Karsten Acked-by: Stanislav Fomichev --- v1 -> v2: - Rebased to apply now that commit b9ca079dd6b0 ("eventpoll: Annotate data-race of busy_poll_usecs") has been picked up from VFS. fs/eventpoll.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/fs/eventpoll.c b/fs/eventpoll.c index 1ae4542f0bd8..f9e0d9307dad 100644 --- a/fs/eventpoll.c +++ b/fs/eventpoll.c @@ -420,7 +420,9 @@ static bool busy_loop_ep_timeout(unsigned long start_ti= me, =20 static bool ep_busy_loop_on(struct eventpoll *ep) { - return !!READ_ONCE(ep->busy_poll_usecs) || net_busy_loop_on(); + return !!READ_ONCE(ep->busy_poll_usecs) || + READ_ONCE(ep->prefer_busy_poll) || + net_busy_loop_on(); } =20 static bool ep_busy_loop_end(void *p, unsigned long start_time) --=20 2.25.1 From nobody Tue Nov 26 07:22:16 2024 Received: from mail-pl1-f172.google.com (mail-pl1-f172.google.com [209.85.214.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 07E7481AB6 for ; Mon, 21 Oct 2024 01:54:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729475644; cv=none; b=VUPRgXpCu+jlRo74PvF5K6gcbXuOxC2Fw6aGn1IhYNtEbthM2aiAzZTLjaJ8cPSrn67BgHPupWuvDh3YvD/CMp9STRveJk0OMNpDWj2uMWi94S1R3amCEdHAMPXVkaTuT137zgLJdmJE/365H07fotK81Rkd+zcOSujQACg6bCY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729475644; c=relaxed/simple; bh=+v/H79GmiAG3naN0zeRg2nElJUiBqVhmyL2Hh1hlU1E=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=nMDL3gAIz4V/u+U0QDgyIQx9Rl5It8KYBnzYjFtTSA2xnJ8LW3YKnNfrfzfrN87yPsu/SyD1NvrjqQmJe1vGQ4teEdy4XRV07z002DDRstgvRBWQu0JEq6/j10UAnYm1CgdoFR4IlciI1kwrr31FkZBjDj3BJOCCq6GmmqMaxIo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=fastly.com; spf=pass smtp.mailfrom=fastly.com; dkim=pass (1024-bit key) header.d=fastly.com header.i=@fastly.com header.b=jPaPMNaz; arc=none smtp.client-ip=209.85.214.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=fastly.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=fastly.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=fastly.com header.i=@fastly.com header.b="jPaPMNaz" Received: by mail-pl1-f172.google.com with SMTP id d9443c01a7336-20c9978a221so45388445ad.1 for ; Sun, 20 Oct 2024 18:54:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fastly.com; s=google; t=1729475642; x=1730080442; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=JmUjBpdJ0f8V/MHrwl+AGewKk5aAt+i1MtpvNR8b1Y8=; b=jPaPMNazIG9wKFrhCiVuME+1DCpbm3buJdEvqMBu/yXUuXMFWHrj7IiEkhnaMV9KZJ m/m5dncBvKrF9U0CB2ZPEaBkp0Xf5w3ftOpoeYIiyM0UI2HBOIQtaAGg+2iSzFbdH23f Sug6HtzSIp23FZ2OWhTySPXE8KVAlx60d5GTo= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729475642; x=1730080442; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=JmUjBpdJ0f8V/MHrwl+AGewKk5aAt+i1MtpvNR8b1Y8=; b=nFzOczKsrQXhEuoyclqmsJitxaeM7YxJcCyxz4nEkkHHbV5p2dxqr+qAJt9zqKUoJ8 2cgt5h7SIGyyCFGXSyYO2fuGxv1i4S8V8i2D8dPnFucLIgw0jx+TCIChrkKZst6xGSKl a8Eue9Ae9tsb53FIuUMCGFtz1frNiklDTcFGYxTPOByEM5Ziu2Eho5ZioQn356ok8Aso aprYRsciKqCok45yBHON/cGTNyakYhNG5DLU3HjCxfPHvfLSPGusZRMl0pmdV8JwtUYk Qe7qiYM/G+mxBZlzW1nYW1dmaE0MaK+UVoxpszLeDMevqKjQ/XUIM8YGx4UFc605WeFs ZaDQ== X-Forwarded-Encrypted: i=1; AJvYcCVv3RAqojqkcTuqsNp/K4LLtkUyAqwjPNcv6Mz15PSWdlBBU1xdxUUlmsx5qbLXQHjtoQvH2DoctZiOERk=@vger.kernel.org X-Gm-Message-State: AOJu0Yz6iWgh28Q2vZXNX7Q245dtE04YXkojEjuNnqbAgcwhXu4/jw+9 LvBcWX6emeNto5yREMldmNWY++fBX9f7Oos6YPsz32NYK27NPFFGC2GBgDCldQY= X-Google-Smtp-Source: AGHT+IGVda21+4TkdnbM/4Q+uHbJcBEZbiTKQt97djp2s9YV3IPbRBsjbBBv8Rl+668P90LjyRMasw== X-Received: by 2002:a17:902:f906:b0:20b:8776:4906 with SMTP id d9443c01a7336-20e5a8ee770mr105564285ad.37.1729475642435; Sun, 20 Oct 2024 18:54:02 -0700 (PDT) Received: from localhost.localdomain ([2620:11a:c019:0:65e:3115:2f58:c5fd]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-20e7eee650bsm15859985ad.34.2024.10.20.18.54.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 20 Oct 2024 18:54:01 -0700 (PDT) From: Joe Damato To: netdev@vger.kernel.org Cc: namangulati@google.com, edumazet@google.com, amritha.nambiar@intel.com, sridhar.samudrala@intel.com, sdf@fomichev.me, peter@typeblog.net, m2shafiei@uwaterloo.ca, bjorn@rivosinc.com, hch@infradead.org, willy@infradead.org, willemdebruijn.kernel@gmail.com, skhawaja@google.com, kuba@kernel.org, Martin Karsten , Joe Damato , Alexander Viro , Christian Brauner , Jan Kara , linux-fsdevel@vger.kernel.org (open list:FILESYSTEMS (VFS and infrastructure)), linux-kernel@vger.kernel.org (open list) Subject: [PATCH net-next v2 5/6] eventpoll: Control irq suspension for prefer_busy_poll Date: Mon, 21 Oct 2024 01:53:00 +0000 Message-Id: <20241021015311.95468-6-jdamato@fastly.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20241021015311.95468-1-jdamato@fastly.com> References: <20241021015311.95468-1-jdamato@fastly.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Martin Karsten When events are reported to userland and prefer_busy_poll is set, irqs are temporarily suspended using napi_suspend_irqs. If no events are found and ep_poll would go to sleep, irq suspension is cancelled using napi_resume_irqs. Signed-off-by: Martin Karsten Co-developed-by: Joe Damato Signed-off-by: Joe Damato Tested-by: Joe Damato Tested-by: Martin Karsten Acked-by: Stanislav Fomichev --- rfc -> v1: - move irq resume code from ep_free to a helper which either resumes IRQs or does nothing if !defined(CONFIG_NET_RX_BUSY_POLL). fs/eventpoll.c | 31 ++++++++++++++++++++++++++++++- 1 file changed, 30 insertions(+), 1 deletion(-) diff --git a/fs/eventpoll.c b/fs/eventpoll.c index f9e0d9307dad..36a657594352 100644 --- a/fs/eventpoll.c +++ b/fs/eventpoll.c @@ -457,6 +457,8 @@ static bool ep_busy_loop(struct eventpoll *ep, int nonb= lock) * it back in when we have moved a socket with a valid NAPI * ID onto the ready list. */ + if (prefer_busy_poll) + napi_resume_irqs(napi_id); ep->napi_id =3D 0; return false; } @@ -540,6 +542,22 @@ static long ep_eventpoll_bp_ioctl(struct file *file, u= nsigned int cmd, } } =20 +static void ep_suspend_napi_irqs(struct eventpoll *ep) +{ + unsigned int napi_id =3D READ_ONCE(ep->napi_id); + + if (napi_id >=3D MIN_NAPI_ID && READ_ONCE(ep->prefer_busy_poll)) + napi_suspend_irqs(napi_id); +} + +static void ep_resume_napi_irqs(struct eventpoll *ep) +{ + unsigned int napi_id =3D READ_ONCE(ep->napi_id); + + if (napi_id >=3D MIN_NAPI_ID && READ_ONCE(ep->prefer_busy_poll)) + napi_resume_irqs(napi_id); +} + #else =20 static inline bool ep_busy_loop(struct eventpoll *ep, int nonblock) @@ -557,6 +575,14 @@ static long ep_eventpoll_bp_ioctl(struct file *file, u= nsigned int cmd, return -EOPNOTSUPP; } =20 +static void ep_suspend_napi_irqs(struct eventpoll *ep) +{ +} + +static void ep_resume_napi_irqs(struct eventpoll *ep) +{ +} + #endif /* CONFIG_NET_RX_BUSY_POLL */ =20 /* @@ -788,6 +814,7 @@ static bool ep_refcount_dec_and_test(struct eventpoll *= ep) =20 static void ep_free(struct eventpoll *ep) { + ep_resume_napi_irqs(ep); mutex_destroy(&ep->mtx); free_uid(ep->user); wakeup_source_unregister(ep->ws); @@ -2005,8 +2032,10 @@ static int ep_poll(struct eventpoll *ep, struct epol= l_event __user *events, * trying again in search of more luck. */ res =3D ep_send_events(ep, events, maxevents); - if (res) + if (res) { + ep_suspend_napi_irqs(ep); return res; + } } =20 if (timed_out) --=20 2.25.1 From nobody Tue Nov 26 07:22:16 2024 Received: from mail-pl1-f181.google.com (mail-pl1-f181.google.com [209.85.214.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 44C2C86277 for ; Mon, 21 Oct 2024 01:54:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729475647; cv=none; b=DksEp/krxEm7eVxzyRXut3bBrQshaS2DA9iQ3Ghrkmm4Ilnk8Bq3HT3vc7XOLa/CMCeW/u7PWiFgyzxQsz2gOGYyIzrE2n/oj+zg6GWnYxRJbqAxwc9NiY1JRdUCqPeHEUoR4xDpvuNBaZW9uqMipAGAnot6cS7t/Ljc/KK7TcM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729475647; c=relaxed/simple; bh=UKcbLgaIhruPiHQVux7frT2vyssaPMGMH3qyAhwocGw=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=jVCsy/+vNddY1R3LyIEW4by+N1a0ZHKJ5ZG5P7qdCImkR5qSfXH9+BijLyv1xbQULygHstf5Llr1FU01Ae5of0qSkMxbrhOZcozvsiMy1I+oqWruB5BEniJ0x2TCfbG7BxmgvNXO4gnD7TwDieERrJNxeCq1wvNR9Kt+v1tCPiE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=fastly.com; spf=pass smtp.mailfrom=fastly.com; dkim=pass (1024-bit key) header.d=fastly.com header.i=@fastly.com header.b=dQV3fgXV; arc=none smtp.client-ip=209.85.214.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=fastly.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=fastly.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=fastly.com header.i=@fastly.com header.b="dQV3fgXV" Received: by mail-pl1-f181.google.com with SMTP id d9443c01a7336-20ca388d242so26579545ad.2 for ; Sun, 20 Oct 2024 18:54:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fastly.com; s=google; t=1729475644; x=1730080444; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ov+a74yyWVssGZ0VVpLag6M6zuR8CKA5GgGeFDgsMcI=; b=dQV3fgXVYxTZflvEtCMJCkH9wPk6AyRT1vExTi+WroPVPIehXv/5vhvLdRIfoRe4iM MxPrXj/K3sHSPtluNRlbHGTP3uDS+1mmLw95ncELP8qAqog2nP+4snjohk3ucVQw+7ff 2ydJrPXGimyuYMfPOyPrZzjScL7DvU0vn9X3A= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729475644; x=1730080444; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ov+a74yyWVssGZ0VVpLag6M6zuR8CKA5GgGeFDgsMcI=; b=pRfMlUxY5u3h6l8iRjWGqdnEY3kDXwa2A2DtHbD3RqjQa2nq+OyYIxR39M6P3qEUCo UmhEX75Xf02A/lbrFZw4Dbe3NMH+29RxHAL6pqf577b+5Y/gt3gdVl5EmgZrUGxhjdX/ UvrcMAC5zR86OH1C278OONstcpvvSGRuUDyEegOyktTsGzyT3zHkrF3Lj/CA91gKKwNu feDQDDRd3J8Wqul+x7jpHU2/qgpuBpMrKak3Cci80eQZZTujuDvP8SUstXrANWp/vTUq INCVzG2Wo12G7spmEWG6g78wZYKJw/QVpSUQeGWWGs2OeNwlKnNPzrNOnpgqcKIjDRbm juLw== X-Forwarded-Encrypted: i=1; AJvYcCUOk9mv1nOcTa1f6yqZ3z0Qcch5li3zkuS9A4hg8bmi6Cqst0bzqx3VK8wK1H9i494ZKx6BrSE0hxPVUvk=@vger.kernel.org X-Gm-Message-State: AOJu0YzQwXIKDdkdpe7IcDG0jEuQL+KoXb2u/yc6Dx7+rACDFXTvW20+ WuMswCN6gSu41yv+BgZG1teTWSA80EN348GNPc8hofmUbW3iAzsAduRocqd9Jdc= X-Google-Smtp-Source: AGHT+IFRh1M+as63n98NkQTDVZCHlHy/fZhBOPIE+Ea8KXRBufysFtL+AR7pOkA5nc0vjrv7IGmlBg== X-Received: by 2002:a17:903:22c2:b0:20c:f9ec:cd8a with SMTP id d9443c01a7336-20e5a79f596mr152490435ad.7.1729475644492; Sun, 20 Oct 2024 18:54:04 -0700 (PDT) Received: from localhost.localdomain ([2620:11a:c019:0:65e:3115:2f58:c5fd]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-20e7eee650bsm15859985ad.34.2024.10.20.18.54.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 20 Oct 2024 18:54:03 -0700 (PDT) From: Joe Damato To: netdev@vger.kernel.org Cc: namangulati@google.com, edumazet@google.com, amritha.nambiar@intel.com, sridhar.samudrala@intel.com, sdf@fomichev.me, peter@typeblog.net, m2shafiei@uwaterloo.ca, bjorn@rivosinc.com, hch@infradead.org, willy@infradead.org, willemdebruijn.kernel@gmail.com, skhawaja@google.com, kuba@kernel.org, Joe Damato , Martin Karsten , "David S. Miller" , Paolo Abeni , Jonathan Corbet , linux-doc@vger.kernel.org (open list:DOCUMENTATION), linux-kernel@vger.kernel.org (open list), bpf@vger.kernel.org (open list:BPF [MISC]:Keyword:(?:\b|_)bpf(?:\b|_)) Subject: [PATCH net-next v2 6/6] docs: networking: Describe irq suspension Date: Mon, 21 Oct 2024 01:53:01 +0000 Message-Id: <20241021015311.95468-7-jdamato@fastly.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20241021015311.95468-1-jdamato@fastly.com> References: <20241021015311.95468-1-jdamato@fastly.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Describe irq suspension, the epoll ioctls, and the tradeoffs of using different gro_flush_timeout values. Signed-off-by: Joe Damato Co-developed-by: Martin Karsten Signed-off-by: Martin Karsten Acked-by: Stanislav Fomichev Reviewed-by: Bagas Sanjaya --- v1 -> v2: - Updated documentation to describe the per-NAPI configuration parameters. Documentation/networking/napi.rst | 132 +++++++++++++++++++++++++++++- 1 file changed, 130 insertions(+), 2 deletions(-) diff --git a/Documentation/networking/napi.rst b/Documentation/networking/n= api.rst index dfa5d549be9c..3b43477a52ce 100644 --- a/Documentation/networking/napi.rst +++ b/Documentation/networking/napi.rst @@ -192,6 +192,28 @@ is reused to control the delay of the timer, while ``napi_defer_hard_irqs`` controls the number of consecutive empty polls before NAPI gives up and goes back to using hardware IRQs. =20 +The above parameters can also be set on a per-NAPI basis using netlink via +netdev-genl. This can be done programmatically in a user application or by +using a script included in the kernel source tree: ``tools/net/ynl/cli.py`= `. + +For example, using the script: + +.. code-block:: bash + + $ kernel-source/tools/net/ynl/cli.py \ + --spec Documentation/netlink/specs/netdev.yaml \ + --do napi-set \ + --json=3D'{"id": 345, + "defer-hard-irqs": 111, + "gro-flush-timeout": 11111}' + +Similarly, the parameter ``irq-suspend-timeout`` can be set using netlink +via netdev-genl. There is no global sysfs parameter for this value. + +``irq_suspend_timeout`` is used to determine how long an application can +completely suspend IRQs. It is used in combination with SO_PREFER_BUSY_POL= L, +which can be set on a per-epoll context basis with ``EPIOCSPARAMS`` ioctl. + .. _poll: =20 Busy polling @@ -207,6 +229,46 @@ selected sockets or using the global ``net.core.busy_p= oll`` and ``net.core.busy_read`` sysctls. An io_uring API for NAPI busy polling also exists. =20 +epoll-based busy polling +------------------------ + +It is possible to trigger packet processing directly from calls to +``epoll_wait``. In order to use this feature, a user application must ensu= re +all file descriptors which are added to an epoll context have the same NAP= I ID. + +If the application uses a dedicated acceptor thread, the application can o= btain +the NAPI ID of the incoming connection using SO_INCOMING_NAPI_ID and then +distribute that file descriptor to a worker thread. The worker thread woul= d add +the file descriptor to its epoll context. This would ensure each worker th= read +has an epoll context with FDs that have the same NAPI ID. + +Alternatively, if the application uses SO_REUSEPORT, a bpf or ebpf program= be +inserted to distribute incoming connections to threads such that each thre= ad is +only given incoming connections with the same NAPI ID. Care must be taken = to +carefully handle cases where a system may have multiple NICs. + +In order to enable busy polling, there are two choices: + +1. ``/proc/sys/net/core/busy_poll`` can be set with a time in useconds to = busy + loop waiting for events. This is a system-wide setting and will cause a= ll + epoll-based applications to busy poll when they call epoll_wait. This m= ay + not be desirable as many applications may not have the need to busy pol= l. + +2. Applications using recent kernels can issue an ioctl on the epoll conte= xt + file descriptor to set (``EPIOCSPARAMS``) or get (``EPIOCGPARAMS``) ``s= truct + epoll_params``:, which user programs can define as follows: + +.. code-block:: c + + struct epoll_params { + uint32_t busy_poll_usecs; + uint16_t busy_poll_budget; + uint8_t prefer_busy_poll; + + /* pad the struct to a multiple of 64bits */ + uint8_t __pad; + }; + IRQ mitigation --------------- =20 @@ -222,12 +284,78 @@ Such applications can pledge to the kernel that they = will perform a busy polling operation periodically, and the driver should keep the device IRQs permanently masked. This mode is enabled by using the ``SO_PREFER_BUSY_POL= L`` socket option. To avoid system misbehavior the pledge is revoked -if ``gro_flush_timeout`` passes without any busy poll call. +if ``gro_flush_timeout`` passes without any busy poll call. For epoll-based +busy polling applications, the ``prefer_busy_poll`` field of ``struct +epoll_params`` can be set to 1 and the ``EPIOCSPARAMS`` ioctl can be issue= d to +enable this mode. See the above section for more details. =20 The NAPI budget for busy polling is lower than the default (which makes sense given the low latency intention of normal busy polling). This is not the case with IRQ mitigation, however, so the budget can be adjusted -with the ``SO_BUSY_POLL_BUDGET`` socket option. +with the ``SO_BUSY_POLL_BUDGET`` socket option. For epoll-based busy polli= ng +applications, the ``busy_poll_budget`` field can be adjusted to the desire= d value +in ``struct epoll_params`` and set on a specific epoll context using the `= `EPIOCSPARAMS`` +ioctl. See the above section for more details. + +It is important to note that choosing a large value for ``gro_flush_timeou= t`` +will defer IRQs to allow for better batch processing, but will induce late= ncy +when the system is not fully loaded. Choosing a small value for +``gro_flush_timeout`` can cause interference of the user application which= is +attempting to busy poll by device IRQs and softirq processing. This value +should be chosen carefully with these tradeoffs in mind. epoll-based busy +polling applications may be able to mitigate how much user processing happ= ens +by choosing an appropriate value for ``maxevents``. + +Users may want to consider an alternate approach, IRQ suspension, to help = deal +with these tradeoffs. + +IRQ suspension +-------------- + +IRQ suspension is a mechanism wherein device IRQs are masked while epoll +triggers NAPI packet processing. + +While application calls to epoll_wait successfully retrieve events, the ke= rnel will +defer the IRQ suspension timer. If the kernel does not retrieve any events +while busy polling (for example, because network traffic levels subsided),= IRQ +suspension is disabled and the IRQ mitigation strategies described above a= re +engaged. + +This allows users to balance CPU consumption with network processing +efficiency. + +To use this mechanism: + + 1. The per-NAPI config parameter ``irq_suspend_timeout`` should be set t= o the + maximum time (in nanoseconds) the application can have its IRQs + suspended. This is done using netlink, as described above. This timeo= ut + serves as a safety mechanism to restart IRQ driver interrupt processi= ng if + the application has stalled. This value should be chosen so that it c= overs + the amount of time the user application needs to process data from its + call to epoll_wait, noting that applications can control how much data + they retrieve by setting ``max_events`` when calling epoll_wait. + + 2. The sysfs parameter or per-NAPI config parameters ``gro_flush_timeout= `` + and ``napi_defer_hard_irqs`` can be set to low values. They will be u= sed + to defer IRQs after busy poll has found no data. + + 3. The ``prefer_busy_poll`` flag must be set to true. This can be done u= sing + the ``EPIOCSPARAMS`` ioctl as described above. + + 4. The application uses epoll as described above to trigger NAPI packet + processing. + +As mentioned above, as long as subsequent calls to epoll_wait return event= s to +userland, the ``irq_suspend_timeout`` is deferred and IRQs are disabled. T= his +allows the application to process data without interference. + +Once a call to epoll_wait results in no events being found, IRQ suspension= is +automatically disabled and the ``gro_flush_timeout`` and +``napi_defer_hard_irqs`` mitigation mechanisms take over. + +It is expected that ``irq_suspend_timeout`` will be set to a value much la= rger +than ``gro_flush_timeout`` as ``irq_suspend_timeout`` should suspend IRQs = for +the duration of one userland processing cycle. =20 .. _threaded: =20 --=20 2.25.1