From nobody Wed Nov 27 13:06:53 2024 Received: from mail-pf1-f177.google.com (mail-pf1-f177.google.com [209.85.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AD76C1BE869 for ; Wed, 9 Oct 2024 17:55:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728496522; cv=none; b=jy3LJWk2ul5GYFJb+NKhppwzWnH+vIPBu7lA7Y0ty631lgyyb9DVF6UoK1ok9zlpMeC8DeeqmCBWWpsICckL0W5FMkYU9HvNCYfsgcJ1dfDHJUIBK5Mh4GXZV9JHYMOjCOQrAciAY4u5XzudOUNpgVrDRs7OcidKIgq0cx6zPj8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728496522; c=relaxed/simple; bh=+KXC5+/RQk1GLo+doGP/ldq3RJbVkY3JcfRB1YrEzsg=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=d8BtHB0cu3crBryY5PvWFj9HBjcn8sv6h0fxIAHltn2Hv1g5xTA7cF+9vTTRtg0j0le061f9s0OLC9BbvScoCJekQkb9QjoPOi9TfPxT1Y+KPp7ZUN1As61Cc3yb661tpU1vXht8VlcX+4p2x3PvYldmmtFAES+3bhVYhEz0tYA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=fastly.com; spf=pass smtp.mailfrom=fastly.com; dkim=pass (1024-bit key) header.d=fastly.com header.i=@fastly.com header.b=pvLZARE1; arc=none smtp.client-ip=209.85.210.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=fastly.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=fastly.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=fastly.com header.i=@fastly.com header.b="pvLZARE1" Received: by mail-pf1-f177.google.com with SMTP id d2e1a72fcca58-71e038f3835so102003b3a.0 for ; Wed, 09 Oct 2024 10:55:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fastly.com; s=google; t=1728496520; x=1729101320; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ox0A4aFXFoAY48o8My0plGAbE+nAuK7cgMCjcB43jYs=; b=pvLZARE15WaplPF9FUNLOfUBmGsQ/5eeRKW6+lwOA2yc1WwHxLq41nij2JlpiDOJBe D0w5rhqhcznfbEwzvo4IIqeittmcHNLikhJF9Ha+btkgAaYXp2y3BR3+p4rp+nPqDdjn u/FQot5zZieap3EmuYkwrQMJs1NflGXeu1WMg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1728496520; x=1729101320; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ox0A4aFXFoAY48o8My0plGAbE+nAuK7cgMCjcB43jYs=; b=HlfVXn3y9LQ41t703Ikqe4tiND4bGNbVABpARXWyzgS+u/fzfgcvE7WTYoJls1O2KI HAHyAe7FJNeannLiZJ4loO3ah6339K6oOITsD9+iXBIJPlT6LBPuPRSF6uWv4ZQnr79z WyDf+sTOSLtdfgmDr0RLW58dn+b1xCj0dbF7gBnOJh/g+BtegwmWmfciHRP/XPRWxcCn XEUMC01JdKl1pkhK8sM3yCBbe7dXRsfbDYKFvYn3xy8TZk10SOS+phA7Je3SPPBgge6e aIc7MMFCno3azaSxpXjazInmAy+XGDjii1uhdf1snwNpKw7yw72BhOOYjPe2aEw5wfo/ L9bg== X-Forwarded-Encrypted: i=1; AJvYcCVuc23n8LqiYtZIM3xo/HbRZWpT39T0ug2MN4ROwB5m3SO0mfITEDbNBByS1btuc5UH5mf77Dgrz+EoYk4=@vger.kernel.org X-Gm-Message-State: AOJu0YxriJpCB42LNyX8MQl58GwBtU1xyDzchWaKwe9cUbW2xBm7AZtX CEijeWV5KYCREPgxzcHWjQYyRirdrZub04CXRzD/cC8GUF/7BqFSoLc+/xjCD7A= X-Google-Smtp-Source: AGHT+IHnnx9FcOwDh3XGIUV14QQ1i/u8PP17M+iakKmWQMQfG22QleKXOKdIGNsIu5tG336OJpcbrQ== X-Received: by 2002:a05:6a00:853:b0:71e:cb5:220a with SMTP id d2e1a72fcca58-71e1db77b92mr4807295b3a.10.1728496520023; Wed, 09 Oct 2024 10:55:20 -0700 (PDT) Received: from localhost.localdomain ([2620:11a:c019:0:65e:3115:2f58:c5fd]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-71df0cbc86csm8044685b3a.27.2024.10.09.10.55.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Oct 2024 10:55:19 -0700 (PDT) From: Joe Damato To: netdev@vger.kernel.org Cc: michael.chan@broadcom.com, pavan.chebbi@broadcom.com, Joe Damato , Michael Chan , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , linux-kernel@vger.kernel.org (open list) Subject: [net-next v4 1/2] tg3: Link IRQs to NAPI instances Date: Wed, 9 Oct 2024 17:55:08 +0000 Message-Id: <20241009175509.31753-2-jdamato@fastly.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20241009175509.31753-1-jdamato@fastly.com> References: <20241009175509.31753-1-jdamato@fastly.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Link IRQs to NAPI instances with netif_napi_set_irq. This information can be queried with the netdev-genl API. Begin by testing my tg3 device in its default state: 1 TX queue and 4 RX queues. Compare the output of /proc/interrupts for my tg3 device with the output of netdev-genl after applying this patch: $ cat /proc/interrupts | grep eth0 343: [...] eth0-tx-0 344: [...] eth0-rx-1 345: [...] eth0-rx-2 346: [...] eth0-rx-3 347: [...] eth0-rx-4 As you can see above, tg3 has named the IRQs such that there is a dedicated tx IRQ and 4 dedicated rx IRQs, for a total of 5 IRQs. $ ./tools/net/ynl/cli.py --spec Documentation/netlink/specs/netdev.yaml \ --dump napi-get --json=3D'{"ifindex": 2}' [{'id': 8197, 'ifindex': 2, 'irq': 347}, {'id': 8196, 'ifindex': 2, 'irq': 346}, {'id': 8195, 'ifindex': 2, 'irq': 345}, {'id': 8194, 'ifindex': 2, 'irq': 344}, {'id': 8193, 'ifindex': 2, 'irq': 343}] Netlink displays the same IRQs as above, noting that each is mapped to a unique NAPI instance. Now, reconfigure the NIC to have 4 TX queues and 4 RX queues: $ sudo ethtool -L eth0 rx 4 tx 4 $ sudo ethtool -l eth0 | tail -5 Current hardware settings: RX: 4 TX: 4 Other: n/a Combined: n/a Examine /proc/interrupts once again, noting that tg3 will now rename the IRQs to suggest that they are combined tx and rx without allocating additional IRQs, so the total IRQ count in /proc/interrupts is unchanged: 343: [...] eth0-0 344: [...] eth0-txrx-1 345: [...] eth0-txrx-2 346: [...] eth0-txrx-3 347: [...] eth0-txrx-4 Check the output from netlink again: $ ./tools/net/ynl/cli.py --spec Documentation/netlink/specs/netdev.yaml \ --dump napi-get --json=3D'{"ifindex": 2}' [{'id': 8973, 'ifindex': 2, 'irq': 347}, {'id': 8972, 'ifindex': 2, 'irq': 346}, {'id': 8971, 'ifindex': 2, 'irq': 345}, {'id': 8970, 'ifindex': 2, 'irq': 344}, {'id': 8969, 'ifindex': 2, 'irq': 343}] Signed-off-by: Joe Damato Reviewed-by: Pavan Chebbi Reviewed-by: Michael Chan --- v4: - Updated commit message rfcv3: - wrapped the netif_napi_add call to 80 characters drivers/net/ethernet/broadcom/tg3.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/drivers/net/ethernet/broadcom/tg3.c b/drivers/net/ethernet/bro= adcom/tg3.c index 378815917741..6564072b47ba 100644 --- a/drivers/net/ethernet/broadcom/tg3.c +++ b/drivers/net/ethernet/broadcom/tg3.c @@ -7413,9 +7413,11 @@ static void tg3_napi_init(struct tg3 *tp) { int i; =20 - netif_napi_add(tp->dev, &tp->napi[0].napi, tg3_poll); - for (i =3D 1; i < tp->irq_cnt; i++) - netif_napi_add(tp->dev, &tp->napi[i].napi, tg3_poll_msix); + for (i =3D 0; i < tp->irq_cnt; i++) { + netif_napi_add(tp->dev, &tp->napi[i].napi, + i ? tg3_poll_msix : tg3_poll); + netif_napi_set_irq(&tp->napi[i].napi, tp->napi[i].irq_vec); + } } =20 static void tg3_napi_fini(struct tg3 *tp) --=20 2.25.1 From nobody Wed Nov 27 13:06:53 2024 Received: from mail-pf1-f172.google.com (mail-pf1-f172.google.com [209.85.210.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2C9591E1A0B for ; Wed, 9 Oct 2024 17:55:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728496524; cv=none; b=VxRLi/BHqf/n11tskBrVsWlAJCnltzy9sqMJ/Pgi/yz9qmwRa2zksHBt3jRkaEb1cgFmIWncFDjD5XkZh3EF3okG1hHnAwMGkxYCj95RykmrKHKSVSoNSrY1Lj9uzeb2IEfV9wBXqsZ835LujQO5pvJIA0IZ6wScpN1Y4rxPKbc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728496524; c=relaxed/simple; bh=1abjD/frGy62ltiWGFym0JBG2lFGEIQw0ergcNbY+TM=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=lmcIl9EhAZXKOS6oqDxUt85PxodaUYuVU/i1KH/nAeyKW5Q4xtPW7Z1nrcIYydZQiLNBf2C1Rv8K9a4Yh3LezVrVWwEoUtY8AHmjqT+lDHnhTWJzX9dWGxf1pCIYAYSLYry/JSovzayTyi8r7nZGwcmVtQD6XLmMmx6t1lfq/hQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=fastly.com; spf=pass smtp.mailfrom=fastly.com; dkim=pass (1024-bit key) header.d=fastly.com header.i=@fastly.com header.b=HzAndZ/d; arc=none smtp.client-ip=209.85.210.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=fastly.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=fastly.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=fastly.com header.i=@fastly.com header.b="HzAndZ/d" Received: by mail-pf1-f172.google.com with SMTP id d2e1a72fcca58-71e01eff831so82029b3a.2 for ; Wed, 09 Oct 2024 10:55:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fastly.com; s=google; t=1728496521; x=1729101321; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=YbS/SuH7apsjdrpWIJvUFHB67kEkphI28+Y3pkQvXLw=; b=HzAndZ/dwvttCG0r8YaMuJFwkIBu7BFGmo+prGYWYCgb0MFiZ2yZeBp8xCqCrpQrN/ ei1sArXgNSi/oqZJXNTzjgSbD9+wadcEx3xwZPtGHSNaTtUV93je3e7RAJymCVgEFQxA vAZIrayxRVfOQUWaOI3r7n7KBkcDe3tNlS7G4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1728496521; x=1729101321; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=YbS/SuH7apsjdrpWIJvUFHB67kEkphI28+Y3pkQvXLw=; b=VemZBwEp/owvpb7tCK89CbrkosEgA7WYCdAS6ogtODUxkLabEz/rQNuyOxh/jH4FiO WrwXPjRZMT0xn118RI16DE98KbawpabfwNnbMwKSoN4oNOf+NgPJ1b+bYaT7AiLpjwLy vVOUdoI1Lmajg9Elgo4J6F8tHE2tlZupLVC5tHhEdWw7NFyEdrfh4L/hRDWfXx+lySkV BW4iP4Rx5ljxjUAxs10lHvTME7wCebSBTL2lA1gWsJQzYzsLSJXAcG5hOEyegB5ocPoV J3Bn4x681egTfFlATVow+faPp3XwQFEMPJYfXNQy4jUnbMQIZjYmSr0X0OOk37FFwBFB kSEg== X-Forwarded-Encrypted: i=1; AJvYcCWz+GwYFdvvYpkwcJCCW2/W441fWe0SpiJysrKWzonh10fHAQuQATH0FSh5sxWS/NnS17SGvZzKxRT5Zxo=@vger.kernel.org X-Gm-Message-State: AOJu0Yx/0h5tvEzT1gPfFnFRiBLB4oumbFzUud7EBma/e5uJIESq19vb 0FtW3Lm8eQ3MLuDzCDCeo/ZrEPgcczQWm2+8wKeU8bJXAP8QOnbT8YORUAC3WFjAZ8VCWNRmTLH a X-Google-Smtp-Source: AGHT+IHJr828q1dctX10jkdC9YI/ok/HdtT7WM3qVf98YF6YrgHs288clcpTjimaKYDEvYCapDZXwA== X-Received: by 2002:a05:6a00:8d3:b0:71e:cf8:d6f1 with SMTP id d2e1a72fcca58-71e1db88911mr4197877b3a.14.1728496521501; Wed, 09 Oct 2024 10:55:21 -0700 (PDT) Received: from localhost.localdomain ([2620:11a:c019:0:65e:3115:2f58:c5fd]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-71df0cbc86csm8044685b3a.27.2024.10.09.10.55.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Oct 2024 10:55:21 -0700 (PDT) From: Joe Damato To: netdev@vger.kernel.org Cc: michael.chan@broadcom.com, pavan.chebbi@broadcom.com, Joe Damato , Michael Chan , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , linux-kernel@vger.kernel.org (open list) Subject: [net-next v4 2/2] tg3: Link queues to NAPIs Date: Wed, 9 Oct 2024 17:55:09 +0000 Message-Id: <20241009175509.31753-3-jdamato@fastly.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20241009175509.31753-1-jdamato@fastly.com> References: <20241009175509.31753-1-jdamato@fastly.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Link queues to NAPIs using the netdev-genl API so this information is queryable. First, test with the default setting on my tg3 NIC at boot with 1 TX queue: $ ./tools/net/ynl/cli.py --spec Documentation/netlink/specs/netdev.yaml \ --dump queue-get --json=3D'{"ifindex": 2}' [{'id': 0, 'ifindex': 2, 'napi-id': 8194, 'type': 'rx'}, {'id': 1, 'ifindex': 2, 'napi-id': 8195, 'type': 'rx'}, {'id': 2, 'ifindex': 2, 'napi-id': 8196, 'type': 'rx'}, {'id': 3, 'ifindex': 2, 'napi-id': 8197, 'type': 'rx'}, {'id': 0, 'ifindex': 2, 'napi-id': 8193, 'type': 'tx'}] Now, adjust the number of TX queues to be 4 via ethtool: $ sudo ethtool -L eth0 tx 4 $ sudo ethtool -l eth0 | tail -5 Current hardware settings: RX: 4 TX: 4 Other: n/a Combined: n/a Despite "Combined: n/a" in the ethtool output, /proc/interrupts shows the tg3 has renamed the IRQs to be combined: 343: [...] eth0-0 344: [...] eth0-txrx-1 345: [...] eth0-txrx-2 346: [...] eth0-txrx-3 347: [...] eth0-txrx-4 Now query this via netlink to ensure the queues are linked properly to their NAPIs: $ ./tools/net/ynl/cli.py --spec Documentation/netlink/specs/netdev.yaml \ --dump queue-get --json=3D'{"ifindex": 2}' [{'id': 0, 'ifindex': 2, 'napi-id': 8960, 'type': 'rx'}, {'id': 1, 'ifindex': 2, 'napi-id': 8961, 'type': 'rx'}, {'id': 2, 'ifindex': 2, 'napi-id': 8962, 'type': 'rx'}, {'id': 3, 'ifindex': 2, 'napi-id': 8963, 'type': 'rx'}, {'id': 0, 'ifindex': 2, 'napi-id': 8960, 'type': 'tx'}, {'id': 1, 'ifindex': 2, 'napi-id': 8961, 'type': 'tx'}, {'id': 2, 'ifindex': 2, 'napi-id': 8962, 'type': 'tx'}, {'id': 3, 'ifindex': 2, 'napi-id': 8963, 'type': 'tx'}] As you can see above, id 0 for both TX and RX share a NAPI, NAPI ID 8960, and so on for each queue index up to 3. Signed-off-by: Joe Damato Reviewed-by: Michael Chan --- v4: - Switch the if ... else if to two ifs as suggested by Michael Chan to handle the case where tg3 might use combined queues - Updated the commit message to test both the default and combined queue cases to ensure correctness rfcv3: - Added running counters for numbering the rx and tx queue IDs to tg3_napi_enable and tg3_napi_disable drivers/net/ethernet/broadcom/tg3.c | 39 ++++++++++++++++++++++++++--- 1 file changed, 35 insertions(+), 4 deletions(-) diff --git a/drivers/net/ethernet/broadcom/tg3.c b/drivers/net/ethernet/bro= adcom/tg3.c index 6564072b47ba..c4bce6493afa 100644 --- a/drivers/net/ethernet/broadcom/tg3.c +++ b/drivers/net/ethernet/broadcom/tg3.c @@ -7395,18 +7395,49 @@ static int tg3_poll(struct napi_struct *napi, int b= udget) =20 static void tg3_napi_disable(struct tg3 *tp) { + int txq_idx =3D tp->txq_cnt - 1; + int rxq_idx =3D tp->rxq_cnt - 1; + struct tg3_napi *tnapi; int i; =20 - for (i =3D tp->irq_cnt - 1; i >=3D 0; i--) - napi_disable(&tp->napi[i].napi); + for (i =3D tp->irq_cnt - 1; i >=3D 0; i--) { + tnapi =3D &tp->napi[i]; + if (tnapi->tx_buffers) { + netif_queue_set_napi(tp->dev, txq_idx, + NETDEV_QUEUE_TYPE_TX, NULL); + txq_idx--; + } + if (tnapi->rx_rcb) { + netif_queue_set_napi(tp->dev, rxq_idx, + NETDEV_QUEUE_TYPE_RX, NULL); + rxq_idx--; + } + napi_disable(&tnapi->napi); + } } =20 static void tg3_napi_enable(struct tg3 *tp) { + int txq_idx =3D 0, rxq_idx =3D 0; + struct tg3_napi *tnapi; int i; =20 - for (i =3D 0; i < tp->irq_cnt; i++) - napi_enable(&tp->napi[i].napi); + for (i =3D 0; i < tp->irq_cnt; i++) { + tnapi =3D &tp->napi[i]; + napi_enable(&tnapi->napi); + if (tnapi->tx_buffers) { + netif_queue_set_napi(tp->dev, txq_idx, + NETDEV_QUEUE_TYPE_TX, + &tnapi->napi); + txq_idx++; + } + if (tnapi->rx_rcb) { + netif_queue_set_napi(tp->dev, rxq_idx, + NETDEV_QUEUE_TYPE_RX, + &tnapi->napi); + rxq_idx++; + } + } } =20 static void tg3_napi_init(struct tg3 *tp) --=20 2.25.1