From nobody Sun Jun 14 06:05:06 2026 Received: from linux.microsoft.com (linux.microsoft.com [13.77.154.182]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 5BED81FE47B; Sat, 2 May 2026 07:46:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=13.77.154.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777707962; cv=none; b=AXRgqpWZIwPsLMbdrrZ0EqvUsLT+IyvfO/OOrlcvmmX9cw/CXnHUcAfcym3qHqcUimkpqIrx8bC9HJJdPtqMilN5BmhxScLLjTOwOQUcfnOGIuaM46Q0Ro3mu+6vraUrPeBUYkZJb8VQVwYBjgQz9VWcXT8PBJTB3W1OhBvYNeU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777707962; c=relaxed/simple; bh=HouUOri904NMOp6S620izooVjSzSuTHhNCYD29MbSGY=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=QdnlFHq/tZ3BGhE83gjVMUl1//9LjPIXTfH6QrmO24wYPDHsmOsaxR2DTpFzzfR8C9KN2FxOvaoIGJUK/06jiuG6gCgzCwhvWCesjuhMw9ELAhHVNpz0S+WA3ui7NOk85GwQL1YA9J0pxovAxgsMvo9yt4vXD3fryHkgRbx7DpI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.microsoft.com; spf=pass smtp.mailfrom=linux.microsoft.com; dkim=pass (1024-bit key) header.d=linux.microsoft.com header.i=@linux.microsoft.com header.b=bb0tQgTR; arc=none smtp.client-ip=13.77.154.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.microsoft.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.microsoft.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.microsoft.com header.i=@linux.microsoft.com header.b="bb0tQgTR" Received: by linux.microsoft.com (Postfix, from userid 1231) id C8E9E20B716C; Sat, 2 May 2026 00:46:00 -0700 (PDT) DKIM-Filter: OpenDKIM Filter v2.11.0 linux.microsoft.com C8E9E20B716C DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.microsoft.com; s=default; t=1777707960; bh=IPauBJl23XVTTPHaHpwNSZN2cNKdTVqw7x6BxTnQO4Q=; h=From:To:Subject:Date:In-Reply-To:References:From; b=bb0tQgTROYid9FHYdW7Ezg4xIh22mzAgfvYJoyyjJr5JXvzQbIRnsywu5Uz8mbey6 lQAaKQ1JipeR2Q7r90STk3hZL238fc46a+cBZqSORGq2ZWUtIH2/sHccP0KVeRJBqL z5g6FU7tS5RQ1yy3csrCHmiRKCJXMaQf2kZSs+H8= From: Aditya Garg To: kys@microsoft.com, haiyangz@microsoft.com, wei.liu@kernel.org, decui@microsoft.com, longli@microsoft.com, andrew+netdev@lunn.ch, davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, kotaranov@microsoft.com, horms@kernel.org, ssengar@linux.microsoft.com, jacob.e.keller@intel.com, dipayanroy@linux.microsoft.com, ernis@linux.microsoft.com, shirazsaleem@microsoft.com, kees@kernel.org, sbhatta@marvell.com, leitao@debian.org, netdev@vger.kernel.org, linux-hyperv@vger.kernel.org, linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, bpf@vger.kernel.org, gargaditya@microsoft.com, gargaditya@linux.microsoft.com Subject: [PATCH net-next v3 1/2] net: mana: Use per-queue allocation for tx_qp to reduce allocation size Date: Sat, 2 May 2026 00:45:33 -0700 Message-ID: <20260502074552.23857-2-gargaditya@linux.microsoft.com> X-Mailer: git-send-email 2.43.7 In-Reply-To: <20260502074552.23857-1-gargaditya@linux.microsoft.com> References: <20260502074552.23857-1-gargaditya@linux.microsoft.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Convert tx_qp from a single contiguous array allocation to per-queue individual allocations. Each mana_tx_qp struct is approximately 35KB. With many queues (e.g., 32/64), the flat array requires a single contiguous allocation that can fail under memory fragmentation. Change mana_tx_qp *tx_qp to mana_tx_qp **tx_qp (array of pointers), allocating each queue's mana_tx_qp individually via kvzalloc. This reduces each allocation to ~35KB and provides vmalloc fallback, avoiding allocation failure due to fragmentation. Signed-off-by: Aditya Garg Reviewed-by: Haiyang Zhang --- .../net/ethernet/microsoft/mana/mana_bpf.c | 2 +- drivers/net/ethernet/microsoft/mana/mana_en.c | 49 ++++++++++++------- .../ethernet/microsoft/mana/mana_ethtool.c | 2 +- include/net/mana/mana.h | 2 +- 4 files changed, 33 insertions(+), 22 deletions(-) diff --git a/drivers/net/ethernet/microsoft/mana/mana_bpf.c b/drivers/net/e= thernet/microsoft/mana/mana_bpf.c index 7697c9b52ed3..b5e9bb184a1d 100644 --- a/drivers/net/ethernet/microsoft/mana/mana_bpf.c +++ b/drivers/net/ethernet/microsoft/mana/mana_bpf.c @@ -68,7 +68,7 @@ int mana_xdp_xmit(struct net_device *ndev, int n, struct = xdp_frame **frames, count++; } =20 - tx_stats =3D &apc->tx_qp[q_idx].txq.stats; + tx_stats =3D &apc->tx_qp[q_idx]->txq.stats; =20 u64_stats_update_begin(&tx_stats->syncp); tx_stats->xdp_xmit +=3D count; diff --git a/drivers/net/ethernet/microsoft/mana/mana_en.c b/drivers/net/et= hernet/microsoft/mana/mana_en.c index a654b3699c4c..8adf72b96145 100644 --- a/drivers/net/ethernet/microsoft/mana/mana_en.c +++ b/drivers/net/ethernet/microsoft/mana/mana_en.c @@ -355,9 +355,9 @@ netdev_tx_t mana_start_xmit(struct sk_buff *skb, struct= net_device *ndev) if (skb_cow_head(skb, MANA_HEADROOM)) goto tx_drop_count; =20 - txq =3D &apc->tx_qp[txq_idx].txq; + txq =3D &apc->tx_qp[txq_idx]->txq; gdma_sq =3D txq->gdma_sq; - cq =3D &apc->tx_qp[txq_idx].tx_cq; + cq =3D &apc->tx_qp[txq_idx]->tx_cq; tx_stats =3D &txq->stats; =20 BUILD_BUG_ON(MAX_TX_WQE_SGL_ENTRIES !=3D MANA_MAX_TX_WQE_SGL_ENTRIES); @@ -614,7 +614,7 @@ static void mana_get_stats64(struct net_device *ndev, } =20 for (q =3D 0; q < num_queues; q++) { - tx_stats =3D &apc->tx_qp[q].txq.stats; + tx_stats =3D &apc->tx_qp[q]->txq.stats; =20 do { start =3D u64_stats_fetch_begin(&tx_stats->syncp); @@ -2321,21 +2321,26 @@ static void mana_destroy_txq(struct mana_port_conte= xt *apc) return; =20 for (i =3D 0; i < apc->num_queues; i++) { - debugfs_remove_recursive(apc->tx_qp[i].mana_tx_debugfs); - apc->tx_qp[i].mana_tx_debugfs =3D NULL; + if (!apc->tx_qp[i]) + continue; + + debugfs_remove_recursive(apc->tx_qp[i]->mana_tx_debugfs); + apc->tx_qp[i]->mana_tx_debugfs =3D NULL; =20 - napi =3D &apc->tx_qp[i].tx_cq.napi; - if (apc->tx_qp[i].txq.napi_initialized) { + napi =3D &apc->tx_qp[i]->tx_cq.napi; + if (apc->tx_qp[i]->txq.napi_initialized) { napi_synchronize(napi); napi_disable_locked(napi); netif_napi_del_locked(napi); - apc->tx_qp[i].txq.napi_initialized =3D false; + apc->tx_qp[i]->txq.napi_initialized =3D false; } - mana_destroy_wq_obj(apc, GDMA_SQ, apc->tx_qp[i].tx_object); + mana_destroy_wq_obj(apc, GDMA_SQ, apc->tx_qp[i]->tx_object); =20 - mana_deinit_cq(apc, &apc->tx_qp[i].tx_cq); + mana_deinit_cq(apc, &apc->tx_qp[i]->tx_cq); =20 - mana_deinit_txq(apc, &apc->tx_qp[i].txq); + mana_deinit_txq(apc, &apc->tx_qp[i]->txq); + + kvfree(apc->tx_qp[i]); } =20 kfree(apc->tx_qp); @@ -2344,7 +2349,7 @@ static void mana_destroy_txq(struct mana_port_context= *apc) =20 static void mana_create_txq_debugfs(struct mana_port_context *apc, int idx) { - struct mana_tx_qp *tx_qp =3D &apc->tx_qp[idx]; + struct mana_tx_qp *tx_qp =3D apc->tx_qp[idx]; char qnum[32]; =20 sprintf(qnum, "TX-%d", idx); @@ -2383,7 +2388,7 @@ static int mana_create_txq(struct mana_port_context *= apc, int err; int i; =20 - apc->tx_qp =3D kzalloc_objs(struct mana_tx_qp, apc->num_queues); + apc->tx_qp =3D kzalloc_objs(struct mana_tx_qp *, apc->num_queues); if (!apc->tx_qp) return -ENOMEM; =20 @@ -2403,10 +2408,16 @@ static int mana_create_txq(struct mana_port_context= *apc, gc =3D gd->gdma_context; =20 for (i =3D 0; i < apc->num_queues; i++) { - apc->tx_qp[i].tx_object =3D INVALID_MANA_HANDLE; + apc->tx_qp[i] =3D kvzalloc_obj(*apc->tx_qp[i]); + if (!apc->tx_qp[i]) { + err =3D -ENOMEM; + goto out; + } + + apc->tx_qp[i]->tx_object =3D INVALID_MANA_HANDLE; =20 /* Create SQ */ - txq =3D &apc->tx_qp[i].txq; + txq =3D &apc->tx_qp[i]->txq; =20 u64_stats_init(&txq->stats.syncp); txq->ndev =3D net; @@ -2424,7 +2435,7 @@ static int mana_create_txq(struct mana_port_context *= apc, goto out; =20 /* Create SQ's CQ */ - cq =3D &apc->tx_qp[i].tx_cq; + cq =3D &apc->tx_qp[i]->tx_cq; cq->type =3D MANA_CQ_TYPE_TX; =20 cq->txq =3D txq; @@ -2453,7 +2464,7 @@ static int mana_create_txq(struct mana_port_context *= apc, =20 err =3D mana_create_wq_obj(apc, apc->port_handle, GDMA_SQ, &wq_spec, &cq_spec, - &apc->tx_qp[i].tx_object); + &apc->tx_qp[i]->tx_object); =20 if (err) goto out; @@ -3288,7 +3299,7 @@ static int mana_dealloc_queues(struct net_device *nde= v) */ =20 for (i =3D 0; i < apc->num_queues; i++) { - txq =3D &apc->tx_qp[i].txq; + txq =3D &apc->tx_qp[i]->txq; tsleep =3D 1000; while (atomic_read(&txq->pending_sends) > 0 && time_before(jiffies, timeout)) { @@ -3307,7 +3318,7 @@ static int mana_dealloc_queues(struct net_device *nde= v) } =20 for (i =3D 0; i < apc->num_queues; i++) { - txq =3D &apc->tx_qp[i].txq; + txq =3D &apc->tx_qp[i]->txq; while ((skb =3D skb_dequeue(&txq->pending_skbs))) { mana_unmap_skb(skb, apc); dev_kfree_skb_any(skb); diff --git a/drivers/net/ethernet/microsoft/mana/mana_ethtool.c b/drivers/n= et/ethernet/microsoft/mana/mana_ethtool.c index 6a4b42fe0944..04350973e19e 100644 --- a/drivers/net/ethernet/microsoft/mana/mana_ethtool.c +++ b/drivers/net/ethernet/microsoft/mana/mana_ethtool.c @@ -260,7 +260,7 @@ static void mana_get_ethtool_stats(struct net_device *n= dev, } =20 for (q =3D 0; q < num_queues; q++) { - tx_stats =3D &apc->tx_qp[q].txq.stats; + tx_stats =3D &apc->tx_qp[q]->txq.stats; =20 do { start =3D u64_stats_fetch_begin(&tx_stats->syncp); diff --git a/include/net/mana/mana.h b/include/net/mana/mana.h index 8f721cd4e4a7..aa90a858c8e3 100644 --- a/include/net/mana/mana.h +++ b/include/net/mana/mana.h @@ -507,7 +507,7 @@ struct mana_port_context { bool tx_shortform_allowed; u16 tx_vp_offset; =20 - struct mana_tx_qp *tx_qp; + struct mana_tx_qp **tx_qp; =20 /* Indirection Table for RX & TX. The values are queue indexes */ u32 *indir_table; --=20 2.43.0 From nobody Sun Jun 14 06:05:06 2026 Received: from linux.microsoft.com (linux.microsoft.com [13.77.154.182]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 77C1726AA91; Sat, 2 May 2026 07:46:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=13.77.154.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777707962; cv=none; b=OX9a2V90HFyk7/GYoKiVNDoWWjEvPKxKSRIWHK4BQkI1sRwOJu3Cq/DKC3WeDidieOwqoIhwToqON6uftiPRkxNAahAhxZ78za0tYZsZyCtRlWZkTvLk+NnltTptEDql6AgzSl9kEYk5k7PruAq3unfYLIMAF/taOF1WRzrCvQo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777707962; c=relaxed/simple; bh=6UnbY4DrqGtksJ4MzzZtojHfq0UuwCW7gM2ovAeSmYg=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=WIeOorbw289wksbDMBENveAorn+iz2GjmcYrimMpz7lj0HJZUIBckJopIihsAJ2SA4umrl0Jz70ge0ytMKGFwCT1oK8XTXPl8nfJSI/x/Mja6fuXU7N6pkVUCz08kLGisV4OmIeao05RTpCcpTl+X4/SsbpzciL68vKpqLx/gdA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.microsoft.com; spf=pass smtp.mailfrom=linux.microsoft.com; dkim=pass (1024-bit key) header.d=linux.microsoft.com header.i=@linux.microsoft.com header.b=mQJ9jCfP; arc=none smtp.client-ip=13.77.154.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.microsoft.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.microsoft.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.microsoft.com header.i=@linux.microsoft.com header.b="mQJ9jCfP" Received: by linux.microsoft.com (Postfix, from userid 1231) id D633920B716F; Sat, 2 May 2026 00:46:00 -0700 (PDT) DKIM-Filter: OpenDKIM Filter v2.11.0 linux.microsoft.com D633920B716F DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.microsoft.com; s=default; t=1777707960; bh=K/dQLWpvtfHpW7x2/2gVgixuqfIkLb/3dZkP2GDm5KU=; h=From:To:Subject:Date:In-Reply-To:References:From; b=mQJ9jCfPm5Mz8TL5FKv8sudWcV2ovb72IpvnkPocdPlCDhfRNM/KcvotcKMkRswLm 0SyA0PZC5AGrWbuc0BK20JGugnX0nIurvYpYP7gzoiWkFR+8jfMisQLTvgXDoT2sHd 5oMnrDRkEnqCv27LHMxxgIQhsXI2W+3xUonHdVR4= From: Aditya Garg To: kys@microsoft.com, haiyangz@microsoft.com, wei.liu@kernel.org, decui@microsoft.com, longli@microsoft.com, andrew+netdev@lunn.ch, davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, kotaranov@microsoft.com, horms@kernel.org, ssengar@linux.microsoft.com, jacob.e.keller@intel.com, dipayanroy@linux.microsoft.com, ernis@linux.microsoft.com, shirazsaleem@microsoft.com, kees@kernel.org, sbhatta@marvell.com, leitao@debian.org, netdev@vger.kernel.org, linux-hyperv@vger.kernel.org, linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, bpf@vger.kernel.org, gargaditya@microsoft.com, gargaditya@linux.microsoft.com Subject: [PATCH net-next v3 2/2] net: mana: Use kvmalloc for large RX queue and buffer allocations Date: Sat, 2 May 2026 00:45:34 -0700 Message-ID: <20260502074552.23857-3-gargaditya@linux.microsoft.com> X-Mailer: git-send-email 2.43.7 In-Reply-To: <20260502074552.23857-1-gargaditya@linux.microsoft.com> References: <20260502074552.23857-1-gargaditya@linux.microsoft.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The RX path allocations for rxbufs_pre, das_pre, and rxq scale with queue count and queue depth. With high queue counts and depth, these can exceed what kmalloc can reliably provide from physically contiguous memory under fragmentation. Switch these from kmalloc to kvmalloc variants so the allocator transparently falls back to vmalloc when contiguous memory is scarce, and update the corresponding frees to kvfree. Signed-off-by: Aditya Garg Reviewed-by: Haiyang Zhang --- drivers/net/ethernet/microsoft/mana/mana_en.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/drivers/net/ethernet/microsoft/mana/mana_en.c b/drivers/net/et= hernet/microsoft/mana/mana_en.c index 8adf72b96145..e1d8ac3417e8 100644 --- a/drivers/net/ethernet/microsoft/mana/mana_en.c +++ b/drivers/net/ethernet/microsoft/mana/mana_en.c @@ -685,11 +685,11 @@ void mana_pre_dealloc_rxbufs(struct mana_port_context= *mpc) put_page(virt_to_head_page(mpc->rxbufs_pre[i])); } =20 - kfree(mpc->das_pre); + kvfree(mpc->das_pre); mpc->das_pre =3D NULL; =20 out2: - kfree(mpc->rxbufs_pre); + kvfree(mpc->rxbufs_pre); mpc->rxbufs_pre =3D NULL; =20 out1: @@ -806,11 +806,11 @@ int mana_pre_alloc_rxbufs(struct mana_port_context *m= pc, int new_mtu, int num_qu num_rxb =3D num_queues * mpc->rx_queue_size; =20 WARN(mpc->rxbufs_pre, "mana rxbufs_pre exists\n"); - mpc->rxbufs_pre =3D kmalloc_array(num_rxb, sizeof(void *), GFP_KERNEL); + mpc->rxbufs_pre =3D kvmalloc_array(num_rxb, sizeof(void *), GFP_KERNEL); if (!mpc->rxbufs_pre) goto error; =20 - mpc->das_pre =3D kmalloc_objs(dma_addr_t, num_rxb); + mpc->das_pre =3D kvmalloc_objs(dma_addr_t, num_rxb); if (!mpc->das_pre) goto error; =20 @@ -2564,7 +2564,7 @@ static void mana_destroy_rxq(struct mana_port_context= *apc, if (rxq->gdma_rq) mana_gd_destroy_queue(gc, rxq->gdma_rq); =20 - kfree(rxq); + kvfree(rxq); } =20 static int mana_fill_rx_oob(struct mana_recv_buf_oob *rx_oob, u32 mem_key, @@ -2704,7 +2704,7 @@ static struct mana_rxq *mana_create_rxq(struct mana_p= ort_context *apc, =20 gc =3D gd->gdma_context; =20 - rxq =3D kzalloc_flex(*rxq, rx_oobs, apc->rx_queue_size); + rxq =3D kvzalloc_flex(*rxq, rx_oobs, apc->rx_queue_size); if (!rxq) return NULL; =20 --=20 2.43.0