From nobody Tue Nov 26 03:41:06 2024 Received: from szxga05-in.huawei.com (szxga05-in.huawei.com [45.249.212.191]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B31DF146D5A; Tue, 22 Oct 2024 03:28:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=45.249.212.191 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729567725; cv=none; b=aUNg1wlPuBjTf06wos/p7mwqbTXhQfzUe/Hh1pwYOjaUZxViNLNvMk+xP+HZ1kDbLfUfp5cB/M+Xg7Fd2h+LtBzX6h/tf1fi02wQT9ZiFseOmLVQIka0uMcSEovkFUFsxZrCixGDnXE7igUodeppHVE/jJ7QNGFf2/x8a/pBIYE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729567725; c=relaxed/simple; bh=4KIp5dZBQ2vgj/5gMVtIqMo/bCJmLB+PJo5m2qX6YdY=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=BJUV+2PG+Tr0jxTlNY/OQWG5HBoIMXjAbFGNuoiE+h5AttgraXQd5PBlyclZOz2YNStFODwQ4QDj+1rMaCKUrcIsmMoT6ab+EPbtPKEP9xy9pkGUXjeMnHNmfSLnucU9HzmsgUQ+A4Pm0O1HH8r/wngoLhqE7MtjKF6GxD+6LtE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com; spf=pass smtp.mailfrom=huawei.com; arc=none smtp.client-ip=45.249.212.191 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=huawei.com Received: from mail.maildlp.com (unknown [172.19.163.17]) by szxga05-in.huawei.com (SkyGuard) with ESMTP id 4XXcwZ50T1z1HLCp; Tue, 22 Oct 2024 11:24:14 +0800 (CST) Received: from dggpemf200006.china.huawei.com (unknown [7.185.36.61]) by mail.maildlp.com (Postfix) with ESMTPS id 38F861A0188; Tue, 22 Oct 2024 11:28:34 +0800 (CST) Received: from localhost.localdomain (10.90.30.45) by dggpemf200006.china.huawei.com (7.185.36.61) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.11; Tue, 22 Oct 2024 11:28:33 +0800 From: Yunsheng Lin To: , , CC: , , , Yunsheng Lin , Wei Fang , Shenwei Wang , Clark Wang , Andrew Lunn , Eric Dumazet , Jeroen de Borst , Praveen Kaligineedi , Shailend Chand , Tony Nguyen , Przemek Kitszel , Alexander Lobakin , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , Saeed Mahameed , Leon Romanovsky , Tariq Toukan , Felix Fietkau , Lorenzo Bianconi , Ryder Lee , Shayne Chen , Sean Wang , Kalle Valo , Matthias Brugger , AngeloGioacchino Del Regno , Ilias Apalodimas , , , , , , , , , Subject: [PATCH net-next v3 1/3] page_pool: introduce page_pool_to_pp() API Date: Tue, 22 Oct 2024 11:22:11 +0800 Message-ID: <20241022032214.3915232-2-linyunsheng@huawei.com> X-Mailer: git-send-email 2.30.0 In-Reply-To: <20241022032214.3915232-1-linyunsheng@huawei.com> References: <20241022032214.3915232-1-linyunsheng@huawei.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: dggems706-chm.china.huawei.com (10.3.19.183) To dggpemf200006.china.huawei.com (7.185.36.61) Content-Type: text/plain; charset="utf-8" introduce page_pool_to_pp() API to avoid caller accessing page->pp directly. Signed-off-by: Yunsheng Lin --- drivers/net/ethernet/freescale/fec_main.c | 8 +++++--- .../net/ethernet/google/gve/gve_buffer_mgmt_dqo.c | 4 ++-- drivers/net/ethernet/intel/iavf/iavf_txrx.c | 6 ++++-- drivers/net/ethernet/intel/idpf/idpf_txrx.c | 14 +++++++++----- drivers/net/ethernet/intel/libeth/rx.c | 2 +- drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c | 3 ++- drivers/net/netdevsim/netdev.c | 6 ++++-- drivers/net/wireless/mediatek/mt76/mt76.h | 2 +- include/net/libeth/rx.h | 3 ++- include/net/page_pool/helpers.h | 5 +++++ net/core/skbuff.c | 3 ++- net/core/xdp.c | 3 ++- 12 files changed, 39 insertions(+), 20 deletions(-) diff --git a/drivers/net/ethernet/freescale/fec_main.c b/drivers/net/ethern= et/freescale/fec_main.c index 1b55047c0237..98fce41d088c 100644 --- a/drivers/net/ethernet/freescale/fec_main.c +++ b/drivers/net/ethernet/freescale/fec_main.c @@ -1009,7 +1009,8 @@ static void fec_enet_bd_init(struct net_device *dev) struct page *page =3D txq->tx_buf[i].buf_p; =20 if (page) - page_pool_put_page(page->pp, page, 0, false); + page_pool_put_page(page_pool_to_pp(page), + page, 0, false); } =20 txq->tx_buf[i].buf_p =3D NULL; @@ -1549,7 +1550,7 @@ fec_enet_tx_queue(struct net_device *ndev, u16 queue_= id, int budget) xdp_return_frame_rx_napi(xdpf); } else { /* recycle pages of XDP_TX frames */ /* The dma_sync_size =3D 0 as XDP_TX has already synced DMA for_device = */ - page_pool_put_page(page->pp, page, 0, true); + page_pool_put_page(page_pool_to_pp(page), page, 0, true); } =20 txq->tx_buf[index].buf_p =3D NULL; @@ -3311,7 +3312,8 @@ static void fec_enet_free_buffers(struct net_device *= ndev) } else { struct page *page =3D txq->tx_buf[i].buf_p; =20 - page_pool_put_page(page->pp, page, 0, false); + page_pool_put_page(page_pool_to_pp(page), + page, 0, false); } =20 txq->tx_buf[i].buf_p =3D NULL; diff --git a/drivers/net/ethernet/google/gve/gve_buffer_mgmt_dqo.c b/driver= s/net/ethernet/google/gve/gve_buffer_mgmt_dqo.c index 05bf1f80a79c..dd5b45adf5cc 100644 --- a/drivers/net/ethernet/google/gve/gve_buffer_mgmt_dqo.c +++ b/drivers/net/ethernet/google/gve/gve_buffer_mgmt_dqo.c @@ -210,8 +210,8 @@ void gve_free_to_page_pool(struct gve_rx_ring *rx, if (!page) return; =20 - page_pool_put_page(page->pp, page, buf_state->page_info.buf_size, - allow_direct); + page_pool_put_page(page_pool_to_pp(page), page, + buf_state->page_info.buf_size, allow_direct); buf_state->page_info.page =3D NULL; } =20 diff --git a/drivers/net/ethernet/intel/iavf/iavf_txrx.c b/drivers/net/ethe= rnet/intel/iavf/iavf_txrx.c index 26b424fd6718..658d8f9a6abb 100644 --- a/drivers/net/ethernet/intel/iavf/iavf_txrx.c +++ b/drivers/net/ethernet/intel/iavf/iavf_txrx.c @@ -1050,7 +1050,8 @@ static void iavf_add_rx_frag(struct sk_buff *skb, const struct libeth_fqe *rx_buffer, unsigned int size) { - u32 hr =3D rx_buffer->page->pp->p.offset; + struct page_pool *pool =3D page_pool_to_pp(rx_buffer->page); + u32 hr =3D pool->p.offset; =20 skb_add_rx_frag(skb, skb_shinfo(skb)->nr_frags, rx_buffer->page, rx_buffer->offset + hr, size, rx_buffer->truesize); @@ -1067,7 +1068,8 @@ static void iavf_add_rx_frag(struct sk_buff *skb, static struct sk_buff *iavf_build_skb(const struct libeth_fqe *rx_buffer, unsigned int size) { - u32 hr =3D rx_buffer->page->pp->p.offset; + struct page_pool *pool =3D page_pool_to_pp(rx_buffer->page); + u32 hr =3D pool->p.offset; struct sk_buff *skb; void *va; =20 diff --git a/drivers/net/ethernet/intel/idpf/idpf_txrx.c b/drivers/net/ethe= rnet/intel/idpf/idpf_txrx.c index d4e6f0e10487..e3389f1a215f 100644 --- a/drivers/net/ethernet/intel/idpf/idpf_txrx.c +++ b/drivers/net/ethernet/intel/idpf/idpf_txrx.c @@ -385,7 +385,8 @@ static void idpf_rx_page_rel(struct libeth_fqe *rx_buf) if (unlikely(!rx_buf->page)) return; =20 - page_pool_put_full_page(rx_buf->page->pp, rx_buf->page, false); + page_pool_put_full_page(page_pool_to_pp(rx_buf->page), rx_buf->page, + false); =20 rx_buf->page =3D NULL; rx_buf->offset =3D 0; @@ -3097,7 +3098,8 @@ idpf_rx_process_skb_fields(struct idpf_rx_queue *rxq,= struct sk_buff *skb, void idpf_rx_add_frag(struct idpf_rx_buf *rx_buf, struct sk_buff *skb, unsigned int size) { - u32 hr =3D rx_buf->page->pp->p.offset; + struct page_pool *pool =3D page_pool_to_pp(rx_buf->page); + u32 hr =3D pool->p.offset; =20 skb_add_rx_frag(skb, skb_shinfo(skb)->nr_frags, rx_buf->page, rx_buf->offset + hr, size, rx_buf->truesize); @@ -3129,8 +3131,10 @@ static u32 idpf_rx_hsplit_wa(const struct libeth_fqe= *hdr, if (!libeth_rx_sync_for_cpu(buf, copy)) return 0; =20 - dst =3D page_address(hdr->page) + hdr->offset + hdr->page->pp->p.offset; - src =3D page_address(buf->page) + buf->offset + buf->page->pp->p.offset; + dst =3D page_address(hdr->page) + hdr->offset + + page_pool_to_pp(hdr->page)->p.offset; + src =3D page_address(buf->page) + buf->offset + + page_pool_to_pp(buf->page)->p.offset; memcpy(dst, src, LARGEST_ALIGN(copy)); =20 buf->offset +=3D copy; @@ -3148,7 +3152,7 @@ static u32 idpf_rx_hsplit_wa(const struct libeth_fqe = *hdr, */ struct sk_buff *idpf_rx_build_skb(const struct libeth_fqe *buf, u32 size) { - u32 hr =3D buf->page->pp->p.offset; + u32 hr =3D page_pool_to_pp(buf->page)->p.offset; struct sk_buff *skb; void *va; =20 diff --git a/drivers/net/ethernet/intel/libeth/rx.c b/drivers/net/ethernet/= intel/libeth/rx.c index f20926669318..385afca0e61d 100644 --- a/drivers/net/ethernet/intel/libeth/rx.c +++ b/drivers/net/ethernet/intel/libeth/rx.c @@ -207,7 +207,7 @@ EXPORT_SYMBOL_NS_GPL(libeth_rx_fq_destroy, LIBETH); */ void libeth_rx_recycle_slow(struct page *page) { - page_pool_recycle_direct(page->pp, page); + page_pool_recycle_direct(page_pool_to_pp(page), page); } EXPORT_SYMBOL_NS_GPL(libeth_rx_recycle_slow, LIBETH); =20 diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c b/drivers/net= /ethernet/mellanox/mlx5/core/en/xdp.c index 4610621a340e..83511a45a6dc 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c @@ -716,7 +716,8 @@ static void mlx5e_free_xdpsq_desc(struct mlx5e_xdpsq *s= q, /* No need to check ((page->pp_magic & ~0x3UL) =3D=3D PP_SIGNATURE) * as we know this is a page_pool page. */ - page_pool_recycle_direct(page->pp, page); + page_pool_recycle_direct(page_pool_to_pp(page), + page); } while (++n < num); =20 break; diff --git a/drivers/net/netdevsim/netdev.c b/drivers/net/netdevsim/netdev.c index cad85bb0cf54..7590ddd14dda 100644 --- a/drivers/net/netdevsim/netdev.c +++ b/drivers/net/netdevsim/netdev.c @@ -632,7 +632,8 @@ nsim_pp_hold_write(struct file *file, const char __user= *data, if (!ns->page) ret =3D -ENOMEM; } else { - page_pool_put_full_page(ns->page->pp, ns->page, false); + page_pool_put_full_page(page_pool_to_pp(ns->page), ns->page, + false); ns->page =3D NULL; } rtnl_unlock(); @@ -827,7 +828,8 @@ void nsim_destroy(struct netdevsim *ns) =20 /* Put this intentionally late to exercise the orphaning path */ if (ns->page) { - page_pool_put_full_page(ns->page->pp, ns->page, false); + page_pool_put_full_page(page_pool_to_pp(ns->page), ns->page, + false); ns->page =3D NULL; } =20 diff --git a/drivers/net/wireless/mediatek/mt76/mt76.h b/drivers/net/wirele= ss/mediatek/mt76/mt76.h index 0b75a45ad2e8..94a277290909 100644 --- a/drivers/net/wireless/mediatek/mt76/mt76.h +++ b/drivers/net/wireless/mediatek/mt76/mt76.h @@ -1688,7 +1688,7 @@ static inline void mt76_put_page_pool_buf(void *buf, = bool allow_direct) { struct page *page =3D virt_to_head_page(buf); =20 - page_pool_put_full_page(page->pp, page, allow_direct); + page_pool_put_full_page(page_pool_to_pp(page), page, allow_direct); } =20 static inline void * diff --git a/include/net/libeth/rx.h b/include/net/libeth/rx.h index 43574bd6612f..beee7ddd77a5 100644 --- a/include/net/libeth/rx.h +++ b/include/net/libeth/rx.h @@ -137,7 +137,8 @@ static inline bool libeth_rx_sync_for_cpu(const struct = libeth_fqe *fqe, return false; } =20 - page_pool_dma_sync_for_cpu(page->pp, page, fqe->offset, len); + page_pool_dma_sync_for_cpu(page_pool_to_pp(page), page, fqe->offset, + len); =20 return true; } diff --git a/include/net/page_pool/helpers.h b/include/net/page_pool/helper= s.h index 793e6fd78bc5..1659f1995985 100644 --- a/include/net/page_pool/helpers.h +++ b/include/net/page_pool/helpers.h @@ -83,6 +83,11 @@ static inline u64 *page_pool_ethtool_stats_get(u64 *data= , const void *stats) } #endif =20 +static inline struct page_pool *page_pool_to_pp(struct page *page) +{ + return page->pp; +} + /** * page_pool_dev_alloc_pages() - allocate a page. * @pool: pool from which to allocate diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 00afeb90c23a..649e02e2cbc8 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -1033,7 +1033,8 @@ bool napi_pp_put_page(netmem_ref netmem) if (unlikely(!is_pp_netmem(netmem))) return false; =20 - page_pool_put_full_netmem(netmem_get_pp(netmem), netmem, false); + page_pool_put_full_netmem(page_pool_to_pp(netmem_to_page(netmem)), + netmem, false); =20 return true; } diff --git a/net/core/xdp.c b/net/core/xdp.c index bcc5551c6424..e8582036b411 100644 --- a/net/core/xdp.c +++ b/net/core/xdp.c @@ -384,7 +384,8 @@ void __xdp_return(void *data, struct xdp_mem_info *mem,= bool napi_direct, /* No need to check ((page->pp_magic & ~0x3UL) =3D=3D PP_SIGNATURE) * as mem->type knows this a page_pool page */ - page_pool_put_full_page(page->pp, page, napi_direct); + page_pool_put_full_page(page_pool_to_pp(page), page, + napi_direct); break; case MEM_TYPE_PAGE_SHARED: page_frag_free(data); --=20 2.33.0 From nobody Tue Nov 26 03:41:06 2024 Received: from szxga02-in.huawei.com (szxga02-in.huawei.com [45.249.212.188]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4285913DDB9; Tue, 22 Oct 2024 03:28:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=45.249.212.188 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729567720; cv=none; b=JkBXknmgvgRt3i4qI6qfB5A8rEqndQypcOTOjw/5QFoJFzbiagi6ZSGN5s9GNuwGIFz3QFRS/WVWe7hiYXIrEmveaBj1Vhbrll5L7868k6PAihHZlZNnU0Ggx+5eYm41/OJm6hA3ipYWD636Z1H42p9yev4L4z+SJDzAdqkHLK8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729567720; c=relaxed/simple; bh=FzKHvWSdH/zegyPx5g6zWcE3t9ZJBNWsqMZut4mO5Fw=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=mWe3/8pWfEb31ighMaaDrLpBNG2tML2Hpq0AhxdI1np+GljX+h2ldPjI8aqj5UIj5sj3UdRiz3PIL4kbN6RDSIY9x5YZcuNjup8xVaaXhRGsrSRmpevoydq+m/xcMdhf3Lg9uRDjF1NzjZAlLXi5yyWjHdtkp8My1tUt2XRf1g0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com; spf=pass smtp.mailfrom=huawei.com; arc=none smtp.client-ip=45.249.212.188 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=huawei.com Received: from mail.maildlp.com (unknown [172.19.162.254]) by szxga02-in.huawei.com (SkyGuard) with ESMTP id 4XXczL59K0zpXCK; Tue, 22 Oct 2024 11:26:38 +0800 (CST) Received: from dggpemf200006.china.huawei.com (unknown [7.185.36.61]) by mail.maildlp.com (Postfix) with ESMTPS id F0D7C180106; Tue, 22 Oct 2024 11:28:35 +0800 (CST) Received: from localhost.localdomain (10.90.30.45) by dggpemf200006.china.huawei.com (7.185.36.61) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.11; Tue, 22 Oct 2024 11:28:35 +0800 From: Yunsheng Lin To: , , CC: , , , Yunsheng Lin , Alexander Lobakin , Jesper Dangaard Brouer , Ilias Apalodimas , Eric Dumazet , , Subject: [PATCH net-next v3 2/3] page_pool: fix timing for checking and disabling napi_local Date: Tue, 22 Oct 2024 11:22:12 +0800 Message-ID: <20241022032214.3915232-3-linyunsheng@huawei.com> X-Mailer: git-send-email 2.30.0 In-Reply-To: <20241022032214.3915232-1-linyunsheng@huawei.com> References: <20241022032214.3915232-1-linyunsheng@huawei.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: dggems706-chm.china.huawei.com (10.3.19.183) To dggpemf200006.china.huawei.com (7.185.36.61) Content-Type: text/plain; charset="utf-8" page_pool page may be freed from skb_defer_free_flush() in softirq context without binding to any specific napi, it may cause use-after-free problem due to the below time window, as below, CPU1 may still access napi->list_owner after CPU0 free the napi memory: CPU 0 CPU1 page_pool_destroy() skb_defer_free_flush() . . . napi =3D READ_ONCE(pool->p.napi); . . page_pool_disable_direct_recycling() . driver free napi memory . . . . napi && READ_ONCE(napi->list_owner) =3D=3D cpuid . . Use rcu mechanism to avoid the above problem. Note, the above was found during code reviewing on how to fix the problem in [1]. 1. https://lore.kernel.org/lkml/8067f204-1380-4d37-8ffd-007fc6f26738@kernel= .org/T/ Fixes: dd64b232deb8 ("page_pool: unlink from napi during destroy") Signed-off-by: Yunsheng Lin CC: Alexander Lobakin Reviewed-by: Xuan Zhuo --- As the IOMMU fix patch depends on synchronize_rcu() added in this patch and the time window is so small that it doesn't seem to be an urgent fix, so target the net-next as the IOMMU fix patch does. --- net/core/page_pool.c | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/net/core/page_pool.c b/net/core/page_pool.c index a813d30d2135..dd497f5c927d 100644 --- a/net/core/page_pool.c +++ b/net/core/page_pool.c @@ -795,6 +795,7 @@ __page_pool_put_page(struct page_pool *pool, netmem_ref= netmem, static bool page_pool_napi_local(const struct page_pool *pool) { const struct napi_struct *napi; + bool napi_local; u32 cpuid; =20 if (unlikely(!in_softirq())) @@ -810,9 +811,15 @@ static bool page_pool_napi_local(const struct page_poo= l *pool) if (READ_ONCE(pool->cpuid) =3D=3D cpuid) return true; =20 + /* Synchronizated with page_pool_destory() to avoid use-after-free + * for 'napi'. + */ + rcu_read_lock(); napi =3D READ_ONCE(pool->p.napi); + napi_local =3D napi && READ_ONCE(napi->list_owner) =3D=3D cpuid; + rcu_read_unlock(); =20 - return napi && READ_ONCE(napi->list_owner) =3D=3D cpuid; + return napi_local; } =20 void page_pool_put_unrefed_netmem(struct page_pool *pool, netmem_ref netme= m, @@ -1126,6 +1133,12 @@ void page_pool_destroy(struct page_pool *pool) if (!page_pool_release(pool)) return; =20 + /* Paired with rcu lock in page_pool_napi_local() to enable clearing + * of pool->p.napi in page_pool_disable_direct_recycling() is seen + * before returning to driver to free the napi instance. + */ + synchronize_rcu(); + page_pool_detached(pool); pool->defer_start =3D jiffies; pool->defer_warn =3D jiffies + DEFER_WARN_INTERVAL; --=20 2.33.0 From nobody Tue Nov 26 03:41:06 2024 Received: from szxga01-in.huawei.com (szxga01-in.huawei.com [45.249.212.187]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 95CC613A86A; Tue, 22 Oct 2024 03:28:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=45.249.212.187 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729567726; cv=none; b=VxmieBjOXo+/oM/IF/1Uz89omUDROMoD60JUrnYrAYmpM2lO+BZwOd64jTyfTwWYXexw3ThGsBZpSJw0gOnD98QqnmF8J2Dun1WJQyr/kpF+5oOiG4Q3bqGUgN2Xz0WLaoau+LYAraYLNR+ZQXUZq9CViyIKmaBWtfDndc1JsoM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729567726; c=relaxed/simple; bh=Z6C9+/PxzKtQMSZofXiwJVdfm/rel4SmJUPNAyKOBpk=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=SgBusEAasTSLxBdUd2lMfnUBOOf3Fi9XuJi0SF6SPnnrGE7SBN0HpiJ5c7VToKiW/pKkdN8Pxvz4ELJbQ2RPtr8/tLzwdOPFmWaNGC0xy1PuWViT/ad8AocAs2AO/pl5AxafbipdDdUiq16c4B/TNCUB7U2rFdPR4spjG3bLCf4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com; spf=pass smtp.mailfrom=huawei.com; arc=none smtp.client-ip=45.249.212.187 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=huawei.com Received: from mail.maildlp.com (unknown [172.19.162.254]) by szxga01-in.huawei.com (SkyGuard) with ESMTP id 4XXczy0dDLzyTVq; Tue, 22 Oct 2024 11:27:10 +0800 (CST) Received: from dggpemf200006.china.huawei.com (unknown [7.185.36.61]) by mail.maildlp.com (Postfix) with ESMTPS id 2E86B180106; Tue, 22 Oct 2024 11:28:40 +0800 (CST) Received: from localhost.localdomain (10.90.30.45) by dggpemf200006.china.huawei.com (7.185.36.61) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.11; Tue, 22 Oct 2024 11:28:39 +0800 From: Yunsheng Lin To: , , CC: , , , Yunsheng Lin , Robin Murphy , Alexander Duyck , IOMMU , Andrew Morton , Eric Dumazet , Jesper Dangaard Brouer , Ilias Apalodimas , , , Subject: [PATCH net-next v3 3/3] page_pool: fix IOMMU crash when driver has already unbound Date: Tue, 22 Oct 2024 11:22:13 +0800 Message-ID: <20241022032214.3915232-4-linyunsheng@huawei.com> X-Mailer: git-send-email 2.30.0 In-Reply-To: <20241022032214.3915232-1-linyunsheng@huawei.com> References: <20241022032214.3915232-1-linyunsheng@huawei.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: dggems706-chm.china.huawei.com (10.3.19.183) To dggpemf200006.china.huawei.com (7.185.36.61) Content-Type: text/plain; charset="utf-8" Networking driver with page_pool support may hand over page still with dma mapping to network stack and try to reuse that page after network stack is done with it and passes it back to page_pool to avoid the penalty of dma mapping/unmapping. With all the caching in the network stack, some pages may be held in the network stack without returning to the page_pool soon enough, and with VF disable causing the driver unbound, the page_pool does not stop the driver from doing it's unbounding work, instead page_pool uses workqueue to check if there is some pages coming back from the network stack periodically, if there is any, it will do the dma unmmapping related cleanup work. As mentioned in [1], attempting DMA unmaps after the driver has already unbound may leak resources or at worst corrupt memory. Fundamentally, the page pool code cannot allow DMA mappings to outlive the driver they belong to. Currently it seems there are at least two cases that the page is not released fast enough causing dma unmmapping done after driver has already unbound: 1. ipv4 packet defragmentation timeout: this seems to cause delay up to 30 secs. 2. skb_defer_free_flush(): this may cause infinite delay if there is no triggering for net_rx_action(). In order not to do the dma unmmapping after driver has already unbound and stall the unloading of the networking driver, add the pool->items array to record all the pages including the ones which are handed over to network stack, so the page_pool can do the dma unmmapping for those pages when page_pool_destroy() is called. As the pool->items need to be large enough to avoid performance degradation, add a 'item_full' stat to indicate the allocation failure due to unavailability of pool->items. Note, the devmem patchset seems to make the bug harder to fix, and may make backporting harder too. As there is no actual user for the devmem and the fixing for devmem is unclear for now, this patch does not consider fixing the case for devmem yet. 1. https://lore.kernel.org/lkml/8067f204-1380-4d37-8ffd-007fc6f26738@kernel= .org/T/ Fixes: f71fec47c2df ("page_pool: make sure struct device is stable") Signed-off-by: Yunsheng Lin Tested-by: Yonglong Liu CC: Robin Murphy CC: Alexander Duyck CC: IOMMU --- include/linux/mm_types.h | 2 +- include/linux/skbuff.h | 1 + include/net/netmem.h | 10 +- include/net/page_pool/helpers.h | 4 +- include/net/page_pool/types.h | 17 ++- net/core/devmem.c | 4 +- net/core/netmem_priv.h | 5 +- net/core/page_pool.c | 213 ++++++++++++++++++++++++++------ net/core/page_pool_priv.h | 10 +- 9 files changed, 210 insertions(+), 56 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 6e3bdf8e38bc..d3e30dcfa021 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -120,7 +120,7 @@ struct page { * page_pool allocated pages. */ unsigned long pp_magic; - struct page_pool *pp; + struct page_pool_item *pp_item; unsigned long _pp_mapping_pad; unsigned long dma_addr; atomic_long_t pp_ref_count; diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index 48f1e0fa2a13..44f3707ec3e3 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -38,6 +38,7 @@ #include #include #include +#include =20 /** * DOC: skb checksums diff --git a/include/net/netmem.h b/include/net/netmem.h index 8a6e20be4b9d..5e7b4d1c1c44 100644 --- a/include/net/netmem.h +++ b/include/net/netmem.h @@ -23,7 +23,7 @@ DECLARE_STATIC_KEY_FALSE(page_pool_mem_providers); struct net_iov { unsigned long __unused_padding; unsigned long pp_magic; - struct page_pool *pp; + struct page_pool_item *pp_item; struct dmabuf_genpool_chunk_owner *owner; unsigned long dma_addr; atomic_long_t pp_ref_count; @@ -33,7 +33,7 @@ struct net_iov { * * struct { * unsigned long pp_magic; - * struct page_pool *pp; + * struct page_pool_item *pp_item; * unsigned long _pp_mapping_pad; * unsigned long dma_addr; * atomic_long_t pp_ref_count; @@ -49,7 +49,7 @@ struct net_iov { static_assert(offsetof(struct page, pg) =3D=3D \ offsetof(struct net_iov, iov)) NET_IOV_ASSERT_OFFSET(pp_magic, pp_magic); -NET_IOV_ASSERT_OFFSET(pp, pp); +NET_IOV_ASSERT_OFFSET(pp_item, pp_item); NET_IOV_ASSERT_OFFSET(dma_addr, dma_addr); NET_IOV_ASSERT_OFFSET(pp_ref_count, pp_ref_count); #undef NET_IOV_ASSERT_OFFSET @@ -127,9 +127,9 @@ static inline struct net_iov *__netmem_clear_lsb(netmem= _ref netmem) return (struct net_iov *)((__force unsigned long)netmem & ~NET_IOV); } =20 -static inline struct page_pool *netmem_get_pp(netmem_ref netmem) +static inline struct page_pool_item *netmem_get_pp_item(netmem_ref netmem) { - return __netmem_clear_lsb(netmem)->pp; + return __netmem_clear_lsb(netmem)->pp_item; } =20 static inline atomic_long_t *netmem_get_pp_ref_count_ref(netmem_ref netmem) diff --git a/include/net/page_pool/helpers.h b/include/net/page_pool/helper= s.h index 1659f1995985..f781c81f8aa9 100644 --- a/include/net/page_pool/helpers.h +++ b/include/net/page_pool/helpers.h @@ -85,7 +85,9 @@ static inline u64 *page_pool_ethtool_stats_get(u64 *data,= const void *stats) =20 static inline struct page_pool *page_pool_to_pp(struct page *page) { - return page->pp; + struct page_pool_item *item =3D page->pp_item; + + return container_of(item, struct page_pool, items[item->pp_idx]); } =20 /** diff --git a/include/net/page_pool/types.h b/include/net/page_pool/types.h index c022c410abe3..194006d2930f 100644 --- a/include/net/page_pool/types.h +++ b/include/net/page_pool/types.h @@ -102,6 +102,7 @@ struct page_pool_params { * @refill: an allocation which triggered a refill of the cache * @waive: pages obtained from the ptr ring that cannot be added to * the cache due to a NUMA mismatch + * @item_full items array is full */ struct page_pool_alloc_stats { u64 fast; @@ -110,6 +111,7 @@ struct page_pool_alloc_stats { u64 empty; u64 refill; u64 waive; + u64 item_full; }; =20 /** @@ -142,6 +144,11 @@ struct page_pool_stats { }; #endif =20 +struct page_pool_item { + netmem_ref pp_netmem; + unsigned int pp_idx; +}; + /* The whole frag API block must stay within one cacheline. On 32-bit syst= ems, * sizeof(long) =3D=3D sizeof(int), so that the block size is ``3 * sizeof= (long)``. * On 64-bit systems, the actual size is ``2 * sizeof(long) + sizeof(int)`= `. @@ -161,6 +168,8 @@ struct page_pool { =20 int cpuid; u32 pages_state_hold_cnt; + unsigned int item_mask; + unsigned int item_idx; =20 bool has_init_callback:1; /* slow::init_callback is set */ bool dma_map:1; /* Perform DMA mapping */ @@ -228,7 +237,11 @@ struct page_pool { */ refcount_t user_cnt; =20 - u64 destroy_cnt; + /* Lock to avoid doing dma unmapping concurrently when + * destroy_cnt > 0. + */ + spinlock_t destroy_lock; + unsigned int destroy_cnt; =20 /* Slow/Control-path information follows */ struct page_pool_params_slow slow; @@ -239,6 +252,8 @@ struct page_pool { u32 napi_id; u32 id; } user; + + struct page_pool_item items[] ____cacheline_aligned_in_smp; }; =20 struct page *page_pool_alloc_pages(struct page_pool *pool, gfp_t gfp); diff --git a/net/core/devmem.c b/net/core/devmem.c index 11b91c12ee11..09c5aa83f12a 100644 --- a/net/core/devmem.c +++ b/net/core/devmem.c @@ -85,7 +85,7 @@ net_devmem_alloc_dmabuf(struct net_devmem_dmabuf_binding = *binding) niov =3D &owner->niovs[index]; =20 niov->pp_magic =3D 0; - niov->pp =3D NULL; + niov->pp_item =3D NULL; atomic_long_set(&niov->pp_ref_count, 0); =20 return niov; @@ -380,7 +380,7 @@ bool mp_dmabuf_devmem_release_page(struct page_pool *po= ol, netmem_ref netmem) if (WARN_ON_ONCE(refcount !=3D 1)) return false; =20 - page_pool_clear_pp_info(netmem); + page_pool_clear_pp_info(pool, netmem); =20 net_devmem_free_dmabuf(netmem_to_net_iov(netmem)); =20 diff --git a/net/core/netmem_priv.h b/net/core/netmem_priv.h index 7eadb8393e00..3173f6070cf7 100644 --- a/net/core/netmem_priv.h +++ b/net/core/netmem_priv.h @@ -18,9 +18,10 @@ static inline void netmem_clear_pp_magic(netmem_ref netm= em) __netmem_clear_lsb(netmem)->pp_magic =3D 0; } =20 -static inline void netmem_set_pp(netmem_ref netmem, struct page_pool *pool) +static inline void netmem_set_pp_item(netmem_ref netmem, + struct page_pool_item *item) { - __netmem_clear_lsb(netmem)->pp =3D pool; + __netmem_clear_lsb(netmem)->pp_item =3D item; } =20 static inline void netmem_set_dma_addr(netmem_ref netmem, diff --git a/net/core/page_pool.c b/net/core/page_pool.c index dd497f5c927d..fa4a98b4fc67 100644 --- a/net/core/page_pool.c +++ b/net/core/page_pool.c @@ -61,6 +61,7 @@ static const char pp_stats[][ETH_GSTRING_LEN] =3D { "rx_pp_alloc_empty", "rx_pp_alloc_refill", "rx_pp_alloc_waive", + "rx_pp_alloc_item_full", "rx_pp_recycle_cached", "rx_pp_recycle_cache_full", "rx_pp_recycle_ring", @@ -94,6 +95,7 @@ bool page_pool_get_stats(const struct page_pool *pool, stats->alloc_stats.empty +=3D pool->alloc_stats.empty; stats->alloc_stats.refill +=3D pool->alloc_stats.refill; stats->alloc_stats.waive +=3D pool->alloc_stats.waive; + stats->alloc_stats.item_full +=3D pool->alloc_stats.item_full; =20 for_each_possible_cpu(cpu) { const struct page_pool_recycle_stats *pcpu =3D @@ -139,6 +141,7 @@ u64 *page_pool_ethtool_stats_get(u64 *data, const void = *stats) *data++ =3D pool_stats->alloc_stats.empty; *data++ =3D pool_stats->alloc_stats.refill; *data++ =3D pool_stats->alloc_stats.waive; + *data++ =3D pool_stats->alloc_stats.item_full; *data++ =3D pool_stats->recycle_stats.cached; *data++ =3D pool_stats->recycle_stats.cache_full; *data++ =3D pool_stats->recycle_stats.ring; @@ -267,14 +270,12 @@ static int page_pool_init(struct page_pool *pool, return -ENOMEM; } =20 + spin_lock_init(&pool->destroy_lock); atomic_set(&pool->pages_state_release_cnt, 0); =20 /* Driver calling page_pool_create() also call page_pool_destroy() */ refcount_set(&pool->user_cnt, 1); =20 - if (pool->dma_map) - get_device(pool->p.dev); - if (pool->slow.flags & PP_FLAG_ALLOW_UNREADABLE_NETMEM) { /* We rely on rtnl_lock()ing to make sure netdev_rx_queue * configuration doesn't change while we're initializing @@ -312,15 +313,93 @@ static void page_pool_uninit(struct page_pool *pool) { ptr_ring_cleanup(&pool->ring, NULL); =20 - if (pool->dma_map) - put_device(pool->p.dev); - #ifdef CONFIG_PAGE_POOL_STATS if (!pool->system) free_percpu(pool->recycle_stats); #endif } =20 +static void page_pool_item_init(struct page_pool *pool, unsigned int item_= cnt) +{ + struct page_pool_item *items =3D pool->items; + unsigned int i; + + WARN_ON_ONCE(!is_power_of_2(item_cnt)); + + for (i =3D 0; i < item_cnt; i++) + items[i].pp_idx =3D i; + + pool->item_mask =3D item_cnt - 1; +} + +static void page_pool_item_uninit(struct page_pool *pool) +{ + struct page_pool_item *items =3D pool->items; + unsigned int mask =3D pool->item_mask; + unsigned int i, unmapped =3D 0; + + if (!pool->dma_map || pool->mp_priv) + return; + + spin_lock_bh(&pool->destroy_lock); + + for (i =3D 0; i <=3D mask; i++) { + struct page *page; + + page =3D netmem_to_page(READ_ONCE(items[i].pp_netmem)); + if (!page) + continue; + + unmapped++; + dma_unmap_page_attrs(pool->p.dev, page_pool_get_dma_addr(page), + PAGE_SIZE << pool->p.order, + pool->p.dma_dir, DMA_ATTR_SKIP_CPU_SYNC | + DMA_ATTR_WEAK_ORDERING); + page_pool_set_dma_addr(page, 0); + } + + WARN_ONCE(page_pool_inflight(pool, false) !=3D unmapped, + "page_pool(%u): unmapped(%u) !=3D inflight page(%d)\n", + pool->user.id, unmapped, page_pool_inflight(pool, false)); + + pool->dma_map =3D false; + spin_unlock_bh(&pool->destroy_lock); +} + +static bool page_pool_item_add(struct page_pool *pool, netmem_ref netmem) +{ + struct page_pool_item *items =3D pool->items; + unsigned int mask =3D pool->item_mask; + unsigned int idx =3D pool->item_idx; + unsigned int i; + + for (i =3D 0; i <=3D mask; i++) { + unsigned int mask_idx =3D idx++ & mask; + + if (!READ_ONCE(items[mask_idx].pp_netmem)) { + WRITE_ONCE(items[mask_idx].pp_netmem, netmem); + netmem_set_pp_item(netmem, &items[mask_idx]); + pool->item_idx =3D idx; + return true; + } + } + + pool->item_idx =3D idx; + alloc_stat_inc(pool, item_full); + return false; +} + +static void page_pool_item_del(struct page_pool *pool, netmem_ref netmem) +{ + struct page_pool_item *item =3D netmem_to_page(netmem)->pp_item; + struct page_pool_item *items =3D pool->items; + unsigned int idx =3D item->pp_idx; + + DEBUG_NET_WARN_ON_ONCE(items[idx].pp_netmem !=3D netmem); + WRITE_ONCE(items[idx].pp_netmem, (netmem_ref)NULL); + netmem_set_pp_item(netmem, NULL); +} + /** * page_pool_create_percpu() - create a page pool for a given cpu. * @params: parameters, see struct page_pool_params @@ -329,10 +408,15 @@ static void page_pool_uninit(struct page_pool *pool) struct page_pool * page_pool_create_percpu(const struct page_pool_params *params, int cpuid) { +#define PAGE_POOL_MIN_INFLIGHT_ITEMS 512 + unsigned int item_cnt =3D (params->pool_size ? : 1024) + + PP_ALLOC_CACHE_SIZE + PAGE_POOL_MIN_INFLIGHT_ITEMS; struct page_pool *pool; int err; =20 - pool =3D kzalloc_node(sizeof(*pool), GFP_KERNEL, params->nid); + item_cnt =3D roundup_pow_of_two(item_cnt); + pool =3D kvzalloc_node(struct_size(pool, items, item_cnt), GFP_KERNEL, + params->nid); if (!pool) return ERR_PTR(-ENOMEM); =20 @@ -340,6 +424,8 @@ page_pool_create_percpu(const struct page_pool_params *= params, int cpuid) if (err < 0) goto err_free; =20 + page_pool_item_init(pool, item_cnt); + err =3D page_pool_list(pool); if (err) goto err_uninit; @@ -350,7 +436,7 @@ page_pool_create_percpu(const struct page_pool_params *= params, int cpuid) page_pool_uninit(pool); err_free: pr_warn("%s() gave up with errno %d\n", __func__, err); - kfree(pool); + kvfree(pool); return ERR_PTR(err); } EXPORT_SYMBOL(page_pool_create_percpu); @@ -365,7 +451,7 @@ struct page_pool *page_pool_create(const struct page_po= ol_params *params) } EXPORT_SYMBOL(page_pool_create); =20 -static void page_pool_return_page(struct page_pool *pool, netmem_ref netme= m); +static void __page_pool_return_page(struct page_pool *pool, netmem_ref net= mem); =20 static noinline netmem_ref page_pool_refill_alloc_cache(struct page_pool *= pool) { @@ -403,7 +489,7 @@ static noinline netmem_ref page_pool_refill_alloc_cache= (struct page_pool *pool) * (2) break out to fallthrough to alloc_pages_node. * This limit stress on page buddy alloactor. */ - page_pool_return_page(pool, netmem); + __page_pool_return_page(pool, netmem); alloc_stat_inc(pool, waive); netmem =3D 0; break; @@ -436,26 +522,32 @@ static netmem_ref __page_pool_get_cached(struct page_= pool *pool) return netmem; } =20 -static void __page_pool_dma_sync_for_device(const struct page_pool *pool, - netmem_ref netmem, - u32 dma_sync_size) +static __always_inline void +__page_pool_dma_sync_for_device(const struct page_pool *pool, netmem_ref n= etmem, + u32 dma_sync_size) { #if defined(CONFIG_HAS_DMA) && defined(CONFIG_DMA_NEED_SYNC) - dma_addr_t dma_addr =3D page_pool_get_dma_addr_netmem(netmem); + if (pool->dma_sync && dma_dev_need_sync(pool->p.dev)) { + dma_addr_t dma_addr =3D page_pool_get_dma_addr_netmem(netmem); =20 - dma_sync_size =3D min(dma_sync_size, pool->p.max_len); - __dma_sync_single_for_device(pool->p.dev, dma_addr + pool->p.offset, - dma_sync_size, pool->p.dma_dir); + dma_sync_size =3D min(dma_sync_size, pool->p.max_len); + __dma_sync_single_for_device(pool->p.dev, + dma_addr + pool->p.offset, + dma_sync_size, pool->p.dma_dir); + } #endif } =20 +/* Called from page_pool_put_*() path, need to synchronizated with + * page_pool_destory() path. + */ static __always_inline void -page_pool_dma_sync_for_device(const struct page_pool *pool, - netmem_ref netmem, +page_pool_dma_sync_for_device(const struct page_pool *pool, netmem_ref net= mem, u32 dma_sync_size) { - if (pool->dma_sync && dma_dev_need_sync(pool->p.dev)) - __page_pool_dma_sync_for_device(pool, netmem, dma_sync_size); + rcu_read_lock(); + __page_pool_dma_sync_for_device(pool, netmem, dma_sync_size); + rcu_read_unlock(); } =20 static bool page_pool_dma_map(struct page_pool *pool, netmem_ref netmem) @@ -477,7 +569,7 @@ static bool page_pool_dma_map(struct page_pool *pool, n= etmem_ref netmem) if (page_pool_set_dma_addr_netmem(netmem, dma)) goto unmap_failed; =20 - page_pool_dma_sync_for_device(pool, netmem, pool->p.max_len); + __page_pool_dma_sync_for_device(pool, netmem, pool->p.max_len); =20 return true; =20 @@ -499,19 +591,24 @@ static struct page *__page_pool_alloc_page_order(stru= ct page_pool *pool, if (unlikely(!page)) return NULL; =20 - if (pool->dma_map && unlikely(!page_pool_dma_map(pool, page_to_netmem(pag= e)))) { - put_page(page); - return NULL; - } + if (unlikely(!page_pool_set_pp_info(pool, page_to_netmem(page)))) + goto err_alloc; + + if (pool->dma_map && unlikely(!page_pool_dma_map(pool, page_to_netmem(pag= e)))) + goto err_set_info; =20 alloc_stat_inc(pool, slow_high_order); - page_pool_set_pp_info(pool, page_to_netmem(page)); =20 /* Track how many pages are held 'in-flight' */ pool->pages_state_hold_cnt++; trace_page_pool_state_hold(pool, page_to_netmem(page), pool->pages_state_hold_cnt); return page; +err_set_info: + page_pool_clear_pp_info(pool, page_to_netmem(page)); +err_alloc: + put_page(page); + return NULL; } =20 /* slow path */ @@ -546,12 +643,18 @@ static noinline netmem_ref __page_pool_alloc_pages_sl= ow(struct page_pool *pool, */ for (i =3D 0; i < nr_pages; i++) { netmem =3D pool->alloc.cache[i]; + + if (unlikely(!page_pool_set_pp_info(pool, netmem))) { + put_page(netmem_to_page(netmem)); + continue; + } + if (dma_map && unlikely(!page_pool_dma_map(pool, netmem))) { + page_pool_clear_pp_info(pool, netmem); put_page(netmem_to_page(netmem)); continue; } =20 - page_pool_set_pp_info(pool, netmem); pool->alloc.cache[pool->alloc.count++] =3D netmem; /* Track how many pages are held 'in-flight' */ pool->pages_state_hold_cnt++; @@ -623,9 +726,11 @@ s32 page_pool_inflight(const struct page_pool *pool, b= ool strict) return inflight; } =20 -void page_pool_set_pp_info(struct page_pool *pool, netmem_ref netmem) +bool page_pool_set_pp_info(struct page_pool *pool, netmem_ref netmem) { - netmem_set_pp(netmem, pool); + if (unlikely(!page_pool_item_add(pool, netmem))) + return false; + netmem_or_pp_magic(netmem, PP_SIGNATURE); =20 /* Ensuring all pages have been split into one fragment initially: @@ -637,12 +742,14 @@ void page_pool_set_pp_info(struct page_pool *pool, ne= tmem_ref netmem) page_pool_fragment_netmem(netmem, 1); if (pool->has_init_callback) pool->slow.init_callback(netmem, pool->slow.init_arg); + + return true; } =20 -void page_pool_clear_pp_info(netmem_ref netmem) +void page_pool_clear_pp_info(struct page_pool *pool, netmem_ref netmem) { netmem_clear_pp_magic(netmem); - netmem_set_pp(netmem, NULL); + page_pool_item_del(pool, netmem); } =20 static __always_inline void __page_pool_release_page_dma(struct page_pool = *pool, @@ -670,7 +777,7 @@ static __always_inline void __page_pool_release_page_dm= a(struct page_pool *pool, * a regular page (that will eventually be returned to the normal * page-allocator via put_page). */ -void page_pool_return_page(struct page_pool *pool, netmem_ref netmem) +void __page_pool_return_page(struct page_pool *pool, netmem_ref netmem) { int count; bool put; @@ -688,7 +795,7 @@ void page_pool_return_page(struct page_pool *pool, netm= em_ref netmem) trace_page_pool_state_release(pool, netmem, count); =20 if (put) { - page_pool_clear_pp_info(netmem); + page_pool_clear_pp_info(pool, netmem); put_page(netmem_to_page(netmem)); } /* An optimization would be to call __free_pages(page, pool->p.order) @@ -697,6 +804,27 @@ void page_pool_return_page(struct page_pool *pool, net= mem_ref netmem) */ } =20 +/* Called from page_pool_put_*() path, need to synchronizated with + * page_pool_destory() path. + */ +static void page_pool_return_page(struct page_pool *pool, netmem_ref netme= m) +{ + unsigned int destroy_cnt; + + rcu_read_lock(); + + destroy_cnt =3D READ_ONCE(pool->destroy_cnt); + if (unlikely(destroy_cnt)) { + spin_lock_bh(&pool->destroy_lock); + __page_pool_return_page(pool, netmem); + spin_unlock_bh(&pool->destroy_lock); + } else { + __page_pool_return_page(pool, netmem); + } + + rcu_read_unlock(); +} + static bool page_pool_recycle_in_ring(struct page_pool *pool, netmem_ref n= etmem) { int ret; @@ -920,11 +1048,11 @@ static netmem_ref page_pool_drain_frag(struct page_p= ool *pool, return 0; =20 if (__page_pool_page_can_be_recycled(netmem)) { - page_pool_dma_sync_for_device(pool, netmem, -1); + __page_pool_dma_sync_for_device(pool, netmem, -1); return netmem; } =20 - page_pool_return_page(pool, netmem); + __page_pool_return_page(pool, netmem); return 0; } =20 @@ -938,7 +1066,7 @@ static void page_pool_free_frag(struct page_pool *pool) if (!netmem || page_pool_unref_netmem(netmem, drain_count)) return; =20 - page_pool_return_page(pool, netmem); + __page_pool_return_page(pool, netmem); } =20 netmem_ref page_pool_alloc_frag_netmem(struct page_pool *pool, @@ -1022,7 +1150,7 @@ static void __page_pool_destroy(struct page_pool *poo= l) static_branch_dec(&page_pool_mem_providers); } =20 - kfree(pool); + kvfree(pool); } =20 static void page_pool_empty_alloc_cache_once(struct page_pool *pool) @@ -1045,7 +1173,7 @@ static void page_pool_empty_alloc_cache_once(struct p= age_pool *pool) static void page_pool_scrub(struct page_pool *pool) { page_pool_empty_alloc_cache_once(pool); - pool->destroy_cnt++; + WRITE_ONCE(pool->destroy_cnt, pool->destroy_cnt + 1); =20 /* No more consumers should exist, but producers could still * be in-flight. @@ -1133,12 +1261,17 @@ void page_pool_destroy(struct page_pool *pool) if (!page_pool_release(pool)) return; =20 + /* Skip dma sync to avoid calling dma API after driver has unbound */ + pool->dma_sync =3D false; + /* Paired with rcu lock in page_pool_napi_local() to enable clearing * of pool->p.napi in page_pool_disable_direct_recycling() is seen * before returning to driver to free the napi instance. */ synchronize_rcu(); =20 + page_pool_item_uninit(pool); + page_pool_detached(pool); pool->defer_start =3D jiffies; pool->defer_warn =3D jiffies + DEFER_WARN_INTERVAL; @@ -1159,7 +1292,7 @@ void page_pool_update_nid(struct page_pool *pool, int= new_nid) /* Flush pool alloc cache, as refill will check NUMA node */ while (pool->alloc.count) { netmem =3D pool->alloc.cache[--pool->alloc.count]; - page_pool_return_page(pool, netmem); + __page_pool_return_page(pool, netmem); } } EXPORT_SYMBOL(page_pool_update_nid); diff --git a/net/core/page_pool_priv.h b/net/core/page_pool_priv.h index 57439787b9c2..5d85f862a30a 100644 --- a/net/core/page_pool_priv.h +++ b/net/core/page_pool_priv.h @@ -36,16 +36,18 @@ static inline bool page_pool_set_dma_addr(struct page *= page, dma_addr_t addr) } =20 #if defined(CONFIG_PAGE_POOL) -void page_pool_set_pp_info(struct page_pool *pool, netmem_ref netmem); -void page_pool_clear_pp_info(netmem_ref netmem); +bool page_pool_set_pp_info(struct page_pool *pool, netmem_ref netmem); +void page_pool_clear_pp_info(struct page_pool *pool, netmem_ref netmem); int page_pool_check_memory_provider(struct net_device *dev, struct netdev_rx_queue *rxq); #else -static inline void page_pool_set_pp_info(struct page_pool *pool, +static inline bool page_pool_set_pp_info(struct page_pool *pool, netmem_ref netmem) { + return true; } -static inline void page_pool_clear_pp_info(netmem_ref netmem) +static inline void page_pool_clear_pp_info(struct page_pool *pool, + netmem_ref netmem) { } static inline int page_pool_check_memory_provider(struct net_device *dev, --=20 2.33.0