From nobody Tue Dec 16 14:20:07 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 35E512620C9; Mon, 28 Apr 2025 09:22:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745832175; cv=none; b=dWleTBAXc3T0BimvyYdh6J3i+w4kFA/5tphNipi1/HQsF9GvxM+4O+zBr1FyEGv8qmm9Dzd6K/lwF+ZJUhh0ID4d5Lp6uaAzDCaGc0tcLCh6eu0NFaOdKJHzkW33uqHrpBUiw9ws6/gYTmKpQiiUD7YBk2dZMd7zV7nmudi3rYk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745832175; c=relaxed/simple; bh=M2YsEsDCT/ILJFtm7dJI15ZdywR83LmuE+/fb4FpI+8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=GRrMzASKsqkpc3r8b7BBWetIif/BBJvtR3sU8pwrmQ9PMWVrEZaiql499Xz1PERMzNDcPW3Deah9GADpnPa6hyOvOwLfpMxZmxMftIjm8jKJo9XpR+FAFHKqGy642859/I+N/85oQ+dQQ2YcHAzDD4BQziq3X7Q7zBgvNJVHDZg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=O+YhCjZ+; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="O+YhCjZ+" Received: by smtp.kernel.org (Postfix) with ESMTPSA id E719BC4CEED; Mon, 28 Apr 2025 09:22:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1745832174; bh=M2YsEsDCT/ILJFtm7dJI15ZdywR83LmuE+/fb4FpI+8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=O+YhCjZ+mbScFKi5vMtdlMKcCCpC9gs+cPnGEWYSJiaaRamFN+NMlxwSlJfqhrg1S pSyNYB8GgLRyAshKQ/h096qEgTdgkUUZgbxtMPbx21VfBdWZimFY+FmNNvhkZLGMtI WDSPgaFMvW2jvGvRekNJbNSm+xJq7N10fmqlOPcnmWBS3kBkcWhXSO41LLnpZ8A3vt w9owbxDfLKRVBom15gYV+q3TWpXnM9ajbovs/NBMHlIf7+L595/rC8clj21BeJ6/iA Rl6d4c0VDyUrKUK5RDHoWiFn9+Nlj74Zso2XJ+cV8IwMYLaYRW6QJTkqJ1T76BxZWW 9BW/bgIzMGXqw== From: Leon Romanovsky To: Marek Szyprowski , Jens Axboe , Christoph Hellwig , Keith Busch Cc: Jake Edge , Jonathan Corbet , Jason Gunthorpe , Zhu Yanjun , Robin Murphy , Joerg Roedel , Will Deacon , Sagi Grimberg , Bjorn Helgaas , Logan Gunthorpe , Yishai Hadas , Shameer Kolothum , Kevin Tian , Alex Williamson , =?UTF-8?q?J=C3=A9r=C3=B4me=20Glisse?= , Andrew Morton , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-rdma@vger.kernel.org, iommu@lists.linux.dev, linux-nvme@lists.infradead.org, linux-pci@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org, Niklas Schnelle , Chuck Lever , Luis Chamberlain , Matthew Wilcox , Dan Williams , Kanchan Joshi , Chaitanya Kulkarni , Leon Romanovsky Subject: [PATCH v10 01/24] PCI/P2PDMA: Refactor the p2pdma mapping helpers Date: Mon, 28 Apr 2025 12:22:07 +0300 Message-ID: <3ad16e0fc3b8f66593a837c9cdcd34bda1e1ab22.1745831017.git.leon@kernel.org> X-Mailer: git-send-email 2.49.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Christoph Hellwig The current scheme with a single helper to determine the P2P status and map a scatterlist segment force users to always use the map_sg helper to DMA map, which we're trying to get away from because they are very cache inefficient. Refactor the code so that there is a single helper that checks the P2P state for a page, including the result that it is not a P2P page to simplify the callers, and a second one to perform the address translation for a bus mapped P2P transfer that does not depend on the scatterlist structure. Signed-off-by: Christoph Hellwig Reviewed-by: Logan Gunthorpe Acked-by: Bjorn Helgaas Tested-by: Jens Axboe Reviewed-by: Luis Chamberlain Signed-off-by: Leon Romanovsky Reviewed-by: Lu Baolu --- drivers/iommu/dma-iommu.c | 47 +++++++++++++++++----------------- drivers/pci/p2pdma.c | 38 ++++----------------------- include/linux/dma-map-ops.h | 51 +++++++++++++++++++++++++++++-------- kernel/dma/direct.c | 43 +++++++++++++++---------------- 4 files changed, 91 insertions(+), 88 deletions(-) diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index a775e4dbe06f..8a89e63c5973 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -1359,7 +1359,6 @@ int iommu_dma_map_sg(struct device *dev, struct scatt= erlist *sg, int nents, struct scatterlist *s, *prev =3D NULL; int prot =3D dma_info_to_prot(dir, dev_is_dma_coherent(dev), attrs); struct pci_p2pdma_map_state p2pdma_state =3D {}; - enum pci_p2pdma_map_type map; dma_addr_t iova; size_t iova_len =3D 0; unsigned long mask =3D dma_get_seg_boundary(dev); @@ -1389,28 +1388,30 @@ int iommu_dma_map_sg(struct device *dev, struct sca= tterlist *sg, int nents, size_t s_length =3D s->length; size_t pad_len =3D (mask - iova_len + 1) & mask; =20 - if (is_pci_p2pdma_page(sg_page(s))) { - map =3D pci_p2pdma_map_segment(&p2pdma_state, dev, s); - switch (map) { - case PCI_P2PDMA_MAP_BUS_ADDR: - /* - * iommu_map_sg() will skip this segment as - * it is marked as a bus address, - * __finalise_sg() will copy the dma address - * into the output segment. - */ - continue; - case PCI_P2PDMA_MAP_THRU_HOST_BRIDGE: - /* - * Mapping through host bridge should be - * mapped with regular IOVAs, thus we - * do nothing here and continue below. - */ - break; - default: - ret =3D -EREMOTEIO; - goto out_restore_sg; - } + switch (pci_p2pdma_state(&p2pdma_state, dev, sg_page(s))) { + case PCI_P2PDMA_MAP_THRU_HOST_BRIDGE: + /* + * Mapping through host bridge should be mapped with + * regular IOVAs, thus we do nothing here and continue + * below. + */ + break; + case PCI_P2PDMA_MAP_NONE: + break; + case PCI_P2PDMA_MAP_BUS_ADDR: + /* + * iommu_map_sg() will skip this segment as it is marked + * as a bus address, __finalise_sg() will copy the dma + * address into the output segment. + */ + s->dma_address =3D pci_p2pdma_bus_addr_map(&p2pdma_state, + sg_phys(s)); + sg_dma_len(s) =3D sg->length; + sg_dma_mark_bus_address(s); + continue; + default: + ret =3D -EREMOTEIO; + goto out_restore_sg; } =20 sg_dma_address(s) =3D s_iova_off; diff --git a/drivers/pci/p2pdma.c b/drivers/pci/p2pdma.c index 19214ec81fbb..8d955c25aed3 100644 --- a/drivers/pci/p2pdma.c +++ b/drivers/pci/p2pdma.c @@ -1004,40 +1004,12 @@ static enum pci_p2pdma_map_type pci_p2pdma_map_type= (struct dev_pagemap *pgmap, return type; } =20 -/** - * pci_p2pdma_map_segment - map an sg segment determining the mapping type - * @state: State structure that should be declared outside of the for_each= _sg() - * loop and initialized to zero. - * @dev: DMA device that's doing the mapping operation - * @sg: scatterlist segment to map - * - * This is a helper to be used by non-IOMMU dma_map_sg() implementations w= here - * the sg segment is the same for the page_link and the dma_address. - * - * Attempt to map a single segment in an SGL with the PCI bus address. - * The segment must point to a PCI P2PDMA page and thus must be - * wrapped in a is_pci_p2pdma_page(sg_page(sg)) check. - * - * Returns the type of mapping used and maps the page if the type is - * PCI_P2PDMA_MAP_BUS_ADDR. - */ -enum pci_p2pdma_map_type -pci_p2pdma_map_segment(struct pci_p2pdma_map_state *state, struct device *= dev, - struct scatterlist *sg) +void __pci_p2pdma_update_state(struct pci_p2pdma_map_state *state, + struct device *dev, struct page *page) { - if (state->pgmap !=3D page_pgmap(sg_page(sg))) { - state->pgmap =3D page_pgmap(sg_page(sg)); - state->map =3D pci_p2pdma_map_type(state->pgmap, dev); - state->bus_off =3D to_p2p_pgmap(state->pgmap)->bus_offset; - } - - if (state->map =3D=3D PCI_P2PDMA_MAP_BUS_ADDR) { - sg->dma_address =3D sg_phys(sg) + state->bus_off; - sg_dma_len(sg) =3D sg->length; - sg_dma_mark_bus_address(sg); - } - - return state->map; + state->pgmap =3D page_pgmap(page); + state->map =3D pci_p2pdma_map_type(state->pgmap, dev); + state->bus_off =3D to_p2p_pgmap(state->pgmap)->bus_offset; } =20 /** diff --git a/include/linux/dma-map-ops.h b/include/linux/dma-map-ops.h index e172522cd936..c3086edeccc6 100644 --- a/include/linux/dma-map-ops.h +++ b/include/linux/dma-map-ops.h @@ -443,6 +443,11 @@ enum pci_p2pdma_map_type { */ PCI_P2PDMA_MAP_UNKNOWN =3D 0, =20 + /* + * Not a PCI P2PDMA transfer. + */ + PCI_P2PDMA_MAP_NONE, + /* * PCI_P2PDMA_MAP_NOT_SUPPORTED: Indicates the transaction will * traverse the host bridge and the host bridge is not in the @@ -471,21 +476,47 @@ enum pci_p2pdma_map_type { =20 struct pci_p2pdma_map_state { struct dev_pagemap *pgmap; - int map; + enum pci_p2pdma_map_type map; u64 bus_off; }; =20 -#ifdef CONFIG_PCI_P2PDMA -enum pci_p2pdma_map_type -pci_p2pdma_map_segment(struct pci_p2pdma_map_state *state, struct device *= dev, - struct scatterlist *sg); -#else /* CONFIG_PCI_P2PDMA */ +/* helper for pci_p2pdma_state(), do not use directly */ +void __pci_p2pdma_update_state(struct pci_p2pdma_map_state *state, + struct device *dev, struct page *page); + +/** + * pci_p2pdma_state - check the P2P transfer state of a page + * @state: P2P state structure + * @dev: device to transfer to/from + * @page: page to map + * + * Check if @page is a PCI P2PDMA page, and if yes of what kind. Returns = the + * map type, and updates @state with all information needed for a P2P tran= sfer. + */ static inline enum pci_p2pdma_map_type -pci_p2pdma_map_segment(struct pci_p2pdma_map_state *state, struct device *= dev, - struct scatterlist *sg) +pci_p2pdma_state(struct pci_p2pdma_map_state *state, struct device *dev, + struct page *page) +{ + if (IS_ENABLED(CONFIG_PCI_P2PDMA) && is_pci_p2pdma_page(page)) { + if (state->pgmap !=3D page_pgmap(page)) + __pci_p2pdma_update_state(state, dev, page); + return state->map; + } + return PCI_P2PDMA_MAP_NONE; +} + +/** + * pci_p2pdma_bus_addr_map - map a PCI_P2PDMA_MAP_BUS_ADDR P2P transfer + * @state: P2P state structure + * @paddr: physical address to map + * + * Map a physically contiguous PCI_P2PDMA_MAP_BUS_ADDR transfer. + */ +static inline dma_addr_t +pci_p2pdma_bus_addr_map(struct pci_p2pdma_map_state *state, phys_addr_t pa= ddr) { - return PCI_P2PDMA_MAP_NOT_SUPPORTED; + WARN_ON_ONCE(state->map !=3D PCI_P2PDMA_MAP_BUS_ADDR); + return paddr + state->bus_off; } -#endif /* CONFIG_PCI_P2PDMA */ =20 #endif /* _LINUX_DMA_MAP_OPS_H */ diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c index b8fe0b3d0ffb..cec43cd5ed62 100644 --- a/kernel/dma/direct.c +++ b/kernel/dma/direct.c @@ -462,34 +462,33 @@ int dma_direct_map_sg(struct device *dev, struct scat= terlist *sgl, int nents, enum dma_data_direction dir, unsigned long attrs) { struct pci_p2pdma_map_state p2pdma_state =3D {}; - enum pci_p2pdma_map_type map; struct scatterlist *sg; int i, ret; =20 for_each_sg(sgl, sg, nents, i) { - if (is_pci_p2pdma_page(sg_page(sg))) { - map =3D pci_p2pdma_map_segment(&p2pdma_state, dev, sg); - switch (map) { - case PCI_P2PDMA_MAP_BUS_ADDR: - continue; - case PCI_P2PDMA_MAP_THRU_HOST_BRIDGE: - /* - * Any P2P mapping that traverses the PCI - * host bridge must be mapped with CPU physical - * address and not PCI bus addresses. This is - * done with dma_direct_map_page() below. - */ - break; - default: - ret =3D -EREMOTEIO; + switch (pci_p2pdma_state(&p2pdma_state, dev, sg_page(sg))) { + case PCI_P2PDMA_MAP_THRU_HOST_BRIDGE: + /* + * Any P2P mapping that traverses the PCI host bridge + * must be mapped with CPU physical address and not PCI + * bus addresses. + */ + break; + case PCI_P2PDMA_MAP_NONE: + sg->dma_address =3D dma_direct_map_page(dev, sg_page(sg), + sg->offset, sg->length, dir, attrs); + if (sg->dma_address =3D=3D DMA_MAPPING_ERROR) { + ret =3D -EIO; goto out_unmap; } - } - - sg->dma_address =3D dma_direct_map_page(dev, sg_page(sg), - sg->offset, sg->length, dir, attrs); - if (sg->dma_address =3D=3D DMA_MAPPING_ERROR) { - ret =3D -EIO; + break; + case PCI_P2PDMA_MAP_BUS_ADDR: + sg->dma_address =3D pci_p2pdma_bus_addr_map(&p2pdma_state, + sg_phys(sg)); + sg_dma_mark_bus_address(sg); + continue; + default: + ret =3D -EREMOTEIO; goto out_unmap; } sg_dma_len(sg) =3D sg->length; --=20 2.49.0 From nobody Tue Dec 16 14:20:07 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4370A25F998; Mon, 28 Apr 2025 09:22:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745832171; cv=none; b=gFtlrZuDeWAZKBcVeQWpBjXrQr7VnLmh2ZUkuoyfkKUSKFTRmjePWX1o82jAMd86P7j2dV3y/dfIo1k9ih/lcPAJSePy/ra6AZ+fB1KkZX0ukXMT60f2rW8UuRtK2PhtmLjxRGu4RaXicg1xMIROUUcuUFtAbTSykd155+69R/k= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745832171; c=relaxed/simple; bh=KNETdLZSwm9vDRb7tfRzLdFkAW3npwZ5glNzbA77Pw0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=EpyFSMq3FXvsHcvGZb0mrS1Ibv9I1P37O9pWyhuWisHDg8El38Y/v5ze/OdI6Tmv9tJx6T/YeiqilqtoO9fqBrhmrHUeev3z7a8z+O+uDfkEwJchof/yjRi5iKdnp05Ehx+F7nH7lN0ySEO/4RdGXDpZ/9qgdS0ITaWAxRm0yTg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=QyNg16EO; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="QyNg16EO" Received: by smtp.kernel.org (Postfix) with ESMTPSA id CFE92C4CEE9; Mon, 28 Apr 2025 09:22:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1745832170; bh=KNETdLZSwm9vDRb7tfRzLdFkAW3npwZ5glNzbA77Pw0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=QyNg16EOsc3ZqETJYD1AU0Ecv7YEeFIaUVFjugCRozHn2KUNMlIdCpZ4dKSw6n91g 4fYomuJsYCESg/2W5f22tHa+x9S6RwqSGJ3sNLoZc8ZuXNBhaugrOA+rVNU3Bonb9t 3K03/IrlWNZy5B3tX0NWpyAPgYrG0s1znb8+bOgMhKSdIgMWewvPHf1qlhLUuosCXR B3aQFlErki3Wyarbnwpc0/bC8pNR/xbJqXk9NlKCaEOwBtMdflnTjGmQnrp2W/F10L fmaDFeaoPp9edXKGVmYjv7azKz8IKVYGDodgURYLHRduvNq7xZi7g1hXqt6VqqjhPJ BWh76EAjTlWMw== From: Leon Romanovsky To: Marek Szyprowski , Jens Axboe , Christoph Hellwig , Keith Busch Cc: Jake Edge , Jonathan Corbet , Jason Gunthorpe , Zhu Yanjun , Robin Murphy , Joerg Roedel , Will Deacon , Sagi Grimberg , Bjorn Helgaas , Logan Gunthorpe , Yishai Hadas , Shameer Kolothum , Kevin Tian , Alex Williamson , =?UTF-8?q?J=C3=A9r=C3=B4me=20Glisse?= , Andrew Morton , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-rdma@vger.kernel.org, iommu@lists.linux.dev, linux-nvme@lists.infradead.org, linux-pci@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org, Niklas Schnelle , Chuck Lever , Luis Chamberlain , Matthew Wilcox , Dan Williams , Kanchan Joshi , Chaitanya Kulkarni , Leon Romanovsky Subject: [PATCH v10 02/24] dma-mapping: move the PCI P2PDMA mapping helpers to pci-p2pdma.h Date: Mon, 28 Apr 2025 12:22:08 +0300 Message-ID: <1c9f2a2e97f77d6eebb5e19651723c0459e0dc29.1745831017.git.leon@kernel.org> X-Mailer: git-send-email 2.49.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Christoph Hellwig To support the upcoming non-scatterlist mapping helpers, we need to go back to have them called outside of the DMA API. Thus move them out of dma-map-ops.h, which is only for DMA API implementations to pci-p2pdma.h, which is for driver use. Note that the core helper is still not exported as the mapping is expected to be done only by very highlevel subsystem code at least for now. Signed-off-by: Christoph Hellwig Reviewed-by: Logan Gunthorpe Acked-by: Bjorn Helgaas Tested-by: Jens Axboe Reviewed-by: Luis Chamberlain Signed-off-by: Leon Romanovsky Reviewed-by: Lu Baolu --- drivers/iommu/dma-iommu.c | 1 + include/linux/dma-map-ops.h | 85 ------------------------------------- include/linux/pci-p2pdma.h | 85 +++++++++++++++++++++++++++++++++++++ kernel/dma/direct.c | 1 + 4 files changed, 87 insertions(+), 85 deletions(-) diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index 8a89e63c5973..9ba8d8bc0ce9 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -27,6 +27,7 @@ #include #include #include +#include #include #include #include diff --git a/include/linux/dma-map-ops.h b/include/linux/dma-map-ops.h index c3086edeccc6..f48e5fb88bd5 100644 --- a/include/linux/dma-map-ops.h +++ b/include/linux/dma-map-ops.h @@ -434,89 +434,4 @@ static inline void debug_dma_dump_mappings(struct devi= ce *dev) #endif /* CONFIG_DMA_API_DEBUG */ =20 extern const struct dma_map_ops dma_dummy_ops; - -enum pci_p2pdma_map_type { - /* - * PCI_P2PDMA_MAP_UNKNOWN: Used internally for indicating the mapping - * type hasn't been calculated yet. Functions that return this enum - * never return this value. - */ - PCI_P2PDMA_MAP_UNKNOWN =3D 0, - - /* - * Not a PCI P2PDMA transfer. - */ - PCI_P2PDMA_MAP_NONE, - - /* - * PCI_P2PDMA_MAP_NOT_SUPPORTED: Indicates the transaction will - * traverse the host bridge and the host bridge is not in the - * allowlist. DMA Mapping routines should return an error when - * this is returned. - */ - PCI_P2PDMA_MAP_NOT_SUPPORTED, - - /* - * PCI_P2PDMA_BUS_ADDR: Indicates that two devices can talk to - * each other directly through a PCI switch and the transaction will - * not traverse the host bridge. Such a mapping should program - * the DMA engine with PCI bus addresses. - */ - PCI_P2PDMA_MAP_BUS_ADDR, - - /* - * PCI_P2PDMA_MAP_THRU_HOST_BRIDGE: Indicates two devices can talk - * to each other, but the transaction traverses a host bridge on the - * allowlist. In this case, a normal mapping either with CPU physical - * addresses (in the case of dma-direct) or IOVA addresses (in the - * case of IOMMUs) should be used to program the DMA engine. - */ - PCI_P2PDMA_MAP_THRU_HOST_BRIDGE, -}; - -struct pci_p2pdma_map_state { - struct dev_pagemap *pgmap; - enum pci_p2pdma_map_type map; - u64 bus_off; -}; - -/* helper for pci_p2pdma_state(), do not use directly */ -void __pci_p2pdma_update_state(struct pci_p2pdma_map_state *state, - struct device *dev, struct page *page); - -/** - * pci_p2pdma_state - check the P2P transfer state of a page - * @state: P2P state structure - * @dev: device to transfer to/from - * @page: page to map - * - * Check if @page is a PCI P2PDMA page, and if yes of what kind. Returns = the - * map type, and updates @state with all information needed for a P2P tran= sfer. - */ -static inline enum pci_p2pdma_map_type -pci_p2pdma_state(struct pci_p2pdma_map_state *state, struct device *dev, - struct page *page) -{ - if (IS_ENABLED(CONFIG_PCI_P2PDMA) && is_pci_p2pdma_page(page)) { - if (state->pgmap !=3D page_pgmap(page)) - __pci_p2pdma_update_state(state, dev, page); - return state->map; - } - return PCI_P2PDMA_MAP_NONE; -} - -/** - * pci_p2pdma_bus_addr_map - map a PCI_P2PDMA_MAP_BUS_ADDR P2P transfer - * @state: P2P state structure - * @paddr: physical address to map - * - * Map a physically contiguous PCI_P2PDMA_MAP_BUS_ADDR transfer. - */ -static inline dma_addr_t -pci_p2pdma_bus_addr_map(struct pci_p2pdma_map_state *state, phys_addr_t pa= ddr) -{ - WARN_ON_ONCE(state->map !=3D PCI_P2PDMA_MAP_BUS_ADDR); - return paddr + state->bus_off; -} - #endif /* _LINUX_DMA_MAP_OPS_H */ diff --git a/include/linux/pci-p2pdma.h b/include/linux/pci-p2pdma.h index 2c07aa6b7665..075c20b161d9 100644 --- a/include/linux/pci-p2pdma.h +++ b/include/linux/pci-p2pdma.h @@ -104,4 +104,89 @@ static inline struct pci_dev *pci_p2pmem_find(struct d= evice *client) return pci_p2pmem_find_many(&client, 1); } =20 +enum pci_p2pdma_map_type { + /* + * PCI_P2PDMA_MAP_UNKNOWN: Used internally as an initial state before + * the mapping type has been calculated. Exported routines for the API + * will never return this value. + */ + PCI_P2PDMA_MAP_UNKNOWN =3D 0, + + /* + * Not a PCI P2PDMA transfer. + */ + PCI_P2PDMA_MAP_NONE, + + /* + * PCI_P2PDMA_MAP_NOT_SUPPORTED: Indicates the transaction will + * traverse the host bridge and the host bridge is not in the + * allowlist. DMA Mapping routines should return an error when + * this is returned. + */ + PCI_P2PDMA_MAP_NOT_SUPPORTED, + + /* + * PCI_P2PDMA_MAP_BUS_ADDR: Indicates that two devices can talk to + * each other directly through a PCI switch and the transaction will + * not traverse the host bridge. Such a mapping should program + * the DMA engine with PCI bus addresses. + */ + PCI_P2PDMA_MAP_BUS_ADDR, + + /* + * PCI_P2PDMA_MAP_THRU_HOST_BRIDGE: Indicates two devices can talk + * to each other, but the transaction traverses a host bridge on the + * allowlist. In this case, a normal mapping either with CPU physical + * addresses (in the case of dma-direct) or IOVA addresses (in the + * case of IOMMUs) should be used to program the DMA engine. + */ + PCI_P2PDMA_MAP_THRU_HOST_BRIDGE, +}; + +struct pci_p2pdma_map_state { + struct dev_pagemap *pgmap; + enum pci_p2pdma_map_type map; + u64 bus_off; +}; + +/* helper for pci_p2pdma_state(), do not use directly */ +void __pci_p2pdma_update_state(struct pci_p2pdma_map_state *state, + struct device *dev, struct page *page); + +/** + * pci_p2pdma_state - check the P2P transfer state of a page + * @state: P2P state structure + * @dev: device to transfer to/from + * @page: page to map + * + * Check if @page is a PCI P2PDMA page, and if yes of what kind. Returns = the + * map type, and updates @state with all information needed for a P2P tran= sfer. + */ +static inline enum pci_p2pdma_map_type +pci_p2pdma_state(struct pci_p2pdma_map_state *state, struct device *dev, + struct page *page) +{ + if (IS_ENABLED(CONFIG_PCI_P2PDMA) && is_pci_p2pdma_page(page)) { + if (state->pgmap !=3D page_pgmap(page)) + __pci_p2pdma_update_state(state, dev, page); + return state->map; + } + return PCI_P2PDMA_MAP_NONE; +} + +/** + * pci_p2pdma_bus_addr_map - Translate a physical address to a bus address + * for a PCI_P2PDMA_MAP_BUS_ADDR transfer. + * @state: P2P state structure + * @paddr: physical address to map + * + * Map a physically contiguous PCI_P2PDMA_MAP_BUS_ADDR transfer. + */ +static inline dma_addr_t +pci_p2pdma_bus_addr_map(struct pci_p2pdma_map_state *state, phys_addr_t pa= ddr) +{ + WARN_ON_ONCE(state->map !=3D PCI_P2PDMA_MAP_BUS_ADDR); + return paddr + state->bus_off; +} + #endif /* _LINUX_PCI_P2P_H */ diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c index cec43cd5ed62..24c359d9c879 100644 --- a/kernel/dma/direct.c +++ b/kernel/dma/direct.c @@ -13,6 +13,7 @@ #include #include #include +#include #include "direct.h" =20 /* --=20 2.49.0 From nobody Tue Dec 16 14:20:07 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D3C6F25F7BC; Mon, 28 Apr 2025 09:23:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745832183; cv=none; b=AP6fG0AOP6KJZvD/U+2Da0Cki4OKmDhWZ3K0lHhQWbGw2gKYld8Khj8533qrhOitIxfMT7Rw92qyOvQO1zF1mAslGaOlEjTkUBthnM4xp3PxVb3Du2JobiZgp+1UGxrsbALDTce5e71Im0qH/Hgfx1T9G5a9Q3C4ttZ7al9hV+c= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745832183; c=relaxed/simple; bh=NSjvHXdN9Auhws02PZWMgoFw38+PyYsmwJSV2Mf4l+E=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=RyTqPogcXK3+V/ou3pSBfuBPfgykxbYZapLz0NGEJ/x6X9HB46gixx84kP7XxXeo1c6axDV9u/INR33KOe4gvq3Vi0alYdigvCjSSY7Aku9ypjfdAE/pHc+3T7eRrHSmGmCsp2bxeyCjh61UPBEm2xrOyDDTTPmeYRXLZ+aD75w= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=GkJSRLTH; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="GkJSRLTH" Received: by smtp.kernel.org (Postfix) with ESMTPSA id CE1CFC4CEE9; Mon, 28 Apr 2025 09:23:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1745832182; bh=NSjvHXdN9Auhws02PZWMgoFw38+PyYsmwJSV2Mf4l+E=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=GkJSRLTHZU22u1UhQ67MPM9g63S7XvmwGzUbJuzqOOoixyYAyVfnA04t4wkZADAmT Kp8RGfRAN5KkYQzvIv85m/MuedCEbIZLkadhGViQy0IoLd+Ybez+IHCgO2/uSYjZND pr/fjH+IU4RZjijmZEtuma8QCzj/V5qs8RmmPj14QRRr0zIjZpIQy4fJvwiCfDn3hb LHesU6Nn6nX3ydTqwP9xqMbEkv2CKn08JumCY+lk4i/fp1QJciqX9Y3b0H4KtHsEAv OgC6Gcr8vk1MCui06+shS3RvU4kjjpK+rarol/LeBfDsV+Z4blbG4gtyfxLsC6nX+g pwpQ+Io0zJd5w== From: Leon Romanovsky To: Marek Szyprowski , Jens Axboe , Christoph Hellwig , Keith Busch Cc: Jake Edge , Jonathan Corbet , Jason Gunthorpe , Zhu Yanjun , Robin Murphy , Joerg Roedel , Will Deacon , Sagi Grimberg , Bjorn Helgaas , Logan Gunthorpe , Yishai Hadas , Shameer Kolothum , Kevin Tian , Alex Williamson , =?UTF-8?q?J=C3=A9r=C3=B4me=20Glisse?= , Andrew Morton , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-rdma@vger.kernel.org, iommu@lists.linux.dev, linux-nvme@lists.infradead.org, linux-pci@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org, Niklas Schnelle , Chuck Lever , Luis Chamberlain , Matthew Wilcox , Dan Williams , Kanchan Joshi , Chaitanya Kulkarni , Jason Gunthorpe , Leon Romanovsky Subject: [PATCH v10 03/24] iommu: generalize the batched sync after map interface Date: Mon, 28 Apr 2025 12:22:09 +0300 Message-ID: <69da19d2cc5df0be5112f0cf2365a0337b00d873.1745831017.git.leon@kernel.org> X-Mailer: git-send-email 2.49.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Christoph Hellwig For the upcoming IOVA-based DMA API we want to batch the ops->iotlb_sync_map() call after mapping multiple IOVAs from dma-iommu without having a scatterlist. Improve the API. Add a wrapper for the map_sync as iommu_sync_map() so that callers don't need to poke into the methods directly. Formalize __iommu_map() into iommu_map_nosync() which requires the caller to call iommu_sync_map() after all maps are completed. Refactor the existing sanity checks from all the different layers into iommu_map_nosync(). Signed-off-by: Christoph Hellwig Acked-by: Will Deacon Tested-by: Jens Axboe Reviewed-by: Jason Gunthorpe Reviewed-by: Luis Chamberlain Signed-off-by: Leon Romanovsky --- drivers/iommu/iommu.c | 65 +++++++++++++++++++------------------------ include/linux/iommu.h | 4 +++ 2 files changed, 33 insertions(+), 36 deletions(-) diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c index 4f91a740c15f..02960585b8d4 100644 --- a/drivers/iommu/iommu.c +++ b/drivers/iommu/iommu.c @@ -2443,8 +2443,8 @@ static size_t iommu_pgsize(struct iommu_domain *domai= n, unsigned long iova, return pgsize; } =20 -static int __iommu_map(struct iommu_domain *domain, unsigned long iova, - phys_addr_t paddr, size_t size, int prot, gfp_t gfp) +int iommu_map_nosync(struct iommu_domain *domain, unsigned long iova, + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) { const struct iommu_domain_ops *ops =3D domain->ops; unsigned long orig_iova =3D iova; @@ -2453,12 +2453,19 @@ static int __iommu_map(struct iommu_domain *domain,= unsigned long iova, phys_addr_t orig_paddr =3D paddr; int ret =3D 0; =20 + might_sleep_if(gfpflags_allow_blocking(gfp)); + if (unlikely(!(domain->type & __IOMMU_DOMAIN_PAGING))) return -EINVAL; =20 if (WARN_ON(!ops->map_pages || domain->pgsize_bitmap =3D=3D 0UL)) return -ENODEV; =20 + /* Discourage passing strange GFP flags */ + if (WARN_ON_ONCE(gfp & (__GFP_COMP | __GFP_DMA | __GFP_DMA32 | + __GFP_HIGHMEM))) + return -EINVAL; + /* find out the minimum page size supported */ min_pagesz =3D 1 << __ffs(domain->pgsize_bitmap); =20 @@ -2506,31 +2513,27 @@ static int __iommu_map(struct iommu_domain *domain,= unsigned long iova, return ret; } =20 -int iommu_map(struct iommu_domain *domain, unsigned long iova, - phys_addr_t paddr, size_t size, int prot, gfp_t gfp) +int iommu_sync_map(struct iommu_domain *domain, unsigned long iova, size_t= size) { const struct iommu_domain_ops *ops =3D domain->ops; - int ret; - - might_sleep_if(gfpflags_allow_blocking(gfp)); =20 - /* Discourage passing strange GFP flags */ - if (WARN_ON_ONCE(gfp & (__GFP_COMP | __GFP_DMA | __GFP_DMA32 | - __GFP_HIGHMEM))) - return -EINVAL; + if (!ops->iotlb_sync_map) + return 0; + return ops->iotlb_sync_map(domain, iova, size); +} =20 - ret =3D __iommu_map(domain, iova, paddr, size, prot, gfp); - if (ret =3D=3D 0 && ops->iotlb_sync_map) { - ret =3D ops->iotlb_sync_map(domain, iova, size); - if (ret) - goto out_err; - } +int iommu_map(struct iommu_domain *domain, unsigned long iova, + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) +{ + int ret; =20 - return ret; + ret =3D iommu_map_nosync(domain, iova, paddr, size, prot, gfp); + if (ret) + return ret; =20 -out_err: - /* undo mappings already done */ - iommu_unmap(domain, iova, size); + ret =3D iommu_sync_map(domain, iova, size); + if (ret) + iommu_unmap(domain, iova, size); =20 return ret; } @@ -2630,26 +2633,17 @@ ssize_t iommu_map_sg(struct iommu_domain *domain, u= nsigned long iova, struct scatterlist *sg, unsigned int nents, int prot, gfp_t gfp) { - const struct iommu_domain_ops *ops =3D domain->ops; size_t len =3D 0, mapped =3D 0; phys_addr_t start; unsigned int i =3D 0; int ret; =20 - might_sleep_if(gfpflags_allow_blocking(gfp)); - - /* Discourage passing strange GFP flags */ - if (WARN_ON_ONCE(gfp & (__GFP_COMP | __GFP_DMA | __GFP_DMA32 | - __GFP_HIGHMEM))) - return -EINVAL; - while (i <=3D nents) { phys_addr_t s_phys =3D sg_phys(sg); =20 if (len && s_phys !=3D start + len) { - ret =3D __iommu_map(domain, iova + mapped, start, + ret =3D iommu_map_nosync(domain, iova + mapped, start, len, prot, gfp); - if (ret) goto out_err; =20 @@ -2672,11 +2666,10 @@ ssize_t iommu_map_sg(struct iommu_domain *domain, u= nsigned long iova, sg =3D sg_next(sg); } =20 - if (ops->iotlb_sync_map) { - ret =3D ops->iotlb_sync_map(domain, iova, mapped); - if (ret) - goto out_err; - } + ret =3D iommu_sync_map(domain, iova, mapped); + if (ret) + goto out_err; + return mapped; =20 out_err: diff --git a/include/linux/iommu.h b/include/linux/iommu.h index ccce8a751e2a..ce472af8e9c3 100644 --- a/include/linux/iommu.h +++ b/include/linux/iommu.h @@ -872,6 +872,10 @@ extern struct iommu_domain *iommu_get_domain_for_dev(s= truct device *dev); extern struct iommu_domain *iommu_get_dma_domain(struct device *dev); extern int iommu_map(struct iommu_domain *domain, unsigned long iova, phys_addr_t paddr, size_t size, int prot, gfp_t gfp); +int iommu_map_nosync(struct iommu_domain *domain, unsigned long iova, + phys_addr_t paddr, size_t size, int prot, gfp_t gfp); +int iommu_sync_map(struct iommu_domain *domain, unsigned long iova, + size_t size); extern size_t iommu_unmap(struct iommu_domain *domain, unsigned long iova, size_t size); extern size_t iommu_unmap_fast(struct iommu_domain *domain, --=20 2.49.0 From nobody Tue Dec 16 14:20:07 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AA15B25F7AC; Mon, 28 Apr 2025 09:22:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745832178; cv=none; b=Gk3QjsoEyBYTn+sp1MslZ+QLK5Mcfd2tSPhVXguJag4PbOtjk9oRyURpF9Gu6sW4MZbbhr+brgmgfHjPJTI1TfEmojqOeugOl/Dns19ptVYPIpsz+heHLFEilGndZlRfNdNF0qkx/DPZsX9Hm3N3ukIc7RnZ6WU0zWMganvLtJk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745832178; c=relaxed/simple; bh=qAutkCz37Tu8LslKD8JBH5oib5UnssVkdsFDkEx7ZAg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=tE27ItbwDYuRCXigwSUpb1io8uWgkx8fILo6Gc+ZQSUriYL5q2FPsSXvprMmAiTKmk3Fjg4g93+KrSzcoGcnScCJXzz0QYaZ6k54VOHw7H6UrRH3TXKDmU9VB/avu10BcPLnMgsY7fXBoS9CzjDjwC7URjqsZVZpHbD4ndSN3bU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=AOOf6Y0m; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="AOOf6Y0m" Received: by smtp.kernel.org (Postfix) with ESMTPSA id D89BCC4CEE9; Mon, 28 Apr 2025 09:22:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1745832178; bh=qAutkCz37Tu8LslKD8JBH5oib5UnssVkdsFDkEx7ZAg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=AOOf6Y0mFh8eaLlqUYaStweSkz4DuWDmM5g52GCOkAPYc1JLbqg3lvf3YSZedB5hq I7YPao5jZ7PVdb5yLDFgUlyfFUAdsvYrD2O+N1GQL1ugH3GyJMizhP+2zRdTDutTtm 6Arc5CR8SWcEOS618cLmUutubaOWsAUZ5S4w8pnYyGc8EOww19gd9qY57pmnTY3446 rDQDtBCTbwMHdkeWYFbIL0Q2+lz4lvUQC5JkKCvPRuSzjFCPq/I4yB/O49lx3fBh5+ nC3MNGBucQEWuM+Rb3Ho6G75E6IdHFTfEgvjbUMPUx4zAeeNoTPV8AE+YhZUqvYG++ 2+GCFe1W9EuRw== From: Leon Romanovsky To: Marek Szyprowski , Jens Axboe , Christoph Hellwig , Keith Busch Cc: Leon Romanovsky , Jake Edge , Jonathan Corbet , Jason Gunthorpe , Zhu Yanjun , Robin Murphy , Joerg Roedel , Will Deacon , Sagi Grimberg , Bjorn Helgaas , Logan Gunthorpe , Yishai Hadas , Shameer Kolothum , Kevin Tian , Alex Williamson , =?UTF-8?q?J=C3=A9r=C3=B4me=20Glisse?= , Andrew Morton , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-rdma@vger.kernel.org, iommu@lists.linux.dev, linux-nvme@lists.infradead.org, linux-pci@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org, Niklas Schnelle , Chuck Lever , Luis Chamberlain , Matthew Wilcox , Dan Williams , Kanchan Joshi , Chaitanya Kulkarni , Jason Gunthorpe Subject: [PATCH v10 04/24] iommu: add kernel-doc for iommu_unmap_fast Date: Mon, 28 Apr 2025 12:22:10 +0300 Message-ID: <6c4bbb539bec7b827b9e9cc24779c9e9c43fc3ed.1745831017.git.leon@kernel.org> X-Mailer: git-send-email 2.49.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Leon Romanovsky Add kernel-doc section for iommu_unmap_fast to document existing limitation of underlying functions which can't split individual ranges. Suggested-by: Jason Gunthorpe Acked-by: Will Deacon Reviewed-by: Christoph Hellwig Tested-by: Jens Axboe Reviewed-by: Jason Gunthorpe Reviewed-by: Luis Chamberlain Signed-off-by: Leon Romanovsky Reviewed-by: Lu Baolu --- drivers/iommu/iommu.c | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c index 02960585b8d4..8619c355ef9c 100644 --- a/drivers/iommu/iommu.c +++ b/drivers/iommu/iommu.c @@ -2621,6 +2621,25 @@ size_t iommu_unmap(struct iommu_domain *domain, } EXPORT_SYMBOL_GPL(iommu_unmap); =20 +/** + * iommu_unmap_fast() - Remove mappings from a range of IOVA without IOTLB= sync + * @domain: Domain to manipulate + * @iova: IO virtual address to start + * @size: Length of the range starting from @iova + * @iotlb_gather: range information for a pending IOTLB flush + * + * iommu_unmap_fast() will remove a translation created by iommu_map(). + * It can't subdivide a mapping created by iommu_map(), so it should be + * called with IOVA ranges that match what was passed to iommu_map(). The + * range can aggregate contiguous iommu_map() calls so long as no individu= al + * range is split. + * + * Basically iommu_unmap_fast() is the same as iommu_unmap() but for calle= rs + * which manage the IOTLB flushing externally to perform a batched sync. + * + * Returns: Number of bytes of IOVA unmapped. iova + res will be the point + * unmapping stopped. + */ size_t iommu_unmap_fast(struct iommu_domain *domain, unsigned long iova, size_t size, struct iommu_iotlb_gather *iotlb_gather) --=20 2.49.0 From nobody Tue Dec 16 14:20:07 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AF3B8267B67; Mon, 28 Apr 2025 09:23:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745832196; cv=none; b=ifgvHmNIeSxEsy4fLdlw9oGw59f8Xq0HoRaJW/qBUyozUNHHrhQavK8XstHQ8Rk8yP0Pc0JptzrDj5UsPrYnnmly4M1RUXAjPZTfzdzObdsDQxuFb3JLqImMr2/ubQwwFmJBJgkB4udG8/p13Bskx1D58Y2RbxJgpgVl7Ngkz7A= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745832196; c=relaxed/simple; bh=7JZyUOUgfxHnGGG2HNKQMfblPgeeA2+t6VQ5mlFxHys=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=NHctt8R9A7gHtXAGXrQ09V/HFVix8rriC7akOeUwIUssYO1TtDFODuv5mupJrbe6LSXg4H4wM8IwSTTItBAM/PXav8K6ansbL3n6JwHFvMeaCAKcAWCPsvQNvZtEnPBd+d2M82Tm/qPB2HQ1r+0E27Rz9NEEm8YdhX2VQH0U5lk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=GfCwVVwd; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="GfCwVVwd" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 793B3C4CEF0; Mon, 28 Apr 2025 09:23:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1745832195; bh=7JZyUOUgfxHnGGG2HNKQMfblPgeeA2+t6VQ5mlFxHys=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=GfCwVVwdyFZmy+bRtQCIHsemKrBs8wbIR2UqsztY5GMF0OIm6MavtPECFiD+SMVDI +a1etzFYrSBq/hrx15Z7WCLjK2G9v22HGYTe8UdZ3YqH/HDl+rz3uDoylfAlaZwt2u 2YY3mUvm1+pvHmj8SEsNj9VVcNA8rvqSDjaWnOkYBdGB7UdEf7U92TRCOmt7Lwp+Bn M989LV/ZRjL+WWCZwb5dI+P4iU/d2OGxVUfpkp0gPNylAN0RBbXVeewsgljmoc+1Ex XZPoE/bywuEsRDcl36Jn1nkR2Ws743Ze2Qn4zXZvtW2lGFPAPZrqEjsCHzhWFmhkZu F0KhMRfOO3NFw== From: Leon Romanovsky To: Marek Szyprowski , Jens Axboe , Christoph Hellwig , Keith Busch Cc: Leon Romanovsky , Jake Edge , Jonathan Corbet , Jason Gunthorpe , Zhu Yanjun , Robin Murphy , Joerg Roedel , Will Deacon , Sagi Grimberg , Bjorn Helgaas , Logan Gunthorpe , Yishai Hadas , Shameer Kolothum , Kevin Tian , Alex Williamson , =?UTF-8?q?J=C3=A9r=C3=B4me=20Glisse?= , Andrew Morton , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-rdma@vger.kernel.org, iommu@lists.linux.dev, linux-nvme@lists.infradead.org, linux-pci@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org, Niklas Schnelle , Chuck Lever , Luis Chamberlain , Matthew Wilcox , Dan Williams , Kanchan Joshi , Chaitanya Kulkarni Subject: [PATCH v10 05/24] dma-mapping: Provide an interface to allow allocate IOVA Date: Mon, 28 Apr 2025 12:22:11 +0300 Message-ID: <30f0601d400711b3859deeb8fef3090f5b2020a4.1745831017.git.leon@kernel.org> X-Mailer: git-send-email 2.49.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Leon Romanovsky The existing .map_page() callback provides both allocating of IOVA and linking DMA pages. That combination works great for most of the callers who use it in control paths, but is less effective in fast paths where there may be multiple calls to map_page(). These advanced callers already manage their data in some sort of database and can perform IOVA allocation in advance, leaving range linkage operation to be in fast path. Provide an interface to allocate/deallocate IOVA and next patch link/unlink DMA ranges to that specific IOVA. In the new API a DMA mapping transaction is identified by a struct dma_iova_state, which holds some recomputed information for the transaction which does not change for each page being mapped, so add a check if IOVA can be used for the specific transaction. The API is exported from dma-iommu as it is the only implementation supported, the namespace is clearly different from iommu_* functions which are not allowed to be used. This code layout allows us to save function call per API call used in datapath as well as a lot of boilerplate code. Reviewed-by: Christoph Hellwig Tested-by: Jens Axboe Reviewed-by: Luis Chamberlain Signed-off-by: Leon Romanovsky --- drivers/iommu/dma-iommu.c | 86 +++++++++++++++++++++++++++++++++++++ include/linux/dma-mapping.h | 48 +++++++++++++++++++++ 2 files changed, 134 insertions(+) diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index 9ba8d8bc0ce9..d3211a8d755e 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -1723,6 +1723,92 @@ size_t iommu_dma_max_mapping_size(struct device *dev) return SIZE_MAX; } =20 +/** + * dma_iova_try_alloc - Try to allocate an IOVA space + * @dev: Device to allocate the IOVA space for + * @state: IOVA state + * @phys: physical address + * @size: IOVA size + * + * Check if @dev supports the IOVA-based DMA API, and if yes allocate IOVA= space + * for the given base address and size. + * + * Note: @phys is only used to calculate the IOVA alignment. Callers that = always + * do PAGE_SIZE aligned transfers can safely pass 0 here. + * + * Returns %true if the IOVA-based DMA API can be used and IOVA space has = been + * allocated, or %false if the regular DMA API should be used. + */ +bool dma_iova_try_alloc(struct device *dev, struct dma_iova_state *state, + phys_addr_t phys, size_t size) +{ + struct iommu_dma_cookie *cookie; + struct iommu_domain *domain; + struct iova_domain *iovad; + size_t iova_off; + dma_addr_t addr; + + memset(state, 0, sizeof(*state)); + if (!use_dma_iommu(dev)) + return false; + + domain =3D iommu_get_dma_domain(dev); + cookie =3D domain->iova_cookie; + iovad =3D &cookie->iovad; + iova_off =3D iova_offset(iovad, phys); + + if (static_branch_unlikely(&iommu_deferred_attach_enabled) && + iommu_deferred_attach(dev, iommu_get_domain_for_dev(dev))) + return false; + + if (WARN_ON_ONCE(!size)) + return false; + + /* + * DMA_IOVA_USE_SWIOTLB is flag which is set by dma-iommu + * internals, make sure that caller didn't set it and/or + * didn't use this interface to map SIZE_MAX. + */ + if (WARN_ON_ONCE((u64)size & DMA_IOVA_USE_SWIOTLB)) + return false; + + addr =3D iommu_dma_alloc_iova(domain, + iova_align(iovad, size + iova_off), + dma_get_mask(dev), dev); + if (!addr) + return false; + + state->addr =3D addr + iova_off; + state->__size =3D size; + return true; +} +EXPORT_SYMBOL_GPL(dma_iova_try_alloc); + +/** + * dma_iova_free - Free an IOVA space + * @dev: Device to free the IOVA space for + * @state: IOVA state + * + * Undoes a successful dma_try_iova_alloc(). + * + * Note that all dma_iova_link() calls need to be undone first. For calle= rs + * that never call dma_iova_unlink(), dma_iova_destroy() can be used inste= ad + * which unlinks all ranges and frees the IOVA space in a single efficient + * operation. + */ +void dma_iova_free(struct device *dev, struct dma_iova_state *state) +{ + struct iommu_domain *domain =3D iommu_get_dma_domain(dev); + struct iommu_dma_cookie *cookie =3D domain->iova_cookie; + struct iova_domain *iovad =3D &cookie->iovad; + size_t iova_start_pad =3D iova_offset(iovad, state->addr); + size_t size =3D dma_iova_size(state); + + iommu_dma_free_iova(domain, state->addr - iova_start_pad, + iova_align(iovad, size + iova_start_pad), NULL); +} +EXPORT_SYMBOL_GPL(dma_iova_free); + void iommu_setup_dma_ops(struct device *dev) { struct iommu_domain *domain =3D iommu_get_domain_for_dev(dev); diff --git a/include/linux/dma-mapping.h b/include/linux/dma-mapping.h index b79925b1c433..de7f73810d54 100644 --- a/include/linux/dma-mapping.h +++ b/include/linux/dma-mapping.h @@ -72,6 +72,22 @@ =20 #define DMA_BIT_MASK(n) (((n) =3D=3D 64) ? ~0ULL : ((1ULL<<(n))-1)) =20 +struct dma_iova_state { + dma_addr_t addr; + u64 __size; +}; + +/* + * Use the high bit to mark if we used swiotlb for one or more ranges. + */ +#define DMA_IOVA_USE_SWIOTLB (1ULL << 63) + +static inline size_t dma_iova_size(struct dma_iova_state *state) +{ + /* Casting is needed for 32-bits systems */ + return (size_t)(state->__size & ~DMA_IOVA_USE_SWIOTLB); +} + #ifdef CONFIG_DMA_API_DEBUG void debug_dma_mapping_error(struct device *dev, dma_addr_t dma_addr); void debug_dma_map_single(struct device *dev, const void *addr, @@ -277,6 +293,38 @@ static inline int dma_mmap_noncontiguous(struct device= *dev, } #endif /* CONFIG_HAS_DMA */ =20 +#ifdef CONFIG_IOMMU_DMA +/** + * dma_use_iova - check if the IOVA API is used for this state + * @state: IOVA state + * + * Return %true if the DMA transfers uses the dma_iova_*() calls or %false= if + * they can't be used. + */ +static inline bool dma_use_iova(struct dma_iova_state *state) +{ + return state->__size !=3D 0; +} + +bool dma_iova_try_alloc(struct device *dev, struct dma_iova_state *state, + phys_addr_t phys, size_t size); +void dma_iova_free(struct device *dev, struct dma_iova_state *state); +#else /* CONFIG_IOMMU_DMA */ +static inline bool dma_use_iova(struct dma_iova_state *state) +{ + return false; +} +static inline bool dma_iova_try_alloc(struct device *dev, + struct dma_iova_state *state, phys_addr_t phys, size_t size) +{ + return false; +} +static inline void dma_iova_free(struct device *dev, + struct dma_iova_state *state) +{ +} +#endif /* CONFIG_IOMMU_DMA */ + #if defined(CONFIG_HAS_DMA) && defined(CONFIG_DMA_NEED_SYNC) void __dma_sync_single_for_cpu(struct device *dev, dma_addr_t addr, size_t= size, enum dma_data_direction dir); --=20 2.49.0 From nobody Tue Dec 16 14:20:07 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1E0E825F96B; Mon, 28 Apr 2025 09:23:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745832187; cv=none; b=NnYUc9WWkUM+e5dPgI0MPW1AWXLNKwFNmPGfOlM6W8X41KwFNmndeyL7eICUezpo9VYhVdHbhfPPHyoNgIGOnBo3yfgb04aXFWTQk+xU4m+jGZ3a4Mt16Gcx8cFKF10qaC3GAfEp/A5TT6RYeXSHfJfTjSzTPEhZ6W6BzTRF/VY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745832187; c=relaxed/simple; bh=0JHkJ3yc+gAWUe6Cn00MEeufheTpl4eLgTynVx9oj5I=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=az+WjG+QnC48y9rkkFq0Hn0o/Cw1WIEr3e4njZF4RQxKP0hO7w+vFLe8Qtw/BPrSeRHiRKdeqfEPVper13xkQDKG1oc/SAu7xH0DmQrxU+zOrvw4ecDE9urtbzBxomPHLLg0W2TbAQ0qowOjkq7HhWXhGp2kWhajgj55vmjnHkI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=PqmGfJdc; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="PqmGfJdc" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1B7CFC4CEE4; Mon, 28 Apr 2025 09:23:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1745832187; bh=0JHkJ3yc+gAWUe6Cn00MEeufheTpl4eLgTynVx9oj5I=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=PqmGfJdcNXSWtK9ZUr/VZuQVATSjN4FCICu9QvOjQ5DFimCvpDJNBXHHRfXKE4d0f JSfdtg+/yURTLqlGkkjWhIP/+bLkaey8ndkjjCWv7Pmda9drOgqolG5sO0/xK8phy1 M1bamGvI/HUYk59FrSxjJodLMEVZQAN0v0cnSkOGKPCsab0uBqMn4YH97NLy505u+y NPLiCd86bDRwc6qTWqjtnq+cD0ihAbz1ZorWks4/3uOYAZsA5mAb9TFncOXGZM5Gnj IOohfCcCDYKXmzfgsFUUlIffxZ68XVFQiK2Q4uEACJ73pihFUgrcFqu4MHI/n1tHxp roqnmUkSZ/rbw== From: Leon Romanovsky To: Marek Szyprowski , Jens Axboe , Christoph Hellwig , Keith Busch Cc: Jake Edge , Jonathan Corbet , Jason Gunthorpe , Zhu Yanjun , Robin Murphy , Joerg Roedel , Will Deacon , Sagi Grimberg , Bjorn Helgaas , Logan Gunthorpe , Yishai Hadas , Shameer Kolothum , Kevin Tian , Alex Williamson , =?UTF-8?q?J=C3=A9r=C3=B4me=20Glisse?= , Andrew Morton , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-rdma@vger.kernel.org, iommu@lists.linux.dev, linux-nvme@lists.infradead.org, linux-pci@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org, Niklas Schnelle , Chuck Lever , Luis Chamberlain , Matthew Wilcox , Dan Williams , Kanchan Joshi , Chaitanya Kulkarni , Leon Romanovsky Subject: [PATCH v10 06/24] iommu/dma: Factor out a iommu_dma_map_swiotlb helper Date: Mon, 28 Apr 2025 12:22:12 +0300 Message-ID: X-Mailer: git-send-email 2.49.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Christoph Hellwig Split the iommu logic from iommu_dma_map_page into a separate helper. This not only keeps the code neatly separated, but will also allow for reuse in another caller. Signed-off-by: Christoph Hellwig Tested-by: Jens Axboe Reviewed-by: Luis Chamberlain Signed-off-by: Leon Romanovsky Reviewed-by: Lu Baolu --- drivers/iommu/dma-iommu.c | 73 ++++++++++++++++++++++----------------- 1 file changed, 41 insertions(+), 32 deletions(-) diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index d3211a8d755e..d7684024c439 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -1138,6 +1138,43 @@ void iommu_dma_sync_sg_for_device(struct device *dev= , struct scatterlist *sgl, arch_sync_dma_for_device(sg_phys(sg), sg->length, dir); } =20 +static phys_addr_t iommu_dma_map_swiotlb(struct device *dev, phys_addr_t p= hys, + size_t size, enum dma_data_direction dir, unsigned long attrs) +{ + struct iommu_domain *domain =3D iommu_get_dma_domain(dev); + struct iova_domain *iovad =3D &domain->iova_cookie->iovad; + + if (!is_swiotlb_active(dev)) { + dev_warn_once(dev, "DMA bounce buffers are inactive, unable to map unali= gned transaction.\n"); + return (phys_addr_t)DMA_MAPPING_ERROR; + } + + trace_swiotlb_bounced(dev, phys, size); + + phys =3D swiotlb_tbl_map_single(dev, phys, size, iova_mask(iovad), dir, + attrs); + + /* + * Untrusted devices should not see padding areas with random leftover + * kernel data, so zero the pre- and post-padding. + * swiotlb_tbl_map_single() has initialized the bounce buffer proper to + * the contents of the original memory buffer. + */ + if (phys !=3D (phys_addr_t)DMA_MAPPING_ERROR && dev_is_untrusted(dev)) { + size_t start, virt =3D (size_t)phys_to_virt(phys); + + /* Pre-padding */ + start =3D iova_align_down(iovad, virt); + memset((void *)start, 0, virt - start); + + /* Post-padding */ + start =3D virt + size; + memset((void *)start, 0, iova_align(iovad, start) - start); + } + + return phys; +} + dma_addr_t iommu_dma_map_page(struct device *dev, struct page *page, unsigned long offset, size_t size, enum dma_data_direction dir, unsigned long attrs) @@ -1151,42 +1188,14 @@ dma_addr_t iommu_dma_map_page(struct device *dev, s= truct page *page, dma_addr_t iova, dma_mask =3D dma_get_mask(dev); =20 /* - * If both the physical buffer start address and size are - * page aligned, we don't need to use a bounce page. + * If both the physical buffer start address and size are page aligned, + * we don't need to use a bounce page. */ if (dev_use_swiotlb(dev, size, dir) && iova_offset(iovad, phys | size)) { - if (!is_swiotlb_active(dev)) { - dev_warn_once(dev, "DMA bounce buffers are inactive, unable to map unal= igned transaction.\n"); - return DMA_MAPPING_ERROR; - } - - trace_swiotlb_bounced(dev, phys, size); - - phys =3D swiotlb_tbl_map_single(dev, phys, size, - iova_mask(iovad), dir, attrs); - - if (phys =3D=3D DMA_MAPPING_ERROR) + phys =3D iommu_dma_map_swiotlb(dev, phys, size, dir, attrs); + if (phys =3D=3D (phys_addr_t)DMA_MAPPING_ERROR) return DMA_MAPPING_ERROR; - - /* - * Untrusted devices should not see padding areas with random - * leftover kernel data, so zero the pre- and post-padding. - * swiotlb_tbl_map_single() has initialized the bounce buffer - * proper to the contents of the original memory buffer. - */ - if (dev_is_untrusted(dev)) { - size_t start, virt =3D (size_t)phys_to_virt(phys); - - /* Pre-padding */ - start =3D iova_align_down(iovad, virt); - memset((void *)start, 0, virt - start); - - /* Post-padding */ - start =3D virt + size; - memset((void *)start, 0, - iova_align(iovad, start) - start); - } } =20 if (!coherent && !(attrs & DMA_ATTR_SKIP_CPU_SYNC)) --=20 2.49.0 From nobody Tue Dec 16 14:20:07 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B354B2676C9; Mon, 28 Apr 2025 09:23:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745832191; cv=none; b=ZNxMDQqrEJWtX/ECAyQ9PeT/VKoPn3FEvBWDYxOe0WfYSMkhhvmJLb3t7+bX2X8NyAW2nq8YPa/aXVd+QI/L1QTDyxUaoFcz2wX1NtcBDFKvjBIA5+LiZu7BloKwrBefWhUoL0w/ljk++JBTA9JJli756/8pLd1VXaZXrPhXIV8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745832191; c=relaxed/simple; bh=OqVY+n6v+7S7pyZo8PdJ2+LmT5/B6VKI0mwJdkJCFKg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=L2iuieKCnyfEE9BxPXJD/uBdNwMQ4z7H+rPAqDyTY6+5bql3njjtQV1Y+LZmsOaWY/8oypqoC5h6n7odQNHYAjbZNjGOM5wCNlPLlVknyutW2SWmRMKF5u0qUJEARB1DxjWMYFcgvWAnABw6Q+F6UGRdt7sUx3zJfJn6lFEs2SQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=X7F1+I1p; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="X7F1+I1p" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7C49CC4CEE4; Mon, 28 Apr 2025 09:23:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1745832191; bh=OqVY+n6v+7S7pyZo8PdJ2+LmT5/B6VKI0mwJdkJCFKg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=X7F1+I1pST5FU77ixPfRu2oeom3Lk8QA17F6CQewgN667Ss9gTq1u+YtQ5ZvdV4ZQ Tmh6txATJr+9/zAcpaBSPiESmHwQHSH9ZtSEuRegWrZl9+G9gqAYxbij02WSppWXAZ h3PGS/xzCWex8B/QYXhKUp/9iw3CN0hL9w01BG2eLBDgZ9hsxfA2kJxbZZIqTtFQ/L 4wSy5AZoyzFoAfovuPO0jkVr9HznsToFDTsWm8YxoKf6NZAx/dPVupv+flVxs/pM7G 5dnZvt2KkpxzmbsUVAqYIF1FsOKMi3H5lmhyPj2YtE1npfuDcXKu5PVQ91HNCM1V8K CqmJfiiihu/8w== From: Leon Romanovsky To: Marek Szyprowski , Jens Axboe , Christoph Hellwig , Keith Busch Cc: Leon Romanovsky , Jake Edge , Jonathan Corbet , Jason Gunthorpe , Zhu Yanjun , Robin Murphy , Joerg Roedel , Will Deacon , Sagi Grimberg , Bjorn Helgaas , Logan Gunthorpe , Yishai Hadas , Shameer Kolothum , Kevin Tian , Alex Williamson , =?UTF-8?q?J=C3=A9r=C3=B4me=20Glisse?= , Andrew Morton , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-rdma@vger.kernel.org, iommu@lists.linux.dev, linux-nvme@lists.infradead.org, linux-pci@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org, Niklas Schnelle , Chuck Lever , Luis Chamberlain , Matthew Wilcox , Dan Williams , Kanchan Joshi , Chaitanya Kulkarni Subject: [PATCH v10 07/24] dma-mapping: Implement link/unlink ranges API Date: Mon, 28 Apr 2025 12:22:13 +0300 Message-ID: <667d4e4602545172ab74c7e6d091fa7d706754fd.1745831017.git.leon@kernel.org> X-Mailer: git-send-email 2.49.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Leon Romanovsky Introduce new DMA APIs to perform DMA linkage of buffers in layers higher than DMA. In proposed API, the callers will perform the following steps. In map path: if (dma_can_use_iova(...)) dma_iova_alloc() for (page in range) dma_iova_link_next(...) dma_iova_sync(...) else /* Fallback to legacy map pages */ for (all pages) dma_map_page(...) In unmap path: if (dma_can_use_iova(...)) dma_iova_destroy() else for (all pages) dma_unmap_page(...) Reviewed-by: Christoph Hellwig Tested-by: Jens Axboe Reviewed-by: Luis Chamberlain Signed-off-by: Leon Romanovsky --- drivers/iommu/dma-iommu.c | 275 +++++++++++++++++++++++++++++++++++- include/linux/dma-mapping.h | 32 +++++ 2 files changed, 306 insertions(+), 1 deletion(-) diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index d7684024c439..98f7205ec8fb 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -1175,6 +1175,17 @@ static phys_addr_t iommu_dma_map_swiotlb(struct devi= ce *dev, phys_addr_t phys, return phys; } =20 +/* + * Checks if a physical buffer has unaligned boundaries with respect to + * the IOMMU granule. Returns non-zero if either the start or end + * address is not aligned to the granule boundary. + */ +static inline size_t iova_unaligned(struct iova_domain *iovad, phys_addr_t= phys, + size_t size) +{ + return iova_offset(iovad, phys | size); +} + dma_addr_t iommu_dma_map_page(struct device *dev, struct page *page, unsigned long offset, size_t size, enum dma_data_direction dir, unsigned long attrs) @@ -1192,7 +1203,7 @@ dma_addr_t iommu_dma_map_page(struct device *dev, str= uct page *page, * we don't need to use a bounce page. */ if (dev_use_swiotlb(dev, size, dir) && - iova_offset(iovad, phys | size)) { + iova_unaligned(iovad, phys, size)) { phys =3D iommu_dma_map_swiotlb(dev, phys, size, dir, attrs); if (phys =3D=3D (phys_addr_t)DMA_MAPPING_ERROR) return DMA_MAPPING_ERROR; @@ -1818,6 +1829,268 @@ void dma_iova_free(struct device *dev, struct dma_i= ova_state *state) } EXPORT_SYMBOL_GPL(dma_iova_free); =20 +static int __dma_iova_link(struct device *dev, dma_addr_t addr, + phys_addr_t phys, size_t size, enum dma_data_direction dir, + unsigned long attrs) +{ + bool coherent =3D dev_is_dma_coherent(dev); + + if (!coherent && !(attrs & DMA_ATTR_SKIP_CPU_SYNC)) + arch_sync_dma_for_device(phys, size, dir); + + return iommu_map_nosync(iommu_get_dma_domain(dev), addr, phys, size, + dma_info_to_prot(dir, coherent, attrs), GFP_ATOMIC); +} + +static int iommu_dma_iova_bounce_and_link(struct device *dev, dma_addr_t a= ddr, + phys_addr_t phys, size_t bounce_len, + enum dma_data_direction dir, unsigned long attrs, + size_t iova_start_pad) +{ + struct iommu_domain *domain =3D iommu_get_dma_domain(dev); + struct iova_domain *iovad =3D &domain->iova_cookie->iovad; + phys_addr_t bounce_phys; + int error; + + bounce_phys =3D iommu_dma_map_swiotlb(dev, phys, bounce_len, dir, attrs); + if (bounce_phys =3D=3D DMA_MAPPING_ERROR) + return -ENOMEM; + + error =3D __dma_iova_link(dev, addr - iova_start_pad, + bounce_phys - iova_start_pad, + iova_align(iovad, bounce_len), dir, attrs); + if (error) + swiotlb_tbl_unmap_single(dev, bounce_phys, bounce_len, dir, + attrs); + return error; +} + +static int iommu_dma_iova_link_swiotlb(struct device *dev, + struct dma_iova_state *state, phys_addr_t phys, size_t offset, + size_t size, enum dma_data_direction dir, unsigned long attrs) +{ + struct iommu_domain *domain =3D iommu_get_dma_domain(dev); + struct iommu_dma_cookie *cookie =3D domain->iova_cookie; + struct iova_domain *iovad =3D &cookie->iovad; + size_t iova_start_pad =3D iova_offset(iovad, phys); + size_t iova_end_pad =3D iova_offset(iovad, phys + size); + dma_addr_t addr =3D state->addr + offset; + size_t mapped =3D 0; + int error; + + if (iova_start_pad) { + size_t bounce_len =3D min(size, iovad->granule - iova_start_pad); + + error =3D iommu_dma_iova_bounce_and_link(dev, addr, phys, + bounce_len, dir, attrs, iova_start_pad); + if (error) + return error; + state->__size |=3D DMA_IOVA_USE_SWIOTLB; + + mapped +=3D bounce_len; + size -=3D bounce_len; + if (!size) + return 0; + } + + size -=3D iova_end_pad; + error =3D __dma_iova_link(dev, addr + mapped, phys + mapped, size, dir, + attrs); + if (error) + goto out_unmap; + mapped +=3D size; + + if (iova_end_pad) { + error =3D iommu_dma_iova_bounce_and_link(dev, addr + mapped, + phys + mapped, iova_end_pad, dir, attrs, 0); + if (error) + goto out_unmap; + state->__size |=3D DMA_IOVA_USE_SWIOTLB; + } + + return 0; + +out_unmap: + dma_iova_unlink(dev, state, 0, mapped, dir, attrs); + return error; +} + +/** + * dma_iova_link - Link a range of IOVA space + * @dev: DMA device + * @state: IOVA state + * @phys: physical address to link + * @offset: offset into the IOVA state to map into + * @size: size of the buffer + * @dir: DMA direction + * @attrs: attributes of mapping properties + * + * Link a range of IOVA space for the given IOVA state without IOTLB sync. + * This function is used to link multiple physical addresses in contiguous + * IOVA space without performing costly IOTLB sync. + * + * The caller is responsible to call to dma_iova_sync() to sync IOTLB at + * the end of linkage. + */ +int dma_iova_link(struct device *dev, struct dma_iova_state *state, + phys_addr_t phys, size_t offset, size_t size, + enum dma_data_direction dir, unsigned long attrs) +{ + struct iommu_domain *domain =3D iommu_get_dma_domain(dev); + struct iommu_dma_cookie *cookie =3D domain->iova_cookie; + struct iova_domain *iovad =3D &cookie->iovad; + size_t iova_start_pad =3D iova_offset(iovad, phys); + + if (WARN_ON_ONCE(iova_start_pad && offset > 0)) + return -EIO; + + if (dev_use_swiotlb(dev, size, dir) && + iova_unaligned(iovad, phys, size)) + return iommu_dma_iova_link_swiotlb(dev, state, phys, offset, + size, dir, attrs); + + return __dma_iova_link(dev, state->addr + offset - iova_start_pad, + phys - iova_start_pad, + iova_align(iovad, size + iova_start_pad), dir, attrs); +} +EXPORT_SYMBOL_GPL(dma_iova_link); + +/** + * dma_iova_sync - Sync IOTLB + * @dev: DMA device + * @state: IOVA state + * @offset: offset into the IOVA state to sync + * @size: size of the buffer + * + * Sync IOTLB for the given IOVA state. This function should be called on + * the IOVA-contiguous range created by one ore more dma_iova_link() calls + * to sync the IOTLB. + */ +int dma_iova_sync(struct device *dev, struct dma_iova_state *state, + size_t offset, size_t size) +{ + struct iommu_domain *domain =3D iommu_get_dma_domain(dev); + struct iommu_dma_cookie *cookie =3D domain->iova_cookie; + struct iova_domain *iovad =3D &cookie->iovad; + dma_addr_t addr =3D state->addr + offset; + size_t iova_start_pad =3D iova_offset(iovad, addr); + + return iommu_sync_map(domain, addr - iova_start_pad, + iova_align(iovad, size + iova_start_pad)); +} +EXPORT_SYMBOL_GPL(dma_iova_sync); + +static void iommu_dma_iova_unlink_range_slow(struct device *dev, + dma_addr_t addr, size_t size, enum dma_data_direction dir, + unsigned long attrs) +{ + struct iommu_domain *domain =3D iommu_get_dma_domain(dev); + struct iommu_dma_cookie *cookie =3D domain->iova_cookie; + struct iova_domain *iovad =3D &cookie->iovad; + size_t iova_start_pad =3D iova_offset(iovad, addr); + dma_addr_t end =3D addr + size; + + do { + phys_addr_t phys; + size_t len; + + phys =3D iommu_iova_to_phys(domain, addr); + if (WARN_ON(!phys)) + /* Something very horrible happen here */ + return; + + len =3D min_t(size_t, + end - addr, iovad->granule - iova_start_pad); + + if (!dev_is_dma_coherent(dev) && + !(attrs & DMA_ATTR_SKIP_CPU_SYNC)) + arch_sync_dma_for_cpu(phys, len, dir); + + swiotlb_tbl_unmap_single(dev, phys, len, dir, attrs); + + addr +=3D len; + iova_start_pad =3D 0; + } while (addr < end); +} + +static void __iommu_dma_iova_unlink(struct device *dev, + struct dma_iova_state *state, size_t offset, size_t size, + enum dma_data_direction dir, unsigned long attrs, + bool free_iova) +{ + struct iommu_domain *domain =3D iommu_get_dma_domain(dev); + struct iommu_dma_cookie *cookie =3D domain->iova_cookie; + struct iova_domain *iovad =3D &cookie->iovad; + dma_addr_t addr =3D state->addr + offset; + size_t iova_start_pad =3D iova_offset(iovad, addr); + struct iommu_iotlb_gather iotlb_gather; + size_t unmapped; + + if ((state->__size & DMA_IOVA_USE_SWIOTLB) || + (!dev_is_dma_coherent(dev) && !(attrs & DMA_ATTR_SKIP_CPU_SYNC))) + iommu_dma_iova_unlink_range_slow(dev, addr, size, dir, attrs); + + iommu_iotlb_gather_init(&iotlb_gather); + iotlb_gather.queued =3D free_iova && READ_ONCE(cookie->fq_domain); + + size =3D iova_align(iovad, size + iova_start_pad); + addr -=3D iova_start_pad; + unmapped =3D iommu_unmap_fast(domain, addr, size, &iotlb_gather); + WARN_ON(unmapped !=3D size); + + if (!iotlb_gather.queued) + iommu_iotlb_sync(domain, &iotlb_gather); + if (free_iova) + iommu_dma_free_iova(domain, addr, size, &iotlb_gather); +} + +/** + * dma_iova_unlink - Unlink a range of IOVA space + * @dev: DMA device + * @state: IOVA state + * @offset: offset into the IOVA state to unlink + * @size: size of the buffer + * @dir: DMA direction + * @attrs: attributes of mapping properties + * + * Unlink a range of IOVA space for the given IOVA state. + */ +void dma_iova_unlink(struct device *dev, struct dma_iova_state *state, + size_t offset, size_t size, enum dma_data_direction dir, + unsigned long attrs) +{ + __iommu_dma_iova_unlink(dev, state, offset, size, dir, attrs, false); +} +EXPORT_SYMBOL_GPL(dma_iova_unlink); + +/** + * dma_iova_destroy - Finish a DMA mapping transaction + * @dev: DMA device + * @state: IOVA state + * @mapped_len: number of bytes to unmap + * @dir: DMA direction + * @attrs: attributes of mapping properties + * + * Unlink the IOVA range up to @mapped_len and free the entire IOVA space.= The + * range of IOVA from dma_addr to @mapped_len must all be linked, and be t= he + * only linked IOVA in state. + */ +void dma_iova_destroy(struct device *dev, struct dma_iova_state *state, + size_t mapped_len, enum dma_data_direction dir, + unsigned long attrs) +{ + if (mapped_len) + __iommu_dma_iova_unlink(dev, state, 0, mapped_len, dir, attrs, + true); + else + /* + * We can be here if first call to dma_iova_link() failed and + * there is nothing to unlink, so let's be more clear. + */ + dma_iova_free(dev, state); +} +EXPORT_SYMBOL_GPL(dma_iova_destroy); + void iommu_setup_dma_ops(struct device *dev) { struct iommu_domain *domain =3D iommu_get_domain_for_dev(dev); diff --git a/include/linux/dma-mapping.h b/include/linux/dma-mapping.h index de7f73810d54..a71e110f1e9d 100644 --- a/include/linux/dma-mapping.h +++ b/include/linux/dma-mapping.h @@ -309,6 +309,17 @@ static inline bool dma_use_iova(struct dma_iova_state = *state) bool dma_iova_try_alloc(struct device *dev, struct dma_iova_state *state, phys_addr_t phys, size_t size); void dma_iova_free(struct device *dev, struct dma_iova_state *state); +void dma_iova_destroy(struct device *dev, struct dma_iova_state *state, + size_t mapped_len, enum dma_data_direction dir, + unsigned long attrs); +int dma_iova_sync(struct device *dev, struct dma_iova_state *state, + size_t offset, size_t size); +int dma_iova_link(struct device *dev, struct dma_iova_state *state, + phys_addr_t phys, size_t offset, size_t size, + enum dma_data_direction dir, unsigned long attrs); +void dma_iova_unlink(struct device *dev, struct dma_iova_state *state, + size_t offset, size_t size, enum dma_data_direction dir, + unsigned long attrs); #else /* CONFIG_IOMMU_DMA */ static inline bool dma_use_iova(struct dma_iova_state *state) { @@ -323,6 +334,27 @@ static inline void dma_iova_free(struct device *dev, struct dma_iova_state *state) { } +static inline void dma_iova_destroy(struct device *dev, + struct dma_iova_state *state, size_t mapped_len, + enum dma_data_direction dir, unsigned long attrs) +{ +} +static inline int dma_iova_sync(struct device *dev, + struct dma_iova_state *state, size_t offset, size_t size) +{ + return -EOPNOTSUPP; +} +static inline int dma_iova_link(struct device *dev, + struct dma_iova_state *state, phys_addr_t phys, size_t offset, + size_t size, enum dma_data_direction dir, unsigned long attrs) +{ + return -EOPNOTSUPP; +} +static inline void dma_iova_unlink(struct device *dev, + struct dma_iova_state *state, size_t offset, size_t size, + enum dma_data_direction dir, unsigned long attrs) +{ +} #endif /* CONFIG_IOMMU_DMA */ =20 #if defined(CONFIG_HAS_DMA) && defined(CONFIG_DMA_NEED_SYNC) --=20 2.49.0 From nobody Tue Dec 16 14:20:07 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6BB1D269B11; Mon, 28 Apr 2025 09:23:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745832213; cv=none; b=Ufle+eFLTeqLEDKkze2EBBqHNT7mX+F40gSHbd7FA7mPm4oquJUQaiHhlCfL5NF7v3Iyca7vlD4bCcZWNNBPV9dNoIlaJ7jHGgTYxVn1bw8eCfXVKH+BIaanp3Jse4zcaeDbPjvUo3dlAArZtWODr4f6h7khwJKD/2XwJdOCBAk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745832213; c=relaxed/simple; bh=zDzlqykv86nTBfYEytc8wm5wkFShzSQKAsrmEhNbBwU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=raiyYjIHc+7gNG19YhNWOr3swRrJhU0/mNkEDe7rnRQ6FFH/4A37C0pmSR5DKUYWcC5gf2OBta1AGmZx8ndyf/nMjwV/6hzAjvj7Jw2/dbv/+EiY0WR3PbWyKACJ4FUs+CXCzirrVddc1AJHGv6IA7wtkwWV2cahgJiSf1C+2Uo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=S1d90qh8; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="S1d90qh8" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 31E69C4CEE9; Mon, 28 Apr 2025 09:23:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1745832212; bh=zDzlqykv86nTBfYEytc8wm5wkFShzSQKAsrmEhNbBwU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=S1d90qh8CUE1R68AnclyKFgyc3M3PH8lRc6WjFsjygRgtWsAEkSuXRwZyPk5H2mWM yCIjmPjwBIzNqNOH29THOBXxLDDuAZw5RIzE+DbuJITHoomDA8eEEMQ24PeQgAyr9A zmhitSbQdvMeIgR+oZ5F+ZZ8YiqyBnwmEK2fSCJajwUgYutk+OX5i7mjlkL88cgAmv AYmthdpUSXspJkugyHwrF2tpdEh2aUssZitwTMgxeS9uC/9y7HeLUu/Ov6P8e9WkBZ RDhA7gkiFAOg5KHkp69Haf5Or2OAtnLQpJilA1Vti+fDK5ZkFfVMcaWVVXQOIyx3E/ fi+Uw3fAUJHUg== From: Leon Romanovsky To: Marek Szyprowski , Jens Axboe , Christoph Hellwig , Keith Busch Cc: Jake Edge , Jonathan Corbet , Jason Gunthorpe , Zhu Yanjun , Robin Murphy , Joerg Roedel , Will Deacon , Sagi Grimberg , Bjorn Helgaas , Logan Gunthorpe , Yishai Hadas , Shameer Kolothum , Kevin Tian , Alex Williamson , =?UTF-8?q?J=C3=A9r=C3=B4me=20Glisse?= , Andrew Morton , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-rdma@vger.kernel.org, iommu@lists.linux.dev, linux-nvme@lists.infradead.org, linux-pci@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org, Niklas Schnelle , Chuck Lever , Luis Chamberlain , Matthew Wilcox , Dan Williams , Kanchan Joshi , Chaitanya Kulkarni , Leon Romanovsky Subject: [PATCH v10 08/24] dma-mapping: add a dma_need_unmap helper Date: Mon, 28 Apr 2025 12:22:14 +0300 Message-ID: <448112d7b47869724d99b00de052127fc9d81612.1745831017.git.leon@kernel.org> X-Mailer: git-send-email 2.49.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Christoph Hellwig Add helper that allows a driver to skip calling dma_unmap_* if the DMA layer can guarantee that they are no-nops. Signed-off-by: Christoph Hellwig Tested-by: Jens Axboe Reviewed-by: Luis Chamberlain Signed-off-by: Leon Romanovsky --- include/linux/dma-mapping.h | 5 +++++ kernel/dma/mapping.c | 18 ++++++++++++++++++ 2 files changed, 23 insertions(+) diff --git a/include/linux/dma-mapping.h b/include/linux/dma-mapping.h index a71e110f1e9d..d2f358c5a25d 100644 --- a/include/linux/dma-mapping.h +++ b/include/linux/dma-mapping.h @@ -406,6 +406,7 @@ static inline bool dma_need_sync(struct device *dev, dm= a_addr_t dma_addr) { return dma_dev_need_sync(dev) ? __dma_need_sync(dev, dma_addr) : false; } +bool dma_need_unmap(struct device *dev); #else /* !CONFIG_HAS_DMA || !CONFIG_DMA_NEED_SYNC */ static inline bool dma_dev_need_sync(const struct device *dev) { @@ -431,6 +432,10 @@ static inline bool dma_need_sync(struct device *dev, d= ma_addr_t dma_addr) { return false; } +static inline bool dma_need_unmap(struct device *dev) +{ + return false; +} #endif /* !CONFIG_HAS_DMA || !CONFIG_DMA_NEED_SYNC */ =20 struct page *dma_alloc_pages(struct device *dev, size_t size, diff --git a/kernel/dma/mapping.c b/kernel/dma/mapping.c index cda127027e48..3c3204ad2839 100644 --- a/kernel/dma/mapping.c +++ b/kernel/dma/mapping.c @@ -443,6 +443,24 @@ bool __dma_need_sync(struct device *dev, dma_addr_t dm= a_addr) } EXPORT_SYMBOL_GPL(__dma_need_sync); =20 +/** + * dma_need_unmap - does this device need dma_unmap_* operations + * @dev: device to check + * + * If this function returns %false, drivers can skip calling dma_unmap_* a= fter + * finishing an I/O. This function must be called after all mappings that= might + * need to be unmapped have been performed. + */ +bool dma_need_unmap(struct device *dev) +{ + if (!dma_map_direct(dev, get_dma_ops(dev))) + return true; + if (!dev->dma_skip_sync) + return true; + return IS_ENABLED(CONFIG_DMA_API_DEBUG); +} +EXPORT_SYMBOL_GPL(dma_need_unmap); + static void dma_setup_need_sync(struct device *dev) { const struct dma_map_ops *ops =3D get_dma_ops(dev); --=20 2.49.0 From nobody Tue Dec 16 14:20:07 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F1DCE267F4D; Mon, 28 Apr 2025 09:23:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745832199; cv=none; b=ObSSUyP5xqzzSU7vLmgGU9QRiqwOdNQXxsJ5YBsPZCnQ0XN5kcLnM7yG0I5Q3dc6jv8qRhfYr1rDGOHGSmobl4ZdBbxHHIY802lPJsm6WNDcoiLiShL7jioslzdrZCSDskaoaCkeOZX57NgZt8GsLTAwmpzcdNrKBbNkaGB2WIc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745832199; c=relaxed/simple; bh=d1+z79zg+kihe5QDvuiZPuBTqwMqD9eJrziwPARATZw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=lVeyjq621AcFCL2p2+3m5ssy8s8lzVqLizMC8cPVdBYhBFOKi9j/VKtLVLNT0bQhxNsPstqJclHadILX6IBCsUPoNErNafAT+4GOX7AJ5eYE+7Vj36ZHq45cHFfAOmylamkm2MNbfHQcHDvwZ/t4/caL2Na45zlTw505k3UfHrA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=QUB1lhoO; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="QUB1lhoO" Received: by smtp.kernel.org (Postfix) with ESMTPSA id C251DC4CEE4; Mon, 28 Apr 2025 09:23:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1745832198; bh=d1+z79zg+kihe5QDvuiZPuBTqwMqD9eJrziwPARATZw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=QUB1lhoO/1KGr0Qc2Y24vqWWVYEtzDQTf470PaEamVy2aHjYRrcXKp64fObLbiKH+ bNxqnX7fKat1w5Xi0X35HNlsQwBZlouj12aYi2WunJrqDdDkEZlSfGcPj6HiaI9DBw U4gv7P24gIYOH7FsCzIuDcSsU8D4IKS4z1awHvLgXavst4fRip90kp2o+6YpEv2CE2 zYvt45+/xEIlk1NMEv8TEY13AZoY9Cl4xN7/bvC563NKc5l8drOkGBh2g4VXmWHRNK pjPmt5XByo10SIhSxFUP3e5wIu8MqQfto5Bo6kx9K8xXZcJiKA5RQmTVg0LE+SeQWw z7Cz3rAsfEiPw== From: Leon Romanovsky To: Marek Szyprowski , Jens Axboe , Christoph Hellwig , Keith Busch Cc: Jake Edge , Jonathan Corbet , Jason Gunthorpe , Zhu Yanjun , Robin Murphy , Joerg Roedel , Will Deacon , Sagi Grimberg , Bjorn Helgaas , Logan Gunthorpe , Yishai Hadas , Shameer Kolothum , Kevin Tian , Alex Williamson , =?UTF-8?q?J=C3=A9r=C3=B4me=20Glisse?= , Andrew Morton , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-rdma@vger.kernel.org, iommu@lists.linux.dev, linux-nvme@lists.infradead.org, linux-pci@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org, Niklas Schnelle , Chuck Lever , Luis Chamberlain , Matthew Wilcox , Dan Williams , Kanchan Joshi , Chaitanya Kulkarni , Leon Romanovsky Subject: [PATCH v10 09/24] docs: core-api: document the IOVA-based API Date: Mon, 28 Apr 2025 12:22:15 +0300 Message-ID: <3c386dfbff489abb871e550c227af65eb90f920d.1745831017.git.leon@kernel.org> X-Mailer: git-send-email 2.49.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Christoph Hellwig Add an explanation of the newly added IOVA-based mapping API. Signed-off-by: Christoph Hellwig Tested-by: Jens Axboe Signed-off-by: Leon Romanovsky --- Documentation/core-api/dma-api.rst | 71 ++++++++++++++++++++++++++++++ 1 file changed, 71 insertions(+) diff --git a/Documentation/core-api/dma-api.rst b/Documentation/core-api/dm= a-api.rst index 8e3cce3d0a23..2ad08517e626 100644 --- a/Documentation/core-api/dma-api.rst +++ b/Documentation/core-api/dma-api.rst @@ -530,6 +530,77 @@ routines, e.g.::: .... } =20 +Part Ie - IOVA-based DMA mappings +--------------------------------- + +These APIs allow a very efficient mapping when using an IOMMU. They are an +optional path that requires extra code and are only recommended for drivers +where DMA mapping performance, or the space usage for storing the DMA addr= esses +matter. All the considerations from the previous section apply here as we= ll. + +:: + + bool dma_iova_try_alloc(struct device *dev, struct dma_iova_state *sta= te, + phys_addr_t phys, size_t size); + +Is used to try to allocate IOVA space for mapping operation. If it returns +false this API can't be used for the given device and the normal streaming +DMA mapping API should be used. The ``struct dma_iova_state`` is allocated +by the driver and must be kept around until unmap time. + +:: + + static inline bool dma_use_iova(struct dma_iova_state *state) + +Can be used by the driver to check if the IOVA-based API is used after a +call to dma_iova_try_alloc. This can be useful in the unmap path. + +:: + + int dma_iova_link(struct device *dev, struct dma_iova_state *state, + phys_addr_t phys, size_t offset, size_t size, + enum dma_data_direction dir, unsigned long attrs); + +Is used to link ranges to the IOVA previously allocated. The start of all +but the first call to dma_iova_link for a given state must be aligned +to the DMA merge boundary returned by ``dma_get_merge_boundary())``, and +the size of all but the last range must be aligned to the DMA merge bounda= ry +as well. + +:: + + int dma_iova_sync(struct device *dev, struct dma_iova_state *state, + size_t offset, size_t size); + +Must be called to sync the IOMMU page tables for IOVA-range mapped by one = or +more calls to ``dma_iova_link()``. + +For drivers that use a one-shot mapping, all ranges can be unmapped and the +IOVA freed by calling: + +:: + + void dma_iova_destroy(struct device *dev, struct dma_iova_state *state, + size_t mapped_len, enum dma_data_direction dir, + unsigned long attrs); + +Alternatively drivers can dynamically manage the IOVA space by unmapping +and mapping individual regions. In that case + +:: + + void dma_iova_unlink(struct device *dev, struct dma_iova_state *state, + size_t offset, size_t size, enum dma_data_direction dir, + unsigned long attrs); + +is used to unmap a range previously mapped, and + +:: + + void dma_iova_free(struct device *dev, struct dma_iova_state *state); + +is used to free the IOVA space. All regions must have been unmapped using +``dma_iova_unlink()`` before calling ``dma_iova_free()``. =20 Part II - Non-coherent DMA allocations -------------------------------------- --=20 2.49.0 From nobody Tue Dec 16 14:20:07 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 879D425F99A; Mon, 28 Apr 2025 09:23:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745832203; cv=none; b=P9/7rOossnq5RMt23Hqq7JnZHJNVvckxHgpbvnZcKy2GrF8jZ9PoXS7ESRJkRn2ZEG/XXJT0aEymD/KUpneI5GxyTVOWARICbj6jQy2cvXMh2Y99KxkXrXBBPpyUSg4XeJ764puyfezkVsoORPpuWHtjqsV0eYlgddajPJu1I5w= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745832203; c=relaxed/simple; bh=DqmELyRaD+HKliKs+yhB7fshFSe2Z6G/xyJIpc/k6wk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=P73msq3t61U9sRwq56T13iDWhvljOdAVU+YkRxUpuRhq99Uu9loKVx5Uyu0DzOQvspY74939ifISoy4LNi3qP5xowXxB6ye79Bh50rVNxVayeAJtvi/JaWp3F3bmzy36iHwtQcQqk5FvryKqUmxGHam3v10rZtgvl32pJ9fj7qA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=i33plADx; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="i33plADx" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 0A77BC4CEE4; Mon, 28 Apr 2025 09:23:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1745832202; bh=DqmELyRaD+HKliKs+yhB7fshFSe2Z6G/xyJIpc/k6wk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=i33plADx4mBDh4Mg0JhS7PdQonorQCsmo5YRKErsBgYKa4z8vOidmkY/AfS0xrH1Z yFDIAo01whKRGniuFCRzbw1+F6BnXoeGa8ey25ORoO6MG2G3/eXqsW1FNher3MmIz0 +9dEW6mpOEwloxkUFgK2y3up8HxeJXMHWV+4XbrTQYL5Or5e//10k5C7G4pNzcKJ/m oANUDjKdfpR/c4GB4V24c0eKKek4BVX9/RgbDOjLjhU8rGCdF1gxhQqNesdw2rfrrL MyYaLBYM5LnbvqLH5GsIk9qtApa0VRVHM28Dg1Mr4VJ5Nm2VYTSsLzNp0HPAb/dB/M 4dv4HRnILcU/A== From: Leon Romanovsky To: Marek Szyprowski , Jens Axboe , Christoph Hellwig , Keith Busch Cc: Leon Romanovsky , Jake Edge , Jonathan Corbet , Jason Gunthorpe , Zhu Yanjun , Robin Murphy , Joerg Roedel , Will Deacon , Sagi Grimberg , Bjorn Helgaas , Logan Gunthorpe , Yishai Hadas , Shameer Kolothum , Kevin Tian , Alex Williamson , =?UTF-8?q?J=C3=A9r=C3=B4me=20Glisse?= , Andrew Morton , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-rdma@vger.kernel.org, iommu@lists.linux.dev, linux-nvme@lists.infradead.org, linux-pci@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org, Niklas Schnelle , Chuck Lever , Luis Chamberlain , Matthew Wilcox , Dan Williams , Kanchan Joshi , Chaitanya Kulkarni , Jason Gunthorpe Subject: [PATCH v10 10/24] mm/hmm: let users to tag specific PFN with DMA mapped bit Date: Mon, 28 Apr 2025 12:22:16 +0300 Message-ID: X-Mailer: git-send-email 2.49.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Leon Romanovsky Introduce new sticky flag (HMM_PFN_DMA_MAPPED), which isn't overwritten by HMM range fault. Such flag allows users to tag specific PFNs with information if this specific PFN was already DMA mapped. Tested-by: Jens Axboe Reviewed-by: Christoph Hellwig Reviewed-by: Jason Gunthorpe Signed-off-by: Leon Romanovsky --- include/linux/hmm.h | 20 ++++++++++++++++-- mm/hmm.c | 51 ++++++++++++++++++++++++++++----------------- 2 files changed, 50 insertions(+), 21 deletions(-) diff --git a/include/linux/hmm.h b/include/linux/hmm.h index 126a36571667..a43e56f273a1 100644 --- a/include/linux/hmm.h +++ b/include/linux/hmm.h @@ -23,6 +23,8 @@ struct mmu_interval_notifier; * HMM_PFN_WRITE - if the page memory can be written to (requires HMM_PFN_= VALID) * HMM_PFN_ERROR - accessing the pfn is impossible and the device should * fail. ie poisoned memory, special pages, no vma, etc + * HMM_PFN_DMA_MAPPED - Flag preserved on input-to-output transformation + * to mark that page is already DMA mapped * * On input: * 0 - Return the current state of the page, do not fault = it. @@ -36,13 +38,19 @@ enum hmm_pfn_flags { HMM_PFN_VALID =3D 1UL << (BITS_PER_LONG - 1), HMM_PFN_WRITE =3D 1UL << (BITS_PER_LONG - 2), HMM_PFN_ERROR =3D 1UL << (BITS_PER_LONG - 3), - HMM_PFN_ORDER_SHIFT =3D (BITS_PER_LONG - 8), + /* + * Sticky flags, carried from input to output, + * don't forget to update HMM_PFN_INOUT_FLAGS + */ + HMM_PFN_DMA_MAPPED =3D 1UL << (BITS_PER_LONG - 4), + + HMM_PFN_ORDER_SHIFT =3D (BITS_PER_LONG - 9), =20 /* Input flags */ HMM_PFN_REQ_FAULT =3D HMM_PFN_VALID, HMM_PFN_REQ_WRITE =3D HMM_PFN_WRITE, =20 - HMM_PFN_FLAGS =3D 0xFFUL << HMM_PFN_ORDER_SHIFT, + HMM_PFN_FLAGS =3D ~((1UL << HMM_PFN_ORDER_SHIFT) - 1), }; =20 /* @@ -57,6 +65,14 @@ static inline struct page *hmm_pfn_to_page(unsigned long= hmm_pfn) return pfn_to_page(hmm_pfn & ~HMM_PFN_FLAGS); } =20 +/* + * hmm_pfn_to_phys() - return physical address pointed to by a device entry + */ +static inline phys_addr_t hmm_pfn_to_phys(unsigned long hmm_pfn) +{ + return __pfn_to_phys(hmm_pfn & ~HMM_PFN_FLAGS); +} + /* * hmm_pfn_to_map_order() - return the CPU mapping size order * diff --git a/mm/hmm.c b/mm/hmm.c index 082f7b7c0b9e..51fe8b011cc7 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -39,13 +39,20 @@ enum { HMM_NEED_ALL_BITS =3D HMM_NEED_FAULT | HMM_NEED_WRITE_FAULT, }; =20 +enum { + /* These flags are carried from input-to-output */ + HMM_PFN_INOUT_FLAGS =3D HMM_PFN_DMA_MAPPED, +}; + static int hmm_pfns_fill(unsigned long addr, unsigned long end, struct hmm_range *range, unsigned long cpu_flags) { unsigned long i =3D (addr - range->start) >> PAGE_SHIFT; =20 - for (; addr < end; addr +=3D PAGE_SIZE, i++) - range->hmm_pfns[i] =3D cpu_flags; + for (; addr < end; addr +=3D PAGE_SIZE, i++) { + range->hmm_pfns[i] &=3D HMM_PFN_INOUT_FLAGS; + range->hmm_pfns[i] |=3D cpu_flags; + } return 0; } =20 @@ -202,8 +209,10 @@ static int hmm_vma_handle_pmd(struct mm_walk *walk, un= signed long addr, return hmm_vma_fault(addr, end, required_fault, walk); =20 pfn =3D pmd_pfn(pmd) + ((addr & ~PMD_MASK) >> PAGE_SHIFT); - for (i =3D 0; addr < end; addr +=3D PAGE_SIZE, i++, pfn++) - hmm_pfns[i] =3D pfn | cpu_flags; + for (i =3D 0; addr < end; addr +=3D PAGE_SIZE, i++, pfn++) { + hmm_pfns[i] &=3D HMM_PFN_INOUT_FLAGS; + hmm_pfns[i] |=3D pfn | cpu_flags; + } return 0; } #else /* CONFIG_TRANSPARENT_HUGEPAGE */ @@ -230,14 +239,14 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, u= nsigned long addr, unsigned long cpu_flags; pte_t pte =3D ptep_get(ptep); uint64_t pfn_req_flags =3D *hmm_pfn; + uint64_t new_pfn_flags =3D 0; =20 if (pte_none_mostly(pte)) { required_fault =3D hmm_pte_need_fault(hmm_vma_walk, pfn_req_flags, 0); if (required_fault) goto fault; - *hmm_pfn =3D 0; - return 0; + goto out; } =20 if (!pte_present(pte)) { @@ -253,16 +262,14 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, u= nsigned long addr, cpu_flags =3D HMM_PFN_VALID; if (is_writable_device_private_entry(entry)) cpu_flags |=3D HMM_PFN_WRITE; - *hmm_pfn =3D swp_offset_pfn(entry) | cpu_flags; - return 0; + new_pfn_flags =3D swp_offset_pfn(entry) | cpu_flags; + goto out; } =20 required_fault =3D hmm_pte_need_fault(hmm_vma_walk, pfn_req_flags, 0); - if (!required_fault) { - *hmm_pfn =3D 0; - return 0; - } + if (!required_fault) + goto out; =20 if (!non_swap_entry(entry)) goto fault; @@ -304,11 +311,13 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, u= nsigned long addr, pte_unmap(ptep); return -EFAULT; } - *hmm_pfn =3D HMM_PFN_ERROR; - return 0; + new_pfn_flags =3D HMM_PFN_ERROR; + goto out; } =20 - *hmm_pfn =3D pte_pfn(pte) | cpu_flags; + new_pfn_flags =3D pte_pfn(pte) | cpu_flags; +out: + *hmm_pfn =3D (*hmm_pfn & HMM_PFN_INOUT_FLAGS) | new_pfn_flags; return 0; =20 fault: @@ -448,8 +457,10 @@ static int hmm_vma_walk_pud(pud_t *pudp, unsigned long= start, unsigned long end, } =20 pfn =3D pud_pfn(pud) + ((addr & ~PUD_MASK) >> PAGE_SHIFT); - for (i =3D 0; i < npages; ++i, ++pfn) - hmm_pfns[i] =3D pfn | cpu_flags; + for (i =3D 0; i < npages; ++i, ++pfn) { + hmm_pfns[i] &=3D HMM_PFN_INOUT_FLAGS; + hmm_pfns[i] |=3D pfn | cpu_flags; + } goto out_unlock; } =20 @@ -507,8 +518,10 @@ static int hmm_vma_walk_hugetlb_entry(pte_t *pte, unsi= gned long hmask, } =20 pfn =3D pte_pfn(entry) + ((start & ~hmask) >> PAGE_SHIFT); - for (; addr < end; addr +=3D PAGE_SIZE, i++, pfn++) - range->hmm_pfns[i] =3D pfn | cpu_flags; + for (; addr < end; addr +=3D PAGE_SIZE, i++, pfn++) { + range->hmm_pfns[i] &=3D HMM_PFN_INOUT_FLAGS; + range->hmm_pfns[i] |=3D pfn | cpu_flags; + } =20 spin_unlock(ptl); return 0; --=20 2.49.0 From nobody Tue Dec 16 14:20:07 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 32D1125FA1F; Mon, 28 Apr 2025 09:23:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745832209; cv=none; b=brudFRnKnypcU39ABg67XL0M78Pq9ANLKCOUFHJcpMBPQ1K5Meesdd6KYQOPHhb9o/SR8YsVkhQA/og4XfY/qOZUANlWOdtfw3CqAj49fQY0Hdqypbbh7zdMV+HUPHFGGFZl131rzyT6jfnlhjpYGLUHtUonQ5k0BKrLAkeguf4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745832209; c=relaxed/simple; bh=UfpnhIUTIzjlDf4zCNY0LaKfq2sLGv79hQGEU8OTNl4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=iO7sqlULbCPloy/PJ9STM3ODu0SHaHOagXbl5wf+ooC9YsoQiMTBurXx2awcrBVUWxuYAacQ4Tou3eYcRt2ydI3Eh1ki/DjTNH7sVIk2I2g3Ku379ZMvh4CFWd90KxcDH8bZ4012ro0c8FUf3POv7wA1XCQcwPHg3KhMW4U90Cg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=g2e1OMPh; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="g2e1OMPh" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 0A27CC4CEE9; Mon, 28 Apr 2025 09:23:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1745832208; bh=UfpnhIUTIzjlDf4zCNY0LaKfq2sLGv79hQGEU8OTNl4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=g2e1OMPhaTBRM5TKyK3GtLmQh40BEO8s/kQDN0FqepNdBbJHYMrx9ZWn2Ip2g7X5/ 36tTxLnjZQg3+LGIXuYaUSpjKXxlN1oD1Bca4VI8ZaZGeJ5SJhAK2rRPDBBU/odWU8 Eizp71jTomV27gw91DmTuIqliRI+lEkuj1wcB/hB2LP/4jFsEZ+Kgm9RGF2l+yFZBS o3SqvK3Z1wI/0q3aEq9nVGtXt6QiOAgF3ai8zQByNwK91Rw+SckIESfJoVyGifd99L 1GXj/pcW3MofL6Bn43AB7JGsN/8fSBujEgbkE8mBuql3c0Q2srMrvJOJffajxK6D2Q WQTXDNxUl5QkA== From: Leon Romanovsky To: Marek Szyprowski , Jens Axboe , Christoph Hellwig , Keith Busch Cc: Leon Romanovsky , Jake Edge , Jonathan Corbet , Jason Gunthorpe , Zhu Yanjun , Robin Murphy , Joerg Roedel , Will Deacon , Sagi Grimberg , Bjorn Helgaas , Logan Gunthorpe , Yishai Hadas , Shameer Kolothum , Kevin Tian , Alex Williamson , =?UTF-8?q?J=C3=A9r=C3=B4me=20Glisse?= , Andrew Morton , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-rdma@vger.kernel.org, iommu@lists.linux.dev, linux-nvme@lists.infradead.org, linux-pci@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org, Niklas Schnelle , Chuck Lever , Luis Chamberlain , Matthew Wilcox , Dan Williams , Kanchan Joshi , Chaitanya Kulkarni , Jason Gunthorpe Subject: [PATCH v10 11/24] mm/hmm: provide generic DMA managing logic Date: Mon, 28 Apr 2025 12:22:17 +0300 Message-ID: X-Mailer: git-send-email 2.49.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Leon Romanovsky HMM callers use PFN list to populate range while calling to hmm_range_fault(), the conversion from PFN to DMA address is done by the callers with help of another DMA list. However, it is wasteful on any modern platform and by doing the right logic, that DMA list can be avoided. Provide generic logic to manage these lists and gave an interface to map/unmap PFNs to DMA addresses, without requiring from the callers to be an experts in DMA core API. Tested-by: Jens Axboe Reviewed-by: Jason Gunthorpe Signed-off-by: Leon Romanovsky --- include/linux/hmm-dma.h | 33 +++++++ include/linux/hmm.h | 6 +- mm/hmm.c | 214 +++++++++++++++++++++++++++++++++++++++- 3 files changed, 251 insertions(+), 2 deletions(-) create mode 100644 include/linux/hmm-dma.h diff --git a/include/linux/hmm-dma.h b/include/linux/hmm-dma.h new file mode 100644 index 000000000000..f58b9fc71999 --- /dev/null +++ b/include/linux/hmm-dma.h @@ -0,0 +1,33 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* Copyright (c) 2024 NVIDIA Corporation & Affiliates */ +#ifndef LINUX_HMM_DMA_H +#define LINUX_HMM_DMA_H + +#include + +struct dma_iova_state; +struct pci_p2pdma_map_state; + +/* + * struct hmm_dma_map - array of PFNs and DMA addresses + * + * @state: DMA IOVA state + * @pfns: array of PFNs + * @dma_list: array of DMA addresses + * @dma_entry_size: size of each DMA entry in the array + */ +struct hmm_dma_map { + struct dma_iova_state state; + unsigned long *pfn_list; + dma_addr_t *dma_list; + size_t dma_entry_size; +}; + +int hmm_dma_map_alloc(struct device *dev, struct hmm_dma_map *map, + size_t nr_entries, size_t dma_entry_size); +void hmm_dma_map_free(struct device *dev, struct hmm_dma_map *map); +dma_addr_t hmm_dma_map_pfn(struct device *dev, struct hmm_dma_map *map, + size_t idx, + struct pci_p2pdma_map_state *p2pdma_state); +bool hmm_dma_unmap_pfn(struct device *dev, struct hmm_dma_map *map, size_t= idx); +#endif /* LINUX_HMM_DMA_H */ diff --git a/include/linux/hmm.h b/include/linux/hmm.h index a43e56f273a1..db75ffc949a7 100644 --- a/include/linux/hmm.h +++ b/include/linux/hmm.h @@ -23,6 +23,8 @@ struct mmu_interval_notifier; * HMM_PFN_WRITE - if the page memory can be written to (requires HMM_PFN_= VALID) * HMM_PFN_ERROR - accessing the pfn is impossible and the device should * fail. ie poisoned memory, special pages, no vma, etc + * HMM_PFN_P2PDMA - P2P page + * HMM_PFN_P2PDMA_BUS - Bus mapped P2P transfer * HMM_PFN_DMA_MAPPED - Flag preserved on input-to-output transformation * to mark that page is already DMA mapped * @@ -43,8 +45,10 @@ enum hmm_pfn_flags { * don't forget to update HMM_PFN_INOUT_FLAGS */ HMM_PFN_DMA_MAPPED =3D 1UL << (BITS_PER_LONG - 4), + HMM_PFN_P2PDMA =3D 1UL << (BITS_PER_LONG - 5), + HMM_PFN_P2PDMA_BUS =3D 1UL << (BITS_PER_LONG - 6), =20 - HMM_PFN_ORDER_SHIFT =3D (BITS_PER_LONG - 9), + HMM_PFN_ORDER_SHIFT =3D (BITS_PER_LONG - 11), =20 /* Input flags */ HMM_PFN_REQ_FAULT =3D HMM_PFN_VALID, diff --git a/mm/hmm.c b/mm/hmm.c index 51fe8b011cc7..a8bf097677f3 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -10,6 +10,7 @@ */ #include #include +#include #include #include #include @@ -23,6 +24,7 @@ #include #include #include +#include #include #include =20 @@ -41,7 +43,8 @@ enum { =20 enum { /* These flags are carried from input-to-output */ - HMM_PFN_INOUT_FLAGS =3D HMM_PFN_DMA_MAPPED, + HMM_PFN_INOUT_FLAGS =3D HMM_PFN_DMA_MAPPED | HMM_PFN_P2PDMA | + HMM_PFN_P2PDMA_BUS, }; =20 static int hmm_pfns_fill(unsigned long addr, unsigned long end, @@ -620,3 +623,212 @@ int hmm_range_fault(struct hmm_range *range) return ret; } EXPORT_SYMBOL(hmm_range_fault); + +/** + * hmm_dma_map_alloc - Allocate HMM map structure + * @dev: device to allocate structure for + * @map: HMM map to allocate + * @nr_entries: number of entries in the map + * @dma_entry_size: size of the DMA entry in the map + * + * Allocate the HMM map structure and all the lists it contains. + * Return 0 on success, -ENOMEM on failure. + */ +int hmm_dma_map_alloc(struct device *dev, struct hmm_dma_map *map, + size_t nr_entries, size_t dma_entry_size) +{ + bool dma_need_sync =3D false; + bool use_iova; + + if (!(nr_entries * PAGE_SIZE / dma_entry_size)) + return -EINVAL; + + /* + * The HMM API violates our normal DMA buffer ownership rules and can't + * transfer buffer ownership. The dma_addressing_limited() check is a + * best approximation to ensure no swiotlb buffering happens. + */ +#ifdef CONFIG_DMA_NEED_SYNC + dma_need_sync =3D !dev->dma_skip_sync; +#endif /* CONFIG_DMA_NEED_SYNC */ + if (dma_need_sync || dma_addressing_limited(dev)) + return -EOPNOTSUPP; + + map->dma_entry_size =3D dma_entry_size; + map->pfn_list =3D kvcalloc(nr_entries, sizeof(*map->pfn_list), + GFP_KERNEL | __GFP_NOWARN); + if (!map->pfn_list) + return -ENOMEM; + + use_iova =3D dma_iova_try_alloc(dev, &map->state, 0, + nr_entries * PAGE_SIZE); + if (!use_iova && dma_need_unmap(dev)) { + map->dma_list =3D kvcalloc(nr_entries, sizeof(*map->dma_list), + GFP_KERNEL | __GFP_NOWARN); + if (!map->dma_list) + goto err_dma; + } + return 0; + +err_dma: + kvfree(map->pfn_list); + return -ENOMEM; +} +EXPORT_SYMBOL_GPL(hmm_dma_map_alloc); + +/** + * hmm_dma_map_free - iFree HMM map structure + * @dev: device to free structure from + * @map: HMM map containing the various lists and state + * + * Free the HMM map structure and all the lists it contains. + */ +void hmm_dma_map_free(struct device *dev, struct hmm_dma_map *map) +{ + if (dma_use_iova(&map->state)) + dma_iova_free(dev, &map->state); + kvfree(map->pfn_list); + kvfree(map->dma_list); +} +EXPORT_SYMBOL_GPL(hmm_dma_map_free); + +/** + * hmm_dma_map_pfn - Map a physical HMM page to DMA address + * @dev: Device to map the page for + * @map: HMM map + * @idx: Index into the PFN and dma address arrays + * @p2pdma_state: PCI P2P state. + * + * dma_alloc_iova() allocates IOVA based on the size specified by their us= e in + * iova->size. Call this function after IOVA allocation to link whole @page + * to get the DMA address. Note that very first call to this function + * will have @offset set to 0 in the IOVA space allocated from + * dma_alloc_iova(). For subsequent calls to this function on same @iova, + * @offset needs to be advanced by the caller with the size of previous + * page that was linked + DMA address returned for the previous page that = was + * linked by this function. + */ +dma_addr_t hmm_dma_map_pfn(struct device *dev, struct hmm_dma_map *map, + size_t idx, + struct pci_p2pdma_map_state *p2pdma_state) +{ + struct dma_iova_state *state =3D &map->state; + dma_addr_t *dma_addrs =3D map->dma_list; + unsigned long *pfns =3D map->pfn_list; + struct page *page =3D hmm_pfn_to_page(pfns[idx]); + phys_addr_t paddr =3D hmm_pfn_to_phys(pfns[idx]); + size_t offset =3D idx * map->dma_entry_size; + unsigned long attrs =3D 0; + dma_addr_t dma_addr; + int ret; + + if ((pfns[idx] & HMM_PFN_DMA_MAPPED) && + !(pfns[idx] & HMM_PFN_P2PDMA_BUS)) { + /* + * We are in this flow when there is a need to resync flags, + * for example when page was already linked in prefetch call + * with READ flag and now we need to add WRITE flag + * + * This page was already programmed to HW and we don't want/need + * to unlink and link it again just to resync flags. + */ + if (dma_use_iova(state)) + return state->addr + offset; + + /* + * Without dma_need_unmap, the dma_addrs array is NULL, thus we + * need to regenerate the address below even if there already + * was a mapping. But !dma_need_unmap implies that the + * mapping stateless, so this is fine. + */ + if (dma_need_unmap(dev)) + return dma_addrs[idx]; + + /* Continue to remapping */ + } + + switch (pci_p2pdma_state(p2pdma_state, dev, page)) { + case PCI_P2PDMA_MAP_NONE: + break; + case PCI_P2PDMA_MAP_THRU_HOST_BRIDGE: + attrs |=3D DMA_ATTR_SKIP_CPU_SYNC; + pfns[idx] |=3D HMM_PFN_P2PDMA; + break; + case PCI_P2PDMA_MAP_BUS_ADDR: + pfns[idx] |=3D HMM_PFN_P2PDMA_BUS | HMM_PFN_DMA_MAPPED; + return pci_p2pdma_bus_addr_map(p2pdma_state, paddr); + default: + return DMA_MAPPING_ERROR; + } + + if (dma_use_iova(state)) { + ret =3D dma_iova_link(dev, state, paddr, offset, + map->dma_entry_size, DMA_BIDIRECTIONAL, + attrs); + if (ret) + goto error; + + ret =3D dma_iova_sync(dev, state, offset, map->dma_entry_size); + if (ret) { + dma_iova_unlink(dev, state, offset, map->dma_entry_size, + DMA_BIDIRECTIONAL, attrs); + goto error; + } + + dma_addr =3D state->addr + offset; + } else { + if (WARN_ON_ONCE(dma_need_unmap(dev) && !dma_addrs)) + goto error; + + dma_addr =3D dma_map_page(dev, page, 0, map->dma_entry_size, + DMA_BIDIRECTIONAL); + if (dma_mapping_error(dev, dma_addr)) + goto error; + + if (dma_need_unmap(dev)) + dma_addrs[idx] =3D dma_addr; + } + pfns[idx] |=3D HMM_PFN_DMA_MAPPED; + return dma_addr; +error: + pfns[idx] &=3D ~HMM_PFN_P2PDMA; + return DMA_MAPPING_ERROR; + +} +EXPORT_SYMBOL_GPL(hmm_dma_map_pfn); + +/** + * hmm_dma_unmap_pfn - Unmap a physical HMM page from DMA address + * @dev: Device to unmap the page from + * @map: HMM map + * @idx: Index of the PFN to unmap + * + * Returns true if the PFN was mapped and has been unmapped, false otherwi= se. + */ +bool hmm_dma_unmap_pfn(struct device *dev, struct hmm_dma_map *map, size_t= idx) +{ + const unsigned long valid_dma =3D HMM_PFN_VALID | HMM_PFN_DMA_MAPPED; + struct dma_iova_state *state =3D &map->state; + dma_addr_t *dma_addrs =3D map->dma_list; + unsigned long *pfns =3D map->pfn_list; + unsigned long attrs =3D 0; + + if ((pfns[idx] & valid_dma) !=3D valid_dma) + return false; + + if (pfns[idx] & HMM_PFN_P2PDMA_BUS) + ; /* no need to unmap bus address P2P mappings */ + else if (dma_use_iova(state)) { + if (pfns[idx] & HMM_PFN_P2PDMA) + attrs |=3D DMA_ATTR_SKIP_CPU_SYNC; + dma_iova_unlink(dev, state, idx * map->dma_entry_size, + map->dma_entry_size, DMA_BIDIRECTIONAL, attrs); + } else if (dma_need_unmap(dev)) + dma_unmap_page(dev, dma_addrs[idx], map->dma_entry_size, + DMA_BIDIRECTIONAL); + + pfns[idx] &=3D + ~(HMM_PFN_DMA_MAPPED | HMM_PFN_P2PDMA | HMM_PFN_P2PDMA_BUS); + return true; +} +EXPORT_SYMBOL_GPL(hmm_dma_unmap_pfn); --=20 2.49.0 From nobody Tue Dec 16 14:20:07 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D68F625F7A2; Mon, 28 Apr 2025 09:23:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745832225; cv=none; b=GVZhNgZeWk40Cm7PdZKeM6nPQNvOBseSakwTSsem8zUwGjpQEiuvh4Da/JVxkntityI5tALqHJk4HlxCvfhhumP6oXRlRadC45ecXpgWKnlu4UKgofO7QgpMLnPTJXG8ho1BMzBoaNHBQsrz+yuN6ztTZA64vpCN/wi5LzRDuko= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745832225; c=relaxed/simple; bh=BHrpmZ7+n44Wwug7Tkul5xfu4MbMbr93E8OVig3WwUA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=E1fkCcChrN4CI98/IlY4YHvUloHWQca2JEmla7m6Q0Jv5u0cK2S9RT4Wgn8Ih6KpicGFOnfDPmBFc0N96cm4CiDepDgIgs9aBZhXHwJwN/GIUuXt+lIdyetQ/eenlZZ7Tyf4V9R51r6vlhFcAN3pAacAPn/XIimDsApWjNKwCLE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=GPO21Dmt; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="GPO21Dmt" Received: by smtp.kernel.org (Postfix) with ESMTPSA id C9187C4CEE4; Mon, 28 Apr 2025 09:23:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1745832224; bh=BHrpmZ7+n44Wwug7Tkul5xfu4MbMbr93E8OVig3WwUA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=GPO21DmtP5aut8YN9Fmpy1EJ5T2SwH0t6xJG37Yv/F+KpLF8RADDddn7mc1YJFzl6 C0785Hq24L/3abOG5FAtwzoMRIbZPhplz6jlqIrPK2Y6PLUuh7GO4ONn3kIBLilWv9 ShacrRkR35u/zf4X8cFKM2Psiy9M5qiFT6c+0ZAuLe9wUDndhx30X1li1TyLMJY2Kw MO9PokUSk437Oet6tu8ESsECMl+qlZgUY7QZW9r2LzVgCkPnQKdjH9pofDx5iSVJcH h6kxN6S/dmXhlLtYFsOJA/2TFHeu6MyK5maSnDmZ8DLakQZv9WAqmr09dIhpa5tPiJ hG+xi/1GGtYxw== From: Leon Romanovsky To: Marek Szyprowski , Jens Axboe , Christoph Hellwig , Keith Busch Cc: Leon Romanovsky , Jake Edge , Jonathan Corbet , Jason Gunthorpe , Zhu Yanjun , Robin Murphy , Joerg Roedel , Will Deacon , Sagi Grimberg , Bjorn Helgaas , Logan Gunthorpe , Yishai Hadas , Shameer Kolothum , Kevin Tian , Alex Williamson , =?UTF-8?q?J=C3=A9r=C3=B4me=20Glisse?= , Andrew Morton , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-rdma@vger.kernel.org, iommu@lists.linux.dev, linux-nvme@lists.infradead.org, linux-pci@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org, Niklas Schnelle , Chuck Lever , Luis Chamberlain , Matthew Wilcox , Dan Williams , Kanchan Joshi , Chaitanya Kulkarni , Jason Gunthorpe Subject: [PATCH v10 12/24] RDMA/umem: Store ODP access mask information in PFN Date: Mon, 28 Apr 2025 12:22:18 +0300 Message-ID: <96f840717ffaae3dff1063cb2dcb249365159e72.1745831017.git.leon@kernel.org> X-Mailer: git-send-email 2.49.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Leon Romanovsky As a preparation to remove dma_list, store access mask in PFN pointer and not in dma_addr_t. Tested-by: Jens Axboe Reviewed-by: Jason Gunthorpe Signed-off-by: Leon Romanovsky --- drivers/infiniband/core/umem_odp.c | 103 +++++++++++---------------- drivers/infiniband/hw/mlx5/mlx5_ib.h | 1 + drivers/infiniband/hw/mlx5/odp.c | 37 +++++----- drivers/infiniband/sw/rxe/rxe_odp.c | 14 ++-- include/rdma/ib_umem_odp.h | 14 +--- 5 files changed, 70 insertions(+), 99 deletions(-) diff --git a/drivers/infiniband/core/umem_odp.c b/drivers/infiniband/core/u= mem_odp.c index c48ef6083020..2a061f377eb1 100644 --- a/drivers/infiniband/core/umem_odp.c +++ b/drivers/infiniband/core/umem_odp.c @@ -298,22 +298,11 @@ EXPORT_SYMBOL(ib_umem_odp_release); static int ib_umem_odp_map_dma_single_page( struct ib_umem_odp *umem_odp, unsigned int dma_index, - struct page *page, - u64 access_mask) + struct page *page) { struct ib_device *dev =3D umem_odp->umem.ibdev; dma_addr_t *dma_addr =3D &umem_odp->dma_list[dma_index]; =20 - if (*dma_addr) { - /* - * If the page is already dma mapped it means it went through - * a non-invalidating trasition, like read-only to writable. - * Resync the flags. - */ - *dma_addr =3D (*dma_addr & ODP_DMA_ADDR_MASK) | access_mask; - return 0; - } - *dma_addr =3D ib_dma_map_page(dev, page, 0, 1 << umem_odp->page_shift, DMA_BIDIRECTIONAL); if (ib_dma_mapping_error(dev, *dma_addr)) { @@ -321,7 +310,6 @@ static int ib_umem_odp_map_dma_single_page( return -EFAULT; } umem_odp->npages++; - *dma_addr |=3D access_mask; return 0; } =20 @@ -357,9 +345,6 @@ int ib_umem_odp_map_dma_and_lock(struct ib_umem_odp *um= em_odp, u64 user_virt, struct hmm_range range =3D {}; unsigned long timeout; =20 - if (access_mask =3D=3D 0) - return -EINVAL; - if (user_virt < ib_umem_start(umem_odp) || user_virt + bcnt > ib_umem_end(umem_odp)) return -EFAULT; @@ -385,7 +370,7 @@ int ib_umem_odp_map_dma_and_lock(struct ib_umem_odp *um= em_odp, u64 user_virt, if (fault) { range.default_flags =3D HMM_PFN_REQ_FAULT; =20 - if (access_mask & ODP_WRITE_ALLOWED_BIT) + if (access_mask & HMM_PFN_WRITE) range.default_flags |=3D HMM_PFN_REQ_WRITE; } =20 @@ -417,22 +402,17 @@ int ib_umem_odp_map_dma_and_lock(struct ib_umem_odp *= umem_odp, u64 user_virt, for (pfn_index =3D 0; pfn_index < num_pfns; pfn_index +=3D 1 << (page_shift - PAGE_SHIFT), dma_index++) { =20 - if (fault) { - /* - * Since we asked for hmm_range_fault() to populate - * pages it shouldn't return an error entry on success. - */ - WARN_ON(range.hmm_pfns[pfn_index] & HMM_PFN_ERROR); - WARN_ON(!(range.hmm_pfns[pfn_index] & HMM_PFN_VALID)); - } else { - if (!(range.hmm_pfns[pfn_index] & HMM_PFN_VALID)) { - WARN_ON(umem_odp->dma_list[dma_index]); - continue; - } - access_mask =3D ODP_READ_ALLOWED_BIT; - if (range.hmm_pfns[pfn_index] & HMM_PFN_WRITE) - access_mask |=3D ODP_WRITE_ALLOWED_BIT; - } + /* + * Since we asked for hmm_range_fault() to populate + * pages it shouldn't return an error entry on success. + */ + WARN_ON(fault && range.hmm_pfns[pfn_index] & HMM_PFN_ERROR); + WARN_ON(fault && !(range.hmm_pfns[pfn_index] & HMM_PFN_VALID)); + if (!(range.hmm_pfns[pfn_index] & HMM_PFN_VALID)) + continue; + + if (range.hmm_pfns[pfn_index] & HMM_PFN_DMA_MAPPED) + continue; =20 hmm_order =3D hmm_pfn_to_map_order(range.hmm_pfns[pfn_index]); /* If a hugepage was detected and ODP wasn't set for, the umem @@ -447,13 +427,14 @@ int ib_umem_odp_map_dma_and_lock(struct ib_umem_odp *= umem_odp, u64 user_virt, } =20 ret =3D ib_umem_odp_map_dma_single_page( - umem_odp, dma_index, hmm_pfn_to_page(range.hmm_pfns[pfn_index]), - access_mask); + umem_odp, dma_index, + hmm_pfn_to_page(range.hmm_pfns[pfn_index])); if (ret < 0) { ibdev_dbg(umem_odp->umem.ibdev, "ib_umem_odp_map_dma_single_page failed with error %d\n", ret); break; } + range.hmm_pfns[pfn_index] |=3D HMM_PFN_DMA_MAPPED; } /* upon success lock should stay on hold for the callee */ if (!ret) @@ -473,7 +454,6 @@ EXPORT_SYMBOL(ib_umem_odp_map_dma_and_lock); void ib_umem_odp_unmap_dma_pages(struct ib_umem_odp *umem_odp, u64 virt, u64 bound) { - dma_addr_t dma_addr; dma_addr_t dma; int idx; u64 addr; @@ -484,34 +464,37 @@ void ib_umem_odp_unmap_dma_pages(struct ib_umem_odp *= umem_odp, u64 virt, virt =3D max_t(u64, virt, ib_umem_start(umem_odp)); bound =3D min_t(u64, bound, ib_umem_end(umem_odp)); for (addr =3D virt; addr < bound; addr +=3D BIT(umem_odp->page_shift)) { + unsigned long pfn_idx =3D (addr - ib_umem_start(umem_odp)) >> + PAGE_SHIFT; + struct page *page =3D + hmm_pfn_to_page(umem_odp->pfn_list[pfn_idx]); + idx =3D (addr - ib_umem_start(umem_odp)) >> umem_odp->page_shift; dma =3D umem_odp->dma_list[idx]; =20 - /* The access flags guaranteed a valid DMA address in case was NULL */ - if (dma) { - unsigned long pfn_idx =3D (addr - ib_umem_start(umem_odp)) >> PAGE_SHIF= T; - struct page *page =3D hmm_pfn_to_page(umem_odp->pfn_list[pfn_idx]); - - dma_addr =3D dma & ODP_DMA_ADDR_MASK; - ib_dma_unmap_page(dev, dma_addr, - BIT(umem_odp->page_shift), - DMA_BIDIRECTIONAL); - if (dma & ODP_WRITE_ALLOWED_BIT) { - struct page *head_page =3D compound_head(page); - /* - * set_page_dirty prefers being called with - * the page lock. However, MMU notifiers are - * called sometimes with and sometimes without - * the lock. We rely on the umem_mutex instead - * to prevent other mmu notifiers from - * continuing and allowing the page mapping to - * be removed. - */ - set_page_dirty(head_page); - } - umem_odp->dma_list[idx] =3D 0; - umem_odp->npages--; + if (!(umem_odp->pfn_list[pfn_idx] & HMM_PFN_VALID)) + goto clear; + if (!(umem_odp->pfn_list[pfn_idx] & HMM_PFN_DMA_MAPPED)) + goto clear; + + ib_dma_unmap_page(dev, dma, BIT(umem_odp->page_shift), + DMA_BIDIRECTIONAL); + if (umem_odp->pfn_list[pfn_idx] & HMM_PFN_WRITE) { + struct page *head_page =3D compound_head(page); + /* + * set_page_dirty prefers being called with + * the page lock. However, MMU notifiers are + * called sometimes with and sometimes without + * the lock. We rely on the umem_mutex instead + * to prevent other mmu notifiers from + * continuing and allowing the page mapping to + * be removed. + */ + set_page_dirty(head_page); } + umem_odp->npages--; +clear: + umem_odp->pfn_list[pfn_idx] &=3D ~HMM_PFN_FLAGS; } } EXPORT_SYMBOL(ib_umem_odp_unmap_dma_pages); diff --git a/drivers/infiniband/hw/mlx5/mlx5_ib.h b/drivers/infiniband/hw/m= lx5/mlx5_ib.h index ace2df3e1d9f..7424b23bb0d9 100644 --- a/drivers/infiniband/hw/mlx5/mlx5_ib.h +++ b/drivers/infiniband/hw/mlx5/mlx5_ib.h @@ -351,6 +351,7 @@ struct mlx5_ib_flow_db { #define MLX5_IB_UPD_XLT_PD BIT(4) #define MLX5_IB_UPD_XLT_ACCESS BIT(5) #define MLX5_IB_UPD_XLT_INDIRECT BIT(6) +#define MLX5_IB_UPD_XLT_DOWNGRADE BIT(7) =20 /* Private QP creation flags to be passed in ib_qp_init_attr.create_flags. * diff --git a/drivers/infiniband/hw/mlx5/odp.c b/drivers/infiniband/hw/mlx5/= odp.c index 86d8fa63bf69..6074c541885c 100644 --- a/drivers/infiniband/hw/mlx5/odp.c +++ b/drivers/infiniband/hw/mlx5/odp.c @@ -34,6 +34,7 @@ #include #include #include +#include =20 #include "mlx5_ib.h" #include "cmd.h" @@ -158,22 +159,12 @@ static void populate_klm(struct mlx5_klm *pklm, size_= t idx, size_t nentries, } } =20 -static u64 umem_dma_to_mtt(dma_addr_t umem_dma) -{ - u64 mtt_entry =3D umem_dma & ODP_DMA_ADDR_MASK; - - if (umem_dma & ODP_READ_ALLOWED_BIT) - mtt_entry |=3D MLX5_IB_MTT_READ; - if (umem_dma & ODP_WRITE_ALLOWED_BIT) - mtt_entry |=3D MLX5_IB_MTT_WRITE; - - return mtt_entry; -} - static void populate_mtt(__be64 *pas, size_t idx, size_t nentries, struct mlx5_ib_mr *mr, int flags) { struct ib_umem_odp *odp =3D to_ib_umem_odp(mr->umem); + bool downgrade =3D flags & MLX5_IB_UPD_XLT_DOWNGRADE; + unsigned long pfn; dma_addr_t pa; size_t i; =20 @@ -181,8 +172,17 @@ static void populate_mtt(__be64 *pas, size_t idx, size= _t nentries, return; =20 for (i =3D 0; i < nentries; i++) { + pfn =3D odp->pfn_list[idx + i]; + if (!(pfn & HMM_PFN_VALID)) + /* ODP initialization */ + continue; + pa =3D odp->dma_list[idx + i]; - pas[i] =3D cpu_to_be64(umem_dma_to_mtt(pa)); + pa |=3D MLX5_IB_MTT_READ; + if ((pfn & HMM_PFN_WRITE) && !downgrade) + pa |=3D MLX5_IB_MTT_WRITE; + + pas[i] =3D cpu_to_be64(pa); } } =20 @@ -303,8 +303,7 @@ static bool mlx5_ib_invalidate_range(struct mmu_interva= l_notifier *mni, * estimate the cost of another UMR vs. the cost of bigger * UMR. */ - if (umem_odp->dma_list[idx] & - (ODP_READ_ALLOWED_BIT | ODP_WRITE_ALLOWED_BIT)) { + if (umem_odp->pfn_list[idx] & HMM_PFN_VALID) { if (!in_block) { blk_start_idx =3D idx; in_block =3D 1; @@ -687,7 +686,7 @@ static int pagefault_real_mr(struct mlx5_ib_mr *mr, str= uct ib_umem_odp *odp, { int page_shift, ret, np; bool downgrade =3D flags & MLX5_PF_FLAGS_DOWNGRADE; - u64 access_mask; + u64 access_mask =3D 0; u64 start_idx; bool fault =3D !(flags & MLX5_PF_FLAGS_SNAPSHOT); u32 xlt_flags =3D MLX5_IB_UPD_XLT_ATOMIC; @@ -695,12 +694,14 @@ static int pagefault_real_mr(struct mlx5_ib_mr *mr, s= truct ib_umem_odp *odp, if (flags & MLX5_PF_FLAGS_ENABLE) xlt_flags |=3D MLX5_IB_UPD_XLT_ENABLE; =20 + if (flags & MLX5_PF_FLAGS_DOWNGRADE) + xlt_flags |=3D MLX5_IB_UPD_XLT_DOWNGRADE; + page_shift =3D odp->page_shift; start_idx =3D (user_va - ib_umem_start(odp)) >> page_shift; - access_mask =3D ODP_READ_ALLOWED_BIT; =20 if (odp->umem.writable && !downgrade) - access_mask |=3D ODP_WRITE_ALLOWED_BIT; + access_mask |=3D HMM_PFN_WRITE; =20 np =3D ib_umem_odp_map_dma_and_lock(odp, user_va, bcnt, access_mask, faul= t); if (np < 0) diff --git a/drivers/infiniband/sw/rxe/rxe_odp.c b/drivers/infiniband/sw/rx= e/rxe_odp.c index 9f6e2bb2a269..caedd9ef2fe3 100644 --- a/drivers/infiniband/sw/rxe/rxe_odp.c +++ b/drivers/infiniband/sw/rxe/rxe_odp.c @@ -26,7 +26,7 @@ static bool rxe_ib_invalidate_range(struct mmu_interval_n= otifier *mni, start =3D max_t(u64, ib_umem_start(umem_odp), range->start); end =3D min_t(u64, ib_umem_end(umem_odp), range->end); =20 - /* update umem_odp->dma_list */ + /* update umem_odp->map.pfn_list */ ib_umem_odp_unmap_dma_pages(umem_odp, start, end); =20 mutex_unlock(&umem_odp->umem_mutex); @@ -44,12 +44,11 @@ static int rxe_odp_do_pagefault_and_lock(struct rxe_mr = *mr, u64 user_va, int bcn { struct ib_umem_odp *umem_odp =3D to_ib_umem_odp(mr->umem); bool fault =3D !(flags & RXE_PAGEFAULT_SNAPSHOT); - u64 access_mask; + u64 access_mask =3D 0; int np; =20 - access_mask =3D ODP_READ_ALLOWED_BIT; if (umem_odp->umem.writable && !(flags & RXE_PAGEFAULT_RDONLY)) - access_mask |=3D ODP_WRITE_ALLOWED_BIT; + access_mask |=3D HMM_PFN_WRITE; =20 /* * ib_umem_odp_map_dma_and_lock() locks umem_mutex on success. @@ -137,7 +136,7 @@ static inline bool rxe_check_pagefault(struct ib_umem_o= dp *umem_odp, while (addr < iova + length) { idx =3D (addr - ib_umem_start(umem_odp)) >> umem_odp->page_shift; =20 - if (!(umem_odp->dma_list[idx] & perm)) { + if (!(umem_odp->map.pfn_list[idx] & perm)) { need_fault =3D true; break; } @@ -151,15 +150,14 @@ static int rxe_odp_map_range_and_lock(struct rxe_mr *= mr, u64 iova, int length, u { struct ib_umem_odp *umem_odp =3D to_ib_umem_odp(mr->umem); bool need_fault; - u64 perm; + u64 perm =3D 0; int err; =20 if (unlikely(length < 1)) return -EINVAL; =20 - perm =3D ODP_READ_ALLOWED_BIT; if (!(flags & RXE_PAGEFAULT_RDONLY)) - perm |=3D ODP_WRITE_ALLOWED_BIT; + perm |=3D HMM_PFN_WRITE; =20 mutex_lock(&umem_odp->umem_mutex); =20 diff --git a/include/rdma/ib_umem_odp.h b/include/rdma/ib_umem_odp.h index 0844c1d05ac6..a345c26a745d 100644 --- a/include/rdma/ib_umem_odp.h +++ b/include/rdma/ib_umem_odp.h @@ -8,6 +8,7 @@ =20 #include #include +#include =20 struct ib_umem_odp { struct ib_umem umem; @@ -67,19 +68,6 @@ static inline size_t ib_umem_odp_num_pages(struct ib_ume= m_odp *umem_odp) umem_odp->page_shift; } =20 -/* - * The lower 2 bits of the DMA address signal the R/W permissions for - * the entry. To upgrade the permissions, provide the appropriate - * bitmask to the map_dma_pages function. - * - * Be aware that upgrading a mapped address might result in change of - * the DMA address for the page. - */ -#define ODP_READ_ALLOWED_BIT (1<<0ULL) -#define ODP_WRITE_ALLOWED_BIT (1<<1ULL) - -#define ODP_DMA_ADDR_MASK (~(ODP_READ_ALLOWED_BIT | ODP_WRITE_ALLOWED_BIT)) - #ifdef CONFIG_INFINIBAND_ON_DEMAND_PAGING =20 struct ib_umem_odp * --=20 2.49.0 From nobody Tue Dec 16 14:20:07 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DB12225F78D; Mon, 28 Apr 2025 09:23:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745832217; cv=none; b=M1OV4Z8pfEvLJQCcqtVLgKnn8h9YalBQ65D+jMdmGMS4MssU18PXi5tikmfLN1k3YVXpJcPdHEMb6DO9UrQg53jWm/gFRcMBn63NO7Zc8irXzCjAJdZRh/mOG4MCyGRAG+c5TKUZWG+gmxU9GV129kcL632WydouwdVv536ffRc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745832217; c=relaxed/simple; bh=kKR0gWMxXwuxAqyCpk8u1WKfpWSe7bGL6lJ6hc6zY74=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Jw4u0sPWA+y1Uxk7KQD2bDJK1YFbgs3R6nqZyX3yG7Awab/W2Sz/60+aO+WREIblAeMaKdhdgnVmgRpYNhEQQ3j7cdeViCG3JErgSMCU4Lu9SydQk8f/XtqBFWVDScldr6UIUKhxlWY7U1YYJM9oxPBL4AOfKmj37+KrcJBKZpY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=I/MRQm1R; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="I/MRQm1R" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1A49AC4CEE4; Mon, 28 Apr 2025 09:23:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1745832216; bh=kKR0gWMxXwuxAqyCpk8u1WKfpWSe7bGL6lJ6hc6zY74=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=I/MRQm1RNUdhdRmyzahXkH9m1dP9HrwVz9Yt0EK2hST4asaK0ilMIJDAuMfKuectq XufC6Ox3dOgjI5FkHyGAJFP20qqXfxI0laK4UMXBiEOfFgUUr48xPpRl5OoVqHaElY 8OmTF7O/GIZVjtlcYeDpoOI+kymIv7ou1hXJGZ+QYrbUYaLhSGkF4BJVNwdDk4B2/t l+LR20o4arELgi5P/SWpzMZT6Mozma3P6JHXd0cRPrhCXlmrwcryOzeuQK/MnnPQvh VbjYfz459r0hLPalUPgduJv5IEJ3V/6UqENjLgMeyKM7ovQtMXtQN7s3rBRTIL3nH3 TiNhbi/0b8W/g== From: Leon Romanovsky To: Marek Szyprowski , Jens Axboe , Christoph Hellwig , Keith Busch Cc: Leon Romanovsky , Jake Edge , Jonathan Corbet , Jason Gunthorpe , Zhu Yanjun , Robin Murphy , Joerg Roedel , Will Deacon , Sagi Grimberg , Bjorn Helgaas , Logan Gunthorpe , Yishai Hadas , Shameer Kolothum , Kevin Tian , Alex Williamson , =?UTF-8?q?J=C3=A9r=C3=B4me=20Glisse?= , Andrew Morton , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-rdma@vger.kernel.org, iommu@lists.linux.dev, linux-nvme@lists.infradead.org, linux-pci@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org, Niklas Schnelle , Chuck Lever , Luis Chamberlain , Matthew Wilcox , Dan Williams , Kanchan Joshi , Chaitanya Kulkarni , Jason Gunthorpe Subject: [PATCH v10 13/24] RDMA/core: Convert UMEM ODP DMA mapping to caching IOVA and page linkage Date: Mon, 28 Apr 2025 12:22:19 +0300 Message-ID: <049945137f8d440071f1f9ecf2fc86afdb4f406c.1745831017.git.leon@kernel.org> X-Mailer: git-send-email 2.49.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Leon Romanovsky Reuse newly added DMA API to cache IOVA and only link/unlink pages in fast path for UMEM ODP flow. Tested-by: Jens Axboe Reviewed-by: Jason Gunthorpe Signed-off-by: Leon Romanovsky --- drivers/infiniband/core/umem_odp.c | 106 ++++++--------------------- drivers/infiniband/hw/mlx5/mlx5_ib.h | 11 ++- drivers/infiniband/hw/mlx5/odp.c | 40 ++++++---- drivers/infiniband/hw/mlx5/umr.c | 12 ++- drivers/infiniband/sw/rxe/rxe_odp.c | 4 +- include/rdma/ib_umem_odp.h | 13 +--- 6 files changed, 71 insertions(+), 115 deletions(-) diff --git a/drivers/infiniband/core/umem_odp.c b/drivers/infiniband/core/u= mem_odp.c index 2a061f377eb1..30cd8f353476 100644 --- a/drivers/infiniband/core/umem_odp.c +++ b/drivers/infiniband/core/umem_odp.c @@ -41,6 +41,7 @@ #include #include #include +#include #include =20 #include @@ -50,6 +51,7 @@ static inline int ib_init_umem_odp(struct ib_umem_odp *umem_odp, const struct mmu_interval_notifier_ops *ops) { + struct ib_device *dev =3D umem_odp->umem.ibdev; int ret; =20 umem_odp->umem.is_odp =3D 1; @@ -59,7 +61,6 @@ static inline int ib_init_umem_odp(struct ib_umem_odp *um= em_odp, size_t page_size =3D 1UL << umem_odp->page_shift; unsigned long start; unsigned long end; - size_t ndmas, npfns; =20 start =3D ALIGN_DOWN(umem_odp->umem.address, page_size); if (check_add_overflow(umem_odp->umem.address, @@ -70,38 +71,23 @@ static inline int ib_init_umem_odp(struct ib_umem_odp *= umem_odp, if (unlikely(end < page_size)) return -EOVERFLOW; =20 - ndmas =3D (end - start) >> umem_odp->page_shift; - if (!ndmas) - return -EINVAL; - - npfns =3D (end - start) >> PAGE_SHIFT; - umem_odp->pfn_list =3D kvcalloc( - npfns, sizeof(*umem_odp->pfn_list), - GFP_KERNEL | __GFP_NOWARN); - if (!umem_odp->pfn_list) - return -ENOMEM; - - umem_odp->dma_list =3D kvcalloc( - ndmas, sizeof(*umem_odp->dma_list), - GFP_KERNEL | __GFP_NOWARN); - if (!umem_odp->dma_list) { - ret =3D -ENOMEM; - goto out_pfn_list; - } + ret =3D hmm_dma_map_alloc(dev->dma_device, &umem_odp->map, + (end - start) >> PAGE_SHIFT, + 1 << umem_odp->page_shift); + if (ret) + return ret; =20 ret =3D mmu_interval_notifier_insert(&umem_odp->notifier, umem_odp->umem.owning_mm, start, end - start, ops); if (ret) - goto out_dma_list; + goto out_free_map; } =20 return 0; =20 -out_dma_list: - kvfree(umem_odp->dma_list); -out_pfn_list: - kvfree(umem_odp->pfn_list); +out_free_map: + hmm_dma_map_free(dev->dma_device, &umem_odp->map); return ret; } =20 @@ -264,6 +250,8 @@ EXPORT_SYMBOL(ib_umem_odp_get); =20 void ib_umem_odp_release(struct ib_umem_odp *umem_odp) { + struct ib_device *dev =3D umem_odp->umem.ibdev; + /* * Ensure that no more pages are mapped in the umem. * @@ -276,48 +264,17 @@ void ib_umem_odp_release(struct ib_umem_odp *umem_odp) ib_umem_end(umem_odp)); mutex_unlock(&umem_odp->umem_mutex); mmu_interval_notifier_remove(&umem_odp->notifier); - kvfree(umem_odp->dma_list); - kvfree(umem_odp->pfn_list); + hmm_dma_map_free(dev->dma_device, &umem_odp->map); } put_pid(umem_odp->tgid); kfree(umem_odp); } EXPORT_SYMBOL(ib_umem_odp_release); =20 -/* - * Map for DMA and insert a single page into the on-demand paging page tab= les. - * - * @umem: the umem to insert the page to. - * @dma_index: index in the umem to add the dma to. - * @page: the page struct to map and add. - * @access_mask: access permissions needed for this page. - * - * The function returns -EFAULT if the DMA mapping operation fails. - * - */ -static int ib_umem_odp_map_dma_single_page( - struct ib_umem_odp *umem_odp, - unsigned int dma_index, - struct page *page) -{ - struct ib_device *dev =3D umem_odp->umem.ibdev; - dma_addr_t *dma_addr =3D &umem_odp->dma_list[dma_index]; - - *dma_addr =3D ib_dma_map_page(dev, page, 0, 1 << umem_odp->page_shift, - DMA_BIDIRECTIONAL); - if (ib_dma_mapping_error(dev, *dma_addr)) { - *dma_addr =3D 0; - return -EFAULT; - } - umem_odp->npages++; - return 0; -} - /** * ib_umem_odp_map_dma_and_lock - DMA map userspace memory in an ODP MR an= d lock it. * * Maps the range passed in the argument to DMA addresses. - * The DMA addresses of the mapped pages is updated in umem_odp->dma_list. * Upon success the ODP MR will be locked to let caller complete its device * page table update. * @@ -374,7 +331,7 @@ int ib_umem_odp_map_dma_and_lock(struct ib_umem_odp *um= em_odp, u64 user_virt, range.default_flags |=3D HMM_PFN_REQ_WRITE; } =20 - range.hmm_pfns =3D &(umem_odp->pfn_list[pfn_start_idx]); + range.hmm_pfns =3D &(umem_odp->map.pfn_list[pfn_start_idx]); timeout =3D jiffies + msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT); =20 retry: @@ -425,16 +382,6 @@ int ib_umem_odp_map_dma_and_lock(struct ib_umem_odp *u= mem_odp, u64 user_virt, __func__, hmm_order, page_shift); break; } - - ret =3D ib_umem_odp_map_dma_single_page( - umem_odp, dma_index, - hmm_pfn_to_page(range.hmm_pfns[pfn_index])); - if (ret < 0) { - ibdev_dbg(umem_odp->umem.ibdev, - "ib_umem_odp_map_dma_single_page failed with error %d\n", ret); - break; - } - range.hmm_pfns[pfn_index] |=3D HMM_PFN_DMA_MAPPED; } /* upon success lock should stay on hold for the callee */ if (!ret) @@ -454,32 +401,23 @@ EXPORT_SYMBOL(ib_umem_odp_map_dma_and_lock); void ib_umem_odp_unmap_dma_pages(struct ib_umem_odp *umem_odp, u64 virt, u64 bound) { - dma_addr_t dma; - int idx; - u64 addr; struct ib_device *dev =3D umem_odp->umem.ibdev; + u64 addr; =20 lockdep_assert_held(&umem_odp->umem_mutex); =20 virt =3D max_t(u64, virt, ib_umem_start(umem_odp)); bound =3D min_t(u64, bound, ib_umem_end(umem_odp)); for (addr =3D virt; addr < bound; addr +=3D BIT(umem_odp->page_shift)) { - unsigned long pfn_idx =3D (addr - ib_umem_start(umem_odp)) >> - PAGE_SHIFT; - struct page *page =3D - hmm_pfn_to_page(umem_odp->pfn_list[pfn_idx]); - - idx =3D (addr - ib_umem_start(umem_odp)) >> umem_odp->page_shift; - dma =3D umem_odp->dma_list[idx]; + u64 offset =3D addr - ib_umem_start(umem_odp); + size_t idx =3D offset >> umem_odp->page_shift; + unsigned long pfn =3D umem_odp->map.pfn_list[idx]; =20 - if (!(umem_odp->pfn_list[pfn_idx] & HMM_PFN_VALID)) - goto clear; - if (!(umem_odp->pfn_list[pfn_idx] & HMM_PFN_DMA_MAPPED)) + if (!hmm_dma_unmap_pfn(dev->dma_device, &umem_odp->map, idx)) goto clear; =20 - ib_dma_unmap_page(dev, dma, BIT(umem_odp->page_shift), - DMA_BIDIRECTIONAL); - if (umem_odp->pfn_list[pfn_idx] & HMM_PFN_WRITE) { + if (pfn & HMM_PFN_WRITE) { + struct page *page =3D hmm_pfn_to_page(pfn); struct page *head_page =3D compound_head(page); /* * set_page_dirty prefers being called with @@ -494,7 +432,7 @@ void ib_umem_odp_unmap_dma_pages(struct ib_umem_odp *um= em_odp, u64 virt, } umem_odp->npages--; clear: - umem_odp->pfn_list[pfn_idx] &=3D ~HMM_PFN_FLAGS; + umem_odp->map.pfn_list[idx] &=3D ~HMM_PFN_FLAGS; } } EXPORT_SYMBOL(ib_umem_odp_unmap_dma_pages); diff --git a/drivers/infiniband/hw/mlx5/mlx5_ib.h b/drivers/infiniband/hw/m= lx5/mlx5_ib.h index 7424b23bb0d9..27fbf5f3a2d6 100644 --- a/drivers/infiniband/hw/mlx5/mlx5_ib.h +++ b/drivers/infiniband/hw/mlx5/mlx5_ib.h @@ -1474,8 +1474,8 @@ void mlx5_ib_odp_cleanup_one(struct mlx5_ib_dev *ibde= v); int __init mlx5_ib_odp_init(void); void mlx5_ib_odp_cleanup(void); int mlx5_odp_init_mkey_cache(struct mlx5_ib_dev *dev); -void mlx5_odp_populate_xlt(void *xlt, size_t idx, size_t nentries, - struct mlx5_ib_mr *mr, int flags); +int mlx5_odp_populate_xlt(void *xlt, size_t idx, size_t nentries, + struct mlx5_ib_mr *mr, int flags); =20 int mlx5_ib_advise_mr_prefetch(struct ib_pd *pd, enum ib_uverbs_advise_mr_advice advice, @@ -1496,8 +1496,11 @@ static inline int mlx5_odp_init_mkey_cache(struct ml= x5_ib_dev *dev) { return 0; } -static inline void mlx5_odp_populate_xlt(void *xlt, size_t idx, size_t nen= tries, - struct mlx5_ib_mr *mr, int flags) {} +static inline int mlx5_odp_populate_xlt(void *xlt, size_t idx, size_t nent= ries, + struct mlx5_ib_mr *mr, int flags) +{ + return -EOPNOTSUPP; +} =20 static inline int mlx5_ib_advise_mr_prefetch(struct ib_pd *pd, diff --git a/drivers/infiniband/hw/mlx5/odp.c b/drivers/infiniband/hw/mlx5/= odp.c index 6074c541885c..eaa2f9f5f3a9 100644 --- a/drivers/infiniband/hw/mlx5/odp.c +++ b/drivers/infiniband/hw/mlx5/odp.c @@ -35,6 +35,8 @@ #include #include #include +#include +#include =20 #include "mlx5_ib.h" #include "cmd.h" @@ -159,40 +161,50 @@ static void populate_klm(struct mlx5_klm *pklm, size_= t idx, size_t nentries, } } =20 -static void populate_mtt(__be64 *pas, size_t idx, size_t nentries, - struct mlx5_ib_mr *mr, int flags) +static int populate_mtt(__be64 *pas, size_t start, size_t nentries, + struct mlx5_ib_mr *mr, int flags) { struct ib_umem_odp *odp =3D to_ib_umem_odp(mr->umem); bool downgrade =3D flags & MLX5_IB_UPD_XLT_DOWNGRADE; - unsigned long pfn; - dma_addr_t pa; + struct pci_p2pdma_map_state p2pdma_state =3D {}; + struct ib_device *dev =3D odp->umem.ibdev; size_t i; =20 if (flags & MLX5_IB_UPD_XLT_ZAP) - return; + return 0; =20 for (i =3D 0; i < nentries; i++) { - pfn =3D odp->pfn_list[idx + i]; + unsigned long pfn =3D odp->map.pfn_list[start + i]; + dma_addr_t dma_addr; + + pfn =3D odp->map.pfn_list[start + i]; if (!(pfn & HMM_PFN_VALID)) /* ODP initialization */ continue; =20 - pa =3D odp->dma_list[idx + i]; - pa |=3D MLX5_IB_MTT_READ; + dma_addr =3D hmm_dma_map_pfn(dev->dma_device, &odp->map, + start + i, &p2pdma_state); + if (ib_dma_mapping_error(dev, dma_addr)) + return -EFAULT; + + dma_addr |=3D MLX5_IB_MTT_READ; if ((pfn & HMM_PFN_WRITE) && !downgrade) - pa |=3D MLX5_IB_MTT_WRITE; + dma_addr |=3D MLX5_IB_MTT_WRITE; =20 - pas[i] =3D cpu_to_be64(pa); + pas[i] =3D cpu_to_be64(dma_addr); + odp->npages++; } + return 0; } =20 -void mlx5_odp_populate_xlt(void *xlt, size_t idx, size_t nentries, - struct mlx5_ib_mr *mr, int flags) +int mlx5_odp_populate_xlt(void *xlt, size_t idx, size_t nentries, + struct mlx5_ib_mr *mr, int flags) { if (flags & MLX5_IB_UPD_XLT_INDIRECT) { populate_klm(xlt, idx, nentries, mr, flags); + return 0; } else { - populate_mtt(xlt, idx, nentries, mr, flags); + return populate_mtt(xlt, idx, nentries, mr, flags); } } =20 @@ -303,7 +315,7 @@ static bool mlx5_ib_invalidate_range(struct mmu_interva= l_notifier *mni, * estimate the cost of another UMR vs. the cost of bigger * UMR. */ - if (umem_odp->pfn_list[idx] & HMM_PFN_VALID) { + if (umem_odp->map.pfn_list[idx] & HMM_PFN_VALID) { if (!in_block) { blk_start_idx =3D idx; in_block =3D 1; diff --git a/drivers/infiniband/hw/mlx5/umr.c b/drivers/infiniband/hw/mlx5/= umr.c index 793f3c5c4d01..5be4426a2884 100644 --- a/drivers/infiniband/hw/mlx5/umr.c +++ b/drivers/infiniband/hw/mlx5/umr.c @@ -840,7 +840,17 @@ int mlx5r_umr_update_xlt(struct mlx5_ib_mr *mr, u64 id= x, int npages, size_to_map =3D npages * desc_size; dma_sync_single_for_cpu(ddev, sg.addr, sg.length, DMA_TO_DEVICE); - mlx5_odp_populate_xlt(xlt, idx, npages, mr, flags); + /* + * npages is the maximum number of pages to map, but we + * can't guarantee that all pages are actually mapped. + * + * For example, if page is p2p of type which is not supported + * for mapping, the number of pages mapped will be less than + * requested. + */ + err =3D mlx5_odp_populate_xlt(xlt, idx, npages, mr, flags); + if (err) + return err; dma_sync_single_for_device(ddev, sg.addr, sg.length, DMA_TO_DEVICE); sg.length =3D ALIGN(size_to_map, MLX5_UMR_FLEX_ALIGNMENT); diff --git a/drivers/infiniband/sw/rxe/rxe_odp.c b/drivers/infiniband/sw/rx= e/rxe_odp.c index caedd9ef2fe3..51b28f5d5027 100644 --- a/drivers/infiniband/sw/rxe/rxe_odp.c +++ b/drivers/infiniband/sw/rxe/rxe_odp.c @@ -194,7 +194,7 @@ static int __rxe_odp_mr_copy(struct rxe_mr *mr, u64 iov= a, void *addr, while (length > 0) { u8 *src, *dest; =20 - page =3D hmm_pfn_to_page(umem_odp->pfn_list[idx]); + page =3D hmm_pfn_to_page(umem_odp->map.pfn_list[idx]); user_va =3D kmap_local_page(page); if (!user_va) return -EFAULT; @@ -277,7 +277,7 @@ static int rxe_odp_do_atomic_op(struct rxe_mr *mr, u64 = iova, int opcode, =20 idx =3D (iova - ib_umem_start(umem_odp)) >> umem_odp->page_shift; page_offset =3D iova & (BIT(umem_odp->page_shift) - 1); - page =3D hmm_pfn_to_page(umem_odp->pfn_list[idx]); + page =3D hmm_pfn_to_page(umem_odp->map.pfn_list[idx]); if (!page) return RESPST_ERR_RKEY_VIOLATION; =20 diff --git a/include/rdma/ib_umem_odp.h b/include/rdma/ib_umem_odp.h index a345c26a745d..2a24bf791c10 100644 --- a/include/rdma/ib_umem_odp.h +++ b/include/rdma/ib_umem_odp.h @@ -8,24 +8,17 @@ =20 #include #include -#include +#include =20 struct ib_umem_odp { struct ib_umem umem; struct mmu_interval_notifier notifier; struct pid *tgid; =20 - /* An array of the pfns included in the on-demand paging umem. */ - unsigned long *pfn_list; + struct hmm_dma_map map; =20 /* - * An array with DMA addresses mapped for pfns in pfn_list. - * The lower two bits designate access permissions. - * See ODP_READ_ALLOWED_BIT and ODP_WRITE_ALLOWED_BIT. - */ - dma_addr_t *dma_list; - /* - * The umem_mutex protects the page_list and dma_list fields of an ODP + * The umem_mutex protects the page_list field of an ODP * umem, allowing only a single thread to map/unmap pages. The mutex * also protects access to the mmu notifier counters. */ --=20 2.49.0 From nobody Tue Dec 16 14:20:07 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F2CEE25F793; Mon, 28 Apr 2025 09:23:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745832221; cv=none; b=Y06bLu8fjslPkOdsHHnMhZlz1Ri7pbkJfG0aUFlX9OyURzCygdN0NbCIKM+Co834/WLAElK6iJx0t2djLyma0SJHMIfvmTBIjU71Dm9E0B96gbEAL6jRYz/myJyBYqiyoyl4flrYzxvdgcv+LQUXgFqROEBf0Z1HCTWGjfrBUUk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745832221; c=relaxed/simple; bh=xVzZEuYjMgXmdxhdT4tLYQ61KOWUbydpHLJY/2wJX2k=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=MKDmRG0y3piFJ4ra4gALVZXxJDytDXHmv24TgPq5GlhZK6g1h+eQ2DB65UUIXQm8tE7bZQlBq/Xwnpti6ULRwoOqwkfykJEWSmfFREpNWiRoH0QpcoqDF93k4XOc1WNVxPvYGPUEIFpH9vyOLKaNz1CYN38wtJFy66oinBmsmCQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=CshCyLDL; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="CshCyLDL" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 36441C4CEE9; Mon, 28 Apr 2025 09:23:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1745832220; bh=xVzZEuYjMgXmdxhdT4tLYQ61KOWUbydpHLJY/2wJX2k=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=CshCyLDL08MB+t5tQ+3u1RLskoUp659Qjdk5H7jw4k/r+x2C++hcJ5Vt1nSb7ZO3U +SSas4L/armZLzifrw9jcpJfzWNrx38YknuiOMHG2P7Zaa7ra2svEohZSHVlr02x8z aZ3bsSdD2+vG4d2Q/SxaaGipOlDQpzU2ODxs8RtimyHJ/JQ2jByTEay9Eub68Ty+w8 4G0V4sO5gNilpnD5KlnuHNDM1FP+erHsxpnhStL+m0vSsyOGDSfZwFUwoyfPajmUiN 3Kocq6tQXP4YylGjbq9dMSNjZrDz9YgG9kg8u2LUEh8kAOMonrxc4GWJ97WOhrA710 UchdfEMlCo6mg== From: Leon Romanovsky To: Marek Szyprowski , Jens Axboe , Christoph Hellwig , Keith Busch Cc: Leon Romanovsky , Jake Edge , Jonathan Corbet , Jason Gunthorpe , Zhu Yanjun , Robin Murphy , Joerg Roedel , Will Deacon , Sagi Grimberg , Bjorn Helgaas , Logan Gunthorpe , Yishai Hadas , Shameer Kolothum , Kevin Tian , Alex Williamson , =?UTF-8?q?J=C3=A9r=C3=B4me=20Glisse?= , Andrew Morton , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-rdma@vger.kernel.org, iommu@lists.linux.dev, linux-nvme@lists.infradead.org, linux-pci@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org, Niklas Schnelle , Chuck Lever , Luis Chamberlain , Matthew Wilcox , Dan Williams , Kanchan Joshi , Chaitanya Kulkarni , Jason Gunthorpe Subject: [PATCH v10 14/24] RDMA/umem: Separate implicit ODP initialization from explicit ODP Date: Mon, 28 Apr 2025 12:22:20 +0300 Message-ID: <25efd635951d87817f72220c8b38757b3e1a555a.1745831017.git.leon@kernel.org> X-Mailer: git-send-email 2.49.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Leon Romanovsky Create separate functions for the implicit ODP initialization which is different from the explicit ODP initialization. Tested-by: Jens Axboe Reviewed-by: Jason Gunthorpe Signed-off-by: Leon Romanovsky --- drivers/infiniband/core/umem_odp.c | 91 +++++++++++++++--------------- 1 file changed, 46 insertions(+), 45 deletions(-) diff --git a/drivers/infiniband/core/umem_odp.c b/drivers/infiniband/core/u= mem_odp.c index 30cd8f353476..51d518989914 100644 --- a/drivers/infiniband/core/umem_odp.c +++ b/drivers/infiniband/core/umem_odp.c @@ -48,41 +48,44 @@ =20 #include "uverbs.h" =20 -static inline int ib_init_umem_odp(struct ib_umem_odp *umem_odp, - const struct mmu_interval_notifier_ops *ops) +static void ib_init_umem_implicit_odp(struct ib_umem_odp *umem_odp) +{ + umem_odp->is_implicit_odp =3D 1; + umem_odp->umem.is_odp =3D 1; + mutex_init(&umem_odp->umem_mutex); +} + +static int ib_init_umem_odp(struct ib_umem_odp *umem_odp, + const struct mmu_interval_notifier_ops *ops) { struct ib_device *dev =3D umem_odp->umem.ibdev; + size_t page_size =3D 1UL << umem_odp->page_shift; + unsigned long start; + unsigned long end; int ret; =20 umem_odp->umem.is_odp =3D 1; mutex_init(&umem_odp->umem_mutex); =20 - if (!umem_odp->is_implicit_odp) { - size_t page_size =3D 1UL << umem_odp->page_shift; - unsigned long start; - unsigned long end; - - start =3D ALIGN_DOWN(umem_odp->umem.address, page_size); - if (check_add_overflow(umem_odp->umem.address, - (unsigned long)umem_odp->umem.length, - &end)) - return -EOVERFLOW; - end =3D ALIGN(end, page_size); - if (unlikely(end < page_size)) - return -EOVERFLOW; - - ret =3D hmm_dma_map_alloc(dev->dma_device, &umem_odp->map, - (end - start) >> PAGE_SHIFT, - 1 << umem_odp->page_shift); - if (ret) - return ret; - - ret =3D mmu_interval_notifier_insert(&umem_odp->notifier, - umem_odp->umem.owning_mm, - start, end - start, ops); - if (ret) - goto out_free_map; - } + start =3D ALIGN_DOWN(umem_odp->umem.address, page_size); + if (check_add_overflow(umem_odp->umem.address, + (unsigned long)umem_odp->umem.length, &end)) + return -EOVERFLOW; + end =3D ALIGN(end, page_size); + if (unlikely(end < page_size)) + return -EOVERFLOW; + + ret =3D hmm_dma_map_alloc(dev->dma_device, &umem_odp->map, + (end - start) >> PAGE_SHIFT, + 1 << umem_odp->page_shift); + if (ret) + return ret; + + ret =3D mmu_interval_notifier_insert(&umem_odp->notifier, + umem_odp->umem.owning_mm, start, + end - start, ops); + if (ret) + goto out_free_map; =20 return 0; =20 @@ -106,7 +109,6 @@ struct ib_umem_odp *ib_umem_odp_alloc_implicit(struct i= b_device *device, { struct ib_umem *umem; struct ib_umem_odp *umem_odp; - int ret; =20 if (access & IB_ACCESS_HUGETLB) return ERR_PTR(-EINVAL); @@ -118,16 +120,10 @@ struct ib_umem_odp *ib_umem_odp_alloc_implicit(struct= ib_device *device, umem->ibdev =3D device; umem->writable =3D ib_access_writable(access); umem->owning_mm =3D current->mm; - umem_odp->is_implicit_odp =3D 1; umem_odp->page_shift =3D PAGE_SHIFT; =20 umem_odp->tgid =3D get_task_pid(current->group_leader, PIDTYPE_PID); - ret =3D ib_init_umem_odp(umem_odp, NULL); - if (ret) { - put_pid(umem_odp->tgid); - kfree(umem_odp); - return ERR_PTR(ret); - } + ib_init_umem_implicit_odp(umem_odp); return umem_odp; } EXPORT_SYMBOL(ib_umem_odp_alloc_implicit); @@ -248,7 +244,7 @@ struct ib_umem_odp *ib_umem_odp_get(struct ib_device *d= evice, } EXPORT_SYMBOL(ib_umem_odp_get); =20 -void ib_umem_odp_release(struct ib_umem_odp *umem_odp) +static void ib_umem_odp_free(struct ib_umem_odp *umem_odp) { struct ib_device *dev =3D umem_odp->umem.ibdev; =20 @@ -258,14 +254,19 @@ void ib_umem_odp_release(struct ib_umem_odp *umem_odp) * It is the driver's responsibility to ensure, before calling us, * that the hardware will not attempt to access the MR any more. */ - if (!umem_odp->is_implicit_odp) { - mutex_lock(&umem_odp->umem_mutex); - ib_umem_odp_unmap_dma_pages(umem_odp, ib_umem_start(umem_odp), - ib_umem_end(umem_odp)); - mutex_unlock(&umem_odp->umem_mutex); - mmu_interval_notifier_remove(&umem_odp->notifier); - hmm_dma_map_free(dev->dma_device, &umem_odp->map); - } + mutex_lock(&umem_odp->umem_mutex); + ib_umem_odp_unmap_dma_pages(umem_odp, ib_umem_start(umem_odp), + ib_umem_end(umem_odp)); + mutex_unlock(&umem_odp->umem_mutex); + mmu_interval_notifier_remove(&umem_odp->notifier); + hmm_dma_map_free(dev->dma_device, &umem_odp->map); +} + +void ib_umem_odp_release(struct ib_umem_odp *umem_odp) +{ + if (!umem_odp->is_implicit_odp) + ib_umem_odp_free(umem_odp); + put_pid(umem_odp->tgid); kfree(umem_odp); } --=20 2.49.0 From nobody Tue Dec 16 14:20:07 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2788A262FC5; Mon, 28 Apr 2025 09:23:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745832233; cv=none; b=RtxMeBJrYqi4zYpfMUFsWpiC/mMN09wwvU4PFu5u/p4OyasBbZdwmmppCL6G0OgQjG/K9QVZIR1VQ3oV0wWoxZq5yvBhQUZC5O6/GJINsqpCLBimHtPmCRz40VYPCRet+jCJf39p/aq/w11WwgpARrydXb9QYVpT1iHQtFnZCwE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745832233; c=relaxed/simple; bh=FRtP0nedIpGKT1lyR+Jn+bulCLXW1n6f8oUL9wMCWYc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=kbrRe3gSmCVsbEpu6H07JS1ivAp5ND8obimYIuA5nTPAC5Jdb2BSxz0dNY2gy2NbkpWy9kOw5pv+zYVrP1kQtnZd6O/tTPhzjRkAmuOT6rGj5clpu7ObYhcpBOihgqS2OEgnDQmtnm5zh/dr6MfzccHvzDUndJB6ksiFk07wiX4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=kM/hpY+C; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="kM/hpY+C" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 4C855C4CEE9; Mon, 28 Apr 2025 09:23:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1745832233; bh=FRtP0nedIpGKT1lyR+Jn+bulCLXW1n6f8oUL9wMCWYc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=kM/hpY+CoyvnZa9O+w2eyvo43Nz9b3lX2VmJy82Pjg21G0aqvqPqV3j/tQELLRfTL LlSjECFZb6nuAnoywpRSplGeXvLysRuNZAQLGF3W2TNIXQzz3E09M9kdcm4pYJnomi bwcKYj5Kd97c69xHNL+ynMc08dQbq7SUppBWOgE+/59P6LBwJOf9900G4LXZVoCBY1 4NpIGX9TmqsnTjQNGAT6rtZOejCaAo40OyuvO4mb1SjiFMUBwjVCP98SS3sQ/BQKzz BNMh+K+iSh94BOvVLlilweUTqvE45tmlUwCXGI/RwIVGX4aQ3Dy2OkfMiSqQ/NdXot DIfxzCzSBCv+w== From: Leon Romanovsky To: Marek Szyprowski , Jens Axboe , Christoph Hellwig , Keith Busch Cc: Leon Romanovsky , Jake Edge , Jonathan Corbet , Jason Gunthorpe , Zhu Yanjun , Robin Murphy , Joerg Roedel , Will Deacon , Sagi Grimberg , Bjorn Helgaas , Logan Gunthorpe , Yishai Hadas , Shameer Kolothum , Kevin Tian , Alex Williamson , =?UTF-8?q?J=C3=A9r=C3=B4me=20Glisse?= , Andrew Morton , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-rdma@vger.kernel.org, iommu@lists.linux.dev, linux-nvme@lists.infradead.org, linux-pci@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org, Niklas Schnelle , Chuck Lever , Luis Chamberlain , Matthew Wilcox , Dan Williams , Kanchan Joshi , Chaitanya Kulkarni , Jason Gunthorpe Subject: [PATCH v10 15/24] vfio/mlx5: Explicitly use number of pages instead of allocated length Date: Mon, 28 Apr 2025 12:22:21 +0300 Message-ID: <4e974a11b74489bfd86822b62231b7f9dc8c1123.1745831017.git.leon@kernel.org> X-Mailer: git-send-email 2.49.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Leon Romanovsky allocated_length is a multiple of page size and number of pages, so let's change the functions to accept number of pages. It opens us a venue to combine receive and send paths together with code readability improvement. Tested-by: Jens Axboe Reviewed-by: Jason Gunthorpe Signed-off-by: Leon Romanovsky --- drivers/vfio/pci/mlx5/cmd.c | 32 ++++++++++----------- drivers/vfio/pci/mlx5/cmd.h | 10 +++---- drivers/vfio/pci/mlx5/main.c | 56 +++++++++++++++++++++++------------- 3 files changed, 57 insertions(+), 41 deletions(-) diff --git a/drivers/vfio/pci/mlx5/cmd.c b/drivers/vfio/pci/mlx5/cmd.c index 11eda6b207f1..377dee7765fb 100644 --- a/drivers/vfio/pci/mlx5/cmd.c +++ b/drivers/vfio/pci/mlx5/cmd.c @@ -318,8 +318,7 @@ static int _create_mkey(struct mlx5_core_dev *mdev, u32= pdn, struct mlx5_vhca_recv_buf *recv_buf, u32 *mkey) { - size_t npages =3D buf ? DIV_ROUND_UP(buf->allocated_length, PAGE_SIZE) : - recv_buf->npages; + size_t npages =3D buf ? buf->npages : recv_buf->npages; int err =3D 0, inlen; __be64 *mtt; void *mkc; @@ -375,7 +374,7 @@ static int mlx5vf_dma_data_buffer(struct mlx5_vhca_data= _buffer *buf) if (mvdev->mdev_detach) return -ENOTCONN; =20 - if (buf->dmaed || !buf->allocated_length) + if (buf->dmaed || !buf->npages) return -EINVAL; =20 ret =3D dma_map_sgtable(mdev->device, &buf->table.sgt, buf->dma_dir, 0); @@ -445,7 +444,7 @@ static int mlx5vf_add_migration_pages(struct mlx5_vhca_= data_buffer *buf, =20 if (ret) goto err_append; - buf->allocated_length +=3D filled * PAGE_SIZE; + buf->npages +=3D filled; /* clean input for another bulk allocation */ memset(page_list, 0, filled * sizeof(*page_list)); to_fill =3D min_t(unsigned int, to_alloc, @@ -464,8 +463,7 @@ static int mlx5vf_add_migration_pages(struct mlx5_vhca_= data_buffer *buf, } =20 struct mlx5_vhca_data_buffer * -mlx5vf_alloc_data_buffer(struct mlx5_vf_migration_file *migf, - size_t length, +mlx5vf_alloc_data_buffer(struct mlx5_vf_migration_file *migf, u32 npages, enum dma_data_direction dma_dir) { struct mlx5_vhca_data_buffer *buf; @@ -477,9 +475,8 @@ mlx5vf_alloc_data_buffer(struct mlx5_vf_migration_file = *migf, =20 buf->dma_dir =3D dma_dir; buf->migf =3D migf; - if (length) { - ret =3D mlx5vf_add_migration_pages(buf, - DIV_ROUND_UP_ULL(length, PAGE_SIZE)); + if (npages) { + ret =3D mlx5vf_add_migration_pages(buf, npages); if (ret) goto end; =20 @@ -505,8 +502,8 @@ void mlx5vf_put_data_buffer(struct mlx5_vhca_data_buffe= r *buf) } =20 struct mlx5_vhca_data_buffer * -mlx5vf_get_data_buffer(struct mlx5_vf_migration_file *migf, - size_t length, enum dma_data_direction dma_dir) +mlx5vf_get_data_buffer(struct mlx5_vf_migration_file *migf, u32 npages, + enum dma_data_direction dma_dir) { struct mlx5_vhca_data_buffer *buf, *temp_buf; struct list_head free_list; @@ -521,7 +518,7 @@ mlx5vf_get_data_buffer(struct mlx5_vf_migration_file *m= igf, list_for_each_entry_safe(buf, temp_buf, &migf->avail_list, buf_elm) { if (buf->dma_dir =3D=3D dma_dir) { list_del_init(&buf->buf_elm); - if (buf->allocated_length >=3D length) { + if (buf->npages >=3D npages) { spin_unlock_irq(&migf->list_lock); goto found; } @@ -535,7 +532,7 @@ mlx5vf_get_data_buffer(struct mlx5_vf_migration_file *m= igf, } } spin_unlock_irq(&migf->list_lock); - buf =3D mlx5vf_alloc_data_buffer(migf, length, dma_dir); + buf =3D mlx5vf_alloc_data_buffer(migf, npages, dma_dir); =20 found: while ((temp_buf =3D list_first_entry_or_null(&free_list, @@ -716,7 +713,7 @@ int mlx5vf_cmd_save_vhca_state(struct mlx5vf_pci_core_d= evice *mvdev, MLX5_SET(save_vhca_state_in, in, op_mod, 0); MLX5_SET(save_vhca_state_in, in, vhca_id, mvdev->vhca_id); MLX5_SET(save_vhca_state_in, in, mkey, buf->mkey); - MLX5_SET(save_vhca_state_in, in, size, buf->allocated_length); + MLX5_SET(save_vhca_state_in, in, size, buf->npages * PAGE_SIZE); MLX5_SET(save_vhca_state_in, in, incremental, inc); MLX5_SET(save_vhca_state_in, in, set_track, track); =20 @@ -738,8 +735,11 @@ int mlx5vf_cmd_save_vhca_state(struct mlx5vf_pci_core_= device *mvdev, } =20 if (!header_buf) { - header_buf =3D mlx5vf_get_data_buffer(migf, - sizeof(struct mlx5_vf_migration_header), DMA_NONE); + header_buf =3D mlx5vf_get_data_buffer( + migf, + DIV_ROUND_UP(sizeof(struct mlx5_vf_migration_header), + PAGE_SIZE), + DMA_NONE); if (IS_ERR(header_buf)) { err =3D PTR_ERR(header_buf); goto err_free; diff --git a/drivers/vfio/pci/mlx5/cmd.h b/drivers/vfio/pci/mlx5/cmd.h index df421dc6de04..7d4a833b6900 100644 --- a/drivers/vfio/pci/mlx5/cmd.h +++ b/drivers/vfio/pci/mlx5/cmd.h @@ -56,7 +56,7 @@ struct mlx5_vhca_data_buffer { struct sg_append_table table; loff_t start_pos; u64 length; - u64 allocated_length; + u32 npages; u32 mkey; enum dma_data_direction dma_dir; u8 dmaed:1; @@ -217,12 +217,12 @@ int mlx5vf_cmd_alloc_pd(struct mlx5_vf_migration_file= *migf); void mlx5vf_cmd_dealloc_pd(struct mlx5_vf_migration_file *migf); void mlx5fv_cmd_clean_migf_resources(struct mlx5_vf_migration_file *migf); struct mlx5_vhca_data_buffer * -mlx5vf_alloc_data_buffer(struct mlx5_vf_migration_file *migf, - size_t length, enum dma_data_direction dma_dir); +mlx5vf_alloc_data_buffer(struct mlx5_vf_migration_file *migf, u32 npages, + enum dma_data_direction dma_dir); void mlx5vf_free_data_buffer(struct mlx5_vhca_data_buffer *buf); struct mlx5_vhca_data_buffer * -mlx5vf_get_data_buffer(struct mlx5_vf_migration_file *migf, - size_t length, enum dma_data_direction dma_dir); +mlx5vf_get_data_buffer(struct mlx5_vf_migration_file *migf, u32 npages, + enum dma_data_direction dma_dir); void mlx5vf_put_data_buffer(struct mlx5_vhca_data_buffer *buf); struct page *mlx5vf_get_migration_page(struct mlx5_vhca_data_buffer *buf, unsigned long offset); diff --git a/drivers/vfio/pci/mlx5/main.c b/drivers/vfio/pci/mlx5/main.c index 709543e7eb04..bc0f468f741b 100644 --- a/drivers/vfio/pci/mlx5/main.c +++ b/drivers/vfio/pci/mlx5/main.c @@ -308,6 +308,7 @@ static struct mlx5_vhca_data_buffer * mlx5vf_mig_file_get_stop_copy_buf(struct mlx5_vf_migration_file *migf, u8 index, size_t required_length) { + u32 npages =3D DIV_ROUND_UP(required_length, PAGE_SIZE); struct mlx5_vhca_data_buffer *buf =3D migf->buf[index]; u8 chunk_num; =20 @@ -315,12 +316,11 @@ mlx5vf_mig_file_get_stop_copy_buf(struct mlx5_vf_migr= ation_file *migf, chunk_num =3D buf->stop_copy_chunk_num; buf->migf->buf[index] =3D NULL; /* Checking whether the pre-allocated buffer can fit */ - if (buf->allocated_length >=3D required_length) + if (buf->npages >=3D npages) return buf; =20 mlx5vf_put_data_buffer(buf); - buf =3D mlx5vf_get_data_buffer(buf->migf, required_length, - DMA_FROM_DEVICE); + buf =3D mlx5vf_get_data_buffer(buf->migf, npages, DMA_FROM_DEVICE); if (IS_ERR(buf)) return buf; =20 @@ -373,7 +373,8 @@ static int mlx5vf_add_stop_copy_header(struct mlx5_vf_m= igration_file *migf, u8 *to_buff; int ret; =20 - header_buf =3D mlx5vf_get_data_buffer(migf, size, DMA_NONE); + header_buf =3D mlx5vf_get_data_buffer(migf, DIV_ROUND_UP(size, PAGE_SIZE), + DMA_NONE); if (IS_ERR(header_buf)) return PTR_ERR(header_buf); =20 @@ -388,7 +389,7 @@ static int mlx5vf_add_stop_copy_header(struct mlx5_vf_m= igration_file *migf, to_buff =3D kmap_local_page(page); memcpy(to_buff, &header, sizeof(header)); header_buf->length =3D sizeof(header); - data.stop_copy_size =3D cpu_to_le64(migf->buf[0]->allocated_length); + data.stop_copy_size =3D cpu_to_le64(migf->buf[0]->npages * PAGE_SIZE); memcpy(to_buff + sizeof(header), &data, sizeof(data)); header_buf->length +=3D sizeof(data); kunmap_local(to_buff); @@ -437,15 +438,20 @@ static int mlx5vf_prep_stop_copy(struct mlx5vf_pci_co= re_device *mvdev, =20 num_chunks =3D mvdev->chunk_mode ? MAX_NUM_CHUNKS : 1; for (i =3D 0; i < num_chunks; i++) { - buf =3D mlx5vf_get_data_buffer(migf, inc_state_size, DMA_FROM_DEVICE); + buf =3D mlx5vf_get_data_buffer( + migf, DIV_ROUND_UP(inc_state_size, PAGE_SIZE), + DMA_FROM_DEVICE); if (IS_ERR(buf)) { ret =3D PTR_ERR(buf); goto err; } =20 migf->buf[i] =3D buf; - buf =3D mlx5vf_get_data_buffer(migf, - sizeof(struct mlx5_vf_migration_header), DMA_NONE); + buf =3D mlx5vf_get_data_buffer( + migf, + DIV_ROUND_UP(sizeof(struct mlx5_vf_migration_header), + PAGE_SIZE), + DMA_NONE); if (IS_ERR(buf)) { ret =3D PTR_ERR(buf); goto err; @@ -553,7 +559,8 @@ static long mlx5vf_precopy_ioctl(struct file *filp, uns= igned int cmd, * We finished transferring the current state and the device has a * dirty state, save a new state to be ready for. */ - buf =3D mlx5vf_get_data_buffer(migf, inc_length, DMA_FROM_DEVICE); + buf =3D mlx5vf_get_data_buffer(migf, DIV_ROUND_UP(inc_length, PAGE_SIZE), + DMA_FROM_DEVICE); if (IS_ERR(buf)) { ret =3D PTR_ERR(buf); mlx5vf_mark_err(migf); @@ -675,8 +682,8 @@ mlx5vf_pci_save_device_data(struct mlx5vf_pci_core_devi= ce *mvdev, bool track) =20 if (track) { /* leave the allocated buffer ready for the stop-copy phase */ - buf =3D mlx5vf_alloc_data_buffer(migf, - migf->buf[0]->allocated_length, DMA_FROM_DEVICE); + buf =3D mlx5vf_alloc_data_buffer(migf, migf->buf[0]->npages, + DMA_FROM_DEVICE); if (IS_ERR(buf)) { ret =3D PTR_ERR(buf); goto out_pd; @@ -917,11 +924,14 @@ static ssize_t mlx5vf_resume_write(struct file *filp,= const char __user *buf, goto out_unlock; break; case MLX5_VF_LOAD_STATE_PREP_HEADER_DATA: - if (vhca_buf_header->allocated_length < migf->record_size) { + { + u32 npages =3D DIV_ROUND_UP(migf->record_size, PAGE_SIZE); + + if (vhca_buf_header->npages < npages) { mlx5vf_free_data_buffer(vhca_buf_header); =20 - migf->buf_header[0] =3D mlx5vf_alloc_data_buffer(migf, - migf->record_size, DMA_NONE); + migf->buf_header[0] =3D mlx5vf_alloc_data_buffer( + migf, npages, DMA_NONE); if (IS_ERR(migf->buf_header[0])) { ret =3D PTR_ERR(migf->buf_header[0]); migf->buf_header[0] =3D NULL; @@ -934,6 +944,7 @@ static ssize_t mlx5vf_resume_write(struct file *filp, c= onst char __user *buf, vhca_buf_header->start_pos =3D migf->max_pos; migf->load_state =3D MLX5_VF_LOAD_STATE_READ_HEADER_DATA; break; + } case MLX5_VF_LOAD_STATE_READ_HEADER_DATA: ret =3D mlx5vf_resume_read_header_data(migf, vhca_buf_header, &buf, &len, pos, &done); @@ -944,12 +955,13 @@ static ssize_t mlx5vf_resume_write(struct file *filp,= const char __user *buf, { u64 size =3D max(migf->record_size, migf->stop_copy_prep_size); + u32 npages =3D DIV_ROUND_UP(size, PAGE_SIZE); =20 - if (vhca_buf->allocated_length < size) { + if (vhca_buf->npages < npages) { mlx5vf_free_data_buffer(vhca_buf); =20 - migf->buf[0] =3D mlx5vf_alloc_data_buffer(migf, - size, DMA_TO_DEVICE); + migf->buf[0] =3D mlx5vf_alloc_data_buffer( + migf, npages, DMA_TO_DEVICE); if (IS_ERR(migf->buf[0])) { ret =3D PTR_ERR(migf->buf[0]); migf->buf[0] =3D NULL; @@ -1037,8 +1049,11 @@ mlx5vf_pci_resume_device_data(struct mlx5vf_pci_core= _device *mvdev) } =20 migf->buf[0] =3D buf; - buf =3D mlx5vf_alloc_data_buffer(migf, - sizeof(struct mlx5_vf_migration_header), DMA_NONE); + buf =3D mlx5vf_alloc_data_buffer( + migf, + DIV_ROUND_UP(sizeof(struct mlx5_vf_migration_header), + PAGE_SIZE), + DMA_NONE); if (IS_ERR(buf)) { ret =3D PTR_ERR(buf); goto out_buf; @@ -1148,7 +1163,8 @@ mlx5vf_pci_step_device_state_locked(struct mlx5vf_pci= _core_device *mvdev, MLX5VF_QUERY_INC | MLX5VF_QUERY_CLEANUP); if (ret) return ERR_PTR(ret); - buf =3D mlx5vf_get_data_buffer(migf, size, DMA_FROM_DEVICE); + buf =3D mlx5vf_get_data_buffer(migf, + DIV_ROUND_UP(size, PAGE_SIZE), DMA_FROM_DEVICE); if (IS_ERR(buf)) return ERR_CAST(buf); /* pre_copy cleanup */ --=20 2.49.0 From nobody Tue Dec 16 14:20:07 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9017826280C; Mon, 28 Apr 2025 09:23:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745832229; cv=none; b=TP+2TVnKLTVw6ydh8SU9nCRNq7TnRBCFfl9d9HF43f+cnK6tXouWfZ/eh79cLRgBPm7Vk9JanT92Sof561d73t89+tHDm2prj2cxMWqGio2u+R09o4JiLVtv2+cym02M6fKxo+E078AyNmhDWzr9EBMzr+LDasbIwO6eWMLrv84= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745832229; c=relaxed/simple; bh=7kmV6sSHRcb488fonoHtZwzAJPM9xkMk8Md8QRxDgl0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=hV9enNx5SpQ7j4WIObzm+BgN1gPsvuo5+cBh0U3bXbX/+j4XgRt1W7u5HYsuzD1c4sEZOj1hQZqlcdQQnq7uqx2kzhftba4ZHN0pLmtUkp6eZ0kCfkS1IEs73W4twhZ0f08RMdTjIEChS6mbs+vtROCLNxF/E5hqi2Py10prf7Q= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=EZ93BsSg; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="EZ93BsSg" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 049BAC4CEEE; Mon, 28 Apr 2025 09:23:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1745832229; bh=7kmV6sSHRcb488fonoHtZwzAJPM9xkMk8Md8QRxDgl0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=EZ93BsSgNAdQJV/FKkAuK1BIMCT6ekG9WjLoM6/NYpTa2Kj0l4O1wMuAKV/wza6Oi XX04fVC21c1JmehRiZQ6bw4QPzyYxXKMutMarjcU4SG/gWZoCjtK/zAnke3zPhkhFm pgXE+nzbYbb5uOHoyVnJRFNsW8VNtcdCZZ6p+gern93fOFW2XcQVOFfF47aLM2pH8U rronrPigo5G9hj7PC5lXCPbDm9zeJtPITEN8QFEDcWU0WwTttSpjDIi7uhRFk/Xzsn rqXGIhJtrdj2SbMjOQGc0E/3/Lsc3JLLv6SxoL5N76exqg/GqOMzyARXDeB+Fn1bTW bUcS5ELKFIlkg== From: Leon Romanovsky To: Marek Szyprowski , Jens Axboe , Christoph Hellwig , Keith Busch Cc: Leon Romanovsky , Jake Edge , Jonathan Corbet , Jason Gunthorpe , Zhu Yanjun , Robin Murphy , Joerg Roedel , Will Deacon , Sagi Grimberg , Bjorn Helgaas , Logan Gunthorpe , Yishai Hadas , Shameer Kolothum , Kevin Tian , Alex Williamson , =?UTF-8?q?J=C3=A9r=C3=B4me=20Glisse?= , Andrew Morton , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-rdma@vger.kernel.org, iommu@lists.linux.dev, linux-nvme@lists.infradead.org, linux-pci@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org, Niklas Schnelle , Chuck Lever , Luis Chamberlain , Matthew Wilcox , Dan Williams , Kanchan Joshi , Chaitanya Kulkarni , Jason Gunthorpe Subject: [PATCH v10 16/24] vfio/mlx5: Rewrite create mkey flow to allow better code reuse Date: Mon, 28 Apr 2025 12:22:22 +0300 Message-ID: X-Mailer: git-send-email 2.49.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Leon Romanovsky Change the creation of mkey to be performed in multiple steps: data allocation, DMA setup and actual call to HW to create that mkey. In this new flow, the whole input to MKEY command is saved to eliminate the need to keep array of pointers for DMA addresses for receive list and in the future patches for send list too. In addition to memory size reduce and elimination of unnecessary data movements to set MKEY input, the code is prepared for future reuse. Tested-by: Jens Axboe Reviewed-by: Jason Gunthorpe Signed-off-by: Leon Romanovsky --- drivers/vfio/pci/mlx5/cmd.c | 157 ++++++++++++++++++++---------------- drivers/vfio/pci/mlx5/cmd.h | 4 +- 2 files changed, 91 insertions(+), 70 deletions(-) diff --git a/drivers/vfio/pci/mlx5/cmd.c b/drivers/vfio/pci/mlx5/cmd.c index 377dee7765fb..84dc3bc128c6 100644 --- a/drivers/vfio/pci/mlx5/cmd.c +++ b/drivers/vfio/pci/mlx5/cmd.c @@ -313,39 +313,21 @@ static int mlx5vf_cmd_get_vhca_id(struct mlx5_core_de= v *mdev, u16 function_id, return ret; } =20 -static int _create_mkey(struct mlx5_core_dev *mdev, u32 pdn, - struct mlx5_vhca_data_buffer *buf, - struct mlx5_vhca_recv_buf *recv_buf, - u32 *mkey) +static u32 *alloc_mkey_in(u32 npages, u32 pdn) { - size_t npages =3D buf ? buf->npages : recv_buf->npages; - int err =3D 0, inlen; - __be64 *mtt; + int inlen; void *mkc; u32 *in; =20 inlen =3D MLX5_ST_SZ_BYTES(create_mkey_in) + - sizeof(*mtt) * round_up(npages, 2); + sizeof(__be64) * round_up(npages, 2); =20 - in =3D kvzalloc(inlen, GFP_KERNEL); + in =3D kvzalloc(inlen, GFP_KERNEL_ACCOUNT); if (!in) - return -ENOMEM; + return NULL; =20 MLX5_SET(create_mkey_in, in, translations_octword_actual_size, DIV_ROUND_UP(npages, 2)); - mtt =3D (__be64 *)MLX5_ADDR_OF(create_mkey_in, in, klm_pas_mtt); - - if (buf) { - struct sg_dma_page_iter dma_iter; - - for_each_sgtable_dma_page(&buf->table.sgt, &dma_iter, 0) - *mtt++ =3D cpu_to_be64(sg_page_iter_dma_address(&dma_iter)); - } else { - int i; - - for (i =3D 0; i < npages; i++) - *mtt++ =3D cpu_to_be64(recv_buf->dma_addrs[i]); - } =20 mkc =3D MLX5_ADDR_OF(create_mkey_in, in, memory_key_mkey_entry); MLX5_SET(mkc, mkc, access_mode_1_0, MLX5_MKC_ACCESS_MODE_MTT); @@ -359,9 +341,30 @@ static int _create_mkey(struct mlx5_core_dev *mdev, u3= 2 pdn, MLX5_SET(mkc, mkc, log_page_size, PAGE_SHIFT); MLX5_SET(mkc, mkc, translations_octword_size, DIV_ROUND_UP(npages, 2)); MLX5_SET64(mkc, mkc, len, npages * PAGE_SIZE); - err =3D mlx5_core_create_mkey(mdev, mkey, in, inlen); - kvfree(in); - return err; + + return in; +} + +static int create_mkey(struct mlx5_core_dev *mdev, u32 npages, + struct mlx5_vhca_data_buffer *buf, u32 *mkey_in, + u32 *mkey) +{ + __be64 *mtt; + int inlen; + + mtt =3D (__be64 *)MLX5_ADDR_OF(create_mkey_in, mkey_in, klm_pas_mtt); + if (buf) { + struct sg_dma_page_iter dma_iter; + + for_each_sgtable_dma_page(&buf->table.sgt, &dma_iter, 0) + *mtt++ =3D cpu_to_be64( + sg_page_iter_dma_address(&dma_iter)); + } + + inlen =3D MLX5_ST_SZ_BYTES(create_mkey_in) + + sizeof(__be64) * round_up(npages, 2); + + return mlx5_core_create_mkey(mdev, mkey, mkey_in, inlen); } =20 static int mlx5vf_dma_data_buffer(struct mlx5_vhca_data_buffer *buf) @@ -374,20 +377,28 @@ static int mlx5vf_dma_data_buffer(struct mlx5_vhca_da= ta_buffer *buf) if (mvdev->mdev_detach) return -ENOTCONN; =20 - if (buf->dmaed || !buf->npages) + if (buf->mkey_in || !buf->npages) return -EINVAL; =20 ret =3D dma_map_sgtable(mdev->device, &buf->table.sgt, buf->dma_dir, 0); if (ret) return ret; =20 - ret =3D _create_mkey(mdev, buf->migf->pdn, buf, NULL, &buf->mkey); - if (ret) + buf->mkey_in =3D alloc_mkey_in(buf->npages, buf->migf->pdn); + if (!buf->mkey_in) { + ret =3D -ENOMEM; goto err; + } =20 - buf->dmaed =3D true; + ret =3D create_mkey(mdev, buf->npages, buf, buf->mkey_in, &buf->mkey); + if (ret) + goto err_create_mkey; =20 return 0; + +err_create_mkey: + kvfree(buf->mkey_in); + buf->mkey_in =3D NULL; err: dma_unmap_sgtable(mdev->device, &buf->table.sgt, buf->dma_dir, 0); return ret; @@ -401,8 +412,9 @@ void mlx5vf_free_data_buffer(struct mlx5_vhca_data_buff= er *buf) lockdep_assert_held(&migf->mvdev->state_mutex); WARN_ON(migf->mvdev->mdev_detach); =20 - if (buf->dmaed) { + if (buf->mkey_in) { mlx5_core_destroy_mkey(migf->mvdev->mdev, buf->mkey); + kvfree(buf->mkey_in); dma_unmap_sgtable(migf->mvdev->mdev->device, &buf->table.sgt, buf->dma_dir, 0); } @@ -783,7 +795,7 @@ int mlx5vf_cmd_load_vhca_state(struct mlx5vf_pci_core_d= evice *mvdev, if (mvdev->mdev_detach) return -ENOTCONN; =20 - if (!buf->dmaed) { + if (!buf->mkey_in) { err =3D mlx5vf_dma_data_buffer(buf); if (err) return err; @@ -1384,56 +1396,54 @@ static int alloc_recv_pages(struct mlx5_vhca_recv_b= uf *recv_buf, kvfree(recv_buf->page_list); return -ENOMEM; } +static void unregister_dma_pages(struct mlx5_core_dev *mdev, u32 npages, + u32 *mkey_in) +{ + dma_addr_t addr; + __be64 *mtt; + int i; + + mtt =3D (__be64 *)MLX5_ADDR_OF(create_mkey_in, mkey_in, klm_pas_mtt); + for (i =3D npages - 1; i >=3D 0; i--) { + addr =3D be64_to_cpu(mtt[i]); + dma_unmap_single(mdev->device, addr, PAGE_SIZE, + DMA_FROM_DEVICE); + } +} =20 -static int register_dma_recv_pages(struct mlx5_core_dev *mdev, - struct mlx5_vhca_recv_buf *recv_buf) +static int register_dma_pages(struct mlx5_core_dev *mdev, u32 npages, + struct page **page_list, u32 *mkey_in) { - int i, j; + dma_addr_t addr; + __be64 *mtt; + int i; =20 - recv_buf->dma_addrs =3D kvcalloc(recv_buf->npages, - sizeof(*recv_buf->dma_addrs), - GFP_KERNEL_ACCOUNT); - if (!recv_buf->dma_addrs) - return -ENOMEM; + mtt =3D (__be64 *)MLX5_ADDR_OF(create_mkey_in, mkey_in, klm_pas_mtt); =20 - for (i =3D 0; i < recv_buf->npages; i++) { - recv_buf->dma_addrs[i] =3D dma_map_page(mdev->device, - recv_buf->page_list[i], - 0, PAGE_SIZE, - DMA_FROM_DEVICE); - if (dma_mapping_error(mdev->device, recv_buf->dma_addrs[i])) + for (i =3D 0; i < npages; i++) { + addr =3D dma_map_page(mdev->device, page_list[i], 0, PAGE_SIZE, + DMA_FROM_DEVICE); + if (dma_mapping_error(mdev->device, addr)) goto error; + + *mtt++ =3D cpu_to_be64(addr); } + return 0; =20 error: - for (j =3D 0; j < i; j++) - dma_unmap_single(mdev->device, recv_buf->dma_addrs[j], - PAGE_SIZE, DMA_FROM_DEVICE); - - kvfree(recv_buf->dma_addrs); + unregister_dma_pages(mdev, i, mkey_in); return -ENOMEM; } =20 -static void unregister_dma_recv_pages(struct mlx5_core_dev *mdev, - struct mlx5_vhca_recv_buf *recv_buf) -{ - int i; - - for (i =3D 0; i < recv_buf->npages; i++) - dma_unmap_single(mdev->device, recv_buf->dma_addrs[i], - PAGE_SIZE, DMA_FROM_DEVICE); - - kvfree(recv_buf->dma_addrs); -} - static void mlx5vf_free_qp_recv_resources(struct mlx5_core_dev *mdev, struct mlx5_vhca_qp *qp) { struct mlx5_vhca_recv_buf *recv_buf =3D &qp->recv_buf; =20 mlx5_core_destroy_mkey(mdev, recv_buf->mkey); - unregister_dma_recv_pages(mdev, recv_buf); + unregister_dma_pages(mdev, recv_buf->npages, recv_buf->mkey_in); + kvfree(recv_buf->mkey_in); free_recv_pages(&qp->recv_buf); } =20 @@ -1449,18 +1459,29 @@ static int mlx5vf_alloc_qp_recv_resources(struct ml= x5_core_dev *mdev, if (err < 0) return err; =20 - err =3D register_dma_recv_pages(mdev, recv_buf); - if (err) + recv_buf->mkey_in =3D alloc_mkey_in(npages, pdn); + if (!recv_buf->mkey_in) { + err =3D -ENOMEM; goto end; + } + + err =3D register_dma_pages(mdev, npages, recv_buf->page_list, + recv_buf->mkey_in); + if (err) + goto err_register_dma; =20 - err =3D _create_mkey(mdev, pdn, NULL, recv_buf, &recv_buf->mkey); + err =3D create_mkey(mdev, npages, NULL, recv_buf->mkey_in, + &recv_buf->mkey); if (err) goto err_create_mkey; =20 return 0; =20 err_create_mkey: - unregister_dma_recv_pages(mdev, recv_buf); + unregister_dma_pages(mdev, npages, recv_buf->mkey_in); +err_register_dma: + kvfree(recv_buf->mkey_in); + recv_buf->mkey_in =3D NULL; end: free_recv_pages(recv_buf); return err; diff --git a/drivers/vfio/pci/mlx5/cmd.h b/drivers/vfio/pci/mlx5/cmd.h index 7d4a833b6900..25dd6ff54591 100644 --- a/drivers/vfio/pci/mlx5/cmd.h +++ b/drivers/vfio/pci/mlx5/cmd.h @@ -58,8 +58,8 @@ struct mlx5_vhca_data_buffer { u64 length; u32 npages; u32 mkey; + u32 *mkey_in; enum dma_data_direction dma_dir; - u8 dmaed:1; u8 stop_copy_chunk_num; struct list_head buf_elm; struct mlx5_vf_migration_file *migf; @@ -133,8 +133,8 @@ struct mlx5_vhca_cq { struct mlx5_vhca_recv_buf { u32 npages; struct page **page_list; - dma_addr_t *dma_addrs; u32 next_rq_offset; + u32 *mkey_in; u32 mkey; }; =20 --=20 2.49.0 From nobody Tue Dec 16 14:20:07 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7835126FDAB; Mon, 28 Apr 2025 09:23:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745832237; cv=none; b=FSM1yM7oNhkzbm+bpnO0NMuuAWDTOD/U2NarQFBfkLLoUpJxw06hhPtplnSq55odnOWOhSLlBXPac4jYD4g3fDFBcSCKpTG00T6qqw4c/piA1p6FJe0rlXJChijheLJt5SSt92S4CqVsfB9iQRBO2PBnSV673yQ6e7/Fhti1zWA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745832237; c=relaxed/simple; bh=W3qb5+sXO2k2PkrP/Voo3AwjNc2CVoiyLQ4V37nfwWs=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=q3mZMe/euK8mfLdwgjlDvWSqGMRZPiFIZe433IgJNXlEl7EaITfmRGl8t7tf+81xpnO6KoRexcxHz1lG5zv3G5absQHZeHbE2MzVFSURP8RoeE1FUE7kmzk0CEpHhlhtsyyw09+9NvGVepSe4AeCgfGmJ3MD6lgkz0fsN3zSYZc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=ti06wAal; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="ti06wAal" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 28103C4CEE9; Mon, 28 Apr 2025 09:23:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1745832237; bh=W3qb5+sXO2k2PkrP/Voo3AwjNc2CVoiyLQ4V37nfwWs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ti06wAal+8WlKC00cBSRuCtD4c3qS46FXuFBGSgU4CrY94LzTYAEHx9yYvlxrFfMW c9FQVeYz7fVFC/ECvu5EyLH2v80yFJiifmyEFdcsV1THBQjlEZiq6Sz7XBhsbp4X2i HbCYETofM4hbfjeSkaannWOl7mmOTn+KLN7i45xeFonqMhNJPXQcDJsc1GfIqWtWU6 YECDYaMKzL1+pMtk30p0OIW3M9UNdvtxMBQ8IILt/K0KKZ6TvUv8do43aEZlDeY7MG 9v5BvT9p57gEuak79//ivUFUN5h/MEkMimrOiBI/SQCBH0M7X6ONuhFCi7JajWkVpM hRy+Ttnu/rW0w== From: Leon Romanovsky To: Marek Szyprowski , Jens Axboe , Christoph Hellwig , Keith Busch Cc: Leon Romanovsky , Jake Edge , Jonathan Corbet , Jason Gunthorpe , Zhu Yanjun , Robin Murphy , Joerg Roedel , Will Deacon , Sagi Grimberg , Bjorn Helgaas , Logan Gunthorpe , Yishai Hadas , Shameer Kolothum , Kevin Tian , Alex Williamson , =?UTF-8?q?J=C3=A9r=C3=B4me=20Glisse?= , Andrew Morton , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-rdma@vger.kernel.org, iommu@lists.linux.dev, linux-nvme@lists.infradead.org, linux-pci@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org, Niklas Schnelle , Chuck Lever , Luis Chamberlain , Matthew Wilcox , Dan Williams , Kanchan Joshi , Chaitanya Kulkarni , Jason Gunthorpe Subject: [PATCH v10 17/24] vfio/mlx5: Enable the DMA link API Date: Mon, 28 Apr 2025 12:22:23 +0300 Message-ID: <2e232c0574f3f0bfe099d63e2cacb7fd61e18d11.1745831017.git.leon@kernel.org> X-Mailer: git-send-email 2.49.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Leon Romanovsky Remove intermediate scatter-gather table completely and enable new DMA link API. Tested-by: Jens Axboe Reviewed-by: Jason Gunthorpe Signed-off-by: Leon Romanovsky --- drivers/vfio/pci/mlx5/cmd.c | 298 ++++++++++++++++------------------- drivers/vfio/pci/mlx5/cmd.h | 21 ++- drivers/vfio/pci/mlx5/main.c | 31 ---- 3 files changed, 147 insertions(+), 203 deletions(-) diff --git a/drivers/vfio/pci/mlx5/cmd.c b/drivers/vfio/pci/mlx5/cmd.c index 84dc3bc128c6..b162e44112fb 100644 --- a/drivers/vfio/pci/mlx5/cmd.c +++ b/drivers/vfio/pci/mlx5/cmd.c @@ -345,26 +345,82 @@ static u32 *alloc_mkey_in(u32 npages, u32 pdn) return in; } =20 -static int create_mkey(struct mlx5_core_dev *mdev, u32 npages, - struct mlx5_vhca_data_buffer *buf, u32 *mkey_in, +static int create_mkey(struct mlx5_core_dev *mdev, u32 npages, u32 *mkey_i= n, u32 *mkey) { + int inlen =3D MLX5_ST_SZ_BYTES(create_mkey_in) + + sizeof(__be64) * round_up(npages, 2); + + return mlx5_core_create_mkey(mdev, mkey, mkey_in, inlen); +} + +static void unregister_dma_pages(struct mlx5_core_dev *mdev, u32 npages, + u32 *mkey_in, struct dma_iova_state *state, + enum dma_data_direction dir) +{ + dma_addr_t addr; __be64 *mtt; - int inlen; + int i; =20 - mtt =3D (__be64 *)MLX5_ADDR_OF(create_mkey_in, mkey_in, klm_pas_mtt); - if (buf) { - struct sg_dma_page_iter dma_iter; + WARN_ON_ONCE(dir =3D=3D DMA_NONE); =20 - for_each_sgtable_dma_page(&buf->table.sgt, &dma_iter, 0) - *mtt++ =3D cpu_to_be64( - sg_page_iter_dma_address(&dma_iter)); + if (dma_use_iova(state)) { + dma_iova_destroy(mdev->device, state, npages * PAGE_SIZE, dir, + 0); + } else { + mtt =3D (__be64 *)MLX5_ADDR_OF(create_mkey_in, mkey_in, + klm_pas_mtt); + for (i =3D npages - 1; i >=3D 0; i--) { + addr =3D be64_to_cpu(mtt[i]); + dma_unmap_page(mdev->device, addr, PAGE_SIZE, dir); + } } +} =20 - inlen =3D MLX5_ST_SZ_BYTES(create_mkey_in) + - sizeof(__be64) * round_up(npages, 2); +static int register_dma_pages(struct mlx5_core_dev *mdev, u32 npages, + struct page **page_list, u32 *mkey_in, + struct dma_iova_state *state, + enum dma_data_direction dir) +{ + dma_addr_t addr; + size_t mapped =3D 0; + __be64 *mtt; + int i, err; =20 - return mlx5_core_create_mkey(mdev, mkey, mkey_in, inlen); + WARN_ON_ONCE(dir =3D=3D DMA_NONE); + + mtt =3D (__be64 *)MLX5_ADDR_OF(create_mkey_in, mkey_in, klm_pas_mtt); + + if (dma_iova_try_alloc(mdev->device, state, 0, npages * PAGE_SIZE)) { + addr =3D state->addr; + for (i =3D 0; i < npages; i++) { + err =3D dma_iova_link(mdev->device, state, + page_to_phys(page_list[i]), mapped, + PAGE_SIZE, dir, 0); + if (err) + goto error; + *mtt++ =3D cpu_to_be64(addr); + addr +=3D PAGE_SIZE; + mapped +=3D PAGE_SIZE; + } + err =3D dma_iova_sync(mdev->device, state, 0, mapped); + if (err) + goto error; + } else { + for (i =3D 0; i < npages; i++) { + addr =3D dma_map_page(mdev->device, page_list[i], 0, + PAGE_SIZE, dir); + err =3D dma_mapping_error(mdev->device, addr); + if (err) + goto error; + *mtt++ =3D cpu_to_be64(addr); + } + } + return 0; + +error: + unregister_dma_pages(mdev, i, mkey_in, state, dir); + return err; } =20 static int mlx5vf_dma_data_buffer(struct mlx5_vhca_data_buffer *buf) @@ -380,98 +436,90 @@ static int mlx5vf_dma_data_buffer(struct mlx5_vhca_da= ta_buffer *buf) if (buf->mkey_in || !buf->npages) return -EINVAL; =20 - ret =3D dma_map_sgtable(mdev->device, &buf->table.sgt, buf->dma_dir, 0); - if (ret) - return ret; - buf->mkey_in =3D alloc_mkey_in(buf->npages, buf->migf->pdn); - if (!buf->mkey_in) { - ret =3D -ENOMEM; - goto err; - } + if (!buf->mkey_in) + return -ENOMEM; =20 - ret =3D create_mkey(mdev, buf->npages, buf, buf->mkey_in, &buf->mkey); + ret =3D register_dma_pages(mdev, buf->npages, buf->page_list, + buf->mkey_in, &buf->state, buf->dma_dir); + if (ret) + goto err_register_dma; + + ret =3D create_mkey(mdev, buf->npages, buf->mkey_in, &buf->mkey); if (ret) goto err_create_mkey; =20 return 0; =20 err_create_mkey: + unregister_dma_pages(mdev, buf->npages, buf->mkey_in, &buf->state, + buf->dma_dir); +err_register_dma: kvfree(buf->mkey_in); buf->mkey_in =3D NULL; -err: - dma_unmap_sgtable(mdev->device, &buf->table.sgt, buf->dma_dir, 0); return ret; } =20 +static void free_page_list(u32 npages, struct page **page_list) +{ + int i; + + /* Undo alloc_pages_bulk() */ + for (i =3D npages - 1; i >=3D 0; i--) + __free_page(page_list[i]); + + kvfree(page_list); +} + void mlx5vf_free_data_buffer(struct mlx5_vhca_data_buffer *buf) { - struct mlx5_vf_migration_file *migf =3D buf->migf; - struct sg_page_iter sg_iter; + struct mlx5vf_pci_core_device *mvdev =3D buf->migf->mvdev; + struct mlx5_core_dev *mdev =3D mvdev->mdev; =20 - lockdep_assert_held(&migf->mvdev->state_mutex); - WARN_ON(migf->mvdev->mdev_detach); + lockdep_assert_held(&mvdev->state_mutex); + WARN_ON(mvdev->mdev_detach); =20 if (buf->mkey_in) { - mlx5_core_destroy_mkey(migf->mvdev->mdev, buf->mkey); + mlx5_core_destroy_mkey(mdev, buf->mkey); + unregister_dma_pages(mdev, buf->npages, buf->mkey_in, + &buf->state, buf->dma_dir); kvfree(buf->mkey_in); - dma_unmap_sgtable(migf->mvdev->mdev->device, &buf->table.sgt, - buf->dma_dir, 0); } =20 - /* Undo alloc_pages_bulk() */ - for_each_sgtable_page(&buf->table.sgt, &sg_iter, 0) - __free_page(sg_page_iter_page(&sg_iter)); - sg_free_append_table(&buf->table); + free_page_list(buf->npages, buf->page_list); kfree(buf); } =20 -static int mlx5vf_add_migration_pages(struct mlx5_vhca_data_buffer *buf, - unsigned int npages) +static int mlx5vf_add_pages(struct page ***page_list, unsigned int npages) { - unsigned int to_alloc =3D npages; - struct page **page_list; - unsigned long filled; - unsigned int to_fill; - int ret; + unsigned int filled, done =3D 0; int i; =20 - to_fill =3D min_t(unsigned int, npages, PAGE_SIZE / sizeof(*page_list)); - page_list =3D kvzalloc(to_fill * sizeof(*page_list), GFP_KERNEL_ACCOUNT); - if (!page_list) + *page_list =3D + kvcalloc(npages, sizeof(struct page *), GFP_KERNEL_ACCOUNT); + if (!*page_list) return -ENOMEM; =20 - do { - filled =3D alloc_pages_bulk(GFP_KERNEL_ACCOUNT, to_fill, - page_list); - if (!filled) { - ret =3D -ENOMEM; + for (;;) { + filled =3D alloc_pages_bulk(GFP_KERNEL_ACCOUNT, npages - done, + *page_list + done); + if (!filled) goto err; - } - to_alloc -=3D filled; - ret =3D sg_alloc_append_table_from_pages( - &buf->table, page_list, filled, 0, - filled << PAGE_SHIFT, UINT_MAX, SG_MAX_SINGLE_ALLOC, - GFP_KERNEL_ACCOUNT); =20 - if (ret) - goto err_append; - buf->npages +=3D filled; - /* clean input for another bulk allocation */ - memset(page_list, 0, filled * sizeof(*page_list)); - to_fill =3D min_t(unsigned int, to_alloc, - PAGE_SIZE / sizeof(*page_list)); - } while (to_alloc > 0); + done +=3D filled; + if (done =3D=3D npages) + break; + } =20 - kvfree(page_list); return 0; =20 -err_append: - for (i =3D filled - 1; i >=3D 0; i--) - __free_page(page_list[i]); err: - kvfree(page_list); - return ret; + for (i =3D 0; i < done; i++) + __free_page(*page_list[i]); + + kvfree(*page_list); + *page_list =3D NULL; + return -ENOMEM; } =20 struct mlx5_vhca_data_buffer * @@ -488,10 +536,12 @@ mlx5vf_alloc_data_buffer(struct mlx5_vf_migration_fil= e *migf, u32 npages, buf->dma_dir =3D dma_dir; buf->migf =3D migf; if (npages) { - ret =3D mlx5vf_add_migration_pages(buf, npages); + ret =3D mlx5vf_add_pages(&buf->page_list, npages); if (ret) goto end; =20 + buf->npages =3D npages; + if (dma_dir !=3D DMA_NONE) { ret =3D mlx5vf_dma_data_buffer(buf); if (ret) @@ -1350,101 +1400,16 @@ static void mlx5vf_destroy_qp(struct mlx5_core_dev= *mdev, kfree(qp); } =20 -static void free_recv_pages(struct mlx5_vhca_recv_buf *recv_buf) -{ - int i; - - /* Undo alloc_pages_bulk() */ - for (i =3D 0; i < recv_buf->npages; i++) - __free_page(recv_buf->page_list[i]); - - kvfree(recv_buf->page_list); -} - -static int alloc_recv_pages(struct mlx5_vhca_recv_buf *recv_buf, - unsigned int npages) -{ - unsigned int filled =3D 0, done =3D 0; - int i; - - recv_buf->page_list =3D kvcalloc(npages, sizeof(*recv_buf->page_list), - GFP_KERNEL_ACCOUNT); - if (!recv_buf->page_list) - return -ENOMEM; - - for (;;) { - filled =3D alloc_pages_bulk(GFP_KERNEL_ACCOUNT, - npages - done, - recv_buf->page_list + done); - if (!filled) - goto err; - - done +=3D filled; - if (done =3D=3D npages) - break; - } - - recv_buf->npages =3D npages; - return 0; - -err: - for (i =3D 0; i < npages; i++) { - if (recv_buf->page_list[i]) - __free_page(recv_buf->page_list[i]); - } - - kvfree(recv_buf->page_list); - return -ENOMEM; -} -static void unregister_dma_pages(struct mlx5_core_dev *mdev, u32 npages, - u32 *mkey_in) -{ - dma_addr_t addr; - __be64 *mtt; - int i; - - mtt =3D (__be64 *)MLX5_ADDR_OF(create_mkey_in, mkey_in, klm_pas_mtt); - for (i =3D npages - 1; i >=3D 0; i--) { - addr =3D be64_to_cpu(mtt[i]); - dma_unmap_single(mdev->device, addr, PAGE_SIZE, - DMA_FROM_DEVICE); - } -} - -static int register_dma_pages(struct mlx5_core_dev *mdev, u32 npages, - struct page **page_list, u32 *mkey_in) -{ - dma_addr_t addr; - __be64 *mtt; - int i; - - mtt =3D (__be64 *)MLX5_ADDR_OF(create_mkey_in, mkey_in, klm_pas_mtt); - - for (i =3D 0; i < npages; i++) { - addr =3D dma_map_page(mdev->device, page_list[i], 0, PAGE_SIZE, - DMA_FROM_DEVICE); - if (dma_mapping_error(mdev->device, addr)) - goto error; - - *mtt++ =3D cpu_to_be64(addr); - } - - return 0; - -error: - unregister_dma_pages(mdev, i, mkey_in); - return -ENOMEM; -} - static void mlx5vf_free_qp_recv_resources(struct mlx5_core_dev *mdev, struct mlx5_vhca_qp *qp) { struct mlx5_vhca_recv_buf *recv_buf =3D &qp->recv_buf; =20 mlx5_core_destroy_mkey(mdev, recv_buf->mkey); - unregister_dma_pages(mdev, recv_buf->npages, recv_buf->mkey_in); + unregister_dma_pages(mdev, recv_buf->npages, recv_buf->mkey_in, + &recv_buf->state, DMA_FROM_DEVICE); kvfree(recv_buf->mkey_in); - free_recv_pages(&qp->recv_buf); + free_page_list(recv_buf->npages, recv_buf->page_list); } =20 static int mlx5vf_alloc_qp_recv_resources(struct mlx5_core_dev *mdev, @@ -1455,10 +1420,12 @@ static int mlx5vf_alloc_qp_recv_resources(struct ml= x5_core_dev *mdev, struct mlx5_vhca_recv_buf *recv_buf =3D &qp->recv_buf; int err; =20 - err =3D alloc_recv_pages(recv_buf, npages); - if (err < 0) + err =3D mlx5vf_add_pages(&recv_buf->page_list, npages); + if (err) return err; =20 + recv_buf->npages =3D npages; + recv_buf->mkey_in =3D alloc_mkey_in(npages, pdn); if (!recv_buf->mkey_in) { err =3D -ENOMEM; @@ -1466,24 +1433,25 @@ static int mlx5vf_alloc_qp_recv_resources(struct ml= x5_core_dev *mdev, } =20 err =3D register_dma_pages(mdev, npages, recv_buf->page_list, - recv_buf->mkey_in); + recv_buf->mkey_in, &recv_buf->state, + DMA_FROM_DEVICE); if (err) goto err_register_dma; =20 - err =3D create_mkey(mdev, npages, NULL, recv_buf->mkey_in, - &recv_buf->mkey); + err =3D create_mkey(mdev, npages, recv_buf->mkey_in, &recv_buf->mkey); if (err) goto err_create_mkey; =20 return 0; =20 err_create_mkey: - unregister_dma_pages(mdev, npages, recv_buf->mkey_in); + unregister_dma_pages(mdev, npages, recv_buf->mkey_in, &recv_buf->state, + DMA_FROM_DEVICE); err_register_dma: kvfree(recv_buf->mkey_in); recv_buf->mkey_in =3D NULL; end: - free_recv_pages(recv_buf); + free_page_list(npages, recv_buf->page_list); return err; } =20 diff --git a/drivers/vfio/pci/mlx5/cmd.h b/drivers/vfio/pci/mlx5/cmd.h index 25dd6ff54591..d7821b5ca772 100644 --- a/drivers/vfio/pci/mlx5/cmd.h +++ b/drivers/vfio/pci/mlx5/cmd.h @@ -53,7 +53,8 @@ struct mlx5_vf_migration_header { }; =20 struct mlx5_vhca_data_buffer { - struct sg_append_table table; + struct page **page_list; + struct dma_iova_state state; loff_t start_pos; u64 length; u32 npages; @@ -63,10 +64,6 @@ struct mlx5_vhca_data_buffer { u8 stop_copy_chunk_num; struct list_head buf_elm; struct mlx5_vf_migration_file *migf; - /* Optimize mlx5vf_get_migration_page() for sequential access */ - struct scatterlist *last_offset_sg; - unsigned int sg_last_entry; - unsigned long last_offset; }; =20 struct mlx5vf_async_data { @@ -133,6 +130,7 @@ struct mlx5_vhca_cq { struct mlx5_vhca_recv_buf { u32 npages; struct page **page_list; + struct dma_iova_state state; u32 next_rq_offset; u32 *mkey_in; u32 mkey; @@ -224,8 +222,17 @@ struct mlx5_vhca_data_buffer * mlx5vf_get_data_buffer(struct mlx5_vf_migration_file *migf, u32 npages, enum dma_data_direction dma_dir); void mlx5vf_put_data_buffer(struct mlx5_vhca_data_buffer *buf); -struct page *mlx5vf_get_migration_page(struct mlx5_vhca_data_buffer *buf, - unsigned long offset); +static inline struct page * +mlx5vf_get_migration_page(struct mlx5_vhca_data_buffer *buf, + unsigned long offset) +{ + int page_entry =3D offset / PAGE_SIZE; + + if (page_entry >=3D buf->npages) + return NULL; + + return buf->page_list[page_entry]; +} void mlx5vf_state_mutex_unlock(struct mlx5vf_pci_core_device *mvdev); void mlx5vf_disable_fds(struct mlx5vf_pci_core_device *mvdev, enum mlx5_vf_migf_state *last_save_state); diff --git a/drivers/vfio/pci/mlx5/main.c b/drivers/vfio/pci/mlx5/main.c index bc0f468f741b..93f894fe60d2 100644 --- a/drivers/vfio/pci/mlx5/main.c +++ b/drivers/vfio/pci/mlx5/main.c @@ -34,37 +34,6 @@ static struct mlx5vf_pci_core_device *mlx5vf_drvdata(str= uct pci_dev *pdev) core_device); } =20 -struct page * -mlx5vf_get_migration_page(struct mlx5_vhca_data_buffer *buf, - unsigned long offset) -{ - unsigned long cur_offset =3D 0; - struct scatterlist *sg; - unsigned int i; - - /* All accesses are sequential */ - if (offset < buf->last_offset || !buf->last_offset_sg) { - buf->last_offset =3D 0; - buf->last_offset_sg =3D buf->table.sgt.sgl; - buf->sg_last_entry =3D 0; - } - - cur_offset =3D buf->last_offset; - - for_each_sg(buf->last_offset_sg, sg, - buf->table.sgt.orig_nents - buf->sg_last_entry, i) { - if (offset < sg->length + cur_offset) { - buf->last_offset_sg =3D sg; - buf->sg_last_entry +=3D i; - buf->last_offset =3D cur_offset; - return nth_page(sg_page(sg), - (offset - cur_offset) / PAGE_SIZE); - } - cur_offset +=3D sg->length; - } - return NULL; -} - static void mlx5vf_disable_fd(struct mlx5_vf_migration_file *migf) { mutex_lock(&migf->lock); --=20 2.49.0 From nobody Tue Dec 16 14:20:07 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E7D0A27A120; Mon, 28 Apr 2025 09:24:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745832257; cv=none; b=KMlV31tGEdvf0ZHQx7Lso7f/f+dnGS4SSdG/lgBmm/fMv9xWcGxVHj7bYzjypZ90kBgAD8pIpXCQc3NsKWoUyHAsSy58QpupO/gBShxDId5JtvO8w3aG9VOxtia7rGkML4POzNfkJ/k+HSSDs1VthScH+RKbOXmGdWkBi9jhUeA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745832257; c=relaxed/simple; bh=4dOjVeYpWJkKZa5Ph2IEMg0CPmd+dfHE5vq3NRZWLwQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Z1Y1bRJ8PryYDNNNDLUAsp7PNxtsIO7R1xacvHBc5s//aqd3y/b43qJ+cUhKYoet97Zz3CWUxkPc8yOrIoTDDjizMZ2EKepwLIjUFrJcRdVGbBWxYOh3rJ90om1CCoxe7XkVN212L/LsQww7tdik6SaQVKdudfBfH4sHFBmnn7s= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=ce3VPJ07; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="ce3VPJ07" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1D8BCC4CEE9; Mon, 28 Apr 2025 09:24:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1745832256; bh=4dOjVeYpWJkKZa5Ph2IEMg0CPmd+dfHE5vq3NRZWLwQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ce3VPJ07DNrV5898GsmpRBIpv34v0Mn1piZte3fq+yUqHqi9q48o9CqXmaMiUVEcR znVULf9fBZG9hXk68O8UxP+wJzq7puvp9eL4agI5Vq7Hv78ajnkd6mpnzVDlI84znw VL1g+tFvIYAR54D2d2n4CL9TATFkhlFsNfVWgPjkNKj3g2yg2TMFMO9c1tTLc+SExB evAOhYpcMSEA8VIAeeAWLqYuRnn2BoO7hk6S+6O3gzb/PPsBE929l1xEtys5zX6LnO UgsP8OUpmVCFAKLp6eVl0pRPDVfXASUaLqX0isM3kIv1f6HDrI1eYBv0zEMsGD5ypt vHx1ExNlBVXSg== From: Leon Romanovsky To: Marek Szyprowski , Jens Axboe , Christoph Hellwig , Keith Busch Cc: Jake Edge , Jonathan Corbet , Jason Gunthorpe , Zhu Yanjun , Robin Murphy , Joerg Roedel , Will Deacon , Sagi Grimberg , Bjorn Helgaas , Logan Gunthorpe , Yishai Hadas , Shameer Kolothum , Kevin Tian , Alex Williamson , =?UTF-8?q?J=C3=A9r=C3=B4me=20Glisse?= , Andrew Morton , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-rdma@vger.kernel.org, iommu@lists.linux.dev, linux-nvme@lists.infradead.org, linux-pci@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org, Niklas Schnelle , Chuck Lever , Luis Chamberlain , Matthew Wilcox , Dan Williams , Kanchan Joshi , Chaitanya Kulkarni , Leon Romanovsky Subject: [PATCH v10 18/24] block: share more code for bio addition helper Date: Mon, 28 Apr 2025 12:22:24 +0300 Message-ID: <0b77dcad1eb6ccb9f5923a2999a89ef779394fb4.1745831017.git.leon@kernel.org> X-Mailer: git-send-email 2.49.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Christoph Hellwig __bio_iov_iter_get_pages currently open codes adding pages to the bio, which duplicates a lot of code from bio_add_page. Add bio_add_page_int helper that pass down the same_page output argument so that __bio_iov_iter_get_pages can reuse the main add bio to page helpers. Note that I'd normally call these helper __bio_add_page, but the former is already taken for an exported API. Signed-off-by: Christoph Hellwig Tested-by: Jens Axboe Signed-off-by: Leon Romanovsky --- block/bio.c | 68 +++++++++++++++++++++++++---------------------------- 1 file changed, 32 insertions(+), 36 deletions(-) diff --git a/block/bio.c b/block/bio.c index 4e6c85a33d74..3047fa3f4b32 100644 --- a/block/bio.c +++ b/block/bio.c @@ -989,20 +989,9 @@ void __bio_add_page(struct bio *bio, struct page *page, } EXPORT_SYMBOL_GPL(__bio_add_page); =20 -/** - * bio_add_page - attempt to add page(s) to bio - * @bio: destination bio - * @page: start page to add - * @len: vec entry length, may cross pages - * @offset: vec entry offset relative to @page, may cross pages - * - * Attempt to add page(s) to the bio_vec maplist. This will only fail - * if either bio->bi_vcnt =3D=3D bio->bi_max_vecs or it's a cloned bio. - */ -int bio_add_page(struct bio *bio, struct page *page, - unsigned int len, unsigned int offset) +static int bio_add_page_int(struct bio *bio, struct page *page, + unsigned int len, unsigned int offset, bool *same_page) { - bool same_page =3D false; =20 if (WARN_ON_ONCE(bio_flagged(bio, BIO_CLONED))) return 0; @@ -1011,7 +1000,7 @@ int bio_add_page(struct bio *bio, struct page *page, =20 if (bio->bi_vcnt > 0 && bvec_try_merge_page(&bio->bi_io_vec[bio->bi_vcnt - 1], - page, len, offset, &same_page)) { + page, len, offset, same_page)) { bio->bi_iter.bi_size +=3D len; return len; } @@ -1021,6 +1010,24 @@ int bio_add_page(struct bio *bio, struct page *page, __bio_add_page(bio, page, len, offset); return len; } + +/** + * bio_add_page - attempt to add page(s) to bio + * @bio: destination bio + * @page: start page to add + * @len: vec entry length, may cross pages + * @offset: vec entry offset relative to @page, may cross pages + * + * Attempt to add page(s) to the bio_vec maplist. Will only fail if the + * bio is full, or it is incorrectly used on a cloned bio. + */ +int bio_add_page(struct bio *bio, struct page *page, + unsigned int len, unsigned int offset) +{ + bool same_page =3D false; + + return bio_add_page_int(bio, page, len, offset, &same_page); +} EXPORT_SYMBOL(bio_add_page); =20 void bio_add_folio_nofail(struct bio *bio, struct folio *folio, size_t len, @@ -1088,27 +1095,6 @@ void bio_iov_bvec_set(struct bio *bio, const struct = iov_iter *iter) bio_set_flag(bio, BIO_CLONED); } =20 -static int bio_iov_add_folio(struct bio *bio, struct folio *folio, size_t = len, - size_t offset) -{ - bool same_page =3D false; - - if (WARN_ON_ONCE(bio->bi_iter.bi_size > UINT_MAX - len)) - return -EIO; - - if (bio->bi_vcnt > 0 && - bvec_try_merge_page(&bio->bi_io_vec[bio->bi_vcnt - 1], - folio_page(folio, 0), len, offset, - &same_page)) { - bio->bi_iter.bi_size +=3D len; - if (same_page && bio_flagged(bio, BIO_PAGE_PINNED)) - unpin_user_folio(folio, 1); - return 0; - } - bio_add_folio_nofail(bio, folio, len, offset); - return 0; -} - static unsigned int get_contig_folio_len(unsigned int *num_pages, struct page **pages, unsigned int i, struct folio *folio, size_t left, @@ -1203,6 +1189,8 @@ static int __bio_iov_iter_get_pages(struct bio *bio, = struct iov_iter *iter) for (left =3D size, i =3D 0; left > 0; left -=3D len, i +=3D num_pages) { struct page *page =3D pages[i]; struct folio *folio =3D page_folio(page); + struct page *first_page =3D folio_page(folio, 0); + bool same_page =3D false; =20 folio_offset =3D ((size_t)folio_page_idx(folio, page) << PAGE_SHIFT) + offset; @@ -1215,7 +1203,15 @@ static int __bio_iov_iter_get_pages(struct bio *bio,= struct iov_iter *iter) len =3D get_contig_folio_len(&num_pages, pages, i, folio, left, offset); =20 - bio_iov_add_folio(bio, folio, len, folio_offset); + if (bio_add_page_int(bio, first_page, len, folio_offset, + &same_page) !=3D len) { + ret =3D -EINVAL; + break; + } + + if (same_page && bio_flagged(bio, BIO_PAGE_PINNED)) + unpin_user_folio(folio, 1); + offset =3D 0; } =20 --=20 2.49.0 From nobody Tue Dec 16 14:20:07 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E24552741D0; Mon, 28 Apr 2025 09:24:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745832241; cv=none; b=m7cVGeETkXV4S36YtBUBU2hSKZQypY0yKzY3wRHI7vXASzC3Qp5g58RBjQA5YK1ooRgPA18NA89ox8FVB1xzcZ4nHRdhV3w1o8dPs4VykNfkGhVC9ce0ay8MTfTRx4VyhZxotCOHc1SWlBF1Ckgg8PjIoSuoZNL6ZWW79oGrkko= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745832241; c=relaxed/simple; bh=cn4ZAo5vXix5AGqKt4IWuh2vUXjY6lms9OP/AMjOBBE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=HJKkBP+BW/paPukFRhLtA5mK0JANQW58xGI3yz3ehJq3wwWd0tQ8XTDSm34DeyDdS1CI6JTXtANYx9Sw3C5hfOY4Hrfaz43VKLuWD5Ergbb5gpi8R4zY2cXYB7KSj5bigc4CtE8L/V1Jo7xmR18FFIn/3XEldR+eYqtjijOgjjg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=oo6f8Lit; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="oo6f8Lit" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 11CAFC4CEEE; Mon, 28 Apr 2025 09:24:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1745832240; bh=cn4ZAo5vXix5AGqKt4IWuh2vUXjY6lms9OP/AMjOBBE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=oo6f8LitBAP+zsGFjpfl+0mKIJC5Ky3ldfas2CEBRMfKaB8MBnnjLgNzm9zM6tZmT gFqE6/2vvpVC3VBAQvhHlTGEYrSHDsE1l5GlwZAkFZ61myGlxGalP4JwuRBNe5Xlms 0LasKT819i+2DFexVc3tcm6U24G3gzB7SPTdpUOmZ0/AxBENdqtZKvrWiW/XO5sPNm kg3YYe08XQScGeBLTSqR5fK19dmrijYQZUR1wb+msSqBR2zHb8K53hQh1vE3wec+WB v8fzSVgfYh4rO84M4uuhfM4r7TSXXi2mlVErAce+d2QxOulbFGJ5HrsUWjnM4mRpF7 d145U3IkIKPog== From: Leon Romanovsky To: Marek Szyprowski , Jens Axboe , Christoph Hellwig , Keith Busch Cc: Jake Edge , Jonathan Corbet , Jason Gunthorpe , Zhu Yanjun , Robin Murphy , Joerg Roedel , Will Deacon , Sagi Grimberg , Bjorn Helgaas , Logan Gunthorpe , Yishai Hadas , Shameer Kolothum , Kevin Tian , Alex Williamson , =?UTF-8?q?J=C3=A9r=C3=B4me=20Glisse?= , Andrew Morton , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-rdma@vger.kernel.org, iommu@lists.linux.dev, linux-nvme@lists.infradead.org, linux-pci@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org, Niklas Schnelle , Chuck Lever , Luis Chamberlain , Matthew Wilcox , Dan Williams , Kanchan Joshi , Chaitanya Kulkarni , Leon Romanovsky Subject: [PATCH v10 19/24] block: don't merge different kinds of P2P transfers in a single bio Date: Mon, 28 Apr 2025 12:22:25 +0300 Message-ID: X-Mailer: git-send-email 2.49.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Christoph Hellwig To get out of the dma mapping helpers having to check every segment for it's P2P status, ensure that bios either contain P2P transfers or non-P2P transfers, and that a P2P bio only contains ranges from a single device. This means we do the page zone access in the bio add path where it should be still page hot, and will only have do the fairly expensive P2P topology lookup once per bio down in the dma mapping path, and only for already marked bios. Signed-off-by: Christoph Hellwig Reviewed-by: Logan Gunthorpe Tested-by: Jens Axboe Signed-off-by: Leon Romanovsky --- block/bio.c | 17 ++++++++++------- block/blk-merge.c | 17 +++++++++++------ include/linux/blk_types.h | 2 ++ 3 files changed, 23 insertions(+), 13 deletions(-) diff --git a/block/bio.c b/block/bio.c index 3047fa3f4b32..279eac2396bf 100644 --- a/block/bio.c +++ b/block/bio.c @@ -928,8 +928,6 @@ static bool bvec_try_merge_page(struct bio_vec *bv, str= uct page *page, return false; if (xen_domain() && !xen_biovec_phys_mergeable(bv, page)) return false; - if (!zone_device_pages_have_same_pgmap(bv->bv_page, page)) - return false; =20 *same_page =3D ((vec_end_addr & PAGE_MASK) =3D=3D ((page_addr + off) & PAGE_MASK)); @@ -998,11 +996,16 @@ static int bio_add_page_int(struct bio *bio, struct p= age *page, if (bio->bi_iter.bi_size > UINT_MAX - len) return 0; =20 - if (bio->bi_vcnt > 0 && - bvec_try_merge_page(&bio->bi_io_vec[bio->bi_vcnt - 1], - page, len, offset, same_page)) { - bio->bi_iter.bi_size +=3D len; - return len; + if (bio->bi_vcnt > 0) { + struct bio_vec *bv =3D &bio->bi_io_vec[bio->bi_vcnt - 1]; + + if (bvec_try_merge_page(bv, page, len, offset, same_page)) { + bio->bi_iter.bi_size +=3D len; + return len; + } + } else { + if (is_pci_p2pdma_page(page)) + bio->bi_opf |=3D REQ_P2PDMA | REQ_NOMERGE; } =20 if (bio->bi_vcnt >=3D bio->bi_max_vecs) diff --git a/block/blk-merge.c b/block/blk-merge.c index fdd4efb54c6c..d9691e900cc6 100644 --- a/block/blk-merge.c +++ b/block/blk-merge.c @@ -320,12 +320,17 @@ int bio_split_rw_at(struct bio *bio, const struct que= ue_limits *lim, unsigned nsegs =3D 0, bytes =3D 0; =20 bio_for_each_bvec(bv, bio, iter) { - /* - * If the queue doesn't support SG gaps and adding this - * offset would create a gap, disallow it. - */ - if (bvprvp && bvec_gap_to_prev(lim, bvprvp, bv.bv_offset)) - goto split; + if (bvprvp) { + /* + * If the queue doesn't support SG gaps and adding this + * offset would create a gap, disallow it. + */ + if (bvec_gap_to_prev(lim, bvprvp, bv.bv_offset)) + goto split; + } else { + if (is_pci_p2pdma_page(bv.bv_page)) + bio->bi_opf |=3D REQ_P2PDMA | REQ_NOMERGE; + } =20 if (nsegs < lim->max_segments && bytes + bv.bv_len <=3D max_bytes && diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h index dce7615c35e7..94cf146e8ce6 100644 --- a/include/linux/blk_types.h +++ b/include/linux/blk_types.h @@ -378,6 +378,7 @@ enum req_flag_bits { __REQ_DRV, /* for driver use */ __REQ_FS_PRIVATE, /* for file system (submitter) use */ __REQ_ATOMIC, /* for atomic write operations */ + __REQ_P2PDMA, /* contains P2P DMA pages */ /* * Command specific flags, keep last: */ @@ -410,6 +411,7 @@ enum req_flag_bits { #define REQ_DRV (__force blk_opf_t)(1ULL << __REQ_DRV) #define REQ_FS_PRIVATE (__force blk_opf_t)(1ULL << __REQ_FS_PRIVATE) #define REQ_ATOMIC (__force blk_opf_t)(1ULL << __REQ_ATOMIC) +#define REQ_P2PDMA (__force blk_opf_t)(1ULL << __REQ_P2PDMA) =20 #define REQ_NOUNMAP (__force blk_opf_t)(1ULL << __REQ_NOUNMAP) =20 --=20 2.49.0 From nobody Tue Dec 16 14:20:07 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0E136262FDF; Mon, 28 Apr 2025 09:24:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745832245; cv=none; b=Monumwci/s9iCNb3/0DnVqgvtmWeoixuoUaC0OiLvPi4U3DB6PC1v9ZuIVFQgffyG0d8CjY9MpPqKs8IqAui5OmQ/0cEqSWmTepAXwKgoEHn7NCbDmBTVCbpleKXFuLqtw9yS+x2xX1IqhvyxquCSTptArDmz6boPYsLUMbwmcE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745832245; c=relaxed/simple; bh=9rhHvPOzSHQlA2XuEjZ/1K4qwc/mwIdcZSmcRZVSrto=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=kkDA4CY7Cv3canHvVIN8ZJO4tJeVrGQBaWvYVd8kpYvpEzotIhn48bu25USU5sV5dHM4c2IIwk28TSB5Vmr64eDng3dMd/5eR82gYjkUvbFdC0jWpoCO4P4T9ZpMGPy/89V1SF7ZsyVWqvx3EWjaTMRaynHXciATJ71DJXOGGaQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=uFvZcReR; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="uFvZcReR" Received: by smtp.kernel.org (Postfix) with ESMTPSA id D3ED5C4CEE4; Mon, 28 Apr 2025 09:24:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1745832244; bh=9rhHvPOzSHQlA2XuEjZ/1K4qwc/mwIdcZSmcRZVSrto=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=uFvZcReRnyppSyxP/Et+YkgbD8FvTtrDWn1JdHg7l/xqdHYtSk6JWl3oO9nEx6jwa JEsC7HRJ1d1OOFKf25ixMoKQIIUzMBTE0oGipToJh69rbiUwq0ibxu+66zm190YNSf wqjB5H2EZFN7yAL1/fKwW1IbFfUZbCfKpERzezPpRpqh5qHk3yxxOUcl63VsLg/CFU C9K72jzKe/+LGntR5KzPj3x4r+5JRCeliq5IDo6uoqnGn2ci3O6WYU9Tx1QPsJBq23 BzPAsJOss5n1uGOuAc5eiBpKyH/R+T/DpVhbmlTDpoJHE4yVQimeHBs5q0A7NWMrIy SucdN/ukBTvrw== From: Leon Romanovsky To: Marek Szyprowski , Jens Axboe , Christoph Hellwig , Keith Busch Cc: Jake Edge , Jonathan Corbet , Jason Gunthorpe , Zhu Yanjun , Robin Murphy , Joerg Roedel , Will Deacon , Sagi Grimberg , Bjorn Helgaas , Logan Gunthorpe , Yishai Hadas , Shameer Kolothum , Kevin Tian , Alex Williamson , =?UTF-8?q?J=C3=A9r=C3=B4me=20Glisse?= , Andrew Morton , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-rdma@vger.kernel.org, iommu@lists.linux.dev, linux-nvme@lists.infradead.org, linux-pci@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org, Niklas Schnelle , Chuck Lever , Luis Chamberlain , Matthew Wilcox , Dan Williams , Kanchan Joshi , Chaitanya Kulkarni , Leon Romanovsky Subject: [PATCH v10 20/24] blk-mq: add scatterlist-less DMA mapping helpers Date: Mon, 28 Apr 2025 12:22:26 +0300 Message-ID: <0ecf38e2b83f8ed60cdc3b9be56012654f284751.1745831017.git.leon@kernel.org> X-Mailer: git-send-email 2.49.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Christoph Hellwig Add a new blk_rq_dma_map / blk_rq_dma_unmap pair that does away with the wasteful scatterlist structure. Instead it uses the mapping iterator to either add segments to the IOVA for IOMMU operations, or just maps them one by one for the direct mapping. For the IOMMU case instead of a scatterlist with an entry for each segment, only a single [dma_addr,len] pair needs to be stored for processing a request, and for the direct mapping the per-segment allocation shrinks from [page,offset,len,dma_addr,dma_len] to just [dma_addr,len]. The major downside of this API is that the IOVA collapsing only works when the driver sets a virt_boundary that matches the IOMMU granule. Note that struct blk_dma_vec, struct blk_dma_mapping and blk_rq_dma_unmap aren't really block specific, but for they are kept with the only mapping routine to keep things simple. Signed-off-by: Christoph Hellwig Tested-by: Jens Axboe Signed-off-by: Leon Romanovsky --- block/blk-merge.c | 163 +++++++++++++++++++++++++++++++++++++ include/linux/blk-mq-dma.h | 63 ++++++++++++++ 2 files changed, 226 insertions(+) create mode 100644 include/linux/blk-mq-dma.h diff --git a/block/blk-merge.c b/block/blk-merge.c index d9691e900cc6..5e2e6db60fda 100644 --- a/block/blk-merge.c +++ b/block/blk-merge.c @@ -7,6 +7,8 @@ #include #include #include +#include +#include #include #include #include @@ -535,6 +537,167 @@ static bool blk_map_iter_next(struct request *req, return true; } =20 +/* + * The IOVA-based DMA API wants to be able to coalesce at the minimal IOMM= U page + * size granularity (which is guaranteed to be <=3D PAGE_SIZE and usually = 4k), so + * we need to ensure our segments are aligned to this as well. + * + * Note that there is no point in using the slightly more complicated IOVA= based + * path for single segment mappings. + */ +static inline bool blk_can_dma_map_iova(struct request *req, + struct device *dma_dev) +{ + return !((queue_virt_boundary(req->q) + 1) & + dma_get_merge_boundary(dma_dev)); +} + +static bool blk_dma_map_bus(struct request *req, struct device *dma_dev, + struct blk_dma_iter *iter, struct phys_vec *vec) +{ + iter->addr =3D pci_p2pdma_bus_addr_map(&iter->p2pdma, vec->paddr); + iter->len =3D vec->len; + return true; +} + +static bool blk_dma_map_direct(struct request *req, struct device *dma_dev, + struct blk_dma_iter *iter, struct phys_vec *vec) +{ + iter->addr =3D dma_map_page(dma_dev, phys_to_page(vec->paddr), + offset_in_page(vec->paddr), vec->len, rq_dma_dir(req)); + if (dma_mapping_error(dma_dev, iter->addr)) { + iter->status =3D BLK_STS_RESOURCE; + return false; + } + iter->len =3D vec->len; + return true; +} + +static bool blk_rq_dma_map_iova(struct request *req, struct device *dma_de= v, + struct dma_iova_state *state, struct blk_dma_iter *iter, + struct phys_vec *vec) +{ + enum dma_data_direction dir =3D rq_dma_dir(req); + unsigned int mapped =3D 0; + int error =3D 0; + + iter->addr =3D state->addr; + iter->len =3D dma_iova_size(state); + + do { + error =3D dma_iova_link(dma_dev, state, vec->paddr, mapped, + vec->len, dir, 0); + if (error) + break; + mapped +=3D vec->len; + } while (blk_map_iter_next(req, &iter->iter, vec)); + + error =3D dma_iova_sync(dma_dev, state, 0, mapped); + if (error) { + iter->status =3D errno_to_blk_status(error); + return false; + } + + return true; +} + +/** + * blk_rq_dma_map_iter_start - map the first DMA segment for a request + * @req: request to map + * @dma_dev: device to map to + * @state: DMA IOVA state + * @iter: block layer DMA iterator + * + * Start DMA mapping @req to @dma_dev. @state and @iter are provided by t= he + * caller and don't need to be initialized. @state needs to be stored for= use + * at unmap time, @iter is only needed at map time. + * + * Returns %false if there is no segment to map, including due to an error= , or + * %true ft it did map a segment. + * + * If a segment was mapped, the DMA address for it is returned in @iter.ad= dr and + * the length in @iter.len. If no segment was mapped the status code is + * returned in @iter.status. + * + * The caller can call blk_rq_dma_map_coalesce() to check if further segme= nts + * need to be mapped after this, or go straight to blk_rq_dma_map_iter_nex= t() + * to try to map the following segments. + */ +bool blk_rq_dma_map_iter_start(struct request *req, struct device *dma_dev, + struct dma_iova_state *state, struct blk_dma_iter *iter) +{ + unsigned int total_len =3D blk_rq_payload_bytes(req); + struct phys_vec vec; + + iter->iter.bio =3D req->bio; + iter->iter.iter =3D req->bio->bi_iter; + memset(&iter->p2pdma, 0, sizeof(iter->p2pdma)); + iter->status =3D BLK_STS_OK; + + /* + * Grab the first segment ASAP because we'll need it to check for P2P + * transfers. + */ + if (!blk_map_iter_next(req, &iter->iter, &vec)) + return false; + + if (IS_ENABLED(CONFIG_PCI_P2PDMA) && (req->cmd_flags & REQ_P2PDMA)) { + switch (pci_p2pdma_state(&iter->p2pdma, dma_dev, + phys_to_page(vec.paddr))) { + case PCI_P2PDMA_MAP_BUS_ADDR: + return blk_dma_map_bus(req, dma_dev, iter, &vec); + case PCI_P2PDMA_MAP_THRU_HOST_BRIDGE: + /* + * P2P transfers through the host bridge are treated the + * same as non-P2P transfers below and during unmap. + */ + req->cmd_flags &=3D ~REQ_P2PDMA; + break; + default: + iter->status =3D BLK_STS_INVAL; + return false; + } + } + + if (blk_can_dma_map_iova(req, dma_dev) && + dma_iova_try_alloc(dma_dev, state, vec.paddr, total_len)) + return blk_rq_dma_map_iova(req, dma_dev, state, iter, &vec); + return blk_dma_map_direct(req, dma_dev, iter, &vec); +} +EXPORT_SYMBOL_GPL(blk_rq_dma_map_iter_start); + +/** + * blk_rq_dma_map_iter_next - map the next DMA segment for a request + * @req: request to map + * @dma_dev: device to map to + * @state: DMA IOVA state + * @iter: block layer DMA iterator + * + * Iterate to the next mapping after a previous call to + * blk_rq_dma_map_iter_start(). See there for a detailed description of t= he + * arguments. + * + * Returns %false if there is no segment to map, including due to an error= , or + * %true ft it did map a segment. + * + * If a segment was mapped, the DMA address for it is returned in @iter.ad= dr and + * the length in @iter.len. If no segment was mapped the status code is + * returned in @iter.status. + */ +bool blk_rq_dma_map_iter_next(struct request *req, struct device *dma_dev, + struct dma_iova_state *state, struct blk_dma_iter *iter) +{ + struct phys_vec vec; + + if (!blk_map_iter_next(req, &iter->iter, &vec)) + return false; + + if (iter->p2pdma.map =3D=3D PCI_P2PDMA_MAP_BUS_ADDR) + return blk_dma_map_bus(req, dma_dev, iter, &vec); + return blk_dma_map_direct(req, dma_dev, iter, &vec); +} +EXPORT_SYMBOL_GPL(blk_rq_dma_map_iter_next); + static inline struct scatterlist *blk_next_sg(struct scatterlist **sg, struct scatterlist *sglist) { diff --git a/include/linux/blk-mq-dma.h b/include/linux/blk-mq-dma.h new file mode 100644 index 000000000000..6d85b4bedcba --- /dev/null +++ b/include/linux/blk-mq-dma.h @@ -0,0 +1,63 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef BLK_MQ_DMA_H +#define BLK_MQ_DMA_H + +#include +#include + +struct blk_dma_iter { + /* Output address range for this iteration */ + dma_addr_t addr; + u32 len; + + /* Status code. Only valid when blk_rq_dma_map_iter_* returned false */ + blk_status_t status; + + /* Internal to blk_rq_dma_map_iter_* */ + struct req_iterator iter; + struct pci_p2pdma_map_state p2pdma; +}; + +bool blk_rq_dma_map_iter_start(struct request *req, struct device *dma_dev, + struct dma_iova_state *state, struct blk_dma_iter *iter); +bool blk_rq_dma_map_iter_next(struct request *req, struct device *dma_dev, + struct dma_iova_state *state, struct blk_dma_iter *iter); + +/** + * blk_rq_dma_map_coalesce - were all segments coalesced? + * @state: DMA state to check + * + * Returns true if blk_rq_dma_map_iter_start coalesced all segments into a + * single DMA range. + */ +static inline bool blk_rq_dma_map_coalesce(struct dma_iova_state *state) +{ + return dma_use_iova(state); +} + +/** + * blk_rq_dma_unmap - try to DMA unmap a request + * @req: request to unmap + * @dma_dev: device to unmap from + * @state: DMA IOVA state + * @mapped_len: number of bytes to unmap + * + * Returns %false if the callers need to manually unmap every DMA segment + * mapped using @iter or %true if no work is left to be done. + */ +static inline bool blk_rq_dma_unmap(struct request *req, struct device *dm= a_dev, + struct dma_iova_state *state, size_t mapped_len) +{ + if (req->cmd_flags & REQ_P2PDMA) + return true; + + if (dma_use_iova(state)) { + dma_iova_destroy(dma_dev, state, mapped_len, rq_dma_dir(req), + 0); + return true; + } + + return !dma_need_unmap(dma_dev); +} + +#endif /* BLK_MQ_DMA_H */ --=20 2.49.0 From nobody Tue Dec 16 14:20:07 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 32439277805; Mon, 28 Apr 2025 09:24:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745832249; cv=none; b=bsjfNdzyQrWFQ/irT8x5dHtCu0y3u7asDPRDcNv4POB9I/JWEgu7iQjMrHiN+zS5NyiQb8o1JSY4FrE7EtRJT4NoP+5T/Wzp+ogUGHTZCXC4d+OytlEjDE3LjCOHmX35nvRTS6QCBSTpBqVMsnAq6cZneCbrAdxV6a5zLTmj/f4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745832249; c=relaxed/simple; bh=UaKpIS/gS7437zv6/V/pxWOCRAoH+oFaLCUvPin30s0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=jAdLsytzEbtjCAdwJURHnTCtsP2mmPKZvS08Xza+Uy6ge+lHLR1PVL3OIyv57dqYqwkhHWCrNbzPVIJO7s3Z93YSLeAHNAigXtEbzPfDogtENGD2VlI/TsXBdmFogYhN60JJ34yAf+826GPXqWkMqZ7bcztjk6fnTdmNl56QUQw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=G/Ukn2zS; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="G/Ukn2zS" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 32880C4CEF1; Mon, 28 Apr 2025 09:24:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1745832249; bh=UaKpIS/gS7437zv6/V/pxWOCRAoH+oFaLCUvPin30s0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=G/Ukn2zSIrjyDxxMBbuxC8G0NNgq427GlA1UdIt9h8WdAt09mSJLJAba2Uk1lLjdc hH7iI67XbyEfiLiAGReOTrPmgM+2w6SXxCJhYj0DC/2JppgeGqGcc8yp0T02GPKplm cxh7af7jGCCF9Z1YVpTA5n7AZuT01VsUpgTbUveUSNG0zsBx2D91spU8nHzbKdRJPk gcuUZzojGyQ2vTalz6AsYECS4vE2AKqovwQhgogBprkg3X9XO2QD386PwCYLTjClDm L2sa6AahrrOtplIHyeQQ0WtZk2mTFeZYNwAoQ0Zr4eUW7xrMGIsTElcXX97TUuMb9O RMHi0c0PYikjA== From: Leon Romanovsky To: Marek Szyprowski , Jens Axboe , Christoph Hellwig , Keith Busch Cc: Jake Edge , Jonathan Corbet , Jason Gunthorpe , Zhu Yanjun , Robin Murphy , Joerg Roedel , Will Deacon , Sagi Grimberg , Bjorn Helgaas , Logan Gunthorpe , Yishai Hadas , Shameer Kolothum , Kevin Tian , Alex Williamson , =?UTF-8?q?J=C3=A9r=C3=B4me=20Glisse?= , Andrew Morton , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-rdma@vger.kernel.org, iommu@lists.linux.dev, linux-nvme@lists.infradead.org, linux-pci@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org, Niklas Schnelle , Chuck Lever , Luis Chamberlain , Matthew Wilcox , Dan Williams , Kanchan Joshi , Chaitanya Kulkarni , Leon Romanovsky Subject: [PATCH v10 21/24] nvme-pci: remove struct nvme_descriptor Date: Mon, 28 Apr 2025 12:22:27 +0300 Message-ID: X-Mailer: git-send-email 2.49.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Christoph Hellwig There is no real point in having a union of two pointer types here, just use a void pointer as we mix and match types between the arms of the union between the allocation and freeing side already. Also rename the nr_allocations field to nr_descriptors to better describe what it does. Signed-off-by: Christoph Hellwig Tested-by: Jens Axboe Signed-off-by: Leon Romanovsky --- drivers/nvme/host/pci.c | 57 +++++++++++++++++------------------------ 1 file changed, 24 insertions(+), 33 deletions(-) diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index b178d52eac1b..638e759b29ad 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -43,7 +43,7 @@ #define NVME_MAX_KB_SZ 8192 #define NVME_MAX_SEGS 128 #define NVME_MAX_META_SEGS 15 -#define NVME_MAX_NR_ALLOCATIONS 5 +#define NVME_MAX_NR_DESCRIPTORS 5 =20 static int use_threaded_interrupts; module_param(use_threaded_interrupts, int, 0444); @@ -219,30 +219,22 @@ struct nvme_queue { struct completion delete_done; }; =20 -union nvme_descriptor { - struct nvme_sgl_desc *sg_list; - __le64 *prp_list; -}; - /* * The nvme_iod describes the data in an I/O. - * - * The sg pointer contains the list of PRP/SGL chunk allocations in additi= on - * to the actual struct scatterlist. */ struct nvme_iod { struct nvme_request req; struct nvme_command cmd; bool aborted; - s8 nr_allocations; /* PRP list pool allocations. 0 means small - pool in use */ + /* # of PRP/SGL descriptors: (0 for small pool) */ + s8 nr_descriptors; unsigned int dma_len; /* length of single DMA segment mapping */ dma_addr_t first_dma; dma_addr_t meta_dma; struct sg_table sgt; struct sg_table meta_sgt; - union nvme_descriptor meta_list; - union nvme_descriptor list[NVME_MAX_NR_ALLOCATIONS]; + void *meta_list; + void *descriptors[NVME_MAX_NR_DESCRIPTORS]; }; =20 static inline unsigned int nvme_dbbuf_size(struct nvme_dev *dev) @@ -544,8 +536,8 @@ static void nvme_free_prps(struct nvme_dev *dev, struct= request *req) dma_addr_t dma_addr =3D iod->first_dma; int i; =20 - for (i =3D 0; i < iod->nr_allocations; i++) { - __le64 *prp_list =3D iod->list[i].prp_list; + for (i =3D 0; i < iod->nr_descriptors; i++) { + __le64 *prp_list =3D iod->descriptors[i]; dma_addr_t next_dma_addr =3D le64_to_cpu(prp_list[last_prp]); =20 dma_pool_free(dev->prp_page_pool, prp_list, dma_addr); @@ -567,11 +559,11 @@ static void nvme_unmap_data(struct nvme_dev *dev, str= uct request *req) =20 dma_unmap_sgtable(dev->dev, &iod->sgt, rq_dma_dir(req), 0); =20 - if (iod->nr_allocations =3D=3D 0) - dma_pool_free(dev->prp_small_pool, iod->list[0].sg_list, + if (iod->nr_descriptors =3D=3D 0) + dma_pool_free(dev->prp_small_pool, iod->descriptors[0], iod->first_dma); - else if (iod->nr_allocations =3D=3D 1) - dma_pool_free(dev->prp_page_pool, iod->list[0].sg_list, + else if (iod->nr_descriptors =3D=3D 1) + dma_pool_free(dev->prp_page_pool, iod->descriptors[0], iod->first_dma); else nvme_free_prps(dev, req); @@ -629,18 +621,18 @@ static blk_status_t nvme_pci_setup_prps(struct nvme_d= ev *dev, nprps =3D DIV_ROUND_UP(length, NVME_CTRL_PAGE_SIZE); if (nprps <=3D (256 / 8)) { pool =3D dev->prp_small_pool; - iod->nr_allocations =3D 0; + iod->nr_descriptors =3D 0; } else { pool =3D dev->prp_page_pool; - iod->nr_allocations =3D 1; + iod->nr_descriptors =3D 1; } =20 prp_list =3D dma_pool_alloc(pool, GFP_ATOMIC, &prp_dma); if (!prp_list) { - iod->nr_allocations =3D -1; + iod->nr_descriptors =3D -1; return BLK_STS_RESOURCE; } - iod->list[0].prp_list =3D prp_list; + iod->descriptors[0] =3D prp_list; iod->first_dma =3D prp_dma; i =3D 0; for (;;) { @@ -649,7 +641,7 @@ static blk_status_t nvme_pci_setup_prps(struct nvme_dev= *dev, prp_list =3D dma_pool_alloc(pool, GFP_ATOMIC, &prp_dma); if (!prp_list) goto free_prps; - iod->list[iod->nr_allocations++].prp_list =3D prp_list; + iod->descriptors[iod->nr_descriptors++] =3D prp_list; prp_list[0] =3D old_prp_list[i - 1]; old_prp_list[i - 1] =3D cpu_to_le64(prp_dma); i =3D 1; @@ -719,19 +711,19 @@ static blk_status_t nvme_pci_setup_sgls(struct nvme_d= ev *dev, =20 if (entries <=3D (256 / sizeof(struct nvme_sgl_desc))) { pool =3D dev->prp_small_pool; - iod->nr_allocations =3D 0; + iod->nr_descriptors =3D 0; } else { pool =3D dev->prp_page_pool; - iod->nr_allocations =3D 1; + iod->nr_descriptors =3D 1; } =20 sg_list =3D dma_pool_alloc(pool, GFP_ATOMIC, &sgl_dma); if (!sg_list) { - iod->nr_allocations =3D -1; + iod->nr_descriptors =3D -1; return BLK_STS_RESOURCE; } =20 - iod->list[0].sg_list =3D sg_list; + iod->descriptors[0] =3D sg_list; iod->first_dma =3D sgl_dma; =20 nvme_pci_sgl_set_seg(&cmd->dptr.sgl, sgl_dma, entries); @@ -870,7 +862,7 @@ static blk_status_t nvme_pci_setup_meta_sgls(struct nvm= e_dev *dev, goto out_unmap_sg; =20 entries =3D iod->meta_sgt.nents; - iod->meta_list.sg_list =3D sg_list; + iod->meta_list =3D sg_list; iod->meta_dma =3D sgl_dma; =20 cmnd->flags =3D NVME_CMD_SGL_METASEG; @@ -923,7 +915,7 @@ static blk_status_t nvme_prep_rq(struct nvme_dev *dev, = struct request *req) blk_status_t ret; =20 iod->aborted =3D false; - iod->nr_allocations =3D -1; + iod->nr_descriptors =3D -1; iod->sgt.nents =3D 0; iod->meta_sgt.nents =3D 0; =20 @@ -1048,8 +1040,7 @@ static __always_inline void nvme_unmap_metadata(struc= t nvme_dev *dev, return; } =20 - dma_pool_free(dev->prp_small_pool, iod->meta_list.sg_list, - iod->meta_dma); + dma_pool_free(dev->prp_small_pool, iod->meta_list, iod->meta_dma); dma_unmap_sgtable(dev->dev, &iod->meta_sgt, rq_dma_dir(req), 0); mempool_free(iod->meta_sgt.sgl, dev->iod_meta_mempool); } @@ -3801,7 +3792,7 @@ static int __init nvme_init(void) BUILD_BUG_ON(IRQ_AFFINITY_MAX_SETS < 2); BUILD_BUG_ON(NVME_MAX_SEGS > SGES_PER_PAGE); BUILD_BUG_ON(sizeof(struct scatterlist) * NVME_MAX_SEGS > PAGE_SIZE); - BUILD_BUG_ON(nvme_pci_npages_prp() > NVME_MAX_NR_ALLOCATIONS); + BUILD_BUG_ON(nvme_pci_npages_prp() > NVME_MAX_NR_DESCRIPTORS); =20 return pci_register_driver(&nvme_driver); } --=20 2.49.0 From nobody Tue Dec 16 14:20:07 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3694E278E44; Mon, 28 Apr 2025 09:24:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745832250; cv=none; b=jEZgmRw2xp5/jFJuI6pXJ0AaA2jIhFFD5wMbpA3725330E+N86h8SL5289qVmoaBGqO66k4bpDQS+wnwe016qD7O9fpz5HICas1N8POvi8m/3FlnoKkHTKOm45B0dH8ywIdMrXsoBcuJumzRxQXNtCHUfPM8rmCFqFIDVwzApXw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745832250; c=relaxed/simple; bh=o9rUJvhmsdRR03YWE52xBlknh79MIWTsCmYUjg7Cbqs=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=SzSVbtxohgsqAbQZesRQFjy/wJTbaafYYFSyLVU8qmzy4p8ZvJHsDF/p9z81tXDZjGlAiVQQnzdia6f2NaRqChsYqpwZIxgjzAvo5UwLznl++xmZXbNyfW1q2evEY/JgHJDtyanXvkIPLoKkI0AOs46chjdnEvKHRMfuOEWSoO4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=ltrLv3YB; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="ltrLv3YB" Received: by smtp.kernel.org (Postfix) with ESMTPSA id D8E77C4CEE9; Mon, 28 Apr 2025 09:24:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1745832249; bh=o9rUJvhmsdRR03YWE52xBlknh79MIWTsCmYUjg7Cbqs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ltrLv3YB0fnSj7N1PfR4mUaSmfOQaR15pMstnUdv3blO6wa7TLBiupO1Qc9uTWN/D qL+Dmw0jaiH/F6BDohX9UQbeve9saTFctay+nWuJmg3kPhQBkYu5ai/9vrvFW5M0MT jt7zztCyEcSj/YyXMxnnO/gFuND0R6EdJ8q4A3kUfd4+Sa5OSwKuHZD3FtoKTvChVU xbVgHwQ1XYVasCaroM1RE0VY4rdG9FRfHhGx0Smf61xyRArCRJ5fe+VCTHRPBaQzEp u2o8MiK33JiaLeDxoJ1LBaOJ0MElGPSOJm1lzFyZkvvzDK1Q9Ei4V7lveThmLGVJ57 8zI62VvtG9iuA== From: Leon Romanovsky To: Marek Szyprowski , Jens Axboe , Christoph Hellwig , Keith Busch Cc: Jake Edge , Jonathan Corbet , Jason Gunthorpe , Zhu Yanjun , Robin Murphy , Joerg Roedel , Will Deacon , Sagi Grimberg , Bjorn Helgaas , Logan Gunthorpe , Yishai Hadas , Shameer Kolothum , Kevin Tian , Alex Williamson , =?UTF-8?q?J=C3=A9r=C3=B4me=20Glisse?= , Andrew Morton , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-rdma@vger.kernel.org, iommu@lists.linux.dev, linux-nvme@lists.infradead.org, linux-pci@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org, Niklas Schnelle , Chuck Lever , Luis Chamberlain , Matthew Wilcox , Dan Williams , Kanchan Joshi , Chaitanya Kulkarni , Leon Romanovsky Subject: [PATCH v10 22/24] nvme-pci: use a better encoding for small prp pool allocations Date: Mon, 28 Apr 2025 12:22:28 +0300 Message-ID: X-Mailer: git-send-email 2.49.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Christoph Hellwig There is plenty of unused space in the iod next to nr_descriptors. Add a separate flag to encode that the transfer is using the full page sized pool, and use a normal 0..n count for the number of descriptors. Signed-off-by: Christoph Hellwig Tested-by: Jens Axboe [ Leon: changed original bool variable to be flag as was proposed by Kancha= n ] Signed-off-by: Kanchan Joshi Signed-off-by: Leon Romanovsky --- drivers/nvme/host/pci.c | 94 ++++++++++++++++++++--------------------- 1 file changed, 47 insertions(+), 47 deletions(-) diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index 638e759b29ad..270fb5146b5b 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -44,6 +44,7 @@ #define NVME_MAX_SEGS 128 #define NVME_MAX_META_SEGS 15 #define NVME_MAX_NR_DESCRIPTORS 5 +#define NVME_SMALL_DESCRIPTOR_SIZE 256 =20 static int use_threaded_interrupts; module_param(use_threaded_interrupts, int, 0444); @@ -219,6 +220,10 @@ struct nvme_queue { struct completion delete_done; }; =20 +enum { + IOD_LARGE_DESCRIPTORS =3D 1 << 0, /* uses the full page sized descriptor = pool */ +}; + /* * The nvme_iod describes the data in an I/O. */ @@ -226,8 +231,8 @@ struct nvme_iod { struct nvme_request req; struct nvme_command cmd; bool aborted; - /* # of PRP/SGL descriptors: (0 for small pool) */ - s8 nr_descriptors; + u8 nr_descriptors; /* # of PRP/SGL descriptors */ + u8 flags; unsigned int dma_len; /* length of single DMA segment mapping */ dma_addr_t first_dma; dma_addr_t meta_dma; @@ -529,13 +534,27 @@ static inline bool nvme_pci_use_sgls(struct nvme_dev = *dev, struct request *req, return true; } =20 -static void nvme_free_prps(struct nvme_dev *dev, struct request *req) +static inline struct dma_pool *nvme_dma_pool(struct nvme_dev *dev, + struct nvme_iod *iod) +{ + if (iod->flags & IOD_LARGE_DESCRIPTORS) + return dev->prp_page_pool; + return dev->prp_small_pool; +} + +static void nvme_free_descriptors(struct nvme_dev *dev, struct request *re= q) { const int last_prp =3D NVME_CTRL_PAGE_SIZE / sizeof(__le64) - 1; struct nvme_iod *iod =3D blk_mq_rq_to_pdu(req); dma_addr_t dma_addr =3D iod->first_dma; int i; =20 + if (iod->nr_descriptors =3D=3D 1) { + dma_pool_free(nvme_dma_pool(dev, iod), iod->descriptors[0], + dma_addr); + return; + } + for (i =3D 0; i < iod->nr_descriptors; i++) { __le64 *prp_list =3D iod->descriptors[i]; dma_addr_t next_dma_addr =3D le64_to_cpu(prp_list[last_prp]); @@ -558,15 +577,7 @@ static void nvme_unmap_data(struct nvme_dev *dev, stru= ct request *req) WARN_ON_ONCE(!iod->sgt.nents); =20 dma_unmap_sgtable(dev->dev, &iod->sgt, rq_dma_dir(req), 0); - - if (iod->nr_descriptors =3D=3D 0) - dma_pool_free(dev->prp_small_pool, iod->descriptors[0], - iod->first_dma); - else if (iod->nr_descriptors =3D=3D 1) - dma_pool_free(dev->prp_page_pool, iod->descriptors[0], - iod->first_dma); - else - nvme_free_prps(dev, req); + nvme_free_descriptors(dev, req); mempool_free(iod->sgt.sgl, dev->iod_mempool); } =20 @@ -588,7 +599,6 @@ static blk_status_t nvme_pci_setup_prps(struct nvme_dev= *dev, struct request *req, struct nvme_rw_command *cmnd) { struct nvme_iod *iod =3D blk_mq_rq_to_pdu(req); - struct dma_pool *pool; int length =3D blk_rq_payload_bytes(req); struct scatterlist *sg =3D iod->sgt.sgl; int dma_len =3D sg_dma_len(sg); @@ -596,7 +606,7 @@ static blk_status_t nvme_pci_setup_prps(struct nvme_dev= *dev, int offset =3D dma_addr & (NVME_CTRL_PAGE_SIZE - 1); __le64 *prp_list; dma_addr_t prp_dma; - int nprps, i; + int i; =20 length -=3D (NVME_CTRL_PAGE_SIZE - offset); if (length <=3D 0) { @@ -618,27 +628,23 @@ static blk_status_t nvme_pci_setup_prps(struct nvme_d= ev *dev, goto done; } =20 - nprps =3D DIV_ROUND_UP(length, NVME_CTRL_PAGE_SIZE); - if (nprps <=3D (256 / 8)) { - pool =3D dev->prp_small_pool; - iod->nr_descriptors =3D 0; - } else { - pool =3D dev->prp_page_pool; - iod->nr_descriptors =3D 1; - } + if (DIV_ROUND_UP(length, NVME_CTRL_PAGE_SIZE) > + NVME_SMALL_DESCRIPTOR_SIZE / sizeof(__le64)) + iod->flags |=3D IOD_LARGE_DESCRIPTORS; =20 - prp_list =3D dma_pool_alloc(pool, GFP_ATOMIC, &prp_dma); - if (!prp_list) { - iod->nr_descriptors =3D -1; + prp_list =3D dma_pool_alloc(nvme_dma_pool(dev, iod), GFP_ATOMIC, + &prp_dma); + if (!prp_list) return BLK_STS_RESOURCE; - } - iod->descriptors[0] =3D prp_list; + iod->descriptors[iod->nr_descriptors++] =3D prp_list; iod->first_dma =3D prp_dma; i =3D 0; for (;;) { if (i =3D=3D NVME_CTRL_PAGE_SIZE >> 3) { __le64 *old_prp_list =3D prp_list; - prp_list =3D dma_pool_alloc(pool, GFP_ATOMIC, &prp_dma); + + prp_list =3D dma_pool_alloc(dev->prp_page_pool, + GFP_ATOMIC, &prp_dma); if (!prp_list) goto free_prps; iod->descriptors[iod->nr_descriptors++] =3D prp_list; @@ -665,7 +671,7 @@ static blk_status_t nvme_pci_setup_prps(struct nvme_dev= *dev, cmnd->dptr.prp2 =3D cpu_to_le64(iod->first_dma); return BLK_STS_OK; free_prps: - nvme_free_prps(dev, req); + nvme_free_descriptors(dev, req); return BLK_STS_RESOURCE; bad_sgl: WARN(DO_ONCE(nvme_print_sgl, iod->sgt.sgl, iod->sgt.nents), @@ -694,7 +700,6 @@ static blk_status_t nvme_pci_setup_sgls(struct nvme_dev= *dev, struct request *req, struct nvme_rw_command *cmd) { struct nvme_iod *iod =3D blk_mq_rq_to_pdu(req); - struct dma_pool *pool; struct nvme_sgl_desc *sg_list; struct scatterlist *sg =3D iod->sgt.sgl; unsigned int entries =3D iod->sgt.nents; @@ -709,21 +714,13 @@ static blk_status_t nvme_pci_setup_sgls(struct nvme_d= ev *dev, return BLK_STS_OK; } =20 - if (entries <=3D (256 / sizeof(struct nvme_sgl_desc))) { - pool =3D dev->prp_small_pool; - iod->nr_descriptors =3D 0; - } else { - pool =3D dev->prp_page_pool; - iod->nr_descriptors =3D 1; - } + if (entries > NVME_SMALL_DESCRIPTOR_SIZE / sizeof(*sg_list)) + iod->flags |=3D IOD_LARGE_DESCRIPTORS; =20 - sg_list =3D dma_pool_alloc(pool, GFP_ATOMIC, &sgl_dma); - if (!sg_list) { - iod->nr_descriptors =3D -1; + sg_list =3D dma_pool_alloc(nvme_dma_pool(dev, iod), GFP_ATOMIC, &sgl_dma); + if (!sg_list) return BLK_STS_RESOURCE; - } - - iod->descriptors[0] =3D sg_list; + iod->descriptors[iod->nr_descriptors++] =3D sg_list; iod->first_dma =3D sgl_dma; =20 nvme_pci_sgl_set_seg(&cmd->dptr.sgl, sgl_dma, entries); @@ -915,7 +912,8 @@ static blk_status_t nvme_prep_rq(struct nvme_dev *dev, = struct request *req) blk_status_t ret; =20 iod->aborted =3D false; - iod->nr_descriptors =3D -1; + iod->nr_descriptors =3D 0; + iod->flags =3D 0; iod->sgt.nents =3D 0; iod->meta_sgt.nents =3D 0; =20 @@ -2833,7 +2831,7 @@ static int nvme_disable_prepare_reset(struct nvme_dev= *dev, bool shutdown) =20 static int nvme_setup_prp_pools(struct nvme_dev *dev) { - size_t small_align =3D 256; + size_t small_align =3D NVME_SMALL_DESCRIPTOR_SIZE; =20 dev->prp_page_pool =3D dma_pool_create("prp list page", dev->dev, NVME_CTRL_PAGE_SIZE, @@ -2841,12 +2839,14 @@ static int nvme_setup_prp_pools(struct nvme_dev *de= v) if (!dev->prp_page_pool) return -ENOMEM; =20 + BUILD_BUG_ON(NVME_SMALL_DESCRIPTOR_SIZE !=3D 256); if (dev->ctrl.quirks & NVME_QUIRK_DMAPOOL_ALIGN_512) - small_align =3D 512; + small_align *=3D 2; =20 /* Optimisation for I/Os between 4k and 128k */ dev->prp_small_pool =3D dma_pool_create("prp list 256", dev->dev, - 256, small_align, 0); + NVME_SMALL_DESCRIPTOR_SIZE, + small_align, 0); if (!dev->prp_small_pool) { dma_pool_destroy(dev->prp_page_pool); return -ENOMEM; --=20 2.49.0 From nobody Tue Dec 16 14:20:07 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 108C5264616; Mon, 28 Apr 2025 09:24:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745832265; cv=none; b=MGlpPj2yCodwFgeG/bp2TA/GsmTf73RaNkG98V244AgUT/rA69cxdgs97HAg5iAqmfVa+AI65+P58IYZ1uPfE43S6vBRIBliPUUDuKseTWzOg0Jq1ldHvMAL0L+Htxxlrkh7iqNLaAjQD0xCMoOK820ZztAb39DCRijNMKIDL7Y= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745832265; c=relaxed/simple; bh=9vP4KAaf7eHx9mt8ZdGXz9CUCxSxRLWpwXr/m0S1xfs=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=qLnc96NaCHGMzaxIZK2E3U56GwLJBJGZC5uEq8JASUtyMKXGzJjmga5lH0FxogTjme/d7Bee6IF1gZWq4aBDIzFdd9GZ7VjS0Hh5cw0Vu9NBcjRnqOErXvdotuzJAGBqGQFFzYbE8d8R2IAWnpBSNH+7/ysJGcfrAzrqLWqa5oU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=DTlAXiAg; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="DTlAXiAg" Received: by smtp.kernel.org (Postfix) with ESMTPSA id DA21AC4CEE4; Mon, 28 Apr 2025 09:24:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1745832264; bh=9vP4KAaf7eHx9mt8ZdGXz9CUCxSxRLWpwXr/m0S1xfs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=DTlAXiAgqfXWqTa/ZRpoyze2KOjtuLs8Hmwqw1wFHLtj0zOYR6v52T4Fnt0GZdS7p V+B2eZtFyDfV6P98uiromPF4qPW+Y6flq/8OxlmrwHybt0FqXw8rKHLpAT+Fn6nUDD PPPN0fuxUud8EjyFn9neTKQ4svCvuxVUC/eFsXE8FGBk0AaL1NgDEF9MMf44YdOST4 TRsBkkxdFmLTUbzLi6WHSD+ibWRgIv/IPX8AmztCQB7y5PuI6/H1NV/wSeSQtuBrzq azRv+KaqDurKP1ubcGlPWrYwHgQSS9I/RA2EeWuyaXsfMiF4FN7zSgLXTKYptxvI8U x7KLvin4isKfg== From: Leon Romanovsky To: Marek Szyprowski , Jens Axboe , Christoph Hellwig , Keith Busch Cc: Jake Edge , Jonathan Corbet , Jason Gunthorpe , Zhu Yanjun , Robin Murphy , Joerg Roedel , Will Deacon , Sagi Grimberg , Bjorn Helgaas , Logan Gunthorpe , Yishai Hadas , Shameer Kolothum , Kevin Tian , Alex Williamson , =?UTF-8?q?J=C3=A9r=C3=B4me=20Glisse?= , Andrew Morton , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-rdma@vger.kernel.org, iommu@lists.linux.dev, linux-nvme@lists.infradead.org, linux-pci@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org, Niklas Schnelle , Chuck Lever , Luis Chamberlain , Matthew Wilcox , Dan Williams , Kanchan Joshi , Chaitanya Kulkarni , Nitesh Shetty , Leon Romanovsky Subject: [PATCH v10 23/24] nvme-pci: convert to blk_rq_dma_map Date: Mon, 28 Apr 2025 12:22:29 +0300 Message-ID: <007e00134d49160d5edab94a72c35b7b91429b09.1745831017.git.leon@kernel.org> X-Mailer: git-send-email 2.49.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Christoph Hellwig Use the blk_rq_dma_map API to DMA map requests instead of scatterlists. Signed-off-by: Christoph Hellwig Tested-by: Jens Axboe [ Leon: squashed optimization patch from Kanchan ] Signed-off-by: Kanchan Joshi Signed-off-by: Nitesh Shetty [ Leon: rewrote original patch due to rebases and addition of metadata supp= ort ] Signed-off-by: Leon Romanovsky --- drivers/nvme/host/pci.c | 593 ++++++++++++++++++++++------------------ 1 file changed, 329 insertions(+), 264 deletions(-) diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index 270fb5146b5b..eb835425b496 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -7,7 +7,7 @@ #include #include #include -#include +#include #include #include #include @@ -26,7 +26,6 @@ #include #include #include -#include =20 #include "trace.h" #include "nvme.h" @@ -144,9 +143,6 @@ struct nvme_dev { bool hmb; struct sg_table *hmb_sgt; =20 - mempool_t *iod_mempool; - mempool_t *iod_meta_mempool; - /* shadow doorbell buffer support: */ __le32 *dbbuf_dbs; dma_addr_t dbbuf_dbs_dma_addr; @@ -222,6 +218,7 @@ struct nvme_queue { =20 enum { IOD_LARGE_DESCRIPTORS =3D 1 << 0, /* uses the full page sized descriptor = pool */ + IOD_SINGLE_SEGMENT =3D 1 << 1, /* single segment dma mapping */ }; =20 /* @@ -233,11 +230,11 @@ struct nvme_iod { bool aborted; u8 nr_descriptors; /* # of PRP/SGL descriptors */ u8 flags; - unsigned int dma_len; /* length of single DMA segment mapping */ - dma_addr_t first_dma; + unsigned int total_len; /* length of the entire transfer */ + unsigned int total_meta_len; /* length of the entire metadata transfer */ dma_addr_t meta_dma; - struct sg_table sgt; - struct sg_table meta_sgt; + struct dma_iova_state dma_state; + struct dma_iova_state dma_meta_state; void *meta_list; void *descriptors[NVME_MAX_NR_DESCRIPTORS]; }; @@ -546,9 +543,14 @@ static void nvme_free_descriptors(struct nvme_dev *dev= , struct request *req) { const int last_prp =3D NVME_CTRL_PAGE_SIZE / sizeof(__le64) - 1; struct nvme_iod *iod =3D blk_mq_rq_to_pdu(req); - dma_addr_t dma_addr =3D iod->first_dma; + dma_addr_t dma_addr; int i; =20 + if (iod->cmd.common.flags & NVME_CMD_SGL_METABUF) + dma_addr =3D le64_to_cpu(iod->cmd.common.dptr.sgl.addr); + else + dma_addr =3D le64_to_cpu(iod->cmd.common.dptr.prp2); + if (iod->nr_descriptors =3D=3D 1) { dma_pool_free(nvme_dma_pool(dev, iod), iod->descriptors[0], dma_addr); @@ -564,67 +566,182 @@ static void nvme_free_descriptors(struct nvme_dev *d= ev, struct request *req) } } =20 +static void nvme_free_prps(struct nvme_dev *dev, struct request *req) +{ + struct nvme_iod *iod =3D blk_mq_rq_to_pdu(req); + enum dma_data_direction dir =3D rq_dma_dir(req); + int length =3D iod->total_len; + dma_addr_t dma_addr; + int i, desc; + __le64 *prp_list; + u32 dma_len; + + dma_addr =3D le64_to_cpu(iod->cmd.common.dptr.prp1); + dma_len =3D min_t(u32, length, + NVME_CTRL_PAGE_SIZE - (dma_addr & (NVME_CTRL_PAGE_SIZE - 1))); + length -=3D dma_len; + if (!length) { + dma_unmap_page(dev->dev, dma_addr, dma_len, dir); + return; + } + + if (length <=3D NVME_CTRL_PAGE_SIZE) { + dma_unmap_page(dev->dev, dma_addr, dma_len, dir); + dma_addr =3D le64_to_cpu(iod->cmd.common.dptr.prp2); + dma_unmap_page(dev->dev, dma_addr, length, dir); + return; + } + + i =3D 0; + desc =3D 0; + prp_list =3D iod->descriptors[desc]; + do { + dma_unmap_page(dev->dev, dma_addr, dma_len, dir); + if (i =3D=3D NVME_CTRL_PAGE_SIZE >> 3) { + prp_list =3D iod->descriptors[++desc]; + i =3D 0; + } + + dma_addr =3D le64_to_cpu(prp_list[i++]); + dma_len =3D min(length, NVME_CTRL_PAGE_SIZE); + length -=3D dma_len; + } while (length); +} + + +static void nvme_free_sgls(struct nvme_dev *dev, struct request *req) +{ + struct nvme_iod *iod =3D blk_mq_rq_to_pdu(req); + dma_addr_t sqe_dma_addr =3D le64_to_cpu(iod->cmd.common.dptr.sgl.addr); + unsigned int sqe_dma_len =3D le32_to_cpu(iod->cmd.common.dptr.sgl.length); + struct nvme_sgl_desc *sg_list =3D iod->descriptors[0]; + enum dma_data_direction dir =3D rq_dma_dir(req); + + if (iod->nr_descriptors) { + unsigned int nr_entries =3D sqe_dma_len / sizeof(*sg_list), i; + + for (i =3D 0; i < nr_entries; i++) + dma_unmap_page(dev->dev, le64_to_cpu(sg_list[i].addr), + le32_to_cpu(sg_list[i].length), dir); + } else { + dma_unmap_page(dev->dev, sqe_dma_addr, sqe_dma_len, dir); + } +} + static void nvme_unmap_data(struct nvme_dev *dev, struct request *req) { struct nvme_iod *iod =3D blk_mq_rq_to_pdu(req); + dma_addr_t dma_addr; =20 - if (iod->dma_len) { - dma_unmap_page(dev->dev, iod->first_dma, iod->dma_len, - rq_dma_dir(req)); + if (iod->flags & IOD_SINGLE_SEGMENT) { + if (iod->cmd.common.flags & + (NVME_CMD_SGL_METABUF | NVME_CMD_SGL_METASEG)) + dma_addr =3D le64_to_cpu(iod->cmd.common.dptr.sgl.addr); + else + dma_addr =3D le64_to_cpu(iod->cmd.common.dptr.prp1); + dma_unmap_page(dev->dev, dma_addr, iod->total_len, + rq_dma_dir(req)); return; } =20 - WARN_ON_ONCE(!iod->sgt.nents); + if (!blk_rq_dma_unmap(req, dev->dev, &iod->dma_state, iod->total_len)) { + if (iod->cmd.common.flags & NVME_CMD_SGL_METABUF) + nvme_free_sgls(dev, req); + else + nvme_free_prps(dev, req); + } =20 - dma_unmap_sgtable(dev->dev, &iod->sgt, rq_dma_dir(req), 0); - nvme_free_descriptors(dev, req); - mempool_free(iod->sgt.sgl, dev->iod_mempool); + if (iod->nr_descriptors) + nvme_free_descriptors(dev, req); } =20 -static void nvme_print_sgl(struct scatterlist *sgl, int nents) +static bool nvme_try_setup_prp_simple(struct nvme_dev *dev, struct request= *req, + struct nvme_rw_command *cmnd, + struct blk_dma_iter *iter) { - int i; - struct scatterlist *sg; + struct nvme_iod *iod =3D blk_mq_rq_to_pdu(req); + struct bio_vec bv =3D req_bvec(req); + unsigned int first_prp_len; + + if (IS_ENABLED(CONFIG_PCI_P2PDMA) && (req->cmd_flags & REQ_P2PDMA)) + return false; + + if ((bv.bv_offset & (NVME_CTRL_PAGE_SIZE - 1)) + bv.bv_len > + NVME_CTRL_PAGE_SIZE * 2) + return false; =20 - for_each_sg(sgl, sg, nents, i) { - dma_addr_t phys =3D sg_phys(sg); - pr_warn("sg[%d] phys_addr:%pad offset:%d length:%d " - "dma_address:%pad dma_length:%d\n", - i, &phys, sg->offset, sg->length, &sg_dma_address(sg), - sg_dma_len(sg)); + iter->addr =3D dma_map_bvec(dev->dev, &bv, rq_dma_dir(req), 0); + if (dma_mapping_error(dev->dev, iter->addr)) { + iter->status =3D BLK_STS_RESOURCE; + return true; } + iod->total_len =3D bv.bv_len; + cmnd->dptr.prp1 =3D cpu_to_le64(iter->addr); + + first_prp_len =3D NVME_CTRL_PAGE_SIZE - + (bv.bv_offset & (NVME_CTRL_PAGE_SIZE - 1)); + if (bv.bv_len > first_prp_len) + cmnd->dptr.prp2 =3D cpu_to_le64(iter->addr + first_prp_len); + else + cmnd->dptr.prp2 =3D 0; + + iter->status =3D BLK_STS_OK; + iod->flags |=3D IOD_SINGLE_SEGMENT; + return true; } =20 static blk_status_t nvme_pci_setup_prps(struct nvme_dev *dev, - struct request *req, struct nvme_rw_command *cmnd) + struct request *req) { struct nvme_iod *iod =3D blk_mq_rq_to_pdu(req); - int length =3D blk_rq_payload_bytes(req); - struct scatterlist *sg =3D iod->sgt.sgl; - int dma_len =3D sg_dma_len(sg); - u64 dma_addr =3D sg_dma_address(sg); - int offset =3D dma_addr & (NVME_CTRL_PAGE_SIZE - 1); + struct nvme_rw_command *cmnd =3D &iod->cmd.rw; + unsigned int length =3D blk_rq_payload_bytes(req); + struct blk_dma_iter iter; + dma_addr_t prp1_dma, prp2_dma =3D 0; + unsigned int prp_len, i; __le64 *prp_list; - dma_addr_t prp_dma; - int i; + unsigned int nr_segments =3D blk_rq_nr_phys_segments(req); =20 - length -=3D (NVME_CTRL_PAGE_SIZE - offset); - if (length <=3D 0) { - iod->first_dma =3D 0; - goto done; + if (nr_segments =3D=3D 1) { + if (nvme_try_setup_prp_simple(dev, req, cmnd, &iter)) + return iter.status; } =20 - dma_len -=3D (NVME_CTRL_PAGE_SIZE - offset); - if (dma_len) { - dma_addr +=3D (NVME_CTRL_PAGE_SIZE - offset); - } else { - sg =3D sg_next(sg); - dma_addr =3D sg_dma_address(sg); - dma_len =3D sg_dma_len(sg); + if (!blk_rq_dma_map_iter_start(req, dev->dev, &iod->dma_state, &iter)) + return iter.status; + + /* + * PRP1 always points to the start of the DMA transfers. + * + * This is the only PRP (except for the list entries) that could be + * non-aligned. + */ + prp1_dma =3D iter.addr; + prp_len =3D min(length, NVME_CTRL_PAGE_SIZE - + (iter.addr & (NVME_CTRL_PAGE_SIZE - 1))); + iod->total_len +=3D prp_len; + iter.addr +=3D prp_len; + iter.len -=3D prp_len; + length -=3D prp_len; + if (!length) + goto done; + + if (!iter.len) { + if (!blk_rq_dma_map_iter_next(req, dev->dev, &iod->dma_state, + &iter)) { + if (WARN_ON_ONCE(!iter.status)) + goto bad_sgl; + goto done; + } } =20 + /* + * PRP2 is usually a list, but can point to data if all data to be + * transferred fits into PRP1 + PRP2: + */ if (length <=3D NVME_CTRL_PAGE_SIZE) { - iod->first_dma =3D dma_addr; + prp2_dma =3D iter.addr; + iod->total_len +=3D length; goto done; } =20 @@ -633,58 +750,83 @@ static blk_status_t nvme_pci_setup_prps(struct nvme_d= ev *dev, iod->flags |=3D IOD_LARGE_DESCRIPTORS; =20 prp_list =3D dma_pool_alloc(nvme_dma_pool(dev, iod), GFP_ATOMIC, - &prp_dma); - if (!prp_list) - return BLK_STS_RESOURCE; + &prp2_dma); + if (!prp_list) { + iter.status =3D BLK_STS_RESOURCE; + goto done; + } iod->descriptors[iod->nr_descriptors++] =3D prp_list; - iod->first_dma =3D prp_dma; + i =3D 0; for (;;) { + prp_list[i++] =3D cpu_to_le64(iter.addr); + prp_len =3D min(length, NVME_CTRL_PAGE_SIZE); + if (WARN_ON_ONCE(iter.len < prp_len)) + goto bad_sgl; + + iod->total_len +=3D prp_len; + iter.addr +=3D prp_len; + iter.len -=3D prp_len; + length -=3D prp_len; + if (!length) + break; + + if (iter.len =3D=3D 0) { + if (!blk_rq_dma_map_iter_next(req, dev->dev, + &iod->dma_state, &iter)) { + if (WARN_ON_ONCE(!iter.status)) + goto bad_sgl; + goto done; + } + } + + /* + * If we've filled the entire descriptor, allocate a new that is + * pointed to be the last entry in the previous PRP list. To + * accommodate for that move the last actual entry to the new + * descriptor. + */ if (i =3D=3D NVME_CTRL_PAGE_SIZE >> 3) { __le64 *old_prp_list =3D prp_list; + dma_addr_t prp_list_dma; =20 prp_list =3D dma_pool_alloc(dev->prp_page_pool, - GFP_ATOMIC, &prp_dma); - if (!prp_list) - goto free_prps; + GFP_ATOMIC, &prp_list_dma); + if (!prp_list) { + iter.status =3D BLK_STS_RESOURCE; + goto done; + } iod->descriptors[iod->nr_descriptors++] =3D prp_list; + prp_list[0] =3D old_prp_list[i - 1]; - old_prp_list[i - 1] =3D cpu_to_le64(prp_dma); + old_prp_list[i - 1] =3D cpu_to_le64(prp_list_dma); i =3D 1; } - prp_list[i++] =3D cpu_to_le64(dma_addr); - dma_len -=3D NVME_CTRL_PAGE_SIZE; - dma_addr +=3D NVME_CTRL_PAGE_SIZE; - length -=3D NVME_CTRL_PAGE_SIZE; - if (length <=3D 0) - break; - if (dma_len > 0) - continue; - if (unlikely(dma_len < 0)) - goto bad_sgl; - sg =3D sg_next(sg); - dma_addr =3D sg_dma_address(sg); - dma_len =3D sg_dma_len(sg); } + done: - cmnd->dptr.prp1 =3D cpu_to_le64(sg_dma_address(iod->sgt.sgl)); - cmnd->dptr.prp2 =3D cpu_to_le64(iod->first_dma); - return BLK_STS_OK; -free_prps: - nvme_free_descriptors(dev, req); - return BLK_STS_RESOURCE; + /* + * nvme_unmap_data uses the DPT field in the SQE to tear down the + * mapping, so initialize it even for failures. + */ + cmnd->dptr.prp1 =3D cpu_to_le64(prp1_dma); + cmnd->dptr.prp2 =3D cpu_to_le64(prp2_dma); + if (unlikely(iter.status)) + nvme_unmap_data(dev, req); + return iter.status; + bad_sgl: - WARN(DO_ONCE(nvme_print_sgl, iod->sgt.sgl, iod->sgt.nents), - "Invalid SGL for payload:%d nents:%d\n", - blk_rq_payload_bytes(req), iod->sgt.nents); + dev_err_once(dev->dev, + "Incorrectly formed request for payload:%d nents:%d\n", + blk_rq_payload_bytes(req), blk_rq_nr_phys_segments(req)); return BLK_STS_IOERR; } =20 static void nvme_pci_sgl_set_data(struct nvme_sgl_desc *sge, - struct scatterlist *sg) + struct blk_dma_iter *iter) { - sge->addr =3D cpu_to_le64(sg_dma_address(sg)); - sge->length =3D cpu_to_le32(sg_dma_len(sg)); + sge->addr =3D cpu_to_le64(iter->addr); + sge->length =3D cpu_to_le32(iter->len); sge->type =3D NVME_SGL_FMT_DATA_DESC << 4; } =20 @@ -696,21 +838,54 @@ static void nvme_pci_sgl_set_seg(struct nvme_sgl_desc= *sge, sge->type =3D NVME_SGL_FMT_LAST_SEG_DESC << 4; } =20 +static bool nvme_try_setup_sgl_simple(struct nvme_dev *dev, struct request= *req, + struct nvme_rw_command *cmnd, + struct blk_dma_iter *iter) +{ + struct nvme_iod *iod =3D blk_mq_rq_to_pdu(req); + struct bio_vec bv =3D req_bvec(req); + + if (IS_ENABLED(CONFIG_PCI_P2PDMA) && (req->cmd_flags & REQ_P2PDMA)) + return false; + + iter->addr =3D dma_map_bvec(dev->dev, &bv, rq_dma_dir(req), 0); + if (dma_mapping_error(dev->dev, iter->addr)) { + iter->status =3D BLK_STS_RESOURCE; + return true; + } + iod->total_len =3D bv.bv_len; + iter->len =3D bv.bv_len; + nvme_pci_sgl_set_data(&cmnd->dptr.sgl, iter); + iter->status =3D BLK_STS_OK; + iod->flags |=3D IOD_SINGLE_SEGMENT; + return true; +} + static blk_status_t nvme_pci_setup_sgls(struct nvme_dev *dev, - struct request *req, struct nvme_rw_command *cmd) + struct request *req) { struct nvme_iod *iod =3D blk_mq_rq_to_pdu(req); + struct nvme_rw_command *cmd =3D &iod->cmd.rw; + unsigned int entries =3D blk_rq_nr_phys_segments(req); struct nvme_sgl_desc *sg_list; - struct scatterlist *sg =3D iod->sgt.sgl; - unsigned int entries =3D iod->sgt.nents; + struct blk_dma_iter iter; dma_addr_t sgl_dma; - int i =3D 0; + unsigned int mapped =3D 0; =20 /* setting the transfer type as SGL */ cmd->flags =3D NVME_CMD_SGL_METABUF; =20 if (entries =3D=3D 1) { - nvme_pci_sgl_set_data(&cmd->dptr.sgl, sg); + if (nvme_try_setup_sgl_simple(dev, req, cmd, &iter)) + return iter.status; + } + + if (!blk_rq_dma_map_iter_start(req, dev->dev, &iod->dma_state, &iter)) + return iter.status; + + if (entries =3D=3D 1 || blk_rq_dma_map_coalesce(&iod->dma_state)) { + nvme_pci_sgl_set_data(&cmd->dptr.sgl, &iter); + iod->total_len +=3D iter.len; return BLK_STS_OK; } =20 @@ -721,168 +896,109 @@ static blk_status_t nvme_pci_setup_sgls(struct nvme= _dev *dev, if (!sg_list) return BLK_STS_RESOURCE; iod->descriptors[iod->nr_descriptors++] =3D sg_list; - iod->first_dma =3D sgl_dma; =20 - nvme_pci_sgl_set_seg(&cmd->dptr.sgl, sgl_dma, entries); do { - nvme_pci_sgl_set_data(&sg_list[i++], sg); - sg =3D sg_next(sg); - } while (--entries > 0); + if (WARN_ON_ONCE(mapped =3D=3D entries)) { + iter.status =3D BLK_STS_IOERR; + break; + } + nvme_pci_sgl_set_data(&sg_list[mapped++], &iter); + iod->total_len +=3D iter.len; + } while (blk_rq_dma_map_iter_next(req, dev->dev, &iod->dma_state, + &iter)); =20 - return BLK_STS_OK; + nvme_pci_sgl_set_seg(&cmd->dptr.sgl, sgl_dma, mapped); + if (unlikely(iter.status)) + nvme_free_sgls(dev, req); + return iter.status; } =20 -static blk_status_t nvme_setup_prp_simple(struct nvme_dev *dev, - struct request *req, struct nvme_rw_command *cmnd, - struct bio_vec *bv) +static blk_status_t nvme_map_data(struct nvme_dev *dev, struct request *re= q) { - struct nvme_iod *iod =3D blk_mq_rq_to_pdu(req); - unsigned int offset =3D bv->bv_offset & (NVME_CTRL_PAGE_SIZE - 1); - unsigned int first_prp_len =3D NVME_CTRL_PAGE_SIZE - offset; - - iod->first_dma =3D dma_map_bvec(dev->dev, bv, rq_dma_dir(req), 0); - if (dma_mapping_error(dev->dev, iod->first_dma)) - return BLK_STS_RESOURCE; - iod->dma_len =3D bv->bv_len; - - cmnd->dptr.prp1 =3D cpu_to_le64(iod->first_dma); - if (bv->bv_len > first_prp_len) - cmnd->dptr.prp2 =3D cpu_to_le64(iod->first_dma + first_prp_len); - else - cmnd->dptr.prp2 =3D 0; - return BLK_STS_OK; + if (nvme_pci_use_sgls(dev, req, blk_rq_nr_phys_segments(req))) + return nvme_pci_setup_sgls(dev, req); + return nvme_pci_setup_prps(dev, req); } =20 -static blk_status_t nvme_setup_sgl_simple(struct nvme_dev *dev, - struct request *req, struct nvme_rw_command *cmnd, - struct bio_vec *bv) +static void nvme_unmap_metadata(struct nvme_dev *dev, struct request *req) { + unsigned int entries =3D req->nr_integrity_segments; struct nvme_iod *iod =3D blk_mq_rq_to_pdu(req); + struct nvme_sgl_desc *sg_list =3D iod->meta_list; + enum dma_data_direction dir =3D rq_dma_dir(req); =20 - iod->first_dma =3D dma_map_bvec(dev->dev, bv, rq_dma_dir(req), 0); - if (dma_mapping_error(dev->dev, iod->first_dma)) - return BLK_STS_RESOURCE; - iod->dma_len =3D bv->bv_len; - - cmnd->flags =3D NVME_CMD_SGL_METABUF; - cmnd->dptr.sgl.addr =3D cpu_to_le64(iod->first_dma); - cmnd->dptr.sgl.length =3D cpu_to_le32(iod->dma_len); - cmnd->dptr.sgl.type =3D NVME_SGL_FMT_DATA_DESC << 4; - return BLK_STS_OK; -} - -static blk_status_t nvme_map_data(struct nvme_dev *dev, struct request *re= q, - struct nvme_command *cmnd) -{ - struct nvme_iod *iod =3D blk_mq_rq_to_pdu(req); - blk_status_t ret =3D BLK_STS_RESOURCE; - int rc; - - if (blk_rq_nr_phys_segments(req) =3D=3D 1) { - struct nvme_queue *nvmeq =3D req->mq_hctx->driver_data; - struct bio_vec bv =3D req_bvec(req); - - if (!is_pci_p2pdma_page(bv.bv_page)) { - if (!nvme_pci_metadata_use_sgls(dev, req) && - (bv.bv_offset & (NVME_CTRL_PAGE_SIZE - 1)) + - bv.bv_len <=3D NVME_CTRL_PAGE_SIZE * 2) - return nvme_setup_prp_simple(dev, req, - &cmnd->rw, &bv); - - if (nvmeq->qid && sgl_threshold && - nvme_ctrl_sgl_supported(&dev->ctrl)) - return nvme_setup_sgl_simple(dev, req, - &cmnd->rw, &bv); - } + /* + * If the NVME_CMD_SGL_METASEG flag is not set and we're using the + * non-SGL linear meta buffer we know that we have a single input + * segment as well. + * + * Note that it would be nice to always use the linear buffer when + * using IOVA mappings and kernel buffers to avoid the SGL + * indirection, but that's left for a future optimization. + */ + if (!(iod->cmd.common.flags & NVME_CMD_SGL_METASEG)) { + dma_unmap_page(dev->dev, le64_to_cpu(iod->cmd.common.dptr.prp1), + iod->total_len, rq_dma_dir(req)); + return; } =20 - iod->dma_len =3D 0; - iod->sgt.sgl =3D mempool_alloc(dev->iod_mempool, GFP_ATOMIC); - if (!iod->sgt.sgl) - return BLK_STS_RESOURCE; - sg_init_table(iod->sgt.sgl, blk_rq_nr_phys_segments(req)); - iod->sgt.orig_nents =3D blk_rq_map_sg(req, iod->sgt.sgl); - if (!iod->sgt.orig_nents) - goto out_free_sg; + if (!blk_rq_dma_unmap(req, dev->dev, &iod->dma_meta_state, + iod->total_meta_len)) { + unsigned int i; =20 - rc =3D dma_map_sgtable(dev->dev, &iod->sgt, rq_dma_dir(req), - DMA_ATTR_NO_WARN); - if (rc) { - if (rc =3D=3D -EREMOTEIO) - ret =3D BLK_STS_TARGET; - goto out_free_sg; + for (i =3D 0; i < entries; i++) + dma_unmap_page(dev->dev, le64_to_cpu(sg_list[i].addr), + le32_to_cpu(sg_list[i].length), dir); } =20 - if (nvme_pci_use_sgls(dev, req, iod->sgt.nents)) - ret =3D nvme_pci_setup_sgls(dev, req, &cmnd->rw); - else - ret =3D nvme_pci_setup_prps(dev, req, &cmnd->rw); - if (ret !=3D BLK_STS_OK) - goto out_unmap_sg; - return BLK_STS_OK; - -out_unmap_sg: - dma_unmap_sgtable(dev->dev, &iod->sgt, rq_dma_dir(req), 0); -out_free_sg: - mempool_free(iod->sgt.sgl, dev->iod_mempool); - return ret; + dma_pool_free(dev->prp_small_pool, iod->meta_list, iod->meta_dma); } =20 static blk_status_t nvme_pci_setup_meta_sgls(struct nvme_dev *dev, struct request *req) { + unsigned int entries =3D req->nr_integrity_segments; struct nvme_iod *iod =3D blk_mq_rq_to_pdu(req); - struct nvme_rw_command *cmnd =3D &iod->cmd.rw; + struct nvme_rw_command *cmd =3D &iod->cmd.rw; struct nvme_sgl_desc *sg_list; - struct scatterlist *sgl, *sg; - unsigned int entries; + struct blk_dma_iter iter; + unsigned int mapped =3D 0; dma_addr_t sgl_dma; - int rc, i; - - iod->meta_sgt.sgl =3D mempool_alloc(dev->iod_meta_mempool, GFP_ATOMIC); - if (!iod->meta_sgt.sgl) - return BLK_STS_RESOURCE; =20 - sg_init_table(iod->meta_sgt.sgl, req->nr_integrity_segments); - iod->meta_sgt.orig_nents =3D blk_rq_map_integrity_sg(req, - iod->meta_sgt.sgl); - if (!iod->meta_sgt.orig_nents) - goto out_free_sg; + cmd->flags =3D NVME_CMD_SGL_METASEG; =20 - rc =3D dma_map_sgtable(dev->dev, &iod->meta_sgt, rq_dma_dir(req), - DMA_ATTR_NO_WARN); - if (rc) - goto out_free_sg; + if (!blk_rq_dma_map_iter_start(req, dev->dev, &iod->dma_meta_state, + &iter)) + return iter.status; =20 sg_list =3D dma_pool_alloc(dev->prp_small_pool, GFP_ATOMIC, &sgl_dma); if (!sg_list) - goto out_unmap_sg; + return BLK_STS_RESOURCE; =20 - entries =3D iod->meta_sgt.nents; iod->meta_list =3D sg_list; iod->meta_dma =3D sgl_dma; + cmd->metadata =3D cpu_to_le64(sgl_dma); =20 - cmnd->flags =3D NVME_CMD_SGL_METASEG; - cmnd->metadata =3D cpu_to_le64(sgl_dma); - - sgl =3D iod->meta_sgt.sgl; - if (entries =3D=3D 1) { - nvme_pci_sgl_set_data(sg_list, sgl); + if (entries =3D=3D 1 || blk_rq_dma_map_coalesce(&iod->dma_meta_state)) { + nvme_pci_sgl_set_data(sg_list, &iter); + iod->total_meta_len +=3D iter.len; return BLK_STS_OK; } =20 - sgl_dma +=3D sizeof(*sg_list); - nvme_pci_sgl_set_seg(sg_list, sgl_dma, entries); - for_each_sg(sgl, sg, entries, i) - nvme_pci_sgl_set_data(&sg_list[i + 1], sg); - - return BLK_STS_OK; + do { + if (WARN_ON_ONCE(mapped =3D=3D entries)) { + iter.status =3D BLK_STS_IOERR; + break; + } + nvme_pci_sgl_set_data(&sg_list[mapped++], &iter); + iod->total_len +=3D iter.len; + } while (blk_rq_dma_map_iter_next(req, dev->dev, &iod->dma_meta_state, + &iter)); =20 -out_unmap_sg: - dma_unmap_sgtable(dev->dev, &iod->meta_sgt, rq_dma_dir(req), 0); -out_free_sg: - mempool_free(iod->meta_sgt.sgl, dev->iod_meta_mempool); - return BLK_STS_RESOURCE; + nvme_pci_sgl_set_seg(sg_list, sgl_dma, mapped); + if (unlikely(iter.status)) + nvme_unmap_metadata(dev, req); + return iter.status; } =20 static blk_status_t nvme_pci_setup_meta_mptr(struct nvme_dev *dev, @@ -914,15 +1030,15 @@ static blk_status_t nvme_prep_rq(struct nvme_dev *de= v, struct request *req) iod->aborted =3D false; iod->nr_descriptors =3D 0; iod->flags =3D 0; - iod->sgt.nents =3D 0; - iod->meta_sgt.nents =3D 0; + iod->total_len =3D 0; + iod->total_meta_len =3D 0; =20 ret =3D nvme_setup_cmd(req->q->queuedata, req); if (ret) return ret; =20 if (blk_rq_nr_phys_segments(req)) { - ret =3D nvme_map_data(dev, req, &iod->cmd); + ret =3D nvme_map_data(dev, req); if (ret) goto out_free_cmd; } @@ -1026,23 +1142,6 @@ static void nvme_queue_rqs(struct rq_list *rqlist) *rqlist =3D requeue_list; } =20 -static __always_inline void nvme_unmap_metadata(struct nvme_dev *dev, - struct request *req) -{ - struct nvme_iod *iod =3D blk_mq_rq_to_pdu(req); - - if (!iod->meta_sgt.nents) { - dma_unmap_page(dev->dev, iod->meta_dma, - rq_integrity_vec(req).bv_len, - rq_dma_dir(req)); - return; - } - - dma_pool_free(dev->prp_small_pool, iod->meta_list, iod->meta_dma); - dma_unmap_sgtable(dev->dev, &iod->meta_sgt, rq_dma_dir(req), 0); - mempool_free(iod->meta_sgt.sgl, dev->iod_meta_mempool); -} - static __always_inline void nvme_pci_unmap_rq(struct request *req) { struct nvme_queue *nvmeq =3D req->mq_hctx->driver_data; @@ -2860,31 +2959,6 @@ static void nvme_release_prp_pools(struct nvme_dev *= dev) dma_pool_destroy(dev->prp_small_pool); } =20 -static int nvme_pci_alloc_iod_mempool(struct nvme_dev *dev) -{ - size_t meta_size =3D sizeof(struct scatterlist) * (NVME_MAX_META_SEGS + 1= ); - size_t alloc_size =3D sizeof(struct scatterlist) * NVME_MAX_SEGS; - - dev->iod_mempool =3D mempool_create_node(1, - mempool_kmalloc, mempool_kfree, - (void *)alloc_size, GFP_KERNEL, - dev_to_node(dev->dev)); - if (!dev->iod_mempool) - return -ENOMEM; - - dev->iod_meta_mempool =3D mempool_create_node(1, - mempool_kmalloc, mempool_kfree, - (void *)meta_size, GFP_KERNEL, - dev_to_node(dev->dev)); - if (!dev->iod_meta_mempool) - goto free; - - return 0; -free: - mempool_destroy(dev->iod_mempool); - return -ENOMEM; -} - static void nvme_free_tagset(struct nvme_dev *dev) { if (dev->tagset.tags) @@ -3253,15 +3327,11 @@ static int nvme_probe(struct pci_dev *pdev, const s= truct pci_device_id *id) if (result) goto out_dev_unmap; =20 - result =3D nvme_pci_alloc_iod_mempool(dev); - if (result) - goto out_release_prp_pools; - dev_info(dev->ctrl.device, "pci function %s\n", dev_name(&pdev->dev)); =20 result =3D nvme_pci_enable(dev); if (result) - goto out_release_iod_mempool; + goto out_release_prp_pools; =20 result =3D nvme_alloc_admin_tag_set(&dev->ctrl, &dev->admin_tagset, &nvme_mq_admin_ops, sizeof(struct nvme_iod)); @@ -3328,9 +3398,6 @@ static int nvme_probe(struct pci_dev *pdev, const str= uct pci_device_id *id) nvme_dev_remove_admin(dev); nvme_dbbuf_dma_free(dev); nvme_free_queues(dev, 0); -out_release_iod_mempool: - mempool_destroy(dev->iod_mempool); - mempool_destroy(dev->iod_meta_mempool); out_release_prp_pools: nvme_release_prp_pools(dev); out_dev_unmap: @@ -3395,8 +3462,6 @@ static void nvme_remove(struct pci_dev *pdev) nvme_dev_remove_admin(dev); nvme_dbbuf_dma_free(dev); nvme_free_queues(dev, 0); - mempool_destroy(dev->iod_mempool); - mempool_destroy(dev->iod_meta_mempool); nvme_release_prp_pools(dev); nvme_dev_unmap(dev); nvme_uninit_ctrl(&dev->ctrl); --=20 2.49.0 From nobody Tue Dec 16 14:20:07 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5E460263F5B; Mon, 28 Apr 2025 09:24:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745832261; cv=none; b=SbuMfK4cuBLLMFvJXVUEdxf/SS3ZR3KTIVaKlyIiWwSkxavAomECMr0nhKQZN1LrymyX1wsfhjq4OFyj3Z/dV4l5VvQH7OrxH/uAogznHZdI2kv3KTLPZEj1PWgmNdbm+6CoKYDPwWkJSytj1yFrHsuYjG/L7LGIB7IAW3nRb7E= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745832261; c=relaxed/simple; bh=JltzEs94lpehyHx6nl6wHRaPQKIRgU/HyWicrIFjUro=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=i8mbF1sFmXl/wShn023sj+q8DSPUPUqTZFh2MVY76MzhYChzQFbhXXuHHbW9mOI4krua6MxY2p5sHGOxjRm+c1RIloccEAHExXMxgEz/i9D3zJVuBq0bgRkzXoECHhj99A5NrFlH1nWiRtX5OJjhmh6gwdls6GqekVyivY6I2KI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=DyqXQ6VB; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="DyqXQ6VB" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 0C09AC4CEE4; Mon, 28 Apr 2025 09:24:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1745832260; bh=JltzEs94lpehyHx6nl6wHRaPQKIRgU/HyWicrIFjUro=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=DyqXQ6VBPGUV60WqF+uTbZzXAsUAIJ9C1BCb2KV9m8F42ozYCzXInlnKQFxIS2b36 s+SctZer4F7Q5NkzQQld7WJZo9qsmkj9svarVHnHcf2ojzaBTAeBo4/HXSAAYmAED9 0xAI51RXFcaNAViqGFTvgw71I5NRyjdQoBDcESxnGQBEZOfbe7wMnLan2Z1+J+qRj+ 8OEN5Dl/+rgO5hpebjkaj1SMCY1/3eRJ/TzmFUnFCx9NWXXoE71dQT6bHrYlzaqCDj cQfDk5LtjtwfeRrBMl5iLWmlcl+SqiMKCEXYnYeUEY8YsO01MnQzVcNjFdveNAbkls vBmDT0QLSZCQA== From: Leon Romanovsky To: Marek Szyprowski , Jens Axboe , Christoph Hellwig , Keith Busch Cc: Leon Romanovsky , Jake Edge , Jonathan Corbet , Jason Gunthorpe , Zhu Yanjun , Robin Murphy , Joerg Roedel , Will Deacon , Sagi Grimberg , Bjorn Helgaas , Logan Gunthorpe , Yishai Hadas , Shameer Kolothum , Kevin Tian , Alex Williamson , =?UTF-8?q?J=C3=A9r=C3=B4me=20Glisse?= , Andrew Morton , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-rdma@vger.kernel.org, iommu@lists.linux.dev, linux-nvme@lists.infradead.org, linux-pci@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org, Niklas Schnelle , Chuck Lever , Luis Chamberlain , Matthew Wilcox , Dan Williams , Kanchan Joshi , Chaitanya Kulkarni Subject: [PATCH v10 24/24] nvme-pci: store aborted state in flags variable Date: Mon, 28 Apr 2025 12:22:30 +0300 Message-ID: X-Mailer: git-send-email 2.49.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Leon Romanovsky Instead of keeping dedicated "bool aborted" variable, let's reuse newly introduced flags variable and save space. Signed-off-by: Leon Romanovsky --- drivers/nvme/host/pci.c | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index eb835425b496..9f3e2d8cbd04 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -219,6 +219,7 @@ struct nvme_queue { enum { IOD_LARGE_DESCRIPTORS =3D 1 << 0, /* uses the full page sized descriptor = pool */ IOD_SINGLE_SEGMENT =3D 1 << 1, /* single segment dma mapping */ + IOD_ABORTED =3D 1 << 2, /* abort timed out commands */ }; =20 /* @@ -227,7 +228,6 @@ enum { struct nvme_iod { struct nvme_request req; struct nvme_command cmd; - bool aborted; u8 nr_descriptors; /* # of PRP/SGL descriptors */ u8 flags; unsigned int total_len; /* length of the entire transfer */ @@ -1027,7 +1027,6 @@ static blk_status_t nvme_prep_rq(struct nvme_dev *dev= , struct request *req) struct nvme_iod *iod =3D blk_mq_rq_to_pdu(req); blk_status_t ret; =20 - iod->aborted =3D false; iod->nr_descriptors =3D 0; iod->flags =3D 0; iod->total_len =3D 0; @@ -1576,7 +1575,7 @@ static enum blk_eh_timer_return nvme_timeout(struct r= equest *req) * returned to the driver, or if this is the admin queue. */ opcode =3D nvme_req(req)->cmd->common.opcode; - if (!nvmeq->qid || iod->aborted) { + if (!nvmeq->qid || (iod->flags & IOD_ABORTED)) { dev_warn(dev->ctrl.device, "I/O tag %d (%04x) opcode %#x (%s) QID %d timeout, reset controller\n", req->tag, nvme_cid(req), opcode, @@ -1589,7 +1588,7 @@ static enum blk_eh_timer_return nvme_timeout(struct r= equest *req) atomic_inc(&dev->ctrl.abort_limit); return BLK_EH_RESET_TIMER; } - iod->aborted =3D true; + iod->flags |=3D IOD_ABORTED; =20 cmd.abort.opcode =3D nvme_admin_abort_cmd; cmd.abort.cid =3D nvme_cid(req); --=20 2.49.0