From nobody Tue Dec 16 16:39:05 2025 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 07689537E7 for ; Tue, 2 Jul 2024 06:37:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.10 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719902257; cv=none; b=pqN3C4TM/5k9SjhyHsfixLICW7Ol1dFF2ucTrTjqnQGhXTDFWK2+b+3bdVxGkQSE64PyK5KXSq8K1tGfzBB4Ncvt73OQV90Qcav3iX/y3Jjb9/aJemWunBIdRk0jHYlWzLIqHSkhovbTmJB1soNrQ5TjnmXrvlXyuuz2ZmfOKFs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719902257; c=relaxed/simple; bh=BZSDz0KzxucLY0QHtPPPO1kqDaVfkMKtM4/CokGnRhQ=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=jPqkUBk/DQ9vj75dUJRMtDdVVkDzZmtgqHNBhYca+4LPMxrO+NZDDTz59Ou7UHVYht/WtpxrhP/kxQxmaEzS8z7MPLhnNf6lWBI0TYcV/G8Irb9cH6pS9dxjT4JXEVm0onYqgCAt0LK2SwNUnL8eEkvVR+CY7eVnEewy+JbAip0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=bsCQrd5v; arc=none smtp.client-ip=192.198.163.10 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="bsCQrd5v" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1719902255; x=1751438255; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=BZSDz0KzxucLY0QHtPPPO1kqDaVfkMKtM4/CokGnRhQ=; b=bsCQrd5v3Cci82ESh8gtmfB67x3W1ak65IJwOKY1na75K5nq02LsQ+SU 9v65AJ/m5qwUZkUiVmVFoH7OEsC7EQ2Gi5NT7LwUI6wsu03mJKdkVdOr3 mB2Prridp2P2odd4LaG2VjJdD/E0mBzy/IiNa7XUbiedsHK7Dr47ERMaj 1L2fHXDX9oIs20z1dHohhozd4L2T72E+BRvOQVhuY6ooXi9VcZLkAKLVi UfLAg7uyHDvuuAzOgps4sXoJ6hR5GV/GVC98ALPWFWBtFk7BjAbr5zYYX SRQEx5djhwVcHW/YtCb+j/IhW9Ex9g/Nee5hXJCfIi1rBxqddZ/JlvJfa g==; X-CSE-ConnectionGUID: +D8hID6JR1mNH3caXKE26A== X-CSE-MsgGUID: PN8REHSCQ7mYuHkGzCjkSA== X-IronPort-AV: E=McAfee;i="6700,10204,11120"; a="28455572" X-IronPort-AV: E=Sophos;i="6.09,178,1716274800"; d="scan'208";a="28455572" Received: from fmviesa003.fm.intel.com ([10.60.135.143]) by fmvoesa104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 01 Jul 2024 23:37:35 -0700 X-CSE-ConnectionGUID: /GQfl16ETLW9bOhKZ8salA== X-CSE-MsgGUID: IJj62i/mTGebSDJKVcQ6ww== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.09,178,1716274800"; d="scan'208";a="50137111" Received: from unknown (HELO allen-box.sh.intel.com) ([10.239.159.127]) by fmviesa003.fm.intel.com with ESMTP; 01 Jul 2024 23:37:31 -0700 From: Lu Baolu To: Jason Gunthorpe , Kevin Tian , Joerg Roedel , Will Deacon , Robin Murphy , Jean-Philippe Brucker , Nicolin Chen , Yi Liu , Jacob Pan , Joel Granados Cc: iommu@lists.linux.dev, virtualization@lists.linux-foundation.org, linux-kernel@vger.kernel.org, Lu Baolu , Jason Gunthorpe Subject: [PATCH v8 01/10] iommu: Introduce domain attachment handle Date: Tue, 2 Jul 2024 14:34:35 +0800 Message-Id: <20240702063444.105814-2-baolu.lu@linux.intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240702063444.105814-1-baolu.lu@linux.intel.com> References: <20240702063444.105814-1-baolu.lu@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Currently, when attaching a domain to a device or its PASID, domain is stored within the iommu group. It could be retrieved for use during the window between attachment and detachment. With new features introduced, there's a need to store more information than just a domain pointer. This information essentially represents the association between a domain and a device. For example, the SVA code already has a custom struct iommu_sva which represents a bond between sva domain and a PASID of a device. Looking forward, the IOMMUFD needs a place to store the iommufd_device pointer in the core, so that the device object ID could be quickly retrieved in the critical fault handling path. Introduce domain attachment handle that explicitly represents the attachment relationship between a domain and a device or its PASID. Co-developed-by: Jason Gunthorpe Signed-off-by: Jason Gunthorpe Signed-off-by: Lu Baolu Reviewed-by: Jason Gunthorpe Reviewed-by: Kevin Tian --- include/linux/iommu.h | 18 +++++++++++++++--- drivers/dma/idxd/init.c | 2 +- drivers/iommu/iommu-sva.c | 13 ++++++++----- drivers/iommu/iommu.c | 26 ++++++++++++++++---------- 4 files changed, 40 insertions(+), 19 deletions(-) diff --git a/include/linux/iommu.h b/include/linux/iommu.h index 17b3f36ad843..afc5af0069bb 100644 --- a/include/linux/iommu.h +++ b/include/linux/iommu.h @@ -989,12 +989,22 @@ struct iommu_fwspec { /* ATS is supported */ #define IOMMU_FWSPEC_PCI_RC_ATS (1 << 0) =20 +/* + * An iommu attach handle represents a relationship between an iommu domain + * and a PASID or RID of a device. It is allocated and managed by the comp= onent + * that manages the domain and is stored in the iommu group during the tim= e the + * domain is attached. + */ +struct iommu_attach_handle { + struct iommu_domain *domain; +}; + /** * struct iommu_sva - handle to a device-mm bond */ struct iommu_sva { + struct iommu_attach_handle handle; struct device *dev; - struct iommu_domain *domain; struct list_head handle_item; refcount_t users; }; @@ -1052,7 +1062,8 @@ int iommu_device_claim_dma_owner(struct device *dev, = void *owner); void iommu_device_release_dma_owner(struct device *dev); =20 int iommu_attach_device_pasid(struct iommu_domain *domain, - struct device *dev, ioasid_t pasid); + struct device *dev, ioasid_t pasid, + struct iommu_attach_handle *handle); void iommu_detach_device_pasid(struct iommu_domain *domain, struct device *dev, ioasid_t pasid); struct iommu_domain * @@ -1388,7 +1399,8 @@ static inline int iommu_device_claim_dma_owner(struct= device *dev, void *owner) } =20 static inline int iommu_attach_device_pasid(struct iommu_domain *domain, - struct device *dev, ioasid_t pasid) + struct device *dev, ioasid_t pasid, + struct iommu_attach_handle *handle) { return -ENODEV; } diff --git a/drivers/dma/idxd/init.c b/drivers/dma/idxd/init.c index a7295943fa22..385c488c9cd1 100644 --- a/drivers/dma/idxd/init.c +++ b/drivers/dma/idxd/init.c @@ -584,7 +584,7 @@ static int idxd_enable_system_pasid(struct idxd_device = *idxd) * DMA domain is owned by the driver, it should support all valid * types such as DMA-FQ, identity, etc. */ - ret =3D iommu_attach_device_pasid(domain, dev, pasid); + ret =3D iommu_attach_device_pasid(domain, dev, pasid, NULL); if (ret) { dev_err(dev, "failed to attach device pasid %d, domain type %d", pasid, domain->type); diff --git a/drivers/iommu/iommu-sva.c b/drivers/iommu/iommu-sva.c index 18a35e798b72..0fb923254062 100644 --- a/drivers/iommu/iommu-sva.c +++ b/drivers/iommu/iommu-sva.c @@ -99,7 +99,9 @@ struct iommu_sva *iommu_sva_bind_device(struct device *de= v, struct mm_struct *mm =20 /* Search for an existing domain. */ list_for_each_entry(domain, &mm->iommu_mm->sva_domains, next) { - ret =3D iommu_attach_device_pasid(domain, dev, iommu_mm->pasid); + handle->handle.domain =3D domain; + ret =3D iommu_attach_device_pasid(domain, dev, iommu_mm->pasid, + &handle->handle); if (!ret) { domain->users++; goto out; @@ -113,7 +115,9 @@ struct iommu_sva *iommu_sva_bind_device(struct device *= dev, struct mm_struct *mm goto out_free_handle; } =20 - ret =3D iommu_attach_device_pasid(domain, dev, iommu_mm->pasid); + handle->handle.domain =3D domain; + ret =3D iommu_attach_device_pasid(domain, dev, iommu_mm->pasid, + &handle->handle); if (ret) goto out_free_domain; domain->users =3D 1; @@ -124,7 +128,6 @@ struct iommu_sva *iommu_sva_bind_device(struct device *= dev, struct mm_struct *mm list_add(&handle->handle_item, &mm->iommu_mm->sva_handles); mutex_unlock(&iommu_sva_lock); handle->dev =3D dev; - handle->domain =3D domain; return handle; =20 out_free_domain: @@ -147,7 +150,7 @@ EXPORT_SYMBOL_GPL(iommu_sva_bind_device); */ void iommu_sva_unbind_device(struct iommu_sva *handle) { - struct iommu_domain *domain =3D handle->domain; + struct iommu_domain *domain =3D handle->handle.domain; struct iommu_mm_data *iommu_mm =3D domain->mm->iommu_mm; struct device *dev =3D handle->dev; =20 @@ -170,7 +173,7 @@ EXPORT_SYMBOL_GPL(iommu_sva_unbind_device); =20 u32 iommu_sva_get_pasid(struct iommu_sva *handle) { - struct iommu_domain *domain =3D handle->domain; + struct iommu_domain *domain =3D handle->handle.domain; =20 return mm_get_enqcmd_pasid(domain->mm); } diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c index 9df7cc75c1bc..a712b0cc3a1d 100644 --- a/drivers/iommu/iommu.c +++ b/drivers/iommu/iommu.c @@ -3352,16 +3352,17 @@ static void __iommu_remove_group_pasid(struct iommu= _group *group, * @domain: the iommu domain. * @dev: the attached device. * @pasid: the pasid of the device. + * @handle: the attach handle. * * Return: 0 on success, or an error. */ int iommu_attach_device_pasid(struct iommu_domain *domain, - struct device *dev, ioasid_t pasid) + struct device *dev, ioasid_t pasid, + struct iommu_attach_handle *handle) { /* Caller must be a probed driver on dev */ struct iommu_group *group =3D dev->iommu_group; struct group_device *device; - void *curr; int ret; =20 if (!domain->ops->set_dev_pasid) @@ -3382,11 +3383,12 @@ int iommu_attach_device_pasid(struct iommu_domain *= domain, } } =20 - curr =3D xa_cmpxchg(&group->pasid_array, pasid, NULL, domain, GFP_KERNEL); - if (curr) { - ret =3D xa_err(curr) ? : -EBUSY; + if (handle) + handle->domain =3D domain; + + ret =3D xa_insert(&group->pasid_array, pasid, handle, GFP_KERNEL); + if (ret) goto out_unlock; - } =20 ret =3D __iommu_set_group_pasid(domain, group, pasid); if (ret) @@ -3414,7 +3416,7 @@ void iommu_detach_device_pasid(struct iommu_domain *d= omain, struct device *dev, =20 mutex_lock(&group->mutex); __iommu_remove_group_pasid(group, pasid, domain); - WARN_ON(xa_erase(&group->pasid_array, pasid) !=3D domain); + xa_erase(&group->pasid_array, pasid); mutex_unlock(&group->mutex); } EXPORT_SYMBOL_GPL(iommu_detach_device_pasid); @@ -3439,15 +3441,19 @@ struct iommu_domain *iommu_get_domain_for_dev_pasid= (struct device *dev, { /* Caller must be a probed driver on dev */ struct iommu_group *group =3D dev->iommu_group; - struct iommu_domain *domain; + struct iommu_attach_handle *handle; + struct iommu_domain *domain =3D NULL; =20 if (!group) return NULL; =20 xa_lock(&group->pasid_array); - domain =3D xa_load(&group->pasid_array, pasid); + handle =3D xa_load(&group->pasid_array, pasid); + if (handle) + domain =3D handle->domain; + if (type && domain && domain->type !=3D type) - domain =3D ERR_PTR(-EBUSY); + domain =3D NULL; xa_unlock(&group->pasid_array); =20 return domain; --=20 2.34.1 From nobody Tue Dec 16 16:39:05 2025 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BDFB47E0FC for ; Tue, 2 Jul 2024 06:37:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.10 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719902260; cv=none; b=dX0j9RYH3eovjHEvOoDCkY5hHcun5SsnpW6uqZUh+WJ+8Hcz9EZRuNxyvjodyR+W4eLJ3NTqCeBq3AOpMA4K8kDC4nv/3FEgo6tn2VYxXk2pnqO3gj+xjbMYRfVH3ourPoYfgrgnLsf8SKJTYGGXBrbec2gDdIa3S+GYK5pgu/I= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719902260; c=relaxed/simple; bh=8+iX1NsitGtjrQyHSg7mn0zvhh5F8O52l6x99Zx88WY=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=rIEBz46ibYcdprfWVXrdNSBEjXcWm84Bp+Co9PnECiUdvOOv19tCAMnLpQQHHGvVfZHngLpvC/WpNgaq0VGu+2CHHjnQrarckrroX8W81WlXFkmC33wNRVB1+wvLw572X8VsZgQjq9GHp9BO9PBWrIIzqphtqQWclX/2t1vIktQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=CCLfT3jk; arc=none smtp.client-ip=192.198.163.10 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="CCLfT3jk" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1719902259; x=1751438259; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=8+iX1NsitGtjrQyHSg7mn0zvhh5F8O52l6x99Zx88WY=; b=CCLfT3jkIgymoAMyRs+B9yVKY3Pk8x3JX1cGTaXqybj3kpdPU22GerCv ZHQnfN615D4m2bi61fpP1FqDt0LkP4zBXSDycJG/6uAxG3YN2RoojsXGV v2JjDm2A8mjetsx077iuQ3OhZtGM/+oaUGueKg4XUcDIJBVGvKLLZVqm1 62rK0fIC/qCGDa/EdtHltTWVhwCP2n256iLHCMl9hfFSgjr9vINSwk8Nw XCwu6F2UTAqh36C2O2Mi4Ml50ok/VoMiI3dhy0mxmZKNciOvC/3GSwFuq u1Sjl3udVK+M54+VpaVzzJv+Q4f5uBpBShuQRaBnfuo7s7iccgL9tdI6t w==; X-CSE-ConnectionGUID: 47h4kGAnTkGPn7xwJHmQ3A== X-CSE-MsgGUID: +UijGfVbSDq/LrI4F7BRSw== X-IronPort-AV: E=McAfee;i="6700,10204,11120"; a="28455586" X-IronPort-AV: E=Sophos;i="6.09,178,1716274800"; d="scan'208";a="28455586" Received: from fmviesa003.fm.intel.com ([10.60.135.143]) by fmvoesa104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 01 Jul 2024 23:37:39 -0700 X-CSE-ConnectionGUID: QIFYx1P3SeOJPKRYYrGTCA== X-CSE-MsgGUID: N7ZCtmPNTKeMTKnpessyjg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.09,178,1716274800"; d="scan'208";a="50137130" Received: from unknown (HELO allen-box.sh.intel.com) ([10.239.159.127]) by fmviesa003.fm.intel.com with ESMTP; 01 Jul 2024 23:37:35 -0700 From: Lu Baolu To: Jason Gunthorpe , Kevin Tian , Joerg Roedel , Will Deacon , Robin Murphy , Jean-Philippe Brucker , Nicolin Chen , Yi Liu , Jacob Pan , Joel Granados Cc: iommu@lists.linux.dev, virtualization@lists.linux-foundation.org, linux-kernel@vger.kernel.org, Lu Baolu , Jason Gunthorpe Subject: [PATCH v8 02/10] iommu: Remove sva handle list Date: Tue, 2 Jul 2024 14:34:36 +0800 Message-Id: <20240702063444.105814-3-baolu.lu@linux.intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240702063444.105814-1-baolu.lu@linux.intel.com> References: <20240702063444.105814-1-baolu.lu@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The struct sva_iommu represents an association between an SVA domain and a PASID of a device. It's stored in the iommu group's pasid array and also tracked by a list in the per-mm data structure. Removes duplicate tracking of sva_iommu by eliminating the list. Signed-off-by: Lu Baolu Reviewed-by: Jason Gunthorpe Reviewed-by: Kevin Tian --- include/linux/iommu.h | 2 -- drivers/iommu/iommu-priv.h | 3 +++ drivers/iommu/iommu-sva.c | 30 ++++++++++++++++++++---------- drivers/iommu/iommu.c | 31 +++++++++++++++++++++++++++++++ 4 files changed, 54 insertions(+), 12 deletions(-) diff --git a/include/linux/iommu.h b/include/linux/iommu.h index afc5af0069bb..87ebcc29020e 100644 --- a/include/linux/iommu.h +++ b/include/linux/iommu.h @@ -1005,14 +1005,12 @@ struct iommu_attach_handle { struct iommu_sva { struct iommu_attach_handle handle; struct device *dev; - struct list_head handle_item; refcount_t users; }; =20 struct iommu_mm_data { u32 pasid; struct list_head sva_domains; - struct list_head sva_handles; }; =20 int iommu_fwspec_init(struct device *dev, struct fwnode_handle *iommu_fwno= de, diff --git a/drivers/iommu/iommu-priv.h b/drivers/iommu/iommu-priv.h index 5f731d994803..f1536a5ebb0d 100644 --- a/drivers/iommu/iommu-priv.h +++ b/drivers/iommu/iommu-priv.h @@ -28,4 +28,7 @@ void iommu_device_unregister_bus(struct iommu_device *iom= mu, const struct bus_type *bus, struct notifier_block *nb); =20 +struct iommu_attach_handle *iommu_attach_handle_get(struct iommu_group *gr= oup, + ioasid_t pasid, + unsigned int type); #endif /* __LINUX_IOMMU_PRIV_H */ diff --git a/drivers/iommu/iommu-sva.c b/drivers/iommu/iommu-sva.c index 0fb923254062..9b7f62517419 100644 --- a/drivers/iommu/iommu-sva.c +++ b/drivers/iommu/iommu-sva.c @@ -41,7 +41,6 @@ static struct iommu_mm_data *iommu_alloc_mm_data(struct m= m_struct *mm, struct de } iommu_mm->pasid =3D pasid; INIT_LIST_HEAD(&iommu_mm->sva_domains); - INIT_LIST_HEAD(&iommu_mm->sva_handles); /* * Make sure the write to mm->iommu_mm is not reordered in front of * initialization to iommu_mm fields. If it does, readers may see a @@ -69,11 +68,16 @@ static struct iommu_mm_data *iommu_alloc_mm_data(struct= mm_struct *mm, struct de */ struct iommu_sva *iommu_sva_bind_device(struct device *dev, struct mm_stru= ct *mm) { + struct iommu_group *group =3D dev->iommu_group; + struct iommu_attach_handle *attach_handle; struct iommu_mm_data *iommu_mm; struct iommu_domain *domain; struct iommu_sva *handle; int ret; =20 + if (!group) + return ERR_PTR(-ENODEV); + mutex_lock(&iommu_sva_lock); =20 /* Allocate mm->pasid if necessary. */ @@ -83,12 +87,22 @@ struct iommu_sva *iommu_sva_bind_device(struct device *= dev, struct mm_struct *mm goto out_unlock; } =20 - list_for_each_entry(handle, &mm->iommu_mm->sva_handles, handle_item) { - if (handle->dev =3D=3D dev) { - refcount_inc(&handle->users); - mutex_unlock(&iommu_sva_lock); - return handle; + /* A bond already exists, just take a reference`. */ + attach_handle =3D iommu_attach_handle_get(group, iommu_mm->pasid, IOMMU_D= OMAIN_SVA); + if (!IS_ERR(attach_handle)) { + handle =3D container_of(attach_handle, struct iommu_sva, handle); + if (attach_handle->domain->mm !=3D mm) { + ret =3D -EBUSY; + goto out_unlock; } + refcount_inc(&handle->users); + mutex_unlock(&iommu_sva_lock); + return handle; + } + + if (PTR_ERR(attach_handle) !=3D -ENOENT) { + ret =3D PTR_ERR(attach_handle); + goto out_unlock; } =20 handle =3D kzalloc(sizeof(*handle), GFP_KERNEL); @@ -99,7 +113,6 @@ struct iommu_sva *iommu_sva_bind_device(struct device *d= ev, struct mm_struct *mm =20 /* Search for an existing domain. */ list_for_each_entry(domain, &mm->iommu_mm->sva_domains, next) { - handle->handle.domain =3D domain; ret =3D iommu_attach_device_pasid(domain, dev, iommu_mm->pasid, &handle->handle); if (!ret) { @@ -115,7 +128,6 @@ struct iommu_sva *iommu_sva_bind_device(struct device *= dev, struct mm_struct *mm goto out_free_handle; } =20 - handle->handle.domain =3D domain; ret =3D iommu_attach_device_pasid(domain, dev, iommu_mm->pasid, &handle->handle); if (ret) @@ -125,7 +137,6 @@ struct iommu_sva *iommu_sva_bind_device(struct device *= dev, struct mm_struct *mm =20 out: refcount_set(&handle->users, 1); - list_add(&handle->handle_item, &mm->iommu_mm->sva_handles); mutex_unlock(&iommu_sva_lock); handle->dev =3D dev; return handle; @@ -159,7 +170,6 @@ void iommu_sva_unbind_device(struct iommu_sva *handle) mutex_unlock(&iommu_sva_lock); return; } - list_del(&handle->handle_item); =20 iommu_detach_device_pasid(domain, dev, iommu_mm->pasid); if (--domain->users =3D=3D 0) { diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c index a712b0cc3a1d..7890bd21dff6 100644 --- a/drivers/iommu/iommu.c +++ b/drivers/iommu/iommu.c @@ -3486,3 +3486,34 @@ void iommu_free_global_pasid(ioasid_t pasid) ida_free(&iommu_global_pasid_ida, pasid); } EXPORT_SYMBOL_GPL(iommu_free_global_pasid); + +/** + * iommu_attach_handle_get - Return the attach handle + * @group: the iommu group that domain was attached to + * @pasid: the pasid within the group + * @type: matched domain type, 0 for any match + * + * Return handle or ERR_PTR(-ENOENT) on none, ERR_PTR(-EBUSY) on mismatch. + * + * Return the attach handle to the caller. The life cycle of an iommu atta= ch + * handle is from the time when the domain is attached to the time when the + * domain is detached. Callers are required to synchronize the call of + * iommu_attach_handle_get() with domain attachment and detachment. The at= tach + * handle can only be used during its life cycle. + */ +struct iommu_attach_handle * +iommu_attach_handle_get(struct iommu_group *group, ioasid_t pasid, unsigne= d int type) +{ + struct iommu_attach_handle *handle; + + xa_lock(&group->pasid_array); + handle =3D xa_load(&group->pasid_array, pasid); + if (!handle) + handle =3D ERR_PTR(-ENOENT); + else if (type && handle->domain->type !=3D type) + handle =3D ERR_PTR(-EBUSY); + xa_unlock(&group->pasid_array); + + return handle; +} +EXPORT_SYMBOL_NS_GPL(iommu_attach_handle_get, IOMMUFD_INTERNAL); --=20 2.34.1 From nobody Tue Dec 16 16:39:05 2025 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D57C9145B09 for ; Tue, 2 Jul 2024 06:37:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.10 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719902265; cv=none; b=g4N02HzIoLdyJE7qVIx/siNOjIBTn8WGsoApYA67P6tBywuXwdXpKi3g+8F+NROpUZcZ/x0F8t7tilrAy7oIMj3eiI6U+A0sd3s0ECJlTTIQfX0yzwiZOuRJYvgkjjC0JA8HZQIW69k+zhET9lPk//QLPYX1xdUjkdfSqQEvJlE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719902265; c=relaxed/simple; bh=g9xlceX+XG1++Meqt1kNrrz3rqWM6bM5bix5CevIJCk=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=RnsBXX7m9nELGyAZxDmdNgC80Z61eGRtFfFcs925RNyiiIdP2hTVzmLaMQKvjbEu/DrXjlxs/KlISk2cKtgRtf4jzKkwXM3T4TICsGDcNDHqCRGTnPAPY9aCjT2ACOiskZN7HBljRaocjMQM5YeT/N+xjn4tuHEyxVFOF5PeadU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=Vq3HJ4VP; arc=none smtp.client-ip=192.198.163.10 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="Vq3HJ4VP" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1719902263; x=1751438263; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=g9xlceX+XG1++Meqt1kNrrz3rqWM6bM5bix5CevIJCk=; b=Vq3HJ4VPZPrKmOehUyKmGwEMdFLsPkgkwKZ7kWTmg5CpAs4MXD6fZpr+ QbKnNBxD85BqYFPGOs0sAxs8+B1p8VayzAm3EyruRD6gJzlR42CiV4xIe p6Qtsw7nOgsGY44QnOw/LQgr3NdkoFm88x+IEArNSP1aqK/VUT7llE9/I iQc55NC+lonT8U8BfkgNXa7NdwLYEPGvfE2UYiEx5MqC7+QICQx8/H7pV uU6TprPRp8fc0ZcEE6Bf3/3cqO40BTniNX7A+hBDZSbJaPDG77IIT6l2L 7AUD3VMSew01Xl+D3RdW469qMpEe3g2V+dGmn2+6je7yXjgjMDaEIvMyn Q==; X-CSE-ConnectionGUID: 6uEBMLsxRgest4atkUwBeQ== X-CSE-MsgGUID: jkjK+KfIRpyXgUt8ppGbkw== X-IronPort-AV: E=McAfee;i="6700,10204,11120"; a="28455599" X-IronPort-AV: E=Sophos;i="6.09,178,1716274800"; d="scan'208";a="28455599" Received: from fmviesa003.fm.intel.com ([10.60.135.143]) by fmvoesa104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 01 Jul 2024 23:37:43 -0700 X-CSE-ConnectionGUID: YjHOGDR3SJCzU2PubHODOw== X-CSE-MsgGUID: xnf7qX/8T0ewf0GBCK4Kuw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.09,178,1716274800"; d="scan'208";a="50137159" Received: from unknown (HELO allen-box.sh.intel.com) ([10.239.159.127]) by fmviesa003.fm.intel.com with ESMTP; 01 Jul 2024 23:37:38 -0700 From: Lu Baolu To: Jason Gunthorpe , Kevin Tian , Joerg Roedel , Will Deacon , Robin Murphy , Jean-Philippe Brucker , Nicolin Chen , Yi Liu , Jacob Pan , Joel Granados Cc: iommu@lists.linux.dev, virtualization@lists.linux-foundation.org, linux-kernel@vger.kernel.org, Lu Baolu , Jason Gunthorpe Subject: [PATCH v8 03/10] iommu: Add attach handle to struct iopf_group Date: Tue, 2 Jul 2024 14:34:37 +0800 Message-Id: <20240702063444.105814-4-baolu.lu@linux.intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240702063444.105814-1-baolu.lu@linux.intel.com> References: <20240702063444.105814-1-baolu.lu@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Previously, the domain that a page fault targets is stored in an iopf_group, which represents a minimal set of page faults. With the introduction of attach handle, replace the domain with the handle so that the fault handler can obtain more information as needed when handling the faults. iommu_report_device_fault() is currently used for SVA page faults, which handles the page fault in an internal cycle. The domain is retrieved with iommu_get_domain_for_dev_pasid() if the pasid in the fault message is valid. This doesn't work in IOMMUFD case, where if the pasid table of a device is wholly managed by user space, there is no domain attached to the PASID of the device, and all page faults are forwarded through a NESTING domain attaching to RID. Add a static flag in iommu ops, which indicates if the IOMMU driver supports user-managed PASID tables. In the iopf deliver path, if no attach handle found for the iopf PASID, roll back to RID domain when the IOMMU driver supports this capability. iommu_get_domain_for_dev_pasid() is no longer used and can be removed. Signed-off-by: Lu Baolu Reviewed-by: Jason Gunthorpe Reviewed-by: Kevin Tian --- include/linux/iommu.h | 17 ++++------- drivers/iommu/io-pgfault.c | 61 +++++++++++++++++++++----------------- drivers/iommu/iommu-sva.c | 3 +- drivers/iommu/iommu.c | 39 ------------------------ 4 files changed, 42 insertions(+), 78 deletions(-) diff --git a/include/linux/iommu.h b/include/linux/iommu.h index 87ebcc29020e..910aec80886e 100644 --- a/include/linux/iommu.h +++ b/include/linux/iommu.h @@ -127,7 +127,7 @@ struct iopf_group { /* list node for iommu_fault_param::faults */ struct list_head pending_node; struct work_struct work; - struct iommu_domain *domain; + struct iommu_attach_handle *attach_handle; /* The device's fault data parameter. */ struct iommu_fault_param *fault_param; }; @@ -547,6 +547,10 @@ static inline int __iommu_copy_struct_from_user_array( * @default_domain: If not NULL this will always be set as the default dom= ain. * This should be an IDENTITY/BLOCKED/PLATFORM domain. * Do not use in new drivers. + * @user_pasid_table: IOMMU driver supports user-managed PASID table. Ther= e is + * no user domain for each PASID and the I/O page fault= s are + * forwarded through the user domain attached to the de= vice + * RID. */ struct iommu_ops { bool (*capable)(struct device *dev, enum iommu_cap); @@ -590,6 +594,7 @@ struct iommu_ops { struct iommu_domain *blocked_domain; struct iommu_domain *release_domain; struct iommu_domain *default_domain; + u8 user_pasid_table:1; }; =20 /** @@ -1064,9 +1069,6 @@ int iommu_attach_device_pasid(struct iommu_domain *do= main, struct iommu_attach_handle *handle); void iommu_detach_device_pasid(struct iommu_domain *domain, struct device *dev, ioasid_t pasid); -struct iommu_domain * -iommu_get_domain_for_dev_pasid(struct device *dev, ioasid_t pasid, - unsigned int type); ioasid_t iommu_alloc_global_pasid(struct device *dev); void iommu_free_global_pasid(ioasid_t pasid); #else /* CONFIG_IOMMU_API */ @@ -1408,13 +1410,6 @@ static inline void iommu_detach_device_pasid(struct = iommu_domain *domain, { } =20 -static inline struct iommu_domain * -iommu_get_domain_for_dev_pasid(struct device *dev, ioasid_t pasid, - unsigned int type) -{ - return NULL; -} - static inline ioasid_t iommu_alloc_global_pasid(struct device *dev) { return IOMMU_PASID_INVALID; diff --git a/drivers/iommu/io-pgfault.c b/drivers/iommu/io-pgfault.c index 06d78fcc79fd..7c9011992d3f 100644 --- a/drivers/iommu/io-pgfault.c +++ b/drivers/iommu/io-pgfault.c @@ -59,30 +59,6 @@ void iopf_free_group(struct iopf_group *group) } EXPORT_SYMBOL_GPL(iopf_free_group); =20 -static struct iommu_domain *get_domain_for_iopf(struct device *dev, - struct iommu_fault *fault) -{ - struct iommu_domain *domain; - - if (fault->prm.flags & IOMMU_FAULT_PAGE_REQUEST_PASID_VALID) { - domain =3D iommu_get_domain_for_dev_pasid(dev, fault->prm.pasid, 0); - if (IS_ERR(domain)) - domain =3D NULL; - } else { - domain =3D iommu_get_domain_for_dev(dev); - } - - if (!domain || !domain->iopf_handler) { - dev_warn_ratelimited(dev, - "iopf (pasid %d) without domain attached or handler installed\n", - fault->prm.pasid); - - return NULL; - } - - return domain; -} - /* Non-last request of a group. Postpone until the last one. */ static int report_partial_fault(struct iommu_fault_param *fault_param, struct iommu_fault *fault) @@ -206,20 +182,51 @@ void iommu_report_device_fault(struct device *dev, st= ruct iopf_fault *evt) if (group =3D=3D &abort_group) goto err_abort; =20 - group->domain =3D get_domain_for_iopf(dev, fault); - if (!group->domain) + if (fault->prm.flags & IOMMU_FAULT_PAGE_REQUEST_PASID_VALID) { + group->attach_handle =3D iommu_attach_handle_get(dev->iommu_group, + fault->prm.pasid, + 0); + if (IS_ERR(group->attach_handle)) { + const struct iommu_ops *ops =3D dev_iommu_ops(dev); + + if (!ops->user_pasid_table) + goto err_abort; + + /* + * The iommu driver for this device supports user- + * managed PASID table. Therefore page faults for + * any PASID should go through the NESTING domain + * attached to the device RID. + */ + group->attach_handle =3D + iommu_attach_handle_get(dev->iommu_group, + IOMMU_NO_PASID, + IOMMU_DOMAIN_NESTED); + if (IS_ERR(group->attach_handle)) + goto err_abort; + } + } else { + group->attach_handle =3D + iommu_attach_handle_get(dev->iommu_group, IOMMU_NO_PASID, 0); + if (IS_ERR(group->attach_handle)) + goto err_abort; + } + + if (!group->attach_handle->domain->iopf_handler) goto err_abort; =20 /* * On success iopf_handler must call iopf_group_response() and * iopf_free_group() */ - if (group->domain->iopf_handler(group)) + if (group->attach_handle->domain->iopf_handler(group)) goto err_abort; =20 return; =20 err_abort: + dev_warn_ratelimited(dev, "iopf with pasid %d aborted\n", + fault->prm.pasid); iopf_group_response(group, IOMMU_PAGE_RESP_FAILURE); if (group =3D=3D &abort_group) __iopf_free_group(group); diff --git a/drivers/iommu/iommu-sva.c b/drivers/iommu/iommu-sva.c index 9b7f62517419..69cde094440e 100644 --- a/drivers/iommu/iommu-sva.c +++ b/drivers/iommu/iommu-sva.c @@ -272,7 +272,8 @@ static void iommu_sva_handle_iopf(struct work_struct *w= ork) if (status !=3D IOMMU_PAGE_RESP_SUCCESS) break; =20 - status =3D iommu_sva_handle_mm(&iopf->fault, group->domain->mm); + status =3D iommu_sva_handle_mm(&iopf->fault, + group->attach_handle->domain->mm); } =20 iopf_group_response(group, status); diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c index 7890bd21dff6..5a7e874abb36 100644 --- a/drivers/iommu/iommu.c +++ b/drivers/iommu/iommu.c @@ -3421,45 +3421,6 @@ void iommu_detach_device_pasid(struct iommu_domain *= domain, struct device *dev, } EXPORT_SYMBOL_GPL(iommu_detach_device_pasid); =20 -/* - * iommu_get_domain_for_dev_pasid() - Retrieve domain for @pasid of @dev - * @dev: the queried device - * @pasid: the pasid of the device - * @type: matched domain type, 0 for any match - * - * This is a variant of iommu_get_domain_for_dev(). It returns the existing - * domain attached to pasid of a device. Callers must hold a lock around t= his - * function, and both iommu_attach/detach_dev_pasid() whenever a domain of - * type is being manipulated. This API does not internally resolve races w= ith - * attach/detach. - * - * Return: attached domain on success, NULL otherwise. - */ -struct iommu_domain *iommu_get_domain_for_dev_pasid(struct device *dev, - ioasid_t pasid, - unsigned int type) -{ - /* Caller must be a probed driver on dev */ - struct iommu_group *group =3D dev->iommu_group; - struct iommu_attach_handle *handle; - struct iommu_domain *domain =3D NULL; - - if (!group) - return NULL; - - xa_lock(&group->pasid_array); - handle =3D xa_load(&group->pasid_array, pasid); - if (handle) - domain =3D handle->domain; - - if (type && domain && domain->type !=3D type) - domain =3D NULL; - xa_unlock(&group->pasid_array); - - return domain; -} -EXPORT_SYMBOL_GPL(iommu_get_domain_for_dev_pasid); - ioasid_t iommu_alloc_global_pasid(struct device *dev) { int ret; --=20 2.34.1 From nobody Tue Dec 16 16:39:05 2025 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6DB171474CC for ; Tue, 2 Jul 2024 06:37:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.10 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719902267; cv=none; b=qGQkm1ZiHB0dEYoS5th/GkiqSTeNEim4u5rP/R55XQs1Clfxn6eD7lSIC5Q0nNKmQIZgqunf2yfF1OX6PeWfHcLJwozdKfaOSVLEsqHIRKsEjmsQ44oA7Jkoc0tPdQnQVauF/QT0eP/yh4eUjKQpbenEArEuyqOLgyDlDcr6CO0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719902267; c=relaxed/simple; bh=dPdzFv08205rJsZbX2ITbLdssatLx9iQwSF9vaAj3+Y=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=td76o6rbGh6sOmk6nwdvhGZkS6PBeg+dy6ZkuB4w3x/fTHydrPvVX1DiOmGg0ZQWh+ZyKdVou3spLDyd+jemks52m1uheCx5bMaNa2dL60SvWJMOnVNGwIkAqKRefNdkAE4gpnA9IHh6JXoQbczTIwUdVRPyOt11Q/uDewlUQgU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=Sclzd2Dq; arc=none smtp.client-ip=192.198.163.10 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="Sclzd2Dq" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1719902267; x=1751438267; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=dPdzFv08205rJsZbX2ITbLdssatLx9iQwSF9vaAj3+Y=; b=Sclzd2DqkUWazcu/3x7Dp62p7oRhReASkKFdJ+FfXo7INqKP9A7EsCl3 W0em+A/USjj2Qcrf9JkHApMj9WOGwUdTiYXk0K7o59S+9VMiVVPIYSVVz cKSjBTiijWHyJ+gYfbNLSJVoiSmLhqAkC9pCJ7jRakOUAofFGvMAm+5GY ObSPbxRFyD+Q+zYkFbG632yjrZRNWieexSZPnu4RxOqfl6t8Cnr8zU0sk fcTGK1otdJLyGIEeXbeU5kE0So/WZm3ZULlqZhSvgQMyo4rdTV0pRH409 6BgQztfUOaVCRzUCdqCWOBpqj0nB0PlErRt+k/sGbgkCHzrPNDwNvIDBr g==; X-CSE-ConnectionGUID: 5TczVRJwRNOY4VonX6bO5w== X-CSE-MsgGUID: TyrC/i4aSMyI9NLxV20KKg== X-IronPort-AV: E=McAfee;i="6700,10204,11120"; a="28455610" X-IronPort-AV: E=Sophos;i="6.09,178,1716274800"; d="scan'208";a="28455610" Received: from fmviesa003.fm.intel.com ([10.60.135.143]) by fmvoesa104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 01 Jul 2024 23:37:46 -0700 X-CSE-ConnectionGUID: j48sfWD0Q9urrdK3mAhPsg== X-CSE-MsgGUID: 1NAE0M01RbqPfy3TsjBiDQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.09,178,1716274800"; d="scan'208";a="50137185" Received: from unknown (HELO allen-box.sh.intel.com) ([10.239.159.127]) by fmviesa003.fm.intel.com with ESMTP; 01 Jul 2024 23:37:42 -0700 From: Lu Baolu To: Jason Gunthorpe , Kevin Tian , Joerg Roedel , Will Deacon , Robin Murphy , Jean-Philippe Brucker , Nicolin Chen , Yi Liu , Jacob Pan , Joel Granados Cc: iommu@lists.linux.dev, virtualization@lists.linux-foundation.org, linux-kernel@vger.kernel.org, Lu Baolu Subject: [PATCH v8 04/10] iommu: Extend domain attach group with handle support Date: Tue, 2 Jul 2024 14:34:38 +0800 Message-Id: <20240702063444.105814-5-baolu.lu@linux.intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240702063444.105814-1-baolu.lu@linux.intel.com> References: <20240702063444.105814-1-baolu.lu@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Unlike the SVA case where each PASID of a device has an SVA domain attached to it, the I/O page faults are handled by the fault handler of the SVA domain. The I/O page faults for a user page table might be handled by the domain attached to RID or the domain attached to the PASID, depending on whether the PASID table is managed by user space or kernel. As a result, there is a need for the domain attach group interfaces to have attach handle support. The attach handle will be forwarded to the fault handler of the user domain. Add some variants of the domain attaching group interfaces so that they could support the attach handle and export them for use in IOMMUFD. Signed-off-by: Lu Baolu Reviewed-by: Kevin Tian --- drivers/iommu/iommu-priv.h | 8 +++ drivers/iommu/iommu.c | 103 +++++++++++++++++++++++++++++++++++++ 2 files changed, 111 insertions(+) diff --git a/drivers/iommu/iommu-priv.h b/drivers/iommu/iommu-priv.h index f1536a5ebb0d..c37801c32f33 100644 --- a/drivers/iommu/iommu-priv.h +++ b/drivers/iommu/iommu-priv.h @@ -31,4 +31,12 @@ void iommu_device_unregister_bus(struct iommu_device *io= mmu, struct iommu_attach_handle *iommu_attach_handle_get(struct iommu_group *gr= oup, ioasid_t pasid, unsigned int type); +int iommu_attach_group_handle(struct iommu_domain *domain, + struct iommu_group *group, + struct iommu_attach_handle *handle); +void iommu_detach_group_handle(struct iommu_domain *domain, + struct iommu_group *group); +int iommu_replace_group_handle(struct iommu_group *group, + struct iommu_domain *new_domain, + struct iommu_attach_handle *handle); #endif /* __LINUX_IOMMU_PRIV_H */ diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c index 5a7e874abb36..8484285fbaa8 100644 --- a/drivers/iommu/iommu.c +++ b/drivers/iommu/iommu.c @@ -3478,3 +3478,106 @@ iommu_attach_handle_get(struct iommu_group *group, = ioasid_t pasid, unsigned int return handle; } EXPORT_SYMBOL_NS_GPL(iommu_attach_handle_get, IOMMUFD_INTERNAL); + +/** + * iommu_attach_group_handle - Attach an IOMMU domain to an IOMMU group + * @domain: IOMMU domain to attach + * @group: IOMMU group that will be attached + * @handle: attach handle + * + * Returns 0 on success and error code on failure. + * + * This is a variant of iommu_attach_group(). It allows the caller to prov= ide + * an attach handle and use it when the domain is attached. This is curren= tly + * used by IOMMUFD to deliver the I/O page faults. + */ +int iommu_attach_group_handle(struct iommu_domain *domain, + struct iommu_group *group, + struct iommu_attach_handle *handle) +{ + int ret; + + if (handle) + handle->domain =3D domain; + + mutex_lock(&group->mutex); + ret =3D xa_insert(&group->pasid_array, IOMMU_NO_PASID, handle, GFP_KERNEL= ); + if (ret) + goto err_unlock; + + ret =3D __iommu_attach_group(domain, group); + if (ret) + goto err_erase; + mutex_unlock(&group->mutex); + + return 0; +err_erase: + xa_erase(&group->pasid_array, IOMMU_NO_PASID); +err_unlock: + mutex_unlock(&group->mutex); + return ret; +} +EXPORT_SYMBOL_NS_GPL(iommu_attach_group_handle, IOMMUFD_INTERNAL); + +/** + * iommu_detach_group_handle - Detach an IOMMU domain from an IOMMU group + * @domain: IOMMU domain to attach + * @group: IOMMU group that will be attached + * + * Detach the specified IOMMU domain from the specified IOMMU group. + * It must be used in conjunction with iommu_attach_group_handle(). + */ +void iommu_detach_group_handle(struct iommu_domain *domain, + struct iommu_group *group) +{ + mutex_lock(&group->mutex); + __iommu_group_set_core_domain(group); + xa_erase(&group->pasid_array, IOMMU_NO_PASID); + mutex_unlock(&group->mutex); +} +EXPORT_SYMBOL_NS_GPL(iommu_detach_group_handle, IOMMUFD_INTERNAL); + +/** + * iommu_replace_group_handle - replace the domain that a group is attache= d to + * @group: IOMMU group that will be attached to the new domain + * @new_domain: new IOMMU domain to replace with + * @handle: attach handle + * + * This is a variant of iommu_group_replace_domain(). It allows the caller= to + * provide an attach handle for the new domain and use it when the domain = is + * attached. + */ +int iommu_replace_group_handle(struct iommu_group *group, + struct iommu_domain *new_domain, + struct iommu_attach_handle *handle) +{ + void *curr; + int ret; + + if (!new_domain) + return -EINVAL; + + mutex_lock(&group->mutex); + if (handle) { + ret =3D xa_reserve(&group->pasid_array, IOMMU_NO_PASID, GFP_KERNEL); + if (ret) + goto err_unlock; + } + + ret =3D __iommu_group_set_domain(group, new_domain); + if (ret) + goto err_release; + + curr =3D xa_store(&group->pasid_array, IOMMU_NO_PASID, handle, GFP_KERNEL= ); + WARN_ON(xa_is_err(curr)); + + mutex_unlock(&group->mutex); + + return 0; +err_release: + xa_release(&group->pasid_array, IOMMU_NO_PASID); +err_unlock: + mutex_unlock(&group->mutex); + return ret; +} +EXPORT_SYMBOL_NS_GPL(iommu_replace_group_handle, IOMMUFD_INTERNAL); --=20 2.34.1 From nobody Tue Dec 16 16:39:05 2025 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D18EB149DF0 for ; Tue, 2 Jul 2024 06:37:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.10 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719902271; cv=none; b=e150tWvrKN9M/vSIUS6YfELdyMbs/mjIOfkoVfgvD/bUqrzApM4Ts+evMRW6v1LHy1HBvnV9sAvi8QYGpwtWcgGHWxr6x99wS61k+tSz4Ym+prspD2F+rF/1AlrMaVfxi89I0KockdjGIoVsyrm93dPt/Jvh0gPqBJrg045uSd8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719902271; c=relaxed/simple; bh=fpDco05MC2sXG8Odatxtzhwad+XNQaPRHu/aJgHwPh8=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=YsdxmBf+bZ8FW+GFhh3ihuDzq63+c/caTag74kRemxYX8Gzx5dY+UMGjvoGPYZVuP6I/rgPCvgBU2Bg3BDevc6a1MxxJqOZ8Qk4mF0fcoB85s9xQ/8OtQ16Qjsb1qBHHQ8S8dVuJv1akFaysgIDv59/RcNGEC17ykPQwlcP2Uzw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=ECvKGeYP; arc=none smtp.client-ip=192.198.163.10 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="ECvKGeYP" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1719902270; x=1751438270; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=fpDco05MC2sXG8Odatxtzhwad+XNQaPRHu/aJgHwPh8=; b=ECvKGeYPDwYHQHQ8KtH74GoZ0FX/HwD2VJveUudfZq/bv6n2Z4FVw55I MqOv4M2yYp7EBAbRqq7hmnU7ow/FEIfLJD79a0uVVyx0mnmBVMBhmb+ca r5us65L7NV+F2THsHEuBldkU2D1GZOHW1dbat/kVGbesY5CSqT2hRAYGJ O5QPsNnNm6naEPfh6cUyHW2/JsmGdPHUPa7G/bP+xgY2+ow8+ixdyfohd HfJeDpyOqZAIcVKqAIYPgYBQb87TXI6H68XU2jEB0Tbb8CSqA5SVrGFBo MRAv90hRPJoecFZVkhinAqVBPj+6Dziiugcj0cVBqz11x/MJ+0JUFuHns A==; X-CSE-ConnectionGUID: 16kdsUXRQsiLLI4EMhRhAw== X-CSE-MsgGUID: oxnmxpLtREGMDjq2l5IesQ== X-IronPort-AV: E=McAfee;i="6700,10204,11120"; a="28455621" X-IronPort-AV: E=Sophos;i="6.09,178,1716274800"; d="scan'208";a="28455621" Received: from fmviesa003.fm.intel.com ([10.60.135.143]) by fmvoesa104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 01 Jul 2024 23:37:50 -0700 X-CSE-ConnectionGUID: fO+8mGL9Ti+fCMroa9g/Gw== X-CSE-MsgGUID: jK4lQfQnRtWH9AcVNkNhXg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.09,178,1716274800"; d="scan'208";a="50137210" Received: from unknown (HELO allen-box.sh.intel.com) ([10.239.159.127]) by fmviesa003.fm.intel.com with ESMTP; 01 Jul 2024 23:37:46 -0700 From: Lu Baolu To: Jason Gunthorpe , Kevin Tian , Joerg Roedel , Will Deacon , Robin Murphy , Jean-Philippe Brucker , Nicolin Chen , Yi Liu , Jacob Pan , Joel Granados Cc: iommu@lists.linux.dev, virtualization@lists.linux-foundation.org, linux-kernel@vger.kernel.org, Lu Baolu Subject: [PATCH v8 05/10] iommufd: Add fault and response message definitions Date: Tue, 2 Jul 2024 14:34:39 +0800 Message-Id: <20240702063444.105814-6-baolu.lu@linux.intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240702063444.105814-1-baolu.lu@linux.intel.com> References: <20240702063444.105814-1-baolu.lu@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" iommu_hwpt_pgfaults represent fault messages that the userspace can retrieve. Multiple iommu_hwpt_pgfaults might be put in an iopf group, with the IOMMU_PGFAULT_FLAGS_LAST_PAGE flag set only for the last iommu_hwpt_pgfault. An iommu_hwpt_page_response is a response message that the userspace should send to the kernel after finishing handling a group of fault messages. The @dev_id, @pasid, and @grpid fields in the message identify an outstanding iopf group for a device. The @cookie field, which matches the cookie field of the last fault in the group, will be used by the kernel to look up the pending message. Signed-off-by: Lu Baolu Reviewed-by: Kevin Tian --- include/uapi/linux/iommufd.h | 83 ++++++++++++++++++++++++++++++++++++ 1 file changed, 83 insertions(+) diff --git a/include/uapi/linux/iommufd.h b/include/uapi/linux/iommufd.h index 1dfeaa2e649e..4d89ed97b533 100644 --- a/include/uapi/linux/iommufd.h +++ b/include/uapi/linux/iommufd.h @@ -692,4 +692,87 @@ struct iommu_hwpt_invalidate { __u32 __reserved; }; #define IOMMU_HWPT_INVALIDATE _IO(IOMMUFD_TYPE, IOMMUFD_CMD_HWPT_INVALIDAT= E) + +/** + * enum iommu_hwpt_pgfault_flags - flags for struct iommu_hwpt_pgfault + * @IOMMU_PGFAULT_FLAGS_PASID_VALID: The pasid field of the fault data is + * valid. + * @IOMMU_PGFAULT_FLAGS_LAST_PAGE: It's the last fault of a fault group. + */ +enum iommu_hwpt_pgfault_flags { + IOMMU_PGFAULT_FLAGS_PASID_VALID =3D (1 << 0), + IOMMU_PGFAULT_FLAGS_LAST_PAGE =3D (1 << 1), +}; + +/** + * enum iommu_hwpt_pgfault_perm - perm bits for struct iommu_hwpt_pgfault + * @IOMMU_PGFAULT_PERM_READ: request for read permission + * @IOMMU_PGFAULT_PERM_WRITE: request for write permission + * @IOMMU_PGFAULT_PERM_EXEC: (PCIE 10.4.1) request with a PASID that has t= he + * Execute Requested bit set in PASID TLP Prefix. + * @IOMMU_PGFAULT_PERM_PRIV: (PCIE 10.4.1) request with a PASID that has t= he + * Privileged Mode Requested bit set in PASID TLP + * Prefix. + */ +enum iommu_hwpt_pgfault_perm { + IOMMU_PGFAULT_PERM_READ =3D (1 << 0), + IOMMU_PGFAULT_PERM_WRITE =3D (1 << 1), + IOMMU_PGFAULT_PERM_EXEC =3D (1 << 2), + IOMMU_PGFAULT_PERM_PRIV =3D (1 << 3), +}; + +/** + * struct iommu_hwpt_pgfault - iommu page fault data + * @flags: Combination of enum iommu_hwpt_pgfault_flags + * @dev_id: id of the originated device + * @pasid: Process Address Space ID + * @grpid: Page Request Group Index + * @perm: Combination of enum iommu_hwpt_pgfault_perm + * @addr: Fault address + * @length: a hint of how much data the requestor is expecting to fetch. F= or + * example, if the PRI initiator knows it is going to do a 10MB + * transfer, it could fill in 10MB and the OS could pre-fault in + * 10MB of IOVA. It's default to 0 if there's no such hint. + * @cookie: kernel-managed cookie identifying a group of fault messages. T= he + * cookie number encoded in the last page fault of the group shou= ld + * be echoed back in the response message. + */ +struct iommu_hwpt_pgfault { + __u32 flags; + __u32 dev_id; + __u32 pasid; + __u32 grpid; + __u32 perm; + __u64 addr; + __u32 length; + __u32 cookie; +}; + +/** + * enum iommufd_page_response_code - Return status of fault handlers + * @IOMMUFD_PAGE_RESP_SUCCESS: Fault has been handled and the page tables + * populated, retry the access. This is the + * "Success" defined in PCI 10.4.2.1. + * @IOMMUFD_PAGE_RESP_INVALID: Could not handle this fault, don't retry the + * access. This is the "Invalid Request" in PCI + * 10.4.2.1. + * @IOMMUFD_PAGE_RESP_FAILURE: General error. Drop all subsequent faults f= rom + * this device if possible. This is the "Respo= nse + * Failure" in PCI 10.4.2.1. + */ +enum iommufd_page_response_code { + IOMMUFD_PAGE_RESP_SUCCESS =3D 0, + IOMMUFD_PAGE_RESP_INVALID, + IOMMUFD_PAGE_RESP_FAILURE, +}; + +/** + * struct iommu_hwpt_page_response - IOMMU page fault response + * @cookie: The kernel-managed cookie reported in the fault message. + * @code: One of response code in enum iommufd_page_response_code. + */ +struct iommu_hwpt_page_response { + __u32 cookie; + __u32 code; +}; #endif --=20 2.34.1 From nobody Tue Dec 16 16:39:05 2025 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0B0C614D2B2 for ; Tue, 2 Jul 2024 06:37:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.10 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719902275; cv=none; b=jHPcse6j6CLQThqBm1pYHXdoR3Fzj0e72HSYLuWvHevyKQJ2n+DXsyBsRwLeLUOQoKYFLltmOmJhYH0ErbixZ81DnFV6dePi4JG7tFAPP9U5Ho6IcmB7GMvy3xeiRJn9r5EhUN4LKvCXgTpuvAIKCJ7RM1X7A2pLSw9w60yG0tE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719902275; c=relaxed/simple; bh=8YNYersmUoR1s6flIO9O0TgZl3l+O3rll+TXDHGAAhc=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=O+QWIJpsqVwAE2CMbwpcIoHoLLCpIJtlE1Ub2kTqAwXsy28rQ77s2p7e6ymGa6gS+RlVQbonNWAs3px/U7wfI2QDfJG27YVVJKp4RYMdLiYhcZpFCkH9YcKk+9E2ZTrQ4RzM3xx5ESnJiSowG4FB22CYRVTB78GtAuU3N67rE9M= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=j5vLha0/; arc=none smtp.client-ip=192.198.163.10 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="j5vLha0/" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1719902274; x=1751438274; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=8YNYersmUoR1s6flIO9O0TgZl3l+O3rll+TXDHGAAhc=; b=j5vLha0/rKdE8nT4ScSfhaSuCbmzQb2Wbel5K8QgJG6tDFwaScghvsJB NGmbC55nyd2vMf7HffKGtcf7CjaNXWN/lwoPj8vkN0JyjXcGDS5QVFePr JXB4DFwZFMoUxSPBZeC6GqTM1I0uaLwU8apYWGIKW3ACk21rs/4Ies4JN 31kHyDz80HFT4S87IbZT6Mt7O2J6lUifkSQ2JpxqhI91ddCJIm6sXDH+w pimTh3djgUd8cSii+GJUx53hXWCJmRGhZq6slBj7PpujBKQ4W1MbM/E9V B/9RluI5aOFXYJtc11VUaF/KSTIiwtTvF13FIbLkY/Jdxeq5ZAiMO7eaR Q==; X-CSE-ConnectionGUID: tRGllxFXRwOkZMMFf/2UBg== X-CSE-MsgGUID: 6Nlqwob4Q3Kd4LJx2grIyA== X-IronPort-AV: E=McAfee;i="6700,10204,11120"; a="28455634" X-IronPort-AV: E=Sophos;i="6.09,178,1716274800"; d="scan'208";a="28455634" Received: from fmviesa003.fm.intel.com ([10.60.135.143]) by fmvoesa104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 01 Jul 2024 23:37:54 -0700 X-CSE-ConnectionGUID: EsR1OtpCTZuVyOgB9OcqJw== X-CSE-MsgGUID: M6HaQndtSv6IG1MiTXwHNA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.09,178,1716274800"; d="scan'208";a="50137248" Received: from unknown (HELO allen-box.sh.intel.com) ([10.239.159.127]) by fmviesa003.fm.intel.com with ESMTP; 01 Jul 2024 23:37:49 -0700 From: Lu Baolu To: Jason Gunthorpe , Kevin Tian , Joerg Roedel , Will Deacon , Robin Murphy , Jean-Philippe Brucker , Nicolin Chen , Yi Liu , Jacob Pan , Joel Granados Cc: iommu@lists.linux.dev, virtualization@lists.linux-foundation.org, linux-kernel@vger.kernel.org, Lu Baolu , Jason Gunthorpe Subject: [PATCH v8 06/10] iommufd: Add iommufd fault object Date: Tue, 2 Jul 2024 14:34:40 +0800 Message-Id: <20240702063444.105814-7-baolu.lu@linux.intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240702063444.105814-1-baolu.lu@linux.intel.com> References: <20240702063444.105814-1-baolu.lu@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" An iommufd fault object provides an interface for delivering I/O page faults to user space. These objects are created and destroyed by user space, and they can be associated with or dissociated from hardware page table objects during page table allocation or destruction. User space interacts with the fault object through a file interface. This interface offers a straightforward and efficient way for user space to handle page faults. It allows user space to read fault messages sequentially and respond to them by writing to the same file. The file interface supports reading messages in poll mode, so it's recommended that user space applications use io_uring to enhance read and write efficiency. A fault object can be associated with any iopf-capable iommufd_hw_pgtable during the pgtable's allocation. All I/O page faults triggered by devices when accessing the I/O addresses of an iommufd_hw_pgtable are routed through the fault object to user space. Similarly, user space's responses to these page faults are routed back to the iommu device driver through the same fault object. Signed-off-by: Lu Baolu Reviewed-by: Jason Gunthorpe Reviewed-by: Kevin Tian --- include/linux/iommu.h | 4 + drivers/iommu/iommufd/iommufd_private.h | 30 ++++ include/uapi/linux/iommufd.h | 18 ++ drivers/iommu/io-pgfault.c | 2 + drivers/iommu/iommufd/fault.c | 226 ++++++++++++++++++++++++ drivers/iommu/iommufd/main.c | 6 + drivers/iommu/iommufd/Makefile | 1 + 7 files changed, 287 insertions(+) create mode 100644 drivers/iommu/iommufd/fault.c diff --git a/include/linux/iommu.h b/include/linux/iommu.h index 910aec80886e..73bc3aee95a1 100644 --- a/include/linux/iommu.h +++ b/include/linux/iommu.h @@ -124,12 +124,16 @@ struct iopf_fault { struct iopf_group { struct iopf_fault last_fault; struct list_head faults; + size_t fault_count; /* list node for iommu_fault_param::faults */ struct list_head pending_node; struct work_struct work; struct iommu_attach_handle *attach_handle; /* The device's fault data parameter. */ struct iommu_fault_param *fault_param; + /* Used by handler provider to hook the group on its own lists. */ + struct list_head node; + u32 cookie; }; =20 /** diff --git a/drivers/iommu/iommufd/iommufd_private.h b/drivers/iommu/iommuf= d/iommufd_private.h index 991f864d1f9b..c8a4519f1405 100644 --- a/drivers/iommu/iommufd/iommufd_private.h +++ b/drivers/iommu/iommufd/iommufd_private.h @@ -128,6 +128,7 @@ enum iommufd_object_type { IOMMUFD_OBJ_HWPT_NESTED, IOMMUFD_OBJ_IOAS, IOMMUFD_OBJ_ACCESS, + IOMMUFD_OBJ_FAULT, #ifdef CONFIG_IOMMUFD_TEST IOMMUFD_OBJ_SELFTEST, #endif @@ -426,6 +427,35 @@ void iopt_remove_access(struct io_pagetable *iopt, u32 iopt_access_list_id); void iommufd_access_destroy_object(struct iommufd_object *obj); =20 +/* + * An iommufd_fault object represents an interface to deliver I/O page fau= lts + * to the user space. These objects are created/destroyed by the user spac= e and + * associated with hardware page table objects during page-table allocatio= n. + */ +struct iommufd_fault { + struct iommufd_object obj; + struct iommufd_ctx *ictx; + struct file *filep; + + /* The lists of outstanding faults protected by below mutex. */ + struct mutex mutex; + struct list_head deliver; + struct xarray response; + + struct wait_queue_head wait_queue; +}; + +struct iommufd_attach_handle { + struct iommu_attach_handle handle; + struct iommufd_device *idev; +}; + +/* Convert an iommu attach handle to iommufd handle. */ +#define to_iommufd_handle(hdl) container_of(hdl, struct iommufd_attach_han= dle, handle) + +int iommufd_fault_alloc(struct iommufd_ucmd *ucmd); +void iommufd_fault_destroy(struct iommufd_object *obj); + #ifdef CONFIG_IOMMUFD_TEST int iommufd_test(struct iommufd_ucmd *ucmd); void iommufd_selftest_destroy(struct iommufd_object *obj); diff --git a/include/uapi/linux/iommufd.h b/include/uapi/linux/iommufd.h index 4d89ed97b533..70b8a38fcd46 100644 --- a/include/uapi/linux/iommufd.h +++ b/include/uapi/linux/iommufd.h @@ -50,6 +50,7 @@ enum { IOMMUFD_CMD_HWPT_SET_DIRTY_TRACKING, IOMMUFD_CMD_HWPT_GET_DIRTY_BITMAP, IOMMUFD_CMD_HWPT_INVALIDATE, + IOMMUFD_CMD_FAULT_QUEUE_ALLOC, }; =20 /** @@ -775,4 +776,21 @@ struct iommu_hwpt_page_response { __u32 cookie; __u32 code; }; + +/** + * struct iommu_fault_alloc - ioctl(IOMMU_FAULT_QUEUE_ALLOC) + * @size: sizeof(struct iommu_fault_alloc) + * @flags: Must be 0 + * @out_fault_id: The ID of the new FAULT + * @out_fault_fd: The fd of the new FAULT + * + * Explicitly allocate a fault handling object. + */ +struct iommu_fault_alloc { + __u32 size; + __u32 flags; + __u32 out_fault_id; + __u32 out_fault_fd; +}; +#define IOMMU_FAULT_QUEUE_ALLOC _IO(IOMMUFD_TYPE, IOMMUFD_CMD_FAULT_QUEUE_= ALLOC) #endif diff --git a/drivers/iommu/io-pgfault.c b/drivers/iommu/io-pgfault.c index 7c9011992d3f..cd679c13752e 100644 --- a/drivers/iommu/io-pgfault.c +++ b/drivers/iommu/io-pgfault.c @@ -110,6 +110,8 @@ static struct iopf_group *iopf_group_alloc(struct iommu= _fault_param *iopf_param, list_add(&group->pending_node, &iopf_param->faults); mutex_unlock(&iopf_param->lock); =20 + group->fault_count =3D list_count_nodes(&group->faults); + return group; } =20 diff --git a/drivers/iommu/iommufd/fault.c b/drivers/iommu/iommufd/fault.c new file mode 100644 index 000000000000..68ff94671d48 --- /dev/null +++ b/drivers/iommu/iommufd/fault.c @@ -0,0 +1,226 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Copyright (C) 2024 Intel Corporation + */ +#define pr_fmt(fmt) "iommufd: " fmt + +#include +#include +#include +#include +#include +#include +#include +#include + +#include "../iommu-priv.h" +#include "iommufd_private.h" + +void iommufd_fault_destroy(struct iommufd_object *obj) +{ + struct iommufd_fault *fault =3D container_of(obj, struct iommufd_fault, o= bj); + struct iopf_group *group, *next; + + /* + * The iommufd object's reference count is zero at this point. + * We can be confident that no other threads are currently + * accessing this pointer. Therefore, acquiring the mutex here + * is unnecessary. + */ + list_for_each_entry_safe(group, next, &fault->deliver, node) { + list_del(&group->node); + iopf_group_response(group, IOMMU_PAGE_RESP_INVALID); + iopf_free_group(group); + } +} + +static void iommufd_compose_fault_message(struct iommu_fault *fault, + struct iommu_hwpt_pgfault *hwpt_fault, + struct iommufd_device *idev, + u32 cookie) +{ + hwpt_fault->flags =3D fault->prm.flags; + hwpt_fault->dev_id =3D idev->obj.id; + hwpt_fault->pasid =3D fault->prm.pasid; + hwpt_fault->grpid =3D fault->prm.grpid; + hwpt_fault->perm =3D fault->prm.perm; + hwpt_fault->addr =3D fault->prm.addr; + hwpt_fault->length =3D 0; + hwpt_fault->cookie =3D cookie; +} + +static ssize_t iommufd_fault_fops_read(struct file *filep, char __user *bu= f, + size_t count, loff_t *ppos) +{ + size_t fault_size =3D sizeof(struct iommu_hwpt_pgfault); + struct iommufd_fault *fault =3D filep->private_data; + struct iommu_hwpt_pgfault data; + struct iommufd_device *idev; + struct iopf_group *group; + struct iopf_fault *iopf; + size_t done =3D 0; + int rc =3D 0; + + if (*ppos || count % fault_size) + return -ESPIPE; + + mutex_lock(&fault->mutex); + while (!list_empty(&fault->deliver) && count > done) { + group =3D list_first_entry(&fault->deliver, + struct iopf_group, node); + + if (group->fault_count * fault_size > count - done) + break; + + rc =3D xa_alloc(&fault->response, &group->cookie, group, + xa_limit_32b, GFP_KERNEL); + if (rc) + break; + + idev =3D to_iommufd_handle(group->attach_handle)->idev; + list_for_each_entry(iopf, &group->faults, list) { + iommufd_compose_fault_message(&iopf->fault, + &data, idev, + group->cookie); + if (copy_to_user(buf + done, &data, fault_size)) { + xa_erase(&fault->response, group->cookie); + rc =3D -EFAULT; + break; + } + done +=3D fault_size; + } + + list_del(&group->node); + } + mutex_unlock(&fault->mutex); + + return done =3D=3D 0 ? rc : done; +} + +static ssize_t iommufd_fault_fops_write(struct file *filep, const char __u= ser *buf, + size_t count, loff_t *ppos) +{ + size_t response_size =3D sizeof(struct iommu_hwpt_page_response); + struct iommufd_fault *fault =3D filep->private_data; + struct iommu_hwpt_page_response response; + struct iopf_group *group; + size_t done =3D 0; + int rc =3D 0; + + if (*ppos || count % response_size) + return -ESPIPE; + + mutex_lock(&fault->mutex); + while (count > done) { + rc =3D copy_from_user(&response, buf + done, response_size); + if (rc) + break; + + group =3D xa_erase(&fault->response, response.cookie); + if (!group) { + rc =3D -EINVAL; + break; + } + + iopf_group_response(group, response.code); + iopf_free_group(group); + done +=3D response_size; + } + mutex_unlock(&fault->mutex); + + return done =3D=3D 0 ? rc : done; +} + +static __poll_t iommufd_fault_fops_poll(struct file *filep, + struct poll_table_struct *wait) +{ + struct iommufd_fault *fault =3D filep->private_data; + __poll_t pollflags =3D EPOLLOUT; + + poll_wait(filep, &fault->wait_queue, wait); + mutex_lock(&fault->mutex); + if (!list_empty(&fault->deliver)) + pollflags |=3D EPOLLIN | EPOLLRDNORM; + mutex_unlock(&fault->mutex); + + return pollflags; +} + +static int iommufd_fault_fops_release(struct inode *inode, struct file *fi= lep) +{ + struct iommufd_fault *fault =3D filep->private_data; + + refcount_dec(&fault->obj.users); + iommufd_ctx_put(fault->ictx); + return 0; +} + +static const struct file_operations iommufd_fault_fops =3D { + .owner =3D THIS_MODULE, + .open =3D nonseekable_open, + .read =3D iommufd_fault_fops_read, + .write =3D iommufd_fault_fops_write, + .poll =3D iommufd_fault_fops_poll, + .release =3D iommufd_fault_fops_release, + .llseek =3D no_llseek, +}; + +int iommufd_fault_alloc(struct iommufd_ucmd *ucmd) +{ + struct iommu_fault_alloc *cmd =3D ucmd->cmd; + struct iommufd_fault *fault; + struct file *filep; + int fdno; + int rc; + + if (cmd->flags) + return -EOPNOTSUPP; + + fault =3D iommufd_object_alloc(ucmd->ictx, fault, IOMMUFD_OBJ_FAULT); + if (IS_ERR(fault)) + return PTR_ERR(fault); + + fault->ictx =3D ucmd->ictx; + INIT_LIST_HEAD(&fault->deliver); + xa_init_flags(&fault->response, XA_FLAGS_ALLOC1); + mutex_init(&fault->mutex); + init_waitqueue_head(&fault->wait_queue); + + filep =3D anon_inode_getfile("[iommufd-pgfault]", &iommufd_fault_fops, + fault, O_RDWR); + if (IS_ERR(filep)) { + rc =3D PTR_ERR(filep); + goto out_abort; + } + + refcount_inc(&fault->obj.users); + iommufd_ctx_get(fault->ictx); + fault->filep =3D filep; + + fdno =3D get_unused_fd_flags(O_CLOEXEC); + if (fdno < 0) { + rc =3D fdno; + goto out_fput; + } + + cmd->out_fault_id =3D fault->obj.id; + cmd->out_fault_fd =3D fdno; + + rc =3D iommufd_ucmd_respond(ucmd, sizeof(*cmd)); + if (rc) + goto out_put_fdno; + iommufd_object_finalize(ucmd->ictx, &fault->obj); + + fd_install(fdno, fault->filep); + + return 0; +out_put_fdno: + put_unused_fd(fdno); +out_fput: + fput(filep); + refcount_dec(&fault->obj.users); + iommufd_ctx_put(fault->ictx); +out_abort: + iommufd_object_abort_and_destroy(ucmd->ictx, &fault->obj); + + return rc; +} diff --git a/drivers/iommu/iommufd/main.c b/drivers/iommu/iommufd/main.c index 39b32932c61e..83bbd7c5d160 100644 --- a/drivers/iommu/iommufd/main.c +++ b/drivers/iommu/iommufd/main.c @@ -319,6 +319,7 @@ static int iommufd_option(struct iommufd_ucmd *ucmd) =20 union ucmd_buffer { struct iommu_destroy destroy; + struct iommu_fault_alloc fault; struct iommu_hw_info info; struct iommu_hwpt_alloc hwpt; struct iommu_hwpt_get_dirty_bitmap get_dirty_bitmap; @@ -355,6 +356,8 @@ struct iommufd_ioctl_op { } static const struct iommufd_ioctl_op iommufd_ioctl_ops[] =3D { IOCTL_OP(IOMMU_DESTROY, iommufd_destroy, struct iommu_destroy, id), + IOCTL_OP(IOMMU_FAULT_QUEUE_ALLOC, iommufd_fault_alloc, struct iommu_fault= _alloc, + out_fault_fd), IOCTL_OP(IOMMU_GET_HW_INFO, iommufd_get_hw_info, struct iommu_hw_info, __reserved), IOCTL_OP(IOMMU_HWPT_ALLOC, iommufd_hwpt_alloc, struct iommu_hwpt_alloc, @@ -513,6 +516,9 @@ static const struct iommufd_object_ops iommufd_object_o= ps[] =3D { .destroy =3D iommufd_hwpt_nested_destroy, .abort =3D iommufd_hwpt_nested_abort, }, + [IOMMUFD_OBJ_FAULT] =3D { + .destroy =3D iommufd_fault_destroy, + }, #ifdef CONFIG_IOMMUFD_TEST [IOMMUFD_OBJ_SELFTEST] =3D { .destroy =3D iommufd_selftest_destroy, diff --git a/drivers/iommu/iommufd/Makefile b/drivers/iommu/iommufd/Makefile index 34b446146961..cf4605962bea 100644 --- a/drivers/iommu/iommufd/Makefile +++ b/drivers/iommu/iommufd/Makefile @@ -1,6 +1,7 @@ # SPDX-License-Identifier: GPL-2.0-only iommufd-y :=3D \ device.o \ + fault.o \ hw_pagetable.o \ io_pagetable.o \ ioas.o \ --=20 2.34.1 From nobody Tue Dec 16 16:39:05 2025 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BC17F14D70E for ; Tue, 2 Jul 2024 06:37:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.10 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719902279; cv=none; b=aOCXGMhKZpTllDK/4D8cY6+HEwKRaeVb1I72CXwzYhmbOx2UrIKQ+7RudIzFWdpg9n1NUDzjT0X3pur95ka88Nkl1NRN7I0WIs8YANATUODjiv0BTo5RmwFaAgjM3hn7UJBQXQoFq732RsiaqwTuntk1otMwKKrw4EdGYsnmU3k= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719902279; c=relaxed/simple; bh=7w9fnJ1pHBsueJ5sOIOy9dsPI+76S3GOS8u5xvy2MNU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=buFBXN3PDSLB+SAQIypYdLPnlsvJB7dkDD1XA3WmfUgGxHPghG8vQgMT3NgkAF2Q20La5mw4RDBShr/A2HIZORaZCeavDZWbQsA8wY6e05c4QOY04YxVYItxH1lJF0+NCdPjGshEAf0VyCIFXKMc/4FhKZPx0Y2ucp0ZwLsuriY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=l5TbFmk1; arc=none smtp.client-ip=192.198.163.10 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="l5TbFmk1" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1719902278; x=1751438278; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=7w9fnJ1pHBsueJ5sOIOy9dsPI+76S3GOS8u5xvy2MNU=; b=l5TbFmk1ImzOfPq1tLgsbB2nu0mETCrePr6bA6vVW++eDRPENm9JenH2 nQji039vWgni7TJwYEX+R2m4+DRR6WIpdNCYRx2dbTkjDfdc/dayCvTKc Y9I/J+eun+dj59EkdcRJc/g/XSNmNos/X6PTJxeJr0zBY1r1dEFLQ/Z7r NdBo8tNte8ZZF1Q6fwqael7S3x/+1ZLpnJLgNouvxuWrYrbgz3dGbYfgi HtDIcS5CP7AN55MpI/foDfenfY1tjvS4kzGohBfVO/uEwe//SyGMbBTJE m7P/QCc4zR0RD08IW/KaywwV4OZERGACm91Dc3fU9Y9vA01WZ1+kGnSeD A==; X-CSE-ConnectionGUID: S5fZiabKQYauPTnWGx2MsA== X-CSE-MsgGUID: Cb9u26gEThypdenfKPEjEg== X-IronPort-AV: E=McAfee;i="6700,10204,11120"; a="28455652" X-IronPort-AV: E=Sophos;i="6.09,178,1716274800"; d="scan'208";a="28455652" Received: from fmviesa003.fm.intel.com ([10.60.135.143]) by fmvoesa104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 01 Jul 2024 23:37:58 -0700 X-CSE-ConnectionGUID: +CnZ7aCPR2Ggo6CdKeB/sQ== X-CSE-MsgGUID: DPrInD/2T7+g5JHDpOnmmg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.09,178,1716274800"; d="scan'208";a="50137266" Received: from unknown (HELO allen-box.sh.intel.com) ([10.239.159.127]) by fmviesa003.fm.intel.com with ESMTP; 01 Jul 2024 23:37:54 -0700 From: Lu Baolu To: Jason Gunthorpe , Kevin Tian , Joerg Roedel , Will Deacon , Robin Murphy , Jean-Philippe Brucker , Nicolin Chen , Yi Liu , Jacob Pan , Joel Granados Cc: iommu@lists.linux.dev, virtualization@lists.linux-foundation.org, linux-kernel@vger.kernel.org, Lu Baolu Subject: [PATCH v8 07/10] iommufd: Fault-capable hwpt attach/detach/replace Date: Tue, 2 Jul 2024 14:34:41 +0800 Message-Id: <20240702063444.105814-8-baolu.lu@linux.intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240702063444.105814-1-baolu.lu@linux.intel.com> References: <20240702063444.105814-1-baolu.lu@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add iopf-capable hw page table attach/detach/replace helpers. The pointer to iommufd_device is stored in the domain attachment handle, so that it can be echo'ed back in the iopf_group. The iopf-capable hw page tables can only be attached to devices that support the IOMMU_DEV_FEAT_IOPF feature. On the first attachment of an iopf-capable hw_pagetable to the device, the IOPF feature is enabled on the device. Similarly, after the last iopf-capable hwpt is detached from the device, the IOPF feature is disabled on the device. The current implementation allows a replacement between iopf-capable and non-iopf-capable hw page tables. This matches the nested translation use case, where a parent domain is attached by default and can then be replaced with a nested user domain with iopf support. Signed-off-by: Lu Baolu Reviewed-by: Kevin Tian Tested-by: Zhangfei Gao --- drivers/iommu/iommufd/iommufd_private.h | 41 +++++ drivers/iommu/iommufd/device.c | 7 +- drivers/iommu/iommufd/fault.c | 190 ++++++++++++++++++++++++ 3 files changed, 235 insertions(+), 3 deletions(-) diff --git a/drivers/iommu/iommufd/iommufd_private.h b/drivers/iommu/iommuf= d/iommufd_private.h index c8a4519f1405..aa4c26c87cb9 100644 --- a/drivers/iommu/iommufd/iommufd_private.h +++ b/drivers/iommu/iommufd/iommufd_private.h @@ -11,6 +11,7 @@ #include #include #include +#include "../iommu-priv.h" =20 struct iommu_domain; struct iommu_group; @@ -293,6 +294,7 @@ int iommufd_check_iova_range(struct io_pagetable *iopt, struct iommufd_hw_pagetable { struct iommufd_object obj; struct iommu_domain *domain; + struct iommufd_fault *fault; }; =20 struct iommufd_hwpt_paging { @@ -396,6 +398,9 @@ struct iommufd_device { /* always the physical device */ struct device *dev; bool enforce_cache_coherency; + /* protect iopf_enabled counter */ + struct mutex iopf_lock; + unsigned int iopf_enabled; }; =20 static inline struct iommufd_device * @@ -456,6 +461,42 @@ struct iommufd_attach_handle { int iommufd_fault_alloc(struct iommufd_ucmd *ucmd); void iommufd_fault_destroy(struct iommufd_object *obj); =20 +int iommufd_fault_domain_attach_dev(struct iommufd_hw_pagetable *hwpt, + struct iommufd_device *idev); +void iommufd_fault_domain_detach_dev(struct iommufd_hw_pagetable *hwpt, + struct iommufd_device *idev); +int iommufd_fault_domain_replace_dev(struct iommufd_device *idev, + struct iommufd_hw_pagetable *hwpt, + struct iommufd_hw_pagetable *old); + +static inline int iommufd_hwpt_attach_device(struct iommufd_hw_pagetable *= hwpt, + struct iommufd_device *idev) +{ + if (hwpt->fault) + return iommufd_fault_domain_attach_dev(hwpt, idev); + + return iommu_attach_group(hwpt->domain, idev->igroup->group); +} + +static inline void iommufd_hwpt_detach_device(struct iommufd_hw_pagetable = *hwpt, + struct iommufd_device *idev) +{ + if (hwpt->fault) + iommufd_fault_domain_detach_dev(hwpt, idev); + + iommu_detach_group(hwpt->domain, idev->igroup->group); +} + +static inline int iommufd_hwpt_replace_device(struct iommufd_device *idev, + struct iommufd_hw_pagetable *hwpt, + struct iommufd_hw_pagetable *old) +{ + if (old->fault || hwpt->fault) + return iommufd_fault_domain_replace_dev(idev, hwpt, old); + + return iommu_group_replace_domain(idev->igroup->group, hwpt->domain); +} + #ifdef CONFIG_IOMMUFD_TEST int iommufd_test(struct iommufd_ucmd *ucmd); void iommufd_selftest_destroy(struct iommufd_object *obj); diff --git a/drivers/iommu/iommufd/device.c b/drivers/iommu/iommufd/device.c index 873630c111c1..9a7ec5997c61 100644 --- a/drivers/iommu/iommufd/device.c +++ b/drivers/iommu/iommufd/device.c @@ -215,6 +215,7 @@ struct iommufd_device *iommufd_device_bind(struct iommu= fd_ctx *ictx, refcount_inc(&idev->obj.users); /* igroup refcount moves into iommufd_device */ idev->igroup =3D igroup; + mutex_init(&idev->iopf_lock); =20 /* * If the caller fails after this success it must call @@ -376,7 +377,7 @@ int iommufd_hw_pagetable_attach(struct iommufd_hw_paget= able *hwpt, * attachment. */ if (list_empty(&idev->igroup->device_list)) { - rc =3D iommu_attach_group(hwpt->domain, idev->igroup->group); + rc =3D iommufd_hwpt_attach_device(hwpt, idev); if (rc) goto err_unresv; idev->igroup->hwpt =3D hwpt; @@ -402,7 +403,7 @@ iommufd_hw_pagetable_detach(struct iommufd_device *idev) mutex_lock(&idev->igroup->lock); list_del(&idev->group_item); if (list_empty(&idev->igroup->device_list)) { - iommu_detach_group(hwpt->domain, idev->igroup->group); + iommufd_hwpt_detach_device(hwpt, idev); idev->igroup->hwpt =3D NULL; } if (hwpt_is_paging(hwpt)) @@ -497,7 +498,7 @@ iommufd_device_do_replace(struct iommufd_device *idev, goto err_unlock; } =20 - rc =3D iommu_group_replace_domain(igroup->group, hwpt->domain); + rc =3D iommufd_hwpt_replace_device(idev, hwpt, old_hwpt); if (rc) goto err_unresv; =20 diff --git a/drivers/iommu/iommufd/fault.c b/drivers/iommu/iommufd/fault.c index 68ff94671d48..4934ae572638 100644 --- a/drivers/iommu/iommufd/fault.c +++ b/drivers/iommu/iommufd/fault.c @@ -8,6 +8,7 @@ #include #include #include +#include #include #include #include @@ -15,6 +16,195 @@ #include "../iommu-priv.h" #include "iommufd_private.h" =20 +static int iommufd_fault_iopf_enable(struct iommufd_device *idev) +{ + struct device *dev =3D idev->dev; + int ret; + + /* + * Once we turn on PCI/PRI support for VF, the response failure code + * should not be forwarded to the hardware due to PRI being a shared + * resource between PF and VFs. There is no coordination for this + * shared capability. This waits for a vPRI reset to recover. + */ + if (dev_is_pci(dev) && to_pci_dev(dev)->is_virtfn) + return -EINVAL; + + mutex_lock(&idev->iopf_lock); + /* Device iopf has already been on. */ + if (++idev->iopf_enabled > 1) { + mutex_unlock(&idev->iopf_lock); + return 0; + } + + ret =3D iommu_dev_enable_feature(dev, IOMMU_DEV_FEAT_IOPF); + if (ret) + --idev->iopf_enabled; + mutex_unlock(&idev->iopf_lock); + + return ret; +} + +static void iommufd_fault_iopf_disable(struct iommufd_device *idev) +{ + mutex_lock(&idev->iopf_lock); + if (!WARN_ON(idev->iopf_enabled =3D=3D 0)) { + if (--idev->iopf_enabled =3D=3D 0) + iommu_dev_disable_feature(idev->dev, IOMMU_DEV_FEAT_IOPF); + } + mutex_unlock(&idev->iopf_lock); +} + +static int __fault_domain_attach_dev(struct iommufd_hw_pagetable *hwpt, + struct iommufd_device *idev) +{ + struct iommufd_attach_handle *handle; + int ret; + + handle =3D kzalloc(sizeof(*handle), GFP_KERNEL); + if (!handle) + return -ENOMEM; + + handle->idev =3D idev; + ret =3D iommu_attach_group_handle(hwpt->domain, idev->igroup->group, + &handle->handle); + if (ret) + kfree(handle); + + return ret; +} + +int iommufd_fault_domain_attach_dev(struct iommufd_hw_pagetable *hwpt, + struct iommufd_device *idev) +{ + int ret; + + if (!hwpt->fault) + return -EINVAL; + + ret =3D iommufd_fault_iopf_enable(idev); + if (ret) + return ret; + + ret =3D __fault_domain_attach_dev(hwpt, idev); + if (ret) + iommufd_fault_iopf_disable(idev); + + return ret; +} + +static void iommufd_auto_response_faults(struct iommufd_hw_pagetable *hwpt, + struct iommufd_attach_handle *handle) +{ + struct iommufd_fault *fault =3D hwpt->fault; + struct iopf_group *group, *next; + unsigned long index; + + if (!fault) + return; + + mutex_lock(&fault->mutex); + list_for_each_entry_safe(group, next, &fault->deliver, node) { + if (group->attach_handle !=3D &handle->handle) + continue; + list_del(&group->node); + iopf_group_response(group, IOMMU_PAGE_RESP_INVALID); + iopf_free_group(group); + } + + xa_for_each(&fault->response, index, group) { + if (group->attach_handle !=3D &handle->handle) + continue; + xa_erase(&fault->response, index); + iopf_group_response(group, IOMMU_PAGE_RESP_INVALID); + iopf_free_group(group); + } + mutex_unlock(&fault->mutex); +} + +static struct iommufd_attach_handle * +iommufd_device_get_attach_handle(struct iommufd_device *idev) +{ + struct iommu_attach_handle *handle; + + handle =3D iommu_attach_handle_get(idev->igroup->group, IOMMU_NO_PASID, 0= ); + if (!handle) + return NULL; + + return to_iommufd_handle(handle); +} + +void iommufd_fault_domain_detach_dev(struct iommufd_hw_pagetable *hwpt, + struct iommufd_device *idev) +{ + struct iommufd_attach_handle *handle; + + handle =3D iommufd_device_get_attach_handle(idev); + iommu_detach_group_handle(hwpt->domain, idev->igroup->group); + iommufd_auto_response_faults(hwpt, handle); + iommufd_fault_iopf_disable(idev); + kfree(handle); +} + +static int __fault_domain_replace_dev(struct iommufd_device *idev, + struct iommufd_hw_pagetable *hwpt, + struct iommufd_hw_pagetable *old) +{ + struct iommufd_attach_handle *handle, *curr =3D NULL; + int ret; + + if (old->fault) + curr =3D iommufd_device_get_attach_handle(idev); + + if (hwpt->fault) { + handle =3D kzalloc(sizeof(*handle), GFP_KERNEL); + if (!handle) + return -ENOMEM; + + handle->handle.domain =3D hwpt->domain; + handle->idev =3D idev; + ret =3D iommu_replace_group_handle(idev->igroup->group, + hwpt->domain, &handle->handle); + } else { + ret =3D iommu_replace_group_handle(idev->igroup->group, + hwpt->domain, NULL); + } + + if (!ret && curr) { + iommufd_auto_response_faults(old, curr); + kfree(curr); + } + + return ret; +} + +int iommufd_fault_domain_replace_dev(struct iommufd_device *idev, + struct iommufd_hw_pagetable *hwpt, + struct iommufd_hw_pagetable *old) +{ + bool iopf_off =3D !hwpt->fault && old->fault; + bool iopf_on =3D hwpt->fault && !old->fault; + int ret; + + if (iopf_on) { + ret =3D iommufd_fault_iopf_enable(idev); + if (ret) + return ret; + } + + ret =3D __fault_domain_replace_dev(idev, hwpt, old); + if (ret) { + if (iopf_on) + iommufd_fault_iopf_disable(idev); + return ret; + } + + if (iopf_off) + iommufd_fault_iopf_disable(idev); + + return 0; +} + void iommufd_fault_destroy(struct iommufd_object *obj) { struct iommufd_fault *fault =3D container_of(obj, struct iommufd_fault, o= bj); --=20 2.34.1 From nobody Tue Dec 16 16:39:05 2025 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6041D14F132 for ; Tue, 2 Jul 2024 06:38:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.10 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719902283; cv=none; b=WOKocW63uuqJGVkUfX5wlw2CyE+EPNDSWEwziX7wo+iAZff+Jp9RmCk21FiYl21P1cxBI2LG73W2sojjRuLSv4RnzGCWQvjwoDwF0f0VOhzixeskf6GPKlBXmXdlk38gr/FmhZB1BMXjY/gEL+mQigJC0SxQhzXUrjSSGC8VZH4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719902283; c=relaxed/simple; bh=TYW4Hxcvr1QiCPoe35bhaALSvHSKSg7HkmF2INs3N84=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=AR5uj1DPCDDhk/mgxacQIHrXbKxlnFNvZUPSrHfMMlzTOYw8c4NqiwakoV0OGE1eMyqNz95DO5dz+E44YZJ5ob1IMDZ1ZkA5SUX2Z+7iRA7l1DKx34TY3MeWGSHR1A4VtmaekUhKL1LcdDlhvN/UyMQIit48022D+vDQiwa3Aew= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=Pr9o75Wn; arc=none smtp.client-ip=192.198.163.10 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="Pr9o75Wn" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1719902282; x=1751438282; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=TYW4Hxcvr1QiCPoe35bhaALSvHSKSg7HkmF2INs3N84=; b=Pr9o75Wn3DvnkebFEUPp35FR/KMiAC2cKAq/zjJgb4tBVQVoJWUv5ixy om3t6FoolnbPRiQihLHnKD1VS/AOYOd06w/CgurE7un6LX8hh1pHAKJSL Qh2Cwz20J2babwwy9VDMYMZHG4jnG7t7ZNojBEXdheEi9Sn8alhcpS2W2 Vyk/NRa0oX5YQ5fHICneIMP0UROMLeJBirfN83PTERbZs713WFu0CyU85 RJOYheWdGPaw3nrGir14aAolIoNiwJwKAyBh6mtVzNwJbKEgj9uuDvZea 2MlpGs7ZvRz8s6hg6xmHIXMdl9LVRQZh8n7zK+zDe2gHqEFzobl/OocTL Q==; X-CSE-ConnectionGUID: m/hGM+glRaWm6gnyLQuT9A== X-CSE-MsgGUID: a2u2EvRXQ+uhQmiRZAkyiQ== X-IronPort-AV: E=McAfee;i="6700,10204,11120"; a="28455670" X-IronPort-AV: E=Sophos;i="6.09,178,1716274800"; d="scan'208";a="28455670" Received: from fmviesa003.fm.intel.com ([10.60.135.143]) by fmvoesa104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 01 Jul 2024 23:38:01 -0700 X-CSE-ConnectionGUID: bmlCrGOmT4W9H+ADM3UqPg== X-CSE-MsgGUID: skIT0fJoQHq/5YnwSmP16w== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.09,178,1716274800"; d="scan'208";a="50137284" Received: from unknown (HELO allen-box.sh.intel.com) ([10.239.159.127]) by fmviesa003.fm.intel.com with ESMTP; 01 Jul 2024 23:37:57 -0700 From: Lu Baolu To: Jason Gunthorpe , Kevin Tian , Joerg Roedel , Will Deacon , Robin Murphy , Jean-Philippe Brucker , Nicolin Chen , Yi Liu , Jacob Pan , Joel Granados Cc: iommu@lists.linux.dev, virtualization@lists.linux-foundation.org, linux-kernel@vger.kernel.org, Lu Baolu Subject: [PATCH v8 08/10] iommufd: Associate fault object with iommufd_hw_pgtable Date: Tue, 2 Jul 2024 14:34:42 +0800 Message-Id: <20240702063444.105814-9-baolu.lu@linux.intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240702063444.105814-1-baolu.lu@linux.intel.com> References: <20240702063444.105814-1-baolu.lu@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" When allocating a user iommufd_hw_pagetable, the user space is allowed to associate a fault object with the hw_pagetable by specifying the fault object ID in the page table allocation data and setting the IOMMU_HWPT_FAULT_ID_VALID flag bit. On a successful return of hwpt allocation, the user can retrieve and respond to page faults by reading and writing the file interface of the fault object. Once a fault object has been associated with a hwpt, the hwpt is iopf-capable, indicated by hwpt->fault is non NULL. Attaching, detaching, or replacing an iopf-capable hwpt to an RID or PASID will differ from those that are not iopf-capable. Signed-off-by: Lu Baolu Reviewed-by: Kevin Tian --- drivers/iommu/iommufd/iommufd_private.h | 9 ++++++ include/uapi/linux/iommufd.h | 8 ++++++ drivers/iommu/iommufd/fault.c | 17 +++++++++++ drivers/iommu/iommufd/hw_pagetable.c | 38 +++++++++++++++++++------ 4 files changed, 64 insertions(+), 8 deletions(-) diff --git a/drivers/iommu/iommufd/iommufd_private.h b/drivers/iommu/iommuf= d/iommufd_private.h index aa4c26c87cb9..92efe30a8f0d 100644 --- a/drivers/iommu/iommufd/iommufd_private.h +++ b/drivers/iommu/iommufd/iommufd_private.h @@ -458,8 +458,17 @@ struct iommufd_attach_handle { /* Convert an iommu attach handle to iommufd handle. */ #define to_iommufd_handle(hdl) container_of(hdl, struct iommufd_attach_han= dle, handle) =20 +static inline struct iommufd_fault * +iommufd_get_fault(struct iommufd_ucmd *ucmd, u32 id) +{ + return container_of(iommufd_get_object(ucmd->ictx, id, + IOMMUFD_OBJ_FAULT), + struct iommufd_fault, obj); +} + int iommufd_fault_alloc(struct iommufd_ucmd *ucmd); void iommufd_fault_destroy(struct iommufd_object *obj); +int iommufd_fault_iopf_handler(struct iopf_group *group); =20 int iommufd_fault_domain_attach_dev(struct iommufd_hw_pagetable *hwpt, struct iommufd_device *idev); diff --git a/include/uapi/linux/iommufd.h b/include/uapi/linux/iommufd.h index 70b8a38fcd46..ede2b464a761 100644 --- a/include/uapi/linux/iommufd.h +++ b/include/uapi/linux/iommufd.h @@ -357,10 +357,13 @@ struct iommu_vfio_ioas { * the parent HWPT in a nesting configurati= on. * @IOMMU_HWPT_ALLOC_DIRTY_TRACKING: Dirty tracking support for device IOM= MU is * enforced on device attachment + * @IOMMU_HWPT_FAULT_ID_VALID: The fault_id field of hwpt allocation data = is + * valid. */ enum iommufd_hwpt_alloc_flags { IOMMU_HWPT_ALLOC_NEST_PARENT =3D 1 << 0, IOMMU_HWPT_ALLOC_DIRTY_TRACKING =3D 1 << 1, + IOMMU_HWPT_FAULT_ID_VALID =3D 1 << 2, }; =20 /** @@ -412,6 +415,9 @@ enum iommu_hwpt_data_type { * @data_type: One of enum iommu_hwpt_data_type * @data_len: Length of the type specific data * @data_uptr: User pointer to the type specific data + * @fault_id: The ID of IOMMUFD_FAULT object. Valid only if flags field of + * IOMMU_HWPT_FAULT_ID_VALID is set. + * @__reserved2: Padding to 64-bit alignment. Must be 0. * * Explicitly allocate a hardware page table object. This is the same obje= ct * type that is returned by iommufd_device_attach() and represents the @@ -442,6 +448,8 @@ struct iommu_hwpt_alloc { __u32 data_type; __u32 data_len; __aligned_u64 data_uptr; + __u32 fault_id; + __u32 __reserved2; }; #define IOMMU_HWPT_ALLOC _IO(IOMMUFD_TYPE, IOMMUFD_CMD_HWPT_ALLOC) =20 diff --git a/drivers/iommu/iommufd/fault.c b/drivers/iommu/iommufd/fault.c index 4934ae572638..54d6cd20a673 100644 --- a/drivers/iommu/iommufd/fault.c +++ b/drivers/iommu/iommufd/fault.c @@ -414,3 +414,20 @@ int iommufd_fault_alloc(struct iommufd_ucmd *ucmd) =20 return rc; } + +int iommufd_fault_iopf_handler(struct iopf_group *group) +{ + struct iommufd_hw_pagetable *hwpt; + struct iommufd_fault *fault; + + hwpt =3D group->attach_handle->domain->fault_data; + fault =3D hwpt->fault; + + mutex_lock(&fault->mutex); + list_add_tail(&group->node, &fault->deliver); + mutex_unlock(&fault->mutex); + + wake_up_interruptible(&fault->wait_queue); + + return 0; +} diff --git a/drivers/iommu/iommufd/hw_pagetable.c b/drivers/iommu/iommufd/h= w_pagetable.c index 33d142f8057d..e63f80f087d1 100644 --- a/drivers/iommu/iommufd/hw_pagetable.c +++ b/drivers/iommu/iommufd/hw_pagetable.c @@ -8,6 +8,15 @@ #include "../iommu-priv.h" #include "iommufd_private.h" =20 +static void __iommufd_hwpt_destroy(struct iommufd_hw_pagetable *hwpt) +{ + if (hwpt->domain) + iommu_domain_free(hwpt->domain); + + if (hwpt->fault) + refcount_dec(&hwpt->fault->obj.users); +} + void iommufd_hwpt_paging_destroy(struct iommufd_object *obj) { struct iommufd_hwpt_paging *hwpt_paging =3D @@ -22,9 +31,7 @@ void iommufd_hwpt_paging_destroy(struct iommufd_object *o= bj) hwpt_paging->common.domain); } =20 - if (hwpt_paging->common.domain) - iommu_domain_free(hwpt_paging->common.domain); - + __iommufd_hwpt_destroy(&hwpt_paging->common); refcount_dec(&hwpt_paging->ioas->obj.users); } =20 @@ -49,9 +56,7 @@ void iommufd_hwpt_nested_destroy(struct iommufd_object *o= bj) struct iommufd_hwpt_nested *hwpt_nested =3D container_of(obj, struct iommufd_hwpt_nested, common.obj); =20 - if (hwpt_nested->common.domain) - iommu_domain_free(hwpt_nested->common.domain); - + __iommufd_hwpt_destroy(&hwpt_nested->common); refcount_dec(&hwpt_nested->parent->common.obj.users); } =20 @@ -213,7 +218,8 @@ iommufd_hwpt_nested_alloc(struct iommufd_ctx *ictx, struct iommufd_hw_pagetable *hwpt; int rc; =20 - if (flags || !user_data->len || !ops->domain_alloc_user) + if ((flags & ~IOMMU_HWPT_FAULT_ID_VALID) || + !user_data->len || !ops->domain_alloc_user) return ERR_PTR(-EOPNOTSUPP); if (parent->auto_domain || !parent->nest_parent) return ERR_PTR(-EINVAL); @@ -227,7 +233,8 @@ iommufd_hwpt_nested_alloc(struct iommufd_ctx *ictx, refcount_inc(&parent->common.obj.users); hwpt_nested->parent =3D parent; =20 - hwpt->domain =3D ops->domain_alloc_user(idev->dev, flags, + hwpt->domain =3D ops->domain_alloc_user(idev->dev, + flags & ~IOMMU_HWPT_FAULT_ID_VALID, parent->common.domain, user_data); if (IS_ERR(hwpt->domain)) { rc =3D PTR_ERR(hwpt->domain); @@ -308,6 +315,21 @@ int iommufd_hwpt_alloc(struct iommufd_ucmd *ucmd) goto out_put_pt; } =20 + if (cmd->flags & IOMMU_HWPT_FAULT_ID_VALID) { + struct iommufd_fault *fault; + + fault =3D iommufd_get_fault(ucmd, cmd->fault_id); + if (IS_ERR(fault)) { + rc =3D PTR_ERR(fault); + goto out_hwpt; + } + hwpt->fault =3D fault; + hwpt->domain->iopf_handler =3D iommufd_fault_iopf_handler; + hwpt->domain->fault_data =3D hwpt; + refcount_inc(&fault->obj.users); + iommufd_put_object(ucmd->ictx, &fault->obj); + } + cmd->out_hwpt_id =3D hwpt->obj.id; rc =3D iommufd_ucmd_respond(ucmd, sizeof(*cmd)); if (rc) --=20 2.34.1 From nobody Tue Dec 16 16:39:05 2025 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E526B14D70E for ; Tue, 2 Jul 2024 06:38:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.10 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719902286; cv=none; b=EAkd4P+QgY9d3YYY55fmY+6YD5y5m5qRpUt5Il7paWv5mMIhUSYL6Mj24VlvH6zc/Rmi6dpmMbhD8E+tWH08Zh4XUBKJi59ZwHDT6GOdNkfSxNRJn8NVlhi7DKi9+Q70UYG6kobuUBb3iX/QdsNtQEEDZyAU7nvNXaTgyal1A6E= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719902286; c=relaxed/simple; bh=UsNgl2+dPtzEBiNUrTHukQdcXrBMGTa4xujfdgev7og=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=XXsuyNYMhOj3lkusfKzjNiWmaZiwKgrOSnPPLJOw5iqnPWGLw0RmzcFze7XjIR7rsMBEFu89Cl1+JotMNHGlTdRclIx+RaE67jHF4awSvMzQZ+mE4gZXPPL616Q4RyuzGeF4kzu4uGcoJDcT6QkHgDzbiIhHeCuUAZwqogvCHOs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=gfDRhv8e; arc=none smtp.client-ip=192.198.163.10 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="gfDRhv8e" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1719902285; x=1751438285; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=UsNgl2+dPtzEBiNUrTHukQdcXrBMGTa4xujfdgev7og=; b=gfDRhv8efDQZVzXF+KZ70s8kAkPlfontj0wst6nQ5zQiYzOSDFnq4DzE DDXIBk1Wn+ubgIyJ04WJWA/zBUZIsrRM9JjhvbGoQHfjiAVH4Z7iO7B1i FigboQw3c4JA/A2cOz+SoomTogD7YyXIrF64kXTI1g1s5WHoPD1WLEk8I cdtEp3YvZdFlugA6ZfURc0jDz+pjtkRajCj7kkBBDrrJ6VTsBMv3AEBxq lo5npGhLFLnrcnx2NpoW37iUZq5mR4NpwXAK1kwTOGACrXlxT3Azfo01Y Oz3S0wz5/wqiOHETyd68Dfq5VtoVaMMGbbed8T0kjOyxx4FAmRRQy2W9p A==; X-CSE-ConnectionGUID: JerpwWo5SC228vr0bfWpVQ== X-CSE-MsgGUID: SfokNartSMifNK51ImFPAw== X-IronPort-AV: E=McAfee;i="6700,10204,11120"; a="28455686" X-IronPort-AV: E=Sophos;i="6.09,178,1716274800"; d="scan'208";a="28455686" Received: from fmviesa003.fm.intel.com ([10.60.135.143]) by fmvoesa104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 01 Jul 2024 23:38:05 -0700 X-CSE-ConnectionGUID: bAVOC67wTfukMupMlZAbEw== X-CSE-MsgGUID: n85bgfLaTJ6yPJjfzFdCuw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.09,178,1716274800"; d="scan'208";a="50137307" Received: from unknown (HELO allen-box.sh.intel.com) ([10.239.159.127]) by fmviesa003.fm.intel.com with ESMTP; 01 Jul 2024 23:38:01 -0700 From: Lu Baolu To: Jason Gunthorpe , Kevin Tian , Joerg Roedel , Will Deacon , Robin Murphy , Jean-Philippe Brucker , Nicolin Chen , Yi Liu , Jacob Pan , Joel Granados Cc: iommu@lists.linux.dev, virtualization@lists.linux-foundation.org, linux-kernel@vger.kernel.org, Lu Baolu Subject: [PATCH v8 09/10] iommufd/selftest: Add IOPF support for mock device Date: Tue, 2 Jul 2024 14:34:43 +0800 Message-Id: <20240702063444.105814-10-baolu.lu@linux.intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240702063444.105814-1-baolu.lu@linux.intel.com> References: <20240702063444.105814-1-baolu.lu@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Extend the selftest mock device to support generating and responding to an IOPF. Also add an ioctl interface to userspace applications to trigger the IOPF on the mock device. This would allow userspace applications to test the IOMMUFD's handling of IOPFs without having to rely on any real hardware. Signed-off-by: Lu Baolu Reviewed-by: Kevin Tian --- drivers/iommu/iommufd/iommufd_test.h | 8 ++++ drivers/iommu/iommufd/selftest.c | 64 ++++++++++++++++++++++++++++ 2 files changed, 72 insertions(+) diff --git a/drivers/iommu/iommufd/iommufd_test.h b/drivers/iommu/iommufd/i= ommufd_test.h index e854d3f67205..acbbba1c6671 100644 --- a/drivers/iommu/iommufd/iommufd_test.h +++ b/drivers/iommu/iommufd/iommufd_test.h @@ -22,6 +22,7 @@ enum { IOMMU_TEST_OP_MOCK_DOMAIN_FLAGS, IOMMU_TEST_OP_DIRTY, IOMMU_TEST_OP_MD_CHECK_IOTLB, + IOMMU_TEST_OP_TRIGGER_IOPF, }; =20 enum { @@ -127,6 +128,13 @@ struct iommu_test_cmd { __u32 id; __u32 iotlb; } check_iotlb; + struct { + __u32 dev_id; + __u32 pasid; + __u32 grpid; + __u32 perm; + __u64 addr; + } trigger_iopf; }; __u32 last; }; diff --git a/drivers/iommu/iommufd/selftest.c b/drivers/iommu/iommufd/selft= est.c index 7a2199470f31..1133f1b2362f 100644 --- a/drivers/iommu/iommufd/selftest.c +++ b/drivers/iommu/iommufd/selftest.c @@ -504,6 +504,8 @@ static bool mock_domain_capable(struct device *dev, enu= m iommu_cap cap) return false; } =20 +static struct iopf_queue *mock_iommu_iopf_queue; + static struct iommu_device mock_iommu_device =3D { }; =20 @@ -514,6 +516,29 @@ static struct iommu_device *mock_probe_device(struct d= evice *dev) return &mock_iommu_device; } =20 +static void mock_domain_page_response(struct device *dev, struct iopf_faul= t *evt, + struct iommu_page_response *msg) +{ +} + +static int mock_dev_enable_feat(struct device *dev, enum iommu_dev_feature= s feat) +{ + if (feat !=3D IOMMU_DEV_FEAT_IOPF || !mock_iommu_iopf_queue) + return -ENODEV; + + return iopf_queue_add_device(mock_iommu_iopf_queue, dev); +} + +static int mock_dev_disable_feat(struct device *dev, enum iommu_dev_featur= es feat) +{ + if (feat !=3D IOMMU_DEV_FEAT_IOPF || !mock_iommu_iopf_queue) + return -ENODEV; + + iopf_queue_remove_device(mock_iommu_iopf_queue, dev); + + return 0; +} + static const struct iommu_ops mock_ops =3D { /* * IOMMU_DOMAIN_BLOCKED cannot be returned from def_domain_type() @@ -529,6 +554,10 @@ static const struct iommu_ops mock_ops =3D { .capable =3D mock_domain_capable, .device_group =3D generic_device_group, .probe_device =3D mock_probe_device, + .page_response =3D mock_domain_page_response, + .dev_enable_feat =3D mock_dev_enable_feat, + .dev_disable_feat =3D mock_dev_disable_feat, + .user_pasid_table =3D true, .default_domain_ops =3D &(struct iommu_domain_ops){ .free =3D mock_domain_free, @@ -1375,6 +1404,31 @@ static int iommufd_test_dirty(struct iommufd_ucmd *u= cmd, unsigned int mockpt_id, return rc; } =20 +static int iommufd_test_trigger_iopf(struct iommufd_ucmd *ucmd, + struct iommu_test_cmd *cmd) +{ + struct iopf_fault event =3D { }; + struct iommufd_device *idev; + + idev =3D iommufd_get_device(ucmd, cmd->trigger_iopf.dev_id); + if (IS_ERR(idev)) + return PTR_ERR(idev); + + event.fault.prm.flags =3D IOMMU_FAULT_PAGE_REQUEST_LAST_PAGE; + if (cmd->trigger_iopf.pasid !=3D IOMMU_NO_PASID) + event.fault.prm.flags |=3D IOMMU_FAULT_PAGE_REQUEST_PASID_VALID; + event.fault.type =3D IOMMU_FAULT_PAGE_REQ; + event.fault.prm.addr =3D cmd->trigger_iopf.addr; + event.fault.prm.pasid =3D cmd->trigger_iopf.pasid; + event.fault.prm.grpid =3D cmd->trigger_iopf.grpid; + event.fault.prm.perm =3D cmd->trigger_iopf.perm; + + iommu_report_device_fault(idev->dev, &event); + iommufd_put_object(ucmd->ictx, &idev->obj); + + return 0; +} + void iommufd_selftest_destroy(struct iommufd_object *obj) { struct selftest_obj *sobj =3D container_of(obj, struct selftest_obj, obj); @@ -1450,6 +1504,8 @@ int iommufd_test(struct iommufd_ucmd *ucmd) cmd->dirty.page_size, u64_to_user_ptr(cmd->dirty.uptr), cmd->dirty.flags); + case IOMMU_TEST_OP_TRIGGER_IOPF: + return iommufd_test_trigger_iopf(ucmd, cmd); default: return -EOPNOTSUPP; } @@ -1491,6 +1547,9 @@ int __init iommufd_test_init(void) &iommufd_mock_bus_type.nb); if (rc) goto err_sysfs; + + mock_iommu_iopf_queue =3D iopf_queue_alloc("mock-iopfq"); + return 0; =20 err_sysfs: @@ -1506,6 +1565,11 @@ int __init iommufd_test_init(void) =20 void iommufd_test_exit(void) { + if (mock_iommu_iopf_queue) { + iopf_queue_free(mock_iommu_iopf_queue); + mock_iommu_iopf_queue =3D NULL; + } + iommu_device_sysfs_remove(&mock_iommu_device); iommu_device_unregister_bus(&mock_iommu_device, &iommufd_mock_bus_type.bus, --=20 2.34.1 From nobody Tue Dec 16 16:39:05 2025 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E193A152179 for ; Tue, 2 Jul 2024 06:38:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.10 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719902290; cv=none; b=Mm0u24gRykpeRDmE0unMpu/WVPPsfvKaypLvHCLouxnJXugpMlw3AlNxv42MpiYdw0pcR4Z77J97VaroBwyRRpO+vcnqiYfe/BECnvvckLI5PBYQ1THS/nbLbn+maNLeSHiFBVyjxIRkbZE0l0EPDzatjCYcBgmobbLqD3WgQbc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719902290; c=relaxed/simple; bh=6jqkIqEZwn4e7Et/J5+bKc0HECtgwLwGNALYtjkl4xI=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Glb6/rGXR6SaU+u6l16NrKbebCYZR2uyDlaZbMEZZrdTFLoqwYVzATkak7cp+xkvYgpYOZoQd+LBcJ6ki3PScA5fOJmULlq3kl18RrkgJHLI/WqZz1fAIerkYvjndldiFMhX1KjKe7LY6tPan6JwC/h/6vFM6rAdJugVMUofaL0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=XdwtA7Oi; arc=none smtp.client-ip=192.198.163.10 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="XdwtA7Oi" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1719902289; x=1751438289; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=6jqkIqEZwn4e7Et/J5+bKc0HECtgwLwGNALYtjkl4xI=; b=XdwtA7OiW623eqTKMwFx5g63S1GH0oWMuakI2gX4d4WbI6XUG0NOuA9c fZtvauaWyoIVxkTf0EB+Txd4qeeAF+KmwN3KRqv4lKsbNmtUE6Jf2BzBW Jok6RcJT9nygqNTGWrOaCfVvaCN5vtcJn5KlVdvbVwdu/RvaRTVHdDbNY 2sgV3idu40Hj1uD5F5JR21Odoz7PS75hdBYAwhKZfH/9SaoHHnmGXrm7N tZ7TVOqIgaaUOegeJkIN23SqTf8Li0dzokLr4g3vn3lbiYRNI7cfVwVx3 V6GuKH+kywEhG4dILVKZ4iaG+Y+aclLp3rOUUSIf8CDKo3Vkzow62zGPS Q==; X-CSE-ConnectionGUID: 284WubZOTMiXyYikE6KXmQ== X-CSE-MsgGUID: kBi9UQu3R1+KvfBgacFKHA== X-IronPort-AV: E=McAfee;i="6700,10204,11120"; a="28455698" X-IronPort-AV: E=Sophos;i="6.09,178,1716274800"; d="scan'208";a="28455698" Received: from fmviesa003.fm.intel.com ([10.60.135.143]) by fmvoesa104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 01 Jul 2024 23:38:09 -0700 X-CSE-ConnectionGUID: qwtQg15+TOGgMMGk4sD4ZA== X-CSE-MsgGUID: gyEaqq1WT4OwGQ6izHg96w== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.09,178,1716274800"; d="scan'208";a="50137337" Received: from unknown (HELO allen-box.sh.intel.com) ([10.239.159.127]) by fmviesa003.fm.intel.com with ESMTP; 01 Jul 2024 23:38:05 -0700 From: Lu Baolu To: Jason Gunthorpe , Kevin Tian , Joerg Roedel , Will Deacon , Robin Murphy , Jean-Philippe Brucker , Nicolin Chen , Yi Liu , Jacob Pan , Joel Granados Cc: iommu@lists.linux.dev, virtualization@lists.linux-foundation.org, linux-kernel@vger.kernel.org, Lu Baolu Subject: [PATCH v8 10/10] iommufd/selftest: Add coverage for IOPF test Date: Tue, 2 Jul 2024 14:34:44 +0800 Message-Id: <20240702063444.105814-11-baolu.lu@linux.intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240702063444.105814-1-baolu.lu@linux.intel.com> References: <20240702063444.105814-1-baolu.lu@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Extend the selftest tool to add coverage of testing IOPF handling. This would include the following tests: - Allocating and destroying an iommufd fault object. - Allocating and destroying an IOPF-capable HWPT. - Attaching/detaching/replacing an IOPF-capable HWPT on a device. - Triggering an IOPF on the mock device. - Retrieving and responding to the IOPF through the file interface. Signed-off-by: Lu Baolu Reviewed-by: Kevin Tian --- tools/testing/selftests/iommu/iommufd_utils.h | 86 +++++++++++++++++-- tools/testing/selftests/iommu/iommufd.c | 22 +++++ .../selftests/iommu/iommufd_fail_nth.c | 2 +- 3 files changed, 104 insertions(+), 6 deletions(-) diff --git a/tools/testing/selftests/iommu/iommufd_utils.h b/tools/testing/= selftests/iommu/iommufd_utils.h index 8d2b46b2114d..a70e35a78584 100644 --- a/tools/testing/selftests/iommu/iommufd_utils.h +++ b/tools/testing/selftests/iommu/iommufd_utils.h @@ -153,7 +153,7 @@ static int _test_cmd_mock_domain_replace(int fd, __u32 = stdev_id, __u32 pt_id, EXPECT_ERRNO(_errno, _test_cmd_mock_domain_replace(self->fd, stdev_id, \ pt_id, NULL)) =20 -static int _test_cmd_hwpt_alloc(int fd, __u32 device_id, __u32 pt_id, +static int _test_cmd_hwpt_alloc(int fd, __u32 device_id, __u32 pt_id, __u3= 2 ft_id, __u32 flags, __u32 *hwpt_id, __u32 data_type, void *data, size_t data_len) { @@ -165,6 +165,7 @@ static int _test_cmd_hwpt_alloc(int fd, __u32 device_id= , __u32 pt_id, .data_type =3D data_type, .data_len =3D data_len, .data_uptr =3D (uint64_t)data, + .fault_id =3D ft_id, }; int ret; =20 @@ -177,24 +178,36 @@ static int _test_cmd_hwpt_alloc(int fd, __u32 device_= id, __u32 pt_id, } =20 #define test_cmd_hwpt_alloc(device_id, pt_id, flags, hwpt_id) = \ - ASSERT_EQ(0, _test_cmd_hwpt_alloc(self->fd, device_id, pt_id, flags, \ + ASSERT_EQ(0, _test_cmd_hwpt_alloc(self->fd, device_id, pt_id, 0, flags, = \ hwpt_id, IOMMU_HWPT_DATA_NONE, NULL, \ 0)) #define test_err_hwpt_alloc(_errno, device_id, pt_id, flags, hwpt_id) \ EXPECT_ERRNO(_errno, _test_cmd_hwpt_alloc( \ - self->fd, device_id, pt_id, flags, \ + self->fd, device_id, pt_id, 0, flags, \ hwpt_id, IOMMU_HWPT_DATA_NONE, NULL, 0)) =20 #define test_cmd_hwpt_alloc_nested(device_id, pt_id, flags, hwpt_id, = \ data_type, data, data_len) \ - ASSERT_EQ(0, _test_cmd_hwpt_alloc(self->fd, device_id, pt_id, flags, \ + ASSERT_EQ(0, _test_cmd_hwpt_alloc(self->fd, device_id, pt_id, 0, flags, \ hwpt_id, data_type, data, data_len)) #define test_err_hwpt_alloc_nested(_errno, device_id, pt_id, flags, hwpt_i= d, \ data_type, data, data_len) \ EXPECT_ERRNO(_errno, \ - _test_cmd_hwpt_alloc(self->fd, device_id, pt_id, flags, \ + _test_cmd_hwpt_alloc(self->fd, device_id, pt_id, 0, flags, \ hwpt_id, data_type, data, data_len)) =20 +#define test_cmd_hwpt_alloc_iopf(device_id, pt_id, fault_id, flags, hwpt_i= d, \ + data_type, data, data_len) \ + ASSERT_EQ(0, _test_cmd_hwpt_alloc(self->fd, device_id, pt_id, fault_id, \ + flags, hwpt_id, data_type, data, \ + data_len)) +#define test_err_hwpt_alloc_iopf(_errno, device_id, pt_id, fault_id, flags= , \ + hwpt_id, data_type, data, data_len) \ + EXPECT_ERRNO(_errno, \ + _test_cmd_hwpt_alloc(self->fd, device_id, pt_id, fault_id, \ + flags, hwpt_id, data_type, data, \ + data_len)) + #define test_cmd_hwpt_check_iotlb(hwpt_id, iotlb_id, expected) = \ ({ \ struct iommu_test_cmd test_cmd =3D { \ @@ -684,3 +697,66 @@ static int _test_cmd_get_hw_info(int fd, __u32 device_= id, void *data, =20 #define test_cmd_get_hw_capabilities(device_id, caps, mask) \ ASSERT_EQ(0, _test_cmd_get_hw_info(self->fd, device_id, NULL, 0, &caps)) + +static int _test_ioctl_fault_alloc(int fd, __u32 *fault_id, __u32 *fault_f= d) +{ + struct iommu_fault_alloc cmd =3D { + .size =3D sizeof(cmd), + }; + int ret; + + ret =3D ioctl(fd, IOMMU_FAULT_QUEUE_ALLOC, &cmd); + if (ret) + return ret; + *fault_id =3D cmd.out_fault_id; + *fault_fd =3D cmd.out_fault_fd; + return 0; +} + +#define test_ioctl_fault_alloc(fault_id, fault_fd) \ + ({ \ + ASSERT_EQ(0, _test_ioctl_fault_alloc(self->fd, fault_id, \ + fault_fd)); \ + ASSERT_NE(0, *(fault_id)); \ + ASSERT_NE(0, *(fault_fd)); \ + }) + +static int _test_cmd_trigger_iopf(int fd, __u32 device_id, __u32 fault_fd) +{ + struct iommu_test_cmd trigger_iopf_cmd =3D { + .size =3D sizeof(trigger_iopf_cmd), + .op =3D IOMMU_TEST_OP_TRIGGER_IOPF, + .trigger_iopf =3D { + .dev_id =3D device_id, + .pasid =3D 0x1, + .grpid =3D 0x2, + .perm =3D IOMMU_PGFAULT_PERM_READ | IOMMU_PGFAULT_PERM_WRITE, + .addr =3D 0xdeadbeaf, + }, + }; + struct iommu_hwpt_page_response response =3D { + .code =3D IOMMUFD_PAGE_RESP_SUCCESS, + }; + struct iommu_hwpt_pgfault fault =3D {}; + ssize_t bytes; + int ret; + + ret =3D ioctl(fd, _IOMMU_TEST_CMD(IOMMU_TEST_OP_TRIGGER_IOPF), &trigger_i= opf_cmd); + if (ret) + return ret; + + bytes =3D read(fault_fd, &fault, sizeof(fault)); + if (bytes <=3D 0) + return -EIO; + + response.cookie =3D fault.cookie; + + bytes =3D write(fault_fd, &response, sizeof(response)); + if (bytes <=3D 0) + return -EIO; + + return 0; +} + +#define test_cmd_trigger_iopf(device_id, fault_fd) \ + ASSERT_EQ(0, _test_cmd_trigger_iopf(self->fd, device_id, fault_fd)) diff --git a/tools/testing/selftests/iommu/iommufd.c b/tools/testing/selfte= sts/iommu/iommufd.c index edf1c99c9936..93634e53e95e 100644 --- a/tools/testing/selftests/iommu/iommufd.c +++ b/tools/testing/selftests/iommu/iommufd.c @@ -279,6 +279,9 @@ TEST_F(iommufd_ioas, alloc_hwpt_nested) uint32_t parent_hwpt_id =3D 0; uint32_t parent_hwpt_id_not_work =3D 0; uint32_t test_hwpt_id =3D 0; + uint32_t iopf_hwpt_id; + uint32_t fault_id; + uint32_t fault_fd; =20 if (self->device_id) { /* Negative tests */ @@ -326,6 +329,7 @@ TEST_F(iommufd_ioas, alloc_hwpt_nested) sizeof(data)); =20 /* Allocate two nested hwpts sharing one common parent hwpt */ + test_ioctl_fault_alloc(&fault_id, &fault_fd); test_cmd_hwpt_alloc_nested(self->device_id, parent_hwpt_id, 0, &nested_hwpt_id[0], IOMMU_HWPT_DATA_SELFTEST, &data, @@ -334,6 +338,14 @@ TEST_F(iommufd_ioas, alloc_hwpt_nested) &nested_hwpt_id[1], IOMMU_HWPT_DATA_SELFTEST, &data, sizeof(data)); + test_err_hwpt_alloc_iopf(ENOENT, self->device_id, parent_hwpt_id, + UINT32_MAX, IOMMU_HWPT_FAULT_ID_VALID, + &iopf_hwpt_id, IOMMU_HWPT_DATA_SELFTEST, + &data, sizeof(data)); + test_cmd_hwpt_alloc_iopf(self->device_id, parent_hwpt_id, fault_id, + IOMMU_HWPT_FAULT_ID_VALID, &iopf_hwpt_id, + IOMMU_HWPT_DATA_SELFTEST, &data, + sizeof(data)); test_cmd_hwpt_check_iotlb_all(nested_hwpt_id[0], IOMMU_TEST_IOTLB_DEFAULT); test_cmd_hwpt_check_iotlb_all(nested_hwpt_id[1], @@ -504,14 +516,24 @@ TEST_F(iommufd_ioas, alloc_hwpt_nested) _test_ioctl_destroy(self->fd, nested_hwpt_id[1])); test_ioctl_destroy(nested_hwpt_id[0]); =20 + /* Switch from nested_hwpt_id[1] to iopf_hwpt_id */ + test_cmd_mock_domain_replace(self->stdev_id, iopf_hwpt_id); + EXPECT_ERRNO(EBUSY, + _test_ioctl_destroy(self->fd, iopf_hwpt_id)); + /* Trigger an IOPF on the device */ + test_cmd_trigger_iopf(self->device_id, fault_fd); + /* Detach from nested_hwpt_id[1] and destroy it */ test_cmd_mock_domain_replace(self->stdev_id, parent_hwpt_id); test_ioctl_destroy(nested_hwpt_id[1]); + test_ioctl_destroy(iopf_hwpt_id); =20 /* Detach from the parent hw_pagetable and destroy it */ test_cmd_mock_domain_replace(self->stdev_id, self->ioas_id); test_ioctl_destroy(parent_hwpt_id); test_ioctl_destroy(parent_hwpt_id_not_work); + close(fault_fd); + test_ioctl_destroy(fault_id); } else { test_err_hwpt_alloc(ENOENT, self->device_id, self->ioas_id, 0, &parent_hwpt_id); diff --git a/tools/testing/selftests/iommu/iommufd_fail_nth.c b/tools/testi= ng/selftests/iommu/iommufd_fail_nth.c index f590417cd67a..c5d5e69452b0 100644 --- a/tools/testing/selftests/iommu/iommufd_fail_nth.c +++ b/tools/testing/selftests/iommu/iommufd_fail_nth.c @@ -615,7 +615,7 @@ TEST_FAIL_NTH(basic_fail_nth, device) if (_test_cmd_get_hw_info(self->fd, idev_id, &info, sizeof(info), NULL)) return -1; =20 - if (_test_cmd_hwpt_alloc(self->fd, idev_id, ioas_id, 0, &hwpt_id, + if (_test_cmd_hwpt_alloc(self->fd, idev_id, ioas_id, 0, 0, &hwpt_id, IOMMU_HWPT_DATA_NONE, 0, 0)) return -1; =20 --=20 2.34.1