From nobody Wed Nov 27 13:06:18 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass header.i=@intel.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=intel.com ARC-Seal: i=1; a=rsa-sha256; t=1699531403; cv=none; d=zohomail.com; s=zohoarc; b=N9f7saiCPhEcas9LeDDqWz5F9sNM7uDuIneDqERThFtxdUNprTe5dg5JZN/XnSXiG4OH5Q4p9TwCqVlLuQHWlGyZEWK3GMpTX3X1Eb5JMZ2MXfBKnu74+kcyI8LSnhoccWvZOHzwoWoZonIIayGNasina7jvxTzWDyq5CSV3HJU= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1699531403; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=oDuXE4L1FN7Eh98Bv1gJs6U2MwTC3YsvLOMPqZdqARQ=; b=JrMKlZdeG6K1G7Wrq/5g7E1zgWkKUJ2zaTf9JAq+RcFe0Qzo1rM+fjmNDuqQz7s4FcG9pXazeXwb+oFqJuZQEjnJz6PNI4DGV01wcnpP/VHyG/fstLfJzVXm2cQyCMqF+PqlFMqQBP46wDEjFy5uzRysuQWEPG6NTC798qDMtO8= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass header.i=@intel.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 169953140375871.66163853684554; Thu, 9 Nov 2023 04:03:23 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1r13jM-0001zG-J7; Thu, 09 Nov 2023 07:01:52 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1r13j9-0001tF-C1 for qemu-devel@nongnu.org; Thu, 09 Nov 2023 07:01:42 -0500 Received: from mgamail.intel.com ([192.55.52.136]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1r13j6-0001ej-Fr for qemu-devel@nongnu.org; Thu, 09 Nov 2023 07:01:39 -0500 Received: from fmviesa001.fm.intel.com ([10.60.135.141]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Nov 2023 04:01:27 -0800 Received: from duan-server-s2600bt.bj.intel.com ([10.240.192.147]) by smtpauth.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Nov 2023 04:01:23 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1699531296; x=1731067296; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=UvojtTk2Tnwz7rhOPFOHVNudkF9K8d5hTFFivDtNhOs=; b=VaH8i04vRCduxQlh/BYR/h3D39mVWtn2kKRSEpGZ3yyAx2BwDXokGnW8 J88Ba8wD5azGGU3Y0gVwwB3Y99/qffNKTdbb0XG0VLxXPmPujSyVO1rhz WGPIjFmKLQ9iirsy7/XXqYM3qLBN13eT8jQ0zfZouQOyi8d7evSCK4xU6 68Vb4WKE5GhkFp4VfMYnBJ95oSCkQpdo+J11aKCr2XdtpTShQhLhxnz/0 s+AQl21mtSyX8h8jqs0ToaaEJoZG3EhAd5tPlwTVkgSrjdEscYCF9q9jw M393ol26aq7UWXjux97md3tcSd7mQZQnd72BBC9pBaQOo9auNBpxaIH9W A==; X-IronPort-AV: E=McAfee;i="6600,9927,10888"; a="369305683" X-IronPort-AV: E=Sophos;i="6.03,289,1694761200"; d="scan'208";a="369305683" X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.03,289,1694761200"; d="scan'208";a="11516016" From: Zhenzhong Duan To: qemu-devel@nongnu.org Cc: alex.williamson@redhat.com, clg@redhat.com, jgg@nvidia.com, nicolinc@nvidia.com, joao.m.martins@oracle.com, eric.auger@redhat.com, peterx@redhat.com, jasowang@redhat.com, kevin.tian@intel.com, yi.l.liu@intel.com, yi.y.sun@intel.com, chao.p.peng@intel.com, Zhenzhong Duan Subject: [PATCH v5 08/20] vfio/iommufd: Enable pci hot reset through iommufd cdev interface Date: Thu, 9 Nov 2023 19:45:17 +0800 Message-Id: <20231109114529.1904193-9-zhenzhong.duan@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231109114529.1904193-1-zhenzhong.duan@intel.com> References: <20231109114529.1904193-1-zhenzhong.duan@intel.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=192.55.52.136; envelope-from=zhenzhong.duan@intel.com; helo=mgamail.intel.com X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @intel.com) X-ZM-MESSAGEID: 1699531405060100001 Content-Type: text/plain; charset="utf-8" Add a new callback iommufd_pci_hot_reset to do iommufd specific check and reset operation. Signed-off-by: Zhenzhong Duan --- hw/vfio/iommufd.c | 142 +++++++++++++++++++++++++++++++++++++++++++ hw/vfio/trace-events | 1 + 2 files changed, 143 insertions(+) diff --git a/hw/vfio/iommufd.c b/hw/vfio/iommufd.c index 958c3e794f..44dc6848bf 100644 --- a/hw/vfio/iommufd.c +++ b/hw/vfio/iommufd.c @@ -24,6 +24,7 @@ #include "sysemu/reset.h" #include "qemu/cutils.h" #include "qemu/chardev_open.h" +#include "pci.h" =20 static int iommufd_map(VFIOContainerBase *bcontainer, hwaddr iova, ram_addr_t size, void *vaddr, bool readonly) @@ -469,9 +470,150 @@ static void iommufd_detach_device(VFIODevice *vbasede= v) close(vbasedev->fd); } =20 +static VFIODevice *vfio_pci_find_by_iommufd_devid(__u32 devid) +{ + VFIODevice *vbasedev_iter; + + QLIST_FOREACH(vbasedev_iter, &vfio_device_list, global_next) { + if (vbasedev_iter->bcontainer->ops !=3D &vfio_iommufd_ops) { + continue; + } + if (devid =3D=3D vbasedev_iter->devid) { + return vbasedev_iter; + } + } + return NULL; +} + +static int iommufd_pci_hot_reset(VFIODevice *vbasedev, bool single) +{ + VFIOPCIDevice *vdev =3D container_of(vbasedev, VFIOPCIDevice, vbasedev= ); + struct vfio_pci_hot_reset_info *info =3D NULL; + struct vfio_pci_dependent_device *devices; + struct vfio_pci_hot_reset *reset; + int ret, i; + bool multi =3D false; + + trace_vfio_pci_hot_reset(vdev->vbasedev.name, single ? "one" : "multi"= ); + + if (!single) { + vfio_pci_pre_reset(vdev); + } + vdev->vbasedev.needs_reset =3D false; + + ret =3D vfio_pci_get_pci_hot_reset_info(vdev, &info); + + if (ret) { + goto out_single; + } + + assert(info->flags & VFIO_PCI_HOT_RESET_FLAG_DEV_ID); + + devices =3D &info->devices[0]; + + if (!(info->flags & VFIO_PCI_HOT_RESET_FLAG_DEV_ID_OWNED)) { + if (!vdev->has_pm_reset) { + for (i =3D 0; i < info->count; i++) { + if (devices[i].devid =3D=3D VFIO_PCI_DEVID_NOT_OWNED) { + error_report("vfio: Cannot reset device %s, " + "depends on device %04x:%02x:%02x.%x " + "which is not owned.", + vdev->vbasedev.name, devices[i].segment, + devices[i].bus, PCI_SLOT(devices[i].devfn= ), + PCI_FUNC(devices[i].devfn)); + } + } + } + ret =3D -EPERM; + goto out_single; + } + + trace_vfio_pci_hot_reset_has_dep_devices(vdev->vbasedev.name); + + for (i =3D 0; i < info->count; i++) { + VFIOPCIDevice *tmp; + VFIODevice *vbasedev_iter; + + trace_vfio_pci_hot_reset_dep_devices_iommufd(devices[i].segment, + devices[i].bus, + PCI_SLOT(devices[i].devfn), + PCI_FUNC(devices[i].devfn), + devices[i].devid); + + /* + * If a VFIO cdev device is resettable, all the dependent devices + * are either bound to same iommufd or within same iommu_groups as + * one of the iommufd bound devices. + */ + assert(devices[i].devid !=3D VFIO_PCI_DEVID_NOT_OWNED); + + if (devices[i].devid =3D=3D vdev->vbasedev.devid || + devices[i].devid =3D=3D VFIO_PCI_DEVID_OWNED) { + continue; + } + + vbasedev_iter =3D vfio_pci_find_by_iommufd_devid(devices[i].devid); + if (!vbasedev_iter || !vbasedev_iter->dev->realized || + vbasedev_iter->type !=3D VFIO_DEVICE_TYPE_PCI) { + continue; + } + tmp =3D container_of(vbasedev_iter, VFIOPCIDevice, vbasedev); + if (single) { + ret =3D -EINVAL; + goto out_single; + } + vfio_pci_pre_reset(tmp); + tmp->vbasedev.needs_reset =3D false; + multi =3D true; + } + + if (!single && !multi) { + ret =3D -EINVAL; + goto out_single; + } + + /* Use zero length array for hot reset with iommufd backend */ + reset =3D g_malloc0(sizeof(*reset)); + reset->argsz =3D sizeof(*reset); + + /* Bus reset! */ + ret =3D ioctl(vdev->vbasedev.fd, VFIO_DEVICE_PCI_HOT_RESET, reset); + g_free(reset); + + trace_vfio_pci_hot_reset_result(vdev->vbasedev.name, + ret ? strerror(errno) : "Success"); + + /* Re-enable INTx on affected devices */ + for (i =3D 0; i < info->count; i++) { + VFIOPCIDevice *tmp; + VFIODevice *vbasedev_iter; + + if (devices[i].devid =3D=3D vdev->vbasedev.devid || + devices[i].devid =3D=3D VFIO_PCI_DEVID_OWNED) { + continue; + } + + vbasedev_iter =3D vfio_pci_find_by_iommufd_devid(devices[i].devid); + if (!vbasedev_iter || !vbasedev_iter->dev->realized || + vbasedev_iter->type !=3D VFIO_DEVICE_TYPE_PCI) { + continue; + } + tmp =3D container_of(vbasedev_iter, VFIOPCIDevice, vbasedev); + vfio_pci_post_reset(tmp); + } +out_single: + if (!single) { + vfio_pci_post_reset(vdev); + } + g_free(info); + + return ret; +} + const VFIOIOMMUOps vfio_iommufd_ops =3D { .dma_map =3D iommufd_map, .dma_unmap =3D iommufd_unmap, .attach_device =3D iommufd_attach_device, .detach_device =3D iommufd_detach_device, + .pci_hot_reset =3D iommufd_pci_hot_reset, }; diff --git a/hw/vfio/trace-events b/hw/vfio/trace-events index 47ae6eede1..a079a2c803 100644 --- a/hw/vfio/trace-events +++ b/hw/vfio/trace-events @@ -34,6 +34,7 @@ vfio_check_af_flr(const char *name) "%s Supports FLR via = AF cap" vfio_pci_hot_reset(const char *name, const char *type) " (%s) %s" vfio_pci_hot_reset_has_dep_devices(const char *name) "%s: hot reset depend= ent devices:" vfio_pci_hot_reset_dep_devices(int domain, int bus, int slot, int function= , int group_id) "\t%04x:%02x:%02x.%x group %d" +vfio_pci_hot_reset_dep_devices_iommufd(int domain, int bus, int slot, int = function, int dev_id) "\t%04x:%02x:%02x.%x devid %d" vfio_pci_hot_reset_result(const char *name, const char *result) "%s hot re= set: %s" vfio_populate_device_config(const char *name, unsigned long size, unsigned= long offset, unsigned long flags) "Device %s config:\n size: 0x%lx, offse= t: 0x%lx, flags: 0x%lx" vfio_populate_device_get_irq_info_failure(const char *errstr) "VFIO_DEVICE= _GET_IRQ_INFO failure: %s" --=20 2.34.1