From nobody Mon May 6 15:29:55 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1490692074947858.5290486521334; Tue, 28 Mar 2017 02:07:54 -0700 (PDT) Received: from localhost ([::1]:51845 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1csn6X-00009p-MA for importer@patchew.org; Tue, 28 Mar 2017 05:07:53 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:44844) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1csn4s-0007QJ-Jb for qemu-devel@nongnu.org; Tue, 28 Mar 2017 05:06:14 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1csn4o-0003y3-OR for qemu-devel@nongnu.org; Tue, 28 Mar 2017 05:06:10 -0400 Received: from ozlabs.ru ([107.173.13.209]:51570) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1csn4o-0003x6-IA; Tue, 28 Mar 2017 05:06:06 -0400 Received: from vpl2.ozlabs.ibm.com (localhost [IPv6:::1]) by ozlabs.ru (Postfix) with ESMTP id 2505F3A602E8; Tue, 28 Mar 2017 05:05:39 -0400 (EDT) From: Alexey Kardashevskiy To: qemu-devel@nongnu.org Date: Tue, 28 Mar 2017 20:05:28 +1100 Message-Id: <20170328090530.20052-2-aik@ozlabs.ru> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20170328090530.20052-1-aik@ozlabs.ru> References: <20170328090530.20052-1-aik@ozlabs.ru> X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [fuzzy] X-Received-From: 107.173.13.209 Subject: [Qemu-devel] [RFC PATCH qemu 1/3] memory: Add get_fd() hook for IOMMU MR X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Alexey Kardashevskiy , Alex Williamson , qemu-ppc@nongnu.org, David Gibson Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Signed-off-by: Alexey Kardashevskiy --- include/exec/memory.h | 2 ++ hw/ppc/spapr_iommu.c | 8 ++++++++ 2 files changed, 10 insertions(+) diff --git a/include/exec/memory.h b/include/exec/memory.h index e39256ad03..925c10b35b 100644 --- a/include/exec/memory.h +++ b/include/exec/memory.h @@ -174,6 +174,8 @@ struct MemoryRegionIOMMUOps { void (*notify_flag_changed)(MemoryRegion *iommu, IOMMUNotifierFlag old_flags, IOMMUNotifierFlag new_flags); + /* Returns a kernel fd for IOMMU */ + int (*get_fd)(MemoryRegion *iommu); }; =20 typedef struct CoalescedMemoryRange CoalescedMemoryRange; diff --git a/hw/ppc/spapr_iommu.c b/hw/ppc/spapr_iommu.c index 9e30e148d6..b61c8f053e 100644 --- a/hw/ppc/spapr_iommu.c +++ b/hw/ppc/spapr_iommu.c @@ -170,6 +170,13 @@ static void spapr_tce_notify_flag_changed(MemoryRegion= *iommu, } } =20 +static int spapr_tce_get_fd(MemoryRegion *iommu) +{ + sPAPRTCETable *tcet =3D container_of(iommu, sPAPRTCETable, iommu); + + return tcet->fd; +} + static int spapr_tce_table_post_load(void *opaque, int version_id) { sPAPRTCETable *tcet =3D SPAPR_TCE_TABLE(opaque); @@ -251,6 +258,7 @@ static MemoryRegionIOMMUOps spapr_iommu_ops =3D { .translate =3D spapr_tce_translate_iommu, .get_min_page_size =3D spapr_tce_get_min_page_size, .notify_flag_changed =3D spapr_tce_notify_flag_changed, + .get_fd =3D spapr_tce_get_fd, }; =20 static int spapr_tce_table_realize(DeviceState *dev) --=20 2.11.0 From nobody Mon May 6 15:29:55 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 149069202399110.404199114650737; Tue, 28 Mar 2017 02:07:03 -0700 (PDT) Received: from localhost ([::1]:51843 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1csn5i-0007rs-Ao for importer@patchew.org; Tue, 28 Mar 2017 05:07:02 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:44679) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1csn4R-0007CX-Pu for qemu-devel@nongnu.org; Tue, 28 Mar 2017 05:05:44 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1csn4N-0003no-5E for qemu-devel@nongnu.org; Tue, 28 Mar 2017 05:05:43 -0400 Received: from ozlabs.ru ([107.173.13.209]:49106) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1csn4N-0003nO-0O; Tue, 28 Mar 2017 05:05:39 -0400 Received: from vpl2.ozlabs.ibm.com (localhost [IPv6:::1]) by ozlabs.ru (Postfix) with ESMTP id CA0E63A6053B; Tue, 28 Mar 2017 05:05:41 -0400 (EDT) From: Alexey Kardashevskiy To: qemu-devel@nongnu.org Date: Tue, 28 Mar 2017 20:05:29 +1100 Message-Id: <20170328090530.20052-3-aik@ozlabs.ru> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20170328090530.20052-1-aik@ozlabs.ru> References: <20170328090530.20052-1-aik@ozlabs.ru> X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [fuzzy] X-Received-From: 107.173.13.209 Subject: [Qemu-devel] [RFC PATCH qemu 2/3] vfio-pci: Reorder group-to-container attaching X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Alexey Kardashevskiy , Alex Williamson , qemu-ppc@nongnu.org, David Gibson Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" At the moment VFIO PCI device initialization works as follows: vfio_realize vfio_get_group vfio_connect_container register memory listeners (1) update QEMU groups lists vfio_kvm_device_add_group Then (example for pseries) the machine reset hook triggers region_add() for all regions where listeners from (1) are listening: ppc_spapr_reset spapr_phb_reset spapr_tce_table_enable memory_region_add_subregion vfio_listener_region_add vfio_spapr_create_window This scheme works fine until we need to handle VFIO PCI device hotplug _and_ we want to enable in-kernel acceleration on, i.e. after PCI hotplug we need a place to call ioctl(vfio_kvm_device_fd, KVM_DEV_VFIO_GROUP_SET_SPAPR_TCE). Since the ioctl needs a LIOBN fd (from sPAPRTCETable) and a IOMMU group fd (from VFIOGroup), vfio_listener_region_add() seems to be the only place for this ioctl(). However this only works during boot time because the machine reset happens strictly after all devices are finalized. When hotplug happens, vfio_listener_region_add() is called when a memory listener is registered but when this happens: 1. new group is not added to the container->group_list yet; 2. VFIO KVM device is unaware of the new IOMMU group. This moves bits around to have all necessary VFIO infrastructure in place for both initial startup and hotplug cases. Signed-off-by: Alexey Kardashevskiy --- hw/vfio/common.c | 21 +++++++++++---------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/hw/vfio/common.c b/hw/vfio/common.c index f3ba9b9007..c75c7594d5 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -1086,6 +1086,16 @@ static int vfio_connect_container(VFIOGroup *group, = AddressSpace *as, goto free_container_exit; } =20 + container->initialized =3D true; + + vfio_kvm_device_add_group(group); + + QLIST_INIT(&container->group_list); + QLIST_INSERT_HEAD(&space->containers, container, next); + + group->container =3D container; + QLIST_INSERT_HEAD(&container->group_list, group, container_next); + container->listener =3D vfio_memory_listener; =20 memory_listener_register(&container->listener, container->space->as); @@ -1097,16 +1107,9 @@ static int vfio_connect_container(VFIOGroup *group, = AddressSpace *as, goto listener_release_exit; } =20 - container->initialized =3D true; - - QLIST_INIT(&container->group_list); - QLIST_INSERT_HEAD(&space->containers, container, next); - - group->container =3D container; - QLIST_INSERT_HEAD(&container->group_list, group, container_next); - return 0; listener_release_exit: + vfio_kvm_device_del_group(group); vfio_listener_release(container); =20 free_container_exit: @@ -1210,8 +1213,6 @@ VFIOGroup *vfio_get_group(int groupid, AddressSpace *= as, Error **errp) =20 QLIST_INSERT_HEAD(&vfio_group_list, group, next); =20 - vfio_kvm_device_add_group(group); - return group; =20 close_fd_exit: --=20 2.11.0 From nobody Mon May 6 15:29:55 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1490692088771437.8632170252798; Tue, 28 Mar 2017 02:08:08 -0700 (PDT) Received: from localhost ([::1]:51846 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1csn6l-0000Np-Fm for importer@patchew.org; Tue, 28 Mar 2017 05:08:07 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:44861) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1csn4t-0007Qt-I2 for qemu-devel@nongnu.org; Tue, 28 Mar 2017 05:06:13 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1csn4s-00041k-Am for qemu-devel@nongnu.org; Tue, 28 Mar 2017 05:06:11 -0400 Received: from ozlabs.ru ([107.173.13.209]:51866) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1csn4s-00041F-2K; Tue, 28 Mar 2017 05:06:10 -0400 Received: from vpl2.ozlabs.ibm.com (localhost [IPv6:::1]) by ozlabs.ru (Postfix) with ESMTP id 7B2493A6053C; Tue, 28 Mar 2017 05:05:43 -0400 (EDT) From: Alexey Kardashevskiy To: qemu-devel@nongnu.org Date: Tue, 28 Mar 2017 20:05:30 +1100 Message-Id: <20170328090530.20052-4-aik@ozlabs.ru> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20170328090530.20052-1-aik@ozlabs.ru> References: <20170328090530.20052-1-aik@ozlabs.ru> X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [fuzzy] X-Received-From: 107.173.13.209 Subject: [Qemu-devel] [RFC PATCH qemu 3/3] vfio: Enable in-kernel acceleration via VFIO KVM device X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Alexey Kardashevskiy , Alex Williamson , qemu-ppc@nongnu.org, David Gibson Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" This enables in-kernel acceleration of TCE update requests via VFIO KVM device. Signed-off-by: Alexey Kardashevskiy --- include/hw/vfio/vfio-common.h | 1 + target/ppc/kvm_ppc.h | 6 ++++++ hw/ppc/spapr_iommu.c | 4 ++++ hw/vfio/common.c | 13 +++++++++++++ hw/vfio/spapr.c | 26 ++++++++++++++++++++++++++ target/ppc/kvm.c | 7 ++++++- hw/vfio/trace-events | 1 + 7 files changed, 57 insertions(+), 1 deletion(-) diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index c582de18c9..ee8c96cc4a 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -175,6 +175,7 @@ extern const MemoryListener vfio_prereg_listener; int vfio_spapr_create_window(VFIOContainer *container, MemoryRegionSection *section, hwaddr *pgsize); +int vfio_spapr_notify_kvm(int vfio_kvm_device_fd, int groupfd, int tablefd= ); int vfio_spapr_remove_window(VFIOContainer *container, hwaddr offset_within_address_space); =20 diff --git a/target/ppc/kvm_ppc.h b/target/ppc/kvm_ppc.h index f48243d13f..ce7327a4e0 100644 --- a/target/ppc/kvm_ppc.h +++ b/target/ppc/kvm_ppc.h @@ -46,6 +46,7 @@ void *kvmppc_create_spapr_tce(uint32_t liobn, uint32_t pa= ge_shift, int kvmppc_remove_spapr_tce(void *table, int pfd, uint32_t window_size); int kvmppc_reset_htab(int shift_hint); uint64_t kvmppc_rma_size(uint64_t current_size, unsigned int hash_shift); +bool kvmppc_has_cap_spapr_vfio(void); #endif /* !CONFIG_USER_ONLY */ bool kvmppc_has_cap_epr(void); int kvmppc_define_rtas_kernel_token(uint32_t token, const char *function); @@ -216,6 +217,11 @@ static inline bool kvmppc_is_mem_backend_page_size_ok(= char *obj_path) return true; } =20 +static inline bool kvmppc_has_cap_spapr_vfio(void) +{ + return false; +} + #endif /* !CONFIG_USER_ONLY */ =20 static inline bool kvmppc_has_cap_epr(void) diff --git a/hw/ppc/spapr_iommu.c b/hw/ppc/spapr_iommu.c index b61c8f053e..fc23d81645 100644 --- a/hw/ppc/spapr_iommu.c +++ b/hw/ppc/spapr_iommu.c @@ -293,6 +293,10 @@ void spapr_tce_set_need_vfio(sPAPRTCETable *tcet, bool= need_vfio) =20 tcet->need_vfio =3D need_vfio; =20 + if (!need_vfio || (tcet->fd !=3D -1 && kvmppc_has_cap_spapr_vfio())) { + return; + } + oldtable =3D tcet->table; =20 tcet->table =3D spapr_tce_alloc_table(tcet->liobn, diff --git a/hw/vfio/common.c b/hw/vfio/common.c index c75c7594d5..9aaf861904 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -440,6 +440,19 @@ static void vfio_listener_region_add(MemoryListener *l= istener, goto fail; } =20 +#ifdef CONFIG_KVM + if (kvm_enabled() && section->mr->iommu_ops->get_fd) { + VFIOGroup *group; + int tablefd =3D section->mr->iommu_ops->get_fd(section->mr); + + if (tablefd !=3D -1) { + QLIST_FOREACH(group, &container->group_list, container_nex= t) { + vfio_spapr_notify_kvm(vfio_kvm_device_fd, + group->fd, tablefd); + } + } + } +#endif vfio_host_win_add(container, section->offset_within_address_space, section->offset_within_address_space + int128_get64(section->size) - 1, pgsize); diff --git a/hw/vfio/spapr.c b/hw/vfio/spapr.c index 4409bcc0d7..dffef3bd5f 100644 --- a/hw/vfio/spapr.c +++ b/hw/vfio/spapr.c @@ -17,6 +17,9 @@ #include "hw/hw.h" #include "qemu/error-report.h" #include "trace.h" +#ifdef CONFIG_KVM +#include "linux/kvm.h" +#endif =20 static bool vfio_prereg_listener_skipped_section(MemoryRegionSection *sect= ion) { @@ -187,6 +190,29 @@ int vfio_spapr_create_window(VFIOContainer *container, return 0; } =20 +int vfio_spapr_notify_kvm(int vfio_kvm_device_fd, int groupfd, int tablefd) +{ +#ifdef CONFIG_KVM + struct kvm_vfio_spapr_tce param =3D { + .groupfd =3D groupfd, + .tablefd =3D tablefd + }; + struct kvm_device_attr attr =3D { + .group =3D KVM_DEV_VFIO_GROUP, + .attr =3D KVM_DEV_VFIO_GROUP_SET_SPAPR_TCE, + .addr =3D (uint64_t)(unsigned long)¶m, + }; + + if (ioctl(vfio_kvm_device_fd, KVM_SET_DEVICE_ATTR, &attr)) { + error_report("vfio: failed to setup fd %d for a group with fd %d: = %s", + param.tablefd, param.groupfd, strerror(errno)); + return -errno; + } + trace_vfio_spapr_notify_kvm(groupfd, tablefd); +#endif + return 0; +} + int vfio_spapr_remove_window(VFIOContainer *container, hwaddr offset_within_address_space) { diff --git a/target/ppc/kvm.c b/target/ppc/kvm.c index 560ce655c7..bca5fe7329 100644 --- a/target/ppc/kvm.c +++ b/target/ppc/kvm.c @@ -131,7 +131,7 @@ int kvm_arch_init(MachineState *ms, KVMState *s) cap_spapr_tce =3D kvm_check_extension(s, KVM_CAP_SPAPR_TCE); cap_spapr_tce_64 =3D kvm_check_extension(s, KVM_CAP_SPAPR_TCE_64); cap_spapr_multitce =3D kvm_check_extension(s, KVM_CAP_SPAPR_MULTITCE); - cap_spapr_vfio =3D false; + cap_spapr_vfio =3D kvm_check_extension(s, KVM_CAP_SPAPR_TCE_VFIO); cap_one_reg =3D kvm_check_extension(s, KVM_CAP_ONE_REG); cap_hior =3D kvm_check_extension(s, KVM_CAP_PPC_HIOR); cap_epr =3D kvm_check_extension(s, KVM_CAP_PPC_EPR); @@ -2416,6 +2416,11 @@ bool kvmppc_has_cap_mmu_hash_v3(void) return cap_mmu_hash_v3; } =20 +bool kvmppc_has_cap_spapr_vfio(void) +{ + return cap_spapr_vfio; +} + static PowerPCCPUClass *ppc_cpu_get_family_class(PowerPCCPUClass *pcc) { ObjectClass *oc =3D OBJECT_CLASS(pcc); diff --git a/hw/vfio/trace-events b/hw/vfio/trace-events index 2561c6d31a..084a92f7c2 100644 --- a/hw/vfio/trace-events +++ b/hw/vfio/trace-events @@ -123,3 +123,4 @@ vfio_prereg_register(uint64_t va, uint64_t size, int re= t) "va=3D%"PRIx64" size=3D%"P vfio_prereg_unregister(uint64_t va, uint64_t size, int ret) "va=3D%"PRIx64= " size=3D%"PRIx64" ret=3D%d" vfio_spapr_create_window(int ps, uint64_t ws, uint64_t off) "pageshift=3D0= x%x winsize=3D0x%"PRIx64" offset=3D0x%"PRIx64 vfio_spapr_remove_window(uint64_t off) "offset=3D%"PRIx64 +vfio_spapr_notify_kvm(int groupfd, int tablefd) "Attached groupfd %d to li= obn fd %d" --=20 2.11.0