From nobody Mon Feb 9 23:01:41 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 15536909517151013.833984673003; Wed, 27 Mar 2019 05:49:11 -0700 (PDT) Received: from localhost ([127.0.0.1]:47240 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1h97zQ-0003y8-GD for importer@patchew.org; Wed, 27 Mar 2019 08:49:08 -0400 Received: from eggs.gnu.org ([209.51.188.92]:42314) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1h97p6-000432-La for qemu-devel@nongnu.org; Wed, 27 Mar 2019 08:38:30 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1h97p5-0002cp-0Y for qemu-devel@nongnu.org; Wed, 27 Mar 2019 08:38:28 -0400 Received: from mail-wr1-x433.google.com ([2a00:1450:4864:20::433]:38346) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1h97p4-0002c3-FS for qemu-devel@nongnu.org; Wed, 27 Mar 2019 08:38:26 -0400 Received: by mail-wr1-x433.google.com with SMTP id k11so10978924wro.5 for ; Wed, 27 Mar 2019 05:38:25 -0700 (PDT) Received: from 640k.localdomain ([93.56.166.5]) by smtp.gmail.com with ESMTPSA id c20sm27722863wre.28.2019.03.27.05.38.23 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 27 Mar 2019 05:38:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references; bh=6jxqpoYv/Ah9HXF2rTQPf1hmVPWdTiYAbI+gtoubbo4=; b=U6wlG9UPGVGwKVyjInZik1X93YV5Svqi1LXPrqVGg8khp3+PTz5mhxHwXK8k7YzpWN YeTl+mCjKco4LJUwDYHqDv0KYXXjeHEuZlXBnK75Of65ouXn4C/ELxsnks7+FCWP6ubZ 9pP9wBqwEJf2VPXfHzA1ropZzAFZBIeJLQtxVE14EsY+U5w582S4NB2txl3fdxkbvHrW jq/Q4N4zkqSHCkgeFhQ45hze9/xMpSUgOsX/CamFRzS/Z+wnfkORTdg5qIRPDOqSlU11 8RxL6uks1Ia69mWvgVTvMYV2v9jqemnyOH8Fkbl0phHrNOHGhVn1CZBqHv2xhY0nhxAe 0g7w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :in-reply-to:references; bh=6jxqpoYv/Ah9HXF2rTQPf1hmVPWdTiYAbI+gtoubbo4=; b=uETdMgq1DeIL7OvtFqCaawWZ0KSiKVbI6GwdDJhni9YYCEpnUMP25HbOecUJksusMI xumdbUnSjp1DBJzAGx+JzkPFSD3aI8sSQW+8R/DZER+3n6Y0CERcTpuqpQeCl5vrAXui hlPSQUNKbfdTh3gj4CNcD2Jt7yOJ+os/HOdBiImBNbXYq2K9tfdlJMRQs2jZjKIsw7V6 l427L6Q9jibMuhN6ETbdO5TLw3HODZrMpjLmm78LfGc2Xv5Z6+KBOIse3jNe/1Rjvrba lf+qA/tx3i6PQ4ExcyZaY40BDAo1QXfHy5hA83BSBh9TxhQD8SROmBoPc/eXuT5ZuyGx 70DA== X-Gm-Message-State: APjAAAUA5D/YlG5d0y1jmIP4ge5210K5QQA5CMGY/gCtfQSDhgLn3ENn 6wurucmz7n1L+OtumIp3e5cDkBzI X-Google-Smtp-Source: APXvYqzsTAwjM5aFqukKDnK2yccynihtPbygD7zlXkAw9gepwkn686vvtruZnUvF2vozvKYWnhlNMw== X-Received: by 2002:adf:f803:: with SMTP id s3mr23797297wrp.54.1553690304470; Wed, 27 Mar 2019 05:38:24 -0700 (PDT) From: Paolo Bonzini To: qemu-devel@nongnu.org Date: Wed, 27 Mar 2019 13:37:58 +0100 Message-Id: <1553690281-81854-21-git-send-email-pbonzini@redhat.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1553690281-81854-1-git-send-email-pbonzini@redhat.com> References: <1553690281-81854-1-git-send-email-pbonzini@redhat.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:4864:20::433 Subject: [Qemu-devel] [PULL 20/23] intel-iommu: optimize nodmar memory regions X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Xu Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: Peter Xu Previously we have per-device system memory aliases when DMAR is disabled by the system. It will slow the system down if there are lots of devices especially when DMAR is disabled, because each of the aliased system address space will contain O(N) slots, and rendering such N address spaces will be O(N^2) complexity. This patch introduces a shared nodmar memory region and for each device we only create an alias to the shared memory region. With the aliasing, QEMU memory core API will be able to detect when devices are sharing the same address space (which is the nodmar address space) when rendering the FlatViews and the total number of FlatViews can be dramatically reduced when there are a lot of devices. Suggested-by: Paolo Bonzini Signed-off-by: Peter Xu Message-Id: <20190313094323.18263-1-peterx@redhat.com> Signed-off-by: Paolo Bonzini --- hw/i386/intel_iommu.c | 91 +++++++++++++++++++++++++++------------= ---- include/hw/i386/intel_iommu.h | 7 +++- 2 files changed, 63 insertions(+), 35 deletions(-) diff --git a/hw/i386/intel_iommu.c b/hw/i386/intel_iommu.c index b90de6c..055a1e8 100644 --- a/hw/i386/intel_iommu.c +++ b/hw/i386/intel_iommu.c @@ -1485,11 +1485,11 @@ static bool vtd_switch_address_space(VTDAddressSpac= e *as) =20 /* Turn off first then on the other */ if (use_iommu) { - memory_region_set_enabled(&as->sys_alias, false); + memory_region_set_enabled(&as->nodmar, false); memory_region_set_enabled(MEMORY_REGION(&as->iommu), true); } else { memory_region_set_enabled(MEMORY_REGION(&as->iommu), false); - memory_region_set_enabled(&as->sys_alias, true); + memory_region_set_enabled(&as->nodmar, true); } =20 if (take_bql) { @@ -3286,7 +3286,8 @@ VTDAddressSpace *vtd_find_add_as(IntelIOMMUState *s, = PCIBus *bus, int devfn) vtd_dev_as =3D vtd_bus->dev_as[devfn]; =20 if (!vtd_dev_as) { - snprintf(name, sizeof(name), "intel_iommu_devfn_%d", devfn); + snprintf(name, sizeof(name), "vtd-%02x.%x", PCI_SLOT(devfn), + PCI_FUNC(devfn)); vtd_bus->dev_as[devfn] =3D vtd_dev_as =3D g_malloc0(sizeof(VTDAddr= essSpace)); =20 vtd_dev_as->bus =3D bus; @@ -3295,44 +3296,53 @@ VTDAddressSpace *vtd_find_add_as(IntelIOMMUState *s= , PCIBus *bus, int devfn) vtd_dev_as->context_cache_entry.context_cache_gen =3D 0; vtd_dev_as->iova_tree =3D iova_tree_new(); =20 + memory_region_init(&vtd_dev_as->root, OBJECT(s), name, UINT64_MAX); + address_space_init(&vtd_dev_as->as, &vtd_dev_as->root, "vtd-root"); + /* - * Memory region relationships looks like (Address range shows - * only lower 32 bits to make it short in length...): - * - * |-----------------+-------------------+----------| - * | Name | Address range | Priority | - * |-----------------+-------------------+----------+ - * | vtd_root | 00000000-ffffffff | 0 | - * | intel_iommu | 00000000-ffffffff | 1 | - * | vtd_sys_alias | 00000000-ffffffff | 1 | - * | intel_iommu_ir | fee00000-feefffff | 64 | - * |-----------------+-------------------+----------| + * Build the DMAR-disabled container with aliases to the + * shared MRs. Note that aliasing to a shared memory region + * could help the memory API to detect same FlatViews so we + * can have devices to share the same FlatView when DMAR is + * disabled (either by not providing "intel_iommu=3Don" or with + * "iommu=3Dpt"). It will greatly reduce the total number of + * FlatViews of the system hence VM runs faster. + */ + memory_region_init_alias(&vtd_dev_as->nodmar, OBJECT(s), + "vtd-nodmar", &s->mr_nodmar, 0, + memory_region_size(&s->mr_nodmar)); + + /* + * Build the per-device DMAR-enabled container. * - * We enable/disable DMAR by switching enablement for - * vtd_sys_alias and intel_iommu regions. IR region is always - * enabled. + * TODO: currently we have per-device IOMMU memory region only + * because we have per-device IOMMU notifiers for devices. If + * one day we can abstract the IOMMU notifiers out of the + * memory regions then we can also share the same memory + * region here just like what we've done above with the nodmar + * region. */ + strcat(name, "-dmar"); memory_region_init_iommu(&vtd_dev_as->iommu, sizeof(vtd_dev_as->io= mmu), TYPE_INTEL_IOMMU_MEMORY_REGION, OBJECT(s), - "intel_iommu_dmar", - UINT64_MAX); - memory_region_init_alias(&vtd_dev_as->sys_alias, OBJECT(s), - "vtd_sys_alias", get_system_memory(), - 0, memory_region_size(get_system_memory()= )); - memory_region_init_io(&vtd_dev_as->iommu_ir, OBJECT(s), - &vtd_mem_ir_ops, s, "intel_iommu_ir", - VTD_INTERRUPT_ADDR_SIZE); - memory_region_init(&vtd_dev_as->root, OBJECT(s), - "vtd_root", UINT64_MAX); - memory_region_add_subregion_overlap(&vtd_dev_as->root, + name, UINT64_MAX); + memory_region_init_alias(&vtd_dev_as->iommu_ir, OBJECT(s), "vtd-ir= ", + &s->mr_ir, 0, memory_region_size(&s->mr_i= r)); + memory_region_add_subregion_overlap(MEMORY_REGION(&vtd_dev_as->iom= mu), VTD_INTERRUPT_ADDR_FIRST, - &vtd_dev_as->iommu_ir, 64); - address_space_init(&vtd_dev_as->as, &vtd_dev_as->root, name); - memory_region_add_subregion_overlap(&vtd_dev_as->root, 0, - &vtd_dev_as->sys_alias, 1); + &vtd_dev_as->iommu_ir, 1); + + /* + * Hook both the containers under the root container, we + * switch between DMAR & noDMAR by enable/disable + * corresponding sub-containers + */ memory_region_add_subregion_overlap(&vtd_dev_as->root, 0, MEMORY_REGION(&vtd_dev_as->iom= mu), - 1); + 0); + memory_region_add_subregion_overlap(&vtd_dev_as->root, 0, + &vtd_dev_as->nodmar, 0); + vtd_switch_address_space(vtd_dev_as); } return vtd_dev_as; @@ -3676,6 +3686,21 @@ static void vtd_realize(DeviceState *dev, Error **er= rp) memset(s->vtd_as_by_bus_num, 0, sizeof(s->vtd_as_by_bus_num)); memory_region_init_io(&s->csrmem, OBJECT(s), &vtd_mem_ops, s, "intel_iommu", DMAR_REG_SIZE); + + /* Create the shared memory regions by all devices */ + memory_region_init(&s->mr_nodmar, OBJECT(s), "vtd-nodmar", + UINT64_MAX); + memory_region_init_io(&s->mr_ir, OBJECT(s), &vtd_mem_ir_ops, + s, "vtd-ir", VTD_INTERRUPT_ADDR_SIZE); + memory_region_init_alias(&s->mr_sys_alias, OBJECT(s), + "vtd-sys-alias", get_system_memory(), 0, + memory_region_size(get_system_memory())); + memory_region_add_subregion_overlap(&s->mr_nodmar, 0, + &s->mr_sys_alias, 0); + memory_region_add_subregion_overlap(&s->mr_nodmar, + VTD_INTERRUPT_ADDR_FIRST, + &s->mr_ir, 1); + sysbus_init_mmio(SYS_BUS_DEVICE(s), &s->csrmem); /* No corresponding destroy */ s->iotlb =3D g_hash_table_new_full(vtd_uint64_hash, vtd_uint64_equal, diff --git a/include/hw/i386/intel_iommu.h b/include/hw/i386/intel_iommu.h index c11e3d5..536f626 100644 --- a/include/hw/i386/intel_iommu.h +++ b/include/hw/i386/intel_iommu.h @@ -105,8 +105,8 @@ struct VTDAddressSpace { uint8_t devfn; AddressSpace as; IOMMUMemoryRegion iommu; - MemoryRegion root; - MemoryRegion sys_alias; + MemoryRegion root; /* The root container of the device */ + MemoryRegion nodmar; /* The alias of shared nodmar MR */ MemoryRegion iommu_ir; /* Interrupt region: 0xfeeXXXXX */ IntelIOMMUState *iommu_state; VTDContextCacheEntry context_cache_entry; @@ -221,6 +221,9 @@ union VTD_IR_MSIAddress { struct IntelIOMMUState { X86IOMMUState x86_iommu; MemoryRegion csrmem; + MemoryRegion mr_nodmar; + MemoryRegion mr_ir; + MemoryRegion mr_sys_alias; uint8_t csr[DMAR_REG_SIZE]; /* register values */ uint8_t wmask[DMAR_REG_SIZE]; /* R/W bytes */ uint8_t w1cmask[DMAR_REG_SIZE]; /* RW1C(Write 1 to Clear) bytes */ --=20 1.8.3.1