From nobody Mon Feb 9 12:25:51 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1552386051176102.578886241958; Tue, 12 Mar 2019 03:20:51 -0700 (PDT) Received: from localhost ([127.0.0.1]:48969 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1h3eWW-0008GT-39 for importer@patchew.org; Tue, 12 Mar 2019 06:20:40 -0400 Received: from eggs.gnu.org ([209.51.188.92]:52411) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1h3dUI-0007ay-Ug for qemu-devel@nongnu.org; Tue, 12 Mar 2019 05:14:21 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1h3dF7-0003nr-1D for qemu-devel@nongnu.org; Tue, 12 Mar 2019 04:58:38 -0400 Received: from mx1.redhat.com ([209.132.183.28]:37812) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1h3dF6-0003n9-Pn; Tue, 12 Mar 2019 04:58:36 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 85E273092643; Tue, 12 Mar 2019 08:53:24 +0000 (UTC) Received: from umbus.redhat.com (vpn2-54-33.bne.redhat.com [10.64.54.33]) by smtp.corp.redhat.com (Postfix) with ESMTP id 2B07B60162; Tue, 12 Mar 2019 08:53:21 +0000 (UTC) From: David Gibson To: peter.maydell@linaro.org Date: Tue, 12 Mar 2019 19:52:15 +1100 Message-Id: <20190312085316.8054-2-dgibson@redhat.com> In-Reply-To: <20190312085316.8054-1-dgibson@redhat.com> References: <20190312085316.8054-1-dgibson@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.43]); Tue, 12 Mar 2019 08:53:24 +0000 (UTC) Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PULL 01/62] vfio/spapr: Fix indirect levels calculation X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lvivier@redhat.com, clg@kaod.org, qemu-ppc@nongnu.org, qemu-devel@nongnu.org, groug@kaod.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Type: text/plain; charset="utf-8" From: Alexey Kardashevskiy The current code assumes that we can address more bits on a PCI bus for DMA than we really can but there is no way knowing the actual limit. This makes a better guess for the number of levels and if the kernel fails to allocate that, this increases the level numbers till succeeded or reached the 64bit limit. This adds levels to the trace point. This may cause the kernel to warn about failed allocation: [65122.837458] Failed to allocate a TCE memory, level shift=3D28 which might happen if MAX_ORDER is not large enough as it can vary: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arc= h/powerpc/Kconfig?h=3Dv5.0-rc2#n727 Signed-off-by: Alexey Kardashevskiy Message-Id: <20190227085149.38596-3-aik@ozlabs.ru> Signed-off-by: David Gibson --- hw/vfio/spapr.c | 43 +++++++++++++++++++++++++++++++++---------- hw/vfio/trace-events | 2 +- 2 files changed, 34 insertions(+), 11 deletions(-) diff --git a/hw/vfio/spapr.c b/hw/vfio/spapr.c index becf71a3fc..88437a79e6 100644 --- a/hw/vfio/spapr.c +++ b/hw/vfio/spapr.c @@ -143,10 +143,10 @@ int vfio_spapr_create_window(VFIOContainer *container, MemoryRegionSection *section, hwaddr *pgsize) { - int ret; + int ret =3D 0; IOMMUMemoryRegion *iommu_mr =3D IOMMU_MEMORY_REGION(section->mr); uint64_t pagesize =3D memory_region_iommu_get_min_page_size(iommu_mr); - unsigned entries, pages; + unsigned entries, bits_total, bits_per_level, max_levels; struct vfio_iommu_spapr_tce_create create =3D { .argsz =3D sizeof(crea= te) }; long systempagesize =3D qemu_getrampagesize(); =20 @@ -176,16 +176,38 @@ int vfio_spapr_create_window(VFIOContainer *container, create.window_size =3D int128_get64(section->size); create.page_shift =3D ctz64(pagesize); /* - * SPAPR host supports multilevel TCE tables, there is some - * heuristic to decide how many levels we want for our table: - * 0..64 =3D 1; 65..4096 =3D 2; 4097..262144 =3D 3; 262145.. =3D 4 + * SPAPR host supports multilevel TCE tables. We try to guess optimal + * levels number and if this fails (for example due to the host memory + * fragmentation), we increase levels. The DMA address structure is: + * rrrrrrrr rxxxxxxx xxxxxxxx xxxxxxxx xxxxxxxx xxxxxxxx xxxxxxxx iii= iiiii + * where: + * r =3D reserved (bits >=3D 55 are reserved in the existing hardwar= e) + * i =3D IOMMU page offset (64K in this example) + * x =3D bits to index a TCE which can be split to equal chunks to i= ndex + * within the level. + * The aim is to split "x" to smaller possible number of levels. */ entries =3D create.window_size >> create.page_shift; - pages =3D MAX((entries * sizeof(uint64_t)) / getpagesize(), 1); - pages =3D MAX(pow2ceil(pages), 1); /* Round up */ - create.levels =3D ctz64(pages) / 6 + 1; - - ret =3D ioctl(container->fd, VFIO_IOMMU_SPAPR_TCE_CREATE, &create); + /* bits_total is number of "x" needed */ + bits_total =3D ctz64(entries * sizeof(uint64_t)); + /* + * bits_per_level is a safe guess of how much we can allocate per leve= l: + * 8 is the current minimum for CONFIG_FORCE_MAX_ZONEORDER and MAX_ORD= ER + * is usually bigger than that. + * Below we look at getpagesize() as TCEs are allocated from system pa= ges. + */ + bits_per_level =3D ctz64(getpagesize()) + 8; + create.levels =3D bits_total / bits_per_level; + if (bits_total % bits_per_level) { + ++create.levels; + } + max_levels =3D (64 - create.page_shift) / ctz64(getpagesize()); + for ( ; create.levels <=3D max_levels; ++create.levels) { + ret =3D ioctl(container->fd, VFIO_IOMMU_SPAPR_TCE_CREATE, &create); + if (!ret) { + break; + } + } if (ret) { error_report("Failed to create a window, ret =3D %d (%m)", ret); return -errno; @@ -200,6 +222,7 @@ int vfio_spapr_create_window(VFIOContainer *container, return -EINVAL; } trace_vfio_spapr_create_window(create.page_shift, + create.levels, create.window_size, create.start_addr); *pgsize =3D pagesize; diff --git a/hw/vfio/trace-events b/hw/vfio/trace-events index ed2f333ad7..cf1e886818 100644 --- a/hw/vfio/trace-events +++ b/hw/vfio/trace-events @@ -129,6 +129,6 @@ vfio_prereg_listener_region_add_skip(uint64_t start, ui= nt64_t end) "0x%"PRIx64" vfio_prereg_listener_region_del_skip(uint64_t start, uint64_t end) "0x%"PR= Ix64" - 0x%"PRIx64 vfio_prereg_register(uint64_t va, uint64_t size, int ret) "va=3D0x%"PRIx64= " size=3D0x%"PRIx64" ret=3D%d" vfio_prereg_unregister(uint64_t va, uint64_t size, int ret) "va=3D0x%"PRIx= 64" size=3D0x%"PRIx64" ret=3D%d" -vfio_spapr_create_window(int ps, uint64_t ws, uint64_t off) "pageshift=3D0= x%x winsize=3D0x%"PRIx64" offset=3D0x%"PRIx64 +vfio_spapr_create_window(int ps, unsigned int levels, uint64_t ws, uint64_= t off) "pageshift=3D0x%x levels=3D%u winsize=3D0x%"PRIx64" offset=3D0x%"PRI= x64 vfio_spapr_remove_window(uint64_t off) "offset=3D0x%"PRIx64 vfio_spapr_group_attach(int groupfd, int tablefd) "Attached groupfd %d to = liobn fd %d" --=20 2.20.1