From nobody Sat May 11 11:46:39 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1688469369; cv=none; d=zohomail.com; s=zohoarc; b=UGrnRo3X9libeKYv3/eCf9fNKbYB6rGGjOYTL9lTt5jNbrP8TMnsploFw+sx40/3d7kKhFptsrb72qK3Gfgvk9MImSgLHe/QOZPXSsdFl4j6kMA/dhdixYQbV4YdA1KbHql8z5zOmgjDb6tLNOAhRQa8hqKlhzUjx3zgcS7A3yQ= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1688469369; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=XzL1+bAVr91n8fic68XSWcMtzYFJAdEtrb5hgT+HHxM=; b=EXSecavWHz1zNbTE2u1mLWfjB3C86HSaAAxQVvabJePmZERBHYgfpElcN1MgKiILHlv3CF4DRWzsViygN89BApgPsy4C17mLbUeSfw3+dIC8IQMZHCNlnLid6VvAW9MKLsCAj4atOhFNlNtMRh5r/azK6V81I/3qA2PE9IZ9hEE= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1688469369832800.220984871471; Tue, 4 Jul 2023 04:16:09 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qGe0Y-0002h5-B0; Tue, 04 Jul 2023 07:15:46 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qGe0V-0002fv-Qn for qemu-devel@nongnu.org; Tue, 04 Jul 2023 07:15:43 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qGe0U-0007xz-2p for qemu-devel@nongnu.org; Tue, 04 Jul 2023 07:15:43 -0400 Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-528-uMQO3GonOp27_Yli2eMq-A-1; Tue, 04 Jul 2023 07:15:38 -0400 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id B6C9D185A78B; Tue, 4 Jul 2023 11:15:37 +0000 (UTC) Received: from laptop.redhat.com (unknown [10.39.194.15]) by smtp.corp.redhat.com (Postfix) with ESMTP id C175E492C13; Tue, 4 Jul 2023 11:15:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1688469341; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=XzL1+bAVr91n8fic68XSWcMtzYFJAdEtrb5hgT+HHxM=; b=cDCQBg3kMmWHYnx2/mk/u8CHe73Yk/PUDecevrRwB5iDGkQUgmYP/bMASYkTUP8NowQMcb lAgsx6S3iieRbHsEi4Ds7coDzkjkt14vSDDqRDaVFZiueBJbDH0ppnVh3h07s5yHTcelT9 ukW9RuplUz2O7nfGSPsyt8ptWGpnQYE= X-MC-Unique: uMQO3GonOp27_Yli2eMq-A-1 From: Eric Auger To: eric.auger.pro@gmail.com, eric.auger@redhat.com, qemu-devel@nongnu.org, qemu-arm@nongnu.org, mst@redhat.com, jean-philippe@linaro.org, zhenzhong.duan@intel.com Cc: alex.williamson@redhat.com, clg@redhap.com, bharat.bhushan@nxp.com, peter.maydell@linaro.org Subject: [PATCH 1/2] virtio-iommu: Fix 64kB host page size VFIO device assignment Date: Tue, 4 Jul 2023 13:15:26 +0200 Message-Id: <20230704111527.3424992-2-eric.auger@redhat.com> In-Reply-To: <20230704111527.3424992-1-eric.auger@redhat.com> References: <20230704111527.3424992-1-eric.auger@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.1 on 10.11.54.10 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=eric.auger@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1688469372080100002 Content-Type: text/plain; charset="utf-8" When running on a 64kB page size host and protecting a VFIO device with the virtio-iommu, qemu crashes with this kind of message: qemu-kvm: virtio-iommu page mask 0xfffffffffffff000 is incompatible with mask 0x20010000 qemu: hardware error: vfio: DMA mapping failed, unable to continue This is due to the fact the IOMMU MR corresponding to the VFIO device is enabled very late on domain attach, after the machine init. The device reports a minimal 64kB page size but it is too late to be applied. virtio_iommu_set_page_size_mask() fails and this causes vfio_listener_region_add() to end up with hw_error(); To work around this issue, we transiently enable the IOMMU MR on machine init to collect the page size requirements and then restore the bypass state. Fixes: 90519b9053 ("virtio-iommu: Add bypass mode support to assigned devic= e") Signed-off-by: Eric Auger --- include/hw/virtio/virtio-iommu.h | 2 ++ hw/virtio/virtio-iommu.c | 30 ++++++++++++++++++++++++++++-- hw/virtio/trace-events | 1 + 3 files changed, 31 insertions(+), 2 deletions(-) diff --git a/include/hw/virtio/virtio-iommu.h b/include/hw/virtio/virtio-io= mmu.h index 2ad5ee320b..a93fc5383e 100644 --- a/include/hw/virtio/virtio-iommu.h +++ b/include/hw/virtio/virtio-iommu.h @@ -61,6 +61,8 @@ struct VirtIOIOMMU { QemuRecMutex mutex; GTree *endpoints; bool boot_bypass; + Notifier machine_done; + bool granule_frozen; }; =20 #endif diff --git a/hw/virtio/virtio-iommu.c b/hw/virtio/virtio-iommu.c index 1cd258135d..1eaf81bab5 100644 --- a/hw/virtio/virtio-iommu.c +++ b/hw/virtio/virtio-iommu.c @@ -24,6 +24,7 @@ #include "hw/virtio/virtio.h" #include "sysemu/kvm.h" #include "sysemu/reset.h" +#include "sysemu/sysemu.h" #include "qapi/error.h" #include "qemu/error-report.h" #include "trace.h" @@ -1106,12 +1107,12 @@ static int virtio_iommu_set_page_size_mask(IOMMUMem= oryRegion *mr, } =20 /* - * After the machine is finalized, we can't change the mask anymore. I= f by + * Once the granule is frozen we can't change the mask anymore. If by * chance the hotplugged device supports the same granule, we can still * accept it. Having a different masks is possible but the guest will = use * sub-optimal block sizes, so warn about it. */ - if (phase_check(PHASE_MACHINE_READY)) { + if (s->granule_frozen) { int new_granule =3D ctz64(new_mask); int cur_granule =3D ctz64(cur_mask); =20 @@ -1146,6 +1147,27 @@ static void virtio_iommu_system_reset(void *opaque) =20 } =20 +static void virtio_iommu_freeze_granule(Notifier *notifier, void *data) +{ + VirtIOIOMMU *s =3D container_of(notifier, VirtIOIOMMU, machine_done); + bool boot_bypass =3D s->config.bypass; + int granule; + + /* + * Transient IOMMU MR enable to collect page_size_mask requirement + * through memory_region_iommu_set_page_size_mask() called by + * VFIO region_add() callback + */ + s->config.bypass =3D 0; + virtio_iommu_switch_address_space_all(s); + /* restore default */ + s->config.bypass =3D boot_bypass; + virtio_iommu_switch_address_space_all(s); + s->granule_frozen =3D true; + granule =3D ctz64(s->config.page_size_mask); + trace_virtio_iommu_freeze_granule(BIT(granule)); +} + static void virtio_iommu_device_realize(DeviceState *dev, Error **errp) { VirtIODevice *vdev =3D VIRTIO_DEVICE(dev); @@ -1189,6 +1211,9 @@ static void virtio_iommu_device_realize(DeviceState *= dev, Error **errp) error_setg(errp, "VIRTIO-IOMMU is not attached to any PCI bus!"); } =20 + s->machine_done.notify =3D virtio_iommu_freeze_granule; + qemu_add_machine_init_done_notifier(&s->machine_done); + qemu_register_reset(virtio_iommu_system_reset, s); } =20 @@ -1198,6 +1223,7 @@ static void virtio_iommu_device_unrealize(DeviceState= *dev) VirtIOIOMMU *s =3D VIRTIO_IOMMU(dev); =20 qemu_unregister_reset(virtio_iommu_system_reset, s); + qemu_remove_machine_init_done_notifier(&s->machine_done); =20 g_hash_table_destroy(s->as_by_busptr); if (s->domains) { diff --git a/hw/virtio/trace-events b/hw/virtio/trace-events index 8f8d05cf9b..68b752e304 100644 --- a/hw/virtio/trace-events +++ b/hw/virtio/trace-events @@ -131,6 +131,7 @@ virtio_iommu_set_page_size_mask(const char *name, uint6= 4_t old, uint64_t new) "m virtio_iommu_notify_flag_add(const char *name) "add notifier to mr %s" virtio_iommu_notify_flag_del(const char *name) "del notifier from mr %s" virtio_iommu_switch_address_space(uint8_t bus, uint8_t slot, uint8_t fn, b= ool on) "Device %02x:%02x.%x switching address space (iommu enabled=3D%d)" +virtio_iommu_freeze_granule(uint64_t page_size_mask) "granule set to 0x%"P= RIx64 =20 # virtio-mem.c virtio_mem_send_response(uint16_t type) "type=3D%" PRIu16 --=20 2.38.1 From nobody Sat May 11 11:46:39 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1688469384; cv=none; d=zohomail.com; s=zohoarc; b=lxO7O7lk6yuKAMIjssVgwisWLgss5I3Uze0veCj4aTo8HQ17v2u9D+LnpUc464iOb96fkPrVF2kA3B9DrYKlQtZn36IcJcbtvoo+xEC6JUir3YUg6iOTi7OfykKRlI0BpuI50ZCaVDVSrvDcmoBsq1vFoofUl/+68e2o+nqOGRM= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1688469384; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=ypEP/NT8IxiszDOugkjgPdfbbvRW0XffouojgQcGeOU=; b=TeSuhf6HRHwj+mFWi5Vm5RjgRAaZXiVphnulXADoUEU/f9RAz6d07sdK0hr4XoheV//aDjxC/JvdvhE30xdXNIsfm2EuIH9Qa4tisHVhSafKfLfYa0vC0hx7ZSAE89mP/xup2jYm48W21cJaosvx8aKFhBU7/TBufmBHTNaMi38= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1688469384855753.3132468739182; Tue, 4 Jul 2023 04:16:24 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qGe0g-0002ja-CH; Tue, 04 Jul 2023 07:15:54 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qGe0a-0002hj-DQ for qemu-devel@nongnu.org; Tue, 04 Jul 2023 07:15:48 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qGe0W-0007yx-Ds for qemu-devel@nongnu.org; Tue, 04 Jul 2023 07:15:46 -0400 Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-656-slBKrhTXMGCzju5lEFZr_g-1; Tue, 04 Jul 2023 07:15:40 -0400 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id F04FA805C3F; Tue, 4 Jul 2023 11:15:39 +0000 (UTC) Received: from laptop.redhat.com (unknown [10.39.194.15]) by smtp.corp.redhat.com (Postfix) with ESMTP id 0722F492C13; Tue, 4 Jul 2023 11:15:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1688469343; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ypEP/NT8IxiszDOugkjgPdfbbvRW0XffouojgQcGeOU=; b=TwKZ2NzhvCkl18ztUZuQGxyWc7DVRy/h/q8xg0kjYp7adpdQ1qyJfvu59Q7MdXWzfuEr3n TFgUk/8KMfPdGW19mOcjd2ZYLQmzvYhbTDwcUJdZGW3SuKeUKcXbW0PbkVPCnIg2Yus+Xu fl/0GqbDAX8WwElX+ZitcqbucyLJNQI= X-MC-Unique: slBKrhTXMGCzju5lEFZr_g-1 From: Eric Auger To: eric.auger.pro@gmail.com, eric.auger@redhat.com, qemu-devel@nongnu.org, qemu-arm@nongnu.org, mst@redhat.com, jean-philippe@linaro.org, zhenzhong.duan@intel.com Cc: alex.williamson@redhat.com, clg@redhap.com, bharat.bhushan@nxp.com, peter.maydell@linaro.org Subject: [PATCH 2/2] virtio-iommu: Rework the trace in virtio_iommu_set_page_size_mask() Date: Tue, 4 Jul 2023 13:15:27 +0200 Message-Id: <20230704111527.3424992-3-eric.auger@redhat.com> In-Reply-To: <20230704111527.3424992-1-eric.auger@redhat.com> References: <20230704111527.3424992-1-eric.auger@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.1 on 10.11.54.10 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=eric.auger@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1688469385240100001 Content-Type: text/plain; charset="utf-8" The current error messages in virtio_iommu_set_page_size_mask() sound quite similar for different situations and miss the IOMMU memory region that causes the issue. Clarify them and rework the comment. Also remove the trace when the new page_size_mask is not applied as the current frozen granule is kept. This message is rather confusing for the end user and anyway the current granule would have been used by the driver Signed-off-by: Eric Auger Reviewed-by: Zhenzhong Duan --- hw/virtio/virtio-iommu.c | 19 +++++++------------ 1 file changed, 7 insertions(+), 12 deletions(-) diff --git a/hw/virtio/virtio-iommu.c b/hw/virtio/virtio-iommu.c index 1eaf81bab5..0d9f7196fe 100644 --- a/hw/virtio/virtio-iommu.c +++ b/hw/virtio/virtio-iommu.c @@ -1101,29 +1101,24 @@ static int virtio_iommu_set_page_size_mask(IOMMUMem= oryRegion *mr, new_mask); =20 if ((cur_mask & new_mask) =3D=3D 0) { - error_setg(errp, "virtio-iommu page mask 0x%"PRIx64 - " is incompatible with mask 0x%"PRIx64, cur_mask, new_m= ask); + error_setg(errp, "virtio-iommu %s reports a page size mask 0x%"PRI= x64 + " incompatible with currently supported mask 0x%"PRIx64, + mr->parent_obj.name, new_mask, cur_mask); return -1; } =20 /* * Once the granule is frozen we can't change the mask anymore. If by * chance the hotplugged device supports the same granule, we can still - * accept it. Having a different masks is possible but the guest will = use - * sub-optimal block sizes, so warn about it. + * accept it. */ if (s->granule_frozen) { - int new_granule =3D ctz64(new_mask); int cur_granule =3D ctz64(cur_mask); =20 - if (new_granule !=3D cur_granule) { - error_setg(errp, "virtio-iommu page mask 0x%"PRIx64 - " is incompatible with mask 0x%"PRIx64, cur_mask, - new_mask); + if (!(BIT(cur_granule) & new_mask)) { + error_setg(errp, "virtio-iommu %s does not support frozen gran= ule 0x%"PRIx64, + mr->parent_obj.name, BIT(cur_granule)); return -1; - } else if (new_mask !=3D cur_mask) { - warn_report("virtio-iommu page mask 0x%"PRIx64 - " does not match 0x%"PRIx64, cur_mask, new_mask); } return 0; } --=20 2.38.1