From nobody Tue Feb 10 15:03:04 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1615916923; cv=none; d=zohomail.com; s=zohoarc; b=IS2fFQTzUaSoUHgxloZ1FyxAIxlOnYGiHm1TmrjUwJ8VDqWKima/T/oi72sIsVbun9qPVUuBk7GLKTrLq1qSHrlXnvSdmgCsMrAoljl7bpmc21vNfNDe2a/mmW3SC2QKe+LYhwHMTVOu3vuUxdwAFh+UC/KMI6KfqLtaQYLWMw0= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1615916923; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=K+5BHNLJ1GDq5HpQgwYYofyXEQ19Z/ZU7zJTadj2dU8=; b=OLV09PaZZVbAFLXcpKYYjc4TaFb7m4lzGY8Whxme+SeI1DYhOgd1xkWCAXUgno3nwDcoMQVlw8koyNu18ms1hIA1ZWMyu5RP0Bec0BpK6IWCk3s8j7oDgM9tAjClvlwlqY/4RT73h7vv9OvH471eEItJnFpzQBsNt7XrUfZ/fQA= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1615916923967897.2439929861418; Tue, 16 Mar 2021 10:48:43 -0700 (PDT) Received: from localhost ([::1]:49812 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lMDoA-0003EE-Hu for importer@patchew.org; Tue, 16 Mar 2021 13:48:42 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:53246) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lMDAk-00050T-Uo for qemu-devel@nongnu.org; Tue, 16 Mar 2021 13:07:58 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:36071) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lMDAi-00060L-Tr for qemu-devel@nongnu.org; Tue, 16 Mar 2021 13:07:58 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-150-0y3UiZ1wNuKU_HAkREVz7g-1; Tue, 16 Mar 2021 13:07:52 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 9F05F801596; Tue, 16 Mar 2021 17:07:51 +0000 (UTC) Received: from rhel8vm.home.shazbot.org (ovpn-115-65.phx2.redhat.com [10.3.115.65]) by smtp.corp.redhat.com (Postfix) with ESMTP id 260105C1A3; Tue, 16 Mar 2021 17:07:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1615914475; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=K+5BHNLJ1GDq5HpQgwYYofyXEQ19Z/ZU7zJTadj2dU8=; b=YXjv/ZZ7UQmBU9PBpwWIz8Az6GDoglbIV+w5Aiv8Q1u1ylg/+lO1172NH1BKcyRkiVlH2B A1VmKnTKGrCpL6M87qHD/NUIjm1zOJsj6wRHJlTlKf377EP0ITTwSAWE/SKW4rBH/lqJAB 7ofbcE+v10upsvmAJWHYMn0+bZmBgRE= X-MC-Unique: 0y3UiZ1wNuKU_HAkREVz7g-1 Subject: [PULL 10/10] vfio/migrate: Move switch of dirty tracking into vfio_memory_listener From: Alex Williamson To: qemu-devel@nongnu.org Date: Tue, 16 Mar 2021 11:07:48 -0600 Message-ID: <161591446879.135549.13699828622046019656.stgit@rhel8vm.home.shazbot.org> In-Reply-To: <161591414009.135549.5804863877511053803.stgit@rhel8vm.home.shazbot.org> References: <161591414009.135549.5804863877511053803.stgit@rhel8vm.home.shazbot.org> User-Agent: StGit/0.23 MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=alex.williamson@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=alex.williamson@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -29 X-Spam_score: -3.0 X-Spam_bar: --- X-Spam_report: (-3.0 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.25, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Zenghui Yu , Paolo Bonzini , Keqian Zhu Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) From: Keqian Zhu For now the switch of vfio dirty page tracking is integrated into @vfio_save_handler. The reason is that some PCI vendor driver may start to track dirty base on _SAVING state of device, so if dirty tracking is started before setting device state, vfio will report full-dirty to QEMU. However, the dirty bmap of all ramblocks are fully set when setup ram saving, so it's not matter whether the device is in _SAVING state when start vfio dirty tracking. Moreover, this logic causes some problems [1]. The object of dirty tracking is guest memory, but the object of @vfio_save_handler is device state, which produces unnecessary coupling and conflicts: 1. Coupling: Their saving granule is different (perVM vs perDevice). vfio will enable dirty_page_tracking for each devices, actually once is enough. 2. Conflicts: The ram_save_setup() traverses all memory_listeners to execute their log_start() and log_sync() hooks to get the first round dirty bitmap, which is used by the bulk stage of ram saving. However, as vfio dirty tracking is not yet started, it can't get dirty bitmap from vfio. Then we give up the chance to handle vfio dirty page at bulk stage. Move the switch of vfio dirty_page_tracking into vfio_memory_listener can solve above problems. Besides, Do not require devices in SAVING state for vfio_sync_dirty_bitmap(). [1] https://www.spinics.net/lists/kvm/msg229967.html Reported-by: Zenghui Yu Signed-off-by: Keqian Zhu Suggested-by: Paolo Bonzini Message-Id: <20210309031913.11508-1-zhukeqian1@huawei.com> Signed-off-by: Alex Williamson --- hw/vfio/common.c | 49 ++++++++++++++++++++++++++++++++++++++++-------= -- hw/vfio/migration.c | 35 ----------------------------------- 2 files changed, 40 insertions(+), 44 deletions(-) diff --git a/hw/vfio/common.c b/hw/vfio/common.c index ad08dfd729b9..ae5654fcdb8d 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -311,7 +311,7 @@ bool vfio_mig_active(void) return true; } =20 -static bool vfio_devices_all_saving(VFIOContainer *container) +static bool vfio_devices_all_dirty_tracking(VFIOContainer *container) { VFIOGroup *group; VFIODevice *vbasedev; @@ -329,13 +329,8 @@ static bool vfio_devices_all_saving(VFIOContainer *con= tainer) return false; } =20 - if (migration->device_state & VFIO_DEVICE_STATE_SAVING) { - if ((vbasedev->pre_copy_dirty_page_tracking =3D=3D ON_OFF_= AUTO_OFF) - && (migration->device_state & VFIO_DEVICE_STATE_RUNNIN= G)) { - return false; - } - continue; - } else { + if ((vbasedev->pre_copy_dirty_page_tracking =3D=3D ON_OFF_AUTO= _OFF) + && (migration->device_state & VFIO_DEVICE_STATE_RUNNING)) { return false; } } @@ -989,6 +984,40 @@ static void vfio_listener_region_del(MemoryListener *l= istener, } } =20 +static void vfio_set_dirty_page_tracking(VFIOContainer *container, bool st= art) +{ + int ret; + struct vfio_iommu_type1_dirty_bitmap dirty =3D { + .argsz =3D sizeof(dirty), + }; + + if (start) { + dirty.flags =3D VFIO_IOMMU_DIRTY_PAGES_FLAG_START; + } else { + dirty.flags =3D VFIO_IOMMU_DIRTY_PAGES_FLAG_STOP; + } + + ret =3D ioctl(container->fd, VFIO_IOMMU_DIRTY_PAGES, &dirty); + if (ret) { + error_report("Failed to set dirty tracking flag 0x%x errno: %d", + dirty.flags, errno); + } +} + +static void vfio_listener_log_global_start(MemoryListener *listener) +{ + VFIOContainer *container =3D container_of(listener, VFIOContainer, lis= tener); + + vfio_set_dirty_page_tracking(container, true); +} + +static void vfio_listener_log_global_stop(MemoryListener *listener) +{ + VFIOContainer *container =3D container_of(listener, VFIOContainer, lis= tener); + + vfio_set_dirty_page_tracking(container, false); +} + static int vfio_get_dirty_bitmap(VFIOContainer *container, uint64_t iova, uint64_t size, ram_addr_t ram_addr) { @@ -1130,7 +1159,7 @@ static void vfio_listener_log_sync(MemoryListener *li= stener, return; } =20 - if (vfio_devices_all_saving(container)) { + if (vfio_devices_all_dirty_tracking(container)) { vfio_sync_dirty_bitmap(container, section); } } @@ -1138,6 +1167,8 @@ static void vfio_listener_log_sync(MemoryListener *li= stener, static const MemoryListener vfio_memory_listener =3D { .region_add =3D vfio_listener_region_add, .region_del =3D vfio_listener_region_del, + .log_global_start =3D vfio_listener_log_global_start, + .log_global_stop =3D vfio_listener_log_global_stop, .log_sync =3D vfio_listener_log_sync, }; =20 diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index eafb778947c3..384576cfc051 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -395,40 +395,10 @@ static int vfio_load_device_config_state(QEMUFile *f,= void *opaque) return qemu_file_get_error(f); } =20 -static int vfio_set_dirty_page_tracking(VFIODevice *vbasedev, bool start) -{ - int ret; - VFIOMigration *migration =3D vbasedev->migration; - VFIOContainer *container =3D vbasedev->group->container; - struct vfio_iommu_type1_dirty_bitmap dirty =3D { - .argsz =3D sizeof(dirty), - }; - - if (start) { - if (migration->device_state & VFIO_DEVICE_STATE_SAVING) { - dirty.flags =3D VFIO_IOMMU_DIRTY_PAGES_FLAG_START; - } else { - return -EINVAL; - } - } else { - dirty.flags =3D VFIO_IOMMU_DIRTY_PAGES_FLAG_STOP; - } - - ret =3D ioctl(container->fd, VFIO_IOMMU_DIRTY_PAGES, &dirty); - if (ret) { - error_report("Failed to set dirty tracking flag 0x%x errno: %d", - dirty.flags, errno); - return -errno; - } - return ret; -} - static void vfio_migration_cleanup(VFIODevice *vbasedev) { VFIOMigration *migration =3D vbasedev->migration; =20 - vfio_set_dirty_page_tracking(vbasedev, false); - if (migration->region.mmaps) { vfio_region_unmap(&migration->region); } @@ -469,11 +439,6 @@ static int vfio_save_setup(QEMUFile *f, void *opaque) return ret; } =20 - ret =3D vfio_set_dirty_page_tracking(vbasedev, true); - if (ret) { - return ret; - } - qemu_put_be64(f, VFIO_MIG_FLAG_END_OF_STATE); =20 ret =3D qemu_file_get_error(f);