From nobody Wed May 15 19:52:38 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1625738126056815.1437994051365; Thu, 8 Jul 2021 02:55:26 -0700 (PDT) Received: from localhost ([::1]:60362 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1m1Qkf-0002ci-0W for importer@patchew.org; Thu, 08 Jul 2021 05:55:25 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:58532) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m1QjL-0000Kz-Qs for qemu-devel@nongnu.org; Thu, 08 Jul 2021 05:54:03 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:28209) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m1QjI-0005I3-PV for qemu-devel@nongnu.org; Thu, 08 Jul 2021 05:54:03 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-45-iZbQ6tUAOBWDI7NIPQvETA-1; Thu, 08 Jul 2021 05:53:58 -0400 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 3853F36319; Thu, 8 Jul 2021 09:53:57 +0000 (UTC) Received: from t480s.redhat.com (ovpn-112-130.ams2.redhat.com [10.36.112.130]) by smtp.corp.redhat.com (Postfix) with ESMTP id E83055D9DD; Thu, 8 Jul 2021 09:53:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1625738039; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=5c4BkJkeAWRDuOL6pIC49RBdHBvgH77WwaWtOu9mmU0=; b=Nrh00/+WjwaYhCi8Vmfx3mJztwYPy3JArBo2Pn/Dy+Eihvmvlynk3zOVQU3FKdaLbmpt1L bTtMislvBfJLatjHVOMP6Scrx2x0jVR6i7Vtf4BYHZrMgE+y6btYi0K7O2ssOffxKV2wyc q1Qk1pFGJBNodtMaTAid7yYLw/XSg2Y= X-MC-Unique: iZbQ6tUAOBWDI7NIPQvETA-1 From: David Hildenbrand To: qemu-devel@nongnu.org Subject: [PATCH v2 1/2] virtio-balloon: don't start free page hinting if postcopy is possible Date: Thu, 8 Jul 2021 11:53:38 +0200 Message-Id: <20210708095339.20274-2-david@redhat.com> In-Reply-To: <20210708095339.20274-1-david@redhat.com> References: <20210708095339.20274-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=david@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=216.205.24.124; envelope-from=david@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1.439, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "Michael S . Tsirkin" , David Hildenbrand , qemu-stable@nongnu.org, Alexander Duyck , "Dr. David Alan Gilbert" , Juan Quintela , Wei Wang , Peter Xu , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZM-MESSAGEID: 1625738127260100001 Postcopy never worked properly with 'free-page-hint=3Don', as there are at least two issues: 1) With postcopy, the guest will never receive a VIRTIO_BALLOON_CMD_ID_DONE and consequently won't release free pages back to the OS once migration finishes. The issue is that for postcopy, we won't do a final bitmap sync while the guest is stopped on the source and virtio_balloon_free_page_hint_notify() will only call virtio_balloon_free_page_done() on the source during PRECOPY_NOTIFY_CLEANUP, after the VM state was already migrated to the destination. 2) Once the VM touches a page on the destination that has been excluded from migration on the source via qemu_guest_free_page_hint() while postcopy is active, that thread will stall until postcopy finishes and all threads are woken up. (with older Linux kernels that won't retry faults when woken up via userfaultfd, we might actually get a SEGFAULT) The issue is that the source will refuse to migrate any pages that are not marked as dirty in the dirty bmap -- for example, because the page might just have been sent. Consequently, the faulting thread will stall, waiting for the page to be migrated -- which could take quite a while and result in guest OS issues. While we could fix 1) comparatively easily, 2) is harder to get right and might require more involved RAM migration changes on source and destination [1]. As it never worked properly, let's not start free page hinting in the precopy notifier if the postcopy migration capability was enabled to fix it easily. Capabilities cannot be enabled once migration is already running. Note 1: in the future we might either adjust migration code on the source to track pages that have actually been sent or adjust migration code on source and destination to eventually send pages multiple times from the source and and deal with pages that are sent multiple times on the destination. Note 2: virtio-mem has similar issues, however, access to "unplugged" memory by the guest is very rare and we would have to be very lucky for it to happen during migration. The spec states "The driver SHOULD NOT read from unplugged memory blocks ..." and "The driver MUST NOT write to unplugged memory blocks". virtio-mem will move away from virtio_balloon_free_page_done() soon and handle this case explicitly on the destination. [1] https://lkml.kernel.org/r/e79fd18c-aa62-c1d8-c7f3-ba3fc2c25fc8@redhat.c= om Fixes: c13c4153f76d ("virtio-balloon: VIRTIO_BALLOON_F_FREE_PAGE_HINT") Cc: qemu-stable@nongnu.org Cc: Wei Wang Cc: Michael S. Tsirkin Cc: Philippe Mathieu-Daud=C3=A9 Cc: Alexander Duyck Cc: Juan Quintela Cc: "Dr. David Alan Gilbert" Cc: Peter Xu Signed-off-by: David Hildenbrand Reviewed-by: Peter Xu --- hw/virtio/virtio-balloon.c | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/hw/virtio/virtio-balloon.c b/hw/virtio/virtio-balloon.c index 4b5d9e5e50..ae7867a8db 100644 --- a/hw/virtio/virtio-balloon.c +++ b/hw/virtio/virtio-balloon.c @@ -30,6 +30,7 @@ #include "trace.h" #include "qemu/error-report.h" #include "migration/misc.h" +#include "migration/migration.h" =20 #include "hw/virtio/virtio-bus.h" #include "hw/virtio/virtio-access.h" @@ -662,6 +663,18 @@ virtio_balloon_free_page_hint_notify(NotifierWithRetur= n *n, void *data) return 0; } =20 + /* + * Pages hinted via qemu_guest_free_page_hint() are cleared from the d= irty + * bitmap and will not get migrated, especially also not when the post= copy + * destination starts using them and requests migration from the sourc= e; the + * faulting thread will stall until postcopy migration finishes and + * all threads are woken up. Let's not start free page hinting if post= copy + * is possible. + */ + if (migrate_postcopy_ram()) { + return 0; + } + switch (pnd->reason) { case PRECOPY_NOTIFY_BEFORE_BITMAP_SYNC: virtio_balloon_free_page_stop(dev); --=20 2.31.1 From nobody Wed May 15 19:52:38 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1625738244672765.2489271496162; Thu, 8 Jul 2021 02:57:24 -0700 (PDT) Received: from localhost ([::1]:37700 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1m1QmZ-0006Md-BB for importer@patchew.org; Thu, 08 Jul 2021 05:57:23 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:58542) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m1QjN-0000Q1-T7 for qemu-devel@nongnu.org; Thu, 08 Jul 2021 05:54:05 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:23811) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m1QjK-0005Iz-HR for qemu-devel@nongnu.org; Thu, 08 Jul 2021 05:54:05 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-2-qO567luNOA-XgCUSS1qkbw-1; Thu, 08 Jul 2021 05:54:00 -0400 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id C022B8030B0; Thu, 8 Jul 2021 09:53:59 +0000 (UTC) Received: from t480s.redhat.com (ovpn-112-130.ams2.redhat.com [10.36.112.130]) by smtp.corp.redhat.com (Postfix) with ESMTP id 95BE25D9DD; Thu, 8 Jul 2021 09:53:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1625738042; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=RCYbme4UFgQccPlQ7jM6iCudvzHrSKM3nTlgSxu0LYg=; b=MCoEuGl8CIPLCHFvhKLMbRHTFlX5TVLFQPt/KOkn7p8dYwxJks6e1gyeHOuhOKl6C2aEQG 8oGJ1pKzs99/jxNxEKmOT6XAn4BzVdyW3AJ9DqMPZqiYFIWRxp5E5QSRytsnuL9Y62u/LM GrdDXRUf6BtOV5HO2i3uT9HWIMdS9M0= X-MC-Unique: qO567luNOA-XgCUSS1qkbw-1 From: David Hildenbrand To: qemu-devel@nongnu.org Subject: [PATCH v2 2/2] virtio-balloon: free page hinting cleanups Date: Thu, 8 Jul 2021 11:53:39 +0200 Message-Id: <20210708095339.20274-3-david@redhat.com> In-Reply-To: <20210708095339.20274-1-david@redhat.com> References: <20210708095339.20274-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=david@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=david@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1.439, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Juan Quintela , "Michael S . Tsirkin" , David Hildenbrand , "Dr. David Alan Gilbert" , Alexander Duyck , Wei Wang , Peter Xu , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZM-MESSAGEID: 1625738245216100003 Let's compress the code a bit to improve readability. We can drop the vm_running check in virtio_balloon_free_page_start() as it's already properly checked in the single caller. Cc: Wei Wang Cc: Michael S. Tsirkin Cc: Philippe Mathieu-Daud=C3=A9 Cc: Alexander Duyck Cc: Juan Quintela Cc: "Dr. David Alan Gilbert" Cc: Peter Xu Signed-off-by: David Hildenbrand --- hw/virtio/virtio-balloon.c | 28 ++++++++-------------------- 1 file changed, 8 insertions(+), 20 deletions(-) diff --git a/hw/virtio/virtio-balloon.c b/hw/virtio/virtio-balloon.c index ae7867a8db..5a69dce35d 100644 --- a/hw/virtio/virtio-balloon.c +++ b/hw/virtio/virtio-balloon.c @@ -534,22 +534,18 @@ static bool get_free_page_hints(VirtIOBalloon *dev) if (dev->free_page_hint_status =3D=3D FREE_PAGE_HINT_S_REQUESTED && id =3D=3D dev->free_page_hint_cmd_id) { dev->free_page_hint_status =3D FREE_PAGE_HINT_S_START; - } else { + } else if (dev->free_page_hint_status =3D=3D FREE_PAGE_HINT_S_STAR= T) { /* * Stop the optimization only when it has started. This * avoids a stale stop sign for the previous command. */ - if (dev->free_page_hint_status =3D=3D FREE_PAGE_HINT_S_START) { - dev->free_page_hint_status =3D FREE_PAGE_HINT_S_STOP; - } + dev->free_page_hint_status =3D FREE_PAGE_HINT_S_STOP; } } =20 - if (elem->in_num) { - if (dev->free_page_hint_status =3D=3D FREE_PAGE_HINT_S_START) { - qemu_guest_free_page_hint(elem->in_sg[0].iov_base, - elem->in_sg[0].iov_len); - } + if (elem->in_num && dev->free_page_hint_status =3D=3D FREE_PAGE_HINT_S= _START) { + qemu_guest_free_page_hint(elem->in_sg[0].iov_base, + elem->in_sg[0].iov_len); } =20 out: @@ -592,16 +588,10 @@ static void virtio_balloon_free_page_start(VirtIOBall= oon *s) { VirtIODevice *vdev =3D VIRTIO_DEVICE(s); =20 - /* For the stop and copy phase, we don't need to start the optimizatio= n */ - if (!vdev->vm_running) { - return; - } - qemu_mutex_lock(&s->free_page_lock); =20 if (s->free_page_hint_cmd_id =3D=3D UINT_MAX) { - s->free_page_hint_cmd_id =3D - VIRTIO_BALLOON_FREE_PAGE_HINT_CMD_ID_MIN; + s->free_page_hint_cmd_id =3D VIRTIO_BALLOON_FREE_PAGE_HINT_CMD_ID_= MIN; } else { s->free_page_hint_cmd_id++; } @@ -649,8 +639,7 @@ static void virtio_balloon_free_page_done(VirtIOBalloon= *s) static int virtio_balloon_free_page_hint_notify(NotifierWithReturn *n, void *data) { - VirtIOBalloon *dev =3D container_of(n, VirtIOBalloon, - free_page_hint_notify); + VirtIOBalloon *dev =3D container_of(n, VirtIOBalloon, free_page_hint_n= otify); VirtIODevice *vdev =3D VIRTIO_DEVICE(dev); PrecopyNotifyData *pnd =3D data; =20 @@ -919,8 +908,7 @@ static void virtio_balloon_device_realize(DeviceState *= dev, Error **errp) s->dvq =3D virtio_add_queue(vdev, 128, virtio_balloon_handle_output); s->svq =3D virtio_add_queue(vdev, 128, virtio_balloon_receive_stats); =20 - if (virtio_has_feature(s->host_features, - VIRTIO_BALLOON_F_FREE_PAGE_HINT)) { + if (virtio_has_feature(s->host_features, VIRTIO_BALLOON_F_FREE_PAGE_HI= NT)) { s->free_page_vq =3D virtio_add_queue(vdev, VIRTQUEUE_MAX_SIZE, virtio_balloon_handle_free_page= _vq); precopy_add_notifier(&s->free_page_hint_notify); --=20 2.31.1