From nobody Sat Oct 25 23:35:03 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1516171936706110.83184516292977; Tue, 16 Jan 2018 22:52:16 -0800 (PST) Received: from localhost ([::1]:41949 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ebha3-00062v-IN for importer@patchew.org; Wed, 17 Jan 2018 01:52:15 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:54616) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ebhYB-00053X-8a for qemu-devel@nongnu.org; Wed, 17 Jan 2018 01:50:21 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ebhY8-0003n9-7r for qemu-devel@nongnu.org; Wed, 17 Jan 2018 01:50:19 -0500 Received: from mga11.intel.com ([192.55.52.93]:59301) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1ebhY7-0003l7-RP for qemu-devel@nongnu.org; Wed, 17 Jan 2018 01:50:16 -0500 Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by fmsmga102.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 16 Jan 2018 22:50:14 -0800 Received: from devel-ww.sh.intel.com ([10.239.48.110]) by fmsmga006.fm.intel.com with ESMTP; 16 Jan 2018 22:50:12 -0800 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.46,371,1511856000"; d="scan'208";a="196306502" From: Wei Wang To: qemu-devel@nongnu.org, virtio-dev@lists.oasis-open.org, mst@redhat.com, quintela@redhat.com, dgilbert@redhat.com Date: Wed, 17 Jan 2018 14:31:57 +0800 Message-Id: <1516170720-4948-2-git-send-email-wei.w.wang@intel.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1516170720-4948-1-git-send-email-wei.w.wang@intel.com> References: <1516170720-4948-1-git-send-email-wei.w.wang@intel.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 192.55.52.93 Subject: [Qemu-devel] [PATCH v1 1/4] virtio-balloon: VIRTIO_BALLOON_F_FREE_PAGE_VQ X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yang.zhang.wz@gmail.com, quan.xu0@gmail.com, liliang.opensource@gmail.com, wei.w.wang@intel.com, pbonzini@redhat.com, nilal@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" The new feature enables the virtio-balloon device to receive the hint of guest free pages from the free page vq, and clears the corresponding bits of the free page from the dirty bitmap, so that those free pages are not transferred by the migration thread. Without this feature, to local live migrate an 8G idle guest takes ~2286 ms. With this featrue, it takes ~260 ms, which redues the migration time to ~11%. Signed-off-by: Wei Wang Signed-off-by: Liang Li CC: Michael S. Tsirkin --- balloon.c | 46 ++++++-- hw/virtio/virtio-balloon.c | 140 ++++++++++++++++++++= +--- include/hw/virtio/virtio-balloon.h | 7 +- include/migration/misc.h | 3 + include/standard-headers/linux/virtio_balloon.h | 4 + include/sysemu/balloon.h | 15 ++- migration/ram.c | 10 ++ 7 files changed, 198 insertions(+), 27 deletions(-) diff --git a/balloon.c b/balloon.c index 1d720ff..480a989 100644 --- a/balloon.c +++ b/balloon.c @@ -36,6 +36,9 @@ =20 static QEMUBalloonEvent *balloon_event_fn; static QEMUBalloonStatus *balloon_stat_fn; +static QEMUBalloonFreePageSupport *balloon_free_page_support_fn; +static QEMUBalloonFreePageStart *balloon_free_page_start_fn; +static QEMUBalloonFreePageStop *balloon_free_page_stop_fn; static void *balloon_opaque; static bool balloon_inhibited; =20 @@ -64,17 +67,41 @@ static bool have_balloon(Error **errp) return true; } =20 -int qemu_add_balloon_handler(QEMUBalloonEvent *event_func, - QEMUBalloonStatus *stat_func, void *opaque) +bool balloon_free_page_support(void) { - if (balloon_event_fn || balloon_stat_fn || balloon_opaque) { - /* We're already registered one balloon handler. How many can - * a guest really have? - */ + return balloon_free_page_support_fn && + balloon_free_page_support_fn(balloon_opaque); +} + +void balloon_free_page_start(void) +{ + balloon_free_page_start_fn(balloon_opaque); +} + +void balloon_free_page_stop(void) +{ + balloon_free_page_stop_fn(balloon_opaque); +} + +int qemu_add_balloon_handler(QEMUBalloonEvent *event_fn, + QEMUBalloonStatus *stat_fn, + QEMUBalloonFreePageSupport *free_page_support= _fn, + QEMUBalloonFreePageStart *free_page_start_fn, + QEMUBalloonFreePageStop *free_page_stop_fn, + void *opaque) +{ + if (balloon_event_fn || balloon_stat_fn || balloon_free_page_support_f= n || + balloon_free_page_start_fn || balloon_free_page_stop_fn || + balloon_opaque) { + /* We already registered one balloon handler. */ return -1; } - balloon_event_fn =3D event_func; - balloon_stat_fn =3D stat_func; + + balloon_event_fn =3D event_fn; + balloon_stat_fn =3D stat_fn; + balloon_free_page_support_fn =3D free_page_support_fn; + balloon_free_page_start_fn =3D free_page_start_fn; + balloon_free_page_stop_fn =3D free_page_stop_fn; balloon_opaque =3D opaque; return 0; } @@ -86,6 +113,9 @@ void qemu_remove_balloon_handler(void *opaque) } balloon_event_fn =3D NULL; balloon_stat_fn =3D NULL; + balloon_free_page_support_fn =3D NULL; + balloon_free_page_start_fn =3D NULL; + balloon_free_page_stop_fn =3D NULL; balloon_opaque =3D NULL; } =20 diff --git a/hw/virtio/virtio-balloon.c b/hw/virtio/virtio-balloon.c index 14e08d2..a2a5536 100644 --- a/hw/virtio/virtio-balloon.c +++ b/hw/virtio/virtio-balloon.c @@ -30,6 +30,7 @@ =20 #include "hw/virtio/virtio-bus.h" #include "hw/virtio/virtio-access.h" +#include "migration/misc.h" =20 #define BALLOON_PAGE_SIZE (1 << VIRTIO_BALLOON_PFN_SHIFT) =20 @@ -73,6 +74,13 @@ static bool balloon_stats_supported(const VirtIOBalloon = *s) return virtio_vdev_has_feature(vdev, VIRTIO_BALLOON_F_STATS_VQ); } =20 +static bool balloon_free_page_supported(const VirtIOBalloon *s) +{ + VirtIODevice *vdev =3D VIRTIO_DEVICE(s); + + return virtio_vdev_has_feature(vdev, VIRTIO_BALLOON_F_FREE_PAGE_VQ); +} + static bool balloon_stats_enabled(const VirtIOBalloon *s) { return s->stats_poll_interval > 0; @@ -305,6 +313,85 @@ out: } } =20 +static void virtio_balloon_handle_free_pages(VirtIODevice *vdev, VirtQueue= *vq) +{ + VirtIOBalloon *dev =3D VIRTIO_BALLOON(vdev); + VirtQueueElement *elem; + uint32_t size, id; + + for (;;) { + elem =3D virtqueue_pop(vq, sizeof(VirtQueueElement)); + if (!elem) { + break; + } + + if (elem->out_num) { + iov_to_buf(elem->out_sg, elem->out_num, 0, &id, sizeof(uint32_= t)); + size =3D elem->out_sg[0].iov_len; + if (id =3D=3D dev->free_page_report_cmd_id) { + atomic_set(&dev->free_page_report_status, + FREE_PAGE_REPORT_S_IN_PROGRESS); + } else { + atomic_set(&dev->free_page_report_status, + FREE_PAGE_REPORT_S_STOP); + } + } + + if (elem->in_num) { + RAMBlock *block; + ram_addr_t offset; + + if (atomic_read(&dev->free_page_report_status) =3D=3D + FREE_PAGE_REPORT_S_IN_PROGRESS) { + block =3D qemu_ram_block_from_host(elem->in_sg[0].iov_base, + false, &offset); + size =3D elem->in_sg[0].iov_len; + skip_free_pages_from_dirty_bitmap(block, offset, size); + } + } + + virtqueue_push(vq, elem, sizeof(id)); + g_free(elem); + } +} + +static bool virtio_balloon_free_page_support(void *opaque) +{ + VirtIOBalloon *s =3D opaque; + + if (!balloon_free_page_supported(s)) { + return false; + } + + return true; +} + +static void virtio_balloon_free_page_start(void *opaque) +{ + VirtIOBalloon *dev =3D opaque; + VirtIODevice *vdev =3D VIRTIO_DEVICE(dev); + + dev->free_page_report_cmd_id++; + virtio_notify_config(vdev); + atomic_set(&dev->free_page_report_status, FREE_PAGE_REPORT_S_START); +} + +static void virtio_balloon_free_page_stop(void *opaque) +{ + VirtIOBalloon *dev =3D opaque; + + /* The guest has done the report */ + if (atomic_read(&dev->free_page_report_status) =3D=3D + FREE_PAGE_REPORT_S_STOP) { + return; + } + + /* Wait till a stop sign is received from the guest */ + while (atomic_read(&dev->free_page_report_status) !=3D + FREE_PAGE_REPORT_S_STOP) + ; +} + static void virtio_balloon_get_config(VirtIODevice *vdev, uint8_t *config_= data) { VirtIOBalloon *dev =3D VIRTIO_BALLOON(vdev); @@ -312,6 +399,7 @@ static void virtio_balloon_get_config(VirtIODevice *vde= v, uint8_t *config_data) =20 config.num_pages =3D cpu_to_le32(dev->num_pages); config.actual =3D cpu_to_le32(dev->actual); + config.free_page_report_cmd_id =3D cpu_to_le32(dev->free_page_report_c= md_id); =20 trace_virtio_balloon_get_config(config.num_pages, config.actual); memcpy(config_data, &config, sizeof(struct virtio_balloon_config)); @@ -418,6 +506,7 @@ static const VMStateDescription vmstate_virtio_balloon_= device =3D { .fields =3D (VMStateField[]) { VMSTATE_UINT32(num_pages, VirtIOBalloon), VMSTATE_UINT32(actual, VirtIOBalloon), + VMSTATE_UINT32(free_page_report_cmd_id, VirtIOBalloon), VMSTATE_END_OF_LIST() }, }; @@ -426,24 +515,18 @@ static void virtio_balloon_device_realize(DeviceState= *dev, Error **errp) { VirtIODevice *vdev =3D VIRTIO_DEVICE(dev); VirtIOBalloon *s =3D VIRTIO_BALLOON(dev); - int ret; =20 virtio_init(vdev, "virtio-balloon", VIRTIO_ID_BALLOON, sizeof(struct virtio_balloon_config)); =20 - ret =3D qemu_add_balloon_handler(virtio_balloon_to_target, - virtio_balloon_stat, s); - - if (ret < 0) { - error_setg(errp, "Only one balloon device is supported"); - virtio_cleanup(vdev); - return; - } - s->ivq =3D virtio_add_queue(vdev, 128, virtio_balloon_handle_output); s->dvq =3D virtio_add_queue(vdev, 128, virtio_balloon_handle_output); s->svq =3D virtio_add_queue(vdev, 128, virtio_balloon_receive_stats); - + if (virtio_has_feature(s->host_features, VIRTIO_BALLOON_F_FREE_PAGE_VQ= )) { + s->free_page_vq =3D virtio_add_queue(vdev, 128, + virtio_balloon_handle_free_page= s); + atomic_set(&s->free_page_report_status, FREE_PAGE_REPORT_S_STOP); + } reset_stats(s); } =20 @@ -471,12 +554,35 @@ static void virtio_balloon_device_reset(VirtIODevice = *vdev) static void virtio_balloon_set_status(VirtIODevice *vdev, uint8_t status) { VirtIOBalloon *s =3D VIRTIO_BALLOON(vdev); + int ret =3D 0; + + if (status & VIRTIO_CONFIG_S_DRIVER_OK) { + if (!s->stats_vq_elem && vdev->vm_running && + virtqueue_rewind(s->svq, 1)) { + /* + * Poll stats queue for the element we have discarded when the= VM + * was stopped. + */ + virtio_balloon_receive_stats(vdev, s->svq); + } =20 - if (!s->stats_vq_elem && vdev->vm_running && - (status & VIRTIO_CONFIG_S_DRIVER_OK) && virtqueue_rewind(s->svq, 1= )) { - /* poll stats queue for the element we have discarded when the VM - * was stopped */ - virtio_balloon_receive_stats(vdev, s->svq); + if (balloon_free_page_supported(s)) { + ret =3D qemu_add_balloon_handler(virtio_balloon_to_target, + virtio_balloon_stat, + virtio_balloon_free_page_suppor= t, + virtio_balloon_free_page_start, + virtio_balloon_free_page_stop, + s); + } else { + ret =3D qemu_add_balloon_handler(virtio_balloon_to_target, + virtio_balloon_stat, NULL, NULL, + NULL, s); + } + if (ret < 0) { + fprintf(stderr, "Only one balloon device is supported\n"); + virtio_cleanup(vdev); + return; + } } } =20 @@ -506,6 +612,8 @@ static const VMStateDescription vmstate_virtio_balloon = =3D { static Property virtio_balloon_properties[] =3D { DEFINE_PROP_BIT("deflate-on-oom", VirtIOBalloon, host_features, VIRTIO_BALLOON_F_DEFLATE_ON_OOM, false), + DEFINE_PROP_BIT("free-page-vq", VirtIOBalloon, host_features, + VIRTIO_BALLOON_F_FREE_PAGE_VQ, false), DEFINE_PROP_END_OF_LIST(), }; =20 diff --git a/include/hw/virtio/virtio-balloon.h b/include/hw/virtio/virtio-= balloon.h index 1ea13bd..b84b4af 100644 --- a/include/hw/virtio/virtio-balloon.h +++ b/include/hw/virtio/virtio-balloon.h @@ -31,11 +31,16 @@ typedef struct virtio_balloon_stat_modern { uint64_t val; } VirtIOBalloonStatModern; =20 +#define FREE_PAGE_REPORT_S_START 0 +#define FREE_PAGE_REPORT_S_IN_PROGRESS 1 +#define FREE_PAGE_REPORT_S_STOP 2 typedef struct VirtIOBalloon { VirtIODevice parent_obj; - VirtQueue *ivq, *dvq, *svq; + VirtQueue *ivq, *dvq, *svq, *free_page_vq; + uint32_t free_page_report_status; uint32_t num_pages; uint32_t actual; + uint32_t free_page_report_cmd_id; uint64_t stats[VIRTIO_BALLOON_S_NR]; VirtQueueElement *stats_vq_elem; size_t stats_vq_offset; diff --git a/include/migration/misc.h b/include/migration/misc.h index 77fd4f5..6df419c 100644 --- a/include/migration/misc.h +++ b/include/migration/misc.h @@ -14,11 +14,14 @@ #ifndef MIGRATION_MISC_H #define MIGRATION_MISC_H =20 +#include "exec/cpu-common.h" #include "qemu/notify.h" =20 /* migration/ram.c */ =20 void ram_mig_init(void); +void skip_free_pages_from_dirty_bitmap(RAMBlock *block, ram_addr_t offset, + size_t len); =20 /* migration/block.c */ =20 diff --git a/include/standard-headers/linux/virtio_balloon.h b/include/stan= dard-headers/linux/virtio_balloon.h index 9d06ccd..596df5d 100644 --- a/include/standard-headers/linux/virtio_balloon.h +++ b/include/standard-headers/linux/virtio_balloon.h @@ -34,15 +34,19 @@ #define VIRTIO_BALLOON_F_MUST_TELL_HOST 0 /* Tell before reclaiming pages = */ #define VIRTIO_BALLOON_F_STATS_VQ 1 /* Memory Stats virtqueue */ #define VIRTIO_BALLOON_F_DEFLATE_ON_OOM 2 /* Deflate balloon on OOM */ +#define VIRTIO_BALLOON_F_FREE_PAGE_VQ 3 /* VQ to report free pages */ =20 /* Size of a PFN in the balloon interface. */ #define VIRTIO_BALLOON_PFN_SHIFT 12 =20 +#define VIRTIO_BALLOON_FREE_PAGE_REPORT_STOP_ID 0 struct virtio_balloon_config { /* Number of pages host wants Guest to give up. */ uint32_t num_pages; /* Number of pages we've actually got in balloon. */ uint32_t actual; + /* The free_page report command id, readonly by guest */ + uint32_t free_page_report_cmd_id; }; =20 #define VIRTIO_BALLOON_S_SWAP_IN 0 /* Amount of memory swapped in */ diff --git a/include/sysemu/balloon.h b/include/sysemu/balloon.h index af49e19..cfee3ca 100644 --- a/include/sysemu/balloon.h +++ b/include/sysemu/balloon.h @@ -18,11 +18,22 @@ =20 typedef void (QEMUBalloonEvent)(void *opaque, ram_addr_t target); typedef void (QEMUBalloonStatus)(void *opaque, BalloonInfo *info); +typedef bool (QEMUBalloonFreePageSupport)(void *opaque); +typedef void (QEMUBalloonFreePageStart)(void *opaque); +typedef void (QEMUBalloonFreePageStop)(void *opaque); =20 -int qemu_add_balloon_handler(QEMUBalloonEvent *event_func, - QEMUBalloonStatus *stat_func, void *opaque); void qemu_remove_balloon_handler(void *opaque); bool qemu_balloon_is_inhibited(void); void qemu_balloon_inhibit(bool state); +bool balloon_free_page_support(void); +void balloon_free_page_start(void); +void balloon_free_page_stop(void); + +int qemu_add_balloon_handler(QEMUBalloonEvent *event_fn, + QEMUBalloonStatus *stat_fn, + QEMUBalloonFreePageSupport *free_page_support= _fn, + QEMUBalloonFreePageStart *free_page_start_fn, + QEMUBalloonFreePageStop *free_page_stop_fn, + void *opaque); =20 #endif diff --git a/migration/ram.c b/migration/ram.c index cb1950f..d6f462c 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -2186,6 +2186,16 @@ static int ram_init_all(RAMState **rsp) return 0; } =20 +void skip_free_pages_from_dirty_bitmap(RAMBlock *block, ram_addr_t offset, + size_t len) +{ + long start =3D offset >> TARGET_PAGE_BITS, + nr =3D len >> TARGET_PAGE_BITS; + + bitmap_clear(block->bmap, start, nr); + ram_state->migration_dirty_pages -=3D nr; +} + /* * Each of ram_save_setup, ram_save_iterate and ram_save_complete has * long-running RCU critical section. When rcu-reclaims in the code --=20 1.8.3.1 From nobody Sat Oct 25 23:35:03 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (208.118.235.17 [208.118.235.17]) by mx.zohomail.com with SMTPS id 151617194009022.260643992789028; Tue, 16 Jan 2018 22:52:20 -0800 (PST) Received: from localhost ([::1]:41948 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ebhZz-0005zf-Pw for importer@patchew.org; Wed, 17 Jan 2018 01:52:11 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:54618) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ebhYB-00053Y-8r for qemu-devel@nongnu.org; Wed, 17 Jan 2018 01:50:20 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ebhY9-0003pg-W7 for qemu-devel@nongnu.org; Wed, 17 Jan 2018 01:50:19 -0500 Received: from mga11.intel.com ([192.55.52.93]:59302) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1ebhY9-0003no-LR for qemu-devel@nongnu.org; Wed, 17 Jan 2018 01:50:17 -0500 Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by fmsmga102.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 16 Jan 2018 22:50:16 -0800 Received: from devel-ww.sh.intel.com ([10.239.48.110]) by fmsmga006.fm.intel.com with ESMTP; 16 Jan 2018 22:50:14 -0800 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.46,371,1511856000"; d="scan'208";a="196306519" From: Wei Wang To: qemu-devel@nongnu.org, virtio-dev@lists.oasis-open.org, mst@redhat.com, quintela@redhat.com, dgilbert@redhat.com Date: Wed, 17 Jan 2018 14:31:58 +0800 Message-Id: <1516170720-4948-3-git-send-email-wei.w.wang@intel.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1516170720-4948-1-git-send-email-wei.w.wang@intel.com> References: <1516170720-4948-1-git-send-email-wei.w.wang@intel.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 192.55.52.93 Subject: [Qemu-devel] [PATCH v1 2/4] migration: call balloon to clear bits of free pages from dirty bitmap X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yang.zhang.wz@gmail.com, quan.xu0@gmail.com, liliang.opensource@gmail.com, wei.w.wang@intel.com, pbonzini@redhat.com, nilal@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" When migration starts, call the related balloon functions to clear the bits of guest free pages from the dirty bitmap. The dirty bitmap should be ready to use when sending pages to the destination, so stop the guest from reporting free pages before sending pages. Signed-off-by: Wei Wang CC: Michael S. Tsirkin --- migration/ram.c | 28 ++++++++++++++++++++++++---- 1 file changed, 24 insertions(+), 4 deletions(-) diff --git a/migration/ram.c b/migration/ram.c index d6f462c..9b8d102 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -49,6 +49,7 @@ #include "qemu/rcu_queue.h" #include "migration/colo.h" #include "migration/block.h" +#include "sysemu/balloon.h" =20 /***********************************************************/ /* ram save/restore */ @@ -206,6 +207,10 @@ struct RAMState { uint32_t last_version; /* We are in the first round */ bool ram_bulk_stage; + /* The feature, skipping the transfer of free pages, is supported */ + bool free_page_support; + /* Skip the transfer of free pages in the bulk stage */ + bool free_page_done; /* How many times we have dirty too many pages */ int dirty_rate_high_cnt; /* these variables are used for bitmap sync */ @@ -773,7 +778,7 @@ unsigned long migration_bitmap_find_dirty(RAMState *rs,= RAMBlock *rb, unsigned long *bitmap =3D rb->bmap; unsigned long next; =20 - if (rs->ram_bulk_stage && start > 0) { + if (rs->ram_bulk_stage && start > 0 && !rs->free_page_support) { next =3D start + 1; } else { next =3D find_next_bit(bitmap, size, start); @@ -1653,6 +1658,8 @@ static void ram_state_reset(RAMState *rs) rs->last_page =3D 0; rs->last_version =3D ram_list.version; rs->ram_bulk_stage =3D true; + rs->free_page_support =3D balloon_free_page_support(); + rs->free_page_done =3D false; } =20 #define MAX_WAIT 50 /* ms, half buffered_file limit */ @@ -2135,7 +2142,7 @@ static int ram_state_init(RAMState **rsp) return 0; } =20 -static void ram_list_init_bitmaps(void) +static void ram_list_init_bitmaps(RAMState *rs) { RAMBlock *block; unsigned long pages; @@ -2145,7 +2152,11 @@ static void ram_list_init_bitmaps(void) QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { pages =3D block->max_length >> TARGET_PAGE_BITS; block->bmap =3D bitmap_new(pages); - bitmap_set(block->bmap, 0, pages); + if (rs->free_page_support) { + bitmap_set(block->bmap, 1, pages); + } else { + bitmap_set(block->bmap, 0, pages); + } if (migrate_postcopy_ram()) { block->unsentmap =3D bitmap_new(pages); bitmap_set(block->unsentmap, 0, pages); @@ -2161,8 +2172,11 @@ static void ram_init_bitmaps(RAMState *rs) qemu_mutex_lock_ramlist(); rcu_read_lock(); =20 - ram_list_init_bitmaps(); + ram_list_init_bitmaps(rs); memory_global_dirty_log_start(); + if (rs->free_page_support) { + balloon_free_page_start(); + } migration_bitmap_sync(rs); =20 rcu_read_unlock(); @@ -2275,6 +2289,12 @@ static int ram_save_iterate(QEMUFile *f, void *opaqu= e) =20 ram_control_before_iterate(f, RAM_CONTROL_ROUND); =20 + /* Before sending the pages, stop the guest from reporting free pages.= */ + if (rs->free_page_support && !rs->free_page_done) { + balloon_free_page_stop(); + rs->free_page_done =3D true; + } + t0 =3D qemu_clock_get_ns(QEMU_CLOCK_REALTIME); i =3D 0; while ((ret =3D qemu_file_rate_limit(f)) =3D=3D 0) { --=20 1.8.3.1 From nobody Sat Oct 25 23:35:03 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (208.118.235.17 [208.118.235.17]) by mx.zohomail.com with SMTPS id 1516171940090616.3872208847764; Tue, 16 Jan 2018 22:52:20 -0800 (PST) Received: from localhost ([::1]:41951 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ebha4-00064L-MK for importer@patchew.org; Wed, 17 Jan 2018 01:52:16 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:54652) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ebhYD-00053q-5b for qemu-devel@nongnu.org; Wed, 17 Jan 2018 01:50:22 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ebhYB-0003rX-Qp for qemu-devel@nongnu.org; Wed, 17 Jan 2018 01:50:21 -0500 Received: from mga11.intel.com ([192.55.52.93]:59302) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1ebhYB-0003no-H1 for qemu-devel@nongnu.org; Wed, 17 Jan 2018 01:50:19 -0500 Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by fmsmga102.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 16 Jan 2018 22:50:19 -0800 Received: from devel-ww.sh.intel.com ([10.239.48.110]) by fmsmga006.fm.intel.com with ESMTP; 16 Jan 2018 22:50:16 -0800 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.46,371,1511856000"; d="scan'208";a="196306541" From: Wei Wang To: qemu-devel@nongnu.org, virtio-dev@lists.oasis-open.org, mst@redhat.com, quintela@redhat.com, dgilbert@redhat.com Date: Wed, 17 Jan 2018 14:31:59 +0800 Message-Id: <1516170720-4948-4-git-send-email-wei.w.wang@intel.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1516170720-4948-1-git-send-email-wei.w.wang@intel.com> References: <1516170720-4948-1-git-send-email-wei.w.wang@intel.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 192.55.52.93 Subject: [Qemu-devel] [PATCH v1 3/4] virtio-balloon: add a timer to limit the free page report wating time X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yang.zhang.wz@gmail.com, quan.xu0@gmail.com, liliang.opensource@gmail.com, wei.w.wang@intel.com, pbonzini@redhat.com, nilal@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" This patch adds a timer to limit the time that the host waits for the free pages reported by the guest. Users can specify the time in ms via "free-page-wait-time" command line option. If a user doesn't specify a time, the host waits till the guest finishes reporting all the free pages. The policy (wait for all the free pages to be reported or use a time limit) is determined by the orchestration layer. Signed-off-by: Wei Wang CC: Michael S. Tsirkin --- hw/virtio/virtio-balloon.c | 85 ++++++++++++++++++++++++++++++++++= +++- hw/virtio/virtio-pci.c | 3 ++ include/hw/virtio/virtio-balloon.h | 4 ++ 3 files changed, 91 insertions(+), 1 deletion(-) diff --git a/hw/virtio/virtio-balloon.c b/hw/virtio/virtio-balloon.c index a2a5536..dcc8cb3 100644 --- a/hw/virtio/virtio-balloon.c +++ b/hw/virtio/virtio-balloon.c @@ -213,6 +213,66 @@ static void balloon_stats_set_poll_interval(Object *ob= j, Visitor *v, balloon_stats_change_timer(s, 0); } =20 +static void balloon_free_page_change_timer(VirtIOBalloon *s, int64_t ms) +{ + timer_mod(s->free_page_timer, + qemu_clock_get_ms(QEMU_CLOCK_REALTIME) + ms); +} + +static void balloon_stop_free_page_report(void *opaque) +{ + VirtIOBalloon *dev =3D opaque; + VirtIODevice *vdev =3D VIRTIO_DEVICE(dev); + + timer_del(dev->free_page_timer); + timer_free(dev->free_page_timer); + dev->free_page_timer =3D NULL; + + if (atomic_read(&dev->free_page_report_status) =3D=3D + FREE_PAGE_REPORT_S_IN_PROGRESS) { + dev->host_stop_free_page =3D true; + virtio_notify_config(vdev); + } +} + +static void balloon_free_page_get_wait_time(Object *obj, Visitor *v, + const char *name, void *opaque, + Error **errp) +{ + VirtIOBalloon *s =3D opaque; + + visit_type_int(v, name, &s->free_page_wait_time, errp); +} + +static void balloon_free_page_set_wait_time(Object *obj, Visitor *v, + const char *name, void *opaque, + Error **errp) +{ + VirtIOBalloon *s =3D opaque; + Error *local_err =3D NULL; + int64_t value; + + visit_type_int(v, name, &value, &local_err); + if (local_err) { + error_propagate(errp, local_err); + return; + } + if (value < 0) { + error_setg(errp, "free page wait time must be greater than zero"); + return; + } + + if (value > UINT32_MAX) { + error_setg(errp, "free page wait time value is too big"); + return; + } + + s->free_page_wait_time =3D value; + g_assert(s->free_page_timer =3D=3D NULL); + s->free_page_timer =3D timer_new_ms(QEMU_CLOCK_REALTIME, + balloon_stop_free_page_report, s); +} + static void virtio_balloon_handle_output(VirtIODevice *vdev, VirtQueue *vq) { VirtIOBalloon *s =3D VIRTIO_BALLOON(vdev); @@ -332,6 +392,7 @@ static void virtio_balloon_handle_free_pages(VirtIODevi= ce *vdev, VirtQueue *vq) atomic_set(&dev->free_page_report_status, FREE_PAGE_REPORT_S_IN_PROGRESS); } else { + dev->host_stop_free_page =3D false; atomic_set(&dev->free_page_report_status, FREE_PAGE_REPORT_S_STOP); } @@ -386,6 +447,10 @@ static void virtio_balloon_free_page_stop(void *opaque) return; } =20 + if (dev->free_page_wait_time) { + balloon_free_page_change_timer(dev, dev->free_page_wait_time); + } + /* Wait till a stop sign is received from the guest */ while (atomic_read(&dev->free_page_report_status) !=3D FREE_PAGE_REPORT_S_STOP) @@ -399,7 +464,19 @@ static void virtio_balloon_get_config(VirtIODevice *vd= ev, uint8_t *config_data) =20 config.num_pages =3D cpu_to_le32(dev->num_pages); config.actual =3D cpu_to_le32(dev->actual); - config.free_page_report_cmd_id =3D cpu_to_le32(dev->free_page_report_c= md_id); + if (dev->host_stop_free_page) { + /* + * Host is actively requesting to stop the free page report, send = the + * stop sign to the guest. This happens when the migration thread = has + * reached the phase to send pages to the destination while the gu= est + * hasn't done the reporting. + */ + config.free_page_report_cmd_id =3D + VIRTIO_BALLOON_FREE_PAGE_REPORT_STOP_I= D; + } else { + config.free_page_report_cmd_id =3D + cpu_to_le32(dev->free_page_report_cmd_= id); + } =20 trace_virtio_balloon_get_config(config.num_pages, config.actual); memcpy(config_data, &config, sizeof(struct virtio_balloon_config)); @@ -526,6 +603,7 @@ static void virtio_balloon_device_realize(DeviceState *= dev, Error **errp) s->free_page_vq =3D virtio_add_queue(vdev, 128, virtio_balloon_handle_free_page= s); atomic_set(&s->free_page_report_status, FREE_PAGE_REPORT_S_STOP); + s->host_stop_free_page =3D false; } reset_stats(s); } @@ -597,6 +675,11 @@ static void virtio_balloon_instance_init(Object *obj) balloon_stats_get_poll_interval, balloon_stats_set_poll_interval, NULL, s, NULL); + + object_property_add(obj, "free-page-wait-time", "int", + balloon_free_page_get_wait_time, + balloon_free_page_set_wait_time, + NULL, s, NULL); } =20 static const VMStateDescription vmstate_virtio_balloon =3D { diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c index 6c75cca..3345104 100644 --- a/hw/virtio/virtio-pci.c +++ b/hw/virtio/virtio-pci.c @@ -2250,6 +2250,9 @@ static void virtio_balloon_pci_instance_init(Object *= obj) object_property_add_alias(obj, "guest-stats-polling-interval", OBJECT(&dev->vdev), "guest-stats-polling-interval", &error_abort= ); + object_property_add_alias(obj, "free-page-wait-time", + OBJECT(&dev->vdev), + "free-page-wait-time", &error_abort); } =20 static const TypeInfo virtio_balloon_pci_info =3D { diff --git a/include/hw/virtio/virtio-balloon.h b/include/hw/virtio/virtio-= balloon.h index b84b4af..268f7d6 100644 --- a/include/hw/virtio/virtio-balloon.h +++ b/include/hw/virtio/virtio-balloon.h @@ -37,6 +37,8 @@ typedef struct virtio_balloon_stat_modern { typedef struct VirtIOBalloon { VirtIODevice parent_obj; VirtQueue *ivq, *dvq, *svq, *free_page_vq; + /* The host is requesting the guest to stop free page reporting */ + bool host_stop_free_page; uint32_t free_page_report_status; uint32_t num_pages; uint32_t actual; @@ -45,8 +47,10 @@ typedef struct VirtIOBalloon { VirtQueueElement *stats_vq_elem; size_t stats_vq_offset; QEMUTimer *stats_timer; + QEMUTimer *free_page_timer; int64_t stats_last_update; int64_t stats_poll_interval; + int64_t free_page_wait_time; uint32_t host_features; } VirtIOBalloon; =20 --=20 1.8.3.1 From nobody Sat Oct 25 23:35:03 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1516172074672756.7217062195137; Tue, 16 Jan 2018 22:54:34 -0800 (PST) Received: from localhost ([::1]:42033 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ebhcE-0007qi-Fn for importer@patchew.org; Wed, 17 Jan 2018 01:54:30 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:54663) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ebhYF-00055a-JQ for qemu-devel@nongnu.org; Wed, 17 Jan 2018 01:50:24 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ebhYE-0003tR-4b for qemu-devel@nongnu.org; Wed, 17 Jan 2018 01:50:23 -0500 Received: from mga11.intel.com ([192.55.52.93]:59302) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1ebhYD-0003no-Sj for qemu-devel@nongnu.org; Wed, 17 Jan 2018 01:50:22 -0500 Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by fmsmga102.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 16 Jan 2018 22:50:21 -0800 Received: from devel-ww.sh.intel.com ([10.239.48.110]) by fmsmga006.fm.intel.com with ESMTP; 16 Jan 2018 22:50:19 -0800 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.46,371,1511856000"; d="scan'208";a="196306562" From: Wei Wang To: qemu-devel@nongnu.org, virtio-dev@lists.oasis-open.org, mst@redhat.com, quintela@redhat.com, dgilbert@redhat.com Date: Wed, 17 Jan 2018 14:32:00 +0800 Message-Id: <1516170720-4948-5-git-send-email-wei.w.wang@intel.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1516170720-4948-1-git-send-email-wei.w.wang@intel.com> References: <1516170720-4948-1-git-send-email-wei.w.wang@intel.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 192.55.52.93 Subject: [Qemu-devel] [PATCH v1 4/4] virtio-balloon: Don't skip free pages if the poison val is non-zero X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yang.zhang.wz@gmail.com, quan.xu0@gmail.com, liliang.opensource@gmail.com, wei.w.wang@intel.com, pbonzini@redhat.com, nilal@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" If the guest is using a non-zero poisoning, we don't skip the transfer of guest free pages. Todo: As a next step optimization, we can try 1) skip the transfer of guest poisoned free pages; 2) send the poison value to destination; and 3) seek a way to poison the guest free pages before the guest starts to run on the destination. Signed-off-by: Wei Wang CC: Michael S. Tsirkin --- hw/virtio/virtio-balloon.c | 3 ++- include/hw/virtio/virtio-balloon.h | 1 + include/standard-headers/linux/virtio_balloon.h | 2 ++ 3 files changed, 5 insertions(+), 1 deletion(-) diff --git a/hw/virtio/virtio-balloon.c b/hw/virtio/virtio-balloon.c index dcc8cb3..0cdc755 100644 --- a/hw/virtio/virtio-balloon.c +++ b/hw/virtio/virtio-balloon.c @@ -403,7 +403,7 @@ static void virtio_balloon_handle_free_pages(VirtIODevi= ce *vdev, VirtQueue *vq) ram_addr_t offset; =20 if (atomic_read(&dev->free_page_report_status) =3D=3D - FREE_PAGE_REPORT_S_IN_PROGRESS) { + FREE_PAGE_REPORT_S_IN_PROGRESS && !dev->poison_val) { block =3D qemu_ram_block_from_host(elem->in_sg[0].iov_base, false, &offset); size =3D elem->in_sg[0].iov_len; @@ -530,6 +530,7 @@ static void virtio_balloon_set_config(VirtIODevice *vde= v, ((ram_addr_t) dev->actual << VIRTIO_BALLOON_PFN_SH= IFT), &error_abort); } + dev->poison_val =3D le32_to_cpu(config.poison_val); trace_virtio_balloon_set_config(dev->actual, oldactual); } =20 diff --git a/include/hw/virtio/virtio-balloon.h b/include/hw/virtio/virtio-= balloon.h index 268f7d6..0bb80cd 100644 --- a/include/hw/virtio/virtio-balloon.h +++ b/include/hw/virtio/virtio-balloon.h @@ -43,6 +43,7 @@ typedef struct VirtIOBalloon { uint32_t num_pages; uint32_t actual; uint32_t free_page_report_cmd_id; + uint32_t poison_val; uint64_t stats[VIRTIO_BALLOON_S_NR]; VirtQueueElement *stats_vq_elem; size_t stats_vq_offset; diff --git a/include/standard-headers/linux/virtio_balloon.h b/include/stan= dard-headers/linux/virtio_balloon.h index 596df5d..f6f4ff3 100644 --- a/include/standard-headers/linux/virtio_balloon.h +++ b/include/standard-headers/linux/virtio_balloon.h @@ -47,6 +47,8 @@ struct virtio_balloon_config { uint32_t actual; /* The free_page report command id, readonly by guest */ uint32_t free_page_report_cmd_id; + /* Stores PAGE_POISON if page poisoning with sanity check is in use */ + uint32_t poison_val; }; =20 #define VIRTIO_BALLOON_S_SWAP_IN 0 /* Amount of memory swapped in */ --=20 1.8.3.1