From nobody Thu May 2 12:39:56 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1517916680013911.7535359667945; Tue, 6 Feb 2018 03:31:20 -0800 (PST) Received: from localhost ([::1]:58487 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ej1T0-0006WK-Ta for importer@patchew.org; Tue, 06 Feb 2018 06:31:14 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:34031) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ej1PK-0003RD-It for qemu-devel@nongnu.org; Tue, 06 Feb 2018 06:27:28 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ej1PD-0000z4-LR for qemu-devel@nongnu.org; Tue, 06 Feb 2018 06:27:26 -0500 Received: from mga14.intel.com ([192.55.52.115]:57906) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1ej1PD-0000xO-9F for qemu-devel@nongnu.org; Tue, 06 Feb 2018 06:27:19 -0500 Received: from orsmga001.jf.intel.com ([10.7.209.18]) by fmsmga103.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 06 Feb 2018 03:27:16 -0800 Received: from devel-ww.sh.intel.com ([10.239.48.110]) by orsmga001.jf.intel.com with ESMTP; 06 Feb 2018 03:27:14 -0800 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.46,468,1511856000"; d="scan'208";a="29184980" From: Wei Wang To: qemu-devel@nongnu.org, virtio-dev@lists.oasis-open.org, mst@redhat.com, quintela@redhat.com, dgilbert@redhat.com Date: Tue, 6 Feb 2018 19:08:17 +0800 Message-Id: <1517915299-15349-2-git-send-email-wei.w.wang@intel.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1517915299-15349-1-git-send-email-wei.w.wang@intel.com> References: <1517915299-15349-1-git-send-email-wei.w.wang@intel.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 192.55.52.115 Subject: [Qemu-devel] [PATCH v2 1/3] virtio-balloon: VIRTIO_BALLOON_F_FREE_PAGE_HINT X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yang.zhang.wz@gmail.com, quan.xu0@gmail.com, liliang.opensource@gmail.com, wei.w.wang@intel.com, pbonzini@redhat.com, nilal@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" The new feature enables the virtio-balloon device to receive the hint of guest free pages from the free page vq, and clears the corresponding bits of the free page from the dirty bitmap, so that those free pages are not transferred by the migration thread. Signed-off-by: Wei Wang Signed-off-by: Liang Li CC: Michael S. Tsirkin CC: Juan Quintela --- balloon.c | 39 +++++-- hw/virtio/virtio-balloon.c | 145 ++++++++++++++++++++= +--- include/hw/virtio/virtio-balloon.h | 11 +- include/migration/misc.h | 3 + include/standard-headers/linux/virtio_balloon.h | 7 ++ include/sysemu/balloon.h | 12 +- migration/ram.c | 10 ++ 7 files changed, 198 insertions(+), 29 deletions(-) diff --git a/balloon.c b/balloon.c index 1d720ff..0f0b30c 100644 --- a/balloon.c +++ b/balloon.c @@ -36,6 +36,8 @@ =20 static QEMUBalloonEvent *balloon_event_fn; static QEMUBalloonStatus *balloon_stat_fn; +static QEMUBalloonFreePageSupport *balloon_free_page_support_fn; +static QEMUBalloonFreePagePoll *balloon_free_page_poll_fn; static void *balloon_opaque; static bool balloon_inhibited; =20 @@ -64,19 +66,34 @@ static bool have_balloon(Error **errp) return true; } =20 -int qemu_add_balloon_handler(QEMUBalloonEvent *event_func, - QEMUBalloonStatus *stat_func, void *opaque) +bool balloon_free_page_support(void) { - if (balloon_event_fn || balloon_stat_fn || balloon_opaque) { - /* We're already registered one balloon handler. How many can - * a guest really have? - */ - return -1; + return balloon_free_page_support_fn && + balloon_free_page_support_fn(balloon_opaque); +} + +void balloon_free_page_poll(void) +{ + balloon_free_page_poll_fn(balloon_opaque); +} + +void qemu_add_balloon_handler(QEMUBalloonEvent *event_fn, + QEMUBalloonStatus *stat_fn, + QEMUBalloonFreePageSupport *free_page_suppor= t_fn, + QEMUBalloonFreePagePoll *free_page_poll_fn, + void *opaque) +{ + if (balloon_event_fn || balloon_stat_fn || balloon_free_page_support_f= n || + balloon_free_page_poll_fn || balloon_opaque) { + /* We already registered one balloon handler. */ + return; } - balloon_event_fn =3D event_func; - balloon_stat_fn =3D stat_func; + + balloon_event_fn =3D event_fn; + balloon_stat_fn =3D stat_fn; + balloon_free_page_support_fn =3D free_page_support_fn; + balloon_free_page_poll_fn =3D free_page_poll_fn; balloon_opaque =3D opaque; - return 0; } =20 void qemu_remove_balloon_handler(void *opaque) @@ -86,6 +103,8 @@ void qemu_remove_balloon_handler(void *opaque) } balloon_event_fn =3D NULL; balloon_stat_fn =3D NULL; + balloon_free_page_support_fn =3D NULL; + balloon_free_page_poll_fn =3D NULL; balloon_opaque =3D NULL; } =20 diff --git a/hw/virtio/virtio-balloon.c b/hw/virtio/virtio-balloon.c index 14e08d2..b424d4e 100644 --- a/hw/virtio/virtio-balloon.c +++ b/hw/virtio/virtio-balloon.c @@ -23,6 +23,7 @@ #include "hw/virtio/virtio-balloon.h" #include "sysemu/kvm.h" #include "exec/address-spaces.h" +#include "exec/ram_addr.h" #include "qapi/visitor.h" #include "qapi-event.h" #include "trace.h" @@ -30,6 +31,7 @@ =20 #include "hw/virtio/virtio-bus.h" #include "hw/virtio/virtio-access.h" +#include "migration/misc.h" =20 #define BALLOON_PAGE_SIZE (1 << VIRTIO_BALLOON_PFN_SHIFT) =20 @@ -305,6 +307,87 @@ out: } } =20 +static void virtio_balloon_poll_free_page_hints(VirtIOBalloon *dev) +{ + VirtQueueElement *elem; + VirtQueue *vq =3D dev->free_page_vq; + VirtIODevice *vdev =3D VIRTIO_DEVICE(dev); + bool page_poisoning =3D virtio_vdev_has_feature(vdev, + VIRTIO_BALLOON_F_PAGE_POISON= ); + uint32_t id; + + /* Poll the vq till a stop cmd id is received */ + while (dev->free_page_report_status !=3D FREE_PAGE_REPORT_S_STOP) { + elem =3D virtqueue_pop(vq, sizeof(VirtQueueElement)); + if (!elem) { + continue; + } + + if (elem->out_num) { + iov_to_buf(elem->out_sg, elem->out_num, 0, &id, sizeof(uint32_= t)); + virtqueue_push(vq, elem, sizeof(id)); + g_free(elem); + if (id =3D=3D dev->free_page_report_cmd_id) { + dev->free_page_report_status =3D FREE_PAGE_REPORT_S_START; + } else { + dev->free_page_report_status =3D FREE_PAGE_REPORT_S_STOP; + break; + } + } + + if (elem->in_num) { + RAMBlock *block; + ram_addr_t offset; + void *base; + size_t total_len, len; + + if (dev->free_page_report_status =3D=3D FREE_PAGE_REPORT_S_STA= RT && + !page_poisoning) { + base =3D elem->in_sg[0].iov_base; + total_len =3D elem->in_sg[0].iov_len; + len =3D total_len; + + while (total_len > 0) { + block =3D qemu_ram_block_from_host(base, false, &offse= t); + if (unlikely(offset + total_len > block->used_length))= { + len =3D block->used_length - offset; + base +=3D len; + } + skip_free_pages_from_dirty_bitmap(block, offset, len); + total_len -=3D len; + } + } + virtqueue_push(vq, elem, total_len); + g_free(elem); + } + } +} + +static bool virtio_balloon_free_page_support(void *opaque) +{ + VirtIOBalloon *s =3D opaque; + VirtIODevice *vdev =3D VIRTIO_DEVICE(s); + + return virtio_vdev_has_feature(vdev, VIRTIO_BALLOON_F_FREE_PAGE_HINT); +} + +static void virtio_balloon_free_page_poll(void *opaque) +{ + VirtIOBalloon *s =3D opaque; + VirtIODevice *vdev =3D VIRTIO_DEVICE(s); + + if (unlikely(s->free_page_report_cmd_id =3D=3D UINT_MAX)) { + s->free_page_report_cmd_id =3D 1; + } else { + s->free_page_report_cmd_id++; + } + + virtio_notify_config(vdev); + s->free_page_report_status =3D FREE_PAGE_REPORT_S_REQUESTED; + + virtio_balloon_poll_free_page_hints(s); +} + static void virtio_balloon_get_config(VirtIODevice *vdev, uint8_t *config_= data) { VirtIOBalloon *dev =3D VIRTIO_BALLOON(vdev); @@ -312,6 +395,7 @@ static void virtio_balloon_get_config(VirtIODevice *vde= v, uint8_t *config_data) =20 config.num_pages =3D cpu_to_le32(dev->num_pages); config.actual =3D cpu_to_le32(dev->actual); + config.free_page_report_cmd_id =3D cpu_to_le32(dev->free_page_report_c= md_id); =20 trace_virtio_balloon_get_config(config.num_pages, config.actual); memcpy(config_data, &config, sizeof(struct virtio_balloon_config)); @@ -365,6 +449,7 @@ static void virtio_balloon_set_config(VirtIODevice *vde= v, ((ram_addr_t) dev->actual << VIRTIO_BALLOON_PFN_SH= IFT), &error_abort); } + dev->poison_val =3D le32_to_cpu(config.poison_val); trace_virtio_balloon_set_config(dev->actual, oldactual); } =20 @@ -374,6 +459,7 @@ static uint64_t virtio_balloon_get_features(VirtIODevic= e *vdev, uint64_t f, VirtIOBalloon *dev =3D VIRTIO_BALLOON(vdev); f |=3D dev->host_features; virtio_add_feature(&f, VIRTIO_BALLOON_F_STATS_VQ); + virtio_add_feature(&f, VIRTIO_BALLOON_F_PAGE_POISON); return f; } =20 @@ -410,6 +496,17 @@ static int virtio_balloon_post_load_device(void *opaqu= e, int version_id) return 0; } =20 +static const VMStateDescription vmstate_virtio_balloon_free_page_report = =3D { + .name =3D "virtio-balloon-device/free-page-report", + .version_id =3D 1, + .minimum_version_id =3D 1, + .needed =3D virtio_balloon_free_page_support, + .fields =3D (VMStateField[]) { + VMSTATE_UINT32(free_page_report_cmd_id, VirtIOBalloon), + VMSTATE_END_OF_LIST() + } +}; + static const VMStateDescription vmstate_virtio_balloon_device =3D { .name =3D "virtio-balloon-device", .version_id =3D 1, @@ -420,30 +517,29 @@ static const VMStateDescription vmstate_virtio_balloo= n_device =3D { VMSTATE_UINT32(actual, VirtIOBalloon), VMSTATE_END_OF_LIST() }, + .subsections =3D (const VMStateDescription * []) { + &vmstate_virtio_balloon_free_page_report, + NULL + } }; =20 static void virtio_balloon_device_realize(DeviceState *dev, Error **errp) { VirtIODevice *vdev =3D VIRTIO_DEVICE(dev); VirtIOBalloon *s =3D VIRTIO_BALLOON(dev); - int ret; =20 virtio_init(vdev, "virtio-balloon", VIRTIO_ID_BALLOON, sizeof(struct virtio_balloon_config)); =20 - ret =3D qemu_add_balloon_handler(virtio_balloon_to_target, - virtio_balloon_stat, s); - - if (ret < 0) { - error_setg(errp, "Only one balloon device is supported"); - virtio_cleanup(vdev); - return; - } - s->ivq =3D virtio_add_queue(vdev, 128, virtio_balloon_handle_output); s->dvq =3D virtio_add_queue(vdev, 128, virtio_balloon_handle_output); s->svq =3D virtio_add_queue(vdev, 128, virtio_balloon_receive_stats); - + if (virtio_has_feature(s->host_features, + VIRTIO_BALLOON_F_FREE_PAGE_HINT)) { + s->free_page_vq =3D virtio_add_queue(vdev, VIRTQUEUE_MAX_SIZE, NUL= L); + s->free_page_report_status =3D FREE_PAGE_REPORT_S_STOP; + s->free_page_report_cmd_id =3D VIRTIO_BALLOON_FREE_PAGE_REPORT_STO= P_ID; + } reset_stats(s); } =20 @@ -472,11 +568,26 @@ static void virtio_balloon_set_status(VirtIODevice *v= dev, uint8_t status) { VirtIOBalloon *s =3D VIRTIO_BALLOON(vdev); =20 - if (!s->stats_vq_elem && vdev->vm_running && - (status & VIRTIO_CONFIG_S_DRIVER_OK) && virtqueue_rewind(s->svq, 1= )) { - /* poll stats queue for the element we have discarded when the VM - * was stopped */ - virtio_balloon_receive_stats(vdev, s->svq); + if (status & VIRTIO_CONFIG_S_DRIVER_OK) { + if (!s->stats_vq_elem && vdev->vm_running && + virtqueue_rewind(s->svq, 1)) { + /* + * Poll stats queue for the element we have discarded when the= VM + * was stopped. + */ + virtio_balloon_receive_stats(vdev, s->svq); + } + + if (virtio_balloon_free_page_support(s)) { + qemu_add_balloon_handler(virtio_balloon_to_target, + virtio_balloon_stat, + virtio_balloon_free_page_support, + virtio_balloon_free_page_poll, + s); + } else { + qemu_add_balloon_handler(virtio_balloon_to_target, + virtio_balloon_stat, NULL, NULL, s); + } } } =20 @@ -506,6 +617,8 @@ static const VMStateDescription vmstate_virtio_balloon = =3D { static Property virtio_balloon_properties[] =3D { DEFINE_PROP_BIT("deflate-on-oom", VirtIOBalloon, host_features, VIRTIO_BALLOON_F_DEFLATE_ON_OOM, false), + DEFINE_PROP_BIT("free-page-hint", VirtIOBalloon, host_features, + VIRTIO_BALLOON_F_FREE_PAGE_HINT, false), DEFINE_PROP_END_OF_LIST(), }; =20 diff --git a/include/hw/virtio/virtio-balloon.h b/include/hw/virtio/virtio-= balloon.h index 1ea13bd..11b4e01 100644 --- a/include/hw/virtio/virtio-balloon.h +++ b/include/hw/virtio/virtio-balloon.h @@ -31,11 +31,20 @@ typedef struct virtio_balloon_stat_modern { uint64_t val; } VirtIOBalloonStatModern; =20 +enum virtio_balloon_free_page_report_status { + FREE_PAGE_REPORT_S_REQUESTED, + FREE_PAGE_REPORT_S_START, + FREE_PAGE_REPORT_S_STOP, +}; + typedef struct VirtIOBalloon { VirtIODevice parent_obj; - VirtQueue *ivq, *dvq, *svq; + VirtQueue *ivq, *dvq, *svq, *free_page_vq; + uint32_t free_page_report_status; uint32_t num_pages; uint32_t actual; + uint32_t free_page_report_cmd_id; + uint32_t poison_val; uint64_t stats[VIRTIO_BALLOON_S_NR]; VirtQueueElement *stats_vq_elem; size_t stats_vq_offset; diff --git a/include/migration/misc.h b/include/migration/misc.h index 77fd4f5..6df419c 100644 --- a/include/migration/misc.h +++ b/include/migration/misc.h @@ -14,11 +14,14 @@ #ifndef MIGRATION_MISC_H #define MIGRATION_MISC_H =20 +#include "exec/cpu-common.h" #include "qemu/notify.h" =20 /* migration/ram.c */ =20 void ram_mig_init(void); +void skip_free_pages_from_dirty_bitmap(RAMBlock *block, ram_addr_t offset, + size_t len); =20 /* migration/block.c */ =20 diff --git a/include/standard-headers/linux/virtio_balloon.h b/include/stan= dard-headers/linux/virtio_balloon.h index 9d06ccd..19d0d8b 100644 --- a/include/standard-headers/linux/virtio_balloon.h +++ b/include/standard-headers/linux/virtio_balloon.h @@ -34,15 +34,22 @@ #define VIRTIO_BALLOON_F_MUST_TELL_HOST 0 /* Tell before reclaiming pages = */ #define VIRTIO_BALLOON_F_STATS_VQ 1 /* Memory Stats virtqueue */ #define VIRTIO_BALLOON_F_DEFLATE_ON_OOM 2 /* Deflate balloon on OOM */ +#define VIRTIO_BALLOON_F_FREE_PAGE_HINT 3 /* VQ to report free pages */ +#define VIRTIO_BALLOON_F_PAGE_POISON 4 /* Guest is using page poisoning */ =20 /* Size of a PFN in the balloon interface. */ #define VIRTIO_BALLOON_PFN_SHIFT 12 =20 +#define VIRTIO_BALLOON_FREE_PAGE_REPORT_STOP_ID 0 struct virtio_balloon_config { /* Number of pages host wants Guest to give up. */ uint32_t num_pages; /* Number of pages we've actually got in balloon. */ uint32_t actual; + /* Free page report command id, readonly by guest */ + uint32_t free_page_report_cmd_id; + /* Stores PAGE_POISON if page poisoning is in use */ + uint32_t poison_val; }; =20 #define VIRTIO_BALLOON_S_SWAP_IN 0 /* Amount of memory swapped in */ diff --git a/include/sysemu/balloon.h b/include/sysemu/balloon.h index af49e19..53db4dc 100644 --- a/include/sysemu/balloon.h +++ b/include/sysemu/balloon.h @@ -18,11 +18,19 @@ =20 typedef void (QEMUBalloonEvent)(void *opaque, ram_addr_t target); typedef void (QEMUBalloonStatus)(void *opaque, BalloonInfo *info); +typedef bool (QEMUBalloonFreePageSupport)(void *opaque); +typedef void (QEMUBalloonFreePagePoll)(void *opaque); =20 -int qemu_add_balloon_handler(QEMUBalloonEvent *event_func, - QEMUBalloonStatus *stat_func, void *opaque); void qemu_remove_balloon_handler(void *opaque); bool qemu_balloon_is_inhibited(void); void qemu_balloon_inhibit(bool state); +bool balloon_free_page_support(void); +void balloon_free_page_poll(void); + +void qemu_add_balloon_handler(QEMUBalloonEvent *event_fn, + QEMUBalloonStatus *stat_fn, + QEMUBalloonFreePageSupport *free_page_suppor= t_fn, + QEMUBalloonFreePagePoll *free_page_poll_fn, + void *opaque); =20 #endif diff --git a/migration/ram.c b/migration/ram.c index cb1950f..d6f462c 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -2186,6 +2186,16 @@ static int ram_init_all(RAMState **rsp) return 0; } =20 +void skip_free_pages_from_dirty_bitmap(RAMBlock *block, ram_addr_t offset, + size_t len) +{ + long start =3D offset >> TARGET_PAGE_BITS, + nr =3D len >> TARGET_PAGE_BITS; + + bitmap_clear(block->bmap, start, nr); + ram_state->migration_dirty_pages -=3D nr; +} + /* * Each of ram_save_setup, ram_save_iterate and ram_save_complete has * long-running RCU critical section. When rcu-reclaims in the code --=20 1.8.3.1 From nobody Thu May 2 12:39:56 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1517916727791451.94190481620126; Tue, 6 Feb 2018 03:32:07 -0800 (PST) Received: from localhost ([::1]:58521 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ej1Tq-0007F5-TM for importer@patchew.org; Tue, 06 Feb 2018 06:32:06 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:34007) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ej1PG-0003Mr-P7 for qemu-devel@nongnu.org; Tue, 06 Feb 2018 06:27:23 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ej1PE-0000zM-4W for qemu-devel@nongnu.org; Tue, 06 Feb 2018 06:27:22 -0500 Received: from mga14.intel.com ([192.55.52.115]:57904) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1ej1PD-0000w9-QQ for qemu-devel@nongnu.org; Tue, 06 Feb 2018 06:27:20 -0500 Received: from orsmga001.jf.intel.com ([10.7.209.18]) by fmsmga103.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 06 Feb 2018 03:27:18 -0800 Received: from devel-ww.sh.intel.com ([10.239.48.110]) by orsmga001.jf.intel.com with ESMTP; 06 Feb 2018 03:27:16 -0800 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.46,468,1511856000"; d="scan'208";a="29184987" From: Wei Wang To: qemu-devel@nongnu.org, virtio-dev@lists.oasis-open.org, mst@redhat.com, quintela@redhat.com, dgilbert@redhat.com Date: Tue, 6 Feb 2018 19:08:18 +0800 Message-Id: <1517915299-15349-3-git-send-email-wei.w.wang@intel.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1517915299-15349-1-git-send-email-wei.w.wang@intel.com> References: <1517915299-15349-1-git-send-email-wei.w.wang@intel.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 192.55.52.115 Subject: [Qemu-devel] [PATCH v2 2/3] migration: use the free page reporting feature from balloon X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yang.zhang.wz@gmail.com, quan.xu0@gmail.com, liliang.opensource@gmail.com, wei.w.wang@intel.com, pbonzini@redhat.com, nilal@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Use the free page reporting feature from the balloon device to clear the bits corresponding to guest free pages from the dirty bitmap, so that the free memory are not sent. Signed-off-by: Wei Wang CC: Michael S. Tsirkin CC: Juan Quintela --- migration/ram.c | 24 ++++++++++++++++++++---- 1 file changed, 20 insertions(+), 4 deletions(-) diff --git a/migration/ram.c b/migration/ram.c index d6f462c..4fe16d2 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -49,6 +49,7 @@ #include "qemu/rcu_queue.h" #include "migration/colo.h" #include "migration/block.h" +#include "sysemu/balloon.h" =20 /***********************************************************/ /* ram save/restore */ @@ -206,6 +207,10 @@ struct RAMState { uint32_t last_version; /* We are in the first round */ bool ram_bulk_stage; + /* The feature, skipping the transfer of free pages, is supported */ + bool free_page_support; + /* Skip the transfer of free pages in the bulk stage */ + bool free_page_done; /* How many times we have dirty too many pages */ int dirty_rate_high_cnt; /* these variables are used for bitmap sync */ @@ -773,7 +778,7 @@ unsigned long migration_bitmap_find_dirty(RAMState *rs,= RAMBlock *rb, unsigned long *bitmap =3D rb->bmap; unsigned long next; =20 - if (rs->ram_bulk_stage && start > 0) { + if (rs->ram_bulk_stage && start > 0 && !rs->free_page_support) { next =3D start + 1; } else { next =3D find_next_bit(bitmap, size, start); @@ -1653,6 +1658,8 @@ static void ram_state_reset(RAMState *rs) rs->last_page =3D 0; rs->last_version =3D ram_list.version; rs->ram_bulk_stage =3D true; + rs->free_page_support =3D balloon_free_page_support(); + rs->free_page_done =3D false; } =20 #define MAX_WAIT 50 /* ms, half buffered_file limit */ @@ -2135,7 +2142,7 @@ static int ram_state_init(RAMState **rsp) return 0; } =20 -static void ram_list_init_bitmaps(void) +static void ram_list_init_bitmaps(RAMState *rs) { RAMBlock *block; unsigned long pages; @@ -2145,7 +2152,11 @@ static void ram_list_init_bitmaps(void) QLIST_FOREACH_RCU(block, &ram_list.blocks, next) { pages =3D block->max_length >> TARGET_PAGE_BITS; block->bmap =3D bitmap_new(pages); - bitmap_set(block->bmap, 0, pages); + if (rs->free_page_support) { + bitmap_set(block->bmap, 1, pages); + } else { + bitmap_set(block->bmap, 0, pages); + } if (migrate_postcopy_ram()) { block->unsentmap =3D bitmap_new(pages); bitmap_set(block->unsentmap, 0, pages); @@ -2161,7 +2172,7 @@ static void ram_init_bitmaps(RAMState *rs) qemu_mutex_lock_ramlist(); rcu_read_lock(); =20 - ram_list_init_bitmaps(); + ram_list_init_bitmaps(rs); memory_global_dirty_log_start(); migration_bitmap_sync(rs); =20 @@ -2275,6 +2286,11 @@ static int ram_save_iterate(QEMUFile *f, void *opaqu= e) =20 ram_control_before_iterate(f, RAM_CONTROL_ROUND); =20 + if (rs->free_page_support && !rs->free_page_done) { + balloon_free_page_poll(); + rs->free_page_done =3D true; + } + t0 =3D qemu_clock_get_ns(QEMU_CLOCK_REALTIME); i =3D 0; while ((ret =3D qemu_file_rate_limit(f)) =3D=3D 0) { --=20 1.8.3.1 From nobody Thu May 2 12:39:56 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (208.118.235.17 [208.118.235.17]) by mx.zohomail.com with SMTPS id 1517916848111786.2187640309394; Tue, 6 Feb 2018 03:34:08 -0800 (PST) Received: from localhost ([::1]:58760 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ej1Vi-0000Vb-3o for importer@patchew.org; Tue, 06 Feb 2018 06:34:02 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:34023) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ej1PH-0003Nh-Dx for qemu-devel@nongnu.org; Tue, 06 Feb 2018 06:27:29 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ej1PF-00010L-SD for qemu-devel@nongnu.org; Tue, 06 Feb 2018 06:27:23 -0500 Received: from mga14.intel.com ([192.55.52.115]:57904) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1ej1PF-0000w9-Gx for qemu-devel@nongnu.org; Tue, 06 Feb 2018 06:27:21 -0500 Received: from orsmga001.jf.intel.com ([10.7.209.18]) by fmsmga103.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 06 Feb 2018 03:27:21 -0800 Received: from devel-ww.sh.intel.com ([10.239.48.110]) by orsmga001.jf.intel.com with ESMTP; 06 Feb 2018 03:27:18 -0800 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.46,468,1511856000"; d="scan'208";a="29184993" From: Wei Wang To: qemu-devel@nongnu.org, virtio-dev@lists.oasis-open.org, mst@redhat.com, quintela@redhat.com, dgilbert@redhat.com Date: Tue, 6 Feb 2018 19:08:19 +0800 Message-Id: <1517915299-15349-4-git-send-email-wei.w.wang@intel.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1517915299-15349-1-git-send-email-wei.w.wang@intel.com> References: <1517915299-15349-1-git-send-email-wei.w.wang@intel.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 192.55.52.115 Subject: [Qemu-devel] [PATCH v2 3/3] virtio-balloon: add a timer to limit the free page report waiting time X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yang.zhang.wz@gmail.com, quan.xu0@gmail.com, liliang.opensource@gmail.com, wei.w.wang@intel.com, pbonzini@redhat.com, nilal@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" This patch adds a timer to limit the time that host waits for the free page hints reported by the guest. Users can specify the time in ms via "free-page-wait-time" command line option. If a user doesn't specify a time, host waits till the guest finishes reporting all the free page hints. The policy (wait for all the free page hints to be reported or use a time limit) is determined by the orchestration layer. Signed-off-by: Wei Wang CC: Michael S. Tsirkin --- hw/virtio/virtio-balloon.c | 84 ++++++++++++++++++++++++++++++++++= +++- hw/virtio/virtio-pci.c | 3 ++ include/hw/virtio/virtio-balloon.h | 4 ++ 3 files changed, 90 insertions(+), 1 deletion(-) diff --git a/hw/virtio/virtio-balloon.c b/hw/virtio/virtio-balloon.c index b424d4e..9ee0de4 100644 --- a/hw/virtio/virtio-balloon.c +++ b/hw/virtio/virtio-balloon.c @@ -207,6 +207,65 @@ static void balloon_stats_set_poll_interval(Object *ob= j, Visitor *v, balloon_stats_change_timer(s, 0); } =20 +static void balloon_free_page_change_timer(VirtIOBalloon *s, int64_t ms) +{ + timer_mod(s->free_page_timer, + qemu_clock_get_ms(QEMU_CLOCK_REALTIME) + ms); +} + +static void balloon_stop_free_page_report(void *opaque) +{ + VirtIOBalloon *dev =3D opaque; + VirtIODevice *vdev =3D VIRTIO_DEVICE(dev); + + timer_del(dev->free_page_timer); + timer_free(dev->free_page_timer); + dev->free_page_timer =3D NULL; + + if (dev->free_page_report_status =3D=3D FREE_PAGE_REPORT_S_START) { + dev->host_stop_free_page =3D true; + virtio_notify_config(vdev); + } +} + +static void balloon_free_page_get_wait_time(Object *obj, Visitor *v, + const char *name, void *opaque, + Error **errp) +{ + VirtIOBalloon *s =3D opaque; + + visit_type_int(v, name, &s->free_page_wait_time, errp); +} + +static void balloon_free_page_set_wait_time(Object *obj, Visitor *v, + const char *name, void *opaque, + Error **errp) +{ + VirtIOBalloon *s =3D opaque; + Error *local_err =3D NULL; + int64_t value; + + visit_type_int(v, name, &value, &local_err); + if (local_err) { + error_propagate(errp, local_err); + return; + } + if (value < 0) { + error_setg(errp, "free page wait time must be greater than zero"); + return; + } + + if (value > UINT32_MAX) { + error_setg(errp, "free page wait time value is too big"); + return; + } + + s->free_page_wait_time =3D value; + g_assert(s->free_page_timer =3D=3D NULL); + s->free_page_timer =3D timer_new_ms(QEMU_CLOCK_REALTIME, + balloon_stop_free_page_report, s); +} + static void virtio_balloon_handle_output(VirtIODevice *vdev, VirtQueue *vq) { VirtIOBalloon *s =3D VIRTIO_BALLOON(vdev); @@ -330,6 +389,7 @@ static void virtio_balloon_poll_free_page_hints(VirtIOB= alloon *dev) if (id =3D=3D dev->free_page_report_cmd_id) { dev->free_page_report_status =3D FREE_PAGE_REPORT_S_START; } else { + dev->host_stop_free_page =3D false; dev->free_page_report_status =3D FREE_PAGE_REPORT_S_STOP; break; } @@ -385,6 +445,10 @@ static void virtio_balloon_free_page_poll(void *opaque) virtio_notify_config(vdev); s->free_page_report_status =3D FREE_PAGE_REPORT_S_REQUESTED; =20 + if (s->free_page_wait_time) { + balloon_free_page_change_timer(s, s->free_page_wait_time); + } + virtio_balloon_poll_free_page_hints(s); } =20 @@ -395,7 +459,19 @@ static void virtio_balloon_get_config(VirtIODevice *vd= ev, uint8_t *config_data) =20 config.num_pages =3D cpu_to_le32(dev->num_pages); config.actual =3D cpu_to_le32(dev->actual); - config.free_page_report_cmd_id =3D cpu_to_le32(dev->free_page_report_c= md_id); + if (dev->host_stop_free_page) { + /* + * Host is actively requesting to stop the free page reporting, se= nd + * the stop sign to the guest. This happens when the migration thr= ead + * has reached the phase to send pages to the destination while the + * guest hasn't done the reporting. + */ + config.free_page_report_cmd_id =3D + VIRTIO_BALLOON_FREE_PAGE_REPORT_STOP_I= D; + } else { + config.free_page_report_cmd_id =3D + cpu_to_le32(dev->free_page_report_cmd_= id); + } =20 trace_virtio_balloon_get_config(config.num_pages, config.actual); memcpy(config_data, &config, sizeof(struct virtio_balloon_config)); @@ -539,6 +615,7 @@ static void virtio_balloon_device_realize(DeviceState *= dev, Error **errp) s->free_page_vq =3D virtio_add_queue(vdev, VIRTQUEUE_MAX_SIZE, NUL= L); s->free_page_report_status =3D FREE_PAGE_REPORT_S_STOP; s->free_page_report_cmd_id =3D VIRTIO_BALLOON_FREE_PAGE_REPORT_STO= P_ID; + s->host_stop_free_page =3D false; } reset_stats(s); } @@ -602,6 +679,11 @@ static void virtio_balloon_instance_init(Object *obj) balloon_stats_get_poll_interval, balloon_stats_set_poll_interval, NULL, s, NULL); + + object_property_add(obj, "free-page-wait-time", "int", + balloon_free_page_get_wait_time, + balloon_free_page_set_wait_time, + NULL, s, NULL); } =20 static const VMStateDescription vmstate_virtio_balloon =3D { diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c index 6c75cca..3345104 100644 --- a/hw/virtio/virtio-pci.c +++ b/hw/virtio/virtio-pci.c @@ -2250,6 +2250,9 @@ static void virtio_balloon_pci_instance_init(Object *= obj) object_property_add_alias(obj, "guest-stats-polling-interval", OBJECT(&dev->vdev), "guest-stats-polling-interval", &error_abort= ); + object_property_add_alias(obj, "free-page-wait-time", + OBJECT(&dev->vdev), + "free-page-wait-time", &error_abort); } =20 static const TypeInfo virtio_balloon_pci_info =3D { diff --git a/include/hw/virtio/virtio-balloon.h b/include/hw/virtio/virtio-= balloon.h index 11b4e01..c16855b 100644 --- a/include/hw/virtio/virtio-balloon.h +++ b/include/hw/virtio/virtio-balloon.h @@ -40,6 +40,8 @@ enum virtio_balloon_free_page_report_status { typedef struct VirtIOBalloon { VirtIODevice parent_obj; VirtQueue *ivq, *dvq, *svq, *free_page_vq; + /* Host is requesting the guest to stop free page reporting */ + bool host_stop_free_page; uint32_t free_page_report_status; uint32_t num_pages; uint32_t actual; @@ -49,8 +51,10 @@ typedef struct VirtIOBalloon { VirtQueueElement *stats_vq_elem; size_t stats_vq_offset; QEMUTimer *stats_timer; + QEMUTimer *free_page_timer; int64_t stats_last_update; int64_t stats_poll_interval; + int64_t free_page_wait_time; uint32_t host_features; } VirtIOBalloon; =20 --=20 1.8.3.1