From nobody Sat Oct 25 08:57:53 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1520421555352364.47093572290544; Wed, 7 Mar 2018 03:19:15 -0800 (PST) Received: from localhost ([::1]:60826 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1etX6I-0004Bu-DD for importer@patchew.org; Wed, 07 Mar 2018 06:19:14 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:59847) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1etWoV-00057R-1O for qemu-devel@nongnu.org; Wed, 07 Mar 2018 06:00:56 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1etWoT-0006NC-E7 for qemu-devel@nongnu.org; Wed, 07 Mar 2018 06:00:51 -0500 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:36528 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1etWoT-0006Mw-7o for qemu-devel@nongnu.org; Wed, 07 Mar 2018 06:00:49 -0500 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id BDFBA8D74A for ; Wed, 7 Mar 2018 11:00:48 +0000 (UTC) Received: from secure.mitica (ovpn-116-34.ams2.redhat.com [10.36.116.34]) by smtp.corp.redhat.com (Postfix) with ESMTP id A2BC1215CDA7; Wed, 7 Mar 2018 11:00:47 +0000 (UTC) From: Juan Quintela To: qemu-devel@nongnu.org Date: Wed, 7 Mar 2018 12:00:08 +0100 Message-Id: <20180307110010.2205-23-quintela@redhat.com> In-Reply-To: <20180307110010.2205-1-quintela@redhat.com> References: <20180307110010.2205-1-quintela@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.6 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.2]); Wed, 07 Mar 2018 11:00:48 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.2]); Wed, 07 Mar 2018 11:00:48 +0000 (UTC) for IP:'10.11.54.6' DOMAIN:'int-mx06.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'quintela@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH v10 22/24] migration: Create ram_multifd_page X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lvivier@redhat.com, dgilbert@redhat.com, peterx@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" The function still don't use multifd, but we have simplified ram_save_page, xbzrle and RDMA stuff is gone. We have added a new counter. Signed-off-by: Juan Quintela -- Add last_page parameter Add commets for done and address Remove multifd field, it is the same than normal pages Merge next patch, now we send multiple pages at a time Remove counter for multifd pages, it is identical to normal pages Use iovec's instead of creating the equivalent. Clear memory used by pages (dave) Use g_new0(danp) define MULTIFD_CONTINUE now pages member is a pointer Fix off-by-one in number of pages in one packet Remove RAM_SAVE_FLAG_MULTIFD_PAGE --- migration/ram.c | 144 +++++++++++++++++++++++++++++++++++++++++++++= +++- migration/trace-events | 3 +- 2 files changed, 144 insertions(+), 3 deletions(-) diff --git a/migration/ram.c b/migration/ram.c index 4efac0c20c..df9646ed2e 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -54,6 +54,7 @@ #include "migration/block.h" #include "sysemu/sysemu.h" #include "qemu/uuid.h" +#include "qemu/iov.h" =20 /***********************************************************/ /* ram save/restore */ @@ -397,7 +398,19 @@ static void compress_threads_save_setup(void) =20 /* Multiple fd's */ =20 +typedef struct { + /* number of used pages */ + uint32_t used; + /* number of allocated pages */ + uint32_t allocated; + /* global number of generated multifd packets */ + uint32_t seq; + struct iovec *iov; + RAMBlock *block; +} multifd_pages_t; + struct MultiFDSendParams { + /* not changed */ uint8_t id; char *name; QemuThread thread; @@ -405,8 +418,15 @@ struct MultiFDSendParams { QemuSemaphore sem; QemuMutex mutex; bool running; + /* protected by param mutex */ bool quit; bool sync; + multifd_pages_t *pages; + /* how many patches has sent this channel */ + uint32_t packets_sent; + /* protected by multifd mutex */ + /* has the thread finish the last submitted job */ + bool done; }; typedef struct MultiFDSendParams MultiFDSendParams; =20 @@ -416,8 +436,31 @@ struct { int count; /* syncs main thread and channels */ QemuSemaphore sem_main; + QemuMutex mutex; + QemuSemaphore sem; + multifd_pages_t *pages; } *multifd_send_state; =20 +static void multifd_pages_init(multifd_pages_t **ppages, size_t size) +{ + multifd_pages_t *pages =3D g_new0(multifd_pages_t, 1); + + pages->allocated =3D size; + pages->iov =3D g_new0(struct iovec, size); + *ppages =3D pages; +} + +static void multifd_pages_clear(multifd_pages_t *pages) +{ + pages->used =3D 0; + pages->allocated =3D 0; + pages->seq =3D 0; + pages->block =3D NULL; + g_free(pages->iov); + pages->iov =3D NULL; + g_free(pages); +} + static void terminate_multifd_send_threads(Error *errp) { int i; @@ -464,10 +507,14 @@ int multifd_save_cleanup(Error **errp) qemu_sem_destroy(&p->sem); g_free(p->name); p->name =3D NULL; + multifd_pages_clear(p->pages); + p->pages =3D NULL; } qemu_sem_destroy(&multifd_send_state->sem_main); g_free(multifd_send_state->params); multifd_send_state->params =3D NULL; + multifd_pages_clear(multifd_send_state->pages); + multifd_send_state->pages =3D NULL; g_free(multifd_send_state); multifd_send_state =3D NULL; return ret; @@ -516,6 +563,7 @@ static void *multifd_send_thread(void *opaque) terminate_multifd_send_threads(local_err); return NULL; } + qemu_sem_post(&multifd_send_state->sem); =20 while (true) { qemu_sem_wait(&p->sem); @@ -530,9 +578,23 @@ static void *multifd_send_thread(void *opaque) qemu_mutex_unlock(&p->mutex); break; } + if (p->pages->used) { + p->pages->used =3D 0; + qemu_mutex_unlock(&p->mutex); + + trace_multifd_send(p->id, p->pages->seq, p->pages->used); + /* ToDo: send page here */ + + qemu_mutex_lock(&multifd_send_state->mutex); + p->done =3D true; + p->packets_sent++; + qemu_mutex_unlock(&multifd_send_state->mutex); + qemu_sem_post(&multifd_send_state->sem); + continue; + } qemu_mutex_unlock(&p->mutex); } - trace_multifd_send_thread_end(p->id); + trace_multifd_send_thread_end(p->id, p->packets_sent); =20 return NULL; } @@ -571,7 +633,10 @@ int multifd_save_setup(void) multifd_send_state->params =3D g_new0(MultiFDSendParams, thread_count); atomic_set(&multifd_send_state->count, 0); qemu_sem_init(&multifd_send_state->sem_main, 0); - + qemu_mutex_init(&multifd_send_state->mutex); + qemu_sem_init(&multifd_send_state->sem, 0); + multifd_pages_init(&multifd_send_state->pages, + migrate_multifd_page_count()); for (i =3D 0; i < thread_count; i++) { MultiFDSendParams *p =3D &multifd_send_state->params[i]; =20 @@ -579,6 +644,8 @@ int multifd_save_setup(void) qemu_sem_init(&p->sem, 0); p->quit =3D false; p->id =3D i; + p->done =3D true; + multifd_pages_init(&p->pages, migrate_multifd_page_count()); p->name =3D g_strdup_printf("multifdsend_%d", i); socket_send_channel_create(multifd_new_send_channel_async, p); } @@ -586,6 +653,50 @@ int multifd_save_setup(void) return 0; } =20 +static void multifd_send_page(RAMBlock *block, ram_addr_t offset, + bool last_page) +{ + int i; + static int next_channel =3D 0; + MultiFDSendParams *p =3D NULL; /* make happy gcc */ + multifd_pages_t *pages =3D multifd_send_state->pages; + + if (!pages->block) { + pages->block =3D block; + } + + pages->iov[pages->used].iov_base =3D block->host + offset; + pages->iov[pages->used].iov_len =3D TARGET_PAGE_SIZE; + pages->used++; + + if (!last_page) { + if (pages->used < pages->allocated) { + return; + } + } + + qemu_sem_wait(&multifd_send_state->sem); + qemu_mutex_lock(&multifd_send_state->mutex); + for (i =3D next_channel;; i =3D (i + 1) % migrate_multifd_channels() )= { + p =3D &multifd_send_state->params[i]; + + if (p->done) { + p->done =3D false; + next_channel =3D (i + 1) % migrate_multifd_channels(); + break; + } + } + qemu_mutex_unlock(&multifd_send_state->mutex); + qemu_mutex_lock(&p->mutex); + p->pages->used =3D 0; + p->pages->seq =3D pages->seq + 1; + p->pages->block =3D NULL; + multifd_send_state->pages =3D p->pages; + p->pages =3D pages; + qemu_mutex_unlock(&p->mutex); + qemu_sem_post(&p->sem); +} + struct MultiFDRecvParams { uint8_t id; char *name; @@ -1220,6 +1331,31 @@ static int ram_save_page(RAMState *rs, PageSearchSta= tus *pss, bool last_stage) return pages; } =20 +static int ram_multifd_page(RAMState *rs, PageSearchStatus *pss, + bool last_stage) +{ + int pages; + uint8_t *p; + RAMBlock *block =3D pss->block; + ram_addr_t offset =3D pss->page << TARGET_PAGE_BITS; + + p =3D block->host + offset; + + pages =3D save_zero_page(rs, block, offset); + if (pages =3D=3D -1) { + ram_counters.transferred +=3D + save_page_header(rs, rs->f, block, + offset | RAM_SAVE_FLAG_PAGE); + multifd_send_page(block, offset, rs->migration_dirty_pages =3D=3D = 1); + qemu_put_buffer(rs->f, p, TARGET_PAGE_SIZE); + ram_counters.transferred +=3D TARGET_PAGE_SIZE; + pages =3D 1; + ram_counters.normal++; + } + + return pages; +} + static int do_compress_ram_page(QEMUFile *f, RAMBlock *block, ram_addr_t offset) { @@ -1648,6 +1784,8 @@ static int ram_save_target_page(RAMState *rs, PageSea= rchStatus *pss, if (migrate_use_compression() && (rs->ram_bulk_stage || !migrate_use_xbzrle())) { res =3D ram_save_compressed_page(rs, pss, last_stage); + } else if (migrate_use_multifd()) { + res =3D ram_multifd_page(rs, pss, last_stage); } else { res =3D ram_save_page(rs, pss, last_stage); } @@ -3047,6 +3185,7 @@ static int ram_load(QEMUFile *f, void *opaque, int ve= rsion_id) if (!migrate_use_compression()) { invalid_flags |=3D RAM_SAVE_FLAG_COMPRESS_PAGE; } + /* This RCU critical section can be very long running. * When RCU reclaims in the code start to become numerous, * it will be necessary to reduce the granularity of this @@ -3166,6 +3305,7 @@ static int ram_load(QEMUFile *f, void *opaque, int ve= rsion_id) break; } break; + case RAM_SAVE_FLAG_EOS: /* normal exit */ multifd_recv_sync_main(); diff --git a/migration/trace-events b/migration/trace-events index db88fa699f..f6ab2c7bcb 100644 --- a/migration/trace-events +++ b/migration/trace-events @@ -80,9 +80,10 @@ ram_save_queue_pages(const char *rbname, size_t start, s= ize_t len) "%s: start: 0 multifd_send_sync_main(void) "" multifd_recv_sync_main(void) "" multifd_send_thread_start(int id) "%d" -multifd_send_thread_end(int id) "%d" +multifd_send_thread_end(char id, uint32_t packets) "channel %d packets %d" multifd_recv_thread_start(int id) "%d" multifd_recv_thread_end(int id) "%d" +multifd_send(char id, int seq, int num) "channel %d sequence %d num pages = %d" =20 # migration/migration.c await_return_path_close_on_source_close(void) "" --=20 2.14.3