From nobody Sun May 5 21:21:29 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (208.118.235.17 [208.118.235.17]) by mx.zohomail.com with SMTPS id 1520875428293804.3353884834112; Mon, 12 Mar 2018 10:23:48 -0700 (PDT) Received: from localhost ([::1]:33375 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1evRAj-0000zS-6E for importer@patchew.org; Mon, 12 Mar 2018 13:23:41 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52889) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1evR8l-00080F-G1 for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:21:40 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1evR8i-0007RU-Cn for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:21:39 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:47213 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1evR8i-0007Qa-70 for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:21:36 -0400 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 391E7406E97D for ; Mon, 12 Mar 2018 17:21:32 +0000 (UTC) Received: from dgilbert-t530.redhat.com (ovpn-117-169.ams2.redhat.com [10.36.117.169]) by smtp.corp.redhat.com (Postfix) with ESMTP id 00BF1202322C; Mon, 12 Mar 2018 17:21:30 +0000 (UTC) From: "Dr. David Alan Gilbert (git)" To: qemu-devel@nongnu.org, mst@redhat.com, maxime.coquelin@redhat.com, marcandre.lureau@redhat.com, peterx@redhat.com, quintela@redhat.com Date: Mon, 12 Mar 2018 17:20:56 +0000 Message-Id: <20180312172124.56461-2-dgilbert@redhat.com> In-Reply-To: <20180312172124.56461-1-dgilbert@redhat.com> References: <20180312172124.56461-1-dgilbert@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.4 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.7]); Mon, 12 Mar 2018 17:21:32 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.7]); Mon, 12 Mar 2018 17:21:32 +0000 (UTC) for IP:'10.11.54.4' DOMAIN:'int-mx04.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'dgilbert@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH v5 01/29] migrate: Update ram_block_discard_range for shared X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: aarcange@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: "Dr. David Alan Gilbert" The choice of call to discard a block is getting more complicated for other cases. We use fallocate PUNCH_HOLE in any file cases; it works for both hugepage and for tmpfs. We use the DONTNEED for non-hugepage cases either where they're anonymous or where they're private. Care should be taken when trying other backing files. Signed-off-by: Dr. David Alan Gilbert Reviewed-by: Peter Xu --- exec.c | 60 ++++++++++++++++++++++++++++++++++++++++++++++----------= ---- trace-events | 3 ++- 2 files changed, 48 insertions(+), 15 deletions(-) diff --git a/exec.c b/exec.c index a9181e6417..34fdfd966e 100644 --- a/exec.c +++ b/exec.c @@ -3721,6 +3721,7 @@ int ram_block_discard_range(RAMBlock *rb, uint64_t st= art, size_t length) } =20 if ((start + length) <=3D rb->used_length) { + bool need_madvise, need_fallocate; uint8_t *host_endaddr =3D host_startaddr + length; if ((uintptr_t)host_endaddr & (rb->page_size - 1)) { error_report("ram_block_discard_range: Unaligned end address: = %p", @@ -3730,29 +3731,60 @@ int ram_block_discard_range(RAMBlock *rb, uint64_t = start, size_t length) =20 errno =3D ENOTSUP; /* If we are missing MADVISE etc */ =20 - if (rb->page_size =3D=3D qemu_host_page_size) { -#if defined(CONFIG_MADVISE) - /* Note: We need the madvise MADV_DONTNEED behaviour of defini= tely - * freeing the page. - */ - ret =3D madvise(host_startaddr, length, MADV_DONTNEED); -#endif - } else { - /* Huge page case - unfortunately it can't do DONTNEED, but - * it can do the equivalent by FALLOC_FL_PUNCH_HOLE in the - * huge page file. + /* The logic here is messy; + * madvise DONTNEED fails for hugepages + * fallocate works on hugepages and shmem + */ + need_madvise =3D (rb->page_size =3D=3D qemu_host_page_size); + need_fallocate =3D rb->fd !=3D -1; + if (need_fallocate) { + /* For a file, this causes the area of the file to be zero'd + * if read, and for hugetlbfs also causes it to be unmapped + * so a userfault will trigger. */ #ifdef CONFIG_FALLOCATE_PUNCH_HOLE ret =3D fallocate(rb->fd, FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEE= P_SIZE, start, length); + if (ret) { + ret =3D -errno; + error_report("ram_block_discard_range: Failed to fallocate= " + "%s:%" PRIx64 " +%zx (%d)", + rb->idstr, start, length, ret); + goto err; + } +#else + ret =3D -ENOSYS; + error_report("ram_block_discard_range: fallocate not available= /file" + "%s:%" PRIx64 " +%zx (%d)", + rb->idstr, start, length, ret); + goto err; #endif } - if (ret) { - ret =3D -errno; - error_report("ram_block_discard_range: Failed to discard range= " + if (need_madvise) { + /* For normal RAM this causes it to be unmapped, + * for shared memory it causes the local mapping to disappear + * and to fall back on the file contents (which we just + * fallocate'd away). + */ +#if defined(CONFIG_MADVISE) + ret =3D madvise(host_startaddr, length, MADV_DONTNEED); + if (ret) { + ret =3D -errno; + error_report("ram_block_discard_range: Failed to discard r= ange " + "%s:%" PRIx64 " +%zx (%d)", + rb->idstr, start, length, ret); + goto err; + } +#else + ret =3D -ENOSYS; + error_report("ram_block_discard_range: MADVISE not available" "%s:%" PRIx64 " +%zx (%d)", rb->idstr, start, length, ret); + goto err; +#endif } + trace_ram_block_discard_range(rb->idstr, host_startaddr, length, + need_madvise, need_fallocate, ret); } else { error_report("ram_block_discard_range: Overrun block '%s' (%" PRIu= 64 "/%zx/" RAM_ADDR_FMT")", diff --git a/trace-events b/trace-events index 89fcad0fd1..196598a6f8 100644 --- a/trace-events +++ b/trace-events @@ -55,9 +55,10 @@ dma_complete(void *dbs, int ret, void *cb) "dbs=3D%p ret= =3D%d cb=3D%p" dma_blk_cb(void *dbs, int ret) "dbs=3D%p ret=3D%d" dma_map_wait(void *dbs) "dbs=3D%p" =20 -# # exec.c +# exec.c find_ram_offset(uint64_t size, uint64_t offset) "size: 0x%" PRIx64 " @ 0x%= " PRIx64 find_ram_offset_loop(uint64_t size, uint64_t candidate, uint64_t offset, u= int64_t next, uint64_t mingap) "trying size: 0x%" PRIx64 " @ 0x%" PRIx64 ",= offset: 0x%" PRIx64" next: 0x%" PRIx64 " mingap: 0x%" PRIx64 +ram_block_discard_range(const char *rbname, void *hva, size_t length, bool= need_madvise, bool need_fallocate, int ret) "%s@%p + 0x%zx: madvise: %d fa= llocate: %d ret: %d" =20 # memory.c memory_region_ops_read(int cpu_index, void *mr, uint64_t addr, uint64_t va= lue, unsigned size) "cpu %d mr %p addr 0x%"PRIx64" value 0x%"PRIx64" size %= u" --=20 2.14.3 From nobody Sun May 5 21:21:29 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (208.118.235.17 [208.118.235.17]) by mx.zohomail.com with SMTPS id 1520875428229644.3777094498357; Mon, 12 Mar 2018 10:23:48 -0700 (PDT) Received: from localhost ([::1]:33376 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1evRAl-00010l-Dk for importer@patchew.org; Mon, 12 Mar 2018 13:23:43 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52891) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1evR8l-00080I-Gb for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:21:40 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1evR8i-0007RB-BJ for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:21:39 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:56678 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1evR8i-0007Qb-6e for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:21:36 -0400 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id AF51F401C9AE for ; Mon, 12 Mar 2018 17:21:33 +0000 (UTC) Received: from dgilbert-t530.redhat.com (ovpn-117-169.ams2.redhat.com [10.36.117.169]) by smtp.corp.redhat.com (Postfix) with ESMTP id 76F9F202322C; Mon, 12 Mar 2018 17:21:32 +0000 (UTC) From: "Dr. David Alan Gilbert (git)" To: qemu-devel@nongnu.org, mst@redhat.com, maxime.coquelin@redhat.com, marcandre.lureau@redhat.com, peterx@redhat.com, quintela@redhat.com Date: Mon, 12 Mar 2018 17:20:57 +0000 Message-Id: <20180312172124.56461-3-dgilbert@redhat.com> In-Reply-To: <20180312172124.56461-1-dgilbert@redhat.com> References: <20180312172124.56461-1-dgilbert@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.4 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.5]); Mon, 12 Mar 2018 17:21:33 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.5]); Mon, 12 Mar 2018 17:21:33 +0000 (UTC) for IP:'10.11.54.4' DOMAIN:'int-mx04.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'dgilbert@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH v5 02/29] qemu_ram_block_host_offset X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: aarcange@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: "Dr. David Alan Gilbert" Utility to give the offset of a host pointer within a RAMBlock (assuming we already know it's in that RAMBlock) Signed-off-by: Dr. David Alan Gilbert Reviewed-by: Peter Xu --- exec.c | 10 ++++++++++ include/exec/cpu-common.h | 1 + 2 files changed, 11 insertions(+) diff --git a/exec.c b/exec.c index 34fdfd966e..2199b093bd 100644 --- a/exec.c +++ b/exec.c @@ -2297,6 +2297,16 @@ static void *qemu_ram_ptr_length(RAMBlock *ram_block= , ram_addr_t addr, return ramblock_ptr(block, addr); } =20 +/* Return the offset of a hostpointer within a ramblock */ +ram_addr_t qemu_ram_block_host_offset(RAMBlock *rb, void *host) +{ + ram_addr_t res =3D (uint8_t *)host - (uint8_t *)rb->host; + assert((uintptr_t)host >=3D (uintptr_t)rb->host); + assert(res < rb->max_length); + + return res; +} + /* * Translates a host ptr back to a RAMBlock, a ram_addr and an offset * in that RAMBlock. diff --git a/include/exec/cpu-common.h b/include/exec/cpu-common.h index 74341b19d2..0d861a6289 100644 --- a/include/exec/cpu-common.h +++ b/include/exec/cpu-common.h @@ -68,6 +68,7 @@ ram_addr_t qemu_ram_addr_from_host(void *ptr); RAMBlock *qemu_ram_block_by_name(const char *name); RAMBlock *qemu_ram_block_from_host(void *ptr, bool round_offset, ram_addr_t *offset); +ram_addr_t qemu_ram_block_host_offset(RAMBlock *rb, void *host); void qemu_ram_set_idstr(RAMBlock *block, const char *name, DeviceState *de= v); void qemu_ram_unset_idstr(RAMBlock *block); const char *qemu_ram_get_idstr(RAMBlock *rb); --=20 2.14.3 From nobody Sun May 5 21:21:29 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1520875433993265.2182590611226; Mon, 12 Mar 2018 10:23:53 -0700 (PDT) Received: from localhost ([::1]:33377 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1evRAn-00011x-1t for importer@patchew.org; Mon, 12 Mar 2018 13:23:45 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52892) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1evR8l-00080J-Gd for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:21:40 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1evR8i-0007RK-Bt for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:21:39 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:38212 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1evR8i-0007Qc-6T for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:21:36 -0400 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 2E99376FB9 for ; Mon, 12 Mar 2018 17:21:35 +0000 (UTC) Received: from dgilbert-t530.redhat.com (ovpn-117-169.ams2.redhat.com [10.36.117.169]) by smtp.corp.redhat.com (Postfix) with ESMTP id EA14C2026E03; Mon, 12 Mar 2018 17:21:33 +0000 (UTC) From: "Dr. David Alan Gilbert (git)" To: qemu-devel@nongnu.org, mst@redhat.com, maxime.coquelin@redhat.com, marcandre.lureau@redhat.com, peterx@redhat.com, quintela@redhat.com Date: Mon, 12 Mar 2018 17:20:58 +0000 Message-Id: <20180312172124.56461-4-dgilbert@redhat.com> In-Reply-To: <20180312172124.56461-1-dgilbert@redhat.com> References: <20180312172124.56461-1-dgilbert@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.4 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.1]); Mon, 12 Mar 2018 17:21:35 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.1]); Mon, 12 Mar 2018 17:21:35 +0000 (UTC) for IP:'10.11.54.4' DOMAIN:'int-mx04.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'dgilbert@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH v5 03/29] postcopy: use UFFDIO_ZEROPAGE only when available X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: aarcange@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: "Dr. David Alan Gilbert" Use a flag on the RAMBlock to state whether it has the UFFDIO_ZEROPAGE capability, use it when it's available. This allows the use of postcopy on tmpfs as well as hugepage backed files. Signed-off-by: Dr. David Alan Gilbert Reviewed-by: Peter Xu --- exec.c | 16 ++++++++++++++++ include/exec/cpu-common.h | 3 +++ migration/postcopy-ram.c | 13 ++++++++++--- 3 files changed, 29 insertions(+), 3 deletions(-) diff --git a/exec.c b/exec.c index 2199b093bd..0eb890dca8 100644 --- a/exec.c +++ b/exec.c @@ -99,6 +99,11 @@ static MemoryRegion io_mem_unassigned; */ #define RAM_RESIZEABLE (1 << 2) =20 +/* UFFDIO_ZEROPAGE is available on this RAMBlock to atomically + * zero the page and wake waiting processes. + * (Set during postcopy) + */ +#define RAM_UF_ZEROPAGE (1 << 3) #endif =20 #ifdef TARGET_PAGE_BITS_VARY @@ -1767,6 +1772,17 @@ bool qemu_ram_is_shared(RAMBlock *rb) return rb->flags & RAM_SHARED; } =20 +/* Note: Only set at the start of postcopy */ +bool qemu_ram_is_uf_zeroable(RAMBlock *rb) +{ + return rb->flags & RAM_UF_ZEROPAGE; +} + +void qemu_ram_set_uf_zeroable(RAMBlock *rb) +{ + rb->flags |=3D RAM_UF_ZEROPAGE; +} + /* Called with iothread lock held. */ void qemu_ram_set_idstr(RAMBlock *new_block, const char *name, DeviceState= *dev) { diff --git a/include/exec/cpu-common.h b/include/exec/cpu-common.h index 0d861a6289..24d335f95d 100644 --- a/include/exec/cpu-common.h +++ b/include/exec/cpu-common.h @@ -73,6 +73,9 @@ void qemu_ram_set_idstr(RAMBlock *block, const char *name= , DeviceState *dev); void qemu_ram_unset_idstr(RAMBlock *block); const char *qemu_ram_get_idstr(RAMBlock *rb); bool qemu_ram_is_shared(RAMBlock *rb); +bool qemu_ram_is_uf_zeroable(RAMBlock *rb); +void qemu_ram_set_uf_zeroable(RAMBlock *rb); + size_t qemu_ram_pagesize(RAMBlock *block); size_t qemu_ram_pagesize_largest(void); =20 diff --git a/migration/postcopy-ram.c b/migration/postcopy-ram.c index 032abfbf1a..a75b5d393f 100644 --- a/migration/postcopy-ram.c +++ b/migration/postcopy-ram.c @@ -481,6 +481,10 @@ static int ram_block_enable_notify(const char *block_n= ame, void *host_addr, error_report("%s userfault: Region doesn't support COPY", __func__= ); return -1; } + if (reg_struct.ioctls & ((__u64)1 << _UFFDIO_ZEROPAGE)) { + RAMBlock *rb =3D qemu_ram_block_by_name(block_name); + qemu_ram_set_uf_zeroable(rb); + } =20 return 0; } @@ -700,11 +704,14 @@ int postcopy_place_page(MigrationIncomingState *mis, = void *host, void *from, int postcopy_place_page_zero(MigrationIncomingState *mis, void *host, RAMBlock *rb) { + size_t pagesize =3D qemu_ram_pagesize(rb); trace_postcopy_place_page_zero(host); =20 - if (qemu_ram_pagesize(rb) =3D=3D getpagesize()) { - if (qemu_ufd_copy_ioctl(mis->userfault_fd, host, NULL, getpagesize= (), - rb)) { + /* Normal RAMBlocks can zero a page using UFFDIO_ZEROPAGE + * but it's not available for everything (e.g. hugetlbpages) + */ + if (qemu_ram_is_uf_zeroable(rb)) { + if (qemu_ufd_copy_ioctl(mis->userfault_fd, host, NULL, pagesize, r= b)) { int e =3D errno; error_report("%s: %s zero host: %p", __func__, strerror(e), host); --=20 2.14.3 From nobody Sun May 5 21:21:29 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1520875751333254.81806928614253; Mon, 12 Mar 2018 10:29:11 -0700 (PDT) Received: from localhost ([::1]:33407 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1evRG0-00065z-7P for importer@patchew.org; Mon, 12 Mar 2018 13:29:08 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52890) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1evR8l-00080G-G9 for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:21:43 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1evR8j-0007S5-7R for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:21:39 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:56680 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1evR8j-0007Rh-1X for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:21:37 -0400 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id A1F13401C9AE for ; Mon, 12 Mar 2018 17:21:36 +0000 (UTC) Received: from dgilbert-t530.redhat.com (ovpn-117-169.ams2.redhat.com [10.36.117.169]) by smtp.corp.redhat.com (Postfix) with ESMTP id 69A72202322B; Mon, 12 Mar 2018 17:21:35 +0000 (UTC) From: "Dr. David Alan Gilbert (git)" To: qemu-devel@nongnu.org, mst@redhat.com, maxime.coquelin@redhat.com, marcandre.lureau@redhat.com, peterx@redhat.com, quintela@redhat.com Date: Mon, 12 Mar 2018 17:20:59 +0000 Message-Id: <20180312172124.56461-5-dgilbert@redhat.com> In-Reply-To: <20180312172124.56461-1-dgilbert@redhat.com> References: <20180312172124.56461-1-dgilbert@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.4 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.5]); Mon, 12 Mar 2018 17:21:36 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.5]); Mon, 12 Mar 2018 17:21:36 +0000 (UTC) for IP:'10.11.54.4' DOMAIN:'int-mx04.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'dgilbert@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH v5 04/29] postcopy: Add notifier chain X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: aarcange@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: "Dr. David Alan Gilbert" Add a notifier chain for postcopy with a 'reason' flag and an opportunity for a notifier member to return an error. Call it when enabling postcopy. This will initially used to enable devices to declare they're unable to postcopy and later to notify of devices of stages within postcopy. Signed-off-by: Dr. David Alan Gilbert Reviewed-by: Peter Xu --- migration/postcopy-ram.c | 36 ++++++++++++++++++++++++++++++++++++ migration/postcopy-ram.h | 26 ++++++++++++++++++++++++++ vl.c | 2 ++ 3 files changed, 64 insertions(+) diff --git a/migration/postcopy-ram.c b/migration/postcopy-ram.c index a75b5d393f..1089814d54 100644 --- a/migration/postcopy-ram.c +++ b/migration/postcopy-ram.c @@ -23,6 +23,8 @@ #include "savevm.h" #include "postcopy-ram.h" #include "ram.h" +#include "qapi/error.h" +#include "qemu/notify.h" #include "sysemu/sysemu.h" #include "sysemu/balloon.h" #include "qemu/error-report.h" @@ -45,6 +47,33 @@ struct PostcopyDiscardState { unsigned int nsentcmds; }; =20 +static NotifierWithReturnList postcopy_notifier_list; + +void postcopy_infrastructure_init(void) +{ + notifier_with_return_list_init(&postcopy_notifier_list); +} + +void postcopy_add_notifier(NotifierWithReturn *nn) +{ + notifier_with_return_list_add(&postcopy_notifier_list, nn); +} + +void postcopy_remove_notifier(NotifierWithReturn *n) +{ + notifier_with_return_remove(n); +} + +int postcopy_notify(enum PostcopyNotifyReason reason, Error **errp) +{ + struct PostcopyNotifyData pnd; + pnd.reason =3D reason; + pnd.errp =3D errp; + + return notifier_with_return_list_notify(&postcopy_notifier_list, + &pnd); +} + /* Postcopy needs to detect accesses to pages that haven't yet been copied * across, and efficiently map new pages in, the techniques for doing this * are target OS specific. @@ -215,6 +244,7 @@ bool postcopy_ram_supported_by_host(MigrationIncomingSt= ate *mis) struct uffdio_register reg_struct; struct uffdio_range range_struct; uint64_t feature_mask; + Error *local_err =3D NULL; =20 if (qemu_target_page_size() > pagesize) { error_report("Target page size bigger than host page size"); @@ -228,6 +258,12 @@ bool postcopy_ram_supported_by_host(MigrationIncomingS= tate *mis) goto out; } =20 + /* Give devices a chance to object */ + if (postcopy_notify(POSTCOPY_NOTIFY_PROBE, &local_err)) { + error_report_err(local_err); + goto out; + } + /* Version and features check */ if (!ufd_check_and_apply(ufd, mis)) { goto out; diff --git a/migration/postcopy-ram.h b/migration/postcopy-ram.h index 14f6cadcbd..2e879bbacb 100644 --- a/migration/postcopy-ram.h +++ b/migration/postcopy-ram.h @@ -116,4 +116,30 @@ PostcopyState postcopy_state_set(PostcopyState new_sta= te); =20 void postcopy_fault_thread_notify(MigrationIncomingState *mis); =20 +/* + * To be called once at the start before any device initialisation + */ +void postcopy_infrastructure_init(void); + +/* Add a notifier to a list to be called when checking whether the devices + * can support postcopy. + * It's data is a *PostcopyNotifyData + * It should return 0 if OK, or a negative value on failure. + * On failure it must set the data->errp to an error. + * + */ +enum PostcopyNotifyReason { + POSTCOPY_NOTIFY_PROBE =3D 0, +}; + +struct PostcopyNotifyData { + enum PostcopyNotifyReason reason; + Error **errp; +}; + +void postcopy_add_notifier(NotifierWithReturn *nn); +void postcopy_remove_notifier(NotifierWithReturn *n); +/* Call the notifier list set by postcopy_add_start_notifier */ +int postcopy_notify(enum PostcopyNotifyReason reason, Error **errp); + #endif diff --git a/vl.c b/vl.c index 3ef04ce991..0b1581107f 100644 --- a/vl.c +++ b/vl.c @@ -94,6 +94,7 @@ int main(int argc, char **argv) #include "audio/audio.h" #include "sysemu/cpus.h" #include "migration/colo.h" +#include "migration/postcopy-ram.h" #include "sysemu/kvm.h" #include "sysemu/hax.h" #include "qapi/qobject-input-visitor.h" @@ -3101,6 +3102,7 @@ int main(int argc, char **argv, char **envp) module_call_init(MODULE_INIT_OPTS); =20 runstate_init(); + postcopy_infrastructure_init(); =20 if (qcrypto_init(&err) < 0) { error_reportf_err(err, "cannot initialize crypto: "); --=20 2.14.3 From nobody Sun May 5 21:21:29 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (208.118.235.17 [208.118.235.17]) by mx.zohomail.com with SMTPS id 1520875755425489.6396434223859; Mon, 12 Mar 2018 10:29:15 -0700 (PDT) Received: from localhost ([::1]:33405 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1evRFx-00061e-5g for importer@patchew.org; Mon, 12 Mar 2018 13:29:05 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52901) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1evR8l-00080R-Ne for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:21:41 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1evR8k-0007Sw-LH for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:21:39 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:56682 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1evR8k-0007Sh-G2 for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:21:38 -0400 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 20E9D401C9AE for ; Mon, 12 Mar 2018 17:21:38 +0000 (UTC) Received: from dgilbert-t530.redhat.com (ovpn-117-169.ams2.redhat.com [10.36.117.169]) by smtp.corp.redhat.com (Postfix) with ESMTP id DCD6C202322B; Mon, 12 Mar 2018 17:21:36 +0000 (UTC) From: "Dr. David Alan Gilbert (git)" To: qemu-devel@nongnu.org, mst@redhat.com, maxime.coquelin@redhat.com, marcandre.lureau@redhat.com, peterx@redhat.com, quintela@redhat.com Date: Mon, 12 Mar 2018 17:21:00 +0000 Message-Id: <20180312172124.56461-6-dgilbert@redhat.com> In-Reply-To: <20180312172124.56461-1-dgilbert@redhat.com> References: <20180312172124.56461-1-dgilbert@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.4 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.5]); Mon, 12 Mar 2018 17:21:38 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.5]); Mon, 12 Mar 2018 17:21:38 +0000 (UTC) for IP:'10.11.54.4' DOMAIN:'int-mx04.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'dgilbert@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH v5 05/29] postcopy: Add vhost-user flag for postcopy and check it X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: aarcange@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: "Dr. David Alan Gilbert" Add a vhost feature flag for postcopy support, and use the postcopy notifier to check it before allowing postcopy. Signed-off-by: Dr. David Alan Gilbert Reviewed-by: Peter Xu --- contrib/libvhost-user/libvhost-user.h | 2 ++ docs/interop/vhost-user.txt | 10 +++++++++ hw/virtio/vhost-user.c | 41 +++++++++++++++++++++++++++++++= +++- 3 files changed, 52 insertions(+), 1 deletion(-) diff --git a/contrib/libvhost-user/libvhost-user.h b/contrib/libvhost-user/= libvhost-user.h index 18f95f65d7..96db29c4ce 100644 --- a/contrib/libvhost-user/libvhost-user.h +++ b/contrib/libvhost-user/libvhost-user.h @@ -48,6 +48,8 @@ enum VhostUserProtocolFeature { VHOST_USER_PROTOCOL_F_NET_MTU =3D 4, VHOST_USER_PROTOCOL_F_SLAVE_REQ =3D 5, VHOST_USER_PROTOCOL_F_CROSS_ENDIAN =3D 6, + VHOST_USER_PROTOCOL_F_CRYPTO_SESSION =3D 7, + VHOST_USER_PROTOCOL_F_PAGEFAULT =3D 8, =20 VHOST_USER_PROTOCOL_F_MAX }; diff --git a/docs/interop/vhost-user.txt b/docs/interop/vhost-user.txt index cb3a7595aa..91a572d781 100644 --- a/docs/interop/vhost-user.txt +++ b/docs/interop/vhost-user.txt @@ -290,6 +290,15 @@ Once the source has finished migration, rings will be = stopped by the source. No further update must be done before rings are restarted. =20 +In postcopy migration the slave is started before all the memory has been +received from the source host, and care must be taken to avoid accessing p= ages +that have yet to be received. The slave opens a 'userfault'-fd and regist= ers +the memory with it; this fd is then passed back over to the master. +The master services requests on the userfaultfd for pages that are accessed +and when the page is available it performs WAKE ioctl's on the userfaultfd +to wake the stalled slave. The client indicates support for this via the +VHOST_USER_PROTOCOL_F_PAGEFAULT feature. + Memory access ------------- =20 @@ -369,6 +378,7 @@ Protocol features #define VHOST_USER_PROTOCOL_F_SLAVE_REQ 5 #define VHOST_USER_PROTOCOL_F_CROSS_ENDIAN 6 #define VHOST_USER_PROTOCOL_F_CRYPTO_SESSION 7 +#define VHOST_USER_PROTOCOL_F_PAGEFAULT 8 =20 Master message types -------------------- diff --git a/hw/virtio/vhost-user.c b/hw/virtio/vhost-user.c index 41ff5cff41..aab35c4845 100644 --- a/hw/virtio/vhost-user.c +++ b/hw/virtio/vhost-user.c @@ -18,6 +18,8 @@ #include "qemu/error-report.h" #include "qemu/sockets.h" #include "sysemu/cryptodev.h" +#include "migration/migration.h" +#include "migration/postcopy-ram.h" =20 #include #include @@ -41,7 +43,7 @@ enum VhostUserProtocolFeature { VHOST_USER_PROTOCOL_F_SLAVE_REQ =3D 5, VHOST_USER_PROTOCOL_F_CROSS_ENDIAN =3D 6, VHOST_USER_PROTOCOL_F_CRYPTO_SESSION =3D 7, - + VHOST_USER_PROTOCOL_F_PAGEFAULT =3D 8, VHOST_USER_PROTOCOL_F_MAX }; =20 @@ -164,8 +166,10 @@ static VhostUserMsg m __attribute__ ((unused)); #define VHOST_USER_VERSION (0x1) =20 struct vhost_user { + struct vhost_dev *dev; CharBackend *chr; int slave_fd; + NotifierWithReturn postcopy_notifier; }; =20 static bool ioeventfd_enabled(void) @@ -791,6 +795,33 @@ out: return ret; } =20 +static int vhost_user_postcopy_notifier(NotifierWithReturn *notifier, + void *opaque) +{ + struct PostcopyNotifyData *pnd =3D opaque; + struct vhost_user *u =3D container_of(notifier, struct vhost_user, + postcopy_notifier); + struct vhost_dev *dev =3D u->dev; + + switch (pnd->reason) { + case POSTCOPY_NOTIFY_PROBE: + if (!virtio_has_feature(dev->protocol_features, + VHOST_USER_PROTOCOL_F_PAGEFAULT)) { + /* TODO: Get the device name into this error somehow */ + error_setg(pnd->errp, + "vhost-user backend not capable of postcopy"); + return -ENOENT; + } + break; + + default: + /* We ignore notifications we don't know */ + break; + } + + return 0; +} + static int vhost_user_init(struct vhost_dev *dev, void *opaque) { uint64_t features, protocol_features; @@ -802,6 +833,7 @@ static int vhost_user_init(struct vhost_dev *dev, void = *opaque) u =3D g_new0(struct vhost_user, 1); u->chr =3D opaque; u->slave_fd =3D -1; + u->dev =3D dev; dev->opaque =3D u; =20 err =3D vhost_user_get_features(dev, &features); @@ -858,6 +890,9 @@ static int vhost_user_init(struct vhost_dev *dev, void = *opaque) return err; } =20 + u->postcopy_notifier.notify =3D vhost_user_postcopy_notifier; + postcopy_add_notifier(&u->postcopy_notifier); + return 0; } =20 @@ -868,6 +903,10 @@ static int vhost_user_cleanup(struct vhost_dev *dev) assert(dev->vhost_ops->backend_type =3D=3D VHOST_BACKEND_TYPE_USER); =20 u =3D dev->opaque; + if (u->postcopy_notifier.notify) { + postcopy_remove_notifier(&u->postcopy_notifier); + u->postcopy_notifier.notify =3D NULL; + } if (u->slave_fd >=3D 0) { qemu_set_fd_handler(u->slave_fd, NULL, NULL, NULL); close(u->slave_fd); --=20 2.14.3 From nobody Sun May 5 21:21:29 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (208.118.235.17 [208.118.235.17]) by mx.zohomail.com with SMTPS id 1520875429207299.8172779708491; Mon, 12 Mar 2018 10:23:49 -0700 (PDT) Received: from localhost ([::1]:33378 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1evRAq-00015H-AX for importer@patchew.org; Mon, 12 Mar 2018 13:23:48 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52947) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1evR8n-00081g-HN for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:21:42 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1evR8m-0007UQ-6I for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:21:41 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:36126 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1evR8l-0007Tz-Ud for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:21:40 -0400 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 94EC68424C for ; Mon, 12 Mar 2018 17:21:39 +0000 (UTC) Received: from dgilbert-t530.redhat.com (ovpn-117-169.ams2.redhat.com [10.36.117.169]) by smtp.corp.redhat.com (Postfix) with ESMTP id 5C0AF202322B; Mon, 12 Mar 2018 17:21:38 +0000 (UTC) From: "Dr. David Alan Gilbert (git)" To: qemu-devel@nongnu.org, mst@redhat.com, maxime.coquelin@redhat.com, marcandre.lureau@redhat.com, peterx@redhat.com, quintela@redhat.com Date: Mon, 12 Mar 2018 17:21:01 +0000 Message-Id: <20180312172124.56461-7-dgilbert@redhat.com> In-Reply-To: <20180312172124.56461-1-dgilbert@redhat.com> References: <20180312172124.56461-1-dgilbert@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.78 on 10.11.54.4 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.2]); Mon, 12 Mar 2018 17:21:39 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.2]); Mon, 12 Mar 2018 17:21:39 +0000 (UTC) for IP:'10.11.54.4' DOMAIN:'int-mx04.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'dgilbert@redhat.com' RCPT:'' Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH v5 06/29] vhost-user: Add 'VHOST_USER_POSTCOPY_ADVISE' message X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: aarcange@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Type: text/plain; charset="utf-8" From: "Dr. David Alan Gilbert" Wire up a notifier to send a VHOST_USER_POSTCOPY_ADVISE message on an incoming advise. Later patches will fill in the behaviour/contents of the message. Signed-off-by: Dr. David Alan Gilbert Reviewed-by: Marc-Andr=C3=A9 Lureau --- contrib/libvhost-user/libvhost-user.c | 11 ++++++++ contrib/libvhost-user/libvhost-user.h | 3 +++ docs/interop/vhost-user.txt | 10 ++++++++ hw/virtio/vhost-user.c | 48 +++++++++++++++++++++++++++++++= ++++ migration/postcopy-ram.h | 1 + migration/savevm.c | 6 +++++ 6 files changed, 79 insertions(+) diff --git a/contrib/libvhost-user/libvhost-user.c b/contrib/libvhost-user/= libvhost-user.c index 2e358b5bce..37d4228193 100644 --- a/contrib/libvhost-user/libvhost-user.c +++ b/contrib/libvhost-user/libvhost-user.c @@ -86,6 +86,7 @@ vu_request_to_string(unsigned int req) REQ(VHOST_USER_SET_VRING_ENDIAN), REQ(VHOST_USER_GET_CONFIG), REQ(VHOST_USER_SET_CONFIG), + REQ(VHOST_USER_POSTCOPY_ADVISE), REQ(VHOST_USER_MAX), }; #undef REQ @@ -856,6 +857,14 @@ vu_set_config(VuDev *dev, VhostUserMsg *vmsg) return false; } =20 +static bool +vu_set_postcopy_advise(VuDev *dev, VhostUserMsg *vmsg) +{ + /* TODO: Open ufd, pass it back in the request */ + vmsg->size =3D 0; + return true; /* =3D send a reply */ +} + static bool vu_process_message(VuDev *dev, VhostUserMsg *vmsg) { @@ -927,6 +936,8 @@ vu_process_message(VuDev *dev, VhostUserMsg *vmsg) return vu_set_config(dev, vmsg); case VHOST_USER_NONE: break; + case VHOST_USER_POSTCOPY_ADVISE: + return vu_set_postcopy_advise(dev, vmsg); default: vmsg_close_fds(vmsg); vu_panic(dev, "Unhandled request: %d", vmsg->request); diff --git a/contrib/libvhost-user/libvhost-user.h b/contrib/libvhost-user/= libvhost-user.h index 96db29c4ce..00d78a8810 100644 --- a/contrib/libvhost-user/libvhost-user.h +++ b/contrib/libvhost-user/libvhost-user.h @@ -83,6 +83,9 @@ typedef enum VhostUserRequest { VHOST_USER_SET_VRING_ENDIAN =3D 23, VHOST_USER_GET_CONFIG =3D 24, VHOST_USER_SET_CONFIG =3D 25, + VHOST_USER_CREATE_CRYPTO_SESSION =3D 26, + VHOST_USER_CLOSE_CRYPTO_SESSION =3D 27, + VHOST_USER_POSTCOPY_ADVISE =3D 28, VHOST_USER_MAX } VhostUserRequest; =20 diff --git a/docs/interop/vhost-user.txt b/docs/interop/vhost-user.txt index 91a572d781..7854e50008 100644 --- a/docs/interop/vhost-user.txt +++ b/docs/interop/vhost-user.txt @@ -699,6 +699,16 @@ Master message types feature has been successfully negotiated. It's a required feature for crypto devices. =20 + * VHOST_USER_POSTCOPY_ADVISE + Id: 28 + Master payload: N/A + Slave payload: userfault fd + + When VHOST_USER_PROTOCOL_F_PAGEFAULT is supported, the + master advises slave that a migration with postcopy enabled is under= way, + the slave must open a userfaultfd for later use. + Note that at this stage the migration is still in precopy mode. + Slave message types ------------------- =20 diff --git a/hw/virtio/vhost-user.c b/hw/virtio/vhost-user.c index aab35c4845..ceb17b0554 100644 --- a/hw/virtio/vhost-user.c +++ b/hw/virtio/vhost-user.c @@ -78,6 +78,7 @@ typedef enum VhostUserRequest { VHOST_USER_SET_CONFIG =3D 25, VHOST_USER_CREATE_CRYPTO_SESSION =3D 26, VHOST_USER_CLOSE_CRYPTO_SESSION =3D 27, + VHOST_USER_POSTCOPY_ADVISE =3D 28, VHOST_USER_MAX } VhostUserRequest; =20 @@ -795,6 +796,50 @@ out: return ret; } =20 +/* + * Called at the start of an inbound postcopy on reception of the + * 'advise' command. + */ +static int vhost_user_postcopy_advise(struct vhost_dev *dev, Error **errp) +{ + struct vhost_user *u =3D dev->opaque; + CharBackend *chr =3D u->chr; + int ufd; + VhostUserMsg msg =3D { + .hdr.request =3D VHOST_USER_POSTCOPY_ADVISE, + .hdr.flags =3D VHOST_USER_VERSION, + }; + + if (vhost_user_write(dev, &msg, NULL, 0) < 0) { + error_setg(errp, "Failed to send postcopy_advise to vhost"); + return -1; + } + + if (vhost_user_read(dev, &msg) < 0) { + error_setg(errp, "Failed to get postcopy_advise reply from vhost"); + return -1; + } + + if (msg.hdr.request !=3D VHOST_USER_POSTCOPY_ADVISE) { + error_setg(errp, "Unexpected msg type. Expected %d received %d", + VHOST_USER_POSTCOPY_ADVISE, msg.hdr.request); + return -1; + } + + if (msg.hdr.size) { + error_setg(errp, "Received bad msg size."); + return -1; + } + ufd =3D qemu_chr_fe_get_msgfd(chr); + if (ufd < 0) { + error_setg(errp, "%s: Failed to get ufd", __func__); + return -1; + } + + /* TODO: register ufd with userfault thread */ + return 0; +} + static int vhost_user_postcopy_notifier(NotifierWithReturn *notifier, void *opaque) { @@ -814,6 +859,9 @@ static int vhost_user_postcopy_notifier(NotifierWithRet= urn *notifier, } break; =20 + case POSTCOPY_NOTIFY_INBOUND_ADVISE: + return vhost_user_postcopy_advise(dev, pnd->errp); + default: /* We ignore notifications we don't know */ break; diff --git a/migration/postcopy-ram.h b/migration/postcopy-ram.h index 2e879bbacb..0421c98d57 100644 --- a/migration/postcopy-ram.h +++ b/migration/postcopy-ram.h @@ -130,6 +130,7 @@ void postcopy_infrastructure_init(void); */ enum PostcopyNotifyReason { POSTCOPY_NOTIFY_PROBE =3D 0, + POSTCOPY_NOTIFY_INBOUND_ADVISE, }; =20 struct PostcopyNotifyData { diff --git a/migration/savevm.c b/migration/savevm.c index 358c5b51e2..1f2bf12a28 100644 --- a/migration/savevm.c +++ b/migration/savevm.c @@ -1386,6 +1386,7 @@ static int loadvm_postcopy_handle_advise(MigrationInc= omingState *mis, { PostcopyState ps =3D postcopy_state_set(POSTCOPY_INCOMING_ADVISE); uint64_t remote_pagesize_summary, local_pagesize_summary, remote_tps; + Error *local_err =3D NULL; =20 trace_loadvm_postcopy_handle_advise(); if (ps !=3D POSTCOPY_INCOMING_NONE) { @@ -1451,6 +1452,11 @@ static int loadvm_postcopy_handle_advise(MigrationIn= comingState *mis, return -1; } =20 + if (postcopy_notify(POSTCOPY_NOTIFY_INBOUND_ADVISE, &local_err)) { + error_report_err(local_err); + return -1; + } + if (ram_postcopy_incoming_init(mis)) { return -1; } --=20 2.14.3 From nobody Sun May 5 21:21:29 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1520875584746712.3153974313349; Mon, 12 Mar 2018 10:26:24 -0700 (PDT) Received: from localhost ([::1]:33393 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1evRDL-0003ey-Rs for importer@patchew.org; Mon, 12 Mar 2018 13:26:23 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52961) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1evR8o-00082f-Mg for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:21:43 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1evR8n-0007VX-K2 for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:21:42 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:56684 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1evR8n-0007VI-Dx for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:21:41 -0400 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 159D0401C9AE for ; Mon, 12 Mar 2018 17:21:41 +0000 (UTC) Received: from dgilbert-t530.redhat.com (ovpn-117-169.ams2.redhat.com [10.36.117.169]) by smtp.corp.redhat.com (Postfix) with ESMTP id D02942026E03; Mon, 12 Mar 2018 17:21:39 +0000 (UTC) From: "Dr. David Alan Gilbert (git)" To: qemu-devel@nongnu.org, mst@redhat.com, maxime.coquelin@redhat.com, marcandre.lureau@redhat.com, peterx@redhat.com, quintela@redhat.com Date: Mon, 12 Mar 2018 17:21:02 +0000 Message-Id: <20180312172124.56461-8-dgilbert@redhat.com> In-Reply-To: <20180312172124.56461-1-dgilbert@redhat.com> References: <20180312172124.56461-1-dgilbert@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.78 on 10.11.54.4 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.5]); Mon, 12 Mar 2018 17:21:41 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.5]); Mon, 12 Mar 2018 17:21:41 +0000 (UTC) for IP:'10.11.54.4' DOMAIN:'int-mx04.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'dgilbert@redhat.com' RCPT:'' Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH v5 07/29] libvhost-user: Support sending fds back to qemu X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: aarcange@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Type: text/plain; charset="utf-8" From: "Dr. David Alan Gilbert" Allow replies with fds (for postcopy) Signed-off-by: Dr. David Alan Gilbert Reviewed-by: Marc-Andr=C3=A9 Lureau --- contrib/libvhost-user/libvhost-user.c | 30 +++++++++++++++++++++++++++++- 1 file changed, 29 insertions(+), 1 deletion(-) diff --git a/contrib/libvhost-user/libvhost-user.c b/contrib/libvhost-user/= libvhost-user.c index 37d4228193..ed9f3148d3 100644 --- a/contrib/libvhost-user/libvhost-user.c +++ b/contrib/libvhost-user/libvhost-user.c @@ -246,6 +246,31 @@ vu_message_write(VuDev *dev, int conn_fd, VhostUserMsg= *vmsg) { int rc; uint8_t *p =3D (uint8_t *)vmsg; + char control[CMSG_SPACE(VHOST_MEMORY_MAX_NREGIONS * sizeof(int))] =3D = { }; + struct iovec iov =3D { + .iov_base =3D (char *)vmsg, + .iov_len =3D VHOST_USER_HDR_SIZE, + }; + struct msghdr msg =3D { + .msg_iov =3D &iov, + .msg_iovlen =3D 1, + .msg_control =3D control, + }; + struct cmsghdr *cmsg; + + memset(control, 0, sizeof(control)); + assert(vmsg->fd_num <=3D VHOST_MEMORY_MAX_NREGIONS); + if (vmsg->fd_num > 0) { + size_t fdsize =3D vmsg->fd_num * sizeof(int); + msg.msg_controllen =3D CMSG_SPACE(fdsize); + cmsg =3D CMSG_FIRSTHDR(&msg); + cmsg->cmsg_len =3D CMSG_LEN(fdsize); + cmsg->cmsg_level =3D SOL_SOCKET; + cmsg->cmsg_type =3D SCM_RIGHTS; + memcpy(CMSG_DATA(cmsg), vmsg->fds, fdsize); + } else { + msg.msg_controllen =3D 0; + } =20 /* Set the version in the flags when sending the reply */ vmsg->flags &=3D ~VHOST_USER_VERSION_MASK; @@ -253,7 +278,7 @@ vu_message_write(VuDev *dev, int conn_fd, VhostUserMsg = *vmsg) vmsg->flags |=3D VHOST_USER_REPLY_MASK; =20 do { - rc =3D write(conn_fd, p, VHOST_USER_HDR_SIZE); + rc =3D sendmsg(conn_fd, &msg, 0); } while (rc < 0 && (errno =3D=3D EINTR || errno =3D=3D EAGAIN)); =20 do { @@ -346,6 +371,7 @@ vu_get_features_exec(VuDev *dev, VhostUserMsg *vmsg) } =20 vmsg->size =3D sizeof(vmsg->payload.u64); + vmsg->fd_num =3D 0; =20 DPRINT("Sending back to guest u64: 0x%016"PRIx64"\n", vmsg->payload.u6= 4); =20 @@ -501,6 +527,7 @@ vu_set_log_base_exec(VuDev *dev, VhostUserMsg *vmsg) dev->log_size =3D log_mmap_size; =20 vmsg->size =3D sizeof(vmsg->payload.u64); + vmsg->fd_num =3D 0; =20 return true; } @@ -759,6 +786,7 @@ vu_get_protocol_features_exec(VuDev *dev, VhostUserMsg = *vmsg) =20 vmsg->payload.u64 =3D features; vmsg->size =3D sizeof(vmsg->payload.u64); + vmsg->fd_num =3D 0; =20 return true; } --=20 2.14.3 From nobody Sun May 5 21:21:29 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1520875928748110.2039100219323; Mon, 12 Mar 2018 10:32:08 -0700 (PDT) Received: from localhost ([::1]:33424 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1evRIp-0000Hp-RK for importer@patchew.org; Mon, 12 Mar 2018 13:32:03 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52977) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1evR8p-000846-U4 for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:21:44 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1evR8p-0007XQ-0s for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:21:43 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:56686 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1evR8o-0007Wi-Sj for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:21:42 -0400 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 886A3401C9AE for ; Mon, 12 Mar 2018 17:21:42 +0000 (UTC) Received: from dgilbert-t530.redhat.com (ovpn-117-169.ams2.redhat.com [10.36.117.169]) by smtp.corp.redhat.com (Postfix) with ESMTP id 5030F202322C; Mon, 12 Mar 2018 17:21:41 +0000 (UTC) From: "Dr. David Alan Gilbert (git)" To: qemu-devel@nongnu.org, mst@redhat.com, maxime.coquelin@redhat.com, marcandre.lureau@redhat.com, peterx@redhat.com, quintela@redhat.com Date: Mon, 12 Mar 2018 17:21:03 +0000 Message-Id: <20180312172124.56461-9-dgilbert@redhat.com> In-Reply-To: <20180312172124.56461-1-dgilbert@redhat.com> References: <20180312172124.56461-1-dgilbert@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.78 on 10.11.54.4 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.5]); Mon, 12 Mar 2018 17:21:42 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.5]); Mon, 12 Mar 2018 17:21:42 +0000 (UTC) for IP:'10.11.54.4' DOMAIN:'int-mx04.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'dgilbert@redhat.com' RCPT:'' Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH v5 08/29] libvhost-user: Open userfaultfd X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: aarcange@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Type: text/plain; charset="utf-8" From: "Dr. David Alan Gilbert" Open a userfaultfd (on a postcopy_advise) and send it back in the reply to the qemu for it to monitor. Signed-off-by: Dr. David Alan Gilbert Reviewed-by: Marc-Andr=C3=A9 Lureau --- contrib/libvhost-user/libvhost-user.c | 42 +++++++++++++++++++++++++++++++= ++-- contrib/libvhost-user/libvhost-user.h | 3 +++ 2 files changed, 43 insertions(+), 2 deletions(-) diff --git a/contrib/libvhost-user/libvhost-user.c b/contrib/libvhost-user/= libvhost-user.c index ed9f3148d3..9e31f47b7a 100644 --- a/contrib/libvhost-user/libvhost-user.c +++ b/contrib/libvhost-user/libvhost-user.c @@ -26,9 +26,20 @@ #include #include #include +#include "qemu/compiler.h" + +#if defined(__linux__) +#include +#include +#include #include =20 -#include "qemu/compiler.h" +#ifdef __NR_userfaultfd +#include +#endif + +#endif + #include "qemu/atomic.h" =20 #include "libvhost-user.h" @@ -888,8 +899,35 @@ vu_set_config(VuDev *dev, VhostUserMsg *vmsg) static bool vu_set_postcopy_advise(VuDev *dev, VhostUserMsg *vmsg) { - /* TODO: Open ufd, pass it back in the request */ + dev->postcopy_ufd =3D -1; +#ifdef UFFDIO_API + struct uffdio_api api_struct; + + dev->postcopy_ufd =3D syscall(__NR_userfaultfd, O_CLOEXEC | O_NONBLOCK= ); vmsg->size =3D 0; +#endif + + if (dev->postcopy_ufd =3D=3D -1) { + vu_panic(dev, "Userfaultfd not available: %s", strerror(errno)); + goto out; + } + +#ifdef UFFDIO_API + api_struct.api =3D UFFD_API; + api_struct.features =3D 0; + if (ioctl(dev->postcopy_ufd, UFFDIO_API, &api_struct)) { + vu_panic(dev, "Failed UFFDIO_API: %s", strerror(errno)); + close(dev->postcopy_ufd); + dev->postcopy_ufd =3D -1; + goto out; + } + /* TODO: Stash feature flags somewhere */ +#endif + +out: + /* Return a ufd to the QEMU */ + vmsg->fd_num =3D 1; + vmsg->fds[0] =3D dev->postcopy_ufd; return true; /* =3D send a reply */ } =20 diff --git a/contrib/libvhost-user/libvhost-user.h b/contrib/libvhost-user/= libvhost-user.h index 00d78a8810..074b7860f6 100644 --- a/contrib/libvhost-user/libvhost-user.h +++ b/contrib/libvhost-user/libvhost-user.h @@ -282,6 +282,9 @@ struct VuDev { * re-initialize */ vu_panic_cb panic; const VuDevIface *iface; + + /* Postcopy data */ + int postcopy_ufd; }; =20 typedef struct VuVirtqElement { --=20 2.14.3 From nobody Sun May 5 21:21:29 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1520875593160828.6711021797466; Mon, 12 Mar 2018 10:26:33 -0700 (PDT) Received: from localhost ([::1]:33395 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1evRDU-0003lk-82 for importer@patchew.org; Mon, 12 Mar 2018 13:26:32 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:53018) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1evR8u-00088S-KZ for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:21:50 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1evR8r-0007ZQ-CG for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:21:48 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:57604 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1evR8r-0007Z0-66 for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:21:45 -0400 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id CD157401DE71 for ; Mon, 12 Mar 2018 17:21:44 +0000 (UTC) Received: from dgilbert-t530.redhat.com (ovpn-117-169.ams2.redhat.com [10.36.117.169]) by smtp.corp.redhat.com (Postfix) with ESMTP id 1A275202322A; Mon, 12 Mar 2018 17:21:42 +0000 (UTC) From: "Dr. David Alan Gilbert (git)" To: qemu-devel@nongnu.org, mst@redhat.com, maxime.coquelin@redhat.com, marcandre.lureau@redhat.com, peterx@redhat.com, quintela@redhat.com Date: Mon, 12 Mar 2018 17:21:04 +0000 Message-Id: <20180312172124.56461-10-dgilbert@redhat.com> In-Reply-To: <20180312172124.56461-1-dgilbert@redhat.com> References: <20180312172124.56461-1-dgilbert@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.4 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.6]); Mon, 12 Mar 2018 17:21:44 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.6]); Mon, 12 Mar 2018 17:21:44 +0000 (UTC) for IP:'10.11.54.4' DOMAIN:'int-mx04.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'dgilbert@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH v5 09/29] postcopy: Allow registering of fd handler X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: aarcange@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: "Dr. David Alan Gilbert" Allow other userfaultfd's to be registered into the fault thread so that handlers for shared memory can get responses. Signed-off-by: Dr. David Alan Gilbert Reviewed-by: Peter Xu --- migration/migration.c | 6 ++ migration/migration.h | 2 + migration/postcopy-ram.c | 210 +++++++++++++++++++++++++++++++++++--------= ---- migration/postcopy-ram.h | 21 +++++ migration/trace-events | 2 + 5 files changed, 188 insertions(+), 53 deletions(-) diff --git a/migration/migration.c b/migration/migration.c index 6a4780ef6f..1f22f463d3 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -155,6 +155,8 @@ MigrationIncomingState *migration_incoming_get_current(= void) if (!once) { mis_current.state =3D MIGRATION_STATUS_NONE; memset(&mis_current, 0, sizeof(MigrationIncomingState)); + mis_current.postcopy_remote_fds =3D g_array_new(FALSE, TRUE, + sizeof(struct PostCopyF= D)); qemu_mutex_init(&mis_current.rp_mutex); qemu_event_init(&mis_current.main_thread_load_event, false); once =3D true; @@ -177,6 +179,10 @@ void migration_incoming_state_destroy(void) qemu_fclose(mis->from_src_file); mis->from_src_file =3D NULL; } + if (mis->postcopy_remote_fds) { + g_array_free(mis->postcopy_remote_fds, TRUE); + mis->postcopy_remote_fds =3D NULL; + } =20 qemu_event_reset(&mis->main_thread_load_event); } diff --git a/migration/migration.h b/migration/migration.h index 08c5d2ded1..d02a759331 100644 --- a/migration/migration.h +++ b/migration/migration.h @@ -51,6 +51,8 @@ struct MigrationIncomingState { QemuMutex rp_mutex; /* We send replies from multiple threads */ void *postcopy_tmp_page; void *postcopy_tmp_zero_page; + /* PostCopyFD's for external userfaultfds & handlers of shared memory = */ + GArray *postcopy_remote_fds; =20 QEMUBH *bh; =20 diff --git a/migration/postcopy-ram.c b/migration/postcopy-ram.c index 1089814d54..4ab1b7d36d 100644 --- a/migration/postcopy-ram.c +++ b/migration/postcopy-ram.c @@ -533,29 +533,44 @@ static void *postcopy_ram_fault_thread(void *opaque) MigrationIncomingState *mis =3D opaque; struct uffd_msg msg; int ret; + size_t index; RAMBlock *rb =3D NULL; RAMBlock *last_rb =3D NULL; /* last RAMBlock we sent part of */ =20 trace_postcopy_ram_fault_thread_entry(); qemu_sem_post(&mis->fault_thread_sem); =20 + struct pollfd *pfd; + size_t pfd_len =3D 2 + mis->postcopy_remote_fds->len; + + pfd =3D g_new0(struct pollfd, pfd_len); + + pfd[0].fd =3D mis->userfault_fd; + pfd[0].events =3D POLLIN; + pfd[1].fd =3D mis->userfault_event_fd; + pfd[1].events =3D POLLIN; /* Waiting for eventfd to go positive */ + trace_postcopy_ram_fault_thread_fds_core(pfd[0].fd, pfd[1].fd); + for (index =3D 0; index < mis->postcopy_remote_fds->len; index++) { + struct PostCopyFD *pcfd =3D &g_array_index(mis->postcopy_remote_fd= s, + struct PostCopyFD, index); + pfd[2 + index].fd =3D pcfd->fd; + pfd[2 + index].events =3D POLLIN; + trace_postcopy_ram_fault_thread_fds_extra(2 + index, pcfd->idstr, + pcfd->fd); + } + while (true) { ram_addr_t rb_offset; - struct pollfd pfd[2]; + int poll_result; =20 /* * We're mainly waiting for the kernel to give us a faulting HVA, * however we can be told to quit via userfault_quit_fd which is * an eventfd */ - pfd[0].fd =3D mis->userfault_fd; - pfd[0].events =3D POLLIN; - pfd[0].revents =3D 0; - pfd[1].fd =3D mis->userfault_event_fd; - pfd[1].events =3D POLLIN; /* Waiting for eventfd to go positive */ - pfd[1].revents =3D 0; - - if (poll(pfd, 2, -1 /* Wait forever */) =3D=3D -1) { + + poll_result =3D poll(pfd, pfd_len, -1 /* Wait forever */); + if (poll_result =3D=3D -1) { error_report("%s: userfault poll: %s", __func__, strerror(errn= o)); break; } @@ -575,57 +590,117 @@ static void *postcopy_ram_fault_thread(void *opaque) } } =20 - ret =3D read(mis->userfault_fd, &msg, sizeof(msg)); - if (ret !=3D sizeof(msg)) { - if (errno =3D=3D EAGAIN) { - /* - * if a wake up happens on the other thread just after - * the poll, there is nothing to read. - */ - continue; + if (pfd[0].revents) { + poll_result--; + ret =3D read(mis->userfault_fd, &msg, sizeof(msg)); + if (ret !=3D sizeof(msg)) { + if (errno =3D=3D EAGAIN) { + /* + * if a wake up happens on the other thread just after + * the poll, there is nothing to read. + */ + continue; + } + if (ret < 0) { + error_report("%s: Failed to read full userfault " + "message: %s", + __func__, strerror(errno)); + break; + } else { + error_report("%s: Read %d bytes from userfaultfd " + "expected %zd", + __func__, ret, sizeof(msg)); + break; /* Lost alignment, don't know what we'd read ne= xt */ + } } - if (ret < 0) { - error_report("%s: Failed to read full userfault message: %= s", - __func__, strerror(errno)); - break; - } else { - error_report("%s: Read %d bytes from userfaultfd expected = %zd", - __func__, ret, sizeof(msg)); - break; /* Lost alignment, don't know what we'd read next */ + if (msg.event !=3D UFFD_EVENT_PAGEFAULT) { + error_report("%s: Read unexpected event %ud from userfault= fd", + __func__, msg.event); + continue; /* It's not a page fault, shouldn't happen */ } - } - if (msg.event !=3D UFFD_EVENT_PAGEFAULT) { - error_report("%s: Read unexpected event %ud from userfaultfd", - __func__, msg.event); - continue; /* It's not a page fault, shouldn't happen */ - } =20 - rb =3D qemu_ram_block_from_host( - (void *)(uintptr_t)msg.arg.pagefault.address, - true, &rb_offset); - if (!rb) { - error_report("postcopy_ram_fault_thread: Fault outside guest: = %" - PRIx64, (uint64_t)msg.arg.pagefault.address); - break; - } + rb =3D qemu_ram_block_from_host( + (void *)(uintptr_t)msg.arg.pagefault.address, + true, &rb_offset); + if (!rb) { + error_report("postcopy_ram_fault_thread: Fault outside gue= st: %" + PRIx64, (uint64_t)msg.arg.pagefault.address); + break; + } =20 - rb_offset &=3D ~(qemu_ram_pagesize(rb) - 1); - trace_postcopy_ram_fault_thread_request(msg.arg.pagefault.address, + rb_offset &=3D ~(qemu_ram_pagesize(rb) - 1); + trace_postcopy_ram_fault_thread_request(msg.arg.pagefault.addr= ess, qemu_ram_get_idstr(rb), rb_offset); + /* + * Send the request to the source - we want to request one + * of our host page sizes (which is >=3D TPS) + */ + if (rb !=3D last_rb) { + last_rb =3D rb; + migrate_send_rp_req_pages(mis, qemu_ram_get_idstr(rb), + rb_offset, qemu_ram_pagesize(rb)); + } else { + /* Save some space */ + migrate_send_rp_req_pages(mis, NULL, + rb_offset, qemu_ram_pagesize(rb)); + } + } =20 - /* - * Send the request to the source - we want to request one - * of our host page sizes (which is >=3D TPS) - */ - if (rb !=3D last_rb) { - last_rb =3D rb; - migrate_send_rp_req_pages(mis, qemu_ram_get_idstr(rb), - rb_offset, qemu_ram_pagesize(rb)); - } else { - /* Save some space */ - migrate_send_rp_req_pages(mis, NULL, - rb_offset, qemu_ram_pagesize(rb)); + /* Now handle any requests from external processes on shared memor= y */ + /* TODO: May need to handle devices deregistering during postcopy = */ + for (index =3D 2; index < pfd_len && poll_result; index++) { + if (pfd[index].revents) { + struct PostCopyFD *pcfd =3D + &g_array_index(mis->postcopy_remote_fds, + struct PostCopyFD, index - 2); + + poll_result--; + if (pfd[index].revents & POLLERR) { + error_report("%s: POLLERR on poll %zd fd=3D%d", + __func__, index, pcfd->fd); + pfd[index].events =3D 0; + continue; + } + + ret =3D read(pcfd->fd, &msg, sizeof(msg)); + if (ret !=3D sizeof(msg)) { + if (errno =3D=3D EAGAIN) { + /* + * if a wake up happens on the other thread just a= fter + * the poll, there is nothing to read. + */ + continue; + } + if (ret < 0) { + error_report("%s: Failed to read full userfault " + "message: %s (shared) revents=3D%d", + __func__, strerror(errno), + pfd[index].revents); + /*TODO: Could just disable this sharer */ + break; + } else { + error_report("%s: Read %d bytes from userfaultfd " + "expected %zd (shared)", + __func__, ret, sizeof(msg)); + /*TODO: Could just disable this sharer */ + break; /*Lost alignment,don't know what we'd read = next*/ + } + } + if (msg.event !=3D UFFD_EVENT_PAGEFAULT) { + error_report("%s: Read unexpected event %ud " + "from userfaultfd (shared)", + __func__, msg.event); + continue; /* It's not a page fault, shouldn't happen */ + } + /* Call the device handler registered with us */ + ret =3D pcfd->handler(pcfd, &msg); + if (ret) { + error_report("%s: Failed to resolve shared fault on %z= d/%s", + __func__, index, pcfd->idstr); + /* TODO: Fail? Disable this sharer? */ + } + } } } trace_postcopy_ram_fault_thread_exit(); @@ -970,3 +1045,32 @@ PostcopyState postcopy_state_set(PostcopyState new_st= ate) { return atomic_xchg(&incoming_postcopy_state, new_state); } + +/* Register a handler for external shared memory postcopy + * called on the destination. + */ +void postcopy_register_shared_ufd(struct PostCopyFD *pcfd) +{ + MigrationIncomingState *mis =3D migration_incoming_get_current(); + + mis->postcopy_remote_fds =3D g_array_append_val(mis->postcopy_remote_f= ds, + *pcfd); +} + +/* Unregister a handler for external shared memory postcopy + */ +void postcopy_unregister_shared_ufd(struct PostCopyFD *pcfd) +{ + guint i; + MigrationIncomingState *mis =3D migration_incoming_get_current(); + GArray *pcrfds =3D mis->postcopy_remote_fds; + + for (i =3D 0; i < pcrfds->len; i++) { + struct PostCopyFD *cur =3D &g_array_index(pcrfds, struct PostCopyF= D, i); + if (cur->fd =3D=3D pcfd->fd) { + mis->postcopy_remote_fds =3D g_array_remove_index(pcrfds, i); + return; + } + } +} + diff --git a/migration/postcopy-ram.h b/migration/postcopy-ram.h index 0421c98d57..f21eef6702 100644 --- a/migration/postcopy-ram.h +++ b/migration/postcopy-ram.h @@ -143,4 +143,25 @@ void postcopy_remove_notifier(NotifierWithReturn *n); /* Call the notifier list set by postcopy_add_start_notifier */ int postcopy_notify(enum PostcopyNotifyReason reason, Error **errp); =20 +struct PostCopyFD; + +/* ufd is a pointer to the struct uffd_msg *TODO: more Portable! */ +typedef int (*pcfdhandler)(struct PostCopyFD *pcfd, void *ufd); + +struct PostCopyFD { + int fd; + /* Data to pass to handler */ + void *data; + /* Handler to be called whenever we get a poll event */ + pcfdhandler handler; + /* A string to use in error messages */ + const char *idstr; +}; + +/* Register a userfaultfd owned by an external process for + * shared memory. + */ +void postcopy_register_shared_ufd(struct PostCopyFD *pcfd); +void postcopy_unregister_shared_ufd(struct PostCopyFD *pcfd); + #endif diff --git a/migration/trace-events b/migration/trace-events index 93961dea16..1e617ad7a6 100644 --- a/migration/trace-events +++ b/migration/trace-events @@ -190,6 +190,8 @@ postcopy_place_page_zero(void *host_addr) "host=3D%p" postcopy_ram_enable_notify(void) "" postcopy_ram_fault_thread_entry(void) "" postcopy_ram_fault_thread_exit(void) "" +postcopy_ram_fault_thread_fds_core(int baseufd, int quitfd) "ufd: %d quitf= d: %d" +postcopy_ram_fault_thread_fds_extra(size_t index, const char *name, int fd= ) "%zd/%s: %d" postcopy_ram_fault_thread_quit(void) "" postcopy_ram_fault_thread_request(uint64_t hostaddr, const char *ramblock,= size_t offset) "Request for HVA=3D0x%" PRIx64 " rb=3D%s offset=3D0x%zx" postcopy_ram_incoming_cleanup_closeuf(void) "" --=20 2.14.3 From nobody Sun May 5 21:21:29 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1520875927346509.67321207590464; Mon, 12 Mar 2018 10:32:07 -0700 (PDT) Received: from localhost ([::1]:33425 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1evRIs-0000K3-Ix for importer@patchew.org; Mon, 12 Mar 2018 13:32:06 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:53019) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1evR8u-00088T-KU for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:21:50 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1evR8t-0007aI-1v for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:21:48 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:57606 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1evR8s-0007Zs-Tq for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:21:46 -0400 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 4EA59401DE71 for ; Mon, 12 Mar 2018 17:21:46 +0000 (UTC) Received: from dgilbert-t530.redhat.com (ovpn-117-169.ams2.redhat.com [10.36.117.169]) by smtp.corp.redhat.com (Postfix) with ESMTP id 13AF9202322A; Mon, 12 Mar 2018 17:21:44 +0000 (UTC) From: "Dr. David Alan Gilbert (git)" To: qemu-devel@nongnu.org, mst@redhat.com, maxime.coquelin@redhat.com, marcandre.lureau@redhat.com, peterx@redhat.com, quintela@redhat.com Date: Mon, 12 Mar 2018 17:21:05 +0000 Message-Id: <20180312172124.56461-11-dgilbert@redhat.com> In-Reply-To: <20180312172124.56461-1-dgilbert@redhat.com> References: <20180312172124.56461-1-dgilbert@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.78 on 10.11.54.4 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.6]); Mon, 12 Mar 2018 17:21:46 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.6]); Mon, 12 Mar 2018 17:21:46 +0000 (UTC) for IP:'10.11.54.4' DOMAIN:'int-mx04.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'dgilbert@redhat.com' RCPT:'' Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH v5 10/29] vhost+postcopy: Register shared ufd with postcopy X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: aarcange@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Type: text/plain; charset="utf-8" From: "Dr. David Alan Gilbert" Register the UFD that comes in as the response to the 'advise' method with the postcopy code. Signed-off-by: Dr. David Alan Gilbert Reviewed-by: Marc-Andr=C3=A9 Lureau --- hw/virtio/vhost-user.c | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/hw/virtio/vhost-user.c b/hw/virtio/vhost-user.c index ceb17b0554..5900583437 100644 --- a/hw/virtio/vhost-user.c +++ b/hw/virtio/vhost-user.c @@ -171,6 +171,7 @@ struct vhost_user { CharBackend *chr; int slave_fd; NotifierWithReturn postcopy_notifier; + struct PostCopyFD postcopy_fd; }; =20 static bool ioeventfd_enabled(void) @@ -796,6 +797,17 @@ out: return ret; } =20 +/* + * Called back from the postcopy fault thread when a fault is received on = our + * ufd. + * TODO: This is Linux specific + */ +static int vhost_user_postcopy_fault_handler(struct PostCopyFD *pcfd, + void *ufd) +{ + return 0; +} + /* * Called at the start of an inbound postcopy on reception of the * 'advise' command. @@ -835,8 +847,14 @@ static int vhost_user_postcopy_advise(struct vhost_dev= *dev, Error **errp) error_setg(errp, "%s: Failed to get ufd", __func__); return -1; } + fcntl(ufd, F_SETFL, O_NONBLOCK); =20 - /* TODO: register ufd with userfault thread */ + /* register ufd with userfault thread */ + u->postcopy_fd.fd =3D ufd; + u->postcopy_fd.data =3D dev; + u->postcopy_fd.handler =3D vhost_user_postcopy_fault_handler; + u->postcopy_fd.idstr =3D "vhost-user"; /* Need to find unique name */ + postcopy_register_shared_ufd(&u->postcopy_fd); return 0; } =20 --=20 2.14.3 From nobody Sun May 5 21:21:29 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1520875589668474.0710302888366; Mon, 12 Mar 2018 10:26:29 -0700 (PDT) Received: from localhost ([::1]:33394 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1evRDQ-0003hY-Hj for importer@patchew.org; Mon, 12 Mar 2018 13:26:28 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:53036) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1evR8v-00088e-R0 for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:21:51 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1evR8u-0007b0-Bi for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:21:49 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:47216 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1evR8u-0007am-5R for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:21:48 -0400 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id C3ADC4067F25 for ; Mon, 12 Mar 2018 17:21:47 +0000 (UTC) Received: from dgilbert-t530.redhat.com (ovpn-117-169.ams2.redhat.com [10.36.117.169]) by smtp.corp.redhat.com (Postfix) with ESMTP id 897692026E03; Mon, 12 Mar 2018 17:21:46 +0000 (UTC) From: "Dr. David Alan Gilbert (git)" To: qemu-devel@nongnu.org, mst@redhat.com, maxime.coquelin@redhat.com, marcandre.lureau@redhat.com, peterx@redhat.com, quintela@redhat.com Date: Mon, 12 Mar 2018 17:21:06 +0000 Message-Id: <20180312172124.56461-12-dgilbert@redhat.com> In-Reply-To: <20180312172124.56461-1-dgilbert@redhat.com> References: <20180312172124.56461-1-dgilbert@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.78 on 10.11.54.4 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.7]); Mon, 12 Mar 2018 17:21:47 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.7]); Mon, 12 Mar 2018 17:21:47 +0000 (UTC) for IP:'10.11.54.4' DOMAIN:'int-mx04.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'dgilbert@redhat.com' RCPT:'' Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH v5 11/29] vhost+postcopy: Transmit 'listen' to slave X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: aarcange@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Type: text/plain; charset="utf-8" From: "Dr. David Alan Gilbert" Notify the vhost-user slave on reception of the 'postcopy-listen' event from the source. Signed-off-by: Dr. David Alan Gilbert Reviewed-by: Marc-Andr=C3=A9 Lureau Reviewed-by: Peter Xu --- contrib/libvhost-user/libvhost-user.c | 19 +++++++++++++++++++ contrib/libvhost-user/libvhost-user.h | 2 ++ docs/interop/vhost-user.txt | 11 +++++++++++ hw/virtio/trace-events | 3 +++ hw/virtio/vhost-user.c | 34 +++++++++++++++++++++++++++++++= +++ migration/postcopy-ram.h | 1 + migration/savevm.c | 7 +++++++ 7 files changed, 77 insertions(+) diff --git a/contrib/libvhost-user/libvhost-user.c b/contrib/libvhost-user/= libvhost-user.c index 9e31f47b7a..e53b1953df 100644 --- a/contrib/libvhost-user/libvhost-user.c +++ b/contrib/libvhost-user/libvhost-user.c @@ -98,6 +98,7 @@ vu_request_to_string(unsigned int req) REQ(VHOST_USER_GET_CONFIG), REQ(VHOST_USER_SET_CONFIG), REQ(VHOST_USER_POSTCOPY_ADVISE), + REQ(VHOST_USER_POSTCOPY_LISTEN), REQ(VHOST_USER_MAX), }; #undef REQ @@ -931,6 +932,22 @@ out: return true; /* =3D send a reply */ } =20 +static bool +vu_set_postcopy_listen(VuDev *dev, VhostUserMsg *vmsg) +{ + vmsg->payload.u64 =3D -1; + vmsg->size =3D sizeof(vmsg->payload.u64); + + if (dev->nregions) { + vu_panic(dev, "Regions already registered at postcopy-listen"); + return true; + } + dev->postcopy_listening =3D true; + + vmsg->flags =3D VHOST_USER_VERSION | VHOST_USER_REPLY_MASK; + vmsg->payload.u64 =3D 0; /* Success */ + return true; +} static bool vu_process_message(VuDev *dev, VhostUserMsg *vmsg) { @@ -1004,6 +1021,8 @@ vu_process_message(VuDev *dev, VhostUserMsg *vmsg) break; case VHOST_USER_POSTCOPY_ADVISE: return vu_set_postcopy_advise(dev, vmsg); + case VHOST_USER_POSTCOPY_LISTEN: + return vu_set_postcopy_listen(dev, vmsg); default: vmsg_close_fds(vmsg); vu_panic(dev, "Unhandled request: %d", vmsg->request); diff --git a/contrib/libvhost-user/libvhost-user.h b/contrib/libvhost-user/= libvhost-user.h index 074b7860f6..ed505cf0c1 100644 --- a/contrib/libvhost-user/libvhost-user.h +++ b/contrib/libvhost-user/libvhost-user.h @@ -86,6 +86,7 @@ typedef enum VhostUserRequest { VHOST_USER_CREATE_CRYPTO_SESSION =3D 26, VHOST_USER_CLOSE_CRYPTO_SESSION =3D 27, VHOST_USER_POSTCOPY_ADVISE =3D 28, + VHOST_USER_POSTCOPY_LISTEN =3D 29, VHOST_USER_MAX } VhostUserRequest; =20 @@ -285,6 +286,7 @@ struct VuDev { =20 /* Postcopy data */ int postcopy_ufd; + bool postcopy_listening; }; =20 typedef struct VuVirtqElement { diff --git a/docs/interop/vhost-user.txt b/docs/interop/vhost-user.txt index 7854e50008..0d24203d31 100644 --- a/docs/interop/vhost-user.txt +++ b/docs/interop/vhost-user.txt @@ -709,6 +709,17 @@ Master message types the slave must open a userfaultfd for later use. Note that at this stage the migration is still in precopy mode. =20 + * VHOST_USER_POSTCOPY_LISTEN + Id: 29 + Master payload: N/A + + Master advises slave that a transition to postcopy mode has happened. + The slave must ensure that shared memory is registered with userfaul= tfd + to cause faulting of non-present pages. + + This is always sent sometime after a VHOST_USER_POSTCOPY_ADVISE, and + thus only when VHOST_USER_PROTOCOL_F_PAGEFAULT is supported. + Slave message types ------------------- =20 diff --git a/hw/virtio/trace-events b/hw/virtio/trace-events index 742ff0f90b..06ec03d6e7 100644 --- a/hw/virtio/trace-events +++ b/hw/virtio/trace-events @@ -6,6 +6,9 @@ vhost_region_add_section(const char *name, uint64_t gpa, ui= nt64_t size, uint64_t vhost_region_add_section_abut(const char *name, uint64_t new_size) "%s: 0x= %"PRIx64 vhost_section(const char *name, int r) "%s:%d" =20 +# hw/virtio/vhost-user.c +vhost_user_postcopy_listen(void) "" + # hw/virtio/virtio.c virtqueue_alloc_element(void *elem, size_t sz, unsigned in_num, unsigned o= ut_num) "elem %p size %zd in_num %u out_num %u" virtqueue_fill(void *vq, const void *elem, unsigned int len, unsigned int = idx) "vq %p elem %p len %u idx %u" diff --git a/hw/virtio/vhost-user.c b/hw/virtio/vhost-user.c index 5900583437..c3ab2994fb 100644 --- a/hw/virtio/vhost-user.c +++ b/hw/virtio/vhost-user.c @@ -20,6 +20,7 @@ #include "sysemu/cryptodev.h" #include "migration/migration.h" #include "migration/postcopy-ram.h" +#include "trace.h" =20 #include #include @@ -79,6 +80,7 @@ typedef enum VhostUserRequest { VHOST_USER_CREATE_CRYPTO_SESSION =3D 26, VHOST_USER_CLOSE_CRYPTO_SESSION =3D 27, VHOST_USER_POSTCOPY_ADVISE =3D 28, + VHOST_USER_POSTCOPY_LISTEN =3D 29, VHOST_USER_MAX } VhostUserRequest; =20 @@ -172,6 +174,8 @@ struct vhost_user { int slave_fd; NotifierWithReturn postcopy_notifier; struct PostCopyFD postcopy_fd; + /* True once we've entered postcopy_listen */ + bool postcopy_listen; }; =20 static bool ioeventfd_enabled(void) @@ -858,6 +862,33 @@ static int vhost_user_postcopy_advise(struct vhost_dev= *dev, Error **errp) return 0; } =20 +/* + * Called at the switch to postcopy on reception of the 'listen' command. + */ +static int vhost_user_postcopy_listen(struct vhost_dev *dev, Error **errp) +{ + struct vhost_user *u =3D dev->opaque; + int ret; + VhostUserMsg msg =3D { + .hdr.request =3D VHOST_USER_POSTCOPY_LISTEN, + .hdr.flags =3D VHOST_USER_VERSION | VHOST_USER_NEED_REPLY_MASK, + }; + u->postcopy_listen =3D true; + trace_vhost_user_postcopy_listen(); + if (vhost_user_write(dev, &msg, NULL, 0) < 0) { + error_setg(errp, "Failed to send postcopy_listen to vhost"); + return -1; + } + + ret =3D process_message_reply(dev, &msg); + if (ret) { + error_setg(errp, "Failed to receive reply to postcopy_listen"); + return ret; + } + + return 0; +} + static int vhost_user_postcopy_notifier(NotifierWithReturn *notifier, void *opaque) { @@ -880,6 +911,9 @@ static int vhost_user_postcopy_notifier(NotifierWithRet= urn *notifier, case POSTCOPY_NOTIFY_INBOUND_ADVISE: return vhost_user_postcopy_advise(dev, pnd->errp); =20 + case POSTCOPY_NOTIFY_INBOUND_LISTEN: + return vhost_user_postcopy_listen(dev, pnd->errp); + default: /* We ignore notifications we don't know */ break; diff --git a/migration/postcopy-ram.h b/migration/postcopy-ram.h index f21eef6702..c8ced3470b 100644 --- a/migration/postcopy-ram.h +++ b/migration/postcopy-ram.h @@ -131,6 +131,7 @@ void postcopy_infrastructure_init(void); enum PostcopyNotifyReason { POSTCOPY_NOTIFY_PROBE =3D 0, POSTCOPY_NOTIFY_INBOUND_ADVISE, + POSTCOPY_NOTIFY_INBOUND_LISTEN, }; =20 struct PostcopyNotifyData { diff --git a/migration/savevm.c b/migration/savevm.c index 1f2bf12a28..305c3ceaf5 100644 --- a/migration/savevm.c +++ b/migration/savevm.c @@ -1618,6 +1618,8 @@ static int loadvm_postcopy_handle_listen(MigrationInc= omingState *mis) { PostcopyState ps =3D postcopy_state_set(POSTCOPY_INCOMING_LISTENING); trace_loadvm_postcopy_handle_listen(); + Error *local_err =3D NULL; + if (ps !=3D POSTCOPY_INCOMING_ADVISE && ps !=3D POSTCOPY_INCOMING_DISC= ARD) { error_report("CMD_POSTCOPY_LISTEN in wrong postcopy state (%d)", p= s); return -1; @@ -1643,6 +1645,11 @@ static int loadvm_postcopy_handle_listen(MigrationIn= comingState *mis) } } =20 + if (postcopy_notify(POSTCOPY_NOTIFY_INBOUND_LISTEN, &local_err)) { + error_report_err(local_err); + return -1; + } + if (mis->have_listen_thread) { error_report("CMD_POSTCOPY_RAM_LISTEN already has a listen thread"= ); return -1; --=20 2.14.3 From nobody Sun May 5 21:21:29 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1520875752001517.4017130612353; Mon, 12 Mar 2018 10:29:12 -0700 (PDT) Received: from localhost ([::1]:33408 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1evRG3-00068D-7O for importer@patchew.org; Mon, 12 Mar 2018 13:29:11 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:53052) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1evR8x-00089N-3X for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:21:52 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1evR8v-0007c2-Q2 for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:21:51 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:56690 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1evR8v-0007bo-Ka for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:21:49 -0400 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 46272401C9AE for ; Mon, 12 Mar 2018 17:21:49 +0000 (UTC) Received: from dgilbert-t530.redhat.com (ovpn-117-169.ams2.redhat.com [10.36.117.169]) by smtp.corp.redhat.com (Postfix) with ESMTP id 0AC32202322C; Mon, 12 Mar 2018 17:21:47 +0000 (UTC) From: "Dr. David Alan Gilbert (git)" To: qemu-devel@nongnu.org, mst@redhat.com, maxime.coquelin@redhat.com, marcandre.lureau@redhat.com, peterx@redhat.com, quintela@redhat.com Date: Mon, 12 Mar 2018 17:21:07 +0000 Message-Id: <20180312172124.56461-13-dgilbert@redhat.com> In-Reply-To: <20180312172124.56461-1-dgilbert@redhat.com> References: <20180312172124.56461-1-dgilbert@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.78 on 10.11.54.4 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.5]); Mon, 12 Mar 2018 17:21:49 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.5]); Mon, 12 Mar 2018 17:21:49 +0000 (UTC) for IP:'10.11.54.4' DOMAIN:'int-mx04.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'dgilbert@redhat.com' RCPT:'' Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH v5 12/29] postcopy+vhost-user: Split set_mem_table for postcopy X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: aarcange@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Type: text/plain; charset="utf-8" From: "Dr. David Alan Gilbert" Split the set_mem_table routines in both qemu and libvhost-user because the postcopy versions are going to be quite different once changes in the later patches are added. However, this patch doesn't produce any functional change, just the split. Signed-off-by: Dr. David Alan Gilbert Reviewed-by: Marc-Andr=C3=A9 Lureau --- contrib/libvhost-user/libvhost-user.c | 53 ++++++++++++++++++++++++ hw/virtio/vhost-user.c | 77 +++++++++++++++++++++++++++++++= +++- 2 files changed, 128 insertions(+), 2 deletions(-) diff --git a/contrib/libvhost-user/libvhost-user.c b/contrib/libvhost-user/= libvhost-user.c index e53b1953df..b2de8ed0a8 100644 --- a/contrib/libvhost-user/libvhost-user.c +++ b/contrib/libvhost-user/libvhost-user.c @@ -448,6 +448,55 @@ vu_reset_device_exec(VuDev *dev, VhostUserMsg *vmsg) return false; } =20 +static bool +vu_set_mem_table_exec_postcopy(VuDev *dev, VhostUserMsg *vmsg) +{ + int i; + VhostUserMemory *memory =3D &vmsg->payload.memory; + dev->nregions =3D memory->nregions; + /* TODO: Postcopy specific code */ + DPRINT("Nregions: %d\n", memory->nregions); + for (i =3D 0; i < dev->nregions; i++) { + void *mmap_addr; + VhostUserMemoryRegion *msg_region =3D &memory->regions[i]; + VuDevRegion *dev_region =3D &dev->regions[i]; + + DPRINT("Region %d\n", i); + DPRINT(" guest_phys_addr: 0x%016"PRIx64"\n", + msg_region->guest_phys_addr); + DPRINT(" memory_size: 0x%016"PRIx64"\n", + msg_region->memory_size); + DPRINT(" userspace_addr 0x%016"PRIx64"\n", + msg_region->userspace_addr); + DPRINT(" mmap_offset 0x%016"PRIx64"\n", + msg_region->mmap_offset); + + dev_region->gpa =3D msg_region->guest_phys_addr; + dev_region->size =3D msg_region->memory_size; + dev_region->qva =3D msg_region->userspace_addr; + dev_region->mmap_offset =3D msg_region->mmap_offset; + + /* We don't use offset argument of mmap() since the + * mapped address has to be page aligned, and we use huge + * pages. */ + mmap_addr =3D mmap(0, dev_region->size + dev_region->mmap_offset, + PROT_READ | PROT_WRITE, MAP_SHARED, + vmsg->fds[i], 0); + + if (mmap_addr =3D=3D MAP_FAILED) { + vu_panic(dev, "region mmap error: %s", strerror(errno)); + } else { + dev_region->mmap_addr =3D (uint64_t)(uintptr_t)mmap_addr; + DPRINT(" mmap_addr: 0x%016"PRIx64"\n", + dev_region->mmap_addr); + } + + close(vmsg->fds[i]); + } + + return false; +} + static bool vu_set_mem_table_exec(VuDev *dev, VhostUserMsg *vmsg) { @@ -464,6 +513,10 @@ vu_set_mem_table_exec(VuDev *dev, VhostUserMsg *vmsg) } dev->nregions =3D memory->nregions; =20 + if (dev->postcopy_listening) { + return vu_set_mem_table_exec_postcopy(dev, vmsg); + } + DPRINT("Nregions: %d\n", memory->nregions); for (i =3D 0; i < dev->nregions; i++) { void *mmap_addr; diff --git a/hw/virtio/vhost-user.c b/hw/virtio/vhost-user.c index c3ab2994fb..b6757ebae3 100644 --- a/hw/virtio/vhost-user.c +++ b/hw/virtio/vhost-user.c @@ -340,15 +340,86 @@ static int vhost_user_set_log_base(struct vhost_dev *= dev, uint64_t base, return 0; } =20 +static int vhost_user_set_mem_table_postcopy(struct vhost_dev *dev, + struct vhost_memory *mem) +{ + int fds[VHOST_MEMORY_MAX_NREGIONS]; + int i, fd; + size_t fd_num =3D 0; + bool reply_supported =3D virtio_has_feature(dev->protocol_features, + VHOST_USER_PROTOCOL_F_REPLY_= ACK); + /* TODO: Add actual postcopy differences */ + VhostUserMsg msg =3D { + .hdr.request =3D VHOST_USER_SET_MEM_TABLE, + .hdr.flags =3D VHOST_USER_VERSION, + }; + + if (reply_supported) { + msg.hdr.flags |=3D VHOST_USER_NEED_REPLY_MASK; + } + + for (i =3D 0; i < dev->mem->nregions; ++i) { + struct vhost_memory_region *reg =3D dev->mem->regions + i; + ram_addr_t offset; + MemoryRegion *mr; + + assert((uintptr_t)reg->userspace_addr =3D=3D reg->userspace_addr); + mr =3D memory_region_from_host((void *)(uintptr_t)reg->userspace_a= ddr, + &offset); + fd =3D memory_region_get_fd(mr); + if (fd > 0) { + msg.payload.memory.regions[fd_num].userspace_addr =3D + reg->userspace_addr; + msg.payload.memory.regions[fd_num].memory_size =3D reg->memor= y_size; + msg.payload.memory.regions[fd_num].guest_phys_addr =3D + reg->guest_phys_addr; + msg.payload.memory.regions[fd_num].mmap_offset =3D offset; + assert(fd_num < VHOST_MEMORY_MAX_NREGIONS); + fds[fd_num++] =3D fd; + } + } + + msg.payload.memory.nregions =3D fd_num; + + if (!fd_num) { + error_report("Failed initializing vhost-user memory map, " + "consider using -object memory-backend-file share=3Do= n"); + return -1; + } + + msg.hdr.size =3D sizeof(msg.payload.memory.nregions); + msg.hdr.size +=3D sizeof(msg.payload.memory.padding); + msg.hdr.size +=3D fd_num * sizeof(VhostUserMemoryRegion); + + if (vhost_user_write(dev, &msg, fds, fd_num) < 0) { + return -1; + } + + if (reply_supported) { + return process_message_reply(dev, &msg); + } + + return 0; +} + static int vhost_user_set_mem_table(struct vhost_dev *dev, struct vhost_memory *mem) { + struct vhost_user *u =3D dev->opaque; int fds[VHOST_MEMORY_MAX_NREGIONS]; int i, fd; size_t fd_num =3D 0; + bool do_postcopy =3D u->postcopy_listen && u->postcopy_fd.handler; bool reply_supported =3D virtio_has_feature(dev->protocol_features, VHOST_USER_PROTOCOL_F_REPLY_= ACK); =20 + if (do_postcopy) { + /* Postcopy has enough differences that it's best done in it's own + * version + */ + return vhost_user_set_mem_table_postcopy(dev, mem); + } + VhostUserMsg msg =3D { .hdr.request =3D VHOST_USER_SET_MEM_TABLE, .hdr.flags =3D VHOST_USER_VERSION, @@ -372,9 +443,11 @@ static int vhost_user_set_mem_table(struct vhost_dev *= dev, error_report("Failed preparing vhost-user memory table msg= "); return -1; } - msg.payload.memory.regions[fd_num].userspace_addr =3D reg->use= rspace_addr; + msg.payload.memory.regions[fd_num].userspace_addr =3D + reg->userspace_addr; msg.payload.memory.regions[fd_num].memory_size =3D reg->memor= y_size; - msg.payload.memory.regions[fd_num].guest_phys_addr =3D reg->gu= est_phys_addr; + msg.payload.memory.regions[fd_num].guest_phys_addr =3D + reg->guest_phys_addr; msg.payload.memory.regions[fd_num].mmap_offset =3D offset; fds[fd_num++] =3D fd; } --=20 2.14.3 From nobody Sun May 5 21:21:29 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (208.118.235.17 [208.118.235.17]) by mx.zohomail.com with SMTPS id 1520875760718379.51313718064296; Mon, 12 Mar 2018 10:29:20 -0700 (PDT) Received: from localhost ([::1]:33409 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1evRGB-0006MW-Qt for importer@patchew.org; Mon, 12 Mar 2018 13:29:19 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:53067) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1evR8y-0008CX-8w for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:21:53 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1evR8x-0007cy-79 for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:21:52 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:36132 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1evR8x-0007cc-3B for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:21:51 -0400 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id B896E8424C for ; Mon, 12 Mar 2018 17:21:50 +0000 (UTC) Received: from dgilbert-t530.redhat.com (ovpn-117-169.ams2.redhat.com [10.36.117.169]) by smtp.corp.redhat.com (Postfix) with ESMTP id 809A7202322C; Mon, 12 Mar 2018 17:21:49 +0000 (UTC) From: "Dr. David Alan Gilbert (git)" To: qemu-devel@nongnu.org, mst@redhat.com, maxime.coquelin@redhat.com, marcandre.lureau@redhat.com, peterx@redhat.com, quintela@redhat.com Date: Mon, 12 Mar 2018 17:21:08 +0000 Message-Id: <20180312172124.56461-14-dgilbert@redhat.com> In-Reply-To: <20180312172124.56461-1-dgilbert@redhat.com> References: <20180312172124.56461-1-dgilbert@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.78 on 10.11.54.4 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.2]); Mon, 12 Mar 2018 17:21:50 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.2]); Mon, 12 Mar 2018 17:21:50 +0000 (UTC) for IP:'10.11.54.4' DOMAIN:'int-mx04.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'dgilbert@redhat.com' RCPT:'' Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH v5 13/29] migration/ram: ramblock_recv_bitmap_test_byte_offset X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: aarcange@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Type: text/plain; charset="utf-8" From: "Dr. David Alan Gilbert" Utility for testing the map when you already know the offset in the RAMBlock. Signed-off-by: Dr. David Alan Gilbert Reviewed-by: Peter Xu Reviewed-by: Marc-Andr=C3=A9 Lureau --- migration/ram.c | 5 +++++ migration/ram.h | 1 + 2 files changed, 6 insertions(+) diff --git a/migration/ram.c b/migration/ram.c index 7266351fd0..6ce7770b8c 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -169,6 +169,11 @@ int ramblock_recv_bitmap_test(RAMBlock *rb, void *host= _addr) rb->receivedmap); } =20 +bool ramblock_recv_bitmap_test_byte_offset(RAMBlock *rb, uint64_t byte_off= set) +{ + return test_bit(byte_offset >> TARGET_PAGE_BITS, rb->receivedmap); +} + void ramblock_recv_bitmap_set(RAMBlock *rb, void *host_addr) { set_bit_atomic(ramblock_recv_bitmap_offset(host_addr, rb), rb->receive= dmap); diff --git a/migration/ram.h b/migration/ram.h index 53f0021c51..5030be110a 100644 --- a/migration/ram.h +++ b/migration/ram.h @@ -60,6 +60,7 @@ int ram_postcopy_incoming_init(MigrationIncomingState *mi= s); void ram_handle_compressed(void *host, uint8_t ch, uint64_t size); =20 int ramblock_recv_bitmap_test(RAMBlock *rb, void *host_addr); +bool ramblock_recv_bitmap_test_byte_offset(RAMBlock *rb, uint64_t byte_off= set); void ramblock_recv_bitmap_set(RAMBlock *rb, void *host_addr); void ramblock_recv_bitmap_set_range(RAMBlock *rb, void *host_addr, size_t = nr); =20 --=20 2.14.3 From nobody Sun May 5 21:21:29 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1520876115396635.9210626826108; Mon, 12 Mar 2018 10:35:15 -0700 (PDT) Received: from localhost ([::1]:33457 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1evRLp-0002rD-62 for importer@patchew.org; Mon, 12 Mar 2018 13:35:09 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:53090) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1evR92-0008HW-L1 for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:21:57 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1evR8y-0007eD-Mn for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:21:56 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:56692 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1evR8y-0007dn-IH for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:21:52 -0400 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 37AAF4040856 for ; Mon, 12 Mar 2018 17:21:52 +0000 (UTC) Received: from dgilbert-t530.redhat.com (ovpn-117-169.ams2.redhat.com [10.36.117.169]) by smtp.corp.redhat.com (Postfix) with ESMTP id F3643202322B; Mon, 12 Mar 2018 17:21:50 +0000 (UTC) From: "Dr. David Alan Gilbert (git)" To: qemu-devel@nongnu.org, mst@redhat.com, maxime.coquelin@redhat.com, marcandre.lureau@redhat.com, peterx@redhat.com, quintela@redhat.com Date: Mon, 12 Mar 2018 17:21:09 +0000 Message-Id: <20180312172124.56461-15-dgilbert@redhat.com> In-Reply-To: <20180312172124.56461-1-dgilbert@redhat.com> References: <20180312172124.56461-1-dgilbert@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.78 on 10.11.54.4 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.5]); Mon, 12 Mar 2018 17:21:52 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.5]); Mon, 12 Mar 2018 17:21:52 +0000 (UTC) for IP:'10.11.54.4' DOMAIN:'int-mx04.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'dgilbert@redhat.com' RCPT:'' Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH v5 14/29] libvhost-user+postcopy: Register new regions with the ufd X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: aarcange@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Type: text/plain; charset="utf-8" From: "Dr. David Alan Gilbert" When new regions are sent to the client using SET_MEM_TABLE, register them with the userfaultfd. Signed-off-by: Dr. David Alan Gilbert Reviewed-by: Marc-Andr=C3=A9 Lureau --- contrib/libvhost-user/libvhost-user.c | 34 +++++++++++++++++++++++++++++++= +++ 1 file changed, 34 insertions(+) diff --git a/contrib/libvhost-user/libvhost-user.c b/contrib/libvhost-user/= libvhost-user.c index b2de8ed0a8..7c8cd5878e 100644 --- a/contrib/libvhost-user/libvhost-user.c +++ b/contrib/libvhost-user/libvhost-user.c @@ -494,6 +494,40 @@ vu_set_mem_table_exec_postcopy(VuDev *dev, VhostUserMs= g *vmsg) close(vmsg->fds[i]); } =20 + /* TODO: Get address back to QEMU */ + for (i =3D 0; i < dev->nregions; i++) { + VuDevRegion *dev_region =3D &dev->regions[i]; +#ifdef UFFDIO_REGISTER + /* We should already have an open ufd. Mark each memory + * range as ufd. + * Note: Do we need any madvises? Well it's not been accessed + * yet, still probably need no THP to be safe, discard to be safe? + */ + struct uffdio_register reg_struct; + reg_struct.range.start =3D (uintptr_t)dev_region->mmap_addr; + reg_struct.range.len =3D dev_region->size + dev_region->mmap_offse= t; + reg_struct.mode =3D UFFDIO_REGISTER_MODE_MISSING; + + if (ioctl(dev->postcopy_ufd, UFFDIO_REGISTER, ®_struct)) { + vu_panic(dev, "%s: Failed to userfault region %d " + "@%p + size:%zx offset: %zx: (ufd=3D%d)%s\n", + __func__, i, + dev_region->mmap_addr, + dev_region->size, dev_region->mmap_offset, + dev->postcopy_ufd, strerror(errno)); + return false; + } + if (!(reg_struct.ioctls & ((__u64)1 << _UFFDIO_COPY))) { + vu_panic(dev, "%s Region (%d) doesn't support COPY", + __func__, i); + return false; + } + DPRINT("%s: region %d: Registered userfault for %llx + %llx\n", + __func__, i, reg_struct.range.start, reg_struct.range.len); + /* TODO: Stash 'zero' support flags somewhere */ +#endif + } + return false; } =20 --=20 2.14.3 From nobody Sun May 5 21:21:29 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1520876301971687.282292532188; Mon, 12 Mar 2018 10:38:21 -0700 (PDT) Received: from localhost ([::1]:33477 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1evROq-0005cF-Kp for importer@patchew.org; Mon, 12 Mar 2018 13:38:16 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:53102) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1evR93-0008ID-CG for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:21:59 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1evR90-0007fG-76 for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:21:57 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:57614 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1evR90-0007eu-1D for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:21:54 -0400 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id AA843401DE7F for ; Mon, 12 Mar 2018 17:21:53 +0000 (UTC) Received: from dgilbert-t530.redhat.com (ovpn-117-169.ams2.redhat.com [10.36.117.169]) by smtp.corp.redhat.com (Postfix) with ESMTP id 72E642026E03; Mon, 12 Mar 2018 17:21:52 +0000 (UTC) From: "Dr. David Alan Gilbert (git)" To: qemu-devel@nongnu.org, mst@redhat.com, maxime.coquelin@redhat.com, marcandre.lureau@redhat.com, peterx@redhat.com, quintela@redhat.com Date: Mon, 12 Mar 2018 17:21:10 +0000 Message-Id: <20180312172124.56461-16-dgilbert@redhat.com> In-Reply-To: <20180312172124.56461-1-dgilbert@redhat.com> References: <20180312172124.56461-1-dgilbert@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.78 on 10.11.54.4 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.6]); Mon, 12 Mar 2018 17:21:53 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.6]); Mon, 12 Mar 2018 17:21:53 +0000 (UTC) for IP:'10.11.54.4' DOMAIN:'int-mx04.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'dgilbert@redhat.com' RCPT:'' Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH v5 15/29] vhost+postcopy: Send address back to qemu X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: aarcange@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Type: text/plain; charset="utf-8" From: "Dr. David Alan Gilbert" We need a better way, but at the moment we need the address of the mappings sent back to qemu so it can interpret the messages on the userfaultfd it reads. This is done as a 3 stage set: QEMU -> client set_mem_table mmap stuff, get addresses client -> qemu here are the addresses qemu -> client OK - now you can use them That ensures that qemu has registered the new addresses in it's userfault code before the client starts accessing them. Note: We don't ask for the default 'ack' reply since we've got our own. Signed-off-by: Dr. David Alan Gilbert Reviewed-by: Marc-Andr=C3=A9 Lureau --- contrib/libvhost-user/libvhost-user.c | 24 ++++++++++++- docs/interop/vhost-user.txt | 9 +++++ hw/virtio/trace-events | 1 + hw/virtio/vhost-user.c | 67 +++++++++++++++++++++++++++++++= ++-- 4 files changed, 98 insertions(+), 3 deletions(-) diff --git a/contrib/libvhost-user/libvhost-user.c b/contrib/libvhost-user/= libvhost-user.c index 7c8cd5878e..6314549b65 100644 --- a/contrib/libvhost-user/libvhost-user.c +++ b/contrib/libvhost-user/libvhost-user.c @@ -491,10 +491,32 @@ vu_set_mem_table_exec_postcopy(VuDev *dev, VhostUserM= sg *vmsg) dev_region->mmap_addr); } =20 + /* Return the address to QEMU so that it can translate the ufd + * fault addresses back. + */ + msg_region->userspace_addr =3D (uintptr_t)(mmap_addr + + dev_region->mmap_offset); close(vmsg->fds[i]); } =20 - /* TODO: Get address back to QEMU */ + /* Send the message back to qemu with the addresses filled in */ + vmsg->fd_num =3D 0; + if (!vu_message_write(dev, dev->sock, vmsg)) { + vu_panic(dev, "failed to respond to set-mem-table for postcopy"); + return false; + } + + /* Wait for QEMU to confirm that it's registered the handler for the + * faults. + */ + if (!vu_message_read(dev, dev->sock, vmsg) || + vmsg->size !=3D sizeof(vmsg->payload.u64) || + vmsg->payload.u64 !=3D 0) { + vu_panic(dev, "failed to receive valid ack for postcopy set-mem-ta= ble"); + return false; + } + + /* OK, now we can go and register the memory and generate faults */ for (i =3D 0; i < dev->nregions; i++) { VuDevRegion *dev_region =3D &dev->regions[i]; #ifdef UFFDIO_REGISTER diff --git a/docs/interop/vhost-user.txt b/docs/interop/vhost-user.txt index 0d24203d31..e295ef12ca 100644 --- a/docs/interop/vhost-user.txt +++ b/docs/interop/vhost-user.txt @@ -455,12 +455,21 @@ Master message types Id: 5 Equivalent ioctl: VHOST_SET_MEM_TABLE Master payload: memory regions description + Slave payload: (postcopy only) memory regions description =20 Sets the memory map regions on the slave so it can translate the vri= ng addresses. In the ancillary data there is an array of file descripto= rs for each memory mapped region. The size and ordering of the fds matc= hes the number and ordering of memory regions. =20 + When VHOST_USER_POSTCOPY_LISTEN has been received, SET_MEM_TABLE rep= lies with + the bases of the memory mapped regions to the master. The slave must + have mmap'd the regions but not yet accessed them and should not yet= generate + a userfault event. Note NEED_REPLY_MASK is not set in this case. + QEMU will then reply back to the list of mappings with an empty + VHOST_USER_SET_MEM_TABLE as an acknowledgment; only upon reception o= f this + message may the guest start accessing the memory and generating faul= ts. + * VHOST_USER_SET_LOG_BASE =20 Id: 6 diff --git a/hw/virtio/trace-events b/hw/virtio/trace-events index 06ec03d6e7..05d18ada77 100644 --- a/hw/virtio/trace-events +++ b/hw/virtio/trace-events @@ -8,6 +8,7 @@ vhost_section(const char *name, int r) "%s:%d" =20 # hw/virtio/vhost-user.c vhost_user_postcopy_listen(void) "" +vhost_user_set_mem_table_postcopy(uint64_t client_addr, uint64_t qhva, int= reply_i, int region_i) "client:0x%"PRIx64" for hva: 0x%"PRIx64" reply %d r= egion %d" =20 # hw/virtio/virtio.c virtqueue_alloc_element(void *elem, size_t sz, unsigned in_num, unsigned o= ut_num) "elem %p size %zd in_num %u out_num %u" diff --git a/hw/virtio/vhost-user.c b/hw/virtio/vhost-user.c index b6757ebae3..1603d70bea 100644 --- a/hw/virtio/vhost-user.c +++ b/hw/virtio/vhost-user.c @@ -174,6 +174,7 @@ struct vhost_user { int slave_fd; NotifierWithReturn postcopy_notifier; struct PostCopyFD postcopy_fd; + uint64_t postcopy_client_bases[VHOST_MEMORY_MAX_NREGIONS]; /* True once we've entered postcopy_listen */ bool postcopy_listen; }; @@ -343,12 +344,15 @@ static int vhost_user_set_log_base(struct vhost_dev *= dev, uint64_t base, static int vhost_user_set_mem_table_postcopy(struct vhost_dev *dev, struct vhost_memory *mem) { + struct vhost_user *u =3D dev->opaque; int fds[VHOST_MEMORY_MAX_NREGIONS]; int i, fd; size_t fd_num =3D 0; bool reply_supported =3D virtio_has_feature(dev->protocol_features, VHOST_USER_PROTOCOL_F_REPLY_= ACK); - /* TODO: Add actual postcopy differences */ + VhostUserMsg msg_reply; + int region_i, msg_i; + VhostUserMsg msg =3D { .hdr.request =3D VHOST_USER_SET_MEM_TABLE, .hdr.flags =3D VHOST_USER_VERSION, @@ -395,6 +399,64 @@ static int vhost_user_set_mem_table_postcopy(struct vh= ost_dev *dev, return -1; } =20 + if (vhost_user_read(dev, &msg_reply) < 0) { + return -1; + } + + if (msg_reply.hdr.request !=3D VHOST_USER_SET_MEM_TABLE) { + error_report("%s: Received unexpected msg type." + "Expected %d received %d", __func__, + VHOST_USER_SET_MEM_TABLE, msg_reply.hdr.request); + return -1; + } + /* We're using the same structure, just reusing one of the + * fields, so it should be the same size. + */ + if (msg_reply.hdr.size !=3D msg.hdr.size) { + error_report("%s: Unexpected size for postcopy reply " + "%d vs %d", __func__, msg_reply.hdr.size, msg.hdr.siz= e); + return -1; + } + + memset(u->postcopy_client_bases, 0, + sizeof(uint64_t) * VHOST_MEMORY_MAX_NREGIONS); + + /* They're in the same order as the regions that were sent + * but some of the regions were skipped (above) if they + * didn't have fd's + */ + for (msg_i =3D 0, region_i =3D 0; + region_i < dev->mem->nregions; + region_i++) { + if (msg_i < fd_num && + msg_reply.payload.memory.regions[msg_i].guest_phys_addr =3D=3D + dev->mem->regions[region_i].guest_phys_addr) { + u->postcopy_client_bases[region_i] =3D + msg_reply.payload.memory.regions[msg_i].userspace_addr; + trace_vhost_user_set_mem_table_postcopy( + msg_reply.payload.memory.regions[msg_i].userspace_addr, + msg.payload.memory.regions[msg_i].userspace_addr, + msg_i, region_i); + msg_i++; + } + } + if (msg_i !=3D fd_num) { + error_report("%s: postcopy reply not fully consumed " + "%d vs %zd", + __func__, msg_i, fd_num); + return -1; + } + /* Now we've registered this with the postcopy code, we ack to the cli= ent, + * because now we're in the position to be able to deal with any faults + * it generates. + */ + /* TODO: Use this for failure cases as well with a bad value */ + msg.hdr.size =3D sizeof(msg.payload.u64); + msg.payload.u64 =3D 0; /* OK */ + if (vhost_user_write(dev, &msg, NULL, 0) < 0) { + return -1; + } + if (reply_supported) { return process_message_reply(dev, &msg); } @@ -411,7 +473,8 @@ static int vhost_user_set_mem_table(struct vhost_dev *d= ev, size_t fd_num =3D 0; bool do_postcopy =3D u->postcopy_listen && u->postcopy_fd.handler; bool reply_supported =3D virtio_has_feature(dev->protocol_features, - VHOST_USER_PROTOCOL_F_REPLY_= ACK); + VHOST_USER_PROTOCOL_F_REPLY_ACK)= && + !do_postcopy; =20 if (do_postcopy) { /* Postcopy has enough differences that it's best done in it's own --=20 2.14.3 From nobody Sun May 5 21:21:29 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1520876180269498.720307021092; Mon, 12 Mar 2018 10:36:20 -0700 (PDT) Received: from localhost ([::1]:33464 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1evRMx-0003tt-Be for importer@patchew.org; Mon, 12 Mar 2018 13:36:19 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:53103) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1evR93-0008IE-CX for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:21:58 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1evR91-0007hO-M7 for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:21:57 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:36138 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1evR91-0007ge-Gj for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:21:55 -0400 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 2AE758424C for ; Mon, 12 Mar 2018 17:21:55 +0000 (UTC) Received: from dgilbert-t530.redhat.com (ovpn-117-169.ams2.redhat.com [10.36.117.169]) by smtp.corp.redhat.com (Postfix) with ESMTP id E5BE3202322A; Mon, 12 Mar 2018 17:21:53 +0000 (UTC) From: "Dr. David Alan Gilbert (git)" To: qemu-devel@nongnu.org, mst@redhat.com, maxime.coquelin@redhat.com, marcandre.lureau@redhat.com, peterx@redhat.com, quintela@redhat.com Date: Mon, 12 Mar 2018 17:21:11 +0000 Message-Id: <20180312172124.56461-17-dgilbert@redhat.com> In-Reply-To: <20180312172124.56461-1-dgilbert@redhat.com> References: <20180312172124.56461-1-dgilbert@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.78 on 10.11.54.4 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.2]); Mon, 12 Mar 2018 17:21:55 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.2]); Mon, 12 Mar 2018 17:21:55 +0000 (UTC) for IP:'10.11.54.4' DOMAIN:'int-mx04.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'dgilbert@redhat.com' RCPT:'' Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH v5 16/29] vhost+postcopy: Stash RAMBlock and offset X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: aarcange@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Type: text/plain; charset="utf-8" From: "Dr. David Alan Gilbert" Stash the RAMBlock and offset for later use looking up addresses. Signed-off-by: Dr. David Alan Gilbert Reviewed-by: Marc-Andr=C3=A9 Lureau --- hw/virtio/trace-events | 1 + hw/virtio/vhost-user.c | 34 ++++++++++++++++++++++++++++++++++ 2 files changed, 35 insertions(+) diff --git a/hw/virtio/trace-events b/hw/virtio/trace-events index 05d18ada77..d7e9e1084b 100644 --- a/hw/virtio/trace-events +++ b/hw/virtio/trace-events @@ -9,6 +9,7 @@ vhost_section(const char *name, int r) "%s:%d" # hw/virtio/vhost-user.c vhost_user_postcopy_listen(void) "" vhost_user_set_mem_table_postcopy(uint64_t client_addr, uint64_t qhva, int= reply_i, int region_i) "client:0x%"PRIx64" for hva: 0x%"PRIx64" reply %d r= egion %d" +vhost_user_set_mem_table_withfd(int index, const char *name, uint64_t memo= ry_size, uint64_t guest_phys_addr, uint64_t userspace_addr, uint64_t offset= ) "%d:%s: size:0x%"PRIx64" GPA:0x%"PRIx64" QVA/userspace:0x%"PRIx64" RB off= set:0x%"PRIx64 =20 # hw/virtio/virtio.c virtqueue_alloc_element(void *elem, size_t sz, unsigned in_num, unsigned o= ut_num) "elem %p size %zd in_num %u out_num %u" diff --git a/hw/virtio/vhost-user.c b/hw/virtio/vhost-user.c index 1603d70bea..b47de62c47 100644 --- a/hw/virtio/vhost-user.c +++ b/hw/virtio/vhost-user.c @@ -175,6 +175,15 @@ struct vhost_user { NotifierWithReturn postcopy_notifier; struct PostCopyFD postcopy_fd; uint64_t postcopy_client_bases[VHOST_MEMORY_MAX_NREGIONS]; + /* Length of the region_rb and region_rb_offset arrays */ + size_t region_rb_len; + /* RAMBlock associated with a given region */ + RAMBlock **region_rb; + /* The offset from the start of the RAMBlock to the start of the + * vhost region. + */ + ram_addr_t *region_rb_offset; + /* True once we've entered postcopy_listen */ bool postcopy_listen; }; @@ -362,6 +371,17 @@ static int vhost_user_set_mem_table_postcopy(struct vh= ost_dev *dev, msg.hdr.flags |=3D VHOST_USER_NEED_REPLY_MASK; } =20 + if (u->region_rb_len < dev->mem->nregions) { + u->region_rb =3D g_renew(RAMBlock*, u->region_rb, dev->mem->nregio= ns); + u->region_rb_offset =3D g_renew(ram_addr_t, u->region_rb_offset, + dev->mem->nregions); + memset(&(u->region_rb[u->region_rb_len]), '\0', + sizeof(RAMBlock *) * (dev->mem->nregions - u->region_rb_len= )); + memset(&(u->region_rb_offset[u->region_rb_len]), '\0', + sizeof(ram_addr_t) * (dev->mem->nregions - u->region_rb_len= )); + u->region_rb_len =3D dev->mem->nregions; + } + for (i =3D 0; i < dev->mem->nregions; ++i) { struct vhost_memory_region *reg =3D dev->mem->regions + i; ram_addr_t offset; @@ -372,6 +392,12 @@ static int vhost_user_set_mem_table_postcopy(struct vh= ost_dev *dev, &offset); fd =3D memory_region_get_fd(mr); if (fd > 0) { + trace_vhost_user_set_mem_table_withfd(fd_num, mr->name, + reg->memory_size, + reg->guest_phys_addr, + reg->userspace_addr, off= set); + u->region_rb_offset[i] =3D offset; + u->region_rb[i] =3D mr->ram_block; msg.payload.memory.regions[fd_num].userspace_addr =3D reg->userspace_addr; msg.payload.memory.regions[fd_num].memory_size =3D reg->memor= y_size; @@ -380,6 +406,9 @@ static int vhost_user_set_mem_table_postcopy(struct vho= st_dev *dev, msg.payload.memory.regions[fd_num].mmap_offset =3D offset; assert(fd_num < VHOST_MEMORY_MAX_NREGIONS); fds[fd_num++] =3D fd; + } else { + u->region_rb_offset[i] =3D 0; + u->region_rb[i] =3D NULL; } } =20 @@ -1148,6 +1177,11 @@ static int vhost_user_cleanup(struct vhost_dev *dev) close(u->slave_fd); u->slave_fd =3D -1; } + g_free(u->region_rb); + u->region_rb =3D NULL; + g_free(u->region_rb_offset); + u->region_rb_offset =3D NULL; + u->region_rb_len =3D 0; g_free(u); dev->opaque =3D 0; =20 --=20 2.14.3 From nobody Sun May 5 21:21:29 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1520875931438908.7386957471175; Mon, 12 Mar 2018 10:32:11 -0700 (PDT) Received: from localhost ([::1]:33426 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1evRIw-0000Ol-At for importer@patchew.org; Mon, 12 Mar 2018 13:32:10 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:53115) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1evR94-0008JF-CB for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:21:59 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1evR93-0007ip-6P for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:21:58 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:36142 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1evR93-0007iV-0u for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:21:57 -0400 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id A06848D6D7 for ; Mon, 12 Mar 2018 17:21:56 +0000 (UTC) Received: from dgilbert-t530.redhat.com (ovpn-117-169.ams2.redhat.com [10.36.117.169]) by smtp.corp.redhat.com (Postfix) with ESMTP id 65C14202322A; Mon, 12 Mar 2018 17:21:55 +0000 (UTC) From: "Dr. David Alan Gilbert (git)" To: qemu-devel@nongnu.org, mst@redhat.com, maxime.coquelin@redhat.com, marcandre.lureau@redhat.com, peterx@redhat.com, quintela@redhat.com Date: Mon, 12 Mar 2018 17:21:12 +0000 Message-Id: <20180312172124.56461-18-dgilbert@redhat.com> In-Reply-To: <20180312172124.56461-1-dgilbert@redhat.com> References: <20180312172124.56461-1-dgilbert@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.78 on 10.11.54.4 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.2]); Mon, 12 Mar 2018 17:21:56 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.2]); Mon, 12 Mar 2018 17:21:56 +0000 (UTC) for IP:'10.11.54.4' DOMAIN:'int-mx04.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'dgilbert@redhat.com' RCPT:'' Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH v5 17/29] vhost+postcopy: Helper to send requests to source for shared pages X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: aarcange@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Type: text/plain; charset="utf-8" From: "Dr. David Alan Gilbert" Provide a helper to be used by shared waker functions to request shared pages from the source. The last_rb pointer is moved into the incoming state since this helper can update it as well as the main fault thread function. Signed-off-by: Dr. David Alan Gilbert Reviewed-by: Marc-Andr=C3=A9 Lureau --- migration/migration.h | 2 ++ migration/postcopy-ram.c | 32 +++++++++++++++++++++++++++++--- migration/postcopy-ram.h | 3 +++ migration/trace-events | 2 ++ 4 files changed, 36 insertions(+), 3 deletions(-) diff --git a/migration/migration.h b/migration/migration.h index d02a759331..83dc36b57a 100644 --- a/migration/migration.h +++ b/migration/migration.h @@ -49,6 +49,8 @@ struct MigrationIncomingState { int userfault_event_fd; QEMUFile *to_src_file; QemuMutex rp_mutex; /* We send replies from multiple threads */ + /* RAMBlock of last request sent to source */ + RAMBlock *last_rb; void *postcopy_tmp_page; void *postcopy_tmp_zero_page; /* PostCopyFD's for external userfaultfds & handlers of shared memory = */ diff --git a/migration/postcopy-ram.c b/migration/postcopy-ram.c index 4ab1b7d36d..83780576ca 100644 --- a/migration/postcopy-ram.c +++ b/migration/postcopy-ram.c @@ -525,6 +525,32 @@ static int ram_block_enable_notify(const char *block_n= ame, void *host_addr, return 0; } =20 +/* + * Callback from shared fault handlers to ask for a page, + * the page must be specified by a RAMBlock and an offset in that rb + * Note: Only for use by shared fault handlers (in fault thread) + */ +int postcopy_request_shared_page(struct PostCopyFD *pcfd, RAMBlock *rb, + uint64_t client_addr, uint64_t rb_offset) +{ + size_t pagesize =3D qemu_ram_pagesize(rb); + uint64_t aligned_rbo =3D rb_offset & ~(pagesize - 1); + MigrationIncomingState *mis =3D migration_incoming_get_current(); + + trace_postcopy_request_shared_page(pcfd->idstr, qemu_ram_get_idstr(rb), + rb_offset); + /* TODO: Check bitmap to see if we already have the page */ + if (rb !=3D mis->last_rb) { + mis->last_rb =3D rb; + migrate_send_rp_req_pages(mis, qemu_ram_get_idstr(rb), + aligned_rbo, pagesize); + } else { + /* Save some space */ + migrate_send_rp_req_pages(mis, NULL, aligned_rbo, pagesize); + } + return 0; +} + /* * Handle faults detected by the USERFAULT markings */ @@ -535,9 +561,9 @@ static void *postcopy_ram_fault_thread(void *opaque) int ret; size_t index; RAMBlock *rb =3D NULL; - RAMBlock *last_rb =3D NULL; /* last RAMBlock we sent part of */ =20 trace_postcopy_ram_fault_thread_entry(); + mis->last_rb =3D NULL; /* last RAMBlock we sent part of */ qemu_sem_post(&mis->fault_thread_sem); =20 struct pollfd *pfd; @@ -636,8 +662,8 @@ static void *postcopy_ram_fault_thread(void *opaque) * Send the request to the source - we want to request one * of our host page sizes (which is >=3D TPS) */ - if (rb !=3D last_rb) { - last_rb =3D rb; + if (rb !=3D mis->last_rb) { + mis->last_rb =3D rb; migrate_send_rp_req_pages(mis, qemu_ram_get_idstr(rb), rb_offset, qemu_ram_pagesize(rb)); } else { diff --git a/migration/postcopy-ram.h b/migration/postcopy-ram.h index c8ced3470b..d7afab07d7 100644 --- a/migration/postcopy-ram.h +++ b/migration/postcopy-ram.h @@ -164,5 +164,8 @@ struct PostCopyFD { */ void postcopy_register_shared_ufd(struct PostCopyFD *pcfd); void postcopy_unregister_shared_ufd(struct PostCopyFD *pcfd); +/* Callback from shared fault handlers to ask for a page */ +int postcopy_request_shared_page(struct PostCopyFD *pcfd, RAMBlock *rb, + uint64_t client_addr, uint64_t offset); =20 #endif diff --git a/migration/trace-events b/migration/trace-events index 1e617ad7a6..7c910b5479 100644 --- a/migration/trace-events +++ b/migration/trace-events @@ -198,6 +198,8 @@ postcopy_ram_incoming_cleanup_closeuf(void) "" postcopy_ram_incoming_cleanup_entry(void) "" postcopy_ram_incoming_cleanup_exit(void) "" postcopy_ram_incoming_cleanup_join(void) "" +postcopy_request_shared_page(const char *sharer, const char *rb, uint64_t = rb_offset) "for %s in %s offset 0x%"PRIx64 + save_xbzrle_page_skipping(void) "" save_xbzrle_page_overflow(void) "" ram_save_iterate_big_wait(uint64_t milliconds, int iterations) "big wait: = %" PRIu64 " milliseconds, %d iterations" --=20 2.14.3 From nobody Sun May 5 21:21:29 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (208.118.235.17 [208.118.235.17]) by mx.zohomail.com with SMTPS id 1520876122591989.0306152649395; Mon, 12 Mar 2018 10:35:22 -0700 (PDT) Received: from localhost ([::1]:33458 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1evRLw-0002wC-O5 for importer@patchew.org; Mon, 12 Mar 2018 13:35:16 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:53130) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1evR95-0008KV-Lk for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:22:00 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1evR94-0007kC-Jx for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:21:59 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:57624 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1evR94-0007jl-Eb for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:21:58 -0400 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 1ECC7401DE93 for ; Mon, 12 Mar 2018 17:21:58 +0000 (UTC) Received: from dgilbert-t530.redhat.com (ovpn-117-169.ams2.redhat.com [10.36.117.169]) by smtp.corp.redhat.com (Postfix) with ESMTP id DB688202322A; Mon, 12 Mar 2018 17:21:56 +0000 (UTC) From: "Dr. David Alan Gilbert (git)" To: qemu-devel@nongnu.org, mst@redhat.com, maxime.coquelin@redhat.com, marcandre.lureau@redhat.com, peterx@redhat.com, quintela@redhat.com Date: Mon, 12 Mar 2018 17:21:13 +0000 Message-Id: <20180312172124.56461-19-dgilbert@redhat.com> In-Reply-To: <20180312172124.56461-1-dgilbert@redhat.com> References: <20180312172124.56461-1-dgilbert@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.78 on 10.11.54.4 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.6]); Mon, 12 Mar 2018 17:21:58 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.6]); Mon, 12 Mar 2018 17:21:58 +0000 (UTC) for IP:'10.11.54.4' DOMAIN:'int-mx04.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'dgilbert@redhat.com' RCPT:'' Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH v5 18/29] vhost+postcopy: Resolve client address X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: aarcange@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Type: text/plain; charset="utf-8" From: "Dr. David Alan Gilbert" Resolve fault addresses read off the clients UFD into RAMBlock and offset, and call back to the postcopy code to ask for the page. Signed-off-by: Dr. David Alan Gilbert Reviewed-by: Peter Xu Reviewed-by: Marc-Andr=C3=A9 Lureau --- hw/virtio/trace-events | 3 +++ hw/virtio/vhost-user.c | 31 ++++++++++++++++++++++++++++++- 2 files changed, 33 insertions(+), 1 deletion(-) diff --git a/hw/virtio/trace-events b/hw/virtio/trace-events index d7e9e1084b..3afd12cfea 100644 --- a/hw/virtio/trace-events +++ b/hw/virtio/trace-events @@ -7,6 +7,9 @@ vhost_region_add_section_abut(const char *name, uint64_t ne= w_size) "%s: 0x%"PRIx vhost_section(const char *name, int r) "%s:%d" =20 # hw/virtio/vhost-user.c +vhost_user_postcopy_fault_handler(const char *name, uint64_t fault_address= , int nregions) "%s: @0x%"PRIx64" nregions:%d" +vhost_user_postcopy_fault_handler_loop(int i, uint64_t client_base, uint64= _t size) "%d: client 0x%"PRIx64" +0x%"PRIx64 +vhost_user_postcopy_fault_handler_found(int i, uint64_t region_offset, uin= t64_t rb_offset) "%d: region_offset: 0x%"PRIx64" rb_offset:0x%"PRIx64 vhost_user_postcopy_listen(void) "" vhost_user_set_mem_table_postcopy(uint64_t client_addr, uint64_t qhva, int= reply_i, int region_i) "client:0x%"PRIx64" for hva: 0x%"PRIx64" reply %d r= egion %d" vhost_user_set_mem_table_withfd(int index, const char *name, uint64_t memo= ry_size, uint64_t guest_phys_addr, uint64_t userspace_addr, uint64_t offset= ) "%d:%s: size:0x%"PRIx64" GPA:0x%"PRIx64" QVA/userspace:0x%"PRIx64" RB off= set:0x%"PRIx64 diff --git a/hw/virtio/vhost-user.c b/hw/virtio/vhost-user.c index b47de62c47..6dee1b565c 100644 --- a/hw/virtio/vhost-user.c +++ b/hw/virtio/vhost-user.c @@ -26,6 +26,7 @@ #include #include #include +#include =20 #define VHOST_MEMORY_MAX_NREGIONS 8 #define VHOST_USER_F_PROTOCOL_FEATURES 30 @@ -974,7 +975,35 @@ out: static int vhost_user_postcopy_fault_handler(struct PostCopyFD *pcfd, void *ufd) { - return 0; + struct vhost_dev *dev =3D pcfd->data; + struct vhost_user *u =3D dev->opaque; + struct uffd_msg *msg =3D ufd; + uint64_t faultaddr =3D msg->arg.pagefault.address; + RAMBlock *rb =3D NULL; + uint64_t rb_offset; + int i; + + trace_vhost_user_postcopy_fault_handler(pcfd->idstr, faultaddr, + dev->mem->nregions); + for (i =3D 0; i < MIN(dev->mem->nregions, u->region_rb_len); i++) { + trace_vhost_user_postcopy_fault_handler_loop(i, + u->postcopy_client_bases[i], dev->mem->regions[i].memory_s= ize); + if (faultaddr >=3D u->postcopy_client_bases[i]) { + /* Ofset of the fault address in the vhost region */ + uint64_t region_offset =3D faultaddr - u->postcopy_client_base= s[i]; + if (region_offset < dev->mem->regions[i].memory_size) { + rb_offset =3D region_offset + u->region_rb_offset[i]; + trace_vhost_user_postcopy_fault_handler_found(i, + region_offset, rb_offset); + rb =3D u->region_rb[i]; + return postcopy_request_shared_page(pcfd, rb, faultaddr, + rb_offset); + } + } + } + error_report("%s: Failed to find region for fault %" PRIx64, + __func__, faultaddr); + return -1; } =20 /* --=20 2.14.3 From nobody Sun May 5 21:21:29 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (208.118.235.17 [208.118.235.17]) by mx.zohomail.com with SMTPS id 1520876471293602.1202080714413; Mon, 12 Mar 2018 10:41:11 -0700 (PDT) Received: from localhost ([::1]:33495 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1evRRV-0000GN-Ah for importer@patchew.org; Mon, 12 Mar 2018 13:41:01 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:53151) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1evR98-0008NI-Bh for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:22:03 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1evR97-0007mO-6u for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:22:02 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:57628 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1evR97-0007ln-11 for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:22:01 -0400 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 733F5401DE71 for ; Mon, 12 Mar 2018 17:22:00 +0000 (UTC) Received: from dgilbert-t530.redhat.com (ovpn-117-169.ams2.redhat.com [10.36.117.169]) by smtp.corp.redhat.com (Postfix) with ESMTP id 5A08B202322A; Mon, 12 Mar 2018 17:21:58 +0000 (UTC) From: "Dr. David Alan Gilbert (git)" To: qemu-devel@nongnu.org, mst@redhat.com, maxime.coquelin@redhat.com, marcandre.lureau@redhat.com, peterx@redhat.com, quintela@redhat.com Date: Mon, 12 Mar 2018 17:21:14 +0000 Message-Id: <20180312172124.56461-20-dgilbert@redhat.com> In-Reply-To: <20180312172124.56461-1-dgilbert@redhat.com> References: <20180312172124.56461-1-dgilbert@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.78 on 10.11.54.4 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.6]); Mon, 12 Mar 2018 17:22:00 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.6]); Mon, 12 Mar 2018 17:22:00 +0000 (UTC) for IP:'10.11.54.4' DOMAIN:'int-mx04.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'dgilbert@redhat.com' RCPT:'' Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH v5 19/29] postcopy: helper for waking shared X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: aarcange@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Type: text/plain; charset="utf-8" From: "Dr. David Alan Gilbert" Provide a helper to send a 'wake' request on a userfaultfd for a shared process. The address in the clients address space is specified together with the RAMBlock it was resolved to. Signed-off-by: Dr. David Alan Gilbert Reviewed-by: Marc-Andr=C3=A9 Lureau --- migration/postcopy-ram.c | 26 ++++++++++++++++++++++++++ migration/postcopy-ram.h | 10 ++++++++++ migration/trace-events | 1 + 3 files changed, 37 insertions(+) diff --git a/migration/postcopy-ram.c b/migration/postcopy-ram.c index 83780576ca..d3e7f10c5a 100644 --- a/migration/postcopy-ram.c +++ b/migration/postcopy-ram.c @@ -525,6 +525,25 @@ static int ram_block_enable_notify(const char *block_n= ame, void *host_addr, return 0; } =20 +int postcopy_wake_shared(struct PostCopyFD *pcfd, + uint64_t client_addr, + RAMBlock *rb) +{ + size_t pagesize =3D qemu_ram_pagesize(rb); + struct uffdio_range range; + int ret; + trace_postcopy_wake_shared(client_addr, qemu_ram_get_idstr(rb)); + range.start =3D client_addr & ~(pagesize - 1); + range.len =3D pagesize; + ret =3D ioctl(pcfd->fd, UFFDIO_WAKE, &range); + if (ret) { + error_report("%s: Failed to wake: %zx in %s (%s)", + __func__, (size_t)client_addr, qemu_ram_get_idstr(rb), + strerror(errno)); + } + return ret; +} + /* * Callback from shared fault handlers to ask for a page, * the page must be specified by a RAMBlock and an offset in that rb @@ -954,6 +973,13 @@ void *postcopy_get_tmp_page(MigrationIncomingState *mi= s) return NULL; } =20 +int postcopy_wake_shared(struct PostCopyFD *pcfd, + uint64_t client_addr, + RAMBlock *rb) +{ + assert(0); + return -1; +} #endif =20 /* -----------------------------------------------------------------------= -- */ diff --git a/migration/postcopy-ram.h b/migration/postcopy-ram.h index d7afab07d7..fcd53b8295 100644 --- a/migration/postcopy-ram.h +++ b/migration/postcopy-ram.h @@ -164,6 +164,16 @@ struct PostCopyFD { */ void postcopy_register_shared_ufd(struct PostCopyFD *pcfd); void postcopy_unregister_shared_ufd(struct PostCopyFD *pcfd); +/* postcopy_wake_shared: Notify a client ufd that a page is available + * + * Returns 0 on success + * + * @pcfd: Structure with fd, handler and name as above + * @client_addr: Address in the client program, not QEMU + * @rb: The RAMBlock the page is in + */ +int postcopy_wake_shared(struct PostCopyFD *pcfd, uint64_t client_addr, + RAMBlock *rb); /* Callback from shared fault handlers to ask for a page */ int postcopy_request_shared_page(struct PostCopyFD *pcfd, RAMBlock *rb, uint64_t client_addr, uint64_t offset); diff --git a/migration/trace-events b/migration/trace-events index 7c910b5479..b0acaaa8a0 100644 --- a/migration/trace-events +++ b/migration/trace-events @@ -199,6 +199,7 @@ postcopy_ram_incoming_cleanup_entry(void) "" postcopy_ram_incoming_cleanup_exit(void) "" postcopy_ram_incoming_cleanup_join(void) "" postcopy_request_shared_page(const char *sharer, const char *rb, uint64_t = rb_offset) "for %s in %s offset 0x%"PRIx64 +postcopy_wake_shared(uint64_t client_addr, const char *rb) "at 0x%"PRIx64"= in %s" =20 save_xbzrle_page_skipping(void) "" save_xbzrle_page_overflow(void) "" --=20 2.14.3 From nobody Sun May 5 21:21:29 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1520876622551697.3386179513602; Mon, 12 Mar 2018 10:43:42 -0700 (PDT) Received: from localhost ([::1]:33508 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1evRU5-0003B4-Mt for importer@patchew.org; Mon, 12 Mar 2018 13:43:41 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:53163) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1evR99-0008OS-F1 for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:22:06 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1evR98-0007no-FB for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:22:03 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:56710 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1evR98-0007nD-Af for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:22:02 -0400 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id E85384040856 for ; Mon, 12 Mar 2018 17:22:01 +0000 (UTC) Received: from dgilbert-t530.redhat.com (ovpn-117-169.ams2.redhat.com [10.36.117.169]) by smtp.corp.redhat.com (Postfix) with ESMTP id AE70D2026E03; Mon, 12 Mar 2018 17:22:00 +0000 (UTC) From: "Dr. David Alan Gilbert (git)" To: qemu-devel@nongnu.org, mst@redhat.com, maxime.coquelin@redhat.com, marcandre.lureau@redhat.com, peterx@redhat.com, quintela@redhat.com Date: Mon, 12 Mar 2018 17:21:15 +0000 Message-Id: <20180312172124.56461-21-dgilbert@redhat.com> In-Reply-To: <20180312172124.56461-1-dgilbert@redhat.com> References: <20180312172124.56461-1-dgilbert@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.78 on 10.11.54.4 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.5]); Mon, 12 Mar 2018 17:22:01 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.5]); Mon, 12 Mar 2018 17:22:01 +0000 (UTC) for IP:'10.11.54.4' DOMAIN:'int-mx04.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'dgilbert@redhat.com' RCPT:'' Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH v5 20/29] postcopy: postcopy_notify_shared_wake X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: aarcange@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Type: text/plain; charset="utf-8" From: "Dr. David Alan Gilbert" Add a hook to allow a client userfaultfd to be 'woken' when a page arrives, and a walker that calls that hook for relevant clients given a RAMBlock and offset. Signed-off-by: Dr. David Alan Gilbert Reviewed-by: Peter Xu Reviewed-by: Marc-Andr=C3=A9 Lureau --- migration/postcopy-ram.c | 16 ++++++++++++++++ migration/postcopy-ram.h | 10 ++++++++++ 2 files changed, 26 insertions(+) diff --git a/migration/postcopy-ram.c b/migration/postcopy-ram.c index d3e7f10c5a..585db9467a 100644 --- a/migration/postcopy-ram.c +++ b/migration/postcopy-ram.c @@ -827,6 +827,22 @@ static int qemu_ufd_copy_ioctl(int userfault_fd, void = *host_addr, return ret; } =20 +int postcopy_notify_shared_wake(RAMBlock *rb, uint64_t offset) +{ + int i; + MigrationIncomingState *mis =3D migration_incoming_get_current(); + GArray *pcrfds =3D mis->postcopy_remote_fds; + + for (i =3D 0; i < pcrfds->len; i++) { + struct PostCopyFD *cur =3D &g_array_index(pcrfds, struct PostCopyF= D, i); + int ret =3D cur->waker(cur, rb, offset); + if (ret) { + return ret; + } + } + return 0; +} + /* * Place a host page (from) at (host) atomically * returns 0 on success diff --git a/migration/postcopy-ram.h b/migration/postcopy-ram.h index fcd53b8295..2c73d77a5c 100644 --- a/migration/postcopy-ram.h +++ b/migration/postcopy-ram.h @@ -148,6 +148,10 @@ struct PostCopyFD; =20 /* ufd is a pointer to the struct uffd_msg *TODO: more Portable! */ typedef int (*pcfdhandler)(struct PostCopyFD *pcfd, void *ufd); +/* Notification to wake, either on place or on reception of + * a fault on something that's already arrived (race) + */ +typedef int (*pcfdwake)(struct PostCopyFD *pcfd, RAMBlock *rb, uint64_t of= fset); =20 struct PostCopyFD { int fd; @@ -155,6 +159,8 @@ struct PostCopyFD { void *data; /* Handler to be called whenever we get a poll event */ pcfdhandler handler; + /* Notification to wake shared client */ + pcfdwake waker; /* A string to use in error messages */ const char *idstr; }; @@ -164,6 +170,10 @@ struct PostCopyFD { */ void postcopy_register_shared_ufd(struct PostCopyFD *pcfd); void postcopy_unregister_shared_ufd(struct PostCopyFD *pcfd); +/* Call each of the shared 'waker's registerd telling them of + * availability of a block. + */ +int postcopy_notify_shared_wake(RAMBlock *rb, uint64_t offset); /* postcopy_wake_shared: Notify a client ufd that a page is available * * Returns 0 on success --=20 2.14.3 From nobody Sun May 5 21:21:29 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1520876762164919.0899707819549; Mon, 12 Mar 2018 10:46:02 -0700 (PDT) Received: from localhost ([::1]:33532 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1evRWL-0005Dp-ES for importer@patchew.org; Mon, 12 Mar 2018 13:46:01 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:53201) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1evR9C-0008Ry-MM for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:22:07 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1evR99-0007oy-TA for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:22:06 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:38224 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1evR99-0007og-Os for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:22:03 -0400 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 6CC4976FB9 for ; Mon, 12 Mar 2018 17:22:03 +0000 (UTC) Received: from dgilbert-t530.redhat.com (ovpn-117-169.ams2.redhat.com [10.36.117.169]) by smtp.corp.redhat.com (Postfix) with ESMTP id 2F8AA202322A; Mon, 12 Mar 2018 17:22:02 +0000 (UTC) From: "Dr. David Alan Gilbert (git)" To: qemu-devel@nongnu.org, mst@redhat.com, maxime.coquelin@redhat.com, marcandre.lureau@redhat.com, peterx@redhat.com, quintela@redhat.com Date: Mon, 12 Mar 2018 17:21:16 +0000 Message-Id: <20180312172124.56461-22-dgilbert@redhat.com> In-Reply-To: <20180312172124.56461-1-dgilbert@redhat.com> References: <20180312172124.56461-1-dgilbert@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.78 on 10.11.54.4 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.1]); Mon, 12 Mar 2018 17:22:03 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.1]); Mon, 12 Mar 2018 17:22:03 +0000 (UTC) for IP:'10.11.54.4' DOMAIN:'int-mx04.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'dgilbert@redhat.com' RCPT:'' Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH v5 21/29] vhost+postcopy: Add vhost waker X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: aarcange@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Type: text/plain; charset="utf-8" From: "Dr. David Alan Gilbert" Register a waker function in vhost-user code to be notified when pages arrive or requests to previously mapped pages get requested. Signed-off-by: Dr. David Alan Gilbert Reviewed-by: Marc-Andr=C3=A9 Lureau --- hw/virtio/trace-events | 3 +++ hw/virtio/vhost-user.c | 30 ++++++++++++++++++++++++++++++ 2 files changed, 33 insertions(+) diff --git a/hw/virtio/trace-events b/hw/virtio/trace-events index 3afd12cfea..fe5e0ff856 100644 --- a/hw/virtio/trace-events +++ b/hw/virtio/trace-events @@ -13,6 +13,9 @@ vhost_user_postcopy_fault_handler_found(int i, uint64_t r= egion_offset, uint64_t vhost_user_postcopy_listen(void) "" vhost_user_set_mem_table_postcopy(uint64_t client_addr, uint64_t qhva, int= reply_i, int region_i) "client:0x%"PRIx64" for hva: 0x%"PRIx64" reply %d r= egion %d" vhost_user_set_mem_table_withfd(int index, const char *name, uint64_t memo= ry_size, uint64_t guest_phys_addr, uint64_t userspace_addr, uint64_t offset= ) "%d:%s: size:0x%"PRIx64" GPA:0x%"PRIx64" QVA/userspace:0x%"PRIx64" RB off= set:0x%"PRIx64 +vhost_user_postcopy_waker(const char *rb, uint64_t rb_offset) "%s + 0x%"PR= Ix64 +vhost_user_postcopy_waker_found(uint64_t client_addr) "0x%"PRIx64 +vhost_user_postcopy_waker_nomatch(const char *rb, uint64_t rb_offset) "%s = + 0x%"PRIx64 =20 # hw/virtio/virtio.c virtqueue_alloc_element(void *elem, size_t sz, unsigned in_num, unsigned o= ut_num) "elem %p size %zd in_num %u out_num %u" diff --git a/hw/virtio/vhost-user.c b/hw/virtio/vhost-user.c index 6dee1b565c..a785aefd3e 100644 --- a/hw/virtio/vhost-user.c +++ b/hw/virtio/vhost-user.c @@ -1006,6 +1006,35 @@ static int vhost_user_postcopy_fault_handler(struct = PostCopyFD *pcfd, return -1; } =20 +static int vhost_user_postcopy_waker(struct PostCopyFD *pcfd, RAMBlock *rb, + uint64_t offset) +{ + struct vhost_dev *dev =3D pcfd->data; + struct vhost_user *u =3D dev->opaque; + int i; + + trace_vhost_user_postcopy_waker(qemu_ram_get_idstr(rb), offset); + + if (!u) { + return 0; + } + /* Translate the offset into an address in the clients address space */ + for (i =3D 0; i < MIN(dev->mem->nregions, u->region_rb_len); i++) { + if (u->region_rb[i] =3D=3D rb && + offset >=3D u->region_rb_offset[i] && + offset < (u->region_rb_offset[i] + + dev->mem->regions[i].memory_size)) { + uint64_t client_addr =3D (offset - u->region_rb_offset[i]) + + u->postcopy_client_bases[i]; + trace_vhost_user_postcopy_waker_found(client_addr); + return postcopy_wake_shared(pcfd, client_addr, rb); + } + } + + trace_vhost_user_postcopy_waker_nomatch(qemu_ram_get_idstr(rb), offset= ); + return 0; +} + /* * Called at the start of an inbound postcopy on reception of the * 'advise' command. @@ -1051,6 +1080,7 @@ static int vhost_user_postcopy_advise(struct vhost_de= v *dev, Error **errp) u->postcopy_fd.fd =3D ufd; u->postcopy_fd.data =3D dev; u->postcopy_fd.handler =3D vhost_user_postcopy_fault_handler; + u->postcopy_fd.waker =3D vhost_user_postcopy_waker; u->postcopy_fd.idstr =3D "vhost-user"; /* Need to find unique name */ postcopy_register_shared_ufd(&u->postcopy_fd); return 0; --=20 2.14.3 From nobody Sun May 5 21:21:29 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 152087635389684.77656339226053; Mon, 12 Mar 2018 10:39:13 -0700 (PDT) Received: from localhost ([::1]:33480 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1evRPl-0006Xc-3C for importer@patchew.org; Mon, 12 Mar 2018 13:39:13 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:53199) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1evR9C-0008Ru-LO for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:22:07 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1evR9B-0007q3-Ca for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:22:06 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:57630 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1evR9B-0007pd-7s for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:22:05 -0400 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id E1078401DE71 for ; Mon, 12 Mar 2018 17:22:04 +0000 (UTC) Received: from dgilbert-t530.redhat.com (ovpn-117-169.ams2.redhat.com [10.36.117.169]) by smtp.corp.redhat.com (Postfix) with ESMTP id A646A2026E03; Mon, 12 Mar 2018 17:22:03 +0000 (UTC) From: "Dr. David Alan Gilbert (git)" To: qemu-devel@nongnu.org, mst@redhat.com, maxime.coquelin@redhat.com, marcandre.lureau@redhat.com, peterx@redhat.com, quintela@redhat.com Date: Mon, 12 Mar 2018 17:21:17 +0000 Message-Id: <20180312172124.56461-23-dgilbert@redhat.com> In-Reply-To: <20180312172124.56461-1-dgilbert@redhat.com> References: <20180312172124.56461-1-dgilbert@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.4 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.6]); Mon, 12 Mar 2018 17:22:04 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.6]); Mon, 12 Mar 2018 17:22:04 +0000 (UTC) for IP:'10.11.54.4' DOMAIN:'int-mx04.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'dgilbert@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH v5 22/29] vhost+postcopy: Call wakeups X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: aarcange@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: "Dr. David Alan Gilbert" Cause the vhost-user client to be woken up whenever: a) We place a page in postcopy mode b) We get a fault and the page has already been received Signed-off-by: Dr. David Alan Gilbert --- migration/postcopy-ram.c | 14 ++++++++++---- migration/trace-events | 1 + 2 files changed, 11 insertions(+), 4 deletions(-) diff --git a/migration/postcopy-ram.c b/migration/postcopy-ram.c index 585db9467a..36db900e8f 100644 --- a/migration/postcopy-ram.c +++ b/migration/postcopy-ram.c @@ -558,7 +558,11 @@ int postcopy_request_shared_page(struct PostCopyFD *pc= fd, RAMBlock *rb, =20 trace_postcopy_request_shared_page(pcfd->idstr, qemu_ram_get_idstr(rb), rb_offset); - /* TODO: Check bitmap to see if we already have the page */ + if (ramblock_recv_bitmap_test_byte_offset(rb, aligned_rbo)) { + trace_postcopy_request_shared_page_present(pcfd->idstr, + qemu_ram_get_idstr(rb), rb_offset); + return postcopy_wake_shared(pcfd, client_addr, rb); + } if (rb !=3D mis->last_rb) { mis->last_rb =3D rb; migrate_send_rp_req_pages(mis, qemu_ram_get_idstr(rb), @@ -866,7 +870,8 @@ int postcopy_place_page(MigrationIncomingState *mis, vo= id *host, void *from, } =20 trace_postcopy_place_page(host); - return 0; + return postcopy_notify_shared_wake(rb, + qemu_ram_block_host_offset(rb, host= )); } =20 /* @@ -890,6 +895,9 @@ int postcopy_place_page_zero(MigrationIncomingState *mi= s, void *host, =20 return -e; } + return postcopy_notify_shared_wake(rb, + qemu_ram_block_host_offset(rb, + host= )); } else { /* The kernel can't use UFFDIO_ZEROPAGE for hugepages */ if (!mis->postcopy_tmp_zero_page) { @@ -909,8 +917,6 @@ int postcopy_place_page_zero(MigrationIncomingState *mi= s, void *host, return postcopy_place_page(mis, host, mis->postcopy_tmp_zero_page, rb); } - - return 0; } =20 /* diff --git a/migration/trace-events b/migration/trace-events index b0acaaa8a0..1e353a317f 100644 --- a/migration/trace-events +++ b/migration/trace-events @@ -199,6 +199,7 @@ postcopy_ram_incoming_cleanup_entry(void) "" postcopy_ram_incoming_cleanup_exit(void) "" postcopy_ram_incoming_cleanup_join(void) "" postcopy_request_shared_page(const char *sharer, const char *rb, uint64_t = rb_offset) "for %s in %s offset 0x%"PRIx64 +postcopy_request_shared_page_present(const char *sharer, const char *rb, u= int64_t rb_offset) "%s already %s offset 0x%"PRIx64 postcopy_wake_shared(uint64_t client_addr, const char *rb) "at 0x%"PRIx64"= in %s" =20 save_xbzrle_page_skipping(void) "" --=20 2.14.3 From nobody Sun May 5 21:21:29 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1520876518744343.04861291873476; Mon, 12 Mar 2018 10:41:58 -0700 (PDT) Received: from localhost ([::1]:33499 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1evRSP-0001MW-T4 for importer@patchew.org; Mon, 12 Mar 2018 13:41:57 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:53223) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1evR9E-0008TU-42 for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:22:09 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1evR9C-0007r3-SV for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:22:08 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:57632 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1evR9C-0007qf-NO for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:22:06 -0400 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 6054D401DE71 for ; Mon, 12 Mar 2018 17:22:06 +0000 (UTC) Received: from dgilbert-t530.redhat.com (ovpn-117-169.ams2.redhat.com [10.36.117.169]) by smtp.corp.redhat.com (Postfix) with ESMTP id 27ACB202322C; Mon, 12 Mar 2018 17:22:05 +0000 (UTC) From: "Dr. David Alan Gilbert (git)" To: qemu-devel@nongnu.org, mst@redhat.com, maxime.coquelin@redhat.com, marcandre.lureau@redhat.com, peterx@redhat.com, quintela@redhat.com Date: Mon, 12 Mar 2018 17:21:18 +0000 Message-Id: <20180312172124.56461-24-dgilbert@redhat.com> In-Reply-To: <20180312172124.56461-1-dgilbert@redhat.com> References: <20180312172124.56461-1-dgilbert@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.78 on 10.11.54.4 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.6]); Mon, 12 Mar 2018 17:22:06 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.6]); Mon, 12 Mar 2018 17:22:06 +0000 (UTC) for IP:'10.11.54.4' DOMAIN:'int-mx04.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'dgilbert@redhat.com' RCPT:'' Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH v5 23/29] libvhost-user: mprotect & madvises for postcopy X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: aarcange@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Type: text/plain; charset="utf-8" From: "Dr. David Alan Gilbert" Clear the area and turn off THP. PROT_NONE the area until after we've userfault advised it to catch any unexpected changes. Signed-off-by: Dr. David Alan Gilbert Reviewed-by: Marc-Andr=C3=A9 Lureau --- contrib/libvhost-user/libvhost-user.c | 47 +++++++++++++++++++++++++++++++= ---- 1 file changed, 42 insertions(+), 5 deletions(-) diff --git a/contrib/libvhost-user/libvhost-user.c b/contrib/libvhost-user/= libvhost-user.c index 6314549b65..5feed52098 100644 --- a/contrib/libvhost-user/libvhost-user.c +++ b/contrib/libvhost-user/libvhost-user.c @@ -454,7 +454,7 @@ vu_set_mem_table_exec_postcopy(VuDev *dev, VhostUserMsg= *vmsg) int i; VhostUserMemory *memory =3D &vmsg->payload.memory; dev->nregions =3D memory->nregions; - /* TODO: Postcopy specific code */ + DPRINT("Nregions: %d\n", memory->nregions); for (i =3D 0; i < dev->nregions; i++) { void *mmap_addr; @@ -478,9 +478,12 @@ vu_set_mem_table_exec_postcopy(VuDev *dev, VhostUserMs= g *vmsg) =20 /* We don't use offset argument of mmap() since the * mapped address has to be page aligned, and we use huge - * pages. */ + * pages. + * In postcopy we're using PROT_NONE here to catch anyone + * accessing it before we userfault + */ mmap_addr =3D mmap(0, dev_region->size + dev_region->mmap_offset, - PROT_READ | PROT_WRITE, MAP_SHARED, + PROT_NONE, MAP_SHARED, vmsg->fds[i], 0); =20 if (mmap_addr =3D=3D MAP_FAILED) { @@ -519,12 +522,38 @@ vu_set_mem_table_exec_postcopy(VuDev *dev, VhostUserM= sg *vmsg) /* OK, now we can go and register the memory and generate faults */ for (i =3D 0; i < dev->nregions; i++) { VuDevRegion *dev_region =3D &dev->regions[i]; + int ret; #ifdef UFFDIO_REGISTER /* We should already have an open ufd. Mark each memory * range as ufd. - * Note: Do we need any madvises? Well it's not been accessed - * yet, still probably need no THP to be safe, discard to be safe? + * Discard any mapping we have here; note I can't use MADV_REMOVE + * or fallocate to make the hole since I don't want to lose + * data that's already arrived in the shared process. + * TODO: How to do hugepage */ + ret =3D madvise((void *)dev_region->mmap_addr, + dev_region->size + dev_region->mmap_offset, + MADV_DONTNEED); + if (ret) { + fprintf(stderr, + "%s: Failed to madvise(DONTNEED) region %d: %s\n", + __func__, i, strerror(errno)); + } + /* Turn off transparent hugepages so we dont get lose wakeups + * in neighbouring pages. + * TODO: Turn this backon later. + */ + ret =3D madvise((void *)dev_region->mmap_addr, + dev_region->size + dev_region->mmap_offset, + MADV_NOHUGEPAGE); + if (ret) { + /* Note: This can happen legally on kernels that are configured + * without madvise'able hugepages + */ + fprintf(stderr, + "%s: Failed to madvise(NOHUGEPAGE) region %d: %s\n", + __func__, i, strerror(errno)); + } struct uffdio_register reg_struct; reg_struct.range.start =3D (uintptr_t)dev_region->mmap_addr; reg_struct.range.len =3D dev_region->size + dev_region->mmap_offse= t; @@ -546,6 +575,14 @@ vu_set_mem_table_exec_postcopy(VuDev *dev, VhostUserMs= g *vmsg) } DPRINT("%s: region %d: Registered userfault for %llx + %llx\n", __func__, i, reg_struct.range.start, reg_struct.range.len); + /* Now it's registered we can let the client at it */ + if (mprotect((void *)dev_region->mmap_addr, + dev_region->size + dev_region->mmap_offset, + PROT_READ | PROT_WRITE)) { + vu_panic(dev, "failed to mprotect region %d for postcopy (%s)", + i, strerror(errno)); + return false; + } /* TODO: Stash 'zero' support flags somewhere */ #endif } --=20 2.14.3 From nobody Sun May 5 21:21:29 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 152087630283885.70686652778511; Mon, 12 Mar 2018 10:38:22 -0700 (PDT) Received: from localhost ([::1]:33478 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1evROv-0005hV-Q3 for importer@patchew.org; Mon, 12 Mar 2018 13:38:21 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:53237) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1evR9F-0008V5-Hw for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:22:11 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1evR9E-0007s7-Cv for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:22:09 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:36146 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1evR9E-0007rk-7d for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:22:08 -0400 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id D32578D6D7 for ; Mon, 12 Mar 2018 17:22:07 +0000 (UTC) Received: from dgilbert-t530.redhat.com (ovpn-117-169.ams2.redhat.com [10.36.117.169]) by smtp.corp.redhat.com (Postfix) with ESMTP id 9B22B202322C; Mon, 12 Mar 2018 17:22:06 +0000 (UTC) From: "Dr. David Alan Gilbert (git)" To: qemu-devel@nongnu.org, mst@redhat.com, maxime.coquelin@redhat.com, marcandre.lureau@redhat.com, peterx@redhat.com, quintela@redhat.com Date: Mon, 12 Mar 2018 17:21:19 +0000 Message-Id: <20180312172124.56461-25-dgilbert@redhat.com> In-Reply-To: <20180312172124.56461-1-dgilbert@redhat.com> References: <20180312172124.56461-1-dgilbert@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.78 on 10.11.54.4 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.2]); Mon, 12 Mar 2018 17:22:07 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.2]); Mon, 12 Mar 2018 17:22:07 +0000 (UTC) for IP:'10.11.54.4' DOMAIN:'int-mx04.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'dgilbert@redhat.com' RCPT:'' Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH v5 24/29] vhost-user: Add VHOST_USER_POSTCOPY_END message X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: aarcange@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Type: text/plain; charset="utf-8" From: "Dr. David Alan Gilbert" This message is sent just before the end of postcopy to get the client to stop using userfault since we wont respond to any more requests. It should close userfaultfd so that any other pages get mapped to the backing file automatically by the kernel, since at this point we know we've received everything. Signed-off-by: Dr. David Alan Gilbert Reviewed-by: Peter Xu Reviewed-by: Marc-Andr=C3=A9 Lureau --- contrib/libvhost-user/libvhost-user.c | 23 +++++++++++++++++++++++ contrib/libvhost-user/libvhost-user.h | 1 + docs/interop/vhost-user.txt | 12 ++++++++++++ hw/virtio/vhost-user.c | 1 + 4 files changed, 37 insertions(+) diff --git a/contrib/libvhost-user/libvhost-user.c b/contrib/libvhost-user/= libvhost-user.c index 5feed52098..504ff5ea59 100644 --- a/contrib/libvhost-user/libvhost-user.c +++ b/contrib/libvhost-user/libvhost-user.c @@ -99,6 +99,7 @@ vu_request_to_string(unsigned int req) REQ(VHOST_USER_SET_CONFIG), REQ(VHOST_USER_POSTCOPY_ADVISE), REQ(VHOST_USER_POSTCOPY_LISTEN), + REQ(VHOST_USER_POSTCOPY_END), REQ(VHOST_USER_MAX), }; #undef REQ @@ -1094,6 +1095,26 @@ vu_set_postcopy_listen(VuDev *dev, VhostUserMsg *vms= g) vmsg->payload.u64 =3D 0; /* Success */ return true; } + +static bool +vu_set_postcopy_end(VuDev *dev, VhostUserMsg *vmsg) +{ + DPRINT("%s: Entry\n", __func__); + dev->postcopy_listening =3D false; + if (dev->postcopy_ufd > 0) { + close(dev->postcopy_ufd); + dev->postcopy_ufd =3D -1; + DPRINT("%s: Done close\n", __func__); + } + + vmsg->fd_num =3D 0; + vmsg->payload.u64 =3D 0; + vmsg->size =3D sizeof(vmsg->payload.u64); + vmsg->flags =3D VHOST_USER_VERSION | VHOST_USER_REPLY_MASK; + DPRINT("%s: exit\n", __func__); + return true; +} + static bool vu_process_message(VuDev *dev, VhostUserMsg *vmsg) { @@ -1169,6 +1190,8 @@ vu_process_message(VuDev *dev, VhostUserMsg *vmsg) return vu_set_postcopy_advise(dev, vmsg); case VHOST_USER_POSTCOPY_LISTEN: return vu_set_postcopy_listen(dev, vmsg); + case VHOST_USER_POSTCOPY_END: + return vu_set_postcopy_end(dev, vmsg); default: vmsg_close_fds(vmsg); vu_panic(dev, "Unhandled request: %d", vmsg->request); diff --git a/contrib/libvhost-user/libvhost-user.h b/contrib/libvhost-user/= libvhost-user.h index ed505cf0c1..79f7a53ee8 100644 --- a/contrib/libvhost-user/libvhost-user.h +++ b/contrib/libvhost-user/libvhost-user.h @@ -87,6 +87,7 @@ typedef enum VhostUserRequest { VHOST_USER_CLOSE_CRYPTO_SESSION =3D 27, VHOST_USER_POSTCOPY_ADVISE =3D 28, VHOST_USER_POSTCOPY_LISTEN =3D 29, + VHOST_USER_POSTCOPY_END =3D 30, VHOST_USER_MAX } VhostUserRequest; =20 diff --git a/docs/interop/vhost-user.txt b/docs/interop/vhost-user.txt index e295ef12ca..c058c407df 100644 --- a/docs/interop/vhost-user.txt +++ b/docs/interop/vhost-user.txt @@ -729,6 +729,18 @@ Master message types This is always sent sometime after a VHOST_USER_POSTCOPY_ADVISE, and thus only when VHOST_USER_PROTOCOL_F_PAGEFAULT is supported. =20 + * VHOST_USER_POSTCOPY_END + Id: 30 + Slave payload: u64 + + Master advises that postcopy migration has now completed. The + slave must disable the userfaultfd. The response is an acknowledgeme= nt + only. + When VHOST_USER_PROTOCOL_F_PAGEFAULT is supported, this message + is sent at the end of the migration, after VHOST_USER_POSTCOPY_LISTEN + was previously sent. + The value returned is an error indication; 0 is success. + Slave message types ------------------- =20 diff --git a/hw/virtio/vhost-user.c b/hw/virtio/vhost-user.c index a785aefd3e..230f2f9d55 100644 --- a/hw/virtio/vhost-user.c +++ b/hw/virtio/vhost-user.c @@ -82,6 +82,7 @@ typedef enum VhostUserRequest { VHOST_USER_CLOSE_CRYPTO_SESSION =3D 27, VHOST_USER_POSTCOPY_ADVISE =3D 28, VHOST_USER_POSTCOPY_LISTEN =3D 29, + VHOST_USER_POSTCOPY_END =3D 30, VHOST_USER_MAX } VhostUserRequest; =20 --=20 2.14.3 From nobody Sun May 5 21:21:29 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1520876668089227.05776190542724; Mon, 12 Mar 2018 10:44:28 -0700 (PDT) Received: from localhost ([::1]:33513 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1evRUo-0003vt-Ri for importer@patchew.org; Mon, 12 Mar 2018 13:44:26 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:53253) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1evR9H-0008WF-Jz for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:22:12 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1evR9F-0007t7-RT for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:22:11 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:47228 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1evR9F-0007sp-MK for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:22:09 -0400 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 533EF406E97D for ; Mon, 12 Mar 2018 17:22:09 +0000 (UTC) Received: from dgilbert-t530.redhat.com (ovpn-117-169.ams2.redhat.com [10.36.117.169]) by smtp.corp.redhat.com (Postfix) with ESMTP id 1A491202322B; Mon, 12 Mar 2018 17:22:07 +0000 (UTC) From: "Dr. David Alan Gilbert (git)" To: qemu-devel@nongnu.org, mst@redhat.com, maxime.coquelin@redhat.com, marcandre.lureau@redhat.com, peterx@redhat.com, quintela@redhat.com Date: Mon, 12 Mar 2018 17:21:20 +0000 Message-Id: <20180312172124.56461-26-dgilbert@redhat.com> In-Reply-To: <20180312172124.56461-1-dgilbert@redhat.com> References: <20180312172124.56461-1-dgilbert@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.78 on 10.11.54.4 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.7]); Mon, 12 Mar 2018 17:22:09 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.7]); Mon, 12 Mar 2018 17:22:09 +0000 (UTC) for IP:'10.11.54.4' DOMAIN:'int-mx04.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'dgilbert@redhat.com' RCPT:'' Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH v5 25/29] vhost+postcopy: Wire up POSTCOPY_END notify X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: aarcange@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Type: text/plain; charset="utf-8" From: "Dr. David Alan Gilbert" Wire up a call to VHOST_USER_POSTCOPY_END message to the vhost clients right before we ask the listener thread to shutdown. Signed-off-by: Dr. David Alan Gilbert Reviewed-by: Marc-Andr=C3=A9 Lureau --- hw/virtio/trace-events | 2 ++ hw/virtio/vhost-user.c | 34 ++++++++++++++++++++++++++++++++++ migration/postcopy-ram.c | 7 +++++++ migration/postcopy-ram.h | 1 + 4 files changed, 44 insertions(+) diff --git a/hw/virtio/trace-events b/hw/virtio/trace-events index fe5e0ff856..857c495e65 100644 --- a/hw/virtio/trace-events +++ b/hw/virtio/trace-events @@ -7,6 +7,8 @@ vhost_region_add_section_abut(const char *name, uint64_t ne= w_size) "%s: 0x%"PRIx vhost_section(const char *name, int r) "%s:%d" =20 # hw/virtio/vhost-user.c +vhost_user_postcopy_end_entry(void) "" +vhost_user_postcopy_end_exit(void) "" vhost_user_postcopy_fault_handler(const char *name, uint64_t fault_address= , int nregions) "%s: @0x%"PRIx64" nregions:%d" vhost_user_postcopy_fault_handler_loop(int i, uint64_t client_base, uint64= _t size) "%d: client 0x%"PRIx64" +0x%"PRIx64 vhost_user_postcopy_fault_handler_found(int i, uint64_t region_offset, uin= t64_t rb_offset) "%d: region_offset: 0x%"PRIx64" rb_offset:0x%"PRIx64 diff --git a/hw/virtio/vhost-user.c b/hw/virtio/vhost-user.c index 230f2f9d55..44aea5c0a8 100644 --- a/hw/virtio/vhost-user.c +++ b/hw/virtio/vhost-user.c @@ -1114,6 +1114,37 @@ static int vhost_user_postcopy_listen(struct vhost_d= ev *dev, Error **errp) return 0; } =20 +/* + * Called at the end of postcopy + */ +static int vhost_user_postcopy_end(struct vhost_dev *dev, Error **errp) +{ + VhostUserMsg msg =3D { + .hdr.request =3D VHOST_USER_POSTCOPY_END, + .hdr.flags =3D VHOST_USER_VERSION | VHOST_USER_NEED_REPLY_MASK, + }; + int ret; + struct vhost_user *u =3D dev->opaque; + + trace_vhost_user_postcopy_end_entry(); + if (vhost_user_write(dev, &msg, NULL, 0) < 0) { + error_setg(errp, "Failed to send postcopy_end to vhost"); + return -1; + } + + ret =3D process_message_reply(dev, &msg); + if (ret) { + error_setg(errp, "Failed to receive reply to postcopy_end"); + return ret; + } + postcopy_unregister_shared_ufd(&u->postcopy_fd); + u->postcopy_fd.handler =3D NULL; + + trace_vhost_user_postcopy_end_exit(); + + return 0; +} + static int vhost_user_postcopy_notifier(NotifierWithReturn *notifier, void *opaque) { @@ -1139,6 +1170,9 @@ static int vhost_user_postcopy_notifier(NotifierWithR= eturn *notifier, case POSTCOPY_NOTIFY_INBOUND_LISTEN: return vhost_user_postcopy_listen(dev, pnd->errp); =20 + case POSTCOPY_NOTIFY_INBOUND_END: + return vhost_user_postcopy_end(dev, pnd->errp); + default: /* We ignore notifications we don't know */ break; diff --git a/migration/postcopy-ram.c b/migration/postcopy-ram.c index 36db900e8f..1379923cfc 100644 --- a/migration/postcopy-ram.c +++ b/migration/postcopy-ram.c @@ -413,6 +413,13 @@ int postcopy_ram_incoming_cleanup(MigrationIncomingSta= te *mis) trace_postcopy_ram_incoming_cleanup_entry(); =20 if (mis->have_fault_thread) { + Error *local_err =3D NULL; + + if (postcopy_notify(POSTCOPY_NOTIFY_INBOUND_END, &local_err)) { + error_report_err(local_err); + return -1; + } + if (qemu_ram_foreach_block(cleanup_range, mis)) { return -1; } diff --git a/migration/postcopy-ram.h b/migration/postcopy-ram.h index 2c73d77a5c..d900d9c34f 100644 --- a/migration/postcopy-ram.h +++ b/migration/postcopy-ram.h @@ -132,6 +132,7 @@ enum PostcopyNotifyReason { POSTCOPY_NOTIFY_PROBE =3D 0, POSTCOPY_NOTIFY_INBOUND_ADVISE, POSTCOPY_NOTIFY_INBOUND_LISTEN, + POSTCOPY_NOTIFY_INBOUND_END, }; =20 struct PostcopyNotifyData { --=20 2.14.3 From nobody Sun May 5 21:21:29 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1520876823414816.9503523845887; Mon, 12 Mar 2018 10:47:03 -0700 (PDT) Received: from localhost ([::1]:33534 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1evRX8-0005qZ-1p for importer@patchew.org; Mon, 12 Mar 2018 13:46:50 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:53270) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1evR9I-000062-To for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:22:17 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1evR9H-0007u0-Oh for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:22:12 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:47230 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1evR9H-0007tk-JH for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:22:11 -0400 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id C5F3D406E97D for ; Mon, 12 Mar 2018 17:22:10 +0000 (UTC) Received: from dgilbert-t530.redhat.com (ovpn-117-169.ams2.redhat.com [10.36.117.169]) by smtp.corp.redhat.com (Postfix) with ESMTP id 8DD2B2026E03; Mon, 12 Mar 2018 17:22:09 +0000 (UTC) From: "Dr. David Alan Gilbert (git)" To: qemu-devel@nongnu.org, mst@redhat.com, maxime.coquelin@redhat.com, marcandre.lureau@redhat.com, peterx@redhat.com, quintela@redhat.com Date: Mon, 12 Mar 2018 17:21:21 +0000 Message-Id: <20180312172124.56461-27-dgilbert@redhat.com> In-Reply-To: <20180312172124.56461-1-dgilbert@redhat.com> References: <20180312172124.56461-1-dgilbert@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.4 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.7]); Mon, 12 Mar 2018 17:22:10 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.7]); Mon, 12 Mar 2018 17:22:10 +0000 (UTC) for IP:'10.11.54.4' DOMAIN:'int-mx04.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'dgilbert@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH v5 26/29] vhost: Huge page align and merge X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: aarcange@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: "Dr. David Alan Gilbert" Align RAMBlocks to page size alignment, and adjust the merging code to deal with partial overlap due to that alignment. This is needed for postcopy so that we can place/fetch whole hugepages when under userfault. Signed-off-by: Dr. David Alan Gilbert --- hw/virtio/trace-events | 3 ++- hw/virtio/vhost.c | 66 ++++++++++++++++++++++++++++++++++++++++++----= ---- 2 files changed, 58 insertions(+), 11 deletions(-) diff --git a/hw/virtio/trace-events b/hw/virtio/trace-events index 857c495e65..1422ff03ab 100644 --- a/hw/virtio/trace-events +++ b/hw/virtio/trace-events @@ -3,7 +3,8 @@ # hw/virtio/vhost.c vhost_commit(bool started, bool changed) "Started: %d Changed: %d" vhost_region_add_section(const char *name, uint64_t gpa, uint64_t size, ui= nt64_t host) "%s: 0x%"PRIx64"+0x%"PRIx64" @ 0x%"PRIx64 -vhost_region_add_section_abut(const char *name, uint64_t new_size) "%s: 0x= %"PRIx64 +vhost_region_add_section_merge(const char *name, uint64_t new_size, uint64= _t gpa, uint64_t owr) "%s: size: 0x%"PRIx64 " gpa: 0x%"PRIx64 " owr: 0x%"PR= Ix64 +vhost_region_add_section_aligned(const char *name, uint64_t gpa, uint64_t = size, uint64_t host) "%s: 0x%"PRIx64"+0x%"PRIx64" @ 0x%"PRIx64 vhost_section(const char *name, int r) "%s:%d" =20 # hw/virtio/vhost-user.c diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c index d8d0ef92e1..250f886acb 100644 --- a/hw/virtio/vhost.c +++ b/hw/virtio/vhost.c @@ -522,10 +522,28 @@ static void vhost_region_add_section(struct vhost_dev= *dev, uint64_t mrs_gpa =3D section->offset_within_address_space; uintptr_t mrs_host =3D (uintptr_t)memory_region_get_ram_ptr(section->m= r) + section->offset_within_region; + RAMBlock *mrs_rb =3D section->mr->ram_block; + size_t mrs_page =3D qemu_ram_pagesize(mrs_rb); =20 trace_vhost_region_add_section(section->mr->name, mrs_gpa, mrs_size, mrs_host); =20 + /* Round the section to it's page size */ + /* First align the start down to a page boundary */ + uint64_t alignage =3D mrs_host & (mrs_page - 1); + if (alignage) { + mrs_host -=3D alignage; + mrs_size +=3D alignage; + mrs_gpa -=3D alignage; + } + /* Now align the size up to a page boundary */ + alignage =3D mrs_size & (mrs_page - 1); + if (alignage) { + mrs_size +=3D mrs_page - alignage; + } + trace_vhost_region_add_section_aligned(section->mr->name, mrs_gpa, mrs= _size, + mrs_host); + if (dev->n_tmp_sections) { /* Since we already have at least one section, lets see if * this extends it; since we're scanning in order, we only @@ -542,18 +560,46 @@ static void vhost_region_add_section(struct vhost_dev= *dev, prev_sec->offset_within_region; uint64_t prev_host_end =3D range_get_last(prev_host_start, prev_= size); =20 - if (prev_gpa_end + 1 =3D=3D mrs_gpa && - prev_host_end + 1 =3D=3D mrs_host && - section->mr =3D=3D prev_sec->mr && - (!dev->vhost_ops->vhost_backend_can_merge || - dev->vhost_ops->vhost_backend_can_merge(dev, + if (mrs_gpa <=3D (prev_gpa_end + 1)) { + /* OK, looks like overlapping/intersecting - it's possible that + * the rounding to page sizes has made them overlap, but they = should + * match up in the same RAMBlock if they do. + */ + if (mrs_gpa < prev_gpa_start) { + error_report("%s:Section rounded to %"PRIx64 + " prior to previous %"PRIx64, + __func__, mrs_gpa, prev_gpa_start); + /* A way to cleanly fail here would be better */ + return; + } + /* Offset from the start of the previous GPA to this GPA */ + size_t offset =3D mrs_gpa - prev_gpa_start; + + if (prev_host_start + offset =3D=3D mrs_host && + section->mr =3D=3D prev_sec->mr && + (!dev->vhost_ops->vhost_backend_can_merge || + dev->vhost_ops->vhost_backend_can_merge(dev, mrs_host, mrs_size, prev_host_start, prev_size))) { - /* The two sections abut */ - need_add =3D false; - prev_sec->size =3D int128_add(prev_sec->size, section->size); - trace_vhost_region_add_section_abut(section->mr->name, - mrs_size + prev_size); + uint64_t max_end =3D MAX(prev_host_end, mrs_host + mrs_siz= e); + need_add =3D false; + prev_sec->offset_within_address_space =3D + MIN(prev_gpa_start, mrs_gpa); + prev_sec->offset_within_region =3D + MIN(prev_host_start, mrs_host) - + (uintptr_t)memory_region_get_ram_ptr(prev_sec->mr); + prev_sec->size =3D int128_make64(max_end - MIN(prev_host_s= tart, + mrs_host)); + trace_vhost_region_add_section_merge(section->mr->name, + int128_get64(prev_sec->size), + prev_sec->offset_within_address_sp= ace, + prev_sec->offset_within_region); + } else { + error_report("%s: Overlapping but not coherent sections " + "at %"PRIx64, + __func__, mrs_gpa); + return; + } } } =20 --=20 2.14.3 From nobody Sun May 5 21:21:29 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 152087646786210.898172336886546; Mon, 12 Mar 2018 10:41:07 -0700 (PDT) Received: from localhost ([::1]:33497 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1evRRb-0000RR-0w for importer@patchew.org; Mon, 12 Mar 2018 13:41:07 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:53290) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1evR9L-000093-UG for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:22:16 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1evR9I-0007ud-Po for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:22:15 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:57636 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1evR9I-0007uP-LJ for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:22:12 -0400 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 47D34401DE6B for ; Mon, 12 Mar 2018 17:22:12 +0000 (UTC) Received: from dgilbert-t530.redhat.com (ovpn-117-169.ams2.redhat.com [10.36.117.169]) by smtp.corp.redhat.com (Postfix) with ESMTP id 0FED4202322C; Mon, 12 Mar 2018 17:22:10 +0000 (UTC) From: "Dr. David Alan Gilbert (git)" To: qemu-devel@nongnu.org, mst@redhat.com, maxime.coquelin@redhat.com, marcandre.lureau@redhat.com, peterx@redhat.com, quintela@redhat.com Date: Mon, 12 Mar 2018 17:21:22 +0000 Message-Id: <20180312172124.56461-28-dgilbert@redhat.com> In-Reply-To: <20180312172124.56461-1-dgilbert@redhat.com> References: <20180312172124.56461-1-dgilbert@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.78 on 10.11.54.4 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.6]); Mon, 12 Mar 2018 17:22:12 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.6]); Mon, 12 Mar 2018 17:22:12 +0000 (UTC) for IP:'10.11.54.4' DOMAIN:'int-mx04.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'dgilbert@redhat.com' RCPT:'' Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH v5 27/29] postcopy: Allow shared memory X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: aarcange@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Type: text/plain; charset="utf-8" From: "Dr. David Alan Gilbert" Now that we have the mechanisms in here, allow shared memory in a postcopy. Note that QEMU can't tell who all the users of shared regions are and thus can't tell whether all the users of the shared regions have appropriate support for postcopy. Those devices that explicitly support shared memory (e.g. vhost-user) must check, but it doesn't stop weirder configurations causing problems. Signed-off-by: Dr. David Alan Gilbert Reviewed-by: Marc-Andr=C3=A9 Lureau --- migration/postcopy-ram.c | 6 ------ 1 file changed, 6 deletions(-) diff --git a/migration/postcopy-ram.c b/migration/postcopy-ram.c index 1379923cfc..964eb1c555 100644 --- a/migration/postcopy-ram.c +++ b/migration/postcopy-ram.c @@ -215,12 +215,6 @@ static int test_ramblock_postcopiable(const char *bloc= k_name, void *host_addr, RAMBlock *rb =3D qemu_ram_block_by_name(block_name); size_t pagesize =3D qemu_ram_pagesize(rb); =20 - if (qemu_ram_is_shared(rb)) { - error_report("Postcopy on shared RAM (%s) is not yet supported", - block_name); - return 1; - } - if (length % pagesize) { error_report("Postcopy requires RAM blocks to be a page size multi= ple," " block %s is 0x" RAM_ADDR_FMT " bytes with a " --=20 2.14.3 From nobody Sun May 5 21:21:29 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1520875942992111.4183745981278; Mon, 12 Mar 2018 10:32:22 -0700 (PDT) Received: from localhost ([::1]:33427 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1evRJ8-0000V5-7v for importer@patchew.org; Mon, 12 Mar 2018 13:32:22 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:53291) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1evR9L-000095-US for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:22:16 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1evR9K-0007vR-95 for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:22:15 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:47236 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1evR9K-0007v5-4C for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:22:14 -0400 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id B9C98406E97D for ; Mon, 12 Mar 2018 17:22:13 +0000 (UTC) Received: from dgilbert-t530.redhat.com (ovpn-117-169.ams2.redhat.com [10.36.117.169]) by smtp.corp.redhat.com (Postfix) with ESMTP id 82F95202322E; Mon, 12 Mar 2018 17:22:12 +0000 (UTC) From: "Dr. David Alan Gilbert (git)" To: qemu-devel@nongnu.org, mst@redhat.com, maxime.coquelin@redhat.com, marcandre.lureau@redhat.com, peterx@redhat.com, quintela@redhat.com Date: Mon, 12 Mar 2018 17:21:23 +0000 Message-Id: <20180312172124.56461-29-dgilbert@redhat.com> In-Reply-To: <20180312172124.56461-1-dgilbert@redhat.com> References: <20180312172124.56461-1-dgilbert@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.78 on 10.11.54.4 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.7]); Mon, 12 Mar 2018 17:22:13 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.7]); Mon, 12 Mar 2018 17:22:13 +0000 (UTC) for IP:'10.11.54.4' DOMAIN:'int-mx04.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'dgilbert@redhat.com' RCPT:'' Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH v5 28/29] libvhost-user: Claim support for postcopy X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: aarcange@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Type: text/plain; charset="utf-8" From: "Dr. David Alan Gilbert" Tell QEMU we understand the protocol features needed for postcopy. Signed-off-by: Dr. David Alan Gilbert Reviewed-by: Marc-Andr=C3=A9 Lureau --- contrib/libvhost-user/libvhost-user.c | 33 +++++++++++++++++++++++++++++++= ++ 1 file changed, 33 insertions(+) diff --git a/contrib/libvhost-user/libvhost-user.c b/contrib/libvhost-user/= libvhost-user.c index 504ff5ea59..beeed0c43f 100644 --- a/contrib/libvhost-user/libvhost-user.c +++ b/contrib/libvhost-user/libvhost-user.c @@ -185,6 +185,35 @@ vmsg_close_fds(VhostUserMsg *vmsg) } } =20 +/* A test to see if we have userfault available */ +static bool +have_userfault(void) +{ +#if defined(__linux__) && defined(__NR_userfaultfd) &&\ + defined(UFFD_FEATURE_MISSING_SHMEM) &&\ + defined(UFFD_FEATURE_MISSING_HUGETLBFS) + /* Now test the kernel we're running on really has the features */ + int ufd =3D syscall(__NR_userfaultfd, O_CLOEXEC | O_NONBLOCK); + struct uffdio_api api_struct; + if (ufd < 0) { + return false; + } + + api_struct.api =3D UFFD_API; + api_struct.features =3D UFFD_FEATURE_MISSING_SHMEM | + UFFD_FEATURE_MISSING_HUGETLBFS; + if (ioctl(ufd, UFFDIO_API, &api_struct)) { + close(ufd); + return false; + } + close(ufd); + return true; + +#else + return false; +#endif +} + static bool vu_message_read(VuDev *dev, int conn_fd, VhostUserMsg *vmsg) { @@ -939,6 +968,10 @@ vu_get_protocol_features_exec(VuDev *dev, VhostUserMsg= *vmsg) uint64_t features =3D 1ULL << VHOST_USER_PROTOCOL_F_LOG_SHMFD | 1ULL << VHOST_USER_PROTOCOL_F_SLAVE_REQ; =20 + if (have_userfault()) { + features |=3D 1ULL << VHOST_USER_PROTOCOL_F_PAGEFAULT; + } + if (dev->iface->get_protocol_features) { features |=3D dev->iface->get_protocol_features(dev); } --=20 2.14.3 From nobody Sun May 5 21:21:29 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1520876627824426.5418661563151; Mon, 12 Mar 2018 10:43:47 -0700 (PDT) Received: from localhost ([::1]:33509 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1evRUB-0003Gh-3U for importer@patchew.org; Mon, 12 Mar 2018 13:43:47 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:53307) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1evR9M-0000A4-Ni for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:22:17 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1evR9L-0007wC-Og for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:22:16 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:38232 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1evR9L-0007vv-Im for qemu-devel@nongnu.org; Mon, 12 Mar 2018 13:22:15 -0400 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 399E476FB9 for ; Mon, 12 Mar 2018 17:22:15 +0000 (UTC) Received: from dgilbert-t530.redhat.com (ovpn-117-169.ams2.redhat.com [10.36.117.169]) by smtp.corp.redhat.com (Postfix) with ESMTP id 012462026E03; Mon, 12 Mar 2018 17:22:13 +0000 (UTC) From: "Dr. David Alan Gilbert (git)" To: qemu-devel@nongnu.org, mst@redhat.com, maxime.coquelin@redhat.com, marcandre.lureau@redhat.com, peterx@redhat.com, quintela@redhat.com Date: Mon, 12 Mar 2018 17:21:24 +0000 Message-Id: <20180312172124.56461-30-dgilbert@redhat.com> In-Reply-To: <20180312172124.56461-1-dgilbert@redhat.com> References: <20180312172124.56461-1-dgilbert@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.78 on 10.11.54.4 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.1]); Mon, 12 Mar 2018 17:22:15 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.1]); Mon, 12 Mar 2018 17:22:15 +0000 (UTC) for IP:'10.11.54.4' DOMAIN:'int-mx04.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'dgilbert@redhat.com' RCPT:'' Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH v5 29/29] postcopy shared docs X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: aarcange@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Type: text/plain; charset="utf-8" From: "Dr. David Alan Gilbert" Add some notes to the migration documentation for shared memory postcopy. Signed-off-by: Dr. David Alan Gilbert Reviewed-by: Marc-Andr=C3=A9 Lureau --- docs/devel/migration.rst | 41 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+) diff --git a/docs/devel/migration.rst b/docs/devel/migration.rst index 9d1b7657f0..e32b087f6e 100644 --- a/docs/devel/migration.rst +++ b/docs/devel/migration.rst @@ -577,3 +577,44 @@ Postcopy now works with hugetlbfs backed memory: hugepages works well, however 1GB hugepages are likely to be problema= tic since it takes ~1 second to transfer a 1GB hugepage across a 10Gbps l= ink, and until the full page is transferred the destination thread is bloc= ked. + +Postcopy with shared memory +--------------------------- + +Postcopy migration with shared memory needs explicit support from the other +processes that share memory and from QEMU. There are restrictions on the t= ype of +memory that userfault can support shared. + +The Linux kernel userfault support works on `/dev/shm` memory and on `huge= tlbfs` +(although the kernel doesn't provide an equivalent to `madvise(MADV_DONTNE= ED)` +for hugetlbfs which may be a problem in some configurations). + +The vhost-user code in QEMU supports clients that have Postcopy support, +and the `vhost-user-bridge` (in `tests/`) and the DPDK package have changes +to support postcopy. + +The client needs to open a userfaultfd and register the areas +of memory that it maps with userfault. The client must then pass the +userfaultfd back to QEMU together with a mapping table that allows +fault addresses in the clients address space to be converted back to +RAMBlock/offsets. The client's userfaultfd is added to the postcopy +fault-thread and page requests are made on behalf of the client by QEMU. +QEMU performs 'wake' operations on the client's userfaultfd to allow it +to continue after a page has arrived. + +.. note:: + There are two future improvements that would be nice: + a) Some way to make QEMU ignorant of the addresses in the clients + address space + b) Avoiding the need for QEMU to perform ufd-wake calls after the + pages have arrived + +Retro-fitting postcopy to existing clients is possible: + a) A mechanism is needed for the registration with userfault as above, + and the registration needs to be coordinated with the phases of + postcopy. In vhost-user extra messages are added to the existing + control channel. + b) Any thread that can block due to guest memory accesses must be + identified and the implication understood; for example if the + guest memory access is made while holding a lock then all other + threads waiting for that lock will also be blocked. --=20 2.14.3