From nobody Thu May 2 16:10:00 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1503603672642450.272393747994; Thu, 24 Aug 2017 12:41:12 -0700 (PDT) Received: from localhost ([::1]:50173 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dky07-000556-Ap for importer@patchew.org; Thu, 24 Aug 2017 15:41:11 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52007) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dkxnA-0001f7-S6 for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:27:49 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dkxn7-00048D-PL for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:27:48 -0400 Received: from mx1.redhat.com ([209.132.183.28]:42298) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dkxn7-00047g-I6 for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:27:45 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 8CF9013A82; Thu, 24 Aug 2017 19:27:44 +0000 (UTC) Received: from dgilbert-t530.redhat.com (ovpn-117-165.ams2.redhat.com [10.36.117.165]) by smtp.corp.redhat.com (Postfix) with ESMTP id 4CAE017CC4; Thu, 24 Aug 2017 19:27:42 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 8CF9013A82 Authentication-Results: ext-mx05.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx05.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=dgilbert@redhat.com From: "Dr. David Alan Gilbert (git)" To: qemu-devel@nongnu.org, maxime.coquelin@redhat.com, a.perevalov@samsung.com, mst@redhat.com, marcandre.lureau@redhat.com Date: Thu, 24 Aug 2017 20:26:59 +0100 Message-Id: <20170824192730.8440-2-dgilbert@redhat.com> In-Reply-To: <20170824192730.8440-1-dgilbert@redhat.com> References: <20170824192730.8440-1-dgilbert@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.29]); Thu, 24 Aug 2017 19:27:44 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [RFC v2 01/32] vhu: vu_queue_started X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lvivier@redhat.com, aarcange@redhat.com, felipe@nutanix.com, peterx@redhat.com, quintela@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: "Dr. David Alan Gilbert" Add a vu_queue_started method to complement vu_queue_enabled. Signed-off-by: Dr. David Alan Gilbert Reviewed-by: Marc-Andr=C3=A9 Lureau --- contrib/libvhost-user/libvhost-user.c | 6 ++++++ contrib/libvhost-user/libvhost-user.h | 9 +++++++++ 2 files changed, 15 insertions(+) diff --git a/contrib/libvhost-user/libvhost-user.c b/contrib/libvhost-user/= libvhost-user.c index 35fa0c5e56..201b9846e9 100644 --- a/contrib/libvhost-user/libvhost-user.c +++ b/contrib/libvhost-user/libvhost-user.c @@ -930,6 +930,12 @@ vu_queue_enabled(VuDev *dev, VuVirtq *vq) return vq->enable; } =20 +bool +vu_queue_started(VuDev *dev, VuVirtq *vq) +{ + return vq->started; +} + static inline uint16_t vring_avail_flags(VuVirtq *vq) { diff --git a/contrib/libvhost-user/libvhost-user.h b/contrib/libvhost-user/= libvhost-user.h index 53ef222c0b..acd019876d 100644 --- a/contrib/libvhost-user/libvhost-user.h +++ b/contrib/libvhost-user/libvhost-user.h @@ -328,6 +328,15 @@ void vu_queue_set_notification(VuDev *dev, VuVirtq *vq= , int enable); bool vu_queue_enabled(VuDev *dev, VuVirtq *vq); =20 /** + * vu_queue_started: + * @dev: a VuDev context + * @vq: a VuVirtq queue + * + * Returns: whether the queue is started. + */ +bool vu_queue_started(VuDev *dev, VuVirtq *vq); + +/** * vu_queue_empty: * @dev: a VuDev context * @vq: a VuVirtq queue --=20 2.13.5 From nobody Thu May 2 16:10:00 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1503603128874555.3620794379246; Thu, 24 Aug 2017 12:32:08 -0700 (PDT) Received: from localhost ([::1]:50126 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dkxrL-0005PA-Mh for importer@patchew.org; Thu, 24 Aug 2017 15:32:07 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52009) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dkxnA-0001fB-T8 for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:27:49 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dkxnA-00049q-2V for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:27:48 -0400 Received: from mx1.redhat.com ([209.132.183.28]:53403) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dkxn9-00049I-St for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:27:48 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id EBEE183F40; Thu, 24 Aug 2017 19:27:46 +0000 (UTC) Received: from dgilbert-t530.redhat.com (ovpn-117-165.ams2.redhat.com [10.36.117.165]) by smtp.corp.redhat.com (Postfix) with ESMTP id DA6CE17CC4; Thu, 24 Aug 2017 19:27:44 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com EBEE183F40 Authentication-Results: ext-mx03.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx03.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=dgilbert@redhat.com From: "Dr. David Alan Gilbert (git)" To: qemu-devel@nongnu.org, maxime.coquelin@redhat.com, a.perevalov@samsung.com, mst@redhat.com, marcandre.lureau@redhat.com Date: Thu, 24 Aug 2017 20:27:00 +0100 Message-Id: <20170824192730.8440-3-dgilbert@redhat.com> In-Reply-To: <20170824192730.8440-1-dgilbert@redhat.com> References: <20170824192730.8440-1-dgilbert@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.27]); Thu, 24 Aug 2017 19:27:47 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [RFC v2 02/32] vhub: Only process received packets on started queues X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lvivier@redhat.com, aarcange@redhat.com, felipe@nutanix.com, peterx@redhat.com, quintela@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: "Dr. David Alan Gilbert" Only process received packets if the queue has been started. Signed-off-by: Dr. David Alan Gilbert Reviewed-by: Marc-Andr=C3=A9 Lureau --- tests/vhost-user-bridge.c | 1 + 1 file changed, 1 insertion(+) diff --git a/tests/vhost-user-bridge.c b/tests/vhost-user-bridge.c index 1e5b5ca3da..324abee53d 100644 --- a/tests/vhost-user-bridge.c +++ b/tests/vhost-user-bridge.c @@ -277,6 +277,7 @@ vubr_backend_recv_cb(int sock, void *ctx) DPRINT(" hdrlen =3D %d\n", hdrlen); =20 if (!vu_queue_enabled(dev, vq) || + !vu_queue_started(dev, vq) || !vu_queue_avail_bytes(dev, vq, hdrlen, 0)) { DPRINT("Got UDP packet, but no available descriptors on RX virtq.\= n"); return; --=20 2.13.5 From nobody Thu May 2 16:10:00 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1503603826009576.0300977033254; Thu, 24 Aug 2017 12:43:46 -0700 (PDT) Received: from localhost ([::1]:50184 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dky2a-0007s8-SF for importer@patchew.org; Thu, 24 Aug 2017 15:43:44 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52036) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dkxnD-0001i7-Js for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:27:52 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dkxnC-0004BT-Kn for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:27:51 -0400 Received: from mx1.redhat.com ([209.132.183.28]:10831) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dkxnC-0004Aj-Bi for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:27:50 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 546BD81235; Thu, 24 Aug 2017 19:27:49 +0000 (UTC) Received: from dgilbert-t530.redhat.com (ovpn-117-165.ams2.redhat.com [10.36.117.165]) by smtp.corp.redhat.com (Postfix) with ESMTP id 407CA7E491; Thu, 24 Aug 2017 19:27:47 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 546BD81235 Authentication-Results: ext-mx01.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx01.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=dgilbert@redhat.com From: "Dr. David Alan Gilbert (git)" To: qemu-devel@nongnu.org, maxime.coquelin@redhat.com, a.perevalov@samsung.com, mst@redhat.com, marcandre.lureau@redhat.com Date: Thu, 24 Aug 2017 20:27:01 +0100 Message-Id: <20170824192730.8440-4-dgilbert@redhat.com> In-Reply-To: <20170824192730.8440-1-dgilbert@redhat.com> References: <20170824192730.8440-1-dgilbert@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.25]); Thu, 24 Aug 2017 19:27:49 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [RFC v2 03/32] migrate: Update ram_block_discard_range for shared X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lvivier@redhat.com, aarcange@redhat.com, felipe@nutanix.com, peterx@redhat.com, quintela@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: "Dr. David Alan Gilbert" The choice of call to discard a block is getting more complicated for other cases. We use fallocate PUNCH_HOLE in any file cases; it works for both hugepage and for tmpfs. We use the DONTNEED for non-hugepage cases either where they're anonymous or where they're private. Care should be taken when trying other backing files. Signed-off-by: Dr. David Alan Gilbert Reviewed-by: Peter Xu --- exec.c | 35 ++++++++++++++++++++++++----------- trace-events | 3 +++ 2 files changed, 27 insertions(+), 11 deletions(-) diff --git a/exec.c b/exec.c index d20c34ca83..67df2909ce 100644 --- a/exec.c +++ b/exec.c @@ -3573,6 +3573,7 @@ int ram_block_discard_range(RAMBlock *rb, uint64_t st= art, size_t length) } =20 if ((start + length) <=3D rb->used_length) { + bool need_madvise, need_fallocate; uint8_t *host_endaddr =3D host_startaddr + length; if ((uintptr_t)host_endaddr & (rb->page_size - 1)) { error_report("ram_block_discard_range: Unaligned end address: = %p", @@ -3582,23 +3583,35 @@ int ram_block_discard_range(RAMBlock *rb, uint64_t = start, size_t length) =20 errno =3D ENOTSUP; /* If we are missing MADVISE etc */ =20 - if (rb->page_size =3D=3D qemu_host_page_size) { -#if defined(CONFIG_MADVISE) - /* Note: We need the madvise MADV_DONTNEED behaviour of defini= tely - * freeing the page. - */ - ret =3D madvise(host_startaddr, length, MADV_DONTNEED); -#endif - } else { - /* Huge page case - unfortunately it can't do DONTNEED, but - * it can do the equivalent by FALLOC_FL_PUNCH_HOLE in the - * huge page file. + /* The logic here is messy; + * madvise DONTNEED fails for hugepages + * fallocate works on hugepages and shmem + */ + need_madvise =3D (rb->page_size =3D=3D qemu_host_page_size); + need_fallocate =3D rb->fd !=3D -1; + if (need_fallocate) { + /* For a file, this causes the area of the file to be zero'd + * if read, and for hugetlbfs also causes it to be unmapped + * so a userfault will trigger. */ #ifdef CONFIG_FALLOCATE_PUNCH_HOLE ret =3D fallocate(rb->fd, FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEE= P_SIZE, start, length); #endif } + /* i.e. need madvise but skip it if the fallocate failed */ + if (need_madvise && (!need_fallocate || (ret =3D=3D 0))) { + /* For normal RAM this causes it to be unmapped, + * for shared memory it causes the local mapping to disappear + * and to fall back on the file contents (which we just + * fallocate'd away). + */ +#if defined(CONFIG_MADVISE) + ret =3D madvise(host_startaddr, length, MADV_DONTNEED); +#endif + } + trace_ram_block_discard_range(rb->idstr, host_startaddr, + need_madvise, need_fallocate, ret); if (ret) { ret =3D -errno; error_report("ram_block_discard_range: Failed to discard range= " diff --git a/trace-events b/trace-events index 1f50f56d9d..213ee34f89 100644 --- a/trace-events +++ b/trace-events @@ -55,6 +55,9 @@ dma_complete(void *dbs, int ret, void *cb) "dbs=3D%p ret= =3D%d cb=3D%p" dma_blk_cb(void *dbs, int ret) "dbs=3D%p ret=3D%d" dma_map_wait(void *dbs) "dbs=3D%p" =20 +# exec.c +ram_block_discard_range(const char *rbname, void *hva, bool need_madvise, = bool need_fallocate, int ret) "%s@%p: madvise: %d fallocate: %d ret: %d" + # memory.c memory_region_ops_read(int cpu_index, void *mr, uint64_t addr, uint64_t va= lue, unsigned size) "cpu %d mr %p addr 0x%"PRIx64" value 0x%"PRIx64" size %= u" memory_region_ops_write(int cpu_index, void *mr, uint64_t addr, uint64_t v= alue, unsigned size) "cpu %d mr %p addr 0x%"PRIx64" value 0x%"PRIx64" size = %u" --=20 2.13.5 From nobody Thu May 2 16:10:00 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 150360299063787.0505862594323; Thu, 24 Aug 2017 12:29:50 -0700 (PDT) Received: from localhost ([::1]:50112 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dkxp7-0003BG-Dt for importer@patchew.org; Thu, 24 Aug 2017 15:29:49 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52084) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dkxnQ-0001uN-7M for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:28:05 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dkxnN-0004IB-2i for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:28:04 -0400 Received: from mx1.redhat.com ([209.132.183.28]:45480) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dkxnM-0004GU-Sz for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:28:01 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id D9FCAC04B941; Thu, 24 Aug 2017 19:27:59 +0000 (UTC) Received: from dgilbert-t530.redhat.com (ovpn-117-165.ams2.redhat.com [10.36.117.165]) by smtp.corp.redhat.com (Postfix) with ESMTP id 9F1A57F8C3; Thu, 24 Aug 2017 19:27:49 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com D9FCAC04B941 Authentication-Results: ext-mx07.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx07.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=dgilbert@redhat.com From: "Dr. David Alan Gilbert (git)" To: qemu-devel@nongnu.org, maxime.coquelin@redhat.com, a.perevalov@samsung.com, mst@redhat.com, marcandre.lureau@redhat.com Date: Thu, 24 Aug 2017 20:27:02 +0100 Message-Id: <20170824192730.8440-5-dgilbert@redhat.com> In-Reply-To: <20170824192730.8440-1-dgilbert@redhat.com> References: <20170824192730.8440-1-dgilbert@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.31]); Thu, 24 Aug 2017 19:28:00 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [RFC v2 04/32] qemu_ram_block_host_offset X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lvivier@redhat.com, aarcange@redhat.com, felipe@nutanix.com, peterx@redhat.com, quintela@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: "Dr. David Alan Gilbert" Utility to give the offset of a host pointer within a RAMBlock (assuming we already know it's in that RAMBlock) Signed-off-by: Dr. David Alan Gilbert Reviewed-by: Peter Xu --- exec.c | 10 ++++++++++ include/exec/cpu-common.h | 1 + 2 files changed, 11 insertions(+) diff --git a/exec.c b/exec.c index 67df2909ce..35b4cea2ed 100644 --- a/exec.c +++ b/exec.c @@ -2231,6 +2231,16 @@ static void *qemu_ram_ptr_length(RAMBlock *ram_block= , ram_addr_t addr, return ramblock_ptr(block, addr); } =20 +/* Return the offset of a hostpointer within a ramblock */ +ram_addr_t qemu_ram_block_host_offset(RAMBlock *rb, void *host) +{ + ram_addr_t res =3D (uint8_t *)host - (uint8_t *)rb->host; + assert((uint8_t *)host >=3D (uint8_t *)rb->host); + assert(res < rb->max_length); + + return res; +} + /* * Translates a host ptr back to a RAMBlock, a ram_addr and an offset * in that RAMBlock. diff --git a/include/exec/cpu-common.h b/include/exec/cpu-common.h index 74341b19d2..0d861a6289 100644 --- a/include/exec/cpu-common.h +++ b/include/exec/cpu-common.h @@ -68,6 +68,7 @@ ram_addr_t qemu_ram_addr_from_host(void *ptr); RAMBlock *qemu_ram_block_by_name(const char *name); RAMBlock *qemu_ram_block_from_host(void *ptr, bool round_offset, ram_addr_t *offset); +ram_addr_t qemu_ram_block_host_offset(RAMBlock *rb, void *host); void qemu_ram_set_idstr(RAMBlock *block, const char *name, DeviceState *de= v); void qemu_ram_unset_idstr(RAMBlock *block); const char *qemu_ram_get_idstr(RAMBlock *rb); --=20 2.13.5 From nobody Thu May 2 16:10:00 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 15036033042583.131056468429165; Thu, 24 Aug 2017 12:35:04 -0700 (PDT) Received: from localhost ([::1]:50139 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dkxu9-00081D-Tu for importer@patchew.org; Thu, 24 Aug 2017 15:35:02 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52086) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dkxnQ-0001vR-F7 for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:28:05 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dkxnP-0004L1-Hy for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:28:04 -0400 Received: from mx1.redhat.com ([209.132.183.28]:40596) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dkxnP-0004KW-Br for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:28:03 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 55DC57E44A; Thu, 24 Aug 2017 19:28:02 +0000 (UTC) Received: from dgilbert-t530.redhat.com (ovpn-117-165.ams2.redhat.com [10.36.117.165]) by smtp.corp.redhat.com (Postfix) with ESMTP id 30D787F383; Thu, 24 Aug 2017 19:28:00 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 55DC57E44A Authentication-Results: ext-mx03.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx03.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=dgilbert@redhat.com From: "Dr. David Alan Gilbert (git)" To: qemu-devel@nongnu.org, maxime.coquelin@redhat.com, a.perevalov@samsung.com, mst@redhat.com, marcandre.lureau@redhat.com Date: Thu, 24 Aug 2017 20:27:03 +0100 Message-Id: <20170824192730.8440-6-dgilbert@redhat.com> In-Reply-To: <20170824192730.8440-1-dgilbert@redhat.com> References: <20170824192730.8440-1-dgilbert@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.27]); Thu, 24 Aug 2017 19:28:02 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [RFC v2 05/32] migration/ram: ramblock_recv_bitmap_test_byte_offset X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lvivier@redhat.com, aarcange@redhat.com, felipe@nutanix.com, peterx@redhat.com, quintela@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: "Dr. David Alan Gilbert" Utility for testing the map when you already know the offset in the RAMBlock. Signed-off-by: Dr. David Alan Gilbert --- migration/ram.c | 5 +++++ migration/ram.h | 1 + 2 files changed, 6 insertions(+) diff --git a/migration/ram.c b/migration/ram.c index affb20cb5a..fbb874fb83 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -164,6 +164,11 @@ int ramblock_recv_bitmap_test(RAMBlock *rb, void *host= _addr) rb->receivedmap); } =20 +bool ramblock_recv_bitmap_test_byte_offset(RAMBlock *rb, uint64_t byte_off= set) +{ + return test_bit(byte_offset >> TARGET_PAGE_BITS, rb->receivedmap); +} + void ramblock_recv_bitmap_set(RAMBlock *rb, void *host_addr) { set_bit_atomic(ramblock_recv_bitmap_offset(host_addr, rb), rb->receive= dmap); diff --git a/migration/ram.h b/migration/ram.h index 4db992298a..8720b9de73 100644 --- a/migration/ram.h +++ b/migration/ram.h @@ -54,6 +54,7 @@ int ram_postcopy_incoming_init(MigrationIncomingState *mi= s); void ram_handle_compressed(void *host, uint8_t ch, uint64_t size); =20 int ramblock_recv_bitmap_test(RAMBlock *rb, void *host_addr); +bool ramblock_recv_bitmap_test_byte_offset(RAMBlock *rb, uint64_t byte_off= set); void ramblock_recv_bitmap_set(RAMBlock *rb, void *host_addr); void ramblock_recv_bitmap_set_range(RAMBlock *rb, void *host_addr, size_t = nr); void ramblock_recv_bitmap_clear(RAMBlock *rb, void *host_addr); --=20 2.13.5 From nobody Thu May 2 16:10:00 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1503603153868173.89202829360363; Thu, 24 Aug 2017 12:32:33 -0700 (PDT) Received: from localhost ([::1]:50127 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dkxrk-0005lh-EI for importer@patchew.org; Thu, 24 Aug 2017 15:32:32 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52108) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dkxnT-0001yF-9C for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:28:08 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dkxnS-0004Mq-50 for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:28:07 -0400 Received: from mx1.redhat.com ([209.132.183.28]:52426) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dkxnR-0004Lu-Rh for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:28:06 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id CCE25C058EA3; Thu, 24 Aug 2017 19:28:04 +0000 (UTC) Received: from dgilbert-t530.redhat.com (ovpn-117-165.ams2.redhat.com [10.36.117.165]) by smtp.corp.redhat.com (Postfix) with ESMTP id A4E847F383; Thu, 24 Aug 2017 19:28:02 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com CCE25C058EA3 Authentication-Results: ext-mx08.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx08.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=dgilbert@redhat.com From: "Dr. David Alan Gilbert (git)" To: qemu-devel@nongnu.org, maxime.coquelin@redhat.com, a.perevalov@samsung.com, mst@redhat.com, marcandre.lureau@redhat.com Date: Thu, 24 Aug 2017 20:27:04 +0100 Message-Id: <20170824192730.8440-7-dgilbert@redhat.com> In-Reply-To: <20170824192730.8440-1-dgilbert@redhat.com> References: <20170824192730.8440-1-dgilbert@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.32]); Thu, 24 Aug 2017 19:28:05 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [RFC v2 06/32] postcopy: use UFFDIO_ZEROPAGE only when available X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lvivier@redhat.com, aarcange@redhat.com, felipe@nutanix.com, peterx@redhat.com, quintela@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: "Dr. David Alan Gilbert" Use a flag on the RAMBlock to state whether it has the UFFDIO_ZEROPAGE capability, use it when it's available. This allows the use of postcopy on tmpfs as well as hugepage backed files. Signed-off-by: Dr. David Alan Gilbert --- exec.c | 15 +++++++++++++++ include/exec/cpu-common.h | 3 +++ migration/postcopy-ram.c | 14 +++++++++++--- 3 files changed, 29 insertions(+), 3 deletions(-) diff --git a/exec.c b/exec.c index 35b4cea2ed..80c3d1d121 100644 --- a/exec.c +++ b/exec.c @@ -103,6 +103,11 @@ static MemoryRegion io_mem_unassigned; */ #define RAM_RESIZEABLE (1 << 2) =20 +/* UFFDIO_ZEROPAGE is available on this RAMBlock to atomically + * zero the page and wake waiting processes. + * (Set during postcopy) + */ +#define RAM_UF_ZEROPAGE (1 << 3) #endif =20 #ifdef TARGET_PAGE_BITS_VARY @@ -1705,6 +1710,16 @@ bool qemu_ram_is_shared(RAMBlock *rb) return rb->flags & RAM_SHARED; } =20 +bool qemu_ram_is_uf_zeroable(RAMBlock *rb) +{ + return rb->flags & RAM_UF_ZEROPAGE; +} + +void qemu_ram_set_uf_zeroable(RAMBlock *rb) +{ + rb->flags |=3D RAM_UF_ZEROPAGE; +} + /* Called with iothread lock held. */ void qemu_ram_set_idstr(RAMBlock *new_block, const char *name, DeviceState= *dev) { diff --git a/include/exec/cpu-common.h b/include/exec/cpu-common.h index 0d861a6289..24d335f95d 100644 --- a/include/exec/cpu-common.h +++ b/include/exec/cpu-common.h @@ -73,6 +73,9 @@ void qemu_ram_set_idstr(RAMBlock *block, const char *name= , DeviceState *dev); void qemu_ram_unset_idstr(RAMBlock *block); const char *qemu_ram_get_idstr(RAMBlock *rb); bool qemu_ram_is_shared(RAMBlock *rb); +bool qemu_ram_is_uf_zeroable(RAMBlock *rb); +void qemu_ram_set_uf_zeroable(RAMBlock *rb); + size_t qemu_ram_pagesize(RAMBlock *block); size_t qemu_ram_pagesize_largest(void); =20 diff --git a/migration/postcopy-ram.c b/migration/postcopy-ram.c index 7a414ebad8..640b72d86d 100644 --- a/migration/postcopy-ram.c +++ b/migration/postcopy-ram.c @@ -408,6 +408,11 @@ static int ram_block_enable_notify(const char *block_n= ame, void *host_addr, error_report("%s userfault: Region doesn't support COPY", __func__= ); return -1; } + if (reg_struct.ioctls & ((__u64)1 << _UFFDIO_ZEROPAGE)) { + RAMBlock *rb =3D qemu_ram_block_by_name(block_name); + qemu_ram_set_uf_zeroable(rb); + } + =20 return 0; } @@ -617,11 +622,14 @@ int postcopy_place_page(MigrationIncomingState *mis, = void *host, void *from, int postcopy_place_page_zero(MigrationIncomingState *mis, void *host, RAMBlock *rb) { + size_t pagesize =3D qemu_ram_pagesize(rb); trace_postcopy_place_page_zero(host); =20 - if (qemu_ram_pagesize(rb) =3D=3D getpagesize()) { - if (qemu_ufd_copy_ioctl(mis->userfault_fd, host, NULL, getpagesize= (), - rb)) { + /* Normal RAMBlocks can zero a page using UFFDIO_ZEROPAGE + * but it's not available for everything (e.g. hugetlbpages) + */ + if (qemu_ram_is_uf_zeroable(rb)) { + if (qemu_ufd_copy_ioctl(mis->userfault_fd, host, NULL, pagesize, r= b)) { int e =3D errno; error_report("%s: %s zero host: %p", __func__, strerror(e), host); --=20 2.13.5 From nobody Thu May 2 16:10:00 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 150360404679178.0112221790614; Thu, 24 Aug 2017 12:47:26 -0700 (PDT) Received: from localhost ([::1]:50207 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dky64-000389-FE for importer@patchew.org; Thu, 24 Aug 2017 15:47:20 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52121) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dkxnW-00021L-0u for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:28:13 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dkxnU-0004QH-Mp for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:28:10 -0400 Received: from mx1.redhat.com ([209.132.183.28]:49376) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dkxnU-0004P6-DM for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:28:08 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 5D9424E33B; Thu, 24 Aug 2017 19:28:07 +0000 (UTC) Received: from dgilbert-t530.redhat.com (ovpn-117-165.ams2.redhat.com [10.36.117.165]) by smtp.corp.redhat.com (Postfix) with ESMTP id 1C51017CC4; Thu, 24 Aug 2017 19:28:04 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 5D9424E33B Authentication-Results: ext-mx09.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx09.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=dgilbert@redhat.com From: "Dr. David Alan Gilbert (git)" To: qemu-devel@nongnu.org, maxime.coquelin@redhat.com, a.perevalov@samsung.com, mst@redhat.com, marcandre.lureau@redhat.com Date: Thu, 24 Aug 2017 20:27:05 +0100 Message-Id: <20170824192730.8440-8-dgilbert@redhat.com> In-Reply-To: <20170824192730.8440-1-dgilbert@redhat.com> References: <20170824192730.8440-1-dgilbert@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.38]); Thu, 24 Aug 2017 19:28:07 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [RFC v2 07/32] postcopy: Add notifier chain X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lvivier@redhat.com, aarcange@redhat.com, felipe@nutanix.com, peterx@redhat.com, quintela@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: "Dr. David Alan Gilbert" Add a notifier chain for postcopy with a 'reason' flag and an opportunity for a notifier member to return an error. Call it when enabling postcopy. This will initially used to enable devices to declare they're unable to postcopy and later to notify of devices of stages within postcopy. Signed-off-by: Dr. David Alan Gilbert Reviewed-by: Peter Xu --- migration/postcopy-ram.c | 41 +++++++++++++++++++++++++++++++++++++++++ migration/postcopy-ram.h | 26 ++++++++++++++++++++++++++ vl.c | 2 ++ 3 files changed, 69 insertions(+) diff --git a/migration/postcopy-ram.c b/migration/postcopy-ram.c index 640b72d86d..95007c00ef 100644 --- a/migration/postcopy-ram.c +++ b/migration/postcopy-ram.c @@ -23,6 +23,8 @@ #include "savevm.h" #include "postcopy-ram.h" #include "ram.h" +#include "qapi/error.h" +#include "qemu/notify.h" #include "sysemu/sysemu.h" #include "sysemu/balloon.h" #include "qemu/error-report.h" @@ -45,6 +47,38 @@ struct PostcopyDiscardState { unsigned int nsentcmds; }; =20 +/* A notifier chain for postcopy + * The notifier should return 0 if it's OK, or a + * -errno on error. + * The notifier should expect an Error ** as it's data + */ +static NotifierWithReturnList postcopy_notifier_list; + +void postcopy_infrastructure_init(void) +{ + notifier_with_return_list_init(&postcopy_notifier_list); +} + +void postcopy_add_notifier(NotifierWithReturn *nn) +{ + notifier_with_return_list_add(&postcopy_notifier_list, nn); +} + +void postcopy_remove_notifier(NotifierWithReturn *n) +{ + notifier_with_return_remove(n); +} + +int postcopy_notify(enum PostcopyNotifyReason reason, Error **errp) +{ + struct PostcopyNotifyData pnd; + pnd.reason =3D reason; + pnd.errp =3D errp; + + return notifier_with_return_list_notify(&postcopy_notifier_list, + &pnd); +} + /* Postcopy needs to detect accesses to pages that haven't yet been copied * across, and efficiently map new pages in, the techniques for doing this * are target OS specific. @@ -133,6 +167,7 @@ bool postcopy_ram_supported_by_host(void) struct uffdio_register reg_struct; struct uffdio_range range_struct; uint64_t feature_mask; + Error *local_err =3D NULL; =20 if (qemu_target_page_size() > pagesize) { error_report("Target page size bigger than host page size"); @@ -146,6 +181,12 @@ bool postcopy_ram_supported_by_host(void) goto out; } =20 + /* Give devices a chance to object */ + if (postcopy_notify(POSTCOPY_NOTIFY_PROBE, &local_err)) { + error_report_err(local_err); + goto out; + } + /* Version and features check */ if (!ufd_version_check(ufd)) { goto out; diff --git a/migration/postcopy-ram.h b/migration/postcopy-ram.h index 78a3591322..d688411674 100644 --- a/migration/postcopy-ram.h +++ b/migration/postcopy-ram.h @@ -114,4 +114,30 @@ PostcopyState postcopy_state_get(void); /* Set the state and return the old state */ PostcopyState postcopy_state_set(PostcopyState new_state); =20 +/* + * To be called once at the start before any device initialisation + */ +void postcopy_infrastructure_init(void); + +/* Add a notifier to a list to be called when checking whether the devices + * can support postcopy. + * It's data is a *PostcopyNotifyData + * It should return 0 if OK, or a negative value on failure. + * On failure it must set the data->errp to an error. + * + */ +enum PostcopyNotifyReason { + POSTCOPY_NOTIFY_PROBE =3D 0, +}; + +struct PostcopyNotifyData { + enum PostcopyNotifyReason reason; + Error **errp; +}; + +void postcopy_add_notifier(NotifierWithReturn *nn); +void postcopy_remove_notifier(NotifierWithReturn *n); +/* Call the notifier list set by postcopy_add_start_notifier */ +int postcopy_notify(enum PostcopyNotifyReason reason, Error **errp); + #endif diff --git a/vl.c b/vl.c index 8e247cc2a2..65dd9dc324 100644 --- a/vl.c +++ b/vl.c @@ -95,6 +95,7 @@ int main(int argc, char **argv) #include "audio/audio.h" #include "sysemu/cpus.h" #include "migration/colo.h" +#include "migration/postcopy-ram.h" #include "sysemu/kvm.h" #include "sysemu/hax.h" #include "qapi/qobject-input-visitor.h" @@ -3082,6 +3083,7 @@ int main(int argc, char **argv, char **envp) module_call_init(MODULE_INIT_OPTS); =20 runstate_init(); + postcopy_infrastructure_init(); =20 if (qcrypto_init(&err) < 0) { error_reportf_err(err, "cannot initialize crypto: "); --=20 2.13.5 From nobody Thu May 2 16:10:00 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1503603475605568.0083618411318; Thu, 24 Aug 2017 12:37:55 -0700 (PDT) Received: from localhost ([::1]:50159 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dkxww-0002Sz-9I for importer@patchew.org; Thu, 24 Aug 2017 15:37:54 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52145) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dkxna-00025K-2O for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:28:15 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dkxnY-0004SW-Qf for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:28:14 -0400 Received: from mx1.redhat.com ([209.132.183.28]:48928) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dkxnY-0004Ri-Ga for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:28:12 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 88F2D61462; Thu, 24 Aug 2017 19:28:11 +0000 (UTC) Received: from dgilbert-t530.redhat.com (ovpn-117-165.ams2.redhat.com [10.36.117.165]) by smtp.corp.redhat.com (Postfix) with ESMTP id A72DF17CC4; Thu, 24 Aug 2017 19:28:07 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 88F2D61462 Authentication-Results: ext-mx10.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx10.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=dgilbert@redhat.com From: "Dr. David Alan Gilbert (git)" To: qemu-devel@nongnu.org, maxime.coquelin@redhat.com, a.perevalov@samsung.com, mst@redhat.com, marcandre.lureau@redhat.com Date: Thu, 24 Aug 2017 20:27:06 +0100 Message-Id: <20170824192730.8440-9-dgilbert@redhat.com> In-Reply-To: <20170824192730.8440-1-dgilbert@redhat.com> References: <20170824192730.8440-1-dgilbert@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.39]); Thu, 24 Aug 2017 19:28:11 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [RFC v2 08/32] postcopy: Add vhost-user flag for postcopy and check it X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lvivier@redhat.com, aarcange@redhat.com, felipe@nutanix.com, peterx@redhat.com, quintela@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: "Dr. David Alan Gilbert" Add a vhost feature flag for postcopy support, and use the postcopy notifier to check it before allowing postcopy. Signed-off-by: Dr. David Alan Gilbert --- contrib/libvhost-user/libvhost-user.h | 1 + docs/interop/vhost-user.txt | 10 +++++++++ hw/virtio/vhost-user.c | 40 +++++++++++++++++++++++++++++++= +++- 3 files changed, 50 insertions(+), 1 deletion(-) diff --git a/contrib/libvhost-user/libvhost-user.h b/contrib/libvhost-user/= libvhost-user.h index acd019876d..95d0d34a28 100644 --- a/contrib/libvhost-user/libvhost-user.h +++ b/contrib/libvhost-user/libvhost-user.h @@ -34,6 +34,7 @@ enum VhostUserProtocolFeature { VHOST_USER_PROTOCOL_F_MQ =3D 0, VHOST_USER_PROTOCOL_F_LOG_SHMFD =3D 1, VHOST_USER_PROTOCOL_F_RARP =3D 2, + VHOST_USER_PROTOCOL_F_PAGEFAULT =3D 7, =20 VHOST_USER_PROTOCOL_F_MAX }; diff --git a/docs/interop/vhost-user.txt b/docs/interop/vhost-user.txt index 954771d0d8..a279560eb0 100644 --- a/docs/interop/vhost-user.txt +++ b/docs/interop/vhost-user.txt @@ -273,6 +273,15 @@ Once the source has finished migration, rings will be = stopped by the source. No further update must be done before rings are restarted. =20 +In postcopy migration the slave is started before all the memory has been +received from the source host, and care must be taken to avoid accessing p= ages +that have yet to be received. The slave opens a 'userfault'-fd and regist= ers +the memory with it; this fd is then passed back over to the master. +The master services requests on the userfaultfd for pages that are accessed +and when the page is available it performs WAKE ioctl's on the userfaultfd +to wake the stalled slave. The client indicates support for this via the +VHOST_USER_PROTOCOL_F_PAGEFAULT feature. + IOMMU support ------------- =20 @@ -327,6 +336,7 @@ Protocol features #define VHOST_USER_PROTOCOL_F_MTU 4 #define VHOST_USER_PROTOCOL_F_SLAVE_REQ 5 #define VHOST_USER_PROTOCOL_F_CROSS_ENDIAN 6 +#define VHOST_USER_PROTOCOL_F_PAGEFAULT 7 =20 Master message types -------------------- diff --git a/hw/virtio/vhost-user.c b/hw/virtio/vhost-user.c index 093675ed98..c51bbd1296 100644 --- a/hw/virtio/vhost-user.c +++ b/hw/virtio/vhost-user.c @@ -17,6 +17,8 @@ #include "sysemu/kvm.h" #include "qemu/error-report.h" #include "qemu/sockets.h" +#include "migration/migration.h" +#include "migration/postcopy-ram.h" =20 #include #include @@ -34,7 +36,7 @@ enum VhostUserProtocolFeature { VHOST_USER_PROTOCOL_F_NET_MTU =3D 4, VHOST_USER_PROTOCOL_F_SLAVE_REQ =3D 5, VHOST_USER_PROTOCOL_F_CROSS_ENDIAN =3D 6, - + VHOST_USER_PROTOCOL_F_PAGEFAULT =3D 7, VHOST_USER_PROTOCOL_F_MAX }; =20 @@ -123,8 +125,10 @@ static VhostUserMsg m __attribute__ ((unused)); #define VHOST_USER_VERSION (0x1) =20 struct vhost_user { + struct vhost_dev *dev; CharBackend *chr; int slave_fd; + NotifierWithReturn postcopy_notifier; }; =20 static bool ioeventfd_enabled(void) @@ -720,6 +724,33 @@ out: return ret; } =20 +static int vhost_user_postcopy_notifier(NotifierWithReturn *notifier, + void *opaque) +{ + struct PostcopyNotifyData *pnd =3D opaque; + struct vhost_user *u =3D container_of(notifier, struct vhost_user, + postcopy_notifier); + struct vhost_dev *dev =3D u->dev; + + switch (pnd->reason) { + case POSTCOPY_NOTIFY_PROBE: + if (!virtio_has_feature(dev->protocol_features, + VHOST_USER_PROTOCOL_F_PAGEFAULT)) { + /* TODO: Get the device name into this error somehow */ + error_setg(pnd->errp, + "vhost-user backend not capable of postcopy"); + return -ENOENT; + } + break; + + default: + /* We ignore notifications we don't know */ + break; + } + + return 0; +} + static int vhost_user_init(struct vhost_dev *dev, void *opaque) { uint64_t features, protocol_features; @@ -731,6 +762,7 @@ static int vhost_user_init(struct vhost_dev *dev, void = *opaque) u =3D g_new0(struct vhost_user, 1); u->chr =3D opaque; u->slave_fd =3D -1; + u->dev =3D dev; dev->opaque =3D u; =20 err =3D vhost_user_get_features(dev, &features); @@ -787,6 +819,9 @@ static int vhost_user_init(struct vhost_dev *dev, void = *opaque) return err; } =20 + u->postcopy_notifier.notify =3D vhost_user_postcopy_notifier; + postcopy_add_notifier(&u->postcopy_notifier); + return 0; } =20 @@ -797,6 +832,9 @@ static int vhost_user_cleanup(struct vhost_dev *dev) assert(dev->vhost_ops->backend_type =3D=3D VHOST_BACKEND_TYPE_USER); =20 u =3D dev->opaque; + if (u->postcopy_notifier.notify) { + postcopy_remove_notifier(&u->postcopy_notifier); + } if (u->slave_fd >=3D 0) { qemu_set_fd_handler(u->slave_fd, NULL, NULL, NULL); close(u->slave_fd); --=20 2.13.5 From nobody Thu May 2 16:10:00 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 150360364916026.66440196365329; Thu, 24 Aug 2017 12:40:49 -0700 (PDT) Received: from localhost ([::1]:50172 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dkxzj-0004if-JJ for importer@patchew.org; Thu, 24 Aug 2017 15:40:47 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52201) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dkxnk-0002FN-0e for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:28:25 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dkxng-0004YJ-Rx for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:28:24 -0400 Received: from mx1.redhat.com ([209.132.183.28]:52722) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dkxng-0004Xl-J1 for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:28:20 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 9970CC0587DB; Thu, 24 Aug 2017 19:28:19 +0000 (UTC) Received: from dgilbert-t530.redhat.com (ovpn-117-165.ams2.redhat.com [10.36.117.165]) by smtp.corp.redhat.com (Postfix) with ESMTP id D3F1217CC4; Thu, 24 Aug 2017 19:28:11 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 9970CC0587DB Authentication-Results: ext-mx08.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx08.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=dgilbert@redhat.com From: "Dr. David Alan Gilbert (git)" To: qemu-devel@nongnu.org, maxime.coquelin@redhat.com, a.perevalov@samsung.com, mst@redhat.com, marcandre.lureau@redhat.com Date: Thu, 24 Aug 2017 20:27:07 +0100 Message-Id: <20170824192730.8440-10-dgilbert@redhat.com> In-Reply-To: <20170824192730.8440-1-dgilbert@redhat.com> References: <20170824192730.8440-1-dgilbert@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.32]); Thu, 24 Aug 2017 19:28:19 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [RFC v2 09/32] vhost-user: Add 'VHOST_USER_POSTCOPY_ADVISE' message X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lvivier@redhat.com, aarcange@redhat.com, felipe@nutanix.com, peterx@redhat.com, quintela@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: "Dr. David Alan Gilbert" Wire up a notifier to send a VHOST_USER_POSTCOPY_ADVISE message on an incoming advise. Later patches will fill in the behaviour/contents of the message. Signed-off-by: Dr. David Alan Gilbert Reviewed-by: Marc-Andr=C3=A9 Lureau --- contrib/libvhost-user/libvhost-user.c | 21 ++++++++++++--- contrib/libvhost-user/libvhost-user.h | 6 ++++- docs/interop/vhost-user.txt | 9 +++++++ hw/virtio/vhost-user.c | 48 +++++++++++++++++++++++++++++++= ++++ migration/postcopy-ram.h | 1 + migration/savevm.c | 6 +++++ 6 files changed, 86 insertions(+), 5 deletions(-) diff --git a/contrib/libvhost-user/libvhost-user.c b/contrib/libvhost-user/= libvhost-user.c index 201b9846e9..8bbdf5fb40 100644 --- a/contrib/libvhost-user/libvhost-user.c +++ b/contrib/libvhost-user/libvhost-user.c @@ -42,9 +42,6 @@ vu_request_to_string(int req) REQ(VHOST_USER_NONE), REQ(VHOST_USER_GET_FEATURES), REQ(VHOST_USER_SET_FEATURES), - REQ(VHOST_USER_NONE), - REQ(VHOST_USER_GET_FEATURES), - REQ(VHOST_USER_SET_FEATURES), REQ(VHOST_USER_SET_OWNER), REQ(VHOST_USER_RESET_OWNER), REQ(VHOST_USER_SET_MEM_TABLE), @@ -62,7 +59,10 @@ vu_request_to_string(int req) REQ(VHOST_USER_GET_QUEUE_NUM), REQ(VHOST_USER_SET_VRING_ENABLE), REQ(VHOST_USER_SEND_RARP), - REQ(VHOST_USER_INPUT_GET_CONFIG), + REQ(VHOST_USER_SET_SLAVE_REQ_FD), + REQ(VHOST_USER_IOTLB_MSG), + REQ(VHOST_USER_SET_VRING_ENDIAN), + REQ(VHOST_USER_POSTCOPY_ADVISE), REQ(VHOST_USER_MAX), }; #undef REQ @@ -744,6 +744,17 @@ vu_set_vring_enable_exec(VuDev *dev, VhostUserMsg *vms= g) } =20 static bool +vu_set_postcopy_advise(VuDev *dev, VhostUserMsg *vmsg) +{ + /* TODO: Open ufd, pass it back in the request + * TODO: Add addresses=20 + */ + vmsg->payload.u64 =3D 0xcafe; + vmsg->size =3D sizeof(vmsg->payload.u64); + return true; /* =3D send a reply */ +} + +static bool vu_process_message(VuDev *dev, VhostUserMsg *vmsg) { int do_reply =3D 0; @@ -808,6 +819,8 @@ vu_process_message(VuDev *dev, VhostUserMsg *vmsg) return vu_set_vring_enable_exec(dev, vmsg); case VHOST_USER_NONE: break; + case VHOST_USER_POSTCOPY_ADVISE: + return vu_set_postcopy_advise(dev, vmsg); default: vmsg_close_fds(vmsg); vu_panic(dev, "Unhandled request: %d", vmsg->request); diff --git a/contrib/libvhost-user/libvhost-user.h b/contrib/libvhost-user/= libvhost-user.h index 95d0d34a28..3987ce643d 100644 --- a/contrib/libvhost-user/libvhost-user.h +++ b/contrib/libvhost-user/libvhost-user.h @@ -62,7 +62,11 @@ typedef enum VhostUserRequest { VHOST_USER_GET_QUEUE_NUM =3D 17, VHOST_USER_SET_VRING_ENABLE =3D 18, VHOST_USER_SEND_RARP =3D 19, - VHOST_USER_INPUT_GET_CONFIG =3D 20, + VHOST_USER_NET_SET_MTU =3D 20, + VHOST_USER_SET_SLAVE_REQ_FD =3D 21, + VHOST_USER_IOTLB_MSG =3D 22, + VHOST_USER_SET_VRING_ENDIAN =3D 23, + VHOST_USER_POSTCOPY_ADVISE =3D 24, VHOST_USER_MAX } VhostUserRequest; =20 diff --git a/docs/interop/vhost-user.txt b/docs/interop/vhost-user.txt index a279560eb0..dad2a1b343 100644 --- a/docs/interop/vhost-user.txt +++ b/docs/interop/vhost-user.txt @@ -606,6 +606,15 @@ Master message types and expect this message once (per VQ) during device configuration (ie. before the master starts the VQ). =20 + * VHOST_USER_POSTCOPY_ADVISE + Id: 24 + Master payload: N/A + Slave payload: userfault fd + u64 + + Master advises slave that a migration with postcopy enabled is under= way, + the slave must open a userfaultfd for later use. + Note that at this stage the migration is still in precopy mode. + Slave message types ------------------- =20 diff --git a/hw/virtio/vhost-user.c b/hw/virtio/vhost-user.c index c51bbd1296..7063e4df61 100644 --- a/hw/virtio/vhost-user.c +++ b/hw/virtio/vhost-user.c @@ -67,6 +67,7 @@ typedef enum VhostUserRequest { VHOST_USER_SET_SLAVE_REQ_FD =3D 21, VHOST_USER_IOTLB_MSG =3D 22, VHOST_USER_SET_VRING_ENDIAN =3D 23, + VHOST_USER_POSTCOPY_ADVISE =3D 24, VHOST_USER_MAX } VhostUserRequest; =20 @@ -724,6 +725,50 @@ out: return ret; } =20 +/* + * Called at the start of an inbound postcopy on reception of the + * 'advise' command. + */ +static int vhost_user_postcopy_advise(struct vhost_dev *dev, Error **errp) +{ + struct vhost_user *u =3D dev->opaque; + CharBackend *chr =3D u->chr; + int ufd; + VhostUserMsg msg =3D { + .request =3D VHOST_USER_POSTCOPY_ADVISE, + .flags =3D VHOST_USER_VERSION, + }; + + if (vhost_user_write(dev, &msg, NULL, 0) < 0) { + error_setg(errp, "Failed to send postcopy_advise to vhost"); + return -1; + } + + if (vhost_user_read(dev, &msg) < 0) { + error_setg(errp, "Failed to get postcopy_advise reply from vhost"); + return -1; + } + + if (msg.request !=3D VHOST_USER_POSTCOPY_ADVISE) { + error_setg(errp, "Unexpected msg type. Expected %d received %d", + VHOST_USER_POSTCOPY_ADVISE, msg.request); + return -1; + } + + if (msg.size !=3D sizeof(msg.payload.u64)) { + error_setg(errp, "Received bad msg size."); + return -1; + } + ufd =3D qemu_chr_fe_get_msgfd(chr); + if (ufd < 0) { + error_setg(errp, "%s: Failed to get ufd", __func__); + return -1; + } + + /* TODO: register ufd with userfault thread */ + return 0; +} + static int vhost_user_postcopy_notifier(NotifierWithReturn *notifier, void *opaque) { @@ -743,6 +788,9 @@ static int vhost_user_postcopy_notifier(NotifierWithRet= urn *notifier, } break; =20 + case POSTCOPY_NOTIFY_INBOUND_ADVISE: + return vhost_user_postcopy_advise(dev, pnd->errp); + default: /* We ignore notifications we don't know */ break; diff --git a/migration/postcopy-ram.h b/migration/postcopy-ram.h index d688411674..70d4b09659 100644 --- a/migration/postcopy-ram.h +++ b/migration/postcopy-ram.h @@ -128,6 +128,7 @@ void postcopy_infrastructure_init(void); */ enum PostcopyNotifyReason { POSTCOPY_NOTIFY_PROBE =3D 0, + POSTCOPY_NOTIFY_INBOUND_ADVISE, }; =20 struct PostcopyNotifyData { diff --git a/migration/savevm.c b/migration/savevm.c index fdd15fa0a7..d35911731d 100644 --- a/migration/savevm.c +++ b/migration/savevm.c @@ -1343,6 +1343,7 @@ static int loadvm_postcopy_handle_advise(MigrationInc= omingState *mis) { PostcopyState ps =3D postcopy_state_set(POSTCOPY_INCOMING_ADVISE); uint64_t remote_pagesize_summary, local_pagesize_summary, remote_tps; + Error *local_err =3D NULL; =20 trace_loadvm_postcopy_handle_advise(); if (ps !=3D POSTCOPY_INCOMING_NONE) { @@ -1390,6 +1391,11 @@ static int loadvm_postcopy_handle_advise(MigrationIn= comingState *mis) return -1; } =20 + if (postcopy_notify(POSTCOPY_NOTIFY_INBOUND_ADVISE, &local_err)) { + error_report_err(local_err); + return -1; + } + if (ram_postcopy_incoming_init(mis)) { return -1; } --=20 2.13.5 From nobody Thu May 2 16:10:00 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1503604210486283.9660044181396; Thu, 24 Aug 2017 12:50:10 -0700 (PDT) Received: from localhost ([::1]:50217 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dky8n-0005IR-B9 for importer@patchew.org; Thu, 24 Aug 2017 15:50:09 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52207) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dkxnk-0002Fk-Dv for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:28:25 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dkxnj-0004ax-Bp for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:28:24 -0400 Received: from mx1.redhat.com ([209.132.183.28]:49894) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dkxnj-0004aM-3E for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:28:23 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 1A7784A6FD; Thu, 24 Aug 2017 19:28:22 +0000 (UTC) Received: from dgilbert-t530.redhat.com (ovpn-117-165.ams2.redhat.com [10.36.117.165]) by smtp.corp.redhat.com (Postfix) with ESMTP id E363717CC4; Thu, 24 Aug 2017 19:28:19 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 1A7784A6FD Authentication-Results: ext-mx09.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx09.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=dgilbert@redhat.com From: "Dr. David Alan Gilbert (git)" To: qemu-devel@nongnu.org, maxime.coquelin@redhat.com, a.perevalov@samsung.com, mst@redhat.com, marcandre.lureau@redhat.com Date: Thu, 24 Aug 2017 20:27:08 +0100 Message-Id: <20170824192730.8440-11-dgilbert@redhat.com> In-Reply-To: <20170824192730.8440-1-dgilbert@redhat.com> References: <20170824192730.8440-1-dgilbert@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.38]); Thu, 24 Aug 2017 19:28:22 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [RFC v2 10/32] vhub: Support sending fds back to qemu X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lvivier@redhat.com, aarcange@redhat.com, felipe@nutanix.com, peterx@redhat.com, quintela@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: "Dr. David Alan Gilbert" Allow replies with fds (for postcopy) Signed-off-by: Dr. David Alan Gilbert Reviewed-by: Marc-Andr=C3=A9 Lureau --- contrib/libvhost-user/libvhost-user.c | 29 ++++++++++++++++++++++++++++- 1 file changed, 28 insertions(+), 1 deletion(-) diff --git a/contrib/libvhost-user/libvhost-user.c b/contrib/libvhost-user/= libvhost-user.c index 8bbdf5fb40..47884c0a15 100644 --- a/contrib/libvhost-user/libvhost-user.c +++ b/contrib/libvhost-user/libvhost-user.c @@ -213,6 +213,30 @@ vu_message_write(VuDev *dev, int conn_fd, VhostUserMsg= *vmsg) { int rc; uint8_t *p =3D (uint8_t *)vmsg; + char control[CMSG_SPACE(VHOST_MEMORY_MAX_NREGIONS * sizeof(int))] =3D = { }; + struct iovec iov =3D { + .iov_base =3D (char *)vmsg, + .iov_len =3D VHOST_USER_HDR_SIZE, + }; + struct msghdr msg =3D { + .msg_iov =3D &iov, + .msg_iovlen =3D 1, + .msg_control =3D control, + }; + struct cmsghdr *cmsg; + + memset(control, 0, sizeof(control)); + if (vmsg->fds) { + size_t fdsize =3D vmsg->fd_num * sizeof(int); + msg.msg_controllen =3D CMSG_SPACE(fdsize); + cmsg =3D CMSG_FIRSTHDR(&msg); + cmsg->cmsg_len =3D CMSG_LEN(fdsize); + cmsg->cmsg_level =3D SOL_SOCKET; + cmsg->cmsg_type =3D SCM_RIGHTS; + memcpy(CMSG_DATA(cmsg), vmsg->fds, fdsize); + } else { + msg.msg_controllen =3D 0; + } =20 /* Set the version in the flags when sending the reply */ vmsg->flags &=3D ~VHOST_USER_VERSION_MASK; @@ -220,7 +244,7 @@ vu_message_write(VuDev *dev, int conn_fd, VhostUserMsg = *vmsg) vmsg->flags |=3D VHOST_USER_REPLY_MASK; =20 do { - rc =3D write(conn_fd, p, VHOST_USER_HDR_SIZE); + rc =3D sendmsg(conn_fd, &msg, 0); } while (rc < 0 && (errno =3D=3D EINTR || errno =3D=3D EAGAIN)); =20 do { @@ -313,6 +337,7 @@ vu_get_features_exec(VuDev *dev, VhostUserMsg *vmsg) } =20 vmsg->size =3D sizeof(vmsg->payload.u64); + vmsg->fd_num =3D 0; =20 DPRINT("Sending back to guest u64: 0x%016"PRIx64"\n", vmsg->payload.u6= 4); =20 @@ -454,6 +479,7 @@ vu_set_log_base_exec(VuDev *dev, VhostUserMsg *vmsg) dev->log_size =3D log_mmap_size; =20 vmsg->size =3D sizeof(vmsg->payload.u64); + vmsg->fd_num =3D 0; =20 return true; } @@ -698,6 +724,7 @@ vu_get_protocol_features_exec(VuDev *dev, VhostUserMsg = *vmsg) =20 vmsg->payload.u64 =3D features; vmsg->size =3D sizeof(vmsg->payload.u64); + vmsg->fd_num =3D 0; =20 return true; } --=20 2.13.5 From nobody Thu May 2 16:10:00 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1503604383852360.6938028265547; Thu, 24 Aug 2017 12:53:03 -0700 (PDT) Received: from localhost ([::1]:50233 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dkyBZ-0008Ef-I1 for importer@patchew.org; Thu, 24 Aug 2017 15:53:01 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52239) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dkxnm-0002I6-Dr for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:28:27 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dkxnl-0004cB-L2 for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:28:26 -0400 Received: from mx1.redhat.com ([209.132.183.28]:52910) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dkxnl-0004bk-Ey for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:28:25 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 7E89DC05683F; Thu, 24 Aug 2017 19:28:24 +0000 (UTC) Received: from dgilbert-t530.redhat.com (ovpn-117-165.ams2.redhat.com [10.36.117.165]) by smtp.corp.redhat.com (Postfix) with ESMTP id 65D2917CC4; Thu, 24 Aug 2017 19:28:22 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 7E89DC05683F Authentication-Results: ext-mx08.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx08.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=dgilbert@redhat.com From: "Dr. David Alan Gilbert (git)" To: qemu-devel@nongnu.org, maxime.coquelin@redhat.com, a.perevalov@samsung.com, mst@redhat.com, marcandre.lureau@redhat.com Date: Thu, 24 Aug 2017 20:27:09 +0100 Message-Id: <20170824192730.8440-12-dgilbert@redhat.com> In-Reply-To: <20170824192730.8440-1-dgilbert@redhat.com> References: <20170824192730.8440-1-dgilbert@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.32]); Thu, 24 Aug 2017 19:28:24 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [RFC v2 11/32] vhub: Open userfaultfd X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lvivier@redhat.com, aarcange@redhat.com, felipe@nutanix.com, peterx@redhat.com, quintela@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: "Dr. David Alan Gilbert" Open a userfaultfd (on a postcopy_advise) and send it back in the reply to the qemu for it to monitor. Signed-off-by: Dr. David Alan Gilbert --- contrib/libvhost-user/libvhost-user.c | 26 +++++++++++++++++++++++--- contrib/libvhost-user/libvhost-user.h | 3 +++ 2 files changed, 26 insertions(+), 3 deletions(-) diff --git a/contrib/libvhost-user/libvhost-user.c b/contrib/libvhost-user/= libvhost-user.c index 47884c0a15..f9b5b12b28 100644 --- a/contrib/libvhost-user/libvhost-user.c +++ b/contrib/libvhost-user/libvhost-user.c @@ -15,6 +15,7 @@ =20 #include #include +#include #include =20 #include "qemu/atomic.h" @@ -773,11 +774,30 @@ vu_set_vring_enable_exec(VuDev *dev, VhostUserMsg *vm= sg) static bool vu_set_postcopy_advise(VuDev *dev, VhostUserMsg *vmsg) { - /* TODO: Open ufd, pass it back in the request - * TODO: Add addresses=20 - */ + struct uffdio_api api_struct; + + dev->postcopy_ufd =3D syscall(__NR_userfaultfd, O_CLOEXEC | O_NONBLOCK= ); + /* TODO: Add addresses */ vmsg->payload.u64 =3D 0xcafe; vmsg->size =3D sizeof(vmsg->payload.u64); + + if (dev->postcopy_ufd =3D=3D -1) { + vu_panic(dev, "Userfaultfd not available: %s", strerror(errno)); + goto out; + } + api_struct.api =3D UFFD_API; + api_struct.features =3D 0; + if (ioctl(dev->postcopy_ufd, UFFDIO_API, &api_struct)) { + vu_panic(dev, "Failed UFFDIO_API: %s", strerror(errno)); + close(dev->postcopy_ufd); + dev->postcopy_ufd =3D -1; + goto out; + } + /* TODO: Stash feature flags somewhere */ +out: + /* Return a ufd to the QEMU */ + vmsg->fd_num =3D 1; + vmsg->fds[0] =3D dev->postcopy_ufd; return true; /* =3D send a reply */ } =20 diff --git a/contrib/libvhost-user/libvhost-user.h b/contrib/libvhost-user/= libvhost-user.h index 3987ce643d..3e8efdd919 100644 --- a/contrib/libvhost-user/libvhost-user.h +++ b/contrib/libvhost-user/libvhost-user.h @@ -234,6 +234,9 @@ struct VuDev { * re-initialize */ vu_panic_cb panic; const VuDevIface *iface; + + /* Postcopy data */ + int postcopy_ufd; }; =20 typedef struct VuVirtqElement { --=20 2.13.5 From nobody Thu May 2 16:10:00 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 150360353598691.00360253289148; Thu, 24 Aug 2017 12:38:55 -0700 (PDT) Received: from localhost ([::1]:50161 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dkxxu-0003C7-OQ for importer@patchew.org; Thu, 24 Aug 2017 15:38:54 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52273) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dkxnq-0002Ly-EV for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:28:33 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dkxno-0004ed-Kh for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:28:30 -0400 Received: from mx1.redhat.com ([209.132.183.28]:50748) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dkxno-0004dc-BL for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:28:28 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 4FF117EA96; Thu, 24 Aug 2017 19:28:27 +0000 (UTC) Received: from dgilbert-t530.redhat.com (ovpn-117-165.ams2.redhat.com [10.36.117.165]) by smtp.corp.redhat.com (Postfix) with ESMTP id C987A17CC4; Thu, 24 Aug 2017 19:28:24 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 4FF117EA96 Authentication-Results: ext-mx04.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx04.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=dgilbert@redhat.com From: "Dr. David Alan Gilbert (git)" To: qemu-devel@nongnu.org, maxime.coquelin@redhat.com, a.perevalov@samsung.com, mst@redhat.com, marcandre.lureau@redhat.com Date: Thu, 24 Aug 2017 20:27:10 +0100 Message-Id: <20170824192730.8440-13-dgilbert@redhat.com> In-Reply-To: <20170824192730.8440-1-dgilbert@redhat.com> References: <20170824192730.8440-1-dgilbert@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.28]); Thu, 24 Aug 2017 19:28:27 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [RFC v2 12/32] postcopy: Allow registering of fd handler X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lvivier@redhat.com, aarcange@redhat.com, felipe@nutanix.com, peterx@redhat.com, quintela@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: "Dr. David Alan Gilbert" Allow other userfaultfd's to be registered into the fault thread so that handlers for shared memory can get responses. Signed-off-by: Dr. David Alan Gilbert --- migration/migration.c | 3 + migration/migration.h | 2 + migration/postcopy-ram.c | 212 +++++++++++++++++++++++++++++++++++--------= ---- migration/postcopy-ram.h | 21 +++++ migration/trace-events | 2 + 5 files changed, 186 insertions(+), 54 deletions(-) diff --git a/migration/migration.c b/migration/migration.c index c3fe0ed9ca..2c43d730e2 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -144,6 +144,8 @@ MigrationIncomingState *migration_incoming_get_current(= void) if (!once) { mis_current.state =3D MIGRATION_STATUS_NONE; memset(&mis_current, 0, sizeof(MigrationIncomingState)); + mis_current.postcopy_remote_fds =3D g_array_new(FALSE, TRUE, + sizeof(struct PostCopyF= D)); qemu_mutex_init(&mis_current.rp_mutex); qemu_event_init(&mis_current.main_thread_load_event, false); once =3D true; @@ -166,6 +168,7 @@ void migration_incoming_state_destroy(void) qemu_fclose(mis->from_src_file); mis->from_src_file =3D NULL; } + g_array_free(mis->postcopy_remote_fds, TRUE); =20 qemu_event_destroy(&mis->main_thread_load_event); } diff --git a/migration/migration.h b/migration/migration.h index 148c9facbc..9fcea6bb25 100644 --- a/migration/migration.h +++ b/migration/migration.h @@ -48,6 +48,8 @@ struct MigrationIncomingState { QemuMutex rp_mutex; /* We send replies from multiple threads */ void *postcopy_tmp_page; void *postcopy_tmp_zero_page; + /* PostCopyFD's for external userfaultfds & handlers of shared memory = */ + GArray *postcopy_remote_fds; =20 QEMUBH *bh; =20 diff --git a/migration/postcopy-ram.c b/migration/postcopy-ram.c index 95007c00ef..faee7708ff 100644 --- a/migration/postcopy-ram.c +++ b/migration/postcopy-ram.c @@ -466,29 +466,43 @@ static void *postcopy_ram_fault_thread(void *opaque) MigrationIncomingState *mis =3D opaque; struct uffd_msg msg; int ret; + size_t index; RAMBlock *rb =3D NULL; RAMBlock *last_rb =3D NULL; /* last RAMBlock we sent part of */ =20 trace_postcopy_ram_fault_thread_entry(); qemu_sem_post(&mis->fault_thread_sem); =20 + struct pollfd *pfd; + size_t pfd_len =3D 2 + mis->postcopy_remote_fds->len; + + pfd =3D g_new0(struct pollfd, pfd_len); + + pfd[0].fd =3D mis->userfault_fd; + pfd[0].events =3D POLLIN; + pfd[1].fd =3D mis->userfault_quit_fd; + pfd[1].events =3D POLLIN; /* Waiting for eventfd to go positive */ + trace_postcopy_ram_fault_thread_fds_core(pfd[0].fd, pfd[1].fd); + for (index =3D 0; index < mis->postcopy_remote_fds->len; index++) { + struct PostCopyFD *pcfd =3D &g_array_index(mis->postcopy_remote_fd= s, + struct PostCopyFD, index); + pfd[2 + index].fd =3D pcfd->fd; + pfd[2 + index].events =3D POLLIN; + trace_postcopy_ram_fault_thread_fds_extra(2 + index, pcfd->idstr, + pcfd->fd); + } + while (true) { ram_addr_t rb_offset; - struct pollfd pfd[2]; + int poll_result; =20 /* * We're mainly waiting for the kernel to give us a faulting HVA, * however we can be told to quit via userfault_quit_fd which is * an eventfd */ - pfd[0].fd =3D mis->userfault_fd; - pfd[0].events =3D POLLIN; - pfd[0].revents =3D 0; - pfd[1].fd =3D mis->userfault_quit_fd; - pfd[1].events =3D POLLIN; /* Waiting for eventfd to go positive */ - pfd[1].revents =3D 0; - - if (poll(pfd, 2, -1 /* Wait forever */) =3D=3D -1) { + poll_result =3D poll(pfd, pfd_len, -1 /* Wait forever */); + if (poll_result =3D=3D -1) { error_report("%s: userfault poll: %s", __func__, strerror(errn= o)); break; } @@ -498,57 +512,118 @@ static void *postcopy_ram_fault_thread(void *opaque) break; } =20 - ret =3D read(mis->userfault_fd, &msg, sizeof(msg)); - if (ret !=3D sizeof(msg)) { - if (errno =3D=3D EAGAIN) { - /* - * if a wake up happens on the other thread just after - * the poll, there is nothing to read. - */ - continue; + if (pfd[0].revents) { + poll_result--; + ret =3D read(mis->userfault_fd, &msg, sizeof(msg)); + if (ret !=3D sizeof(msg)) { + if (errno =3D=3D EAGAIN) { + /* + * if a wake up happens on the other thread just after + * the poll, there is nothing to read. + */ + continue; + } + if (ret < 0) { + error_report("%s: Failed to read full userfault " + "message: %s", + __func__, strerror(errno)); + break; + } else { + error_report("%s: Read %d bytes from userfaultfd " + "expected %zd", + __func__, ret, sizeof(msg)); + break; /* Lost alignment, don't know what we'd read ne= xt */ + } + } + if (msg.event !=3D UFFD_EVENT_PAGEFAULT) { + error_report("%s: Read unexpected event %ud from userfault= fd", + __func__, msg.event); + continue; /* It's not a page fault, shouldn't happen */ } - if (ret < 0) { - error_report("%s: Failed to read full userfault message: %= s", - __func__, strerror(errno)); + + rb =3D qemu_ram_block_from_host( + (void *)(uintptr_t)msg.arg.pagefault.address, + true, &rb_offset); + if (!rb) { + error_report("postcopy_ram_fault_thread: Fault outside gue= st: %" + PRIx64, (uint64_t)msg.arg.pagefault.address); break; - } else { - error_report("%s: Read %d bytes from userfaultfd expected = %zd", - __func__, ret, sizeof(msg)); - break; /* Lost alignment, don't know what we'd read next */ } - } - if (msg.event !=3D UFFD_EVENT_PAGEFAULT) { - error_report("%s: Read unexpected event %ud from userfaultfd", - __func__, msg.event); - continue; /* It's not a page fault, shouldn't happen */ - } =20 - rb =3D qemu_ram_block_from_host( - (void *)(uintptr_t)msg.arg.pagefault.address, - true, &rb_offset); - if (!rb) { - error_report("postcopy_ram_fault_thread: Fault outside guest: = %" - PRIx64, (uint64_t)msg.arg.pagefault.address); - break; - } + rb_offset &=3D ~(qemu_ram_pagesize(rb) - 1); + trace_postcopy_ram_fault_thread_request(msg.arg.pagefault.addr= ess, + qemu_ram_get_idstr(rb), + rb_offset); =20 - rb_offset &=3D ~(qemu_ram_pagesize(rb) - 1); - trace_postcopy_ram_fault_thread_request(msg.arg.pagefault.address, - qemu_ram_get_idstr(rb), - rb_offset); + /* + * Send the request to the source - we want to request one + * of our host page sizes (which is >=3D TPS) + */ + if (rb !=3D last_rb) { + last_rb =3D rb; + migrate_send_rp_req_pages(mis, qemu_ram_get_idstr(rb), + rb_offset, qemu_ram_pagesize(rb)); + } else { + /* Save some space */ + migrate_send_rp_req_pages(mis, NULL, + rb_offset, qemu_ram_pagesize(rb)); + } + } =20 - /* - * Send the request to the source - we want to request one - * of our host page sizes (which is >=3D TPS) - */ - if (rb !=3D last_rb) { - last_rb =3D rb; - migrate_send_rp_req_pages(mis, qemu_ram_get_idstr(rb), - rb_offset, qemu_ram_pagesize(rb)); - } else { - /* Save some space */ - migrate_send_rp_req_pages(mis, NULL, - rb_offset, qemu_ram_pagesize(rb)); + /* Now handle any requests from external processes on shared memor= y */ + /* TODO: May need to handle devices deregistering during postcopy = */ + for (index =3D 2; index < pfd_len && poll_result; index++) { + if (pfd[index].revents) { + struct PostCopyFD *pcfd =3D + &g_array_index(mis->postcopy_remote_fds, + struct PostCopyFD, index - 2); + + poll_result--; + if (pfd[index].revents & POLLERR) { + error_report("%s: POLLERR on poll %zd fd=3D%d", + __func__, index, pcfd->fd); + pfd[index].events =3D 0; + continue; + } + + ret =3D read(pcfd->fd, &msg, sizeof(msg)); + if (ret !=3D sizeof(msg)) { + if (errno =3D=3D EAGAIN) { + /* + * if a wake up happens on the other thread just a= fter + * the poll, there is nothing to read. + */ + continue; + } + if (ret < 0) { + error_report("%s: Failed to read full userfault " + "message: %s (shared) revents=3D%d", + __func__, strerror(errno), + pfd[index].revents); + /*TODO: Could just disable this sharer */ + break; + } else { + error_report("%s: Read %d bytes from userfaultfd " + "expected %zd (shared)", + __func__, ret, sizeof(msg)); + /*TODO: Could just disable this sharer */ + break; /*Lost alignment,don't know what we'd read = next*/ + } + } + if (msg.event !=3D UFFD_EVENT_PAGEFAULT) { + error_report("%s: Read unexpected event %ud " + "from userfaultfd (shared)", + __func__, msg.event); + continue; /* It's not a page fault, shouldn't happen */ + } + /* Call the device handler registered with us */ + ret =3D pcfd->handler(pcfd, &msg); + if (ret) { + error_report("%s: Failed to resolve shared fault on %z= d/%s", + __func__, index, pcfd->idstr); + /* TODO: Fail? Disable this sharer? */ + } + } } } trace_postcopy_ram_fault_thread_exit(); @@ -878,3 +953,32 @@ PostcopyState postcopy_state_set(PostcopyState new_sta= te) { return atomic_xchg(&incoming_postcopy_state, new_state); } + +/* Register a handler for external shared memory postcopy + * called on the destination. + */ +void postcopy_register_shared_ufd(struct PostCopyFD *pcfd) +{ + MigrationIncomingState *mis =3D migration_incoming_get_current(); + + mis->postcopy_remote_fds =3D g_array_append_val(mis->postcopy_remote_f= ds, + *pcfd); +} + +/* Unregister a handler for external shared memory postcopy + */ +void postcopy_unregister_shared_ufd(struct PostCopyFD *pcfd) +{ + guint i; + MigrationIncomingState *mis =3D migration_incoming_get_current(); + GArray *pcrfds =3D mis->postcopy_remote_fds; + + for (i =3D 0; i < pcrfds->len; i++) { + struct PostCopyFD *cur =3D &g_array_index(pcrfds, struct PostCopyF= D, i); + if (cur->fd =3D=3D pcfd->fd) { + mis->postcopy_remote_fds =3D g_array_remove_index(pcrfds, i); + return; + } + } +} + diff --git a/migration/postcopy-ram.h b/migration/postcopy-ram.h index 70d4b09659..ba8a8ffec5 100644 --- a/migration/postcopy-ram.h +++ b/migration/postcopy-ram.h @@ -141,4 +141,25 @@ void postcopy_remove_notifier(NotifierWithReturn *n); /* Call the notifier list set by postcopy_add_start_notifier */ int postcopy_notify(enum PostcopyNotifyReason reason, Error **errp); =20 +struct PostCopyFD; + +/* ufd is a pointer to the struct uffd_msg *TODO: more Portable! */ +typedef int (*pcfdhandler)(struct PostCopyFD *pcfd, void *ufd); + +struct PostCopyFD { + int fd; + /* Data to pass to handler */ + void *data; + /* Handler to be called whenever we get a poll event */ + pcfdhandler handler; + /* A string to use in error messages */ + char *idstr; +}; + +/* Register a userfaultfd owned by an external process for + * shared memory. + */ +void postcopy_register_shared_ufd(struct PostCopyFD *pcfd); +void postcopy_unregister_shared_ufd(struct PostCopyFD *pcfd); + #endif diff --git a/migration/trace-events b/migration/trace-events index 7a3b5144ff..23f4e5339b 100644 --- a/migration/trace-events +++ b/migration/trace-events @@ -189,6 +189,8 @@ postcopy_place_page_zero(void *host_addr) "host=3D%p" postcopy_ram_enable_notify(void) "" postcopy_ram_fault_thread_entry(void) "" postcopy_ram_fault_thread_exit(void) "" +postcopy_ram_fault_thread_fds_core(int baseufd, int quitfd) "ufd: %d quitf= d: %d" +postcopy_ram_fault_thread_fds_extra(size_t index, const char *name, int fd= ) "%zd/%s: %d" postcopy_ram_fault_thread_quit(void) "" postcopy_ram_fault_thread_request(uint64_t hostaddr, const char *ramblock,= size_t offset) "Request for HVA=3D0x%" PRIx64 " rb=3D%s offset=3D0x%zx" postcopy_ram_incoming_cleanup_closeuf(void) "" --=20 2.13.5 From nobody Thu May 2 16:10:00 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1503603701072182.85483901337864; Thu, 24 Aug 2017 12:41:41 -0700 (PDT) Received: from localhost ([::1]:50177 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dky0Z-0005f1-Rw for importer@patchew.org; Thu, 24 Aug 2017 15:41:39 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52351) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dkxo2-0002Wc-8x for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:28:43 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dkxnz-0004ps-54 for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:28:42 -0400 Received: from mx1.redhat.com ([209.132.183.28]:49644) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dkxny-0004p3-VB for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:28:39 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id F33A461479; Thu, 24 Aug 2017 19:28:37 +0000 (UTC) Received: from dgilbert-t530.redhat.com (ovpn-117-165.ams2.redhat.com [10.36.117.165]) by smtp.corp.redhat.com (Postfix) with ESMTP id 954CD17CC4; Thu, 24 Aug 2017 19:28:27 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com F33A461479 Authentication-Results: ext-mx10.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx10.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=dgilbert@redhat.com From: "Dr. David Alan Gilbert (git)" To: qemu-devel@nongnu.org, maxime.coquelin@redhat.com, a.perevalov@samsung.com, mst@redhat.com, marcandre.lureau@redhat.com Date: Thu, 24 Aug 2017 20:27:11 +0100 Message-Id: <20170824192730.8440-14-dgilbert@redhat.com> In-Reply-To: <20170824192730.8440-1-dgilbert@redhat.com> References: <20170824192730.8440-1-dgilbert@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.39]); Thu, 24 Aug 2017 19:28:38 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [RFC v2 13/32] vhost+postcopy: Register shared ufd with postcopy X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lvivier@redhat.com, aarcange@redhat.com, felipe@nutanix.com, peterx@redhat.com, quintela@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: "Dr. David Alan Gilbert" Register the UFD that comes in as the response to the 'advise' method with the postcopy code. Signed-off-by: Dr. David Alan Gilbert --- hw/virtio/vhost-user.c | 21 ++++++++++++++++++++- migration/postcopy-ram.h | 2 +- 2 files changed, 21 insertions(+), 2 deletions(-) diff --git a/hw/virtio/vhost-user.c b/hw/virtio/vhost-user.c index 7063e4df61..b7898f8939 100644 --- a/hw/virtio/vhost-user.c +++ b/hw/virtio/vhost-user.c @@ -24,6 +24,7 @@ #include #include #include +#include =20 #define VHOST_MEMORY_MAX_NREGIONS 8 #define VHOST_USER_F_PROTOCOL_FEATURES 30 @@ -130,6 +131,7 @@ struct vhost_user { CharBackend *chr; int slave_fd; NotifierWithReturn postcopy_notifier; + struct PostCopyFD postcopy_fd; }; =20 static bool ioeventfd_enabled(void) @@ -726,6 +728,17 @@ out: } =20 /* + * Called back from the postcopy fault thread when a fault is received on = our + * ufd. + * TODO: This is Linux specific + */ +static int vhost_user_postcopy_fault_handler(struct PostCopyFD *pcfd, + void *ufd) +{ + return 0; +} + +/* * Called at the start of an inbound postcopy on reception of the * 'advise' command. */ @@ -764,8 +777,14 @@ static int vhost_user_postcopy_advise(struct vhost_dev= *dev, Error **errp) error_setg(errp, "%s: Failed to get ufd", __func__); return -1; } + fcntl(ufd, F_SETFL, O_NONBLOCK); =20 - /* TODO: register ufd with userfault thread */ + /* register ufd with userfault thread */ + u->postcopy_fd.fd =3D ufd; + u->postcopy_fd.data =3D dev; + u->postcopy_fd.handler =3D vhost_user_postcopy_fault_handler; + u->postcopy_fd.idstr =3D "vhost-user"; /* Need to find unique name */ + postcopy_register_shared_ufd(&u->postcopy_fd); return 0; } =20 diff --git a/migration/postcopy-ram.h b/migration/postcopy-ram.h index ba8a8ffec5..28c216cc7a 100644 --- a/migration/postcopy-ram.h +++ b/migration/postcopy-ram.h @@ -153,7 +153,7 @@ struct PostCopyFD { /* Handler to be called whenever we get a poll event */ pcfdhandler handler; /* A string to use in error messages */ - char *idstr; + const char *idstr; }; =20 /* Register a userfaultfd owned by an external process for --=20 2.13.5 From nobody Thu May 2 16:10:00 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1503603852840437.0948831365273; Thu, 24 Aug 2017 12:44:12 -0700 (PDT) Received: from localhost ([::1]:50186 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dky31-0008EB-Ef for importer@patchew.org; Thu, 24 Aug 2017 15:44:11 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52361) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dkxo3-0002XS-6A for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:28:44 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dkxo1-0004tx-Rh for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:28:43 -0400 Received: from mx1.redhat.com ([209.132.183.28]:33704) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dkxo1-0004sF-IE for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:28:41 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 88CF51F56A; Thu, 24 Aug 2017 19:28:40 +0000 (UTC) Received: from dgilbert-t530.redhat.com (ovpn-117-165.ams2.redhat.com [10.36.117.165]) by smtp.corp.redhat.com (Postfix) with ESMTP id 4AED117CC4; Thu, 24 Aug 2017 19:28:38 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 88CF51F56A Authentication-Results: ext-mx06.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx06.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=dgilbert@redhat.com From: "Dr. David Alan Gilbert (git)" To: qemu-devel@nongnu.org, maxime.coquelin@redhat.com, a.perevalov@samsung.com, mst@redhat.com, marcandre.lureau@redhat.com Date: Thu, 24 Aug 2017 20:27:12 +0100 Message-Id: <20170824192730.8440-15-dgilbert@redhat.com> In-Reply-To: <20170824192730.8440-1-dgilbert@redhat.com> References: <20170824192730.8440-1-dgilbert@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.30]); Thu, 24 Aug 2017 19:28:40 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [RFC v2 14/32] vhost+postcopy: Transmit 'listen' to client X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lvivier@redhat.com, aarcange@redhat.com, felipe@nutanix.com, peterx@redhat.com, quintela@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: "Dr. David Alan Gilbert" Notify the vhost-user client on reception of the 'postcopy-listen' event from the source. Signed-off-by: Dr. David Alan Gilbert Reviewed-by: Marc-Andr=C3=A9 Lureau --- contrib/libvhost-user/libvhost-user.c | 21 +++++++++++++++++++++ contrib/libvhost-user/libvhost-user.h | 2 ++ docs/interop/vhost-user.txt | 6 ++++++ hw/virtio/trace-events | 3 +++ hw/virtio/vhost-user.c | 30 ++++++++++++++++++++++++++++++ migration/postcopy-ram.h | 1 + migration/savevm.c | 7 +++++++ 7 files changed, 70 insertions(+) diff --git a/contrib/libvhost-user/libvhost-user.c b/contrib/libvhost-user/= libvhost-user.c index f9b5b12b28..e8accf11db 100644 --- a/contrib/libvhost-user/libvhost-user.c +++ b/contrib/libvhost-user/libvhost-user.c @@ -15,7 +15,9 @@ =20 #include #include +#include #include +#include #include =20 #include "qemu/atomic.h" @@ -64,6 +66,7 @@ vu_request_to_string(int req) REQ(VHOST_USER_IOTLB_MSG), REQ(VHOST_USER_SET_VRING_ENDIAN), REQ(VHOST_USER_POSTCOPY_ADVISE), + REQ(VHOST_USER_POSTCOPY_LISTEN), REQ(VHOST_USER_MAX), }; #undef REQ @@ -802,6 +805,22 @@ out: } =20 static bool +vu_set_postcopy_listen(VuDev *dev, VhostUserMsg *vmsg) +{ + vmsg->payload.u64 =3D -1; + vmsg->size =3D sizeof(vmsg->payload.u64); + + if (dev->nregions) { + vu_panic(dev, "Regions already registered at postcopy-listen"); + return true; + } + dev->postcopy_listening =3D true; + + vmsg->flags =3D VHOST_USER_VERSION | VHOST_USER_REPLY_MASK; + vmsg->payload.u64 =3D 0; /* Success */ + return true; +} +static bool vu_process_message(VuDev *dev, VhostUserMsg *vmsg) { int do_reply =3D 0; @@ -868,6 +887,8 @@ vu_process_message(VuDev *dev, VhostUserMsg *vmsg) break; case VHOST_USER_POSTCOPY_ADVISE: return vu_set_postcopy_advise(dev, vmsg); + case VHOST_USER_POSTCOPY_LISTEN: + return vu_set_postcopy_listen(dev, vmsg); default: vmsg_close_fds(vmsg); vu_panic(dev, "Unhandled request: %d", vmsg->request); diff --git a/contrib/libvhost-user/libvhost-user.h b/contrib/libvhost-user/= libvhost-user.h index 3e8efdd919..29c11ba56c 100644 --- a/contrib/libvhost-user/libvhost-user.h +++ b/contrib/libvhost-user/libvhost-user.h @@ -67,6 +67,7 @@ typedef enum VhostUserRequest { VHOST_USER_IOTLB_MSG =3D 22, VHOST_USER_SET_VRING_ENDIAN =3D 23, VHOST_USER_POSTCOPY_ADVISE =3D 24, + VHOST_USER_POSTCOPY_LISTEN =3D 25, VHOST_USER_MAX } VhostUserRequest; =20 @@ -237,6 +238,7 @@ struct VuDev { =20 /* Postcopy data */ int postcopy_ufd; + bool postcopy_listening; }; =20 typedef struct VuVirtqElement { diff --git a/docs/interop/vhost-user.txt b/docs/interop/vhost-user.txt index dad2a1b343..73c3dd74db 100644 --- a/docs/interop/vhost-user.txt +++ b/docs/interop/vhost-user.txt @@ -615,6 +615,12 @@ Master message types the slave must open a userfaultfd for later use. Note that at this stage the migration is still in precopy mode. =20 + * VHOST_USER_POSTCOPY_LISTEN + Id: 25 + Master payload: N/A + + Master advises slave that a transition to postcopy mode has happened. + Slave message types ------------------- =20 diff --git a/hw/virtio/trace-events b/hw/virtio/trace-events index 775461ae98..f736c7c84f 100644 --- a/hw/virtio/trace-events +++ b/hw/virtio/trace-events @@ -1,5 +1,8 @@ # See docs/devel/tracing.txt for syntax documentation. =20 +# hw/virtio/vhost-user.c +vhost_user_postcopy_listen(void) "" + # hw/virtio/virtio.c virtqueue_alloc_element(void *elem, size_t sz, unsigned in_num, unsigned o= ut_num) "elem %p size %zd in_num %u out_num %u" virtqueue_fill(void *vq, const void *elem, unsigned int len, unsigned int = idx) "vq %p elem %p len %u idx %u" diff --git a/hw/virtio/vhost-user.c b/hw/virtio/vhost-user.c index b7898f8939..9178271ab2 100644 --- a/hw/virtio/vhost-user.c +++ b/hw/virtio/vhost-user.c @@ -69,6 +69,7 @@ typedef enum VhostUserRequest { VHOST_USER_IOTLB_MSG =3D 22, VHOST_USER_SET_VRING_ENDIAN =3D 23, VHOST_USER_POSTCOPY_ADVISE =3D 24, + VHOST_USER_POSTCOPY_LISTEN =3D 25, VHOST_USER_MAX } VhostUserRequest; =20 @@ -788,6 +789,32 @@ static int vhost_user_postcopy_advise(struct vhost_dev= *dev, Error **errp) return 0; } =20 +/* + * Called at the switch to postcopy on reception of the 'listen' command. + */ +static int vhost_user_postcopy_listen(struct vhost_dev *dev, Error **errp) +{ + int ret; + VhostUserMsg msg =3D { + .request =3D VHOST_USER_POSTCOPY_LISTEN, + .flags =3D VHOST_USER_VERSION | VHOST_USER_NEED_REPLY_MASK, + }; + + trace_vhost_user_postcopy_listen(); + if (vhost_user_write(dev, &msg, NULL, 0) < 0) { + error_setg(errp, "Failed to send postcopy_listen to vhost"); + return -1; + } + + ret =3D process_message_reply(dev, &msg); + if (ret) { + error_setg(errp, "Failed to receive reply to postcopy_listen"); + return ret; + } + + return 0; +} + static int vhost_user_postcopy_notifier(NotifierWithReturn *notifier, void *opaque) { @@ -810,6 +837,9 @@ static int vhost_user_postcopy_notifier(NotifierWithRet= urn *notifier, case POSTCOPY_NOTIFY_INBOUND_ADVISE: return vhost_user_postcopy_advise(dev, pnd->errp); =20 + case POSTCOPY_NOTIFY_INBOUND_LISTEN: + return vhost_user_postcopy_listen(dev, pnd->errp); + default: /* We ignore notifications we don't know */ break; diff --git a/migration/postcopy-ram.h b/migration/postcopy-ram.h index 28c216cc7a..873c147b68 100644 --- a/migration/postcopy-ram.h +++ b/migration/postcopy-ram.h @@ -129,6 +129,7 @@ void postcopy_infrastructure_init(void); enum PostcopyNotifyReason { POSTCOPY_NOTIFY_PROBE =3D 0, POSTCOPY_NOTIFY_INBOUND_ADVISE, + POSTCOPY_NOTIFY_INBOUND_LISTEN, }; =20 struct PostcopyNotifyData { diff --git a/migration/savevm.c b/migration/savevm.c index d35911731d..72f084e10d 100644 --- a/migration/savevm.c +++ b/migration/savevm.c @@ -1557,6 +1557,8 @@ static int loadvm_postcopy_handle_listen(MigrationInc= omingState *mis) { PostcopyState ps =3D postcopy_state_set(POSTCOPY_INCOMING_LISTENING); trace_loadvm_postcopy_handle_listen(); + Error *local_err =3D NULL; + if (ps !=3D POSTCOPY_INCOMING_ADVISE && ps !=3D POSTCOPY_INCOMING_DISC= ARD) { error_report("CMD_POSTCOPY_LISTEN in wrong postcopy state (%d)", p= s); return -1; @@ -1578,6 +1580,11 @@ static int loadvm_postcopy_handle_listen(MigrationIn= comingState *mis) return -1; } =20 + if (postcopy_notify(POSTCOPY_NOTIFY_INBOUND_LISTEN, &local_err)) { + error_report_err(local_err); + return -1; + } + if (mis->have_listen_thread) { error_report("CMD_POSTCOPY_RAM_LISTEN already has a listen thread"= ); return -1; --=20 2.13.5 From nobody Thu May 2 16:10:00 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1503603804385412.440586110023; Thu, 24 Aug 2017 12:43:24 -0700 (PDT) Received: from localhost ([::1]:50183 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dky2F-0007Xc-5E for importer@patchew.org; Thu, 24 Aug 2017 15:43:23 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52387) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dkxo5-0002ZD-8y for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:28:46 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dkxo4-0004vw-BE for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:28:45 -0400 Received: from mx1.redhat.com ([209.132.183.28]:57329) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dkxo4-0004uz-5o for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:28:44 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 33ACA83F40; Thu, 24 Aug 2017 19:28:43 +0000 (UTC) Received: from dgilbert-t530.redhat.com (ovpn-117-165.ams2.redhat.com [10.36.117.165]) by smtp.corp.redhat.com (Postfix) with ESMTP id D398C17CC4; Thu, 24 Aug 2017 19:28:40 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 33ACA83F40 Authentication-Results: ext-mx03.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx03.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=dgilbert@redhat.com From: "Dr. David Alan Gilbert (git)" To: qemu-devel@nongnu.org, maxime.coquelin@redhat.com, a.perevalov@samsung.com, mst@redhat.com, marcandre.lureau@redhat.com Date: Thu, 24 Aug 2017 20:27:13 +0100 Message-Id: <20170824192730.8440-16-dgilbert@redhat.com> In-Reply-To: <20170824192730.8440-1-dgilbert@redhat.com> References: <20170824192730.8440-1-dgilbert@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.27]); Thu, 24 Aug 2017 19:28:43 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [RFC v2 15/32] vhost+postcopy: Register new regions with the ufd X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lvivier@redhat.com, aarcange@redhat.com, felipe@nutanix.com, peterx@redhat.com, quintela@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: "Dr. David Alan Gilbert" When new regions are sent to the client using SET_MEM_TABLE, register them with the userfaultfd. Signed-off-by: Dr. David Alan Gilbert --- contrib/libvhost-user/libvhost-user.c | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) diff --git a/contrib/libvhost-user/libvhost-user.c b/contrib/libvhost-user/= libvhost-user.c index e8accf11db..e6ab059a03 100644 --- a/contrib/libvhost-user/libvhost-user.c +++ b/contrib/libvhost-user/libvhost-user.c @@ -449,6 +449,38 @@ vu_set_mem_table_exec(VuDev *dev, VhostUserMsg *vmsg) dev_region->mmap_addr); } =20 + if (dev->postcopy_listening) { + /* We should already have an open ufd need to mark each memory + * range as ufd. + * Note: Do we need any madvises? Well it's not been accessed + * yet, still probably need no THP to be safe, discard to be s= afe? + */ + struct uffdio_register reg_struct; + reg_struct.range.start =3D (uintptr_t)dev_region->mmap_addr; + reg_struct.range.len =3D dev_region->size + dev_region->mmap_o= ffset; + reg_struct.mode =3D UFFDIO_REGISTER_MODE_MISSING; + + if (ioctl(dev->postcopy_ufd, UFFDIO_REGISTER, ®_struct)) { + vu_panic(dev, "%s: Failed to userfault region %d " + "@%p + %zx: (ufd=3D%d)%s\n", + __func__, i, + dev_region->mmap_addr, + dev_region->size + dev_region->mmap_offset, + dev->postcopy_ufd, strerror(errno)); + continue; + } + if (!(reg_struct.ioctls & ((__u64)1 << _UFFDIO_COPY))) { + vu_panic(dev, "%s Region (%d) doesn't support COPY", + __func__, i); + continue; + } + DPRINT("%s: region %d: Registered userfault for %llx + %llx\n", + __func__, i, reg_struct.range.start, reg_struct.range.= len); + /* TODO: Stash 'zero' support flags somewhere */ + /* TODO: Get address back to QEMU */ + + } + close(vmsg->fds[i]); } =20 --=20 2.13.5 From nobody Thu May 2 16:10:00 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1503604009365743.3462663178067; Thu, 24 Aug 2017 12:46:49 -0700 (PDT) Received: from localhost ([::1]:50204 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dky5Y-0002ic-5E for importer@patchew.org; Thu, 24 Aug 2017 15:46:48 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52415) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dkxo8-0002cj-AS for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:28:51 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dkxo7-0004zk-1E for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:28:48 -0400 Received: from mx1.redhat.com ([209.132.183.28]:49838) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dkxo6-0004yL-OU for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:28:46 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id C242461479; Thu, 24 Aug 2017 19:28:45 +0000 (UTC) Received: from dgilbert-t530.redhat.com (ovpn-117-165.ams2.redhat.com [10.36.117.165]) by smtp.corp.redhat.com (Postfix) with ESMTP id 7E0A517CC4; Thu, 24 Aug 2017 19:28:43 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com C242461479 Authentication-Results: ext-mx10.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx10.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=dgilbert@redhat.com From: "Dr. David Alan Gilbert (git)" To: qemu-devel@nongnu.org, maxime.coquelin@redhat.com, a.perevalov@samsung.com, mst@redhat.com, marcandre.lureau@redhat.com Date: Thu, 24 Aug 2017 20:27:14 +0100 Message-Id: <20170824192730.8440-17-dgilbert@redhat.com> In-Reply-To: <20170824192730.8440-1-dgilbert@redhat.com> References: <20170824192730.8440-1-dgilbert@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.39]); Thu, 24 Aug 2017 19:28:45 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [RFC v2 16/32] vhost+postcopy: Send address back to qemu X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lvivier@redhat.com, aarcange@redhat.com, felipe@nutanix.com, peterx@redhat.com, quintela@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: "Dr. David Alan Gilbert" We need a better way, but at the moment we need the address of the mappings sent back to qemu so it can interpret the messages on the userfaultfd it reads. Note: We don't ask for the default 'ack' reply since we've got our own. Signed-off-by: Dr. David Alan Gilbert --- contrib/libvhost-user/libvhost-user.c | 15 ++++++++- docs/interop/vhost-user.txt | 6 ++++ hw/virtio/trace-events | 1 + hw/virtio/vhost-user.c | 57 +++++++++++++++++++++++++++++++= +++- 4 files changed, 77 insertions(+), 2 deletions(-) diff --git a/contrib/libvhost-user/libvhost-user.c b/contrib/libvhost-user/= libvhost-user.c index e6ab059a03..5ec54f7d60 100644 --- a/contrib/libvhost-user/libvhost-user.c +++ b/contrib/libvhost-user/libvhost-user.c @@ -477,13 +477,26 @@ vu_set_mem_table_exec(VuDev *dev, VhostUserMsg *vmsg) DPRINT("%s: region %d: Registered userfault for %llx + %llx\n", __func__, i, reg_struct.range.start, reg_struct.range.= len); /* TODO: Stash 'zero' support flags somewhere */ - /* TODO: Get address back to QEMU */ =20 + /* TODO: We need to find a way for the qemu not to see the vir= tual + * addresses of the clients, so as to keep better separation. + */ + /* Return the address to QEMU so that it can translate the ufd + * fault addresses back. + */ + msg_region->userspace_addr =3D (uintptr_t)(mmap_addr + + dev_region->mmap_offs= et); } =20 close(vmsg->fds[i]); } =20 + if (dev->postcopy_listening) { + /* Need to return the addresses - send the updated message back */ + vmsg->fd_num =3D 0; + return true; + } + return false; } =20 diff --git a/docs/interop/vhost-user.txt b/docs/interop/vhost-user.txt index 73c3dd74db..b2a548c94d 100644 --- a/docs/interop/vhost-user.txt +++ b/docs/interop/vhost-user.txt @@ -413,12 +413,18 @@ Master message types Id: 5 Equivalent ioctl: VHOST_SET_MEM_TABLE Master payload: memory regions description + Slave payload: (postcopy only) memory regions description =20 Sets the memory map regions on the slave so it can translate the vri= ng addresses. In the ancillary data there is an array of file descripto= rs for each memory mapped region. The size and ordering of the fds matc= hes the number and ordering of memory regions. =20 + When postcopy-listening has been received, SET_MEM_TABLE replies with + the bases of the memory mapped regions to the master. It must have = mmap'd + the regions and enabled userfaultfd on them. Note NEED_REPLY_MASK + is not set in this case. + * VHOST_USER_SET_LOG_BASE =20 Id: 6 diff --git a/hw/virtio/trace-events b/hw/virtio/trace-events index f736c7c84f..63fd4a79cf 100644 --- a/hw/virtio/trace-events +++ b/hw/virtio/trace-events @@ -2,6 +2,7 @@ =20 # hw/virtio/vhost-user.c vhost_user_postcopy_listen(void) "" +vhost_user_set_mem_table_postcopy(uint64_t client_addr, uint64_t qhva, int= reply_i, int region_i) "client:0x%"PRIx64" for hva: 0x%"PRIx64" reply %d r= egion %d" =20 # hw/virtio/virtio.c virtqueue_alloc_element(void *elem, size_t sz, unsigned in_num, unsigned o= ut_num) "elem %p size %zd in_num %u out_num %u" diff --git a/hw/virtio/vhost-user.c b/hw/virtio/vhost-user.c index 9178271ab2..2e4eb0864a 100644 --- a/hw/virtio/vhost-user.c +++ b/hw/virtio/vhost-user.c @@ -19,6 +19,7 @@ #include "qemu/sockets.h" #include "migration/migration.h" #include "migration/postcopy-ram.h" +#include "trace.h" =20 #include #include @@ -133,6 +134,7 @@ struct vhost_user { int slave_fd; NotifierWithReturn postcopy_notifier; struct PostCopyFD postcopy_fd; + uint64_t postcopy_client_bases[VHOST_MEMORY_MAX_NREGIONS]; }; =20 static bool ioeventfd_enabled(void) @@ -300,11 +302,13 @@ static int vhost_user_set_log_base(struct vhost_dev *= dev, uint64_t base, static int vhost_user_set_mem_table(struct vhost_dev *dev, struct vhost_memory *mem) { + struct vhost_user *u =3D dev->opaque; int fds[VHOST_MEMORY_MAX_NREGIONS]; int i, fd; size_t fd_num =3D 0; bool reply_supported =3D virtio_has_feature(dev->protocol_features, - VHOST_USER_PROTOCOL_F_REPLY_= ACK); + VHOST_USER_PROTOCOL_F_REPLY_ACK)= && + !u->postcopy_fd.handler; =20 VhostUserMsg msg =3D { .request =3D VHOST_USER_SET_MEM_TABLE, @@ -350,6 +354,57 @@ static int vhost_user_set_mem_table(struct vhost_dev *= dev, return -1; } =20 + if (u->postcopy_fd.handler) { + VhostUserMsg msg_reply; + int region_i, reply_i; + if (vhost_user_read(dev, &msg_reply) < 0) { + return -1; + } + + if (msg_reply.request !=3D VHOST_USER_SET_MEM_TABLE) { + error_report("%s: Received unexpected msg type." + "Expected %d received %d", __func__, + VHOST_USER_SET_MEM_TABLE, msg_reply.request); + return -1; + } + /* We're using the same structure, just reusing one of the + * fields, so it should be the same size. + */ + if (msg_reply.size !=3D msg.size) { + error_report("%s: Unexpected size for postcopy reply " + "%d vs %d", __func__, msg_reply.size, msg.size); + return -1; + } + + memset(u->postcopy_client_bases, 0, + sizeof(uint64_t) * VHOST_MEMORY_MAX_NREGIONS); + + /* They're in the same order as the regions that were sent + * but some of the regions were skipped (above) if they + * didn't have fd's + */ + for (reply_i =3D 0, region_i =3D 0; + region_i < dev->mem->nregions; + region_i++) { + if (reply_i < fd_num && + msg_reply.payload.memory.regions[region_i].guest_phys_addr= =3D=3D + dev->mem->regions[region_i].guest_phys_addr) { + u->postcopy_client_bases[region_i] =3D + msg_reply.payload.memory.regions[reply_i].userspace_ad= dr; + trace_vhost_user_set_mem_table_postcopy( + msg_reply.payload.memory.regions[reply_i].userspace_ad= dr, + msg.payload.memory.regions[reply_i].userspace_addr, + reply_i, region_i); + reply_i++; + } + } + if (reply_i !=3D fd_num) { + error_report("%s: postcopy reply not fully consumed " + "%d vs %zd", + __func__, reply_i, fd_num); + return -1; + } + } if (reply_supported) { return process_message_reply(dev, &msg); } --=20 2.13.5 From nobody Thu May 2 16:10:00 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1503604532184611.393794694984; Thu, 24 Aug 2017 12:55:32 -0700 (PDT) Received: from localhost ([::1]:50256 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dkyDy-00024o-OZ for importer@patchew.org; Thu, 24 Aug 2017 15:55:30 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52573) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dkxoN-0002re-4k for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:29:04 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dkxoK-00057n-13 for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:29:03 -0400 Received: from mx1.redhat.com ([209.132.183.28]:50780) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dkxoJ-00056V-OD for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:28:59 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 7E1FD4A6FD; Thu, 24 Aug 2017 19:28:58 +0000 (UTC) Received: from dgilbert-t530.redhat.com (ovpn-117-165.ams2.redhat.com [10.36.117.165]) by smtp.corp.redhat.com (Postfix) with ESMTP id 194C017CC4; Thu, 24 Aug 2017 19:28:45 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 7E1FD4A6FD Authentication-Results: ext-mx09.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx09.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=dgilbert@redhat.com From: "Dr. David Alan Gilbert (git)" To: qemu-devel@nongnu.org, maxime.coquelin@redhat.com, a.perevalov@samsung.com, mst@redhat.com, marcandre.lureau@redhat.com Date: Thu, 24 Aug 2017 20:27:15 +0100 Message-Id: <20170824192730.8440-18-dgilbert@redhat.com> In-Reply-To: <20170824192730.8440-1-dgilbert@redhat.com> References: <20170824192730.8440-1-dgilbert@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.38]); Thu, 24 Aug 2017 19:28:58 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [RFC v2 17/32] vhost+postcopy: Stash RAMBlock and offset X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lvivier@redhat.com, aarcange@redhat.com, felipe@nutanix.com, peterx@redhat.com, quintela@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: "Dr. David Alan Gilbert" Stash the RAMBlock and offset for later use looking up addresses. Signed-off-by: Dr. David Alan Gilbert --- hw/virtio/trace-events | 1 + hw/virtio/vhost-user.c | 30 ++++++++++++++++++++++++++++++ 2 files changed, 31 insertions(+) diff --git a/hw/virtio/trace-events b/hw/virtio/trace-events index 63fd4a79cf..5067dee19b 100644 --- a/hw/virtio/trace-events +++ b/hw/virtio/trace-events @@ -3,6 +3,7 @@ # hw/virtio/vhost-user.c vhost_user_postcopy_listen(void) "" vhost_user_set_mem_table_postcopy(uint64_t client_addr, uint64_t qhva, int= reply_i, int region_i) "client:0x%"PRIx64" for hva: 0x%"PRIx64" reply %d r= egion %d" +vhost_user_set_mem_table_withfd(int index, const char *name, uint64_t memo= ry_size, uint64_t guest_phys_addr, uint64_t userspace_addr, uint64_t offset= ) "%d:%s: size:0x%"PRIx64" GPA:0x%"PRIx64" QVA/userspace:0x%"PRIx64" RB off= set:0x%"PRIx64 =20 # hw/virtio/virtio.c virtqueue_alloc_element(void *elem, size_t sz, unsigned in_num, unsigned o= ut_num) "elem %p size %zd in_num %u out_num %u" diff --git a/hw/virtio/vhost-user.c b/hw/virtio/vhost-user.c index 2e4eb0864a..fbe2743298 100644 --- a/hw/virtio/vhost-user.c +++ b/hw/virtio/vhost-user.c @@ -135,6 +135,14 @@ struct vhost_user { NotifierWithReturn postcopy_notifier; struct PostCopyFD postcopy_fd; uint64_t postcopy_client_bases[VHOST_MEMORY_MAX_NREGIONS]; + /* Length of the region_rb and region_rb_offset arrays */ + size_t region_rb_len; + /* RAMBlock associated with a given region */ + RAMBlock **region_rb; + /* The offset from the start of the RAMBlock to the start of the + * vhost region. + */ + ram_addr_t *region_rb_offset; }; =20 static bool ioeventfd_enabled(void) @@ -319,6 +327,17 @@ static int vhost_user_set_mem_table(struct vhost_dev *= dev, msg.flags |=3D VHOST_USER_NEED_REPLY_MASK; } =20 + if (u->region_rb_len < dev->mem->nregions) { + u->region_rb =3D g_renew(RAMBlock*, u->region_rb, dev->mem->nregio= ns); + u->region_rb_offset =3D g_renew(ram_addr_t, u->region_rb_offset, + dev->mem->nregions); + memset(&(u->region_rb[u->region_rb_len]), '\0', + sizeof(RAMBlock *) * (dev->mem->nregions - u->region_rb_len= )); + memset(&(u->region_rb_offset[u->region_rb_len]), '\0', + sizeof(ram_addr_t) * (dev->mem->nregions - u->region_rb_len= )); + u->region_rb_len =3D dev->mem->nregions; + } + for (i =3D 0; i < dev->mem->nregions; ++i) { struct vhost_memory_region *reg =3D dev->mem->regions + i; ram_addr_t offset; @@ -327,8 +346,14 @@ static int vhost_user_set_mem_table(struct vhost_dev *= dev, assert((uintptr_t)reg->userspace_addr =3D=3D reg->userspace_addr); mr =3D memory_region_from_host((void *)(uintptr_t)reg->userspace_a= ddr, &offset); + u->region_rb_offset[i] =3D offset; + u->region_rb[i] =3D mr->ram_block; fd =3D memory_region_get_fd(mr); if (fd > 0) { + trace_vhost_user_set_mem_table_withfd(fd_num, mr->name, + reg->memory_size, + reg->guest_phys_addr, + reg->userspace_addr, off= set); msg.payload.memory.regions[fd_num].userspace_addr =3D reg->use= rspace_addr; msg.payload.memory.regions[fd_num].memory_size =3D reg->memor= y_size; msg.payload.memory.regions[fd_num].guest_phys_addr =3D reg->gu= est_phys_addr; @@ -992,6 +1017,11 @@ static int vhost_user_cleanup(struct vhost_dev *dev) close(u->slave_fd); u->slave_fd =3D -1; } + g_free(u->region_rb); + u->region_rb =3D NULL; + g_free(u->region_rb_offset); + u->region_rb_offset =3D NULL; + u->region_rb_len =3D 0; g_free(u); dev->opaque =3D 0; =20 --=20 2.13.5 From nobody Thu May 2 16:10:00 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1503603955082828.9941175652627; Thu, 24 Aug 2017 12:45:55 -0700 (PDT) Received: from localhost ([::1]:50199 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dky4f-0001yd-T8 for importer@patchew.org; Thu, 24 Aug 2017 15:45:53 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52578) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dkxoN-0002s8-IP for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:29:04 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dkxoM-00059r-7X for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:29:03 -0400 Received: from mx1.redhat.com ([209.132.183.28]:50298) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dkxoL-00059G-VU for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:29:02 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id E11D75F742; Thu, 24 Aug 2017 19:29:00 +0000 (UTC) Received: from dgilbert-t530.redhat.com (ovpn-117-165.ams2.redhat.com [10.36.117.165]) by smtp.corp.redhat.com (Postfix) with ESMTP id C971217CC4; Thu, 24 Aug 2017 19:28:58 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com E11D75F742 Authentication-Results: ext-mx10.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx10.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=dgilbert@redhat.com From: "Dr. David Alan Gilbert (git)" To: qemu-devel@nongnu.org, maxime.coquelin@redhat.com, a.perevalov@samsung.com, mst@redhat.com, marcandre.lureau@redhat.com Date: Thu, 24 Aug 2017 20:27:16 +0100 Message-Id: <20170824192730.8440-19-dgilbert@redhat.com> In-Reply-To: <20170824192730.8440-1-dgilbert@redhat.com> References: <20170824192730.8440-1-dgilbert@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.39]); Thu, 24 Aug 2017 19:29:01 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [RFC v2 18/32] vhost+postcopy: Send requests to source for shared pages X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lvivier@redhat.com, aarcange@redhat.com, felipe@nutanix.com, peterx@redhat.com, quintela@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: "Dr. David Alan Gilbert" Send requests back to the source for shared page requests. Signed-off-by: Dr. David Alan Gilbert --- migration/migration.h | 2 ++ migration/postcopy-ram.c | 31 ++++++++++++++++++++++++++++--- migration/postcopy-ram.h | 3 +++ migration/trace-events | 2 ++ 4 files changed, 35 insertions(+), 3 deletions(-) diff --git a/migration/migration.h b/migration/migration.h index 9fcea6bb25..214b0b6afd 100644 --- a/migration/migration.h +++ b/migration/migration.h @@ -46,6 +46,8 @@ struct MigrationIncomingState { int userfault_quit_fd; QEMUFile *to_src_file; QemuMutex rp_mutex; /* We send replies from multiple threads */ + /* RAMBlock of last request sent to source */ + RAMBlock *last_rb; void *postcopy_tmp_page; void *postcopy_tmp_zero_page; /* PostCopyFD's for external userfaultfds & handlers of shared memory = */ diff --git a/migration/postcopy-ram.c b/migration/postcopy-ram.c index faee7708ff..2d77674b94 100644 --- a/migration/postcopy-ram.c +++ b/migration/postcopy-ram.c @@ -459,6 +459,31 @@ static int ram_block_enable_notify(const char *block_n= ame, void *host_addr, } =20 /* + * Callback from shared fault handlers to ask for a page, + * the page must be specified by a RAMBlock and an offset in that rb + */ +int postcopy_request_shared_page(struct PostCopyFD *pcfd, RAMBlock *rb, + uint64_t client_addr, uint64_t rb_offset) +{ + size_t pagesize =3D qemu_ram_pagesize(rb); + uint64_t aligned_rbo =3D rb_offset & ~(pagesize - 1); + MigrationIncomingState *mis =3D migration_incoming_get_current(); + + trace_postcopy_request_shared_page(pcfd->idstr, qemu_ram_get_idstr(rb), + rb_offset); + /* TODO: Check bitmap to see if we already have the page */ + if (rb !=3D mis->last_rb) { + mis->last_rb =3D rb; + migrate_send_rp_req_pages(mis, qemu_ram_get_idstr(rb), + aligned_rbo, pagesize); + } else { + /* Save some space */ + migrate_send_rp_req_pages(mis, NULL, aligned_rbo, pagesize); + } + return 0; +} + +/* * Handle faults detected by the USERFAULT markings */ static void *postcopy_ram_fault_thread(void *opaque) @@ -468,9 +493,9 @@ static void *postcopy_ram_fault_thread(void *opaque) int ret; size_t index; RAMBlock *rb =3D NULL; - RAMBlock *last_rb =3D NULL; /* last RAMBlock we sent part of */ =20 trace_postcopy_ram_fault_thread_entry(); + mis->last_rb =3D NULL; /* last RAMBlock we sent part of */ qemu_sem_post(&mis->fault_thread_sem); =20 struct pollfd *pfd; @@ -559,8 +584,8 @@ static void *postcopy_ram_fault_thread(void *opaque) * Send the request to the source - we want to request one * of our host page sizes (which is >=3D TPS) */ - if (rb !=3D last_rb) { - last_rb =3D rb; + if (rb !=3D mis->last_rb) { + mis->last_rb =3D rb; migrate_send_rp_req_pages(mis, qemu_ram_get_idstr(rb), rb_offset, qemu_ram_pagesize(rb)); } else { diff --git a/migration/postcopy-ram.h b/migration/postcopy-ram.h index 873c147b68..69e88b0174 100644 --- a/migration/postcopy-ram.h +++ b/migration/postcopy-ram.h @@ -162,5 +162,8 @@ struct PostCopyFD { */ void postcopy_register_shared_ufd(struct PostCopyFD *pcfd); void postcopy_unregister_shared_ufd(struct PostCopyFD *pcfd); +/* Callback from shared fault handlers to ask for a page */ +int postcopy_request_shared_page(struct PostCopyFD *pcfd, RAMBlock *rb, + uint64_t client_addr, uint64_t offset); =20 #endif diff --git a/migration/trace-events b/migration/trace-events index 23f4e5339b..3a0b143f7e 100644 --- a/migration/trace-events +++ b/migration/trace-events @@ -197,6 +197,8 @@ postcopy_ram_incoming_cleanup_closeuf(void) "" postcopy_ram_incoming_cleanup_entry(void) "" postcopy_ram_incoming_cleanup_exit(void) "" postcopy_ram_incoming_cleanup_join(void) "" +postcopy_request_shared_page(const char *sharer, const char *rb, uint64_t = rb_offset) "for %s in %s offset 0x%"PRIx64 + save_xbzrle_page_skipping(void) "" save_xbzrle_page_overflow(void) "" ram_save_iterate_big_wait(uint64_t milliconds, int iterations) "big wait: = %" PRIu64 " milliseconds, %d iterations" --=20 2.13.5 From nobody Thu May 2 16:10:00 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 15036033430181016.796123967748; Thu, 24 Aug 2017 12:35:43 -0700 (PDT) Received: from localhost ([::1]:50147 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dkxun-0000N7-Qv for importer@patchew.org; Thu, 24 Aug 2017 15:35:41 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52609) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dkxoP-0002vT-PC for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:29:06 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dkxoO-0005Bk-Mj for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:29:05 -0400 Received: from mx1.redhat.com ([209.132.183.28]:53988) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dkxoO-0005B1-Dr for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:29:04 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 6C5F1C058ECB; Thu, 24 Aug 2017 19:29:03 +0000 (UTC) Received: from dgilbert-t530.redhat.com (ovpn-117-165.ams2.redhat.com [10.36.117.165]) by smtp.corp.redhat.com (Postfix) with ESMTP id 340DC17CC4; Thu, 24 Aug 2017 19:29:01 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 6C5F1C058ECB Authentication-Results: ext-mx08.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx08.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=dgilbert@redhat.com From: "Dr. David Alan Gilbert (git)" To: qemu-devel@nongnu.org, maxime.coquelin@redhat.com, a.perevalov@samsung.com, mst@redhat.com, marcandre.lureau@redhat.com Date: Thu, 24 Aug 2017 20:27:17 +0100 Message-Id: <20170824192730.8440-20-dgilbert@redhat.com> In-Reply-To: <20170824192730.8440-1-dgilbert@redhat.com> References: <20170824192730.8440-1-dgilbert@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.32]); Thu, 24 Aug 2017 19:29:03 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [RFC v2 19/32] vhost+postcopy: Resolve client address X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lvivier@redhat.com, aarcange@redhat.com, felipe@nutanix.com, peterx@redhat.com, quintela@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: "Dr. David Alan Gilbert" Resolve fault addresses read off the clients UFD into RAMBlock and offset, and call back to the postcopy code to ask for the page. Signed-off-by: Dr. David Alan Gilbert --- hw/virtio/trace-events | 3 +++ hw/virtio/vhost-user.c | 30 +++++++++++++++++++++++++++++- 2 files changed, 32 insertions(+), 1 deletion(-) diff --git a/hw/virtio/trace-events b/hw/virtio/trace-events index 5067dee19b..f7d4b831fe 100644 --- a/hw/virtio/trace-events +++ b/hw/virtio/trace-events @@ -1,6 +1,9 @@ # See docs/devel/tracing.txt for syntax documentation. =20 # hw/virtio/vhost-user.c +vhost_user_postcopy_fault_handler(const char *name, uint64_t fault_address= , int nregions) "%s: @0x%"PRIx64" nregions:%d" +vhost_user_postcopy_fault_handler_loop(int i, uint64_t client_base, uint64= _t size) "%d: client 0x%"PRIx64" +0x%"PRIx64 +vhost_user_postcopy_fault_handler_found(int i, uint64_t region_offset, uin= t64_t rb_offset) "%d: region_offset: 0x%"PRIx64" rb_offset:0x%"PRIx64 vhost_user_postcopy_listen(void) "" vhost_user_set_mem_table_postcopy(uint64_t client_addr, uint64_t qhva, int= reply_i, int region_i) "client:0x%"PRIx64" for hva: 0x%"PRIx64" reply %d r= egion %d" vhost_user_set_mem_table_withfd(int index, const char *name, uint64_t memo= ry_size, uint64_t guest_phys_addr, uint64_t userspace_addr, uint64_t offset= ) "%d:%s: size:0x%"PRIx64" GPA:0x%"PRIx64" QVA/userspace:0x%"PRIx64" RB off= set:0x%"PRIx64 diff --git a/hw/virtio/vhost-user.c b/hw/virtio/vhost-user.c index fbe2743298..2897ff70b3 100644 --- a/hw/virtio/vhost-user.c +++ b/hw/virtio/vhost-user.c @@ -816,7 +816,35 @@ out: static int vhost_user_postcopy_fault_handler(struct PostCopyFD *pcfd, void *ufd) { - return 0; + struct vhost_dev *dev =3D pcfd->data; + struct vhost_user *u =3D dev->opaque; + struct uffd_msg *msg =3D ufd; + uint64_t faultaddr =3D msg->arg.pagefault.address; + RAMBlock *rb =3D NULL; + uint64_t rb_offset; + int i; + + trace_vhost_user_postcopy_fault_handler(pcfd->idstr, faultaddr, + dev->mem->nregions); + for (i =3D 0; i < MIN(dev->mem->nregions, u->region_rb_len); i++) { + trace_vhost_user_postcopy_fault_handler_loop(i, + u->postcopy_client_bases[i], dev->mem->regions[i].memory_s= ize); + if (faultaddr >=3D u->postcopy_client_bases[i]) { + /* Ofset of the fault address in the vhost region */ + uint64_t region_offset =3D faultaddr - u->postcopy_client_base= s[i]; + if (region_offset <=3D dev->mem->regions[i].memory_size) { + rb_offset =3D region_offset + u->region_rb_offset[i]; + trace_vhost_user_postcopy_fault_handler_found(i, + region_offset, rb_offset); + rb =3D u->region_rb[i]; + return postcopy_request_shared_page(pcfd, rb, faultaddr, + rb_offset); + } + } + } + error_report("%s: Failed to find region for fault %" PRIx64, + __func__, faultaddr); + return -1; } =20 /* --=20 2.13.5 From nobody Thu May 2 16:10:00 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1503604182650206.47536959668446; Thu, 24 Aug 2017 12:49:42 -0700 (PDT) Received: from localhost ([::1]:50215 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dky8L-0004vV-DR for importer@patchew.org; Thu, 24 Aug 2017 15:49:41 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52677) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dkxob-000369-Ug for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:29:18 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dkxoY-0005Hq-QW for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:29:17 -0400 Received: from mx1.redhat.com ([209.132.183.28]:51628) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dkxoY-0005HB-IV for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:29:14 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 90ACA883BA; Thu, 24 Aug 2017 19:29:13 +0000 (UTC) Received: from dgilbert-t530.redhat.com (ovpn-117-165.ams2.redhat.com [10.36.117.165]) by smtp.corp.redhat.com (Postfix) with ESMTP id B65C217CC4; Thu, 24 Aug 2017 19:29:03 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 90ACA883BA Authentication-Results: ext-mx02.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx02.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=dgilbert@redhat.com From: "Dr. David Alan Gilbert (git)" To: qemu-devel@nongnu.org, maxime.coquelin@redhat.com, a.perevalov@samsung.com, mst@redhat.com, marcandre.lureau@redhat.com Date: Thu, 24 Aug 2017 20:27:18 +0100 Message-Id: <20170824192730.8440-21-dgilbert@redhat.com> In-Reply-To: <20170824192730.8440-1-dgilbert@redhat.com> References: <20170824192730.8440-1-dgilbert@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.26]); Thu, 24 Aug 2017 19:29:13 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [RFC v2 20/32] postcopy: wake shared X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lvivier@redhat.com, aarcange@redhat.com, felipe@nutanix.com, peterx@redhat.com, quintela@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: "Dr. David Alan Gilbert" Send a 'wake' request on a userfaultfd for a shared process. The address in the clients address space is specified together with the RAMBlock it was resolved to. Signed-off-by: Dr. David Alan Gilbert --- migration/postcopy-ram.c | 26 ++++++++++++++++++++++++++ migration/postcopy-ram.h | 6 ++++++ migration/trace-events | 1 + 3 files changed, 33 insertions(+) diff --git a/migration/postcopy-ram.c b/migration/postcopy-ram.c index 2d77674b94..2c9680ef7a 100644 --- a/migration/postcopy-ram.c +++ b/migration/postcopy-ram.c @@ -458,6 +458,25 @@ static int ram_block_enable_notify(const char *block_n= ame, void *host_addr, return 0; } =20 +int postcopy_wake_shared(struct PostCopyFD *pcfd, + uint64_t client_addr, + RAMBlock *rb) +{ + size_t pagesize =3D qemu_ram_pagesize(rb); + struct uffdio_range range; + int ret; + trace_postcopy_wake_shared(client_addr, qemu_ram_get_idstr(rb)); + range.start =3D client_addr & ~(pagesize - 1); + range.len =3D pagesize; + ret =3D ioctl(pcfd->fd, UFFDIO_WAKE, &range); + if (ret) { + error_report("%s: Failed to wake: %zx in %s (%s)", + __func__, client_addr, qemu_ram_get_idstr(rb), + strerror(errno)); + } + return ret; +} + /* * Callback from shared fault handlers to ask for a page, * the page must be specified by a RAMBlock and an offset in that rb @@ -876,6 +895,13 @@ void *postcopy_get_tmp_page(MigrationIncomingState *mi= s) return NULL; } =20 +int postcopy_wake_shared(struct PostCopyFD *pcfd, + uint64_t client_addr, + RAMBlock *rb) +{ + assert(0); + return -1; +} #endif =20 /* -----------------------------------------------------------------------= -- */ diff --git a/migration/postcopy-ram.h b/migration/postcopy-ram.h index 69e88b0174..d2b2f5f4aa 100644 --- a/migration/postcopy-ram.h +++ b/migration/postcopy-ram.h @@ -162,6 +162,12 @@ struct PostCopyFD { */ void postcopy_register_shared_ufd(struct PostCopyFD *pcfd); void postcopy_unregister_shared_ufd(struct PostCopyFD *pcfd); +/* Notify a client ufd that a page is available + * Note: The 'client_address' is in the address space of the client + * program not QEMU + */ +int postcopy_wake_shared(struct PostCopyFD *pcfd, uint64_t client_addr, + RAMBlock *rb); /* Callback from shared fault handlers to ask for a page */ int postcopy_request_shared_page(struct PostCopyFD *pcfd, RAMBlock *rb, uint64_t client_addr, uint64_t offset); diff --git a/migration/trace-events b/migration/trace-events index 3a0b143f7e..535e7ad84b 100644 --- a/migration/trace-events +++ b/migration/trace-events @@ -198,6 +198,7 @@ postcopy_ram_incoming_cleanup_entry(void) "" postcopy_ram_incoming_cleanup_exit(void) "" postcopy_ram_incoming_cleanup_join(void) "" postcopy_request_shared_page(const char *sharer, const char *rb, uint64_t = rb_offset) "for %s in %s offset 0x%"PRIx64 +postcopy_wake_shared(uint64_t client_addr, const char *rb) "at 0x%"PRIx64"= in %s" =20 save_xbzrle_page_skipping(void) "" save_xbzrle_page_overflow(void) "" --=20 2.13.5 From nobody Thu May 2 16:10:00 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1503604665567115.12010173802776; Thu, 24 Aug 2017 12:57:45 -0700 (PDT) Received: from localhost ([::1]:50272 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dkyG8-0004Lt-Eg for importer@patchew.org; Thu, 24 Aug 2017 15:57:44 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52725) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dkxoe-00038T-JP for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:29:21 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dkxod-0005LI-Lk for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:29:20 -0400 Received: from mx1.redhat.com ([209.132.183.28]:51880) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dkxod-0005KL-CZ for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:29:19 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 673E8883BA; Thu, 24 Aug 2017 19:29:18 +0000 (UTC) Received: from dgilbert-t530.redhat.com (ovpn-117-165.ams2.redhat.com [10.36.117.165]) by smtp.corp.redhat.com (Postfix) with ESMTP id DAFCE17CC4; Thu, 24 Aug 2017 19:29:13 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 673E8883BA Authentication-Results: ext-mx02.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx02.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=dgilbert@redhat.com From: "Dr. David Alan Gilbert (git)" To: qemu-devel@nongnu.org, maxime.coquelin@redhat.com, a.perevalov@samsung.com, mst@redhat.com, marcandre.lureau@redhat.com Date: Thu, 24 Aug 2017 20:27:19 +0100 Message-Id: <20170824192730.8440-22-dgilbert@redhat.com> In-Reply-To: <20170824192730.8440-1-dgilbert@redhat.com> References: <20170824192730.8440-1-dgilbert@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.26]); Thu, 24 Aug 2017 19:29:18 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [RFC v2 21/32] postcopy: postcopy_notify_shared_wake X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lvivier@redhat.com, aarcange@redhat.com, felipe@nutanix.com, peterx@redhat.com, quintela@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: "Dr. David Alan Gilbert" Add a hook to allow a client userfaultfd to be 'woken' when a page arrives, and a walker that calls that hook for relevant clients given a RAMBlock and offset. Signed-off-by: Dr. David Alan Gilbert --- migration/postcopy-ram.c | 16 ++++++++++++++++ migration/postcopy-ram.h | 10 ++++++++++ 2 files changed, 26 insertions(+) diff --git a/migration/postcopy-ram.c b/migration/postcopy-ram.c index 2c9680ef7a..40b58a7912 100644 --- a/migration/postcopy-ram.c +++ b/migration/postcopy-ram.c @@ -749,6 +749,22 @@ static int qemu_ufd_copy_ioctl(int userfault_fd, void = *host_addr, return ret; } =20 +int postcopy_notify_shared_wake(RAMBlock *rb, uint64_t offset) +{ + int i; + MigrationIncomingState *mis =3D migration_incoming_get_current(); + GArray *pcrfds =3D mis->postcopy_remote_fds; + + for (i =3D 0; i < pcrfds->len; i++) { + struct PostCopyFD *cur =3D &g_array_index(pcrfds, struct PostCopyF= D, i); + int ret =3D cur->waker(cur, rb, offset); + if (ret) { + return ret; + } + } + return 0; +} + /* * Place a host page (from) at (host) atomically * returns 0 on success diff --git a/migration/postcopy-ram.h b/migration/postcopy-ram.h index d2b2f5f4aa..ecf731c689 100644 --- a/migration/postcopy-ram.h +++ b/migration/postcopy-ram.h @@ -146,6 +146,10 @@ struct PostCopyFD; =20 /* ufd is a pointer to the struct uffd_msg *TODO: more Portable! */ typedef int (*pcfdhandler)(struct PostCopyFD *pcfd, void *ufd); +/* Notification to wake, either on place or on reception of + * a fault on something that's already arrived (race) + */ +typedef int (*pcfdwake)(struct PostCopyFD *pcfd, RAMBlock *rb, uint64_t of= fset); =20 struct PostCopyFD { int fd; @@ -153,6 +157,8 @@ struct PostCopyFD { void *data; /* Handler to be called whenever we get a poll event */ pcfdhandler handler; + /* Notification to wake shared client */ + pcfdwake waker; /* A string to use in error messages */ const char *idstr; }; @@ -162,6 +168,10 @@ struct PostCopyFD { */ void postcopy_register_shared_ufd(struct PostCopyFD *pcfd); void postcopy_unregister_shared_ufd(struct PostCopyFD *pcfd); +/* Call each of the shared 'waker's registerd telling them of + * availability of a block. + */ +int postcopy_notify_shared_wake(RAMBlock *rb, uint64_t offset); /* Notify a client ufd that a page is available * Note: The 'client_address' is in the address space of the client * program not QEMU --=20 2.13.5 From nobody Thu May 2 16:10:00 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1503604348024258.07992602709226; Thu, 24 Aug 2017 12:52:28 -0700 (PDT) Received: from localhost ([::1]:50230 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dkyB0-0007nW-Ln for importer@patchew.org; Thu, 24 Aug 2017 15:52:26 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52748) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dkxoh-0003Bf-Cd for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:29:26 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dkxog-0005O1-Al for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:29:23 -0400 Received: from mx1.redhat.com ([209.132.183.28]:51240) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dkxog-0005ML-2C for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:29:22 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id EAB6F61462; Thu, 24 Aug 2017 19:29:20 +0000 (UTC) Received: from dgilbert-t530.redhat.com (ovpn-117-165.ams2.redhat.com [10.36.117.165]) by smtp.corp.redhat.com (Postfix) with ESMTP id A380A17CC4; Thu, 24 Aug 2017 19:29:18 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com EAB6F61462 Authentication-Results: ext-mx10.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx10.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=dgilbert@redhat.com From: "Dr. David Alan Gilbert (git)" To: qemu-devel@nongnu.org, maxime.coquelin@redhat.com, a.perevalov@samsung.com, mst@redhat.com, marcandre.lureau@redhat.com Date: Thu, 24 Aug 2017 20:27:20 +0100 Message-Id: <20170824192730.8440-23-dgilbert@redhat.com> In-Reply-To: <20170824192730.8440-1-dgilbert@redhat.com> References: <20170824192730.8440-1-dgilbert@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.39]); Thu, 24 Aug 2017 19:29:21 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [RFC v2 22/32] vhost+postcopy: Add vhost waker X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lvivier@redhat.com, aarcange@redhat.com, felipe@nutanix.com, peterx@redhat.com, quintela@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: "Dr. David Alan Gilbert" Register a waker function in vhost-user code to be notified when pages arrive or requests to previously mapped pages get requested. Signed-off-by: Dr. David Alan Gilbert --- hw/virtio/trace-events | 3 +++ hw/virtio/vhost-user.c | 26 ++++++++++++++++++++++++++ 2 files changed, 29 insertions(+) diff --git a/hw/virtio/trace-events b/hw/virtio/trace-events index f7d4b831fe..adebf6dc6b 100644 --- a/hw/virtio/trace-events +++ b/hw/virtio/trace-events @@ -7,6 +7,9 @@ vhost_user_postcopy_fault_handler_found(int i, uint64_t reg= ion_offset, uint64_t vhost_user_postcopy_listen(void) "" vhost_user_set_mem_table_postcopy(uint64_t client_addr, uint64_t qhva, int= reply_i, int region_i) "client:0x%"PRIx64" for hva: 0x%"PRIx64" reply %d r= egion %d" vhost_user_set_mem_table_withfd(int index, const char *name, uint64_t memo= ry_size, uint64_t guest_phys_addr, uint64_t userspace_addr, uint64_t offset= ) "%d:%s: size:0x%"PRIx64" GPA:0x%"PRIx64" QVA/userspace:0x%"PRIx64" RB off= set:0x%"PRIx64 +vhost_user_postcopy_waker(const char *rb, uint64_t rb_offset) "%s + 0x%"PR= Ix64 +vhost_user_postcopy_waker_found(uint64_t client_addr) "0x%"PRIx64 +vhost_user_postcopy_waker_nomatch(const char *rb, uint64_t rb_offset) "%s = + 0x%"PRIx64 =20 # hw/virtio/virtio.c virtqueue_alloc_element(void *elem, size_t sz, unsigned in_num, unsigned o= ut_num) "elem %p size %zd in_num %u out_num %u" diff --git a/hw/virtio/vhost-user.c b/hw/virtio/vhost-user.c index 2897ff70b3..3bff33a1a6 100644 --- a/hw/virtio/vhost-user.c +++ b/hw/virtio/vhost-user.c @@ -847,6 +847,31 @@ static int vhost_user_postcopy_fault_handler(struct Po= stCopyFD *pcfd, return -1; } =20 +static int vhost_user_postcopy_waker(struct PostCopyFD *pcfd, RAMBlock *rb, + uint64_t offset) +{ + struct vhost_dev *dev =3D pcfd->data; + struct vhost_user *u =3D dev->opaque; + int i; + + trace_vhost_user_postcopy_waker(qemu_ram_get_idstr(rb), offset); + /* Translate the offset into an address in the clients address space */ + for (i =3D 0; i < MIN(dev->mem->nregions, u->region_rb_len); i++) { + if (u->region_rb[i] =3D=3D rb && + offset >=3D u->region_rb_offset[i] && + offset < (u->region_rb_offset[i] + + dev->mem->regions[i].memory_size)) { + uint64_t client_addr =3D (offset - u->region_rb_offset[i]) + + u->postcopy_client_bases[i]; + trace_vhost_user_postcopy_waker_found(client_addr); + return postcopy_wake_shared(pcfd, client_addr, rb); + } + } + + trace_vhost_user_postcopy_waker_nomatch(qemu_ram_get_idstr(rb), offset= ); + return 0; +} + /* * Called at the start of an inbound postcopy on reception of the * 'advise' command. @@ -892,6 +917,7 @@ static int vhost_user_postcopy_advise(struct vhost_dev = *dev, Error **errp) u->postcopy_fd.fd =3D ufd; u->postcopy_fd.data =3D dev; u->postcopy_fd.handler =3D vhost_user_postcopy_fault_handler; + u->postcopy_fd.waker =3D vhost_user_postcopy_waker; u->postcopy_fd.idstr =3D "vhost-user"; /* Need to find unique name */ postcopy_register_shared_ufd(&u->postcopy_fd); return 0; --=20 2.13.5 From nobody Thu May 2 16:10:00 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1503603513116762.273763439121; Thu, 24 Aug 2017 12:38:33 -0700 (PDT) Received: from localhost ([::1]:50160 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dkxxX-0002vN-VA for importer@patchew.org; Thu, 24 Aug 2017 15:38:31 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52796) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dkxok-0003EO-KF for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:29:27 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dkxoi-0005PM-Jg for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:29:26 -0400 Received: from mx1.redhat.com ([209.132.183.28]:43544) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dkxoi-0005Ow-AV for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:29:24 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 5B84880F75; Thu, 24 Aug 2017 19:29:23 +0000 (UTC) Received: from dgilbert-t530.redhat.com (ovpn-117-165.ams2.redhat.com [10.36.117.165]) by smtp.corp.redhat.com (Postfix) with ESMTP id 3FE3A17CC4; Thu, 24 Aug 2017 19:29:21 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 5B84880F75 Authentication-Results: ext-mx03.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx03.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=dgilbert@redhat.com From: "Dr. David Alan Gilbert (git)" To: qemu-devel@nongnu.org, maxime.coquelin@redhat.com, a.perevalov@samsung.com, mst@redhat.com, marcandre.lureau@redhat.com Date: Thu, 24 Aug 2017 20:27:21 +0100 Message-Id: <20170824192730.8440-24-dgilbert@redhat.com> In-Reply-To: <20170824192730.8440-1-dgilbert@redhat.com> References: <20170824192730.8440-1-dgilbert@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.27]); Thu, 24 Aug 2017 19:29:23 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [RFC v2 23/32] vhost+postcopy: Call wakeups X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lvivier@redhat.com, aarcange@redhat.com, felipe@nutanix.com, peterx@redhat.com, quintela@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: "Dr. David Alan Gilbert" Cause the vhost-user client to be woken up whenever: a) We place a page in postcopy mode b) We get a fault and the page has already been received Signed-off-by: Dr. David Alan Gilbert --- migration/postcopy-ram.c | 14 ++++++++++---- migration/trace-events | 1 + 2 files changed, 11 insertions(+), 4 deletions(-) diff --git a/migration/postcopy-ram.c b/migration/postcopy-ram.c index 40b58a7912..7d0786ff04 100644 --- a/migration/postcopy-ram.c +++ b/migration/postcopy-ram.c @@ -490,7 +490,11 @@ int postcopy_request_shared_page(struct PostCopyFD *pc= fd, RAMBlock *rb, =20 trace_postcopy_request_shared_page(pcfd->idstr, qemu_ram_get_idstr(rb), rb_offset); - /* TODO: Check bitmap to see if we already have the page */ + if (ramblock_recv_bitmap_test_byte_offset(rb, aligned_rbo)) { + trace_postcopy_request_shared_page_present(pcfd->idstr, + qemu_ram_get_idstr(rb), rb_offset); + return postcopy_wake_shared(pcfd, client_addr, rb); + } if (rb !=3D mis->last_rb) { mis->last_rb =3D rb; migrate_send_rp_req_pages(mis, qemu_ram_get_idstr(rb), @@ -788,7 +792,8 @@ int postcopy_place_page(MigrationIncomingState *mis, vo= id *host, void *from, } =20 trace_postcopy_place_page(host); - return 0; + return postcopy_notify_shared_wake(rb, + qemu_ram_block_host_offset(rb, host= )); } =20 /* @@ -812,6 +817,9 @@ int postcopy_place_page_zero(MigrationIncomingState *mi= s, void *host, =20 return -e; } + return postcopy_notify_shared_wake(rb, + qemu_ram_block_host_offset(rb, + host= )); } else { /* The kernel can't use UFFDIO_ZEROPAGE for hugepages */ if (!mis->postcopy_tmp_zero_page) { @@ -831,8 +839,6 @@ int postcopy_place_page_zero(MigrationIncomingState *mi= s, void *host, return postcopy_place_page(mis, host, mis->postcopy_tmp_zero_page, rb); } - - return 0; } =20 /* diff --git a/migration/trace-events b/migration/trace-events index 535e7ad84b..10cff5a068 100644 --- a/migration/trace-events +++ b/migration/trace-events @@ -198,6 +198,7 @@ postcopy_ram_incoming_cleanup_entry(void) "" postcopy_ram_incoming_cleanup_exit(void) "" postcopy_ram_incoming_cleanup_join(void) "" postcopy_request_shared_page(const char *sharer, const char *rb, uint64_t = rb_offset) "for %s in %s offset 0x%"PRIx64 +postcopy_request_shared_page_present(const char *sharer, const char *rb, u= int64_t rb_offset) "%s already %s offset 0x%"PRIx64 postcopy_wake_shared(uint64_t client_addr, const char *rb) "at 0x%"PRIx64"= in %s" =20 save_xbzrle_page_skipping(void) "" --=20 2.13.5 From nobody Thu May 2 16:10:00 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 150360412641012.318039850794662; Thu, 24 Aug 2017 12:48:46 -0700 (PDT) Received: from localhost ([::1]:50211 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dky7J-00048L-TO for importer@patchew.org; Thu, 24 Aug 2017 15:48:37 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52819) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dkxou-0003NM-UF for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:29:37 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dkxor-0005TR-Q7 for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:29:36 -0400 Received: from mx1.redhat.com ([209.132.183.28]:45552) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dkxor-0005T4-K0 for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:29:33 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id A4BE437E65; Thu, 24 Aug 2017 19:29:32 +0000 (UTC) Received: from dgilbert-t530.redhat.com (ovpn-117-165.ams2.redhat.com [10.36.117.165]) by smtp.corp.redhat.com (Postfix) with ESMTP id A729817CC4; Thu, 24 Aug 2017 19:29:23 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com A4BE437E65 Authentication-Results: ext-mx05.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx05.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=dgilbert@redhat.com From: "Dr. David Alan Gilbert (git)" To: qemu-devel@nongnu.org, maxime.coquelin@redhat.com, a.perevalov@samsung.com, mst@redhat.com, marcandre.lureau@redhat.com Date: Thu, 24 Aug 2017 20:27:22 +0100 Message-Id: <20170824192730.8440-25-dgilbert@redhat.com> In-Reply-To: <20170824192730.8440-1-dgilbert@redhat.com> References: <20170824192730.8440-1-dgilbert@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.29]); Thu, 24 Aug 2017 19:29:32 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [RFC v2 24/32] vub+postcopy: madvises X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lvivier@redhat.com, aarcange@redhat.com, felipe@nutanix.com, peterx@redhat.com, quintela@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: "Dr. David Alan Gilbert" Clear the area and turn off THP. Signed-off-by: Dr. David Alan Gilbert Reviewed-by: Marc-Andr=C3=A9 Lureau --- contrib/libvhost-user/libvhost-user.c | 32 ++++++++++++++++++++++++++++++-- 1 file changed, 30 insertions(+), 2 deletions(-) diff --git a/contrib/libvhost-user/libvhost-user.c b/contrib/libvhost-user/= libvhost-user.c index 5ec54f7d60..d816851c6d 100644 --- a/contrib/libvhost-user/libvhost-user.c +++ b/contrib/libvhost-user/libvhost-user.c @@ -450,11 +450,39 @@ vu_set_mem_table_exec(VuDev *dev, VhostUserMsg *vmsg) } =20 if (dev->postcopy_listening) { + int ret; /* We should already have an open ufd need to mark each memory * range as ufd. - * Note: Do we need any madvises? Well it's not been accessed - * yet, still probably need no THP to be safe, discard to be s= afe? */ + + /* Discard any mapping we have here; note I can't use MADV_REM= OVE + * or fallocate to make the hole since I don't want to lose + * data that's already arrived in the shared process. + * TODO: How to do hugepage + */ + ret =3D madvise((void *)dev_region->mmap_addr, + dev_region->size + dev_region->mmap_offset, + MADV_DONTNEED); + if (ret) { + fprintf(stderr, + "%s: Failed to madvise(DONTNEED) region %d: %s\n", + __func__, i, strerror(errno)); + } + /* Turn off transparent hugepages so we dont get lose wakeups + * in neighbouring pages. + * TODO: Turn this backon later. + */ + ret =3D madvise((void *)dev_region->mmap_addr, + dev_region->size + dev_region->mmap_offset, + MADV_NOHUGEPAGE); + if (ret) { + /* Note: This can happen legally on kernels that are confi= gured + * without madvise'able hugepages + */ + fprintf(stderr, + "%s: Failed to madvise(NOHUGEPAGE) region %d: %s\n= ", + __func__, i, strerror(errno)); + } struct uffdio_register reg_struct; reg_struct.range.start =3D (uintptr_t)dev_region->mmap_addr; reg_struct.range.len =3D dev_region->size + dev_region->mmap_o= ffset; --=20 2.13.5 From nobody Thu May 2 16:10:00 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1503603682425808.0802233590892; Thu, 24 Aug 2017 12:41:22 -0700 (PDT) Received: from localhost ([::1]:50175 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dky0H-0005Bt-5Q for importer@patchew.org; Thu, 24 Aug 2017 15:41:21 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52838) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dkxox-0003Pv-M5 for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:29:40 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dkxow-0005Vn-EN for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:29:39 -0400 Received: from mx1.redhat.com ([209.132.183.28]:48584) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dkxow-0005VK-4k for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:29:38 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 1EA76C047B61; Thu, 24 Aug 2017 19:29:37 +0000 (UTC) Received: from dgilbert-t530.redhat.com (ovpn-117-165.ams2.redhat.com [10.36.117.165]) by smtp.corp.redhat.com (Postfix) with ESMTP id EEDE07E491; Thu, 24 Aug 2017 19:29:32 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 1EA76C047B61 Authentication-Results: ext-mx07.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx07.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=dgilbert@redhat.com From: "Dr. David Alan Gilbert (git)" To: qemu-devel@nongnu.org, maxime.coquelin@redhat.com, a.perevalov@samsung.com, mst@redhat.com, marcandre.lureau@redhat.com Date: Thu, 24 Aug 2017 20:27:23 +0100 Message-Id: <20170824192730.8440-26-dgilbert@redhat.com> In-Reply-To: <20170824192730.8440-1-dgilbert@redhat.com> References: <20170824192730.8440-1-dgilbert@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.31]); Thu, 24 Aug 2017 19:29:37 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [RFC v2 25/32] vhost+postcopy: Lock around set_mem_table X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lvivier@redhat.com, aarcange@redhat.com, felipe@nutanix.com, peterx@redhat.com, quintela@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: "Dr. David Alan Gilbert" **HACK - better solution needed ** We have the situation where: qemu bridge send set_mem_table map memory a) mark area with UFD send reply with map addresses b) start using c) receive reply As soon as (a) happens qemu might start seeing faults from memory accesses (but doesn't until b); but it can't process those faults until (c) when it's received the mmap addresses. Make the fault handler spin until it gets the reply in (c). At the very least this needs some proper locks, but preferably we need to split the message. Signed-off-by: Dr. David Alan Gilbert --- hw/virtio/trace-events | 1 + hw/virtio/vhost-user.c | 17 ++++++++++++++++- 2 files changed, 17 insertions(+), 1 deletion(-) diff --git a/hw/virtio/trace-events b/hw/virtio/trace-events index adebf6dc6b..065822c70a 100644 --- a/hw/virtio/trace-events +++ b/hw/virtio/trace-events @@ -10,6 +10,7 @@ vhost_user_set_mem_table_withfd(int index, const char *na= me, uint64_t memory_siz vhost_user_postcopy_waker(const char *rb, uint64_t rb_offset) "%s + 0x%"PR= Ix64 vhost_user_postcopy_waker_found(uint64_t client_addr) "0x%"PRIx64 vhost_user_postcopy_waker_nomatch(const char *rb, uint64_t rb_offset) "%s = + 0x%"PRIx64 +vhost_user_postcopy_waker_spin(const char *rb) "%s" =20 # hw/virtio/virtio.c virtqueue_alloc_element(void *elem, size_t sz, unsigned in_num, unsigned o= ut_num) "elem %p size %zd in_num %u out_num %u" diff --git a/hw/virtio/vhost-user.c b/hw/virtio/vhost-user.c index 3bff33a1a6..4d03383a66 100644 --- a/hw/virtio/vhost-user.c +++ b/hw/virtio/vhost-user.c @@ -143,6 +143,7 @@ struct vhost_user { * vhost region. */ ram_addr_t *region_rb_offset; + uint64_t in_set_mem_table; /*Hack! 1 while waiting for set_m= em_table reply */ }; =20 static bool ioeventfd_enabled(void) @@ -338,6 +339,7 @@ static int vhost_user_set_mem_table(struct vhost_dev *d= ev, u->region_rb_len =3D dev->mem->nregions; } =20 + atomic_set(&u->in_set_mem_table, true); for (i =3D 0; i < dev->mem->nregions; ++i) { struct vhost_memory_region *reg =3D dev->mem->regions + i; ram_addr_t offset; @@ -368,14 +370,15 @@ static int vhost_user_set_mem_table(struct vhost_dev = *dev, if (!fd_num) { error_report("Failed initializing vhost-user memory map, " "consider using -object memory-backend-file share=3Do= n"); + atomic_set(&u->in_set_mem_table, false); return -1; } =20 msg.size =3D sizeof(msg.payload.memory.nregions); msg.size +=3D sizeof(msg.payload.memory.padding); msg.size +=3D fd_num * sizeof(VhostUserMemoryRegion); - if (vhost_user_write(dev, &msg, fds, fd_num) < 0) { + atomic_set(&u->in_set_mem_table, false); return -1; } =20 @@ -390,6 +393,7 @@ static int vhost_user_set_mem_table(struct vhost_dev *d= ev, error_report("%s: Received unexpected msg type." "Expected %d received %d", __func__, VHOST_USER_SET_MEM_TABLE, msg_reply.request); + atomic_set(&u->in_set_mem_table, false); return -1; } /* We're using the same structure, just reusing one of the @@ -398,6 +402,7 @@ static int vhost_user_set_mem_table(struct vhost_dev *d= ev, if (msg_reply.size !=3D msg.size) { error_report("%s: Unexpected size for postcopy reply " "%d vs %d", __func__, msg_reply.size, msg.size); + atomic_set(&u->in_set_mem_table, false); return -1; } =20 @@ -427,9 +432,11 @@ static int vhost_user_set_mem_table(struct vhost_dev *= dev, error_report("%s: postcopy reply not fully consumed " "%d vs %zd", __func__, reply_i, fd_num); + atomic_set(&u->in_set_mem_table, false); return -1; } } + atomic_set(&u->in_set_mem_table, false); if (reply_supported) { return process_message_reply(dev, &msg); } @@ -855,6 +862,14 @@ static int vhost_user_postcopy_waker(struct PostCopyFD= *pcfd, RAMBlock *rb, int i; =20 trace_vhost_user_postcopy_waker(qemu_ram_get_idstr(rb), offset); + if (!u) { + return 0; + } + while (atomic_mb_read(&u->in_set_mem_table)) { + trace_vhost_user_postcopy_waker_spin(qemu_ram_get_idstr(rb)); + usleep(1000*100); + } + /* Translate the offset into an address in the clients address space */ for (i =3D 0; i < MIN(dev->mem->nregions, u->region_rb_len); i++) { if (u->region_rb[i] =3D=3D rb && --=20 2.13.5 From nobody Thu May 2 16:10:00 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1503604287909934.052283931536; Thu, 24 Aug 2017 12:51:27 -0700 (PDT) Received: from localhost ([::1]:50226 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dkyA2-0006ek-Nc for importer@patchew.org; Thu, 24 Aug 2017 15:51:26 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52858) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dkxoz-0003Rz-Ry for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:29:45 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dkxoy-0005Wn-PQ for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:29:41 -0400 Received: from mx1.redhat.com ([209.132.183.28]:55132) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dkxoy-0005WQ-GS for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:29:40 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 7DD0BC0587DF; Thu, 24 Aug 2017 19:29:39 +0000 (UTC) Received: from dgilbert-t530.redhat.com (ovpn-117-165.ams2.redhat.com [10.36.117.165]) by smtp.corp.redhat.com (Postfix) with ESMTP id 69F5517CC4; Thu, 24 Aug 2017 19:29:37 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 7DD0BC0587DF Authentication-Results: ext-mx08.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx08.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=dgilbert@redhat.com From: "Dr. David Alan Gilbert (git)" To: qemu-devel@nongnu.org, maxime.coquelin@redhat.com, a.perevalov@samsung.com, mst@redhat.com, marcandre.lureau@redhat.com Date: Thu, 24 Aug 2017 20:27:24 +0100 Message-Id: <20170824192730.8440-27-dgilbert@redhat.com> In-Reply-To: <20170824192730.8440-1-dgilbert@redhat.com> References: <20170824192730.8440-1-dgilbert@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.32]); Thu, 24 Aug 2017 19:29:39 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [RFC v2 26/32] vhost: Add VHOST_USER_POSTCOPY_END message X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lvivier@redhat.com, aarcange@redhat.com, felipe@nutanix.com, peterx@redhat.com, quintela@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: "Dr. David Alan Gilbert" This message is sent just before the end of postcopy to get the client to stop using userfault since we wont respond to any more requests. It should close userfaultfd so that any other pages get mapped to the backing file automatically by the kernel, since at this point we know we've received everything. Signed-off-by: Dr. David Alan Gilbert Reviewed-by: Peter Xu --- contrib/libvhost-user/libvhost-user.c | 23 +++++++++++++++++++++++ contrib/libvhost-user/libvhost-user.h | 1 + docs/interop/vhost-user.txt | 8 ++++++++ hw/virtio/vhost-user.c | 1 + 4 files changed, 33 insertions(+) diff --git a/contrib/libvhost-user/libvhost-user.c b/contrib/libvhost-user/= libvhost-user.c index d816851c6d..23bff47649 100644 --- a/contrib/libvhost-user/libvhost-user.c +++ b/contrib/libvhost-user/libvhost-user.c @@ -67,6 +67,7 @@ vu_request_to_string(int req) REQ(VHOST_USER_SET_VRING_ENDIAN), REQ(VHOST_USER_POSTCOPY_ADVISE), REQ(VHOST_USER_POSTCOPY_LISTEN), + REQ(VHOST_USER_POSTCOPY_END), REQ(VHOST_USER_MAX), }; #undef REQ @@ -893,6 +894,26 @@ vu_set_postcopy_listen(VuDev *dev, VhostUserMsg *vmsg) vmsg->payload.u64 =3D 0; /* Success */ return true; } + +static bool +vu_set_postcopy_end(VuDev *dev, VhostUserMsg *vmsg) +{ + DPRINT("%s: Entry\n", __func__); + dev->postcopy_listening =3D false; + if (dev->postcopy_ufd > 0) { + close(dev->postcopy_ufd); + dev->postcopy_ufd =3D -1; + DPRINT("%s: Done close\n", __func__); + } + + vmsg->fd_num =3D 0; + vmsg->payload.u64 =3D 0; + vmsg->size =3D sizeof(vmsg->payload.u64); + vmsg->flags =3D VHOST_USER_VERSION | VHOST_USER_REPLY_MASK; + DPRINT("%s: exit\n", __func__); + return true; +} + static bool vu_process_message(VuDev *dev, VhostUserMsg *vmsg) { @@ -962,6 +983,8 @@ vu_process_message(VuDev *dev, VhostUserMsg *vmsg) return vu_set_postcopy_advise(dev, vmsg); case VHOST_USER_POSTCOPY_LISTEN: return vu_set_postcopy_listen(dev, vmsg); + case VHOST_USER_POSTCOPY_END: + return vu_set_postcopy_end(dev, vmsg); default: vmsg_close_fds(vmsg); vu_panic(dev, "Unhandled request: %d", vmsg->request); diff --git a/contrib/libvhost-user/libvhost-user.h b/contrib/libvhost-user/= libvhost-user.h index 29c11ba56c..a78596e6fd 100644 --- a/contrib/libvhost-user/libvhost-user.h +++ b/contrib/libvhost-user/libvhost-user.h @@ -68,6 +68,7 @@ typedef enum VhostUserRequest { VHOST_USER_SET_VRING_ENDIAN =3D 23, VHOST_USER_POSTCOPY_ADVISE =3D 24, VHOST_USER_POSTCOPY_LISTEN =3D 25, + VHOST_USER_POSTCOPY_END =3D 26, VHOST_USER_MAX } VhostUserRequest; =20 diff --git a/docs/interop/vhost-user.txt b/docs/interop/vhost-user.txt index b2a548c94d..d6586e0b43 100644 --- a/docs/interop/vhost-user.txt +++ b/docs/interop/vhost-user.txt @@ -627,6 +627,14 @@ Master message types =20 Master advises slave that a transition to postcopy mode has happened. =20 + * VHOST_USER_POSTCOPY_END + Id: 26 + Slave payload: u64 + + Master advises that postcopy migration has now completed. The + slave must disable the userfaultfd. The response is an acknowledgeme= nt + only. + Slave message types ------------------- =20 diff --git a/hw/virtio/vhost-user.c b/hw/virtio/vhost-user.c index 4d03383a66..c2e55be0fd 100644 --- a/hw/virtio/vhost-user.c +++ b/hw/virtio/vhost-user.c @@ -71,6 +71,7 @@ typedef enum VhostUserRequest { VHOST_USER_SET_VRING_ENDIAN =3D 23, VHOST_USER_POSTCOPY_ADVISE =3D 24, VHOST_USER_POSTCOPY_LISTEN =3D 25, + VHOST_USER_POSTCOPY_END =3D 26, VHOST_USER_MAX } VhostUserRequest; =20 --=20 2.13.5 From nobody Thu May 2 16:10:00 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1503604507387388.3766439514709; Thu, 24 Aug 2017 12:55:07 -0700 (PDT) Received: from localhost ([::1]:50245 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dkyDa-0001VM-2K for importer@patchew.org; Thu, 24 Aug 2017 15:55:06 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52885) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dkxp3-0003Uh-4Z for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:29:46 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dkxp1-0005Xu-4o for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:29:45 -0400 Received: from mx1.redhat.com ([209.132.183.28]:45848) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dkxp0-0005XM-SX for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:29:43 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id DB7886DB; Thu, 24 Aug 2017 19:29:41 +0000 (UTC) Received: from dgilbert-t530.redhat.com (ovpn-117-165.ams2.redhat.com [10.36.117.165]) by smtp.corp.redhat.com (Postfix) with ESMTP id C8D0717CC4; Thu, 24 Aug 2017 19:29:39 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com DB7886DB Authentication-Results: ext-mx05.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx05.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=dgilbert@redhat.com From: "Dr. David Alan Gilbert (git)" To: qemu-devel@nongnu.org, maxime.coquelin@redhat.com, a.perevalov@samsung.com, mst@redhat.com, marcandre.lureau@redhat.com Date: Thu, 24 Aug 2017 20:27:25 +0100 Message-Id: <20170824192730.8440-28-dgilbert@redhat.com> In-Reply-To: <20170824192730.8440-1-dgilbert@redhat.com> References: <20170824192730.8440-1-dgilbert@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.29]); Thu, 24 Aug 2017 19:29:42 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [RFC v2 27/32] vhost+postcopy: Wire up POSTCOPY_END notify X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lvivier@redhat.com, aarcange@redhat.com, felipe@nutanix.com, peterx@redhat.com, quintela@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: "Dr. David Alan Gilbert" Wire up a call to VHOST_USER_POSTCOPY_END message to the vhost clients right before we ask the listener thread to shutdown. Signed-off-by: Dr. David Alan Gilbert Reviewed-by: Peter Xu --- hw/virtio/trace-events | 2 ++ hw/virtio/vhost-user.c | 30 ++++++++++++++++++++++++++++++ migration/postcopy-ram.c | 5 +++++ migration/postcopy-ram.h | 1 + 4 files changed, 38 insertions(+) diff --git a/hw/virtio/trace-events b/hw/virtio/trace-events index 065822c70a..5b599617a1 100644 --- a/hw/virtio/trace-events +++ b/hw/virtio/trace-events @@ -1,6 +1,8 @@ # See docs/devel/tracing.txt for syntax documentation. =20 # hw/virtio/vhost-user.c +vhost_user_postcopy_end_entry(void) "" +vhost_user_postcopy_end_exit(void) "" vhost_user_postcopy_fault_handler(const char *name, uint64_t fault_address= , int nregions) "%s: @0x%"PRIx64" nregions:%d" vhost_user_postcopy_fault_handler_loop(int i, uint64_t client_base, uint64= _t size) "%d: client 0x%"PRIx64" +0x%"PRIx64 vhost_user_postcopy_fault_handler_found(int i, uint64_t region_offset, uin= t64_t rb_offset) "%d: region_offset: 0x%"PRIx64" rb_offset:0x%"PRIx64 diff --git a/hw/virtio/vhost-user.c b/hw/virtio/vhost-user.c index c2e55be0fd..d4461459fe 100644 --- a/hw/virtio/vhost-user.c +++ b/hw/virtio/vhost-user.c @@ -965,6 +965,33 @@ static int vhost_user_postcopy_listen(struct vhost_dev= *dev, Error **errp) return 0; } =20 +/* + * Called at the end of postcopy + */ +static int vhost_user_postcopy_end(struct vhost_dev *dev, Error **errp) +{ + VhostUserMsg msg =3D { + .request =3D VHOST_USER_POSTCOPY_END, + .flags =3D VHOST_USER_VERSION | VHOST_USER_NEED_REPLY_MASK, + }; + int ret; + + trace_vhost_user_postcopy_end_entry(); + if (vhost_user_write(dev, &msg, NULL, 0) < 0) { + error_setg(errp, "Failed to send postcopy_end to vhost"); + return -1; + } + + ret =3D process_message_reply(dev, &msg); + if (ret) { + error_setg(errp, "Failed to receive reply to postcopy_end"); + return ret; + } + trace_vhost_user_postcopy_end_exit(); + + return 0; +} + static int vhost_user_postcopy_notifier(NotifierWithReturn *notifier, void *opaque) { @@ -990,6 +1017,9 @@ static int vhost_user_postcopy_notifier(NotifierWithRe= turn *notifier, case POSTCOPY_NOTIFY_INBOUND_LISTEN: return vhost_user_postcopy_listen(dev, pnd->errp); =20 + case POSTCOPY_NOTIFY_INBOUND_END: + return vhost_user_postcopy_end(dev, pnd->errp); + default: /* We ignore notifications we don't know */ break; diff --git a/migration/postcopy-ram.c b/migration/postcopy-ram.c index 7d0786ff04..28791cf1f1 100644 --- a/migration/postcopy-ram.c +++ b/migration/postcopy-ram.c @@ -337,7 +337,12 @@ int postcopy_ram_incoming_cleanup(MigrationIncomingSta= te *mis) =20 if (mis->have_fault_thread) { uint64_t tmp64; + Error *local_err =3D NULL; =20 + if (postcopy_notify(POSTCOPY_NOTIFY_INBOUND_END, &local_err)) { + error_report_err(local_err); + return -1; + } if (qemu_ram_foreach_block(cleanup_range, mis)) { return -1; } diff --git a/migration/postcopy-ram.h b/migration/postcopy-ram.h index ecf731c689..d0dc838001 100644 --- a/migration/postcopy-ram.h +++ b/migration/postcopy-ram.h @@ -130,6 +130,7 @@ enum PostcopyNotifyReason { POSTCOPY_NOTIFY_PROBE =3D 0, POSTCOPY_NOTIFY_INBOUND_ADVISE, POSTCOPY_NOTIFY_INBOUND_LISTEN, + POSTCOPY_NOTIFY_INBOUND_END, }; =20 struct PostcopyNotifyData { --=20 2.13.5 From nobody Thu May 2 16:10:00 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1503603836948278.92861701164827; Thu, 24 Aug 2017 12:43:56 -0700 (PDT) Received: from localhost ([::1]:50185 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dky2l-00080T-Pr for importer@patchew.org; Thu, 24 Aug 2017 15:43:55 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52950) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dkxpC-0003es-Rs for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:29:55 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dkxp9-0005cW-OK for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:29:54 -0400 Received: from mx1.redhat.com ([209.132.183.28]:52142) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dkxp9-0005c8-Ih for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:29:51 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 8F17A5F7B4; Thu, 24 Aug 2017 19:29:50 +0000 (UTC) Received: from dgilbert-t530.redhat.com (ovpn-117-165.ams2.redhat.com [10.36.117.165]) by smtp.corp.redhat.com (Postfix) with ESMTP id 3164C17CC4; Thu, 24 Aug 2017 19:29:42 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 8F17A5F7B4 Authentication-Results: ext-mx10.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx10.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=dgilbert@redhat.com From: "Dr. David Alan Gilbert (git)" To: qemu-devel@nongnu.org, maxime.coquelin@redhat.com, a.perevalov@samsung.com, mst@redhat.com, marcandre.lureau@redhat.com Date: Thu, 24 Aug 2017 20:27:26 +0100 Message-Id: <20170824192730.8440-29-dgilbert@redhat.com> In-Reply-To: <20170824192730.8440-1-dgilbert@redhat.com> References: <20170824192730.8440-1-dgilbert@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.39]); Thu, 24 Aug 2017 19:29:50 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [RFC v2 28/32] postcopy: Allow shared memory X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lvivier@redhat.com, aarcange@redhat.com, felipe@nutanix.com, peterx@redhat.com, quintela@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: "Dr. David Alan Gilbert" Now that we have the mechanisms in here, allow shared memory in a postcopy. Note that QEMU can't tell who all the users of shared regions are and thus can't tell whether all the users of the shared regions have appropriate support for postcopy. Those devices that explicitly support shared memory (e.g. vhost-user) must check, but it doesn't stop weirder configurations causing problems. Signed-off-by: Dr. David Alan Gilbert Reviewed-by: Marc-Andr=C3=A9 Lureau --- migration/postcopy-ram.c | 6 ------ 1 file changed, 6 deletions(-) diff --git a/migration/postcopy-ram.c b/migration/postcopy-ram.c index 28791cf1f1..89c3aadda1 100644 --- a/migration/postcopy-ram.c +++ b/migration/postcopy-ram.c @@ -138,12 +138,6 @@ static int test_ramblock_postcopiable(const char *bloc= k_name, void *host_addr, RAMBlock *rb =3D qemu_ram_block_by_name(block_name); size_t pagesize =3D qemu_ram_pagesize(rb); =20 - if (qemu_ram_is_shared(rb)) { - error_report("Postcopy on shared RAM (%s) is not yet supported", - block_name); - return 1; - } - if (length % pagesize) { error_report("Postcopy requires RAM blocks to be a page size multi= ple," " block %s is 0x" RAM_ADDR_FMT " bytes with a " --=20 2.13.5 From nobody Thu May 2 16:10:00 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1503603988181773.0908711645687; Thu, 24 Aug 2017 12:46:28 -0700 (PDT) Received: from localhost ([::1]:50202 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dky5C-0002Qz-Sw for importer@patchew.org; Thu, 24 Aug 2017 15:46:26 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52965) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dkxpG-0003ix-Od for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:29:59 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dkxpF-0005f5-QE for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:29:58 -0400 Received: from mx1.redhat.com ([209.132.183.28]:35702) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dkxpF-0005es-Ju for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:29:57 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 98986267F8; Thu, 24 Aug 2017 19:29:56 +0000 (UTC) Received: from dgilbert-t530.redhat.com (ovpn-117-165.ams2.redhat.com [10.36.117.165]) by smtp.corp.redhat.com (Postfix) with ESMTP id DB41A7F38A; Thu, 24 Aug 2017 19:29:50 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 98986267F8 Authentication-Results: ext-mx06.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx06.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=dgilbert@redhat.com From: "Dr. David Alan Gilbert (git)" To: qemu-devel@nongnu.org, maxime.coquelin@redhat.com, a.perevalov@samsung.com, mst@redhat.com, marcandre.lureau@redhat.com Date: Thu, 24 Aug 2017 20:27:27 +0100 Message-Id: <20170824192730.8440-30-dgilbert@redhat.com> In-Reply-To: <20170824192730.8440-1-dgilbert@redhat.com> References: <20170824192730.8440-1-dgilbert@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.30]); Thu, 24 Aug 2017 19:29:56 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [RFC v2 29/32] vhost-user: Claim support for postcopy X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lvivier@redhat.com, aarcange@redhat.com, felipe@nutanix.com, peterx@redhat.com, quintela@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: "Dr. David Alan Gilbert" Tell QEMU we understand the protocol features needed for postcopy. Signed-off-by: Dr. David Alan Gilbert Reviewed-by: Marc-Andr=C3=A9 Lureau --- contrib/libvhost-user/libvhost-user.c | 33 +++++++++++++++++++++++++++++++= ++ 1 file changed, 33 insertions(+) diff --git a/contrib/libvhost-user/libvhost-user.c b/contrib/libvhost-user/= libvhost-user.c index 23bff47649..290748733b 100644 --- a/contrib/libvhost-user/libvhost-user.c +++ b/contrib/libvhost-user/libvhost-user.c @@ -144,6 +144,35 @@ vmsg_close_fds(VhostUserMsg *vmsg) } } =20 +/* A test to see if we have userfault available */ +static bool +have_userfault(void) +{ +#if defined(__linux__) && defined(__NR_userfaultfd) &&\ + defined(UFFD_FEATURE_MISSING_SHMEM) &&\ + defined(UFFD_FEATURE_MISSING_HUGETLBFS) + /* Now test the kernel we're running on really has the features */ + int ufd =3D syscall(__NR_userfaultfd, O_CLOEXEC | O_NONBLOCK); + struct uffdio_api api_struct; + if (ufd < 0) { + return false; + } + + api_struct.api =3D UFFD_API; + api_struct.features =3D UFFD_FEATURE_MISSING_SHMEM | + UFFD_FEATURE_MISSING_HUGETLBFS; + if (ioctl(ufd, UFFDIO_API, &api_struct)) { + close(ufd); + return false; + } + close(ufd); + return true; + +#else + return false; +#endif +} + static bool vu_message_read(VuDev *dev, int conn_fd, VhostUserMsg *vmsg) { @@ -796,6 +825,10 @@ vu_get_protocol_features_exec(VuDev *dev, VhostUserMsg= *vmsg) { uint64_t features =3D 1ULL << VHOST_USER_PROTOCOL_F_LOG_SHMFD; =20 + if (have_userfault()) { + features |=3D 1ULL << VHOST_USER_PROTOCOL_F_PAGEFAULT; + } + if (dev->iface->get_protocol_features) { features |=3D dev->iface->get_protocol_features(dev); } --=20 2.13.5 From nobody Thu May 2 16:10:00 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1503604157875793.3771997135096; Thu, 24 Aug 2017 12:49:17 -0700 (PDT) Received: from localhost ([::1]:50214 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dky7w-0004bb-Fl for importer@patchew.org; Thu, 24 Aug 2017 15:49:16 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52985) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dkxpL-0003nO-L2 for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:30:06 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dkxpI-0005gM-Dt for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:30:03 -0400 Received: from mx1.redhat.com ([209.132.183.28]:53380) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dkxpI-0005ft-4m for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:30:00 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 29BA1883C3; Thu, 24 Aug 2017 19:29:59 +0000 (UTC) Received: from dgilbert-t530.redhat.com (ovpn-117-165.ams2.redhat.com [10.36.117.165]) by smtp.corp.redhat.com (Postfix) with ESMTP id E52907F383; Thu, 24 Aug 2017 19:29:56 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 29BA1883C3 Authentication-Results: ext-mx02.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx02.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=dgilbert@redhat.com From: "Dr. David Alan Gilbert (git)" To: qemu-devel@nongnu.org, maxime.coquelin@redhat.com, a.perevalov@samsung.com, mst@redhat.com, marcandre.lureau@redhat.com Date: Thu, 24 Aug 2017 20:27:28 +0100 Message-Id: <20170824192730.8440-31-dgilbert@redhat.com> In-Reply-To: <20170824192730.8440-1-dgilbert@redhat.com> References: <20170824192730.8440-1-dgilbert@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.26]); Thu, 24 Aug 2017 19:29:59 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [RFC v2 30/32] vhost: Merge neighbouring hugepage regions where appropriate X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lvivier@redhat.com, aarcange@redhat.com, felipe@nutanix.com, peterx@redhat.com, quintela@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: "Dr. David Alan Gilbert" Where two regions are created with a gap such that when aligned to hugepage boundaries, the two regions overlap, merge them. I also add quite a few trace events to see what's going on. Note: This doesn't handle all the cases, but does handle the common case on a PC due to the 640k hole. Signed-off-by: Dr. David Alan Gilbert Signed-off-by: Maxime Coquelin --- hw/virtio/trace-events | 11 +++++++ hw/virtio/vhost.c | 79 ++++++++++++++++++++++++++++++++++++++++++++++= +++- 2 files changed, 89 insertions(+), 1 deletion(-) diff --git a/hw/virtio/trace-events b/hw/virtio/trace-events index 5b599617a1..f98efb39fd 100644 --- a/hw/virtio/trace-events +++ b/hw/virtio/trace-events @@ -1,5 +1,16 @@ # See docs/devel/tracing.txt for syntax documentation. =20 +# hw/virtio/vhost.c +vhost_dev_assign_memory_merged(int from, int to, uint64_t size, uint64_t s= tart_addr, uint64_t uaddr) "f/t=3D%d/%d 0x%"PRIx64" @ P: 0x%"PRIx64" U: 0x%= "PRIx64 +vhost_dev_assign_memory_not_merged(uint64_t size, uint64_t start_addr, uin= t64_t uaddr) "0x%"PRIx64" @ P: 0x%"PRIx64" U: 0x%"PRIx64 +vhost_dev_assign_memory_entry(uint64_t size, uint64_t start_addr, uint64_t= uaddr) "0x%"PRIx64" @ P: 0x%"PRIx64" U: 0x%"PRIx64 +vhost_dev_assign_memory_exit(uint32_t nregions) "%"PRId32 +vhost_huge_page_stretch_and_merge_entry(uint32_t nregions) "%"PRId32 +vhost_huge_page_stretch_and_merge_can(void) "" +vhost_huge_page_stretch_and_merge_size_align(int d, uint64_t gpa, uint64_t= align) "%d: gpa: 0x%"PRIx64" align: 0x%"PRIx64 +vhost_huge_page_stretch_and_merge_start_align(int d, uint64_t gpa, uint64_= t align) "%d: gpa: 0x%"PRIx64" align: 0x%"PRIx64 +vhost_section(const char *name, int r) "%s:%d" + # hw/virtio/vhost-user.c vhost_user_postcopy_end_entry(void) "" vhost_user_postcopy_end_exit(void) "" diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c index 6eddb099b0..fb506e747f 100644 --- a/hw/virtio/vhost.c +++ b/hw/virtio/vhost.c @@ -27,6 +27,7 @@ #include "hw/virtio/virtio-access.h" #include "migration/blocker.h" #include "sysemu/dma.h" +#include "trace.h" =20 /* enabled until disconnected backend stabilizes */ #define _VHOST_DEBUG 1 @@ -250,6 +251,8 @@ static void vhost_dev_assign_memory(struct vhost_dev *d= ev, { int from, to; struct vhost_memory_region *merged =3D NULL; + trace_vhost_dev_assign_memory_entry(size, start_addr, uaddr); + for (from =3D 0, to =3D 0; from < dev->mem->nregions; ++from, ++to) { struct vhost_memory_region *reg =3D dev->mem->regions + to; uint64_t prlast, urlast; @@ -293,11 +296,13 @@ static void vhost_dev_assign_memory(struct vhost_dev = *dev, uaddr =3D merged->userspace_addr =3D u; start_addr =3D merged->guest_phys_addr =3D s; size =3D merged->memory_size =3D e - s + 1; + trace_vhost_dev_assign_memory_merged(from, to, size, start_addr, u= addr); assert(merged->memory_size); } =20 if (!merged) { struct vhost_memory_region *reg =3D dev->mem->regions + to; + trace_vhost_dev_assign_memory_not_merged(size, start_addr, uaddr); memset(reg, 0, sizeof *reg); reg->memory_size =3D size; assert(reg->memory_size); @@ -307,6 +312,7 @@ static void vhost_dev_assign_memory(struct vhost_dev *d= ev, } assert(to <=3D dev->mem->nregions + 1); dev->mem->nregions =3D to; + trace_vhost_dev_assign_memory_exit(to); } =20 static uint64_t vhost_get_log_size(struct vhost_dev *dev) @@ -610,8 +616,12 @@ static void vhost_set_memory(MemoryListener *listener, =20 static bool vhost_section(MemoryRegionSection *section) { - return memory_region_is_ram(section->mr) && + bool result; + result =3D memory_region_is_ram(section->mr) && !memory_region_is_rom(section->mr); + + trace_vhost_section(section->mr->name, result); + return result; } =20 static void vhost_begin(MemoryListener *listener) @@ -622,6 +632,68 @@ static void vhost_begin(MemoryListener *listener) dev->mem_changed_start_addr =3D -1; } =20 +/* Look for regions that are hugepage backed but not aligned + * and fix them up to be aligned. + * TODO: For now this is just enough to deal with the 640k hole + */ +static bool vhost_huge_page_stretch_and_merge(struct vhost_dev *dev) +{ + int i, j; + bool result =3D true; + trace_vhost_huge_page_stretch_and_merge_entry(dev->mem->nregions); + + for (i =3D 0; i < dev->mem->nregions; i++) { + struct vhost_memory_region *reg =3D dev->mem->regions + i; + ram_addr_t offset; + RAMBlock *rb =3D qemu_ram_block_from_host((void *)reg->userspace_a= ddr, + false, &offset); + size_t pagesize =3D qemu_ram_pagesize(rb); + uint64_t alignage; + alignage =3D reg->guest_phys_addr & (pagesize - 1); + if (alignage) { + + trace_vhost_huge_page_stretch_and_merge_start_align(i, + (uint64_t)reg->guest_phys_= addr, + alignage); + for (j =3D 0; j < dev->mem->nregions; j++) { + struct vhost_memory_region *oreg =3D dev->mem->regions + j; + if (j =3D=3D i) { + continue; + } + + if (oreg->guest_phys_addr =3D=3D + (reg->guest_phys_addr - alignage) && + oreg->userspace_addr =3D=3D + (reg->userspace_addr - alignage)) { + struct vhost_memory_region treg =3D *reg; + trace_vhost_huge_page_stretch_and_merge_can(); + vhost_dev_unassign_memory(dev, oreg->guest_phys_addr, + oreg->memory_size); + vhost_dev_unassign_memory(dev, treg.guest_phys_addr, + treg.memory_size); + vhost_dev_assign_memory(dev, + treg.guest_phys_addr - alignag= e, + treg.memory_size + alignage, + treg.userspace_addr - alignage= ); + return vhost_huge_page_stretch_and_merge(dev); + } + } + } + alignage =3D reg->memory_size & (pagesize - 1); + if (alignage) { + trace_vhost_huge_page_stretch_and_merge_size_align(i, + (uint64_t)reg->guest_phys_a= ddr, + alignage); + /* We ignore this if we find something else to merge, + * so we only return false if we're left with this + */ + result =3D false; + } + } + + return result; +} + static void vhost_commit(MemoryListener *listener) { struct vhost_dev *dev =3D container_of(listener, struct vhost_dev, @@ -641,6 +713,7 @@ static void vhost_commit(MemoryListener *listener) return; } =20 + vhost_huge_page_stretch_and_merge(dev); if (dev->started) { start_addr =3D dev->mem_changed_start_addr; size =3D dev->mem_changed_end_addr - dev->mem_changed_start_addr += 1; @@ -1512,6 +1585,10 @@ int vhost_dev_start(struct vhost_dev *hdev, VirtIODe= vice *vdev) goto fail_features; } =20 + if (!vhost_huge_page_stretch_and_merge(hdev)) { + VHOST_OPS_DEBUG("vhost_huge_page_stretch_and_merge failed"); + goto fail_mem; + } if (vhost_dev_has_iommu(hdev)) { memory_listener_register(&hdev->iommu_listener, vdev->dma_as); } --=20 2.13.5 From nobody Thu May 2 16:10:00 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1503604335014986.9500175458116; Thu, 24 Aug 2017 12:52:15 -0700 (PDT) Received: from localhost ([::1]:50229 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dkyAn-0007cl-Us for importer@patchew.org; Thu, 24 Aug 2017 15:52:13 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:53000) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dkxpM-0003nw-9v for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:30:10 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dkxpL-0005iU-4H for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:30:04 -0400 Received: from mx1.redhat.com ([209.132.183.28]:52558) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dkxpK-0005h4-RU for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:30:03 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id C339E5F742; Thu, 24 Aug 2017 19:30:01 +0000 (UTC) Received: from dgilbert-t530.redhat.com (ovpn-117-165.ams2.redhat.com [10.36.117.165]) by smtp.corp.redhat.com (Postfix) with ESMTP id 7449A7F383; Thu, 24 Aug 2017 19:29:59 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com C339E5F742 Authentication-Results: ext-mx10.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx10.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=dgilbert@redhat.com From: "Dr. David Alan Gilbert (git)" To: qemu-devel@nongnu.org, maxime.coquelin@redhat.com, a.perevalov@samsung.com, mst@redhat.com, marcandre.lureau@redhat.com Date: Thu, 24 Aug 2017 20:27:29 +0100 Message-Id: <20170824192730.8440-32-dgilbert@redhat.com> In-Reply-To: <20170824192730.8440-1-dgilbert@redhat.com> References: <20170824192730.8440-1-dgilbert@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.39]); Thu, 24 Aug 2017 19:30:02 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [RFC v2 31/32] vhost: Don't break merged regions on small remove/non-adds X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lvivier@redhat.com, aarcange@redhat.com, felipe@nutanix.com, peterx@redhat.com, quintela@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: "Dr. David Alan Gilbert" The previous patch merges hugepage regions with small gaps back into one; this patch stops some of the cases that split them up again. vhost_set_memory avoids adding small regions that are being dirty-monitored (typically VGA regions), but vhost_set_memory does remove these regions from any region they overlap. Avoid doing that removal if we'll just merge them again. The typical case is where the VGA regions dynamically change at run time after we've done the merge; although the merge works, the result is that the memory is marked as having changed and a set_mem_table message is transmitted. By avoiding the split we avoid the join and we avoid marking the regions as having changed, and thus avoid sending the set_mem_table. Signed-off-by: Dr. David Alan Gilbert --- hw/virtio/trace-events | 2 ++ hw/virtio/vhost.c | 42 +++++++++++++++++++++++++++++++++++++++++- 2 files changed, 43 insertions(+), 1 deletion(-) diff --git a/hw/virtio/trace-events b/hw/virtio/trace-events index f98efb39fd..e22d2055b0 100644 --- a/hw/virtio/trace-events +++ b/hw/virtio/trace-events @@ -10,6 +10,8 @@ vhost_huge_page_stretch_and_merge_can(void) "" vhost_huge_page_stretch_and_merge_size_align(int d, uint64_t gpa, uint64_t= align) "%d: gpa: 0x%"PRIx64" align: 0x%"PRIx64 vhost_huge_page_stretch_and_merge_start_align(int d, uint64_t gpa, uint64_= t align) "%d: gpa: 0x%"PRIx64" align: 0x%"PRIx64 vhost_section(const char *name, int r) "%s:%d" +vhost_dev_would_remerge_no(uint64_t start_addr, uint64_t size) "0x%"PRIx64= " + 0x%"PRIx64 +vhost_dev_would_remerge_yes(uint64_t start_addr, uint64_t size) "0x%"PRIx6= 4" + 0x%"PRIx64 =20 # hw/virtio/vhost-user.c vhost_user_postcopy_end_entry(void) "" diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c index fb506e747f..18714f6d03 100644 --- a/hw/virtio/vhost.c +++ b/hw/virtio/vhost.c @@ -564,6 +564,43 @@ static bool vhost_dev_cmp_memory(struct vhost_dev *dev, return uaddr !=3D reg->userspace_addr + start_addr - reg->guest_phys_a= ddr; } =20 +/* Called by vhost_set_memory on removal where start_addr/size have + * been found to correspond to an existing region (reg). + * Returns True iff removing the section would be undone later + * by merging (for hugepage) and thus there's no point in removing it. + */ +static bool vhost_dev_would_remerge(struct vhost_dev *dev, + struct vhost_memory_region *reg, + hwaddr start_addr, ram_addr_t size) +{ + uint64_t reglast =3D range_get_last(reg->guest_phys_addr, reg->memory_= size); + uint64_t memlast =3D range_get_last(start_addr, size); + ram_addr_t offset; + RAMBlock *rb =3D qemu_ram_block_from_host((void *)reg->userspace_addr, + false, &offset); + size_t reg_pagesize =3D qemu_ram_pagesize(rb); + + /* If the region being deleted hangs over the end of the existing + * region, it's not a case we're interested in. + * If it's just a normal sized page then there won't normally be any + * alignment merging going on. + * and if the hole being cut is larger than the pagesize of the + * region then assume it won't be remerged. + */ + if (memlast > reglast || start_addr < reg->guest_phys_addr || + reg_pagesize =3D=3D getpagesize() || + size >=3D reg_pagesize) { + trace_vhost_dev_would_remerge_no(start_addr, size); + return false; + } + + /* This is a small chunk overlapping a big hugepage region, + * deleting it will get remerged + */ + trace_vhost_dev_would_remerge_yes(start_addr, size); + return true; +} + static void vhost_set_memory(MemoryListener *listener, MemoryRegionSection *section, bool add) @@ -594,7 +631,10 @@ static void vhost_set_memory(MemoryListener *listener, return; } } else { - if (!vhost_dev_find_reg(dev, start_addr, size)) { + struct vhost_memory_region *reg =3D vhost_dev_find_reg(dev, + start_addr, + size); + if (!reg || vhost_dev_would_remerge(dev, reg, start_addr, size)) { /* Removing region that we don't access. Nothing to do. */ return; } --=20 2.13.5 From nobody Thu May 2 16:10:00 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 150360445474915.530610940632982; Thu, 24 Aug 2017 12:54:14 -0700 (PDT) Received: from localhost ([::1]:50238 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dkyCj-0000kW-FS for importer@patchew.org; Thu, 24 Aug 2017 15:54:13 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:53106) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dkxpd-000491-MV for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:30:22 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dkxpa-0005uY-JB for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:30:21 -0400 Received: from mx1.redhat.com ([209.132.183.28]:54124) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dkxpa-0005ty-AR for qemu-devel@nongnu.org; Thu, 24 Aug 2017 15:30:18 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 3CF8C883D6; Thu, 24 Aug 2017 19:30:17 +0000 (UTC) Received: from dgilbert-t530.redhat.com (ovpn-117-165.ams2.redhat.com [10.36.117.165]) by smtp.corp.redhat.com (Postfix) with ESMTP id 10BB87F383; Thu, 24 Aug 2017 19:30:01 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 3CF8C883D6 Authentication-Results: ext-mx02.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx02.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=dgilbert@redhat.com From: "Dr. David Alan Gilbert (git)" To: qemu-devel@nongnu.org, maxime.coquelin@redhat.com, a.perevalov@samsung.com, mst@redhat.com, marcandre.lureau@redhat.com Date: Thu, 24 Aug 2017 20:27:30 +0100 Message-Id: <20170824192730.8440-33-dgilbert@redhat.com> In-Reply-To: <20170824192730.8440-1-dgilbert@redhat.com> References: <20170824192730.8440-1-dgilbert@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.26]); Thu, 24 Aug 2017 19:30:17 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [RFC v2 32/32] postcopy shared docs X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lvivier@redhat.com, aarcange@redhat.com, felipe@nutanix.com, peterx@redhat.com, quintela@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: "Dr. David Alan Gilbert" Add some notes to the migration documentation for shared memory postcopy. Signed-off-by: Dr. David Alan Gilbert --- docs/devel/migration.txt | 39 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+) diff --git a/docs/devel/migration.txt b/docs/devel/migration.txt index 1b940a829b..d4c344c671 100644 --- a/docs/devel/migration.txt +++ b/docs/devel/migration.txt @@ -553,3 +553,42 @@ Postcopy now works with hugetlbfs backed memory: hugepages works well, however 1GB hugepages are likely to be problema= tic since it takes ~1 second to transfer a 1GB hugepage across a 10Gbps l= ink, and until the full page is transferred the destination thread is bloc= ked. + +=3D=3D=3D Postcopy with shared memory =3D=3D=3D + +Postcopy migration with shared memory needs explicit support from the other +processes that share memory and from QEMU. There are restrictions on the t= ype of +memory that userfault can support shared. + +The Linux kernel userfault support works on /dev/shm memory and on hugetlb= fs +(although the kernel doesn't provide an equivalent to madvise(MADV_DONTNEE= D) +for hugetlbfs which may be a problem in some configurations). + +The vhost-user code in QEMU supports clients that have Postcopy support, +and the vhost-user-bridge (in tests/) and the DPDK package have changes +to support postcopy. + +The client needs to open a userfaultfd and register the areas +of memory that it maps with userfault. The client must then pass the +userfaultfd back to QEMU together with a mapping table that allows +fault addresses in the clients address space to be converted back to +RAMBlock/offsets. The client's userfaultfd is added to the postcopy +fault-thread and page requests are made on behalf of the client by QEMU. +QEMU performs 'wake' operations on the client's userfaultfd to allow it +to continue after a page has arrived. + + Note: There are two future improvements that would be nice: + a) Some way to make QEMU ignorant of the addresses in the clients + address space + b) Avoiding the need for QEMU to perform ufd-wake calls after the + pages have arrived + +Retro-fitting postcopy to existing clients is possible: + a) A mechanism is needed for the registration with userfault as above, + and the registration needs to be coordinated with the phases of + postcopy. In vhost-user extra messages are added to the existing + control channel. + b) Any thread that can block due to guest memory accesses must be + identified and the implication understood; for example if the + guest memory access is made while holding a lock then all other + threads waiting for that lock will also be blocked. --=20 2.13.5