From nobody Sun May 19 06:50:49 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1670314793; cv=none; d=zohomail.com; s=zohoarc; b=lpTMQj+LshRIa4TYQ2J/9fESBTumGhlQX7Cyz7CEhOfdIxy8Y4tgeXv4NJhdo76ELyhGvOLSHYcJlwXc/lvFJq9AYNlBUmN3DFCYwxVwwejoZ1AwK+cfKZuiEG/o0oZtdlkUGaWbzfuMfiUEDl1WOmK/zjaqiZXGv7uCdd4X0Dc= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1670314793; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:Reply-To:References:Sender:Subject:To; bh=7Lu0tBvVBDbZItuSCpyeW0EnPLGxEpOADudMTAfvpaA=; b=BZUhEFMplQyp1q0mP9XQ6HOIG1U0u4IgpPRZpP6rvXRJ4JZ1/8AncQAFa3loCgSAYz0BzP+6aJ3p0IEu61PtcK8ARBrZzN/VSFLbnuGzMPMsjhQDqbpoXWSKcQ5D9y5kyNb/mCrZLKlQ9Vve9KarOzXjeBh+6vO0/yt59kwlTqc= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1670314793257527.1438006537513; Tue, 6 Dec 2022 00:19:53 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1p2TAs-0007Ta-5p; Tue, 06 Dec 2022 03:19:34 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1p2TAk-0007Ob-Jw for qemu-devel@nongnu.org; Tue, 06 Dec 2022 03:19:27 -0500 Received: from szxga01-in.huawei.com ([45.249.212.187]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1p2TAg-0005Dt-Va for qemu-devel@nongnu.org; Tue, 06 Dec 2022 03:19:26 -0500 Received: from kwepemi100025.china.huawei.com (unknown [172.30.72.56]) by szxga01-in.huawei.com (SkyGuard) with ESMTP id 4NRCwh1KwPzmWPw; Tue, 6 Dec 2022 16:18:28 +0800 (CST) Received: from DESKTOP-27KDQMV.china.huawei.com (10.174.148.223) by kwepemi100025.china.huawei.com (7.221.188.158) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Tue, 6 Dec 2022 16:18:44 +0800 To: , , , CC: , , , , , , Longpeng Subject: [PATCH v2 1/2] vhost: configure all host notifiers in a single MR transaction Date: Tue, 6 Dec 2022 16:18:40 +0800 Message-ID: <20221206081841.2458-2-longpeng2@huawei.com> X-Mailer: git-send-email 2.25.0.windows.1 In-Reply-To: <20221206081841.2458-1-longpeng2@huawei.com> References: <20221206081841.2458-1-longpeng2@huawei.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Originating-IP: [10.174.148.223] X-ClientProxiedBy: dggems706-chm.china.huawei.com (10.3.19.183) To kwepemi100025.china.huawei.com (7.221.188.158) X-CFilter-Loop: Reflected Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=45.249.212.187; envelope-from=longpeng2@huawei.com; helo=szxga01-in.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: "Longpeng(Mike)" From: "Longpeng(Mike)" via Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZM-MESSAGEID: 1670314794258100003 Content-Type: text/plain; charset="utf-8" From: Longpeng This allows the vhost device to batch the setup of all its host notifiers. This significantly reduces the device starting time, e.g. the time spend on enabling notifiers reduce from 376ms to 9.1ms for a VM with 64 vCPUs and 3 vhost-vDPA generic devices[1] (64vq per device) [1] https://www.mail-archive.com/qemu-devel@nongnu.org/msg921541.html Signed-off-by: Longpeng --- hw/virtio/vhost.c | 40 ++++++++++++++++++++++++++++++++++++++-- 1 file changed, 38 insertions(+), 2 deletions(-) diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c index 7fb008bc9e..16f8391d86 100644 --- a/hw/virtio/vhost.c +++ b/hw/virtio/vhost.c @@ -1507,7 +1507,7 @@ void vhost_dev_cleanup(struct vhost_dev *hdev) int vhost_dev_enable_notifiers(struct vhost_dev *hdev, VirtIODevice *vdev) { BusState *qbus =3D BUS(qdev_get_parent_bus(DEVICE(vdev))); - int i, r, e; + int i, n, r, e; =20 /* We will pass the notifiers to the kernel, make sure that QEMU * doesn't interfere. @@ -1518,6 +1518,12 @@ int vhost_dev_enable_notifiers(struct vhost_dev *hde= v, VirtIODevice *vdev) goto fail; } =20 + /* + * Batch all the host notifiers in a single transaction to avoid + * quadratic time complexity in address_space_update_ioeventfds(). + */ + memory_region_transaction_begin(); + for (i =3D 0; i < hdev->nvqs; ++i) { r =3D virtio_bus_set_host_notifier(VIRTIO_BUS(qbus), hdev->vq_inde= x + i, true); @@ -1527,8 +1533,12 @@ int vhost_dev_enable_notifiers(struct vhost_dev *hde= v, VirtIODevice *vdev) } } =20 + memory_region_transaction_commit(); + return 0; fail_vq: + /* save i for a second iteration after transaction is committed. */ + n =3D i; while (--i >=3D 0) { e =3D virtio_bus_set_host_notifier(VIRTIO_BUS(qbus), hdev->vq_inde= x + i, false); @@ -1536,8 +1546,18 @@ fail_vq: error_report("vhost VQ %d notifier cleanup error: %d", i, -r); } assert (e >=3D 0); - virtio_bus_cleanup_host_notifier(VIRTIO_BUS(qbus), hdev->vq_index = + i); } + + /* + * The transaction expects the ioeventfds to be open when it + * commits. Do it now, before the cleanup loop. + */ + memory_region_transaction_commit(); + + while (--n >=3D 0) { + virtio_bus_cleanup_host_notifier(VIRTIO_BUS(qbus), hdev->vq_index = + n); + } + virtio_device_release_ioeventfd(vdev); fail: return r; @@ -1553,6 +1573,12 @@ void vhost_dev_disable_notifiers(struct vhost_dev *h= dev, VirtIODevice *vdev) BusState *qbus =3D BUS(qdev_get_parent_bus(DEVICE(vdev))); int i, r; =20 + /* + * Batch all the host notifiers in a single transaction to avoid + * quadratic time complexity in address_space_update_ioeventfds(). + */ + memory_region_transaction_begin(); + for (i =3D 0; i < hdev->nvqs; ++i) { r =3D virtio_bus_set_host_notifier(VIRTIO_BUS(qbus), hdev->vq_inde= x + i, false); @@ -1560,8 +1586,18 @@ void vhost_dev_disable_notifiers(struct vhost_dev *h= dev, VirtIODevice *vdev) error_report("vhost VQ %d notifier cleanup failed: %d", i, -r); } assert (r >=3D 0); + } + + /* + * The transaction expects the ioeventfds to be open when it + * commits. Do it now, before the cleanup loop. + */ + memory_region_transaction_commit(); + + for (i =3D 0; i < hdev->nvqs; ++i) { virtio_bus_cleanup_host_notifier(VIRTIO_BUS(qbus), hdev->vq_index = + i); } + virtio_device_release_ioeventfd(vdev); } =20 --=20 2.23.0 From nobody Sun May 19 06:50:49 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1670314787; cv=none; d=zohomail.com; s=zohoarc; b=k6HRc45OOKuKEd2j+7qbXurgtz/SHwRNyPnzkJAmVZoPabvFOGv5878tKzpFN0Dxz6gvjGMnOJGa8KhqdBMFfDs1MWWSJjjGxbi5oal1RCEMw9vdGgaGKrXKV8pwldtnoiXE+ynKDvME6R3uoTTnqKEHhIuK5wJhlmNW+uccAZ8= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1670314787; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:Reply-To:References:Sender:Subject:To; bh=NIOWvuH+k3hAFgqyXkgLV2vgl5pJBKTbD1JwStp33Dw=; b=VmTfEMk0tyqsqH92jYxEDNOEtksP1ojf2w9PNHpIutJ0pbMWL9Ex8ELkflnynHblaW1mtH2SVsr8IAnsyCIL+JNVJNRCvT0/9trWKcpmur6uZNMfMaXo+E1iYRr6UYlQUlU/Y6CrlK320bfmRQSrOtjDqFkvbzny82VzorEoYhY= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1670314787021674.8150664496242; Tue, 6 Dec 2022 00:19:47 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1p2TAs-0007UO-NK; Tue, 06 Dec 2022 03:19:34 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1p2TAj-0007OT-Ef for qemu-devel@nongnu.org; Tue, 06 Dec 2022 03:19:27 -0500 Received: from szxga01-in.huawei.com ([45.249.212.187]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1p2TAg-0005Ds-LZ for qemu-devel@nongnu.org; Tue, 06 Dec 2022 03:19:25 -0500 Received: from kwepemi100025.china.huawei.com (unknown [172.30.72.56]) by szxga01-in.huawei.com (SkyGuard) with ESMTP id 4NRCwh39NKzmWQF; Tue, 6 Dec 2022 16:18:28 +0800 (CST) Received: from DESKTOP-27KDQMV.china.huawei.com (10.174.148.223) by kwepemi100025.china.huawei.com (7.221.188.158) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Tue, 6 Dec 2022 16:18:45 +0800 To: , , , CC: , , , , , , Longpeng Subject: [PATCH v2 2/2] vdpa: commit all host notifier MRs in a single MR transaction Date: Tue, 6 Dec 2022 16:18:41 +0800 Message-ID: <20221206081841.2458-3-longpeng2@huawei.com> X-Mailer: git-send-email 2.25.0.windows.1 In-Reply-To: <20221206081841.2458-1-longpeng2@huawei.com> References: <20221206081841.2458-1-longpeng2@huawei.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Originating-IP: [10.174.148.223] X-ClientProxiedBy: dggems706-chm.china.huawei.com (10.3.19.183) To kwepemi100025.china.huawei.com (7.221.188.158) X-CFilter-Loop: Reflected Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=45.249.212.187; envelope-from=longpeng2@huawei.com; helo=szxga01-in.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: "Longpeng(Mike)" From: "Longpeng(Mike)" via Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZM-MESSAGEID: 1670314788227100003 Content-Type: text/plain; charset="utf-8" From: Longpeng This allows the vhost-vdpa device to batch the setup of all its MRs of host notifiers. This significantly reduces the device starting time, e.g. the time spend on setup the host notifier MRs reduce from 423ms to 32ms for a VM with 64 vCPUs and 3 vhost-vDPA generic devices[1] (64vq per device). [1] https://www.mail-archive.com/qemu-devel@nongnu.org/msg921541.html Signed-off-by: Longpeng --- hw/virtio/vhost-vdpa.c | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c index 7468e44b87..eb233cf08a 100644 --- a/hw/virtio/vhost-vdpa.c +++ b/hw/virtio/vhost-vdpa.c @@ -547,9 +547,18 @@ static void vhost_vdpa_host_notifiers_uninit(struct vh= ost_dev *dev, int n) { int i; =20 + /* + * Pack all the changes to the memory regions in a single + * transaction to avoid a few updating of the address space + * topology. + */ + memory_region_transaction_begin(); + for (i =3D dev->vq_index; i < dev->vq_index + n; i++) { vhost_vdpa_host_notifier_uninit(dev, i); } + + memory_region_transaction_commit(); } =20 static void vhost_vdpa_host_notifiers_init(struct vhost_dev *dev) @@ -562,16 +571,25 @@ static void vhost_vdpa_host_notifiers_init(struct vho= st_dev *dev) return; } =20 + /* + * Pack all the changes to the memory regions in a single + * transaction to avoid a few updating of the address space + * topology. + */ + memory_region_transaction_begin(); + for (i =3D dev->vq_index; i < dev->vq_index + dev->nvqs; i++) { if (vhost_vdpa_host_notifier_init(dev, i)) { goto err; } } =20 + memory_region_transaction_commit(); return; =20 err: vhost_vdpa_host_notifiers_uninit(dev, i - dev->vq_index); + memory_region_transaction_commit(); return; } =20 --=20 2.23.0