From nobody Sun Apr 13 18:17:55 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1612546459; cv=none; d=zohomail.com; s=zohoarc; b=c6rib/SKAs/ItLwtLXTonTvERBdWK63eNEmqVSkMM5pfJ+/q2z2aNHbWufOlxi+RpT2xrJx4+zdQrpUZ6XWcNGNTFBU7+pE9VC/o9zgkN8N0Wxfbshn4RQb0w3/EFmBulnwveamWrZvuMVjSwNjVpUbHkfBcKqKxe9DX0p8RBBo= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1612546459; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=dNurtIFNm+Trbo8N7qQEKTqr1ZEtax4PCeh/XzV4iJ0=; b=eDpX4HQKs8NYpZrsVTUFjNxvUQD9zvGUBNHYeiMzhkoUYbgISAYvaknIe8n50eBM6f63avD+LfhXipRATfrlPcsBXpsryieXoGwmy6AHoQEo72Zon0SXNcwAsytTq241qWb5i1btjHqaI2Co3FFKGSJ1PPkIYJNRmKdRvP7JAlk= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1612546459722923.1666374798772; Fri, 5 Feb 2021 09:34:19 -0800 (PST) Received: from localhost ([::1]:51116 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l84zq-0006xD-FJ for importer@patchew.org; Fri, 05 Feb 2021 12:34:18 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:43316) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l84Ic-0004aK-RP for qemu-devel@nongnu.org; Fri, 05 Feb 2021 11:49:38 -0500 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:38406) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1l84IZ-0007aU-Q1 for qemu-devel@nongnu.org; Fri, 05 Feb 2021 11:49:38 -0500 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-27-oWtRJEqFMO2TjBrF8Q21dg-1; Fri, 05 Feb 2021 11:49:30 -0500 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 1FDFA15721; Fri, 5 Feb 2021 16:49:29 +0000 (UTC) Received: from localhost (ovpn-112-54.ams2.redhat.com [10.36.112.54]) by smtp.corp.redhat.com (Postfix) with ESMTP id B2B315C648; Fri, 5 Feb 2021 16:49:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1612543775; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=dNurtIFNm+Trbo8N7qQEKTqr1ZEtax4PCeh/XzV4iJ0=; b=NKzs+P9dXr2w9jVwg7fZokS65LgIFTxpU/xJqxl5VgaevbZrnp1CLXhCazlWQYO4TTonYK gcs45n+wHV0fg5R0iv0Ql4NDAbAcwSxCuAhdoklX4idREZe4gila9RKAk961kjOyfDoVkt XU1P+SM4w8swKDJdp/sqSRMQGqkAHeo= X-MC-Unique: oWtRJEqFMO2TjBrF8Q21dg-1 From: Stefan Hajnoczi To: Peter Maydell , qemu-devel@nongnu.org Subject: [PULL v3 23/27] multi-process: Synchronize remote memory Date: Fri, 5 Feb 2021 16:44:55 +0000 Message-Id: <20210205164459.432714-24-stefanha@redhat.com> In-Reply-To: <20210205164459.432714-1-stefanha@redhat.com> References: <20210205164459.432714-1-stefanha@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=stefanha@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=216.205.24.124; envelope-from=stefanha@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -30 X-Spam_score: -3.1 X-Spam_bar: --- X-Spam_report: (-3.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.352, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Fam Zheng , John G Johnson , Thomas Huth , Jagannathan Raman , Stefan Hajnoczi , qemu-block@nongnu.org, "Michael S. Tsirkin" , "Denis V. Lunev" , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , =?UTF-8?q?Daniel=20P=2E=20Berrang=C3=A9?= , Elena Ufimtseva , Wainer dos Santos Moschetta , Igor Mammedov , Paolo Bonzini , =?UTF-8?q?Alex=20Benn=C3=A9e?= , Eduardo Habkost Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Type: text/plain; charset="utf-8" From: Jagannathan Raman Add ProxyMemoryListener object which is used to keep the view of the RAM in sync between QEMU and remote process. A MemoryListener is registered for system-memory AddressSpace. The listener sends SYNC_SYSMEM message to the remote process when memory listener commits the changes to memory, the remote process receives the message and processes it in the handler for SYNC_SYSMEM message. Signed-off-by: Jagannathan Raman Signed-off-by: John G Johnson Signed-off-by: Elena Ufimtseva Reviewed-by: Stefan Hajnoczi Message-id: 04fe4e6a9ca90d4f11ab6f59be7652f5b086a071.1611938319.git.jag.ram= an@oracle.com Signed-off-by: Stefan Hajnoczi --- MAINTAINERS | 2 + include/hw/remote/proxy-memory-listener.h | 28 +++ include/hw/remote/proxy.h | 2 + hw/remote/message.c | 4 + hw/remote/proxy-memory-listener.c | 227 ++++++++++++++++++++++ hw/remote/proxy.c | 6 + hw/remote/meson.build | 1 + 7 files changed, 270 insertions(+) create mode 100644 include/hw/remote/proxy-memory-listener.h create mode 100644 hw/remote/proxy-memory-listener.c diff --git a/MAINTAINERS b/MAINTAINERS index 51a8859357..3b0ea950fc 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -3213,6 +3213,8 @@ F: include/hw/remote/memory.h F: hw/remote/memory.c F: hw/remote/proxy.c F: include/hw/remote/proxy.h +F: hw/remote/proxy-memory-listener.c +F: include/hw/remote/proxy-memory-listener.h =20 Build and test automation ------------------------- diff --git a/include/hw/remote/proxy-memory-listener.h b/include/hw/remote/= proxy-memory-listener.h new file mode 100644 index 0000000000..c4f3efb928 --- /dev/null +++ b/include/hw/remote/proxy-memory-listener.h @@ -0,0 +1,28 @@ +/* + * Copyright =C2=A9 2018, 2021 Oracle and/or its affiliates. + * + * This work is licensed under the terms of the GNU GPL, version 2 or late= r. + * See the COPYING file in the top-level directory. + * + */ + +#ifndef PROXY_MEMORY_LISTENER_H +#define PROXY_MEMORY_LISTENER_H + +#include "exec/memory.h" +#include "io/channel.h" + +typedef struct ProxyMemoryListener { + MemoryListener listener; + + int n_mr_sections; + MemoryRegionSection *mr_sections; + + QIOChannel *ioc; +} ProxyMemoryListener; + +void proxy_memory_listener_configure(ProxyMemoryListener *proxy_listener, + QIOChannel *ioc); +void proxy_memory_listener_deconfigure(ProxyMemoryListener *proxy_listener= ); + +#endif diff --git a/include/hw/remote/proxy.h b/include/hw/remote/proxy.h index ea7fa4fb3c..12888b4f90 100644 --- a/include/hw/remote/proxy.h +++ b/include/hw/remote/proxy.h @@ -11,6 +11,7 @@ =20 #include "hw/pci/pci.h" #include "io/channel.h" +#include "hw/remote/proxy-memory-listener.h" =20 #define TYPE_PCI_PROXY_DEV "x-pci-proxy-dev" OBJECT_DECLARE_SIMPLE_TYPE(PCIProxyDev, PCI_PROXY_DEV) @@ -36,6 +37,7 @@ struct PCIProxyDev { QemuMutex io_mutex; QIOChannel *ioc; Error *migration_blocker; + ProxyMemoryListener proxy_listener; ProxyMemoryRegion region[PCI_NUM_REGIONS]; }; =20 diff --git a/hw/remote/message.c b/hw/remote/message.c index f2e84457e0..25341d8ad2 100644 --- a/hw/remote/message.c +++ b/hw/remote/message.c @@ -17,6 +17,7 @@ #include "sysemu/runstate.h" #include "hw/pci/pci.h" #include "exec/memattrs.h" +#include "hw/remote/memory.h" =20 static void process_config_write(QIOChannel *ioc, PCIDevice *dev, MPQemuMsg *msg, Error **errp); @@ -61,6 +62,9 @@ void coroutine_fn mpqemu_remote_msg_loop_co(void *data) case MPQEMU_CMD_BAR_READ: process_bar_read(com->ioc, &msg, &local_err); break; + case MPQEMU_CMD_SYNC_SYSMEM: + remote_sysmem_reconfig(&msg, &local_err); + break; default: error_setg(&local_err, "Unknown command (%d) received for device %s" diff --git a/hw/remote/proxy-memory-listener.c b/hw/remote/proxy-memory-lis= tener.c new file mode 100644 index 0000000000..af1fa6f5aa --- /dev/null +++ b/hw/remote/proxy-memory-listener.c @@ -0,0 +1,227 @@ +/* + * Copyright =C2=A9 2018, 2021 Oracle and/or its affiliates. + * + * This work is licensed under the terms of the GNU GPL, version 2 or late= r. + * See the COPYING file in the top-level directory. + * + */ + +#include "qemu/osdep.h" +#include "qemu-common.h" + +#include "qemu/compiler.h" +#include "qemu/int128.h" +#include "qemu/range.h" +#include "exec/memory.h" +#include "exec/cpu-common.h" +#include "cpu.h" +#include "exec/ram_addr.h" +#include "exec/address-spaces.h" +#include "qapi/error.h" +#include "hw/remote/mpqemu-link.h" +#include "hw/remote/proxy-memory-listener.h" + +/* + * TODO: get_fd_from_hostaddr(), proxy_mrs_can_merge() and + * proxy_memory_listener_commit() defined below perform tasks similar to t= he + * functions defined in vhost-user.c. These functions are good candidates + * for refactoring. + * + */ + +static void proxy_memory_listener_reset(MemoryListener *listener) +{ + ProxyMemoryListener *proxy_listener =3D container_of(listener, + ProxyMemoryListener, + listener); + int mrs; + + for (mrs =3D 0; mrs < proxy_listener->n_mr_sections; mrs++) { + memory_region_unref(proxy_listener->mr_sections[mrs].mr); + } + + g_free(proxy_listener->mr_sections); + proxy_listener->mr_sections =3D NULL; + proxy_listener->n_mr_sections =3D 0; +} + +static int get_fd_from_hostaddr(uint64_t host, ram_addr_t *offset) +{ + MemoryRegion *mr; + ram_addr_t off; + + /** + * Assumes that the host address is a valid address as it's + * coming from the MemoryListener system. In the case host + * address is not valid, the following call would return + * the default subregion of "system_memory" region, and + * not NULL. So it's not possible to check for NULL here. + */ + mr =3D memory_region_from_host((void *)(uintptr_t)host, &off); + + if (offset) { + *offset =3D off; + } + + return memory_region_get_fd(mr); +} + +static bool proxy_mrs_can_merge(uint64_t host, uint64_t prev_host, size_t = size) +{ + if (((prev_host + size) !=3D host)) { + return false; + } + + if (get_fd_from_hostaddr(host, NULL) !=3D + get_fd_from_hostaddr(prev_host, NULL)) { + return false; + } + + return true; +} + +static bool try_merge(ProxyMemoryListener *proxy_listener, + MemoryRegionSection *section) +{ + uint64_t mrs_size, mrs_gpa, mrs_page; + MemoryRegionSection *prev_sec; + bool merged =3D false; + uintptr_t mrs_host; + RAMBlock *mrs_rb; + + if (!proxy_listener->n_mr_sections) { + return false; + } + + mrs_rb =3D section->mr->ram_block; + mrs_page =3D (uint64_t)qemu_ram_pagesize(mrs_rb); + mrs_size =3D int128_get64(section->size); + mrs_gpa =3D section->offset_within_address_space; + mrs_host =3D (uintptr_t)memory_region_get_ram_ptr(section->mr) + + section->offset_within_region; + + if (get_fd_from_hostaddr(mrs_host, NULL) < 0) { + return true; + } + + mrs_host =3D mrs_host & ~(mrs_page - 1); + mrs_gpa =3D mrs_gpa & ~(mrs_page - 1); + mrs_size =3D ROUND_UP(mrs_size, mrs_page); + + prev_sec =3D proxy_listener->mr_sections + + (proxy_listener->n_mr_sections - 1); + uint64_t prev_gpa_start =3D prev_sec->offset_within_address_space; + uint64_t prev_size =3D int128_get64(prev_sec->size); + uint64_t prev_gpa_end =3D range_get_last(prev_gpa_start, prev_size); + uint64_t prev_host_start =3D + (uintptr_t)memory_region_get_ram_ptr(prev_sec->mr) + + prev_sec->offset_within_region; + uint64_t prev_host_end =3D range_get_last(prev_host_start, prev_size); + + if (mrs_gpa <=3D (prev_gpa_end + 1)) { + g_assert(mrs_gpa > prev_gpa_start); + + if ((section->mr =3D=3D prev_sec->mr) && + proxy_mrs_can_merge(mrs_host, prev_host_start, + (mrs_gpa - prev_gpa_start))) { + uint64_t max_end =3D MAX(prev_host_end, mrs_host + mrs_size); + merged =3D true; + prev_sec->offset_within_address_space =3D + MIN(prev_gpa_start, mrs_gpa); + prev_sec->offset_within_region =3D + MIN(prev_host_start, mrs_host) - + (uintptr_t)memory_region_get_ram_ptr(prev_sec->mr); + prev_sec->size =3D int128_make64(max_end - MIN(prev_host_start, + mrs_host)); + } + } + + return merged; +} + +static void proxy_memory_listener_region_addnop(MemoryListener *listener, + MemoryRegionSection *secti= on) +{ + ProxyMemoryListener *proxy_listener =3D container_of(listener, + ProxyMemoryListener, + listener); + + if (!memory_region_is_ram(section->mr) || + memory_region_is_rom(section->mr)) { + return; + } + + if (try_merge(proxy_listener, section)) { + return; + } + + ++proxy_listener->n_mr_sections; + proxy_listener->mr_sections =3D g_renew(MemoryRegionSection, + proxy_listener->mr_sections, + proxy_listener->n_mr_sections); + proxy_listener->mr_sections[proxy_listener->n_mr_sections - 1] =3D *se= ction; + proxy_listener->mr_sections[proxy_listener->n_mr_sections - 1].fv =3D = NULL; + memory_region_ref(section->mr); +} + +static void proxy_memory_listener_commit(MemoryListener *listener) +{ + ProxyMemoryListener *proxy_listener =3D container_of(listener, + ProxyMemoryListener, + listener); + MPQemuMsg msg; + MemoryRegionSection *section; + ram_addr_t offset; + uintptr_t host_addr; + int region; + Error *local_err =3D NULL; + + memset(&msg, 0, sizeof(MPQemuMsg)); + + msg.cmd =3D MPQEMU_CMD_SYNC_SYSMEM; + msg.num_fds =3D proxy_listener->n_mr_sections; + msg.size =3D sizeof(SyncSysmemMsg); + if (msg.num_fds > REMOTE_MAX_FDS) { + error_report("Number of fds is more than %d", REMOTE_MAX_FDS); + return; + } + + for (region =3D 0; region < proxy_listener->n_mr_sections; region++) { + section =3D &proxy_listener->mr_sections[region]; + msg.data.sync_sysmem.gpas[region] =3D + section->offset_within_address_space; + msg.data.sync_sysmem.sizes[region] =3D int128_get64(section->size); + host_addr =3D (uintptr_t)memory_region_get_ram_ptr(section->mr) + + section->offset_within_region; + msg.fds[region] =3D get_fd_from_hostaddr(host_addr, &offset); + msg.data.sync_sysmem.offsets[region] =3D offset; + } + if (!mpqemu_msg_send(&msg, proxy_listener->ioc, &local_err)) { + error_report_err(local_err); + } +} + +void proxy_memory_listener_deconfigure(ProxyMemoryListener *proxy_listener) +{ + memory_listener_unregister(&proxy_listener->listener); + + proxy_memory_listener_reset(&proxy_listener->listener); +} + +void proxy_memory_listener_configure(ProxyMemoryListener *proxy_listener, + QIOChannel *ioc) +{ + proxy_listener->n_mr_sections =3D 0; + proxy_listener->mr_sections =3D NULL; + + proxy_listener->ioc =3D ioc; + + proxy_listener->listener.begin =3D proxy_memory_listener_reset; + proxy_listener->listener.commit =3D proxy_memory_listener_commit; + proxy_listener->listener.region_add =3D proxy_memory_listener_region_a= ddnop; + proxy_listener->listener.region_nop =3D proxy_memory_listener_region_a= ddnop; + proxy_listener->listener.priority =3D 10; + + memory_listener_register(&proxy_listener->listener, + &address_space_memory); +} diff --git a/hw/remote/proxy.c b/hw/remote/proxy.c index 22eb422231..472b2df335 100644 --- a/hw/remote/proxy.c +++ b/hw/remote/proxy.c @@ -19,6 +19,8 @@ #include "qemu/sockets.h" #include "hw/remote/mpqemu-link.h" #include "qemu/error-report.h" +#include "hw/remote/proxy-memory-listener.h" +#include "qom/object.h" =20 static void pci_proxy_dev_realize(PCIDevice *device, Error **errp) { @@ -52,6 +54,8 @@ static void pci_proxy_dev_realize(PCIDevice *device, Erro= r **errp) =20 qemu_mutex_init(&dev->io_mutex); qio_channel_set_blocking(dev->ioc, true, NULL); + + proxy_memory_listener_configure(&dev->proxy_listener, dev->ioc); } =20 static void pci_proxy_dev_exit(PCIDevice *pdev) @@ -65,6 +69,8 @@ static void pci_proxy_dev_exit(PCIDevice *pdev) migrate_del_blocker(dev->migration_blocker); =20 error_free(dev->migration_blocker); + + proxy_memory_listener_deconfigure(&dev->proxy_listener); } =20 static void config_op_send(PCIProxyDev *pdev, uint32_t addr, uint32_t *val, diff --git a/hw/remote/meson.build b/hw/remote/meson.build index 569cd20edf..7f11be4736 100644 --- a/hw/remote/meson.build +++ b/hw/remote/meson.build @@ -7,5 +7,6 @@ remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('= remote-obj.c')) remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('proxy.c')) =20 specific_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('memory.c')) +specific_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('proxy-memory-= listener.c')) =20 softmmu_ss.add_all(when: 'CONFIG_MULTIPROCESS', if_true: remote_ss) --=20 2.29.2