From nobody Wed Apr 16 17:29:01 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1612434155; cv=none; d=zohomail.com; s=zohoarc; b=dUEV4b11TWAxIo9d9KQM1Zbo+q0Gb9vZb6BzXJCDPQ56AiJPOrKkC+8TgitCkrxupxSifE9AhHxX9WPci1IbjgoZMxxHctxy/roHAiuGQCuiEsuu6JoJgpGmnZCPhINIfXkQLKSQZiAb7lUdxV5+G7VXvPVzDEcjfyWyPEexy9M= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1612434155; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=dNurtIFNm+Trbo8N7qQEKTqr1ZEtax4PCeh/XzV4iJ0=; b=jwCUtAdkZS/RdsRbufNuQifUuBOAKYel1tn2NjQDvJ7AyJkn9SXp5hSu6VU60ZdnfwIDPge17PHyp/3cMJuX8PB4qeGK5//TnJK2TuY5BMsj43BTAJU2rifpa+haK0fJJhRiBtgrGnXIRgPAWjQR9Qyf13Ei6J7WUsNAFQDPPzc= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1612434155064430.2667752420233; Thu, 4 Feb 2021 02:22:35 -0800 (PST) Received: from localhost ([::1]:45360 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l7bmT-0004cu-Mf for importer@patchew.org; Thu, 04 Feb 2021 05:22:33 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:52908) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l7bTO-0006Nb-LW for qemu-devel@nongnu.org; Thu, 04 Feb 2021 05:02:50 -0500 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:55633) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1l7bTL-000228-K5 for qemu-devel@nongnu.org; Thu, 04 Feb 2021 05:02:50 -0500 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-551-dzKFjtpiPEiSeADfNHrQFA-1; Thu, 04 Feb 2021 05:02:45 -0500 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 9A540107ACE6; Thu, 4 Feb 2021 10:02:43 +0000 (UTC) Received: from localhost (ovpn-115-89.ams2.redhat.com [10.36.115.89]) by smtp.corp.redhat.com (Postfix) with ESMTP id 224D95D762; Thu, 4 Feb 2021 10:02:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1612432967; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=dNurtIFNm+Trbo8N7qQEKTqr1ZEtax4PCeh/XzV4iJ0=; b=FN82orO4vskp3HU1x9iCVU4iuGgTtoCqDeezv9eVUomshLJXGkhamS9y+pg57KPdIa9Lk8 2KOyI7OvT1ZGhWv1NJmt+g2a+uireFyO+UHzq0B9x8il/WeRF3VkpuXs2FA4YtrB7ARLxo AaJZ8bvvm5eWt3KR2AZrWnbqWNNudcs= X-MC-Unique: dzKFjtpiPEiSeADfNHrQFA-1 From: Stefan Hajnoczi To: Peter Maydell , qemu-devel@nongnu.org Subject: [PULL 23/27] multi-process: Synchronize remote memory Date: Thu, 4 Feb 2021 09:58:30 +0000 Message-Id: <20210204095834.345749-24-stefanha@redhat.com> In-Reply-To: <20210204095834.345749-1-stefanha@redhat.com> References: <20210204095834.345749-1-stefanha@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.15 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=stefanha@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=216.205.24.124; envelope-from=stefanha@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -32 X-Spam_score: -3.3 X-Spam_bar: --- X-Spam_report: (-3.3 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.539, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Elena Ufimtseva , Fam Zheng , Thomas Huth , =?UTF-8?q?Daniel=20P=2E=20Berrang=C3=A9?= , Eduardo Habkost , qemu-block@nongnu.org, John G Johnson , "Denis V. Lunev" , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , "Michael S. Tsirkin" , Wainer dos Santos Moschetta , Stefan Hajnoczi , Paolo Bonzini , Igor Mammedov , Jagannathan Raman , =?UTF-8?q?Alex=20Benn=C3=A9e?= Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Type: text/plain; charset="utf-8" From: Jagannathan Raman Add ProxyMemoryListener object which is used to keep the view of the RAM in sync between QEMU and remote process. A MemoryListener is registered for system-memory AddressSpace. The listener sends SYNC_SYSMEM message to the remote process when memory listener commits the changes to memory, the remote process receives the message and processes it in the handler for SYNC_SYSMEM message. Signed-off-by: Jagannathan Raman Signed-off-by: John G Johnson Signed-off-by: Elena Ufimtseva Reviewed-by: Stefan Hajnoczi Message-id: 04fe4e6a9ca90d4f11ab6f59be7652f5b086a071.1611938319.git.jag.ram= an@oracle.com Signed-off-by: Stefan Hajnoczi --- MAINTAINERS | 2 + include/hw/remote/proxy-memory-listener.h | 28 +++ include/hw/remote/proxy.h | 2 + hw/remote/message.c | 4 + hw/remote/proxy-memory-listener.c | 227 ++++++++++++++++++++++ hw/remote/proxy.c | 6 + hw/remote/meson.build | 1 + 7 files changed, 270 insertions(+) create mode 100644 include/hw/remote/proxy-memory-listener.h create mode 100644 hw/remote/proxy-memory-listener.c diff --git a/MAINTAINERS b/MAINTAINERS index 51a8859357..3b0ea950fc 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -3213,6 +3213,8 @@ F: include/hw/remote/memory.h F: hw/remote/memory.c F: hw/remote/proxy.c F: include/hw/remote/proxy.h +F: hw/remote/proxy-memory-listener.c +F: include/hw/remote/proxy-memory-listener.h =20 Build and test automation ------------------------- diff --git a/include/hw/remote/proxy-memory-listener.h b/include/hw/remote/= proxy-memory-listener.h new file mode 100644 index 0000000000..c4f3efb928 --- /dev/null +++ b/include/hw/remote/proxy-memory-listener.h @@ -0,0 +1,28 @@ +/* + * Copyright =C2=A9 2018, 2021 Oracle and/or its affiliates. + * + * This work is licensed under the terms of the GNU GPL, version 2 or late= r. + * See the COPYING file in the top-level directory. + * + */ + +#ifndef PROXY_MEMORY_LISTENER_H +#define PROXY_MEMORY_LISTENER_H + +#include "exec/memory.h" +#include "io/channel.h" + +typedef struct ProxyMemoryListener { + MemoryListener listener; + + int n_mr_sections; + MemoryRegionSection *mr_sections; + + QIOChannel *ioc; +} ProxyMemoryListener; + +void proxy_memory_listener_configure(ProxyMemoryListener *proxy_listener, + QIOChannel *ioc); +void proxy_memory_listener_deconfigure(ProxyMemoryListener *proxy_listener= ); + +#endif diff --git a/include/hw/remote/proxy.h b/include/hw/remote/proxy.h index ea7fa4fb3c..12888b4f90 100644 --- a/include/hw/remote/proxy.h +++ b/include/hw/remote/proxy.h @@ -11,6 +11,7 @@ =20 #include "hw/pci/pci.h" #include "io/channel.h" +#include "hw/remote/proxy-memory-listener.h" =20 #define TYPE_PCI_PROXY_DEV "x-pci-proxy-dev" OBJECT_DECLARE_SIMPLE_TYPE(PCIProxyDev, PCI_PROXY_DEV) @@ -36,6 +37,7 @@ struct PCIProxyDev { QemuMutex io_mutex; QIOChannel *ioc; Error *migration_blocker; + ProxyMemoryListener proxy_listener; ProxyMemoryRegion region[PCI_NUM_REGIONS]; }; =20 diff --git a/hw/remote/message.c b/hw/remote/message.c index f2e84457e0..25341d8ad2 100644 --- a/hw/remote/message.c +++ b/hw/remote/message.c @@ -17,6 +17,7 @@ #include "sysemu/runstate.h" #include "hw/pci/pci.h" #include "exec/memattrs.h" +#include "hw/remote/memory.h" =20 static void process_config_write(QIOChannel *ioc, PCIDevice *dev, MPQemuMsg *msg, Error **errp); @@ -61,6 +62,9 @@ void coroutine_fn mpqemu_remote_msg_loop_co(void *data) case MPQEMU_CMD_BAR_READ: process_bar_read(com->ioc, &msg, &local_err); break; + case MPQEMU_CMD_SYNC_SYSMEM: + remote_sysmem_reconfig(&msg, &local_err); + break; default: error_setg(&local_err, "Unknown command (%d) received for device %s" diff --git a/hw/remote/proxy-memory-listener.c b/hw/remote/proxy-memory-lis= tener.c new file mode 100644 index 0000000000..af1fa6f5aa --- /dev/null +++ b/hw/remote/proxy-memory-listener.c @@ -0,0 +1,227 @@ +/* + * Copyright =C2=A9 2018, 2021 Oracle and/or its affiliates. + * + * This work is licensed under the terms of the GNU GPL, version 2 or late= r. + * See the COPYING file in the top-level directory. + * + */ + +#include "qemu/osdep.h" +#include "qemu-common.h" + +#include "qemu/compiler.h" +#include "qemu/int128.h" +#include "qemu/range.h" +#include "exec/memory.h" +#include "exec/cpu-common.h" +#include "cpu.h" +#include "exec/ram_addr.h" +#include "exec/address-spaces.h" +#include "qapi/error.h" +#include "hw/remote/mpqemu-link.h" +#include "hw/remote/proxy-memory-listener.h" + +/* + * TODO: get_fd_from_hostaddr(), proxy_mrs_can_merge() and + * proxy_memory_listener_commit() defined below perform tasks similar to t= he + * functions defined in vhost-user.c. These functions are good candidates + * for refactoring. + * + */ + +static void proxy_memory_listener_reset(MemoryListener *listener) +{ + ProxyMemoryListener *proxy_listener =3D container_of(listener, + ProxyMemoryListener, + listener); + int mrs; + + for (mrs =3D 0; mrs < proxy_listener->n_mr_sections; mrs++) { + memory_region_unref(proxy_listener->mr_sections[mrs].mr); + } + + g_free(proxy_listener->mr_sections); + proxy_listener->mr_sections =3D NULL; + proxy_listener->n_mr_sections =3D 0; +} + +static int get_fd_from_hostaddr(uint64_t host, ram_addr_t *offset) +{ + MemoryRegion *mr; + ram_addr_t off; + + /** + * Assumes that the host address is a valid address as it's + * coming from the MemoryListener system. In the case host + * address is not valid, the following call would return + * the default subregion of "system_memory" region, and + * not NULL. So it's not possible to check for NULL here. + */ + mr =3D memory_region_from_host((void *)(uintptr_t)host, &off); + + if (offset) { + *offset =3D off; + } + + return memory_region_get_fd(mr); +} + +static bool proxy_mrs_can_merge(uint64_t host, uint64_t prev_host, size_t = size) +{ + if (((prev_host + size) !=3D host)) { + return false; + } + + if (get_fd_from_hostaddr(host, NULL) !=3D + get_fd_from_hostaddr(prev_host, NULL)) { + return false; + } + + return true; +} + +static bool try_merge(ProxyMemoryListener *proxy_listener, + MemoryRegionSection *section) +{ + uint64_t mrs_size, mrs_gpa, mrs_page; + MemoryRegionSection *prev_sec; + bool merged =3D false; + uintptr_t mrs_host; + RAMBlock *mrs_rb; + + if (!proxy_listener->n_mr_sections) { + return false; + } + + mrs_rb =3D section->mr->ram_block; + mrs_page =3D (uint64_t)qemu_ram_pagesize(mrs_rb); + mrs_size =3D int128_get64(section->size); + mrs_gpa =3D section->offset_within_address_space; + mrs_host =3D (uintptr_t)memory_region_get_ram_ptr(section->mr) + + section->offset_within_region; + + if (get_fd_from_hostaddr(mrs_host, NULL) < 0) { + return true; + } + + mrs_host =3D mrs_host & ~(mrs_page - 1); + mrs_gpa =3D mrs_gpa & ~(mrs_page - 1); + mrs_size =3D ROUND_UP(mrs_size, mrs_page); + + prev_sec =3D proxy_listener->mr_sections + + (proxy_listener->n_mr_sections - 1); + uint64_t prev_gpa_start =3D prev_sec->offset_within_address_space; + uint64_t prev_size =3D int128_get64(prev_sec->size); + uint64_t prev_gpa_end =3D range_get_last(prev_gpa_start, prev_size); + uint64_t prev_host_start =3D + (uintptr_t)memory_region_get_ram_ptr(prev_sec->mr) + + prev_sec->offset_within_region; + uint64_t prev_host_end =3D range_get_last(prev_host_start, prev_size); + + if (mrs_gpa <=3D (prev_gpa_end + 1)) { + g_assert(mrs_gpa > prev_gpa_start); + + if ((section->mr =3D=3D prev_sec->mr) && + proxy_mrs_can_merge(mrs_host, prev_host_start, + (mrs_gpa - prev_gpa_start))) { + uint64_t max_end =3D MAX(prev_host_end, mrs_host + mrs_size); + merged =3D true; + prev_sec->offset_within_address_space =3D + MIN(prev_gpa_start, mrs_gpa); + prev_sec->offset_within_region =3D + MIN(prev_host_start, mrs_host) - + (uintptr_t)memory_region_get_ram_ptr(prev_sec->mr); + prev_sec->size =3D int128_make64(max_end - MIN(prev_host_start, + mrs_host)); + } + } + + return merged; +} + +static void proxy_memory_listener_region_addnop(MemoryListener *listener, + MemoryRegionSection *secti= on) +{ + ProxyMemoryListener *proxy_listener =3D container_of(listener, + ProxyMemoryListener, + listener); + + if (!memory_region_is_ram(section->mr) || + memory_region_is_rom(section->mr)) { + return; + } + + if (try_merge(proxy_listener, section)) { + return; + } + + ++proxy_listener->n_mr_sections; + proxy_listener->mr_sections =3D g_renew(MemoryRegionSection, + proxy_listener->mr_sections, + proxy_listener->n_mr_sections); + proxy_listener->mr_sections[proxy_listener->n_mr_sections - 1] =3D *se= ction; + proxy_listener->mr_sections[proxy_listener->n_mr_sections - 1].fv =3D = NULL; + memory_region_ref(section->mr); +} + +static void proxy_memory_listener_commit(MemoryListener *listener) +{ + ProxyMemoryListener *proxy_listener =3D container_of(listener, + ProxyMemoryListener, + listener); + MPQemuMsg msg; + MemoryRegionSection *section; + ram_addr_t offset; + uintptr_t host_addr; + int region; + Error *local_err =3D NULL; + + memset(&msg, 0, sizeof(MPQemuMsg)); + + msg.cmd =3D MPQEMU_CMD_SYNC_SYSMEM; + msg.num_fds =3D proxy_listener->n_mr_sections; + msg.size =3D sizeof(SyncSysmemMsg); + if (msg.num_fds > REMOTE_MAX_FDS) { + error_report("Number of fds is more than %d", REMOTE_MAX_FDS); + return; + } + + for (region =3D 0; region < proxy_listener->n_mr_sections; region++) { + section =3D &proxy_listener->mr_sections[region]; + msg.data.sync_sysmem.gpas[region] =3D + section->offset_within_address_space; + msg.data.sync_sysmem.sizes[region] =3D int128_get64(section->size); + host_addr =3D (uintptr_t)memory_region_get_ram_ptr(section->mr) + + section->offset_within_region; + msg.fds[region] =3D get_fd_from_hostaddr(host_addr, &offset); + msg.data.sync_sysmem.offsets[region] =3D offset; + } + if (!mpqemu_msg_send(&msg, proxy_listener->ioc, &local_err)) { + error_report_err(local_err); + } +} + +void proxy_memory_listener_deconfigure(ProxyMemoryListener *proxy_listener) +{ + memory_listener_unregister(&proxy_listener->listener); + + proxy_memory_listener_reset(&proxy_listener->listener); +} + +void proxy_memory_listener_configure(ProxyMemoryListener *proxy_listener, + QIOChannel *ioc) +{ + proxy_listener->n_mr_sections =3D 0; + proxy_listener->mr_sections =3D NULL; + + proxy_listener->ioc =3D ioc; + + proxy_listener->listener.begin =3D proxy_memory_listener_reset; + proxy_listener->listener.commit =3D proxy_memory_listener_commit; + proxy_listener->listener.region_add =3D proxy_memory_listener_region_a= ddnop; + proxy_listener->listener.region_nop =3D proxy_memory_listener_region_a= ddnop; + proxy_listener->listener.priority =3D 10; + + memory_listener_register(&proxy_listener->listener, + &address_space_memory); +} diff --git a/hw/remote/proxy.c b/hw/remote/proxy.c index 22eb422231..472b2df335 100644 --- a/hw/remote/proxy.c +++ b/hw/remote/proxy.c @@ -19,6 +19,8 @@ #include "qemu/sockets.h" #include "hw/remote/mpqemu-link.h" #include "qemu/error-report.h" +#include "hw/remote/proxy-memory-listener.h" +#include "qom/object.h" =20 static void pci_proxy_dev_realize(PCIDevice *device, Error **errp) { @@ -52,6 +54,8 @@ static void pci_proxy_dev_realize(PCIDevice *device, Erro= r **errp) =20 qemu_mutex_init(&dev->io_mutex); qio_channel_set_blocking(dev->ioc, true, NULL); + + proxy_memory_listener_configure(&dev->proxy_listener, dev->ioc); } =20 static void pci_proxy_dev_exit(PCIDevice *pdev) @@ -65,6 +69,8 @@ static void pci_proxy_dev_exit(PCIDevice *pdev) migrate_del_blocker(dev->migration_blocker); =20 error_free(dev->migration_blocker); + + proxy_memory_listener_deconfigure(&dev->proxy_listener); } =20 static void config_op_send(PCIProxyDev *pdev, uint32_t addr, uint32_t *val, diff --git a/hw/remote/meson.build b/hw/remote/meson.build index 569cd20edf..7f11be4736 100644 --- a/hw/remote/meson.build +++ b/hw/remote/meson.build @@ -7,5 +7,6 @@ remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('= remote-obj.c')) remote_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('proxy.c')) =20 specific_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('memory.c')) +specific_ss.add(when: 'CONFIG_MULTIPROCESS', if_true: files('proxy-memory-= listener.c')) =20 softmmu_ss.add_all(when: 'CONFIG_MULTIPROCESS', if_true: remote_ss) --=20 2.29.2