From nobody Mon Nov 25 03:38:18 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1717503359; cv=none; d=zohomail.com; s=zohoarc; b=AG2IhHkJ67w3dujWvCQ9kY3tMx5TQSXVwsIVQQ+15ruKd+LGXPhnyiLXKSKSaz1J4GkYxD+ygGtPc/c4mrUXVzCAOqH73Y5yCr+iK9ZoNLHe/XqR6NEii+tqU8KpCtDvSH9O4jfn449hRYidBDwBJ6j8DryhQ+z9IJEb2cNMe2g= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1717503359; h=Content-Type:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:Reply-To:Reply-To:References:Sender:Subject:Subject:To:To:Message-Id; bh=HUSAhxZGVZw/KvXgdyZO6dxTTAfy+mtyaHRgsds+Pj0=; b=Pqj+wLTX8zoAfZTHYxZXEx34NbpG1jRNf0XIYXKqxR5nnQVhtSvOLtuDRPu7U0eB8V7EUqKkuh6k6R0Bbqk/RQCeYK3XXo2Ixx9Zj5gaNOaXgL0JhPFK6zxpZWrGkcHhL0ZtarnFe8TxI09FvOdd0aOFv1bFiCAfpNH2ulknUTc= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 17175033590751010.409026438111; Tue, 4 Jun 2024 05:15:59 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sET3t-0003lz-Qb; Tue, 04 Jun 2024 08:14:45 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sET3r-0003jo-Gi for qemu-devel@nongnu.org; Tue, 04 Jun 2024 08:14:43 -0400 Received: from szxga02-in.huawei.com ([45.249.212.188]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sET3i-0006wV-15 for qemu-devel@nongnu.org; Tue, 04 Jun 2024 08:14:42 -0400 Received: from mail.maildlp.com (unknown [172.19.163.48]) by szxga02-in.huawei.com (SkyGuard) with ESMTP id 4VtqHD2wdYzclR7; Tue, 4 Jun 2024 20:12:56 +0800 (CST) Received: from dggpemf200006.china.huawei.com (unknown [7.185.36.61]) by mail.maildlp.com (Postfix) with ESMTPS id 085BB180060; Tue, 4 Jun 2024 20:14:20 +0800 (CST) Received: from DESKTOP-8LI8G6S.china.huawei.com (10.173.124.235) by dggpemf200006.china.huawei.com (7.185.36.61) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.11; Tue, 4 Jun 2024 20:14:18 +0800 To: CC: , , , , , , , , , , , , , , , Jialin Wang Subject: [PATCH 1/6] migration: remove RDMA live migration temporarily Date: Tue, 4 Jun 2024 20:14:07 +0800 Message-ID: <1717503252-51884-2-git-send-email-arei.gonglei@huawei.com> X-Mailer: git-send-email 2.8.2.windows.1 In-Reply-To: <1717503252-51884-1-git-send-email-arei.gonglei@huawei.com> References: <1717503252-51884-1-git-send-email-arei.gonglei@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.173.124.235] X-ClientProxiedBy: dggems704-chm.china.huawei.com (10.3.19.181) To dggpemf200006.china.huawei.com (7.185.36.61) Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=45.249.212.188; envelope-from=arei.gonglei@huawei.com; helo=szxga02-in.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Gonglei From: Gonglei via Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZM-MESSAGEID: 1717503360856100001 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Jialin Wang The new RDMA live migration will be introduced in the upcoming few commits. Signed-off-by: Jialin Wang Signed-off-by: Gonglei --- docs/rdma.txt | 420 ---- meson.build | 6 - migration/meson.build | 1 - migration/migration-stats.c | 5 +- migration/migration-stats.h | 4 - migration/migration.c | 20 - migration/migration.h | 9 - migration/options.c | 16 - migration/options.h | 2 - migration/qemu-file.c | 1 - migration/ram.c | 90 +- migration/rdma.c | 4184 ---------------------------------- migration/rdma.h | 69 - migration/savevm.c | 2 +- migration/trace-events | 68 +- qapi/migration.json | 13 +- scripts/analyze-migration.py | 3 - 17 files changed, 10 insertions(+), 4903 deletions(-) delete mode 100644 docs/rdma.txt delete mode 100644 migration/rdma.c delete mode 100644 migration/rdma.h diff --git a/docs/rdma.txt b/docs/rdma.txt deleted file mode 100644 index bd8dd799a9..0000000000 --- a/docs/rdma.txt +++ /dev/null @@ -1,420 +0,0 @@ -(RDMA: Remote Direct Memory Access) -RDMA Live Migration Specification, Version # 1 -=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D -Wiki: https://wiki.qemu.org/Features/RDMALiveMigration -Github: git@github.com:hinesmr/qemu.git, 'rdma' branch - -Copyright (C) 2013 Michael R. Hines - -An *exhaustive* paper (2010) shows additional performance details -linked on the QEMU wiki above. - -Contents: -=3D=3D=3D=3D=3D=3D=3D=3D=3D -* Introduction -* Before running -* Running -* Performance -* RDMA Migration Protocol Description -* Versioning and Capabilities -* QEMUFileRDMA Interface -* Migration of VM's ram -* Error handling -* TODO - -Introduction: -=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D - -RDMA helps make your migration more deterministic under heavy load because -of the significantly lower latency and higher throughput over TCP/IP. This= is -because the RDMA I/O architecture reduces the number of interrupts and -data copies by bypassing the host networking stack. In particular, a TCP-b= ased -migration, under certain types of memory-bound workloads, may take a more -unpredictable amount of time to complete the migration if the amount of -memory tracked during each live migration iteration round cannot keep pace -with the rate of dirty memory produced by the workload. - -RDMA currently comes in two flavors: both Ethernet based (RoCE, or RDMA -over Converged Ethernet) as well as Infiniband-based. This implementation = of -migration using RDMA is capable of using both technologies because of -the use of the OpenFabrics OFED software stack that abstracts out the -programming model irrespective of the underlying hardware. - -Refer to openfabrics.org or your respective RDMA hardware vendor for -an understanding on how to verify that you have the OFED software stack -installed in your environment. You should be able to successfully link -against the "librdmacm" and "libibverbs" libraries and development headers -for a working build of QEMU to run successfully using RDMA Migration. - -BEFORE RUNNING: -=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D - -Use of RDMA during migration requires pinning and registering memory -with the hardware. This means that memory must be physically resident -before the hardware can transmit that memory to another machine. -If this is not acceptable for your application or product, then the use -of RDMA migration may in fact be harmful to co-located VMs or other -software on the machine if there is not sufficient memory available to -relocate the entire footprint of the virtual machine. If so, then the -use of RDMA is discouraged and it is recommended to use standard TCP migra= tion. - -Experimental: Next, decide if you want dynamic page registration. -For example, if you have an 8GB RAM virtual machine, but only 1GB -is in active use, then enabling this feature will cause all 8GB to -be pinned and resident in memory. This feature mostly affects the -bulk-phase round of the migration and can be enabled for extremely -high-performance RDMA hardware using the following command: - -QEMU Monitor Command: -$ migrate_set_capability rdma-pin-all on # disabled by default - -Performing this action will cause all 8GB to be pinned, so if that's -not what you want, then please ignore this step altogether. - -On the other hand, this will also significantly speed up the bulk round -of the migration, which can greatly reduce the "total" time of your migrat= ion. -Example performance of this using an idle VM in the previous example -can be found in the "Performance" section. - -Note: for very large virtual machines (hundreds of GBs), pinning all -*all* of the memory of your virtual machine in the kernel is very expensive -may extend the initial bulk iteration time by many seconds, -and thus extending the total migration time. However, this will not -affect the determinism or predictability of your migration you will -still gain from the benefits of advanced pinning with RDMA. - -RUNNING: -=3D=3D=3D=3D=3D=3D=3D=3D - -First, set the migration speed to match your hardware's capabilities: - -QEMU Monitor Command: -$ migrate_set_parameter max-bandwidth 40g # or whatever is the MAX of your= RDMA device - -Next, on the destination machine, add the following to the QEMU command li= ne: - -qemu ..... -incoming rdma:host:port - -Finally, perform the actual migration on the source machine: - -QEMU Monitor Command: -$ migrate -d rdma:host:port - -PERFORMANCE -=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D - -Here is a brief summary of total migration time and downtime using RDMA: -Using a 40gbps infiniband link performing a worst-case stress test, -using an 8GB RAM virtual machine: - -Using the following command: -$ apt-get install stress -$ stress --vm-bytes 7500M --vm 1 --vm-keep - -1. Migration throughput: 26 gigabits/second. -2. Downtime (stop time) varies between 15 and 100 milliseconds. - -EFFECTS of memory registration on bulk phase round: - -For example, in the same 8GB RAM example with all 8GB of memory in -active use and the VM itself is completely idle using the same 40 gbps -infiniband link: - -1. rdma-pin-all disabled total time: approximately 7.5 seconds @ 9.5 Gbps -2. rdma-pin-all enabled total time: approximately 4 seconds @ 26 Gbps - -These numbers would of course scale up to whatever size virtual machine -you have to migrate using RDMA. - -Enabling this feature does *not* have any measurable affect on -migration *downtime*. This is because, without this feature, all of the -memory will have already been registered already in advance during -the bulk round and does not need to be re-registered during the successive -iteration rounds. - -RDMA Protocol Description: -=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D - -Migration with RDMA is separated into two parts: - -1. The transmission of the pages using RDMA -2. Everything else (a control channel is introduced) - -"Everything else" is transmitted using a formal -protocol now, consisting of infiniband SEND messages. - -An infiniband SEND message is the standard ibverbs -message used by applications of infiniband hardware. -The only difference between a SEND message and an RDMA -message is that SEND messages cause notifications -to be posted to the completion queue (CQ) on the -infiniband receiver side, whereas RDMA messages (used -for VM's ram) do not (to behave like an actual DMA). - -Messages in infiniband require two things: - -1. registration of the memory that will be transmitted -2. (SEND only) work requests to be posted on both - sides of the network before the actual transmission - can occur. - -RDMA messages are much easier to deal with. Once the memory -on the receiver side is registered and pinned, we're -basically done. All that is required is for the sender -side to start dumping bytes onto the link. - -(Memory is not released from pinning until the migration -completes, given that RDMA migrations are very fast.) - -SEND messages require more coordination because the -receiver must have reserved space (using a receive -work request) on the receive queue (RQ) before QEMUFileRDMA -can start using them to carry all the bytes as -a control transport for migration of device state. - -To begin the migration, the initial connection setup is -as follows (migration-rdma.c): - -1. Receiver and Sender are started (command line or libvirt): -2. Both sides post two RQ work requests -3. Receiver does listen() -4. Sender does connect() -5. Receiver accept() -6. Check versioning and capabilities (described later) - -At this point, we define a control channel on top of SEND messages -which is described by a formal protocol. Each SEND message has a -header portion and a data portion (but together are transmitted -as a single SEND message). - -Header: - * Length (of the data portion, uint32, network byte orde= r) - * Type (what command to perform, uint32, network byte = order) - * Repeat (Number of commands in data portion, same type = only) - -The 'Repeat' field is here to support future multiple page registrations -in a single message without any need to change the protocol itself -so that the protocol is compatible against multiple versions of QEMU. -Version #1 requires that all server implementations of the protocol must -check this field and register all requests found in the array of commands = located -in the data portion and return an equal number of results in the response. -The maximum number of repeats is hard-coded to 4096. This is a conservative -limit based on the maximum size of a SEND message along with empirical -observations on the maximum future benefit of simultaneous page registrati= ons. - -The 'type' field has 12 different command values: - 1. Unused - 2. Error (sent to the source during bad things) - 3. Ready (control-channel is available) - 4. QEMU File (for sending non-live device state) - 5. RAM Blocks request (used right after connection setup) - 6. RAM Blocks result (used right after connection setup) - 7. Compress page (zap zero page and skip registration) - 8. Register request (dynamic chunk registration) - 9. Register result ('rkey' to be used by sender) - 10. Register finished (registration for current iteration fin= ished) - 11. Unregister request (unpin previously registered memory) - 12. Unregister finished (confirmation that unpin completed) - -A single control message, as hinted above, can contain within the data -portion an array of many commands of the same type. If there is more than -one command, then the 'repeat' field will be greater than 1. - -After connection setup, message 5 & 6 are used to exchange ram block -information and optionally pin all the memory if requested by the user. - -After ram block exchange is completed, we have two protocol-level -functions, responsible for communicating control-channel commands -using the above list of values: - -Logically: - -qemu_rdma_exchange_recv(header, expected command type) - -1. We transmit a READY command to let the sender know that - we are *ready* to receive some data bytes on the control channel. -2. Before attempting to receive the expected command, we post another - RQ work request to replace the one we just used up. -3. Block on a CQ event channel and wait for the SEND to arrive. -4. When the send arrives, librdmacm will unblock us. -5. Verify that the command-type and version received matches the one we ex= pected. - -qemu_rdma_exchange_send(header, data, optional response header & data): - -1. Block on the CQ event channel waiting for a READY command - from the receiver to tell us that the receiver - is *ready* for us to transmit some new bytes. -2. Optionally: if we are expecting a response from the command - (that we have not yet transmitted), let's post an RQ - work request to receive that data a few moments later. -3. When the READY arrives, librdmacm will - unblock us and we immediately post a RQ work request - to replace the one we just used up. -4. Now, we can actually post the work request to SEND - the requested command type of the header we were asked for. -5. Optionally, if we are expecting a response (as before), - we block again and wait for that response using the additional - work request we previously posted. (This is used to carry - 'Register result' commands #6 back to the sender which - hold the rkey need to perform RDMA. Note that the virtual address - corresponding to this rkey was already exchanged at the beginning - of the connection (described below). - -All of the remaining command types (not including 'ready') -described above all use the aforementioned two functions to do the hard wo= rk: - -1. After connection setup, RAMBlock information is exchanged using - this protocol before the actual migration begins. This information incl= udes - a description of each RAMBlock on the server side as well as the virtua= l addresses - and lengths of each RAMBlock. This is used by the client to determine t= he - start and stop locations of chunks and how to register them dynamically - before performing the RDMA operations. -2. During runtime, once a 'chunk' becomes full of pages ready to - be sent with RDMA, the registration commands are used to ask the - other side to register the memory for this chunk and respond - with the result (rkey) of the registration. -3. Also, the QEMUFile interfaces also call these functions (described belo= w) - when transmitting non-live state, such as devices or to send - its own protocol information during the migration process. -4. Finally, zero pages are only checked if a page has not yet been registe= red - using chunk registration (or not checked at all and unconditionally - written if chunk registration is disabled. This is accomplished using - the "Compress" command listed above. If the page *has* been registered - then we check the entire chunk for zero. Only if the entire chunk is - zero, then we send a compress command to zap the page on the other side. - -Versioning and Capabilities -=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D -Current version of the protocol is version #1. - -The same version applies to both for protocol traffic and capabilities -negotiation. (i.e. There is only one version number that is referred to -by all communication). - -librdmacm provides the user with a 'private data' area to be exchanged -at connection-setup time before any infiniband traffic is generated. - -Header: - * Version (protocol version validated before send/recv occurs), - uint32, network byte order - * Flags (bitwise OR of each capability), - uint32, network byte order - -There is no data portion of this header right now, so there is -no length field. The maximum size of the 'private data' section -is only 192 bytes per the Infiniband specification, so it's not -very useful for data anyway. This structure needs to remain small. - -This private data area is a convenient place to check for protocol -versioning because the user does not need to register memory to -transmit a few bytes of version information. - -This is also a convenient place to negotiate capabilities -(like dynamic page registration). - -If the version is invalid, we throw an error. - -If the version is new, we only negotiate the capabilities that the -requested version is able to perform and ignore the rest. - -Currently there is only one capability in Version #1: dynamic page registr= ation - -Finally: Negotiation happens with the Flags field: If the primary-VM -sets a flag, but the destination does not support this capability, it -will return a zero-bit for that flag and the primary-VM will understand -that as not being an available capability and will thus disable that -capability on the primary-VM side. - -QEMUFileRDMA Interface: -=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D - -QEMUFileRDMA introduces a couple of new functions: - -1. qemu_rdma_get_buffer() (QEMUFileOps rdma_read_ops) -2. qemu_rdma_put_buffer() (QEMUFileOps rdma_write_ops) - -These two functions are very short and simply use the protocol -describe above to deliver bytes without changing the upper-level -users of QEMUFile that depend on a bytestream abstraction. - -Finally, how do we handoff the actual bytes to get_buffer()? - -Again, because we're trying to "fake" a bytestream abstraction -using an analogy not unlike individual UDP frames, we have -to hold on to the bytes received from control-channel's SEND -messages in memory. - -Each time we receive a complete "QEMU File" control-channel -message, the bytes from SEND are copied into a small local holding area. - -Then, we return the number of bytes requested by get_buffer() -and leave the remaining bytes in the holding area until get_buffer() -comes around for another pass. - -If the buffer is empty, then we follow the same steps -listed above and issue another "QEMU File" protocol command, -asking for a new SEND message to re-fill the buffer. - -Migration of VM's ram: -=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D - -At the beginning of the migration, (migration-rdma.c), -the sender and the receiver populate the list of RAMBlocks -to be registered with each other into a structure. -Then, using the aforementioned protocol, they exchange a -description of these blocks with each other, to be used later -during the iteration of main memory. This description includes -a list of all the RAMBlocks, their offsets and lengths, virtual -addresses and possibly includes pre-registered RDMA keys in case dynamic -page registration was disabled on the server-side, otherwise not. - -Main memory is not migrated with the aforementioned protocol, -but is instead migrated with normal RDMA Write operations. - -Pages are migrated in "chunks" (hard-coded to 1 Megabyte right now). -Chunk size is not dynamic, but it could be in a future implementation. -There's nothing to indicate that this is useful right now. - -When a chunk is full (or a flush() occurs), the memory backed by -the chunk is registered with librdmacm is pinned in memory on -both sides using the aforementioned protocol. -After pinning, an RDMA Write is generated and transmitted -for the entire chunk. - -Chunks are also transmitted in batches: This means that we -do not request that the hardware signal the completion queue -for the completion of *every* chunk. The current batch size -is about 64 chunks (corresponding to 64 MB of memory). -Only the last chunk in a batch must be signaled. -This helps keep everything as asynchronous as possible -and helps keep the hardware busy performing RDMA operations. - -Error-handling: -=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D - -Infiniband has what is called a "Reliable, Connected" -link (one of 4 choices). This is the mode in which -we use for RDMA migration. - -If a *single* message fails, -the decision is to abort the migration entirely and -cleanup all the RDMA descriptors and unregister all -the memory. - -After cleanup, the Virtual Machine is returned to normal -operation the same way that would happen if the TCP -socket is broken during a non-RDMA based migration. - -TODO: -=3D=3D=3D=3D=3D -1. Currently, 'ulimit -l' mlock() limits as well as cgroups swap limits - are not compatible with infiniband memory pinning and will result in - an aborted migration (but with the source VM left unaffected). -2. Use of the recent /proc//pagemap would likely speed up - the use of KSM and ballooning while using RDMA. -3. Also, some form of balloon-device usage tracking would also - help alleviate some issues. -4. Use LRU to provide more fine-grained direction of UNREGISTER - requests for unpinning memory in an overcommitted environment. -5. Expose UNREGISTER support to the user by way of workload-specific - hints about application behavior. diff --git a/meson.build b/meson.build index 6386607144..3894f1f942 100644 --- a/meson.build +++ b/meson.build @@ -2425,12 +2425,6 @@ if rbd.found() dependencies: rbd, prefix: '#include ')) endif -if rdma.found() - config_host_data.set('HAVE_IBV_ADVISE_MR', - cc.has_function('ibv_advise_mr', - dependencies: rdma, - prefix: '#include ')) -endif =20 have_asan_fiber =3D false if get_option('sanitizers') and \ diff --git a/migration/meson.build b/migration/meson.build index bdc3244bce..4e8a9ccf3e 100644 --- a/migration/meson.build +++ b/migration/meson.build @@ -37,7 +37,6 @@ else system_ss.add(files('colo-stubs.c')) endif =20 -system_ss.add(when: rdma, if_true: files('rdma.c')) system_ss.add(when: zstd, if_true: files('multifd-zstd.c')) =20 specific_ss.add(when: 'CONFIG_SYSTEM_ONLY', diff --git a/migration/migration-stats.c b/migration/migration-stats.c index f690b98a03..9bc8d7018f 100644 --- a/migration/migration-stats.c +++ b/migration/migration-stats.c @@ -62,9 +62,8 @@ void migration_rate_reset(void) uint64_t migration_transferred_bytes(void) { uint64_t multifd =3D stat64_get(&mig_stats.multifd_bytes); - uint64_t rdma =3D stat64_get(&mig_stats.rdma_bytes); uint64_t qemu_file =3D stat64_get(&mig_stats.qemu_file_transferred); =20 - trace_migration_transferred_bytes(qemu_file, multifd, rdma); - return qemu_file + multifd + rdma; + trace_migration_transferred_bytes(qemu_file, multifd); + return qemu_file + multifd; } diff --git a/migration/migration-stats.h b/migration/migration-stats.h index 05290ade76..6b87e133f1 100644 --- a/migration/migration-stats.h +++ b/migration/migration-stats.h @@ -93,10 +93,6 @@ typedef struct { * Maximum amount of data we can send in a cycle. */ Stat64 rate_limit_max; - /* - * Number of bytes sent through RDMA. - */ - Stat64 rdma_bytes; /* * Number of pages transferred that were full of zeros. */ diff --git a/migration/migration.c b/migration/migration.c index e1b269624c..6b9ad4ff5f 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -25,7 +25,6 @@ #include "sysemu/runstate.h" #include "sysemu/sysemu.h" #include "sysemu/cpu-throttle.h" -#include "rdma.h" #include "ram.h" #include "migration/global_state.h" #include "migration/misc.h" @@ -645,18 +644,6 @@ static void qemu_start_incoming_migration(const char *= uri, bool has_channels, } else if (saddr->type =3D=3D SOCKET_ADDRESS_TYPE_FD) { fd_start_incoming_migration(saddr->u.fd.str, errp); } -#ifdef CONFIG_RDMA - } else if (addr->transport =3D=3D MIGRATION_ADDRESS_TYPE_RDMA) { - if (migrate_xbzrle()) { - error_setg(errp, "RDMA and XBZRLE can't be used together"); - return; - } - if (migrate_multifd()) { - error_setg(errp, "RDMA and multifd can't be used together"); - return; - } - rdma_start_incoming_migration(&addr->u.rdma, errp); -#endif } else if (addr->transport =3D=3D MIGRATION_ADDRESS_TYPE_EXEC) { exec_start_incoming_migration(addr->u.exec.args, errp); } else if (addr->transport =3D=3D MIGRATION_ADDRESS_TYPE_FILE) { @@ -744,9 +731,7 @@ process_incoming_migration_co(void *opaque) migrate_set_state(&mis->state, MIGRATION_STATUS_SETUP, MIGRATION_STATUS_ACTIVE); =20 - mis->loadvm_co =3D qemu_coroutine_self(); ret =3D qemu_loadvm_state(mis->from_src_file); - mis->loadvm_co =3D NULL; =20 trace_vmstate_downtime_checkpoint("dst-precopy-loadvm-completed"); =20 @@ -1668,7 +1653,6 @@ int migrate_init(MigrationState *s, Error **errp) s->iteration_initial_bytes =3D 0; s->threshold_size =3D 0; s->switchover_acked =3D false; - s->rdma_migration =3D false; /* * set mig_stats memory to zero for a new migration */ @@ -2062,10 +2046,6 @@ void qmp_migrate(const char *uri, bool has_channels, } else if (saddr->type =3D=3D SOCKET_ADDRESS_TYPE_FD) { fd_start_outgoing_migration(s, saddr->u.fd.str, &local_err); } -#ifdef CONFIG_RDMA - } else if (addr->transport =3D=3D MIGRATION_ADDRESS_TYPE_RDMA) { - rdma_start_outgoing_migration(s, &addr->u.rdma, &local_err); -#endif } else if (addr->transport =3D=3D MIGRATION_ADDRESS_TYPE_EXEC) { exec_start_outgoing_migration(s, addr->u.exec.args, &local_err); } else if (addr->transport =3D=3D MIGRATION_ADDRESS_TYPE_FILE) { diff --git a/migration/migration.h b/migration/migration.h index 6af01362d4..714643fe7e 100644 --- a/migration/migration.h +++ b/migration/migration.h @@ -162,13 +162,6 @@ struct MigrationIncomingState { =20 int state; =20 - /* - * The incoming migration coroutine, non-NULL during qemu_loadvm_state= (). - * Used to wake the migration incoming coroutine from rdma code. How m= uch is - * it safe - it's a question. - */ - Coroutine *loadvm_co; - /* The coroutine we should enter (back) after failover */ Coroutine *colo_incoming_co; QemuSemaphore colo_incoming_sem; @@ -455,8 +448,6 @@ struct MigrationState { * switchover has been received. */ bool switchover_acked; - /* Is this a rdma migration */ - bool rdma_migration; }; =20 void migrate_set_state(int *state, int old_state, int new_state); diff --git a/migration/options.c b/migration/options.c index 5ab5b6d85d..601cd712b7 100644 --- a/migration/options.c +++ b/migration/options.c @@ -165,7 +165,6 @@ Property migration_properties[] =3D { =20 /* Migration capabilities */ DEFINE_PROP_MIG_CAP("x-xbzrle", MIGRATION_CAPABILITY_XBZRLE), - DEFINE_PROP_MIG_CAP("x-rdma-pin-all", MIGRATION_CAPABILITY_RDMA_PIN_AL= L), DEFINE_PROP_MIG_CAP("x-auto-converge", MIGRATION_CAPABILITY_AUTO_CONVE= RGE), DEFINE_PROP_MIG_CAP("x-zero-blocks", MIGRATION_CAPABILITY_ZERO_BLOCKS), DEFINE_PROP_MIG_CAP("x-events", MIGRATION_CAPABILITY_EVENTS), @@ -287,13 +286,6 @@ bool migrate_postcopy_ram(void) return s->capabilities[MIGRATION_CAPABILITY_POSTCOPY_RAM]; } =20 -bool migrate_rdma_pin_all(void) -{ - MigrationState *s =3D migrate_get_current(); - - return s->capabilities[MIGRATION_CAPABILITY_RDMA_PIN_ALL]; -} - bool migrate_release_ram(void) { MigrationState *s =3D migrate_get_current(); @@ -357,13 +349,6 @@ bool migrate_postcopy(void) return migrate_postcopy_ram() || migrate_dirty_bitmaps(); } =20 -bool migrate_rdma(void) -{ - MigrationState *s =3D migrate_get_current(); - - return s->rdma_migration; -} - bool migrate_tls(void) { MigrationState *s =3D migrate_get_current(); @@ -422,7 +407,6 @@ INITIALIZE_MIGRATE_CAPS_SET(check_caps_background_snaps= hot, MIGRATION_CAPABILITY_PAUSE_BEFORE_SWITCHOVER, MIGRATION_CAPABILITY_AUTO_CONVERGE, MIGRATION_CAPABILITY_RELEASE_RAM, - MIGRATION_CAPABILITY_RDMA_PIN_ALL, MIGRATION_CAPABILITY_XBZRLE, MIGRATION_CAPABILITY_X_COLO, MIGRATION_CAPABILITY_VALIDATE_UUID, diff --git a/migration/options.h b/migration/options.h index 4b21cc2669..cb26708ebf 100644 --- a/migration/options.h +++ b/migration/options.h @@ -35,7 +35,6 @@ bool migrate_multifd(void); bool migrate_pause_before_switchover(void); bool migrate_postcopy_blocktime(void); bool migrate_postcopy_preempt(void); -bool migrate_rdma_pin_all(void); bool migrate_release_ram(void); bool migrate_return_path(void); bool migrate_validate_uuid(void); @@ -52,7 +51,6 @@ bool migrate_zero_copy_send(void); =20 bool migrate_multifd_flush_after_each_section(void); bool migrate_postcopy(void); -bool migrate_rdma(void); bool migrate_tls(void); =20 /* capabilities helpers */ diff --git a/migration/qemu-file.c b/migration/qemu-file.c index b6d2f588bd..09fdfc2b4d 100644 --- a/migration/qemu-file.c +++ b/migration/qemu-file.c @@ -31,7 +31,6 @@ #include "trace.h" #include "options.h" #include "qapi/error.h" -#include "rdma.h" #include "io/channel-file.h" =20 #define IO_BUF_SIZE 32768 diff --git a/migration/ram.c b/migration/ram.c index ceea586b06..6b027c7fd7 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -57,7 +57,6 @@ #include "qemu/iov.h" #include "multifd.h" #include "sysemu/runstate.h" -#include "rdma.h" #include "options.h" #include "sysemu/dirtylimit.h" #include "sysemu/kvm.h" @@ -88,7 +87,6 @@ #define RAM_SAVE_FLAG_EOS 0x10 #define RAM_SAVE_FLAG_CONTINUE 0x20 #define RAM_SAVE_FLAG_XBZRLE 0x40 -/* 0x80 is reserved in rdma.h for RAM_SAVE_FLAG_HOOK */ #define RAM_SAVE_FLAG_MULTIFD_FLUSH 0x200 /* We can't use any flag that is bigger than 0x200 */ =20 @@ -1168,32 +1166,6 @@ static int save_zero_page(RAMState *rs, PageSearchSt= atus *pss, return len; } =20 -/* - * @pages: the number of pages written by the control path, - * < 0 - error - * > 0 - number of pages written - * - * Return true if the pages has been saved, otherwise false is returned. - */ -static bool control_save_page(PageSearchStatus *pss, - ram_addr_t offset, int *pages) -{ - int ret; - - ret =3D rdma_control_save_page(pss->pss_channel, pss->block->offset, o= ffset, - TARGET_PAGE_SIZE); - if (ret =3D=3D RAM_SAVE_CONTROL_NOT_SUPP) { - return false; - } - - if (ret =3D=3D RAM_SAVE_CONTROL_DELAYED) { - *pages =3D 1; - return true; - } - *pages =3D ret; - return true; -} - /* * directly send the page to the stream * @@ -1997,11 +1969,6 @@ int ram_save_queue_pages(const char *rbname, ram_add= r_t start, ram_addr_t len, static int ram_save_target_page_legacy(RAMState *rs, PageSearchStatus *pss) { ram_addr_t offset =3D ((ram_addr_t)pss->page) << TARGET_PAGE_BITS; - int res; - - if (control_save_page(pss, offset, &res)) { - return res; - } =20 if (save_zero_page(rs, pss, offset)) { return 1; @@ -3041,20 +3008,6 @@ static int ram_save_setup(QEMUFile *f, void *opaque,= Error **errp) } } =20 - ret =3D rdma_registration_start(f, RAM_CONTROL_SETUP); - if (ret < 0) { - error_setg(errp, "%s: failed to start RDMA registration", __func__= ); - qemu_file_set_error(f, ret); - return ret; - } - - ret =3D rdma_registration_stop(f, RAM_CONTROL_SETUP); - if (ret < 0) { - error_setg(errp, "%s: failed to stop RDMA registration", __func__); - qemu_file_set_error(f, ret); - return ret; - } - migration_ops =3D g_malloc0(sizeof(MigrationOps)); =20 if (migrate_multifd()) { @@ -3148,12 +3101,6 @@ static int ram_save_iterate(QEMUFile *f, void *opaqu= e) /* Read version before ram_list.blocks */ smp_rmb(); =20 - ret =3D rdma_registration_start(f, RAM_CONTROL_ROUND); - if (ret < 0) { - qemu_file_set_error(f, ret); - goto out; - } - t0 =3D qemu_clock_get_ns(QEMU_CLOCK_REALTIME); i =3D 0; while ((ret =3D migration_rate_exceeded(f)) =3D=3D 0 || @@ -3197,16 +3144,6 @@ static int ram_save_iterate(QEMUFile *f, void *opaqu= e) } } =20 - /* - * Must occur before EOS (or any QEMUFile operation) - * because of RDMA protocol. - */ - ret =3D rdma_registration_stop(f, RAM_CONTROL_ROUND); - if (ret < 0) { - qemu_file_set_error(f, ret); - } - -out: if (ret >=3D 0 && migration_is_setup_or_active()) { if (migrate_multifd() && migrate_multifd_flush_after_each_section(= ) && @@ -3251,12 +3188,6 @@ static int ram_save_complete(QEMUFile *f, void *opaq= ue) migration_bitmap_sync_precopy(rs, true); } =20 - ret =3D rdma_registration_start(f, RAM_CONTROL_FINISH); - if (ret < 0) { - qemu_file_set_error(f, ret); - return ret; - } - /* try transferring iterative blocks of memory */ =20 /* flush all remaining blocks regardless of rate limiting */ @@ -3275,12 +3206,6 @@ static int ram_save_complete(QEMUFile *f, void *opaq= ue) } } qemu_mutex_unlock(&rs->bitmap_mutex); - - ret =3D rdma_registration_stop(f, RAM_CONTROL_FINISH); - if (ret < 0) { - qemu_file_set_error(f, ret); - return ret; - } } =20 ret =3D multifd_send_sync_main(); @@ -3493,8 +3418,7 @@ static inline void *colo_cache_from_block_offset(RAMB= lock *block, /** * ram_handle_zero: handle the zero page case * - * If a page (or a whole RDMA chunk) has been - * determined to be zero, then zap it. + * If a page has been determined to be zero, then zap it. * * @host: host address for the zero page * @ch: what the page is filled from. We only support zero @@ -4071,10 +3995,6 @@ static int parse_ramblock(QEMUFile *f, RAMBlock *blo= ck, ram_addr_t length) return -EINVAL; } } - ret =3D rdma_block_notification_handle(f, block->idstr); - if (ret < 0) { - qemu_file_set_error(f, ret); - } =20 return ret; } @@ -4124,7 +4044,7 @@ static int ram_load_precopy(QEMUFile *f) int flags =3D 0, ret =3D 0, invalid_flags =3D 0, i =3D 0; =20 if (migrate_mapped_ram()) { - invalid_flags |=3D (RAM_SAVE_FLAG_HOOK | RAM_SAVE_FLAG_MULTIFD_FLU= SH | + invalid_flags |=3D (RAM_SAVE_FLAG_MULTIFD_FLUSH | RAM_SAVE_FLAG_PAGE | RAM_SAVE_FLAG_XBZRLE | RAM_SAVE_FLAG_ZERO); } @@ -4255,12 +4175,6 @@ static int ram_load_precopy(QEMUFile *f) multifd_recv_sync_main(); } break; - case RAM_SAVE_FLAG_HOOK: - ret =3D rdma_registration_handle(f); - if (ret < 0) { - qemu_file_set_error(f, ret); - } - break; default: error_report("Unknown combination of migration flags: 0x%x", f= lags); ret =3D -EINVAL; diff --git a/migration/rdma.c b/migration/rdma.c deleted file mode 100644 index 855753c671..0000000000 --- a/migration/rdma.c +++ /dev/null @@ -1,4184 +0,0 @@ -/* - * RDMA protocol and interfaces - * - * Copyright IBM, Corp. 2010-2013 - * Copyright Red Hat, Inc. 2015-2016 - * - * Authors: - * Michael R. Hines - * Jiuxing Liu - * Daniel P. Berrange - * - * This work is licensed under the terms of the GNU GPL, version 2 or - * later. See the COPYING file in the top-level directory. - * - */ - -#include "qemu/osdep.h" -#include "qapi/error.h" -#include "qemu/cutils.h" -#include "exec/target_page.h" -#include "rdma.h" -#include "migration.h" -#include "migration-stats.h" -#include "qemu-file.h" -#include "ram.h" -#include "qemu/error-report.h" -#include "qemu/main-loop.h" -#include "qemu/module.h" -#include "qemu/rcu.h" -#include "qemu/sockets.h" -#include "qemu/bitmap.h" -#include "qemu/coroutine.h" -#include "exec/memory.h" -#include -#include -#include -#include -#include "trace.h" -#include "qom/object.h" -#include "options.h" -#include - -#define RDMA_RESOLVE_TIMEOUT_MS 10000 - -/* Do not merge data if larger than this. */ -#define RDMA_MERGE_MAX (2 * 1024 * 1024) -#define RDMA_SIGNALED_SEND_MAX (RDMA_MERGE_MAX / 4096) - -#define RDMA_REG_CHUNK_SHIFT 20 /* 1 MB */ - -/* - * This is only for non-live state being migrated. - * Instead of RDMA_WRITE messages, we use RDMA_SEND - * messages for that state, which requires a different - * delivery design than main memory. - */ -#define RDMA_SEND_INCREMENT 32768 - -/* - * Maximum size infiniband SEND message - */ -#define RDMA_CONTROL_MAX_BUFFER (512 * 1024) -#define RDMA_CONTROL_MAX_COMMANDS_PER_MESSAGE 4096 - -#define RDMA_CONTROL_VERSION_CURRENT 1 -/* - * Capabilities for negotiation. - */ -#define RDMA_CAPABILITY_PIN_ALL 0x01 - -/* - * Add the other flags above to this list of known capabilities - * as they are introduced. - */ -static uint32_t known_capabilities =3D RDMA_CAPABILITY_PIN_ALL; - -/* - * A work request ID is 64-bits and we split up these bits - * into 3 parts: - * - * bits 0-15 : type of control message, 2^16 - * bits 16-29: ram block index, 2^14 - * bits 30-63: ram block chunk number, 2^34 - * - * The last two bit ranges are only used for RDMA writes, - * in order to track their completion and potentially - * also track unregistration status of the message. - */ -#define RDMA_WRID_TYPE_SHIFT 0UL -#define RDMA_WRID_BLOCK_SHIFT 16UL -#define RDMA_WRID_CHUNK_SHIFT 30UL - -#define RDMA_WRID_TYPE_MASK \ - ((1UL << RDMA_WRID_BLOCK_SHIFT) - 1UL) - -#define RDMA_WRID_BLOCK_MASK \ - (~RDMA_WRID_TYPE_MASK & ((1UL << RDMA_WRID_CHUNK_SHIFT) - 1UL)) - -#define RDMA_WRID_CHUNK_MASK (~RDMA_WRID_BLOCK_MASK & ~RDMA_WRID_TYPE_MASK) - -/* - * RDMA migration protocol: - * 1. RDMA Writes (data messages, i.e. RAM) - * 2. IB Send/Recv (control channel messages) - */ -enum { - RDMA_WRID_NONE =3D 0, - RDMA_WRID_RDMA_WRITE =3D 1, - RDMA_WRID_SEND_CONTROL =3D 2000, - RDMA_WRID_RECV_CONTROL =3D 4000, -}; - -/* - * Work request IDs for IB SEND messages only (not RDMA writes). - * This is used by the migration protocol to transmit - * control messages (such as device state and registration commands) - * - * We could use more WRs, but we have enough for now. - */ -enum { - RDMA_WRID_READY =3D 0, - RDMA_WRID_DATA, - RDMA_WRID_CONTROL, - RDMA_WRID_MAX, -}; - -/* - * SEND/RECV IB Control Messages. - */ -enum { - RDMA_CONTROL_NONE =3D 0, - RDMA_CONTROL_ERROR, - RDMA_CONTROL_READY, /* ready to receive */ - RDMA_CONTROL_QEMU_FILE, /* QEMUFile-transmitted bytes */ - RDMA_CONTROL_RAM_BLOCKS_REQUEST, /* RAMBlock synchronization */ - RDMA_CONTROL_RAM_BLOCKS_RESULT, /* RAMBlock synchronization */ - RDMA_CONTROL_COMPRESS, /* page contains repeat values */ - RDMA_CONTROL_REGISTER_REQUEST, /* dynamic page registration */ - RDMA_CONTROL_REGISTER_RESULT, /* key to use after registration */ - RDMA_CONTROL_REGISTER_FINISHED, /* current iteration finished */ - RDMA_CONTROL_UNREGISTER_REQUEST, /* dynamic UN-registration */ - RDMA_CONTROL_UNREGISTER_FINISHED, /* unpinning finished */ -}; - - -/* - * Memory and MR structures used to represent an IB Send/Recv work request. - * This is *not* used for RDMA writes, only IB Send/Recv. - */ -typedef struct { - uint8_t control[RDMA_CONTROL_MAX_BUFFER]; /* actual buffer to registe= r */ - struct ibv_mr *control_mr; /* registration metadata */ - size_t control_len; /* length of the message */ - uint8_t *control_curr; /* start of unconsumed byte= s */ -} RDMAWorkRequestData; - -/* - * Negotiate RDMA capabilities during connection-setup time. - */ -typedef struct { - uint32_t version; - uint32_t flags; -} RDMACapabilities; - -static void caps_to_network(RDMACapabilities *cap) -{ - cap->version =3D htonl(cap->version); - cap->flags =3D htonl(cap->flags); -} - -static void network_to_caps(RDMACapabilities *cap) -{ - cap->version =3D ntohl(cap->version); - cap->flags =3D ntohl(cap->flags); -} - -/* - * Representation of a RAMBlock from an RDMA perspective. - * This is not transmitted, only local. - * This and subsequent structures cannot be linked lists - * because we're using a single IB message to transmit - * the information. It's small anyway, so a list is overkill. - */ -typedef struct RDMALocalBlock { - char *block_name; - uint8_t *local_host_addr; /* local virtual address */ - uint64_t remote_host_addr; /* remote virtual address */ - uint64_t offset; - uint64_t length; - struct ibv_mr **pmr; /* MRs for chunk-level registration */ - struct ibv_mr *mr; /* MR for non-chunk-level registration= */ - uint32_t *remote_keys; /* rkeys for chunk-level registration = */ - uint32_t remote_rkey; /* rkeys for non-chunk-level registrat= ion */ - int index; /* which block are we */ - unsigned int src_index; /* (Only used on dest) */ - bool is_ram_block; - int nb_chunks; - unsigned long *transit_bitmap; - unsigned long *unregister_bitmap; -} RDMALocalBlock; - -/* - * Also represents a RAMblock, but only on the dest. - * This gets transmitted by the dest during connection-time - * to the source VM and then is used to populate the - * corresponding RDMALocalBlock with - * the information needed to perform the actual RDMA. - */ -typedef struct QEMU_PACKED RDMADestBlock { - uint64_t remote_host_addr; - uint64_t offset; - uint64_t length; - uint32_t remote_rkey; - uint32_t padding; -} RDMADestBlock; - -static const char *control_desc(unsigned int rdma_control) -{ - static const char *strs[] =3D { - [RDMA_CONTROL_NONE] =3D "NONE", - [RDMA_CONTROL_ERROR] =3D "ERROR", - [RDMA_CONTROL_READY] =3D "READY", - [RDMA_CONTROL_QEMU_FILE] =3D "QEMU FILE", - [RDMA_CONTROL_RAM_BLOCKS_REQUEST] =3D "RAM BLOCKS REQUEST", - [RDMA_CONTROL_RAM_BLOCKS_RESULT] =3D "RAM BLOCKS RESULT", - [RDMA_CONTROL_COMPRESS] =3D "COMPRESS", - [RDMA_CONTROL_REGISTER_REQUEST] =3D "REGISTER REQUEST", - [RDMA_CONTROL_REGISTER_RESULT] =3D "REGISTER RESULT", - [RDMA_CONTROL_REGISTER_FINISHED] =3D "REGISTER FINISHED", - [RDMA_CONTROL_UNREGISTER_REQUEST] =3D "UNREGISTER REQUEST", - [RDMA_CONTROL_UNREGISTER_FINISHED] =3D "UNREGISTER FINISHED", - }; - - if (rdma_control > RDMA_CONTROL_UNREGISTER_FINISHED) { - return "??BAD CONTROL VALUE??"; - } - - return strs[rdma_control]; -} - -#if !defined(htonll) -static uint64_t htonll(uint64_t v) -{ - union { uint32_t lv[2]; uint64_t llv; } u; - u.lv[0] =3D htonl(v >> 32); - u.lv[1] =3D htonl(v & 0xFFFFFFFFULL); - return u.llv; -} -#endif - -#if !defined(ntohll) -static uint64_t ntohll(uint64_t v) -{ - union { uint32_t lv[2]; uint64_t llv; } u; - u.llv =3D v; - return ((uint64_t)ntohl(u.lv[0]) << 32) | (uint64_t) ntohl(u.lv[1]); -} -#endif - -static void dest_block_to_network(RDMADestBlock *db) -{ - db->remote_host_addr =3D htonll(db->remote_host_addr); - db->offset =3D htonll(db->offset); - db->length =3D htonll(db->length); - db->remote_rkey =3D htonl(db->remote_rkey); -} - -static void network_to_dest_block(RDMADestBlock *db) -{ - db->remote_host_addr =3D ntohll(db->remote_host_addr); - db->offset =3D ntohll(db->offset); - db->length =3D ntohll(db->length); - db->remote_rkey =3D ntohl(db->remote_rkey); -} - -/* - * Virtual address of the above structures used for transmitting - * the RAMBlock descriptions at connection-time. - * This structure is *not* transmitted. - */ -typedef struct RDMALocalBlocks { - int nb_blocks; - bool init; /* main memory init complete */ - RDMALocalBlock *block; -} RDMALocalBlocks; - -/* - * Main data structure for RDMA state. - * While there is only one copy of this structure being allocated right no= w, - * this is the place where one would start if you wanted to consider - * having more than one RDMA connection open at the same time. - */ -typedef struct RDMAContext { - char *host; - int port; - - RDMAWorkRequestData wr_data[RDMA_WRID_MAX]; - - /* - * This is used by *_exchange_send() to figure out whether or not - * the initial "READY" message has already been received or not. - * This is because other functions may potentially poll() and detect - * the READY message before send() does, in which case we need to - * know if it completed. - */ - int control_ready_expected; - - /* number of outstanding writes */ - int nb_sent; - - /* store info about current buffer so that we can - merge it with future sends */ - uint64_t current_addr; - uint64_t current_length; - /* index of ram block the current buffer belongs to */ - int current_index; - /* index of the chunk in the current ram block */ - int current_chunk; - - bool pin_all; - - /* - * infiniband-specific variables for opening the device - * and maintaining connection state and so forth. - * - * cm_id also has ibv_context, rdma_event_channel, and ibv_qp in - * cm_id->verbs, cm_id->channel, and cm_id->qp. - */ - struct rdma_cm_id *cm_id; /* connection manager ID */ - struct rdma_cm_id *listen_id; - bool connected; - - struct ibv_context *verbs; - struct rdma_event_channel *channel; - struct ibv_qp *qp; /* queue pair */ - struct ibv_comp_channel *recv_comp_channel; /* recv completion channe= l */ - struct ibv_comp_channel *send_comp_channel; /* send completion channe= l */ - struct ibv_pd *pd; /* protection domain */ - struct ibv_cq *recv_cq; /* recvieve completion queue */ - struct ibv_cq *send_cq; /* send completion queue */ - - /* - * If a previous write failed (perhaps because of a failed - * memory registration, then do not attempt any future work - * and remember the error state. - */ - bool errored; - bool error_reported; - bool received_error; - - /* - * Description of ram blocks used throughout the code. - */ - RDMALocalBlocks local_ram_blocks; - RDMADestBlock *dest_blocks; - - /* Index of the next RAMBlock received during block registration */ - unsigned int next_src_index; - - /* - * Migration on *destination* started. - * Then use coroutine yield function. - * Source runs in a thread, so we don't care. - */ - int migration_started_on_destination; - - int total_registrations; - int total_writes; - - int unregister_current, unregister_next; - uint64_t unregistrations[RDMA_SIGNALED_SEND_MAX]; - - GHashTable *blockmap; - - /* the RDMAContext for return path */ - struct RDMAContext *return_path; - bool is_return_path; -} RDMAContext; - -#define TYPE_QIO_CHANNEL_RDMA "qio-channel-rdma" -OBJECT_DECLARE_SIMPLE_TYPE(QIOChannelRDMA, QIO_CHANNEL_RDMA) - - - -struct QIOChannelRDMA { - QIOChannel parent; - RDMAContext *rdmain; - RDMAContext *rdmaout; - QEMUFile *file; - bool blocking; /* XXX we don't actually honour this yet */ -}; - -/* - * Main structure for IB Send/Recv control messages. - * This gets prepended at the beginning of every Send/Recv. - */ -typedef struct QEMU_PACKED { - uint32_t len; /* Total length of data portion */ - uint32_t type; /* which control command to perform */ - uint32_t repeat; /* number of commands in data portion of same type */ - uint32_t padding; -} RDMAControlHeader; - -static void control_to_network(RDMAControlHeader *control) -{ - control->type =3D htonl(control->type); - control->len =3D htonl(control->len); - control->repeat =3D htonl(control->repeat); -} - -static void network_to_control(RDMAControlHeader *control) -{ - control->type =3D ntohl(control->type); - control->len =3D ntohl(control->len); - control->repeat =3D ntohl(control->repeat); -} - -/* - * Register a single Chunk. - * Information sent by the source VM to inform the dest - * to register an single chunk of memory before we can perform - * the actual RDMA operation. - */ -typedef struct QEMU_PACKED { - union QEMU_PACKED { - uint64_t current_addr; /* offset into the ram_addr_t space */ - uint64_t chunk; /* chunk to lookup if unregistering */ - } key; - uint32_t current_index; /* which ramblock the chunk belongs to */ - uint32_t padding; - uint64_t chunks; /* how many sequential chunks to register = */ -} RDMARegister; - -static bool rdma_errored(RDMAContext *rdma) -{ - if (rdma->errored && !rdma->error_reported) { - error_report("RDMA is in an error state waiting migration" - " to abort!"); - rdma->error_reported =3D true; - } - return rdma->errored; -} - -static void register_to_network(RDMAContext *rdma, RDMARegister *reg) -{ - RDMALocalBlock *local_block; - local_block =3D &rdma->local_ram_blocks.block[reg->current_index]; - - if (local_block->is_ram_block) { - /* - * current_addr as passed in is an address in the local ram_addr_t - * space, we need to translate this for the destination - */ - reg->key.current_addr -=3D local_block->offset; - reg->key.current_addr +=3D rdma->dest_blocks[reg->current_index].o= ffset; - } - reg->key.current_addr =3D htonll(reg->key.current_addr); - reg->current_index =3D htonl(reg->current_index); - reg->chunks =3D htonll(reg->chunks); -} - -static void network_to_register(RDMARegister *reg) -{ - reg->key.current_addr =3D ntohll(reg->key.current_addr); - reg->current_index =3D ntohl(reg->current_index); - reg->chunks =3D ntohll(reg->chunks); -} - -typedef struct QEMU_PACKED { - uint32_t value; /* if zero, we will madvise() */ - uint32_t block_idx; /* which ram block index */ - uint64_t offset; /* Address in remote ram_addr_t space */ - uint64_t length; /* length of the chunk */ -} RDMACompress; - -static void compress_to_network(RDMAContext *rdma, RDMACompress *comp) -{ - comp->value =3D htonl(comp->value); - /* - * comp->offset as passed in is an address in the local ram_addr_t - * space, we need to translate this for the destination - */ - comp->offset -=3D rdma->local_ram_blocks.block[comp->block_idx].offset; - comp->offset +=3D rdma->dest_blocks[comp->block_idx].offset; - comp->block_idx =3D htonl(comp->block_idx); - comp->offset =3D htonll(comp->offset); - comp->length =3D htonll(comp->length); -} - -static void network_to_compress(RDMACompress *comp) -{ - comp->value =3D ntohl(comp->value); - comp->block_idx =3D ntohl(comp->block_idx); - comp->offset =3D ntohll(comp->offset); - comp->length =3D ntohll(comp->length); -} - -/* - * The result of the dest's memory registration produces an "rkey" - * which the source VM must reference in order to perform - * the RDMA operation. - */ -typedef struct QEMU_PACKED { - uint32_t rkey; - uint32_t padding; - uint64_t host_addr; -} RDMARegisterResult; - -static void result_to_network(RDMARegisterResult *result) -{ - result->rkey =3D htonl(result->rkey); - result->host_addr =3D htonll(result->host_addr); -}; - -static void network_to_result(RDMARegisterResult *result) -{ - result->rkey =3D ntohl(result->rkey); - result->host_addr =3D ntohll(result->host_addr); -}; - -static int qemu_rdma_exchange_send(RDMAContext *rdma, RDMAControlHeader *h= ead, - uint8_t *data, RDMAControlHeader *resp, - int *resp_idx, - int (*callback)(RDMAContext *rdma, - Error **errp), - Error **errp); - -static inline uint64_t ram_chunk_index(const uint8_t *start, - const uint8_t *host) -{ - return ((uintptr_t) host - (uintptr_t) start) >> RDMA_REG_CHUNK_SHIFT; -} - -static inline uint8_t *ram_chunk_start(const RDMALocalBlock *rdma_ram_bloc= k, - uint64_t i) -{ - return (uint8_t *)(uintptr_t)(rdma_ram_block->local_host_addr + - (i << RDMA_REG_CHUNK_SHIFT)); -} - -static inline uint8_t *ram_chunk_end(const RDMALocalBlock *rdma_ram_block, - uint64_t i) -{ - uint8_t *result =3D ram_chunk_start(rdma_ram_block, i) + - (1UL << RDMA_REG_CHUNK_SHIFT); - - if (result > (rdma_ram_block->local_host_addr + rdma_ram_block->length= )) { - result =3D rdma_ram_block->local_host_addr + rdma_ram_block->lengt= h; - } - - return result; -} - -static void rdma_add_block(RDMAContext *rdma, const char *block_name, - void *host_addr, - ram_addr_t block_offset, uint64_t length) -{ - RDMALocalBlocks *local =3D &rdma->local_ram_blocks; - RDMALocalBlock *block; - RDMALocalBlock *old =3D local->block; - - local->block =3D g_new0(RDMALocalBlock, local->nb_blocks + 1); - - if (local->nb_blocks) { - if (rdma->blockmap) { - for (int x =3D 0; x < local->nb_blocks; x++) { - g_hash_table_remove(rdma->blockmap, - (void *)(uintptr_t)old[x].offset); - g_hash_table_insert(rdma->blockmap, - (void *)(uintptr_t)old[x].offset, - &local->block[x]); - } - } - memcpy(local->block, old, sizeof(RDMALocalBlock) * local->nb_block= s); - g_free(old); - } - - block =3D &local->block[local->nb_blocks]; - - block->block_name =3D g_strdup(block_name); - block->local_host_addr =3D host_addr; - block->offset =3D block_offset; - block->length =3D length; - block->index =3D local->nb_blocks; - block->src_index =3D ~0U; /* Filled in by the receipt of the block lis= t */ - block->nb_chunks =3D ram_chunk_index(host_addr, host_addr + length) + = 1UL; - block->transit_bitmap =3D bitmap_new(block->nb_chunks); - bitmap_clear(block->transit_bitmap, 0, block->nb_chunks); - block->unregister_bitmap =3D bitmap_new(block->nb_chunks); - bitmap_clear(block->unregister_bitmap, 0, block->nb_chunks); - block->remote_keys =3D g_new0(uint32_t, block->nb_chunks); - - block->is_ram_block =3D local->init ? false : true; - - if (rdma->blockmap) { - g_hash_table_insert(rdma->blockmap, (void *)(uintptr_t)block_offse= t, block); - } - - trace_rdma_add_block(block_name, local->nb_blocks, - (uintptr_t) block->local_host_addr, - block->offset, block->length, - (uintptr_t) (block->local_host_addr + block->leng= th), - BITS_TO_LONGS(block->nb_chunks) * - sizeof(unsigned long) * 8, - block->nb_chunks); - - local->nb_blocks++; -} - -/* - * Memory regions need to be registered with the device and queue pairs se= tup - * in advanced before the migration starts. This tells us where the RAM bl= ocks - * are so that we can register them individually. - */ -static int qemu_rdma_init_one_block(RAMBlock *rb, void *opaque) -{ - const char *block_name =3D qemu_ram_get_idstr(rb); - void *host_addr =3D qemu_ram_get_host_addr(rb); - ram_addr_t block_offset =3D qemu_ram_get_offset(rb); - ram_addr_t length =3D qemu_ram_get_used_length(rb); - rdma_add_block(opaque, block_name, host_addr, block_offset, length); - return 0; -} - -/* - * Identify the RAMBlocks and their quantity. They will be references to - * identify chunk boundaries inside each RAMBlock and also be referenced - * during dynamic page registration. - */ -static void qemu_rdma_init_ram_blocks(RDMAContext *rdma) -{ - RDMALocalBlocks *local =3D &rdma->local_ram_blocks; - int ret; - - assert(rdma->blockmap =3D=3D NULL); - memset(local, 0, sizeof *local); - ret =3D foreach_not_ignored_block(qemu_rdma_init_one_block, rdma); - assert(!ret); - trace_qemu_rdma_init_ram_blocks(local->nb_blocks); - rdma->dest_blocks =3D g_new0(RDMADestBlock, - rdma->local_ram_blocks.nb_blocks); - local->init =3D true; -} - -/* - * Note: If used outside of cleanup, the caller must ensure that the desti= nation - * block structures are also updated - */ -static void rdma_delete_block(RDMAContext *rdma, RDMALocalBlock *block) -{ - RDMALocalBlocks *local =3D &rdma->local_ram_blocks; - RDMALocalBlock *old =3D local->block; - - if (rdma->blockmap) { - g_hash_table_remove(rdma->blockmap, (void *)(uintptr_t)block->offs= et); - } - if (block->pmr) { - for (int j =3D 0; j < block->nb_chunks; j++) { - if (!block->pmr[j]) { - continue; - } - ibv_dereg_mr(block->pmr[j]); - rdma->total_registrations--; - } - g_free(block->pmr); - block->pmr =3D NULL; - } - - if (block->mr) { - ibv_dereg_mr(block->mr); - rdma->total_registrations--; - block->mr =3D NULL; - } - - g_free(block->transit_bitmap); - block->transit_bitmap =3D NULL; - - g_free(block->unregister_bitmap); - block->unregister_bitmap =3D NULL; - - g_free(block->remote_keys); - block->remote_keys =3D NULL; - - g_free(block->block_name); - block->block_name =3D NULL; - - if (rdma->blockmap) { - for (int x =3D 0; x < local->nb_blocks; x++) { - g_hash_table_remove(rdma->blockmap, - (void *)(uintptr_t)old[x].offset); - } - } - - if (local->nb_blocks > 1) { - - local->block =3D g_new0(RDMALocalBlock, local->nb_blocks - 1); - - if (block->index) { - memcpy(local->block, old, sizeof(RDMALocalBlock) * block->inde= x); - } - - if (block->index < (local->nb_blocks - 1)) { - memcpy(local->block + block->index, old + (block->index + 1), - sizeof(RDMALocalBlock) * - (local->nb_blocks - (block->index + 1))); - for (int x =3D block->index; x < local->nb_blocks - 1; x++) { - local->block[x].index--; - } - } - } else { - assert(block =3D=3D local->block); - local->block =3D NULL; - } - - trace_rdma_delete_block(block, (uintptr_t)block->local_host_addr, - block->offset, block->length, - (uintptr_t)(block->local_host_addr + block->le= ngth), - BITS_TO_LONGS(block->nb_chunks) * - sizeof(unsigned long) * 8, block->nb_chunks= ); - - g_free(old); - - local->nb_blocks--; - - if (local->nb_blocks && rdma->blockmap) { - for (int x =3D 0; x < local->nb_blocks; x++) { - g_hash_table_insert(rdma->blockmap, - (void *)(uintptr_t)local->block[x].offset, - &local->block[x]); - } - } -} - -/* - * Trace RDMA device open, with device details. - */ -static void qemu_rdma_dump_id(const char *who, struct ibv_context *verbs) -{ - struct ibv_port_attr port; - - if (ibv_query_port(verbs, 1, &port)) { - trace_qemu_rdma_dump_id_failed(who); - return; - } - - trace_qemu_rdma_dump_id(who, - verbs->device->name, - verbs->device->dev_name, - verbs->device->dev_path, - verbs->device->ibdev_path, - port.link_layer, - port.link_layer =3D=3D IBV_LINK_LAYER_INFINIBAND ? "Infini= band" - : port.link_layer =3D=3D IBV_LINK_LAYER_ETHERNET ? "Ethern= et" - : "Unknown"); -} - -/* - * Trace RDMA gid addressing information. - * Useful for understanding the RDMA device hierarchy in the kernel. - */ -static void qemu_rdma_dump_gid(const char *who, struct rdma_cm_id *id) -{ - char sgid[33]; - char dgid[33]; - inet_ntop(AF_INET6, &id->route.addr.addr.ibaddr.sgid, sgid, sizeof sgi= d); - inet_ntop(AF_INET6, &id->route.addr.addr.ibaddr.dgid, dgid, sizeof dgi= d); - trace_qemu_rdma_dump_gid(who, sgid, dgid); -} - -/* - * As of now, IPv6 over RoCE / iWARP is not supported by linux. - * We will try the next addrinfo struct, and fail if there are - * no other valid addresses to bind against. - * - * If user is listening on '[::]', then we will not have a opened a device - * yet and have no way of verifying if the device is RoCE or not. - * - * In this case, the source VM will throw an error for ALL types of - * connections (both IPv4 and IPv6) if the destination machine does not ha= ve - * a regular infiniband network available for use. - * - * The only way to guarantee that an error is thrown for broken kernels is - * for the management software to choose a *specific* interface at bind ti= me - * and validate what time of hardware it is. - * - * Unfortunately, this puts the user in a fix: - * - * If the source VM connects with an IPv4 address without knowing that the - * destination has bound to '[::]' the migration will unconditionally fail - * unless the management software is explicitly listening on the IPv4 - * address while using a RoCE-based device. - * - * If the source VM connects with an IPv6 address, then we're OK because = we can - * throw an error on the source (and similarly on the destination). - * - * But in mixed environments, this will be broken for a while until it is= fixed - * inside linux. - * - * We do provide a *tiny* bit of help in this function: We can list all of= the - * devices in the system and check to see if all the devices are RoCE or - * Infiniband. - * - * If we detect that we have a *pure* RoCE environment, then we can safely - * thrown an error even if the management software has specified '[::]' as= the - * bind address. - * - * However, if there is are multiple hetergeneous devices, then we cannot = make - * this assumption and the user just has to be sure they know what they are - * doing. - * - * Patches are being reviewed on linux-rdma. - */ -static int qemu_rdma_broken_ipv6_kernel(struct ibv_context *verbs, Error *= *errp) -{ - /* This bug only exists in linux, to our knowledge. */ -#ifdef CONFIG_LINUX - struct ibv_port_attr port_attr; - - /* - * Verbs are only NULL if management has bound to '[::]'. - * - * Let's iterate through all the devices and see if there any pure IB - * devices (non-ethernet). - * - * If not, then we can safely proceed with the migration. - * Otherwise, there are no guarantees until the bug is fixed in linux. - */ - if (!verbs) { - int num_devices; - struct ibv_device **dev_list =3D ibv_get_device_list(&num_devices); - bool roce_found =3D false; - bool ib_found =3D false; - - for (int x =3D 0; x < num_devices; x++) { - verbs =3D ibv_open_device(dev_list[x]); - /* - * ibv_open_device() is not documented to set errno. If - * it does, it's somebody else's doc bug. If it doesn't, - * the use of errno below is wrong. - * TODO Find out whether ibv_open_device() sets errno. - */ - if (!verbs) { - if (errno =3D=3D EPERM) { - continue; - } else { - error_setg_errno(errp, errno, - "could not open RDMA device context"); - return -1; - } - } - - if (ibv_query_port(verbs, 1, &port_attr)) { - ibv_close_device(verbs); - error_setg(errp, - "RDMA ERROR: Could not query initial IB port"); - return -1; - } - - if (port_attr.link_layer =3D=3D IBV_LINK_LAYER_INFINIBAND) { - ib_found =3D true; - } else if (port_attr.link_layer =3D=3D IBV_LINK_LAYER_ETHERNET= ) { - roce_found =3D true; - } - - ibv_close_device(verbs); - - } - - if (roce_found) { - if (ib_found) { - warn_report("migrations may fail:" - " IPv6 over RoCE / iWARP in linux" - " is broken. But since you appear to have a" - " mixed RoCE / IB environment, be sure to only" - " migrate over the IB fabric until the kernel " - " fixes the bug."); - } else { - error_setg(errp, "RDMA ERROR: " - "You only have RoCE / iWARP devices in your sys= tems" - " and your management software has specified '[= ::]'" - ", but IPv6 over RoCE / iWARP is not supported = in Linux."); - return -1; - } - } - - return 0; - } - - /* - * If we have a verbs context, that means that some other than '[::]' = was - * used by the management software for binding. In which case we can - * actually warn the user about a potentially broken kernel. - */ - - /* IB ports start with 1, not 0 */ - if (ibv_query_port(verbs, 1, &port_attr)) { - error_setg(errp, "RDMA ERROR: Could not query initial IB port"); - return -1; - } - - if (port_attr.link_layer =3D=3D IBV_LINK_LAYER_ETHERNET) { - error_setg(errp, "RDMA ERROR: " - "Linux kernel's RoCE / iWARP does not support IPv6 " - "(but patches on linux-rdma in progress)"); - return -1; - } - -#endif - - return 0; -} - -/* - * Figure out which RDMA device corresponds to the requested IP hostname - * Also create the initial connection manager identifiers for opening - * the connection. - */ -static int qemu_rdma_resolve_host(RDMAContext *rdma, Error **errp) -{ - Error *err =3D NULL; - int ret; - struct rdma_addrinfo *res; - char port_str[16]; - struct rdma_cm_event *cm_event; - char ip[40] =3D "unknown"; - - if (rdma->host =3D=3D NULL || !strcmp(rdma->host, "")) { - error_setg(errp, "RDMA ERROR: RDMA hostname has not been set"); - return -1; - } - - /* create CM channel */ - rdma->channel =3D rdma_create_event_channel(); - if (!rdma->channel) { - error_setg(errp, "RDMA ERROR: could not create CM channel"); - return -1; - } - - /* create CM id */ - ret =3D rdma_create_id(rdma->channel, &rdma->cm_id, NULL, RDMA_PS_TCP); - if (ret < 0) { - error_setg(errp, "RDMA ERROR: could not create channel id"); - goto err_resolve_create_id; - } - - snprintf(port_str, 16, "%d", rdma->port); - port_str[15] =3D '\0'; - - ret =3D rdma_getaddrinfo(rdma->host, port_str, NULL, &res); - if (ret) { - error_setg(errp, "RDMA ERROR: could not rdma_getaddrinfo address %= s", - rdma->host); - goto err_resolve_get_addr; - } - - /* Try all addresses, saving the first error in @err */ - for (struct rdma_addrinfo *e =3D res; e !=3D NULL; e =3D e->ai_next) { - Error **local_errp =3D err ? NULL : &err; - - inet_ntop(e->ai_family, - &((struct sockaddr_in *) e->ai_dst_addr)->sin_addr, ip, sizeof= ip); - trace_qemu_rdma_resolve_host_trying(rdma->host, ip); - - ret =3D rdma_resolve_addr(rdma->cm_id, NULL, e->ai_dst_addr, - RDMA_RESOLVE_TIMEOUT_MS); - if (ret >=3D 0) { - if (e->ai_family =3D=3D AF_INET6) { - ret =3D qemu_rdma_broken_ipv6_kernel(rdma->cm_id->verbs, - local_errp); - if (ret < 0) { - continue; - } - } - error_free(err); - goto route; - } - } - - rdma_freeaddrinfo(res); - if (err) { - error_propagate(errp, err); - } else { - error_setg(errp, "RDMA ERROR: could not resolve address %s", - rdma->host); - } - goto err_resolve_get_addr; - -route: - rdma_freeaddrinfo(res); - qemu_rdma_dump_gid("source_resolve_addr", rdma->cm_id); - - ret =3D rdma_get_cm_event(rdma->channel, &cm_event); - if (ret < 0) { - error_setg(errp, "RDMA ERROR: could not perform event_addr_resolve= d"); - goto err_resolve_get_addr; - } - - if (cm_event->event !=3D RDMA_CM_EVENT_ADDR_RESOLVED) { - error_setg(errp, - "RDMA ERROR: result not equal to event_addr_resolved %s= ", - rdma_event_str(cm_event->event)); - rdma_ack_cm_event(cm_event); - goto err_resolve_get_addr; - } - rdma_ack_cm_event(cm_event); - - /* resolve route */ - ret =3D rdma_resolve_route(rdma->cm_id, RDMA_RESOLVE_TIMEOUT_MS); - if (ret < 0) { - error_setg(errp, "RDMA ERROR: could not resolve rdma route"); - goto err_resolve_get_addr; - } - - ret =3D rdma_get_cm_event(rdma->channel, &cm_event); - if (ret < 0) { - error_setg(errp, "RDMA ERROR: could not perform event_route_resolv= ed"); - goto err_resolve_get_addr; - } - if (cm_event->event !=3D RDMA_CM_EVENT_ROUTE_RESOLVED) { - error_setg(errp, "RDMA ERROR: " - "result not equal to event_route_resolved: %s", - rdma_event_str(cm_event->event)); - rdma_ack_cm_event(cm_event); - goto err_resolve_get_addr; - } - rdma_ack_cm_event(cm_event); - rdma->verbs =3D rdma->cm_id->verbs; - qemu_rdma_dump_id("source_resolve_host", rdma->cm_id->verbs); - qemu_rdma_dump_gid("source_resolve_host", rdma->cm_id); - return 0; - -err_resolve_get_addr: - rdma_destroy_id(rdma->cm_id); - rdma->cm_id =3D NULL; -err_resolve_create_id: - rdma_destroy_event_channel(rdma->channel); - rdma->channel =3D NULL; - return -1; -} - -/* - * Create protection domain and completion queues - */ -static int qemu_rdma_alloc_pd_cq(RDMAContext *rdma, Error **errp) -{ - /* allocate pd */ - rdma->pd =3D ibv_alloc_pd(rdma->verbs); - if (!rdma->pd) { - error_setg(errp, "failed to allocate protection domain"); - return -1; - } - - /* create receive completion channel */ - rdma->recv_comp_channel =3D ibv_create_comp_channel(rdma->verbs); - if (!rdma->recv_comp_channel) { - error_setg(errp, "failed to allocate receive completion channel"); - goto err_alloc_pd_cq; - } - - /* - * Completion queue can be filled by read work requests. - */ - rdma->recv_cq =3D ibv_create_cq(rdma->verbs, (RDMA_SIGNALED_SEND_MAX *= 3), - NULL, rdma->recv_comp_channel, 0); - if (!rdma->recv_cq) { - error_setg(errp, "failed to allocate receive completion queue"); - goto err_alloc_pd_cq; - } - - /* create send completion channel */ - rdma->send_comp_channel =3D ibv_create_comp_channel(rdma->verbs); - if (!rdma->send_comp_channel) { - error_setg(errp, "failed to allocate send completion channel"); - goto err_alloc_pd_cq; - } - - rdma->send_cq =3D ibv_create_cq(rdma->verbs, (RDMA_SIGNALED_SEND_MAX *= 3), - NULL, rdma->send_comp_channel, 0); - if (!rdma->send_cq) { - error_setg(errp, "failed to allocate send completion queue"); - goto err_alloc_pd_cq; - } - - return 0; - -err_alloc_pd_cq: - if (rdma->pd) { - ibv_dealloc_pd(rdma->pd); - } - if (rdma->recv_comp_channel) { - ibv_destroy_comp_channel(rdma->recv_comp_channel); - } - if (rdma->send_comp_channel) { - ibv_destroy_comp_channel(rdma->send_comp_channel); - } - if (rdma->recv_cq) { - ibv_destroy_cq(rdma->recv_cq); - rdma->recv_cq =3D NULL; - } - rdma->pd =3D NULL; - rdma->recv_comp_channel =3D NULL; - rdma->send_comp_channel =3D NULL; - return -1; - -} - -/* - * Create queue pairs. - */ -static int qemu_rdma_alloc_qp(RDMAContext *rdma) -{ - struct ibv_qp_init_attr attr =3D { 0 }; - - attr.cap.max_send_wr =3D RDMA_SIGNALED_SEND_MAX; - attr.cap.max_recv_wr =3D 3; - attr.cap.max_send_sge =3D 1; - attr.cap.max_recv_sge =3D 1; - attr.send_cq =3D rdma->send_cq; - attr.recv_cq =3D rdma->recv_cq; - attr.qp_type =3D IBV_QPT_RC; - - if (rdma_create_qp(rdma->cm_id, rdma->pd, &attr) < 0) { - return -1; - } - - rdma->qp =3D rdma->cm_id->qp; - return 0; -} - -/* Check whether On-Demand Paging is supported by RDAM device */ -static bool rdma_support_odp(struct ibv_context *dev) -{ - struct ibv_device_attr_ex attr =3D {0}; - - if (ibv_query_device_ex(dev, NULL, &attr)) { - return false; - } - - if (attr.odp_caps.general_caps & IBV_ODP_SUPPORT) { - return true; - } - - return false; -} - -/* - * ibv_advise_mr to avoid RNR NAK error as far as possible. - * The responder mr registering with ODP will sent RNR NAK back to - * the requester in the face of the page fault. - */ -static void qemu_rdma_advise_prefetch_mr(struct ibv_pd *pd, uint64_t addr, - uint32_t len, uint32_t lkey, - const char *name, bool wr) -{ -#ifdef HAVE_IBV_ADVISE_MR - int ret; - int advice =3D wr ? IBV_ADVISE_MR_ADVICE_PREFETCH_WRITE : - IBV_ADVISE_MR_ADVICE_PREFETCH; - struct ibv_sge sg_list =3D {.lkey =3D lkey, .addr =3D addr, .length = =3D len}; - - ret =3D ibv_advise_mr(pd, advice, - IBV_ADVISE_MR_FLAG_FLUSH, &sg_list, 1); - /* ignore the error */ - trace_qemu_rdma_advise_mr(name, len, addr, strerror(ret)); -#endif -} - -static int qemu_rdma_reg_whole_ram_blocks(RDMAContext *rdma, Error **errp) -{ - int i; - RDMALocalBlocks *local =3D &rdma->local_ram_blocks; - - for (i =3D 0; i < local->nb_blocks; i++) { - int access =3D IBV_ACCESS_LOCAL_WRITE | IBV_ACCESS_REMOTE_WRITE; - - local->block[i].mr =3D - ibv_reg_mr(rdma->pd, - local->block[i].local_host_addr, - local->block[i].length, access - ); - /* - * ibv_reg_mr() is not documented to set errno. If it does, - * it's somebody else's doc bug. If it doesn't, the use of - * errno below is wrong. - * TODO Find out whether ibv_reg_mr() sets errno. - */ - if (!local->block[i].mr && - errno =3D=3D ENOTSUP && rdma_support_odp(rdma->verbs)) { - access |=3D IBV_ACCESS_ON_DEMAND; - /* register ODP mr */ - local->block[i].mr =3D - ibv_reg_mr(rdma->pd, - local->block[i].local_host_addr, - local->block[i].length, access); - trace_qemu_rdma_register_odp_mr(local->block[i].block_name= ); - - if (local->block[i].mr) { - qemu_rdma_advise_prefetch_mr(rdma->pd, - (uintptr_t)local->block[i].local_host_= addr, - local->block[i].length, - local->block[i].mr->lkey, - local->block[i].block_name, - true); - } - } - - if (!local->block[i].mr) { - error_setg_errno(errp, errno, - "Failed to register local dest ram block!"); - goto err; - } - rdma->total_registrations++; - } - - return 0; - -err: - for (i--; i >=3D 0; i--) { - ibv_dereg_mr(local->block[i].mr); - local->block[i].mr =3D NULL; - rdma->total_registrations--; - } - - return -1; - -} - -/* - * Find the ram block that corresponds to the page requested to be - * transmitted by QEMU. - * - * Once the block is found, also identify which 'chunk' within that - * block that the page belongs to. - */ -static void qemu_rdma_search_ram_block(RDMAContext *rdma, - uintptr_t block_offset, - uint64_t offset, - uint64_t length, - uint64_t *block_index, - uint64_t *chunk_index) -{ - uint64_t current_addr =3D block_offset + offset; - RDMALocalBlock *block =3D g_hash_table_lookup(rdma->blockmap, - (void *) block_offset); - assert(block); - assert(current_addr >=3D block->offset); - assert((current_addr + length) <=3D (block->offset + block->length)); - - *block_index =3D block->index; - *chunk_index =3D ram_chunk_index(block->local_host_addr, - block->local_host_addr + (current_addr - block->offset)); -} - -/* - * Register a chunk with IB. If the chunk was already registered - * previously, then skip. - * - * Also return the keys associated with the registration needed - * to perform the actual RDMA operation. - */ -static int qemu_rdma_register_and_get_keys(RDMAContext *rdma, - RDMALocalBlock *block, uintptr_t host_addr, - uint32_t *lkey, uint32_t *rkey, int chunk, - uint8_t *chunk_start, uint8_t *chunk_end) -{ - if (block->mr) { - if (lkey) { - *lkey =3D block->mr->lkey; - } - if (rkey) { - *rkey =3D block->mr->rkey; - } - return 0; - } - - /* allocate memory to store chunk MRs */ - if (!block->pmr) { - block->pmr =3D g_new0(struct ibv_mr *, block->nb_chunks); - } - - /* - * If 'rkey', then we're the destination, so grant access to the sourc= e. - * - * If 'lkey', then we're the source VM, so grant access only to oursel= ves. - */ - if (!block->pmr[chunk]) { - uint64_t len =3D chunk_end - chunk_start; - int access =3D rkey ? IBV_ACCESS_LOCAL_WRITE | IBV_ACCESS_REMOTE_W= RITE : - 0; - - trace_qemu_rdma_register_and_get_keys(len, chunk_start); - - block->pmr[chunk] =3D ibv_reg_mr(rdma->pd, chunk_start, len, acces= s); - /* - * ibv_reg_mr() is not documented to set errno. If it does, - * it's somebody else's doc bug. If it doesn't, the use of - * errno below is wrong. - * TODO Find out whether ibv_reg_mr() sets errno. - */ - if (!block->pmr[chunk] && - errno =3D=3D ENOTSUP && rdma_support_odp(rdma->verbs)) { - access |=3D IBV_ACCESS_ON_DEMAND; - /* register ODP mr */ - block->pmr[chunk] =3D ibv_reg_mr(rdma->pd, chunk_start, len, a= ccess); - trace_qemu_rdma_register_odp_mr(block->block_name); - - if (block->pmr[chunk]) { - qemu_rdma_advise_prefetch_mr(rdma->pd, (uintptr_t)chunk_st= art, - len, block->pmr[chunk]->lkey, - block->block_name, rkey); - - } - } - } - if (!block->pmr[chunk]) { - return -1; - } - rdma->total_registrations++; - - if (lkey) { - *lkey =3D block->pmr[chunk]->lkey; - } - if (rkey) { - *rkey =3D block->pmr[chunk]->rkey; - } - return 0; -} - -/* - * Register (at connection time) the memory used for control - * channel messages. - */ -static int qemu_rdma_reg_control(RDMAContext *rdma, int idx) -{ - rdma->wr_data[idx].control_mr =3D ibv_reg_mr(rdma->pd, - rdma->wr_data[idx].control, RDMA_CONTROL_MAX_BUFFER, - IBV_ACCESS_LOCAL_WRITE | IBV_ACCESS_REMOTE_WRITE); - if (rdma->wr_data[idx].control_mr) { - rdma->total_registrations++; - return 0; - } - return -1; -} - -/* - * Perform a non-optimized memory unregistration after every transfer - * for demonstration purposes, only if pin-all is not requested. - * - * Potential optimizations: - * 1. Start a new thread to run this function continuously - - for bit clearing - - and for receipt of unregister messages - * 2. Use an LRU. - * 3. Use workload hints. - */ -static int qemu_rdma_unregister_waiting(RDMAContext *rdma) -{ - Error *err =3D NULL; - - while (rdma->unregistrations[rdma->unregister_current]) { - int ret; - uint64_t wr_id =3D rdma->unregistrations[rdma->unregister_current]; - uint64_t chunk =3D - (wr_id & RDMA_WRID_CHUNK_MASK) >> RDMA_WRID_CHUNK_SHIFT; - uint64_t index =3D - (wr_id & RDMA_WRID_BLOCK_MASK) >> RDMA_WRID_BLOCK_SHIFT; - RDMALocalBlock *block =3D - &(rdma->local_ram_blocks.block[index]); - RDMARegister reg =3D { .current_index =3D index }; - RDMAControlHeader resp =3D { .type =3D RDMA_CONTROL_UNREGISTER_FIN= ISHED, - }; - RDMAControlHeader head =3D { .len =3D sizeof(RDMARegister), - .type =3D RDMA_CONTROL_UNREGISTER_REQUE= ST, - .repeat =3D 1, - }; - - trace_qemu_rdma_unregister_waiting_proc(chunk, - rdma->unregister_current); - - rdma->unregistrations[rdma->unregister_current] =3D 0; - rdma->unregister_current++; - - if (rdma->unregister_current =3D=3D RDMA_SIGNALED_SEND_MAX) { - rdma->unregister_current =3D 0; - } - - - /* - * Unregistration is speculative (because migration is single-thre= aded - * and we cannot break the protocol's inifinband message ordering). - * Thus, if the memory is currently being used for transmission, - * then abort the attempt to unregister and try again - * later the next time a completion is received for this memory. - */ - clear_bit(chunk, block->unregister_bitmap); - - if (test_bit(chunk, block->transit_bitmap)) { - trace_qemu_rdma_unregister_waiting_inflight(chunk); - continue; - } - - trace_qemu_rdma_unregister_waiting_send(chunk); - - ret =3D ibv_dereg_mr(block->pmr[chunk]); - block->pmr[chunk] =3D NULL; - block->remote_keys[chunk] =3D 0; - - if (ret !=3D 0) { - error_report("unregistration chunk failed: %s", - strerror(ret)); - return -1; - } - rdma->total_registrations--; - - reg.key.chunk =3D chunk; - register_to_network(rdma, ®); - ret =3D qemu_rdma_exchange_send(rdma, &head, (uint8_t *) ®, - &resp, NULL, NULL, &err); - if (ret < 0) { - error_report_err(err); - return -1; - } - - trace_qemu_rdma_unregister_waiting_complete(chunk); - } - - return 0; -} - -static uint64_t qemu_rdma_make_wrid(uint64_t wr_id, uint64_t index, - uint64_t chunk) -{ - uint64_t result =3D wr_id & RDMA_WRID_TYPE_MASK; - - result |=3D (index << RDMA_WRID_BLOCK_SHIFT); - result |=3D (chunk << RDMA_WRID_CHUNK_SHIFT); - - return result; -} - -/* - * Consult the connection manager to see a work request - * (of any kind) has completed. - * Return the work request ID that completed. - */ -static int qemu_rdma_poll(RDMAContext *rdma, struct ibv_cq *cq, - uint64_t *wr_id_out, uint32_t *byte_len) -{ - int ret; - struct ibv_wc wc; - uint64_t wr_id; - - ret =3D ibv_poll_cq(cq, 1, &wc); - - if (!ret) { - *wr_id_out =3D RDMA_WRID_NONE; - return 0; - } - - if (ret < 0) { - return -1; - } - - wr_id =3D wc.wr_id & RDMA_WRID_TYPE_MASK; - - if (wc.status !=3D IBV_WC_SUCCESS) { - return -1; - } - - if (rdma->control_ready_expected && - (wr_id >=3D RDMA_WRID_RECV_CONTROL)) { - trace_qemu_rdma_poll_recv(wr_id - RDMA_WRID_RECV_CONTROL, wr_id, - rdma->nb_sent); - rdma->control_ready_expected =3D 0; - } - - if (wr_id =3D=3D RDMA_WRID_RDMA_WRITE) { - uint64_t chunk =3D - (wc.wr_id & RDMA_WRID_CHUNK_MASK) >> RDMA_WRID_CHUNK_SHIFT; - uint64_t index =3D - (wc.wr_id & RDMA_WRID_BLOCK_MASK) >> RDMA_WRID_BLOCK_SHIFT; - RDMALocalBlock *block =3D &(rdma->local_ram_blocks.block[index]); - - trace_qemu_rdma_poll_write(wr_id, rdma->nb_sent, - index, chunk, block->local_host_addr, - (void *)(uintptr_t)block->remote_host_a= ddr); - - clear_bit(chunk, block->transit_bitmap); - - if (rdma->nb_sent > 0) { - rdma->nb_sent--; - } - } else { - trace_qemu_rdma_poll_other(wr_id, rdma->nb_sent); - } - - *wr_id_out =3D wc.wr_id; - if (byte_len) { - *byte_len =3D wc.byte_len; - } - - return 0; -} - -/* Wait for activity on the completion channel. - * Returns 0 on success, none-0 on error. - */ -static int qemu_rdma_wait_comp_channel(RDMAContext *rdma, - struct ibv_comp_channel *comp_chann= el) -{ - struct rdma_cm_event *cm_event; - - /* - * Coroutine doesn't start until migration_fd_process_incoming() - * so don't yield unless we know we're running inside of a coroutine. - */ - if (rdma->migration_started_on_destination && - migration_incoming_get_current()->state =3D=3D MIGRATION_STATUS_AC= TIVE) { - yield_until_fd_readable(comp_channel->fd); - } else { - /* This is the source side, we're in a separate thread - * or destination prior to migration_fd_process_incoming() - * after postcopy, the destination also in a separate thread. - * we can't yield; so we have to poll the fd. - * But we need to be able to handle 'cancel' or an error - * without hanging forever. - */ - while (!rdma->errored && !rdma->received_error) { - GPollFD pfds[2]; - pfds[0].fd =3D comp_channel->fd; - pfds[0].events =3D G_IO_IN | G_IO_HUP | G_IO_ERR; - pfds[0].revents =3D 0; - - pfds[1].fd =3D rdma->channel->fd; - pfds[1].events =3D G_IO_IN | G_IO_HUP | G_IO_ERR; - pfds[1].revents =3D 0; - - /* 0.1s timeout, should be fine for a 'cancel' */ - switch (qemu_poll_ns(pfds, 2, 100 * 1000 * 1000)) { - case 2: - case 1: /* fd active */ - if (pfds[0].revents) { - return 0; - } - - if (pfds[1].revents) { - if (rdma_get_cm_event(rdma->channel, &cm_event) < 0) { - return -1; - } - - if (cm_event->event =3D=3D RDMA_CM_EVENT_DISCONNECTED = || - cm_event->event =3D=3D RDMA_CM_EVENT_DEVICE_REMOVA= L) { - rdma_ack_cm_event(cm_event); - return -1; - } - rdma_ack_cm_event(cm_event); - } - break; - - case 0: /* Timeout, go around again */ - break; - - default: /* Error of some type - - * I don't trust errno from qemu_poll_ns - */ - return -1; - } - - if (migrate_get_current()->state =3D=3D MIGRATION_STATUS_CANCE= LLING) { - /* Bail out and let the cancellation happen */ - return -1; - } - } - } - - if (rdma->received_error) { - return -1; - } - return -rdma->errored; -} - -static struct ibv_comp_channel *to_channel(RDMAContext *rdma, uint64_t wri= d) -{ - return wrid < RDMA_WRID_RECV_CONTROL ? rdma->send_comp_channel : - rdma->recv_comp_channel; -} - -static struct ibv_cq *to_cq(RDMAContext *rdma, uint64_t wrid) -{ - return wrid < RDMA_WRID_RECV_CONTROL ? rdma->send_cq : rdma->recv_cq; -} - -/* - * Block until the next work request has completed. - * - * First poll to see if a work request has already completed, - * otherwise block. - * - * If we encounter completed work requests for IDs other than - * the one we're interested in, then that's generally an error. - * - * The only exception is actual RDMA Write completions. These - * completions only need to be recorded, but do not actually - * need further processing. - */ -static int qemu_rdma_block_for_wrid(RDMAContext *rdma, - uint64_t wrid_requested, - uint32_t *byte_len) -{ - int num_cq_events =3D 0, ret; - struct ibv_cq *cq; - void *cq_ctx; - uint64_t wr_id =3D RDMA_WRID_NONE, wr_id_in; - struct ibv_comp_channel *ch =3D to_channel(rdma, wrid_requested); - struct ibv_cq *poll_cq =3D to_cq(rdma, wrid_requested); - - if (ibv_req_notify_cq(poll_cq, 0)) { - return -1; - } - /* poll cq first */ - while (wr_id !=3D wrid_requested) { - ret =3D qemu_rdma_poll(rdma, poll_cq, &wr_id_in, byte_len); - if (ret < 0) { - return -1; - } - - wr_id =3D wr_id_in & RDMA_WRID_TYPE_MASK; - - if (wr_id =3D=3D RDMA_WRID_NONE) { - break; - } - if (wr_id !=3D wrid_requested) { - trace_qemu_rdma_block_for_wrid_miss(wrid_requested, wr_id); - } - } - - if (wr_id =3D=3D wrid_requested) { - return 0; - } - - while (1) { - ret =3D qemu_rdma_wait_comp_channel(rdma, ch); - if (ret < 0) { - goto err_block_for_wrid; - } - - ret =3D ibv_get_cq_event(ch, &cq, &cq_ctx); - if (ret < 0) { - goto err_block_for_wrid; - } - - num_cq_events++; - - if (ibv_req_notify_cq(cq, 0)) { - goto err_block_for_wrid; - } - - while (wr_id !=3D wrid_requested) { - ret =3D qemu_rdma_poll(rdma, poll_cq, &wr_id_in, byte_len); - if (ret < 0) { - goto err_block_for_wrid; - } - - wr_id =3D wr_id_in & RDMA_WRID_TYPE_MASK; - - if (wr_id =3D=3D RDMA_WRID_NONE) { - break; - } - if (wr_id !=3D wrid_requested) { - trace_qemu_rdma_block_for_wrid_miss(wrid_requested, wr_id); - } - } - - if (wr_id =3D=3D wrid_requested) { - goto success_block_for_wrid; - } - } - -success_block_for_wrid: - if (num_cq_events) { - ibv_ack_cq_events(cq, num_cq_events); - } - return 0; - -err_block_for_wrid: - if (num_cq_events) { - ibv_ack_cq_events(cq, num_cq_events); - } - - rdma->errored =3D true; - return -1; -} - -/* - * Post a SEND message work request for the control channel - * containing some data and block until the post completes. - */ -static int qemu_rdma_post_send_control(RDMAContext *rdma, uint8_t *buf, - RDMAControlHeader *head, - Error **errp) -{ - int ret; - RDMAWorkRequestData *wr =3D &rdma->wr_data[RDMA_WRID_CONTROL]; - struct ibv_send_wr *bad_wr; - struct ibv_sge sge =3D { - .addr =3D (uintptr_t)(wr->control), - .length =3D head->len + sizeof(RDMAControlHeade= r), - .lkey =3D wr->control_mr->lkey, - }; - struct ibv_send_wr send_wr =3D { - .wr_id =3D RDMA_WRID_SEND_CONTROL, - .opcode =3D IBV_WR_SEND, - .send_flags =3D IBV_SEND_SIGNALED, - .sg_list =3D &sge, - .num_sge =3D 1, - }; - - trace_qemu_rdma_post_send_control(control_desc(head->type)); - - /* - * We don't actually need to do a memcpy() in here if we used - * the "sge" properly, but since we're only sending control messages - * (not RAM in a performance-critical path), then its OK for now. - * - * The copy makes the RDMAControlHeader simpler to manipulate - * for the time being. - */ - assert(head->len <=3D RDMA_CONTROL_MAX_BUFFER - sizeof(*head)); - memcpy(wr->control, head, sizeof(RDMAControlHeader)); - control_to_network((void *) wr->control); - - if (buf) { - memcpy(wr->control + sizeof(RDMAControlHeader), buf, head->len); - } - - - ret =3D ibv_post_send(rdma->qp, &send_wr, &bad_wr); - - if (ret > 0) { - error_setg(errp, "Failed to use post IB SEND for control"); - return -1; - } - - ret =3D qemu_rdma_block_for_wrid(rdma, RDMA_WRID_SEND_CONTROL, NULL); - if (ret < 0) { - error_setg(errp, "rdma migration: send polling control error"); - return -1; - } - - return 0; -} - -/* - * Post a RECV work request in anticipation of some future receipt - * of data on the control channel. - */ -static int qemu_rdma_post_recv_control(RDMAContext *rdma, int idx, - Error **errp) -{ - struct ibv_recv_wr *bad_wr; - struct ibv_sge sge =3D { - .addr =3D (uintptr_t)(rdma->wr_data[idx].contr= ol), - .length =3D RDMA_CONTROL_MAX_BUFFER, - .lkey =3D rdma->wr_data[idx].control_mr->lkey, - }; - - struct ibv_recv_wr recv_wr =3D { - .wr_id =3D RDMA_WRID_RECV_CONTROL + id= x, - .sg_list =3D &sge, - .num_sge =3D 1, - }; - - - if (ibv_post_recv(rdma->qp, &recv_wr, &bad_wr)) { - error_setg(errp, "error posting control recv"); - return -1; - } - - return 0; -} - -/* - * Block and wait for a RECV control channel message to arrive. - */ -static int qemu_rdma_exchange_get_response(RDMAContext *rdma, - RDMAControlHeader *head, uint32_t expecting, int idx, - Error **errp) -{ - uint32_t byte_len; - int ret =3D qemu_rdma_block_for_wrid(rdma, RDMA_WRID_RECV_CONTROL + id= x, - &byte_len); - - if (ret < 0) { - error_setg(errp, "rdma migration: recv polling control error!"); - return -1; - } - - network_to_control((void *) rdma->wr_data[idx].control); - memcpy(head, rdma->wr_data[idx].control, sizeof(RDMAControlHeader)); - - trace_qemu_rdma_exchange_get_response_start(control_desc(expecting)); - - if (expecting =3D=3D RDMA_CONTROL_NONE) { - trace_qemu_rdma_exchange_get_response_none(control_desc(head->type= ), - head->type); - } else if (head->type !=3D expecting || head->type =3D=3D RDMA_CONTROL= _ERROR) { - error_setg(errp, "Was expecting a %s (%d) control message" - ", but got: %s (%d), length: %d", - control_desc(expecting), expecting, - control_desc(head->type), head->type, head->len); - if (head->type =3D=3D RDMA_CONTROL_ERROR) { - rdma->received_error =3D true; - } - return -1; - } - if (head->len > RDMA_CONTROL_MAX_BUFFER - sizeof(*head)) { - error_setg(errp, "too long length: %d", head->len); - return -1; - } - if (sizeof(*head) + head->len !=3D byte_len) { - error_setg(errp, "Malformed length: %d byte_len %d", - head->len, byte_len); - return -1; - } - - return 0; -} - -/* - * When a RECV work request has completed, the work request's - * buffer is pointed at the header. - * - * This will advance the pointer to the data portion - * of the control message of the work request's buffer that - * was populated after the work request finished. - */ -static void qemu_rdma_move_header(RDMAContext *rdma, int idx, - RDMAControlHeader *head) -{ - rdma->wr_data[idx].control_len =3D head->len; - rdma->wr_data[idx].control_curr =3D - rdma->wr_data[idx].control + sizeof(RDMAControlHeader); -} - -/* - * This is an 'atomic' high-level operation to deliver a single, unified - * control-channel message. - * - * Additionally, if the user is expecting some kind of reply to this messa= ge, - * they can request a 'resp' response message be filled in by posting an - * additional work request on behalf of the user and waiting for an additi= onal - * completion. - * - * The extra (optional) response is used during registration to us from ha= ving - * to perform an *additional* exchange of message just to provide a respon= se by - * instead piggy-backing on the acknowledgement. - */ -static int qemu_rdma_exchange_send(RDMAContext *rdma, RDMAControlHeader *h= ead, - uint8_t *data, RDMAControlHeader *resp, - int *resp_idx, - int (*callback)(RDMAContext *rdma, - Error **errp), - Error **errp) -{ - int ret; - - /* - * Wait until the dest is ready before attempting to deliver the messa= ge - * by waiting for a READY message. - */ - if (rdma->control_ready_expected) { - RDMAControlHeader resp_ignored; - - ret =3D qemu_rdma_exchange_get_response(rdma, &resp_ignored, - RDMA_CONTROL_READY, - RDMA_WRID_READY, errp); - if (ret < 0) { - return -1; - } - } - - /* - * If the user is expecting a response, post a WR in anticipation of i= t. - */ - if (resp) { - ret =3D qemu_rdma_post_recv_control(rdma, RDMA_WRID_DATA, errp); - if (ret < 0) { - return -1; - } - } - - /* - * Post a WR to replace the one we just consumed for the READY message. - */ - ret =3D qemu_rdma_post_recv_control(rdma, RDMA_WRID_READY, errp); - if (ret < 0) { - return -1; - } - - /* - * Deliver the control message that was requested. - */ - ret =3D qemu_rdma_post_send_control(rdma, data, head, errp); - - if (ret < 0) { - return -1; - } - - /* - * If we're expecting a response, block and wait for it. - */ - if (resp) { - if (callback) { - trace_qemu_rdma_exchange_send_issue_callback(); - ret =3D callback(rdma, errp); - if (ret < 0) { - return -1; - } - } - - trace_qemu_rdma_exchange_send_waiting(control_desc(resp->type)); - ret =3D qemu_rdma_exchange_get_response(rdma, resp, - resp->type, RDMA_WRID_DATA, - errp); - - if (ret < 0) { - return -1; - } - - qemu_rdma_move_header(rdma, RDMA_WRID_DATA, resp); - if (resp_idx) { - *resp_idx =3D RDMA_WRID_DATA; - } - trace_qemu_rdma_exchange_send_received(control_desc(resp->type)); - } - - rdma->control_ready_expected =3D 1; - - return 0; -} - -/* - * This is an 'atomic' high-level operation to receive a single, unified - * control-channel message. - */ -static int qemu_rdma_exchange_recv(RDMAContext *rdma, RDMAControlHeader *h= ead, - uint32_t expecting, Error **errp) -{ - RDMAControlHeader ready =3D { - .len =3D 0, - .type =3D RDMA_CONTROL_READY, - .repeat =3D 1, - }; - int ret; - - /* - * Inform the source that we're ready to receive a message. - */ - ret =3D qemu_rdma_post_send_control(rdma, NULL, &ready, errp); - - if (ret < 0) { - return -1; - } - - /* - * Block and wait for the message. - */ - ret =3D qemu_rdma_exchange_get_response(rdma, head, - expecting, RDMA_WRID_READY, errp= ); - - if (ret < 0) { - return -1; - } - - qemu_rdma_move_header(rdma, RDMA_WRID_READY, head); - - /* - * Post a new RECV work request to replace the one we just consumed. - */ - ret =3D qemu_rdma_post_recv_control(rdma, RDMA_WRID_READY, errp); - if (ret < 0) { - return -1; - } - - return 0; -} - -/* - * Write an actual chunk of memory using RDMA. - * - * If we're using dynamic registration on the dest-side, we have to - * send a registration command first. - */ -static int qemu_rdma_write_one(RDMAContext *rdma, - int current_index, uint64_t current_addr, - uint64_t length, Error **errp) -{ - struct ibv_sge sge; - struct ibv_send_wr send_wr =3D { 0 }; - struct ibv_send_wr *bad_wr; - int reg_result_idx, ret, count =3D 0; - uint64_t chunk, chunks; - uint8_t *chunk_start, *chunk_end; - RDMALocalBlock *block =3D &(rdma->local_ram_blocks.block[current_index= ]); - RDMARegister reg; - RDMARegisterResult *reg_result; - RDMAControlHeader resp =3D { .type =3D RDMA_CONTROL_REGISTER_RESULT }; - RDMAControlHeader head =3D { .len =3D sizeof(RDMARegister), - .type =3D RDMA_CONTROL_REGISTER_REQUEST, - .repeat =3D 1, - }; - -retry: - sge.addr =3D (uintptr_t)(block->local_host_addr + - (current_addr - block->offset)); - sge.length =3D length; - - chunk =3D ram_chunk_index(block->local_host_addr, - (uint8_t *)(uintptr_t)sge.addr); - chunk_start =3D ram_chunk_start(block, chunk); - - if (block->is_ram_block) { - chunks =3D length / (1UL << RDMA_REG_CHUNK_SHIFT); - - if (chunks && ((length % (1UL << RDMA_REG_CHUNK_SHIFT)) =3D=3D 0))= { - chunks--; - } - } else { - chunks =3D block->length / (1UL << RDMA_REG_CHUNK_SHIFT); - - if (chunks && ((block->length % (1UL << RDMA_REG_CHUNK_SHIFT)) =3D= =3D 0)) { - chunks--; - } - } - - trace_qemu_rdma_write_one_top(chunks + 1, - (chunks + 1) * - (1UL << RDMA_REG_CHUNK_SHIFT) / 1024 / 1= 024); - - chunk_end =3D ram_chunk_end(block, chunk + chunks); - - - while (test_bit(chunk, block->transit_bitmap)) { - (void)count; - trace_qemu_rdma_write_one_block(count++, current_index, chunk, - sge.addr, length, rdma->nb_sent, block->nb_chunks); - - ret =3D qemu_rdma_block_for_wrid(rdma, RDMA_WRID_RDMA_WRITE, NULL); - - if (ret < 0) { - error_setg(errp, "Failed to Wait for previous write to complet= e " - "block %d chunk %" PRIu64 - " current %" PRIu64 " len %" PRIu64 " %d", - current_index, chunk, sge.addr, length, rdma->nb_sent); - return -1; - } - } - - if (!rdma->pin_all || !block->is_ram_block) { - if (!block->remote_keys[chunk]) { - /* - * This chunk has not yet been registered, so first check to s= ee - * if the entire chunk is zero. If so, tell the other size to - * memset() + madvise() the entire chunk without RDMA. - */ - - if (buffer_is_zero((void *)(uintptr_t)sge.addr, length)) { - RDMACompress comp =3D { - .offset =3D current_addr, - .value =3D 0, - .block_idx =3D current_index, - .length =3D length, - }; - - head.len =3D sizeof(comp); - head.type =3D RDMA_CONTROL_COMPRESS; - - trace_qemu_rdma_write_one_zero(chunk, sge.length, - current_index, current_addr= ); - - compress_to_network(rdma, &comp); - ret =3D qemu_rdma_exchange_send(rdma, &head, - (uint8_t *) &comp, NULL, NULL, NULL, errp); - - if (ret < 0) { - return -1; - } - - /* - * TODO: Here we are sending something, but we are not - * accounting for anything transferred. The following is = wrong: - * - * stat64_add(&mig_stats.rdma_bytes, sge.length); - * - * because we are using some kind of compression. I - * would think that head.len would be the more similar - * thing to a correct value. - */ - stat64_add(&mig_stats.zero_pages, - sge.length / qemu_target_page_size()); - return 1; - } - - /* - * Otherwise, tell other side to register. - */ - reg.current_index =3D current_index; - if (block->is_ram_block) { - reg.key.current_addr =3D current_addr; - } else { - reg.key.chunk =3D chunk; - } - reg.chunks =3D chunks; - - trace_qemu_rdma_write_one_sendreg(chunk, sge.length, current_i= ndex, - current_addr); - - register_to_network(rdma, ®); - ret =3D qemu_rdma_exchange_send(rdma, &head, (uint8_t *) ®, - &resp, ®_result_idx, NULL, errp); - if (ret < 0) { - return -1; - } - - /* try to overlap this single registration with the one we sen= t. */ - if (qemu_rdma_register_and_get_keys(rdma, block, sge.addr, - &sge.lkey, NULL, chunk, - chunk_start, chunk_end)) { - error_setg(errp, "cannot get lkey"); - return -1; - } - - reg_result =3D (RDMARegisterResult *) - rdma->wr_data[reg_result_idx].control_curr; - - network_to_result(reg_result); - - trace_qemu_rdma_write_one_recvregres(block->remote_keys[chunk], - reg_result->rkey, chunk); - - block->remote_keys[chunk] =3D reg_result->rkey; - block->remote_host_addr =3D reg_result->host_addr; - } else { - /* already registered before */ - if (qemu_rdma_register_and_get_keys(rdma, block, sge.addr, - &sge.lkey, NULL, chunk, - chunk_start, chunk_end)) { - error_setg(errp, "cannot get lkey!"); - return -1; - } - } - - send_wr.wr.rdma.rkey =3D block->remote_keys[chunk]; - } else { - send_wr.wr.rdma.rkey =3D block->remote_rkey; - - if (qemu_rdma_register_and_get_keys(rdma, block, sge.addr, - &sge.lkey, NULL, chun= k, - chunk_start, chunk_en= d)) { - error_setg(errp, "cannot get lkey!"); - return -1; - } - } - - /* - * Encode the ram block index and chunk within this wrid. - * We will use this information at the time of completion - * to figure out which bitmap to check against and then which - * chunk in the bitmap to look for. - */ - send_wr.wr_id =3D qemu_rdma_make_wrid(RDMA_WRID_RDMA_WRITE, - current_index, chunk); - - send_wr.opcode =3D IBV_WR_RDMA_WRITE; - send_wr.send_flags =3D IBV_SEND_SIGNALED; - send_wr.sg_list =3D &sge; - send_wr.num_sge =3D 1; - send_wr.wr.rdma.remote_addr =3D block->remote_host_addr + - (current_addr - block->offset); - - trace_qemu_rdma_write_one_post(chunk, sge.addr, send_wr.wr.rdma.remote= _addr, - sge.length); - - /* - * ibv_post_send() does not return negative error numbers, - * per the specification they are positive - no idea why. - */ - ret =3D ibv_post_send(rdma->qp, &send_wr, &bad_wr); - - if (ret =3D=3D ENOMEM) { - trace_qemu_rdma_write_one_queue_full(); - ret =3D qemu_rdma_block_for_wrid(rdma, RDMA_WRID_RDMA_WRITE, NULL); - if (ret < 0) { - error_setg(errp, "rdma migration: failed to make " - "room in full send queue!"); - return -1; - } - - goto retry; - - } else if (ret > 0) { - error_setg_errno(errp, ret, - "rdma migration: post rdma write failed"); - return -1; - } - - set_bit(chunk, block->transit_bitmap); - stat64_add(&mig_stats.normal_pages, sge.length / qemu_target_page_size= ()); - /* - * We are adding to transferred the amount of data written, but no - * overhead at all. I will assume that RDMA is magicaly and don't - * need to transfer (at least) the addresses where it wants to - * write the pages. Here it looks like it should be something - * like: - * sizeof(send_wr) + sge.length - * but this being RDMA, who knows. - */ - stat64_add(&mig_stats.rdma_bytes, sge.length); - ram_transferred_add(sge.length); - rdma->total_writes++; - - return 0; -} - -/* - * Push out any unwritten RDMA operations. - * - * We support sending out multiple chunks at the same time. - * Not all of them need to get signaled in the completion queue. - */ -static int qemu_rdma_write_flush(RDMAContext *rdma, Error **errp) -{ - int ret; - - if (!rdma->current_length) { - return 0; - } - - ret =3D qemu_rdma_write_one(rdma, rdma->current_index, rdma->current_a= ddr, - rdma->current_length, errp); - - if (ret < 0) { - return -1; - } - - if (ret =3D=3D 0) { - rdma->nb_sent++; - trace_qemu_rdma_write_flush(rdma->nb_sent); - } - - rdma->current_length =3D 0; - rdma->current_addr =3D 0; - - return 0; -} - -static inline bool qemu_rdma_buffer_mergeable(RDMAContext *rdma, - uint64_t offset, uint64_t len) -{ - RDMALocalBlock *block; - uint8_t *host_addr; - uint8_t *chunk_end; - - if (rdma->current_index < 0) { - return false; - } - - if (rdma->current_chunk < 0) { - return false; - } - - block =3D &(rdma->local_ram_blocks.block[rdma->current_index]); - host_addr =3D block->local_host_addr + (offset - block->offset); - chunk_end =3D ram_chunk_end(block, rdma->current_chunk); - - if (rdma->current_length =3D=3D 0) { - return false; - } - - /* - * Only merge into chunk sequentially. - */ - if (offset !=3D (rdma->current_addr + rdma->current_length)) { - return false; - } - - if (offset < block->offset) { - return false; - } - - if ((offset + len) > (block->offset + block->length)) { - return false; - } - - if ((host_addr + len) > chunk_end) { - return false; - } - - return true; -} - -/* - * We're not actually writing here, but doing three things: - * - * 1. Identify the chunk the buffer belongs to. - * 2. If the chunk is full or the buffer doesn't belong to the current - * chunk, then start a new chunk and flush() the old chunk. - * 3. To keep the hardware busy, we also group chunks into batches - * and only require that a batch gets acknowledged in the completion - * queue instead of each individual chunk. - */ -static int qemu_rdma_write(RDMAContext *rdma, - uint64_t block_offset, uint64_t offset, - uint64_t len, Error **errp) -{ - uint64_t current_addr =3D block_offset + offset; - uint64_t index =3D rdma->current_index; - uint64_t chunk =3D rdma->current_chunk; - - /* If we cannot merge it, we flush the current buffer first. */ - if (!qemu_rdma_buffer_mergeable(rdma, current_addr, len)) { - if (qemu_rdma_write_flush(rdma, errp) < 0) { - return -1; - } - rdma->current_length =3D 0; - rdma->current_addr =3D current_addr; - - qemu_rdma_search_ram_block(rdma, block_offset, - offset, len, &index, &chunk); - rdma->current_index =3D index; - rdma->current_chunk =3D chunk; - } - - /* merge it */ - rdma->current_length +=3D len; - - /* flush it if buffer is too large */ - if (rdma->current_length >=3D RDMA_MERGE_MAX) { - return qemu_rdma_write_flush(rdma, errp); - } - - return 0; -} - -static void qemu_rdma_cleanup(RDMAContext *rdma) -{ - Error *err =3D NULL; - - if (rdma->cm_id && rdma->connected) { - if ((rdma->errored || - migrate_get_current()->state =3D=3D MIGRATION_STATUS_CANCELLI= NG) && - !rdma->received_error) { - RDMAControlHeader head =3D { .len =3D 0, - .type =3D RDMA_CONTROL_ERROR, - .repeat =3D 1, - }; - warn_report("Early error. Sending error."); - if (qemu_rdma_post_send_control(rdma, NULL, &head, &err) < 0) { - warn_report_err(err); - } - } - - rdma_disconnect(rdma->cm_id); - trace_qemu_rdma_cleanup_disconnect(); - rdma->connected =3D false; - } - - if (rdma->channel) { - qemu_set_fd_handler(rdma->channel->fd, NULL, NULL, NULL); - } - g_free(rdma->dest_blocks); - rdma->dest_blocks =3D NULL; - - for (int i =3D 0; i < RDMA_WRID_MAX; i++) { - if (rdma->wr_data[i].control_mr) { - rdma->total_registrations--; - ibv_dereg_mr(rdma->wr_data[i].control_mr); - } - rdma->wr_data[i].control_mr =3D NULL; - } - - if (rdma->local_ram_blocks.block) { - while (rdma->local_ram_blocks.nb_blocks) { - rdma_delete_block(rdma, &rdma->local_ram_blocks.block[0]); - } - } - - if (rdma->qp) { - rdma_destroy_qp(rdma->cm_id); - rdma->qp =3D NULL; - } - if (rdma->recv_cq) { - ibv_destroy_cq(rdma->recv_cq); - rdma->recv_cq =3D NULL; - } - if (rdma->send_cq) { - ibv_destroy_cq(rdma->send_cq); - rdma->send_cq =3D NULL; - } - if (rdma->recv_comp_channel) { - ibv_destroy_comp_channel(rdma->recv_comp_channel); - rdma->recv_comp_channel =3D NULL; - } - if (rdma->send_comp_channel) { - ibv_destroy_comp_channel(rdma->send_comp_channel); - rdma->send_comp_channel =3D NULL; - } - if (rdma->pd) { - ibv_dealloc_pd(rdma->pd); - rdma->pd =3D NULL; - } - if (rdma->cm_id) { - rdma_destroy_id(rdma->cm_id); - rdma->cm_id =3D NULL; - } - - /* the destination side, listen_id and channel is shared */ - if (rdma->listen_id) { - if (!rdma->is_return_path) { - rdma_destroy_id(rdma->listen_id); - } - rdma->listen_id =3D NULL; - - if (rdma->channel) { - if (!rdma->is_return_path) { - rdma_destroy_event_channel(rdma->channel); - } - rdma->channel =3D NULL; - } - } - - if (rdma->channel) { - rdma_destroy_event_channel(rdma->channel); - rdma->channel =3D NULL; - } - g_free(rdma->host); - rdma->host =3D NULL; -} - - -static int qemu_rdma_source_init(RDMAContext *rdma, bool pin_all, Error **= errp) -{ - int ret; - - /* - * Will be validated against destination's actual capabilities - * after the connect() completes. - */ - rdma->pin_all =3D pin_all; - - ret =3D qemu_rdma_resolve_host(rdma, errp); - if (ret < 0) { - goto err_rdma_source_init; - } - - ret =3D qemu_rdma_alloc_pd_cq(rdma, errp); - if (ret < 0) { - goto err_rdma_source_init; - } - - ret =3D qemu_rdma_alloc_qp(rdma); - if (ret < 0) { - error_setg(errp, "RDMA ERROR: rdma migration: error allocating qp!= "); - goto err_rdma_source_init; - } - - qemu_rdma_init_ram_blocks(rdma); - - /* Build the hash that maps from offset to RAMBlock */ - rdma->blockmap =3D g_hash_table_new(g_direct_hash, g_direct_equal); - for (int i =3D 0; i < rdma->local_ram_blocks.nb_blocks; i++) { - g_hash_table_insert(rdma->blockmap, - (void *)(uintptr_t)rdma->local_ram_blocks.block[i].offset, - &rdma->local_ram_blocks.block[i]); - } - - for (int i =3D 0; i < RDMA_WRID_MAX; i++) { - ret =3D qemu_rdma_reg_control(rdma, i); - if (ret < 0) { - error_setg(errp, "RDMA ERROR: rdma migration: error " - "registering %d control!", i); - goto err_rdma_source_init; - } - } - - return 0; - -err_rdma_source_init: - qemu_rdma_cleanup(rdma); - return -1; -} - -static int qemu_get_cm_event_timeout(RDMAContext *rdma, - struct rdma_cm_event **cm_event, - long msec, Error **errp) -{ - int ret; - struct pollfd poll_fd =3D { - .fd =3D rdma->channel->fd, - .events =3D POLLIN, - .revents =3D 0 - }; - - do { - ret =3D poll(&poll_fd, 1, msec); - } while (ret < 0 && errno =3D=3D EINTR); - - if (ret =3D=3D 0) { - error_setg(errp, "RDMA ERROR: poll cm event timeout"); - return -1; - } else if (ret < 0) { - error_setg(errp, "RDMA ERROR: failed to poll cm event, errno=3D%i", - errno); - return -1; - } else if (poll_fd.revents & POLLIN) { - if (rdma_get_cm_event(rdma->channel, cm_event) < 0) { - error_setg(errp, "RDMA ERROR: failed to get cm event"); - return -1; - } - return 0; - } else { - error_setg(errp, "RDMA ERROR: no POLLIN event, revent=3D%x", - poll_fd.revents); - return -1; - } -} - -static int qemu_rdma_connect(RDMAContext *rdma, bool return_path, - Error **errp) -{ - RDMACapabilities cap =3D { - .version =3D RDMA_CONTROL_VERSION_CURRENT, - .flags =3D 0, - }; - struct rdma_conn_param conn_param =3D { .initiator_depth =3D 2, - .retry_count =3D 5, - .private_data =3D &cap, - .private_data_len =3D sizeof(cap= ), - }; - struct rdma_cm_event *cm_event; - int ret; - - /* - * Only negotiate the capability with destination if the user - * on the source first requested the capability. - */ - if (rdma->pin_all) { - trace_qemu_rdma_connect_pin_all_requested(); - cap.flags |=3D RDMA_CAPABILITY_PIN_ALL; - } - - caps_to_network(&cap); - - ret =3D qemu_rdma_post_recv_control(rdma, RDMA_WRID_READY, errp); - if (ret < 0) { - goto err_rdma_source_connect; - } - - ret =3D rdma_connect(rdma->cm_id, &conn_param); - if (ret < 0) { - error_setg_errno(errp, errno, - "RDMA ERROR: connecting to destination!"); - goto err_rdma_source_connect; - } - - if (return_path) { - ret =3D qemu_get_cm_event_timeout(rdma, &cm_event, 5000, errp); - } else { - ret =3D rdma_get_cm_event(rdma->channel, &cm_event); - if (ret < 0) { - error_setg_errno(errp, errno, - "RDMA ERROR: failed to get cm event"); - } - } - if (ret < 0) { - goto err_rdma_source_connect; - } - - if (cm_event->event !=3D RDMA_CM_EVENT_ESTABLISHED) { - error_setg(errp, "RDMA ERROR: connecting to destination!"); - rdma_ack_cm_event(cm_event); - goto err_rdma_source_connect; - } - rdma->connected =3D true; - - memcpy(&cap, cm_event->param.conn.private_data, sizeof(cap)); - network_to_caps(&cap); - - /* - * Verify that the *requested* capabilities are supported by the desti= nation - * and disable them otherwise. - */ - if (rdma->pin_all && !(cap.flags & RDMA_CAPABILITY_PIN_ALL)) { - warn_report("RDMA: Server cannot support pinning all memory. " - "Will register memory dynamically."); - rdma->pin_all =3D false; - } - - trace_qemu_rdma_connect_pin_all_outcome(rdma->pin_all); - - rdma_ack_cm_event(cm_event); - - rdma->control_ready_expected =3D 1; - rdma->nb_sent =3D 0; - return 0; - -err_rdma_source_connect: - qemu_rdma_cleanup(rdma); - return -1; -} - -static int qemu_rdma_dest_init(RDMAContext *rdma, Error **errp) -{ - Error *err =3D NULL; - int ret; - struct rdma_cm_id *listen_id; - char ip[40] =3D "unknown"; - struct rdma_addrinfo *res, *e; - char port_str[16]; - int reuse =3D 1; - - for (int i =3D 0; i < RDMA_WRID_MAX; i++) { - rdma->wr_data[i].control_len =3D 0; - rdma->wr_data[i].control_curr =3D NULL; - } - - if (!rdma->host || !rdma->host[0]) { - error_setg(errp, "RDMA ERROR: RDMA host is not set!"); - rdma->errored =3D true; - return -1; - } - /* create CM channel */ - rdma->channel =3D rdma_create_event_channel(); - if (!rdma->channel) { - error_setg(errp, "RDMA ERROR: could not create rdma event channel"= ); - rdma->errored =3D true; - return -1; - } - - /* create CM id */ - ret =3D rdma_create_id(rdma->channel, &listen_id, NULL, RDMA_PS_TCP); - if (ret < 0) { - error_setg(errp, "RDMA ERROR: could not create cm_id!"); - goto err_dest_init_create_listen_id; - } - - snprintf(port_str, 16, "%d", rdma->port); - port_str[15] =3D '\0'; - - ret =3D rdma_getaddrinfo(rdma->host, port_str, NULL, &res); - if (ret) { - error_setg(errp, "RDMA ERROR: could not rdma_getaddrinfo address %= s", - rdma->host); - goto err_dest_init_bind_addr; - } - - ret =3D rdma_set_option(listen_id, RDMA_OPTION_ID, RDMA_OPTION_ID_REUS= EADDR, - &reuse, sizeof reuse); - if (ret < 0) { - error_setg(errp, "RDMA ERROR: Error: could not set REUSEADDR optio= n"); - goto err_dest_init_bind_addr; - } - - /* Try all addresses, saving the first error in @err */ - for (e =3D res; e !=3D NULL; e =3D e->ai_next) { - Error **local_errp =3D err ? NULL : &err; - - inet_ntop(e->ai_family, - &((struct sockaddr_in *) e->ai_dst_addr)->sin_addr, ip, sizeof= ip); - trace_qemu_rdma_dest_init_trying(rdma->host, ip); - ret =3D rdma_bind_addr(listen_id, e->ai_dst_addr); - if (ret < 0) { - continue; - } - if (e->ai_family =3D=3D AF_INET6) { - ret =3D qemu_rdma_broken_ipv6_kernel(listen_id->verbs, - local_errp); - if (ret < 0) { - continue; - } - } - error_free(err); - break; - } - - rdma_freeaddrinfo(res); - if (!e) { - if (err) { - error_propagate(errp, err); - } else { - error_setg(errp, "RDMA ERROR: Error: could not rdma_bind_addr!= "); - } - goto err_dest_init_bind_addr; - } - - rdma->listen_id =3D listen_id; - qemu_rdma_dump_gid("dest_init", listen_id); - return 0; - -err_dest_init_bind_addr: - rdma_destroy_id(listen_id); -err_dest_init_create_listen_id: - rdma_destroy_event_channel(rdma->channel); - rdma->channel =3D NULL; - rdma->errored =3D true; - return -1; - -} - -static void qemu_rdma_return_path_dest_init(RDMAContext *rdma_return_path, - RDMAContext *rdma) -{ - for (int i =3D 0; i < RDMA_WRID_MAX; i++) { - rdma_return_path->wr_data[i].control_len =3D 0; - rdma_return_path->wr_data[i].control_curr =3D NULL; - } - - /*the CM channel and CM id is shared*/ - rdma_return_path->channel =3D rdma->channel; - rdma_return_path->listen_id =3D rdma->listen_id; - - rdma->return_path =3D rdma_return_path; - rdma_return_path->return_path =3D rdma; - rdma_return_path->is_return_path =3D true; -} - -static RDMAContext *qemu_rdma_data_init(InetSocketAddress *saddr, Error **= errp) -{ - RDMAContext *rdma =3D NULL; - - rdma =3D g_new0(RDMAContext, 1); - rdma->current_index =3D -1; - rdma->current_chunk =3D -1; - - rdma->host =3D g_strdup(saddr->host); - rdma->port =3D atoi(saddr->port); - return rdma; -} - -/* - * QEMUFile interface to the control channel. - * SEND messages for control only. - * VM's ram is handled with regular RDMA messages. - */ -static ssize_t qio_channel_rdma_writev(QIOChannel *ioc, - const struct iovec *iov, - size_t niov, - int *fds, - size_t nfds, - int flags, - Error **errp) -{ - QIOChannelRDMA *rioc =3D QIO_CHANNEL_RDMA(ioc); - RDMAContext *rdma; - int ret; - ssize_t done =3D 0; - size_t len; - - RCU_READ_LOCK_GUARD(); - rdma =3D qatomic_rcu_read(&rioc->rdmaout); - - if (!rdma) { - error_setg(errp, "RDMA control channel output is not set"); - return -1; - } - - if (rdma->errored) { - error_setg(errp, - "RDMA is in an error state waiting migration to abort!"= ); - return -1; - } - - /* - * Push out any writes that - * we're queued up for VM's ram. - */ - ret =3D qemu_rdma_write_flush(rdma, errp); - if (ret < 0) { - rdma->errored =3D true; - return -1; - } - - for (int i =3D 0; i < niov; i++) { - size_t remaining =3D iov[i].iov_len; - uint8_t * data =3D (void *)iov[i].iov_base; - while (remaining) { - RDMAControlHeader head =3D {}; - - len =3D MIN(remaining, RDMA_SEND_INCREMENT); - remaining -=3D len; - - head.len =3D len; - head.type =3D RDMA_CONTROL_QEMU_FILE; - - ret =3D qemu_rdma_exchange_send(rdma, &head, - data, NULL, NULL, NULL, errp); - - if (ret < 0) { - rdma->errored =3D true; - return -1; - } - - data +=3D len; - done +=3D len; - } - } - - return done; -} - -static size_t qemu_rdma_fill(RDMAContext *rdma, uint8_t *buf, - size_t size, int idx) -{ - size_t len =3D 0; - - if (rdma->wr_data[idx].control_len) { - trace_qemu_rdma_fill(rdma->wr_data[idx].control_len, size); - - len =3D MIN(size, rdma->wr_data[idx].control_len); - memcpy(buf, rdma->wr_data[idx].control_curr, len); - rdma->wr_data[idx].control_curr +=3D len; - rdma->wr_data[idx].control_len -=3D len; - } - - return len; -} - -/* - * QEMUFile interface to the control channel. - * RDMA links don't use bytestreams, so we have to - * return bytes to QEMUFile opportunistically. - */ -static ssize_t qio_channel_rdma_readv(QIOChannel *ioc, - const struct iovec *iov, - size_t niov, - int **fds, - size_t *nfds, - int flags, - Error **errp) -{ - QIOChannelRDMA *rioc =3D QIO_CHANNEL_RDMA(ioc); - RDMAContext *rdma; - RDMAControlHeader head; - int ret; - ssize_t done =3D 0; - size_t len; - - RCU_READ_LOCK_GUARD(); - rdma =3D qatomic_rcu_read(&rioc->rdmain); - - if (!rdma) { - error_setg(errp, "RDMA control channel input is not set"); - return -1; - } - - if (rdma->errored) { - error_setg(errp, - "RDMA is in an error state waiting migration to abort!"= ); - return -1; - } - - for (int i =3D 0; i < niov; i++) { - size_t want =3D iov[i].iov_len; - uint8_t *data =3D (void *)iov[i].iov_base; - - /* - * First, we hold on to the last SEND message we - * were given and dish out the bytes until we run - * out of bytes. - */ - len =3D qemu_rdma_fill(rdma, data, want, 0); - done +=3D len; - want -=3D len; - /* Got what we needed, so go to next iovec */ - if (want =3D=3D 0) { - continue; - } - - /* If we got any data so far, then don't wait - * for more, just return what we have */ - if (done > 0) { - break; - } - - - /* We've got nothing at all, so lets wait for - * more to arrive - */ - ret =3D qemu_rdma_exchange_recv(rdma, &head, RDMA_CONTROL_QEMU_FIL= E, - errp); - - if (ret < 0) { - rdma->errored =3D true; - return -1; - } - - /* - * SEND was received with new bytes, now try again. - */ - len =3D qemu_rdma_fill(rdma, data, want, 0); - done +=3D len; - want -=3D len; - - /* Still didn't get enough, so lets just return */ - if (want) { - if (done =3D=3D 0) { - return QIO_CHANNEL_ERR_BLOCK; - } else { - break; - } - } - } - return done; -} - -/* - * Block until all the outstanding chunks have been delivered by the hardw= are. - */ -static int qemu_rdma_drain_cq(RDMAContext *rdma) -{ - Error *err =3D NULL; - - if (qemu_rdma_write_flush(rdma, &err) < 0) { - error_report_err(err); - return -1; - } - - while (rdma->nb_sent) { - if (qemu_rdma_block_for_wrid(rdma, RDMA_WRID_RDMA_WRITE, NULL) < 0= ) { - error_report("rdma migration: complete polling error!"); - return -1; - } - } - - qemu_rdma_unregister_waiting(rdma); - - return 0; -} - - -static int qio_channel_rdma_set_blocking(QIOChannel *ioc, - bool blocking, - Error **errp) -{ - QIOChannelRDMA *rioc =3D QIO_CHANNEL_RDMA(ioc); - /* XXX we should make readv/writev actually honour this :-) */ - rioc->blocking =3D blocking; - return 0; -} - - -typedef struct QIOChannelRDMASource QIOChannelRDMASource; -struct QIOChannelRDMASource { - GSource parent; - QIOChannelRDMA *rioc; - GIOCondition condition; -}; - -static gboolean -qio_channel_rdma_source_prepare(GSource *source, - gint *timeout) -{ - QIOChannelRDMASource *rsource =3D (QIOChannelRDMASource *)source; - RDMAContext *rdma; - GIOCondition cond =3D 0; - *timeout =3D -1; - - RCU_READ_LOCK_GUARD(); - if (rsource->condition =3D=3D G_IO_IN) { - rdma =3D qatomic_rcu_read(&rsource->rioc->rdmain); - } else { - rdma =3D qatomic_rcu_read(&rsource->rioc->rdmaout); - } - - if (!rdma) { - error_report("RDMAContext is NULL when prepare Gsource"); - return FALSE; - } - - if (rdma->wr_data[0].control_len) { - cond |=3D G_IO_IN; - } - cond |=3D G_IO_OUT; - - return cond & rsource->condition; -} - -static gboolean -qio_channel_rdma_source_check(GSource *source) -{ - QIOChannelRDMASource *rsource =3D (QIOChannelRDMASource *)source; - RDMAContext *rdma; - GIOCondition cond =3D 0; - - RCU_READ_LOCK_GUARD(); - if (rsource->condition =3D=3D G_IO_IN) { - rdma =3D qatomic_rcu_read(&rsource->rioc->rdmain); - } else { - rdma =3D qatomic_rcu_read(&rsource->rioc->rdmaout); - } - - if (!rdma) { - error_report("RDMAContext is NULL when check Gsource"); - return FALSE; - } - - if (rdma->wr_data[0].control_len) { - cond |=3D G_IO_IN; - } - cond |=3D G_IO_OUT; - - return cond & rsource->condition; -} - -static gboolean -qio_channel_rdma_source_dispatch(GSource *source, - GSourceFunc callback, - gpointer user_data) -{ - QIOChannelFunc func =3D (QIOChannelFunc)callback; - QIOChannelRDMASource *rsource =3D (QIOChannelRDMASource *)source; - RDMAContext *rdma; - GIOCondition cond =3D 0; - - RCU_READ_LOCK_GUARD(); - if (rsource->condition =3D=3D G_IO_IN) { - rdma =3D qatomic_rcu_read(&rsource->rioc->rdmain); - } else { - rdma =3D qatomic_rcu_read(&rsource->rioc->rdmaout); - } - - if (!rdma) { - error_report("RDMAContext is NULL when dispatch Gsource"); - return FALSE; - } - - if (rdma->wr_data[0].control_len) { - cond |=3D G_IO_IN; - } - cond |=3D G_IO_OUT; - - return (*func)(QIO_CHANNEL(rsource->rioc), - (cond & rsource->condition), - user_data); -} - -static void -qio_channel_rdma_source_finalize(GSource *source) -{ - QIOChannelRDMASource *ssource =3D (QIOChannelRDMASource *)source; - - object_unref(OBJECT(ssource->rioc)); -} - -static GSourceFuncs qio_channel_rdma_source_funcs =3D { - qio_channel_rdma_source_prepare, - qio_channel_rdma_source_check, - qio_channel_rdma_source_dispatch, - qio_channel_rdma_source_finalize -}; - -static GSource *qio_channel_rdma_create_watch(QIOChannel *ioc, - GIOCondition condition) -{ - QIOChannelRDMA *rioc =3D QIO_CHANNEL_RDMA(ioc); - QIOChannelRDMASource *ssource; - GSource *source; - - source =3D g_source_new(&qio_channel_rdma_source_funcs, - sizeof(QIOChannelRDMASource)); - ssource =3D (QIOChannelRDMASource *)source; - - ssource->rioc =3D rioc; - object_ref(OBJECT(rioc)); - - ssource->condition =3D condition; - - return source; -} - -static void qio_channel_rdma_set_aio_fd_handler(QIOChannel *ioc, - AioContext *read_ctx, - IOHandler *io_read, - AioContext *write_ctx, - IOHandler *io_write, - void *opaque) -{ - QIOChannelRDMA *rioc =3D QIO_CHANNEL_RDMA(ioc); - if (io_read) { - aio_set_fd_handler(read_ctx, rioc->rdmain->recv_comp_channel->fd, - io_read, io_write, NULL, NULL, opaque); - aio_set_fd_handler(read_ctx, rioc->rdmain->send_comp_channel->fd, - io_read, io_write, NULL, NULL, opaque); - } else { - aio_set_fd_handler(write_ctx, rioc->rdmaout->recv_comp_channel->fd, - io_read, io_write, NULL, NULL, opaque); - aio_set_fd_handler(write_ctx, rioc->rdmaout->send_comp_channel->fd, - io_read, io_write, NULL, NULL, opaque); - } -} - -struct rdma_close_rcu { - struct rcu_head rcu; - RDMAContext *rdmain; - RDMAContext *rdmaout; -}; - -/* callback from qio_channel_rdma_close via call_rcu */ -static void qio_channel_rdma_close_rcu(struct rdma_close_rcu *rcu) -{ - if (rcu->rdmain) { - qemu_rdma_cleanup(rcu->rdmain); - } - - if (rcu->rdmaout) { - qemu_rdma_cleanup(rcu->rdmaout); - } - - g_free(rcu->rdmain); - g_free(rcu->rdmaout); - g_free(rcu); -} - -static int qio_channel_rdma_close(QIOChannel *ioc, - Error **errp) -{ - QIOChannelRDMA *rioc =3D QIO_CHANNEL_RDMA(ioc); - RDMAContext *rdmain, *rdmaout; - struct rdma_close_rcu *rcu =3D g_new(struct rdma_close_rcu, 1); - - trace_qemu_rdma_close(); - - rdmain =3D rioc->rdmain; - if (rdmain) { - qatomic_rcu_set(&rioc->rdmain, NULL); - } - - rdmaout =3D rioc->rdmaout; - if (rdmaout) { - qatomic_rcu_set(&rioc->rdmaout, NULL); - } - - rcu->rdmain =3D rdmain; - rcu->rdmaout =3D rdmaout; - call_rcu(rcu, qio_channel_rdma_close_rcu, rcu); - - return 0; -} - -static int -qio_channel_rdma_shutdown(QIOChannel *ioc, - QIOChannelShutdown how, - Error **errp) -{ - QIOChannelRDMA *rioc =3D QIO_CHANNEL_RDMA(ioc); - RDMAContext *rdmain, *rdmaout; - - RCU_READ_LOCK_GUARD(); - - rdmain =3D qatomic_rcu_read(&rioc->rdmain); - rdmaout =3D qatomic_rcu_read(&rioc->rdmain); - - switch (how) { - case QIO_CHANNEL_SHUTDOWN_READ: - if (rdmain) { - rdmain->errored =3D true; - } - break; - case QIO_CHANNEL_SHUTDOWN_WRITE: - if (rdmaout) { - rdmaout->errored =3D true; - } - break; - case QIO_CHANNEL_SHUTDOWN_BOTH: - default: - if (rdmain) { - rdmain->errored =3D true; - } - if (rdmaout) { - rdmaout->errored =3D true; - } - break; - } - - return 0; -} - -/* - * Parameters: - * @offset =3D=3D 0 : - * This means that 'block_offset' is a full virtual address that do= es not - * belong to a RAMBlock of the virtual machine and instead - * represents a private malloc'd memory area that the caller wishes= to - * transfer. - * - * @offset !=3D 0 : - * Offset is an offset to be added to block_offset and used - * to also lookup the corresponding RAMBlock. - * - * @size : Number of bytes to transfer - * - * @pages_sent : User-specificed pointer to indicate how many pages were - * sent. Usually, this will not be more than a few bytes = of - * the protocol because most transfers are sent asynchron= ously. - */ -static int qemu_rdma_save_page(QEMUFile *f, ram_addr_t block_offset, - ram_addr_t offset, size_t size) -{ - QIOChannelRDMA *rioc =3D QIO_CHANNEL_RDMA(qemu_file_get_ioc(f)); - Error *err =3D NULL; - RDMAContext *rdma; - int ret; - - RCU_READ_LOCK_GUARD(); - rdma =3D qatomic_rcu_read(&rioc->rdmaout); - - if (!rdma) { - return -1; - } - - if (rdma_errored(rdma)) { - return -1; - } - - qemu_fflush(f); - - /* - * Add this page to the current 'chunk'. If the chunk - * is full, or the page doesn't belong to the current chunk, - * an actual RDMA write will occur and a new chunk will be formed. - */ - ret =3D qemu_rdma_write(rdma, block_offset, offset, size, &err); - if (ret < 0) { - error_report_err(err); - goto err; - } - - /* - * Drain the Completion Queue if possible, but do not block, - * just poll. - * - * If nothing to poll, the end of the iteration will do this - * again to make sure we don't overflow the request queue. - */ - while (1) { - uint64_t wr_id, wr_id_in; - ret =3D qemu_rdma_poll(rdma, rdma->recv_cq, &wr_id_in, NULL); - - if (ret < 0) { - error_report("rdma migration: polling error"); - goto err; - } - - wr_id =3D wr_id_in & RDMA_WRID_TYPE_MASK; - - if (wr_id =3D=3D RDMA_WRID_NONE) { - break; - } - } - - while (1) { - uint64_t wr_id, wr_id_in; - ret =3D qemu_rdma_poll(rdma, rdma->send_cq, &wr_id_in, NULL); - - if (ret < 0) { - error_report("rdma migration: polling error"); - goto err; - } - - wr_id =3D wr_id_in & RDMA_WRID_TYPE_MASK; - - if (wr_id =3D=3D RDMA_WRID_NONE) { - break; - } - } - - return RAM_SAVE_CONTROL_DELAYED; - -err: - rdma->errored =3D true; - return -1; -} - -int rdma_control_save_page(QEMUFile *f, ram_addr_t block_offset, - ram_addr_t offset, size_t size) -{ - if (!migrate_rdma() || migration_in_postcopy()) { - return RAM_SAVE_CONTROL_NOT_SUPP; - } - - int ret =3D qemu_rdma_save_page(f, block_offset, offset, size); - - if (ret !=3D RAM_SAVE_CONTROL_DELAYED && - ret !=3D RAM_SAVE_CONTROL_NOT_SUPP) { - if (ret < 0) { - qemu_file_set_error(f, ret); - } - } - return ret; -} - -static void rdma_accept_incoming_migration(void *opaque); - -static void rdma_cm_poll_handler(void *opaque) -{ - RDMAContext *rdma =3D opaque; - struct rdma_cm_event *cm_event; - MigrationIncomingState *mis =3D migration_incoming_get_current(); - - if (rdma_get_cm_event(rdma->channel, &cm_event) < 0) { - error_report("get_cm_event failed %d", errno); - return; - } - - if (cm_event->event =3D=3D RDMA_CM_EVENT_DISCONNECTED || - cm_event->event =3D=3D RDMA_CM_EVENT_DEVICE_REMOVAL) { - if (!rdma->errored && - migration_incoming_get_current()->state !=3D - MIGRATION_STATUS_COMPLETED) { - error_report("receive cm event, cm event is %d", cm_event->eve= nt); - rdma->errored =3D true; - if (rdma->return_path) { - rdma->return_path->errored =3D true; - } - } - rdma_ack_cm_event(cm_event); - if (mis->loadvm_co) { - qemu_coroutine_enter(mis->loadvm_co); - } - return; - } - rdma_ack_cm_event(cm_event); -} - -static int qemu_rdma_accept(RDMAContext *rdma) -{ - Error *err =3D NULL; - RDMACapabilities cap; - struct rdma_conn_param conn_param =3D { - .responder_resources =3D 2, - .private_data =3D &cap, - .private_data_len =3D sizeof(c= ap), - }; - RDMAContext *rdma_return_path =3D NULL; - g_autoptr(InetSocketAddress) isock =3D g_new0(InetSocketAddress, 1); - struct rdma_cm_event *cm_event; - struct ibv_context *verbs; - int ret; - - ret =3D rdma_get_cm_event(rdma->channel, &cm_event); - if (ret < 0) { - goto err_rdma_dest_wait; - } - - if (cm_event->event !=3D RDMA_CM_EVENT_CONNECT_REQUEST) { - rdma_ack_cm_event(cm_event); - goto err_rdma_dest_wait; - } - - isock->host =3D g_strdup(rdma->host); - isock->port =3D g_strdup_printf("%d", rdma->port); - - /* - * initialize the RDMAContext for return path for postcopy after first - * connection request reached. - */ - if ((migrate_postcopy() || migrate_return_path()) - && !rdma->is_return_path) { - rdma_return_path =3D qemu_rdma_data_init(isock, NULL); - if (rdma_return_path =3D=3D NULL) { - rdma_ack_cm_event(cm_event); - goto err_rdma_dest_wait; - } - - qemu_rdma_return_path_dest_init(rdma_return_path, rdma); - } - - memcpy(&cap, cm_event->param.conn.private_data, sizeof(cap)); - - network_to_caps(&cap); - - if (cap.version < 1 || cap.version > RDMA_CONTROL_VERSION_CURRENT) { - error_report("Unknown source RDMA version: %d, bailing...", - cap.version); - rdma_ack_cm_event(cm_event); - goto err_rdma_dest_wait; - } - - /* - * Respond with only the capabilities this version of QEMU knows about. - */ - cap.flags &=3D known_capabilities; - - /* - * Enable the ones that we do know about. - * Add other checks here as new ones are introduced. - */ - if (cap.flags & RDMA_CAPABILITY_PIN_ALL) { - rdma->pin_all =3D true; - } - - rdma->cm_id =3D cm_event->id; - verbs =3D cm_event->id->verbs; - - rdma_ack_cm_event(cm_event); - - trace_qemu_rdma_accept_pin_state(rdma->pin_all); - - caps_to_network(&cap); - - trace_qemu_rdma_accept_pin_verbsc(verbs); - - if (!rdma->verbs) { - rdma->verbs =3D verbs; - } else if (rdma->verbs !=3D verbs) { - error_report("ibv context not matching %p, %p!", rdma->verbs, - verbs); - goto err_rdma_dest_wait; - } - - qemu_rdma_dump_id("dest_init", verbs); - - ret =3D qemu_rdma_alloc_pd_cq(rdma, &err); - if (ret < 0) { - error_report_err(err); - goto err_rdma_dest_wait; - } - - ret =3D qemu_rdma_alloc_qp(rdma); - if (ret < 0) { - error_report("rdma migration: error allocating qp!"); - goto err_rdma_dest_wait; - } - - qemu_rdma_init_ram_blocks(rdma); - - for (int i =3D 0; i < RDMA_WRID_MAX; i++) { - ret =3D qemu_rdma_reg_control(rdma, i); - if (ret < 0) { - error_report("rdma: error registering %d control", i); - goto err_rdma_dest_wait; - } - } - - /* Accept the second connection request for return path */ - if ((migrate_postcopy() || migrate_return_path()) - && !rdma->is_return_path) { - qemu_set_fd_handler(rdma->channel->fd, rdma_accept_incoming_migrat= ion, - NULL, - (void *)(intptr_t)rdma->return_path); - } else { - qemu_set_fd_handler(rdma->channel->fd, rdma_cm_poll_handler, - NULL, rdma); - } - - ret =3D rdma_accept(rdma->cm_id, &conn_param); - if (ret < 0) { - error_report("rdma_accept failed"); - goto err_rdma_dest_wait; - } - - ret =3D rdma_get_cm_event(rdma->channel, &cm_event); - if (ret < 0) { - error_report("rdma_accept get_cm_event failed"); - goto err_rdma_dest_wait; - } - - if (cm_event->event !=3D RDMA_CM_EVENT_ESTABLISHED) { - error_report("rdma_accept not event established"); - rdma_ack_cm_event(cm_event); - goto err_rdma_dest_wait; - } - - rdma_ack_cm_event(cm_event); - rdma->connected =3D true; - - ret =3D qemu_rdma_post_recv_control(rdma, RDMA_WRID_READY, &err); - if (ret < 0) { - error_report_err(err); - goto err_rdma_dest_wait; - } - - qemu_rdma_dump_gid("dest_connect", rdma->cm_id); - - return 0; - -err_rdma_dest_wait: - rdma->errored =3D true; - qemu_rdma_cleanup(rdma); - g_free(rdma_return_path); - return -1; -} - -static int dest_ram_sort_func(const void *a, const void *b) -{ - unsigned int a_index =3D ((const RDMALocalBlock *)a)->src_index; - unsigned int b_index =3D ((const RDMALocalBlock *)b)->src_index; - - return (a_index < b_index) ? -1 : (a_index !=3D b_index); -} - -/* - * During each iteration of the migration, we listen for instructions - * by the source VM to perform dynamic page registrations before they - * can perform RDMA operations. - * - * We respond with the 'rkey'. - * - * Keep doing this until the source tells us to stop. - */ -int rdma_registration_handle(QEMUFile *f) -{ - RDMAControlHeader reg_resp =3D { .len =3D sizeof(RDMARegisterResult), - .type =3D RDMA_CONTROL_REGISTER_RESULT, - .repeat =3D 0, - }; - RDMAControlHeader unreg_resp =3D { .len =3D 0, - .type =3D RDMA_CONTROL_UNREGISTER_FINISHED, - .repeat =3D 0, - }; - RDMAControlHeader blocks =3D { .type =3D RDMA_CONTROL_RAM_BLOCKS_RESUL= T, - .repeat =3D 1 }; - QIOChannelRDMA *rioc; - Error *err =3D NULL; - RDMAContext *rdma; - RDMALocalBlocks *local; - RDMAControlHeader head; - RDMARegister *reg, *registers; - RDMACompress *comp; - RDMARegisterResult *reg_result; - static RDMARegisterResult results[RDMA_CONTROL_MAX_COMMANDS_PER_MESSAG= E]; - RDMALocalBlock *block; - void *host_addr; - int ret; - int idx =3D 0; - - if (!migrate_rdma()) { - return 0; - } - - RCU_READ_LOCK_GUARD(); - rioc =3D QIO_CHANNEL_RDMA(qemu_file_get_ioc(f)); - rdma =3D qatomic_rcu_read(&rioc->rdmain); - - if (!rdma) { - return -1; - } - - if (rdma_errored(rdma)) { - return -1; - } - - local =3D &rdma->local_ram_blocks; - do { - trace_rdma_registration_handle_wait(); - - ret =3D qemu_rdma_exchange_recv(rdma, &head, RDMA_CONTROL_NONE, &e= rr); - - if (ret < 0) { - error_report_err(err); - break; - } - - if (head.repeat > RDMA_CONTROL_MAX_COMMANDS_PER_MESSAGE) { - error_report("rdma: Too many requests in this message (%d)." - "Bailing.", head.repeat); - break; - } - - switch (head.type) { - case RDMA_CONTROL_COMPRESS: - comp =3D (RDMACompress *) rdma->wr_data[idx].control_curr; - network_to_compress(comp); - - trace_rdma_registration_handle_compress(comp->length, - comp->block_idx, - comp->offset); - if (comp->block_idx >=3D rdma->local_ram_blocks.nb_blocks) { - error_report("rdma: 'compress' bad block index %u (vs %d)", - (unsigned int)comp->block_idx, - rdma->local_ram_blocks.nb_blocks); - goto err; - } - block =3D &(rdma->local_ram_blocks.block[comp->block_idx]); - - host_addr =3D block->local_host_addr + - (comp->offset - block->offset); - if (comp->value) { - error_report("rdma: Zero page with non-zero (%d) value", - comp->value); - goto err; - } - ram_handle_zero(host_addr, comp->length); - break; - - case RDMA_CONTROL_REGISTER_FINISHED: - trace_rdma_registration_handle_finished(); - return 0; - - case RDMA_CONTROL_RAM_BLOCKS_REQUEST: - trace_rdma_registration_handle_ram_blocks(); - - /* Sort our local RAM Block list so it's the same as the sourc= e, - * we can do this since we've filled in a src_index in the list - * as we received the RAMBlock list earlier. - */ - qsort(rdma->local_ram_blocks.block, - rdma->local_ram_blocks.nb_blocks, - sizeof(RDMALocalBlock), dest_ram_sort_func); - for (int i =3D 0; i < local->nb_blocks; i++) { - local->block[i].index =3D i; - } - - if (rdma->pin_all) { - ret =3D qemu_rdma_reg_whole_ram_blocks(rdma, &err); - if (ret < 0) { - error_report_err(err); - goto err; - } - } - - /* - * Dest uses this to prepare to transmit the RAMBlock descript= ions - * to the source VM after connection setup. - * Both sides use the "remote" structure to communicate and up= date - * their "local" descriptions with what was sent. - */ - for (int i =3D 0; i < local->nb_blocks; i++) { - rdma->dest_blocks[i].remote_host_addr =3D - (uintptr_t)(local->block[i].local_host_addr); - - if (rdma->pin_all) { - rdma->dest_blocks[i].remote_rkey =3D local->block[i].m= r->rkey; - } - - rdma->dest_blocks[i].offset =3D local->block[i].offset; - rdma->dest_blocks[i].length =3D local->block[i].length; - - dest_block_to_network(&rdma->dest_blocks[i]); - trace_rdma_registration_handle_ram_blocks_loop( - local->block[i].block_name, - local->block[i].offset, - local->block[i].length, - local->block[i].local_host_addr, - local->block[i].src_index); - } - - blocks.len =3D rdma->local_ram_blocks.nb_blocks - * sizeof(RDMADestBlock); - - - ret =3D qemu_rdma_post_send_control(rdma, - (uint8_t *) rdma->dest_blocks, &blocks, - &err); - - if (ret < 0) { - error_report_err(err); - goto err; - } - - break; - case RDMA_CONTROL_REGISTER_REQUEST: - trace_rdma_registration_handle_register(head.repeat); - - reg_resp.repeat =3D head.repeat; - registers =3D (RDMARegister *) rdma->wr_data[idx].control_curr; - - for (int count =3D 0; count < head.repeat; count++) { - uint64_t chunk; - uint8_t *chunk_start, *chunk_end; - - reg =3D ®isters[count]; - network_to_register(reg); - - reg_result =3D &results[count]; - - trace_rdma_registration_handle_register_loop(count, - reg->current_index, reg->key.current_addr, reg->c= hunks); - - if (reg->current_index >=3D rdma->local_ram_blocks.nb_bloc= ks) { - error_report("rdma: 'register' bad block index %u (vs = %d)", - (unsigned int)reg->current_index, - rdma->local_ram_blocks.nb_blocks); - goto err; - } - block =3D &(rdma->local_ram_blocks.block[reg->current_inde= x]); - if (block->is_ram_block) { - if (block->offset > reg->key.current_addr) { - error_report("rdma: bad register address for block= %s" - " offset: %" PRIx64 " current_addr: %" PRIx64, - block->block_name, block->offset, - reg->key.current_addr); - goto err; - } - host_addr =3D (block->local_host_addr + - (reg->key.current_addr - block->offset)); - chunk =3D ram_chunk_index(block->local_host_addr, - (uint8_t *) host_addr); - } else { - chunk =3D reg->key.chunk; - host_addr =3D block->local_host_addr + - (reg->key.chunk * (1UL << RDMA_REG_CHUNK_SHIFT)); - /* Check for particularly bad chunk value */ - if (host_addr < (void *)block->local_host_addr) { - error_report("rdma: bad chunk for block %s" - " chunk: %" PRIx64, - block->block_name, reg->key.chunk); - goto err; - } - } - chunk_start =3D ram_chunk_start(block, chunk); - chunk_end =3D ram_chunk_end(block, chunk + reg->chunks); - /* avoid "-Waddress-of-packed-member" warning */ - uint32_t tmp_rkey =3D 0; - if (qemu_rdma_register_and_get_keys(rdma, block, - (uintptr_t)host_addr, NULL, &tmp_rkey, - chunk, chunk_start, chunk_end)) { - error_report("cannot get rkey"); - goto err; - } - reg_result->rkey =3D tmp_rkey; - - reg_result->host_addr =3D (uintptr_t)block->local_host_add= r; - - trace_rdma_registration_handle_register_rkey(reg_result->r= key); - - result_to_network(reg_result); - } - - ret =3D qemu_rdma_post_send_control(rdma, - (uint8_t *) results, ®_resp, &err); - - if (ret < 0) { - error_report_err(err); - goto err; - } - break; - case RDMA_CONTROL_UNREGISTER_REQUEST: - trace_rdma_registration_handle_unregister(head.repeat); - unreg_resp.repeat =3D head.repeat; - registers =3D (RDMARegister *) rdma->wr_data[idx].control_curr; - - for (int count =3D 0; count < head.repeat; count++) { - reg =3D ®isters[count]; - network_to_register(reg); - - trace_rdma_registration_handle_unregister_loop(count, - reg->current_index, reg->key.chunk); - - block =3D &(rdma->local_ram_blocks.block[reg->current_inde= x]); - - ret =3D ibv_dereg_mr(block->pmr[reg->key.chunk]); - block->pmr[reg->key.chunk] =3D NULL; - - if (ret !=3D 0) { - error_report("rdma unregistration chunk failed: %s", - strerror(errno)); - goto err; - } - - rdma->total_registrations--; - - trace_rdma_registration_handle_unregister_success(reg->key= .chunk); - } - - ret =3D qemu_rdma_post_send_control(rdma, NULL, &unreg_resp, &= err); - - if (ret < 0) { - error_report_err(err); - goto err; - } - break; - case RDMA_CONTROL_REGISTER_RESULT: - error_report("Invalid RESULT message at dest."); - goto err; - default: - error_report("Unknown control message %s", control_desc(head.t= ype)); - goto err; - } - } while (1); - -err: - rdma->errored =3D true; - return -1; -} - -/* Destination: - * Called during the initial RAM load section which lists the - * RAMBlocks by name. This lets us know the order of the RAMBlocks on - * the source. We've already built our local RAMBlock list, but not - * yet sent the list to the source. - */ -int rdma_block_notification_handle(QEMUFile *f, const char *name) -{ - int curr; - int found =3D -1; - - if (!migrate_rdma()) { - return 0; - } - - RCU_READ_LOCK_GUARD(); - QIOChannelRDMA *rioc =3D QIO_CHANNEL_RDMA(qemu_file_get_ioc(f)); - RDMAContext *rdma =3D qatomic_rcu_read(&rioc->rdmain); - - if (!rdma) { - return -1; - } - - /* Find the matching RAMBlock in our local list */ - for (curr =3D 0; curr < rdma->local_ram_blocks.nb_blocks; curr++) { - if (!strcmp(rdma->local_ram_blocks.block[curr].block_name, name)) { - found =3D curr; - break; - } - } - - if (found =3D=3D -1) { - error_report("RAMBlock '%s' not found on destination", name); - return -1; - } - - rdma->local_ram_blocks.block[curr].src_index =3D rdma->next_src_index; - trace_rdma_block_notification_handle(name, rdma->next_src_index); - rdma->next_src_index++; - - return 0; -} - -int rdma_registration_start(QEMUFile *f, uint64_t flags) -{ - if (!migrate_rdma() || migration_in_postcopy()) { - return 0; - } - - QIOChannelRDMA *rioc =3D QIO_CHANNEL_RDMA(qemu_file_get_ioc(f)); - RCU_READ_LOCK_GUARD(); - RDMAContext *rdma =3D qatomic_rcu_read(&rioc->rdmaout); - if (!rdma) { - return -1; - } - - if (rdma_errored(rdma)) { - return -1; - } - - trace_rdma_registration_start(flags); - qemu_put_be64(f, RAM_SAVE_FLAG_HOOK); - return qemu_fflush(f); -} - -/* - * Inform dest that dynamic registrations are done for now. - * First, flush writes, if any. - */ -int rdma_registration_stop(QEMUFile *f, uint64_t flags) -{ - QIOChannelRDMA *rioc; - Error *err =3D NULL; - RDMAContext *rdma; - RDMAControlHeader head =3D { .len =3D 0, .repeat =3D 1 }; - int ret; - - if (!migrate_rdma() || migration_in_postcopy()) { - return 0; - } - - RCU_READ_LOCK_GUARD(); - rioc =3D QIO_CHANNEL_RDMA(qemu_file_get_ioc(f)); - rdma =3D qatomic_rcu_read(&rioc->rdmaout); - if (!rdma) { - return -1; - } - - if (rdma_errored(rdma)) { - return -1; - } - - qemu_fflush(f); - ret =3D qemu_rdma_drain_cq(rdma); - - if (ret < 0) { - goto err; - } - - if (flags =3D=3D RAM_CONTROL_SETUP) { - RDMAControlHeader resp =3D {.type =3D RDMA_CONTROL_RAM_BLOCKS_RESU= LT }; - RDMALocalBlocks *local =3D &rdma->local_ram_blocks; - int reg_result_idx, nb_dest_blocks; - - head.type =3D RDMA_CONTROL_RAM_BLOCKS_REQUEST; - trace_rdma_registration_stop_ram(); - - /* - * Make sure that we parallelize the pinning on both sides. - * For very large guests, doing this serially takes a really - * long time, so we have to 'interleave' the pinning locally - * with the control messages by performing the pinning on this - * side before we receive the control response from the other - * side that the pinning has completed. - */ - ret =3D qemu_rdma_exchange_send(rdma, &head, NULL, &resp, - ®_result_idx, rdma->pin_all ? - qemu_rdma_reg_whole_ram_blocks : NULL, - &err); - if (ret < 0) { - error_report_err(err); - return -1; - } - - nb_dest_blocks =3D resp.len / sizeof(RDMADestBlock); - - /* - * The protocol uses two different sets of rkeys (mutually exclusi= ve): - * 1. One key to represent the virtual address of the entire ram b= lock. - * (dynamic chunk registration disabled - pin everything with o= ne rkey.) - * 2. One to represent individual chunks within a ram block. - * (dynamic chunk registration enabled - pin individual chunks.) - * - * Once the capability is successfully negotiated, the destination= transmits - * the keys to use (or sends them later) including the virtual add= resses - * and then propagates the remote ram block descriptions to his lo= cal copy. - */ - - if (local->nb_blocks !=3D nb_dest_blocks) { - error_report("ram blocks mismatch (Number of blocks %d vs %d)", - local->nb_blocks, nb_dest_blocks); - error_printf("Your QEMU command line parameters are probably " - "not identical on both the source and destination= ."); - rdma->errored =3D true; - return -1; - } - - qemu_rdma_move_header(rdma, reg_result_idx, &resp); - memcpy(rdma->dest_blocks, - rdma->wr_data[reg_result_idx].control_curr, resp.len); - for (int i =3D 0; i < nb_dest_blocks; i++) { - network_to_dest_block(&rdma->dest_blocks[i]); - - /* We require that the blocks are in the same order */ - if (rdma->dest_blocks[i].length !=3D local->block[i].length) { - error_report("Block %s/%d has a different length %" PRIu64 - "vs %" PRIu64, - local->block[i].block_name, i, - local->block[i].length, - rdma->dest_blocks[i].length); - rdma->errored =3D true; - return -1; - } - local->block[i].remote_host_addr =3D - rdma->dest_blocks[i].remote_host_addr; - local->block[i].remote_rkey =3D rdma->dest_blocks[i].remote_rk= ey; - } - } - - trace_rdma_registration_stop(flags); - - head.type =3D RDMA_CONTROL_REGISTER_FINISHED; - ret =3D qemu_rdma_exchange_send(rdma, &head, NULL, NULL, NULL, NULL, &= err); - - if (ret < 0) { - error_report_err(err); - goto err; - } - - return 0; -err: - rdma->errored =3D true; - return -1; -} - -static void qio_channel_rdma_finalize(Object *obj) -{ - QIOChannelRDMA *rioc =3D QIO_CHANNEL_RDMA(obj); - if (rioc->rdmain) { - qemu_rdma_cleanup(rioc->rdmain); - g_free(rioc->rdmain); - rioc->rdmain =3D NULL; - } - if (rioc->rdmaout) { - qemu_rdma_cleanup(rioc->rdmaout); - g_free(rioc->rdmaout); - rioc->rdmaout =3D NULL; - } -} - -static void qio_channel_rdma_class_init(ObjectClass *klass, - void *class_data G_GNUC_UNUSED) -{ - QIOChannelClass *ioc_klass =3D QIO_CHANNEL_CLASS(klass); - - ioc_klass->io_writev =3D qio_channel_rdma_writev; - ioc_klass->io_readv =3D qio_channel_rdma_readv; - ioc_klass->io_set_blocking =3D qio_channel_rdma_set_blocking; - ioc_klass->io_close =3D qio_channel_rdma_close; - ioc_klass->io_create_watch =3D qio_channel_rdma_create_watch; - ioc_klass->io_set_aio_fd_handler =3D qio_channel_rdma_set_aio_fd_handl= er; - ioc_klass->io_shutdown =3D qio_channel_rdma_shutdown; -} - -static const TypeInfo qio_channel_rdma_info =3D { - .parent =3D TYPE_QIO_CHANNEL, - .name =3D TYPE_QIO_CHANNEL_RDMA, - .instance_size =3D sizeof(QIOChannelRDMA), - .instance_finalize =3D qio_channel_rdma_finalize, - .class_init =3D qio_channel_rdma_class_init, -}; - -static void qio_channel_rdma_register_types(void) -{ - type_register_static(&qio_channel_rdma_info); -} - -type_init(qio_channel_rdma_register_types); - -static QEMUFile *rdma_new_input(RDMAContext *rdma) -{ - QIOChannelRDMA *rioc =3D QIO_CHANNEL_RDMA(object_new(TYPE_QIO_CHANNEL_= RDMA)); - - rioc->file =3D qemu_file_new_input(QIO_CHANNEL(rioc)); - rioc->rdmain =3D rdma; - rioc->rdmaout =3D rdma->return_path; - - return rioc->file; -} - -static QEMUFile *rdma_new_output(RDMAContext *rdma) -{ - QIOChannelRDMA *rioc =3D QIO_CHANNEL_RDMA(object_new(TYPE_QIO_CHANNEL_= RDMA)); - - rioc->file =3D qemu_file_new_output(QIO_CHANNEL(rioc)); - rioc->rdmaout =3D rdma; - rioc->rdmain =3D rdma->return_path; - - return rioc->file; -} - -static void rdma_accept_incoming_migration(void *opaque) -{ - RDMAContext *rdma =3D opaque; - QEMUFile *f; - - trace_qemu_rdma_accept_incoming_migration(); - if (qemu_rdma_accept(rdma) < 0) { - error_report("RDMA ERROR: Migration initialization failed"); - return; - } - - trace_qemu_rdma_accept_incoming_migration_accepted(); - - if (rdma->is_return_path) { - return; - } - - f =3D rdma_new_input(rdma); - if (f =3D=3D NULL) { - error_report("RDMA ERROR: could not open RDMA for input"); - qemu_rdma_cleanup(rdma); - return; - } - - rdma->migration_started_on_destination =3D 1; - migration_fd_process_incoming(f); -} - -void rdma_start_incoming_migration(InetSocketAddress *host_port, - Error **errp) -{ - MigrationState *s =3D migrate_get_current(); - int ret; - RDMAContext *rdma; - - trace_rdma_start_incoming_migration(); - - /* Avoid ram_block_discard_disable(), cannot change during migration. = */ - if (ram_block_discard_is_required()) { - error_setg(errp, "RDMA: cannot disable RAM discard"); - return; - } - - rdma =3D qemu_rdma_data_init(host_port, errp); - if (rdma =3D=3D NULL) { - goto err; - } - - ret =3D qemu_rdma_dest_init(rdma, errp); - if (ret < 0) { - goto err; - } - - trace_rdma_start_incoming_migration_after_dest_init(); - - ret =3D rdma_listen(rdma->listen_id, 5); - - if (ret < 0) { - error_setg(errp, "RDMA ERROR: listening on socket!"); - goto cleanup_rdma; - } - - trace_rdma_start_incoming_migration_after_rdma_listen(); - s->rdma_migration =3D true; - qemu_set_fd_handler(rdma->channel->fd, rdma_accept_incoming_migration, - NULL, (void *)(intptr_t)rdma); - return; - -cleanup_rdma: - qemu_rdma_cleanup(rdma); -err: - if (rdma) { - g_free(rdma->host); - } - g_free(rdma); -} - -void rdma_start_outgoing_migration(void *opaque, - InetSocketAddress *host_port, Error **errp) -{ - MigrationState *s =3D opaque; - RDMAContext *rdma_return_path =3D NULL; - RDMAContext *rdma; - int ret; - - /* Avoid ram_block_discard_disable(), cannot change during migration. = */ - if (ram_block_discard_is_required()) { - error_setg(errp, "RDMA: cannot disable RAM discard"); - return; - } - - rdma =3D qemu_rdma_data_init(host_port, errp); - if (rdma =3D=3D NULL) { - goto err; - } - - ret =3D qemu_rdma_source_init(rdma, migrate_rdma_pin_all(), errp); - - if (ret < 0) { - goto err; - } - - trace_rdma_start_outgoing_migration_after_rdma_source_init(); - ret =3D qemu_rdma_connect(rdma, false, errp); - - if (ret < 0) { - goto err; - } - - /* RDMA postcopy need a separate queue pair for return path */ - if (migrate_postcopy() || migrate_return_path()) { - rdma_return_path =3D qemu_rdma_data_init(host_port, errp); - - if (rdma_return_path =3D=3D NULL) { - goto return_path_err; - } - - ret =3D qemu_rdma_source_init(rdma_return_path, - migrate_rdma_pin_all(), errp); - - if (ret < 0) { - goto return_path_err; - } - - ret =3D qemu_rdma_connect(rdma_return_path, true, errp); - - if (ret < 0) { - goto return_path_err; - } - - rdma->return_path =3D rdma_return_path; - rdma_return_path->return_path =3D rdma; - rdma_return_path->is_return_path =3D true; - } - - trace_rdma_start_outgoing_migration_after_rdma_connect(); - - s->to_dst_file =3D rdma_new_output(rdma); - s->rdma_migration =3D true; - migrate_fd_connect(s, NULL); - return; -return_path_err: - qemu_rdma_cleanup(rdma); -err: - g_free(rdma); - g_free(rdma_return_path); -} diff --git a/migration/rdma.h b/migration/rdma.h deleted file mode 100644 index a8d27f33b8..0000000000 --- a/migration/rdma.h +++ /dev/null @@ -1,69 +0,0 @@ -/* - * RDMA protocol and interfaces - * - * Copyright IBM, Corp. 2010-2013 - * Copyright Red Hat, Inc. 2015-2016 - * - * Authors: - * Michael R. Hines - * Jiuxing Liu - * Daniel P. Berrange - * - * This work is licensed under the terms of the GNU GPL, version 2 or - * later. See the COPYING file in the top-level directory. - * - */ - -#include "qemu/sockets.h" - -#ifndef QEMU_MIGRATION_RDMA_H -#define QEMU_MIGRATION_RDMA_H - -#include "exec/memory.h" - -void rdma_start_outgoing_migration(void *opaque, InetSocketAddress *host_p= ort, - Error **errp); - -void rdma_start_incoming_migration(InetSocketAddress *host_port, Error **e= rrp); - -/* - * Constants used by rdma return codes - */ -#define RAM_CONTROL_SETUP 0 -#define RAM_CONTROL_ROUND 1 -#define RAM_CONTROL_FINISH 3 - -/* - * Whenever this is found in the data stream, the flags - * will be passed to rdma functions in the incoming-migration - * side. - */ -#define RAM_SAVE_FLAG_HOOK 0x80 - -#define RAM_SAVE_CONTROL_NOT_SUPP -1000 -#define RAM_SAVE_CONTROL_DELAYED -2000 - -#ifdef CONFIG_RDMA -int rdma_registration_handle(QEMUFile *f); -int rdma_registration_start(QEMUFile *f, uint64_t flags); -int rdma_registration_stop(QEMUFile *f, uint64_t flags); -int rdma_block_notification_handle(QEMUFile *f, const char *name); -int rdma_control_save_page(QEMUFile *f, ram_addr_t block_offset, - ram_addr_t offset, size_t size); -#else -static inline -int rdma_registration_handle(QEMUFile *f) { return 0; } -static inline -int rdma_registration_start(QEMUFile *f, uint64_t flags) { return 0; } -static inline -int rdma_registration_stop(QEMUFile *f, uint64_t flags) { return 0; } -static inline -int rdma_block_notification_handle(QEMUFile *f, const char *name) { return= 0; } -static inline -int rdma_control_save_page(QEMUFile *f, ram_addr_t block_offset, - ram_addr_t offset, size_t size) -{ - return RAM_SAVE_CONTROL_NOT_SUPP; -} -#endif -#endif diff --git a/migration/savevm.c b/migration/savevm.c index c621f2359b..3941d65693 100644 --- a/migration/savevm.c +++ b/migration/savevm.c @@ -2996,7 +2996,7 @@ int qemu_loadvm_state(QEMUFile *f) =20 /* We've got to be careful; if we don't read the data and just shut th= e fd * then the sender can error if we close while it's still sending. - * We also mustn't read data that isn't there; some transports (RDMA) + * We also mustn't read data that isn't there; some transports * will stall waiting for that data when the source has already closed. */ if (ret =3D=3D 0 && should_send_vmdesc()) { diff --git a/migration/trace-events b/migration/trace-events index 0b7c3324fb..72e0517f09 100644 --- a/migration/trace-events +++ b/migration/trace-events @@ -193,7 +193,7 @@ process_incoming_migration_co_postcopy_end_main(void) "" postcopy_preempt_enabled(bool value) "%d" =20 # migration-stats -migration_transferred_bytes(uint64_t qemu_file, uint64_t multifd, uint64_t= rdma) "qemu_file %" PRIu64 " multifd %" PRIu64 " RDMA %" PRIu64 +migration_transferred_bytes(uint64_t qemu_file, uint64_t multifd) "qemu_fi= le %" PRIu64 " multifd %" PRIu64 =20 # channel.c migration_set_incoming_channel(void *ioc, const char *ioctype) "ioc=3D%p i= octype=3D%s" @@ -204,72 +204,6 @@ migrate_state_too_big(void) "" migrate_global_state_post_load(const char *state) "loaded state: %s" migrate_global_state_pre_save(const char *state) "saved state: %s" =20 -# rdma.c -qemu_rdma_accept_incoming_migration(void) "" -qemu_rdma_accept_incoming_migration_accepted(void) "" -qemu_rdma_accept_pin_state(bool pin) "%d" -qemu_rdma_accept_pin_verbsc(void *verbs) "Verbs context after listen: %p" -qemu_rdma_block_for_wrid_miss(uint64_t wcomp, uint64_t req) "A Wanted wrid= %" PRIu64 " but got %" PRIu64 -qemu_rdma_cleanup_disconnect(void) "" -qemu_rdma_close(void) "" -qemu_rdma_connect_pin_all_requested(void) "" -qemu_rdma_connect_pin_all_outcome(bool pin) "%d" -qemu_rdma_dest_init_trying(const char *host, const char *ip) "%s =3D> %s" -qemu_rdma_dump_id_failed(const char *who) "%s RDMA Device opened, but can'= t query port information" -qemu_rdma_dump_id(const char *who, const char *name, const char *dev_name,= const char *dev_path, const char *ibdev_path, int transport, const char *t= ransport_name) "%s RDMA Device opened: kernel name %s uverbs device name %s= , infiniband_verbs class device path %s, infiniband class device path %s, t= ransport: (%d) %s" -qemu_rdma_dump_gid(const char *who, const char *src, const char *dst) "%s = Source GID: %s, Dest GID: %s" -qemu_rdma_exchange_get_response_start(const char *desc) "CONTROL: %s recei= ving..." -qemu_rdma_exchange_get_response_none(const char *desc, int type) "Surprise= : got %s (%d)" -qemu_rdma_exchange_send_issue_callback(void) "" -qemu_rdma_exchange_send_waiting(const char *desc) "Waiting for response %s" -qemu_rdma_exchange_send_received(const char *desc) "Response %s received." -qemu_rdma_fill(size_t control_len, size_t size) "RDMA %zd of %zd bytes alr= eady in buffer" -qemu_rdma_init_ram_blocks(int blocks) "Allocated %d local ram block struct= ures" -qemu_rdma_poll_recv(uint64_t comp, int64_t id, int sent) "completion %" PR= Iu64 " received (%" PRId64 ") left %d" -qemu_rdma_poll_write(uint64_t comp, int left, uint64_t block, uint64_t chu= nk, void *local, void *remote) "completions %" PRIu64 " left %d, block %" P= RIu64 ", chunk: %" PRIu64 " %p %p" -qemu_rdma_poll_other(uint64_t comp, int left) "other completion %" PRIu64 = " received left %d" -qemu_rdma_post_send_control(const char *desc) "CONTROL: sending %s.." -qemu_rdma_register_and_get_keys(uint64_t len, void *start) "Registering %"= PRIu64 " bytes @ %p" -qemu_rdma_register_odp_mr(const char *name) "Try to register On-Demand Pag= ing memory region: %s" -qemu_rdma_advise_mr(const char *name, uint32_t len, uint64_t addr, const c= har *res) "Try to advise block %s prefetch at %" PRIu32 "@0x%" PRIx64 ": %s" -qemu_rdma_resolve_host_trying(const char *host, const char *ip) "Trying %s= =3D> %s" -qemu_rdma_signal_unregister_append(uint64_t chunk, int pos) "Appending unr= egister chunk %" PRIu64 " at position %d" -qemu_rdma_signal_unregister_already(uint64_t chunk) "Unregister chunk %" P= RIu64 " already in queue" -qemu_rdma_unregister_waiting_inflight(uint64_t chunk) "Cannot unregister i= nflight chunk: %" PRIu64 -qemu_rdma_unregister_waiting_proc(uint64_t chunk, int pos) "Processing unr= egister for chunk: %" PRIu64 " at position %d" -qemu_rdma_unregister_waiting_send(uint64_t chunk) "Sending unregister for = chunk: %" PRIu64 -qemu_rdma_unregister_waiting_complete(uint64_t chunk) "Unregister for chun= k: %" PRIu64 " complete." -qemu_rdma_write_flush(int sent) "sent total: %d" -qemu_rdma_write_one_block(int count, int block, uint64_t chunk, uint64_t c= urrent, uint64_t len, int nb_sent, int nb_chunks) "(%d) Not clobbering: blo= ck: %d chunk %" PRIu64 " current %" PRIu64 " len %" PRIu64 " %d %d" -qemu_rdma_write_one_post(uint64_t chunk, long addr, long remote, uint32_t = len) "Posting chunk: %" PRIu64 ", addr: 0x%lx remote: 0x%lx, bytes %" PRIu32 -qemu_rdma_write_one_queue_full(void) "" -qemu_rdma_write_one_recvregres(int mykey, int theirkey, uint64_t chunk) "R= eceived registration result: my key: 0x%x their key 0x%x, chunk %" PRIu64 -qemu_rdma_write_one_sendreg(uint64_t chunk, int len, int index, int64_t of= fset) "Sending registration request chunk %" PRIu64 " for %d bytes, index: = %d, offset: %" PRId64 -qemu_rdma_write_one_top(uint64_t chunks, uint64_t size) "Writing %" PRIu64= " chunks, (%" PRIu64 " MB)" -qemu_rdma_write_one_zero(uint64_t chunk, int len, int index, int64_t offse= t) "Entire chunk is zero, sending compress: %" PRIu64 " for %d bytes, index= : %d, offset: %" PRId64 -rdma_add_block(const char *block_name, int block, uint64_t addr, uint64_t = offset, uint64_t len, uint64_t end, uint64_t bits, int chunks) "Added Block= : '%s':%d, addr: %" PRIu64 ", offset: %" PRIu64 " length: %" PRIu64 " end: = %" PRIu64 " bits %" PRIu64 " chunks %d" -rdma_block_notification_handle(const char *name, int index) "%s at %d" -rdma_delete_block(void *block, uint64_t addr, uint64_t offset, uint64_t le= n, uint64_t end, uint64_t bits, int chunks) "Deleted Block: %p, addr: %" PR= Iu64 ", offset: %" PRIu64 " length: %" PRIu64 " end: %" PRIu64 " bits %" PR= Iu64 " chunks %d" -rdma_registration_handle_compress(int64_t length, int index, int64_t offse= t) "Zapping zero chunk: %" PRId64 " bytes, index %d, offset %" PRId64 -rdma_registration_handle_finished(void) "" -rdma_registration_handle_ram_blocks(void) "" -rdma_registration_handle_ram_blocks_loop(const char *name, uint64_t offset= , uint64_t length, void *local_host_addr, unsigned int src_index) "%s: @0x%= " PRIx64 "/%" PRIu64 " host:@%p src_index: %u" -rdma_registration_handle_register(int requests) "%d requests" -rdma_registration_handle_register_loop(int req, int index, uint64_t addr, = uint64_t chunks) "Registration request (%d): index %d, current_addr %" PRIu= 64 " chunks: %" PRIu64 -rdma_registration_handle_register_rkey(int rkey) "0x%x" -rdma_registration_handle_unregister(int requests) "%d requests" -rdma_registration_handle_unregister_loop(int count, int index, uint64_t ch= unk) "Unregistration request (%d): index %d, chunk %" PRIu64 -rdma_registration_handle_unregister_success(uint64_t chunk) "%" PRIu64 -rdma_registration_handle_wait(void) "" -rdma_registration_start(uint64_t flags) "%" PRIu64 -rdma_registration_stop(uint64_t flags) "%" PRIu64 -rdma_registration_stop_ram(void) "" -rdma_start_incoming_migration(void) "" -rdma_start_incoming_migration_after_dest_init(void) "" -rdma_start_incoming_migration_after_rdma_listen(void) "" -rdma_start_outgoing_migration_after_rdma_connect(void) "" -rdma_start_outgoing_migration_after_rdma_source_init(void) "" - # postcopy-ram.c postcopy_discard_send_finish(const char *ramblock, int nwords, int ncmds) = "%s mask words sent=3D%d in %d commands" postcopy_discard_send_range(const char *ramblock, unsigned long start, uns= igned long length) "%s:%lx/%lx" diff --git a/qapi/migration.json b/qapi/migration.json index a351fd3714..4d7d49bfec 100644 --- a/qapi/migration.json +++ b/qapi/migration.json @@ -210,9 +210,9 @@ # # @setup-time: amount of setup time in milliseconds *before* the # iterations begin but *after* the QMP command is issued. This is -# designed to provide an accounting of any activities (such as -# RDMA pinning) which may be expensive, but do not actually occur -# during the iterative migration rounds themselves. (since 1.6) +# designed to provide an accounting of any activities which may be +# expensive, but do not actually occur during the iterative migration +# rounds themselves. (since 1.6) # # @cpu-throttle-percentage: percentage of time guest cpus are being # throttled during auto-converge. This is only present when @@ -378,10 +378,6 @@ # for certain work loads, by sending compressed difference of the # pages # -# @rdma-pin-all: Controls whether or not the entire VM memory -# footprint is mlock()'d on demand or all at once. Refer to -# docs/rdma.txt for usage. Disabled by default. (since 2.0) -# # @zero-blocks: During storage migration encode blocks of zeroes # efficiently. This essentially saves 1MB of zeroes per block on # the wire. Enabling requires source and target VM to support @@ -476,7 +472,7 @@ # Since: 1.2 ## { 'enum': 'MigrationCapability', - 'data': ['xbzrle', 'rdma-pin-all', 'auto-converge', 'zero-blocks', + 'data': ['xbzrle', 'auto-converge', 'zero-blocks', 'events', 'postcopy-ram', { 'name': 'x-colo', 'features': [ 'unstable' ] }, 'release-ram', @@ -533,7 +529,6 @@ # -> { "execute": "query-migrate-capabilities" } # <- { "return": [ # {"state": false, "capability": "xbzrle"}, -# {"state": false, "capability": "rdma-pin-all"}, # {"state": false, "capability": "auto-converge"}, # {"state": false, "capability": "zero-blocks"}, # {"state": true, "capability": "events"}, diff --git a/scripts/analyze-migration.py b/scripts/analyze-migration.py index 8a254a5b6a..70e77622d3 100755 --- a/scripts/analyze-migration.py +++ b/scripts/analyze-migration.py @@ -110,7 +110,6 @@ class RamSection(object): RAM_SAVE_FLAG_EOS =3D 0x10 RAM_SAVE_FLAG_CONTINUE =3D 0x20 RAM_SAVE_FLAG_XBZRLE =3D 0x40 - RAM_SAVE_FLAG_HOOK =3D 0x80 RAM_SAVE_FLAG_COMPRESS_PAGE =3D 0x100 RAM_SAVE_FLAG_MULTIFD_FLUSH =3D 0x200 =20 @@ -203,8 +202,6 @@ def read(self): flags &=3D ~self.RAM_SAVE_FLAG_PAGE elif flags & self.RAM_SAVE_FLAG_XBZRLE: raise Exception("XBZRLE RAM compression is not supported y= et") - elif flags & self.RAM_SAVE_FLAG_HOOK: - raise Exception("RAM hooks don't make sense with files") if flags & self.RAM_SAVE_FLAG_MULTIFD_FLUSH: continue =20 --=20 2.43.0 From nobody Mon Nov 25 03:38:18 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1717503297; cv=none; d=zohomail.com; s=zohoarc; b=ZB7Gp50wPDEHgglZHxbog6/EhA7qxghcA0G1NojksgaJluWakkTVvDMNZzGCi8LVaJWEcJuRw/Vqx9J/035Zmhf+IJU/51nlrJiwMVe9/ovoNY6bPCXjFc/15VM9Qikh7YbhlF3d2H/XVZv2Rm7hX8lzYk0q0RAzIAJDE5pKSoM= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1717503297; h=Content-Type:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:Reply-To:Reply-To:References:Sender:Subject:Subject:To:To:Message-Id; bh=klqeD9bWTfGAUpZQV1OObBBU+YmFe5MuxCH0l3XzpFw=; b=mkE9ZvqhGR5EJs43s3JPseX8RffFq8zx2X+XQv8+Zk8+dXaXa2aZqiOLV7OZafTUdMibKziDgGQePrYfoJwDT5gkygwWtEcB6BVqYRaY9dNX8nkyhlZ7XE0lFKFCs8V9YuxQ0sKhWYMf43CRLdcR0Zw5d5ZyLjC8Fwrx5WfLJy0= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1717503297986516.7067440253988; Tue, 4 Jun 2024 05:14:57 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sET3n-0003hf-7K; Tue, 04 Jun 2024 08:14:39 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sET3k-0003fz-Lr for qemu-devel@nongnu.org; Tue, 04 Jun 2024 08:14:36 -0400 Received: from szxga03-in.huawei.com ([45.249.212.189]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sET3g-0006wW-3c for qemu-devel@nongnu.org; Tue, 04 Jun 2024 08:14:36 -0400 Received: from mail.maildlp.com (unknown [172.19.163.174]) by szxga03-in.huawei.com (SkyGuard) with ESMTP id 4VtqF41gQQzPpHv; Tue, 4 Jun 2024 20:11:04 +0800 (CST) Received: from dggpemf200006.china.huawei.com (unknown [7.185.36.61]) by mail.maildlp.com (Postfix) with ESMTPS id 97E5D14022E; Tue, 4 Jun 2024 20:14:20 +0800 (CST) Received: from DESKTOP-8LI8G6S.china.huawei.com (10.173.124.235) by dggpemf200006.china.huawei.com (7.185.36.61) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.11; Tue, 4 Jun 2024 20:14:19 +0800 To: CC: , , , , , , , , , , , , , , , Jialin Wang Subject: [PATCH 2/6] io: add QIOChannelRDMA class Date: Tue, 4 Jun 2024 20:14:08 +0800 Message-ID: <1717503252-51884-3-git-send-email-arei.gonglei@huawei.com> X-Mailer: git-send-email 2.8.2.windows.1 In-Reply-To: <1717503252-51884-1-git-send-email-arei.gonglei@huawei.com> References: <1717503252-51884-1-git-send-email-arei.gonglei@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.173.124.235] X-ClientProxiedBy: dggems704-chm.china.huawei.com (10.3.19.181) To dggpemf200006.china.huawei.com (7.185.36.61) Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=45.249.212.189; envelope-from=arei.gonglei@huawei.com; helo=szxga03-in.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Gonglei From: Gonglei via Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZM-MESSAGEID: 1717503300058100003 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Jialin Wang Implement a QIOChannelRDMA subclass that is based on the rsocket API (similar to socket API). Signed-off-by: Jialin Wang Signed-off-by: Gonglei --- include/io/channel-rdma.h | 152 +++++++++++++ io/channel-rdma.c | 437 ++++++++++++++++++++++++++++++++++++++ io/meson.build | 1 + io/trace-events | 14 ++ 4 files changed, 604 insertions(+) create mode 100644 include/io/channel-rdma.h create mode 100644 io/channel-rdma.c diff --git a/include/io/channel-rdma.h b/include/io/channel-rdma.h new file mode 100644 index 0000000000..8cab2459e5 --- /dev/null +++ b/include/io/channel-rdma.h @@ -0,0 +1,152 @@ +/* + * QEMU I/O channels RDMA driver + * + * Copyright (c) 2024 HUAWEI TECHNOLOGIES CO., LTD. + * + * Authors: + * Jialin Wang + * Gonglei + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library; if not, see . + */ + +#ifndef QIO_CHANNEL_RDMA_H +#define QIO_CHANNEL_RDMA_H + +#include "io/channel.h" +#include "io/task.h" +#include "qemu/sockets.h" +#include "qom/object.h" + +#define TYPE_QIO_CHANNEL_RDMA "qio-channel-rdma" +OBJECT_DECLARE_SIMPLE_TYPE(QIOChannelRDMA, QIO_CHANNEL_RDMA) + +/** + * QIOChannelRDMA: + * + * The QIOChannelRDMA object provides a channel implementation + * that discards all writes and returns EOF for all reads. + */ +struct QIOChannelRDMA { + QIOChannel parent; + /* the rsocket fd */ + int fd; + + struct sockaddr_storage localAddr; + socklen_t localAddrLen; + struct sockaddr_storage remoteAddr; + socklen_t remoteAddrLen; +}; + +/** + * qio_channel_rdma_new: + * + * Create a channel for performing I/O on a rdma + * connection, that is initially closed. After + * creating the rdma, it must be setup as a client + * connection or server. + * + * Returns: the rdma channel object + */ +QIOChannelRDMA *qio_channel_rdma_new(void); + +/** + * qio_channel_rdma_connect_sync: + * @ioc: the rdma channel object + * @addr: the address to connect to + * @errp: pointer to a NULL-initialized error object + * + * Attempt to connect to the address @addr. This method + * will run in the foreground so the caller will not regain + * execution control until the connection is established or + * an error occurs. + */ +int qio_channel_rdma_connect_sync(QIOChannelRDMA *ioc, InetSocketAddress *= addr, + Error **errp); + +/** + * qio_channel_rdma_connect_async: + * @ioc: the rdma channel object + * @addr: the address to connect to + * @callback: the function to invoke on completion + * @opaque: user data to pass to @callback + * @destroy: the function to free @opaque + * @context: the context to run the async task. If %NULL, the default + * context will be used. + * + * Attempt to connect to the address @addr. This method + * will run in the background so the caller will regain + * execution control immediately. The function @callback + * will be invoked on completion or failure. The @addr + * parameter will be copied, so may be freed as soon + * as this function returns without waiting for completion. + */ +void qio_channel_rdma_connect_async(QIOChannelRDMA *ioc, + InetSocketAddress *addr, + QIOTaskFunc callback, gpointer opaque, + GDestroyNotify destroy, + GMainContext *context); + +/** + * qio_channel_rdma_listen_sync: + * @ioc: the rdma channel object + * @addr: the address to listen to + * @num: the expected amount of connections + * @errp: pointer to a NULL-initialized error object + * + * Attempt to listen to the address @addr. This method + * will run in the foreground so the caller will not regain + * execution control until the connection is established or + * an error occurs. + */ +int qio_channel_rdma_listen_sync(QIOChannelRDMA *ioc, InetSocketAddress *a= ddr, + int num, Error **errp); + +/** + * qio_channel_rdma_listen_async: + * @ioc: the rdma channel object + * @addr: the address to listen to + * @num: the expected amount of connections + * @callback: the function to invoke on completion + * @opaque: user data to pass to @callback + * @destroy: the function to free @opaque + * @context: the context to run the async task. If %NULL, the default + * context will be used. + * + * Attempt to listen to the address @addr. This method + * will run in the background so the caller will regain + * execution control immediately. The function @callback + * will be invoked on completion or failure. The @addr + * parameter will be copied, so may be freed as soon + * as this function returns without waiting for completion. + */ +void qio_channel_rdma_listen_async(QIOChannelRDMA *ioc, InetSocketAddress = *addr, + int num, QIOTaskFunc callback, + gpointer opaque, GDestroyNotify destroy, + GMainContext *context); + +/** + * qio_channel_rdma_accept: + * @ioc: the rdma channel object + * @errp: pointer to a NULL-initialized error object + * + * If the rdma represents a server, then this accepts + * a new client connection. The returned channel will + * represent the connected client rdma. + * + * Returns: the new client channel, or NULL on error + */ +QIOChannelRDMA *qio_channel_rdma_accept(QIOChannelRDMA *ioc, Error **errp); + +#endif /* QIO_CHANNEL_RDMA_H */ diff --git a/io/channel-rdma.c b/io/channel-rdma.c new file mode 100644 index 0000000000..92c362df52 --- /dev/null +++ b/io/channel-rdma.c @@ -0,0 +1,437 @@ +/* + * QEMU I/O channels RDMA driver + * + * Copyright (c) 2024 HUAWEI TECHNOLOGIES CO., LTD. + * + * Authors: + * Jialin Wang + * Gonglei + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library; if not, see . + */ + +#include "qemu/osdep.h" +#include "io/channel-rdma.h" +#include "io/channel.h" +#include "qapi/clone-visitor.h" +#include "qapi/error.h" +#include "qapi/qapi-visit-sockets.h" +#include "trace.h" +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +QIOChannelRDMA *qio_channel_rdma_new(void) +{ + QIOChannelRDMA *rioc; + QIOChannel *ioc; + + rioc =3D QIO_CHANNEL_RDMA(object_new(TYPE_QIO_CHANNEL_RDMA)); + ioc =3D QIO_CHANNEL(rioc); + qio_channel_set_feature(ioc, QIO_CHANNEL_FEATURE_SHUTDOWN); + + trace_qio_channel_rdma_new(ioc); + + return rioc; +} + +static int qio_channel_rdma_set_fd(QIOChannelRDMA *rioc, int fd, Error **e= rrp) +{ + if (rioc->fd !=3D -1) { + error_setg(errp, "Socket is already open"); + return -1; + } + + rioc->fd =3D fd; + rioc->remoteAddrLen =3D sizeof(rioc->remoteAddr); + rioc->localAddrLen =3D sizeof(rioc->localAddr); + + if (rgetpeername(fd, (struct sockaddr *)&rioc->remoteAddr, + &rioc->remoteAddrLen) < 0) { + if (errno =3D=3D ENOTCONN) { + memset(&rioc->remoteAddr, 0, sizeof(rioc->remoteAddr)); + rioc->remoteAddrLen =3D sizeof(rioc->remoteAddr); + } else { + error_setg_errno(errp, errno, + "Unable to query remote rsocket address"); + goto error; + } + } + + if (rgetsockname(fd, (struct sockaddr *)&rioc->localAddr, + &rioc->localAddrLen) < 0) { + error_setg_errno(errp, errno, "Unable to query local rsocket addre= ss"); + goto error; + } + + return 0; + +error: + rioc->fd =3D -1; /* Let the caller close FD on failure */ + return -1; +} + +int qio_channel_rdma_connect_sync(QIOChannelRDMA *rioc, InetSocketAddress = *addr, + Error **errp) +{ + int ret, fd =3D -1; + struct rdma_addrinfo *ai; + + trace_qio_channel_rdma_connect_sync(rioc, addr); + ret =3D rdma_getaddrinfo(addr->host, addr->port, NULL, &ai); + if (ret) { + error_setg(errp, "Failed to rdma_getaddrinfo: %s", gai_strerror(re= t)); + goto out; + } + + fd =3D rsocket(ai->ai_family, SOCK_STREAM, 0); + if (fd < 0) { + error_setg_errno(errp, errno, "Failed to create rsocket"); + goto out; + } + qemu_set_cloexec(fd); + +retry: + ret =3D rconnect(fd, ai->ai_dst_addr, ai->ai_dst_len); + if (ret) { + if (errno =3D=3D EINTR) { + goto retry; + } + error_setg_errno(errp, errno, "Failed to rconnect"); + goto out; + } + + trace_qio_channel_rdma_connect_complete(rioc, fd); + ret =3D qio_channel_rdma_set_fd(rioc, fd, errp); + if (ret) { + goto out; + } + +out: + if (ret) { + trace_qio_channel_rdma_connect_fail(rioc); + if (fd >=3D 0) { + rclose(fd); + } + } + if (ai) { + rdma_freeaddrinfo(ai); + } + + return ret; +} + +static void qio_channel_rdma_connect_worker(QIOTask *task, gpointer opaque) +{ + QIOChannelRDMA *ioc =3D QIO_CHANNEL_RDMA(qio_task_get_source(task)); + InetSocketAddress *addr =3D opaque; + Error *err =3D NULL; + + qio_channel_rdma_connect_sync(ioc, addr, &err); + + qio_task_set_error(task, err); +} + +void qio_channel_rdma_connect_async(QIOChannelRDMA *ioc, + InetSocketAddress *addr, + QIOTaskFunc callback, gpointer opaque, + GDestroyNotify destroy, + GMainContext *context) +{ + QIOTask *task =3D qio_task_new(OBJECT(ioc), callback, opaque, destroy); + InetSocketAddress *addrCopy; + + addrCopy =3D QAPI_CLONE(InetSocketAddress, addr); + + /* rdma_getaddrinfo() blocks in DNS lookups, so we must use a thread */ + trace_qio_channel_rdma_connect_async(ioc, addr); + qio_task_run_in_thread(task, qio_channel_rdma_connect_worker, addrCopy, + (GDestroyNotify)qapi_free_InetSocketAddress, + context); +} + +int qio_channel_rdma_listen_sync(QIOChannelRDMA *rioc, InetSocketAddress *= addr, + int num, Error **errp) +{ + int ret, fd =3D -1; + struct rdma_addrinfo *ai; + struct rdma_addrinfo ai_hints =3D { 0 }; + + trace_qio_channel_rdma_listen_sync(rioc, addr, num); + ai_hints.ai_port_space =3D RDMA_PS_TCP; + ai_hints.ai_flags |=3D RAI_PASSIVE; + ret =3D rdma_getaddrinfo(addr->host, addr->port, &ai_hints, &ai); + if (ret) { + error_setg(errp, "Failed to rdma_getaddrinfo: %s", gai_strerror(re= t)); + goto out; + } + + fd =3D rsocket(ai->ai_family, SOCK_STREAM, 0); + if (fd < 0) { + error_setg_errno(errp, errno, "Failed to create rsocket"); + goto out; + } + qemu_set_cloexec(fd); + + ret =3D rbind(fd, ai->ai_src_addr, ai->ai_src_len); + if (ret) { + error_setg_errno(errp, errno, "Failed to rbind"); + goto out; + } + + ret =3D rlisten(fd, num); + if (ret) { + error_setg_errno(errp, errno, "Failed to rlisten"); + goto out; + } + + ret =3D qio_channel_rdma_set_fd(rioc, fd, errp); + if (ret) { + goto out; + } + + qio_channel_set_feature(QIO_CHANNEL(rioc), QIO_CHANNEL_FEATURE_LISTEN); + trace_qio_channel_rdma_listen_complete(rioc, fd); + +out: + if (ret) { + trace_qio_channel_rdma_listen_fail(rioc); + if (fd >=3D 0) { + rclose(fd); + } + } + if (ai) { + rdma_freeaddrinfo(ai); + } + + return ret; +} + +struct QIOChannelListenWorkerData { + InetSocketAddress *addr; + int num; /* amount of expected connections */ +}; + +static void qio_channel_listen_worker_free(gpointer opaque) +{ + struct QIOChannelListenWorkerData *data =3D opaque; + + qapi_free_InetSocketAddress(data->addr); + g_free(data); +} + +static void qio_channel_rdma_listen_worker(QIOTask *task, gpointer opaque) +{ + QIOChannelRDMA *ioc =3D QIO_CHANNEL_RDMA(qio_task_get_source(task)); + struct QIOChannelListenWorkerData *data =3D opaque; + Error *err =3D NULL; + + qio_channel_rdma_listen_sync(ioc, data->addr, data->num, &err); + + qio_task_set_error(task, err); +} + +void qio_channel_rdma_listen_async(QIOChannelRDMA *ioc, InetSocketAddress = *addr, + int num, QIOTaskFunc callback, + gpointer opaque, GDestroyNotify destroy, + GMainContext *context) +{ + QIOTask *task =3D qio_task_new(OBJECT(ioc), callback, opaque, destroy); + struct QIOChannelListenWorkerData *data; + + data =3D g_new0(struct QIOChannelListenWorkerData, 1); + data->addr =3D QAPI_CLONE(InetSocketAddress, addr); + data->num =3D num; + + /* rdma_getaddrinfo() blocks in DNS lookups, so we must use a thread */ + trace_qio_channel_rdma_listen_async(ioc, addr, num); + qio_task_run_in_thread(task, qio_channel_rdma_listen_worker, data, + qio_channel_listen_worker_free, context); +} + +QIOChannelRDMA *qio_channel_rdma_accept(QIOChannelRDMA *rioc, Error **errp) +{ + QIOChannelRDMA *cioc; + + cioc =3D qio_channel_rdma_new(); + cioc->remoteAddrLen =3D sizeof(rioc->remoteAddr); + cioc->localAddrLen =3D sizeof(rioc->localAddr); + + trace_qio_channel_rdma_accept(rioc); +retry: + cioc->fd =3D raccept(rioc->fd, (struct sockaddr *)&cioc->remoteAddr, + &cioc->remoteAddrLen); + if (cioc->fd < 0) { + if (errno =3D=3D EINTR) { + goto retry; + } + error_setg_errno(errp, errno, "Unable to accept connection"); + goto error; + } + qemu_set_cloexec(cioc->fd); + + if (rgetsockname(cioc->fd, (struct sockaddr *)&cioc->localAddr, + &cioc->localAddrLen) < 0) { + error_setg_errno(errp, errno, "Unable to query local rsocket addre= ss"); + goto error; + } + + trace_qio_channel_rdma_accept_complete(rioc, cioc, cioc->fd); + return cioc; + +error: + trace_qio_channel_rdma_accept_fail(rioc); + object_unref(OBJECT(cioc)); + return NULL; +} + +static void qio_channel_rdma_init(Object *obj) +{ + QIOChannelRDMA *ioc =3D QIO_CHANNEL_RDMA(obj); + ioc->fd =3D -1; +} + +static void qio_channel_rdma_finalize(Object *obj) +{ + QIOChannelRDMA *ioc =3D QIO_CHANNEL_RDMA(obj); + + if (ioc->fd !=3D -1) { + rclose(ioc->fd); + ioc->fd =3D -1; + } +} + +static ssize_t qio_channel_rdma_readv(QIOChannel *ioc, const struct iovec = *iov, + size_t niov, int **fds G_GNUC_UNUSED, + size_t *nfds G_GNUC_UNUSED, + int flags G_GNUC_UNUSED, Error **err= p) +{ + QIOChannelRDMA *rioc =3D QIO_CHANNEL_RDMA(ioc); + ssize_t ret; + +retry: + ret =3D rreadv(rioc->fd, iov, niov); + if (ret < 0) { + if (errno =3D=3D EINTR) { + goto retry; + } + error_setg_errno(errp, errno, "Unable to write to rsocket"); + return -1; + } + + return ret; +} + +static ssize_t qio_channel_rdma_writev(QIOChannel *ioc, const struct iovec= *iov, + size_t niov, int *fds G_GNUC_UNUSED, + size_t nfds G_GNUC_UNUSED, + int flags G_GNUC_UNUSED, Error **er= rp) +{ + QIOChannelRDMA *rioc =3D QIO_CHANNEL_RDMA(ioc); + ssize_t ret; + +retry: + ret =3D rwritev(rioc->fd, iov, niov); + if (ret <=3D 0) { + if (errno =3D=3D EINTR) { + goto retry; + } + error_setg_errno(errp, errno, "Unable to write to rsocket"); + return -1; + } + + return ret; +} + +static void qio_channel_rdma_set_delay(QIOChannel *ioc, bool enabled) +{ + QIOChannelRDMA *rioc =3D QIO_CHANNEL_RDMA(ioc); + int v =3D enabled ? 0 : 1; + + rsetsockopt(rioc->fd, IPPROTO_TCP, TCP_NODELAY, &v, sizeof(v)); +} + +static int qio_channel_rdma_close(QIOChannel *ioc, Error **errp) +{ + QIOChannelRDMA *rioc =3D QIO_CHANNEL_RDMA(ioc); + + if (rioc->fd !=3D -1) { + rclose(rioc->fd); + rioc->fd =3D -1; + } + + return 0; +} + +static int qio_channel_rdma_shutdown(QIOChannel *ioc, QIOChannelShutdown h= ow, + Error **errp) +{ + QIOChannelRDMA *rioc =3D QIO_CHANNEL_RDMA(ioc); + int sockhow; + + switch (how) { + case QIO_CHANNEL_SHUTDOWN_READ: + sockhow =3D SHUT_RD; + break; + case QIO_CHANNEL_SHUTDOWN_WRITE: + sockhow =3D SHUT_WR; + break; + case QIO_CHANNEL_SHUTDOWN_BOTH: + default: + sockhow =3D SHUT_RDWR; + break; + } + + if (rshutdown(rioc->fd, sockhow) < 0) { + error_setg_errno(errp, errno, "Unable to shutdown rsocket"); + return -1; + } + + return 0; +} + +static void qio_channel_rdma_class_init(ObjectClass *klass, + void *class_data G_GNUC_UNUSED) +{ + QIOChannelClass *ioc_klass =3D QIO_CHANNEL_CLASS(klass); + + ioc_klass->io_writev =3D qio_channel_rdma_writev; + ioc_klass->io_readv =3D qio_channel_rdma_readv; + ioc_klass->io_close =3D qio_channel_rdma_close; + ioc_klass->io_shutdown =3D qio_channel_rdma_shutdown; + ioc_klass->io_set_delay =3D qio_channel_rdma_set_delay; +} + +static const TypeInfo qio_channel_rdma_info =3D { + .parent =3D TYPE_QIO_CHANNEL, + .name =3D TYPE_QIO_CHANNEL_RDMA, + .instance_size =3D sizeof(QIOChannelRDMA), + .instance_init =3D qio_channel_rdma_init, + .instance_finalize =3D qio_channel_rdma_finalize, + .class_init =3D qio_channel_rdma_class_init, +}; + +static void qio_channel_rdma_register_types(void) +{ + type_register_static(&qio_channel_rdma_info); +} + +type_init(qio_channel_rdma_register_types); diff --git a/io/meson.build b/io/meson.build index 283b9b2bdb..e0dbd5183f 100644 --- a/io/meson.build +++ b/io/meson.build @@ -14,3 +14,4 @@ io_ss.add(files( 'net-listener.c', 'task.c', ), gnutls) +io_ss.add(when: rdma, if_true: files('channel-rdma.c')) diff --git a/io/trace-events b/io/trace-events index d4c0f84a9a..33026a2224 100644 --- a/io/trace-events +++ b/io/trace-events @@ -67,3 +67,17 @@ qio_channel_command_new_pid(void *ioc, int writefd, int = readfd, int pid) "Comman qio_channel_command_new_spawn(void *ioc, const char *binary, int flags) "C= ommand new spawn ioc=3D%p binary=3D%s flags=3D%d" qio_channel_command_abort(void *ioc, int pid) "Command abort ioc=3D%p pid= =3D%d" qio_channel_command_wait(void *ioc, int pid, int ret, int status) "Command= abort ioc=3D%p pid=3D%d ret=3D%d status=3D%d" + +# channel-rdma.c +qio_channel_rdma_new(void *ioc) "RDMA rsocket new ioc=3D%p" +qio_channel_rdma_connect_sync(void *ioc, void *addr) "RDMA rsocket connect= sync ioc=3D%p addr=3D%p" +qio_channel_rdma_connect_async(void *ioc, void *addr) "RDMA rsocket connec= t async ioc=3D%p addr=3D%p" +qio_channel_rdma_connect_fail(void *ioc) "RDMA rsocket connect fail ioc=3D= %p" +qio_channel_rdma_connect_complete(void *ioc, int fd) "RDMA rsocket connect= complete ioc=3D%p fd=3D%d" +qio_channel_rdma_listen_sync(void *ioc, void *addr, int num) "RDMA rsocket= listen sync ioc=3D%p addr=3D%p num=3D%d" +qio_channel_rdma_listen_fail(void *ioc) "RDMA rsocket listen fail ioc=3D%p" +qio_channel_rdma_listen_async(void *ioc, void *addr, int num) "RDMA rsocke= t listen async ioc=3D%p addr=3D%p num=3D%d" +qio_channel_rdma_listen_complete(void *ioc, int fd) "RDMA rsocket listen c= omplete ioc=3D%p fd=3D%d" +qio_channel_rdma_accept(void *ioc) "Socket accept start ioc=3D%p" +qio_channel_rdma_accept_fail(void *ioc) "RDMA rsocket accept fail ioc=3D%p" +qio_channel_rdma_accept_complete(void *ioc, void *cioc, int fd) "RDMA rsoc= ket accept complete ioc=3D%p cioc=3D%p fd=3D%d" --=20 2.43.0 From nobody Mon Nov 25 03:38:18 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1717503328; cv=none; d=zohomail.com; s=zohoarc; b=gvLqSuY+igAGOQ3eWCfpXhc64OgRpl9F3+EuO+0vj52+rAmHJfH1ZlZnSqDI5Yzg2IETd3wAchlM1e6WkD27m1ph1VGwJKX6urtvKP6oUoNBxdkR/x8R2juEEd/eQ111jQ0Qt/79BKVHAQt3tunzmC6T5vA2M6ncAjoRODDRsvA= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1717503328; h=Content-Type:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:Reply-To:Reply-To:References:Sender:Subject:Subject:To:To:Message-Id; bh=YVvT3RFgZaIqyf+Z/LFIls77jROzVJvfDyHcGACa1Ps=; b=in4swNpelwqZw9UhnarsRGLpVeRlJ3h46IaHUnaePTDpVJ3nUeguQ5B8qqJwxnog7WYruFb2ajsIp9l1aMpmYIH/tmVwnuVpRxsEaw6MsLIrdQIY0D+WfmNRRte5NUaG2f37W2rhMAkCWylHAgyQ5s+zIdeSc5F2ovR6doWAc9w= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1717503328623879.509428279886; Tue, 4 Jun 2024 05:15:28 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sET3r-0003k6-NG; Tue, 04 Jun 2024 08:14:43 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sET3o-0003iN-RU for qemu-devel@nongnu.org; Tue, 04 Jun 2024 08:14:40 -0400 Received: from szxga08-in.huawei.com ([45.249.212.255]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sET3j-0006wf-91 for qemu-devel@nongnu.org; Tue, 04 Jun 2024 08:14:40 -0400 Received: from mail.maildlp.com (unknown [172.19.163.48]) by szxga08-in.huawei.com (SkyGuard) with ESMTP id 4VtqDL10fyz1S9Lw; Tue, 4 Jun 2024 20:10:26 +0800 (CST) Received: from dggpemf200006.china.huawei.com (unknown [7.185.36.61]) by mail.maildlp.com (Postfix) with ESMTPS id 5BC93180060; Tue, 4 Jun 2024 20:14:21 +0800 (CST) Received: from DESKTOP-8LI8G6S.china.huawei.com (10.173.124.235) by dggpemf200006.china.huawei.com (7.185.36.61) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.11; Tue, 4 Jun 2024 20:14:20 +0800 To: CC: , , , , , , , , , , , , , , , Jialin Wang Subject: [PATCH 3/6] io/channel-rdma: support working in coroutine Date: Tue, 4 Jun 2024 20:14:09 +0800 Message-ID: <1717503252-51884-4-git-send-email-arei.gonglei@huawei.com> X-Mailer: git-send-email 2.8.2.windows.1 In-Reply-To: <1717503252-51884-1-git-send-email-arei.gonglei@huawei.com> References: <1717503252-51884-1-git-send-email-arei.gonglei@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.173.124.235] X-ClientProxiedBy: dggems704-chm.china.huawei.com (10.3.19.181) To dggpemf200006.china.huawei.com (7.185.36.61) Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=45.249.212.255; envelope-from=arei.gonglei@huawei.com; helo=szxga08-in.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Gonglei From: Gonglei via Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZM-MESSAGEID: 1717503330246100001 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Jialin Wang It is not feasible to obtain RDMA completion queue notifications through poll/ppoll on the rsocket fd. Therefore, we create a thread named rpoller for each rsocket fd and two eventfds: pollin_eventfd and pollout_eventfd. When using io_create_watch or io_set_aio_fd_handler waits for POLLIN or POLLOUT events, it will actually poll/ppoll on the pollin_eventfd and pollout_eventfd instead of the rsocket fd. The rpoller rpoll() on the rsocket fd to receive POLLIN and POLLOUT events. When a POLLIN event occurs, the rpoller write the pollin_eventfd, and then poll/ppoll will return the POLLIN event. When a POLLOUT event occurs, the rpoller read the pollout_eventfd, and then poll/ppoll will return the POLLOUT event. For a non-blocking rsocket fd, if rread/rwrite returns EAGAIN, it will read/write the pollin/pollout_eventfd, preventing poll/ppoll from returning POLLIN/POLLOUT events. Known limitations: For a blocking rsocket fd, if we use io_create_watch to wait for POLLIN or POLLOUT events, since the rsocket fd is blocking, we cannot determine when it is not ready to read/write as we can with non-blocking fds. Therefore, when an event occurs, it will occurs always, potentially leave the qemu hanging. So we need be cautious to avoid hanging when using io_create_watch . Luckily, channel-rdma works well in coroutines :) Signed-off-by: Jialin Wang Signed-off-by: Gonglei --- include/io/channel-rdma.h | 15 +- io/channel-rdma.c | 363 +++++++++++++++++++++++++++++++++++++- 2 files changed, 376 insertions(+), 2 deletions(-) diff --git a/include/io/channel-rdma.h b/include/io/channel-rdma.h index 8cab2459e5..cb56127d76 100644 --- a/include/io/channel-rdma.h +++ b/include/io/channel-rdma.h @@ -47,6 +47,18 @@ struct QIOChannelRDMA { socklen_t localAddrLen; struct sockaddr_storage remoteAddr; socklen_t remoteAddrLen; + + /* private */ + + /* qemu g_poll/ppoll() POLLIN event on it */ + int pollin_eventfd; + /* qemu g_poll/ppoll() POLLOUT event on it */ + int pollout_eventfd; + + /* the index in the rpoller's fds array */ + int index; + /* rpoller will rpoll() rpoll_events on the rsocket fd */ + short int rpoll_events; }; =20 /** @@ -147,6 +159,7 @@ void qio_channel_rdma_listen_async(QIOChannelRDMA *ioc,= InetSocketAddress *addr, * * Returns: the new client channel, or NULL on error */ -QIOChannelRDMA *qio_channel_rdma_accept(QIOChannelRDMA *ioc, Error **errp); +QIOChannelRDMA *coroutine_mixed_fn qio_channel_rdma_accept(QIOChannelRDMA = *ioc, + Error **errp); =20 #endif /* QIO_CHANNEL_RDMA_H */ diff --git a/io/channel-rdma.c b/io/channel-rdma.c index 92c362df52..9792add5cf 100644 --- a/io/channel-rdma.c +++ b/io/channel-rdma.c @@ -23,10 +23,15 @@ =20 #include "qemu/osdep.h" #include "io/channel-rdma.h" +#include "io/channel-util.h" +#include "io/channel-watch.h" #include "io/channel.h" #include "qapi/clone-visitor.h" #include "qapi/error.h" #include "qapi/qapi-visit-sockets.h" +#include "qemu/atomic.h" +#include "qemu/error-report.h" +#include "qemu/thread.h" #include "trace.h" #include #include @@ -39,11 +44,274 @@ #include #include =20 +typedef enum { + CLEAR_POLLIN, + CLEAR_POLLOUT, + SET_POLLIN, + SET_POLLOUT, +} UpdateEvent; + +typedef enum { + RP_CMD_ADD_IOC, + RP_CMD_DEL_IOC, + RP_CMD_UPDATE, +} RpollerCMD; + +typedef struct { + RpollerCMD cmd; + QIOChannelRDMA *rioc; +} RpollerMsg; + +/* + * rpoll() on the rsocket fd with rpoll_events, when POLLIN/POLLOUT event + * occurs, it will write/read the pollin_eventfd/pollout_eventfd to allow + * qemu g_poll/ppoll() get the POLLIN/POLLOUT event + */ +static struct Rpoller { + QemuThread thread; + bool is_running; + int sock[2]; + int count; /* the number of rsocket fds being rpoll() */ + int size; /* the size of fds/riocs */ + struct pollfd *fds; + QIOChannelRDMA **riocs; +} rpoller; + +static void qio_channel_rdma_notify_rpoller(QIOChannelRDMA *rioc, + RpollerCMD cmd) +{ + RpollerMsg msg; + int ret; + + msg.cmd =3D cmd; + msg.rioc =3D rioc; + + ret =3D RETRY_ON_EINTR(write(rpoller.sock[0], &msg, sizeof msg)); + if (ret !=3D sizeof msg) { + error_report("%s: failed to send msg, errno: %d", __func__, errno); + } +} + +static void qio_channel_rdma_update_poll_event(QIOChannelRDMA *rioc, + UpdateEvent action, + bool notify_rpoller) +{ + /* An eventfd with the value of ULLONG_MAX - 1 is readable but unwrita= ble */ + unsigned long long buf =3D ULLONG_MAX - 1; + + switch (action) { + /* only rpoller do SET_* action, to allow qemu ppoll() get the event */ + case SET_POLLIN: + RETRY_ON_EINTR(write(rioc->pollin_eventfd, &buf, sizeof buf)); + rioc->rpoll_events &=3D ~POLLIN; + break; + case SET_POLLOUT: + RETRY_ON_EINTR(read(rioc->pollout_eventfd, &buf, sizeof buf)); + rioc->rpoll_events &=3D ~POLLOUT; + break; + + /* the rsocket fd is not ready to rread/rwrite */ + case CLEAR_POLLIN: + RETRY_ON_EINTR(read(rioc->pollin_eventfd, &buf, sizeof buf)); + rioc->rpoll_events |=3D POLLIN; + break; + case CLEAR_POLLOUT: + RETRY_ON_EINTR(write(rioc->pollout_eventfd, &buf, sizeof buf)); + rioc->rpoll_events |=3D POLLOUT; + break; + default: + break; + } + + /* notify rpoller to rpoll() POLLIN/POLLOUT events */ + if (notify_rpoller) { + qio_channel_rdma_notify_rpoller(rioc, RP_CMD_UPDATE); + } +} + +static void qio_channel_rdma_rpoller_add_rioc(QIOChannelRDMA *rioc) +{ + if (rioc->index !=3D -1) { + error_report("%s: rioc already exsits", __func__); + return; + } + + rioc->index =3D ++rpoller.count; + + if (rpoller.count + 1 > rpoller.size) { + rpoller.size *=3D 2; + rpoller.fds =3D g_renew(struct pollfd, rpoller.fds, rpoller.size); + rpoller.riocs =3D g_renew(QIOChannelRDMA *, rpoller.riocs, rpoller= .size); + } + + rpoller.fds[rioc->index].fd =3D rioc->fd; + rpoller.fds[rioc->index].events =3D rioc->rpoll_events; + rpoller.riocs[rioc->index] =3D rioc; +} + +static void qio_channel_rdma_rpoller_del_rioc(QIOChannelRDMA *rioc) +{ + if (rioc->index =3D=3D -1) { + error_report("%s: rioc not exsits", __func__); + return; + } + + rpoller.fds[rioc->index] =3D rpoller.fds[rpoller.count]; + rpoller.riocs[rioc->index] =3D rpoller.riocs[rpoller.count]; + rpoller.riocs[rioc->index]->index =3D rioc->index; + rpoller.count--; + + close(rioc->pollin_eventfd); + close(rioc->pollout_eventfd); + rioc->index =3D -1; + rioc->rpoll_events =3D 0; +} + +static void qio_channel_rdma_rpoller_update_ioc(QIOChannelRDMA *rioc) +{ + if (rioc->index =3D=3D -1) { + error_report("%s: rioc not exsits", __func__); + return; + } + + rpoller.fds[rioc->index].fd =3D rioc->fd; + rpoller.fds[rioc->index].events =3D rioc->rpoll_events; +} + +static void qio_channel_rdma_rpoller_process_msg(void) +{ + RpollerMsg msg; + int ret; + + ret =3D RETRY_ON_EINTR(read(rpoller.sock[1], &msg, sizeof msg)); + if (ret !=3D sizeof msg) { + error_report("%s: rpoller failed to recv msg: %s", __func__, + strerror(errno)); + return; + } + + switch (msg.cmd) { + case RP_CMD_ADD_IOC: + qio_channel_rdma_rpoller_add_rioc(msg.rioc); + break; + case RP_CMD_DEL_IOC: + qio_channel_rdma_rpoller_del_rioc(msg.rioc); + break; + case RP_CMD_UPDATE: + qio_channel_rdma_rpoller_update_ioc(msg.rioc); + break; + default: + break; + } +} + +static void qio_channel_rdma_rpoller_cleanup(void) +{ + close(rpoller.sock[0]); + close(rpoller.sock[1]); + rpoller.sock[0] =3D -1; + rpoller.sock[1] =3D -1; + g_free(rpoller.fds); + g_free(rpoller.riocs); + rpoller.fds =3D NULL; + rpoller.riocs =3D NULL; + rpoller.count =3D 0; + rpoller.size =3D 0; + rpoller.is_running =3D false; +} + +static void *qio_channel_rdma_rpoller_thread(void *opaque) +{ + int i, ret, error_events =3D POLLERR | POLLHUP | POLLNVAL; + + do { + ret =3D rpoll(rpoller.fds, rpoller.count + 1, -1); + if (ret < 0 && errno !=3D -EINTR) { + error_report("%s: rpoll() error: %s", __func__, strerror(errno= )); + break; + } + + for (i =3D 1; i <=3D rpoller.count; i++) { + if (rpoller.fds[i].revents & (POLLIN | error_events)) { + qio_channel_rdma_update_poll_event(rpoller.riocs[i], SET_P= OLLIN, + false); + rpoller.fds[i].events &=3D ~POLLIN; + } + if (rpoller.fds[i].revents & (POLLOUT | error_events)) { + qio_channel_rdma_update_poll_event(rpoller.riocs[i], + SET_POLLOUT, false); + rpoller.fds[i].events &=3D ~POLLOUT; + } + /* ignore this fd */ + if (rpoller.fds[i].revents & (error_events)) { + rpoller.fds[i].fd =3D -1; + } + } + + if (rpoller.fds[0].revents) { + qio_channel_rdma_rpoller_process_msg(); + } + } while (rpoller.count >=3D 1); + + qio_channel_rdma_rpoller_cleanup(); + + return NULL; +} + +static void qio_channel_rdma_rpoller_start(void) +{ + if (qatomic_xchg(&rpoller.is_running, true)) { + return; + } + + if (qemu_socketpair(AF_UNIX, SOCK_STREAM, 0, rpoller.sock)) { + rpoller.is_running =3D false; + error_report("%s: failed to create socketpair %s", __func__, + strerror(errno)); + return; + } + + rpoller.count =3D 0; + rpoller.size =3D 4; + rpoller.fds =3D g_malloc0_n(rpoller.size, sizeof(struct pollfd)); + rpoller.riocs =3D g_malloc0_n(rpoller.size, sizeof(QIOChannelRDMA *)); + rpoller.fds[0].fd =3D rpoller.sock[1]; + rpoller.fds[0].events =3D POLLIN; + + qemu_thread_create(&rpoller.thread, "qio-channel-rdma-rpoller", + qio_channel_rdma_rpoller_thread, NULL, + QEMU_THREAD_JOINABLE); +} + +static void qio_channel_rdma_add_rioc_to_rpoller(QIOChannelRDMA *rioc) +{ + int flags =3D EFD_CLOEXEC | EFD_NONBLOCK; + + /* + * A single eventfd is either readable or writable. A single eventfd c= annot + * represent a state where it is neither readable nor writable. so use= two + * eventfds here. + */ + rioc->pollin_eventfd =3D eventfd(0, flags); + rioc->pollout_eventfd =3D eventfd(0, flags); + /* pollout_eventfd with the value 0, means writable, make it unwritabl= e */ + qio_channel_rdma_update_poll_event(rioc, CLEAR_POLLOUT, false); + + /* tell the rpoller to rpoll() events on rioc->socketfd */ + rioc->rpoll_events =3D POLLIN | POLLOUT; + qio_channel_rdma_notify_rpoller(rioc, RP_CMD_ADD_IOC); +} + QIOChannelRDMA *qio_channel_rdma_new(void) { QIOChannelRDMA *rioc; QIOChannel *ioc; =20 + qio_channel_rdma_rpoller_start(); + if (!rpoller.is_running) { + return NULL; + } + rioc =3D QIO_CHANNEL_RDMA(object_new(TYPE_QIO_CHANNEL_RDMA)); ioc =3D QIO_CHANNEL(rioc); qio_channel_set_feature(ioc, QIO_CHANNEL_FEATURE_SHUTDOWN); @@ -125,6 +393,8 @@ retry: goto out; } =20 + qio_channel_rdma_add_rioc_to_rpoller(rioc); + out: if (ret) { trace_qio_channel_rdma_connect_fail(rioc); @@ -211,6 +481,8 @@ int qio_channel_rdma_listen_sync(QIOChannelRDMA *rioc, = InetSocketAddress *addr, qio_channel_set_feature(QIO_CHANNEL(rioc), QIO_CHANNEL_FEATURE_LISTEN); trace_qio_channel_rdma_listen_complete(rioc, fd); =20 + qio_channel_rdma_add_rioc_to_rpoller(rioc); + out: if (ret) { trace_qio_channel_rdma_listen_fail(rioc); @@ -267,8 +539,10 @@ void qio_channel_rdma_listen_async(QIOChannelRDMA *ioc= , InetSocketAddress *addr, qio_channel_listen_worker_free, context); } =20 -QIOChannelRDMA *qio_channel_rdma_accept(QIOChannelRDMA *rioc, Error **errp) +QIOChannelRDMA *coroutine_mixed_fn qio_channel_rdma_accept(QIOChannelRDMA = *rioc, + Error **errp) { + QIOChannel *ioc =3D QIO_CHANNEL(rioc); QIOChannelRDMA *cioc; =20 cioc =3D qio_channel_rdma_new(); @@ -283,6 +557,17 @@ retry: if (errno =3D=3D EINTR) { goto retry; } + if (errno =3D=3D EAGAIN) { + if (!(rioc->rpoll_events & POLLIN)) { + qio_channel_rdma_update_poll_event(rioc, CLEAR_POLLIN, tru= e); + } + if (qemu_in_coroutine()) { + qio_channel_yield(ioc, G_IO_IN); + } else { + qio_channel_wait(ioc, G_IO_IN); + } + goto retry; + } error_setg_errno(errp, errno, "Unable to accept connection"); goto error; } @@ -294,6 +579,8 @@ retry: goto error; } =20 + qio_channel_rdma_add_rioc_to_rpoller(cioc); + trace_qio_channel_rdma_accept_complete(rioc, cioc, cioc->fd); return cioc; =20 @@ -307,6 +594,10 @@ static void qio_channel_rdma_init(Object *obj) { QIOChannelRDMA *ioc =3D QIO_CHANNEL_RDMA(obj); ioc->fd =3D -1; + ioc->pollin_eventfd =3D -1; + ioc->pollout_eventfd =3D -1; + ioc->index =3D -1; + ioc->rpoll_events =3D 0; } =20 static void qio_channel_rdma_finalize(Object *obj) @@ -314,6 +605,7 @@ static void qio_channel_rdma_finalize(Object *obj) QIOChannelRDMA *ioc =3D QIO_CHANNEL_RDMA(obj); =20 if (ioc->fd !=3D -1) { + qio_channel_rdma_notify_rpoller(ioc, RP_CMD_DEL_IOC); rclose(ioc->fd); ioc->fd =3D -1; } @@ -330,6 +622,12 @@ static ssize_t qio_channel_rdma_readv(QIOChannel *ioc,= const struct iovec *iov, retry: ret =3D rreadv(rioc->fd, iov, niov); if (ret < 0) { + if (errno =3D=3D EAGAIN) { + if (!(rioc->rpoll_events & POLLIN)) { + qio_channel_rdma_update_poll_event(rioc, CLEAR_POLLIN, tru= e); + } + return QIO_CHANNEL_ERR_BLOCK; + } if (errno =3D=3D EINTR) { goto retry; } @@ -351,6 +649,12 @@ static ssize_t qio_channel_rdma_writev(QIOChannel *ioc= , const struct iovec *iov, retry: ret =3D rwritev(rioc->fd, iov, niov); if (ret <=3D 0) { + if (errno =3D=3D EAGAIN) { + if (!(rioc->rpoll_events & POLLOUT)) { + qio_channel_rdma_update_poll_event(rioc, CLEAR_POLLOUT, tr= ue); + } + return QIO_CHANNEL_ERR_BLOCK; + } if (errno =3D=3D EINTR) { goto retry; } @@ -361,6 +665,28 @@ retry: return ret; } =20 +static int qio_channel_rdma_set_blocking(QIOChannel *ioc, bool enabled, + Error **errp G_GNUC_UNUSED) +{ + QIOChannelRDMA *rioc =3D QIO_CHANNEL_RDMA(ioc); + int flags, ret; + + flags =3D rfcntl(rioc->fd, F_GETFL); + if (enabled) { + flags &=3D ~O_NONBLOCK; + } else { + flags |=3D O_NONBLOCK; + } + + ret =3D rfcntl(rioc->fd, F_SETFL, flags); + if (ret) { + error_setg_errno(errp, errno, + "Unable to rfcntl rsocket fd with flags %d", flag= s); + } + + return ret; +} + static void qio_channel_rdma_set_delay(QIOChannel *ioc, bool enabled) { QIOChannelRDMA *rioc =3D QIO_CHANNEL_RDMA(ioc); @@ -374,6 +700,7 @@ static int qio_channel_rdma_close(QIOChannel *ioc, Erro= r **errp) QIOChannelRDMA *rioc =3D QIO_CHANNEL_RDMA(ioc); =20 if (rioc->fd !=3D -1) { + qio_channel_rdma_notify_rpoller(rioc, RP_CMD_DEL_IOC); rclose(rioc->fd); rioc->fd =3D -1; } @@ -408,6 +735,37 @@ static int qio_channel_rdma_shutdown(QIOChannel *ioc, = QIOChannelShutdown how, return 0; } =20 +static void +qio_channel_rdma_set_aio_fd_handler(QIOChannel *ioc, AioContext *read_ctx, + IOHandler *io_read, AioContext *write_= ctx, + IOHandler *io_write, void *opaque) +{ + QIOChannelRDMA *rioc =3D QIO_CHANNEL_RDMA(ioc); + + qio_channel_util_set_aio_fd_handler(rioc->pollin_eventfd, read_ctx, io= _read, + rioc->pollout_eventfd, write_ctx, + io_write, opaque); +} + +static GSource *qio_channel_rdma_create_watch(QIOChannel *ioc, + GIOCondition condition) +{ + QIOChannelRDMA *rioc =3D QIO_CHANNEL_RDMA(ioc); + + switch (condition) { + case G_IO_IN: + return qio_channel_create_fd_watch(ioc, rioc->pollin_eventfd, + condition); + case G_IO_OUT: + return qio_channel_create_fd_watch(ioc, rioc->pollout_eventfd, + condition); + default: + error_report("%s: do not support watch 0x%x event", __func__, + condition); + return NULL; + } +} + static void qio_channel_rdma_class_init(ObjectClass *klass, void *class_data G_GNUC_UNUSED) { @@ -415,9 +773,12 @@ static void qio_channel_rdma_class_init(ObjectClass *k= lass, =20 ioc_klass->io_writev =3D qio_channel_rdma_writev; ioc_klass->io_readv =3D qio_channel_rdma_readv; + ioc_klass->io_set_blocking =3D qio_channel_rdma_set_blocking; ioc_klass->io_close =3D qio_channel_rdma_close; ioc_klass->io_shutdown =3D qio_channel_rdma_shutdown; ioc_klass->io_set_delay =3D qio_channel_rdma_set_delay; + ioc_klass->io_create_watch =3D qio_channel_rdma_create_watch; + ioc_klass->io_set_aio_fd_handler =3D qio_channel_rdma_set_aio_fd_handl= er; } =20 static const TypeInfo qio_channel_rdma_info =3D { --=20 2.43.0 From nobody Mon Nov 25 03:38:18 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1717503322; cv=none; d=zohomail.com; s=zohoarc; b=E7FsbqaQikUhxvpQEgumYwFDGVlXTBNnWZ0PpTlpCA2WJCEm0y+GaChHIuqiV91yedQaVJDwDT7fMcoo1DWR6ExVvZqpEtoMdyKxzopCVnvI1KzVQZo0W1eVrqCQQ6N11EJWG/t8oGC7w8wdRLDX221yf7nke8NJ3QrCgFll+mo= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1717503322; h=Content-Type:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:Reply-To:Reply-To:References:Sender:Subject:Subject:To:To:Message-Id; bh=Jlxj5hAUxBLwqzbYblEX9AjeaQKVFRfrKjY/HWHl1vI=; b=Oq8LoKzlAlZldzZiXiPjsuEegzutUrKXlOcwpZ+tBwZf0nQvzZcJwBLJVfrqa22NqOITb0MKzK/5hOhK5+T/jOMKuk1z1RluUHg+WbMLj5Pwb0NPoReHAyOOOoXxW1E+tdZMjNqtu47lGxszfUI0ai+1hHJpnRtWxHTc2KfbT60= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1717503322137687.6484990459834; Tue, 4 Jun 2024 05:15:22 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sET3o-0003i7-9U; Tue, 04 Jun 2024 08:14:40 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sET3l-0003gL-F5 for qemu-devel@nongnu.org; Tue, 04 Jun 2024 08:14:37 -0400 Received: from szxga03-in.huawei.com ([45.249.212.189]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sET3g-0006wr-3v for qemu-devel@nongnu.org; Tue, 04 Jun 2024 08:14:36 -0400 Received: from mail.maildlp.com (unknown [172.19.163.48]) by szxga03-in.huawei.com (SkyGuard) with ESMTP id 4VtqF55KFlzPpNw; Tue, 4 Jun 2024 20:11:05 +0800 (CST) Received: from dggpemf200006.china.huawei.com (unknown [7.185.36.61]) by mail.maildlp.com (Postfix) with ESMTPS id 20034180060; Tue, 4 Jun 2024 20:14:22 +0800 (CST) Received: from DESKTOP-8LI8G6S.china.huawei.com (10.173.124.235) by dggpemf200006.china.huawei.com (7.185.36.61) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.11; Tue, 4 Jun 2024 20:14:21 +0800 To: CC: , , , , , , , , , , , , , , , Jialin Wang Subject: [PATCH 4/6] tests/unit: add test-io-channel-rdma.c Date: Tue, 4 Jun 2024 20:14:10 +0800 Message-ID: <1717503252-51884-5-git-send-email-arei.gonglei@huawei.com> X-Mailer: git-send-email 2.8.2.windows.1 In-Reply-To: <1717503252-51884-1-git-send-email-arei.gonglei@huawei.com> References: <1717503252-51884-1-git-send-email-arei.gonglei@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.173.124.235] X-ClientProxiedBy: dggems704-chm.china.huawei.com (10.3.19.181) To dggpemf200006.china.huawei.com (7.185.36.61) Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=45.249.212.189; envelope-from=arei.gonglei@huawei.com; helo=szxga03-in.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Gonglei From: Gonglei via Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZM-MESSAGEID: 1717503323949100003 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Jialin Wang Signed-off-by: Jialin Wang Signed-off-by: Gonglei --- tests/unit/meson.build | 1 + tests/unit/test-io-channel-rdma.c | 276 ++++++++++++++++++++++++++++++ 2 files changed, 277 insertions(+) create mode 100644 tests/unit/test-io-channel-rdma.c diff --git a/tests/unit/meson.build b/tests/unit/meson.build index 26c109c968..c44020a3b5 100644 --- a/tests/unit/meson.build +++ b/tests/unit/meson.build @@ -85,6 +85,7 @@ if have_block 'test-authz-listfile': [authz], 'test-io-task': [testblock], 'test-io-channel-socket': ['socket-helpers.c', 'io-channel-helpers.c',= io], + 'test-io-channel-rdma': ['io-channel-helpers.c', io], 'test-io-channel-file': ['io-channel-helpers.c', io], 'test-io-channel-command': ['io-channel-helpers.c', io], 'test-io-channel-buffer': ['io-channel-helpers.c', io], diff --git a/tests/unit/test-io-channel-rdma.c b/tests/unit/test-io-channel= -rdma.c new file mode 100644 index 0000000000..e96b55c8c7 --- /dev/null +++ b/tests/unit/test-io-channel-rdma.c @@ -0,0 +1,276 @@ +/* + * QEMU I/O channel RDMA test + * + * Copyright (c) 2024 HUAWEI TECHNOLOGIES CO., LTD. + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library; if not, see . + */ + +#include "qemu/osdep.h" +#include "io/channel-rdma.h" +#include "qapi/error.h" +#include "qemu/main-loop.h" +#include "qemu/module.h" +#include "io-channel-helpers.h" +#include "qapi-types-sockets.h" +#include + +static SocketAddress *l_addr; +static SocketAddress *c_addr; + +static void test_io_channel_set_rdma_bufs(QIOChannel *src, QIOChannel *dst) +{ + int buflen =3D 64 * 1024; + + /* + * Make the socket buffers small so that we see + * the effects of partial reads/writes + */ + rsetsockopt(((QIOChannelRDMA *)src)->fd, SOL_SOCKET, SO_SNDBUF, + (char *)&buflen, sizeof(buflen)); + + rsetsockopt(((QIOChannelRDMA *)dst)->fd, SOL_SOCKET, SO_SNDBUF, + (char *)&buflen, sizeof(buflen)); +} + +static void test_io_channel_setup_sync(InetSocketAddress *listen_addr, + InetSocketAddress *connect_addr, + QIOChannel **srv, QIOChannel **src, + QIOChannel **dst) +{ + QIOChannelRDMA *lioc; + + lioc =3D qio_channel_rdma_new(); + qio_channel_rdma_listen_sync(lioc, listen_addr, 1, &error_abort); + + *src =3D QIO_CHANNEL(qio_channel_rdma_new()); + qio_channel_rdma_connect_sync(QIO_CHANNEL_RDMA(*src), connect_addr, + &error_abort); + qio_channel_set_delay(*src, false); + + qio_channel_wait(QIO_CHANNEL(lioc), G_IO_IN); + *dst =3D QIO_CHANNEL(qio_channel_rdma_accept(lioc, &error_abort)); + g_assert(*dst); + + test_io_channel_set_rdma_bufs(*src, *dst); + + *srv =3D QIO_CHANNEL(lioc); +} + +struct TestIOChannelData { + bool err; + GMainLoop *loop; +}; + +static void test_io_channel_complete(QIOTask *task, gpointer opaque) +{ + struct TestIOChannelData *data =3D opaque; + data->err =3D qio_task_propagate_error(task, NULL); + g_main_loop_quit(data->loop); +} + +static void test_io_channel_setup_async(InetSocketAddress *listen_addr, + InetSocketAddress *connect_addr, + QIOChannel **srv, QIOChannel **src, + QIOChannel **dst) +{ + QIOChannelRDMA *lioc; + struct TestIOChannelData data; + + data.loop =3D g_main_loop_new(g_main_context_default(), TRUE); + + lioc =3D qio_channel_rdma_new(); + qio_channel_rdma_listen_async(lioc, listen_addr, 1, + test_io_channel_complete, &data, NULL, N= ULL); + + g_main_loop_run(data.loop); + g_main_context_iteration(g_main_context_default(), FALSE); + + g_assert(!data.err); + + *src =3D QIO_CHANNEL(qio_channel_rdma_new()); + + qio_channel_rdma_connect_async(QIO_CHANNEL_RDMA(*src), connect_addr, + test_io_channel_complete, &data, NULL, = NULL); + + g_main_loop_run(data.loop); + g_main_context_iteration(g_main_context_default(), FALSE); + + g_assert(!data.err); + + if (qemu_in_coroutine()) { + qio_channel_yield(QIO_CHANNEL(lioc), G_IO_IN); + } else { + qio_channel_wait(QIO_CHANNEL(lioc), G_IO_IN); + } + *dst =3D QIO_CHANNEL(qio_channel_rdma_accept(lioc, &error_abort)); + g_assert(*dst); + + qio_channel_set_delay(*src, false); + test_io_channel_set_rdma_bufs(*src, *dst); + + *srv =3D QIO_CHANNEL(lioc); + + g_main_loop_unref(data.loop); +} + +static void test_io_channel(bool async, InetSocketAddress *listen_addr, + InetSocketAddress *connect_addr) +{ + QIOChannel *src, *dst, *srv; + QIOChannelTest *test; + + if (async) { + /* async + blocking */ + + test_io_channel_setup_async(listen_addr, connect_addr, &srv, &src, + &dst); + + g_assert(qio_channel_has_feature(src, QIO_CHANNEL_FEATURE_SHUTDOWN= )); + g_assert(qio_channel_has_feature(dst, QIO_CHANNEL_FEATURE_SHUTDOWN= )); + + test =3D qio_channel_test_new(); + qio_channel_test_run_threads(test, true, src, dst); + qio_channel_test_validate(test); + + /* unref without close, to ensure finalize() cleans up */ + + object_unref(OBJECT(src)); + object_unref(OBJECT(dst)); + object_unref(OBJECT(srv)); + + /* async + non-blocking */ + + test_io_channel_setup_async(listen_addr, connect_addr, &srv, &src, + &dst); + + g_assert(qio_channel_has_feature(src, QIO_CHANNEL_FEATURE_SHUTDOWN= )); + g_assert(qio_channel_has_feature(dst, QIO_CHANNEL_FEATURE_SHUTDOWN= )); + + test =3D qio_channel_test_new(); + qio_channel_test_run_threads(test, false, src, dst); + qio_channel_test_validate(test); + + /* close before unref, to ensure finalize copes with already close= d */ + + qio_channel_close(src, &error_abort); + qio_channel_close(dst, &error_abort); + object_unref(OBJECT(src)); + object_unref(OBJECT(dst)); + + qio_channel_close(srv, &error_abort); + object_unref(OBJECT(srv)); + } else { + /* sync + blocking */ + + test_io_channel_setup_sync(listen_addr, connect_addr, &srv, &src, = &dst); + + g_assert(qio_channel_has_feature(src, QIO_CHANNEL_FEATURE_SHUTDOWN= )); + g_assert(qio_channel_has_feature(dst, QIO_CHANNEL_FEATURE_SHUTDOWN= )); + + test =3D qio_channel_test_new(); + qio_channel_test_run_threads(test, true, src, dst); + qio_channel_test_validate(test); + + /* unref without close, to ensure finalize() cleans up */ + + object_unref(OBJECT(src)); + object_unref(OBJECT(dst)); + object_unref(OBJECT(srv)); + + /* sync + non-blocking */ + + test_io_channel_setup_sync(listen_addr, connect_addr, &srv, &src, = &dst); + + g_assert(qio_channel_has_feature(src, QIO_CHANNEL_FEATURE_SHUTDOWN= )); + g_assert(qio_channel_has_feature(dst, QIO_CHANNEL_FEATURE_SHUTDOWN= )); + + test =3D qio_channel_test_new(); + qio_channel_test_run_threads(test, false, src, dst); + qio_channel_test_validate(test); + + /* close before unref, to ensure finalize copes with already close= d */ + + qio_channel_close(src, &error_abort); + qio_channel_close(dst, &error_abort); + object_unref(OBJECT(src)); + object_unref(OBJECT(dst)); + + qio_channel_close(srv, &error_abort); + object_unref(OBJECT(srv)); + } +} + +static void test_io_channel_rdma(bool async) +{ + InetSocketAddress *listen_addr; + InetSocketAddress *connect_addr; + + listen_addr =3D &l_addr->u.inet; + connect_addr =3D &l_addr->u.inet; + + test_io_channel(async, listen_addr, connect_addr); +} + +static void test_io_channel_rdma_sync(void) +{ + test_io_channel_rdma(false); +} + +static void test_io_channel_rdma_async(void) +{ + test_io_channel_rdma(true); +} + +static void test_io_channel_rdma_co(void *opaque) +{ + test_io_channel_rdma(true); +} + +static void test_io_channel_rdma_coroutine(void) +{ + Coroutine *coroutine; + + coroutine =3D qemu_coroutine_create(test_io_channel_rdma_co, NULL); + qemu_coroutine_enter(coroutine); +} + +int main(int argc, char **argv) +{ + module_call_init(MODULE_INIT_QOM); + qemu_init_main_loop(&error_abort); + + if (argc !=3D 3) { + fprintf(stderr, "Usage: %s listen_addr connect_addr\n", argv[0]); + exit(-1); + } + + l_addr =3D socket_parse(argv[1], NULL); + c_addr =3D socket_parse(argv[2], NULL); + if (l_addr =3D=3D NULL || c_addr =3D=3D NULL || + l_addr->type !=3D SOCKET_ADDRESS_TYPE_INET || + c_addr->type !=3D SOCKET_ADDRESS_TYPE_INET) { + fprintf(stderr, "Only socket address types 'inet' is supported\n"); + exit(-1); + } + + g_test_init(&argc, &argv, NULL); + + g_test_add_func("/io/channel/rdma/sync", test_io_channel_rdma_sync); + g_test_add_func("/io/channel/rdma/async", test_io_channel_rdma_async); + g_test_add_func("/io/channel/rdma/coroutine", + test_io_channel_rdma_coroutine); + + return g_test_run(); +} --=20 2.43.0 From nobody Mon Nov 25 03:38:18 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1717503306; cv=none; d=zohomail.com; s=zohoarc; b=n1U3bZF5nb7Mej7UUnm1qeDeYwweAXMUbNVh+Zgb+1gwu0P4pQ8uXkBVsCCNNFUF2MBNryX6+jhm97H64DtZ8JJlSORNQJcKRF/JVRlednRcfsR5s13AwwCsL6/nXLhLucDY9aTFZaDVreD2NiCx7TBKbEA5unkDhKRxGFu/y/s= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1717503306; h=Content-Type:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:Reply-To:Reply-To:References:Sender:Subject:Subject:To:To:Message-Id; bh=XZ1uiPQmQrPdYPLVweGoc/EHbMf8brAvj3mTAO0shuY=; b=QU4+MFbEWc1xwwYZUvp0ZlUBKjKyr94rZ+O8ob8tSRNMV7iQrBp3+Uz728FIJTY3OEp2B3Jdy7dZrWruwbO0FtTbqZSU3rq3eJlEX6TtP7H7/7ISr9EyN65fuG4Y4Tel5FZbVVlJWyIkO9/yCAAsPHdm1RY4FiVKiMTJYr4gPAk= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1717503306814942.0355163505163; Tue, 4 Jun 2024 05:15:06 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sET3q-0003iw-0Z; Tue, 04 Jun 2024 08:14:42 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sET3l-0003gk-Rx for qemu-devel@nongnu.org; Tue, 04 Jun 2024 08:14:37 -0400 Received: from szxga04-in.huawei.com ([45.249.212.190]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sET3g-0006xF-Qf for qemu-devel@nongnu.org; Tue, 04 Jun 2024 08:14:37 -0400 Received: from mail.maildlp.com (unknown [172.19.88.214]) by szxga04-in.huawei.com (SkyGuard) with ESMTP id 4VtqDf2DGjz2Cjml; Tue, 4 Jun 2024 20:10:42 +0800 (CST) Received: from dggpemf200006.china.huawei.com (unknown [7.185.36.61]) by mail.maildlp.com (Postfix) with ESMTPS id DA27A1A016F; Tue, 4 Jun 2024 20:14:22 +0800 (CST) Received: from DESKTOP-8LI8G6S.china.huawei.com (10.173.124.235) by dggpemf200006.china.huawei.com (7.185.36.61) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.11; Tue, 4 Jun 2024 20:14:22 +0800 To: CC: , , , , , , , , , , , , , , , Jialin Wang Subject: [PATCH 5/6] migration: introduce new RDMA live migration Date: Tue, 4 Jun 2024 20:14:11 +0800 Message-ID: <1717503252-51884-6-git-send-email-arei.gonglei@huawei.com> X-Mailer: git-send-email 2.8.2.windows.1 In-Reply-To: <1717503252-51884-1-git-send-email-arei.gonglei@huawei.com> References: <1717503252-51884-1-git-send-email-arei.gonglei@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.173.124.235] X-ClientProxiedBy: dggems704-chm.china.huawei.com (10.3.19.181) To dggpemf200006.china.huawei.com (7.185.36.61) Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=45.249.212.190; envelope-from=arei.gonglei@huawei.com; helo=szxga04-in.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Gonglei From: Gonglei via Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZM-MESSAGEID: 1717503307955100001 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Jialin Wang Signed-off-by: Jialin Wang Signed-off-by: Gonglei --- migration/meson.build | 2 + migration/migration.c | 11 +++++- migration/rdma.c | 88 +++++++++++++++++++++++++++++++++++++++++++ migration/rdma.h | 24 ++++++++++++ 4 files changed, 124 insertions(+), 1 deletion(-) create mode 100644 migration/rdma.c create mode 100644 migration/rdma.h diff --git a/migration/meson.build b/migration/meson.build index 4e8a9ccf3e..04e2e16239 100644 --- a/migration/meson.build +++ b/migration/meson.build @@ -42,3 +42,5 @@ system_ss.add(when: zstd, if_true: files('multifd-zstd.c'= )) specific_ss.add(when: 'CONFIG_SYSTEM_ONLY', if_true: files('ram.c', 'target.c')) + +system_ss.add(when: rdma, if_true: files('rdma.c')) diff --git a/migration/migration.c b/migration/migration.c index 6b9ad4ff5f..77c301d351 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -25,6 +25,7 @@ #include "sysemu/runstate.h" #include "sysemu/sysemu.h" #include "sysemu/cpu-throttle.h" +#include "rdma.h" #include "ram.h" #include "migration/global_state.h" #include "migration/misc.h" @@ -145,7 +146,7 @@ static bool transport_supports_multi_channels(Migration= Address *addr) } else if (addr->transport =3D=3D MIGRATION_ADDRESS_TYPE_FILE) { return migrate_mapped_ram(); } else { - return false; + return addr->transport =3D=3D MIGRATION_ADDRESS_TYPE_RDMA; } } =20 @@ -644,6 +645,10 @@ static void qemu_start_incoming_migration(const char *= uri, bool has_channels, } else if (saddr->type =3D=3D SOCKET_ADDRESS_TYPE_FD) { fd_start_incoming_migration(saddr->u.fd.str, errp); } +#ifdef CONFIG_RDMA + } else if (addr->transport =3D=3D MIGRATION_ADDRESS_TYPE_RDMA) { + rdma_start_incoming_migration(&addr->u.rdma, errp); +#endif } else if (addr->transport =3D=3D MIGRATION_ADDRESS_TYPE_EXEC) { exec_start_incoming_migration(addr->u.exec.args, errp); } else if (addr->transport =3D=3D MIGRATION_ADDRESS_TYPE_FILE) { @@ -2046,6 +2051,10 @@ void qmp_migrate(const char *uri, bool has_channels, } else if (saddr->type =3D=3D SOCKET_ADDRESS_TYPE_FD) { fd_start_outgoing_migration(s, saddr->u.fd.str, &local_err); } +#ifdef CONFIG_RDMA + } else if (addr->transport =3D=3D MIGRATION_ADDRESS_TYPE_RDMA) { + rdma_start_outgoing_migration(s, &addr->u.rdma, &local_err); +#endif } else if (addr->transport =3D=3D MIGRATION_ADDRESS_TYPE_EXEC) { exec_start_outgoing_migration(s, addr->u.exec.args, &local_err); } else if (addr->transport =3D=3D MIGRATION_ADDRESS_TYPE_FILE) { diff --git a/migration/rdma.c b/migration/rdma.c new file mode 100644 index 0000000000..09a4de7f59 --- /dev/null +++ b/migration/rdma.c @@ -0,0 +1,88 @@ +/* + * QEMU live migration via RDMA + * + * Copyright (c) 2024 HUAWEI TECHNOLOGIES CO., LTD. + * + * Authors: + * Jialin Wang + * Gonglei + * + * This work is licensed under the terms of the GNU GPL, version 2 or late= r. + * See the COPYING file in the top-level directory. + */ + +#include "qemu/osdep.h" +#include "io/channel-rdma.h" +#include "io/channel.h" +#include "qapi/clone-visitor.h" +#include "qapi/qapi-types-sockets.h" +#include "qapi/qapi-visit-sockets.h" +#include "channel.h" +#include "migration.h" +#include "rdma.h" +#include "trace.h" +#include + +static struct RDMAOutgoingArgs { + InetSocketAddress *addr; +} outgoing_args; + +static void rdma_outgoing_migration(QIOTask *task, gpointer opaque) +{ + MigrationState *s =3D opaque; + QIOChannelRDMA *rioc =3D QIO_CHANNEL_RDMA(qio_task_get_source(task)); + + migration_channel_connect(s, QIO_CHANNEL(rioc), outgoing_args.addr->ho= st, + NULL); + object_unref(OBJECT(rioc)); +} + +void rdma_start_outgoing_migration(MigrationState *s, InetSocketAddress *i= addr, + Error **errp) +{ + QIOChannelRDMA *rioc =3D qio_channel_rdma_new(); + + /* in case previous migration leaked it */ + qapi_free_InetSocketAddress(outgoing_args.addr); + outgoing_args.addr =3D QAPI_CLONE(InetSocketAddress, iaddr); + + qio_channel_set_name(QIO_CHANNEL(rioc), "migration-rdma-outgoing"); + qio_channel_rdma_connect_async(rioc, iaddr, rdma_outgoing_migration, s, + NULL, NULL); +} + +static void coroutine_fn rdma_accept_incoming_migration(void *opaque) +{ + QIOChannelRDMA *rioc =3D opaque; + QIOChannelRDMA *cioc; + + while (!migration_has_all_channels()) { + cioc =3D qio_channel_rdma_accept(rioc, NULL); + + qio_channel_set_name(QIO_CHANNEL(cioc), "migration-rdma-incoming"); + migration_channel_process_incoming(QIO_CHANNEL(cioc)); + object_unref(OBJECT(cioc)); + } +} + +void rdma_start_incoming_migration(InetSocketAddress *addr, Error **errp) +{ + QIOChannelRDMA *rioc =3D qio_channel_rdma_new(); + MigrationIncomingState *mis =3D migration_incoming_get_current(); + Coroutine *co; + int num =3D 1; + + qio_channel_set_name(QIO_CHANNEL(rioc), "migration-rdma-listener"); + + if (qio_channel_rdma_listen_sync(rioc, addr, num, errp) < 0) { + object_unref(OBJECT(rioc)); + return; + } + + mis->transport_data =3D rioc; + mis->transport_cleanup =3D object_unref; + + qio_channel_set_blocking(QIO_CHANNEL(rioc), false, NULL); + co =3D qemu_coroutine_create(rdma_accept_incoming_migration, rioc); + aio_co_schedule(qemu_get_current_aio_context(), co); +} diff --git a/migration/rdma.h b/migration/rdma.h new file mode 100644 index 0000000000..4c3eb9a972 --- /dev/null +++ b/migration/rdma.h @@ -0,0 +1,24 @@ +/* + * QEMU live migration via RDMA + * + * Copyright (c) 2024 HUAWEI TECHNOLOGIES CO., LTD. + * + * Authors: + * Jialin Wang + * Gonglei + * + * This work is licensed under the terms of the GNU GPL, version 2 or late= r. + * See the COPYING file in the top-level directory. + */ + +#ifndef QEMU_MIGRATION_RDMA_H +#define QEMU_MIGRATION_RDMA_H + +#include "qemu/sockets.h" + +void rdma_start_outgoing_migration(MigrationState *s, InetSocketAddress *a= ddr, + Error **errp); + +void rdma_start_incoming_migration(InetSocketAddress *addr, Error **errp); + +#endif /* QEMU_MIGRATION_RDMA_H */ --=20 2.43.0 From nobody Mon Nov 25 03:38:18 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1717503360; cv=none; d=zohomail.com; s=zohoarc; b=irvx2VoF7V2qBd+D03mo9Bd5u9xtjBhjXQIUMorfBDKbJ0kZkcyVWwrUMoIYRY4AHjk60fA7DLl9l8w9lPhE6SkLZqPnrInJUC7LLSIviqADANau6JudTEwAodNnzoqBojzn/DBpR4z4lwE+vDFkTiQICfzIE3awoPtkF6JSnaE= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1717503360; h=Content-Type:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:Reply-To:Reply-To:References:Sender:Subject:Subject:To:To:Message-Id; bh=Xj84flbXsIyrK7BKdUlvipoc3sqIVn91HxeFGfOP4Ig=; b=YB8IRDBQSgM0MjQk2Z1hPcbDXsxywUe1Vya1wVhNoNTpaM0ybUYghZlfn5zGiGK0BpQBYjxpRPY8tXMSv9d3SOVsr16SYWVzMHhUHJ5AClS1xCnlzCTyDRLaMyXzXcA0KgaDV9XImhII0wVjnDEj8XQZlKCYF1WF0rzOsIAdYQY= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1717503360581586.565269908597; Tue, 4 Jun 2024 05:16:00 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sET3r-0003jU-I1; Tue, 04 Jun 2024 08:14:43 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sET3l-0003gl-Rv for qemu-devel@nongnu.org; Tue, 04 Jun 2024 08:14:37 -0400 Received: from szxga05-in.huawei.com ([45.249.212.191]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sET3i-0006xV-92 for qemu-devel@nongnu.org; Tue, 04 Jun 2024 08:14:37 -0400 Received: from mail.maildlp.com (unknown [172.19.163.17]) by szxga05-in.huawei.com (SkyGuard) with ESMTP id 4VtqDg1ybZz355fS; Tue, 4 Jun 2024 20:10:43 +0800 (CST) Received: from dggpemf200006.china.huawei.com (unknown [7.185.36.61]) by mail.maildlp.com (Postfix) with ESMTPS id A0ED81A0188; Tue, 4 Jun 2024 20:14:23 +0800 (CST) Received: from DESKTOP-8LI8G6S.china.huawei.com (10.173.124.235) by dggpemf200006.china.huawei.com (7.185.36.61) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.11; Tue, 4 Jun 2024 20:14:22 +0800 To: CC: , , , , , , , , , , , , , , , Jialin Wang Subject: [PATCH 6/6] migration/rdma: support multifd for RDMA migration Date: Tue, 4 Jun 2024 20:14:12 +0800 Message-ID: <1717503252-51884-7-git-send-email-arei.gonglei@huawei.com> X-Mailer: git-send-email 2.8.2.windows.1 In-Reply-To: <1717503252-51884-1-git-send-email-arei.gonglei@huawei.com> References: <1717503252-51884-1-git-send-email-arei.gonglei@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.173.124.235] X-ClientProxiedBy: dggems704-chm.china.huawei.com (10.3.19.181) To dggpemf200006.china.huawei.com (7.185.36.61) Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=45.249.212.191; envelope-from=arei.gonglei@huawei.com; helo=szxga05-in.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Gonglei From: Gonglei via Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZM-MESSAGEID: 1717503363541100010 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Jialin Wang Signed-off-by: Jialin Wang Signed-off-by: Gonglei --- migration/multifd.c | 10 ++++++++++ migration/rdma.c | 27 +++++++++++++++++++++++++++ migration/rdma.h | 6 ++++++ 3 files changed, 43 insertions(+) diff --git a/migration/multifd.c b/migration/multifd.c index f317bff077..cee9858ad1 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -32,6 +32,7 @@ #include "io/channel-file.h" #include "io/channel-socket.h" #include "yank_functions.h" +#include "rdma.h" =20 /* Multiple fd's */ =20 @@ -793,6 +794,9 @@ static bool multifd_send_cleanup_channel(MultiFDSendPar= ams *p, Error **errp) static void multifd_send_cleanup_state(void) { file_cleanup_outgoing_migration(); +#ifdef CONFIG_RDMA + rdma_cleanup_outgoing_migration(); +#endif socket_cleanup_outgoing_migration(); qemu_sem_destroy(&multifd_send_state->channels_created); qemu_sem_destroy(&multifd_send_state->channels_ready); @@ -1139,6 +1143,12 @@ static bool multifd_new_send_channel_create(gpointer= opaque, Error **errp) return file_send_channel_create(opaque, errp); } =20 +#ifdef CONFIG_RDMA + if (rdma_send_channel_create(multifd_new_send_channel_async, opaque)) { + return true; + } +#endif + socket_send_channel_create(multifd_new_send_channel_async, opaque); return true; } diff --git a/migration/rdma.c b/migration/rdma.c index 09a4de7f59..af4d2b5a5a 100644 --- a/migration/rdma.c +++ b/migration/rdma.c @@ -19,6 +19,7 @@ #include "qapi/qapi-visit-sockets.h" #include "channel.h" #include "migration.h" +#include "options.h" #include "rdma.h" #include "trace.h" #include @@ -27,6 +28,28 @@ static struct RDMAOutgoingArgs { InetSocketAddress *addr; } outgoing_args; =20 +bool rdma_send_channel_create(QIOTaskFunc f, void *data) +{ + QIOChannelRDMA *rioc; + + if (!outgoing_args.addr) { + return false; + } + + rioc =3D qio_channel_rdma_new(); + qio_channel_rdma_connect_async(rioc, outgoing_args.addr, f, data, NULL, + NULL); + return true; +} + +void rdma_cleanup_outgoing_migration(void) +{ + if (outgoing_args.addr) { + qapi_free_InetSocketAddress(outgoing_args.addr); + outgoing_args.addr =3D NULL; + } +} + static void rdma_outgoing_migration(QIOTask *task, gpointer opaque) { MigrationState *s =3D opaque; @@ -74,6 +97,10 @@ void rdma_start_incoming_migration(InetSocketAddress *ad= dr, Error **errp) =20 qio_channel_set_name(QIO_CHANNEL(rioc), "migration-rdma-listener"); =20 + if (migrate_multifd()) { + num =3D migrate_multifd_channels(); + } + if (qio_channel_rdma_listen_sync(rioc, addr, num, errp) < 0) { object_unref(OBJECT(rioc)); return; diff --git a/migration/rdma.h b/migration/rdma.h index 4c3eb9a972..cefccac61c 100644 --- a/migration/rdma.h +++ b/migration/rdma.h @@ -16,6 +16,12 @@ =20 #include "qemu/sockets.h" =20 +#include + +bool rdma_send_channel_create(QIOTaskFunc f, void *data); + +void rdma_cleanup_outgoing_migration(void); + void rdma_start_outgoing_migration(MigrationState *s, InetSocketAddress *a= ddr, Error **errp); =20 --=20 2.43.0