From nobody Thu May 9 22:02:47 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass header.i=lukasstraub2@web.de; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=web.de ARC-Seal: i=1; a=rsa-sha256; t=1683507249; cv=none; d=zohomail.com; s=zohoarc; b=JwMSgzuqdHAOeAFu4KgGe3PvXbT6Ai2iX7FkjtZWxihMmO7w+DFGabAWD7w0/mewJ/9Vwx+tZ4rZZGWlmEtOSkddsrwIY5bCwBvKN+TVy5HW84LfdIKToroL5XvhsLi3+IPA4jc+NXALeB/uaulvtf6n4pN3TIOHLz7jvAMcjGA= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1683507249; h=Content-Type:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=hUyLjUlg02B/kO9b28C9DHMZRohOQOoG5bXqoHbk/7I=; b=VdGJIMJzNOJ//zcZrlO8dUlCM2pdIMX/7Ece6zyCQ/I0ESVAxXBYFe0iK5cAiiyZmj98guny7BQMQ88khjh4I333BD5NoaFyS7Ch/OpnaAQmMwZ0miakRsj+o1EOyzTLbWiUh6jngR46kDzM5Zl/2z6nP0jvdEHadvmJBP31mlQ= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass header.i=lukasstraub2@web.de; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1683507249797753.1869076970462; Sun, 7 May 2023 17:54:09 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pvp7O-00071R-IM; Sun, 07 May 2023 20:52:46 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pvp7N-000717-60 for qemu-devel@nongnu.org; Sun, 07 May 2023 20:52:45 -0400 Received: from mout.web.de ([212.227.15.4]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pvp7K-0004Xg-EA for qemu-devel@nongnu.org; Sun, 07 May 2023 20:52:44 -0400 Received: from gecko.fritz.box ([82.207.254.115]) by smtp.web.de (mrweb005 [213.165.67.108]) with ESMTPSA (Nemesis) id 1MY5fd-1phmkJ3uub-00YTAp; Mon, 08 May 2023 02:52:34 +0200 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=web.de; s=s29768273; t=1683507155; i=lukasstraub2@web.de; bh=8wn4GkQ7/OJiG7CL3mo7ScQO5fAcXwo9fps/8sZHERQ=; h=X-UI-Sender-Class:Date:From:To:Cc:Subject:In-Reply-To:References; b=TxB00nqzyrApdF5k8UAbrFGi7Q6KWeJBJYLgbTjdZlXz4qXU+Zh7C/L0xKb8kMwNI sJC1jxbKWAZfH6ekUDneNF3aWuHfV0V0btLyJDdWKenvlPYeFBE81GioKwSvCnaGxL hEs8csQt8IEjGymOxGMfoboW+/N7CzJxX9qo4YZbqQ7998zFSNVvaOn6Rb3ZKdoO/R 3WjAXDZdkDCjd51kIl5mrdMRP1Zm7epgL2tu8UEEMNgxQXxwTWU/QPwhXK7X3S7Ok7 DivY1o4p3xFHtn+V6dsTpeXBpxHkQtaAm/KfrQvu7txc4YmdFdLilzuEM8mEfxpYX+ etIVVdZ44+Fgw== X-UI-Sender-Class: 814a7b36-bfc1-4dae-8640-3722d8ec6cd6 Date: Mon, 8 May 2023 02:52:33 +0200 From: Lukas Straub To: qemu-devel Cc: Juan Quintela , Peter Xu , Leonardo Bras Subject: [PATCH 1/5] ram: Add public helper to set colo bitmap Message-ID: In-Reply-To: References: MIME-Version: 1.0 Content-Type: multipart/signed; boundary="Sig_/I3MFa6umhU/ggOdWlvjgxSS"; protocol="application/pgp-signature"; micalg=pgp-sha256 X-Provags-ID: V03:K1:chPg89AQaGkjW/sC7RNkKim5n5sZ1UilWxzt/qQLMeThAYRym4d g9hpCpeeOjpjPr1wjBTi9DWesw5f2vU7GeqqDjEdV2OJmEh+FmscoZ7YzTjN2c5BfyS+TfA YZsPcmw2r8C0LEIMlU94xqJNskBhEwiL/0qOnoyXAEGSFmmWq+Sftm/fVNXRagc86BuiL1T Dg2WtBty94zhoEpkrfH4g== UI-OutboundReport: notjunk:1;M01:P0:4+kJoJfF8w8=;LZ25wXWBlFC6bIsdwtD6KGHf6/W dQoKoyzCaodSXvVRok5GUSOJOFZpdrJz+oKeJW7DvbuhChdKh+NUbqmsTcxB/pT36wlGrFa2w QW4qeMNTSrgFnWSZwARLEEy9waZzN4eOineLch4bJ5xhpXE7mnMw1dP5K3XIUAsgYzM69hS41 Fe7dcTXNaCFsVirr2r9tc9x0Js6jImMKzs+7dE2+vzckEc04yBzlBrB8Sq/cyNT/TIHiC9TCp sAHCuhuGa1oLMS4DDoBsF0rFi1HwQ91VWj5PvKjx6PTphBoLIKi6eLzvfWn449SV+j4Fi1MSv LjNZFBIwu6c99pEGms5f7brDOfQzrai8QWhLoO5s4x0iAkhzzX9uW0iGaw+RVFpXqdgWWlUGn m8+o6SYkMtQZjNHUTke5ZYn9QnnrSODpy4o1D2ac2HrWtZc3X43gHabwFopqrsgeAYsqbcImP /bmDkJRWh7tdcHDpbtP5Qbqf1xX7NxKOlVuSrZyvoSVqyNVgrTBdV1QKiAVqJchkaogryioL6 tYxYsNvfhWo6wtzjpqbzVASvu3vBT9I6bhmpFgWIs6Kdz3T+SOq5f7T9e63l26yekPs/nD80+ 6shkFc0sM+D/rHQ65acyNQZOGjBGb5u8C39oMP707YtPrTeiILImTsUgG16+zYvM1mL19AmSF kvuryqII+39U9IC+v8kFiLtBWRpzFjizsxJA/4rLb/7nyYtw9YWURIhvDlXdlC//kJiuyoxjr NQNn8CUtn827kJjpiJVjUbFVTIWg+Gwke4eVs7KP20Mz7cWiPjGw6BEw3bLMyYC+tGulAB0V5 Fid9YFESRioCe3gVxlYvdXogvqOt8t+GSn5CBALZrsVOdRkZJTMBZIBGbQH8kNj7E3vkkuJwa mcsDQWOjG2Hsggd7ltzYbkPEXQ28eDw4q8YH4zjDHJNaPkRNvMpYASMvjtGTkfR0HwTsuE0dP p8nesA== Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=212.227.15.4; envelope-from=lukasstraub2@web.de; helo=mout.web.de X-Spam_score_int: -25 X-Spam_score: -2.6 X-Spam_bar: -- X-Spam_report: (-2.6 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-Zoho-Virus-Status: 1 X-ZohoMail-DKIM: pass (identity lukasstraub2@web.de) X-ZM-MESSAGEID: 1683507250083100001 --Sig_/I3MFa6umhU/ggOdWlvjgxSS Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" The overhead of the mutex in non-multifd mode is negligible, because in that case its just the single thread taking the mutex. This will be used in the next commits to add colo support to multifd. Signed-off-by: Lukas Straub --- migration/ram.c | 17 ++++++++++++++--- migration/ram.h | 1 + 2 files changed, 15 insertions(+), 3 deletions(-) diff --git a/migration/ram.c b/migration/ram.c index 5e7bf20ca5..2d3fd2112a 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -3633,6 +3633,18 @@ static ram_addr_t host_page_offset_from_ram_block_of= fset(RAMBlock *block, return ((uintptr_t)block->host + offset) & (block->page_size - 1); } =20 +void colo_record_bitmap(RAMBlock *block, ram_addr_t *normal, uint normal_n= um) +{ + qemu_mutex_lock(&ram_state->bitmap_mutex); + for (int i =3D 0; i < normal_num; i++) { + ram_addr_t offset =3D normal[i]; + ram_state->migration_dirty_pages +=3D !test_and_set_bit( + offset >> TARGET_PAGE_BITS, + block->bmap); + } + qemu_mutex_unlock(&ram_state->bitmap_mutex); +} + static inline void *colo_cache_from_block_offset(RAMBlock *block, ram_addr_t offset, bool record_bitmap) { @@ -3650,9 +3662,8 @@ static inline void *colo_cache_from_block_offset(RAMB= lock *block, * It help us to decide which pages in ram cache should be flushed * into VM's RAM later. */ - if (record_bitmap && - !test_and_set_bit(offset >> TARGET_PAGE_BITS, block->bmap)) { - ram_state->migration_dirty_pages++; + if (record_bitmap) { + colo_record_bitmap(block, &offset, 1); } return block->colo_cache + offset; } diff --git a/migration/ram.h b/migration/ram.h index 6fffbeb5f1..887d1fbae6 100644 --- a/migration/ram.h +++ b/migration/ram.h @@ -82,6 +82,7 @@ int colo_init_ram_cache(void); void colo_flush_ram_cache(void); void colo_release_ram_cache(void); void colo_incoming_start_dirty_log(void); +void colo_record_bitmap(RAMBlock *block, ram_addr_t *normal, uint normal_n= um); =20 /* Background snapshot */ bool ram_write_tracking_available(void); --=20 2.39.2 --Sig_/I3MFa6umhU/ggOdWlvjgxSS Content-Type: application/pgp-signature Content-Description: OpenPGP digital signature -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEg/qxWKDZuPtyYo+kNasLKJxdslgFAmRYR9EACgkQNasLKJxd slgizxAAnPToGeBo8BuYUwmb6px8xRhcqxYavf5CaHIvTS7yk7gDBxa579QG4rb6 Syyoo1n0jIwHXyOoy+8xLZw/FcU5vI4aS74OmNnqqiDX6EVe5LCRRUx8iNoziMIq ULjGSsk3I5Wb+X5vwi2O/sXQVwcsqsGSxGPm23qjB+gsYZoE6ZvLfMh6IG4bFNuO XmCB/XEKP4fhxW7RiAbTDhHACriZyiyPOPRAKgHIdniUZeLz34VXL2QTIpiqvM0U KZmplGCGJEctVNysqIMWlqTEyoXKT+K1ZFm0/HCr8qB9ApGi67YY9vE3HB4P96US vRdz84JvcHSiau2ySSWY94U7B+O84GWxH0uOmNK05wSlCVyAf2KI4P8yFfQBSEkv wPgV0QaojkY5kDH1H/UNHn4EypcYnWDtjXGxiImS+Ky6YviuShxCC8pARyVPQELx y8ZC+0Mv4sZXAdC4sIPN38W2y0FtiOiZakbw/bKkAGMbUHZAg+MlJ6H3vTidv60d IRkyHyYzWZgNiF1FzrsF4KzS+a5guxqkaPn4ssc3OsD+VAi3jW1VHvllovKfMzMs kkMDjCgntxsBUQqKhYy85Ar+LRz0ofIW8DfPbINAFyWvR11ljc6Vu6KNvbtU3n+N QuOm1UotS8DT0RhibWWTXeUuQa8/aMIcYmlRLX4DmHdUjgsTWBo= =639H -----END PGP SIGNATURE----- --Sig_/I3MFa6umhU/ggOdWlvjgxSS-- From nobody Thu May 9 22:02:47 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass header.i=lukasstraub2@web.de; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=web.de ARC-Seal: i=1; a=rsa-sha256; t=1683507201; cv=none; d=zohomail.com; s=zohoarc; b=eiOUoKOjRs//Sgsj8HGbZBpk78tO3a1cexVTucN42xMzib+o7jiRRQi1GB803FXtGyBUG1jsrB4zqGKVntTixt8sjp6IMqE5RWBMSukQ/NWRS4R56NQVD+hf5YiVbJ9HCncjaO6G6+nksWXKrUaJ0RTaaHMHLJxl6QduY8xDaqw= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1683507201; h=Content-Type:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=m+DzkYKmnEGzPASKc3B20wJuHdoN1Qs8PWDE7d51lvQ=; b=GidZcgewD+b4CAuXBjypeRfp2gdvy8vESAjbAr/IrS9CKj2WI0qOKm8k/S8jTbhySAP8nkwS5mYR5Yd9dC9CSynd5l8Zz/t3eR9flKDK7xD4oiADFg7UMMNlvRbFiWK3y/GTPjTn1lmrl1ZDpZoFcf/zAJEn3XuL5bDGs1YFWkk= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass header.i=lukasstraub2@web.de; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 168350720161414.237924236411232; Sun, 7 May 2023 17:53:21 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pvp7Q-00071w-59; Sun, 07 May 2023 20:52:48 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pvp7O-00071K-8e for qemu-devel@nongnu.org; Sun, 07 May 2023 20:52:46 -0400 Received: from mout.web.de ([212.227.15.3]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pvp7M-0004bK-M9 for qemu-devel@nongnu.org; Sun, 07 May 2023 20:52:46 -0400 Received: from gecko.fritz.box ([82.207.254.115]) by smtp.web.de (mrweb005 [213.165.67.108]) with ESMTPSA (Nemesis) id 1N8Vsr-1qIDQP1wKv-014VGZ; Mon, 08 May 2023 02:52:40 +0200 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=web.de; s=s29768273; t=1683507160; i=lukasstraub2@web.de; bh=IF2rLFsIMJ/UTQ+z/EK3wC/8lHx5BXBAM9BnhOAPBLc=; h=X-UI-Sender-Class:Date:From:To:Cc:Subject:In-Reply-To:References; b=QOFxp/jLKXQ70fvwP1f7Mqwo3CewyWVEVH6NWmhOU5CIYoMIUJbhJGPHVhZ4OXHuU C/hITLhwFU0C8P9XQ9ii/0KGYZzTaIREzWQLaj1Mq8c9x8x9LEQwLGKEQzkXeVRHHl seHtjgEymIjkctjtpJZPQD/9CVRSrQvxdzrejY48I/sdbhIg/RIp0uowt+b0UkBgnp WAh5Ogcu5gxYjOJkZKmiBtw29Wdwp9MP51UVLYR0F9QtcIY296HVS5QW2+euYjnHgj m2PZpa1qELE7XqLzUsKGT0Ucwq465oQSxUrqmxzwZ3I/jJVfrQxSz2qawGqaBitN+K AJSccUR5WJFow== X-UI-Sender-Class: 814a7b36-bfc1-4dae-8640-3722d8ec6cd6 Date: Mon, 8 May 2023 02:52:39 +0200 From: Lukas Straub To: qemu-devel Cc: Juan Quintela , Peter Xu , Leonardo Bras Subject: [PATCH 2/5] ram: Let colo_flush_ram_cache take the bitmap_mutex Message-ID: In-Reply-To: References: MIME-Version: 1.0 Content-Type: multipart/signed; boundary="Sig_/8gpWrRVMZCwlNjXxelYuQUI"; protocol="application/pgp-signature"; micalg=pgp-sha256 X-Provags-ID: V03:K1:43EvQ0cPrAbJ+gxTI9+sZuUeDDpjymSecUhermJzPgkIKIzIzRm UaqbM7q0sNjQfsf2N1kISlHseVEE2Kiqd3S934jZVgewkTTk3Diy9jR1tTXJlc4GoB0uN/d bfZg0/MBvc0UBPefSkzPeQuPbh0oEtp6V2SAFuujDU+Uuaec63uX86PKc5gGCBHA/HnJgNY veyvPT+hCex2YueUnYwxA== UI-OutboundReport: notjunk:1;M01:P0:lY0/pg5DNNo=;Cla8xEX0M58NcwASk6YT7OaEkCr 6YMaKmeWfqwTEuhbfwYIpXGmqQ6etMhs1HfHVYhn15ZWqpPENLmChOxHhLiWfoG1+uKmFhr5V xB6bTsEjFtEwRgj1B6aFCG3Q9BTrGveWIT0LwizujsCI42+tJ6fPDjiHoQDEXVraTL8JQyC3p DwuXZnNqFg2HqY9GusFjwQqVItqS+DxwEI3AF0wHYGy3DM/RvKok+vvLFrGeIxN3pzUD20iqt cM5CJeD4yKLTjcIqyO8ixAq8/+wq8Qmn3GLS3NhafUa0WXZYs+Dz/hkbgooSfz8bitTLrPddE 2XxGY2NtZ4zmgp+mguLa70pOAsEZub/JhPOds4nPUGXQZ8d+Pzo/h5oFrx2wwCkKzabn+F7bN NZCAiW8yUftuCRDbrTXeomurSzMOtFeOH7aU6DJU/cueF7IljyxoaBzCTIXNycGenLCIZRsDi 2tANBMtKlLzLpSE5OLvCfOhHomsk1soW976gBcD6xfV/nPqvyZnYcOZjkI76giitJ0LZHhfY4 s9iNT0ChRQ6uQJ5xj5Wk5JbsMH7TkmhnNVCXt+JWLy0Ms1yYMLyhfxsIt7J7W394/upn1a33i fF3EIXQTP1xkL58Hl9EPXESaNwMu7l0yKzqnWnGmAt43sQz8KeHhTCx0xuuB81vcyyolppA3O jP36/MAQRy/YcacjPOHI7pwVffmu1q6ZBvCyQ+nQ/Ukk61X2oW+B2ZYcUhy32yxYmPSrjVsdw ZkLjzB1DGtIlWXuwurkeViGtxjk9Szqjbw4wQRxNVF2jjXldAOcm7436vs+bjKzGjDvqX+AAm QYD75DPARTYfImsgnUPJL1pT8bBy8YiMRc+IxYIyNMYk7kC8VKfyLYG2M5rIVps4USjg+xhuZ cQso2Fg+dDRgplaRHFhmJD9lzE1Dl4Cz33vg4hb9cYFFpL0jMFvr/MlhDOjeWPawdH3XJL/Lt mtDCpFkd69NG3tSylllJXS/AMVA= Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=212.227.15.3; envelope-from=lukasstraub2@web.de; helo=mout.web.de X-Spam_score_int: -25 X-Spam_score: -2.6 X-Spam_bar: -- X-Spam_report: (-2.6 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-Zoho-Virus-Status: 1 X-ZohoMail-DKIM: pass (identity lukasstraub2@web.de) X-ZM-MESSAGEID: 1683507202700100005 --Sig_/8gpWrRVMZCwlNjXxelYuQUI Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" This will be used in the next commits to add colo support to multifd. Signed-off-by: Lukas Straub --- migration/ram.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/migration/ram.c b/migration/ram.c index 2d3fd2112a..f9e7aeda12 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -4230,6 +4230,7 @@ void colo_flush_ram_cache(void) unsigned long offset =3D 0; =20 memory_global_dirty_log_sync(); + qemu_mutex_lock(&ram_state->bitmap_mutex); WITH_RCU_READ_LOCK_GUARD() { RAMBLOCK_FOREACH_NOT_IGNORED(block) { ramblock_sync_dirty_bitmap(ram_state, block); @@ -4264,6 +4265,7 @@ void colo_flush_ram_cache(void) } } } + qemu_mutex_unlock(&ram_state->bitmap_mutex); trace_colo_flush_ram_cache_end(); } =20 --=20 2.39.2 --Sig_/8gpWrRVMZCwlNjXxelYuQUI Content-Type: application/pgp-signature Content-Description: OpenPGP digital signature -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEg/qxWKDZuPtyYo+kNasLKJxdslgFAmRYR9cACgkQNasLKJxd slhLeQ/+OzrlEXqJcIImE6nYh3fbEURQMnsr5+YK6XYZljfSwess6od0P36K5WhW nR58S8TCxMCe5KsRM7gZZz/OS0zVJCS7FogE0uIVrUGrBQEnKDn77q525cvkg0Ag DuvQ3SXhQmk5HJWTvQOFWmpLJWLmk38VI5NC7P5fXA96+59EGbQeFOGFMvRo609p izrENiItvkLj7NoLbk9/nY2ICVTm4Ut12DtDjD71Z9Wi5fdK5PshqLwEIi8/iPf7 +9AHV1rJHnqOi6Cf5R63ykQ7CI+RNhvHlMdb0PHmf8vD+Sdr6NOatwbvr4FzPy8w MdnnTA5tYlgMO3J8bTaNzerZgEF0q4NPm1dNd0engQV8LNKUGXLRMI1ilqEh57o7 hCJ6SNUjSiOfaRDu8HJC1EzJ8lXvbzUs3xuF5MFRdgcaZa9sLkQCkl+oztzrudiC NdKyYisvMxxl4cSFRK1Rw7tYyBPkfDZZB1JSyoUDQW1C2LAWNtU7SS6u77W0WN8n 9u8GjSLKe0+/3odd1S/dyh/0QlZtsS3FtEeHQB3RJ2soa7oahpctGXpkGtVf+Ja8 VcdCjrmsTi8u0bEvPqC8+o81Ak+Y7Y1AJxyy8a4PImypHOVmYqiGZptnwR509TL6 Jh3Rb8ZYvlMnv+0p7nYH9+21FU1UBpI76Bu4r5TDCRTQxTGkxEk= =5OqQ -----END PGP SIGNATURE----- --Sig_/8gpWrRVMZCwlNjXxelYuQUI-- From nobody Thu May 9 22:02:47 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass header.i=lukasstraub2@web.de; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=web.de ARC-Seal: i=1; a=rsa-sha256; t=1683507199; cv=none; d=zohomail.com; s=zohoarc; b=KFsym/a4P/xsxJVCmaxy3f0RiS5M5vXARwzuvC/KOg3LSsxPnpAW+KgE4BqnRmfxDr3ZlmiuaRR6y2izEWfi7r9XChFY6Nigh8qYHyybJmKkVj+HORtqFMPZZ0SAFrIpzgUv/vsLbUZhGw62UVEYMwNN9u8pJ1m31teTTpwHS5s= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1683507199; h=Content-Type:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=sQqQcN/xnXAhhxG3gmNsgMNimRt/nvpWT17gqXpUnfs=; b=TOAw27pOUwOnouYsEnlOB4eqImCMh3hH5bQjgFECPCsge5HpZ4Qq1gt5kCh5aKYw0jdlc3HAed+y+ka0VdqPq739FsB4os9r6aFCBAvVbStgywaP4wijG5Q4dd30eTSN4Z8GUJS3FXUJTip+VJne2ruMgKdB15qdsXIBMRuCuUc= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass header.i=lukasstraub2@web.de; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1683507199742127.00007821100678; Sun, 7 May 2023 17:53:19 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pvp7f-00072n-VO; Sun, 07 May 2023 20:53:03 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pvp7Y-00072f-UM for qemu-devel@nongnu.org; Sun, 07 May 2023 20:52:56 -0400 Received: from mout.web.de ([212.227.15.3]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pvp7X-0004cy-9A for qemu-devel@nongnu.org; Sun, 07 May 2023 20:52:56 -0400 Received: from gecko.fritz.box ([82.207.254.115]) by smtp.web.de (mrweb006 [213.165.67.108]) with ESMTPSA (Nemesis) id 1N2SL5-1qM3Aa20b9-013gYG; Mon, 08 May 2023 02:52:51 +0200 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=web.de; s=s29768273; t=1683507171; i=lukasstraub2@web.de; bh=5e+xMzjmHT59gTlPxFHHRCM0p3PCapn6o0VgG/pG2i4=; h=X-UI-Sender-Class:Date:From:To:Cc:Subject:In-Reply-To:References; b=USmHOvmh//Xe+xULCHO6IT2dMnKcsuf7HoUUKoBXt+2RiCJDwRpM8XsjdNk5tvxnN COB+SqHItNwk5fhbpQJGXX5NQjtA4cZ3KHhYpfTWeLdX3qoODi3lrpj9iGNT0FUbeY PTynVYV7UFr//LsF0mf+0yEVU8VoTpMqFxjfNTMjSB8vsTyhJG6EBn+ACZLNNeV9+X tbYrZ0naCIP7vnVVZmQUDff2TkZ8QR3kWFPY+6mI+dF9K7z26IkmPh5ucJumO5bOWE U2V3Ig8jGp6GHeN0jHa2JPM/vrA+Odsp/+3ZxwHjJq3g3SWOqRkbNUGTPl0OqLfn5/ dx8BVWYDzdZsA== X-UI-Sender-Class: 814a7b36-bfc1-4dae-8640-3722d8ec6cd6 Date: Mon, 8 May 2023 02:52:50 +0200 From: Lukas Straub To: qemu-devel Cc: Juan Quintela , Peter Xu , Leonardo Bras Subject: [PATCH 3/5] multifd: Introduce multifd-internal.h Message-ID: <20230508005250.7866d810@gecko.fritz.box> In-Reply-To: References: MIME-Version: 1.0 Content-Type: multipart/signed; boundary="Sig_/gMPqQhQVjMS5XYdYnCmcEbD"; protocol="application/pgp-signature"; micalg=pgp-sha256 X-Provags-ID: V03:K1:a/lgCR8yQmMqaZcjcg4MVB1lACvwzMtS/q4vH4ELVCcCTxD81Iy 1TaVNr9YUQfFx8C1UiE+9HO56yxwGnPSD9KSMlHYr7lOL/7MrPg8As6RPAefetdrZbaNZ1i MplSa4+gZdjNqJFy0vIdYvoCajYnvIT+V2O+Bx6yerQPS3sIkDXmq49sg3gv5TwXf/vKi0B bqZX3i3xrOGnPX9lwd4IA== UI-OutboundReport: notjunk:1;M01:P0:6KM+BGKgoXg=;dqXEZEr5fjIG+qdmZys1yMrwMgO FG+tvhb1IupttxrtZuqle54/ckFJM9E4PG3Iv0qRjtlmwg61FJYbD9OauQC1F5zkn+vsO3Cj5 wPXbVzKQSLJ7UcxdKJPRcyoEz4or/jA7nNHS8AlIJsIRi9bV9adtoqPvCU/kuXhhvZ5DD1QQf ByKSv+uu9IG5cVJ6OfHZCVlcLj0+w41jhYIxqzB5VJypTote0CeuTRhVXU3cL6jSJLkCAbP7D 2vlTdpMTWttiP30YuKY8KLwBY3/RLk+lqA/v0i92UMgzjv6gpSfeJe95D/3PntE8ydA1xWPSS WtMMXwqElzq+Jafg+YRDITDPagSWa2MpujmJyVsd6hKIzdWiuYX7ca4DetXaWYvmJid8A4wAp L58mFXy8nx3Oz13NTI9h1AwR1YKWOuokpqD4821vu/H0s67RgaR8uZLIhPh13Y/X+IaYe2J2l BlSP6fjelYBTfHfufBxEhCHVkrNrwukh6bV7HKEre2VENyrhsa2vzqmWEqQdrVgkSq84g3w2D ZHaQ7P1GLBUVlK4xqwoksocE5RQXBhzm273sCaOh2Q7+TegxdrYqjXDZd/KuGFwjAUX0iJGI8 hcJmKVZH4leyx74MmLD6Y7E5hd54ltvkK+2tbr1R7KdC1aSEI79yv1ZYq4Xx9g9SYJQUZJ2Ta A8pMTVXDv9siEkv2eWEejA6ofAlG0D2p7vPDgGfov9faPDQzyDMr0kjUmywo51lZ1M0q8QyDF ypYceEI6+uDvQz44jxQlfnxDpsAFxt/ffqfVAmR2y4g9Je0ryMxzdv3NSHQUpOg93B0ebIvBW eTv7CuhIqIlYvQsUgNHYSjCTObmiOfXZVL5KfP011FBdvSe+i/HzvYFKzj68ewt3g2olN3b+r jNYaS6XYQec+BWB6Mw4NgHAv22kFbTTAgZqXOnMpCoea8GVzoXJ2KUXIa81ZEc+FvkwTmGtR4 /EfN/xKozBRq7BMUThP10ISiOVY= Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=212.227.15.3; envelope-from=lukasstraub2@web.de; helo=mout.web.de X-Spam_score_int: -25 X-Spam_score: -2.6 X-Spam_bar: -- X-Spam_report: (-2.6 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-Zoho-Virus-Status: 1 X-ZohoMail-DKIM: pass (identity lukasstraub2@web.de) X-ZM-MESSAGEID: 1683507200716100001 --Sig_/gMPqQhQVjMS5XYdYnCmcEbD Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Introduce multifd-internal.h so code that would normally go into multifd.c can go into an extra file. This way, multifd.c hopefully won't grow to 4000 lines like ram.c This will be used in the next commits to add colo support to multifd. Signed-off-by: Lukas Straub --- migration/multifd-internal.h | 34 ++++++++++++++++++++++++++++++++++ migration/multifd.c | 15 ++++----------- 2 files changed, 38 insertions(+), 11 deletions(-) create mode 100644 migration/multifd-internal.h diff --git a/migration/multifd-internal.h b/migration/multifd-internal.h new file mode 100644 index 0000000000..6eeaa028e7 --- /dev/null +++ b/migration/multifd-internal.h @@ -0,0 +1,34 @@ +/* + * Internal Multifd header + * + * Copyright (c) 2019-2020 Red Hat Inc + * + * Authors: + * Juan Quintela + * + * This work is licensed under the terms of the GNU GPL, version 2 or late= r. + * See the COPYING file in the top-level directory. + */ + +#ifdef QEMU_MIGRATION_MULTIFD_INTERNAL_H +#error Only include this header directly +#endif +#define QEMU_MIGRATION_MULTIFD_INTERNAL_H + +#ifndef MULTIFD_INTERNAL +#error This header is internal to multifd +#endif + +struct MultiFDRecvState { + MultiFDRecvParams *params; + /* number of created threads */ + int count; + /* syncs main thread and channels */ + QemuSemaphore sem_sync; + /* global number of generated multifd packets */ + uint64_t packet_num; + /* multifd ops */ + MultiFDMethods *ops; +}; + +extern struct MultiFDRecvState *multifd_recv_state; diff --git a/migration/multifd.c b/migration/multifd.c index 4e71c19292..f6bad69b6c 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -31,6 +31,9 @@ #include "io/channel-socket.h" #include "yank_functions.h" =20 +#define MULTIFD_INTERNAL +#include "multifd-internal.h" + /* Multiple fd's */ =20 #define MULTIFD_MAGIC 0x11223344U @@ -967,17 +970,7 @@ int multifd_save_setup(Error **errp) return 0; } =20 -struct { - MultiFDRecvParams *params; - /* number of created threads */ - int count; - /* syncs main thread and channels */ - QemuSemaphore sem_sync; - /* global number of generated multifd packets */ - uint64_t packet_num; - /* multifd ops */ - MultiFDMethods *ops; -} *multifd_recv_state; +struct MultiFDRecvState *multifd_recv_state; =20 static void multifd_recv_terminate_threads(Error *err) { --=20 2.39.2 --Sig_/gMPqQhQVjMS5XYdYnCmcEbD Content-Type: application/pgp-signature Content-Description: OpenPGP digital signature -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEg/qxWKDZuPtyYo+kNasLKJxdslgFAmRYR+IACgkQNasLKJxd slgp0Q/+NfEAD8XYU/j97Wqx+cq80ocqdmdXq0VtEOMfz9ztgJVAtflQ52apWmng nSYY2rbGg5LCoVNMAiGW5ov+6OuRlxk1o6BkJjKTU1NaXyNPlPctsumvFn+Edpoh KRoh8/9/L0HuKvK2wzpGHwQX1VOoF6NS2LQpEuT4C8hE8b4GynE48NCI+d6WAJG3 dB9vxMNnxx1bdDil4kChWhfQKFXpL7n+4rEsyDD40weNZx5Mfq2ee6M8rc74s5Ew YfKULYuEK/pB+ITh6knBTR+2ZkLwhJtXfUc+4OaZLxPTQ+kTxOgSW4IFwezb4UwV FCkB+yrB6if9qjzAdWp3v/vk2RvrEWWAaLWTGF81XDgdcTjZ7a87kzic5XVuf19l f1pvcwGO1sKuk2MBWVOKzyV/KGLqI31ODOtu3TO+geSIUOv5yHPFnS3DH0CKHPO3 ZMZv6aXt5M+qvqErGfD7N+vU/jfu6RmaG9UJ2xbWhCB4FN51RCV94JtDsJ81CgU3 F66lRECCG2JJ9iVlZkIHOuM6uWbB1vcwx+bN/JpymX/7w4efjd7RYc4OEzy10ecu EjDte0LKgBueuGc6+X4rl/WJe+S1xCsLIAOM7VquorOSTCpGR++Y+HEULWWosGe1 x2miB1KUBXKE9SQJHapIUhP731kY2EmnZc6ZnmYmupZIuKMycGc= =FZWo -----END PGP SIGNATURE----- --Sig_/gMPqQhQVjMS5XYdYnCmcEbD-- From nobody Thu May 9 22:02:47 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass header.i=lukasstraub2@web.de; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=web.de ARC-Seal: i=1; a=rsa-sha256; t=1683507220; cv=none; d=zohomail.com; s=zohoarc; b=ORPctdukjUmKFWqdNxT1z5VjUR6kDuq0odbF9tuzBYu2gCJ5Owy2KYR8AFWVzqd7d9BHGAc0aFf9OTx4cZ1kLN7Ca+Wu2VLmt/r7Lw5VQdsxvn270N5YSLBYEd+E7sREmJwWCBVDEcKg2bobqs2nfOi1xoJHHnUxy/s2fdmg57w= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1683507220; h=Content-Type:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=JcfIf64LecY+mqysnXSpWl8nMEdCZz+n8yXnz+EBd64=; b=AYTID1ENrkGs3GHv/t751Lk2wTnHh5xjzB2j8D/LTx5nEdZqJ/4fASWGb0aGsI01lnt1X4TgDRHnSkEr9jLU0hhT5CQNc+SI/YHRVIh4lNjdLwDNv/ltT2YA3/1vzIGkUlFh6NhItQPN1CZUS3GBRTcCpzOKh/9R4hcD3imdA5A= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass header.i=lukasstraub2@web.de; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1683507220488577.1467423687795; Sun, 7 May 2023 17:53:40 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pvp7j-00074W-EW; Sun, 07 May 2023 20:53:07 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pvp7f-00073Q-Hm for qemu-devel@nongnu.org; Sun, 07 May 2023 20:53:03 -0400 Received: from mout.web.de ([212.227.15.4]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pvp7c-0004dc-Cf for qemu-devel@nongnu.org; Sun, 07 May 2023 20:53:02 -0400 Received: from gecko.fritz.box ([82.207.254.115]) by smtp.web.de (mrweb006 [213.165.67.108]) with ESMTPSA (Nemesis) id 1MdwNW-1qWnle2D5i-00bJoO; Mon, 08 May 2023 02:52:56 +0200 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=web.de; s=s29768273; t=1683507176; i=lukasstraub2@web.de; bh=VzcD1vAT2sl9FLRE5djaiokWkXqOmnEfb7dazRiopis=; h=X-UI-Sender-Class:Date:From:To:Cc:Subject:In-Reply-To:References; b=LZ84NMzrdZQXOl6UDmVamp4zFTcP60z+IdYDKyfDU6kTOTiE9cNWtMBTCoM8GlwLC oIKsE1dzh9rC831XjY2scXnRT3/kx9itahEhvbA2aDHpJC/oS8FMClU7od2gFFMHEf yVSXCw1lVm9mRS4px963I0/7/4TzrgAzfqcZsmPZtAifcYk1jdcOGz3i13JMbQpveJ U9wrSxkC1P4PDTbUAAEX/F2p2hjTGK8irwS9jnWeo2OkKxr+qPsWSB5jMe1MSqKFd/ KaRNtMgrR9yBXvfMW3N99Uhpdwqo2xDIyUcaJ9ohFvLq8rXbh2M2x4kLnYB52SMAHS 0LAVBMpWKbNuw== X-UI-Sender-Class: 814a7b36-bfc1-4dae-8640-3722d8ec6cd6 Date: Mon, 8 May 2023 02:52:55 +0200 From: Lukas Straub To: qemu-devel Cc: Juan Quintela , Peter Xu , Leonardo Bras Subject: [PATCH 4/5] multifd: Introduce a overridable revc_pages method Message-ID: In-Reply-To: References: MIME-Version: 1.0 Content-Type: multipart/signed; boundary="Sig_/qeBBQnnHi6_H_tu38Z_ESLu"; protocol="application/pgp-signature"; micalg=pgp-sha256 X-Provags-ID: V03:K1:dek4LtcPWo+liw2WIGKO0Yg2BShSavyQYR3Cvaezc9Obr2yqAwE aNrGsuZoLhT+M7JcjGaqdJs0um/l8SaIIGwA/8f/TGurx4Ztp+Ed4Nbx6MRznq2S+NmU1/S eCj+ZqwJ0c2225grqjgnxzJi5Qng3uMPlPANUkdJqGHnkyTE4Z9Pcmzp8PqewsKO7vf0vuM 8SNrpPQ6M0WudejmlfMkw== UI-OutboundReport: notjunk:1;M01:P0:0r79VP30mgI=;iJdaHxNkAtugZ7glDwGpVzU3rLZ XkWayjIW9j96HEFjKO/CWUib/IaBMXuNu1LKt0s3umgOOsAVTj+CqwpnswPYmfzlrYaNNBsKE MenclW0Hjiy8XL9tx9wiJtZDbuPFwyVs2yMk6EwfycHu3x0+vjmZlGYYWyfTZQrasHy8HBUoR mAsAdx8F8Xeu2TOJs8ZBN1PuSqaAVUMh23wCnT9T6nDafk9PPcZNECZxExazBkdnGxk1Sp6Rb ETwoh7KGq+eNu7jgtNr+2mZIOS1Mi0vGjeclC/oksQ7ZLG0kAnUNJkwZwgGzu8TtfUoBbmF2i uBGbNogiqSodnnGSgYi6j2e882W/IgVM8lDdPXxkOB10MiCgl4xGI++uTqSrfTySRX4t5W0nf rNh8vv4o1dJP3qbzw9hsDHkQkDkVmFmWUo5PMNA8yvA38HCWi6gcn1F+0NQk+70mVTcHm5M01 NTkSFsZrQwpx1STBjUri3OVfsDJz0DvGfrkasaV7varJdIICz7AsV82PdFWJ6UkbkbMiw8baX b8jW9wMmp4GigHBZNoPrtJ5oFaR2c5wy/7u+nI5R0WDBmGWHsCxQ+322G14UrGUC7zB6uRu7U gwGGxjswy0h26LWOoxQ71E8vJ6YIlFrj+SH+985wDTyQX9FCbOYUAm3O1ksYjhy+Uqnsp+Ej8 gZxowcdbuqfIkG5Re64qLyBHgTgEQ5BmY50H21c2QKbzEu3iEeWdrEVS4inznBXFmCEaDP+2b AB1ZxHB3NUQhZN4z81wjjF12S+btnOYnjfrlfL8qMbBkcRrU0cmE4aXYkaWjfm9fse57pOZLx jargiZuSrqmlUVqisz5/g9Z4uMP//QDHLI9az+ywBnrADuNdXihDXgPj+sfSf+kWqbvJgW18r WIpFW9QjTVb8rSFR5thWM33mNz8ywXyaeLR9MU4Z6XV/vPOnyLPkBC+15+SPX4AAj3a7Z8Hv7 wdz7fg+2U31f/ImESFXuF2zPCB4= Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=212.227.15.4; envelope-from=lukasstraub2@web.de; helo=mout.web.de X-Spam_score_int: -25 X-Spam_score: -2.6 X-Spam_bar: -- X-Spam_report: (-2.6 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-Zoho-Virus-Status: 1 X-ZohoMail-DKIM: pass (identity lukasstraub2@web.de) X-ZM-MESSAGEID: 1683507221785100003 --Sig_/qeBBQnnHi6_H_tu38Z_ESLu Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" This allows to override the behaviour around recv_pages. Think of it like a "multifd_colo" child class of multifd. This will be used in the next commits to add colo support to multifd. Signed-off-by: Lukas Straub --- migration/meson.build | 1 + migration/multifd-colo.c | 39 +++++++++++++++++++++++++++++ migration/multifd-internal.h | 5 ++++ migration/multifd.c | 48 ++++++++++++++++++++++++++++-------- 4 files changed, 83 insertions(+), 10 deletions(-) create mode 100644 migration/multifd-colo.c diff --git a/migration/meson.build b/migration/meson.build index da1897fadf..22ab6c6d73 100644 --- a/migration/meson.build +++ b/migration/meson.build @@ -23,6 +23,7 @@ softmmu_ss.add(files( 'migration.c', 'multifd.c', 'multifd-zlib.c', + 'multifd-colo.c', 'options.c', 'postcopy-ram.c', 'savevm.c', diff --git a/migration/multifd-colo.c b/migration/multifd-colo.c new file mode 100644 index 0000000000..c035d15e87 --- /dev/null +++ b/migration/multifd-colo.c @@ -0,0 +1,39 @@ +/* + * multifd colo implementation + * + * Copyright (c) Lukas Straub + * + * This work is licensed under the terms of the GNU GPL, version 2 or late= r. + * See the COPYING file in the top-level directory. + */ + +#include "qemu/osdep.h" +#include "exec/target_page.h" +#include "exec/ramblock.h" +#include "qemu/error-report.h" +#include "qapi/error.h" +#include "ram.h" +#include "multifd.h" +#include "io/channel-socket.h" + +#define MULTIFD_INTERNAL +#include "multifd-internal.h" + +static int multifd_colo_recv_pages(MultiFDRecvParams *p, Error **errp) +{ + return multifd_recv_state->ops->recv_pages(p, errp); +} + +int multifd_colo_load_setup(Error **errp) +{ + int ret; + + ret =3D _multifd_load_setup(errp); + if (ret) { + return ret; + } + + multifd_recv_state->recv_pages =3D multifd_colo_recv_pages; + + return 0; +} diff --git a/migration/multifd-internal.h b/migration/multifd-internal.h index 6eeaa028e7..82357f1d88 100644 --- a/migration/multifd-internal.h +++ b/migration/multifd-internal.h @@ -29,6 +29,11 @@ struct MultiFDRecvState { uint64_t packet_num; /* multifd ops */ MultiFDMethods *ops; + /* overridable recv method */ + int (*recv_pages)(MultiFDRecvParams *p, Error **errp); }; =20 extern struct MultiFDRecvState *multifd_recv_state; + +int _multifd_load_setup(Error **errp); +int multifd_colo_load_setup(Error **errp); diff --git a/migration/multifd.c b/migration/multifd.c index f6bad69b6c..fb5e8859de 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -1126,7 +1126,7 @@ static void *multifd_recv_thread(void *opaque) qemu_mutex_unlock(&p->mutex); =20 if (p->normal_num) { - ret =3D multifd_recv_state->ops->recv_pages(p, &local_err); + ret =3D multifd_recv_state->recv_pages(p, &local_err); if (ret !=3D 0) { break; } @@ -1152,20 +1152,12 @@ static void *multifd_recv_thread(void *opaque) return NULL; } =20 -int multifd_load_setup(Error **errp) +int _multifd_load_setup(Error **errp) { int thread_count; uint32_t page_count =3D MULTIFD_PACKET_SIZE / qemu_target_page_size(); uint8_t i; =20 - /* - * Return successfully if multiFD recv state is already initialised - * or multiFD is not enabled. - */ - if (multifd_recv_state || !migrate_multifd()) { - return 0; - } - thread_count =3D migrate_multifd_channels(); multifd_recv_state =3D g_malloc0(sizeof(*multifd_recv_state)); multifd_recv_state->params =3D g_new0(MultiFDRecvParams, thread_count); @@ -1204,6 +1196,42 @@ int multifd_load_setup(Error **errp) return 0; } =20 +static int multifd_normal_recv_pages(MultiFDRecvParams *p, Error **errp) +{ + return multifd_recv_state->ops->recv_pages(p, errp); +} + +static int multifd_normal_load_setup(Error **errp) +{ + int ret; + + ret =3D _multifd_load_setup(errp); + if (ret) { + return ret; + } + + multifd_recv_state->recv_pages =3D multifd_normal_recv_pages; + + return 0; +} + +int multifd_load_setup(Error **errp) +{ + /* + * Return successfully if multiFD recv state is already initialised + * or multiFD is not enabled. + */ + if (multifd_recv_state || !migrate_multifd()) { + return 0; + } + + if (migrate_colo()) { + return multifd_colo_load_setup(errp); + } else { + return multifd_normal_load_setup(errp); + } +} + bool multifd_recv_all_channels_created(void) { int thread_count =3D migrate_multifd_channels(); --=20 2.39.2 --Sig_/qeBBQnnHi6_H_tu38Z_ESLu Content-Type: application/pgp-signature Content-Description: OpenPGP digital signature -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEg/qxWKDZuPtyYo+kNasLKJxdslgFAmRYR+cACgkQNasLKJxd slgAGhAAss651RBtH1wSAdC83uI646V6Jk8tF3OSaWjL6mtVyspEKe8uB3trZOsL iGliU1KyCi7JPmruZ70PF6Efo+zpKrC0aKF61eEw9Fx/GYUHP7nRtT4ZcsGbJXsU 4snpF8ceKvaxskbq4UHIILninoRHW/bj9Ci+colkCsDcRFMgij5DuxhyrfFc/VWE LAFINAO/fjb37AAm7q9U4JIO5sCgoO6yHcaQ7bdp7rF0YRyHIRrOOzs1cxWPOEeX epGi3VXHH7/rQhfWyokP4E27uALgokWspdT+f2+zV9viqJrNSMVCTm8iVnkxb+O4 QcNuPHUM7cmF2N+s01mFw5XKkRqAG2rMbUS7vnsgeDw1c04xIROvYIR+P2Kz+L1S w5g7kHsEQceUnKrNow2zPZwpIFPHtlHnsXL4X4WHOJ9XnBwXgsKCfpVZV13Fqhsi 69YoKzyHm5ryPHdI3+nEuYahwvSagpXAIUtTPdY2T8/rToDvJH/Cr9POfed/yiYj e9sJDKsxt1XTTLQh0NgXYgshZSWQYFZjyf6GhSUBi8XYMhQRQ72QPkpbQMsPWkj/ 2dunq8FzZVqGnqwuAR8oCVNAChyDK+ELTyiTkHGDkmeSxPfxIZNk16z4eoYywvW2 FXIWe601/pri+6ZstmhJob1P2dW40221N40S8/5CB7hJy/5HZ78= =EMhA -----END PGP SIGNATURE----- --Sig_/qeBBQnnHi6_H_tu38Z_ESLu-- From nobody Thu May 9 22:02:47 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass header.i=lukasstraub2@web.de; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=web.de ARC-Seal: i=1; a=rsa-sha256; t=1683507245; cv=none; d=zohomail.com; s=zohoarc; b=B+vzzUYBXyzv+tPoHnd5zWnGJ93Db+Ge79/V19RcfhI4paBIBMKGn1NHXoawRs3zsrkEaZ7Wy1+R7uWAUbpCR3cg61P5ZhV8bieS7493d/Wv97ZLC+4Nf55qPOhtHuCGmH3rxNtODulksgRpGomFae2rLNe3FnqMxqRNl6uNVxg= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1683507245; h=Content-Type:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=A2MIwLwfx5P/bNAjYBrwb9K4UTpRTixo0egH+hFWYUA=; b=Vk3lyLkBV39Ap/H+hMhKdPCexiB7yGf69IOTHSmqbM57bECIh5vpAIl9nfsro02jxSS1D6z/rJaHHwEewzlCfZw/8xE6uDYf1eH26dJGdTUGBbF0IH5inD+Qb4/NUUH6OlZ4veSn11v8qo2hHy32G8QYhBxzu8Lf/HwQbtLCcZY= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass header.i=lukasstraub2@web.de; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 168350724534567.31200042311036; Sun, 7 May 2023 17:54:05 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pvp7p-00074n-Nx; Sun, 07 May 2023 20:53:13 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pvp7i-00074K-EU for qemu-devel@nongnu.org; Sun, 07 May 2023 20:53:06 -0400 Received: from mout.web.de ([212.227.15.14]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pvp7g-0004e0-Mi for qemu-devel@nongnu.org; Sun, 07 May 2023 20:53:06 -0400 Received: from gecko.fritz.box ([82.207.254.115]) by smtp.web.de (mrweb006 [213.165.67.108]) with ESMTPSA (Nemesis) id 1N6Jxd-1qG1Yr1Q2T-016PW0; Mon, 08 May 2023 02:53:00 +0200 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=web.de; s=s29768273; t=1683507180; i=lukasstraub2@web.de; bh=QqQu9UpnrMOOeENfQVOCRiuOPULjx7ymu7Dgeocpbcg=; h=X-UI-Sender-Class:Date:From:To:Cc:Subject:In-Reply-To:References; b=lWnrNpqRoTQXwBUZ95y3s3d9CF64iLhr5ilELhUELnftdEJhLQYfiU+uVVwttpZKr +/VlF9sXBxQJ87WFDUAnwgdTk1r/S7DhRvNf8CPAtn985bjoUq28r8lBKaHp+AS7Jv w7+uK4UT7U/rK8KO9yz3PDj8wjzAjG2oOYbJtgqST7JOa/F7mzuOk6ULeOtys1avgt XRdy3CGyd3eeiqw0EbSJuOV/ArH0NxPhjrl3eLsJbnsUdllYbSA6nzLSxNKmQhwlQ7 gHMZzYJn/risJ+1t/C06Otpc6td6n4M7fr4gEq/eiIla8GgPAgahHETOiKc3ikGlYU /m4e29sE7EFXQ== X-UI-Sender-Class: 814a7b36-bfc1-4dae-8640-3722d8ec6cd6 Date: Mon, 8 May 2023 02:52:59 +0200 From: Lukas Straub To: qemu-devel Cc: Juan Quintela , Peter Xu , Leonardo Bras Subject: [PATCH 5/5] multifd: Add colo support Message-ID: <6461809211524b24210fd4ac2ea6b2593fc7538f.1683506133.git.lukasstraub2@web.de> In-Reply-To: References: MIME-Version: 1.0 Content-Type: multipart/signed; boundary="Sig_/uAkkJE8iJQBkQhm3pEjlxLi"; protocol="application/pgp-signature"; micalg=pgp-sha256 X-Provags-ID: V03:K1:tZW2VOOupCD3OHINsQ1OyaW7u1OgAR1Qz9Xu8UvpXlkiVcwgx97 cz6WbUzXCMKL7dhRNA9crum+fG0Xg7wsX4vWIGsS8fL5g4YSRMMaJ+4o3Cf4Fv6CqwLbHWr FcrdWWFKPugjRxcHEN+xoq7msuH0sEYRN7Pme2uNHRH4B04bM2GCWmny0gwwjKdR0mAPW9H EoqlZbqhMdkwedcmD5SBA== UI-OutboundReport: notjunk:1;M01:P0:jN7hdHy0iW8=;yjQO8qvhTXsNKZ9pWQL9wP6K6UX 938Pm8gpTwlNjNeQWZ8Q3l6jIHM5s+/hfFG8IKUJ8UJV0QOTOT7xL5Mj3J1pKiCXd6mpi+e32 og3FUHdYdMKrGSXvL96Wq0GrQpHoXZQBmBBL06Iz3PfUw6R4iSrF2LuAXFbVQa1/TNX7s0ttq nCDvibUN5A4Qh1djuk8S4xFNSgUevyxN1ZLC3igeR1qVCMwVrAwlQVYc7DcGq/hSe5AhaIO7D SqsV2ZUnrCIeGMWAjOBJoX51LhRfS/WuXkTzD/T6ocM4TwXQO++yfvmIk1JeVmrLDVIs3pvIr /Vzd6b3LGHOIuzAkKITZZN16Q4QRMNW673uT/+z2FDjK9bVSPs82M0SBetDlQqGvba82jmGHA CtZjTi7aa5HALUTbCXnmBci8Khht51OH/YLTO+jqW9yysnRg4LXxpZxZadEJkK3j/xKGOayHS zSusYantLAcV6SNmAcFL9+R1scbnAMDvLJN/Ch9GlLOfZDxoWS8aQn1lT1Wn9ET6Tl0SMauHc GIEbI11tC1Mj/TIp6t8AFMjSfZFoktOCTeeDzGxzXRlvXgoVMbFhPl8I7y5w253iXEXVkDncT d+CHFZV/N5rtwfK7F8bQG8ZFYICSvfHxQJc7+cFYqEYtH4LgDzGAJG3DuR/Ua1mJkFQ6B+/J2 wMP5YdfbRQrVMlAbSEA6FTgj0nDQN8CI7Vq4Vrq3qVZ/mpP5VQSDIE03wpoXEXhfJqVwsXJ6K +Ec8W+LQ59Di1LN029Ov2kH0Sba/Tilp7/s0lLOl0zBNFmNfhJjN8wc82R1UYiGmwFf2wiY1E WbPrm9HNiGFa7z3v+J1nXE1D9nTHZiEW4Aftiz4aX4ZYYc4c3eaPndJRd4QXIb75b+ty68Sq5 YTma+R1b7CPbOqtq/7nAEI7086G3wQ0jnx3TvFSHuLvtG8rzC7tty6U5NA0mrn+rjkUikH5EB eQvxPDGUVbbX0z3CPvEju7dFEIY= Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=212.227.15.14; envelope-from=lukasstraub2@web.de; helo=mout.web.de X-Spam_score_int: -25 X-Spam_score: -2.6 X-Spam_bar: -- X-Spam_report: (-2.6 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-Zoho-Virus-Status: 1 X-ZohoMail-DKIM: pass (identity lukasstraub2@web.de) X-ZM-MESSAGEID: 1683507252750100001 --Sig_/uAkkJE8iJQBkQhm3pEjlxLi Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Signed-off-by: Lukas Straub --- migration/multifd-colo.c | 30 +++++++++++++++++++++++++++++- migration/multifd.c | 11 +++++------ migration/multifd.h | 2 ++ 3 files changed, 36 insertions(+), 7 deletions(-) diff --git a/migration/multifd-colo.c b/migration/multifd-colo.c index c035d15e87..305a1b7000 100644 --- a/migration/multifd-colo.c +++ b/migration/multifd-colo.c @@ -15,13 +15,41 @@ #include "ram.h" #include "multifd.h" #include "io/channel-socket.h" +#include "migration/colo.h" =20 #define MULTIFD_INTERNAL #include "multifd-internal.h" =20 static int multifd_colo_recv_pages(MultiFDRecvParams *p, Error **errp) { - return multifd_recv_state->ops->recv_pages(p, errp); + int ret =3D 0; + + /* + * While we're still in precopy mode, we copy received pages to both g= uest + * and cache. No need to set dirty bits, since guest and cache memory = are + * in sync. + */ + if (migration_incoming_in_colo_state()) { + colo_record_bitmap(p->block, p->normal, p->normal_num); + } + + p->host =3D p->block->colo_cache; + ret =3D multifd_recv_state->ops->recv_pages(p, errp); + if (ret !=3D 0) { + p->host =3D p->block->host; + return ret; + } + + if (!migration_incoming_in_colo_state()) { + for (int i =3D 0; i < p->normal_num; i++) { + void *guest =3D p->block->host + p->normal[i]; + void *cache =3D p->host + p->normal[i]; + memcpy(guest, cache, p->page_size); + } + } + + p->host =3D p->block->host; + return ret; } =20 int multifd_colo_load_setup(Error **errp) diff --git a/migration/multifd.c b/migration/multifd.c index fb5e8859de..fddbf86596 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -284,7 +284,6 @@ static void multifd_send_fill_packet(MultiFDSendParams = *p) static int multifd_recv_unfill_packet(MultiFDRecvParams *p, Error **errp) { MultiFDPacket_t *packet =3D p->packet; - RAMBlock *block; int i; =20 packet->magic =3D be32_to_cpu(packet->magic); @@ -334,21 +333,21 @@ static int multifd_recv_unfill_packet(MultiFDRecvPara= ms *p, Error **errp) =20 /* make sure that ramblock is 0 terminated */ packet->ramblock[255] =3D 0; - block =3D qemu_ram_block_by_name(packet->ramblock); - if (!block) { + p->block =3D qemu_ram_block_by_name(packet->ramblock); + if (!p->block) { error_setg(errp, "multifd: unknown ram block %s", packet->ramblock); return -1; } =20 - p->host =3D block->host; + p->host =3D p->block->host; for (i =3D 0; i < p->normal_num; i++) { uint64_t offset =3D be64_to_cpu(packet->offset[i]); =20 - if (offset > (block->used_length - p->page_size)) { + if (offset > (p->block->used_length - p->page_size)) { error_setg(errp, "multifd: offset too long %" PRIu64 " (max " RAM_ADDR_FMT ")", - offset, block->used_length); + offset, p->block->used_length); return -1; } p->normal[i] =3D offset; diff --git a/migration/multifd.h b/migration/multifd.h index 7cfc265148..a835643b48 100644 --- a/migration/multifd.h +++ b/migration/multifd.h @@ -175,6 +175,8 @@ typedef struct { uint32_t next_packet_size; /* packets sent through this channel */ uint64_t num_packets; + /* ramblock */ + RAMBlock *block; /* ramblock host address */ uint8_t *host; /* non zero pages recv through this channel */ --=20 2.39.2 --Sig_/uAkkJE8iJQBkQhm3pEjlxLi Content-Type: application/pgp-signature Content-Description: OpenPGP digital signature -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEg/qxWKDZuPtyYo+kNasLKJxdslgFAmRYR+sACgkQNasLKJxd slhbiA/5AZcuTVqmqkbIwY8P7HkekgC1wqmMMDRZLoyCKsA/OHZlLiFcZ7oWRRaY qADkC9Hoe40m8irCMS55N8OqRmfTLL9WSDy6FlcqCIrx/fq//myyBAUYRDEJ2x+y 1dWE50WS3ACc0/dYSxLL08cNwqYhNkHQRLhYslumQYfaD4kfKCAi+WgaOCL9oDxB H0uLK4li3yrJsJzXMjTum+/k4XUDRZmz4Xz/yzU8afN51cfNJ7fsI1BMErlGnftV lM93xc+peib5SwYl7Nkwr43J2KU+dOzkKJkuwkcfBJYxtS03SfcxF+7s4mDazONF uV0UpzV/oxjCgoTvuY/K1RAi7GN1ScYptHEXUC15c2Y6zCDYXzn7I0Q+UDSf1jxM 1An//ky99Toj5FiZdUe0iFpElo7UcUy4BSNv1xdDH+kScXrLTVpNc/bbcZuanAdS toMa1kSFSUVxq76300JjLBYiiZEwadQmsMV7+15Y1EqCQE+T7hprwFMPs5Zis4Gk ehJjRD7fuGmo9vlNAYcBD+WRfLsv7XmMVXei0rSiZqGQ4DTaLT6OI1785Vf8ca9s mkOoN3dT9mICXl2ZNhMLsfmNqj8OcsFLTUMYDTplDLgp2vXkrmpOs/46T0CK1yy1 +ZelKuMMauSQFfzgGqO0sJG/Zpp0X1nI0dV2NdMcz4xczJYM6D4= =LAtb -----END PGP SIGNATURE----- --Sig_/uAkkJE8iJQBkQhm3pEjlxLi--