From nobody Sat Nov 23 20:02:00 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=smartx.com ARC-Seal: i=1; a=rsa-sha256; t=1729649470; cv=none; d=zohomail.com; s=zohoarc; b=BnwF9gqX3OQEWIaLR0+n9C5qgqSZ2iYPk2grpqSfqtIm7ZhfPqNSYTvDjzxW9eRCuPW4As8zRcOYl6SCm1yi3aAt0czCPzBa48eScaxuiUqsnegNkGi8oxbnsSxt0lBwOqdmPwP7zBWh83x+AtohXoh4fZWhLkJTbphRC4ciZcQ= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1729649470; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=BcW/CUwOVFNCzVyQxL/mv8bi+/kDzT9IfUfKurNJVmk=; b=ERqBkwoaW6bMddQcV4/Rohw13RVj1FsJFDPmp07kqkz520uBpTuistdZHXa4zyfrP4E7BOjDhCorB5g9wWVSqqCIntsZ/niotX6moLGjmlDWE15eKJfVZ1gG9HRLq8OUoGwX0LJM628IZIDeHzDLWmvQJGLZ8fi3LbDL/K8woa8= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1729649470601496.7862639793268; Tue, 22 Oct 2024 19:11:10 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1t3QpF-0001XZ-AM; Tue, 22 Oct 2024 22:10:17 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1t3QpB-0001XG-Lg for qemu-devel@nongnu.org; Tue, 22 Oct 2024 22:10:13 -0400 Received: from mail-ot1-x32a.google.com ([2607:f8b0:4864:20::32a]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1t3Qp7-0007dZ-Ln for qemu-devel@nongnu.org; Tue, 22 Oct 2024 22:10:11 -0400 Received: by mail-ot1-x32a.google.com with SMTP id 46e09a7af769-7180f2f5fb0so3201065a34.1 for ; Tue, 22 Oct 2024 19:10:09 -0700 (PDT) Received: from Hyman-Dev-Euler.zelin.local ([103.172.41.198]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7eaeab5839dsm5861849a12.47.2024.10.22.19.10.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Oct 2024 19:10:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=smartx-com.20230601.gappssmtp.com; s=20230601; t=1729649408; x=1730254208; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=BcW/CUwOVFNCzVyQxL/mv8bi+/kDzT9IfUfKurNJVmk=; b=PDxssTyIJMCTHY0jdEVQqAHdM9yPviC0XTL5O5LMoybET2Lcbg48Pt2Ao3RL3yBD9G 5bFk3Fb3gA9bO05UpqJu5Qj/jDjjyUxOejvgVkOfwHOdJSbgi+lSt46t/8a5ZCj8j+tc pDJJ79BjPtbPeWmCQcmbyllk+GC9svj16xVm/XvKiBaYDomjLDxkF1DHUFaY4V6n7mgX TMuF/PZScOVTFqL3eedwfYm5i6vDxLXKoLb8v9cFCtn9QA3Tcsjh6ArJjMTGJ050S+85 Ri42eiljyVnpkj3M2FMzw73CNkdb6eFI3QgQYzerZYHGq4gBxGsany27ZcGjgdJ92os7 uoNw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729649408; x=1730254208; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=BcW/CUwOVFNCzVyQxL/mv8bi+/kDzT9IfUfKurNJVmk=; b=HvOLs0oaFPplKvulAqZnj05fpWK3C95imsvIsxtUQtVqkq+Z5HZfAA3V85ystlMuV5 zlDDBW2PPJDVs6MHKiQ8kDCimw8O2yKfhESFGZXltg2+tWEb0ikYGhMm1MqvFN82o7z7 hUIIMcYyLF1mx236wf9HUJEDmhls83lPoEhQ14+1vzsK56kdx9qfacQaNk7JnxmWhdBG 7Sio3LTPcmbCVWwM4K8LasnsELo7Al0qkZdrZ0pWp03iBBGEKffnrO3T/3pp2pd6Sai/ huw3jrXbDbmmMcHbBQkJwnoLa4cUP1g/upkV4FXogWKox8sx87yh3L2TDXKT/M3/789o jXYA== X-Gm-Message-State: AOJu0YwrGVMarFvfODdhEBT9pcxW4PX4rp5M+UBm4DtcmHyGPQBmWY++ qoX0D8wxRCY5+FnDDJE1nRdDufQCD6UFPMIZjx2X79FjX8+QhkF9662ZtNnWUUCFkp3yWM/DhgZ LiPF3iQ== X-Google-Smtp-Source: AGHT+IGKHxHiu3vXEONduE4iDUz/KshhtlVz+8STFHvcTODS+BeaeV8m0FKdtbEtHp3x8AST6iWxdA== X-Received: by 2002:a05:6830:628d:b0:718:1235:8b8a with SMTP id 46e09a7af769-7184b330c4fmr1381320a34.21.1729649407475; Tue, 22 Oct 2024 19:10:07 -0700 (PDT) From: yong.huang@smartx.com To: qemu-devel@nongnu.org Cc: Peter Xu , Fabiano Rosas , Paolo Bonzini , yong.huang@smartx.com Subject: [PATCH RFC 1/2] accel/kvm: Introduce kvm_dirty_log_manual_enabled Date: Wed, 23 Oct 2024 10:09:50 +0800 Message-Id: <6df7d9ca9c90cf021857ef4f06b7f59c3664441b.1729648695.git.yong.huang@smartx.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2607:f8b0:4864:20::32a; envelope-from=yong.huang@smartx.com; helo=mail-ot1-x32a.google.com X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_NONE=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @smartx-com.20230601.gappssmtp.com) X-ZM-MESSAGEID: 1729649471501116600 Content-Type: text/plain; charset="utf-8" From: Hyman Huang Introudce kvm_dirty_log_manual_enabled to indicate if dirty logging manually was enabled in KVM. The kvm_dirty_log_manual_enabled will be used in the next commit. Signed-off-by: Hyman Huang --- accel/kvm/kvm-all.c | 2 ++ include/sysemu/kvm.h | 8 ++++++++ 2 files changed, 10 insertions(+) diff --git a/accel/kvm/kvm-all.c b/accel/kvm/kvm-all.c index 801cff16a5..a0f3e6e493 100644 --- a/accel/kvm/kvm-all.c +++ b/accel/kvm/kvm-all.c @@ -93,6 +93,7 @@ bool kvm_allowed; bool kvm_readonly_mem_allowed; bool kvm_vm_attributes_allowed; bool kvm_msi_use_devid; +bool kvm_dirty_log_manual; static bool kvm_has_guest_debug; static int kvm_sstep_flags; static bool kvm_immediate_exit; @@ -2537,6 +2538,7 @@ static int kvm_setup_dirty_ring(KVMState *s) dirty_log_manual_caps); s->manual_dirty_log_protect =3D 0; } + kvm_dirty_log_manual =3D true; } } =20 diff --git a/include/sysemu/kvm.h b/include/sysemu/kvm.h index c3a60b2890..f38f2818e1 100644 --- a/include/sysemu/kvm.h +++ b/include/sysemu/kvm.h @@ -42,6 +42,7 @@ extern bool kvm_gsi_routing_allowed; extern bool kvm_gsi_direct_mapping; extern bool kvm_readonly_mem_allowed; extern bool kvm_msi_use_devid; +extern bool kvm_dirty_log_manual; =20 #define kvm_enabled() (kvm_allowed) /** @@ -143,6 +144,12 @@ extern bool kvm_msi_use_devid; */ #define kvm_msi_devid_required() (kvm_msi_use_devid) =20 +/** + * kvm_dirty_log_manual_enabled: + * Returns: true if dirty logging manually was enabled. + */ +#define kvm_dirty_log_manual_enabled() (kvm_dirty_log_manual) + #else =20 #define kvm_enabled() (0) @@ -157,6 +164,7 @@ extern bool kvm_msi_use_devid; #define kvm_gsi_direct_mapping() (false) #define kvm_readonly_mem_enabled() (false) #define kvm_msi_devid_required() (false) +#define kvm_dirty_log_manual_enabled() (false) =20 #endif /* CONFIG_KVM_IS_POSSIBLE */ =20 --=20 2.27.0 From nobody Sat Nov 23 20:02:00 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=smartx.com ARC-Seal: i=1; a=rsa-sha256; t=1729649470; cv=none; d=zohomail.com; s=zohoarc; b=nmk3w4GiI3/AE7XXyHkndIgagWBhW7wMgi+qWPajCEKPNLp7zpI29lx0ByhHFNEIsw8CuSxb49PJPxrRHwyXSwLN/1mL8MICISIDawVVpCZUC3Nkf4TcqVuuKr/ZOaiNThPV6WZuN9dgUYizoW3ahsHUxjAdLyQ3qBtUxMSI/5I= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1729649470; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=Hb2OkrkcHnno8Skwipf6IKRUV4pB/qnGZMYm94e+N8U=; b=Cr0t6f3SOKhi7t/myE9lzaIOdmk3lus4g0NolQUbYVYP+f8/j88p5dcxPIiXr0FuLpbZLAdhOyvhtGs5SpRQAs5g9qv1Dbx014x5bNz9yQY6c632X0eI3ukRfJ2RKHk3kGoajwV7Sk85ZDM0MvUlQlDwst4xDLr9L0Z3z5Dc898= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1729649470199528.3608924587674; Tue, 22 Oct 2024 19:11:10 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1t3QpL-0001Ya-0h; Tue, 22 Oct 2024 22:10:23 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1t3QpF-0001Xq-Gn for qemu-devel@nongnu.org; Tue, 22 Oct 2024 22:10:17 -0400 Received: from mail-oi1-x235.google.com ([2607:f8b0:4864:20::235]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1t3QpB-0007dr-E4 for qemu-devel@nongnu.org; Tue, 22 Oct 2024 22:10:15 -0400 Received: by mail-oi1-x235.google.com with SMTP id 5614622812f47-3e600add5dcso2357593b6e.2 for ; Tue, 22 Oct 2024 19:10:12 -0700 (PDT) Received: from Hyman-Dev-Euler.zelin.local ([103.172.41.198]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7eaeab5839dsm5861849a12.47.2024.10.22.19.10.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Oct 2024 19:10:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=smartx-com.20230601.gappssmtp.com; s=20230601; t=1729649411; x=1730254211; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Hb2OkrkcHnno8Skwipf6IKRUV4pB/qnGZMYm94e+N8U=; b=OJqfoMp69tzYYmeXrFVtJ2SL/96+Ub4lCRXrnyZRflKYjIViUmSU4Oh4V+rsO7N/rW lwfVsWTFw8Ckt24du4ZCW5KTQAEJorw2juQ0H95lbWJnUz1LX974EbDR79uDy7Gq/lGJ bhYqAZaf+CzS72X5v5LoItNxdfle61lA7NW/3/gjmNY9NwNeMiftrAO9RbNYsY97rseR t3pNWC3VrgGSGdVNJv5ApH1fRhDt8Asg3nTlV/0iOJ4XGYup61IzHiVwWtTT++Sjc4aW WQKXtYwdcrjG8YzW4tFYzhSXHsm3uCZvwuGW6yu2Qn+DRX8eMCuDM9+OSClmyEblXw5s 0gbA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729649411; x=1730254211; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Hb2OkrkcHnno8Skwipf6IKRUV4pB/qnGZMYm94e+N8U=; b=gb+9ZMtuN2/1p5juPQ09a5q08s0Ahx33ngDZhJ/AK2rJkyXzYGEx5guL5wBmUChzy0 VNO54cxB118UKGfKIim4ff6iLKs21f+9SLUm2uaonIoLiOl7ADVRscvXyoJ17bsv3Puj Xfq3Rk+niv2P/vbu+/iVmFmXdLAMGj0UyhmR1tx0XiMWoJ86+6MkOTu42pR7DMdnFEsO bmoQYcXZraEGYsigYxPDVILyTmB09LmFBgpD4YRj7ZqulHD78D4y6/YjAnvx3rZDHgtl p0O6QYkSzVrMLYDgE4OABclGYBncGz8oF7RFOKuY4OB/rn4lA4rp9EXPMXzgVWr+isio 5zew== X-Gm-Message-State: AOJu0YwKVy3L8pvctBOvebdrl54j1RAgoYHD4yLGQf01Fuv/iYOAx8sH /DAWsMP0UPKKXYIA9UwJ+9syzlPMRF5iBDDEeKGHrTYuuq5kNB3ixMsXryWMasGViBw3yNhpPC8 FQ8wBnA== X-Google-Smtp-Source: AGHT+IH8o55cPEB9JfYynoU0C7q9SflAqczFJsFZ4FJvgvttwPgkKnI7DsNlsuXAxjm5HWLn18fHEg== X-Received: by 2002:a05:6808:23c8:b0:3e6:1356:ad5e with SMTP id 5614622812f47-3e624500ac4mr1076410b6e.13.1729649410173; Tue, 22 Oct 2024 19:10:10 -0700 (PDT) From: yong.huang@smartx.com To: qemu-devel@nongnu.org Cc: Peter Xu , Fabiano Rosas , Paolo Bonzini , yong.huang@smartx.com Subject: [PATCH RFC 2/2] migration: Avoid doing RAMBlock dirty sync in the initial iteration Date: Wed, 23 Oct 2024 10:09:51 +0800 Message-Id: <76f0ea57ac7f4c2a68203d17fec1c34bb3d16a4a.1729648695.git.yong.huang@smartx.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2607:f8b0:4864:20::235; envelope-from=yong.huang@smartx.com; helo=mail-oi1-x235.google.com X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_NONE=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @smartx-com.20230601.gappssmtp.com) X-ZM-MESSAGEID: 1729649471488116600 Content-Type: text/plain; charset="utf-8" From: Hyman Huang KVM always returns 1 when userspace retrieves a dirty bitmap for the first time when KVM_DIRTY_LOG_INITIALLY_SET is enabled; in such scenario, the RAMBlock dirty sync of the initial iteration can be skipped. Signed-off-by: Hyman Huang --- migration/cpu-throttle.c | 3 ++- migration/ram.c | 30 +++++++++++++++++++++++++++--- 2 files changed, 29 insertions(+), 4 deletions(-) diff --git a/migration/cpu-throttle.c b/migration/cpu-throttle.c index 342681cdd4..06e3b1be78 100644 --- a/migration/cpu-throttle.c +++ b/migration/cpu-throttle.c @@ -27,6 +27,7 @@ #include "hw/core/cpu.h" #include "qemu/main-loop.h" #include "sysemu/cpus.h" +#include "sysemu/kvm.h" #include "cpu-throttle.h" #include "migration.h" #include "migration-stats.h" @@ -141,7 +142,7 @@ void cpu_throttle_dirty_sync_timer_tick(void *opaque) * effect on guest performance, therefore omit it to avoid * paying extra for the sync penalty. */ - if (sync_cnt <=3D 1) { + if (sync_cnt <=3D (kvm_dirty_log_manual_enabled() ? 0 : 1)) { goto end; } =20 diff --git a/migration/ram.c b/migration/ram.c index d284f63854..b312ebd69d 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -2718,7 +2718,7 @@ static void ram_list_init_bitmaps(void) { MigrationState *ms =3D migrate_get_current(); RAMBlock *block; - unsigned long pages; + unsigned long pages, clear_bmap_pages; uint8_t shift; =20 /* Skip setting bitmap if there is no RAM */ @@ -2736,6 +2736,7 @@ static void ram_list_init_bitmaps(void) =20 RAMBLOCK_FOREACH_NOT_IGNORED(block) { pages =3D block->max_length >> TARGET_PAGE_BITS; + clear_bmap_pages =3D clear_bmap_size(pages, shift); /* * The initial dirty bitmap for migration must be set with all * ones to make sure we'll migrate every guest RAM page to @@ -2751,7 +2752,17 @@ static void ram_list_init_bitmaps(void) block->file_bmap =3D bitmap_new(pages); } block->clear_bmap_shift =3D shift; - block->clear_bmap =3D bitmap_new(clear_bmap_size(pages, shift)= ); + block->clear_bmap =3D bitmap_new(clear_bmap_pages); + /* + * Set the clear bitmap by default to enable dirty logging. + * + * Note that with KVM_DIRTY_LOG_INITIALLY_SET, dirty logging + * will be enabled gradually in small chunks using + * KVM_CLEAR_DIRTY_LOG + */ + if (kvm_dirty_log_manual_enabled()) { + bitmap_set(block->clear_bmap, 0, clear_bmap_pages); + } } } } @@ -2771,6 +2782,7 @@ static void migration_bitmap_clear_discarded_pages(RA= MState *rs) =20 static bool ram_init_bitmaps(RAMState *rs, Error **errp) { + Error *local_err =3D NULL; bool ret =3D true; =20 qemu_mutex_lock_ramlist(); @@ -2783,7 +2795,19 @@ static bool ram_init_bitmaps(RAMState *rs, Error **e= rrp) if (!ret) { goto out_unlock; } - migration_bitmap_sync_precopy(false); + + if (kvm_dirty_log_manual_enabled()) { + /* + * Bypass the RAMBlock dirty sync and still publish a + * notification. + */ + if (precopy_notify(PRECOPY_NOTIFY_AFTER_BITMAP_SYNC, + &local_err)) { + error_report_err(local_err); + } + } else { + migration_bitmap_sync_precopy(false); + } } } out_unlock: --=20 2.27.0