From nobody Wed Nov 19 12:30:30 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1615983598; cv=none; d=zohomail.com; s=zohoarc; b=WheA6Q9Cht3LbBA8vApDnuPQHD+EhyvhGQH2wQBdxqeJBNx3fYHX6a78uZ5bl1SExNPFtrwwvF2v2hTqGUh1yV3nx4IQqzMybEr+3Qg7l8c5GqRzandpVx5k5I+Z5X0sUQqEoBAUXpVFh6l1mz8AgoYYoeTqrZWilonVgL0k7eQ= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1615983598; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=uioFA7kSJG6dSbTCJnInaEvZmY5RBbbDHDyMmMYCNBw=; b=El7Lb5JYAhZKZgiUJg6ElGa4uwHW4nd+6Eh6koa4+jg/r6q6WhZVrzymE3Lc2iMo63TrghVQ1zxP8Hskgip8QSQRTYu3TYIIDJ01d+SEUcVDPqhg0h8APQ2okz88czlyKs6ThLkr297bop2S9uAYzxCyKM6RgQLh3c49YnlcsHc= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1615983598177362.064645625357; Wed, 17 Mar 2021 05:19:58 -0700 (PDT) Received: from localhost ([::1]:53786 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lMV9Z-0005H6-2V for importer@patchew.org; Wed, 17 Mar 2021 08:19:57 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:57220) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lMV6Y-0002Dm-JI; Wed, 17 Mar 2021 08:16:50 -0400 Received: from mail-ed1-x52f.google.com ([2a00:1450:4864:20::52f]:43670) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lMV6X-0007i9-4f; Wed, 17 Mar 2021 08:16:50 -0400 Received: by mail-ed1-x52f.google.com with SMTP id e7so1949315edu.10; Wed, 17 Mar 2021 05:16:44 -0700 (PDT) Received: from avogadro.lan ([2001:b07:6468:f312:c8dd:75d4:99ab:290a]) by smtp.gmail.com with ESMTPSA id gq25sm11520996ejb.85.2021.03.17.05.16.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 17 Mar 2021 05:16:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=uioFA7kSJG6dSbTCJnInaEvZmY5RBbbDHDyMmMYCNBw=; b=gzheDU8k9NmiDG5YgtzWM2S501Ue3ugVRA2DEITq4Jps0hJFsXaWujVQcbRQBpR4MC UJrg4hpBLKO6aowt67Qiy50aq2zl1saCZbjJ5TMRqtR5Yk4bY9OCw5WXg0isJeHsio0N RqvN2hsCjLKrmLjI3GwyCIjn/UzqQVE+scxo6jPa2Xt9V/rAo2l6vTf6a5nIBJzJZgwH uZxjRv6xDSQli83CFbKjqw2Vcuq4mLqrhhYiEb7Oh9dC3iMrP+U/Vz3qMHg2lxfUHkyy aKRC9AlDlN++rFG3XDRjG85mTT1mNUVRL2/ljfQL2WEhhu++U909gETKeJJzD7QoWwj3 5WzA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :in-reply-to:references:mime-version:content-transfer-encoding; bh=uioFA7kSJG6dSbTCJnInaEvZmY5RBbbDHDyMmMYCNBw=; b=sCX4Os3WOmhp7mjUi6E46q/L0DwbB9YjrNgM85lveoVTUXjiyZYEnMstm5lJHhz5Yn YFswvi9U8il9sRRrANkwV6UIlaOQTz1ZdWb7CjM3aq51cXvRBRGMg9ag6Q0WM9mo9Ag8 TrXOm9iQqQ5+fUrEXQ8U+MtG3GJsveT2e4tubEREJpyG00O5RyS0knfxvU+BKROBNqCm H4LOOluqi6JdlEEuPlqQArwzHMYf+qJMWdU0FULD87naLzMesP00GjQNbsMD2b0RbQnW NPucivHhrj7YIe/KkOuR0n9fwyoGfs//fZ158D7DZ7I3UBsc1wqkbc2g20e5c4Wnf2dY uirQ== X-Gm-Message-State: AOAM532kTMXvfLmGJuXxXE/9/Qns74uvAHbpScyFQjfCxWpSf5M/Ddde epJU/zUzx3R9GLxf3ziztgcYEZF9zPw= X-Google-Smtp-Source: ABdhPJxxEt/D3g0duWrjXpKGsTxGMkk2so1ZxnebEoXMuMryrpsx6gnz8d43YlebTztPuIfjc1FBmw== X-Received: by 2002:aa7:df14:: with SMTP id c20mr41604268edy.197.1615983403683; Wed, 17 Mar 2021 05:16:43 -0700 (PDT) From: Paolo Bonzini To: qemu-devel@nongnu.org Subject: [PATCH 1/6] block/vdi: When writing new bmap entry fails, don't leak the buffer Date: Wed, 17 Mar 2021 13:16:36 +0100 Message-Id: <20210317121641.215714-2-pbonzini@redhat.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210317121641.215714-1-pbonzini@redhat.com> References: <20210317121641.215714-1-pbonzini@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2a00:1450:4864:20::52f; envelope-from=paolo.bonzini@gmail.com; helo=mail-ed1-x52f.google.com X-Spam_score_int: -14 X-Spam_score: -1.5 X-Spam_bar: - X-Spam_report: (-1.5 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FORGED_FROMDOMAIN=0.249, FREEMAIL_FROM=0.001, HEADER_FROM_DIFFERENT_DOMAINS=0.25, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: david.edmondson@oracle.com, kwolf@redhat.com, =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , qemu-block@nongnu.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) From: David Edmondson If a new bitmap entry is allocated, requiring the entire block to be written, avoiding leaking the buffer allocated for the block should the write fail. Reviewed-by: Philippe Mathieu-Daud=C3=A9 Signed-off-by: David Edmondson Message-Id: <20210309144015.557477-2-david.edmondson@oracle.com> Signed-off-by: Paolo Bonzini --- block/vdi.c | 1 + 1 file changed, 1 insertion(+) diff --git a/block/vdi.c b/block/vdi.c index 5627e7d764..2a6dc26124 100644 --- a/block/vdi.c +++ b/block/vdi.c @@ -690,6 +690,7 @@ nonallocating_write: =20 logout("finished data write\n"); if (ret < 0) { + g_free(block); return ret; } =20 --=20 2.29.2 From nobody Wed Nov 19 12:30:30 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1615983898; cv=none; d=zohomail.com; s=zohoarc; b=OH0AdSh38/xV++96IvupYUUy4KW+FDX4eEWVBD4mfQGWqicv/h7pvoCJ3ESpWij8vEOzjqyrdJY3yNZlmsJVF2ak3iVWRbEmDwaDRj08fl99ai/g0ha7OPxBdVgMqz+aVQ2JGdZoG8Jen9Iu/qWvR8eHYdxA6Z+U+mRUkLGIxzA= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1615983898; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=sB3+w5MPiq5YtSsSHBffd76Yw89YjLssV56LPOeeU7o=; b=L0l1cPPyLCcZfQJd3KpYGG/F2Q4npvu7wZSAelnj07WWGUs2dIDmbVN7BTHYxGOCtl+ryFAyrRPg6ef+IAQIEDJRXD5Zw7ueHY8ZEbZyHfsPSljk38fVzM6WSJdya1oiJtQMC62T+rtLJZoCi85iphnFkob5YnRdzEeXdyQ9+8U= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1615983898875403.4482405306974; Wed, 17 Mar 2021 05:24:58 -0700 (PDT) Received: from localhost ([::1]:32872 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lMVEP-0008PN-QE for importer@patchew.org; Wed, 17 Mar 2021 08:24:57 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:57256) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lMV6Z-0002H2-TS; Wed, 17 Mar 2021 08:16:52 -0400 Received: from mail-ej1-x62b.google.com ([2a00:1450:4864:20::62b]:39006) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lMV6X-0007ip-Dv; Wed, 17 Mar 2021 08:16:51 -0400 Received: by mail-ej1-x62b.google.com with SMTP id p7so2239528eju.6; Wed, 17 Mar 2021 05:16:45 -0700 (PDT) Received: from avogadro.lan ([2001:b07:6468:f312:c8dd:75d4:99ab:290a]) by smtp.gmail.com with ESMTPSA id gq25sm11520996ejb.85.2021.03.17.05.16.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 17 Mar 2021 05:16:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=sB3+w5MPiq5YtSsSHBffd76Yw89YjLssV56LPOeeU7o=; b=VPa7OkfIBFuMIPg6IXR6b2rNuIdaNT2vqM7FR5TnUc5IDHTgJku2n8YeUsZdu7QctD fSh2lJ26+N5OxiYx5Pg84+bDVzo6DsOIoaO/xhRvf2bFItf8LANrCbKDx+iO6qSrghke 6FqmsFZwBcO5ger5S/MQhENcDqs80hgwleN3uj1srKBweAKWSYQWZJMTneU88SgO++bP EfoQWRRfi0amPcAXjIU+GsBS9JkbQG+1jXjPMm2Ux361Z9KYLi2z3InUrFNdUhep19ts c9k9W8Y4g5dRfjQmWhAqlM/m5XtT2cwm44u9oGK1qojPvtMS4Nh7ekpAVTkXZD4UYWda mNFQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :in-reply-to:references:mime-version:content-transfer-encoding; bh=sB3+w5MPiq5YtSsSHBffd76Yw89YjLssV56LPOeeU7o=; b=CKRt1cYpZ5yKWnfu7o7MaghVsFpE9PrjepZSQO8YriP8JKEDZR02MGfrnHNtZo8Adv N3jcGzZaqnPNoKSKRgXmn3/hLJyfDtIIRuB9T1qS/OwR5Hj83sBNDDDUANEW/qtffypK jHTdFybTiDVPWa7g0jaoGZK7iZ5JmUOgQRgcEFRx8YK5an9w8oaFw/5xtoLPRYQraUkN Rh+NuIroTCuPz0m+qSHjd+8JBm2PN0fiL99ZvFhXOvEGM7QceJa6TnhNZ0wP8lwSFYCQ gUrgSZ7FRYyRjtTTZAGhfCndklzB/DtO8ZmXri7E84y/xK+RVSBRTg9pwI6tH66vuKSa W42A== X-Gm-Message-State: AOAM532L26pfbvjo5POTj2F17XNZlMwtaCH/ns9xYaRkWtBY+j1qwpMJ R6pPR5NhQEycXkV9HZHtrkkgJhCVFTk= X-Google-Smtp-Source: ABdhPJy3dE9kGJ7QL8WMi98Ffukvg8Y9sTxeBUyR/opbydoGRQTbu9Sl7jgBkJWxc49nRWb5l5BheA== X-Received: by 2002:a17:906:2816:: with SMTP id r22mr34459159ejc.2.1615983404471; Wed, 17 Mar 2021 05:16:44 -0700 (PDT) From: Paolo Bonzini To: qemu-devel@nongnu.org Subject: [PATCH 2/6] block/vdi: Don't assume that blocks are larger than VdiHeader Date: Wed, 17 Mar 2021 13:16:37 +0100 Message-Id: <20210317121641.215714-3-pbonzini@redhat.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210317121641.215714-1-pbonzini@redhat.com> References: <20210317121641.215714-1-pbonzini@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2a00:1450:4864:20::62b; envelope-from=paolo.bonzini@gmail.com; helo=mail-ej1-x62b.google.com X-Spam_score_int: -14 X-Spam_score: -1.5 X-Spam_bar: - X-Spam_report: (-1.5 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FORGED_FROMDOMAIN=0.249, FREEMAIL_FROM=0.001, HEADER_FROM_DIFFERENT_DOMAINS=0.25, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: david.edmondson@oracle.com, kwolf@redhat.com, qemu-block@nongnu.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Type: text/plain; charset="utf-8" From: David Edmondson Given that the block size is read from the header of the VDI file, a wide variety of sizes might be seen. Rather than re-using a block sized memory region when writing the VDI header, allocate an appropriately sized buffer. Signed-off-by: David Edmondson Message-Id: <20210309144015.557477-3-david.edmondson@oracle.com> Signed-off-by: Paolo Bonzini --- block/vdi.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/block/vdi.c b/block/vdi.c index 2a6dc26124..548f8a057b 100644 --- a/block/vdi.c +++ b/block/vdi.c @@ -696,18 +696,20 @@ nonallocating_write: =20 if (block) { /* One or more new blocks were allocated. */ - VdiHeader *header =3D (VdiHeader *) block; + VdiHeader *header; uint8_t *base; uint64_t offset; uint32_t n_sectors; =20 + g_free(block); + header =3D g_malloc(sizeof(*header)); + logout("now writing modified header\n"); assert(VDI_IS_ALLOCATED(bmap_first)); *header =3D s->header; vdi_header_to_le(header); - ret =3D bdrv_pwrite(bs->file, 0, block, sizeof(VdiHeader)); - g_free(block); - block =3D NULL; + ret =3D bdrv_pwrite(bs->file, 0, header, sizeof(*header)); + g_free(header); =20 if (ret < 0) { return ret; --=20 2.29.2 From nobody Wed Nov 19 12:30:30 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1615983512; cv=none; d=zohomail.com; s=zohoarc; b=cp+RPjJxS7b6+ruF5M/xm8wD/FHhJMvtn7pnfa2cxHg1kkXShqz3w4clzY9pEv97Sn+kdpRFaFLkkirj6EYkAhwdkpQ4U1Y1qDxzKJh3ex3/x2zYFKiyTtRz/UQ9l8yOOg+VF8wDNtE2HC/O7D/2ABwfhr2JLzTPGX7czCeG6Zc= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1615983512; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=Xgtdm62nkB77KDEB509RPm6RJ/Eax7jgJ8G9QzqE+DA=; b=TJPyOqcWo0gq6ZBMP8qTpojzhA32b1G7d8V5/e11wWAUFQcj0ln3qovREQJW10i++ZBCIGZ++QnVbjIbPDem1cLi5wEAknGg/MBJz0pr6EwA6EFs+OQ8EM6zRToCoID4uMkmpnWyTCVvZYZZnVMOzMp+B9ubiIVwphUmYZznkp8= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1615983512100615.7535273650316; Wed, 17 Mar 2021 05:18:32 -0700 (PDT) Received: from localhost ([::1]:50284 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lMV8B-0003qa-2F for importer@patchew.org; Wed, 17 Mar 2021 08:18:31 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:57218) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lMV6Y-0002De-FT; Wed, 17 Mar 2021 08:16:50 -0400 Received: from mail-ed1-x52e.google.com ([2a00:1450:4864:20::52e]:34437) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lMV6V-0007iv-7U; Wed, 17 Mar 2021 08:16:50 -0400 Received: by mail-ed1-x52e.google.com with SMTP id y6so1990303eds.1; Wed, 17 Mar 2021 05:16:46 -0700 (PDT) Received: from avogadro.lan ([2001:b07:6468:f312:c8dd:75d4:99ab:290a]) by smtp.gmail.com with ESMTPSA id gq25sm11520996ejb.85.2021.03.17.05.16.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 17 Mar 2021 05:16:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Xgtdm62nkB77KDEB509RPm6RJ/Eax7jgJ8G9QzqE+DA=; b=CkDfnq2nsQUGebocK+FZmp2+91b7X8No6vVzYCfFgLEbdqvE2IBmB9W0D6rFPBR8BZ Wvi1XK9UN78vZ5Qk1kaDT3juFp75BltgQfP13Xwfnew4KdO9SKLOh65O2XGGmu3mPCLA CMx409tlhYU2p/lVhM6SJO312b4AnxTE+dXymMNk45pscieOKPWQHuM87ppAlm2QOY1Z HdARLROnK7eUvgaWHWedPo/huFroXr3dPQFvuRxUZxfE8Jx2Hc3XFv6M8vnw9aZN2eDQ sqycxfjVqh+PybPb8OHej3/+br+OKr/Tn0OaV5bon5W6tbhkVbvDncwETlpFLXd9X5EI xweQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :in-reply-to:references:mime-version:content-transfer-encoding; bh=Xgtdm62nkB77KDEB509RPm6RJ/Eax7jgJ8G9QzqE+DA=; b=X4jp3O7AHLZiaMZw8I86Hsl4Z2sMOlyOkHHYasZ0vsma6r1RD1LT/EHHhTfpF9F0r6 jJBHOLTJmxiSKN/xCEWHsogjXauo0PemoioXllSl8xWvMdTJv3o/iEQ+9mshj8fW/mpO cBb0XERwZ9eL1uShs/pWENYyTTFFMB5ad6A4KidQxRQOUMkUvIjvFM2ir14YBZ4XB3gf Z5sArRkRh2W1NlrwXVACDpOOQLemHP1Ow6oeJWOINNf7yOP27wmJIqzkkss7UyBza/8x EN/EWHEXVDQHoKexUSfxwDup2ITHBrx82d9HpORSj7NTFv95dDC80PqYT+uiMRPPIN9/ AB9g== X-Gm-Message-State: AOAM531rZHGH0jj7yXISXJ2kk1ZVYfx/K5ZDkPqifwlK0ZtvxiKtRoWD OB5m6fUy7L49/jXq4ySffI+jpavtfAk= X-Google-Smtp-Source: ABdhPJw0pEgq1SqlrnTx7Elrj4nB12+aduC3VdHdg14sHvPThfeOGpRR0T+SW5gJHpfLUjmVgG/Lig== X-Received: by 2002:aa7:d954:: with SMTP id l20mr41746369eds.1.1615983405213; Wed, 17 Mar 2021 05:16:45 -0700 (PDT) From: Paolo Bonzini To: qemu-devel@nongnu.org Subject: [PATCH 3/6] coroutine/mutex: Store the coroutine in the CoWaitRecord only once Date: Wed, 17 Mar 2021 13:16:38 +0100 Message-Id: <20210317121641.215714-4-pbonzini@redhat.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210317121641.215714-1-pbonzini@redhat.com> References: <20210317121641.215714-1-pbonzini@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2a00:1450:4864:20::52e; envelope-from=paolo.bonzini@gmail.com; helo=mail-ed1-x52e.google.com X-Spam_score_int: -14 X-Spam_score: -1.5 X-Spam_bar: - X-Spam_report: (-1.5 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FORGED_FROMDOMAIN=0.249, FREEMAIL_FROM=0.001, HEADER_FROM_DIFFERENT_DOMAINS=0.25, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: david.edmondson@oracle.com, kwolf@redhat.com, =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , qemu-block@nongnu.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) From: David Edmondson When taking the slow path for mutex acquisition, set the coroutine value in the CoWaitRecord in push_waiter(), rather than both there and in the caller. Reviewed-by: Paolo Bonzini Reviewed-by: Philippe Mathieu-Daud=C3=A9 Signed-off-by: David Edmondson Message-Id: <20210309144015.557477-4-david.edmondson@oracle.com> Signed-off-by: Paolo Bonzini --- util/qemu-coroutine-lock.c | 1 - 1 file changed, 1 deletion(-) diff --git a/util/qemu-coroutine-lock.c b/util/qemu-coroutine-lock.c index 5816bf8900..eb73cf11dc 100644 --- a/util/qemu-coroutine-lock.c +++ b/util/qemu-coroutine-lock.c @@ -204,7 +204,6 @@ static void coroutine_fn qemu_co_mutex_lock_slowpath(Ai= oContext *ctx, unsigned old_handoff; =20 trace_qemu_co_mutex_lock_entry(mutex, self); - w.co =3D self; push_waiter(mutex, &w); =20 /* This is the "Responsibility Hand-Off" protocol; a lock() picks from --=20 2.29.2 From nobody Wed Nov 19 12:30:30 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1615984003; cv=none; d=zohomail.com; s=zohoarc; b=dWO0FzBVClFQJxn7h3qW7qGCeVtEmLj5tBRGoN8UWnFgRv04tFvUDB2lkgBywBdAicLJRYT6kZfHkAxr1mqIbESbQBrwc+N8VvgZbzZafO+EW/vNsRpKc6RxF7sISTOnj8eJhYB7XiHv7A/38GdaZWxU/39F8e2wdpLG6JwjbZw= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1615984003; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=tYCbgh+KOfRbcZnCn6zMIAeTgkH/DDMxlNrT19NgzJE=; b=c4XfsEa/p7CcKc+WiY3x6KcQ97OYbLxyOHW47gaPMAjDFg4rqbU+5TNau9YLOfasV/1WG3L41qrVugDb/GM2LkZ9Bp2YSM+ZrS8E7I/4a5rtJ9txcR+tGKWpXCQUoMWYorF28ckG6zlnHzbquI587/sgzi0uASFXqp+sHuIXaQM= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1615984003802141.9918626691242; Wed, 17 Mar 2021 05:26:43 -0700 (PDT) Received: from localhost ([::1]:35510 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lMVG6-0001D6-N2 for importer@patchew.org; Wed, 17 Mar 2021 08:26:42 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:57260) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lMV6a-0002Hu-ED; Wed, 17 Mar 2021 08:16:52 -0400 Received: from mail-ej1-x62a.google.com ([2a00:1450:4864:20::62a]:43174) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lMV6X-0007j2-HB; Wed, 17 Mar 2021 08:16:52 -0400 Received: by mail-ej1-x62a.google.com with SMTP id p8so2202351ejb.10; Wed, 17 Mar 2021 05:16:47 -0700 (PDT) Received: from avogadro.lan ([2001:b07:6468:f312:c8dd:75d4:99ab:290a]) by smtp.gmail.com with ESMTPSA id gq25sm11520996ejb.85.2021.03.17.05.16.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 17 Mar 2021 05:16:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=tYCbgh+KOfRbcZnCn6zMIAeTgkH/DDMxlNrT19NgzJE=; b=i0bWDOEELuRTIAM7HbY3woLKx8GHenTkxAIXZ3LQe2w/xKH2Z6itIWrPQV5Ha9P+jA OYmNJ/+uYkgn0GUZ5EPJW7TjtsQYIobeitYHycJsuiQgKWIqys51fCyLY4vyK2a1eesQ nAy9NrB50NOoMD5I8NYikFkqTUzbfRle/ijHvfmbCC3ZPWGXPw47SL9RUb58CnBf/z8P 5yFQimla8jI7W6Wy18V9df5rGLL/p5x80QtwDw1rQHqABJNrPYj+Vts9kXIPwxdXgo+t 6AgJgDHS+mmF79RNHKYpcAUNLi2mQn2cOGQQHYIp1nHmgkbgKj7gdx3InibJd9aouOit uuMg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :in-reply-to:references:mime-version:content-transfer-encoding; bh=tYCbgh+KOfRbcZnCn6zMIAeTgkH/DDMxlNrT19NgzJE=; b=r70XYWytDe6u6I4su7D03BYhIWqHWFjCbK20l4vkv6qsHMhE4vy/JsfbnEiW4XRpD4 xseub1c6OjFQoAM/8kzRWD/4rkkwiExv7bkyzavapRExIKQzo4eoXKuiFVFTBk/cFZ37 bxGK5bn57DN2cHUx3VCR07NIX7gOX74ljYaIYH/1/fgdAiJ+r/0SLS+W3a2lt3hJW+nH mFoPElC4FtlNsuNj6RswE/Lmae4rezZon7xzwNCt/lMurVx0H2+d0ExH8MiqgXNmAo+O J0LRGBbgslUt8vydKI78zENToykjS5miYn+b/HaflI7szWtvZ1b3b6iOoHzHNYIkiQpE r99A== X-Gm-Message-State: AOAM533cB1eKCyqiu21fndBPUloFNiioLDA/5B3r0r3+0iJ8zzR30qaD aKyzFYU6OATLmqo7e+KeoScmD7E8ej4= X-Google-Smtp-Source: ABdhPJxK5QYMAIWTjZF4su3t6cpoAd04+3P5ZVJuxkHGsdrPHh0Ug4HE+hrqGl5Rf4c1ENTOx4aHDA== X-Received: by 2002:a17:906:12db:: with SMTP id l27mr34495455ejb.500.1615983406353; Wed, 17 Mar 2021 05:16:46 -0700 (PDT) From: Paolo Bonzini To: qemu-devel@nongnu.org Subject: [PATCH 4/6] coroutine-lock: reimplement CoRwLock to fix downgrade bug Date: Wed, 17 Mar 2021 13:16:39 +0100 Message-Id: <20210317121641.215714-5-pbonzini@redhat.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210317121641.215714-1-pbonzini@redhat.com> References: <20210317121641.215714-1-pbonzini@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2a00:1450:4864:20::62a; envelope-from=paolo.bonzini@gmail.com; helo=mail-ej1-x62a.google.com X-Spam_score_int: -14 X-Spam_score: -1.5 X-Spam_bar: - X-Spam_report: (-1.5 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FORGED_FROMDOMAIN=0.249, FREEMAIL_FROM=0.001, HEADER_FROM_DIFFERENT_DOMAINS=0.25, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: david.edmondson@oracle.com, kwolf@redhat.com, qemu-block@nongnu.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Type: text/plain; charset="utf-8" An invariant of the current rwlock is that if multiple coroutines hold a reader lock, all must be runnable. The unlock implementation relies on this, choosing to wake a single coroutine when the final read lock holder exits the critical section, assuming that it will wake a coroutine attempting to acquire a write lock. The downgrade implementation violates this assumption by creating a read lock owning coroutine that is exclusively runnable - any other coroutines that are waiting to acquire a read lock are *not* made runnable when the write lock holder converts its ownership to read only. More in general, the old implementation had lots of other fairness bugs. The root cause of the bugs was that CoQueue would wake up readers even if there were pending writers, and would wake up writers even if there were readers. In that case, the coroutine would go back to sleep *at the end* of the CoQueue, losing its place at the head of the line. To fix this, keep the queue of waiters explicitly in the CoRwLock instead of using CoQueue, and store for each whether it is a potential reader or a writer. This way, downgrade can look at the first queued coroutines and wake it only if it is a reader, causing all other readers in line to be released in turn. Reported-by: David Edmondson Signed-off-by: Paolo Bonzini Reviewed-by: David Edmondson --- v3->v4: clean up the code and fix upgrade logic. Fix upgrade comment too. include/qemu/coroutine.h | 17 +++-- util/qemu-coroutine-lock.c | 148 ++++++++++++++++++++++++------------- 2 files changed, 106 insertions(+), 59 deletions(-) diff --git a/include/qemu/coroutine.h b/include/qemu/coroutine.h index 84eab6e3bf..7919d3bb62 100644 --- a/include/qemu/coroutine.h +++ b/include/qemu/coroutine.h @@ -237,11 +237,15 @@ bool qemu_co_enter_next_impl(CoQueue *queue, QemuLock= able *lock); bool qemu_co_queue_empty(CoQueue *queue); =20 =20 +typedef struct CoRwTicket CoRwTicket; typedef struct CoRwlock { - int pending_writer; - int reader; CoMutex mutex; - CoQueue queue; + + /* Number of readers, of -1 if owned for writing. */ + int owners; + + /* Waiting coroutines. */ + QSIMPLEQ_HEAD(, CoRwTicket) tickets; } CoRwlock; =20 /** @@ -260,10 +264,9 @@ void qemu_co_rwlock_rdlock(CoRwlock *lock); /** * Write Locks the CoRwlock from a reader. This is a bit more efficient t= han * @qemu_co_rwlock_unlock followed by a separate @qemu_co_rwlock_wrlock. - * However, if the lock cannot be upgraded immediately, control is transfe= rred - * to the caller of the current coroutine. Also, @qemu_co_rwlock_upgrade - * only overrides CoRwlock fairness if there are no concurrent readers, so - * another writer might run while @qemu_co_rwlock_upgrade blocks. + * Note that if the lock cannot be upgraded immediately, control is transf= erred + * to the caller of the current coroutine; another writer might run while + * @qemu_co_rwlock_upgrade blocks. */ void qemu_co_rwlock_upgrade(CoRwlock *lock); =20 diff --git a/util/qemu-coroutine-lock.c b/util/qemu-coroutine-lock.c index eb73cf11dc..2669403839 100644 --- a/util/qemu-coroutine-lock.c +++ b/util/qemu-coroutine-lock.c @@ -327,11 +327,51 @@ void coroutine_fn qemu_co_mutex_unlock(CoMutex *mutex) trace_qemu_co_mutex_unlock_return(mutex, self); } =20 +struct CoRwTicket { + bool read; + Coroutine *co; + QSIMPLEQ_ENTRY(CoRwTicket) next; +}; + void qemu_co_rwlock_init(CoRwlock *lock) { - memset(lock, 0, sizeof(*lock)); - qemu_co_queue_init(&lock->queue); qemu_co_mutex_init(&lock->mutex); + lock->owners =3D 0; + QSIMPLEQ_INIT(&lock->tickets); +} + +/* Releases the internal CoMutex. */ +static void qemu_co_rwlock_maybe_wake_one(CoRwlock *lock) +{ + CoRwTicket *tkt =3D QSIMPLEQ_FIRST(&lock->tickets); + Coroutine *co =3D NULL; + + /* + * Setting lock->owners here prevents rdlock and wrlock from + * sneaking in between unlock and wake. + */ + + if (tkt) { + if (tkt->read) { + if (lock->owners >=3D 0) { + lock->owners++; + co =3D tkt->co; + } + } else { + if (lock->owners =3D=3D 0) { + lock->owners =3D -1; + co =3D tkt->co; + } + } + } + + if (co) { + QSIMPLEQ_REMOVE_HEAD(&lock->tickets, next); + qemu_co_mutex_unlock(&lock->mutex); + aio_co_wake(co); + } else { + qemu_co_mutex_unlock(&lock->mutex); + } } =20 void qemu_co_rwlock_rdlock(CoRwlock *lock) @@ -340,13 +380,22 @@ void qemu_co_rwlock_rdlock(CoRwlock *lock) =20 qemu_co_mutex_lock(&lock->mutex); /* For fairness, wait if a writer is in line. */ - while (lock->pending_writer) { - qemu_co_queue_wait(&lock->queue, &lock->mutex); + if (lock->owners =3D=3D 0 || (lock->owners > 0 && QSIMPLEQ_EMPTY(&lock= ->tickets))) { + lock->owners++; + qemu_co_mutex_unlock(&lock->mutex); + } else { + CoRwTicket my_ticket =3D { true, self }; + + QSIMPLEQ_INSERT_TAIL(&lock->tickets, &my_ticket, next); + qemu_co_mutex_unlock(&lock->mutex); + qemu_coroutine_yield(); + assert(lock->owners >=3D 1); + + /* Possibly wake another reader, which will wake the next in line.= */ + qemu_co_mutex_lock(&lock->mutex); + qemu_co_rwlock_maybe_wake_one(lock); } - lock->reader++; - qemu_co_mutex_unlock(&lock->mutex); =20 - /* The rest of the read-side critical section is run without the mutex= . */ self->locks_held++; } =20 @@ -355,69 +404,64 @@ void qemu_co_rwlock_unlock(CoRwlock *lock) Coroutine *self =3D qemu_coroutine_self(); =20 assert(qemu_in_coroutine()); - if (!lock->reader) { - /* The critical section started in qemu_co_rwlock_wrlock. */ - qemu_co_queue_restart_all(&lock->queue); - } else { - self->locks_held--; + self->locks_held--; =20 - qemu_co_mutex_lock(&lock->mutex); - lock->reader--; - assert(lock->reader >=3D 0); - /* Wakeup only one waiting writer */ - if (!lock->reader) { - qemu_co_queue_next(&lock->queue); - } + qemu_co_mutex_lock(&lock->mutex); + if (lock->owners > 0) { + lock->owners--; + } else { + assert(lock->owners =3D=3D -1); + lock->owners =3D 0; } - qemu_co_mutex_unlock(&lock->mutex); + + qemu_co_rwlock_maybe_wake_one(lock); } =20 void qemu_co_rwlock_downgrade(CoRwlock *lock) { - Coroutine *self =3D qemu_coroutine_self(); - - /* lock->mutex critical section started in qemu_co_rwlock_wrlock or - * qemu_co_rwlock_upgrade. - */ - assert(lock->reader =3D=3D 0); - lock->reader++; - qemu_co_mutex_unlock(&lock->mutex); + qemu_co_mutex_lock(&lock->mutex); + assert(lock->owners =3D=3D -1); + lock->owners =3D 1; =20 - /* The rest of the read-side critical section is run without the mutex= . */ - self->locks_held++; + /* Possibly wake another reader, which will wake the next in line. */ + qemu_co_rwlock_maybe_wake_one(lock); } =20 void qemu_co_rwlock_wrlock(CoRwlock *lock) { + Coroutine *self =3D qemu_coroutine_self(); + qemu_co_mutex_lock(&lock->mutex); - lock->pending_writer++; - while (lock->reader) { - qemu_co_queue_wait(&lock->queue, &lock->mutex); + if (lock->owners =3D=3D 0) { + lock->owners =3D -1; + qemu_co_mutex_unlock(&lock->mutex); + } else { + CoRwTicket my_ticket =3D { false, qemu_coroutine_self() }; + + QSIMPLEQ_INSERT_TAIL(&lock->tickets, &my_ticket, next); + qemu_co_mutex_unlock(&lock->mutex); + qemu_coroutine_yield(); + assert(lock->owners =3D=3D -1); } - lock->pending_writer--; =20 - /* The rest of the write-side critical section is run with - * the mutex taken, so that lock->reader remains zero. - * There is no need to update self->locks_held. - */ + self->locks_held++; } =20 void qemu_co_rwlock_upgrade(CoRwlock *lock) { - Coroutine *self =3D qemu_coroutine_self(); - qemu_co_mutex_lock(&lock->mutex); - assert(lock->reader > 0); - lock->reader--; - lock->pending_writer++; - while (lock->reader) { - qemu_co_queue_wait(&lock->queue, &lock->mutex); - } - lock->pending_writer--; + assert(lock->owners > 0); + /* For fairness, wait if a writer is in line. */ + if (lock->owners =3D=3D 1 && QSIMPLEQ_EMPTY(&lock->tickets)) { + lock->owners =3D -1; + qemu_co_mutex_unlock(&lock->mutex); + } else { + CoRwTicket my_ticket =3D { false, qemu_coroutine_self() }; =20 - /* The rest of the write-side critical section is run with - * the mutex taken, similar to qemu_co_rwlock_wrlock. Do - * not account for the lock twice in self->locks_held. - */ - self->locks_held--; + lock->owners--; + QSIMPLEQ_INSERT_TAIL(&lock->tickets, &my_ticket, next); + qemu_co_rwlock_maybe_wake_one(lock); + qemu_coroutine_yield(); + assert(lock->owners =3D=3D -1); + } } --=20 2.29.2 From nobody Wed Nov 19 12:30:30 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1615983777; cv=none; d=zohomail.com; s=zohoarc; b=XnhJWlFI0ND3uV4rqJYblMrD9qlbLaboIaONmNyBBhAWqtxbjZm3BdyueT0vtLEBsGWJhJ+rwEbLcPFSdyuba3sVnb4oqaVxWcofaRn2j2hb+DUWyvNEBSUNdQGx/Ud/ERPkxbTlHQMozFToie2yfVWuffOIcCex51rovJCDLRo= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1615983777; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=Xvvv2l70qQm9QCjFzsbGDAhjoM/FYrf9gqA1A6uPvNg=; b=lX1bDAulf98YvnlzbKZ8FE9AKGy3QOrmMJ3D4O45+oJRZys9lI/+8jaW34T1lB9tJZGEIoHZrFRhAuGTFNCY34Ih7GPnsRw7j+syEQ+adveHbKdYqgp3Fh47enIug9M5GqTYKq0NfYJSEX19HKhVg4WbYLigbuuhvfjmh2KTb+k= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1615983777121160.85001286918384; Wed, 17 Mar 2021 05:22:57 -0700 (PDT) Received: from localhost ([::1]:58418 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lMVCR-0007Gv-1Z for importer@patchew.org; Wed, 17 Mar 2021 08:22:55 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:57254) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lMV6Z-0002Gr-SK; Wed, 17 Mar 2021 08:16:52 -0400 Received: from mail-ed1-x530.google.com ([2a00:1450:4864:20::530]:39623) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lMV6X-0007jN-DZ; Wed, 17 Mar 2021 08:16:51 -0400 Received: by mail-ed1-x530.google.com with SMTP id bf3so1963126edb.6; Wed, 17 Mar 2021 05:16:47 -0700 (PDT) Received: from avogadro.lan ([2001:b07:6468:f312:c8dd:75d4:99ab:290a]) by smtp.gmail.com with ESMTPSA id gq25sm11520996ejb.85.2021.03.17.05.16.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 17 Mar 2021 05:16:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Xvvv2l70qQm9QCjFzsbGDAhjoM/FYrf9gqA1A6uPvNg=; b=n/MR5Z++Q7RJ18bEJ1Od2M4AzLNhj0+MUecAyA5DS7FYoLRbJjITQFdVyvl12yRYE/ P640Oltdbm1vlYWKKLJwdA7oD+GBS32qat+V8W2UDntoZimIU0pJRKSFAKYGKT/DLcLJ 5TaRSuRPa5udr0GNIZvHcuBlWtcfHleF3pNgdOh0/Jpm9X2MgqVw4YRDSuQPC/srkvIJ nm71l4BcnYiJtpHXvsInjll9Jl21N+jqSCnx9mo9TFQ0Z94v+Vs4eBjYx/YPSdR+i5PO GMQsDNV8lCly988BcTNN0rVSFXiebR/C89eP5NntbN87852NUaDZBS8dCacYX+UA/h62 7GkQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :in-reply-to:references:mime-version:content-transfer-encoding; bh=Xvvv2l70qQm9QCjFzsbGDAhjoM/FYrf9gqA1A6uPvNg=; b=LloySkQoMqxnvNEUKbh8tN9VpjfGNEIhzbu7WglAkSaHgzffJDwpFEYt/D9FeU6ArA p3flgNm2Y9BEIqRNXOuvT7v5EqpS8hxIr0fPyK97LT8FA4D7g/ZjAgRePHsW/02g0fL9 EMLEFBmw5hIxKvGlCvh6VOyWZXfSdi8aW7F7h7woosQaHtJSXSn27/TuZhsXz1prLXmE RASmP7qPfVXbjxL0ixdTnwnpTh/E5rrYFwXh51xk1sJ6NiyfYk8cidMMOFNFvkHqy7I0 7hTNT/o2YCxrLMU5TiAUF9ya5nJ51fsGecwp6AZrUb6Mswmnmnj21wcT1ncvoyLMVEcK GZuA== X-Gm-Message-State: AOAM530RSyKhHVEo15vsVNSNay6TROxB2RxqEJf1UDp0NEpbv0OjjKd5 uEJtZg+qvMRzQqWJ3UYmd+PPVUP+9c8= X-Google-Smtp-Source: ABdhPJzPTZU+TuiDdExmCF1cuEAuZ/nh3Wpso1zwoeukD7aCi4BLCzXF8xzfu7H9xjBgWMx9NOd3pw== X-Received: by 2002:aa7:d656:: with SMTP id v22mr41141009edr.119.1615983407060; Wed, 17 Mar 2021 05:16:47 -0700 (PDT) From: Paolo Bonzini To: qemu-devel@nongnu.org Subject: [PATCH 5/6] test-coroutine: add rwlock upgrade test Date: Wed, 17 Mar 2021 13:16:40 +0100 Message-Id: <20210317121641.215714-6-pbonzini@redhat.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210317121641.215714-1-pbonzini@redhat.com> References: <20210317121641.215714-1-pbonzini@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2a00:1450:4864:20::530; envelope-from=paolo.bonzini@gmail.com; helo=mail-ed1-x530.google.com X-Spam_score_int: -14 X-Spam_score: -1.5 X-Spam_bar: - X-Spam_report: (-1.5 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FORGED_FROMDOMAIN=0.249, FREEMAIL_FROM=0.001, HEADER_FROM_DIFFERENT_DOMAINS=0.25, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: david.edmondson@oracle.com, kwolf@redhat.com, qemu-block@nongnu.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Type: text/plain; charset="utf-8" Test that rwlock upgrade is fair, and readers go back to sleep if a writer is in line. Signed-off-by: Paolo Bonzini Reviewed-by: David Edmondson --- tests/unit/test-coroutine.c | 62 +++++++++++++++++++++++++++++++++++++ 1 file changed, 62 insertions(+) diff --git a/tests/unit/test-coroutine.c b/tests/unit/test-coroutine.c index e946d93a65..6e6f51d480 100644 --- a/tests/unit/test-coroutine.c +++ b/tests/unit/test-coroutine.c @@ -264,6 +264,67 @@ static void test_co_mutex_lockable(void) g_assert(QEMU_MAKE_LOCKABLE(null_pointer) =3D=3D NULL); } =20 +static CoRwlock rwlock; + +/* Test that readers are properly sent back to the queue when upgrading, + * even if they are the sole readers. The test scenario is as follows: + * + * + * | c1 | c2 | + * |--------------+------------+ + * | rdlock | | + * | yield | | + * | | wrlock | + * | | | + * | upgrade | | + * | | | + * | | unlock | + * | | | + * | unlock | | + */ + +static void coroutine_fn rwlock_yield_upgrade(void *opaque) +{ + qemu_co_rwlock_rdlock(&rwlock); + qemu_coroutine_yield(); + + qemu_co_rwlock_upgrade(&rwlock); + qemu_co_rwlock_unlock(&rwlock); + + *(bool *)opaque =3D true; +} + +static void coroutine_fn rwlock_wrlock_yield(void *opaque) +{ + qemu_co_rwlock_wrlock(&rwlock); + qemu_coroutine_yield(); + + qemu_co_rwlock_unlock(&rwlock); + *(bool *)opaque =3D true; +} + +static void test_co_rwlock_upgrade(void) +{ + bool c1_done =3D false; + bool c2_done =3D false; + Coroutine *c1, *c2; + + qemu_co_rwlock_init(&rwlock); + c1 =3D qemu_coroutine_create(rwlock_yield_upgrade, &c1_done); + c2 =3D qemu_coroutine_create(rwlock_wrlock_yield, &c2_done); + + qemu_coroutine_enter(c1); + qemu_coroutine_enter(c2); + + /* c1 now should go to sleep. */ + qemu_coroutine_enter(c1); + g_assert(!c1_done); + + qemu_coroutine_enter(c2); + g_assert(c1_done); + g_assert(c2_done); +} + /* * Check that creation, enter, and return work */ @@ -501,6 +562,7 @@ int main(int argc, char **argv) g_test_add_func("/basic/order", test_order); g_test_add_func("/locking/co-mutex", test_co_mutex); g_test_add_func("/locking/co-mutex/lockable", test_co_mutex_lockable); + g_test_add_func("/locking/co-rwlock/upgrade", test_co_rwlock_upgrade); if (g_test_perf()) { g_test_add_func("/perf/lifecycle", perf_lifecycle); g_test_add_func("/perf/nesting", perf_nesting); --=20 2.29.2 From nobody Wed Nov 19 12:30:30 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1615983710; cv=none; d=zohomail.com; s=zohoarc; b=EvS+AImiIHBnxLPPKVFdf6BplwJZXFzGXkpaT9sG7mfT0xczq8QlTXL8FYmu2v44G0DbaWTsK1AKTw/xHWH9J//R7pobqlx1K6irXM1wqom4nNzjS7dA/zfLk3d8e4zmu/vt7awKvHhJGeVJEqyFhNNz9wtYjEGMAlgKHWqr2FU= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1615983710; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=smfqshe5rWL1zIyiqGH7+GywYlbeI5eZY7MNQuJ+Muk=; b=UEU+F7utUWZaUYIwQvYjhWRuYiGqkzEZoCStd3Cwob7553173xAQAjsvpdIMTbG+rWIZXkfaFkUNF6kcMD9VcFMzeYdsP50qAKloCpCNbqW81UxdIDnMuDQVC9LF6emiAZ7+e0FaBg3FnAEsH8n/k0585enp/RcR8dO8v/0hRMM= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1615983710413306.55800855759674; Wed, 17 Mar 2021 05:21:50 -0700 (PDT) Received: from localhost ([::1]:56186 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lMVBN-0006MH-7a for importer@patchew.org; Wed, 17 Mar 2021 08:21:49 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:57258) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lMV6a-0002HR-BF; Wed, 17 Mar 2021 08:16:52 -0400 Received: from mail-ej1-x629.google.com ([2a00:1450:4864:20::629]:34835) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lMV6X-0007jm-Gd; Wed, 17 Mar 2021 08:16:51 -0400 Received: by mail-ej1-x629.google.com with SMTP id dx17so2243256ejb.2; Wed, 17 Mar 2021 05:16:48 -0700 (PDT) Received: from avogadro.lan ([2001:b07:6468:f312:c8dd:75d4:99ab:290a]) by smtp.gmail.com with ESMTPSA id gq25sm11520996ejb.85.2021.03.17.05.16.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 17 Mar 2021 05:16:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=smfqshe5rWL1zIyiqGH7+GywYlbeI5eZY7MNQuJ+Muk=; b=ff/BMM0yYL2KOKRLefc27F7D0Mg4L1QZPERQBm/dtapY0VMiYI0hn8Xjc+TdPLhRIW 5RiA0p9hSgyS3JJ1B6EGuPlLLlBv6ZmXwf/rb6xPYeFiBRvuAEiNVSjyPXxKk6KTjCxa /iGRTp8+XZxtDsq6fhjmS8tstboUW32x4zK42c/qhZpO8UgifgYhpoLIq5ax4LtnLJHS kCnCmymwPoNG0UKSlONMEe1BBNLHkgNDJA+Ur/katxy7SwgtlbKZera61shzGx7QSaTv RCKeUYA5d3K94S+ooCiMOYhtgTrVaN23zlwcQHHYUPTHKV3qW9DH6fS+BsyOHHrCee0a kCEA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :in-reply-to:references:mime-version:content-transfer-encoding; bh=smfqshe5rWL1zIyiqGH7+GywYlbeI5eZY7MNQuJ+Muk=; b=JtJxirP36pG8I8MLqgsfMz2tplblMdQU4Hbe2VkITvT2MOikSOTaHpfZLgZa94MNkS dlnjEAkPsmSpIHJiraewnY1kjc57oB/eJJLMeog1jsWDIHiKcl4fvGITmk9FJEYfaRoA bYFUplU0uopAcySQLn0G2JKEzR/BveFCAS2urdhwL/DodE1Zrz8oPEEuESgU6i0XBo4Z HJYeTalFNXT8Nrm/PYikD7h4/+IlZSG4TxZ67RowCUQ6ABqF3aqq0Qtt6X/7ug6Nst0R MGbIP2JdPQ1f+S7WR30r8zpikExDd4ODsbEP9GBox0tf2gJgiuMqIwKikCYzDWUXZEm+ CSKQ== X-Gm-Message-State: AOAM533ywveV2+xvK6A+LIw+U7d8JstS/1HchdbsLu4++F0g4K1icgRE 0q0FKIx0tmHQ92XlmDBywdrfE0xKoIw= X-Google-Smtp-Source: ABdhPJy7aglDZLedtVIG91Ey87ZuCH5kokcUgj7oB25TmAXtsyFhd6gCrPrhsSg6xP8oq0USxWW8mA== X-Received: by 2002:a17:906:5611:: with SMTP id f17mr35491731ejq.208.1615983407836; Wed, 17 Mar 2021 05:16:47 -0700 (PDT) From: Paolo Bonzini To: qemu-devel@nongnu.org Subject: [PATCH 6/6] test-coroutine: Add rwlock downgrade test Date: Wed, 17 Mar 2021 13:16:41 +0100 Message-Id: <20210317121641.215714-7-pbonzini@redhat.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210317121641.215714-1-pbonzini@redhat.com> References: <20210317121641.215714-1-pbonzini@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2a00:1450:4864:20::629; envelope-from=paolo.bonzini@gmail.com; helo=mail-ej1-x629.google.com X-Spam_score_int: -14 X-Spam_score: -1.5 X-Spam_bar: - X-Spam_report: (-1.5 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FORGED_FROMDOMAIN=0.249, FREEMAIL_FROM=0.001, HEADER_FROM_DIFFERENT_DOMAINS=0.25, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: david.edmondson@oracle.com, kwolf@redhat.com, qemu-block@nongnu.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Type: text/plain; charset="utf-8" From: David Edmondson Test that downgrading an rwlock does not result in a failure to schedule coroutines queued on the rwlock. The diagram associated with test_co_rwlock_downgrade() describes the intended behaviour, but what was observed previously corresponds to: | c1 | c2 | c3 | c4 | |--------+------------+------------+----------| | rdlock | | | | | yield | | | | | | wrlock | | | | | | | | | | | rdlock | | | | | | | | | | | wrlock | | | | | | | unlock | | | | | yield | | | | | | | | | | | downgrade | | | | | ... | | | | | unlock | | | | | | | | | | | | | This results in a failure... ERROR:../tests/test-coroutine.c:369:test_co_rwlock_downgrade: assertion fai= led: (c3_done) Bail out! ERROR:../tests/test-coroutine.c:369:test_co_rwlock_downgrade: ass= ertion failed: (c3_done) ...as a result of the c3 coroutine failing to run to completion. Signed-off-by: David Edmondson Message-Id: <20210309144015.557477-5-david.edmondson@oracle.com> Signed-off-by: Paolo Bonzini --- tests/unit/test-coroutine.c | 99 +++++++++++++++++++++++++++++++++++++ 1 file changed, 99 insertions(+) diff --git a/tests/unit/test-coroutine.c b/tests/unit/test-coroutine.c index 6e6f51d480..aa77a3bcb3 100644 --- a/tests/unit/test-coroutine.c +++ b/tests/unit/test-coroutine.c @@ -325,6 +325,104 @@ static void test_co_rwlock_upgrade(void) g_assert(c2_done); } =20 +static void coroutine_fn rwlock_rdlock_yield(void *opaque) +{ + qemu_co_rwlock_rdlock(&rwlock); + qemu_coroutine_yield(); + + qemu_co_rwlock_unlock(&rwlock); + qemu_coroutine_yield(); + + *(bool *)opaque =3D true; +} + +static void coroutine_fn rwlock_wrlock_downgrade(void *opaque) +{ + qemu_co_rwlock_wrlock(&rwlock); + + qemu_co_rwlock_downgrade(&rwlock); + qemu_co_rwlock_unlock(&rwlock); + *(bool *)opaque =3D true; +} + +static void coroutine_fn rwlock_rdlock(void *opaque) +{ + qemu_co_rwlock_rdlock(&rwlock); + + qemu_co_rwlock_unlock(&rwlock); + *(bool *)opaque =3D true; +} + +static void coroutine_fn rwlock_wrlock(void *opaque) +{ + qemu_co_rwlock_wrlock(&rwlock); + + qemu_co_rwlock_unlock(&rwlock); + *(bool *)opaque =3D true; +} + +/* + * Check that downgrading a reader-writer lock does not cause a hang. + * + * Four coroutines are used to produce a situation where there are + * both reader and writer hopefuls waiting to acquire an rwlock that + * is held by a reader. + * + * The correct sequence of operations we aim to provoke can be + * represented as: + * + * | c1 | c2 | c3 | c4 | + * |--------+------------+------------+------------| + * | rdlock | | | | + * | yield | | | | + * | | wrlock | | | + * | | | | | + * | | | rdlock | | + * | | | | | + * | | | | wrlock | + * | | | | | + * | unlock | | | | + * | yield | | | | + * | | | | | + * | | downgrade | | | + * | | | | | + * | | | unlock | | + * | | ... | | | + * | | unlock | | | + * | | | | | + * | | | | unlock | + */ +static void test_co_rwlock_downgrade(void) +{ + bool c1_done =3D false; + bool c2_done =3D false; + bool c3_done =3D false; + bool c4_done =3D false; + Coroutine *c1, *c2, *c3, *c4; + + qemu_co_rwlock_init(&rwlock); + + c1 =3D qemu_coroutine_create(rwlock_rdlock_yield, &c1_done); + c2 =3D qemu_coroutine_create(rwlock_wrlock_downgrade, &c2_done); + c3 =3D qemu_coroutine_create(rwlock_rdlock, &c3_done); + c4 =3D qemu_coroutine_create(rwlock_wrlock, &c4_done); + + qemu_coroutine_enter(c1); + qemu_coroutine_enter(c2); + qemu_coroutine_enter(c3); + qemu_coroutine_enter(c4); + + qemu_coroutine_enter(c1); + + g_assert(c2_done); + g_assert(c3_done); + g_assert(c4_done); + + qemu_coroutine_enter(c1); + + g_assert(c1_done); +} + /* * Check that creation, enter, and return work */ @@ -563,6 +661,7 @@ int main(int argc, char **argv) g_test_add_func("/locking/co-mutex", test_co_mutex); g_test_add_func("/locking/co-mutex/lockable", test_co_mutex_lockable); g_test_add_func("/locking/co-rwlock/upgrade", test_co_rwlock_upgrade); + g_test_add_func("/locking/co-rwlock/downgrade", test_co_rwlock_downgra= de); if (g_test_perf()) { g_test_add_func("/perf/lifecycle", perf_lifecycle); g_test_add_func("/perf/nesting", perf_nesting); --=20 2.29.2