From nobody Tue Apr 30 19:35:43 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; arc=pass (i=1dmarc=pass fromdomain=virtuozzo.com); dmarc=pass(p=quarantine dis=none) header.from=virtuozzo.com ARC-Seal: i=2; a=rsa-sha256; t=1611941018; cv=pass; d=zohomail.com; s=zohoarc; b=AfEhqpkkK4jnYpxTen8abTmBB3/M4PGBfTGY0sTewHmav9ZJfn0HHU/aAvZ8hkenXpj/86Z63qAgFjTY9Jv3TXO7OfLOOcviO/Z90VP7EMCujQFtZAX8OBzeR0E3OxVh5DVK7znYtUk7SjfVcRhP6T8+jJDzhOtjSJKhJN5puEY= ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1611941018; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=7sACqhT3LC0le2qy95oJqyBeh8bjD+/wB/gzlkAwfxs=; b=f5T/jBdVFqIKM14Qn6fjX2rANcU7tGCa2l5XJNgXUrMrns5bxPJH4xQ5CAVkYgj9+h0YDGhqaA9hZfIUHnSqBwDR92VK46zJju0U5zkhVvNV1Ek7lvyqcosuaSqzYBBJCVuO7Lv6YPn012kDAi1b6/en9dWKkHb9Z0QYAmb1FOw= ARC-Authentication-Results: i=2; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; arc=pass (i=1dmarc=pass fromdomain=virtuozzo.com); dmarc=pass header.from= (p=quarantine dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 161194101849992.95922092601245; Fri, 29 Jan 2021 09:23:38 -0800 (PST) Received: from localhost ([::1]:51714 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l5XUf-0006in-BX for importer@patchew.org; Fri, 29 Jan 2021 12:23:37 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:42700) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l5X07-0001Uj-OK; Fri, 29 Jan 2021 11:52:07 -0500 Received: from mail-eopbgr60091.outbound.protection.outlook.com ([40.107.6.91]:6275 helo=EUR04-DB3-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l5Wzv-0007Pt-HN; Fri, 29 Jan 2021 11:52:03 -0500 Received: from AM7PR08MB5494.eurprd08.prod.outlook.com (2603:10a6:20b:dc::15) by AM6PR08MB3621.eurprd08.prod.outlook.com (2603:10a6:20b:4b::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3784.17; Fri, 29 Jan 2021 16:51:42 +0000 Received: from AM7PR08MB5494.eurprd08.prod.outlook.com ([fe80::f1f0:6610:11f5:5e4a]) by AM7PR08MB5494.eurprd08.prod.outlook.com ([fe80::f1f0:6610:11f5:5e4a%8]) with mapi id 15.20.3784.019; Fri, 29 Jan 2021 16:51:42 +0000 Received: from kvm.sw.ru (185.215.60.152) by AM0PR02CA0001.eurprd02.prod.outlook.com (2603:10a6:208:3e::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3805.16 via Frontend Transport; Fri, 29 Jan 2021 16:51:40 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Yh9qRYH7NUYjNZVi4F1Idy3wMZ6xfOUm2N0Jlw1CPh4bskHuZL3/Sf+V1L35GGcovXRJk6QEBR7rZfQ4etngVzLzjhmmMCqgms7jfkxdFsvCbyAengk+sym5ouhv5/csTG6iOLHF3nE7H2AcXTycDzidCRNhchDMhs22kGLnF8pcyAcEXq9WfAgaeoQxQvuJKkEPUte9pK/mWXEnFdHupxHDVPaewcEdUJ5WCMObZonfBkoblpAceF8MS8p/ILV59H9BtMNTxwk5y4qBNUhIG3YKHEPxpq9fvXFF1EOYrJf/Fg1crE+8FhFeB+NJgW4EHe3/Gz+RMeZDeqfMXFWJbA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=7sACqhT3LC0le2qy95oJqyBeh8bjD+/wB/gzlkAwfxs=; b=XwOLhKK+gMmf4lZKVG+2RkeHaKy+3tV8S89EVDfa9kywxlLMtHUiDGgHsQvYCYBzSn2PoXD02brSkH3AUrICRfxBMQBHntvT8Y/3z5uH8kCy4U0QORFEtNqYBoeFZ3CoYC6KUc8yFPiqxwIQA2XVn/lsTSwM1CSGTiRs/PcEuDEBCqg6XsP0m3jxvmcVsolGbkJwQQGL/Zq+r5UNVBF+rmmr561ZWQzCnJZudfqpibFdSLaR4wHNVR7ItePXYY7fPgov+L0fFO+ChwldIIveU8dxA+gD5ZQmXEjGK+kikdGQRU8lE2tlkqqFduAXpJ/6jKmJYIl8i665h8vwVjyrRg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=virtuozzo.com; dmarc=pass action=none header.from=virtuozzo.com; dkim=pass header.d=virtuozzo.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=virtuozzo.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=7sACqhT3LC0le2qy95oJqyBeh8bjD+/wB/gzlkAwfxs=; b=BD3MylcwV/ZawMROAKnWS4pJaeaPFmstPq9butVx+B77rJUQeA/t22Wlg6ktgH9v4IQhgjOTgUDGHtMyI6K6TvXAIwmrreuBF0/w6Z+20qqPnxjoYh2h9UwL2yegpOcz0jTxjEO8rnkxAKIS3guwsu2Y9M6hNcGfhWYCNsSI3vs= Authentication-Results: nongnu.org; dkim=none (message not signed) header.d=none;nongnu.org; dmarc=none action=none header.from=virtuozzo.com; From: Vladimir Sementsov-Ogievskiy To: qemu-block@nongnu.org Subject: [PATCH 1/7] qemu/queue: add some useful QLIST_ and QTAILQ_ macros Date: Fri, 29 Jan 2021 19:50:24 +0300 Message-Id: <20210129165030.640169-2-vsementsov@virtuozzo.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210129165030.640169-1-vsementsov@virtuozzo.com> References: <20210129165030.640169-1-vsementsov@virtuozzo.com> Content-Transfer-Encoding: quoted-printable X-Originating-IP: [185.215.60.152] X-ClientProxiedBy: AM0PR02CA0001.eurprd02.prod.outlook.com (2603:10a6:208:3e::14) To AM7PR08MB5494.eurprd08.prod.outlook.com (2603:10a6:20b:dc::15) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: f3c58f20-dc11-4144-6899-08d8c47626fe X-MS-TrafficTypeDiagnostic: AM6PR08MB3621: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:110; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: BeIsebOtuclbXDL5TkzTNuJt67ed8IBw60kJP5nFI7yWoDQaYo6nf9hWCsQ02V5rb44DYAGPEzD/WdsTJhBliFaOZGvMtoYPXVHkjcrKnpIS930OOMz+r82u0HEQuU6x4bCpe7a2b5R6EvxT6Yeh2cWBEEXGB63iLd3RzxDs//vrizllQXXFYF14ubEpfV7Yvhs3NLHM/hTYlKIJfA+pR9fBc/C8dXzWHnY7I+dgyFFwRU2en5BEhMjKN3bI948AZj76nMITqcRg+SjKWi3IP1+S6GP8sHPn9QfrAnvnmD9Xh6taDeiMc74DK1he0pyC5gfkhMcRplnTJCRFgh4zdN5jJreDu6cQaqJhxRN8+tqy3fxC+P96ZhlLICqKrb2tcxdUYYdSFMVifURYZfBaomupgJDMgtA+W4sByK5QPQ4pZ1NUGWlTVjBeRmXL8AJjl8UZ708wnLrqvs7d9Ofuuuk6Vx2CmHU48vI5sYaAEA0/xBI9pvbswgXXh1OLTJw0nfC5aeR6NXNQ618tO0ba7w== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:AM7PR08MB5494.eurprd08.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(346002)(366004)(396003)(136003)(39840400004)(376002)(66476007)(83380400001)(1076003)(86362001)(8936002)(6486002)(316002)(52116002)(6512007)(66946007)(8676002)(2616005)(2906002)(66556008)(36756003)(4326008)(16526019)(6506007)(26005)(186003)(107886003)(956004)(6916009)(6666004)(5660300002)(478600001); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData: =?us-ascii?Q?t/M2RQhvFuLbg3LiRTZ7zRoOhzq/dEaiICZP3Xq+llqIhAdq2YT6CS5TejLU?= =?us-ascii?Q?KNgxSx1a8KzN9E5RQbuDSYo1XgQkioBPLqLdjR0H8FqcRs9V7O6LM2sGNnE5?= =?us-ascii?Q?706PzzY0VPYzvqypywFrIOhZechvwXR6qoBzbvDYTETACKxHBFV/Hn7AiWeY?= =?us-ascii?Q?noQVT662gFm3cDJqX8gW7wJh6L3tzMtmHclSzdqwSnpOU8yBnJLE0DG/ivFV?= =?us-ascii?Q?gyu+W7IxHt6WRTNJ1jpCEnUqmwcm4XJjUjMmyMxmGS1iB2zGVwNbUxeXsZ3C?= =?us-ascii?Q?ESmq01U/JtLY/SIttmOZd8ZO0SHVRBKoXNg3uuevbqkz4XYv+Fh19xTJ7DAN?= =?us-ascii?Q?49Cbw2QIoYqg22Lg7bMtAnLNbHa+Is3YEAZXeNP475OC+vzZuBMpvT0sUHTH?= =?us-ascii?Q?lC07s5fE10W4WkTjFysaQtV+hiGQ0a1UqOy2CZqE3RndP9UDUwUyz694mM2Y?= =?us-ascii?Q?4tWGY9lHyr7M2WHaMvnSDKDcnxUepVTf7fo6Ht2qiFEadxw/OunPJpzGLomg?= =?us-ascii?Q?6W4NAN7aOU2IcUR+zFR7pqSfRdP0wGuMr/7+CLq19t2AXV21lzy61OId2C5Y?= =?us-ascii?Q?4neHdTxnX0LYuzh7nG4sdcs7Q65Fyp6szEcbiu2kQ0QKBHZs394zTulumqo8?= =?us-ascii?Q?q7aHDZfH20B7Oo2NT9OKoL3GC4tMSFDA3QVpp03Nc8gQcA9ety+oRxoWQ1Gz?= =?us-ascii?Q?RBEYpBbdloOqn64WvYxKES2uso4gjnAuG8ZacBSCl/7cWF4tjbM8Hxk6z6iO?= =?us-ascii?Q?LRuToiYUo826t4QB12DUzC7adh01KYKJoIT3op2j30y4S7ho1LXp+lQASGZV?= =?us-ascii?Q?z253le+z2/k4fTw0UR6WH4zYvWH1wkynBqUXjW+4BXtKXi43daaLisXoMX8a?= =?us-ascii?Q?WEVXgfZKtcQb4zNSeIEi9MHoQP494qjrmYPl+/WE315sLW6UvF2Z+3VSyOWG?= =?us-ascii?Q?wpm6zHUA69CNXspySBum3A8PcfvYxDJ6cwydyOaNKgX03ouKmNWHK/H/GA/5?= =?us-ascii?Q?PUW4ijb6/DkRFNf8zJB9ictvqrC82eR59vPOZdj3S1i4aI7Pww3SspA8vMsV?= =?us-ascii?Q?XcDkuh2i?= X-OriginatorOrg: virtuozzo.com X-MS-Exchange-CrossTenant-Network-Message-Id: f3c58f20-dc11-4144-6899-08d8c47626fe X-MS-Exchange-CrossTenant-AuthSource: AM7PR08MB5494.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 29 Jan 2021 16:51:41.4836 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 0bc7f26d-0264-416e-a6fc-8352af79c58f X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: vscyH4l1DVCHRJ87QfkR62vA3Dnk/k3qYHd9dsmOmlRP1gym32xETkzgtvtko2IW2wRurn1kDJ4mJcVIE4OdTGo6G0veT94tfFCvYX80Cig= X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM6PR08MB3621 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=40.107.6.91; envelope-from=vsementsov@virtuozzo.com; helo=EUR04-DB3-obe.outbound.protection.outlook.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, MSGID_FROM_MTA_HEADER=0.001, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, vsementsov@virtuozzo.com, armbru@redhat.com, qemu-devel@nongnu.org, mreitz@redhat.com, den@openvz.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @virtuozzo.com) Content-Type: text/plain; charset="utf-8" Add QLIST_FOREACH_FUNC_SAFE(), QTAILQ_FOREACH_FUNC_SAFE() and QTAILQ_POP_HEAD(), to be used in following commit. Signed-off-by: Vladimir Sementsov-Ogievskiy --- include/qemu/queue.h | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/include/qemu/queue.h b/include/qemu/queue.h index e029e7bf66..03e1fce85f 100644 --- a/include/qemu/queue.h +++ b/include/qemu/queue.h @@ -173,6 +173,13 @@ struct { = \ (var) && ((next_var) =3D ((var)->field.le_next), 1); \ (var) =3D (next_var)) =20 +#define QLIST_FOREACH_FUNC_SAFE(head, field, func) do { \ + typeof(*QLIST_FIRST(head)) *qffs_var, *qffs_next_var; \ + QLIST_FOREACH_SAFE(qffs_var, (head), field, qffs_next_var) { \ + (func)(qffs_var); \ + } \ +} while (/*CONSTCOND*/0) + /* * List access methods. */ @@ -490,6 +497,13 @@ union { = \ (var) && ((prev_var) =3D QTAILQ_PREV(var, field), 1); \ (var) =3D (prev_var)) =20 +#define QTAILQ_FOREACH_FUNC_SAFE(head, field, func) do { \ + typeof(*QTAILQ_FIRST(head)) *qffs_var, *qffs_next_var; \ + QTAILQ_FOREACH_SAFE(qffs_var, (head), field, qffs_next_var) { \ + (func)(qffs_var); \ + } \ +} while (/*CONSTCOND*/0) + /* * Tail queue access methods. */ --=20 2.29.2 From nobody Tue Apr 30 19:35:43 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; arc=pass (i=1dmarc=pass fromdomain=virtuozzo.com); dmarc=pass(p=quarantine dis=none) header.from=virtuozzo.com ARC-Seal: i=2; a=rsa-sha256; t=1611940424; cv=pass; d=zohomail.com; s=zohoarc; b=FE4RvGfV4nSPM8da3H/suOUx4up71nn6O19ZzE8tlnQRIi8KYnehqFSNb8TteCJ0bT9FYNM9Cn/q45vjnJLZkfTrAw/7RXxdNugmZlruH/y9ci07AWV31HpEyMXR4LmJLfTdPjc2BKbw0S5d3zBrgXuh7RCmaCD5N49wZGAyfxc= ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1611940424; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=CE+Hj0hy6s/TEo5B94bzPZsfhHsPlkA7o2W7jA/5GMM=; b=k5CoOYTfryXrYaB1PWzbBCrKVsU3i2WmP/efvyNB6ODXV9yHBPzn8reqgqwTWUdqrnDu5AY3nFfr1AtPk94gm/kCCBWdYqw+ZstKqtJHeTnx/SLdk+NhOBdS+4Jzy/Avg+dXZu28LoeGLhnXEe62W7JvSxRd39AsjZLhVgWQatc= ARC-Authentication-Results: i=2; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; arc=pass (i=1dmarc=pass fromdomain=virtuozzo.com); dmarc=pass header.from= (p=quarantine dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1611940424919383.3650966603784; Fri, 29 Jan 2021 09:13:44 -0800 (PST) Received: from localhost ([::1]:55428 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l5XL5-0004mP-FB for importer@patchew.org; Fri, 29 Jan 2021 12:13:43 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:42832) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l5X0P-0001YM-3V; Fri, 29 Jan 2021 11:52:22 -0500 Received: from mail-eopbgr60091.outbound.protection.outlook.com ([40.107.6.91]:6275 helo=EUR04-DB3-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l5X08-0007Pt-Bl; Fri, 29 Jan 2021 11:52:20 -0500 Received: from AM7PR08MB5494.eurprd08.prod.outlook.com (2603:10a6:20b:dc::15) by AM6PR08MB3621.eurprd08.prod.outlook.com (2603:10a6:20b:4b::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3784.17; Fri, 29 Jan 2021 16:51:43 +0000 Received: from AM7PR08MB5494.eurprd08.prod.outlook.com ([fe80::f1f0:6610:11f5:5e4a]) by AM7PR08MB5494.eurprd08.prod.outlook.com ([fe80::f1f0:6610:11f5:5e4a%8]) with mapi id 15.20.3784.019; Fri, 29 Jan 2021 16:51:43 +0000 Received: from kvm.sw.ru (185.215.60.152) by AM0PR02CA0001.eurprd02.prod.outlook.com (2603:10a6:208:3e::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3805.16 via Frontend Transport; Fri, 29 Jan 2021 16:51:41 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ZLyZnZbr3VONvwl1GXi/QMABSLJlQAXNNtJm4/F++v3j3YTLV95S3zV0TGXDCWO6VTKVBvf9CLoF20on45Tt/53Pgm16wbVw6afIU9jG6uOs3Xh306Mvxxa0raqujIcofJeGzAMZn2rfL4x1oWB92Uzxo4s2yGrmN4yM3M0P0Uu/ixvXHspqWn3GRLjRtZwBGvUeIINtRZWgefF0cccDFCXpFQY/15CkfgmDPh+wMOYaemo19zjffY+OXx3eVvXLvCgHkAGyk0Sgiu4kgpG4/P7ISywM2EkXd9aD8inT1d40aQGbihz10cGjz504L33oA2GbOffcB0EBblFhx6lz7g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=CE+Hj0hy6s/TEo5B94bzPZsfhHsPlkA7o2W7jA/5GMM=; b=izQectLE3or5AaG1wZsQ8zSDLsCTHdN9cnzXcYZ2A/UrNDZI0tWN5PokpP8cIljXifPgKK/E+HWJ5xE0TIrX3yhYX+bSktDvOU44vogZzuQLO2VtqeElEBpbY4bCvgA5XkXdRDBAUnJ0AHIBi1mrTYWDnyw+fFro2RXPHXw5g6MaBNov0+gjtsM0CzjgiXjHBxK2KSoKQ6sA+++bJyUfjtxwPFFH1N9nJaqwbwJhWDXKOtm9FpKxlQCCVXQEXX9re7zkZNPIendVbBVfYc+B87/Gdf1IReGFD0oPkrrxIz4tbkunLOxd931w2WZ8NBj10XfeZaUqk6nWx7fKszmMlQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=virtuozzo.com; dmarc=pass action=none header.from=virtuozzo.com; dkim=pass header.d=virtuozzo.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=virtuozzo.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=CE+Hj0hy6s/TEo5B94bzPZsfhHsPlkA7o2W7jA/5GMM=; b=KPwcjNGIk1o4XM2/iwAawg/rGSjDeCgikHu3lJHAbP4I1JngkTCHC2gtRU+ZPJWf/htdQMeAtAWDoFkJVEKr0lXY8lWHLiXWMfms9h/1CLv8Q7PqQTAc79GVEu51Qiiq8jolQ2al6D7wYgdy5Xzpcz0LtoPG2HNxMgFCMUOF/xo= Authentication-Results: nongnu.org; dkim=none (message not signed) header.d=none;nongnu.org; dmarc=none action=none header.from=virtuozzo.com; From: Vladimir Sementsov-Ogievskiy To: qemu-block@nongnu.org Subject: [PATCH 2/7] block/qcow2: introduce cache for compressed writes Date: Fri, 29 Jan 2021 19:50:25 +0300 Message-Id: <20210129165030.640169-3-vsementsov@virtuozzo.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210129165030.640169-1-vsementsov@virtuozzo.com> References: <20210129165030.640169-1-vsementsov@virtuozzo.com> Content-Transfer-Encoding: quoted-printable X-Originating-IP: [185.215.60.152] X-ClientProxiedBy: AM0PR02CA0001.eurprd02.prod.outlook.com (2603:10a6:208:3e::14) To AM7PR08MB5494.eurprd08.prod.outlook.com (2603:10a6:20b:dc::15) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 7723d43e-f33d-4fb8-b4f6-08d8c4762784 X-MS-TrafficTypeDiagnostic: AM6PR08MB3621: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:1169; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: SHCnd1u5ZmPEJSGprzxAeU5r37T5K77NS5wZqUXGpneOntLIJ2hDurldZqJB5z0IbRMrZD+QuuwJ72B+at2ttZYH+zTE0C22JsA8KrZejSWDijw9OwyVtibBdEJqp7eVtH/2bUnObq/OmPoIXXvwIQ0iRjCQKN3YpIqp7RaPlOncUU2YyM5qBwbctPniz74N84n62WxunotnCQptys5h4F4Pd9kwTWrIObNDA3JJySQAOqhILiQdaAS4Bgr6+UbdHwbmIvOzoixNcZtVl9cvGf3KQJY8WiFfVoh1KXwg+FAwyAMVBEyDiGV+qXbftb3sbe69RM45vT4YovtgO54iq3P/himdE/+1bsp9Ti/ddeAClmcitFuM8biqNpl20mqe3E61OM6Tgy9hYWk4o/sivX+zySdhxT1Iq+eeLi0Ksb6o2cJFKadk2OnnxZSzBJtV12Bib3H1jX+YzAKPwkD+ipf1EEYo7Bk1TgI8IxKs7EJ0sLyT/Jl33qepJ314f9CEsP/tDZ2O+NDX3MO8Z96hog== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:AM7PR08MB5494.eurprd08.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(346002)(366004)(396003)(136003)(39840400004)(376002)(66476007)(19627235002)(83380400001)(30864003)(1076003)(86362001)(8936002)(6486002)(316002)(52116002)(6512007)(66946007)(8676002)(2616005)(2906002)(66556008)(36756003)(4326008)(16526019)(6506007)(26005)(186003)(107886003)(956004)(6916009)(6666004)(5660300002)(478600001); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData: =?us-ascii?Q?LeumfMrAxTPe9/fMgZXUnWcIzyfyMfVV1ClDWPz6mdZk/WnLeIC07RZU04HU?= =?us-ascii?Q?6CmZBz0iI58Co6fFiz2f1f/QWeN1NZW5hyS+aH5Pp9qui+mOeyrC9VARNvC+?= =?us-ascii?Q?5aaZTQa6yjjgM2AlTLVqrS1NflJPQcAvFOSbjZLHBRdAdFFqNYQdtbsDsv7T?= =?us-ascii?Q?hVYU21cdxmLXF6C+Np5y7huqqVeNejyuQXv79i7Lru/QK1nl4ZvS4vZe0O6w?= =?us-ascii?Q?F2Yy0faSDawBn7hinraFvJjksZ/e4i0Skpf4qNxh/m9IoFld2oyrYs4CUqqG?= =?us-ascii?Q?lt66s9GXbq2NC+//BCo30nOaoAgPLe5XBNg3pB8htH6kkoS7VG6ENJo6cg79?= =?us-ascii?Q?/+I7y6ZlBvDEWSr+Ih6kPZ6PC2FVnomS32XujJkTmeJgezD1CLP2ovS6tCAw?= =?us-ascii?Q?czMLsmPanXLIWml2hxieOvsxlCuNen9imbnO79r+icRS5/v7+DknwvRL8ayS?= =?us-ascii?Q?DyPvHaVc2R+xZ9J9Ql4o6YllviXoPYoPg5yz8GHT1FVEfR6zucetP6VcxjKT?= =?us-ascii?Q?EmRjOyBxyzXixO0SU7saWGqDRRp1SQoXu4nvTerKf+IErKKTBv88BtuXQzqt?= =?us-ascii?Q?F/S4TsXtKRJox4LnJRhN4O4WOlVEBQd5vrcjCWCDpgoRPhhpfyedpauYxgsv?= =?us-ascii?Q?Cm3r5/2EVB8e4E5/FUzxD7V5s3GJP7ob2qL45XvMCoY4PFP4jpfp1CjHkxKh?= =?us-ascii?Q?d21+iS6PZL7WHB0ID2bxANOOndDlxlxnh3jy3LH5eLIl421vz/R/jiDklUfk?= =?us-ascii?Q?7Rvo6zVwCBL7adpAOnfYCbe8aOKhjZz2v5B1rpZR1h7Y1reJ7WGTcE64eJw8?= =?us-ascii?Q?HxY9+NDN/FsqpR/IgAHzCO7zsgIsfjUeDTcokSMCOd+B0ohzNebn5pBTPnJN?= =?us-ascii?Q?8Zr2zWCc+wbmwEE7WjIH7RuzIRCbm/5bxwHQV4ikP3Ape7qs/gQi0DnbeaUI?= =?us-ascii?Q?I38PHCzUiz2riL5yyMH4YUTNy5QA0had90WzqSMHYLpmExD4jfIGHmwALzVf?= =?us-ascii?Q?6eUMGPZeKCIt16iNj65PZXIediSg1Q+LrGE1Z33+3Yj2VBikLXmFxo1/YG+A?= =?us-ascii?Q?fi50FarH?= X-OriginatorOrg: virtuozzo.com X-MS-Exchange-CrossTenant-Network-Message-Id: 7723d43e-f33d-4fb8-b4f6-08d8c4762784 X-MS-Exchange-CrossTenant-AuthSource: AM7PR08MB5494.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 29 Jan 2021 16:51:42.4215 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 0bc7f26d-0264-416e-a6fc-8352af79c58f X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: CnHyWc9TEYQSU7zzW4Tq7XMHd3L25DTi6BeWtigM60MdPdBfKXpDWUXucPOZmvselpJHHX6wRMYaukG8vHeXOtf6ugBRGdejDhTaAFzZuLU= X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM6PR08MB3621 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=40.107.6.91; envelope-from=vsementsov@virtuozzo.com; helo=EUR04-DB3-obe.outbound.protection.outlook.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, MSGID_FROM_MTA_HEADER=0.001, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, vsementsov@virtuozzo.com, armbru@redhat.com, qemu-devel@nongnu.org, mreitz@redhat.com, den@openvz.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @virtuozzo.com) Content-Type: text/plain; charset="utf-8" Compressed writes and O_DIRECT are not friends: they works too slow, because compressed writes does many small unaligned to 512 writes. Let's introduce an internal cache, so that compressed writes may work well when O_DIRECT is on. Signed-off-by: Vladimir Sementsov-Ogievskiy --- block/qcow2.h | 29 + block/qcow2-compressed-write-cache.c | 770 +++++++++++++++++++++++++++ block/meson.build | 1 + 3 files changed, 800 insertions(+) create mode 100644 block/qcow2-compressed-write-cache.c diff --git a/block/qcow2.h b/block/qcow2.h index 0678073b74..fbdedf89fa 100644 --- a/block/qcow2.h +++ b/block/qcow2.h @@ -322,6 +322,8 @@ typedef struct Qcow2BitmapHeaderExt { uint64_t bitmap_directory_offset; } QEMU_PACKED Qcow2BitmapHeaderExt; =20 +typedef struct Qcow2CompressedWriteCache Qcow2CompressedWriteCache; + #define QCOW2_MAX_THREADS 4 =20 typedef struct BDRVQcow2State { @@ -1010,4 +1012,31 @@ int coroutine_fn qcow2_co_decrypt(BlockDriverState *bs, uint64_t host_offset, uint64_t guest_offset, void *buf, size_t len); =20 +Qcow2CompressedWriteCache *qcow2_compressed_cache_new(BdrvChild *data_file, + int64_t cluster_size, + int64_t cache_size); +void qcow2_compressed_cache_free(Qcow2CompressedWriteCache *s); +int coroutine_fn +qcow2_compressed_cache_co_read(Qcow2CompressedWriteCache *s, int64_t offse= t, + int64_t bytes, void *buf); +int coroutine_fn +qcow2_compressed_cache_co_write(Qcow2CompressedWriteCache *s, int64_t offs= et, + int64_t bytes, void *buf); +void coroutine_fn +qcow2_compressed_cache_co_set_cluster_end(Qcow2CompressedWriteCache *s, + int64_t cluster_data_end); +int coroutine_fn +qcow2_compressed_cache_co_flush(Qcow2CompressedWriteCache *s); +int qcow2_compressed_cache_flush(BlockDriverState *bs, + Qcow2CompressedWriteCache *state); +int coroutine_fn +qcow2_compressed_cache_co_stop_flush(Qcow2CompressedWriteCache *s); +int qcow2_compressed_cache_stop_flush(BlockDriverState *bs, + Qcow2CompressedWriteCache *s); +void qcow2_compressed_cache_set_size(Qcow2CompressedWriteCache *s, + int64_t size); +void coroutine_fn +qcow2_compressed_cache_co_discard(Qcow2CompressedWriteCache *s, + int64_t cluster_offset); + #endif diff --git a/block/qcow2-compressed-write-cache.c b/block/qcow2-compressed-= write-cache.c new file mode 100644 index 0000000000..7bb92cb550 --- /dev/null +++ b/block/qcow2-compressed-write-cache.c @@ -0,0 +1,770 @@ +/* + * Write cache for qcow2 compressed writes + * + * Copyright (c) 2021 Virtuozzo International GmbH. + * + * Permission is hereby granted, free of charge, to any person obtaining a= copy + * of this software and associated documentation files (the "Software"), t= o deal + * in the Software without restriction, including without limitation the r= ights + * to use, copy, modify, merge, publish, distribute, sublicense, and/or se= ll + * copies of the Software, and to permit persons to whom the Software is + * furnished to do so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included= in + * all copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS= OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL + * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OT= HER + * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING= FROM, + * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS = IN + * THE SOFTWARE. + */ + +#include "qemu/osdep.h" + +#include "block/block_int.h" +#include "block/block-gen.h" +#include "qemu/coroutine.h" +#include "qapi/qapi-events-block-core.h" +#include "qcow2.h" + +typedef struct CacheExtent { + int64_t offset; + int64_t bytes; + void *buf; + QLIST_ENTRY(CacheExtent) next; +} CacheExtent; + +typedef struct CacheCluster { + int64_t cluster_offset; + int64_t n_bytes; /* sum of extents lengths */ + + /* + * data_end: cluster is full if data_end reached and ready to be flush= ed. + * data_end is absoluted offset, not relative. + */ + int64_t data_end; + + bool in_flight; /* cluster is being flushed now */ + + /* + * waiters: coroutines to wake after flush. Must be empty when in_flig= ht is + * false + */ + CoQueue waiters; + + QTAILQ_ENTRY(CacheCluster) next; + QLIST_HEAD(, CacheExtent) extents; /* sorted by offset */ +} CacheCluster; + +typedef QTAILQ_HEAD(, CacheCluster) CacheClusterQueue; + +struct Qcow2CompressedWriteCache { + /* + * @data_file and @cluster_size are copied from qcow2 state. Not huge + * duplications, seems better to avoid accessing the whole qcow2 state + * instead. + */ + BdrvChild *data_file; + int64_t cluster_size; + + CoQueue waiters; /* coroutines, waiting for free place in the cache */ + + /* + * Cache limits not total number of clusters but total number of active + * clusters. Active clusters are clusters with non-zero @n_bytes (and + * therefor non-empty @extents). This is done so we don't need to wait= for + * cache flush qcow2_compressed_cache_set_cluster_end() (which may cre= ate + * cluster with @data_end set but zero @n_bytes), as + * qcow2_compressed_cache_set_cluster_end() is intended to be called f= rom + * qcow2 mutex critical section. + */ + int nb_active_clusters; + + /* + * If max_active_clusters is 0 it means that cache is inactive: all new + * writes should fallthrough to data_file immediately. + */ + int max_active_clusters; + + CacheClusterQueue clusters; +}; + +/* New extent takes ownership of @buf */ +static CacheExtent *cache_extent_new(int64_t offset, int64_t bytes, void *= buf) +{ + CacheExtent *e =3D g_new(CacheExtent, 1); + + *e =3D (CacheExtent) { + .offset =3D offset, + .bytes =3D bytes, + .buf =3D buf, + }; + + return e; +} + +static void cache_extent_free(CacheExtent *e) +{ + if (e) { + g_free(e->buf); + g_free(e); + } +} + +static CacheCluster *find_cluster(Qcow2CompressedWriteCache *s, + int64_t cluster_offset) +{ + CacheCluster *c; + + assert(QEMU_IS_ALIGNED(cluster_offset, s->cluster_size)); + + QTAILQ_FOREACH(c, &s->clusters, next) { + if (cluster_offset =3D=3D c->cluster_offset) { + return c; + } + } + + return NULL; +} + +/* Creates "inactive" cluster, which doesn't influence s->nb_active_cluste= rs */ +static CacheCluster *cache_cluster_new(Qcow2CompressedWriteCache *s, + int64_t cluster_offset) +{ + CacheCluster *c; + + assert(!find_cluster(s, cluster_offset)); + + c =3D g_new(CacheCluster, 1); + *c =3D (CacheCluster) { + .cluster_offset =3D cluster_offset, + .data_end =3D cluster_offset + s->cluster_size + }; + + qemu_co_queue_init(&c->waiters); + QTAILQ_INSERT_TAIL(&s->clusters, c, next); + + return c; +} + +static void cache_cluster_free(CacheCluster *cluster) +{ + assert(!cluster->in_flight); + assert(qemu_co_queue_empty(&cluster->waiters)); + QLIST_FOREACH_FUNC_SAFE(&cluster->extents, next, cache_extent_free); + g_free(cluster); +} + +static bool cache_cluster_is_full(CacheCluster *cluster) +{ + assert(cluster->cluster_offset + cluster->n_bytes <=3D cluster->data_e= nd); + return cluster->cluster_offset + cluster->n_bytes =3D=3D cluster->data= _end; +} + +static void cache_cluster_remove(Qcow2CompressedWriteCache *s, + CacheCluster *cluster) +{ + if (cluster->n_bytes) { + s->nb_active_clusters--; + } + QTAILQ_REMOVE(&s->clusters, cluster, next); + cache_cluster_free(cluster); +} + +/* + * Return number of consequtive clusters starting from @first. Store next = after + * last extent pointer into @next, store end offset of last extent into + * @end_off. + */ +static int count_consequitive_extents(CacheExtent *first, + CacheExtent **next, + int64_t *end_off) +{ + CacheExtent *e; + int64_t off =3D first->offset; + int nb =3D 0; + + for (e =3D first; e; e =3D QLIST_NEXT(e, next)) { + assert(e->offset >=3D off); + if (e->offset > off) { + break; + } + off +=3D e->bytes; + nb++; + } + + if (next) { + *next =3D e; + } + + if (end_off) { + *end_off =3D off; + } + + return nb; +} + +/* + * Write consequtive extents, starting from @firest. Store next after last + * extent pointer into @next. If align > 1, align end of the whole write by + * zero. + */ +static int coroutine_fn flush_consequitive_extents(Qcow2CompressedWriteCac= he *s, + CacheExtent *first, + CacheExtent **next, + int64_t align) +{ + CacheExtent *e, *end_extent; + int64_t end; + int nb_extents =3D count_consequitive_extents(first, &end_extent, &end= ); + int64_t aligned_end =3D QEMU_ALIGN_UP(end, align); + int64_t tail =3D aligned_end - end; + int64_t len =3D aligned_end - first->offset; + + /* + * Alignment if for flushing full cluster, first extent offset is alwa= ys + * aligned. + */ + assert(QEMU_IS_ALIGNED(first->offset, align)); + + if (next) { + *next =3D end_extent; + } + + if (tail) { + nb_extents++; + } + + if (nb_extents > IOV_MAX) { + g_autofree void *buf =3D g_malloc(len); + char *p =3D buf; + + for (e =3D first; e !=3D end_extent; e =3D QLIST_NEXT(e, next)) { + memcpy(p, e->buf, e->bytes); + p +=3D e->bytes; + } + + if (tail) { + memset(p, 0, tail); + } + + BLKDBG_EVENT(s->data_file, BLKDBG_WRITE_COMPRESSED); + return bdrv_co_pwrite(s->data_file, first->offset, len, buf, 0); + } else { + int ret; + QEMUIOVector qiov; + g_autofree void *tail_buf =3D NULL; + + qemu_iovec_init(&qiov, nb_extents); + for (e =3D first; e !=3D end_extent; e =3D QLIST_NEXT(e, next)) { + qemu_iovec_add(&qiov, e->buf, e->bytes); + } + + if (tail) { + tail_buf =3D g_malloc0(tail); + qemu_iovec_add(&qiov, tail_buf, tail); + } + + BLKDBG_EVENT(s->data_file, BLKDBG_WRITE_COMPRESSED); + ret =3D bdrv_co_pwritev(s->data_file, first->offset, len, &qiov, 0= ); + qemu_iovec_destroy(&qiov); + return ret; + } +} + +static int coroutine_fn cache_cluster_flush_full(Qcow2CompressedWriteCache= *s, + CacheCluster *cluster) +{ + int ret; + CacheExtent *end_extent; + int64_t align =3D MIN(s->cluster_size, + MAX(s->data_file->bs->bl.request_alignment, 4 * 10= 24)); + + assert(cache_cluster_is_full(cluster)); + + ret =3D flush_consequitive_extents(s, QLIST_FIRST(&cluster->extents), + &end_extent, align); + + assert(end_extent =3D=3D NULL); /* all extents flushed */ + + return ret; +} + +static int coroutine_fn cache_cluster_flush(Qcow2CompressedWriteCache *s, + CacheCluster *c) +{ + int ret; + CacheExtent *e =3D QLIST_FIRST(&c->extents); + + if (cache_cluster_is_full(c)) { + return cache_cluster_flush_full(s, c); + } + + while (e) { + ret =3D flush_consequitive_extents(s, e, &e, 1); + if (ret < 0) { + return ret; + } + } + + return 0; +} + +int coroutine_fn qcow2_compressed_cache_co_flush(Qcow2CompressedWriteCache= *s) +{ + int ret =3D 0; + CacheCluster *c; + GList *local_clusters =3D NULL, *p; + + /* + * Make a snapshot of current state: we will not flush clusters create= d in + * parallel with flush operations and don't allow adding more extents = to + * staged clusters. We are also protected from parallel flush operatio= ns + * flushing the same clusters. + */ + QTAILQ_FOREACH(c, &s->clusters, next) { + if (!c->in_flight && c->n_bytes) { + c->in_flight =3D true; + local_clusters =3D g_list_append(local_clusters, c); + } + } + + for (p =3D local_clusters; p; p =3D p->next) { + CacheCluster *c =3D p->data; + + if (ret =3D=3D 0) { + ret =3D cache_cluster_flush(s, c); + } + + c->in_flight =3D false; + qemu_co_queue_restart_all(&c->waiters); + + if (ret =3D=3D 0) { + cache_cluster_remove(s, c); + } + } + + g_list_free(local_clusters); + + return ret; +} + +int coroutine_fn +qcow2_compressed_cache_co_stop_flush(Qcow2CompressedWriteCache *s) +{ + int ret, save; + + save =3D s->max_active_clusters; + s->max_active_clusters =3D 0; /* No more extents */ + + ret =3D qcow2_compressed_cache_co_flush(s); + if (ret < 0) { + s->max_active_clusters =3D save; + return ret; + } + + assert(QTAILQ_EMPTY(&s->clusters)); + return 0; +} + +/* @cluster takes ownership of @extent */ +static void cluster_add_extent(Qcow2CompressedWriteCache *s, + CacheCluster *cluster, CacheExtent *extent) +{ + CacheExtent *e; + + assert(extent->bytes); + assert(extent->offset >=3D cluster->cluster_offset); + assert(extent->offset + extent->bytes <=3D cluster->data_end); + assert(!cluster->in_flight); + + e =3D QLIST_FIRST(&cluster->extents); + if (!e) { + /* inactive cluster */ + assert(!cluster->n_bytes); + s->nb_active_clusters++; + assert(s->nb_active_clusters <=3D s->max_active_clusters); + QLIST_INSERT_HEAD(&cluster->extents, extent, next); + } else if (e->offset > extent->offset) { + assert(extent->offset + extent->bytes <=3D e->offset); + QLIST_INSERT_HEAD(&cluster->extents, extent, next); + } else { + while (QLIST_NEXT(e, next) && + QLIST_NEXT(e, next)->offset < extent->offset) { + e =3D QLIST_NEXT(e, next); + } + + /* Now e is last element with offset < extent->offset */ + assert(e->offset + e->bytes <=3D extent->offset); + + QLIST_INSERT_AFTER(e, extent, next); + + e =3D QLIST_NEXT(extent, next); + if (e) { + assert(extent->offset + extent->bytes <=3D e->offset); + } + } + + cluster->n_bytes +=3D extent->bytes; +} + +static CacheCluster *find_cluster_to_flush(Qcow2CompressedWriteCache *s) +{ + CacheCluster *c; + + QTAILQ_FOREACH(c, &s->clusters, next) { + if (!c->in_flight && cache_cluster_is_full(c)) { + return c; + } + } + + return NULL; +} + +/* Cache an extent if there is a place */ +static bool coroutine_fn +try_cache_extent(Qcow2CompressedWriteCache *s, CacheExtent *extent, + bool *cluster_in_flight) +{ + CacheCluster *c; + int64_t cluster_offset =3D QEMU_ALIGN_DOWN(extent->offset, s->cluster_= size); + + assert(extent->bytes); + + if (s->max_active_clusters =3D=3D 0) { + *cluster_in_flight =3D true; + return false; + } + + *cluster_in_flight =3D false; + + c =3D find_cluster(s, cluster_offset); + if (c && c->in_flight) { + *cluster_in_flight =3D true; + return false; + } + if (s->nb_active_clusters >=3D s->max_active_clusters && + (!c || !c->n_bytes)) + { + /* + * Cache is full, we can't allocate a new cluster and can't activa= te + * existing inactive cluster + */ + return false; + } + + if (!c) { + c =3D cache_cluster_new(s, cluster_offset); + } + + cluster_add_extent(s, c, extent); + + if (cache_cluster_is_full(c)) { + qemu_co_queue_restart_all(&s->waiters); + } + + return true; +} + +/* Takes ownership of @buf, don't free it after call! */ +int coroutine_fn +qcow2_compressed_cache_co_write(Qcow2CompressedWriteCache *s, int64_t offs= et, + int64_t bytes, void *buf) +{ + int ret; + int i; + CacheExtent *extents[] =3D {NULL, NULL}; + int nb_extents =3D 0; /* number of non-NULL elements in @extents */ + int64_t e0_len; + + assert(bytes > 0); + assert(bytes < s->cluster_size); + + e0_len =3D MIN(bytes, QEMU_ALIGN_UP(offset + 1, s->cluster_size) - off= set); + extents[0] =3D cache_extent_new(offset, e0_len, buf); + nb_extents =3D 1; + + if (bytes > e0_len) { + int64_t e1_len =3D bytes - e0_len; + /* + * We have to allocate a new buffer, so that clusters are independ= ent + * and can free their extents when needed. + */ + void *e1_buf =3D g_memdup(((const char *)buf) + e0_len, e1_len); + + extents[1] =3D cache_extent_new(offset + e0_len, e1_len, e1_buf); + nb_extents =3D 2; + } + + while (nb_extents) { + bool do_write =3D false; + + for (i =3D 0; i < 2; i++) { + CacheExtent *e =3D extents[i]; + + do_write =3D false; + + if (!e) { + continue; + } + + if (try_cache_extent(s, e, &do_write)) { + extents[i] =3D NULL; + nb_extents--; + continue; + } + + if (do_write) { + ret =3D bdrv_co_pwrite(s->data_file, e->offset, e->bytes, + e->buf, 0); + + cache_extent_free(e); + extents[i] =3D NULL; + nb_extents--; + + if (ret < 0) { + goto out; + } + } + } + + if (do_write) { + /* + * We yielded during second extent writing. Probably the cache= is + * already free and we can now cache the first extent. + */ + continue; + } + + if (nb_extents) { + CacheCluster *cluster =3D find_cluster_to_flush(s); + + if (cluster) { + cluster->in_flight =3D true; + ret =3D cache_cluster_flush_full(s, cluster); + cluster->in_flight =3D false; + qemu_co_queue_restart_all(&cluster->waiters); + qemu_co_queue_restart_all(&s->waiters); + if (ret < 0) { + goto out; + } + cache_cluster_remove(s, cluster); + continue; + } + + qemu_co_queue_wait(&s->waiters, NULL); + } + } + + ret =3D 0; + +out: + for (i =3D 0; i < 2; i++) { + cache_extent_free(extents[i]); + } + + return 0; +} + +int coroutine_fn +qcow2_compressed_cache_co_read(Qcow2CompressedWriteCache *s, int64_t offse= t, + int64_t bytes, void *buf) +{ + CacheCluster *c; + CacheExtent *e; + int64_t cluster_offset =3D QEMU_ALIGN_DOWN(offset, s->cluster_size); + + c =3D find_cluster(s, cluster_offset); + if (c) { + QLIST_FOREACH(e, &c->extents, next) { + if (e->offset =3D=3D offset && e->bytes <=3D bytes) { + memcpy(buf, e->buf, e->bytes); + return 0; + } + } + } + + return bdrv_co_pread(s->data_file, offset, bytes, buf, 0); +} + +/* + * Caller states, that there would be no writes to this cluster beyond + * specified @cluster_data_end. So, it's OK to flush cluster when it is fi= lled + * up to @cluster_data_end and it's OK to align flushing write operation u= p to + * some alignment (not greater than cluster_size of course). + */ +void coroutine_fn +qcow2_compressed_cache_co_set_cluster_end(Qcow2CompressedWriteCache *s, + int64_t cluster_data_end) +{ + int64_t cluster_offset =3D QEMU_ALIGN_DOWN(cluster_data_end, s->cluste= r_size); + CacheExtent *e; + CacheCluster *c; + + c =3D find_cluster(s, cluster_offset); + if (!c) { + c =3D cache_cluster_new(s, cluster_offset); + } + + QLIST_FOREACH(e, &c->extents, next) { + assert(e->offset + e->bytes <=3D cluster_data_end); + } + + /* Shouldn't set data_end several times */ + assert(c->data_end =3D=3D c->cluster_offset + s->cluster_size); + + c->data_end =3D cluster_data_end; + if (cache_cluster_is_full(c)) { + qemu_co_queue_restart_all(&s->waiters); + } +} + +Qcow2CompressedWriteCache *qcow2_compressed_cache_new(BdrvChild *data_file, + int64_t cluster_size, + int64_t cache_size) +{ + Qcow2CompressedWriteCache *s =3D g_new(Qcow2CompressedWriteCache, 1); + + assert(cache_size >=3D cluster_size); + + *s =3D (Qcow2CompressedWriteCache) { + .data_file =3D data_file, + .cluster_size =3D cluster_size, + .max_active_clusters =3D cache_size / cluster_size, + }; + + qemu_co_queue_init(&s->waiters); + QTAILQ_INIT(&s->clusters); + + return s; +} + +void qcow2_compressed_cache_free(Qcow2CompressedWriteCache *s) +{ + if (!s) { + return; + } + + QTAILQ_FOREACH_FUNC_SAFE(&s->clusters, next, cache_cluster_free); + g_free(s); +} + +void qcow2_compressed_cache_set_size(Qcow2CompressedWriteCache *s, + int64_t size) +{ + /* + * We don't do flush here. Don't care too much, it's safe to have cache + * larger than maximum, it will only decrease until it reaches the new + * maximum. + */ + assert(size >=3D s->cluster_size); + s->max_active_clusters =3D size / s->cluster_size; +} + +void coroutine_fn +qcow2_compressed_cache_co_discard(Qcow2CompressedWriteCache *s, + int64_t cluster_offset) +{ + CacheCluster *c; + + while (true) { + c =3D find_cluster(s, cluster_offset); + if (!c) { + return; + } + if (!c->in_flight) { + cache_cluster_remove(s, c); + return; + } + qemu_co_queue_wait(&c->waiters, NULL); + } +} + + +/* + * Wrappers for qcow2_compressed_cache_co_flush + * + * TODO: update scripts/block-coroutine-wrapper to generate this too + */ + +typedef struct Qcow2CompressedCacheCoFlush { + BdrvPollCo poll_state; + Qcow2CompressedWriteCache *state; +} Qcow2CompressedCacheCoFlush; + +static void coroutine_fn +qcow2_compressed_cache_co_flush_entry(void *opaque) +{ + Qcow2CompressedCacheCoFlush *s =3D opaque; + + s->poll_state.ret =3D qcow2_compressed_cache_co_flush(s->state); + s->poll_state.in_progress =3D false; + + aio_wait_kick(); +} + +int qcow2_compressed_cache_flush(BlockDriverState *bs, + Qcow2CompressedWriteCache *state) +{ + if (qemu_in_coroutine()) { + return qcow2_compressed_cache_co_flush(state); + } else { + Qcow2CompressedCacheCoFlush s =3D { + .poll_state.bs =3D bs, + .poll_state.in_progress =3D true, + + .state =3D state, + }; + + s.poll_state.co =3D + qemu_coroutine_create(qcow2_compressed_cache_co_flush_entry, &= s); + + return bdrv_poll_co(&s.poll_state); + } +} + +/* + * Wrappers for qcow2_compressed_cache_co_stop_flush + * + * TODO: update scripts/block-coroutine-wrapper to generate this too + */ + +typedef struct Qcow2CompressedCacheCoStopFlush { + BdrvPollCo poll_state; + Qcow2CompressedWriteCache *state; +} Qcow2CompressedCacheCoStopFlush; + +static void coroutine_fn +qcow2_compressed_cache_co_stop_flush_entry(void *opaque) +{ + Qcow2CompressedCacheCoStopFlush *s =3D opaque; + + s->poll_state.ret =3D qcow2_compressed_cache_co_stop_flush(s->state); + s->poll_state.in_progress =3D false; + + aio_wait_kick(); +} + +int qcow2_compressed_cache_stop_flush(BlockDriverState *bs, + Qcow2CompressedWriteCache *state) +{ + if (qemu_in_coroutine()) { + return qcow2_compressed_cache_co_stop_flush(state); + } else { + Qcow2CompressedCacheCoStopFlush s =3D { + .poll_state.bs =3D bs, + .poll_state.in_progress =3D true, + + .state =3D state, + }; + + s.poll_state.co =3D + qemu_coroutine_create(qcow2_compressed_cache_co_stop_flush_ent= ry, + &s); + + return bdrv_poll_co(&s.poll_state); + } +} diff --git a/block/meson.build b/block/meson.build index eeaefe5809..7b94794c28 100644 --- a/block/meson.build +++ b/block/meson.build @@ -25,6 +25,7 @@ block_ss.add(files( 'qcow2-bitmap.c', 'qcow2-cache.c', 'qcow2-cluster.c', + 'qcow2-compressed-write-cache.c', 'qcow2-refcount.c', 'qcow2-snapshot.c', 'qcow2-threads.c', --=20 2.29.2 From nobody Tue Apr 30 19:35:43 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; arc=pass (i=1dmarc=pass fromdomain=virtuozzo.com); dmarc=pass(p=quarantine dis=none) header.from=virtuozzo.com ARC-Seal: i=2; a=rsa-sha256; t=1611940607; cv=pass; d=zohomail.com; s=zohoarc; b=GIj0/qR6rHDpv0YwkCw7T4RNykhBuVL5HBq/+ToPBIeaY8fKxJJb8AP3aw2QEsnCYSJ3w6ynDqSzH9AbUb4u2KSwobI5ewcQAI6HwSo7yPPV4mwASnAex4bTihF6whZ5QzN+OAa6cczr4aEa1ot8ru4WPO0gkbZwL7F6ZHWCg/Q= ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1611940607; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=zyMwim8yPEUIbanVJxCv8TO8vczyvOeItKq/9B8heJE=; b=QC/Z5SZE5GYQLGGTbpgPzroWh8hnFj++cS/g5CcCM8yTXzJHuAP5cL0+I9LYCL+UT04cWqpxP2Hsa0w7LBRdf2Y1CyRbs7DDR42Vuati37Mtl5ZrgdhA1HIxZm91kQ2eVCbuzjNsp/Rmp9bEs+b5GslIcjlstyTbitlTtflsjeY= ARC-Authentication-Results: i=2; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; arc=pass (i=1dmarc=pass fromdomain=virtuozzo.com); dmarc=pass header.from= (p=quarantine dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1611940607418828.6375409958589; Fri, 29 Jan 2021 09:16:47 -0800 (PST) Received: from localhost ([::1]:35882 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l5XO2-0008Rg-5V for importer@patchew.org; Fri, 29 Jan 2021 12:16:46 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:42772) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l5X0M-0001Xq-4z; Fri, 29 Jan 2021 11:52:19 -0500 Received: from mail-eopbgr60100.outbound.protection.outlook.com ([40.107.6.100]:29349 helo=EUR04-DB3-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l5X07-0007OV-VT; Fri, 29 Jan 2021 11:52:17 -0500 Received: from AM7PR08MB5494.eurprd08.prod.outlook.com (2603:10a6:20b:dc::15) by AM6PR08MB3621.eurprd08.prod.outlook.com (2603:10a6:20b:4b::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3784.17; Fri, 29 Jan 2021 16:51:43 +0000 Received: from AM7PR08MB5494.eurprd08.prod.outlook.com ([fe80::f1f0:6610:11f5:5e4a]) by AM7PR08MB5494.eurprd08.prod.outlook.com ([fe80::f1f0:6610:11f5:5e4a%8]) with mapi id 15.20.3784.019; Fri, 29 Jan 2021 16:51:43 +0000 Received: from kvm.sw.ru (185.215.60.152) by AM0PR02CA0001.eurprd02.prod.outlook.com (2603:10a6:208:3e::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3805.16 via Frontend Transport; Fri, 29 Jan 2021 16:51:42 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=BkFxHtwza4uwDKEu8ypnYJHP9flcCIMT1rr/1/avtipApH7d1vO6UD/c0dCVR2EHVIk8tsaxEHT1lVCWn9xsHPFJeuHUgeY3HwXEV3uet7STvWnXmb8sigFP05py+xfKU7WxFRlByS2LQekFb8woZ9t6SEtC+nZaIcPs91nlZ0utXH28MuZA4HWupw3lr7hBHJmCSgd4989JmJ3UR/NoeqZxuq+YDOLKUOxi4ZsvfBX6yDI78igFVXy+ptHkudlyf/zFmjq3DI9GgBqJbBdeNFa5VU4DuJ0CEpFXVvpX7u4GWWRq3iKnOEDG2aI3a8McL7/hCwuom8ohoSqNwj4HHg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=zyMwim8yPEUIbanVJxCv8TO8vczyvOeItKq/9B8heJE=; b=ITOdBUuiDLOmPYErF6hIp9MqbwEXUa1xr76WHeBiuCqylUTWDhzUk473qJQBHT5I2gQM1us+sJ3w/5vSUqu/Qz8pvd3G8mSDuMMXiSFzI2faLvNCDZZBVRue6yduOBn4bk+IY7IsWm6sBn6ubFzTaHTAsfl2hhxduTKWX9pBvJylYb0dZBNyFGFmjNLHfvCKkewck/GosFafffr4YezrVNlnMUdRcDAWxJi+GuhbZ/qhCu0ZYKwJ1jj+QlgKFGoE5H1UZpOW9E00kxo4b3LshcgJlR1ObcTcooI37LqQKTMboPEtBPEJvj8xhTKcmingv3+cUlHTNUZVsU9+s3mhHQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=virtuozzo.com; dmarc=pass action=none header.from=virtuozzo.com; dkim=pass header.d=virtuozzo.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=virtuozzo.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=zyMwim8yPEUIbanVJxCv8TO8vczyvOeItKq/9B8heJE=; b=DPli5vphhIZDc6uSWSmUnquGDAqEafNkMXv2ZvvtOG9UFVEv/MwoTgwOLaHIr3vbFLg6ZMX94hcTgoHMN887ooA1Q2JDVonwItGsExQ0Sui+sRJIBSr84WfqHPSXbgGQfBphqFamNgQ/pgEJMswJVhAC/Fo/dgupvAB0K8VVCAs= Authentication-Results: nongnu.org; dkim=none (message not signed) header.d=none;nongnu.org; dmarc=none action=none header.from=virtuozzo.com; From: Vladimir Sementsov-Ogievskiy To: qemu-block@nongnu.org Subject: [PATCH 3/7] block/qcow2: use compressed write cache Date: Fri, 29 Jan 2021 19:50:26 +0300 Message-Id: <20210129165030.640169-4-vsementsov@virtuozzo.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210129165030.640169-1-vsementsov@virtuozzo.com> References: <20210129165030.640169-1-vsementsov@virtuozzo.com> Content-Transfer-Encoding: quoted-printable X-Originating-IP: [185.215.60.152] X-ClientProxiedBy: AM0PR02CA0001.eurprd02.prod.outlook.com (2603:10a6:208:3e::14) To AM7PR08MB5494.eurprd08.prod.outlook.com (2603:10a6:20b:dc::15) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 00834972-3e29-4450-d7f4-08d8c476281d X-MS-TrafficTypeDiagnostic: AM6PR08MB3621: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:2657; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: a4dReTWVmdu518xNc+HM+F+YyKuYtiCZQ8zYfSL6+OmmAycVyeM0GihtxRAbauWYVRjv8YFKVhaQaspLciUODo6efL60ratOydJz9qUqy7oEYG5VCnM0ZUKCJMO7gORCkZan0nEXdLorVUoVBCi/R5xWt+LnRVvOyQvihJXDQ5Asu36GK+HPMWEwUTBGWUqBPR49y6Bab1vd23wRQw4Vn6teEwSMZUecm78ngZAi7XKpDZluyZuEinlx1NE9AzFIa9/b9McWS9JzFHK+BKxPKtODzn0w0TgSE3jMcHz7i85wcItKGlblUYQ4EikPronap7Ebf9xqLrCD2w3adcriQ232dOoe5c5Wsi24CdKXdb5+FtwdnJlm/KAhntSdol5CUCgQ4wkO7lkfdH6Uymsgc8EqGla4KdNI3xPly9NzCtmjvDWZ69i1837dgu7sRTKbRRGjyG6WQRGsy3RCwJEVIZOvyrfJ9Mh9bmUjwTO6AdiB6AQdINhjS/UvSS0No0w1mAzulUTJZxTyP6holOjwMg== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:AM7PR08MB5494.eurprd08.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(346002)(366004)(396003)(136003)(39840400004)(376002)(66476007)(19627235002)(83380400001)(1076003)(86362001)(8936002)(6486002)(316002)(52116002)(6512007)(66946007)(8676002)(2616005)(2906002)(66556008)(36756003)(4326008)(16526019)(6506007)(26005)(186003)(107886003)(956004)(6916009)(6666004)(5660300002)(478600001); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData: =?us-ascii?Q?ZiChEs49V7UTLt4txUp6resjmF8iFkziK8tdVukGrt3OTmi9mwqlMQQysET1?= =?us-ascii?Q?SUevkBvdZG+3UIlq/xMgWHb0dmG+b1l4V5V2QsM3zY6OvSXvT3bFh2QbS69D?= =?us-ascii?Q?m4WSWK9N3LCrk0zIwGWbmbuPMb3LvEisnPUSJV4RCeB93T2ul5BVZordEJly?= =?us-ascii?Q?28X3TNCrSjjjp4LK3rWaWssWyECQ1z3U8QLxm5zWnDG/zpOEDIEG6alEsthD?= =?us-ascii?Q?OiA0zcFa46orhwh6S6xHqaSPLfl/CB7zr4YIdRVOZJEaVaiqZB/ASqC7mz7a?= =?us-ascii?Q?ZG3jp+y7Qc7SngRYcO2yYp90scJOmUyudbgtYv7c/0XGtvh2UCDpNwx4veBK?= =?us-ascii?Q?aL/CiM0jGsBDYtq1y7dRqzERoxTau3mmeDduFV1+jZ8YvpyZKyeQ4xF0dfFD?= =?us-ascii?Q?L79zHyWQX5rlPgjif8d52zaOCBWzO4Ch/yLT+VGEHynPDhmnQZmm+RSgsgz9?= =?us-ascii?Q?wnL3wV68QYiww0xIdt5UEHX3cKtvrNuO+gy7AI2kfDAFJj//Xq1G7Tnl2xbg?= =?us-ascii?Q?aqU0MOmYlMNdp4Qqwxz9C1kVh61PACpsOwcLD51wkOoa0KmZTXGHoe88sVzh?= =?us-ascii?Q?W5+6TaysWxpMpABc2/WqT5WH0Hbzvv52pJRwFmK7Um7EDOkZoFdl+gL4n9y2?= =?us-ascii?Q?XqZaL3mqxp+xnKu8HT3eJGEEnHUyObelMHsRtkqWgELK3ongYul81xlxuRLL?= =?us-ascii?Q?8YUDVzzf42QQ3Nw8yB9q5CYFoiChd6F42j8sRf69Ffx9MqehLi3Ik8PTJ8dS?= =?us-ascii?Q?ga+aOxdpI/lBWW6XtnUR16uAL0sKYLTrs8zowoO/OiblrQT1vFym3sNlrhUe?= =?us-ascii?Q?6Gc9JxJTI6XUaa8QJ0Kym1ALcwhDd9p6jf1ueKUu2QV1a93KNMehoNzDlmrQ?= =?us-ascii?Q?zoFmMdzHYPnaGnGS3Bkov5WE15gDzTo4jBjZRNLUGKvjjDwZ4CO4PhaJwg7q?= =?us-ascii?Q?D0mJvwmdVsumOPF6BEe4z/gCvF0kcRzUTJJWtaBxnIQ+3/j3EMf0BkylHmwo?= =?us-ascii?Q?zpMs2D7IxpjWGTr088AeNK00b+qgNn7bm0/j9GVYJXMVaUjVx0ZF7l0sczLM?= =?us-ascii?Q?c1sCbd+H?= X-OriginatorOrg: virtuozzo.com X-MS-Exchange-CrossTenant-Network-Message-Id: 00834972-3e29-4450-d7f4-08d8c476281d X-MS-Exchange-CrossTenant-AuthSource: AM7PR08MB5494.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 29 Jan 2021 16:51:43.2747 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 0bc7f26d-0264-416e-a6fc-8352af79c58f X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: V73DYQMuv3J/jcO2TNH0vsJ2BR0gAl9EDm1yohvYQT8haptUWFx+iL8wPADaoH1ijN5FiDRmoBC73P/RISsQ8Uy6ZJY2CWCu1jC1+YXD8V8= X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM6PR08MB3621 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=40.107.6.100; envelope-from=vsementsov@virtuozzo.com; helo=EUR04-DB3-obe.outbound.protection.outlook.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, MSGID_FROM_MTA_HEADER=0.001, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, vsementsov@virtuozzo.com, armbru@redhat.com, qemu-devel@nongnu.org, mreitz@redhat.com, den@openvz.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @virtuozzo.com) Content-Type: text/plain; charset="utf-8" Introduce a new option: compressed-cache-size, with default to 64 clusters (to be not less than 64 default max-workers for backup job). Signed-off-by: Vladimir Sementsov-Ogievskiy --- qapi/block-core.json | 8 +++- block/qcow2.h | 4 ++ block/qcow2-refcount.c | 13 +++++++ block/qcow2.c | 87 ++++++++++++++++++++++++++++++++++++++++-- 4 files changed, 108 insertions(+), 4 deletions(-) diff --git a/qapi/block-core.json b/qapi/block-core.json index 9f555d5c1d..e0be6657f3 100644 --- a/qapi/block-core.json +++ b/qapi/block-core.json @@ -3202,6 +3202,11 @@ # an image, the data file name is loaded from the image # file. (since 4.0) # +# @compressed-cache-size: The maximum size of compressed write cache in +# bytes. If positive must be not less than +# cluster size. 0 disables the feature. Default +# is 64 * cluster_size. (since 6.0) +# # Since: 2.9 ## { 'struct': 'BlockdevOptionsQcow2', @@ -3217,7 +3222,8 @@ '*refcount-cache-size': 'int', '*cache-clean-interval': 'int', '*encrypt': 'BlockdevQcow2Encryption', - '*data-file': 'BlockdevRef' } } + '*data-file': 'BlockdevRef', + '*compressed-cache-size': 'int' } } =20 ## # @SshHostKeyCheckMode: diff --git a/block/qcow2.h b/block/qcow2.h index fbdedf89fa..b86aa06006 100644 --- a/block/qcow2.h +++ b/block/qcow2.h @@ -150,6 +150,7 @@ #define QCOW2_OPT_L2_CACHE_ENTRY_SIZE "l2-cache-entry-size" #define QCOW2_OPT_REFCOUNT_CACHE_SIZE "refcount-cache-size" #define QCOW2_OPT_CACHE_CLEAN_INTERVAL "cache-clean-interval" +#define QCOW2_OPT_COMPRESSED_CACHE_SIZE "compressed-cache-size" =20 typedef struct QCowHeader { uint32_t magic; @@ -422,6 +423,9 @@ typedef struct BDRVQcow2State { * is to convert the image with the desired compression type set. */ Qcow2CompressionType compression_type; + + uint64_t compressed_cache_size; + Qcow2CompressedWriteCache *compressed_cache; } BDRVQcow2State; =20 typedef struct Qcow2COWRegion { diff --git a/block/qcow2-refcount.c b/block/qcow2-refcount.c index 8e649b008e..5720591460 100644 --- a/block/qcow2-refcount.c +++ b/block/qcow2-refcount.c @@ -899,6 +899,11 @@ static int QEMU_WARN_UNUSED_RESULT update_refcount(Blo= ckDriverState *bs, qcow2_cache_discard(s->l2_table_cache, table); } =20 + if (s->compressed_cache) { + qcow2_compressed_cache_co_discard(s->compressed_cache, + cluster_offset); + } + if (s->discard_passthrough[type]) { update_refcount_discard(bs, cluster_offset, s->cluster_siz= e); } @@ -1110,6 +1115,14 @@ int64_t qcow2_alloc_bytes(BlockDriverState *bs, int = size) } =20 if (!offset || ROUND_UP(offset, s->cluster_size) !=3D new_clus= ter) { + if (offset && s->compressed_cache) { + /* + * Previous cluster is unfinished, but we'll not write= more + * data to it. We should inform compressed cache. + */ + qcow2_compressed_cache_co_set_cluster_end( + s->compressed_cache, offset); + } offset =3D new_cluster; free_in_cluster =3D s->cluster_size; } else { diff --git a/block/qcow2.c b/block/qcow2.c index 5d94f45be9..3997d8c143 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -808,6 +808,13 @@ static QemuOptsList qcow2_runtime_opts =3D { .type =3D QEMU_OPT_NUMBER, .help =3D "Clean unused cache entries after this time (in seco= nds)", }, + { + .name =3D QCOW2_OPT_COMPRESSED_CACHE_SIZE, + .type =3D QEMU_OPT_NUMBER, + .help =3D "The maximum size of compressed write cache in bytes= . If " + "positive must be not less than cluster size. 0 disabl= es " + "the feature. Default is 64 * cluster_size", + }, BLOCK_CRYPTO_OPT_DEF_KEY_SECRET("encrypt.", "ID of secret providing qcow2 AES key or LUKS passphrase"), { /* end of list */ } @@ -969,6 +976,38 @@ typedef struct Qcow2ReopenState { QCryptoBlockOpenOptions *crypto_opts; /* Disk encryption runtime optio= ns */ } Qcow2ReopenState; =20 +static int qcow2_update_compressed_cache_option(BlockDriverState *bs, + QemuOpts *opts, Error **er= rp) +{ + BDRVQcow2State *s =3D bs->opaque; + uint64_t new_size; + int ret; + + new_size =3D qemu_opt_get_size(opts, QCOW2_OPT_COMPRESSED_CACHE_SIZE, + 64 * s->cluster_size); + if (new_size > 0 && new_size < s->cluster_size) { + error_setg(errp, "compressed cache size is too small"); + return -EINVAL; + } + + if (s->compressed_cache && !new_size) { + ret =3D qcow2_compressed_cache_stop_flush(bs, s->compressed_cache); + if (ret < 0) { + error_report("Failed to flush the compressed write cache: %s", + strerror(-ret)); + return ret; + } + qcow2_compressed_cache_free(s->compressed_cache); + s->compressed_cache =3D NULL; + } else if (s->compressed_cache) { + qcow2_compressed_cache_set_size(s->compressed_cache, new_size); + } + + s->compressed_cache_size =3D new_size; + + return 0; +} + static int qcow2_update_options_prepare(BlockDriverState *bs, Qcow2ReopenState *r, QDict *options, int flags, @@ -994,6 +1033,11 @@ static int qcow2_update_options_prepare(BlockDriverSt= ate *bs, goto fail; } =20 + ret =3D qcow2_update_compressed_cache_option(bs, opts, errp); + if (ret < 0) { + goto fail; + } + /* get L2 table/refcount block cache size from command line options */ read_cache_sizes(bs, opts, &l2_cache_size, &l2_cache_entry_size, &refcount_cache_size, &local_err); @@ -2660,6 +2704,17 @@ static int qcow2_inactivate(BlockDriverState *bs) bdrv_get_device_or_node_name(bs)); } =20 + if (s->compressed_cache) { + ret =3D qcow2_compressed_cache_stop_flush(bs, s->compressed_cache); + if (ret < 0) { + result =3D ret; + error_report("Failed to flush the compressed write cache: %s", + strerror(-ret)); + } + qcow2_compressed_cache_free(s->compressed_cache); + s->compressed_cache =3D NULL; + } + ret =3D qcow2_cache_flush(bs, s->l2_table_cache); if (ret) { result =3D ret; @@ -2692,6 +2747,8 @@ static void qcow2_close(BlockDriverState *bs) qcow2_inactivate(bs); } =20 + assert(!s->compressed_cache); + cache_clean_timer_del(bs); qcow2_cache_destroy(s->l2_table_cache); qcow2_cache_destroy(s->refcount_block_cache); @@ -4539,8 +4596,14 @@ qcow2_co_pwritev_compressed_task(BlockDriverState *b= s, goto fail; } =20 - BLKDBG_EVENT(s->data_file, BLKDBG_WRITE_COMPRESSED); - ret =3D bdrv_co_pwrite(s->data_file, cluster_offset, out_len, out_buf,= 0); + if (s->compressed_cache) { + ret =3D qcow2_compressed_cache_co_write(s->compressed_cache, + cluster_offset, out_len, out= _buf); + out_buf =3D NULL; + } else { + BLKDBG_EVENT(s->data_file, BLKDBG_WRITE_COMPRESSED); + ret =3D bdrv_co_pwrite(s->data_file, cluster_offset, out_len, out_= buf, 0); + } if (ret < 0) { goto fail; } @@ -4601,6 +4664,12 @@ qcow2_co_pwritev_compressed_part(BlockDriverState *b= s, return -EINVAL; } =20 + if (!s->compressed_cache && s->compressed_cache_size) { + s->compressed_cache =3D + qcow2_compressed_cache_new(s->data_file, s->cluster_size, + s->compressed_cache_size); + } + while (bytes && aio_task_pool_status(aio) =3D=3D 0) { uint64_t chunk_size =3D MIN(bytes, s->cluster_size); =20 @@ -4656,7 +4725,12 @@ qcow2_co_preadv_compressed(BlockDriverState *bs, out_buf =3D qemu_blockalign(bs, s->cluster_size); =20 BLKDBG_EVENT(bs->file, BLKDBG_READ_COMPRESSED); - ret =3D bdrv_co_pread(bs->file, coffset, csize, buf, 0); + if (s->compressed_cache) { + ret =3D qcow2_compressed_cache_co_read(s->compressed_cache, + coffset, csize, buf); + } else { + ret =3D bdrv_co_pread(bs->file, coffset, csize, buf, 0); + } if (ret < 0) { goto fail; } @@ -4875,6 +4949,13 @@ static coroutine_fn int qcow2_co_flush_to_os(BlockDr= iverState *bs) BDRVQcow2State *s =3D bs->opaque; int ret; =20 + if (s->compressed_cache) { + ret =3D qcow2_compressed_cache_flush(bs, s->compressed_cache); + if (ret < 0) { + return ret; + } + } + qemu_co_mutex_lock(&s->lock); ret =3D qcow2_write_caches(bs); qemu_co_mutex_unlock(&s->lock); --=20 2.29.2 From nobody Tue Apr 30 19:35:43 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; arc=pass (i=1dmarc=pass fromdomain=virtuozzo.com); dmarc=pass(p=quarantine dis=none) header.from=virtuozzo.com ARC-Seal: i=2; a=rsa-sha256; t=1611940173; cv=pass; d=zohomail.com; s=zohoarc; b=gLMtY/nwX/TwZm18M0gaEgLIvK2/0ZnEKMSgoY9kVFA2wFaQjSZE7rl876T6cj97zzVhav2tH4CyEZ6om/09PubyM+R4v4gagInp0QicK4uAai0UXvAsclMkOJFnEQu6SZrjU+qVciTG7tZLl90FkQIuHuUL8aYBtl1DCr1jNG8= ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1611940173; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=IuU5U8ExdiQchUZ41oPhpNoMvYCUBoVyADNBAItP90I=; b=AmaJHu8YqTF14+xY97B4kxFz/2++pLeD4EWZtg76WJhpnkhMizmSewBfI31ocaR3c8jWvzkkTCpOccp4S/5ht/3j6mzO4YZ1U2xKkJbdBdrLj1yb8KzozyIOmsL5Tr2mRmknBEtsDT03WE+djiOf6kiNUHc8A7sut3G+HcbsVzY= ARC-Authentication-Results: i=2; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; arc=pass (i=1dmarc=pass fromdomain=virtuozzo.com); dmarc=pass header.from= (p=quarantine dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 16119401739381002.7548469205958; Fri, 29 Jan 2021 09:09:33 -0800 (PST) Received: from localhost ([::1]:45884 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l5XH1-0000da-Io for importer@patchew.org; Fri, 29 Jan 2021 12:09:31 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:42836) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l5X0Q-0001YN-50; Fri, 29 Jan 2021 11:52:22 -0500 Received: from mail-eopbgr60100.outbound.protection.outlook.com ([40.107.6.100]:29349 helo=EUR04-DB3-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l5X0M-0007OV-Pt; Fri, 29 Jan 2021 11:52:21 -0500 Received: from AM7PR08MB5494.eurprd08.prod.outlook.com (2603:10a6:20b:dc::15) by AM6PR08MB3621.eurprd08.prod.outlook.com (2603:10a6:20b:4b::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3784.17; Fri, 29 Jan 2021 16:51:44 +0000 Received: from AM7PR08MB5494.eurprd08.prod.outlook.com ([fe80::f1f0:6610:11f5:5e4a]) by AM7PR08MB5494.eurprd08.prod.outlook.com ([fe80::f1f0:6610:11f5:5e4a%8]) with mapi id 15.20.3784.019; Fri, 29 Jan 2021 16:51:44 +0000 Received: from kvm.sw.ru (185.215.60.152) by AM0PR02CA0001.eurprd02.prod.outlook.com (2603:10a6:208:3e::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3805.16 via Frontend Transport; Fri, 29 Jan 2021 16:51:43 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=RBFDrj301vRYLnfnzHOGZrCHDIl/QukRtz/hIKHwrIW+J24eVwD6WzfjSjK+9pPZ/S918Mb3IdOluDrx6+mzi8xlocRWRqmGo8MZbPMqih+hx9BL6zyPp7Gafku/MYgzZMoxmGI3UkGHRleAhkD50Q6402S6a9BkX+tJp/RJG3leCoo/yLPUj1cGtmjumuYHrPpFfe9NHvRriwK+CUgmnliNPyau5+SDRhYfQlNWZInoYBCXusD9XUfYrkcZu+gvcsgeW52rBx0Rv2waHuQ/4fVyf1TuKIqFfdIJn63kccOwRkNJNxOtTQLTfV3IhGomK8/3zZqS0DYFCJeohIOQkw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=IuU5U8ExdiQchUZ41oPhpNoMvYCUBoVyADNBAItP90I=; b=TNVk8Pc81zUtOpc9ptEexf/dGuTuVAWLns5V74uydeflIlzjSWaN+IyNi7p7vyiJx7EyxIYOZUcb4uzLlhb5Y03TfKSGQuGgsMLjmim81IQkjUE++tEsKRBlz26UqKuNhzzoZF4q+OaznU/6LLXTBW0q+6NcRMFbwi6wg6//uUwmQnMlx+7k6YyQANr0dlB1CJCWSR5fMX0eYJLLMSUotTImoajeAXfltjVPw7Wnmwr5GrYWKvvixR19y5B5fFrYpGZrPGOmX+tMnUcja8az7a/183uJ2LRHcPe2ehXF19izlEbiZIFqaWefBzDY2BbwxIGavp8Q/1ETEpWpHI+Pqg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=virtuozzo.com; dmarc=pass action=none header.from=virtuozzo.com; dkim=pass header.d=virtuozzo.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=virtuozzo.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=IuU5U8ExdiQchUZ41oPhpNoMvYCUBoVyADNBAItP90I=; b=uT4Y1Lo4oXJbtrrquMh9D1zw5Ivxq/CyEtCsyydrMShsGNqQYtXKlrM9K8u+5o0VzKWlm02+nJ47/8ZcG618DSu7sxiGw0lzF7Q7r6jlt5FI84Kcv8Mgqr1tmLWU1lnfY9lSwSsQrDKYgFvtJUK+MyghZLjm2ISDscblW+SdSLE= Authentication-Results: nongnu.org; dkim=none (message not signed) header.d=none;nongnu.org; dmarc=none action=none header.from=virtuozzo.com; From: Vladimir Sementsov-Ogievskiy To: qemu-block@nongnu.org Subject: [PATCH 4/7] simplebench: bench_one(): add slow_limit argument Date: Fri, 29 Jan 2021 19:50:27 +0300 Message-Id: <20210129165030.640169-5-vsementsov@virtuozzo.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210129165030.640169-1-vsementsov@virtuozzo.com> References: <20210129165030.640169-1-vsementsov@virtuozzo.com> Content-Transfer-Encoding: quoted-printable X-Originating-IP: [185.215.60.152] X-ClientProxiedBy: AM0PR02CA0001.eurprd02.prod.outlook.com (2603:10a6:208:3e::14) To AM7PR08MB5494.eurprd08.prod.outlook.com (2603:10a6:20b:dc::15) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 258df72e-7957-40ff-f3a1-08d8c47628a0 X-MS-TrafficTypeDiagnostic: AM6PR08MB3621: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:6430; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: fqnViCFSv6sWoBh/X/lyKcXSYlWCoZt5vtQXl3wOnb0v198mYOKgXDjLw5+Rk2/m+PkG7XyT9hYRWEKwX28w9RLMQ51HLSRjp50cQKCd/BtyLBcy1JwKhut58gVphwglGyxdMWWUJCAOV38R4F2BBOaRravpT4NkrxOEz3qDf+dl5YH8qFG/Mit2s/9w2FTwi7EDCv4YDcAifPkL6kAjj1wgkrCnwyj4XpSxcxP/QlPeYTEOycn31BekoVf3QEAC1hJe97CKk7jtcGujTtXzYGhWUP/8ixUfBDh9/hUT+3nvCh4lreSE3fszKlYuvIrGlo156KE2yXrq233/CHwD2PCXhvHMxwZHDHBlNARnFpmFaWTXfSgnL0A8ZoogBnFnp/HuxCFi46mJBLCt2X6GP2fXkPVo9QlNJ5tJnasa8O4pkO5S2v+WyBl/bR7u+nUr16HdzwuNCAnC5qdaf6FJ+NQO0Dh2AMknTePkxP27T1iriamZ0bogzmI9UliKHD3uBQm5aM1XVCZ6/+zC4OUSbg== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:AM7PR08MB5494.eurprd08.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(346002)(366004)(396003)(136003)(39840400004)(376002)(66476007)(83380400001)(1076003)(86362001)(8936002)(6486002)(316002)(52116002)(6512007)(66946007)(8676002)(2616005)(2906002)(66556008)(36756003)(4326008)(16526019)(6506007)(26005)(186003)(107886003)(956004)(6916009)(6666004)(5660300002)(478600001); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData: =?us-ascii?Q?SN2eTLdxYrvMIF7RgUWw8ghwVytnSQYHGDS5T9D3Z4Wz5lz18BpGaqeRKnp7?= =?us-ascii?Q?l6GDI29zUf27krolgp1BU8BG8JvxqQfNd4jv+4bgY1xVU2/+LE2fb8qrqWz0?= =?us-ascii?Q?5Q4uVywj2pQUtr0aRc26AUown8Y6uH1uYkOmA9NNdRjxjs5RP4dS14y4iwnh?= =?us-ascii?Q?aR3EvP8tykw+bfE2x75f3HwIUyIcWXiMbfcYiIH2W3+AXGotcYXfBYFbSOEY?= =?us-ascii?Q?3/eC3p78wBdEVfa38fTWR1Vmd4x2OaYlNGiW2tydgadn2cnERVLExkAYvfWU?= =?us-ascii?Q?tdOhoiD1ef5w2LhG9YxI/uhzm9wM0OOFNNWGfsFsVqCfUnCBQhTWBBCC+Wha?= =?us-ascii?Q?QDiUdIHH8EXxzqgsgsXCyAvbCvXWgEcx72Ua1lvh5UqcsNKt4eWoUwrCD8X8?= =?us-ascii?Q?F6KeIN0rKQm0k2mxHfOoMLTEd9w1aKZpn9mOuYYmY6bz0p5+pFEvE2jnZEaj?= =?us-ascii?Q?rohXR5QFRSDNFcunVWhNEXB3biglSihCSxBtkOZ9OMDyXU00H4L0hWeE6Yaa?= =?us-ascii?Q?h2Jc6n6k5MhiJ24385JcF+rRPsIYaWxdxG9P52LuPbgRrnHTqvXBfAbNgYM5?= =?us-ascii?Q?MTygFFGXMatgOm5kafPKGvfyXlZ2bCQQsR37J9BWMiW4a1uc7cLKS+b23Dsq?= =?us-ascii?Q?/Mmty0U4F9MV1488wjVmRAV9rgBhe0jiret1F7bOG2bANycqBU8XRr2RXXW2?= =?us-ascii?Q?Z2m1seZ4cPQWy4u8s/EzmUpC1/AE1AQhVXPqNEPr7gNecjU+RKuXZKMEDFWO?= =?us-ascii?Q?1bAjGQpwitaTHZlxiu2a+vI/D3CmNTKzSKcMfDkszQTh72Q8mFTQKGH4LPGD?= =?us-ascii?Q?eaJa9iFZ0i85Lk2800VURs737r11vZEs5KpKuS1LhUCRvzeE/qPmiBYM01ik?= =?us-ascii?Q?uVcSZ6A3H25+QRCG1Tfh5Tdqs6s8ENM0u9ljHuwbqIZA+cElK+YJxeuq+0Jv?= =?us-ascii?Q?oZ7yb47uKBD/VPuzp2G+IJ+Qdea7b61i+9aBWllP1AKAHEPYacK9vOJvTC2V?= =?us-ascii?Q?jq20w/xl2bJPVQfSpbQGc1uRbn9sOG4JLTnVsyRft4+n08AGuGREHzi4S/Fg?= =?us-ascii?Q?R8EKvXmi?= X-OriginatorOrg: virtuozzo.com X-MS-Exchange-CrossTenant-Network-Message-Id: 258df72e-7957-40ff-f3a1-08d8c47628a0 X-MS-Exchange-CrossTenant-AuthSource: AM7PR08MB5494.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 29 Jan 2021 16:51:44.1459 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 0bc7f26d-0264-416e-a6fc-8352af79c58f X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: ngwWV7/oXuOV5BUIoEP3xS7y677as8cYDcVHnGIDIcg+JMIZ3mp8yV3GHWbpYCOiiEdxJzZ+RnKdh9p4UpS75dS+bPmAiryzKRjVSDfSkFE= X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM6PR08MB3621 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=40.107.6.100; envelope-from=vsementsov@virtuozzo.com; helo=EUR04-DB3-obe.outbound.protection.outlook.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, MSGID_FROM_MTA_HEADER=0.001, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, vsementsov@virtuozzo.com, armbru@redhat.com, qemu-devel@nongnu.org, mreitz@redhat.com, den@openvz.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @virtuozzo.com) Content-Type: text/plain; charset="utf-8" Sometimes one of cells in a testing table runs too slow. And we really don't want to wait so long. Limit number of runs in this case. Signed-off-by: Vladimir Sementsov-Ogievskiy --- scripts/simplebench/simplebench.py | 29 +++++++++++++++++++++++++---- 1 file changed, 25 insertions(+), 4 deletions(-) diff --git a/scripts/simplebench/simplebench.py b/scripts/simplebench/simpl= ebench.py index f61513af90..78d90faf4d 100644 --- a/scripts/simplebench/simplebench.py +++ b/scripts/simplebench/simplebench.py @@ -19,9 +19,11 @@ # =20 import statistics +import time =20 =20 -def bench_one(test_func, test_env, test_case, count=3D5, initial_run=3DTru= e): +def bench_one(test_func, test_env, test_case, count=3D5, initial_run=3DTru= e, + slow_limit=3D100): """Benchmark one test-case =20 test_func -- benchmarking function with prototype @@ -36,6 +38,8 @@ def bench_one(test_func, test_env, test_case, count=3D5, = initial_run=3DTrue): test_case -- test case - opaque second argument for test_func count -- how many times to call test_func, to calculate average initial_run -- do initial run of test_func, which don't get into result + slow_limit -- reduce test runs to 2, if current run exceedes the limit + (in seconds) =20 Returns dict with the following fields: 'runs': list of test_func results @@ -47,17 +51,34 @@ def bench_one(test_func, test_env, test_case, count=3D5= , initial_run=3DTrue): 'n-failed': number of failed runs (exists only if at least one run failed) """ + runs =3D [] + i =3D 0 if initial_run: + t =3D time.time() + print(' #initial run:') - print(' ', test_func(test_env, test_case)) + res =3D test_func(test_env, test_case) + print(' ', res) + + if time.time() - t > 50: + print(' - initial run is too slow, so it counts') + runs.append(res) + i =3D 1 + + for i in range(i, count): + t =3D time.time() =20 - runs =3D [] - for i in range(count): print(' #run {}'.format(i+1)) res =3D test_func(test_env, test_case) print(' ', res) runs.append(res) =20 + if time.time() - t > 50 and len(runs) >=3D 2: + print(' - run is too slow, and we have enough runs, stop he= re') + break + + count =3D len(runs) + result =3D {'runs': runs} =20 succeeded =3D [r for r in runs if ('seconds' in r or 'iops' in r)] --=20 2.29.2 From nobody Tue Apr 30 19:35:43 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; arc=pass (i=1dmarc=pass fromdomain=virtuozzo.com); dmarc=pass(p=quarantine dis=none) header.from=virtuozzo.com ARC-Seal: i=2; a=rsa-sha256; t=1611940720; cv=pass; d=zohomail.com; s=zohoarc; b=Z+YLRJURK6vAQilyxJaq35q47+ss6zKcRVEStrVT72Yt3wDwGZYK6+EiiRmuLRXSVeRyujPIVkgsK9pjCWxRTx1/p/bVHIawjsm5Lqh6cK9Qd7sUU+b+JOV81Kv5gOaX0qzHi4lAu8KjDolXQ7kiEeYVpNk2JUVzHDTubQo/t6g= ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1611940720; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=AYzBXpgmuISa5N1uD0Uv9BisBpP67e04XXPa5K10Hes=; b=e4jVfdJpn/sknGHMFcbyh/2IvY6Bw57ukJ8kljW1jpehuk5v0SuNV+hrdP9Z0Uny3LwAFYQfBFv4EkpuACJF+lLpSqUY+jMn/QhWVZs9OQlEstX/D+KVmB1W6WqUaYJqx/pwqsrfDKuutHCzHc1wJGA6La5cv95suNn3btFEImc= ARC-Authentication-Results: i=2; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; arc=pass (i=1dmarc=pass fromdomain=virtuozzo.com); dmarc=pass header.from= (p=quarantine dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1611940720059883.6039224604498; Fri, 29 Jan 2021 09:18:40 -0800 (PST) Received: from localhost ([::1]:39610 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l5XPq-0001Xx-RG for importer@patchew.org; Fri, 29 Jan 2021 12:18:38 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:42884) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l5X0Z-0001cF-IO; Fri, 29 Jan 2021 11:52:31 -0500 Received: from mail-eopbgr60091.outbound.protection.outlook.com ([40.107.6.91]:6275 helo=EUR04-DB3-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l5X0Q-0007Pt-AL; Fri, 29 Jan 2021 11:52:31 -0500 Received: from AM7PR08MB5494.eurprd08.prod.outlook.com (2603:10a6:20b:dc::15) by AM6PR08MB3621.eurprd08.prod.outlook.com (2603:10a6:20b:4b::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3784.17; Fri, 29 Jan 2021 16:51:45 +0000 Received: from AM7PR08MB5494.eurprd08.prod.outlook.com ([fe80::f1f0:6610:11f5:5e4a]) by AM7PR08MB5494.eurprd08.prod.outlook.com ([fe80::f1f0:6610:11f5:5e4a%8]) with mapi id 15.20.3784.019; Fri, 29 Jan 2021 16:51:44 +0000 Received: from kvm.sw.ru (185.215.60.152) by AM0PR02CA0001.eurprd02.prod.outlook.com (2603:10a6:208:3e::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3805.16 via Frontend Transport; Fri, 29 Jan 2021 16:51:44 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=oM2JzjjStRb80cxNhoNsFsoyCCd34M6R1M78F/B+v5eCMmrX+eKrOnFPV4yU7G1SCbsD1taa3dOZ9IqiNI3M5LMV0dDVP+E4VJtWkzshnYyK0XvNm+lkoSnyWYO6FiL35mN3sdY4AutLSZCp853MpRSbmNyA1c8+akYN1hL7KcZE7Rm/gKoBpJxBGs8m0WcxwIUL10CebnJbLqI9aruUFGnVXSTzWEm3h5tQj4oQmwQxX9XLCm+VoquGSODGsfm6c8yJ+EHhg/shvCuw/m+J3Psa+KfGP47nXeFHag0fXHX22YB/2QT4vqWg29AK3VkCoLF53CpTICdl7kFCjkXCNA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=AYzBXpgmuISa5N1uD0Uv9BisBpP67e04XXPa5K10Hes=; b=Ol4xwY4DPHVjdCkVipDLA+gFaDhbP4vXnSKha/42AEbA53k1vxdR/uYOKM9NgVYruq/swPePCzxzMBk4NU9qGn/E/Rv8trzRNRnrS+yyYdgk+oQ4vBwYKWGKyxzkXKe97Asc67z0mRAiiBCx9ZxUMtGvbzkNPtp6J684uIFDCR/9o/N6RBy4HIpgxIJ/MnyK+qpKvLNp1Ps/kvgSV+oBE0tt/TaYT9GHZlV6EyrNvRNxsSEFUMHvMrp93047tDLK4A7Si/8ptsodxvGHAgrzTY2+AwGN1YmcAR4d/sZvcoXJZeJLx4QGgeK3NKjSOeOGLGIqUr1dlF6hoD+w/2+t/w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=virtuozzo.com; dmarc=pass action=none header.from=virtuozzo.com; dkim=pass header.d=virtuozzo.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=virtuozzo.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=AYzBXpgmuISa5N1uD0Uv9BisBpP67e04XXPa5K10Hes=; b=X1bdGJx2glnVK8Zte7ojC/G2VRKJULf26Cd4jhio6I0lqWMz9d5o8xV/6pAzbQh/0d0IS9RndJRbKL5IDVfezIQ2+fTX5PQCylC1K6qVQNeBSoFNx4QIYyyxvfTstmWl2QTw/6/l9a/mkcuaXJ85nbgZx3TQXD6YgHgFou1X5dQ= Authentication-Results: nongnu.org; dkim=none (message not signed) header.d=none;nongnu.org; dmarc=none action=none header.from=virtuozzo.com; From: Vladimir Sementsov-Ogievskiy To: qemu-block@nongnu.org Subject: [PATCH 5/7] simplebench: bench_one(): support count=1 Date: Fri, 29 Jan 2021 19:50:28 +0300 Message-Id: <20210129165030.640169-6-vsementsov@virtuozzo.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210129165030.640169-1-vsementsov@virtuozzo.com> References: <20210129165030.640169-1-vsementsov@virtuozzo.com> Content-Transfer-Encoding: quoted-printable X-Originating-IP: [185.215.60.152] X-ClientProxiedBy: AM0PR02CA0001.eurprd02.prod.outlook.com (2603:10a6:208:3e::14) To AM7PR08MB5494.eurprd08.prod.outlook.com (2603:10a6:20b:dc::15) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 82dc5d1a-1132-4aef-45fb-08d8c4762919 X-MS-TrafficTypeDiagnostic: AM6PR08MB3621: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:2043; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: ED1t+wOYCtVhqtStw8RSrV4rTYUT/nn8ywvzBF2MKlYPYSCMqfnwVxBKgzxiJWo0iz16CETeVuWknOEhUvEVQ7EzzJ4lMEjqUV8UN6Vk6QR9O6TZaQEvAJWnV5l3ISv0xOGptqtdJu0onmsf9jBV2MMbgBiTiC+l9pGwDyhBItSbErZGYVTVE4/Mh4aXEodIEtKTVQkuiFXuF9bDRdh83HgSurc6aONPR2wxakv8AlGlkgLndAXu7LD53GKjzExgA5xa1IMJpQJ7YeSjNHlQwFKtg2iHCjYKIGq0E85PSgVrJvw04MIlPl4EClbWB4uVbOXlJm599SL06ONVDnWN0E29PEOKX82nqKfmYzdXZrbnY/Plv1mdw6MK9cQukUeQSEyMeA8HmJXeygbGRerhFZDR4CHAKR9SVTq5phJmDFfsZ4yJDT1G4TqOP/AJkhK6FhaiPT7RXcYxN6Rs6XlPMTtpOvVDKZBThEqD406JgYLsCnApOC2EeXrEbvk99Wx53n4frzkpAA45HO1338xCng== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:AM7PR08MB5494.eurprd08.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(346002)(366004)(396003)(136003)(39840400004)(376002)(66476007)(83380400001)(1076003)(86362001)(8936002)(6486002)(316002)(52116002)(6512007)(66946007)(8676002)(2616005)(2906002)(66556008)(36756003)(4744005)(4326008)(16526019)(6506007)(26005)(186003)(107886003)(956004)(6916009)(6666004)(5660300002)(478600001); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData: =?us-ascii?Q?pFeD1sYpG9/CWgkGqe62C49NhDfq4XVenlVqdSGytgTcef437oaqVik0z2Vl?= =?us-ascii?Q?FB9M+Ex9EHd5L0m3cfQOpkMX/RcLNdlkquPXXxMU3HFMrTz7/z3wbzzC2rFD?= =?us-ascii?Q?UMCYvG/m+H0TY003OhSpM4yRgOsH3CJgiFZkzzS+ALzZ+XKJDc+oAlb2z221?= =?us-ascii?Q?/lEcMmZ834u1yRzqSw4jQcMihAU/XYMDUb1V661QXyzQBSSpa7STHJ7X9Z2y?= =?us-ascii?Q?xozS9sdWgFw2C9jWEiRWeV5j/SPFzVbg3xxMfCDPCGRNxUbzfZUN5U/xMti5?= =?us-ascii?Q?sDrWSLbD4betCJHxQbZP2fUIaYd2uxWV2yBnUlxE3PR2ejYXXVXqNQC+yH7x?= =?us-ascii?Q?vgHmzahaC+oQ3iCdQNlaa2BYhoatJeVR2G2jaJ6QU+VjZD8hruACoNjgkXeI?= =?us-ascii?Q?u5mwKeJXhpvpHAsT6+PxTJFUl5tznMi6o0TOtfj6Q3Lz1/wfbVJdh/vETimH?= =?us-ascii?Q?4ScyjlTlYArKySWicizzJPApzQ9Q9P9YzBrEdXBuIx7bOxzEVxyOqWD9nnUq?= =?us-ascii?Q?H2v1WLesf8CpAOP3glSKUwQF/MZTF2MknbSwbt1qvxuZLs41vPC8BAhodgrv?= =?us-ascii?Q?b6rrCRJi3of+sfD3+Otafrtdm43KnZCvJQ96iQo+6KS5yL9QpP/1FGaySaCS?= =?us-ascii?Q?fAw+ZeihWlDusmfcXidUbhfgPtb1h8y6SWsyWw6ge8cOZHJT4gvCwXc/TKtL?= =?us-ascii?Q?Fo5i/2HsSgOOJSg1Z/6fltiT586NMbOpCUd3tIGKRXqegwu5wCtkmSPjumzS?= =?us-ascii?Q?EOisCUV7iudo3PWaXBFfo80dxDVr9ubMzzYhcOx6UZYl35E76DyZVCihOC72?= =?us-ascii?Q?m6F/EyNiqkhGZlGK0on1zkczxIEsDBtRdVIiWW+IYIQlCGLXTaEOmLfwW2bp?= =?us-ascii?Q?b/Md802i51vDUeBYCKVUg/DTRTQJzHREiznDppqtE3/M6ypeXgOgRRjYK43q?= =?us-ascii?Q?cgSRYw6vCyRWou81gqnESHozynT2Sb9XB8ZRqW+kj55ez+A0XQ6klgEbriCk?= =?us-ascii?Q?iVRsG/aCozPS+HQNCCntegX1xZCGwfrFvc87DT73DevxMuiJUi0w2smTzaAJ?= =?us-ascii?Q?ItOg6qJU?= X-OriginatorOrg: virtuozzo.com X-MS-Exchange-CrossTenant-Network-Message-Id: 82dc5d1a-1132-4aef-45fb-08d8c4762919 X-MS-Exchange-CrossTenant-AuthSource: AM7PR08MB5494.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 29 Jan 2021 16:51:44.9135 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 0bc7f26d-0264-416e-a6fc-8352af79c58f X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: lEBobpzE4HPXyaUZEs1NvrLCYvwAQaPHbfV1w1/3ByTKe4q/lIutw5+LJOxr9b7O9GkmyPrBRrBQSfR8G4tIfyZxb4T7Zelmg61qIW6oRPU= X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM6PR08MB3621 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=40.107.6.91; envelope-from=vsementsov@virtuozzo.com; helo=EUR04-DB3-obe.outbound.protection.outlook.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, MSGID_FROM_MTA_HEADER=0.001, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, vsementsov@virtuozzo.com, armbru@redhat.com, qemu-devel@nongnu.org, mreitz@redhat.com, den@openvz.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @virtuozzo.com) Content-Type: text/plain; charset="utf-8" statistics.stdev raises if sequence length is less than two. Support that case by hand. Signed-off-by: Vladimir Sementsov-Ogievskiy --- scripts/simplebench/simplebench.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/scripts/simplebench/simplebench.py b/scripts/simplebench/simpl= ebench.py index 78d90faf4d..407e95cec3 100644 --- a/scripts/simplebench/simplebench.py +++ b/scripts/simplebench/simplebench.py @@ -92,7 +92,10 @@ def bench_one(test_func, test_env, test_case, count=3D5,= initial_run=3DTrue, dim =3D 'seconds' result['dimension'] =3D dim result['average'] =3D statistics.mean(r[dim] for r in succeeded) - result['stdev'] =3D statistics.stdev(r[dim] for r in succeeded) + if len(succeeded) =3D=3D 1: + result['stdev'] =3D 0 + else: + result['stdev'] =3D statistics.stdev(r[dim] for r in succeeded) =20 if len(succeeded) < count: result['n-failed'] =3D count - len(succeeded) --=20 2.29.2 From nobody Tue Apr 30 19:35:43 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; arc=pass (i=1dmarc=pass fromdomain=virtuozzo.com); dmarc=pass(p=quarantine dis=none) header.from=virtuozzo.com ARC-Seal: i=2; a=rsa-sha256; t=1611941247; cv=pass; d=zohomail.com; s=zohoarc; b=n9ZcxZgheFICFiDDwjm38jtZ/Ro6/ldG8ykTRQtbajtM9yGSGCtlmqbeYUvr2KgckJi8yGry64w55K6Ho4iu7jtKVw+6PAdGx6e8oAwUrRc2px21DpoLquo7SMRvd68z4Au04fgcpuyh2H9LFyhPZqx9+OAt01CAg2SzT48n78A= ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1611941247; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=T2ex50jNXYIs75xlJ8miTYosT5mQnjvdw+cBwQtNH7Q=; b=fdoI63aq2kipWx5QJXutzQYIuPbN7bTjo5Ul+3wsmTm2hBepp5/UyoG3heIzOEdrYacEYroNhRXxdnBek++nwHHBJS436Kt/BcHHbnAA2zJwomibEtMgwBqbSd4zRZiFGxPEY7XjLarOVH1I2oGZ6SrWpMZIWVRqUY2YsNhgH4c= ARC-Authentication-Results: i=2; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; arc=pass (i=1dmarc=pass fromdomain=virtuozzo.com); dmarc=pass header.from= (p=quarantine dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1611941247028709.3806553035328; Fri, 29 Jan 2021 09:27:27 -0800 (PST) Received: from localhost ([::1]:60912 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l5XYL-0002Fb-Tn for importer@patchew.org; Fri, 29 Jan 2021 12:27:25 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:42886) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l5X0a-0001dY-9V; Fri, 29 Jan 2021 11:52:32 -0500 Received: from mail-eopbgr60100.outbound.protection.outlook.com ([40.107.6.100]:29349 helo=EUR04-DB3-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l5X0Q-0007OV-Rh; Fri, 29 Jan 2021 11:52:31 -0500 Received: from AM7PR08MB5494.eurprd08.prod.outlook.com (2603:10a6:20b:dc::15) by AM6PR08MB3621.eurprd08.prod.outlook.com (2603:10a6:20b:4b::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3784.17; Fri, 29 Jan 2021 16:51:45 +0000 Received: from AM7PR08MB5494.eurprd08.prod.outlook.com ([fe80::f1f0:6610:11f5:5e4a]) by AM7PR08MB5494.eurprd08.prod.outlook.com ([fe80::f1f0:6610:11f5:5e4a%8]) with mapi id 15.20.3784.019; Fri, 29 Jan 2021 16:51:45 +0000 Received: from kvm.sw.ru (185.215.60.152) by AM0PR02CA0001.eurprd02.prod.outlook.com (2603:10a6:208:3e::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3805.16 via Frontend Transport; Fri, 29 Jan 2021 16:51:45 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=MVHebQ0rkJxtayeFHZbjIcU628yBGi/Mq/oXMmiUGzDs6PvHXeHaxNy7oK6PdBT8L1yNCriW14UCGzKjZGmKiAbKU9VZ+WUAxUCPhZunFT1cYZSq02bLnwHEqKIJV+0FTPVtHgqkUflcy17q+JqN24KVUWD/ectGhvuQtYDOSQFZ9s5ohtFTh5w/ehtqY+OTQis9tvsbsTs1wNeUUmCZmWoJRiW6MOx1CGdTlJIKu9z09QRakYGbDL2IlHHP/R4awFxaan2f9nddx6ApEJPMZ/x7Q6R4Ljlr1ExGh54o/yheyCgXZzmqfxQVbGPKT42aaMfapAtoOWvD0634FGV3ag== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=T2ex50jNXYIs75xlJ8miTYosT5mQnjvdw+cBwQtNH7Q=; b=DMALUtfn8m7OwnR2PKJ3DQvKGhEXlJMroe6xkZS8LzDLh6stfvXz+EJ/54CPFHv6W1TyAs3dudHQfRpUUGAF8vQUV2/QIAqrm6SKhHQBgCMqEkwBIjYPxbF3CAPbyDIRpK2ohIJuanZwR53hg2p/lp9jEpiiSnU7VFHlB0JvVUx4RvHaLglANsZqz473axIFQwBx1e5FTVB7hXhveamdTdiyQz2l4/ZRF75rsKCbWvakKxIeA5kiuR7JKw885GreSBCq52Sy2tq7TaUK2/CRmoVYIZ3G2dZdwndKz1vfNPVaP7twQt4t7gjWEz5OQu6xpR3kTdfbIH/9NFPdXYvW8Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=virtuozzo.com; dmarc=pass action=none header.from=virtuozzo.com; dkim=pass header.d=virtuozzo.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=virtuozzo.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=T2ex50jNXYIs75xlJ8miTYosT5mQnjvdw+cBwQtNH7Q=; b=CMxk4/D2/1Bl7uRam/32xpoM/DOIE7fEPuK9Bjz0I3tQpelR8hfu0jSNjkIbhBL1+Tu1ltD24/Nhc0MJk+PScTjVvWt4OwxUZbGVi8d9QKpMMkt3IACb1cCxlMkyrAz5BTxHU9vtaThmfSKE7rzbeCTBQTT1e9h2fQwM23ePnqA= Authentication-Results: nongnu.org; dkim=none (message not signed) header.d=none;nongnu.org; dmarc=none action=none header.from=virtuozzo.com; From: Vladimir Sementsov-Ogievskiy To: qemu-block@nongnu.org Subject: [PATCH 6/7] simplebench/bench-backup: add --compressed option Date: Fri, 29 Jan 2021 19:50:29 +0300 Message-Id: <20210129165030.640169-7-vsementsov@virtuozzo.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210129165030.640169-1-vsementsov@virtuozzo.com> References: <20210129165030.640169-1-vsementsov@virtuozzo.com> Content-Transfer-Encoding: quoted-printable X-Originating-IP: [185.215.60.152] X-ClientProxiedBy: AM0PR02CA0001.eurprd02.prod.outlook.com (2603:10a6:208:3e::14) To AM7PR08MB5494.eurprd08.prod.outlook.com (2603:10a6:20b:dc::15) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 4f75c1b8-2a6c-4af7-5ace-08d8c4762992 X-MS-TrafficTypeDiagnostic: AM6PR08MB3621: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:6430; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: EOcSaJafw3rs9n9CR5ayCXY/i4jIEyawWMbgNASMZrdPrr9WhXzc3iagDBR6mphtEvghXJy0Ua9Dw3PaVBdgVD3fiSb+9rmbkX8UHD/+oxXboU1QSJiBZDNSpCfS4xtkvOWXIH6CWZjU0CwVyq5LogOrUOJQ5D1kOKnmXra3oTn95H9dRS7WBKc72oZHZIBog/F0XDggoNfLBrYvKg8We5I/7TxuyDdJTQ9ZaTNk1bxcyRGYjWo6akse+U9DzkaNGNYF3qweT0nzzIWfzcnJ5DHlZKfdjvp80MuOWK+WDRSyt3neoENu/Rm4QgYIbYQHUzWWHMbPjt4d4p4Az+/G0i2VH1EDopKfC8jUUHQwDEztK7MduZEUR1kiGoAcSif7HFoslIQQugxWRx9fL6PANOU68/2SwWq7oEGsHLRsjRc8RGPEPEhIlUZga7XGMjg/mG9RCZX8iTHiFt4pLf+zLCJALLNHw5TYUACIyMlFI9cC797pNnXktEOTwEvYUCpOHMqiRNmzvyu9AK3dSSOYlQ== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:AM7PR08MB5494.eurprd08.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(346002)(366004)(396003)(136003)(39840400004)(376002)(66476007)(83380400001)(1076003)(86362001)(8936002)(6486002)(316002)(52116002)(6512007)(66946007)(8676002)(2616005)(2906002)(66556008)(36756003)(4326008)(16526019)(6506007)(26005)(186003)(107886003)(956004)(6916009)(6666004)(5660300002)(478600001); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData: =?us-ascii?Q?8xhW07+Oo6RNS2ayXebcB+Qsfc0SaOhkl5lsDuSdbPnXn1JXREm5GJGJlhd4?= =?us-ascii?Q?nalzfAJrP4+i3qCH6mFA9Na+7KgKRf6jn5wVG5ZfqLP7Nj6plG3tSwCMIFZp?= =?us-ascii?Q?fBvzj7a/UVtoCf1evB3HafDrBQvatU0w2o6jRp35mO/u2Z/1u8pMY35MbRIl?= =?us-ascii?Q?TlRytPZLOqNb1kFN+D3uPMcCNsgJb01DzWy5KBZ9H+ug9VU9cN228pj3mYJ2?= =?us-ascii?Q?5ld6WE2VpBoOjDvKlcTIxemBy1tzhIS+i6H7YFxb/RafQp7KW2j66zK0rRov?= =?us-ascii?Q?+pxHDro0J/CujrrbTtKiZpI4Y78Rltj+tr5kFHtOpyhKN4Kv7CPT/XGFChMC?= =?us-ascii?Q?GuQuAk6PGMJaAAhQHuV9UCqPB67Fml/hBjT3I9VYwA0Nw18DjBJt2VCLcZ71?= =?us-ascii?Q?sZZAL6V2gf5V7o504gKrEJ2v65uYXpYP3GCJMGxIWD92k1+FFY2KnGMieR4s?= =?us-ascii?Q?yjvoJqo7LVgIU8g9SwKh0r07SKCTAeuLHo/3ukznDQrVXwmGpq5uIXd3aHWo?= =?us-ascii?Q?XyOvK/WckmUj9GshRcA3lKIPMGrqn9NXq/qL90I/LHDmCpcbwdKH1f8AO03M?= =?us-ascii?Q?w3rh5tpwPKwOS5OF/rPR5MZPnuBqgRc4I17pUtLZM3iLpadeDfmAwO6akocL?= =?us-ascii?Q?Qd+beXt/pfyXCdAdm2hDtw6pBT268iMz+1BTfVrWJ1hZ8mc3PujDGvItEGsI?= =?us-ascii?Q?3OibPwBw7sfKbiyZO6+yAC0GRVlKaN2ZzkSrO5JC2EwYAU6K4V4FsdIShnm1?= =?us-ascii?Q?rULV5VjI2LxESpbTB6iQuCUzzRozmsRfeBDNzJVuUnkAGpq9nqSyaZqNbnDS?= =?us-ascii?Q?xjoMcNLrHB7rFdWidGQD8LcRxtY/iNRhV2IfAGTFdmsH/aVa7BxKKyzw2+sD?= =?us-ascii?Q?qKEIfdXVo1c63LUqPYMWk1a/qGmEdw+2q5r3XmEn8INrGlDRHnbBXiy5/A88?= =?us-ascii?Q?tB1aQIeSTT4CQ0OQLDx6wBfy/24rDg3ZVghKpjp5ComVokpEou6YbbmW+FzY?= =?us-ascii?Q?WB1QVvV3yeGRbzNalaZV+/Gy0sFV5fKLnNQTNp64VU7ZKcGipGeW3/ZOQ9Wf?= =?us-ascii?Q?yTjBGEeq?= X-OriginatorOrg: virtuozzo.com X-MS-Exchange-CrossTenant-Network-Message-Id: 4f75c1b8-2a6c-4af7-5ace-08d8c4762992 X-MS-Exchange-CrossTenant-AuthSource: AM7PR08MB5494.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 29 Jan 2021 16:51:45.7031 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 0bc7f26d-0264-416e-a6fc-8352af79c58f X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: m2Y8kC4hp7AZ65T6T4OHmqTqBZZ8qsHEGTyRdwe4hW6jrkcnYwjbCztXrZ3dIvwHvuEIcWbm5u1c8Ldu+CQSElhg0g9Y5ilfEb+VUXrsTB0= X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM6PR08MB3621 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=40.107.6.100; envelope-from=vsementsov@virtuozzo.com; helo=EUR04-DB3-obe.outbound.protection.outlook.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, MSGID_FROM_MTA_HEADER=0.001, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, vsementsov@virtuozzo.com, armbru@redhat.com, qemu-devel@nongnu.org, mreitz@redhat.com, den@openvz.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @virtuozzo.com) Content-Type: text/plain; charset="utf-8" Allow bench compressed backup. Signed-off-by: Vladimir Sementsov-Ogievskiy --- scripts/simplebench/bench-backup.py | 55 ++++++++++++++++++-------- scripts/simplebench/bench_block_job.py | 23 +++++++++++ 2 files changed, 62 insertions(+), 16 deletions(-) diff --git a/scripts/simplebench/bench-backup.py b/scripts/simplebench/benc= h-backup.py index 33a1ecfefa..72eae85bb1 100755 --- a/scripts/simplebench/bench-backup.py +++ b/scripts/simplebench/bench-backup.py @@ -23,7 +23,7 @@ import json =20 import simplebench from results_to_text import results_to_text -from bench_block_job import bench_block_copy, drv_file, drv_nbd +from bench_block_job import bench_block_copy, drv_file, drv_nbd, drv_qcow2 =20 =20 def bench_func(env, case): @@ -37,29 +37,41 @@ def bench_func(env, case): def bench(args): test_cases =3D [] =20 - sources =3D {} - targets =3D {} - for d in args.dir: - label, path =3D d.split(':') # paths with colon not supported - sources[label] =3D drv_file(path + '/test-source') - targets[label] =3D drv_file(path + '/test-target') + # paths with colon not supported, so we just split by ':' + dirs =3D dict(d.split(':') for d in args.dir) =20 + nbd_drv =3D None if args.nbd: nbd =3D args.nbd.split(':') host =3D nbd[0] port =3D '10809' if len(nbd) =3D=3D 1 else nbd[1] - drv =3D drv_nbd(host, port) - sources['nbd'] =3D drv - targets['nbd'] =3D drv + nbd_drv =3D drv_nbd(host, port) =20 for t in args.test: src, dst =3D t.split(':') =20 - test_cases.append({ - 'id': t, - 'source': sources[src], - 'target': targets[dst] - }) + if src =3D=3D 'nbd' and dst =3D=3D 'nbd': + raise ValueError("Can't use 'nbd' label for both src and dst") + + if (src =3D=3D 'nbd' or dst =3D=3D 'nbd') and not nbd_drv: + raise ValueError("'nbd' label used but --nbd is not given") + + if src =3D=3D 'nbd': + source =3D nbd_drv + else: + source =3D drv_file(dirs[src] + '/test-source') + + if dst =3D=3D 'nbd': + test_cases.append({'id': t, 'source': source, 'target': nbd_dr= v}) + continue + + fname =3D dirs[dst] + '/test-target' + if args.compressed: + fname +=3D '.qcow2' + target =3D drv_file(fname) + if args.compressed: + target =3D drv_qcow2(target) + test_cases.append({'id': t, 'source': source, 'target': target}) =20 binaries =3D [] # list of (