From nobody Fri May 17 13:29:24 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; arc=pass (i=1dmarc=pass fromdomain=virtuozzo.com); dmarc=pass(p=quarantine dis=none) header.from=virtuozzo.com ARC-Seal: i=2; a=rsa-sha256; t=1614968312; cv=pass; d=zohomail.com; s=zohoarc; b=UvwXTtUWvLlbn/FBY6AUe9sAjo9olt5GpiKBG35ZDkvBeZQn+8Srye/t2lmTps8lKFyKPR4ikYkRMsykbWC5TB6xaXz4z92I7by/Y05WlyxqSdf+3xN2dPVb7w6bck40o4kzqKDe4hWK8tkFKbJrbDir8+qoxCWCjgRKkTU1YjU= ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1614968312; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=1CdTBvjQtHCu8YdqT4pSi7KK3mGuUddtjwrGa38khIM=; b=Z7UW0iL8Nw7i7kYTrLNxelz/LZmlqPGIZ5GP8zpcnIFkKBZB36jSEtEkFwjSZr+D6ieTN9I3YbFN7uJFXy3rhrPcTQoI+KP+K3858rD0H1VjxClJRTJKcqseT/ghPbrr8Vsdev5PvocufmYOBqT8ae0fHhVh4tiwBypENMfL5Y8= ARC-Authentication-Results: i=2; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; arc=pass (i=1dmarc=pass fromdomain=virtuozzo.com); dmarc=pass header.from= (p=quarantine dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1614968312502893.5054533496876; Fri, 5 Mar 2021 10:18:32 -0800 (PST) Received: from localhost ([::1]:53048 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lIF1z-0002GD-7X for importer@patchew.org; Fri, 05 Mar 2021 13:18:31 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:42764) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lIEMT-0004fH-8W; Fri, 05 Mar 2021 12:35:37 -0500 Received: from mail-eopbgr30124.outbound.protection.outlook.com ([40.107.3.124]:32132 helo=EUR03-AM5-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lIEMO-0005C4-Tp; Fri, 05 Mar 2021 12:35:37 -0500 Received: from AM7PR08MB5494.eurprd08.prod.outlook.com (2603:10a6:20b:dc::15) by AM6PR08MB4408.eurprd08.prod.outlook.com (2603:10a6:20b:be::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3890.19; Fri, 5 Mar 2021 17:35:24 +0000 Received: from AM7PR08MB5494.eurprd08.prod.outlook.com ([fe80::f1f0:6610:11f5:5e4a]) by AM7PR08MB5494.eurprd08.prod.outlook.com ([fe80::f1f0:6610:11f5:5e4a%8]) with mapi id 15.20.3912.024; Fri, 5 Mar 2021 17:35:24 +0000 Received: from localhost.localdomain (185.215.60.202) by HE1PR02CA0119.eurprd02.prod.outlook.com (2603:10a6:7:29::48) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3912.17 via Frontend Transport; Fri, 5 Mar 2021 17:35:23 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=mZc6ol6sz7Y8aVr9vxfyl4/CqdWeTaufRrncjSdzoeUzlWCO1M2+MMxqBhZcnZ7Pp4JDaQ9eVunySGBgMBfqwXqm7pugS/x+W5Z6QOoZ3027E+jd7uC3UyTGgot9uXep+BotAWjKl0sX/gNgTy/kiuZ2v2JdF50/cqHpbX8u2p6zw2AQQO2rXdA3gOhYHZBSmVSmMxVUq2cWRTehxICv4cjJN6z1eeAsJsjYKL4PWx5n5TYqTg2Q6RRmkWgvrFB9Pne6dDxAfYl8YBmJ72DfWCEW0dC1iRZqbLLTZQxIEjWqmoKImaiyVuqxqGgQoMQMLpU+4fALcGpS+7mS9cJWoA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=1CdTBvjQtHCu8YdqT4pSi7KK3mGuUddtjwrGa38khIM=; b=Ghszr4PK/+BcQOmL0v3pkIGu6W4vto4l+vKldRt7V9izax6CjatX7j6aOwOjUgamBBdsu5jZHHWA8UzI3qP7ZhLKQ5b9jLnzZpdzKlVNI3Wrr/Ya9yD36JE/mhzxeK3jtUdghjnfO6qy3uwFuJLMewleBlvhM4eddmRieNS7FdE2aZgJy71EHdAewZw0w0P3EDVQMY8jBCQmjwFieb9hDGGZDISTgNi62PWZOupqblEW1R/1GJ96Ai3H+77VyYgckFGAJb57QwcVajs5f4K7U9Xjs2P8fnDyWUIEcYY+vNFSkRHyzHBG60o2wRFbE+daJ7ouKtfMDJTOMzo1hXVp+w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=virtuozzo.com; dmarc=pass action=none header.from=virtuozzo.com; dkim=pass header.d=virtuozzo.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=virtuozzo.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=1CdTBvjQtHCu8YdqT4pSi7KK3mGuUddtjwrGa38khIM=; b=fgdHrzau3UQ7ayesEyu2c5AwvXH7rhK59NCUaNJJj81eszRk4/GLEriWHBKYtcOo3FgPLUWI2ZPDrjlm24w7ScMXVHyJg4HQolkHs2rwop3c1NyerK3SuX52cUzbQZqN5EPU9DNQDpvEkufqqlQOILjYQeZM+DctbCFIvW2uqeM= Authentication-Results: nongnu.org; dkim=none (message not signed) header.d=none;nongnu.org; dmarc=none action=none header.from=virtuozzo.com; From: Vladimir Sementsov-Ogievskiy To: qemu-block@nongnu.org Cc: qemu-devel@nongnu.org, crosa@redhat.com, ehabkost@redhat.com, mreitz@redhat.com, kwolf@redhat.com, vsementsov@virtuozzo.com, jsnow@redhat.com Subject: [PATCH v3 1/6] block-jobs: flush target at the end of .run() Date: Fri, 5 Mar 2021 20:35:02 +0300 Message-Id: <20210305173507.393137-2-vsementsov@virtuozzo.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210305173507.393137-1-vsementsov@virtuozzo.com> References: <20210305173507.393137-1-vsementsov@virtuozzo.com> Content-Transfer-Encoding: quoted-printable X-Originating-IP: [185.215.60.202] X-ClientProxiedBy: HE1PR02CA0119.eurprd02.prod.outlook.com (2603:10a6:7:29::48) To AM7PR08MB5494.eurprd08.prod.outlook.com (2603:10a6:20b:dc::15) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 1f3e3e65-23aa-4d20-bfc2-08d8dffd0f17 X-MS-TrafficTypeDiagnostic: AM6PR08MB4408: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:655; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: AlgZqjWXujqEXbDvJDgWLMloJuqux/rPz0dm+5PPxaa7zV+I8sPpxX7wsJQbWLD96K03/QBMWlTaUSDOWYy3+EPQqq1yucaWM9GKAxISf6PFLJ5H2y6z3bs1e8UC4MCzXU6sVdOGm9r+t2Kh16lOsTdZrEz+Jmx5vhZkbRzUucf2Ns5wj4yj+I1LVe6M2+uR3FKNq8yTxXdYi76SEW4vpq8AzEki7saLNMfhQCyfxUfztLJ6bSK2RrTIqyeBJp5MuU7tLC0COieR3HKUwM7q+en6GQ1JvY2uQ/ePXrAqRBPxf5I2W32277qjVoSxOBE5kJ4qgUHxnWOglDfV8zQreRy8JMymnXwR7qXCPNWGnVr6cXADdq7CUApXLYeke8wm59s6DNNXtEDS04wAERT0eoHv4ojaZp39seXtM/Yh2XN2008xkv3Y2hkBSA6UHXmd4hiGCf7vyYprBqlH1VYZKKMtCKqks09gN12TeL6Iiqlt3x5dO3a74XgXEzsdYD8Ix1jieHSEEAsn5KGGcfevqmeFXOaoBUC+sTKsJ0tgwELI4LgVzAT+OZLReB78sGT0/ezTSB/Xir+soWReVEz8S4v3PJZQ+cA/W9mGLd3hhrw= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:AM7PR08MB5494.eurprd08.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(136003)(366004)(39850400004)(396003)(376002)(346002)(16526019)(66476007)(478600001)(186003)(8936002)(8676002)(6916009)(1076003)(956004)(6486002)(83380400001)(69590400012)(52116002)(26005)(6666004)(6506007)(2906002)(2616005)(4326008)(66556008)(316002)(6512007)(36756003)(66946007)(86362001)(5660300002)(142923001); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData: =?us-ascii?Q?UVkSQvnhNiE8lw8kA4GjBCV7TaMcg/Ni54rtitolVrddUG5IwdOS/7OqQZAU?= =?us-ascii?Q?CDg+2oetAyokjDEmFnKB2uXqn9tfzo4jP7JHhsdq7n2gxyTURBxcJtdJoooU?= =?us-ascii?Q?cmGCUTt/DiG6DuoJuekADNbKpDgAdlek/BvSh4+Ha4iS9vsF58fJa5dl3wzB?= =?us-ascii?Q?dLS7vsX9y20GDU2QcAgzZ7G5DuzYips49PTlxr8a3MfKdW5g6g2xZByS6el6?= =?us-ascii?Q?LUtjMgsOg19b3G0m/l79G8MAE6qsgBpPIEC2m5FzJtJnKMGDkhvJ3Dk5jwcS?= =?us-ascii?Q?3G/WDx+iLL9cVSGXF7qbK2FX5gF4LS/S85sCp2AbiDtefD/uYrOOHZVS6JJr?= =?us-ascii?Q?7aOgeHG0MQemPmdYnWQC9YdDxPb7mteI9qY7CVs2ZDEk+RFuKb5tZuThBkEH?= =?us-ascii?Q?qulmrIGXGYVzKl8YlZKv4QOCf1aC1IniOgKRrpVClFs7FvfeYMtKTUR0OoY6?= =?us-ascii?Q?lbdAQllKWRzZ0LiHq4Uc79bRvx1McFitNRqzx7i95ojkQTiBLJeZsm35q+Ou?= =?us-ascii?Q?cVWvCeRDsyJnuzuyRnLgarBrrn9Lb46FbCk+Qu1Kul+5QUQU2YcQhH75IWKR?= =?us-ascii?Q?sJCUDSOmGkpg7UtadV+N+9BFdt0Lq1Mov1zp6qbkzRVZoPHc08KCNvwKQVKh?= =?us-ascii?Q?nZShkqDhYg9h08f5AO7GLOY57RQShmlEbkTfJyF3mjHIxGEacveHFpkM3oT1?= =?us-ascii?Q?I8MFZyzMLDxno3/f9xDI3noHYg8/fmjdP7VSnh7aCPCDcBgsWZtOmqbaPNkd?= =?us-ascii?Q?sK4pnCemna95Ry2EB6RSx30mZhZYzwoM8Ox26B/ipq+bqKKo4uq+coiYzghQ?= =?us-ascii?Q?Sryok4KqTg4ocDUaeJAoWEU/Too+qUA4hYxu4gaMMMNNvOnzYwnpXd/wZU4D?= =?us-ascii?Q?4pcTH+wVlFJdlmdwdhQbAwl70vGAcfNSge9f5FchWCIdOfcCRGCOBNceIzOp?= =?us-ascii?Q?YfGvlWyqgOkDGNwTVQMtCKgekONar6F0GeBMyTQ31hpp0cx+efu0gzNHuZTE?= =?us-ascii?Q?00bvp2vHvdOcd78Xj6POG3YPYGWDqJorE5zJoQRVhsqGtuwdlWEeoKFFdYzH?= =?us-ascii?Q?k/HaUAgodGikUyOXMkzLpaAc90xSFrLXUhdjreklwrrnIw9eX+USsm7haAl0?= =?us-ascii?Q?0QCxWS1fJElQU9b7WLzW2meMPECiCLzJW1H7s18P4h2mcEhMgUyiWeXpsPdN?= =?us-ascii?Q?ez9ZovGAeJrtJbeaYwt+DoDDda+PuP2kNgwVa/2I4/DOjsW6gP3W17YoPI1T?= =?us-ascii?Q?xcoY1WQTit29uOcU47uJURFyhz1HMbb2obxnRYr7dJB4QtsWx2ZSGZpFnEzN?= =?us-ascii?Q?qqYgH0oXi5PQbK2zrLd8gclm?= X-OriginatorOrg: virtuozzo.com X-MS-Exchange-CrossTenant-Network-Message-Id: 1f3e3e65-23aa-4d20-bfc2-08d8dffd0f17 X-MS-Exchange-CrossTenant-AuthSource: AM7PR08MB5494.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Mar 2021 17:35:24.7298 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 0bc7f26d-0264-416e-a6fc-8352af79c58f X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: g0TJz5ySohXcbx9kUnrvEbuYDAXhHkNfETTUqhWn6mXuSLdsGJN+WElotEdI/DLDe6isNxcV/G+tg2TEnYFmh4Wh6tNS68X1+wuoHK4tEKo= X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM6PR08MB4408 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=40.107.3.124; envelope-from=vsementsov@virtuozzo.com; helo=EUR03-AM5-obe.outbound.protection.outlook.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, MSGID_FROM_MTA_HEADER=0.001, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @virtuozzo.com) Content-Type: text/plain; charset="utf-8" We are going to implement compressed write cache to improve performance of compressed backup when target is opened in O_DIRECT mode. We definitely want to flush the cache at backup finish, and if flush fails it should be reported as block-job failure, not simply ignored in bdrv_close(). So, teach all block-jobs to flush their targets at the end. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz --- include/block/blockjob_int.h | 18 ++++++++++++++++++ block/backup.c | 8 +++++--- block/commit.c | 2 ++ block/mirror.c | 2 ++ block/stream.c | 2 ++ blockjob.c | 16 ++++++++++++++++ 6 files changed, 45 insertions(+), 3 deletions(-) diff --git a/include/block/blockjob_int.h b/include/block/blockjob_int.h index 6633d83da2..6ef3123120 100644 --- a/include/block/blockjob_int.h +++ b/include/block/blockjob_int.h @@ -119,4 +119,22 @@ int64_t block_job_ratelimit_get_delay(BlockJob *job, u= int64_t n); BlockErrorAction block_job_error_action(BlockJob *job, BlockdevOnError on_= err, int is_read, int error); =20 +/** + * block_job_final_target_flush: + * @job: The job to signal an error for if flush failed. + * @target_bs: The bs to flush. + * @ret: Will be updated (to return code of bdrv_flush()) only if it is ze= ro + * now. This is a bit unusual interface but all callers are comforta= ble + * with it. + * + * The function is intended to be called at the end of .run() for any data + * copying job. + * + * There are may be some internal caches in format layers of target, + * like compressed_cache in qcow2 format. So we should call flush to + * be sure that all data reached the destination protocol layer. + */ +void block_job_final_target_flush(BlockJob *job, BlockDriverState *target_= bs, + int *ret); + #endif diff --git a/block/backup.c b/block/backup.c index 94e6dcd72e..d3ba8e0f75 100644 --- a/block/backup.c +++ b/block/backup.c @@ -255,7 +255,7 @@ static void backup_init_bcs_bitmap(BackupBlockJob *job) static int coroutine_fn backup_run(Job *job, Error **errp) { BackupBlockJob *s =3D container_of(job, BackupBlockJob, common.job); - int ret; + int ret =3D 0; =20 backup_init_bcs_bitmap(s); =20 @@ -297,10 +297,12 @@ static int coroutine_fn backup_run(Job *job, Error **= errp) job_yield(job); } } else { - return backup_loop(s); + ret =3D backup_loop(s); } =20 - return 0; + block_job_final_target_flush(&s->common, s->target_bs, &ret); + + return ret; } =20 static void coroutine_fn backup_pause(Job *job) diff --git a/block/commit.c b/block/commit.c index dd9ba87349..1b61b60ccd 100644 --- a/block/commit.c +++ b/block/commit.c @@ -193,6 +193,8 @@ static int coroutine_fn commit_run(Job *job, Error **er= rp) ret =3D 0; =20 out: + block_job_final_target_flush(&s->common, blk_bs(s->base), &ret); + qemu_vfree(buf); =20 return ret; diff --git a/block/mirror.c b/block/mirror.c index 1803c6988b..bc559bd053 100644 --- a/block/mirror.c +++ b/block/mirror.c @@ -1095,6 +1095,8 @@ immediate_exit: g_free(s->in_flight_bitmap); bdrv_dirty_iter_free(s->dbi); =20 + block_job_final_target_flush(&s->common, blk_bs(s->target), &ret); + if (need_drain) { s->in_drain =3D true; bdrv_drained_begin(bs); diff --git a/block/stream.c b/block/stream.c index 1fa742b0db..cda41e4a64 100644 --- a/block/stream.c +++ b/block/stream.c @@ -182,6 +182,8 @@ static int coroutine_fn stream_run(Job *job, Error **er= rp) } } =20 + block_job_final_target_flush(&s->common, s->target_bs, &error); + /* Do not remove the backing file if an error was there but ignored. */ return error; } diff --git a/blockjob.c b/blockjob.c index f2feff051d..e226bfbbfb 100644 --- a/blockjob.c +++ b/blockjob.c @@ -525,3 +525,19 @@ BlockErrorAction block_job_error_action(BlockJob *job,= BlockdevOnError on_err, } return action; } + +void block_job_final_target_flush(BlockJob *job, BlockDriverState *target_= bs, + int *ret) +{ + int flush_ret =3D bdrv_flush(target_bs); + + if (flush_ret < 0 && !block_job_is_internal(job)) { + qapi_event_send_block_job_error(job->job.id, + IO_OPERATION_TYPE_WRITE, + BLOCK_ERROR_ACTION_REPORT); + } + + if (*ret =3D=3D 0) { + *ret =3D flush_ret; + } +} --=20 2.29.2 From nobody Fri May 17 13:29:24 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; arc=pass (i=1dmarc=pass fromdomain=virtuozzo.com); dmarc=pass(p=quarantine dis=none) header.from=virtuozzo.com ARC-Seal: i=2; a=rsa-sha256; t=1614969397; cv=pass; d=zohomail.com; s=zohoarc; b=iCsTB97sM+d8MdZ6sTc5QK6xHWIHig91q0WkVuN0VtvMvqETUBUomHuhOwaEWGPKOEAT86e8nYK7obN5o6Grz9K2SAG7annaiDUwNbc2UH9OMBXy+S03DIrMlEYKw/p5jWOcUXLLTNDnn9jOv1nZIb4zaa74/PKYOXY5y/vzlWc= ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1614969397; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=iXZs9GFBvUAXNvJ2vlQnOgXar2kFLPug2NV9BHHGUJM=; b=B0uZDujGEBa2cWa9zJ9C3AhgiveDjzt/jbjiGM5pK3X6jhqHJw2TJBLVGgjGC0BtOTahbg+0+KC2Cbd/tDPik/Az6lm5dQeEBbA/3UNH3r+uv+5EMPvo1pRo94kM/pwI82HSShmOBjDzLqrRO9FJnkUYb1mxBUpTJFFCmvUYM6I= ARC-Authentication-Results: i=2; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; arc=pass (i=1dmarc=pass fromdomain=virtuozzo.com); dmarc=pass header.from= (p=quarantine dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1614969397236463.3842227144637; Fri, 5 Mar 2021 10:36:37 -0800 (PST) Received: from localhost ([::1]:57322 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lIFJU-00028B-3k for importer@patchew.org; Fri, 05 Mar 2021 13:36:36 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:42778) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lIEMW-0004my-1M; Fri, 05 Mar 2021 12:35:40 -0500 Received: from mail-eopbgr30124.outbound.protection.outlook.com ([40.107.3.124]:32132 helo=EUR03-AM5-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lIEMU-0005C4-0t; Fri, 05 Mar 2021 12:35:39 -0500 Received: from AM7PR08MB5494.eurprd08.prod.outlook.com (2603:10a6:20b:dc::15) by AM6PR08MB4408.eurprd08.prod.outlook.com (2603:10a6:20b:be::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3890.19; Fri, 5 Mar 2021 17:35:26 +0000 Received: from AM7PR08MB5494.eurprd08.prod.outlook.com ([fe80::f1f0:6610:11f5:5e4a]) by AM7PR08MB5494.eurprd08.prod.outlook.com ([fe80::f1f0:6610:11f5:5e4a%8]) with mapi id 15.20.3912.024; Fri, 5 Mar 2021 17:35:26 +0000 Received: from localhost.localdomain (185.215.60.202) by HE1PR02CA0119.eurprd02.prod.outlook.com (2603:10a6:7:29::48) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3912.17 via Frontend Transport; Fri, 5 Mar 2021 17:35:24 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=BOiNeRriohHXg8wlftsSnnPWfMXP03xx3mpeyMCs6kird8t6NBZ5YJsYXqOecIHhmeHxrnwuTB8o3cQI1MQB7LH5SiRineIRIivWfPKaX2tl4u7r9xKoiiBimzuMPvrLyCyh4+lVJDa82ocLxCy2iHqq3szpvPK6NPCyU0LDWIqEKfwqf8zFYuNJSaDpJYMwrzSzeSnXIjSDHsf7U6VQZlmnKGyIXG+Tqu2EoyGKpmD0lAxaRGe6GApEJZnBg3fH0ysAYnzTpTXr4+BbyoHGk0CrVKD1B8LIX/nW2ee1sQxXlo2gVUMOkagHkD83bz/ythkG9AWLtltIC3ZCBHXi1A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=iXZs9GFBvUAXNvJ2vlQnOgXar2kFLPug2NV9BHHGUJM=; b=Xk6pZYc52P6Mr8Pr0WR5sDTco15/WDONDFAGTREV4X5VSbyFeWE7ZdT0wIv6a5d56q7h4569OJ34jSXa54XblQMbq40S1aOt5jrJQ17GzraFSma8Om91/4gpI0GxZ67/45+R3NpcUk/WYI5A6KPgMsKpq9DYxHhvsWIKqDuSxMdsROvq5e4Klzk0bBd8c7ZcxzVtcKV9RGLD5tMPnBHJ7c1wHy4sBes3wBy1rJjCjmIpATiHrMtXDub+8rQawTTTYO8RJ+PSsyFeb5gBi+LCFJ8Mxr8ubMum1CZTWKGKmWCYwoR3MqD3jNmJWs53DLhYupQVaEj3EES3p6UKmxeykQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=virtuozzo.com; dmarc=pass action=none header.from=virtuozzo.com; dkim=pass header.d=virtuozzo.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=virtuozzo.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=iXZs9GFBvUAXNvJ2vlQnOgXar2kFLPug2NV9BHHGUJM=; b=t0vYq4dlB6otliWusoBSiLhMX4LoGLUMXN3mmc4iX29n6mRVUrI0HlNB4EZfrZQ4kOGL4/C2yml5YXNO148Vqh7QF5eE2gQjYMMTOOvZQ+qSyI5bRS61/qvV6DjWXBlZ9p1kfaGEWLWwE5nd/XYGou5j1WI+bUSXMxI6L0HQGHc= Authentication-Results: nongnu.org; dkim=none (message not signed) header.d=none;nongnu.org; dmarc=none action=none header.from=virtuozzo.com; From: Vladimir Sementsov-Ogievskiy To: qemu-block@nongnu.org Cc: qemu-devel@nongnu.org, crosa@redhat.com, ehabkost@redhat.com, mreitz@redhat.com, kwolf@redhat.com, vsementsov@virtuozzo.com, jsnow@redhat.com Subject: [PATCH v3 2/6] iotests: add qcow2-discard-during-rewrite Date: Fri, 5 Mar 2021 20:35:03 +0300 Message-Id: <20210305173507.393137-3-vsementsov@virtuozzo.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210305173507.393137-1-vsementsov@virtuozzo.com> References: <20210305173507.393137-1-vsementsov@virtuozzo.com> Content-Transfer-Encoding: quoted-printable X-Originating-IP: [185.215.60.202] X-ClientProxiedBy: HE1PR02CA0119.eurprd02.prod.outlook.com (2603:10a6:7:29::48) To AM7PR08MB5494.eurprd08.prod.outlook.com (2603:10a6:20b:dc::15) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: eb09c3f9-4313-4d06-6278-08d8dffd0fd8 X-MS-TrafficTypeDiagnostic: AM6PR08MB4408: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:3826; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: zrFI6hczVfJQ472FMlivgjBHm/3mNc89moELM9Cn9oy3zCkHeMSBJnyUp5M9AUfEgGM0UUu//5wzt6QYPsvhK9NTa3OKHv4Q2wBB3NxgXG80+PaiveTfnbXGUGbCmpw79JafYER0xSKtzhrTHv0QRZhO5G2CzeY8L8xPUhc337M9XKDOdFHD0uyQAS3u/M/L5UP03Pn+kyfHn+aBApTC2PRxOcXnMYk1iVgvAleNqmqgFljGmLjIy6fmsEeTFjDPWjtZo/d6XFbP0lEX4gMLrU7fIv4PGL1QV7yUYNJ/7czsJsLh25ELnjOVPbjMhNg3N34RcCevYplRu0kmegVJvtMJ0WaQyCv4ZI9HZ51qEhpwGlWZCCxCARxot+klEVOLV4ybliu7I7RDD0DHP2P1ghXHuh3dma9Z/VO+d6/gQa9KlcDDEyOAwjmRIGyd2TND2Mntlg9ams113oF4kJCBnpht+m1X9tw5xo2AhQJQr9V4MlFd7/vrEHCIktTTFtt1S8JIZ+18PUJ9M7E+IYy9jLpErC0avgRm2T+gLG62D7+hIyPzfcFD46uDNxe6/fYoZgIZZwsibCauPIUKy2eSVg0oQlxS2Xqrdi37zeNhcyhZsRoog/dfXljSj8MmiZtUU6lDWCKzkkmWmMYfZdVHXawKtDO5taGWXajBuy56wzCWo371wrj+ibdj600htERRXwxPogB6pKX7QI793bm6JQ== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:AM7PR08MB5494.eurprd08.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(136003)(366004)(39850400004)(396003)(376002)(346002)(16526019)(66476007)(478600001)(186003)(8936002)(8676002)(6916009)(1076003)(956004)(6486002)(83380400001)(69590400012)(52116002)(26005)(6666004)(6506007)(2906002)(2616005)(4326008)(66556008)(316002)(6512007)(36756003)(66946007)(86362001)(5660300002)(2004002)(21314003); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData: =?us-ascii?Q?49nsfyEgaMwBsFKsz9zfV7gXo0PrYEM+2FdjUv5mleWq36KRrrycyfVKYZd9?= =?us-ascii?Q?BKgs1Xbn2IgxdSDxin/pc1iSwUPVyqjyUO5LQTjjzQVYSKx6bWZKjRo0D7ne?= =?us-ascii?Q?7TDsZ2GiTPhuQjljsC+C2fbYHw8B8u3eH6egpbk5Xj4F1eEG1AjQkvIaELj/?= =?us-ascii?Q?vdeB5lCZT31MnYQQQojsbq/fRGfyaXSeYdhxX1M2rXUOTB4eurwamegX4UMV?= =?us-ascii?Q?s5RvzOB5VECRrP4G5imYugJycyVsEgXClUFFJb71IjSZEj6vLiiQ+M2FuJHr?= =?us-ascii?Q?Cd/qHwXjgwRjQGmbqNK+yZRh6DLR4f9S7K/dnOOCHkTsxcPpN9ibLVWKCndO?= =?us-ascii?Q?im7inexKkg7qL9aF7lbigFuBHUTW0otRawzow6kuXvgmmtRVWcsfKfiNBIym?= =?us-ascii?Q?eaR4j6h2IP+0ygQ+hAWjP2e8gl2NIE4NqnaH2hfAFpDk4SWlbtYvRPEHIfmn?= =?us-ascii?Q?oEZxtbM75r3zXDivGeHYiRfoZ57MrCpnLvYXUuCeo5Breav2jhrmMWlPcqsn?= =?us-ascii?Q?nrux7reCIuhBWLfDEi8+oWWOwXGKDqTKhoLyQD4DNZ8g3W2x9lKDyNCzwh/Z?= =?us-ascii?Q?YaSbiVxgwpLuQbkQ5TTTT0Vsi3q13VM+/rwTgkPGP17dDUb5LWoJ74RLjpo1?= =?us-ascii?Q?yyF1tbXGbTk5beLzzlw7pjfKIJ0LN2pDcdXzW6w8zjTInEKOrJMAyvt0tRsS?= =?us-ascii?Q?esQS44hxrI74qG2QRonpCc73XajM1bIot1N7g0mabIJ1Y2SHp9O9K4QePUR0?= =?us-ascii?Q?R9TwjVmizsvV8zQly3gHKSBXLcvks0sxhCGAim3tGBs/NcN43xTKZ6plJpaH?= =?us-ascii?Q?whjRJwmanOddCcWbeDDSaJjMFFUxj9Qh7ROH8MuS9Kny/T4s8h4erkvVgDcJ?= =?us-ascii?Q?uQBVEpvxegzV/hbdXgepOBvOKmYyH55D5PAaF1KXfsyFjjhfuaGPsInnvdAm?= =?us-ascii?Q?4FOS+SiLrkacsaE10oNHjs3WYkfvrtoQckKLfezpzTnQykv/mbC7Unc8OopU?= =?us-ascii?Q?krUrSZKHlSYMFoGYKVSZ7IJlPQaOg9dV3pymKsULRg3wA94+KIhisuwwy0js?= =?us-ascii?Q?YjgtKQ6Rggp9+ACNixftKUvnQUh2zqg8k6uCSGKlE3jdA/sbNK0R6BAFNLsB?= =?us-ascii?Q?wS6WWTUnHSaaY0xD1M7fuXziYUV+XdnX8e3b02YPABo8jw71TQz1FY6EOXtL?= =?us-ascii?Q?oPNQYydbxYXo+0BZ0QgPy0Lg9TJW0EXK7lNxwPeUOOjrbkNQMI6c5ISYdKCN?= =?us-ascii?Q?OaB8tFgKcnjwkx31m41wWbN045eyolXpeGYWAR5nFuxkd/T4wC2KhYRby1Ws?= =?us-ascii?Q?uzEkaKDKQ1O6qqjK7quKW1Vh?= X-OriginatorOrg: virtuozzo.com X-MS-Exchange-CrossTenant-Network-Message-Id: eb09c3f9-4313-4d06-6278-08d8dffd0fd8 X-MS-Exchange-CrossTenant-AuthSource: AM7PR08MB5494.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Mar 2021 17:35:26.0057 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 0bc7f26d-0264-416e-a6fc-8352af79c58f X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 4+XEAk/JEV+hHh8VNCINx1s+d4LWYKs/1k48rcULCRfE0NDpe2I3rp456kmRv1C1sIrJ+QnXpSyVRKPaG6CRE24zg9zf36EUXHZn4uQxDbI= X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM6PR08MB4408 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=40.107.3.124; envelope-from=vsementsov@virtuozzo.com; helo=EUR03-AM5-obe.outbound.protection.outlook.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, MSGID_FROM_MTA_HEADER=0.001, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @virtuozzo.com) Content-Type: text/plain; charset="utf-8" Simple test: - start writing to allocated cluster A - discard this cluster - write to another unallocated cluster B (it's allocated in same place where A was allocated) - continue writing to A For now last action pollutes cluster B which is a bug fixed by the following commit. For now, add test to "disabled" group, so that it doesn't run automatically. Signed-off-by: Vladimir Sementsov-Ogievskiy --- .../tests/qcow2-discard-during-rewrite | 72 +++++++++++++++++++ .../tests/qcow2-discard-during-rewrite.out | 21 ++++++ 2 files changed, 93 insertions(+) create mode 100755 tests/qemu-iotests/tests/qcow2-discard-during-rewrite create mode 100644 tests/qemu-iotests/tests/qcow2-discard-during-rewrite.o= ut diff --git a/tests/qemu-iotests/tests/qcow2-discard-during-rewrite b/tests/= qemu-iotests/tests/qcow2-discard-during-rewrite new file mode 100755 index 0000000000..7f0d8a107a --- /dev/null +++ b/tests/qemu-iotests/tests/qcow2-discard-during-rewrite @@ -0,0 +1,72 @@ +#!/usr/bin/env bash +# group: quick disabled +# +# Test discarding (and reusing) host cluster during writing data to it. +# +# Copyright (c) 2021 Virtuozzo International GmbH. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . +# + +# creator +owner=3Dvsementsov@virtuozzo.com + +seq=3D`basename $0` +echo "QA output created by $seq" + +status=3D1 # failure is the default! + +_cleanup() +{ + _cleanup_test_img +} +trap "_cleanup; exit \$status" 0 1 2 3 15 + +# get standard environment, filters and checks +. ./../common.rc +. ./../common.filter + +_supported_fmt qcow2 +_supported_proto file fuse +_supported_os Linux + +size=3D1M +_make_test_img $size + +( +cat < (p=quarantine dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1614969597610691.6358401527242; Fri, 5 Mar 2021 10:39:57 -0800 (PST) Received: from localhost ([::1]:37498 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lIFMi-0005ax-A9 for importer@patchew.org; Fri, 05 Mar 2021 13:39:56 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:42804) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lIEMZ-0004rV-86; Fri, 05 Mar 2021 12:35:43 -0500 Received: from mail-eopbgr30124.outbound.protection.outlook.com ([40.107.3.124]:32132 helo=EUR03-AM5-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lIEMW-0005C4-Ll; Fri, 05 Mar 2021 12:35:42 -0500 Received: from AM7PR08MB5494.eurprd08.prod.outlook.com (2603:10a6:20b:dc::15) by AM6PR08MB4408.eurprd08.prod.outlook.com (2603:10a6:20b:be::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3890.19; Fri, 5 Mar 2021 17:35:27 +0000 Received: from AM7PR08MB5494.eurprd08.prod.outlook.com ([fe80::f1f0:6610:11f5:5e4a]) by AM7PR08MB5494.eurprd08.prod.outlook.com ([fe80::f1f0:6610:11f5:5e4a%8]) with mapi id 15.20.3912.024; Fri, 5 Mar 2021 17:35:27 +0000 Received: from localhost.localdomain (185.215.60.202) by HE1PR02CA0119.eurprd02.prod.outlook.com (2603:10a6:7:29::48) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3912.17 via Frontend Transport; Fri, 5 Mar 2021 17:35:26 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=kFlAWi6+qMQyZPP4C/wrajV0kMwdQk2klpWUTledoMuiEU1My9EDcAgnuyxG0KoSA+6wDydz55Q1IBcLEWfCqcVPdbsOfr+r7wpB+l8+3vUJRW5UcBUJ6dt7yopmWHHsAd7Mp3hUiSxXZLDGhytJaeAFZXNKuvmKfSd62iEyeTQuyJ9cxcHPYi5l7J926aq3nS0LIJeZOnyL1Ul/+5IU7qmbGb/ObuSmYEBOkXmjahozQmLf0oG+kPydEeSsrnBOgd85mOLARBG4kIfTZ+bJSPSgOEzWeLF78eaLMOyBOQ+50B2RXAHavQ/I21F1nZOGsZFICiRrzrFgcK3GwmmiGg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=rlYMDCEtFyErQ51N92hVFCzADbE3xM2AY8XpqIQu35g=; b=UIx5bwjur00+b1fgDJA0JZI1pbGUj7BYmXGihgY944Iv2MNRxhu9GdhvwVhX0ED43WIOItUURjezPLq1RWNouIC71IcMr8Si73s1xAtbuOl6PBa/PbXv+kTBit8iLCe48sxY6N14gDsmyg6MG4FnnWVgd9UgSbQN+nHqPvDPqCVXjR3gLhh6iZpTYgBGMq5PM9z0wjOl6y1VMrqTMBxwR2vgklVKyzEi4cUsv+FBUsUgGiakC5uKcu5qNJETpS8v3rnR4G74BtKm7R/bo2bQ9HeVJLFC81IgJZivFVhmkIplzC2UOwfmdQufwt9Rt5xYl38XC23J6yvJVKMf83yN5A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=virtuozzo.com; dmarc=pass action=none header.from=virtuozzo.com; dkim=pass header.d=virtuozzo.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=virtuozzo.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=rlYMDCEtFyErQ51N92hVFCzADbE3xM2AY8XpqIQu35g=; b=XKsiwTnRQ/2B5EKKj0t+VSCP6LyN5bIClvrfOLGF2rcEpGbUf6/o62o31B+kkG7blrrSwm+3mVQ8Bs0ghmWyoxiQ4+Cp5rxo9ObUfL0CNGNlF+JZU91dFvAsqnHJZqINPqEfnkEDFp41SOir4MQUTKhXxhgk2xsR4mB0KtB2srg= Authentication-Results: nongnu.org; dkim=none (message not signed) header.d=none;nongnu.org; dmarc=none action=none header.from=virtuozzo.com; From: Vladimir Sementsov-Ogievskiy To: qemu-block@nongnu.org Cc: qemu-devel@nongnu.org, crosa@redhat.com, ehabkost@redhat.com, mreitz@redhat.com, kwolf@redhat.com, vsementsov@virtuozzo.com, jsnow@redhat.com Subject: [PATCH v3 3/6] block/qcow2: introduce inflight writes counters: fix discard Date: Fri, 5 Mar 2021 20:35:04 +0300 Message-Id: <20210305173507.393137-4-vsementsov@virtuozzo.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210305173507.393137-1-vsementsov@virtuozzo.com> References: <20210305173507.393137-1-vsementsov@virtuozzo.com> Content-Transfer-Encoding: quoted-printable X-Originating-IP: [185.215.60.202] X-ClientProxiedBy: HE1PR02CA0119.eurprd02.prod.outlook.com (2603:10a6:7:29::48) To AM7PR08MB5494.eurprd08.prod.outlook.com (2603:10a6:20b:dc::15) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: cb1eeca8-41f8-4f14-4ffc-08d8dffd10a0 X-MS-TrafficTypeDiagnostic: AM6PR08MB4408: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:5516; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: rcIa60XTATVjwJPWcdGNFF6ioafiQ7/04V7Nqv9Ki5bGjR1RerXMkKPmR2Lri+tIWeWdn5NbbptSuM/gabGASkMIGVkO0BAo7BAUzDSM+qfGAzP55C2E8pzr01EdrDZBGXzxJ+UKnq5I9S2DIYuVGBu+sY/IhBkZc9b/Gi5CBDczuXGRTSgT6Gwbos3bqWHx19VQ/5TG8qCoXs6DOvloDfRHduOqAM85gMmhZ//qx1ZnRHZuKdFSilutjT9sCZRMwKWq6Z8AJ3Fw4frP+JNjrGp53KI7CjRlZ5BwoD59gzC1riEJnx+kiXYaUAso701gm7WA3ugwcFFMOot5/4q+wZwRIzwYJPw4VufIjvSXmsXC1v04OkPaJJfjSxdenLtxYJBnplMZKNTqPICt8jss7xAcwcC8nwVEQz0d2md8p4JmWgRqPRF88fgcT0kdE64Yfjb2S25Sg5/kZ0j/dZSCG+uZMUcul6G/flOscf9hCyC1r+1vtSPVejDrvcyxu1ZW0IBfEwJr8U7k4KOR1mS6gTsLQ5gfyraivGNslV5ynDNx+xvjhysvusJUXJ4E1gsOOnl11DmucMrr7/R3xzAPX666t5qVkcKhfqFD4PnVHfEX5CE8Ey2lldnlJrncvObr X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:AM7PR08MB5494.eurprd08.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(136003)(366004)(39850400004)(396003)(376002)(346002)(16526019)(66476007)(478600001)(186003)(8936002)(8676002)(6916009)(1076003)(956004)(6486002)(83380400001)(19627235002)(69590400012)(52116002)(26005)(6666004)(6506007)(2906002)(2616005)(4326008)(66556008)(316002)(6512007)(36756003)(66946007)(86362001)(5660300002)(21314003)(309714004); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData: =?us-ascii?Q?JcB6kDMbQrAZtJkUNNVCrV11ub6bF170wu2PMjwwnKUZ5ASZeKuvuca66yLd?= =?us-ascii?Q?6FgNAUj9njXsJHnZUyqJXNjrbBKcOLQboJP8VAL+IZW3n71ss8p+lwAjGx2E?= =?us-ascii?Q?SKAmaq1eqrCw8A14zZZrFfcIxLCoYPyq2YAlnyuuMmOGiPNKOIbwlNc6NqL1?= =?us-ascii?Q?ayNenOE6dPv2pwhY3QX4gy78RpNVSWa/kObi6/meIJ3WBaGFNoOuVgKB2Vn9?= =?us-ascii?Q?Zy3rUXBjhW8GtWUd9WksEwyU25OC5Qca6qEF+jGfhAnJvxLOQLLLpXwVtrvr?= =?us-ascii?Q?/0jBGg8+WSPNVKJUaD72mXQ9mQYY4Hqm/jLfEa9NmT1trssq1zBrQjwpUsG6?= =?us-ascii?Q?liOGIHo/Ut2TlwCDVefimSzTbwTRHzYhqZLZ3z0MAkKzXuCzKtSecKZN34kt?= =?us-ascii?Q?ddKO1TVUmuOplikingDj4knwOm2UJqOy2mfN+RavIxF+dYrn9rTU7qKi7pTs?= =?us-ascii?Q?wBCrNZmKOFO99qdNWJ9HR1sOBiGhQJiisb8jDfvAPIwyEZdlaZiK2gnpgGLR?= =?us-ascii?Q?2RekddL3vrFDgOfwQpItIl/4ZnLUhPCJ6hdb5s15xW81ec1YiqwbkF6b/x9d?= =?us-ascii?Q?5pFvfdQ+mNmxX55KO8a8Hp2jBCT/a3dKtlVQB/pdOETdEnJQonvXjrHmlKxc?= =?us-ascii?Q?N+rDgiWSJhSEZqfok+/AjT1XKU7geKPD9DIccnNrBqAwOyt0BBqpN9dBXfVE?= =?us-ascii?Q?SqM5SQa3rRKxc13anlLHqv/uDNkLtuZ21MEpSqi03xgz57LCUcA9J1z7XTdY?= =?us-ascii?Q?zT3oCwM8X6UpPvi7Q4OOhzpeUfaPisv7jJcEg9Gqz7SfZdpBrDaWRo0ok9iD?= =?us-ascii?Q?T5AEespOtmL/WEpoZ/fHXX6I5sLH/Ug+saVQKGCCjseMwEwzpw02egQTgu16?= =?us-ascii?Q?nbUxMzIMAMr8HrFmmEUWgrTT91rcF6QjC9/1XZuiSkq/nWHMMDGAbpXV/0v/?= =?us-ascii?Q?3rwVSCH32MezEXZb5qnAfH4OjZMUxP/xOcm2d55mAKDXXnC/NJUnecTCrHw1?= =?us-ascii?Q?mM5hzCHz6uhcz0hIEEgwGC1KU4qkfEDsAYh2fgcz25CpOWrv8aozyTemcTYU?= =?us-ascii?Q?oojMxNtfb/onKVJe61xgDmz/TBnayVpQpexGkcLeSCmqImrmCd04N1wRhRoB?= =?us-ascii?Q?6495Obekzv9yKpQPAXZzRzacwjP9OCFYR4CMYzjxZ+PNT0drNW135rLn5YaU?= =?us-ascii?Q?LFdiQxilyZQwzVk8ytFFSxhMHqJXgE5ttu4VkVnEt50ONQb+yQivF3AHHp73?= =?us-ascii?Q?FYFr6OAr8UvYgMVp/DbxKTxAcicWcoo08zsXOz+7NwrWnCoe7dG1PVatcB1e?= =?us-ascii?Q?6ehjn2cjBLDKkVP1Z4JLkz2/?= X-OriginatorOrg: virtuozzo.com X-MS-Exchange-CrossTenant-Network-Message-Id: cb1eeca8-41f8-4f14-4ffc-08d8dffd10a0 X-MS-Exchange-CrossTenant-AuthSource: AM7PR08MB5494.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Mar 2021 17:35:27.3896 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 0bc7f26d-0264-416e-a6fc-8352af79c58f X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: axmasRP71YQJ4oiX3vFzrJLtAlg00MkjP+tX/h3yMFjvEUsAioKsiKjdsXq5wK8S9jhQdDoHmCjAFNBB4HKQ9X4HLdjnW9WjN1vx39ED9NA= X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM6PR08MB4408 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=40.107.3.124; envelope-from=vsementsov@virtuozzo.com; helo=EUR03-AM5-obe.outbound.protection.outlook.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, MSGID_FROM_MTA_HEADER=0.001, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @virtuozzo.com) Content-Type: text/plain; charset="utf-8" There is a bug in qcow2: host cluster can be discarded (refcount becomes 0) and reused during data write. In this case data write may pollute another cluster (recently allocated) or even metadata. To fix the issue let's track inflight writes to host cluster in the hash table and consider new counter when discarding and reusing host clusters. Enable qcow2-discard-during-rewrite as it is fixed. Signed-off-by: Vladimir Sementsov-Ogievskiy --- block/qcow2.h | 9 ++ block/qcow2-refcount.c | 149 +++++++++++++++++- block/qcow2.c | 26 ++- .../tests/qcow2-discard-during-rewrite | 2 +- 4 files changed, 181 insertions(+), 5 deletions(-) diff --git a/block/qcow2.h b/block/qcow2.h index 0678073b74..e18d8dfbae 100644 --- a/block/qcow2.h +++ b/block/qcow2.h @@ -420,6 +420,8 @@ typedef struct BDRVQcow2State { * is to convert the image with the desired compression type set. */ Qcow2CompressionType compression_type; + + GHashTable *inflight_writes_counters; } BDRVQcow2State; =20 typedef struct Qcow2COWRegion { @@ -896,6 +898,13 @@ int qcow2_shrink_reftable(BlockDriverState *bs); int64_t qcow2_get_last_cluster(BlockDriverState *bs, int64_t size); int qcow2_detect_metadata_preallocation(BlockDriverState *bs); =20 +int qcow2_inflight_writes_inc(BlockDriverState *bs, int64_t offset, + int64_t length); +int qcow2_inflight_writes_dec(BlockDriverState *bs, int64_t offset, + int64_t length); +int qcow2_inflight_writes_dec_locked(BlockDriverState *bs, int64_t offset, + int64_t length); + /* qcow2-cluster.c functions */ int qcow2_grow_l1_table(BlockDriverState *bs, uint64_t min_size, bool exact_size); diff --git a/block/qcow2-refcount.c b/block/qcow2-refcount.c index 8e649b008e..464d133368 100644 --- a/block/qcow2-refcount.c +++ b/block/qcow2-refcount.c @@ -799,6 +799,140 @@ found: } } =20 +/* + * Qcow2InFlightRefcount is a type for values of s->inflight_writes_counte= rs + * hasm map. And it's keys are cluster indices. + */ +typedef struct Qcow2InFlightRefcount { + /* + * Number of in-flight writes to the cluster, always > 0, as when beco= mes + * 0 the entry is removed from s->inflight_writes_counters. + */ + uint64_t inflight_writes_cnt; + + /* Cluster refcount is known to be zero */ + bool refcount_zero; + + /* Cluster refcount was made zero with this discard-type */ + enum qcow2_discard_type type; +} Qcow2InFlightRefcount; + +static Qcow2InFlightRefcount *find_infl_wr(BDRVQcow2State *s, + int64_t cluster_index) +{ + Qcow2InFlightRefcount *infl; + + if (!s->inflight_writes_counters) { + return NULL; + } + + infl =3D g_hash_table_lookup(s->inflight_writes_counters, &cluster_ind= ex); + + if (infl) { + assert(infl->inflight_writes_cnt > 0); + } + + return infl; +} + +/* + * Returns true if there are any in-flight writes to the cluster blocking + * its reallocation. + */ +static bool has_infl_wr(BDRVQcow2State *s, int64_t cluster_index) +{ + return !!find_infl_wr(s, cluster_index); +} + +static int update_inflight_write_cnt(BlockDriverState *bs, + int64_t offset, int64_t length, + bool decrease, bool locked) +{ + BDRVQcow2State *s =3D bs->opaque; + int64_t start, last, cluster_offset; + + if (locked) { + qemu_co_mutex_assert_locked(&s->lock); + } + + start =3D start_of_cluster(s, offset); + last =3D start_of_cluster(s, offset + length - 1); + for (cluster_offset =3D start; cluster_offset <=3D last; + cluster_offset +=3D s->cluster_size) + { + int64_t cluster_index =3D cluster_offset >> s->cluster_bits; + Qcow2InFlightRefcount *infl =3D find_infl_wr(s, cluster_index); + + if (!infl) { + infl =3D g_new0(Qcow2InFlightRefcount, 1); + g_hash_table_insert(s->inflight_writes_counters, + g_memdup(&cluster_index, sizeof(cluster_in= dex)), + infl); + } + + if (decrease) { + assert(infl->inflight_writes_cnt >=3D 1); + infl->inflight_writes_cnt--; + } else { + infl->inflight_writes_cnt++; + } + + if (infl->inflight_writes_cnt =3D=3D 0) { + bool refcount_zero =3D infl->refcount_zero; + enum qcow2_discard_type type =3D infl->type; + + g_hash_table_remove(s->inflight_writes_counters, &cluster_inde= x); + + if (refcount_zero) { + /* + * Slow path. We must reset normal refcount to actually re= lease + * the cluster. + */ + int ret; + + if (!locked) { + qemu_co_mutex_lock(&s->lock); + } + ret =3D qcow2_update_cluster_refcount(bs, cluster_index, 0, + true, type); + if (!locked) { + qemu_co_mutex_unlock(&s->lock); + } + + if (ret < 0) { + return ret; + } + } + } + + } + + return 0; +} + +int qcow2_inflight_writes_inc(BlockDriverState *bs, int64_t offset, + int64_t length) +{ + return update_inflight_write_cnt(bs, offset, length, false, false); +} + +/* + * Called with s->lock not locked by caller. Will take s->lock only if nee= d to + * release the cluster (refcount is 0 and inflight-write-cnt becomes zero). + */ +int qcow2_inflight_writes_dec(BlockDriverState *bs, int64_t offset, + int64_t length) +{ + return update_inflight_write_cnt(bs, offset, length, true, false); +} + +/* Called with s->lock locked. */ +int qcow2_inflight_writes_dec_locked(BlockDriverState *bs, int64_t offset, + int64_t length) +{ + return update_inflight_write_cnt(bs, offset, length, true, true); +} + /* XXX: cache several refcount block clusters ? */ /* @addend is the absolute value of the addend; if @decrease is set, @adde= nd * will be subtracted from the current refcount, otherwise it will be adde= d */ @@ -885,6 +1019,13 @@ static int QEMU_WARN_UNUSED_RESULT update_refcount(Bl= ockDriverState *bs, =20 if (refcount =3D=3D 0) { void *table; + Qcow2InFlightRefcount *infl =3D find_infl_wr(s, cluster_index); + + if (infl) { + infl->refcount_zero =3D true; + infl->type =3D type; + continue; + } =20 table =3D qcow2_cache_is_table_offset(s->refcount_block_cache, offset); @@ -983,7 +1124,7 @@ retry: =20 if (ret < 0) { return ret; - } else if (refcount !=3D 0) { + } else if (refcount !=3D 0 || has_infl_wr(s, next_cluster_index)) { goto retry; } } @@ -1046,7 +1187,7 @@ int64_t qcow2_alloc_clusters_at(BlockDriverState *bs,= uint64_t offset, ret =3D qcow2_get_refcount(bs, cluster_index++, &refcount); if (ret < 0) { return ret; - } else if (refcount !=3D 0) { + } else if (refcount !=3D 0 || has_infl_wr(s, cluster_index)) { break; } } @@ -2294,7 +2435,9 @@ static int64_t alloc_clusters_imrt(BlockDriverState *= bs, contiguous_free_clusters < cluster_count; cluster++) { - if (!s->get_refcount(*refcount_table, cluster)) { + if (!s->get_refcount(*refcount_table, cluster) && + !has_infl_wr(s, cluster)) + { contiguous_free_clusters++; if (first_gap) { /* If this is the first free cluster found, update diff --git a/block/qcow2.c b/block/qcow2.c index d9f49a52e7..6ee94421e0 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -1831,6 +1831,8 @@ static int coroutine_fn qcow2_do_open(BlockDriverStat= e *bs, QDict *options, #endif =20 qemu_co_queue_init(&s->thread_task_queue); + s->inflight_writes_counters =3D + g_hash_table_new_full(g_int64_hash, g_int64_equal, g_free, g_free); =20 return ret; =20 @@ -2546,6 +2548,9 @@ out_unlocked: =20 out_locked: qcow2_handle_l2meta(bs, &l2meta, false); + + qcow2_inflight_writes_dec_locked(bs, host_offset, bytes); + qemu_co_mutex_unlock(&s->lock); =20 qemu_vfree(crypt_buf); @@ -2605,6 +2610,8 @@ static coroutine_fn int qcow2_co_pwritev_part( goto out_locked; } =20 + qcow2_inflight_writes_inc(bs, host_offset, cur_bytes); + qemu_co_mutex_unlock(&s->lock); =20 if (!aio && cur_bytes !=3D bytes) { @@ -2707,6 +2714,9 @@ static void qcow2_close(BlockDriverState *bs) g_free(s->image_backing_file); g_free(s->image_backing_format); =20 + assert(g_hash_table_size(s->inflight_writes_counters) =3D=3D 0); + g_hash_table_unref(s->inflight_writes_counters); + if (has_data_file(bs)) { bdrv_unref_child(bs, s->data_file); s->data_file =3D NULL; @@ -4097,10 +4107,17 @@ qcow2_co_copy_range_to(BlockDriverState *bs, goto fail; } =20 + qcow2_inflight_writes_inc(bs, host_offset, cur_bytes); + qemu_co_mutex_unlock(&s->lock); + ret =3D bdrv_co_copy_range_to(src, src_offset, s->data_file, host_= offset, cur_bytes, read_flags, write_flags); + qemu_co_mutex_lock(&s->lock); + + qcow2_inflight_writes_dec_locked(bs, host_offset, cur_bytes); + if (ret < 0) { goto fail; } @@ -4536,13 +4553,20 @@ qcow2_co_pwritev_compressed_task(BlockDriverState *= bs, } =20 ret =3D qcow2_pre_write_overlap_check(bs, 0, cluster_offset, out_len, = true); - qemu_co_mutex_unlock(&s->lock); if (ret < 0) { + qemu_co_mutex_unlock(&s->lock); goto fail; } =20 + qcow2_inflight_writes_inc(bs, cluster_offset, out_len); + + qemu_co_mutex_unlock(&s->lock); + BLKDBG_EVENT(s->data_file, BLKDBG_WRITE_COMPRESSED); ret =3D bdrv_co_pwrite(s->data_file, cluster_offset, out_len, out_buf,= 0); + + qcow2_inflight_writes_dec(bs, cluster_offset, out_len); + if (ret < 0) { goto fail; } diff --git a/tests/qemu-iotests/tests/qcow2-discard-during-rewrite b/tests/= qemu-iotests/tests/qcow2-discard-during-rewrite index 7f0d8a107a..2e2e0d2cb0 100755 --- a/tests/qemu-iotests/tests/qcow2-discard-during-rewrite +++ b/tests/qemu-iotests/tests/qcow2-discard-during-rewrite @@ -1,5 +1,5 @@ #!/usr/bin/env bash -# group: quick disabled +# group: quick # # Test discarding (and reusing) host cluster during writing data to it. # --=20 2.29.2 From nobody Fri May 17 13:29:24 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; arc=pass (i=1dmarc=pass fromdomain=virtuozzo.com); dmarc=pass(p=quarantine dis=none) header.from=virtuozzo.com ARC-Seal: i=2; a=rsa-sha256; t=1614968496; cv=pass; d=zohomail.com; s=zohoarc; b=AKDwB1Bz2KATuKSaimjGUUcrpDva1TB6XbgJYhvpYE+StuTJUuFp2Q6kOqNuSuTvaYk1/nyF+tmoLZzufmn+tU3fTt02FvyOHRVePzefFdez4vzwWBrYuS36nQ/1daAMu4mL1/+kW++61mv5cbyrLU+l3qW4pxgkKhxtfhF5Vyo= ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1614968496; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=j0CAWZrvLtoFXm/TNktW68d0kmwrGlEe2Otw+N5QY34=; b=Qz/n4/mHQ+uRZDWF2hVYPAsV1OYc1k4bwnd1GMfRrdQOPh0Hc2DqpG/JacViyiyYjnWNLpNIs2HMTwBSY8qctVWZx8UmKm6JoXK9CbB2X1WEG+s1c70o8UQJjkQFybZaoNZ1OTAulYm8MQLGFECvsioWKoZErZnN1xDsKljqTwY= ARC-Authentication-Results: i=2; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; arc=pass (i=1dmarc=pass fromdomain=virtuozzo.com); dmarc=pass header.from= (p=quarantine dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1614968495998278.310245484946; Fri, 5 Mar 2021 10:21:35 -0800 (PST) Received: from localhost ([::1]:35744 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lIF4w-0006hP-S2 for importer@patchew.org; Fri, 05 Mar 2021 13:21:34 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:42822) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lIEMc-0004wL-Qx; Fri, 05 Mar 2021 12:35:47 -0500 Received: from mail-eopbgr30124.outbound.protection.outlook.com ([40.107.3.124]:32132 helo=EUR03-AM5-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lIEMZ-0005C4-SQ; Fri, 05 Mar 2021 12:35:46 -0500 Received: from AM7PR08MB5494.eurprd08.prod.outlook.com (2603:10a6:20b:dc::15) by AM6PR08MB4408.eurprd08.prod.outlook.com (2603:10a6:20b:be::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3890.19; Fri, 5 Mar 2021 17:35:28 +0000 Received: from AM7PR08MB5494.eurprd08.prod.outlook.com ([fe80::f1f0:6610:11f5:5e4a]) by AM7PR08MB5494.eurprd08.prod.outlook.com ([fe80::f1f0:6610:11f5:5e4a%8]) with mapi id 15.20.3912.024; Fri, 5 Mar 2021 17:35:28 +0000 Received: from localhost.localdomain (185.215.60.202) by HE1PR02CA0119.eurprd02.prod.outlook.com (2603:10a6:7:29::48) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3912.17 via Frontend Transport; Fri, 5 Mar 2021 17:35:27 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=mTq2atMXVWOrLFIpZHz4oKJGEdHD3MgSJ3zoo+rDe8xwbKF3bDAwgE23RAeb+oqrisMeLjMdTh2m+akthL7H7pV4W9MSHOlAYsZz7gU+KOrF2UHd/OZahzGrk4hqJ0FHXxCOfXLTyFDiHN4eRS/Z6rfZMlaqVwtcJj5/pF/zMdwSdzlZoFYcxPO6o2Ce/4WleUuSLHImzI8qZ2JE6PiFOTbHyS4wnQjeRPHRPvm9/o1Edg0bsSLYpZxlaceW5FYiTj2nhq+Mz6fYyRuP5psvuxeEWVG1dXKUhZ5Hv1/d3bItRTJHOcacmVzglKuKm3p1n2PhTnsYRvHqFqlJw/AW8A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=j0CAWZrvLtoFXm/TNktW68d0kmwrGlEe2Otw+N5QY34=; b=nv36VpWOuU5P9ddBezL7RMp09B5QvZo//CGEmEt8hcF1g9p80QN2EEyAMMfPvcezTVvdInbbOFNpHS8QnZUEQaxUfAoTJ5P1p4y6sBIpUbCnze9kB5bScoI+bBNTbEzCzQeXLtms/TAGy41M9fu8DYqs5RTcN2t8//Yhmp6bPv99z1QX+aP+zQ85ZImWebIi9lLgjiEuDP5E324wiHe2CKyBZwwdJiJkJhvPOisLwk+kLOGrOlKRZHIRw9bVju+6D62OAdbObSESKPJYKCSo9V1Q3SbKgwHNA2AxVTyG7drw8xGcL9I8NpDHErBFNjn2BEp6YtGIzFh81mhMuaDI4g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=virtuozzo.com; dmarc=pass action=none header.from=virtuozzo.com; dkim=pass header.d=virtuozzo.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=virtuozzo.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=j0CAWZrvLtoFXm/TNktW68d0kmwrGlEe2Otw+N5QY34=; b=p+vSwm1vZoUt7jPyn5L6mY+1KnVSGb2zlifP0pYAqeINB+YZeW4dJz38Xlt9jqFLTW5+tn7jMWO11rcL4k3xNruzWjOkFPHmN1lWM5q61hUyPmm3x9dJCd4943uX/I6TTUS0+rbQisA06JD1P+uJEiG1YOF2LcIfZffo78OqxpI= Authentication-Results: nongnu.org; dkim=none (message not signed) header.d=none;nongnu.org; dmarc=none action=none header.from=virtuozzo.com; From: Vladimir Sementsov-Ogievskiy To: qemu-block@nongnu.org Cc: qemu-devel@nongnu.org, crosa@redhat.com, ehabkost@redhat.com, mreitz@redhat.com, kwolf@redhat.com, vsementsov@virtuozzo.com, jsnow@redhat.com Subject: [PATCH v3 4/6] util: implement seqcache Date: Fri, 5 Mar 2021 20:35:05 +0300 Message-Id: <20210305173507.393137-5-vsementsov@virtuozzo.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210305173507.393137-1-vsementsov@virtuozzo.com> References: <20210305173507.393137-1-vsementsov@virtuozzo.com> Content-Transfer-Encoding: quoted-printable X-Originating-IP: [185.215.60.202] X-ClientProxiedBy: HE1PR02CA0119.eurprd02.prod.outlook.com (2603:10a6:7:29::48) To AM7PR08MB5494.eurprd08.prod.outlook.com (2603:10a6:20b:dc::15) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 28059b9c-e12b-4f59-6e70-08d8dffd1171 X-MS-TrafficTypeDiagnostic: AM6PR08MB4408: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:1332; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: FZzJiKpJLMeVN8XLkbBqF81HYiWHlPHBEgH69Zq0Eq+QmmKJfabqJ//t3l8qKsvYCcyALuWsTvQLlq2yYRh4GJjYTs360SWBsNJlyqdPiUePHgcZEte9aUqGpMK6nmLWKklxXqHk9GjNqmBkQ/9bqVuFUT+TrMOM8yscDLUz4LBItHDM1t8825gQU0WIhq5UPTUCvFyv2quoKOMD4vjhINFEgbXS5H9abRiAbnX/PUNPR2MuLu8Ic3mUHbNo0qmhTZeutNAwGJhxJSJV9Wq8AjmWev1XCwWE5sD3ncWIq/3ZElvvgoW2UGtkAzD6eDtyS63lp7An24hpeF2ggNDQl/q9V6Q0huT25V6qd12BofSQRDnahgZLfE6gprGhKJYS5mcFR06rArpE5vwhjV3zV4Zxm9NeIMEh6+2BqZ2qNdQJnRk5nxTY0YUIZtV8q2Nv59S4qx6JTN+osrxS4+cE/Vl+iinfoGpwSZIJD0hw5PtklQs4wH0io4GYBoKi3EmvLviExvSqq62LYS9wviP18+qzsvtWpWfFmR2G0Wk7IQ/xIC80RnVZKRxtc4fuP5QXREi4c/XGv/egZVEt2nED6g== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:AM7PR08MB5494.eurprd08.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(136003)(366004)(39850400004)(396003)(376002)(346002)(16526019)(66476007)(478600001)(186003)(8936002)(8676002)(6916009)(1076003)(956004)(6486002)(83380400001)(69590400012)(52116002)(26005)(6666004)(6506007)(2906002)(2616005)(4326008)(66556008)(316002)(6512007)(36756003)(66946007)(30864003)(86362001)(5660300002); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData: =?us-ascii?Q?nusAoJIRp2iwUt7oicbIxHpXqMorWvg/utvROlal5bWl7fBxSfBmmNpi1JEX?= =?us-ascii?Q?erG9yJyNlqdV9xFM52Z+h+2ZbfdkG32qlFCeU+Ci9goaNFBGFaMQaIXaRgWe?= =?us-ascii?Q?hLSKpRK6DXMjG8vm9jlOMEBYVx0S7IGXENfUVbV2yeIF1hFwVfSR8RXx2RIm?= =?us-ascii?Q?pp0bcVYapzPoX1bUYGUHeFB1SGSavPlDcHLRyZVHry7DvnCmpK2ADKJBekm8?= =?us-ascii?Q?Su0whY6jAfRv6MoXcrPKdONcH161tLGFyKAY2vxqGIpfe9d+JQaWuHDY1MiS?= =?us-ascii?Q?dnzZr0FARIhaj1+D+gnEF5/mhmQqvq7LNoB7H+gI56fH9TB3tmnG0uxB4j6V?= =?us-ascii?Q?OIXA5v7a+MDxEJ45tyx/ZzTR4vTIesRH1YSVBoMMWpiVOkvGIw/nmVVwyKUp?= =?us-ascii?Q?wxn0ZQ7RIXQAuQgrKZY7rlyyRob0qdPUgkjR/H3gLv4mcAai6tX9bS50cl3N?= =?us-ascii?Q?WJLhX5JZkowIFG3FJSwKEvPK2C7EhGoA0Cbk14UtTSLVs2E9GiRXrE5zNkjT?= =?us-ascii?Q?z/i1lJM8ERvpBB2YALoCh5W8VKdGpMKy88F9QSPbfXOfw6lNs79xzJUNtXzL?= =?us-ascii?Q?SuQoHwOAPXuc78YHZjBBnmd4dtEl+i+Q9G3j06KCz0seUfbbSwPoVXXUA6HO?= =?us-ascii?Q?ZwhK0e+vUZWn8O1ggRAcvoCi/BkFyoQRKXAhLigCGBYBnOfAHE9HlywBFZ99?= =?us-ascii?Q?OS/1MDy2GGk9AXmQ0LX82r7SLQkpUl4tT9Re5gOKFMej9PJUfhLKl8Ss+A13?= =?us-ascii?Q?FDsl1gMsjLLkIzZZOjc8WwFTEfnIZ89b//2eSQhNa8relW3a/VJvNytmnjbN?= =?us-ascii?Q?ulUzqhcy48gkM3IBmceyNcnT0NmyaZJ2gYL0b8RwwAyWJo/Co2hVI14ePd7e?= =?us-ascii?Q?8VWUIzT8PoWPJmlHSn34u4jidMUtHpvytCZ2CRMPfMcLtK602qgdCi6Ar5Z1?= =?us-ascii?Q?1yizP2zutjPMG25K3u+6YoJxB9GI4TxD6vl/l5U+YxVrj79v2m+mPYrC4WDc?= =?us-ascii?Q?GpTwha2MwzArFqDusZVRMKvkJw/JYb1DSyjz9ximcosqrMudx/OZi7JgXAIj?= =?us-ascii?Q?z+vBYBiHsQeWlrnI0empiYt8vdJFeqlYFNedSomHAiBJ+qcZ1imAEwpRedni?= =?us-ascii?Q?0p2DaeYQaBe93D797QLYCE3frpyvBRLgHpJLw9dXX3Jor0foAtCMFGDlpe+a?= =?us-ascii?Q?quKxY+Mhe1O5j/9tznjzgUGJRIxKfn7Pt33XMbCW8Xqg0hM47DBtFVeO4LLr?= =?us-ascii?Q?V6HHhYgMXvqqosZwe5uYQEeywMVOjw82RBn23kPHFmqsAXAnRhJbKXfsmEit?= =?us-ascii?Q?BMCb6XGrhqMdS9Oq2j73EreO?= X-OriginatorOrg: virtuozzo.com X-MS-Exchange-CrossTenant-Network-Message-Id: 28059b9c-e12b-4f59-6e70-08d8dffd1171 X-MS-Exchange-CrossTenant-AuthSource: AM7PR08MB5494.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Mar 2021 17:35:28.7646 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 0bc7f26d-0264-416e-a6fc-8352af79c58f X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: ypONcR4uPtbM1JfFIBN6Vi8xX31mIQNQY71DRGnEFDXEhn7BnF3WUM8gYHGLMHXJKmN1z8qbBTQmL6040PdoywzFdGytJGSjt50/wDyUUcw= X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM6PR08MB4408 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=40.107.3.124; envelope-from=vsementsov@virtuozzo.com; helo=EUR03-AM5-obe.outbound.protection.outlook.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, MSGID_FROM_MTA_HEADER=0.001, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @virtuozzo.com) Content-Type: text/plain; charset="utf-8" Implement cache for small sequential unaligned writes, so that they may be cached until we get a complete cluster and then write it. The cache is intended to be used for backup to qcow2 compressed target opened in O_DIRECT mode, but can be reused for any similar (even not block-layer related) task. Signed-off-by: Vladimir Sementsov-Ogievskiy --- include/qemu/seqcache.h | 42 +++++ util/seqcache.c | 361 ++++++++++++++++++++++++++++++++++++++++ MAINTAINERS | 6 + util/meson.build | 1 + 4 files changed, 410 insertions(+) create mode 100644 include/qemu/seqcache.h create mode 100644 util/seqcache.c diff --git a/include/qemu/seqcache.h b/include/qemu/seqcache.h new file mode 100644 index 0000000000..a308d54b01 --- /dev/null +++ b/include/qemu/seqcache.h @@ -0,0 +1,42 @@ +/* + * Cache for small sequential write requests. + * + * Copyright (c) 2021 Virtuozzo International GmbH. + * + * Permission is hereby granted, free of charge, to any person obtaining a= copy + * of this software and associated documentation files (the "Software"), t= o deal + * in the Software without restriction, including without limitation the r= ights + * to use, copy, modify, merge, publish, distribute, sublicense, and/or se= ll + * copies of the Software, and to permit persons to whom the Software is + * furnished to do so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included= in + * all copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS= OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL + * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OT= HER + * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING= FROM, + * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS = IN + * THE SOFTWARE. + */ + +#ifndef QEMU_SEQCACHE_H +#define QEMU_SEQCACHE_H + +typedef struct SeqCache SeqCache; + +SeqCache *seqcache_new(int64_t cluster_size); +void seqcache_free(SeqCache *s); + +void seqcache_write(SeqCache *s, int64_t offset, int64_t bytes, uint8_t *b= uf); +int64_t seqcache_read(SeqCache *s, int64_t offset, int64_t bytes, uint8_t = *buf); + +bool seqcache_get_next_flush(SeqCache *s, int64_t *offset, int64_t *bytes, + uint8_t **buf, bool *unfinished); +void seqcache_discard_cluster(SeqCache *s, int64_t offset); + +uint64_t seqcache_nb_clusters(SeqCache *s); + +#endif /* QEMU_SEQCACHE_H */ diff --git a/util/seqcache.c b/util/seqcache.c new file mode 100644 index 0000000000..d923560eab --- /dev/null +++ b/util/seqcache.c @@ -0,0 +1,361 @@ +/* + * Cache for small sequential write requests. + * + * Copyright (c) 2021 Virtuozzo International GmbH. + * + * Permission is hereby granted, free of charge, to any person obtaining a= copy + * of this software and associated documentation files (the "Software"), t= o deal + * in the Software without restriction, including without limitation the r= ights + * to use, copy, modify, merge, publish, distribute, sublicense, and/or se= ll + * copies of the Software, and to permit persons to whom the Software is + * furnished to do so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included= in + * all copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS= OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL + * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OT= HER + * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING= FROM, + * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS = IN + * THE SOFTWARE. + * + * + * =3D Description =3D + * + * SeqCache is an abbreviation for Sequential Cache. + * + * The Cache is intended to improve performance of small unaligned sequent= ial + * writes. Cache has a cluster_size parameter and the unit of caching is a= ligned + * cluster. Cache has a list of cached clusters, several "finished" ones = and at + * most one "unfinished". "unfinished" cluster is a cluster where last byt= e of + * last write operation is cached and there is a free place after that byt= e to + * the end of cluster. "finished" clusters are just stored in cache to be = read + * or flushed and don't allow any writes to them. + * + * If write to the cache intersects cluster bounds, it's splat into several + * requests by cluster bounds. So, consider a write that doesn't intersect + * cluster bounds to describe the whole picture: + * + * There are two cases allowed: + * + * 1. Sequential write to "unfinished" cluster. Actually it's a write sequ= ential + * previous write. The data goes to "unfinished" cluster. If "unfinishe= d" is + * filled up to the cluster bound it becomes "finished". + * + * 2. Write to new cluster, not existing in the cache. It may be sequentia= l to + * previous write or not. Current "unfinshed" cluster (if exists) becom= es + * "finished" and new "unfinished" cluster is started. Note also that o= ffset + * of the write to new cluster is not required to be aligned. + * + * Any other write operation (non-sequential write to "unfinished" clus= ter + * or write to any of "finished" clusters) will crash. + */ + +#include "qemu/osdep.h" + +#include "qemu/queue.h" +#include "qemu/seqcache.h" + +/* + * Cluster + * + * Representation of one cached cluster, aligned to SeqCache::cluster_size. + * Caches only one subregion of the cluster, started at @offset (may be + * unaligned to cluster_size) and of @bytes length (may be unaligned as we= ll). + * The whole subregion always lay in one aligned cluster: + * + * QEMU_ALIGN_DOWN(offset, cluster_size) =3D=3D + * QEMU_ALIGN_DOWN(offset + bytes - 1, cluster_size) + * + * @buf is allocated to be able to fill the cluster up to the cluster end,= i.e. + * allocated @buf length is at least: + * + * cluster_size - offset % cluster_size + */ +typedef struct Cluster { + int64_t offset; + int64_t bytes; + uint8_t *buf; + QSIMPLEQ_ENTRY(Cluster) entry; +} Cluster; + +/* + * SeqCache caches small sequential writes into "unfinished" @cur_write cl= uster, + * until entire cluster (of @cluster_size bytes) is filled by seqcache_wri= te() + * calls. + * + * @cur_write->offset may be unaligned to @cluster_size if first write off= set is + * not aligned (for example, if there was a flush request and cache was fl= ushed, + * then we continue from the middle of the cluster with an empty cache). + * + * @cur_write may be NULL, which means we don't have current cluster and n= ext + * seqcache_write() will start a new one. + * + * @all is a list of all clusters cached in the cache, some "finished" and= one + * "unfinished" @cur_write (if exists). If @cur_write is not NULL it is a = last + * one in the list. + * + * @nb_clusters is number of elements in @all list. + * + * @next_flush is an iterator for flushing. SeqCache knows nothing about h= ow + * data should be flushing, so for flush user calls seqcache_get_next_flus= h() + * (which moves @next_flush iterator) and when data is flushed somehow and= cache + * is not needed anymore, user can call seqcache_discard_cluster(). + */ +typedef struct SeqCache { + int64_t cluster_size; + int64_t nb_clusters; + Cluster *cur_write; + Cluster *next_flush; + QSIMPLEQ_HEAD(, Cluster) all; +} SeqCache; + +static void cluster_free(Cluster *req) +{ + if (req) { + g_free(req->buf); + g_free(req); + } +} + +SeqCache *seqcache_new(int64_t cluster_size) +{ + SeqCache *s =3D g_new(SeqCache, 1); + + *s =3D (SeqCache) { + .cluster_size =3D cluster_size, + }; + + QSIMPLEQ_INIT(&s->all); + + return s; +} + +/* + * User should clean the cache calling seqcache_get_next_flush() and + * seqcache_discard_cluster() sequentially before freeing it. + */ +void seqcache_free(SeqCache *s) +{ + if (s) { + assert(QSIMPLEQ_EMPTY(&s->all)); + g_free(s); + } +} + +/* End of cached region inside one cluster */ +static int64_t cached_end(Cluster *cl) +{ + return cl->offset + cl->bytes; +} + +/* Start of aligned cluster containing the @offset */ +static int64_t cluster_start(SeqCache *s, int64_t offset) +{ + return QEMU_ALIGN_DOWN(offset, s->cluster_size); +} + +/* End of aligned cluster containing the @offset */ +static int64_t cluster_end(SeqCache *s, int64_t offset) +{ + return cluster_start(s, offset) + s->cluster_size; +} + +/* Align down @offset to s->cluster_size and search for corresponding clus= ter */ +static Cluster *seqcache_find_cluster(SeqCache *s, int64_t offset) +{ + Cluster *cl; + int64_t cl_start =3D cluster_start(s, offset); + + QSIMPLEQ_FOREACH(cl, &s->all, entry) { + if (cluster_start(s, cl->offset) =3D=3D cl_start) { + return cl; + } + } + + return NULL; +} + +/* Makes current "unfinished" cluster the "finished" one. */ +static void seqcache_finalize_current_cluster(SeqCache *s) +{ + if (s->cur_write && !s->next_flush) { + s->next_flush =3D s->cur_write; + } + s->cur_write =3D NULL; +} + +/* + * Write an @offset, @bytes, @buf request into the cache. The requests MUS= T not + * intersect cluster bounds. + */ +static void seqcache_write_one(SeqCache *s, int64_t offset, int64_t bytes, + uint8_t *buf) +{ + assert(bytes > 0); + assert(cluster_start(s, offset) =3D=3D cluster_start(s, offset + bytes= - 1)); + + if (s->cur_write && offset =3D=3D cached_end(s->cur_write)) { + /* Continue sequential process */ + memcpy(s->cur_write->buf + s->cur_write->bytes, buf, bytes); + s->cur_write->bytes +=3D bytes; + + if (cached_end(s->cur_write) =3D=3D cluster_end(s, s->cur_write->o= ffset)) { + seqcache_finalize_current_cluster(s); + } + + return; + } + + /* We are starting a new cluster. Check that it's really new */ + assert(!seqcache_find_cluster(s, offset)); + + seqcache_finalize_current_cluster(s); + + s->cur_write =3D g_new(Cluster, 1); + *s->cur_write =3D (Cluster) { + .offset =3D offset, + .bytes =3D bytes, + .buf =3D g_malloc(s->cluster_size), + }; + + memcpy(s->cur_write->buf, buf, bytes); + QSIMPLEQ_INSERT_TAIL(&s->all, s->cur_write, entry); + s->nb_clusters++; +} + +/* Write an @offset, @bytes, @buf request into the cache. */ +void seqcache_write(SeqCache *s, int64_t offset, int64_t bytes, uint8_t *b= uf) +{ + while (bytes) { + int64_t cl_end =3D cluster_end(s, offset); + int64_t chunk =3D MIN(bytes, cl_end - offset); + + seqcache_write_one(s, offset, chunk, buf); + offset +=3D chunk; + bytes -=3D chunk; + buf +=3D chunk; + } +} + +/* + * Read from the cache. + * + * If @offset misses the cache, return 0. + * + * If @offset is inside the cache, copy corresponding available data to @b= uf + * (may be less than required @bytes if hole reached earlier) and return n= umber + * of bytes copied. + */ +int64_t seqcache_read(SeqCache *s, int64_t offset, int64_t bytes, uint8_t = *buf) +{ + uint8_t *pos =3D buf; + + while (bytes) { + Cluster *cl =3D seqcache_find_cluster(s, offset); + int64_t chunk; + + if (!cl || offset < cl->offset || offset >=3D cached_end(cl)) { + break; + } + + chunk =3D MIN(bytes, cached_end(cl) - offset); + memcpy(pos, cl->buf + offset - cl->offset, chunk); + offset +=3D chunk; + bytes -=3D chunk; + pos +=3D chunk; + + if (!QEMU_IS_ALIGNED(offset, s->cluster_size)) { + break; + } + } + + return pos - buf; +} + +/* + * Get next region for flushing. @offset, @bytes and @buf are out-paramete= rs + * to return the region. + * + * @unfinished is in-out argument which means that user is interested in + * flushing unfinished cluster too: + * + * If there are "finished" clusters, "finished" cluster is returned and + * *@unfinished is set to false, independently of its original value. + * + * If there are no "finished" clusters but "unfinished" exists (i.e. + * s->cur_write !=3D NULL and it is the only element of s->all), then *@un= finished + * value remains the same and the following logic works: + * + * If *@unfinished: + * return s->cur_write unfinished cluster for flushing + * Else + * return nothing + * + * + * Returns true and set @offset, @bytes, @buf and @unfinished if there is + * something to flush (accordingly to @unfinished value), returns false + * otherwise. + * + * Nothing is removed from the cache. + */ +bool seqcache_get_next_flush(SeqCache *s, int64_t *offset, int64_t *bytes, + uint8_t **buf, bool *unfinished) +{ + Cluster *req =3D s->next_flush; + + if (s->next_flush) { + *unfinished =3D false; + req =3D s->next_flush; + s->next_flush =3D QSIMPLEQ_NEXT(req, entry); + if (s->next_flush =3D=3D s->cur_write) { + s->next_flush =3D NULL; + } + } else if (s->cur_write && *unfinished) { + req =3D s->cur_write; + } else { + return false; + } + + *offset =3D req->offset; + *bytes =3D req->bytes; + *buf =3D req->buf; + + return true; +} + +/* + * Find corresponding cluster and drop it. No matter does requested @offse= t is + * cached itself or not. + */ +void seqcache_discard_cluster(SeqCache *s, int64_t offset) +{ + Cluster *cl =3D seqcache_find_cluster(s, offset); + + if (!cl) { + return; + } + + if (cl =3D=3D s->cur_write) { + assert(cl !=3D s->next_flush); + s->cur_write =3D NULL; + } else if (cl =3D=3D s->next_flush) { + assert(cl !=3D s->cur_write); + s->next_flush =3D QSIMPLEQ_NEXT(s->next_flush, entry); + if (s->next_flush =3D=3D s->cur_write) { + s->next_flush =3D NULL; + } + } + + QSIMPLEQ_REMOVE(&s->all, cl, Cluster, entry); + cluster_free(cl); + s->nb_clusters--; +} + +/* Returns number of cached clusters including unfinished */ +uint64_t seqcache_nb_clusters(SeqCache *s) +{ + return s->nb_clusters; +} diff --git a/MAINTAINERS b/MAINTAINERS index 9b2aa18e1f..5e263c3343 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -3347,3 +3347,9 @@ Performance Tools and Tests M: Ahmed Karaman S: Maintained F: scripts/performance/ + +Sequential Cache +M: Vladimir Sementsov-Ogievskiy +S: Supported +F: util/seqcache.c +F: include/qemu/seqcache.h diff --git a/util/meson.build b/util/meson.build index 984fba965f..6682e463ac 100644 --- a/util/meson.build +++ b/util/meson.build @@ -75,6 +75,7 @@ if have_block util_ss.add(files('block-helpers.c')) util_ss.add(files('qemu-coroutine-sleep.c')) util_ss.add(files('qemu-co-shared-resource.c')) + util_ss.add(files('seqcache.c')) util_ss.add(files('thread-pool.c', 'qemu-timer.c')) util_ss.add(files('readline.c')) util_ss.add(files('throttle.c')) --=20 2.29.2 From nobody Fri May 17 13:29:24 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; arc=pass (i=1dmarc=pass fromdomain=virtuozzo.com); dmarc=pass(p=quarantine dis=none) header.from=virtuozzo.com ARC-Seal: i=2; a=rsa-sha256; t=1614969581; cv=pass; d=zohomail.com; s=zohoarc; b=L99RslcH1629+MFX7dqkklPZsOYKZSimzxSAczIrBU11BovKqlF7B5XRQ5ZTMKYSZAbPsxptVilJyh6xSnmdp03KLq5brDi8/CQXeB8OUGeGQ5sx56R1oiWIe0eKSg6LvxqHkjDbAiq7/WG0OPpGlknATfc/RnLgvknnY7KjQo0= ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1614969581; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=H/+shfLISwkOirf04DiUtMpKyo+BE3BeVeHJm/68kK4=; b=A6iiPE/IiUXwJBDtZ8oeewNgKwbogVU60BoDDixsHW8YgbW+lMCSbaRD5UGwgwRkxCBieOq/2CTDGufmK17s1xjvl3fWQVCQbK4U4ocxUkB9h83stOP9PI+pi541VVYNrbPA6xZwG7wxeG7wBblcL+tqLhan7smrmTf9ozTEjSU= ARC-Authentication-Results: i=2; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; arc=pass (i=1dmarc=pass fromdomain=virtuozzo.com); dmarc=pass header.from= (p=quarantine dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1614969581684994.1431330188417; Fri, 5 Mar 2021 10:39:41 -0800 (PST) Received: from localhost ([::1]:36982 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lIFMS-0005Mf-AP for importer@patchew.org; Fri, 05 Mar 2021 13:39:40 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:42846) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lIEMf-0004zZ-1o; Fri, 05 Mar 2021 12:35:49 -0500 Received: from mail-eopbgr30124.outbound.protection.outlook.com ([40.107.3.124]:32132 helo=EUR03-AM5-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lIEMd-0005C4-Bx; Fri, 05 Mar 2021 12:35:48 -0500 Received: from AM7PR08MB5494.eurprd08.prod.outlook.com (2603:10a6:20b:dc::15) by AM6PR08MB4408.eurprd08.prod.outlook.com (2603:10a6:20b:be::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3890.19; Fri, 5 Mar 2021 17:35:30 +0000 Received: from AM7PR08MB5494.eurprd08.prod.outlook.com ([fe80::f1f0:6610:11f5:5e4a]) by AM7PR08MB5494.eurprd08.prod.outlook.com ([fe80::f1f0:6610:11f5:5e4a%8]) with mapi id 15.20.3912.024; Fri, 5 Mar 2021 17:35:30 +0000 Received: from localhost.localdomain (185.215.60.202) by HE1PR02CA0119.eurprd02.prod.outlook.com (2603:10a6:7:29::48) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3912.17 via Frontend Transport; Fri, 5 Mar 2021 17:35:28 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=BscOvyc8IOSNGM2ZDLVNI86pwhvd19ID1Jj72zPO5B2WpRTeNIg8iqkCGp7Thn1QS0r5jbSWS2ttw0fuLipqG1RrWP1IY+EoAdsD2qU/FbpUAaC0JQ1uITkZO7BFeTy5j+B5g2aHV5PW7KW2eBJQfA4OAyAjscMGTMwwEpsicBROuTEySv01W/7Fq2MXv7bGqU4D39Ry36eqQ01llX9IIkkviMvwj5tIEeH6vkV9CdORF8ef7vtD0HUakdGdv2TpO4IWYHcIpFwXuqaGNVM+Ox/uB6o+BEceZhJVjp6io1rh+RpQM3vqTvTcBoELTFh/AT38C6sitbjwWgnu6Sz+nQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=H/+shfLISwkOirf04DiUtMpKyo+BE3BeVeHJm/68kK4=; b=I9yMeBi1pecMGUDB+ijx/gncL25kW2cDwVwhGqfy0dQxjE1DuQvEPhfWfAVTnELrFYwsggdyLKSqYefyVooAIQG/2KwRrZt0Z32k751blqzzwmLaqt2uQ6/iZ3VdGSIhQQpH1bn5/z4FS0hFB/oJUUWr7UtRC6FwoL4L6NYeu5XgVrfBqHlJvK0ELBMM6NJsJO3zGA+53M45Iby5ZFswMqKJmlQxuFgGu9tmvz/2JquHCfubPYzNfeC9BIPY3w7Af7+sdjIhEb+RQcUebsN8yjsq61vLmKkzurisHsvX1EMMw3Gt8v2ib0CHS5X5PW4LbdFMI2nWT1Q1NJiJRL94yQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=virtuozzo.com; dmarc=pass action=none header.from=virtuozzo.com; dkim=pass header.d=virtuozzo.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=virtuozzo.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=H/+shfLISwkOirf04DiUtMpKyo+BE3BeVeHJm/68kK4=; b=bVBtgQdrZh0/AJrb1mTUbirzwpxn3hoTHG245ZqDG4qvGGRJ91GB9b+q/g3fz67ohwroAjm4kEDzmM9S/EyiYC7Z3yMh43PZ9otcDtoiFvPG68tjz5fIn5x6sW7wwyRxqquWBcKgcWr4ubaaTKnPiTR2eoEb0jDtlTIJFu1etMU= Authentication-Results: nongnu.org; dkim=none (message not signed) header.d=none;nongnu.org; dmarc=none action=none header.from=virtuozzo.com; From: Vladimir Sementsov-Ogievskiy To: qemu-block@nongnu.org Cc: qemu-devel@nongnu.org, crosa@redhat.com, ehabkost@redhat.com, mreitz@redhat.com, kwolf@redhat.com, vsementsov@virtuozzo.com, jsnow@redhat.com Subject: [PATCH v3 5/6] block-coroutine-wrapper: allow non bdrv_ prefix Date: Fri, 5 Mar 2021 20:35:06 +0300 Message-Id: <20210305173507.393137-6-vsementsov@virtuozzo.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210305173507.393137-1-vsementsov@virtuozzo.com> References: <20210305173507.393137-1-vsementsov@virtuozzo.com> Content-Transfer-Encoding: quoted-printable X-Originating-IP: [185.215.60.202] X-ClientProxiedBy: HE1PR02CA0119.eurprd02.prod.outlook.com (2603:10a6:7:29::48) To AM7PR08MB5494.eurprd08.prod.outlook.com (2603:10a6:20b:dc::15) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: b238ec45-a0ee-465d-7641-08d8dffd1243 X-MS-TrafficTypeDiagnostic: AM6PR08MB4408: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:608; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: In9l2GfvblUw5X2n+d5ZMnF3q+A3Gw8YLBBJ071pzDLTfGSphd8SzLBQCNz4/O703AOPmV7Te883f3YNlhFkz5M6A/fN64m8vn8MTMeqULshlCzmlrp0OmVn01w3NiinsJsH7h4ISeKCh08iXPyNi/8ZVTZAemXvo8r4qvHxsWq9UWz/FJVqpq6UbmOY54910WX+TplWXoNLtMptw+9pcd9f1THLwG5g49O/+lv1VD7eQQdxsSR2eQ+eQsG8Cve/bcaxo1XoX+JXGMIJykOYfQqEoA+r5YXo4kroYIMbPU8F4rz5BHKsFmO+D7u4qwhUsdReI18Lekyg4JVPC2kS7UnCQnTRB9h/LQ5LfBFwj3dIIAFxOWrYDDrRqHG1uLURMU3HWq4JYFMqoeYMH/pVwJ9Fldau6cDLLLe/N/Dw/p7I92YnK98RN8tGj+bhpYKB1UgFr3XkhAdkA25WvbPMVXhHX6GYOaJ3lxt7Wb+cBM0Cc7AnzOw8o9dMxCywmdRDH6cHItUSOuH1RdoiTSd3RMEbVvPzuonIJ0faeRv22/ZBXEGVN2ytAw+QmMTAuHCu X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:AM7PR08MB5494.eurprd08.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(136003)(366004)(39850400004)(396003)(376002)(346002)(16526019)(66476007)(478600001)(186003)(8936002)(8676002)(6916009)(1076003)(956004)(6486002)(83380400001)(69590400012)(52116002)(26005)(6666004)(6506007)(2906002)(2616005)(4326008)(66556008)(316002)(6512007)(36756003)(66946007)(86362001)(5660300002); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData: =?us-ascii?Q?avsfbKwDUx92VsIYNgPZ2C4mH+qjUnWCYRfzA9Ui66CMLvvkbpfVFNuqyEq3?= =?us-ascii?Q?1WbR1hioGbXWwXJwVIjKKlF27UdNZZRgtDTDFk3hpAd7ZpPxA8GpY0JSKm3p?= =?us-ascii?Q?nAs+repQOw4fkI5lZwQnnKLk4suj/InJVVyR83T+yuNoOD2a+Q95ggISne/h?= =?us-ascii?Q?UdKVFFT0urwNSBjmh/tyKapZofsFmQA6mc6sVOABh6bAW9jF0s6IYMoVUwAE?= =?us-ascii?Q?eC3F2mBkkViodYh8kbCLjisC7rM5+cM5um1mD4nhq8vj1ciBF5BlZNQRo63M?= =?us-ascii?Q?AA69uwc/NBqlrT7nBBdlIBaT0Uk1JwNeW9HJ88vPYnM7uw3JtOcHCMzSKd++?= =?us-ascii?Q?sw22IxBm6ymt8XSewVMDD37cM1aA65Juw3kAmnoj4G3jsTCSpS1H2eViwKDC?= =?us-ascii?Q?GwLodspKj+Wy6tVQmNHP6BO9d4QwVzu+1duKIkp55fvMFMUdLJT6h4V5TYCR?= =?us-ascii?Q?1wYDZsnsyYv4Ok5TXe+CeYKMDFWCXGUZeivbCIw+UJijuMCDJWw1x+Fc73BW?= =?us-ascii?Q?qgujq2XnRjuC818ii4SE5zFYETKmC+OYiUzYr2bpqsmovDLzIMKYBZtZ114h?= =?us-ascii?Q?6EyLaInr/iUzNhB1aM+ispyGe76p5npYYwe5ATDOJNxdSdGL6OCTHX7hIIm5?= =?us-ascii?Q?fjVcWDOhFx5KgAcNYFtvoqR5D+P1E2G7xNtP3ZA8AosbtoGWc3P0xR2gBu4+?= =?us-ascii?Q?74J/ypMWUz4II7EkIjRZ0lPCw36qbBEVC1Ic7lcm0gbS+8W+0s9R6SpfANtt?= =?us-ascii?Q?MxTIBVh5QUZPJ1hbDjQXrNXEjGfG7HGq9FfK0yFSG+EkrrOR6rXLjLf2lBae?= =?us-ascii?Q?z4G/zWuCq0PbaqLEFmt0BPxLDxBQpXSAl6eFNiYac0KLH+rJ0/F2AOKjeQTZ?= =?us-ascii?Q?ie3BiBqDBZ9bezoN5S2zRQFzfzLpAbmL4MPJD58lWxkZMjGpmc8xaco7K+e+?= =?us-ascii?Q?yN+T/qaqzDwr97/iuhiEYk8HmUSrDg2vFPzlBEKkjMLFaUeyUpqnNeIum5Ls?= =?us-ascii?Q?yLwCZs5xT8QtQYppCreYjDR0USUJ3O3rxhq7uJRMWq+g46d+V0rVVpDd+uN5?= =?us-ascii?Q?rWpYJs6fLaYM86DBpmEMoB1FrALT8GUXq3DRTXj8Nbrt8bsH+aORUrZypG00?= =?us-ascii?Q?rbW5q5estZWXo3TxnAVl2QKUVT5VTn5l77gpC5/AeCD6bKef3Evp4Q2YJHly?= =?us-ascii?Q?VRU6inlzJKFiQ/qNlkPMQ4tcvlMPlPaOrAiq4hGOjDk+HOWDZOHO5FYGxbZ1?= =?us-ascii?Q?CfsAYPQRXYEIRkPAl+t2VOE/Ok904hCOFA6g+CZU2NV4tvsNZsNF07huexPa?= =?us-ascii?Q?PYkrynBPWFc7gKLozqQdh+uL?= X-OriginatorOrg: virtuozzo.com X-MS-Exchange-CrossTenant-Network-Message-Id: b238ec45-a0ee-465d-7641-08d8dffd1243 X-MS-Exchange-CrossTenant-AuthSource: AM7PR08MB5494.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Mar 2021 17:35:30.0778 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 0bc7f26d-0264-416e-a6fc-8352af79c58f X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: drcExzDQ2L/yoYt0ElNdYqSuGsdQGmdcD5BQzaRmndfTu1XaBMga5jgDQ3C9sD49SRiaiANjPiu0aHZmTfizFta0juNBv8CMWCicP7CgQe8= X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM6PR08MB4408 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=40.107.3.124; envelope-from=vsementsov@virtuozzo.com; helo=EUR03-AM5-obe.outbound.protection.outlook.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, MSGID_FROM_MTA_HEADER=0.001, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @virtuozzo.com) Content-Type: text/plain; charset="utf-8" We are going to reuse the script to generate a qcow2_ function in further commit. Prepare the script now. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz --- scripts/block-coroutine-wrapper.py | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/scripts/block-coroutine-wrapper.py b/scripts/block-coroutine-w= rapper.py index 0461fd1c45..85dbeb9ecf 100644 --- a/scripts/block-coroutine-wrapper.py +++ b/scripts/block-coroutine-wrapper.py @@ -98,12 +98,13 @@ def snake_to_camel(func_name: str) -> str: =20 =20 def gen_wrapper(func: FuncDecl) -> str: - assert func.name.startswith('bdrv_') - assert not func.name.startswith('bdrv_co_') + assert not '_co_' in func.name assert func.return_type =3D=3D 'int' assert func.args[0].type in ['BlockDriverState *', 'BdrvChild *'] =20 - name =3D 'bdrv_co_' + func.name[5:] + subsystem, subname =3D func.name.split('_', 1) + + name =3D f'{subsystem}_co_{subname}' bs =3D 'bs' if func.args[0].type =3D=3D 'BlockDriverState *' else 'chi= ld->bs' struct_name =3D snake_to_camel(name) =20 --=20 2.29.2 From nobody Fri May 17 13:29:24 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; arc=pass (i=1dmarc=pass fromdomain=virtuozzo.com); dmarc=pass(p=quarantine dis=none) header.from=virtuozzo.com ARC-Seal: i=2; a=rsa-sha256; t=1614969313; cv=pass; d=zohomail.com; s=zohoarc; b=msj5L0GsGhvzM479y9iJIgYkrDgvrCMbHnxayaS5FIsbSulpuNmJ9CcSYXgO4z3uhGxxjBnDXcwa+hI4FsuD0ORTVhKYAOkboaOkpnOIBkikThSr8+qSD5rZrCQEbtBCuYZ+nV1aMIj6WSITsGi6YbOhW3PVzmMTw83wwLWP0r8= ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1614969313; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=42OGlUD99CJV/hXRGe+/X0EzOEnI74ar1pVUsDwjesw=; b=mIl2aq8K6WU24ytfKQQitY27mdTjncqzC8IT1rk0XCuo/hxbT4PpbFGeqoomn6SGLADX5rKFshEpvrPl3R7E7XB62j2+MRHqgD9NeatuXo12QkjHD0pzDfLa4z6OMnTlO+UIPkWoeXv3rVxetSy51Hd9XcvXSC4r7VHo7Vurb6I= ARC-Authentication-Results: i=2; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; arc=pass (i=1dmarc=pass fromdomain=virtuozzo.com); dmarc=pass header.from= (p=quarantine dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1614969313918650.5144902256907; Fri, 5 Mar 2021 10:35:13 -0800 (PST) Received: from localhost ([::1]:52402 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lIFI8-0008SU-Qa for importer@patchew.org; Fri, 05 Mar 2021 13:35:12 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:42766) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lIEMT-0004h1-UC; Fri, 05 Mar 2021 12:35:38 -0500 Received: from mail-eopbgr20134.outbound.protection.outlook.com ([40.107.2.134]:49806 helo=EUR02-VE1-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lIEMR-0005EG-98; Fri, 05 Mar 2021 12:35:37 -0500 Received: from AM7PR08MB5494.eurprd08.prod.outlook.com (2603:10a6:20b:dc::15) by AM6PR08MB4850.eurprd08.prod.outlook.com (2603:10a6:20b:d2::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3890.26; Fri, 5 Mar 2021 17:35:31 +0000 Received: from AM7PR08MB5494.eurprd08.prod.outlook.com ([fe80::f1f0:6610:11f5:5e4a]) by AM7PR08MB5494.eurprd08.prod.outlook.com ([fe80::f1f0:6610:11f5:5e4a%8]) with mapi id 15.20.3912.024; Fri, 5 Mar 2021 17:35:31 +0000 Received: from localhost.localdomain (185.215.60.202) by HE1PR02CA0119.eurprd02.prod.outlook.com (2603:10a6:7:29::48) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3912.17 via Frontend Transport; Fri, 5 Mar 2021 17:35:30 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=mt0+DCZoPNMTwdHTPXZDMKXSde+k02TpR7wmqbkMOdhpRH5fBhOEoClVNWhlLGGpk6G3x62noN8p+qT9/ICQoHlxNLb2yGrcP4IBsNvAJe+p+GGcEW+Nv+RYK9gY32lbTDQX/vtG83xnKvOG6rXWaV7pd6jzJJQGQJDwUfs5PAdu25KKY+Oym5U3IOYfNl9q/KQSss0VDfEgV/rc0e/e7sQJBH3ql1RlVZ26bklxVNZQCerMeQBc+Kx9g16xjbbSCMa00XjHa15kpZ2K+HAq9Y868Dswui+2FJAUnQ4B5gV/1/qChZDNBOYw2jsHuM40Jc5zcmhYCpkXU9TJGnJ9Zw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=42OGlUD99CJV/hXRGe+/X0EzOEnI74ar1pVUsDwjesw=; b=DBu+3bYLaZQjvjEBfFZPRQqS2vg5fR9voLKOH4QAETsK5pOcv2nCxf8y61jDWGsTYSq3/A2yjgslRjrAOD1BJ3uMyxB0wqpHLnP6cmgaqUKzal/5VQW6SPkdXTawY0XToQBpYhsSjMwbhaglmqU8x+3mGkBhWetyOkjKoDvqUv7rRjRPCmDM+Q139LTsT6f5XR/Yus06Qu2X7O7FjyY3OJIZGn7cJyza6xPn6iXZ0ss7KpLuhQfmzqLHswXUqfW1TUJRdzNYpgCV7tYqvAOLGES0q+CpAyMQ9ag2tR8UUqtuk9q5VSKYKo2FkqV/NbbnCoP1OE2yxdHu+lb5ZeQYpQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=virtuozzo.com; dmarc=pass action=none header.from=virtuozzo.com; dkim=pass header.d=virtuozzo.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=virtuozzo.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=42OGlUD99CJV/hXRGe+/X0EzOEnI74ar1pVUsDwjesw=; b=fN3OpmhHl+Vf0zybfwidf7Jd2bmPvWRSc7cl8VYJZWMfQ6yDpE06lduvukBf6iEOXlJ6S5mXa5C8GUoJ+fZF4C2e31dzibF1jvrfzt9x+CzPDiorptLrRHV7uapY1Lio13aCqg6ijsUYARcFeqS4vCv/uddi49bxnV5xUmlfMMw= Authentication-Results: nongnu.org; dkim=none (message not signed) header.d=none;nongnu.org; dmarc=none action=none header.from=virtuozzo.com; From: Vladimir Sementsov-Ogievskiy To: qemu-block@nongnu.org Cc: qemu-devel@nongnu.org, crosa@redhat.com, ehabkost@redhat.com, mreitz@redhat.com, kwolf@redhat.com, vsementsov@virtuozzo.com, jsnow@redhat.com Subject: [PATCH v3 6/6] block/qcow2: use seqcache for compressed writes Date: Fri, 5 Mar 2021 20:35:07 +0300 Message-Id: <20210305173507.393137-7-vsementsov@virtuozzo.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210305173507.393137-1-vsementsov@virtuozzo.com> References: <20210305173507.393137-1-vsementsov@virtuozzo.com> Content-Transfer-Encoding: quoted-printable X-Originating-IP: [185.215.60.202] X-ClientProxiedBy: HE1PR02CA0119.eurprd02.prod.outlook.com (2603:10a6:7:29::48) To AM7PR08MB5494.eurprd08.prod.outlook.com (2603:10a6:20b:dc::15) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 6ec31d52-192e-43f4-e8a8-08d8dffd130c X-MS-TrafficTypeDiagnostic: AM6PR08MB4850: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:2043; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: a7S4D3nuC5YSvNlZRxJisKedY7cJtE9HSpWL7KtxPkch4ZZ8kovIKTpVa6yKZNy0R9Wx7QaVOOsJpOKGoJHQ4rzXnou1PslcIpLhtO5cX3vWrFwOqdkY1CgfnMj/zSX/Ntf0A9HJKvvwOINp3wW9+ICVuQgJwTxrFanuOba7+svKHqhwhXNuaskvBuAdpfBZsg2oiq78WM6eTOo4efTrEoFvuNyI3u8+15bQgbzwmYL/ALxbIM5dDdL6igp0kjaNct0y8Z2VFPtt0FEQ1pi20ShuOUQVHwlqm6gHSGrWkz+ivDDUX4+s92UqWApG4Cf4zMnMnELvql09XdpR7U/wSL81xnoK/mKtkbppnhEtW8LdAnO7ozDHN7Dclh8E9ht/y9zE0MJp17jsM0cib4JuWLgflM5qN7YlW7aNBxuwJbtoqXdi12OGG8jqUI+DbEWq5KhEw2OtItDGWGhhj6G4K/PnHz//4IRUDK/uNCWLDChpUXg+GfzY+AJcXB3Ghl7k2lLAmY00WOIvLhLWCKiqxx6PIiDnIk/RMIHpTyXjm0Cqw4pfIJHapWYl9trRopAKTDqtzWWg14jXOK/awnnLbQ== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:AM7PR08MB5494.eurprd08.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(39850400004)(396003)(136003)(376002)(366004)(346002)(66476007)(52116002)(66946007)(478600001)(316002)(66556008)(6486002)(19627235002)(6512007)(16526019)(5660300002)(186003)(83380400001)(8676002)(4326008)(2906002)(2616005)(86362001)(956004)(6666004)(26005)(6506007)(6916009)(69590400012)(36756003)(8936002)(1076003); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData: =?us-ascii?Q?YcLDAte2uFR2ip0a3ZIRoa3z1lxf24Xo1/nnv1F0/i4rIyvr1M6qSw12NcCZ?= =?us-ascii?Q?66KsaYBurdsmo0ztE/Kd8FwqVmUhLoX4x7khOXwGvAkna6dv7GGjdxCSJVDv?= =?us-ascii?Q?1+3DGCAzlDwRLKuGILZPRTKlC5jDXTt+3j42AT86dzXsfSr8PuzY2l66hXP9?= =?us-ascii?Q?XO0iH4TZx+LMW5larg2a4D8W8PjpKua7ArhmVDwxAGZWmY81EKXMnRvmqzcj?= =?us-ascii?Q?zjd3H5pvpUm6Yqq156QIKtx5NhZ+/77VUgnNPwAfqluijOyjsWWrD3wxd5Lp?= =?us-ascii?Q?oTHZZ/dH5O6oH3u7Em8ZZBKPxcDNQthWL0d7I86vF+7mHNeKoQpjI/2+nUFZ?= =?us-ascii?Q?ImGqv/knqP8TM9XGkwhf3CIdmaKqA7coa65wkGOvx9jvDTVeWJgGfNYZAvFK?= =?us-ascii?Q?vxSm7BXFw0DEz0Jmc9FKzu17oOuDGbUhoDUu4cAY8HrZ2M4y5byBWqIH36Wx?= =?us-ascii?Q?vYd46+CHdot/DyFHJIGbcey/SIMSpNkHVjnyZrkIRW2zM/sTFMTuMJpZWDDm?= =?us-ascii?Q?OeS2ZUErrjy9EoKgNeu8a2mnAUTq0xh5VrJzmksVT6nr/olIaK6e1rofJPcd?= =?us-ascii?Q?U4oQYh9xvqDNfHWqOA13j7zyT0rc0yywHi5qiGKh2mNCmG5wa0/w3XM2js5G?= =?us-ascii?Q?F/enuGsTC01cjw22vMputCUhkThu2ICOHmcK4L+2WqmzmKIzMy0+g1DYx7vX?= =?us-ascii?Q?ZXfoEemys6G94vbncef6yeuAVnMDiiXJFiyBp/NwRrfo1rmkCsxHyes0oI+n?= =?us-ascii?Q?1i5FQ4+GszyfwZ0vaTRpsDKIIkxH4p5UJotK0HE5MryKqdsdEXqcldPKt26t?= =?us-ascii?Q?QTyiQMoHCoFs4Skj7hQtNaJcSZb982jqbbo+y5+/Tr5e2SWzxd4H7Y+PHPhw?= =?us-ascii?Q?+KfedMmPpUnKlTRNJCzAgQ1ZVT+djTIHyKiK91FDCn/DD/fO2PsuNMra/qN7?= =?us-ascii?Q?pvz6zh/AwqwbFkcX+QsVnBAzlonE9TdoG6DNk0fX6ApTOe/M0JbJ66jys3Gq?= =?us-ascii?Q?zvXVcGhiIosLThXuDqxkAtQleL+RADv0RLiQmtZQn2XYaXAKTwySaLwNbwHu?= =?us-ascii?Q?wCRHfZHeelvUBdB/7QevsSusTEIIEM5SDEbdduv1DEir+GnyWjQzVkr35FRd?= =?us-ascii?Q?LXqkOs1zM0G0RwUe/002NohXGfLBD5LkFvuM6Oc96TUrLwk9EZZTzjg2X0mB?= =?us-ascii?Q?zeTWtQVEW4s3si39NQlNKvp9b2VNGGbg/GXifd38/xPfmkBcaC3e3NXmVWW6?= =?us-ascii?Q?XCERAAq4KoKarjYL/OSpOu4UTxCe2Tv0x1bu6B5GvtWNy06pzXbou4RJUPHs?= =?us-ascii?Q?5z0epmzVJWx3q2mcX818bza/?= X-OriginatorOrg: virtuozzo.com X-MS-Exchange-CrossTenant-Network-Message-Id: 6ec31d52-192e-43f4-e8a8-08d8dffd130c X-MS-Exchange-CrossTenant-AuthSource: AM7PR08MB5494.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Mar 2021 17:35:31.3816 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 0bc7f26d-0264-416e-a6fc-8352af79c58f X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: EoUOsNs2tyvQQOyI80GVdAR2Mqj3sAnr74AULf517eJNV5Utt2EBgWbV4+KrW1mSuYhfXwXnlqg61jFw6DGeOcbwP0dSssblWauGabQPcUs= X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM6PR08MB4850 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=40.107.2.134; envelope-from=vsementsov@virtuozzo.com; helo=EUR02-VE1-obe.outbound.protection.outlook.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, MSGID_FROM_MTA_HEADER=0.001, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @virtuozzo.com) Content-Type: text/plain; charset="utf-8" Compressed writes are unaligned to 512, which works very slow in O_DIRECT mode. Let's use the cache. Signed-off-by: Vladimir Sementsov-Ogievskiy --- block/coroutines.h | 3 + block/qcow2.h | 4 ++ block/qcow2-refcount.c | 10 +++ block/qcow2.c | 158 ++++++++++++++++++++++++++++++++++++++--- 4 files changed, 164 insertions(+), 11 deletions(-) diff --git a/block/coroutines.h b/block/coroutines.h index 4cfb4946e6..cb8e05830e 100644 --- a/block/coroutines.h +++ b/block/coroutines.h @@ -66,4 +66,7 @@ int coroutine_fn bdrv_co_readv_vmstate(BlockDriverState *= bs, int coroutine_fn bdrv_co_writev_vmstate(BlockDriverState *bs, QEMUIOVector *qiov, int64_t pos); =20 +int coroutine_fn qcow2_co_flush_compressed_cache(BlockDriverState *bs); +int generated_co_wrapper qcow2_flush_compressed_cache(BlockDriverState *bs= ); + #endif /* BLOCK_COROUTINES_INT_H */ diff --git a/block/qcow2.h b/block/qcow2.h index e18d8dfbae..8b8fed0950 100644 --- a/block/qcow2.h +++ b/block/qcow2.h @@ -28,6 +28,7 @@ #include "crypto/block.h" #include "qemu/coroutine.h" #include "qemu/units.h" +#include "qemu/seqcache.h" #include "block/block_int.h" =20 //#define DEBUG_ALLOC @@ -422,6 +423,9 @@ typedef struct BDRVQcow2State { Qcow2CompressionType compression_type; =20 GHashTable *inflight_writes_counters; + + SeqCache *compressed_cache; + int compressed_flush_ret; } BDRVQcow2State; =20 typedef struct Qcow2COWRegion { diff --git a/block/qcow2-refcount.c b/block/qcow2-refcount.c index 464d133368..218917308e 100644 --- a/block/qcow2-refcount.c +++ b/block/qcow2-refcount.c @@ -29,6 +29,7 @@ #include "qemu/bswap.h" #include "qemu/cutils.h" #include "trace.h" +#include "block/coroutines.h" =20 static int64_t alloc_clusters_noref(BlockDriverState *bs, uint64_t size, uint64_t max); @@ -1040,6 +1041,10 @@ static int QEMU_WARN_UNUSED_RESULT update_refcount(B= lockDriverState *bs, qcow2_cache_discard(s->l2_table_cache, table); } =20 + if (s->compressed_cache) { + seqcache_discard_cluster(s->compressed_cache, cluster_offs= et); + } + if (s->discard_passthrough[type]) { update_refcount_discard(bs, cluster_offset, s->cluster_siz= e); } @@ -1349,6 +1354,11 @@ int coroutine_fn qcow2_write_caches(BlockDriverState= *bs) BDRVQcow2State *s =3D bs->opaque; int ret; =20 + ret =3D qcow2_flush_compressed_cache(bs); + if (ret < 0) { + return ret; + } + ret =3D qcow2_cache_write(bs, s->l2_table_cache); if (ret < 0) { return ret; diff --git a/block/qcow2.c b/block/qcow2.c index 6ee94421e0..5f3713cd6f 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -42,6 +42,7 @@ #include "qapi/qapi-visit-block-core.h" #include "crypto.h" #include "block/aio_task.h" +#include "block/coroutines.h" =20 /* Differences with QCOW: @@ -1834,6 +1835,10 @@ static int coroutine_fn qcow2_do_open(BlockDriverSta= te *bs, QDict *options, s->inflight_writes_counters =3D g_hash_table_new_full(g_int64_hash, g_int64_equal, g_free, g_free); =20 + if (!has_data_file(bs) && (bs->file->bs->open_flags & BDRV_O_NOCACHE))= { + s->compressed_cache =3D seqcache_new(s->cluster_size); + } + return ret; =20 fail: @@ -2653,6 +2658,91 @@ fail_nometa: return ret; } =20 +/* + * Flush one cluster of compressed cache if exists. If @unfinished is true= and + * there is no finished cluster to flush, flush the unfinished one. Otherw= ise + * flush only finished clusters. + * + * Return 0 if nothing to flush, 1 if one cluster successfully flushed and= <0 on + * error. + */ +static int coroutine_fn +qcow2_co_compressed_flush_one(BlockDriverState *bs, bool unfinished) +{ + BDRVQcow2State *s =3D bs->opaque; + int ret; + int64_t align =3D 1; + int64_t offset, bytes; + uint8_t *buf; + + if (!s->compressed_cache) { + return 0; + } + + if (!seqcache_get_next_flush(s->compressed_cache, &offset, &bytes, &bu= f, + &unfinished)) + { + return 0; + } + + qcow2_inflight_writes_inc(bs, offset, bytes); + + /* + * Don't try to align-up unfinished cached cluster, parallel write to = it is + * possible! For finished host clusters data in the tail of the cluste= r will + * be never used. So, take some good alignment for speed. + * + * Note also, that seqcache guarantees that allocated size of @buf is = enough + * to fill the cluster up to the end, so we are safe to align up with + * align <=3D cluster_size. + */ + if (!unfinished) { + align =3D MIN(s->cluster_size, + MAX(s->data_file->bs->bl.request_alignment, + bdrv_opt_mem_align(bs))); + } + + BLKDBG_EVENT(s->data_file, BLKDBG_WRITE_COMPRESSED); + ret =3D bdrv_co_pwrite(s->data_file, offset, + QEMU_ALIGN_UP(offset + bytes, align) - offset, + buf, 0); + if (ret < 0 && s->compressed_flush_ret =3D=3D 0) { + /* + * The data that was "written" earlier is lost. We don't want to + * care with storing it somehow to retry flushing later. Also, how= much + * data will we have to store, if not drop it after failed flush? + * After this point qcow2_co_flush_compressed_cache() (and + * qcow2_flush_compressed_cache()) never return success. + */ + s->compressed_flush_ret =3D ret; + } + + seqcache_discard_cluster(s->compressed_cache, offset); + + qcow2_inflight_writes_dec(bs, offset, bytes); + + return ret < 0 ? ret : 1; +} + +int coroutine_fn qcow2_co_flush_compressed_cache(BlockDriverState *bs) +{ + BDRVQcow2State *s =3D bs->opaque; + + if (s->compressed_cache) { + uint64_t nb_clusters =3D seqcache_nb_clusters(s->compressed_cache); + + /* + * If parallel writes are possible we don't want to loop forever. = So + * flush only clusters available at start of flush operation. + */ + while (nb_clusters--) { + qcow2_co_compressed_flush_one(bs, true); + } + } + + return s->compressed_flush_ret; +} + static int qcow2_inactivate(BlockDriverState *bs) { BDRVQcow2State *s =3D bs->opaque; @@ -2667,6 +2757,13 @@ static int qcow2_inactivate(BlockDriverState *bs) bdrv_get_device_or_node_name(bs)); } =20 + ret =3D qcow2_flush_compressed_cache(bs); + if (ret) { + result =3D ret; + error_report("Failed to flush the compressed write cache: %s", + strerror(-ret)); + } + ret =3D qcow2_cache_flush(bs, s->l2_table_cache); if (ret) { result =3D ret; @@ -2699,6 +2796,12 @@ static void qcow2_close(BlockDriverState *bs) qcow2_inactivate(bs); } =20 + /* + * Cache should be flushed in qcow2_inactivate() and should be empty in + * inactive mode. So we are safe to free it. + */ + seqcache_free(s->compressed_cache); + cache_clean_timer_del(bs); qcow2_cache_destroy(s->l2_table_cache); qcow2_cache_destroy(s->refcount_block_cache); @@ -4558,18 +4661,42 @@ qcow2_co_pwritev_compressed_task(BlockDriverState *= bs, goto fail; } =20 - qcow2_inflight_writes_inc(bs, cluster_offset, out_len); + if (s->compressed_cache) { + /* + * It's important to do seqcache_write() in the same critical sect= ion + * (by s->lock) as qcow2_alloc_compressed_cluster_offset(), so tha= t the + * cache is filled sequentially. + */ + seqcache_write(s->compressed_cache, cluster_offset, out_len, out_b= uf); =20 - qemu_co_mutex_unlock(&s->lock); + qemu_co_mutex_unlock(&s->lock); =20 - BLKDBG_EVENT(s->data_file, BLKDBG_WRITE_COMPRESSED); - ret =3D bdrv_co_pwrite(s->data_file, cluster_offset, out_len, out_buf,= 0); + ret =3D qcow2_co_compressed_flush_one(bs, false); + if (ret < 0) { + /* + * if ret < 0 it probably not this request which failed, but s= ome + * previous one that was cached. Moreover, when we write to the + * cache we probably may always report success, because + * seqcache_write() never fails. Still, it's better to inform + * compressed backup now then wait until final flush. + */ + goto fail; + } + } else { + qcow2_inflight_writes_inc(bs, cluster_offset, out_len); =20 - qcow2_inflight_writes_dec(bs, cluster_offset, out_len); + qemu_co_mutex_unlock(&s->lock); =20 - if (ret < 0) { - goto fail; + BLKDBG_EVENT(s->data_file, BLKDBG_WRITE_COMPRESSED); + ret =3D bdrv_co_pwrite(s->data_file, cluster_offset, out_len, out_= buf, 0); + + qcow2_inflight_writes_dec(bs, cluster_offset, out_len); + + if (ret < 0) { + goto fail; + } } + success: ret =3D 0; fail: @@ -4681,10 +4808,19 @@ qcow2_co_preadv_compressed(BlockDriverState *bs, =20 out_buf =3D qemu_blockalign(bs, s->cluster_size); =20 - BLKDBG_EVENT(bs->file, BLKDBG_READ_COMPRESSED); - ret =3D bdrv_co_pread(bs->file, coffset, csize, buf, 0); - if (ret < 0) { - goto fail; + /* + * seqcache_read may return less bytes than csize, as csize may exceed + * actual compressed data size. So we are OK if seqcache_read returns + * something > 0. + */ + if (!s->compressed_cache || + seqcache_read(s->compressed_cache, coffset, csize, buf) <=3D 0) + { + BLKDBG_EVENT(bs->file, BLKDBG_READ_COMPRESSED); + ret =3D bdrv_co_pread(bs->file, coffset, csize, buf, 0); + if (ret < 0) { + goto fail; + } } =20 if (qcow2_co_decompress(bs, out_buf, s->cluster_size, buf, csize) < 0)= { --=20 2.29.2