From nobody Sun May 19 07:11:47 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1598444315; cv=none; d=zohomail.com; s=zohoarc; b=RN+fai/GnBRMwDKo6vJ2MmMxrBACbvRhBLJKNhdx/t3x2G542c01P1E3L7r70SUcqrjibNyuF1490dLs6ewuILdkaZx2fQeCwQDsN4Ea3nJbHObRHYFpa7TSj1dr4/BTi6cl7TrNJ2+iJWPMhB8CAoNE4s2j2Ut2A/GThYHF7Po= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1598444315; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=Tm1fxchLk7NawaLv6HoG1045kx32Nb0XMC07Gg7Ubyw=; b=jMxkMjorqYy4YbUOJLe6xWPoWGkL4ztS+dW1tqzZ+PhBvDC6UHMSYF0v1vv8ESAeicgSNvnusZikkoB+gw4jHrI2YatkgKzBgruRn0vHdYN9s9AVPE+2A6CDZc2153+jZQKBbDi1z/UOXRUFInfhNzu3wcgKi7mc1ZCgZorX0bk= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1598444315014378.75225745119155; Wed, 26 Aug 2020 05:18:35 -0700 (PDT) Received: from localhost ([::1]:52060 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kAuNq-0004wl-VU for importer@patchew.org; Wed, 26 Aug 2020 08:18:30 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:37508) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kAuMe-0003aQ-74; Wed, 26 Aug 2020 08:17:16 -0400 Received: from proxmox-new.maurer-it.com ([212.186.127.180]:11365) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kAuMb-0004Ck-Nd; Wed, 26 Aug 2020 08:17:15 -0400 Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id 7BE28432AA; Wed, 26 Aug 2020 14:17:01 +0200 (CEST) From: Stefan Reiter To: qemu-block@nongnu.org Subject: [PATCH 1/3] job: add sequential transaction support Date: Wed, 26 Aug 2020 14:13:57 +0200 Message-Id: <20200826121359.15450-2-s.reiter@proxmox.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200826121359.15450-1-s.reiter@proxmox.com> References: <20200826121359.15450-1-s.reiter@proxmox.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=212.186.127.180; envelope-from=s.reiter@proxmox.com; helo=proxmox-new.maurer-it.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/08/26 08:17:01 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, w.bumiller@proxmox.com, armbru@redhat.com, qemu-devel@nongnu.org, mreitz@redhat.com, jsnow@redhat.com, dietmar@proxmox.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Type: text/plain; charset="utf-8" Jobs in a sequential transaction should never be started with job_start manually. job_txn_start_seq and the sequentially called job_start will take care of it, 'assert'ing in case a job is already running or has finished. Signed-off-by: Stefan Reiter --- include/qemu/job.h | 12 ++++++++++++ job.c | 24 ++++++++++++++++++++++++ 2 files changed, 36 insertions(+) diff --git a/include/qemu/job.h b/include/qemu/job.h index 32aabb1c60..f7a6a0926a 100644 --- a/include/qemu/job.h +++ b/include/qemu/job.h @@ -280,6 +280,18 @@ typedef enum JobCreateFlags { */ JobTxn *job_txn_new(void); =20 +/** + * Create a new transaction and set it to sequential mode, i.e. run all jo= bs + * one after the other instead of at the same time. + */ +JobTxn *job_txn_new_seq(void); + +/** + * Helper method to start the first job in a sequential transaction to kic= k it + * off. Other jobs will be run after this one completes. + */ +void job_txn_start_seq(JobTxn *txn); + /** * Release a reference that was previously acquired with job_txn_add_job or * job_txn_new. If it's the last reference to the object, it will be freed. diff --git a/job.c b/job.c index 8fecf38960..4df7c1d2ca 100644 --- a/job.c +++ b/job.c @@ -72,6 +72,8 @@ struct JobTxn { =20 /* Reference count */ int refcnt; + + bool sequential; }; =20 /* Right now, this mutex is only needed to synchronize accesses to job->bu= sy @@ -102,6 +104,25 @@ JobTxn *job_txn_new(void) return txn; } =20 +JobTxn *job_txn_new_seq(void) +{ + JobTxn *txn =3D job_txn_new(); + txn->sequential =3D true; + return txn; +} + +void job_txn_start_seq(JobTxn *txn) +{ + assert(txn->sequential); + assert(!txn->aborting); + + Job *first =3D QLIST_FIRST(&txn->jobs); + assert(first); + assert(first->status =3D=3D JOB_STATUS_CREATED); + + job_start(first); +} + static void job_txn_ref(JobTxn *txn) { txn->refcnt++; @@ -840,6 +861,9 @@ static void job_completed_txn_success(Job *job) */ QLIST_FOREACH(other_job, &txn->jobs, txn_list) { if (!job_is_completed(other_job)) { + if (txn->sequential) { + job_start(other_job); + } return; } assert(other_job->ret =3D=3D 0); --=20 2.20.1 From nobody Sun May 19 07:11:47 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1598444379; cv=none; d=zohomail.com; s=zohoarc; b=erb6TRs42IYKnydNoYXFKMBeGnoo9ay7+pAkA+vr2e4s6HnIdDSzmT+x2WCWMEPTn0M2KACqXmplT1hiT5R9ybZAExwCVJf1zS0GYTBeyKUkRuivL7TeeH3IYHtuVrmdvucYlpYm9iuMoggtDXBG7Sn482fzSE/gJeWut28gr2E= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1598444379; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=hznE0HYPbrfzqfwbAPHtjivY80KkuSjdX2qjQxjFArs=; b=SdFJ6+WwaOW0GZEzkhRVJhK71eak5ADHuzqM+u7BHWX4IXp4jw0OCfRQ+8J4lzb3DT3DKwF7YAv8Hgt22ki455fmCDjMyVNd0DaPnkdBbH7vUEZqOdHCypChfhUHDfG78Lnk6ua3YGje6adGXhqtHo87ZZCcVgfRfKceuNhfI+Q= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 159844437978511.65800844616308; Wed, 26 Aug 2020 05:19:39 -0700 (PDT) Received: from localhost ([::1]:56800 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kAuOw-0006qg-HE for importer@patchew.org; Wed, 26 Aug 2020 08:19:38 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:37506) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kAuMe-0003aO-77; Wed, 26 Aug 2020 08:17:16 -0400 Received: from proxmox-new.maurer-it.com ([212.186.127.180]:18191) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kAuMb-0004Cj-JJ; Wed, 26 Aug 2020 08:17:15 -0400 Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id 91E08448B6; Wed, 26 Aug 2020 14:17:01 +0200 (CEST) From: Stefan Reiter To: qemu-block@nongnu.org Subject: [PATCH 2/3] blockdev: add sequential mode to *-backup transactions Date: Wed, 26 Aug 2020 14:13:58 +0200 Message-Id: <20200826121359.15450-3-s.reiter@proxmox.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200826121359.15450-1-s.reiter@proxmox.com> References: <20200826121359.15450-1-s.reiter@proxmox.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=212.186.127.180; envelope-from=s.reiter@proxmox.com; helo=proxmox-new.maurer-it.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/08/26 08:17:01 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, w.bumiller@proxmox.com, armbru@redhat.com, qemu-devel@nongnu.org, mreitz@redhat.com, jsnow@redhat.com, dietmar@proxmox.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Type: text/plain; charset="utf-8" Only supported with completion-mode 'grouped', since it relies on a JobTxn to exist. This means that for now it is only available for {drive,blockdev}-backup transactions. Since only one job will be running at a time, bandwidth-limits can be applied effectively. It can also prevent overloading a host's IO capabilities in general. Signed-off-by: Stefan Reiter --- blockdev.c | 25 ++++++++++++++++++++++--- qapi/transaction.json | 6 +++++- 2 files changed, 27 insertions(+), 4 deletions(-) diff --git a/blockdev.c b/blockdev.c index 3848a9c8ab..3691e5e791 100644 --- a/blockdev.c +++ b/blockdev.c @@ -1826,7 +1826,10 @@ static void drive_backup_commit(BlkActionState *comm= on) aio_context_acquire(aio_context); =20 assert(state->job); - job_start(&state->job->job); + + if (!common->txn_props->sequential) { + job_start(&state->job->job); + } =20 aio_context_release(aio_context); } @@ -1927,7 +1930,9 @@ static void blockdev_backup_commit(BlkActionState *co= mmon) aio_context_acquire(aio_context); =20 assert(state->job); - job_start(&state->job->job); + if (!common->txn_props->sequential) { + job_start(&state->job->job); + } =20 aio_context_release(aio_context); } @@ -2303,6 +2308,11 @@ static TransactionProperties *get_transaction_proper= ties( props->completion_mode =3D ACTION_COMPLETION_MODE_INDIVIDUAL; } =20 + if (!props->has_sequential) { + props->has_sequential =3D true; + props->sequential =3D false; + } + return props; } =20 @@ -2328,7 +2338,11 @@ void qmp_transaction(TransactionActionList *dev_list, */ props =3D get_transaction_properties(props); if (props->completion_mode !=3D ACTION_COMPLETION_MODE_INDIVIDUAL) { - block_job_txn =3D job_txn_new(); + block_job_txn =3D props->sequential ? job_txn_new_seq() : job_txn_= new(); + } else if (props->sequential) { + error_setg(errp, "Sequential transaction mode is not supported wit= h " + "completion-mode =3D individual"); + return; } =20 /* drain all i/o before any operations */ @@ -2367,6 +2381,11 @@ void qmp_transaction(TransactionActionList *dev_list, } } =20 + /* jobs in sequential txns don't start themselves on commit */ + if (block_job_txn && props->sequential) { + job_txn_start_seq(block_job_txn); + } + /* success */ goto exit; =20 diff --git a/qapi/transaction.json b/qapi/transaction.json index 15ddebdbc3..4808383088 100644 --- a/qapi/transaction.json +++ b/qapi/transaction.json @@ -84,11 +84,15 @@ # Actions will complete or fail as a group. # See @ActionCompletionMode for details. # +# @sequential: Run the jobs in the transaction one after the other, instead +# of all at once. Not supported for completion-mode 'individu= al'. +# # Since: 2.5 ## { 'struct': 'TransactionProperties', 'data': { - '*completion-mode': 'ActionCompletionMode' + '*completion-mode': 'ActionCompletionMode', + '*sequential': 'bool' } } =20 --=20 2.20.1 From nobody Sun May 19 07:11:47 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1598444315; cv=none; d=zohomail.com; s=zohoarc; b=GKiYXqCu049ngyWyOna+MaG2OsEqc9JjY/ztAWhQ5YLQKHdk+wRQLbkbTTD7n5QyY3+TJYw9hFgr0OGsLNonzEAuirDCwqfKR0dmUi6zo6EYMvtWMi9fGQoYvlG/DkHoCRhunRi9OWunl94ad1+6ZxVACBnv4Hpzf2yryeRlFlU= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1598444315; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=8qpUvndftLJI0tCOmudBGnAuo0hzOAfD2TEeHijwwlg=; b=U/ARFE4AR11374ypM4Tew/uPWk7m04cd5cnq7jNRLrcApBOsadeMXawb6htZdlZaEj5nngzqMBEyrpsoo6ih7i81ji7j2E21FYcsUncfuDRqBZkMXIWAOQW3rnZuSsfgvZ/wOqLbbUbJbygmo37u4WJsT7E4KkhLQMr6gzmlxvg= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1598444315825496.9344699522644; Wed, 26 Aug 2020 05:18:35 -0700 (PDT) Received: from localhost ([::1]:52454 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kAuNu-00056N-Jf for importer@patchew.org; Wed, 26 Aug 2020 08:18:34 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:37514) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kAuMf-0003ax-BV; Wed, 26 Aug 2020 08:17:17 -0400 Received: from proxmox-new.maurer-it.com ([212.186.127.180]:39896) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kAuMb-0004Cl-UM; Wed, 26 Aug 2020 08:17:17 -0400 Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id 98364448C0; Wed, 26 Aug 2020 14:17:01 +0200 (CEST) From: Stefan Reiter To: qemu-block@nongnu.org Subject: [PATCH 3/3] backup: initialize bcs bitmap on job create, not start Date: Wed, 26 Aug 2020 14:13:59 +0200 Message-Id: <20200826121359.15450-4-s.reiter@proxmox.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200826121359.15450-1-s.reiter@proxmox.com> References: <20200826121359.15450-1-s.reiter@proxmox.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=212.186.127.180; envelope-from=s.reiter@proxmox.com; helo=proxmox-new.maurer-it.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/08/26 08:17:01 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, w.bumiller@proxmox.com, armbru@redhat.com, qemu-devel@nongnu.org, mreitz@redhat.com, jsnow@redhat.com, dietmar@proxmox.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Type: text/plain; charset="utf-8" After backup_init_bcs_bitmap the copy-before-write behaviour is active. This way, multiple backup jobs created at once but running in a sequential transaction will still represent the same point in time. Signed-off-by: Stefan Reiter --- I'd imagine this was done on job start for a purpose, so this is potentially wrong. In testing it works fine. Sent along for feedback, since it would be necessary to really make use of = the sequential backup feature (without it, the individual backup jobs would not= have a consistent view of the guest). block/backup.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/block/backup.c b/block/backup.c index 4f13bb20a5..14660eef45 100644 --- a/block/backup.c +++ b/block/backup.c @@ -237,8 +237,6 @@ static int coroutine_fn backup_run(Job *job, Error **er= rp) BackupBlockJob *s =3D container_of(job, BackupBlockJob, common.job); int ret =3D 0; =20 - backup_init_bcs_bitmap(s); - if (s->sync_mode =3D=3D MIRROR_SYNC_MODE_TOP) { int64_t offset =3D 0; int64_t count; @@ -471,6 +469,8 @@ BlockJob *backup_job_create(const char *job_id, BlockDr= iverState *bs, block_job_add_bdrv(&job->common, "target", target, 0, BLK_PERM_ALL, &error_abort); =20 + backup_init_bcs_bitmap(job); + return &job->common; =20 error: --=20 2.20.1