The following changes since commit 248b23735645f7cbb503d9be6f5bf825f2a603ab: Update version for v2.10.0-rc4 release (2017-08-24 17:34:26 +0100) are available in the git repository at: git://github.com/stefanha/qemu.git tags/block-pull-request for you to fetch changes up to 3e4c705212abfe8c9882a00beb2d1466a8a53cec: qcow2: allocate cluster_cache/cluster_data on demand (2017-08-30 18:02:10 +0100) ---------------------------------------------------------------- ---------------------------------------------------------------- Alberto Garcia (8): throttle: Fix wrong variable name in the header documentation throttle: Update the throttle_fix_bucket() documentation throttle: Make throttle_is_valid() a bit less verbose throttle: Remove throttle_fix_bucket() / throttle_unfix_bucket() throttle: Make LeakyBucket.avg and LeakyBucket.max integer types throttle: Make burst_length 64bit and add range checks throttle: Test the valid range of config values misc: Remove unused Error variables Dan Aloni (1): nvme: Fix get/set number of queues feature, again Eduardo Habkost (1): oslib-posix: Print errors before aborting on qemu_alloc_stack() Fred Rolland (1): qemu-doc: Add UUID support in initiator name Stefan Hajnoczi (4): scripts: add argparse module for Python 2.6 compatibility docker.py: Python 2.6 argparse compatibility tests: migration/guestperf Python 2.6 argparse compatibility qcow2: allocate cluster_cache/cluster_data on demand include/qemu/throttle.h | 8 +- block/qcow.c | 12 +- block/qcow2-cluster.c | 17 + block/qcow2.c | 20 +- dump.c | 4 +- hw/block/nvme.c | 4 +- tests/test-throttle.c | 80 +- util/oslib-posix.c | 2 + util/throttle.c | 86 +- COPYING.PYTHON | 270 ++++ qemu-doc.texi | 5 +- scripts/argparse.py | 2406 ++++++++++++++++++++++++++++++++++++ tests/docker/docker.py | 4 +- tests/migration/guestperf/shell.py | 8 +- 14 files changed, 2831 insertions(+), 95 deletions(-) create mode 100644 COPYING.PYTHON create mode 100644 scripts/argparse.py -- 2.13.5 From: Dan Aloni <dan@kernelim.com> The number of queues that should be return by the admin command should: 1) Only mention the number of non-admin queues. 2) It is zero-based, meaning that '0 == one non-admin queue', '1 == two non-admin queues', and so forth. Because our `num_queues` means the number of queues _plus_ the admin queue, then the right calculation for the number returned from the admin command is `num_queues - 2`, combining the two requirements mentioned. The issue was discovered by reducing num_queues from 64 to 8 and running a Linux VM with an SMP parameter larger than that (e.g. 22). It tries to utilize all queues, and therefore fails with an invalid queue number when trying to queue I/Os on the last queue. Signed-off-by: Dan Aloni <dan@kernelim.com> CC: Alex Friedman <alex@e8storage.com> CC: Keith Busch <keith.busch@intel.com> CC: Stefan Hajnoczi <stefanha@redhat.com> Reviewed-by: Keith Busch <keith.busch@intel.com> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com> --- hw/block/nvme.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/hw/block/nvme.c b/hw/block/nvme.c index XXXXXXX..XXXXXXX 100644 --- a/hw/block/nvme.c +++ b/hw/block/nvme.c @@ -XXX,XX +XXX,XX @@ static uint16_t nvme_get_feature(NvmeCtrl *n, NvmeCmd *cmd, NvmeRequest *req) result = blk_enable_write_cache(n->conf.blk); break; case NVME_NUMBER_OF_QUEUES: - result = cpu_to_le32((n->num_queues - 1) | ((n->num_queues - 1) << 16)); + result = cpu_to_le32((n->num_queues - 2) | ((n->num_queues - 2) << 16)); break; default: return NVME_INVALID_FIELD | NVME_DNR; @@ -XXX,XX +XXX,XX @@ static uint16_t nvme_set_feature(NvmeCtrl *n, NvmeCmd *cmd, NvmeRequest *req) break; case NVME_NUMBER_OF_QUEUES: req->cqe.result = - cpu_to_le32((n->num_queues - 1) | ((n->num_queues - 1) << 16)); + cpu_to_le32((n->num_queues - 2) | ((n->num_queues - 2) << 16)); break; default: return NVME_INVALID_FIELD | NVME_DNR; -- 2.13.5 From: Alberto Garcia <berto@igalia.com> The level of the burst bucket is stored in bkt.burst_level, not bkt.burst_length. Signed-off-by: Alberto Garcia <berto@igalia.com> Reviewed-by: Manos Pitsidianakis <el13635@mail.ntua.gr> Message-id: 49aab2711d02f285567f3b3b13a113847af33812.1503580370.git.berto@igalia.com Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com> --- include/qemu/throttle.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/include/qemu/throttle.h b/include/qemu/throttle.h index XXXXXXX..XXXXXXX 100644 --- a/include/qemu/throttle.h +++ b/include/qemu/throttle.h @@ -XXX,XX +XXX,XX @@ typedef enum { * - The bkt.avg rate does not apply until the bucket is full, * allowing the user to do bursts until then. The I/O limit during * bursts is bkt.max. To enforce this limit we keep an additional - * bucket in bkt.burst_length that leaks at a rate of bkt.max units + * bucket in bkt.burst_level that leaks at a rate of bkt.max units * per second. * * - Because of all of the above, the user can perform I/O at a -- 2.13.5 From: Alberto Garcia <berto@igalia.com> The way the throttling algorithm works is that requests start being throttled once the bucket level exceeds the burst limit. When we get there the bucket leaks at the level set by the user (bkt->avg), and that leak rate is what prevents guest I/O from exceeding the desired limit. If we don't allow bursts (i.e. bkt->max == 0) then we can start throttling requests immediately. The problem with keeping the threshold at 0 is that it only allows one request at a time, and as soon as there's a bit of I/O from the guest every other request will be throttled and performance will suffer considerably. That can even make the guest unable to reach the throttle limit if that limit is high enough, and that happens regardless of the block scheduler used by the guest. Increasing that threshold gives flexibility to the guest, allowing it to perform short bursts of I/O before being throttled. Increasing the threshold too much does not make a difference in the long run (because it's the leak rate what defines the actual throughput) but it does allow the guest to perform longer initial bursts and exceed the throttle limit for a short while. A burst value of bkt->avg / 10 allows the guest to perform 100ms' worth of I/O at the target rate without being throttled. Signed-off-by: Alberto Garcia <berto@igalia.com> Message-id: 31aae6645f0d1fbf3860fb2b528b757236f0c0a7.1503580370.git.berto@igalia.com Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com> --- util/throttle.c | 11 +++-------- 1 file changed, 3 insertions(+), 8 deletions(-) diff --git a/util/throttle.c b/util/throttle.c index XXXXXXX..XXXXXXX 100644 --- a/util/throttle.c +++ b/util/throttle.c @@ -XXX,XX +XXX,XX @@ static void throttle_fix_bucket(LeakyBucket *bkt) /* zero bucket level */ bkt->level = bkt->burst_level = 0; - /* The following is done to cope with the Linux CFQ block scheduler - * which regroup reads and writes by block of 100ms in the guest. - * When they are two process one making reads and one making writes cfq - * make a pattern looking like the following: - * WWWWWWWWWWWRRRRRRRRRRRRRRWWWWWWWWWWWWWwRRRRRRRRRRRRRRRRR - * Having a max burst value of 100ms of the average will help smooth the - * throttling - */ + /* If bkt->max is 0 we still want to allow short bursts of I/O + * from the guest, otherwise every other request will be throttled + * and performance will suffer considerably. */ min = bkt->avg / 10; if (bkt->avg && !bkt->max) { bkt->max = min; -- 2.13.5 From: Alberto Garcia <berto@igalia.com> Use a pointer to the bucket instead of repeating cfg->buckets[i] all the time. This makes the code more concise and will help us expand the checks later and save a few line breaks. Signed-off-by: Alberto Garcia <berto@igalia.com> Message-id: 763ffc40a26b17d54cf93f5a999e4656049fcf0c.1503580370.git.berto@igalia.com Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com> --- util/throttle.c | 15 +++++++-------- 1 file changed, 7 insertions(+), 8 deletions(-) diff --git a/util/throttle.c b/util/throttle.c index XXXXXXX..XXXXXXX 100644 --- a/util/throttle.c +++ b/util/throttle.c @@ -XXX,XX +XXX,XX @@ bool throttle_is_valid(ThrottleConfig *cfg, Error **errp) } for (i = 0; i < BUCKETS_COUNT; i++) { - if (cfg->buckets[i].avg < 0 || - cfg->buckets[i].max < 0 || - cfg->buckets[i].avg > THROTTLE_VALUE_MAX || - cfg->buckets[i].max > THROTTLE_VALUE_MAX) { + LeakyBucket *bkt = &cfg->buckets[i]; + if (bkt->avg < 0 || bkt->max < 0 || + bkt->avg > THROTTLE_VALUE_MAX || bkt->max > THROTTLE_VALUE_MAX) { error_setg(errp, "bps/iops/max values must be within [0, %lld]", THROTTLE_VALUE_MAX); return false; } - if (!cfg->buckets[i].burst_length) { + if (!bkt->burst_length) { error_setg(errp, "the burst length cannot be 0"); return false; } - if (cfg->buckets[i].burst_length > 1 && !cfg->buckets[i].max) { + if (bkt->burst_length > 1 && !bkt->max) { error_setg(errp, "burst length set without burst rate"); return false; } - if (cfg->buckets[i].max && !cfg->buckets[i].avg) { + if (bkt->max && !bkt->avg) { error_setg(errp, "bps_max/iops_max require corresponding" " bps/iops values"); return false; } - if (cfg->buckets[i].max && cfg->buckets[i].max < cfg->buckets[i].avg) { + if (bkt->max && bkt->max < bkt->avg) { error_setg(errp, "bps_max/iops_max cannot be lower than bps/iops"); return false; } -- 2.13.5 From: Alberto Garcia <berto@igalia.com> The throttling code can change internally the value of bkt->max if it hasn't been set by the user. The problem with this is that if we want to retrieve the original value we have to undo this change first. This is ugly and unnecessary: this patch removes the throttle_fix_bucket() and throttle_unfix_bucket() functions completely and moves the logic to throttle_compute_wait(). Signed-off-by: Alberto Garcia <berto@igalia.com> Reviewed-by: Manos Pitsidianakis <el13635@mail.ntua.gr> Message-id: 5b0b9e1ac6eb208d709eddc7b09e7669a523bff3.1503580370.git.berto@igalia.com Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com> --- util/throttle.c | 62 +++++++++++++++++++++------------------------------------ 1 file changed, 23 insertions(+), 39 deletions(-) diff --git a/util/throttle.c b/util/throttle.c index XXXXXXX..XXXXXXX 100644 --- a/util/throttle.c +++ b/util/throttle.c @@ -XXX,XX +XXX,XX @@ static int64_t throttle_do_compute_wait(double limit, double extra) int64_t throttle_compute_wait(LeakyBucket *bkt) { double extra; /* the number of extra units blocking the io */ + double bucket_size; /* I/O before throttling to bkt->avg */ + double burst_bucket_size; /* Before throttling to bkt->max */ if (!bkt->avg) { return 0; } - /* If the bucket is full then we have to wait */ - extra = bkt->level - bkt->max * bkt->burst_length; + if (!bkt->max) { + /* If bkt->max is 0 we still want to allow short bursts of I/O + * from the guest, otherwise every other request will be throttled + * and performance will suffer considerably. */ + bucket_size = bkt->avg / 10; + burst_bucket_size = 0; + } else { + /* If we have a burst limit then we have to wait until all I/O + * at burst rate has finished before throttling to bkt->avg */ + bucket_size = bkt->max * bkt->burst_length; + burst_bucket_size = bkt->max / 10; + } + + /* If the main bucket is full then we have to wait */ + extra = bkt->level - bucket_size; if (extra > 0) { return throttle_do_compute_wait(bkt->avg, extra); } - /* If the bucket is not full yet we have to make sure that we - * fulfill the goal of bkt->max units per second. */ + /* If the main bucket is not full yet we still have to check the + * burst bucket in order to enforce the burst limit */ if (bkt->burst_length > 1) { - /* We use 1/10 of the max value to smooth the throttling. - * See throttle_fix_bucket() for more details. */ - extra = bkt->burst_level - bkt->max / 10; + extra = bkt->burst_level - burst_bucket_size; if (extra > 0) { return throttle_do_compute_wait(bkt->max, extra); } @@ -XXX,XX +XXX,XX @@ bool throttle_is_valid(ThrottleConfig *cfg, Error **errp) return true; } -/* fix bucket parameters */ -static void throttle_fix_bucket(LeakyBucket *bkt) -{ - double min; - - /* zero bucket level */ - bkt->level = bkt->burst_level = 0; - - /* If bkt->max is 0 we still want to allow short bursts of I/O - * from the guest, otherwise every other request will be throttled - * and performance will suffer considerably. */ - min = bkt->avg / 10; - if (bkt->avg && !bkt->max) { - bkt->max = min; - } -} - -/* undo internal bucket parameter changes (see throttle_fix_bucket()) */ -static void throttle_unfix_bucket(LeakyBucket *bkt) -{ - if (bkt->max < bkt->avg) { - bkt->max = 0; - } -} - /* Used to configure the throttle * * @ts: the throttle state we are working on @@ -XXX,XX +XXX,XX @@ void throttle_config(ThrottleState *ts, ts->cfg = *cfg; + /* Zero bucket level */ for (i = 0; i < BUCKETS_COUNT; i++) { - throttle_fix_bucket(&ts->cfg.buckets[i]); + ts->cfg.buckets[i].level = 0; + ts->cfg.buckets[i].burst_level = 0; } ts->previous_leak = qemu_clock_get_ns(clock_type); @@ -XXX,XX +XXX,XX @@ void throttle_config(ThrottleState *ts, */ void throttle_get_config(ThrottleState *ts, ThrottleConfig *cfg) { - int i; - *cfg = ts->cfg; - - for (i = 0; i < BUCKETS_COUNT; i++) { - throttle_unfix_bucket(&cfg->buckets[i]); - } } -- 2.13.5 From: Alberto Garcia <berto@igalia.com> Both the throttling limits set with the throttling.iops-* and throttling.bps-* options and their QMP equivalents defined in the BlockIOThrottle struct are integer values. Those limits are also reported in the BlockDeviceInfo struct and they are integers there as well. Therefore there's no reason to store them internally as double and do the conversion everytime we're setting or querying them, so this patch uses uint64_t for those types. Let's also use an unsigned type because we don't allow negative values anyway. LeakyBucket.level and LeakyBucket.burst_level do however remain double because their value changes depending on the fraction of time elapsed since the previous I/O operation. Signed-off-by: Alberto Garcia <berto@igalia.com> Message-id: f29b840422767b5be2c41c2dfdbbbf6c5f8fedf8.1503580370.git.berto@igalia.com Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com> --- include/qemu/throttle.h | 4 ++-- tests/test-throttle.c | 3 ++- util/throttle.c | 7 +++---- 3 files changed, 7 insertions(+), 7 deletions(-) diff --git a/include/qemu/throttle.h b/include/qemu/throttle.h index XXXXXXX..XXXXXXX 100644 --- a/include/qemu/throttle.h +++ b/include/qemu/throttle.h @@ -XXX,XX +XXX,XX @@ typedef enum { */ typedef struct LeakyBucket { - double avg; /* average goal in units per second */ - double max; /* leaky bucket max burst in units */ + uint64_t avg; /* average goal in units per second */ + uint64_t max; /* leaky bucket max burst in units */ double level; /* bucket level in units */ double burst_level; /* bucket level in units (for computing bursts) */ unsigned burst_length; /* max length of the burst period, in seconds */ diff --git a/tests/test-throttle.c b/tests/test-throttle.c index XXXXXXX..XXXXXXX 100644 --- a/tests/test-throttle.c +++ b/tests/test-throttle.c @@ -XXX,XX +XXX,XX @@ static void test_enabled(void) for (i = 0; i < BUCKETS_COUNT; i++) { throttle_config_init(&cfg); set_cfg_value(false, i, 150); + g_assert(throttle_is_valid(&cfg, NULL)); g_assert(throttle_enabled(&cfg)); } for (i = 0; i < BUCKETS_COUNT; i++) { throttle_config_init(&cfg); set_cfg_value(false, i, -150); - g_assert(!throttle_enabled(&cfg)); + g_assert(!throttle_is_valid(&cfg, NULL)); } } diff --git a/util/throttle.c b/util/throttle.c index XXXXXXX..XXXXXXX 100644 --- a/util/throttle.c +++ b/util/throttle.c @@ -XXX,XX +XXX,XX @@ int64_t throttle_compute_wait(LeakyBucket *bkt) /* If bkt->max is 0 we still want to allow short bursts of I/O * from the guest, otherwise every other request will be throttled * and performance will suffer considerably. */ - bucket_size = bkt->avg / 10; + bucket_size = (double) bkt->avg / 10; burst_bucket_size = 0; } else { /* If we have a burst limit then we have to wait until all I/O * at burst rate has finished before throttling to bkt->avg */ bucket_size = bkt->max * bkt->burst_length; - burst_bucket_size = bkt->max / 10; + burst_bucket_size = (double) bkt->max / 10; } /* If the main bucket is full then we have to wait */ @@ -XXX,XX +XXX,XX @@ bool throttle_is_valid(ThrottleConfig *cfg, Error **errp) for (i = 0; i < BUCKETS_COUNT; i++) { LeakyBucket *bkt = &cfg->buckets[i]; - if (bkt->avg < 0 || bkt->max < 0 || - bkt->avg > THROTTLE_VALUE_MAX || bkt->max > THROTTLE_VALUE_MAX) { + if (bkt->avg > THROTTLE_VALUE_MAX || bkt->max > THROTTLE_VALUE_MAX) { error_setg(errp, "bps/iops/max values must be within [0, %lld]", THROTTLE_VALUE_MAX); return false; -- 2.13.5 From: Alberto Garcia <berto@igalia.com> LeakyBucket.burst_length is defined as an unsigned integer but the code never checks for overflows and it only makes sure that the value is not 0. In practice this means that the user can set something like throttling.iops-total-max-length=4294967300 despite being larger than UINT_MAX and the final value after casting to unsigned int will be 4. This patch changes the data type to uint64_t. This does not increase the storage size of LeakyBucket, and allows us to assign the value directly from qemu_opt_get_number() or BlockIOThrottle and then do the checks directly in throttle_is_valid(). The value of burst_length does not have a specific upper limit, but since the bucket size is defined by max * burst_length we have to prevent overflows. Instead of going for UINT64_MAX or something similar this patch reuses THROTTLE_VALUE_MAX, which allows I/O bursts of 1 GiB/s for 10 days in a row. Signed-off-by: Alberto Garcia <berto@igalia.com> Message-id: 1b2e3049803f71cafb2e1fa1be4fb47147a0d398.1503580370.git.berto@igalia.com Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com> --- include/qemu/throttle.h | 2 +- util/throttle.c | 5 +++++ 2 files changed, 6 insertions(+), 1 deletion(-) diff --git a/include/qemu/throttle.h b/include/qemu/throttle.h index XXXXXXX..XXXXXXX 100644 --- a/include/qemu/throttle.h +++ b/include/qemu/throttle.h @@ -XXX,XX +XXX,XX @@ typedef struct LeakyBucket { uint64_t max; /* leaky bucket max burst in units */ double level; /* bucket level in units */ double burst_level; /* bucket level in units (for computing bursts) */ - unsigned burst_length; /* max length of the burst period, in seconds */ + uint64_t burst_length; /* max length of the burst period, in seconds */ } LeakyBucket; /* The following structure is used to configure a ThrottleState diff --git a/util/throttle.c b/util/throttle.c index XXXXXXX..XXXXXXX 100644 --- a/util/throttle.c +++ b/util/throttle.c @@ -XXX,XX +XXX,XX @@ bool throttle_is_valid(ThrottleConfig *cfg, Error **errp) return false; } + if (bkt->max && bkt->burst_length > THROTTLE_VALUE_MAX / bkt->max) { + error_setg(errp, "burst length too high for this burst rate"); + return false; + } + if (bkt->max && !bkt->avg) { error_setg(errp, "bps_max/iops_max require corresponding" " bps/iops values"); -- 2.13.5 From: Alberto Garcia <berto@igalia.com> Signed-off-by: Alberto Garcia <berto@igalia.com> Message-id: a57dd6274e1b6dc9c28769fec4c7ea543be5c5e3.1503580370.git.berto@igalia.com Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com> --- tests/test-throttle.c | 77 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 77 insertions(+) diff --git a/tests/test-throttle.c b/tests/test-throttle.c index XXXXXXX..XXXXXXX 100644 --- a/tests/test-throttle.c +++ b/tests/test-throttle.c @@ -XXX,XX +XXX,XX @@ static void test_is_valid(void) test_is_valid_for_value(1, true); } +static void test_ranges(void) +{ + int i; + + for (i = 0; i < BUCKETS_COUNT; i++) { + LeakyBucket *b = &cfg.buckets[i]; + throttle_config_init(&cfg); + + /* avg = 0 means throttling is disabled, but the config is valid */ + b->avg = 0; + g_assert(throttle_is_valid(&cfg, NULL)); + g_assert(!throttle_enabled(&cfg)); + + /* These are valid configurations (values <= THROTTLE_VALUE_MAX) */ + b->avg = 1; + g_assert(throttle_is_valid(&cfg, NULL)); + + b->avg = THROTTLE_VALUE_MAX; + g_assert(throttle_is_valid(&cfg, NULL)); + + b->avg = THROTTLE_VALUE_MAX; + b->max = THROTTLE_VALUE_MAX; + g_assert(throttle_is_valid(&cfg, NULL)); + + /* Values over THROTTLE_VALUE_MAX are not allowed */ + b->avg = THROTTLE_VALUE_MAX + 1; + g_assert(!throttle_is_valid(&cfg, NULL)); + + b->avg = THROTTLE_VALUE_MAX; + b->max = THROTTLE_VALUE_MAX + 1; + g_assert(!throttle_is_valid(&cfg, NULL)); + + /* burst_length must be between 1 and THROTTLE_VALUE_MAX */ + b->avg = 1; + b->max = 1; + b->burst_length = 0; + g_assert(!throttle_is_valid(&cfg, NULL)); + + b->avg = 1; + b->max = 1; + b->burst_length = 1; + g_assert(throttle_is_valid(&cfg, NULL)); + + b->avg = 1; + b->max = 1; + b->burst_length = THROTTLE_VALUE_MAX; + g_assert(throttle_is_valid(&cfg, NULL)); + + b->avg = 1; + b->max = 1; + b->burst_length = THROTTLE_VALUE_MAX + 1; + g_assert(!throttle_is_valid(&cfg, NULL)); + + /* burst_length * max cannot exceed THROTTLE_VALUE_MAX */ + b->avg = 1; + b->max = 2; + b->burst_length = THROTTLE_VALUE_MAX / 2; + g_assert(throttle_is_valid(&cfg, NULL)); + + b->avg = 1; + b->max = 3; + b->burst_length = THROTTLE_VALUE_MAX / 2; + g_assert(!throttle_is_valid(&cfg, NULL)); + + b->avg = 1; + b->max = THROTTLE_VALUE_MAX; + b->burst_length = 1; + g_assert(throttle_is_valid(&cfg, NULL)); + + b->avg = 1; + b->max = THROTTLE_VALUE_MAX; + b->burst_length = 2; + g_assert(!throttle_is_valid(&cfg, NULL)); + } +} + static void test_max_is_missing_limit(void) { int i; @@ -XXX,XX +XXX,XX @@ int main(int argc, char **argv) g_test_add_func("/throttle/config/enabled", test_enabled); g_test_add_func("/throttle/config/conflicting", test_conflicting_config); g_test_add_func("/throttle/config/is_valid", test_is_valid); + g_test_add_func("/throttle/config/ranges", test_ranges); g_test_add_func("/throttle/config/max", test_max_is_missing_limit); g_test_add_func("/throttle/config/iops_size", test_iops_size_is_missing_limit); -- 2.13.5 From: Eduardo Habkost <ehabkost@redhat.com> If QEMU is running on a system that's out of memory and mmap() fails, QEMU aborts with no error message at all, making it hard to debug the reason for the failure. Add perror() calls that will print error information before aborting. Signed-off-by: Eduardo Habkost <ehabkost@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Message-id: 20170829212053.6003-1-ehabkost@redhat.com Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com> --- util/oslib-posix.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/util/oslib-posix.c b/util/oslib-posix.c index XXXXXXX..XXXXXXX 100644 --- a/util/oslib-posix.c +++ b/util/oslib-posix.c @@ -XXX,XX +XXX,XX @@ void *qemu_alloc_stack(size_t *sz) ptr = mmap(NULL, *sz, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); if (ptr == MAP_FAILED) { + perror("failed to allocate memory for stack"); abort(); } @@ -XXX,XX +XXX,XX @@ void *qemu_alloc_stack(size_t *sz) guardpage = ptr; #endif if (mprotect(guardpage, pagesz, PROT_NONE) != 0) { + perror("failed to set up stack guard page"); abort(); } -- 2.13.5 From: Alberto Garcia <berto@igalia.com> There's a few cases which we're passing an Error pointer to a function only to discard it immediately afterwards without checking it. In these cases we can simply remove the variable and pass NULL instead. Signed-off-by: Alberto Garcia <berto@igalia.com> Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Reviewed-by: Eric Blake <eblake@redhat.com> Message-id: 20170829120836.16091-1-berto@igalia.com Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com> --- block/qcow.c | 12 +++--------- block/qcow2.c | 8 ++------ dump.c | 4 +--- 3 files changed, 6 insertions(+), 18 deletions(-) diff --git a/block/qcow.c b/block/qcow.c index XXXXXXX..XXXXXXX 100644 --- a/block/qcow.c +++ b/block/qcow.c @@ -XXX,XX +XXX,XX @@ static uint64_t get_cluster_offset(BlockDriverState *bs, start_sect = (offset & ~(s->cluster_size - 1)) >> 9; for(i = 0; i < s->cluster_sectors; i++) { if (i < n_start || i >= n_end) { - Error *err = NULL; memset(s->cluster_data, 0x00, 512); if (qcrypto_block_encrypt(s->crypto, start_sect + i, s->cluster_data, BDRV_SECTOR_SIZE, - &err) < 0) { - error_free(err); + NULL) < 0) { errno = EIO; return -1; } @@ -XXX,XX +XXX,XX @@ static coroutine_fn int qcow_co_readv(BlockDriverState *bs, int64_t sector_num, QEMUIOVector hd_qiov; uint8_t *buf; void *orig_buf; - Error *err = NULL; if (qiov->niov > 1) { buf = orig_buf = qemu_try_blockalign(bs, qiov->size); @@ -XXX,XX +XXX,XX @@ static coroutine_fn int qcow_co_readv(BlockDriverState *bs, int64_t sector_num, if (bs->encrypted) { assert(s->crypto); if (qcrypto_block_decrypt(s->crypto, sector_num, buf, - n * BDRV_SECTOR_SIZE, &err) < 0) { + n * BDRV_SECTOR_SIZE, NULL) < 0) { goto fail; } } @@ -XXX,XX +XXX,XX @@ done: return ret; fail: - error_free(err); ret = -EIO; goto done; } @@ -XXX,XX +XXX,XX @@ static coroutine_fn int qcow_co_writev(BlockDriverState *bs, int64_t sector_num, break; } if (bs->encrypted) { - Error *err = NULL; assert(s->crypto); if (qcrypto_block_encrypt(s->crypto, sector_num, buf, - n * BDRV_SECTOR_SIZE, &err) < 0) { - error_free(err); + n * BDRV_SECTOR_SIZE, NULL) < 0) { ret = -EIO; break; } diff --git a/block/qcow2.c b/block/qcow2.c index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -XXX,XX +XXX,XX @@ static coroutine_fn int qcow2_co_preadv(BlockDriverState *bs, uint64_t offset, assert(s->crypto); assert((offset & (BDRV_SECTOR_SIZE - 1)) == 0); assert((cur_bytes & (BDRV_SECTOR_SIZE - 1)) == 0); - Error *err = NULL; if (qcrypto_block_decrypt(s->crypto, (s->crypt_physical_offset ? cluster_offset + offset_in_cluster : offset) >> BDRV_SECTOR_BITS, cluster_data, cur_bytes, - &err) < 0) { - error_free(err); + NULL) < 0) { ret = -EIO; goto fail; } @@ -XXX,XX +XXX,XX @@ static coroutine_fn int qcow2_co_pwritev(BlockDriverState *bs, uint64_t offset, qemu_iovec_concat(&hd_qiov, qiov, bytes_done, cur_bytes); if (bs->encrypted) { - Error *err = NULL; assert(s->crypto); if (!cluster_data) { cluster_data = qemu_try_blockalign(bs->file->bs, @@ -XXX,XX +XXX,XX @@ static coroutine_fn int qcow2_co_pwritev(BlockDriverState *bs, uint64_t offset, cluster_offset + offset_in_cluster : offset) >> BDRV_SECTOR_BITS, cluster_data, - cur_bytes, &err) < 0) { - error_free(err); + cur_bytes, NULL) < 0) { ret = -EIO; goto fail; } diff --git a/dump.c b/dump.c index XXXXXXX..XXXXXXX 100644 --- a/dump.c +++ b/dump.c @@ -XXX,XX +XXX,XX @@ static void dump_process(DumpState *s, Error **errp) static void *dump_thread(void *data) { - Error *err = NULL; DumpState *s = (DumpState *)data; - dump_process(s, &err); - error_free(err); + dump_process(s, NULL); return NULL; } -- 2.13.5 The minimum Python version supported by QEMU is 2.6. The argparse standard library module was only added in Python 2.7. Many scripts would like to use argparse because it supports command-line sub-commands. This patch adds argparse. See the top of argparse.py for details. Suggested-by: Daniel P. Berrange <berrange@redhat.com> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com> Acked-by: John Snow <jsnow@redhat.com> Message-id: 20170825155732.15665-2-stefanha@redhat.com Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com> --- COPYING.PYTHON | 270 ++++++ scripts/argparse.py | 2406 +++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 2676 insertions(+) create mode 100644 COPYING.PYTHON create mode 100644 scripts/argparse.py diff --git a/COPYING.PYTHON b/COPYING.PYTHON new file mode 100644 index XXXXXXX..XXXXXXX --- /dev/null +++ b/COPYING.PYTHON @@ -XXX,XX +XXX,XX @@ +A. HISTORY OF THE SOFTWARE +========================== + +Python was created in the early 1990s by Guido van Rossum at Stichting +Mathematisch Centrum (CWI, see http://www.cwi.nl) in the Netherlands +as a successor of a language called ABC. Guido remains Python's +principal author, although it includes many contributions from others. + +In 1995, Guido continued his work on Python at the Corporation for +National Research Initiatives (CNRI, see http://www.cnri.reston.va.us) +in Reston, Virginia where he released several versions of the +software. + +In May 2000, Guido and the Python core development team moved to +BeOpen.com to form the BeOpen PythonLabs team. In October of the same +year, the PythonLabs team moved to Digital Creations (now Zope +Corporation, see http://www.zope.com). In 2001, the Python Software +Foundation (PSF, see http://www.python.org/psf/) was formed, a +non-profit organization created specifically to own Python-related +Intellectual Property. Zope Corporation is a sponsoring member of +the PSF. + +All Python releases are Open Source (see http://www.opensource.org for +the Open Source Definition). Historically, most, but not all, Python +releases have also been GPL-compatible; the table below summarizes +the various releases. + + Release Derived Year Owner GPL- + from compatible? (1) + + 0.9.0 thru 1.2 1991-1995 CWI yes + 1.3 thru 1.5.2 1.2 1995-1999 CNRI yes + 1.6 1.5.2 2000 CNRI no + 2.0 1.6 2000 BeOpen.com no + 1.6.1 1.6 2001 CNRI yes (2) + 2.1 2.0+1.6.1 2001 PSF no + 2.0.1 2.0+1.6.1 2001 PSF yes + 2.1.1 2.1+2.0.1 2001 PSF yes + 2.2 2.1.1 2001 PSF yes + 2.1.2 2.1.1 2002 PSF yes + 2.1.3 2.1.2 2002 PSF yes + 2.2.1 2.2 2002 PSF yes + 2.2.2 2.2.1 2002 PSF yes + 2.2.3 2.2.2 2003 PSF yes + 2.3 2.2.2 2002-2003 PSF yes + 2.3.1 2.3 2002-2003 PSF yes + 2.3.2 2.3.1 2002-2003 PSF yes + 2.3.3 2.3.2 2002-2003 PSF yes + 2.3.4 2.3.3 2004 PSF yes + 2.3.5 2.3.4 2005 PSF yes + 2.4 2.3 2004 PSF yes + 2.4.1 2.4 2005 PSF yes + 2.4.2 2.4.1 2005 PSF yes + 2.4.3 2.4.2 2006 PSF yes + 2.5 2.4 2006 PSF yes + 2.7 2.6 2010 PSF yes + +Footnotes: + +(1) GPL-compatible doesn't mean that we're distributing Python under + the GPL. All Python licenses, unlike the GPL, let you distribute + a modified version without making your changes open source. The + GPL-compatible licenses make it possible to combine Python with + other software that is released under the GPL; the others don't. + +(2) According to Richard Stallman, 1.6.1 is not GPL-compatible, + because its license has a choice of law clause. According to + CNRI, however, Stallman's lawyer has told CNRI's lawyer that 1.6.1 + is "not incompatible" with the GPL. + +Thanks to the many outside volunteers who have worked under Guido's +direction to make these releases possible. + + +B. TERMS AND CONDITIONS FOR ACCESSING OR OTHERWISE USING PYTHON +=============================================================== + +PYTHON SOFTWARE FOUNDATION LICENSE VERSION 2 +-------------------------------------------- + +1. This LICENSE AGREEMENT is between the Python Software Foundation +("PSF"), and the Individual or Organization ("Licensee") accessing and +otherwise using this software ("Python") in source or binary form and +its associated documentation. + +2. Subject to the terms and conditions of this License Agreement, PSF +hereby grants Licensee a nonexclusive, royalty-free, world-wide +license to reproduce, analyze, test, perform and/or display publicly, +prepare derivative works, distribute, and otherwise use Python +alone or in any derivative version, provided, however, that PSF's +License Agreement and PSF's notice of copyright, i.e., "Copyright (c) +2001, 2002, 2003, 2004, 2005, 2006 Python Software Foundation; All Rights +Reserved" are retained in Python alone or in any derivative version +prepared by Licensee. + +3. In the event Licensee prepares a derivative work that is based on +or incorporates Python or any part thereof, and wants to make +the derivative work available to others as provided herein, then +Licensee hereby agrees to include in any such work a brief summary of +the changes made to Python. + +4. PSF is making Python available to Licensee on an "AS IS" +basis. PSF MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR +IMPLIED. BY WAY OF EXAMPLE, BUT NOT LIMITATION, PSF MAKES NO AND +DISCLAIMS ANY REPRESENTATION OR WARRANTY OF MERCHANTABILITY OR FITNESS +FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF PYTHON WILL NOT +INFRINGE ANY THIRD PARTY RIGHTS. + +5. PSF SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF PYTHON +FOR ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR LOSS AS +A RESULT OF MODIFYING, DISTRIBUTING, OR OTHERWISE USING PYTHON, +OR ANY DERIVATIVE THEREOF, EVEN IF ADVISED OF THE POSSIBILITY THEREOF. + +6. This License Agreement will automatically terminate upon a material +breach of its terms and conditions. + +7. Nothing in this License Agreement shall be deemed to create any +relationship of agency, partnership, or joint venture between PSF and +Licensee. This License Agreement does not grant permission to use PSF +trademarks or trade name in a trademark sense to endorse or promote +products or services of Licensee, or any third party. + +8. By copying, installing or otherwise using Python, Licensee +agrees to be bound by the terms and conditions of this License +Agreement. + + +BEOPEN.COM LICENSE AGREEMENT FOR PYTHON 2.0 +------------------------------------------- + +BEOPEN PYTHON OPEN SOURCE LICENSE AGREEMENT VERSION 1 + +1. This LICENSE AGREEMENT is between BeOpen.com ("BeOpen"), having an +office at 160 Saratoga Avenue, Santa Clara, CA 95051, and the +Individual or Organization ("Licensee") accessing and otherwise using +this software in source or binary form and its associated +documentation ("the Software"). + +2. Subject to the terms and conditions of this BeOpen Python License +Agreement, BeOpen hereby grants Licensee a non-exclusive, +royalty-free, world-wide license to reproduce, analyze, test, perform +and/or display publicly, prepare derivative works, distribute, and +otherwise use the Software alone or in any derivative version, +provided, however, that the BeOpen Python License is retained in the +Software, alone or in any derivative version prepared by Licensee. + +3. BeOpen is making the Software available to Licensee on an "AS IS" +basis. BEOPEN MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR +IMPLIED. BY WAY OF EXAMPLE, BUT NOT LIMITATION, BEOPEN MAKES NO AND +DISCLAIMS ANY REPRESENTATION OR WARRANTY OF MERCHANTABILITY OR FITNESS +FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF THE SOFTWARE WILL NOT +INFRINGE ANY THIRD PARTY RIGHTS. + +4. BEOPEN SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF THE +SOFTWARE FOR ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR LOSS +AS A RESULT OF USING, MODIFYING OR DISTRIBUTING THE SOFTWARE, OR ANY +DERIVATIVE THEREOF, EVEN IF ADVISED OF THE POSSIBILITY THEREOF. + +5. This License Agreement will automatically terminate upon a material +breach of its terms and conditions. + +6. This License Agreement shall be governed by and interpreted in all +respects by the law of the State of California, excluding conflict of +law provisions. Nothing in this License Agreement shall be deemed to +create any relationship of agency, partnership, or joint venture +between BeOpen and Licensee. This License Agreement does not grant +permission to use BeOpen trademarks or trade names in a trademark +sense to endorse or promote products or services of Licensee, or any +third party. As an exception, the "BeOpen Python" logos available at +http://www.pythonlabs.com/logos.html may be used according to the +permissions granted on that web page. + +7. By copying, installing or otherwise using the software, Licensee +agrees to be bound by the terms and conditions of this License +Agreement. + + +CNRI LICENSE AGREEMENT FOR PYTHON 1.6.1 +--------------------------------------- + +1. This LICENSE AGREEMENT is between the Corporation for National +Research Initiatives, having an office at 1895 Preston White Drive, +Reston, VA 20191 ("CNRI"), and the Individual or Organization +("Licensee") accessing and otherwise using Python 1.6.1 software in +source or binary form and its associated documentation. + +2. Subject to the terms and conditions of this License Agreement, CNRI +hereby grants Licensee a nonexclusive, royalty-free, world-wide +license to reproduce, analyze, test, perform and/or display publicly, +prepare derivative works, distribute, and otherwise use Python 1.6.1 +alone or in any derivative version, provided, however, that CNRI's +License Agreement and CNRI's notice of copyright, i.e., "Copyright (c) +1995-2001 Corporation for National Research Initiatives; All Rights +Reserved" are retained in Python 1.6.1 alone or in any derivative +version prepared by Licensee. Alternately, in lieu of CNRI's License +Agreement, Licensee may substitute the following text (omitting the +quotes): "Python 1.6.1 is made available subject to the terms and +conditions in CNRI's License Agreement. This Agreement together with +Python 1.6.1 may be located on the Internet using the following +unique, persistent identifier (known as a handle): 1895.22/1013. This +Agreement may also be obtained from a proxy server on the Internet +using the following URL: http://hdl.handle.net/1895.22/1013". + +3. In the event Licensee prepares a derivative work that is based on +or incorporates Python 1.6.1 or any part thereof, and wants to make +the derivative work available to others as provided herein, then +Licensee hereby agrees to include in any such work a brief summary of +the changes made to Python 1.6.1. + +4. CNRI is making Python 1.6.1 available to Licensee on an "AS IS" +basis. CNRI MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR +IMPLIED. BY WAY OF EXAMPLE, BUT NOT LIMITATION, CNRI MAKES NO AND +DISCLAIMS ANY REPRESENTATION OR WARRANTY OF MERCHANTABILITY OR FITNESS +FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF PYTHON 1.6.1 WILL NOT +INFRINGE ANY THIRD PARTY RIGHTS. + +5. CNRI SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF PYTHON +1.6.1 FOR ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR LOSS AS +A RESULT OF MODIFYING, DISTRIBUTING, OR OTHERWISE USING PYTHON 1.6.1, +OR ANY DERIVATIVE THEREOF, EVEN IF ADVISED OF THE POSSIBILITY THEREOF. + +6. This License Agreement will automatically terminate upon a material +breach of its terms and conditions. + +7. This License Agreement shall be governed by the federal +intellectual property law of the United States, including without +limitation the federal copyright law, and, to the extent such +U.S. federal law does not apply, by the law of the Commonwealth of +Virginia, excluding Virginia's conflict of law provisions. +Notwithstanding the foregoing, with regard to derivative works based +on Python 1.6.1 that incorporate non-separable material that was +previously distributed under the GNU General Public License (GPL), the +law of the Commonwealth of Virginia shall govern this License +Agreement only as to issues arising under or with respect to +Paragraphs 4, 5, and 7 of this License Agreement. Nothing in this +License Agreement shall be deemed to create any relationship of +agency, partnership, or joint venture between CNRI and Licensee. This +License Agreement does not grant permission to use CNRI trademarks or +trade name in a trademark sense to endorse or promote products or +services of Licensee, or any third party. + +8. By clicking on the "ACCEPT" button where indicated, or by copying, +installing or otherwise using Python 1.6.1, Licensee agrees to be +bound by the terms and conditions of this License Agreement. + + ACCEPT + + +CWI LICENSE AGREEMENT FOR PYTHON 0.9.0 THROUGH 1.2 +-------------------------------------------------- + +Copyright (c) 1991 - 1995, Stichting Mathematisch Centrum Amsterdam, +The Netherlands. All rights reserved. + +Permission to use, copy, modify, and distribute this software and its +documentation for any purpose and without fee is hereby granted, +provided that the above copyright notice appear in all copies and that +both that copyright notice and this permission notice appear in +supporting documentation, and that the name of Stichting Mathematisch +Centrum or CWI not be used in advertising or publicity pertaining to +distribution of the software without specific, written prior +permission. + +STICHTING MATHEMATISCH CENTRUM DISCLAIMS ALL WARRANTIES WITH REGARD TO +THIS SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND +FITNESS, IN NO EVENT SHALL STICHTING MATHEMATISCH CENTRUM BE LIABLE +FOR ANY SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES +WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN +ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT +OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. diff --git a/scripts/argparse.py b/scripts/argparse.py new file mode 100644 index XXXXXXX..XXXXXXX --- /dev/null +++ b/scripts/argparse.py @@ -XXX,XX +XXX,XX @@ +# This is a local copy of the standard library argparse module taken from PyPI. +# It is licensed under the Python Software Foundation License. This is a +# fallback for Python 2.6 which does not include this module. Python 2.7+ and +# 3+ will never load this module because built-in modules are loaded before +# anything in sys.path. +# +# If your script is not located in the same directory as this file, import it +# like this: +# +# import os +# import sys +# sys.path.append(os.path.join(os.path.dirname(__file__), ..., 'scripts')) +# import argparse + +# Author: Steven J. Bethard <steven.bethard@gmail.com>. +# Maintainer: Thomas Waldmann <tw@waldmann-edv.de> + +"""Command-line parsing library + +This module is an optparse-inspired command-line parsing library that: + + - handles both optional and positional arguments + - produces highly informative usage messages + - supports parsers that dispatch to sub-parsers + +The following is a simple usage example that sums integers from the +command-line and writes the result to a file:: + + parser = argparse.ArgumentParser( + description='sum the integers at the command line') + parser.add_argument( + 'integers', metavar='int', nargs='+', type=int, + help='an integer to be summed') + parser.add_argument( + '--log', default=sys.stdout, type=argparse.FileType('w'), + help='the file where the sum should be written') + args = parser.parse_args() + args.log.write('%s' % sum(args.integers)) + args.log.close() + +The module contains the following public classes: + + - ArgumentParser -- The main entry point for command-line parsing. As the + example above shows, the add_argument() method is used to populate + the parser with actions for optional and positional arguments. Then + the parse_args() method is invoked to convert the args at the + command-line into an object with attributes. + + - ArgumentError -- The exception raised by ArgumentParser objects when + there are errors with the parser's actions. Errors raised while + parsing the command-line are caught by ArgumentParser and emitted + as command-line messages. + + - FileType -- A factory for defining types of files to be created. As the + example above shows, instances of FileType are typically passed as + the type= argument of add_argument() calls. + + - Action -- The base class for parser actions. Typically actions are + selected by passing strings like 'store_true' or 'append_const' to + the action= argument of add_argument(). However, for greater + customization of ArgumentParser actions, subclasses of Action may + be defined and passed as the action= argument. + + - HelpFormatter, RawDescriptionHelpFormatter, RawTextHelpFormatter, + ArgumentDefaultsHelpFormatter -- Formatter classes which + may be passed as the formatter_class= argument to the + ArgumentParser constructor. HelpFormatter is the default, + RawDescriptionHelpFormatter and RawTextHelpFormatter tell the parser + not to change the formatting for help text, and + ArgumentDefaultsHelpFormatter adds information about argument defaults + to the help. + +All other classes in this module are considered implementation details. +(Also note that HelpFormatter and RawDescriptionHelpFormatter are only +considered public as object names -- the API of the formatter objects is +still considered an implementation detail.) +""" + +__version__ = '1.4.0' # we use our own version number independant of the + # one in stdlib and we release this on pypi. + +__external_lib__ = True # to make sure the tests really test THIS lib, + # not the builtin one in Python stdlib + +__all__ = [ + 'ArgumentParser', + 'ArgumentError', + 'ArgumentTypeError', + 'FileType', + 'HelpFormatter', + 'ArgumentDefaultsHelpFormatter', + 'RawDescriptionHelpFormatter', + 'RawTextHelpFormatter', + 'Namespace', + 'Action', + 'ONE_OR_MORE', + 'OPTIONAL', + 'PARSER', + 'REMAINDER', + 'SUPPRESS', + 'ZERO_OR_MORE', +] + + +import copy as _copy +import os as _os +import re as _re +import sys as _sys +import textwrap as _textwrap + +from gettext import gettext as _ + +try: + set +except NameError: + # for python < 2.4 compatibility (sets module is there since 2.3): + from sets import Set as set + +try: + basestring +except NameError: + basestring = str + +try: + sorted +except NameError: + # for python < 2.4 compatibility: + def sorted(iterable, reverse=False): + result = list(iterable) + result.sort() + if reverse: + result.reverse() + return result + + +def _callable(obj): + return hasattr(obj, '__call__') or hasattr(obj, '__bases__') + + +SUPPRESS = '==SUPPRESS==' + +OPTIONAL = '?' +ZERO_OR_MORE = '*' +ONE_OR_MORE = '+' +PARSER = 'A...' +REMAINDER = '...' +_UNRECOGNIZED_ARGS_ATTR = '_unrecognized_args' + +# ============================= +# Utility functions and classes +# ============================= + +class _AttributeHolder(object): + """Abstract base class that provides __repr__. + + The __repr__ method returns a string in the format:: + ClassName(attr=name, attr=name, ...) + The attributes are determined either by a class-level attribute, + '_kwarg_names', or by inspecting the instance __dict__. + """ + + def __repr__(self): + type_name = type(self).__name__ + arg_strings = [] + for arg in self._get_args(): + arg_strings.append(repr(arg)) + for name, value in self._get_kwargs(): + arg_strings.append('%s=%r' % (name, value)) + return '%s(%s)' % (type_name, ', '.join(arg_strings)) + + def _get_kwargs(self): + return sorted(self.__dict__.items()) + + def _get_args(self): + return [] + + +def _ensure_value(namespace, name, value): + if getattr(namespace, name, None) is None: + setattr(namespace, name, value) + return getattr(namespace, name) + + +# =============== +# Formatting Help +# =============== + +class HelpFormatter(object): + """Formatter for generating usage messages and argument help strings. + + Only the name of this class is considered a public API. All the methods + provided by the class are considered an implementation detail. + """ + + def __init__(self, + prog, + indent_increment=2, + max_help_position=24, + width=None): + + # default setting for width + if width is None: + try: + width = int(_os.environ['COLUMNS']) + except (KeyError, ValueError): + width = 80 + width -= 2 + + self._prog = prog + self._indent_increment = indent_increment + self._max_help_position = max_help_position + self._width = width + + self._current_indent = 0 + self._level = 0 + self._action_max_length = 0 + + self._root_section = self._Section(self, None) + self._current_section = self._root_section + + self._whitespace_matcher = _re.compile(r'\s+') + self._long_break_matcher = _re.compile(r'\n\n\n+') + + # =============================== + # Section and indentation methods + # =============================== + def _indent(self): + self._current_indent += self._indent_increment + self._level += 1 + + def _dedent(self): + self._current_indent -= self._indent_increment + assert self._current_indent >= 0, 'Indent decreased below 0.' + self._level -= 1 + + class _Section(object): + + def __init__(self, formatter, parent, heading=None): + self.formatter = formatter + self.parent = parent + self.heading = heading + self.items = [] + + def format_help(self): + # format the indented section + if self.parent is not None: + self.formatter._indent() + join = self.formatter._join_parts + for func, args in self.items: + func(*args) + item_help = join([func(*args) for func, args in self.items]) + if self.parent is not None: + self.formatter._dedent() + + # return nothing if the section was empty + if not item_help: + return '' + + # add the heading if the section was non-empty + if self.heading is not SUPPRESS and self.heading is not None: + current_indent = self.formatter._current_indent + heading = '%*s%s:\n' % (current_indent, '', self.heading) + else: + heading = '' + + # join the section-initial newline, the heading and the help + return join(['\n', heading, item_help, '\n']) + + def _add_item(self, func, args): + self._current_section.items.append((func, args)) + + # ======================== + # Message building methods + # ======================== + def start_section(self, heading): + self._indent() + section = self._Section(self, self._current_section, heading) + self._add_item(section.format_help, []) + self._current_section = section + + def end_section(self): + self._current_section = self._current_section.parent + self._dedent() + + def add_text(self, text): + if text is not SUPPRESS and text is not None: + self._add_item(self._format_text, [text]) + + def add_usage(self, usage, actions, groups, prefix=None): + if usage is not SUPPRESS: + args = usage, actions, groups, prefix + self._add_item(self._format_usage, args) + + def add_argument(self, action): + if action.help is not SUPPRESS: + + # find all invocations + get_invocation = self._format_action_invocation + invocations = [get_invocation(action)] + for subaction in self._iter_indented_subactions(action): + invocations.append(get_invocation(subaction)) + + # update the maximum item length + invocation_length = max([len(s) for s in invocations]) + action_length = invocation_length + self._current_indent + self._action_max_length = max(self._action_max_length, + action_length) + + # add the item to the list + self._add_item(self._format_action, [action]) + + def add_arguments(self, actions): + for action in actions: + self.add_argument(action) + + # ======================= + # Help-formatting methods + # ======================= + def format_help(self): + help = self._root_section.format_help() + if help: + help = self._long_break_matcher.sub('\n\n', help) + help = help.strip('\n') + '\n' + return help + + def _join_parts(self, part_strings): + return ''.join([part + for part in part_strings + if part and part is not SUPPRESS]) + + def _format_usage(self, usage, actions, groups, prefix): + if prefix is None: + prefix = _('usage: ') + + # if usage is specified, use that + if usage is not None: + usage = usage % dict(prog=self._prog) + + # if no optionals or positionals are available, usage is just prog + elif usage is None and not actions: + usage = '%(prog)s' % dict(prog=self._prog) + + # if optionals and positionals are available, calculate usage + elif usage is None: + prog = '%(prog)s' % dict(prog=self._prog) + + # split optionals from positionals + optionals = [] + positionals = [] + for action in actions: + if action.option_strings: + optionals.append(action) + else: + positionals.append(action) + + # build full usage string + format = self._format_actions_usage + action_usage = format(optionals + positionals, groups) + usage = ' '.join([s for s in [prog, action_usage] if s]) + + # wrap the usage parts if it's too long + text_width = self._width - self._current_indent + if len(prefix) + len(usage) > text_width: + + # break usage into wrappable parts + part_regexp = r'\(.*?\)+|\[.*?\]+|\S+' + opt_usage = format(optionals, groups) + pos_usage = format(positionals, groups) + opt_parts = _re.findall(part_regexp, opt_usage) + pos_parts = _re.findall(part_regexp, pos_usage) + assert ' '.join(opt_parts) == opt_usage + assert ' '.join(pos_parts) == pos_usage + + # helper for wrapping lines + def get_lines(parts, indent, prefix=None): + lines = [] + line = [] + if prefix is not None: + line_len = len(prefix) - 1 + else: + line_len = len(indent) - 1 + for part in parts: + if line_len + 1 + len(part) > text_width: + lines.append(indent + ' '.join(line)) + line = [] + line_len = len(indent) - 1 + line.append(part) + line_len += len(part) + 1 + if line: + lines.append(indent + ' '.join(line)) + if prefix is not None: + lines[0] = lines[0][len(indent):] + return lines + + # if prog is short, follow it with optionals or positionals + if len(prefix) + len(prog) <= 0.75 * text_width: + indent = ' ' * (len(prefix) + len(prog) + 1) + if opt_parts: + lines = get_lines([prog] + opt_parts, indent, prefix) + lines.extend(get_lines(pos_parts, indent)) + elif pos_parts: + lines = get_lines([prog] + pos_parts, indent, prefix) + else: + lines = [prog] + + # if prog is long, put it on its own line + else: + indent = ' ' * len(prefix) + parts = opt_parts + pos_parts + lines = get_lines(parts, indent) + if len(lines) > 1: + lines = [] + lines.extend(get_lines(opt_parts, indent)) + lines.extend(get_lines(pos_parts, indent)) + lines = [prog] + lines + + # join lines into usage + usage = '\n'.join(lines) + + # prefix with 'usage:' + return '%s%s\n\n' % (prefix, usage) + + def _format_actions_usage(self, actions, groups): + # find group indices and identify actions in groups + group_actions = set() + inserts = {} + for group in groups: + try: + start = actions.index(group._group_actions[0]) + except ValueError: + continue + else: + end = start + len(group._group_actions) + if actions[start:end] == group._group_actions: + for action in group._group_actions: + group_actions.add(action) + if not group.required: + if start in inserts: + inserts[start] += ' [' + else: + inserts[start] = '[' + inserts[end] = ']' + else: + if start in inserts: + inserts[start] += ' (' + else: + inserts[start] = '(' + inserts[end] = ')' + for i in range(start + 1, end): + inserts[i] = '|' + + # collect all actions format strings + parts = [] + for i, action in enumerate(actions): + + # suppressed arguments are marked with None + # remove | separators for suppressed arguments + if action.help is SUPPRESS: + parts.append(None) + if inserts.get(i) == '|': + inserts.pop(i) + elif inserts.get(i + 1) == '|': + inserts.pop(i + 1) + + # produce all arg strings + elif not action.option_strings: + part = self._format_args(action, action.dest) + + # if it's in a group, strip the outer [] + if action in group_actions: + if part[0] == '[' and part[-1] == ']': + part = part[1:-1] + + # add the action string to the list + parts.append(part) + + # produce the first way to invoke the option in brackets + else: + option_string = action.option_strings[0] + + # if the Optional doesn't take a value, format is: + # -s or --long + if action.nargs == 0: + part = '%s' % option_string + + # if the Optional takes a value, format is: + # -s ARGS or --long ARGS + else: + default = action.dest.upper() + args_string = self._format_args(action, default) + part = '%s %s' % (option_string, args_string) + + # make it look optional if it's not required or in a group + if not action.required and action not in group_actions: + part = '[%s]' % part + + # add the action string to the list + parts.append(part) + + # insert things at the necessary indices + for i in sorted(inserts, reverse=True): + parts[i:i] = [inserts[i]] + + # join all the action items with spaces + text = ' '.join([item for item in parts if item is not None]) + + # clean up separators for mutually exclusive groups + open = r'[\[(]' + close = r'[\])]' + text = _re.sub(r'(%s) ' % open, r'\1', text) + text = _re.sub(r' (%s)' % close, r'\1', text) + text = _re.sub(r'%s *%s' % (open, close), r'', text) + text = _re.sub(r'\(([^|]*)\)', r'\1', text) + text = text.strip() + + # return the text + return text + + def _format_text(self, text): + if '%(prog)' in text: + text = text % dict(prog=self._prog) + text_width = self._width - self._current_indent + indent = ' ' * self._current_indent + return self._fill_text(text, text_width, indent) + '\n\n' + + def _format_action(self, action): + # determine the required width and the entry label + help_position = min(self._action_max_length + 2, + self._max_help_position) + help_width = self._width - help_position + action_width = help_position - self._current_indent - 2 + action_header = self._format_action_invocation(action) + + # ho nelp; start on same line and add a final newline + if not action.help: + tup = self._current_indent, '', action_header + action_header = '%*s%s\n' % tup + + # short action name; start on the same line and pad two spaces + elif len(action_header) <= action_width: + tup = self._current_indent, '', action_width, action_header + action_header = '%*s%-*s ' % tup + indent_first = 0 + + # long action name; start on the next line + else: + tup = self._current_indent, '', action_header + action_header = '%*s%s\n' % tup + indent_first = help_position + + # collect the pieces of the action help + parts = [action_header] + + # if there was help for the action, add lines of help text + if action.help: + help_text = self._expand_help(action) + help_lines = self._split_lines(help_text, help_width) + parts.append('%*s%s\n' % (indent_first, '', help_lines[0])) + for line in help_lines[1:]: + parts.append('%*s%s\n' % (help_position, '', line)) + + # or add a newline if the description doesn't end with one + elif not action_header.endswith('\n'): + parts.append('\n') + + # if there are any sub-actions, add their help as well + for subaction in self._iter_indented_subactions(action): + parts.append(self._format_action(subaction)) + + # return a single string + return self._join_parts(parts) + + def _format_action_invocation(self, action): + if not action.option_strings: + metavar, = self._metavar_formatter(action, action.dest)(1) + return metavar + + else: + parts = [] + + # if the Optional doesn't take a value, format is: + # -s, --long + if action.nargs == 0: + parts.extend(action.option_strings) + + # if the Optional takes a value, format is: + # -s ARGS, --long ARGS + else: + default = action.dest.upper() + args_string = self._format_args(action, default) + for option_string in action.option_strings: + parts.append('%s %s' % (option_string, args_string)) + + return ', '.join(parts) + + def _metavar_formatter(self, action, default_metavar): + if action.metavar is not None: + result = action.metavar + elif action.choices is not None: + choice_strs = [str(choice) for choice in action.choices] + result = '{%s}' % ','.join(choice_strs) + else: + result = default_metavar + + def format(tuple_size): + if isinstance(result, tuple): + return result + else: + return (result, ) * tuple_size + return format + + def _format_args(self, action, default_metavar): + get_metavar = self._metavar_formatter(action, default_metavar) + if action.nargs is None: + result = '%s' % get_metavar(1) + elif action.nargs == OPTIONAL: + result = '[%s]' % get_metavar(1) + elif action.nargs == ZERO_OR_MORE: + result = '[%s [%s ...]]' % get_metavar(2) + elif action.nargs == ONE_OR_MORE: + result = '%s [%s ...]' % get_metavar(2) + elif action.nargs == REMAINDER: + result = '...' + elif action.nargs == PARSER: + result = '%s ...' % get_metavar(1) + else: + formats = ['%s' for _ in range(action.nargs)] + result = ' '.join(formats) % get_metavar(action.nargs) + return result + + def _expand_help(self, action): + params = dict(vars(action), prog=self._prog) + for name in list(params): + if params[name] is SUPPRESS: + del params[name] + for name in list(params): + if hasattr(params[name], '__name__'): + params[name] = params[name].__name__ + if params.get('choices') is not None: + choices_str = ', '.join([str(c) for c in params['choices']]) + params['choices'] = choices_str + return self._get_help_string(action) % params + + def _iter_indented_subactions(self, action): + try: + get_subactions = action._get_subactions + except AttributeError: + pass + else: + self._indent() + for subaction in get_subactions(): + yield subaction + self._dedent() + + def _split_lines(self, text, width): + text = self._whitespace_matcher.sub(' ', text).strip() + return _textwrap.wrap(text, width) + + def _fill_text(self, text, width, indent): + text = self._whitespace_matcher.sub(' ', text).strip() + return _textwrap.fill(text, width, initial_indent=indent, + subsequent_indent=indent) + + def _get_help_string(self, action): + return action.help + + +class RawDescriptionHelpFormatter(HelpFormatter): + """Help message formatter which retains any formatting in descriptions. + + Only the name of this class is considered a public API. All the methods + provided by the class are considered an implementation detail. + """ + + def _fill_text(self, text, width, indent): + return ''.join([indent + line for line in text.splitlines(True)]) + + +class RawTextHelpFormatter(RawDescriptionHelpFormatter): + """Help message formatter which retains formatting of all help text. + + Only the name of this class is considered a public API. All the methods + provided by the class are considered an implementation detail. + """ + + def _split_lines(self, text, width): + return text.splitlines() + + +class ArgumentDefaultsHelpFormatter(HelpFormatter): + """Help message formatter which adds default values to argument help. + + Only the name of this class is considered a public API. All the methods + provided by the class are considered an implementation detail. + """ + + def _get_help_string(self, action): + help = action.help + if '%(default)' not in action.help: + if action.default is not SUPPRESS: + defaulting_nargs = [OPTIONAL, ZERO_OR_MORE] + if action.option_strings or action.nargs in defaulting_nargs: + help += ' (default: %(default)s)' + return help + + +# ===================== +# Options and Arguments +# ===================== + +def _get_action_name(argument): + if argument is None: + return None + elif argument.option_strings: + return '/'.join(argument.option_strings) + elif argument.metavar not in (None, SUPPRESS): + return argument.metavar + elif argument.dest not in (None, SUPPRESS): + return argument.dest + else: + return None + + +class ArgumentError(Exception): + """An error from creating or using an argument (optional or positional). + + The string value of this exception is the message, augmented with + information about the argument that caused it. + """ + + def __init__(self, argument, message): + self.argument_name = _get_action_name(argument) + self.message = message + + def __str__(self): + if self.argument_name is None: + format = '%(message)s' + else: + format = 'argument %(argument_name)s: %(message)s' + return format % dict(message=self.message, + argument_name=self.argument_name) + + +class ArgumentTypeError(Exception): + """An error from trying to convert a command line string to a type.""" + pass + + +# ============== +# Action classes +# ============== + +class Action(_AttributeHolder): + """Information about how to convert command line strings to Python objects. + + Action objects are used by an ArgumentParser to represent the information + needed to parse a single argument from one or more strings from the + command line. The keyword arguments to the Action constructor are also + all attributes of Action instances. + + Keyword Arguments: + + - option_strings -- A list of command-line option strings which + should be associated with this action. + + - dest -- The name of the attribute to hold the created object(s) + + - nargs -- The number of command-line arguments that should be + consumed. By default, one argument will be consumed and a single + value will be produced. Other values include: + - N (an integer) consumes N arguments (and produces a list) + - '?' consumes zero or one arguments + - '*' consumes zero or more arguments (and produces a list) + - '+' consumes one or more arguments (and produces a list) + Note that the difference between the default and nargs=1 is that + with the default, a single value will be produced, while with + nargs=1, a list containing a single value will be produced. + + - const -- The value to be produced if the option is specified and the + option uses an action that takes no values. + + - default -- The value to be produced if the option is not specified. + + - type -- The type which the command-line arguments should be converted + to, should be one of 'string', 'int', 'float', 'complex' or a + callable object that accepts a single string argument. If None, + 'string' is assumed. + + - choices -- A container of values that should be allowed. If not None, + after a command-line argument has been converted to the appropriate + type, an exception will be raised if it is not a member of this + collection. + + - required -- True if the action must always be specified at the + command line. This is only meaningful for optional command-line + arguments. + + - help -- The help string describing the argument. + + - metavar -- The name to be used for the option's argument with the + help string. If None, the 'dest' value will be used as the name. + """ + + def __init__(self, + option_strings, + dest, + nargs=None, + const=None, + default=None, + type=None, + choices=None, + required=False, + help=None, + metavar=None): + self.option_strings = option_strings + self.dest = dest + self.nargs = nargs + self.const = const + self.default = default + self.type = type + self.choices = choices + self.required = required + self.help = help + self.metavar = metavar + + def _get_kwargs(self): + names = [ + 'option_strings', + 'dest', + 'nargs', + 'const', + 'default', + 'type', + 'choices', + 'help', + 'metavar', + ] + return [(name, getattr(self, name)) for name in names] + + def __call__(self, parser, namespace, values, option_string=None): + raise NotImplementedError(_('.__call__() not defined')) + + +class _StoreAction(Action): + + def __init__(self, + option_strings, + dest, + nargs=None, + const=None, + default=None, + type=None, + choices=None, + required=False, + help=None, + metavar=None): + if nargs == 0: + raise ValueError('nargs for store actions must be > 0; if you ' + 'have nothing to store, actions such as store ' + 'true or store const may be more appropriate') + if const is not None and nargs != OPTIONAL: + raise ValueError('nargs must be %r to supply const' % OPTIONAL) + super(_StoreAction, self).__init__( + option_strings=option_strings, + dest=dest, + nargs=nargs, + const=const, + default=default, + type=type, + choices=choices, + required=required, + help=help, + metavar=metavar) + + def __call__(self, parser, namespace, values, option_string=None): + setattr(namespace, self.dest, values) + + +class _StoreConstAction(Action): + + def __init__(self, + option_strings, + dest, + const, + default=None, + required=False, + help=None, + metavar=None): + super(_StoreConstAction, self).__init__( + option_strings=option_strings, + dest=dest, + nargs=0, + const=const, + default=default, + required=required, + help=help) + + def __call__(self, parser, namespace, values, option_string=None): + setattr(namespace, self.dest, self.const) + + +class _StoreTrueAction(_StoreConstAction): + + def __init__(self, + option_strings, + dest, + default=False, + required=False, + help=None): + super(_StoreTrueAction, self).__init__( + option_strings=option_strings, + dest=dest, + const=True, + default=default, + required=required, + help=help) + + +class _StoreFalseAction(_StoreConstAction): + + def __init__(self, + option_strings, + dest, + default=True, + required=False, + help=None): + super(_StoreFalseAction, self).__init__( + option_strings=option_strings, + dest=dest, + const=False, + default=default, + required=required, + help=help) + + +class _AppendAction(Action): + + def __init__(self, + option_strings, + dest, + nargs=None, + const=None, + default=None, + type=None, + choices=None, + required=False, + help=None, + metavar=None): + if nargs == 0: + raise ValueError('nargs for append actions must be > 0; if arg ' + 'strings are not supplying the value to append, ' + 'the append const action may be more appropriate') + if const is not None and nargs != OPTIONAL: + raise ValueError('nargs must be %r to supply const' % OPTIONAL) + super(_AppendAction, self).__init__( + option_strings=option_strings, + dest=dest, + nargs=nargs, + const=const, + default=default, + type=type, + choices=choices, + required=required, + help=help, + metavar=metavar) + + def __call__(self, parser, namespace, values, option_string=None): + items = _copy.copy(_ensure_value(namespace, self.dest, [])) + items.append(values) + setattr(namespace, self.dest, items) + + +class _AppendConstAction(Action): + + def __init__(self, + option_strings, + dest, + const, + default=None, + required=False, + help=None, + metavar=None): + super(_AppendConstAction, self).__init__( + option_strings=option_strings, + dest=dest, + nargs=0, + const=const, + default=default, + required=required, + help=help, + metavar=metavar) + + def __call__(self, parser, namespace, values, option_string=None): + items = _copy.copy(_ensure_value(namespace, self.dest, [])) + items.append(self.const) + setattr(namespace, self.dest, items) + + +class _CountAction(Action): + + def __init__(self, + option_strings, + dest, + default=None, + required=False, + help=None): + super(_CountAction, self).__init__( + option_strings=option_strings, + dest=dest, + nargs=0, + default=default, + required=required, + help=help) + + def __call__(self, parser, namespace, values, option_string=None): + new_count = _ensure_value(namespace, self.dest, 0) + 1 + setattr(namespace, self.dest, new_count) + + +class _HelpAction(Action): + + def __init__(self, + option_strings, + dest=SUPPRESS, + default=SUPPRESS, + help=None): + super(_HelpAction, self).__init__( + option_strings=option_strings, + dest=dest, + default=default, + nargs=0, + help=help) + + def __call__(self, parser, namespace, values, option_string=None): + parser.print_help() + parser.exit() + + +class _VersionAction(Action): + + def __init__(self, + option_strings, + version=None, + dest=SUPPRESS, + default=SUPPRESS, + help="show program's version number and exit"): + super(_VersionAction, self).__init__( + option_strings=option_strings, + dest=dest, + default=default, + nargs=0, + help=help) + self.version = version + + def __call__(self, parser, namespace, values, option_string=None): + version = self.version + if version is None: + version = parser.version + formatter = parser._get_formatter() + formatter.add_text(version) + parser.exit(message=formatter.format_help()) + + +class _SubParsersAction(Action): + + class _ChoicesPseudoAction(Action): + + def __init__(self, name, aliases, help): + metavar = dest = name + if aliases: + metavar += ' (%s)' % ', '.join(aliases) + sup = super(_SubParsersAction._ChoicesPseudoAction, self) + sup.__init__(option_strings=[], dest=dest, help=help, + metavar=metavar) + + def __init__(self, + option_strings, + prog, + parser_class, + dest=SUPPRESS, + help=None, + metavar=None): + + self._prog_prefix = prog + self._parser_class = parser_class + self._name_parser_map = {} + self._choices_actions = [] + + super(_SubParsersAction, self).__init__( + option_strings=option_strings, + dest=dest, + nargs=PARSER, + choices=self._name_parser_map, + help=help, + metavar=metavar) + + def add_parser(self, name, **kwargs): + # set prog from the existing prefix + if kwargs.get('prog') is None: + kwargs['prog'] = '%s %s' % (self._prog_prefix, name) + + aliases = kwargs.pop('aliases', ()) + + # create a pseudo-action to hold the choice help + if 'help' in kwargs: + help = kwargs.pop('help') + choice_action = self._ChoicesPseudoAction(name, aliases, help) + self._choices_actions.append(choice_action) + + # create the parser and add it to the map + parser = self._parser_class(**kwargs) + self._name_parser_map[name] = parser + + # make parser available under aliases also + for alias in aliases: + self._name_parser_map[alias] = parser + + return parser + + def _get_subactions(self): + return self._choices_actions + + def __call__(self, parser, namespace, values, option_string=None): + parser_name = values[0] + arg_strings = values[1:] + + # set the parser name if requested + if self.dest is not SUPPRESS: + setattr(namespace, self.dest, parser_name) + + # select the parser + try: + parser = self._name_parser_map[parser_name] + except KeyError: + tup = parser_name, ', '.join(self._name_parser_map) + msg = _('unknown parser %r (choices: %s)' % tup) + raise ArgumentError(self, msg) + + # parse all the remaining options into the namespace + # store any unrecognized options on the object, so that the top + # level parser can decide what to do with them + namespace, arg_strings = parser.parse_known_args(arg_strings, namespace) + if arg_strings: + vars(namespace).setdefault(_UNRECOGNIZED_ARGS_ATTR, []) + getattr(namespace, _UNRECOGNIZED_ARGS_ATTR).extend(arg_strings) + + +# ============== +# Type classes +# ============== + +class FileType(object): + """Factory for creating file object types + + Instances of FileType are typically passed as type= arguments to the + ArgumentParser add_argument() method. + + Keyword Arguments: + - mode -- A string indicating how the file is to be opened. Accepts the + same values as the builtin open() function. + - bufsize -- The file's desired buffer size. Accepts the same values as + the builtin open() function. + """ + + def __init__(self, mode='r', bufsize=None): + self._mode = mode + self._bufsize = bufsize + + def __call__(self, string): + # the special argument "-" means sys.std{in,out} + if string == '-': + if 'r' in self._mode: + return _sys.stdin + elif 'w' in self._mode: + return _sys.stdout + else: + msg = _('argument "-" with mode %r' % self._mode) + raise ValueError(msg) + + try: + # all other arguments are used as file names + if self._bufsize: + return open(string, self._mode, self._bufsize) + else: + return open(string, self._mode) + except IOError: + err = _sys.exc_info()[1] + message = _("can't open '%s': %s") + raise ArgumentTypeError(message % (string, err)) + + def __repr__(self): + args = [self._mode, self._bufsize] + args_str = ', '.join([repr(arg) for arg in args if arg is not None]) + return '%s(%s)' % (type(self).__name__, args_str) + +# =========================== +# Optional and Positional Parsing +# =========================== + +class Namespace(_AttributeHolder): + """Simple object for storing attributes. + + Implements equality by attribute names and values, and provides a simple + string representation. + """ + + def __init__(self, **kwargs): + for name in kwargs: + setattr(self, name, kwargs[name]) + + __hash__ = None + + def __eq__(self, other): + return vars(self) == vars(other) + + def __ne__(self, other): + return not (self == other) + + def __contains__(self, key): + return key in self.__dict__ + + +class _ActionsContainer(object): + + def __init__(self, + description, + prefix_chars, + argument_default, + conflict_handler): + super(_ActionsContainer, self).__init__() + + self.description = description + self.argument_default = argument_default + self.prefix_chars = prefix_chars + self.conflict_handler = conflict_handler + + # set up registries + self._registries = {} + + # register actions + self.register('action', None, _StoreAction) + self.register('action', 'store', _StoreAction) + self.register('action', 'store_const', _StoreConstAction) + self.register('action', 'store_true', _StoreTrueAction) + self.register('action', 'store_false', _StoreFalseAction) + self.register('action', 'append', _AppendAction) + self.register('action', 'append_const', _AppendConstAction) + self.register('action', 'count', _CountAction) + self.register('action', 'help', _HelpAction) + self.register('action', 'version', _VersionAction) + self.register('action', 'parsers', _SubParsersAction) + + # raise an exception if the conflict handler is invalid + self._get_handler() + + # action storage + self._actions = [] + self._option_string_actions = {} + + # groups + self._action_groups = [] + self._mutually_exclusive_groups = [] + + # defaults storage + self._defaults = {} + + # determines whether an "option" looks like a negative number + self._negative_number_matcher = _re.compile(r'^-\d+$|^-\d*\.\d+$') + + # whether or not there are any optionals that look like negative + # numbers -- uses a list so it can be shared and edited + self._has_negative_number_optionals = [] + + # ==================== + # Registration methods + # ==================== + def register(self, registry_name, value, object): + registry = self._registries.setdefault(registry_name, {}) + registry[value] = object + + def _registry_get(self, registry_name, value, default=None): + return self._registries[registry_name].get(value, default) + + # ================================== + # Namespace default accessor methods + # ================================== + def set_defaults(self, **kwargs): + self._defaults.update(kwargs) + + # if these defaults match any existing arguments, replace + # the previous default on the object with the new one + for action in self._actions: + if action.dest in kwargs: + action.default = kwargs[action.dest] + + def get_default(self, dest): + for action in self._actions: + if action.dest == dest and action.default is not None: + return action.default + return self._defaults.get(dest, None) + + + # ======================= + # Adding argument actions + # ======================= + def add_argument(self, *args, **kwargs): + """ + add_argument(dest, ..., name=value, ...) + add_argument(option_string, option_string, ..., name=value, ...) + """ + + # if no positional args are supplied or only one is supplied and + # it doesn't look like an option string, parse a positional + # argument + chars = self.prefix_chars + if not args or len(args) == 1 and args[0][0] not in chars: + if args and 'dest' in kwargs: + raise ValueError('dest supplied twice for positional argument') + kwargs = self._get_positional_kwargs(*args, **kwargs) + + # otherwise, we're adding an optional argument + else: + kwargs = self._get_optional_kwargs(*args, **kwargs) + + # if no default was supplied, use the parser-level default + if 'default' not in kwargs: + dest = kwargs['dest'] + if dest in self._defaults: + kwargs['default'] = self._defaults[dest] + elif self.argument_default is not None: + kwargs['default'] = self.argument_default + + # create the action object, and add it to the parser + action_class = self._pop_action_class(kwargs) + if not _callable(action_class): + raise ValueError('unknown action "%s"' % action_class) + action = action_class(**kwargs) + + # raise an error if the action type is not callable + type_func = self._registry_get('type', action.type, action.type) + if not _callable(type_func): + raise ValueError('%r is not callable' % type_func) + + return self._add_action(action) + + def add_argument_group(self, *args, **kwargs): + group = _ArgumentGroup(self, *args, **kwargs) + self._action_groups.append(group) + return group + + def add_mutually_exclusive_group(self, **kwargs): + group = _MutuallyExclusiveGroup(self, **kwargs) + self._mutually_exclusive_groups.append(group) + return group + + def _add_action(self, action): + # resolve any conflicts + self._check_conflict(action) + + # add to actions list + self._actions.append(action) + action.container = self + + # index the action by any option strings it has + for option_string in action.option_strings: + self._option_string_actions[option_string] = action + + # set the flag if any option strings look like negative numbers + for option_string in action.option_strings: + if self._negative_number_matcher.match(option_string): + if not self._has_negative_number_optionals: + self._has_negative_number_optionals.append(True) + + # return the created action + return action + + def _remove_action(self, action): + self._actions.remove(action) + + def _add_container_actions(self, container): + # collect groups by titles + title_group_map = {} + for group in self._action_groups: + if group.title in title_group_map: + msg = _('cannot merge actions - two groups are named %r') + raise ValueError(msg % (group.title)) + title_group_map[group.title] = group + + # map each action to its group + group_map = {} + for group in container._action_groups: + + # if a group with the title exists, use that, otherwise + # create a new group matching the container's group + if group.title not in title_group_map: + title_group_map[group.title] = self.add_argument_group( + title=group.title, + description=group.description, + conflict_handler=group.conflict_handler) + + # map the actions to their new group + for action in group._group_actions: + group_map[action] = title_group_map[group.title] + + # add container's mutually exclusive groups + # NOTE: if add_mutually_exclusive_group ever gains title= and + # description= then this code will need to be expanded as above + for group in container._mutually_exclusive_groups: + mutex_group = self.add_mutually_exclusive_group( + required=group.required) + + # map the actions to their new mutex group + for action in group._group_actions: + group_map[action] = mutex_group + + # add all actions to this container or their group + for action in container._actions: + group_map.get(action, self)._add_action(action) + + def _get_positional_kwargs(self, dest, **kwargs): + # make sure required is not specified + if 'required' in kwargs: + msg = _("'required' is an invalid argument for positionals") + raise TypeError(msg) + + # mark positional arguments as required if at least one is + # always required + if kwargs.get('nargs') not in [OPTIONAL, ZERO_OR_MORE]: + kwargs['required'] = True + if kwargs.get('nargs') == ZERO_OR_MORE and 'default' not in kwargs: + kwargs['required'] = True + + # return the keyword arguments with no option strings + return dict(kwargs, dest=dest, option_strings=[]) + + def _get_optional_kwargs(self, *args, **kwargs): + # determine short and long option strings + option_strings = [] + long_option_strings = [] + for option_string in args: + # error on strings that don't start with an appropriate prefix + if not option_string[0] in self.prefix_chars: + msg = _('invalid option string %r: ' + 'must start with a character %r') + tup = option_string, self.prefix_chars + raise ValueError(msg % tup) + + # strings starting with two prefix characters are long options + option_strings.append(option_string) + if option_string[0] in self.prefix_chars: + if len(option_string) > 1: + if option_string[1] in self.prefix_chars: + long_option_strings.append(option_string) + + # infer destination, '--foo-bar' -> 'foo_bar' and '-x' -> 'x' + dest = kwargs.pop('dest', None) + if dest is None: + if long_option_strings: + dest_option_string = long_option_strings[0] + else: + dest_option_string = option_strings[0] + dest = dest_option_string.lstrip(self.prefix_chars) + if not dest: + msg = _('dest= is required for options like %r') + raise ValueError(msg % option_string) + dest = dest.replace('-', '_') + + # return the updated keyword arguments + return dict(kwargs, dest=dest, option_strings=option_strings) + + def _pop_action_class(self, kwargs, default=None): + action = kwargs.pop('action', default) + return self._registry_get('action', action, action) + + def _get_handler(self): + # determine function from conflict handler string + handler_func_name = '_handle_conflict_%s' % self.conflict_handler + try: + return getattr(self, handler_func_name) + except AttributeError: + msg = _('invalid conflict_resolution value: %r') + raise ValueError(msg % self.conflict_handler) + + def _check_conflict(self, action): + + # find all options that conflict with this option + confl_optionals = [] + for option_string in action.option_strings: + if option_string in self._option_string_actions: + confl_optional = self._option_string_actions[option_string] + confl_optionals.append((option_string, confl_optional)) + + # resolve any conflicts + if confl_optionals: + conflict_handler = self._get_handler() + conflict_handler(action, confl_optionals) + + def _handle_conflict_error(self, action, conflicting_actions): + message = _('conflicting option string(s): %s') + conflict_string = ', '.join([option_string + for option_string, action + in conflicting_actions]) + raise ArgumentError(action, message % conflict_string) + + def _handle_conflict_resolve(self, action, conflicting_actions): + + # remove all conflicting options + for option_string, action in conflicting_actions: + + # remove the conflicting option + action.option_strings.remove(option_string) + self._option_string_actions.pop(option_string, None) + + # if the option now has no option string, remove it from the + # container holding it + if not action.option_strings: + action.container._remove_action(action) + + +class _ArgumentGroup(_ActionsContainer): + + def __init__(self, container, title=None, description=None, **kwargs): + # add any missing keyword arguments by checking the container + update = kwargs.setdefault + update('conflict_handler', container.conflict_handler) + update('prefix_chars', container.prefix_chars) + update('argument_default', container.argument_default) + super_init = super(_ArgumentGroup, self).__init__ + super_init(description=description, **kwargs) + + # group attributes + self.title = title + self._group_actions = [] + + # share most attributes with the container + self._registries = container._registries + self._actions = container._actions + self._option_string_actions = container._option_string_actions + self._defaults = container._defaults + self._has_negative_number_optionals = \ + container._has_negative_number_optionals + + def _add_action(self, action): + action = super(_ArgumentGroup, self)._add_action(action) + self._group_actions.append(action) + return action + + def _remove_action(self, action): + super(_ArgumentGroup, self)._remove_action(action) + self._group_actions.remove(action) + + +class _MutuallyExclusiveGroup(_ArgumentGroup): + + def __init__(self, container, required=False): + super(_MutuallyExclusiveGroup, self).__init__(container) + self.required = required + self._container = container + + def _add_action(self, action): + if action.required: + msg = _('mutually exclusive arguments must be optional') + raise ValueError(msg) + action = self._container._add_action(action) + self._group_actions.append(action) + return action + + def _remove_action(self, action): + self._container._remove_action(action) + self._group_actions.remove(action) + + +class ArgumentParser(_AttributeHolder, _ActionsContainer): + """Object for parsing command line strings into Python objects. + + Keyword Arguments: + - prog -- The name of the program (default: sys.argv[0]) + - usage -- A usage message (default: auto-generated from arguments) + - description -- A description of what the program does + - epilog -- Text following the argument descriptions + - parents -- Parsers whose arguments should be copied into this one + - formatter_class -- HelpFormatter class for printing help messages + - prefix_chars -- Characters that prefix optional arguments + - fromfile_prefix_chars -- Characters that prefix files containing + additional arguments + - argument_default -- The default value for all arguments + - conflict_handler -- String indicating how to handle conflicts + - add_help -- Add a -h/-help option + """ + + def __init__(self, + prog=None, + usage=None, + description=None, + epilog=None, + version=None, + parents=[], + formatter_class=HelpFormatter, + prefix_chars='-', + fromfile_prefix_chars=None, + argument_default=None, + conflict_handler='error', + add_help=True): + + if version is not None: + import warnings + warnings.warn( + """The "version" argument to ArgumentParser is deprecated. """ + """Please use """ + """"add_argument(..., action='version', version="N", ...)" """ + """instead""", DeprecationWarning) + + superinit = super(ArgumentParser, self).__init__ + superinit(description=description, + prefix_chars=prefix_chars, + argument_default=argument_default, + conflict_handler=conflict_handler) + + # default setting for prog + if prog is None: + prog = _os.path.basename(_sys.argv[0]) + + self.prog = prog + self.usage = usage + self.epilog = epilog + self.version = version + self.formatter_class = formatter_class + self.fromfile_prefix_chars = fromfile_prefix_chars + self.add_help = add_help + + add_group = self.add_argument_group + self._positionals = add_group(_('positional arguments')) + self._optionals = add_group(_('optional arguments')) + self._subparsers = None + + # register types + def identity(string): + return string + self.register('type', None, identity) + + # add help and version arguments if necessary + # (using explicit default to override global argument_default) + if '-' in prefix_chars: + default_prefix = '-' + else: + default_prefix = prefix_chars[0] + if self.add_help: + self.add_argument( + default_prefix+'h', default_prefix*2+'help', + action='help', default=SUPPRESS, + help=_('show this help message and exit')) + if self.version: + self.add_argument( + default_prefix+'v', default_prefix*2+'version', + action='version', default=SUPPRESS, + version=self.version, + help=_("show program's version number and exit")) + + # add parent arguments and defaults + for parent in parents: + self._add_container_actions(parent) + try: + defaults = parent._defaults + except AttributeError: + pass + else: + self._defaults.update(defaults) + + # ======================= + # Pretty __repr__ methods + # ======================= + def _get_kwargs(self): + names = [ + 'prog', + 'usage', + 'description', + 'version', + 'formatter_class', + 'conflict_handler', + 'add_help', + ] + return [(name, getattr(self, name)) for name in names] + + # ================================== + # Optional/Positional adding methods + # ================================== + def add_subparsers(self, **kwargs): + if self._subparsers is not None: + self.error(_('cannot have multiple subparser arguments')) + + # add the parser class to the arguments if it's not present + kwargs.setdefault('parser_class', type(self)) + + if 'title' in kwargs or 'description' in kwargs: + title = _(kwargs.pop('title', 'subcommands')) + description = _(kwargs.pop('description', None)) + self._subparsers = self.add_argument_group(title, description) + else: + self._subparsers = self._positionals + + # prog defaults to the usage message of this parser, skipping + # optional arguments and with no "usage:" prefix + if kwargs.get('prog') is None: + formatter = self._get_formatter() + positionals = self._get_positional_actions() + groups = self._mutually_exclusive_groups + formatter.add_usage(self.usage, positionals, groups, '') + kwargs['prog'] = formatter.format_help().strip() + + # create the parsers action and add it to the positionals list + parsers_class = self._pop_action_class(kwargs, 'parsers') + action = parsers_class(option_strings=[], **kwargs) + self._subparsers._add_action(action) + + # return the created parsers action + return action + + def _add_action(self, action): + if action.option_strings: + self._optionals._add_action(action) + else: + self._positionals._add_action(action) + return action + + def _get_optional_actions(self): + return [action + for action in self._actions + if action.option_strings] + + def _get_positional_actions(self): + return [action + for action in self._actions + if not action.option_strings] + + # ===================================== + # Command line argument parsing methods + # ===================================== + def parse_args(self, args=None, namespace=None): + args, argv = self.parse_known_args(args, namespace) + if argv: + msg = _('unrecognized arguments: %s') + self.error(msg % ' '.join(argv)) + return args + + def parse_known_args(self, args=None, namespace=None): + # args default to the system args + if args is None: + args = _sys.argv[1:] + + # default Namespace built from parser defaults + if namespace is None: + namespace = Namespace() + + # add any action defaults that aren't present + for action in self._actions: + if action.dest is not SUPPRESS: + if not hasattr(namespace, action.dest): + if action.default is not SUPPRESS: + setattr(namespace, action.dest, action.default) + + # add any parser defaults that aren't present + for dest in self._defaults: + if not hasattr(namespace, dest): + setattr(namespace, dest, self._defaults[dest]) + + # parse the arguments and exit if there are any errors + try: + namespace, args = self._parse_known_args(args, namespace) + if hasattr(namespace, _UNRECOGNIZED_ARGS_ATTR): + args.extend(getattr(namespace, _UNRECOGNIZED_ARGS_ATTR)) + delattr(namespace, _UNRECOGNIZED_ARGS_ATTR) + return namespace, args + except ArgumentError: + err = _sys.exc_info()[1] + self.error(str(err)) + + def _parse_known_args(self, arg_strings, namespace): + # replace arg strings that are file references + if self.fromfile_prefix_chars is not None: + arg_strings = self._read_args_from_files(arg_strings) + + # map all mutually exclusive arguments to the other arguments + # they can't occur with + action_conflicts = {} + for mutex_group in self._mutually_exclusive_groups: + group_actions = mutex_group._group_actions + for i, mutex_action in enumerate(mutex_group._group_actions): + conflicts = action_conflicts.setdefault(mutex_action, []) + conflicts.extend(group_actions[:i]) + conflicts.extend(group_actions[i + 1:]) + + # find all option indices, and determine the arg_string_pattern + # which has an 'O' if there is an option at an index, + # an 'A' if there is an argument, or a '-' if there is a '--' + option_string_indices = {} + arg_string_pattern_parts = [] + arg_strings_iter = iter(arg_strings) + for i, arg_string in enumerate(arg_strings_iter): + + # all args after -- are non-options + if arg_string == '--': + arg_string_pattern_parts.append('-') + for arg_string in arg_strings_iter: + arg_string_pattern_parts.append('A') + + # otherwise, add the arg to the arg strings + # and note the index if it was an option + else: + option_tuple = self._parse_optional(arg_string) + if option_tuple is None: + pattern = 'A' + else: + option_string_indices[i] = option_tuple + pattern = 'O' + arg_string_pattern_parts.append(pattern) + + # join the pieces together to form the pattern + arg_strings_pattern = ''.join(arg_string_pattern_parts) + + # converts arg strings to the appropriate and then takes the action + seen_actions = set() + seen_non_default_actions = set() + + def take_action(action, argument_strings, option_string=None): + seen_actions.add(action) + argument_values = self._get_values(action, argument_strings) + + # error if this argument is not allowed with other previously + # seen arguments, assuming that actions that use the default + # value don't really count as "present" + if argument_values is not action.default: + seen_non_default_actions.add(action) + for conflict_action in action_conflicts.get(action, []): + if conflict_action in seen_non_default_actions: + msg = _('not allowed with argument %s') + action_name = _get_action_name(conflict_action) + raise ArgumentError(action, msg % action_name) + + # take the action if we didn't receive a SUPPRESS value + # (e.g. from a default) + if argument_values is not SUPPRESS: + action(self, namespace, argument_values, option_string) + + # function to convert arg_strings into an optional action + def consume_optional(start_index): + + # get the optional identified at this index + option_tuple = option_string_indices[start_index] + action, option_string, explicit_arg = option_tuple + + # identify additional optionals in the same arg string + # (e.g. -xyz is the same as -x -y -z if no args are required) + match_argument = self._match_argument + action_tuples = [] + while True: + + # if we found no optional action, skip it + if action is None: + extras.append(arg_strings[start_index]) + return start_index + 1 + + # if there is an explicit argument, try to match the + # optional's string arguments to only this + if explicit_arg is not None: + arg_count = match_argument(action, 'A') + + # if the action is a single-dash option and takes no + # arguments, try to parse more single-dash options out + # of the tail of the option string + chars = self.prefix_chars + if arg_count == 0 and option_string[1] not in chars: + action_tuples.append((action, [], option_string)) + char = option_string[0] + option_string = char + explicit_arg[0] + new_explicit_arg = explicit_arg[1:] or None + optionals_map = self._option_string_actions + if option_string in optionals_map: + action = optionals_map[option_string] + explicit_arg = new_explicit_arg + else: + msg = _('ignored explicit argument %r') + raise ArgumentError(action, msg % explicit_arg) + + # if the action expect exactly one argument, we've + # successfully matched the option; exit the loop + elif arg_count == 1: + stop = start_index + 1 + args = [explicit_arg] + action_tuples.append((action, args, option_string)) + break + + # error if a double-dash option did not use the + # explicit argument + else: + msg = _('ignored explicit argument %r') + raise ArgumentError(action, msg % explicit_arg) + + # if there is no explicit argument, try to match the + # optional's string arguments with the following strings + # if successful, exit the loop + else: + start = start_index + 1 + selected_patterns = arg_strings_pattern[start:] + arg_count = match_argument(action, selected_patterns) + stop = start + arg_count + args = arg_strings[start:stop] + action_tuples.append((action, args, option_string)) + break + + # add the Optional to the list and return the index at which + # the Optional's string args stopped + assert action_tuples + for action, args, option_string in action_tuples: + take_action(action, args, option_string) + return stop + + # the list of Positionals left to be parsed; this is modified + # by consume_positionals() + positionals = self._get_positional_actions() + + # function to convert arg_strings into positional actions + def consume_positionals(start_index): + # match as many Positionals as possible + match_partial = self._match_arguments_partial + selected_pattern = arg_strings_pattern[start_index:] + arg_counts = match_partial(positionals, selected_pattern) + + # slice off the appropriate arg strings for each Positional + # and add the Positional and its args to the list + for action, arg_count in zip(positionals, arg_counts): + args = arg_strings[start_index: start_index + arg_count] + start_index += arg_count + take_action(action, args) + + # slice off the Positionals that we just parsed and return the + # index at which the Positionals' string args stopped + positionals[:] = positionals[len(arg_counts):] + return start_index + + # consume Positionals and Optionals alternately, until we have + # passed the last option string + extras = [] + start_index = 0 + if option_string_indices: + max_option_string_index = max(option_string_indices) + else: + max_option_string_index = -1 + while start_index <= max_option_string_index: + + # consume any Positionals preceding the next option + next_option_string_index = min([ + index + for index in option_string_indices + if index >= start_index]) + if start_index != next_option_string_index: + positionals_end_index = consume_positionals(start_index) + + # only try to parse the next optional if we didn't consume + # the option string during the positionals parsing + if positionals_end_index > start_index: + start_index = positionals_end_index + continue + else: + start_index = positionals_end_index + + # if we consumed all the positionals we could and we're not + # at the index of an option string, there were extra arguments + if start_index not in option_string_indices: + strings = arg_strings[start_index:next_option_string_index] + extras.extend(strings) + start_index = next_option_string_index + + # consume the next optional and any arguments for it + start_index = consume_optional(start_index) + + # consume any positionals following the last Optional + stop_index = consume_positionals(start_index) + + # if we didn't consume all the argument strings, there were extras + extras.extend(arg_strings[stop_index:]) + + # if we didn't use all the Positional objects, there were too few + # arg strings supplied. + if positionals: + self.error(_('too few arguments')) + + # make sure all required actions were present, and convert defaults. + for action in self._actions: + if action not in seen_actions: + if action.required: + name = _get_action_name(action) + self.error(_('argument %s is required') % name) + else: + # Convert action default now instead of doing it before + # parsing arguments to avoid calling convert functions + # twice (which may fail) if the argument was given, but + # only if it was defined already in the namespace + if (action.default is not None and + isinstance(action.default, basestring) and + hasattr(namespace, action.dest) and + action.default is getattr(namespace, action.dest)): + setattr(namespace, action.dest, + self._get_value(action, action.default)) + + # make sure all required groups had one option present + for group in self._mutually_exclusive_groups: + if group.required: + for action in group._group_actions: + if action in seen_non_default_actions: + break + + # if no actions were used, report the error + else: + names = [_get_action_name(action) + for action in group._group_actions + if action.help is not SUPPRESS] + msg = _('one of the arguments %s is required') + self.error(msg % ' '.join(names)) + + # return the updated namespace and the extra arguments + return namespace, extras + + def _read_args_from_files(self, arg_strings): + # expand arguments referencing files + new_arg_strings = [] + for arg_string in arg_strings: + + # for regular arguments, just add them back into the list + if arg_string[0] not in self.fromfile_prefix_chars: + new_arg_strings.append(arg_string) + + # replace arguments referencing files with the file content + else: + try: + args_file = open(arg_string[1:]) + try: + arg_strings = [] + for arg_line in args_file.read().splitlines(): + for arg in self.convert_arg_line_to_args(arg_line): + arg_strings.append(arg) + arg_strings = self._read_args_from_files(arg_strings) + new_arg_strings.extend(arg_strings) + finally: + args_file.close() + except IOError: + err = _sys.exc_info()[1] + self.error(str(err)) + + # return the modified argument list + return new_arg_strings + + def convert_arg_line_to_args(self, arg_line): + return [arg_line] + + def _match_argument(self, action, arg_strings_pattern): + # match the pattern for this action to the arg strings + nargs_pattern = self._get_nargs_pattern(action) + match = _re.match(nargs_pattern, arg_strings_pattern) + + # raise an exception if we weren't able to find a match + if match is None: + nargs_errors = { + None: _('expected one argument'), + OPTIONAL: _('expected at most one argument'), + ONE_OR_MORE: _('expected at least one argument'), + } + default = _('expected %s argument(s)') % action.nargs + msg = nargs_errors.get(action.nargs, default) + raise ArgumentError(action, msg) + + # return the number of arguments matched + return len(match.group(1)) + + def _match_arguments_partial(self, actions, arg_strings_pattern): + # progressively shorten the actions list by slicing off the + # final actions until we find a match + result = [] + for i in range(len(actions), 0, -1): + actions_slice = actions[:i] + pattern = ''.join([self._get_nargs_pattern(action) + for action in actions_slice]) + match = _re.match(pattern, arg_strings_pattern) + if match is not None: + result.extend([len(string) for string in match.groups()]) + break + + # return the list of arg string counts + return result + + def _parse_optional(self, arg_string): + # if it's an empty string, it was meant to be a positional + if not arg_string: + return None + + # if it doesn't start with a prefix, it was meant to be positional + if not arg_string[0] in self.prefix_chars: + return None + + # if the option string is present in the parser, return the action + if arg_string in self._option_string_actions: + action = self._option_string_actions[arg_string] + return action, arg_string, None + + # if it's just a single character, it was meant to be positional + if len(arg_string) == 1: + return None + + # if the option string before the "=" is present, return the action + if '=' in arg_string: + option_string, explicit_arg = arg_string.split('=', 1) + if option_string in self._option_string_actions: + action = self._option_string_actions[option_string] + return action, option_string, explicit_arg + + # search through all possible prefixes of the option string + # and all actions in the parser for possible interpretations + option_tuples = self._get_option_tuples(arg_string) + + # if multiple actions match, the option string was ambiguous + if len(option_tuples) > 1: + options = ', '.join([option_string + for action, option_string, explicit_arg in option_tuples]) + tup = arg_string, options + self.error(_('ambiguous option: %s could match %s') % tup) + + # if exactly one action matched, this segmentation is good, + # so return the parsed action + elif len(option_tuples) == 1: + option_tuple, = option_tuples + return option_tuple + + # if it was not found as an option, but it looks like a negative + # number, it was meant to be positional + # unless there are negative-number-like options + if self._negative_number_matcher.match(arg_string): + if not self._has_negative_number_optionals: + return None + + # if it contains a space, it was meant to be a positional + if ' ' in arg_string: + return None + + # it was meant to be an optional but there is no such option + # in this parser (though it might be a valid option in a subparser) + return None, arg_string, None + + def _get_option_tuples(self, option_string): + result = [] + + # option strings starting with two prefix characters are only + # split at the '=' + chars = self.prefix_chars + if option_string[0] in chars and option_string[1] in chars: + if '=' in option_string: + option_prefix, explicit_arg = option_string.split('=', 1) + else: + option_prefix = option_string + explicit_arg = None + for option_string in self._option_string_actions: + if option_string.startswith(option_prefix): + action = self._option_string_actions[option_string] + tup = action, option_string, explicit_arg + result.append(tup) + + # single character options can be concatenated with their arguments + # but multiple character options always have to have their argument + # separate + elif option_string[0] in chars and option_string[1] not in chars: + option_prefix = option_string + explicit_arg = None + short_option_prefix = option_string[:2] + short_explicit_arg = option_string[2:] + + for option_string in self._option_string_actions: + if option_string == short_option_prefix: + action = self._option_string_actions[option_string] + tup = action, option_string, short_explicit_arg + result.append(tup) + elif option_string.startswith(option_prefix): + action = self._option_string_actions[option_string] + tup = action, option_string, explicit_arg + result.append(tup) + + # shouldn't ever get here + else: + self.error(_('unexpected option string: %s') % option_string) + + # return the collected option tuples + return result + + def _get_nargs_pattern(self, action): + # in all examples below, we have to allow for '--' args + # which are represented as '-' in the pattern + nargs = action.nargs + + # the default (None) is assumed to be a single argument + if nargs is None: + nargs_pattern = '(-*A-*)' + + # allow zero or one arguments + elif nargs == OPTIONAL: + nargs_pattern = '(-*A?-*)' + + # allow zero or more arguments + elif nargs == ZERO_OR_MORE: + nargs_pattern = '(-*[A-]*)' + + # allow one or more arguments + elif nargs == ONE_OR_MORE: + nargs_pattern = '(-*A[A-]*)' + + # allow any number of options or arguments + elif nargs == REMAINDER: + nargs_pattern = '([-AO]*)' + + # allow one argument followed by any number of options or arguments + elif nargs == PARSER: + nargs_pattern = '(-*A[-AO]*)' + + # all others should be integers + else: + nargs_pattern = '(-*%s-*)' % '-*'.join('A' * nargs) + + # if this is an optional action, -- is not allowed + if action.option_strings: + nargs_pattern = nargs_pattern.replace('-*', '') + nargs_pattern = nargs_pattern.replace('-', '') + + # return the pattern + return nargs_pattern + + # ======================== + # Value conversion methods + # ======================== + def _get_values(self, action, arg_strings): + # for everything but PARSER args, strip out '--' + if action.nargs not in [PARSER, REMAINDER]: + arg_strings = [s for s in arg_strings if s != '--'] + + # optional argument produces a default when not present + if not arg_strings and action.nargs == OPTIONAL: + if action.option_strings: + value = action.const + else: + value = action.default + if isinstance(value, basestring): + value = self._get_value(action, value) + self._check_value(action, value) + + # when nargs='*' on a positional, if there were no command-line + # args, use the default if it is anything other than None + elif (not arg_strings and action.nargs == ZERO_OR_MORE and + not action.option_strings): + if action.default is not None: + value = action.default + else: + value = arg_strings + self._check_value(action, value) + + # single argument or optional argument produces a single value + elif len(arg_strings) == 1 and action.nargs in [None, OPTIONAL]: + arg_string, = arg_strings + value = self._get_value(action, arg_string) + self._check_value(action, value) + + # REMAINDER arguments convert all values, checking none + elif action.nargs == REMAINDER: + value = [self._get_value(action, v) for v in arg_strings] + + # PARSER arguments convert all values, but check only the first + elif action.nargs == PARSER: + value = [self._get_value(action, v) for v in arg_strings] + self._check_value(action, value[0]) + + # all other types of nargs produce a list + else: + value = [self._get_value(action, v) for v in arg_strings] + for v in value: + self._check_value(action, v) + + # return the converted value + return value + + def _get_value(self, action, arg_string): + type_func = self._registry_get('type', action.type, action.type) + if not _callable(type_func): + msg = _('%r is not callable') + raise ArgumentError(action, msg % type_func) + + # convert the value to the appropriate type + try: + result = type_func(arg_string) + + # ArgumentTypeErrors indicate errors + except ArgumentTypeError: + name = getattr(action.type, '__name__', repr(action.type)) + msg = str(_sys.exc_info()[1]) + raise ArgumentError(action, msg) + + # TypeErrors or ValueErrors also indicate errors + except (TypeError, ValueError): + name = getattr(action.type, '__name__', repr(action.type)) + msg = _('invalid %s value: %r') + raise ArgumentError(action, msg % (name, arg_string)) + + # return the converted value + return result + + def _check_value(self, action, value): + # converted value must be one of the choices (if specified) + if action.choices is not None and value not in action.choices: + tup = value, ', '.join(map(repr, action.choices)) + msg = _('invalid choice: %r (choose from %s)') % tup + raise ArgumentError(action, msg) + + # ======================= + # Help-formatting methods + # ======================= + def format_usage(self): + formatter = self._get_formatter() + formatter.add_usage(self.usage, self._actions, + self._mutually_exclusive_groups) + return formatter.format_help() + + def format_help(self): + formatter = self._get_formatter() + + # usage + formatter.add_usage(self.usage, self._actions, + self._mutually_exclusive_groups) + + # description + formatter.add_text(self.description) + + # positionals, optionals and user-defined groups + for action_group in self._action_groups: + formatter.start_section(action_group.title) + formatter.add_text(action_group.description) + formatter.add_arguments(action_group._group_actions) + formatter.end_section() + + # epilog + formatter.add_text(self.epilog) + + # determine help from format above + return formatter.format_help() + + def format_version(self): + import warnings + warnings.warn( + 'The format_version method is deprecated -- the "version" ' + 'argument to ArgumentParser is no longer supported.', + DeprecationWarning) + formatter = self._get_formatter() + formatter.add_text(self.version) + return formatter.format_help() + + def _get_formatter(self): + return self.formatter_class(prog=self.prog) + + # ===================== + # Help-printing methods + # ===================== + def print_usage(self, file=None): + if file is None: + file = _sys.stdout + self._print_message(self.format_usage(), file) + + def print_help(self, file=None): + if file is None: + file = _sys.stdout + self._print_message(self.format_help(), file) + + def print_version(self, file=None): + import warnings + warnings.warn( + 'The print_version method is deprecated -- the "version" ' + 'argument to ArgumentParser is no longer supported.', + DeprecationWarning) + self._print_message(self.format_version(), file) + + def _print_message(self, message, file=None): + if message: + if file is None: + file = _sys.stderr + file.write(message) + + # =============== + # Exiting methods + # =============== + def exit(self, status=0, message=None): + if message: + self._print_message(message, _sys.stderr) + _sys.exit(status) + + def error(self, message): + """error(message: string) + + Prints a usage message incorporating the message to stderr and + exits. + + If you override this in a subclass, it should not return -- it + should either exit or raise an exception. + """ + self.print_usage(_sys.stderr) + self.exit(2, _('%s: error: %s\n') % (self.prog, message)) -- 2.13.5 Add the scripts/ directory to sys.path so Python 2.6 will be able to import argparse. Cc: Fam Zheng <famz@redhat.com> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com> Acked-by: John Snow <jsnow@redhat.com> Acked-by: Fam Zheng <famz@redhat.com> Message-id: 20170825155732.15665-3-stefanha@redhat.com Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com> --- tests/docker/docker.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/tests/docker/docker.py b/tests/docker/docker.py index XXXXXXX..XXXXXXX 100755 --- a/tests/docker/docker.py +++ b/tests/docker/docker.py @@ -XXX,XX +XXX,XX @@ import os import sys +sys.path.append(os.path.join(os.path.dirname(__file__), + '..', '..', 'scripts')) +import argparse import subprocess import json import hashlib import atexit import uuid -import argparse import tempfile import re import signal -- 2.13.5 Add the scripts/ directory to sys.path so Python 2.6 will be able to import argparse. Cc: Daniel P. Berrange <berrange@redhat.com> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com> Acked-by: John Snow <jsnow@redhat.com> Acked-by: Fam Zheng <famz@redhat.com> Message-id: 20170825155732.15665-4-stefanha@redhat.com Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com> --- tests/migration/guestperf/shell.py | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/tests/migration/guestperf/shell.py b/tests/migration/guestperf/shell.py index XXXXXXX..XXXXXXX 100644 --- a/tests/migration/guestperf/shell.py +++ b/tests/migration/guestperf/shell.py @@ -XXX,XX +XXX,XX @@ # -import argparse -import fnmatch import os import os.path -import platform import sys +sys.path.append(os.path.join(os.path.dirname(__file__), + '..', '..', '..', 'scripts')) +import argparse +import fnmatch +import platform from guestperf.hardware import Hardware from guestperf.engine import Engine -- 2.13.5 From: Fred Rolland <rollandf@gmail.com> Update doc with the usage of UUID for initiator name. Related-To: https://bugzilla.redhat.com/1006468 Signed-off-by: Fred Rolland <frolland@redhat.com> Message-id: 20170823084830.30500-1-frolland@redhat.com Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com> --- qemu-doc.texi | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/qemu-doc.texi b/qemu-doc.texi index XXXXXXX..XXXXXXX 100644 --- a/qemu-doc.texi +++ b/qemu-doc.texi @@ -XXX,XX +XXX,XX @@ in a configuration file provided via '-readconfig' or directly on the command line. If the initiator-name is not specified qemu will use a default name -of 'iqn.2008-11.org.linux-kvm[:<name>'] where <name> is the name of the +of 'iqn.2008-11.org.linux-kvm[:<uuid>'] where <uuid> is the UUID of the +virtual machine. If the UUID is not specified qemu will use +'iqn.2008-11.org.linux-kvm[:<name>'] where <name> is the name of the virtual machine. - @example Setting a specific initiator name to use when logging in to the target -iscsi initiator-name=iqn.qemu.test:my-initiator -- 2.13.5 Most qcow2 files are uncompressed so it is wasteful to allocate (32 + 1) * cluster_size + 512 bytes upfront. Allocate s->cluster_cache and s->cluster_data when the first read operation is performance on a compressed cluster. The buffers are freed in .bdrv_close(). .bdrv_open() no longer has any code paths that can allocate these buffers, so remove the free functions in the error code path. This patch can result in significant memory savings when many qcow2 disks are attached or backing file chains are long: Before 12.81% (1,023,193,088B) After 5.36% (393,893,888B) Reported-by: Alexey Kardashevskiy <aik@ozlabs.ru> Tested-by: Alexey Kardashevskiy <aik@ozlabs.ru> Reviewed-by: Eric Blake <eblake@redhat.com> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com> Message-id: 20170821135530.32344-1-stefanha@redhat.com Cc: Kevin Wolf <kwolf@redhat.com> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com> --- block/qcow2-cluster.c | 17 +++++++++++++++++ block/qcow2.c | 12 ------------ 2 files changed, 17 insertions(+), 12 deletions(-) diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2-cluster.c +++ b/block/qcow2-cluster.c @@ -XXX,XX +XXX,XX @@ int qcow2_decompress_cluster(BlockDriverState *bs, uint64_t cluster_offset) nb_csectors = ((cluster_offset >> s->csize_shift) & s->csize_mask) + 1; sector_offset = coffset & 511; csize = nb_csectors * 512 - sector_offset; + + /* Allocate buffers on first decompress operation, most images are + * uncompressed and the memory overhead can be avoided. The buffers + * are freed in .bdrv_close(). + */ + if (!s->cluster_data) { + /* one more sector for decompressed data alignment */ + s->cluster_data = qemu_try_blockalign(bs->file->bs, + QCOW_MAX_CRYPT_CLUSTERS * s->cluster_size + 512); + if (!s->cluster_data) { + return -ENOMEM; + } + } + if (!s->cluster_cache) { + s->cluster_cache = g_malloc(s->cluster_size); + } + BLKDBG_EVENT(bs->file, BLKDBG_READ_COMPRESSED); ret = bdrv_read(bs->file, coffset >> 9, s->cluster_data, nb_csectors); diff --git a/block/qcow2.c b/block/qcow2.c index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -XXX,XX +XXX,XX @@ static int qcow2_do_open(BlockDriverState *bs, QDict *options, int flags, goto fail; } - s->cluster_cache = g_malloc(s->cluster_size); - /* one more sector for decompressed data alignment */ - s->cluster_data = qemu_try_blockalign(bs->file->bs, QCOW_MAX_CRYPT_CLUSTERS - * s->cluster_size + 512); - if (s->cluster_data == NULL) { - error_setg(errp, "Could not allocate temporary cluster buffer"); - ret = -ENOMEM; - goto fail; - } - s->cluster_cache_offset = -1; s->flags = flags; @@ -XXX,XX +XXX,XX @@ static int qcow2_do_open(BlockDriverState *bs, QDict *options, int flags, if (s->refcount_block_cache) { qcow2_cache_destroy(bs, s->refcount_block_cache); } - g_free(s->cluster_cache); - qemu_vfree(s->cluster_data); qcrypto_block_free(s->crypto); qapi_free_QCryptoBlockOpenOptions(s->crypto_opts); return ret; -- 2.13.5 The following changes since commit 31ee895047bdcf7387e3570cbd2a473c6f744b08: Merge remote-tracking branch 'remotes/jasowang/tags/net-pull-request' into staging (2021-01-25 15:56:13 +0000) are available in the Git repository at: https://github.com/XanClic/qemu.git tags/pull-block-2021-01-26 for you to fetch changes up to bb24cdc5efee580e81f71c5ff0fd980f2cc179d0: iotests/178: Pass value to invalid option (2021-01-26 14:36:37 +0100) ---------------------------------------------------------------- Block patches: - Make backup block jobs use asynchronous requests with the block-copy module - Use COR filter node for stream block jobs - Make coroutine-sigaltstack’s qemu_coroutine_new() function thread-safe - Report error string when file locking fails with an unexpected errno - iotest fixes, additions, and some refactoring ---------------------------------------------------------------- Alberto Garcia (1): iotests: Add test for the regression fixed in c8bf9a9169 Andrey Shinkevich (10): copy-on-read: support preadv/pwritev_part functions block: add API function to insert a node copy-on-read: add filter drop function qapi: add filter-node-name to block-stream qapi: copy-on-read filter: add 'bottom' option iotests: add #310 to test bottom node in COR driver block: include supported_read_flags into BDS structure copy-on-read: skip non-guest reads if no copy needed stream: rework backing-file changing block: apply COR-filter to block-stream jobs David Edmondson (1): block: report errno when flock fcntl fails Max Reitz (14): iotests.py: Assume a couple of variables as given iotests/297: Rewrite in Python and extend reach iotests: Move try_remove to iotests.py iotests/129: Remove test images in tearDown() iotests/129: Do not check @busy iotests/129: Use throttle node iotests/129: Actually test a commit job iotests/129: Limit mirror job's buffer size iotests/129: Clean up pylint and mypy complaints iotests/300: Clean up pylint and mypy complaints coroutine-sigaltstack: Add SIGUSR2 mutex iotests/129: Limit backup's max-chunk/max-workers iotests/118: Drop 'change' test iotests/178: Pass value to invalid option Vladimir Sementsov-Ogievskiy (27): iotests: fix _check_o_direct qapi: block-stream: add "bottom" argument iotests: 30: prepare to COR filter insertion by stream job block/stream: add s->target_bs qapi: backup: add perf.use-copy-range parameter block/block-copy: More explicit call_state block/block-copy: implement block_copy_async block/block-copy: add max_chunk and max_workers parameters block/block-copy: add list of all call-states block/block-copy: add ratelimit to block-copy block/block-copy: add block_copy_cancel blockjob: add set_speed to BlockJobDriver job: call job_enter from job_pause qapi: backup: add max-chunk and max-workers to x-perf struct iotests: 56: prepare for backup over block-copy iotests: 185: prepare for backup over block-copy iotests: 219: prepare for backup over block-copy iotests: 257: prepare for backup over block-copy block/block-copy: make progress_bytes_callback optional block/backup: drop extra gotos from backup_run() backup: move to block-copy qapi: backup: disable copy_range by default block/block-copy: drop unused block_copy_set_progress_callback() block/block-copy: drop unused argument of block_copy() simplebench/bench_block_job: use correct shebang line with python3 simplebench: bench_block_job: add cmd_options argument simplebench: add bench-backup.py qapi/block-core.json | 66 +++++- block/backup-top.h | 1 + block/copy-on-read.h | 32 +++ include/block/block-copy.h | 61 ++++- include/block/block.h | 10 +- include/block/block_int.h | 15 +- include/block/blockjob_int.h | 2 + block.c | 25 ++ block/backup-top.c | 6 +- block/backup.c | 233 ++++++++++++------- block/block-copy.c | 227 +++++++++++++++--- block/copy-on-read.c | 184 ++++++++++++++- block/file-posix.c | 38 ++- block/io.c | 10 +- block/monitor/block-hmp-cmds.c | 7 +- block/replication.c | 2 + block/stream.c | 185 +++++++++------ blockdev.c | 83 +++++-- blockjob.c | 6 + job.c | 3 + util/coroutine-sigaltstack.c | 9 + scripts/simplebench/bench-backup.py | 167 ++++++++++++++ scripts/simplebench/bench-example.py | 2 +- scripts/simplebench/bench_block_job.py | 13 +- tests/qemu-iotests/030 | 12 +- tests/qemu-iotests/056 | 9 +- tests/qemu-iotests/109.out | 24 ++ tests/qemu-iotests/118 | 20 +- tests/qemu-iotests/118.out | 4 +- tests/qemu-iotests/124 | 8 +- tests/qemu-iotests/129 | 79 ++++--- tests/qemu-iotests/141.out | 2 +- tests/qemu-iotests/178 | 2 +- tests/qemu-iotests/178.out.qcow2 | 2 +- tests/qemu-iotests/178.out.raw | 2 +- tests/qemu-iotests/185 | 3 +- tests/qemu-iotests/185.out | 3 +- tests/qemu-iotests/219 | 13 +- tests/qemu-iotests/245 | 20 +- tests/qemu-iotests/257 | 1 + tests/qemu-iotests/257.out | 306 ++++++++++++------------- tests/qemu-iotests/297 | 112 +++++++-- tests/qemu-iotests/297.out | 5 +- tests/qemu-iotests/300 | 19 +- tests/qemu-iotests/310 | 117 ++++++++++ tests/qemu-iotests/310.out | 15 ++ tests/qemu-iotests/313 | 104 +++++++++ tests/qemu-iotests/313.out | 29 +++ tests/qemu-iotests/common.rc | 7 +- tests/qemu-iotests/group | 2 + tests/qemu-iotests/iotests.py | 37 +-- 51 files changed, 1797 insertions(+), 547 deletions(-) create mode 100644 block/copy-on-read.h create mode 100755 scripts/simplebench/bench-backup.py create mode 100755 tests/qemu-iotests/310 create mode 100644 tests/qemu-iotests/310.out create mode 100755 tests/qemu-iotests/313 create mode 100644 tests/qemu-iotests/313.out -- 2.29.2 From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Unfortunately commit "iotests: handle tmpfs" breaks running iotests with -nbd -nocache, as _check_o_direct tries to create $TEST_IMG.test_o_direct, but in case of nbd TEST_IMG is something like nbd+unix:///... , and test fails with message qemu-img: nbd+unix:///?socket[...]test_o_direct: Protocol driver 'nbd' does not support image creation, and opening the image failed: Failed to connect to '/tmp/tmp.[...]/nbd/test_o_direct': No such file or directory Use TEST_DIR instead. Fixes: cfdca2b9f9d4ca26bb2b2dfe8de3149092e39170 Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Message-Id: <20201218182012.47607-1-vsementsov@virtuozzo.com> Signed-off-by: Max Reitz <mreitz@redhat.com> --- tests/qemu-iotests/common.rc | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/tests/qemu-iotests/common.rc b/tests/qemu-iotests/common.rc index XXXXXXX..XXXXXXX 100644 --- a/tests/qemu-iotests/common.rc +++ b/tests/qemu-iotests/common.rc @@ -XXX,XX +XXX,XX @@ _supported_cache_modes() # Check whether the filesystem supports O_DIRECT _check_o_direct() { - $QEMU_IMG create -f raw "$TEST_IMG".test_o_direct 1M > /dev/null - out=$($QEMU_IO -f raw -t none -c quit "$TEST_IMG".test_o_direct 2>&1) - rm -f "$TEST_IMG".test_o_direct + testfile="$TEST_DIR"/_check_o_direct + $QEMU_IMG create -f raw "$testfile" 1M > /dev/null + out=$($QEMU_IO -f raw -t none -c quit "$testfile" 2>&1) + rm -f "$testfile" [[ "$out" != *"O_DIRECT"* ]] } -- 2.29.2 From: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com> Add support for the recently introduced functions bdrv_co_preadv_part() and bdrv_co_pwritev_part() to the COR-filter driver. Signed-off-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Message-Id: <20201216061703.70908-2-vsementsov@virtuozzo.com> Signed-off-by: Max Reitz <mreitz@redhat.com> --- block/copy-on-read.c | 28 ++++++++++++++++------------ 1 file changed, 16 insertions(+), 12 deletions(-) diff --git a/block/copy-on-read.c b/block/copy-on-read.c index XXXXXXX..XXXXXXX 100644 --- a/block/copy-on-read.c +++ b/block/copy-on-read.c @@ -XXX,XX +XXX,XX @@ static int64_t cor_getlength(BlockDriverState *bs) } -static int coroutine_fn cor_co_preadv(BlockDriverState *bs, - uint64_t offset, uint64_t bytes, - QEMUIOVector *qiov, int flags) +static int coroutine_fn cor_co_preadv_part(BlockDriverState *bs, + uint64_t offset, uint64_t bytes, + QEMUIOVector *qiov, + size_t qiov_offset, + int flags) { - return bdrv_co_preadv(bs->file, offset, bytes, qiov, - flags | BDRV_REQ_COPY_ON_READ); + return bdrv_co_preadv_part(bs->file, offset, bytes, qiov, qiov_offset, + flags | BDRV_REQ_COPY_ON_READ); } -static int coroutine_fn cor_co_pwritev(BlockDriverState *bs, - uint64_t offset, uint64_t bytes, - QEMUIOVector *qiov, int flags) +static int coroutine_fn cor_co_pwritev_part(BlockDriverState *bs, + uint64_t offset, + uint64_t bytes, + QEMUIOVector *qiov, + size_t qiov_offset, int flags) { - - return bdrv_co_pwritev(bs->file, offset, bytes, qiov, flags); + return bdrv_co_pwritev_part(bs->file, offset, bytes, qiov, qiov_offset, + flags); } @@ -XXX,XX +XXX,XX @@ static BlockDriver bdrv_copy_on_read = { .bdrv_getlength = cor_getlength, - .bdrv_co_preadv = cor_co_preadv, - .bdrv_co_pwritev = cor_co_pwritev, + .bdrv_co_preadv_part = cor_co_preadv_part, + .bdrv_co_pwritev_part = cor_co_pwritev_part, .bdrv_co_pwrite_zeroes = cor_co_pwrite_zeroes, .bdrv_co_pdiscard = cor_co_pdiscard, .bdrv_co_pwritev_compressed = cor_co_pwritev_compressed, -- 2.29.2 From: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com> Provide API for insertion a node to backing chain. Suggested-by: Max Reitz <mreitz@redhat.com> Signed-off-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Message-Id: <20201216061703.70908-3-vsementsov@virtuozzo.com> Signed-off-by: Max Reitz <mreitz@redhat.com> --- include/block/block.h | 2 ++ block.c | 25 +++++++++++++++++++++++++ 2 files changed, 27 insertions(+) diff --git a/include/block/block.h b/include/block/block.h index XXXXXXX..XXXXXXX 100644 --- a/include/block/block.h +++ b/include/block/block.h @@ -XXX,XX +XXX,XX @@ void bdrv_append(BlockDriverState *bs_new, BlockDriverState *bs_top, Error **errp); void bdrv_replace_node(BlockDriverState *from, BlockDriverState *to, Error **errp); +BlockDriverState *bdrv_insert_node(BlockDriverState *bs, QDict *node_options, + int flags, Error **errp); int bdrv_parse_aio(const char *mode, int *flags); int bdrv_parse_cache_mode(const char *mode, int *flags, bool *writethrough); diff --git a/block.c b/block.c index XXXXXXX..XXXXXXX 100644 --- a/block.c +++ b/block.c @@ -XXX,XX +XXX,XX @@ static void bdrv_delete(BlockDriverState *bs) g_free(bs); } +BlockDriverState *bdrv_insert_node(BlockDriverState *bs, QDict *node_options, + int flags, Error **errp) +{ + BlockDriverState *new_node_bs; + Error *local_err = NULL; + + new_node_bs = bdrv_open(NULL, NULL, node_options, flags, errp); + if (new_node_bs == NULL) { + error_prepend(errp, "Could not create node: "); + return NULL; + } + + bdrv_drained_begin(bs); + bdrv_replace_node(bs, new_node_bs, &local_err); + bdrv_drained_end(bs); + + if (local_err) { + bdrv_unref(new_node_bs); + error_propagate(errp, local_err); + return NULL; + } + + return new_node_bs; +} + /* * Run consistency checks on an image * -- 2.29.2 From: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com> Provide API for the COR-filter removal. Also, drop the filter child permissions for an inactive state when the filter node is being removed. To insert the filter, the block generic layer function bdrv_insert_node() can be used. The new function bdrv_cor_filter_drop() may be considered as an intermediate solution before the QEMU permission update system has overhauled. Then we are able to implement the API function bdrv_remove_node() on the block generic layer. Signed-off-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Message-Id: <20201216061703.70908-4-vsementsov@virtuozzo.com> Signed-off-by: Max Reitz <mreitz@redhat.com> --- block/copy-on-read.h | 32 +++++++++++++++++++++++++ block/copy-on-read.c | 56 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 88 insertions(+) create mode 100644 block/copy-on-read.h diff --git a/block/copy-on-read.h b/block/copy-on-read.h new file mode 100644 index XXXXXXX..XXXXXXX --- /dev/null +++ b/block/copy-on-read.h @@ -XXX,XX +XXX,XX @@ +/* + * Copy-on-read filter block driver + * + * The filter driver performs Copy-On-Read (COR) operations + * + * Copyright (c) 2018-2020 Virtuozzo International GmbH. + * + * Author: + * Andrey Shinkevich <andrey.shinkevich@virtuozzo.com> + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; either version 2 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program. If not, see <http://www.gnu.org/licenses/>. + */ + +#ifndef BLOCK_COPY_ON_READ +#define BLOCK_COPY_ON_READ + +#include "block/block_int.h" + +void bdrv_cor_filter_drop(BlockDriverState *cor_filter_bs); + +#endif /* BLOCK_COPY_ON_READ */ diff --git a/block/copy-on-read.c b/block/copy-on-read.c index XXXXXXX..XXXXXXX 100644 --- a/block/copy-on-read.c +++ b/block/copy-on-read.c @@ -XXX,XX +XXX,XX @@ #include "qemu/osdep.h" #include "block/block_int.h" #include "qemu/module.h" +#include "qapi/error.h" +#include "block/copy-on-read.h" + + +typedef struct BDRVStateCOR { + bool active; +} BDRVStateCOR; static int cor_open(BlockDriverState *bs, QDict *options, int flags, Error **errp) { + BDRVStateCOR *state = bs->opaque; + bs->file = bdrv_open_child(NULL, options, "file", bs, &child_of_bds, BDRV_CHILD_FILTERED | BDRV_CHILD_PRIMARY, false, errp); @@ -XXX,XX +XXX,XX @@ static int cor_open(BlockDriverState *bs, QDict *options, int flags, ((BDRV_REQ_FUA | BDRV_REQ_MAY_UNMAP | BDRV_REQ_NO_FALLBACK) & bs->file->bs->supported_zero_flags); + state->active = true; + + /* + * We don't need to call bdrv_child_refresh_perms() now as the permissions + * will be updated later when the filter node gets its parent. + */ + return 0; } @@ -XXX,XX +XXX,XX @@ static void cor_child_perm(BlockDriverState *bs, BdrvChild *c, uint64_t perm, uint64_t shared, uint64_t *nperm, uint64_t *nshared) { + BDRVStateCOR *s = bs->opaque; + + if (!s->active) { + /* + * While the filter is being removed + */ + *nperm = 0; + *nshared = BLK_PERM_ALL; + return; + } + *nperm = perm & PERM_PASSTHROUGH; *nshared = (shared & PERM_PASSTHROUGH) | PERM_UNCHANGED; @@ -XXX,XX +XXX,XX @@ static void cor_lock_medium(BlockDriverState *bs, bool locked) static BlockDriver bdrv_copy_on_read = { .format_name = "copy-on-read", + .instance_size = sizeof(BDRVStateCOR), .bdrv_open = cor_open, .bdrv_child_perm = cor_child_perm, @@ -XXX,XX +XXX,XX @@ static BlockDriver bdrv_copy_on_read = { .is_filter = true, }; + +void bdrv_cor_filter_drop(BlockDriverState *cor_filter_bs) +{ + BdrvChild *child; + BlockDriverState *bs; + BDRVStateCOR *s = cor_filter_bs->opaque; + + child = bdrv_filter_child(cor_filter_bs); + if (!child) { + return; + } + bs = child->bs; + + /* Retain the BDS until we complete the graph change. */ + bdrv_ref(bs); + /* Hold a guest back from writing while permissions are being reset. */ + bdrv_drained_begin(bs); + /* Drop permissions before the graph change. */ + s->active = false; + bdrv_child_refresh_perms(cor_filter_bs, child, &error_abort); + bdrv_replace_node(cor_filter_bs, bs, &error_abort); + + bdrv_drained_end(bs); + bdrv_unref(bs); + bdrv_unref(cor_filter_bs); +} + + static void bdrv_copy_on_read_init(void) { bdrv_register(&bdrv_copy_on_read); -- 2.29.2 From: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com> Provide the possibility to pass the 'filter-node-name' parameter to the block-stream job as it is done for the commit block job. Signed-off-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> [vsementsov: comment indentation, s/Since: 5.2/Since: 6.0/] Reviewed-by: Max Reitz <mreitz@redhat.com> Message-Id: <20201216061703.70908-5-vsementsov@virtuozzo.com> [mreitz: s/commit/stream/] Signed-off-by: Max Reitz <mreitz@redhat.com> --- qapi/block-core.json | 6 ++++++ include/block/block_int.h | 7 ++++++- block/monitor/block-hmp-cmds.c | 4 ++-- block/stream.c | 4 +++- blockdev.c | 4 +++- 5 files changed, 20 insertions(+), 5 deletions(-) diff --git a/qapi/block-core.json b/qapi/block-core.json index XXXXXXX..XXXXXXX 100644 --- a/qapi/block-core.json +++ b/qapi/block-core.json @@ -XXX,XX +XXX,XX @@ # 'stop' and 'enospc' can only be used if the block device # supports io-status (see BlockInfo). Since 1.3. # +# @filter-node-name: the node name that should be assigned to the +# filter driver that the stream job inserts into the graph +# above @device. If this option is not given, a node name is +# autogenerated. (Since: 6.0) +# # @auto-finalize: When false, this job will wait in a PENDING state after it has # finished its work, waiting for @block-job-finalize before # making any block graph changes. @@ -XXX,XX +XXX,XX @@ 'data': { '*job-id': 'str', 'device': 'str', '*base': 'str', '*base-node': 'str', '*backing-file': 'str', '*speed': 'int', '*on-error': 'BlockdevOnError', + '*filter-node-name': 'str', '*auto-finalize': 'bool', '*auto-dismiss': 'bool' } } ## diff --git a/include/block/block_int.h b/include/block/block_int.h index XXXXXXX..XXXXXXX 100644 --- a/include/block/block_int.h +++ b/include/block/block_int.h @@ -XXX,XX +XXX,XX @@ int is_windows_drive(const char *filename); * See @BlockJobCreateFlags * @speed: The maximum speed, in bytes per second, or 0 for unlimited. * @on_error: The action to take upon error. + * @filter_node_name: The node name that should be assigned to the filter + * driver that the stream job inserts into the graph above + * @bs. NULL means that a node name should be autogenerated. * @errp: Error object. * * Start a streaming operation on @bs. Clusters that are unallocated @@ -XXX,XX +XXX,XX @@ int is_windows_drive(const char *filename); void stream_start(const char *job_id, BlockDriverState *bs, BlockDriverState *base, const char *backing_file_str, int creation_flags, int64_t speed, - BlockdevOnError on_error, Error **errp); + BlockdevOnError on_error, + const char *filter_node_name, + Error **errp); /** * commit_start: diff --git a/block/monitor/block-hmp-cmds.c b/block/monitor/block-hmp-cmds.c index XXXXXXX..XXXXXXX 100644 --- a/block/monitor/block-hmp-cmds.c +++ b/block/monitor/block-hmp-cmds.c @@ -XXX,XX +XXX,XX @@ void hmp_block_stream(Monitor *mon, const QDict *qdict) qmp_block_stream(true, device, device, base != NULL, base, false, NULL, false, NULL, qdict_haskey(qdict, "speed"), speed, true, - BLOCKDEV_ON_ERROR_REPORT, false, false, false, false, - &error); + BLOCKDEV_ON_ERROR_REPORT, false, NULL, false, false, false, + false, &error); hmp_handle_error(mon, error); } diff --git a/block/stream.c b/block/stream.c index XXXXXXX..XXXXXXX 100644 --- a/block/stream.c +++ b/block/stream.c @@ -XXX,XX +XXX,XX @@ static const BlockJobDriver stream_job_driver = { void stream_start(const char *job_id, BlockDriverState *bs, BlockDriverState *base, const char *backing_file_str, int creation_flags, int64_t speed, - BlockdevOnError on_error, Error **errp) + BlockdevOnError on_error, + const char *filter_node_name, + Error **errp) { StreamBlockJob *s; BlockDriverState *iter; diff --git a/blockdev.c b/blockdev.c index XXXXXXX..XXXXXXX 100644 --- a/blockdev.c +++ b/blockdev.c @@ -XXX,XX +XXX,XX @@ void qmp_block_stream(bool has_job_id, const char *job_id, const char *device, bool has_backing_file, const char *backing_file, bool has_speed, int64_t speed, bool has_on_error, BlockdevOnError on_error, + bool has_filter_node_name, const char *filter_node_name, bool has_auto_finalize, bool auto_finalize, bool has_auto_dismiss, bool auto_dismiss, Error **errp) @@ -XXX,XX +XXX,XX @@ void qmp_block_stream(bool has_job_id, const char *job_id, const char *device, } stream_start(has_job_id ? job_id : NULL, bs, base_bs, base_name, - job_flags, has_speed ? speed : 0, on_error, &local_err); + job_flags, has_speed ? speed : 0, on_error, + filter_node_name, &local_err); if (local_err) { error_propagate(errp, local_err); goto out; -- 2.29.2 From: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com> Add an option to limit copy-on-read operations to specified sub-chain of backing-chain, to make copy-on-read filter useful for block-stream job. Suggested-by: Max Reitz <mreitz@redhat.com> Suggested-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Signed-off-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> [vsementsov: change subject, modified to freeze the chain, do some fixes] Message-Id: <20201216061703.70908-6-vsementsov@virtuozzo.com> Signed-off-by: Max Reitz <mreitz@redhat.com> --- qapi/block-core.json | 20 ++++++++- block/copy-on-read.c | 98 +++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 115 insertions(+), 3 deletions(-) diff --git a/qapi/block-core.json b/qapi/block-core.json index XXXXXXX..XXXXXXX 100644 --- a/qapi/block-core.json +++ b/qapi/block-core.json @@ -XXX,XX +XXX,XX @@ 'data': { 'throttle-group': 'str', 'file' : 'BlockdevRef' } } + +## +# @BlockdevOptionsCor: +# +# Driver specific block device options for the copy-on-read driver. +# +# @bottom: The name of a non-filter node (allocation-bearing layer) that +# limits the COR operations in the backing chain (inclusive), so +# that no data below this node will be copied by this filter. +# If option is absent, the limit is not applied, so that data +# from all backing layers may be copied. +# +# Since: 6.0 +## +{ 'struct': 'BlockdevOptionsCor', + 'base': 'BlockdevOptionsGenericFormat', + 'data': { '*bottom': 'str' } } + ## # @BlockdevOptions: # @@ -XXX,XX +XXX,XX @@ 'bochs': 'BlockdevOptionsGenericFormat', 'cloop': 'BlockdevOptionsGenericFormat', 'compress': 'BlockdevOptionsGenericFormat', - 'copy-on-read':'BlockdevOptionsGenericFormat', + 'copy-on-read':'BlockdevOptionsCor', 'dmg': 'BlockdevOptionsGenericFormat', 'file': 'BlockdevOptionsFile', 'ftp': 'BlockdevOptionsCurlFtp', diff --git a/block/copy-on-read.c b/block/copy-on-read.c index XXXXXXX..XXXXXXX 100644 --- a/block/copy-on-read.c +++ b/block/copy-on-read.c @@ -XXX,XX +XXX,XX @@ #include "block/block_int.h" #include "qemu/module.h" #include "qapi/error.h" +#include "qapi/qmp/qdict.h" #include "block/copy-on-read.h" typedef struct BDRVStateCOR { bool active; + BlockDriverState *bottom_bs; + bool chain_frozen; } BDRVStateCOR; static int cor_open(BlockDriverState *bs, QDict *options, int flags, Error **errp) { + BlockDriverState *bottom_bs = NULL; BDRVStateCOR *state = bs->opaque; + /* Find a bottom node name, if any */ + const char *bottom_node = qdict_get_try_str(options, "bottom"); bs->file = bdrv_open_child(NULL, options, "file", bs, &child_of_bds, BDRV_CHILD_FILTERED | BDRV_CHILD_PRIMARY, @@ -XXX,XX +XXX,XX @@ static int cor_open(BlockDriverState *bs, QDict *options, int flags, ((BDRV_REQ_FUA | BDRV_REQ_MAY_UNMAP | BDRV_REQ_NO_FALLBACK) & bs->file->bs->supported_zero_flags); + if (bottom_node) { + bottom_bs = bdrv_find_node(bottom_node); + if (!bottom_bs) { + error_setg(errp, "Bottom node '%s' not found", bottom_node); + qdict_del(options, "bottom"); + return -EINVAL; + } + qdict_del(options, "bottom"); + + if (!bottom_bs->drv) { + error_setg(errp, "Bottom node '%s' not opened", bottom_node); + return -EINVAL; + } + + if (bottom_bs->drv->is_filter) { + error_setg(errp, "Bottom node '%s' is a filter", bottom_node); + return -EINVAL; + } + + if (bdrv_freeze_backing_chain(bs, bottom_bs, errp) < 0) { + return -EINVAL; + } + state->chain_frozen = true; + + /* + * We do freeze the chain, so it shouldn't be removed. Still, storing a + * pointer worth bdrv_ref(). + */ + bdrv_ref(bottom_bs); + } state->active = true; + state->bottom_bs = bottom_bs; /* * We don't need to call bdrv_child_refresh_perms() now as the permissions @@ -XXX,XX +XXX,XX @@ static int coroutine_fn cor_co_preadv_part(BlockDriverState *bs, size_t qiov_offset, int flags) { - return bdrv_co_preadv_part(bs->file, offset, bytes, qiov, qiov_offset, - flags | BDRV_REQ_COPY_ON_READ); + int64_t n; + int local_flags; + int ret; + BDRVStateCOR *state = bs->opaque; + + if (!state->bottom_bs) { + return bdrv_co_preadv_part(bs->file, offset, bytes, qiov, qiov_offset, + flags | BDRV_REQ_COPY_ON_READ); + } + + while (bytes) { + local_flags = flags; + + /* In case of failure, try to copy-on-read anyway */ + ret = bdrv_is_allocated(bs->file->bs, offset, bytes, &n); + if (ret <= 0) { + ret = bdrv_is_allocated_above(bdrv_backing_chain_next(bs->file->bs), + state->bottom_bs, true, offset, + n, &n); + if (ret > 0 || ret < 0) { + local_flags |= BDRV_REQ_COPY_ON_READ; + } + /* Finish earlier if the end of a backing file has been reached */ + if (n == 0) { + break; + } + } + + ret = bdrv_co_preadv_part(bs->file, offset, n, qiov, qiov_offset, + local_flags); + if (ret < 0) { + return ret; + } + + offset += n; + qiov_offset += n; + bytes -= n; + } + + return 0; } @@ -XXX,XX +XXX,XX @@ static void cor_lock_medium(BlockDriverState *bs, bool locked) } +static void cor_close(BlockDriverState *bs) +{ + BDRVStateCOR *s = bs->opaque; + + if (s->chain_frozen) { + s->chain_frozen = false; + bdrv_unfreeze_backing_chain(bs, s->bottom_bs); + } + + bdrv_unref(s->bottom_bs); +} + + static BlockDriver bdrv_copy_on_read = { .format_name = "copy-on-read", .instance_size = sizeof(BDRVStateCOR), .bdrv_open = cor_open, + .bdrv_close = cor_close, .bdrv_child_perm = cor_child_perm, .bdrv_getlength = cor_getlength, @@ -XXX,XX +XXX,XX @@ void bdrv_cor_filter_drop(BlockDriverState *cor_filter_bs) bdrv_drained_begin(bs); /* Drop permissions before the graph change. */ s->active = false; + /* unfreeze, as otherwise bdrv_replace_node() will fail */ + if (s->chain_frozen) { + s->chain_frozen = false; + bdrv_unfreeze_backing_chain(cor_filter_bs, s->bottom_bs); + } bdrv_child_refresh_perms(cor_filter_bs, child, &error_abort); bdrv_replace_node(cor_filter_bs, bs, &error_abort); -- 2.29.2 From: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com> The test case #310 is similar to #216 by Max Reitz. The difference is that the test #310 involves a bottom node to the COR filter driver. Signed-off-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> [vsementsov: detach backing to test reads from top, limit to qcow2] Message-Id: <20201216061703.70908-7-vsementsov@virtuozzo.com> [mreitz: Add "# group:" line] Signed-off-by: Max Reitz <mreitz@redhat.com> --- tests/qemu-iotests/310 | 117 +++++++++++++++++++++++++++++++++++++ tests/qemu-iotests/310.out | 15 +++++ tests/qemu-iotests/group | 1 + 3 files changed, 133 insertions(+) create mode 100755 tests/qemu-iotests/310 create mode 100644 tests/qemu-iotests/310.out diff --git a/tests/qemu-iotests/310 b/tests/qemu-iotests/310 new file mode 100755 index XXXXXXX..XXXXXXX --- /dev/null +++ b/tests/qemu-iotests/310 @@ -XXX,XX +XXX,XX @@ +#!/usr/bin/env python3 +# group: rw quick +# +# Copy-on-read tests using a COR filter with a bottom node +# +# Copyright (C) 2018 Red Hat, Inc. +# Copyright (c) 2020 Virtuozzo International GmbH +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# + +import iotests +from iotests import log, qemu_img, qemu_io_silent + +# Need backing file support +iotests.script_initialize(supported_fmts=['qcow2'], + supported_platforms=['linux']) + +log('') +log('=== Copy-on-read across nodes ===') +log('') + +# This test is similar to the 216 one by Max Reitz <mreitz@redhat.com> +# The difference is that this test case involves a bottom node to the +# COR filter driver. + +with iotests.FilePath('base.img') as base_img_path, \ + iotests.FilePath('mid.img') as mid_img_path, \ + iotests.FilePath('top.img') as top_img_path, \ + iotests.VM() as vm: + + log('--- Setting up images ---') + log('') + + assert qemu_img('create', '-f', iotests.imgfmt, base_img_path, '64M') == 0 + assert qemu_io_silent(base_img_path, '-c', 'write -P 1 0M 1M') == 0 + assert qemu_io_silent(base_img_path, '-c', 'write -P 1 3M 1M') == 0 + assert qemu_img('create', '-f', iotests.imgfmt, '-b', base_img_path, + '-F', iotests.imgfmt, mid_img_path) == 0 + assert qemu_io_silent(mid_img_path, '-c', 'write -P 3 2M 1M') == 0 + assert qemu_io_silent(mid_img_path, '-c', 'write -P 3 4M 1M') == 0 + assert qemu_img('create', '-f', iotests.imgfmt, '-b', mid_img_path, + '-F', iotests.imgfmt, top_img_path) == 0 + assert qemu_io_silent(top_img_path, '-c', 'write -P 2 1M 1M') == 0 + +# 0 1 2 3 4 +# top 2 +# mid 3 3 +# base 1 1 + + log('Done') + + log('') + log('--- Doing COR ---') + log('') + + vm.launch() + + log(vm.qmp('blockdev-add', + node_name='node0', + driver='copy-on-read', + bottom='node2', + file={ + 'driver': iotests.imgfmt, + 'file': { + 'driver': 'file', + 'filename': top_img_path + }, + 'backing': { + 'node-name': 'node2', + 'driver': iotests.imgfmt, + 'file': { + 'driver': 'file', + 'filename': mid_img_path + }, + 'backing': { + 'driver': iotests.imgfmt, + 'file': { + 'driver': 'file', + 'filename': base_img_path + } + }, + } + })) + + # Trigger COR + log(vm.qmp('human-monitor-command', + command_line='qemu-io node0 "read 0 5M"')) + + vm.shutdown() + + log('') + log('--- Checking COR result ---') + log('') + + # Detach backing to check that we can read the data from the top level now + assert qemu_img('rebase', '-u', '-b', '', '-f', iotests.imgfmt, + top_img_path) == 0 + + assert qemu_io_silent(top_img_path, '-c', 'read -P 0 0 1M') == 0 + assert qemu_io_silent(top_img_path, '-c', 'read -P 2 1M 1M') == 0 + assert qemu_io_silent(top_img_path, '-c', 'read -P 3 2M 1M') == 0 + assert qemu_io_silent(top_img_path, '-c', 'read -P 0 3M 1M') == 0 + assert qemu_io_silent(top_img_path, '-c', 'read -P 3 4M 1M') == 0 + + log('Done') diff --git a/tests/qemu-iotests/310.out b/tests/qemu-iotests/310.out new file mode 100644 index XXXXXXX..XXXXXXX --- /dev/null +++ b/tests/qemu-iotests/310.out @@ -XXX,XX +XXX,XX @@ + +=== Copy-on-read across nodes === + +--- Setting up images --- + +Done + +--- Doing COR --- + +{"return": {}} +{"return": ""} + +--- Checking COR result --- + +Done diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group index XXXXXXX..XXXXXXX 100644 --- a/tests/qemu-iotests/group +++ b/tests/qemu-iotests/group @@ -XXX,XX +XXX,XX @@ 307 rw quick export 308 rw 309 rw auto quick +310 rw quick 312 rw quick -- 2.29.2 From: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com> Add the new member supported_read_flags to the BlockDriverState structure. It will control the flags set for copy-on-read operations. Make the block generic layer evaluate supported read flags before they go to a block driver. Suggested-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Signed-off-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> [vsementsov: use assert instead of abort] Reviewed-by: Max Reitz <mreitz@redhat.com> Message-Id: <20201216061703.70908-8-vsementsov@virtuozzo.com> Signed-off-by: Max Reitz <mreitz@redhat.com> --- include/block/block_int.h | 4 ++++ block/io.c | 10 ++++++++-- 2 files changed, 12 insertions(+), 2 deletions(-) diff --git a/include/block/block_int.h b/include/block/block_int.h index XXXXXXX..XXXXXXX 100644 --- a/include/block/block_int.h +++ b/include/block/block_int.h @@ -XXX,XX +XXX,XX @@ struct BlockDriverState { /* I/O Limits */ BlockLimits bl; + /* + * Flags honored during pread + */ + unsigned int supported_read_flags; /* Flags honored during pwrite (so far: BDRV_REQ_FUA, * BDRV_REQ_WRITE_UNCHANGED). * If a driver does not support BDRV_REQ_WRITE_UNCHANGED, those diff --git a/block/io.c b/block/io.c index XXXXXXX..XXXXXXX 100644 --- a/block/io.c +++ b/block/io.c @@ -XXX,XX +XXX,XX @@ static int coroutine_fn bdrv_aligned_preadv(BdrvChild *child, if (flags & BDRV_REQ_COPY_ON_READ) { int64_t pnum; + /* The flag BDRV_REQ_COPY_ON_READ has reached its addressee */ + flags &= ~BDRV_REQ_COPY_ON_READ; + ret = bdrv_is_allocated(bs, offset, bytes, &pnum); if (ret < 0) { goto out; @@ -XXX,XX +XXX,XX @@ static int coroutine_fn bdrv_aligned_preadv(BdrvChild *child, goto out; } + assert(!(flags & ~bs->supported_read_flags)); + max_bytes = ROUND_UP(MAX(0, total_bytes - offset), align); if (bytes <= max_bytes && bytes <= max_transfer) { - ret = bdrv_driver_preadv(bs, offset, bytes, qiov, qiov_offset, 0); + ret = bdrv_driver_preadv(bs, offset, bytes, qiov, qiov_offset, flags); goto out; } @@ -XXX,XX +XXX,XX @@ static int coroutine_fn bdrv_aligned_preadv(BdrvChild *child, ret = bdrv_driver_preadv(bs, offset + bytes - bytes_remaining, num, qiov, - qiov_offset + bytes - bytes_remaining, 0); + qiov_offset + bytes - bytes_remaining, + flags); max_bytes -= num; } else { num = bytes_remaining; -- 2.29.2 From: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com> If the flag BDRV_REQ_PREFETCH was set, skip idling read/write operations in COR-driver. It can be taken into account for the COR-algorithms optimization. That check is being made during the block stream job by the moment. Add the BDRV_REQ_PREFETCH flag to the supported_read_flags of the COR-filter. block: Modify the comment for the flag BDRV_REQ_PREFETCH as we are going to use it alone and pass it to the COR-filter driver for further processing. Signed-off-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Message-Id: <20201216061703.70908-9-vsementsov@virtuozzo.com> Signed-off-by: Max Reitz <mreitz@redhat.com> --- include/block/block.h | 8 +++++--- block/copy-on-read.c | 14 ++++++++++---- 2 files changed, 15 insertions(+), 7 deletions(-) diff --git a/include/block/block.h b/include/block/block.h index XXXXXXX..XXXXXXX 100644 --- a/include/block/block.h +++ b/include/block/block.h @@ -XXX,XX +XXX,XX @@ typedef enum { BDRV_REQ_NO_FALLBACK = 0x100, /* - * BDRV_REQ_PREFETCH may be used only together with BDRV_REQ_COPY_ON_READ - * on read request and means that caller doesn't really need data to be - * written to qiov parameter which may be NULL. + * BDRV_REQ_PREFETCH makes sense only in the context of copy-on-read + * (i.e., together with the BDRV_REQ_COPY_ON_READ flag or when a COR + * filter is involved), in which case it signals that the COR operation + * need not read the data into memory (qiov) but only ensure they are + * copied to the top layer (i.e., that COR operation is done). */ BDRV_REQ_PREFETCH = 0x200, diff --git a/block/copy-on-read.c b/block/copy-on-read.c index XXXXXXX..XXXXXXX 100644 --- a/block/copy-on-read.c +++ b/block/copy-on-read.c @@ -XXX,XX +XXX,XX @@ static int cor_open(BlockDriverState *bs, QDict *options, int flags, return -EINVAL; } + bs->supported_read_flags = BDRV_REQ_PREFETCH; + bs->supported_write_flags = BDRV_REQ_WRITE_UNCHANGED | (BDRV_REQ_FUA & bs->file->bs->supported_write_flags); @@ -XXX,XX +XXX,XX @@ static int coroutine_fn cor_co_preadv_part(BlockDriverState *bs, } } - ret = bdrv_co_preadv_part(bs->file, offset, n, qiov, qiov_offset, - local_flags); - if (ret < 0) { - return ret; + /* Skip if neither read nor write are needed */ + if ((local_flags & (BDRV_REQ_PREFETCH | BDRV_REQ_COPY_ON_READ)) != + BDRV_REQ_PREFETCH) { + ret = bdrv_co_preadv_part(bs->file, offset, n, qiov, qiov_offset, + local_flags); + if (ret < 0) { + return ret; + } } offset += n; -- 2.29.2 From: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com> Stream in stream_prepare calls bdrv_change_backing_file() to change backing-file in the metadata of bs. It may use either backing-file parameter given by user or just take filename of base on job start. Backing file format is determined by base on job finish. There are some problems with this design, we solve only two by this patch: 1. Consider scenario with backing-file unset. Current concept of stream supports changing of the base during the job (we don't freeze link to the base). So, we should not save base filename at job start, - let's determine name of the base on job finish. 2. Using direct base to determine filename and format is not very good: base node may be a filter, so its filename may be JSON, and format_name is not good for storing into qcow2 metadata as backing file format. - let's use unfiltered_base Signed-off-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> [vsementsov: change commit subject, change logic in stream_prepare] Message-Id: <20201216061703.70908-10-vsementsov@virtuozzo.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Signed-off-by: Max Reitz <mreitz@redhat.com> --- block/stream.c | 9 +++++---- blockdev.c | 8 +------- 2 files changed, 6 insertions(+), 11 deletions(-) diff --git a/block/stream.c b/block/stream.c index XXXXXXX..XXXXXXX 100644 --- a/block/stream.c +++ b/block/stream.c @@ -XXX,XX +XXX,XX @@ static int stream_prepare(Job *job) BlockDriverState *bs = blk_bs(bjob->blk); BlockDriverState *unfiltered_bs = bdrv_skip_filters(bs); BlockDriverState *base = bdrv_filter_or_cow_bs(s->above_base); + BlockDriverState *unfiltered_base = bdrv_skip_filters(base); Error *local_err = NULL; int ret = 0; @@ -XXX,XX +XXX,XX @@ static int stream_prepare(Job *job) if (bdrv_cow_child(unfiltered_bs)) { const char *base_id = NULL, *base_fmt = NULL; - if (base) { - base_id = s->backing_file_str; - if (base->drv) { - base_fmt = base->drv->format_name; + if (unfiltered_base) { + base_id = s->backing_file_str ?: unfiltered_base->filename; + if (unfiltered_base->drv) { + base_fmt = unfiltered_base->drv->format_name; } } bdrv_set_backing_hd(unfiltered_bs, base, &local_err); diff --git a/blockdev.c b/blockdev.c index XXXXXXX..XXXXXXX 100644 --- a/blockdev.c +++ b/blockdev.c @@ -XXX,XX +XXX,XX @@ void qmp_block_stream(bool has_job_id, const char *job_id, const char *device, BlockDriverState *base_bs = NULL; AioContext *aio_context; Error *local_err = NULL; - const char *base_name = NULL; int job_flags = JOB_DEFAULT; if (!has_on_error) { @@ -XXX,XX +XXX,XX @@ void qmp_block_stream(bool has_job_id, const char *job_id, const char *device, goto out; } assert(bdrv_get_aio_context(base_bs) == aio_context); - base_name = base; } if (has_base_node) { @@ -XXX,XX +XXX,XX @@ void qmp_block_stream(bool has_job_id, const char *job_id, const char *device, } assert(bdrv_get_aio_context(base_bs) == aio_context); bdrv_refresh_filename(base_bs); - base_name = base_bs->filename; } /* Check for op blockers in the whole chain between bs and base */ @@ -XXX,XX +XXX,XX @@ void qmp_block_stream(bool has_job_id, const char *job_id, const char *device, goto out; } - /* backing_file string overrides base bs filename */ - base_name = has_backing_file ? backing_file : base_name; - if (has_auto_finalize && !auto_finalize) { job_flags |= JOB_MANUAL_FINALIZE; } @@ -XXX,XX +XXX,XX @@ void qmp_block_stream(bool has_job_id, const char *job_id, const char *device, job_flags |= JOB_MANUAL_DISMISS; } - stream_start(has_job_id ? job_id : NULL, bs, base_bs, base_name, + stream_start(has_job_id ? job_id : NULL, bs, base_bs, backing_file, job_flags, has_speed ? speed : 0, on_error, filter_node_name, &local_err); if (local_err) { -- 2.29.2 From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> The code already don't freeze base node and we try to make it prepared for the situation when base node is changed during the operation. In other words, block-stream doesn't own base node. Let's introduce a new interface which should replace the current one, which will in better relations with the code. Specifying bottom node instead of base, and requiring it to be non-filter gives us the following benefits: - drop difference between above_base and base_overlay, which will be renamed to just bottom, when old interface dropped - clean way to work with parallel streams/commits on the same backing chain, which otherwise become a problem when we introduce a filter for stream job - cleaner interface. Nobody will surprised the fact that base node may disappear during block-stream, when there is no word about "base" in the interface. Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Message-Id: <20201216061703.70908-11-vsementsov@virtuozzo.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Signed-off-by: Max Reitz <mreitz@redhat.com> --- qapi/block-core.json | 12 ++++--- include/block/block_int.h | 1 + block/monitor/block-hmp-cmds.c | 3 +- block/stream.c | 50 +++++++++++++++++++--------- blockdev.c | 59 ++++++++++++++++++++++++++++------ 5 files changed, 94 insertions(+), 31 deletions(-) diff --git a/qapi/block-core.json b/qapi/block-core.json index XXXXXXX..XXXXXXX 100644 --- a/qapi/block-core.json +++ b/qapi/block-core.json @@ -XXX,XX +XXX,XX @@ # @device: the device or node name of the top image # # @base: the common backing file name. -# It cannot be set if @base-node is also set. +# It cannot be set if @base-node or @bottom is also set. # # @base-node: the node name of the backing file. -# It cannot be set if @base is also set. (Since 2.8) +# It cannot be set if @base or @bottom is also set. (Since 2.8) +# +# @bottom: the last node in the chain that should be streamed into +# top. It cannot be set if @base or @base-node is also set. +# It cannot be filter node. (Since 6.0) # # @backing-file: The backing file string to write into the top # image. This filename is not validated. @@ -XXX,XX +XXX,XX @@ ## { 'command': 'block-stream', 'data': { '*job-id': 'str', 'device': 'str', '*base': 'str', - '*base-node': 'str', '*backing-file': 'str', '*speed': 'int', - '*on-error': 'BlockdevOnError', + '*base-node': 'str', '*backing-file': 'str', '*bottom': 'str', + '*speed': 'int', '*on-error': 'BlockdevOnError', '*filter-node-name': 'str', '*auto-finalize': 'bool', '*auto-dismiss': 'bool' } } diff --git a/include/block/block_int.h b/include/block/block_int.h index XXXXXXX..XXXXXXX 100644 --- a/include/block/block_int.h +++ b/include/block/block_int.h @@ -XXX,XX +XXX,XX @@ int is_windows_drive(const char *filename); */ void stream_start(const char *job_id, BlockDriverState *bs, BlockDriverState *base, const char *backing_file_str, + BlockDriverState *bottom, int creation_flags, int64_t speed, BlockdevOnError on_error, const char *filter_node_name, diff --git a/block/monitor/block-hmp-cmds.c b/block/monitor/block-hmp-cmds.c index XXXXXXX..XXXXXXX 100644 --- a/block/monitor/block-hmp-cmds.c +++ b/block/monitor/block-hmp-cmds.c @@ -XXX,XX +XXX,XX @@ void hmp_block_stream(Monitor *mon, const QDict *qdict) int64_t speed = qdict_get_try_int(qdict, "speed", 0); qmp_block_stream(true, device, device, base != NULL, base, false, NULL, - false, NULL, qdict_haskey(qdict, "speed"), speed, true, + false, NULL, false, NULL, + qdict_haskey(qdict, "speed"), speed, true, BLOCKDEV_ON_ERROR_REPORT, false, NULL, false, false, false, false, &error); diff --git a/block/stream.c b/block/stream.c index XXXXXXX..XXXXXXX 100644 --- a/block/stream.c +++ b/block/stream.c @@ -XXX,XX +XXX,XX @@ static const BlockJobDriver stream_job_driver = { void stream_start(const char *job_id, BlockDriverState *bs, BlockDriverState *base, const char *backing_file_str, + BlockDriverState *bottom, int creation_flags, int64_t speed, BlockdevOnError on_error, const char *filter_node_name, @@ -XXX,XX +XXX,XX @@ void stream_start(const char *job_id, BlockDriverState *bs, BlockDriverState *iter; bool bs_read_only; int basic_flags = BLK_PERM_CONSISTENT_READ | BLK_PERM_WRITE_UNCHANGED; - BlockDriverState *base_overlay = bdrv_find_overlay(bs, base); + BlockDriverState *base_overlay; BlockDriverState *above_base; - if (!base_overlay) { - error_setg(errp, "'%s' is not in the backing chain of '%s'", - base->node_name, bs->node_name); - return; - } + assert(!(base && bottom)); + assert(!(backing_file_str && bottom)); + + if (bottom) { + /* + * New simple interface. The code is written in terms of old interface + * with @base parameter (still, it doesn't freeze link to base, so in + * this mean old code is correct for new interface). So, for now, just + * emulate base_overlay and above_base. Still, when old interface + * finally removed, we should refactor code to use only "bottom", but + * not "*base*" things. + */ + assert(!bottom->drv->is_filter); + base_overlay = above_base = bottom; + } else { + base_overlay = bdrv_find_overlay(bs, base); + if (!base_overlay) { + error_setg(errp, "'%s' is not in the backing chain of '%s'", + base->node_name, bs->node_name); + return; + } - /* - * Find the node directly above @base. @base_overlay is a COW overlay, so - * it must have a bdrv_cow_child(), but it is the immediate overlay of - * @base, so between the two there can only be filters. - */ - above_base = base_overlay; - if (bdrv_cow_bs(above_base) != base) { - above_base = bdrv_cow_bs(above_base); - while (bdrv_filter_bs(above_base) != base) { - above_base = bdrv_filter_bs(above_base); + /* + * Find the node directly above @base. @base_overlay is a COW overlay, + * so it must have a bdrv_cow_child(), but it is the immediate overlay + * of @base, so between the two there can only be filters. + */ + above_base = base_overlay; + if (bdrv_cow_bs(above_base) != base) { + above_base = bdrv_cow_bs(above_base); + while (bdrv_filter_bs(above_base) != base) { + above_base = bdrv_filter_bs(above_base); + } } } diff --git a/blockdev.c b/blockdev.c index XXXXXXX..XXXXXXX 100644 --- a/blockdev.c +++ b/blockdev.c @@ -XXX,XX +XXX,XX @@ void qmp_block_stream(bool has_job_id, const char *job_id, const char *device, bool has_base, const char *base, bool has_base_node, const char *base_node, bool has_backing_file, const char *backing_file, + bool has_bottom, const char *bottom, bool has_speed, int64_t speed, bool has_on_error, BlockdevOnError on_error, bool has_filter_node_name, const char *filter_node_name, @@ -XXX,XX +XXX,XX @@ void qmp_block_stream(bool has_job_id, const char *job_id, const char *device, bool has_auto_dismiss, bool auto_dismiss, Error **errp) { - BlockDriverState *bs, *iter; + BlockDriverState *bs, *iter, *iter_end; BlockDriverState *base_bs = NULL; + BlockDriverState *bottom_bs = NULL; AioContext *aio_context; Error *local_err = NULL; int job_flags = JOB_DEFAULT; + if (has_base && has_base_node) { + error_setg(errp, "'base' and 'base-node' cannot be specified " + "at the same time"); + return; + } + + if (has_base && has_bottom) { + error_setg(errp, "'base' and 'bottom' cannot be specified " + "at the same time"); + return; + } + + if (has_bottom && has_base_node) { + error_setg(errp, "'bottom' and 'base-node' cannot be specified " + "at the same time"); + return; + } + if (!has_on_error) { on_error = BLOCKDEV_ON_ERROR_REPORT; } @@ -XXX,XX +XXX,XX @@ void qmp_block_stream(bool has_job_id, const char *job_id, const char *device, aio_context = bdrv_get_aio_context(bs); aio_context_acquire(aio_context); - if (has_base && has_base_node) { - error_setg(errp, "'base' and 'base-node' cannot be specified " - "at the same time"); - goto out; - } - if (has_base) { base_bs = bdrv_find_backing_image(bs, base); if (base_bs == NULL) { @@ -XXX,XX +XXX,XX @@ void qmp_block_stream(bool has_job_id, const char *job_id, const char *device, bdrv_refresh_filename(base_bs); } - /* Check for op blockers in the whole chain between bs and base */ - for (iter = bs; iter && iter != base_bs; + if (has_bottom) { + bottom_bs = bdrv_lookup_bs(NULL, bottom, errp); + if (!bottom_bs) { + goto out; + } + if (!bottom_bs->drv) { + error_setg(errp, "Node '%s' is not open", bottom); + goto out; + } + if (bottom_bs->drv->is_filter) { + error_setg(errp, "Node '%s' is a filter, use a non-filter node " + "as 'bottom'", bottom); + goto out; + } + if (!bdrv_chain_contains(bs, bottom_bs)) { + error_setg(errp, "Node '%s' is not in a chain starting from '%s'", + bottom, device); + goto out; + } + assert(bdrv_get_aio_context(bottom_bs) == aio_context); + } + + /* + * Check for op blockers in the whole chain between bs and base (or bottom) + */ + iter_end = has_bottom ? bdrv_filter_or_cow_bs(bottom_bs) : base_bs; + for (iter = bs; iter && iter != iter_end; iter = bdrv_filter_or_cow_bs(iter)) { if (bdrv_op_is_blocked(iter, BLOCK_OP_TYPE_STREAM, errp)) { @@ -XXX,XX +XXX,XX @@ void qmp_block_stream(bool has_job_id, const char *job_id, const char *device, } stream_start(has_job_id ? job_id : NULL, bs, base_bs, backing_file, - job_flags, has_speed ? speed : 0, on_error, + bottom_bs, job_flags, has_speed ? speed : 0, on_error, filter_node_name, &local_err); if (local_err) { error_propagate(errp, local_err); -- 2.29.2 From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> test_stream_parallel run parallel stream jobs, intersecting so that top of one is base of another. It's OK now, but it would be a problem if insert the filter, as one job will want to use another job's filter as above_base node. Correct thing to do is move to new interface: "bottom" argument instead of base. This guarantees that jobs don't intersect by their actions. Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Message-Id: <20201216061703.70908-12-vsementsov@virtuozzo.com> Signed-off-by: Max Reitz <mreitz@redhat.com> --- tests/qemu-iotests/030 | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/tests/qemu-iotests/030 b/tests/qemu-iotests/030 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/030 +++ b/tests/qemu-iotests/030 @@ -XXX,XX +XXX,XX @@ class TestParallelOps(iotests.QMPTestCase): node_name = 'node%d' % i job_id = 'stream-%s' % node_name pending_jobs.append(job_id) - result = self.vm.qmp('block-stream', device=node_name, job_id=job_id, base=self.imgs[i-2], speed=1024) + result = self.vm.qmp('block-stream', device=node_name, + job_id=job_id, bottom=f'node{i-1}', + speed=1024) self.assert_qmp(result, 'return', {}) for job in pending_jobs: -- 2.29.2 From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Add a direct link to target bs for convenience and to simplify following commit which will insert COR filter above target bs. This is a part of original commit written by Andrey. Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Message-Id: <20201216061703.70908-13-vsementsov@virtuozzo.com> Signed-off-by: Max Reitz <mreitz@redhat.com> --- block/stream.c | 23 ++++++++++------------- 1 file changed, 10 insertions(+), 13 deletions(-) diff --git a/block/stream.c b/block/stream.c index XXXXXXX..XXXXXXX 100644 --- a/block/stream.c +++ b/block/stream.c @@ -XXX,XX +XXX,XX @@ typedef struct StreamBlockJob { BlockJob common; BlockDriverState *base_overlay; /* COW overlay (stream from this) */ BlockDriverState *above_base; /* Node directly above the base */ + BlockDriverState *target_bs; BlockdevOnError on_error; char *backing_file_str; bool bs_read_only; @@ -XXX,XX +XXX,XX @@ static void stream_abort(Job *job) StreamBlockJob *s = container_of(job, StreamBlockJob, common.job); if (s->chain_frozen) { - BlockJob *bjob = &s->common; - bdrv_unfreeze_backing_chain(blk_bs(bjob->blk), s->above_base); + bdrv_unfreeze_backing_chain(s->target_bs, s->above_base); } } static int stream_prepare(Job *job) { StreamBlockJob *s = container_of(job, StreamBlockJob, common.job); - BlockJob *bjob = &s->common; - BlockDriverState *bs = blk_bs(bjob->blk); - BlockDriverState *unfiltered_bs = bdrv_skip_filters(bs); + BlockDriverState *unfiltered_bs = bdrv_skip_filters(s->target_bs); BlockDriverState *base = bdrv_filter_or_cow_bs(s->above_base); BlockDriverState *unfiltered_base = bdrv_skip_filters(base); Error *local_err = NULL; int ret = 0; - bdrv_unfreeze_backing_chain(bs, s->above_base); + bdrv_unfreeze_backing_chain(s->target_bs, s->above_base); s->chain_frozen = false; if (bdrv_cow_child(unfiltered_bs)) { @@ -XXX,XX +XXX,XX @@ static void stream_clean(Job *job) { StreamBlockJob *s = container_of(job, StreamBlockJob, common.job); BlockJob *bjob = &s->common; - BlockDriverState *bs = blk_bs(bjob->blk); /* Reopen the image back in read-only mode if necessary */ if (s->bs_read_only) { /* Give up write permissions before making it read-only */ blk_set_perm(bjob->blk, 0, BLK_PERM_ALL, &error_abort); - bdrv_reopen_set_read_only(bs, true, NULL); + bdrv_reopen_set_read_only(s->target_bs, true, NULL); } g_free(s->backing_file_str); @@ -XXX,XX +XXX,XX @@ static int coroutine_fn stream_run(Job *job, Error **errp) { StreamBlockJob *s = container_of(job, StreamBlockJob, common.job); BlockBackend *blk = s->common.blk; - BlockDriverState *bs = blk_bs(blk); - BlockDriverState *unfiltered_bs = bdrv_skip_filters(bs); + BlockDriverState *unfiltered_bs = bdrv_skip_filters(s->target_bs); bool enable_cor = !bdrv_cow_child(s->base_overlay); int64_t len; int64_t offset = 0; @@ -XXX,XX +XXX,XX @@ static int coroutine_fn stream_run(Job *job, Error **errp) return 0; } - len = bdrv_getlength(bs); + len = bdrv_getlength(s->target_bs); if (len < 0) { return len; } @@ -XXX,XX +XXX,XX @@ static int coroutine_fn stream_run(Job *job, Error **errp) * account. */ if (enable_cor) { - bdrv_enable_copy_on_read(bs); + bdrv_enable_copy_on_read(s->target_bs); } for ( ; offset < len; offset += n) { @@ -XXX,XX +XXX,XX @@ static int coroutine_fn stream_run(Job *job, Error **errp) } if (enable_cor) { - bdrv_disable_copy_on_read(bs); + bdrv_disable_copy_on_read(s->target_bs); } /* Do not remove the backing file if an error was there but ignored. */ @@ -XXX,XX +XXX,XX @@ void stream_start(const char *job_id, BlockDriverState *bs, s->base_overlay = base_overlay; s->above_base = above_base; s->backing_file_str = g_strdup(backing_file_str); + s->target_bs = bs; s->bs_read_only = bs_read_only; s->chain_frozen = true; -- 2.29.2 From: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com> This patch completes the series with the COR-filter applied to block-stream operations. Adding the filter makes it possible in future implement discarding copied regions in backing files during the block-stream job, to reduce the disk overuse (we need control on permissions). Also, the filter now is smart enough to do copy-on-read with specified base, so we have benefit on guest reads even when doing block-stream of the part of the backing chain. Several iotests are slightly modified due to filter insertion. Signed-off-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Message-Id: <20201216061703.70908-14-vsementsov@virtuozzo.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Signed-off-by: Max Reitz <mreitz@redhat.com> --- block/stream.c | 105 ++++++++++++++++++++++--------------- tests/qemu-iotests/030 | 8 +-- tests/qemu-iotests/141.out | 2 +- tests/qemu-iotests/245 | 20 ++++--- 4 files changed, 80 insertions(+), 55 deletions(-) diff --git a/block/stream.c b/block/stream.c index XXXXXXX..XXXXXXX 100644 --- a/block/stream.c +++ b/block/stream.c @@ -XXX,XX +XXX,XX @@ #include "block/blockjob_int.h" #include "qapi/error.h" #include "qapi/qmp/qerror.h" +#include "qapi/qmp/qdict.h" #include "qemu/ratelimit.h" #include "sysemu/block-backend.h" +#include "block/copy-on-read.h" enum { /* @@ -XXX,XX +XXX,XX @@ typedef struct StreamBlockJob { BlockJob common; BlockDriverState *base_overlay; /* COW overlay (stream from this) */ BlockDriverState *above_base; /* Node directly above the base */ + BlockDriverState *cor_filter_bs; BlockDriverState *target_bs; BlockdevOnError on_error; char *backing_file_str; bool bs_read_only; - bool chain_frozen; } StreamBlockJob; static int coroutine_fn stream_populate(BlockBackend *blk, @@ -XXX,XX +XXX,XX @@ static int coroutine_fn stream_populate(BlockBackend *blk, { assert(bytes < SIZE_MAX); - return blk_co_preadv(blk, offset, bytes, NULL, - BDRV_REQ_COPY_ON_READ | BDRV_REQ_PREFETCH); -} - -static void stream_abort(Job *job) -{ - StreamBlockJob *s = container_of(job, StreamBlockJob, common.job); - - if (s->chain_frozen) { - bdrv_unfreeze_backing_chain(s->target_bs, s->above_base); - } + return blk_co_preadv(blk, offset, bytes, NULL, BDRV_REQ_PREFETCH); } static int stream_prepare(Job *job) @@ -XXX,XX +XXX,XX @@ static int stream_prepare(Job *job) Error *local_err = NULL; int ret = 0; - bdrv_unfreeze_backing_chain(s->target_bs, s->above_base); - s->chain_frozen = false; + /* We should drop filter at this point, as filter hold the backing chain */ + bdrv_cor_filter_drop(s->cor_filter_bs); + s->cor_filter_bs = NULL; if (bdrv_cow_child(unfiltered_bs)) { const char *base_id = NULL, *base_fmt = NULL; @@ -XXX,XX +XXX,XX @@ static void stream_clean(Job *job) StreamBlockJob *s = container_of(job, StreamBlockJob, common.job); BlockJob *bjob = &s->common; + if (s->cor_filter_bs) { + bdrv_cor_filter_drop(s->cor_filter_bs); + s->cor_filter_bs = NULL; + } + /* Reopen the image back in read-only mode if necessary */ if (s->bs_read_only) { /* Give up write permissions before making it read-only */ @@ -XXX,XX +XXX,XX @@ static int coroutine_fn stream_run(Job *job, Error **errp) StreamBlockJob *s = container_of(job, StreamBlockJob, common.job); BlockBackend *blk = s->common.blk; BlockDriverState *unfiltered_bs = bdrv_skip_filters(s->target_bs); - bool enable_cor = !bdrv_cow_child(s->base_overlay); int64_t len; int64_t offset = 0; uint64_t delay_ns = 0; @@ -XXX,XX +XXX,XX @@ static int coroutine_fn stream_run(Job *job, Error **errp) } job_progress_set_remaining(&s->common.job, len); - /* Turn on copy-on-read for the whole block device so that guest read - * requests help us make progress. Only do this when copying the entire - * backing chain since the copy-on-read operation does not take base into - * account. - */ - if (enable_cor) { - bdrv_enable_copy_on_read(s->target_bs); - } - for ( ; offset < len; offset += n) { bool copy; int ret; @@ -XXX,XX +XXX,XX @@ static int coroutine_fn stream_run(Job *job, Error **errp) } } - if (enable_cor) { - bdrv_disable_copy_on_read(s->target_bs); - } - /* Do not remove the backing file if an error was there but ignored. */ return error; } @@ -XXX,XX +XXX,XX @@ static const BlockJobDriver stream_job_driver = { .free = block_job_free, .run = stream_run, .prepare = stream_prepare, - .abort = stream_abort, .clean = stream_clean, .user_resume = block_job_user_resume, }, @@ -XXX,XX +XXX,XX @@ void stream_start(const char *job_id, BlockDriverState *bs, bool bs_read_only; int basic_flags = BLK_PERM_CONSISTENT_READ | BLK_PERM_WRITE_UNCHANGED; BlockDriverState *base_overlay; + BlockDriverState *cor_filter_bs = NULL; BlockDriverState *above_base; + QDict *opts; assert(!(base && bottom)); assert(!(backing_file_str && bottom)); @@ -XXX,XX +XXX,XX @@ void stream_start(const char *job_id, BlockDriverState *bs, } } - if (bdrv_freeze_backing_chain(bs, above_base, errp) < 0) { - return; - } - /* Make sure that the image is opened in read-write mode */ bs_read_only = bdrv_is_read_only(bs); if (bs_read_only) { - if (bdrv_reopen_set_read_only(bs, false, errp) != 0) { - bs_read_only = false; - goto fail; + int ret; + /* Hold the chain during reopen */ + if (bdrv_freeze_backing_chain(bs, above_base, errp) < 0) { + return; + } + + ret = bdrv_reopen_set_read_only(bs, false, errp); + + /* failure, or cor-filter will hold the chain */ + bdrv_unfreeze_backing_chain(bs, above_base); + + if (ret < 0) { + return; } } - /* Prevent concurrent jobs trying to modify the graph structure here, we - * already have our own plans. Also don't allow resize as the image size is - * queried only at the job start and then cached. */ - s = block_job_create(job_id, &stream_job_driver, NULL, bs, - basic_flags | BLK_PERM_GRAPH_MOD, + opts = qdict_new(); + + qdict_put_str(opts, "driver", "copy-on-read"); + qdict_put_str(opts, "file", bdrv_get_node_name(bs)); + /* Pass the base_overlay node name as 'bottom' to COR driver */ + qdict_put_str(opts, "bottom", base_overlay->node_name); + if (filter_node_name) { + qdict_put_str(opts, "node-name", filter_node_name); + } + + cor_filter_bs = bdrv_insert_node(bs, opts, BDRV_O_RDWR, errp); + if (!cor_filter_bs) { + goto fail; + } + + if (!filter_node_name) { + cor_filter_bs->implicit = true; + } + + s = block_job_create(job_id, &stream_job_driver, NULL, cor_filter_bs, + BLK_PERM_CONSISTENT_READ, basic_flags | BLK_PERM_WRITE, speed, creation_flags, NULL, NULL, errp); if (!s) { goto fail; } + /* + * Prevent concurrent jobs trying to modify the graph structure here, we + * already have our own plans. Also don't allow resize as the image size is + * queried only at the job start and then cached. + */ + if (block_job_add_bdrv(&s->common, "active node", bs, 0, + basic_flags | BLK_PERM_WRITE, &error_abort)) { + goto fail; + } + /* Block all intermediate nodes between bs and base, because they will * disappear from the chain after this operation. The streaming job reads * every block only once, assuming that it doesn't change, so forbid writes @@ -XXX,XX +XXX,XX @@ void stream_start(const char *job_id, BlockDriverState *bs, s->base_overlay = base_overlay; s->above_base = above_base; s->backing_file_str = g_strdup(backing_file_str); + s->cor_filter_bs = cor_filter_bs; s->target_bs = bs; s->bs_read_only = bs_read_only; - s->chain_frozen = true; s->on_error = on_error; trace_stream_start(bs, base, s); @@ -XXX,XX +XXX,XX @@ void stream_start(const char *job_id, BlockDriverState *bs, return; fail: + if (cor_filter_bs) { + bdrv_cor_filter_drop(cor_filter_bs); + } if (bs_read_only) { bdrv_reopen_set_read_only(bs, true, NULL); } - bdrv_unfreeze_backing_chain(bs, above_base); } diff --git a/tests/qemu-iotests/030 b/tests/qemu-iotests/030 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/030 +++ b/tests/qemu-iotests/030 @@ -XXX,XX +XXX,XX @@ class TestParallelOps(iotests.QMPTestCase): self.assert_no_active_block_jobs() # Set a speed limit to make sure that this job blocks the rest - result = self.vm.qmp('block-stream', device='node4', job_id='stream-node4', base=self.imgs[1], speed=1024*1024) + result = self.vm.qmp('block-stream', device='node4', + job_id='stream-node4', base=self.imgs[1], + filter_node_name='stream-filter', speed=1024*1024) self.assert_qmp(result, 'return', {}) result = self.vm.qmp('block-stream', device='node5', job_id='stream-node5', base=self.imgs[2]) self.assert_qmp(result, 'error/desc', - "Node 'node4' is busy: block device is in use by block job: stream") + "Node 'stream-filter' is busy: block device is in use by block job: stream") result = self.vm.qmp('block-stream', device='node3', job_id='stream-node3', base=self.imgs[2]) self.assert_qmp(result, 'error/desc', @@ -XXX,XX +XXX,XX @@ class TestParallelOps(iotests.QMPTestCase): # block-commit should also fail if it touches nodes used by the stream job result = self.vm.qmp('block-commit', device='drive0', base=self.imgs[4], job_id='commit-node4') self.assert_qmp(result, 'error/desc', - "Node 'node4' is busy: block device is in use by block job: stream") + "Node 'stream-filter' is busy: block device is in use by block job: stream") result = self.vm.qmp('block-commit', device='drive0', base=self.imgs[1], top=self.imgs[3], job_id='commit-node1') self.assert_qmp(result, 'error/desc', diff --git a/tests/qemu-iotests/141.out b/tests/qemu-iotests/141.out index XXXXXXX..XXXXXXX 100644 --- a/tests/qemu-iotests/141.out +++ b/tests/qemu-iotests/141.out @@ -XXX,XX +XXX,XX @@ wrote 1048576/1048576 bytes at offset 0 {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "running", "id": "job0"}} {'execute': 'blockdev-del', 'arguments': {'node-name': 'drv0'}} -{"error": {"class": "GenericError", "desc": "Node drv0 is in use"}} +{"error": {"class": "GenericError", "desc": "Node 'drv0' is busy: block device is in use by block job: stream"}} {'execute': 'block-job-cancel', 'arguments': {'device': 'job0'}} {"return": {}} diff --git a/tests/qemu-iotests/245 b/tests/qemu-iotests/245 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/245 +++ b/tests/qemu-iotests/245 @@ -XXX,XX +XXX,XX @@ class TestBlockdevReopen(iotests.QMPTestCase): # hd1 <- hd0 result = self.vm.qmp('block-stream', conv_keys = True, job_id = 'stream0', - device = 'hd1', auto_finalize = False) + device = 'hd1', filter_node_name='cor', + auto_finalize = False) self.assert_qmp(result, 'return', {}) - # We can't reopen with the original options because that would - # make hd1 read-only and block-stream requires it to be read-write - # (Which error message appears depends on whether the stream job is - # already done with copying at this point.) + # We can't reopen with the original options because there is a filter + # inserted by stream job above hd1. self.reopen(opts, {}, - ["Can't set node 'hd1' to r/o with copy-on-read enabled", - "Cannot make block node read-only, there is a writer on it"]) + "Cannot change the option 'backing.backing.file.node-name'") + + # We can't reopen hd1 to read-only, as block-stream requires it to be + # read-write + self.reopen(opts['backing'], {'read-only': True}, + "Cannot make block node read-only, there is a writer on it") # We can't remove hd2 while the stream job is ongoing opts['backing']['backing'] = None - self.reopen(opts, {'backing.read-only': False}, "Cannot change 'backing' link from 'hd1' to 'hd2'") + self.reopen(opts['backing'], {'read-only': False}, + "Cannot change 'backing' link from 'hd1' to 'hd2'") # We can detach hd1 from hd0 because it doesn't affect the stream job opts['backing'] = None -- 2.29.2 There are a couple of environment variables that we fetch with os.environ.get() without supplying a default. Clearly they are required and expected to be set by the ./check script (as evidenced by execute_setup_common(), which checks for test_dir and qemu_default_machine to be set, and aborts if they are not). Using .get() this way has the disadvantage of returning an Optional[str] type, which mypy will complain about when tests just assume these values to be str. Use [] instead, which raises a KeyError for environment variables that are not set. When this exception is raised, catch it and move the abort code from execute_setup_common() there. Drop the 'assert iotests.sock_dir is not None' from iotest 300, because that sort of thing is precisely what this patch wants to prevent. Signed-off-by: Max Reitz <mreitz@redhat.com> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Reviewed-by: Willian Rampazzo <willianr@redhat.com> Message-Id: <20210118105720.14824-2-mreitz@redhat.com> --- tests/qemu-iotests/300 | 1 - tests/qemu-iotests/iotests.py | 26 +++++++++++++------------- 2 files changed, 13 insertions(+), 14 deletions(-) diff --git a/tests/qemu-iotests/300 b/tests/qemu-iotests/300 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/300 +++ b/tests/qemu-iotests/300 @@ -XXX,XX +XXX,XX @@ import qemu BlockBitmapMapping = List[Dict[str, Union[str, List[Dict[str, str]]]]] -assert iotests.sock_dir is not None mig_sock = os.path.join(iotests.sock_dir, 'mig_sock') diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py index XXXXXXX..XXXXXXX 100644 --- a/tests/qemu-iotests/iotests.py +++ b/tests/qemu-iotests/iotests.py @@ -XXX,XX +XXX,XX @@ qemu_opts = os.environ.get('QEMU_OPTIONS', '').strip().split(' ') imgfmt = os.environ.get('IMGFMT', 'raw') imgproto = os.environ.get('IMGPROTO', 'file') -test_dir = os.environ.get('TEST_DIR') -sock_dir = os.environ.get('SOCK_DIR') output_dir = os.environ.get('OUTPUT_DIR', '.') -cachemode = os.environ.get('CACHEMODE') -aiomode = os.environ.get('AIOMODE') -qemu_default_machine = os.environ.get('QEMU_DEFAULT_MACHINE') + +try: + test_dir = os.environ['TEST_DIR'] + sock_dir = os.environ['SOCK_DIR'] + cachemode = os.environ['CACHEMODE'] + aiomode = os.environ['AIOMODE'] + qemu_default_machine = os.environ['QEMU_DEFAULT_MACHINE'] +except KeyError: + # We are using these variables as proxies to indicate that we're + # not being run via "check". There may be other things set up by + # "check" that individual test cases rely on. + sys.stderr.write('Please run this test via the "check" script\n') + sys.exit(os.EX_USAGE) socket_scm_helper = os.environ.get('SOCKET_SCM_HELPER', 'socket_scm_helper') @@ -XXX,XX +XXX,XX @@ def execute_setup_common(supported_fmts: Sequence[str] = (), """ # Note: Python 3.6 and pylint do not like 'Collection' so use 'Sequence'. - # We are using TEST_DIR and QEMU_DEFAULT_MACHINE as proxies to - # indicate that we're not being run via "check". There may be - # other things set up by "check" that individual test cases rely - # on. - if test_dir is None or qemu_default_machine is None: - sys.stderr.write('Please run this test via the "check" script\n') - sys.exit(os.EX_USAGE) - debug = '-d' in sys.argv if debug: sys.argv.remove('-d') -- 2.29.2 Instead of checking iotests.py only, check all Python files in the qemu-iotests/ directory. Of course, most of them do not pass, so there is an extensive skip list for now. (The only files that do pass are 209, 254, 283, and iotests.py.) (Alternatively, we could have the opposite, i.e. an explicit list of files that we do want to check, but I think it is better to check files by default.) Unless started in debug mode (./check -d), the output has no information on which files are tested, so we will not have a problem e.g. with backports, where some files may be missing when compared to upstream. Besides the technical rewrite, some more things are changed: - For the pylint invocation, PYTHONPATH is adjusted. This mirrors setting MYPYPATH for mypy. - Also, MYPYPATH is now derived from PYTHONPATH, so that we include paths set by the environment. Maybe at some point we want to let the check script add '../../python/' to PYTHONPATH so that iotests.py does not need to do that. - Passing --notes=FIXME,XXX to pylint suppresses warnings for TODO comments. TODO is fine, we do not need 297 to complain about such comments. - The "Success" line from mypy's output is suppressed, because (A) it does not add useful information, and (B) it would leak information about the files having been tested to the reference output, which we decidedly do not want. Suggested-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Signed-off-by: Max Reitz <mreitz@redhat.com> Message-Id: <20210118105720.14824-3-mreitz@redhat.com> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> --- tests/qemu-iotests/297 | 112 +++++++++++++++++++++++++++++-------- tests/qemu-iotests/297.out | 5 +- 2 files changed, 92 insertions(+), 25 deletions(-) diff --git a/tests/qemu-iotests/297 b/tests/qemu-iotests/297 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/297 +++ b/tests/qemu-iotests/297 @@ -XXX,XX +XXX,XX @@ -#!/usr/bin/env bash +#!/usr/bin/env python3 # group: meta # # Copyright (C) 2020 Red Hat, Inc. @@ -XXX,XX +XXX,XX @@ # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. -seq=$(basename $0) -echo "QA output created by $seq" +import os +import re +import shutil +import subprocess +import sys -status=1 # failure is the default! +import iotests -# get standard environment -. ./common.rc -if ! type -p "pylint-3" > /dev/null; then - _notrun "pylint-3 not found" -fi -if ! type -p "mypy" > /dev/null; then - _notrun "mypy not found" -fi +# TODO: Empty this list! +SKIP_FILES = ( + '030', '040', '041', '044', '045', '055', '056', '057', '065', '093', + '096', '118', '124', '129', '132', '136', '139', '147', '148', '149', + '151', '152', '155', '163', '165', '169', '194', '196', '199', '202', + '203', '205', '206', '207', '208', '210', '211', '212', '213', '216', + '218', '219', '222', '224', '228', '234', '235', '236', '237', '238', + '240', '242', '245', '246', '248', '255', '256', '257', '258', '260', + '262', '264', '266', '274', '277', '280', '281', '295', '296', '298', + '299', '300', '302', '303', '304', '307', + 'nbd-fault-injector.py', 'qcow2.py', 'qcow2_format.py', 'qed.py' +) -pylint-3 --score=n iotests.py -MYPYPATH=../../python/ mypy --warn-unused-configs --disallow-subclassing-any \ - --disallow-any-generics --disallow-incomplete-defs \ - --disallow-untyped-decorators --no-implicit-optional \ - --warn-redundant-casts --warn-unused-ignores \ - --no-implicit-reexport iotests.py +def is_python_file(filename): + if not os.path.isfile(filename): + return False -# success, all done -echo "*** done" -rm -f $seq.full -status=0 + if filename.endswith('.py'): + return True + + with open(filename) as f: + try: + first_line = f.readline() + return re.match('^#!.*python', first_line) is not None + except UnicodeDecodeError: # Ignore binary files + return False + + +def run_linters(): + files = [filename for filename in (set(os.listdir('.')) - set(SKIP_FILES)) + if is_python_file(filename)] + + iotests.logger.debug('Files to be checked:') + iotests.logger.debug(', '.join(sorted(files))) + + print('=== pylint ===') + sys.stdout.flush() + + # Todo notes are fine, but fixme's or xxx's should probably just be + # fixed (in tests, at least) + env = os.environ.copy() + qemu_module_path = os.path.join(os.path.dirname(__file__), + '..', '..', 'python') + try: + env['PYTHONPATH'] += os.pathsep + qemu_module_path + except KeyError: + env['PYTHONPATH'] = qemu_module_path + subprocess.run(('pylint-3', '--score=n', '--notes=FIXME,XXX', *files), + env=env, check=False) + + print('=== mypy ===') + sys.stdout.flush() + + # We have to call mypy separately for each file. Otherwise, it + # will interpret all given files as belonging together (i.e., they + # may not both define the same classes, etc.; most notably, they + # must not both define the __main__ module). + env['MYPYPATH'] = env['PYTHONPATH'] + for filename in files: + p = subprocess.run(('mypy', + '--warn-unused-configs', + '--disallow-subclassing-any', + '--disallow-any-generics', + '--disallow-incomplete-defs', + '--disallow-untyped-decorators', + '--no-implicit-optional', + '--warn-redundant-casts', + '--warn-unused-ignores', + '--no-implicit-reexport', + filename), + env=env, + check=False, + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, + universal_newlines=True) + + if p.returncode != 0: + print(p.stdout) + + +for linter in ('pylint-3', 'mypy'): + if shutil.which(linter) is None: + iotests.notrun(f'{linter} not found') + +iotests.script_main(run_linters) diff --git a/tests/qemu-iotests/297.out b/tests/qemu-iotests/297.out index XXXXXXX..XXXXXXX 100644 --- a/tests/qemu-iotests/297.out +++ b/tests/qemu-iotests/297.out @@ -XXX,XX +XXX,XX @@ -QA output created by 297 -Success: no issues found in 1 source file -*** done +=== pylint === +=== mypy === -- 2.29.2 Signed-off-by: Max Reitz <mreitz@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Reviewed-by: Willian Rampazzo <willianr@redhat.com> Message-Id: <20210118105720.14824-4-mreitz@redhat.com> --- tests/qemu-iotests/124 | 8 +------- tests/qemu-iotests/iotests.py | 11 +++++++---- 2 files changed, 8 insertions(+), 11 deletions(-) diff --git a/tests/qemu-iotests/124 b/tests/qemu-iotests/124 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/124 +++ b/tests/qemu-iotests/124 @@ -XXX,XX +XXX,XX @@ import os import iotests +from iotests import try_remove def io_write_patterns(img, patterns): @@ -XXX,XX +XXX,XX @@ def io_write_patterns(img, patterns): iotests.qemu_io('-c', 'write -P%s %s %s' % pattern, img) -def try_remove(img): - try: - os.remove(img) - except OSError: - pass - - def transaction_action(action, **kwargs): return { 'type': action, diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py index XXXXXXX..XXXXXXX 100644 --- a/tests/qemu-iotests/iotests.py +++ b/tests/qemu-iotests/iotests.py @@ -XXX,XX +XXX,XX @@ class FilePath: return False +def try_remove(img): + try: + os.remove(img) + except OSError: + pass + def file_path_remover(): for path in reversed(file_path_remover.paths): - try: - os.remove(path) - except OSError: - pass + try_remove(path) def file_path(*names, base_dir=test_dir): -- 2.29.2 Signed-off-by: Max Reitz <mreitz@redhat.com> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Reviewed-by: Eric Blake <eblake@redhat.com> Reviewed-by: Willian Rampazzo <willianr@redhat.com> Message-Id: <20210118105720.14824-5-mreitz@redhat.com> --- tests/qemu-iotests/129 | 2 ++ 1 file changed, 2 insertions(+) diff --git a/tests/qemu-iotests/129 b/tests/qemu-iotests/129 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/129 +++ b/tests/qemu-iotests/129 @@ -XXX,XX +XXX,XX @@ class TestStopWithBlockJob(iotests.QMPTestCase): result = self.vm.qmp("block_set_io_throttle", conv_keys=False, **params) self.vm.shutdown() + for img in (self.test_img, self.target_img, self.base_img): + iotests.try_remove(img) def do_test_stop(self, cmd, **args): """Test 'stop' while block job is running on a throttled drive. -- 2.29.2 @busy is false when the job is paused, which happens all the time because that is how jobs yield (e.g. for mirror at least since commit 565ac01f8d3). Back when 129 was added (2015), perhaps there was no better way of checking whether the job was still actually running. Now we have the @status field (as of 58b295ba52c, i.e. 2018), which can give us exactly that information. Signed-off-by: Max Reitz <mreitz@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Reviewed-by: Willian Rampazzo <willianr@redhat.com> Message-Id: <20210118105720.14824-6-mreitz@redhat.com> --- tests/qemu-iotests/129 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/qemu-iotests/129 b/tests/qemu-iotests/129 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/129 +++ b/tests/qemu-iotests/129 @@ -XXX,XX +XXX,XX @@ class TestStopWithBlockJob(iotests.QMPTestCase): result = self.vm.qmp("stop") self.assert_qmp(result, 'return', {}) result = self.vm.qmp("query-block-jobs") - self.assert_qmp(result, 'return[0]/busy', True) + self.assert_qmp(result, 'return[0]/status', 'running') self.assert_qmp(result, 'return[0]/ready', False) def test_drive_mirror(self): -- 2.29.2 Throttling on the BB has not affected block jobs in a while, so it is possible that one of the jobs in 129 finishes before the VM is stopped. We can fix that by running the job from a throttle node. Signed-off-by: Max Reitz <mreitz@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Reviewed-by: Willian Rampazzo <willianr@redhat.com> Message-Id: <20210118105720.14824-7-mreitz@redhat.com> --- tests/qemu-iotests/129 | 37 +++++++++++++------------------------ 1 file changed, 13 insertions(+), 24 deletions(-) diff --git a/tests/qemu-iotests/129 b/tests/qemu-iotests/129 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/129 +++ b/tests/qemu-iotests/129 @@ -XXX,XX +XXX,XX @@ class TestStopWithBlockJob(iotests.QMPTestCase): iotests.qemu_img('create', '-f', iotests.imgfmt, self.test_img, "-b", self.base_img, '-F', iotests.imgfmt) iotests.qemu_io('-f', iotests.imgfmt, '-c', 'write -P0x5d 1M 128M', self.test_img) - self.vm = iotests.VM().add_drive(self.test_img) + self.vm = iotests.VM() + self.vm.add_object('throttle-group,id=tg0,x-bps-total=1024') + + source_drive = 'driver=throttle,' \ + 'throttle-group=tg0,' \ + f'file.driver={iotests.imgfmt},' \ + f'file.file.filename={self.test_img}' + + self.vm.add_drive(None, source_drive) self.vm.launch() def tearDown(self): - params = {"device": "drive0", - "bps": 0, - "bps_rd": 0, - "bps_wr": 0, - "iops": 0, - "iops_rd": 0, - "iops_wr": 0, - } - result = self.vm.qmp("block_set_io_throttle", conv_keys=False, - **params) self.vm.shutdown() for img in (self.test_img, self.target_img, self.base_img): iotests.try_remove(img) @@ -XXX,XX +XXX,XX @@ class TestStopWithBlockJob(iotests.QMPTestCase): def do_test_stop(self, cmd, **args): """Test 'stop' while block job is running on a throttled drive. The 'stop' command shouldn't drain the job""" - params = {"device": "drive0", - "bps": 1024, - "bps_rd": 0, - "bps_wr": 0, - "iops": 0, - "iops_rd": 0, - "iops_wr": 0, - } - result = self.vm.qmp("block_set_io_throttle", conv_keys=False, - **params) - self.assert_qmp(result, 'return', {}) result = self.vm.qmp(cmd, **args) self.assert_qmp(result, 'return', {}) + result = self.vm.qmp("stop") self.assert_qmp(result, 'return', {}) result = self.vm.qmp("query-block-jobs") + self.assert_qmp(result, 'return[0]/status', 'running') self.assert_qmp(result, 'return[0]/ready', False) def test_drive_mirror(self): self.do_test_stop("drive-mirror", device="drive0", - target=self.target_img, + target=self.target_img, format=iotests.imgfmt, sync="full") def test_drive_backup(self): self.do_test_stop("drive-backup", device="drive0", - target=self.target_img, + target=self.target_img, format=iotests.imgfmt, sync="full") def test_block_commit(self): -- 2.29.2 Before this patch, test_block_commit() performs an active commit, which under the hood is a mirror job. If we want to test various different block jobs, we should perhaps run an actual commit job instead. Doing so requires adding an overlay above the source node before the commit is done (and then specifying the source node as the top node for the commit job). Signed-off-by: Max Reitz <mreitz@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Reviewed-by: Willian Rampazzo <willianr@redhat.com> Message-Id: <20210118105720.14824-8-mreitz@redhat.com> --- tests/qemu-iotests/129 | 27 +++++++++++++++++++++++++-- 1 file changed, 25 insertions(+), 2 deletions(-) diff --git a/tests/qemu-iotests/129 b/tests/qemu-iotests/129 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/129 +++ b/tests/qemu-iotests/129 @@ -XXX,XX +XXX,XX @@ class TestStopWithBlockJob(iotests.QMPTestCase): test_img = os.path.join(iotests.test_dir, 'test.img') target_img = os.path.join(iotests.test_dir, 'target.img') base_img = os.path.join(iotests.test_dir, 'base.img') + overlay_img = os.path.join(iotests.test_dir, 'overlay.img') def setUp(self): iotests.qemu_img('create', '-f', iotests.imgfmt, self.base_img, "1G") @@ -XXX,XX +XXX,XX @@ class TestStopWithBlockJob(iotests.QMPTestCase): self.vm.add_object('throttle-group,id=tg0,x-bps-total=1024') source_drive = 'driver=throttle,' \ + 'node-name=source,' \ 'throttle-group=tg0,' \ f'file.driver={iotests.imgfmt},' \ f'file.file.filename={self.test_img}' @@ -XXX,XX +XXX,XX @@ class TestStopWithBlockJob(iotests.QMPTestCase): def tearDown(self): self.vm.shutdown() - for img in (self.test_img, self.target_img, self.base_img): + for img in (self.test_img, self.target_img, self.base_img, + self.overlay_img): iotests.try_remove(img) def do_test_stop(self, cmd, **args): @@ -XXX,XX +XXX,XX @@ class TestStopWithBlockJob(iotests.QMPTestCase): sync="full") def test_block_commit(self): - self.do_test_stop("block-commit", device="drive0") + # Add overlay above the source node so that we actually use a + # commit job instead of a mirror job + + iotests.qemu_img('create', '-f', iotests.imgfmt, self.overlay_img, + '1G') + + result = self.vm.qmp('blockdev-add', **{ + 'node-name': 'overlay', + 'driver': iotests.imgfmt, + 'file': { + 'driver': 'file', + 'filename': self.overlay_img + } + }) + self.assert_qmp(result, 'return', {}) + + result = self.vm.qmp('blockdev-snapshot', + node='source', overlay='overlay') + self.assert_qmp(result, 'return', {}) + + self.do_test_stop('block-commit', device='drive0', top_node='source') if __name__ == '__main__': iotests.main(supported_fmts=["qcow2"], -- 2.29.2 Issuing 'stop' on the VM drains all nodes. If the mirror job has many large requests in flight, this may lead to significant I/O that looks a bit like 'stop' would make the job try to complete (which is what 129 should verify not to happen). We can limit the I/O in flight by limiting the buffer size, so mirror will make very little progress during the 'stop' drain. (We do not need to do anything about commit, which has a buffer size of 512 kB by default; or backup, which goes cluster by cluster. Once we have asynchronous requests for backup, that will change, but then we can fine-tune the backup job to only perform a single request on a very small chunk, too.) Signed-off-by: Max Reitz <mreitz@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Reviewed-by: Willian Rampazzo <willianr@redhat.com> Message-Id: <20210118105720.14824-9-mreitz@redhat.com> --- tests/qemu-iotests/129 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/qemu-iotests/129 b/tests/qemu-iotests/129 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/129 +++ b/tests/qemu-iotests/129 @@ -XXX,XX +XXX,XX @@ class TestStopWithBlockJob(iotests.QMPTestCase): def test_drive_mirror(self): self.do_test_stop("drive-mirror", device="drive0", target=self.target_img, format=iotests.imgfmt, - sync="full") + sync="full", buf_size=65536) def test_drive_backup(self): self.do_test_stop("drive-backup", device="drive0", -- 2.29.2 And consequentially drop it from 297's skip list. Signed-off-by: Max Reitz <mreitz@redhat.com> Reviewed-by: Willian Rampazzo <willianr@redhat.com> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Message-Id: <20210118105720.14824-10-mreitz@redhat.com> --- tests/qemu-iotests/129 | 4 ++-- tests/qemu-iotests/297 | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/tests/qemu-iotests/129 b/tests/qemu-iotests/129 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/129 +++ b/tests/qemu-iotests/129 @@ -XXX,XX +XXX,XX @@ import os import iotests -import time class TestStopWithBlockJob(iotests.QMPTestCase): test_img = os.path.join(iotests.test_dir, 'test.img') @@ -XXX,XX +XXX,XX @@ class TestStopWithBlockJob(iotests.QMPTestCase): iotests.qemu_img('create', '-f', iotests.imgfmt, self.base_img, "1G") iotests.qemu_img('create', '-f', iotests.imgfmt, self.test_img, "-b", self.base_img, '-F', iotests.imgfmt) - iotests.qemu_io('-f', iotests.imgfmt, '-c', 'write -P0x5d 1M 128M', self.test_img) + iotests.qemu_io('-f', iotests.imgfmt, '-c', 'write -P0x5d 1M 128M', + self.test_img) self.vm = iotests.VM() self.vm.add_object('throttle-group,id=tg0,x-bps-total=1024') diff --git a/tests/qemu-iotests/297 b/tests/qemu-iotests/297 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/297 +++ b/tests/qemu-iotests/297 @@ -XXX,XX +XXX,XX @@ import iotests # TODO: Empty this list! SKIP_FILES = ( '030', '040', '041', '044', '045', '055', '056', '057', '065', '093', - '096', '118', '124', '129', '132', '136', '139', '147', '148', '149', + '096', '118', '124', '132', '136', '139', '147', '148', '149', '151', '152', '155', '163', '165', '169', '194', '196', '199', '202', '203', '205', '206', '207', '208', '210', '211', '212', '213', '216', '218', '219', '222', '224', '228', '234', '235', '236', '237', '238', -- 2.29.2 And consequentially drop it from 297's skip list. Signed-off-by: Max Reitz <mreitz@redhat.com> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Reviewed-by: Willian Rampazzo <willianr@redhat.com> Message-Id: <20210118105720.14824-11-mreitz@redhat.com> --- tests/qemu-iotests/297 | 2 +- tests/qemu-iotests/300 | 18 +++++++++++++++--- 2 files changed, 16 insertions(+), 4 deletions(-) diff --git a/tests/qemu-iotests/297 b/tests/qemu-iotests/297 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/297 +++ b/tests/qemu-iotests/297 @@ -XXX,XX +XXX,XX @@ SKIP_FILES = ( '218', '219', '222', '224', '228', '234', '235', '236', '237', '238', '240', '242', '245', '246', '248', '255', '256', '257', '258', '260', '262', '264', '266', '274', '277', '280', '281', '295', '296', '298', - '299', '300', '302', '303', '304', '307', + '299', '302', '303', '304', '307', 'nbd-fault-injector.py', 'qcow2.py', 'qcow2_format.py', 'qed.py' ) diff --git a/tests/qemu-iotests/300 b/tests/qemu-iotests/300 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/300 +++ b/tests/qemu-iotests/300 @@ -XXX,XX +XXX,XX @@ import os import random import re from typing import Dict, List, Optional, Union + import iotests + +# Import qemu after iotests.py has amended sys.path +# pylint: disable=wrong-import-order import qemu BlockBitmapMapping = List[Dict[str, Union[str, List[Dict[str, str]]]]] @@ -XXX,XX +XXX,XX @@ class TestDirtyBitmapMigration(iotests.QMPTestCase): If @msg is None, check that there has not been any error. """ self.vm_b.shutdown() + + log = self.vm_b.get_log() + assert log is not None # Loaded after shutdown + if msg is None: - self.assertNotIn('qemu-system-', self.vm_b.get_log()) + self.assertNotIn('qemu-system-', log) else: - self.assertIn(msg, self.vm_b.get_log()) + self.assertIn(msg, log) @staticmethod def mapping(node_name: str, node_alias: str, @@ -XXX,XX +XXX,XX @@ class TestBlockBitmapMappingErrors(TestDirtyBitmapMigration): # Check for the error in the source's log self.vm_a.shutdown() + + log = self.vm_a.get_log() + assert log is not None # Loaded after shutdown + self.assertIn(f"Cannot migrate bitmap '{name}' on node " f"'{self.src_node_name}': Name is longer than 255 bytes", - self.vm_a.get_log()) + log) # Expect abnormal shutdown of the destination VM because of # the failed migration -- 2.29.2 Disposition (action) for any given signal is global for the process. When two threads run coroutine-sigaltstack's qemu_coroutine_new() concurrently, they may interfere with each other: One of them may revert the SIGUSR2 handler to SIG_DFL, between the other thread (a) setting up coroutine_trampoline() as the handler and (b) raising SIGUSR2. That SIGUSR2 will then terminate the QEMU process abnormally. We have to ensure that only one thread at a time can modify the process-global SIGUSR2 handler. To do so, wrap the whole section where that is done in a mutex. Alternatively, we could for example have the SIGUSR2 handler always be coroutine_trampoline(), so there would be no need to invoke sigaction() in qemu_coroutine_new(). Laszlo has posted a patch to do so here: https://lists.nongnu.org/archive/html/qemu-devel/2021-01/msg05962.html However, given that coroutine-sigaltstack is more of a fallback implementation for platforms that do not support ucontext, that change may be a bit too invasive to be comfortable with it. The mutex proposed here may negatively impact performance, but the change is much simpler. Signed-off-by: Max Reitz <mreitz@redhat.com> Message-Id: <20210125120305.19520-1-mreitz@redhat.com> Reviewed-by: Laszlo Ersek <lersek@redhat.com> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> --- util/coroutine-sigaltstack.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/util/coroutine-sigaltstack.c b/util/coroutine-sigaltstack.c index XXXXXXX..XXXXXXX 100644 --- a/util/coroutine-sigaltstack.c +++ b/util/coroutine-sigaltstack.c @@ -XXX,XX +XXX,XX @@ Coroutine *qemu_coroutine_new(void) sigset_t sigs; sigset_t osigs; sigjmp_buf old_env; + static pthread_mutex_t sigusr2_mutex = PTHREAD_MUTEX_INITIALIZER; /* The way to manipulate stack is with the sigaltstack function. We * prepare a stack, with it delivering a signal to ourselves and then @@ -XXX,XX +XXX,XX @@ Coroutine *qemu_coroutine_new(void) sa.sa_handler = coroutine_trampoline; sigfillset(&sa.sa_mask); sa.sa_flags = SA_ONSTACK; + + /* + * sigaction() is a process-global operation. We must not run + * this code in multiple threads at once. + */ + pthread_mutex_lock(&sigusr2_mutex); if (sigaction(SIGUSR2, &sa, &osa) != 0) { abort(); } @@ -XXX,XX +XXX,XX @@ Coroutine *qemu_coroutine_new(void) * Restore the old SIGUSR2 signal handler and mask */ sigaction(SIGUSR2, &osa, NULL); + pthread_mutex_unlock(&sigusr2_mutex); + pthread_sigmask(SIG_SETMASK, &osigs, NULL); /* -- 2.29.2 From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Experiments show, that copy_range is not always making things faster. So, to make experimentation simpler, let's add a parameter. Some more perf parameters will be added soon, so here is a new struct. For now, add new backup qmp parameter with x- prefix for the following reasons: - We are going to add more performance parameters, some will be related to the whole block-copy process, some only to background copying in backup (ignored for copy-before-write operations). - On the other hand, we are going to use block-copy interface in other block jobs, which will need performance options as well.. And it should be the same structure or at least somehow related. So, there are too much unclean things about how the interface and now we need the new options mostly for testing. Let's keep them experimental for a while. In do_backup_common() new x-perf parameter handled in a way to make further options addition simpler. We add use-copy-range with default=true, and we'll change the default in further patch, after moving backup to use block-copy. Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Message-Id: <20210116214705.822267-2-vsementsov@virtuozzo.com> [mreitz: s/5\.2/6.0/] Signed-off-by: Max Reitz <mreitz@redhat.com> --- qapi/block-core.json | 17 ++++++++++++++++- block/backup-top.h | 1 + include/block/block-copy.h | 2 +- include/block/block_int.h | 3 +++ block/backup-top.c | 4 +++- block/backup.c | 6 +++++- block/block-copy.c | 4 ++-- block/replication.c | 2 ++ blockdev.c | 8 ++++++++ 9 files changed, 41 insertions(+), 6 deletions(-) diff --git a/qapi/block-core.json b/qapi/block-core.json index XXXXXXX..XXXXXXX 100644 --- a/qapi/block-core.json +++ b/qapi/block-core.json @@ -XXX,XX +XXX,XX @@ { 'struct': 'BlockdevSnapshot', 'data': { 'node': 'str', 'overlay': 'str' } } +## +# @BackupPerf: +# +# Optional parameters for backup. These parameters don't affect +# functionality, but may significantly affect performance. +# +# @use-copy-range: Use copy offloading. Default true. +# +# Since: 6.0 +## +{ 'struct': 'BackupPerf', + 'data': { '*use-copy-range': 'bool' }} + ## # @BackupCommon: # @@ -XXX,XX +XXX,XX @@ # above node specified by @drive. If this option is not given, # a node name is autogenerated. (Since: 4.2) # +# @x-perf: Performance options. (Since 6.0) +# # Note: @on-source-error and @on-target-error only affect background # I/O. If an error occurs during a guest write request, the device's # rerror/werror actions will be used. @@ -XXX,XX +XXX,XX @@ '*on-source-error': 'BlockdevOnError', '*on-target-error': 'BlockdevOnError', '*auto-finalize': 'bool', '*auto-dismiss': 'bool', - '*filter-node-name': 'str' } } + '*filter-node-name': 'str', '*x-perf': 'BackupPerf' } } ## # @DriveBackup: diff --git a/block/backup-top.h b/block/backup-top.h index XXXXXXX..XXXXXXX 100644 --- a/block/backup-top.h +++ b/block/backup-top.h @@ -XXX,XX +XXX,XX @@ BlockDriverState *bdrv_backup_top_append(BlockDriverState *source, BlockDriverState *target, const char *filter_node_name, uint64_t cluster_size, + BackupPerf *perf, BdrvRequestFlags write_flags, BlockCopyState **bcs, Error **errp); diff --git a/include/block/block-copy.h b/include/block/block-copy.h index XXXXXXX..XXXXXXX 100644 --- a/include/block/block-copy.h +++ b/include/block/block-copy.h @@ -XXX,XX +XXX,XX @@ typedef void (*ProgressBytesCallbackFunc)(int64_t bytes, void *opaque); typedef struct BlockCopyState BlockCopyState; BlockCopyState *block_copy_state_new(BdrvChild *source, BdrvChild *target, - int64_t cluster_size, + int64_t cluster_size, bool use_copy_range, BdrvRequestFlags write_flags, Error **errp); diff --git a/include/block/block_int.h b/include/block/block_int.h index XXXXXXX..XXXXXXX 100644 --- a/include/block/block_int.h +++ b/include/block/block_int.h @@ -XXX,XX +XXX,XX @@ void mirror_start(const char *job_id, BlockDriverState *bs, * @sync_mode: What parts of the disk image should be copied to the destination. * @sync_bitmap: The dirty bitmap if sync_mode is 'bitmap' or 'incremental' * @bitmap_mode: The bitmap synchronization policy to use. + * @perf: Performance options. All actual fields assumed to be present, + * all ".has_*" fields are ignored. * @on_source_error: The action to take upon error reading from the source. * @on_target_error: The action to take upon error writing to the target. * @creation_flags: Flags that control the behavior of the Job lifetime. @@ -XXX,XX +XXX,XX @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs, BitmapSyncMode bitmap_mode, bool compress, const char *filter_node_name, + BackupPerf *perf, BlockdevOnError on_source_error, BlockdevOnError on_target_error, int creation_flags, diff --git a/block/backup-top.c b/block/backup-top.c index XXXXXXX..XXXXXXX 100644 --- a/block/backup-top.c +++ b/block/backup-top.c @@ -XXX,XX +XXX,XX @@ BlockDriverState *bdrv_backup_top_append(BlockDriverState *source, BlockDriverState *target, const char *filter_node_name, uint64_t cluster_size, + BackupPerf *perf, BdrvRequestFlags write_flags, BlockCopyState **bcs, Error **errp) @@ -XXX,XX +XXX,XX @@ BlockDriverState *bdrv_backup_top_append(BlockDriverState *source, state->cluster_size = cluster_size; state->bcs = block_copy_state_new(top->backing, state->target, - cluster_size, write_flags, &local_err); + cluster_size, perf->use_copy_range, + write_flags, &local_err); if (local_err) { error_prepend(&local_err, "Cannot create block-copy-state: "); goto fail; diff --git a/block/backup.c b/block/backup.c index XXXXXXX..XXXXXXX 100644 --- a/block/backup.c +++ b/block/backup.c @@ -XXX,XX +XXX,XX @@ typedef struct BackupBlockJob { uint64_t len; uint64_t bytes_read; int64_t cluster_size; + BackupPerf perf; BlockCopyState *bcs; } BackupBlockJob; @@ -XXX,XX +XXX,XX @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs, BitmapSyncMode bitmap_mode, bool compress, const char *filter_node_name, + BackupPerf *perf, BlockdevOnError on_source_error, BlockdevOnError on_target_error, int creation_flags, @@ -XXX,XX +XXX,XX @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs, (compress ? BDRV_REQ_WRITE_COMPRESSED : 0), backup_top = bdrv_backup_top_append(bs, target, filter_node_name, - cluster_size, write_flags, &bcs, errp); + cluster_size, perf, + write_flags, &bcs, errp); if (!backup_top) { goto error; } @@ -XXX,XX +XXX,XX @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs, job->bcs = bcs; job->cluster_size = cluster_size; job->len = len; + job->perf = *perf; block_copy_set_progress_callback(bcs, backup_progress_bytes_callback, job); block_copy_set_progress_meter(bcs, &job->common.job.progress); diff --git a/block/block-copy.c b/block/block-copy.c index XXXXXXX..XXXXXXX 100644 --- a/block/block-copy.c +++ b/block/block-copy.c @@ -XXX,XX +XXX,XX @@ static uint32_t block_copy_max_transfer(BdrvChild *source, BdrvChild *target) } BlockCopyState *block_copy_state_new(BdrvChild *source, BdrvChild *target, - int64_t cluster_size, + int64_t cluster_size, bool use_copy_range, BdrvRequestFlags write_flags, Error **errp) { BlockCopyState *s; @@ -XXX,XX +XXX,XX @@ BlockCopyState *block_copy_state_new(BdrvChild *source, BdrvChild *target, * We enable copy-range, but keep small copy_size, until first * successful copy_range (look at block_copy_do_copy). */ - s->use_copy_range = true; + s->use_copy_range = use_copy_range; s->copy_size = MAX(s->cluster_size, BLOCK_COPY_MAX_BUFFER); } diff --git a/block/replication.c b/block/replication.c index XXXXXXX..XXXXXXX 100644 --- a/block/replication.c +++ b/block/replication.c @@ -XXX,XX +XXX,XX @@ static void replication_start(ReplicationState *rs, ReplicationMode mode, int64_t active_length, hidden_length, disk_length; AioContext *aio_context; Error *local_err = NULL; + BackupPerf perf = { .use_copy_range = true }; aio_context = bdrv_get_aio_context(bs); aio_context_acquire(aio_context); @@ -XXX,XX +XXX,XX @@ static void replication_start(ReplicationState *rs, ReplicationMode mode, s->backup_job = backup_job_create( NULL, s->secondary_disk->bs, s->hidden_disk->bs, 0, MIRROR_SYNC_MODE_NONE, NULL, 0, false, NULL, + &perf, BLOCKDEV_ON_ERROR_REPORT, BLOCKDEV_ON_ERROR_REPORT, JOB_INTERNAL, backup_job_completed, bs, NULL, &local_err); diff --git a/blockdev.c b/blockdev.c index XXXXXXX..XXXXXXX 100644 --- a/blockdev.c +++ b/blockdev.c @@ -XXX,XX +XXX,XX @@ static BlockJob *do_backup_common(BackupCommon *backup, { BlockJob *job = NULL; BdrvDirtyBitmap *bmap = NULL; + BackupPerf perf = { .use_copy_range = true }; int job_flags = JOB_DEFAULT; if (!backup->has_speed) { @@ -XXX,XX +XXX,XX @@ static BlockJob *do_backup_common(BackupCommon *backup, backup->compress = false; } + if (backup->x_perf) { + if (backup->x_perf->has_use_copy_range) { + perf.use_copy_range = backup->x_perf->use_copy_range; + } + } + if ((backup->sync == MIRROR_SYNC_MODE_BITMAP) || (backup->sync == MIRROR_SYNC_MODE_INCREMENTAL)) { /* done before desugaring 'incremental' to print the right message */ @@ -XXX,XX +XXX,XX @@ static BlockJob *do_backup_common(BackupCommon *backup, backup->sync, bmap, backup->bitmap_mode, backup->compress, backup->filter_node_name, + &perf, backup->on_source_error, backup->on_target_error, job_flags, NULL, NULL, txn, errp); -- 2.29.2 From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Refactor common path to use BlockCopyCallState pointer as parameter, to prepare it for use in asynchronous block-copy (at least, we'll need to run block-copy in a coroutine, passing the whole parameters as one pointer). Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Message-Id: <20210116214705.822267-3-vsementsov@virtuozzo.com> Signed-off-by: Max Reitz <mreitz@redhat.com> --- block/block-copy.c | 51 ++++++++++++++++++++++++++++++++++------------ 1 file changed, 38 insertions(+), 13 deletions(-) diff --git a/block/block-copy.c b/block/block-copy.c index XXXXXXX..XXXXXXX 100644 --- a/block/block-copy.c +++ b/block/block-copy.c @@ -XXX,XX +XXX,XX @@ static coroutine_fn int block_copy_task_entry(AioTask *task); typedef struct BlockCopyCallState { + /* IN parameters */ + BlockCopyState *s; + int64_t offset; + int64_t bytes; + + /* State */ bool failed; + + /* OUT parameters */ bool error_is_read; } BlockCopyCallState; @@ -XXX,XX +XXX,XX @@ int64_t block_copy_reset_unallocated(BlockCopyState *s, * Returns 1 if dirty clusters found and successfully copied, 0 if no dirty * clusters found and -errno on failure. */ -static int coroutine_fn block_copy_dirty_clusters(BlockCopyState *s, - int64_t offset, int64_t bytes, - bool *error_is_read) +static int coroutine_fn +block_copy_dirty_clusters(BlockCopyCallState *call_state) { + BlockCopyState *s = call_state->s; + int64_t offset = call_state->offset; + int64_t bytes = call_state->bytes; + int ret = 0; bool found_dirty = false; int64_t end = offset + bytes; AioTaskPool *aio = NULL; - BlockCopyCallState call_state = {false, false}; /* * block_copy() user is responsible for keeping source and target in same @@ -XXX,XX +XXX,XX @@ static int coroutine_fn block_copy_dirty_clusters(BlockCopyState *s, BlockCopyTask *task; int64_t status_bytes; - task = block_copy_task_create(s, &call_state, offset, bytes); + task = block_copy_task_create(s, call_state, offset, bytes); if (!task) { /* No more dirty bits in the bitmap */ trace_block_copy_skip_range(s, offset, bytes); @@ -XXX,XX +XXX,XX @@ out: aio_task_pool_free(aio); } - if (error_is_read && ret < 0) { - *error_is_read = call_state.error_is_read; - } return ret < 0 ? ret : found_dirty; } /* - * block_copy + * block_copy_common * * Copy requested region, accordingly to dirty bitmap. * Collaborate with parallel block_copy requests: if they succeed it will help @@ -XXX,XX +XXX,XX @@ out: * it means that some I/O operation failed in context of _this_ block_copy call, * not some parallel operation. */ -int coroutine_fn block_copy(BlockCopyState *s, int64_t offset, int64_t bytes, - bool *error_is_read) +static int coroutine_fn block_copy_common(BlockCopyCallState *call_state) { int ret; do { - ret = block_copy_dirty_clusters(s, offset, bytes, error_is_read); + ret = block_copy_dirty_clusters(call_state); if (ret == 0) { - ret = block_copy_wait_one(s, offset, bytes); + ret = block_copy_wait_one(call_state->s, call_state->offset, + call_state->bytes); } /* @@ -XXX,XX +XXX,XX @@ int coroutine_fn block_copy(BlockCopyState *s, int64_t offset, int64_t bytes, return ret; } +int coroutine_fn block_copy(BlockCopyState *s, int64_t start, int64_t bytes, + bool *error_is_read) +{ + BlockCopyCallState call_state = { + .s = s, + .offset = start, + .bytes = bytes, + }; + + int ret = block_copy_common(&call_state); + + if (error_is_read && ret < 0) { + *error_is_read = call_state.error_is_read; + } + + return ret; +} + BdrvDirtyBitmap *block_copy_dirty_bitmap(BlockCopyState *s) { return s->copy_bitmap; -- 2.29.2 From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> We'll need async block-copy invocation to use in backup directly. Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Message-Id: <20210116214705.822267-4-vsementsov@virtuozzo.com> Signed-off-by: Max Reitz <mreitz@redhat.com> --- include/block/block-copy.h | 29 ++++++++++++++ block/block-copy.c | 81 ++++++++++++++++++++++++++++++++++++-- 2 files changed, 106 insertions(+), 4 deletions(-) diff --git a/include/block/block-copy.h b/include/block/block-copy.h index XXXXXXX..XXXXXXX 100644 --- a/include/block/block-copy.h +++ b/include/block/block-copy.h @@ -XXX,XX +XXX,XX @@ #include "qemu/co-shared-resource.h" typedef void (*ProgressBytesCallbackFunc)(int64_t bytes, void *opaque); +typedef void (*BlockCopyAsyncCallbackFunc)(void *opaque); typedef struct BlockCopyState BlockCopyState; +typedef struct BlockCopyCallState BlockCopyCallState; BlockCopyState *block_copy_state_new(BdrvChild *source, BdrvChild *target, int64_t cluster_size, bool use_copy_range, @@ -XXX,XX +XXX,XX @@ int64_t block_copy_reset_unallocated(BlockCopyState *s, int coroutine_fn block_copy(BlockCopyState *s, int64_t offset, int64_t bytes, bool *error_is_read); +/* + * Run block-copy in a coroutine, create corresponding BlockCopyCallState + * object and return pointer to it. Never returns NULL. + * + * Caller is responsible to call block_copy_call_free() to free + * BlockCopyCallState object. + */ +BlockCopyCallState *block_copy_async(BlockCopyState *s, + int64_t offset, int64_t bytes, + BlockCopyAsyncCallbackFunc cb, + void *cb_opaque); + +/* + * Free finished BlockCopyCallState. Trying to free running + * block-copy will crash. + */ +void block_copy_call_free(BlockCopyCallState *call_state); + +/* + * Note, that block-copy call is marked finished prior to calling + * the callback. + */ +bool block_copy_call_finished(BlockCopyCallState *call_state); +bool block_copy_call_succeeded(BlockCopyCallState *call_state); +bool block_copy_call_failed(BlockCopyCallState *call_state); +int block_copy_call_status(BlockCopyCallState *call_state, bool *error_is_read); + BdrvDirtyBitmap *block_copy_dirty_bitmap(BlockCopyState *s); void block_copy_set_skip_unallocated(BlockCopyState *s, bool skip); diff --git a/block/block-copy.c b/block/block-copy.c index XXXXXXX..XXXXXXX 100644 --- a/block/block-copy.c +++ b/block/block-copy.c @@ -XXX,XX +XXX,XX @@ static coroutine_fn int block_copy_task_entry(AioTask *task); typedef struct BlockCopyCallState { - /* IN parameters */ + /* IN parameters. Initialized in block_copy_async() and never changed. */ BlockCopyState *s; int64_t offset; int64_t bytes; + BlockCopyAsyncCallbackFunc cb; + void *cb_opaque; + + /* Coroutine where async block-copy is running */ + Coroutine *co; /* State */ - bool failed; + int ret; + bool finished; /* OUT parameters */ bool error_is_read; @@ -XXX,XX +XXX,XX @@ static coroutine_fn int block_copy_task_entry(AioTask *task) ret = block_copy_do_copy(t->s, t->offset, t->bytes, t->zeroes, &error_is_read); - if (ret < 0 && !t->call_state->failed) { - t->call_state->failed = true; + if (ret < 0 && !t->call_state->ret) { + t->call_state->ret = ret; t->call_state->error_is_read = error_is_read; } else { progress_work_done(t->s->progress, t->bytes); @@ -XXX,XX +XXX,XX @@ static int coroutine_fn block_copy_common(BlockCopyCallState *call_state) */ } while (ret > 0); + call_state->finished = true; + + if (call_state->cb) { + call_state->cb(call_state->cb_opaque); + } + return ret; } @@ -XXX,XX +XXX,XX @@ int coroutine_fn block_copy(BlockCopyState *s, int64_t start, int64_t bytes, return ret; } +static void coroutine_fn block_copy_async_co_entry(void *opaque) +{ + block_copy_common(opaque); +} + +BlockCopyCallState *block_copy_async(BlockCopyState *s, + int64_t offset, int64_t bytes, + BlockCopyAsyncCallbackFunc cb, + void *cb_opaque) +{ + BlockCopyCallState *call_state = g_new(BlockCopyCallState, 1); + + *call_state = (BlockCopyCallState) { + .s = s, + .offset = offset, + .bytes = bytes, + .cb = cb, + .cb_opaque = cb_opaque, + + .co = qemu_coroutine_create(block_copy_async_co_entry, call_state), + }; + + qemu_coroutine_enter(call_state->co); + + return call_state; +} + +void block_copy_call_free(BlockCopyCallState *call_state) +{ + if (!call_state) { + return; + } + + assert(call_state->finished); + g_free(call_state); +} + +bool block_copy_call_finished(BlockCopyCallState *call_state) +{ + return call_state->finished; +} + +bool block_copy_call_succeeded(BlockCopyCallState *call_state) +{ + return call_state->finished && call_state->ret == 0; +} + +bool block_copy_call_failed(BlockCopyCallState *call_state) +{ + return call_state->finished && call_state->ret < 0; +} + +int block_copy_call_status(BlockCopyCallState *call_state, bool *error_is_read) +{ + assert(call_state->finished); + if (error_is_read) { + *error_is_read = call_state->error_is_read; + } + return call_state->ret; +} + BdrvDirtyBitmap *block_copy_dirty_bitmap(BlockCopyState *s) { return s->copy_bitmap; -- 2.29.2 From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> They will be used for backup. Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Message-Id: <20210116214705.822267-5-vsementsov@virtuozzo.com> Signed-off-by: Max Reitz <mreitz@redhat.com> --- include/block/block-copy.h | 6 ++++++ block/block-copy.c | 11 +++++++++-- 2 files changed, 15 insertions(+), 2 deletions(-) diff --git a/include/block/block-copy.h b/include/block/block-copy.h index XXXXXXX..XXXXXXX 100644 --- a/include/block/block-copy.h +++ b/include/block/block-copy.h @@ -XXX,XX +XXX,XX @@ int coroutine_fn block_copy(BlockCopyState *s, int64_t offset, int64_t bytes, * * Caller is responsible to call block_copy_call_free() to free * BlockCopyCallState object. + * + * @max_workers means maximum of parallel coroutines to execute sub-requests, + * must be > 0. + * + * @max_chunk means maximum length for one IO operation. Zero means unlimited. */ BlockCopyCallState *block_copy_async(BlockCopyState *s, int64_t offset, int64_t bytes, + int max_workers, int64_t max_chunk, BlockCopyAsyncCallbackFunc cb, void *cb_opaque); diff --git a/block/block-copy.c b/block/block-copy.c index XXXXXXX..XXXXXXX 100644 --- a/block/block-copy.c +++ b/block/block-copy.c @@ -XXX,XX +XXX,XX @@ typedef struct BlockCopyCallState { BlockCopyState *s; int64_t offset; int64_t bytes; + int max_workers; + int64_t max_chunk; BlockCopyAsyncCallbackFunc cb; void *cb_opaque; @@ -XXX,XX +XXX,XX @@ static BlockCopyTask *block_copy_task_create(BlockCopyState *s, int64_t offset, int64_t bytes) { BlockCopyTask *task; + int64_t max_chunk = MIN_NON_ZERO(s->copy_size, call_state->max_chunk); if (!bdrv_dirty_bitmap_next_dirty_area(s->copy_bitmap, offset, offset + bytes, - s->copy_size, &offset, &bytes)) + max_chunk, &offset, &bytes)) { return NULL; } @@ -XXX,XX +XXX,XX @@ block_copy_dirty_clusters(BlockCopyCallState *call_state) bytes = end - offset; if (!aio && bytes) { - aio = aio_task_pool_new(BLOCK_COPY_MAX_WORKERS); + aio = aio_task_pool_new(call_state->max_workers); } ret = block_copy_task_run(aio, task); @@ -XXX,XX +XXX,XX @@ int coroutine_fn block_copy(BlockCopyState *s, int64_t start, int64_t bytes, .s = s, .offset = start, .bytes = bytes, + .max_workers = BLOCK_COPY_MAX_WORKERS, }; int ret = block_copy_common(&call_state); @@ -XXX,XX +XXX,XX @@ static void coroutine_fn block_copy_async_co_entry(void *opaque) BlockCopyCallState *block_copy_async(BlockCopyState *s, int64_t offset, int64_t bytes, + int max_workers, int64_t max_chunk, BlockCopyAsyncCallbackFunc cb, void *cb_opaque) { @@ -XXX,XX +XXX,XX @@ BlockCopyCallState *block_copy_async(BlockCopyState *s, .s = s, .offset = offset, .bytes = bytes, + .max_workers = max_workers, + .max_chunk = max_chunk, .cb = cb, .cb_opaque = cb_opaque, -- 2.29.2 From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> It simplifies debugging. Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Message-Id: <20210116214705.822267-6-vsementsov@virtuozzo.com> Signed-off-by: Max Reitz <mreitz@redhat.com> --- block/block-copy.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/block/block-copy.c b/block/block-copy.c index XXXXXXX..XXXXXXX 100644 --- a/block/block-copy.c +++ b/block/block-copy.c @@ -XXX,XX +XXX,XX @@ typedef struct BlockCopyCallState { /* Coroutine where async block-copy is running */ Coroutine *co; + /* To reference all call states from BlockCopyState */ + QLIST_ENTRY(BlockCopyCallState) list; + /* State */ int ret; bool finished; @@ -XXX,XX +XXX,XX @@ typedef struct BlockCopyState { bool use_copy_range; int64_t copy_size; uint64_t len; - QLIST_HEAD(, BlockCopyTask) tasks; + QLIST_HEAD(, BlockCopyTask) tasks; /* All tasks from all block-copy calls */ + QLIST_HEAD(, BlockCopyCallState) calls; BdrvRequestFlags write_flags; @@ -XXX,XX +XXX,XX @@ BlockCopyState *block_copy_state_new(BdrvChild *source, BdrvChild *target, } QLIST_INIT(&s->tasks); + QLIST_INIT(&s->calls); return s; } @@ -XXX,XX +XXX,XX @@ static int coroutine_fn block_copy_common(BlockCopyCallState *call_state) { int ret; + QLIST_INSERT_HEAD(&call_state->s->calls, call_state, list); + do { ret = block_copy_dirty_clusters(call_state); @@ -XXX,XX +XXX,XX @@ static int coroutine_fn block_copy_common(BlockCopyCallState *call_state) call_state->cb(call_state->cb_opaque); } + QLIST_REMOVE(call_state, list); + return ret; } -- 2.29.2 From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> We are going to directly use one async block-copy operation for backup job, so we need rate limiter. We want to maintain current backup behavior: only background copying is limited and copy-before-write operations only participate in limit calculation. Therefore we need one rate limiter for block-copy state and boolean flag for block-copy call state for actual limitation. Note, that we can't just calculate each chunk in limiter after successful copying: it will not save us from starting a lot of async sub-requests which will exceed limit too much. Instead let's use the following scheme on sub-request creation: 1. If at the moment limit is not exceeded, create the request and account it immediately. 2. If at the moment limit is already exceeded, drop create sub-request and handle limit instead (by sleep). With this approach we'll never exceed the limit more than by one sub-request (which pretty much matches current backup behavior). Note also, that if there is in-flight block-copy async call, block_copy_kick() should be used after set-speed to apply new setup faster. For that block_copy_kick() published in this patch. Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Message-Id: <20210116214705.822267-7-vsementsov@virtuozzo.com> Signed-off-by: Max Reitz <mreitz@redhat.com> --- include/block/block-copy.h | 5 ++++- block/backup-top.c | 2 +- block/backup.c | 2 +- block/block-copy.c | 46 +++++++++++++++++++++++++++++++++++++- 4 files changed, 51 insertions(+), 4 deletions(-) diff --git a/include/block/block-copy.h b/include/block/block-copy.h index XXXXXXX..XXXXXXX 100644 --- a/include/block/block-copy.h +++ b/include/block/block-copy.h @@ -XXX,XX +XXX,XX @@ int64_t block_copy_reset_unallocated(BlockCopyState *s, int64_t offset, int64_t *count); int coroutine_fn block_copy(BlockCopyState *s, int64_t offset, int64_t bytes, - bool *error_is_read); + bool ignore_ratelimit, bool *error_is_read); /* * Run block-copy in a coroutine, create corresponding BlockCopyCallState @@ -XXX,XX +XXX,XX @@ bool block_copy_call_succeeded(BlockCopyCallState *call_state); bool block_copy_call_failed(BlockCopyCallState *call_state); int block_copy_call_status(BlockCopyCallState *call_state, bool *error_is_read); +void block_copy_set_speed(BlockCopyState *s, uint64_t speed); +void block_copy_kick(BlockCopyCallState *call_state); + BdrvDirtyBitmap *block_copy_dirty_bitmap(BlockCopyState *s); void block_copy_set_skip_unallocated(BlockCopyState *s, bool skip); diff --git a/block/backup-top.c b/block/backup-top.c index XXXXXXX..XXXXXXX 100644 --- a/block/backup-top.c +++ b/block/backup-top.c @@ -XXX,XX +XXX,XX @@ static coroutine_fn int backup_top_cbw(BlockDriverState *bs, uint64_t offset, off = QEMU_ALIGN_DOWN(offset, s->cluster_size); end = QEMU_ALIGN_UP(offset + bytes, s->cluster_size); - return block_copy(s->bcs, off, end - off, NULL); + return block_copy(s->bcs, off, end - off, true, NULL); } static int coroutine_fn backup_top_co_pdiscard(BlockDriverState *bs, diff --git a/block/backup.c b/block/backup.c index XXXXXXX..XXXXXXX 100644 --- a/block/backup.c +++ b/block/backup.c @@ -XXX,XX +XXX,XX @@ static int coroutine_fn backup_do_cow(BackupBlockJob *job, trace_backup_do_cow_enter(job, start, offset, bytes); - ret = block_copy(job->bcs, start, end - start, error_is_read); + ret = block_copy(job->bcs, start, end - start, true, error_is_read); trace_backup_do_cow_return(job, offset, bytes, ret); diff --git a/block/block-copy.c b/block/block-copy.c index XXXXXXX..XXXXXXX 100644 --- a/block/block-copy.c +++ b/block/block-copy.c @@ -XXX,XX +XXX,XX @@ #define BLOCK_COPY_MAX_BUFFER (1 * MiB) #define BLOCK_COPY_MAX_MEM (128 * MiB) #define BLOCK_COPY_MAX_WORKERS 64 +#define BLOCK_COPY_SLICE_TIME 100000000ULL /* ns */ static coroutine_fn int block_copy_task_entry(AioTask *task); @@ -XXX,XX +XXX,XX @@ typedef struct BlockCopyCallState { int64_t bytes; int max_workers; int64_t max_chunk; + bool ignore_ratelimit; BlockCopyAsyncCallbackFunc cb; void *cb_opaque; @@ -XXX,XX +XXX,XX @@ typedef struct BlockCopyCallState { /* State */ int ret; bool finished; + QemuCoSleepState *sleep_state; /* OUT parameters */ bool error_is_read; @@ -XXX,XX +XXX,XX @@ typedef struct BlockCopyState { void *progress_opaque; SharedResource *mem; + + uint64_t speed; + RateLimit rate_limit; } BlockCopyState; static BlockCopyTask *find_conflicting_task(BlockCopyState *s, @@ -XXX,XX +XXX,XX @@ block_copy_dirty_clusters(BlockCopyCallState *call_state) } task->zeroes = ret & BDRV_BLOCK_ZERO; + if (s->speed) { + if (!call_state->ignore_ratelimit) { + uint64_t ns = ratelimit_calculate_delay(&s->rate_limit, 0); + if (ns > 0) { + block_copy_task_end(task, -EAGAIN); + g_free(task); + qemu_co_sleep_ns_wakeable(QEMU_CLOCK_REALTIME, ns, + &call_state->sleep_state); + continue; + } + } + + ratelimit_calculate_delay(&s->rate_limit, task->bytes); + } + trace_block_copy_process(s, task->offset); co_get_from_shres(s->mem, task->bytes); @@ -XXX,XX +XXX,XX @@ out: return ret < 0 ? ret : found_dirty; } +void block_copy_kick(BlockCopyCallState *call_state) +{ + if (call_state->sleep_state) { + qemu_co_sleep_wake(call_state->sleep_state); + } +} + /* * block_copy_common * @@ -XXX,XX +XXX,XX @@ static int coroutine_fn block_copy_common(BlockCopyCallState *call_state) } int coroutine_fn block_copy(BlockCopyState *s, int64_t start, int64_t bytes, - bool *error_is_read) + bool ignore_ratelimit, bool *error_is_read) { BlockCopyCallState call_state = { .s = s, .offset = start, .bytes = bytes, + .ignore_ratelimit = ignore_ratelimit, .max_workers = BLOCK_COPY_MAX_WORKERS, }; @@ -XXX,XX +XXX,XX @@ void block_copy_set_skip_unallocated(BlockCopyState *s, bool skip) { s->skip_unallocated = skip; } + +void block_copy_set_speed(BlockCopyState *s, uint64_t speed) +{ + s->speed = speed; + if (speed > 0) { + ratelimit_set_speed(&s->rate_limit, speed, BLOCK_COPY_SLICE_TIME); + } + + /* + * Note: it's good to kick all call states from here, but it should be done + * only from a coroutine, to not crash if s->calls list changed while + * entering one call. So for now, the only user of this function kicks its + * only one call_state by hand. + */ +} -- 2.29.2 From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Add function to cancel running async block-copy call. It will be used in backup. Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Message-Id: <20210116214705.822267-8-vsementsov@virtuozzo.com> Signed-off-by: Max Reitz <mreitz@redhat.com> --- include/block/block-copy.h | 13 +++++++++++++ block/block-copy.c | 24 +++++++++++++++++++----- 2 files changed, 32 insertions(+), 5 deletions(-) diff --git a/include/block/block-copy.h b/include/block/block-copy.h index XXXXXXX..XXXXXXX 100644 --- a/include/block/block-copy.h +++ b/include/block/block-copy.h @@ -XXX,XX +XXX,XX @@ void block_copy_call_free(BlockCopyCallState *call_state); bool block_copy_call_finished(BlockCopyCallState *call_state); bool block_copy_call_succeeded(BlockCopyCallState *call_state); bool block_copy_call_failed(BlockCopyCallState *call_state); +bool block_copy_call_cancelled(BlockCopyCallState *call_state); int block_copy_call_status(BlockCopyCallState *call_state, bool *error_is_read); void block_copy_set_speed(BlockCopyState *s, uint64_t speed); void block_copy_kick(BlockCopyCallState *call_state); +/* + * Cancel running block-copy call. + * + * Cancel leaves block-copy state valid: dirty bits are correct and you may use + * cancel + <run block_copy with same parameters> to emulate pause/resume. + * + * Note also, that the cancel is async: it only marks block-copy call to be + * cancelled. So, the call may be cancelled (block_copy_call_cancelled() reports + * true) but not yet finished (block_copy_call_finished() reports false). + */ +void block_copy_call_cancel(BlockCopyCallState *call_state); + BdrvDirtyBitmap *block_copy_dirty_bitmap(BlockCopyState *s); void block_copy_set_skip_unallocated(BlockCopyState *s, bool skip); diff --git a/block/block-copy.c b/block/block-copy.c index XXXXXXX..XXXXXXX 100644 --- a/block/block-copy.c +++ b/block/block-copy.c @@ -XXX,XX +XXX,XX @@ typedef struct BlockCopyCallState { int ret; bool finished; QemuCoSleepState *sleep_state; + bool cancelled; /* OUT parameters */ bool error_is_read; @@ -XXX,XX +XXX,XX @@ block_copy_dirty_clusters(BlockCopyCallState *call_state) assert(QEMU_IS_ALIGNED(offset, s->cluster_size)); assert(QEMU_IS_ALIGNED(bytes, s->cluster_size)); - while (bytes && aio_task_pool_status(aio) == 0) { + while (bytes && aio_task_pool_status(aio) == 0 && !call_state->cancelled) { BlockCopyTask *task; int64_t status_bytes; @@ -XXX,XX +XXX,XX @@ static int coroutine_fn block_copy_common(BlockCopyCallState *call_state) do { ret = block_copy_dirty_clusters(call_state); - if (ret == 0) { + if (ret == 0 && !call_state->cancelled) { ret = block_copy_wait_one(call_state->s, call_state->offset, call_state->bytes); } @@ -XXX,XX +XXX,XX @@ static int coroutine_fn block_copy_common(BlockCopyCallState *call_state) * 2. We have waited for some intersecting block-copy request * It may have failed and produced new dirty bits. */ - } while (ret > 0); + } while (ret > 0 && !call_state->cancelled); call_state->finished = true; @@ -XXX,XX +XXX,XX @@ bool block_copy_call_finished(BlockCopyCallState *call_state) bool block_copy_call_succeeded(BlockCopyCallState *call_state) { - return call_state->finished && call_state->ret == 0; + return call_state->finished && !call_state->cancelled && + call_state->ret == 0; } bool block_copy_call_failed(BlockCopyCallState *call_state) { - return call_state->finished && call_state->ret < 0; + return call_state->finished && !call_state->cancelled && + call_state->ret < 0; +} + +bool block_copy_call_cancelled(BlockCopyCallState *call_state) +{ + return call_state->cancelled; } int block_copy_call_status(BlockCopyCallState *call_state, bool *error_is_read) @@ -XXX,XX +XXX,XX @@ int block_copy_call_status(BlockCopyCallState *call_state, bool *error_is_read) return call_state->ret; } +void block_copy_call_cancel(BlockCopyCallState *call_state) +{ + call_state->cancelled = true; + block_copy_kick(call_state); +} + BdrvDirtyBitmap *block_copy_dirty_bitmap(BlockCopyState *s) { return s->copy_bitmap; -- 2.29.2 From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> We are going to use async block-copy call in backup, so we'll need to passthrough setting backup speed to block-copy call. Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Message-Id: <20210116214705.822267-9-vsementsov@virtuozzo.com> Signed-off-by: Max Reitz <mreitz@redhat.com> --- include/block/blockjob_int.h | 2 ++ blockjob.c | 6 ++++++ 2 files changed, 8 insertions(+) diff --git a/include/block/blockjob_int.h b/include/block/blockjob_int.h index XXXXXXX..XXXXXXX 100644 --- a/include/block/blockjob_int.h +++ b/include/block/blockjob_int.h @@ -XXX,XX +XXX,XX @@ struct BlockJobDriver { * besides job->blk to the new AioContext. */ void (*attached_aio_context)(BlockJob *job, AioContext *new_context); + + void (*set_speed)(BlockJob *job, int64_t speed); }; /** diff --git a/blockjob.c b/blockjob.c index XXXXXXX..XXXXXXX 100644 --- a/blockjob.c +++ b/blockjob.c @@ -XXX,XX +XXX,XX @@ static bool job_timer_pending(Job *job) void block_job_set_speed(BlockJob *job, int64_t speed, Error **errp) { + const BlockJobDriver *drv = block_job_driver(job); int64_t old_speed = job->speed; if (job_apply_verb(&job->job, JOB_VERB_SET_SPEED, errp)) { @@ -XXX,XX +XXX,XX @@ void block_job_set_speed(BlockJob *job, int64_t speed, Error **errp) ratelimit_set_speed(&job->limit, speed, BLOCK_JOB_SLICE_TIME); job->speed = speed; + + if (drv->set_speed) { + drv->set_speed(job, speed); + } + if (speed && speed <= old_speed) { return; } -- 2.29.2 From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> If main job coroutine called job_yield (while some background process is in progress), we should give it a chance to call job_pause_point(). It will be used in backup, when moved on async block-copy. Note, that job_user_pause is not enough: we want to handle child_job_drained_begin() as well, which call job_pause(). Still, if job is already in job_do_yield() in job_pause_point() we should not enter it. iotest 109 output is modified: on stop we do bdrv_drain_all() which now triggers job pause immediately (and pause after ready is standby). Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Message-Id: <20210116214705.822267-10-vsementsov@virtuozzo.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Signed-off-by: Max Reitz <mreitz@redhat.com> --- job.c | 3 +++ tests/qemu-iotests/109.out | 24 ++++++++++++++++++++++++ 2 files changed, 27 insertions(+) diff --git a/job.c b/job.c index XXXXXXX..XXXXXXX 100644 --- a/job.c +++ b/job.c @@ -XXX,XX +XXX,XX @@ static bool job_timer_not_pending(Job *job) void job_pause(Job *job) { job->pause_count++; + if (!job->paused) { + job_enter(job); + } } void job_resume(Job *job) diff --git a/tests/qemu-iotests/109.out b/tests/qemu-iotests/109.out index XXXXXXX..XXXXXXX 100644 --- a/tests/qemu-iotests/109.out +++ b/tests/qemu-iotests/109.out @@ -XXX,XX +XXX,XX @@ read 512/512 bytes at offset 0 {"execute":"quit"} {"return": {}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false, "reason": "host-qmp-quit"}} +{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "standby", "id": "src"}} +{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "ready", "id": "src"}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "waiting", "id": "src"}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "pending", "id": "src"}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "BLOCK_JOB_COMPLETED", "data": {"device": "src", "len": 1024, "offset": 1024, "speed": 0, "type": "mirror"}} @@ -XXX,XX +XXX,XX @@ read 512/512 bytes at offset 0 {"execute":"quit"} {"return": {}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false, "reason": "host-qmp-quit"}} +{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "standby", "id": "src"}} +{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "ready", "id": "src"}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "waiting", "id": "src"}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "pending", "id": "src"}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "BLOCK_JOB_COMPLETED", "data": {"device": "src", "len": 197120, "offset": 197120, "speed": 0, "type": "mirror"}} @@ -XXX,XX +XXX,XX @@ read 512/512 bytes at offset 0 {"execute":"quit"} {"return": {}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false, "reason": "host-qmp-quit"}} +{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "standby", "id": "src"}} +{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "ready", "id": "src"}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "waiting", "id": "src"}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "pending", "id": "src"}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "BLOCK_JOB_COMPLETED", "data": {"device": "src", "len": 327680, "offset": 327680, "speed": 0, "type": "mirror"}} @@ -XXX,XX +XXX,XX @@ read 512/512 bytes at offset 0 {"execute":"quit"} {"return": {}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false, "reason": "host-qmp-quit"}} +{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "standby", "id": "src"}} +{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "ready", "id": "src"}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "waiting", "id": "src"}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "pending", "id": "src"}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "BLOCK_JOB_COMPLETED", "data": {"device": "src", "len": 1024, "offset": 1024, "speed": 0, "type": "mirror"}} @@ -XXX,XX +XXX,XX @@ read 512/512 bytes at offset 0 {"execute":"quit"} {"return": {}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false, "reason": "host-qmp-quit"}} +{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "standby", "id": "src"}} +{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "ready", "id": "src"}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "waiting", "id": "src"}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "pending", "id": "src"}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "BLOCK_JOB_COMPLETED", "data": {"device": "src", "len": 65536, "offset": 65536, "speed": 0, "type": "mirror"}} @@ -XXX,XX +XXX,XX @@ read 512/512 bytes at offset 0 {"execute":"quit"} {"return": {}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false, "reason": "host-qmp-quit"}} +{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "standby", "id": "src"}} +{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "ready", "id": "src"}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "waiting", "id": "src"}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "pending", "id": "src"}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "BLOCK_JOB_COMPLETED", "data": {"device": "src", "len": 2560, "offset": 2560, "speed": 0, "type": "mirror"}} @@ -XXX,XX +XXX,XX @@ read 512/512 bytes at offset 0 {"execute":"quit"} {"return": {}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false, "reason": "host-qmp-quit"}} +{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "standby", "id": "src"}} +{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "ready", "id": "src"}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "waiting", "id": "src"}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "pending", "id": "src"}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "BLOCK_JOB_COMPLETED", "data": {"device": "src", "len": 2560, "offset": 2560, "speed": 0, "type": "mirror"}} @@ -XXX,XX +XXX,XX @@ read 512/512 bytes at offset 0 {"execute":"quit"} {"return": {}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false, "reason": "host-qmp-quit"}} +{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "standby", "id": "src"}} +{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "ready", "id": "src"}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "waiting", "id": "src"}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "pending", "id": "src"}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "BLOCK_JOB_COMPLETED", "data": {"device": "src", "len": 31457280, "offset": 31457280, "speed": 0, "type": "mirror"}} @@ -XXX,XX +XXX,XX @@ read 512/512 bytes at offset 0 {"execute":"quit"} {"return": {}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false, "reason": "host-qmp-quit"}} +{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "standby", "id": "src"}} +{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "ready", "id": "src"}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "waiting", "id": "src"}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "pending", "id": "src"}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "BLOCK_JOB_COMPLETED", "data": {"device": "src", "len": 327680, "offset": 327680, "speed": 0, "type": "mirror"}} @@ -XXX,XX +XXX,XX @@ read 512/512 bytes at offset 0 {"execute":"quit"} {"return": {}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false, "reason": "host-qmp-quit"}} +{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "standby", "id": "src"}} +{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "ready", "id": "src"}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "waiting", "id": "src"}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "pending", "id": "src"}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "BLOCK_JOB_COMPLETED", "data": {"device": "src", "len": 2048, "offset": 2048, "speed": 0, "type": "mirror"}} @@ -XXX,XX +XXX,XX @@ WARNING: Image format was not specified for 'TEST_DIR/t.raw' and probing guessed {"execute":"quit"} {"return": {}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false, "reason": "host-qmp-quit"}} +{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "standby", "id": "src"}} +{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "ready", "id": "src"}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "waiting", "id": "src"}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "pending", "id": "src"}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "BLOCK_JOB_COMPLETED", "data": {"device": "src", "len": 512, "offset": 512, "speed": 0, "type": "mirror"}} @@ -XXX,XX +XXX,XX @@ Images are identical. {"execute":"quit"} {"return": {}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false, "reason": "host-qmp-quit"}} +{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "standby", "id": "src"}} +{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "ready", "id": "src"}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "waiting", "id": "src"}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "pending", "id": "src"}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "BLOCK_JOB_COMPLETED", "data": {"device": "src", "len": 512, "offset": 512, "speed": 0, "type": "mirror"}} -- 2.29.2 From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Add new parameters to configure future backup features. The patch doesn't introduce aio backup requests (so we actually have only one worker) neither requests larger than one cluster. Still, formally we satisfy these maximums anyway, so add the parameters now, to facilitate further patch which will really change backup job behavior. Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Message-Id: <20210116214705.822267-11-vsementsov@virtuozzo.com> Signed-off-by: Max Reitz <mreitz@redhat.com> --- qapi/block-core.json | 13 ++++++++++++- block/backup.c | 28 +++++++++++++++++++++++----- block/replication.c | 2 +- blockdev.c | 8 +++++++- 4 files changed, 43 insertions(+), 8 deletions(-) diff --git a/qapi/block-core.json b/qapi/block-core.json index XXXXXXX..XXXXXXX 100644 --- a/qapi/block-core.json +++ b/qapi/block-core.json @@ -XXX,XX +XXX,XX @@ # # @use-copy-range: Use copy offloading. Default true. # +# @max-workers: Maximum number of parallel requests for the sustained background +# copying process. Doesn't influence copy-before-write operations. +# Default 64. +# +# @max-chunk: Maximum request length for the sustained background copying +# process. Doesn't influence copy-before-write operations. +# 0 means unlimited. If max-chunk is non-zero then it should not be +# less than job cluster size which is calculated as maximum of +# target image cluster size and 64k. Default 0. +# # Since: 6.0 ## { 'struct': 'BackupPerf', - 'data': { '*use-copy-range': 'bool' }} + 'data': { '*use-copy-range': 'bool', + '*max-workers': 'int', '*max-chunk': 'int64' } } ## # @BackupCommon: diff --git a/block/backup.c b/block/backup.c index XXXXXXX..XXXXXXX 100644 --- a/block/backup.c +++ b/block/backup.c @@ -XXX,XX +XXX,XX @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs, return NULL; } + cluster_size = backup_calculate_cluster_size(target, errp); + if (cluster_size < 0) { + goto error; + } + + if (perf->max_workers < 1) { + error_setg(errp, "max-workers must be greater than zero"); + return NULL; + } + + if (perf->max_chunk < 0) { + error_setg(errp, "max-chunk must be zero (which means no limit) or " + "positive"); + return NULL; + } + + if (perf->max_chunk && perf->max_chunk < cluster_size) { + error_setg(errp, "Required max-chunk (%" PRIi64 ") is less than backup " + "cluster size (%" PRIi64 ")", perf->max_chunk, cluster_size); + return NULL; + } + + if (sync_bitmap) { /* If we need to write to this bitmap, check that we can: */ if (bitmap_mode != BITMAP_SYNC_MODE_NEVER && @@ -XXX,XX +XXX,XX @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs, goto error; } - cluster_size = backup_calculate_cluster_size(target, errp); - if (cluster_size < 0) { - goto error; - } - /* * If source is in backing chain of target assume that target is going to be * used for "image fleecing", i.e. it should represent a kind of snapshot of diff --git a/block/replication.c b/block/replication.c index XXXXXXX..XXXXXXX 100644 --- a/block/replication.c +++ b/block/replication.c @@ -XXX,XX +XXX,XX @@ static void replication_start(ReplicationState *rs, ReplicationMode mode, int64_t active_length, hidden_length, disk_length; AioContext *aio_context; Error *local_err = NULL; - BackupPerf perf = { .use_copy_range = true }; + BackupPerf perf = { .use_copy_range = true, .max_workers = 1 }; aio_context = bdrv_get_aio_context(bs); aio_context_acquire(aio_context); diff --git a/blockdev.c b/blockdev.c index XXXXXXX..XXXXXXX 100644 --- a/blockdev.c +++ b/blockdev.c @@ -XXX,XX +XXX,XX @@ static BlockJob *do_backup_common(BackupCommon *backup, { BlockJob *job = NULL; BdrvDirtyBitmap *bmap = NULL; - BackupPerf perf = { .use_copy_range = true }; + BackupPerf perf = { .use_copy_range = true, .max_workers = 64 }; int job_flags = JOB_DEFAULT; if (!backup->has_speed) { @@ -XXX,XX +XXX,XX @@ static BlockJob *do_backup_common(BackupCommon *backup, if (backup->x_perf->has_use_copy_range) { perf.use_copy_range = backup->x_perf->use_copy_range; } + if (backup->x_perf->has_max_workers) { + perf.max_workers = backup->x_perf->max_workers; + } + if (backup->x_perf->has_max_chunk) { + perf.max_chunk = backup->x_perf->max_chunk; + } } if ((backup->sync == MIRROR_SYNC_MODE_BITMAP) || -- 2.29.2 From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> After introducing parallel async copy requests instead of plain cluster-by-cluster copying loop, we'll have to wait for paused status, as we need to wait for several parallel request. So, let's gently wait instead of just asserting that job already paused. Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Message-Id: <20210116214705.822267-12-vsementsov@virtuozzo.com> Signed-off-by: Max Reitz <mreitz@redhat.com> --- tests/qemu-iotests/056 | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/tests/qemu-iotests/056 b/tests/qemu-iotests/056 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/056 +++ b/tests/qemu-iotests/056 @@ -XXX,XX +XXX,XX @@ class BackupTest(iotests.QMPTestCase): event = self.vm.event_wait(name="BLOCK_JOB_ERROR", match={'data': {'device': 'drive0'}}) self.assertNotEqual(event, None) - # OK, job should be wedged - res = self.vm.qmp('query-block-jobs') + # OK, job should pause, but it can't do it immediately, as it can't + # cancel other parallel requests (which didn't fail) + with iotests.Timeout(60, "Timeout waiting for backup actually paused"): + while True: + res = self.vm.qmp('query-block-jobs') + if res['return'][0]['status'] == 'paused': + break self.assert_qmp(res, 'return[0]/status', 'paused') res = self.vm.qmp('block-job-dismiss', id='drive0') self.assert_qmp(res, 'error/desc', -- 2.29.2 Right now, this does not change anything, because backup ignores max-chunk and max-workers. However, as soon as backup is switched over to block-copy for the background copying process, we will need it to keep 129 passing. Signed-off-by: Max Reitz <mreitz@redhat.com> Message-Id: <20210120102043.28346-1-mreitz@redhat.com> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> --- tests/qemu-iotests/129 | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/tests/qemu-iotests/129 b/tests/qemu-iotests/129 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/129 +++ b/tests/qemu-iotests/129 @@ -XXX,XX +XXX,XX @@ class TestStopWithBlockJob(iotests.QMPTestCase): sync="full", buf_size=65536) def test_drive_backup(self): + # Limit max-chunk and max-workers so that block-copy will not + # launch so many workers working on so much data each that + # stop's bdrv_drain_all() would finish the job self.do_test_stop("drive-backup", device="drive0", target=self.target_img, format=iotests.imgfmt, - sync="full") + sync="full", + x_perf={ 'max-chunk': 65536, + 'max-workers': 8 }) def test_block_commit(self): # Add overlay above the source node so that we actually use a -- 2.29.2 From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> The further change of moving backup to be a one block-copy call will make copying chunk-size and cluster-size two separate things. So, even with 64k cluster sized qcow2 image, default chunk would be 1M. 185 test however assumes, that with speed limited to 64K, one iteration would result in offset=64K. It will change, as first iteration would result in offset=1M independently of speed. So, let's explicitly specify, what test wants: set max-chunk to 64K, so that one iteration is 64K. Note, that we don't need to limit max-workers, as block-copy rate limiter will handle the situation and wouldn't start new workers when speed limit is obviously reached. Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Message-Id: <20210116214705.822267-13-vsementsov@virtuozzo.com> Signed-off-by: Max Reitz <mreitz@redhat.com> --- tests/qemu-iotests/185 | 3 ++- tests/qemu-iotests/185.out | 3 ++- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/tests/qemu-iotests/185 b/tests/qemu-iotests/185 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/185 +++ b/tests/qemu-iotests/185 @@ -XXX,XX +XXX,XX @@ _send_qemu_cmd $h \ 'target': '$TEST_IMG.copy', 'format': '$IMGFMT', 'sync': 'full', - 'speed': 65536 } }" \ + 'speed': 65536, + 'x-perf': {'max-chunk': 65536} } }" \ "return" # If we don't sleep here 'quit' command races with disk I/O diff --git a/tests/qemu-iotests/185.out b/tests/qemu-iotests/185.out index XXXXXXX..XXXXXXX 100644 --- a/tests/qemu-iotests/185.out +++ b/tests/qemu-iotests/185.out @@ -XXX,XX +XXX,XX @@ Formatting 'TEST_DIR/t.qcow2.copy', fmt=qcow2 cluster_size=65536 extended_l2=off 'target': 'TEST_DIR/t.IMGFMT.copy', 'format': 'IMGFMT', 'sync': 'full', - 'speed': 65536 } } + 'speed': 65536, + 'x-perf': { 'max-chunk': 65536 } } } Formatting 'TEST_DIR/t.qcow2.copy', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=67108864 lazy_refcounts=off refcount_bits=16 {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "created", "id": "disk"}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "running", "id": "disk"}} -- 2.29.2 From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> The further change of moving backup to be a one block-copy call will make copying chunk-size and cluster-size two separate things. So, even with 64k cluster sized qcow2 image, default chunk would be 1M. Test 219 depends on specified chunk-size. Update it for explicit chunk-size for backup as for mirror. Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Message-Id: <20210116214705.822267-14-vsementsov@virtuozzo.com> Signed-off-by: Max Reitz <mreitz@redhat.com> --- tests/qemu-iotests/219 | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/tests/qemu-iotests/219 b/tests/qemu-iotests/219 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/219 +++ b/tests/qemu-iotests/219 @@ -XXX,XX +XXX,XX @@ with iotests.FilePath('disk.img') as disk_path, \ # but related to this also automatic state transitions like job # completion), but still get pause points often enough to avoid making this # test very slow, it's important to have the right ratio between speed and - # buf_size. + # copy-chunk-size. # - # For backup, buf_size is hard-coded to the source image cluster size (64k), - # so we'll pick the same for mirror. The slice time, i.e. the granularity - # of the rate limiting is 100ms. With a speed of 256k per second, we can - # get four pause points per second. This gives us 250ms per iteration, - # which should be enough to stay deterministic. + # Chose 64k copy-chunk-size both for mirror (by buf_size) and backup (by + # x-max-chunk). The slice time, i.e. the granularity of the rate limiting + # is 100ms. With a speed of 256k per second, we can get four pause points + # per second. This gives us 250ms per iteration, which should be enough to + # stay deterministic. test_job_lifecycle(vm, 'drive-mirror', has_ready=True, job_args={ 'device': 'drive0-node', @@ -XXX,XX +XXX,XX @@ with iotests.FilePath('disk.img') as disk_path, \ 'target': copy_path, 'sync': 'full', 'speed': 262144, + 'x-perf': {'max-chunk': 65536}, 'auto-finalize': auto_finalize, 'auto-dismiss': auto_dismiss, }) -- 2.29.2 From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Iotest 257 dumps a lot of in-progress information of backup job, such as offset and bitmap dirtiness. Further commit will move backup to be one block-copy call, which will introduce async parallel requests instead of plain cluster-by-cluster copying. To keep things deterministic, allow only one worker (only one copy request at a time) for this test. Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Message-Id: <20210116214705.822267-15-vsementsov@virtuozzo.com> Signed-off-by: Max Reitz <mreitz@redhat.com> --- tests/qemu-iotests/257 | 1 + tests/qemu-iotests/257.out | 306 ++++++++++++++++++------------------- 2 files changed, 154 insertions(+), 153 deletions(-) diff --git a/tests/qemu-iotests/257 b/tests/qemu-iotests/257 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/257 +++ b/tests/qemu-iotests/257 @@ -XXX,XX +XXX,XX @@ def blockdev_backup(vm, device, target, sync, **kwargs): target=target, sync=sync, filter_node_name='backup-top', + x_perf={'max-workers': 1}, **kwargs) return result diff --git a/tests/qemu-iotests/257.out b/tests/qemu-iotests/257.out index XXXXXXX..XXXXXXX 100644 --- a/tests/qemu-iotests/257.out +++ b/tests/qemu-iotests/257.out @@ -XXX,XX +XXX,XX @@ write -P0x76 0x3ff0000 0x10000 {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_0", "sync": "full", "target": "ref_target_0"}} +{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_0", "sync": "full", "target": "ref_target_0", "x-perf": {"max-workers": 1}}} {"return": {}} {"data": {"device": "ref_backup_0", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} @@ -XXX,XX +XXX,XX @@ expecting 6 dirty sectors; have 6. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_1", "sync": "full", "target": "ref_target_1"}} +{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_1", "sync": "full", "target": "ref_target_1", "x-perf": {"max-workers": 1}}} {"return": {}} {"data": {"device": "ref_backup_1", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} @@ -XXX,XX +XXX,XX @@ expecting 6 dirty sectors; have 6. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "never", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_1", "sync": "bitmap", "target": "backup_target_1"}} +{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "never", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_1", "sync": "bitmap", "target": "backup_target_1", "x-perf": {"max-workers": 1}}} {"return": {}} --- Write #2 --- @@ -XXX,XX +XXX,XX @@ expecting 15 dirty sectors; have 15. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_2", "sync": "full", "target": "ref_target_2"}} +{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_2", "sync": "full", "target": "ref_target_2", "x-perf": {"max-workers": 1}}} {"return": {}} {"data": {"device": "ref_backup_2", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} @@ -XXX,XX +XXX,XX @@ expecting 15 dirty sectors; have 15. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "never", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_2", "sync": "bitmap", "target": "backup_target_2"}} +{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "never", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_2", "sync": "bitmap", "target": "backup_target_2", "x-perf": {"max-workers": 1}}} {"return": {}} {"execute": "job-finalize", "arguments": {"id": "backup_2"}} {"return": {}} @@ -XXX,XX +XXX,XX @@ write -P0x76 0x3ff0000 0x10000 {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_0", "sync": "full", "target": "ref_target_0"}} +{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_0", "sync": "full", "target": "ref_target_0", "x-perf": {"max-workers": 1}}} {"return": {}} {"data": {"device": "ref_backup_0", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} @@ -XXX,XX +XXX,XX @@ expecting 6 dirty sectors; have 6. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_1", "sync": "full", "target": "ref_target_1"}} +{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_1", "sync": "full", "target": "ref_target_1", "x-perf": {"max-workers": 1}}} {"return": {}} {"data": {"device": "ref_backup_1", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} @@ -XXX,XX +XXX,XX @@ expecting 6 dirty sectors; have 6. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "never", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_1", "sync": "bitmap", "target": "backup_target_1"}} +{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "never", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_1", "sync": "bitmap", "target": "backup_target_1", "x-perf": {"max-workers": 1}}} {"return": {}} {"data": {"action": "report", "device": "backup_1", "operation": "read"}, "event": "BLOCK_JOB_ERROR", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} {"data": {"device": "backup_1", "error": "Input/output error", "len": 393216, "offset": 65536, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} @@ -XXX,XX +XXX,XX @@ expecting 14 dirty sectors; have 14. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_2", "sync": "full", "target": "ref_target_2"}} +{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_2", "sync": "full", "target": "ref_target_2", "x-perf": {"max-workers": 1}}} {"return": {}} {"data": {"device": "ref_backup_2", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} @@ -XXX,XX +XXX,XX @@ expecting 14 dirty sectors; have 14. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "never", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_2", "sync": "bitmap", "target": "backup_target_2"}} +{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "never", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_2", "sync": "bitmap", "target": "backup_target_2", "x-perf": {"max-workers": 1}}} {"return": {}} {"execute": "job-finalize", "arguments": {"id": "backup_2"}} {"return": {}} @@ -XXX,XX +XXX,XX @@ write -P0x76 0x3ff0000 0x10000 {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_0", "sync": "full", "target": "ref_target_0"}} +{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_0", "sync": "full", "target": "ref_target_0", "x-perf": {"max-workers": 1}}} {"return": {}} {"data": {"device": "ref_backup_0", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} @@ -XXX,XX +XXX,XX @@ expecting 6 dirty sectors; have 6. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_1", "sync": "full", "target": "ref_target_1"}} +{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_1", "sync": "full", "target": "ref_target_1", "x-perf": {"max-workers": 1}}} {"return": {}} {"data": {"device": "ref_backup_1", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} @@ -XXX,XX +XXX,XX @@ expecting 6 dirty sectors; have 6. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "never", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_1", "sync": "bitmap", "target": "backup_target_1"}} +{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "never", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_1", "sync": "bitmap", "target": "backup_target_1", "x-perf": {"max-workers": 1}}} {"return": {}} --- Write #2 --- @@ -XXX,XX +XXX,XX @@ expecting 15 dirty sectors; have 15. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_2", "sync": "full", "target": "ref_target_2"}} +{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_2", "sync": "full", "target": "ref_target_2", "x-perf": {"max-workers": 1}}} {"return": {}} {"data": {"device": "ref_backup_2", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} @@ -XXX,XX +XXX,XX @@ expecting 15 dirty sectors; have 15. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "never", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_2", "sync": "bitmap", "target": "backup_target_2"}} +{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "never", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_2", "sync": "bitmap", "target": "backup_target_2", "x-perf": {"max-workers": 1}}} {"return": {}} {"execute": "job-finalize", "arguments": {"id": "backup_2"}} {"return": {}} @@ -XXX,XX +XXX,XX @@ write -P0x76 0x3ff0000 0x10000 {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_0", "sync": "full", "target": "ref_target_0"}} +{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_0", "sync": "full", "target": "ref_target_0", "x-perf": {"max-workers": 1}}} {"return": {}} {"data": {"device": "ref_backup_0", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} @@ -XXX,XX +XXX,XX @@ expecting 6 dirty sectors; have 6. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_1", "sync": "full", "target": "ref_target_1"}} +{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_1", "sync": "full", "target": "ref_target_1", "x-perf": {"max-workers": 1}}} {"return": {}} {"data": {"device": "ref_backup_1", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} @@ -XXX,XX +XXX,XX @@ expecting 6 dirty sectors; have 6. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "on-success", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_1", "sync": "bitmap", "target": "backup_target_1"}} +{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "on-success", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_1", "sync": "bitmap", "target": "backup_target_1", "x-perf": {"max-workers": 1}}} {"return": {}} --- Write #2 --- @@ -XXX,XX +XXX,XX @@ expecting 15 dirty sectors; have 15. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_2", "sync": "full", "target": "ref_target_2"}} +{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_2", "sync": "full", "target": "ref_target_2", "x-perf": {"max-workers": 1}}} {"return": {}} {"data": {"device": "ref_backup_2", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} @@ -XXX,XX +XXX,XX @@ expecting 15 dirty sectors; have 15. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "on-success", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_2", "sync": "bitmap", "target": "backup_target_2"}} +{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "on-success", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_2", "sync": "bitmap", "target": "backup_target_2", "x-perf": {"max-workers": 1}}} {"return": {}} {"execute": "job-finalize", "arguments": {"id": "backup_2"}} {"return": {}} @@ -XXX,XX +XXX,XX @@ write -P0x76 0x3ff0000 0x10000 {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_0", "sync": "full", "target": "ref_target_0"}} +{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_0", "sync": "full", "target": "ref_target_0", "x-perf": {"max-workers": 1}}} {"return": {}} {"data": {"device": "ref_backup_0", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} @@ -XXX,XX +XXX,XX @@ expecting 6 dirty sectors; have 6. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_1", "sync": "full", "target": "ref_target_1"}} +{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_1", "sync": "full", "target": "ref_target_1", "x-perf": {"max-workers": 1}}} {"return": {}} {"data": {"device": "ref_backup_1", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} @@ -XXX,XX +XXX,XX @@ expecting 6 dirty sectors; have 6. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "on-success", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_1", "sync": "bitmap", "target": "backup_target_1"}} +{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "on-success", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_1", "sync": "bitmap", "target": "backup_target_1", "x-perf": {"max-workers": 1}}} {"return": {}} {"data": {"action": "report", "device": "backup_1", "operation": "read"}, "event": "BLOCK_JOB_ERROR", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} {"data": {"device": "backup_1", "error": "Input/output error", "len": 393216, "offset": 65536, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} @@ -XXX,XX +XXX,XX @@ expecting 14 dirty sectors; have 14. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_2", "sync": "full", "target": "ref_target_2"}} +{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_2", "sync": "full", "target": "ref_target_2", "x-perf": {"max-workers": 1}}} {"return": {}} {"data": {"device": "ref_backup_2", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} @@ -XXX,XX +XXX,XX @@ expecting 14 dirty sectors; have 14. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "on-success", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_2", "sync": "bitmap", "target": "backup_target_2"}} +{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "on-success", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_2", "sync": "bitmap", "target": "backup_target_2", "x-perf": {"max-workers": 1}}} {"return": {}} {"execute": "job-finalize", "arguments": {"id": "backup_2"}} {"return": {}} @@ -XXX,XX +XXX,XX @@ write -P0x76 0x3ff0000 0x10000 {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_0", "sync": "full", "target": "ref_target_0"}} +{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_0", "sync": "full", "target": "ref_target_0", "x-perf": {"max-workers": 1}}} {"return": {}} {"data": {"device": "ref_backup_0", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} @@ -XXX,XX +XXX,XX @@ expecting 6 dirty sectors; have 6. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_1", "sync": "full", "target": "ref_target_1"}} +{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_1", "sync": "full", "target": "ref_target_1", "x-perf": {"max-workers": 1}}} {"return": {}} {"data": {"device": "ref_backup_1", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} @@ -XXX,XX +XXX,XX @@ expecting 6 dirty sectors; have 6. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "on-success", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_1", "sync": "bitmap", "target": "backup_target_1"}} +{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "on-success", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_1", "sync": "bitmap", "target": "backup_target_1", "x-perf": {"max-workers": 1}}} {"return": {}} --- Write #2 --- @@ -XXX,XX +XXX,XX @@ expecting 12 dirty sectors; have 12. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_2", "sync": "full", "target": "ref_target_2"}} +{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_2", "sync": "full", "target": "ref_target_2", "x-perf": {"max-workers": 1}}} {"return": {}} {"data": {"device": "ref_backup_2", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} @@ -XXX,XX +XXX,XX @@ expecting 12 dirty sectors; have 12. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "on-success", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_2", "sync": "bitmap", "target": "backup_target_2"}} +{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "on-success", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_2", "sync": "bitmap", "target": "backup_target_2", "x-perf": {"max-workers": 1}}} {"return": {}} {"execute": "job-finalize", "arguments": {"id": "backup_2"}} {"return": {}} @@ -XXX,XX +XXX,XX @@ write -P0x76 0x3ff0000 0x10000 {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_0", "sync": "full", "target": "ref_target_0"}} +{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_0", "sync": "full", "target": "ref_target_0", "x-perf": {"max-workers": 1}}} {"return": {}} {"data": {"device": "ref_backup_0", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} @@ -XXX,XX +XXX,XX @@ expecting 6 dirty sectors; have 6. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_1", "sync": "full", "target": "ref_target_1"}} +{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_1", "sync": "full", "target": "ref_target_1", "x-perf": {"max-workers": 1}}} {"return": {}} {"data": {"device": "ref_backup_1", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} @@ -XXX,XX +XXX,XX @@ expecting 6 dirty sectors; have 6. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "always", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_1", "sync": "bitmap", "target": "backup_target_1"}} +{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "always", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_1", "sync": "bitmap", "target": "backup_target_1", "x-perf": {"max-workers": 1}}} {"return": {}} --- Write #2 --- @@ -XXX,XX +XXX,XX @@ expecting 12 dirty sectors; have 12. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_2", "sync": "full", "target": "ref_target_2"}} +{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_2", "sync": "full", "target": "ref_target_2", "x-perf": {"max-workers": 1}}} {"return": {}} {"data": {"device": "ref_backup_2", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} @@ -XXX,XX +XXX,XX @@ expecting 12 dirty sectors; have 12. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "always", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_2", "sync": "bitmap", "target": "backup_target_2"}} +{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "always", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_2", "sync": "bitmap", "target": "backup_target_2", "x-perf": {"max-workers": 1}}} {"return": {}} {"execute": "job-finalize", "arguments": {"id": "backup_2"}} {"return": {}} @@ -XXX,XX +XXX,XX @@ write -P0x76 0x3ff0000 0x10000 {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_0", "sync": "full", "target": "ref_target_0"}} +{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_0", "sync": "full", "target": "ref_target_0", "x-perf": {"max-workers": 1}}} {"return": {}} {"data": {"device": "ref_backup_0", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} @@ -XXX,XX +XXX,XX @@ expecting 6 dirty sectors; have 6. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_1", "sync": "full", "target": "ref_target_1"}} +{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_1", "sync": "full", "target": "ref_target_1", "x-perf": {"max-workers": 1}}} {"return": {}} {"data": {"device": "ref_backup_1", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} @@ -XXX,XX +XXX,XX @@ expecting 6 dirty sectors; have 6. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "always", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_1", "sync": "bitmap", "target": "backup_target_1"}} +{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "always", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_1", "sync": "bitmap", "target": "backup_target_1", "x-perf": {"max-workers": 1}}} {"return": {}} {"data": {"action": "report", "device": "backup_1", "operation": "read"}, "event": "BLOCK_JOB_ERROR", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} {"data": {"device": "backup_1", "error": "Input/output error", "len": 393216, "offset": 65536, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} @@ -XXX,XX +XXX,XX @@ expecting 13 dirty sectors; have 13. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_2", "sync": "full", "target": "ref_target_2"}} +{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_2", "sync": "full", "target": "ref_target_2", "x-perf": {"max-workers": 1}}} {"return": {}} {"data": {"device": "ref_backup_2", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} @@ -XXX,XX +XXX,XX @@ expecting 13 dirty sectors; have 13. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "always", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_2", "sync": "bitmap", "target": "backup_target_2"}} +{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "always", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_2", "sync": "bitmap", "target": "backup_target_2", "x-perf": {"max-workers": 1}}} {"return": {}} {"execute": "job-finalize", "arguments": {"id": "backup_2"}} {"return": {}} @@ -XXX,XX +XXX,XX @@ write -P0x76 0x3ff0000 0x10000 {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_0", "sync": "full", "target": "ref_target_0"}} +{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_0", "sync": "full", "target": "ref_target_0", "x-perf": {"max-workers": 1}}} {"return": {}} {"data": {"device": "ref_backup_0", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} @@ -XXX,XX +XXX,XX @@ expecting 6 dirty sectors; have 6. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_1", "sync": "full", "target": "ref_target_1"}} +{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_1", "sync": "full", "target": "ref_target_1", "x-perf": {"max-workers": 1}}} {"return": {}} {"data": {"device": "ref_backup_1", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} @@ -XXX,XX +XXX,XX @@ expecting 6 dirty sectors; have 6. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "always", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_1", "sync": "bitmap", "target": "backup_target_1"}} +{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "always", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_1", "sync": "bitmap", "target": "backup_target_1", "x-perf": {"max-workers": 1}}} {"return": {}} --- Write #2 --- @@ -XXX,XX +XXX,XX @@ expecting 12 dirty sectors; have 12. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_2", "sync": "full", "target": "ref_target_2"}} +{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_2", "sync": "full", "target": "ref_target_2", "x-perf": {"max-workers": 1}}} {"return": {}} {"data": {"device": "ref_backup_2", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} @@ -XXX,XX +XXX,XX @@ expecting 12 dirty sectors; have 12. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "always", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_2", "sync": "bitmap", "target": "backup_target_2"}} +{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "always", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_2", "sync": "bitmap", "target": "backup_target_2", "x-perf": {"max-workers": 1}}} {"return": {}} {"execute": "job-finalize", "arguments": {"id": "backup_2"}} {"return": {}} @@ -XXX,XX +XXX,XX @@ write -P0x76 0x3ff0000 0x10000 {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_0", "sync": "full", "target": "ref_target_0"}} +{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_0", "sync": "full", "target": "ref_target_0", "x-perf": {"max-workers": 1}}} {"return": {}} {"data": {"device": "ref_backup_0", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} @@ -XXX,XX +XXX,XX @@ expecting 6 dirty sectors; have 6. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_1", "sync": "full", "target": "ref_target_1"}} +{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_1", "sync": "full", "target": "ref_target_1", "x-perf": {"max-workers": 1}}} {"return": {}} {"data": {"device": "ref_backup_1", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} @@ -XXX,XX +XXX,XX @@ expecting 6 dirty sectors; have 6. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "on-success", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_1", "sync": "full", "target": "backup_target_1"}} +{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "on-success", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_1", "sync": "full", "target": "backup_target_1", "x-perf": {"max-workers": 1}}} {"return": {}} --- Write #2 --- @@ -XXX,XX +XXX,XX @@ expecting 15 dirty sectors; have 15. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_2", "sync": "full", "target": "ref_target_2"}} +{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_2", "sync": "full", "target": "ref_target_2", "x-perf": {"max-workers": 1}}} {"return": {}} {"data": {"device": "ref_backup_2", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} @@ -XXX,XX +XXX,XX @@ expecting 15 dirty sectors; have 15. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "on-success", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_2", "sync": "bitmap", "target": "backup_target_2"}} +{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "on-success", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_2", "sync": "bitmap", "target": "backup_target_2", "x-perf": {"max-workers": 1}}} {"return": {}} {"execute": "job-finalize", "arguments": {"id": "backup_2"}} {"return": {}} @@ -XXX,XX +XXX,XX @@ write -P0x76 0x3ff0000 0x10000 {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_0", "sync": "full", "target": "ref_target_0"}} +{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_0", "sync": "full", "target": "ref_target_0", "x-perf": {"max-workers": 1}}} {"return": {}} {"data": {"device": "ref_backup_0", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} @@ -XXX,XX +XXX,XX @@ expecting 6 dirty sectors; have 6. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_1", "sync": "full", "target": "ref_target_1"}} +{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_1", "sync": "full", "target": "ref_target_1", "x-perf": {"max-workers": 1}}} {"return": {}} {"data": {"device": "ref_backup_1", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} @@ -XXX,XX +XXX,XX @@ expecting 6 dirty sectors; have 6. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "on-success", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_1", "sync": "full", "target": "backup_target_1"}} +{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "on-success", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_1", "sync": "full", "target": "backup_target_1", "x-perf": {"max-workers": 1}}} {"return": {}} {"data": {"action": "report", "device": "backup_1", "operation": "read"}, "event": "BLOCK_JOB_ERROR", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} {"data": {"device": "backup_1", "error": "Input/output error", "len": 67108864, "offset": 983040, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} @@ -XXX,XX +XXX,XX @@ expecting 14 dirty sectors; have 14. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_2", "sync": "full", "target": "ref_target_2"}} +{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_2", "sync": "full", "target": "ref_target_2", "x-perf": {"max-workers": 1}}} {"return": {}} {"data": {"device": "ref_backup_2", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} @@ -XXX,XX +XXX,XX @@ expecting 14 dirty sectors; have 14. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "on-success", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_2", "sync": "bitmap", "target": "backup_target_2"}} +{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "on-success", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_2", "sync": "bitmap", "target": "backup_target_2", "x-perf": {"max-workers": 1}}} {"return": {}} {"execute": "job-finalize", "arguments": {"id": "backup_2"}} {"return": {}} @@ -XXX,XX +XXX,XX @@ write -P0x76 0x3ff0000 0x10000 {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_0", "sync": "full", "target": "ref_target_0"}} +{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_0", "sync": "full", "target": "ref_target_0", "x-perf": {"max-workers": 1}}} {"return": {}} {"data": {"device": "ref_backup_0", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} @@ -XXX,XX +XXX,XX @@ expecting 6 dirty sectors; have 6. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_1", "sync": "full", "target": "ref_target_1"}} +{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_1", "sync": "full", "target": "ref_target_1", "x-perf": {"max-workers": 1}}} {"return": {}} {"data": {"device": "ref_backup_1", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} @@ -XXX,XX +XXX,XX @@ expecting 6 dirty sectors; have 6. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "on-success", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_1", "sync": "full", "target": "backup_target_1"}} +{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "on-success", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_1", "sync": "full", "target": "backup_target_1", "x-perf": {"max-workers": 1}}} {"return": {}} --- Write #2 --- @@ -XXX,XX +XXX,XX @@ expecting 12 dirty sectors; have 12. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_2", "sync": "full", "target": "ref_target_2"}} +{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_2", "sync": "full", "target": "ref_target_2", "x-perf": {"max-workers": 1}}} {"return": {}} {"data": {"device": "ref_backup_2", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} @@ -XXX,XX +XXX,XX @@ expecting 12 dirty sectors; have 12. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "on-success", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_2", "sync": "bitmap", "target": "backup_target_2"}} +{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "on-success", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_2", "sync": "bitmap", "target": "backup_target_2", "x-perf": {"max-workers": 1}}} {"return": {}} {"execute": "job-finalize", "arguments": {"id": "backup_2"}} {"return": {}} @@ -XXX,XX +XXX,XX @@ write -P0x76 0x3ff0000 0x10000 {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_0", "sync": "full", "target": "ref_target_0"}} +{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_0", "sync": "full", "target": "ref_target_0", "x-perf": {"max-workers": 1}}} {"return": {}} {"data": {"device": "ref_backup_0", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} @@ -XXX,XX +XXX,XX @@ expecting 6 dirty sectors; have 6. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_1", "sync": "full", "target": "ref_target_1"}} +{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_1", "sync": "full", "target": "ref_target_1", "x-perf": {"max-workers": 1}}} {"return": {}} {"data": {"device": "ref_backup_1", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} @@ -XXX,XX +XXX,XX @@ expecting 6 dirty sectors; have 6. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "always", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_1", "sync": "full", "target": "backup_target_1"}} +{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "always", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_1", "sync": "full", "target": "backup_target_1", "x-perf": {"max-workers": 1}}} {"return": {}} --- Write #2 --- @@ -XXX,XX +XXX,XX @@ expecting 12 dirty sectors; have 12. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_2", "sync": "full", "target": "ref_target_2"}} +{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_2", "sync": "full", "target": "ref_target_2", "x-perf": {"max-workers": 1}}} {"return": {}} {"data": {"device": "ref_backup_2", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} @@ -XXX,XX +XXX,XX @@ expecting 12 dirty sectors; have 12. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "always", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_2", "sync": "bitmap", "target": "backup_target_2"}} +{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "always", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_2", "sync": "bitmap", "target": "backup_target_2", "x-perf": {"max-workers": 1}}} {"return": {}} {"execute": "job-finalize", "arguments": {"id": "backup_2"}} {"return": {}} @@ -XXX,XX +XXX,XX @@ write -P0x76 0x3ff0000 0x10000 {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_0", "sync": "full", "target": "ref_target_0"}} +{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_0", "sync": "full", "target": "ref_target_0", "x-perf": {"max-workers": 1}}} {"return": {}} {"data": {"device": "ref_backup_0", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} @@ -XXX,XX +XXX,XX @@ expecting 6 dirty sectors; have 6. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_1", "sync": "full", "target": "ref_target_1"}} +{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_1", "sync": "full", "target": "ref_target_1", "x-perf": {"max-workers": 1}}} {"return": {}} {"data": {"device": "ref_backup_1", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} @@ -XXX,XX +XXX,XX @@ expecting 6 dirty sectors; have 6. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "always", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_1", "sync": "full", "target": "backup_target_1"}} +{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "always", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_1", "sync": "full", "target": "backup_target_1", "x-perf": {"max-workers": 1}}} {"return": {}} {"data": {"action": "report", "device": "backup_1", "operation": "read"}, "event": "BLOCK_JOB_ERROR", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} {"data": {"device": "backup_1", "error": "Input/output error", "len": 67108864, "offset": 983040, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} @@ -XXX,XX +XXX,XX @@ expecting 1014 dirty sectors; have 1014. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_2", "sync": "full", "target": "ref_target_2"}} +{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_2", "sync": "full", "target": "ref_target_2", "x-perf": {"max-workers": 1}}} {"return": {}} {"data": {"device": "ref_backup_2", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} @@ -XXX,XX +XXX,XX @@ expecting 1014 dirty sectors; have 1014. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "always", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_2", "sync": "bitmap", "target": "backup_target_2"}} +{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "always", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_2", "sync": "bitmap", "target": "backup_target_2", "x-perf": {"max-workers": 1}}} {"return": {}} {"execute": "job-finalize", "arguments": {"id": "backup_2"}} {"return": {}} @@ -XXX,XX +XXX,XX @@ write -P0x76 0x3ff0000 0x10000 {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_0", "sync": "full", "target": "ref_target_0"}} +{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_0", "sync": "full", "target": "ref_target_0", "x-perf": {"max-workers": 1}}} {"return": {}} {"data": {"device": "ref_backup_0", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} @@ -XXX,XX +XXX,XX @@ expecting 6 dirty sectors; have 6. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_1", "sync": "full", "target": "ref_target_1"}} +{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_1", "sync": "full", "target": "ref_target_1", "x-perf": {"max-workers": 1}}} {"return": {}} {"data": {"device": "ref_backup_1", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} @@ -XXX,XX +XXX,XX @@ expecting 6 dirty sectors; have 6. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "always", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_1", "sync": "full", "target": "backup_target_1"}} +{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "always", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_1", "sync": "full", "target": "backup_target_1", "x-perf": {"max-workers": 1}}} {"return": {}} --- Write #2 --- @@ -XXX,XX +XXX,XX @@ expecting 12 dirty sectors; have 12. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_2", "sync": "full", "target": "ref_target_2"}} +{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_2", "sync": "full", "target": "ref_target_2", "x-perf": {"max-workers": 1}}} {"return": {}} {"data": {"device": "ref_backup_2", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} @@ -XXX,XX +XXX,XX @@ expecting 12 dirty sectors; have 12. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "always", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_2", "sync": "bitmap", "target": "backup_target_2"}} +{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "always", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_2", "sync": "bitmap", "target": "backup_target_2", "x-perf": {"max-workers": 1}}} {"return": {}} {"execute": "job-finalize", "arguments": {"id": "backup_2"}} {"return": {}} @@ -XXX,XX +XXX,XX @@ write -P0x76 0x3ff0000 0x10000 {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_0", "sync": "full", "target": "ref_target_0"}} +{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_0", "sync": "full", "target": "ref_target_0", "x-perf": {"max-workers": 1}}} {"return": {}} {"data": {"device": "ref_backup_0", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} @@ -XXX,XX +XXX,XX @@ expecting 6 dirty sectors; have 6. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_1", "sync": "full", "target": "ref_target_1"}} +{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_1", "sync": "full", "target": "ref_target_1", "x-perf": {"max-workers": 1}}} {"return": {}} {"data": {"device": "ref_backup_1", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} @@ -XXX,XX +XXX,XX @@ expecting 6 dirty sectors; have 6. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "on-success", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_1", "sync": "top", "target": "backup_target_1"}} +{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "on-success", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_1", "sync": "top", "target": "backup_target_1", "x-perf": {"max-workers": 1}}} {"return": {}} --- Write #2 --- @@ -XXX,XX +XXX,XX @@ expecting 15 dirty sectors; have 15. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_2", "sync": "full", "target": "ref_target_2"}} +{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_2", "sync": "full", "target": "ref_target_2", "x-perf": {"max-workers": 1}}} {"return": {}} {"data": {"device": "ref_backup_2", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} @@ -XXX,XX +XXX,XX @@ expecting 15 dirty sectors; have 15. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "on-success", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_2", "sync": "bitmap", "target": "backup_target_2"}} +{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "on-success", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_2", "sync": "bitmap", "target": "backup_target_2", "x-perf": {"max-workers": 1}}} {"return": {}} {"execute": "job-finalize", "arguments": {"id": "backup_2"}} {"return": {}} @@ -XXX,XX +XXX,XX @@ write -P0x76 0x3ff0000 0x10000 {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_0", "sync": "full", "target": "ref_target_0"}} +{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_0", "sync": "full", "target": "ref_target_0", "x-perf": {"max-workers": 1}}} {"return": {}} {"data": {"device": "ref_backup_0", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} @@ -XXX,XX +XXX,XX @@ expecting 6 dirty sectors; have 6. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_1", "sync": "full", "target": "ref_target_1"}} +{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_1", "sync": "full", "target": "ref_target_1", "x-perf": {"max-workers": 1}}} {"return": {}} {"data": {"device": "ref_backup_1", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} @@ -XXX,XX +XXX,XX @@ expecting 6 dirty sectors; have 6. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "on-success", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_1", "sync": "top", "target": "backup_target_1"}} +{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "on-success", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_1", "sync": "top", "target": "backup_target_1", "x-perf": {"max-workers": 1}}} {"return": {}} {"data": {"action": "report", "device": "backup_1", "operation": "read"}, "event": "BLOCK_JOB_ERROR", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} {"data": {"device": "backup_1", "error": "Input/output error", "len": 458752, "offset": 65536, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} @@ -XXX,XX +XXX,XX @@ expecting 14 dirty sectors; have 14. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_2", "sync": "full", "target": "ref_target_2"}} +{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_2", "sync": "full", "target": "ref_target_2", "x-perf": {"max-workers": 1}}} {"return": {}} {"data": {"device": "ref_backup_2", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} @@ -XXX,XX +XXX,XX @@ expecting 14 dirty sectors; have 14. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "on-success", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_2", "sync": "bitmap", "target": "backup_target_2"}} +{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "on-success", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_2", "sync": "bitmap", "target": "backup_target_2", "x-perf": {"max-workers": 1}}} {"return": {}} {"execute": "job-finalize", "arguments": {"id": "backup_2"}} {"return": {}} @@ -XXX,XX +XXX,XX @@ write -P0x76 0x3ff0000 0x10000 {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_0", "sync": "full", "target": "ref_target_0"}} +{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_0", "sync": "full", "target": "ref_target_0", "x-perf": {"max-workers": 1}}} {"return": {}} {"data": {"device": "ref_backup_0", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} @@ -XXX,XX +XXX,XX @@ expecting 6 dirty sectors; have 6. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_1", "sync": "full", "target": "ref_target_1"}} +{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_1", "sync": "full", "target": "ref_target_1", "x-perf": {"max-workers": 1}}} {"return": {}} {"data": {"device": "ref_backup_1", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} @@ -XXX,XX +XXX,XX @@ expecting 6 dirty sectors; have 6. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "on-success", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_1", "sync": "top", "target": "backup_target_1"}} +{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "on-success", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_1", "sync": "top", "target": "backup_target_1", "x-perf": {"max-workers": 1}}} {"return": {}} --- Write #2 --- @@ -XXX,XX +XXX,XX @@ expecting 12 dirty sectors; have 12. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_2", "sync": "full", "target": "ref_target_2"}} +{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_2", "sync": "full", "target": "ref_target_2", "x-perf": {"max-workers": 1}}} {"return": {}} {"data": {"device": "ref_backup_2", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} @@ -XXX,XX +XXX,XX @@ expecting 12 dirty sectors; have 12. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "on-success", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_2", "sync": "bitmap", "target": "backup_target_2"}} +{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "on-success", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_2", "sync": "bitmap", "target": "backup_target_2", "x-perf": {"max-workers": 1}}} {"return": {}} {"execute": "job-finalize", "arguments": {"id": "backup_2"}} {"return": {}} @@ -XXX,XX +XXX,XX @@ write -P0x76 0x3ff0000 0x10000 {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_0", "sync": "full", "target": "ref_target_0"}} +{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_0", "sync": "full", "target": "ref_target_0", "x-perf": {"max-workers": 1}}} {"return": {}} {"data": {"device": "ref_backup_0", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} @@ -XXX,XX +XXX,XX @@ expecting 6 dirty sectors; have 6. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_1", "sync": "full", "target": "ref_target_1"}} +{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_1", "sync": "full", "target": "ref_target_1", "x-perf": {"max-workers": 1}}} {"return": {}} {"data": {"device": "ref_backup_1", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} @@ -XXX,XX +XXX,XX @@ expecting 6 dirty sectors; have 6. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "always", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_1", "sync": "top", "target": "backup_target_1"}} +{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "always", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_1", "sync": "top", "target": "backup_target_1", "x-perf": {"max-workers": 1}}} {"return": {}} --- Write #2 --- @@ -XXX,XX +XXX,XX @@ expecting 12 dirty sectors; have 12. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_2", "sync": "full", "target": "ref_target_2"}} +{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_2", "sync": "full", "target": "ref_target_2", "x-perf": {"max-workers": 1}}} {"return": {}} {"data": {"device": "ref_backup_2", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} @@ -XXX,XX +XXX,XX @@ expecting 12 dirty sectors; have 12. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "always", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_2", "sync": "bitmap", "target": "backup_target_2"}} +{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "always", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_2", "sync": "bitmap", "target": "backup_target_2", "x-perf": {"max-workers": 1}}} {"return": {}} {"execute": "job-finalize", "arguments": {"id": "backup_2"}} {"return": {}} @@ -XXX,XX +XXX,XX @@ write -P0x76 0x3ff0000 0x10000 {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_0", "sync": "full", "target": "ref_target_0"}} +{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_0", "sync": "full", "target": "ref_target_0", "x-perf": {"max-workers": 1}}} {"return": {}} {"data": {"device": "ref_backup_0", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} @@ -XXX,XX +XXX,XX @@ expecting 6 dirty sectors; have 6. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_1", "sync": "full", "target": "ref_target_1"}} +{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_1", "sync": "full", "target": "ref_target_1", "x-perf": {"max-workers": 1}}} {"return": {}} {"data": {"device": "ref_backup_1", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} @@ -XXX,XX +XXX,XX @@ expecting 6 dirty sectors; have 6. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "always", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_1", "sync": "top", "target": "backup_target_1"}} +{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "always", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_1", "sync": "top", "target": "backup_target_1", "x-perf": {"max-workers": 1}}} {"return": {}} {"data": {"action": "report", "device": "backup_1", "operation": "read"}, "event": "BLOCK_JOB_ERROR", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} {"data": {"device": "backup_1", "error": "Input/output error", "len": 458752, "offset": 65536, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} @@ -XXX,XX +XXX,XX @@ expecting 14 dirty sectors; have 14. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_2", "sync": "full", "target": "ref_target_2"}} +{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_2", "sync": "full", "target": "ref_target_2", "x-perf": {"max-workers": 1}}} {"return": {}} {"data": {"device": "ref_backup_2", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} @@ -XXX,XX +XXX,XX @@ expecting 14 dirty sectors; have 14. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "always", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_2", "sync": "bitmap", "target": "backup_target_2"}} +{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "always", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_2", "sync": "bitmap", "target": "backup_target_2", "x-perf": {"max-workers": 1}}} {"return": {}} {"execute": "job-finalize", "arguments": {"id": "backup_2"}} {"return": {}} @@ -XXX,XX +XXX,XX @@ write -P0x76 0x3ff0000 0x10000 {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_0", "sync": "full", "target": "ref_target_0"}} +{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_0", "sync": "full", "target": "ref_target_0", "x-perf": {"max-workers": 1}}} {"return": {}} {"data": {"device": "ref_backup_0", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} @@ -XXX,XX +XXX,XX @@ expecting 6 dirty sectors; have 6. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_1", "sync": "full", "target": "ref_target_1"}} +{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_1", "sync": "full", "target": "ref_target_1", "x-perf": {"max-workers": 1}}} {"return": {}} {"data": {"device": "ref_backup_1", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} @@ -XXX,XX +XXX,XX @@ expecting 6 dirty sectors; have 6. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "always", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_1", "sync": "top", "target": "backup_target_1"}} +{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "always", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_1", "sync": "top", "target": "backup_target_1", "x-perf": {"max-workers": 1}}} {"return": {}} --- Write #2 --- @@ -XXX,XX +XXX,XX @@ expecting 12 dirty sectors; have 12. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_2", "sync": "full", "target": "ref_target_2"}} +{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "ref_backup_2", "sync": "full", "target": "ref_target_2", "x-perf": {"max-workers": 1}}} {"return": {}} {"data": {"device": "ref_backup_2", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} @@ -XXX,XX +XXX,XX @@ expecting 12 dirty sectors; have 12. OK! {"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}} {"return": {}} {} -{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "always", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_2", "sync": "bitmap", "target": "backup_target_2"}} +{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "always", "device": "drive0", "filter-node-name": "backup-top", "job-id": "backup_2", "sync": "bitmap", "target": "backup_target_2", "x-perf": {"max-workers": 1}}} {"return": {}} {"execute": "job-finalize", "arguments": {"id": "backup_2"}} {"return": {}} @@ -XXX,XX +XXX,XX @@ qemu_img compare "TEST_DIR/PID-img" "TEST_DIR/PID-fbackup2" ==> Identical, OK! -- Sync mode incremental tests -- -{"execute": "blockdev-backup", "arguments": {"bitmap-mode": "on-success", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "incremental", "target": "backup_target"}} +{"execute": "blockdev-backup", "arguments": {"bitmap-mode": "on-success", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "incremental", "target": "backup_target", "x-perf": {"max-workers": 1}}} {"error": {"class": "GenericError", "desc": "must provide a valid bitmap name for 'incremental' sync mode"}} -{"execute": "blockdev-backup", "arguments": {"bitmap-mode": "always", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "incremental", "target": "backup_target"}} +{"execute": "blockdev-backup", "arguments": {"bitmap-mode": "always", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "incremental", "target": "backup_target", "x-perf": {"max-workers": 1}}} {"error": {"class": "GenericError", "desc": "must provide a valid bitmap name for 'incremental' sync mode"}} -{"execute": "blockdev-backup", "arguments": {"bitmap-mode": "never", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "incremental", "target": "backup_target"}} +{"execute": "blockdev-backup", "arguments": {"bitmap-mode": "never", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "incremental", "target": "backup_target", "x-perf": {"max-workers": 1}}} {"error": {"class": "GenericError", "desc": "must provide a valid bitmap name for 'incremental' sync mode"}} -{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "incremental", "target": "backup_target"}} +{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "incremental", "target": "backup_target", "x-perf": {"max-workers": 1}}} {"error": {"class": "GenericError", "desc": "must provide a valid bitmap name for 'incremental' sync mode"}} -{"execute": "blockdev-backup", "arguments": {"bitmap": "bitmap404", "bitmap-mode": "on-success", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "incremental", "target": "backup_target"}} +{"execute": "blockdev-backup", "arguments": {"bitmap": "bitmap404", "bitmap-mode": "on-success", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "incremental", "target": "backup_target", "x-perf": {"max-workers": 1}}} {"error": {"class": "GenericError", "desc": "Bitmap 'bitmap404' could not be found"}} -{"execute": "blockdev-backup", "arguments": {"bitmap": "bitmap404", "bitmap-mode": "always", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "incremental", "target": "backup_target"}} +{"execute": "blockdev-backup", "arguments": {"bitmap": "bitmap404", "bitmap-mode": "always", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "incremental", "target": "backup_target", "x-perf": {"max-workers": 1}}} {"error": {"class": "GenericError", "desc": "Bitmap sync mode must be 'on-success' when using sync mode 'incremental'"}} -{"execute": "blockdev-backup", "arguments": {"bitmap": "bitmap404", "bitmap-mode": "never", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "incremental", "target": "backup_target"}} +{"execute": "blockdev-backup", "arguments": {"bitmap": "bitmap404", "bitmap-mode": "never", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "incremental", "target": "backup_target", "x-perf": {"max-workers": 1}}} {"error": {"class": "GenericError", "desc": "Bitmap sync mode must be 'on-success' when using sync mode 'incremental'"}} -{"execute": "blockdev-backup", "arguments": {"bitmap": "bitmap404", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "incremental", "target": "backup_target"}} +{"execute": "blockdev-backup", "arguments": {"bitmap": "bitmap404", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "incremental", "target": "backup_target", "x-perf": {"max-workers": 1}}} {"error": {"class": "GenericError", "desc": "Bitmap 'bitmap404' could not be found"}} -{"execute": "blockdev-backup", "arguments": {"bitmap": "bitmap0", "bitmap-mode": "always", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "incremental", "target": "backup_target"}} +{"execute": "blockdev-backup", "arguments": {"bitmap": "bitmap0", "bitmap-mode": "always", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "incremental", "target": "backup_target", "x-perf": {"max-workers": 1}}} {"error": {"class": "GenericError", "desc": "Bitmap sync mode must be 'on-success' when using sync mode 'incremental'"}} -{"execute": "blockdev-backup", "arguments": {"bitmap": "bitmap0", "bitmap-mode": "never", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "incremental", "target": "backup_target"}} +{"execute": "blockdev-backup", "arguments": {"bitmap": "bitmap0", "bitmap-mode": "never", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "incremental", "target": "backup_target", "x-perf": {"max-workers": 1}}} {"error": {"class": "GenericError", "desc": "Bitmap sync mode must be 'on-success' when using sync mode 'incremental'"}} -- Sync mode bitmap tests -- -{"execute": "blockdev-backup", "arguments": {"bitmap-mode": "on-success", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "bitmap", "target": "backup_target"}} +{"execute": "blockdev-backup", "arguments": {"bitmap-mode": "on-success", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "bitmap", "target": "backup_target", "x-perf": {"max-workers": 1}}} {"error": {"class": "GenericError", "desc": "must provide a valid bitmap name for 'bitmap' sync mode"}} -{"execute": "blockdev-backup", "arguments": {"bitmap-mode": "always", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "bitmap", "target": "backup_target"}} +{"execute": "blockdev-backup", "arguments": {"bitmap-mode": "always", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "bitmap", "target": "backup_target", "x-perf": {"max-workers": 1}}} {"error": {"class": "GenericError", "desc": "must provide a valid bitmap name for 'bitmap' sync mode"}} -{"execute": "blockdev-backup", "arguments": {"bitmap-mode": "never", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "bitmap", "target": "backup_target"}} +{"execute": "blockdev-backup", "arguments": {"bitmap-mode": "never", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "bitmap", "target": "backup_target", "x-perf": {"max-workers": 1}}} {"error": {"class": "GenericError", "desc": "must provide a valid bitmap name for 'bitmap' sync mode"}} -{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "bitmap", "target": "backup_target"}} +{"execute": "blockdev-backup", "arguments": {"device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "bitmap", "target": "backup_target", "x-perf": {"max-workers": 1}}} {"error": {"class": "GenericError", "desc": "must provide a valid bitmap name for 'bitmap' sync mode"}} -{"execute": "blockdev-backup", "arguments": {"bitmap": "bitmap404", "bitmap-mode": "on-success", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "bitmap", "target": "backup_target"}} +{"execute": "blockdev-backup", "arguments": {"bitmap": "bitmap404", "bitmap-mode": "on-success", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "bitmap", "target": "backup_target", "x-perf": {"max-workers": 1}}} {"error": {"class": "GenericError", "desc": "Bitmap 'bitmap404' could not be found"}} -{"execute": "blockdev-backup", "arguments": {"bitmap": "bitmap404", "bitmap-mode": "always", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "bitmap", "target": "backup_target"}} +{"execute": "blockdev-backup", "arguments": {"bitmap": "bitmap404", "bitmap-mode": "always", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "bitmap", "target": "backup_target", "x-perf": {"max-workers": 1}}} {"error": {"class": "GenericError", "desc": "Bitmap 'bitmap404' could not be found"}} -{"execute": "blockdev-backup", "arguments": {"bitmap": "bitmap404", "bitmap-mode": "never", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "bitmap", "target": "backup_target"}} +{"execute": "blockdev-backup", "arguments": {"bitmap": "bitmap404", "bitmap-mode": "never", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "bitmap", "target": "backup_target", "x-perf": {"max-workers": 1}}} {"error": {"class": "GenericError", "desc": "Bitmap 'bitmap404' could not be found"}} -{"execute": "blockdev-backup", "arguments": {"bitmap": "bitmap404", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "bitmap", "target": "backup_target"}} +{"execute": "blockdev-backup", "arguments": {"bitmap": "bitmap404", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "bitmap", "target": "backup_target", "x-perf": {"max-workers": 1}}} {"error": {"class": "GenericError", "desc": "Bitmap 'bitmap404' could not be found"}} -{"execute": "blockdev-backup", "arguments": {"bitmap": "bitmap0", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "bitmap", "target": "backup_target"}} +{"execute": "blockdev-backup", "arguments": {"bitmap": "bitmap0", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "bitmap", "target": "backup_target", "x-perf": {"max-workers": 1}}} {"error": {"class": "GenericError", "desc": "Bitmap sync mode must be given when providing a bitmap"}} -- Sync mode full tests -- -{"execute": "blockdev-backup", "arguments": {"bitmap-mode": "on-success", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "full", "target": "backup_target"}} +{"execute": "blockdev-backup", "arguments": {"bitmap-mode": "on-success", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "full", "target": "backup_target", "x-perf": {"max-workers": 1}}} {"error": {"class": "GenericError", "desc": "Cannot specify bitmap sync mode without a bitmap"}} -{"execute": "blockdev-backup", "arguments": {"bitmap-mode": "always", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "full", "target": "backup_target"}} +{"execute": "blockdev-backup", "arguments": {"bitmap-mode": "always", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "full", "target": "backup_target", "x-perf": {"max-workers": 1}}} {"error": {"class": "GenericError", "desc": "Cannot specify bitmap sync mode without a bitmap"}} -{"execute": "blockdev-backup", "arguments": {"bitmap-mode": "never", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "full", "target": "backup_target"}} +{"execute": "blockdev-backup", "arguments": {"bitmap-mode": "never", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "full", "target": "backup_target", "x-perf": {"max-workers": 1}}} {"error": {"class": "GenericError", "desc": "Cannot specify bitmap sync mode without a bitmap"}} -{"execute": "blockdev-backup", "arguments": {"bitmap": "bitmap404", "bitmap-mode": "on-success", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "full", "target": "backup_target"}} +{"execute": "blockdev-backup", "arguments": {"bitmap": "bitmap404", "bitmap-mode": "on-success", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "full", "target": "backup_target", "x-perf": {"max-workers": 1}}} {"error": {"class": "GenericError", "desc": "Bitmap 'bitmap404' could not be found"}} -{"execute": "blockdev-backup", "arguments": {"bitmap": "bitmap404", "bitmap-mode": "always", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "full", "target": "backup_target"}} +{"execute": "blockdev-backup", "arguments": {"bitmap": "bitmap404", "bitmap-mode": "always", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "full", "target": "backup_target", "x-perf": {"max-workers": 1}}} {"error": {"class": "GenericError", "desc": "Bitmap 'bitmap404' could not be found"}} -{"execute": "blockdev-backup", "arguments": {"bitmap": "bitmap404", "bitmap-mode": "never", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "full", "target": "backup_target"}} +{"execute": "blockdev-backup", "arguments": {"bitmap": "bitmap404", "bitmap-mode": "never", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "full", "target": "backup_target", "x-perf": {"max-workers": 1}}} {"error": {"class": "GenericError", "desc": "Bitmap 'bitmap404' could not be found"}} -{"execute": "blockdev-backup", "arguments": {"bitmap": "bitmap404", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "full", "target": "backup_target"}} +{"execute": "blockdev-backup", "arguments": {"bitmap": "bitmap404", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "full", "target": "backup_target", "x-perf": {"max-workers": 1}}} {"error": {"class": "GenericError", "desc": "Bitmap 'bitmap404' could not be found"}} -{"execute": "blockdev-backup", "arguments": {"bitmap": "bitmap0", "bitmap-mode": "never", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "full", "target": "backup_target"}} +{"execute": "blockdev-backup", "arguments": {"bitmap": "bitmap0", "bitmap-mode": "never", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "full", "target": "backup_target", "x-perf": {"max-workers": 1}}} {"error": {"class": "GenericError", "desc": "Bitmap sync mode 'never' has no meaningful effect when combined with sync mode 'full'"}} -{"execute": "blockdev-backup", "arguments": {"bitmap": "bitmap0", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "full", "target": "backup_target"}} +{"execute": "blockdev-backup", "arguments": {"bitmap": "bitmap0", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "full", "target": "backup_target", "x-perf": {"max-workers": 1}}} {"error": {"class": "GenericError", "desc": "Bitmap sync mode must be given when providing a bitmap"}} -- Sync mode top tests -- -{"execute": "blockdev-backup", "arguments": {"bitmap-mode": "on-success", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "top", "target": "backup_target"}} +{"execute": "blockdev-backup", "arguments": {"bitmap-mode": "on-success", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "top", "target": "backup_target", "x-perf": {"max-workers": 1}}} {"error": {"class": "GenericError", "desc": "Cannot specify bitmap sync mode without a bitmap"}} -{"execute": "blockdev-backup", "arguments": {"bitmap-mode": "always", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "top", "target": "backup_target"}} +{"execute": "blockdev-backup", "arguments": {"bitmap-mode": "always", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "top", "target": "backup_target", "x-perf": {"max-workers": 1}}} {"error": {"class": "GenericError", "desc": "Cannot specify bitmap sync mode without a bitmap"}} -{"execute": "blockdev-backup", "arguments": {"bitmap-mode": "never", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "top", "target": "backup_target"}} +{"execute": "blockdev-backup", "arguments": {"bitmap-mode": "never", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "top", "target": "backup_target", "x-perf": {"max-workers": 1}}} {"error": {"class": "GenericError", "desc": "Cannot specify bitmap sync mode without a bitmap"}} -{"execute": "blockdev-backup", "arguments": {"bitmap": "bitmap404", "bitmap-mode": "on-success", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "top", "target": "backup_target"}} +{"execute": "blockdev-backup", "arguments": {"bitmap": "bitmap404", "bitmap-mode": "on-success", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "top", "target": "backup_target", "x-perf": {"max-workers": 1}}} {"error": {"class": "GenericError", "desc": "Bitmap 'bitmap404' could not be found"}} -{"execute": "blockdev-backup", "arguments": {"bitmap": "bitmap404", "bitmap-mode": "always", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "top", "target": "backup_target"}} +{"execute": "blockdev-backup", "arguments": {"bitmap": "bitmap404", "bitmap-mode": "always", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "top", "target": "backup_target", "x-perf": {"max-workers": 1}}} {"error": {"class": "GenericError", "desc": "Bitmap 'bitmap404' could not be found"}} -{"execute": "blockdev-backup", "arguments": {"bitmap": "bitmap404", "bitmap-mode": "never", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "top", "target": "backup_target"}} +{"execute": "blockdev-backup", "arguments": {"bitmap": "bitmap404", "bitmap-mode": "never", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "top", "target": "backup_target", "x-perf": {"max-workers": 1}}} {"error": {"class": "GenericError", "desc": "Bitmap 'bitmap404' could not be found"}} -{"execute": "blockdev-backup", "arguments": {"bitmap": "bitmap404", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "top", "target": "backup_target"}} +{"execute": "blockdev-backup", "arguments": {"bitmap": "bitmap404", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "top", "target": "backup_target", "x-perf": {"max-workers": 1}}} {"error": {"class": "GenericError", "desc": "Bitmap 'bitmap404' could not be found"}} -{"execute": "blockdev-backup", "arguments": {"bitmap": "bitmap0", "bitmap-mode": "never", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "top", "target": "backup_target"}} +{"execute": "blockdev-backup", "arguments": {"bitmap": "bitmap0", "bitmap-mode": "never", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "top", "target": "backup_target", "x-perf": {"max-workers": 1}}} {"error": {"class": "GenericError", "desc": "Bitmap sync mode 'never' has no meaningful effect when combined with sync mode 'top'"}} -{"execute": "blockdev-backup", "arguments": {"bitmap": "bitmap0", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "top", "target": "backup_target"}} +{"execute": "blockdev-backup", "arguments": {"bitmap": "bitmap0", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "top", "target": "backup_target", "x-perf": {"max-workers": 1}}} {"error": {"class": "GenericError", "desc": "Bitmap sync mode must be given when providing a bitmap"}} -- Sync mode none tests -- -{"execute": "blockdev-backup", "arguments": {"bitmap-mode": "on-success", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "none", "target": "backup_target"}} +{"execute": "blockdev-backup", "arguments": {"bitmap-mode": "on-success", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "none", "target": "backup_target", "x-perf": {"max-workers": 1}}} {"error": {"class": "GenericError", "desc": "Cannot specify bitmap sync mode without a bitmap"}} -{"execute": "blockdev-backup", "arguments": {"bitmap-mode": "always", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "none", "target": "backup_target"}} +{"execute": "blockdev-backup", "arguments": {"bitmap-mode": "always", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "none", "target": "backup_target", "x-perf": {"max-workers": 1}}} {"error": {"class": "GenericError", "desc": "Cannot specify bitmap sync mode without a bitmap"}} -{"execute": "blockdev-backup", "arguments": {"bitmap-mode": "never", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "none", "target": "backup_target"}} +{"execute": "blockdev-backup", "arguments": {"bitmap-mode": "never", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "none", "target": "backup_target", "x-perf": {"max-workers": 1}}} {"error": {"class": "GenericError", "desc": "Cannot specify bitmap sync mode without a bitmap"}} -{"execute": "blockdev-backup", "arguments": {"bitmap": "bitmap404", "bitmap-mode": "on-success", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "none", "target": "backup_target"}} +{"execute": "blockdev-backup", "arguments": {"bitmap": "bitmap404", "bitmap-mode": "on-success", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "none", "target": "backup_target", "x-perf": {"max-workers": 1}}} {"error": {"class": "GenericError", "desc": "Bitmap 'bitmap404' could not be found"}} -{"execute": "blockdev-backup", "arguments": {"bitmap": "bitmap404", "bitmap-mode": "always", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "none", "target": "backup_target"}} +{"execute": "blockdev-backup", "arguments": {"bitmap": "bitmap404", "bitmap-mode": "always", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "none", "target": "backup_target", "x-perf": {"max-workers": 1}}} {"error": {"class": "GenericError", "desc": "Bitmap 'bitmap404' could not be found"}} -{"execute": "blockdev-backup", "arguments": {"bitmap": "bitmap404", "bitmap-mode": "never", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "none", "target": "backup_target"}} +{"execute": "blockdev-backup", "arguments": {"bitmap": "bitmap404", "bitmap-mode": "never", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "none", "target": "backup_target", "x-perf": {"max-workers": 1}}} {"error": {"class": "GenericError", "desc": "Bitmap 'bitmap404' could not be found"}} -{"execute": "blockdev-backup", "arguments": {"bitmap": "bitmap404", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "none", "target": "backup_target"}} +{"execute": "blockdev-backup", "arguments": {"bitmap": "bitmap404", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "none", "target": "backup_target", "x-perf": {"max-workers": 1}}} {"error": {"class": "GenericError", "desc": "Bitmap 'bitmap404' could not be found"}} -{"execute": "blockdev-backup", "arguments": {"bitmap": "bitmap0", "bitmap-mode": "on-success", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "none", "target": "backup_target"}} +{"execute": "blockdev-backup", "arguments": {"bitmap": "bitmap0", "bitmap-mode": "on-success", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "none", "target": "backup_target", "x-perf": {"max-workers": 1}}} {"error": {"class": "GenericError", "desc": "sync mode 'none' does not produce meaningful bitmap outputs"}} -{"execute": "blockdev-backup", "arguments": {"bitmap": "bitmap0", "bitmap-mode": "always", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "none", "target": "backup_target"}} +{"execute": "blockdev-backup", "arguments": {"bitmap": "bitmap0", "bitmap-mode": "always", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "none", "target": "backup_target", "x-perf": {"max-workers": 1}}} {"error": {"class": "GenericError", "desc": "sync mode 'none' does not produce meaningful bitmap outputs"}} -{"execute": "blockdev-backup", "arguments": {"bitmap": "bitmap0", "bitmap-mode": "never", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "none", "target": "backup_target"}} +{"execute": "blockdev-backup", "arguments": {"bitmap": "bitmap0", "bitmap-mode": "never", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "none", "target": "backup_target", "x-perf": {"max-workers": 1}}} {"error": {"class": "GenericError", "desc": "sync mode 'none' does not produce meaningful bitmap outputs"}} -{"execute": "blockdev-backup", "arguments": {"bitmap": "bitmap0", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "none", "target": "backup_target"}} +{"execute": "blockdev-backup", "arguments": {"bitmap": "bitmap0", "device": "drive0", "filter-node-name": "backup-top", "job-id": "api_job", "sync": "none", "target": "backup_target", "x-perf": {"max-workers": 1}}} {"error": {"class": "GenericError", "desc": "Bitmap sync mode must be given when providing a bitmap"}} -- 2.29.2 From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> We are going to stop use of this callback in the following commit. Still the callback handling code will be dropped in a separate commit. So, for now let's make it optional. Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Message-Id: <20210116214705.822267-16-vsementsov@virtuozzo.com> Signed-off-by: Max Reitz <mreitz@redhat.com> --- block/block-copy.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/block/block-copy.c b/block/block-copy.c index XXXXXXX..XXXXXXX 100644 --- a/block/block-copy.c +++ b/block/block-copy.c @@ -XXX,XX +XXX,XX @@ static coroutine_fn int block_copy_task_entry(AioTask *task) t->call_state->error_is_read = error_is_read; } else { progress_work_done(t->s->progress, t->bytes); - t->s->progress_bytes_callback(t->bytes, t->s->progress_opaque); + if (t->s->progress_bytes_callback) { + t->s->progress_bytes_callback(t->bytes, t->s->progress_opaque); + } } co_put_to_shres(t->s->mem, t->bytes); block_copy_task_end(t, ret); -- 2.29.2 From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Message-Id: <20210116214705.822267-17-vsementsov@virtuozzo.com> Signed-off-by: Max Reitz <mreitz@redhat.com> --- block/backup.c | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-) diff --git a/block/backup.c b/block/backup.c index XXXXXXX..XXXXXXX 100644 --- a/block/backup.c +++ b/block/backup.c @@ -XXX,XX +XXX,XX @@ static void backup_init_bcs_bitmap(BackupBlockJob *job) static int coroutine_fn backup_run(Job *job, Error **errp) { BackupBlockJob *s = container_of(job, BackupBlockJob, common.job); - int ret = 0; + int ret; backup_init_bcs_bitmap(s); @@ -XXX,XX +XXX,XX @@ static int coroutine_fn backup_run(Job *job, Error **errp) for (offset = 0; offset < s->len; ) { if (yield_and_check(s)) { - ret = -ECANCELED; - goto out; + return -ECANCELED; } ret = block_copy_reset_unallocated(s->bcs, offset, &count); if (ret < 0) { - goto out; + return ret; } offset += count; @@ -XXX,XX +XXX,XX @@ static int coroutine_fn backup_run(Job *job, Error **errp) job_yield(job); } } else { - ret = backup_loop(s); + return backup_loop(s); } - out: - return ret; + return 0; } static const BlockJobDriver backup_job_driver = { -- 2.29.2 From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> This brings async request handling and block-status driven chunk sizes to backup out of the box, which improves backup performance. Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Message-Id: <20210116214705.822267-18-vsementsov@virtuozzo.com> Signed-off-by: Max Reitz <mreitz@redhat.com> --- block/backup.c | 187 +++++++++++++++++++++++++++++++------------------ 1 file changed, 120 insertions(+), 67 deletions(-) diff --git a/block/backup.c b/block/backup.c index XXXXXXX..XXXXXXX 100644 --- a/block/backup.c +++ b/block/backup.c @@ -XXX,XX +XXX,XX @@ #include "block/block-copy.h" #include "qapi/error.h" #include "qapi/qmp/qerror.h" -#include "qemu/ratelimit.h" #include "qemu/cutils.h" #include "sysemu/block-backend.h" #include "qemu/bitmap.h" @@ -XXX,XX +XXX,XX @@ typedef struct BackupBlockJob { BlockdevOnError on_source_error; BlockdevOnError on_target_error; uint64_t len; - uint64_t bytes_read; int64_t cluster_size; BackupPerf perf; BlockCopyState *bcs; + + bool wait; + BlockCopyCallState *bg_bcs_call; } BackupBlockJob; static const BlockJobDriver backup_job_driver; -static void backup_progress_bytes_callback(int64_t bytes, void *opaque) -{ - BackupBlockJob *s = opaque; - - s->bytes_read += bytes; -} - -static int coroutine_fn backup_do_cow(BackupBlockJob *job, - int64_t offset, uint64_t bytes, - bool *error_is_read) -{ - int ret = 0; - int64_t start, end; /* bytes */ - - start = QEMU_ALIGN_DOWN(offset, job->cluster_size); - end = QEMU_ALIGN_UP(bytes + offset, job->cluster_size); - - trace_backup_do_cow_enter(job, start, offset, bytes); - - ret = block_copy(job->bcs, start, end - start, true, error_is_read); - - trace_backup_do_cow_return(job, offset, bytes, ret); - - return ret; -} - static void backup_cleanup_sync_bitmap(BackupBlockJob *job, int ret) { BdrvDirtyBitmap *bm; @@ -XXX,XX +XXX,XX @@ static BlockErrorAction backup_error_action(BackupBlockJob *job, } } -static bool coroutine_fn yield_and_check(BackupBlockJob *job) +static void coroutine_fn backup_block_copy_callback(void *opaque) { - uint64_t delay_ns; - - if (job_is_cancelled(&job->common.job)) { - return true; - } - - /* - * We need to yield even for delay_ns = 0 so that bdrv_drain_all() can - * return. Without a yield, the VM would not reboot. - */ - delay_ns = block_job_ratelimit_get_delay(&job->common, job->bytes_read); - job->bytes_read = 0; - job_sleep_ns(&job->common.job, delay_ns); + BackupBlockJob *s = opaque; - if (job_is_cancelled(&job->common.job)) { - return true; + if (s->wait) { + s->wait = false; + aio_co_wake(s->common.job.co); + } else { + job_enter(&s->common.job); } - - return false; } static int coroutine_fn backup_loop(BackupBlockJob *job) { - bool error_is_read; - int64_t offset; - BdrvDirtyBitmapIter *bdbi; + BlockCopyCallState *s = NULL; int ret = 0; + bool error_is_read; + BlockErrorAction act; + + while (true) { /* retry loop */ + job->bg_bcs_call = s = block_copy_async(job->bcs, 0, + QEMU_ALIGN_UP(job->len, job->cluster_size), + job->perf.max_workers, job->perf.max_chunk, + backup_block_copy_callback, job); + + while (!block_copy_call_finished(s) && + !job_is_cancelled(&job->common.job)) + { + job_yield(&job->common.job); + } - bdbi = bdrv_dirty_iter_new(block_copy_dirty_bitmap(job->bcs)); - while ((offset = bdrv_dirty_iter_next(bdbi)) != -1) { - do { - if (yield_and_check(job)) { - goto out; - } - ret = backup_do_cow(job, offset, job->cluster_size, &error_is_read); - if (ret < 0 && backup_error_action(job, error_is_read, -ret) == - BLOCK_ERROR_ACTION_REPORT) - { - goto out; - } - } while (ret < 0); + if (!block_copy_call_finished(s)) { + assert(job_is_cancelled(&job->common.job)); + /* + * Note that we can't use job_yield() here, as it doesn't work for + * cancelled job. + */ + block_copy_call_cancel(s); + job->wait = true; + qemu_coroutine_yield(); + assert(block_copy_call_finished(s)); + ret = 0; + goto out; + } + + if (job_is_cancelled(&job->common.job) || + block_copy_call_succeeded(s)) + { + ret = 0; + goto out; + } + + if (block_copy_call_cancelled(s)) { + /* + * Job is not cancelled but only block-copy call. This is possible + * after job pause. Now the pause is finished, start new block-copy + * iteration. + */ + block_copy_call_free(s); + continue; + } + + /* The only remaining case is failed block-copy call. */ + assert(block_copy_call_failed(s)); + + ret = block_copy_call_status(s, &error_is_read); + act = backup_error_action(job, error_is_read, -ret); + switch (act) { + case BLOCK_ERROR_ACTION_REPORT: + goto out; + case BLOCK_ERROR_ACTION_STOP: + /* + * Go to pause prior to starting new block-copy call on the next + * iteration. + */ + job_pause_point(&job->common.job); + break; + case BLOCK_ERROR_ACTION_IGNORE: + /* Proceed to new block-copy call to retry. */ + break; + default: + abort(); + } + + block_copy_call_free(s); } - out: - bdrv_dirty_iter_free(bdbi); +out: + block_copy_call_free(s); + job->bg_bcs_call = NULL; return ret; } @@ -XXX,XX +XXX,XX @@ static int coroutine_fn backup_run(Job *job, Error **errp) int64_t count; for (offset = 0; offset < s->len; ) { - if (yield_and_check(s)) { + if (job_is_cancelled(job)) { + return -ECANCELED; + } + + job_pause_point(job); + + if (job_is_cancelled(job)) { return -ECANCELED; } @@ -XXX,XX +XXX,XX @@ static int coroutine_fn backup_run(Job *job, Error **errp) return 0; } +static void coroutine_fn backup_pause(Job *job) +{ + BackupBlockJob *s = container_of(job, BackupBlockJob, common.job); + + if (s->bg_bcs_call && !block_copy_call_finished(s->bg_bcs_call)) { + block_copy_call_cancel(s->bg_bcs_call); + s->wait = true; + qemu_coroutine_yield(); + } +} + +static void coroutine_fn backup_set_speed(BlockJob *job, int64_t speed) +{ + BackupBlockJob *s = container_of(job, BackupBlockJob, common); + + /* + * block_job_set_speed() is called first from block_job_create(), when we + * don't yet have s->bcs. + */ + if (s->bcs) { + block_copy_set_speed(s->bcs, speed); + if (s->bg_bcs_call) { + block_copy_kick(s->bg_bcs_call); + } + } +} + static const BlockJobDriver backup_job_driver = { .job_driver = { .instance_size = sizeof(BackupBlockJob), @@ -XXX,XX +XXX,XX @@ static const BlockJobDriver backup_job_driver = { .commit = backup_commit, .abort = backup_abort, .clean = backup_clean, - } + .pause = backup_pause, + }, + .set_speed = backup_set_speed, }; static int64_t backup_calculate_cluster_size(BlockDriverState *target, @@ -XXX,XX +XXX,XX @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs, job->len = len; job->perf = *perf; - block_copy_set_progress_callback(bcs, backup_progress_bytes_callback, job); block_copy_set_progress_meter(bcs, &job->common.job.progress); + block_copy_set_speed(bcs, speed); /* Required permissions are already taken by backup-top target */ block_job_add_bdrv(&job->common, "target", target, 0, BLK_PERM_ALL, -- 2.29.2 From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Further commit will add a benchmark (scripts/simplebench/bench-backup.py), which will show that backup works better with async parallel requests (previous commit) and disabled copy_range. So, let's disable copy_range by default. Note: the option was added several commits ago with default to true, to follow old behavior (the feature was enabled unconditionally), and only now we are going to change the default behavior. Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Message-Id: <20210116214705.822267-19-vsementsov@virtuozzo.com> Signed-off-by: Max Reitz <mreitz@redhat.com> --- qapi/block-core.json | 2 +- blockdev.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/qapi/block-core.json b/qapi/block-core.json index XXXXXXX..XXXXXXX 100644 --- a/qapi/block-core.json +++ b/qapi/block-core.json @@ -XXX,XX +XXX,XX @@ # Optional parameters for backup. These parameters don't affect # functionality, but may significantly affect performance. # -# @use-copy-range: Use copy offloading. Default true. +# @use-copy-range: Use copy offloading. Default false. # # @max-workers: Maximum number of parallel requests for the sustained background # copying process. Doesn't influence copy-before-write operations. diff --git a/blockdev.c b/blockdev.c index XXXXXXX..XXXXXXX 100644 --- a/blockdev.c +++ b/blockdev.c @@ -XXX,XX +XXX,XX @@ static BlockJob *do_backup_common(BackupCommon *backup, { BlockJob *job = NULL; BdrvDirtyBitmap *bmap = NULL; - BackupPerf perf = { .use_copy_range = true, .max_workers = 64 }; + BackupPerf perf = { .max_workers = 64 }; int job_flags = JOB_DEFAULT; if (!backup->has_speed) { -- 2.29.2 From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Drop unused code. Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Message-Id: <20210116214705.822267-20-vsementsov@virtuozzo.com> Signed-off-by: Max Reitz <mreitz@redhat.com> --- include/block/block-copy.h | 6 ------ block/block-copy.c | 15 --------------- 2 files changed, 21 deletions(-) diff --git a/include/block/block-copy.h b/include/block/block-copy.h index XXXXXXX..XXXXXXX 100644 --- a/include/block/block-copy.h +++ b/include/block/block-copy.h @@ -XXX,XX +XXX,XX @@ #include "block/block.h" #include "qemu/co-shared-resource.h" -typedef void (*ProgressBytesCallbackFunc)(int64_t bytes, void *opaque); typedef void (*BlockCopyAsyncCallbackFunc)(void *opaque); typedef struct BlockCopyState BlockCopyState; typedef struct BlockCopyCallState BlockCopyCallState; @@ -XXX,XX +XXX,XX @@ BlockCopyState *block_copy_state_new(BdrvChild *source, BdrvChild *target, BdrvRequestFlags write_flags, Error **errp); -void block_copy_set_progress_callback( - BlockCopyState *s, - ProgressBytesCallbackFunc progress_bytes_callback, - void *progress_opaque); - void block_copy_set_progress_meter(BlockCopyState *s, ProgressMeter *pm); void block_copy_state_free(BlockCopyState *s); diff --git a/block/block-copy.c b/block/block-copy.c index XXXXXXX..XXXXXXX 100644 --- a/block/block-copy.c +++ b/block/block-copy.c @@ -XXX,XX +XXX,XX @@ typedef struct BlockCopyState { bool skip_unallocated; ProgressMeter *progress; - /* progress_bytes_callback: called when some copying progress is done. */ - ProgressBytesCallbackFunc progress_bytes_callback; - void *progress_opaque; SharedResource *mem; @@ -XXX,XX +XXX,XX @@ BlockCopyState *block_copy_state_new(BdrvChild *source, BdrvChild *target, return s; } -void block_copy_set_progress_callback( - BlockCopyState *s, - ProgressBytesCallbackFunc progress_bytes_callback, - void *progress_opaque) -{ - s->progress_bytes_callback = progress_bytes_callback; - s->progress_opaque = progress_opaque; -} - void block_copy_set_progress_meter(BlockCopyState *s, ProgressMeter *pm) { s->progress = pm; @@ -XXX,XX +XXX,XX @@ static coroutine_fn int block_copy_task_entry(AioTask *task) t->call_state->error_is_read = error_is_read; } else { progress_work_done(t->s->progress, t->bytes); - if (t->s->progress_bytes_callback) { - t->s->progress_bytes_callback(t->bytes, t->s->progress_opaque); - } } co_put_to_shres(t->s->mem, t->bytes); block_copy_task_end(t, ret); -- 2.29.2 From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Message-Id: <20210116214705.822267-21-vsementsov@virtuozzo.com> Signed-off-by: Max Reitz <mreitz@redhat.com> --- include/block/block-copy.h | 2 +- block/backup-top.c | 2 +- block/block-copy.c | 10 ++-------- 3 files changed, 4 insertions(+), 10 deletions(-) diff --git a/include/block/block-copy.h b/include/block/block-copy.h index XXXXXXX..XXXXXXX 100644 --- a/include/block/block-copy.h +++ b/include/block/block-copy.h @@ -XXX,XX +XXX,XX @@ int64_t block_copy_reset_unallocated(BlockCopyState *s, int64_t offset, int64_t *count); int coroutine_fn block_copy(BlockCopyState *s, int64_t offset, int64_t bytes, - bool ignore_ratelimit, bool *error_is_read); + bool ignore_ratelimit); /* * Run block-copy in a coroutine, create corresponding BlockCopyCallState diff --git a/block/backup-top.c b/block/backup-top.c index XXXXXXX..XXXXXXX 100644 --- a/block/backup-top.c +++ b/block/backup-top.c @@ -XXX,XX +XXX,XX @@ static coroutine_fn int backup_top_cbw(BlockDriverState *bs, uint64_t offset, off = QEMU_ALIGN_DOWN(offset, s->cluster_size); end = QEMU_ALIGN_UP(offset + bytes, s->cluster_size); - return block_copy(s->bcs, off, end - off, true, NULL); + return block_copy(s->bcs, off, end - off, true); } static int coroutine_fn backup_top_co_pdiscard(BlockDriverState *bs, diff --git a/block/block-copy.c b/block/block-copy.c index XXXXXXX..XXXXXXX 100644 --- a/block/block-copy.c +++ b/block/block-copy.c @@ -XXX,XX +XXX,XX @@ static int coroutine_fn block_copy_common(BlockCopyCallState *call_state) } int coroutine_fn block_copy(BlockCopyState *s, int64_t start, int64_t bytes, - bool ignore_ratelimit, bool *error_is_read) + bool ignore_ratelimit) { BlockCopyCallState call_state = { .s = s, @@ -XXX,XX +XXX,XX @@ int coroutine_fn block_copy(BlockCopyState *s, int64_t start, int64_t bytes, .max_workers = BLOCK_COPY_MAX_WORKERS, }; - int ret = block_copy_common(&call_state); - - if (error_is_read && ret < 0) { - *error_is_read = call_state.error_is_read; - } - - return ret; + return block_copy_common(&call_state); } static void coroutine_fn block_copy_async_co_entry(void *opaque) -- 2.29.2 From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Message-Id: <20210116214705.822267-22-vsementsov@virtuozzo.com> Signed-off-by: Max Reitz <mreitz@redhat.com> --- scripts/simplebench/bench_block_job.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/simplebench/bench_block_job.py b/scripts/simplebench/bench_block_job.py index XXXXXXX..XXXXXXX 100755 --- a/scripts/simplebench/bench_block_job.py +++ b/scripts/simplebench/bench_block_job.py @@ -XXX,XX +XXX,XX @@ -#!/usr/bin/env python +#!/usr/bin/env python3 # # Benchmark block jobs # -- 2.29.2 From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Add argument to allow additional block-job options. Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Message-Id: <20210116214705.822267-23-vsementsov@virtuozzo.com> Signed-off-by: Max Reitz <mreitz@redhat.com> --- scripts/simplebench/bench-example.py | 2 +- scripts/simplebench/bench_block_job.py | 11 +++++++---- 2 files changed, 8 insertions(+), 5 deletions(-) diff --git a/scripts/simplebench/bench-example.py b/scripts/simplebench/bench-example.py index XXXXXXX..XXXXXXX 100644 --- a/scripts/simplebench/bench-example.py +++ b/scripts/simplebench/bench-example.py @@ -XXX,XX +XXX,XX @@ from bench_block_job import bench_block_copy, drv_file, drv_nbd def bench_func(env, case): """ Handle one "cell" of benchmarking table. """ - return bench_block_copy(env['qemu_binary'], env['cmd'], + return bench_block_copy(env['qemu_binary'], env['cmd'], {} case['source'], case['target']) diff --git a/scripts/simplebench/bench_block_job.py b/scripts/simplebench/bench_block_job.py index XXXXXXX..XXXXXXX 100755 --- a/scripts/simplebench/bench_block_job.py +++ b/scripts/simplebench/bench_block_job.py @@ -XXX,XX +XXX,XX @@ def bench_block_job(cmd, cmd_args, qemu_args): # Bench backup or mirror -def bench_block_copy(qemu_binary, cmd, source, target): +def bench_block_copy(qemu_binary, cmd, cmd_options, source, target): """Helper to run bench_block_job() for mirror or backup""" assert cmd in ('blockdev-backup', 'blockdev-mirror') source['node-name'] = 'source' target['node-name'] = 'target' - return bench_block_job(cmd, - {'job-id': 'job0', 'device': 'source', - 'target': 'target', 'sync': 'full'}, + cmd_options['job-id'] = 'job0' + cmd_options['device'] = 'source' + cmd_options['target'] = 'target' + cmd_options['sync'] = 'full' + + return bench_block_job(cmd, cmd_options, [qemu_binary, '-blockdev', json.dumps(source), '-blockdev', json.dumps(target)]) -- 2.29.2 From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Add script to benchmark new backup architecture. Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Message-Id: <20210116214705.822267-24-vsementsov@virtuozzo.com> [mreitz: s/not unsupported/not supported/] Signed-off-by: Max Reitz <mreitz@redhat.com> --- scripts/simplebench/bench-backup.py | 167 ++++++++++++++++++++++++++++ 1 file changed, 167 insertions(+) create mode 100755 scripts/simplebench/bench-backup.py diff --git a/scripts/simplebench/bench-backup.py b/scripts/simplebench/bench-backup.py new file mode 100755 index XXXXXXX..XXXXXXX --- /dev/null +++ b/scripts/simplebench/bench-backup.py @@ -XXX,XX +XXX,XX @@ +#!/usr/bin/env python3 +# +# Bench backup block-job +# +# Copyright (c) 2020 Virtuozzo International GmbH. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# + +import argparse +import json + +import simplebench +from results_to_text import results_to_text +from bench_block_job import bench_block_copy, drv_file, drv_nbd + + +def bench_func(env, case): + """ Handle one "cell" of benchmarking table. """ + cmd_options = env['cmd-options'] if 'cmd-options' in env else {} + return bench_block_copy(env['qemu-binary'], env['cmd'], + cmd_options, + case['source'], case['target']) + + +def bench(args): + test_cases = [] + + sources = {} + targets = {} + for d in args.dir: + label, path = d.split(':') # paths with colon not supported + sources[label] = drv_file(path + '/test-source') + targets[label] = drv_file(path + '/test-target') + + if args.nbd: + nbd = args.nbd.split(':') + host = nbd[0] + port = '10809' if len(nbd) == 1 else nbd[1] + drv = drv_nbd(host, port) + sources['nbd'] = drv + targets['nbd'] = drv + + for t in args.test: + src, dst = t.split(':') + + test_cases.append({ + 'id': t, + 'source': sources[src], + 'target': targets[dst] + }) + + binaries = [] # list of (<label>, <path>, [<options>]) + for i, q in enumerate(args.env): + name_path = q.split(':') + if len(name_path) == 1: + label = f'q{i}' + path_opts = name_path[0].split(',') + else: + assert len(name_path) == 2 # paths with colon not supported + label = name_path[0] + path_opts = name_path[1].split(',') + + binaries.append((label, path_opts[0], path_opts[1:])) + + test_envs = [] + + bin_paths = {} + for i, q in enumerate(args.env): + opts = q.split(',') + label_path = opts[0] + opts = opts[1:] + + if ':' in label_path: + # path with colon inside is not supported + label, path = label_path.split(':') + bin_paths[label] = path + elif label_path in bin_paths: + label = label_path + path = bin_paths[label] + else: + path = label_path + label = f'q{i}' + bin_paths[label] = path + + x_perf = {} + is_mirror = False + for opt in opts: + if opt == 'mirror': + is_mirror = True + elif opt == 'copy-range=on': + x_perf['use-copy-range'] = True + elif opt == 'copy-range=off': + x_perf['use-copy-range'] = False + elif opt.startswith('max-workers='): + x_perf['max-workers'] = int(opt.split('=')[1]) + + if is_mirror: + assert not x_perf + test_envs.append({ + 'id': f'mirror({label})', + 'cmd': 'blockdev-mirror', + 'qemu-binary': path + }) + else: + test_envs.append({ + 'id': f'backup({label})\n' + '\n'.join(opts), + 'cmd': 'blockdev-backup', + 'cmd-options': {'x-perf': x_perf} if x_perf else {}, + 'qemu-binary': path + }) + + result = simplebench.bench(bench_func, test_envs, test_cases, count=3) + with open('results.json', 'w') as f: + json.dump(result, f, indent=4) + print(results_to_text(result)) + + +class ExtendAction(argparse.Action): + def __call__(self, parser, namespace, values, option_string=None): + items = getattr(namespace, self.dest) or [] + items.extend(values) + setattr(namespace, self.dest, items) + + +if __name__ == '__main__': + p = argparse.ArgumentParser('Backup benchmark', epilog=''' +ENV format + + (LABEL:PATH|LABEL|PATH)[,max-workers=N][,use-copy-range=(on|off)][,mirror] + + LABEL short name for the binary + PATH path to the binary + max-workers set x-perf.max-workers of backup job + use-copy-range set x-perf.use-copy-range of backup job + mirror use mirror job instead of backup''', + formatter_class=argparse.RawTextHelpFormatter) + p.add_argument('--env', nargs='+', help='''\ +Qemu binaries with labels and options, see below +"ENV format" section''', + action=ExtendAction) + p.add_argument('--dir', nargs='+', help='''\ +Directories, each containing "test-source" and/or +"test-target" files, raw images to used in +benchmarking. File path with label, like +label:/path/to/directory''', + action=ExtendAction) + p.add_argument('--nbd', help='''\ +host:port for remote NBD image, (or just host, for +default port 10809). Use it in tests, label is "nbd" +(but you cannot create test nbd:nbd).''') + p.add_argument('--test', nargs='+', help='''\ +Tests, in form source-dir-label:target-dir-label''', + action=ExtendAction) + + bench(p.parse_args()) -- 2.29.2 From: David Edmondson <david.edmondson@oracle.com> When a call to fcntl(2) for the purpose of adding file locks fails with an error other than EAGAIN or EACCES, report the error returned by fcntl. EAGAIN or EACCES are elided as they are considered to be common failures, indicating that a conflicting lock is held by another process. No errors are elided when removing file locks. Signed-off-by: David Edmondson <david.edmondson@oracle.com> Message-Id: <20210113164447.2545785-1-david.edmondson@oracle.com> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Signed-off-by: Max Reitz <mreitz@redhat.com> --- block/file-posix.c | 38 ++++++++++++++++++++++++++++---------- 1 file changed, 28 insertions(+), 10 deletions(-) diff --git a/block/file-posix.c b/block/file-posix.c index XXXXXXX..XXXXXXX 100644 --- a/block/file-posix.c +++ b/block/file-posix.c @@ -XXX,XX +XXX,XX @@ typedef struct RawPosixAIOData { static int cdrom_reopen(BlockDriverState *bs); #endif +/* + * Elide EAGAIN and EACCES details when failing to lock, as this + * indicates that the specified file region is already locked by + * another process, which is considered a common scenario. + */ +#define raw_lock_error_setg_errno(errp, err, fmt, ...) \ + do { \ + if ((err) == EAGAIN || (err) == EACCES) { \ + error_setg((errp), (fmt), ## __VA_ARGS__); \ + } else { \ + error_setg_errno((errp), (err), (fmt), ## __VA_ARGS__); \ + } \ + } while (0) + #if defined(__NetBSD__) static int raw_normalize_devicepath(const char **filename, Error **errp) { @@ -XXX,XX +XXX,XX @@ static int raw_apply_lock_bytes(BDRVRawState *s, int fd, if ((perm_lock_bits & bit) && !(locked_perm & bit)) { ret = qemu_lock_fd(fd, off, 1, false); if (ret) { - error_setg(errp, "Failed to lock byte %d", off); + raw_lock_error_setg_errno(errp, -ret, "Failed to lock byte %d", + off); return ret; } else if (s) { s->locked_perm |= bit; @@ -XXX,XX +XXX,XX @@ static int raw_apply_lock_bytes(BDRVRawState *s, int fd, } else if (unlock && (locked_perm & bit) && !(perm_lock_bits & bit)) { ret = qemu_unlock_fd(fd, off, 1); if (ret) { - error_setg(errp, "Failed to unlock byte %d", off); + error_setg_errno(errp, -ret, "Failed to unlock byte %d", off); return ret; } else if (s) { s->locked_perm &= ~bit; @@ -XXX,XX +XXX,XX @@ static int raw_apply_lock_bytes(BDRVRawState *s, int fd, if ((shared_perm_lock_bits & bit) && !(locked_shared_perm & bit)) { ret = qemu_lock_fd(fd, off, 1, false); if (ret) { - error_setg(errp, "Failed to lock byte %d", off); + raw_lock_error_setg_errno(errp, -ret, "Failed to lock byte %d", + off); return ret; } else if (s) { s->locked_shared_perm |= bit; @@ -XXX,XX +XXX,XX @@ static int raw_apply_lock_bytes(BDRVRawState *s, int fd, !(shared_perm_lock_bits & bit)) { ret = qemu_unlock_fd(fd, off, 1); if (ret) { - error_setg(errp, "Failed to unlock byte %d", off); + error_setg_errno(errp, -ret, "Failed to unlock byte %d", off); return ret; } else if (s) { s->locked_shared_perm &= ~bit; @@ -XXX,XX +XXX,XX @@ static int raw_check_lock_bytes(int fd, uint64_t perm, uint64_t shared_perm, ret = qemu_lock_fd_test(fd, off, 1, true); if (ret) { char *perm_name = bdrv_perm_names(p); - error_setg(errp, - "Failed to get \"%s\" lock", - perm_name); + + raw_lock_error_setg_errno(errp, -ret, + "Failed to get \"%s\" lock", + perm_name); g_free(perm_name); return ret; } @@ -XXX,XX +XXX,XX @@ static int raw_check_lock_bytes(int fd, uint64_t perm, uint64_t shared_perm, ret = qemu_lock_fd_test(fd, off, 1, true); if (ret) { char *perm_name = bdrv_perm_names(p); - error_setg(errp, - "Failed to get shared \"%s\" lock", - perm_name); + + raw_lock_error_setg_errno(errp, -ret, + "Failed to get shared \"%s\" lock", + perm_name); g_free(perm_name); return ret; } -- 2.29.2 From: Alberto Garcia <berto@igalia.com> Signed-off-by: Alberto Garcia <berto@igalia.com> Suggested-by: Maxim Levitsky <mlevitsk@redhat.com> Reviewed-by: Maxim Levitsky <mlevitsk@redhat.com> Message-Id: <20210112170540.2912-1-berto@igalia.com> [mreitz: Add "# group:" line] Signed-off-by: Max Reitz <mreitz@redhat.com> --- tests/qemu-iotests/313 | 104 +++++++++++++++++++++++++++++++++++++ tests/qemu-iotests/313.out | 29 +++++++++++ tests/qemu-iotests/group | 1 + 3 files changed, 134 insertions(+) create mode 100755 tests/qemu-iotests/313 create mode 100644 tests/qemu-iotests/313.out diff --git a/tests/qemu-iotests/313 b/tests/qemu-iotests/313 new file mode 100755 index XXXXXXX..XXXXXXX --- /dev/null +++ b/tests/qemu-iotests/313 @@ -XXX,XX +XXX,XX @@ +#!/usr/bin/env bash +# group: rw auto quick +# +# Test for the regression fixed in commit c8bf9a9169 +# +# Copyright (C) 2020 Igalia, S.L. +# Author: Alberto Garcia <berto@igalia.com> +# Based on a test case by Maxim Levitsky <mlevitsk@redhat.com> +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# + +# creator +owner=berto@igalia.com + +seq=`basename $0` +echo "QA output created by $seq" + +status=1 # failure is the default! + +_cleanup() +{ + _cleanup_test_img +} +trap "_cleanup; exit \$status" 0 1 2 3 15 + +# get standard environment, filters and checks +. ./common.rc +. ./common.filter + +_supported_fmt qcow2 +_supported_proto file +_supported_os Linux +_unsupported_imgopts cluster_size refcount_bits extended_l2 compat=0.10 data_file + +# The cluster size must be at least the granularity of the mirror job (4KB) +# Note that larger cluster sizes will produce very large images (several GBs) +cluster_size=4096 +refcount_bits=64 # Make it equal to the L2 entry size for convenience +options="cluster_size=${cluster_size},refcount_bits=${refcount_bits}" + +# Number of refcount entries per refcount blocks +ref_entries=$(( ${cluster_size} * 8 / ${refcount_bits} )) + +# Number of data clusters needed to fill a refcount block +# Equals ${ref_entries} minus two (one L2 table and one refcount block) +data_clusters_per_refblock=$(( ${ref_entries} - 2 )) + +# Number of entries in the refcount cache +ref_blocks=4 + +# Write enough data clusters to fill the refcount cache and allocate +# one more refcount block. +# Subtract 3 clusters from the total: qcow2 header, refcount table, L1 table +total_data_clusters=$(( ${data_clusters_per_refblock} * ${ref_blocks} + 1 - 3 )) + +# Total size to write in bytes +total_size=$(( ${total_data_clusters} * ${cluster_size} )) + +echo +echo '### Create the image' +echo +TEST_IMG_FILE=$TEST_IMG.base _make_test_img -o $options $total_size | _filter_img_create_size + +echo +echo '### Write data to allocate more refcount blocks than the cache can hold' +echo +$QEMU_IO -c "write -P 1 0 $total_size" $TEST_IMG.base | _filter_qemu_io + +echo +echo '### Create an overlay' +echo +_make_test_img -F $IMGFMT -b $TEST_IMG.base -o $options | _filter_img_create_size + +echo +echo '### Fill the overlay with zeroes' +echo +$QEMU_IO -c "write -z 0 $total_size" $TEST_IMG | _filter_qemu_io + +echo +echo '### Commit changes to the base image' +echo +$QEMU_IMG commit $TEST_IMG + +echo +echo '### Check the base image' +echo +$QEMU_IMG check $TEST_IMG.base + +# success, all done +echo "*** done" +rm -f $seq.full +status=0 diff --git a/tests/qemu-iotests/313.out b/tests/qemu-iotests/313.out new file mode 100644 index XXXXXXX..XXXXXXX --- /dev/null +++ b/tests/qemu-iotests/313.out @@ -XXX,XX +XXX,XX @@ +QA output created by 313 + +### Create the image + +Formatting 'TEST_DIR/t.IMGFMT.base', fmt=IMGFMT size=SIZE + +### Write data to allocate more refcount blocks than the cache can hold + +wrote 8347648/8347648 bytes at offset 0 +7.961 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +### Create an overlay + +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=SIZE backing_file=TEST_DIR/t.IMGFMT.base backing_fmt=IMGFMT + +### Fill the overlay with zeroes + +wrote 8347648/8347648 bytes at offset 0 +7.961 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +### Commit changes to the base image + +Image committed. + +### Check the base image + +No errors were found on the image. +Image end offset: 8396800 +*** done diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group index XXXXXXX..XXXXXXX 100644 --- a/tests/qemu-iotests/group +++ b/tests/qemu-iotests/group @@ -XXX,XX +XXX,XX @@ 309 rw auto quick 310 rw quick 312 rw quick +313 rw auto quick -- 2.29.2 Commit 0afec75734331 removed the 'change' QMP command, so we can no longer test it in 118. Fixes: 0afec75734331a0b52fa3aa4235220eda8c7846f ('qmp: remove deprecated "change" command') Signed-off-by: Max Reitz <mreitz@redhat.com> Message-Id: <20210126104833.57026-1-mreitz@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> --- tests/qemu-iotests/118 | 20 +------------------- tests/qemu-iotests/118.out | 4 ++-- 2 files changed, 3 insertions(+), 21 deletions(-) diff --git a/tests/qemu-iotests/118 b/tests/qemu-iotests/118 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/118 +++ b/tests/qemu-iotests/118 @@ -XXX,XX +XXX,XX @@ #!/usr/bin/env python3 # group: rw # -# Test case for the QMP 'change' command and all other associated -# commands +# Test case for media change monitor commands # # Copyright (C) 2015 Red Hat, Inc. # @@ -XXX,XX +XXX,XX @@ class ChangeBaseClass(iotests.QMPTestCase): class GeneralChangeTestsBaseClass(ChangeBaseClass): - def test_change(self): - # 'change' requires a drive name, so skip the test for blockdev - if not self.use_drive: - return - - result = self.vm.qmp('change', device='drive0', target=new_img, - arg=iotests.imgfmt) - self.assert_qmp(result, 'return', {}) - - self.wait_for_open() - self.wait_for_close() - - result = self.vm.qmp('query-block') - if self.has_real_tray: - self.assert_qmp(result, 'return[0]/tray_open', False) - self.assert_qmp(result, 'return[0]/inserted/image/filename', new_img) - def test_blockdev_change_medium(self): result = self.vm.qmp('blockdev-change-medium', id=self.device_name, filename=new_img, diff --git a/tests/qemu-iotests/118.out b/tests/qemu-iotests/118.out index XXXXXXX..XXXXXXX 100644 --- a/tests/qemu-iotests/118.out +++ b/tests/qemu-iotests/118.out @@ -XXX,XX +XXX,XX @@ -....................................................................................................................................................................... +........................................................................................................................................................... ---------------------------------------------------------------------- -Ran 167 tests +Ran 155 tests OK -- 2.29.2 ccd3b3b8112 has deprecated short-hand boolean options (i.e., options with values). All options without values are interpreted as boolean options, so this includes the invalid option "snapshot.foo" used in iotest 178. So after ccd3b3b8112, 178 fails with: +qemu-img: warning: short-form boolean option 'snapshot.foo' deprecated +Please use snapshot.foo=on instead Suppress that deprecation warning by passing some value to it (it does not matter which, because the option is invalid anyway). Fixes: ccd3b3b8112b670fdccf8a392b8419b173ffccb4 ("qemu-option: warn for short-form boolean options") Signed-off-by: Max Reitz <mreitz@redhat.com> Message-Id: <20210126123834.115915-1-mreitz@redhat.com> --- tests/qemu-iotests/178 | 2 +- tests/qemu-iotests/178.out.qcow2 | 2 +- tests/qemu-iotests/178.out.raw | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/tests/qemu-iotests/178 b/tests/qemu-iotests/178 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/178 +++ b/tests/qemu-iotests/178 @@ -XXX,XX +XXX,XX @@ $QEMU_IMG measure --image-opts # missing filename $QEMU_IMG measure -f qcow2 # missing filename $QEMU_IMG measure -l snap1 # missing filename $QEMU_IMG measure -o , # invalid option list -$QEMU_IMG measure -l snapshot.foo # invalid snapshot option +$QEMU_IMG measure -l snapshot.foo=bar # invalid snapshot option $QEMU_IMG measure --output foo # invalid output format $QEMU_IMG measure --size -1 # invalid image size $QEMU_IMG measure -O foo "$TEST_IMG" # unknown image file format diff --git a/tests/qemu-iotests/178.out.qcow2 b/tests/qemu-iotests/178.out.qcow2 index XXXXXXX..XXXXXXX 100644 --- a/tests/qemu-iotests/178.out.qcow2 +++ b/tests/qemu-iotests/178.out.qcow2 @@ -XXX,XX +XXX,XX @@ qemu-img: --image-opts, -f, and -l require a filename argument. qemu-img: --image-opts, -f, and -l require a filename argument. qemu-img: Invalid option list: , qemu-img: Invalid parameter 'snapshot.foo' -qemu-img: Failed in parsing snapshot param 'snapshot.foo' +qemu-img: Failed in parsing snapshot param 'snapshot.foo=bar' qemu-img: --output must be used with human or json as argument. qemu-img: Invalid image size specified. Must be between 0 and 9223372036854775807. qemu-img: Unknown file format 'foo' diff --git a/tests/qemu-iotests/178.out.raw b/tests/qemu-iotests/178.out.raw index XXXXXXX..XXXXXXX 100644 --- a/tests/qemu-iotests/178.out.raw +++ b/tests/qemu-iotests/178.out.raw @@ -XXX,XX +XXX,XX @@ qemu-img: --image-opts, -f, and -l require a filename argument. qemu-img: --image-opts, -f, and -l require a filename argument. qemu-img: Invalid option list: , qemu-img: Invalid parameter 'snapshot.foo' -qemu-img: Failed in parsing snapshot param 'snapshot.foo' +qemu-img: Failed in parsing snapshot param 'snapshot.foo=bar' qemu-img: --output must be used with human or json as argument. qemu-img: Invalid image size specified. Must be between 0 and 9223372036854775807. qemu-img: Unknown file format 'foo' -- 2.29.2