The following changes since commit 248b23735645f7cbb503d9be6f5bf825f2a603ab: Update version for v2.10.0-rc4 release (2017-08-24 17:34:26 +0100) are available in the git repository at: git://github.com/stefanha/qemu.git tags/block-pull-request for you to fetch changes up to 3e4c705212abfe8c9882a00beb2d1466a8a53cec: qcow2: allocate cluster_cache/cluster_data on demand (2017-08-30 18:02:10 +0100) ---------------------------------------------------------------- ---------------------------------------------------------------- Alberto Garcia (8): throttle: Fix wrong variable name in the header documentation throttle: Update the throttle_fix_bucket() documentation throttle: Make throttle_is_valid() a bit less verbose throttle: Remove throttle_fix_bucket() / throttle_unfix_bucket() throttle: Make LeakyBucket.avg and LeakyBucket.max integer types throttle: Make burst_length 64bit and add range checks throttle: Test the valid range of config values misc: Remove unused Error variables Dan Aloni (1): nvme: Fix get/set number of queues feature, again Eduardo Habkost (1): oslib-posix: Print errors before aborting on qemu_alloc_stack() Fred Rolland (1): qemu-doc: Add UUID support in initiator name Stefan Hajnoczi (4): scripts: add argparse module for Python 2.6 compatibility docker.py: Python 2.6 argparse compatibility tests: migration/guestperf Python 2.6 argparse compatibility qcow2: allocate cluster_cache/cluster_data on demand include/qemu/throttle.h | 8 +- block/qcow.c | 12 +- block/qcow2-cluster.c | 17 + block/qcow2.c | 20 +- dump.c | 4 +- hw/block/nvme.c | 4 +- tests/test-throttle.c | 80 +- util/oslib-posix.c | 2 + util/throttle.c | 86 +- COPYING.PYTHON | 270 ++++ qemu-doc.texi | 5 +- scripts/argparse.py | 2406 ++++++++++++++++++++++++++++++++++++ tests/docker/docker.py | 4 +- tests/migration/guestperf/shell.py | 8 +- 14 files changed, 2831 insertions(+), 95 deletions(-) create mode 100644 COPYING.PYTHON create mode 100644 scripts/argparse.py -- 2.13.5 From: Dan Aloni <dan@kernelim.com> The number of queues that should be return by the admin command should: 1) Only mention the number of non-admin queues. 2) It is zero-based, meaning that '0 == one non-admin queue', '1 == two non-admin queues', and so forth. Because our `num_queues` means the number of queues _plus_ the admin queue, then the right calculation for the number returned from the admin command is `num_queues - 2`, combining the two requirements mentioned. The issue was discovered by reducing num_queues from 64 to 8 and running a Linux VM with an SMP parameter larger than that (e.g. 22). It tries to utilize all queues, and therefore fails with an invalid queue number when trying to queue I/Os on the last queue. Signed-off-by: Dan Aloni <dan@kernelim.com> CC: Alex Friedman <alex@e8storage.com> CC: Keith Busch <keith.busch@intel.com> CC: Stefan Hajnoczi <stefanha@redhat.com> Reviewed-by: Keith Busch <keith.busch@intel.com> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com> --- hw/block/nvme.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/hw/block/nvme.c b/hw/block/nvme.c index XXXXXXX..XXXXXXX 100644 --- a/hw/block/nvme.c +++ b/hw/block/nvme.c @@ -XXX,XX +XXX,XX @@ static uint16_t nvme_get_feature(NvmeCtrl *n, NvmeCmd *cmd, NvmeRequest *req) result = blk_enable_write_cache(n->conf.blk); break; case NVME_NUMBER_OF_QUEUES: - result = cpu_to_le32((n->num_queues - 1) | ((n->num_queues - 1) << 16)); + result = cpu_to_le32((n->num_queues - 2) | ((n->num_queues - 2) << 16)); break; default: return NVME_INVALID_FIELD | NVME_DNR; @@ -XXX,XX +XXX,XX @@ static uint16_t nvme_set_feature(NvmeCtrl *n, NvmeCmd *cmd, NvmeRequest *req) break; case NVME_NUMBER_OF_QUEUES: req->cqe.result = - cpu_to_le32((n->num_queues - 1) | ((n->num_queues - 1) << 16)); + cpu_to_le32((n->num_queues - 2) | ((n->num_queues - 2) << 16)); break; default: return NVME_INVALID_FIELD | NVME_DNR; -- 2.13.5 From: Alberto Garcia <berto@igalia.com> The level of the burst bucket is stored in bkt.burst_level, not bkt.burst_length. Signed-off-by: Alberto Garcia <berto@igalia.com> Reviewed-by: Manos Pitsidianakis <el13635@mail.ntua.gr> Message-id: 49aab2711d02f285567f3b3b13a113847af33812.1503580370.git.berto@igalia.com Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com> --- include/qemu/throttle.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/include/qemu/throttle.h b/include/qemu/throttle.h index XXXXXXX..XXXXXXX 100644 --- a/include/qemu/throttle.h +++ b/include/qemu/throttle.h @@ -XXX,XX +XXX,XX @@ typedef enum { * - The bkt.avg rate does not apply until the bucket is full, * allowing the user to do bursts until then. The I/O limit during * bursts is bkt.max. To enforce this limit we keep an additional - * bucket in bkt.burst_length that leaks at a rate of bkt.max units + * bucket in bkt.burst_level that leaks at a rate of bkt.max units * per second. * * - Because of all of the above, the user can perform I/O at a -- 2.13.5 From: Alberto Garcia <berto@igalia.com> The way the throttling algorithm works is that requests start being throttled once the bucket level exceeds the burst limit. When we get there the bucket leaks at the level set by the user (bkt->avg), and that leak rate is what prevents guest I/O from exceeding the desired limit. If we don't allow bursts (i.e. bkt->max == 0) then we can start throttling requests immediately. The problem with keeping the threshold at 0 is that it only allows one request at a time, and as soon as there's a bit of I/O from the guest every other request will be throttled and performance will suffer considerably. That can even make the guest unable to reach the throttle limit if that limit is high enough, and that happens regardless of the block scheduler used by the guest. Increasing that threshold gives flexibility to the guest, allowing it to perform short bursts of I/O before being throttled. Increasing the threshold too much does not make a difference in the long run (because it's the leak rate what defines the actual throughput) but it does allow the guest to perform longer initial bursts and exceed the throttle limit for a short while. A burst value of bkt->avg / 10 allows the guest to perform 100ms' worth of I/O at the target rate without being throttled. Signed-off-by: Alberto Garcia <berto@igalia.com> Message-id: 31aae6645f0d1fbf3860fb2b528b757236f0c0a7.1503580370.git.berto@igalia.com Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com> --- util/throttle.c | 11 +++-------- 1 file changed, 3 insertions(+), 8 deletions(-) diff --git a/util/throttle.c b/util/throttle.c index XXXXXXX..XXXXXXX 100644 --- a/util/throttle.c +++ b/util/throttle.c @@ -XXX,XX +XXX,XX @@ static void throttle_fix_bucket(LeakyBucket *bkt) /* zero bucket level */ bkt->level = bkt->burst_level = 0; - /* The following is done to cope with the Linux CFQ block scheduler - * which regroup reads and writes by block of 100ms in the guest. - * When they are two process one making reads and one making writes cfq - * make a pattern looking like the following: - * WWWWWWWWWWWRRRRRRRRRRRRRRWWWWWWWWWWWWWwRRRRRRRRRRRRRRRRR - * Having a max burst value of 100ms of the average will help smooth the - * throttling - */ + /* If bkt->max is 0 we still want to allow short bursts of I/O + * from the guest, otherwise every other request will be throttled + * and performance will suffer considerably. */ min = bkt->avg / 10; if (bkt->avg && !bkt->max) { bkt->max = min; -- 2.13.5 From: Alberto Garcia <berto@igalia.com> Use a pointer to the bucket instead of repeating cfg->buckets[i] all the time. This makes the code more concise and will help us expand the checks later and save a few line breaks. Signed-off-by: Alberto Garcia <berto@igalia.com> Message-id: 763ffc40a26b17d54cf93f5a999e4656049fcf0c.1503580370.git.berto@igalia.com Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com> --- util/throttle.c | 15 +++++++-------- 1 file changed, 7 insertions(+), 8 deletions(-) diff --git a/util/throttle.c b/util/throttle.c index XXXXXXX..XXXXXXX 100644 --- a/util/throttle.c +++ b/util/throttle.c @@ -XXX,XX +XXX,XX @@ bool throttle_is_valid(ThrottleConfig *cfg, Error **errp) } for (i = 0; i < BUCKETS_COUNT; i++) { - if (cfg->buckets[i].avg < 0 || - cfg->buckets[i].max < 0 || - cfg->buckets[i].avg > THROTTLE_VALUE_MAX || - cfg->buckets[i].max > THROTTLE_VALUE_MAX) { + LeakyBucket *bkt = &cfg->buckets[i]; + if (bkt->avg < 0 || bkt->max < 0 || + bkt->avg > THROTTLE_VALUE_MAX || bkt->max > THROTTLE_VALUE_MAX) { error_setg(errp, "bps/iops/max values must be within [0, %lld]", THROTTLE_VALUE_MAX); return false; } - if (!cfg->buckets[i].burst_length) { + if (!bkt->burst_length) { error_setg(errp, "the burst length cannot be 0"); return false; } - if (cfg->buckets[i].burst_length > 1 && !cfg->buckets[i].max) { + if (bkt->burst_length > 1 && !bkt->max) { error_setg(errp, "burst length set without burst rate"); return false; } - if (cfg->buckets[i].max && !cfg->buckets[i].avg) { + if (bkt->max && !bkt->avg) { error_setg(errp, "bps_max/iops_max require corresponding" " bps/iops values"); return false; } - if (cfg->buckets[i].max && cfg->buckets[i].max < cfg->buckets[i].avg) { + if (bkt->max && bkt->max < bkt->avg) { error_setg(errp, "bps_max/iops_max cannot be lower than bps/iops"); return false; } -- 2.13.5 From: Alberto Garcia <berto@igalia.com> The throttling code can change internally the value of bkt->max if it hasn't been set by the user. The problem with this is that if we want to retrieve the original value we have to undo this change first. This is ugly and unnecessary: this patch removes the throttle_fix_bucket() and throttle_unfix_bucket() functions completely and moves the logic to throttle_compute_wait(). Signed-off-by: Alberto Garcia <berto@igalia.com> Reviewed-by: Manos Pitsidianakis <el13635@mail.ntua.gr> Message-id: 5b0b9e1ac6eb208d709eddc7b09e7669a523bff3.1503580370.git.berto@igalia.com Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com> --- util/throttle.c | 62 +++++++++++++++++++++------------------------------------ 1 file changed, 23 insertions(+), 39 deletions(-) diff --git a/util/throttle.c b/util/throttle.c index XXXXXXX..XXXXXXX 100644 --- a/util/throttle.c +++ b/util/throttle.c @@ -XXX,XX +XXX,XX @@ static int64_t throttle_do_compute_wait(double limit, double extra) int64_t throttle_compute_wait(LeakyBucket *bkt) { double extra; /* the number of extra units blocking the io */ + double bucket_size; /* I/O before throttling to bkt->avg */ + double burst_bucket_size; /* Before throttling to bkt->max */ if (!bkt->avg) { return 0; } - /* If the bucket is full then we have to wait */ - extra = bkt->level - bkt->max * bkt->burst_length; + if (!bkt->max) { + /* If bkt->max is 0 we still want to allow short bursts of I/O + * from the guest, otherwise every other request will be throttled + * and performance will suffer considerably. */ + bucket_size = bkt->avg / 10; + burst_bucket_size = 0; + } else { + /* If we have a burst limit then we have to wait until all I/O + * at burst rate has finished before throttling to bkt->avg */ + bucket_size = bkt->max * bkt->burst_length; + burst_bucket_size = bkt->max / 10; + } + + /* If the main bucket is full then we have to wait */ + extra = bkt->level - bucket_size; if (extra > 0) { return throttle_do_compute_wait(bkt->avg, extra); } - /* If the bucket is not full yet we have to make sure that we - * fulfill the goal of bkt->max units per second. */ + /* If the main bucket is not full yet we still have to check the + * burst bucket in order to enforce the burst limit */ if (bkt->burst_length > 1) { - /* We use 1/10 of the max value to smooth the throttling. - * See throttle_fix_bucket() for more details. */ - extra = bkt->burst_level - bkt->max / 10; + extra = bkt->burst_level - burst_bucket_size; if (extra > 0) { return throttle_do_compute_wait(bkt->max, extra); } @@ -XXX,XX +XXX,XX @@ bool throttle_is_valid(ThrottleConfig *cfg, Error **errp) return true; } -/* fix bucket parameters */ -static void throttle_fix_bucket(LeakyBucket *bkt) -{ - double min; - - /* zero bucket level */ - bkt->level = bkt->burst_level = 0; - - /* If bkt->max is 0 we still want to allow short bursts of I/O - * from the guest, otherwise every other request will be throttled - * and performance will suffer considerably. */ - min = bkt->avg / 10; - if (bkt->avg && !bkt->max) { - bkt->max = min; - } -} - -/* undo internal bucket parameter changes (see throttle_fix_bucket()) */ -static void throttle_unfix_bucket(LeakyBucket *bkt) -{ - if (bkt->max < bkt->avg) { - bkt->max = 0; - } -} - /* Used to configure the throttle * * @ts: the throttle state we are working on @@ -XXX,XX +XXX,XX @@ void throttle_config(ThrottleState *ts, ts->cfg = *cfg; + /* Zero bucket level */ for (i = 0; i < BUCKETS_COUNT; i++) { - throttle_fix_bucket(&ts->cfg.buckets[i]); + ts->cfg.buckets[i].level = 0; + ts->cfg.buckets[i].burst_level = 0; } ts->previous_leak = qemu_clock_get_ns(clock_type); @@ -XXX,XX +XXX,XX @@ void throttle_config(ThrottleState *ts, */ void throttle_get_config(ThrottleState *ts, ThrottleConfig *cfg) { - int i; - *cfg = ts->cfg; - - for (i = 0; i < BUCKETS_COUNT; i++) { - throttle_unfix_bucket(&cfg->buckets[i]); - } } -- 2.13.5 From: Alberto Garcia <berto@igalia.com> Both the throttling limits set with the throttling.iops-* and throttling.bps-* options and their QMP equivalents defined in the BlockIOThrottle struct are integer values. Those limits are also reported in the BlockDeviceInfo struct and they are integers there as well. Therefore there's no reason to store them internally as double and do the conversion everytime we're setting or querying them, so this patch uses uint64_t for those types. Let's also use an unsigned type because we don't allow negative values anyway. LeakyBucket.level and LeakyBucket.burst_level do however remain double because their value changes depending on the fraction of time elapsed since the previous I/O operation. Signed-off-by: Alberto Garcia <berto@igalia.com> Message-id: f29b840422767b5be2c41c2dfdbbbf6c5f8fedf8.1503580370.git.berto@igalia.com Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com> --- include/qemu/throttle.h | 4 ++-- tests/test-throttle.c | 3 ++- util/throttle.c | 7 +++---- 3 files changed, 7 insertions(+), 7 deletions(-) diff --git a/include/qemu/throttle.h b/include/qemu/throttle.h index XXXXXXX..XXXXXXX 100644 --- a/include/qemu/throttle.h +++ b/include/qemu/throttle.h @@ -XXX,XX +XXX,XX @@ typedef enum { */ typedef struct LeakyBucket { - double avg; /* average goal in units per second */ - double max; /* leaky bucket max burst in units */ + uint64_t avg; /* average goal in units per second */ + uint64_t max; /* leaky bucket max burst in units */ double level; /* bucket level in units */ double burst_level; /* bucket level in units (for computing bursts) */ unsigned burst_length; /* max length of the burst period, in seconds */ diff --git a/tests/test-throttle.c b/tests/test-throttle.c index XXXXXXX..XXXXXXX 100644 --- a/tests/test-throttle.c +++ b/tests/test-throttle.c @@ -XXX,XX +XXX,XX @@ static void test_enabled(void) for (i = 0; i < BUCKETS_COUNT; i++) { throttle_config_init(&cfg); set_cfg_value(false, i, 150); + g_assert(throttle_is_valid(&cfg, NULL)); g_assert(throttle_enabled(&cfg)); } for (i = 0; i < BUCKETS_COUNT; i++) { throttle_config_init(&cfg); set_cfg_value(false, i, -150); - g_assert(!throttle_enabled(&cfg)); + g_assert(!throttle_is_valid(&cfg, NULL)); } } diff --git a/util/throttle.c b/util/throttle.c index XXXXXXX..XXXXXXX 100644 --- a/util/throttle.c +++ b/util/throttle.c @@ -XXX,XX +XXX,XX @@ int64_t throttle_compute_wait(LeakyBucket *bkt) /* If bkt->max is 0 we still want to allow short bursts of I/O * from the guest, otherwise every other request will be throttled * and performance will suffer considerably. */ - bucket_size = bkt->avg / 10; + bucket_size = (double) bkt->avg / 10; burst_bucket_size = 0; } else { /* If we have a burst limit then we have to wait until all I/O * at burst rate has finished before throttling to bkt->avg */ bucket_size = bkt->max * bkt->burst_length; - burst_bucket_size = bkt->max / 10; + burst_bucket_size = (double) bkt->max / 10; } /* If the main bucket is full then we have to wait */ @@ -XXX,XX +XXX,XX @@ bool throttle_is_valid(ThrottleConfig *cfg, Error **errp) for (i = 0; i < BUCKETS_COUNT; i++) { LeakyBucket *bkt = &cfg->buckets[i]; - if (bkt->avg < 0 || bkt->max < 0 || - bkt->avg > THROTTLE_VALUE_MAX || bkt->max > THROTTLE_VALUE_MAX) { + if (bkt->avg > THROTTLE_VALUE_MAX || bkt->max > THROTTLE_VALUE_MAX) { error_setg(errp, "bps/iops/max values must be within [0, %lld]", THROTTLE_VALUE_MAX); return false; -- 2.13.5 From: Alberto Garcia <berto@igalia.com> LeakyBucket.burst_length is defined as an unsigned integer but the code never checks for overflows and it only makes sure that the value is not 0. In practice this means that the user can set something like throttling.iops-total-max-length=4294967300 despite being larger than UINT_MAX and the final value after casting to unsigned int will be 4. This patch changes the data type to uint64_t. This does not increase the storage size of LeakyBucket, and allows us to assign the value directly from qemu_opt_get_number() or BlockIOThrottle and then do the checks directly in throttle_is_valid(). The value of burst_length does not have a specific upper limit, but since the bucket size is defined by max * burst_length we have to prevent overflows. Instead of going for UINT64_MAX or something similar this patch reuses THROTTLE_VALUE_MAX, which allows I/O bursts of 1 GiB/s for 10 days in a row. Signed-off-by: Alberto Garcia <berto@igalia.com> Message-id: 1b2e3049803f71cafb2e1fa1be4fb47147a0d398.1503580370.git.berto@igalia.com Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com> --- include/qemu/throttle.h | 2 +- util/throttle.c | 5 +++++ 2 files changed, 6 insertions(+), 1 deletion(-) diff --git a/include/qemu/throttle.h b/include/qemu/throttle.h index XXXXXXX..XXXXXXX 100644 --- a/include/qemu/throttle.h +++ b/include/qemu/throttle.h @@ -XXX,XX +XXX,XX @@ typedef struct LeakyBucket { uint64_t max; /* leaky bucket max burst in units */ double level; /* bucket level in units */ double burst_level; /* bucket level in units (for computing bursts) */ - unsigned burst_length; /* max length of the burst period, in seconds */ + uint64_t burst_length; /* max length of the burst period, in seconds */ } LeakyBucket; /* The following structure is used to configure a ThrottleState diff --git a/util/throttle.c b/util/throttle.c index XXXXXXX..XXXXXXX 100644 --- a/util/throttle.c +++ b/util/throttle.c @@ -XXX,XX +XXX,XX @@ bool throttle_is_valid(ThrottleConfig *cfg, Error **errp) return false; } + if (bkt->max && bkt->burst_length > THROTTLE_VALUE_MAX / bkt->max) { + error_setg(errp, "burst length too high for this burst rate"); + return false; + } + if (bkt->max && !bkt->avg) { error_setg(errp, "bps_max/iops_max require corresponding" " bps/iops values"); -- 2.13.5 From: Alberto Garcia <berto@igalia.com> Signed-off-by: Alberto Garcia <berto@igalia.com> Message-id: a57dd6274e1b6dc9c28769fec4c7ea543be5c5e3.1503580370.git.berto@igalia.com Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com> --- tests/test-throttle.c | 77 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 77 insertions(+) diff --git a/tests/test-throttle.c b/tests/test-throttle.c index XXXXXXX..XXXXXXX 100644 --- a/tests/test-throttle.c +++ b/tests/test-throttle.c @@ -XXX,XX +XXX,XX @@ static void test_is_valid(void) test_is_valid_for_value(1, true); } +static void test_ranges(void) +{ + int i; + + for (i = 0; i < BUCKETS_COUNT; i++) { + LeakyBucket *b = &cfg.buckets[i]; + throttle_config_init(&cfg); + + /* avg = 0 means throttling is disabled, but the config is valid */ + b->avg = 0; + g_assert(throttle_is_valid(&cfg, NULL)); + g_assert(!throttle_enabled(&cfg)); + + /* These are valid configurations (values <= THROTTLE_VALUE_MAX) */ + b->avg = 1; + g_assert(throttle_is_valid(&cfg, NULL)); + + b->avg = THROTTLE_VALUE_MAX; + g_assert(throttle_is_valid(&cfg, NULL)); + + b->avg = THROTTLE_VALUE_MAX; + b->max = THROTTLE_VALUE_MAX; + g_assert(throttle_is_valid(&cfg, NULL)); + + /* Values over THROTTLE_VALUE_MAX are not allowed */ + b->avg = THROTTLE_VALUE_MAX + 1; + g_assert(!throttle_is_valid(&cfg, NULL)); + + b->avg = THROTTLE_VALUE_MAX; + b->max = THROTTLE_VALUE_MAX + 1; + g_assert(!throttle_is_valid(&cfg, NULL)); + + /* burst_length must be between 1 and THROTTLE_VALUE_MAX */ + b->avg = 1; + b->max = 1; + b->burst_length = 0; + g_assert(!throttle_is_valid(&cfg, NULL)); + + b->avg = 1; + b->max = 1; + b->burst_length = 1; + g_assert(throttle_is_valid(&cfg, NULL)); + + b->avg = 1; + b->max = 1; + b->burst_length = THROTTLE_VALUE_MAX; + g_assert(throttle_is_valid(&cfg, NULL)); + + b->avg = 1; + b->max = 1; + b->burst_length = THROTTLE_VALUE_MAX + 1; + g_assert(!throttle_is_valid(&cfg, NULL)); + + /* burst_length * max cannot exceed THROTTLE_VALUE_MAX */ + b->avg = 1; + b->max = 2; + b->burst_length = THROTTLE_VALUE_MAX / 2; + g_assert(throttle_is_valid(&cfg, NULL)); + + b->avg = 1; + b->max = 3; + b->burst_length = THROTTLE_VALUE_MAX / 2; + g_assert(!throttle_is_valid(&cfg, NULL)); + + b->avg = 1; + b->max = THROTTLE_VALUE_MAX; + b->burst_length = 1; + g_assert(throttle_is_valid(&cfg, NULL)); + + b->avg = 1; + b->max = THROTTLE_VALUE_MAX; + b->burst_length = 2; + g_assert(!throttle_is_valid(&cfg, NULL)); + } +} + static void test_max_is_missing_limit(void) { int i; @@ -XXX,XX +XXX,XX @@ int main(int argc, char **argv) g_test_add_func("/throttle/config/enabled", test_enabled); g_test_add_func("/throttle/config/conflicting", test_conflicting_config); g_test_add_func("/throttle/config/is_valid", test_is_valid); + g_test_add_func("/throttle/config/ranges", test_ranges); g_test_add_func("/throttle/config/max", test_max_is_missing_limit); g_test_add_func("/throttle/config/iops_size", test_iops_size_is_missing_limit); -- 2.13.5 From: Eduardo Habkost <ehabkost@redhat.com> If QEMU is running on a system that's out of memory and mmap() fails, QEMU aborts with no error message at all, making it hard to debug the reason for the failure. Add perror() calls that will print error information before aborting. Signed-off-by: Eduardo Habkost <ehabkost@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Message-id: 20170829212053.6003-1-ehabkost@redhat.com Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com> --- util/oslib-posix.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/util/oslib-posix.c b/util/oslib-posix.c index XXXXXXX..XXXXXXX 100644 --- a/util/oslib-posix.c +++ b/util/oslib-posix.c @@ -XXX,XX +XXX,XX @@ void *qemu_alloc_stack(size_t *sz) ptr = mmap(NULL, *sz, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); if (ptr == MAP_FAILED) { + perror("failed to allocate memory for stack"); abort(); } @@ -XXX,XX +XXX,XX @@ void *qemu_alloc_stack(size_t *sz) guardpage = ptr; #endif if (mprotect(guardpage, pagesz, PROT_NONE) != 0) { + perror("failed to set up stack guard page"); abort(); } -- 2.13.5 From: Alberto Garcia <berto@igalia.com> There's a few cases which we're passing an Error pointer to a function only to discard it immediately afterwards without checking it. In these cases we can simply remove the variable and pass NULL instead. Signed-off-by: Alberto Garcia <berto@igalia.com> Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Reviewed-by: Eric Blake <eblake@redhat.com> Message-id: 20170829120836.16091-1-berto@igalia.com Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com> --- block/qcow.c | 12 +++--------- block/qcow2.c | 8 ++------ dump.c | 4 +--- 3 files changed, 6 insertions(+), 18 deletions(-) diff --git a/block/qcow.c b/block/qcow.c index XXXXXXX..XXXXXXX 100644 --- a/block/qcow.c +++ b/block/qcow.c @@ -XXX,XX +XXX,XX @@ static uint64_t get_cluster_offset(BlockDriverState *bs, start_sect = (offset & ~(s->cluster_size - 1)) >> 9; for(i = 0; i < s->cluster_sectors; i++) { if (i < n_start || i >= n_end) { - Error *err = NULL; memset(s->cluster_data, 0x00, 512); if (qcrypto_block_encrypt(s->crypto, start_sect + i, s->cluster_data, BDRV_SECTOR_SIZE, - &err) < 0) { - error_free(err); + NULL) < 0) { errno = EIO; return -1; } @@ -XXX,XX +XXX,XX @@ static coroutine_fn int qcow_co_readv(BlockDriverState *bs, int64_t sector_num, QEMUIOVector hd_qiov; uint8_t *buf; void *orig_buf; - Error *err = NULL; if (qiov->niov > 1) { buf = orig_buf = qemu_try_blockalign(bs, qiov->size); @@ -XXX,XX +XXX,XX @@ static coroutine_fn int qcow_co_readv(BlockDriverState *bs, int64_t sector_num, if (bs->encrypted) { assert(s->crypto); if (qcrypto_block_decrypt(s->crypto, sector_num, buf, - n * BDRV_SECTOR_SIZE, &err) < 0) { + n * BDRV_SECTOR_SIZE, NULL) < 0) { goto fail; } } @@ -XXX,XX +XXX,XX @@ done: return ret; fail: - error_free(err); ret = -EIO; goto done; } @@ -XXX,XX +XXX,XX @@ static coroutine_fn int qcow_co_writev(BlockDriverState *bs, int64_t sector_num, break; } if (bs->encrypted) { - Error *err = NULL; assert(s->crypto); if (qcrypto_block_encrypt(s->crypto, sector_num, buf, - n * BDRV_SECTOR_SIZE, &err) < 0) { - error_free(err); + n * BDRV_SECTOR_SIZE, NULL) < 0) { ret = -EIO; break; } diff --git a/block/qcow2.c b/block/qcow2.c index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -XXX,XX +XXX,XX @@ static coroutine_fn int qcow2_co_preadv(BlockDriverState *bs, uint64_t offset, assert(s->crypto); assert((offset & (BDRV_SECTOR_SIZE - 1)) == 0); assert((cur_bytes & (BDRV_SECTOR_SIZE - 1)) == 0); - Error *err = NULL; if (qcrypto_block_decrypt(s->crypto, (s->crypt_physical_offset ? cluster_offset + offset_in_cluster : offset) >> BDRV_SECTOR_BITS, cluster_data, cur_bytes, - &err) < 0) { - error_free(err); + NULL) < 0) { ret = -EIO; goto fail; } @@ -XXX,XX +XXX,XX @@ static coroutine_fn int qcow2_co_pwritev(BlockDriverState *bs, uint64_t offset, qemu_iovec_concat(&hd_qiov, qiov, bytes_done, cur_bytes); if (bs->encrypted) { - Error *err = NULL; assert(s->crypto); if (!cluster_data) { cluster_data = qemu_try_blockalign(bs->file->bs, @@ -XXX,XX +XXX,XX @@ static coroutine_fn int qcow2_co_pwritev(BlockDriverState *bs, uint64_t offset, cluster_offset + offset_in_cluster : offset) >> BDRV_SECTOR_BITS, cluster_data, - cur_bytes, &err) < 0) { - error_free(err); + cur_bytes, NULL) < 0) { ret = -EIO; goto fail; } diff --git a/dump.c b/dump.c index XXXXXXX..XXXXXXX 100644 --- a/dump.c +++ b/dump.c @@ -XXX,XX +XXX,XX @@ static void dump_process(DumpState *s, Error **errp) static void *dump_thread(void *data) { - Error *err = NULL; DumpState *s = (DumpState *)data; - dump_process(s, &err); - error_free(err); + dump_process(s, NULL); return NULL; } -- 2.13.5 The minimum Python version supported by QEMU is 2.6. The argparse standard library module was only added in Python 2.7. Many scripts would like to use argparse because it supports command-line sub-commands. This patch adds argparse. See the top of argparse.py for details. Suggested-by: Daniel P. Berrange <berrange@redhat.com> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com> Acked-by: John Snow <jsnow@redhat.com> Message-id: 20170825155732.15665-2-stefanha@redhat.com Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com> --- COPYING.PYTHON | 270 ++++++ scripts/argparse.py | 2406 +++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 2676 insertions(+) create mode 100644 COPYING.PYTHON create mode 100644 scripts/argparse.py diff --git a/COPYING.PYTHON b/COPYING.PYTHON new file mode 100644 index XXXXXXX..XXXXXXX --- /dev/null +++ b/COPYING.PYTHON @@ -XXX,XX +XXX,XX @@ +A. HISTORY OF THE SOFTWARE +========================== + +Python was created in the early 1990s by Guido van Rossum at Stichting +Mathematisch Centrum (CWI, see http://www.cwi.nl) in the Netherlands +as a successor of a language called ABC. Guido remains Python's +principal author, although it includes many contributions from others. + +In 1995, Guido continued his work on Python at the Corporation for +National Research Initiatives (CNRI, see http://www.cnri.reston.va.us) +in Reston, Virginia where he released several versions of the +software. + +In May 2000, Guido and the Python core development team moved to +BeOpen.com to form the BeOpen PythonLabs team. In October of the same +year, the PythonLabs team moved to Digital Creations (now Zope +Corporation, see http://www.zope.com). In 2001, the Python Software +Foundation (PSF, see http://www.python.org/psf/) was formed, a +non-profit organization created specifically to own Python-related +Intellectual Property. Zope Corporation is a sponsoring member of +the PSF. + +All Python releases are Open Source (see http://www.opensource.org for +the Open Source Definition). Historically, most, but not all, Python +releases have also been GPL-compatible; the table below summarizes +the various releases. + + Release Derived Year Owner GPL- + from compatible? (1) + + 0.9.0 thru 1.2 1991-1995 CWI yes + 1.3 thru 1.5.2 1.2 1995-1999 CNRI yes + 1.6 1.5.2 2000 CNRI no + 2.0 1.6 2000 BeOpen.com no + 1.6.1 1.6 2001 CNRI yes (2) + 2.1 2.0+1.6.1 2001 PSF no + 2.0.1 2.0+1.6.1 2001 PSF yes + 2.1.1 2.1+2.0.1 2001 PSF yes + 2.2 2.1.1 2001 PSF yes + 2.1.2 2.1.1 2002 PSF yes + 2.1.3 2.1.2 2002 PSF yes + 2.2.1 2.2 2002 PSF yes + 2.2.2 2.2.1 2002 PSF yes + 2.2.3 2.2.2 2003 PSF yes + 2.3 2.2.2 2002-2003 PSF yes + 2.3.1 2.3 2002-2003 PSF yes + 2.3.2 2.3.1 2002-2003 PSF yes + 2.3.3 2.3.2 2002-2003 PSF yes + 2.3.4 2.3.3 2004 PSF yes + 2.3.5 2.3.4 2005 PSF yes + 2.4 2.3 2004 PSF yes + 2.4.1 2.4 2005 PSF yes + 2.4.2 2.4.1 2005 PSF yes + 2.4.3 2.4.2 2006 PSF yes + 2.5 2.4 2006 PSF yes + 2.7 2.6 2010 PSF yes + +Footnotes: + +(1) GPL-compatible doesn't mean that we're distributing Python under + the GPL. All Python licenses, unlike the GPL, let you distribute + a modified version without making your changes open source. The + GPL-compatible licenses make it possible to combine Python with + other software that is released under the GPL; the others don't. + +(2) According to Richard Stallman, 1.6.1 is not GPL-compatible, + because its license has a choice of law clause. According to + CNRI, however, Stallman's lawyer has told CNRI's lawyer that 1.6.1 + is "not incompatible" with the GPL. + +Thanks to the many outside volunteers who have worked under Guido's +direction to make these releases possible. + + +B. TERMS AND CONDITIONS FOR ACCESSING OR OTHERWISE USING PYTHON +=============================================================== + +PYTHON SOFTWARE FOUNDATION LICENSE VERSION 2 +-------------------------------------------- + +1. This LICENSE AGREEMENT is between the Python Software Foundation +("PSF"), and the Individual or Organization ("Licensee") accessing and +otherwise using this software ("Python") in source or binary form and +its associated documentation. + +2. Subject to the terms and conditions of this License Agreement, PSF +hereby grants Licensee a nonexclusive, royalty-free, world-wide +license to reproduce, analyze, test, perform and/or display publicly, +prepare derivative works, distribute, and otherwise use Python +alone or in any derivative version, provided, however, that PSF's +License Agreement and PSF's notice of copyright, i.e., "Copyright (c) +2001, 2002, 2003, 2004, 2005, 2006 Python Software Foundation; All Rights +Reserved" are retained in Python alone or in any derivative version +prepared by Licensee. + +3. In the event Licensee prepares a derivative work that is based on +or incorporates Python or any part thereof, and wants to make +the derivative work available to others as provided herein, then +Licensee hereby agrees to include in any such work a brief summary of +the changes made to Python. + +4. PSF is making Python available to Licensee on an "AS IS" +basis. PSF MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR +IMPLIED. BY WAY OF EXAMPLE, BUT NOT LIMITATION, PSF MAKES NO AND +DISCLAIMS ANY REPRESENTATION OR WARRANTY OF MERCHANTABILITY OR FITNESS +FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF PYTHON WILL NOT +INFRINGE ANY THIRD PARTY RIGHTS. + +5. PSF SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF PYTHON +FOR ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR LOSS AS +A RESULT OF MODIFYING, DISTRIBUTING, OR OTHERWISE USING PYTHON, +OR ANY DERIVATIVE THEREOF, EVEN IF ADVISED OF THE POSSIBILITY THEREOF. + +6. This License Agreement will automatically terminate upon a material +breach of its terms and conditions. + +7. Nothing in this License Agreement shall be deemed to create any +relationship of agency, partnership, or joint venture between PSF and +Licensee. This License Agreement does not grant permission to use PSF +trademarks or trade name in a trademark sense to endorse or promote +products or services of Licensee, or any third party. + +8. By copying, installing or otherwise using Python, Licensee +agrees to be bound by the terms and conditions of this License +Agreement. + + +BEOPEN.COM LICENSE AGREEMENT FOR PYTHON 2.0 +------------------------------------------- + +BEOPEN PYTHON OPEN SOURCE LICENSE AGREEMENT VERSION 1 + +1. This LICENSE AGREEMENT is between BeOpen.com ("BeOpen"), having an +office at 160 Saratoga Avenue, Santa Clara, CA 95051, and the +Individual or Organization ("Licensee") accessing and otherwise using +this software in source or binary form and its associated +documentation ("the Software"). + +2. Subject to the terms and conditions of this BeOpen Python License +Agreement, BeOpen hereby grants Licensee a non-exclusive, +royalty-free, world-wide license to reproduce, analyze, test, perform +and/or display publicly, prepare derivative works, distribute, and +otherwise use the Software alone or in any derivative version, +provided, however, that the BeOpen Python License is retained in the +Software, alone or in any derivative version prepared by Licensee. + +3. BeOpen is making the Software available to Licensee on an "AS IS" +basis. BEOPEN MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR +IMPLIED. BY WAY OF EXAMPLE, BUT NOT LIMITATION, BEOPEN MAKES NO AND +DISCLAIMS ANY REPRESENTATION OR WARRANTY OF MERCHANTABILITY OR FITNESS +FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF THE SOFTWARE WILL NOT +INFRINGE ANY THIRD PARTY RIGHTS. + +4. BEOPEN SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF THE +SOFTWARE FOR ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR LOSS +AS A RESULT OF USING, MODIFYING OR DISTRIBUTING THE SOFTWARE, OR ANY +DERIVATIVE THEREOF, EVEN IF ADVISED OF THE POSSIBILITY THEREOF. + +5. This License Agreement will automatically terminate upon a material +breach of its terms and conditions. + +6. This License Agreement shall be governed by and interpreted in all +respects by the law of the State of California, excluding conflict of +law provisions. Nothing in this License Agreement shall be deemed to +create any relationship of agency, partnership, or joint venture +between BeOpen and Licensee. This License Agreement does not grant +permission to use BeOpen trademarks or trade names in a trademark +sense to endorse or promote products or services of Licensee, or any +third party. As an exception, the "BeOpen Python" logos available at +http://www.pythonlabs.com/logos.html may be used according to the +permissions granted on that web page. + +7. By copying, installing or otherwise using the software, Licensee +agrees to be bound by the terms and conditions of this License +Agreement. + + +CNRI LICENSE AGREEMENT FOR PYTHON 1.6.1 +--------------------------------------- + +1. This LICENSE AGREEMENT is between the Corporation for National +Research Initiatives, having an office at 1895 Preston White Drive, +Reston, VA 20191 ("CNRI"), and the Individual or Organization +("Licensee") accessing and otherwise using Python 1.6.1 software in +source or binary form and its associated documentation. + +2. Subject to the terms and conditions of this License Agreement, CNRI +hereby grants Licensee a nonexclusive, royalty-free, world-wide +license to reproduce, analyze, test, perform and/or display publicly, +prepare derivative works, distribute, and otherwise use Python 1.6.1 +alone or in any derivative version, provided, however, that CNRI's +License Agreement and CNRI's notice of copyright, i.e., "Copyright (c) +1995-2001 Corporation for National Research Initiatives; All Rights +Reserved" are retained in Python 1.6.1 alone or in any derivative +version prepared by Licensee. Alternately, in lieu of CNRI's License +Agreement, Licensee may substitute the following text (omitting the +quotes): "Python 1.6.1 is made available subject to the terms and +conditions in CNRI's License Agreement. This Agreement together with +Python 1.6.1 may be located on the Internet using the following +unique, persistent identifier (known as a handle): 1895.22/1013. This +Agreement may also be obtained from a proxy server on the Internet +using the following URL: http://hdl.handle.net/1895.22/1013". + +3. In the event Licensee prepares a derivative work that is based on +or incorporates Python 1.6.1 or any part thereof, and wants to make +the derivative work available to others as provided herein, then +Licensee hereby agrees to include in any such work a brief summary of +the changes made to Python 1.6.1. + +4. CNRI is making Python 1.6.1 available to Licensee on an "AS IS" +basis. CNRI MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR +IMPLIED. BY WAY OF EXAMPLE, BUT NOT LIMITATION, CNRI MAKES NO AND +DISCLAIMS ANY REPRESENTATION OR WARRANTY OF MERCHANTABILITY OR FITNESS +FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF PYTHON 1.6.1 WILL NOT +INFRINGE ANY THIRD PARTY RIGHTS. + +5. CNRI SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF PYTHON +1.6.1 FOR ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR LOSS AS +A RESULT OF MODIFYING, DISTRIBUTING, OR OTHERWISE USING PYTHON 1.6.1, +OR ANY DERIVATIVE THEREOF, EVEN IF ADVISED OF THE POSSIBILITY THEREOF. + +6. This License Agreement will automatically terminate upon a material +breach of its terms and conditions. + +7. This License Agreement shall be governed by the federal +intellectual property law of the United States, including without +limitation the federal copyright law, and, to the extent such +U.S. federal law does not apply, by the law of the Commonwealth of +Virginia, excluding Virginia's conflict of law provisions. +Notwithstanding the foregoing, with regard to derivative works based +on Python 1.6.1 that incorporate non-separable material that was +previously distributed under the GNU General Public License (GPL), the +law of the Commonwealth of Virginia shall govern this License +Agreement only as to issues arising under or with respect to +Paragraphs 4, 5, and 7 of this License Agreement. Nothing in this +License Agreement shall be deemed to create any relationship of +agency, partnership, or joint venture between CNRI and Licensee. This +License Agreement does not grant permission to use CNRI trademarks or +trade name in a trademark sense to endorse or promote products or +services of Licensee, or any third party. + +8. By clicking on the "ACCEPT" button where indicated, or by copying, +installing or otherwise using Python 1.6.1, Licensee agrees to be +bound by the terms and conditions of this License Agreement. + + ACCEPT + + +CWI LICENSE AGREEMENT FOR PYTHON 0.9.0 THROUGH 1.2 +-------------------------------------------------- + +Copyright (c) 1991 - 1995, Stichting Mathematisch Centrum Amsterdam, +The Netherlands. All rights reserved. + +Permission to use, copy, modify, and distribute this software and its +documentation for any purpose and without fee is hereby granted, +provided that the above copyright notice appear in all copies and that +both that copyright notice and this permission notice appear in +supporting documentation, and that the name of Stichting Mathematisch +Centrum or CWI not be used in advertising or publicity pertaining to +distribution of the software without specific, written prior +permission. + +STICHTING MATHEMATISCH CENTRUM DISCLAIMS ALL WARRANTIES WITH REGARD TO +THIS SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND +FITNESS, IN NO EVENT SHALL STICHTING MATHEMATISCH CENTRUM BE LIABLE +FOR ANY SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES +WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN +ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT +OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. diff --git a/scripts/argparse.py b/scripts/argparse.py new file mode 100644 index XXXXXXX..XXXXXXX --- /dev/null +++ b/scripts/argparse.py @@ -XXX,XX +XXX,XX @@ +# This is a local copy of the standard library argparse module taken from PyPI. +# It is licensed under the Python Software Foundation License. This is a +# fallback for Python 2.6 which does not include this module. Python 2.7+ and +# 3+ will never load this module because built-in modules are loaded before +# anything in sys.path. +# +# If your script is not located in the same directory as this file, import it +# like this: +# +# import os +# import sys +# sys.path.append(os.path.join(os.path.dirname(__file__), ..., 'scripts')) +# import argparse + +# Author: Steven J. Bethard <steven.bethard@gmail.com>. +# Maintainer: Thomas Waldmann <tw@waldmann-edv.de> + +"""Command-line parsing library + +This module is an optparse-inspired command-line parsing library that: + + - handles both optional and positional arguments + - produces highly informative usage messages + - supports parsers that dispatch to sub-parsers + +The following is a simple usage example that sums integers from the +command-line and writes the result to a file:: + + parser = argparse.ArgumentParser( + description='sum the integers at the command line') + parser.add_argument( + 'integers', metavar='int', nargs='+', type=int, + help='an integer to be summed') + parser.add_argument( + '--log', default=sys.stdout, type=argparse.FileType('w'), + help='the file where the sum should be written') + args = parser.parse_args() + args.log.write('%s' % sum(args.integers)) + args.log.close() + +The module contains the following public classes: + + - ArgumentParser -- The main entry point for command-line parsing. As the + example above shows, the add_argument() method is used to populate + the parser with actions for optional and positional arguments. Then + the parse_args() method is invoked to convert the args at the + command-line into an object with attributes. + + - ArgumentError -- The exception raised by ArgumentParser objects when + there are errors with the parser's actions. Errors raised while + parsing the command-line are caught by ArgumentParser and emitted + as command-line messages. + + - FileType -- A factory for defining types of files to be created. As the + example above shows, instances of FileType are typically passed as + the type= argument of add_argument() calls. + + - Action -- The base class for parser actions. Typically actions are + selected by passing strings like 'store_true' or 'append_const' to + the action= argument of add_argument(). However, for greater + customization of ArgumentParser actions, subclasses of Action may + be defined and passed as the action= argument. + + - HelpFormatter, RawDescriptionHelpFormatter, RawTextHelpFormatter, + ArgumentDefaultsHelpFormatter -- Formatter classes which + may be passed as the formatter_class= argument to the + ArgumentParser constructor. HelpFormatter is the default, + RawDescriptionHelpFormatter and RawTextHelpFormatter tell the parser + not to change the formatting for help text, and + ArgumentDefaultsHelpFormatter adds information about argument defaults + to the help. + +All other classes in this module are considered implementation details. +(Also note that HelpFormatter and RawDescriptionHelpFormatter are only +considered public as object names -- the API of the formatter objects is +still considered an implementation detail.) +""" + +__version__ = '1.4.0' # we use our own version number independant of the + # one in stdlib and we release this on pypi. + +__external_lib__ = True # to make sure the tests really test THIS lib, + # not the builtin one in Python stdlib + +__all__ = [ + 'ArgumentParser', + 'ArgumentError', + 'ArgumentTypeError', + 'FileType', + 'HelpFormatter', + 'ArgumentDefaultsHelpFormatter', + 'RawDescriptionHelpFormatter', + 'RawTextHelpFormatter', + 'Namespace', + 'Action', + 'ONE_OR_MORE', + 'OPTIONAL', + 'PARSER', + 'REMAINDER', + 'SUPPRESS', + 'ZERO_OR_MORE', +] + + +import copy as _copy +import os as _os +import re as _re +import sys as _sys +import textwrap as _textwrap + +from gettext import gettext as _ + +try: + set +except NameError: + # for python < 2.4 compatibility (sets module is there since 2.3): + from sets import Set as set + +try: + basestring +except NameError: + basestring = str + +try: + sorted +except NameError: + # for python < 2.4 compatibility: + def sorted(iterable, reverse=False): + result = list(iterable) + result.sort() + if reverse: + result.reverse() + return result + + +def _callable(obj): + return hasattr(obj, '__call__') or hasattr(obj, '__bases__') + + +SUPPRESS = '==SUPPRESS==' + +OPTIONAL = '?' +ZERO_OR_MORE = '*' +ONE_OR_MORE = '+' +PARSER = 'A...' +REMAINDER = '...' +_UNRECOGNIZED_ARGS_ATTR = '_unrecognized_args' + +# ============================= +# Utility functions and classes +# ============================= + +class _AttributeHolder(object): + """Abstract base class that provides __repr__. + + The __repr__ method returns a string in the format:: + ClassName(attr=name, attr=name, ...) + The attributes are determined either by a class-level attribute, + '_kwarg_names', or by inspecting the instance __dict__. + """ + + def __repr__(self): + type_name = type(self).__name__ + arg_strings = [] + for arg in self._get_args(): + arg_strings.append(repr(arg)) + for name, value in self._get_kwargs(): + arg_strings.append('%s=%r' % (name, value)) + return '%s(%s)' % (type_name, ', '.join(arg_strings)) + + def _get_kwargs(self): + return sorted(self.__dict__.items()) + + def _get_args(self): + return [] + + +def _ensure_value(namespace, name, value): + if getattr(namespace, name, None) is None: + setattr(namespace, name, value) + return getattr(namespace, name) + + +# =============== +# Formatting Help +# =============== + +class HelpFormatter(object): + """Formatter for generating usage messages and argument help strings. + + Only the name of this class is considered a public API. All the methods + provided by the class are considered an implementation detail. + """ + + def __init__(self, + prog, + indent_increment=2, + max_help_position=24, + width=None): + + # default setting for width + if width is None: + try: + width = int(_os.environ['COLUMNS']) + except (KeyError, ValueError): + width = 80 + width -= 2 + + self._prog = prog + self._indent_increment = indent_increment + self._max_help_position = max_help_position + self._width = width + + self._current_indent = 0 + self._level = 0 + self._action_max_length = 0 + + self._root_section = self._Section(self, None) + self._current_section = self._root_section + + self._whitespace_matcher = _re.compile(r'\s+') + self._long_break_matcher = _re.compile(r'\n\n\n+') + + # =============================== + # Section and indentation methods + # =============================== + def _indent(self): + self._current_indent += self._indent_increment + self._level += 1 + + def _dedent(self): + self._current_indent -= self._indent_increment + assert self._current_indent >= 0, 'Indent decreased below 0.' + self._level -= 1 + + class _Section(object): + + def __init__(self, formatter, parent, heading=None): + self.formatter = formatter + self.parent = parent + self.heading = heading + self.items = [] + + def format_help(self): + # format the indented section + if self.parent is not None: + self.formatter._indent() + join = self.formatter._join_parts + for func, args in self.items: + func(*args) + item_help = join([func(*args) for func, args in self.items]) + if self.parent is not None: + self.formatter._dedent() + + # return nothing if the section was empty + if not item_help: + return '' + + # add the heading if the section was non-empty + if self.heading is not SUPPRESS and self.heading is not None: + current_indent = self.formatter._current_indent + heading = '%*s%s:\n' % (current_indent, '', self.heading) + else: + heading = '' + + # join the section-initial newline, the heading and the help + return join(['\n', heading, item_help, '\n']) + + def _add_item(self, func, args): + self._current_section.items.append((func, args)) + + # ======================== + # Message building methods + # ======================== + def start_section(self, heading): + self._indent() + section = self._Section(self, self._current_section, heading) + self._add_item(section.format_help, []) + self._current_section = section + + def end_section(self): + self._current_section = self._current_section.parent + self._dedent() + + def add_text(self, text): + if text is not SUPPRESS and text is not None: + self._add_item(self._format_text, [text]) + + def add_usage(self, usage, actions, groups, prefix=None): + if usage is not SUPPRESS: + args = usage, actions, groups, prefix + self._add_item(self._format_usage, args) + + def add_argument(self, action): + if action.help is not SUPPRESS: + + # find all invocations + get_invocation = self._format_action_invocation + invocations = [get_invocation(action)] + for subaction in self._iter_indented_subactions(action): + invocations.append(get_invocation(subaction)) + + # update the maximum item length + invocation_length = max([len(s) for s in invocations]) + action_length = invocation_length + self._current_indent + self._action_max_length = max(self._action_max_length, + action_length) + + # add the item to the list + self._add_item(self._format_action, [action]) + + def add_arguments(self, actions): + for action in actions: + self.add_argument(action) + + # ======================= + # Help-formatting methods + # ======================= + def format_help(self): + help = self._root_section.format_help() + if help: + help = self._long_break_matcher.sub('\n\n', help) + help = help.strip('\n') + '\n' + return help + + def _join_parts(self, part_strings): + return ''.join([part + for part in part_strings + if part and part is not SUPPRESS]) + + def _format_usage(self, usage, actions, groups, prefix): + if prefix is None: + prefix = _('usage: ') + + # if usage is specified, use that + if usage is not None: + usage = usage % dict(prog=self._prog) + + # if no optionals or positionals are available, usage is just prog + elif usage is None and not actions: + usage = '%(prog)s' % dict(prog=self._prog) + + # if optionals and positionals are available, calculate usage + elif usage is None: + prog = '%(prog)s' % dict(prog=self._prog) + + # split optionals from positionals + optionals = [] + positionals = [] + for action in actions: + if action.option_strings: + optionals.append(action) + else: + positionals.append(action) + + # build full usage string + format = self._format_actions_usage + action_usage = format(optionals + positionals, groups) + usage = ' '.join([s for s in [prog, action_usage] if s]) + + # wrap the usage parts if it's too long + text_width = self._width - self._current_indent + if len(prefix) + len(usage) > text_width: + + # break usage into wrappable parts + part_regexp = r'\(.*?\)+|\[.*?\]+|\S+' + opt_usage = format(optionals, groups) + pos_usage = format(positionals, groups) + opt_parts = _re.findall(part_regexp, opt_usage) + pos_parts = _re.findall(part_regexp, pos_usage) + assert ' '.join(opt_parts) == opt_usage + assert ' '.join(pos_parts) == pos_usage + + # helper for wrapping lines + def get_lines(parts, indent, prefix=None): + lines = [] + line = [] + if prefix is not None: + line_len = len(prefix) - 1 + else: + line_len = len(indent) - 1 + for part in parts: + if line_len + 1 + len(part) > text_width: + lines.append(indent + ' '.join(line)) + line = [] + line_len = len(indent) - 1 + line.append(part) + line_len += len(part) + 1 + if line: + lines.append(indent + ' '.join(line)) + if prefix is not None: + lines[0] = lines[0][len(indent):] + return lines + + # if prog is short, follow it with optionals or positionals + if len(prefix) + len(prog) <= 0.75 * text_width: + indent = ' ' * (len(prefix) + len(prog) + 1) + if opt_parts: + lines = get_lines([prog] + opt_parts, indent, prefix) + lines.extend(get_lines(pos_parts, indent)) + elif pos_parts: + lines = get_lines([prog] + pos_parts, indent, prefix) + else: + lines = [prog] + + # if prog is long, put it on its own line + else: + indent = ' ' * len(prefix) + parts = opt_parts + pos_parts + lines = get_lines(parts, indent) + if len(lines) > 1: + lines = [] + lines.extend(get_lines(opt_parts, indent)) + lines.extend(get_lines(pos_parts, indent)) + lines = [prog] + lines + + # join lines into usage + usage = '\n'.join(lines) + + # prefix with 'usage:' + return '%s%s\n\n' % (prefix, usage) + + def _format_actions_usage(self, actions, groups): + # find group indices and identify actions in groups + group_actions = set() + inserts = {} + for group in groups: + try: + start = actions.index(group._group_actions[0]) + except ValueError: + continue + else: + end = start + len(group._group_actions) + if actions[start:end] == group._group_actions: + for action in group._group_actions: + group_actions.add(action) + if not group.required: + if start in inserts: + inserts[start] += ' [' + else: + inserts[start] = '[' + inserts[end] = ']' + else: + if start in inserts: + inserts[start] += ' (' + else: + inserts[start] = '(' + inserts[end] = ')' + for i in range(start + 1, end): + inserts[i] = '|' + + # collect all actions format strings + parts = [] + for i, action in enumerate(actions): + + # suppressed arguments are marked with None + # remove | separators for suppressed arguments + if action.help is SUPPRESS: + parts.append(None) + if inserts.get(i) == '|': + inserts.pop(i) + elif inserts.get(i + 1) == '|': + inserts.pop(i + 1) + + # produce all arg strings + elif not action.option_strings: + part = self._format_args(action, action.dest) + + # if it's in a group, strip the outer [] + if action in group_actions: + if part[0] == '[' and part[-1] == ']': + part = part[1:-1] + + # add the action string to the list + parts.append(part) + + # produce the first way to invoke the option in brackets + else: + option_string = action.option_strings[0] + + # if the Optional doesn't take a value, format is: + # -s or --long + if action.nargs == 0: + part = '%s' % option_string + + # if the Optional takes a value, format is: + # -s ARGS or --long ARGS + else: + default = action.dest.upper() + args_string = self._format_args(action, default) + part = '%s %s' % (option_string, args_string) + + # make it look optional if it's not required or in a group + if not action.required and action not in group_actions: + part = '[%s]' % part + + # add the action string to the list + parts.append(part) + + # insert things at the necessary indices + for i in sorted(inserts, reverse=True): + parts[i:i] = [inserts[i]] + + # join all the action items with spaces + text = ' '.join([item for item in parts if item is not None]) + + # clean up separators for mutually exclusive groups + open = r'[\[(]' + close = r'[\])]' + text = _re.sub(r'(%s) ' % open, r'\1', text) + text = _re.sub(r' (%s)' % close, r'\1', text) + text = _re.sub(r'%s *%s' % (open, close), r'', text) + text = _re.sub(r'\(([^|]*)\)', r'\1', text) + text = text.strip() + + # return the text + return text + + def _format_text(self, text): + if '%(prog)' in text: + text = text % dict(prog=self._prog) + text_width = self._width - self._current_indent + indent = ' ' * self._current_indent + return self._fill_text(text, text_width, indent) + '\n\n' + + def _format_action(self, action): + # determine the required width and the entry label + help_position = min(self._action_max_length + 2, + self._max_help_position) + help_width = self._width - help_position + action_width = help_position - self._current_indent - 2 + action_header = self._format_action_invocation(action) + + # ho nelp; start on same line and add a final newline + if not action.help: + tup = self._current_indent, '', action_header + action_header = '%*s%s\n' % tup + + # short action name; start on the same line and pad two spaces + elif len(action_header) <= action_width: + tup = self._current_indent, '', action_width, action_header + action_header = '%*s%-*s ' % tup + indent_first = 0 + + # long action name; start on the next line + else: + tup = self._current_indent, '', action_header + action_header = '%*s%s\n' % tup + indent_first = help_position + + # collect the pieces of the action help + parts = [action_header] + + # if there was help for the action, add lines of help text + if action.help: + help_text = self._expand_help(action) + help_lines = self._split_lines(help_text, help_width) + parts.append('%*s%s\n' % (indent_first, '', help_lines[0])) + for line in help_lines[1:]: + parts.append('%*s%s\n' % (help_position, '', line)) + + # or add a newline if the description doesn't end with one + elif not action_header.endswith('\n'): + parts.append('\n') + + # if there are any sub-actions, add their help as well + for subaction in self._iter_indented_subactions(action): + parts.append(self._format_action(subaction)) + + # return a single string + return self._join_parts(parts) + + def _format_action_invocation(self, action): + if not action.option_strings: + metavar, = self._metavar_formatter(action, action.dest)(1) + return metavar + + else: + parts = [] + + # if the Optional doesn't take a value, format is: + # -s, --long + if action.nargs == 0: + parts.extend(action.option_strings) + + # if the Optional takes a value, format is: + # -s ARGS, --long ARGS + else: + default = action.dest.upper() + args_string = self._format_args(action, default) + for option_string in action.option_strings: + parts.append('%s %s' % (option_string, args_string)) + + return ', '.join(parts) + + def _metavar_formatter(self, action, default_metavar): + if action.metavar is not None: + result = action.metavar + elif action.choices is not None: + choice_strs = [str(choice) for choice in action.choices] + result = '{%s}' % ','.join(choice_strs) + else: + result = default_metavar + + def format(tuple_size): + if isinstance(result, tuple): + return result + else: + return (result, ) * tuple_size + return format + + def _format_args(self, action, default_metavar): + get_metavar = self._metavar_formatter(action, default_metavar) + if action.nargs is None: + result = '%s' % get_metavar(1) + elif action.nargs == OPTIONAL: + result = '[%s]' % get_metavar(1) + elif action.nargs == ZERO_OR_MORE: + result = '[%s [%s ...]]' % get_metavar(2) + elif action.nargs == ONE_OR_MORE: + result = '%s [%s ...]' % get_metavar(2) + elif action.nargs == REMAINDER: + result = '...' + elif action.nargs == PARSER: + result = '%s ...' % get_metavar(1) + else: + formats = ['%s' for _ in range(action.nargs)] + result = ' '.join(formats) % get_metavar(action.nargs) + return result + + def _expand_help(self, action): + params = dict(vars(action), prog=self._prog) + for name in list(params): + if params[name] is SUPPRESS: + del params[name] + for name in list(params): + if hasattr(params[name], '__name__'): + params[name] = params[name].__name__ + if params.get('choices') is not None: + choices_str = ', '.join([str(c) for c in params['choices']]) + params['choices'] = choices_str + return self._get_help_string(action) % params + + def _iter_indented_subactions(self, action): + try: + get_subactions = action._get_subactions + except AttributeError: + pass + else: + self._indent() + for subaction in get_subactions(): + yield subaction + self._dedent() + + def _split_lines(self, text, width): + text = self._whitespace_matcher.sub(' ', text).strip() + return _textwrap.wrap(text, width) + + def _fill_text(self, text, width, indent): + text = self._whitespace_matcher.sub(' ', text).strip() + return _textwrap.fill(text, width, initial_indent=indent, + subsequent_indent=indent) + + def _get_help_string(self, action): + return action.help + + +class RawDescriptionHelpFormatter(HelpFormatter): + """Help message formatter which retains any formatting in descriptions. + + Only the name of this class is considered a public API. All the methods + provided by the class are considered an implementation detail. + """ + + def _fill_text(self, text, width, indent): + return ''.join([indent + line for line in text.splitlines(True)]) + + +class RawTextHelpFormatter(RawDescriptionHelpFormatter): + """Help message formatter which retains formatting of all help text. + + Only the name of this class is considered a public API. All the methods + provided by the class are considered an implementation detail. + """ + + def _split_lines(self, text, width): + return text.splitlines() + + +class ArgumentDefaultsHelpFormatter(HelpFormatter): + """Help message formatter which adds default values to argument help. + + Only the name of this class is considered a public API. All the methods + provided by the class are considered an implementation detail. + """ + + def _get_help_string(self, action): + help = action.help + if '%(default)' not in action.help: + if action.default is not SUPPRESS: + defaulting_nargs = [OPTIONAL, ZERO_OR_MORE] + if action.option_strings or action.nargs in defaulting_nargs: + help += ' (default: %(default)s)' + return help + + +# ===================== +# Options and Arguments +# ===================== + +def _get_action_name(argument): + if argument is None: + return None + elif argument.option_strings: + return '/'.join(argument.option_strings) + elif argument.metavar not in (None, SUPPRESS): + return argument.metavar + elif argument.dest not in (None, SUPPRESS): + return argument.dest + else: + return None + + +class ArgumentError(Exception): + """An error from creating or using an argument (optional or positional). + + The string value of this exception is the message, augmented with + information about the argument that caused it. + """ + + def __init__(self, argument, message): + self.argument_name = _get_action_name(argument) + self.message = message + + def __str__(self): + if self.argument_name is None: + format = '%(message)s' + else: + format = 'argument %(argument_name)s: %(message)s' + return format % dict(message=self.message, + argument_name=self.argument_name) + + +class ArgumentTypeError(Exception): + """An error from trying to convert a command line string to a type.""" + pass + + +# ============== +# Action classes +# ============== + +class Action(_AttributeHolder): + """Information about how to convert command line strings to Python objects. + + Action objects are used by an ArgumentParser to represent the information + needed to parse a single argument from one or more strings from the + command line. The keyword arguments to the Action constructor are also + all attributes of Action instances. + + Keyword Arguments: + + - option_strings -- A list of command-line option strings which + should be associated with this action. + + - dest -- The name of the attribute to hold the created object(s) + + - nargs -- The number of command-line arguments that should be + consumed. By default, one argument will be consumed and a single + value will be produced. Other values include: + - N (an integer) consumes N arguments (and produces a list) + - '?' consumes zero or one arguments + - '*' consumes zero or more arguments (and produces a list) + - '+' consumes one or more arguments (and produces a list) + Note that the difference between the default and nargs=1 is that + with the default, a single value will be produced, while with + nargs=1, a list containing a single value will be produced. + + - const -- The value to be produced if the option is specified and the + option uses an action that takes no values. + + - default -- The value to be produced if the option is not specified. + + - type -- The type which the command-line arguments should be converted + to, should be one of 'string', 'int', 'float', 'complex' or a + callable object that accepts a single string argument. If None, + 'string' is assumed. + + - choices -- A container of values that should be allowed. If not None, + after a command-line argument has been converted to the appropriate + type, an exception will be raised if it is not a member of this + collection. + + - required -- True if the action must always be specified at the + command line. This is only meaningful for optional command-line + arguments. + + - help -- The help string describing the argument. + + - metavar -- The name to be used for the option's argument with the + help string. If None, the 'dest' value will be used as the name. + """ + + def __init__(self, + option_strings, + dest, + nargs=None, + const=None, + default=None, + type=None, + choices=None, + required=False, + help=None, + metavar=None): + self.option_strings = option_strings + self.dest = dest + self.nargs = nargs + self.const = const + self.default = default + self.type = type + self.choices = choices + self.required = required + self.help = help + self.metavar = metavar + + def _get_kwargs(self): + names = [ + 'option_strings', + 'dest', + 'nargs', + 'const', + 'default', + 'type', + 'choices', + 'help', + 'metavar', + ] + return [(name, getattr(self, name)) for name in names] + + def __call__(self, parser, namespace, values, option_string=None): + raise NotImplementedError(_('.__call__() not defined')) + + +class _StoreAction(Action): + + def __init__(self, + option_strings, + dest, + nargs=None, + const=None, + default=None, + type=None, + choices=None, + required=False, + help=None, + metavar=None): + if nargs == 0: + raise ValueError('nargs for store actions must be > 0; if you ' + 'have nothing to store, actions such as store ' + 'true or store const may be more appropriate') + if const is not None and nargs != OPTIONAL: + raise ValueError('nargs must be %r to supply const' % OPTIONAL) + super(_StoreAction, self).__init__( + option_strings=option_strings, + dest=dest, + nargs=nargs, + const=const, + default=default, + type=type, + choices=choices, + required=required, + help=help, + metavar=metavar) + + def __call__(self, parser, namespace, values, option_string=None): + setattr(namespace, self.dest, values) + + +class _StoreConstAction(Action): + + def __init__(self, + option_strings, + dest, + const, + default=None, + required=False, + help=None, + metavar=None): + super(_StoreConstAction, self).__init__( + option_strings=option_strings, + dest=dest, + nargs=0, + const=const, + default=default, + required=required, + help=help) + + def __call__(self, parser, namespace, values, option_string=None): + setattr(namespace, self.dest, self.const) + + +class _StoreTrueAction(_StoreConstAction): + + def __init__(self, + option_strings, + dest, + default=False, + required=False, + help=None): + super(_StoreTrueAction, self).__init__( + option_strings=option_strings, + dest=dest, + const=True, + default=default, + required=required, + help=help) + + +class _StoreFalseAction(_StoreConstAction): + + def __init__(self, + option_strings, + dest, + default=True, + required=False, + help=None): + super(_StoreFalseAction, self).__init__( + option_strings=option_strings, + dest=dest, + const=False, + default=default, + required=required, + help=help) + + +class _AppendAction(Action): + + def __init__(self, + option_strings, + dest, + nargs=None, + const=None, + default=None, + type=None, + choices=None, + required=False, + help=None, + metavar=None): + if nargs == 0: + raise ValueError('nargs for append actions must be > 0; if arg ' + 'strings are not supplying the value to append, ' + 'the append const action may be more appropriate') + if const is not None and nargs != OPTIONAL: + raise ValueError('nargs must be %r to supply const' % OPTIONAL) + super(_AppendAction, self).__init__( + option_strings=option_strings, + dest=dest, + nargs=nargs, + const=const, + default=default, + type=type, + choices=choices, + required=required, + help=help, + metavar=metavar) + + def __call__(self, parser, namespace, values, option_string=None): + items = _copy.copy(_ensure_value(namespace, self.dest, [])) + items.append(values) + setattr(namespace, self.dest, items) + + +class _AppendConstAction(Action): + + def __init__(self, + option_strings, + dest, + const, + default=None, + required=False, + help=None, + metavar=None): + super(_AppendConstAction, self).__init__( + option_strings=option_strings, + dest=dest, + nargs=0, + const=const, + default=default, + required=required, + help=help, + metavar=metavar) + + def __call__(self, parser, namespace, values, option_string=None): + items = _copy.copy(_ensure_value(namespace, self.dest, [])) + items.append(self.const) + setattr(namespace, self.dest, items) + + +class _CountAction(Action): + + def __init__(self, + option_strings, + dest, + default=None, + required=False, + help=None): + super(_CountAction, self).__init__( + option_strings=option_strings, + dest=dest, + nargs=0, + default=default, + required=required, + help=help) + + def __call__(self, parser, namespace, values, option_string=None): + new_count = _ensure_value(namespace, self.dest, 0) + 1 + setattr(namespace, self.dest, new_count) + + +class _HelpAction(Action): + + def __init__(self, + option_strings, + dest=SUPPRESS, + default=SUPPRESS, + help=None): + super(_HelpAction, self).__init__( + option_strings=option_strings, + dest=dest, + default=default, + nargs=0, + help=help) + + def __call__(self, parser, namespace, values, option_string=None): + parser.print_help() + parser.exit() + + +class _VersionAction(Action): + + def __init__(self, + option_strings, + version=None, + dest=SUPPRESS, + default=SUPPRESS, + help="show program's version number and exit"): + super(_VersionAction, self).__init__( + option_strings=option_strings, + dest=dest, + default=default, + nargs=0, + help=help) + self.version = version + + def __call__(self, parser, namespace, values, option_string=None): + version = self.version + if version is None: + version = parser.version + formatter = parser._get_formatter() + formatter.add_text(version) + parser.exit(message=formatter.format_help()) + + +class _SubParsersAction(Action): + + class _ChoicesPseudoAction(Action): + + def __init__(self, name, aliases, help): + metavar = dest = name + if aliases: + metavar += ' (%s)' % ', '.join(aliases) + sup = super(_SubParsersAction._ChoicesPseudoAction, self) + sup.__init__(option_strings=[], dest=dest, help=help, + metavar=metavar) + + def __init__(self, + option_strings, + prog, + parser_class, + dest=SUPPRESS, + help=None, + metavar=None): + + self._prog_prefix = prog + self._parser_class = parser_class + self._name_parser_map = {} + self._choices_actions = [] + + super(_SubParsersAction, self).__init__( + option_strings=option_strings, + dest=dest, + nargs=PARSER, + choices=self._name_parser_map, + help=help, + metavar=metavar) + + def add_parser(self, name, **kwargs): + # set prog from the existing prefix + if kwargs.get('prog') is None: + kwargs['prog'] = '%s %s' % (self._prog_prefix, name) + + aliases = kwargs.pop('aliases', ()) + + # create a pseudo-action to hold the choice help + if 'help' in kwargs: + help = kwargs.pop('help') + choice_action = self._ChoicesPseudoAction(name, aliases, help) + self._choices_actions.append(choice_action) + + # create the parser and add it to the map + parser = self._parser_class(**kwargs) + self._name_parser_map[name] = parser + + # make parser available under aliases also + for alias in aliases: + self._name_parser_map[alias] = parser + + return parser + + def _get_subactions(self): + return self._choices_actions + + def __call__(self, parser, namespace, values, option_string=None): + parser_name = values[0] + arg_strings = values[1:] + + # set the parser name if requested + if self.dest is not SUPPRESS: + setattr(namespace, self.dest, parser_name) + + # select the parser + try: + parser = self._name_parser_map[parser_name] + except KeyError: + tup = parser_name, ', '.join(self._name_parser_map) + msg = _('unknown parser %r (choices: %s)' % tup) + raise ArgumentError(self, msg) + + # parse all the remaining options into the namespace + # store any unrecognized options on the object, so that the top + # level parser can decide what to do with them + namespace, arg_strings = parser.parse_known_args(arg_strings, namespace) + if arg_strings: + vars(namespace).setdefault(_UNRECOGNIZED_ARGS_ATTR, []) + getattr(namespace, _UNRECOGNIZED_ARGS_ATTR).extend(arg_strings) + + +# ============== +# Type classes +# ============== + +class FileType(object): + """Factory for creating file object types + + Instances of FileType are typically passed as type= arguments to the + ArgumentParser add_argument() method. + + Keyword Arguments: + - mode -- A string indicating how the file is to be opened. Accepts the + same values as the builtin open() function. + - bufsize -- The file's desired buffer size. Accepts the same values as + the builtin open() function. + """ + + def __init__(self, mode='r', bufsize=None): + self._mode = mode + self._bufsize = bufsize + + def __call__(self, string): + # the special argument "-" means sys.std{in,out} + if string == '-': + if 'r' in self._mode: + return _sys.stdin + elif 'w' in self._mode: + return _sys.stdout + else: + msg = _('argument "-" with mode %r' % self._mode) + raise ValueError(msg) + + try: + # all other arguments are used as file names + if self._bufsize: + return open(string, self._mode, self._bufsize) + else: + return open(string, self._mode) + except IOError: + err = _sys.exc_info()[1] + message = _("can't open '%s': %s") + raise ArgumentTypeError(message % (string, err)) + + def __repr__(self): + args = [self._mode, self._bufsize] + args_str = ', '.join([repr(arg) for arg in args if arg is not None]) + return '%s(%s)' % (type(self).__name__, args_str) + +# =========================== +# Optional and Positional Parsing +# =========================== + +class Namespace(_AttributeHolder): + """Simple object for storing attributes. + + Implements equality by attribute names and values, and provides a simple + string representation. + """ + + def __init__(self, **kwargs): + for name in kwargs: + setattr(self, name, kwargs[name]) + + __hash__ = None + + def __eq__(self, other): + return vars(self) == vars(other) + + def __ne__(self, other): + return not (self == other) + + def __contains__(self, key): + return key in self.__dict__ + + +class _ActionsContainer(object): + + def __init__(self, + description, + prefix_chars, + argument_default, + conflict_handler): + super(_ActionsContainer, self).__init__() + + self.description = description + self.argument_default = argument_default + self.prefix_chars = prefix_chars + self.conflict_handler = conflict_handler + + # set up registries + self._registries = {} + + # register actions + self.register('action', None, _StoreAction) + self.register('action', 'store', _StoreAction) + self.register('action', 'store_const', _StoreConstAction) + self.register('action', 'store_true', _StoreTrueAction) + self.register('action', 'store_false', _StoreFalseAction) + self.register('action', 'append', _AppendAction) + self.register('action', 'append_const', _AppendConstAction) + self.register('action', 'count', _CountAction) + self.register('action', 'help', _HelpAction) + self.register('action', 'version', _VersionAction) + self.register('action', 'parsers', _SubParsersAction) + + # raise an exception if the conflict handler is invalid + self._get_handler() + + # action storage + self._actions = [] + self._option_string_actions = {} + + # groups + self._action_groups = [] + self._mutually_exclusive_groups = [] + + # defaults storage + self._defaults = {} + + # determines whether an "option" looks like a negative number + self._negative_number_matcher = _re.compile(r'^-\d+$|^-\d*\.\d+$') + + # whether or not there are any optionals that look like negative + # numbers -- uses a list so it can be shared and edited + self._has_negative_number_optionals = [] + + # ==================== + # Registration methods + # ==================== + def register(self, registry_name, value, object): + registry = self._registries.setdefault(registry_name, {}) + registry[value] = object + + def _registry_get(self, registry_name, value, default=None): + return self._registries[registry_name].get(value, default) + + # ================================== + # Namespace default accessor methods + # ================================== + def set_defaults(self, **kwargs): + self._defaults.update(kwargs) + + # if these defaults match any existing arguments, replace + # the previous default on the object with the new one + for action in self._actions: + if action.dest in kwargs: + action.default = kwargs[action.dest] + + def get_default(self, dest): + for action in self._actions: + if action.dest == dest and action.default is not None: + return action.default + return self._defaults.get(dest, None) + + + # ======================= + # Adding argument actions + # ======================= + def add_argument(self, *args, **kwargs): + """ + add_argument(dest, ..., name=value, ...) + add_argument(option_string, option_string, ..., name=value, ...) + """ + + # if no positional args are supplied or only one is supplied and + # it doesn't look like an option string, parse a positional + # argument + chars = self.prefix_chars + if not args or len(args) == 1 and args[0][0] not in chars: + if args and 'dest' in kwargs: + raise ValueError('dest supplied twice for positional argument') + kwargs = self._get_positional_kwargs(*args, **kwargs) + + # otherwise, we're adding an optional argument + else: + kwargs = self._get_optional_kwargs(*args, **kwargs) + + # if no default was supplied, use the parser-level default + if 'default' not in kwargs: + dest = kwargs['dest'] + if dest in self._defaults: + kwargs['default'] = self._defaults[dest] + elif self.argument_default is not None: + kwargs['default'] = self.argument_default + + # create the action object, and add it to the parser + action_class = self._pop_action_class(kwargs) + if not _callable(action_class): + raise ValueError('unknown action "%s"' % action_class) + action = action_class(**kwargs) + + # raise an error if the action type is not callable + type_func = self._registry_get('type', action.type, action.type) + if not _callable(type_func): + raise ValueError('%r is not callable' % type_func) + + return self._add_action(action) + + def add_argument_group(self, *args, **kwargs): + group = _ArgumentGroup(self, *args, **kwargs) + self._action_groups.append(group) + return group + + def add_mutually_exclusive_group(self, **kwargs): + group = _MutuallyExclusiveGroup(self, **kwargs) + self._mutually_exclusive_groups.append(group) + return group + + def _add_action(self, action): + # resolve any conflicts + self._check_conflict(action) + + # add to actions list + self._actions.append(action) + action.container = self + + # index the action by any option strings it has + for option_string in action.option_strings: + self._option_string_actions[option_string] = action + + # set the flag if any option strings look like negative numbers + for option_string in action.option_strings: + if self._negative_number_matcher.match(option_string): + if not self._has_negative_number_optionals: + self._has_negative_number_optionals.append(True) + + # return the created action + return action + + def _remove_action(self, action): + self._actions.remove(action) + + def _add_container_actions(self, container): + # collect groups by titles + title_group_map = {} + for group in self._action_groups: + if group.title in title_group_map: + msg = _('cannot merge actions - two groups are named %r') + raise ValueError(msg % (group.title)) + title_group_map[group.title] = group + + # map each action to its group + group_map = {} + for group in container._action_groups: + + # if a group with the title exists, use that, otherwise + # create a new group matching the container's group + if group.title not in title_group_map: + title_group_map[group.title] = self.add_argument_group( + title=group.title, + description=group.description, + conflict_handler=group.conflict_handler) + + # map the actions to their new group + for action in group._group_actions: + group_map[action] = title_group_map[group.title] + + # add container's mutually exclusive groups + # NOTE: if add_mutually_exclusive_group ever gains title= and + # description= then this code will need to be expanded as above + for group in container._mutually_exclusive_groups: + mutex_group = self.add_mutually_exclusive_group( + required=group.required) + + # map the actions to their new mutex group + for action in group._group_actions: + group_map[action] = mutex_group + + # add all actions to this container or their group + for action in container._actions: + group_map.get(action, self)._add_action(action) + + def _get_positional_kwargs(self, dest, **kwargs): + # make sure required is not specified + if 'required' in kwargs: + msg = _("'required' is an invalid argument for positionals") + raise TypeError(msg) + + # mark positional arguments as required if at least one is + # always required + if kwargs.get('nargs') not in [OPTIONAL, ZERO_OR_MORE]: + kwargs['required'] = True + if kwargs.get('nargs') == ZERO_OR_MORE and 'default' not in kwargs: + kwargs['required'] = True + + # return the keyword arguments with no option strings + return dict(kwargs, dest=dest, option_strings=[]) + + def _get_optional_kwargs(self, *args, **kwargs): + # determine short and long option strings + option_strings = [] + long_option_strings = [] + for option_string in args: + # error on strings that don't start with an appropriate prefix + if not option_string[0] in self.prefix_chars: + msg = _('invalid option string %r: ' + 'must start with a character %r') + tup = option_string, self.prefix_chars + raise ValueError(msg % tup) + + # strings starting with two prefix characters are long options + option_strings.append(option_string) + if option_string[0] in self.prefix_chars: + if len(option_string) > 1: + if option_string[1] in self.prefix_chars: + long_option_strings.append(option_string) + + # infer destination, '--foo-bar' -> 'foo_bar' and '-x' -> 'x' + dest = kwargs.pop('dest', None) + if dest is None: + if long_option_strings: + dest_option_string = long_option_strings[0] + else: + dest_option_string = option_strings[0] + dest = dest_option_string.lstrip(self.prefix_chars) + if not dest: + msg = _('dest= is required for options like %r') + raise ValueError(msg % option_string) + dest = dest.replace('-', '_') + + # return the updated keyword arguments + return dict(kwargs, dest=dest, option_strings=option_strings) + + def _pop_action_class(self, kwargs, default=None): + action = kwargs.pop('action', default) + return self._registry_get('action', action, action) + + def _get_handler(self): + # determine function from conflict handler string + handler_func_name = '_handle_conflict_%s' % self.conflict_handler + try: + return getattr(self, handler_func_name) + except AttributeError: + msg = _('invalid conflict_resolution value: %r') + raise ValueError(msg % self.conflict_handler) + + def _check_conflict(self, action): + + # find all options that conflict with this option + confl_optionals = [] + for option_string in action.option_strings: + if option_string in self._option_string_actions: + confl_optional = self._option_string_actions[option_string] + confl_optionals.append((option_string, confl_optional)) + + # resolve any conflicts + if confl_optionals: + conflict_handler = self._get_handler() + conflict_handler(action, confl_optionals) + + def _handle_conflict_error(self, action, conflicting_actions): + message = _('conflicting option string(s): %s') + conflict_string = ', '.join([option_string + for option_string, action + in conflicting_actions]) + raise ArgumentError(action, message % conflict_string) + + def _handle_conflict_resolve(self, action, conflicting_actions): + + # remove all conflicting options + for option_string, action in conflicting_actions: + + # remove the conflicting option + action.option_strings.remove(option_string) + self._option_string_actions.pop(option_string, None) + + # if the option now has no option string, remove it from the + # container holding it + if not action.option_strings: + action.container._remove_action(action) + + +class _ArgumentGroup(_ActionsContainer): + + def __init__(self, container, title=None, description=None, **kwargs): + # add any missing keyword arguments by checking the container + update = kwargs.setdefault + update('conflict_handler', container.conflict_handler) + update('prefix_chars', container.prefix_chars) + update('argument_default', container.argument_default) + super_init = super(_ArgumentGroup, self).__init__ + super_init(description=description, **kwargs) + + # group attributes + self.title = title + self._group_actions = [] + + # share most attributes with the container + self._registries = container._registries + self._actions = container._actions + self._option_string_actions = container._option_string_actions + self._defaults = container._defaults + self._has_negative_number_optionals = \ + container._has_negative_number_optionals + + def _add_action(self, action): + action = super(_ArgumentGroup, self)._add_action(action) + self._group_actions.append(action) + return action + + def _remove_action(self, action): + super(_ArgumentGroup, self)._remove_action(action) + self._group_actions.remove(action) + + +class _MutuallyExclusiveGroup(_ArgumentGroup): + + def __init__(self, container, required=False): + super(_MutuallyExclusiveGroup, self).__init__(container) + self.required = required + self._container = container + + def _add_action(self, action): + if action.required: + msg = _('mutually exclusive arguments must be optional') + raise ValueError(msg) + action = self._container._add_action(action) + self._group_actions.append(action) + return action + + def _remove_action(self, action): + self._container._remove_action(action) + self._group_actions.remove(action) + + +class ArgumentParser(_AttributeHolder, _ActionsContainer): + """Object for parsing command line strings into Python objects. + + Keyword Arguments: + - prog -- The name of the program (default: sys.argv[0]) + - usage -- A usage message (default: auto-generated from arguments) + - description -- A description of what the program does + - epilog -- Text following the argument descriptions + - parents -- Parsers whose arguments should be copied into this one + - formatter_class -- HelpFormatter class for printing help messages + - prefix_chars -- Characters that prefix optional arguments + - fromfile_prefix_chars -- Characters that prefix files containing + additional arguments + - argument_default -- The default value for all arguments + - conflict_handler -- String indicating how to handle conflicts + - add_help -- Add a -h/-help option + """ + + def __init__(self, + prog=None, + usage=None, + description=None, + epilog=None, + version=None, + parents=[], + formatter_class=HelpFormatter, + prefix_chars='-', + fromfile_prefix_chars=None, + argument_default=None, + conflict_handler='error', + add_help=True): + + if version is not None: + import warnings + warnings.warn( + """The "version" argument to ArgumentParser is deprecated. """ + """Please use """ + """"add_argument(..., action='version', version="N", ...)" """ + """instead""", DeprecationWarning) + + superinit = super(ArgumentParser, self).__init__ + superinit(description=description, + prefix_chars=prefix_chars, + argument_default=argument_default, + conflict_handler=conflict_handler) + + # default setting for prog + if prog is None: + prog = _os.path.basename(_sys.argv[0]) + + self.prog = prog + self.usage = usage + self.epilog = epilog + self.version = version + self.formatter_class = formatter_class + self.fromfile_prefix_chars = fromfile_prefix_chars + self.add_help = add_help + + add_group = self.add_argument_group + self._positionals = add_group(_('positional arguments')) + self._optionals = add_group(_('optional arguments')) + self._subparsers = None + + # register types + def identity(string): + return string + self.register('type', None, identity) + + # add help and version arguments if necessary + # (using explicit default to override global argument_default) + if '-' in prefix_chars: + default_prefix = '-' + else: + default_prefix = prefix_chars[0] + if self.add_help: + self.add_argument( + default_prefix+'h', default_prefix*2+'help', + action='help', default=SUPPRESS, + help=_('show this help message and exit')) + if self.version: + self.add_argument( + default_prefix+'v', default_prefix*2+'version', + action='version', default=SUPPRESS, + version=self.version, + help=_("show program's version number and exit")) + + # add parent arguments and defaults + for parent in parents: + self._add_container_actions(parent) + try: + defaults = parent._defaults + except AttributeError: + pass + else: + self._defaults.update(defaults) + + # ======================= + # Pretty __repr__ methods + # ======================= + def _get_kwargs(self): + names = [ + 'prog', + 'usage', + 'description', + 'version', + 'formatter_class', + 'conflict_handler', + 'add_help', + ] + return [(name, getattr(self, name)) for name in names] + + # ================================== + # Optional/Positional adding methods + # ================================== + def add_subparsers(self, **kwargs): + if self._subparsers is not None: + self.error(_('cannot have multiple subparser arguments')) + + # add the parser class to the arguments if it's not present + kwargs.setdefault('parser_class', type(self)) + + if 'title' in kwargs or 'description' in kwargs: + title = _(kwargs.pop('title', 'subcommands')) + description = _(kwargs.pop('description', None)) + self._subparsers = self.add_argument_group(title, description) + else: + self._subparsers = self._positionals + + # prog defaults to the usage message of this parser, skipping + # optional arguments and with no "usage:" prefix + if kwargs.get('prog') is None: + formatter = self._get_formatter() + positionals = self._get_positional_actions() + groups = self._mutually_exclusive_groups + formatter.add_usage(self.usage, positionals, groups, '') + kwargs['prog'] = formatter.format_help().strip() + + # create the parsers action and add it to the positionals list + parsers_class = self._pop_action_class(kwargs, 'parsers') + action = parsers_class(option_strings=[], **kwargs) + self._subparsers._add_action(action) + + # return the created parsers action + return action + + def _add_action(self, action): + if action.option_strings: + self._optionals._add_action(action) + else: + self._positionals._add_action(action) + return action + + def _get_optional_actions(self): + return [action + for action in self._actions + if action.option_strings] + + def _get_positional_actions(self): + return [action + for action in self._actions + if not action.option_strings] + + # ===================================== + # Command line argument parsing methods + # ===================================== + def parse_args(self, args=None, namespace=None): + args, argv = self.parse_known_args(args, namespace) + if argv: + msg = _('unrecognized arguments: %s') + self.error(msg % ' '.join(argv)) + return args + + def parse_known_args(self, args=None, namespace=None): + # args default to the system args + if args is None: + args = _sys.argv[1:] + + # default Namespace built from parser defaults + if namespace is None: + namespace = Namespace() + + # add any action defaults that aren't present + for action in self._actions: + if action.dest is not SUPPRESS: + if not hasattr(namespace, action.dest): + if action.default is not SUPPRESS: + setattr(namespace, action.dest, action.default) + + # add any parser defaults that aren't present + for dest in self._defaults: + if not hasattr(namespace, dest): + setattr(namespace, dest, self._defaults[dest]) + + # parse the arguments and exit if there are any errors + try: + namespace, args = self._parse_known_args(args, namespace) + if hasattr(namespace, _UNRECOGNIZED_ARGS_ATTR): + args.extend(getattr(namespace, _UNRECOGNIZED_ARGS_ATTR)) + delattr(namespace, _UNRECOGNIZED_ARGS_ATTR) + return namespace, args + except ArgumentError: + err = _sys.exc_info()[1] + self.error(str(err)) + + def _parse_known_args(self, arg_strings, namespace): + # replace arg strings that are file references + if self.fromfile_prefix_chars is not None: + arg_strings = self._read_args_from_files(arg_strings) + + # map all mutually exclusive arguments to the other arguments + # they can't occur with + action_conflicts = {} + for mutex_group in self._mutually_exclusive_groups: + group_actions = mutex_group._group_actions + for i, mutex_action in enumerate(mutex_group._group_actions): + conflicts = action_conflicts.setdefault(mutex_action, []) + conflicts.extend(group_actions[:i]) + conflicts.extend(group_actions[i + 1:]) + + # find all option indices, and determine the arg_string_pattern + # which has an 'O' if there is an option at an index, + # an 'A' if there is an argument, or a '-' if there is a '--' + option_string_indices = {} + arg_string_pattern_parts = [] + arg_strings_iter = iter(arg_strings) + for i, arg_string in enumerate(arg_strings_iter): + + # all args after -- are non-options + if arg_string == '--': + arg_string_pattern_parts.append('-') + for arg_string in arg_strings_iter: + arg_string_pattern_parts.append('A') + + # otherwise, add the arg to the arg strings + # and note the index if it was an option + else: + option_tuple = self._parse_optional(arg_string) + if option_tuple is None: + pattern = 'A' + else: + option_string_indices[i] = option_tuple + pattern = 'O' + arg_string_pattern_parts.append(pattern) + + # join the pieces together to form the pattern + arg_strings_pattern = ''.join(arg_string_pattern_parts) + + # converts arg strings to the appropriate and then takes the action + seen_actions = set() + seen_non_default_actions = set() + + def take_action(action, argument_strings, option_string=None): + seen_actions.add(action) + argument_values = self._get_values(action, argument_strings) + + # error if this argument is not allowed with other previously + # seen arguments, assuming that actions that use the default + # value don't really count as "present" + if argument_values is not action.default: + seen_non_default_actions.add(action) + for conflict_action in action_conflicts.get(action, []): + if conflict_action in seen_non_default_actions: + msg = _('not allowed with argument %s') + action_name = _get_action_name(conflict_action) + raise ArgumentError(action, msg % action_name) + + # take the action if we didn't receive a SUPPRESS value + # (e.g. from a default) + if argument_values is not SUPPRESS: + action(self, namespace, argument_values, option_string) + + # function to convert arg_strings into an optional action + def consume_optional(start_index): + + # get the optional identified at this index + option_tuple = option_string_indices[start_index] + action, option_string, explicit_arg = option_tuple + + # identify additional optionals in the same arg string + # (e.g. -xyz is the same as -x -y -z if no args are required) + match_argument = self._match_argument + action_tuples = [] + while True: + + # if we found no optional action, skip it + if action is None: + extras.append(arg_strings[start_index]) + return start_index + 1 + + # if there is an explicit argument, try to match the + # optional's string arguments to only this + if explicit_arg is not None: + arg_count = match_argument(action, 'A') + + # if the action is a single-dash option and takes no + # arguments, try to parse more single-dash options out + # of the tail of the option string + chars = self.prefix_chars + if arg_count == 0 and option_string[1] not in chars: + action_tuples.append((action, [], option_string)) + char = option_string[0] + option_string = char + explicit_arg[0] + new_explicit_arg = explicit_arg[1:] or None + optionals_map = self._option_string_actions + if option_string in optionals_map: + action = optionals_map[option_string] + explicit_arg = new_explicit_arg + else: + msg = _('ignored explicit argument %r') + raise ArgumentError(action, msg % explicit_arg) + + # if the action expect exactly one argument, we've + # successfully matched the option; exit the loop + elif arg_count == 1: + stop = start_index + 1 + args = [explicit_arg] + action_tuples.append((action, args, option_string)) + break + + # error if a double-dash option did not use the + # explicit argument + else: + msg = _('ignored explicit argument %r') + raise ArgumentError(action, msg % explicit_arg) + + # if there is no explicit argument, try to match the + # optional's string arguments with the following strings + # if successful, exit the loop + else: + start = start_index + 1 + selected_patterns = arg_strings_pattern[start:] + arg_count = match_argument(action, selected_patterns) + stop = start + arg_count + args = arg_strings[start:stop] + action_tuples.append((action, args, option_string)) + break + + # add the Optional to the list and return the index at which + # the Optional's string args stopped + assert action_tuples + for action, args, option_string in action_tuples: + take_action(action, args, option_string) + return stop + + # the list of Positionals left to be parsed; this is modified + # by consume_positionals() + positionals = self._get_positional_actions() + + # function to convert arg_strings into positional actions + def consume_positionals(start_index): + # match as many Positionals as possible + match_partial = self._match_arguments_partial + selected_pattern = arg_strings_pattern[start_index:] + arg_counts = match_partial(positionals, selected_pattern) + + # slice off the appropriate arg strings for each Positional + # and add the Positional and its args to the list + for action, arg_count in zip(positionals, arg_counts): + args = arg_strings[start_index: start_index + arg_count] + start_index += arg_count + take_action(action, args) + + # slice off the Positionals that we just parsed and return the + # index at which the Positionals' string args stopped + positionals[:] = positionals[len(arg_counts):] + return start_index + + # consume Positionals and Optionals alternately, until we have + # passed the last option string + extras = [] + start_index = 0 + if option_string_indices: + max_option_string_index = max(option_string_indices) + else: + max_option_string_index = -1 + while start_index <= max_option_string_index: + + # consume any Positionals preceding the next option + next_option_string_index = min([ + index + for index in option_string_indices + if index >= start_index]) + if start_index != next_option_string_index: + positionals_end_index = consume_positionals(start_index) + + # only try to parse the next optional if we didn't consume + # the option string during the positionals parsing + if positionals_end_index > start_index: + start_index = positionals_end_index + continue + else: + start_index = positionals_end_index + + # if we consumed all the positionals we could and we're not + # at the index of an option string, there were extra arguments + if start_index not in option_string_indices: + strings = arg_strings[start_index:next_option_string_index] + extras.extend(strings) + start_index = next_option_string_index + + # consume the next optional and any arguments for it + start_index = consume_optional(start_index) + + # consume any positionals following the last Optional + stop_index = consume_positionals(start_index) + + # if we didn't consume all the argument strings, there were extras + extras.extend(arg_strings[stop_index:]) + + # if we didn't use all the Positional objects, there were too few + # arg strings supplied. + if positionals: + self.error(_('too few arguments')) + + # make sure all required actions were present, and convert defaults. + for action in self._actions: + if action not in seen_actions: + if action.required: + name = _get_action_name(action) + self.error(_('argument %s is required') % name) + else: + # Convert action default now instead of doing it before + # parsing arguments to avoid calling convert functions + # twice (which may fail) if the argument was given, but + # only if it was defined already in the namespace + if (action.default is not None and + isinstance(action.default, basestring) and + hasattr(namespace, action.dest) and + action.default is getattr(namespace, action.dest)): + setattr(namespace, action.dest, + self._get_value(action, action.default)) + + # make sure all required groups had one option present + for group in self._mutually_exclusive_groups: + if group.required: + for action in group._group_actions: + if action in seen_non_default_actions: + break + + # if no actions were used, report the error + else: + names = [_get_action_name(action) + for action in group._group_actions + if action.help is not SUPPRESS] + msg = _('one of the arguments %s is required') + self.error(msg % ' '.join(names)) + + # return the updated namespace and the extra arguments + return namespace, extras + + def _read_args_from_files(self, arg_strings): + # expand arguments referencing files + new_arg_strings = [] + for arg_string in arg_strings: + + # for regular arguments, just add them back into the list + if arg_string[0] not in self.fromfile_prefix_chars: + new_arg_strings.append(arg_string) + + # replace arguments referencing files with the file content + else: + try: + args_file = open(arg_string[1:]) + try: + arg_strings = [] + for arg_line in args_file.read().splitlines(): + for arg in self.convert_arg_line_to_args(arg_line): + arg_strings.append(arg) + arg_strings = self._read_args_from_files(arg_strings) + new_arg_strings.extend(arg_strings) + finally: + args_file.close() + except IOError: + err = _sys.exc_info()[1] + self.error(str(err)) + + # return the modified argument list + return new_arg_strings + + def convert_arg_line_to_args(self, arg_line): + return [arg_line] + + def _match_argument(self, action, arg_strings_pattern): + # match the pattern for this action to the arg strings + nargs_pattern = self._get_nargs_pattern(action) + match = _re.match(nargs_pattern, arg_strings_pattern) + + # raise an exception if we weren't able to find a match + if match is None: + nargs_errors = { + None: _('expected one argument'), + OPTIONAL: _('expected at most one argument'), + ONE_OR_MORE: _('expected at least one argument'), + } + default = _('expected %s argument(s)') % action.nargs + msg = nargs_errors.get(action.nargs, default) + raise ArgumentError(action, msg) + + # return the number of arguments matched + return len(match.group(1)) + + def _match_arguments_partial(self, actions, arg_strings_pattern): + # progressively shorten the actions list by slicing off the + # final actions until we find a match + result = [] + for i in range(len(actions), 0, -1): + actions_slice = actions[:i] + pattern = ''.join([self._get_nargs_pattern(action) + for action in actions_slice]) + match = _re.match(pattern, arg_strings_pattern) + if match is not None: + result.extend([len(string) for string in match.groups()]) + break + + # return the list of arg string counts + return result + + def _parse_optional(self, arg_string): + # if it's an empty string, it was meant to be a positional + if not arg_string: + return None + + # if it doesn't start with a prefix, it was meant to be positional + if not arg_string[0] in self.prefix_chars: + return None + + # if the option string is present in the parser, return the action + if arg_string in self._option_string_actions: + action = self._option_string_actions[arg_string] + return action, arg_string, None + + # if it's just a single character, it was meant to be positional + if len(arg_string) == 1: + return None + + # if the option string before the "=" is present, return the action + if '=' in arg_string: + option_string, explicit_arg = arg_string.split('=', 1) + if option_string in self._option_string_actions: + action = self._option_string_actions[option_string] + return action, option_string, explicit_arg + + # search through all possible prefixes of the option string + # and all actions in the parser for possible interpretations + option_tuples = self._get_option_tuples(arg_string) + + # if multiple actions match, the option string was ambiguous + if len(option_tuples) > 1: + options = ', '.join([option_string + for action, option_string, explicit_arg in option_tuples]) + tup = arg_string, options + self.error(_('ambiguous option: %s could match %s') % tup) + + # if exactly one action matched, this segmentation is good, + # so return the parsed action + elif len(option_tuples) == 1: + option_tuple, = option_tuples + return option_tuple + + # if it was not found as an option, but it looks like a negative + # number, it was meant to be positional + # unless there are negative-number-like options + if self._negative_number_matcher.match(arg_string): + if not self._has_negative_number_optionals: + return None + + # if it contains a space, it was meant to be a positional + if ' ' in arg_string: + return None + + # it was meant to be an optional but there is no such option + # in this parser (though it might be a valid option in a subparser) + return None, arg_string, None + + def _get_option_tuples(self, option_string): + result = [] + + # option strings starting with two prefix characters are only + # split at the '=' + chars = self.prefix_chars + if option_string[0] in chars and option_string[1] in chars: + if '=' in option_string: + option_prefix, explicit_arg = option_string.split('=', 1) + else: + option_prefix = option_string + explicit_arg = None + for option_string in self._option_string_actions: + if option_string.startswith(option_prefix): + action = self._option_string_actions[option_string] + tup = action, option_string, explicit_arg + result.append(tup) + + # single character options can be concatenated with their arguments + # but multiple character options always have to have their argument + # separate + elif option_string[0] in chars and option_string[1] not in chars: + option_prefix = option_string + explicit_arg = None + short_option_prefix = option_string[:2] + short_explicit_arg = option_string[2:] + + for option_string in self._option_string_actions: + if option_string == short_option_prefix: + action = self._option_string_actions[option_string] + tup = action, option_string, short_explicit_arg + result.append(tup) + elif option_string.startswith(option_prefix): + action = self._option_string_actions[option_string] + tup = action, option_string, explicit_arg + result.append(tup) + + # shouldn't ever get here + else: + self.error(_('unexpected option string: %s') % option_string) + + # return the collected option tuples + return result + + def _get_nargs_pattern(self, action): + # in all examples below, we have to allow for '--' args + # which are represented as '-' in the pattern + nargs = action.nargs + + # the default (None) is assumed to be a single argument + if nargs is None: + nargs_pattern = '(-*A-*)' + + # allow zero or one arguments + elif nargs == OPTIONAL: + nargs_pattern = '(-*A?-*)' + + # allow zero or more arguments + elif nargs == ZERO_OR_MORE: + nargs_pattern = '(-*[A-]*)' + + # allow one or more arguments + elif nargs == ONE_OR_MORE: + nargs_pattern = '(-*A[A-]*)' + + # allow any number of options or arguments + elif nargs == REMAINDER: + nargs_pattern = '([-AO]*)' + + # allow one argument followed by any number of options or arguments + elif nargs == PARSER: + nargs_pattern = '(-*A[-AO]*)' + + # all others should be integers + else: + nargs_pattern = '(-*%s-*)' % '-*'.join('A' * nargs) + + # if this is an optional action, -- is not allowed + if action.option_strings: + nargs_pattern = nargs_pattern.replace('-*', '') + nargs_pattern = nargs_pattern.replace('-', '') + + # return the pattern + return nargs_pattern + + # ======================== + # Value conversion methods + # ======================== + def _get_values(self, action, arg_strings): + # for everything but PARSER args, strip out '--' + if action.nargs not in [PARSER, REMAINDER]: + arg_strings = [s for s in arg_strings if s != '--'] + + # optional argument produces a default when not present + if not arg_strings and action.nargs == OPTIONAL: + if action.option_strings: + value = action.const + else: + value = action.default + if isinstance(value, basestring): + value = self._get_value(action, value) + self._check_value(action, value) + + # when nargs='*' on a positional, if there were no command-line + # args, use the default if it is anything other than None + elif (not arg_strings and action.nargs == ZERO_OR_MORE and + not action.option_strings): + if action.default is not None: + value = action.default + else: + value = arg_strings + self._check_value(action, value) + + # single argument or optional argument produces a single value + elif len(arg_strings) == 1 and action.nargs in [None, OPTIONAL]: + arg_string, = arg_strings + value = self._get_value(action, arg_string) + self._check_value(action, value) + + # REMAINDER arguments convert all values, checking none + elif action.nargs == REMAINDER: + value = [self._get_value(action, v) for v in arg_strings] + + # PARSER arguments convert all values, but check only the first + elif action.nargs == PARSER: + value = [self._get_value(action, v) for v in arg_strings] + self._check_value(action, value[0]) + + # all other types of nargs produce a list + else: + value = [self._get_value(action, v) for v in arg_strings] + for v in value: + self._check_value(action, v) + + # return the converted value + return value + + def _get_value(self, action, arg_string): + type_func = self._registry_get('type', action.type, action.type) + if not _callable(type_func): + msg = _('%r is not callable') + raise ArgumentError(action, msg % type_func) + + # convert the value to the appropriate type + try: + result = type_func(arg_string) + + # ArgumentTypeErrors indicate errors + except ArgumentTypeError: + name = getattr(action.type, '__name__', repr(action.type)) + msg = str(_sys.exc_info()[1]) + raise ArgumentError(action, msg) + + # TypeErrors or ValueErrors also indicate errors + except (TypeError, ValueError): + name = getattr(action.type, '__name__', repr(action.type)) + msg = _('invalid %s value: %r') + raise ArgumentError(action, msg % (name, arg_string)) + + # return the converted value + return result + + def _check_value(self, action, value): + # converted value must be one of the choices (if specified) + if action.choices is not None and value not in action.choices: + tup = value, ', '.join(map(repr, action.choices)) + msg = _('invalid choice: %r (choose from %s)') % tup + raise ArgumentError(action, msg) + + # ======================= + # Help-formatting methods + # ======================= + def format_usage(self): + formatter = self._get_formatter() + formatter.add_usage(self.usage, self._actions, + self._mutually_exclusive_groups) + return formatter.format_help() + + def format_help(self): + formatter = self._get_formatter() + + # usage + formatter.add_usage(self.usage, self._actions, + self._mutually_exclusive_groups) + + # description + formatter.add_text(self.description) + + # positionals, optionals and user-defined groups + for action_group in self._action_groups: + formatter.start_section(action_group.title) + formatter.add_text(action_group.description) + formatter.add_arguments(action_group._group_actions) + formatter.end_section() + + # epilog + formatter.add_text(self.epilog) + + # determine help from format above + return formatter.format_help() + + def format_version(self): + import warnings + warnings.warn( + 'The format_version method is deprecated -- the "version" ' + 'argument to ArgumentParser is no longer supported.', + DeprecationWarning) + formatter = self._get_formatter() + formatter.add_text(self.version) + return formatter.format_help() + + def _get_formatter(self): + return self.formatter_class(prog=self.prog) + + # ===================== + # Help-printing methods + # ===================== + def print_usage(self, file=None): + if file is None: + file = _sys.stdout + self._print_message(self.format_usage(), file) + + def print_help(self, file=None): + if file is None: + file = _sys.stdout + self._print_message(self.format_help(), file) + + def print_version(self, file=None): + import warnings + warnings.warn( + 'The print_version method is deprecated -- the "version" ' + 'argument to ArgumentParser is no longer supported.', + DeprecationWarning) + self._print_message(self.format_version(), file) + + def _print_message(self, message, file=None): + if message: + if file is None: + file = _sys.stderr + file.write(message) + + # =============== + # Exiting methods + # =============== + def exit(self, status=0, message=None): + if message: + self._print_message(message, _sys.stderr) + _sys.exit(status) + + def error(self, message): + """error(message: string) + + Prints a usage message incorporating the message to stderr and + exits. + + If you override this in a subclass, it should not return -- it + should either exit or raise an exception. + """ + self.print_usage(_sys.stderr) + self.exit(2, _('%s: error: %s\n') % (self.prog, message)) -- 2.13.5 Add the scripts/ directory to sys.path so Python 2.6 will be able to import argparse. Cc: Fam Zheng <famz@redhat.com> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com> Acked-by: John Snow <jsnow@redhat.com> Acked-by: Fam Zheng <famz@redhat.com> Message-id: 20170825155732.15665-3-stefanha@redhat.com Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com> --- tests/docker/docker.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/tests/docker/docker.py b/tests/docker/docker.py index XXXXXXX..XXXXXXX 100755 --- a/tests/docker/docker.py +++ b/tests/docker/docker.py @@ -XXX,XX +XXX,XX @@ import os import sys +sys.path.append(os.path.join(os.path.dirname(__file__), + '..', '..', 'scripts')) +import argparse import subprocess import json import hashlib import atexit import uuid -import argparse import tempfile import re import signal -- 2.13.5 Add the scripts/ directory to sys.path so Python 2.6 will be able to import argparse. Cc: Daniel P. Berrange <berrange@redhat.com> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com> Acked-by: John Snow <jsnow@redhat.com> Acked-by: Fam Zheng <famz@redhat.com> Message-id: 20170825155732.15665-4-stefanha@redhat.com Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com> --- tests/migration/guestperf/shell.py | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/tests/migration/guestperf/shell.py b/tests/migration/guestperf/shell.py index XXXXXXX..XXXXXXX 100644 --- a/tests/migration/guestperf/shell.py +++ b/tests/migration/guestperf/shell.py @@ -XXX,XX +XXX,XX @@ # -import argparse -import fnmatch import os import os.path -import platform import sys +sys.path.append(os.path.join(os.path.dirname(__file__), + '..', '..', '..', 'scripts')) +import argparse +import fnmatch +import platform from guestperf.hardware import Hardware from guestperf.engine import Engine -- 2.13.5 From: Fred Rolland <rollandf@gmail.com> Update doc with the usage of UUID for initiator name. Related-To: https://bugzilla.redhat.com/1006468 Signed-off-by: Fred Rolland <frolland@redhat.com> Message-id: 20170823084830.30500-1-frolland@redhat.com Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com> --- qemu-doc.texi | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/qemu-doc.texi b/qemu-doc.texi index XXXXXXX..XXXXXXX 100644 --- a/qemu-doc.texi +++ b/qemu-doc.texi @@ -XXX,XX +XXX,XX @@ in a configuration file provided via '-readconfig' or directly on the command line. If the initiator-name is not specified qemu will use a default name -of 'iqn.2008-11.org.linux-kvm[:<name>'] where <name> is the name of the +of 'iqn.2008-11.org.linux-kvm[:<uuid>'] where <uuid> is the UUID of the +virtual machine. If the UUID is not specified qemu will use +'iqn.2008-11.org.linux-kvm[:<name>'] where <name> is the name of the virtual machine. - @example Setting a specific initiator name to use when logging in to the target -iscsi initiator-name=iqn.qemu.test:my-initiator -- 2.13.5 Most qcow2 files are uncompressed so it is wasteful to allocate (32 + 1) * cluster_size + 512 bytes upfront. Allocate s->cluster_cache and s->cluster_data when the first read operation is performance on a compressed cluster. The buffers are freed in .bdrv_close(). .bdrv_open() no longer has any code paths that can allocate these buffers, so remove the free functions in the error code path. This patch can result in significant memory savings when many qcow2 disks are attached or backing file chains are long: Before 12.81% (1,023,193,088B) After 5.36% (393,893,888B) Reported-by: Alexey Kardashevskiy <aik@ozlabs.ru> Tested-by: Alexey Kardashevskiy <aik@ozlabs.ru> Reviewed-by: Eric Blake <eblake@redhat.com> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com> Message-id: 20170821135530.32344-1-stefanha@redhat.com Cc: Kevin Wolf <kwolf@redhat.com> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com> --- block/qcow2-cluster.c | 17 +++++++++++++++++ block/qcow2.c | 12 ------------ 2 files changed, 17 insertions(+), 12 deletions(-) diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2-cluster.c +++ b/block/qcow2-cluster.c @@ -XXX,XX +XXX,XX @@ int qcow2_decompress_cluster(BlockDriverState *bs, uint64_t cluster_offset) nb_csectors = ((cluster_offset >> s->csize_shift) & s->csize_mask) + 1; sector_offset = coffset & 511; csize = nb_csectors * 512 - sector_offset; + + /* Allocate buffers on first decompress operation, most images are + * uncompressed and the memory overhead can be avoided. The buffers + * are freed in .bdrv_close(). + */ + if (!s->cluster_data) { + /* one more sector for decompressed data alignment */ + s->cluster_data = qemu_try_blockalign(bs->file->bs, + QCOW_MAX_CRYPT_CLUSTERS * s->cluster_size + 512); + if (!s->cluster_data) { + return -ENOMEM; + } + } + if (!s->cluster_cache) { + s->cluster_cache = g_malloc(s->cluster_size); + } + BLKDBG_EVENT(bs->file, BLKDBG_READ_COMPRESSED); ret = bdrv_read(bs->file, coffset >> 9, s->cluster_data, nb_csectors); diff --git a/block/qcow2.c b/block/qcow2.c index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -XXX,XX +XXX,XX @@ static int qcow2_do_open(BlockDriverState *bs, QDict *options, int flags, goto fail; } - s->cluster_cache = g_malloc(s->cluster_size); - /* one more sector for decompressed data alignment */ - s->cluster_data = qemu_try_blockalign(bs->file->bs, QCOW_MAX_CRYPT_CLUSTERS - * s->cluster_size + 512); - if (s->cluster_data == NULL) { - error_setg(errp, "Could not allocate temporary cluster buffer"); - ret = -ENOMEM; - goto fail; - } - s->cluster_cache_offset = -1; s->flags = flags; @@ -XXX,XX +XXX,XX @@ static int qcow2_do_open(BlockDriverState *bs, QDict *options, int flags, if (s->refcount_block_cache) { qcow2_cache_destroy(bs, s->refcount_block_cache); } - g_free(s->cluster_cache); - qemu_vfree(s->cluster_data); qcrypto_block_free(s->crypto); qapi_free_QCryptoBlockOpenOptions(s->crypto_opts); return ret; -- 2.13.5 The following changes since commit 5375af3cd7b8adcc10c18d8083b7be63976c9645: Merge remote-tracking branch 'remotes/mst/tags/for_upstream' into staging (2020-05-04 15:51:09 +0100) are available in the Git repository at: https://github.com/XanClic/qemu.git tags/pull-block-2020-05-05 for you to fetch changes up to 4ce5dd3e9b5ee0fac18625860eb3727399ee965e: block/block-copy: use aio-task-pool API (2020-05-05 14:03:28 +0200) ---------------------------------------------------------------- Block patches: - Asynchronous copying for block-copy (i.e., the backup job) - Allow resizing of qcow2 images when they have internal snapshots - iotests: Logging improvements for Python tests - iotest 153 fix, and block comment cleanups ---------------------------------------------------------------- Eric Blake (4): block: Add blk_new_with_bs() helper qcow2: Allow resize of images with internal snapshots qcow2: Tweak comment about bitmaps vs. resize block: Comment cleanups John Snow (14): iotests: do a light delinting iotests: don't use 'format' for drive_add iotests: ignore import warnings from pylint iotests: replace mutable list default args iotests: add pylintrc file iotests: alphabetize standard imports iotests: drop pre-Python 3.4 compatibility code iotests: touch up log function signature iotests: limit line length to 79 chars iotests: add hmp helper with logging iotests: add script_initialize iotest 258: use script_main iotests: Mark verify functions as private iotests: use python logging for iotests.log() Maxim Levitsky (1): Fix iotest 153 Vladimir Sementsov-Ogievskiy (5): block/block-copy: rename in-flight requests to tasks block/block-copy: alloc task on each iteration block/block-copy: add state pointer to BlockCopyTask block/block-copy: refactor task creation block/block-copy: use aio-task-pool API include/sysemu/block-backend.h | 2 + block/block-backend.c | 23 +++ block/block-copy.c | 279 +++++++++++++++++-------- block/crypto.c | 9 +- block/io.c | 3 +- block/parallels.c | 8 +- block/qcow.c | 8 +- block/qcow2-refcount.c | 2 +- block/qcow2-snapshot.c | 20 +- block/qcow2.c | 45 ++-- block/qed.c | 8 +- block/sheepdog.c | 10 +- block/vdi.c | 8 +- block/vhdx.c | 8 +- block/vmdk.c | 9 +- block/vpc.c | 8 +- block/vvfat.c | 10 +- blockdev.c | 8 +- blockjob.c | 7 +- tests/qemu-iotests/001 | 2 +- tests/qemu-iotests/030 | 4 +- tests/qemu-iotests/052 | 2 +- tests/qemu-iotests/055 | 3 +- tests/qemu-iotests/061 | 35 ++++ tests/qemu-iotests/061.out | 28 +++ tests/qemu-iotests/134 | 2 +- tests/qemu-iotests/149 | 3 +- tests/qemu-iotests/153 | 2 +- tests/qemu-iotests/153.out | 12 +- tests/qemu-iotests/155 | 2 +- tests/qemu-iotests/188 | 2 +- tests/qemu-iotests/194 | 4 +- tests/qemu-iotests/202 | 4 +- tests/qemu-iotests/203 | 4 +- tests/qemu-iotests/206 | 2 +- tests/qemu-iotests/207 | 6 +- tests/qemu-iotests/208 | 2 +- tests/qemu-iotests/209 | 2 +- tests/qemu-iotests/210 | 6 +- tests/qemu-iotests/211 | 6 +- tests/qemu-iotests/212 | 6 +- tests/qemu-iotests/213 | 6 +- tests/qemu-iotests/216 | 4 +- tests/qemu-iotests/218 | 2 +- tests/qemu-iotests/219 | 2 +- tests/qemu-iotests/222 | 7 +- tests/qemu-iotests/224 | 4 +- tests/qemu-iotests/228 | 6 +- tests/qemu-iotests/234 | 4 +- tests/qemu-iotests/235 | 4 +- tests/qemu-iotests/236 | 2 +- tests/qemu-iotests/237 | 2 +- tests/qemu-iotests/238 | 2 + tests/qemu-iotests/242 | 2 +- tests/qemu-iotests/245 | 1 + tests/qemu-iotests/245.out | 10 +- tests/qemu-iotests/246 | 2 +- tests/qemu-iotests/248 | 2 +- tests/qemu-iotests/254 | 2 +- tests/qemu-iotests/255 | 2 +- tests/qemu-iotests/256 | 2 +- tests/qemu-iotests/258 | 10 +- tests/qemu-iotests/260 | 4 +- tests/qemu-iotests/262 | 4 +- tests/qemu-iotests/264 | 4 +- tests/qemu-iotests/274 | 4 +- tests/qemu-iotests/277 | 2 + tests/qemu-iotests/280 | 8 +- tests/qemu-iotests/283 | 4 +- tests/qemu-iotests/iotests.py | 366 ++++++++++++++++++++------------- tests/qemu-iotests/pylintrc | 26 +++ 71 files changed, 728 insertions(+), 386 deletions(-) create mode 100644 tests/qemu-iotests/pylintrc -- 2.26.2 From: John Snow <jsnow@redhat.com> This doesn't fix everything in here, but it does help clean up the pylint report considerably. This should be 100% style changes only; the intent is to make pylint more useful by working on establishing a baseline for iotests that we can gate against in the future. Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Message-Id: <20200331000014.11581-2-jsnow@redhat.com> Reviewed-by: Kevin Wolf <kwolf@redhat.com> Signed-off-by: Max Reitz <mreitz@redhat.com> --- tests/qemu-iotests/iotests.py | 83 ++++++++++++++++++----------------- 1 file changed, 43 insertions(+), 40 deletions(-) diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py index XXXXXXX..XXXXXXX 100644 --- a/tests/qemu-iotests/iotests.py +++ b/tests/qemu-iotests/iotests.py @@ -XXX,XX +XXX,XX @@ # along with this program. If not, see <http://www.gnu.org/licenses/>. # -import errno import os import re import subprocess -import string import unittest import sys import struct @@ -XXX,XX +XXX,XX @@ import faulthandler sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python')) from qemu import qtest -assert sys.version_info >= (3,6) +assert sys.version_info >= (3, 6) faulthandler.enable() @@ -XXX,XX +XXX,XX @@ def qemu_img_log(*args): return result def img_info_log(filename, filter_path=None, imgopts=False, extra_args=[]): - args = [ 'info' ] + args = ['info'] if imgopts: args.append('--image-opts') else: - args += [ '-f', imgfmt ] + args += ['-f', imgfmt] args += extra_args args.append(filename) @@ -XXX,XX +XXX,XX @@ class QemuIoInteractive: # quit command is in close(), '\n' is added automatically assert '\n' not in cmd cmd = cmd.strip() - assert cmd != 'q' and cmd != 'quit' + assert cmd not in ('q', 'quit') self._p.stdin.write(cmd + '\n') self._p.stdin.flush() return self._read_output() @@ -XXX,XX +XXX,XX @@ def qemu_nbd_early_pipe(*args): sys.stderr.write('qemu-nbd received signal %i: %s\n' % (-exitcode, ' '.join(qemu_nbd_args + ['--fork'] + list(args)))) - if exitcode == 0: - return exitcode, '' - else: - return exitcode, subp.communicate()[0] + + return exitcode, subp.communicate()[0] if exitcode else '' def qemu_nbd_popen(*args): '''Run qemu-nbd in daemon mode and return the parent's exit code''' @@ -XXX,XX +XXX,XX @@ def filter_qmp(qmsg, filter_fn): items = qmsg.items() for k, v in items: - if isinstance(v, list) or isinstance(v, dict): + if isinstance(v, (dict, list)): qmsg[k] = filter_qmp(v, filter_fn) else: qmsg[k] = filter_fn(k, v) @@ -XXX,XX +XXX,XX @@ def filter_testfiles(msg): return msg.replace(prefix, 'TEST_DIR/PID-') def filter_qmp_testfiles(qmsg): - def _filter(key, value): + def _filter(_key, value): if is_str(value): return filter_testfiles(value) return value @@ -XXX,XX +XXX,XX @@ def filter_imgfmt(msg): return msg.replace(imgfmt, 'IMGFMT') def filter_qmp_imgfmt(qmsg): - def _filter(key, value): + def _filter(_key, value): if is_str(value): return filter_imgfmt(value) return value @@ -XXX,XX +XXX,XX @@ def log(msg, filters=[], indent=None): If indent is provided, JSON serializable messages are pretty-printed.''' for flt in filters: msg = flt(msg) - if isinstance(msg, dict) or isinstance(msg, list): + if isinstance(msg, (dict, list)): # Python < 3.4 needs to know not to add whitespace when pretty-printing: separators = (', ', ': ') if indent is None else (',', ': ') # Don't sort if it's already sorted @@ -XXX,XX +XXX,XX @@ def log(msg, filters=[], indent=None): print(msg) class Timeout: - def __init__(self, seconds, errmsg = "Timeout"): + def __init__(self, seconds, errmsg="Timeout"): self.seconds = seconds self.errmsg = errmsg def __enter__(self): signal.signal(signal.SIGALRM, self.timeout) signal.setitimer(signal.ITIMER_REAL, self.seconds) return self - def __exit__(self, type, value, traceback): + def __exit__(self, exc_type, value, traceback): signal.setitimer(signal.ITIMER_REAL, 0) return False def timeout(self, signum, frame): @@ -XXX,XX +XXX,XX @@ class Timeout: def file_pattern(name): return "{0}-{1}".format(os.getpid(), name) -class FilePaths(object): +class FilePaths: """ FilePaths is an auto-generated filename that cleans itself up. @@ -XXX,XX +XXX,XX @@ class VM(qtest.QEMUQtestMachine): self.pause_drive(drive, "write_aio") return self.qmp('human-monitor-command', - command_line='qemu-io %s "break %s bp_%s"' % (drive, event, drive)) + command_line='qemu-io %s "break %s bp_%s"' % (drive, event, drive)) def resume_drive(self, drive): self.qmp('human-monitor-command', - command_line='qemu-io %s "remove_break bp_%s"' % (drive, drive)) + command_line='qemu-io %s "remove_break bp_%s"' % (drive, drive)) def hmp_qemu_io(self, drive, cmd): '''Write to a given drive using an HMP command''' @@ -XXX,XX +XXX,XX @@ class VM(qtest.QEMUQtestMachine): if output is None: output = dict() if isinstance(obj, list): - for i in range(len(obj)): - self.flatten_qmp_object(obj[i], output, basestr + str(i) + '.') + for i, item in enumerate(obj): + self.flatten_qmp_object(item, output, basestr + str(i) + '.') elif isinstance(obj, dict): for key in obj: self.flatten_qmp_object(obj[key], output, basestr + key + '.') @@ -XXX,XX +XXX,XX @@ class VM(qtest.QEMUQtestMachine): for bitmap in bitmaps[node_name]: if bitmap.get('name', '') == bitmap_name: - if recording is None: - return bitmap - elif bitmap.get('recording') == recording: + if recording is None or bitmap.get('recording') == recording: return bitmap return None @@ -XXX,XX +XXX,XX @@ class VM(qtest.QEMUQtestMachine): assert node is not None, 'Cannot follow path %s%s' % (root, path) try: - node_id = next(edge['child'] for edge in graph['edges'] \ - if edge['parent'] == node['id'] and - edge['name'] == child_name) + node_id = next(edge['child'] for edge in graph['edges'] + if (edge['parent'] == node['id'] and + edge['name'] == child_name)) + + node = next(node for node in graph['nodes'] + if node['id'] == node_id) - node = next(node for node in graph['nodes'] \ - if node['id'] == node_id) except StopIteration: node = None @@ -XXX,XX +XXX,XX @@ index_re = re.compile(r'([^\[]+)\[([^\]]+)\]') class QMPTestCase(unittest.TestCase): '''Abstract base class for QMP test cases''' + def __init__(self, *args, **kwargs): + super().__init__(*args, **kwargs) + # Many users of this class set a VM property we rely on heavily + # in the methods below. + self.vm = None + def dictpath(self, d, path): '''Traverse a path in a nested dict''' for component in path.split('/'): @@ -XXX,XX +XXX,XX @@ class QMPTestCase(unittest.TestCase): else: self.assertEqual(result, value, '"%s" is "%s", expected "%s"' - % (path, str(result), str(value))) + % (path, str(result), str(value))) def assert_no_active_block_jobs(self): result = self.vm.qmp('query-block-jobs') @@ -XXX,XX +XXX,XX @@ class QMPTestCase(unittest.TestCase): """Issue a query-named-block-nodes and assert node_name and/or file_name is present in the result""" def check_equal_or_none(a, b): - return a == None or b == None or a == b + return a is None or b is None or a == b assert node_name or file_name result = self.vm.qmp('query-named-block-nodes') for x in result["return"]: if check_equal_or_none(x.get("node-name"), node_name) and \ check_equal_or_none(x.get("file"), file_name): return - self.assertTrue(False, "Cannot find %s %s in result:\n%s" % \ - (node_name, file_name, result)) + self.fail("Cannot find %s %s in result:\n%s" % + (node_name, file_name, result)) def assert_json_filename_equal(self, json_filename, reference): '''Asserts that the given filename is a json: filename and that its @@ -XXX,XX +XXX,XX @@ class QMPTestCase(unittest.TestCase): self.assert_qmp(event, 'data/error', error) self.assert_no_active_block_jobs() return event - elif event['event'] == 'JOB_STATUS_CHANGE': + if event['event'] == 'JOB_STATUS_CHANGE': self.assert_qmp(event, 'data/id', drive) def wait_ready(self, drive='drive0'): - '''Wait until a block job BLOCK_JOB_READY event''' - f = {'data': {'type': 'mirror', 'device': drive } } - event = self.vm.event_wait(name='BLOCK_JOB_READY', match=f) + """Wait until a BLOCK_JOB_READY event, and return the event.""" + f = {'data': {'type': 'mirror', 'device': drive}} + return self.vm.event_wait(name='BLOCK_JOB_READY', match=f) def wait_ready_and_cancel(self, drive='drive0'): self.wait_ready(drive=drive) @@ -XXX,XX +XXX,XX @@ class QMPTestCase(unittest.TestCase): for job in result['return']: if job['device'] == job_id: found = True - if job['paused'] == True and job['busy'] == False: + if job['paused'] and not job['busy']: return job break assert found @@ -XXX,XX +XXX,XX @@ def qemu_pipe(*args): universal_newlines=True) exitcode = subp.wait() if exitcode < 0: - sys.stderr.write('qemu received signal %i: %s\n' % (-exitcode, - ' '.join(args))) + sys.stderr.write('qemu received signal %i: %s\n' % + (-exitcode, ' '.join(args))) return subp.communicate()[0] def supported_formats(read_only=False): @@ -XXX,XX +XXX,XX @@ def skip_if_unsupported(required_formats=[], read_only=False): if usf_list: test_case.case_skip('{}: formats {} are not whitelisted'.format( test_case, usf_list)) + return None else: return func(test_case, *args, **kwargs) return func_wrapper @@ -XXX,XX +XXX,XX @@ def skip_if_user_is_root(func): def func_wrapper(*args, **kwargs): if os.getuid() == 0: case_notrun('{}: cannot be run as root'.format(args[0])) + return None else: return func(*args, **kwargs) return func_wrapper -- 2.26.2 From: John Snow <jsnow@redhat.com> It shadows (with a different type) the built-in format. Use something else. Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Message-Id: <20200331000014.11581-3-jsnow@redhat.com> Reviewed-by: Kevin Wolf <kwolf@redhat.com> Signed-off-by: Max Reitz <mreitz@redhat.com> --- tests/qemu-iotests/055 | 3 ++- tests/qemu-iotests/iotests.py | 6 +++--- 2 files changed, 5 insertions(+), 4 deletions(-) diff --git a/tests/qemu-iotests/055 b/tests/qemu-iotests/055 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/055 +++ b/tests/qemu-iotests/055 @@ -XXX,XX +XXX,XX @@ class TestDriveCompression(iotests.QMPTestCase): qemu_img('create', '-f', fmt, blockdev_target_img, str(TestDriveCompression.image_len), *args) if attach_target: - self.vm.add_drive(blockdev_target_img, format=fmt, interface="none") + self.vm.add_drive(blockdev_target_img, + img_format=fmt, interface="none") self.vm.launch() diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py index XXXXXXX..XXXXXXX 100644 --- a/tests/qemu-iotests/iotests.py +++ b/tests/qemu-iotests/iotests.py @@ -XXX,XX +XXX,XX @@ class VM(qtest.QEMUQtestMachine): self._args.append(opts) return self - def add_drive(self, path, opts='', interface='virtio', format=imgfmt): + def add_drive(self, path, opts='', interface='virtio', img_format=imgfmt): '''Add a virtio-blk drive to the VM''' options = ['if=%s' % interface, 'id=drive%d' % self._num_drives] if path is not None: options.append('file=%s' % path) - options.append('format=%s' % format) + options.append('format=%s' % img_format) options.append('cache=%s' % cachemode) options.append('aio=%s' % aiomode) if opts: options.append(opts) - if format == 'luks' and 'key-secret' not in opts: + if img_format == 'luks' and 'key-secret' not in opts: # default luks support if luks_default_secret_object not in self._args: self.add_object(luks_default_secret_object) -- 2.26.2 From: John Snow <jsnow@redhat.com> The right way to solve this is to come up with a virtual environment infrastructure that sets all the paths correctly, and/or to create installable python modules that can be imported normally. That's hard, so just silence this error for now. Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Signed-off-by: John Snow <jsnow@redhat.com> Message-Id: <20200331000014.11581-4-jsnow@redhat.com> Reviewed-by: Kevin Wolf <kwolf@redhat.com> Signed-off-by: Max Reitz <mreitz@redhat.com> --- tests/qemu-iotests/iotests.py | 1 + 1 file changed, 1 insertion(+) diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py index XXXXXXX..XXXXXXX 100644 --- a/tests/qemu-iotests/iotests.py +++ b/tests/qemu-iotests/iotests.py @@ -XXX,XX +XXX,XX @@ import io from collections import OrderedDict import faulthandler +# pylint: disable=import-error, wrong-import-position sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python')) from qemu import qtest -- 2.26.2 From: John Snow <jsnow@redhat.com> It's bad hygiene: if we modify this list, it will be modified across all invocations. (Remaining bad usages are fixed in a subsequent patch which changes the function signature anyway.) Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Message-Id: <20200331000014.11581-5-jsnow@redhat.com> Reviewed-by: Kevin Wolf <kwolf@redhat.com> Signed-off-by: Max Reitz <mreitz@redhat.com> --- tests/qemu-iotests/iotests.py | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py index XXXXXXX..XXXXXXX 100644 --- a/tests/qemu-iotests/iotests.py +++ b/tests/qemu-iotests/iotests.py @@ -XXX,XX +XXX,XX @@ def qemu_img_log(*args): log(result, filters=[filter_testfiles]) return result -def img_info_log(filename, filter_path=None, imgopts=False, extra_args=[]): +def img_info_log(filename, filter_path=None, imgopts=False, extra_args=()): args = ['info'] if imgopts: args.append('--image-opts') @@ -XXX,XX +XXX,XX @@ def filter_qmp_imgfmt(qmsg): return value return filter_qmp(qmsg, _filter) -def log(msg, filters=[], indent=None): +def log(msg, filters=(), indent=None): '''Logs either a string message or a JSON serializable message (like QMP). If indent is provided, JSON serializable messages are pretty-printed.''' for flt in filters: @@ -XXX,XX +XXX,XX @@ class VM(qtest.QEMUQtestMachine): result.append(filter_qmp_event(ev)) return result - def qmp_log(self, cmd, filters=[], indent=None, **kwargs): + def qmp_log(self, cmd, filters=(), indent=None, **kwargs): full_cmd = OrderedDict(( ("execute", cmd), ("arguments", ordered_qmp(kwargs)) @@ -XXX,XX +XXX,XX @@ def case_notrun(reason): open('%s/%s.casenotrun' % (output_dir, seq), 'a').write( ' [case not run] ' + reason + '\n') -def verify_image_format(supported_fmts=[], unsupported_fmts=[]): +def verify_image_format(supported_fmts=(), unsupported_fmts=()): assert not (supported_fmts and unsupported_fmts) if 'generic' in supported_fmts and \ @@ -XXX,XX +XXX,XX @@ def verify_image_format(supported_fmts=[], unsupported_fmts=[]): if not_sup or (imgfmt in unsupported_fmts): notrun('not suitable for this image format: %s' % imgfmt) -def verify_protocol(supported=[], unsupported=[]): +def verify_protocol(supported=(), unsupported=()): assert not (supported and unsupported) if 'generic' in supported: @@ -XXX,XX +XXX,XX @@ def verify_platform(supported=None, unsupported=None): if not any((sys.platform.startswith(x) for x in supported)): notrun('not suitable for this OS: %s' % sys.platform) -def verify_cache_mode(supported_cache_modes=[]): +def verify_cache_mode(supported_cache_modes=()): if supported_cache_modes and (cachemode not in supported_cache_modes): notrun('not suitable for this cache mode: %s' % cachemode) -def verify_aio_mode(supported_aio_modes=[]): +def verify_aio_mode(supported_aio_modes=()): if supported_aio_modes and (aiomode not in supported_aio_modes): notrun('not suitable for this aio mode: %s' % aiomode) @@ -XXX,XX +XXX,XX @@ def supported_formats(read_only=False): return supported_formats.formats[read_only] -def skip_if_unsupported(required_formats=[], read_only=False): +def skip_if_unsupported(required_formats=(), read_only=False): '''Skip Test Decorator Runs the test if all the required formats are whitelisted''' def skip_test_decorator(func): @@ -XXX,XX +XXX,XX @@ def execute_unittest(output, verbosity, debug): sys.stderr.write(out) def execute_test(test_function=None, - supported_fmts=[], + supported_fmts=(), supported_platforms=None, - supported_cache_modes=[], supported_aio_modes={}, - unsupported_fmts=[], supported_protocols=[], - unsupported_protocols=[]): + supported_cache_modes=(), supported_aio_modes=(), + unsupported_fmts=(), supported_protocols=(), + unsupported_protocols=()): """Run either unittest or script-style tests.""" # We are using TEST_DIR and QEMU_DEFAULT_MACHINE as proxies to -- 2.26.2 From: John Snow <jsnow@redhat.com> This allows others to get repeatable results with pylint. If you run `pylint iotests.py`, you should see a 100% pass. Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Message-Id: <20200331000014.11581-6-jsnow@redhat.com> Reviewed-by: Kevin Wolf <kwolf@redhat.com> Signed-off-by: Max Reitz <mreitz@redhat.com> --- tests/qemu-iotests/pylintrc | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) create mode 100644 tests/qemu-iotests/pylintrc diff --git a/tests/qemu-iotests/pylintrc b/tests/qemu-iotests/pylintrc new file mode 100644 index XXXXXXX..XXXXXXX --- /dev/null +++ b/tests/qemu-iotests/pylintrc @@ -XXX,XX +XXX,XX @@ +[MESSAGES CONTROL] + +# Disable the message, report, category or checker with the given id(s). You +# can either give multiple identifiers separated by comma (,) or put this +# option multiple times (only on the command line, not in the configuration +# file where it should appear only once). You can also use "--disable=all" to +# disable everything first and then reenable specific checks. For example, if +# you want to run only the similarities checker, you can use "--disable=all +# --enable=similarities". If you want to run only the classes checker, but have +# no Warning level messages displayed, use "--disable=all --enable=classes +# --disable=W". +disable=invalid-name, + no-else-return, + too-few-public-methods, + too-many-arguments, + too-many-branches, + too-many-lines, + too-many-locals, + too-many-public-methods, + # These are temporary, and should be removed: + line-too-long, + missing-docstring, -- 2.26.2 From: John Snow <jsnow@redhat.com> I had to fix a merge conflict, so do this tiny harmless thing while I'm here. Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Message-Id: <20200331000014.11581-7-jsnow@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Reviewed-by: Kevin Wolf <kwolf@redhat.com> Signed-off-by: Max Reitz <mreitz@redhat.com> --- tests/qemu-iotests/iotests.py | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py index XXXXXXX..XXXXXXX 100644 --- a/tests/qemu-iotests/iotests.py +++ b/tests/qemu-iotests/iotests.py @@ -XXX,XX +XXX,XX @@ # along with this program. If not, see <http://www.gnu.org/licenses/>. # +import atexit +from collections import OrderedDict +import faulthandler +import io +import json +import logging import os import re +import signal +import struct import subprocess -import unittest import sys -import struct -import json -import signal -import logging -import atexit -import io -from collections import OrderedDict -import faulthandler +import unittest # pylint: disable=import-error, wrong-import-position sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python')) -- 2.26.2 From: John Snow <jsnow@redhat.com> We no longer need to accommodate <3.4, drop this code. (The lines were > 79 chars and it stood out.) Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Message-Id: <20200331000014.11581-8-jsnow@redhat.com> Reviewed-by: Kevin Wolf <kwolf@redhat.com> Signed-off-by: Max Reitz <mreitz@redhat.com> --- tests/qemu-iotests/iotests.py | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py index XXXXXXX..XXXXXXX 100644 --- a/tests/qemu-iotests/iotests.py +++ b/tests/qemu-iotests/iotests.py @@ -XXX,XX +XXX,XX @@ def log(msg, filters=(), indent=None): for flt in filters: msg = flt(msg) if isinstance(msg, (dict, list)): - # Python < 3.4 needs to know not to add whitespace when pretty-printing: - separators = (', ', ': ') if indent is None else (',', ': ') # Don't sort if it's already sorted do_sort = not isinstance(msg, OrderedDict) - print(json.dumps(msg, sort_keys=do_sort, - indent=indent, separators=separators)) + print(json.dumps(msg, sort_keys=do_sort, indent=indent)) else: print(msg) -- 2.26.2 From: John Snow <jsnow@redhat.com> Representing nested, recursive data structures in mypy is notoriously difficult; the best we can reliably do right now is denote the leaf types as "Any" while describing the general shape of the data. Regardless, this fully annotates the log() function. Typing notes: TypeVar is a Type variable that can optionally be constrained by a sequence of possible types. This variable is bound to a specific type per-invocation, like a Generic. log() behaves as log<Msg>() now, where the incoming type informs the signature it expects for any filter arguments passed in. If Msg is a str, then filter should take and return a str. Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Message-Id: <20200331000014.11581-9-jsnow@redhat.com> Reviewed-by: Kevin Wolf <kwolf@redhat.com> Signed-off-by: Max Reitz <mreitz@redhat.com> --- tests/qemu-iotests/iotests.py | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py index XXXXXXX..XXXXXXX 100644 --- a/tests/qemu-iotests/iotests.py +++ b/tests/qemu-iotests/iotests.py @@ -XXX,XX +XXX,XX @@ import signal import struct import subprocess import sys +from typing import (Any, Callable, Dict, Iterable, List, Optional, TypeVar) import unittest # pylint: disable=import-error, wrong-import-position @@ -XXX,XX +XXX,XX @@ def filter_qmp_imgfmt(qmsg): return value return filter_qmp(qmsg, _filter) -def log(msg, filters=(), indent=None): - '''Logs either a string message or a JSON serializable message (like QMP). - If indent is provided, JSON serializable messages are pretty-printed.''' + +Msg = TypeVar('Msg', Dict[str, Any], List[Any], str) + +def log(msg: Msg, + filters: Iterable[Callable[[Msg], Msg]] = (), + indent: Optional[int] = None) -> None: + """ + Logs either a string message or a JSON serializable message (like QMP). + If indent is provided, JSON serializable messages are pretty-printed. + """ for flt in filters: msg = flt(msg) if isinstance(msg, (dict, list)): -- 2.26.2 From: John Snow <jsnow@redhat.com> 79 is the PEP8 recommendation. This recommendation works well for reading patch diffs in TUI email clients. Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Message-Id: <20200331000014.11581-10-jsnow@redhat.com> Reviewed-by: Kevin Wolf <kwolf@redhat.com> Signed-off-by: Max Reitz <mreitz@redhat.com> --- tests/qemu-iotests/iotests.py | 64 +++++++++++++++++++++++------------ tests/qemu-iotests/pylintrc | 6 +++- 2 files changed, 47 insertions(+), 23 deletions(-) diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py index XXXXXXX..XXXXXXX 100644 --- a/tests/qemu-iotests/iotests.py +++ b/tests/qemu-iotests/iotests.py @@ -XXX,XX +XXX,XX @@ luks_default_key_secret_opt = 'key-secret=keysec0' def qemu_img(*args): '''Run qemu-img and return the exit code''' devnull = open('/dev/null', 'r+') - exitcode = subprocess.call(qemu_img_args + list(args), stdin=devnull, stdout=devnull) + exitcode = subprocess.call(qemu_img_args + list(args), + stdin=devnull, stdout=devnull) if exitcode < 0: - sys.stderr.write('qemu-img received signal %i: %s\n' % (-exitcode, ' '.join(qemu_img_args + list(args)))) + sys.stderr.write('qemu-img received signal %i: %s\n' + % (-exitcode, ' '.join(qemu_img_args + list(args)))) return exitcode def ordered_qmp(qmsg, conv_keys=True): @@ -XXX,XX +XXX,XX @@ def qemu_img_verbose(*args): '''Run qemu-img without suppressing its output and return the exit code''' exitcode = subprocess.call(qemu_img_args + list(args)) if exitcode < 0: - sys.stderr.write('qemu-img received signal %i: %s\n' % (-exitcode, ' '.join(qemu_img_args + list(args)))) + sys.stderr.write('qemu-img received signal %i: %s\n' + % (-exitcode, ' '.join(qemu_img_args + list(args)))) return exitcode def qemu_img_pipe(*args): @@ -XXX,XX +XXX,XX @@ def qemu_img_pipe(*args): universal_newlines=True) exitcode = subp.wait() if exitcode < 0: - sys.stderr.write('qemu-img received signal %i: %s\n' % (-exitcode, ' '.join(qemu_img_args + list(args)))) + sys.stderr.write('qemu-img received signal %i: %s\n' + % (-exitcode, ' '.join(qemu_img_args + list(args)))) return subp.communicate()[0] def qemu_img_log(*args): @@ -XXX,XX +XXX,XX @@ def qemu_io(*args): universal_newlines=True) exitcode = subp.wait() if exitcode < 0: - sys.stderr.write('qemu-io received signal %i: %s\n' % (-exitcode, ' '.join(args))) + sys.stderr.write('qemu-io received signal %i: %s\n' + % (-exitcode, ' '.join(args))) return subp.communicate()[0] def qemu_io_log(*args): @@ -XXX,XX +XXX,XX @@ win32_re = re.compile(r"\r") def filter_win32(msg): return win32_re.sub("", msg) -qemu_io_re = re.compile(r"[0-9]* ops; [0-9\/:. sec]* \([0-9\/.inf]* [EPTGMKiBbytes]*\/sec and [0-9\/.inf]* ops\/sec\)") +qemu_io_re = re.compile(r"[0-9]* ops; [0-9\/:. sec]* " + r"\([0-9\/.inf]* [EPTGMKiBbytes]*\/sec " + r"and [0-9\/.inf]* ops\/sec\)") def filter_qemu_io(msg): msg = filter_win32(msg) - return qemu_io_re.sub("X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)", msg) + return qemu_io_re.sub("X ops; XX:XX:XX.X " + "(XXX YYY/sec and XXX ops/sec)", msg) chown_re = re.compile(r"chown [0-9]+:[0-9]+") def filter_chown(msg): @@ -XXX,XX +XXX,XX @@ def filter_img_info(output, filename): line = filter_testfiles(line) line = line.replace(imgfmt, 'IMGFMT') line = re.sub('iters: [0-9]+', 'iters: XXX', line) - line = re.sub('uuid: [-a-f0-9]+', 'uuid: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX', line) + line = re.sub('uuid: [-a-f0-9]+', + 'uuid: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX', + line) line = re.sub('cid: [0-9]+', 'cid: XXXXXXXXXX', line) lines.append(line) return '\n'.join(lines) @@ -XXX,XX +XXX,XX @@ class VM(qtest.QEMUQtestMachine): self.pause_drive(drive, "write_aio") return self.qmp('human-monitor-command', - command_line='qemu-io %s "break %s bp_%s"' % (drive, event, drive)) + command_line='qemu-io %s "break %s bp_%s"' + % (drive, event, drive)) def resume_drive(self, drive): self.qmp('human-monitor-command', - command_line='qemu-io %s "remove_break bp_%s"' % (drive, drive)) + command_line='qemu-io %s "remove_break bp_%s"' + % (drive, drive)) def hmp_qemu_io(self, drive, cmd): '''Write to a given drive using an HMP command''' @@ -XXX,XX +XXX,XX @@ class QMPTestCase(unittest.TestCase): idx = int(idx) if not isinstance(d, dict) or component not in d: - self.fail('failed path traversal for "%s" in "%s"' % (path, str(d))) + self.fail(f'failed path traversal for "{path}" in "{d}"') d = d[component] if m: if not isinstance(d, list): - self.fail('path component "%s" in "%s" is not a list in "%s"' % (component, path, str(d))) + self.fail(f'path component "{component}" in "{path}" ' + f'is not a list in "{d}"') try: d = d[idx] except IndexError: - self.fail('invalid index "%s" in path "%s" in "%s"' % (idx, path, str(d))) + self.fail(f'invalid index "{idx}" in path "{path}" ' + f'in "{d}"') return d def assert_qmp_absent(self, d, path): @@ -XXX,XX +XXX,XX @@ class QMPTestCase(unittest.TestCase): '''Asserts that the given filename is a json: filename and that its content is equal to the given reference object''' self.assertEqual(json_filename[:5], 'json:') - self.assertEqual(self.vm.flatten_qmp_object(json.loads(json_filename[5:])), - self.vm.flatten_qmp_object(reference)) + self.assertEqual( + self.vm.flatten_qmp_object(json.loads(json_filename[5:])), + self.vm.flatten_qmp_object(reference) + ) - def cancel_and_wait(self, drive='drive0', force=False, resume=False, wait=60.0): + def cancel_and_wait(self, drive='drive0', force=False, + resume=False, wait=60.0): '''Cancel a block job and wait for it to finish, returning the event''' result = self.vm.qmp('block-job-cancel', device=drive, force=force) self.assert_qmp(result, 'return', {}) @@ -XXX,XX +XXX,XX @@ class QMPTestCase(unittest.TestCase): self.assert_no_active_block_jobs() return result - def wait_until_completed(self, drive='drive0', check_offset=True, wait=60.0, - error=None): + def wait_until_completed(self, drive='drive0', check_offset=True, + wait=60.0, error=None): '''Wait for a block job to finish, returning the event''' while True: for event in self.vm.get_qmp_events(wait=wait): @@ -XXX,XX +XXX,XX @@ def verify_quorum(): notrun('quorum support missing') def qemu_pipe(*args): - '''Run qemu with an option to print something and exit (e.g. a help option), - and return its output''' + """ + Run qemu with an option to print something and exit (e.g. a help option). + + :return: QEMU's stdout output. + """ args = [qemu_prog] + qemu_opts + list(args) subp = subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, @@ -XXX,XX +XXX,XX @@ def skip_if_unsupported(required_formats=(), read_only=False): usf_list = list(set(fmts) - set(supported_formats(read_only))) if usf_list: - test_case.case_skip('{}: formats {} are not whitelisted'.format( - test_case, usf_list)) + msg = f'{test_case}: formats {usf_list} are not whitelisted' + test_case.case_skip(msg) return None else: return func(test_case, *args, **kwargs) diff --git a/tests/qemu-iotests/pylintrc b/tests/qemu-iotests/pylintrc index XXXXXXX..XXXXXXX 100644 --- a/tests/qemu-iotests/pylintrc +++ b/tests/qemu-iotests/pylintrc @@ -XXX,XX +XXX,XX @@ disable=invalid-name, too-many-locals, too-many-public-methods, # These are temporary, and should be removed: - line-too-long, missing-docstring, + +[FORMAT] + +# Maximum number of characters on a single line. +max-line-length=79 -- 2.26.2 From: John Snow <jsnow@redhat.com> Minor cleanup for HMP functions; helps with line length and consolidates HMP helpers through one implementation function. Although we are adding a universal toggle to turn QMP logging on or off, many existing callers to hmp functions don't expect that output to be logged, which causes quite a few changes in the test output. For now, offer a use_log parameter. Typing notes: QMPResponse is just an alias for Dict[str, Any]. It holds no special meanings and it is not a formal subtype of Dict[str, Any]. It is best thought of as a lexical synonym. We may well wish to add stricter subtypes in the future for certain shapes of data that are not formalized as Python objects, at which point we can simply retire the alias and allow mypy to more strictly check usages of the name. Signed-off-by: John Snow <jsnow@redhat.com> Message-Id: <20200331000014.11581-11-jsnow@redhat.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Reviewed-by: Kevin Wolf <kwolf@redhat.com> Signed-off-by: Max Reitz <mreitz@redhat.com> --- tests/qemu-iotests/iotests.py | 39 +++++++++++++++++++++-------------- 1 file changed, 24 insertions(+), 15 deletions(-) diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py index XXXXXXX..XXXXXXX 100644 --- a/tests/qemu-iotests/iotests.py +++ b/tests/qemu-iotests/iotests.py @@ -XXX,XX +XXX,XX @@ from qemu import qtest assert sys.version_info >= (3, 6) +# Type Aliases +QMPResponse = Dict[str, Any] + + faulthandler.enable() # This will not work if arguments contain spaces but is necessary if we @@ -XXX,XX +XXX,XX @@ class VM(qtest.QEMUQtestMachine): self._args.append(addr) return self - def pause_drive(self, drive, event=None): - '''Pause drive r/w operations''' + def hmp(self, command_line: str, use_log: bool = False) -> QMPResponse: + cmd = 'human-monitor-command' + kwargs = {'command-line': command_line} + if use_log: + return self.qmp_log(cmd, **kwargs) + else: + return self.qmp(cmd, **kwargs) + + def pause_drive(self, drive: str, event: Optional[str] = None) -> None: + """Pause drive r/w operations""" if not event: self.pause_drive(drive, "read_aio") self.pause_drive(drive, "write_aio") return - self.qmp('human-monitor-command', - command_line='qemu-io %s "break %s bp_%s"' - % (drive, event, drive)) - - def resume_drive(self, drive): - self.qmp('human-monitor-command', - command_line='qemu-io %s "remove_break bp_%s"' - % (drive, drive)) - - def hmp_qemu_io(self, drive, cmd): - '''Write to a given drive using an HMP command''' - return self.qmp('human-monitor-command', - command_line='qemu-io %s "%s"' % (drive, cmd)) + self.hmp(f'qemu-io {drive} "break {event} bp_{drive}"') + + def resume_drive(self, drive: str) -> None: + """Resume drive r/w operations""" + self.hmp(f'qemu-io {drive} "remove_break bp_{drive}"') + + def hmp_qemu_io(self, drive: str, cmd: str, + use_log: bool = False) -> QMPResponse: + """Write to a given drive using an HMP command""" + return self.hmp(f'qemu-io {drive} "{cmd}"', use_log=use_log) def flatten_qmp_object(self, obj, output=None, basestr=''): if output is None: -- 2.26.2 From: John Snow <jsnow@redhat.com> Like script_main, but doesn't require a single point of entry. Replace all existing initialization sections with this drop-in replacement. This brings debug support to all existing script-style iotests. Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Message-Id: <20200331000014.11581-12-jsnow@redhat.com> Reviewed-by: Kevin Wolf <kwolf@redhat.com> [mreitz: Give 274 the same treatment] Signed-off-by: Max Reitz <mreitz@redhat.com> --- tests/qemu-iotests/149 | 3 +- tests/qemu-iotests/194 | 4 +- tests/qemu-iotests/202 | 4 +- tests/qemu-iotests/203 | 4 +- tests/qemu-iotests/206 | 2 +- tests/qemu-iotests/207 | 6 ++- tests/qemu-iotests/208 | 2 +- tests/qemu-iotests/209 | 2 +- tests/qemu-iotests/210 | 6 ++- tests/qemu-iotests/211 | 6 ++- tests/qemu-iotests/212 | 6 ++- tests/qemu-iotests/213 | 6 ++- tests/qemu-iotests/216 | 4 +- tests/qemu-iotests/218 | 2 +- tests/qemu-iotests/219 | 2 +- tests/qemu-iotests/222 | 7 ++-- tests/qemu-iotests/224 | 4 +- tests/qemu-iotests/228 | 6 ++- tests/qemu-iotests/234 | 4 +- tests/qemu-iotests/235 | 4 +- tests/qemu-iotests/236 | 2 +- tests/qemu-iotests/237 | 2 +- tests/qemu-iotests/238 | 2 + tests/qemu-iotests/242 | 2 +- tests/qemu-iotests/246 | 2 +- tests/qemu-iotests/248 | 2 +- tests/qemu-iotests/254 | 2 +- tests/qemu-iotests/255 | 2 +- tests/qemu-iotests/256 | 2 +- tests/qemu-iotests/258 | 7 ++-- tests/qemu-iotests/260 | 4 +- tests/qemu-iotests/262 | 4 +- tests/qemu-iotests/264 | 4 +- tests/qemu-iotests/274 | 4 +- tests/qemu-iotests/277 | 2 + tests/qemu-iotests/280 | 8 ++-- tests/qemu-iotests/283 | 4 +- tests/qemu-iotests/iotests.py | 76 +++++++++++++++++++++++------------ 38 files changed, 132 insertions(+), 83 deletions(-) diff --git a/tests/qemu-iotests/149 b/tests/qemu-iotests/149 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/149 +++ b/tests/qemu-iotests/149 @@ -XXX,XX +XXX,XX @@ def test_once(config, qemu_img=False): # Obviously we only work with the luks image format -iotests.verify_image_format(supported_fmts=['luks']) -iotests.verify_platform() +iotests.script_initialize(supported_fmts=['luks']) # We need sudo in order to run cryptsetup to create # dm-crypt devices. This is safe to use on any diff --git a/tests/qemu-iotests/194 b/tests/qemu-iotests/194 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/194 +++ b/tests/qemu-iotests/194 @@ -XXX,XX +XXX,XX @@ import iotests -iotests.verify_image_format(supported_fmts=['qcow2', 'qed', 'raw']) -iotests.verify_platform(['linux']) +iotests.script_initialize(supported_fmts=['qcow2', 'qed', 'raw'], + supported_platforms=['linux']) with iotests.FilePath('source.img') as source_img_path, \ iotests.FilePath('dest.img') as dest_img_path, \ diff --git a/tests/qemu-iotests/202 b/tests/qemu-iotests/202 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/202 +++ b/tests/qemu-iotests/202 @@ -XXX,XX +XXX,XX @@ import iotests -iotests.verify_image_format(supported_fmts=['qcow2']) -iotests.verify_platform(['linux']) +iotests.script_initialize(supported_fmts=['qcow2'], + supported_platforms=['linux']) with iotests.FilePath('disk0.img') as disk0_img_path, \ iotests.FilePath('disk1.img') as disk1_img_path, \ diff --git a/tests/qemu-iotests/203 b/tests/qemu-iotests/203 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/203 +++ b/tests/qemu-iotests/203 @@ -XXX,XX +XXX,XX @@ import iotests -iotests.verify_image_format(supported_fmts=['qcow2']) -iotests.verify_platform(['linux']) +iotests.script_initialize(supported_fmts=['qcow2'], + supported_platforms=['linux']) with iotests.FilePath('disk0.img') as disk0_img_path, \ iotests.FilePath('disk1.img') as disk1_img_path, \ diff --git a/tests/qemu-iotests/206 b/tests/qemu-iotests/206 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/206 +++ b/tests/qemu-iotests/206 @@ -XXX,XX +XXX,XX @@ import iotests from iotests import imgfmt -iotests.verify_image_format(supported_fmts=['qcow2']) +iotests.script_initialize(supported_fmts=['qcow2']) with iotests.FilePath('t.qcow2') as disk_path, \ iotests.FilePath('t.qcow2.base') as backing_path, \ diff --git a/tests/qemu-iotests/207 b/tests/qemu-iotests/207 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/207 +++ b/tests/qemu-iotests/207 @@ -XXX,XX +XXX,XX @@ import iotests import subprocess import re -iotests.verify_image_format(supported_fmts=['raw']) -iotests.verify_protocol(supported=['ssh']) +iotests.script_initialize( + supported_fmts=['raw'], + supported_protocols=['ssh'], +) def filter_hash(qmsg): def _filter(key, value): diff --git a/tests/qemu-iotests/208 b/tests/qemu-iotests/208 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/208 +++ b/tests/qemu-iotests/208 @@ -XXX,XX +XXX,XX @@ import iotests -iotests.verify_image_format(supported_fmts=['generic']) +iotests.script_initialize(supported_fmts=['generic']) with iotests.FilePath('disk.img') as disk_img_path, \ iotests.FilePath('disk-snapshot.img') as disk_snapshot_img_path, \ diff --git a/tests/qemu-iotests/209 b/tests/qemu-iotests/209 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/209 +++ b/tests/qemu-iotests/209 @@ -XXX,XX +XXX,XX @@ import iotests from iotests import qemu_img_create, qemu_io, qemu_img_verbose, qemu_nbd, \ file_path -iotests.verify_image_format(supported_fmts=['qcow2']) +iotests.script_initialize(supported_fmts=['qcow2']) disk = file_path('disk') nbd_sock = file_path('nbd-sock', base_dir=iotests.sock_dir) diff --git a/tests/qemu-iotests/210 b/tests/qemu-iotests/210 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/210 +++ b/tests/qemu-iotests/210 @@ -XXX,XX +XXX,XX @@ import iotests from iotests import imgfmt -iotests.verify_image_format(supported_fmts=['luks']) -iotests.verify_protocol(supported=['file']) +iotests.script_initialize( + supported_fmts=['luks'], + supported_protocols=['file'], +) with iotests.FilePath('t.luks') as disk_path, \ iotests.VM() as vm: diff --git a/tests/qemu-iotests/211 b/tests/qemu-iotests/211 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/211 +++ b/tests/qemu-iotests/211 @@ -XXX,XX +XXX,XX @@ import iotests from iotests import imgfmt -iotests.verify_image_format(supported_fmts=['vdi']) -iotests.verify_protocol(supported=['file']) +iotests.script_initialize( + supported_fmts=['vdi'], + supported_protocols=['file'], +) def blockdev_create(vm, options): error = vm.blockdev_create(options) diff --git a/tests/qemu-iotests/212 b/tests/qemu-iotests/212 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/212 +++ b/tests/qemu-iotests/212 @@ -XXX,XX +XXX,XX @@ import iotests from iotests import imgfmt -iotests.verify_image_format(supported_fmts=['parallels']) -iotests.verify_protocol(supported=['file']) +iotests.script_initialize( + supported_fmts=['parallels'], + supported_protocols=['file'], +) with iotests.FilePath('t.parallels') as disk_path, \ iotests.VM() as vm: diff --git a/tests/qemu-iotests/213 b/tests/qemu-iotests/213 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/213 +++ b/tests/qemu-iotests/213 @@ -XXX,XX +XXX,XX @@ import iotests from iotests import imgfmt -iotests.verify_image_format(supported_fmts=['vhdx']) -iotests.verify_protocol(supported=['file']) +iotests.script_initialize( + supported_fmts=['vhdx'], + supported_protocols=['file'], +) with iotests.FilePath('t.vhdx') as disk_path, \ iotests.VM() as vm: diff --git a/tests/qemu-iotests/216 b/tests/qemu-iotests/216 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/216 +++ b/tests/qemu-iotests/216 @@ -XXX,XX +XXX,XX @@ import iotests from iotests import log, qemu_img, qemu_io_silent # Need backing file support -iotests.verify_image_format(supported_fmts=['qcow2', 'qcow', 'qed', 'vmdk']) -iotests.verify_platform(['linux']) +iotests.script_initialize(supported_fmts=['qcow2', 'qcow', 'qed', 'vmdk'], + supported_platforms=['linux']) log('') log('=== Copy-on-read across nodes ===') diff --git a/tests/qemu-iotests/218 b/tests/qemu-iotests/218 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/218 +++ b/tests/qemu-iotests/218 @@ -XXX,XX +XXX,XX @@ import iotests from iotests import log, qemu_img, qemu_io_silent -iotests.verify_image_format(supported_fmts=['qcow2', 'raw']) +iotests.script_initialize(supported_fmts=['qcow2', 'raw']) # Launches the VM, adds two null-co nodes (source and target), and diff --git a/tests/qemu-iotests/219 b/tests/qemu-iotests/219 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/219 +++ b/tests/qemu-iotests/219 @@ -XXX,XX +XXX,XX @@ import iotests -iotests.verify_image_format(supported_fmts=['qcow2']) +iotests.script_initialize(supported_fmts=['qcow2']) img_size = 4 * 1024 * 1024 diff --git a/tests/qemu-iotests/222 b/tests/qemu-iotests/222 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/222 +++ b/tests/qemu-iotests/222 @@ -XXX,XX +XXX,XX @@ import iotests from iotests import log, qemu_img, qemu_io, qemu_io_silent -iotests.verify_platform(['linux']) -iotests.verify_image_format(supported_fmts=['qcow2', 'qcow', 'qed', 'vmdk', - 'vhdx', 'raw']) +iotests.script_initialize( + supported_fmts=['qcow2', 'qcow', 'qed', 'vmdk', 'vhdx', 'raw'], + supported_platforms=['linux'], +) patterns = [("0x5d", "0", "64k"), ("0xd5", "1M", "64k"), diff --git a/tests/qemu-iotests/224 b/tests/qemu-iotests/224 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/224 +++ b/tests/qemu-iotests/224 @@ -XXX,XX +XXX,XX @@ from iotests import log, qemu_img, qemu_io_silent, filter_qmp_testfiles, \ import json # Need backing file support (for arbitrary backing formats) -iotests.verify_image_format(supported_fmts=['qcow2', 'qcow', 'qed']) -iotests.verify_platform(['linux']) +iotests.script_initialize(supported_fmts=['qcow2', 'qcow', 'qed'], + supported_platforms=['linux']) # There are two variations of this test: diff --git a/tests/qemu-iotests/228 b/tests/qemu-iotests/228 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/228 +++ b/tests/qemu-iotests/228 @@ -XXX,XX +XXX,XX @@ from iotests import log, qemu_img, filter_testfiles, filter_imgfmt, \ filter_qmp_testfiles, filter_qmp_imgfmt # Need backing file and change-backing-file support -iotests.verify_image_format(supported_fmts=['qcow2', 'qed']) -iotests.verify_platform(['linux']) +iotests.script_initialize( + supported_fmts=['qcow2', 'qed'], + supported_platforms=['linux'], +) def log_node_info(node): diff --git a/tests/qemu-iotests/234 b/tests/qemu-iotests/234 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/234 +++ b/tests/qemu-iotests/234 @@ -XXX,XX +XXX,XX @@ import iotests import os -iotests.verify_image_format(supported_fmts=['qcow2']) -iotests.verify_platform(['linux']) +iotests.script_initialize(supported_fmts=['qcow2'], + supported_platforms=['linux']) with iotests.FilePath('img') as img_path, \ iotests.FilePath('backing') as backing_path, \ diff --git a/tests/qemu-iotests/235 b/tests/qemu-iotests/235 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/235 +++ b/tests/qemu-iotests/235 @@ -XXX,XX +XXX,XX @@ sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python')) from qemu.machine import QEMUMachine +iotests.script_initialize(supported_fmts=['qcow2']) + # Note: # This test was added to check that mirror dead-lock was fixed (see previous # commit before this test addition). @@ -XXX,XX +XXX,XX @@ from qemu.machine import QEMUMachine size = 1 * 1024 * 1024 * 1024 -iotests.verify_image_format(supported_fmts=['qcow2']) - disk = file_path('disk') # prepare source image diff --git a/tests/qemu-iotests/236 b/tests/qemu-iotests/236 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/236 +++ b/tests/qemu-iotests/236 @@ -XXX,XX +XXX,XX @@ import iotests from iotests import log -iotests.verify_image_format(supported_fmts=['generic']) +iotests.script_initialize(supported_fmts=['generic']) size = 64 * 1024 * 1024 granularity = 64 * 1024 diff --git a/tests/qemu-iotests/237 b/tests/qemu-iotests/237 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/237 +++ b/tests/qemu-iotests/237 @@ -XXX,XX +XXX,XX @@ import math import iotests from iotests import imgfmt -iotests.verify_image_format(supported_fmts=['vmdk']) +iotests.script_initialize(supported_fmts=['vmdk']) with iotests.FilePath('t.vmdk') as disk_path, \ iotests.FilePath('t.vmdk.1') as extent1_path, \ diff --git a/tests/qemu-iotests/238 b/tests/qemu-iotests/238 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/238 +++ b/tests/qemu-iotests/238 @@ -XXX,XX +XXX,XX @@ import os import iotests from iotests import log +iotests.script_initialize() + virtio_scsi_device = iotests.get_virtio_scsi_device() vm = iotests.VM() diff --git a/tests/qemu-iotests/242 b/tests/qemu-iotests/242 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/242 +++ b/tests/qemu-iotests/242 @@ -XXX,XX +XXX,XX @@ import struct from iotests import qemu_img_create, qemu_io, qemu_img_pipe, \ file_path, img_info_log, log, filter_qemu_io -iotests.verify_image_format(supported_fmts=['qcow2']) +iotests.script_initialize(supported_fmts=['qcow2']) disk = file_path('disk') chunk = 256 * 1024 diff --git a/tests/qemu-iotests/246 b/tests/qemu-iotests/246 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/246 +++ b/tests/qemu-iotests/246 @@ -XXX,XX +XXX,XX @@ import iotests from iotests import log -iotests.verify_image_format(supported_fmts=['qcow2']) +iotests.script_initialize(supported_fmts=['qcow2']) size = 64 * 1024 * 1024 * 1024 gran_small = 32 * 1024 gran_large = 128 * 1024 diff --git a/tests/qemu-iotests/248 b/tests/qemu-iotests/248 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/248 +++ b/tests/qemu-iotests/248 @@ -XXX,XX +XXX,XX @@ import iotests from iotests import qemu_img_create, qemu_io, file_path, filter_qmp_testfiles -iotests.verify_image_format(supported_fmts=['qcow2']) +iotests.script_initialize(supported_fmts=['qcow2']) source, target = file_path('source', 'target') size = 5 * 1024 * 1024 diff --git a/tests/qemu-iotests/254 b/tests/qemu-iotests/254 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/254 +++ b/tests/qemu-iotests/254 @@ -XXX,XX +XXX,XX @@ import iotests from iotests import qemu_img_create, file_path, log -iotests.verify_image_format(supported_fmts=['qcow2']) +iotests.script_initialize(supported_fmts=['qcow2']) disk, top = file_path('disk', 'top') size = 1024 * 1024 diff --git a/tests/qemu-iotests/255 b/tests/qemu-iotests/255 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/255 +++ b/tests/qemu-iotests/255 @@ -XXX,XX +XXX,XX @@ import iotests from iotests import imgfmt -iotests.verify_image_format(supported_fmts=['qcow2']) +iotests.script_initialize(supported_fmts=['qcow2']) iotests.log('Finishing a commit job with background reads') iotests.log('============================================') diff --git a/tests/qemu-iotests/256 b/tests/qemu-iotests/256 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/256 +++ b/tests/qemu-iotests/256 @@ -XXX,XX +XXX,XX @@ import os import iotests from iotests import log -iotests.verify_image_format(supported_fmts=['qcow2']) +iotests.script_initialize(supported_fmts=['qcow2']) size = 64 * 1024 * 1024 with iotests.FilePath('img0') as img0_path, \ diff --git a/tests/qemu-iotests/258 b/tests/qemu-iotests/258 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/258 +++ b/tests/qemu-iotests/258 @@ -XXX,XX +XXX,XX @@ from iotests import log, qemu_img, qemu_io_silent, \ filter_qmp_testfiles, filter_qmp_imgfmt # Need backing file and change-backing-file support -iotests.verify_image_format(supported_fmts=['qcow2', 'qed']) -iotests.verify_platform(['linux']) - +iotests.script_initialize( + supported_fmts=['qcow2', 'qed'], + supported_platforms=['linux'], +) # Returns a node for blockdev-add def node(node_name, path, backing=None, fmt=None, throttle=None): diff --git a/tests/qemu-iotests/260 b/tests/qemu-iotests/260 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/260 +++ b/tests/qemu-iotests/260 @@ -XXX,XX +XXX,XX @@ import iotests from iotests import qemu_img_create, file_path, log, filter_qmp_event -iotests.verify_image_format(supported_fmts=['qcow2']) +iotests.script_initialize( + supported_fmts=['qcow2'] +) base, top = file_path('base', 'top') size = 64 * 1024 * 3 diff --git a/tests/qemu-iotests/262 b/tests/qemu-iotests/262 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/262 +++ b/tests/qemu-iotests/262 @@ -XXX,XX +XXX,XX @@ import iotests import os -iotests.verify_image_format(supported_fmts=['qcow2']) -iotests.verify_platform(['linux']) +iotests.script_initialize(supported_fmts=['qcow2'], + supported_platforms=['linux']) with iotests.FilePath('img') as img_path, \ iotests.FilePath('mig_fifo') as fifo, \ diff --git a/tests/qemu-iotests/264 b/tests/qemu-iotests/264 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/264 +++ b/tests/qemu-iotests/264 @@ -XXX,XX +XXX,XX @@ import iotests from iotests import qemu_img_create, qemu_io_silent_check, file_path, \ qemu_nbd_popen, log -iotests.verify_image_format(supported_fmts=['qcow2']) +iotests.script_initialize( + supported_fmts=['qcow2'], +) disk_a, disk_b, nbd_sock = file_path('disk_a', 'disk_b', 'nbd-sock') nbd_uri = 'nbd+unix:///?socket=' + nbd_sock diff --git a/tests/qemu-iotests/274 b/tests/qemu-iotests/274 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/274 +++ b/tests/qemu-iotests/274 @@ -XXX,XX +XXX,XX @@ import iotests -iotests.verify_image_format(supported_fmts=['qcow2']) -iotests.verify_platform(['linux']) +iotests.script_initialize(supported_fmts=['qcow2'], + supported_platforms=['linux']) size_short = 1 * 1024 * 1024 size_long = 2 * 1024 * 1024 diff --git a/tests/qemu-iotests/277 b/tests/qemu-iotests/277 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/277 +++ b/tests/qemu-iotests/277 @@ -XXX,XX +XXX,XX @@ import subprocess import iotests from iotests import file_path, log +iotests.script_initialize() + nbd_sock, conf_file = file_path('nbd-sock', 'nbd-fault-injector.conf') diff --git a/tests/qemu-iotests/280 b/tests/qemu-iotests/280 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/280 +++ b/tests/qemu-iotests/280 @@ -XXX,XX +XXX,XX @@ import iotests import os -iotests.verify_image_format(supported_fmts=['qcow2']) -iotests.verify_protocol(supported=['file']) -iotests.verify_platform(['linux']) +iotests.script_initialize( + supported_fmts=['qcow2'], + supported_protocols=['file'], + supported_platforms=['linux'], +) with iotests.FilePath('base') as base_path , \ iotests.FilePath('top') as top_path, \ diff --git a/tests/qemu-iotests/283 b/tests/qemu-iotests/283 index XXXXXXX..XXXXXXX 100644 --- a/tests/qemu-iotests/283 +++ b/tests/qemu-iotests/283 @@ -XXX,XX +XXX,XX @@ import iotests # The test is unrelated to formats, restrict it to qcow2 to avoid extra runs -iotests.verify_image_format(supported_fmts=['qcow2']) +iotests.script_initialize( + supported_fmts=['qcow2'], +) size = 1024 * 1024 diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py index XXXXXXX..XXXXXXX 100644 --- a/tests/qemu-iotests/iotests.py +++ b/tests/qemu-iotests/iotests.py @@ -XXX,XX +XXX,XX @@ import signal import struct import subprocess import sys -from typing import (Any, Callable, Dict, Iterable, List, Optional, TypeVar) +from typing import (Any, Callable, Dict, Iterable, + List, Optional, Sequence, TypeVar) import unittest # pylint: disable=import-error, wrong-import-position @@ -XXX,XX +XXX,XX @@ def verify_protocol(supported=(), unsupported=()): if not_sup or (imgproto in unsupported): notrun('not suitable for this protocol: %s' % imgproto) -def verify_platform(supported=None, unsupported=None): - if unsupported is not None: - if any((sys.platform.startswith(x) for x in unsupported)): - notrun('not suitable for this OS: %s' % sys.platform) +def verify_platform(supported=(), unsupported=()): + if any((sys.platform.startswith(x) for x in unsupported)): + notrun('not suitable for this OS: %s' % sys.platform) - if supported is not None: + if supported: if not any((sys.platform.startswith(x) for x in supported)): notrun('not suitable for this OS: %s' % sys.platform) @@ -XXX,XX +XXX,XX @@ def skip_if_user_is_root(func): return func(*args, **kwargs) return func_wrapper -def execute_unittest(output, verbosity, debug): +def execute_unittest(debug=False): + """Executes unittests within the calling module.""" + + verbosity = 2 if debug else 1 + + if debug: + output = sys.stdout + else: + # We need to filter out the time taken from the output so that + # qemu-iotest can reliably diff the results against master output. + output = io.StringIO() + runner = unittest.TextTestRunner(stream=output, descriptions=True, verbosity=verbosity) try: @@ -XXX,XX +XXX,XX @@ def execute_unittest(output, verbosity, debug): # exception unittest.main(testRunner=runner) finally: + # We need to filter out the time taken from the output so that + # qemu-iotest can reliably diff the results against master output. if not debug: out = output.getvalue() out = re.sub(r'Ran (\d+) tests? in [\d.]+s', r'Ran \1 tests', out) @@ -XXX,XX +XXX,XX @@ def execute_unittest(output, verbosity, debug): sys.stderr.write(out) -def execute_test(test_function=None, - supported_fmts=(), - supported_platforms=None, - supported_cache_modes=(), supported_aio_modes=(), - unsupported_fmts=(), supported_protocols=(), - unsupported_protocols=()): - """Run either unittest or script-style tests.""" +def execute_setup_common(supported_fmts: Sequence[str] = (), + supported_platforms: Sequence[str] = (), + supported_cache_modes: Sequence[str] = (), + supported_aio_modes: Sequence[str] = (), + unsupported_fmts: Sequence[str] = (), + supported_protocols: Sequence[str] = (), + unsupported_protocols: Sequence[str] = ()) -> bool: + """ + Perform necessary setup for either script-style or unittest-style tests. + + :return: Bool; Whether or not debug mode has been requested via the CLI. + """ + # Note: Python 3.6 and pylint do not like 'Collection' so use 'Sequence'. # We are using TEST_DIR and QEMU_DEFAULT_MACHINE as proxies to # indicate that we're not being run via "check". There may be @@ -XXX,XX +XXX,XX @@ def execute_test(test_function=None, sys.stderr.write('Please run this test via the "check" script\n') sys.exit(os.EX_USAGE) - debug = '-d' in sys.argv - verbosity = 1 verify_image_format(supported_fmts, unsupported_fmts) verify_protocol(supported_protocols, unsupported_protocols) verify_platform(supported=supported_platforms) verify_cache_mode(supported_cache_modes) verify_aio_mode(supported_aio_modes) + debug = '-d' in sys.argv if debug: - output = sys.stdout - verbosity = 2 sys.argv.remove('-d') - else: - # We need to filter out the time taken from the output so that - # qemu-iotest can reliably diff the results against master output. - output = io.StringIO() - logging.basicConfig(level=(logging.DEBUG if debug else logging.WARN)) + return debug + +def execute_test(*args, test_function=None, **kwargs): + """Run either unittest or script-style tests.""" + + debug = execute_setup_common(*args, **kwargs) if not test_function: - execute_unittest(output, verbosity, debug) + execute_unittest(debug) else: test_function() +# This is called from script-style iotests without a single point of entry +def script_initialize(*args, **kwargs): + """Initialize script-style tests without running any tests.""" + execute_setup_common(*args, **kwargs) + +# This is called from script-style iotests with a single point of entry def script_main(test_function, *args, **kwargs): """Run script-style tests outside of the unittest framework""" - execute_test(test_function, *args, **kwargs) + execute_test(*args, test_function=test_function, **kwargs) +# This is called from unittest style iotests def main(*args, **kwargs): """Run tests using the unittest framework""" - execute_test(None, *args, **kwargs) + execute_test(*args, **kwargs) -- 2.26.2 From: John Snow <jsnow@redhat.com> Since this one is nicely factored to use a single entry point, use script_main to run the tests. Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Message-Id: <20200331000014.11581-13-jsnow@redhat.com> Reviewed-by: Kevin Wolf <kwolf@redhat.com> Signed-off-by: Max Reitz <mreitz@redhat.com> --- tests/qemu-iotests/258 | 11 ++++------- 1 file changed, 4 insertions(+), 7 deletions(-) diff --git a/tests/qemu-iotests/258 b/tests/qemu-iotests/258 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/258 +++ b/tests/qemu-iotests/258 @@ -XXX,XX +XXX,XX @@ import iotests from iotests import log, qemu_img, qemu_io_silent, \ filter_qmp_testfiles, filter_qmp_imgfmt -# Need backing file and change-backing-file support -iotests.script_initialize( - supported_fmts=['qcow2', 'qed'], - supported_platforms=['linux'], -) - # Returns a node for blockdev-add def node(node_name, path, backing=None, fmt=None, throttle=None): if fmt is None: @@ -XXX,XX +XXX,XX @@ def main(): test_concurrent_finish(False) if __name__ == '__main__': - main() + # Need backing file and change-backing-file support + iotests.script_main(main, + supported_fmts=['qcow2', 'qed'], + supported_platforms=['linux']) -- 2.26.2 From: John Snow <jsnow@redhat.com> Mark the verify functions as "private" with a leading underscore, to discourage their use. Update type signatures while we're here. (Also, make pending patches not yet using the new entry points fail in a very obvious way.) Signed-off-by: John Snow <jsnow@redhat.com> Message-Id: <20200331000014.11581-14-jsnow@redhat.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Reviewed-by: Kevin Wolf <kwolf@redhat.com> Signed-off-by: Max Reitz <mreitz@redhat.com> --- tests/qemu-iotests/iotests.py | 23 +++++++++++++---------- 1 file changed, 13 insertions(+), 10 deletions(-) diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py index XXXXXXX..XXXXXXX 100644 --- a/tests/qemu-iotests/iotests.py +++ b/tests/qemu-iotests/iotests.py @@ -XXX,XX +XXX,XX @@ def case_notrun(reason): open('%s/%s.casenotrun' % (output_dir, seq), 'a').write( ' [case not run] ' + reason + '\n') -def verify_image_format(supported_fmts=(), unsupported_fmts=()): +def _verify_image_format(supported_fmts: Sequence[str] = (), + unsupported_fmts: Sequence[str] = ()) -> None: assert not (supported_fmts and unsupported_fmts) if 'generic' in supported_fmts and \ @@ -XXX,XX +XXX,XX @@ def verify_image_format(supported_fmts=(), unsupported_fmts=()): if not_sup or (imgfmt in unsupported_fmts): notrun('not suitable for this image format: %s' % imgfmt) -def verify_protocol(supported=(), unsupported=()): +def _verify_protocol(supported: Sequence[str] = (), + unsupported: Sequence[str] = ()) -> None: assert not (supported and unsupported) if 'generic' in supported: @@ -XXX,XX +XXX,XX @@ def verify_protocol(supported=(), unsupported=()): if not_sup or (imgproto in unsupported): notrun('not suitable for this protocol: %s' % imgproto) -def verify_platform(supported=(), unsupported=()): +def _verify_platform(supported: Sequence[str] = (), + unsupported: Sequence[str] = ()) -> None: if any((sys.platform.startswith(x) for x in unsupported)): notrun('not suitable for this OS: %s' % sys.platform) @@ -XXX,XX +XXX,XX @@ def verify_platform(supported=(), unsupported=()): if not any((sys.platform.startswith(x) for x in supported)): notrun('not suitable for this OS: %s' % sys.platform) -def verify_cache_mode(supported_cache_modes=()): +def _verify_cache_mode(supported_cache_modes: Sequence[str] = ()) -> None: if supported_cache_modes and (cachemode not in supported_cache_modes): notrun('not suitable for this cache mode: %s' % cachemode) -def verify_aio_mode(supported_aio_modes=()): +def _verify_aio_mode(supported_aio_modes: Sequence[str] = ()): if supported_aio_modes and (aiomode not in supported_aio_modes): notrun('not suitable for this aio mode: %s' % aiomode) @@ -XXX,XX +XXX,XX @@ def execute_setup_common(supported_fmts: Sequence[str] = (), sys.stderr.write('Please run this test via the "check" script\n') sys.exit(os.EX_USAGE) - verify_image_format(supported_fmts, unsupported_fmts) - verify_protocol(supported_protocols, unsupported_protocols) - verify_platform(supported=supported_platforms) - verify_cache_mode(supported_cache_modes) - verify_aio_mode(supported_aio_modes) + _verify_image_format(supported_fmts, unsupported_fmts) + _verify_protocol(supported_protocols, unsupported_protocols) + _verify_platform(supported=supported_platforms) + _verify_cache_mode(supported_cache_modes) + _verify_aio_mode(supported_aio_modes) debug = '-d' in sys.argv if debug: -- 2.26.2 From: John Snow <jsnow@redhat.com> We can turn logging on/off globally instead of per-function. Remove use_log from run_job, and use python logging to turn on diffable output when we run through a script entry point. iotest 245 changes output order due to buffering reasons. An extended note on python logging: A NullHandler is added to `qemu.iotests` to stop output from being generated if this code is used as a library without configuring logging. A NullHandler is only needed at the root, so a duplicate handler is not needed for `qemu.iotests.diff_io`. When logging is not configured, messages at the 'WARNING' levels or above are printed with default settings. The NullHandler stops this from occurring, which is considered good hygiene for code used as a library. See https://docs.python.org/3/howto/logging.html#library-config When logging is actually enabled (always at the behest of an explicit call by a client script), a root logger is implicitly created at the root, which allows messages to propagate upwards and be handled/emitted from the root logger with default settings. When we want iotest logging, we attach a handler to the qemu.iotests.diff_io logger and disable propagation to avoid possible double-printing. For more information on python logging infrastructure, I highly recommend downloading the pip package `logging_tree`, which provides convenient visualizations of the hierarchical logging configuration under different circumstances. See https://pypi.org/project/logging_tree/ for more information. Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Message-Id: <20200331000014.11581-15-jsnow@redhat.com> Reviewed-by: Kevin Wolf <kwolf@redhat.com> Signed-off-by: Max Reitz <mreitz@redhat.com> --- tests/qemu-iotests/030 | 4 +-- tests/qemu-iotests/155 | 2 +- tests/qemu-iotests/245 | 1 + tests/qemu-iotests/245.out | 10 +++---- tests/qemu-iotests/iotests.py | 53 ++++++++++++++++++++--------------- 5 files changed, 39 insertions(+), 31 deletions(-) diff --git a/tests/qemu-iotests/030 b/tests/qemu-iotests/030 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/030 +++ b/tests/qemu-iotests/030 @@ -XXX,XX +XXX,XX @@ class TestParallelOps(iotests.QMPTestCase): result = self.vm.qmp('block-job-set-speed', device='drive0', speed=0) self.assert_qmp(result, 'return', {}) - self.vm.run_job(job='drive0', auto_dismiss=True, use_log=False) - self.vm.run_job(job='node4', auto_dismiss=True, use_log=False) + self.vm.run_job(job='drive0', auto_dismiss=True) + self.vm.run_job(job='node4', auto_dismiss=True) self.assert_no_active_block_jobs() # Test a block-stream and a block-commit job in parallel diff --git a/tests/qemu-iotests/155 b/tests/qemu-iotests/155 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/155 +++ b/tests/qemu-iotests/155 @@ -XXX,XX +XXX,XX @@ class MirrorBaseClass(BaseClass): self.assert_qmp(result, 'return', {}) - self.vm.run_job('mirror-job', use_log=False, auto_finalize=False, + self.vm.run_job('mirror-job', auto_finalize=False, pre_finalize=self.openBacking, auto_dismiss=True) def testFull(self): diff --git a/tests/qemu-iotests/245 b/tests/qemu-iotests/245 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/245 +++ b/tests/qemu-iotests/245 @@ -XXX,XX +XXX,XX @@ class TestBlockdevReopen(iotests.QMPTestCase): self.run_test_iothreads(None, 'iothread0') if __name__ == '__main__': + iotests.activate_logging() iotests.main(supported_fmts=["qcow2"], supported_protocols=["file"]) diff --git a/tests/qemu-iotests/245.out b/tests/qemu-iotests/245.out index XXXXXXX..XXXXXXX 100644 --- a/tests/qemu-iotests/245.out +++ b/tests/qemu-iotests/245.out @@ -XXX,XX +XXX,XX @@ -..................... ----------------------------------------------------------------------- -Ran 21 tests - -OK {"execute": "job-finalize", "arguments": {"id": "commit0"}} {"return": {}} {"data": {"id": "commit0", "type": "commit"}, "event": "BLOCK_JOB_PENDING", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} @@ -XXX,XX +XXX,XX @@ OK {"return": {}} {"data": {"id": "stream0", "type": "stream"}, "event": "BLOCK_JOB_PENDING", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} {"data": {"device": "stream0", "len": 3145728, "offset": 3145728, "speed": 0, "type": "stream"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} +..................... +---------------------------------------------------------------------- +Ran 21 tests + +OK diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py index XXXXXXX..XXXXXXX 100644 --- a/tests/qemu-iotests/iotests.py +++ b/tests/qemu-iotests/iotests.py @@ -XXX,XX +XXX,XX @@ assert sys.version_info >= (3, 6) QMPResponse = Dict[str, Any] +# Use this logger for logging messages directly from the iotests module +logger = logging.getLogger('qemu.iotests') +logger.addHandler(logging.NullHandler()) + +# Use this logger for messages that ought to be used for diff output. +test_logger = logging.getLogger('qemu.iotests.diff_io') + + faulthandler.enable() # This will not work if arguments contain spaces but is necessary if we @@ -XXX,XX +XXX,XX @@ def log(msg: Msg, if isinstance(msg, (dict, list)): # Don't sort if it's already sorted do_sort = not isinstance(msg, OrderedDict) - print(json.dumps(msg, sort_keys=do_sort, indent=indent)) + test_logger.info(json.dumps(msg, sort_keys=do_sort, indent=indent)) else: - print(msg) + test_logger.info(msg) class Timeout: def __init__(self, seconds, errmsg="Timeout"): @@ -XXX,XX +XXX,XX @@ class VM(qtest.QEMUQtestMachine): # Returns None on success, and an error string on failure def run_job(self, job, auto_finalize=True, auto_dismiss=False, - pre_finalize=None, cancel=False, use_log=True, wait=60.0): + pre_finalize=None, cancel=False, wait=60.0): """ run_job moves a job from creation through to dismissal. @@ -XXX,XX +XXX,XX @@ class VM(qtest.QEMUQtestMachine): invoked prior to issuing job-finalize, if any. :param cancel: Bool. When true, cancels the job after the pre_finalize callback. - :param use_log: Bool. When false, does not log QMP messages. :param wait: Float. Timeout value specifying how long to wait for any event, in seconds. Defaults to 60.0. """ @@ -XXX,XX +XXX,XX @@ class VM(qtest.QEMUQtestMachine): while True: ev = filter_qmp_event(self.events_wait(events, timeout=wait)) if ev['event'] != 'JOB_STATUS_CHANGE': - if use_log: - log(ev) + log(ev) continue status = ev['data']['status'] if status == 'aborting': @@ -XXX,XX +XXX,XX @@ class VM(qtest.QEMUQtestMachine): for j in result['return']: if j['id'] == job: error = j['error'] - if use_log: - log('Job failed: %s' % (j['error'])) + log('Job failed: %s' % (j['error'])) elif status == 'ready': - if use_log: - self.qmp_log('job-complete', id=job) - else: - self.qmp('job-complete', id=job) + self.qmp_log('job-complete', id=job) elif status == 'pending' and not auto_finalize: if pre_finalize: pre_finalize() - if cancel and use_log: + if cancel: self.qmp_log('job-cancel', id=job) - elif cancel: - self.qmp('job-cancel', id=job) - elif use_log: - self.qmp_log('job-finalize', id=job) else: - self.qmp('job-finalize', id=job) + self.qmp_log('job-finalize', id=job) elif status == 'concluded' and not auto_dismiss: - if use_log: - self.qmp_log('job-dismiss', id=job) - else: - self.qmp('job-dismiss', id=job) + self.qmp_log('job-dismiss', id=job) elif status == 'null': return error @@ -XXX,XX +XXX,XX @@ def notrun(reason): seq = os.path.basename(sys.argv[0]) open('%s/%s.notrun' % (output_dir, seq), 'w').write(reason + '\n') - print('%s not run: %s' % (seq, reason)) + logger.warning("%s not run: %s", seq, reason) sys.exit(0) def case_notrun(reason): @@ -XXX,XX +XXX,XX @@ def execute_setup_common(supported_fmts: Sequence[str] = (), if debug: sys.argv.remove('-d') logging.basicConfig(level=(logging.DEBUG if debug else logging.WARN)) + logger.debug("iotests debugging messages active") return debug @@ -XXX,XX +XXX,XX @@ def execute_test(*args, test_function=None, **kwargs): else: test_function() +def activate_logging(): + """Activate iotests.log() output to stdout for script-style tests.""" + handler = logging.StreamHandler(stream=sys.stdout) + formatter = logging.Formatter('%(message)s') + handler.setFormatter(formatter) + test_logger.addHandler(handler) + test_logger.setLevel(logging.INFO) + test_logger.propagate = False + # This is called from script-style iotests without a single point of entry def script_initialize(*args, **kwargs): """Initialize script-style tests without running any tests.""" + activate_logging() execute_setup_common(*args, **kwargs) # This is called from script-style iotests with a single point of entry def script_main(test_function, *args, **kwargs): """Run script-style tests outside of the unittest framework""" + activate_logging() execute_test(*args, test_function=test_function, **kwargs) # This is called from unittest style iotests -- 2.26.2 From: Eric Blake <eblake@redhat.com> There are several callers that need to create a new block backend from an existing BDS; make the task slightly easier with a common helper routine. Suggested-by: Max Reitz <mreitz@redhat.com> Signed-off-by: Eric Blake <eblake@redhat.com> Message-Id: <20200424190903.522087-2-eblake@redhat.com> [mreitz: Set @ret only in error paths, see https://lists.nongnu.org/archive/html/qemu-block/2020-04/msg01216.html] Signed-off-by: Max Reitz <mreitz@redhat.com> Message-Id: <20200428192648.749066-2-eblake@redhat.com> Signed-off-by: Max Reitz <mreitz@redhat.com> --- include/sysemu/block-backend.h | 2 ++ block/block-backend.c | 23 +++++++++++++++++++++++ block/crypto.c | 9 ++++----- block/parallels.c | 8 ++++---- block/qcow.c | 8 ++++---- block/qcow2.c | 18 ++++++++---------- block/qed.c | 8 ++++---- block/sheepdog.c | 10 +++++----- block/vdi.c | 8 ++++---- block/vhdx.c | 8 ++++---- block/vmdk.c | 9 ++++----- block/vpc.c | 8 ++++---- blockdev.c | 8 +++----- blockjob.c | 7 ++----- 14 files changed, 75 insertions(+), 59 deletions(-) diff --git a/include/sysemu/block-backend.h b/include/sysemu/block-backend.h index XXXXXXX..XXXXXXX 100644 --- a/include/sysemu/block-backend.h +++ b/include/sysemu/block-backend.h @@ -XXX,XX +XXX,XX @@ typedef struct BlockBackendPublic { } BlockBackendPublic; BlockBackend *blk_new(AioContext *ctx, uint64_t perm, uint64_t shared_perm); +BlockBackend *blk_new_with_bs(BlockDriverState *bs, uint64_t perm, + uint64_t shared_perm, Error **errp); BlockBackend *blk_new_open(const char *filename, const char *reference, QDict *options, int flags, Error **errp); int blk_get_refcnt(BlockBackend *blk); diff --git a/block/block-backend.c b/block/block-backend.c index XXXXXXX..XXXXXXX 100644 --- a/block/block-backend.c +++ b/block/block-backend.c @@ -XXX,XX +XXX,XX @@ BlockBackend *blk_new(AioContext *ctx, uint64_t perm, uint64_t shared_perm) return blk; } +/* + * Create a new BlockBackend connected to an existing BlockDriverState. + * + * @perm is a bitmasks of BLK_PERM_* constants which describes the + * permissions to request for @bs that is attached to this + * BlockBackend. @shared_perm is a bitmask which describes which + * permissions may be granted to other users of the attached node. + * Both sets of permissions can be changed later using blk_set_perm(). + * + * Return the new BlockBackend on success, null on failure. + */ +BlockBackend *blk_new_with_bs(BlockDriverState *bs, uint64_t perm, + uint64_t shared_perm, Error **errp) +{ + BlockBackend *blk = blk_new(bdrv_get_aio_context(bs), perm, shared_perm); + + if (blk_insert_bs(blk, bs, errp) < 0) { + blk_unref(blk); + return NULL; + } + return blk; +} + /* * Creates a new BlockBackend, opens a new BlockDriverState, and connects both. * The new BlockBackend is in the main AioContext. diff --git a/block/crypto.c b/block/crypto.c index XXXXXXX..XXXXXXX 100644 --- a/block/crypto.c +++ b/block/crypto.c @@ -XXX,XX +XXX,XX @@ static int block_crypto_co_create_generic(BlockDriverState *bs, QCryptoBlock *crypto = NULL; struct BlockCryptoCreateData data; - blk = blk_new(bdrv_get_aio_context(bs), - BLK_PERM_WRITE | BLK_PERM_RESIZE, BLK_PERM_ALL); - - ret = blk_insert_bs(blk, bs, errp); - if (ret < 0) { + blk = blk_new_with_bs(bs, BLK_PERM_WRITE | BLK_PERM_RESIZE, BLK_PERM_ALL, + errp); + if (!blk) { + ret = -EPERM; goto cleanup; } diff --git a/block/parallels.c b/block/parallels.c index XXXXXXX..XXXXXXX 100644 --- a/block/parallels.c +++ b/block/parallels.c @@ -XXX,XX +XXX,XX @@ static int coroutine_fn parallels_co_create(BlockdevCreateOptions* opts, return -EIO; } - blk = blk_new(bdrv_get_aio_context(bs), - BLK_PERM_WRITE | BLK_PERM_RESIZE, BLK_PERM_ALL); - ret = blk_insert_bs(blk, bs, errp); - if (ret < 0) { + blk = blk_new_with_bs(bs, BLK_PERM_WRITE | BLK_PERM_RESIZE, BLK_PERM_ALL, + errp); + if (!blk) { + ret = -EPERM; goto out; } blk_set_allow_write_beyond_eof(blk, true); diff --git a/block/qcow.c b/block/qcow.c index XXXXXXX..XXXXXXX 100644 --- a/block/qcow.c +++ b/block/qcow.c @@ -XXX,XX +XXX,XX @@ static int coroutine_fn qcow_co_create(BlockdevCreateOptions *opts, return -EIO; } - qcow_blk = blk_new(bdrv_get_aio_context(bs), - BLK_PERM_WRITE | BLK_PERM_RESIZE, BLK_PERM_ALL); - ret = blk_insert_bs(qcow_blk, bs, errp); - if (ret < 0) { + qcow_blk = blk_new_with_bs(bs, BLK_PERM_WRITE | BLK_PERM_RESIZE, + BLK_PERM_ALL, errp); + if (!qcow_blk) { + ret = -EPERM; goto exit; } blk_set_allow_write_beyond_eof(qcow_blk, true); diff --git a/block/qcow2.c b/block/qcow2.c index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -XXX,XX +XXX,XX @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp) } /* Create BlockBackend to write to the image */ - blk = blk_new(bdrv_get_aio_context(bs), - BLK_PERM_WRITE | BLK_PERM_RESIZE, BLK_PERM_ALL); - ret = blk_insert_bs(blk, bs, errp); - if (ret < 0) { + blk = blk_new_with_bs(bs, BLK_PERM_WRITE | BLK_PERM_RESIZE, BLK_PERM_ALL, + errp); + if (!blk) { + ret = -EPERM; goto out; } blk_set_allow_write_beyond_eof(blk, true); @@ -XXX,XX +XXX,XX @@ static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts, } if (new_size) { - BlockBackend *blk = blk_new(bdrv_get_aio_context(bs), - BLK_PERM_RESIZE, BLK_PERM_ALL); - ret = blk_insert_bs(blk, bs, errp); - if (ret < 0) { - blk_unref(blk); - return ret; + BlockBackend *blk = blk_new_with_bs(bs, BLK_PERM_RESIZE, BLK_PERM_ALL, + errp); + if (!blk) { + return -EPERM; } /* diff --git a/block/qed.c b/block/qed.c index XXXXXXX..XXXXXXX 100644 --- a/block/qed.c +++ b/block/qed.c @@ -XXX,XX +XXX,XX @@ static int coroutine_fn bdrv_qed_co_create(BlockdevCreateOptions *opts, return -EIO; } - blk = blk_new(bdrv_get_aio_context(bs), - BLK_PERM_WRITE | BLK_PERM_RESIZE, BLK_PERM_ALL); - ret = blk_insert_bs(blk, bs, errp); - if (ret < 0) { + blk = blk_new_with_bs(bs, BLK_PERM_WRITE | BLK_PERM_RESIZE, BLK_PERM_ALL, + errp); + if (!blk) { + ret = -EPERM; goto out; } blk_set_allow_write_beyond_eof(blk, true); diff --git a/block/sheepdog.c b/block/sheepdog.c index XXXXXXX..XXXXXXX 100644 --- a/block/sheepdog.c +++ b/block/sheepdog.c @@ -XXX,XX +XXX,XX @@ static int sd_prealloc(BlockDriverState *bs, int64_t old_size, int64_t new_size, void *buf = NULL; int ret; - blk = blk_new(bdrv_get_aio_context(bs), - BLK_PERM_CONSISTENT_READ | BLK_PERM_WRITE | BLK_PERM_RESIZE, - BLK_PERM_ALL); + blk = blk_new_with_bs(bs, + BLK_PERM_CONSISTENT_READ | BLK_PERM_WRITE | BLK_PERM_RESIZE, + BLK_PERM_ALL, errp); - ret = blk_insert_bs(blk, bs, errp); - if (ret < 0) { + if (!blk) { + ret = -EPERM; goto out_with_err_set; } diff --git a/block/vdi.c b/block/vdi.c index XXXXXXX..XXXXXXX 100644 --- a/block/vdi.c +++ b/block/vdi.c @@ -XXX,XX +XXX,XX @@ static int coroutine_fn vdi_co_do_create(BlockdevCreateOptions *create_options, goto exit; } - blk = blk_new(bdrv_get_aio_context(bs_file), - BLK_PERM_WRITE | BLK_PERM_RESIZE, BLK_PERM_ALL); - ret = blk_insert_bs(blk, bs_file, errp); - if (ret < 0) { + blk = blk_new_with_bs(bs_file, BLK_PERM_WRITE | BLK_PERM_RESIZE, + BLK_PERM_ALL, errp); + if (!blk) { + ret = -EPERM; goto exit; } diff --git a/block/vhdx.c b/block/vhdx.c index XXXXXXX..XXXXXXX 100644 --- a/block/vhdx.c +++ b/block/vhdx.c @@ -XXX,XX +XXX,XX @@ static int coroutine_fn vhdx_co_create(BlockdevCreateOptions *opts, return -EIO; } - blk = blk_new(bdrv_get_aio_context(bs), - BLK_PERM_WRITE | BLK_PERM_RESIZE, BLK_PERM_ALL); - ret = blk_insert_bs(blk, bs, errp); - if (ret < 0) { + blk = blk_new_with_bs(bs, BLK_PERM_WRITE | BLK_PERM_RESIZE, BLK_PERM_ALL, + errp); + if (!blk) { + ret = -EPERM; goto delete_and_exit; } blk_set_allow_write_beyond_eof(blk, true); diff --git a/block/vmdk.c b/block/vmdk.c index XXXXXXX..XXXXXXX 100644 --- a/block/vmdk.c +++ b/block/vmdk.c @@ -XXX,XX +XXX,XX @@ static BlockBackend *vmdk_co_create_cb(int64_t size, int idx, if (!bs) { return NULL; } - blk = blk_new(bdrv_get_aio_context(bs), - BLK_PERM_CONSISTENT_READ | BLK_PERM_WRITE | BLK_PERM_RESIZE, - BLK_PERM_ALL); - if (blk_insert_bs(blk, bs, errp)) { - bdrv_unref(bs); + blk = blk_new_with_bs(bs, + BLK_PERM_CONSISTENT_READ | BLK_PERM_WRITE | BLK_PERM_RESIZE, + BLK_PERM_ALL, errp); + if (!blk) { return NULL; } blk_set_allow_write_beyond_eof(blk, true); diff --git a/block/vpc.c b/block/vpc.c index XXXXXXX..XXXXXXX 100644 --- a/block/vpc.c +++ b/block/vpc.c @@ -XXX,XX +XXX,XX @@ static int coroutine_fn vpc_co_create(BlockdevCreateOptions *opts, return -EIO; } - blk = blk_new(bdrv_get_aio_context(bs), - BLK_PERM_WRITE | BLK_PERM_RESIZE, BLK_PERM_ALL); - ret = blk_insert_bs(blk, bs, errp); - if (ret < 0) { + blk = blk_new_with_bs(bs, BLK_PERM_WRITE | BLK_PERM_RESIZE, BLK_PERM_ALL, + errp); + if (!blk) { + ret = -EPERM; goto out; } blk_set_allow_write_beyond_eof(blk, true); diff --git a/blockdev.c b/blockdev.c index XXXXXXX..XXXXXXX 100644 --- a/blockdev.c +++ b/blockdev.c @@ -XXX,XX +XXX,XX @@ void qmp_block_resize(bool has_device, const char *device, BlockBackend *blk = NULL; BlockDriverState *bs; AioContext *aio_context; - int ret; bs = bdrv_lookup_bs(has_device ? device : NULL, has_node_name ? node_name : NULL, @@ -XXX,XX +XXX,XX @@ void qmp_block_resize(bool has_device, const char *device, goto out; } - blk = blk_new(bdrv_get_aio_context(bs), BLK_PERM_RESIZE, BLK_PERM_ALL); - ret = blk_insert_bs(blk, bs, errp); - if (ret < 0) { + blk = blk_new_with_bs(bs, BLK_PERM_RESIZE, BLK_PERM_ALL, errp); + if (!blk) { goto out; } bdrv_drained_begin(bs); - ret = blk_truncate(blk, size, false, PREALLOC_MODE_OFF, 0, errp); + blk_truncate(blk, size, false, PREALLOC_MODE_OFF, 0, errp); bdrv_drained_end(bs); out: diff --git a/blockjob.c b/blockjob.c index XXXXXXX..XXXXXXX 100644 --- a/blockjob.c +++ b/blockjob.c @@ -XXX,XX +XXX,XX @@ void *block_job_create(const char *job_id, const BlockJobDriver *driver, { BlockBackend *blk; BlockJob *job; - int ret; if (job_id == NULL && !(flags & JOB_INTERNAL)) { job_id = bdrv_get_device_name(bs); } - blk = blk_new(bdrv_get_aio_context(bs), perm, shared_perm); - ret = blk_insert_bs(blk, bs, errp); - if (ret < 0) { - blk_unref(blk); + blk = blk_new_with_bs(bs, perm, shared_perm, errp); + if (!blk) { return NULL; } -- 2.26.2 From: Eric Blake <eblake@redhat.com> We originally refused to allow resize of images with internal snapshots because the v2 image format did not require the tracking of snapshot size, making it impossible to safely revert to a snapshot with a different size than the current view of the image. But the snapshot size tracking was rectified in v3, and our recent fixes to qemu-img amend (see 0a85af35) guarantee that we always have a valid snapshot size. Thus, we no longer need to artificially limit image resizes, but it does become one more thing that would prevent a downgrade back to v2. And now that we support different-sized snapshots, it's also easy to fix reverting to a snapshot to apply the new size. Upgrade iotest 61 to cover this (we previously had NO coverage of refusal to resize while snapshots exist). Note that the amend process can fail but still have effects: in particular, since we break things into upgrade, resize, downgrade, a failure during resize does not roll back changes made during upgrade, nor does failure in downgrade roll back a resize. But this situation is pre-existing even without this patch; and without journaling, the best we could do is minimize the chance of partial failure by collecting all changes prior to doing any writes - which adds a lot of complexity but could still fail with EIO. On the other hand, we are careful that even if we have partial modification but then fail, the image is left viable (that is, we are careful to sequence things so that after each successful cluster write, there may be transient leaked clusters but no corrupt metadata). And complicating the code to make it more transaction-like is not worth the effort: a user can always request multiple 'qemu-img amend' changing one thing each, if they need finer-grained control over detecting the first failure than what they get by letting qemu decide how to sequence multiple changes. Signed-off-by: Eric Blake <eblake@redhat.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Message-Id: <20200428192648.749066-3-eblake@redhat.com> Signed-off-by: Max Reitz <mreitz@redhat.com> --- block/qcow2-snapshot.c | 20 ++++++++++++++++---- block/qcow2.c | 25 ++++++++++++++++++++++--- tests/qemu-iotests/061 | 35 +++++++++++++++++++++++++++++++++++ tests/qemu-iotests/061.out | 28 ++++++++++++++++++++++++++++ 4 files changed, 101 insertions(+), 7 deletions(-) diff --git a/block/qcow2-snapshot.c b/block/qcow2-snapshot.c index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2-snapshot.c +++ b/block/qcow2-snapshot.c @@ -XXX,XX +XXX,XX @@ */ #include "qemu/osdep.h" +#include "sysemu/block-backend.h" #include "qapi/error.h" #include "qcow2.h" #include "qemu/bswap.h" @@ -XXX,XX +XXX,XX @@ int qcow2_snapshot_goto(BlockDriverState *bs, const char *snapshot_id) } if (sn->disk_size != bs->total_sectors * BDRV_SECTOR_SIZE) { - error_report("qcow2: Loading snapshots with different disk " - "size is not implemented"); - ret = -ENOTSUP; - goto fail; + BlockBackend *blk = blk_new_with_bs(bs, BLK_PERM_RESIZE, BLK_PERM_ALL, + &local_err); + if (!blk) { + error_report_err(local_err); + ret = -ENOTSUP; + goto fail; + } + + ret = blk_truncate(blk, sn->disk_size, true, PREALLOC_MODE_OFF, 0, + &local_err); + blk_unref(blk); + if (ret < 0) { + error_report_err(local_err); + goto fail; + } } /* diff --git a/block/qcow2.c b/block/qcow2.c index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -XXX,XX +XXX,XX @@ static int coroutine_fn qcow2_co_truncate(BlockDriverState *bs, int64_t offset, qemu_co_mutex_lock(&s->lock); - /* cannot proceed if image has snapshots */ - if (s->nb_snapshots) { - error_setg(errp, "Can't resize an image which has snapshots"); + /* + * Even though we store snapshot size for all images, it was not + * required until v3, so it is not safe to proceed for v2. + */ + if (s->nb_snapshots && s->qcow_version < 3) { + error_setg(errp, "Can't resize a v2 image which has snapshots"); ret = -ENOTSUP; goto fail; } @@ -XXX,XX +XXX,XX @@ static int qcow2_downgrade(BlockDriverState *bs, int target_version, BDRVQcow2State *s = bs->opaque; int current_version = s->qcow_version; int ret; + int i; /* This is qcow2_downgrade(), not qcow2_upgrade() */ assert(target_version < current_version); @@ -XXX,XX +XXX,XX @@ static int qcow2_downgrade(BlockDriverState *bs, int target_version, return -ENOTSUP; } + /* + * If any internal snapshot has a different size than the current + * image size, or VM state size that exceeds 32 bits, downgrading + * is unsafe. Even though we would still use v3-compliant output + * to preserve that data, other v2 programs might not realize + * those optional fields are important. + */ + for (i = 0; i < s->nb_snapshots; i++) { + if (s->snapshots[i].vm_state_size > UINT32_MAX || + s->snapshots[i].disk_size != bs->total_sectors * BDRV_SECTOR_SIZE) { + error_setg(errp, "Internal snapshots prevent downgrade of image"); + return -ENOTSUP; + } + } + /* clear incompatible features */ if (s->incompatible_features & QCOW2_INCOMPAT_DIRTY) { ret = qcow2_mark_clean(bs); diff --git a/tests/qemu-iotests/061 b/tests/qemu-iotests/061 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/061 +++ b/tests/qemu-iotests/061 @@ -XXX,XX +XXX,XX @@ $PYTHON qcow2.py "$TEST_IMG" dump-header $QEMU_IO -c "read -P 0x2a 42M 64k" "$TEST_IMG" | _filter_qemu_io _check_test_img +echo +echo "=== Testing resize with snapshots ===" +echo +_make_test_img -o "compat=0.10" 32M +$QEMU_IO -c "write -P 0x2a 24M 64k" "$TEST_IMG" | _filter_qemu_io +$QEMU_IMG snapshot -c foo "$TEST_IMG" +$QEMU_IMG resize "$TEST_IMG" 64M && + echo "unexpected pass" +$PYTHON qcow2.py "$TEST_IMG" dump-header | grep '^\(version\|size\|nb_snap\)' + +$QEMU_IMG amend -o "compat=1.1,size=128M" "$TEST_IMG" || + echo "unexpected fail" +$PYTHON qcow2.py "$TEST_IMG" dump-header | grep '^\(version\|size\|nb_snap\)' + +$QEMU_IMG snapshot -c bar "$TEST_IMG" +$QEMU_IMG resize --shrink "$TEST_IMG" 64M || + echo "unexpected fail" +$PYTHON qcow2.py "$TEST_IMG" dump-header | grep '^\(version\|size\|nb_snap\)' + +$QEMU_IMG amend -o "compat=0.10,size=32M" "$TEST_IMG" && + echo "unexpected pass" +$PYTHON qcow2.py "$TEST_IMG" dump-header | grep '^\(version\|size\|nb_snap\)' + +$QEMU_IMG snapshot -a bar "$TEST_IMG" || + echo "unexpected fail" +$PYTHON qcow2.py "$TEST_IMG" dump-header | grep '^\(version\|size\|nb_snap\)' + +$QEMU_IMG snapshot -d bar "$TEST_IMG" +$QEMU_IMG amend -o "compat=0.10,size=32M" "$TEST_IMG" || + echo "unexpected fail" +$PYTHON qcow2.py "$TEST_IMG" dump-header | grep '^\(version\|size\|nb_snap\)' + +_check_test_img + + echo echo "=== Testing dirty lazy_refcounts=off ===" echo diff --git a/tests/qemu-iotests/061.out b/tests/qemu-iotests/061.out index XXXXXXX..XXXXXXX 100644 --- a/tests/qemu-iotests/061.out +++ b/tests/qemu-iotests/061.out @@ -XXX,XX +XXX,XX @@ read 65536/65536 bytes at offset 44040192 64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) No errors were found on the image. +=== Testing resize with snapshots === + +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=33554432 +wrote 65536/65536 bytes at offset 25165824 +64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +qemu-img: Can't resize a v2 image which has snapshots +version 2 +size 33554432 +nb_snapshots 1 +version 3 +size 134217728 +nb_snapshots 1 +Image resized. +version 3 +size 67108864 +nb_snapshots 2 +qemu-img: Internal snapshots prevent downgrade of image +version 3 +size 33554432 +nb_snapshots 2 +version 3 +size 134217728 +nb_snapshots 2 +version 2 +size 33554432 +nb_snapshots 1 +No errors were found on the image. + === Testing dirty lazy_refcounts=off === Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 -- 2.26.2 From: Eric Blake <eblake@redhat.com> Our comment did not actually match the code. Rewrite the comment to be less sensitive to any future changes to qcow2-bitmap.c that might implement scenarios that we currently reject. Signed-off-by: Eric Blake <eblake@redhat.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Message-Id: <20200428192648.749066-4-eblake@redhat.com> Signed-off-by: Max Reitz <mreitz@redhat.com> --- block/qcow2.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/block/qcow2.c b/block/qcow2.c index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -XXX,XX +XXX,XX @@ static int coroutine_fn qcow2_co_truncate(BlockDriverState *bs, int64_t offset, goto fail; } - /* cannot proceed if image has bitmaps */ + /* See qcow2-bitmap.c for which bitmap scenarios prevent a resize. */ if (qcow2_truncate_bitmaps_check(bs, errp)) { ret = -ENOTSUP; goto fail; -- 2.26.2 From: Eric Blake <eblake@redhat.com> It's been a while since we got rid of the sector-based bdrv_read and bdrv_write (commit 2e11d756); let's finish the job on a few remaining comments. Signed-off-by: Eric Blake <eblake@redhat.com> Message-Id: <20200428213807.776655-1-eblake@redhat.com> Reviewed-by: Alberto Garcia <berto@igalia.com> Signed-off-by: Max Reitz <mreitz@redhat.com> --- block/io.c | 3 ++- block/qcow2-refcount.c | 2 +- block/vvfat.c | 10 +++++----- tests/qemu-iotests/001 | 2 +- tests/qemu-iotests/052 | 2 +- tests/qemu-iotests/134 | 2 +- tests/qemu-iotests/188 | 2 +- 7 files changed, 12 insertions(+), 11 deletions(-) diff --git a/block/io.c b/block/io.c index XXXXXXX..XXXXXXX 100644 --- a/block/io.c +++ b/block/io.c @@ -XXX,XX +XXX,XX @@ int bdrv_pwrite_zeroes(BdrvChild *child, int64_t offset, * flags are passed through to bdrv_pwrite_zeroes (e.g. BDRV_REQ_MAY_UNMAP, * BDRV_REQ_FUA). * - * Returns < 0 on error, 0 on success. For error codes see bdrv_write(). + * Returns < 0 on error, 0 on success. For error codes see bdrv_pwrite(). */ int bdrv_make_zero(BdrvChild *child, BdrvRequestFlags flags) { @@ -XXX,XX +XXX,XX @@ int bdrv_make_zero(BdrvChild *child, BdrvRequestFlags flags) } } +/* return < 0 if error. See bdrv_pwrite() for the return codes */ int bdrv_preadv(BdrvChild *child, int64_t offset, QEMUIOVector *qiov) { int ret; diff --git a/block/qcow2-refcount.c b/block/qcow2-refcount.c index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2-refcount.c +++ b/block/qcow2-refcount.c @@ -XXX,XX +XXX,XX @@ fail: * - 0 if writing to this offset will not affect the mentioned metadata * - a positive QCow2MetadataOverlap value indicating one overlapping section * - a negative value (-errno) indicating an error while performing a check, - * e.g. when bdrv_read failed on QCOW2_OL_INACTIVE_L2 + * e.g. when bdrv_pread failed on QCOW2_OL_INACTIVE_L2 */ int qcow2_check_metadata_overlap(BlockDriverState *bs, int ign, int64_t offset, int64_t size) diff --git a/block/vvfat.c b/block/vvfat.c index XXXXXXX..XXXXXXX 100644 --- a/block/vvfat.c +++ b/block/vvfat.c @@ -XXX,XX +XXX,XX @@ DLOG(checkpoint()); * - get modified FAT * - compare the two FATs (TODO) * - get buffer for marking used clusters - * - recurse direntries from root (using bs->bdrv_read to make + * - recurse direntries from root (using bs->bdrv_pread to make * sure to get the new data) * - check that the FAT agrees with the size * - count the number of clusters occupied by this directory and @@ -XXX,XX +XXX,XX @@ static int handle_deletes(BDRVVVFATState* s) /* * synchronize mapping with new state: * - * - copy FAT (with bdrv_read) + * - copy FAT (with bdrv_pread) * - mark all filenames corresponding to mappings as deleted - * - recurse direntries from root (using bs->bdrv_read) + * - recurse direntries from root (using bs->bdrv_pread) * - delete files corresponding to mappings marked as deleted */ static int do_commit(BDRVVVFATState* s) @@ -XXX,XX +XXX,XX @@ static int do_commit(BDRVVVFATState* s) return ret; } - /* copy FAT (with bdrv_read) */ + /* copy FAT (with bdrv_pread) */ memcpy(s->fat.pointer, s->fat2, 0x200 * s->sectors_per_fat); - /* recurse direntries from root (using bs->bdrv_read) */ + /* recurse direntries from root (using bs->bdrv_pread) */ ret = commit_direntries(s, 0, -1); if (ret) { fprintf(stderr, "Fatal: error while committing (%d)\n", ret); diff --git a/tests/qemu-iotests/001 b/tests/qemu-iotests/001 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/001 +++ b/tests/qemu-iotests/001 @@ -XXX,XX +XXX,XX @@ #!/usr/bin/env bash # -# Test simple read/write using plain bdrv_read/bdrv_write +# Test simple read/write using plain bdrv_pread/bdrv_pwrite # # Copyright (C) 2009 Red Hat, Inc. # diff --git a/tests/qemu-iotests/052 b/tests/qemu-iotests/052 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/052 +++ b/tests/qemu-iotests/052 @@ -XXX,XX +XXX,XX @@ #!/usr/bin/env bash # -# Test bdrv_read/bdrv_write using BDRV_O_SNAPSHOT +# Test bdrv_pread/bdrv_pwrite using BDRV_O_SNAPSHOT # # Copyright (C) 2013 Red Hat, Inc. # diff --git a/tests/qemu-iotests/134 b/tests/qemu-iotests/134 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/134 +++ b/tests/qemu-iotests/134 @@ -XXX,XX +XXX,XX @@ #!/usr/bin/env bash # -# Test encrypted read/write using plain bdrv_read/bdrv_write +# Test encrypted read/write using plain bdrv_pread/bdrv_pwrite # # Copyright (C) 2015 Red Hat, Inc. # diff --git a/tests/qemu-iotests/188 b/tests/qemu-iotests/188 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/188 +++ b/tests/qemu-iotests/188 @@ -XXX,XX +XXX,XX @@ #!/usr/bin/env bash # -# Test encrypted read/write using plain bdrv_read/bdrv_write +# Test encrypted read/write using plain bdrv_pread/bdrv_pwrite # # Copyright (C) 2017 Red Hat, Inc. # -- 2.26.2 From: Maxim Levitsky <mlevitsk@redhat.com> Commit f62514b3def5fb2acbef64d0e053c0c31fa45aff made qemu-img reject -o "" but this test uses it. Since this test only tries to do a dry-run run of qemu-img amend, replace the -o "" with dummy -o "size=$size". Fixes: f62514b3def5fb2acbef64d0e053c0c31fa45aff Signed-off-by: Maxim Levitsky <mlevitsk@redhat.com> Message-Id: <20200504131959.9533-1-mlevitsk@redhat.com> Signed-off-by: Max Reitz <mreitz@redhat.com> --- tests/qemu-iotests/153 | 2 +- tests/qemu-iotests/153.out | 12 ++++++------ 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/tests/qemu-iotests/153 b/tests/qemu-iotests/153 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/153 +++ b/tests/qemu-iotests/153 @@ -XXX,XX +XXX,XX @@ for opts1 in "" "read-only=on" "read-only=on,force-share=on"; do _run_cmd $QEMU_IMG check $L "${TEST_IMG}" _run_cmd $QEMU_IMG compare $L "${TEST_IMG}" "${TEST_IMG}" _run_cmd $QEMU_IMG map $L "${TEST_IMG}" - _run_cmd $QEMU_IMG amend -o "" $L "${TEST_IMG}" + _run_cmd $QEMU_IMG amend -o "size=$size" $L "${TEST_IMG}" _run_cmd $QEMU_IMG commit $L "${TEST_IMG}" _run_cmd $QEMU_IMG resize $L "${TEST_IMG}" $size _run_cmd $QEMU_IMG rebase $L "${TEST_IMG}" -b "${TEST_IMG}.base" diff --git a/tests/qemu-iotests/153.out b/tests/qemu-iotests/153.out index XXXXXXX..XXXXXXX 100644 --- a/tests/qemu-iotests/153.out +++ b/tests/qemu-iotests/153.out @@ -XXX,XX +XXX,XX @@ _qemu_img_wrapper map TEST_DIR/t.qcow2 qemu-img: Could not open 'TEST_DIR/t.qcow2': Failed to get shared "write" lock Is another process using the image [TEST_DIR/t.qcow2]? -_qemu_img_wrapper amend -o TEST_DIR/t.qcow2 +_qemu_img_wrapper amend -o size=32M TEST_DIR/t.qcow2 qemu-img: Could not open 'TEST_DIR/t.qcow2': Failed to get "write" lock Is another process using the image [TEST_DIR/t.qcow2]? @@ -XXX,XX +XXX,XX @@ _qemu_img_wrapper compare -U TEST_DIR/t.qcow2 TEST_DIR/t.qcow2 _qemu_img_wrapper map -U TEST_DIR/t.qcow2 -_qemu_img_wrapper amend -o -U TEST_DIR/t.qcow2 +_qemu_img_wrapper amend -o size=32M -U TEST_DIR/t.qcow2 qemu-img: unrecognized option '-U' Try 'qemu-img --help' for more information @@ -XXX,XX +XXX,XX @@ _qemu_img_wrapper compare TEST_DIR/t.qcow2 TEST_DIR/t.qcow2 _qemu_img_wrapper map TEST_DIR/t.qcow2 -_qemu_img_wrapper amend -o TEST_DIR/t.qcow2 +_qemu_img_wrapper amend -o size=32M TEST_DIR/t.qcow2 qemu-img: Could not open 'TEST_DIR/t.qcow2': Failed to get "write" lock Is another process using the image [TEST_DIR/t.qcow2]? @@ -XXX,XX +XXX,XX @@ _qemu_img_wrapper compare -U TEST_DIR/t.qcow2 TEST_DIR/t.qcow2 _qemu_img_wrapper map -U TEST_DIR/t.qcow2 -_qemu_img_wrapper amend -o -U TEST_DIR/t.qcow2 +_qemu_img_wrapper amend -o size=32M -U TEST_DIR/t.qcow2 qemu-img: unrecognized option '-U' Try 'qemu-img --help' for more information @@ -XXX,XX +XXX,XX @@ _qemu_img_wrapper compare TEST_DIR/t.qcow2 TEST_DIR/t.qcow2 _qemu_img_wrapper map TEST_DIR/t.qcow2 -_qemu_img_wrapper amend -o TEST_DIR/t.qcow2 +_qemu_img_wrapper amend -o size=32M TEST_DIR/t.qcow2 _qemu_img_wrapper commit TEST_DIR/t.qcow2 @@ -XXX,XX +XXX,XX @@ _qemu_img_wrapper compare -U TEST_DIR/t.qcow2 TEST_DIR/t.qcow2 _qemu_img_wrapper map -U TEST_DIR/t.qcow2 -_qemu_img_wrapper amend -o -U TEST_DIR/t.qcow2 +_qemu_img_wrapper amend -o size=32M -U TEST_DIR/t.qcow2 qemu-img: unrecognized option '-U' Try 'qemu-img --help' for more information -- 2.26.2 From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> We are going to use aio-task-pool API and extend in-flight request structure to be a successor of AioTask, so rename things appropriately. Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Message-Id: <20200429130847.28124-2-vsementsov@virtuozzo.com> Signed-off-by: Max Reitz <mreitz@redhat.com> --- block/block-copy.c | 98 +++++++++++++++++++++++----------------------- 1 file changed, 48 insertions(+), 50 deletions(-) diff --git a/block/block-copy.c b/block/block-copy.c index XXXXXXX..XXXXXXX 100644 --- a/block/block-copy.c +++ b/block/block-copy.c @@ -XXX,XX +XXX,XX @@ #define BLOCK_COPY_MAX_BUFFER (1 * MiB) #define BLOCK_COPY_MAX_MEM (128 * MiB) -typedef struct BlockCopyInFlightReq { +typedef struct BlockCopyTask { int64_t offset; int64_t bytes; - QLIST_ENTRY(BlockCopyInFlightReq) list; - CoQueue wait_queue; /* coroutines blocked on this request */ -} BlockCopyInFlightReq; + QLIST_ENTRY(BlockCopyTask) list; + CoQueue wait_queue; /* coroutines blocked on this task */ +} BlockCopyTask; typedef struct BlockCopyState { /* @@ -XXX,XX +XXX,XX @@ typedef struct BlockCopyState { bool use_copy_range; int64_t copy_size; uint64_t len; - QLIST_HEAD(, BlockCopyInFlightReq) inflight_reqs; + QLIST_HEAD(, BlockCopyTask) tasks; BdrvRequestFlags write_flags; @@ -XXX,XX +XXX,XX @@ typedef struct BlockCopyState { SharedResource *mem; } BlockCopyState; -static BlockCopyInFlightReq *find_conflicting_inflight_req(BlockCopyState *s, - int64_t offset, - int64_t bytes) +static BlockCopyTask *find_conflicting_task(BlockCopyState *s, + int64_t offset, int64_t bytes) { - BlockCopyInFlightReq *req; + BlockCopyTask *t; - QLIST_FOREACH(req, &s->inflight_reqs, list) { - if (offset + bytes > req->offset && offset < req->offset + req->bytes) { - return req; + QLIST_FOREACH(t, &s->tasks, list) { + if (offset + bytes > t->offset && offset < t->offset + t->bytes) { + return t; } } @@ -XXX,XX +XXX,XX @@ static BlockCopyInFlightReq *find_conflicting_inflight_req(BlockCopyState *s, } /* - * If there are no intersecting requests return false. Otherwise, wait for the - * first found intersecting request to finish and return true. + * If there are no intersecting tasks return false. Otherwise, wait for the + * first found intersecting tasks to finish and return true. */ static bool coroutine_fn block_copy_wait_one(BlockCopyState *s, int64_t offset, int64_t bytes) { - BlockCopyInFlightReq *req = find_conflicting_inflight_req(s, offset, bytes); + BlockCopyTask *task = find_conflicting_task(s, offset, bytes); - if (!req) { + if (!task) { return false; } - qemu_co_queue_wait(&req->wait_queue, NULL); + qemu_co_queue_wait(&task->wait_queue, NULL); return true; } /* Called only on full-dirty region */ -static void block_copy_inflight_req_begin(BlockCopyState *s, - BlockCopyInFlightReq *req, - int64_t offset, int64_t bytes) +static void block_copy_task_begin(BlockCopyState *s, BlockCopyTask *task, + int64_t offset, int64_t bytes) { - assert(!find_conflicting_inflight_req(s, offset, bytes)); + assert(!find_conflicting_task(s, offset, bytes)); bdrv_reset_dirty_bitmap(s->copy_bitmap, offset, bytes); s->in_flight_bytes += bytes; - req->offset = offset; - req->bytes = bytes; - qemu_co_queue_init(&req->wait_queue); - QLIST_INSERT_HEAD(&s->inflight_reqs, req, list); + task->offset = offset; + task->bytes = bytes; + qemu_co_queue_init(&task->wait_queue); + QLIST_INSERT_HEAD(&s->tasks, task, list); } /* - * block_copy_inflight_req_shrink + * block_copy_task_shrink * - * Drop the tail of the request to be handled later. Set dirty bits back and - * wake up all requests waiting for us (may be some of them are not intersecting - * with shrunk request) + * Drop the tail of the task to be handled later. Set dirty bits back and + * wake up all tasks waiting for us (may be some of them are not intersecting + * with shrunk task) */ -static void coroutine_fn block_copy_inflight_req_shrink(BlockCopyState *s, - BlockCopyInFlightReq *req, int64_t new_bytes) +static void coroutine_fn block_copy_task_shrink(BlockCopyState *s, + BlockCopyTask *task, + int64_t new_bytes) { - if (new_bytes == req->bytes) { + if (new_bytes == task->bytes) { return; } - assert(new_bytes > 0 && new_bytes < req->bytes); + assert(new_bytes > 0 && new_bytes < task->bytes); - s->in_flight_bytes -= req->bytes - new_bytes; + s->in_flight_bytes -= task->bytes - new_bytes; bdrv_set_dirty_bitmap(s->copy_bitmap, - req->offset + new_bytes, req->bytes - new_bytes); + task->offset + new_bytes, task->bytes - new_bytes); - req->bytes = new_bytes; - qemu_co_queue_restart_all(&req->wait_queue); + task->bytes = new_bytes; + qemu_co_queue_restart_all(&task->wait_queue); } -static void coroutine_fn block_copy_inflight_req_end(BlockCopyState *s, - BlockCopyInFlightReq *req, - int ret) +static void coroutine_fn block_copy_task_end(BlockCopyState *s, + BlockCopyTask *task, int ret) { - s->in_flight_bytes -= req->bytes; + s->in_flight_bytes -= task->bytes; if (ret < 0) { - bdrv_set_dirty_bitmap(s->copy_bitmap, req->offset, req->bytes); + bdrv_set_dirty_bitmap(s->copy_bitmap, task->offset, task->bytes); } - QLIST_REMOVE(req, list); - qemu_co_queue_restart_all(&req->wait_queue); + QLIST_REMOVE(task, list); + qemu_co_queue_restart_all(&task->wait_queue); } void block_copy_state_free(BlockCopyState *s) @@ -XXX,XX +XXX,XX @@ BlockCopyState *block_copy_state_new(BdrvChild *source, BdrvChild *target, s->copy_size = MAX(s->cluster_size, BLOCK_COPY_MAX_BUFFER); } - QLIST_INIT(&s->inflight_reqs); + QLIST_INIT(&s->tasks); return s; } @@ -XXX,XX +XXX,XX @@ static int coroutine_fn block_copy_dirty_clusters(BlockCopyState *s, assert(QEMU_IS_ALIGNED(bytes, s->cluster_size)); while (bytes) { - BlockCopyInFlightReq req; + BlockCopyTask task; int64_t next_zero, cur_bytes, status_bytes; if (!bdrv_dirty_bitmap_get(s->copy_bitmap, offset)) { @@ -XXX,XX +XXX,XX @@ static int coroutine_fn block_copy_dirty_clusters(BlockCopyState *s, assert(next_zero < offset + cur_bytes); /* no need to do MIN() */ cur_bytes = next_zero - offset; } - block_copy_inflight_req_begin(s, &req, offset, cur_bytes); + block_copy_task_begin(s, &task, offset, cur_bytes); ret = block_copy_block_status(s, offset, cur_bytes, &status_bytes); assert(ret >= 0); /* never fail */ cur_bytes = MIN(cur_bytes, status_bytes); - block_copy_inflight_req_shrink(s, &req, cur_bytes); + block_copy_task_shrink(s, &task, cur_bytes); if (s->skip_unallocated && !(ret & BDRV_BLOCK_ALLOCATED)) { - block_copy_inflight_req_end(s, &req, 0); + block_copy_task_end(s, &task, 0); progress_set_remaining(s->progress, bdrv_get_dirty_count(s->copy_bitmap) + s->in_flight_bytes); @@ -XXX,XX +XXX,XX @@ static int coroutine_fn block_copy_dirty_clusters(BlockCopyState *s, ret = block_copy_do_copy(s, offset, cur_bytes, ret & BDRV_BLOCK_ZERO, error_is_read); co_put_to_shres(s->mem, cur_bytes); - block_copy_inflight_req_end(s, &req, ret); + block_copy_task_end(s, &task, ret); if (ret < 0) { return ret; } -- 2.26.2 From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> We are going to use aio-task-pool API, so tasks will be handled in parallel. We need therefore separate allocated task on each iteration. Introduce this logic now. Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Message-Id: <20200429130847.28124-3-vsementsov@virtuozzo.com> Signed-off-by: Max Reitz <mreitz@redhat.com> --- block/block-copy.c | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/block/block-copy.c b/block/block-copy.c index XXXXXXX..XXXXXXX 100644 --- a/block/block-copy.c +++ b/block/block-copy.c @@ -XXX,XX +XXX,XX @@ static bool coroutine_fn block_copy_wait_one(BlockCopyState *s, int64_t offset, } /* Called only on full-dirty region */ -static void block_copy_task_begin(BlockCopyState *s, BlockCopyTask *task, - int64_t offset, int64_t bytes) +static BlockCopyTask *block_copy_task_create(BlockCopyState *s, + int64_t offset, int64_t bytes) { + BlockCopyTask *task = g_new(BlockCopyTask, 1); + assert(!find_conflicting_task(s, offset, bytes)); bdrv_reset_dirty_bitmap(s->copy_bitmap, offset, bytes); @@ -XXX,XX +XXX,XX @@ static void block_copy_task_begin(BlockCopyState *s, BlockCopyTask *task, task->bytes = bytes; qemu_co_queue_init(&task->wait_queue); QLIST_INSERT_HEAD(&s->tasks, task, list); + + return task; } /* @@ -XXX,XX +XXX,XX @@ static int coroutine_fn block_copy_dirty_clusters(BlockCopyState *s, assert(QEMU_IS_ALIGNED(bytes, s->cluster_size)); while (bytes) { - BlockCopyTask task; + g_autofree BlockCopyTask *task = NULL; int64_t next_zero, cur_bytes, status_bytes; if (!bdrv_dirty_bitmap_get(s->copy_bitmap, offset)) { @@ -XXX,XX +XXX,XX @@ static int coroutine_fn block_copy_dirty_clusters(BlockCopyState *s, assert(next_zero < offset + cur_bytes); /* no need to do MIN() */ cur_bytes = next_zero - offset; } - block_copy_task_begin(s, &task, offset, cur_bytes); + task = block_copy_task_create(s, offset, cur_bytes); ret = block_copy_block_status(s, offset, cur_bytes, &status_bytes); assert(ret >= 0); /* never fail */ cur_bytes = MIN(cur_bytes, status_bytes); - block_copy_task_shrink(s, &task, cur_bytes); + block_copy_task_shrink(s, task, cur_bytes); if (s->skip_unallocated && !(ret & BDRV_BLOCK_ALLOCATED)) { - block_copy_task_end(s, &task, 0); + block_copy_task_end(s, task, 0); progress_set_remaining(s->progress, bdrv_get_dirty_count(s->copy_bitmap) + s->in_flight_bytes); @@ -XXX,XX +XXX,XX @@ static int coroutine_fn block_copy_dirty_clusters(BlockCopyState *s, ret = block_copy_do_copy(s, offset, cur_bytes, ret & BDRV_BLOCK_ZERO, error_is_read); co_put_to_shres(s->mem, cur_bytes); - block_copy_task_end(s, &task, ret); + block_copy_task_end(s, task, ret); if (ret < 0) { return ret; } -- 2.26.2 From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> We are going to use aio-task-pool API, so we'll need state pointer in BlockCopyTask anyway. Add it now and use where possible. Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Message-Id: <20200429130847.28124-4-vsementsov@virtuozzo.com> Signed-off-by: Max Reitz <mreitz@redhat.com> --- block/block-copy.c | 28 +++++++++++++++------------- 1 file changed, 15 insertions(+), 13 deletions(-) diff --git a/block/block-copy.c b/block/block-copy.c index XXXXXXX..XXXXXXX 100644 --- a/block/block-copy.c +++ b/block/block-copy.c @@ -XXX,XX +XXX,XX @@ #define BLOCK_COPY_MAX_MEM (128 * MiB) typedef struct BlockCopyTask { + BlockCopyState *s; int64_t offset; int64_t bytes; QLIST_ENTRY(BlockCopyTask) list; @@ -XXX,XX +XXX,XX @@ static BlockCopyTask *block_copy_task_create(BlockCopyState *s, bdrv_reset_dirty_bitmap(s->copy_bitmap, offset, bytes); s->in_flight_bytes += bytes; - task->offset = offset; - task->bytes = bytes; + *task = (BlockCopyTask) { + .s = s, + .offset = offset, + .bytes = bytes, + }; qemu_co_queue_init(&task->wait_queue); QLIST_INSERT_HEAD(&s->tasks, task, list); @@ -XXX,XX +XXX,XX @@ static BlockCopyTask *block_copy_task_create(BlockCopyState *s, * wake up all tasks waiting for us (may be some of them are not intersecting * with shrunk task) */ -static void coroutine_fn block_copy_task_shrink(BlockCopyState *s, - BlockCopyTask *task, +static void coroutine_fn block_copy_task_shrink(BlockCopyTask *task, int64_t new_bytes) { if (new_bytes == task->bytes) { @@ -XXX,XX +XXX,XX @@ static void coroutine_fn block_copy_task_shrink(BlockCopyState *s, assert(new_bytes > 0 && new_bytes < task->bytes); - s->in_flight_bytes -= task->bytes - new_bytes; - bdrv_set_dirty_bitmap(s->copy_bitmap, + task->s->in_flight_bytes -= task->bytes - new_bytes; + bdrv_set_dirty_bitmap(task->s->copy_bitmap, task->offset + new_bytes, task->bytes - new_bytes); task->bytes = new_bytes; qemu_co_queue_restart_all(&task->wait_queue); } -static void coroutine_fn block_copy_task_end(BlockCopyState *s, - BlockCopyTask *task, int ret) +static void coroutine_fn block_copy_task_end(BlockCopyTask *task, int ret) { - s->in_flight_bytes -= task->bytes; + task->s->in_flight_bytes -= task->bytes; if (ret < 0) { - bdrv_set_dirty_bitmap(s->copy_bitmap, task->offset, task->bytes); + bdrv_set_dirty_bitmap(task->s->copy_bitmap, task->offset, task->bytes); } QLIST_REMOVE(task, list); qemu_co_queue_restart_all(&task->wait_queue); @@ -XXX,XX +XXX,XX @@ static int coroutine_fn block_copy_dirty_clusters(BlockCopyState *s, ret = block_copy_block_status(s, offset, cur_bytes, &status_bytes); assert(ret >= 0); /* never fail */ cur_bytes = MIN(cur_bytes, status_bytes); - block_copy_task_shrink(s, task, cur_bytes); + block_copy_task_shrink(task, cur_bytes); if (s->skip_unallocated && !(ret & BDRV_BLOCK_ALLOCATED)) { - block_copy_task_end(s, task, 0); + block_copy_task_end(task, 0); progress_set_remaining(s->progress, bdrv_get_dirty_count(s->copy_bitmap) + s->in_flight_bytes); @@ -XXX,XX +XXX,XX @@ static int coroutine_fn block_copy_dirty_clusters(BlockCopyState *s, ret = block_copy_do_copy(s, offset, cur_bytes, ret & BDRV_BLOCK_ZERO, error_is_read); co_put_to_shres(s->mem, cur_bytes); - block_copy_task_end(s, task, ret); + block_copy_task_end(task, ret); if (ret < 0) { return ret; } -- 2.26.2 From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Instead of just relying on the comment "Called only on full-dirty region" in block_copy_task_create() let's move initial dirty area search directly to block_copy_task_create(). Let's also use effective bdrv_dirty_bitmap_next_dirty_area instead of looping through all non-dirty clusters. Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Message-Id: <20200429130847.28124-5-vsementsov@virtuozzo.com> Signed-off-by: Max Reitz <mreitz@redhat.com> --- block/block-copy.c | 80 ++++++++++++++++++++++++++-------------------- 1 file changed, 46 insertions(+), 34 deletions(-) diff --git a/block/block-copy.c b/block/block-copy.c index XXXXXXX..XXXXXXX 100644 --- a/block/block-copy.c +++ b/block/block-copy.c @@ -XXX,XX +XXX,XX @@ typedef struct BlockCopyTask { CoQueue wait_queue; /* coroutines blocked on this task */ } BlockCopyTask; +static int64_t task_end(BlockCopyTask *task) +{ + return task->offset + task->bytes; +} + typedef struct BlockCopyState { /* * BdrvChild objects are not owned or managed by block-copy. They are @@ -XXX,XX +XXX,XX @@ static bool coroutine_fn block_copy_wait_one(BlockCopyState *s, int64_t offset, return true; } -/* Called only on full-dirty region */ +/* + * Search for the first dirty area in offset/bytes range and create task at + * the beginning of it. + */ static BlockCopyTask *block_copy_task_create(BlockCopyState *s, int64_t offset, int64_t bytes) { - BlockCopyTask *task = g_new(BlockCopyTask, 1); + BlockCopyTask *task; + if (!bdrv_dirty_bitmap_next_dirty_area(s->copy_bitmap, + offset, offset + bytes, + s->copy_size, &offset, &bytes)) + { + return NULL; + } + + /* region is dirty, so no existent tasks possible in it */ assert(!find_conflicting_task(s, offset, bytes)); bdrv_reset_dirty_bitmap(s->copy_bitmap, offset, bytes); s->in_flight_bytes += bytes; + task = g_new(BlockCopyTask, 1); *task = (BlockCopyTask) { .s = s, .offset = offset, @@ -XXX,XX +XXX,XX @@ static int coroutine_fn block_copy_dirty_clusters(BlockCopyState *s, { int ret = 0; bool found_dirty = false; + int64_t end = offset + bytes; /* * block_copy() user is responsible for keeping source and target in same @@ -XXX,XX +XXX,XX @@ static int coroutine_fn block_copy_dirty_clusters(BlockCopyState *s, while (bytes) { g_autofree BlockCopyTask *task = NULL; - int64_t next_zero, cur_bytes, status_bytes; + int64_t status_bytes; - if (!bdrv_dirty_bitmap_get(s->copy_bitmap, offset)) { - trace_block_copy_skip(s, offset); - offset += s->cluster_size; - bytes -= s->cluster_size; - continue; /* already copied */ + task = block_copy_task_create(s, offset, bytes); + if (!task) { + /* No more dirty bits in the bitmap */ + trace_block_copy_skip_range(s, offset, bytes); + break; + } + if (task->offset > offset) { + trace_block_copy_skip_range(s, offset, task->offset - offset); } found_dirty = true; - cur_bytes = MIN(bytes, s->copy_size); - - next_zero = bdrv_dirty_bitmap_next_zero(s->copy_bitmap, offset, - cur_bytes); - if (next_zero >= 0) { - assert(next_zero > offset); /* offset is dirty */ - assert(next_zero < offset + cur_bytes); /* no need to do MIN() */ - cur_bytes = next_zero - offset; - } - task = block_copy_task_create(s, offset, cur_bytes); - - ret = block_copy_block_status(s, offset, cur_bytes, &status_bytes); + ret = block_copy_block_status(s, task->offset, task->bytes, + &status_bytes); assert(ret >= 0); /* never fail */ - cur_bytes = MIN(cur_bytes, status_bytes); - block_copy_task_shrink(task, cur_bytes); + if (status_bytes < task->bytes) { + block_copy_task_shrink(task, status_bytes); + } if (s->skip_unallocated && !(ret & BDRV_BLOCK_ALLOCATED)) { block_copy_task_end(task, 0); progress_set_remaining(s->progress, bdrv_get_dirty_count(s->copy_bitmap) + s->in_flight_bytes); - trace_block_copy_skip_range(s, offset, status_bytes); - offset += status_bytes; - bytes -= status_bytes; + trace_block_copy_skip_range(s, task->offset, task->bytes); + offset = task_end(task); + bytes = end - offset; continue; } - trace_block_copy_process(s, offset); + trace_block_copy_process(s, task->offset); - co_get_from_shres(s->mem, cur_bytes); - ret = block_copy_do_copy(s, offset, cur_bytes, ret & BDRV_BLOCK_ZERO, - error_is_read); - co_put_to_shres(s->mem, cur_bytes); + co_get_from_shres(s->mem, task->bytes); + ret = block_copy_do_copy(s, task->offset, task->bytes, + ret & BDRV_BLOCK_ZERO, error_is_read); + co_put_to_shres(s->mem, task->bytes); block_copy_task_end(task, ret); if (ret < 0) { return ret; } - progress_work_done(s->progress, cur_bytes); - s->progress_bytes_callback(cur_bytes, s->progress_opaque); - offset += cur_bytes; - bytes -= cur_bytes; + progress_work_done(s->progress, task->bytes); + s->progress_bytes_callback(task->bytes, s->progress_opaque); + offset = task_end(task); + bytes = end - offset; } return found_dirty; -- 2.26.2 From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Run block_copy iterations in parallel in aio tasks. Changes: - BlockCopyTask becomes aio task structure. Add zeroes field to pass it to block_copy_do_copy - add call state - it's a state of one call of block_copy(), shared between parallel tasks. For now used only to keep information about first error: is it read or not. - convert block_copy_dirty_clusters to aio-task loop. Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Message-Id: <20200429130847.28124-6-vsementsov@virtuozzo.com> Signed-off-by: Max Reitz <mreitz@redhat.com> --- block/block-copy.c | 119 ++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 106 insertions(+), 13 deletions(-) diff --git a/block/block-copy.c b/block/block-copy.c index XXXXXXX..XXXXXXX 100644 --- a/block/block-copy.c +++ b/block/block-copy.c @@ -XXX,XX +XXX,XX @@ #include "block/block-copy.h" #include "sysemu/block-backend.h" #include "qemu/units.h" +#include "qemu/coroutine.h" +#include "block/aio_task.h" #define BLOCK_COPY_MAX_COPY_RANGE (16 * MiB) #define BLOCK_COPY_MAX_BUFFER (1 * MiB) #define BLOCK_COPY_MAX_MEM (128 * MiB) +#define BLOCK_COPY_MAX_WORKERS 64 + +static coroutine_fn int block_copy_task_entry(AioTask *task); + +typedef struct BlockCopyCallState { + bool failed; + bool error_is_read; +} BlockCopyCallState; typedef struct BlockCopyTask { + AioTask task; + BlockCopyState *s; + BlockCopyCallState *call_state; int64_t offset; int64_t bytes; + bool zeroes; QLIST_ENTRY(BlockCopyTask) list; CoQueue wait_queue; /* coroutines blocked on this task */ } BlockCopyTask; @@ -XXX,XX +XXX,XX @@ static bool coroutine_fn block_copy_wait_one(BlockCopyState *s, int64_t offset, * the beginning of it. */ static BlockCopyTask *block_copy_task_create(BlockCopyState *s, + BlockCopyCallState *call_state, int64_t offset, int64_t bytes) { BlockCopyTask *task; @@ -XXX,XX +XXX,XX @@ static BlockCopyTask *block_copy_task_create(BlockCopyState *s, task = g_new(BlockCopyTask, 1); *task = (BlockCopyTask) { + .task.func = block_copy_task_entry, .s = s, + .call_state = call_state, .offset = offset, .bytes = bytes, }; @@ -XXX,XX +XXX,XX @@ void block_copy_set_progress_meter(BlockCopyState *s, ProgressMeter *pm) s->progress = pm; } +/* + * Takes ownership of @task + * + * If pool is NULL directly run the task, otherwise schedule it into the pool. + * + * Returns: task.func return code if pool is NULL + * otherwise -ECANCELED if pool status is bad + * otherwise 0 (successfully scheduled) + */ +static coroutine_fn int block_copy_task_run(AioTaskPool *pool, + BlockCopyTask *task) +{ + if (!pool) { + int ret = task->task.func(&task->task); + + g_free(task); + return ret; + } + + aio_task_pool_wait_slot(pool); + if (aio_task_pool_status(pool) < 0) { + co_put_to_shres(task->s->mem, task->bytes); + block_copy_task_end(task, -ECANCELED); + g_free(task); + return -ECANCELED; + } + + aio_task_pool_start_task(pool, &task->task); + + return 0; +} + /* * block_copy_do_copy * @@ -XXX,XX +XXX,XX @@ out: return ret; } +static coroutine_fn int block_copy_task_entry(AioTask *task) +{ + BlockCopyTask *t = container_of(task, BlockCopyTask, task); + bool error_is_read; + int ret; + + ret = block_copy_do_copy(t->s, t->offset, t->bytes, t->zeroes, + &error_is_read); + if (ret < 0 && !t->call_state->failed) { + t->call_state->failed = true; + t->call_state->error_is_read = error_is_read; + } else { + progress_work_done(t->s->progress, t->bytes); + t->s->progress_bytes_callback(t->bytes, t->s->progress_opaque); + } + co_put_to_shres(t->s->mem, t->bytes); + block_copy_task_end(t, ret); + + return ret; +} + static int block_copy_block_status(BlockCopyState *s, int64_t offset, int64_t bytes, int64_t *pnum) { @@ -XXX,XX +XXX,XX @@ static int coroutine_fn block_copy_dirty_clusters(BlockCopyState *s, int ret = 0; bool found_dirty = false; int64_t end = offset + bytes; + AioTaskPool *aio = NULL; + BlockCopyCallState call_state = {false, false}; /* * block_copy() user is responsible for keeping source and target in same @@ -XXX,XX +XXX,XX @@ static int coroutine_fn block_copy_dirty_clusters(BlockCopyState *s, assert(QEMU_IS_ALIGNED(offset, s->cluster_size)); assert(QEMU_IS_ALIGNED(bytes, s->cluster_size)); - while (bytes) { - g_autofree BlockCopyTask *task = NULL; + while (bytes && aio_task_pool_status(aio) == 0) { + BlockCopyTask *task; int64_t status_bytes; - task = block_copy_task_create(s, offset, bytes); + task = block_copy_task_create(s, &call_state, offset, bytes); if (!task) { /* No more dirty bits in the bitmap */ trace_block_copy_skip_range(s, offset, bytes); @@ -XXX,XX +XXX,XX @@ static int coroutine_fn block_copy_dirty_clusters(BlockCopyState *s, } if (s->skip_unallocated && !(ret & BDRV_BLOCK_ALLOCATED)) { block_copy_task_end(task, 0); + g_free(task); progress_set_remaining(s->progress, bdrv_get_dirty_count(s->copy_bitmap) + s->in_flight_bytes); @@ -XXX,XX +XXX,XX @@ static int coroutine_fn block_copy_dirty_clusters(BlockCopyState *s, bytes = end - offset; continue; } + task->zeroes = ret & BDRV_BLOCK_ZERO; trace_block_copy_process(s, task->offset); co_get_from_shres(s->mem, task->bytes); - ret = block_copy_do_copy(s, task->offset, task->bytes, - ret & BDRV_BLOCK_ZERO, error_is_read); - co_put_to_shres(s->mem, task->bytes); - block_copy_task_end(task, ret); - if (ret < 0) { - return ret; - } - progress_work_done(s->progress, task->bytes); - s->progress_bytes_callback(task->bytes, s->progress_opaque); offset = task_end(task); bytes = end - offset; + + if (!aio && bytes) { + aio = aio_task_pool_new(BLOCK_COPY_MAX_WORKERS); + } + + ret = block_copy_task_run(aio, task); + if (ret < 0) { + goto out; + } + } + +out: + if (aio) { + aio_task_pool_wait_all(aio); + + /* + * We are not really interested in -ECANCELED returned from + * block_copy_task_run. If it fails, it means some task already failed + * for real reason, let's return first failure. + * Still, assert that we don't rewrite failure by success. + */ + assert(ret == 0 || aio_task_pool_status(aio) < 0); + ret = aio_task_pool_status(aio); + + aio_task_pool_free(aio); + } + if (error_is_read && ret < 0) { + *error_is_read = call_state.error_is_read; } - return found_dirty; + return ret < 0 ? ret : found_dirty; } /* -- 2.26.2