The following changes since commit 5375af3cd7b8adcc10c18d8083b7be63976c9645:

Merge remote-tracking branch 'remotes/mst/tags/for_upstream' into staging (2020-05-04 15:51:09 +0100)

are available in the Git repository at:

https://github.com/XanClic/qemu.git tags/pull-block-2020-05-05

for you to fetch changes up to 4ce5dd3e9b5ee0fac18625860eb3727399ee965e:

block/block-copy: use aio-task-pool API (2020-05-05 14:03:28 +0200)

Block patches:

- Asynchronous copying for block-copy (i.e., the backup job)

- Allow resizing of qcow2 images when they have internal snapshots

- iotests: Logging improvements for Python tests

- iotest 153 fix, and block comment cleanups

John Snow (14):

iotests: do a light delinting

iotests: don't use 'format' for drive_add

iotests: ignore import warnings from pylint

iotests: replace mutable list default args

iotests: add pylintrc file

iotests: alphabetize standard imports

iotests: drop pre-Python 3.4 compatibility code

iotests: touch up log function signature

iotests: limit line length to 79 chars

iotests: add hmp helper with logging

iotests: add script_initialize

iotest 258: use script_main

iotests: Mark verify functions as private

iotests: use python logging for iotests.log()

Maxim Levitsky (1):

Fix iotest 153

Vladimir Sementsov-Ogievskiy (5):

block/block-copy: rename in-flight requests to tasks

block/block-copy: alloc task on each iteration

block/block-copy: add state pointer to BlockCopyTask

block/block-copy: refactor task creation

block/block-copy: use aio-task-pool API

include/sysemu/block-backend.h | 2 +

block/block-backend.c | 23 +++

block/block-copy.c | 279 +++++++++++++++++--------

block/crypto.c | 9 +-

block/io.c | 3 +-

block/parallels.c | 8 +-

block/qcow.c | 8 +-

block/qcow2-refcount.c | 2 +-

block/qcow2-snapshot.c | 20 +-

block/qcow2.c | 45 ++--

block/qed.c | 8 +-

block/sheepdog.c | 10 +-

block/vdi.c | 8 +-

block/vhdx.c | 8 +-

block/vmdk.c | 9 +-

block/vpc.c | 8 +-

block/vvfat.c | 10 +-

blockdev.c | 8 +-

blockjob.c | 7 +-

tests/qemu-iotests/001 | 2 +-

tests/qemu-iotests/030 | 4 +-

tests/qemu-iotests/052 | 2 +-

tests/qemu-iotests/055 | 3 +-

tests/qemu-iotests/061 | 35 ++++

tests/qemu-iotests/061.out | 28 +++

tests/qemu-iotests/134 | 2 +-

tests/qemu-iotests/149 | 3 +-

tests/qemu-iotests/153 | 2 +-

tests/qemu-iotests/153.out | 12 +-

tests/qemu-iotests/155 | 2 +-

tests/qemu-iotests/188 | 2 +-

tests/qemu-iotests/194 | 4 +-

tests/qemu-iotests/202 | 4 +-

tests/qemu-iotests/203 | 4 +-

tests/qemu-iotests/206 | 2 +-

tests/qemu-iotests/207 | 6 +-

tests/qemu-iotests/208 | 2 +-

tests/qemu-iotests/209 | 2 +-

tests/qemu-iotests/210 | 6 +-

tests/qemu-iotests/211 | 6 +-

tests/qemu-iotests/212 | 6 +-

tests/qemu-iotests/213 | 6 +-

tests/qemu-iotests/216 | 4 +-

tests/qemu-iotests/218 | 2 +-

tests/qemu-iotests/219 | 2 +-

tests/qemu-iotests/222 | 7 +-

tests/qemu-iotests/224 | 4 +-

tests/qemu-iotests/228 | 6 +-

tests/qemu-iotests/234 | 4 +-

tests/qemu-iotests/235 | 4 +-

tests/qemu-iotests/236 | 2 +-

tests/qemu-iotests/237 | 2 +-

tests/qemu-iotests/238 | 2 +

tests/qemu-iotests/242 | 2 +-

tests/qemu-iotests/245 | 1 +

tests/qemu-iotests/245.out | 10 +-

tests/qemu-iotests/246 | 2 +-

tests/qemu-iotests/248 | 2 +-

tests/qemu-iotests/254 | 2 +-

tests/qemu-iotests/255 | 2 +-

tests/qemu-iotests/256 | 2 +-

tests/qemu-iotests/258 | 10 +-

tests/qemu-iotests/260 | 4 +-

tests/qemu-iotests/262 | 4 +-

tests/qemu-iotests/264 | 4 +-

tests/qemu-iotests/274 | 4 +-

tests/qemu-iotests/277 | 2 +

tests/qemu-iotests/280 | 8 +-

tests/qemu-iotests/283 | 4 +-

tests/qemu-iotests/iotests.py | 366 ++++++++++++++++++++-------------

tests/qemu-iotests/pylintrc | 26 +++

71 files changed, 728 insertions(+), 386 deletions(-)

create mode 100644 tests/qemu-iotests/pylintrc

2.26.2

From: John Snow <jsnow@redhat.com>

We can turn logging on/off globally instead of per-function.

Remove use_log from run_job, and use python logging to turn on

diffable output when we run through a script entry point.

iotest 245 changes output order due to buffering reasons.

An extended note on python logging:

A NullHandler is added to `qemu.iotests` to stop output from being

generated if this code is used as a library without configuring logging.

A NullHandler is only needed at the root, so a duplicate handler is not

needed for `qemu.iotests.diff_io`.

When logging is not configured, messages at the 'WARNING' levels or

above are printed with default settings. The NullHandler stops this from

occurring, which is considered good hygiene for code used as a library.

See https://docs.python.org/3/howto/logging.html#library-config

When logging is actually enabled (always at the behest of an explicit

call by a client script), a root logger is implicitly created at the

root, which allows messages to propagate upwards and be handled/emitted

from the root logger with default settings.

When we want iotest logging, we attach a handler to the

qemu.iotests.diff_io logger and disable propagation to avoid possible

double-printing.

For more information on python logging infrastructure, I highly

recommend downloading the pip package `logging_tree`, which provides

convenient visualizations of the hierarchical logging configuration

under different circumstances.

See https://pypi.org/project/logging_tree/ for more information.

Signed-off-by: John Snow <jsnow@redhat.com>

Reviewed-by: Max Reitz <mreitz@redhat.com>

Message-Id: <20200331000014.11581-15-jsnow@redhat.com>

Reviewed-by: Kevin Wolf <kwolf@redhat.com>

Signed-off-by: Max Reitz <mreitz@redhat.com>

tests/qemu-iotests/030 | 4 +--

tests/qemu-iotests/155 | 2 +-

tests/qemu-iotests/245 | 1 +

tests/qemu-iotests/245.out | 10 +++----

tests/qemu-iotests/iotests.py | 53 ++++++++++++++++++++---------------

5 files changed, 39 insertions(+), 31 deletions(-)

diff --git a/tests/qemu-iotests/030 b/tests/qemu-iotests/030

index XXXXXXX..XXXXXXX 100755

--- a/tests/qemu-iotests/030

+++ b/tests/qemu-iotests/030

@@ -XXX,XX +XXX,XX @@ class TestParallelOps(iotests.QMPTestCase):

result = self.vm.qmp('block-job-set-speed', device='drive0', speed=0)

self.assert_qmp(result, 'return', {})

- self.vm.run_job(job='drive0', auto_dismiss=True, use_log=False)

- self.vm.run_job(job='node4', auto_dismiss=True, use_log=False)

+ self.vm.run_job(job='drive0', auto_dismiss=True)

+ self.vm.run_job(job='node4', auto_dismiss=True)

self.assert_no_active_block_jobs()

# Test a block-stream and a block-commit job in parallel

diff --git a/tests/qemu-iotests/155 b/tests/qemu-iotests/155

index XXXXXXX..XXXXXXX 100755

--- a/tests/qemu-iotests/155

+++ b/tests/qemu-iotests/155

@@ -XXX,XX +XXX,XX @@ class MirrorBaseClass(BaseClass):

self.assert_qmp(result, 'return', {})

- self.vm.run_job('mirror-job', use_log=False, auto_finalize=False,

+ self.vm.run_job('mirror-job', auto_finalize=False,

pre_finalize=self.openBacking, auto_dismiss=True)

def testFull(self):

diff --git a/tests/qemu-iotests/245 b/tests/qemu-iotests/245

index XXXXXXX..XXXXXXX 100755

--- a/tests/qemu-iotests/245

+++ b/tests/qemu-iotests/245

@@ -XXX,XX +XXX,XX @@ class TestBlockdevReopen(iotests.QMPTestCase):

self.run_test_iothreads(None, 'iothread0')

if __name__ == '__main__':

+ iotests.activate_logging()

iotests.main(supported_fmts=["qcow2"],

supported_protocols=["file"])

diff --git a/tests/qemu-iotests/245.out b/tests/qemu-iotests/245.out

index XXXXXXX..XXXXXXX 100644

--- a/tests/qemu-iotests/245.out

+++ b/tests/qemu-iotests/245.out

@@ -XXX,XX +XXX,XX @@

-.....................

-----------------------------------------------------------------------

-Ran 21 tests

-

-OK

{"execute": "job-finalize", "arguments": {"id": "commit0"}}

{"return": {}}

{"data": {"id": "commit0", "type": "commit"}, "event": "BLOCK_JOB_PENDING", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}

@@ -XXX,XX +XXX,XX @@ OK

{"return": {}}

{"data": {"id": "stream0", "type": "stream"}, "event": "BLOCK_JOB_PENDING", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}

{"data": {"device": "stream0", "len": 3145728, "offset": 3145728, "speed": 0, "type": "stream"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}

+.....................

+----------------------------------------------------------------------

+Ran 21 tests

+

+OK

diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py

index XXXXXXX..XXXXXXX 100644

--- a/tests/qemu-iotests/iotests.py

+++ b/tests/qemu-iotests/iotests.py

@@ -XXX,XX +XXX,XX @@ assert sys.version_info >= (3, 6)

QMPResponse = Dict[str, Any]

+# Use this logger for logging messages directly from the iotests module

+logger = logging.getLogger('qemu.iotests')

+logger.addHandler(logging.NullHandler())

+

+# Use this logger for messages that ought to be used for diff output.

+test_logger = logging.getLogger('qemu.iotests.diff_io')

+

+

faulthandler.enable()

# This will not work if arguments contain spaces but is necessary if we

@@ -XXX,XX +XXX,XX @@ def log(msg: Msg,

if isinstance(msg, (dict, list)):

# Don't sort if it's already sorted

do_sort = not isinstance(msg, OrderedDict)

- print(json.dumps(msg, sort_keys=do_sort, indent=indent))

+ test_logger.info(json.dumps(msg, sort_keys=do_sort, indent=indent))

else:

- print(msg)

+ test_logger.info(msg)

class Timeout:

def __init__(self, seconds, errmsg="Timeout"):

@@ -XXX,XX +XXX,XX @@ class VM(qtest.QEMUQtestMachine):

# Returns None on success, and an error string on failure

def run_job(self, job, auto_finalize=True, auto_dismiss=False,

- pre_finalize=None, cancel=False, use_log=True, wait=60.0):

+ pre_finalize=None, cancel=False, wait=60.0):

"""

run_job moves a job from creation through to dismissal.

@@ -XXX,XX +XXX,XX @@ class VM(qtest.QEMUQtestMachine):

invoked prior to issuing job-finalize, if any.

:param cancel: Bool. When true, cancels the job after the pre_finalize

callback.

- :param use_log: Bool. When false, does not log QMP messages.

:param wait: Float. Timeout value specifying how long to wait for any

event, in seconds. Defaults to 60.0.

"""

@@ -XXX,XX +XXX,XX @@ class VM(qtest.QEMUQtestMachine):

while True:

ev = filter_qmp_event(self.events_wait(events, timeout=wait))

if ev['event'] != 'JOB_STATUS_CHANGE':

- if use_log:

- log(ev)

+ log(ev)

continue

status = ev['data']['status']

if status == 'aborting':

@@ -XXX,XX +XXX,XX @@ class VM(qtest.QEMUQtestMachine):

for j in result['return']:

if j['id'] == job:

error = j['error']

- if use_log:

- log('Job failed: %s' % (j['error']))

+ log('Job failed: %s' % (j['error']))

elif status == 'ready':

- if use_log:

- self.qmp_log('job-complete', id=job)

- else:

- self.qmp('job-complete', id=job)

+ self.qmp_log('job-complete', id=job)

elif status == 'pending' and not auto_finalize:

if pre_finalize:

pre_finalize()

- if cancel and use_log:

+ if cancel:

self.qmp_log('job-cancel', id=job)

- elif cancel:

- self.qmp('job-cancel', id=job)

- elif use_log:

- self.qmp_log('job-finalize', id=job)

else:

- self.qmp('job-finalize', id=job)

+ self.qmp_log('job-finalize', id=job)

elif status == 'concluded' and not auto_dismiss:

- if use_log:

- self.qmp_log('job-dismiss', id=job)

- else:

- self.qmp('job-dismiss', id=job)

+ self.qmp_log('job-dismiss', id=job)

elif status == 'null':

return error

@@ -XXX,XX +XXX,XX @@ def notrun(reason):

seq = os.path.basename(sys.argv[0])

open('%s/%s.notrun' % (output_dir, seq), 'w').write(reason + '\n')

- print('%s not run: %s' % (seq, reason))

+ logger.warning("%s not run: %s", seq, reason)

sys.exit(0)

def case_notrun(reason):

@@ -XXX,XX +XXX,XX @@ def execute_setup_common(supported_fmts: Sequence[str] = (),

if debug:

sys.argv.remove('-d')

logging.basicConfig(level=(logging.DEBUG if debug else logging.WARN))

+ logger.debug("iotests debugging messages active")

return debug

@@ -XXX,XX +XXX,XX @@ def execute_test(*args, test_function=None, **kwargs):

else:

test_function()

+def activate_logging():

+ """Activate iotests.log() output to stdout for script-style tests."""

+ handler = logging.StreamHandler(stream=sys.stdout)

+ formatter = logging.Formatter('%(message)s')

+ handler.setFormatter(formatter)

+ test_logger.addHandler(handler)

+ test_logger.setLevel(logging.INFO)

+ test_logger.propagate = False

+

# This is called from script-style iotests without a single point of entry

def script_initialize(*args, **kwargs):

"""Initialize script-style tests without running any tests."""

+ activate_logging()

execute_setup_common(*args, **kwargs)

# This is called from script-style iotests with a single point of entry

def script_main(test_function, *args, **kwargs):

"""Run script-style tests outside of the unittest framework"""

+ activate_logging()

execute_test(*args, test_function=test_function, **kwargs)

# This is called from unittest style iotests

2.26.2

From: John Snow <jsnow@redhat.com>

This allows others to get repeatable results with pylint. If you run

`pylint iotests.py`, you should see a 100% pass.

Signed-off-by: John Snow <jsnow@redhat.com>

Reviewed-by: Max Reitz <mreitz@redhat.com>

Message-Id: <20200331000014.11581-6-jsnow@redhat.com>

Reviewed-by: Kevin Wolf <kwolf@redhat.com>

Signed-off-by: Max Reitz <mreitz@redhat.com>

tests/qemu-iotests/pylintrc | 22 ++++++++++++++++++++++

1 file changed, 22 insertions(+)

create mode 100644 tests/qemu-iotests/pylintrc

diff --git a/tests/qemu-iotests/pylintrc b/tests/qemu-iotests/pylintrc

+++ b/tests/qemu-iotests/pylintrc

+[MESSAGES CONTROL]

+

+# Disable the message, report, category or checker with the given id(s). You

+# can either give multiple identifiers separated by comma (,) or put this

+# option multiple times (only on the command line, not in the configuration

+# file where it should appear only once). You can also use "--disable=all" to

+# disable everything first and then reenable specific checks. For example, if

+# you want to run only the similarities checker, you can use "--disable=all

+# --enable=similarities". If you want to run only the classes checker, but have

+# no Warning level messages displayed, use "--disable=all --enable=classes

+# --disable=W".

+disable=invalid-name,

+ no-else-return,

+ too-few-public-methods,

+ too-many-arguments,

+ too-many-branches,

+ too-many-lines,

+ too-many-locals,

+ too-many-public-methods,

+ # These are temporary, and should be removed:

+ line-too-long,

+ missing-docstring,

2.26.2

From: John Snow <jsnow@redhat.com>

Mark the verify functions as "private" with a leading underscore, to

discourage their use. Update type signatures while we're here.

(Also, make pending patches not yet using the new entry points fail in a

very obvious way.)

Signed-off-by: John Snow <jsnow@redhat.com>

Message-Id: <20200331000014.11581-14-jsnow@redhat.com>

Reviewed-by: Max Reitz <mreitz@redhat.com>