include/block/block.h | 17 +- include/block/block_int.h | 26 +-- include/block/throttle-groups.h | 2 +- include/qemu/iov.h | 2 +- block/blkverify.c | 2 +- block/file-posix.c | 2 +- block/io.c | 274 ++++++++++++++++++++++---------- block/throttle-groups.c | 5 +- tests/test-write-threshold.c | 5 +- util/iov.c | 25 ++- block/trace-events | 12 +- 11 files changed, 252 insertions(+), 120 deletions(-)
Hi all!
We want 64bit write-zeroes, and for this, convert all io functions to
64bit.
We chose signed type, to be consistent with off_t (which is signed) and
with possibility for signed return type (where negative value means
error).
Please refer to initial cover-letter
https://lists.gnu.org/archive/html/qemu-devel/2020-03/msg08723.html
for more info.
v4: I found, that some more work is needed for block/block-backend, so
decided to make partI, converting block/io
v4 is based on Kevin's block branch ([PULL 00/34] Block layer patches)
for BDRV_MAX_LENGTH
changes:
01-05: new
06: add Alberto's r-b
07: new
08-16: rebase, add new-style request check, improve commit-msg, drop r-bs
Based-on: <20201211170812.228643-1-kwolf@redhat.com>
Vladimir Sementsov-Ogievskiy (16):
block: refactor bdrv_check_request: add errp
util/iov: make qemu_iovec_init_extended() honest
block: fix theoretical overflow in bdrv_init_padding()
block/io: refactor bdrv_pad_request(): move bdrv_pad_request() up
block/io: bdrv_pad_request(): support qemu_iovec_init_extended failure
block/throttle-groups: throttle_group_co_io_limits_intercept(): 64bit
bytes
block/io: improve bdrv_check_request: check qiov too
block: use int64_t as bytes type in tracked requests
block/io: use int64_t bytes in driver wrappers
block/io: support int64_t bytes in bdrv_co_do_pwrite_zeroes()
block/io: support int64_t bytes in bdrv_aligned_pwritev()
block/io: support int64_t bytes in bdrv_co_do_copy_on_readv()
block/io: support int64_t bytes in bdrv_aligned_preadv()
block/io: support int64_t bytes in bdrv_co_p{read,write}v_part()
block/io: support int64_t bytes in read/write wrappers
block/io: use int64_t bytes in copy_range
include/block/block.h | 17 +-
include/block/block_int.h | 26 +--
include/block/throttle-groups.h | 2 +-
include/qemu/iov.h | 2 +-
block/blkverify.c | 2 +-
block/file-posix.c | 2 +-
block/io.c | 274 ++++++++++++++++++++++----------
block/throttle-groups.c | 5 +-
tests/test-write-threshold.c | 5 +-
util/iov.c | 25 ++-
block/trace-events | 12 +-
11 files changed, 252 insertions(+), 120 deletions(-)
--
2.25.4
On 12/11/20 12:39 PM, Vladimir Sementsov-Ogievskiy wrote:
> Hi all!
>
> We want 64bit write-zeroes, and for this, convert all io functions to
> 64bit.
>
> We chose signed type, to be consistent with off_t (which is signed) and
> with possibility for signed return type (where negative value means
> error).
>
> Please refer to initial cover-letter
> https://lists.gnu.org/archive/html/qemu-devel/2020-03/msg08723.html
> for more info.
>
> v4: I found, that some more work is needed for block/block-backend, so
> decided to make partI, converting block/io
>
> v4 is based on Kevin's block branch ([PULL 00/34] Block layer patches)
> for BDRV_MAX_LENGTH
>
> changes:
> 01-05: new
> 06: add Alberto's r-b
> 07: new
> 08-16: rebase, add new-style request check, improve commit-msg, drop r-bs
I had planned to send a pull request for this series today, but ran into
a snag. Without this series applied, './check -qcow2' fails 030, 185,
and 297. With it applied, I now also get a failure in 206. I'm trying
to bisect which patch caused the problem, but here's the failure:
206 fail [20:54:54] [20:55:01] 6.9s (last: 6.7s) output
mismatch (see 206.out.bad)
--- /home/eblake/qemu/tests/qemu-iotests/206.out
+++ 206.out.bad
@@ -180,7 +180,7 @@
{"execute": "blockdev-create", "arguments": {"job-id": "job0",
"options": {"driver": "qcow2", "file": "node0", "size":
9223372036854775296}}}
{"return": {}}
-Job failed: Could not resize image: Required too big image size, it
must be not greater than 9223372035781033984
+Job failed: Could not resize image: offset(9223372036854775296) exceeds
maximum(9223372035781033984)
{"execute": "job-dismiss", "arguments": {"id": "job0"}}
{"return": {}}
Looks like it is just a changed error message, so I can touch up the
correct patch and then repackage the pull request tomorrow (it's too
late for me today). Oh, and the 0 exit status of ./check when a test
fails is something I see you already plan on fixing...
--
Eric Blake, Principal Software Engineer
Red Hat, Inc. +1-919-301-3226
Virtualization: qemu.org | libvirt.org
02.02.2021 05:56, Eric Blake wrote:
> On 12/11/20 12:39 PM, Vladimir Sementsov-Ogievskiy wrote:
>> Hi all!
>>
>> We want 64bit write-zeroes, and for this, convert all io functions to
>> 64bit.
>>
>> We chose signed type, to be consistent with off_t (which is signed) and
>> with possibility for signed return type (where negative value means
>> error).
>>
>> Please refer to initial cover-letter
>> https://lists.gnu.org/archive/html/qemu-devel/2020-03/msg08723.html
>> for more info.
>>
>> v4: I found, that some more work is needed for block/block-backend, so
>> decided to make partI, converting block/io
>>
>> v4 is based on Kevin's block branch ([PULL 00/34] Block layer patches)
>> for BDRV_MAX_LENGTH
>>
>> changes:
>> 01-05: new
>> 06: add Alberto's r-b
>> 07: new
>> 08-16: rebase, add new-style request check, improve commit-msg, drop r-bs
>
> I had planned to send a pull request for this series today, but ran into
> a snag. Without this series applied, './check -qcow2' fails 030, 185,
> and 297. With it applied, I now also get a failure in 206. I'm trying
> to bisect which patch caused the problem, but here's the failure:
>
> 206 fail [20:54:54] [20:55:01] 6.9s (last: 6.7s) output
> mismatch (see 206.out.bad)
> --- /home/eblake/qemu/tests/qemu-iotests/206.out
> +++ 206.out.bad
> @@ -180,7 +180,7 @@
>
> {"execute": "blockdev-create", "arguments": {"job-id": "job0",
> "options": {"driver": "qcow2", "file": "node0", "size":
> 9223372036854775296}}}
> {"return": {}}
> -Job failed: Could not resize image: Required too big image size, it
> must be not greater than 9223372035781033984
> +Job failed: Could not resize image: offset(9223372036854775296) exceeds
> maximum(9223372035781033984)
> {"execute": "job-dismiss", "arguments": {"id": "job0"}}
> {"return": {}}
>
> Looks like it is just a changed error message, so I can touch up the
> correct patch and then repackage the pull request tomorrow (it's too
> late for me today). Oh, and the 0 exit status of ./check when a test
> fails is something I see you already plan on fixing...
>
Yes, Kevin have already sent a pull with "iotests: check: return 1 on failure"
--
Best regards,
Vladimir
On 2/1/21 8:56 PM, Eric Blake wrote:
> I had planned to send a pull request for this series today, but ran into
> a snag. Without this series applied, './check -qcow2' fails 030, 185,
> and 297. With it applied, I now also get a failure in 206. I'm trying
> to bisect which patch caused the problem, but here's the failure:
>
> 206 fail [20:54:54] [20:55:01] 6.9s (last: 6.7s) output
> mismatch (see 206.out.bad)
> --- /home/eblake/qemu/tests/qemu-iotests/206.out
> +++ 206.out.bad
> @@ -180,7 +180,7 @@
>
> {"execute": "blockdev-create", "arguments": {"job-id": "job0",
> "options": {"driver": "qcow2", "file": "node0", "size":
> 9223372036854775296}}}
> {"return": {}}
> -Job failed: Could not resize image: Required too big image size, it
> must be not greater than 9223372035781033984
> +Job failed: Could not resize image: offset(9223372036854775296) exceeds
> maximum(9223372035781033984)
> {"execute": "job-dismiss", "arguments": {"id": "job0"}}
> {"return": {}}
>
> Looks like it is just a changed error message, so I can touch up the
> correct patch and then repackage the pull request
Culprit was "block: refactor bdrv_check_request: add errp". I'm
preparing the pull request now.
--
Eric Blake, Principal Software Engineer
Red Hat, Inc. +1-919-301-3226
Virtualization: qemu.org | libvirt.org
On 2/1/21 8:56 PM, Eric Blake wrote:
> I had planned to send a pull request for this series today, but ran into
> a snag. Without this series applied, './check -qcow2' fails 030, 185,
> and 297.
297 appears to be fixed once Kevin's pull request lands (well, that may
be needing a v2). 185 appears to be just a whitespace difference that
missed fixing in 362ef77f9 and similar:
--- /home/eblake/qemu/tests/qemu-iotests/185.out
+++ 185.out.bad
@@ -89,7 +89,7 @@
'format': 'IMGFMT',
'sync': 'full',
'speed': 65536,
- 'x-perf': { 'max-chunk': 65536 } } }
+ 'x-perf': {'max-chunk': 65536} } }
030 is a bit tougher to figure out.
030 fail [09:40:32] [09:40:48] 16.9s (last: 15.4s) failed,
exit status 1
--- /home/eblake/qemu/tests/qemu-iotests/030.out
+++ 030.out.bad
@@ -1,5 +1,45 @@
-...........................
+WARNING:qemu.machine:qemu received signal 11; command:
"/home/eblake/qemu/build/tests/qemu-iotests/../../qemu-system-x86_64
-display none -vga none -chardev
socket,id=mon,path=/tmp/tmpedy9c_uf/qemu-421866-monitor.sock -mon
chardev=mon,mode=control -qtest
unix:path=/tmp/tmpedy9c_uf/qemu-421866-qtest.sock -accel qtest
-nodefaults -display none -accel qtest -drive
if=virtio,id=drive0,file=/home/eblake/qemu/build/tests/qemu-iotests/scratch/img-8.img,format=qcow2,cache=writeback,aio=threads,backing.backing.backing.backing.backing.backing.backing.backing.node-name=node0,backing.backing.backing.backing.backing.backing.backing.node-name=node1,backing.backing.backing.backing.backing.backing.node-name=node2,backing.backing.backing.backing.backing.node-name=node3,backing.backing.backing.backing.node-name=node4,backing.backing.backing.node-name=node5,backing.backing.node-name=node6,backing.node-name=node7,node-name=node8"
+.............EE.............
+======================================================================
+ERROR: test_stream_parallel (__main__.TestParallelOps)
+----------------------------------------------------------------------
+Traceback (most recent call last):
+ File "/home/eblake/qemu/tests/qemu-iotests/030", line 260, in
test_stream_parallel
+ for event in self.vm.get_qmp_events(wait=True):
+ File
"/home/eblake/qemu/tests/qemu-iotests/../../python/qemu/machine.py",
line 585, in get_qmp_events
+ events = self._qmp.get_events(wait=wait)
+ File "/home/eblake/qemu/tests/qemu-iotests/../../python/qemu/qmp.py",
line 328, in get_events
+ self.__get_events(wait)
+ File "/home/eblake/qemu/tests/qemu-iotests/../../python/qemu/qmp.py",
line 197, in __get_events
+ raise QMPConnectError("Error while reading from socket")
+qemu.qmp.QMPConnectError: Error while reading from socket
+
+======================================================================
+ERROR: test_stream_parallel (__main__.TestParallelOps)
+----------------------------------------------------------------------
+Traceback (most recent call last):
+ File
"/home/eblake/qemu/tests/qemu-iotests/../../python/qemu/machine.py",
line 477, in _do_shutdown
+ self._soft_shutdown(timeout, has_quit)
+ File
"/home/eblake/qemu/tests/qemu-iotests/../../python/qemu/machine.py",
line 457, in _soft_shutdown
+ self._qmp.cmd('quit')
+ File "/home/eblake/qemu/tests/qemu-iotests/../../python/qemu/qmp.py",
line 278, in cmd
+ return self.cmd_obj(qmp_cmd)
+ File "/home/eblake/qemu/tests/qemu-iotests/../../python/qemu/qmp.py",
line 256, in cmd_obj
+ self.__sock.sendall(json.dumps(qmp_cmd).encode('utf-8'))
+BrokenPipeError: [Errno 32] Broken pipe
+
+The above exception was the direct cause of the following exception:
+
+Traceback (most recent call last):
+ File "/home/eblake/qemu/tests/qemu-iotests/030", line 227, in tearDown
+ self.vm.shutdown()
+ File
"/home/eblake/qemu/tests/qemu-iotests/../../python/qemu/machine.py",
line 507, in shutdown
+ self._do_shutdown(timeout, has_quit)
+ File
"/home/eblake/qemu/tests/qemu-iotests/../../python/qemu/machine.py",
line 480, in _do_shutdown
+ raise AbnormalShutdown("Could not perform graceful shutdown") \
+qemu.machine.AbnormalShutdown: Could not perform graceful shutdown
+
----------------------------------------------------------------------
Ran 27 tests
-OK
+FAILED (errors=2)
--
Eric Blake, Principal Software Engineer
Red Hat, Inc. +1-919-301-3226
Virtualization: qemu.org | libvirt.org
Am 02.02.2021 um 17:13 hat Eric Blake geschrieben:
> On 2/1/21 8:56 PM, Eric Blake wrote:
>
> > I had planned to send a pull request for this series today, but ran into
> > a snag. Without this series applied, './check -qcow2' fails 030, 185,
> > and 297.
>
> 297 appears to be fixed once Kevin's pull request lands (well, that may
> be needing a v2). 185 appears to be just a whitespace difference that
> missed fixing in 362ef77f9 and similar:
>
> --- /home/eblake/qemu/tests/qemu-iotests/185.out
> +++ 185.out.bad
> @@ -89,7 +89,7 @@
> 'format': 'IMGFMT',
> 'sync': 'full',
> 'speed': 65536,
> - 'x-perf': { 'max-chunk': 65536 } } }
> + 'x-perf': {'max-chunk': 65536} } }
>
> 030 is a bit tougher to figure out.
>
> 030 fail [09:40:32] [09:40:48] 16.9s (last: 15.4s) failed,
> exit status 1
> --- /home/eblake/qemu/tests/qemu-iotests/030.out
> +++ 030.out.bad
> @@ -1,5 +1,45 @@
> -...........................
> +WARNING:qemu.machine:qemu received signal 11; command:
So some qemu process segfaulted. Did you have a look at the resulting
coredump?
Kevin
02.02.2021 19:23, Kevin Wolf wrote:
> Am 02.02.2021 um 17:13 hat Eric Blake geschrieben:
>> On 2/1/21 8:56 PM, Eric Blake wrote:
>>
>>> I had planned to send a pull request for this series today, but ran into
>>> a snag. Without this series applied, './check -qcow2' fails 030, 185,
>>> and 297.
>>
>> 297 appears to be fixed once Kevin's pull request lands (well, that may
>> be needing a v2). 185 appears to be just a whitespace difference that
>> missed fixing in 362ef77f9 and similar:
>>
>> --- /home/eblake/qemu/tests/qemu-iotests/185.out
>> +++ 185.out.bad
>> @@ -89,7 +89,7 @@
>> 'format': 'IMGFMT',
>> 'sync': 'full',
>> 'speed': 65536,
>> - 'x-perf': { 'max-chunk': 65536 } } }
>> + 'x-perf': {'max-chunk': 65536} } }
>>
>> 030 is a bit tougher to figure out.
>>
>> 030 fail [09:40:32] [09:40:48] 16.9s (last: 15.4s) failed,
>> exit status 1
>> --- /home/eblake/qemu/tests/qemu-iotests/030.out
>> +++ 030.out.bad
>> @@ -1,5 +1,45 @@
>> -...........................
>> +WARNING:qemu.machine:qemu received signal 11; command:
>
> So some qemu process segfaulted. Did you have a look at the resulting
> coredump?
>
> Kevin
>
Note that 30 is known to crash sometimes. Look at
"[PATCH RFC 0/5] Fix accidental crash in iotest 30"
https://patchew.org/QEMU/20201120161622.1537-1-vsementsov@virtuozzo.com/
--
Best regards,
Vladimir
02.02.2021 19:29, Vladimir Sementsov-Ogievskiy wrote:
> 02.02.2021 19:23, Kevin Wolf wrote:
>> Am 02.02.2021 um 17:13 hat Eric Blake geschrieben:
>>> On 2/1/21 8:56 PM, Eric Blake wrote:
>>>
>>>> I had planned to send a pull request for this series today, but ran into
>>>> a snag. Without this series applied, './check -qcow2' fails 030, 185,
>>>> and 297.
>>>
>>> 297 appears to be fixed once Kevin's pull request lands (well, that may
>>> be needing a v2). 185 appears to be just a whitespace difference that
>>> missed fixing in 362ef77f9 and similar:
>>>
>>> --- /home/eblake/qemu/tests/qemu-iotests/185.out
>>> +++ 185.out.bad
>>> @@ -89,7 +89,7 @@
>>> 'format': 'IMGFMT',
>>> 'sync': 'full',
>>> 'speed': 65536,
>>> - 'x-perf': { 'max-chunk': 65536 } } }
>>> + 'x-perf': {'max-chunk': 65536} } }
>>>
>>> 030 is a bit tougher to figure out.
>>>
>>> 030 fail [09:40:32] [09:40:48] 16.9s (last: 15.4s) failed,
>>> exit status 1
>>> --- /home/eblake/qemu/tests/qemu-iotests/030.out
>>> +++ 030.out.bad
>>> @@ -1,5 +1,45 @@
>>> -...........................
>>> +WARNING:qemu.machine:qemu received signal 11; command:
>>
>> So some qemu process segfaulted. Did you have a look at the resulting
>> coredump?
>>
>> Kevin
>>
>
> Note that 30 is known to crash sometimes. Look at
>
> "[PATCH RFC 0/5] Fix accidental crash in iotest 30"
>
> https://patchew.org/QEMU/20201120161622.1537-1-vsementsov@virtuozzo.com/
>
Also, I see that this series diverged with master.. Did you already rebased it? If so, could you push it in your tree?
--
Best regards,
Vladimir
On Tue, 2 Feb 2021 at 17:09, Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> wrote: > Note that 30 is known to crash sometimes. Look at > > "[PATCH RFC 0/5] Fix accidental crash in iotest 30" > > https://patchew.org/QEMU/20201120161622.1537-1-vsementsov@virtuozzo.com/ It certainly seems to be the least reliable iotest in my experience. For example it just fell over on ppc64 on MST's latest pullreq merge: https://lore.kernel.org/qemu-devel/CAFEAcA8aZ6qTLjp00FyqYUwtqk0tAFYUpjW0FeepPMMVfOUbPg@mail.gmail.com/ thanks -- PMM
On Tue, 2 Feb 2021 at 22:47, Peter Maydell <peter.maydell@linaro.org> wrote: > > On Tue, 2 Feb 2021 at 17:09, Vladimir Sementsov-Ogievskiy > <vsementsov@virtuozzo.com> wrote: > > Note that 30 is known to crash sometimes. Look at > > > > "[PATCH RFC 0/5] Fix accidental crash in iotest 30" > > > > https://patchew.org/QEMU/20201120161622.1537-1-vsementsov@virtuozzo.com/ > > It certainly seems to be the least reliable iotest in my experience. > For example it just fell over on ppc64 on MST's latest pullreq merge: > > https://lore.kernel.org/qemu-devel/CAFEAcA8aZ6qTLjp00FyqYUwtqk0tAFYUpjW0FeepPMMVfOUbPg@mail.gmail.com/ Just saw this same error again, on FreeBSD rather than OpenBSD this time. Can we please just disable 030 if we can't fix it ? thanks -- PMM
On Wed, 3 Feb 2021 at 10:45, Peter Maydell <peter.maydell@linaro.org> wrote: > > On Tue, 2 Feb 2021 at 22:47, Peter Maydell <peter.maydell@linaro.org> wrote: > > > > On Tue, 2 Feb 2021 at 17:09, Vladimir Sementsov-Ogievskiy > > <vsementsov@virtuozzo.com> wrote: > > > Note that 30 is known to crash sometimes. Look at > > > > > > "[PATCH RFC 0/5] Fix accidental crash in iotest 30" > > > > > > https://patchew.org/QEMU/20201120161622.1537-1-vsementsov@virtuozzo.com/ > > > > It certainly seems to be the least reliable iotest in my experience. > > For example it just fell over on ppc64 on MST's latest pullreq merge: > > > > https://lore.kernel.org/qemu-devel/CAFEAcA8aZ6qTLjp00FyqYUwtqk0tAFYUpjW0FeepPMMVfOUbPg@mail.gmail.com/ > > Just saw this same error again, on FreeBSD rather than OpenBSD > this time. Can we please just disable 030 if we can't fix it ? And another 030 failure, this time in gitlib CI: https://gitlab.com/qemu-project/qemu/-/jobs/1008038881 thanks -- PMM
ping
11.12.2020 21:39, Vladimir Sementsov-Ogievskiy wrote:
> Hi all!
>
> We want 64bit write-zeroes, and for this, convert all io functions to
> 64bit.
>
> We chose signed type, to be consistent with off_t (which is signed) and
> with possibility for signed return type (where negative value means
> error).
>
> Please refer to initial cover-letter
> https://lists.gnu.org/archive/html/qemu-devel/2020-03/msg08723.html
> for more info.
>
> v4: I found, that some more work is needed for block/block-backend, so
> decided to make partI, converting block/io
>
> v4 is based on Kevin's block branch ([PULL 00/34] Block layer patches)
> for BDRV_MAX_LENGTH
>
> changes:
> 01-05: new
> 06: add Alberto's r-b
> 07: new
> 08-16: rebase, add new-style request check, improve commit-msg, drop r-bs
>
> Based-on: <20201211170812.228643-1-kwolf@redhat.com>
>
> Vladimir Sementsov-Ogievskiy (16):
> block: refactor bdrv_check_request: add errp
> util/iov: make qemu_iovec_init_extended() honest
> block: fix theoretical overflow in bdrv_init_padding()
> block/io: refactor bdrv_pad_request(): move bdrv_pad_request() up
> block/io: bdrv_pad_request(): support qemu_iovec_init_extended failure
> block/throttle-groups: throttle_group_co_io_limits_intercept(): 64bit
> bytes
> block/io: improve bdrv_check_request: check qiov too
> block: use int64_t as bytes type in tracked requests
> block/io: use int64_t bytes in driver wrappers
> block/io: support int64_t bytes in bdrv_co_do_pwrite_zeroes()
> block/io: support int64_t bytes in bdrv_aligned_pwritev()
> block/io: support int64_t bytes in bdrv_co_do_copy_on_readv()
> block/io: support int64_t bytes in bdrv_aligned_preadv()
> block/io: support int64_t bytes in bdrv_co_p{read,write}v_part()
> block/io: support int64_t bytes in read/write wrappers
> block/io: use int64_t bytes in copy_range
>
> include/block/block.h | 17 +-
> include/block/block_int.h | 26 +--
> include/block/throttle-groups.h | 2 +-
> include/qemu/iov.h | 2 +-
> block/blkverify.c | 2 +-
> block/file-posix.c | 2 +-
> block/io.c | 274 ++++++++++++++++++++++----------
> block/throttle-groups.c | 5 +-
> tests/test-write-threshold.c | 5 +-
> util/iov.c | 25 ++-
> block/trace-events | 12 +-
> 11 files changed, 252 insertions(+), 120 deletions(-)
>
--
Best regards,
Vladimir
11.12.2020 21:39, Vladimir Sementsov-Ogievskiy wrote: > Hi all! > > We want 64bit write-zeroes, and for this, convert all io functions to > 64bit. > > We chose signed type, to be consistent with off_t (which is signed) and > with possibility for signed return type (where negative value means > error). > > Please refer to initial cover-letter > https://lists.gnu.org/archive/html/qemu-devel/2020-03/msg08723.html > for more info. > > v4: I found, that some more work is needed for block/block-backend, so > decided to make partI, converting block/io > > v4 is based on Kevin's block branch ([PULL 00/34] Block layer patches) > for BDRV_MAX_LENGTH > > changes: > 01-05: new > 06: add Alberto's r-b > 07: new > 08-16: rebase, add new-style request check, improve commit-msg, drop r-bs > > Based-on:<20201211170812.228643-1-kwolf@redhat.com> Now based on master. -- Best regards, Vladimir
© 2016 - 2024 Red Hat, Inc.