iotest failures in head [was: [PATCH v4 00/16] 64bit block-layer: part I]

Posted by Eric Blake 4 years, 8 months ago

On 2/1/21 8:56 PM, Eric Blake wrote:

> I had planned to send a pull request for this series today, but ran into
> a snag.  Without this series applied, './check -qcow2' fails 030, 185,
> and 297.

297 appears to be fixed once Kevin's pull request lands (well, that may
be needing a v2).  185 appears to be just a whitespace difference that
missed fixing in 362ef77f9 and similar:

--- /home/eblake/qemu/tests/qemu-iotests/185.out
+++ 185.out.bad
@@ -89,7 +89,7 @@
                       'format': 'IMGFMT',
                       'sync': 'full',
                       'speed': 65536,
-                      'x-perf': { 'max-chunk': 65536 } } }
+                      'x-perf': {'max-chunk': 65536} } }

030 is a bit tougher to figure out.

030   fail       [09:40:32] [09:40:48]   16.9s  (last: 15.4s) failed,
exit status 1
--- /home/eblake/qemu/tests/qemu-iotests/030.out
+++ 030.out.bad
@@ -1,5 +1,45 @@
-...........................
+WARNING:qemu.machine:qemu received signal 11; command:
"/home/eblake/qemu/build/tests/qemu-iotests/../../qemu-system-x86_64
-display none -vga none -chardev
socket,id=mon,path=/tmp/tmpedy9c_uf/qemu-421866-monitor.sock -mon
chardev=mon,mode=control -qtest
unix:path=/tmp/tmpedy9c_uf/qemu-421866-qtest.sock -accel qtest
-nodefaults -display none -accel qtest -drive
if=virtio,id=drive0,file=/home/eblake/qemu/build/tests/qemu-iotests/scratch/img-8.img,format=qcow2,cache=writeback,aio=threads,backing.backing.backing.backing.backing.backing.backing.backing.node-name=node0,backing.backing.backing.backing.backing.backing.backing.node-name=node1,backing.backing.backing.backing.backing.backing.node-name=node2,backing.backing.backing.backing.backing.node-name=node3,backing.backing.backing.backing.node-name=node4,backing.backing.backing.node-name=node5,backing.backing.node-name=node6,backing.node-name=node7,node-name=node8"
+.............EE.............
+======================================================================
+ERROR: test_stream_parallel (__main__.TestParallelOps)
+----------------------------------------------------------------------
+Traceback (most recent call last):
+  File "/home/eblake/qemu/tests/qemu-iotests/030", line 260, in
test_stream_parallel
+    for event in self.vm.get_qmp_events(wait=True):
+  File
"/home/eblake/qemu/tests/qemu-iotests/../../python/qemu/machine.py",
line 585, in get_qmp_events
+    events = self._qmp.get_events(wait=wait)
+  File "/home/eblake/qemu/tests/qemu-iotests/../../python/qemu/qmp.py",
line 328, in get_events
+    self.__get_events(wait)
+  File "/home/eblake/qemu/tests/qemu-iotests/../../python/qemu/qmp.py",
line 197, in __get_events
+    raise QMPConnectError("Error while reading from socket")
+qemu.qmp.QMPConnectError: Error while reading from socket
+
+======================================================================
+ERROR: test_stream_parallel (__main__.TestParallelOps)
+----------------------------------------------------------------------
+Traceback (most recent call last):
+  File
"/home/eblake/qemu/tests/qemu-iotests/../../python/qemu/machine.py",
line 477, in _do_shutdown
+    self._soft_shutdown(timeout, has_quit)
+  File
"/home/eblake/qemu/tests/qemu-iotests/../../python/qemu/machine.py",
line 457, in _soft_shutdown
+    self._qmp.cmd('quit')
+  File "/home/eblake/qemu/tests/qemu-iotests/../../python/qemu/qmp.py",
line 278, in cmd
+    return self.cmd_obj(qmp_cmd)
+  File "/home/eblake/qemu/tests/qemu-iotests/../../python/qemu/qmp.py",
line 256, in cmd_obj
+    self.__sock.sendall(json.dumps(qmp_cmd).encode('utf-8'))
+BrokenPipeError: [Errno 32] Broken pipe
+
+The above exception was the direct cause of the following exception:
+
+Traceback (most recent call last):
+  File "/home/eblake/qemu/tests/qemu-iotests/030", line 227, in tearDown
+    self.vm.shutdown()
+  File
"/home/eblake/qemu/tests/qemu-iotests/../../python/qemu/machine.py",
line 507, in shutdown
+    self._do_shutdown(timeout, has_quit)
+  File
"/home/eblake/qemu/tests/qemu-iotests/../../python/qemu/machine.py",
line 480, in _do_shutdown
+    raise AbnormalShutdown("Could not perform graceful shutdown") \
+qemu.machine.AbnormalShutdown: Could not perform graceful shutdown
+
 ----------------------------------------------------------------------
 Ran 27 tests

-OK
+FAILED (errors=2)


-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org

Re: iotest failures in head [was: [PATCH v4 00/16] 64bit block-layer: part I]

Posted by Kevin Wolf 4 years, 8 months ago

Am 02.02.2021 um 17:13 hat Eric Blake geschrieben:
> On 2/1/21 8:56 PM, Eric Blake wrote:
> 
> > I had planned to send a pull request for this series today, but ran into
> > a snag.  Without this series applied, './check -qcow2' fails 030, 185,
> > and 297.
> 
> 297 appears to be fixed once Kevin's pull request lands (well, that may
> be needing a v2).  185 appears to be just a whitespace difference that
> missed fixing in 362ef77f9 and similar:
> 
> --- /home/eblake/qemu/tests/qemu-iotests/185.out
> +++ 185.out.bad
> @@ -89,7 +89,7 @@
>                        'format': 'IMGFMT',
>                        'sync': 'full',
>                        'speed': 65536,
> -                      'x-perf': { 'max-chunk': 65536 } } }
> +                      'x-perf': {'max-chunk': 65536} } }
> 
> 030 is a bit tougher to figure out.
> 
> 030   fail       [09:40:32] [09:40:48]   16.9s  (last: 15.4s) failed,
> exit status 1
> --- /home/eblake/qemu/tests/qemu-iotests/030.out
> +++ 030.out.bad
> @@ -1,5 +1,45 @@
> -...........................
> +WARNING:qemu.machine:qemu received signal 11; command:

So some qemu process segfaulted. Did you have a look at the resulting
coredump?

Kevin

Re: iotest failures in head [was: [PATCH v4 00/16] 64bit block-layer: part I]

Posted by Vladimir Sementsov-Ogievskiy 4 years, 8 months ago

02.02.2021 19:23, Kevin Wolf wrote:
> Am 02.02.2021 um 17:13 hat Eric Blake geschrieben:
>> On 2/1/21 8:56 PM, Eric Blake wrote:
>>
>>> I had planned to send a pull request for this series today, but ran into
>>> a snag.  Without this series applied, './check -qcow2' fails 030, 185,
>>> and 297.
>>
>> 297 appears to be fixed once Kevin's pull request lands (well, that may
>> be needing a v2).  185 appears to be just a whitespace difference that
>> missed fixing in 362ef77f9 and similar:
>>
>> --- /home/eblake/qemu/tests/qemu-iotests/185.out
>> +++ 185.out.bad
>> @@ -89,7 +89,7 @@
>>                         'format': 'IMGFMT',
>>                         'sync': 'full',
>>                         'speed': 65536,
>> -                      'x-perf': { 'max-chunk': 65536 } } }
>> +                      'x-perf': {'max-chunk': 65536} } }
>>
>> 030 is a bit tougher to figure out.
>>
>> 030   fail       [09:40:32] [09:40:48]   16.9s  (last: 15.4s) failed,
>> exit status 1
>> --- /home/eblake/qemu/tests/qemu-iotests/030.out
>> +++ 030.out.bad
>> @@ -1,5 +1,45 @@
>> -...........................
>> +WARNING:qemu.machine:qemu received signal 11; command:
> 
> So some qemu process segfaulted. Did you have a look at the resulting
> coredump?
> 
> Kevin
> 

Note that 30 is known to crash sometimes. Look at

"[PATCH RFC 0/5] Fix accidental crash in iotest 30"

https://patchew.org/QEMU/20201120161622.1537-1-vsementsov@virtuozzo.com/

-- 
Best regards,
Vladimir

Re: iotest failures in head [was: [PATCH v4 00/16] 64bit block-layer: part I]

Posted by Vladimir Sementsov-Ogievskiy 4 years, 8 months ago

02.02.2021 19:29, Vladimir Sementsov-Ogievskiy wrote:
> 02.02.2021 19:23, Kevin Wolf wrote:
>> Am 02.02.2021 um 17:13 hat Eric Blake geschrieben:
>>> On 2/1/21 8:56 PM, Eric Blake wrote:
>>>
>>>> I had planned to send a pull request for this series today, but ran into
>>>> a snag.  Without this series applied, './check -qcow2' fails 030, 185,
>>>> and 297.
>>>
>>> 297 appears to be fixed once Kevin's pull request lands (well, that may
>>> be needing a v2).  185 appears to be just a whitespace difference that
>>> missed fixing in 362ef77f9 and similar:
>>>
>>> --- /home/eblake/qemu/tests/qemu-iotests/185.out
>>> +++ 185.out.bad
>>> @@ -89,7 +89,7 @@
>>>                         'format': 'IMGFMT',
>>>                         'sync': 'full',
>>>                         'speed': 65536,
>>> -                      'x-perf': { 'max-chunk': 65536 } } }
>>> +                      'x-perf': {'max-chunk': 65536} } }
>>>
>>> 030 is a bit tougher to figure out.
>>>
>>> 030   fail       [09:40:32] [09:40:48]   16.9s  (last: 15.4s) failed,
>>> exit status 1
>>> --- /home/eblake/qemu/tests/qemu-iotests/030.out
>>> +++ 030.out.bad
>>> @@ -1,5 +1,45 @@
>>> -...........................
>>> +WARNING:qemu.machine:qemu received signal 11; command:
>>
>> So some qemu process segfaulted. Did you have a look at the resulting
>> coredump?
>>
>> Kevin
>>
> 
> Note that 30 is known to crash sometimes. Look at
> 
> "[PATCH RFC 0/5] Fix accidental crash in iotest 30"
> 
> https://patchew.org/QEMU/20201120161622.1537-1-vsementsov@virtuozzo.com/
> 

Also, I see that this series diverged with master.. Did you already rebased it? If so, could you push it in your tree?

-- 
Best regards,
Vladimir

Re: iotest failures in head [was: [PATCH v4 00/16] 64bit block-layer: part I]

Posted by Peter Maydell 4 years, 8 months ago

On Tue, 2 Feb 2021 at 17:09, Vladimir Sementsov-Ogievskiy
<vsementsov@virtuozzo.com> wrote:
> Note that 30 is known to crash sometimes. Look at
>
> "[PATCH RFC 0/5] Fix accidental crash in iotest 30"
>
> https://patchew.org/QEMU/20201120161622.1537-1-vsementsov@virtuozzo.com/

It certainly seems to be the least reliable iotest in my experience.
For example it just fell over on ppc64 on MST's latest pullreq merge:

https://lore.kernel.org/qemu-devel/CAFEAcA8aZ6qTLjp00FyqYUwtqk0tAFYUpjW0FeepPMMVfOUbPg@mail.gmail.com/

thanks
-- PMM

Re: iotest failures in head [was: [PATCH v4 00/16] 64bit block-layer: part I]

Posted by Peter Maydell 4 years, 8 months ago

On Tue, 2 Feb 2021 at 22:47, Peter Maydell <peter.maydell@linaro.org> wrote:
>
> On Tue, 2 Feb 2021 at 17:09, Vladimir Sementsov-Ogievskiy
> <vsementsov@virtuozzo.com> wrote:
> > Note that 30 is known to crash sometimes. Look at
> >
> > "[PATCH RFC 0/5] Fix accidental crash in iotest 30"
> >
> > https://patchew.org/QEMU/20201120161622.1537-1-vsementsov@virtuozzo.com/
>
> It certainly seems to be the least reliable iotest in my experience.
> For example it just fell over on ppc64 on MST's latest pullreq merge:
>
> https://lore.kernel.org/qemu-devel/CAFEAcA8aZ6qTLjp00FyqYUwtqk0tAFYUpjW0FeepPMMVfOUbPg@mail.gmail.com/

Just saw this same error again, on FreeBSD rather than OpenBSD
this time. Can we please just disable 030 if we can't fix it ?

thanks
-- PMM

Re: iotest failures in head [was: [PATCH v4 00/16] 64bit block-layer: part I]

Posted by Peter Maydell 4 years, 8 months ago

On Wed, 3 Feb 2021 at 10:45, Peter Maydell <peter.maydell@linaro.org> wrote:
>
> On Tue, 2 Feb 2021 at 22:47, Peter Maydell <peter.maydell@linaro.org> wrote:
> >
> > On Tue, 2 Feb 2021 at 17:09, Vladimir Sementsov-Ogievskiy
> > <vsementsov@virtuozzo.com> wrote:
> > > Note that 30 is known to crash sometimes. Look at
> > >
> > > "[PATCH RFC 0/5] Fix accidental crash in iotest 30"
> > >
> > > https://patchew.org/QEMU/20201120161622.1537-1-vsementsov@virtuozzo.com/
> >
> > It certainly seems to be the least reliable iotest in my experience.
> > For example it just fell over on ppc64 on MST's latest pullreq merge:
> >
> > https://lore.kernel.org/qemu-devel/CAFEAcA8aZ6qTLjp00FyqYUwtqk0tAFYUpjW0FeepPMMVfOUbPg@mail.gmail.com/
>
> Just saw this same error again, on FreeBSD rather than OpenBSD
> this time. Can we please just disable 030 if we can't fix it ?

And another 030 failure, this time in gitlib CI:
https://gitlab.com/qemu-project/qemu/-/jobs/1008038881

thanks
-- PMM