1. Patchew
  2. QEMU
  3. View series

[PATCH for-11.1?] qemu-iotest 161: fix race

Michael Tokarev posted 1 patch 2 months ago
Failed in applying to current master (apply log)
[PATCH for-11.1?] qemu-iotest 161: fix race
Posted by Michael Tokarev 2 months ago 
qemu-iotest 161 fails intermittedly due to a race condition:

qemu:block / io-qcow2-161             ERROR            2.08s   exit status 1
――――――――――――――――――――――――――――――――――― ✀  ―――――――――――――――――――――――――――――――――――――
stderr:
--- tests/qemu-iotests/161.out
+++ scratch/qcow2-file-161/161.out.bad
@@ -34,6 +34,8 @@
 *** Commit and then change an option on the backing file

 Formatting 'TEST_DIR/t.IMGFMT.base', fmt=IMGFMT size=1048576
+qemu-img: TEST_DIR/t.IMGFMT.base: Failed to get "write" lock
+Is another process using the image [TEST_DIR/t.IMGFMT.base]?
 Formatting 'TEST_DIR/t.IMGFMT.int', fmt=IMGFMT size=1048576 backing_file=TEST_DIR/t.IMGFMT.base backing_fmt=IMGFMT
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 backing_file=TEST_DIR/t.IMGFMT.int backing_fmt=IMGFMT
 { 'execute': 'qmp_capabilities' }

This test is run right after a previous test, where we've
killed qemu process which holds the test image open.
It looks like when the next test is started (with creating
the image files), old qemu is still running, so we're racing
with that qemu, and qemu-img is unable to lock the image file
properly.

We can add some `sleep $little-amount` here, but it is simpler
and more reliable to just remove the image files, so that new
ones are created.

Resolves: https://gitlab.com/qemu-project/qemu/-/work_items/2867
Signed-off-by: Michael Tokarev <mjt@tls.msk.ru>

diff --git a/tests/qemu-iotests/161 b/tests/qemu-iotests/161
index f25effab936..0b8ba0b19e2 100755
--- a/tests/qemu-iotests/161
+++ b/tests/qemu-iotests/161
@@ -106,6 +106,7 @@ echo
 echo "*** Commit and then change an option on the backing file"
 echo
 # Create the images again
+rm -f -- "$TEST_IMG.base" "$TEST_IMG.int"
 TEST_IMG="$TEST_IMG.base" _make_test_img $IMG_SIZE
 TEST_IMG="$TEST_IMG.int" _make_test_img -b "$TEST_IMG.base" -F $IMGFMT
 _make_test_img -b "$TEST_IMG.int" -F $IMGFMT
Re: [PATCH for-11.1?] qemu-iotest 161: fix race
Posted by Daniel P. Berrangé 2 months ago 
On Tue, Jul 28, 2026 at 07:39:16PM +0300, Michael Tokarev wrote:
> qemu-iotest 161 fails intermittedly due to a race condition:
> 
> qemu:block / io-qcow2-161             ERROR            2.08s   exit status 1
> ――――――――――――――――――――――――――――――――――― ✀  ―――――――――――――――――――――――――――――――――――――
> stderr:
> --- tests/qemu-iotests/161.out
> +++ scratch/qcow2-file-161/161.out.bad
> @@ -34,6 +34,8 @@
>  *** Commit and then change an option on the backing file
> 
>  Formatting 'TEST_DIR/t.IMGFMT.base', fmt=IMGFMT size=1048576
> +qemu-img: TEST_DIR/t.IMGFMT.base: Failed to get "write" lock
> +Is another process using the image [TEST_DIR/t.IMGFMT.base]?
>  Formatting 'TEST_DIR/t.IMGFMT.int', fmt=IMGFMT size=1048576 backing_file=TEST_DIR/t.IMGFMT.base backing_fmt=IMGFMT
>  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 backing_file=TEST_DIR/t.IMGFMT.int backing_fmt=IMGFMT
>  { 'execute': 'qmp_capabilities' }
> 
> This test is run right after a previous test, where we've
> killed qemu process which holds the test image open.

When you say "previous test" can you clarify the scenario you're
hitting. The tests run with TEST_DIR set to a name that is unique
to the test name + format. eg

   qemu.git/build/scratch/raw-file-101

so...

> It looks like when the next test is started (with creating
> the image files), old qemu is still running, so we're racing
> with that qemu, and qemu-img is unable to lock the image file
> properly.

...iiuc this can only happen if the exact same test + format is
re-run ?

Is that what you're doing ?

If so, rather than deleting the image we should do two
things

 * Make sure tests clean up after themselves if we find one that
   is buggy and leaves stuff running
 
 * Make TEST_DIR more unique by adding the current test hardness
   PID to is, so re-runs can never clash

diff --git a/tests/qemu-iotests/testrunner.py b/tests/qemu-iotests/testrunner.py
index dbe2dddc32..9c7f9d733e 100644
--- a/tests/qemu-iotests/testrunner.py
+++ b/tests/qemu-iotests/testrunner.py
@@ -259,7 +259,7 @@ def do_run_test(self, test: str) -> TestResult:
         for d in ['TEST_DIR', 'SOCK_DIR']:
             env[d] = os.path.join(
                 env[d],
-                f"{self.env.imgfmt}-{self.env.imgproto}-{f_test.name}")
+                f"{self.env.imgfmt}-{self.env.imgproto}-{f_test.name}-{os.getpid()}")
             Path(env[d]).mkdir(parents=True, exist_ok=True)
 
         test_dir = env['TEST_DIR']



With regards,
Daniel
-- 
|: https://berrange.com       ~~        https://hachyderm.io/@berrange :|
|: https://libvirt.org          ~~          https://entangle-photo.org :|
|: https://pixelfed.art/berrange   ~~    https://fstop138.berrange.com :|
Re: [PATCH for-11.1?] qemu-iotest 161: fix race
Posted by Michael Tokarev 2 months ago 
On 7/29/26 14:04, Daniel P. Berrangé wrote:
> On Tue, Jul 28, 2026 at 07:39:16PM +0300, Michael Tokarev wrote:


>> This test is run right after a previous test, where we've
>> killed qemu process which holds the test image open.
> 
> When you say "previous test" can you clarify the scenario you're
> hitting. The tests run with TEST_DIR set to a name that is unique
> to the test name + format. eg
> 
>     qemu.git/build/scratch/raw-file-101

It is the same SUBtest in qemu-iotests/161.

/mjt
Re: [PATCH for-11.1?] qemu-iotest 161: fix race
Posted by Kevin Wolf 2 months ago 
Am 28.07.2026 um 18:39 hat Michael Tokarev geschrieben:
> qemu-iotest 161 fails intermittedly due to a race condition:
> 
> qemu:block / io-qcow2-161             ERROR            2.08s   exit status 1
> ――――――――――――――――――――――――――――――――――― ✀  ―――――――――――――――――――――――――――――――――――――
> stderr:
> --- tests/qemu-iotests/161.out
> +++ scratch/qcow2-file-161/161.out.bad
> @@ -34,6 +34,8 @@
>  *** Commit and then change an option on the backing file
> 
>  Formatting 'TEST_DIR/t.IMGFMT.base', fmt=IMGFMT size=1048576
> +qemu-img: TEST_DIR/t.IMGFMT.base: Failed to get "write" lock
> +Is another process using the image [TEST_DIR/t.IMGFMT.base]?
>  Formatting 'TEST_DIR/t.IMGFMT.int', fmt=IMGFMT size=1048576 backing_file=TEST_DIR/t.IMGFMT.base backing_fmt=IMGFMT
>  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 backing_file=TEST_DIR/t.IMGFMT.int backing_fmt=IMGFMT
>  { 'execute': 'qmp_capabilities' }
> 
> This test is run right after a previous test, where we've
> killed qemu process which holds the test image open.
> It looks like when the next test is started (with creating
> the image files), old qemu is still running, so we're racing
> with that qemu, and qemu-img is unable to lock the image file
> properly.
> 
> We can add some `sleep $little-amount` here, but it is simpler
> and more reliable to just remove the image files, so that new
> ones are created.
> 
> Resolves: https://gitlab.com/qemu-project/qemu/-/work_items/2867
> Signed-off-by: Michael Tokarev <mjt@tls.msk.ru>
> 
> diff --git a/tests/qemu-iotests/161 b/tests/qemu-iotests/161
> index f25effab936..0b8ba0b19e2 100755
> --- a/tests/qemu-iotests/161
> +++ b/tests/qemu-iotests/161
> @@ -106,6 +106,7 @@ echo
>  echo "*** Commit and then change an option on the backing file"
>  echo
>  # Create the images again
> +rm -f -- "$TEST_IMG.base" "$TEST_IMG.int"
>  TEST_IMG="$TEST_IMG.base" _make_test_img $IMG_SIZE
>  TEST_IMG="$TEST_IMG.int" _make_test_img -b "$TEST_IMG.base" -F $IMGFMT
>  _make_test_img -b "$TEST_IMG.int" -F $IMGFMT

This can only work for the file protocol, but the test also supports
fuse, which is now broken.

Kevin


161   fail       [19:51:58] [19:52:03]   5.3s                 output mismatch (see /home/kwolf/source/qemu/build-clang/scratch/qcow2-fuse-161/161.out.bad)
--- /home/kwolf/source/qemu/tests/qemu-iotests/161.out
+++ /home/kwolf/source/qemu/build-clang/scratch/qcow2-fuse-161/161.out.bad
@@ -33,6 +33,8 @@

 *** Commit and then change an option on the backing file

+rm: cannot remove '/tmp/qemu-iotests-9_5xqdbm/qcow2-fuse-161/fuse-t.qcow2.base': Device or resource busy
+rm: cannot remove '/tmp/qemu-iotests-9_5xqdbm/qcow2-fuse-161/fuse-t.qcow2.int': Device or resource busy
 Formatting 'TEST_DIR/t.IMGFMT.base', fmt=IMGFMT size=1048576
 Formatting 'TEST_DIR/t.IMGFMT.int', fmt=IMGFMT size=1048576 backing_file=TEST_DIR/t.IMGFMT.base backing_fmt=IMGFMT
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 backing_file=TEST_DIR/t.IMGFMT.int backing_fmt=IMGFMT
Failures: 161
Failed 1 of 1 iotests
Re: [PATCH for-11.1?] qemu-iotest 161: fix race
Posted by Michael Tokarev 2 months ago 
On 7/28/26 20:53, Kevin Wolf wrote:

>> diff --git a/tests/qemu-iotests/161 b/tests/qemu-iotests/161
>> index f25effab936..0b8ba0b19e2 100755
>> --- a/tests/qemu-iotests/161
>> +++ b/tests/qemu-iotests/161
>> @@ -106,6 +106,7 @@ echo
>>   echo "*** Commit and then change an option on the backing file"
>>   echo
>>   # Create the images again
>> +rm -f -- "$TEST_IMG.base" "$TEST_IMG.int"
>>   TEST_IMG="$TEST_IMG.base" _make_test_img $IMG_SIZE
>>   TEST_IMG="$TEST_IMG.int" _make_test_img -b "$TEST_IMG.base" -F $IMGFMT
>>   _make_test_img -b "$TEST_IMG.int" -F $IMGFMT
> 
> This can only work for the file protocol, but the test also supports
> fuse, which is now broken.

I haven't noticed _rm_test_img function.  I wonder maybe we can add
this rm into _make_test_img directly, and fix this for all tests?

_make_test_img already calls _rm_test_img - for fuse protocol exactly.
It can call _rm_test_img unconditionally, I guess.

I'll give it a try tomorrow.

Thanks,

/mjt
Re: [PATCH for-11.1?] qemu-iotest 161: fix race
Posted by Kevin Wolf 2 months ago 
Am 28.07.2026 um 22:36 hat Michael Tokarev geschrieben:
> On 7/28/26 20:53, Kevin Wolf wrote:
> 
> > > diff --git a/tests/qemu-iotests/161 b/tests/qemu-iotests/161
> > > index f25effab936..0b8ba0b19e2 100755
> > > --- a/tests/qemu-iotests/161
> > > +++ b/tests/qemu-iotests/161
> > > @@ -106,6 +106,7 @@ echo
> > >   echo "*** Commit and then change an option on the backing file"
> > >   echo
> > >   # Create the images again
> > > +rm -f -- "$TEST_IMG.base" "$TEST_IMG.int"
> > >   TEST_IMG="$TEST_IMG.base" _make_test_img $IMG_SIZE
> > >   TEST_IMG="$TEST_IMG.int" _make_test_img -b "$TEST_IMG.base" -F $IMGFMT
> > >   _make_test_img -b "$TEST_IMG.int" -F $IMGFMT
> > 
> > This can only work for the file protocol, but the test also supports
> > fuse, which is now broken.
> 
> I haven't noticed _rm_test_img function.  I wonder maybe we can add
> this rm into _make_test_img directly, and fix this for all tests?
> 
> _make_test_img already calls _rm_test_img - for fuse protocol exactly.
> It can call _rm_test_img unconditionally, I guess.

Using _rm_test_img should do the trick for 161 because it only supports
file and fuse anyway. Those are the obvious two protocols that work with
_rm_test_img.

Some others might work, too, because they just export a local file that
can be deleted normally (nbd, ssh, nfs). _make_test_img() seems to take
care of restarting the NBD server so that it would then serve the new
image. I suppose SSH and NFS don't need to be notified and would
automatically serve the new image. It's not entirely obvious, but it
looks like it could work.

rbd probably breaks, so we might need to exclude that.

Kevin
Re: [PATCH for-11.1?] qemu-iotest 161: fix race
Posted by Michael Tokarev 2 months ago 
On 7/29/26 12:17, Kevin Wolf wrote:
..
> Using _rm_test_img should do the trick for 161 because it only supports
> file and fuse anyway. Those are the obvious two protocols that work with
> _rm_test_img.

actually it wont.  Because among others, _rm_test_img run `qemu-img info`
on the image in question, and this is not working either while it's in use:

  $ qemu-img info test.qcow2
  qemu-img: Could not open 'test.qcow2': Failed to get shared "write" lock
  Is another process using the image [test.qcow2]?

this is exactly the problem we're trying to avoid :)

/mjt
Re: [PATCH for-11.1?] qemu-iotest 161: fix race
Posted by Stefan Hajnoczi 2 months ago 
On Tue, Jul 28, 2026 at 12:40 PM Michael Tokarev <mjt@tls.msk.ru> wrote:
>
> qemu-iotest 161 fails intermittedly due to a race condition:
>
> qemu:block / io-qcow2-161             ERROR            2.08s   exit status 1
> ――――――――――――――――――――――――――――――――――― ✀  ―――――――――――――――――――――――――――――――――――――
> stderr:
> --- tests/qemu-iotests/161.out
> +++ scratch/qcow2-file-161/161.out.bad
> @@ -34,6 +34,8 @@
>  *** Commit and then change an option on the backing file
>
>  Formatting 'TEST_DIR/t.IMGFMT.base', fmt=IMGFMT size=1048576
> +qemu-img: TEST_DIR/t.IMGFMT.base: Failed to get "write" lock
> +Is another process using the image [TEST_DIR/t.IMGFMT.base]?
>  Formatting 'TEST_DIR/t.IMGFMT.int', fmt=IMGFMT size=1048576 backing_file=TEST_DIR/t.IMGFMT.base backing_fmt=IMGFMT
>  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 backing_file=TEST_DIR/t.IMGFMT.int backing_fmt=IMGFMT
>  { 'execute': 'qmp_capabilities' }
>
> This test is run right after a previous test, where we've
> killed qemu process which holds the test image open.
> It looks like when the next test is started (with creating
> the image files), old qemu is still running, so we're racing
> with that qemu, and qemu-img is unable to lock the image file
> properly.
>
> We can add some `sleep $little-amount` here, but it is simpler
> and more reliable to just remove the image files, so that new
> ones are created.
>
> Resolves: https://gitlab.com/qemu-project/qemu/-/work_items/2867
> Signed-off-by: Michael Tokarev <mjt@tls.msk.ru>

Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>

Patchew 2.1-devel-ea86937

© 2016 - 2026 Red Hat, Inc.