[v1] qemu-img: Support bitmap --merge into backing image

[PATCH] qemu-img: Support bitmap --merge into backing image

Posted by Eric Blake 3 years, 7 months ago

If you have the chain 'base.qcow2 <- top.qcow2' and want to merge a
bitmap from top into base, qemu-img was failing with:

qemu-img: Could not open 'top.qcow2': Could not open backing file: Failed to get shared "write" lock
Is another process using the image [base.qcow2]?

The easiest fix is to not open the entire backing chain of the source
image, so that we aren't worrying about competing BDS visiting the
backing image as both a read-only backing of the source and the
writeable destination.

Fixes: http://bugzilla.redhat.com/1877209
Reported-by: Eyal Shenitzky <eshenitz@redhat.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
---
 qemu-img.c                 |  3 +-
 tests/qemu-iotests/291     | 12 ++++++++
 tests/qemu-iotests/291.out | 56 ++++++++++++++++++++++++++++++++++++++
 3 files changed, 70 insertions(+), 1 deletion(-)

diff --git a/qemu-img.c b/qemu-img.c
index eb2fc1f86243..b15098a2f9b3 100644
--- a/qemu-img.c
+++ b/qemu-img.c
@@ -4755,7 +4755,8 @@ static int img_bitmap(int argc, char **argv)
     }
     bs = blk_bs(blk);
     if (src_filename) {
-        src = img_open(false, src_filename, src_fmt, 0, false, false, false);
+        src = img_open(false, src_filename, src_fmt, BDRV_O_NO_BACKING,
+                       false, false, false);
         if (!src) {
             goto out;
         }
diff --git a/tests/qemu-iotests/291 b/tests/qemu-iotests/291
index 1e0bb76959bb..4f837b205655 100755
--- a/tests/qemu-iotests/291
+++ b/tests/qemu-iotests/291
@@ -91,6 +91,15 @@ $QEMU_IMG bitmap --remove --image-opts \
     driver=$IMGFMT,file.driver=file,file.filename="$TEST_IMG" tmp
 _img_info --format-specific

+echo
+echo "=== Merge from top layer into backing image ==="
+echo
+
+$QEMU_IMG rebase -u -F qcow2 -b "$TEST_IMG.base" "$TEST_IMG"
+$QEMU_IMG bitmap --add --merge b2 -b "$TEST_IMG" -F $IMGFMT \
+     -f $IMGFMT "$TEST_IMG.base" b3
+_img_info --format-specific --backing-chain
+
 echo
 echo "=== Check bitmap contents ==="
 echo
@@ -107,6 +116,9 @@ $QEMU_IMG map --output=json --image-opts \
 nbd_server_start_unix_socket -r -f qcow2 -B b2 "$TEST_IMG"
 $QEMU_IMG map --output=json --image-opts \
     "$IMG,x-dirty-bitmap=qemu:dirty-bitmap:b2" | _filter_qemu_img_map
+nbd_server_start_unix_socket -r -f qcow2 -B b3 "$TEST_IMG"
+$QEMU_IMG map --output=json --image-opts \
+    "$IMG,x-dirty-bitmap=qemu:dirty-bitmap:b3" | _filter_qemu_img_map

 # success, all done
 echo '*** done'
diff --git a/tests/qemu-iotests/291.out b/tests/qemu-iotests/291.out
index ee89a728855f..3990f7aacc7b 100644
--- a/tests/qemu-iotests/291.out
+++ b/tests/qemu-iotests/291.out
@@ -68,6 +68,59 @@ Format specific information:
     corrupt: false
     extended l2: false

+=== Merge from top layer into backing image ===
+
+image: TEST_DIR/t.IMGFMT
+file format: IMGFMT
+virtual size: 10 MiB (10485760 bytes)
+cluster_size: 65536
+backing file: TEST_DIR/t.IMGFMT.base
+backing file format: IMGFMT
+Format specific information:
+    compat: 1.1
+    compression type: zlib
+    lazy refcounts: false
+    bitmaps:
+        [0]:
+            flags:
+            name: b1
+            granularity: 524288
+        [1]:
+            flags:
+                [0]: auto
+            name: b2
+            granularity: 65536
+        [2]:
+            flags:
+            name: b0
+            granularity: 65536
+    refcount bits: 16
+    corrupt: false
+    extended l2: false
+
+image: TEST_DIR/t.IMGFMT.base
+file format: IMGFMT
+virtual size: 10 MiB (10485760 bytes)
+cluster_size: 65536
+Format specific information:
+    compat: 1.1
+    compression type: zlib
+    lazy refcounts: false
+    bitmaps:
+        [0]:
+            flags:
+                [0]: auto
+            name: b0
+            granularity: 65536
+        [1]:
+            flags:
+                [0]: auto
+            name: b3
+            granularity: 65536
+    refcount bits: 16
+    corrupt: false
+    extended l2: false
+
 === Check bitmap contents ===

 [{ "start": 0, "length": 3145728, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
@@ -79,4 +132,7 @@ Format specific information:
 [{ "start": 0, "length": 2097152, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
 { "start": 2097152, "length": 1048576, "depth": 0, "zero": false, "data": false},
 { "start": 3145728, "length": 7340032, "depth": 0, "zero": false, "data": true, "offset": OFFSET}]
+[{ "start": 0, "length": 2097152, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
+{ "start": 2097152, "length": 1048576, "depth": 0, "zero": false, "data": false},
+{ "start": 3145728, "length": 7340032, "depth": 0, "zero": false, "data": true, "offset": OFFSET}]
 *** done
-- 
2.28.0

Re: [PATCH] qemu-img: Support bitmap --merge into backing image

Posted by Max Reitz 3 years, 7 months ago

On 09.09.20 14:33, Eric Blake wrote:
> If you have the chain 'base.qcow2 <- top.qcow2' and want to merge a
> bitmap from top into base, qemu-img was failing with:
> 
> qemu-img: Could not open 'top.qcow2': Could not open backing file: Failed to get shared "write" lock
> Is another process using the image [base.qcow2]?
> 
> The easiest fix is to not open the entire backing chain of the source
> image, so that we aren't worrying about competing BDS visiting the
> backing image as both a read-only backing of the source and the
> writeable destination.
> 
> Fixes: http://bugzilla.redhat.com/1877209
> Reported-by: Eyal Shenitzky <eshenitz@redhat.com>
> Signed-off-by: Eric Blake <eblake@redhat.com>
> ---
>  qemu-img.c                 |  3 +-
>  tests/qemu-iotests/291     | 12 ++++++++
>  tests/qemu-iotests/291.out | 56 ++++++++++++++++++++++++++++++++++++++
>  3 files changed, 70 insertions(+), 1 deletion(-)
> 
> diff --git a/qemu-img.c b/qemu-img.c
> index eb2fc1f86243..b15098a2f9b3 100644
> --- a/qemu-img.c
> +++ b/qemu-img.c
> @@ -4755,7 +4755,8 @@ static int img_bitmap(int argc, char **argv)
>      }
>      bs = blk_bs(blk);
>      if (src_filename) {
> -        src = img_open(false, src_filename, src_fmt, 0, false, false, false);
> +        src = img_open(false, src_filename, src_fmt, BDRV_O_NO_BACKING,
> +                       false, false, false);

Why not do the same for the destination BB?

>          if (!src) {
>              goto out;
>          }
> diff --git a/tests/qemu-iotests/291 b/tests/qemu-iotests/291
> index 1e0bb76959bb..4f837b205655 100755
> --- a/tests/qemu-iotests/291
> +++ b/tests/qemu-iotests/291

[...]

> @@ -107,6 +116,9 @@ $QEMU_IMG map --output=json --image-opts \
>  nbd_server_start_unix_socket -r -f qcow2 -B b2 "$TEST_IMG"
>  $QEMU_IMG map --output=json --image-opts \
>      "$IMG,x-dirty-bitmap=qemu:dirty-bitmap:b2" | _filter_qemu_img_map
> +nbd_server_start_unix_socket -r -f qcow2 -B b3 "$TEST_IMG"

Why not look into $TEST_IMG.base to see specifically whether the bitmap
is there?

(I also am quite surprised that it’s even possible to export bitmaps
from backing nodes, but, well.)

Max

Re: [PATCH] qemu-img: Support bitmap --merge into backing image

Posted by Eric Blake 3 years, 7 months ago

On 9/11/20 3:31 AM, Max Reitz wrote:
> On 09.09.20 14:33, Eric Blake wrote:
>> If you have the chain 'base.qcow2 <- top.qcow2' and want to merge a
>> bitmap from top into base, qemu-img was failing with:
>>
>> qemu-img: Could not open 'top.qcow2': Could not open backing file: Failed to get shared "write" lock
>> Is another process using the image [base.qcow2]?
>>
>> The easiest fix is to not open the entire backing chain of the source
>> image, so that we aren't worrying about competing BDS visiting the
>> backing image as both a read-only backing of the source and the
>> writeable destination.
>>
>> Fixes: http://bugzilla.redhat.com/1877209
>> Reported-by: Eyal Shenitzky <eshenitz@redhat.com>
>> Signed-off-by: Eric Blake <eblake@redhat.com>
>> ---
>>   qemu-img.c                 |  3 +-
>>   tests/qemu-iotests/291     | 12 ++++++++
>>   tests/qemu-iotests/291.out | 56 ++++++++++++++++++++++++++++++++++++++
>>   3 files changed, 70 insertions(+), 1 deletion(-)
>>
>> diff --git a/qemu-img.c b/qemu-img.c
>> index eb2fc1f86243..b15098a2f9b3 100644
>> --- a/qemu-img.c
>> +++ b/qemu-img.c
>> @@ -4755,7 +4755,8 @@ static int img_bitmap(int argc, char **argv)
>>       }
>>       bs = blk_bs(blk);
>>       if (src_filename) {
>> -        src = img_open(false, src_filename, src_fmt, 0, false, false, false);
>> +        src = img_open(false, src_filename, src_fmt, BDRV_O_NO_BACKING,
>> +                       false, false, false);
> 
> Why not do the same for the destination BB?

Yeah, that should work, too.

> 
>>           if (!src) {
>>               goto out;
>>           }
>> diff --git a/tests/qemu-iotests/291 b/tests/qemu-iotests/291
>> index 1e0bb76959bb..4f837b205655 100755
>> --- a/tests/qemu-iotests/291
>> +++ b/tests/qemu-iotests/291
> 
> [...]
> 
>> @@ -107,6 +116,9 @@ $QEMU_IMG map --output=json --image-opts \
>>   nbd_server_start_unix_socket -r -f qcow2 -B b2 "$TEST_IMG"
>>   $QEMU_IMG map --output=json --image-opts \
>>       "$IMG,x-dirty-bitmap=qemu:dirty-bitmap:b2" | _filter_qemu_img_map
>> +nbd_server_start_unix_socket -r -f qcow2 -B b3 "$TEST_IMG"
> 
> Why not look into $TEST_IMG.base to see specifically whether the bitmap
> is there?

We did just that, several lines earlier, with the qemu-img info 
--backing-chain.

> 
> (I also am quite surprised that it’s even possible to export bitmaps
> from backing nodes, but, well.)

I actually ought to call that out in the commit message.  It used to be 
that we were inconsistent on what we could see from the backing chain (a 
filter would make it so we can't), but as soon as your filter patches 
land, then we _do_ want to always be able to find a bitmap from the 
backing chain (incremental backup depends on that: we create an overlay 
disk to run the block-copy job as a filter, and _want_ to expose that 
overlay image with the bitmap it inherits from the original image).  So 
being able to export bitmaps from a backing node is normally a feature; 
and it is only in 'qemu-img bitmap' where we don't want accidental 
inheritance to get in the way from what we are actually merging.

I'll send a v2.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org

Re: [PATCH] qemu-img: Support bitmap --merge into backing image

Posted by Max Reitz 3 years, 7 months ago

On 11.09.20 17:17, Eric Blake wrote:
> On 9/11/20 3:31 AM, Max Reitz wrote:
>> On 09.09.20 14:33, Eric Blake wrote:
>>> If you have the chain 'base.qcow2 <- top.qcow2' and want to merge a
>>> bitmap from top into base, qemu-img was failing with:
>>>
>>> qemu-img: Could not open 'top.qcow2': Could not open backing file:
>>> Failed to get shared "write" lock
>>> Is another process using the image [base.qcow2]?
>>>
>>> The easiest fix is to not open the entire backing chain of the source
>>> image, so that we aren't worrying about competing BDS visiting the
>>> backing image as both a read-only backing of the source and the
>>> writeable destination.
>>>
>>> Fixes: http://bugzilla.redhat.com/1877209
>>> Reported-by: Eyal Shenitzky <eshenitz@redhat.com>
>>> Signed-off-by: Eric Blake <eblake@redhat.com>
>>> ---
>>>   qemu-img.c                 |  3 +-
>>>   tests/qemu-iotests/291     | 12 ++++++++
>>>   tests/qemu-iotests/291.out | 56 ++++++++++++++++++++++++++++++++++++++
>>>   3 files changed, 70 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/qemu-img.c b/qemu-img.c
>>> index eb2fc1f86243..b15098a2f9b3 100644
>>> --- a/qemu-img.c
>>> +++ b/qemu-img.c
>>> @@ -4755,7 +4755,8 @@ static int img_bitmap(int argc, char **argv)
>>>       }
>>>       bs = blk_bs(blk);
>>>       if (src_filename) {
>>> -        src = img_open(false, src_filename, src_fmt, 0, false,
>>> false, false);
>>> +        src = img_open(false, src_filename, src_fmt, BDRV_O_NO_BACKING,
>>> +                       false, false, false);
>>
>> Why not do the same for the destination BB?
> 
> Yeah, that should work, too.
> 
>>
>>>           if (!src) {
>>>               goto out;
>>>           }
>>> diff --git a/tests/qemu-iotests/291 b/tests/qemu-iotests/291
>>> index 1e0bb76959bb..4f837b205655 100755
>>> --- a/tests/qemu-iotests/291
>>> +++ b/tests/qemu-iotests/291
>>
>> [...]
>>
>>> @@ -107,6 +116,9 @@ $QEMU_IMG map --output=json --image-opts \
>>>   nbd_server_start_unix_socket -r -f qcow2 -B b2 "$TEST_IMG"
>>>   $QEMU_IMG map --output=json --image-opts \
>>>       "$IMG,x-dirty-bitmap=qemu:dirty-bitmap:b2" | _filter_qemu_img_map
>>> +nbd_server_start_unix_socket -r -f qcow2 -B b3 "$TEST_IMG"
>>
>> Why not look into $TEST_IMG.base to see specifically whether the bitmap
>> is there?
> 
> We did just that, several lines earlier, with the qemu-img info
> --backing-chain.

OK, perfect.

>>
>> (I also am quite surprised that it’s even possible to export bitmaps
>> from backing nodes, but, well.)
> 
> I actually ought to call that out in the commit message.  It used to be
> that we were inconsistent on what we could see from the backing chain (a
> filter would make it so we can't), but as soon as your filter patches
> land, then we _do_ want to always be able to find a bitmap from the
> backing chain (incremental backup depends on that: we create an overlay
> disk to run the block-copy job as a filter, and _want_ to expose that
> overlay image with the bitmap it inherits from the original image).  So
> being able to export bitmaps from a backing node is normally a feature;
> and it is only in 'qemu-img bitmap' where we don't want accidental
> inheritance to get in the way from what we are actually merging.

Understood.

Max