Let's document how we use file locks in file-posix driver, to allow
external programs to "communicate" in this way with Qemu.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
---

Hi all!

We need to access disk images from non-Qemu code and coordinate with
Qemu utilities which may use same image. So, we want to support Qemu
file locking in the external code.

So, here is a patch to document how Qemu locking works, and make this
thing "public".

This is an RFC, because I'm unsure how should we actually document
different operations we have.

For example greaph-mod is a strange thing, I think we should get rid
of it at all.. And at least, no sense in locking corresponding byte in a
raw file.

The other thing is write-unchanged.. What it means when we consider a
raw file opened in several processes? Probably we don't need it too..

 docs/system/qemu-block-drivers.rst.inc | 55 ++++++++++++++++++++++++++
 1 file changed, 55 insertions(+)

diff --git a/docs/system/qemu-block-drivers.rst.inc b/docs/system/qemu-block-drivers.rst.inc
index b052a6d14e..3cd708b3dc 100644
--- a/docs/system/qemu-block-drivers.rst.inc
+++ b/docs/system/qemu-block-drivers.rst.inc
@@ -952,3 +952,58 @@ on host and see if there are locks held by the QEMU process on the image file.
 More than one byte could be locked by the QEMU instance, each byte of which
 reflects a particular permission that is acquired or protected by the running
 block driver.
+
+Image locking protocol
+~~~~~~~~~~~~~~~~~~~~~~
+
+QEMU holds rd locks and never rw locks. Instead, GETLK fcntl is used with F_WRLCK
+to handle permissions as described below.
+QEMU process may rd-lock the following bytes of the image with corresponding
+meaning:
+
+Permission bytes. If permission byte is rd-locked, it means that some process
+uses corresponding permission on that file.
+
+Byte    Operation
+100     read
+          Lock holder can read
+101     write
+          Lock holder can write
+102     write-unchanged
+          Lock holder can write same data
+103     resize
+          Lock holder can resize the file
+104     graph-mod
+          Undefined. QEMU sometimes locks this byte, but external programs
+          should not. QEMU will stop locking this byte in future
+
+Unshare bytes. If permission byte is rd-locked, it means that some process
+does not allow the others use corresponding options on that file.
+
+Byte    Operation
+200     read
+          Lock holder don't allow read operation to other processes.
+201     write
+          Lock holder don't allow write operation to other processes.
+202     write-unchanged
+          Lock holder don't allow write-unchanged operation to other processes.
+203     resize
+          Lock holder don't allow resizing the file by other processes.
+204     graph-mod
+          Undefined. QEMU sometimes locks this byte, but external programs
+          should not. QEMU will stop locking this byte in future
+
+Handling the permissions works as follows: assume we want to open the file to do
+some operations and in the same time want to disallow some operation to other
+processes. So, we want to lock some of the bytes described above. We operate as
+follows:
+
+1. rd-lock all needed bytes, both "permission" bytes and "unshare" bytes.
+
+2. For each "unshare" byte we rd-locked, do GETLK that "tries" to wr-lock
+corresponding "permission" byte. So, we check is there any other process that
+uses the permission we want to unshare. If it exists we fail.
+
+3. For each "permission" byte we rd-locked, do GETLK that "tries" to wr-lock
+corresponding "unshare" byte. So, we check is there any other process that
+unshares the permission we want to have. If it exists we fail.
-- 
2.29.2

Ping.

We need to synchronize access to qcow2 images between Qemu and our internal program. For this solution to be reliable, the locking protocol should be documented at least..

22.03.2021 21:27, Vladimir Sementsov-Ogievskiy wrote:
> Let's document how we use file locks in file-posix driver, to allow
> external programs to "communicate" in this way with Qemu.
> 
> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
> ---
> 
> Hi all!
> 
> We need to access disk images from non-Qemu code and coordinate with
> Qemu utilities which may use same image. So, we want to support Qemu
> file locking in the external code.
> 
> So, here is a patch to document how Qemu locking works, and make this
> thing "public".
> 
> This is an RFC, because I'm unsure how should we actually document
> different operations we have.
> 
> For example greaph-mod is a strange thing, I think we should get rid
> of it at all.. And at least, no sense in locking corresponding byte in a
> raw file.
> 
> The other thing is write-unchanged.. What it means when we consider a
> raw file opened in several processes? Probably we don't need it too..
> 
>   docs/system/qemu-block-drivers.rst.inc | 55 ++++++++++++++++++++++++++
>   1 file changed, 55 insertions(+)
> 
> diff --git a/docs/system/qemu-block-drivers.rst.inc b/docs/system/qemu-block-drivers.rst.inc
> index b052a6d14e..3cd708b3dc 100644
> --- a/docs/system/qemu-block-drivers.rst.inc
> +++ b/docs/system/qemu-block-drivers.rst.inc
> @@ -952,3 +952,58 @@ on host and see if there are locks held by the QEMU process on the image file.
>   More than one byte could be locked by the QEMU instance, each byte of which
>   reflects a particular permission that is acquired or protected by the running
>   block driver.
> +
> +Image locking protocol
> +~~~~~~~~~~~~~~~~~~~~~~
> +
> +QEMU holds rd locks and never rw locks. Instead, GETLK fcntl is used with F_WRLCK
> +to handle permissions as described below.
> +QEMU process may rd-lock the following bytes of the image with corresponding
> +meaning:
> +
> +Permission bytes. If permission byte is rd-locked, it means that some process
> +uses corresponding permission on that file.
> +
> +Byte    Operation
> +100     read
> +          Lock holder can read
> +101     write
> +          Lock holder can write
> +102     write-unchanged
> +          Lock holder can write same data
> +103     resize
> +          Lock holder can resize the file
> +104     graph-mod
> +          Undefined. QEMU sometimes locks this byte, but external programs
> +          should not. QEMU will stop locking this byte in future
> +
> +Unshare bytes. If permission byte is rd-locked, it means that some process
> +does not allow the others use corresponding options on that file.
> +
> +Byte    Operation
> +200     read
> +          Lock holder don't allow read operation to other processes.
> +201     write
> +          Lock holder don't allow write operation to other processes.
> +202     write-unchanged
> +          Lock holder don't allow write-unchanged operation to other processes.
> +203     resize
> +          Lock holder don't allow resizing the file by other processes.
> +204     graph-mod
> +          Undefined. QEMU sometimes locks this byte, but external programs
> +          should not. QEMU will stop locking this byte in future
> +
> +Handling the permissions works as follows: assume we want to open the file to do
> +some operations and in the same time want to disallow some operation to other
> +processes. So, we want to lock some of the bytes described above. We operate as
> +follows:
> +
> +1. rd-lock all needed bytes, both "permission" bytes and "unshare" bytes.
> +
> +2. For each "unshare" byte we rd-locked, do GETLK that "tries" to wr-lock
> +corresponding "permission" byte. So, we check is there any other process that
> +uses the permission we want to unshare. If it exists we fail.
> +
> +3. For each "permission" byte we rd-locked, do GETLK that "tries" to wr-lock
> +corresponding "unshare" byte. So, we check is there any other process that
> +unshares the permission we want to have. If it exists we fail.
> 


-- 
Best regards,
Vladimir

Ping. Any thoughts?

24.05.2021 17:40, Vladimir Sementsov-Ogievskiy wrote:
> Ping.
> 
> We need to synchronize access to qcow2 images between Qemu and our internal program. For this solution to be reliable, the locking protocol should be documented at least..
> 
> 22.03.2021 21:27, Vladimir Sementsov-Ogievskiy wrote:
>> Let's document how we use file locks in file-posix driver, to allow
>> external programs to "communicate" in this way with Qemu.
>>
>> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
>> ---
>>
>> Hi all!
>>
>> We need to access disk images from non-Qemu code and coordinate with
>> Qemu utilities which may use same image. So, we want to support Qemu
>> file locking in the external code.
>>
>> So, here is a patch to document how Qemu locking works, and make this
>> thing "public".
>>
>> This is an RFC, because I'm unsure how should we actually document
>> different operations we have.
>>
>> For example greaph-mod is a strange thing, I think we should get rid
>> of it at all.. And at least, no sense in locking corresponding byte in a
>> raw file.
>>
>> The other thing is write-unchanged.. What it means when we consider a
>> raw file opened in several processes? Probably we don't need it too..
>>
>>   docs/system/qemu-block-drivers.rst.inc | 55 ++++++++++++++++++++++++++
>>   1 file changed, 55 insertions(+)
>>
>> diff --git a/docs/system/qemu-block-drivers.rst.inc b/docs/system/qemu-block-drivers.rst.inc
>> index b052a6d14e..3cd708b3dc 100644
>> --- a/docs/system/qemu-block-drivers.rst.inc
>> +++ b/docs/system/qemu-block-drivers.rst.inc
>> @@ -952,3 +952,58 @@ on host and see if there are locks held by the QEMU process on the image file.
>>   More than one byte could be locked by the QEMU instance, each byte of which
>>   reflects a particular permission that is acquired or protected by the running
>>   block driver.
>> +
>> +Image locking protocol
>> +~~~~~~~~~~~~~~~~~~~~~~
>> +
>> +QEMU holds rd locks and never rw locks. Instead, GETLK fcntl is used with F_WRLCK
>> +to handle permissions as described below.
>> +QEMU process may rd-lock the following bytes of the image with corresponding
>> +meaning:
>> +
>> +Permission bytes. If permission byte is rd-locked, it means that some process
>> +uses corresponding permission on that file.
>> +
>> +Byte    Operation
>> +100     read
>> +          Lock holder can read
>> +101     write
>> +          Lock holder can write
>> +102     write-unchanged
>> +          Lock holder can write same data
>> +103     resize
>> +          Lock holder can resize the file
>> +104     graph-mod
>> +          Undefined. QEMU sometimes locks this byte, but external programs
>> +          should not. QEMU will stop locking this byte in future
>> +
>> +Unshare bytes. If permission byte is rd-locked, it means that some process
>> +does not allow the others use corresponding options on that file.
>> +
>> +Byte    Operation
>> +200     read
>> +          Lock holder don't allow read operation to other processes.
>> +201     write
>> +          Lock holder don't allow write operation to other processes.
>> +202     write-unchanged
>> +          Lock holder don't allow write-unchanged operation to other processes.
>> +203     resize
>> +          Lock holder don't allow resizing the file by other processes.
>> +204     graph-mod
>> +          Undefined. QEMU sometimes locks this byte, but external programs
>> +          should not. QEMU will stop locking this byte in future
>> +
>> +Handling the permissions works as follows: assume we want to open the file to do
>> +some operations and in the same time want to disallow some operation to other
>> +processes. So, we want to lock some of the bytes described above. We operate as
>> +follows:
>> +
>> +1. rd-lock all needed bytes, both "permission" bytes and "unshare" bytes.
>> +
>> +2. For each "unshare" byte we rd-locked, do GETLK that "tries" to wr-lock
>> +corresponding "permission" byte. So, we check is there any other process that
>> +uses the permission we want to unshare. If it exists we fail.
>> +
>> +3. For each "permission" byte we rd-locked, do GETLK that "tries" to wr-lock
>> +corresponding "unshare" byte. So, we check is there any other process that
>> +unshares the permission we want to have. If it exists we fail.
>>
> 
> 


-- 
Best regards,
Vladimir

On Mon, Mar 22, 2021 at 09:27:38PM +0300, Vladimir Sementsov-Ogievskiy wrote:
> Let's document how we use file locks in file-posix driver, to allow
> external programs to "communicate" in this way with Qemu.
> 
> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
> ---
> 
> Hi all!
> 
> We need to access disk images from non-Qemu code and coordinate with
> Qemu utilities which may use same image. So, we want to support Qemu
> file locking in the external code.
> 
> So, here is a patch to document how Qemu locking works, and make this
> thing "public".
> 
> This is an RFC, because I'm unsure how should we actually document
> different operations we have.
> 
> For example greaph-mod is a strange thing, I think we should get rid
> of it at all.. And at least, no sense in locking corresponding byte in a
> raw file.
> 
> The other thing is write-unchanged.. What it means when we consider a
> raw file opened in several processes? Probably we don't need it too..
> 
>  docs/system/qemu-block-drivers.rst.inc | 55 ++++++++++++++++++++++++++
>  1 file changed, 55 insertions(+)
> 
> diff --git a/docs/system/qemu-block-drivers.rst.inc b/docs/system/qemu-block-drivers.rst.inc
> index b052a6d14e..3cd708b3dc 100644
> --- a/docs/system/qemu-block-drivers.rst.inc
> +++ b/docs/system/qemu-block-drivers.rst.inc
> @@ -952,3 +952,58 @@ on host and see if there are locks held by the QEMU process on the image file.
>  More than one byte could be locked by the QEMU instance, each byte of which
>  reflects a particular permission that is acquired or protected by the running
>  block driver.
> +
> +Image locking protocol
> +~~~~~~~~~~~~~~~~~~~~~~
> +
> +QEMU holds rd locks and never rw locks. Instead, GETLK fcntl is used with F_WRLCK
> +to handle permissions as described below.
> +QEMU process may rd-lock the following bytes of the image with corresponding
> +meaning:
> +
> +Permission bytes. If permission byte is rd-locked, it means that some process
> +uses corresponding permission on that file.
> +
> +Byte    Operation
> +100     read
> +          Lock holder can read
> +101     write
> +          Lock holder can write
> +102     write-unchanged
> +          Lock holder can write same data
> +103     resize
> +          Lock holder can resize the file
> +104     graph-mod
> +          Undefined. QEMU sometimes locks this byte, but external programs
> +          should not. QEMU will stop locking this byte in future
> +
> +Unshare bytes. If permission byte is rd-locked, it means that some process
> +does not allow the others use corresponding options on that file.
> +
> +Byte    Operation
> +200     read
> +          Lock holder don't allow read operation to other processes.
> +201     write
> +          Lock holder don't allow write operation to other processes.
> +202     write-unchanged
> +          Lock holder don't allow write-unchanged operation to other processes.
> +203     resize
> +          Lock holder don't allow resizing the file by other processes.
> +204     graph-mod
> +          Undefined. QEMU sometimes locks this byte, but external programs
> +          should not. QEMU will stop locking this byte in future
> +
> +Handling the permissions works as follows: assume we want to open the file to do
> +some operations and in the same time want to disallow some operation to other
> +processes. So, we want to lock some of the bytes described above. We operate as
> +follows:
> +
> +1. rd-lock all needed bytes, both "permission" bytes and "unshare" bytes.
> +
> +2. For each "unshare" byte we rd-locked, do GETLK that "tries" to wr-lock
> +corresponding "permission" byte. So, we check is there any other process that
> +uses the permission we want to unshare. If it exists we fail.
> +
> +3. For each "permission" byte we rd-locked, do GETLK that "tries" to wr-lock
> +corresponding "unshare" byte. So, we check is there any other process that
> +unshares the permission we want to have. If it exists we fail.

I'd perhaps illustrate the few common scenarios - read-only, shared access,
read-write, exclusive and read-write share access, using C-psuedo-code, from
opening image, locking, doing operations, unlocking and closing image.

This would make it explicit that these locks need to be held open
for the duration of the i/o operations.

I'd also probably warn about the dangers of traditional fcntl locks  in
threaded programs, compared to the saner OFD locks.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|