From: Oleksandr Andrushchenko <oleksandr_andrushchenko@epam.com>

While working with Xen's libxenvchan library I have faced an issue with
unmap notifications sent in wrong order if both UNMAP_NOTIFY_SEND_EVENT
and UNMAP_NOTIFY_CLEAR_BYTE were requested: first we send an event channel
notification and then clear the notification byte which renders in the below
inconsistency (cli_live is the byte which was requested to be cleared on unmap):

[  444.514243] gntdev_put_map UNMAP_NOTIFY_SEND_EVENT map->notify.event 6
libxenvchan_is_open cli_live 1
[  444.515239] __unmap_grant_pages UNMAP_NOTIFY_CLEAR_BYTE at 14

Thus it is not possible to reliably implement the checks like
- wait for the notification (UNMAP_NOTIFY_SEND_EVENT)
- check the variable (UNMAP_NOTIFY_CLEAR_BYTE)
because it is possible that the variable gets checked before it is cleared
by the kernel.

To fix that we need to re-order the notifications, so the variable is first
gets cleared and then the event channel notification is sent.
With this fix I can see the correct order of execution:

[   54.522611] __unmap_grant_pages UNMAP_NOTIFY_CLEAR_BYTE at 14
[   54.537966] gntdev_put_map UNMAP_NOTIFY_SEND_EVENT map->notify.event 6
libxenvchan_is_open cli_live 0

Cc: stable@vger.kernel.org
Signed-off-by: Oleksandr Andrushchenko <oleksandr_andrushchenko@epam.com>
---
 drivers/xen/gntdev.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/drivers/xen/gntdev.c b/drivers/xen/gntdev.c
index fec1b6537166..59ffea800079 100644
--- a/drivers/xen/gntdev.c
+++ b/drivers/xen/gntdev.c
@@ -250,13 +250,13 @@ void gntdev_put_map(struct gntdev_priv *priv, struct gntdev_grant_map *map)
 	if (!refcount_dec_and_test(&map->users))
 		return;
 
+	if (map->pages && !use_ptemod)
+		unmap_grant_pages(map, 0, map->count);
+
 	if (map->notify.flags & UNMAP_NOTIFY_SEND_EVENT) {
 		notify_remote_via_evtchn(map->notify.event);
 		evtchn_put(map->notify.event);
 	}
-
-	if (map->pages && !use_ptemod)
-		unmap_grant_pages(map, 0, map->count);
 	gntdev_free_map(map);
 }
 
-- 
2.25.1

On 12/10/21 4:28 AM, Oleksandr Andrushchenko wrote:
> From: Oleksandr Andrushchenko <oleksandr_andrushchenko@epam.com>
>
> While working with Xen's libxenvchan library I have faced an issue with
> unmap notifications sent in wrong order if both UNMAP_NOTIFY_SEND_EVENT
> and UNMAP_NOTIFY_CLEAR_BYTE were requested: first we send an event channel
> notification and then clear the notification byte which renders in the below
> inconsistency (cli_live is the byte which was requested to be cleared on unmap):
>
> [  444.514243] gntdev_put_map UNMAP_NOTIFY_SEND_EVENT map->notify.event 6
> libxenvchan_is_open cli_live 1
> [  444.515239] __unmap_grant_pages UNMAP_NOTIFY_CLEAR_BYTE at 14
>
> Thus it is not possible to reliably implement the checks like
> - wait for the notification (UNMAP_NOTIFY_SEND_EVENT)
> - check the variable (UNMAP_NOTIFY_CLEAR_BYTE)
> because it is possible that the variable gets checked before it is cleared
> by the kernel.
>
> To fix that we need to re-order the notifications, so the variable is first
> gets cleared and then the event channel notification is sent.
> With this fix I can see the correct order of execution:
>
> [   54.522611] __unmap_grant_pages UNMAP_NOTIFY_CLEAR_BYTE at 14
> [   54.537966] gntdev_put_map UNMAP_NOTIFY_SEND_EVENT map->notify.event 6
> libxenvchan_is_open cli_live 0
>
> Cc: stable@vger.kernel.org
> Signed-off-by: Oleksandr Andrushchenko <oleksandr_andrushchenko@epam.com>



Reviewed-by: Boris Ostrovsky <boris.ostrovsky@oracle.com>

On 12/10/21 4:28 AM, Oleksandr Andrushchenko wrote:
> From: Oleksandr Andrushchenko <oleksandr_andrushchenko@epam.com>
>
> While working with Xen's libxenvchan library I have faced an issue with
> unmap notifications sent in wrong order if both UNMAP_NOTIFY_SEND_EVENT
> and UNMAP_NOTIFY_CLEAR_BYTE were requested: first we send an event channel
> notification and then clear the notification byte which renders in the below
> inconsistency (cli_live is the byte which was requested to be cleared on unmap):
>
> [  444.514243] gntdev_put_map UNMAP_NOTIFY_SEND_EVENT map->notify.event 6
> libxenvchan_is_open cli_live 1
> [  444.515239] __unmap_grant_pages UNMAP_NOTIFY_CLEAR_BYTE at 14
>
> Thus it is not possible to reliably implement the checks like
> - wait for the notification (UNMAP_NOTIFY_SEND_EVENT)
> - check the variable (UNMAP_NOTIFY_CLEAR_BYTE)
> because it is possible that the variable gets checked before it is cleared
> by the kernel.
>
> To fix that we need to re-order the notifications, so the variable is first
> gets cleared and then the event channel notification is sent.
> With this fix I can see the correct order of execution:
>
> [   54.522611] __unmap_grant_pages UNMAP_NOTIFY_CLEAR_BYTE at 14
> [   54.537966] gntdev_put_map UNMAP_NOTIFY_SEND_EVENT map->notify.event 6
> libxenvchan_is_open cli_live 0
>
> Cc: stable@vger.kernel.org
> Signed-off-by: Oleksandr Andrushchenko <oleksandr_andrushchenko@epam.com>



Applied to for-linus-5.16c


-boris