virtio devices support separate iothreads waiting for
events from file descriptors. These are asynchronous
events that can't be recorded and replayed, therefore
this patch disables ioeventfd for all devices when
record or replay is enabled.

Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgalyuk@ispras.ru>
---
 hw/virtio/virtio-pci.c |    6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c
index 094c36aa3e..76fbc111ea 100644
--- a/hw/virtio/virtio-pci.c
+++ b/hw/virtio/virtio-pci.c
@@ -37,6 +37,7 @@
 #include "qemu/range.h"
 #include "hw/virtio/virtio-bus.h"
 #include "qapi/visitor.h"
+#include "sysemu/replay.h"
 
 #define VIRTIO_PCI_REGION_SIZE(dev)     VIRTIO_PCI_CONFIG_OFF(msix_present(dev))
 
@@ -1746,6 +1747,11 @@ static void virtio_pci_realize(PCIDevice *pci_dev, Error **errp)
         proxy->flags &= ~VIRTIO_PCI_FLAG_USE_IOEVENTFD;
     }
 
+    /* fd-based ioevents can't be synchronized in record/replay */
+    if (replay_mode != REPLAY_MODE_NONE) {
+        proxy->flags &= ~VIRTIO_PCI_FLAG_USE_IOEVENTFD;
+    }
+
     /*
      * virtio pci bar layout used by default.
      * subclasses can re-arrange things if needed.

Pavel Dovgalyuk <pavel.dovgalyuk@ispras.ru> writes:

> virtio devices support separate iothreads waiting for
> events from file descriptors. These are asynchronous
> events that can't be recorded and replayed, therefore
> this patch disables ioeventfd for all devices when
> record or replay is enabled.

Does it always have to be the case? Can a virtio device forgo iothreads
and poll events from the main loop?

>
> Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgalyuk@ispras.ru>
> ---
>  hw/virtio/virtio-pci.c |    6 ++++++
>  1 file changed, 6 insertions(+)
>
> diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c
> index 094c36aa3e..76fbc111ea 100644
> --- a/hw/virtio/virtio-pci.c
> +++ b/hw/virtio/virtio-pci.c
> @@ -37,6 +37,7 @@
>  #include "qemu/range.h"
>  #include "hw/virtio/virtio-bus.h"
>  #include "qapi/visitor.h"
> +#include "sysemu/replay.h"
>  
>  #define VIRTIO_PCI_REGION_SIZE(dev)     VIRTIO_PCI_CONFIG_OFF(msix_present(dev))
>  
> @@ -1746,6 +1747,11 @@ static void virtio_pci_realize(PCIDevice *pci_dev, Error **errp)
>          proxy->flags &= ~VIRTIO_PCI_FLAG_USE_IOEVENTFD;
>      }
>  
> +    /* fd-based ioevents can't be synchronized in record/replay */
> +    if (replay_mode != REPLAY_MODE_NONE) {
> +        proxy->flags &= ~VIRTIO_PCI_FLAG_USE_IOEVENTFD;
> +    }
> +
>      /*
>       * virtio pci bar layout used by default.
>       * subclasses can re-arrange things if needed.

I think we need a better solution for this. Are we going to end up
sprinkling replay annotations throughout the device backends for each an
every device supported by QEMU?

This seems to be it should be the other way around. We should identify
the devices (or classes of devices) that support the logging and
consistency requirements needed for replay and mark them with a QOM
parameter. Then we can walk the tree of devices and either claim
compatibility with the request to record/replay or report to the user
"can't use record/replay because device %s is not supported".

-- 
Alex Bennée

On 11/02/21 09:58, Alex Bennée wrote:
>> virtio devices support separate iothreads waiting for
>> events from file descriptors. These are asynchronous
>> events that can't be recorded and replayed, therefore
>> this patch disables ioeventfd for all devices when
>> record or replay is enabled.
> Does it always have to be the case? Can a virtio device forgo iothreads
> and poll events from the main loop?

Yes and in fact it's the default, but ioeventfd has the same issue I 
think even if the polling happens from the main thread.

Paolo

On Thu, 11 Feb 2021 11:34:53 +0300
Pavel Dovgalyuk <pavel.dovgalyuk@ispras.ru> wrote:

> virtio devices support separate iothreads waiting for
> events from file descriptors. These are asynchronous
> events that can't be recorded and replayed, therefore
> this patch disables ioeventfd for all devices when
> record or replay is enabled.
> 
> Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgalyuk@ispras.ru>
> ---
>  hw/virtio/virtio-pci.c |    6 ++++++
>  1 file changed, 6 insertions(+)
> 
> diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c
> index 094c36aa3e..76fbc111ea 100644
> --- a/hw/virtio/virtio-pci.c
> +++ b/hw/virtio/virtio-pci.c
> @@ -37,6 +37,7 @@
>  #include "qemu/range.h"
>  #include "hw/virtio/virtio-bus.h"
>  #include "qapi/visitor.h"
> +#include "sysemu/replay.h"
>  
>  #define VIRTIO_PCI_REGION_SIZE(dev)     VIRTIO_PCI_CONFIG_OFF(msix_present(dev))
>  
> @@ -1746,6 +1747,11 @@ static void virtio_pci_realize(PCIDevice *pci_dev, Error **errp)
>          proxy->flags &= ~VIRTIO_PCI_FLAG_USE_IOEVENTFD;
>      }
>  
> +    /* fd-based ioevents can't be synchronized in record/replay */
> +    if (replay_mode != REPLAY_MODE_NONE) {
> +        proxy->flags &= ~VIRTIO_PCI_FLAG_USE_IOEVENTFD;
> +    }
> +
>      /*
>       * virtio pci bar layout used by default.
>       * subclasses can re-arrange things if needed.
> 
> 

I think you would need to do the same for any device using ioeventfd,
no?

On 11.02.2021 11:44, Cornelia Huck wrote:
> On Thu, 11 Feb 2021 11:34:53 +0300
> Pavel Dovgalyuk <pavel.dovgalyuk@ispras.ru> wrote:
> 
>> virtio devices support separate iothreads waiting for
>> events from file descriptors. These are asynchronous
>> events that can't be recorded and replayed, therefore
>> this patch disables ioeventfd for all devices when
>> record or replay is enabled.
>>
>> Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgalyuk@ispras.ru>
>> ---
>>   hw/virtio/virtio-pci.c |    6 ++++++
>>   1 file changed, 6 insertions(+)
>>
>> diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c
>> index 094c36aa3e..76fbc111ea 100644
>> --- a/hw/virtio/virtio-pci.c
>> +++ b/hw/virtio/virtio-pci.c
>> @@ -37,6 +37,7 @@
>>   #include "qemu/range.h"
>>   #include "hw/virtio/virtio-bus.h"
>>   #include "qapi/visitor.h"
>> +#include "sysemu/replay.h"
>>   
>>   #define VIRTIO_PCI_REGION_SIZE(dev)     VIRTIO_PCI_CONFIG_OFF(msix_present(dev))
>>   
>> @@ -1746,6 +1747,11 @@ static void virtio_pci_realize(PCIDevice *pci_dev, Error **errp)
>>           proxy->flags &= ~VIRTIO_PCI_FLAG_USE_IOEVENTFD;
>>       }
>>   
>> +    /* fd-based ioevents can't be synchronized in record/replay */
>> +    if (replay_mode != REPLAY_MODE_NONE) {
>> +        proxy->flags &= ~VIRTIO_PCI_FLAG_USE_IOEVENTFD;
>> +    }
>> +
>>       /*
>>        * virtio pci bar layout used by default.
>>        * subclasses can re-arrange things if needed.
>>
>>
> 
> I think you would need to do the same for any device using ioeventfd,
> no?

That's right.
It seems, that ioeventfd is used in virtio-ccw, virtio-mmio, and 
virtio-pci. The second one is related to KVM only, right?

And the first one should be updated too for replay.

On 11/02/21 09:56, Pavel Dovgalyuk wrote:
> 
> That's right.
> It seems, that ioeventfd is used in virtio-ccw, virtio-mmio, and 
> virtio-pci. The second one is related to KVM only, right?

No, it's used for TCG as well (with ARM "-M virt" or x86 "-M microvm").

Paolo

> And the first one should be updated too for replay.

On 11.02.2021 12:05, Paolo Bonzini wrote:
> On 11/02/21 09:56, Pavel Dovgalyuk wrote:
>>
>> That's right.
>> It seems, that ioeventfd is used in virtio-ccw, virtio-mmio, and 
>> virtio-pci. The second one is related to KVM only, right?
> 
> No, it's used for TCG as well (with ARM "-M virt" or x86 "-M microvm").

Thanks for pointing at these machines.
But virtio-mmio does not support ioeventfd disabling as in PCI and CCW 
devices.
Should this option be added first?
Or the following code means that ioeventfd is used only in KVM mode?

static bool virtio_mmio_ioeventfd_enabled(DeviceState *d)
{
     return kvm_eventfds_enabled();
}

Pavel Dovgalyuk

On 24/02/21 10:56, Pavel Dovgalyuk wrote:
>>
> 
> Thanks for pointing at these machines.
> But virtio-mmio does not support ioeventfd disabling as in PCI and CCW 
> devices.
> Should this option be added first?

Yes, that was a good idea.

Paolo