Sometimes devices have different features depending of things outside
of qemu.  For instance the kernel.  Document how to handle that cases.

Signed-off-by: Juan Quintela <quintela@redhat.com>
Acked-by: Peter Xu <peterx@redhat.com>

---

If you have some example to put here, I am all ears.  I guess that
virtio-* with some features that are on qemu but not on all kernel
would do the trick, but I am not a virtio guru myself.  Patches
welcome.
---
 docs/devel/migration.rst | 96 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 96 insertions(+)

diff --git a/docs/devel/migration.rst b/docs/devel/migration.rst
index 5ef2b36e9e..e671df729e 100644
--- a/docs/devel/migration.rst
+++ b/docs/devel/migration.rst
@@ -358,6 +358,102 @@ machine types to have the right value: ::
          ...
      };
 
+A device with diferent features on both sides
+---------------------------------------------
+
+Let's assume that we are using the same QEMU binary on both sides,
+just to make the things easier.  But we have a device that has
+different features on both sides of the migration.  That can be
+because the devices are different, because the kernel driver of both
+devices have different features, whatever.
+
+How can we get this to work with migration.  The way to do that is
+"theoretically" easy.  You have to get the features that the device
+has in the source of the migration.  The features that the device has
+on the target of the migration, you get the intersection of the
+features of both sides, and that is the way that you should launch
+QEMU.
+
+Notice that this is not completely related to QEMU.  The most
+important thing here is that this should be handle by the managing
+application that launches QEMU.  If QEMU is configured correctly, the
+migration will suceeed.
+
+Once that we have defined that, doing this is complicated.  Almost all
+devices are bad at being able to be launched with only some features
+enabled.  With one big exception: cpus.
+
+You can read the documentation for QEMU x86 cpu models here:
+
+https://qemu-project.gitlab.io/qemu/system/qemu-cpu-models.html
+
+See when they talk about migration they recommend that one chooses the
+newest cpu model that is supported for all cpus.
+
+Let's say that we have:
+
+Host A:
+
+Device X has the feature Y
+
+Host B:
+
+Device X has not the feature Y
+
+If we try to migrate without any care from host A to host B, it will
+fail because when migration tries to load the feature Y on
+destination, it will find that the hardware is not there.
+
+Doing this would be the equivalent of doing with cpus:
+
+Host A:
+
+$ qemu-system-x86_64 -cpu host
+
+Host B:
+
+$ qemu-system-x86_64 -cpu host
+
+When both hosts have different cpu features this is waranteed to fail.
+Especially if Host B has less features than host A.  If host A has
+less features than host B, sometimes it works.  Important word of last
+sentence is "sometimes".
+
+So, forgetting about cpu models and continuing with the -cpu host
+example, let's see that the differences of the cpus is that Host A and
+B have the following features:
+
+Features:   'pcid'  'stibp' 'taa-no'
+Host A:        X       X
+Host B:                        X
+
+And we want to migrate between them, the way configure both QEMU cpu
+will be:
+
+Host A:
+
+$ qemu-system-x86_64 -cpu host,pcid=off,stibp=off
+
+Host B:
+
+$ qemu-system-x86_64 -cpu host,taa-no=off
+
+And you would be able to migrate between them.  It is responsability
+of the management application or of the user to make sure that the
+configuration is correct.  QEMU don't know how to look at this kind of
+features in general.
+
+Notice that we don't recomend to use -cpu host for migration.  It is
+used in this example because it makes the exampler simpler.
+
+Other devices have worse control about individual features.  If they
+want to be able to migrate between hosts that show different features,
+the device needs a way to configure which ones it is going to use.
+
+In this section we have considered that we are using the same QEMU
+binary in both sides of the migration.  If we use different QEMU
+versions process, then we need to have into account all other
+differences and the examples become even more complicated.
 
 VMState
 -------
-- 
2.41.0

Juan Quintela <quintela@redhat.com> writes:

> Sometimes devices have different features depending of things outside
> of qemu.  For instance the kernel.  Document how to handle that cases.
>
> Signed-off-by: Juan Quintela <quintela@redhat.com>
> Acked-by: Peter Xu <peterx@redhat.com>
>
> ---
>
> If you have some example to put here, I am all ears.  I guess that
> virtio-* with some features that are on qemu but not on all kernel
> would do the trick, but I am not a virtio guru myself.  Patches
> welcome.
> ---
>  docs/devel/migration.rst | 96 ++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 96 insertions(+)
>
> diff --git a/docs/devel/migration.rst b/docs/devel/migration.rst
> index 5ef2b36e9e..e671df729e 100644
> --- a/docs/devel/migration.rst
> +++ b/docs/devel/migration.rst
> @@ -358,6 +358,102 @@ machine types to have the right value: ::
>           ...
>       };
>  
> +A device with diferent features on both sides
> +---------------------------------------------
> +
> +Let's assume that we are using the same QEMU binary on both sides,
> +just to make the things easier.  But we have a device that has
> +different features on both sides of the migration.  That can be
> +because the devices are different, because the kernel driver of both
> +devices have different features, whatever.
> +
> +How can we get this to work with migration.  The way to do that is
> +"theoretically" easy.  You have to get the features that the device
> +has in the source of the migration.  The features that the device has
> +on the target of the migration, you get the intersection of the
> +features of both sides, and that is the way that you should launch
> +QEMU.
> +
> +Notice that this is not completely related to QEMU.  The most
> +important thing here is that this should be handle by the managing

s/handle/handled/

> +application that launches QEMU.  If QEMU is configured correctly, the
> +migration will suceeed.

s/suceeed/succeed/

> +
> +Once that we have defined that, doing this is complicated.  Almost all

I get what you mean here, but it is slightly confusing. Maybe

"Once we have defined that, doing it is complicated." or
"That said, actually doing it is complicated."

> +devices are bad at being able to be launched with only some features
> +enabled.  With one big exception: cpus.
> +
> +You can read the documentation for QEMU x86 cpu models here:
> +
> +https://qemu-project.gitlab.io/qemu/system/qemu-cpu-models.html
> +
> +See when they talk about migration they recommend that one chooses the
> +newest cpu model that is supported for all cpus.
> +
> +Let's say that we have:
> +
> +Host A:
> +
> +Device X has the feature Y
> +
> +Host B:
> +
> +Device X has not the feature Y
> +
> +If we try to migrate without any care from host A to host B, it will
> +fail because when migration tries to load the feature Y on
> +destination, it will find that the hardware is not there.
> +
> +Doing this would be the equivalent of doing with cpus:
> +
> +Host A:
> +
> +$ qemu-system-x86_64 -cpu host
> +
> +Host B:
> +
> +$ qemu-system-x86_64 -cpu host
> +
> +When both hosts have different cpu features this is waranteed to
> fail.

s/waranteed/guaranteed/

> +Especially if Host B has less features than host A.  If host A has
> +less features than host B, sometimes it works.  Important word of last
> +sentence is "sometimes".
> +
> +So, forgetting about cpu models and continuing with the -cpu host
> +example, let's see that the differences of the cpus is that Host A and
> +B have the following features:
> +
> +Features:   'pcid'  'stibp' 'taa-no'
> +Host A:        X       X
> +Host B:                        X
> +
> +And we want to migrate between them, the way configure both QEMU cpu
> +will be:
> +
> +Host A:
> +
> +$ qemu-system-x86_64 -cpu host,pcid=off,stibp=off
> +
> +Host B:
> +
> +$ qemu-system-x86_64 -cpu host,taa-no=off
> +
> +And you would be able to migrate between them.  It is responsability
> +of the management application or of the user to make sure that the
> +configuration is correct.  QEMU don't know how to look at this kind of

s/don't/doesn't/

> +features in general.
> +
> +Notice that we don't recomend to use -cpu host for migration.  It is
> +used in this example because it makes the exampler simpler.

s/exampler/example/

> +
> +Other devices have worse control about individual features.  If they
> +want to be able to migrate between hosts that show different features,
> +the device needs a way to configure which ones it is going to use.
> +
> +In this section we have considered that we are using the same QEMU
> +binary in both sides of the migration.  If we use different QEMU
> +versions process, then we need to have into account all other
> +differences and the examples become even more complicated.
>  
>  VMState
>  -------

Fabiano Rosas <farosas@suse.de> wrote:
> Juan Quintela <quintela@redhat.com> writes:
>> +Notice that this is not completely related to QEMU.  The most
>> +important thing here is that this should be handle by the managing
>
> s/handle/handled/

Done.

>> +application that launches QEMU.  If QEMU is configured correctly, the
>> +migration will suceeed.
>
> s/suceeed/succeed/

Done.

>
>> +
>> +Once that we have defined that, doing this is complicated.  Almost all
>
> I get what you mean here, but it is slightly confusing. Maybe
>
> "Once we have defined that, doing it is complicated." or
> "That said, actually doing it is complicated."

Last one wins.

>> +When both hosts have different cpu features this is waranteed to
>> fail.
>
> s/waranteed/guaranteed/

Done.
>> +And you would be able to migrate between them.  It is responsability
>> +of the management application or of the user to make sure that the
>> +configuration is correct.  QEMU don't know how to look at this kind of
>
> s/don't/doesn't/

Done

>> +features in general.
>> +
>> +Notice that we don't recomend to use -cpu host for migration.  It is
>> +used in this example because it makes the exampler simpler.
>
> s/exampler/example/

Thanks.