From: Peter Xu <peterx@redhat.com>

The current article is not extremely easy to follow, and may contain too
much information for someone looking for solutions on VMSD compatibility
issues.  Meanwhile, VMSD versioning is not discussed.

I'm not yet sure whether we should just obsolete VMSD versioning; it's
still used quite a lot.  And I had a feeling that for simple use cases
where backward migration is not strongly required, device developers can
still consider using it.  So in this patch I decided to keep it (anyway, we
can't drop it in the near future because of massive existing users), and we
can still allow user to use it in contexts where forward-only migration
might be enough.

This doc patch does below changes:

  - Rename the page from "Backward compatibility" to "Migration
  compatibility", to avoid using "backward" as a word (because we'll want
  to identify "forward" / "backward" migrations in the new doc)

  - Add a TOC for this page for better indexing

  - A new section to explain what is forward/backward migration

  - A new small section for VMSD just to make things complete

  - Explain the two ways to make VMSD compatible with old qemu binaries

    For this one, I added a small section on how to use VMSD versioning for
    new fields just to start such documents.  Rename the old "How backwards
    compatibility works" section to "machine type based (forward+backward
    migration)" to be the 2nd solution (I called it machine type based
    solution). When at it, I provided a summary and a TODO for the 2nd
    solution.

  - A new section to explain which solution to choose

  - Moved the other two existing sections into "Extended readings", because
  they can be even further away to most device developers

Signed-off-by: Peter Xu <peterx@redhat.com>
---
 docs/devel/migration/compatibility.rst | 140 ++++++++++++++++++++++++-
 1 file changed, 137 insertions(+), 3 deletions(-)

diff --git a/docs/devel/migration/compatibility.rst b/docs/devel/migration/compatibility.rst
index 5a5417ef06..ea9da201ef 100644
--- a/docs/devel/migration/compatibility.rst
+++ b/docs/devel/migration/compatibility.rst
@@ -1,8 +1,139 @@
-Backwards compatibility
 =======================
+Migration compatibility
+=======================
+
+Migration is a hard topic sometimes.  One of the major reason is that it
+has a strict compatibility requirement - a migration (live or not) can
+happen between two different versions of QEMUs, so QEMU needs to make sure
+the migration can work across different versions of QEMU binaries.
+
+This document majorly discusses the compatibility requirement of forward /
+backward migrations that QEMU need to maintain, and what QEMU developers
+should do to achieve such compatibility requirements across different QEMU
+versions.
+
+.. contents::
+
+Types of migrations (forward / backward)
+========================================
+
+Let's firstly define the terms **forward migration** and **backward
+migration**.
+
+.. note::
+
+    To simplify the use case, we always discuss between two consecutive
+    versions of QEMU major releases (between QEMU version *N* and QEMU
+    version *N-1*).  But logically it applies to the case where the two
+    QEMU binaries involved contains more than one major version difference.
+
+.. _forward_migration:
+
+**Forward migration**: can be seen as the use case where a VM cluster can
+upgrade its nodes to a newer version of QEMU (version *N*) from an older
+version of QEMU (version *N-1*).
+
+.. _backward_migration:
+
+**Backward migration**: can be seen as the use case where a VM cluster
+would like to migrate from a newer version of QEMU (version *N*) back to an
+even older version of QEMU (version *N-1*).
+
+A forward migration is more common, where system upgrades are needed.  In
+this case, the upgrade can be done seamlessly by live migrating the old VMs
+to the new VMs with the new binaries.
+
+A backward migration can be less common OTOH, because downgrade is less
+common than upgrade for whatever reasons.  However for a production level
+system setup, this should also be allowed, because a cluster can contain
+different versions of QEMU binary.  It should be always allowed to migrate
+between old and new hosts as long as the machine type is supported across
+all the relevant hosts / nodes.
+
+VMState description data structure (VMSD)
+=========================================
+
+VMSD (or in the complete form, **VMStateDescription**) is the data
+structure that QEMU uses to describe data to be migrated for devices.
+Each device should provide its own VMSD structure to describe what it needs
+to be migrated when a VM live migration is requested.
+
+Device VMSD compatibility
+=========================
+
+Then if the VMSD structures need changing, how does the device maintain
+compatibilty?
 
-How backwards compatibility works
----------------------------------
+Here we only discuss VMSD-based migrations.  If one device is not using
+VMSD to migrate its device data, it's considered part of "advanced users",
+then this document may not apply anymore.  If you're writting a new device,
+please always consider starting with VMSD-based migration model.
+
+Consider the case where a device can start to support a new feature in the
+current release, where it wasn't supported before.  The new feature may
+require some new device states to be migrated (which can be new VMSD fields
+to be added, or new subsections).  The same question needs to be answered
+when one would like to modify an existing VMSD fields / layouts to fix a
+bug, and so on.
+
+Depending on the goal, the solution to this problem may vary.
+
+If one would like to provide a full support of migration between whatever
+versions, one can try to implement it using :ref:`machine_type_compat`
+solution.  If one would like to provide a fundamental upgrade-only
+compatibility, one could consider to use the simpler
+:ref:`vmsd_versioning_compat` solution.
+
+Solutions
+=========
+
+.. _vmsd_versioning_compat:
+
+VMSD versioning (forward migration only)
+----------------------------------------
+
+This is normally the simplest way to support cross-version QEMU live
+migration. The trade-off is backward migration will not be supported. It
+means migrations from new QEMU binaries to old QEMU binaries can fail. It's
+because even if the new QEMU can understand the old version of VMSD by
+proper versioning of the VMSD fields, the old QEMU will not be able to
+understand the new version of VMSD layout.  Then when someone migrates a VM
+using the new VMSD to an older version of QEMU, the old QEMU will not
+accept the new migration stream, reporting that the VSMD version too new.
+
+Please have a look at **include/migration/vmstate.h** for more information
+on how to use VMSD versioning.
+
+Taking an example of adding a new field for migration.  The change will
+need to at least contain two parts:
+
+  - Boost existing VMSD version.
+
+  - Add the new VMSD field with the boosted version, with specific
+    **VMSTATE_\*_V()** macros.  For example, **VMSTATE_UINT8_V()** will
+    define an uint8 typed VMSD field with version specified.
+
+.. _machine_type_compat:
+
+Machine type based (forward+backward migration)
+-----------------------------------------------
+
+QEMU developers can leverage machine type compatibile properties to provide
+a fully migratable device / protocol, so the migration behavior will be
+defined by the machine type, no matter which QEMU binary will be used.  One
+can reference the entries defined in **hw_compat_\*** global properties for
+examples.
+
+Comparing to VMSD versioning approach above, this may require more code
+changes, but provide a higher level of compatibility that is bound to the
+machine type being used.  To be explicit, since the migration behavior is
+bound to machine type, it will support both forward migration and backward
+migration as long as the machine type is supported.
+
+.. note::
+
+   Currently this section is pretty long.  TODO: rewrite this section to
+   make it easier for QEMU developers to understand.
 
 When we do migration, we have two QEMU processes: the source and the
 target.  There are two cases, they are the same version or they are
@@ -217,6 +348,9 @@ machine types to have the right value::
          ...
      };
 
+Extended readings
+=================
+
 A device with different features on both sides
 ----------------------------------------------
 
-- 
2.43.0

peterx@redhat.com writes:

> From: Peter Xu <peterx@redhat.com>
>
> The current article is not extremely easy to follow, and may contain too
> much information for someone looking for solutions on VMSD compatibility
> issues.  Meanwhile, VMSD versioning is not discussed.
>
> I'm not yet sure whether we should just obsolete VMSD versioning; it's
> still used quite a lot.  And I had a feeling that for simple use cases
> where backward migration is not strongly required, device developers can
> still consider using it.  So in this patch I decided to keep it (anyway, we
> can't drop it in the near future because of massive existing users), and we
> can still allow user to use it in contexts where forward-only migration
> might be enough.
>
> This doc patch does below changes:
>
>   - Rename the page from "Backward compatibility" to "Migration
>   compatibility", to avoid using "backward" as a word (because we'll want
>   to identify "forward" / "backward" migrations in the new doc)
>
>   - Add a TOC for this page for better indexing
>
>   - A new section to explain what is forward/backward migration
>
>   - A new small section for VMSD just to make things complete
>
>   - Explain the two ways to make VMSD compatible with old qemu binaries
>
>     For this one, I added a small section on how to use VMSD versioning for
>     new fields just to start such documents.  Rename the old "How backwards
>     compatibility works" section to "machine type based (forward+backward
>     migration)" to be the 2nd solution (I called it machine type based
>     solution). When at it, I provided a summary and a TODO for the 2nd
>     solution.
>
>   - A new section to explain which solution to choose
>
>   - Moved the other two existing sections into "Extended readings", because
>   they can be even further away to most device developers
>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>  docs/devel/migration/compatibility.rst | 140 ++++++++++++++++++++++++-
>  1 file changed, 137 insertions(+), 3 deletions(-)
>
> diff --git a/docs/devel/migration/compatibility.rst b/docs/devel/migration/compatibility.rst
> index 5a5417ef06..ea9da201ef 100644
> --- a/docs/devel/migration/compatibility.rst
> +++ b/docs/devel/migration/compatibility.rst
> @@ -1,8 +1,139 @@
> -Backwards compatibility
>  =======================
> +Migration compatibility
> +=======================
> +
> +Migration is a hard topic sometimes.  One of the major reason is that it
> +has a strict compatibility requirement - a migration (live or not) can
> +happen between two different versions of QEMUs, so QEMU needs to make sure

S/QEMUs/QEMU/

> +the migration can work across different versions of QEMU binaries.
> +
> +This document majorly discusses the compatibility requirement of forward /

s/majorly/mainly/

> +backward migrations that QEMU need to maintain, and what QEMU developers

s/need/needs/

> +should do to achieve such compatibility requirements across different QEMU

maybe s/achieve/maintain/ ?

> +versions.
> +
> +.. contents::
> +
> +Types of migrations (forward / backward)
> +========================================
> +
> +Let's firstly define the terms **forward migration** and **backward
> +migration**.
> +
> +.. note::
> +
> +    To simplify the use case, we always discuss between two consecutive
> +    versions of QEMU major releases (between QEMU version *N* and QEMU
> +    version *N-1*).  But logically it applies to the case where the two
> +    QEMU binaries involved contains more than one major version difference.
> +
> +.. _forward_migration:
> +
> +**Forward migration**: can be seen as the use case where a VM cluster can
> +upgrade its nodes to a newer version of QEMU (version *N*) from an older
> +version of QEMU (version *N-1*).
> +
> +.. _backward_migration:
> +
> +**Backward migration**: can be seen as the use case where a VM cluster
> +would like to migrate from a newer version of QEMU (version *N*) back to an
> +even older version of QEMU (version *N-1*).

I'd drop the VM cluster part from these. Define the terms in a more
strict manner (QEMU versions, n/n-1, that's it). Then the parts below
could be second paragraphs further detailing the use-cases of the two
types of migration.

> +
> +A forward migration is more common, where system upgrades are needed.  In
> +this case, the upgrade can be done seamlessly by live migrating the old VMs
> +to the new VMs with the new binaries.

I got a bit confused whether this was describing migration to a
different host or within the same host. I suggest we spell it out some
more:

"live migrating an existing VM that uses an old QEMU binary to another
VM using the newly updated QEMU binary."

> +
> +A backward migration can be less common OTOH, because downgrade is less
> +common than upgrade for whatever reasons.  However for a production level
> +system setup, this should also be allowed, because a cluster can contain
> +different versions of QEMU binary.  It should be always allowed to migrate
> +between old and new hosts as long as the machine type is supported across
> +all the relevant hosts / nodes.
> +
> +VMState description data structure (VMSD)
> +=========================================
> +
> +VMSD (or in the complete form, **VMStateDescription**) is the data
> +structure that QEMU uses to describe data to be migrated for devices.
> +Each device should provide its own VMSD structure to describe what it needs
> +to be migrated when a VM live migration is requested.
> +
> +Device VMSD compatibility
> +=========================
> +
> +Then if the VMSD structures need changing, how does the device maintain
> +compatibilty?

s/compatibilty/compatibility/

Hm, there's a logical jump here from the _device_ changing to this
requiring a change in the VMSD. Also the device is not the thing that
maintains compatibility. Again we might need to spell it out:

"When a device data needs to change from one QEMU version to another,
how to maintain compatibility?" ... or something like this

>  
> -How backwards compatibility works
> ----------------------------------
> +Here we only discuss VMSD-based migrations.  If one device is not using
> +VMSD to migrate its device data, it's considered part of "advanced users",
> +then this document may not apply anymore.  If you're writting a new device,
> +please always consider starting with VMSD-based migration model.
> +
> +Consider the case where a device can start to support a new feature in the
> +current release, where it wasn't supported before.  The new feature may

s/, where it/ that/

> +require some new device states to be migrated (which can be new VMSD fields
> +to be added, or new subsections).  The same question needs to be answered
> +when one would like to modify an existing VMSD fields / layouts to fix a

s/an //

> +bug, and so on.
> +
> +Depending on the goal, the solution to this problem may vary.
> +
> +If one would like to provide a full support of migration between whatever
> +versions, one can try to implement it using :ref:`machine_type_compat`
> +solution.  If one would like to provide a fundamental upgrade-only
> +compatibility, one could consider to use the simpler
> +:ref:`vmsd_versioning_compat` solution.

Should this whole paragraph be put before mentioning VMSD? Otherwise
here we are talking about machine type compat after having said: "Here
we only discuss VMSD-based migrations".

All in all this could be:

up top:
Migration compatibility strategies
==================================

When a device data needs to change from one QEMU version to another, how
to maintain compatibility?

Depending on the goal, the solution to this problem may vary.

If one would like to provide a full support of migration between
arbitrary versions, one can try to implement it using
:ref:`machine_type_compat` solution.  If one would like to provide a
fundamental upgrade-only compatibility, one could consider to use the
simpler :ref:`vmsd_versioning_compat` solution.

VMState description data structure (VMSD)
-----------------------------------------

VMSD (or in the complete form, **VMStateDescription**) is the data ...
...

Machine type properties
-----------------------

The QEMU machine type is versioned (-machine q35 resolves is equivalent
to pc-q35-9.0) and has properties that can be set to different values
across machine versions. Those can be used to enable/disable different
parts of the code and can be used to help migration compatibility.

Migration compatibility solutions
=================================

VMSD versioning (forward migration only)
----------------------------------------
...

Machine type based (forward+backward migration)
-----------------------------------------------
...

> +
> +Solutions
> +=========
> +
> +.. _vmsd_versioning_compat:
> +
> +VMSD versioning (forward migration only)
> +----------------------------------------
> +
> +This is normally the simplest way to support cross-version QEMU live
> +migration. The trade-off is backward migration will not be supported. It
> +means migrations from new QEMU binaries to old QEMU binaries can fail. It's
> +because even if the new QEMU can understand the old version of VMSD by
> +proper versioning of the VMSD fields, the old QEMU will not be able to
> +understand the new version of VMSD layout.  Then when someone migrates a VM
> +using the new VMSD to an older version of QEMU, the old QEMU will not
> +accept the new migration stream, reporting that the VSMD version too new.

But we still have _some_ form of backward compat for _some_ kinds of
problems by using dummy fields in the VMSD for instance, right?

> +
> +Please have a look at **include/migration/vmstate.h** for more information
> +on how to use VMSD versioning.
> +
> +Taking an example of adding a new field for migration.  The change will
> +need to at least contain two parts:
> +
> +  - Boost existing VMSD version.
> +
> +  - Add the new VMSD field with the boosted version, with specific
> +    **VMSTATE_\*_V()** macros.  For example, **VMSTATE_UINT8_V()** will
> +    define an uint8 typed VMSD field with version specified.
> +
> +.. _machine_type_compat:
> +
> +Machine type based (forward+backward migration)
> +-----------------------------------------------
> +
> +QEMU developers can leverage machine type compatibile properties to provide

compatible

> +a fully migratable device / protocol, so the migration behavior will be
> +defined by the machine type, no matter which QEMU binary will be used.  One
> +can reference the entries defined in **hw_compat_\*** global properties for
> +examples.
> +
> +Comparing to VMSD versioning approach above, this may require more code
> +changes, but provide a higher level of compatibility that is bound to the
> +machine type being used.  To be explicit, since the migration behavior is
> +bound to machine type, it will support both forward migration and backward
> +migration as long as the machine type is supported.
> +
> +.. note::
> +
> +   Currently this section is pretty long.  TODO: rewrite this section to
> +   make it easier for QEMU developers to understand.
>  
>  When we do migration, we have two QEMU processes: the source and the
>  target.  There are two cases, they are the same version or they are
> @@ -217,6 +348,9 @@ machine types to have the right value::
>           ...
>       };
>  
> +Extended readings
> +=================
> +
>  A device with different features on both sides
>  ----------------------------------------------

On Mon, Jan 22, 2024 at 12:39:06PM -0300, Fabiano Rosas wrote:
> peterx@redhat.com writes:
> 
> > From: Peter Xu <peterx@redhat.com>
> >
> > The current article is not extremely easy to follow, and may contain too
> > much information for someone looking for solutions on VMSD compatibility
> > issues.  Meanwhile, VMSD versioning is not discussed.
> >
> > I'm not yet sure whether we should just obsolete VMSD versioning; it's
> > still used quite a lot.  And I had a feeling that for simple use cases
> > where backward migration is not strongly required, device developers can
> > still consider using it.  So in this patch I decided to keep it (anyway, we
> > can't drop it in the near future because of massive existing users), and we
> > can still allow user to use it in contexts where forward-only migration
> > might be enough.
> >
> > This doc patch does below changes:
> >
> >   - Rename the page from "Backward compatibility" to "Migration
> >   compatibility", to avoid using "backward" as a word (because we'll want
> >   to identify "forward" / "backward" migrations in the new doc)
> >
> >   - Add a TOC for this page for better indexing
> >
> >   - A new section to explain what is forward/backward migration
> >
> >   - A new small section for VMSD just to make things complete
> >
> >   - Explain the two ways to make VMSD compatible with old qemu binaries
> >
> >     For this one, I added a small section on how to use VMSD versioning for
> >     new fields just to start such documents.  Rename the old "How backwards
> >     compatibility works" section to "machine type based (forward+backward
> >     migration)" to be the 2nd solution (I called it machine type based
> >     solution). When at it, I provided a summary and a TODO for the 2nd
> >     solution.
> >
> >   - A new section to explain which solution to choose
> >
> >   - Moved the other two existing sections into "Extended readings", because
> >   they can be even further away to most device developers
> >
> > Signed-off-by: Peter Xu <peterx@redhat.com>
> > ---
> >  docs/devel/migration/compatibility.rst | 140 ++++++++++++++++++++++++-
> >  1 file changed, 137 insertions(+), 3 deletions(-)
> >
> > diff --git a/docs/devel/migration/compatibility.rst b/docs/devel/migration/compatibility.rst
> > index 5a5417ef06..ea9da201ef 100644
> > --- a/docs/devel/migration/compatibility.rst
> > +++ b/docs/devel/migration/compatibility.rst
> > @@ -1,8 +1,139 @@
> > -Backwards compatibility
> >  =======================
> > +Migration compatibility
> > +=======================
> > +
> > +Migration is a hard topic sometimes.  One of the major reason is that it
> > +has a strict compatibility requirement - a migration (live or not) can
> > +happen between two different versions of QEMUs, so QEMU needs to make sure
> 
> S/QEMUs/QEMU/

I'll fix all these up that you suggested as English issues.

> 
> > +the migration can work across different versions of QEMU binaries.
> > +
> > +This document majorly discusses the compatibility requirement of forward /
> 
> s/majorly/mainly/
> 
> > +backward migrations that QEMU need to maintain, and what QEMU developers
> 
> s/need/needs/
> 
> > +should do to achieve such compatibility requirements across different QEMU
> 
> maybe s/achieve/maintain/ ?
> 
> > +versions.
> > +
> > +.. contents::
> > +
> > +Types of migrations (forward / backward)
> > +========================================
> > +
> > +Let's firstly define the terms **forward migration** and **backward
> > +migration**.
> > +
> > +.. note::
> > +
> > +    To simplify the use case, we always discuss between two consecutive
> > +    versions of QEMU major releases (between QEMU version *N* and QEMU
> > +    version *N-1*).  But logically it applies to the case where the two
> > +    QEMU binaries involved contains more than one major version difference.
> > +
> > +.. _forward_migration:
> > +
> > +**Forward migration**: can be seen as the use case where a VM cluster can
> > +upgrade its nodes to a newer version of QEMU (version *N*) from an older
> > +version of QEMU (version *N-1*).
> > +
> > +.. _backward_migration:
> > +
> > +**Backward migration**: can be seen as the use case where a VM cluster
> > +would like to migrate from a newer version of QEMU (version *N*) back to an
> > +even older version of QEMU (version *N-1*).
> 
> I'd drop the VM cluster part from these. Define the terms in a more
> strict manner (QEMU versions, n/n-1, that's it). Then the parts below
> could be second paragraphs further detailing the use-cases of the two
> types of migration.

OK.

> 
> > +
> > +A forward migration is more common, where system upgrades are needed.  In
> > +this case, the upgrade can be done seamlessly by live migrating the old VMs
> > +to the new VMs with the new binaries.
> 
> I got a bit confused whether this was describing migration to a
> different host or within the same host. I suggest we spell it out some
> more:

It can be same-host or across-host.

> 
> "live migrating an existing VM that uses an old QEMU binary to another
> VM using the newly updated QEMU binary."

Even though I don't see a major difference on how it was reworded.. but I
can use your version here.

> 
> > +
> > +A backward migration can be less common OTOH, because downgrade is less
> > +common than upgrade for whatever reasons.  However for a production level
> > +system setup, this should also be allowed, because a cluster can contain
> > +different versions of QEMU binary.  It should be always allowed to migrate
> > +between old and new hosts as long as the machine type is supported across
> > +all the relevant hosts / nodes.
> > +
> > +VMState description data structure (VMSD)
> > +=========================================
> > +
> > +VMSD (or in the complete form, **VMStateDescription**) is the data
> > +structure that QEMU uses to describe data to be migrated for devices.
> > +Each device should provide its own VMSD structure to describe what it needs
> > +to be migrated when a VM live migration is requested.
> > +
> > +Device VMSD compatibility
> > +=========================
> > +
> > +Then if the VMSD structures need changing, how does the device maintain
> > +compatibilty?
> 
> s/compatibilty/compatibility/
> 
> Hm, there's a logical jump here from the _device_ changing to this
> requiring a change in the VMSD. Also the device is not the thing that
> maintains compatibility. Again we might need to spell it out:
> 
> "When a device data needs to change from one QEMU version to another,
> how to maintain compatibility?" ... or something like this

OK.

> 
> >  
> > -How backwards compatibility works
> > ----------------------------------
> > +Here we only discuss VMSD-based migrations.  If one device is not using
> > +VMSD to migrate its device data, it's considered part of "advanced users",
> > +then this document may not apply anymore.  If you're writting a new device,
> > +please always consider starting with VMSD-based migration model.
> > +
> > +Consider the case where a device can start to support a new feature in the
> > +current release, where it wasn't supported before.  The new feature may
> 
> s/, where it/ that/
> 
> > +require some new device states to be migrated (which can be new VMSD fields
> > +to be added, or new subsections).  The same question needs to be answered
> > +when one would like to modify an existing VMSD fields / layouts to fix a
> 
> s/an //
> 
> > +bug, and so on.
> > +
> > +Depending on the goal, the solution to this problem may vary.
> > +
> > +If one would like to provide a full support of migration between whatever
> > +versions, one can try to implement it using :ref:`machine_type_compat`
> > +solution.  If one would like to provide a fundamental upgrade-only
> > +compatibility, one could consider to use the simpler
> > +:ref:`vmsd_versioning_compat` solution.
> 
> Should this whole paragraph be put before mentioning VMSD? Otherwise
> here we are talking about machine type compat after having said: "Here
> we only discuss VMSD-based migrations".

Note that even with the machine compat properties solution we mentioned
below, the device will still need some VMSD entry changes, and VMSD is
still the core of the problem.

The difference is instead of using VMSD versioning, one may need to link
the VMSD entry to the machine compat properties via either:

  - VMStateDescription.needed(), as a sub-vmsd or,

  - VMStateField.field_exists() as a vmsd field.

That's why I want to keep VMSD a separate section, because it's always the
core of the problem, no matter for either of the two solutions below.

> 
> All in all this could be:
> 
> up top:
> Migration compatibility strategies
> ==================================
> 
> When a device data needs to change from one QEMU version to another, how
> to maintain compatibility?
> 
> Depending on the goal, the solution to this problem may vary.
> 
> If one would like to provide a full support of migration between
> arbitrary versions, one can try to implement it using
> :ref:`machine_type_compat` solution.  If one would like to provide a
> fundamental upgrade-only compatibility, one could consider to use the
> simpler :ref:`vmsd_versioning_compat` solution.
> 
> VMState description data structure (VMSD)
> -----------------------------------------
> 
> VMSD (or in the complete form, **VMStateDescription**) is the data ...
> ...
> 
> Machine type properties
> -----------------------
> 
> The QEMU machine type is versioned (-machine q35 resolves is equivalent
> to pc-q35-9.0) and has properties that can be set to different values
> across machine versions. Those can be used to enable/disable different
> parts of the code and can be used to help migration compatibility.

Some statement on machine types may be good, but I'm afraid then we'll also
need to describe more on machine compat properties, which is IMHO even more
important.  That part is actually more or less covered by the old document
below (which is pretty long that Juan drafted just a while ago).  That's
why I didn't mention much on machine types here, leaving that for the rest.

PS: I added a "TODO" though right below for a possible future rewrite [1].
The current explanation is, IMHO, too much, and not as clear.

The major goal of this doc patch is adding vmsd versioning into the
picutre, and describe both solutions we can use, while in Juan's old
version there's no mention of VMSD versioning.  The ultimate goal is in the
future we can share the doc to other developers when asking similar
questions.

Since we're at it, I would also like to know how you think about whether we
should still suggest people using VMSD versioning, as we know that it won't
work for backward migrations.

My current thoughts is it is still fine, as it's easier to use, and it
should still be applicable to the cases where a strict migration semantics
are not required.  However it's hard to justify which device needs that
strictness.

For example, any device to be used in migration-test must be forward +
backward migration compatible at least, because you just added the n-1
regression tests to cover both directions.  Said that, only a few devices
are involved because currently our migration-test qemu cmdline is pretty
simple.

Fundamentally, IMHO it's because QEMU as a project is used both in
enterprise and personal emulations.  I think it might be too strict to
always request backward migration capability if we know some device / arch
is only used for personal, or educational, purposes.

The other reason is so far using machine compat properties can sometimes be
complicated, and I'm not sure whether it's worthwhile we keep requesting
that to device developers.  Maybe some day we can provide some VMSD macros
to make that even easier, then we can obsolete VMSD versionings.  But I
haven't really thought it through.

> 
> Migration compatibility solutions
> =================================
> 
> VMSD versioning (forward migration only)
> ----------------------------------------
> ...
> 
> Machine type based (forward+backward migration)
> -----------------------------------------------
> ...
> 
> > +
> > +Solutions
> > +=========
> > +
> > +.. _vmsd_versioning_compat:
> > +
> > +VMSD versioning (forward migration only)
> > +----------------------------------------
> > +
> > +This is normally the simplest way to support cross-version QEMU live
> > +migration. The trade-off is backward migration will not be supported. It
> > +means migrations from new QEMU binaries to old QEMU binaries can fail. It's
> > +because even if the new QEMU can understand the old version of VMSD by
> > +proper versioning of the VMSD fields, the old QEMU will not be able to
> > +understand the new version of VMSD layout.  Then when someone migrates a VM
> > +using the new VMSD to an older version of QEMU, the old QEMU will not
> > +accept the new migration stream, reporting that the VSMD version too new.
> 
> But we still have _some_ form of backward compat for _some_ kinds of
> problems by using dummy fields in the VMSD for instance, right?

Just to make sure we're on the same page: UNUSED is definitely needed for
forward migration, since if without those the old stream will still contain
the removed field, which on the new binary will be wrongly recognized as
the "next" field if ever existed, or "extra / unknown" fields, probably
messing up the next device state.

For backward, I assume it only works if all-zero is a valid state first on
that field being removed on the old binary.  Otherwise it might still fail
in weird ways, afaiu.  For that, I'd say that shouldn't be stated as
supported either as a general approach.

> 
> > +
> > +Please have a look at **include/migration/vmstate.h** for more information
> > +on how to use VMSD versioning.
> > +
> > +Taking an example of adding a new field for migration.  The change will
> > +need to at least contain two parts:
> > +
> > +  - Boost existing VMSD version.
> > +
> > +  - Add the new VMSD field with the boosted version, with specific
> > +    **VMSTATE_\*_V()** macros.  For example, **VMSTATE_UINT8_V()** will
> > +    define an uint8 typed VMSD field with version specified.
> > +
> > +.. _machine_type_compat:
> > +
> > +Machine type based (forward+backward migration)
> > +-----------------------------------------------
> > +
> > +QEMU developers can leverage machine type compatibile properties to provide
> 
> compatible
> 
> > +a fully migratable device / protocol, so the migration behavior will be
> > +defined by the machine type, no matter which QEMU binary will be used.  One
> > +can reference the entries defined in **hw_compat_\*** global properties for
> > +examples.
> > +
> > +Comparing to VMSD versioning approach above, this may require more code
> > +changes, but provide a higher level of compatibility that is bound to the
> > +machine type being used.  To be explicit, since the migration behavior is
> > +bound to machine type, it will support both forward migration and backward
> > +migration as long as the machine type is supported.
> > +
> > +.. note::
> > +
> > +   Currently this section is pretty long.  TODO: rewrite this section to
> > +   make it easier for QEMU developers to understand.

[1]

> >  
> >  When we do migration, we have two QEMU processes: the source and the
> >  target.  There are two cases, they are the same version or they are
> > @@ -217,6 +348,9 @@ machine types to have the right value::
> >           ...
> >       };
> >  
> > +Extended readings
> > +=================
> > +
> >  A device with different features on both sides
> >  ----------------------------------------------
> 

-- 
Peter Xu

Peter Xu <peterx@redhat.com> writes:

> On Mon, Jan 22, 2024 at 12:39:06PM -0300, Fabiano Rosas wrote:
>> peterx@redhat.com writes:
>> 
>> > From: Peter Xu <peterx@redhat.com>
>> >
>> > The current article is not extremely easy to follow, and may contain too
>> > much information for someone looking for solutions on VMSD compatibility
>> > issues.  Meanwhile, VMSD versioning is not discussed.
>> >
>> > I'm not yet sure whether we should just obsolete VMSD versioning; it's
>> > still used quite a lot.  And I had a feeling that for simple use cases
>> > where backward migration is not strongly required, device developers can
>> > still consider using it.  So in this patch I decided to keep it (anyway, we
>> > can't drop it in the near future because of massive existing users), and we
>> > can still allow user to use it in contexts where forward-only migration
>> > might be enough.
>> >
>> > This doc patch does below changes:
>> >
>> >   - Rename the page from "Backward compatibility" to "Migration
>> >   compatibility", to avoid using "backward" as a word (because we'll want
>> >   to identify "forward" / "backward" migrations in the new doc)
>> >
>> >   - Add a TOC for this page for better indexing
>> >
>> >   - A new section to explain what is forward/backward migration
>> >
>> >   - A new small section for VMSD just to make things complete
>> >
>> >   - Explain the two ways to make VMSD compatible with old qemu binaries
>> >
>> >     For this one, I added a small section on how to use VMSD versioning for
>> >     new fields just to start such documents.  Rename the old "How backwards
>> >     compatibility works" section to "machine type based (forward+backward
>> >     migration)" to be the 2nd solution (I called it machine type based
>> >     solution). When at it, I provided a summary and a TODO for the 2nd
>> >     solution.
>> >
>> >   - A new section to explain which solution to choose
>> >
>> >   - Moved the other two existing sections into "Extended readings", because
>> >   they can be even further away to most device developers
>> >
>> > Signed-off-by: Peter Xu <peterx@redhat.com>
>> > ---
>> >  docs/devel/migration/compatibility.rst | 140 ++++++++++++++++++++++++-
>> >  1 file changed, 137 insertions(+), 3 deletions(-)
>> >
>> > diff --git a/docs/devel/migration/compatibility.rst b/docs/devel/migration/compatibility.rst
>> > index 5a5417ef06..ea9da201ef 100644
>> > --- a/docs/devel/migration/compatibility.rst
>> > +++ b/docs/devel/migration/compatibility.rst
>> > @@ -1,8 +1,139 @@
>> > -Backwards compatibility
>> >  =======================
>> > +Migration compatibility
>> > +=======================
>> > +
>> > +Migration is a hard topic sometimes.  One of the major reason is that it
>> > +has a strict compatibility requirement - a migration (live or not) can
>> > +happen between two different versions of QEMUs, so QEMU needs to make sure
>> 
>> S/QEMUs/QEMU/
>
> I'll fix all these up that you suggested as English issues.
>
>> 
>> > +the migration can work across different versions of QEMU binaries.
>> > +
>> > +This document majorly discusses the compatibility requirement of forward /
>> 
>> s/majorly/mainly/
>> 
>> > +backward migrations that QEMU need to maintain, and what QEMU developers
>> 
>> s/need/needs/
>> 
>> > +should do to achieve such compatibility requirements across different QEMU
>> 
>> maybe s/achieve/maintain/ ?
>> 
>> > +versions.
>> > +
>> > +.. contents::
>> > +
>> > +Types of migrations (forward / backward)
>> > +========================================
>> > +
>> > +Let's firstly define the terms **forward migration** and **backward
>> > +migration**.
>> > +
>> > +.. note::
>> > +
>> > +    To simplify the use case, we always discuss between two consecutive
>> > +    versions of QEMU major releases (between QEMU version *N* and QEMU
>> > +    version *N-1*).  But logically it applies to the case where the two
>> > +    QEMU binaries involved contains more than one major version difference.
>> > +
>> > +.. _forward_migration:
>> > +
>> > +**Forward migration**: can be seen as the use case where a VM cluster can
>> > +upgrade its nodes to a newer version of QEMU (version *N*) from an older
>> > +version of QEMU (version *N-1*).
>> > +
>> > +.. _backward_migration:
>> > +
>> > +**Backward migration**: can be seen as the use case where a VM cluster
>> > +would like to migrate from a newer version of QEMU (version *N*) back to an
>> > +even older version of QEMU (version *N-1*).
>> 
>> I'd drop the VM cluster part from these. Define the terms in a more
>> strict manner (QEMU versions, n/n-1, that's it). Then the parts below
>> could be second paragraphs further detailing the use-cases of the two
>> types of migration.
>
> OK.
>
>> 
>> > +
>> > +A forward migration is more common, where system upgrades are needed.  In
>> > +this case, the upgrade can be done seamlessly by live migrating the old VMs
>> > +to the new VMs with the new binaries.
>> 
>> I got a bit confused whether this was describing migration to a
>> different host or within the same host. I suggest we spell it out some
>> more:
>
> It can be same-host or across-host.
>
>> 
>> "live migrating an existing VM that uses an old QEMU binary to another
>> VM using the newly updated QEMU binary."
>
> Even though I don't see a major difference on how it was reworded.. but I
> can use your version here.
>
>> 
>> > +
>> > +A backward migration can be less common OTOH, because downgrade is less
>> > +common than upgrade for whatever reasons.  However for a production level
>> > +system setup, this should also be allowed, because a cluster can contain
>> > +different versions of QEMU binary.  It should be always allowed to migrate
>> > +between old and new hosts as long as the machine type is supported across
>> > +all the relevant hosts / nodes.
>> > +
>> > +VMState description data structure (VMSD)
>> > +=========================================
>> > +
>> > +VMSD (or in the complete form, **VMStateDescription**) is the data
>> > +structure that QEMU uses to describe data to be migrated for devices.
>> > +Each device should provide its own VMSD structure to describe what it needs
>> > +to be migrated when a VM live migration is requested.
>> > +
>> > +Device VMSD compatibility
>> > +=========================
>> > +
>> > +Then if the VMSD structures need changing, how does the device maintain
>> > +compatibilty?
>> 
>> s/compatibilty/compatibility/
>> 
>> Hm, there's a logical jump here from the _device_ changing to this
>> requiring a change in the VMSD. Also the device is not the thing that
>> maintains compatibility. Again we might need to spell it out:
>> 
>> "When a device data needs to change from one QEMU version to another,
>> how to maintain compatibility?" ... or something like this
>
> OK.
>
>> 
>> >  
>> > -How backwards compatibility works
>> > ----------------------------------
>> > +Here we only discuss VMSD-based migrations.  If one device is not using
>> > +VMSD to migrate its device data, it's considered part of "advanced users",
>> > +then this document may not apply anymore.  If you're writting a new device,
>> > +please always consider starting with VMSD-based migration model.
>> > +
>> > +Consider the case where a device can start to support a new feature in the
>> > +current release, where it wasn't supported before.  The new feature may
>> 
>> s/, where it/ that/
>> 
>> > +require some new device states to be migrated (which can be new VMSD fields
>> > +to be added, or new subsections).  The same question needs to be answered
>> > +when one would like to modify an existing VMSD fields / layouts to fix a
>> 
>> s/an //
>> 
>> > +bug, and so on.
>> > +
>> > +Depending on the goal, the solution to this problem may vary.
>> > +
>> > +If one would like to provide a full support of migration between whatever
>> > +versions, one can try to implement it using :ref:`machine_type_compat`
>> > +solution.  If one would like to provide a fundamental upgrade-only
>> > +compatibility, one could consider to use the simpler
>> > +:ref:`vmsd_versioning_compat` solution.
>> 
>> Should this whole paragraph be put before mentioning VMSD? Otherwise
>> here we are talking about machine type compat after having said: "Here
>> we only discuss VMSD-based migrations".
>
> Note that even with the machine compat properties solution we mentioned
> below, the device will still need some VMSD entry changes, and VMSD is
> still the core of the problem.
>
> The difference is instead of using VMSD versioning, one may need to link
> the VMSD entry to the machine compat properties via either:
>
>   - VMStateDescription.needed(), as a sub-vmsd or,
>
>   - VMStateField.field_exists() as a vmsd field.
>
> That's why I want to keep VMSD a separate section, because it's always the
> core of the problem, no matter for either of the two solutions below.
>
>> 
>> All in all this could be:
>> 
>> up top:
>> Migration compatibility strategies
>> ==================================
>> 
>> When a device data needs to change from one QEMU version to another, how
>> to maintain compatibility?
>> 
>> Depending on the goal, the solution to this problem may vary.
>> 
>> If one would like to provide a full support of migration between
>> arbitrary versions, one can try to implement it using
>> :ref:`machine_type_compat` solution.  If one would like to provide a
>> fundamental upgrade-only compatibility, one could consider to use the
>> simpler :ref:`vmsd_versioning_compat` solution.
>> 
>> VMState description data structure (VMSD)
>> -----------------------------------------
>> 
>> VMSD (or in the complete form, **VMStateDescription**) is the data ...
>> ...
>> 
>> Machine type properties
>> -----------------------
>> 
>> The QEMU machine type is versioned (-machine q35 resolves is equivalent
>> to pc-q35-9.0) and has properties that can be set to different values
>> across machine versions. Those can be used to enable/disable different
>> parts of the code and can be used to help migration compatibility.
>
> Some statement on machine types may be good, but I'm afraid then we'll also
> need to describe more on machine compat properties, which is IMHO even more
> important.  That part is actually more or less covered by the old document
> below (which is pretty long that Juan drafted just a while ago).  That's
> why I didn't mention much on machine types here, leaving that for the rest.
>
> PS: I added a "TODO" though right below for a possible future rewrite [1].
> The current explanation is, IMHO, too much, and not as clear.
>
> The major goal of this doc patch is adding vmsd versioning into the
> picutre, and describe both solutions we can use, while in Juan's old
> version there's no mention of VMSD versioning.  The ultimate goal is in the
> future we can share the doc to other developers when asking similar
> questions.
>
> Since we're at it, I would also like to know how you think about whether we
> should still suggest people using VMSD versioning, as we know that it won't
> work for backward migrations.
>
> My current thoughts is it is still fine, as it's easier to use, and it
> should still be applicable to the cases where a strict migration semantics
> are not required.  However it's hard to justify which device needs that
> strictness.

I'd prefer if we kept things strict. However I don't think we can do
that without having enough testing and specially, clear recipes on how
to add compatibility back once it gets lost. Think of that recent thread
were we discussed an old powerpc issue. How come we can see the fix
today in the code but cannot tell which problem it was trying to solve?
That's bonkers. Ideally every type of breakage would have a mapping into
why it breaks and how to fix it.

So with testing to catch the issue early and a clear step-by-step on how
to identify and fix compatibility, then we could require strict
compatibility for every device.

>
> For example, any device to be used in migration-test must be forward +
> backward migration compatible at least, because you just added the n-1
> regression tests to cover both directions.  Said that, only a few devices
> are involved because currently our migration-test qemu cmdline is pretty
> simple.

We might want to make a distinction between migration core vs. device
state testing. I see n-1 testing more like migration core testing. It's
bad to break migration, but it's really bad to break migration for
everyone because we refactored something deep within migration/.

I also wouldn't mind if we had some simple way for device developers to
add migration tests that cover their code. Currently it's infeasible to
edit migration-test with new command lines for every device of
interest. Maybe we could have a little framework that takes a command
line and spits a migration stream? Something really self-contained,
behind the device's CONFIG in meson.

>
> Fundamentally, IMHO it's because QEMU as a project is used both in
> enterprise and personal emulations.  I think it might be too strict to
> always request backward migration capability if we know some device / arch
> is only used for personal, or educational, purposes.

Do we need migration support tiers? =)

If a machine supports KVM, that's already a good indication that we will
want to have migration working both ways.

If the machine supports only TCG, then migration will probably be a nice
to have for development, but no one would be relying on it in
production. It might be ok to only support forward migration.

Could that be a first filter to reason about this? That might narrow the
list of devices enough.

> The other reason is so far using machine compat properties can sometimes be
> complicated, and I'm not sure whether it's worthwhile we keep requesting
> that to device developers.  Maybe some day we can provide some VMSD macros
> to make that even easier, then we can obsolete VMSD versionings.  But I
> haven't really thought it through.
>
>> 
>> Migration compatibility solutions
>> =================================
>> 
>> VMSD versioning (forward migration only)
>> ----------------------------------------
>> ...
>> 
>> Machine type based (forward+backward migration)
>> -----------------------------------------------
>> ...
>> 
>> > +
>> > +Solutions
>> > +=========
>> > +
>> > +.. _vmsd_versioning_compat:
>> > +
>> > +VMSD versioning (forward migration only)
>> > +----------------------------------------
>> > +
>> > +This is normally the simplest way to support cross-version QEMU live
>> > +migration. The trade-off is backward migration will not be supported. It
>> > +means migrations from new QEMU binaries to old QEMU binaries can fail. It's
>> > +because even if the new QEMU can understand the old version of VMSD by
>> > +proper versioning of the VMSD fields, the old QEMU will not be able to
>> > +understand the new version of VMSD layout.  Then when someone migrates a VM
>> > +using the new VMSD to an older version of QEMU, the old QEMU will not
>> > +accept the new migration stream, reporting that the VSMD version too new.
>> 
>> But we still have _some_ form of backward compat for _some_ kinds of
>> problems by using dummy fields in the VMSD for instance, right?
>
> Just to make sure we're on the same page: UNUSED is definitely needed for
> forward migration, since if without those the old stream will still contain
> the removed field, which on the new binary will be wrongly recognized as
> the "next" field if ever existed, or "extra / unknown" fields, probably
> messing up the next device state.
>
> For backward, I assume it only works if all-zero is a valid state first on
> that field being removed on the old binary.  Otherwise it might still fail
> in weird ways, afaiu.  For that, I'd say that shouldn't be stated as
> supported either as a general approach.
>
>> 
>> > +
>> > +Please have a look at **include/migration/vmstate.h** for more information
>> > +on how to use VMSD versioning.
>> > +
>> > +Taking an example of adding a new field for migration.  The change will
>> > +need to at least contain two parts:
>> > +
>> > +  - Boost existing VMSD version.
>> > +
>> > +  - Add the new VMSD field with the boosted version, with specific
>> > +    **VMSTATE_\*_V()** macros.  For example, **VMSTATE_UINT8_V()** will
>> > +    define an uint8 typed VMSD field with version specified.
>> > +
>> > +.. _machine_type_compat:
>> > +
>> > +Machine type based (forward+backward migration)
>> > +-----------------------------------------------
>> > +
>> > +QEMU developers can leverage machine type compatibile properties to provide
>> 
>> compatible
>> 
>> > +a fully migratable device / protocol, so the migration behavior will be
>> > +defined by the machine type, no matter which QEMU binary will be used.  One
>> > +can reference the entries defined in **hw_compat_\*** global properties for
>> > +examples.
>> > +
>> > +Comparing to VMSD versioning approach above, this may require more code
>> > +changes, but provide a higher level of compatibility that is bound to the
>> > +machine type being used.  To be explicit, since the migration behavior is
>> > +bound to machine type, it will support both forward migration and backward
>> > +migration as long as the machine type is supported.
>> > +
>> > +.. note::
>> > +
>> > +   Currently this section is pretty long.  TODO: rewrite this section to
>> > +   make it easier for QEMU developers to understand.
>
> [1]
>
>> >  
>> >  When we do migration, we have two QEMU processes: the source and the
>> >  target.  There are two cases, they are the same version or they are
>> > @@ -217,6 +348,9 @@ machine types to have the right value::
>> >           ...
>> >       };
>> >  
>> > +Extended readings
>> > +=================
>> > +
>> >  A device with different features on both sides
>> >  ----------------------------------------------
>>

On Mon, Jan 29, 2024 at 10:44:46AM -0300, Fabiano Rosas wrote:
> > Since we're at it, I would also like to know how you think about whether we
> > should still suggest people using VMSD versioning, as we know that it won't
> > work for backward migrations.
> >
> > My current thoughts is it is still fine, as it's easier to use, and it
> > should still be applicable to the cases where a strict migration semantics
> > are not required.  However it's hard to justify which device needs that
> > strictness.
> 
> I'd prefer if we kept things strict. However I don't think we can do
> that without having enough testing and specially, clear recipes on how
> to add compatibility back once it gets lost. Think of that recent thread

If it was broken, IMHO we should just fix it and backport to stable.

I think Juan used to worry on what happens if someone already used an old
version of old release, e.g., someone using 8.2.0 may not be able to
migrate to 8.2.1 if we fix that breakage in 9.0 and backport that to 8.2.1.
My take is that maybe that's overcomplicated, and maybe we should simply
only maintain the latest stable version, rather than all.  In this case,
IMHO it will be less burden if we only guarantee 8.2.1 will be working,
e.g., when migrating from 8.1.z -> 8.2.1.  Then we should just state a
known issue in 8.2.0 that it is broken, and both:

  (1) 8.1.z -> 8.2.0, and
  (2) 8.2.0 -> 8.2.1

will expect to fail.

> were we discussed an old powerpc issue. How come we can see the fix
> today in the code but cannot tell which problem it was trying to solve?
> That's bonkers. Ideally every type of breakage would have a mapping into
> why it breaks and how to fix it.
> 
> So with testing to catch the issue early and a clear step-by-step on how
> to identify and fix compatibility, then we could require strict
> compatibility for every device.

I don't think we can guarantee no bug there, but indeed we can do better on
providing some test framework for device VMSDs.

> 
> >
> > For example, any device to be used in migration-test must be forward +
> > backward migration compatible at least, because you just added the n-1
> > regression tests to cover both directions.  Said that, only a few devices
> > are involved because currently our migration-test qemu cmdline is pretty
> > simple.
> 
> We might want to make a distinction between migration core vs. device
> state testing. I see n-1 testing more like migration core testing. It's
> bad to break migration, but it's really bad to break migration for
> everyone because we refactored something deep within migration/.
> 
> I also wouldn't mind if we had some simple way for device developers to
> add migration tests that cover their code. Currently it's infeasible to
> edit migration-test with new command lines for every device of
> interest. Maybe we could have a little framework that takes a command
> line and spits a migration stream? Something really self-contained,
> behind the device's CONFIG in meson.

I added one more todo:

https://wiki.qemu.org/ToDo/LiveMigration#Device_migration_stream_test_framework

How's that look?  Feel free to modify on your will.

-- 
Peter Xu

Peter Xu <peterx@redhat.com> writes:

> On Mon, Jan 29, 2024 at 10:44:46AM -0300, Fabiano Rosas wrote:
>> > Since we're at it, I would also like to know how you think about whether we
>> > should still suggest people using VMSD versioning, as we know that it won't
>> > work for backward migrations.
>> >
>> > My current thoughts is it is still fine, as it's easier to use, and it
>> > should still be applicable to the cases where a strict migration semantics
>> > are not required.  However it's hard to justify which device needs that
>> > strictness.
>> 
>> I'd prefer if we kept things strict. However I don't think we can do
>> that without having enough testing and specially, clear recipes on how
>> to add compatibility back once it gets lost. Think of that recent thread
>
> If it was broken, IMHO we should just fix it and backport to stable.

(tangent)
Sure, but I'm talking about how do we instruct device developers on
fixing migration bugs. We cannot simply yell "regression!" and expect
people to care.

Once something breaks there's no easy way to determine what's the right
fix. It will always involve copying the migration maintainers and some
back and forth with the device people before we reach an agreement on
what's even broken.

When I say "clear recipes" what I mean is we'd have a "catalogue" of
types of failures that could happen. Those would be both documented in
plain english and also have some instrumentation in the code to produce
a clear error/message.

  E.g.: "Device 'foo' failed to migrate because of error type X: the src
  machine provided more state than the dst was expecting around the
  value Y".

And that "error type X" would come with some docs listing examples of
other similar errors and what strategies we suggest do deal with them.

Currently most migration failures are just a completely helpless:
"blergh, error -5". And the only thing we can say about it upfront is
"well, something must have changed in the stream".

Real migration failures I have seen recently (all fixed already):

1- Some feature bit was mistakenly removed from an arm cpu. Migration
   complains about a 'length' field being different.

2- A group of devices was moved from the machine init to the cpu init on
   pseries. Migration spews some nonsense about an "index".

3- Recent (invalid) bug on -cpu max on arm, a couple of bits were set in
   a register. Migration barfs incomprehensibly with: "error while
   loading state for instance 0x0 of device 'cpu', Operation not
   permitted".

So I bet we could improve these error cases to be a bit more predictable
and that would help device developers to be able to maintain migration
compatibility without making it seem like an arbitrary, hard to achieve
requirement.
(/tangent)

>
> I think Juan used to worry on what happens if someone already used an old
> version of old release, e.g., someone using 8.2.0 may not be able to
> migrate to 8.2.1 if we fix that breakage in 9.0 and backport that to 8.2.1.
> My take is that maybe that's overcomplicated, and maybe we should simply
> only maintain the latest stable version, rather than all.  In this case,
> IMHO it will be less burden if we only guarantee 8.2.1 will be working,
> e.g., when migrating from 8.1.z -> 8.2.1.  Then we should just state a
> known issue in 8.2.0 that it is broken, and both:
>
>   (1) 8.1.z -> 8.2.0, and

Fair enough.

>   (2) 8.2.0 -> 8.2.1

Do you think we may not be able to always ensure that the user can get
out of the broken version? Or do you simply think that's too much work?

I think I agree with you. It's better to have a clear statement of what
we support and make sure that works, rather than having _some_ scenarios
where the user _may_ need to shutdown the VM and _some_ where they _may_
be able to migrate out of the situation. It creates a confusing message
that I imagine would just cause people to avoid using migration
altogether.

> will expect to fail.
>
>> were we discussed an old powerpc issue. How come we can see the fix
>> today in the code but cannot tell which problem it was trying to solve?
>> That's bonkers. Ideally every type of breakage would have a mapping into
>> why it breaks and how to fix it.
>> 
>> So with testing to catch the issue early and a clear step-by-step on how
>> to identify and fix compatibility, then we could require strict
>> compatibility for every device.
>
> I don't think we can guarantee no bug there, but indeed we can do better on
> providing some test framework for device VMSDs.
>
>> 
>> >
>> > For example, any device to be used in migration-test must be forward +
>> > backward migration compatible at least, because you just added the n-1
>> > regression tests to cover both directions.  Said that, only a few devices
>> > are involved because currently our migration-test qemu cmdline is pretty
>> > simple.
>> 
>> We might want to make a distinction between migration core vs. device
>> state testing. I see n-1 testing more like migration core testing. It's
>> bad to break migration, but it's really bad to break migration for
>> everyone because we refactored something deep within migration/.
>> 
>> I also wouldn't mind if we had some simple way for device developers to
>> add migration tests that cover their code. Currently it's infeasible to
>> edit migration-test with new command lines for every device of
>> interest. Maybe we could have a little framework that takes a command
>> line and spits a migration stream? Something really self-contained,
>> behind the device's CONFIG in meson.
>
> I added one more todo:
>
> https://wiki.qemu.org/ToDo/LiveMigration#Device_migration_stream_test_framework
>
> How's that look?  Feel free to modify on your will.

Looks good.

The point about the guest behavior influence is something that Juan has
mentioned as a blocker for testing with static data. I don't think it
would be impossible to have some unit testing at the vmstate with some
artificial values, but it might be too much work to be worth it.

On Tue, Jan 30, 2024 at 10:58:24AM -0300, Fabiano Rosas wrote:
> Peter Xu <peterx@redhat.com> writes:
> 
> > On Mon, Jan 29, 2024 at 10:44:46AM -0300, Fabiano Rosas wrote:
> >> > Since we're at it, I would also like to know how you think about whether we
> >> > should still suggest people using VMSD versioning, as we know that it won't
> >> > work for backward migrations.
> >> >
> >> > My current thoughts is it is still fine, as it's easier to use, and it
> >> > should still be applicable to the cases where a strict migration semantics
> >> > are not required.  However it's hard to justify which device needs that
> >> > strictness.
> >> 
> >> I'd prefer if we kept things strict. However I don't think we can do
> >> that without having enough testing and specially, clear recipes on how
> >> to add compatibility back once it gets lost. Think of that recent thread
> >
> > If it was broken, IMHO we should just fix it and backport to stable.
> 
> (tangent)
> Sure, but I'm talking about how do we instruct device developers on
> fixing migration bugs. We cannot simply yell "regression!" and expect
> people to care.
> 
> Once something breaks there's no easy way to determine what's the right
> fix. It will always involve copying the migration maintainers and some
> back and forth with the device people before we reach an agreement on
> what's even broken.
> 
> When I say "clear recipes" what I mean is we'd have a "catalogue" of
> types of failures that could happen. Those would be both documented in
> plain english and also have some instrumentation in the code to produce
> a clear error/message.
> 
>   E.g.: "Device 'foo' failed to migrate because of error type X: the src
>   machine provided more state than the dst was expecting around the
>   value Y".
> 
> And that "error type X" would come with some docs listing examples of
> other similar errors and what strategies we suggest do deal with them.
> 
> Currently most migration failures are just a completely helpless:
> "blergh, error -5". And the only thing we can say about it upfront is
> "well, something must have changed in the stream".

Yes, IMHO it's because of the current design of VMSD isn't self describing,
then if some VMSD added new fields without boosting the version_id, what
can happen is when the destination reads that new VMSD field added it'll
assume it is the next thing to read, and that can be a completely different
VMSD for another device.  We just don't have anything to describe it.

E.g. for sending an uint32_t field, vmstate_info_uint32 only does
qemu_put_be32s(), we keep pushing data onto the wire without proper
description of that field.

We used to have some attempt describing these fields so it might be easier
at least for debugging, see:

commit 8118f0950fc77cce7873002a5021172dd6e040b5
Author: Alexander Graf <agraf@csgraf.de>
Date:   Thu Jan 22 15:01:39 2015 +0100

    migration: Append JSON description of migration stream

But that seems only for debugging.  E.g., that happens _after_ all vmstate
loaded.  So the reported error could be the same confusing, as when an
error happens it is before the JSON chunk ready.

We may be able to do better in this regard in the future, but that'll take
some thoughts and effort.

> 
> Real migration failures I have seen recently (all fixed already):
> 
> 1- Some feature bit was mistakenly removed from an arm cpu. Migration
>    complains about a 'length' field being different.
> 
> 2- A group of devices was moved from the machine init to the cpu init on
>    pseries. Migration spews some nonsense about an "index".
> 
> 3- Recent (invalid) bug on -cpu max on arm, a couple of bits were set in
>    a register. Migration barfs incomprehensibly with: "error while
>    loading state for instance 0x0 of device 'cpu', Operation not
>    permitted".
> 
> So I bet we could improve these error cases to be a bit more predictable
> and that would help device developers to be able to maintain migration
> compatibility without making it seem like an arbitrary, hard to achieve
> requirement.
> (/tangent)

Right, there can be multiple ways to fail, and we may need to look into
them one by one.  They all take quite some effort.

> 
> >
> > I think Juan used to worry on what happens if someone already used an old
> > version of old release, e.g., someone using 8.2.0 may not be able to
> > migrate to 8.2.1 if we fix that breakage in 9.0 and backport that to 8.2.1.
> > My take is that maybe that's overcomplicated, and maybe we should simply
> > only maintain the latest stable version, rather than all.  In this case,
> > IMHO it will be less burden if we only guarantee 8.2.1 will be working,
> > e.g., when migrating from 8.1.z -> 8.2.1.  Then we should just state a
> > known issue in 8.2.0 that it is broken, and both:
> >
> >   (1) 8.1.z -> 8.2.0, and
> 
> Fair enough.
> 
> >   (2) 8.2.0 -> 8.2.1
> 
> Do you think we may not be able to always ensure that the user can get
> out of the broken version? Or do you simply think that's too much work?

Yes.  Allowing such compatibility means we declare support over all
binaries we released, including buggy ones.  But a bug is a bug, and it can
happen.  Just like QEMU, as one normal piece of software, can crash when
hit a bug, and it can happen to any binary because we can't guarantee it's
bug-free.

It's just too much to me.  It's also hard if not impossible, because I
don't think it always has a solution if the src QEMU is simply broken.
When such happens, we request the user to reboot with a correct version of
QEMU.

> 
> I think I agree with you. It's better to have a clear statement of what
> we support and make sure that works, rather than having _some_ scenarios
> where the user _may_ need to shutdown the VM and _some_ where they _may_
> be able to migrate out of the situation. It creates a confusing message
> that I imagine would just cause people to avoid using migration
> altogether.
> 
> > will expect to fail.
> >
> >> were we discussed an old powerpc issue. How come we can see the fix
> >> today in the code but cannot tell which problem it was trying to solve?
> >> That's bonkers. Ideally every type of breakage would have a mapping into
> >> why it breaks and how to fix it.
> >> 
> >> So with testing to catch the issue early and a clear step-by-step on how
> >> to identify and fix compatibility, then we could require strict
> >> compatibility for every device.
> >
> > I don't think we can guarantee no bug there, but indeed we can do better on
> > providing some test framework for device VMSDs.
> >
> >> 
> >> >
> >> > For example, any device to be used in migration-test must be forward +
> >> > backward migration compatible at least, because you just added the n-1
> >> > regression tests to cover both directions.  Said that, only a few devices
> >> > are involved because currently our migration-test qemu cmdline is pretty
> >> > simple.
> >> 
> >> We might want to make a distinction between migration core vs. device
> >> state testing. I see n-1 testing more like migration core testing. It's
> >> bad to break migration, but it's really bad to break migration for
> >> everyone because we refactored something deep within migration/.
> >> 
> >> I also wouldn't mind if we had some simple way for device developers to
> >> add migration tests that cover their code. Currently it's infeasible to
> >> edit migration-test with new command lines for every device of
> >> interest. Maybe we could have a little framework that takes a command
> >> line and spits a migration stream? Something really self-contained,
> >> behind the device's CONFIG in meson.
> >
> > I added one more todo:
> >
> > https://wiki.qemu.org/ToDo/LiveMigration#Device_migration_stream_test_framework
> >
> > How's that look?  Feel free to modify on your will.
> 
> Looks good.
> 
> The point about the guest behavior influence is something that Juan has
> mentioned as a blocker for testing with static data. I don't think it
> would be impossible to have some unit testing at the vmstate with some
> artificial values, but it might be too much work to be worth it.
> 

-- 
Peter Xu

On Mon, 29 Jan 2024 at 13:45, Fabiano Rosas <farosas@suse.de> wrote:
>
> Peter Xu <peterx@redhat.com> writes:
> > Fundamentally, IMHO it's because QEMU as a project is used both in
> > enterprise and personal emulations.  I think it might be too strict to
> > always request backward migration capability if we know some device / arch
> > is only used for personal, or educational, purposes.
>
> Do we need migration support tiers? =)

We already have them. The tier list is:

 * if the machine type is a versioned one, then we maintain
   forwards compatibility for the versioned machine
   (i.e. can migrate machine-X.Y of QEMU A.B to the
    machine-X.Y of a QEMU C.D which is newer than A.B).
 * if the machine type is not versioned, then we do not make
   any guarantee of migration compatibility across QEMU versions.
   Instead the aim is that if the user tries it it either works
   or gives an error message that the migration failed
   (e.g. because the version field in a VMState struct was bumped).
   Migration breaks are generally called out in commit messages.
   Often for machines in this tier the user is really interested
   in state-save snapshots for debugging purposes, rather than
   in a true cross-host-machine migration.
 * some machine types do not support migration/savevm/loadvm
   at all, because of devices missing VMState structs. This
   is not desirable, and for new machine models we try to
   ensure that they have vmstate structs as part of the minimum
   quality bar, but it is true of some legacy machine types.

AIUI we, in the sense of the upstream project, do not support
backwards migration compatibility (i.e. migrating a machine-X.Y
from QEMU C.D to QEMU A.B where A.B is an older version than C.D);
though some downstreams (read: RedHat) may do so.

thanks
-- PMM

Peter Maydell <peter.maydell@linaro.org> writes:

> On Mon, 29 Jan 2024 at 13:45, Fabiano Rosas <farosas@suse.de> wrote:
>>
>> Peter Xu <peterx@redhat.com> writes:
>> > Fundamentally, IMHO it's because QEMU as a project is used both in
>> > enterprise and personal emulations.  I think it might be too strict to
>> > always request backward migration capability if we know some device / arch
>> > is only used for personal, or educational, purposes.
>>
>> Do we need migration support tiers? =)
>
> We already have them. The tier list is:

Ah that's good. Thanks, Peter.

>
>  * if the machine type is a versioned one, then we maintain
>    forwards compatibility for the versioned machine
>    (i.e. can migrate machine-X.Y of QEMU A.B to the
>     machine-X.Y of a QEMU C.D which is newer than A.B).
>  * if the machine type is not versioned, then we do not make
>    any guarantee of migration compatibility across QEMU versions.
>    Instead the aim is that if the user tries it it either works
>    or gives an error message that the migration failed
>    (e.g. because the version field in a VMState struct was bumped).
>    Migration breaks are generally called out in commit messages.
>    Often for machines in this tier the user is really interested
>    in state-save snapshots for debugging purposes, rather than
>    in a true cross-host-machine migration.
>  * some machine types do not support migration/savevm/loadvm
>    at all, because of devices missing VMState structs. This
>    is not desirable, and for new machine models we try to
>    ensure that they have vmstate structs as part of the minimum
>    quality bar, but it is true of some legacy machine types.

Hm, does this mean in some cases we're requiring new models to have
vmstate only to never look at them again? Or do you mean some versioned
machines are currently broken?

> AIUI we, in the sense of the upstream project, do not support
> backwards migration compatibility (i.e. migrating a machine-X.Y
> from QEMU C.D to QEMU A.B where A.B is an older version than C.D);
> though some downstreams (read: RedHat) may do so.

Here we still need to make a distinction between migration code and
vmstate. If we simply ignore backwards migration then it might become
impossible for downstreams to provide it without major
modifications. But luckily this is the easy case.

>
> thanks
> -- PMM

On Mon, 29 Jan 2024 at 15:18, Fabiano Rosas <farosas@suse.de> wrote:
>
> Peter Maydell <peter.maydell@linaro.org> writes:
>
> > On Mon, 29 Jan 2024 at 13:45, Fabiano Rosas <farosas@suse.de> wrote:
> >>
> >> Peter Xu <peterx@redhat.com> writes:
> >> > Fundamentally, IMHO it's because QEMU as a project is used both in
> >> > enterprise and personal emulations.  I think it might be too strict to
> >> > always request backward migration capability if we know some device / arch
> >> > is only used for personal, or educational, purposes.
> >>
> >> Do we need migration support tiers? =)
> >
> > We already have them. The tier list is:
>
> Ah that's good. Thanks, Peter.
>
> >
> >  * if the machine type is a versioned one, then we maintain
> >    forwards compatibility for the versioned machine
> >    (i.e. can migrate machine-X.Y of QEMU A.B to the
> >     machine-X.Y of a QEMU C.D which is newer than A.B).
> >  * if the machine type is not versioned, then we do not make
> >    any guarantee of migration compatibility across QEMU versions.
> >    Instead the aim is that if the user tries it it either works
> >    or gives an error message that the migration failed
> >    (e.g. because the version field in a VMState struct was bumped).
> >    Migration breaks are generally called out in commit messages.
> >    Often for machines in this tier the user is really interested
> >    in state-save snapshots for debugging purposes, rather than
> >    in a true cross-host-machine migration.
> >  * some machine types do not support migration/savevm/loadvm
> >    at all, because of devices missing VMState structs. This
> >    is not desirable, and for new machine models we try to
> >    ensure that they have vmstate structs as part of the minimum
> >    quality bar, but it is true of some legacy machine types.
>
> Hm, does this mean in some cases we're requiring new models to have
> vmstate only to never look at them again? Or do you mean some versioned
> machines are currently broken?

New device models have vmstate; we don't actively test that
savevm/loadvm works, but as with most device models we fix bugs
if anybody reports them. Some older device models simply omit
the vmstate struct completely (which results in the guest not
behaving right after savevm/loadvm); a few at least register a
migration blocker. Usually if somebody's doing a refactoring
and cleanup of an old device they'll add the vmstate while they're
doing it.

Any device which is used by a versioned machine type is supposed
to have the vmstate support.

> > AIUI we, in the sense of the upstream project, do not support
> > backwards migration compatibility (i.e. migrating a machine-X.Y
> > from QEMU C.D to QEMU A.B where A.B is an older version than C.D);
> > though some downstreams (read: RedHat) may do so.
>
> Here we still need to make a distinction between migration code and
> vmstate. If we simply ignore backwards migration then it might become
> impossible for downstreams to provide it without major
> modifications. But luckily this is the easy case.

Yeah, there's no reason for us to make our downstreams' lives
harder; the "not supported upstream" part is a mix of
(a) we don't test it so it probably doesn't work and
(b) we're not going to insist on patch submitters tying themselves
in knots over trying to implement a level of compatibility for
a device when we don't advertise that it's supposed to work

thanks
-- PMM

On Mon, Jan 29, 2024 at 03:51:07PM +0000, Peter Maydell wrote:
> On Mon, 29 Jan 2024 at 15:18, Fabiano Rosas <farosas@suse.de> wrote:
> >
> > Peter Maydell <peter.maydell@linaro.org> writes:
> >
> > > On Mon, 29 Jan 2024 at 13:45, Fabiano Rosas <farosas@suse.de> wrote:
> > >>
> > >> Peter Xu <peterx@redhat.com> writes:
> > >> > Fundamentally, IMHO it's because QEMU as a project is used both in
> > >> > enterprise and personal emulations.  I think it might be too strict to
> > >> > always request backward migration capability if we know some device / arch
> > >> > is only used for personal, or educational, purposes.
> > >>
> > >> Do we need migration support tiers? =)
> > >
> > > We already have them. The tier list is:
> >
> > Ah that's good. Thanks, Peter.
> >
> > >
> > >  * if the machine type is a versioned one, then we maintain
> > >    forwards compatibility for the versioned machine
> > >    (i.e. can migrate machine-X.Y of QEMU A.B to the
> > >     machine-X.Y of a QEMU C.D which is newer than A.B).
> > >  * if the machine type is not versioned, then we do not make
> > >    any guarantee of migration compatibility across QEMU versions.
> > >    Instead the aim is that if the user tries it it either works
> > >    or gives an error message that the migration failed
> > >    (e.g. because the version field in a VMState struct was bumped).
> > >    Migration breaks are generally called out in commit messages.
> > >    Often for machines in this tier the user is really interested
> > >    in state-save snapshots for debugging purposes, rather than
> > >    in a true cross-host-machine migration.
> > >  * some machine types do not support migration/savevm/loadvm
> > >    at all, because of devices missing VMState structs. This
> > >    is not desirable, and for new machine models we try to
> > >    ensure that they have vmstate structs as part of the minimum
> > >    quality bar, but it is true of some legacy machine types.
> >
> > Hm, does this mean in some cases we're requiring new models to have
> > vmstate only to never look at them again? Or do you mean some versioned
> > machines are currently broken?
> 
> New device models have vmstate; we don't actively test that
> savevm/loadvm works, but as with most device models we fix bugs
> if anybody reports them. Some older device models simply omit
> the vmstate struct completely (which results in the guest not
> behaving right after savevm/loadvm); a few at least register a
> migration blocker. Usually if somebody's doing a refactoring
> and cleanup of an old device they'll add the vmstate while they're
> doing it.
> 
> Any device which is used by a versioned machine type is supposed
> to have the vmstate support.
> 
> > > AIUI we, in the sense of the upstream project, do not support
> > > backwards migration compatibility (i.e. migrating a machine-X.Y
> > > from QEMU C.D to QEMU A.B where A.B is an older version than C.D);
> > > though some downstreams (read: RedHat) may do so.
> >
> > Here we still need to make a distinction between migration code and
> > vmstate. If we simply ignore backwards migration then it might become
> > impossible for downstreams to provide it without major
> > modifications. But luckily this is the easy case.
> 
> Yeah, there's no reason for us to make our downstreams' lives
> harder; the "not supported upstream" part is a mix of
> (a) we don't test it so it probably doesn't work and
> (b) we're not going to insist on patch submitters tying themselves
> in knots over trying to implement a level of compatibility for
> a device when we don't advertise that it's supposed to work

(b) makes sense.  I suppose that further justified what this document
wanted to make clear of, on that we should probably allow vmsd versioning
to be used, which is already better than either migration incompatible, or
even not support migration at all on the system.

But shouldn't we still have that tier to request "backward migration"?  Yes
it's not covered yet by any test, but maybe some day we can have it.
Besides, we're more discussing the goal which can apply to suggestions to
patch submitters and reviewers.  I think that's still a fair goal so that
on extremely popular devices + architectures, we still ask for bi-direction
migration capability.

Do we have a place where such tiering is documented?  Should I add one more
patch to describe it?

I think the hard thing is still how to justify which change will require
which tier: we were discussing machine type numbers, but normally IIUC many
devices can be used in multiple machine types, in that case should we pick
the strictest rule out of those machine types that such device support?
Maybe it'll be easier we just justify that during reviews.

-- 
Peter Xu