Hi

This is the migration PULL request.  It has:
- Fabiano rosas trheadinfo cleanups
- Hyman Huang dirtylimit changes
- Part of my changes
- Peter Xu documentation
- Tejus updato to migration descriptions
- Wei want improvements for postocpy and multifd setup

Please apply.

Now a not on CI, thas has been really bad.  After too many problems
with last PULLS, I decided to learn to use qemu CI.  On one hand, it
is not so difficult, even I can use it O:-)

On the other hand, the amount of problems that I got is inmense.  Some
of them dissapear when I rerun the checks, but I never know if it is
my PULL request, the CI system or the tests themselves.

So it ends going something like:

while (true); do
- git pull
- git rebase
- git push ci blah, blah
- Next day cames, and too many errors, so I rebase again

The last step takes more time than expected and not always trivial to
know how the failure is.

This (last) patch is not part of the PULL request, but I have found
that it _always_ makes gcov fail.  I had to use bisect to find where
the problem was.

https://gitlab.com/juan.quintela/qemu/-/jobs/4571878922

I could use help to know how a change in test/qtest/migration-test.c
can break block layer tests, I am all ears.

Yes, I tried several times.  It always fails on that patch.  The
passes with flying colors.

Later, Juan.

Fabiano Rosas (2):
  migration/multifd: Rename threadinfo.c functions
  migration/multifd: Protect accesses to migration_threads

Hyman Huang(黄勇) (8):
  softmmu/dirtylimit: Add parameter check for hmp "set_vcpu_dirty_limit"
  qapi/migration: Introduce x-vcpu-dirty-limit-period parameter
  qapi/migration: Introduce vcpu-dirty-limit parameters
  migration: Introduce dirty-limit capability
  migration: Refactor auto-converge capability logic
  migration: Put the detection logic before auto-converge checking
  migration: Implement dirty-limit convergence algo
  migration: Extend query-migrate to provide dirty page limit info

Juan Quintela (12):
  migration-test: Be consistent for ppc
  migration-test: Make machine_opts regular with other options
  migration-test: Create arch_opts
  migration-test: machine_opts is really arch specific
  migration.json: Don't use space before colon
  migration: skipped field is really obsolete.
  qemu-file: Rename qemu_file_transferred_ fast -> noflush
  migration: Change qemu_file_transferred to noflush
  qemu_file: Make qemu_file_is_writable() static
  qemu-file: Simplify qemu_file_shutdown()
  qemu-file: Make qemu_file_get_error_obj() static
  migration/rdma: Split qemu_fopen_rdma() into input/output functions

Peter Xu (1):
  docs/migration: Update postcopy bits

Tejus GK (1):
  migration: Update error description whenever migration fails

Wei Wang (2):
  migration: enforce multifd and postcopy preempt to be set before
    incoming
  qtest/migration-tests.c: use "-incoming defer" for postcopy tests

 docs/about/deprecated.rst      |  10 +++
 docs/devel/migration.rst       |  94 ++++++++++++++++++++---------
 qapi/migration.json            | 107 ++++++++++++++++++++++++++-------
 include/sysemu/dirtylimit.h    |   2 +
 migration/options.h            |   1 +
 migration/qemu-file.h          |  14 ++---
 migration/threadinfo.h         |   7 +--
 migration/migration-hmp-cmds.c |  26 ++++++++
 migration/migration.c          |  36 ++++++++---
 migration/multifd.c            |   4 +-
 migration/options.c            |  87 ++++++++++++++++++++++++++-
 migration/qemu-file.c          |  24 ++------
 migration/ram.c                |  59 +++++++++++++++---
 migration/rdma.c               |  39 ++++++------
 migration/savevm.c             |   6 +-
 migration/threadinfo.c         |  19 +++++-
 migration/vmstate.c            |   4 +-
 softmmu/dirtylimit.c           |  97 +++++++++++++++++++++++++++---
 tests/qtest/migration-test.c   |  48 +++++++--------
 migration/trace-events         |   1 +
 20 files changed, 520 insertions(+), 165 deletions(-)

-- 
2.40.1

On 24/07/2023 15.06, Juan Quintela wrote:
> Hi
> 
> This is the migration PULL request.

Maybe it would better to use "PULL" instead of "PATCH" in the subject?

> Now a not on CI, thas has been really bad.  After too many problems
> with last PULLS, I decided to learn to use qemu CI.  On one hand, it
> is not so difficult, even I can use it O:-)
> 
> On the other hand, the amount of problems that I got is inmense.  Some
> of them dissapear when I rerun the checks, but I never know if it is
> my PULL request, the CI system or the tests themselves.

I normally peek at https://gitlab.com/qemu-project/qemu/-/pipelines to see 
whether the problem occurred in one of the last staging CI runs already ... 
or I push the master branch to my own repo to see whether it reproduces with 
a clean state. That often helps in judging whether it's a new problem or a 
pre-existing one.

> This (last) patch is not part of the PULL request, but I have found
> that it _always_ makes gcov fail.  I had to use bisect to find where
> the problem was.
> 
> https://gitlab.com/juan.quintela/qemu/-/jobs/4571878922
> 
> I could use help to know how a change in test/qtest/migration-test.c
> can break block layer tests, I am all ears.
> 
> Yes, I tried several times.  It always fails on that patch.  The
> passes with flying colors.

Can you reproduce it locally by running "make check-block"?

The tests/qemu-iotests/tests/copy-before-write test seems to be doing some 
things with snapshots ... maybe that's related?

  Thomas

Thomas Huth <thuth@redhat.com> wrote:
> On 24/07/2023 15.06, Juan Quintela wrote:
>> Hi
>> This is the migration PULL request.
>
> Maybe it would better to use "PULL" instead of "PATCH" in the subject?

Grrrr.

Resending.  Thanks.

>> Now a not on CI, thas has been really bad.  After too many problems
>> with last PULLS, I decided to learn to use qemu CI.  On one hand, it
>> is not so difficult, even I can use it O:-)
>> On the other hand, the amount of problems that I got is inmense.
>> Some
>> of them dissapear when I rerun the checks, but I never know if it is
>> my PULL request, the CI system or the tests themselves.
>
> I normally peek at https://gitlab.com/qemu-project/qemu/-/pipelines to
> see whether the problem occurred in one of the last staging CI runs
> already ... or I push the master branch to my own repo to see whether
> it reproduces with a clean state. That often helps in judging whether
> it's a new problem or a pre-existing one.

It don't happens for master branch at the time.  It only happens with my
changes.

But the change previous to that one runs well.  That one always fails in
the block layer.  And the changes on that "series" were only for
migration-test.c, so it shouldn't break any other tests.  No other files
are touched.

Yes, in the PULL request more files are touched, but the tests I was
doing on CI there weren't.

I have no clue what gcov is adding to those tests really (I know what
gcov is, not what gcov is trying to do on that test.)

>> This (last) patch is not part of the PULL request, but I have found
>> that it _always_ makes gcov fail.  I had to use bisect to find where
>> the problem was.
>> https://gitlab.com/juan.quintela/qemu/-/jobs/4571878922
>> I could use help to know how a change in test/qtest/migration-test.c
>> can break block layer tests, I am all ears.
>> Yes, I tried several times.  It always fails on that patch.  The
>> passes with flying colors.
>
> Can you reproduce it locally by running "make check-block"?

No.  make check with all architectures under the sun works as expected.

I have learn my lesson here, and know I have to terminals open.  One
compiles x86_64 natively and test natively.

The other compiles aarch64 and test it using TCG.

(I do more tests, but that is run after each patch got reviewed and
integrated for the PULL request)

> The tests/qemu-iotests/tests/copy-before-write test seems to be doing
> some things with snapshots ... maybe that's related?

It could.  But I am not changing that.  I am only changing
migration-test.c.

As Daniel answered on list, problably it is just a race that changing
timing makes it more probable.

Later, juan.