This series refactors the ram compress code.

It first removes ram.c dependencies from the core compress code, then
moves it out to its own file. Finally, on the migration destination side
the initialisation and cleanup of compress threads is moved out of ram.c
to migration.c. This allows using COLO with compress enabled.

This series is based on the following series:
https://lore.kernel.org/qemu-devel/af76761aa6978071c5b8e9b872b697db465a5520.1680457631.git.lukasstraub2@web.de/T/#t

Lukas Straub (14):
  ram.c: Let the compress threads return a CompressResult enum
  ram.c: Dont change param->block in the compress thread
  ram.c: Reset result after sending queued data
  ram.c: Do not call save_page_header() from compress threads
  ram.c: Call update_compress_thread_counts from
    compress_send_queued_data
  ram.c: Remove last ram.c dependency from the core compress code
  ram.c: Introduce whitespace (squash with next patch)
  ram.c: Move core compression code into its own file
  ram.c: Remove whitespace (squash with previous patch)
  ram.c: Move core decompression code into its own file
  ram compress: Assert that the file buffer matches the result
  ram.c: Remove unused include after moving out code
  ram-compress.c: Make target independent
  migration: Initialize and cleanup decompression in migration.c

 migration/meson.build    |   5 +-
 migration/migration.c    |   9 +
 migration/qemu-file.c    |  11 +
 migration/qemu-file.h    |   1 +
 migration/ram-compress.c | 483 ++++++++++++++++++++++++++++++++++++++
 migration/ram-compress.h |  70 ++++++
 migration/ram.c          | 490 +++------------------------------------
 7 files changed, 615 insertions(+), 454 deletions(-)
 create mode 100644 migration/ram-compress.c
 create mode 100644 migration/ram-compress.h

--
2.30.2

> -----Original Message-----
> From: qemu-devel-bounces+chen.zhang=intel.com@nongnu.org <qemu-
> devel-bounces+chen.zhang=intel.com@nongnu.org> On Behalf Of Lukas
> Straub
> Sent: Monday, April 3, 2023 1:56 AM
> To: qemu-devel <qemu-devel@nongnu.org>
> Cc: Dr. David Alan Gilbert <dgilbert@redhat.com>; Juan Quintela
> <quintela@redhat.com>; Peter Xu <peterx@redhat.com>
> Subject: [PATCH 00/14] migration/ram.c: Refactor compress code
> 
> This series refactors the ram compress code.
> 
> It first removes ram.c dependencies from the core compress code, then
> moves it out to its own file. Finally, on the migration destination side the
> initialisation and cleanup of compress threads is moved out of ram.c to
> migration.c. This allows using COLO with compress enabled.
> 

Nice to see this optimization~
Support the ram compress feature will reduce COLO checkpoint network bandwidth requirements
And may improve COLO protected VM's performance.

Thanks
Chen

> This series is based on the following series:
> https://lore.kernel.org/qemu-
> devel/af76761aa6978071c5b8e9b872b697db465a5520.1680457631.git.lukasstr
> aub2@web.de/T/#t
> 
> Lukas Straub (14):
>   ram.c: Let the compress threads return a CompressResult enum
>   ram.c: Dont change param->block in the compress thread
>   ram.c: Reset result after sending queued data
>   ram.c: Do not call save_page_header() from compress threads
>   ram.c: Call update_compress_thread_counts from
>     compress_send_queued_data
>   ram.c: Remove last ram.c dependency from the core compress code
>   ram.c: Introduce whitespace (squash with next patch)
>   ram.c: Move core compression code into its own file
>   ram.c: Remove whitespace (squash with previous patch)
>   ram.c: Move core decompression code into its own file
>   ram compress: Assert that the file buffer matches the result
>   ram.c: Remove unused include after moving out code
>   ram-compress.c: Make target independent
>   migration: Initialize and cleanup decompression in migration.c
> 
>  migration/meson.build    |   5 +-
>  migration/migration.c    |   9 +
>  migration/qemu-file.c    |  11 +
>  migration/qemu-file.h    |   1 +
>  migration/ram-compress.c | 483
> ++++++++++++++++++++++++++++++++++++++
>  migration/ram-compress.h |  70 ++++++
>  migration/ram.c          | 490 +++------------------------------------
>  7 files changed, 615 insertions(+), 454 deletions(-)  create mode 100644
> migration/ram-compress.c  create mode 100644 migration/ram-compress.h
> 
> --
> 2.30.2

On Sun, 2 Apr 2023 17:55:59 +0000
Lukas Straub <lukasstraub2@web.de> wrote:

> This series refactors the ram compress code.
> 
> It first removes ram.c dependencies from the core compress code, then
> moves it out to its own file. Finally, on the migration destination side
> the initialisation and cleanup of compress threads is moved out of ram.c
> to migration.c. This allows using COLO with compress enabled.
> 
> This series is based on the following series:
> https://lore.kernel.org/qemu-devel/af76761aa6978071c5b8e9b872b697db465a5520.1680457631.git.lukasstraub2@web.de/T/#t
> 

Hmm, I'm wondering if I should add postcopy+compress tests in this
series? Because if we start this, the test matrix really will explode.

Regards,
Lukas Straub

-- 

On Wed, Apr 05, 2023 at 02:44:13PM +0000, Lukas Straub wrote:
> On Sun, 2 Apr 2023 17:55:59 +0000
> Lukas Straub <lukasstraub2@web.de> wrote:
> 
> > This series refactors the ram compress code.
> > 
> > It first removes ram.c dependencies from the core compress code, then
> > moves it out to its own file. Finally, on the migration destination side
> > the initialisation and cleanup of compress threads is moved out of ram.c
> > to migration.c. This allows using COLO with compress enabled.
> > 
> > This series is based on the following series:
> > https://lore.kernel.org/qemu-devel/af76761aa6978071c5b8e9b872b697db465a5520.1680457631.git.lukasstraub2@web.de/T/#t
> > 
> 
> Hmm, I'm wondering if I should add postcopy+compress tests in this
> series? Because if we start this, the test matrix really will explode.

Do you perhaps mean when compress added to the existing (and already large)
postcopy test matrix?

I assume we can still add one for vanilla postcopy+compress, so it still
covers the major parts.  Or, add the compress test with the postcopy test
with most features enabled? The most comprehensive one in qtest should be:

/x86_64/migration/postcopy/preempt/recovery/tls/psk

That covers postcopy+preempt+tls, meanwhile there'll be an emulation of
network failure too during postcopy and test recovering from it.

-- 
Peter Xu