[Qemu-devel] [PATCH 00/59] RAMState + qdev

Juan Quintela posted 59 patches 7 years ago
Patches applied successfully (tree, apply log)
git fetch https://github.com/patchew-project/qemu tags/patchew/20170419205923.8808-1-quintela@redhat.com
Test checkpatch failed
Test docker passed
Test s390x passed
exec.c                        |   11 +-
hmp.c                         |    3 +
hw/core/qdev.c                |   40 +-
include/exec/ram_addr.h       |    8 +-
include/hw/qdev-core.h        |    3 +-
include/migration/migration.h |   41 +-
include/sysemu/sysemu.h       |    2 +-
migration/migration.c         |   44 +-
migration/postcopy-ram.c      |   14 +-
migration/ram.c               | 1210 +++++++++++++++++++++--------------------
migration/savevm.c            |   15 +-
migration/trace-events        |    4 +-
qapi-schema.json              |    5 +-
qdev-monitor.c                |   45 ++
14 files changed, 741 insertions(+), 704 deletions(-)
[Qemu-devel] [PATCH 00/59] RAMState + qdev
Posted by Juan Quintela 7 years ago
Hi

Hi

This is just v3 rebased, with review-by, and the small changes suggested.
I added the removal of hotmplug because they are already reviewed.

I plan to send this as migration/next for when 2.0 open.



- put all reviews
- rebase
- small changes on compression-switch
- add all the hotplug removal comments/typos/fixes

Please, comment.

Thanks, Juan.

[v3]
Contination of the previous series, addressed comments from Peter and Dave.

- All docs suggestions changed
- rename flush_page_queue to migration_page_queue()
- rebase on top of latest
- move the compression-switch logic
- change parameters to page_abs everywhere (notice that we end dropping them)


[v2]
Continuation of previous series, all review comments addressed. New things:
- Consolidate all function comments in the same style (yes, docs)
- Be much more careful with maintaining comments correct
- Move all postcopy fields to RAMState
- Move QEMUFile to RAMState
- rename qemu_target_page_bits() to qemu_target_page_size() to reflect use
- Remove MigrationState from functions that don't need it
- reorganize last_sent_block to the place where it is used/needed
- Move several places from offsets to pages
- Rename last_ram_offset() to last_ram_page() to refect use

Please comment.


[v1]
Currently, we have several places where we store informaticon about
ram for migration pruposes:
- global variables on migration/ram.c
- inside the accounting_info struct in migration/ram.c
  notice that not all the accounting vars are inside there
- some stuff is in MigrationState, althought it belongs to migrate/ram.c

So, this series does:
- move everything related to ram.c to RAMState struct
- make all the statistics consistent, exporting them with an accessor
  function

Why now?

Because I am trying to do some more optimizations about how we send
data around and it is basically impossible to do with current code, we
still need to add more variables.  Notice that there are things like that:
- accounting info was only reset if we had xbzrle enabled
- How/where to initialize variables are completely inconsistent.



To Do:

- There are still places that access directly the global struct.
  Mainly postcopy.  We could finfd a way to make a pointer to the
  current migration.  If people like the approach, I will search where
  to put it.
- I haven't posted any real change here, this is just the move of
  variables to the struct and pass the struct around.  Optimizations
  will came after.

- Consolidate XBZRLE, Compression params, etc in its own structs
  (inside or not RAMState, to be able to allocate ones, others, or
  ...)


[v2 no-hotplug]
This updates patches with all the comments received.
I move qdev_unplug() to make linux-user compile.

Please, review.


[RFC - v1 no-hotplug]
This series disable hotplug/unplug during migration.  Thank to Markus
for explaining where I had to put the checks.  Why?  Because during
migration we will fail if there are changes.  For instance, in
postcopy, if we add a memory region, we would failing.  Same for other
devices if they are not setup exactly the same on destination.

Iidea would be to disable it, andthen enable for the thing that we know that work.

This series are on top of my previous RAMState v2 serie.

Commets, please?

Thanks, Juan.


Comments, please.


Chao Fan (1):
  ram: Add page-size to output in 'info migrate'

Juan Quintela (58):
  ram: Update all functions comments
  ram: Rename flush_page_queue() to migration_page_queue_free()
  ram: Rename block_name to rbname
  ram: Create RAMState
  ram: Add dirty_rate_high_cnt to RAMState
  ram: Move bitmap_sync_count into RAMState
  ram: Move start time into RAMState
  ram: Move bytes_xfer_prev into RAMState
  ram: Change byte_xfer_{prev,now} type to uint64_t
  ram: Move num_dirty_pages_period into RAMState
  ram: Change num_dirty_pages_period type to uint64_t
  ram: Move xbzrle_cache_miss_prev into RAMState
  ram: Move iterations_prev into RAMState
  ram: Move dup_pages into RAMState
  ram: Remove unused dup_mig_bytes_transferred()
  ram: Remove unused pages_skipped variable
  ram: Move norm_pages to RAMState
  ram: Remove norm_mig_bytes_transferred
  ram: Move iterations into RAMState
  ram: Move xbzrle_bytes into RAMState
  ram: Move xbzrle_pages into RAMState
  ram: Move xbzrle_cache_miss into RAMState
  ram: Move xbzrle_cache_miss_rate into RAMState
  ram: Move xbzrle_overflows into RAMState
  ram: Move migration_dirty_pages to RAMState
  ram: Everything was init to zero, so use memset
  ram: Move migration_bitmap_mutex into RAMState
  ram: Move migration_bitmap_rcu into RAMState
  ram: Move bytes_transferred into RAMState
  ram: Use the RAMState bytes_transferred parameter
  ram: Remove ram_save_remaining
  ram: Move last_req_rb to RAMState
  ram: Move src_page_req* to RAMState
  ram: Create ram_dirty_sync_count()
  ram: Remove dirty_bytes_rate
  ram: Move dirty_pages_rate to RAMState
  ram: Move postcopy_requests into RAMState
  ram: Add QEMUFile to RAMState
  ram: Move QEMUFile into RAMState
  ram: Remove compression_switch and inline its logic
  migration: Remove MigrationState from migration_in_postcopy
  ram: We don't need MigrationState parameter anymore
  ram: Rename qemu_target_page_bits() to qemu_target_page_size()
  ram: Pass RAMBlock to bitmap_sync
  ram: ram_discard_range() don't use the mis parameter
  ram: reorganize last_sent_block
  ram: Use page number instead of an address for the bitmap operations
  ram: Remember last_page instead of last_offset
  ram: Change offset field in PageSearchStatus to page
  ram: Use ramblock and page offset instead of absolute offset
  ram: rename last_ram_offset() last_ram_pages()
  ram: Use RAMBitmap type for coherence
  migration: Remove MigrationState parameter from migration_is_idle()
  qdev: qdev_hotplug is really a bool
  qdev: Export qdev_hot_removed
  qdev: Move qdev_unplug() to qdev-monitor.c
  migration: Disable hotplug/unplug during migration
  ram: Remove migration_bitmap_extend()

 exec.c                        |   11 +-
 hmp.c                         |    3 +
 hw/core/qdev.c                |   40 +-
 include/exec/ram_addr.h       |    8 +-
 include/hw/qdev-core.h        |    3 +-
 include/migration/migration.h |   41 +-
 include/sysemu/sysemu.h       |    2 +-
 migration/migration.c         |   44 +-
 migration/postcopy-ram.c      |   14 +-
 migration/ram.c               | 1210 +++++++++++++++++++++--------------------
 migration/savevm.c            |   15 +-
 migration/trace-events        |    4 +-
 qapi-schema.json              |    5 +-
 qdev-monitor.c                |   45 ++
 14 files changed, 741 insertions(+), 704 deletions(-)

-- 
2.9.3


Re: [Qemu-devel] [PATCH 00/59] RAMState + qdev
Posted by no-reply@patchew.org 7 years ago
Hi,

This series seems to have some coding style problems. See output below for
more information:

Type: series
Message-id: 20170419205923.8808-1-quintela@redhat.com
Subject: [Qemu-devel] [PATCH 00/59] RAMState + qdev

=== TEST SCRIPT BEGIN ===
#!/bin/bash

BASE=base
n=1
total=$(git log --oneline $BASE.. | wc -l)
failed=0

# Useful git options
git config --local diff.renamelimit 0
git config --local diff.renames True

commits="$(git log --format=%H --reverse $BASE..)"
for c in $commits; do
    echo "Checking PATCH $n/$total: $(git log -n 1 --format=%s $c)..."
    if ! git show $c --format=email | ./scripts/checkpatch.pl --mailback -; then
        failed=1
        echo
    fi
    n=$((n+1))
done

exit $failed
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
From https://github.com/patchew-project/qemu
 * [new tag]         patchew/20170419205923.8808-1-quintela@redhat.com -> patchew/20170419205923.8808-1-quintela@redhat.com
Switched to a new branch 'test'
6d30711 ram: Remove migration_bitmap_extend()
123c47a migration: Disable hotplug/unplug during migration
df49a84 qdev: Move qdev_unplug() to qdev-monitor.c
9c87aac qdev: Export qdev_hot_removed
8d1cce2 qdev: qdev_hotplug is really a bool
ea0729a migration: Remove MigrationState parameter from migration_is_idle()
e13e7c0 ram: Use RAMBitmap type for coherence
3178a8b ram: rename last_ram_offset() last_ram_pages()
0954f82 ram: Use ramblock and page offset instead of absolute offset
f097dac ram: Change offset field in PageSearchStatus to page
5cd2c21 ram: Remember last_page instead of last_offset
bd26e98 ram: Use page number instead of an address for the bitmap operations
32451bd ram: reorganize last_sent_block
7f5e04a ram: ram_discard_range() don't use the mis parameter
801f4ba ram: Pass RAMBlock to bitmap_sync
22fd5ab ram: Add page-size to output in 'info migrate'
db3f4d1 ram: Rename qemu_target_page_bits() to qemu_target_page_size()
7bf9556 ram: We don't need MigrationState parameter anymore
700c985 migration: Remove MigrationState from migration_in_postcopy
f029ef9 ram: Remove compression_switch and inline its logic
c646ad4 ram: Move QEMUFile into RAMState
939622e ram: Add QEMUFile to RAMState
6f20a3d ram: Move postcopy_requests into RAMState
048ab66 ram: Move dirty_pages_rate to RAMState
b0d70d8 ram: Remove dirty_bytes_rate
3f41071 ram: Create ram_dirty_sync_count()
377a2ed ram: Move src_page_req* to RAMState
89103e7 ram: Move last_req_rb to RAMState
7801c98 ram: Remove ram_save_remaining
8043ead ram: Use the RAMState bytes_transferred parameter
09ee03e ram: Move bytes_transferred into RAMState
6a9cf97 ram: Move migration_bitmap_rcu into RAMState
ea114cd ram: Move migration_bitmap_mutex into RAMState
3d75400 ram: Everything was init to zero, so use memset
3e21ead ram: Move migration_dirty_pages to RAMState
cda540b ram: Move xbzrle_overflows into RAMState
dc059c2 ram: Move xbzrle_cache_miss_rate into RAMState
293ce6f ram: Move xbzrle_cache_miss into RAMState
6502c29 ram: Move xbzrle_pages into RAMState
e4f0f58 ram: Move xbzrle_bytes into RAMState
98fe82c ram: Move iterations into RAMState
34be799 ram: Remove norm_mig_bytes_transferred
8e345bc ram: Move norm_pages to RAMState
a4ffc19 ram: Remove unused pages_skipped variable
8953f8b ram: Remove unused dup_mig_bytes_transferred()
280281f ram: Move dup_pages into RAMState
b3ccc2e ram: Move iterations_prev into RAMState
bdf3604 ram: Move xbzrle_cache_miss_prev into RAMState
1d3322b ram: Change num_dirty_pages_period type to uint64_t
a6254ab ram: Move num_dirty_pages_period into RAMState
aacfe07 ram: Change byte_xfer_{prev, now} type to uint64_t
49f2369 ram: Move bytes_xfer_prev into RAMState
9bb96a0 ram: Move start time into RAMState
e0c33c4 ram: Move bitmap_sync_count into RAMState
ffb4fda ram: Add dirty_rate_high_cnt to RAMState
e2435c3 ram: Create RAMState
8392453 ram: Rename block_name to rbname
f68407c ram: Rename flush_page_queue() to migration_page_queue_free()
c12f2ff ram: Update all functions comments

=== OUTPUT BEGIN ===
Checking PATCH 1/59: ram: Update all functions comments...
Checking PATCH 2/59: ram: Rename flush_page_queue() to migration_page_queue_free()...
Checking PATCH 3/59: ram: Rename block_name to rbname...
Checking PATCH 4/59: ram: Create RAMState...
Checking PATCH 5/59: ram: Add dirty_rate_high_cnt to RAMState...
Checking PATCH 6/59: ram: Move bitmap_sync_count into RAMState...
Checking PATCH 7/59: ram: Move start time into RAMState...
Checking PATCH 8/59: ram: Move bytes_xfer_prev into RAMState...
Checking PATCH 9/59: ram: Change byte_xfer_{prev, now} type to uint64_t...
Checking PATCH 10/59: ram: Move num_dirty_pages_period into RAMState...
Checking PATCH 11/59: ram: Change num_dirty_pages_period type to uint64_t...
Checking PATCH 12/59: ram: Move xbzrle_cache_miss_prev into RAMState...
Checking PATCH 13/59: ram: Move iterations_prev into RAMState...
Checking PATCH 14/59: ram: Move dup_pages into RAMState...
Checking PATCH 15/59: ram: Remove unused dup_mig_bytes_transferred()...
Checking PATCH 16/59: ram: Remove unused pages_skipped variable...
Checking PATCH 17/59: ram: Move norm_pages to RAMState...
Checking PATCH 18/59: ram: Remove norm_mig_bytes_transferred...
Checking PATCH 19/59: ram: Move iterations into RAMState...
Checking PATCH 20/59: ram: Move xbzrle_bytes into RAMState...
Checking PATCH 21/59: ram: Move xbzrle_pages into RAMState...
Checking PATCH 22/59: ram: Move xbzrle_cache_miss into RAMState...
Checking PATCH 23/59: ram: Move xbzrle_cache_miss_rate into RAMState...
Checking PATCH 24/59: ram: Move xbzrle_overflows into RAMState...
Checking PATCH 25/59: ram: Move migration_dirty_pages to RAMState...
Checking PATCH 26/59: ram: Everything was init to zero, so use memset...
Checking PATCH 27/59: ram: Move migration_bitmap_mutex into RAMState...
Checking PATCH 28/59: ram: Move migration_bitmap_rcu into RAMState...
Checking PATCH 29/59: ram: Move bytes_transferred into RAMState...
Checking PATCH 30/59: ram: Use the RAMState bytes_transferred parameter...
Checking PATCH 31/59: ram: Remove ram_save_remaining...
Checking PATCH 32/59: ram: Move last_req_rb to RAMState...
Checking PATCH 33/59: ram: Move src_page_req* to RAMState...
Checking PATCH 34/59: ram: Create ram_dirty_sync_count()...
Checking PATCH 35/59: ram: Remove dirty_bytes_rate...
Checking PATCH 36/59: ram: Move dirty_pages_rate to RAMState...
Checking PATCH 37/59: ram: Move postcopy_requests into RAMState...
Checking PATCH 38/59: ram: Add QEMUFile to RAMState...
Checking PATCH 39/59: ram: Move QEMUFile into RAMState...
Checking PATCH 40/59: ram: Remove compression_switch and inline its logic...
Checking PATCH 41/59: migration: Remove MigrationState from migration_in_postcopy...
Checking PATCH 42/59: ram: We don't need MigrationState parameter anymore...
Checking PATCH 43/59: ram: Rename qemu_target_page_bits() to qemu_target_page_size()...
Checking PATCH 44/59: ram: Add page-size to output in 'info migrate'...
Checking PATCH 45/59: ram: Pass RAMBlock to bitmap_sync...
Checking PATCH 46/59: ram: ram_discard_range() don't use the mis parameter...
Checking PATCH 47/59: ram: reorganize last_sent_block...
Checking PATCH 48/59: ram: Use page number instead of an address for the bitmap operations...
Checking PATCH 49/59: ram: Remember last_page instead of last_offset...
Checking PATCH 50/59: ram: Change offset field in PageSearchStatus to page...
Checking PATCH 51/59: ram: Use ramblock and page offset instead of absolute offset...
Checking PATCH 52/59: ram: rename last_ram_offset() last_ram_pages()...
Checking PATCH 53/59: ram: Use RAMBitmap type for coherence...
Checking PATCH 54/59: migration: Remove MigrationState parameter from migration_is_idle()...
Checking PATCH 55/59: qdev: qdev_hotplug is really a bool...
ERROR: do not initialise globals to 0 or NULL
#24: FILE: hw/core/qdev.c:42:
+bool qdev_hotplug = false;

total: 1 errors, 0 warnings, 24 lines checked

Your patch has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.

Checking PATCH 56/59: qdev: Export qdev_hot_removed...
ERROR: do not initialise globals to 0 or NULL
#23: FILE: hw/core/qdev.c:44:
+bool qdev_hot_removed = false;

total: 1 errors, 0 warnings, 15 lines checked

Your patch has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.

Checking PATCH 57/59: qdev: Move qdev_unplug() to qdev-monitor.c...
Checking PATCH 58/59: migration: Disable hotplug/unplug during migration...
Checking PATCH 59/59: ram: Remove migration_bitmap_extend()...
=== OUTPUT END ===

Test command exited with code: 1


---
Email generated automatically by Patchew [http://patchew.org/].
Please send your feedback to patchew-devel@freelists.org