This reverts commit cfd66f30fb0f735df06ff4220e5000290a43dad3.

The simplification of unqueue_page() introduced a bug that sometimes
breaks migration on s390x hosts. Seems like there are still pages here
that do not have their dirty bit set.
The problem is not fully understood yet, but since we are already in
the freeze for QEMU 7.1 and we need something working there, let's
revert this patch for the upcoming release. The optimization can be
redone later again in a proper way if necessary.

Buglink: https://bugzilla.redhat.com/show_bug.cgi?id=2099934
Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 migration/ram.c        | 37 ++++++++++++++++++++++++++-----------
 migration/trace-events |  3 ++-
 2 files changed, 28 insertions(+), 12 deletions(-)

diff --git a/migration/ram.c b/migration/ram.c
index b94669ba5d..dc1de9ddbc 100644
--- a/migration/ram.c
+++ b/migration/ram.c
@@ -1612,7 +1612,6 @@ static RAMBlock *unqueue_page(RAMState *rs, ram_addr_t *offset)
 {
     struct RAMSrcPageRequest *entry;
     RAMBlock *block = NULL;
-    size_t page_size;
 
     if (!postcopy_has_request(rs)) {
         return NULL;
@@ -1629,13 +1628,10 @@ static RAMBlock *unqueue_page(RAMState *rs, ram_addr_t *offset)
     entry = QSIMPLEQ_FIRST(&rs->src_page_requests);
     block = entry->rb;
     *offset = entry->offset;
-    page_size = qemu_ram_pagesize(block);
-    /* Each page request should only be multiple page size of the ramblock */
-    assert((entry->len % page_size) == 0);
 
-    if (entry->len > page_size) {
-        entry->len -= page_size;
-        entry->offset += page_size;
+    if (entry->len > TARGET_PAGE_SIZE) {
+        entry->len -= TARGET_PAGE_SIZE;
+        entry->offset += TARGET_PAGE_SIZE;
     } else {
         memory_region_unref(block->mr);
         QSIMPLEQ_REMOVE_HEAD(&rs->src_page_requests, next_req);
@@ -1643,9 +1639,6 @@ static RAMBlock *unqueue_page(RAMState *rs, ram_addr_t *offset)
         migration_consume_urgent_request();
     }
 
-    trace_unqueue_page(block->idstr, *offset,
-                       test_bit((*offset >> TARGET_PAGE_BITS), block->bmap));
-
     return block;
 }
 
@@ -2069,8 +2062,30 @@ static bool get_queued_page(RAMState *rs, PageSearchStatus *pss)
 {
     RAMBlock  *block;
     ram_addr_t offset;
+    bool dirty;
+
+    do {
+        block = unqueue_page(rs, &offset);
+        /*
+         * We're sending this page, and since it's postcopy nothing else
+         * will dirty it, and we must make sure it doesn't get sent again
+         * even if this queue request was received after the background
+         * search already sent it.
+         */
+        if (block) {
+            unsigned long page;
+
+            page = offset >> TARGET_PAGE_BITS;
+            dirty = test_bit(page, block->bmap);
+            if (!dirty) {
+                trace_get_queued_page_not_dirty(block->idstr, (uint64_t)offset,
+                                                page);
+            } else {
+                trace_get_queued_page(block->idstr, (uint64_t)offset, page);
+            }
+        }
 
-    block = unqueue_page(rs, &offset);
+    } while (block && !dirty);
 
     if (block) {
         /* See comment above postcopy_preempted_contains() */
diff --git a/migration/trace-events b/migration/trace-events
index a34afe7b85..57003edcbd 100644
--- a/migration/trace-events
+++ b/migration/trace-events
@@ -85,6 +85,8 @@ put_qlist_end(const char *field_name, const char *vmsd_name) "%s(%s)"
 qemu_file_fclose(void) ""
 
 # ram.c
+get_queued_page(const char *block_name, uint64_t tmp_offset, unsigned long page_abs) "%s/0x%" PRIx64 " page_abs=0x%lx"
+get_queued_page_not_dirty(const char *block_name, uint64_t tmp_offset, unsigned long page_abs) "%s/0x%" PRIx64 " page_abs=0x%lx"
 migration_bitmap_sync_start(void) ""
 migration_bitmap_sync_end(uint64_t dirty_pages) "dirty_pages %" PRIu64
 migration_bitmap_clear_dirty(char *str, uint64_t start, uint64_t size, unsigned long page) "rb %s start 0x%"PRIx64" size 0x%"PRIx64" page 0x%lx"
@@ -110,7 +112,6 @@ ram_save_iterate_big_wait(uint64_t milliconds, int iterations) "big wait: %" PRI
 ram_load_complete(int ret, uint64_t seq_iter) "exit_code %d seq iteration %" PRIu64
 ram_write_tracking_ramblock_start(const char *block_id, size_t page_size, void *addr, size_t length) "%s: page_size: %zu addr: %p length: %zu"
 ram_write_tracking_ramblock_stop(const char *block_id, size_t page_size, void *addr, size_t length) "%s: page_size: %zu addr: %p length: %zu"
-unqueue_page(char *block, uint64_t offset, bool dirty) "ramblock '%s' offset 0x%"PRIx64" dirty %d"
 postcopy_preempt_triggered(char *str, unsigned long page) "during sending ramblock %s offset 0x%lx"
 postcopy_preempt_restored(char *str, unsigned long page) "ramblock %s offset 0x%lx"
 postcopy_preempt_hit(char *str, uint64_t offset) "ramblock %s offset 0x%"PRIx64
-- 
2.31.1

* Thomas Huth (thuth@redhat.com) wrote:
> This reverts commit cfd66f30fb0f735df06ff4220e5000290a43dad3.
> 
> The simplification of unqueue_page() introduced a bug that sometimes
> breaks migration on s390x hosts. Seems like there are still pages here
> that do not have their dirty bit set.

I don't think it's about 'not having their dirty bit set' - it's
perfectly fine to have the bits clear (which indicates the page has
already been sent to the destination, sometime inbetween the page request
being sent from the destination and it being unqueued).
My suspicion is that either:
  * you're hitting a case where the removal of the loop
    causes it to stop when it hits a !dirty page, even though there are some
    diries left behind it.
  * We're retransmitting a page that is already marked !dirty
    which would cause overwriting of a now modified page on the
    destination.

I have no idea why either of these would be s390 specific.


> The problem is not fully understood yet, but since we are already in
> the freeze for QEMU 7.1 and we need something working there, let's
> revert this patch for the upcoming release. The optimization can be
> redone later again in a proper way if necessary.

Yeh OK

Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>

> Buglink: https://bugzilla.redhat.com/show_bug.cgi?id=2099934
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
>  migration/ram.c        | 37 ++++++++++++++++++++++++++-----------
>  migration/trace-events |  3 ++-
>  2 files changed, 28 insertions(+), 12 deletions(-)
> 
> diff --git a/migration/ram.c b/migration/ram.c
> index b94669ba5d..dc1de9ddbc 100644
> --- a/migration/ram.c
> +++ b/migration/ram.c
> @@ -1612,7 +1612,6 @@ static RAMBlock *unqueue_page(RAMState *rs, ram_addr_t *offset)
>  {
>      struct RAMSrcPageRequest *entry;
>      RAMBlock *block = NULL;
> -    size_t page_size;
>  
>      if (!postcopy_has_request(rs)) {
>          return NULL;
> @@ -1629,13 +1628,10 @@ static RAMBlock *unqueue_page(RAMState *rs, ram_addr_t *offset)
>      entry = QSIMPLEQ_FIRST(&rs->src_page_requests);
>      block = entry->rb;
>      *offset = entry->offset;
> -    page_size = qemu_ram_pagesize(block);
> -    /* Each page request should only be multiple page size of the ramblock */
> -    assert((entry->len % page_size) == 0);
>  
> -    if (entry->len > page_size) {
> -        entry->len -= page_size;
> -        entry->offset += page_size;
> +    if (entry->len > TARGET_PAGE_SIZE) {
> +        entry->len -= TARGET_PAGE_SIZE;
> +        entry->offset += TARGET_PAGE_SIZE;
>      } else {
>          memory_region_unref(block->mr);
>          QSIMPLEQ_REMOVE_HEAD(&rs->src_page_requests, next_req);
> @@ -1643,9 +1639,6 @@ static RAMBlock *unqueue_page(RAMState *rs, ram_addr_t *offset)
>          migration_consume_urgent_request();
>      }
>  
> -    trace_unqueue_page(block->idstr, *offset,
> -                       test_bit((*offset >> TARGET_PAGE_BITS), block->bmap));
> -
>      return block;
>  }
>  
> @@ -2069,8 +2062,30 @@ static bool get_queued_page(RAMState *rs, PageSearchStatus *pss)
>  {
>      RAMBlock  *block;
>      ram_addr_t offset;
> +    bool dirty;
> +
> +    do {
> +        block = unqueue_page(rs, &offset);
> +        /*
> +         * We're sending this page, and since it's postcopy nothing else
> +         * will dirty it, and we must make sure it doesn't get sent again
> +         * even if this queue request was received after the background
> +         * search already sent it.
> +         */
> +        if (block) {
> +            unsigned long page;
> +
> +            page = offset >> TARGET_PAGE_BITS;
> +            dirty = test_bit(page, block->bmap);
> +            if (!dirty) {
> +                trace_get_queued_page_not_dirty(block->idstr, (uint64_t)offset,
> +                                                page);
> +            } else {
> +                trace_get_queued_page(block->idstr, (uint64_t)offset, page);
> +            }
> +        }
>  
> -    block = unqueue_page(rs, &offset);
> +    } while (block && !dirty);
>  
>      if (block) {
>          /* See comment above postcopy_preempted_contains() */
> diff --git a/migration/trace-events b/migration/trace-events
> index a34afe7b85..57003edcbd 100644
> --- a/migration/trace-events
> +++ b/migration/trace-events
> @@ -85,6 +85,8 @@ put_qlist_end(const char *field_name, const char *vmsd_name) "%s(%s)"
>  qemu_file_fclose(void) ""
>  
>  # ram.c
> +get_queued_page(const char *block_name, uint64_t tmp_offset, unsigned long page_abs) "%s/0x%" PRIx64 " page_abs=0x%lx"
> +get_queued_page_not_dirty(const char *block_name, uint64_t tmp_offset, unsigned long page_abs) "%s/0x%" PRIx64 " page_abs=0x%lx"
>  migration_bitmap_sync_start(void) ""
>  migration_bitmap_sync_end(uint64_t dirty_pages) "dirty_pages %" PRIu64
>  migration_bitmap_clear_dirty(char *str, uint64_t start, uint64_t size, unsigned long page) "rb %s start 0x%"PRIx64" size 0x%"PRIx64" page 0x%lx"
> @@ -110,7 +112,6 @@ ram_save_iterate_big_wait(uint64_t milliconds, int iterations) "big wait: %" PRI
>  ram_load_complete(int ret, uint64_t seq_iter) "exit_code %d seq iteration %" PRIu64
>  ram_write_tracking_ramblock_start(const char *block_id, size_t page_size, void *addr, size_t length) "%s: page_size: %zu addr: %p length: %zu"
>  ram_write_tracking_ramblock_stop(const char *block_id, size_t page_size, void *addr, size_t length) "%s: page_size: %zu addr: %p length: %zu"
> -unqueue_page(char *block, uint64_t offset, bool dirty) "ramblock '%s' offset 0x%"PRIx64" dirty %d"
>  postcopy_preempt_triggered(char *str, unsigned long page) "during sending ramblock %s offset 0x%lx"
>  postcopy_preempt_restored(char *str, unsigned long page) "ramblock %s offset 0x%lx"
>  postcopy_preempt_hit(char *str, uint64_t offset) "ramblock %s offset 0x%"PRIx64
> -- 
> 2.31.1
> 
-- 
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

On 02/08/2022 10.47, Dr. David Alan Gilbert wrote:
> * Thomas Huth (thuth@redhat.com) wrote:
>> This reverts commit cfd66f30fb0f735df06ff4220e5000290a43dad3.
>>
>> The simplification of unqueue_page() introduced a bug that sometimes
>> breaks migration on s390x hosts. Seems like there are still pages here
>> that do not have their dirty bit set.
> 
> I don't think it's about 'not having their dirty bit set' - it's
> perfectly fine to have the bits clear (which indicates the page has
> already been sent to the destination, sometime inbetween the page request
> being sent from the destination and it being unqueued).

Ok, could you maybe simply drop that sentence from the commit description 
when picking up the patch? Or shall I resend a v2?

  Thomas

* Thomas Huth (thuth@redhat.com) wrote:
> On 02/08/2022 10.47, Dr. David Alan Gilbert wrote:
> > * Thomas Huth (thuth@redhat.com) wrote:
> > > This reverts commit cfd66f30fb0f735df06ff4220e5000290a43dad3.
> > > 
> > > The simplification of unqueue_page() introduced a bug that sometimes
> > > breaks migration on s390x hosts. Seems like there are still pages here
> > > that do not have their dirty bit set.
> > 
> > I don't think it's about 'not having their dirty bit set' - it's
> > perfectly fine to have the bits clear (which indicates the page has
> > already been sent to the destination, sometime inbetween the page request
> > being sent from the destination and it being unqueued).
> 
> Ok, could you maybe simply drop that sentence from the commit description
> when picking up the patch? Or shall I resend a v2?

Queued

>  Thomas
> 
> 
-- 
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

* Thomas Huth (thuth@redhat.com) wrote:
> On 02/08/2022 10.47, Dr. David Alan Gilbert wrote:
> > * Thomas Huth (thuth@redhat.com) wrote:
> > > This reverts commit cfd66f30fb0f735df06ff4220e5000290a43dad3.
> > > 
> > > The simplification of unqueue_page() introduced a bug that sometimes
> > > breaks migration on s390x hosts. Seems like there are still pages here
> > > that do not have their dirty bit set.
> > 
> > I don't think it's about 'not having their dirty bit set' - it's
> > perfectly fine to have the bits clear (which indicates the page has
> > already been sent to the destination, sometime inbetween the page request
> > being sent from the destination and it being unqueued).
> 
> Ok, could you maybe simply drop that sentence from the commit description
> when picking up the patch? Or shall I resend a v2?

Sure, I'll reword

>  Thomas
> 
-- 
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK