[Qemu-devel] [PATCH v3] migration: do not rom_reset() during incoming migration

Catherine Ho posted 1 patch 4 years, 11 months ago
Test docker-mingw@fedora passed
Test docker-clang@ubuntu passed
Test checkpatch passed
Test asan passed
Patches applied successfully (tree, apply log)
git fetch https://github.com/patchew-project/qemu tags/patchew/1554688616-18583-1-git-send-email-catherine.hecx@gmail.com
There is a newer version of this series
hw/core/loader.c | 7 +++++++
1 file changed, 7 insertions(+)
[Qemu-devel] [PATCH v3] migration: do not rom_reset() during incoming migration
Posted by Catherine Ho 4 years, 11 months ago
Commit 18269069c310 ("migration: Introduce ignore-shared capability")
addes ignore-shared capability to bypass the shared ramblock (e,g,
membackend + numa node). It does good to live migration.

As told by Yury,this commit expectes that QEMU doesn't write to guest RAM
until VM starts, but it does on aarch64 qemu:
Backtrace:
1  0x000055f4a296dd84 in address_space_write_rom_internal () at
exec.c:3458
2  0x000055f4a296de3a in address_space_write_rom () at exec.c:3479
3  0x000055f4a2d519ff in rom_reset () at hw/core/loader.c:1101
4  0x000055f4a2d475ec in qemu_devices_reset () at hw/core/reset.c:69
5  0x000055f4a2c90a28 in qemu_system_reset () at vl.c:1675
6  0x000055f4a2c9851d in main () at vl.c:4552

Actually, on arm64 virt marchine, ramblock "dtb" will be filled into ram
druing rom_reset. In ignore-shared incoming case, this rom filling
is not required since all the data has been stored in memory backend
file.

Further more, as suggested by Peter Xu, if we do rom_reset() now with
these ROMs then the RAM data should be re-filled again too with the
migration stream coming in.

Fixes: commit 18269069c310 ("migration: Introduce ignore-shared
capability")
Suggested-by: Yury Kotov <yury-kotov@yandex-team.ru>
Suggested-by: Peter Xu <peterx@redhat.com>
Signed-off-by: Catherine Ho <catherine.hecx@gmail.com>
---
 hw/core/loader.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/hw/core/loader.c b/hw/core/loader.c
index fe5cb24122..946bb8ff00 100644
--- a/hw/core/loader.c
+++ b/hw/core/loader.c
@@ -1087,6 +1087,13 @@ static void rom_reset(void *unused)
 {
     Rom *rom;
 
+    /*
+     * If we do rom_reset() now with these ROMs then the RAM data should be
+     * re-filled again too with the migration stream coming in.
+     */
+    if (runstate_check(RUN_STATE_INMIGRATE))
+        return;
+
     QTAILQ_FOREACH(rom, &roms, next) {
         if (rom->fw_file) {
             continue;
-- 
2.17.1


Re: [Qemu-devel] [PATCH v3] migration: do not rom_reset() during incoming migration
Posted by Peter Xu 4 years, 11 months ago
On Sun, Apr 07, 2019 at 09:56:56PM -0400, Catherine Ho wrote:
> Commit 18269069c310 ("migration: Introduce ignore-shared capability")
> addes ignore-shared capability to bypass the shared ramblock (e,g,
> membackend + numa node). It does good to live migration.
> 
> As told by Yury,this commit expectes that QEMU doesn't write to guest RAM
> until VM starts, but it does on aarch64 qemu:
> Backtrace:
> 1  0x000055f4a296dd84 in address_space_write_rom_internal () at
> exec.c:3458
> 2  0x000055f4a296de3a in address_space_write_rom () at exec.c:3479
> 3  0x000055f4a2d519ff in rom_reset () at hw/core/loader.c:1101
> 4  0x000055f4a2d475ec in qemu_devices_reset () at hw/core/reset.c:69
> 5  0x000055f4a2c90a28 in qemu_system_reset () at vl.c:1675
> 6  0x000055f4a2c9851d in main () at vl.c:4552
> 
> Actually, on arm64 virt marchine, ramblock "dtb" will be filled into ram
> druing rom_reset. In ignore-shared incoming case, this rom filling
> is not required since all the data has been stored in memory backend
> file.
> 
> Further more, as suggested by Peter Xu, if we do rom_reset() now with
> these ROMs then the RAM data should be re-filled again too with the
> migration stream coming in.
> 
> Fixes: commit 18269069c310 ("migration: Introduce ignore-shared
> capability")
> Suggested-by: Yury Kotov <yury-kotov@yandex-team.ru>
> Suggested-by: Peter Xu <peterx@redhat.com>
> Signed-off-by: Catherine Ho <catherine.hecx@gmail.com>
> ---
>  hw/core/loader.c | 7 +++++++
>  1 file changed, 7 insertions(+)
> 
> diff --git a/hw/core/loader.c b/hw/core/loader.c
> index fe5cb24122..946bb8ff00 100644
> --- a/hw/core/loader.c
> +++ b/hw/core/loader.c
> @@ -1087,6 +1087,13 @@ static void rom_reset(void *unused)
>  {
>      Rom *rom;
>  
> +    /*
> +     * If we do rom_reset() now with these ROMs then the RAM data should be
> +     * re-filled again too with the migration stream coming in.

I'm unconfident about correcting English in patches, but it does look
a bit odd to me...  I would say:

  We don't need to fill in the RAM with ROM data because we'll fill
  the data in during the next incoming migration in all cases.  Note
  that some of those RAMs can actually be modified by the guest on ARM
  so this is probably the only right thing to do here.

> +     */
> +    if (runstate_check(RUN_STATE_INMIGRATE))
> +        return;

The change looks good to me but of course it'll be nicer if some other
people can review it.

> +
>      QTAILQ_FOREACH(rom, &roms, next) {
>          if (rom->fw_file) {
>              continue;
> -- 
> 2.17.1
> 

Regards,

-- 
Peter Xu

[Qemu-devel] [PATCH v4] migration: do not rom_reset() during incoming migration
Posted by Catherine Ho 4 years, 11 months ago
Commit 18269069c310 ("migration: Introduce ignore-shared capability")
addes ignore-shared capability to bypass the shared ramblock (e,g,
membackend + numa node). It does good to live migration.

As told by Yury,this commit expectes that QEMU doesn't write to guest RAM
until VM starts, but it does on aarch64 qemu:
Backtrace:
1  0x000055f4a296dd84 in address_space_write_rom_internal () at
exec.c:3458
2  0x000055f4a296de3a in address_space_write_rom () at exec.c:3479
3  0x000055f4a2d519ff in rom_reset () at hw/core/loader.c:1101
4  0x000055f4a2d475ec in qemu_devices_reset () at hw/core/reset.c:69
5  0x000055f4a2c90a28 in qemu_system_reset () at vl.c:1675
6  0x000055f4a2c9851d in main () at vl.c:4552

Actually, on arm64 virt marchine, ramblock "dtb" will be filled into ram
druing rom_reset. In ignore-shared incoming case, this rom filling
is not required since all the data has been stored in memory backend
file.

Further more, as suggested by Peter Xu, if we do rom_reset() now with
these ROMs then the RAM data should be re-filled again too with the
migration stream coming in.

Fixes: commit 18269069c310 ("migration: Introduce ignore-shared
capability")
Suggested-by: Yury Kotov <yury-kotov@yandex-team.ru>
Suggested-by: Peter Xu <peterx@redhat.com>
Signed-off-by: Catherine Ho <catherine.hecx@gmail.com>
---
 hw/core/loader.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/hw/core/loader.c b/hw/core/loader.c
index fe5cb24122..040109464b 100644
--- a/hw/core/loader.c
+++ b/hw/core/loader.c
@@ -1087,6 +1087,15 @@ static void rom_reset(void *unused)
 {
     Rom *rom;
 
+    /*
+     * We don't need to fill in the RAM with ROM data because we'll fill
+     * the data in during the next incoming migration in all cases.  Note
+     * that some of those RAMs can actually be modified by the guest on ARM
+     * so this is probably the only right thing to do here.
+     */
+    if (runstate_check(RUN_STATE_INMIGRATE))
+        return;
+
     QTAILQ_FOREACH(rom, &roms, next) {
         if (rom->fw_file) {
             continue;
-- 
2.17.1


Re: [Qemu-devel] [PATCH v4] migration: do not rom_reset() during incoming migration
Posted by Catherine Ho 4 years, 11 months ago
Ping, thanks

B.R.
Catherine

On Mon, 8 Apr 2019 at 16:43, Catherine Ho <catherine.hecx@gmail.com> wrote:

> Commit 18269069c310 ("migration: Introduce ignore-shared capability")
> addes ignore-shared capability to bypass the shared ramblock (e,g,
> membackend + numa node). It does good to live migration.
>
> As told by Yury,this commit expectes that QEMU doesn't write to guest RAM
> until VM starts, but it does on aarch64 qemu:
> Backtrace:
> 1  0x000055f4a296dd84 in address_space_write_rom_internal () at
> exec.c:3458
> 2  0x000055f4a296de3a in address_space_write_rom () at exec.c:3479
> 3  0x000055f4a2d519ff in rom_reset () at hw/core/loader.c:1101
> 4  0x000055f4a2d475ec in qemu_devices_reset () at hw/core/reset.c:69
> 5  0x000055f4a2c90a28 in qemu_system_reset () at vl.c:1675
> 6  0x000055f4a2c9851d in main () at vl.c:4552
>
> Actually, on arm64 virt marchine, ramblock "dtb" will be filled into ram
> druing rom_reset. In ignore-shared incoming case, this rom filling
> is not required since all the data has been stored in memory backend
> file.
>
> Further more, as suggested by Peter Xu, if we do rom_reset() now with
> these ROMs then the RAM data should be re-filled again too with the
> migration stream coming in.
>
> Fixes: commit 18269069c310 ("migration: Introduce ignore-shared
> capability")
> Suggested-by: Yury Kotov <yury-kotov@yandex-team.ru>
> Suggested-by: Peter Xu <peterx@redhat.com>
> Signed-off-by: Catherine Ho <catherine.hecx@gmail.com>
> ---
>  hw/core/loader.c | 9 +++++++++
>  1 file changed, 9 insertions(+)
>
> diff --git a/hw/core/loader.c b/hw/core/loader.c
> index fe5cb24122..040109464b 100644
> --- a/hw/core/loader.c
> +++ b/hw/core/loader.c
> @@ -1087,6 +1087,15 @@ static void rom_reset(void *unused)
>  {
>      Rom *rom;
>
> +    /*
> +     * We don't need to fill in the RAM with ROM data because we'll fill
> +     * the data in during the next incoming migration in all cases.  Note
> +     * that some of those RAMs can actually be modified by the guest on
> ARM
> +     * so this is probably the only right thing to do here.
> +     */
> +    if (runstate_check(RUN_STATE_INMIGRATE))
> +        return;
> +
>      QTAILQ_FOREACH(rom, &roms, next) {
>          if (rom->fw_file) {
>              continue;
> --
> 2.17.1
>
>
Re: [Qemu-devel] [PATCH v4] migration: do not rom_reset() during incoming migration
Posted by Dr. David Alan Gilbert 4 years, 9 months ago
Paolo, can you take this one please.

* Catherine Ho (catherine.hecx@gmail.com) wrote:
> Commit 18269069c310 ("migration: Introduce ignore-shared capability")
> addes ignore-shared capability to bypass the shared ramblock (e,g,
> membackend + numa node). It does good to live migration.
> 
> As told by Yury,this commit expectes that QEMU doesn't write to guest RAM
> until VM starts, but it does on aarch64 qemu:
> Backtrace:
> 1  0x000055f4a296dd84 in address_space_write_rom_internal () at
> exec.c:3458
> 2  0x000055f4a296de3a in address_space_write_rom () at exec.c:3479
> 3  0x000055f4a2d519ff in rom_reset () at hw/core/loader.c:1101
> 4  0x000055f4a2d475ec in qemu_devices_reset () at hw/core/reset.c:69
> 5  0x000055f4a2c90a28 in qemu_system_reset () at vl.c:1675
> 6  0x000055f4a2c9851d in main () at vl.c:4552
> 
> Actually, on arm64 virt marchine, ramblock "dtb" will be filled into ram
> druing rom_reset. In ignore-shared incoming case, this rom filling
> is not required since all the data has been stored in memory backend
> file.
> 
> Further more, as suggested by Peter Xu, if we do rom_reset() now with
> these ROMs then the RAM data should be re-filled again too with the
> migration stream coming in.
> 
> Fixes: commit 18269069c310 ("migration: Introduce ignore-shared
> capability")
> Suggested-by: Yury Kotov <yury-kotov@yandex-team.ru>
> Suggested-by: Peter Xu <peterx@redhat.com>
> Signed-off-by: Catherine Ho <catherine.hecx@gmail.com>
> ---
>  hw/core/loader.c | 9 +++++++++
>  1 file changed, 9 insertions(+)
> 
> diff --git a/hw/core/loader.c b/hw/core/loader.c
> index fe5cb24122..040109464b 100644
> --- a/hw/core/loader.c
> +++ b/hw/core/loader.c
> @@ -1087,6 +1087,15 @@ static void rom_reset(void *unused)
>  {
>      Rom *rom;
>  
> +    /*
> +     * We don't need to fill in the RAM with ROM data because we'll fill
> +     * the data in during the next incoming migration in all cases.  Note
> +     * that some of those RAMs can actually be modified by the guest on ARM
> +     * so this is probably the only right thing to do here.
> +     */
> +    if (runstate_check(RUN_STATE_INMIGRATE))
> +        return;
> +
>      QTAILQ_FOREACH(rom, &roms, next) {
>          if (rom->fw_file) {
>              continue;
> -- 
> 2.17.1
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] [PATCH v4] migration: do not rom_reset() during incoming migration
Posted by Catherine Ho 4 years, 7 months ago
Hi Paolo
Ping, is any other comment I hadn't addressed?

Cheers
Catherine

On Thu, 6 Jun 2019 at 02:31, Dr. David Alan Gilbert <dgilbert@redhat.com>
wrote:

> Paolo, can you take this one please.
>
> * Catherine Ho (catherine.hecx@gmail.com) wrote:
> > Commit 18269069c310 ("migration: Introduce ignore-shared capability")
> > addes ignore-shared capability to bypass the shared ramblock (e,g,
> > membackend + numa node). It does good to live migration.
> >
> > As told by Yury,this commit expectes that QEMU doesn't write to guest RAM
> > until VM starts, but it does on aarch64 qemu:
> > Backtrace:
> > 1  0x000055f4a296dd84 in address_space_write_rom_internal () at
> > exec.c:3458
> > 2  0x000055f4a296de3a in address_space_write_rom () at exec.c:3479
> > 3  0x000055f4a2d519ff in rom_reset () at hw/core/loader.c:1101
> > 4  0x000055f4a2d475ec in qemu_devices_reset () at hw/core/reset.c:69
> > 5  0x000055f4a2c90a28 in qemu_system_reset () at vl.c:1675
> > 6  0x000055f4a2c9851d in main () at vl.c:4552
> >
> > Actually, on arm64 virt marchine, ramblock "dtb" will be filled into ram
> > druing rom_reset. In ignore-shared incoming case, this rom filling
> > is not required since all the data has been stored in memory backend
> > file.
> >
> > Further more, as suggested by Peter Xu, if we do rom_reset() now with
> > these ROMs then the RAM data should be re-filled again too with the
> > migration stream coming in.
> >
> > Fixes: commit 18269069c310 ("migration: Introduce ignore-shared
> > capability")
> > Suggested-by: Yury Kotov <yury-kotov@yandex-team.ru>
> > Suggested-by: Peter Xu <peterx@redhat.com>
> > Signed-off-by: Catherine Ho <catherine.hecx@gmail.com>
> > ---
> >  hw/core/loader.c | 9 +++++++++
> >  1 file changed, 9 insertions(+)
> >
> > diff --git a/hw/core/loader.c b/hw/core/loader.c
> > index fe5cb24122..040109464b 100644
> > --- a/hw/core/loader.c
> > +++ b/hw/core/loader.c
> > @@ -1087,6 +1087,15 @@ static void rom_reset(void *unused)
> >  {
> >      Rom *rom;
> >
> > +    /*
> > +     * We don't need to fill in the RAM with ROM data because we'll fill
> > +     * the data in during the next incoming migration in all cases.
> Note
> > +     * that some of those RAMs can actually be modified by the guest on
> ARM
> > +     * so this is probably the only right thing to do here.
> > +     */
> > +    if (runstate_check(RUN_STATE_INMIGRATE))
> > +        return;
> > +
> >      QTAILQ_FOREACH(rom, &roms, next) {
> >          if (rom->fw_file) {
> >              continue;
> > --
> > 2.17.1
> >
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>
Re: [Qemu-devel] [PATCH v4] migration: do not rom_reset() during incoming migration
Posted by Paolo Bonzini 4 years, 7 months ago
On 14/08/19 12:40, Catherine Ho wrote:
> Hi Paolo
> Ping, is any other comment I hadn't addressed?

No, I queued the patch now.

Paolo

> Cheers
> Catherine
> 
> On Thu, 6 Jun 2019 at 02:31, Dr. David Alan Gilbert <dgilbert@redhat.com
> <mailto:dgilbert@redhat.com>> wrote:
> 
>     Paolo, can you take this one please.
> 
>     * Catherine Ho (catherine.hecx@gmail.com
>     <mailto:catherine.hecx@gmail.com>) wrote:
>     > Commit 18269069c310 ("migration: Introduce ignore-shared capability")
>     > addes ignore-shared capability to bypass the shared ramblock (e,g,
>     > membackend + numa node). It does good to live migration.
>     >
>     > As told by Yury,this commit expectes that QEMU doesn't write to
>     guest RAM
>     > until VM starts, but it does on aarch64 qemu:
>     > Backtrace:
>     > 1  0x000055f4a296dd84 in address_space_write_rom_internal () at
>     > exec.c:3458
>     > 2  0x000055f4a296de3a in address_space_write_rom () at exec.c:3479
>     > 3  0x000055f4a2d519ff in rom_reset () at hw/core/loader.c:1101
>     > 4  0x000055f4a2d475ec in qemu_devices_reset () at hw/core/reset.c:69
>     > 5  0x000055f4a2c90a28 in qemu_system_reset () at vl.c:1675
>     > 6  0x000055f4a2c9851d in main () at vl.c:4552
>     >
>     > Actually, on arm64 virt marchine, ramblock "dtb" will be filled
>     into ram
>     > druing rom_reset. In ignore-shared incoming case, this rom filling
>     > is not required since all the data has been stored in memory backend
>     > file.
>     >
>     > Further more, as suggested by Peter Xu, if we do rom_reset() now with
>     > these ROMs then the RAM data should be re-filled again too with the
>     > migration stream coming in.
>     >
>     > Fixes: commit 18269069c310 ("migration: Introduce ignore-shared
>     > capability")
>     > Suggested-by: Yury Kotov <yury-kotov@yandex-team.ru
>     <mailto:yury-kotov@yandex-team.ru>>
>     > Suggested-by: Peter Xu <peterx@redhat.com <mailto:peterx@redhat.com>>
>     > Signed-off-by: Catherine Ho <catherine.hecx@gmail.com
>     <mailto:catherine.hecx@gmail.com>>
>     > ---
>     >  hw/core/loader.c | 9 +++++++++
>     >  1 file changed, 9 insertions(+)
>     >
>     > diff --git a/hw/core/loader.c b/hw/core/loader.c
>     > index fe5cb24122..040109464b 100644
>     > --- a/hw/core/loader.c
>     > +++ b/hw/core/loader.c
>     > @@ -1087,6 +1087,15 @@ static void rom_reset(void *unused)
>     >  {
>     >      Rom *rom;
>     > 
>     > +    /*
>     > +     * We don't need to fill in the RAM with ROM data because
>     we'll fill
>     > +     * the data in during the next incoming migration in all
>     cases.  Note
>     > +     * that some of those RAMs can actually be modified by the
>     guest on ARM
>     > +     * so this is probably the only right thing to do here.
>     > +     */
>     > +    if (runstate_check(RUN_STATE_INMIGRATE))
>     > +        return;
>     > +
>     >      QTAILQ_FOREACH(rom, &roms, next) {
>     >          if (rom->fw_file) {
>     >              continue;
>     > --
>     > 2.17.1
>     >
>     --
>     Dr. David Alan Gilbert / dgilbert@redhat.com
>     <mailto:dgilbert@redhat.com> / Manchester, UK
> 


Re: [Qemu-devel] [PATCH v4] migration: do not rom_reset() during incoming migration
Posted by Peter Xu 4 years, 11 months ago
On Mon, Apr 08, 2019 at 04:42:13AM -0400, Catherine Ho wrote:
> Commit 18269069c310 ("migration: Introduce ignore-shared capability")
> addes ignore-shared capability to bypass the shared ramblock (e,g,
> membackend + numa node). It does good to live migration.
> 
> As told by Yury,this commit expectes that QEMU doesn't write to guest RAM
> until VM starts, but it does on aarch64 qemu:
> Backtrace:
> 1  0x000055f4a296dd84 in address_space_write_rom_internal () at
> exec.c:3458
> 2  0x000055f4a296de3a in address_space_write_rom () at exec.c:3479
> 3  0x000055f4a2d519ff in rom_reset () at hw/core/loader.c:1101
> 4  0x000055f4a2d475ec in qemu_devices_reset () at hw/core/reset.c:69
> 5  0x000055f4a2c90a28 in qemu_system_reset () at vl.c:1675
> 6  0x000055f4a2c9851d in main () at vl.c:4552
> 
> Actually, on arm64 virt marchine, ramblock "dtb" will be filled into ram
> druing rom_reset. In ignore-shared incoming case, this rom filling
> is not required since all the data has been stored in memory backend
> file.
> 
> Further more, as suggested by Peter Xu, if we do rom_reset() now with
> these ROMs then the RAM data should be re-filled again too with the
> migration stream coming in.
> 
> Fixes: commit 18269069c310 ("migration: Introduce ignore-shared
> capability")
> Suggested-by: Yury Kotov <yury-kotov@yandex-team.ru>
> Suggested-by: Peter Xu <peterx@redhat.com>
> Signed-off-by: Catherine Ho <catherine.hecx@gmail.com>

Acked-by: Peter Xu <peterx@redhat.com>

-- 
Peter Xu

Re: [Qemu-devel] [PATCH v4] migration: do not rom_reset() during incoming migration
Posted by Catherine Ho 4 years, 10 months ago
Sorry for the noise, is there any more comment for this patch?
Without this patch, ignore shared capabilities can not be used on arm64

B.R.
Catherine

On Tue, 16 Apr 2019 at 10:51, Peter Xu <peterx@redhat.com> wrote:

> On Mon, Apr 08, 2019 at 04:42:13AM -0400, Catherine Ho wrote:
> > Commit 18269069c310 ("migration: Introduce ignore-shared capability")
> > addes ignore-shared capability to bypass the shared ramblock (e,g,
> > membackend + numa node). It does good to live migration.
> >
> > As told by Yury,this commit expectes that QEMU doesn't write to guest RAM
> > until VM starts, but it does on aarch64 qemu:
> > Backtrace:
> > 1  0x000055f4a296dd84 in address_space_write_rom_internal () at
> > exec.c:3458
> > 2  0x000055f4a296de3a in address_space_write_rom () at exec.c:3479
> > 3  0x000055f4a2d519ff in rom_reset () at hw/core/loader.c:1101
> > 4  0x000055f4a2d475ec in qemu_devices_reset () at hw/core/reset.c:69
> > 5  0x000055f4a2c90a28 in qemu_system_reset () at vl.c:1675
> > 6  0x000055f4a2c9851d in main () at vl.c:4552
> >
> > Actually, on arm64 virt marchine, ramblock "dtb" will be filled into ram
> > druing rom_reset. In ignore-shared incoming case, this rom filling
> > is not required since all the data has been stored in memory backend
> > file.
> >
> > Further more, as suggested by Peter Xu, if we do rom_reset() now with
> > these ROMs then the RAM data should be re-filled again too with the
> > migration stream coming in.
> >
> > Fixes: commit 18269069c310 ("migration: Introduce ignore-shared
> > capability")
> > Suggested-by: Yury Kotov <yury-kotov@yandex-team.ru>
> > Suggested-by: Peter Xu <peterx@redhat.com>
> > Signed-off-by: Catherine Ho <catherine.hecx@gmail.com>
>
> Acked-by: Peter Xu <peterx@redhat.com>
>
> --
> Peter Xu
>