It will be used when we want to resume one paused migration.
Signed-off-by: Peter Xu <peterx@redhat.com>
---
hmp-commands.hx | 7 ++++---
hmp.c | 4 +++-
migration/migration.c | 2 +-
qapi-schema.json | 5 ++++-
4 files changed, 12 insertions(+), 6 deletions(-)
diff --git a/hmp-commands.hx b/hmp-commands.hx
index 1941e19..7adb029 100644
--- a/hmp-commands.hx
+++ b/hmp-commands.hx
@@ -928,13 +928,14 @@ ETEXI
{
.name = "migrate",
- .args_type = "detach:-d,blk:-b,inc:-i,uri:s",
- .params = "[-d] [-b] [-i] uri",
+ .args_type = "detach:-d,blk:-b,inc:-i,resume:-r,uri:s",
+ .params = "[-d] [-b] [-i] [-r] uri",
.help = "migrate to URI (using -d to not wait for completion)"
"\n\t\t\t -b for migration without shared storage with"
" full copy of disk\n\t\t\t -i for migration without "
"shared storage with incremental copy of disk "
- "(base image shared between src and destination)",
+ "(base image shared between src and destination)"
+ "\n\t\t\t -r to resume a paused migration",
.cmd = hmp_migrate,
},
diff --git a/hmp.c b/hmp.c
index fd80dce..ebc1563 100644
--- a/hmp.c
+++ b/hmp.c
@@ -1891,10 +1891,12 @@ void hmp_migrate(Monitor *mon, const QDict *qdict)
bool detach = qdict_get_try_bool(qdict, "detach", false);
bool blk = qdict_get_try_bool(qdict, "blk", false);
bool inc = qdict_get_try_bool(qdict, "inc", false);
+ bool resume = qdict_get_try_bool(qdict, "resume", false);
const char *uri = qdict_get_str(qdict, "uri");
Error *err = NULL;
- qmp_migrate(uri, !!blk, blk, !!inc, inc, false, false, &err);
+ qmp_migrate(uri, !!blk, blk, !!inc, inc,
+ false, false, true, resume, &err);
if (err) {
error_report_err(err);
return;
diff --git a/migration/migration.c b/migration/migration.c
index 9d93836..36ff8c3 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -1238,7 +1238,7 @@ bool migration_is_blocked(Error **errp)
void qmp_migrate(const char *uri, bool has_blk, bool blk,
bool has_inc, bool inc, bool has_detach, bool detach,
- Error **errp)
+ bool has_resume, bool resume, Error **errp)
{
Error *local_err = NULL;
MigrationState *s = migrate_get_current();
diff --git a/qapi-schema.json b/qapi-schema.json
index 2a36b80..27b7c4c 100644
--- a/qapi-schema.json
+++ b/qapi-schema.json
@@ -3208,6 +3208,8 @@
# @detach: this argument exists only for compatibility reasons and
# is ignored by QEMU
#
+# @resume: resume one paused migration
+#
# Returns: nothing on success
#
# Since: 0.14.0
@@ -3229,7 +3231,8 @@
#
##
{ 'command': 'migrate',
- 'data': {'uri': 'str', '*blk': 'bool', '*inc': 'bool', '*detach': 'bool' } }
+ 'data': {'uri': 'str', '*blk': 'bool', '*inc': 'bool',
+ '*detach': 'bool', '*resume': 'bool' } }
##
# @migrate-incoming:
--
2.7.4
On 07/28/2017 03:06 AM, Peter Xu wrote: > It will be used when we want to resume one paused migration. > > Signed-off-by: Peter Xu <peterx@redhat.com> > --- > hmp-commands.hx | 7 ++++--- > hmp.c | 4 +++- > migration/migration.c | 2 +- > qapi-schema.json | 5 ++++- > 4 files changed, 12 insertions(+), 6 deletions(-) > > +++ b/qapi-schema.json > @@ -3208,6 +3208,8 @@ > # @detach: this argument exists only for compatibility reasons and > # is ignored by QEMU > # > +# @resume: resume one paused migration Mention default false, and that it is since 2.11. -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
On Fri, Jul 28, 2017 at 10:57:12AM -0500, Eric Blake wrote: > On 07/28/2017 03:06 AM, Peter Xu wrote: > > It will be used when we want to resume one paused migration. > > > > Signed-off-by: Peter Xu <peterx@redhat.com> > > --- > > hmp-commands.hx | 7 ++++--- > > hmp.c | 4 +++- > > migration/migration.c | 2 +- > > qapi-schema.json | 5 ++++- > > 4 files changed, 12 insertions(+), 6 deletions(-) > > > > > +++ b/qapi-schema.json > > @@ -3208,6 +3208,8 @@ > > # @detach: this argument exists only for compatibility reasons and > > # is ignored by QEMU > > # > > +# @resume: resume one paused migration > > Mention default false, and that it is since 2.11. Will fix. Thanks, -- Peter Xu
* Peter Xu (peterx@redhat.com) wrote:
> It will be used when we want to resume one paused migration.
>
> Signed-off-by: Peter Xu <peterx@redhat.com>
Other than Eric's comments:
Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
> ---
> hmp-commands.hx | 7 ++++---
> hmp.c | 4 +++-
> migration/migration.c | 2 +-
> qapi-schema.json | 5 ++++-
> 4 files changed, 12 insertions(+), 6 deletions(-)
>
> diff --git a/hmp-commands.hx b/hmp-commands.hx
> index 1941e19..7adb029 100644
> --- a/hmp-commands.hx
> +++ b/hmp-commands.hx
> @@ -928,13 +928,14 @@ ETEXI
>
> {
> .name = "migrate",
> - .args_type = "detach:-d,blk:-b,inc:-i,uri:s",
> - .params = "[-d] [-b] [-i] uri",
> + .args_type = "detach:-d,blk:-b,inc:-i,resume:-r,uri:s",
> + .params = "[-d] [-b] [-i] [-r] uri",
> .help = "migrate to URI (using -d to not wait for completion)"
> "\n\t\t\t -b for migration without shared storage with"
> " full copy of disk\n\t\t\t -i for migration without "
> "shared storage with incremental copy of disk "
> - "(base image shared between src and destination)",
> + "(base image shared between src and destination)"
> + "\n\t\t\t -r to resume a paused migration",
> .cmd = hmp_migrate,
> },
>
> diff --git a/hmp.c b/hmp.c
> index fd80dce..ebc1563 100644
> --- a/hmp.c
> +++ b/hmp.c
> @@ -1891,10 +1891,12 @@ void hmp_migrate(Monitor *mon, const QDict *qdict)
> bool detach = qdict_get_try_bool(qdict, "detach", false);
> bool blk = qdict_get_try_bool(qdict, "blk", false);
> bool inc = qdict_get_try_bool(qdict, "inc", false);
> + bool resume = qdict_get_try_bool(qdict, "resume", false);
> const char *uri = qdict_get_str(qdict, "uri");
> Error *err = NULL;
>
> - qmp_migrate(uri, !!blk, blk, !!inc, inc, false, false, &err);
> + qmp_migrate(uri, !!blk, blk, !!inc, inc,
> + false, false, true, resume, &err);
> if (err) {
> error_report_err(err);
> return;
> diff --git a/migration/migration.c b/migration/migration.c
> index 9d93836..36ff8c3 100644
> --- a/migration/migration.c
> +++ b/migration/migration.c
> @@ -1238,7 +1238,7 @@ bool migration_is_blocked(Error **errp)
>
> void qmp_migrate(const char *uri, bool has_blk, bool blk,
> bool has_inc, bool inc, bool has_detach, bool detach,
> - Error **errp)
> + bool has_resume, bool resume, Error **errp)
> {
> Error *local_err = NULL;
> MigrationState *s = migrate_get_current();
> diff --git a/qapi-schema.json b/qapi-schema.json
> index 2a36b80..27b7c4c 100644
> --- a/qapi-schema.json
> +++ b/qapi-schema.json
> @@ -3208,6 +3208,8 @@
> # @detach: this argument exists only for compatibility reasons and
> # is ignored by QEMU
> #
> +# @resume: resume one paused migration
> +#
> # Returns: nothing on success
> #
> # Since: 0.14.0
> @@ -3229,7 +3231,8 @@
> #
> ##
> { 'command': 'migrate',
> - 'data': {'uri': 'str', '*blk': 'bool', '*inc': 'bool', '*detach': 'bool' } }
> + 'data': {'uri': 'str', '*blk': 'bool', '*inc': 'bool',
> + '*detach': 'bool', '*resume': 'bool' } }
>
> ##
> # @migrate-incoming:
> --
> 2.7.4
>
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
On Fri, Jul 28, 2017 at 04:06:25PM +0800, Peter Xu wrote: > It will be used when we want to resume one paused migration. > > Signed-off-by: Peter Xu <peterx@redhat.com> > --- > hmp-commands.hx | 7 ++++--- > hmp.c | 4 +++- > migration/migration.c | 2 +- > qapi-schema.json | 5 ++++- > 4 files changed, 12 insertions(+), 6 deletions(-) I'm not seeing explicit info about how we handle the original failure and how it relates to this resume command, but this feels like a potentially racy approach to me. If we have a network problem between source & target, we could see two results. Either the TCP stream will simply hang (it'll still appear open to QEMU but no traffic will be flowing), or the connection may actually break such that we get EOF and end up closing the file descriptor. In the latter case, we're ok because the original channel is now gone and we can safely establish the new one by issuing the new 'migrate --resume URI' command. In the former case, however, there is the possibility that the hang may come back to life at some point, concurrently with us trying to do 'migrate --resume URI' and I'm unclear on the semantics if that happens. Should the original connection carry on, and thus cause the 'migrate --resume' command to fail, or will we forcably terminate the original connection no matter what and use the new "resumed" connection. There's also synchronization with the target host - at the time we want to recover, we need to be able to tell the target to accept new incoming clients again, but we don't want to do that if the original connection comes back to life. It feels to me that if the mgmt app or admin believes the migration is in a stuck state, we should be able to explicitly terminate the existing connection via a monitor command. Then setup the target host to accept new client, and then issue this migrate resume on the source. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Tue, Aug 01, 2017 at 12:03:48PM +0100, Daniel P. Berrange wrote:
> On Fri, Jul 28, 2017 at 04:06:25PM +0800, Peter Xu wrote:
> > It will be used when we want to resume one paused migration.
> >
> > Signed-off-by: Peter Xu <peterx@redhat.com>
> > ---
> > hmp-commands.hx | 7 ++++---
> > hmp.c | 4 +++-
> > migration/migration.c | 2 +-
> > qapi-schema.json | 5 ++++-
> > 4 files changed, 12 insertions(+), 6 deletions(-)
>
> I'm not seeing explicit info about how we handle the original failure
> and how it relates to this resume command, but this feels like a
> potentially racy approach to me.
>
> If we have a network problem between source & target, we could see
> two results. Either the TCP stream will simply hang (it'll still
> appear open to QEMU but no traffic will be flowing),
(let's say this is the "1st condition")
> or the connection
> may actually break such that we get EOF and end up closing the file
> descriptor.
(let's say this is the "2nd condition")
>
> In the latter case, we're ok because the original channel is now
> gone and we can safely establish the new one by issuing the new
> 'migrate --resume URI' command.
>
> In the former case, however, there is the possibility that the
> hang may come back to life at some point, concurrently with us
> trying to do 'migrate --resume URI' and I'm unclear on the
> semantics if that happens.
>
> Should the original connection carry on, and thus cause the
> 'migrate --resume' command to fail, or will we forcably terminate
> the original connection no matter what and use the new "resumed"
> connection.
Hmm yes, this is a good question. Currently this series is only
handling the 2nd condition, say, when we can detect the error via
system calls (IIUC we can know nothing when the 1st condition is
encountered, we just e.g. block at the system calls as usual when
reading the file handle). And currently the "resume" command is only
allowed if the 2nd condition is detected (so it will never destroy an
existing channel).
If you see the next following patch, there is something like:
if (has_resume && resume) {
if (s->state != MIGRATION_STATUS_POSTCOPY_PAUSED) {
error_setg(errp, "Cannot resume if there is no "
"paused migration");
return;
}
goto do_resume;
}
And here MIGRATION_STATUS_POSTCOPY_PAUSED will only be set when the
2nd condition is met.
>
> There's also synchronization with the target host - at the time we
> want to recover, we need to be able to tell the target to accept
> new incoming clients again, but we don't want to do that if the
> original connection comes back to life.
Yeah, I hacked this part in this v1 series (as you may have seen) to
keep the ports open-forever. I am not sure whether that is acceptable,
but looks not. :)
How about this: when destination detected 2nd condition, it firstly
switch to "postcopy-pause" state, then re-opens the accept channels.
And it can turns the accept channels off when the state moves out of
"postcopy-pause".
>
> It feels to me that if the mgmt app or admin believes the migration
> is in a stuck state, we should be able to explicitly terminate the
> existing connection via a monitor command. Then setup the target
> host to accept new client, and then issue this migrate resume on
> the source.
Totally agree. That should be the only way to handle 1st condition
well. However, would you mind if I postpone it a bit? IMHO as long as
we can solve the 2nd condition nicely (which is the goal of this
series), then it won't be too hard to continue support the 1st
condition.
Since we are at here discussing the usage model... maybe I can further
extend it a bit to gain more input.
IMHO in general there are two phases for the recovery (assume we are
always talking about postcopy):
active --> paused --> recovery --> active
[1] [2]
For [1]: the 1st condition we discussed above can be seen as "manual
pause" - user can provide a command to forcely discard existing
migration channel. While 2nd condition is the "automatic pause" (what
this series does): when qemu detected network problem, it
automatically switch to the paused state.
For [2]: we are always doing it in the "manual" way: we need a command
to trigger the recovery.
What I am thinking is whether it would make sense in the future to do
the "automatic" thing for [2] as well. In that sense, source
periodically detects connectability of existing migration channel
(which is broken), and it will auto-reconnect if it finds that the
network is recovered. We can add a new capability bit for it (e.g.,
"postcopy-auto-recovery"), showing whether we would like the
"automatic recovery" happen.
If we put these into a matrix:
|------------+---------------+----------------------------------------|
| Pause mode | Recovery mode | Use case |
|------------+---------------+----------------------------------------|
| manual | manual | 1st condition mentioned above |
| | auto | (I *guess* we don't need this one) |
|------------+---------------+----------------------------------------|
| auto | manual | 2nd condition mentioned above |
| | auto | (will we want this one in the future?) |
|------------+---------------+----------------------------------------|
Thanks,
--
Peter Xu
On Wed, Aug 02, 2017 at 01:56:46PM +0800, Peter Xu wrote:
> On Tue, Aug 01, 2017 at 12:03:48PM +0100, Daniel P. Berrange wrote:
> > On Fri, Jul 28, 2017 at 04:06:25PM +0800, Peter Xu wrote:
> > > It will be used when we want to resume one paused migration.
> > >
> > > Signed-off-by: Peter Xu <peterx@redhat.com>
> > > ---
> > > hmp-commands.hx | 7 ++++---
> > > hmp.c | 4 +++-
> > > migration/migration.c | 2 +-
> > > qapi-schema.json | 5 ++++-
> > > 4 files changed, 12 insertions(+), 6 deletions(-)
> >
> > I'm not seeing explicit info about how we handle the original failure
> > and how it relates to this resume command, but this feels like a
> > potentially racy approach to me.
> >
> > If we have a network problem between source & target, we could see
> > two results. Either the TCP stream will simply hang (it'll still
> > appear open to QEMU but no traffic will be flowing),
>
> (let's say this is the "1st condition")
>
> > or the connection
> > may actually break such that we get EOF and end up closing the file
> > descriptor.
>
> (let's say this is the "2nd condition")
>
> >
> > In the latter case, we're ok because the original channel is now
> > gone and we can safely establish the new one by issuing the new
> > 'migrate --resume URI' command.
> >
> > In the former case, however, there is the possibility that the
> > hang may come back to life at some point, concurrently with us
> > trying to do 'migrate --resume URI' and I'm unclear on the
> > semantics if that happens.
> >
> > Should the original connection carry on, and thus cause the
> > 'migrate --resume' command to fail, or will we forcably terminate
> > the original connection no matter what and use the new "resumed"
> > connection.
>
> Hmm yes, this is a good question. Currently this series is only
> handling the 2nd condition, say, when we can detect the error via
> system calls (IIUC we can know nothing when the 1st condition is
> encountered, we just e.g. block at the system calls as usual when
> reading the file handle). And currently the "resume" command is only
> allowed if the 2nd condition is detected (so it will never destroy an
> existing channel).
>
> If you see the next following patch, there is something like:
>
> if (has_resume && resume) {
> if (s->state != MIGRATION_STATUS_POSTCOPY_PAUSED) {
> error_setg(errp, "Cannot resume if there is no "
> "paused migration");
> return;
> }
> goto do_resume;
> }
>
> And here MIGRATION_STATUS_POSTCOPY_PAUSED will only be set when the
> 2nd condition is met.
>
> >
> > There's also synchronization with the target host - at the time we
> > want to recover, we need to be able to tell the target to accept
> > new incoming clients again, but we don't want to do that if the
> > original connection comes back to life.
>
> Yeah, I hacked this part in this v1 series (as you may have seen) to
> keep the ports open-forever. I am not sure whether that is acceptable,
> but looks not. :)
>
> How about this: when destination detected 2nd condition, it firstly
> switch to "postcopy-pause" state, then re-opens the accept channels.
> And it can turns the accept channels off when the state moves out of
> "postcopy-pause".
>
> >
> > It feels to me that if the mgmt app or admin believes the migration
> > is in a stuck state, we should be able to explicitly terminate the
> > existing connection via a monitor command. Then setup the target
> > host to accept new client, and then issue this migrate resume on
> > the source.
>
> Totally agree. That should be the only way to handle 1st condition
> well. However, would you mind if I postpone it a bit? IMHO as long as
> we can solve the 2nd condition nicely (which is the goal of this
> series), then it won't be too hard to continue support the 1st
> condition.
Sure, the 1st scenario is an easy bolt on to the second scenario. I
just wanted to be clear about what the target of these patches is,
because I think the 1st scenario is probably the most common one.
I guess if you have TCP keepalives enabled with a reasonably short
timeout, the 1st scenario will turn into the 2nd scenario fairly
quickly.
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
© 2016 - 2026 Red Hat, Inc.