include/migration/cpr.h | 3 -- include/migration/misc.h | 5 +++ migration/migration.h | 3 ++ migration/cpr.c | 18 ++++---- migration/migration.c | 91 +++++++++++++++++++++++++++++++++++++++- migration/options.c | 10 ++--- system/vl.c | 7 ++++ qemu-options.hx | 18 +++++++- 8 files changed, 133 insertions(+), 22 deletions(-)
CI: https://gitlab.com/peterx/qemu/-/pipelines/2560168907 This series introduces a generic way to specify migration parameters that can be used even during the early boot phase of QEMU. One example use case that already existed is CPR-transfer / CPR-exec. currently QEMU has a temporary global variable (incoming_mode) to achieve this, but it's hard to understand and this hack bleeded into quite a few places that we could have avoided. The lines in patch 2 touched may provide some idea. With a generic approach of setting migration parameters with cmdlines, we can remove this hack meanwhile QEMU should be able to keep the CPR behavior as before. To CPR maintainers and reviewers: please have a closer look, even better if it can be smoke tested, to see if this works for Oracle's environment, TIA. The 1st patch implemented that new semantics. It is straightforward: now we can setup any migration parameter using an extra line of: -incoming config:key1=value1,key2=value2,... So far only one such instance is allowed for simplicity, but it should be enough. We still allow multiple -incoming for specifying channels, used together with one "-incoming config:*". Then parameters set this way will be visible almost anytime for QEMU, for example, during initialization of device backends (which is before migration object created). I posted this series majorly because I want to see if this will make a possible new user for the new "local" migration parameter proposed in Vladimir's series: https://lore.kernel.org/r/20260522120534.77653-1-vsementsov@yandex-team.ru Especially, there're some context on this idea too in this email: https://lore.kernel.org/all/ahdI7Vl5KraK566D@x1.local/ With this series, we should be able to drop "incoming-fds" TAP property from the other series, instead relying on the existing "local" parameters both in migration core and in TAP's property should suffice. One thing to mention is I didn't further make only-migratable into a migration parameter. Logically it will also work now with only-migratable, but it then also means I'll need to convert it to a parameter, which will be mutable even after VM started. It will change how only-migratable used to work, hence I skipped. After this, we also almost have no reason to use -global for migration parameters. Capabilities are still not supported in -incoming cmdline, though. Thanks, Peter Xu (2): migration/vl: Allow set parameters with -incoming config:* migration/cpr: Opt-in "mode" parameter for early boot access include/migration/cpr.h | 3 -- include/migration/misc.h | 5 +++ migration/migration.h | 3 ++ migration/cpr.c | 18 ++++---- migration/migration.c | 91 +++++++++++++++++++++++++++++++++++++++- migration/options.c | 10 ++--- system/vl.c | 7 ++++ qemu-options.hx | 18 +++++++- 8 files changed, 133 insertions(+), 22 deletions(-) -- 2.53.0
Peter Xu <peterx@redhat.com> writes: > CI: https://gitlab.com/peterx/qemu/-/pipelines/2560168907 > > This series introduces a generic way to specify migration parameters that > can be used even during the early boot phase of QEMU. > > One example use case that already existed is CPR-transfer / CPR-exec. > currently QEMU has a temporary global variable (incoming_mode) to achieve > this, but it's hard to understand and this hack bleeded into quite a few > places that we could have avoided. The lines in patch 2 touched may > provide some idea. > > With a generic approach of setting migration parameters with cmdlines, we > can remove this hack meanwhile QEMU should be able to keep the CPR behavior > as before. To CPR maintainers and reviewers: please have a closer look, > even better if it can be smoke tested, to see if this works for Oracle's > environment, TIA. > > The 1st patch implemented that new semantics. It is straightforward: now > we can setup any migration parameter using an extra line of: > > -incoming config:key1=value1,key2=value2,... Just merge my code mate. https://lore.kernel.org/r/20251215220041.12657-25-farosas@suse.de A couple of differences from my patch: - old-style keyval vs. json; Isn't json preferred nowadays? - using "config:" This makes it non-uniform with uri and channels which don't have a keyword in front of them. I guess I could live with it, but it seems odd. I see that it makes parsing way easier. > > So far only one such instance is allowed for simplicity, but it should be > enough. We still allow multiple -incoming for specifying channels, used > together with one "-incoming config:*". > > Then parameters set this way will be visible almost anytime for QEMU, for > example, during initialization of device backends (which is before > migration object created). > > I posted this series majorly because I want to see if this will make a > possible new user for the new "local" migration parameter proposed in > Vladimir's series: > > https://lore.kernel.org/r/20260522120534.77653-1-vsementsov@yandex-team.ru > > Especially, there're some context on this idea too in this email: > > https://lore.kernel.org/all/ahdI7Vl5KraK566D@x1.local/ > I like the idea overall. > With this series, we should be able to drop "incoming-fds" TAP property > from the other series, instead relying on the existing "local" parameters > both in migration core and in TAP's property should suffice. > > One thing to mention is I didn't further make only-migratable into a > migration parameter. Logically it will also work now with only-migratable, > but it then also means I'll need to convert it to a parameter, which will > be mutable even after VM started. It will change how only-migratable used > to work, hence I skipped. > > After this, we also almost have no reason to use -global for migration > parameters. Capabilities are still not supported in -incoming cmdline, > though. > Will we still merge capabilities and parameters? Then it would be a free upgrade. > Thanks, > > Peter Xu (2): > migration/vl: Allow set parameters with -incoming config:* > migration/cpr: Opt-in "mode" parameter for early boot access > > include/migration/cpr.h | 3 -- > include/migration/misc.h | 5 +++ > migration/migration.h | 3 ++ > migration/cpr.c | 18 ++++---- > migration/migration.c | 91 +++++++++++++++++++++++++++++++++++++++- > migration/options.c | 10 ++--- > system/vl.c | 7 ++++ > qemu-options.hx | 18 +++++++- > 8 files changed, 133 insertions(+), 22 deletions(-)
On Thu, May 28, 2026 at 07:01:50PM -0300, Fabiano Rosas wrote: > Peter Xu <peterx@redhat.com> writes: > > > CI: https://gitlab.com/peterx/qemu/-/pipelines/2560168907 > > > > This series introduces a generic way to specify migration parameters that > > can be used even during the early boot phase of QEMU. > > > > One example use case that already existed is CPR-transfer / CPR-exec. > > currently QEMU has a temporary global variable (incoming_mode) to achieve > > this, but it's hard to understand and this hack bleeded into quite a few > > places that we could have avoided. The lines in patch 2 touched may > > provide some idea. > > > > With a generic approach of setting migration parameters with cmdlines, we > > can remove this hack meanwhile QEMU should be able to keep the CPR behavior > > as before. To CPR maintainers and reviewers: please have a closer look, > > even better if it can be smoke tested, to see if this works for Oracle's > > environment, TIA. > > > > The 1st patch implemented that new semantics. It is straightforward: now > > we can setup any migration parameter using an extra line of: > > > > -incoming config:key1=value1,key2=value2,... > > Just merge my code mate. > https://lore.kernel.org/r/20251215220041.12657-25-farosas@suse.de Heh, I apologize. I totally forgot it even if I seem to have read it.. The goal is differnent, though, which is to make it available before migration object initialization. Say, I wished -global would work too but it doesn't, and it just can't. So it's not "something good to have" anymore, but functionally required for either CPR and TAP in the future if we adopt this, they'll all read them early. > > A couple of differences from my patch: > > - old-style keyval vs. json; > > Isn't json preferred nowadays? I didn't mention it in doc, but yeah this series also supports JSON due to the same qapi helper used. I don't suggest json though for this entry (and that's also why I didn't mention it..) because 99% of migration parameters are scalars and I don't see why we should use JSON here.. the only one not scalar is block-bitmap-mapping but it's not intended to be used in -incoming at all (even if it'll work..). > > - using "config:" > > This makes it non-uniform with uri and channels which don't have a > keyword in front of them. I guess I could live with it, but it seems > odd. I see that it makes parsing way easier. Yeah, having some identifier would be nice. I wished channels also have identifiers if we don't need to keep compatibility. > > > > > So far only one such instance is allowed for simplicity, but it should be > > enough. We still allow multiple -incoming for specifying channels, used > > together with one "-incoming config:*". > > > > Then parameters set this way will be visible almost anytime for QEMU, for > > example, during initialization of device backends (which is before > > migration object created). > > > > I posted this series majorly because I want to see if this will make a > > possible new user for the new "local" migration parameter proposed in > > Vladimir's series: > > > > https://lore.kernel.org/r/20260522120534.77653-1-vsementsov@yandex-team.ru > > > > Especially, there're some context on this idea too in this email: > > > > https://lore.kernel.org/all/ahdI7Vl5KraK566D@x1.local/ > > > > I like the idea overall. > > > With this series, we should be able to drop "incoming-fds" TAP property > > from the other series, instead relying on the existing "local" parameters > > both in migration core and in TAP's property should suffice. > > > > One thing to mention is I didn't further make only-migratable into a > > migration parameter. Logically it will also work now with only-migratable, > > but it then also means I'll need to convert it to a parameter, which will > > be mutable even after VM started. It will change how only-migratable used > > to work, hence I skipped. > > > > After this, we also almost have no reason to use -global for migration > > parameters. Capabilities are still not supported in -incoming cmdline, > > though. > > > > Will we still merge capabilities and parameters? Then it would be a free > upgrade. Yes! /me waiting for your patches. -- Peter Xu
On 29/05/2026 15:15, Peter Xu wrote: > On Thu, May 28, 2026 at 07:01:50PM -0300, Fabiano Rosas wrote: >> Peter Xu <peterx@redhat.com> writes: >> >>> CI: https://urldefense.proofpoint.com/v2/url?u=https-3A__gitlab.com_peterx_qemu_-2D_pipelines_2560168907&d=DwIBaQ&c=s883GpUCOChKOHiocYtGcg&r=c23RpsaH4D2MKyD3EPJTDa0BAxz6tV8aUJqVSoytEiY&m=UheDdoNwrmPPc3shvbHQ-dAHk7N6GGtWRAbXjZasaJCgV123Fp6foXXlMUC1mXjb&s=KXo-AcgkWswgIUOgetjfuqaSAGVBUC-c78vno6rnNV0&e= >>> >>> This series introduces a generic way to specify migration parameters that >>> can be used even during the early boot phase of QEMU. >>> >>> One example use case that already existed is CPR-transfer / CPR-exec. >>> currently QEMU has a temporary global variable (incoming_mode) to achieve >>> this, but it's hard to understand and this hack bleeded into quite a few >>> places that we could have avoided. The lines in patch 2 touched may >>> provide some idea. >>> >>> With a generic approach of setting migration parameters with cmdlines, we >>> can remove this hack meanwhile QEMU should be able to keep the CPR behavior >>> as before. To CPR maintainers and reviewers: please have a closer look, >>> even better if it can be smoke tested, to see if this works for Oracle's >>> environment, TIA. >>> >>> The 1st patch implemented that new semantics. It is straightforward: now >>> we can setup any migration parameter using an extra line of: >>> >>> -incoming config:key1=value1,key2=value2,... >> >> Just merge my code mate. >> https://urldefense.proofpoint.com/v2/url?u=https-3A__lore.kernel.org_r_20251215220041.12657-2D25-2Dfarosas-40suse.de&d=DwIBaQ&c=s883GpUCOChKOHiocYtGcg&r=c23RpsaH4D2MKyD3EPJTDa0BAxz6tV8aUJqVSoytEiY&m=UheDdoNwrmPPc3shvbHQ-dAHk7N6GGtWRAbXjZasaJCgV123Fp6foXXlMUC1mXjb&s=jaAeqTGqX39wQ3zMnICUZhIcIJl6a0lCALABkjC-HxI&e= > > Heh, I apologize. I totally forgot it even if I seem to have read it.. > > The goal is differnent, though, which is to make it available before > migration object initialization. Say, I wished -global would work too but > it doesn't, and it just can't. So it's not "something good to have" > anymore, but functionally required for either CPR and TAP in the future if > we adopt this, they'll all read them early. One of the reasons CPR requires a separate channel is because it needs to access the stream before the devices are created. Is this part of the same problem in that the migration object needs to be created much earlier than it is currently? And could doing this potentially remove the need for a separate CPR channel? >> A couple of differences from my patch: >> >> - old-style keyval vs. json; >> >> Isn't json preferred nowadays? > > I didn't mention it in doc, but yeah this series also supports JSON due to > the same qapi helper used. I don't suggest json though for this entry (and > that's also why I didn't mention it..) because 99% of migration parameters > are scalars and I don't see why we should use JSON here.. the only one not > scalar is block-bitmap-mapping but it's not intended to be used in > -incoming at all (even if it'll work..). > >> >> - using "config:" >> >> This makes it non-uniform with uri and channels which don't have a >> keyword in front of them. I guess I could live with it, but it seems >> odd. I see that it makes parsing way easier. > > Yeah, having some identifier would be nice. I wished channels also have > identifiers if we don't need to keep compatibility. > >> >>> >>> So far only one such instance is allowed for simplicity, but it should be >>> enough. We still allow multiple -incoming for specifying channels, used >>> together with one "-incoming config:*". >>> >>> Then parameters set this way will be visible almost anytime for QEMU, for >>> example, during initialization of device backends (which is before >>> migration object created). >>> >>> I posted this series majorly because I want to see if this will make a >>> possible new user for the new "local" migration parameter proposed in >>> Vladimir's series: >>> >>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lore.kernel.org_r_20260522120534.77653-2D1-2Dvsementsov-40yandex-2Dteam.ru&d=DwIBaQ&c=s883GpUCOChKOHiocYtGcg&r=c23RpsaH4D2MKyD3EPJTDa0BAxz6tV8aUJqVSoytEiY&m=UheDdoNwrmPPc3shvbHQ-dAHk7N6GGtWRAbXjZasaJCgV123Fp6foXXlMUC1mXjb&s=T6LBhQkrn3e5gBGAQuTCshuzq04UbUaFAO1OaFnRAhY&e= >>> >>> Especially, there're some context on this idea too in this email: >>> >>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lore.kernel.org_all_ahdI7Vl5KraK566D-40x1.local_&d=DwIBaQ&c=s883GpUCOChKOHiocYtGcg&r=c23RpsaH4D2MKyD3EPJTDa0BAxz6tV8aUJqVSoytEiY&m=UheDdoNwrmPPc3shvbHQ-dAHk7N6GGtWRAbXjZasaJCgV123Fp6foXXlMUC1mXjb&s=Ix4yc95OzsoKMBcFrEc51cVrtKzUZKarzIBlN-IeYaI&e= >>> >> >> I like the idea overall. >> >>> With this series, we should be able to drop "incoming-fds" TAP property >>> from the other series, instead relying on the existing "local" parameters >>> both in migration core and in TAP's property should suffice. >>> >>> One thing to mention is I didn't further make only-migratable into a >>> migration parameter. Logically it will also work now with only-migratable, >>> but it then also means I'll need to convert it to a parameter, which will >>> be mutable even after VM started. It will change how only-migratable used >>> to work, hence I skipped. >>> >>> After this, we also almost have no reason to use -global for migration >>> parameters. Capabilities are still not supported in -incoming cmdline, >>> though. >>> >> >> Will we still merge capabilities and parameters? Then it would be a free >> upgrade. > > Yes! /me waiting for your patches. ATB, Mark.
On Wed, Jun 03, 2026 at 04:41:31PM +0100, Mark Cave-Ayland wrote: > One of the reasons CPR requires a separate channel is because it needs to > access the stream before the devices are created. Yes. > Is this part of the same problem in that the migration object needs to be > created much earlier than it is currently? And could doing this > potentially remove the need for a separate CPR channel? The problem is the migration object can't yet be created too early, due to the fact it needs to apply compat properties. That's why this series used another approach to have a temporary early MigrationParameters object to keep things until migration object is initialized (during which the temp object will be destroyed). Thanks, -- Peter Xu
On 03/06/2026 16:59, Peter Xu wrote: >> Is this part of the same problem in that the migration object needs to be >> created much earlier than it is currently? And could doing this >> potentially remove the need for a separate CPR channel? > > The problem is the migration object can't yet be created too early, due to > the fact it needs to apply compat properties. That's why this series used > another approach to have a temporary early MigrationParameters object to > keep things until migration object is initialized (during which the temp > object will be destroyed). If the MigrationParameters are needed earlier, then presumably devices can only depend upon the value of migration capabilities, and not the value of migration compat props? In that case could you not just build the migration compat props with .realized_set_allowed = true in a similar way to e.g. hw/nvram/xlnx-bbram.c? ATB, Mark.
On Thu, Jun 04, 2026 at 11:00:17AM +0100, Mark Cave-Ayland wrote: > On 03/06/2026 16:59, Peter Xu wrote: > > > > Is this part of the same problem in that the migration object needs to be > > > created much earlier than it is currently? And could doing this > > > potentially remove the need for a separate CPR channel? > > > > The problem is the migration object can't yet be created too early, due to > > the fact it needs to apply compat properties. That's why this series used > > another approach to have a temporary early MigrationParameters object to > > keep things until migration object is initialized (during which the temp > > object will be destroyed). > > If the MigrationParameters are needed earlier, then presumably devices can > only depend upon the value of migration capabilities, and not the value of > migration compat props? > > In that case could you not just build the migration compat props with > .realized_set_allowed = true in a similar way to e.g. hw/nvram/xlnx-bbram.c? Could you elaborate? A few contexts to provide: Migration object (even if now a TYPE_DEVICE) doesn't use realize(), and it will be TYPE_OBJECT soon, which is at least the current plan. Direct use of anything qdev specific won't work. The other thing is, migration parameters by default should be allowed to be set anytime after QEMU starts but before migration starts. I don't yet understand the reference of realized_set_allowed use case, and how that idea can apply to migration. Thanks, -- Peter Xu
On 29/05/2026 15:15, Peter Xu wrote: (cut) >> I like the idea overall. >> >>> With this series, we should be able to drop "incoming-fds" TAP property >>> from the other series, instead relying on the existing "local" parameters >>> both in migration core and in TAP's property should suffice. >>> >>> One thing to mention is I didn't further make only-migratable into a >>> migration parameter. Logically it will also work now with only-migratable, >>> but it then also means I'll need to convert it to a parameter, which will >>> be mutable even after VM started. It will change how only-migratable used >>> to work, hence I skipped. >>> >>> After this, we also almost have no reason to use -global for migration >>> parameters. Capabilities are still not supported in -incoming cmdline, >>> though. >>> >> >> Will we still merge capabilities and parameters? Then it would be a free >> upgrade. > > Yes! /me waiting for your patches. Is this related to the series posted at https://patchew.org/QEMU/20251215220041.12657-1-farosas@suse.de/? What's the current status with this work? ATB, Mark.
Mark Cave-Ayland <mark.caveayland@nutanix.com> writes: > On 29/05/2026 15:15, Peter Xu wrote: > > (cut) > >>> I like the idea overall. >>> >>>> With this series, we should be able to drop "incoming-fds" TAP property >>>> from the other series, instead relying on the existing "local" parameters >>>> both in migration core and in TAP's property should suffice. >>>> >>>> One thing to mention is I didn't further make only-migratable into a >>>> migration parameter. Logically it will also work now with only-migratable, >>>> but it then also means I'll need to convert it to a parameter, which will >>>> be mutable even after VM started. It will change how only-migratable used >>>> to work, hence I skipped. >>>> >>>> After this, we also almost have no reason to use -global for migration >>>> parameters. Capabilities are still not supported in -incoming cmdline, >>>> though. >>>> >>> >>> Will we still merge capabilities and parameters? Then it would be a free >>> upgrade. >> >> Yes! /me waiting for your patches. > > Is this related to the series posted at > https://patchew.org/QEMU/20251215220041.12657-1-farosas@suse.de/? What's > the current status with this work? > Yes, the plan is still to merge parameters and capabilities into a single 'options' term. This is largely about internal abstractions. I plan to get back to that part of the work as soon as I find some cycles. The further changes involving deprecations and adding the 'config' parameter to migration commands have encountered some push-back, so I don't think we'll get back to them at least before the above is all finished. I still think those changes are worthwhile, but we need to find consensus. > > ATB, > > Mark.
On Fri, May 29, 2026 at 10:15:35AM -0400, Peter Xu wrote: > On Thu, May 28, 2026 at 07:01:50PM -0300, Fabiano Rosas wrote: > > Peter Xu <peterx@redhat.com> writes: > > > > > CI: https://gitlab.com/peterx/qemu/-/pipelines/2560168907 > > > > > > This series introduces a generic way to specify migration parameters that > > > can be used even during the early boot phase of QEMU. > > > > > > One example use case that already existed is CPR-transfer / CPR-exec. > > > currently QEMU has a temporary global variable (incoming_mode) to achieve > > > this, but it's hard to understand and this hack bleeded into quite a few > > > places that we could have avoided. The lines in patch 2 touched may > > > provide some idea. > > > > > > With a generic approach of setting migration parameters with cmdlines, we > > > can remove this hack meanwhile QEMU should be able to keep the CPR behavior > > > as before. To CPR maintainers and reviewers: please have a closer look, > > > even better if it can be smoke tested, to see if this works for Oracle's > > > environment, TIA. > > > > > > The 1st patch implemented that new semantics. It is straightforward: now > > > we can setup any migration parameter using an extra line of: > > > > > > -incoming config:key1=value1,key2=value2,... > > > > Just merge my code mate. > > https://lore.kernel.org/r/20251215220041.12657-25-farosas@suse.de > > Heh, I apologize. I totally forgot it even if I seem to have read it.. > > The goal is differnent, though, which is to make it available before > migration object initialization. Say, I wished -global would work too but > it doesn't, and it just can't. So it's not "something good to have" > anymore, but functionally required for either CPR and TAP in the future if > we adopt this, they'll all read them early. > > > > > A couple of differences from my patch: > > > > - old-style keyval vs. json; > > > > Isn't json preferred nowadays? > > I didn't mention it in doc, but yeah this series also supports JSON due to > the same qapi helper used. I don't suggest json though for this entry (and > that's also why I didn't mention it..) because 99% of migration parameters > are scalars and I don't see why we should use JSON here.. the only one not > scalar is block-bitmap-mapping but it's not intended to be used in > -incoming at all (even if it'll work..). 99% of **todays** parameters are scalars. The reason we are actively moving to CLI args which are modelled with QAPI and expressable using JSON is because that future proofs the command line for any kind of parameters we might need. We should actively document and promote the use of JSON. > > - using "config:" > > > > This makes it non-uniform with uri and channels which don't have a > > keyword in front of them. I guess I could live with it, but it seems > > odd. I see that it makes parsing way easier. > > Yeah, having some identifier would be nice. I wished channels also have > identifiers if we don't need to keep compatibility. IMHO having a magic "config:" prefix is an anti-pattern because it involves custom command line parsing logic. Does "-incoming config" imply '-incoming defer' semantics or is it independent ? > > > > > > > > > So far only one such instance is allowed for simplicity, but it should be > > > enough. We still allow multiple -incoming for specifying channels, used > > > together with one "-incoming config:*". > > > > > > Then parameters set this way will be visible almost anytime for QEMU, for > > > example, during initialization of device backends (which is before > > > migration object created). > > > > > > I posted this series majorly because I want to see if this will make a > > > possible new user for the new "local" migration parameter proposed in > > > Vladimir's series: > > > > > > https://lore.kernel.org/r/20260522120534.77653-1-vsementsov@yandex-team.ru > > > > > > Especially, there're some context on this idea too in this email: > > > > > > https://lore.kernel.org/all/ahdI7Vl5KraK566D@x1.local/ > > > > > > > I like the idea overall. > > > > > With this series, we should be able to drop "incoming-fds" TAP property > > > from the other series, instead relying on the existing "local" parameters > > > both in migration core and in TAP's property should suffice. > > > > > > One thing to mention is I didn't further make only-migratable into a > > > migration parameter. Logically it will also work now with only-migratable, > > > but it then also means I'll need to convert it to a parameter, which will > > > be mutable even after VM started. It will change how only-migratable used > > > to work, hence I skipped. > > > > > > After this, we also almost have no reason to use -global for migration > > > parameters. Capabilities are still not supported in -incoming cmdline, > > > though. > > > > > > > Will we still merge capabilities and parameters? Then it would be a free > > upgrade. > > Yes! /me waiting for your patches. > > -- > Peter Xu > > With regards, Daniel -- |: https://berrange.com ~~ https://hachyderm.io/@berrange :| |: https://libvirt.org ~~ https://entangle-photo.org :| |: https://pixelfed.art/berrange ~~ https://fstop138.berrange.com :|
On Wed, Jun 03, 2026 at 10:48:54AM +0100, Daniel P. Berrangé wrote:
> On Fri, May 29, 2026 at 10:15:35AM -0400, Peter Xu wrote:
> > On Thu, May 28, 2026 at 07:01:50PM -0300, Fabiano Rosas wrote:
> > > Peter Xu <peterx@redhat.com> writes:
> > >
> > > > CI: https://gitlab.com/peterx/qemu/-/pipelines/2560168907
> > > >
> > > > This series introduces a generic way to specify migration parameters that
> > > > can be used even during the early boot phase of QEMU.
> > > >
> > > > One example use case that already existed is CPR-transfer / CPR-exec.
> > > > currently QEMU has a temporary global variable (incoming_mode) to achieve
> > > > this, but it's hard to understand and this hack bleeded into quite a few
> > > > places that we could have avoided. The lines in patch 2 touched may
> > > > provide some idea.
> > > >
> > > > With a generic approach of setting migration parameters with cmdlines, we
> > > > can remove this hack meanwhile QEMU should be able to keep the CPR behavior
> > > > as before. To CPR maintainers and reviewers: please have a closer look,
> > > > even better if it can be smoke tested, to see if this works for Oracle's
> > > > environment, TIA.
> > > >
> > > > The 1st patch implemented that new semantics. It is straightforward: now
> > > > we can setup any migration parameter using an extra line of:
> > > >
> > > > -incoming config:key1=value1,key2=value2,...
> > >
> > > Just merge my code mate.
> > > https://lore.kernel.org/r/20251215220041.12657-25-farosas@suse.de
> >
> > Heh, I apologize. I totally forgot it even if I seem to have read it..
> >
> > The goal is differnent, though, which is to make it available before
> > migration object initialization. Say, I wished -global would work too but
> > it doesn't, and it just can't. So it's not "something good to have"
> > anymore, but functionally required for either CPR and TAP in the future if
> > we adopt this, they'll all read them early.
> >
> > >
> > > A couple of differences from my patch:
> > >
> > > - old-style keyval vs. json;
> > >
> > > Isn't json preferred nowadays?
> >
> > I didn't mention it in doc, but yeah this series also supports JSON due to
> > the same qapi helper used. I don't suggest json though for this entry (and
> > that's also why I didn't mention it..) because 99% of migration parameters
> > are scalars and I don't see why we should use JSON here.. the only one not
> > scalar is block-bitmap-mapping but it's not intended to be used in
> > -incoming at all (even if it'll work..).
>
> 99% of **todays** parameters are scalars.
>
> The reason we are actively moving to CLI args which are modelled with QAPI
> and expressable using JSON is because that future proofs the command line
> for any kind of parameters we might need.
>
> We should actively document and promote the use of JSON.
Ah OK, sure, will do when respin.
>
> > > - using "config:"
> > >
> > > This makes it non-uniform with uri and channels which don't have a
> > > keyword in front of them. I guess I could live with it, but it seems
> > > odd. I see that it makes parsing way easier.
> >
> > Yeah, having some identifier would be nice. I wished channels also have
> > identifiers if we don't need to keep compatibility.
>
> IMHO having a magic "config:" prefix is an anti-pattern because it
> involves custom command line parsing logic.
Yes, it's not elegant. I wished we had something like this though when
introducing the cpr channels; right now anything wasn't "defer" or URI
implies it's a "channel"..
I also wanted to avoid introducing new cmdline parameters, so migration
incoming cmdlines can stick with the same option. If there's better
suggestion please shoot.
>
> Does "-incoming config" imply '-incoming defer' semantics or is it
> independent ?
They're independent. Examples:
a) "-incoming config:* -incoming tcp:*", setup parameters for TCP incoming
migration without further deferral
b) "-incoming config:*" only, setup global parameters for a possible
upcoming outgoing migration
Thanks,
--
Peter Xu
On Wed, Jun 03, 2026 at 11:50:53AM -0400, Peter Xu wrote: > On Wed, Jun 03, 2026 at 10:48:54AM +0100, Daniel P. Berrangé wrote: > > On Fri, May 29, 2026 at 10:15:35AM -0400, Peter Xu wrote: > > > On Thu, May 28, 2026 at 07:01:50PM -0300, Fabiano Rosas wrote: > > > > Peter Xu <peterx@redhat.com> writes: > > > > > > > > > CI: https://gitlab.com/peterx/qemu/-/pipelines/2560168907 > > > > > > > > > > This series introduces a generic way to specify migration parameters that > > > > > can be used even during the early boot phase of QEMU. > > > > > > > > > > One example use case that already existed is CPR-transfer / CPR-exec. > > > > > currently QEMU has a temporary global variable (incoming_mode) to achieve > > > > > this, but it's hard to understand and this hack bleeded into quite a few > > > > > places that we could have avoided. The lines in patch 2 touched may > > > > > provide some idea. > > > > > > > > > > With a generic approach of setting migration parameters with cmdlines, we > > > > > can remove this hack meanwhile QEMU should be able to keep the CPR behavior > > > > > as before. To CPR maintainers and reviewers: please have a closer look, > > > > > even better if it can be smoke tested, to see if this works for Oracle's > > > > > environment, TIA. > > > > > > > > > > The 1st patch implemented that new semantics. It is straightforward: now > > > > > we can setup any migration parameter using an extra line of: > > > > > > > > > > -incoming config:key1=value1,key2=value2,... snip > > > > - using "config:" > > > > > > > > This makes it non-uniform with uri and channels which don't have a > > > > keyword in front of them. I guess I could live with it, but it seems > > > > odd. I see that it makes parsing way easier. > > > > > > Yeah, having some identifier would be nice. I wished channels also have > > > identifiers if we don't need to keep compatibility. > > > > IMHO having a magic "config:" prefix is an anti-pattern because it > > involves custom command line parsing logic. > > Yes, it's not elegant. I wished we had something like this though when > introducing the cpr channels; right now anything wasn't "defer" or URI > implies it's a "channel".. > > I also wanted to avoid introducing new cmdline parameters, so migration > incoming cmdlines can stick with the same option. If there's better > suggestion please shoot. > > > > > Does "-incoming config" imply '-incoming defer' semantics or is it > > independent ? > > They're independent. Examples: > > a) "-incoming config:* -incoming tcp:*", setup parameters for TCP incoming > migration without further deferral > > b) "-incoming config:*" only, setup global parameters for a possible > upcoming outgoing migration In that case they should definitely be independent command line options. The old "-incoming" design is already broken / limited in the non-'defer' case because it is hardcoded to use URI syntax which can't express all the address formats we accept in QAPI syntax. Adding more special cases onto -incoming just makes the bad situation even worse and is not a forward looking design. If we need the ability to specify migration parameters on the CLI, then as a starting point for the design, we should assume that -incoming does not exist, and design a complete solution from scratch that uses QAPI exclusively, both for addresses and configuration. As discussed before, IMHO the "migrate" and "migrate-incoming" commands need to accept both the address(s) and parameters/capabilities as inline data items rather than relying on pre-configured global state from the 'migrate-parameter' / 'migrate-capability' commands. If we did that modelling for 'migrate-incoming' then that modelling of command parmaeters could map directly to a new '-migrate-incoming' command line argument that accepted exactly the same data model. With regards, Daniel -- |: https://berrange.com ~~ https://hachyderm.io/@berrange :| |: https://libvirt.org ~~ https://entangle-photo.org :| |: https://pixelfed.art/berrange ~~ https://fstop138.berrange.com :|
On 03/06/2026 18:15, Daniel P. Berrangé wrote: > On Wed, Jun 03, 2026 at 11:50:53AM -0400, Peter Xu wrote: >> On Wed, Jun 03, 2026 at 10:48:54AM +0100, Daniel P. Berrangé wrote: >>> On Fri, May 29, 2026 at 10:15:35AM -0400, Peter Xu wrote: >>>> On Thu, May 28, 2026 at 07:01:50PM -0300, Fabiano Rosas wrote: >>>>> Peter Xu <peterx@redhat.com> writes: >>>>> >>>>>> CI: https://urldefense.proofpoint.com/v2/url?u=https-3A__gitlab.com_peterx_qemu_-2D_pipelines_2560168907&d=DwIDaQ&c=s883GpUCOChKOHiocYtGcg&r=c23RpsaH4D2MKyD3EPJTDa0BAxz6tV8aUJqVSoytEiY&m=kl69Otn4IxRhV6bx7NvgCZttOrO1SL4fZ-Qd3Jb-ODVtspTqSiQ6t1ujrrf-AB5J&s=nlkn4_NSgXkwTmC4JK_Le8z4ElYZ_cmlKRBCjJhTrIY&e= >>>>>> >>>>>> This series introduces a generic way to specify migration parameters that >>>>>> can be used even during the early boot phase of QEMU. >>>>>> >>>>>> One example use case that already existed is CPR-transfer / CPR-exec. >>>>>> currently QEMU has a temporary global variable (incoming_mode) to achieve >>>>>> this, but it's hard to understand and this hack bleeded into quite a few >>>>>> places that we could have avoided. The lines in patch 2 touched may >>>>>> provide some idea. >>>>>> >>>>>> With a generic approach of setting migration parameters with cmdlines, we >>>>>> can remove this hack meanwhile QEMU should be able to keep the CPR behavior >>>>>> as before. To CPR maintainers and reviewers: please have a closer look, >>>>>> even better if it can be smoke tested, to see if this works for Oracle's >>>>>> environment, TIA. >>>>>> >>>>>> The 1st patch implemented that new semantics. It is straightforward: now >>>>>> we can setup any migration parameter using an extra line of: >>>>>> >>>>>> -incoming config:key1=value1,key2=value2,... > > snip > >>>>> - using "config:" >>>>> >>>>> This makes it non-uniform with uri and channels which don't have a >>>>> keyword in front of them. I guess I could live with it, but it seems >>>>> odd. I see that it makes parsing way easier. >>>> >>>> Yeah, having some identifier would be nice. I wished channels also have >>>> identifiers if we don't need to keep compatibility. >>> >>> IMHO having a magic "config:" prefix is an anti-pattern because it >>> involves custom command line parsing logic. >> >> Yes, it's not elegant. I wished we had something like this though when >> introducing the cpr channels; right now anything wasn't "defer" or URI >> implies it's a "channel".. >> >> I also wanted to avoid introducing new cmdline parameters, so migration >> incoming cmdlines can stick with the same option. If there's better >> suggestion please shoot. >> >>> >>> Does "-incoming config" imply '-incoming defer' semantics or is it >>> independent ? >> >> They're independent. Examples: >> >> a) "-incoming config:* -incoming tcp:*", setup parameters for TCP incoming >> migration without further deferral >> >> b) "-incoming config:*" only, setup global parameters for a possible >> upcoming outgoing migration > > In that case they should definitely be independent command line > options. The old "-incoming" design is already broken / limited > in the non-'defer' case because it is hardcoded to use URI syntax > which can't express all the address formats we accept in QAPI > syntax. Adding more special cases onto -incoming just makes the > bad situation even worse and is not a forward looking design. > > If we need the ability to specify migration parameters on the > CLI, then as a starting point for the design, we should assume that > -incoming does not exist, and design a complete solution from scratch > that uses QAPI exclusively, both for addresses and configuration. Yes, absolutely. There has been a lot of effort to standardise our parsing, so something that will accept the current (legacy?) -incoming command line along with a new JSON notation allowing the extra parameters feels like the right approach. > As discussed before, IMHO the "migrate" and "migrate-incoming" > commands need to accept both the address(s) and parameters/capabilities > as inline data items rather than relying on pre-configured global state > from the 'migrate-parameter' / 'migrate-capability' commands. > > If we did that modelling for 'migrate-incoming' then that modelling of > command parmaeters could map directly to a new '-migrate-incoming' > command line argument that accepted exactly the same data model. > > With regards, > Daniel ATB, Mark.
Daniel P. Berrangé <berrange@redhat.com> writes: > On Wed, Jun 03, 2026 at 11:50:53AM -0400, Peter Xu wrote: >> On Wed, Jun 03, 2026 at 10:48:54AM +0100, Daniel P. Berrangé wrote: >> > On Fri, May 29, 2026 at 10:15:35AM -0400, Peter Xu wrote: >> > > On Thu, May 28, 2026 at 07:01:50PM -0300, Fabiano Rosas wrote: >> > > > Peter Xu <peterx@redhat.com> writes: >> > > > >> > > > > CI: https://gitlab.com/peterx/qemu/-/pipelines/2560168907 >> > > > > >> > > > > This series introduces a generic way to specify migration parameters that >> > > > > can be used even during the early boot phase of QEMU. >> > > > > >> > > > > One example use case that already existed is CPR-transfer / CPR-exec. >> > > > > currently QEMU has a temporary global variable (incoming_mode) to achieve >> > > > > this, but it's hard to understand and this hack bleeded into quite a few >> > > > > places that we could have avoided. The lines in patch 2 touched may >> > > > > provide some idea. >> > > > > >> > > > > With a generic approach of setting migration parameters with cmdlines, we >> > > > > can remove this hack meanwhile QEMU should be able to keep the CPR behavior >> > > > > as before. To CPR maintainers and reviewers: please have a closer look, >> > > > > even better if it can be smoke tested, to see if this works for Oracle's >> > > > > environment, TIA. >> > > > > >> > > > > The 1st patch implemented that new semantics. It is straightforward: now >> > > > > we can setup any migration parameter using an extra line of: >> > > > > >> > > > > -incoming config:key1=value1,key2=value2,... > > snip > >> > > > - using "config:" >> > > > >> > > > This makes it non-uniform with uri and channels which don't have a >> > > > keyword in front of them. I guess I could live with it, but it seems >> > > > odd. I see that it makes parsing way easier. >> > > >> > > Yeah, having some identifier would be nice. I wished channels also have >> > > identifiers if we don't need to keep compatibility. >> > >> > IMHO having a magic "config:" prefix is an anti-pattern because it >> > involves custom command line parsing logic. >> >> Yes, it's not elegant. I wished we had something like this though when >> introducing the cpr channels; right now anything wasn't "defer" or URI >> implies it's a "channel".. >> >> I also wanted to avoid introducing new cmdline parameters, so migration >> incoming cmdlines can stick with the same option. If there's better >> suggestion please shoot. >> >> > >> > Does "-incoming config" imply '-incoming defer' semantics or is it >> > independent ? >> >> They're independent. Examples: >> >> a) "-incoming config:* -incoming tcp:*", setup parameters for TCP incoming >> migration without further deferral >> >> b) "-incoming config:*" only, setup global parameters for a possible >> upcoming outgoing migration > > In that case they should definitely be independent command line > options. The old "-incoming" design is already broken / limited > in the non-'defer' case because it is hardcoded to use URI syntax > which can't express all the address formats we accept in QAPI > syntax. Adding more special cases onto -incoming just makes the > bad situation even worse and is not a forward looking design. > > If we need the ability to specify migration parameters on the > CLI, then as a starting point for the design, we should assume that > -incoming does not exist, and design a complete solution from scratch > that uses QAPI exclusively, both for addresses and configuration. > > As discussed before, IMHO the "migrate" and "migrate-incoming" > commands need to accept both the address(s) and parameters/capabilities > as inline data items rather than relying on pre-configured global state > from the 'migrate-parameter' / 'migrate-capability' commands. > > If we did that modelling for 'migrate-incoming' then that modelling of > command parmaeters could map directly to a new '-migrate-incoming' > command line argument that accepted exactly the same data model. > Maybe a user creatable object would better fit this use case instead of a new command. We could expose what is today MigrationParameters plus the few compat options from migration_properties. It could work more or less the same for the QMP migration commands, QMP set/get commands, source and destination command lines, the compatibility use-case and the debugging use-case.
On Wed, Jun 03, 2026 at 02:51:30PM -0300, Fabiano Rosas wrote: > Daniel P. Berrangé <berrange@redhat.com> writes: > > > In that case they should definitely be independent command line > > options. The old "-incoming" design is already broken / limited > > in the non-'defer' case because it is hardcoded to use URI syntax > > which can't express all the address formats we accept in QAPI > > syntax. Adding more special cases onto -incoming just makes the > > bad situation even worse and is not a forward looking design. > > > > If we need the ability to specify migration parameters on the > > CLI, then as a starting point for the design, we should assume that > > -incoming does not exist, and design a complete solution from scratch > > that uses QAPI exclusively, both for addresses and configuration. > > > > As discussed before, IMHO the "migrate" and "migrate-incoming" > > commands need to accept both the address(s) and parameters/capabilities > > as inline data items rather than relying on pre-configured global state > > from the 'migrate-parameter' / 'migrate-capability' commands. > > > > If we did that modelling for 'migrate-incoming' then that modelling of > > command parmaeters could map directly to a new '-migrate-incoming' > > command line argument that accepted exactly the same data model. > > > > Maybe a user creatable object would better fit this use case instead of > a new command. We could expose what is today MigrationParameters plus > the few compat options from migration_properties. It could work more or > less the same for the QMP migration commands, QMP set/get commands, > source and destination command lines, the compatibility use-case and the > debugging use-case. If we can do something using "-object" that is useful both at command line time and in QMP runtime, then that would be interesting too. My general feeling wrt changes to the current command line is that any suggestion should be desgined with a nod towards a future scenario where QEMU is 100% configured with QMP. ie the full command line is qemu-system-x86_64 -qmp <address> In this world, the current -incoming argument is already largely unsatisfactory for anything other than "defer". A model that uses -object would fit in well, as would a hypothetical -migrate-incoming that directly mapped to 'migrate-incoming' QMP including capabilities and parameters. With regards, Daniel -- |: https://berrange.com ~~ https://hachyderm.io/@berrange :| |: https://libvirt.org ~~ https://entangle-photo.org :| |: https://pixelfed.art/berrange ~~ https://fstop138.berrange.com :|
On Wed, Jun 03, 2026 at 07:00:07PM +0100, Daniel P. Berrangé wrote: > On Wed, Jun 03, 2026 at 02:51:30PM -0300, Fabiano Rosas wrote: > > Daniel P. Berrangé <berrange@redhat.com> writes: > > > > > In that case they should definitely be independent command line > > > options. The old "-incoming" design is already broken / limited > > > in the non-'defer' case because it is hardcoded to use URI syntax > > > which can't express all the address formats we accept in QAPI > > > syntax. Adding more special cases onto -incoming just makes the > > > bad situation even worse and is not a forward looking design. > > > > > > If we need the ability to specify migration parameters on the > > > CLI, then as a starting point for the design, we should assume that > > > -incoming does not exist, and design a complete solution from scratch > > > that uses QAPI exclusively, both for addresses and configuration. > > > > > > As discussed before, IMHO the "migrate" and "migrate-incoming" > > > commands need to accept both the address(s) and parameters/capabilities > > > as inline data items rather than relying on pre-configured global state > > > from the 'migrate-parameter' / 'migrate-capability' commands. > > > > > > If we did that modelling for 'migrate-incoming' then that modelling of > > > command parmaeters could map directly to a new '-migrate-incoming' > > > command line argument that accepted exactly the same data model. > > > > > > > Maybe a user creatable object would better fit this use case instead of > > a new command. We could expose what is today MigrationParameters plus > > the few compat options from migration_properties. It could work more or > > less the same for the QMP migration commands, QMP set/get commands, > > source and destination command lines, the compatibility use-case and the > > debugging use-case. > > If we can do something using "-object" that is useful both at command > line time and in QMP runtime, then that would be interesting too. > > My general feeling wrt changes to the current command line is that > any suggestion should be desgined with a nod towards a future scenario > where QEMU is 100% configured with QMP. ie the full command line is > > qemu-system-x86_64 -qmp <address> > > In this world, the current -incoming argument is already largely > unsatisfactory for anything other than "defer". A model that uses > -object would fit in well, as would a hypothetical -migrate-incoming > that directly mapped to 'migrate-incoming' QMP including capabilities > and parameters. I don't think even the current -incoming config:* is a blocker or add too much complexity to the full-QMP-based solution. When it comes, we can simply map all -incoming config:* (JSON or not) to QMP command migrate-set-parameters, what we need is teaching that QMP handler to apply parameters to the temporary MigrationParameters rather than migration object when the latter hasn't been initialized. I was talking to Fabiano and he suggested me to mention one more thing I said on the list. It's a matter of whether we should still invest time on supporting migration caps/parameters in QMP migrate/migrate-incoming commands. Requirements like this (allow migration parameters to be accessible during early stage of QEMU) is going towards the other direction of the that idea. That means even if we put all configs (caps/params) into QMP command migrate[-incoming], libvirt will still need to manage these global setup and making sure when invoking migrate[-incoming] QMP commands they match with the globals. Say, if one start QEMU with -incoming config:local=on but then invoke "migrate-incoming,local=off" it's illegal. Considering that it looks like there're solid use cases that we want to support (after CPR's "mode" parameter, now "local"), I want to discuss again whether we want to still spend effort supporting "allow migrate QMP command to specify capabilities/parameters". Ultimately, we still seem to need these global parameters. Thanks, -- Peter Xu
On Wed, Jun 03, 2026 at 02:46:52PM -0400, Peter Xu wrote: > On Wed, Jun 03, 2026 at 07:00:07PM +0100, Daniel P. Berrangé wrote: > > On Wed, Jun 03, 2026 at 02:51:30PM -0300, Fabiano Rosas wrote: > > > Daniel P. Berrangé <berrange@redhat.com> writes: > > > > > > > In that case they should definitely be independent command line > > > > options. The old "-incoming" design is already broken / limited > > > > in the non-'defer' case because it is hardcoded to use URI syntax > > > > which can't express all the address formats we accept in QAPI > > > > syntax. Adding more special cases onto -incoming just makes the > > > > bad situation even worse and is not a forward looking design. > > > > > > > > If we need the ability to specify migration parameters on the > > > > CLI, then as a starting point for the design, we should assume that > > > > -incoming does not exist, and design a complete solution from scratch > > > > that uses QAPI exclusively, both for addresses and configuration. > > > > > > > > As discussed before, IMHO the "migrate" and "migrate-incoming" > > > > commands need to accept both the address(s) and parameters/capabilities > > > > as inline data items rather than relying on pre-configured global state > > > > from the 'migrate-parameter' / 'migrate-capability' commands. > > > > > > > > If we did that modelling for 'migrate-incoming' then that modelling of > > > > command parmaeters could map directly to a new '-migrate-incoming' > > > > command line argument that accepted exactly the same data model. > > > > > > > > > > Maybe a user creatable object would better fit this use case instead of > > > a new command. We could expose what is today MigrationParameters plus > > > the few compat options from migration_properties. It could work more or > > > less the same for the QMP migration commands, QMP set/get commands, > > > source and destination command lines, the compatibility use-case and the > > > debugging use-case. > > > > If we can do something using "-object" that is useful both at command > > line time and in QMP runtime, then that would be interesting too. > > > > My general feeling wrt changes to the current command line is that > > any suggestion should be desgined with a nod towards a future scenario > > where QEMU is 100% configured with QMP. ie the full command line is > > > > qemu-system-x86_64 -qmp <address> > > > > In this world, the current -incoming argument is already largely > > unsatisfactory for anything other than "defer". A model that uses > > -object would fit in well, as would a hypothetical -migrate-incoming > > that directly mapped to 'migrate-incoming' QMP including capabilities > > and parameters. > > I don't think even the current -incoming config:* is a blocker or add too > much complexity to the full-QMP-based solution. > > When it comes, we can simply map all -incoming config:* (JSON or not) to > QMP command migrate-set-parameters, what we need is teaching that QMP > handler to apply parameters to the temporary MigrationParameters rather > than migration object when the latter hasn't been initialized. > > I was talking to Fabiano and he suggested me to mention one more thing I > said on the list. It's a matter of whether we should still invest time on > supporting migration caps/parameters in QMP migrate/migrate-incoming > commands. > > Requirements like this (allow migration parameters to be accessible during > early stage of QEMU) is going towards the other direction of the that idea. > That means even if we put all configs (caps/params) into QMP command > migrate[-incoming], libvirt will still need to manage these global setup > and making sure when invoking migrate[-incoming] QMP commands they match > with the globals. Say, if one start QEMU with -incoming config:local=on > but then invoke "migrate-incoming,local=off" it's illegal. > > Considering that it looks like there're solid use cases that we want to > support (after CPR's "mode" parameter, now "local"), I want to discuss > again whether we want to still spend effort supporting "allow migrate QMP > command to specify capabilities/parameters". Ultimately, we still seem to > need these global parameters. This is saying that incoming migration is a multi phase/stage task. There is a "prepare" phase which sets up QEMU ready to receive an incoming migration. There is a "running" phase where we are waiting for incoming connection and/or handling the migration data stream. The current "migrate_incoming" overloads both tasks into the same QMP command such that they always happen at the same point in time. "-incoming defer" was a crude hack to partially separate them, allowing parameters/capabilities to be set before the real migrate_incoming QMP command is invoked. We can address that by separating the tasks explicitly with a "migrate_prepare" command / -migrate-prepare CLI arg that sets all the parameters/capabilities atomically, and a "migrate_run" command that initiates the processing. The "jobs" QAPI design has this explicit concept of different phases that a job can be in, and provides commands for lifecycle handling. With regards, Daniel -- |: https://berrange.com ~~ https://hachyderm.io/@berrange :| |: https://libvirt.org ~~ https://entangle-photo.org :| |: https://pixelfed.art/berrange ~~ https://fstop138.berrange.com :|
On Thu, Jun 04, 2026 at 09:18:26AM +0100, Daniel P. Berrangé wrote: > On Wed, Jun 03, 2026 at 02:46:52PM -0400, Peter Xu wrote: > > On Wed, Jun 03, 2026 at 07:00:07PM +0100, Daniel P. Berrangé wrote: > > > On Wed, Jun 03, 2026 at 02:51:30PM -0300, Fabiano Rosas wrote: > > > > Daniel P. Berrangé <berrange@redhat.com> writes: > > > > > > > > > In that case they should definitely be independent command line > > > > > options. The old "-incoming" design is already broken / limited > > > > > in the non-'defer' case because it is hardcoded to use URI syntax > > > > > which can't express all the address formats we accept in QAPI > > > > > syntax. Adding more special cases onto -incoming just makes the > > > > > bad situation even worse and is not a forward looking design. > > > > > > > > > > If we need the ability to specify migration parameters on the > > > > > CLI, then as a starting point for the design, we should assume that > > > > > -incoming does not exist, and design a complete solution from scratch > > > > > that uses QAPI exclusively, both for addresses and configuration. > > > > > > > > > > As discussed before, IMHO the "migrate" and "migrate-incoming" > > > > > commands need to accept both the address(s) and parameters/capabilities > > > > > as inline data items rather than relying on pre-configured global state > > > > > from the 'migrate-parameter' / 'migrate-capability' commands. > > > > > > > > > > If we did that modelling for 'migrate-incoming' then that modelling of > > > > > command parmaeters could map directly to a new '-migrate-incoming' > > > > > command line argument that accepted exactly the same data model. > > > > > > > > > > > > > Maybe a user creatable object would better fit this use case instead of > > > > a new command. We could expose what is today MigrationParameters plus > > > > the few compat options from migration_properties. It could work more or > > > > less the same for the QMP migration commands, QMP set/get commands, > > > > source and destination command lines, the compatibility use-case and the > > > > debugging use-case. > > > > > > If we can do something using "-object" that is useful both at command > > > line time and in QMP runtime, then that would be interesting too. > > > > > > My general feeling wrt changes to the current command line is that > > > any suggestion should be desgined with a nod towards a future scenario > > > where QEMU is 100% configured with QMP. ie the full command line is > > > > > > qemu-system-x86_64 -qmp <address> > > > > > > In this world, the current -incoming argument is already largely > > > unsatisfactory for anything other than "defer". A model that uses > > > -object would fit in well, as would a hypothetical -migrate-incoming > > > that directly mapped to 'migrate-incoming' QMP including capabilities > > > and parameters. > > > > I don't think even the current -incoming config:* is a blocker or add too > > much complexity to the full-QMP-based solution. > > > > When it comes, we can simply map all -incoming config:* (JSON or not) to > > QMP command migrate-set-parameters, what we need is teaching that QMP > > handler to apply parameters to the temporary MigrationParameters rather > > than migration object when the latter hasn't been initialized. > > > > I was talking to Fabiano and he suggested me to mention one more thing I > > said on the list. It's a matter of whether we should still invest time on > > supporting migration caps/parameters in QMP migrate/migrate-incoming > > commands. > > > > Requirements like this (allow migration parameters to be accessible during > > early stage of QEMU) is going towards the other direction of the that idea. > > That means even if we put all configs (caps/params) into QMP command > > migrate[-incoming], libvirt will still need to manage these global setup > > and making sure when invoking migrate[-incoming] QMP commands they match > > with the globals. Say, if one start QEMU with -incoming config:local=on > > but then invoke "migrate-incoming,local=off" it's illegal. > > > > Considering that it looks like there're solid use cases that we want to > > support (after CPR's "mode" parameter, now "local"), I want to discuss > > again whether we want to still spend effort supporting "allow migrate QMP > > command to specify capabilities/parameters". Ultimately, we still seem to > > need these global parameters. > > This is saying that incoming migration is a multi phase/stage task. > > There is a "prepare" phase which sets up QEMU ready to receive an > incoming migration. > > There is a "running" phase where we are waiting for incoming connection > and/or handling the migration data stream. > > The current "migrate_incoming" overloads both tasks into the same QMP > command such that they always happen at the same point in time. > "-incoming defer" was a crude hack to partially separate them, allowing > parameters/capabilities to be set before the real migrate_incoming > QMP command is invoked. > > We can address that by separating the tasks explicitly with a > "migrate_prepare" command / -migrate-prepare CLI arg that sets all > the parameters/capabilities atomically, and a "migrate_run" command > that initiates the processing. I'm OK with this, or IMHO we can also stick with -incoming treating it as the preparation phase. For most of the time, I would slightly prefer keeping old things working but building new things on top. Obsolete should only happen if very needed and properly justified. For this one, I hope the QAPI helpers I used should introduce minimum overhead even if we're reusing an old paramaeter.. we can still consider introducing another parmeter besides -incoming, but it'll work similarly like what this patch does, then. > > The "jobs" QAPI design has this explicit concept of different phases > that a job can be in, and provides commands for lifecycle handling. We have some rich discussion before on Jobs: https://lore.kernel.org/qemu-devel/878qg1uhbd.fsf_-_@pond.sub.org/#t IIRC the consensus was it's only about the interfacing that can be shared or not, which doesn't seem to be a huge mount of code that we can reuse; what we can reuse is minimum and we may need to overload the interface when migration joins the party. We're definitely open to adopt anything Jobs did right. We don't necessarily need to switch to Jobs until further justified, which won't be a trivial task. Thanks, -- Peter Xu
On 03.06.26 21:46, Peter Xu wrote:
>> My general feeling wrt changes to the current command line is that
>> any suggestion should be desgined with a nod towards a future scenario
>> where QEMU is 100% configured with QMP. ie the full command line is
>>
>> qemu-system-x86_64 -qmp <address>
>>
[..]
>
> Requirements like this (allow migration parameters to be accessible during
> early stage of QEMU) is going towards the other direction of the that idea.
> That means even if we put all configs (caps/params) into QMP command
> migrate[-incoming], libvirt will still need to manage these global setup
> and making sure when invoking migrate[-incoming] QMP commands they match
> with the globals. Say, if one start QEMU with -incomingconfig:local=on
> but then invoke "migrate-incoming,local=off" it's illegal.
Right.. And this show, that moving "local" from migrate-set-parameters to
commandline is not a clear solution for the whole problem.
We don't need cmdline argument. We need two things:
1. "local" must be set before initializing tap-device.
It not actually requires it being a cmdline parameter. Calling
migrate-set-parameters before netdev_add is also OK.
So, moving to cmdline solves this [1] point, but may be too restrictive.
2. "local" must not be changed after initializing tap-device.
And this one is not guaranteed anyway, with cmdline or with QMP.
---
So, in my series, if drop "incoming-fds" and rely on "local" instead, we
actually want "local" be immutable after first read in tap initialization
code.
Maybe, just implement this feature? So, in code it will look like:
set_migration_parameters(...) {
...
if "local" value is changing and "local" is immutable:
fail
...
}
tap_initializaion(...) {
...
local = migrate_get_local_and_make_it_immutable(err_text="you can not change 'local' parameter value after TAP device initialization");
...
}
---
For user it should provide simple error messages
"you can not change 'local' parameter value after TAP device initialization".
if user tries to change local when it's not allowed.
--
Best regards,
Vladimir
On Thu, Jun 04, 2026 at 01:00:10AM +0300, Vladimir Sementsov-Ogievskiy wrote:
> On 03.06.26 21:46, Peter Xu wrote:
> > > My general feeling wrt changes to the current command line is that
> > > any suggestion should be desgined with a nod towards a future scenario
> > > where QEMU is 100% configured with QMP. ie the full command line is
> > >
> > > qemu-system-x86_64 -qmp <address>
> > >
>
> [..]
>
> >
> > Requirements like this (allow migration parameters to be accessible during
> > early stage of QEMU) is going towards the other direction of the that idea.
> > That means even if we put all configs (caps/params) into QMP command
> > migrate[-incoming], libvirt will still need to manage these global setup
> > and making sure when invoking migrate[-incoming] QMP commands they match
> > with the globals. Say, if one start QEMU with -incomingconfig:local=on
> > but then invoke "migrate-incoming,local=off" it's illegal.
>
> Right.. And this show, that moving "local" from migrate-set-parameters to
> commandline is not a clear solution for the whole problem.
>
> We don't need cmdline argument. We need two things:
>
> 1. "local" must be set before initializing tap-device.
>
> It not actually requires it being a cmdline parameter. Calling
> migrate-set-parameters before netdev_add is also OK.
Ohh is it ok? I thought it was not OK so we look at this.
Say, this is what I see on init TAP device when without hotplug:
qemu_create_late_backends() -> net_init_clients()
Such happens before migration object initialization.
>
> So, moving to cmdline solves this [1] point, but may be too restrictive.
>
>
> 2. "local" must not be changed after initializing tap-device.
>
> And this one is not guaranteed anyway, with cmdline or with QMP.
>
> ---
>
>
> So, in my series, if drop "incoming-fds" and rely on "local" instead, we
> actually want "local" be immutable after first read in tap initialization
> code.
>
> Maybe, just implement this feature? So, in code it will look like:
>
> set_migration_parameters(...) {
>
> ...
>
> if "local" value is changing and "local" is immutable:
> fail
We can't do that, can we?
Imagine we need to further migrate this VM to another host, where we need
to turn "local" off after this incoming migration.. we can forbid only
during incoming phase and re-enable the mutability, but it seems too much.
My understanding is such protection is fine but not strongly necessary.
IMHO we rely on a lot of things that admin needs to do right. I hope this
isn't a major issue to offload that to admin to say the admin should always
do the right things.
We have a bunch of similar issues in QEMU IIUC, e.g. we have known issue
that some -device needs to be ordered in some way otherwise it'll stop
working. We then need admin (or in this case libvirt) do the right thing
too.
>
> ...
>
> }
>
>
> tap_initializaion(...) {
>
> ...
>
> local = migrate_get_local_and_make_it_immutable(err_text="you can not change 'local' parameter value after TAP device initialization");
>
> ...
>
>
> }
>
> ---
>
> For user it should provide simple error messages
>
> "you can not change 'local' parameter value after TAP device initialization".
>
> if user tries to change local when it's not allowed.
>
>
> --
> Best regards,
> Vladimir
>
--
Peter Xu
On 04.06.26 21:01, Peter Xu wrote:
> On Thu, Jun 04, 2026 at 01:00:10AM +0300, Vladimir Sementsov-Ogievskiy wrote:
>> On 03.06.26 21:46, Peter Xu wrote:
>>>> My general feeling wrt changes to the current command line is that
>>>> any suggestion should be desgined with a nod towards a future scenario
>>>> where QEMU is 100% configured with QMP. ie the full command line is
>>>>
>>>> qemu-system-x86_64 -qmp <address>
>>>>
>>
>> [..]
>>
>>>
>>> Requirements like this (allow migration parameters to be accessible during
>>> early stage of QEMU) is going towards the other direction of the that idea.
>>> That means even if we put all configs (caps/params) into QMP command
>>> migrate[-incoming], libvirt will still need to manage these global setup
>>> and making sure when invoking migrate[-incoming] QMP commands they match
>>> with the globals. Say, if one start QEMU with -incomingconfig:local=on
>>> but then invoke "migrate-incoming,local=off" it's illegal.
>>
>> Right.. And this show, that moving "local" from migrate-set-parameters to
>> commandline is not a clear solution for the whole problem.
>>
>> We don't need cmdline argument. We need two things:
>>
>> 1. "local" must be set before initializing tap-device.
>>
>> It not actually requires it being a cmdline parameter. Calling
>> migrate-set-parameters before netdev_add is also OK.
>
> Ohh is it ok? I thought it was not OK so we look at this.
>
> Say, this is what I see on init TAP device when without hotplug:
>
> qemu_create_late_backends() -> net_init_clients()
>
> Such happens before migration object initialization.
Yes, it doesn't work for TAP devices added through commandline. But
if devices are added only throuhg QMP (isn't libvirt do so?), it
should work.
>
>>
>> So, moving to cmdline solves this [1] point, but may be too restrictive.
>>
>>
>> 2. "local" must not be changed after initializing tap-device.
>>
>> And this one is not guaranteed anyway, with cmdline or with QMP.
>>
>> ---
>>
>>
>> So, in my series, if drop "incoming-fds" and rely on "local" instead, we
>> actually want "local" be immutable after first read in tap initialization
>> code.
>>
>> Maybe, just implement this feature? So, in code it will look like:
>>
>> set_migration_parameters(...) {
>>
>> ...
>>
>> if "local" value is changing and "local" is immutable:
>> fail
>
> We can't do that, can we?
>
> Imagine we need to further migrate this VM to another host, where we need
> to turn "local" off after this incoming migration.. we can forbid only
> during incoming phase and re-enable the mutability, but it seems too much.
Agree, right. I missed further outgoing migration.
>
> My understanding is such protection is fine but not strongly necessary.
> IMHO we rely on a lot of things that admin needs to do right. I hope this
> isn't a major issue to offload that to admin to say the admin should always
> do the right things.
>
> We have a bunch of similar issues in QEMU IIUC, e.g. we have known issue
> that some -device needs to be ordered in some way otherwise it'll stop
> working. We then need admin (or in this case libvirt) do the right thing
> too.
So, possible way is:
1. net devices are only added by QMP commands
2. migration parameter "local" is set before adding devices
So, we can omit "incoming-fds" and rely on "local" && "support-local-migration".
And we can check in .pre_incoming of TAP device, does current value of "local"
equal to what it was at time of TAP initialization, and fail if it differs.
Still, having explicit incoming-fds looks safer, as it's more difficult to do
a mistake.
With incoming-fds=true, user is sure, that new target vm will not try
to connect/open resources, which are still used by running source vm,
regardless of migration configuration correctness.
For example, if user accidentally set "local" _after_ creating TAP device,
we can of-course check it in .pre_incoming and fail, but at the moment we
already break networking in running guest, by attaching to same backend on
target QEMU.
--
Best regards,
Vladimir
On Fri, Jun 05, 2026 at 10:35:52AM +0300, Vladimir Sementsov-Ogievskiy wrote:
> On 04.06.26 21:01, Peter Xu wrote:
> > On Thu, Jun 04, 2026 at 01:00:10AM +0300, Vladimir Sementsov-Ogievskiy wrote:
> > > On 03.06.26 21:46, Peter Xu wrote:
> > > > > My general feeling wrt changes to the current command line is that
> > > > > any suggestion should be desgined with a nod towards a future scenario
> > > > > where QEMU is 100% configured with QMP. ie the full command line is
> > > > >
> > > > > qemu-system-x86_64 -qmp <address>
> > > > >
> > >
> > > [..]
> > >
> > > >
> > > > Requirements like this (allow migration parameters to be accessible during
> > > > early stage of QEMU) is going towards the other direction of the that idea.
> > > > That means even if we put all configs (caps/params) into QMP command
> > > > migrate[-incoming], libvirt will still need to manage these global setup
> > > > and making sure when invoking migrate[-incoming] QMP commands they match
> > > > with the globals. Say, if one start QEMU with -incomingconfig:local=on
> > > > but then invoke "migrate-incoming,local=off" it's illegal.
> > >
> > > Right.. And this show, that moving "local" from migrate-set-parameters to
> > > commandline is not a clear solution for the whole problem.
> > >
> > > We don't need cmdline argument. We need two things:
> > >
> > > 1. "local" must be set before initializing tap-device.
> > >
> > > It not actually requires it being a cmdline parameter. Calling
> > > migrate-set-parameters before netdev_add is also OK.
> >
> > Ohh is it ok? I thought it was not OK so we look at this.
> >
> > Say, this is what I see on init TAP device when without hotplug:
> >
> > qemu_create_late_backends() -> net_init_clients()
> >
> > Such happens before migration object initialization.
>
> Yes, it doesn't work for TAP devices added through commandline. But
> if devices are added only throuhg QMP (isn't libvirt do so?), it
> should work.
>
> >
> > >
> > > So, moving to cmdline solves this [1] point, but may be too restrictive.
> > >
> > >
> > > 2. "local" must not be changed after initializing tap-device.
> > >
> > > And this one is not guaranteed anyway, with cmdline or with QMP.
> > >
> > > ---
> > >
> > >
> > > So, in my series, if drop "incoming-fds" and rely on "local" instead, we
> > > actually want "local" be immutable after first read in tap initialization
> > > code.
> > >
> > > Maybe, just implement this feature? So, in code it will look like:
> > >
> > > set_migration_parameters(...) {
> > >
> > > ...
> > >
> > > if "local" value is changing and "local" is immutable:
> > > fail
> >
> > We can't do that, can we?
> >
> > Imagine we need to further migrate this VM to another host, where we need
> > to turn "local" off after this incoming migration.. we can forbid only
> > during incoming phase and re-enable the mutability, but it seems too much.
>
> Agree, right. I missed further outgoing migration.
>
> >
> > My understanding is such protection is fine but not strongly necessary.
> > IMHO we rely on a lot of things that admin needs to do right. I hope this
> > isn't a major issue to offload that to admin to say the admin should always
> > do the right things.
> >
> > We have a bunch of similar issues in QEMU IIUC, e.g. we have known issue
> > that some -device needs to be ordered in some way otherwise it'll stop
> > working. We then need admin (or in this case libvirt) do the right thing
> > too.
>
> So, possible way is:
>
> 1. net devices are only added by QMP commands
> 2. migration parameter "local" is set before adding devices
>
> So, we can omit "incoming-fds" and rely on "local" && "support-local-migration".
>
> And we can check in .pre_incoming of TAP device, does current value of "local"
> equal to what it was at time of TAP initialization, and fail if it differs.
>
> Still, having explicit incoming-fds looks safer, as it's more difficult to do
> a mistake.
If this will be the only outlier, then maybe yes, an extra option in a
special device backend isn't much of an issue.
The question is if there'll be more things relying on "local=on". We may
not want to keep introducing per-device flags even if all of them can play
the same role as migration's "local".
>
> With incoming-fds=true, user is sure, that new target vm will not try
> to connect/open resources, which are still used by running source vm,
> regardless of migration configuration correctness.
>
> For example, if user accidentally set "local" _after_ creating TAP device,
> we can of-course check it in .pre_incoming and fail, but at the moment we
> already break networking in running guest, by attaching to same backend on
> target QEMU.
>
> --
> Best regards,
> Vladimir
>
--
Peter Xu
On 05.06.26 17:08, Peter Xu wrote:
> On Fri, Jun 05, 2026 at 10:35:52AM +0300, Vladimir Sementsov-Ogievskiy wrote:
>> On 04.06.26 21:01, Peter Xu wrote:
>>> On Thu, Jun 04, 2026 at 01:00:10AM +0300, Vladimir Sementsov-Ogievskiy wrote:
>>>> On 03.06.26 21:46, Peter Xu wrote:
>>>>>> My general feeling wrt changes to the current command line is that
>>>>>> any suggestion should be desgined with a nod towards a future scenario
>>>>>> where QEMU is 100% configured with QMP. ie the full command line is
>>>>>>
>>>>>> qemu-system-x86_64 -qmp <address>
>>>>>>
>>>> [..]
>>>>
>>>>> Requirements like this (allow migration parameters to be accessible during
>>>>> early stage of QEMU) is going towards the other direction of the that idea.
>>>>> That means even if we put all configs (caps/params) into QMP command
>>>>> migrate[-incoming], libvirt will still need to manage these global setup
>>>>> and making sure when invoking migrate[-incoming] QMP commands they match
>>>>> with the globals. Say, if one start QEMU with -incomingconfig:local=on
>>>>> but then invoke "migrate-incoming,local=off" it's illegal.
>>>> Right.. And this show, that moving "local" from migrate-set-parameters to
>>>> commandline is not a clear solution for the whole problem.
>>>>
>>>> We don't need cmdline argument. We need two things:
>>>>
>>>> 1. "local" must be set before initializing tap-device.
>>>>
>>>> It not actually requires it being a cmdline parameter. Calling
>>>> migrate-set-parameters before netdev_add is also OK.
>>> Ohh is it ok? I thought it was not OK so we look at this.
>>>
>>> Say, this is what I see on init TAP device when without hotplug:
>>>
>>> qemu_create_late_backends() -> net_init_clients()
>>>
>>> Such happens before migration object initialization.
>> Yes, it doesn't work for TAP devices added through commandline. But
>> if devices are added only throuhg QMP (isn't libvirt do so?), it
>> should work.
>>
>>>> So, moving to cmdline solves this [1] point, but may be too restrictive.
>>>>
>>>>
>>>> 2. "local" must not be changed after initializing tap-device.
>>>>
>>>> And this one is not guaranteed anyway, with cmdline or with QMP.
>>>>
>>>> ---
>>>>
>>>>
>>>> So, in my series, if drop "incoming-fds" and rely on "local" instead, we
>>>> actually want "local" be immutable after first read in tap initialization
>>>> code.
>>>>
>>>> Maybe, just implement this feature? So, in code it will look like:
>>>>
>>>> set_migration_parameters(...) {
>>>>
>>>> ...
>>>>
>>>> if "local" value is changing and "local" is immutable:
>>>> fail
>>> We can't do that, can we?
>>>
>>> Imagine we need to further migrate this VM to another host, where we need
>>> to turn "local" off after this incoming migration.. we can forbid only
>>> during incoming phase and re-enable the mutability, but it seems too much.
>> Agree, right. I missed further outgoing migration.
>>
>>> My understanding is such protection is fine but not strongly necessary.
>>> IMHO we rely on a lot of things that admin needs to do right. I hope this
>>> isn't a major issue to offload that to admin to say the admin should always
>>> do the right things.
>>>
>>> We have a bunch of similar issues in QEMU IIUC, e.g. we have known issue
>>> that some -device needs to be ordered in some way otherwise it'll stop
>>> working. We then need admin (or in this case libvirt) do the right thing
>>> too.
>> So, possible way is:
>>
>> 1. net devices are only added by QMP commands
>> 2. migration parameter "local" is set before adding devices
>>
>> So, we can omit "incoming-fds" and rely on "local" && "support-local-migration".
>>
>> And we can check in .pre_incoming of TAP device, does current value of "local"
>> equal to what it was at time of TAP initialization, and fail if it differs.
>>
>> Still, having explicit incoming-fds looks safer, as it's more difficult to do
>> a mistake.
> If this will be the only outlier, then maybe yes, an extra option in a
> special device backend isn't much of an issue.
>
> The question is if there'll be more things relying on "local=on". We may
> not want to keep introducing per-device flags even if all of them can play
> the same role as migration's "local".
Agree.
Hmm, so, may be, I just go this way for my series? It will just work with
these two restrictions above, which is OK for me. And no conflict with
new "-incoming" design.
--
Best regards,
Vladimir
On Fri, Jun 05, 2026 at 05:37:03PM +0300, Vladimir Sementsov-Ogievskiy wrote: > Hmm, so, may be, I just go this way for my series? It will just work with > these two restrictions above, which is OK for me. And no conflict with > new "-incoming" design. Sounds good. IIUC that's also our original plan to make it experimental and see if that can lands earlier, so no worry on whatever happens with this series so far, feel free to go ahead. Thanks, -- Peter Xu
On Wed, Jun 03, 2026 at 06:15:57PM +0100, Daniel P. Berrangé wrote: > On Wed, Jun 03, 2026 at 11:50:53AM -0400, Peter Xu wrote: > > On Wed, Jun 03, 2026 at 10:48:54AM +0100, Daniel P. Berrangé wrote: > > > On Fri, May 29, 2026 at 10:15:35AM -0400, Peter Xu wrote: > > > > On Thu, May 28, 2026 at 07:01:50PM -0300, Fabiano Rosas wrote: > > > > > Peter Xu <peterx@redhat.com> writes: > > > > > > > > > > > CI: https://gitlab.com/peterx/qemu/-/pipelines/2560168907 > > > > > > > > > > > > This series introduces a generic way to specify migration parameters that > > > > > > can be used even during the early boot phase of QEMU. > > > > > > > > > > > > One example use case that already existed is CPR-transfer / CPR-exec. > > > > > > currently QEMU has a temporary global variable (incoming_mode) to achieve > > > > > > this, but it's hard to understand and this hack bleeded into quite a few > > > > > > places that we could have avoided. The lines in patch 2 touched may > > > > > > provide some idea. > > > > > > > > > > > > With a generic approach of setting migration parameters with cmdlines, we > > > > > > can remove this hack meanwhile QEMU should be able to keep the CPR behavior > > > > > > as before. To CPR maintainers and reviewers: please have a closer look, > > > > > > even better if it can be smoke tested, to see if this works for Oracle's > > > > > > environment, TIA. > > > > > > > > > > > > The 1st patch implemented that new semantics. It is straightforward: now > > > > > > we can setup any migration parameter using an extra line of: > > > > > > > > > > > > -incoming config:key1=value1,key2=value2,... > > snip > > > > > > - using "config:" > > > > > > > > > > This makes it non-uniform with uri and channels which don't have a > > > > > keyword in front of them. I guess I could live with it, but it seems > > > > > odd. I see that it makes parsing way easier. > > > > > > > > Yeah, having some identifier would be nice. I wished channels also have > > > > identifiers if we don't need to keep compatibility. > > > > > > IMHO having a magic "config:" prefix is an anti-pattern because it > > > involves custom command line parsing logic. > > > > Yes, it's not elegant. I wished we had something like this though when > > introducing the cpr channels; right now anything wasn't "defer" or URI > > implies it's a "channel".. > > > > I also wanted to avoid introducing new cmdline parameters, so migration > > incoming cmdlines can stick with the same option. If there's better > > suggestion please shoot. > > > > > > > > Does "-incoming config" imply '-incoming defer' semantics or is it > > > independent ? > > > > They're independent. Examples: > > > > a) "-incoming config:* -incoming tcp:*", setup parameters for TCP incoming > > migration without further deferral > > > > b) "-incoming config:*" only, setup global parameters for a possible > > upcoming outgoing migration > > In that case they should definitely be independent command line > options. The old "-incoming" design is already broken / limited > in the non-'defer' case because it is hardcoded to use URI syntax > which can't express all the address formats we accept in QAPI > syntax. Adding more special cases onto -incoming just makes the > bad situation even worse and is not a forward looking design. > > If we need the ability to specify migration parameters on the > CLI, then as a starting point for the design, we should assume that > -incoming does not exist, and design a complete solution from scratch > that uses QAPI exclusively, both for addresses and configuration. Do we want to obsolete -incoming? > > As discussed before, IMHO the "migrate" and "migrate-incoming" > commands need to accept both the address(s) and parameters/capabilities > as inline data items rather than relying on pre-configured global state > from the 'migrate-parameter' / 'migrate-capability' commands. This is one example that "relying only on migrate/migrate-incoming to specify parameters" won't work. Essentially parameters like CPR's and the new "local" parameter wants to be global so that it will be visible and will have an impact on how other QEMU modules behave. Here for TAP when "local" is globally on we may want to initialize the device differently from the default operations. That will need to happen before any QMP commands. > > If we did that modelling for 'migrate-incoming' then that modelling of > command parmaeters could map directly to a new '-migrate-incoming' > command line argument that accepted exactly the same data model. This is true. Then we will make this series to depend on Fabiano's previous work (to be posted; per we talked yesterday). I wanted to see if we can reduce the impact to minimum, -incoming is indeed not well designed, but in real life we have control over what can happen with URIs and the current "config:" won't conflict with any of them. But if we strongly wish to obsolete -incoming completely, I'm OK too. Thanks, -- Peter Xu
Peter Xu <peterx@redhat.com> writes: > On Wed, Jun 03, 2026 at 06:15:57PM +0100, Daniel P. Berrangé wrote: >> On Wed, Jun 03, 2026 at 11:50:53AM -0400, Peter Xu wrote: >> > On Wed, Jun 03, 2026 at 10:48:54AM +0100, Daniel P. Berrangé wrote: >> > > On Fri, May 29, 2026 at 10:15:35AM -0400, Peter Xu wrote: >> > > > On Thu, May 28, 2026 at 07:01:50PM -0300, Fabiano Rosas wrote: >> > > > > Peter Xu <peterx@redhat.com> writes: >> > > > > >> > > > > > CI: https://gitlab.com/peterx/qemu/-/pipelines/2560168907 >> > > > > > >> > > > > > This series introduces a generic way to specify migration parameters that >> > > > > > can be used even during the early boot phase of QEMU. >> > > > > > >> > > > > > One example use case that already existed is CPR-transfer / CPR-exec. >> > > > > > currently QEMU has a temporary global variable (incoming_mode) to achieve >> > > > > > this, but it's hard to understand and this hack bleeded into quite a few >> > > > > > places that we could have avoided. The lines in patch 2 touched may >> > > > > > provide some idea. >> > > > > > >> > > > > > With a generic approach of setting migration parameters with cmdlines, we >> > > > > > can remove this hack meanwhile QEMU should be able to keep the CPR behavior >> > > > > > as before. To CPR maintainers and reviewers: please have a closer look, >> > > > > > even better if it can be smoke tested, to see if this works for Oracle's >> > > > > > environment, TIA. >> > > > > > >> > > > > > The 1st patch implemented that new semantics. It is straightforward: now >> > > > > > we can setup any migration parameter using an extra line of: >> > > > > > >> > > > > > -incoming config:key1=value1,key2=value2,... >> >> snip >> >> > > > > - using "config:" >> > > > > >> > > > > This makes it non-uniform with uri and channels which don't have a >> > > > > keyword in front of them. I guess I could live with it, but it seems >> > > > > odd. I see that it makes parsing way easier. >> > > > >> > > > Yeah, having some identifier would be nice. I wished channels also have >> > > > identifiers if we don't need to keep compatibility. >> > > >> > > IMHO having a magic "config:" prefix is an anti-pattern because it >> > > involves custom command line parsing logic. >> > >> > Yes, it's not elegant. I wished we had something like this though when >> > introducing the cpr channels; right now anything wasn't "defer" or URI >> > implies it's a "channel".. >> > >> > I also wanted to avoid introducing new cmdline parameters, so migration >> > incoming cmdlines can stick with the same option. If there's better >> > suggestion please shoot. >> > >> > > >> > > Does "-incoming config" imply '-incoming defer' semantics or is it >> > > independent ? >> > >> > They're independent. Examples: >> > >> > a) "-incoming config:* -incoming tcp:*", setup parameters for TCP incoming >> > migration without further deferral >> > >> > b) "-incoming config:*" only, setup global parameters for a possible >> > upcoming outgoing migration >> >> In that case they should definitely be independent command line >> options. The old "-incoming" design is already broken / limited >> in the non-'defer' case because it is hardcoded to use URI syntax >> which can't express all the address formats we accept in QAPI >> syntax. Adding more special cases onto -incoming just makes the >> bad situation even worse and is not a forward looking design. >> >> If we need the ability to specify migration parameters on the >> CLI, then as a starting point for the design, we should assume that >> -incoming does not exist, and design a complete solution from scratch >> that uses QAPI exclusively, both for addresses and configuration. > > Do we want to obsolete -incoming? > >> >> As discussed before, IMHO the "migrate" and "migrate-incoming" >> commands need to accept both the address(s) and parameters/capabilities >> as inline data items rather than relying on pre-configured global state >> from the 'migrate-parameter' / 'migrate-capability' commands. > > This is one example that "relying only on migrate/migrate-incoming to > specify parameters" won't work. > > Essentially parameters like CPR's and the new "local" parameter wants to be > global so that it will be visible and will have an impact on how other QEMU > modules behave. Here for TAP when "local" is globally on we may want to > initialize the device differently from the default operations. That will > need to happen before any QMP commands. > >> >> If we did that modelling for 'migrate-incoming' then that modelling of >> command parmaeters could map directly to a new '-migrate-incoming' >> command line argument that accepted exactly the same data model. > > This is true. Then we will make this series to depend on Fabiano's > previous work (to be posted; per we talked yesterday). > I don't see the dependency. Take the 'config' out from this series and pass a JSON into -incoming. It can be parsed into MigrationParameters all the same. > I wanted to see if we can reduce the impact to minimum, -incoming is indeed > not well designed, but in real life we have control over what can happen > with URIs and the current "config:" won't conflict with any of them. > > But if we strongly wish to obsolete -incoming completely, I'm OK too. > > Thanks,
© 2016 - 2026 Red Hat, Inc.