We want to switch both from QemuOpts to the keyval parser in the future,
which results in some incompatibilities, mainly around list handling.
Mark the non-JSON version of both as unstable syntax so that management
tools switch to JSON and we can later make the change without breaking
things.

Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
 docs/about/deprecated.rst | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/docs/about/deprecated.rst b/docs/about/deprecated.rst
index 3c2be84d80..42f6a478fb 100644
--- a/docs/about/deprecated.rst
+++ b/docs/about/deprecated.rst
@@ -160,6 +160,17 @@ Use ``-display sdl`` instead.
 
 Use ``-display curses`` instead.
 
+Stable non-JSON ``-device`` and ``-object`` syntax (since 6.2)
+''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
+
+If you rely on a stable interface for ``-device`` and ``-object`` that doesn't
+change incompatibly between QEMU versions (e.g. because you are using the QEMU
+command line as a machine interface in scripts rather than interactively), use
+JSON syntax for these options instead.
+
+There is no intention to remove support for non-JSON syntax entirely, but
+future versions may change the way to spell some options.
+
 
 Plugin argument passing through ``arg=<string>`` (since 6.1)
 ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
-- 
2.31.1

On Fri, Sep 24, 2021 at 11:04:27AM +0200, Kevin Wolf wrote:
> We want to switch both from QemuOpts to the keyval parser in the future,
> which results in some incompatibilities, mainly around list handling.
> Mark the non-JSON version of both as unstable syntax so that management
> tools switch to JSON and we can later make the change without breaking
> things.
> 
> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
> ---
>  docs/about/deprecated.rst | 11 +++++++++++
>  1 file changed, 11 insertions(+)

Reviewed-by: Eric Blake <eblake@redhat.com>

> 
> diff --git a/docs/about/deprecated.rst b/docs/about/deprecated.rst
> index 3c2be84d80..42f6a478fb 100644
> --- a/docs/about/deprecated.rst
> +++ b/docs/about/deprecated.rst
> @@ -160,6 +160,17 @@ Use ``-display sdl`` instead.
>  
>  Use ``-display curses`` instead.
>  
> +Stable non-JSON ``-device`` and ``-object`` syntax (since 6.2)
> +''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
> +
> +If you rely on a stable interface for ``-device`` and ``-object`` that doesn't
> +change incompatibly between QEMU versions (e.g. because you are using the QEMU
> +command line as a machine interface in scripts rather than interactively), use
> +JSON syntax for these options instead.
> +
> +There is no intention to remove support for non-JSON syntax entirely, but
> +future versions may change the way to spell some options.
> +
>  
>  Plugin argument passing through ``arg=<string>`` (since 6.1)
>  ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
> -- 
> 2.31.1
> 

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org

On 24/09/21 11:04, Kevin Wolf wrote:
> We want to switch both from QemuOpts to the keyval parser in the future,
> which results in some incompatibilities, mainly around list handling.
> Mark the non-JSON version of both as unstable syntax so that management
> tools switch to JSON and we can later make the change without breaking
> things.

Maybe we need a different section for unstable syntaxes, rather than 
overloading deprecated.rst?

Paolo

On Fri, 24 Sept 2021 at 10:14, Kevin Wolf <kwolf@redhat.com> wrote:
>
> We want to switch both from QemuOpts to the keyval parser in the future,
> which results in some incompatibilities, mainly around list handling.
> Mark the non-JSON version of both as unstable syntax so that management
> tools switch to JSON and we can later make the change without breaking
> things.
>
> Signed-off-by: Kevin Wolf <kwolf@redhat.com>

> +Stable non-JSON ``-device`` and ``-object`` syntax (since 6.2)
> +''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
> +
> +If you rely on a stable interface for ``-device`` and ``-object`` that doesn't
> +change incompatibly between QEMU versions (e.g. because you are using the QEMU
> +command line as a machine interface in scripts rather than interactively), use
> +JSON syntax for these options instead.
> +
> +There is no intention to remove support for non-JSON syntax entirely, but
> +future versions may change the way to spell some options.

As it stands, this is basically saying "pretty much anybody
using the command line, your stuff may break in future, instead
use some other interface you've never heard of, which doesn't
appear to be documented in the manual and which none of the
documentation's examples use". Is there some more limited
deprecation we can do rather than "the entire commandline
for almost all users" ?

thanks
-- PMM

Am 27.09.2021 um 11:00 hat Peter Maydell geschrieben:
> On Fri, 24 Sept 2021 at 10:14, Kevin Wolf <kwolf@redhat.com> wrote:
> >
> > We want to switch both from QemuOpts to the keyval parser in the future,
> > which results in some incompatibilities, mainly around list handling.
> > Mark the non-JSON version of both as unstable syntax so that management
> > tools switch to JSON and we can later make the change without breaking
> > things.
> >
> > Signed-off-by: Kevin Wolf <kwolf@redhat.com>
> 
> > +Stable non-JSON ``-device`` and ``-object`` syntax (since 6.2)
> > +''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
> > +
> > +If you rely on a stable interface for ``-device`` and ``-object`` that doesn't
> > +change incompatibly between QEMU versions (e.g. because you are using the QEMU
> > +command line as a machine interface in scripts rather than interactively), use
> > +JSON syntax for these options instead.
> > +
> > +There is no intention to remove support for non-JSON syntax entirely, but
> > +future versions may change the way to spell some options.
> 
> As it stands, this is basically saying "pretty much anybody
> using the command line, your stuff may break in future, instead
> use some other interface you've never heard of, which doesn't
> appear to be documented in the manual and which none of the
> documentation's examples use".

The documentation is a valid criticism. We need to document the JSON
interfaces properly (which will really mostly be a pointer to the
existing QMP documentation at least for -object, but it's important to
tell people where to look for the details).

> Is there some more limited deprecation we can do rather than "the
> entire commandline for almost all users" ?

I don't think "almost all" users is true. I see three groups of users:

1. Using a management tool that is probably using libvirt. This is
   likely the vast majority of users. They won't notice a difference
   because libvirt abstracts it away. libvirt developers are actively
   asking us for JSON (and QAPI) based interfaces because using the same
   representation to describe configurations in QMP and on the CLI makes
   their life easier.

2. People starting QEMU on the command line manually. This is
   essentially the same situation as HMP: If something changes, you get
   an error message, you look up the new syntax, done. Small
   inconvenience, but that's it. This includes simple scripts that just
   start QEMU and are only used to store a long command line somewhere.

3. People writing more complex scripts or applications that invoke QEMU
   manually instead of using libvirt. They may actually be hurt by such
   changes. They should probably be using a proper machine interface,
   i.e. JSON mode, so the deprecation notice is for them to change
   their code. This is probably a small minority and not "almost all
   users".

Yes, we could in theory do a more limited deprecation. The planned
change from my side is just going from QemuOpts to the keyval parser,
which doesn't change anything in the vast majority of cases.

But we have the separation in the monitor between QMP and HMP for a good
reason: Requirements for a good machine interface are different from a
good human interface. The same is true for the command line.

So it seems to make a lot of sense to me to have both a machine
interface (JSON based) and a human interface (non-JSON) on the command
line, too, and take the same liberties for evolving the human interface
as we already do in HMP - which means that it's technically an unstable
interface where compatibility doesn't prevent improvements, but not that
it looks completely different in every QEMU version.

Kevin

On Mon, 27 Sept 2021 at 12:27, Kevin Wolf <kwolf@redhat.com> wrote:
>
> Am 27.09.2021 um 11:00 hat Peter Maydell geschrieben:
> > On Fri, 24 Sept 2021 at 10:14, Kevin Wolf <kwolf@redhat.com> wrote:
> > >
> > > We want to switch both from QemuOpts to the keyval parser in the future,
> > > which results in some incompatibilities, mainly around list handling.
> > > Mark the non-JSON version of both as unstable syntax so that management
> > > tools switch to JSON and we can later make the change without breaking
> > > things.
> > >
> > > Signed-off-by: Kevin Wolf <kwolf@redhat.com>
> >
> > > +Stable non-JSON ``-device`` and ``-object`` syntax (since 6.2)
> > > +''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
> > > +
> > > +If you rely on a stable interface for ``-device`` and ``-object`` that doesn't
> > > +change incompatibly between QEMU versions (e.g. because you are using the QEMU
> > > +command line as a machine interface in scripts rather than interactively), use
> > > +JSON syntax for these options instead.
> > > +
> > > +There is no intention to remove support for non-JSON syntax entirely, but
> > > +future versions may change the way to spell some options.
> >
> > As it stands, this is basically saying "pretty much anybody
> > using the command line, your stuff may break in future, instead
> > use some other interface you've never heard of, which doesn't
> > appear to be documented in the manual and which none of the
> > documentation's examples use".
>
> The documentation is a valid criticism. We need to document the JSON
> interfaces properly (which will really mostly be a pointer to the
> existing QMP documentation at least for -object, but it's important to
> tell people where to look for the details).
>
> > Is there some more limited deprecation we can do rather than "the
> > entire commandline for almost all users" ?
>
> I don't think "almost all" users is true.

> I see three groups of users

...all of whom "rely on a stable interface for -device and -object",
and only two of whom it's reasonable to say "use the JSON version" to.

> 1. Using a management tool that is probably using libvirt. This is
>    likely the vast majority of users. They won't notice a difference
>    because libvirt abstracts it away. libvirt developers are actively
>    asking us for JSON (and QAPI) based interfaces because using the same
>    representation to describe configurations in QMP and on the CLI makes
>    their life easier.

Yes, absolutely we should be recommending that libvirt and
other management interfaces use the JSON.

> 2. People starting QEMU on the command line manually. This is
>    essentially the same situation as HMP: If something changes, you get
>    an error message, you look up the new syntax, done. Small
>    inconvenience, but that's it. This includes simple scripts that just
>    start QEMU and are only used to store a long command line somewhere.

It's a small inconvenience that we seem to be imposing on our
users on a pretty frequent basis. Moreover, each one of these
really needs to be its own deprecation, so that users actually
can have some advance notice if they need it and look up the
new syntax. We shouldn't hide them all under this umbrella
"we might break anything at any time" entry, which I think
will pretty much encourage breaking compatibility more often
because you don't have to think about "oh, we should deprecate
this and maybe print warnings about use of deprecated syntax
and then drop it later", you can just break things and point
to this "we said -device wasn't going to be stable" entry.

As a concrete example, the commit message for this patch vaguely
mentions some issues "around list handling", which gives me no
idea at all about what syntax is going to break in future or
whether it is likely to affect scripts I've written.

> 3. People writing more complex scripts or applications that invoke QEMU
>    manually instead of using libvirt. They may actually be hurt by such
>    changes. They should probably be using a proper machine interface,
>    i.e. JSON mode, so the deprecation notice is for them to change
>    their code. This is probably a small minority and not "almost all
>    users".

Yeah, this group is kind of similar to group 1 (well, at one
end it shades into group 1 and at the other into group 2).

More generally, I think I'd rather see the deprecation of
the old approach appear after some period when the JSON
command line version has been available to users and adopted
by major consumers like libvirt, not as a patch in the same
series which is introducing the JSON syntax in the first plaec.

-- PMM

Am 27.09.2021 um 14:52 hat Peter Maydell geschrieben:
> On Mon, 27 Sept 2021 at 12:27, Kevin Wolf <kwolf@redhat.com> wrote:
> >
> > Am 27.09.2021 um 11:00 hat Peter Maydell geschrieben:
> > > On Fri, 24 Sept 2021 at 10:14, Kevin Wolf <kwolf@redhat.com> wrote:
> > > >
> > > > We want to switch both from QemuOpts to the keyval parser in the future,
> > > > which results in some incompatibilities, mainly around list handling.
> > > > Mark the non-JSON version of both as unstable syntax so that management
> > > > tools switch to JSON and we can later make the change without breaking
> > > > things.
> > > >
> > > > Signed-off-by: Kevin Wolf <kwolf@redhat.com>
> > >
> > > > +Stable non-JSON ``-device`` and ``-object`` syntax (since 6.2)
> > > > +''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
> > > > +
> > > > +If you rely on a stable interface for ``-device`` and ``-object`` that doesn't
> > > > +change incompatibly between QEMU versions (e.g. because you are using the QEMU
> > > > +command line as a machine interface in scripts rather than interactively), use
> > > > +JSON syntax for these options instead.
> > > > +
> > > > +There is no intention to remove support for non-JSON syntax entirely, but
> > > > +future versions may change the way to spell some options.
> > >
> > > As it stands, this is basically saying "pretty much anybody
> > > using the command line, your stuff may break in future, instead
> > > use some other interface you've never heard of, which doesn't
> > > appear to be documented in the manual and which none of the
> > > documentation's examples use".
> >
> > The documentation is a valid criticism. We need to document the JSON
> > interfaces properly (which will really mostly be a pointer to the
> > existing QMP documentation at least for -object, but it's important to
> > tell people where to look for the details).
> >
> > > Is there some more limited deprecation we can do rather than "the
> > > entire commandline for almost all users" ?
> >
> > I don't think "almost all" users is true.
> 
> > I see three groups of users
> 
> ...all of whom "rely on a stable interface for -device and -object",
> and only two of whom it's reasonable to say "use the JSON version" to.

I'm not sure that the human interactive case requires unchanged syntax
as strictly as the other cases do.

After each distro upgrade (or even a browser upgrade within the same
distro), some UI changes somewhere and I have to adapt. I don't think
anyone ever makes promises like "this button is going to stay in the
exact same place forever". And our policy is already that we're not
making such promises for HMP either.

> > 1. Using a management tool that is probably using libvirt. This is
> >    likely the vast majority of users. They won't notice a difference
> >    because libvirt abstracts it away. libvirt developers are actively
> >    asking us for JSON (and QAPI) based interfaces because using the same
> >    representation to describe configurations in QMP and on the CLI makes
> >    their life easier.
> 
> Yes, absolutely we should be recommending that libvirt and
> other management interfaces use the JSON.
> 
> > 2. People starting QEMU on the command line manually. This is
> >    essentially the same situation as HMP: If something changes, you get
> >    an error message, you look up the new syntax, done. Small
> >    inconvenience, but that's it. This includes simple scripts that just
> >    start QEMU and are only used to store a long command line somewhere.
> 
> It's a small inconvenience that we seem to be imposing on our
> users on a pretty frequent basis. Moreover, each one of these
> really needs to be its own deprecation, so that users actually
> can have some advance notice if they need it and look up the
> new syntax. We shouldn't hide them all under this umbrella
> "we might break anything at any time" entry, which I think
> will pretty much encourage breaking compatibility more often
> because you don't have to think about "oh, we should deprecate
> this and maybe print warnings about use of deprecated syntax
> and then drop it later", you can just break things and point
> to this "we said -device wasn't going to be stable" entry.

Are you suggesting bringing back stricter compatibility rules for HMP
then?

The problem with the deprecation period is that you need to have a time
where both options work. But when you have two different parsers, you
can't just magically have the union of their accepted syntaxes. Unless
you invest a lot of time and effort, you have a flag day where the
syntax changes. And I think this is perfectly reasonable for interactive
use.

Deprecations are the wrong tool for human interfaces. You don't need to
know months in advance that something will change in the UI (you'll
forget it again anyway until the change happens and the information
becomes relevant), but this is a case for the changelog.

If you do need to know months in advance, JSON is probably better suited
for your use case.

> As a concrete example, the commit message for this patch vaguely
> mentions some issues "around list handling", which gives me no
> idea at all about what syntax is going to break in future or
> whether it is likely to affect scripts I've written.

The one known incompatible case for -object is the 'host-nodes' property
of memory-backend, which is a list. QemuOpts doesn't support lists at
all, but the string visitor allows using a comma separated list of
integer ranges. (By the way, the existing syntax of it seems to be
undocumented, so not sure how many users even know about how to use this
list support.)

For -device, who knows. I'm positive that the string visitor list
support isn't used there and a patch in this series removes the string
visitor from -device therefore.
What other incompatibilities and hacks there are in some devices, we'll
find out when we actually try to describe devices in QAPI. I'm not even
entirely sure yet that there will be incompatibilities. But if we do
find a case, I don't want to have to stop and delay work for another
year to have a deprecation period for interactive human interfaces.

Another similar case is -drive: We only still have it in its current
form because even figuring out what the exact cases are that would be
incompatible is hard enough to stop us from even trying to deprecate
them individually. But actually it would seem reasonable enough to me to
just change -drive because that makes sense as a convenient option for
interactive use, while -blockdev is more verbose and covers the stable
API part.

> > 3. People writing more complex scripts or applications that invoke QEMU
> >    manually instead of using libvirt. They may actually be hurt by such
> >    changes. They should probably be using a proper machine interface,
> >    i.e. JSON mode, so the deprecation notice is for them to change
> >    their code. This is probably a small minority and not "almost all
> >    users".
> 
> Yeah, this group is kind of similar to group 1 (well, at one
> end it shades into group 1 and at the other into group 2).
> 
> More generally, I think I'd rather see the deprecation of
> the old approach appear after some period when the JSON
> command line version has been available to users and adopted
> by major consumers like libvirt, not as a patch in the same
> series which is introducing the JSON syntax in the first plaec.

Okay, that can be done. So I can post this final patch separately and
only for -object for now, and we'll do -device separately once libvirt
uses JSON for it.

Kevin