Sphinx, through Pygments, does not like annotated json examples very
much. In some versions of Sphinx (1.7), it will render the non-json
portions of code blocks in red, but in newer versions (2.0) it will
throw an exception and not highlight the block at all. Though we can
suppress this warning, it doesn't bring back highlighting on non-strict
json blocks.

We can alleviate this by creating a custom lexer for QMP examples that
allows us to properly highlight these examples in a robust way, keeping
our directionality notations.

Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
Signed-off-by: John Snow <jsnow@redhat.com>
---
 docs/conf.py             |  4 ++--
 docs/sphinx/qmp_lexer.py | 34 ++++++++++++++++++++++++++++++++++
 2 files changed, 36 insertions(+), 2 deletions(-)
 create mode 100644 docs/sphinx/qmp_lexer.py

diff --git a/docs/conf.py b/docs/conf.py
index befbcc6c3e..e46b299b71 100644
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -41,7 +41,7 @@ except NameError:
 # add these directories to sys.path here. If the directory is relative to the
 # documentation root, use an absolute path starting from qemu_docdir.
 #
-# sys.path.insert(0, os.path.join(qemu_docdir, "my_subdir"))
+sys.path.insert(0, os.path.join(qemu_docdir, "sphinx"))
 
 
 # -- General configuration ------------------------------------------------
@@ -54,7 +54,7 @@ needs_sphinx = '1.3'
 # Add any Sphinx extension module names here, as strings. They can be
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
 # ones.
-extensions = []
+extensions = ['qmp_lexer']
 
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ['_templates']
diff --git a/docs/sphinx/qmp_lexer.py b/docs/sphinx/qmp_lexer.py
new file mode 100644
index 0000000000..f619f11139
--- /dev/null
+++ b/docs/sphinx/qmp_lexer.py
@@ -0,0 +1,34 @@
+# QEMU Monitor Protocol Lexer Extension
+#
+# Copyright (C) 2019, Red Hat Inc.
+#
+# Authors:
+#  Eduardo Habkost <ehabkost@redhat.com>
+#  John Snow <jsnow@redhat.com>
+#
+# This work is licensed under the terms of the GNU GPL, version 2.  See
+# the COPYING file in the top-level directory.
+"""qmp_lexer is a Sphinx extension that provides a QMP lexer for code blocks."""
+
+from pygments.lexer import RegexLexer, DelegatingLexer
+from pygments.lexers.data import JsonLexer
+import pygments.token
+
+class QMPExampleMarkersLexer(RegexLexer):
+    """QMPExampleMarkersLexer highlights directionality flow indicators."""
+    tokens = {
+        'root': [
+            (r'-> ', pygments.token.Generic.Prompt),
+            (r'<- ', pygments.token.Generic.Prompt),
+        ]
+    }
+
+class QMPExampleLexer(DelegatingLexer):
+    """QMPExampleLexer highlights annotated QMP examples."""
+    def __init__(self, **options):
+        super(QMPExampleLexer, self).__init__(JsonLexer, QMPExampleMarkersLexer,
+                                              pygments.token.Error, **options)
+
+def setup(sphinx):
+    """For use by Sphinx() extensions API."""
+    sphinx.add_lexer("QMP", QMPExampleLexer())
-- 
2.20.1

On Tue, May 21, 2019 at 04:06:56PM -0400, John Snow wrote:
> Sphinx, through Pygments, does not like annotated json examples very
> much. In some versions of Sphinx (1.7), it will render the non-json
> portions of code blocks in red, but in newer versions (2.0) it will
> throw an exception and not highlight the block at all. Though we can
> suppress this warning, it doesn't bring back highlighting on non-strict
> json blocks.
> 
> We can alleviate this by creating a custom lexer for QMP examples that
> allows us to properly highlight these examples in a robust way, keeping
> our directionality notations.
> 
> Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
> Signed-off-by: John Snow <jsnow@redhat.com>

Thanks for figuring out how to integrate it into Sphinx!

Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>

-- 
Eduardo

On Tue, 21 May 2019 at 21:07, John Snow <jsnow@redhat.com> wrote:
>
> Sphinx, through Pygments, does not like annotated json examples very
> much. In some versions of Sphinx (1.7), it will render the non-json
> portions of code blocks in red, but in newer versions (2.0) it will
> throw an exception and not highlight the block at all. Though we can
> suppress this warning, it doesn't bring back highlighting on non-strict
> json blocks.
>
> We can alleviate this by creating a custom lexer for QMP examples that
> allows us to properly highlight these examples in a robust way, keeping
> our directionality notations.

> diff --git a/docs/sphinx/qmp_lexer.py b/docs/sphinx/qmp_lexer.py
> new file mode 100644
> index 0000000000..f619f11139
> --- /dev/null
> +++ b/docs/sphinx/qmp_lexer.py
> @@ -0,0 +1,34 @@
> +# QEMU Monitor Protocol Lexer Extension
> +#
> +# Copyright (C) 2019, Red Hat Inc.
> +#
> +# Authors:
> +#  Eduardo Habkost <ehabkost@redhat.com>
> +#  John Snow <jsnow@redhat.com>
> +#
> +# This work is licensed under the terms of the GNU GPL, version 2.  See
> +# the COPYING file in the top-level directory.

Did you definitely mean 2-only and not 2-or-later ?

thanks
-- PMM

On 5/22/19 4:49 AM, Peter Maydell wrote:
> On Tue, 21 May 2019 at 21:07, John Snow <jsnow@redhat.com> wrote:
>>
>> Sphinx, through Pygments, does not like annotated json examples very
>> much. In some versions of Sphinx (1.7), it will render the non-json
>> portions of code blocks in red, but in newer versions (2.0) it will
>> throw an exception and not highlight the block at all. Though we can
>> suppress this warning, it doesn't bring back highlighting on non-strict
>> json blocks.
>>
>> We can alleviate this by creating a custom lexer for QMP examples that
>> allows us to properly highlight these examples in a robust way, keeping
>> our directionality notations.
> 
>> diff --git a/docs/sphinx/qmp_lexer.py b/docs/sphinx/qmp_lexer.py
>> new file mode 100644
>> index 0000000000..f619f11139
>> --- /dev/null
>> +++ b/docs/sphinx/qmp_lexer.py
>> @@ -0,0 +1,34 @@
>> +# QEMU Monitor Protocol Lexer Extension
>> +#
>> +# Copyright (C) 2019, Red Hat Inc.
>> +#
>> +# Authors:
>> +#  Eduardo Habkost <ehabkost@redhat.com>
>> +#  John Snow <jsnow@redhat.com>
>> +#
>> +# This work is licensed under the terms of the GNU GPL, version 2.  See
>> +# the COPYING file in the top-level directory.
> 
> Did you definitely mean 2-only and not 2-or-later ?
> 
> thanks
> -- PMM
> 

Copy-paste pulled from another Python script. 2 or later is fine by me;
I can resend if desired (or, I'd be fine with anyone touching it up in
post.)

On Wed, 22 May 2019 at 20:02, John Snow <jsnow@redhat.com> wrote:
>
>
>
> On 5/22/19 4:49 AM, Peter Maydell wrote:
> > On Tue, 21 May 2019 at 21:07, John Snow <jsnow@redhat.com> wrote:
> >>
> >> Sphinx, through Pygments, does not like annotated json examples very
> >> much. In some versions of Sphinx (1.7), it will render the non-json
> >> portions of code blocks in red, but in newer versions (2.0) it will
> >> throw an exception and not highlight the block at all. Though we can
> >> suppress this warning, it doesn't bring back highlighting on non-strict
> >> json blocks.
> >>
> >> We can alleviate this by creating a custom lexer for QMP examples that
> >> allows us to properly highlight these examples in a robust way, keeping
> >> our directionality notations.
> >
> >> diff --git a/docs/sphinx/qmp_lexer.py b/docs/sphinx/qmp_lexer.py
> >> new file mode 100644
> >> index 0000000000..f619f11139
> >> --- /dev/null
> >> +++ b/docs/sphinx/qmp_lexer.py
> >> @@ -0,0 +1,34 @@
> >> +# QEMU Monitor Protocol Lexer Extension
> >> +#
> >> +# Copyright (C) 2019, Red Hat Inc.
> >> +#
> >> +# Authors:
> >> +#  Eduardo Habkost <ehabkost@redhat.com>
> >> +#  John Snow <jsnow@redhat.com>
> >> +#
> >> +# This work is licensed under the terms of the GNU GPL, version 2.  See
> >> +# the COPYING file in the top-level directory.
> >
> > Did you definitely mean 2-only and not 2-or-later ?

> Copy-paste pulled from another Python script. 2 or later is fine by me;
> I can resend if desired (or, I'd be fine with anyone touching it up in
> post.)

Our default-preference as a project is 2-or-later, so if
you're both happy with that I think we should use that.

thanks
-- PMM

On Wed, May 22, 2019 at 09:49:27AM +0100, Peter Maydell wrote:
> On Tue, 21 May 2019 at 21:07, John Snow <jsnow@redhat.com> wrote:
> >
> > Sphinx, through Pygments, does not like annotated json examples very
> > much. In some versions of Sphinx (1.7), it will render the non-json
> > portions of code blocks in red, but in newer versions (2.0) it will
> > throw an exception and not highlight the block at all. Though we can
> > suppress this warning, it doesn't bring back highlighting on non-strict
> > json blocks.
> >
> > We can alleviate this by creating a custom lexer for QMP examples that
> > allows us to properly highlight these examples in a robust way, keeping
> > our directionality notations.
> 
> > diff --git a/docs/sphinx/qmp_lexer.py b/docs/sphinx/qmp_lexer.py
> > new file mode 100644
> > index 0000000000..f619f11139
> > --- /dev/null
> > +++ b/docs/sphinx/qmp_lexer.py
> > @@ -0,0 +1,34 @@
> > +# QEMU Monitor Protocol Lexer Extension
> > +#
> > +# Copyright (C) 2019, Red Hat Inc.
> > +#
> > +# Authors:
> > +#  Eduardo Habkost <ehabkost@redhat.com>
> > +#  John Snow <jsnow@redhat.com>
> > +#
> > +# This work is licensed under the terms of the GNU GPL, version 2.  See
> > +# the COPYING file in the top-level directory.
> 
> Did you definitely mean 2-only and not 2-or-later ?

John asked if I was OK with GPLv2 before submitting the patch,
but I'm also OK with 2-or-later.

-- 
Eduardo