[v2] meson: Pass -j option to sphinx

[PATCH v2] meson: Pass -j option to sphinx

Posted by Fabiano Rosas 1 year ago

Save a bit of build time by passing the number of jobs option to
sphinx.

We cannot use the -j option from make because meson does not support
setting build time parameters for custom targets. Use nproc instead or
the equivalent sphinx option "-j auto", if that is available.

Also make sure our plugins support parallelism and report it properly
to sphinx. Particularly, implement the merge_domaindata method in
DBusDomain that is used to merge in data from other subprocesses.

before:
  $ time make man html
  ...
  [1/2] Generating docs/QEMU manual with a custom command
  [2/2] Generating docs/QEMU man pages with a custom command

  real    0m43.157s
  user    0m42.642s
  sys     0m0.576s

after:
  $ time make man html
  ...
  [1/2] Generating docs/QEMU manual with a custom command
  [2/2] Generating docs/QEMU man pages with a custom command

  real    0m25.014s
  user    0m51.288s
  sys     0m2.085s

Signed-off-by: Fabiano Rosas <farosas@suse.de>
---
 docs/meson.build           | 12 ++++++++++++
 docs/sphinx/dbusdomain.py  |  4 ++++
 docs/sphinx/fakedbusdoc.py |  5 +++++
 docs/sphinx/qmp_lexer.py   |  5 +++++
 4 files changed, 26 insertions(+)

diff --git a/docs/meson.build b/docs/meson.build
index f220800e3e..9e4bed6fa0 100644
--- a/docs/meson.build
+++ b/docs/meson.build
@@ -10,6 +10,18 @@ if sphinx_build.found()
     SPHINX_ARGS += [ '-W', '-Dkerneldoc_werror=1' ]
   endif
 
+  sphinx_version = run_command(SPHINX_ARGS + ['--version'],
+                               check: false).stdout().split()[1]
+  if sphinx_version.version_compare('>=5.1.2')
+    SPHINX_ARGS += ['-j', 'auto']
+  else
+    nproc = find_program('nproc')
+    if nproc.found()
+      jobs = run_command(nproc, check:false).stdout()
+      SPHINX_ARGS += ['-j', jobs]
+    endif
+  endif
+
   # This is a bit awkward but works: create a trivial document and
   # try to run it with our configuration file (which enforces a
   # version requirement). This will fail if sphinx-build is too old.
diff --git a/docs/sphinx/dbusdomain.py b/docs/sphinx/dbusdomain.py
index 2ea95af623..9872fd5bf6 100644
--- a/docs/sphinx/dbusdomain.py
+++ b/docs/sphinx/dbusdomain.py
@@ -400,6 +400,10 @@ def get_objects(self) -> Iterator[Tuple[str, str, str, str, str, int]]:
         for refname, obj in self.objects.items():
             yield (refname, refname, obj.objtype, obj.docname, obj.node_id, 1)
 
+    def merge_domaindata(self, docnames, otherdata):
+        for name, obj in otherdata['objects'].items():
+            if obj.docname in docnames:
+                self.data['objects'][name] = obj
 
 def setup(app):
     app.add_domain(DBusDomain)
diff --git a/docs/sphinx/fakedbusdoc.py b/docs/sphinx/fakedbusdoc.py
index d2c5079046..2d2e6ef640 100644
--- a/docs/sphinx/fakedbusdoc.py
+++ b/docs/sphinx/fakedbusdoc.py
@@ -23,3 +23,8 @@ def run(self):
 def setup(app: Sphinx) -> Dict[str, Any]:
     """Register a fake dbus-doc directive with Sphinx"""
     app.add_directive("dbus-doc", FakeDBusDocDirective)
+
+    return dict(
+        parallel_read_safe = True,
+        parallel_write_safe = True
+    )
diff --git a/docs/sphinx/qmp_lexer.py b/docs/sphinx/qmp_lexer.py
index f7e4c0e198..a59de8a079 100644
--- a/docs/sphinx/qmp_lexer.py
+++ b/docs/sphinx/qmp_lexer.py
@@ -41,3 +41,8 @@ def setup(sphinx):
         sphinx.add_lexer('QMP', QMPExampleLexer)
     except errors.VersionRequirementError:
         sphinx.add_lexer('QMP', QMPExampleLexer())
+
+    return dict(
+        parallel_read_safe = True,
+        parallel_write_safe = True
+    )
-- 
2.35.3

Re: [PATCH v2] meson: Pass -j option to sphinx

Posted by Daniel P. Berrangé 1 year ago

On Thu, Apr 27, 2023 at 02:25:16PM -0300, Fabiano Rosas wrote:
> Save a bit of build time by passing the number of jobs option to
> sphinx.
> 
> We cannot use the -j option from make because meson does not support
> setting build time parameters for custom targets. Use nproc instead or
> the equivalent sphinx option "-j auto", if that is available.
> 
> Also make sure our plugins support parallelism and report it properly
> to sphinx. Particularly, implement the merge_domaindata method in
> DBusDomain that is used to merge in data from other subprocesses.
> 
> before:
>   $ time make man html
>   ...
>   [1/2] Generating docs/QEMU manual with a custom command
>   [2/2] Generating docs/QEMU man pages with a custom command
> 
>   real    0m43.157s
>   user    0m42.642s
>   sys     0m0.576s
> 
> after:
>   $ time make man html
>   ...
>   [1/2] Generating docs/QEMU manual with a custom command
>   [2/2] Generating docs/QEMU man pages with a custom command
> 
>   real    0m25.014s
>   user    0m51.288s
>   sys     0m2.085s

On my 12 CPU laptop I see a similar magnitude benefit - about
20 seconds is cut from the docs build time - 50 down to 30 secs.

Watching the CPU usage I see sphinx is not very good at keeping
all CPUs busy. For perhaps 2 seconds I'll see  8 sphinx processes
burning CPUs, but the majority of the time it'll only be 1 or 2
sphinx processes.

IOW, we do get a benefit, but it is not nearly as good as one
might hope for given the number of CPUs potentially available.

> Signed-off-by: Fabiano Rosas <farosas@suse.de>
> ---
>  docs/meson.build           | 12 ++++++++++++
>  docs/sphinx/dbusdomain.py  |  4 ++++
>  docs/sphinx/fakedbusdoc.py |  5 +++++
>  docs/sphinx/qmp_lexer.py   |  5 +++++
>  4 files changed, 26 insertions(+)

Tested-by: Daniel P. Berrangé <berrange@redhat.com>

> 
> diff --git a/docs/meson.build b/docs/meson.build
> index f220800e3e..9e4bed6fa0 100644
> --- a/docs/meson.build
> +++ b/docs/meson.build
> @@ -10,6 +10,18 @@ if sphinx_build.found()
>      SPHINX_ARGS += [ '-W', '-Dkerneldoc_werror=1' ]
>    endif
>  
> +  sphinx_version = run_command(SPHINX_ARGS + ['--version'],
> +                               check: false).stdout().split()[1]
> +  if sphinx_version.version_compare('>=5.1.2')
> +    SPHINX_ARGS += ['-j', 'auto']
> +  else
> +    nproc = find_program('nproc')
> +    if nproc.found()
> +      jobs = run_command(nproc, check:false).stdout()
> +      SPHINX_ARGS += ['-j', jobs]
> +    endif
> +  endif

ANy reason for check: false in these 2 run_command calls ?

They'll both return 0 on success, so I would have though
'check: true' was more robust at error reporting ?


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v2] meson: Pass -j option to sphinx

Posted by Fabiano Rosas 1 year ago

Daniel P. Berrangé <berrange@redhat.com> writes:

> On Thu, Apr 27, 2023 at 02:25:16PM -0300, Fabiano Rosas wrote:
>> Save a bit of build time by passing the number of jobs option to
>> sphinx.
>> 
>> We cannot use the -j option from make because meson does not support
>> setting build time parameters for custom targets. Use nproc instead or
>> the equivalent sphinx option "-j auto", if that is available.
>> 
>> Also make sure our plugins support parallelism and report it properly
>> to sphinx. Particularly, implement the merge_domaindata method in
>> DBusDomain that is used to merge in data from other subprocesses.
>> 
>> before:
>>   $ time make man html
>>   ...
>>   [1/2] Generating docs/QEMU manual with a custom command
>>   [2/2] Generating docs/QEMU man pages with a custom command
>> 
>>   real    0m43.157s
>>   user    0m42.642s
>>   sys     0m0.576s
>> 
>> after:
>>   $ time make man html
>>   ...
>>   [1/2] Generating docs/QEMU manual with a custom command
>>   [2/2] Generating docs/QEMU man pages with a custom command
>> 
>>   real    0m25.014s
>>   user    0m51.288s
>>   sys     0m2.085s
>
> On my 12 CPU laptop I see a similar magnitude benefit - about
> 20 seconds is cut from the docs build time - 50 down to 30 secs.
>
> Watching the CPU usage I see sphinx is not very good at keeping
> all CPUs busy. For perhaps 2 seconds I'll see  8 sphinx processes
> burning CPUs, but the majority of the time it'll only be 1 or 2
> sphinx processes.
>
> IOW, we do get a benefit, but it is not nearly as good as one
> might hope for given the number of CPUs potentially available.
>
>> Signed-off-by: Fabiano Rosas <farosas@suse.de>
>> ---
>>  docs/meson.build           | 12 ++++++++++++
>>  docs/sphinx/dbusdomain.py  |  4 ++++
>>  docs/sphinx/fakedbusdoc.py |  5 +++++
>>  docs/sphinx/qmp_lexer.py   |  5 +++++
>>  4 files changed, 26 insertions(+)
>
> Tested-by: Daniel P. Berrangé <berrange@redhat.com>
>
>> 
>> diff --git a/docs/meson.build b/docs/meson.build
>> index f220800e3e..9e4bed6fa0 100644
>> --- a/docs/meson.build
>> +++ b/docs/meson.build
>> @@ -10,6 +10,18 @@ if sphinx_build.found()
>>      SPHINX_ARGS += [ '-W', '-Dkerneldoc_werror=1' ]
>>    endif
>>  
>> +  sphinx_version = run_command(SPHINX_ARGS + ['--version'],
>> +                               check: false).stdout().split()[1]
>> +  if sphinx_version.version_compare('>=5.1.2')
>> +    SPHINX_ARGS += ['-j', 'auto']
>> +  else
>> +    nproc = find_program('nproc')
>> +    if nproc.found()
>> +      jobs = run_command(nproc, check:false).stdout()
>> +      SPHINX_ARGS += ['-j', jobs]
>> +    endif
>> +  endif
>
> ANy reason for check: false in these 2 run_command calls ?
>

No, I haven't thought about it. I'll change them to true.