This serie introduce doxygen in the sphinx html docs generation.
One benefit is to keep most of the documentation in the source
files of xen so that it's more maintainable, on the other hand
there are some limitation of doxygen that must be addressed
modifying the current codebase (for example doxygen can't parse
anonymous structure/union).

To reproduce the documentation xen must be compiled because
most of the headers are generated on compilation time from
the makefiles.

Here follows the steps to generate the sphinx html docs, some
package may be required on your machine, everything is suggested
by the autoconf script.
Here I'm building the arm64 docs (the only introduced for now by
this serie):

./configure
make -C xen XEN_TARGET_ARCH="arm64" CROSS_COMPILE="aarch64-linux-gnu-" menuconfig
make -C xen XEN_TARGET_ARCH="arm64" CROSS_COMPILE="aarch64-linux-gnu-"
make -C docs XEN_TARGET_ARCH="arm64" sphinx-html

now in docs/sphinx/html/ we have the generated docs starting
from the index.html page.


Luca Fancellu (3):
  docs: add doxygen support for html documentation
  docs: hypercalls sphinx skeleton for generated html
  docs/doxygen: doxygen documentation for grant_table.h

 .gitignore                                    |    7 +
 config/Docs.mk.in                             |    2 +
 docs/Makefile                                 |   51 +-
 docs/conf.py                                  |   48 +-
 docs/configure                                |  258 ++
 docs/configure.ac                             |   15 +
 docs/hypercall-interfaces/arm32.rst           |    4 +
 docs/hypercall-interfaces/arm64.rst           |   33 +
 .../arm64/grant_tables.rst                    |    8 +
 docs/hypercall-interfaces/index.rst.in        |    7 +
 docs/hypercall-interfaces/x86_64.rst          |    4 +
 docs/index.rst                                |    8 +
 docs/xen-doxygen/customdoxygen.css            |   36 +
 docs/xen-doxygen/doxy_input.list              |    1 +
 docs/xen-doxygen/doxy_input_template.in       |   21 +
 docs/xen-doxygen/footer.html                  |   21 +
 docs/xen-doxygen/header.html                  |   56 +
 docs/xen-doxygen/mainpage.md                  |    5 +
 docs/xen-doxygen/xen_project_logo_165x67.png  |  Bin 0 -> 18223 bytes
 docs/xen.doxyfile.in                          | 2314 +++++++++++++++++
 m4/ax_python_module.m4                        |   56 +
 m4/docs_tool.m4                               |    9 +
 xen/include/public/grant_table.h              |   79 +-
 23 files changed, 3007 insertions(+), 36 deletions(-)
 create mode 100644 docs/hypercall-interfaces/arm32.rst
 create mode 100644 docs/hypercall-interfaces/arm64.rst
 create mode 100644 docs/hypercall-interfaces/arm64/grant_tables.rst
 create mode 100644 docs/hypercall-interfaces/index.rst.in
 create mode 100644 docs/hypercall-interfaces/x86_64.rst
 create mode 100644 docs/xen-doxygen/customdoxygen.css
 create mode 100644 docs/xen-doxygen/doxy_input.list
 create mode 100644 docs/xen-doxygen/doxy_input_template.in
 create mode 100644 docs/xen-doxygen/footer.html
 create mode 100644 docs/xen-doxygen/header.html
 create mode 100644 docs/xen-doxygen/mainpage.md
 create mode 100644 docs/xen-doxygen/xen_project_logo_165x67.png
 create mode 100644 docs/xen.doxyfile.in
 create mode 100644 m4/ax_python_module.m4

-- 
2.17.1

On 06/04/2021 11:36, Luca Fancellu wrote:
> This serie introduce doxygen in the sphinx html docs generation.
> One benefit is to keep most of the documentation in the source
> files of xen so that it's more maintainable, on the other hand
> there are some limitation of doxygen that must be addressed
> modifying the current codebase (for example doxygen can't parse
> anonymous structure/union).
> 
> To reproduce the documentation xen must be compiled because
> most of the headers are generated on compilation time from
> the makefiles.
> 
> Here follows the steps to generate the sphinx html docs, some
> package may be required on your machine, everything is suggested
> by the autoconf script.
> Here I'm building the arm64 docs (the only introduced for now by
> this serie):
> 
> ./configure
> make -C xen XEN_TARGET_ARCH="arm64" CROSS_COMPILE="aarch64-linux-gnu-" menuconfig
> make -C xen XEN_TARGET_ARCH="arm64" CROSS_COMPILE="aarch64-linux-gnu-"
> make -C docs XEN_TARGET_ARCH="arm64" sphinx-html

I have tried this instruction and got:

make: Entering directory '/home/ANT.AMAZON.COM/jgrall/works/oss/xen/docs'
xen.doxyfile
make: xen.doxyfile: Command not found
Makefile:67: recipe for target 'sphinx-html' failed
make: *** [sphinx-html] Error 127
make: Leaving directory '/home/ANT.AMAZON.COM/jgrall/works/oss/xen/docs'

AFAICT, $DOXYGEN will only get defined when sphinx-build is installed.
When sphinx-build is not installed SPHINXBUILD will be equal to 'no', 
but the Makefile check for emptiness.

Cheers,

-- 
Julien Grall

> On 7 Apr 2021, at 14:07, Julien Grall <julien@xen.org> wrote:
> 
> 
> 
> On 06/04/2021 11:36, Luca Fancellu wrote:
>> This serie introduce doxygen in the sphinx html docs generation.
>> One benefit is to keep most of the documentation in the source
>> files of xen so that it's more maintainable, on the other hand
>> there are some limitation of doxygen that must be addressed
>> modifying the current codebase (for example doxygen can't parse
>> anonymous structure/union).
>> To reproduce the documentation xen must be compiled because
>> most of the headers are generated on compilation time from
>> the makefiles.
>> Here follows the steps to generate the sphinx html docs, some
>> package may be required on your machine, everything is suggested
>> by the autoconf script.
>> Here I'm building the arm64 docs (the only introduced for now by
>> this serie):
>> ./configure
>> make -C xen XEN_TARGET_ARCH="arm64" CROSS_COMPILE="aarch64-linux-gnu-" menuconfig
>> make -C xen XEN_TARGET_ARCH="arm64" CROSS_COMPILE="aarch64-linux-gnu-"
>> make -C docs XEN_TARGET_ARCH="arm64" sphinx-html
> 
> I have tried this instruction and got:
> 
> make: Entering directory '/home/ANT.AMAZON.COM/jgrall/works/oss/xen/docs'
> xen.doxyfile
> make: xen.doxyfile: Command not found
> Makefile:67: recipe for target 'sphinx-html' failed
> make: *** [sphinx-html] Error 127
> make: Leaving directory '/home/ANT.AMAZON.COM/jgrall/works/oss/xen/docs'
> 
> AFAICT, $DOXYGEN will only get defined when sphinx-build is installed.
> When sphinx-build is not installed SPHINXBUILD will be equal to 'no', but the Makefile check for emptiness.
> 

Hi Julien,

Thank you for spotting it, I’ll fix it in the v2 patch

Cheers,
Luca

> Cheers,
> 
> -- 
> Julien Grall

On 06/04/2021 11:36, Luca Fancellu wrote:
> This serie introduce doxygen in the sphinx html docs generation.
> One benefit is to keep most of the documentation in the source
> files of xen so that it's more maintainable, on the other hand
> there are some limitation of doxygen that must be addressed
> modifying the current codebase (for example doxygen can't parse
> anonymous structure/union).
>
> To reproduce the documentation xen must be compiled because
> most of the headers are generated on compilation time from
> the makefiles.
>
> Here follows the steps to generate the sphinx html docs, some
> package may be required on your machine, everything is suggested
> by the autoconf script.
> Here I'm building the arm64 docs (the only introduced for now by
> this serie):
>
> ./configure
> make -C xen XEN_TARGET_ARCH="arm64" CROSS_COMPILE="aarch64-linux-gnu-" menuconfig
> make -C xen XEN_TARGET_ARCH="arm64" CROSS_COMPILE="aarch64-linux-gnu-"
> make -C docs XEN_TARGET_ARCH="arm64" sphinx-html
>
> now in docs/sphinx/html/ we have the generated docs starting
> from the index.html page.

Do you have a sample rendered output?

The plan was to try and use Linux's kernel-doc plugin for Sphinx, which
is very doxygen-like.  Did you experiment with this option?

~Andrew

On Tue, 6 Apr 2021, Andrew Cooper wrote:
> On 06/04/2021 11:36, Luca Fancellu wrote:
> > This serie introduce doxygen in the sphinx html docs generation.
> > One benefit is to keep most of the documentation in the source
> > files of xen so that it's more maintainable, on the other hand
> > there are some limitation of doxygen that must be addressed
> > modifying the current codebase (for example doxygen can't parse
> > anonymous structure/union).
> >
> > To reproduce the documentation xen must be compiled because
> > most of the headers are generated on compilation time from
> > the makefiles.
> >
> > Here follows the steps to generate the sphinx html docs, some
> > package may be required on your machine, everything is suggested
> > by the autoconf script.
> > Here I'm building the arm64 docs (the only introduced for now by
> > this serie):
> >
> > ./configure
> > make -C xen XEN_TARGET_ARCH="arm64" CROSS_COMPILE="aarch64-linux-gnu-" menuconfig
> > make -C xen XEN_TARGET_ARCH="arm64" CROSS_COMPILE="aarch64-linux-gnu-"
> > make -C docs XEN_TARGET_ARCH="arm64" sphinx-html
> >
> > now in docs/sphinx/html/ we have the generated docs starting
> > from the index.html page.
> 
> Do you have a sample rendered output?
> 
> The plan was to try and use Linux's kernel-doc plugin for Sphinx, which
> is very doxygen-like.  Did you experiment with this option?

As you probably know the end goal for Luca (and the Xen FuSa SIG as a
whole) is to generate all FuSa documents, including requirements docs,
interface docs, etc.

FuSa requires us to follow the famous V model, where the high level
requirements are linked to the lower level requirements, which are
linked to the interface docs, which are linked all the way down to the
code.

Maintaining the linking is difficult and typically done with expensive
proprietary FuSa tools.

Fortunately, an engineer working with the Zephyr project developed a set
of scripts for Doxygen that are able to generate the required FuSa docs
and also links from in-code comments and markdown/rst docs in the tree.

This is great work, and in the FuSa SIG we thought it would be best to
align ourselves with Zephyr to be able to pull our efforts together on
the tooling side instead of doing the same thing again on our own.

This is the reason why we ended up with Doxygen.

On 06/04/2021 22:24, Stefano Stabellini wrote:
> On Tue, 6 Apr 2021, Andrew Cooper wrote:
>> On 06/04/2021 11:36, Luca Fancellu wrote:
>>> This serie introduce doxygen in the sphinx html docs generation.
>>> One benefit is to keep most of the documentation in the source
>>> files of xen so that it's more maintainable, on the other hand
>>> there are some limitation of doxygen that must be addressed
>>> modifying the current codebase (for example doxygen can't parse
>>> anonymous structure/union).
>>>
>>> To reproduce the documentation xen must be compiled because
>>> most of the headers are generated on compilation time from
>>> the makefiles.
>>>
>>> Here follows the steps to generate the sphinx html docs, some
>>> package may be required on your machine, everything is suggested
>>> by the autoconf script.
>>> Here I'm building the arm64 docs (the only introduced for now by
>>> this serie):
>>>
>>> ./configure
>>> make -C xen XEN_TARGET_ARCH="arm64" CROSS_COMPILE="aarch64-linux-gnu-" menuconfig
>>> make -C xen XEN_TARGET_ARCH="arm64" CROSS_COMPILE="aarch64-linux-gnu-"
>>> make -C docs XEN_TARGET_ARCH="arm64" sphinx-html
>>>
>>> now in docs/sphinx/html/ we have the generated docs starting
>>> from the index.html page.
>> Do you have a sample rendered output?
>>
>> The plan was to try and use Linux's kernel-doc plugin for Sphinx, which
>> is very doxygen-like.  Did you experiment with this option?
> As you probably know the end goal for Luca (and the Xen FuSa SIG as a
> whole) is to generate all FuSa documents, including requirements docs,
> interface docs, etc.
>
> FuSa requires us to follow the famous V model, where the high level
> requirements are linked to the lower level requirements, which are
> linked to the interface docs, which are linked all the way down to the
> code.
>
> Maintaining the linking is difficult and typically done with expensive
> proprietary FuSa tools.
>
> Fortunately, an engineer working with the Zephyr project developed a set
> of scripts for Doxygen that are able to generate the required FuSa docs
> and also links from in-code comments and markdown/rst docs in the tree.
>
> This is great work, and in the FuSa SIG we thought it would be best to
> align ourselves with Zephyr to be able to pull our efforts together on
> the tooling side instead of doing the same thing again on our own.
>
> This is the reason why we ended up with Doxygen.

So are we're saying that Doxygen is a hard dependency because there is
an extension for Doxygen to generate other safety docs?

~Andrew

Hi Andrew,

> On 7 Apr 2021, at 12:15, Andrew Cooper <andrew.cooper3@citrix.com> wrote:
> 
> On 06/04/2021 22:24, Stefano Stabellini wrote:
>> On Tue, 6 Apr 2021, Andrew Cooper wrote:
>>> On 06/04/2021 11:36, Luca Fancellu wrote:
>>>> This serie introduce doxygen in the sphinx html docs generation.
>>>> One benefit is to keep most of the documentation in the source
>>>> files of xen so that it's more maintainable, on the other hand
>>>> there are some limitation of doxygen that must be addressed
>>>> modifying the current codebase (for example doxygen can't parse
>>>> anonymous structure/union).
>>>> 
>>>> To reproduce the documentation xen must be compiled because
>>>> most of the headers are generated on compilation time from
>>>> the makefiles.
>>>> 
>>>> Here follows the steps to generate the sphinx html docs, some
>>>> package may be required on your machine, everything is suggested
>>>> by the autoconf script.
>>>> Here I'm building the arm64 docs (the only introduced for now by
>>>> this serie):
>>>> 
>>>> ./configure
>>>> make -C xen XEN_TARGET_ARCH="arm64" CROSS_COMPILE="aarch64-linux-gnu-" menuconfig
>>>> make -C xen XEN_TARGET_ARCH="arm64" CROSS_COMPILE="aarch64-linux-gnu-"
>>>> make -C docs XEN_TARGET_ARCH="arm64" sphinx-html
>>>> 
>>>> now in docs/sphinx/html/ we have the generated docs starting
>>>> from the index.html page.
>>> Do you have a sample rendered output?
>>> 
>>> The plan was to try and use Linux's kernel-doc plugin for Sphinx, which
>>> is very doxygen-like.  Did you experiment with this option?
>> As you probably know the end goal for Luca (and the Xen FuSa SIG as a
>> whole) is to generate all FuSa documents, including requirements docs,
>> interface docs, etc.
>> 
>> FuSa requires us to follow the famous V model, where the high level
>> requirements are linked to the lower level requirements, which are
>> linked to the interface docs, which are linked all the way down to the
>> code.
>> 
>> Maintaining the linking is difficult and typically done with expensive
>> proprietary FuSa tools.
>> 
>> Fortunately, an engineer working with the Zephyr project developed a set
>> of scripts for Doxygen that are able to generate the required FuSa docs
>> and also links from in-code comments and markdown/rst docs in the tree.
>> 
>> This is great work, and in the FuSa SIG we thought it would be best to
>> align ourselves with Zephyr to be able to pull our efforts together on
>> the tooling side instead of doing the same thing again on our own.
>> 
>> This is the reason why we ended up with Doxygen.
> 
> So are we're saying that Doxygen is a hard dependency because there is
> an extension for Doxygen to generate other safety docs?

hard no as we could find other solution but the current strategy we take together
with Zephyr project is based on using Doxygen for requirement linking.

Also an other argument is that the documentation generated is actually nice and
that could also be useful for developers and users (see [1] for Zephyr doc).

Our global idea is also that using doxygen we can extract a big part of the
documentation (which will partly be used as certification) directly from the
code which will make it a lot easier for developers to maintain.

Regards
Bertrand


[1] https://docs.zephyrproject.org/latest/


> 
> ~Andrew
>