Currently we don't have much information on how testing is done in
libvirt and the little we have is scattered among multiple files. This
patch creates a common landing page containing all important bits about
testing in libvirt, providing links to respective sections which
deserve their own articles.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
---
 docs/meson.build |   1 +
 docs/testing.rst | 176 +++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 177 insertions(+)
 create mode 100644 docs/testing.rst

diff --git a/docs/meson.build b/docs/meson.build
index 0c7c54a75f..6d61e18385 100644
--- a/docs/meson.build
+++ b/docs/meson.build
@@ -105,6 +105,7 @@ docs_rst_files = [
   'submitting-patches',
   'support',
   'testapi',
+  'testing',
   'testsuites',
   'testtck',
   'uri',
diff --git a/docs/testing.rst b/docs/testing.rst
new file mode 100644
index 0000000000..af0f334633
--- /dev/null
+++ b/docs/testing.rst
@@ -0,0 +1,176 @@
+=======
+Testing
+=======
+
+.. contents::
+
+Different types of tests are available to libvirt developers for testing a
+given libvirt release.
+
+Unit tests
+----------
+
+The unit test suite present in the source code is mainly used to test our
+XML parser/formatter, QEMU command line generator, QEMU capabilities probing,
+etc. It is run by developers before submitting patches upstream and is
+mandatory to pass for any contribution to be accepted upstream. One can run
+the test suite in the source tree with the following
+
+::
+
+    make check (libvirt 6.6.0 and older)
+
+::
+
+    ninja test (libvirt 6.7.0 and newer)
+
+
+Container builds
+----------------
+
+Technically speaking these are not tests in the common sense. However, usage of
+public container images to build libvirt in predefined and widely accessible
+environments makes it possible to expand our coverage across distros,
+architectures, toolchain flavors and library versions and as such is a very
+valuable marker when accepting upstream contributions. Therefore, it is
+recommended to run libvirt builds against your changes in various containers to
+either locally or by using GitLab's shared CI runners to make sure everything
+runs cleanly before submitting your patches. The images themselves come from
+libvirt's GitLab container registry, but this can be overriden if needed, see
+below.
+
+Registry
+~~~~~~~~
+
+Libvirt project has its container registry hosted by GitLab at
+``registry.gitlab.com/libvirt/libvirt`` which will automatically be
+used to pull in pre-built layers. This avoids the need to build all the
+containers locally using the Dockerfile recipes found in ``ci/containers/``.
+
+
+Running container builds with GitLab CI
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+As long as your GitLab account has CI minutes available, pipelines will run
+automatically on every branch push to your fork.
+
+Running container builds locally
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+In order to run container builds locally, we have a ``helper`` script inside
+the ``ci`` directory that can pull, build, and test (if applicable) changes on
+your current local branch. It supports both the Docker and Podman runtimes
+with an automatic selection of whichever runtime is configured on your system.
+In case neither has been enabled/configured, please go through the following
+prerequisites.
+
+Docker Prerequisites
+~~~~~~~~~~~~~~~~~~~~
+
+Install "docker" with the system package manager and start the Docker service
+on your development machine, then make sure you have the privilege to run
+Docker commands. Typically it means setting up passwordless ``sudo docker``
+command or login as root. For example:
+
+.. code::
+
+  $ sudo yum install docker
+  $ # or `apt-get install docker` for Ubuntu, etc.
+  $ sudo systemctl start docker
+  $ sudo docker ps
+
+The last command should print an empty table, to verify the system is ready.
+
+An alternative method to set up permissions is by adding the current user to
+"docker" group and making the docker daemon socket file (by default
+``/var/run/docker.sock``) accessible to the group:
+
+.. code::
+
+  $ sudo groupadd docker
+  $ sudo usermod $USER -a -G docker
+  $ sudo chown :docker /var/run/docker.sock
+
+Note that any one of above configurations makes it possible for the user to
+exploit the whole host with Docker bind mounting or other privileged
+operations.  So only do it on development machines.
+
+Podman Prerequisites
+~~~~~~~~~~~~~~~~~~~~
+
+Install "podman" with the system package manager.
+
+.. code::
+
+  $ sudo dnf install podman
+  $ podman ps
+
+The last command should print an empty table, to verify the system is ready.
+
+Examples of executing local container builds
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+All of the following examples will utilize ``helper`` script mentioned earlier
+sections. Let's start with the basics - listing available container images in
+the default libvirt registry:
+
+::
+
+    $ cd <libvirt_git>/ci
+    $ ./helper --help
+    $ ./helper list-images
+    Available x86 container images:
+
+    ...
+    alpine-edge
+    fedora-rawhide
+    ...
+
+    Available cross-compiler container images:
+
+    ...
+    debian-sid-cross-s390x
+    fedora-rawhide-cross-mingw32
+    fedora-rawhide-cross-mingw64
+    ...
+
+Now let's say one would want to build their local libvirt changes on Alpine
+Edge using their own GitLab's registry container. They'd then proceed with
+
+::
+
+    $ ci/helper build --image-prefix registry.gitlab.com/<user>/libvirt/ci- alpine-edge
+
+Finally, it would be nice if one could get an interactive shell inside the
+test environment to debug potential build issues. This can be achieved with the
+following:
+
+::
+
+    $ ci/helper shell alpine-edge
+
+
+Integration tests
+-----------------
+
+There are a few frameworks for writing and running functional tests in libvirt
+with TCK being the one that runs in our upstream CI.
+
+-  the `TCK test suite <testtck.html>`__ is a functional test suite implemented
+   using the `Perl bindings <https://search.cpan.org/dist/Sys-Virt/>`__ of
+   libvirt. This is the recommended framework to use for writing upstream
+   functional tests at the moment. You can start by cloning the
+   `TCK git repo <https://gitlab.com/libvirt/libvirt-tck>`__.
+
+-  the `Avocado VT <https://github.com/avocado-framework/avocado-vt>`__ test
+   suite with the libvirt plugin is another framework implementing functional
+   testing utilizing the Avocado test framework underneath. Although written in
+   Python, the vast majority of the tests are exercising libvirt through the
+   command line client ``virsh``.
+
+-  the `libvirt-test-API <testapi.html>`__ is also a functional test suite, but
+   implemented using the `Python bindings <python.html>`__ of libvirt.
+   Unfortunately this framework is the least recommended one as it's largely
+   unmaintained and may be completely deprecated in the future in favour of TCK.
+   You can get it by cloning the
+   `git repo <https://gitlab.com/libvirt/libvirt-test-API/>`__.
-- 
2.36.1

On Tue, Jul 12, 2022 at 02:44:36PM +0200, Erik Skultety wrote:
> Currently we don't have much information on how testing is done in
> libvirt and the little we have is scattered among multiple files. This
> patch creates a common landing page containing all important bits about
> testing in libvirt, providing links to respective sections which
> deserve their own articles.
> 
> Signed-off-by: Erik Skultety <eskultet@redhat.com>
> ---
>  docs/meson.build |   1 +
>  docs/testing.rst | 176 +++++++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 177 insertions(+)
>  create mode 100644 docs/testing.rst
> 
> diff --git a/docs/meson.build b/docs/meson.build
> index 0c7c54a75f..6d61e18385 100644
> --- a/docs/meson.build
> +++ b/docs/meson.build
> @@ -105,6 +105,7 @@ docs_rst_files = [
>    'submitting-patches',
>    'support',
>    'testapi',
> +  'testing',
>    'testsuites',
>    'testtck',
>    'uri',
> diff --git a/docs/testing.rst b/docs/testing.rst
> new file mode 100644
> index 0000000000..af0f334633
> --- /dev/null
> +++ b/docs/testing.rst
> @@ -0,0 +1,176 @@
> +=======
> +Testing
> +=======
> +
> +.. contents::
> +
> +Different types of tests are available to libvirt developers for testing a
> +given libvirt release.
> +
> +Unit tests
> +----------
> +
> +The unit test suite present in the source code is mainly used to test our
> +XML parser/formatter, QEMU command line generator, QEMU capabilities probing,
> +etc. It is run by developers before submitting patches upstream and is
> +mandatory to pass for any contribution to be accepted upstream. One can run
> +the test suite in the source tree with the following
> +
> +::
> +
> +    make check (libvirt 6.6.0 and older)
> +

IMHO we can ignore documenting such old releases. 

> +::
> +
> +    ninja test (libvirt 6.7.0 and newer)
> +
> +



> +Docker Prerequisites
> +~~~~~~~~~~~~~~~~~~~~
> +
> +Install "docker" with the system package manager and start the Docker service
> +on your development machine, then make sure you have the privilege to run
> +Docker commands. Typically it means setting up passwordless ``sudo docker``
> +command or login as root. For example:
> +
> +.. code::
> +
> +  $ sudo yum install docker

s/yum/dnf/   and perhaps also   'install -y'

> +  $ # or `apt-get install docker` for Ubuntu, etc.
> +  $ sudo systemctl start docker
> +  $ sudo docker ps
> +
> +The last command should print an empty table, to verify the system is ready.
> +
> +An alternative method to set up permissions is by adding the current user to
> +"docker" group and making the docker daemon socket file (by default
> +``/var/run/docker.sock``) accessible to the group:
> +
> +.. code::
> +
> +  $ sudo groupadd docker
> +  $ sudo usermod $USER -a -G docker
> +  $ sudo chown :docker /var/run/docker.sock
> +
> +Note that any one of above configurations makes it possible for the user to
> +exploit the whole host with Docker bind mounting or other privileged
> +operations.  So only do it on development machines.
> +
> +Podman Prerequisites
> +~~~~~~~~~~~~~~~~~~~~
> +
> +Install "podman" with the system package manager.
> +
> +.. code::
> +
> +  $ sudo dnf install podman

  Also 'install -y'

> +  $ podman ps
> +
> +The last command should print an empty table, to verify the system is ready.

Given it has much simpler setup, and doesn't open a local root shell
I'd suggest we document podman first, and explicitly recommend its
usage, over docker.




With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|