From: Tomeu Vizoso <tomeu.vizoso@collabora.com>

Developers can easily execute several tests on different devices
by just pushing their branch to their fork in a repository hosted
on gitlab.freedesktop.org which has an infrastructure to run jobs
in several runners and farms with different devices.

There are also other automated tools that uprev dependencies,
monitor the infra, and so on that are already used by the Mesa
project, and we can reuse them too.

Also, store expectations about what the DRM drivers are supposed
to pass in the IGT test suite. By storing the test expectations
along with the code, we can make sure both stay in sync with each
other so we can know when a code change breaks those expectations.

Also, include a configuration file that points to the out-of-tree
CI scripts.

This will allow all contributors to drm to reuse the infrastructure
already in gitlab.freedesktop.org to test the driver on several
generations of the hardware.

Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Signed-off-by: Helen Koike <helen.koike@collabora.com>
Acked-by: Daniel Stone <daniels@collabora.com>
Acked-by: Rob Clark <robdclark@gmail.com>
Tested-by: Rob Clark <robdclark@gmail.com>
---

You can also see this patch on
https://gitlab.freedesktop.org/helen.fornazier/linux/-/tree/drm-ci-tests

Thanks!

v2:
  - Fix names of result expectation files to match SoC
  - Don't execute tests that are going to skip on all boards

v3:
  - Remove tracking of dmesg output during test execution

v4:
  - Move up to drivers/gpu/drm
  - Add support for a bunch of other drivers
  - Explain how to incorporate fixes for CI from a
    ${TARGET_BRANCH}-external-fixes branch
  - Remove tests that pass from expected results file, to reduce the
    size of in-tree files
  - Add docs about how to deal with outages in automated testing labs
  - Specify the exact SHA of the CI scripts to be used

v5:
  - Remove unneeded skips from Meson expectations file
  - Use a more advanced runner that detects flakes automatically
  - Use a more succint format for the expectations
  - Run many more tests (and use sharding to finish in time)
  - Use skip lists to avoid hanging machines
  - Add some build testing
  - Build IGT in each pipeline for faster uprevs
  - List failures in the GitLab UI

v6:
  - Rebase on top of latest drm-next
  - Lower priority of LAVA jobs to not impact Mesa CI as much
  - Update docs

v7:
  - Rebase on top of latest drm-next

v8:
  - Move all files specific to testing the kernel into the kernel tree
    (thus I have dropped the r-bs I had collected so far)
  - Uprev Gitlab CI infrastructure scripts to the latest from Mesa
  - Add MAINTAINERS entry
  - Fix boot on MT8173 by adding some Kconfigs that are now needed
  - Link to the docs from index.rst and hard-wrap the file

v9:
  - Only automatically run the pipelines for merge requests
  - Switch to zstd for the build artifacts to align with Mesa
  - Add Qcom USB PHYs to config as they are now =m in the defconfig

v10:
  - Include ci yml files from mesa/mesa (where the development is
    current active) instead of a spin off project.
  - Uprev Gitlab CI infrastructure scripts to the latest from Mesa
  - Update MAINTAINERS entry
  - Uprev igt tool
  - add LAVA_JOB_PRIORITY: 30
  - pipeline example:
  https://gitlab.freedesktop.org/helen.fornazier/linux/-/pipelines/940506

v11:
  - Fix dts paths
  - Update xfails
  - Remove meanless comment
  - Disable clang-format job

---
 Documentation/gpu/automated_testing.rst       |  144 +
 Documentation/gpu/index.rst                   |    1 +
 MAINTAINERS                                   |    8 +
 drivers/gpu/drm/ci/arm.config                 |   69 +
 drivers/gpu/drm/ci/arm64.config               |  199 ++
 drivers/gpu/drm/ci/build-igt.sh               |   35 +
 drivers/gpu/drm/ci/build.sh                   |  157 +
 drivers/gpu/drm/ci/build.yml                  |  110 +
 drivers/gpu/drm/ci/check-patch.py             |   57 +
 drivers/gpu/drm/ci/container.yml              |   65 +
 drivers/gpu/drm/ci/gitlab-ci.yml              |  251 ++
 drivers/gpu/drm/ci/igt_runner.sh              |   77 +
 drivers/gpu/drm/ci/image-tags.yml             |   15 +
 drivers/gpu/drm/ci/lava-submit.sh             |   57 +
 drivers/gpu/drm/ci/static-checks.yml          |   12 +
 drivers/gpu/drm/ci/test.yml                   |  335 ++
 drivers/gpu/drm/ci/testlist.txt               | 2912 +++++++++++++++++
 drivers/gpu/drm/ci/x86_64.config              |  111 +
 .../gpu/drm/ci/xfails/amdgpu-stoney-fails.txt |   19 +
 .../drm/ci/xfails/amdgpu-stoney-flakes.txt    |   21 +
 .../gpu/drm/ci/xfails/amdgpu-stoney-skips.txt |    2 +
 drivers/gpu/drm/ci/xfails/i915-amly-fails.txt |   17 +
 .../gpu/drm/ci/xfails/i915-amly-flakes.txt    |   32 +
 drivers/gpu/drm/ci/xfails/i915-amly-skips.txt |    4 +
 drivers/gpu/drm/ci/xfails/i915-apl-fails.txt  |   58 +
 drivers/gpu/drm/ci/xfails/i915-apl-flakes.txt |    1 +
 drivers/gpu/drm/ci/xfails/i915-apl-skips.txt  |    6 +
 drivers/gpu/drm/ci/xfails/i915-cml-fails.txt  |   18 +
 drivers/gpu/drm/ci/xfails/i915-cml-flakes.txt |   38 +
 drivers/gpu/drm/ci/xfails/i915-cml-skips.txt  |    2 +
 drivers/gpu/drm/ci/xfails/i915-glk-fails.txt  |   19 +
 drivers/gpu/drm/ci/xfails/i915-glk-flakes.txt |   41 +
 drivers/gpu/drm/ci/xfails/i915-glk-skips.txt  |    5 +
 drivers/gpu/drm/ci/xfails/i915-kbl-fails.txt  |   25 +
 drivers/gpu/drm/ci/xfails/i915-kbl-flakes.txt |   26 +
 drivers/gpu/drm/ci/xfails/i915-kbl-skips.txt  |    5 +
 drivers/gpu/drm/ci/xfails/i915-tgl-fails.txt  |   37 +
 drivers/gpu/drm/ci/xfails/i915-tgl-flakes.txt |    5 +
 drivers/gpu/drm/ci/xfails/i915-tgl-skips.txt  |   11 +
 drivers/gpu/drm/ci/xfails/i915-whl-fails.txt  |   48 +
 drivers/gpu/drm/ci/xfails/i915-whl-flakes.txt |    1 +
 drivers/gpu/drm/ci/xfails/i915-whl-skips.txt  |    2 +
 .../drm/ci/xfails/mediatek-mt8173-fails.txt   |   29 +
 .../drm/ci/xfails/mediatek-mt8173-flakes.txt  |    0
 .../drm/ci/xfails/mediatek-mt8183-fails.txt   |   10 +
 .../drm/ci/xfails/mediatek-mt8183-flakes.txt  |   14 +
 .../gpu/drm/ci/xfails/meson-g12b-fails.txt    |   12 +
 .../gpu/drm/ci/xfails/meson-g12b-flakes.txt   |    4 +
 .../gpu/drm/ci/xfails/msm-apq8016-fails.txt   |   15 +
 .../gpu/drm/ci/xfails/msm-apq8016-flakes.txt  |    4 +
 .../gpu/drm/ci/xfails/msm-apq8096-fails.txt   |    2 +
 .../gpu/drm/ci/xfails/msm-apq8096-flakes.txt  |    4 +
 .../gpu/drm/ci/xfails/msm-apq8096-skips.txt   |    2 +
 .../gpu/drm/ci/xfails/msm-sc7180-fails.txt    |   25 +
 .../gpu/drm/ci/xfails/msm-sc7180-flakes.txt   |    7 +
 .../gpu/drm/ci/xfails/msm-sc7180-skips.txt    |   23 +
 .../gpu/drm/ci/xfails/msm-sdm845-fails.txt    |   68 +
 .../gpu/drm/ci/xfails/msm-sdm845-flakes.txt   |   11 +
 .../gpu/drm/ci/xfails/msm-sdm845-skips.txt    |    2 +
 .../drm/ci/xfails/rockchip-rk3288-fails.txt   |   48 +
 .../drm/ci/xfails/rockchip-rk3288-flakes.txt  |    9 +
 .../drm/ci/xfails/rockchip-rk3288-skips.txt   |   52 +
 .../drm/ci/xfails/rockchip-rk3399-fails.txt   |   37 +
 .../drm/ci/xfails/rockchip-rk3399-flakes.txt  |   23 +
 .../drm/ci/xfails/rockchip-rk3399-skips.txt   |    5 +
 .../drm/ci/xfails/virtio_gpu-none-fails.txt   |   38 +
 .../drm/ci/xfails/virtio_gpu-none-flakes.txt  |    0
 .../drm/ci/xfails/virtio_gpu-none-skips.txt   |    6 +
 test                                          |    0
 69 files changed, 5508 insertions(+)
 create mode 100644 Documentation/gpu/automated_testing.rst
 create mode 100644 drivers/gpu/drm/ci/arm.config
 create mode 100644 drivers/gpu/drm/ci/arm64.config
 create mode 100644 drivers/gpu/drm/ci/build-igt.sh
 create mode 100644 drivers/gpu/drm/ci/build.sh
 create mode 100644 drivers/gpu/drm/ci/build.yml
 create mode 100755 drivers/gpu/drm/ci/check-patch.py
 create mode 100644 drivers/gpu/drm/ci/container.yml
 create mode 100644 drivers/gpu/drm/ci/gitlab-ci.yml
 create mode 100755 drivers/gpu/drm/ci/igt_runner.sh
 create mode 100644 drivers/gpu/drm/ci/image-tags.yml
 create mode 100755 drivers/gpu/drm/ci/lava-submit.sh
 create mode 100644 drivers/gpu/drm/ci/static-checks.yml
 create mode 100644 drivers/gpu/drm/ci/test.yml
 create mode 100644 drivers/gpu/drm/ci/testlist.txt
 create mode 100644 drivers/gpu/drm/ci/x86_64.config
 create mode 100644 drivers/gpu/drm/ci/xfails/amdgpu-stoney-fails.txt
 create mode 100644 drivers/gpu/drm/ci/xfails/amdgpu-stoney-flakes.txt
 create mode 100644 drivers/gpu/drm/ci/xfails/amdgpu-stoney-skips.txt
 create mode 100644 drivers/gpu/drm/ci/xfails/i915-amly-fails.txt
 create mode 100644 drivers/gpu/drm/ci/xfails/i915-amly-flakes.txt
 create mode 100644 drivers/gpu/drm/ci/xfails/i915-amly-skips.txt
 create mode 100644 drivers/gpu/drm/ci/xfails/i915-apl-fails.txt
 create mode 100644 drivers/gpu/drm/ci/xfails/i915-apl-flakes.txt
 create mode 100644 drivers/gpu/drm/ci/xfails/i915-apl-skips.txt
 create mode 100644 drivers/gpu/drm/ci/xfails/i915-cml-fails.txt
 create mode 100644 drivers/gpu/drm/ci/xfails/i915-cml-flakes.txt
 create mode 100644 drivers/gpu/drm/ci/xfails/i915-cml-skips.txt
 create mode 100644 drivers/gpu/drm/ci/xfails/i915-glk-fails.txt
 create mode 100644 drivers/gpu/drm/ci/xfails/i915-glk-flakes.txt
 create mode 100644 drivers/gpu/drm/ci/xfails/i915-glk-skips.txt
 create mode 100644 drivers/gpu/drm/ci/xfails/i915-kbl-fails.txt
 create mode 100644 drivers/gpu/drm/ci/xfails/i915-kbl-flakes.txt
 create mode 100644 drivers/gpu/drm/ci/xfails/i915-kbl-skips.txt
 create mode 100644 drivers/gpu/drm/ci/xfails/i915-tgl-fails.txt
 create mode 100644 drivers/gpu/drm/ci/xfails/i915-tgl-flakes.txt
 create mode 100644 drivers/gpu/drm/ci/xfails/i915-tgl-skips.txt
 create mode 100644 drivers/gpu/drm/ci/xfails/i915-whl-fails.txt
 create mode 100644 drivers/gpu/drm/ci/xfails/i915-whl-flakes.txt
 create mode 100644 drivers/gpu/drm/ci/xfails/i915-whl-skips.txt
 create mode 100644 drivers/gpu/drm/ci/xfails/mediatek-mt8173-fails.txt
 create mode 100644 drivers/gpu/drm/ci/xfails/mediatek-mt8173-flakes.txt
 create mode 100644 drivers/gpu/drm/ci/xfails/mediatek-mt8183-fails.txt
 create mode 100644 drivers/gpu/drm/ci/xfails/mediatek-mt8183-flakes.txt
 create mode 100644 drivers/gpu/drm/ci/xfails/meson-g12b-fails.txt
 create mode 100644 drivers/gpu/drm/ci/xfails/meson-g12b-flakes.txt
 create mode 100644 drivers/gpu/drm/ci/xfails/msm-apq8016-fails.txt
 create mode 100644 drivers/gpu/drm/ci/xfails/msm-apq8016-flakes.txt
 create mode 100644 drivers/gpu/drm/ci/xfails/msm-apq8096-fails.txt
 create mode 100644 drivers/gpu/drm/ci/xfails/msm-apq8096-flakes.txt
 create mode 100644 drivers/gpu/drm/ci/xfails/msm-apq8096-skips.txt
 create mode 100644 drivers/gpu/drm/ci/xfails/msm-sc7180-fails.txt
 create mode 100644 drivers/gpu/drm/ci/xfails/msm-sc7180-flakes.txt
 create mode 100644 drivers/gpu/drm/ci/xfails/msm-sc7180-skips.txt
 create mode 100644 drivers/gpu/drm/ci/xfails/msm-sdm845-fails.txt
 create mode 100644 drivers/gpu/drm/ci/xfails/msm-sdm845-flakes.txt
 create mode 100644 drivers/gpu/drm/ci/xfails/msm-sdm845-skips.txt
 create mode 100644 drivers/gpu/drm/ci/xfails/rockchip-rk3288-fails.txt
 create mode 100644 drivers/gpu/drm/ci/xfails/rockchip-rk3288-flakes.txt
 create mode 100644 drivers/gpu/drm/ci/xfails/rockchip-rk3288-skips.txt
 create mode 100644 drivers/gpu/drm/ci/xfails/rockchip-rk3399-fails.txt
 create mode 100644 drivers/gpu/drm/ci/xfails/rockchip-rk3399-flakes.txt
 create mode 100644 drivers/gpu/drm/ci/xfails/rockchip-rk3399-skips.txt
 create mode 100644 drivers/gpu/drm/ci/xfails/virtio_gpu-none-fails.txt
 create mode 100644 drivers/gpu/drm/ci/xfails/virtio_gpu-none-flakes.txt
 create mode 100644 drivers/gpu/drm/ci/xfails/virtio_gpu-none-skips.txt
 create mode 100644 test

diff --git a/Documentation/gpu/automated_testing.rst b/Documentation/gpu/automated_testing.rst
new file mode 100644
index 000000000000..1b87b802ac7f
--- /dev/null
+++ b/Documentation/gpu/automated_testing.rst
@@ -0,0 +1,144 @@
+.. SPDX-License-Identifier: GPL-2.0+
+
+=========================================
+Automated testing of the DRM subsystem
+=========================================
+
+Introduction
+============
+
+Making sure that changes to the core or drivers don't introduce regressions can
+be very time-consuming when lots of different hardware configurations need to
+be tested. Moreover, it isn't practical for each person interested in this
+testing to have to acquire and maintain what can be a considerable amount of
+hardware.
+
+Also, it is desirable for developers to check for regressions in their code by
+themselves, instead of relying on the maintainers to find them and then
+reporting back.
+
+There are facilities in gitlab.freedesktop.org to automatically test Mesa that
+can be used as well for testing the DRM subsystem. This document explains how
+people interested in testing it can use this shared infrastructure to save
+quite some time and effort.
+
+
+Relevant files
+==============
+
+drivers/gpu/drm/ci/gitlab-ci.yml
+--------------------------------
+
+This is the root configuration file for GitLab CI. Among other less interesting
+bits, it specifies the specific version of the scripts to be used. There are
+some variables that can be modified to change the behavior of the pipeline:
+
+DRM_CI_PROJECT_PATH
+    Repository that contains the Mesa software infrastructure for CI
+
+DRM_CI_COMMIT_SHA
+    A particular revision to use from that repository
+
+UPSTREAM_REPO
+    URL to git repository containing the target branch
+
+TARGET_BRANCH
+    Branch to which this branch is to be merged into
+
+IGT_VERSION
+    Revision of igt-gpu-tools being used, from
+    https://gitlab.freedesktop.org/drm/igt-gpu-tools
+
+drivers/gpu/drm/ci/testlist.txt
+-------------------------------
+
+IGT tests to be run on all drivers (unless mentioned in a driver's *-skips.txt
+file, see below).
+
+drivers/gpu/drm/ci/${DRIVER_NAME}-${HW_REVISION}-fails.txt
+----------------------------------------------------------
+
+Lists the known failures for a given driver on a specific hardware revision.
+
+drivers/gpu/drm/ci/${DRIVER_NAME}-${HW_REVISION}-flakes.txt
+-----------------------------------------------------------
+
+Lists the tests that for a given driver on a specific hardware revision are
+known to behave unreliably. These tests won't cause a job to fail regardless of
+the result. They will still be run.
+
+drivers/gpu/drm/ci/${DRIVER_NAME}-${HW_REVISION}-skips.txt
+-----------------------------------------------------------
+
+Lists the tests that won't be run for a given driver on a specific hardware
+revision. These are usually tests that interfere with the running of the test
+list due to hanging the machine, causing OOM, taking too long, etc.
+
+
+How to enable automated testing on your tree
+============================================
+
+1. Create a Linux tree in https://gitlab.freedesktop.org/ if you don't have one
+yet
+
+2. In your kernel repo's configuration (eg.
+https://gitlab.freedesktop.org/janedoe/linux/-/settings/ci_cd), change the
+CI/CD configuration file from .gitlab-ci.yml to
+drivers/gpu/drm/ci/gitlab-ci.yml.
+
+3. Next time you push to this repository, you will see a CI pipeline being
+created (eg. https://gitlab.freedesktop.org/janedoe/linux/-/pipelines)
+
+4. The various jobs will be run and when the pipeline is finished, all jobs
+should be green unless a regression has been found.
+
+
+How to update test expectations
+===============================
+
+If your changes to the code fix any tests, you will have to remove one or more
+lines from one or more of the files in
+drivers/gpu/drm/ci/${DRIVER_NAME}_*_fails.txt, for each of the test platforms
+affected by the change.
+
+
+How to expand coverage
+======================
+
+If your code changes make it possible to run more tests (by solving reliability
+issues, for example), you can remove tests from the flakes and/or skips lists,
+and then the expected results if there are any known failures.
+
+If there is a need for updating the version of IGT being used (maybe you have
+added more tests to it), update the IGT_VERSION variable at the top of the
+gitlab-ci.yml file.
+
+
+How to test your changes to the scripts
+=======================================
+
+For testing changes to the scripts in the drm-ci repo, change the
+DRM_CI_PROJECT_PATH and DRM_CI_COMMIT_SHA variables in
+drivers/gpu/drm/ci/gitlab-ci.yml to match your fork of the project (eg.
+janedoe/drm-ci). This fork needs to be in https://gitlab.freedesktop.org/.
+
+
+How to incorporate external fixes in your testing
+=================================================
+
+Often, regressions in other trees will prevent testing changes local to the
+tree under test. These fixes will be automatically merged in during the build
+jobs from a branch in the target tree that is named as
+${TARGET_BRANCH}-external-fixes.
+
+If the pipeline is not in a merge request and a branch with the same name
+exists in the local tree, commits from that branch will be merged in as well.
+
+
+How to deal with automated testing labs that may be down
+========================================================
+
+If a hardware farm is down and thus causing pipelines to fail that would
+otherwise pass, one can disable all jobs that would be submitted to that farm
+by editing the file at
+https://gitlab.freedesktop.org/gfx-ci/lab-status/-/blob/main/lab-status.yml.
diff --git a/Documentation/gpu/index.rst b/Documentation/gpu/index.rst
index eee5996acf2c..e45ff0915246 100644
--- a/Documentation/gpu/index.rst
+++ b/Documentation/gpu/index.rst
@@ -17,6 +17,7 @@ GPU Driver Developer's Guide
    backlight
    vga-switcheroo
    vgaarbiter
+   automated_testing
    todo
    rfc/index
 
diff --git a/MAINTAINERS b/MAINTAINERS
index cd882b87a3c6..d6f163ec7b31 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -7131,6 +7131,14 @@ T:	git git://anongit.freedesktop.org/drm/drm-misc
 F:	drivers/gpu/drm/ttm/
 F:	include/drm/ttm/
 
+DRM AUTOMATED TESTING
+M:	Helen Koike <helen.koike@collabora.com>
+L:	dri-devel@lists.freedesktop.org
+S:	Maintained
+T:	git git://anongit.freedesktop.org/drm/drm-misc
+F:	Documentation/gpu/automated_testing.rst
+F:	drivers/gpu/drm/ci/
+
 DSBR100 USB FM RADIO DRIVER
 M:	Alexey Klimov <klimov.linux@gmail.com>
 L:	linux-media@vger.kernel.org
diff --git a/drivers/gpu/drm/ci/arm.config b/drivers/gpu/drm/ci/arm.config
new file mode 100644
index 000000000000..871f4de063ad
--- /dev/null
+++ b/drivers/gpu/drm/ci/arm.config
@@ -0,0 +1,69 @@
+CONFIG_LOCALVERSION_AUTO=y
+CONFIG_DEBUG_KERNEL=y
+
+CONFIG_CRYPTO_ZSTD=y
+CONFIG_ZRAM_MEMORY_TRACKING=y
+CONFIG_ZRAM_WRITEBACK=y
+CONFIG_ZRAM=y
+CONFIG_ZSMALLOC_STAT=y
+
+# abootimg with a 'dummy' rootfs fails with root=/dev/nfs
+CONFIG_BLK_DEV_INITRD=n
+
+CONFIG_DEVFREQ_GOV_PERFORMANCE=y
+CONFIG_DEVFREQ_GOV_POWERSAVE=y
+CONFIG_DEVFREQ_GOV_USERSPACE=y
+CONFIG_DEVFREQ_GOV_PASSIVE=y
+CONFIG_DEVFREQ_GOV_SIMPLE_ONDEMAND=y
+
+CONFIG_DRM=y
+CONFIG_DRM_ETNAVIV=y
+CONFIG_DRM_ROCKCHIP=y
+CONFIG_DRM_PANFROST=y
+CONFIG_DRM_LIMA=y
+CONFIG_DRM_PANEL_SIMPLE=y
+CONFIG_PWM_CROS_EC=y
+CONFIG_BACKLIGHT_PWM=y
+
+CONFIG_ROCKCHIP_CDN_DP=n
+
+CONFIG_SPI_ROCKCHIP=y
+CONFIG_PWM_ROCKCHIP=y
+CONFIG_PHY_ROCKCHIP_DP=y
+CONFIG_DWMAC_ROCKCHIP=y
+
+CONFIG_MFD_RK808=y
+CONFIG_REGULATOR_RK808=y
+CONFIG_RTC_DRV_RK808=y
+CONFIG_COMMON_CLK_RK808=y
+
+CONFIG_REGULATOR_FAN53555=y
+CONFIG_REGULATOR=y
+
+CONFIG_REGULATOR_VCTRL=y
+
+CONFIG_KASAN=n
+CONFIG_KASAN_INLINE=n
+CONFIG_STACKTRACE=n
+
+CONFIG_TMPFS=y
+
+CONFIG_PROVE_LOCKING=n
+CONFIG_DEBUG_LOCKDEP=n
+CONFIG_SOFTLOCKUP_DETECTOR=n
+CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC=n
+
+CONFIG_FW_LOADER_COMPRESS=y
+
+CONFIG_USB_USBNET=y
+CONFIG_NETDEVICES=y
+CONFIG_USB_NET_DRIVERS=y
+CONFIG_USB_RTL8152=y
+CONFIG_USB_NET_AX8817X=y
+CONFIG_USB_NET_SMSC95XX=y
+
+# TK1
+CONFIG_ARM_TEGRA_DEVFREQ=y
+
+# 32-bit build failure
+CONFIG_DRM_MSM=n
diff --git a/drivers/gpu/drm/ci/arm64.config b/drivers/gpu/drm/ci/arm64.config
new file mode 100644
index 000000000000..817e18ddfd4f
--- /dev/null
+++ b/drivers/gpu/drm/ci/arm64.config
@@ -0,0 +1,199 @@
+CONFIG_LOCALVERSION_AUTO=y
+CONFIG_DEBUG_KERNEL=y
+
+CONFIG_CRYPTO_ZSTD=y
+CONFIG_ZRAM_MEMORY_TRACKING=y
+CONFIG_ZRAM_WRITEBACK=y
+CONFIG_ZRAM=y
+CONFIG_ZSMALLOC_STAT=y
+
+# abootimg with a 'dummy' rootfs fails with root=/dev/nfs
+CONFIG_BLK_DEV_INITRD=n
+
+CONFIG_DEVFREQ_GOV_PERFORMANCE=y
+CONFIG_DEVFREQ_GOV_POWERSAVE=y
+CONFIG_DEVFREQ_GOV_USERSPACE=y
+CONFIG_DEVFREQ_GOV_PASSIVE=y
+
+CONFIG_DRM=y
+CONFIG_DRM_ROCKCHIP=y
+CONFIG_DRM_PANFROST=y
+CONFIG_DRM_LIMA=y
+CONFIG_DRM_PANEL_SIMPLE=y
+CONFIG_DRM_PANEL_EDP=y
+CONFIG_DRM_MSM=y
+CONFIG_DRM_ETNAVIV=y
+CONFIG_DRM_I2C_ADV7511=y
+CONFIG_PWM_CROS_EC=y
+CONFIG_BACKLIGHT_PWM=y
+
+CONFIG_ROCKCHIP_CDN_DP=n
+
+CONFIG_SPI_ROCKCHIP=y
+CONFIG_PWM_ROCKCHIP=y
+CONFIG_PHY_ROCKCHIP_DP=y
+CONFIG_DWMAC_ROCKCHIP=y
+CONFIG_STMMAC_ETH=y
+CONFIG_TYPEC_FUSB302=y
+CONFIG_TYPEC=y
+CONFIG_TYPEC_TCPM=y
+
+# MSM platform bits
+
+# For CONFIG_QCOM_LMH
+CONFIG_OF=y
+
+CONFIG_ARM_SMMU_QCOM=y
+CONFIG_QCOM_COMMAND_DB=y
+CONFIG_QCOM_RPMHPD=y
+CONFIG_QCOM_RPMPD=y
+CONFIG_QCOM_OCMEM=y
+CONFIG_SDM_GPUCC_845=y
+CONFIG_SDM_VIDEOCC_845=y
+CONFIG_SDM_DISPCC_845=y
+CONFIG_SDM_LPASSCC_845=y
+CONFIG_SDM_CAMCC_845=y
+CONFIG_RESET_QCOM_PDC=y
+CONFIG_DRM_TI_SN65DSI86=y
+CONFIG_I2C_QCOM_GENI=y
+CONFIG_SPI_QCOM_GENI=y
+CONFIG_PHY_QCOM_QUSB2=y
+CONFIG_PHY_QCOM_QMP=y
+CONFIG_MSM_GCC_8996=y
+CONFIG_QCOM_CLK_APCC_MSM8996=y
+CONFIG_QCOM_LLCC=y
+CONFIG_QCOM_LMH=y
+CONFIG_QCOM_SPMI_TEMP_ALARM=y
+CONFIG_QCOM_WDT=y
+CONFIG_POWER_RESET_QCOM_PON=y
+CONFIG_RTC_DRV_PM8XXX=y
+CONFIG_INTERCONNECT=y
+CONFIG_INTERCONNECT_QCOM=y
+CONFIG_INTERCONNECT_QCOM_MSM8996=y
+CONFIG_INTERCONNECT_QCOM_SDM845=y
+CONFIG_INTERCONNECT_QCOM_MSM8916=y
+CONFIG_INTERCONNECT_QCOM_MSM8996=y
+CONFIG_INTERCONNECT_QCOM_OSM_L3=y
+CONFIG_INTERCONNECT_QCOM_SC7180=y
+CONFIG_INTERCONNECT_QCOM_SM8350=y
+CONFIG_CRYPTO_DEV_QCOM_RNG=y
+CONFIG_SC_DISPCC_7180=y
+CONFIG_SC_GPUCC_7180=y
+CONFIG_SM_GPUCC_8350=y
+CONFIG_QCOM_SPMI_ADC5=y
+CONFIG_DRM_PARADE_PS8640=y
+CONFIG_DRM_LONTIUM_LT9611UXC=y
+CONFIG_PHY_QCOM_USB_HS=y
+CONFIG_QCOM_GPI_DMA=y
+CONFIG_USB_ONBOARD_HUB=y
+CONFIG_NVMEM_QCOM_QFPROM=y
+CONFIG_PHY_QCOM_USB_SNPS_FEMTO_V2=y
+
+
+# db410c ethernet
+CONFIG_USB_RTL8152=y
+# db820c ethernet
+CONFIG_ATL1C=y
+# Chromebooks ethernet
+CONFIG_USB_ONBOARD_HUB=y
+# 888 HDK ethernet
+CONFIG_USB_LAN78XX=y
+
+CONFIG_ARCH_ALPINE=n
+CONFIG_ARCH_BCM2835=y
+CONFIG_ARCH_BCM_IPROC=n
+CONFIG_ARCH_BERLIN=n
+CONFIG_ARCH_BRCMSTB=n
+CONFIG_ARCH_EXYNOS=n
+CONFIG_ARCH_K3=n
+CONFIG_ARCH_LAYERSCAPE=n
+CONFIG_ARCH_LG1K=n
+CONFIG_ARCH_HISI=n
+CONFIG_ARCH_MVEBU=n
+CONFIG_ARCH_SEATTLE=n
+CONFIG_ARCH_SYNQUACER=n
+CONFIG_ARCH_RENESAS=n
+CONFIG_ARCH_R8A774A1=n
+CONFIG_ARCH_R8A774C0=n
+CONFIG_ARCH_R8A7795=n
+CONFIG_ARCH_R8A7796=n
+CONFIG_ARCH_R8A77965=n
+CONFIG_ARCH_R8A77970=n
+CONFIG_ARCH_R8A77980=n
+CONFIG_ARCH_R8A77990=n
+CONFIG_ARCH_R8A77995=n
+CONFIG_ARCH_STRATIX10=n
+CONFIG_ARCH_TEGRA=n
+CONFIG_ARCH_SPRD=n
+CONFIG_ARCH_THUNDER=n
+CONFIG_ARCH_THUNDER2=n
+CONFIG_ARCH_UNIPHIER=n
+CONFIG_ARCH_VEXPRESS=n
+CONFIG_ARCH_XGENE=n
+CONFIG_ARCH_ZX=n
+CONFIG_ARCH_ZYNQMP=n
+
+# Strip out some stuff we don't need for graphics testing, to reduce
+# the build.
+CONFIG_CAN=n
+CONFIG_WIRELESS=n
+CONFIG_RFKILL=n
+CONFIG_WLAN=n
+
+CONFIG_REGULATOR_FAN53555=y
+CONFIG_REGULATOR=y
+
+CONFIG_REGULATOR_VCTRL=y
+
+CONFIG_KASAN=n
+CONFIG_KASAN_INLINE=n
+CONFIG_STACKTRACE=n
+
+CONFIG_TMPFS=y
+
+CONFIG_PROVE_LOCKING=n
+CONFIG_DEBUG_LOCKDEP=n
+CONFIG_SOFTLOCKUP_DETECTOR=y
+CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC=y
+
+CONFIG_DETECT_HUNG_TASK=y
+
+CONFIG_FW_LOADER_COMPRESS=y
+CONFIG_FW_LOADER_USER_HELPER=n
+
+CONFIG_USB_USBNET=y
+CONFIG_NETDEVICES=y
+CONFIG_USB_NET_DRIVERS=y
+CONFIG_USB_RTL8152=y
+CONFIG_USB_NET_AX8817X=y
+CONFIG_USB_NET_SMSC95XX=y
+
+# For amlogic
+CONFIG_MESON_GXL_PHY=y
+CONFIG_MDIO_BUS_MUX_MESON_G12A=y
+CONFIG_DRM_MESON=y
+
+# For Mediatek
+CONFIG_DRM_MEDIATEK=y
+CONFIG_PWM_MEDIATEK=y
+CONFIG_DRM_MEDIATEK_HDMI=y
+CONFIG_GNSS=y
+CONFIG_GNSS_MTK_SERIAL=y
+CONFIG_HW_RANDOM=y
+CONFIG_HW_RANDOM_MTK=y
+CONFIG_MTK_DEVAPC=y
+CONFIG_PWM_MTK_DISP=y
+CONFIG_MTK_CMDQ=y
+
+# For nouveau.  Note that DRM must be a module so that it's loaded after NFS is up to provide the firmware.
+CONFIG_ARCH_TEGRA=y
+CONFIG_DRM_NOUVEAU=m
+CONFIG_DRM_TEGRA=m
+CONFIG_R8169=y
+CONFIG_STAGING=y
+CONFIG_DRM_TEGRA_STAGING=y
+CONFIG_TEGRA_HOST1X=y
+CONFIG_ARM_TEGRA_DEVFREQ=y
+CONFIG_TEGRA_SOCTHERM=y
+CONFIG_DRM_TEGRA_DEBUG=y
+CONFIG_PWM_TEGRA=y
diff --git a/drivers/gpu/drm/ci/build-igt.sh b/drivers/gpu/drm/ci/build-igt.sh
new file mode 100644
index 000000000000..500fa4f5c30a
--- /dev/null
+++ b/drivers/gpu/drm/ci/build-igt.sh
@@ -0,0 +1,35 @@
+#!/bin/bash
+# SPDX-License-Identifier: MIT
+
+set -ex
+
+git clone https://gitlab.freedesktop.org/drm/igt-gpu-tools.git --single-branch --no-checkout
+cd igt-gpu-tools
+git checkout $IGT_VERSION
+
+if [[ "$KERNEL_ARCH" = "arm" ]]; then
+    . ../.gitlab-ci/container/create-cross-file.sh armhf
+    EXTRA_MESON_ARGS="--cross-file /cross_file-armhf.txt"
+fi
+
+MESON_OPTIONS="-Doverlay=disabled                    \
+               -Dchamelium=disabled                  \
+               -Dvalgrind=disabled                   \
+               -Dman=enabled                         \
+               -Dtests=enabled                       \
+               -Drunner=enabled                      \
+               -Dlibunwind=enabled                   \
+               -Dprefix=/igt"
+
+mkdir -p /igt
+meson build $MESON_OPTIONS $EXTRA_MESON_ARGS
+ninja -C build -j${FDO_CI_CONCURRENT:-4} || ninja -C build -j 1
+ninja -C build install
+
+mkdir -p artifacts/
+tar -cf artifacts/igt.tar /igt
+
+# Pass needed files to the test stage
+S3_ARTIFACT_NAME="igt.tar.gz"
+gzip -c artifacts/igt.tar > ${S3_ARTIFACT_NAME}
+ci-fairy s3cp --token-file "${CI_JOB_JWT_FILE}" ${S3_ARTIFACT_NAME} https://${PIPELINE_ARTIFACTS_BASE}/${KERNEL_ARCH}/${S3_ARTIFACT_NAME}
diff --git a/drivers/gpu/drm/ci/build.sh b/drivers/gpu/drm/ci/build.sh
new file mode 100644
index 000000000000..7b014287a041
--- /dev/null
+++ b/drivers/gpu/drm/ci/build.sh
@@ -0,0 +1,157 @@
+#!/bin/bash
+# SPDX-License-Identifier: MIT
+
+set -ex
+
+# Clean up stale rebases that GitLab might not have removed when reusing a checkout dir
+rm -rf .git/rebase-apply
+
+. .gitlab-ci/container/container_pre_build.sh
+
+# libssl-dev was uninstalled because it was considered an ephemeral package
+apt-get update
+apt-get install -y libssl-dev
+
+if [[ "$KERNEL_ARCH" = "arm64" ]]; then
+    GCC_ARCH="aarch64-linux-gnu"
+    DEBIAN_ARCH="arm64"
+    DEVICE_TREES="arch/arm64/boot/dts/rockchip/rk3399-gru-kevin.dtb"
+    DEVICE_TREES+=" arch/arm64/boot/dts/amlogic/meson-gxl-s805x-libretech-ac.dtb"
+    DEVICE_TREES+=" arch/arm64/boot/dts/allwinner/sun50i-h6-pine-h64.dtb"
+    DEVICE_TREES+=" arch/arm64/boot/dts/amlogic/meson-gxm-khadas-vim2.dtb"
+    DEVICE_TREES+=" arch/arm64/boot/dts/qcom/apq8016-sbc.dtb"
+    DEVICE_TREES+=" arch/arm64/boot/dts/qcom/apq8096-db820c.dtb"
+    DEVICE_TREES+=" arch/arm64/boot/dts/amlogic/meson-g12b-a311d-khadas-vim3.dtb"
+    DEVICE_TREES+=" arch/arm64/boot/dts/mediatek/mt8173-elm-hana.dtb"
+    DEVICE_TREES+=" arch/arm64/boot/dts/mediatek/mt8183-kukui-jacuzzi-juniper-sku16.dtb"
+    DEVICE_TREES+=" arch/arm64/boot/dts/mediatek/mt8192-asurada-spherion-r0.dtb"
+    DEVICE_TREES+=" arch/arm64/boot/dts/qcom/sc7180-trogdor-lazor-limozeen-nots-r5.dtb"
+elif [[ "$KERNEL_ARCH" = "arm" ]]; then
+    GCC_ARCH="arm-linux-gnueabihf"
+    DEBIAN_ARCH="armhf"
+    DEVICE_TREES="arch/arm/boot/dts/rockchip/rk3288-veyron-jaq.dtb"
+    DEVICE_TREES+=" arch/arm/boot/dts/allwinner/sun8i-h3-libretech-all-h3-cc.dtb"
+    DEVICE_TREES+=" arch/arm/boot/dts/nxp/imx/imx6q-cubox-i.dtb"
+    apt-get install -y libssl-dev:armhf
+else
+    GCC_ARCH="x86_64-linux-gnu"
+    DEBIAN_ARCH="x86_64"
+    DEVICE_TREES=""
+fi
+
+export ARCH=${KERNEL_ARCH}
+export CROSS_COMPILE="${GCC_ARCH}-"
+
+# The kernel doesn't like the gold linker (or the old lld in our debians).
+# Sneak in some override symlinks during kernel build until we can update
+# debian.
+mkdir -p ld-links
+for i in /usr/bin/*-ld /usr/bin/ld; do
+    i=$(basename $i)
+    ln -sf /usr/bin/$i.bfd ld-links/$i
+done
+
+NEWPATH=$(pwd)/ld-links
+export PATH=$NEWPATH:$PATH
+
+git config --global user.email "fdo@example.com"
+git config --global user.name "freedesktop.org CI"
+git config --global pull.rebase true
+
+# Try to merge fixes from target repo
+if [ "$(git ls-remote --exit-code --heads ${UPSTREAM_REPO} ${TARGET_BRANCH}-external-fixes)" ]; then
+    git pull ${UPSTREAM_REPO} ${TARGET_BRANCH}-external-fixes
+fi
+
+# Try to merge fixes from local repo if this isn't a merge request
+if [ -z "$CI_MERGE_REQUEST_PROJECT_PATH" ]; then
+    if [ "$(git ls-remote --exit-code --heads origin ${TARGET_BRANCH}-external-fixes)" ]; then
+        git pull origin ${TARGET_BRANCH}-external-fixes
+    fi
+fi
+
+for opt in $ENABLE_KCONFIGS; do
+  echo CONFIG_$opt=y >> drivers/gpu/drm/ci/${KERNEL_ARCH}.config
+done
+for opt in $DISABLE_KCONFIGS; do
+  echo CONFIG_$opt=n >> drivers/gpu/drm/ci/${KERNEL_ARCH}.config
+done
+
+if [[ -n "${MERGE_FRAGMENT}" ]]; then
+    ./scripts/kconfig/merge_config.sh ${DEFCONFIG} drivers/gpu/drm/ci/${MERGE_FRAGMENT}
+else
+    make `basename ${DEFCONFIG}`
+fi
+
+make ${KERNEL_IMAGE_NAME}
+
+mkdir -p /lava-files/
+for image in ${KERNEL_IMAGE_NAME}; do
+    cp arch/${KERNEL_ARCH}/boot/${image} /lava-files/.
+done
+
+if [[ -n ${DEVICE_TREES} ]]; then
+    make dtbs
+    cp ${DEVICE_TREES} /lava-files/.
+fi
+
+make modules
+mkdir -p install/modules/
+INSTALL_MOD_PATH=install/modules/ make modules_install
+
+if [[ ${DEBIAN_ARCH} = "arm64" ]]; then
+    make Image.lzma
+    mkimage \
+        -f auto \
+        -A arm \
+        -O linux \
+        -d arch/arm64/boot/Image.lzma \
+        -C lzma\
+        -b arch/arm64/boot/dts/qcom/sdm845-cheza-r3.dtb \
+        /lava-files/cheza-kernel
+    KERNEL_IMAGE_NAME+=" cheza-kernel"
+
+    # Make a gzipped copy of the Image for db410c.
+    gzip -k /lava-files/Image
+    KERNEL_IMAGE_NAME+=" Image.gz"
+fi
+
+# Pass needed files to the test stage
+mkdir -p install
+cp -rfv .gitlab-ci/* install/.
+cp -rfv install/common install/ci-common
+cp -rfv drivers/gpu/drm/ci/* install/.
+
+. .gitlab-ci/container/container_post_build.sh
+
+if [[ "$UPLOAD_TO_MINIO" = "1" ]]; then
+    xz -7 -c -T${FDO_CI_CONCURRENT:-4} vmlinux > /lava-files/vmlinux.xz
+    FILES_TO_UPLOAD="$KERNEL_IMAGE_NAME vmlinux.xz"
+
+    if [[ -n $DEVICE_TREES ]]; then
+        FILES_TO_UPLOAD="$FILES_TO_UPLOAD $(basename -a $DEVICE_TREES)"
+    fi
+
+    for f in $FILES_TO_UPLOAD; do
+        ci-fairy s3cp --token-file "${CI_JOB_JWT_FILE}" /lava-files/$f \
+                https://${PIPELINE_ARTIFACTS_BASE}/${DEBIAN_ARCH}/$f
+    done
+
+    S3_ARTIFACT_NAME="kernel-files.tar.zst"
+    tar --zstd -cf $S3_ARTIFACT_NAME install
+    ci-fairy s3cp --token-file "${CI_JOB_JWT_FILE}" ${S3_ARTIFACT_NAME} https://${PIPELINE_ARTIFACTS_BASE}/${DEBIAN_ARCH}/${S3_ARTIFACT_NAME}
+
+    echo "Download vmlinux.xz from https://${PIPELINE_ARTIFACTS_BASE}/${DEBIAN_ARCH}/vmlinux.xz"
+fi
+
+mkdir -p artifacts/install/lib
+mv install/* artifacts/install/.
+rm -rf artifacts/install/modules
+ln -s common artifacts/install/ci-common
+
+for image in ${KERNEL_IMAGE_NAME}; do
+    cp /lava-files/$image artifacts/install/.
+done
+
+tar -C artifacts -cf artifacts/install.tar install
+rm -rf artifacts/install
diff --git a/drivers/gpu/drm/ci/build.yml b/drivers/gpu/drm/ci/build.yml
new file mode 100644
index 000000000000..e6503f1c5927
--- /dev/null
+++ b/drivers/gpu/drm/ci/build.yml
@@ -0,0 +1,110 @@
+.build:
+  extends:
+    - .build-rules
+  stage: build
+  artifacts:
+    paths:
+      - artifacts
+  script:
+    - FDO_CI_CONCURRENT=${FDO_CI_CONCURRENT} bash drivers/gpu/drm/ci/build.sh
+
+.build:arm32:
+  extends:
+    - .build
+    - .use-debian/arm64_build
+  tags:
+    - aarch64
+  variables:
+    DEFCONFIG: "arch/arm/configs/multi_v7_defconfig"
+    KERNEL_IMAGE_NAME: "zImage"
+    KERNEL_ARCH: "arm"
+
+.build:arm64:
+  extends:
+    - .build
+    - .use-debian/arm64_build
+  tags:
+    - aarch64
+  variables:
+    DEFCONFIG: "arch/arm64/configs/defconfig"
+    KERNEL_IMAGE_NAME: "Image"
+    KERNEL_ARCH: "arm64"
+
+.build:x86_64:
+  extends:
+    - .build
+    - .use-debian/x86_64_build
+  variables:
+    DEFCONFIG: "arch/x86/configs/x86_64_defconfig"
+    KERNEL_IMAGE_NAME: "bzImage"
+    KERNEL_ARCH: "x86_64"
+
+
+# Build IGT for testing on devices
+
+igt:arm32:
+  extends: .build:arm32
+  script:
+    - FDO_CI_CONCURRENT=${FDO_CI_CONCURRENT} bash drivers/gpu/drm/ci/build-igt.sh
+
+igt:arm64:
+  extends: .build:arm64
+  script:
+    - FDO_CI_CONCURRENT=${FDO_CI_CONCURRENT} bash drivers/gpu/drm/ci/build-igt.sh
+
+igt:x86_64:
+  extends: .build:x86_64
+  script:
+    - FDO_CI_CONCURRENT=${FDO_CI_CONCURRENT} bash drivers/gpu/drm/ci/build-igt.sh
+
+# Build kernels for testing on devices
+
+testing:arm32:
+  extends: .build:arm32
+  variables:
+    # Would be good to have DEBUG_KMEMLEAK, but it doesn't work well with any of
+    # PROVE_LOCKING and KASAN as of 5.17.
+    #
+    # db410c and db820c don't boot with KASAN_INLINE, probably due to the kernel
+    # becoming too big for their bootloaders.
+    ENABLE_KCONFIGS: "PROVE_LOCKING DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT"
+    UPLOAD_TO_MINIO: 1
+    MERGE_FRAGMENT: arm.config
+
+testing:arm64:
+  extends: .build:arm64
+  variables:
+    # Would be good to have DEBUG_KMEMLEAK, but it doesn't work well with any of
+    # PROVE_LOCKING and KASAN as of 5.17.
+    #
+    # db410c and db820c don't boot with KASAN_INLINE, probably due to the kernel
+    # becoming too big for their bootloaders.
+    ENABLE_KCONFIGS: "PROVE_LOCKING DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT"
+    UPLOAD_TO_MINIO: 1
+    MERGE_FRAGMENT: arm64.config
+
+testing:x86_64:
+  extends: .build:x86_64
+  variables:
+    # Would be good to have DEBUG_KMEMLEAK, but it doesn't work well with any of
+    # PROVE_LOCKING and KASAN as of 5.17.
+    #
+    # db410c and db820c don't boot with KASAN_INLINE, probably due to the kernel
+    # becoming too big for their bootloaders.
+    ENABLE_KCONFIGS: "PROVE_LOCKING DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT"
+    UPLOAD_TO_MINIO: 1
+    MERGE_FRAGMENT: x86_64.config
+
+
+# Jobs for build-testing different configurations
+
+build:arm32:
+  extends: .build:arm32
+
+build-nodebugfs:arm64:
+  extends: .build:arm64
+  variables:
+    DISABLE_KCONFIGS: "DEBUG_FS"
+
+build:x86_64:
+  extends: .build:x86_64
diff --git a/drivers/gpu/drm/ci/check-patch.py b/drivers/gpu/drm/ci/check-patch.py
new file mode 100755
index 000000000000..a5f399a20e25
--- /dev/null
+++ b/drivers/gpu/drm/ci/check-patch.py
@@ -0,0 +1,57 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: GPL-2.0-or-later
+#
+# check-patch.py: run checkpatch.pl across all commits in a branch
+#
+# Based on qemu/.gitlab-ci.d/check-patch.py
+#
+# Copyright (C) 2020 Red Hat, Inc.
+# Copyright (C) 2022 Collabora Ltd.
+
+import os
+import os.path
+import sys
+import subprocess
+
+repourl = "https://gitlab.freedesktop.org/%s.git" % os.environ["CI_MERGE_REQUEST_PROJECT_PATH"]
+
+# GitLab CI environment does not give us any direct info about the
+# base for the user's branch. We thus need to figure out a common
+# ancestor between the user's branch and current git master.
+os.environ["GIT_DEPTH"] = "1000"
+subprocess.call(["git", "remote", "remove", "check-patch"], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
+subprocess.check_call(["git", "remote", "add", "check-patch", repourl])
+subprocess.check_call(["git", "fetch", "check-patch", os.environ["CI_MERGE_REQUEST_TARGET_BRANCH_NAME"]],
+                      stdout=subprocess.DEVNULL,
+                      stderr=subprocess.DEVNULL)
+
+ancestor = subprocess.check_output(["git", "merge-base",
+                                    "check-patch/%s" % os.environ["CI_MERGE_REQUEST_TARGET_BRANCH_NAME"], "HEAD"],
+                                   universal_newlines=True)
+
+ancestor = ancestor.strip()
+
+log = subprocess.check_output(["git", "log", "--format=%H %s",
+                               ancestor + "..."],
+                              universal_newlines=True)
+
+subprocess.check_call(["git", "remote", "rm", "check-patch"])
+
+if log == "":
+    print("\nNo commits since %s, skipping checks\n" % ancestor)
+    sys.exit(0)
+
+errors = False
+
+print("\nChecking all commits since %s...\n" % ancestor, flush=True)
+
+ret = subprocess.run(["scripts/checkpatch.pl",
+                      "--terse",
+                      "--types", os.environ["CHECKPATCH_TYPES"],
+                      "--git", ancestor + "..."])
+
+if ret.returncode != 0:
+    print("    ❌ FAIL one or more commits failed scripts/checkpatch.pl")
+    sys.exit(1)
+
+sys.exit(0)
diff --git a/drivers/gpu/drm/ci/container.yml b/drivers/gpu/drm/ci/container.yml
new file mode 100644
index 000000000000..9764e7921a4f
--- /dev/null
+++ b/drivers/gpu/drm/ci/container.yml
@@ -0,0 +1,65 @@
+.container:
+  variables:
+    CI_REPOSITORY_URL: ${DRM_CI_PROJECT_URL}.git # So ci-templates clones drm-ci instead of the repo to test
+    CI_COMMIT_SHA: ${DRM_CI_COMMIT_SHA}
+
+debian/x86_64_build-base:
+  variables:
+    EXTRA_LOCAL_PACKAGES: "libcairo-dev libdw-dev libjson-c-dev libkmod2 libkmod-dev libpciaccess-dev libproc2-dev libudev-dev libunwind-dev python3-docutils bc python3-ply libssl-dev bc"
+
+debian/x86_64_test-gl:
+  variables:
+    EXTRA_LOCAL_PACKAGES: "jq libasound2 libcairo2 libdw1 libglib2.0-0 libjson-c5 libkmod-dev libkmod2 libgles2 libproc2-dev"
+
+debian/arm64_build:
+  variables:
+    EXTRA_LOCAL_PACKAGES: "libcairo-dev libdw-dev libjson-c-dev libproc2-dev libkmod2 libkmod-dev libpciaccess-dev libudev-dev libunwind-dev python3-docutils libssl-dev crossbuild-essential-armhf libkmod-dev:armhf libproc2-dev:armhf libunwind-dev:armhf libdw-dev:armhf libpixman-1-dev:armhf libcairo-dev:armhf libudev-dev:armhf libjson-c-dev:armhf"
+
+.kernel+rootfs:
+  variables:
+    EXTRA_LOCAL_PACKAGES: "jq libasound2 libcairo2 libdw1 libglib2.0-0 libjson-c5"
+
+# Disable container jobs that we won't use
+alpine/x86_64_build:
+  rules:
+    - when: never
+
+debian/x86_64_test-vk:
+  rules:
+    - when: never
+
+fedora/x86_64_build:
+  rules:
+    - when: never
+
+debian/android_build:
+  rules:
+    - when: never
+
+debian/x86_64_test-android:
+  rules:
+    - when: never
+
+windows_build_vs2019:
+  rules:
+    - when: never
+
+windows_test_vs2019:
+  rules:
+    - when: never
+
+.debian/x86_64_build-mingw:
+   rules:
+    - when: never
+
+rustfmt:
+   rules:
+    - when: never
+
+windows_vs2019:
+   rules:
+    - when: never
+
+clang-format:
+   rules:
+    - when: never
\ No newline at end of file
diff --git a/drivers/gpu/drm/ci/gitlab-ci.yml b/drivers/gpu/drm/ci/gitlab-ci.yml
new file mode 100644
index 000000000000..2c4df53f5dfe
--- /dev/null
+++ b/drivers/gpu/drm/ci/gitlab-ci.yml
@@ -0,0 +1,251 @@
+variables:
+  DRM_CI_PROJECT_PATH: &drm-ci-project-path mesa/mesa
+  DRM_CI_COMMIT_SHA: &drm-ci-commit-sha 0dc961645c4f0241f8512cb0ec3ad59635842072
+
+  UPSTREAM_REPO: git://anongit.freedesktop.org/drm/drm
+  TARGET_BRANCH: drm-next
+
+  IGT_VERSION: 471bfababd070e1dac0ebb87470ac4f2ae85e663
+
+  DEQP_RUNNER_GIT_URL: https://gitlab.freedesktop.org/anholt/deqp-runner.git
+  DEQP_RUNNER_GIT_TAG: v0.15.0
+
+  FDO_UPSTREAM_REPO: helen.fornazier/linux   # The repo where the git-archive daily runs
+  MESA_TEMPLATES_COMMIT: &ci-templates-commit d5aa3941aa03c2f716595116354fb81eb8012acb
+  DRM_CI_PROJECT_URL: https://gitlab.freedesktop.org/${DRM_CI_PROJECT_PATH}
+  CI_PRE_CLONE_SCRIPT: |-
+          set -o xtrace
+          curl -L --retry 4 -f --retry-all-errors --retry-delay 60 -s ${DRM_CI_PROJECT_URL}/-/raw/${DRM_CI_COMMIT_SHA}/.gitlab-ci/download-git-cache.sh -o download-git-cache.sh
+          bash download-git-cache.sh
+          rm download-git-cache.sh
+          set +o xtrace
+  S3_HOST: s3.freedesktop.org
+  # per-pipeline artifact storage on MinIO
+  PIPELINE_ARTIFACTS_BASE: ${S3_HOST}/artifacts/${CI_PROJECT_PATH}/${CI_PIPELINE_ID}
+  # per-job artifact storage on MinIO
+  JOB_ARTIFACTS_BASE: ${PIPELINE_ARTIFACTS_BASE}/${CI_JOB_ID}
+
+  LAVA_JOB_PRIORITY: 30
+
+default:
+  before_script:
+    - export SCRIPTS_DIR=$(mktemp -d)
+    - curl -L -s --retry 4 -f --retry-all-errors --retry-delay 60 -O --output-dir "${SCRIPTS_DIR}" "${DRM_CI_PROJECT_URL}/-/raw/${DRM_CI_COMMIT_SHA}/.gitlab-ci/setup-test-env.sh"
+    - source ${SCRIPTS_DIR}/setup-test-env.sh
+    - echo -e "\e[0Ksection_start:$(date +%s):unset_env_vars_section[collapsed=true]\r\e[0KUnsetting vulnerable environment variables"
+    - export CI_JOB_JWT_FILE="${CI_JOB_JWT_FILE:-$(mktemp)}"
+    - echo -n "${CI_JOB_JWT}" > "${CI_JOB_JWT_FILE}"
+    - unset CI_JOB_JWT
+    - echo -e "\e[0Ksection_end:$(date +%s):unset_env_vars_section\r\e[0K"
+
+    - echo -e "\e[0Ksection_start:$(date +%s):drm_ci_download_section[collapsed=true]\r\e[0KDownloading mesa from $DRM_CI_PROJECT_URL/-/archive/$DRM_CI_COMMIT_SHA/mesa-$DRM_CI_COMMIT_SHA.tar.gz"
+    - cd $CI_PROJECT_DIR
+    - curl --output - $DRM_CI_PROJECT_URL/-/archive/$DRM_CI_COMMIT_SHA/mesa-$DRM_CI_COMMIT_SHA.tar.gz | tar -xz
+    - mv mesa-$DRM_CI_COMMIT_SHA/.gitlab-ci* .
+    - rm -rf mesa-$DRM_CI_COMMIT_SHA/
+    - echo -e "\e[0Ksection_end:$(date +%s):drm_ci_download_section\r\e[0K"
+
+  after_script:
+    - >
+      set +x
+
+      test -e "${CI_JOB_JWT_FILE}" &&
+      export CI_JOB_JWT="$(<${CI_JOB_JWT_FILE})" &&
+      rm "${CI_JOB_JWT_FILE}"
+
+  # Retry when job fails.
+  retry:
+    max: 1
+    # Ignore runner_unsupported, stale_schedule, archived_failure, or
+    # unmet_prerequisites
+    when:
+      - api_failure
+      - runner_system_failure
+      - script_failure
+      - job_execution_timeout
+      - scheduler_failure
+      - data_integrity_failure
+      - unknown_failure
+
+include:
+  - project: 'freedesktop/ci-templates'
+    ref: 16bc29078de5e0a067ff84a1a199a3760d3b3811
+    file:
+      - '/templates/ci-fairy.yml'
+  - project: 'freedesktop/ci-templates'
+    ref: *ci-templates-commit
+    file:
+      - '/templates/alpine.yml'
+      - '/templates/debian.yml'
+      - '/templates/fedora.yml'
+  - project: *drm-ci-project-path
+    ref: *drm-ci-commit-sha
+    file:
+      - '/.gitlab-ci/farm-rules.yml'
+      - '/.gitlab-ci/test-source-dep.yml'
+      - '/.gitlab-ci/container/gitlab-ci.yml'
+      - '/.gitlab-ci/test/gitlab-ci.yml'
+      - '/.gitlab-ci/lava/lava-gitlab-ci.yml'
+  - drivers/gpu/drm/ci/image-tags.yml
+  - drivers/gpu/drm/ci/container.yml
+  - drivers/gpu/drm/ci/static-checks.yml
+  - drivers/gpu/drm/ci/build.yml
+  - drivers/gpu/drm/ci/test.yml
+  - 'https://gitlab.freedesktop.org/gfx-ci/lab-status/-/raw/main/lab-status.yml'
+
+
+stages:
+  - sanity
+  - container
+  - git-archive
+  - build
+  - amdgpu
+  - i915
+  - mediatek
+  - meson
+  - msm
+  - rockchip
+  - virtio-gpu
+  - lint
+
+# YAML anchors for rule conditions
+# --------------------------------
+.rules-anchors:
+  rules:
+    # Pipeline for forked project branch
+    - if: &is-forked-branch '$CI_COMMIT_BRANCH && $CI_PROJECT_NAMESPACE != "mesa"'
+      when: manual
+    # Forked project branch / pre-merge pipeline not for Marge bot
+    - if: &is-forked-branch-or-pre-merge-not-for-marge '$CI_PROJECT_NAMESPACE != "mesa" || ($GITLAB_USER_LOGIN != "marge-bot" && $CI_PIPELINE_SOURCE == "merge_request_event")'
+      when: manual
+    # Pipeline runs for the main branch of the upstream Mesa project
+    - if: &is-mesa-main '$CI_PROJECT_NAMESPACE == "mesa" && $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH && $CI_COMMIT_BRANCH'
+      when: always
+    # Post-merge pipeline
+    - if: &is-post-merge '$CI_PROJECT_NAMESPACE == "mesa" && $CI_COMMIT_BRANCH'
+      when: on_success
+    # Post-merge pipeline, not for Marge Bot
+    - if: &is-post-merge-not-for-marge '$CI_PROJECT_NAMESPACE == "mesa" && $GITLAB_USER_LOGIN != "marge-bot" && $CI_COMMIT_BRANCH'
+      when: on_success
+    # Pre-merge pipeline
+    - if: &is-pre-merge '$CI_PIPELINE_SOURCE == "merge_request_event"'
+      when: on_success
+    # Pre-merge pipeline for Marge Bot
+    - if: &is-pre-merge-for-marge '$GITLAB_USER_LOGIN == "marge-bot" && $CI_PIPELINE_SOURCE == "merge_request_event"'
+      when: on_success
+
+# Rule to filter for only scheduled pipelines.
+.scheduled_pipeline-rules:
+  rules:
+    - if: &is-scheduled-pipeline '$CI_PIPELINE_SOURCE == "schedule"'
+      when: on_success
+
+# Generic rule to not run the job during scheduled pipelines. Jobs that aren't
+# something like a nightly run should include this rule.
+.no_scheduled_pipelines-rules:
+  rules:
+    - if: *is-scheduled-pipeline
+      when: never
+
+# When to automatically run the CI for build jobs
+.build-rules:
+  rules:
+    - !reference [.no_scheduled_pipelines-rules, rules]
+    # Run automatically once all dependency jobs have passed
+    - when: on_success
+
+
+.ci-deqp-artifacts:
+  artifacts:
+    name: "mesa_${CI_JOB_NAME}"
+    when: always
+    untracked: false
+    paths:
+      # Watch out!  Artifacts are relative to the build dir.
+      # https://gitlab.com/gitlab-org/gitlab-ce/commit/8788fb925706cad594adf6917a6c5f6587dd1521
+      - artifacts
+      - _build/meson-logs/*.txt
+      - _build/meson-logs/strace
+
+
+.container-rules:
+  rules:
+    - !reference [.no_scheduled_pipelines-rules, rules]
+    # Run pipeline by default in the main project if any CI pipeline
+    # configuration files were changed, to ensure docker images are up to date
+    - if: *is-post-merge
+      changes:
+      - drivers/gpu/drm/ci/**/*
+      when: on_success
+    # Run pipeline by default if it was triggered by Marge Bot, is for a
+    # merge request, and any files affecting the pipeline were changed
+    - if: *is-pre-merge-for-marge
+      when: on_success
+    # Run pipeline by default in the main project if it was not triggered by
+    # Marge Bot, and any files affecting the pipeline were changed
+    - if: *is-post-merge-not-for-marge
+      when: on_success
+    # Allow triggering jobs manually in other cases
+    - when: manual
+
+
+
+# Git archive
+
+make git archive:
+  extends:
+    - .fdo.ci-fairy
+  stage: git-archive
+  rules:
+    - !reference [.scheduled_pipeline-rules, rules]
+  # ensure we are running on packet
+  tags:
+    - packet.net
+  script:
+    # Remove drm-ci files we just added
+    - rm -rf .gitlab-ci.*
+
+    # Compactify the .git directory
+    - git gc --aggressive
+    # compress the current folder
+    - tar -cvzf ../$CI_PROJECT_NAME.tar.gz .
+
+    # login with the JWT token file
+    - ci-fairy s3cp --token-file "${CI_JOB_JWT_FILE}" ../$CI_PROJECT_NAME.tar.gz https://$S3_HOST/git-cache/$CI_PROJECT_NAMESPACE/$CI_PROJECT_NAME/$CI_PROJECT_NAME.tar.gz
+
+
+# Sanity checks of MR settings and commit logs
+sanity:
+  extends:
+    - .fdo.ci-fairy
+  stage: sanity
+  rules:
+    - if: *is-pre-merge
+      when: on_success
+    # Other cases default to never
+  variables:
+    GIT_STRATEGY: none
+  script:
+    # ci-fairy check-commits --junit-xml=check-commits.xml
+    - ci-fairy check-merge-request --require-allow-collaboration --junit-xml=check-merge-request.xml
+  artifacts:
+    when: on_failure
+    reports:
+      junit: check-*.xml
+
+# Rules for tests that should not block merging, but should be available to
+# optionally run with the "play" button in the UI in pre-merge non-marge
+# pipelines.  This should appear in "extends:" after any includes of
+# test-source-dep.yml rules, so that these rules replace those.
+.test-manual-mr:
+  rules:
+    - !reference [.no_scheduled_pipelines-rules, rules]
+    - if: *is-forked-branch-or-pre-merge-not-for-marge
+      when: manual
+  variables:
+    JOB_TIMEOUT: 80
+
+
+# Jobs that need to pass before spending hardware resources on further testing
+.required-for-hardware-jobs:
+  needs: []
\ No newline at end of file
diff --git a/drivers/gpu/drm/ci/igt_runner.sh b/drivers/gpu/drm/ci/igt_runner.sh
new file mode 100755
index 000000000000..2bb759165063
--- /dev/null
+++ b/drivers/gpu/drm/ci/igt_runner.sh
@@ -0,0 +1,77 @@
+#!/bin/sh
+# SPDX-License-Identifier: MIT
+
+set -ex
+
+export IGT_FORCE_DRIVER=${DRIVER_NAME}
+export PATH=$PATH:/igt/bin/
+export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/igt/lib/aarch64-linux-gnu/:/igt/lib/x86_64-linux-gnu:/igt/lib:/igt/lib64
+
+# Uncomment the below to debug problems with driver probing
+: '
+ls -l /dev/dri/
+cat /sys/kernel/debug/devices_deferred
+cat /sys/kernel/debug/device_component/*
+'
+
+# Dump drm state to confirm that kernel was able to find a connected display:
+# TODO this path might not exist for all drivers.. maybe run modetest instead?
+set +e
+cat /sys/kernel/debug/dri/*/state
+set -e
+
+# Cannot use HWCI_KERNEL_MODULES as at that point we don't have the module in /lib
+if [ "$IGT_FORCE_DRIVER" = "amdgpu" ]; then
+    mv /install/modules/lib/modules/* /lib/modules/.
+    modprobe amdgpu
+fi
+
+if [ -e "/install/xfails/$DRIVER_NAME-$GPU_VERSION-skips.txt" ]; then
+    IGT_SKIPS="--skips /install/xfails/$DRIVER_NAME-$GPU_VERSION-skips.txt"
+fi
+
+if [ -e "/install/xfails/$DRIVER_NAME-$GPU_VERSION-flakes.txt" ]; then
+    IGT_FLAKES="--flakes /install/xfails/$DRIVER_NAME-$GPU_VERSION-flakes.txt"
+fi
+
+if [ -e "/install/xfails/$DRIVER_NAME-$GPU_VERSION-fails.txt" ]; then
+    IGT_FAILS="--baseline /install/xfails/$DRIVER_NAME-$GPU_VERSION-fails.txt"
+fi
+
+if [ "`uname -m`" = "aarch64" ]; then
+    ARCH="arm64"
+elif [ "`uname -m`" = "armv7l" ]; then
+    ARCH="arm"
+else
+    ARCH="x86_64"
+fi
+
+curl -L --retry 4 -f --retry-all-errors --retry-delay 60 -s ${FDO_HTTP_CACHE_URI:-}$PIPELINE_ARTIFACTS_BASE/$ARCH/igt.tar.gz | tar --zstd -v -x -C /
+
+set +e
+igt-runner \
+    run \
+    --igt-folder /igt/libexec/igt-gpu-tools \
+    --caselist /install/testlist.txt \
+    --output /results \
+    $IGT_SKIPS \
+    $IGT_FLAKES \
+    $IGT_FAILS \
+    --fraction-start $CI_NODE_INDEX \
+    --fraction $CI_NODE_TOTAL \
+    --jobs 1
+ret=$?
+set -e
+
+deqp-runner junit \
+   --testsuite IGT \
+   --results /results/failures.csv \
+   --output /results/junit.xml \
+   --limit 50 \
+   --template "See https://$CI_PROJECT_ROOT_NAMESPACE.pages.freedesktop.org/-/$CI_PROJECT_NAME/-/jobs/$CI_JOB_ID/artifacts/results/{{testcase}}.xml"
+
+# Store the results also in the simpler format used by the runner in ChromeOS CI
+#sed -r 's/(dmesg-warn|pass)/success/g' /results/results.txt > /results/results_simple.txt
+
+cd $oldpath
+exit $ret
diff --git a/drivers/gpu/drm/ci/image-tags.yml b/drivers/gpu/drm/ci/image-tags.yml
new file mode 100644
index 000000000000..f051b6c547c5
--- /dev/null
+++ b/drivers/gpu/drm/ci/image-tags.yml
@@ -0,0 +1,15 @@
+variables:
+   CONTAINER_TAG: "2023-08-10-mesa-uprev"
+   DEBIAN_X86_64_BUILD_BASE_IMAGE: "debian/x86_64_build-base"
+   DEBIAN_BASE_TAG: "${CONTAINER_TAG}"
+
+   DEBIAN_X86_64_BUILD_IMAGE_PATH: "debian/x86_64_build"
+   DEBIAN_BUILD_TAG: "${CONTAINER_TAG}"
+
+   KERNEL_ROOTFS_TAG: "${CONTAINER_TAG}"
+
+   DEBIAN_X86_64_TEST_BASE_IMAGE: "debian/x86_64_test-base"
+   DEBIAN_X86_64_TEST_IMAGE_GL_PATH: "debian/x86_64_test-gl"
+   DEBIAN_X86_64_TEST_GL_TAG: "${CONTAINER_TAG}"
+
+   ALPINE_X86_64_LAVA_SSH_TAG: "${CONTAINER_TAG}"
\ No newline at end of file
diff --git a/drivers/gpu/drm/ci/lava-submit.sh b/drivers/gpu/drm/ci/lava-submit.sh
new file mode 100755
index 000000000000..0c4456b21b0f
--- /dev/null
+++ b/drivers/gpu/drm/ci/lava-submit.sh
@@ -0,0 +1,57 @@
+#!/bin/bash
+# SPDX-License-Identifier: MIT
+
+set -e
+set -x
+
+# Try to use the kernel and rootfs built in mainline first, so we're more
+# likely to hit cache
+if curl -L --retry 4 -f --retry-all-errors --retry-delay 60 -s "https://${BASE_SYSTEM_MAINLINE_HOST_PATH}/done"; then
+	BASE_SYSTEM_HOST_PATH="${BASE_SYSTEM_MAINLINE_HOST_PATH}"
+else
+	BASE_SYSTEM_HOST_PATH="${BASE_SYSTEM_FORK_HOST_PATH}"
+fi
+
+rm -rf results
+mkdir -p results/job-rootfs-overlay/
+
+cp artifacts/ci-common/capture-devcoredump.sh results/job-rootfs-overlay/
+cp artifacts/ci-common/init-*.sh results/job-rootfs-overlay/
+cp artifacts/ci-common/intel-gpu-freq.sh results/job-rootfs-overlay/
+cp "$SCRIPTS_DIR"/setup-test-env.sh results/job-rootfs-overlay/
+
+# Prepare env vars for upload.
+section_start variables "Variables passed through:"
+KERNEL_IMAGE_BASE_URL="https://${BASE_SYSTEM_HOST_PATH}" \
+	artifacts/ci-common/generate-env.sh | tee results/job-rootfs-overlay/set-job-env-vars.sh
+section_end variables
+
+tar zcf job-rootfs-overlay.tar.gz -C results/job-rootfs-overlay/ .
+ci-fairy s3cp --token-file "${CI_JOB_JWT_FILE}" job-rootfs-overlay.tar.gz "https://${JOB_ROOTFS_OVERLAY_PATH}"
+
+touch results/lava.log
+tail -f results/lava.log &
+
+PYTHONPATH=artifacts/ artifacts/lava/lava_job_submitter.py \
+	submit \
+	--dump-yaml \
+	--pipeline-info "$CI_JOB_NAME: $CI_PIPELINE_URL on $CI_COMMIT_REF_NAME ${CI_NODE_INDEX}/${CI_NODE_TOTAL}" \
+	--rootfs-url-prefix "https://${BASE_SYSTEM_HOST_PATH}" \
+	--kernel-url-prefix "https://${PIPELINE_ARTIFACTS_BASE}/${ARCH}" \
+	--build-url "${FDO_HTTP_CACHE_URI:-}https://${PIPELINE_ARTIFACTS_BASE}/${ARCH}/kernel-files.tar.zst" \
+	--job-rootfs-overlay-url "${FDO_HTTP_CACHE_URI:-}https://${JOB_ROOTFS_OVERLAY_PATH}" \
+	--job-timeout-min ${JOB_TIMEOUT:-80} \
+	--first-stage-init artifacts/ci-common/init-stage1.sh \
+	--ci-project-dir "${CI_PROJECT_DIR}" \
+	--device-type "${DEVICE_TYPE}" \
+	--dtb-filename "${DTB}" \
+	--jwt-file "${CI_JOB_JWT_FILE}" \
+	--kernel-image-name "${KERNEL_IMAGE_NAME}" \
+	--kernel-image-type "${KERNEL_IMAGE_TYPE}" \
+	--boot-method "${BOOT_METHOD}" \
+	--visibility-group "${VISIBILITY_GROUP}" \
+	--lava-tags "${LAVA_TAGS}" \
+	--mesa-job-name "$CI_JOB_NAME" \
+	--structured-log-file "results/lava_job_detail.json" \
+	--ssh-client-image "${LAVA_SSH_CLIENT_IMAGE}" \
+	>> results/lava.log
diff --git a/drivers/gpu/drm/ci/static-checks.yml b/drivers/gpu/drm/ci/static-checks.yml
new file mode 100644
index 000000000000..13ffa827b7fa
--- /dev/null
+++ b/drivers/gpu/drm/ci/static-checks.yml
@@ -0,0 +1,12 @@
+check-patch:
+  extends:
+    - .build
+    - .use-debian/x86_64_build
+  script:
+    - drivers/gpu/drm/ci/check-patch.py
+  variables:
+    CHECKPATCH_TYPES: "BAD_SIGN_OFF,BAD_STABLE_ADDRESS_STYLE,COMMIT_COMMENT_SYMBOL,COMMIT_MESSAGE,EMAIL_SUBJECT,FROM_SIGN_OFF_MISMATCH,MISSING_SIGN_OFF,NO_AUTHOR_SIGN_OFF,DIFF_IN_COMMIT_MSG,GERRIT_CHANGE_ID,GIT_COMMIT_ID,UNKNOWN_COMMIT_ID,CODE_INDENT,BIT_MACRO,DOS_LINE_ENDINGS"
+  rules:
+    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
+      when: on_success
+    # Other cases default to never
diff --git a/drivers/gpu/drm/ci/test.yml b/drivers/gpu/drm/ci/test.yml
new file mode 100644
index 000000000000..6473cddaa7a9
--- /dev/null
+++ b/drivers/gpu/drm/ci/test.yml
@@ -0,0 +1,335 @@
+.test-rules:
+  rules:
+    - if: '$FD_FARM == "offline" && $RUNNER_TAG =~ /^google-freedreno-/'
+      when: never
+    - if: '$COLLABORA_FARM == "offline" && $RUNNER_TAG =~ /^mesa-ci-x86-64-lava-/'
+      when: never
+    - !reference [.no_scheduled_pipelines-rules, rules]
+    - when: on_success
+
+.lava-test:
+  extends:
+    - .test-rules
+  script:
+    # Note: Build dir (and thus install) may be dirty due to GIT_STRATEGY
+    - rm -rf install
+    - tar -xf artifacts/install.tar
+    - mv install/* artifacts/.
+    # Override it with our lava-submit.sh script
+    - ./artifacts/lava-submit.sh
+
+.lava-igt:arm32:
+  extends:
+    - .lava-test:arm32
+  variables:
+    HWCI_TEST_SCRIPT: "/install/igt_runner.sh"
+    ARCH: "armhf"
+  dependencies:
+    - testing:arm32
+  needs:
+    - alpine/x86_64_lava_ssh_client
+    - kernel+rootfs_arm32
+    - debian/x86_64_build
+    - testing:arm32
+    - igt:arm32
+
+.lava-igt:arm64:
+  extends:
+    - .lava-test:arm64
+  variables:
+    HWCI_TEST_SCRIPT: "/install/igt_runner.sh"
+    ARCH: "arm64"
+  dependencies:
+    - testing:arm64
+  needs:
+    - alpine/x86_64_lava_ssh_client
+    - kernel+rootfs_arm64
+    - debian/x86_64_build
+    - testing:arm64
+    - igt:arm64
+
+.lava-igt:x86_64:
+  extends:
+    - .lava-test:x86_64
+  variables:
+    HWCI_TEST_SCRIPT: "/install/igt_runner.sh"
+    ARCH: "x86_64"
+  dependencies:
+    - testing:x86_64
+  needs:
+    - alpine/x86_64_lava_ssh_client
+    - kernel+rootfs_x86_64
+    - debian/x86_64_build
+    - testing:x86_64
+    - igt:x86_64
+
+.baremetal-igt-arm64:
+  extends:
+    - .baremetal-test-arm64
+    - .use-debian/arm64_test
+    - .test-rules
+  variables:
+    FDO_CI_CONCURRENT: 10
+    HWCI_TEST_SCRIPT: "/install/igt_runner.sh"
+    S3_ARTIFACT_NAME: "arm64/kernel-files"
+    BM_KERNEL: https://${PIPELINE_ARTIFACTS_BASE}/arm64/Image.gz
+    BM_CMDLINE: "ip=dhcp console=ttyMSM0,115200n8 $BM_KERNEL_EXTRA_ARGS root=/dev/nfs rw nfsrootdebug nfsroot=,tcp,nfsvers=4.2 init=/init $BM_KERNELARGS"
+  needs:
+    - debian/arm64_test
+    - job: testing:arm64
+      artifacts: false
+    - igt:arm64
+  tags:
+    - $RUNNER_TAG
+
+msm:sc7180:
+  extends:
+    - .lava-igt:arm64
+  stage: msm
+  parallel: 2
+  variables:
+    DRIVER_NAME: msm
+    DEVICE_TYPE: sc7180-trogdor-lazor-limozeen
+    DTB: sc7180-trogdor-lazor-limozeen-nots-r5
+    BOOT_METHOD: depthcharge
+    KERNEL_IMAGE_TYPE: ""
+    GPU_VERSION: sc7180
+    RUNNER_TAG: mesa-ci-x86-64-lava-sc7180-trogdor-lazor-limozeen
+
+msm:apq8016:
+  extends:
+    - .baremetal-igt-arm64
+  stage: msm
+  variables:
+    DRIVER_NAME: msm
+    BM_DTB: https://${PIPELINE_ARTIFACTS_BASE}/arm64/apq8016-sbc.dtb
+    GPU_VERSION: apq8016
+    BM_CMDLINE: "ip=dhcp console=ttyMSM0,115200n8 $BM_KERNEL_EXTRA_ARGS root=/dev/nfs rw nfsrootdebug nfsroot=,tcp,nfsvers=4.2 init=/init $BM_KERNELARGS"
+    RUNNER_TAG: google-freedreno-db410c
+  script:
+    - ./install/bare-metal/fastboot.sh
+  rules:
+    # TODO: current issue: it is not fiding the NFS root. Fix and remove this rule.
+    - when: never
+
+msm:apq8096:
+  extends:
+    - .baremetal-igt-arm64
+  stage: msm
+  variables:
+    DRIVER_NAME: msm
+    BM_KERNEL_EXTRA_ARGS: maxcpus=2
+    BM_DTB: https://${PIPELINE_ARTIFACTS_BASE}/arm64/apq8096-db820c.dtb
+    GPU_VERSION: apq8096
+    RUNNER_TAG: google-freedreno-db820c
+  script:
+    - ./install/bare-metal/fastboot.sh
+
+msm:sdm845:
+  extends:
+    - .baremetal-igt-arm64
+  stage: msm
+  parallel: 6
+  variables:
+    DRIVER_NAME: msm
+    BM_KERNEL: https://${PIPELINE_ARTIFACTS_BASE}/arm64/cheza-kernel
+    GPU_VERSION: sdm845
+    RUNNER_TAG: google-freedreno-cheza
+  script:
+    - ./install/bare-metal/cros-servo.sh
+
+rockchip:rk3288:
+  extends:
+    - .lava-igt:arm32
+  stage: rockchip
+  variables:
+    DRIVER_NAME: rockchip
+    DEVICE_TYPE: rk3288-veyron-jaq
+    DTB: ${DEVICE_TYPE}
+    BOOT_METHOD: depthcharge
+    KERNEL_IMAGE_TYPE: "zimage"
+    GPU_VERSION: rk3288
+    RUNNER_TAG: mesa-ci-x86-64-lava-rk3288-veyron-jaq
+
+rockchip:rk3399:
+  extends:
+    - .lava-igt:arm64
+  stage: rockchip
+  parallel: 3
+  variables:
+    DRIVER_NAME: rockchip
+    DEVICE_TYPE: rk3399-gru-kevin
+    DTB: ${DEVICE_TYPE}
+    BOOT_METHOD: depthcharge
+    KERNEL_IMAGE_TYPE: ""
+    GPU_VERSION: rk3399
+    RUNNER_TAG: mesa-ci-x86-64-lava-rk3399-gru-kevin
+
+.i915:
+  extends:
+    - .lava-igt:x86_64
+  stage: i915
+  variables:
+    DRIVER_NAME: i915
+    DTB: ""
+    BOOT_METHOD: depthcharge
+    KERNEL_IMAGE_TYPE: ""
+
+i915:apl:
+  extends:
+    - .i915
+  parallel: 12
+  variables:
+    DEVICE_TYPE: asus-C523NA-A20057-coral
+    GPU_VERSION: apl
+    RUNNER_TAG: mesa-ci-x86-64-lava-asus-C523NA-A20057-coral
+
+i915:glk:
+  extends:
+    - .i915
+  parallel: 5
+  variables:
+    DEVICE_TYPE: hp-x360-12b-ca0010nr-n4020-octopus
+    GPU_VERSION: glk
+    RUNNER_TAG: mesa-ci-x86-64-lava-hp-x360-12b-ca0010nr-n4020-octopus
+
+i915:amly:
+  extends:
+    - .i915
+  parallel: 8
+  variables:
+    DEVICE_TYPE: asus-C433TA-AJ0005-rammus
+    GPU_VERSION: amly
+    RUNNER_TAG: mesa-ci-x86-64-lava-asus-C433TA-AJ0005-rammus
+
+i915:kbl:
+  extends:
+    - .i915
+  parallel: 5
+  variables:
+    DEVICE_TYPE: hp-x360-14-G1-sona
+    GPU_VERSION: kbl
+    RUNNER_TAG: mesa-ci-x86-64-lava-hp-x360-14-G1-sona
+
+i915:whl:
+  extends:
+    - .i915
+  parallel: 8
+  variables:
+    DEVICE_TYPE: dell-latitude-5400-8665U-sarien
+    GPU_VERSION: whl
+    RUNNER_TAG: mesa-ci-x86-64-lava-dell-latitude-5400-8665U-sarien
+
+i915:cml:
+  extends:
+    - .i915
+  parallel: 6
+  variables:
+    DEVICE_TYPE: asus-C436FA-Flip-hatch
+    GPU_VERSION: cml
+    RUNNER_TAG: mesa-ci-x86-64-lava-asus-C436FA-flip-hatch
+
+i915:tgl:
+  extends:
+    - .i915
+  parallel: 6
+  variables:
+    DEVICE_TYPE: asus-cx9400-volteer
+    GPU_VERSION: tgl
+    RUNNER_TAG: mesa-ci-x86-64-lava-asus-cx9400-volteer
+
+.amdgpu:
+  extends:
+    - .lava-igt:x86_64
+  stage: amdgpu
+  variables:
+    DRIVER_NAME: amdgpu
+    DTB: ""
+    BOOT_METHOD: depthcharge
+    KERNEL_IMAGE_TYPE: ""
+
+amdgpu:stoney:
+  extends:
+    - .amdgpu
+  variables:
+    DEVICE_TYPE: hp-11A-G6-EE-grunt
+    GPU_VERSION: stoney
+    RUNNER_TAG: mesa-ci-x86-64-lava-hp-11A-G6-EE-grunt
+
+.mediatek:
+  extends:
+    - .lava-igt:arm64
+  stage: mediatek
+  variables:
+    DRIVER_NAME: mediatek
+    DTB: ${DEVICE_TYPE}
+    BOOT_METHOD: depthcharge
+    KERNEL_IMAGE_TYPE: ""
+
+mediatek:mt8173:
+  extends:
+    - .mediatek
+  variables:
+    DEVICE_TYPE: mt8173-elm-hana
+    GPU_VERSION: mt8173
+    RUNNER_TAG: mesa-ci-x86-64-lava-mt8173-elm-hana
+  rules:
+    # TODO: current issue: device is hanging. Fix and remove this rule.
+    - when: never
+
+mediatek:mt8183:
+  extends:
+    - .mediatek
+  variables:
+    DEVICE_TYPE: mt8183-kukui-jacuzzi-juniper-sku16
+    GPU_VERSION: mt8183
+    RUNNER_TAG: mesa-ci-x86-64-lava-mt8183-kukui-jacuzzi-juniper-sku16
+
+# drm-mtk doesn't even probe yet in mainline for mt8192
+.mediatek:mt8192:
+  extends:
+    - .mediatek
+  variables:
+    DEVICE_TYPE: mt8192-asurada-spherion-r0
+    GPU_VERSION: mt8192
+    RUNNER_TAG: mesa-ci-x86-64-lava-mt8192-asurada-spherion-r0
+
+.meson:
+  extends:
+    - .lava-igt:arm64
+  stage: meson
+  variables:
+    DRIVER_NAME: meson
+    DTB: ${DEVICE_TYPE}
+    BOOT_METHOD: u-boot
+    KERNEL_IMAGE_TYPE: "image"
+
+meson:g12b:
+  extends:
+    - .meson
+  variables:
+    DEVICE_TYPE: meson-g12b-a311d-khadas-vim3
+    GPU_VERSION: g12b
+    RUNNER_TAG: mesa-ci-x86-64-lava-meson-g12b-a311d-khadas-vim3
+
+virtio_gpu:none:
+  stage: virtio-gpu
+  variables:
+    CROSVM_GALLIUM_DRIVER: llvmpipe
+    DRIVER_NAME: virtio_gpu
+    GPU_VERSION: none
+  extends:
+    - .test-gl
+  tags:
+    - kvm
+  script:
+    - ln -sf $CI_PROJECT_DIR/install /install
+    - mv install/bzImage /lava-files/bzImage
+    - install/crosvm-runner.sh install/igt_runner.sh
+  needs:
+    - debian/x86_64_test-gl
+    - testing:x86_64
+    - igt:x86_64
+  rules:
+    # TODO: current issue: malloc(): corrupted top size. Fix and remove this rule.
+    - when: never
\ No newline at end of file
diff --git a/drivers/gpu/drm/ci/testlist.txt b/drivers/gpu/drm/ci/testlist.txt
new file mode 100644
index 000000000000..f82cd90372f4
--- /dev/null
+++ b/drivers/gpu/drm/ci/testlist.txt
@@ -0,0 +1,2912 @@
+core_auth@getclient-simple
+core_auth@getclient-master-drop
+core_auth@basic-auth
+core_auth@many-magics
+core_getclient
+core_getstats
+core_getversion
+core_setmaster_vs_auth
+drm_read@invalid-buffer
+drm_read@fault-buffer
+drm_read@empty-block
+drm_read@empty-nonblock
+drm_read@short-buffer-block
+drm_read@short-buffer-nonblock
+drm_read@short-buffer-wakeup
+gem_eio@throttle
+gem_eio@create
+gem_eio@create-ext
+gem_eio@context-create
+gem_eio@execbuf
+gem_eio@banned
+gem_eio@suspend
+gem_eio@hibernate
+gem_eio@in-flight-external
+gem_eio@in-flight-suspend
+gem_eio@reset-stress
+gem_eio@unwedge-stress
+gem_eio@wait-immediate
+gem_eio@wait-wedge-immediate
+gem_eio@in-flight-immediate
+gem_eio@in-flight-contexts-immediate
+gem_eio@in-flight-internal-immediate
+gem_eio@wait-1us
+gem_eio@wait-wedge-1us
+gem_eio@in-flight-1us
+gem_eio@in-flight-contexts-1us
+gem_eio@in-flight-internal-1us
+gem_eio@wait-10ms
+gem_eio@wait-wedge-10ms
+gem_eio@in-flight-10ms
+gem_eio@in-flight-contexts-10ms
+gem_eio@in-flight-internal-10ms
+gem_eio@kms
+kms_3d
+kms_addfb_basic@unused-handle
+kms_addfb_basic@unused-pitches
+kms_addfb_basic@unused-offsets
+kms_addfb_basic@unused-modifier
+kms_addfb_basic@clobberred-modifier
+kms_addfb_basic@invalid-smem-bo-on-discrete
+kms_addfb_basic@legacy-format
+kms_addfb_basic@no-handle
+kms_addfb_basic@basic
+kms_addfb_basic@bad-pitch-0
+kms_addfb_basic@bad-pitch-32
+kms_addfb_basic@bad-pitch-63
+kms_addfb_basic@bad-pitch-128
+kms_addfb_basic@bad-pitch-256
+kms_addfb_basic@bad-pitch-1024
+kms_addfb_basic@bad-pitch-999
+kms_addfb_basic@bad-pitch-65536
+kms_addfb_basic@invalid-get-prop-any
+kms_addfb_basic@invalid-get-prop
+kms_addfb_basic@invalid-set-prop-any
+kms_addfb_basic@invalid-set-prop
+kms_addfb_basic@master-rmfb
+kms_addfb_basic@addfb25-modifier-no-flag
+kms_addfb_basic@addfb25-bad-modifier
+kms_addfb_basic@addfb25-x-tiled-mismatch-legacy
+kms_addfb_basic@addfb25-x-tiled-legacy
+kms_addfb_basic@addfb25-framebuffer-vs-set-tiling
+kms_addfb_basic@basic-x-tiled-legacy
+kms_addfb_basic@framebuffer-vs-set-tiling
+kms_addfb_basic@tile-pitch-mismatch
+kms_addfb_basic@basic-y-tiled-legacy
+kms_addfb_basic@size-max
+kms_addfb_basic@too-wide
+kms_addfb_basic@too-high
+kms_addfb_basic@bo-too-small
+kms_addfb_basic@small-bo
+kms_addfb_basic@bo-too-small-due-to-tiling
+kms_addfb_basic@addfb25-y-tiled-legacy
+kms_addfb_basic@addfb25-yf-tiled-legacy
+kms_addfb_basic@addfb25-y-tiled-small-legacy
+kms_addfb_basic@addfb25-4-tiled
+kms_async_flips@async-flip-with-page-flip-events
+kms_async_flips@alternate-sync-async-flip
+kms_async_flips@test-time-stamp
+kms_async_flips@test-cursor
+kms_async_flips@invalid-async-flip
+kms_async_flips@crc
+kms_atomic@plane-overlay-legacy
+kms_atomic@plane-primary-legacy
+kms_atomic@plane-primary-overlay-mutable-zpos
+kms_atomic@plane-immutable-zpos
+kms_atomic@test-only
+kms_atomic@plane-cursor-legacy
+kms_atomic@plane-invalid-params
+kms_atomic@plane-invalid-params-fence
+kms_atomic@crtc-invalid-params
+kms_atomic@crtc-invalid-params-fence
+kms_atomic@atomic-invalid-params
+kms_atomic@atomic_plane_damage
+kms_atomic_interruptible@legacy-setmode
+kms_atomic_interruptible@atomic-setmode
+kms_atomic_interruptible@legacy-dpms
+kms_atomic_interruptible@legacy-pageflip
+kms_atomic_interruptible@legacy-cursor
+kms_atomic_interruptible@universal-setplane-primary
+kms_atomic_interruptible@universal-setplane-cursor
+kms_atomic_transition@plane-primary-toggle-with-vblank-wait
+kms_atomic_transition@plane-all-transition
+kms_atomic_transition@plane-all-transition-fencing
+kms_atomic_transition@plane-all-transition-nonblocking
+kms_atomic_transition@plane-all-transition-nonblocking-fencing
+kms_atomic_transition@plane-use-after-nonblocking-unbind
+kms_atomic_transition@plane-use-after-nonblocking-unbind-fencing
+kms_atomic_transition@plane-all-modeset-transition
+kms_atomic_transition@plane-all-modeset-transition-fencing
+kms_atomic_transition@plane-all-modeset-transition-internal-panels
+kms_atomic_transition@plane-all-modeset-transition-fencing-internal-panels
+kms_atomic_transition@plane-toggle-modeset-transition
+kms_atomic_transition@modeset-transition
+kms_atomic_transition@modeset-transition-fencing
+kms_atomic_transition@modeset-transition-nonblocking
+kms_atomic_transition@modeset-transition-nonblocking-fencing
+kms_big_fb@x-tiled-addfb-size-overflow
+kms_big_fb@y-tiled-addfb-size-overflow
+kms_big_fb@yf-tiled-addfb-size-overflow
+kms_big_fb@4-tiled-addfb-size-overflow
+kms_big_fb@x-tiled-addfb-size-offset-overflow
+kms_big_fb@y-tiled-addfb-size-offset-overflow
+kms_big_fb@yf-tiled-addfb-size-offset-overflow
+kms_big_fb@4-tiled-addfb-size-offset-overflow
+kms_big_fb@linear-addfb
+kms_big_fb@x-tiled-addfb
+kms_big_fb@y-tiled-addfb
+kms_big_fb@yf-tiled-addfb
+kms_big_fb@4-tiled-addfb
+kms_big_fb@linear-8bpp-rotate-0
+kms_big_fb@linear-8bpp-rotate-90
+kms_big_fb@linear-8bpp-rotate-180
+kms_big_fb@linear-8bpp-rotate-270
+kms_big_fb@linear-16bpp-rotate-0
+kms_big_fb@linear-16bpp-rotate-90
+kms_big_fb@linear-16bpp-rotate-180
+kms_big_fb@linear-16bpp-rotate-270
+kms_big_fb@linear-32bpp-rotate-0
+kms_big_fb@linear-32bpp-rotate-90
+kms_big_fb@linear-32bpp-rotate-180
+kms_big_fb@linear-32bpp-rotate-270
+kms_big_fb@linear-64bpp-rotate-0
+kms_big_fb@linear-64bpp-rotate-90
+kms_big_fb@linear-64bpp-rotate-180
+kms_big_fb@linear-64bpp-rotate-270
+kms_big_fb@x-tiled-8bpp-rotate-0
+kms_big_fb@x-tiled-8bpp-rotate-90
+kms_big_fb@x-tiled-8bpp-rotate-180
+kms_big_fb@x-tiled-8bpp-rotate-270
+kms_big_fb@x-tiled-16bpp-rotate-0
+kms_big_fb@x-tiled-16bpp-rotate-90
+kms_big_fb@x-tiled-16bpp-rotate-180
+kms_big_fb@x-tiled-16bpp-rotate-270
+kms_big_fb@x-tiled-32bpp-rotate-0
+kms_big_fb@x-tiled-32bpp-rotate-90
+kms_big_fb@x-tiled-32bpp-rotate-180
+kms_big_fb@x-tiled-32bpp-rotate-270
+kms_big_fb@x-tiled-64bpp-rotate-0
+kms_big_fb@x-tiled-64bpp-rotate-90
+kms_big_fb@x-tiled-64bpp-rotate-180
+kms_big_fb@x-tiled-64bpp-rotate-270
+kms_big_fb@y-tiled-8bpp-rotate-0
+kms_big_fb@y-tiled-8bpp-rotate-90
+kms_big_fb@y-tiled-8bpp-rotate-180
+kms_big_fb@y-tiled-8bpp-rotate-270
+kms_big_fb@y-tiled-16bpp-rotate-0
+kms_big_fb@y-tiled-16bpp-rotate-90
+kms_big_fb@y-tiled-16bpp-rotate-180
+kms_big_fb@y-tiled-16bpp-rotate-270
+kms_big_fb@y-tiled-32bpp-rotate-0
+kms_big_fb@y-tiled-32bpp-rotate-90
+kms_big_fb@y-tiled-32bpp-rotate-180
+kms_big_fb@y-tiled-32bpp-rotate-270
+kms_big_fb@y-tiled-64bpp-rotate-0
+kms_big_fb@y-tiled-64bpp-rotate-90
+kms_big_fb@y-tiled-64bpp-rotate-180
+kms_big_fb@y-tiled-64bpp-rotate-270
+kms_big_fb@yf-tiled-8bpp-rotate-0
+kms_big_fb@yf-tiled-8bpp-rotate-90
+kms_big_fb@yf-tiled-8bpp-rotate-180
+kms_big_fb@yf-tiled-8bpp-rotate-270
+kms_big_fb@yf-tiled-16bpp-rotate-0
+kms_big_fb@yf-tiled-16bpp-rotate-90
+kms_big_fb@yf-tiled-16bpp-rotate-180
+kms_big_fb@yf-tiled-16bpp-rotate-270
+kms_big_fb@yf-tiled-32bpp-rotate-0
+kms_big_fb@yf-tiled-32bpp-rotate-90
+kms_big_fb@yf-tiled-32bpp-rotate-180
+kms_big_fb@yf-tiled-32bpp-rotate-270
+kms_big_fb@yf-tiled-64bpp-rotate-0
+kms_big_fb@yf-tiled-64bpp-rotate-90
+kms_big_fb@yf-tiled-64bpp-rotate-180
+kms_big_fb@yf-tiled-64bpp-rotate-270
+kms_big_fb@4-tiled-8bpp-rotate-0
+kms_big_fb@4-tiled-8bpp-rotate-90
+kms_big_fb@4-tiled-8bpp-rotate-180
+kms_big_fb@4-tiled-8bpp-rotate-270
+kms_big_fb@4-tiled-16bpp-rotate-0
+kms_big_fb@4-tiled-16bpp-rotate-90
+kms_big_fb@4-tiled-16bpp-rotate-180
+kms_big_fb@4-tiled-16bpp-rotate-270
+kms_big_fb@4-tiled-32bpp-rotate-0
+kms_big_fb@4-tiled-32bpp-rotate-90
+kms_big_fb@4-tiled-32bpp-rotate-180
+kms_big_fb@4-tiled-32bpp-rotate-270
+kms_big_fb@4-tiled-64bpp-rotate-0
+kms_big_fb@4-tiled-64bpp-rotate-90
+kms_big_fb@4-tiled-64bpp-rotate-180
+kms_big_fb@4-tiled-64bpp-rotate-270
+kms_big_fb@linear-max-hw-stride-32bpp-rotate-0
+kms_big_fb@linear-max-hw-stride-32bpp-rotate-180
+kms_big_fb@linear-max-hw-stride-64bpp-rotate-0
+kms_big_fb@linear-max-hw-stride-64bpp-rotate-180
+kms_big_fb@x-tiled-max-hw-stride-32bpp-rotate-0
+kms_big_fb@x-tiled-max-hw-stride-32bpp-rotate-0-async-flip
+kms_big_fb@x-tiled-max-hw-stride-32bpp-rotate-180
+kms_big_fb@x-tiled-max-hw-stride-32bpp-rotate-180-async-flip
+kms_big_fb@x-tiled-max-hw-stride-64bpp-rotate-0
+kms_big_fb@x-tiled-max-hw-stride-64bpp-rotate-0-async-flip
+kms_big_fb@x-tiled-max-hw-stride-64bpp-rotate-180
+kms_big_fb@x-tiled-max-hw-stride-64bpp-rotate-180-async-flip
+kms_big_fb@x-tiled-max-hw-stride-32bpp-rotate-0-hflip
+kms_big_fb@x-tiled-max-hw-stride-32bpp-rotate-0-hflip-async-flip
+kms_big_fb@x-tiled-max-hw-stride-32bpp-rotate-180-hflip
+kms_big_fb@x-tiled-max-hw-stride-32bpp-rotate-180-hflip-async-flip
+kms_big_fb@x-tiled-max-hw-stride-64bpp-rotate-0-hflip
+kms_big_fb@x-tiled-max-hw-stride-64bpp-rotate-0-hflip-async-flip
+kms_big_fb@x-tiled-max-hw-stride-64bpp-rotate-180-hflip
+kms_big_fb@x-tiled-max-hw-stride-64bpp-rotate-180-hflip-async-flip
+kms_big_fb@y-tiled-max-hw-stride-32bpp-rotate-0
+kms_big_fb@y-tiled-max-hw-stride-32bpp-rotate-0-async-flip
+kms_big_fb@y-tiled-max-hw-stride-32bpp-rotate-180
+kms_big_fb@y-tiled-max-hw-stride-32bpp-rotate-180-async-flip
+kms_big_fb@y-tiled-max-hw-stride-64bpp-rotate-0
+kms_big_fb@y-tiled-max-hw-stride-64bpp-rotate-0-async-flip
+kms_big_fb@y-tiled-max-hw-stride-64bpp-rotate-180
+kms_big_fb@y-tiled-max-hw-stride-64bpp-rotate-180-async-flip
+kms_big_fb@y-tiled-max-hw-stride-32bpp-rotate-0-hflip
+kms_big_fb@y-tiled-max-hw-stride-32bpp-rotate-0-hflip-async-flip
+kms_big_fb@y-tiled-max-hw-stride-32bpp-rotate-180-hflip
+kms_big_fb@y-tiled-max-hw-stride-32bpp-rotate-180-hflip-async-flip
+kms_big_fb@y-tiled-max-hw-stride-64bpp-rotate-0-hflip
+kms_big_fb@y-tiled-max-hw-stride-64bpp-rotate-0-hflip-async-flip
+kms_big_fb@y-tiled-max-hw-stride-64bpp-rotate-180-hflip
+kms_big_fb@y-tiled-max-hw-stride-64bpp-rotate-180-hflip-async-flip
+kms_big_fb@yf-tiled-max-hw-stride-32bpp-rotate-0
+kms_big_fb@yf-tiled-max-hw-stride-32bpp-rotate-0-async-flip
+kms_big_fb@yf-tiled-max-hw-stride-32bpp-rotate-180
+kms_big_fb@yf-tiled-max-hw-stride-32bpp-rotate-180-async-flip
+kms_big_fb@yf-tiled-max-hw-stride-64bpp-rotate-0
+kms_big_fb@yf-tiled-max-hw-stride-64bpp-rotate-0-async-flip
+kms_big_fb@yf-tiled-max-hw-stride-64bpp-rotate-180
+kms_big_fb@yf-tiled-max-hw-stride-64bpp-rotate-180-async-flip
+kms_big_fb@yf-tiled-max-hw-stride-32bpp-rotate-0-hflip
+kms_big_fb@yf-tiled-max-hw-stride-32bpp-rotate-0-hflip-async-flip
+kms_big_fb@yf-tiled-max-hw-stride-32bpp-rotate-180-hflip
+kms_big_fb@yf-tiled-max-hw-stride-32bpp-rotate-180-hflip-async-flip
+kms_big_fb@yf-tiled-max-hw-stride-64bpp-rotate-0-hflip
+kms_big_fb@yf-tiled-max-hw-stride-64bpp-rotate-0-hflip-async-flip
+kms_big_fb@yf-tiled-max-hw-stride-64bpp-rotate-180-hflip
+kms_big_fb@yf-tiled-max-hw-stride-64bpp-rotate-180-hflip-async-flip
+kms_big_fb@4-tiled-max-hw-stride-32bpp-rotate-0
+kms_big_fb@4-tiled-max-hw-stride-32bpp-rotate-0-async-flip
+kms_big_fb@4-tiled-max-hw-stride-32bpp-rotate-180
+kms_big_fb@4-tiled-max-hw-stride-32bpp-rotate-180-async-flip
+kms_big_fb@4-tiled-max-hw-stride-64bpp-rotate-0
+kms_big_fb@4-tiled-max-hw-stride-64bpp-rotate-0-async-flip
+kms_big_fb@4-tiled-max-hw-stride-64bpp-rotate-180
+kms_big_fb@4-tiled-max-hw-stride-64bpp-rotate-180-async-flip
+kms_big_fb@4-tiled-max-hw-stride-32bpp-rotate-0-hflip
+kms_big_fb@4-tiled-max-hw-stride-32bpp-rotate-0-hflip-async-flip
+kms_big_fb@4-tiled-max-hw-stride-32bpp-rotate-180-hflip
+kms_big_fb@4-tiled-max-hw-stride-32bpp-rotate-180-hflip-async-flip
+kms_big_fb@4-tiled-max-hw-stride-64bpp-rotate-0-hflip
+kms_big_fb@4-tiled-max-hw-stride-64bpp-rotate-0-hflip-async-flip
+kms_big_fb@4-tiled-max-hw-stride-64bpp-rotate-180-hflip
+kms_big_fb@4-tiled-max-hw-stride-64bpp-rotate-180-hflip-async-flip
+kms_big_joiner@basic
+kms_big_joiner@invalid-modeset
+kms_big_joiner@2x-modeset
+kms_busy@basic
+kms_busy@basic-hang
+kms_busy@extended-pageflip-modeset-hang-oldfb
+kms_busy@extended-pageflip-hang-oldfb
+kms_busy@extended-pageflip-hang-newfb
+kms_busy@extended-modeset-hang-oldfb
+kms_busy@extended-modeset-hang-newfb
+kms_busy@extended-modeset-hang-oldfb-with-reset
+kms_busy@extended-modeset-hang-newfb-with-reset
+kms_bw@linear-tiling-1-displays-1920x1080p
+kms_bw@linear-tiling-1-displays-2560x1440p
+kms_bw@linear-tiling-1-displays-3840x2160p
+kms_bw@linear-tiling-2-displays-1920x1080p
+kms_bw@linear-tiling-2-displays-2560x1440p
+kms_bw@linear-tiling-2-displays-3840x2160p
+kms_bw@linear-tiling-3-displays-1920x1080p
+kms_bw@linear-tiling-3-displays-2560x1440p
+kms_bw@linear-tiling-3-displays-3840x2160p
+kms_bw@linear-tiling-4-displays-1920x1080p
+kms_bw@linear-tiling-4-displays-2560x1440p
+kms_bw@linear-tiling-4-displays-3840x2160p
+kms_bw@linear-tiling-5-displays-1920x1080p
+kms_bw@linear-tiling-5-displays-2560x1440p
+kms_bw@linear-tiling-5-displays-3840x2160p
+kms_bw@linear-tiling-6-displays-1920x1080p
+kms_bw@linear-tiling-6-displays-2560x1440p
+kms_bw@linear-tiling-6-displays-3840x2160p
+kms_bw@linear-tiling-7-displays-1920x1080p
+kms_bw@linear-tiling-7-displays-2560x1440p
+kms_bw@linear-tiling-7-displays-3840x2160p
+kms_bw@linear-tiling-8-displays-1920x1080p
+kms_bw@linear-tiling-8-displays-2560x1440p
+kms_bw@linear-tiling-8-displays-3840x2160p
+kms_ccs@pipe-A-bad-pixel-format-y_tiled_ccs
+kms_ccs@pipe-A-bad-pixel-format-yf_tiled_ccs
+kms_ccs@pipe-A-bad-pixel-format-y_tiled_gen12_rc_ccs
+kms_ccs@pipe-A-bad-pixel-format-y_tiled_gen12_rc_ccs_cc
+kms_ccs@pipe-A-bad-pixel-format-y_tiled_gen12_mc_ccs
+kms_ccs@pipe-A-bad-pixel-format-4_tiled_dg2_rc_ccs
+kms_ccs@pipe-A-bad-pixel-format-4_tiled_dg2_mc_ccs
+kms_ccs@pipe-A-bad-pixel-format-4_tiled_dg2_rc_ccs_cc
+kms_ccs@pipe-A-bad-pixel-format-4_tiled_mtl_rc_ccs
+kms_ccs@pipe-A-bad-pixel-format-4_tiled_mtl_mc_ccs
+kms_ccs@pipe-A-bad-pixel-format-4_tiled_mtl_rc_ccs_cc
+kms_ccs@pipe-A-bad-rotation-90-y_tiled_ccs
+kms_ccs@pipe-A-bad-rotation-90-yf_tiled_ccs
+kms_ccs@pipe-A-bad-rotation-90-y_tiled_gen12_rc_ccs
+kms_ccs@pipe-A-bad-rotation-90-y_tiled_gen12_rc_ccs_cc
+kms_ccs@pipe-A-bad-rotation-90-y_tiled_gen12_mc_ccs
+kms_ccs@pipe-A-bad-rotation-90-4_tiled_dg2_rc_ccs
+kms_ccs@pipe-A-bad-rotation-90-4_tiled_dg2_mc_ccs
+kms_ccs@pipe-A-bad-rotation-90-4_tiled_dg2_rc_ccs_cc
+kms_ccs@pipe-A-bad-rotation-90-4_tiled_mtl_rc_ccs
+kms_ccs@pipe-A-bad-rotation-90-4_tiled_mtl_mc_ccs
+kms_ccs@pipe-A-bad-rotation-90-4_tiled_mtl_rc_ccs_cc
+kms_ccs@pipe-A-crc-primary-basic-y_tiled_ccs
+kms_ccs@pipe-A-crc-primary-basic-yf_tiled_ccs
+kms_ccs@pipe-A-crc-primary-basic-y_tiled_gen12_rc_ccs
+kms_ccs@pipe-A-crc-primary-basic-y_tiled_gen12_rc_ccs_cc
+kms_ccs@pipe-A-crc-primary-basic-y_tiled_gen12_mc_ccs
+kms_ccs@pipe-A-crc-primary-basic-4_tiled_dg2_rc_ccs
+kms_ccs@pipe-A-crc-primary-basic-4_tiled_dg2_mc_ccs
+kms_ccs@pipe-A-crc-primary-basic-4_tiled_dg2_rc_ccs_cc
+kms_ccs@pipe-A-crc-primary-basic-4_tiled_mtl_rc_ccs
+kms_ccs@pipe-A-crc-primary-basic-4_tiled_mtl_mc_ccs
+kms_ccs@pipe-A-crc-primary-basic-4_tiled_mtl_rc_ccs_cc
+kms_ccs@pipe-A-crc-primary-rotation-180-y_tiled_ccs
+kms_ccs@pipe-A-crc-primary-rotation-180-yf_tiled_ccs
+kms_ccs@pipe-A-crc-primary-rotation-180-y_tiled_gen12_rc_ccs
+kms_ccs@pipe-A-crc-primary-rotation-180-y_tiled_gen12_rc_ccs_cc
+kms_ccs@pipe-A-crc-primary-rotation-180-y_tiled_gen12_mc_ccs
+kms_ccs@pipe-A-crc-primary-rotation-180-4_tiled_dg2_rc_ccs
+kms_ccs@pipe-A-crc-primary-rotation-180-4_tiled_dg2_mc_ccs
+kms_ccs@pipe-A-crc-primary-rotation-180-4_tiled_dg2_rc_ccs_cc
+kms_ccs@pipe-A-crc-primary-rotation-180-4_tiled_mtl_rc_ccs
+kms_ccs@pipe-A-crc-primary-rotation-180-4_tiled_mtl_mc_ccs
+kms_ccs@pipe-A-crc-primary-rotation-180-4_tiled_mtl_rc_ccs_cc
+kms_ccs@pipe-A-random-ccs-data-y_tiled_ccs
+kms_ccs@pipe-A-random-ccs-data-yf_tiled_ccs
+kms_ccs@pipe-A-random-ccs-data-y_tiled_gen12_rc_ccs
+kms_ccs@pipe-A-random-ccs-data-y_tiled_gen12_rc_ccs_cc
+kms_ccs@pipe-A-random-ccs-data-y_tiled_gen12_mc_ccs
+kms_ccs@pipe-A-random-ccs-data-4_tiled_dg2_rc_ccs
+kms_ccs@pipe-A-random-ccs-data-4_tiled_dg2_mc_ccs
+kms_ccs@pipe-A-random-ccs-data-4_tiled_dg2_rc_ccs_cc
+kms_ccs@pipe-A-random-ccs-data-4_tiled_mtl_rc_ccs
+kms_ccs@pipe-A-random-ccs-data-4_tiled_mtl_mc_ccs
+kms_ccs@pipe-A-random-ccs-data-4_tiled_mtl_rc_ccs_cc
+kms_ccs@pipe-A-missing-ccs-buffer-y_tiled_ccs
+kms_ccs@pipe-A-missing-ccs-buffer-yf_tiled_ccs
+kms_ccs@pipe-A-missing-ccs-buffer-y_tiled_gen12_rc_ccs
+kms_ccs@pipe-A-missing-ccs-buffer-y_tiled_gen12_rc_ccs_cc
+kms_ccs@pipe-A-missing-ccs-buffer-y_tiled_gen12_mc_ccs
+kms_ccs@pipe-A-missing-ccs-buffer-4_tiled_mtl_rc_ccs
+kms_ccs@pipe-A-missing-ccs-buffer-4_tiled_mtl_mc_ccs
+kms_ccs@pipe-A-missing-ccs-buffer-4_tiled_mtl_rc_ccs_cc
+kms_ccs@pipe-A-ccs-on-another-bo-y_tiled_ccs
+kms_ccs@pipe-A-ccs-on-another-bo-yf_tiled_ccs
+kms_ccs@pipe-A-ccs-on-another-bo-y_tiled_gen12_rc_ccs
+kms_ccs@pipe-A-ccs-on-another-bo-y_tiled_gen12_rc_ccs_cc
+kms_ccs@pipe-A-ccs-on-another-bo-y_tiled_gen12_mc_ccs
+kms_ccs@pipe-A-ccs-on-another-bo-4_tiled_mtl_rc_ccs
+kms_ccs@pipe-A-ccs-on-another-bo-4_tiled_mtl_mc_ccs
+kms_ccs@pipe-A-ccs-on-another-bo-4_tiled_mtl_rc_ccs_cc
+kms_ccs@pipe-A-bad-aux-stride-y_tiled_ccs
+kms_ccs@pipe-A-bad-aux-stride-yf_tiled_ccs
+kms_ccs@pipe-A-bad-aux-stride-y_tiled_gen12_rc_ccs
+kms_ccs@pipe-A-bad-aux-stride-y_tiled_gen12_rc_ccs_cc
+kms_ccs@pipe-A-bad-aux-stride-y_tiled_gen12_mc_ccs
+kms_ccs@pipe-A-bad-aux-stride-4_tiled_mtl_rc_ccs
+kms_ccs@pipe-A-bad-aux-stride-4_tiled_mtl_mc_ccs
+kms_ccs@pipe-A-bad-aux-stride-4_tiled_mtl_rc_ccs_cc
+kms_ccs@pipe-A-crc-sprite-planes-basic-y_tiled_ccs
+kms_ccs@pipe-A-crc-sprite-planes-basic-yf_tiled_ccs
+kms_ccs@pipe-A-crc-sprite-planes-basic-y_tiled_gen12_rc_ccs
+kms_ccs@pipe-A-crc-sprite-planes-basic-y_tiled_gen12_rc_ccs_cc
+kms_ccs@pipe-A-crc-sprite-planes-basic-y_tiled_gen12_mc_ccs
+kms_ccs@pipe-A-crc-sprite-planes-basic-4_tiled_dg2_rc_ccs
+kms_ccs@pipe-A-crc-sprite-planes-basic-4_tiled_dg2_mc_ccs
+kms_ccs@pipe-A-crc-sprite-planes-basic-4_tiled_dg2_rc_ccs_cc
+kms_ccs@pipe-A-crc-sprite-planes-basic-4_tiled_mtl_rc_ccs
+kms_ccs@pipe-A-crc-sprite-planes-basic-4_tiled_mtl_mc_ccs
+kms_ccs@pipe-A-crc-sprite-planes-basic-4_tiled_mtl_rc_ccs_cc
+kms_ccs@pipe-B-bad-pixel-format-y_tiled_ccs
+kms_ccs@pipe-B-bad-pixel-format-yf_tiled_ccs
+kms_ccs@pipe-B-bad-pixel-format-y_tiled_gen12_rc_ccs
+kms_ccs@pipe-B-bad-pixel-format-y_tiled_gen12_rc_ccs_cc
+kms_ccs@pipe-B-bad-pixel-format-y_tiled_gen12_mc_ccs
+kms_ccs@pipe-B-bad-pixel-format-4_tiled_dg2_rc_ccs
+kms_ccs@pipe-B-bad-pixel-format-4_tiled_dg2_mc_ccs
+kms_ccs@pipe-B-bad-pixel-format-4_tiled_dg2_rc_ccs_cc
+kms_ccs@pipe-B-bad-pixel-format-4_tiled_mtl_rc_ccs
+kms_ccs@pipe-B-bad-pixel-format-4_tiled_mtl_mc_ccs
+kms_ccs@pipe-B-bad-pixel-format-4_tiled_mtl_rc_ccs_cc
+kms_ccs@pipe-B-bad-rotation-90-y_tiled_ccs
+kms_ccs@pipe-B-bad-rotation-90-yf_tiled_ccs
+kms_ccs@pipe-B-bad-rotation-90-y_tiled_gen12_rc_ccs
+kms_ccs@pipe-B-bad-rotation-90-y_tiled_gen12_rc_ccs_cc
+kms_ccs@pipe-B-bad-rotation-90-y_tiled_gen12_mc_ccs
+kms_ccs@pipe-B-bad-rotation-90-4_tiled_dg2_rc_ccs
+kms_ccs@pipe-B-bad-rotation-90-4_tiled_dg2_mc_ccs
+kms_ccs@pipe-B-bad-rotation-90-4_tiled_dg2_rc_ccs_cc
+kms_ccs@pipe-B-bad-rotation-90-4_tiled_mtl_rc_ccs
+kms_ccs@pipe-B-bad-rotation-90-4_tiled_mtl_mc_ccs
+kms_ccs@pipe-B-bad-rotation-90-4_tiled_mtl_rc_ccs_cc
+kms_ccs@pipe-B-crc-primary-basic-y_tiled_ccs
+kms_ccs@pipe-B-crc-primary-basic-yf_tiled_ccs
+kms_ccs@pipe-B-crc-primary-basic-y_tiled_gen12_rc_ccs
+kms_ccs@pipe-B-crc-primary-basic-y_tiled_gen12_rc_ccs_cc
+kms_ccs@pipe-B-crc-primary-basic-y_tiled_gen12_mc_ccs
+kms_ccs@pipe-B-crc-primary-basic-4_tiled_dg2_rc_ccs
+kms_ccs@pipe-B-crc-primary-basic-4_tiled_dg2_mc_ccs
+kms_ccs@pipe-B-crc-primary-basic-4_tiled_dg2_rc_ccs_cc
+kms_ccs@pipe-B-crc-primary-basic-4_tiled_mtl_rc_ccs
+kms_ccs@pipe-B-crc-primary-basic-4_tiled_mtl_mc_ccs
+kms_ccs@pipe-B-crc-primary-basic-4_tiled_mtl_rc_ccs_cc
+kms_ccs@pipe-B-crc-primary-rotation-180-y_tiled_ccs
+kms_ccs@pipe-B-crc-primary-rotation-180-yf_tiled_ccs
+kms_ccs@pipe-B-crc-primary-rotation-180-y_tiled_gen12_rc_ccs
+kms_ccs@pipe-B-crc-primary-rotation-180-y_tiled_gen12_rc_ccs_cc
+kms_ccs@pipe-B-crc-primary-rotation-180-y_tiled_gen12_mc_ccs
+kms_ccs@pipe-B-crc-primary-rotation-180-4_tiled_dg2_rc_ccs
+kms_ccs@pipe-B-crc-primary-rotation-180-4_tiled_dg2_mc_ccs
+kms_ccs@pipe-B-crc-primary-rotation-180-4_tiled_dg2_rc_ccs_cc
+kms_ccs@pipe-B-crc-primary-rotation-180-4_tiled_mtl_rc_ccs
+kms_ccs@pipe-B-crc-primary-rotation-180-4_tiled_mtl_mc_ccs
+kms_ccs@pipe-B-crc-primary-rotation-180-4_tiled_mtl_rc_ccs_cc
+kms_ccs@pipe-B-random-ccs-data-y_tiled_ccs
+kms_ccs@pipe-B-random-ccs-data-yf_tiled_ccs
+kms_ccs@pipe-B-random-ccs-data-y_tiled_gen12_rc_ccs
+kms_ccs@pipe-B-random-ccs-data-y_tiled_gen12_rc_ccs_cc
+kms_ccs@pipe-B-random-ccs-data-y_tiled_gen12_mc_ccs
+kms_ccs@pipe-B-random-ccs-data-4_tiled_dg2_rc_ccs
+kms_ccs@pipe-B-random-ccs-data-4_tiled_dg2_mc_ccs
+kms_ccs@pipe-B-random-ccs-data-4_tiled_dg2_rc_ccs_cc
+kms_ccs@pipe-B-random-ccs-data-4_tiled_mtl_rc_ccs
+kms_ccs@pipe-B-random-ccs-data-4_tiled_mtl_mc_ccs
+kms_ccs@pipe-B-random-ccs-data-4_tiled_mtl_rc_ccs_cc
+kms_ccs@pipe-B-missing-ccs-buffer-y_tiled_ccs
+kms_ccs@pipe-B-missing-ccs-buffer-yf_tiled_ccs
+kms_ccs@pipe-B-missing-ccs-buffer-y_tiled_gen12_rc_ccs
+kms_ccs@pipe-B-missing-ccs-buffer-y_tiled_gen12_rc_ccs_cc
+kms_ccs@pipe-B-missing-ccs-buffer-y_tiled_gen12_mc_ccs
+kms_ccs@pipe-B-missing-ccs-buffer-4_tiled_mtl_rc_ccs
+kms_ccs@pipe-B-missing-ccs-buffer-4_tiled_mtl_mc_ccs
+kms_ccs@pipe-B-missing-ccs-buffer-4_tiled_mtl_rc_ccs_cc
+kms_ccs@pipe-B-ccs-on-another-bo-y_tiled_ccs
+kms_ccs@pipe-B-ccs-on-another-bo-yf_tiled_ccs
+kms_ccs@pipe-B-ccs-on-another-bo-y_tiled_gen12_rc_ccs
+kms_ccs@pipe-B-ccs-on-another-bo-y_tiled_gen12_rc_ccs_cc
+kms_ccs@pipe-B-ccs-on-another-bo-y_tiled_gen12_mc_ccs
+kms_ccs@pipe-B-ccs-on-another-bo-4_tiled_mtl_rc_ccs
+kms_ccs@pipe-B-ccs-on-another-bo-4_tiled_mtl_mc_ccs
+kms_ccs@pipe-B-ccs-on-another-bo-4_tiled_mtl_rc_ccs_cc
+kms_ccs@pipe-B-bad-aux-stride-y_tiled_ccs
+kms_ccs@pipe-B-bad-aux-stride-yf_tiled_ccs
+kms_ccs@pipe-B-bad-aux-stride-y_tiled_gen12_rc_ccs
+kms_ccs@pipe-B-bad-aux-stride-y_tiled_gen12_rc_ccs_cc
+kms_ccs@pipe-B-bad-aux-stride-y_tiled_gen12_mc_ccs
+kms_ccs@pipe-B-bad-aux-stride-4_tiled_mtl_rc_ccs
+kms_ccs@pipe-B-bad-aux-stride-4_tiled_mtl_mc_ccs
+kms_ccs@pipe-B-bad-aux-stride-4_tiled_mtl_rc_ccs_cc
+kms_ccs@pipe-B-crc-sprite-planes-basic-y_tiled_ccs
+kms_ccs@pipe-B-crc-sprite-planes-basic-yf_tiled_ccs
+kms_ccs@pipe-B-crc-sprite-planes-basic-y_tiled_gen12_rc_ccs
+kms_ccs@pipe-B-crc-sprite-planes-basic-y_tiled_gen12_rc_ccs_cc
+kms_ccs@pipe-B-crc-sprite-planes-basic-y_tiled_gen12_mc_ccs
+kms_ccs@pipe-B-crc-sprite-planes-basic-4_tiled_dg2_rc_ccs
+kms_ccs@pipe-B-crc-sprite-planes-basic-4_tiled_dg2_mc_ccs
+kms_ccs@pipe-B-crc-sprite-planes-basic-4_tiled_dg2_rc_ccs_cc
+kms_ccs@pipe-B-crc-sprite-planes-basic-4_tiled_mtl_rc_ccs
+kms_ccs@pipe-B-crc-sprite-planes-basic-4_tiled_mtl_mc_ccs
+kms_ccs@pipe-B-crc-sprite-planes-basic-4_tiled_mtl_rc_ccs_cc
+kms_ccs@pipe-C-bad-pixel-format-y_tiled_ccs
+kms_ccs@pipe-C-bad-pixel-format-yf_tiled_ccs
+kms_ccs@pipe-C-bad-pixel-format-y_tiled_gen12_rc_ccs
+kms_ccs@pipe-C-bad-pixel-format-y_tiled_gen12_rc_ccs_cc
+kms_ccs@pipe-C-bad-pixel-format-y_tiled_gen12_mc_ccs
+kms_ccs@pipe-C-bad-pixel-format-4_tiled_dg2_rc_ccs
+kms_ccs@pipe-C-bad-pixel-format-4_tiled_dg2_mc_ccs
+kms_ccs@pipe-C-bad-pixel-format-4_tiled_dg2_rc_ccs_cc
+kms_ccs@pipe-C-bad-pixel-format-4_tiled_mtl_rc_ccs
+kms_ccs@pipe-C-bad-pixel-format-4_tiled_mtl_mc_ccs
+kms_ccs@pipe-C-bad-pixel-format-4_tiled_mtl_rc_ccs_cc
+kms_ccs@pipe-C-bad-rotation-90-y_tiled_ccs
+kms_ccs@pipe-C-bad-rotation-90-yf_tiled_ccs
+kms_ccs@pipe-C-bad-rotation-90-y_tiled_gen12_rc_ccs
+kms_ccs@pipe-C-bad-rotation-90-y_tiled_gen12_rc_ccs_cc
+kms_ccs@pipe-C-bad-rotation-90-y_tiled_gen12_mc_ccs
+kms_ccs@pipe-C-bad-rotation-90-4_tiled_dg2_rc_ccs
+kms_ccs@pipe-C-bad-rotation-90-4_tiled_dg2_mc_ccs
+kms_ccs@pipe-C-bad-rotation-90-4_tiled_dg2_rc_ccs_cc
+kms_ccs@pipe-C-bad-rotation-90-4_tiled_mtl_rc_ccs
+kms_ccs@pipe-C-bad-rotation-90-4_tiled_mtl_mc_ccs
+kms_ccs@pipe-C-bad-rotation-90-4_tiled_mtl_rc_ccs_cc
+kms_ccs@pipe-C-crc-primary-basic-y_tiled_ccs
+kms_ccs@pipe-C-crc-primary-basic-yf_tiled_ccs
+kms_ccs@pipe-C-crc-primary-basic-y_tiled_gen12_rc_ccs
+kms_ccs@pipe-C-crc-primary-basic-y_tiled_gen12_rc_ccs_cc
+kms_ccs@pipe-C-crc-primary-basic-y_tiled_gen12_mc_ccs
+kms_ccs@pipe-C-crc-primary-basic-4_tiled_dg2_rc_ccs
+kms_ccs@pipe-C-crc-primary-basic-4_tiled_dg2_mc_ccs
+kms_ccs@pipe-C-crc-primary-basic-4_tiled_dg2_rc_ccs_cc
+kms_ccs@pipe-C-crc-primary-basic-4_tiled_mtl_rc_ccs
+kms_ccs@pipe-C-crc-primary-basic-4_tiled_mtl_mc_ccs
+kms_ccs@pipe-C-crc-primary-basic-4_tiled_mtl_rc_ccs_cc
+kms_ccs@pipe-C-crc-primary-rotation-180-y_tiled_ccs
+kms_ccs@pipe-C-crc-primary-rotation-180-yf_tiled_ccs
+kms_ccs@pipe-C-crc-primary-rotation-180-y_tiled_gen12_rc_ccs
+kms_ccs@pipe-C-crc-primary-rotation-180-y_tiled_gen12_rc_ccs_cc
+kms_ccs@pipe-C-crc-primary-rotation-180-y_tiled_gen12_mc_ccs
+kms_ccs@pipe-C-crc-primary-rotation-180-4_tiled_dg2_rc_ccs
+kms_ccs@pipe-C-crc-primary-rotation-180-4_tiled_dg2_mc_ccs
+kms_ccs@pipe-C-crc-primary-rotation-180-4_tiled_dg2_rc_ccs_cc
+kms_ccs@pipe-C-crc-primary-rotation-180-4_tiled_mtl_rc_ccs
+kms_ccs@pipe-C-crc-primary-rotation-180-4_tiled_mtl_mc_ccs
+kms_ccs@pipe-C-crc-primary-rotation-180-4_tiled_mtl_rc_ccs_cc
+kms_ccs@pipe-C-random-ccs-data-y_tiled_ccs
+kms_ccs@pipe-C-random-ccs-data-yf_tiled_ccs
+kms_ccs@pipe-C-random-ccs-data-y_tiled_gen12_rc_ccs
+kms_ccs@pipe-C-random-ccs-data-y_tiled_gen12_rc_ccs_cc
+kms_ccs@pipe-C-random-ccs-data-y_tiled_gen12_mc_ccs
+kms_ccs@pipe-C-random-ccs-data-4_tiled_dg2_rc_ccs
+kms_ccs@pipe-C-random-ccs-data-4_tiled_dg2_mc_ccs
+kms_ccs@pipe-C-random-ccs-data-4_tiled_dg2_rc_ccs_cc
+kms_ccs@pipe-C-random-ccs-data-4_tiled_mtl_rc_ccs
+kms_ccs@pipe-C-random-ccs-data-4_tiled_mtl_mc_ccs
+kms_ccs@pipe-C-random-ccs-data-4_tiled_mtl_rc_ccs_cc
+kms_ccs@pipe-C-missing-ccs-buffer-y_tiled_ccs
+kms_ccs@pipe-C-missing-ccs-buffer-yf_tiled_ccs
+kms_ccs@pipe-C-missing-ccs-buffer-y_tiled_gen12_rc_ccs
+kms_ccs@pipe-C-missing-ccs-buffer-y_tiled_gen12_rc_ccs_cc
+kms_ccs@pipe-C-missing-ccs-buffer-y_tiled_gen12_mc_ccs
+kms_ccs@pipe-C-missing-ccs-buffer-4_tiled_mtl_rc_ccs
+kms_ccs@pipe-C-missing-ccs-buffer-4_tiled_mtl_mc_ccs
+kms_ccs@pipe-C-missing-ccs-buffer-4_tiled_mtl_rc_ccs_cc
+kms_ccs@pipe-C-ccs-on-another-bo-y_tiled_ccs
+kms_ccs@pipe-C-ccs-on-another-bo-yf_tiled_ccs
+kms_ccs@pipe-C-ccs-on-another-bo-y_tiled_gen12_rc_ccs
+kms_ccs@pipe-C-ccs-on-another-bo-y_tiled_gen12_rc_ccs_cc
+kms_ccs@pipe-C-ccs-on-another-bo-y_tiled_gen12_mc_ccs
+kms_ccs@pipe-C-ccs-on-another-bo-4_tiled_mtl_rc_ccs
+kms_ccs@pipe-C-ccs-on-another-bo-4_tiled_mtl_mc_ccs
+kms_ccs@pipe-C-ccs-on-another-bo-4_tiled_mtl_rc_ccs_cc
+kms_ccs@pipe-C-bad-aux-stride-y_tiled_ccs
+kms_ccs@pipe-C-bad-aux-stride-yf_tiled_ccs
+kms_ccs@pipe-C-bad-aux-stride-y_tiled_gen12_rc_ccs
+kms_ccs@pipe-C-bad-aux-stride-y_tiled_gen12_rc_ccs_cc
+kms_ccs@pipe-C-bad-aux-stride-y_tiled_gen12_mc_ccs
+kms_ccs@pipe-C-bad-aux-stride-4_tiled_mtl_rc_ccs
+kms_ccs@pipe-C-bad-aux-stride-4_tiled_mtl_mc_ccs
+kms_ccs@pipe-C-bad-aux-stride-4_tiled_mtl_rc_ccs_cc
+kms_ccs@pipe-C-crc-sprite-planes-basic-y_tiled_ccs
+kms_ccs@pipe-C-crc-sprite-planes-basic-yf_tiled_ccs
+kms_ccs@pipe-C-crc-sprite-planes-basic-y_tiled_gen12_rc_ccs
+kms_ccs@pipe-C-crc-sprite-planes-basic-y_tiled_gen12_rc_ccs_cc
+kms_ccs@pipe-C-crc-sprite-planes-basic-y_tiled_gen12_mc_ccs
+kms_ccs@pipe-C-crc-sprite-planes-basic-4_tiled_dg2_rc_ccs
+kms_ccs@pipe-C-crc-sprite-planes-basic-4_tiled_dg2_mc_ccs
+kms_ccs@pipe-C-crc-sprite-planes-basic-4_tiled_dg2_rc_ccs_cc
+kms_ccs@pipe-C-crc-sprite-planes-basic-4_tiled_mtl_rc_ccs
+kms_ccs@pipe-C-crc-sprite-planes-basic-4_tiled_mtl_mc_ccs
+kms_ccs@pipe-C-crc-sprite-planes-basic-4_tiled_mtl_rc_ccs_cc
+kms_ccs@pipe-D-bad-pixel-format-y_tiled_ccs
+kms_ccs@pipe-D-bad-pixel-format-yf_tiled_ccs
+kms_ccs@pipe-D-bad-pixel-format-y_tiled_gen12_rc_ccs
+kms_ccs@pipe-D-bad-pixel-format-y_tiled_gen12_rc_ccs_cc
+kms_ccs@pipe-D-bad-pixel-format-y_tiled_gen12_mc_ccs
+kms_ccs@pipe-D-bad-pixel-format-4_tiled_dg2_rc_ccs
+kms_ccs@pipe-D-bad-pixel-format-4_tiled_dg2_mc_ccs
+kms_ccs@pipe-D-bad-pixel-format-4_tiled_dg2_rc_ccs_cc
+kms_ccs@pipe-D-bad-pixel-format-4_tiled_mtl_rc_ccs
+kms_ccs@pipe-D-bad-pixel-format-4_tiled_mtl_mc_ccs
+kms_ccs@pipe-D-bad-pixel-format-4_tiled_mtl_rc_ccs_cc
+kms_ccs@pipe-D-bad-rotation-90-y_tiled_ccs
+kms_ccs@pipe-D-bad-rotation-90-yf_tiled_ccs
+kms_ccs@pipe-D-bad-rotation-90-y_tiled_gen12_rc_ccs
+kms_ccs@pipe-D-bad-rotation-90-y_tiled_gen12_rc_ccs_cc
+kms_ccs@pipe-D-bad-rotation-90-y_tiled_gen12_mc_ccs
+kms_ccs@pipe-D-bad-rotation-90-4_tiled_dg2_rc_ccs
+kms_ccs@pipe-D-bad-rotation-90-4_tiled_dg2_mc_ccs
+kms_ccs@pipe-D-bad-rotation-90-4_tiled_dg2_rc_ccs_cc
+kms_ccs@pipe-D-bad-rotation-90-4_tiled_mtl_rc_ccs
+kms_ccs@pipe-D-bad-rotation-90-4_tiled_mtl_mc_ccs
+kms_ccs@pipe-D-bad-rotation-90-4_tiled_mtl_rc_ccs_cc
+kms_ccs@pipe-D-crc-primary-basic-y_tiled_ccs
+kms_ccs@pipe-D-crc-primary-basic-yf_tiled_ccs
+kms_ccs@pipe-D-crc-primary-basic-y_tiled_gen12_rc_ccs
+kms_ccs@pipe-D-crc-primary-basic-y_tiled_gen12_rc_ccs_cc
+kms_ccs@pipe-D-crc-primary-basic-y_tiled_gen12_mc_ccs
+kms_ccs@pipe-D-crc-primary-basic-4_tiled_dg2_rc_ccs
+kms_ccs@pipe-D-crc-primary-basic-4_tiled_dg2_mc_ccs
+kms_ccs@pipe-D-crc-primary-basic-4_tiled_dg2_rc_ccs_cc
+kms_ccs@pipe-D-crc-primary-basic-4_tiled_mtl_rc_ccs
+kms_ccs@pipe-D-crc-primary-basic-4_tiled_mtl_mc_ccs
+kms_ccs@pipe-D-crc-primary-basic-4_tiled_mtl_rc_ccs_cc
+kms_ccs@pipe-D-crc-primary-rotation-180-y_tiled_ccs
+kms_ccs@pipe-D-crc-primary-rotation-180-yf_tiled_ccs
+kms_ccs@pipe-D-crc-primary-rotation-180-y_tiled_gen12_rc_ccs
+kms_ccs@pipe-D-crc-primary-rotation-180-y_tiled_gen12_rc_ccs_cc
+kms_ccs@pipe-D-crc-primary-rotation-180-y_tiled_gen12_mc_ccs
+kms_ccs@pipe-D-crc-primary-rotation-180-4_tiled_dg2_rc_ccs
+kms_ccs@pipe-D-crc-primary-rotation-180-4_tiled_dg2_mc_ccs
+kms_ccs@pipe-D-crc-primary-rotation-180-4_tiled_dg2_rc_ccs_cc
+kms_ccs@pipe-D-crc-primary-rotation-180-4_tiled_mtl_rc_ccs
+kms_ccs@pipe-D-crc-primary-rotation-180-4_tiled_mtl_mc_ccs
+kms_ccs@pipe-D-crc-primary-rotation-180-4_tiled_mtl_rc_ccs_cc
+kms_ccs@pipe-D-random-ccs-data-y_tiled_ccs
+kms_ccs@pipe-D-random-ccs-data-yf_tiled_ccs
+kms_ccs@pipe-D-random-ccs-data-y_tiled_gen12_rc_ccs
+kms_ccs@pipe-D-random-ccs-data-y_tiled_gen12_rc_ccs_cc
+kms_ccs@pipe-D-random-ccs-data-y_tiled_gen12_mc_ccs
+kms_ccs@pipe-D-random-ccs-data-4_tiled_dg2_rc_ccs
+kms_ccs@pipe-D-random-ccs-data-4_tiled_dg2_mc_ccs
+kms_ccs@pipe-D-random-ccs-data-4_tiled_dg2_rc_ccs_cc
+kms_ccs@pipe-D-random-ccs-data-4_tiled_mtl_rc_ccs
+kms_ccs@pipe-D-random-ccs-data-4_tiled_mtl_mc_ccs
+kms_ccs@pipe-D-random-ccs-data-4_tiled_mtl_rc_ccs_cc
+kms_ccs@pipe-D-missing-ccs-buffer-y_tiled_ccs
+kms_ccs@pipe-D-missing-ccs-buffer-yf_tiled_ccs
+kms_ccs@pipe-D-missing-ccs-buffer-y_tiled_gen12_rc_ccs
+kms_ccs@pipe-D-missing-ccs-buffer-y_tiled_gen12_rc_ccs_cc
+kms_ccs@pipe-D-missing-ccs-buffer-y_tiled_gen12_mc_ccs
+kms_ccs@pipe-D-missing-ccs-buffer-4_tiled_mtl_rc_ccs
+kms_ccs@pipe-D-missing-ccs-buffer-4_tiled_mtl_mc_ccs
+kms_ccs@pipe-D-missing-ccs-buffer-4_tiled_mtl_rc_ccs_cc
+kms_ccs@pipe-D-ccs-on-another-bo-y_tiled_ccs
+kms_ccs@pipe-D-ccs-on-another-bo-yf_tiled_ccs
+kms_ccs@pipe-D-ccs-on-another-bo-y_tiled_gen12_rc_ccs
+kms_ccs@pipe-D-ccs-on-another-bo-y_tiled_gen12_rc_ccs_cc
+kms_ccs@pipe-D-ccs-on-another-bo-y_tiled_gen12_mc_ccs
+kms_ccs@pipe-D-ccs-on-another-bo-4_tiled_mtl_rc_ccs
+kms_ccs@pipe-D-ccs-on-another-bo-4_tiled_mtl_mc_ccs
+kms_ccs@pipe-D-ccs-on-another-bo-4_tiled_mtl_rc_ccs_cc
+kms_ccs@pipe-D-bad-aux-stride-y_tiled_ccs
+kms_ccs@pipe-D-bad-aux-stride-yf_tiled_ccs
+kms_ccs@pipe-D-bad-aux-stride-y_tiled_gen12_rc_ccs
+kms_ccs@pipe-D-bad-aux-stride-y_tiled_gen12_rc_ccs_cc
+kms_ccs@pipe-D-bad-aux-stride-y_tiled_gen12_mc_ccs
+kms_ccs@pipe-D-bad-aux-stride-4_tiled_mtl_rc_ccs
+kms_ccs@pipe-D-bad-aux-stride-4_tiled_mtl_mc_ccs
+kms_ccs@pipe-D-bad-aux-stride-4_tiled_mtl_rc_ccs_cc
+kms_ccs@pipe-D-crc-sprite-planes-basic-y_tiled_ccs
+kms_ccs@pipe-D-crc-sprite-planes-basic-yf_tiled_ccs
+kms_ccs@pipe-D-crc-sprite-planes-basic-y_tiled_gen12_rc_ccs
+kms_ccs@pipe-D-crc-sprite-planes-basic-y_tiled_gen12_rc_ccs_cc
+kms_ccs@pipe-D-crc-sprite-planes-basic-y_tiled_gen12_mc_ccs
+kms_ccs@pipe-D-crc-sprite-planes-basic-4_tiled_dg2_rc_ccs
+kms_ccs@pipe-D-crc-sprite-planes-basic-4_tiled_dg2_mc_ccs
+kms_ccs@pipe-D-crc-sprite-planes-basic-4_tiled_dg2_rc_ccs_cc
+kms_ccs@pipe-D-crc-sprite-planes-basic-4_tiled_mtl_rc_ccs
+kms_ccs@pipe-D-crc-sprite-planes-basic-4_tiled_mtl_mc_ccs
+kms_ccs@pipe-D-crc-sprite-planes-basic-4_tiled_mtl_rc_ccs_cc
+kms_ccs@pipe-E-bad-pixel-format-y_tiled_ccs
+kms_ccs@pipe-E-bad-pixel-format-yf_tiled_ccs
+kms_ccs@pipe-E-bad-pixel-format-y_tiled_gen12_rc_ccs
+kms_ccs@pipe-E-bad-pixel-format-y_tiled_gen12_rc_ccs_cc
+kms_ccs@pipe-E-bad-pixel-format-y_tiled_gen12_mc_ccs
+kms_ccs@pipe-E-bad-pixel-format-4_tiled_dg2_rc_ccs
+kms_ccs@pipe-E-bad-pixel-format-4_tiled_dg2_mc_ccs
+kms_ccs@pipe-E-bad-pixel-format-4_tiled_dg2_rc_ccs_cc
+kms_ccs@pipe-E-bad-pixel-format-4_tiled_mtl_rc_ccs
+kms_ccs@pipe-E-bad-pixel-format-4_tiled_mtl_mc_ccs
+kms_ccs@pipe-E-bad-pixel-format-4_tiled_mtl_rc_ccs_cc
+kms_ccs@pipe-E-bad-rotation-90-y_tiled_ccs
+kms_ccs@pipe-E-bad-rotation-90-yf_tiled_ccs
+kms_ccs@pipe-E-bad-rotation-90-y_tiled_gen12_rc_ccs
+kms_ccs@pipe-E-bad-rotation-90-y_tiled_gen12_rc_ccs_cc
+kms_ccs@pipe-E-bad-rotation-90-y_tiled_gen12_mc_ccs
+kms_ccs@pipe-E-bad-rotation-90-4_tiled_dg2_rc_ccs
+kms_ccs@pipe-E-bad-rotation-90-4_tiled_dg2_mc_ccs
+kms_ccs@pipe-E-bad-rotation-90-4_tiled_dg2_rc_ccs_cc
+kms_ccs@pipe-E-bad-rotation-90-4_tiled_mtl_rc_ccs
+kms_ccs@pipe-E-bad-rotation-90-4_tiled_mtl_mc_ccs
+kms_ccs@pipe-E-bad-rotation-90-4_tiled_mtl_rc_ccs_cc
+kms_ccs@pipe-E-crc-primary-basic-y_tiled_ccs
+kms_ccs@pipe-E-crc-primary-basic-yf_tiled_ccs
+kms_ccs@pipe-E-crc-primary-basic-y_tiled_gen12_rc_ccs
+kms_ccs@pipe-E-crc-primary-basic-y_tiled_gen12_rc_ccs_cc
+kms_ccs@pipe-E-crc-primary-basic-y_tiled_gen12_mc_ccs
+kms_ccs@pipe-E-crc-primary-basic-4_tiled_dg2_rc_ccs
+kms_ccs@pipe-E-crc-primary-basic-4_tiled_dg2_mc_ccs
+kms_ccs@pipe-E-crc-primary-basic-4_tiled_dg2_rc_ccs_cc
+kms_ccs@pipe-E-crc-primary-basic-4_tiled_mtl_rc_ccs
+kms_ccs@pipe-E-crc-primary-basic-4_tiled_mtl_mc_ccs
+kms_ccs@pipe-E-crc-primary-basic-4_tiled_mtl_rc_ccs_cc
+kms_ccs@pipe-E-crc-primary-rotation-180-y_tiled_ccs
+kms_ccs@pipe-E-crc-primary-rotation-180-yf_tiled_ccs
+kms_ccs@pipe-E-crc-primary-rotation-180-y_tiled_gen12_rc_ccs
+kms_ccs@pipe-E-crc-primary-rotation-180-y_tiled_gen12_rc_ccs_cc
+kms_ccs@pipe-E-crc-primary-rotation-180-y_tiled_gen12_mc_ccs
+kms_ccs@pipe-E-crc-primary-rotation-180-4_tiled_dg2_rc_ccs
+kms_ccs@pipe-E-crc-primary-rotation-180-4_tiled_dg2_mc_ccs
+kms_ccs@pipe-E-crc-primary-rotation-180-4_tiled_dg2_rc_ccs_cc
+kms_ccs@pipe-E-crc-primary-rotation-180-4_tiled_mtl_rc_ccs
+kms_ccs@pipe-E-crc-primary-rotation-180-4_tiled_mtl_mc_ccs
+kms_ccs@pipe-E-crc-primary-rotation-180-4_tiled_mtl_rc_ccs_cc
+kms_ccs@pipe-E-random-ccs-data-y_tiled_ccs
+kms_ccs@pipe-E-random-ccs-data-yf_tiled_ccs
+kms_ccs@pipe-E-random-ccs-data-y_tiled_gen12_rc_ccs
+kms_ccs@pipe-E-random-ccs-data-y_tiled_gen12_rc_ccs_cc
+kms_ccs@pipe-E-random-ccs-data-y_tiled_gen12_mc_ccs
+kms_ccs@pipe-E-random-ccs-data-4_tiled_dg2_rc_ccs
+kms_ccs@pipe-E-random-ccs-data-4_tiled_dg2_mc_ccs
+kms_ccs@pipe-E-random-ccs-data-4_tiled_dg2_rc_ccs_cc
+kms_ccs@pipe-E-random-ccs-data-4_tiled_mtl_rc_ccs
+kms_ccs@pipe-E-random-ccs-data-4_tiled_mtl_mc_ccs
+kms_ccs@pipe-E-random-ccs-data-4_tiled_mtl_rc_ccs_cc
+kms_ccs@pipe-E-missing-ccs-buffer-y_tiled_ccs
+kms_ccs@pipe-E-missing-ccs-buffer-yf_tiled_ccs
+kms_ccs@pipe-E-missing-ccs-buffer-y_tiled_gen12_rc_ccs
+kms_ccs@pipe-E-missing-ccs-buffer-y_tiled_gen12_rc_ccs_cc
+kms_ccs@pipe-E-missing-ccs-buffer-y_tiled_gen12_mc_ccs
+kms_ccs@pipe-E-missing-ccs-buffer-4_tiled_mtl_rc_ccs
+kms_ccs@pipe-E-missing-ccs-buffer-4_tiled_mtl_mc_ccs
+kms_ccs@pipe-E-missing-ccs-buffer-4_tiled_mtl_rc_ccs_cc
+kms_ccs@pipe-E-ccs-on-another-bo-y_tiled_ccs
+kms_ccs@pipe-E-ccs-on-another-bo-yf_tiled_ccs
+kms_ccs@pipe-E-ccs-on-another-bo-y_tiled_gen12_rc_ccs
+kms_ccs@pipe-E-ccs-on-another-bo-y_tiled_gen12_rc_ccs_cc
+kms_ccs@pipe-E-ccs-on-another-bo-y_tiled_gen12_mc_ccs
+kms_ccs@pipe-E-ccs-on-another-bo-4_tiled_mtl_rc_ccs
+kms_ccs@pipe-E-ccs-on-another-bo-4_tiled_mtl_mc_ccs
+kms_ccs@pipe-E-ccs-on-another-bo-4_tiled_mtl_rc_ccs_cc
+kms_ccs@pipe-E-bad-aux-stride-y_tiled_ccs
+kms_ccs@pipe-E-bad-aux-stride-yf_tiled_ccs
+kms_ccs@pipe-E-bad-aux-stride-y_tiled_gen12_rc_ccs
+kms_ccs@pipe-E-bad-aux-stride-y_tiled_gen12_rc_ccs_cc
+kms_ccs@pipe-E-bad-aux-stride-y_tiled_gen12_mc_ccs
+kms_ccs@pipe-E-bad-aux-stride-4_tiled_mtl_rc_ccs
+kms_ccs@pipe-E-bad-aux-stride-4_tiled_mtl_mc_ccs
+kms_ccs@pipe-E-bad-aux-stride-4_tiled_mtl_rc_ccs_cc
+kms_ccs@pipe-E-crc-sprite-planes-basic-y_tiled_ccs
+kms_ccs@pipe-E-crc-sprite-planes-basic-yf_tiled_ccs
+kms_ccs@pipe-E-crc-sprite-planes-basic-y_tiled_gen12_rc_ccs
+kms_ccs@pipe-E-crc-sprite-planes-basic-y_tiled_gen12_rc_ccs_cc
+kms_ccs@pipe-E-crc-sprite-planes-basic-y_tiled_gen12_mc_ccs
+kms_ccs@pipe-E-crc-sprite-planes-basic-4_tiled_dg2_rc_ccs
+kms_ccs@pipe-E-crc-sprite-planes-basic-4_tiled_dg2_mc_ccs
+kms_ccs@pipe-E-crc-sprite-planes-basic-4_tiled_dg2_rc_ccs_cc
+kms_ccs@pipe-E-crc-sprite-planes-basic-4_tiled_mtl_rc_ccs
+kms_ccs@pipe-E-crc-sprite-planes-basic-4_tiled_mtl_mc_ccs
+kms_ccs@pipe-E-crc-sprite-planes-basic-4_tiled_mtl_rc_ccs_cc
+kms_ccs@pipe-F-bad-pixel-format-y_tiled_ccs
+kms_ccs@pipe-F-bad-pixel-format-yf_tiled_ccs
+kms_ccs@pipe-F-bad-pixel-format-y_tiled_gen12_rc_ccs
+kms_ccs@pipe-F-bad-pixel-format-y_tiled_gen12_rc_ccs_cc
+kms_ccs@pipe-F-bad-pixel-format-y_tiled_gen12_mc_ccs
+kms_ccs@pipe-F-bad-pixel-format-4_tiled_dg2_rc_ccs
+kms_ccs@pipe-F-bad-pixel-format-4_tiled_dg2_mc_ccs
+kms_ccs@pipe-F-bad-pixel-format-4_tiled_dg2_rc_ccs_cc
+kms_ccs@pipe-F-bad-pixel-format-4_tiled_mtl_rc_ccs
+kms_ccs@pipe-F-bad-pixel-format-4_tiled_mtl_mc_ccs
+kms_ccs@pipe-F-bad-pixel-format-4_tiled_mtl_rc_ccs_cc
+kms_ccs@pipe-F-bad-rotation-90-y_tiled_ccs
+kms_ccs@pipe-F-bad-rotation-90-yf_tiled_ccs
+kms_ccs@pipe-F-bad-rotation-90-y_tiled_gen12_rc_ccs
+kms_ccs@pipe-F-bad-rotation-90-y_tiled_gen12_rc_ccs_cc
+kms_ccs@pipe-F-bad-rotation-90-y_tiled_gen12_mc_ccs
+kms_ccs@pipe-F-bad-rotation-90-4_tiled_dg2_rc_ccs
+kms_ccs@pipe-F-bad-rotation-90-4_tiled_dg2_mc_ccs
+kms_ccs@pipe-F-bad-rotation-90-4_tiled_dg2_rc_ccs_cc
+kms_ccs@pipe-F-bad-rotation-90-4_tiled_mtl_rc_ccs
+kms_ccs@pipe-F-bad-rotation-90-4_tiled_mtl_mc_ccs
+kms_ccs@pipe-F-bad-rotation-90-4_tiled_mtl_rc_ccs_cc
+kms_ccs@pipe-F-crc-primary-basic-y_tiled_ccs
+kms_ccs@pipe-F-crc-primary-basic-yf_tiled_ccs
+kms_ccs@pipe-F-crc-primary-basic-y_tiled_gen12_rc_ccs
+kms_ccs@pipe-F-crc-primary-basic-y_tiled_gen12_rc_ccs_cc
+kms_ccs@pipe-F-crc-primary-basic-y_tiled_gen12_mc_ccs
+kms_ccs@pipe-F-crc-primary-basic-4_tiled_dg2_rc_ccs
+kms_ccs@pipe-F-crc-primary-basic-4_tiled_dg2_mc_ccs
+kms_ccs@pipe-F-crc-primary-basic-4_tiled_dg2_rc_ccs_cc
+kms_ccs@pipe-F-crc-primary-basic-4_tiled_mtl_rc_ccs
+kms_ccs@pipe-F-crc-primary-basic-4_tiled_mtl_mc_ccs
+kms_ccs@pipe-F-crc-primary-basic-4_tiled_mtl_rc_ccs_cc
+kms_ccs@pipe-F-crc-primary-rotation-180-y_tiled_ccs
+kms_ccs@pipe-F-crc-primary-rotation-180-yf_tiled_ccs
+kms_ccs@pipe-F-crc-primary-rotation-180-y_tiled_gen12_rc_ccs
+kms_ccs@pipe-F-crc-primary-rotation-180-y_tiled_gen12_rc_ccs_cc
+kms_ccs@pipe-F-crc-primary-rotation-180-y_tiled_gen12_mc_ccs
+kms_ccs@pipe-F-crc-primary-rotation-180-4_tiled_dg2_rc_ccs
+kms_ccs@pipe-F-crc-primary-rotation-180-4_tiled_dg2_mc_ccs
+kms_ccs@pipe-F-crc-primary-rotation-180-4_tiled_dg2_rc_ccs_cc
+kms_ccs@pipe-F-crc-primary-rotation-180-4_tiled_mtl_rc_ccs
+kms_ccs@pipe-F-crc-primary-rotation-180-4_tiled_mtl_mc_ccs
+kms_ccs@pipe-F-crc-primary-rotation-180-4_tiled_mtl_rc_ccs_cc
+kms_ccs@pipe-F-random-ccs-data-y_tiled_ccs
+kms_ccs@pipe-F-random-ccs-data-yf_tiled_ccs
+kms_ccs@pipe-F-random-ccs-data-y_tiled_gen12_rc_ccs
+kms_ccs@pipe-F-random-ccs-data-y_tiled_gen12_rc_ccs_cc
+kms_ccs@pipe-F-random-ccs-data-y_tiled_gen12_mc_ccs
+kms_ccs@pipe-F-random-ccs-data-4_tiled_dg2_rc_ccs
+kms_ccs@pipe-F-random-ccs-data-4_tiled_dg2_mc_ccs
+kms_ccs@pipe-F-random-ccs-data-4_tiled_dg2_rc_ccs_cc
+kms_ccs@pipe-F-random-ccs-data-4_tiled_mtl_rc_ccs
+kms_ccs@pipe-F-random-ccs-data-4_tiled_mtl_mc_ccs
+kms_ccs@pipe-F-random-ccs-data-4_tiled_mtl_rc_ccs_cc
+kms_ccs@pipe-F-missing-ccs-buffer-y_tiled_ccs
+kms_ccs@pipe-F-missing-ccs-buffer-yf_tiled_ccs
+kms_ccs@pipe-F-missing-ccs-buffer-y_tiled_gen12_rc_ccs
+kms_ccs@pipe-F-missing-ccs-buffer-y_tiled_gen12_rc_ccs_cc
+kms_ccs@pipe-F-missing-ccs-buffer-y_tiled_gen12_mc_ccs
+kms_ccs@pipe-F-missing-ccs-buffer-4_tiled_mtl_rc_ccs
+kms_ccs@pipe-F-missing-ccs-buffer-4_tiled_mtl_mc_ccs
+kms_ccs@pipe-F-missing-ccs-buffer-4_tiled_mtl_rc_ccs_cc
+kms_ccs@pipe-F-ccs-on-another-bo-y_tiled_ccs
+kms_ccs@pipe-F-ccs-on-another-bo-yf_tiled_ccs
+kms_ccs@pipe-F-ccs-on-another-bo-y_tiled_gen12_rc_ccs
+kms_ccs@pipe-F-ccs-on-another-bo-y_tiled_gen12_rc_ccs_cc
+kms_ccs@pipe-F-ccs-on-another-bo-y_tiled_gen12_mc_ccs
+kms_ccs@pipe-F-ccs-on-another-bo-4_tiled_mtl_rc_ccs
+kms_ccs@pipe-F-ccs-on-another-bo-4_tiled_mtl_mc_ccs
+kms_ccs@pipe-F-ccs-on-another-bo-4_tiled_mtl_rc_ccs_cc
+kms_ccs@pipe-F-bad-aux-stride-y_tiled_ccs
+kms_ccs@pipe-F-bad-aux-stride-yf_tiled_ccs
+kms_ccs@pipe-F-bad-aux-stride-y_tiled_gen12_rc_ccs
+kms_ccs@pipe-F-bad-aux-stride-y_tiled_gen12_rc_ccs_cc
+kms_ccs@pipe-F-bad-aux-stride-y_tiled_gen12_mc_ccs
+kms_ccs@pipe-F-bad-aux-stride-4_tiled_mtl_rc_ccs
+kms_ccs@pipe-F-bad-aux-stride-4_tiled_mtl_mc_ccs
+kms_ccs@pipe-F-bad-aux-stride-4_tiled_mtl_rc_ccs_cc
+kms_ccs@pipe-F-crc-sprite-planes-basic-y_tiled_ccs
+kms_ccs@pipe-F-crc-sprite-planes-basic-yf_tiled_ccs
+kms_ccs@pipe-F-crc-sprite-planes-basic-y_tiled_gen12_rc_ccs
+kms_ccs@pipe-F-crc-sprite-planes-basic-y_tiled_gen12_rc_ccs_cc
+kms_ccs@pipe-F-crc-sprite-planes-basic-y_tiled_gen12_mc_ccs
+kms_ccs@pipe-F-crc-sprite-planes-basic-4_tiled_dg2_rc_ccs
+kms_ccs@pipe-F-crc-sprite-planes-basic-4_tiled_dg2_mc_ccs
+kms_ccs@pipe-F-crc-sprite-planes-basic-4_tiled_dg2_rc_ccs_cc
+kms_ccs@pipe-F-crc-sprite-planes-basic-4_tiled_mtl_rc_ccs
+kms_ccs@pipe-F-crc-sprite-planes-basic-4_tiled_mtl_mc_ccs
+kms_ccs@pipe-F-crc-sprite-planes-basic-4_tiled_mtl_rc_ccs_cc
+kms_ccs@pipe-G-bad-pixel-format-y_tiled_ccs
+kms_ccs@pipe-G-bad-pixel-format-yf_tiled_ccs
+kms_ccs@pipe-G-bad-pixel-format-y_tiled_gen12_rc_ccs
+kms_ccs@pipe-G-bad-pixel-format-y_tiled_gen12_rc_ccs_cc
+kms_ccs@pipe-G-bad-pixel-format-y_tiled_gen12_mc_ccs
+kms_ccs@pipe-G-bad-pixel-format-4_tiled_dg2_rc_ccs
+kms_ccs@pipe-G-bad-pixel-format-4_tiled_dg2_mc_ccs
+kms_ccs@pipe-G-bad-pixel-format-4_tiled_dg2_rc_ccs_cc
+kms_ccs@pipe-G-bad-pixel-format-4_tiled_mtl_rc_ccs
+kms_ccs@pipe-G-bad-pixel-format-4_tiled_mtl_mc_ccs
+kms_ccs@pipe-G-bad-pixel-format-4_tiled_mtl_rc_ccs_cc
+kms_ccs@pipe-G-bad-rotation-90-y_tiled_ccs
+kms_ccs@pipe-G-bad-rotation-90-yf_tiled_ccs
+kms_ccs@pipe-G-bad-rotation-90-y_tiled_gen12_rc_ccs
+kms_ccs@pipe-G-bad-rotation-90-y_tiled_gen12_rc_ccs_cc
+kms_ccs@pipe-G-bad-rotation-90-y_tiled_gen12_mc_ccs
+kms_ccs@pipe-G-bad-rotation-90-4_tiled_dg2_rc_ccs
+kms_ccs@pipe-G-bad-rotation-90-4_tiled_dg2_mc_ccs
+kms_ccs@pipe-G-bad-rotation-90-4_tiled_dg2_rc_ccs_cc
+kms_ccs@pipe-G-bad-rotation-90-4_tiled_mtl_rc_ccs
+kms_ccs@pipe-G-bad-rotation-90-4_tiled_mtl_mc_ccs
+kms_ccs@pipe-G-bad-rotation-90-4_tiled_mtl_rc_ccs_cc
+kms_ccs@pipe-G-crc-primary-basic-y_tiled_ccs
+kms_ccs@pipe-G-crc-primary-basic-yf_tiled_ccs
+kms_ccs@pipe-G-crc-primary-basic-y_tiled_gen12_rc_ccs
+kms_ccs@pipe-G-crc-primary-basic-y_tiled_gen12_rc_ccs_cc
+kms_ccs@pipe-G-crc-primary-basic-y_tiled_gen12_mc_ccs
+kms_ccs@pipe-G-crc-primary-basic-4_tiled_dg2_rc_ccs
+kms_ccs@pipe-G-crc-primary-basic-4_tiled_dg2_mc_ccs
+kms_ccs@pipe-G-crc-primary-basic-4_tiled_dg2_rc_ccs_cc
+kms_ccs@pipe-G-crc-primary-basic-4_tiled_mtl_rc_ccs
+kms_ccs@pipe-G-crc-primary-basic-4_tiled_mtl_mc_ccs
+kms_ccs@pipe-G-crc-primary-basic-4_tiled_mtl_rc_ccs_cc
+kms_ccs@pipe-G-crc-primary-rotation-180-y_tiled_ccs
+kms_ccs@pipe-G-crc-primary-rotation-180-yf_tiled_ccs
+kms_ccs@pipe-G-crc-primary-rotation-180-y_tiled_gen12_rc_ccs
+kms_ccs@pipe-G-crc-primary-rotation-180-y_tiled_gen12_rc_ccs_cc
+kms_ccs@pipe-G-crc-primary-rotation-180-y_tiled_gen12_mc_ccs
+kms_ccs@pipe-G-crc-primary-rotation-180-4_tiled_dg2_rc_ccs
+kms_ccs@pipe-G-crc-primary-rotation-180-4_tiled_dg2_mc_ccs
+kms_ccs@pipe-G-crc-primary-rotation-180-4_tiled_dg2_rc_ccs_cc
+kms_ccs@pipe-G-crc-primary-rotation-180-4_tiled_mtl_rc_ccs
+kms_ccs@pipe-G-crc-primary-rotation-180-4_tiled_mtl_mc_ccs
+kms_ccs@pipe-G-crc-primary-rotation-180-4_tiled_mtl_rc_ccs_cc
+kms_ccs@pipe-G-random-ccs-data-y_tiled_ccs
+kms_ccs@pipe-G-random-ccs-data-yf_tiled_ccs
+kms_ccs@pipe-G-random-ccs-data-y_tiled_gen12_rc_ccs
+kms_ccs@pipe-G-random-ccs-data-y_tiled_gen12_rc_ccs_cc
+kms_ccs@pipe-G-random-ccs-data-y_tiled_gen12_mc_ccs
+kms_ccs@pipe-G-random-ccs-data-4_tiled_dg2_rc_ccs
+kms_ccs@pipe-G-random-ccs-data-4_tiled_dg2_mc_ccs
+kms_ccs@pipe-G-random-ccs-data-4_tiled_dg2_rc_ccs_cc
+kms_ccs@pipe-G-random-ccs-data-4_tiled_mtl_rc_ccs
+kms_ccs@pipe-G-random-ccs-data-4_tiled_mtl_mc_ccs
+kms_ccs@pipe-G-random-ccs-data-4_tiled_mtl_rc_ccs_cc
+kms_ccs@pipe-G-missing-ccs-buffer-y_tiled_ccs
+kms_ccs@pipe-G-missing-ccs-buffer-yf_tiled_ccs
+kms_ccs@pipe-G-missing-ccs-buffer-y_tiled_gen12_rc_ccs
+kms_ccs@pipe-G-missing-ccs-buffer-y_tiled_gen12_rc_ccs_cc
+kms_ccs@pipe-G-missing-ccs-buffer-y_tiled_gen12_mc_ccs
+kms_ccs@pipe-G-missing-ccs-buffer-4_tiled_mtl_rc_ccs
+kms_ccs@pipe-G-missing-ccs-buffer-4_tiled_mtl_mc_ccs
+kms_ccs@pipe-G-missing-ccs-buffer-4_tiled_mtl_rc_ccs_cc
+kms_ccs@pipe-G-ccs-on-another-bo-y_tiled_ccs
+kms_ccs@pipe-G-ccs-on-another-bo-yf_tiled_ccs
+kms_ccs@pipe-G-ccs-on-another-bo-y_tiled_gen12_rc_ccs
+kms_ccs@pipe-G-ccs-on-another-bo-y_tiled_gen12_rc_ccs_cc
+kms_ccs@pipe-G-ccs-on-another-bo-y_tiled_gen12_mc_ccs
+kms_ccs@pipe-G-ccs-on-another-bo-4_tiled_mtl_rc_ccs
+kms_ccs@pipe-G-ccs-on-another-bo-4_tiled_mtl_mc_ccs
+kms_ccs@pipe-G-ccs-on-another-bo-4_tiled_mtl_rc_ccs_cc
+kms_ccs@pipe-G-bad-aux-stride-y_tiled_ccs
+kms_ccs@pipe-G-bad-aux-stride-yf_tiled_ccs
+kms_ccs@pipe-G-bad-aux-stride-y_tiled_gen12_rc_ccs
+kms_ccs@pipe-G-bad-aux-stride-y_tiled_gen12_rc_ccs_cc
+kms_ccs@pipe-G-bad-aux-stride-y_tiled_gen12_mc_ccs
+kms_ccs@pipe-G-bad-aux-stride-4_tiled_mtl_rc_ccs
+kms_ccs@pipe-G-bad-aux-stride-4_tiled_mtl_mc_ccs
+kms_ccs@pipe-G-bad-aux-stride-4_tiled_mtl_rc_ccs_cc
+kms_ccs@pipe-G-crc-sprite-planes-basic-y_tiled_ccs
+kms_ccs@pipe-G-crc-sprite-planes-basic-yf_tiled_ccs
+kms_ccs@pipe-G-crc-sprite-planes-basic-y_tiled_gen12_rc_ccs
+kms_ccs@pipe-G-crc-sprite-planes-basic-y_tiled_gen12_rc_ccs_cc
+kms_ccs@pipe-G-crc-sprite-planes-basic-y_tiled_gen12_mc_ccs
+kms_ccs@pipe-G-crc-sprite-planes-basic-4_tiled_dg2_rc_ccs
+kms_ccs@pipe-G-crc-sprite-planes-basic-4_tiled_dg2_mc_ccs
+kms_ccs@pipe-G-crc-sprite-planes-basic-4_tiled_dg2_rc_ccs_cc
+kms_ccs@pipe-G-crc-sprite-planes-basic-4_tiled_mtl_rc_ccs
+kms_ccs@pipe-G-crc-sprite-planes-basic-4_tiled_mtl_mc_ccs
+kms_ccs@pipe-G-crc-sprite-planes-basic-4_tiled_mtl_rc_ccs_cc
+kms_ccs@pipe-H-bad-pixel-format-y_tiled_ccs
+kms_ccs@pipe-H-bad-pixel-format-yf_tiled_ccs
+kms_ccs@pipe-H-bad-pixel-format-y_tiled_gen12_rc_ccs
+kms_ccs@pipe-H-bad-pixel-format-y_tiled_gen12_rc_ccs_cc
+kms_ccs@pipe-H-bad-pixel-format-y_tiled_gen12_mc_ccs
+kms_ccs@pipe-H-bad-pixel-format-4_tiled_dg2_rc_ccs
+kms_ccs@pipe-H-bad-pixel-format-4_tiled_dg2_mc_ccs
+kms_ccs@pipe-H-bad-pixel-format-4_tiled_dg2_rc_ccs_cc
+kms_ccs@pipe-H-bad-pixel-format-4_tiled_mtl_rc_ccs
+kms_ccs@pipe-H-bad-pixel-format-4_tiled_mtl_mc_ccs
+kms_ccs@pipe-H-bad-pixel-format-4_tiled_mtl_rc_ccs_cc
+kms_ccs@pipe-H-bad-rotation-90-y_tiled_ccs
+kms_ccs@pipe-H-bad-rotation-90-yf_tiled_ccs
+kms_ccs@pipe-H-bad-rotation-90-y_tiled_gen12_rc_ccs
+kms_ccs@pipe-H-bad-rotation-90-y_tiled_gen12_rc_ccs_cc
+kms_ccs@pipe-H-bad-rotation-90-y_tiled_gen12_mc_ccs
+kms_ccs@pipe-H-bad-rotation-90-4_tiled_dg2_rc_ccs
+kms_ccs@pipe-H-bad-rotation-90-4_tiled_dg2_mc_ccs
+kms_ccs@pipe-H-bad-rotation-90-4_tiled_dg2_rc_ccs_cc
+kms_ccs@pipe-H-bad-rotation-90-4_tiled_mtl_rc_ccs
+kms_ccs@pipe-H-bad-rotation-90-4_tiled_mtl_mc_ccs
+kms_ccs@pipe-H-bad-rotation-90-4_tiled_mtl_rc_ccs_cc
+kms_ccs@pipe-H-crc-primary-basic-y_tiled_ccs
+kms_ccs@pipe-H-crc-primary-basic-yf_tiled_ccs
+kms_ccs@pipe-H-crc-primary-basic-y_tiled_gen12_rc_ccs
+kms_ccs@pipe-H-crc-primary-basic-y_tiled_gen12_rc_ccs_cc
+kms_ccs@pipe-H-crc-primary-basic-y_tiled_gen12_mc_ccs
+kms_ccs@pipe-H-crc-primary-basic-4_tiled_dg2_rc_ccs
+kms_ccs@pipe-H-crc-primary-basic-4_tiled_dg2_mc_ccs
+kms_ccs@pipe-H-crc-primary-basic-4_tiled_dg2_rc_ccs_cc
+kms_ccs@pipe-H-crc-primary-basic-4_tiled_mtl_rc_ccs
+kms_ccs@pipe-H-crc-primary-basic-4_tiled_mtl_mc_ccs
+kms_ccs@pipe-H-crc-primary-basic-4_tiled_mtl_rc_ccs_cc
+kms_ccs@pipe-H-crc-primary-rotation-180-y_tiled_ccs
+kms_ccs@pipe-H-crc-primary-rotation-180-yf_tiled_ccs
+kms_ccs@pipe-H-crc-primary-rotation-180-y_tiled_gen12_rc_ccs
+kms_ccs@pipe-H-crc-primary-rotation-180-y_tiled_gen12_rc_ccs_cc
+kms_ccs@pipe-H-crc-primary-rotation-180-y_tiled_gen12_mc_ccs
+kms_ccs@pipe-H-crc-primary-rotation-180-4_tiled_dg2_rc_ccs
+kms_ccs@pipe-H-crc-primary-rotation-180-4_tiled_dg2_mc_ccs
+kms_ccs@pipe-H-crc-primary-rotation-180-4_tiled_dg2_rc_ccs_cc
+kms_ccs@pipe-H-crc-primary-rotation-180-4_tiled_mtl_rc_ccs
+kms_ccs@pipe-H-crc-primary-rotation-180-4_tiled_mtl_mc_ccs
+kms_ccs@pipe-H-crc-primary-rotation-180-4_tiled_mtl_rc_ccs_cc
+kms_ccs@pipe-H-random-ccs-data-y_tiled_ccs
+kms_ccs@pipe-H-random-ccs-data-yf_tiled_ccs
+kms_ccs@pipe-H-random-ccs-data-y_tiled_gen12_rc_ccs
+kms_ccs@pipe-H-random-ccs-data-y_tiled_gen12_rc_ccs_cc
+kms_ccs@pipe-H-random-ccs-data-y_tiled_gen12_mc_ccs
+kms_ccs@pipe-H-random-ccs-data-4_tiled_dg2_rc_ccs
+kms_ccs@pipe-H-random-ccs-data-4_tiled_dg2_mc_ccs
+kms_ccs@pipe-H-random-ccs-data-4_tiled_dg2_rc_ccs_cc
+kms_ccs@pipe-H-random-ccs-data-4_tiled_mtl_rc_ccs
+kms_ccs@pipe-H-random-ccs-data-4_tiled_mtl_mc_ccs
+kms_ccs@pipe-H-random-ccs-data-4_tiled_mtl_rc_ccs_cc
+kms_ccs@pipe-H-missing-ccs-buffer-y_tiled_ccs
+kms_ccs@pipe-H-missing-ccs-buffer-yf_tiled_ccs
+kms_ccs@pipe-H-missing-ccs-buffer-y_tiled_gen12_rc_ccs
+kms_ccs@pipe-H-missing-ccs-buffer-y_tiled_gen12_rc_ccs_cc
+kms_ccs@pipe-H-missing-ccs-buffer-y_tiled_gen12_mc_ccs
+kms_ccs@pipe-H-missing-ccs-buffer-4_tiled_mtl_rc_ccs
+kms_ccs@pipe-H-missing-ccs-buffer-4_tiled_mtl_mc_ccs
+kms_ccs@pipe-H-missing-ccs-buffer-4_tiled_mtl_rc_ccs_cc
+kms_ccs@pipe-H-ccs-on-another-bo-y_tiled_ccs
+kms_ccs@pipe-H-ccs-on-another-bo-yf_tiled_ccs
+kms_ccs@pipe-H-ccs-on-another-bo-y_tiled_gen12_rc_ccs
+kms_ccs@pipe-H-ccs-on-another-bo-y_tiled_gen12_rc_ccs_cc
+kms_ccs@pipe-H-ccs-on-another-bo-y_tiled_gen12_mc_ccs
+kms_ccs@pipe-H-ccs-on-another-bo-4_tiled_mtl_rc_ccs
+kms_ccs@pipe-H-ccs-on-another-bo-4_tiled_mtl_mc_ccs
+kms_ccs@pipe-H-ccs-on-another-bo-4_tiled_mtl_rc_ccs_cc
+kms_ccs@pipe-H-bad-aux-stride-y_tiled_ccs
+kms_ccs@pipe-H-bad-aux-stride-yf_tiled_ccs
+kms_ccs@pipe-H-bad-aux-stride-y_tiled_gen12_rc_ccs
+kms_ccs@pipe-H-bad-aux-stride-y_tiled_gen12_rc_ccs_cc
+kms_ccs@pipe-H-bad-aux-stride-y_tiled_gen12_mc_ccs
+kms_ccs@pipe-H-bad-aux-stride-4_tiled_mtl_rc_ccs
+kms_ccs@pipe-H-bad-aux-stride-4_tiled_mtl_mc_ccs
+kms_ccs@pipe-H-bad-aux-stride-4_tiled_mtl_rc_ccs_cc
+kms_ccs@pipe-H-crc-sprite-planes-basic-y_tiled_ccs
+kms_ccs@pipe-H-crc-sprite-planes-basic-yf_tiled_ccs
+kms_ccs@pipe-H-crc-sprite-planes-basic-y_tiled_gen12_rc_ccs
+kms_ccs@pipe-H-crc-sprite-planes-basic-y_tiled_gen12_rc_ccs_cc
+kms_ccs@pipe-H-crc-sprite-planes-basic-y_tiled_gen12_mc_ccs
+kms_ccs@pipe-H-crc-sprite-planes-basic-4_tiled_dg2_rc_ccs
+kms_ccs@pipe-H-crc-sprite-planes-basic-4_tiled_dg2_mc_ccs
+kms_ccs@pipe-H-crc-sprite-planes-basic-4_tiled_dg2_rc_ccs_cc
+kms_ccs@pipe-H-crc-sprite-planes-basic-4_tiled_mtl_rc_ccs
+kms_ccs@pipe-H-crc-sprite-planes-basic-4_tiled_mtl_mc_ccs
+kms_ccs@pipe-H-crc-sprite-planes-basic-4_tiled_mtl_rc_ccs_cc
+kms_cdclk@plane-scaling
+kms_cdclk@mode-transition
+kms_cdclk@mode-transition-all-outputs
+kms_color@degamma
+kms_color@gamma
+kms_color@legacy-gamma
+kms_color@legacy-gamma-reset
+kms_color@ctm-red-to-blue
+kms_color@ctm-green-to-red
+kms_color@ctm-blue-to-red
+kms_color@ctm-max
+kms_color@ctm-negative
+kms_color@ctm-0-25
+kms_color@ctm-0-50
+kms_color@ctm-0-75
+kms_color@ctm-signed
+kms_color@deep-color
+kms_color@invalid-gamma-lut-sizes
+kms_color@invalid-degamma-lut-sizes
+kms_color@invalid-ctm-matrix-sizes
+kms_concurrent@pipe-A
+kms_concurrent@pipe-B
+kms_concurrent@pipe-C
+kms_concurrent@pipe-D
+kms_concurrent@pipe-E
+kms_concurrent@pipe-F
+kms_concurrent@pipe-G
+kms_concurrent@pipe-H
+kms_content_protection@legacy
+kms_content_protection@atomic
+kms_content_protection@atomic-dpms
+kms_content_protection@LIC
+kms_content_protection@type1
+kms_content_protection@mei_interface
+kms_content_protection@content_type_change
+kms_content_protection@uevent
+kms_content_protection@srm
+kms_content_protection@dp-mst-type-0
+kms_content_protection@dp-mst-lic-type-0
+kms_content_protection@dp-mst-type-1
+kms_content_protection@dp-mst-lic-type-1
+kms_cursor_crc@cursor-size-change
+kms_cursor_crc@cursor-alpha-opaque
+kms_cursor_crc@cursor-alpha-transparent
+kms_cursor_crc@cursor-dpms
+kms_cursor_crc@cursor-suspend
+kms_cursor_crc@cursor-onscreen-32x32
+kms_cursor_crc@cursor-offscreen-32x32
+kms_cursor_crc@cursor-sliding-32x32
+kms_cursor_crc@cursor-random-32x32
+kms_cursor_crc@cursor-rapid-movement-32x32
+kms_cursor_crc@cursor-onscreen-32x10
+kms_cursor_crc@cursor-offscreen-32x10
+kms_cursor_crc@cursor-sliding-32x10
+kms_cursor_crc@cursor-random-32x10
+kms_cursor_crc@cursor-rapid-movement-32x10
+kms_cursor_crc@cursor-onscreen-64x64
+kms_cursor_crc@cursor-offscreen-64x64
+kms_cursor_crc@cursor-sliding-64x64
+kms_cursor_crc@cursor-random-64x64
+kms_cursor_crc@cursor-rapid-movement-64x64
+kms_cursor_crc@cursor-onscreen-64x21
+kms_cursor_crc@cursor-offscreen-64x21
+kms_cursor_crc@cursor-sliding-64x21
+kms_cursor_crc@cursor-random-64x21
+kms_cursor_crc@cursor-rapid-movement-64x21
+kms_cursor_crc@cursor-onscreen-128x128
+kms_cursor_crc@cursor-offscreen-128x128
+kms_cursor_crc@cursor-sliding-128x128
+kms_cursor_crc@cursor-random-128x128
+kms_cursor_crc@cursor-rapid-movement-128x128
+kms_cursor_crc@cursor-onscreen-128x42
+kms_cursor_crc@cursor-offscreen-128x42
+kms_cursor_crc@cursor-sliding-128x42
+kms_cursor_crc@cursor-random-128x42
+kms_cursor_crc@cursor-rapid-movement-128x42
+kms_cursor_crc@cursor-onscreen-256x256
+kms_cursor_crc@cursor-offscreen-256x256
+kms_cursor_crc@cursor-sliding-256x256
+kms_cursor_crc@cursor-random-256x256
+kms_cursor_crc@cursor-rapid-movement-256x256
+kms_cursor_crc@cursor-onscreen-256x85
+kms_cursor_crc@cursor-offscreen-256x85
+kms_cursor_crc@cursor-sliding-256x85
+kms_cursor_crc@cursor-random-256x85
+kms_cursor_crc@cursor-rapid-movement-256x85
+kms_cursor_crc@cursor-onscreen-512x512
+kms_cursor_crc@cursor-offscreen-512x512
+kms_cursor_crc@cursor-sliding-512x512
+kms_cursor_crc@cursor-random-512x512
+kms_cursor_crc@cursor-rapid-movement-512x512
+kms_cursor_crc@cursor-onscreen-512x170
+kms_cursor_crc@cursor-offscreen-512x170
+kms_cursor_crc@cursor-sliding-512x170
+kms_cursor_crc@cursor-random-512x170
+kms_cursor_crc@cursor-rapid-movement-512x170
+kms_cursor_crc@cursor-onscreen-max-size
+kms_cursor_crc@cursor-offscreen-max-size
+kms_cursor_crc@cursor-sliding-max-size
+kms_cursor_crc@cursor-random-max-size
+kms_cursor_crc@cursor-rapid-movement-max-size
+kms_cursor_legacy@single-bo
+kms_cursor_legacy@single-move
+kms_cursor_legacy@forked-bo
+kms_cursor_legacy@forked-move
+kms_cursor_legacy@torture-bo
+kms_cursor_legacy@torture-move
+kms_cursor_legacy@nonblocking-modeset-vs-cursor-atomic
+kms_cursor_legacy@long-nonblocking-modeset-vs-cursor-atomic
+kms_cursor_legacy@2x-flip-vs-cursor-legacy
+kms_cursor_legacy@2x-flip-vs-cursor-atomic
+kms_cursor_legacy@2x-long-flip-vs-cursor-legacy
+kms_cursor_legacy@2x-long-flip-vs-cursor-atomic
+kms_cursor_legacy@2x-nonblocking-modeset-vs-cursor-atomic
+kms_cursor_legacy@2x-long-nonblocking-modeset-vs-cursor-atomic
+kms_cursor_legacy@2x-cursor-vs-flip-legacy
+kms_cursor_legacy@2x-long-cursor-vs-flip-legacy
+kms_cursor_legacy@2x-cursor-vs-flip-atomic
+kms_cursor_legacy@2x-long-cursor-vs-flip-atomic
+kms_cursor_legacy@flip-vs-cursor-crc-legacy
+kms_cursor_legacy@flip-vs-cursor-crc-atomic
+kms_cursor_legacy@flip-vs-cursor-busy-crc-legacy
+kms_cursor_legacy@flip-vs-cursor-busy-crc-atomic
+kms_cursor_legacy@basic-flip-before-cursor-legacy
+kms_cursor_legacy@basic-busy-flip-before-cursor-legacy
+kms_cursor_legacy@basic-flip-after-cursor-legacy
+kms_cursor_legacy@basic-flip-before-cursor-varying-size
+kms_cursor_legacy@basic-busy-flip-before-cursor-varying-size
+kms_cursor_legacy@basic-flip-after-cursor-varying-size
+kms_cursor_legacy@short-flip-before-cursor-toggle
+kms_cursor_legacy@short-busy-flip-before-cursor-toggle
+kms_cursor_legacy@short-flip-after-cursor-toggle
+kms_cursor_legacy@basic-flip-before-cursor-atomic
+kms_cursor_legacy@basic-busy-flip-before-cursor-atomic
+kms_cursor_legacy@basic-flip-after-cursor-atomic
+kms_cursor_legacy@short-flip-before-cursor-atomic-transitions
+kms_cursor_legacy@short-busy-flip-before-cursor-atomic-transitions
+kms_cursor_legacy@short-flip-after-cursor-atomic-transitions
+kms_cursor_legacy@short-flip-before-cursor-atomic-transitions-varying-size
+kms_cursor_legacy@short-busy-flip-before-cursor-atomic-transitions-varying-size
+kms_cursor_legacy@short-flip-after-cursor-atomic-transitions-varying-size
+kms_cursor_legacy@cursor-vs-flip-legacy
+kms_cursor_legacy@flip-vs-cursor-legacy
+kms_cursor_legacy@cursorA-vs-flipA-legacy
+kms_cursor_legacy@cursorA-vs-flipB-legacy
+kms_cursor_legacy@cursorB-vs-flipA-legacy
+kms_cursor_legacy@cursorB-vs-flipB-legacy
+kms_cursor_legacy@cursor-vs-flip-varying-size
+kms_cursor_legacy@flip-vs-cursor-varying-size
+kms_cursor_legacy@cursorA-vs-flipA-varying-size
+kms_cursor_legacy@cursorA-vs-flipB-varying-size
+kms_cursor_legacy@cursorB-vs-flipA-varying-size
+kms_cursor_legacy@cursorB-vs-flipB-varying-size
+kms_cursor_legacy@cursor-vs-flip-toggle
+kms_cursor_legacy@flip-vs-cursor-toggle
+kms_cursor_legacy@cursorA-vs-flipA-toggle
+kms_cursor_legacy@cursorA-vs-flipB-toggle
+kms_cursor_legacy@cursorB-vs-flipA-toggle
+kms_cursor_legacy@cursorB-vs-flipB-toggle
+kms_cursor_legacy@cursor-vs-flip-atomic
+kms_cursor_legacy@flip-vs-cursor-atomic
+kms_cursor_legacy@cursorA-vs-flipA-atomic
+kms_cursor_legacy@cursorA-vs-flipB-atomic
+kms_cursor_legacy@cursorB-vs-flipA-atomic
+kms_cursor_legacy@cursorB-vs-flipB-atomic
+kms_cursor_legacy@cursor-vs-flip-atomic-transitions
+kms_cursor_legacy@flip-vs-cursor-atomic-transitions
+kms_cursor_legacy@cursorA-vs-flipA-atomic-transitions
+kms_cursor_legacy@cursorA-vs-flipB-atomic-transitions
+kms_cursor_legacy@cursorB-vs-flipA-atomic-transitions
+kms_cursor_legacy@cursorB-vs-flipB-atomic-transitions
+kms_cursor_legacy@cursor-vs-flip-atomic-transitions-varying-size
+kms_cursor_legacy@flip-vs-cursor-atomic-transitions-varying-size
+kms_cursor_legacy@cursorA-vs-flipA-atomic-transitions-varying-size
+kms_cursor_legacy@cursorA-vs-flipB-atomic-transitions-varying-size
+kms_cursor_legacy@cursorB-vs-flipA-atomic-transitions-varying-size
+kms_cursor_legacy@cursorB-vs-flipB-atomic-transitions-varying-size
+kms_dither@FB-8BPC-Vs-Panel-6BPC
+kms_dither@FB-8BPC-Vs-Panel-8BPC
+kms_dp_aux_dev
+kms_tiled_display@basic-test-pattern
+kms_tiled_display@basic-test-pattern-with-chamelium
+kms_draw_crc@draw-method-mmap-cpu
+kms_draw_crc@draw-method-mmap-gtt
+kms_draw_crc@draw-method-mmap-wc
+kms_draw_crc@draw-method-pwrite
+kms_draw_crc@draw-method-blt
+kms_draw_crc@draw-method-render
+kms_draw_crc@fill-fb
+kms_dsc@dsc-basic
+kms_dsc@dsc-with-formats
+kms_dsc@dsc-with-bpc
+kms_dsc@dsc-with-bpc-formats
+kms_dsc@dsc-with-output-formats
+kms_fbcon_fbt@fbc
+kms_fbcon_fbt@psr
+kms_fbcon_fbt@fbc-suspend
+kms_fbcon_fbt@psr-suspend
+kms_fence_pin_leak
+kms_flip@nonblocking-read
+kms_flip@wf_vblank-ts-check
+kms_flip@2x-wf_vblank-ts-check
+kms_flip@blocking-wf_vblank
+kms_flip@2x-blocking-wf_vblank
+kms_flip@absolute-wf_vblank
+kms_flip@2x-absolute-wf_vblank
+kms_flip@blocking-absolute-wf_vblank
+kms_flip@2x-blocking-absolute-wf_vblank
+kms_flip@basic-plain-flip
+kms_flip@2x-plain-flip
+kms_flip@busy-flip
+kms_flip@2x-busy-flip
+kms_flip@flip-vs-fences
+kms_flip@2x-flip-vs-fences
+kms_flip@plain-flip-ts-check
+kms_flip@2x-plain-flip-ts-check
+kms_flip@plain-flip-fb-recreate
+kms_flip@2x-plain-flip-fb-recreate
+kms_flip@flip-vs-rmfb
+kms_flip@2x-flip-vs-rmfb
+kms_flip@basic-flip-vs-dpms
+kms_flip@2x-flip-vs-dpms
+kms_flip@flip-vs-panning
+kms_flip@2x-flip-vs-panning
+kms_flip@basic-flip-vs-modeset
+kms_flip@2x-flip-vs-modeset
+kms_flip@flip-vs-expired-vblank
+kms_flip@2x-flip-vs-expired-vblank
+kms_flip@flip-vs-absolute-wf_vblank
+kms_flip@2x-flip-vs-absolute-wf_vblank
+kms_flip@basic-flip-vs-wf_vblank
+kms_flip@2x-flip-vs-wf_vblank
+kms_flip@flip-vs-blocking-wf-vblank
+kms_flip@2x-flip-vs-blocking-wf-vblank
+kms_flip@flip-vs-modeset-vs-hang
+kms_flip@2x-flip-vs-modeset-vs-hang
+kms_flip@flip-vs-panning-vs-hang
+kms_flip@2x-flip-vs-panning-vs-hang
+kms_flip@flip-vs-dpms-off-vs-modeset
+kms_flip@2x-flip-vs-dpms-off-vs-modeset
+kms_flip@single-buffer-flip-vs-dpms-off-vs-modeset
+kms_flip@2x-single-buffer-flip-vs-dpms-off-vs-modeset
+kms_flip@dpms-off-confusion
+kms_flip@nonexisting-fb
+kms_flip@2x-nonexisting-fb
+kms_flip@dpms-vs-vblank-race
+kms_flip@2x-dpms-vs-vblank-race
+kms_flip@modeset-vs-vblank-race
+kms_flip@2x-modeset-vs-vblank-race
+kms_flip@bo-too-big
+kms_flip@flip-vs-suspend
+kms_flip@2x-flip-vs-suspend
+kms_flip@wf_vblank-ts-check-interruptible
+kms_flip@2x-wf_vblank-ts-check-interruptible
+kms_flip@absolute-wf_vblank-interruptible
+kms_flip@2x-absolute-wf_vblank-interruptible
+kms_flip@blocking-absolute-wf_vblank-interruptible
+kms_flip@2x-blocking-absolute-wf_vblank-interruptible
+kms_flip@plain-flip-interruptible
+kms_flip@2x-plain-flip-interruptible
+kms_flip@flip-vs-fences-interruptible
+kms_flip@2x-flip-vs-fences-interruptible
+kms_flip@plain-flip-ts-check-interruptible
+kms_flip@2x-plain-flip-ts-check-interruptible
+kms_flip@plain-flip-fb-recreate-interruptible
+kms_flip@2x-plain-flip-fb-recreate-interruptible
+kms_flip@flip-vs-rmfb-interruptible
+kms_flip@2x-flip-vs-rmfb-interruptible
+kms_flip@flip-vs-panning-interruptible
+kms_flip@2x-flip-vs-panning-interruptible
+kms_flip@flip-vs-expired-vblank-interruptible
+kms_flip@2x-flip-vs-expired-vblank-interruptible
+kms_flip@flip-vs-absolute-wf_vblank-interruptible
+kms_flip@2x-flip-vs-absolute-wf_vblank-interruptible
+kms_flip@flip-vs-wf_vblank-interruptible
+kms_flip@2x-flip-vs-wf_vblank-interruptible
+kms_flip@flip-vs-dpms-off-vs-modeset-interruptible
+kms_flip@2x-flip-vs-dpms-off-vs-modeset-interruptible
+kms_flip@single-buffer-flip-vs-dpms-off-vs-modeset-interruptible
+kms_flip@2x-single-buffer-flip-vs-dpms-off-vs-modeset-interruptible
+kms_flip@dpms-off-confusion-interruptible
+kms_flip@nonexisting-fb-interruptible
+kms_flip@2x-nonexisting-fb-interruptible
+kms_flip@dpms-vs-vblank-race-interruptible
+kms_flip@2x-dpms-vs-vblank-race-interruptible
+kms_flip@modeset-vs-vblank-race-interruptible
+kms_flip@2x-modeset-vs-vblank-race-interruptible
+kms_flip@bo-too-big-interruptible
+kms_flip@flip-vs-suspend-interruptible
+kms_flip@2x-flip-vs-suspend-interruptible
+kms_flip_event_leak@basic
+kms_flip_scaled_crc@flip-32bpp-ytile-to-64bpp-ytile-downscaling
+kms_flip_scaled_crc@flip-32bpp-yftile-to-64bpp-yftile-downscaling
+kms_flip_scaled_crc@flip-32bpp-xtile-to-64bpp-xtile-downscaling
+kms_flip_scaled_crc@flip-32bpp-4tile-to-64bpp-4tile-downscaling
+kms_flip_scaled_crc@flip-32bpp-linear-to-64bpp-linear-downscaling
+kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytile-downscaling
+kms_flip_scaled_crc@flip-64bpp-yftile-to-32bpp-yftile-downscaling
+kms_flip_scaled_crc@flip-64bpp-xtile-to-32bpp-xtile-downscaling
+kms_flip_scaled_crc@flip-64bpp-4tile-to-32bpp-4tile-downscaling
+kms_flip_scaled_crc@flip-64bpp-linear-to-32bpp-linear-downscaling
+kms_flip_scaled_crc@flip-64bpp-ytile-to-16bpp-ytile-downscaling
+kms_flip_scaled_crc@flip-64bpp-yftile-to-16bpp-yftile-downscaling
+kms_flip_scaled_crc@flip-64bpp-xtile-to-16bpp-xtile-downscaling
+kms_flip_scaled_crc@flip-64bpp-4tile-to-16bpp-4tile-downscaling
+kms_flip_scaled_crc@flip-64bpp-linear-to-16bpp-linear-downscaling
+kms_flip_scaled_crc@flip-32bpp-ytileccs-to-64bpp-ytile-downscaling
+kms_flip_scaled_crc@flip-32bpp-yftileccs-to-64bpp-yftile-downscaling
+kms_flip_scaled_crc@flip-32bpp-ytile-to-32bpp-ytilegen12rcccs-downscaling
+kms_flip_scaled_crc@flip-32bpp-4tile-to-32bpp-4tiledg2rcccs-downscaling
+kms_flip_scaled_crc@flip-32bpp-ytile-to-32bpp-ytileccs-downscaling
+kms_flip_scaled_crc@flip-32bpp-yftile-to-32bpp-yftileccs-downscaling
+kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytilercccs-downscaling
+kms_flip_scaled_crc@flip-64bpp-4tile-to-32bpp-4tiledg2rcccs-downscaling
+kms_flip_scaled_crc@flip-32bpp-ytile-to-64bpp-ytile-upscaling
+kms_flip_scaled_crc@flip-32bpp-yftile-to-64bpp-yftile-upscaling
+kms_flip_scaled_crc@flip-32bpp-xtile-to-64bpp-xtile-upscaling
+kms_flip_scaled_crc@flip-32bpp-4tile-to-64bpp-4tile-upscaling
+kms_flip_scaled_crc@flip-32bpp-linear-to-64bpp-linear-upscaling
+kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytile-upscaling
+kms_flip_scaled_crc@flip-64bpp-yftile-to-32bpp-yftile-upscaling
+kms_flip_scaled_crc@flip-64bpp-xtile-to-32bpp-xtile-upscaling
+kms_flip_scaled_crc@flip-64bpp-4tile-to-32bpp-4tile-upscaling
+kms_flip_scaled_crc@flip-64bpp-linear-to-32bpp-linear-upscaling
+kms_flip_scaled_crc@flip-64bpp-ytile-to-16bpp-ytile-upscaling
+kms_flip_scaled_crc@flip-64bpp-yftile-to-16bpp-yftile-upscaling
+kms_flip_scaled_crc@flip-64bpp-xtile-to-16bpp-xtile-upscaling
+kms_flip_scaled_crc@flip-64bpp-4tile-to-16bpp-4tile-upscaling
+kms_flip_scaled_crc@flip-64bpp-linear-to-16bpp-linear-upscaling
+kms_flip_scaled_crc@flip-32bpp-ytileccs-to-64bpp-ytile-upscaling
+kms_flip_scaled_crc@flip-32bpp-yftileccs-to-64bpp-yftile-upscaling
+kms_flip_scaled_crc@flip-32bpp-ytile-to-32bpp-ytilegen12rcccs-upscaling
+kms_flip_scaled_crc@flip-32bpp-4tile-to-32bpp-4tiledg2rcccs-upscaling
+kms_flip_scaled_crc@flip-32bpp-ytile-to-32bpp-ytileccs-upscaling
+kms_flip_scaled_crc@flip-32bpp-yftile-to-32bpp-yftileccs-upscaling
+kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytilegen12rcccs-upscaling
+kms_flip_scaled_crc@flip-64bpp-4tile-to-32bpp-4tiledg2rcccs-upscaling
+kms_force_connector_basic@force-load-detect
+kms_force_connector_basic@force-connector-state
+kms_force_connector_basic@force-edid
+kms_force_connector_basic@prune-stale-modes
+kms_frontbuffer_tracking@fbc-1p-rte
+kms_frontbuffer_tracking@fbc-2p-rte
+kms_frontbuffer_tracking@psr-1p-rte
+kms_frontbuffer_tracking@psr-2p-rte
+kms_frontbuffer_tracking@fbcpsr-1p-rte
+kms_frontbuffer_tracking@fbcpsr-2p-rte
+kms_frontbuffer_tracking@drrs-1p-rte
+kms_frontbuffer_tracking@drrs-2p-rte
+kms_frontbuffer_tracking@fbcdrrs-1p-rte
+kms_frontbuffer_tracking@fbcdrrs-2p-rte
+kms_frontbuffer_tracking@psrdrrs-1p-rte
+kms_frontbuffer_tracking@psrdrrs-2p-rte
+kms_frontbuffer_tracking@fbcpsrdrrs-1p-rte
+kms_frontbuffer_tracking@fbcpsrdrrs-2p-rte
+kms_frontbuffer_tracking@fbc-1p-primscrn-pri-indfb-draw-mmap-cpu
+kms_frontbuffer_tracking@fbc-1p-primscrn-pri-indfb-draw-mmap-gtt
+kms_frontbuffer_tracking@fbc-1p-primscrn-pri-indfb-draw-mmap-wc
+kms_frontbuffer_tracking@fbc-1p-primscrn-pri-indfb-draw-pwrite
+kms_frontbuffer_tracking@fbc-1p-primscrn-pri-indfb-draw-blt
+kms_frontbuffer_tracking@fbc-1p-primscrn-pri-indfb-draw-render
+kms_frontbuffer_tracking@fbc-1p-primscrn-pri-shrfb-draw-mmap-cpu
+kms_frontbuffer_tracking@fbc-1p-primscrn-pri-shrfb-draw-mmap-gtt
+kms_frontbuffer_tracking@fbc-1p-primscrn-pri-shrfb-draw-mmap-wc
+kms_frontbuffer_tracking@fbc-1p-primscrn-pri-shrfb-draw-pwrite
+kms_frontbuffer_tracking@fbc-1p-primscrn-pri-shrfb-draw-blt
+kms_frontbuffer_tracking@fbc-1p-primscrn-pri-shrfb-draw-render
+kms_frontbuffer_tracking@fbc-1p-primscrn-cur-indfb-draw-mmap-cpu
+kms_frontbuffer_tracking@fbc-1p-primscrn-cur-indfb-draw-mmap-gtt
+kms_frontbuffer_tracking@fbc-1p-primscrn-cur-indfb-draw-mmap-wc
+kms_frontbuffer_tracking@fbc-1p-primscrn-cur-indfb-draw-pwrite
+kms_frontbuffer_tracking@fbc-1p-primscrn-cur-indfb-draw-blt
+kms_frontbuffer_tracking@fbc-1p-primscrn-cur-indfb-draw-render
+kms_frontbuffer_tracking@fbc-1p-primscrn-spr-indfb-draw-mmap-cpu
+kms_frontbuffer_tracking@fbc-1p-primscrn-spr-indfb-draw-mmap-gtt
+kms_frontbuffer_tracking@fbc-1p-primscrn-spr-indfb-draw-mmap-wc
+kms_frontbuffer_tracking@fbc-1p-primscrn-spr-indfb-draw-pwrite
+kms_frontbuffer_tracking@fbc-1p-primscrn-spr-indfb-draw-blt
+kms_frontbuffer_tracking@fbc-1p-primscrn-spr-indfb-draw-render
+kms_frontbuffer_tracking@fbc-1p-offscren-pri-indfb-draw-mmap-cpu
+kms_frontbuffer_tracking@fbc-1p-offscren-pri-indfb-draw-mmap-gtt
+kms_frontbuffer_tracking@fbc-1p-offscren-pri-indfb-draw-mmap-wc
+kms_frontbuffer_tracking@fbc-1p-offscren-pri-indfb-draw-pwrite
+kms_frontbuffer_tracking@fbc-1p-offscren-pri-indfb-draw-blt
+kms_frontbuffer_tracking@fbc-1p-offscren-pri-indfb-draw-render
+kms_frontbuffer_tracking@fbc-1p-offscren-pri-shrfb-draw-mmap-cpu
+kms_frontbuffer_tracking@fbc-1p-offscren-pri-shrfb-draw-mmap-gtt
+kms_frontbuffer_tracking@fbc-1p-offscren-pri-shrfb-draw-mmap-wc
+kms_frontbuffer_tracking@fbc-1p-offscren-pri-shrfb-draw-pwrite
+kms_frontbuffer_tracking@fbc-1p-offscren-pri-shrfb-draw-blt
+kms_frontbuffer_tracking@fbc-1p-offscren-pri-shrfb-draw-render
+kms_frontbuffer_tracking@fbc-2p-primscrn-pri-indfb-draw-mmap-cpu
+kms_frontbuffer_tracking@fbc-2p-primscrn-pri-indfb-draw-mmap-gtt
+kms_frontbuffer_tracking@fbc-2p-primscrn-pri-indfb-draw-mmap-wc
+kms_frontbuffer_tracking@fbc-2p-primscrn-pri-indfb-draw-pwrite
+kms_frontbuffer_tracking@fbc-2p-primscrn-pri-indfb-draw-blt
+kms_frontbuffer_tracking@fbc-2p-primscrn-pri-indfb-draw-render
+kms_frontbuffer_tracking@fbc-2p-primscrn-pri-shrfb-draw-mmap-cpu
+kms_frontbuffer_tracking@fbc-2p-primscrn-pri-shrfb-draw-mmap-gtt
+kms_frontbuffer_tracking@fbc-2p-primscrn-pri-shrfb-draw-mmap-wc
+kms_frontbuffer_tracking@fbc-2p-primscrn-pri-shrfb-draw-pwrite
+kms_frontbuffer_tracking@fbc-2p-primscrn-pri-shrfb-draw-blt
+kms_frontbuffer_tracking@fbc-2p-primscrn-pri-shrfb-draw-render
+kms_frontbuffer_tracking@fbc-2p-primscrn-cur-indfb-draw-mmap-cpu
+kms_frontbuffer_tracking@fbc-2p-primscrn-cur-indfb-draw-mmap-gtt
+kms_frontbuffer_tracking@fbc-2p-primscrn-cur-indfb-draw-mmap-wc
+kms_frontbuffer_tracking@fbc-2p-primscrn-cur-indfb-draw-pwrite
+kms_frontbuffer_tracking@fbc-2p-primscrn-cur-indfb-draw-blt
+kms_frontbuffer_tracking@fbc-2p-primscrn-cur-indfb-draw-render
+kms_frontbuffer_tracking@fbc-2p-primscrn-spr-indfb-draw-mmap-cpu
+kms_frontbuffer_tracking@fbc-2p-primscrn-spr-indfb-draw-mmap-gtt
+kms_frontbuffer_tracking@fbc-2p-primscrn-spr-indfb-draw-mmap-wc
+kms_frontbuffer_tracking@fbc-2p-primscrn-spr-indfb-draw-pwrite
+kms_frontbuffer_tracking@fbc-2p-primscrn-spr-indfb-draw-blt
+kms_frontbuffer_tracking@fbc-2p-primscrn-spr-indfb-draw-render
+kms_frontbuffer_tracking@fbc-2p-scndscrn-pri-indfb-draw-mmap-cpu
+kms_frontbuffer_tracking@fbc-2p-scndscrn-pri-indfb-draw-mmap-gtt
+kms_frontbuffer_tracking@fbc-2p-scndscrn-pri-indfb-draw-mmap-wc
+kms_frontbuffer_tracking@fbc-2p-scndscrn-pri-indfb-draw-pwrite
+kms_frontbuffer_tracking@fbc-2p-scndscrn-pri-indfb-draw-blt
+kms_frontbuffer_tracking@fbc-2p-scndscrn-pri-indfb-draw-render
+kms_frontbuffer_tracking@fbc-2p-scndscrn-pri-shrfb-draw-mmap-cpu
+kms_frontbuffer_tracking@fbc-2p-scndscrn-pri-shrfb-draw-mmap-gtt
+kms_frontbuffer_tracking@fbc-2p-scndscrn-pri-shrfb-draw-mmap-wc
+kms_frontbuffer_tracking@fbc-2p-scndscrn-pri-shrfb-draw-pwrite
+kms_frontbuffer_tracking@fbc-2p-scndscrn-pri-shrfb-draw-blt
+kms_frontbuffer_tracking@fbc-2p-scndscrn-pri-shrfb-draw-render
+kms_frontbuffer_tracking@fbc-2p-scndscrn-cur-indfb-draw-mmap-cpu
+kms_frontbuffer_tracking@fbc-2p-scndscrn-cur-indfb-draw-mmap-gtt
+kms_frontbuffer_tracking@fbc-2p-scndscrn-cur-indfb-draw-mmap-wc
+kms_frontbuffer_tracking@fbc-2p-scndscrn-cur-indfb-draw-pwrite
+kms_frontbuffer_tracking@fbc-2p-scndscrn-cur-indfb-draw-blt
+kms_frontbuffer_tracking@fbc-2p-scndscrn-cur-indfb-draw-render
+kms_frontbuffer_tracking@fbc-2p-scndscrn-spr-indfb-draw-mmap-cpu
+kms_frontbuffer_tracking@fbc-2p-scndscrn-spr-indfb-draw-mmap-gtt
+kms_frontbuffer_tracking@fbc-2p-scndscrn-spr-indfb-draw-mmap-wc
+kms_frontbuffer_tracking@fbc-2p-scndscrn-spr-indfb-draw-pwrite
+kms_frontbuffer_tracking@fbc-2p-scndscrn-spr-indfb-draw-blt
+kms_frontbuffer_tracking@fbc-2p-scndscrn-spr-indfb-draw-render
+kms_frontbuffer_tracking@psr-1p-primscrn-pri-indfb-draw-mmap-cpu
+kms_frontbuffer_tracking@psr-1p-primscrn-pri-indfb-draw-mmap-gtt
+kms_frontbuffer_tracking@psr-1p-primscrn-pri-indfb-draw-mmap-wc
+kms_frontbuffer_tracking@psr-1p-primscrn-pri-indfb-draw-pwrite
+kms_frontbuffer_tracking@psr-1p-primscrn-pri-indfb-draw-blt
+kms_frontbuffer_tracking@psr-1p-primscrn-pri-indfb-draw-render
+kms_frontbuffer_tracking@psr-1p-primscrn-pri-shrfb-draw-mmap-cpu
+kms_frontbuffer_tracking@psr-1p-primscrn-pri-shrfb-draw-mmap-gtt
+kms_frontbuffer_tracking@psr-1p-primscrn-pri-shrfb-draw-mmap-wc
+kms_frontbuffer_tracking@psr-1p-primscrn-pri-shrfb-draw-pwrite
+kms_frontbuffer_tracking@psr-1p-primscrn-pri-shrfb-draw-blt
+kms_frontbuffer_tracking@psr-1p-primscrn-pri-shrfb-draw-render
+kms_frontbuffer_tracking@psr-1p-primscrn-cur-indfb-draw-mmap-cpu
+kms_frontbuffer_tracking@psr-1p-primscrn-cur-indfb-draw-mmap-gtt
+kms_frontbuffer_tracking@psr-1p-primscrn-cur-indfb-draw-mmap-wc
+kms_frontbuffer_tracking@psr-1p-primscrn-cur-indfb-draw-pwrite
+kms_frontbuffer_tracking@psr-1p-primscrn-cur-indfb-draw-blt
+kms_frontbuffer_tracking@psr-1p-primscrn-cur-indfb-draw-render
+kms_frontbuffer_tracking@psr-1p-primscrn-spr-indfb-draw-mmap-cpu
+kms_frontbuffer_tracking@psr-1p-primscrn-spr-indfb-draw-mmap-gtt
+kms_frontbuffer_tracking@psr-1p-primscrn-spr-indfb-draw-mmap-wc
+kms_frontbuffer_tracking@psr-1p-primscrn-spr-indfb-draw-pwrite
+kms_frontbuffer_tracking@psr-1p-primscrn-spr-indfb-draw-blt
+kms_frontbuffer_tracking@psr-1p-primscrn-spr-indfb-draw-render
+kms_frontbuffer_tracking@psr-1p-offscren-pri-indfb-draw-mmap-cpu
+kms_frontbuffer_tracking@psr-1p-offscren-pri-indfb-draw-mmap-gtt
+kms_frontbuffer_tracking@psr-1p-offscren-pri-indfb-draw-mmap-wc
+kms_frontbuffer_tracking@psr-1p-offscren-pri-indfb-draw-pwrite
+kms_frontbuffer_tracking@psr-1p-offscren-pri-indfb-draw-blt
+kms_frontbuffer_tracking@psr-1p-offscren-pri-indfb-draw-render
+kms_frontbuffer_tracking@psr-1p-offscren-pri-shrfb-draw-mmap-cpu
+kms_frontbuffer_tracking@psr-1p-offscren-pri-shrfb-draw-mmap-gtt
+kms_frontbuffer_tracking@psr-1p-offscren-pri-shrfb-draw-mmap-wc
+kms_frontbuffer_tracking@psr-1p-offscren-pri-shrfb-draw-pwrite
+kms_frontbuffer_tracking@psr-1p-offscren-pri-shrfb-draw-blt
+kms_frontbuffer_tracking@psr-1p-offscren-pri-shrfb-draw-render
+kms_frontbuffer_tracking@psr-2p-primscrn-pri-indfb-draw-mmap-cpu
+kms_frontbuffer_tracking@psr-2p-primscrn-pri-indfb-draw-mmap-gtt
+kms_frontbuffer_tracking@psr-2p-primscrn-pri-indfb-draw-mmap-wc
+kms_frontbuffer_tracking@psr-2p-primscrn-pri-indfb-draw-pwrite
+kms_frontbuffer_tracking@psr-2p-primscrn-pri-indfb-draw-blt
+kms_frontbuffer_tracking@psr-2p-primscrn-pri-indfb-draw-render
+kms_frontbuffer_tracking@psr-2p-primscrn-pri-shrfb-draw-mmap-cpu
+kms_frontbuffer_tracking@psr-2p-primscrn-pri-shrfb-draw-mmap-gtt
+kms_frontbuffer_tracking@psr-2p-primscrn-pri-shrfb-draw-mmap-wc
+kms_frontbuffer_tracking@psr-2p-primscrn-pri-shrfb-draw-pwrite
+kms_frontbuffer_tracking@psr-2p-primscrn-pri-shrfb-draw-blt
+kms_frontbuffer_tracking@psr-2p-primscrn-pri-shrfb-draw-render
+kms_frontbuffer_tracking@psr-2p-primscrn-cur-indfb-draw-mmap-cpu
+kms_frontbuffer_tracking@psr-2p-primscrn-cur-indfb-draw-mmap-gtt
+kms_frontbuffer_tracking@psr-2p-primscrn-cur-indfb-draw-mmap-wc
+kms_frontbuffer_tracking@psr-2p-primscrn-cur-indfb-draw-pwrite
+kms_frontbuffer_tracking@psr-2p-primscrn-cur-indfb-draw-blt
+kms_frontbuffer_tracking@psr-2p-primscrn-cur-indfb-draw-render
+kms_frontbuffer_tracking@psr-2p-primscrn-spr-indfb-draw-mmap-cpu
+kms_frontbuffer_tracking@psr-2p-primscrn-spr-indfb-draw-mmap-gtt
+kms_frontbuffer_tracking@psr-2p-primscrn-spr-indfb-draw-mmap-wc
+kms_frontbuffer_tracking@psr-2p-primscrn-spr-indfb-draw-pwrite
+kms_frontbuffer_tracking@psr-2p-primscrn-spr-indfb-draw-blt
+kms_frontbuffer_tracking@psr-2p-primscrn-spr-indfb-draw-render
+kms_frontbuffer_tracking@psr-2p-scndscrn-pri-indfb-draw-mmap-cpu
+kms_frontbuffer_tracking@psr-2p-scndscrn-pri-indfb-draw-mmap-gtt
+kms_frontbuffer_tracking@psr-2p-scndscrn-pri-indfb-draw-mmap-wc
+kms_frontbuffer_tracking@psr-2p-scndscrn-pri-indfb-draw-pwrite
+kms_frontbuffer_tracking@psr-2p-scndscrn-pri-indfb-draw-blt
+kms_frontbuffer_tracking@psr-2p-scndscrn-pri-indfb-draw-render
+kms_frontbuffer_tracking@psr-2p-scndscrn-pri-shrfb-draw-mmap-cpu
+kms_frontbuffer_tracking@psr-2p-scndscrn-pri-shrfb-draw-mmap-gtt
+kms_frontbuffer_tracking@psr-2p-scndscrn-pri-shrfb-draw-mmap-wc
+kms_frontbuffer_tracking@psr-2p-scndscrn-pri-shrfb-draw-pwrite
+kms_frontbuffer_tracking@psr-2p-scndscrn-pri-shrfb-draw-blt
+kms_frontbuffer_tracking@psr-2p-scndscrn-pri-shrfb-draw-render
+kms_frontbuffer_tracking@psr-2p-scndscrn-cur-indfb-draw-mmap-cpu
+kms_frontbuffer_tracking@psr-2p-scndscrn-cur-indfb-draw-mmap-gtt
+kms_frontbuffer_tracking@psr-2p-scndscrn-cur-indfb-draw-mmap-wc
+kms_frontbuffer_tracking@psr-2p-scndscrn-cur-indfb-draw-pwrite
+kms_frontbuffer_tracking@psr-2p-scndscrn-cur-indfb-draw-blt
+kms_frontbuffer_tracking@psr-2p-scndscrn-cur-indfb-draw-render
+kms_frontbuffer_tracking@psr-2p-scndscrn-spr-indfb-draw-mmap-cpu
+kms_frontbuffer_tracking@psr-2p-scndscrn-spr-indfb-draw-mmap-gtt
+kms_frontbuffer_tracking@psr-2p-scndscrn-spr-indfb-draw-mmap-wc
+kms_frontbuffer_tracking@psr-2p-scndscrn-spr-indfb-draw-pwrite
+kms_frontbuffer_tracking@psr-2p-scndscrn-spr-indfb-draw-blt
+kms_frontbuffer_tracking@psr-2p-scndscrn-spr-indfb-draw-render
+kms_frontbuffer_tracking@fbcpsr-1p-primscrn-pri-indfb-draw-mmap-cpu
+kms_frontbuffer_tracking@fbcpsr-1p-primscrn-pri-indfb-draw-mmap-gtt
+kms_frontbuffer_tracking@fbcpsr-1p-primscrn-pri-indfb-draw-mmap-wc
+kms_frontbuffer_tracking@fbcpsr-1p-primscrn-pri-indfb-draw-pwrite
+kms_frontbuffer_tracking@fbcpsr-1p-primscrn-pri-indfb-draw-blt
+kms_frontbuffer_tracking@fbcpsr-1p-primscrn-pri-indfb-draw-render
+kms_frontbuffer_tracking@fbcpsr-1p-primscrn-pri-shrfb-draw-mmap-cpu
+kms_frontbuffer_tracking@fbcpsr-1p-primscrn-pri-shrfb-draw-mmap-gtt
+kms_frontbuffer_tracking@fbcpsr-1p-primscrn-pri-shrfb-draw-mmap-wc
+kms_frontbuffer_tracking@fbcpsr-1p-primscrn-pri-shrfb-draw-pwrite
+kms_frontbuffer_tracking@fbcpsr-1p-primscrn-pri-shrfb-draw-blt
+kms_frontbuffer_tracking@fbcpsr-1p-primscrn-pri-shrfb-draw-render
+kms_frontbuffer_tracking@fbcpsr-1p-primscrn-cur-indfb-draw-mmap-cpu
+kms_frontbuffer_tracking@fbcpsr-1p-primscrn-cur-indfb-draw-mmap-gtt
+kms_frontbuffer_tracking@fbcpsr-1p-primscrn-cur-indfb-draw-mmap-wc
+kms_frontbuffer_tracking@fbcpsr-1p-primscrn-cur-indfb-draw-pwrite
+kms_frontbuffer_tracking@fbcpsr-1p-primscrn-cur-indfb-draw-blt
+kms_frontbuffer_tracking@fbcpsr-1p-primscrn-cur-indfb-draw-render
+kms_frontbuffer_tracking@fbcpsr-1p-primscrn-spr-indfb-draw-mmap-cpu
+kms_frontbuffer_tracking@fbcpsr-1p-primscrn-spr-indfb-draw-mmap-gtt
+kms_frontbuffer_tracking@fbcpsr-1p-primscrn-spr-indfb-draw-mmap-wc
+kms_frontbuffer_tracking@fbcpsr-1p-primscrn-spr-indfb-draw-pwrite
+kms_frontbuffer_tracking@fbcpsr-1p-primscrn-spr-indfb-draw-blt
+kms_frontbuffer_tracking@fbcpsr-1p-primscrn-spr-indfb-draw-render
+kms_frontbuffer_tracking@fbcpsr-1p-offscren-pri-indfb-draw-mmap-cpu
+kms_frontbuffer_tracking@fbcpsr-1p-offscren-pri-indfb-draw-mmap-gtt
+kms_frontbuffer_tracking@fbcpsr-1p-offscren-pri-indfb-draw-mmap-wc
+kms_frontbuffer_tracking@fbcpsr-1p-offscren-pri-indfb-draw-pwrite
+kms_frontbuffer_tracking@fbcpsr-1p-offscren-pri-indfb-draw-blt
+kms_frontbuffer_tracking@fbcpsr-1p-offscren-pri-indfb-draw-render
+kms_frontbuffer_tracking@fbcpsr-1p-offscren-pri-shrfb-draw-mmap-cpu
+kms_frontbuffer_tracking@fbcpsr-1p-offscren-pri-shrfb-draw-mmap-gtt
+kms_frontbuffer_tracking@fbcpsr-1p-offscren-pri-shrfb-draw-mmap-wc
+kms_frontbuffer_tracking@fbcpsr-1p-offscren-pri-shrfb-draw-pwrite
+kms_frontbuffer_tracking@fbcpsr-1p-offscren-pri-shrfb-draw-blt
+kms_frontbuffer_tracking@fbcpsr-1p-offscren-pri-shrfb-draw-render
+kms_frontbuffer_tracking@fbcpsr-2p-primscrn-pri-indfb-draw-mmap-cpu
+kms_frontbuffer_tracking@fbcpsr-2p-primscrn-pri-indfb-draw-mmap-gtt
+kms_frontbuffer_tracking@fbcpsr-2p-primscrn-pri-indfb-draw-mmap-wc
+kms_frontbuffer_tracking@fbcpsr-2p-primscrn-pri-indfb-draw-pwrite
+kms_frontbuffer_tracking@fbcpsr-2p-primscrn-pri-indfb-draw-blt
+kms_frontbuffer_tracking@fbcpsr-2p-primscrn-pri-indfb-draw-render
+kms_frontbuffer_tracking@fbcpsr-2p-primscrn-pri-shrfb-draw-mmap-cpu
+kms_frontbuffer_tracking@fbcpsr-2p-primscrn-pri-shrfb-draw-mmap-gtt
+kms_frontbuffer_tracking@fbcpsr-2p-primscrn-pri-shrfb-draw-mmap-wc
+kms_frontbuffer_tracking@fbcpsr-2p-primscrn-pri-shrfb-draw-pwrite
+kms_frontbuffer_tracking@fbcpsr-2p-primscrn-pri-shrfb-draw-blt
+kms_frontbuffer_tracking@fbcpsr-2p-primscrn-pri-shrfb-draw-render
+kms_frontbuffer_tracking@fbcpsr-2p-primscrn-cur-indfb-draw-mmap-cpu
+kms_frontbuffer_tracking@fbcpsr-2p-primscrn-cur-indfb-draw-mmap-gtt
+kms_frontbuffer_tracking@fbcpsr-2p-primscrn-cur-indfb-draw-mmap-wc
+kms_frontbuffer_tracking@fbcpsr-2p-primscrn-cur-indfb-draw-pwrite
+kms_frontbuffer_tracking@fbcpsr-2p-primscrn-cur-indfb-draw-blt
+kms_frontbuffer_tracking@fbcpsr-2p-primscrn-cur-indfb-draw-render
+kms_frontbuffer_tracking@fbcpsr-2p-primscrn-spr-indfb-draw-mmap-cpu
+kms_frontbuffer_tracking@fbcpsr-2p-primscrn-spr-indfb-draw-mmap-gtt
+kms_frontbuffer_tracking@fbcpsr-2p-primscrn-spr-indfb-draw-mmap-wc
+kms_frontbuffer_tracking@fbcpsr-2p-primscrn-spr-indfb-draw-pwrite
+kms_frontbuffer_tracking@fbcpsr-2p-primscrn-spr-indfb-draw-blt
+kms_frontbuffer_tracking@fbcpsr-2p-primscrn-spr-indfb-draw-render
+kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-pri-indfb-draw-mmap-cpu
+kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-pri-indfb-draw-mmap-gtt
+kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-pri-indfb-draw-mmap-wc
+kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-pri-indfb-draw-pwrite
+kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-pri-indfb-draw-blt
+kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-pri-indfb-draw-render
+kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-pri-shrfb-draw-mmap-cpu
+kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-pri-shrfb-draw-mmap-gtt
+kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-pri-shrfb-draw-mmap-wc
+kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-pri-shrfb-draw-pwrite
+kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-pri-shrfb-draw-blt
+kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-pri-shrfb-draw-render
+kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-cur-indfb-draw-mmap-cpu
+kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-cur-indfb-draw-mmap-gtt
+kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-cur-indfb-draw-mmap-wc
+kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-cur-indfb-draw-pwrite
+kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-cur-indfb-draw-blt
+kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-cur-indfb-draw-render
+kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-spr-indfb-draw-mmap-cpu
+kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-spr-indfb-draw-mmap-gtt
+kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-spr-indfb-draw-mmap-wc
+kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-spr-indfb-draw-pwrite
+kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-spr-indfb-draw-blt
+kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-spr-indfb-draw-render
+kms_frontbuffer_tracking@drrs-1p-primscrn-pri-indfb-draw-mmap-cpu
+kms_frontbuffer_tracking@drrs-1p-primscrn-pri-indfb-draw-mmap-gtt
+kms_frontbuffer_tracking@drrs-1p-primscrn-pri-indfb-draw-mmap-wc
+kms_frontbuffer_tracking@drrs-1p-primscrn-pri-indfb-draw-pwrite
+kms_frontbuffer_tracking@drrs-1p-primscrn-pri-indfb-draw-blt
+kms_frontbuffer_tracking@drrs-1p-primscrn-pri-indfb-draw-render
+kms_frontbuffer_tracking@drrs-1p-primscrn-pri-shrfb-draw-mmap-cpu
+kms_frontbuffer_tracking@drrs-1p-primscrn-pri-shrfb-draw-mmap-gtt
+kms_frontbuffer_tracking@drrs-1p-primscrn-pri-shrfb-draw-mmap-wc
+kms_frontbuffer_tracking@drrs-1p-primscrn-pri-shrfb-draw-pwrite
+kms_frontbuffer_tracking@drrs-1p-primscrn-pri-shrfb-draw-blt
+kms_frontbuffer_tracking@drrs-1p-primscrn-pri-shrfb-draw-render
+kms_frontbuffer_tracking@drrs-1p-primscrn-cur-indfb-draw-mmap-cpu
+kms_frontbuffer_tracking@drrs-1p-primscrn-cur-indfb-draw-mmap-gtt
+kms_frontbuffer_tracking@drrs-1p-primscrn-cur-indfb-draw-mmap-wc
+kms_frontbuffer_tracking@drrs-1p-primscrn-cur-indfb-draw-pwrite
+kms_frontbuffer_tracking@drrs-1p-primscrn-cur-indfb-draw-blt
+kms_frontbuffer_tracking@drrs-1p-primscrn-cur-indfb-draw-render
+kms_frontbuffer_tracking@drrs-1p-primscrn-spr-indfb-draw-mmap-cpu
+kms_frontbuffer_tracking@drrs-1p-primscrn-spr-indfb-draw-mmap-gtt
+kms_frontbuffer_tracking@drrs-1p-primscrn-spr-indfb-draw-mmap-wc
+kms_frontbuffer_tracking@drrs-1p-primscrn-spr-indfb-draw-pwrite
+kms_frontbuffer_tracking@drrs-1p-primscrn-spr-indfb-draw-blt
+kms_frontbuffer_tracking@drrs-1p-primscrn-spr-indfb-draw-render
+kms_frontbuffer_tracking@drrs-1p-offscren-pri-indfb-draw-mmap-cpu
+kms_frontbuffer_tracking@drrs-1p-offscren-pri-indfb-draw-mmap-gtt
+kms_frontbuffer_tracking@drrs-1p-offscren-pri-indfb-draw-mmap-wc
+kms_frontbuffer_tracking@drrs-1p-offscren-pri-indfb-draw-pwrite
+kms_frontbuffer_tracking@drrs-1p-offscren-pri-indfb-draw-blt
+kms_frontbuffer_tracking@drrs-1p-offscren-pri-indfb-draw-render
+kms_frontbuffer_tracking@drrs-1p-offscren-pri-shrfb-draw-mmap-cpu
+kms_frontbuffer_tracking@drrs-1p-offscren-pri-shrfb-draw-mmap-gtt
+kms_frontbuffer_tracking@drrs-1p-offscren-pri-shrfb-draw-mmap-wc
+kms_frontbuffer_tracking@drrs-1p-offscren-pri-shrfb-draw-pwrite
+kms_frontbuffer_tracking@drrs-1p-offscren-pri-shrfb-draw-blt
+kms_frontbuffer_tracking@drrs-1p-offscren-pri-shrfb-draw-render
+kms_frontbuffer_tracking@drrs-2p-primscrn-pri-indfb-draw-mmap-cpu
+kms_frontbuffer_tracking@drrs-2p-primscrn-pri-indfb-draw-mmap-gtt
+kms_frontbuffer_tracking@drrs-2p-primscrn-pri-indfb-draw-mmap-wc
+kms_frontbuffer_tracking@drrs-2p-primscrn-pri-indfb-draw-pwrite
+kms_frontbuffer_tracking@drrs-2p-primscrn-pri-indfb-draw-blt
+kms_frontbuffer_tracking@drrs-2p-primscrn-pri-indfb-draw-render
+kms_frontbuffer_tracking@drrs-2p-primscrn-pri-shrfb-draw-mmap-cpu
+kms_frontbuffer_tracking@drrs-2p-primscrn-pri-shrfb-draw-mmap-gtt
+kms_frontbuffer_tracking@drrs-2p-primscrn-pri-shrfb-draw-mmap-wc
+kms_frontbuffer_tracking@drrs-2p-primscrn-pri-shrfb-draw-pwrite
+kms_frontbuffer_tracking@drrs-2p-primscrn-pri-shrfb-draw-blt
+kms_frontbuffer_tracking@drrs-2p-primscrn-pri-shrfb-draw-render
+kms_frontbuffer_tracking@drrs-2p-primscrn-cur-indfb-draw-mmap-cpu
+kms_frontbuffer_tracking@drrs-2p-primscrn-cur-indfb-draw-mmap-gtt
+kms_frontbuffer_tracking@drrs-2p-primscrn-cur-indfb-draw-mmap-wc
+kms_frontbuffer_tracking@drrs-2p-primscrn-cur-indfb-draw-pwrite
+kms_frontbuffer_tracking@drrs-2p-primscrn-cur-indfb-draw-blt
+kms_frontbuffer_tracking@drrs-2p-primscrn-cur-indfb-draw-render
+kms_frontbuffer_tracking@drrs-2p-primscrn-spr-indfb-draw-mmap-cpu
+kms_frontbuffer_tracking@drrs-2p-primscrn-spr-indfb-draw-mmap-gtt
+kms_frontbuffer_tracking@drrs-2p-primscrn-spr-indfb-draw-mmap-wc
+kms_frontbuffer_tracking@drrs-2p-primscrn-spr-indfb-draw-pwrite
+kms_frontbuffer_tracking@drrs-2p-primscrn-spr-indfb-draw-blt
+kms_frontbuffer_tracking@drrs-2p-primscrn-spr-indfb-draw-render
+kms_frontbuffer_tracking@drrs-2p-scndscrn-pri-indfb-draw-mmap-cpu
+kms_frontbuffer_tracking@drrs-2p-scndscrn-pri-indfb-draw-mmap-gtt
+kms_frontbuffer_tracking@drrs-2p-scndscrn-pri-indfb-draw-mmap-wc
+kms_frontbuffer_tracking@drrs-2p-scndscrn-pri-indfb-draw-pwrite
+kms_frontbuffer_tracking@drrs-2p-scndscrn-pri-indfb-draw-blt
+kms_frontbuffer_tracking@drrs-2p-scndscrn-pri-indfb-draw-render
+kms_frontbuffer_tracking@drrs-2p-scndscrn-pri-shrfb-draw-mmap-cpu
+kms_frontbuffer_tracking@drrs-2p-scndscrn-pri-shrfb-draw-mmap-gtt
+kms_frontbuffer_tracking@drrs-2p-scndscrn-pri-shrfb-draw-mmap-wc
+kms_frontbuffer_tracking@drrs-2p-scndscrn-pri-shrfb-draw-pwrite
+kms_frontbuffer_tracking@drrs-2p-scndscrn-pri-shrfb-draw-blt
+kms_frontbuffer_tracking@drrs-2p-scndscrn-pri-shrfb-draw-render
+kms_frontbuffer_tracking@drrs-2p-scndscrn-cur-indfb-draw-mmap-cpu
+kms_frontbuffer_tracking@drrs-2p-scndscrn-cur-indfb-draw-mmap-gtt
+kms_frontbuffer_tracking@drrs-2p-scndscrn-cur-indfb-draw-mmap-wc
+kms_frontbuffer_tracking@drrs-2p-scndscrn-cur-indfb-draw-pwrite
+kms_frontbuffer_tracking@drrs-2p-scndscrn-cur-indfb-draw-blt
+kms_frontbuffer_tracking@drrs-2p-scndscrn-cur-indfb-draw-render
+kms_frontbuffer_tracking@drrs-2p-scndscrn-spr-indfb-draw-mmap-cpu
+kms_frontbuffer_tracking@drrs-2p-scndscrn-spr-indfb-draw-mmap-gtt
+kms_frontbuffer_tracking@drrs-2p-scndscrn-spr-indfb-draw-mmap-wc
+kms_frontbuffer_tracking@drrs-2p-scndscrn-spr-indfb-draw-pwrite
+kms_frontbuffer_tracking@drrs-2p-scndscrn-spr-indfb-draw-blt
+kms_frontbuffer_tracking@drrs-2p-scndscrn-spr-indfb-draw-render
+kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-pri-indfb-draw-mmap-cpu
+kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-pri-indfb-draw-mmap-gtt
+kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-pri-indfb-draw-mmap-wc
+kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-pri-indfb-draw-pwrite
+kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-pri-indfb-draw-blt
+kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-pri-indfb-draw-render
+kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-pri-shrfb-draw-mmap-cpu
+kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-pri-shrfb-draw-mmap-gtt
+kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-pri-shrfb-draw-mmap-wc
+kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-pri-shrfb-draw-pwrite
+kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-pri-shrfb-draw-blt
+kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-pri-shrfb-draw-render
+kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-cur-indfb-draw-mmap-cpu
+kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-cur-indfb-draw-mmap-gtt
+kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-cur-indfb-draw-mmap-wc
+kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-cur-indfb-draw-pwrite
+kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-cur-indfb-draw-blt
+kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-cur-indfb-draw-render
+kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-spr-indfb-draw-mmap-cpu
+kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-spr-indfb-draw-mmap-gtt
+kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-spr-indfb-draw-mmap-wc
+kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-spr-indfb-draw-pwrite
+kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-spr-indfb-draw-blt
+kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-spr-indfb-draw-render
+kms_frontbuffer_tracking@fbcdrrs-1p-offscren-pri-indfb-draw-mmap-cpu
+kms_frontbuffer_tracking@fbcdrrs-1p-offscren-pri-indfb-draw-mmap-gtt
+kms_frontbuffer_tracking@fbcdrrs-1p-offscren-pri-indfb-draw-mmap-wc
+kms_frontbuffer_tracking@fbcdrrs-1p-offscren-pri-indfb-draw-pwrite
+kms_frontbuffer_tracking@fbcdrrs-1p-offscren-pri-indfb-draw-blt
+kms_frontbuffer_tracking@fbcdrrs-1p-offscren-pri-indfb-draw-render
+kms_frontbuffer_tracking@fbcdrrs-1p-offscren-pri-shrfb-draw-mmap-cpu
+kms_frontbuffer_tracking@fbcdrrs-1p-offscren-pri-shrfb-draw-mmap-gtt
+kms_frontbuffer_tracking@fbcdrrs-1p-offscren-pri-shrfb-draw-mmap-wc
+kms_frontbuffer_tracking@fbcdrrs-1p-offscren-pri-shrfb-draw-pwrite
+kms_frontbuffer_tracking@fbcdrrs-1p-offscren-pri-shrfb-draw-blt
+kms_frontbuffer_tracking@fbcdrrs-1p-offscren-pri-shrfb-draw-render
+kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-pri-indfb-draw-mmap-cpu
+kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-pri-indfb-draw-mmap-gtt
+kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-pri-indfb-draw-mmap-wc
+kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-pri-indfb-draw-pwrite
+kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-pri-indfb-draw-blt
+kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-pri-indfb-draw-render
+kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-pri-shrfb-draw-mmap-cpu
+kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-pri-shrfb-draw-mmap-gtt
+kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-pri-shrfb-draw-mmap-wc
+kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-pri-shrfb-draw-pwrite
+kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-pri-shrfb-draw-blt
+kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-pri-shrfb-draw-render
+kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-cur-indfb-draw-mmap-cpu
+kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-cur-indfb-draw-mmap-gtt
+kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-cur-indfb-draw-mmap-wc
+kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-cur-indfb-draw-pwrite
+kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-cur-indfb-draw-blt
+kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-cur-indfb-draw-render
+kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-spr-indfb-draw-mmap-cpu
+kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-spr-indfb-draw-mmap-gtt
+kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-spr-indfb-draw-mmap-wc
+kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-spr-indfb-draw-pwrite
+kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-spr-indfb-draw-blt
+kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-spr-indfb-draw-render
+kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-pri-indfb-draw-mmap-cpu
+kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-pri-indfb-draw-mmap-gtt
+kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-pri-indfb-draw-mmap-wc
+kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-pri-indfb-draw-pwrite
+kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-pri-indfb-draw-blt
+kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-pri-indfb-draw-render
+kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-pri-shrfb-draw-mmap-cpu
+kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-pri-shrfb-draw-mmap-gtt
+kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-pri-shrfb-draw-mmap-wc
+kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-pri-shrfb-draw-pwrite
+kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-pri-shrfb-draw-blt
+kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-pri-shrfb-draw-render
+kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-cur-indfb-draw-mmap-cpu
+kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-cur-indfb-draw-mmap-gtt
+kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-cur-indfb-draw-mmap-wc
+kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-cur-indfb-draw-pwrite
+kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-cur-indfb-draw-blt
+kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-cur-indfb-draw-render
+kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-spr-indfb-draw-mmap-cpu
+kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-spr-indfb-draw-mmap-gtt
+kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-spr-indfb-draw-mmap-wc
+kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-spr-indfb-draw-pwrite
+kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-spr-indfb-draw-blt
+kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-spr-indfb-draw-render
+kms_frontbuffer_tracking@psrdrrs-1p-primscrn-pri-indfb-draw-mmap-cpu
+kms_frontbuffer_tracking@psrdrrs-1p-primscrn-pri-indfb-draw-mmap-gtt
+kms_frontbuffer_tracking@psrdrrs-1p-primscrn-pri-indfb-draw-mmap-wc
+kms_frontbuffer_tracking@psrdrrs-1p-primscrn-pri-indfb-draw-pwrite
+kms_frontbuffer_tracking@psrdrrs-1p-primscrn-pri-indfb-draw-blt
+kms_frontbuffer_tracking@psrdrrs-1p-primscrn-pri-indfb-draw-render
+kms_frontbuffer_tracking@psrdrrs-1p-primscrn-pri-shrfb-draw-mmap-cpu
+kms_frontbuffer_tracking@psrdrrs-1p-primscrn-pri-shrfb-draw-mmap-gtt
+kms_frontbuffer_tracking@psrdrrs-1p-primscrn-pri-shrfb-draw-mmap-wc
+kms_frontbuffer_tracking@psrdrrs-1p-primscrn-pri-shrfb-draw-pwrite
+kms_frontbuffer_tracking@psrdrrs-1p-primscrn-pri-shrfb-draw-blt
+kms_frontbuffer_tracking@psrdrrs-1p-primscrn-pri-shrfb-draw-render
+kms_frontbuffer_tracking@psrdrrs-1p-primscrn-cur-indfb-draw-mmap-cpu
+kms_frontbuffer_tracking@psrdrrs-1p-primscrn-cur-indfb-draw-mmap-gtt
+kms_frontbuffer_tracking@psrdrrs-1p-primscrn-cur-indfb-draw-mmap-wc
+kms_frontbuffer_tracking@psrdrrs-1p-primscrn-cur-indfb-draw-pwrite
+kms_frontbuffer_tracking@psrdrrs-1p-primscrn-cur-indfb-draw-blt
+kms_frontbuffer_tracking@psrdrrs-1p-primscrn-cur-indfb-draw-render
+kms_frontbuffer_tracking@psrdrrs-1p-primscrn-spr-indfb-draw-mmap-cpu
+kms_frontbuffer_tracking@psrdrrs-1p-primscrn-spr-indfb-draw-mmap-gtt
+kms_frontbuffer_tracking@psrdrrs-1p-primscrn-spr-indfb-draw-mmap-wc
+kms_frontbuffer_tracking@psrdrrs-1p-primscrn-spr-indfb-draw-pwrite
+kms_frontbuffer_tracking@psrdrrs-1p-primscrn-spr-indfb-draw-blt
+kms_frontbuffer_tracking@psrdrrs-1p-primscrn-spr-indfb-draw-render
+kms_frontbuffer_tracking@psrdrrs-1p-offscren-pri-indfb-draw-mmap-cpu
+kms_frontbuffer_tracking@psrdrrs-1p-offscren-pri-indfb-draw-mmap-gtt
+kms_frontbuffer_tracking@psrdrrs-1p-offscren-pri-indfb-draw-mmap-wc
+kms_frontbuffer_tracking@psrdrrs-1p-offscren-pri-indfb-draw-pwrite
+kms_frontbuffer_tracking@psrdrrs-1p-offscren-pri-indfb-draw-blt
+kms_frontbuffer_tracking@psrdrrs-1p-offscren-pri-indfb-draw-render
+kms_frontbuffer_tracking@psrdrrs-1p-offscren-pri-shrfb-draw-mmap-cpu
+kms_frontbuffer_tracking@psrdrrs-1p-offscren-pri-shrfb-draw-mmap-gtt
+kms_frontbuffer_tracking@psrdrrs-1p-offscren-pri-shrfb-draw-mmap-wc
+kms_frontbuffer_tracking@psrdrrs-1p-offscren-pri-shrfb-draw-pwrite
+kms_frontbuffer_tracking@psrdrrs-1p-offscren-pri-shrfb-draw-blt
+kms_frontbuffer_tracking@psrdrrs-1p-offscren-pri-shrfb-draw-render
+kms_frontbuffer_tracking@psrdrrs-2p-primscrn-pri-indfb-draw-mmap-cpu
+kms_frontbuffer_tracking@psrdrrs-2p-primscrn-pri-indfb-draw-mmap-gtt
+kms_frontbuffer_tracking@psrdrrs-2p-primscrn-pri-indfb-draw-mmap-wc
+kms_frontbuffer_tracking@psrdrrs-2p-primscrn-pri-indfb-draw-pwrite
+kms_frontbuffer_tracking@psrdrrs-2p-primscrn-pri-indfb-draw-blt
+kms_frontbuffer_tracking@psrdrrs-2p-primscrn-pri-indfb-draw-render
+kms_frontbuffer_tracking@psrdrrs-2p-primscrn-pri-shrfb-draw-mmap-cpu
+kms_frontbuffer_tracking@psrdrrs-2p-primscrn-pri-shrfb-draw-mmap-gtt
+kms_frontbuffer_tracking@psrdrrs-2p-primscrn-pri-shrfb-draw-mmap-wc
+kms_frontbuffer_tracking@psrdrrs-2p-primscrn-pri-shrfb-draw-pwrite
+kms_frontbuffer_tracking@psrdrrs-2p-primscrn-pri-shrfb-draw-blt
+kms_frontbuffer_tracking@psrdrrs-2p-primscrn-pri-shrfb-draw-render
+kms_frontbuffer_tracking@psrdrrs-2p-primscrn-cur-indfb-draw-mmap-cpu
+kms_frontbuffer_tracking@psrdrrs-2p-primscrn-cur-indfb-draw-mmap-gtt
+kms_frontbuffer_tracking@psrdrrs-2p-primscrn-cur-indfb-draw-mmap-wc
+kms_frontbuffer_tracking@psrdrrs-2p-primscrn-cur-indfb-draw-pwrite
+kms_frontbuffer_tracking@psrdrrs-2p-primscrn-cur-indfb-draw-blt
+kms_frontbuffer_tracking@psrdrrs-2p-primscrn-cur-indfb-draw-render
+kms_frontbuffer_tracking@psrdrrs-2p-primscrn-spr-indfb-draw-mmap-cpu
+kms_frontbuffer_tracking@psrdrrs-2p-primscrn-spr-indfb-draw-mmap-gtt
+kms_frontbuffer_tracking@psrdrrs-2p-primscrn-spr-indfb-draw-mmap-wc
+kms_frontbuffer_tracking@psrdrrs-2p-primscrn-spr-indfb-draw-pwrite
+kms_frontbuffer_tracking@psrdrrs-2p-primscrn-spr-indfb-draw-blt
+kms_frontbuffer_tracking@psrdrrs-2p-primscrn-spr-indfb-draw-render
+kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-pri-indfb-draw-mmap-cpu
+kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-pri-indfb-draw-mmap-gtt
+kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-pri-indfb-draw-mmap-wc
+kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-pri-indfb-draw-pwrite
+kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-pri-indfb-draw-blt
+kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-pri-indfb-draw-render
+kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-pri-shrfb-draw-mmap-cpu
+kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-pri-shrfb-draw-mmap-gtt
+kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-pri-shrfb-draw-mmap-wc
+kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-pri-shrfb-draw-pwrite
+kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-pri-shrfb-draw-blt
+kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-pri-shrfb-draw-render
+kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-cur-indfb-draw-mmap-cpu
+kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-cur-indfb-draw-mmap-gtt
+kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-cur-indfb-draw-mmap-wc
+kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-cur-indfb-draw-pwrite
+kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-cur-indfb-draw-blt
+kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-cur-indfb-draw-render
+kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-spr-indfb-draw-mmap-cpu
+kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-spr-indfb-draw-mmap-gtt
+kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-spr-indfb-draw-mmap-wc
+kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-spr-indfb-draw-pwrite
+kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-spr-indfb-draw-blt
+kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-spr-indfb-draw-render
+kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-pri-indfb-draw-mmap-cpu
+kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-pri-indfb-draw-mmap-gtt
+kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-pri-indfb-draw-mmap-wc
+kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-pri-indfb-draw-pwrite
+kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-pri-indfb-draw-blt
+kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-pri-indfb-draw-render
+kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-pri-shrfb-draw-mmap-cpu
+kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-pri-shrfb-draw-mmap-gtt
+kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-pri-shrfb-draw-mmap-wc
+kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-pri-shrfb-draw-pwrite
+kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-pri-shrfb-draw-blt
+kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-pri-shrfb-draw-render
+kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-cur-indfb-draw-mmap-cpu
+kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-cur-indfb-draw-mmap-gtt
+kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-cur-indfb-draw-mmap-wc
+kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-cur-indfb-draw-pwrite
+kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-cur-indfb-draw-blt
+kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-cur-indfb-draw-render
+kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-spr-indfb-draw-mmap-cpu
+kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-spr-indfb-draw-mmap-gtt
+kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-spr-indfb-draw-mmap-wc
+kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-spr-indfb-draw-pwrite
+kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-spr-indfb-draw-blt
+kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-spr-indfb-draw-render
+kms_frontbuffer_tracking@fbcpsrdrrs-1p-offscren-pri-indfb-draw-mmap-cpu
+kms_frontbuffer_tracking@fbcpsrdrrs-1p-offscren-pri-indfb-draw-mmap-gtt
+kms_frontbuffer_tracking@fbcpsrdrrs-1p-offscren-pri-indfb-draw-mmap-wc
+kms_frontbuffer_tracking@fbcpsrdrrs-1p-offscren-pri-indfb-draw-pwrite
+kms_frontbuffer_tracking@fbcpsrdrrs-1p-offscren-pri-indfb-draw-blt
+kms_frontbuffer_tracking@fbcpsrdrrs-1p-offscren-pri-indfb-draw-render
+kms_frontbuffer_tracking@fbcpsrdrrs-1p-offscren-pri-shrfb-draw-mmap-cpu
+kms_frontbuffer_tracking@fbcpsrdrrs-1p-offscren-pri-shrfb-draw-mmap-gtt
+kms_frontbuffer_tracking@fbcpsrdrrs-1p-offscren-pri-shrfb-draw-mmap-wc
+kms_frontbuffer_tracking@fbcpsrdrrs-1p-offscren-pri-shrfb-draw-pwrite
+kms_frontbuffer_tracking@fbcpsrdrrs-1p-offscren-pri-shrfb-draw-blt
+kms_frontbuffer_tracking@fbcpsrdrrs-1p-offscren-pri-shrfb-draw-render
+kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-pri-indfb-draw-mmap-cpu
+kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-pri-indfb-draw-mmap-gtt
+kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-pri-indfb-draw-mmap-wc
+kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-pri-indfb-draw-pwrite
+kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-pri-indfb-draw-blt
+kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-pri-indfb-draw-render
+kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-pri-shrfb-draw-mmap-cpu
+kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-pri-shrfb-draw-mmap-gtt
+kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-pri-shrfb-draw-mmap-wc
+kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-pri-shrfb-draw-pwrite
+kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-pri-shrfb-draw-blt
+kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-pri-shrfb-draw-render
+kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-cur-indfb-draw-mmap-cpu
+kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-cur-indfb-draw-mmap-gtt
+kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-cur-indfb-draw-mmap-wc
+kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-cur-indfb-draw-pwrite
+kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-cur-indfb-draw-blt
+kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-cur-indfb-draw-render
+kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-spr-indfb-draw-mmap-cpu
+kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-spr-indfb-draw-mmap-gtt
+kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-spr-indfb-draw-mmap-wc
+kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-spr-indfb-draw-pwrite
+kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-spr-indfb-draw-blt
+kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-spr-indfb-draw-render
+kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-pri-indfb-draw-mmap-cpu
+kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-pri-indfb-draw-mmap-gtt
+kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-pri-indfb-draw-mmap-wc
+kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-pri-indfb-draw-pwrite
+kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-pri-indfb-draw-blt
+kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-pri-indfb-draw-render
+kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-pri-shrfb-draw-mmap-cpu
+kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-pri-shrfb-draw-mmap-gtt
+kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-pri-shrfb-draw-mmap-wc
+kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-pri-shrfb-draw-pwrite
+kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-pri-shrfb-draw-blt
+kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-pri-shrfb-draw-render
+kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-cur-indfb-draw-mmap-cpu
+kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-cur-indfb-draw-mmap-gtt
+kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-cur-indfb-draw-mmap-wc
+kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-cur-indfb-draw-pwrite
+kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-cur-indfb-draw-blt
+kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-cur-indfb-draw-render
+kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-spr-indfb-draw-mmap-cpu
+kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-spr-indfb-draw-mmap-gtt
+kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-spr-indfb-draw-mmap-wc
+kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-spr-indfb-draw-pwrite
+kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-spr-indfb-draw-blt
+kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-spr-indfb-draw-render
+kms_frontbuffer_tracking@fbc-1p-primscrn-indfb-pgflip-blt
+kms_frontbuffer_tracking@fbc-1p-primscrn-indfb-msflip-blt
+kms_frontbuffer_tracking@fbc-1p-primscrn-indfb-plflip-blt
+kms_frontbuffer_tracking@fbc-1p-primscrn-shrfb-pgflip-blt
+kms_frontbuffer_tracking@fbc-1p-primscrn-shrfb-msflip-blt
+kms_frontbuffer_tracking@fbc-1p-primscrn-shrfb-plflip-blt
+kms_frontbuffer_tracking@fbc-2p-primscrn-indfb-pgflip-blt
+kms_frontbuffer_tracking@fbc-2p-primscrn-indfb-msflip-blt
+kms_frontbuffer_tracking@fbc-2p-primscrn-indfb-plflip-blt
+kms_frontbuffer_tracking@fbc-2p-primscrn-shrfb-pgflip-blt
+kms_frontbuffer_tracking@fbc-2p-primscrn-shrfb-msflip-blt
+kms_frontbuffer_tracking@fbc-2p-primscrn-shrfb-plflip-blt
+kms_frontbuffer_tracking@fbc-2p-scndscrn-indfb-pgflip-blt
+kms_frontbuffer_tracking@fbc-2p-scndscrn-indfb-msflip-blt
+kms_frontbuffer_tracking@fbc-2p-scndscrn-indfb-plflip-blt
+kms_frontbuffer_tracking@fbc-2p-scndscrn-shrfb-pgflip-blt
+kms_frontbuffer_tracking@fbc-2p-scndscrn-shrfb-msflip-blt
+kms_frontbuffer_tracking@fbc-2p-scndscrn-shrfb-plflip-blt
+kms_frontbuffer_tracking@psr-1p-primscrn-indfb-pgflip-blt
+kms_frontbuffer_tracking@psr-1p-primscrn-indfb-msflip-blt
+kms_frontbuffer_tracking@psr-1p-primscrn-indfb-plflip-blt
+kms_frontbuffer_tracking@psr-1p-primscrn-shrfb-pgflip-blt
+kms_frontbuffer_tracking@psr-1p-primscrn-shrfb-msflip-blt
+kms_frontbuffer_tracking@psr-1p-primscrn-shrfb-plflip-blt
+kms_frontbuffer_tracking@psr-2p-primscrn-indfb-pgflip-blt
+kms_frontbuffer_tracking@psr-2p-primscrn-indfb-msflip-blt
+kms_frontbuffer_tracking@psr-2p-primscrn-indfb-plflip-blt
+kms_frontbuffer_tracking@psr-2p-primscrn-shrfb-pgflip-blt
+kms_frontbuffer_tracking@psr-2p-primscrn-shrfb-msflip-blt
+kms_frontbuffer_tracking@psr-2p-primscrn-shrfb-plflip-blt
+kms_frontbuffer_tracking@psr-2p-scndscrn-indfb-pgflip-blt
+kms_frontbuffer_tracking@psr-2p-scndscrn-indfb-msflip-blt
+kms_frontbuffer_tracking@psr-2p-scndscrn-indfb-plflip-blt
+kms_frontbuffer_tracking@psr-2p-scndscrn-shrfb-pgflip-blt
+kms_frontbuffer_tracking@psr-2p-scndscrn-shrfb-msflip-blt
+kms_frontbuffer_tracking@psr-2p-scndscrn-shrfb-plflip-blt
+kms_frontbuffer_tracking@fbcpsr-1p-primscrn-indfb-pgflip-blt
+kms_frontbuffer_tracking@fbcpsr-1p-primscrn-indfb-msflip-blt
+kms_frontbuffer_tracking@fbcpsr-1p-primscrn-indfb-plflip-blt
+kms_frontbuffer_tracking@fbcpsr-1p-primscrn-shrfb-pgflip-blt
+kms_frontbuffer_tracking@fbcpsr-1p-primscrn-shrfb-msflip-blt
+kms_frontbuffer_tracking@fbcpsr-1p-primscrn-shrfb-plflip-blt
+kms_frontbuffer_tracking@fbcpsr-2p-primscrn-indfb-pgflip-blt
+kms_frontbuffer_tracking@fbcpsr-2p-primscrn-indfb-msflip-blt
+kms_frontbuffer_tracking@fbcpsr-2p-primscrn-indfb-plflip-blt
+kms_frontbuffer_tracking@fbcpsr-2p-primscrn-shrfb-pgflip-blt
+kms_frontbuffer_tracking@fbcpsr-2p-primscrn-shrfb-msflip-blt
+kms_frontbuffer_tracking@fbcpsr-2p-primscrn-shrfb-plflip-blt
+kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-indfb-pgflip-blt
+kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-indfb-msflip-blt
+kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-indfb-plflip-blt
+kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-shrfb-pgflip-blt
+kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-shrfb-msflip-blt
+kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-shrfb-plflip-blt
+kms_frontbuffer_tracking@drrs-1p-primscrn-indfb-pgflip-blt
+kms_frontbuffer_tracking@drrs-1p-primscrn-indfb-msflip-blt
+kms_frontbuffer_tracking@drrs-1p-primscrn-indfb-plflip-blt
+kms_frontbuffer_tracking@drrs-1p-primscrn-shrfb-pgflip-blt
+kms_frontbuffer_tracking@drrs-1p-primscrn-shrfb-msflip-blt
+kms_frontbuffer_tracking@drrs-1p-primscrn-shrfb-plflip-blt
+kms_frontbuffer_tracking@drrs-2p-primscrn-indfb-pgflip-blt
+kms_frontbuffer_tracking@drrs-2p-primscrn-indfb-msflip-blt
+kms_frontbuffer_tracking@drrs-2p-primscrn-indfb-plflip-blt
+kms_frontbuffer_tracking@drrs-2p-primscrn-shrfb-pgflip-blt
+kms_frontbuffer_tracking@drrs-2p-primscrn-shrfb-msflip-blt
+kms_frontbuffer_tracking@drrs-2p-primscrn-shrfb-plflip-blt
+kms_frontbuffer_tracking@drrs-2p-scndscrn-indfb-pgflip-blt
+kms_frontbuffer_tracking@drrs-2p-scndscrn-indfb-msflip-blt
+kms_frontbuffer_tracking@drrs-2p-scndscrn-indfb-plflip-blt
+kms_frontbuffer_tracking@drrs-2p-scndscrn-shrfb-pgflip-blt
+kms_frontbuffer_tracking@drrs-2p-scndscrn-shrfb-msflip-blt
+kms_frontbuffer_tracking@drrs-2p-scndscrn-shrfb-plflip-blt
+kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-indfb-pgflip-blt
+kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-indfb-msflip-blt
+kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-indfb-plflip-blt
+kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-shrfb-pgflip-blt
+kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-shrfb-msflip-blt
+kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-shrfb-plflip-blt
+kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-indfb-pgflip-blt
+kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-indfb-msflip-blt
+kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-indfb-plflip-blt
+kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-shrfb-pgflip-blt
+kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-shrfb-msflip-blt
+kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-shrfb-plflip-blt
+kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-indfb-pgflip-blt
+kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-indfb-msflip-blt
+kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-indfb-plflip-blt
+kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-shrfb-pgflip-blt
+kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-shrfb-msflip-blt
+kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-shrfb-plflip-blt
+kms_frontbuffer_tracking@psrdrrs-1p-primscrn-indfb-pgflip-blt
+kms_frontbuffer_tracking@psrdrrs-1p-primscrn-indfb-msflip-blt
+kms_frontbuffer_tracking@psrdrrs-1p-primscrn-indfb-plflip-blt
+kms_frontbuffer_tracking@psrdrrs-1p-primscrn-shrfb-pgflip-blt
+kms_frontbuffer_tracking@psrdrrs-1p-primscrn-shrfb-msflip-blt
+kms_frontbuffer_tracking@psrdrrs-1p-primscrn-shrfb-plflip-blt
+kms_frontbuffer_tracking@psrdrrs-2p-primscrn-indfb-pgflip-blt
+kms_frontbuffer_tracking@psrdrrs-2p-primscrn-indfb-msflip-blt
+kms_frontbuffer_tracking@psrdrrs-2p-primscrn-indfb-plflip-blt
+kms_frontbuffer_tracking@psrdrrs-2p-primscrn-shrfb-pgflip-blt
+kms_frontbuffer_tracking@psrdrrs-2p-primscrn-shrfb-msflip-blt
+kms_frontbuffer_tracking@psrdrrs-2p-primscrn-shrfb-plflip-blt
+kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-indfb-pgflip-blt
+kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-indfb-msflip-blt
+kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-indfb-plflip-blt
+kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-shrfb-pgflip-blt
+kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-shrfb-msflip-blt
+kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-shrfb-plflip-blt
+kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-indfb-pgflip-blt
+kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-indfb-msflip-blt
+kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-indfb-plflip-blt
+kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-shrfb-pgflip-blt
+kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-shrfb-msflip-blt
+kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-shrfb-plflip-blt
+kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-indfb-pgflip-blt
+kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-indfb-msflip-blt
+kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-indfb-plflip-blt
+kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-shrfb-pgflip-blt
+kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-shrfb-msflip-blt
+kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-shrfb-plflip-blt
+kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-indfb-pgflip-blt
+kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-indfb-msflip-blt
+kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-indfb-plflip-blt
+kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-shrfb-pgflip-blt
+kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-shrfb-msflip-blt
+kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-shrfb-plflip-blt
+kms_frontbuffer_tracking@fbc-1p-indfb-fliptrack-mmap-gtt
+kms_frontbuffer_tracking@fbc-1p-shrfb-fliptrack-mmap-gtt
+kms_frontbuffer_tracking@fbc-2p-indfb-fliptrack-mmap-gtt
+kms_frontbuffer_tracking@fbc-2p-shrfb-fliptrack-mmap-gtt
+kms_frontbuffer_tracking@fbcpsr-1p-indfb-fliptrack-mmap-gtt
+kms_frontbuffer_tracking@fbcpsr-1p-shrfb-fliptrack-mmap-gtt
+kms_frontbuffer_tracking@fbcpsr-2p-indfb-fliptrack-mmap-gtt
+kms_frontbuffer_tracking@fbcpsr-2p-shrfb-fliptrack-mmap-gtt
+kms_frontbuffer_tracking@fbcdrrs-1p-indfb-fliptrack-mmap-gtt
+kms_frontbuffer_tracking@fbcdrrs-1p-shrfb-fliptrack-mmap-gtt
+kms_frontbuffer_tracking@fbcdrrs-2p-indfb-fliptrack-mmap-gtt
+kms_frontbuffer_tracking@fbcdrrs-2p-shrfb-fliptrack-mmap-gtt
+kms_frontbuffer_tracking@fbcpsrdrrs-1p-indfb-fliptrack-mmap-gtt
+kms_frontbuffer_tracking@fbcpsrdrrs-1p-shrfb-fliptrack-mmap-gtt
+kms_frontbuffer_tracking@fbcpsrdrrs-2p-indfb-fliptrack-mmap-gtt
+kms_frontbuffer_tracking@fbcpsrdrrs-2p-shrfb-fliptrack-mmap-gtt
+kms_frontbuffer_tracking@fbc-1p-primscrn-cur-indfb-move
+kms_frontbuffer_tracking@fbc-1p-primscrn-cur-indfb-onoff
+kms_frontbuffer_tracking@fbc-1p-primscrn-spr-indfb-move
+kms_frontbuffer_tracking@fbc-1p-primscrn-spr-indfb-onoff
+kms_frontbuffer_tracking@fbc-2p-primscrn-cur-indfb-move
+kms_frontbuffer_tracking@fbc-2p-primscrn-cur-indfb-onoff
+kms_frontbuffer_tracking@fbc-2p-primscrn-spr-indfb-move
+kms_frontbuffer_tracking@fbc-2p-primscrn-spr-indfb-onoff
+kms_frontbuffer_tracking@fbc-2p-scndscrn-cur-indfb-move
+kms_frontbuffer_tracking@fbc-2p-scndscrn-cur-indfb-onoff
+kms_frontbuffer_tracking@fbc-2p-scndscrn-spr-indfb-move
+kms_frontbuffer_tracking@fbc-2p-scndscrn-spr-indfb-onoff
+kms_frontbuffer_tracking@psr-1p-primscrn-cur-indfb-move
+kms_frontbuffer_tracking@psr-1p-primscrn-cur-indfb-onoff
+kms_frontbuffer_tracking@psr-1p-primscrn-spr-indfb-move
+kms_frontbuffer_tracking@psr-1p-primscrn-spr-indfb-onoff
+kms_frontbuffer_tracking@psr-2p-primscrn-cur-indfb-move
+kms_frontbuffer_tracking@psr-2p-primscrn-cur-indfb-onoff
+kms_frontbuffer_tracking@psr-2p-primscrn-spr-indfb-move
+kms_frontbuffer_tracking@psr-2p-primscrn-spr-indfb-onoff
+kms_frontbuffer_tracking@psr-2p-scndscrn-cur-indfb-move
+kms_frontbuffer_tracking@psr-2p-scndscrn-cur-indfb-onoff
+kms_frontbuffer_tracking@psr-2p-scndscrn-spr-indfb-move
+kms_frontbuffer_tracking@psr-2p-scndscrn-spr-indfb-onoff
+kms_frontbuffer_tracking@fbcpsr-1p-primscrn-cur-indfb-move
+kms_frontbuffer_tracking@fbcpsr-1p-primscrn-cur-indfb-onoff
+kms_frontbuffer_tracking@fbcpsr-1p-primscrn-spr-indfb-move
+kms_frontbuffer_tracking@fbcpsr-1p-primscrn-spr-indfb-onoff
+kms_frontbuffer_tracking@fbcpsr-2p-primscrn-cur-indfb-move
+kms_frontbuffer_tracking@fbcpsr-2p-primscrn-cur-indfb-onoff
+kms_frontbuffer_tracking@fbcpsr-2p-primscrn-spr-indfb-move
+kms_frontbuffer_tracking@fbcpsr-2p-primscrn-spr-indfb-onoff
+kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-cur-indfb-move
+kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-cur-indfb-onoff
+kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-spr-indfb-move
+kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-spr-indfb-onoff
+kms_frontbuffer_tracking@drrs-1p-primscrn-cur-indfb-move
+kms_frontbuffer_tracking@drrs-1p-primscrn-cur-indfb-onoff
+kms_frontbuffer_tracking@drrs-1p-primscrn-spr-indfb-move
+kms_frontbuffer_tracking@drrs-1p-primscrn-spr-indfb-onoff
+kms_frontbuffer_tracking@drrs-2p-primscrn-cur-indfb-move
+kms_frontbuffer_tracking@drrs-2p-primscrn-cur-indfb-onoff
+kms_frontbuffer_tracking@drrs-2p-primscrn-spr-indfb-move
+kms_frontbuffer_tracking@drrs-2p-primscrn-spr-indfb-onoff
+kms_frontbuffer_tracking@drrs-2p-scndscrn-cur-indfb-move
+kms_frontbuffer_tracking@drrs-2p-scndscrn-cur-indfb-onoff
+kms_frontbuffer_tracking@drrs-2p-scndscrn-spr-indfb-move
+kms_frontbuffer_tracking@drrs-2p-scndscrn-spr-indfb-onoff
+kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-cur-indfb-move
+kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-cur-indfb-onoff
+kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-spr-indfb-move
+kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-spr-indfb-onoff
+kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-cur-indfb-move
+kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-cur-indfb-onoff
+kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-spr-indfb-move
+kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-spr-indfb-onoff
+kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-cur-indfb-move
+kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-cur-indfb-onoff
+kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-spr-indfb-move
+kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-spr-indfb-onoff
+kms_frontbuffer_tracking@psrdrrs-1p-primscrn-cur-indfb-move
+kms_frontbuffer_tracking@psrdrrs-1p-primscrn-cur-indfb-onoff
+kms_frontbuffer_tracking@psrdrrs-1p-primscrn-spr-indfb-move
+kms_frontbuffer_tracking@psrdrrs-1p-primscrn-spr-indfb-onoff
+kms_frontbuffer_tracking@psrdrrs-2p-primscrn-cur-indfb-move
+kms_frontbuffer_tracking@psrdrrs-2p-primscrn-cur-indfb-onoff
+kms_frontbuffer_tracking@psrdrrs-2p-primscrn-spr-indfb-move
+kms_frontbuffer_tracking@psrdrrs-2p-primscrn-spr-indfb-onoff
+kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-cur-indfb-move
+kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-cur-indfb-onoff
+kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-spr-indfb-move
+kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-spr-indfb-onoff
+kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-cur-indfb-move
+kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-cur-indfb-onoff
+kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-spr-indfb-move
+kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-spr-indfb-onoff
+kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-cur-indfb-move
+kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-cur-indfb-onoff
+kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-spr-indfb-move
+kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-spr-indfb-onoff
+kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-cur-indfb-move
+kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-cur-indfb-onoff
+kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-spr-indfb-move
+kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-spr-indfb-onoff
+kms_frontbuffer_tracking@fbc-1p-primscrn-spr-indfb-fullscreen
+kms_frontbuffer_tracking@fbc-2p-primscrn-spr-indfb-fullscreen
+kms_frontbuffer_tracking@fbc-2p-scndscrn-spr-indfb-fullscreen
+kms_frontbuffer_tracking@psr-1p-primscrn-spr-indfb-fullscreen
+kms_frontbuffer_tracking@psr-2p-primscrn-spr-indfb-fullscreen
+kms_frontbuffer_tracking@psr-2p-scndscrn-spr-indfb-fullscreen
+kms_frontbuffer_tracking@fbcpsr-1p-primscrn-spr-indfb-fullscreen
+kms_frontbuffer_tracking@fbcpsr-2p-primscrn-spr-indfb-fullscreen
+kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-spr-indfb-fullscreen
+kms_frontbuffer_tracking@drrs-1p-primscrn-spr-indfb-fullscreen
+kms_frontbuffer_tracking@drrs-2p-primscrn-spr-indfb-fullscreen
+kms_frontbuffer_tracking@drrs-2p-scndscrn-spr-indfb-fullscreen
+kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-spr-indfb-fullscreen
+kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-spr-indfb-fullscreen
+kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-spr-indfb-fullscreen
+kms_frontbuffer_tracking@psrdrrs-1p-primscrn-spr-indfb-fullscreen
+kms_frontbuffer_tracking@psrdrrs-2p-primscrn-spr-indfb-fullscreen
+kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-spr-indfb-fullscreen
+kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-spr-indfb-fullscreen
+kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-spr-indfb-fullscreen
+kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-spr-indfb-fullscreen
+kms_frontbuffer_tracking@fbc-1p-pri-indfb-multidraw
+kms_frontbuffer_tracking@fbc-2p-pri-indfb-multidraw
+kms_frontbuffer_tracking@psr-1p-pri-indfb-multidraw
+kms_frontbuffer_tracking@psr-2p-pri-indfb-multidraw
+kms_frontbuffer_tracking@fbcpsr-1p-pri-indfb-multidraw
+kms_frontbuffer_tracking@fbcpsr-2p-pri-indfb-multidraw
+kms_frontbuffer_tracking@drrs-1p-pri-indfb-multidraw
+kms_frontbuffer_tracking@drrs-2p-pri-indfb-multidraw
+kms_frontbuffer_tracking@fbcdrrs-1p-pri-indfb-multidraw
+kms_frontbuffer_tracking@fbcdrrs-2p-pri-indfb-multidraw
+kms_frontbuffer_tracking@psrdrrs-1p-pri-indfb-multidraw
+kms_frontbuffer_tracking@psrdrrs-2p-pri-indfb-multidraw
+kms_frontbuffer_tracking@fbcpsrdrrs-1p-pri-indfb-multidraw
+kms_frontbuffer_tracking@fbcpsrdrrs-2p-pri-indfb-multidraw
+kms_frontbuffer_tracking@fbc-farfromfence-mmap-gtt
+kms_frontbuffer_tracking@psr-farfromfence-mmap-gtt
+kms_frontbuffer_tracking@fbcpsr-farfromfence-mmap-gtt
+kms_frontbuffer_tracking@drrs-farfromfence-mmap-gtt
+kms_frontbuffer_tracking@fbcdrrs-farfromfence-mmap-gtt
+kms_frontbuffer_tracking@psrdrrs-farfromfence-mmap-gtt
+kms_frontbuffer_tracking@fbcpsrdrrs-farfromfence-mmap-gtt
+kms_frontbuffer_tracking@fbc-rgb565-draw-mmap-cpu
+kms_frontbuffer_tracking@fbc-rgb101010-draw-mmap-cpu
+kms_frontbuffer_tracking@fbc-rgb565-draw-mmap-gtt
+kms_frontbuffer_tracking@fbc-rgb101010-draw-mmap-gtt
+kms_frontbuffer_tracking@fbc-rgb565-draw-mmap-wc
+kms_frontbuffer_tracking@fbc-rgb101010-draw-mmap-wc
+kms_frontbuffer_tracking@fbc-rgb565-draw-pwrite
+kms_frontbuffer_tracking@fbc-rgb101010-draw-pwrite
+kms_frontbuffer_tracking@fbc-rgb565-draw-blt
+kms_frontbuffer_tracking@fbc-rgb101010-draw-blt
+kms_frontbuffer_tracking@fbc-rgb565-draw-render
+kms_frontbuffer_tracking@fbc-rgb101010-draw-render
+kms_frontbuffer_tracking@psr-rgb565-draw-mmap-cpu
+kms_frontbuffer_tracking@psr-rgb101010-draw-mmap-cpu
+kms_frontbuffer_tracking@psr-rgb565-draw-mmap-gtt
+kms_frontbuffer_tracking@psr-rgb101010-draw-mmap-gtt
+kms_frontbuffer_tracking@psr-rgb565-draw-mmap-wc
+kms_frontbuffer_tracking@psr-rgb101010-draw-mmap-wc
+kms_frontbuffer_tracking@psr-rgb565-draw-pwrite
+kms_frontbuffer_tracking@psr-rgb101010-draw-pwrite
+kms_frontbuffer_tracking@psr-rgb565-draw-blt
+kms_frontbuffer_tracking@psr-rgb101010-draw-blt
+kms_frontbuffer_tracking@psr-rgb565-draw-render
+kms_frontbuffer_tracking@psr-rgb101010-draw-render
+kms_frontbuffer_tracking@fbcpsr-rgb565-draw-mmap-cpu
+kms_frontbuffer_tracking@fbcpsr-rgb101010-draw-mmap-cpu
+kms_frontbuffer_tracking@fbcpsr-rgb565-draw-mmap-gtt
+kms_frontbuffer_tracking@fbcpsr-rgb101010-draw-mmap-gtt
+kms_frontbuffer_tracking@fbcpsr-rgb565-draw-mmap-wc
+kms_frontbuffer_tracking@fbcpsr-rgb101010-draw-mmap-wc
+kms_frontbuffer_tracking@fbcpsr-rgb565-draw-pwrite
+kms_frontbuffer_tracking@fbcpsr-rgb101010-draw-pwrite
+kms_frontbuffer_tracking@fbcpsr-rgb565-draw-blt
+kms_frontbuffer_tracking@fbcpsr-rgb101010-draw-blt
+kms_frontbuffer_tracking@fbcpsr-rgb565-draw-render
+kms_frontbuffer_tracking@fbcpsr-rgb101010-draw-render
+kms_frontbuffer_tracking@drrs-rgb565-draw-mmap-cpu
+kms_frontbuffer_tracking@drrs-rgb101010-draw-mmap-cpu
+kms_frontbuffer_tracking@drrs-rgb565-draw-mmap-gtt
+kms_frontbuffer_tracking@drrs-rgb101010-draw-mmap-gtt
+kms_frontbuffer_tracking@drrs-rgb565-draw-mmap-wc
+kms_frontbuffer_tracking@drrs-rgb101010-draw-mmap-wc
+kms_frontbuffer_tracking@drrs-rgb565-draw-pwrite
+kms_frontbuffer_tracking@drrs-rgb101010-draw-pwrite
+kms_frontbuffer_tracking@drrs-rgb565-draw-blt
+kms_frontbuffer_tracking@drrs-rgb101010-draw-blt
+kms_frontbuffer_tracking@drrs-rgb565-draw-render
+kms_frontbuffer_tracking@drrs-rgb101010-draw-render
+kms_frontbuffer_tracking@fbcdrrs-rgb565-draw-mmap-cpu
+kms_frontbuffer_tracking@fbcdrrs-rgb101010-draw-mmap-cpu
+kms_frontbuffer_tracking@fbcdrrs-rgb565-draw-mmap-gtt
+kms_frontbuffer_tracking@fbcdrrs-rgb101010-draw-mmap-gtt
+kms_frontbuffer_tracking@fbcdrrs-rgb565-draw-mmap-wc
+kms_frontbuffer_tracking@fbcdrrs-rgb101010-draw-mmap-wc
+kms_frontbuffer_tracking@fbcdrrs-rgb565-draw-pwrite
+kms_frontbuffer_tracking@fbcdrrs-rgb101010-draw-pwrite
+kms_frontbuffer_tracking@fbcdrrs-rgb565-draw-blt
+kms_frontbuffer_tracking@fbcdrrs-rgb101010-draw-blt
+kms_frontbuffer_tracking@fbcdrrs-rgb565-draw-render
+kms_frontbuffer_tracking@fbcdrrs-rgb101010-draw-render
+kms_frontbuffer_tracking@psrdrrs-rgb565-draw-mmap-cpu
+kms_frontbuffer_tracking@psrdrrs-rgb101010-draw-mmap-cpu
+kms_frontbuffer_tracking@psrdrrs-rgb565-draw-mmap-gtt
+kms_frontbuffer_tracking@psrdrrs-rgb101010-draw-mmap-gtt
+kms_frontbuffer_tracking@psrdrrs-rgb565-draw-mmap-wc
+kms_frontbuffer_tracking@psrdrrs-rgb101010-draw-mmap-wc
+kms_frontbuffer_tracking@psrdrrs-rgb565-draw-pwrite
+kms_frontbuffer_tracking@psrdrrs-rgb101010-draw-pwrite
+kms_frontbuffer_tracking@psrdrrs-rgb565-draw-blt
+kms_frontbuffer_tracking@psrdrrs-rgb101010-draw-blt
+kms_frontbuffer_tracking@psrdrrs-rgb565-draw-render
+kms_frontbuffer_tracking@psrdrrs-rgb101010-draw-render
+kms_frontbuffer_tracking@fbcpsrdrrs-rgb565-draw-mmap-cpu
+kms_frontbuffer_tracking@fbcpsrdrrs-rgb101010-draw-mmap-cpu
+kms_frontbuffer_tracking@fbcpsrdrrs-rgb565-draw-mmap-gtt
+kms_frontbuffer_tracking@fbcpsrdrrs-rgb101010-draw-mmap-gtt
+kms_frontbuffer_tracking@fbcpsrdrrs-rgb565-draw-mmap-wc
+kms_frontbuffer_tracking@fbcpsrdrrs-rgb101010-draw-mmap-wc
+kms_frontbuffer_tracking@fbcpsrdrrs-rgb565-draw-pwrite
+kms_frontbuffer_tracking@fbcpsrdrrs-rgb101010-draw-pwrite
+kms_frontbuffer_tracking@fbcpsrdrrs-rgb565-draw-blt
+kms_frontbuffer_tracking@fbcpsrdrrs-rgb101010-draw-blt
+kms_frontbuffer_tracking@fbcpsrdrrs-rgb565-draw-render
+kms_frontbuffer_tracking@fbcpsrdrrs-rgb101010-draw-render
+kms_frontbuffer_tracking@fbc-indfb-scaledprimary
+kms_frontbuffer_tracking@fbc-shrfb-scaledprimary
+kms_frontbuffer_tracking@psr-indfb-scaledprimary
+kms_frontbuffer_tracking@psr-shrfb-scaledprimary
+kms_frontbuffer_tracking@fbcpsr-indfb-scaledprimary
+kms_frontbuffer_tracking@fbcpsr-shrfb-scaledprimary
+kms_frontbuffer_tracking@drrs-indfb-scaledprimary
+kms_frontbuffer_tracking@drrs-shrfb-scaledprimary
+kms_frontbuffer_tracking@fbcdrrs-indfb-scaledprimary
+kms_frontbuffer_tracking@fbcdrrs-shrfb-scaledprimary
+kms_frontbuffer_tracking@psrdrrs-indfb-scaledprimary
+kms_frontbuffer_tracking@psrdrrs-shrfb-scaledprimary
+kms_frontbuffer_tracking@fbcpsrdrrs-indfb-scaledprimary
+kms_frontbuffer_tracking@fbcpsrdrrs-shrfb-scaledprimary
+kms_frontbuffer_tracking@fbc-modesetfrombusy
+kms_frontbuffer_tracking@fbc-badstride
+kms_frontbuffer_tracking@fbc-stridechange
+kms_frontbuffer_tracking@fbc-tiling-linear
+kms_frontbuffer_tracking@fbc-tiling-y
+kms_frontbuffer_tracking@fbc-tiling-4
+kms_frontbuffer_tracking@fbc-suspend
+kms_frontbuffer_tracking@psr-modesetfrombusy
+kms_frontbuffer_tracking@psr-slowdraw
+kms_frontbuffer_tracking@psr-suspend
+kms_frontbuffer_tracking@fbcpsr-modesetfrombusy
+kms_frontbuffer_tracking@fbcpsr-badstride
+kms_frontbuffer_tracking@fbcpsr-stridechange
+kms_frontbuffer_tracking@fbcpsr-tiling-linear
+kms_frontbuffer_tracking@fbcpsr-tiling-y
+kms_frontbuffer_tracking@fbcpsr-tiling-4
+kms_frontbuffer_tracking@fbcpsr-slowdraw
+kms_frontbuffer_tracking@fbcpsr-suspend
+kms_frontbuffer_tracking@drrs-modesetfrombusy
+kms_frontbuffer_tracking@drrs-slowdraw
+kms_frontbuffer_tracking@drrs-suspend
+kms_frontbuffer_tracking@fbcdrrs-modesetfrombusy
+kms_frontbuffer_tracking@fbcdrrs-badstride
+kms_frontbuffer_tracking@fbcdrrs-stridechange
+kms_frontbuffer_tracking@fbcdrrs-tiling-linear
+kms_frontbuffer_tracking@fbcdrrs-tiling-y
+kms_frontbuffer_tracking@fbcdrrs-tiling-4
+kms_frontbuffer_tracking@fbcdrrs-slowdraw
+kms_frontbuffer_tracking@fbcdrrs-suspend
+kms_frontbuffer_tracking@psrdrrs-modesetfrombusy
+kms_frontbuffer_tracking@psrdrrs-slowdraw
+kms_frontbuffer_tracking@psrdrrs-suspend
+kms_frontbuffer_tracking@fbcpsrdrrs-modesetfrombusy
+kms_frontbuffer_tracking@fbcpsrdrrs-badstride
+kms_frontbuffer_tracking@fbcpsrdrrs-stridechange
+kms_frontbuffer_tracking@fbcpsrdrrs-tiling-linear
+kms_frontbuffer_tracking@fbcpsrdrrs-tiling-y
+kms_frontbuffer_tracking@fbcpsrdrrs-tiling-4
+kms_frontbuffer_tracking@fbcpsrdrrs-slowdraw
+kms_frontbuffer_tracking@fbcpsrdrrs-suspend
+kms_frontbuffer_tracking@basic
+kms_getfb@getfb-handle-zero
+kms_getfb@getfb-handle-valid
+kms_getfb@getfb-handle-closed
+kms_getfb@getfb-handle-not-fb
+kms_getfb@getfb-addfb-different-handles
+kms_getfb@getfb-repeated-different-handles
+kms_getfb@getfb-reject-ccs
+kms_getfb@getfb2-handle-zero
+kms_getfb@getfb2-handle-closed
+kms_getfb@getfb2-handle-not-fb
+kms_getfb@getfb2-accept-ccs
+kms_getfb@getfb2-into-addfb2
+kms_getfb@getfb-handle-protection
+kms_getfb@getfb2-handle-protection
+kms_hdmi_inject@inject-4k
+kms_hdmi_inject@inject-audio
+kms_hdr@bpc-switch
+kms_hdr@bpc-switch-dpms
+kms_hdr@bpc-switch-suspend
+kms_hdr@static-toggle
+kms_hdr@static-toggle-dpms
+kms_hdr@static-toggle-suspend
+kms_hdr@static-swap
+kms_hdr@invalid-metadata-sizes
+kms_hdr@invalid-hdr
+kms_invalid_mode@clock-too-high
+kms_invalid_mode@zero-clock
+kms_invalid_mode@int-max-clock
+kms_invalid_mode@uint-max-clock
+kms_invalid_mode@zero-hdisplay
+kms_invalid_mode@zero-vdisplay
+kms_invalid_mode@bad-hsync-start
+kms_invalid_mode@bad-vsync-start
+kms_invalid_mode@bad-hsync-end
+kms_invalid_mode@bad-vsync-end
+kms_invalid_mode@bad-htotal
+kms_invalid_mode@bad-vtotal
+kms_legacy_colorkey@basic
+kms_legacy_colorkey@invalid-plane
+kms_multipipe_modeset@basic-max-pipe-crc-check
+kms_panel_fitting@legacy
+kms_panel_fitting@atomic-fastset
+kms_pipe_b_c_ivb@pipe-B-dpms-off-modeset-pipe-C
+kms_pipe_b_c_ivb@pipe-B-double-modeset-then-modeset-pipe-C
+kms_pipe_b_c_ivb@disable-pipe-B-enable-pipe-C
+kms_pipe_b_c_ivb@from-pipe-C-to-B-with-3-lanes
+kms_pipe_b_c_ivb@enable-pipe-C-while-B-has-3-lanes
+kms_pipe_crc_basic@bad-source
+kms_pipe_crc_basic@read-crc
+kms_pipe_crc_basic@read-crc-frame-sequence
+kms_pipe_crc_basic@nonblocking-crc
+kms_pipe_crc_basic@nonblocking-crc-frame-sequence
+kms_pipe_crc_basic@suspend-read-crc
+kms_pipe_crc_basic@hang-read-crc
+kms_pipe_crc_basic@disable-crc-after-crtc
+kms_pipe_crc_basic@compare-crc-sanitycheck-xr24
+kms_pipe_crc_basic@compare-crc-sanitycheck-nv12
+kms_plane@pixel-format
+kms_plane@pixel-format-source-clamping
+kms_plane@plane-position-covered
+kms_plane@plane-position-hole
+kms_plane@plane-position-hole-dpms
+kms_plane@plane-panning-top-left
+kms_plane@plane-panning-bottom-right
+kms_plane@plane-panning-bottom-right-suspend
+kms_plane@invalid-pixel-format-settings
+kms_plane_alpha_blend@alpha-basic
+kms_plane_alpha_blend@alpha-7efc
+kms_plane_alpha_blend@coverage-7efc
+kms_plane_alpha_blend@coverage-vs-premult-vs-constant
+kms_plane_alpha_blend@alpha-transparent-fb
+kms_plane_alpha_blend@alpha-opaque-fb
+kms_plane_alpha_blend@constant-alpha-min
+kms_plane_alpha_blend@constant-alpha-mid
+kms_plane_alpha_blend@constant-alpha-max
+kms_plane_cursor@primary
+kms_plane_cursor@overlay
+kms_plane_cursor@viewport
+kms_plane_lowres@tiling-none
+kms_plane_lowres@tiling-x
+kms_plane_lowres@tiling-y
+kms_plane_lowres@tiling-yf
+kms_plane_lowres@tiling-4
+kms_plane_multiple@tiling-none
+kms_plane_multiple@tiling-x
+kms_plane_multiple@tiling-y
+kms_plane_multiple@tiling-yf
+kms_plane_multiple@tiling-4
+kms_plane_scaling@plane-upscale-with-pixel-format-20x20
+kms_plane_scaling@plane-upscale-with-pixel-format-factor-0-25
+kms_plane_scaling@plane-downscale-with-pixel-format-factor-0-25
+kms_plane_scaling@plane-downscale-with-pixel-format-factor-0-5
+kms_plane_scaling@plane-downscale-with-pixel-format-factor-0-75
+kms_plane_scaling@plane-scaler-with-pixel-format-unity-scaling
+kms_plane_scaling@plane-upscale-with-rotation-20x20
+kms_plane_scaling@plane-upscale-with-rotation-factor-0-25
+kms_plane_scaling@plane-downscale-with-rotation-factor-0-25
+kms_plane_scaling@plane-downscale-with-rotation-factor-0-5
+kms_plane_scaling@plane-downscale-with-rotation-factor-0-75
+kms_plane_scaling@plane-scaler-with-rotation-unity-scaling
+kms_plane_scaling@plane-upscale-with-modifiers-20x20
+kms_plane_scaling@plane-upscale-with-modifiers-factor-0-25
+kms_plane_scaling@plane-downscale-with-modifiers-factor-0-25
+kms_plane_scaling@plane-downscale-with-modifiers-factor-0-5
+kms_plane_scaling@plane-downscale-with-modifiers-factor-0-75
+kms_plane_scaling@plane-scaler-with-modifiers-unity-scaling
+kms_plane_scaling@plane-scaler-with-clipping-clamping-pixel-formats
+kms_plane_scaling@plane-scaler-with-clipping-clamping-rotation
+kms_plane_scaling@plane-scaler-with-clipping-clamping-modifiers
+kms_plane_scaling@planes-upscale-20x20
+kms_plane_scaling@planes-upscale-factor-0-25
+kms_plane_scaling@planes-scaler-unity-scaling
+kms_plane_scaling@planes-downscale-factor-0-25
+kms_plane_scaling@planes-downscale-factor-0-5
+kms_plane_scaling@planes-downscale-factor-0-75
+kms_plane_scaling@planes-upscale-20x20-downscale-factor-0-25
+kms_plane_scaling@planes-upscale-20x20-downscale-factor-0-5
+kms_plane_scaling@planes-upscale-20x20-downscale-factor-0-75
+kms_plane_scaling@planes-upscale-factor-0-25-downscale-factor-0-25
+kms_plane_scaling@planes-upscale-factor-0-25-downscale-factor-0-5
+kms_plane_scaling@planes-upscale-factor-0-25-downscale-factor-0-75
+kms_plane_scaling@planes-unity-scaling-downscale-factor-0-25
+kms_plane_scaling@planes-unity-scaling-downscale-factor-0-5
+kms_plane_scaling@planes-unity-scaling-downscale-factor-0-75
+kms_plane_scaling@planes-downscale-factor-0-25-upscale-20x20
+kms_plane_scaling@planes-downscale-factor-0-25-upscale-factor-0-25
+kms_plane_scaling@planes-downscale-factor-0-25-unity-scaling
+kms_plane_scaling@planes-downscale-factor-0-5-upscale-20x20
+kms_plane_scaling@planes-downscale-factor-0-5-upscale-factor-0-25
+kms_plane_scaling@planes-downscale-factor-0-5-unity-scaling
+kms_plane_scaling@planes-downscale-factor-0-75-upscale-20x20
+kms_plane_scaling@planes-downscale-factor-0-75-upscale-factor-0-25
+kms_plane_scaling@planes-downscale-factor-0-75-unity-scaling
+kms_plane_scaling@intel-max-src-size
+kms_plane_scaling@invalid-num-scalers
+kms_plane_scaling@invalid-parameters
+kms_plane_scaling@2x-scaler-multi-pipe
+kms_prime@basic-crc-hybrid
+kms_prime@basic-modeset-hybrid
+kms_prime@D3hot
+kms_prime@basic-crc-vgem
+kms_prop_blob@basic
+kms_prop_blob@blob-prop-core
+kms_prop_blob@blob-prop-validate
+kms_prop_blob@blob-prop-lifetime
+kms_prop_blob@blob-multiple
+kms_prop_blob@invalid-get-prop-any
+kms_prop_blob@invalid-get-prop
+kms_prop_blob@invalid-set-prop-any
+kms_prop_blob@invalid-set-prop
+kms_properties@plane-properties-legacy
+kms_properties@plane-properties-atomic
+kms_properties@crtc-properties-legacy
+kms_properties@crtc-properties-atomic
+kms_properties@connector-properties-legacy
+kms_properties@connector-properties-atomic
+kms_properties@invalid-properties-legacy
+kms_properties@invalid-properties-atomic
+kms_properties@get_properties-sanity-atomic
+kms_properties@get_properties-sanity-non-atomic
+kms_psr@basic
+kms_psr@no_drrs
+kms_psr@primary_page_flip
+kms_psr@primary_mmap_gtt
+kms_psr@primary_mmap_cpu
+kms_psr@primary_blt
+kms_psr@primary_render
+kms_psr@sprite_mmap_gtt
+kms_psr@cursor_mmap_gtt
+kms_psr@sprite_mmap_cpu
+kms_psr@cursor_mmap_cpu
+kms_psr@sprite_blt
+kms_psr@cursor_blt
+kms_psr@sprite_render
+kms_psr@cursor_render
+kms_psr@sprite_plane_move
+kms_psr@cursor_plane_move
+kms_psr@sprite_plane_onoff
+kms_psr@cursor_plane_onoff
+kms_psr@dpms
+kms_psr@suspend
+kms_psr@psr2_basic
+kms_psr@psr2_no_drrs
+kms_psr@psr2_primary_page_flip
+kms_psr@psr2_primary_mmap_gtt
+kms_psr@psr2_primary_mmap_cpu
+kms_psr@psr2_primary_blt
+kms_psr@psr2_primary_render
+kms_psr@psr2_sprite_mmap_gtt
+kms_psr@psr2_cursor_mmap_gtt
+kms_psr@psr2_sprite_mmap_cpu
+kms_psr@psr2_cursor_mmap_cpu
+kms_psr@psr2_sprite_blt
+kms_psr@psr2_cursor_blt
+kms_psr@psr2_sprite_render
+kms_psr@psr2_cursor_render
+kms_psr@psr2_sprite_plane_move
+kms_psr@psr2_cursor_plane_move
+kms_psr@psr2_sprite_plane_onoff
+kms_psr@psr2_cursor_plane_onoff
+kms_psr@psr2_dpms
+kms_psr@psr2_suspend
+kms_psr2_sf@primary-plane-update-sf-dmg-area
+kms_psr2_sf@primary-plane-update-sf-dmg-area-big-fb
+kms_psr2_sf@overlay-plane-update-sf-dmg-area
+kms_psr2_sf@cursor-plane-update-sf
+kms_psr2_sf@cursor-plane-move-continuous-sf
+kms_psr2_sf@cursor-plane-move-continuous-exceed-sf
+kms_psr2_sf@cursor-plane-move-continuous-exceed-fully-sf
+kms_psr2_sf@plane-move-sf-dmg-area
+kms_psr2_sf@overlay-plane-move-continuous-sf
+kms_psr2_sf@overlay-plane-move-continuous-exceed-sf
+kms_psr2_sf@overlay-plane-move-continuous-exceed-fully-sf
+kms_psr2_sf@overlay-primary-update-sf-dmg-area
+kms_psr2_sf@overlay-plane-update-continuous-sf
+kms_psr2_su@page_flip-XRGB8888
+kms_psr2_su@page_flip-NV12
+kms_psr2_su@page_flip-P010
+kms_psr2_su@frontbuffer-XRGB8888
+kms_pwrite_crc
+kms_rmfb@rmfb-ioctl
+kms_rmfb@close-fd
+kms_rotation_crc@primary-rotation-90
+kms_rotation_crc@primary-rotation-180
+kms_rotation_crc@primary-rotation-270
+kms_rotation_crc@sprite-rotation-90
+kms_rotation_crc@sprite-rotation-180
+kms_rotation_crc@sprite-rotation-270
+kms_rotation_crc@cursor-rotation-180
+kms_rotation_crc@sprite-rotation-90-pos-100-0
+kms_rotation_crc@bad-pixel-format
+kms_rotation_crc@bad-tiling
+kms_rotation_crc@primary-x-tiled-reflect-x-0
+kms_rotation_crc@primary-x-tiled-reflect-x-180
+kms_rotation_crc@primary-y-tiled-reflect-x-0
+kms_rotation_crc@primary-y-tiled-reflect-x-90
+kms_rotation_crc@primary-y-tiled-reflect-x-180
+kms_rotation_crc@primary-y-tiled-reflect-x-270
+kms_rotation_crc@primary-yf-tiled-reflect-x-0
+kms_rotation_crc@primary-yf-tiled-reflect-x-90
+kms_rotation_crc@primary-yf-tiled-reflect-x-180
+kms_rotation_crc@primary-yf-tiled-reflect-x-270
+kms_rotation_crc@primary-4-tiled-reflect-x-0
+kms_rotation_crc@primary-4-tiled-reflect-x-180
+kms_rotation_crc@multiplane-rotation
+kms_rotation_crc@multiplane-rotation-cropping-top
+kms_rotation_crc@multiplane-rotation-cropping-bottom
+kms_rotation_crc@exhaust-fences
+kms_scaling_modes@scaling-mode-full
+kms_scaling_modes@scaling-mode-center
+kms_scaling_modes@scaling-mode-full-aspect
+kms_scaling_modes@scaling-mode-none
+kms_selftest@drm_cmdline
+kms_selftest@drm_damage
+kms_selftest@drm_dp_mst
+kms_selftest@drm_format_helper
+kms_selftest@drm_format
+kms_selftest@framebuffer
+kms_selftest@drm_plane
+kms_setmode@basic
+kms_setmode@basic-clone-single-crtc
+kms_setmode@invalid-clone-single-crtc
+kms_setmode@invalid-clone-exclusive-crtc
+kms_setmode@clone-exclusive-crtc
+kms_setmode@invalid-clone-single-crtc-stealing
+kms_sysfs_edid_timing
+kms_tv_load_detect@load-detect
+kms_universal_plane@universal-plane-pipe-A-functional
+kms_universal_plane@universal-plane-pipe-A-sanity
+kms_universal_plane@disable-primary-vs-flip-pipe-A
+kms_universal_plane@cursor-fb-leak-pipe-A
+kms_universal_plane@universal-plane-pageflip-windowed-pipe-A
+kms_universal_plane@universal-plane-pipe-B-functional
+kms_universal_plane@universal-plane-pipe-B-sanity
+kms_universal_plane@disable-primary-vs-flip-pipe-B
+kms_universal_plane@cursor-fb-leak-pipe-B
+kms_universal_plane@universal-plane-pageflip-windowed-pipe-B
+kms_universal_plane@universal-plane-pipe-C-functional
+kms_universal_plane@universal-plane-pipe-C-sanity
+kms_universal_plane@disable-primary-vs-flip-pipe-C
+kms_universal_plane@cursor-fb-leak-pipe-C
+kms_universal_plane@universal-plane-pageflip-windowed-pipe-C
+kms_universal_plane@universal-plane-pipe-D-functional
+kms_universal_plane@universal-plane-pipe-D-sanity
+kms_universal_plane@disable-primary-vs-flip-pipe-D
+kms_universal_plane@cursor-fb-leak-pipe-D
+kms_universal_plane@universal-plane-pageflip-windowed-pipe-D
+kms_universal_plane@universal-plane-pipe-E-functional
+kms_universal_plane@universal-plane-pipe-E-sanity
+kms_universal_plane@disable-primary-vs-flip-pipe-E
+kms_universal_plane@cursor-fb-leak-pipe-E
+kms_universal_plane@universal-plane-pageflip-windowed-pipe-E
+kms_universal_plane@universal-plane-pipe-F-functional
+kms_universal_plane@universal-plane-pipe-F-sanity
+kms_universal_plane@disable-primary-vs-flip-pipe-F
+kms_universal_plane@cursor-fb-leak-pipe-F
+kms_universal_plane@universal-plane-pageflip-windowed-pipe-F
+kms_universal_plane@universal-plane-pipe-G-functional
+kms_universal_plane@universal-plane-pipe-G-sanity
+kms_universal_plane@disable-primary-vs-flip-pipe-G
+kms_universal_plane@cursor-fb-leak-pipe-G
+kms_universal_plane@universal-plane-pageflip-windowed-pipe-G
+kms_universal_plane@universal-plane-pipe-H-functional
+kms_universal_plane@universal-plane-pipe-H-sanity
+kms_universal_plane@disable-primary-vs-flip-pipe-H
+kms_universal_plane@cursor-fb-leak-pipe-H
+kms_universal_plane@universal-plane-pageflip-windowed-pipe-H
+kms_vblank@invalid
+kms_vblank@crtc-id
+kms_vblank@pipe-A-accuracy-idle
+kms_vblank@pipe-A-query-idle
+kms_vblank@pipe-A-query-idle-hang
+kms_vblank@pipe-A-query-forked
+kms_vblank@pipe-A-query-forked-hang
+kms_vblank@pipe-A-query-busy
+kms_vblank@pipe-A-query-busy-hang
+kms_vblank@pipe-A-query-forked-busy
+kms_vblank@pipe-A-query-forked-busy-hang
+kms_vblank@pipe-A-wait-idle
+kms_vblank@pipe-A-wait-idle-hang
+kms_vblank@pipe-A-wait-forked
+kms_vblank@pipe-A-wait-forked-hang
+kms_vblank@pipe-A-wait-busy
+kms_vblank@pipe-A-wait-busy-hang
+kms_vblank@pipe-A-wait-forked-busy
+kms_vblank@pipe-A-wait-forked-busy-hang
+kms_vblank@pipe-A-ts-continuation-idle
+kms_vblank@pipe-A-ts-continuation-idle-hang
+kms_vblank@pipe-A-ts-continuation-dpms-rpm
+kms_vblank@pipe-A-ts-continuation-dpms-suspend
+kms_vblank@pipe-A-ts-continuation-suspend
+kms_vblank@pipe-A-ts-continuation-modeset
+kms_vblank@pipe-A-ts-continuation-modeset-hang
+kms_vblank@pipe-A-ts-continuation-modeset-rpm
+kms_vblank@pipe-B-accuracy-idle
+kms_vblank@pipe-B-query-idle
+kms_vblank@pipe-B-query-idle-hang
+kms_vblank@pipe-B-query-forked
+kms_vblank@pipe-B-query-forked-hang
+kms_vblank@pipe-B-query-busy
+kms_vblank@pipe-B-query-busy-hang
+kms_vblank@pipe-B-query-forked-busy
+kms_vblank@pipe-B-query-forked-busy-hang
+kms_vblank@pipe-B-wait-idle
+kms_vblank@pipe-B-wait-idle-hang
+kms_vblank@pipe-B-wait-forked
+kms_vblank@pipe-B-wait-forked-hang
+kms_vblank@pipe-B-wait-busy
+kms_vblank@pipe-B-wait-busy-hang
+kms_vblank@pipe-B-wait-forked-busy
+kms_vblank@pipe-B-wait-forked-busy-hang
+kms_vblank@pipe-B-ts-continuation-idle
+kms_vblank@pipe-B-ts-continuation-idle-hang
+kms_vblank@pipe-B-ts-continuation-dpms-rpm
+kms_vblank@pipe-B-ts-continuation-dpms-suspend
+kms_vblank@pipe-B-ts-continuation-suspend
+kms_vblank@pipe-B-ts-continuation-modeset
+kms_vblank@pipe-B-ts-continuation-modeset-hang
+kms_vblank@pipe-B-ts-continuation-modeset-rpm
+kms_vblank@pipe-C-accuracy-idle
+kms_vblank@pipe-C-query-idle
+kms_vblank@pipe-C-query-idle-hang
+kms_vblank@pipe-C-query-forked
+kms_vblank@pipe-C-query-forked-hang
+kms_vblank@pipe-C-query-busy
+kms_vblank@pipe-C-query-busy-hang
+kms_vblank@pipe-C-query-forked-busy
+kms_vblank@pipe-C-query-forked-busy-hang
+kms_vblank@pipe-C-wait-idle
+kms_vblank@pipe-C-wait-idle-hang
+kms_vblank@pipe-C-wait-forked
+kms_vblank@pipe-C-wait-forked-hang
+kms_vblank@pipe-C-wait-busy
+kms_vblank@pipe-C-wait-busy-hang
+kms_vblank@pipe-C-wait-forked-busy
+kms_vblank@pipe-C-wait-forked-busy-hang
+kms_vblank@pipe-C-ts-continuation-idle
+kms_vblank@pipe-C-ts-continuation-idle-hang
+kms_vblank@pipe-C-ts-continuation-dpms-rpm
+kms_vblank@pipe-C-ts-continuation-dpms-suspend
+kms_vblank@pipe-C-ts-continuation-suspend
+kms_vblank@pipe-C-ts-continuation-modeset
+kms_vblank@pipe-C-ts-continuation-modeset-hang
+kms_vblank@pipe-C-ts-continuation-modeset-rpm
+kms_vblank@pipe-D-accuracy-idle
+kms_vblank@pipe-D-query-idle
+kms_vblank@pipe-D-query-idle-hang
+kms_vblank@pipe-D-query-forked
+kms_vblank@pipe-D-query-forked-hang
+kms_vblank@pipe-D-query-busy
+kms_vblank@pipe-D-query-busy-hang
+kms_vblank@pipe-D-query-forked-busy
+kms_vblank@pipe-D-query-forked-busy-hang
+kms_vblank@pipe-D-wait-idle
+kms_vblank@pipe-D-wait-idle-hang
+kms_vblank@pipe-D-wait-forked
+kms_vblank@pipe-D-wait-forked-hang
+kms_vblank@pipe-D-wait-busy
+kms_vblank@pipe-D-wait-busy-hang
+kms_vblank@pipe-D-wait-forked-busy
+kms_vblank@pipe-D-wait-forked-busy-hang
+kms_vblank@pipe-D-ts-continuation-idle
+kms_vblank@pipe-D-ts-continuation-idle-hang
+kms_vblank@pipe-D-ts-continuation-dpms-rpm
+kms_vblank@pipe-D-ts-continuation-dpms-suspend
+kms_vblank@pipe-D-ts-continuation-suspend
+kms_vblank@pipe-D-ts-continuation-modeset
+kms_vblank@pipe-D-ts-continuation-modeset-hang
+kms_vblank@pipe-D-ts-continuation-modeset-rpm
+kms_vblank@pipe-E-accuracy-idle
+kms_vblank@pipe-E-query-idle
+kms_vblank@pipe-E-query-idle-hang
+kms_vblank@pipe-E-query-forked
+kms_vblank@pipe-E-query-forked-hang
+kms_vblank@pipe-E-query-busy
+kms_vblank@pipe-E-query-busy-hang
+kms_vblank@pipe-E-query-forked-busy
+kms_vblank@pipe-E-query-forked-busy-hang
+kms_vblank@pipe-E-wait-idle
+kms_vblank@pipe-E-wait-idle-hang
+kms_vblank@pipe-E-wait-forked
+kms_vblank@pipe-E-wait-forked-hang
+kms_vblank@pipe-E-wait-busy
+kms_vblank@pipe-E-wait-busy-hang
+kms_vblank@pipe-E-wait-forked-busy
+kms_vblank@pipe-E-wait-forked-busy-hang
+kms_vblank@pipe-E-ts-continuation-idle
+kms_vblank@pipe-E-ts-continuation-idle-hang
+kms_vblank@pipe-E-ts-continuation-dpms-rpm
+kms_vblank@pipe-E-ts-continuation-dpms-suspend
+kms_vblank@pipe-E-ts-continuation-suspend
+kms_vblank@pipe-E-ts-continuation-modeset
+kms_vblank@pipe-E-ts-continuation-modeset-hang
+kms_vblank@pipe-E-ts-continuation-modeset-rpm
+kms_vblank@pipe-F-accuracy-idle
+kms_vblank@pipe-F-query-idle
+kms_vblank@pipe-F-query-idle-hang
+kms_vblank@pipe-F-query-forked
+kms_vblank@pipe-F-query-forked-hang
+kms_vblank@pipe-F-query-busy
+kms_vblank@pipe-F-query-busy-hang
+kms_vblank@pipe-F-query-forked-busy
+kms_vblank@pipe-F-query-forked-busy-hang
+kms_vblank@pipe-F-wait-idle
+kms_vblank@pipe-F-wait-idle-hang
+kms_vblank@pipe-F-wait-forked
+kms_vblank@pipe-F-wait-forked-hang
+kms_vblank@pipe-F-wait-busy
+kms_vblank@pipe-F-wait-busy-hang
+kms_vblank@pipe-F-wait-forked-busy
+kms_vblank@pipe-F-wait-forked-busy-hang
+kms_vblank@pipe-F-ts-continuation-idle
+kms_vblank@pipe-F-ts-continuation-idle-hang
+kms_vblank@pipe-F-ts-continuation-dpms-rpm
+kms_vblank@pipe-F-ts-continuation-dpms-suspend
+kms_vblank@pipe-F-ts-continuation-suspend
+kms_vblank@pipe-F-ts-continuation-modeset
+kms_vblank@pipe-F-ts-continuation-modeset-hang
+kms_vblank@pipe-F-ts-continuation-modeset-rpm
+kms_vblank@pipe-G-accuracy-idle
+kms_vblank@pipe-G-query-idle
+kms_vblank@pipe-G-query-idle-hang
+kms_vblank@pipe-G-query-forked
+kms_vblank@pipe-G-query-forked-hang
+kms_vblank@pipe-G-query-busy
+kms_vblank@pipe-G-query-busy-hang
+kms_vblank@pipe-G-query-forked-busy
+kms_vblank@pipe-G-query-forked-busy-hang
+kms_vblank@pipe-G-wait-idle
+kms_vblank@pipe-G-wait-idle-hang
+kms_vblank@pipe-G-wait-forked
+kms_vblank@pipe-G-wait-forked-hang
+kms_vblank@pipe-G-wait-busy
+kms_vblank@pipe-G-wait-busy-hang
+kms_vblank@pipe-G-wait-forked-busy
+kms_vblank@pipe-G-wait-forked-busy-hang
+kms_vblank@pipe-G-ts-continuation-idle
+kms_vblank@pipe-G-ts-continuation-idle-hang
+kms_vblank@pipe-G-ts-continuation-dpms-rpm
+kms_vblank@pipe-G-ts-continuation-dpms-suspend
+kms_vblank@pipe-G-ts-continuation-suspend
+kms_vblank@pipe-G-ts-continuation-modeset
+kms_vblank@pipe-G-ts-continuation-modeset-hang
+kms_vblank@pipe-G-ts-continuation-modeset-rpm
+kms_vblank@pipe-H-accuracy-idle
+kms_vblank@pipe-H-query-idle
+kms_vblank@pipe-H-query-idle-hang
+kms_vblank@pipe-H-query-forked
+kms_vblank@pipe-H-query-forked-hang
+kms_vblank@pipe-H-query-busy
+kms_vblank@pipe-H-query-busy-hang
+kms_vblank@pipe-H-query-forked-busy
+kms_vblank@pipe-H-query-forked-busy-hang
+kms_vblank@pipe-H-wait-idle
+kms_vblank@pipe-H-wait-idle-hang
+kms_vblank@pipe-H-wait-forked
+kms_vblank@pipe-H-wait-forked-hang
+kms_vblank@pipe-H-wait-busy
+kms_vblank@pipe-H-wait-busy-hang
+kms_vblank@pipe-H-wait-forked-busy
+kms_vblank@pipe-H-wait-forked-busy-hang
+kms_vblank@pipe-H-ts-continuation-idle
+kms_vblank@pipe-H-ts-continuation-idle-hang
+kms_vblank@pipe-H-ts-continuation-dpms-rpm
+kms_vblank@pipe-H-ts-continuation-dpms-suspend
+kms_vblank@pipe-H-ts-continuation-suspend
+kms_vblank@pipe-H-ts-continuation-modeset
+kms_vblank@pipe-H-ts-continuation-modeset-hang
+kms_vblank@pipe-H-ts-continuation-modeset-rpm
+kms_vrr@flip-basic
+kms_vrr@flip-dpms
+kms_vrr@flip-suspend
+kms_vrr@flipline
+kms_vrr@negative-basic
+kms_writeback@writeback-pixel-formats
+kms_writeback@writeback-invalid-parameters
+kms_writeback@writeback-fb-id
+kms_writeback@writeback-check-output
+prime_mmap_kms@buffer-sharing
diff --git a/drivers/gpu/drm/ci/x86_64.config b/drivers/gpu/drm/ci/x86_64.config
new file mode 100644
index 000000000000..1cbd49a5b23a
--- /dev/null
+++ b/drivers/gpu/drm/ci/x86_64.config
@@ -0,0 +1,111 @@
+CONFIG_LOCALVERSION_AUTO=y
+CONFIG_DEBUG_KERNEL=y
+
+CONFIG_CRYPTO_ZSTD=y
+CONFIG_ZRAM_MEMORY_TRACKING=y
+CONFIG_ZRAM_WRITEBACK=y
+CONFIG_ZRAM=y
+CONFIG_ZSMALLOC_STAT=y
+
+CONFIG_PWM=y
+CONFIG_PM_DEVFREQ=y
+CONFIG_OF=y
+CONFIG_CROS_EC=y
+
+# abootimg with a 'dummy' rootfs fails with root=/dev/nfs
+CONFIG_BLK_DEV_INITRD=n
+
+CONFIG_DEVFREQ_GOV_PERFORMANCE=y
+CONFIG_DEVFREQ_GOV_POWERSAVE=y
+CONFIG_DEVFREQ_GOV_USERSPACE=y
+CONFIG_DEVFREQ_GOV_PASSIVE=y
+
+CONFIG_DRM=y
+CONFIG_DRM_PANEL_SIMPLE=y
+CONFIG_PWM_CROS_EC=y
+CONFIG_BACKLIGHT_PWM=y
+
+# Strip out some stuff we don't need for graphics testing, to reduce
+# the build.
+CONFIG_CAN=n
+CONFIG_WIRELESS=n
+CONFIG_RFKILL=n
+CONFIG_WLAN=n
+
+CONFIG_REGULATOR_FAN53555=y
+CONFIG_REGULATOR=y
+
+CONFIG_REGULATOR_VCTRL=y
+
+CONFIG_KASAN=n
+CONFIG_KASAN_INLINE=n
+CONFIG_STACKTRACE=n
+
+CONFIG_TMPFS=y
+
+CONFIG_PROVE_LOCKING=n
+CONFIG_DEBUG_LOCKDEP=n
+CONFIG_SOFTLOCKUP_DETECTOR=y
+CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC=y
+
+CONFIG_DETECT_HUNG_TASK=y
+
+CONFIG_USB_USBNET=y
+CONFIG_NETDEVICES=y
+CONFIG_USB_NET_DRIVERS=y
+CONFIG_USB_RTL8152=y
+CONFIG_USB_NET_AX8817X=y
+CONFIG_USB_NET_SMSC95XX=y
+CONFIG_USB_GADGET=y
+CONFIG_USB_ETH=y
+
+CONFIG_FW_LOADER_COMPRESS=y
+
+# options for AMD devices
+CONFIG_X86_AMD_PLATFORM_DEVICE=y
+CONFIG_ACPI_VIDEO=y
+CONFIG_X86_AMD_FREQ_SENSITIVITY=y
+CONFIG_PINCTRL=y
+CONFIG_PINCTRL_AMD=y
+CONFIG_DRM_AMDGPU=m
+CONFIG_DRM_AMDGPU_SI=y
+CONFIG_DRM_AMDGPU_USERPTR=y
+CONFIG_DRM_AMD_ACP=n
+CONFIG_ACPI_WMI=y
+CONFIG_MXM_WMI=y
+CONFIG_PARPORT=y
+CONFIG_PARPORT_PC=y
+CONFIG_PARPORT_SERIAL=y
+CONFIG_SERIAL_8250_DW=y
+CONFIG_CHROME_PLATFORMS=y
+CONFIG_KVM_AMD=m
+
+#options for Intel devices
+CONFIG_MFD_INTEL_LPSS_PCI=y
+CONFIG_KVM_INTEL=m
+
+#options for KVM guests
+CONFIG_FUSE_FS=y
+CONFIG_HYPERVISOR_GUEST=y
+CONFIG_KVM=y
+CONFIG_KVM_GUEST=y
+CONFIG_VIRT_DRIVERS=y
+CONFIG_VIRTIO_FS=y
+CONFIG_DRM_VIRTIO_GPU=y
+CONFIG_SERIAL_8250_CONSOLE=y
+CONFIG_VIRTIO_NET=y
+CONFIG_VIRTIO_CONSOLE=y
+CONFIG_PARAVIRT=y
+CONFIG_VIRTIO_BLK=y
+CONFIG_VIRTUALIZATION=y
+CONFIG_VIRTIO=y
+CONFIG_VIRTIO_PCI=y
+CONFIG_VIRTIO_MMIO=y
+CONFIG_VIRTIO_MMIO_CMDLINE_DEVICES=y
+CONFIG_CRYPTO_DEV_VIRTIO=y
+CONFIG_HW_RANDOM_VIRTIO=y
+CONFIG_BLK_MQ_VIRTIO=y
+CONFIG_TUN=y
+CONFIG_VSOCKETS=y
+CONFIG_VIRTIO_VSOCKETS=y
+CONFIG_VHOST_VSOCK=m
diff --git a/drivers/gpu/drm/ci/xfails/amdgpu-stoney-fails.txt b/drivers/gpu/drm/ci/xfails/amdgpu-stoney-fails.txt
new file mode 100644
index 000000000000..bd9392536e7c
--- /dev/null
+++ b/drivers/gpu/drm/ci/xfails/amdgpu-stoney-fails.txt
@@ -0,0 +1,19 @@
+kms_addfb_basic@bad-pitch-65536,Fail
+kms_addfb_basic@bo-too-small,Fail
+kms_async_flips@invalid-async-flip,Fail
+kms_atomic@plane-immutable-zpos,Fail
+kms_atomic_transition@plane-toggle-modeset-transition,Fail
+kms_bw@linear-tiling-1-displays-2560x1440p,Fail
+kms_bw@linear-tiling-1-displays-3840x2160p,Fail
+kms_bw@linear-tiling-2-displays-3840x2160p,Fail
+kms_bw@linear-tiling-3-displays-1920x1080p,Fail
+kms_color@degamma,Fail
+kms_cursor_crc@cursor-size-change,Fail
+kms_cursor_crc@pipe-A-cursor-size-change,Fail
+kms_cursor_crc@pipe-B-cursor-size-change,Fail
+kms_cursor_legacy@forked-move,Fail
+kms_hdr@bpc-switch,Fail
+kms_hdr@bpc-switch-dpms,Fail
+kms_plane_multiple@atomic-pipe-A-tiling-none,Fail
+kms_rmfb@close-fd,Fail
+kms_rotation_crc@primary-rotation-180,Fail
diff --git a/drivers/gpu/drm/ci/xfails/amdgpu-stoney-flakes.txt b/drivers/gpu/drm/ci/xfails/amdgpu-stoney-flakes.txt
new file mode 100644
index 000000000000..f8defa0f9e67
--- /dev/null
+++ b/drivers/gpu/drm/ci/xfails/amdgpu-stoney-flakes.txt
@@ -0,0 +1,21 @@
+kms_addfb_basic@too-high
+kms_async_flips@alternate-sync-async-flip
+kms_async_flips@async-flip-with-page-flip-events
+kms_async_flips@crc
+kms_async_flips@test-cursor
+kms_async_flips@test-time-stamp
+kms_atomic_transition@plane-all-modeset-transition-internal-panels
+kms_atomic_transition@plane-all-transition
+kms_atomic_transition@plane-use-after-nonblocking-unbind
+kms_bw@linear-tiling-1-displays-1920x1080p
+kms_bw@linear-tiling-2-displays-1920x1080p
+kms_bw@linear-tiling-2-displays-2560x1440p
+kms_bw@linear-tiling-3-displays-2560x1440p
+kms_bw@linear-tiling-3-displays-3840x2160p
+kms_cursor_crc@pipe-A-cursor-alpha-opaque
+kms_cursor_crc@pipe-B-cursor-alpha-opaque
+kms_plane@pixel-format
+kms_plane_multiple@atomic-pipe-B-tiling-none
+kms_plane_scaling@downscale-with-rotation-factor-0-5
+kms_universal_plane@disable-primary-vs-flip-pipe-A
+kms_universal_plane@disable-primary-vs-flip-pipe-B
diff --git a/drivers/gpu/drm/ci/xfails/amdgpu-stoney-skips.txt b/drivers/gpu/drm/ci/xfails/amdgpu-stoney-skips.txt
new file mode 100644
index 000000000000..e2c538a0f954
--- /dev/null
+++ b/drivers/gpu/drm/ci/xfails/amdgpu-stoney-skips.txt
@@ -0,0 +1,2 @@
+# Suspend to RAM seems to be broken on this machine
+.*suspend.*
\ No newline at end of file
diff --git a/drivers/gpu/drm/ci/xfails/i915-amly-fails.txt b/drivers/gpu/drm/ci/xfails/i915-amly-fails.txt
new file mode 100644
index 000000000000..5f513c638beb
--- /dev/null
+++ b/drivers/gpu/drm/ci/xfails/i915-amly-fails.txt
@@ -0,0 +1,17 @@
+kms_flip_scaled_crc@flip-32bpp-linear-to-64bpp-linear-downscaling,Fail
+kms_flip_scaled_crc@flip-32bpp-linear-to-64bpp-linear-upscaling,Fail
+kms_flip_scaled_crc@flip-32bpp-xtile-to-64bpp-xtile-downscaling,Fail
+kms_flip_scaled_crc@flip-32bpp-xtile-to-64bpp-xtile-upscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-linear-to-16bpp-linear-downscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-linear-to-16bpp-linear-upscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-linear-to-32bpp-linear-downscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-linear-to-32bpp-linear-upscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-xtile-to-16bpp-xtile-downscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-xtile-to-16bpp-xtile-upscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-xtile-to-32bpp-xtile-downscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-xtile-to-32bpp-xtile-upscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytilegen12rcccs-upscaling,Fail
+kms_plane_alpha_blend@alpha-basic,Fail
+kms_plane_alpha_blend@alpha-opaque-fb,Fail
+kms_plane_alpha_blend@alpha-transparent-fb,Fail
+kms_plane_alpha_blend@constant-alpha-max,Fail
diff --git a/drivers/gpu/drm/ci/xfails/i915-amly-flakes.txt b/drivers/gpu/drm/ci/xfails/i915-amly-flakes.txt
new file mode 100644
index 000000000000..d5000515a315
--- /dev/null
+++ b/drivers/gpu/drm/ci/xfails/i915-amly-flakes.txt
@@ -0,0 +1,32 @@
+kms_bw@linear-tiling-2-displays-1920x1080p
+kms_bw@linear-tiling-2-displays-2560x1440p
+kms_bw@linear-tiling-2-displays-3840x2160p
+kms_bw@linear-tiling-3-displays-1920x1080p
+kms_bw@linear-tiling-3-displays-2560x1440p
+kms_bw@linear-tiling-3-displays-3840x2160p
+kms_bw@linear-tiling-4-displays-1920x1080p
+kms_bw@linear-tiling-4-displays-2560x1440p
+kms_bw@linear-tiling-4-displays-3840x2160p
+kms_flip_scaled_crc@flip-32bpp-ytile-to-64bpp-ytile-downscaling
+kms_flip_scaled_crc@flip-32bpp-ytile-to-64bpp-ytile-upscaling
+kms_flip_scaled_crc@flip-32bpp-ytileccs-to-64bpp-ytile-downscaling
+kms_flip_scaled_crc@flip-32bpp-ytileccs-to-64bpp-ytile-upscaling
+kms_flip_scaled_crc@flip-64bpp-ytile-to-16bpp-ytile-downscaling
+kms_flip_scaled_crc@flip-64bpp-ytile-to-16bpp-ytile-upscaling
+kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytile-downscaling
+kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytile-upscaling
+kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytilercccs-downscaling
+kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytilercccs-upscaling
+kms_plane_alpha_blend@pipe-A-alpha-basic
+kms_plane_alpha_blend@pipe-A-alpha-opaque-fb
+kms_plane_alpha_blend@pipe-A-alpha-transparent-fb
+kms_plane_alpha_blend@pipe-A-constant-alpha-max
+kms_plane_alpha_blend@pipe-B-alpha-basic
+kms_plane_alpha_blend@pipe-B-alpha-opaque-fb
+kms_plane_alpha_blend@pipe-B-alpha-transparent-fb
+kms_plane_alpha_blend@pipe-B-constant-alpha-max
+kms_plane_alpha_blend@pipe-C-alpha-basic
+kms_plane_alpha_blend@pipe-C-alpha-opaque-fb
+kms_plane_alpha_blend@pipe-C-alpha-transparent-fb
+kms_plane_alpha_blend@pipe-C-constant-alpha-max
+kms_sysfs_edid_timing
diff --git a/drivers/gpu/drm/ci/xfails/i915-amly-skips.txt b/drivers/gpu/drm/ci/xfails/i915-amly-skips.txt
new file mode 100644
index 000000000000..fe55540a3f9a
--- /dev/null
+++ b/drivers/gpu/drm/ci/xfails/i915-amly-skips.txt
@@ -0,0 +1,4 @@
+# Suspend to RAM seems to be broken on this machine
+.*suspend.*
+# This is generating kernel oops with divide error
+kms_plane_scaling@invalid-parameters
\ No newline at end of file
diff --git a/drivers/gpu/drm/ci/xfails/i915-apl-fails.txt b/drivers/gpu/drm/ci/xfails/i915-apl-fails.txt
new file mode 100644
index 000000000000..46397ce38d5a
--- /dev/null
+++ b/drivers/gpu/drm/ci/xfails/i915-apl-fails.txt
@@ -0,0 +1,58 @@
+kms_3d,Timeout
+kms_bw@linear-tiling-2-displays-1920x1080p,Fail
+kms_bw@linear-tiling-2-displays-2560x1440p,Fail
+kms_bw@linear-tiling-2-displays-3840x2160p,Fail
+kms_bw@linear-tiling-3-displays-1920x1080p,Fail
+kms_bw@linear-tiling-3-displays-2560x1440p,Fail
+kms_bw@linear-tiling-3-displays-3840x2160p,Fail
+kms_bw@linear-tiling-4-displays-1920x1080p,Fail
+kms_bw@linear-tiling-4-displays-2560x1440p,Fail
+kms_bw@linear-tiling-4-displays-3840x2160p,Fail
+kms_color@ctm-0-25,Fail
+kms_color@ctm-0-50,Fail
+kms_color@ctm-0-75,Fail
+kms_color@ctm-max,Fail
+kms_color@ctm-negative,Fail
+kms_color@ctm-red-to-blue,Fail
+kms_color@ctm-signed,Fail
+kms_flip_scaled_crc@flip-32bpp-linear-to-64bpp-linear-downscaling,Fail
+kms_flip_scaled_crc@flip-32bpp-linear-to-64bpp-linear-upscaling,Fail
+kms_flip_scaled_crc@flip-32bpp-xtile-to-64bpp-xtile-downscaling,Fail
+kms_flip_scaled_crc@flip-32bpp-xtile-to-64bpp-xtile-upscaling,Fail
+kms_flip_scaled_crc@flip-32bpp-ytile-to-64bpp-ytile-downscaling,Fail
+kms_flip_scaled_crc@flip-32bpp-ytile-to-64bpp-ytile-upscaling,Fail
+kms_flip_scaled_crc@flip-32bpp-ytileccs-to-64bpp-ytile-downscaling,Fail
+kms_flip_scaled_crc@flip-32bpp-ytileccs-to-64bpp-ytile-upscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-linear-to-16bpp-linear-downscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-linear-to-16bpp-linear-upscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-linear-to-32bpp-linear-downscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-linear-to-32bpp-linear-upscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-xtile-to-16bpp-xtile-downscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-xtile-to-16bpp-xtile-upscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-xtile-to-32bpp-xtile-downscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-xtile-to-32bpp-xtile-upscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-ytile-to-16bpp-ytile-downscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-ytile-to-16bpp-ytile-upscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytile-downscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytile-upscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytilegen12rcccs-upscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytilercccs-downscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytilercccs-upscaling,Fail
+kms_hdmi_inject@inject-4k,Timeout
+kms_plane@plane-position-hole,Timeout
+kms_plane_alpha_blend@alpha-basic,Fail
+kms_plane_alpha_blend@alpha-opaque-fb,Fail
+kms_plane_alpha_blend@alpha-transparent-fb,Fail
+kms_plane_alpha_blend@constant-alpha-max,Fail
+kms_plane_alpha_blend@pipe-A-alpha-opaque-fb,Fail
+kms_plane_alpha_blend@pipe-A-alpha-transparent-fb,Fail
+kms_plane_alpha_blend@pipe-A-constant-alpha-max,Fail
+kms_plane_alpha_blend@pipe-B-alpha-opaque-fb,Fail
+kms_plane_alpha_blend@pipe-B-alpha-transparent-fb,Fail
+kms_plane_alpha_blend@pipe-B-constant-alpha-max,Fail
+kms_plane_alpha_blend@pipe-C-alpha-opaque-fb,Fail
+kms_plane_alpha_blend@pipe-C-alpha-transparent-fb,Fail
+kms_plane_alpha_blend@pipe-C-constant-alpha-max,Fail
+kms_plane_multiple@tiling-y,Timeout
+kms_pwrite_crc,Timeout
+kms_sysfs_edid_timing,Fail
diff --git a/drivers/gpu/drm/ci/xfails/i915-apl-flakes.txt b/drivers/gpu/drm/ci/xfails/i915-apl-flakes.txt
new file mode 100644
index 000000000000..331c5841bb41
--- /dev/null
+++ b/drivers/gpu/drm/ci/xfails/i915-apl-flakes.txt
@@ -0,0 +1 @@
+kms_frontbuffer_tracking@fbc-tiling-linear
diff --git a/drivers/gpu/drm/ci/xfails/i915-apl-skips.txt b/drivers/gpu/drm/ci/xfails/i915-apl-skips.txt
new file mode 100644
index 000000000000..3430b215c06e
--- /dev/null
+++ b/drivers/gpu/drm/ci/xfails/i915-apl-skips.txt
@@ -0,0 +1,6 @@
+# Suspend to RAM seems to be broken on this machine
+.*suspend.*
+# This is generating kernel oops with divide error
+kms_plane_scaling@invalid-parameters
+# This is cascading issues
+kms_3d
\ No newline at end of file
diff --git a/drivers/gpu/drm/ci/xfails/i915-cml-fails.txt b/drivers/gpu/drm/ci/xfails/i915-cml-fails.txt
new file mode 100644
index 000000000000..6139b410e767
--- /dev/null
+++ b/drivers/gpu/drm/ci/xfails/i915-cml-fails.txt
@@ -0,0 +1,18 @@
+kms_color@ctm-0-25,Fail
+kms_flip_scaled_crc@flip-32bpp-linear-to-64bpp-linear-downscaling,Fail
+kms_flip_scaled_crc@flip-32bpp-linear-to-64bpp-linear-upscaling,Fail
+kms_flip_scaled_crc@flip-32bpp-xtile-to-64bpp-xtile-downscaling,Fail
+kms_flip_scaled_crc@flip-32bpp-xtile-to-64bpp-xtile-upscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-linear-to-16bpp-linear-downscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-linear-to-16bpp-linear-upscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-linear-to-32bpp-linear-downscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-linear-to-32bpp-linear-upscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-xtile-to-16bpp-xtile-downscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-xtile-to-16bpp-xtile-upscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-xtile-to-32bpp-xtile-downscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-xtile-to-32bpp-xtile-upscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytilegen12rcccs-upscaling,Fail
+kms_plane_alpha_blend@alpha-basic,Fail
+kms_plane_alpha_blend@alpha-opaque-fb,Fail
+kms_plane_alpha_blend@alpha-transparent-fb,Fail
+kms_plane_alpha_blend@constant-alpha-max,Fail
diff --git a/drivers/gpu/drm/ci/xfails/i915-cml-flakes.txt b/drivers/gpu/drm/ci/xfails/i915-cml-flakes.txt
new file mode 100644
index 000000000000..0514a7b3fdb0
--- /dev/null
+++ b/drivers/gpu/drm/ci/xfails/i915-cml-flakes.txt
@@ -0,0 +1,38 @@
+kms_bw@linear-tiling-2-displays-1920x1080p
+kms_bw@linear-tiling-2-displays-2560x1440p
+kms_bw@linear-tiling-2-displays-3840x2160p
+kms_bw@linear-tiling-3-displays-1920x1080p
+kms_bw@linear-tiling-3-displays-2560x1440p
+kms_bw@linear-tiling-3-displays-3840x2160p
+kms_bw@linear-tiling-4-displays-1920x1080p
+kms_bw@linear-tiling-4-displays-2560x1440p
+kms_bw@linear-tiling-4-displays-3840x2160p
+kms_draw_crc@draw-method-xrgb8888-render-xtiled
+kms_flip@flip-vs-suspend
+kms_flip_scaled_crc@flip-32bpp-ytile-to-64bpp-ytile-downscaling
+kms_flip_scaled_crc@flip-32bpp-ytile-to-64bpp-ytile-upscaling
+kms_flip_scaled_crc@flip-32bpp-ytileccs-to-64bpp-ytile-downscaling
+kms_flip_scaled_crc@flip-32bpp-ytileccs-to-64bpp-ytile-upscaling
+kms_flip_scaled_crc@flip-64bpp-ytile-to-16bpp-ytile-downscaling
+kms_flip_scaled_crc@flip-64bpp-ytile-to-16bpp-ytile-upscaling
+kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytile-downscaling
+kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytile-upscaling
+kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytilercccs-downscaling
+kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytilercccs-upscaling
+kms_hdr@bpc-switch-suspend
+kms_plane_alpha_blend@constant-alpha-min
+kms_plane_alpha_blend@pipe-A-alpha-basic
+kms_plane_alpha_blend@pipe-A-alpha-opaque-fb
+kms_plane_alpha_blend@pipe-A-alpha-transparent-fb
+kms_plane_alpha_blend@pipe-A-constant-alpha-max
+kms_plane_alpha_blend@pipe-B-alpha-basic
+kms_plane_alpha_blend@pipe-B-alpha-opaque-fb
+kms_plane_alpha_blend@pipe-B-alpha-transparent-fb
+kms_plane_alpha_blend@pipe-B-constant-alpha-max
+kms_plane_alpha_blend@pipe-C-alpha-basic
+kms_plane_alpha_blend@pipe-C-alpha-opaque-fb
+kms_plane_alpha_blend@pipe-C-alpha-transparent-fb
+kms_plane_alpha_blend@pipe-C-constant-alpha-max
+kms_psr2_su@page_flip-NV12
+kms_psr2_su@page_flip-P010
+kms_setmode@basic
diff --git a/drivers/gpu/drm/ci/xfails/i915-cml-skips.txt b/drivers/gpu/drm/ci/xfails/i915-cml-skips.txt
new file mode 100644
index 000000000000..6d3d7ddc377f
--- /dev/null
+++ b/drivers/gpu/drm/ci/xfails/i915-cml-skips.txt
@@ -0,0 +1,2 @@
+# This is generating kernel oops with divide error
+kms_plane_scaling@invalid-parameters
diff --git a/drivers/gpu/drm/ci/xfails/i915-glk-fails.txt b/drivers/gpu/drm/ci/xfails/i915-glk-fails.txt
new file mode 100644
index 000000000000..5bd432e78129
--- /dev/null
+++ b/drivers/gpu/drm/ci/xfails/i915-glk-fails.txt
@@ -0,0 +1,19 @@
+kms_fbcon_fbt@fbc,Fail
+kms_flip_scaled_crc@flip-32bpp-linear-to-64bpp-linear-downscaling,Fail
+kms_flip_scaled_crc@flip-32bpp-linear-to-64bpp-linear-upscaling,Fail
+kms_flip_scaled_crc@flip-32bpp-xtile-to-64bpp-xtile-downscaling,Fail
+kms_flip_scaled_crc@flip-32bpp-xtile-to-64bpp-xtile-upscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-linear-to-16bpp-linear-downscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-linear-to-16bpp-linear-upscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-linear-to-32bpp-linear-downscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-linear-to-32bpp-linear-upscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-xtile-to-16bpp-xtile-downscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-xtile-to-16bpp-xtile-upscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-xtile-to-32bpp-xtile-downscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-xtile-to-32bpp-xtile-upscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytilegen12rcccs-upscaling,Fail
+kms_frontbuffer_tracking@fbcdrrs-tiling-linear,Fail
+kms_plane_alpha_blend@alpha-basic,Fail
+kms_plane_alpha_blend@alpha-opaque-fb,Fail
+kms_plane_alpha_blend@alpha-transparent-fb,Fail
+kms_plane_alpha_blend@constant-alpha-max,Fail
diff --git a/drivers/gpu/drm/ci/xfails/i915-glk-flakes.txt b/drivers/gpu/drm/ci/xfails/i915-glk-flakes.txt
new file mode 100644
index 000000000000..fc41d13a2d56
--- /dev/null
+++ b/drivers/gpu/drm/ci/xfails/i915-glk-flakes.txt
@@ -0,0 +1,41 @@
+kms_bw@linear-tiling-1-displays-3840x2160p
+kms_bw@linear-tiling-2-displays-1920x1080p
+kms_bw@linear-tiling-2-displays-2560x1440p
+kms_bw@linear-tiling-2-displays-3840x2160p
+kms_bw@linear-tiling-3-displays-1920x1080p
+kms_bw@linear-tiling-3-displays-2560x1440p
+kms_bw@linear-tiling-3-displays-3840x2160p
+kms_bw@linear-tiling-4-displays-1920x1080p
+kms_bw@linear-tiling-4-displays-2560x1440p
+kms_bw@linear-tiling-4-displays-3840x2160p
+kms_flip@blocking-wf_vblank
+kms_flip@wf_vblank-ts-check
+kms_flip@wf_vblank-ts-check-interruptible
+kms_flip_scaled_crc@flip-32bpp-ytile-to-64bpp-ytile-downscaling
+kms_flip_scaled_crc@flip-32bpp-ytile-to-64bpp-ytile-upscaling
+kms_flip_scaled_crc@flip-32bpp-ytileccs-to-64bpp-ytile-downscaling
+kms_flip_scaled_crc@flip-32bpp-ytileccs-to-64bpp-ytile-upscaling
+kms_flip_scaled_crc@flip-64bpp-ytile-to-16bpp-ytile-downscaling
+kms_flip_scaled_crc@flip-64bpp-ytile-to-16bpp-ytile-upscaling
+kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytile-downscaling
+kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytile-upscaling
+kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytilercccs-downscaling
+kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytilercccs-upscaling
+kms_frontbuffer_tracking@fbc-tiling-linear
+kms_plane_alpha_blend@pipe-A-alpha-basic
+kms_plane_alpha_blend@pipe-A-alpha-opaque-fb
+kms_plane_alpha_blend@pipe-A-alpha-transparent-fb
+kms_plane_alpha_blend@pipe-A-constant-alpha-max
+kms_plane_alpha_blend@pipe-B-alpha-basic
+kms_plane_alpha_blend@pipe-B-alpha-opaque-fb
+kms_plane_alpha_blend@pipe-B-alpha-transparent-fb
+kms_plane_alpha_blend@pipe-B-constant-alpha-max
+kms_plane_alpha_blend@pipe-C-alpha-basic
+kms_plane_alpha_blend@pipe-C-alpha-opaque-fb
+kms_plane_alpha_blend@pipe-C-alpha-transparent-fb
+kms_plane_alpha_blend@pipe-C-constant-alpha-max
+kms_prop_blob@invalid-set-prop-any
+kms_rotation_crc@multiplane-rotation
+kms_rotation_crc@multiplane-rotation-cropping-bottom
+kms_rotation_crc@multiplane-rotation-cropping-top
+kms_setmode@basic
diff --git a/drivers/gpu/drm/ci/xfails/i915-glk-skips.txt b/drivers/gpu/drm/ci/xfails/i915-glk-skips.txt
new file mode 100644
index 000000000000..4c7d00ce14bc
--- /dev/null
+++ b/drivers/gpu/drm/ci/xfails/i915-glk-skips.txt
@@ -0,0 +1,5 @@
+# Suspend to RAM seems to be broken on this machine
+.*suspend.*
+
+# This is generating kernel oops with divide error
+kms_plane_scaling@invalid-parameters
\ No newline at end of file
diff --git a/drivers/gpu/drm/ci/xfails/i915-kbl-fails.txt b/drivers/gpu/drm/ci/xfails/i915-kbl-fails.txt
new file mode 100644
index 000000000000..56ec021a7679
--- /dev/null
+++ b/drivers/gpu/drm/ci/xfails/i915-kbl-fails.txt
@@ -0,0 +1,25 @@
+kms_bw@linear-tiling-2-displays-2560x1440p,Fail
+kms_bw@linear-tiling-4-displays-2560x1440p,Fail
+kms_flip_scaled_crc@flip-32bpp-linear-to-64bpp-linear-downscaling,Fail
+kms_flip_scaled_crc@flip-32bpp-linear-to-64bpp-linear-upscaling,Fail
+kms_flip_scaled_crc@flip-32bpp-xtile-to-64bpp-xtile-downscaling,Fail
+kms_flip_scaled_crc@flip-32bpp-xtile-to-64bpp-xtile-upscaling,Fail
+kms_flip_scaled_crc@flip-32bpp-ytile-to-64bpp-ytile-upscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-linear-to-16bpp-linear-downscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-linear-to-16bpp-linear-upscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-linear-to-32bpp-linear-downscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-linear-to-32bpp-linear-upscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-xtile-to-16bpp-xtile-downscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-xtile-to-16bpp-xtile-upscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-xtile-to-32bpp-xtile-downscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-xtile-to-32bpp-xtile-upscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytile-downscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytilegen12rcccs-upscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytilercccs-upscaling,Fail
+kms_plane_alpha_blend@alpha-basic,Fail
+kms_plane_alpha_blend@alpha-opaque-fb,Fail
+kms_plane_alpha_blend@alpha-transparent-fb,Fail
+kms_plane_alpha_blend@constant-alpha-max,Fail
+kms_plane_alpha_blend@pipe-A-constant-alpha-max,Fail
+kms_plane_alpha_blend@pipe-B-alpha-opaque-fb,Fail
+kms_plane_alpha_blend@pipe-C-constant-alpha-max,Fail
diff --git a/drivers/gpu/drm/ci/xfails/i915-kbl-flakes.txt b/drivers/gpu/drm/ci/xfails/i915-kbl-flakes.txt
new file mode 100644
index 000000000000..f3ba1c4c5d46
--- /dev/null
+++ b/drivers/gpu/drm/ci/xfails/i915-kbl-flakes.txt
@@ -0,0 +1,26 @@
+kms_async_flips@crc
+kms_bw@linear-tiling-2-displays-1920x1080p
+kms_bw@linear-tiling-2-displays-3840x2160p
+kms_bw@linear-tiling-3-displays-1920x1080p
+kms_bw@linear-tiling-3-displays-2560x1440p
+kms_bw@linear-tiling-3-displays-3840x2160p
+kms_bw@linear-tiling-4-displays-1920x1080p
+kms_bw@linear-tiling-4-displays-3840x2160p
+kms_color@ctm-0-25
+kms_flip_scaled_crc@flip-32bpp-ytile-to-64bpp-ytile-downscaling
+kms_flip_scaled_crc@flip-32bpp-ytileccs-to-64bpp-ytile-downscaling
+kms_flip_scaled_crc@flip-32bpp-ytileccs-to-64bpp-ytile-upscaling
+kms_flip_scaled_crc@flip-64bpp-ytile-to-16bpp-ytile-downscaling
+kms_flip_scaled_crc@flip-64bpp-ytile-to-16bpp-ytile-upscaling
+kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytile-upscaling
+kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytilercccs-downscaling
+kms_plane_alpha_blend@pipe-A-alpha-basic
+kms_plane_alpha_blend@pipe-A-alpha-opaque-fb
+kms_plane_alpha_blend@pipe-A-alpha-transparent-fb
+kms_plane_alpha_blend@pipe-B-alpha-basic
+kms_plane_alpha_blend@pipe-B-alpha-transparent-fb
+kms_plane_alpha_blend@pipe-B-constant-alpha-max
+kms_plane_alpha_blend@pipe-C-alpha-basic
+kms_plane_alpha_blend@pipe-C-alpha-opaque-fb
+kms_plane_alpha_blend@pipe-C-alpha-transparent-fb
+kms_sysfs_edid_timing
diff --git a/drivers/gpu/drm/ci/xfails/i915-kbl-skips.txt b/drivers/gpu/drm/ci/xfails/i915-kbl-skips.txt
new file mode 100644
index 000000000000..4c7d00ce14bc
--- /dev/null
+++ b/drivers/gpu/drm/ci/xfails/i915-kbl-skips.txt
@@ -0,0 +1,5 @@
+# Suspend to RAM seems to be broken on this machine
+.*suspend.*
+
+# This is generating kernel oops with divide error
+kms_plane_scaling@invalid-parameters
\ No newline at end of file
diff --git a/drivers/gpu/drm/ci/xfails/i915-tgl-fails.txt b/drivers/gpu/drm/ci/xfails/i915-tgl-fails.txt
new file mode 100644
index 000000000000..a6da5544e198
--- /dev/null
+++ b/drivers/gpu/drm/ci/xfails/i915-tgl-fails.txt
@@ -0,0 +1,37 @@
+kms_bw@linear-tiling-2-displays-3840x2160p,Fail
+kms_bw@linear-tiling-3-displays-1920x1080p,Fail
+kms_bw@linear-tiling-3-displays-2560x1440p,Fail
+kms_bw@linear-tiling-3-displays-3840x2160p,Fail
+kms_bw@linear-tiling-4-displays-1920x1080p,Fail
+kms_bw@linear-tiling-4-displays-2560x1440p,Fail
+kms_bw@linear-tiling-4-displays-3840x2160p,Fail
+kms_bw@linear-tiling-5-displays-1920x1080p,Fail
+kms_bw@linear-tiling-5-displays-2560x1440p,Fail
+kms_bw@linear-tiling-5-displays-3840x2160p,Fail
+kms_color@ctm-0-25,Fail
+kms_flip@flip-vs-panning-vs-hang,Timeout
+kms_flip_scaled_crc@flip-32bpp-linear-to-64bpp-linear-downscaling,Fail
+kms_flip_scaled_crc@flip-32bpp-linear-to-64bpp-linear-upscaling,Fail
+kms_flip_scaled_crc@flip-32bpp-xtile-to-64bpp-xtile-downscaling,Fail
+kms_flip_scaled_crc@flip-32bpp-xtile-to-64bpp-xtile-upscaling,Fail
+kms_flip_scaled_crc@flip-32bpp-ytile-to-64bpp-ytile-downscaling,Fail
+kms_flip_scaled_crc@flip-32bpp-ytile-to-64bpp-ytile-upscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-linear-to-16bpp-linear-downscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-linear-to-16bpp-linear-upscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-linear-to-32bpp-linear-downscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-linear-to-32bpp-linear-upscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-xtile-to-16bpp-xtile-downscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-xtile-to-16bpp-xtile-upscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-xtile-to-32bpp-xtile-downscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-xtile-to-32bpp-xtile-upscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-ytile-to-16bpp-ytile-downscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-ytile-to-16bpp-ytile-upscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytile-downscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytile-upscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytilegen12rcccs-upscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytilercccs-downscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytilercccs-upscaling,Fail
+kms_rotation_crc@bad-pixel-format,Fail
+kms_rotation_crc@multiplane-rotation,Fail
+kms_rotation_crc@multiplane-rotation-cropping-bottom,Fail
+kms_rotation_crc@multiplane-rotation-cropping-top,Fail
diff --git a/drivers/gpu/drm/ci/xfails/i915-tgl-flakes.txt b/drivers/gpu/drm/ci/xfails/i915-tgl-flakes.txt
new file mode 100644
index 000000000000..1cd910ee06df
--- /dev/null
+++ b/drivers/gpu/drm/ci/xfails/i915-tgl-flakes.txt
@@ -0,0 +1,5 @@
+kms_draw_crc@.*
+kms_flip@blocking-absolute-wf_vblank
+kms_flip@bo-too-big-interruptible
+kms_flip@busy-flip
+kms_flip@flip-vs-rmfb-interruptible
diff --git a/drivers/gpu/drm/ci/xfails/i915-tgl-skips.txt b/drivers/gpu/drm/ci/xfails/i915-tgl-skips.txt
new file mode 100644
index 000000000000..1d0621750b14
--- /dev/null
+++ b/drivers/gpu/drm/ci/xfails/i915-tgl-skips.txt
@@ -0,0 +1,11 @@
+# Suspend to RAM seems to be broken on this machine
+.*suspend.*
+
+# GPU hangs, then the whole machine
+gem_eio.*
+
+# Whole machine hangs
+kms_flip@absolute-wf_vblank@a-edp1
+
+# This is generating kernel oops with divide error
+kms_plane_scaling@invalid-parameters
\ No newline at end of file
diff --git a/drivers/gpu/drm/ci/xfails/i915-whl-fails.txt b/drivers/gpu/drm/ci/xfails/i915-whl-fails.txt
new file mode 100644
index 000000000000..967327ddc1ac
--- /dev/null
+++ b/drivers/gpu/drm/ci/xfails/i915-whl-fails.txt
@@ -0,0 +1,48 @@
+kms_bw@linear-tiling-2-displays-1920x1080p,Fail
+kms_bw@linear-tiling-2-displays-2560x1440p,Fail
+kms_bw@linear-tiling-2-displays-3840x2160p,Fail
+kms_bw@linear-tiling-3-displays-1920x1080p,Fail
+kms_bw@linear-tiling-3-displays-2560x1440p,Fail
+kms_bw@linear-tiling-3-displays-3840x2160p,Fail
+kms_bw@linear-tiling-4-displays-1920x1080p,Fail
+kms_bw@linear-tiling-4-displays-2560x1440p,Fail
+kms_bw@linear-tiling-4-displays-3840x2160p,Fail
+kms_fbcon_fbt@fbc,Fail
+kms_fbcon_fbt@fbc-suspend,Fail
+kms_flip_scaled_crc@flip-32bpp-linear-to-64bpp-linear-downscaling,Fail
+kms_flip_scaled_crc@flip-32bpp-linear-to-64bpp-linear-upscaling,Fail
+kms_flip_scaled_crc@flip-32bpp-xtile-to-64bpp-xtile-downscaling,Fail
+kms_flip_scaled_crc@flip-32bpp-xtile-to-64bpp-xtile-upscaling,Fail
+kms_flip_scaled_crc@flip-32bpp-ytile-to-64bpp-ytile-downscaling,Fail
+kms_flip_scaled_crc@flip-32bpp-ytile-to-64bpp-ytile-upscaling,Fail
+kms_flip_scaled_crc@flip-32bpp-ytileccs-to-64bpp-ytile-downscaling,Fail
+kms_flip_scaled_crc@flip-32bpp-ytileccs-to-64bpp-ytile-upscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-linear-to-16bpp-linear-downscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-linear-to-16bpp-linear-upscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-linear-to-32bpp-linear-downscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-linear-to-32bpp-linear-upscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-xtile-to-16bpp-xtile-downscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-xtile-to-16bpp-xtile-upscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-xtile-to-32bpp-xtile-downscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-xtile-to-32bpp-xtile-upscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-ytile-to-16bpp-ytile-downscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-ytile-to-16bpp-ytile-upscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytile-downscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytile-upscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytilegen12rcccs-upscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytilercccs-downscaling,Fail
+kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytilercccs-upscaling,Fail
+kms_frontbuffer_tracking@fbc-tiling-linear,Fail
+kms_plane_alpha_blend@alpha-basic,Fail
+kms_plane_alpha_blend@alpha-opaque-fb,Fail
+kms_plane_alpha_blend@alpha-transparent-fb,Fail
+kms_plane_alpha_blend@constant-alpha-max,Fail
+kms_plane_alpha_blend@pipe-A-alpha-opaque-fb,Fail
+kms_plane_alpha_blend@pipe-A-alpha-transparent-fb,Fail
+kms_plane_alpha_blend@pipe-A-constant-alpha-max,Fail
+kms_plane_alpha_blend@pipe-B-alpha-opaque-fb,Fail
+kms_plane_alpha_blend@pipe-B-alpha-transparent-fb,Fail
+kms_plane_alpha_blend@pipe-B-constant-alpha-max,Fail
+kms_plane_alpha_blend@pipe-C-alpha-opaque-fb,Fail
+kms_plane_alpha_blend@pipe-C-alpha-transparent-fb,Fail
+kms_plane_alpha_blend@pipe-C-constant-alpha-max,Fail
diff --git a/drivers/gpu/drm/ci/xfails/i915-whl-flakes.txt b/drivers/gpu/drm/ci/xfails/i915-whl-flakes.txt
new file mode 100644
index 000000000000..c33202e7e2a1
--- /dev/null
+++ b/drivers/gpu/drm/ci/xfails/i915-whl-flakes.txt
@@ -0,0 +1 @@
+kms_flip@flip-vs-suspend
diff --git a/drivers/gpu/drm/ci/xfails/i915-whl-skips.txt b/drivers/gpu/drm/ci/xfails/i915-whl-skips.txt
new file mode 100644
index 000000000000..f3be0888a214
--- /dev/null
+++ b/drivers/gpu/drm/ci/xfails/i915-whl-skips.txt
@@ -0,0 +1,2 @@
+# This is generating kernel oops with divide error
+kms_plane_scaling@invalid-parameters
\ No newline at end of file
diff --git a/drivers/gpu/drm/ci/xfails/mediatek-mt8173-fails.txt b/drivers/gpu/drm/ci/xfails/mediatek-mt8173-fails.txt
new file mode 100644
index 000000000000..671916067dba
--- /dev/null
+++ b/drivers/gpu/drm/ci/xfails/mediatek-mt8173-fails.txt
@@ -0,0 +1,29 @@
+kms_3d,Fail
+kms_addfb_basic@addfb25-bad-modifier,Fail
+kms_bw@linear-tiling-1-displays-1920x1080p,Fail
+kms_bw@linear-tiling-1-displays-2560x1440p,Fail
+kms_bw@linear-tiling-1-displays-3840x2160p,Fail
+kms_bw@linear-tiling-2-displays-1920x1080p,Fail
+kms_bw@linear-tiling-2-displays-2560x1440p,Fail
+kms_bw@linear-tiling-2-displays-3840x2160p,Fail
+kms_bw@linear-tiling-3-displays-1920x1080p,Fail
+kms_bw@linear-tiling-3-displays-2560x1440p,Fail
+kms_bw@linear-tiling-3-displays-3840x2160p,Fail
+kms_color@pipe-A-invalid-gamma-lut-sizes,Fail
+kms_color@pipe-B-invalid-gamma-lut-sizes,Fail
+kms_force_connector_basic@force-connector-state,Fail
+kms_force_connector_basic@force-edid,Fail
+kms_force_connector_basic@force-load-detect,Fail
+kms_force_connector_basic@prune-stale-modes,Fail
+kms_invalid_mode@int-max-clock,Fail
+kms_plane_scaling@planes-upscale-20x20,Fail
+kms_plane_scaling@planes-upscale-20x20-downscale-factor-0-25,Fail
+kms_plane_scaling@planes-upscale-20x20-downscale-factor-0-5,Fail
+kms_plane_scaling@planes-upscale-20x20-downscale-factor-0-75,Fail
+kms_plane_scaling@upscale-with-modifier-20x20,Fail
+kms_plane_scaling@upscale-with-pixel-format-20x20,Fail
+kms_plane_scaling@upscale-with-rotation-20x20,Fail
+kms_properties@get_properties-sanity-atomic,Fail
+kms_properties@plane-properties-atomic,Fail
+kms_properties@plane-properties-legacy,Fail
+kms_rmfb@close-fd,Fail
diff --git a/drivers/gpu/drm/ci/xfails/mediatek-mt8173-flakes.txt b/drivers/gpu/drm/ci/xfails/mediatek-mt8173-flakes.txt
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/drivers/gpu/drm/ci/xfails/mediatek-mt8183-fails.txt b/drivers/gpu/drm/ci/xfails/mediatek-mt8183-fails.txt
new file mode 100644
index 000000000000..6ff81d00e84e
--- /dev/null
+++ b/drivers/gpu/drm/ci/xfails/mediatek-mt8183-fails.txt
@@ -0,0 +1,10 @@
+kms_addfb_basic@addfb25-bad-modifier,Fail
+kms_bw@linear-tiling-1-displays-2560x1440p,Fail
+kms_bw@linear-tiling-2-displays-1920x1080p,Fail
+kms_bw@linear-tiling-2-displays-2560x1440p,Fail
+kms_bw@linear-tiling-2-displays-3840x2160p,Fail
+kms_bw@linear-tiling-3-displays-2560x1440p,Fail
+kms_bw@linear-tiling-3-displays-3840x2160p,Fail
+kms_color@pipe-A-invalid-gamma-lut-sizes,Fail
+kms_plane_scaling@upscale-with-rotation-20x20,Fail
+kms_rmfb@close-fd,Fail
\ No newline at end of file
diff --git a/drivers/gpu/drm/ci/xfails/mediatek-mt8183-flakes.txt b/drivers/gpu/drm/ci/xfails/mediatek-mt8183-flakes.txt
new file mode 100644
index 000000000000..208890b79eb0
--- /dev/null
+++ b/drivers/gpu/drm/ci/xfails/mediatek-mt8183-flakes.txt
@@ -0,0 +1,14 @@
+core_setmaster_vs_auth
+kms_bw@linear-tiling-1-displays-1920x1080p
+kms_bw@linear-tiling-1-displays-3840x2160p
+kms_bw@linear-tiling-3-displays-1920x1080p
+kms_cursor_legacy@cursor-vs-flip-atomic
+kms_plane_scaling@invalid-num-scalers
+kms_plane_scaling@planes-upscale-20x20
+kms_plane_scaling@planes-upscale-20x20-downscale-factor-0-5
+kms_plane_scaling@upscale-with-modifier-20x20
+kms_plane_scaling@upscale-with-pixel-format-20x20
+kms_prop_blob@invalid-set-prop-any
+kms_properties@get_properties-sanity-atomic
+kms_properties@plane-properties-atomic
+kms_properties@plane-properties-legacy
\ No newline at end of file
diff --git a/drivers/gpu/drm/ci/xfails/meson-g12b-fails.txt b/drivers/gpu/drm/ci/xfails/meson-g12b-fails.txt
new file mode 100644
index 000000000000..860e702091e2
--- /dev/null
+++ b/drivers/gpu/drm/ci/xfails/meson-g12b-fails.txt
@@ -0,0 +1,12 @@
+kms_3d,Fail
+kms_properties@connector-properties-atomic,Fail
+kms_properties@get_properties-sanity-atomic,Fail
+kms_properties@get_properties-sanity-non-atomic,Fail
+kms_properties@connector-properties-legacy,Fail
+kms_cursor_legacy@forked-bo,Fail
+kms_cursor_legacy@forked-move,Fail
+kms_cursor_legacy@single-bo,Fail
+kms_cursor_legacy@single-move,Fail
+kms_cursor_legacy@torture-bo,Fail
+kms_cursor_legacy@torture-move,Fail
+kms_hdmi_inject@inject-4k,Fail
\ No newline at end of file
diff --git a/drivers/gpu/drm/ci/xfails/meson-g12b-flakes.txt b/drivers/gpu/drm/ci/xfails/meson-g12b-flakes.txt
new file mode 100644
index 000000000000..b63329d06767
--- /dev/null
+++ b/drivers/gpu/drm/ci/xfails/meson-g12b-flakes.txt
@@ -0,0 +1,4 @@
+kms_force_connector_basic@force-connector-state
+kms_force_connector_basic@force-edid
+kms_force_connector_basic@force-load-detect
+kms_force_connector_basic@prune-stale-modes
\ No newline at end of file
diff --git a/drivers/gpu/drm/ci/xfails/msm-apq8016-fails.txt b/drivers/gpu/drm/ci/xfails/msm-apq8016-fails.txt
new file mode 100644
index 000000000000..9981682feab2
--- /dev/null
+++ b/drivers/gpu/drm/ci/xfails/msm-apq8016-fails.txt
@@ -0,0 +1,15 @@
+kms_3d,Fail
+kms_addfb_basic@addfb25-bad-modifier,Fail
+kms_cursor_legacy@all-pipes-forked-bo,Fail
+kms_cursor_legacy@all-pipes-forked-move,Fail
+kms_cursor_legacy@all-pipes-single-bo,Fail
+kms_cursor_legacy@all-pipes-single-move,Fail
+kms_cursor_legacy@all-pipes-torture-bo,Fail
+kms_cursor_legacy@all-pipes-torture-move,Fail
+kms_cursor_legacy@pipe-A-forked-bo,Fail
+kms_cursor_legacy@pipe-A-forked-move,Fail
+kms_cursor_legacy@pipe-A-single-bo,Fail
+kms_cursor_legacy@pipe-A-single-move,Fail
+kms_cursor_legacy@pipe-A-torture-bo,Fail
+kms_cursor_legacy@pipe-A-torture-move,Fail
+kms_hdmi_inject@inject-4k,Fail
diff --git a/drivers/gpu/drm/ci/xfails/msm-apq8016-flakes.txt b/drivers/gpu/drm/ci/xfails/msm-apq8016-flakes.txt
new file mode 100644
index 000000000000..0e3b60d3fade
--- /dev/null
+++ b/drivers/gpu/drm/ci/xfails/msm-apq8016-flakes.txt
@@ -0,0 +1,4 @@
+kms_force_connector_basic@force-connector-state
+kms_force_connector_basic@force-edid
+kms_force_connector_basic@force-load-detect
+kms_force_connector_basic@prune-stale-modes
diff --git a/drivers/gpu/drm/ci/xfails/msm-apq8096-fails.txt b/drivers/gpu/drm/ci/xfails/msm-apq8096-fails.txt
new file mode 100644
index 000000000000..88a1fc0a3b0d
--- /dev/null
+++ b/drivers/gpu/drm/ci/xfails/msm-apq8096-fails.txt
@@ -0,0 +1,2 @@
+kms_3d,Fail
+kms_addfb_basic@addfb25-bad-modifier,Fail
diff --git a/drivers/gpu/drm/ci/xfails/msm-apq8096-flakes.txt b/drivers/gpu/drm/ci/xfails/msm-apq8096-flakes.txt
new file mode 100644
index 000000000000..0e3b60d3fade
--- /dev/null
+++ b/drivers/gpu/drm/ci/xfails/msm-apq8096-flakes.txt
@@ -0,0 +1,4 @@
+kms_force_connector_basic@force-connector-state
+kms_force_connector_basic@force-edid
+kms_force_connector_basic@force-load-detect
+kms_force_connector_basic@prune-stale-modes
diff --git a/drivers/gpu/drm/ci/xfails/msm-apq8096-skips.txt b/drivers/gpu/drm/ci/xfails/msm-apq8096-skips.txt
new file mode 100644
index 000000000000..cd49c8ce2059
--- /dev/null
+++ b/drivers/gpu/drm/ci/xfails/msm-apq8096-skips.txt
@@ -0,0 +1,2 @@
+# Whole machine hangs
+kms_cursor_legacy@all-pipes-torture-move
\ No newline at end of file
diff --git a/drivers/gpu/drm/ci/xfails/msm-sc7180-fails.txt b/drivers/gpu/drm/ci/xfails/msm-sc7180-fails.txt
new file mode 100644
index 000000000000..14adeba3b62d
--- /dev/null
+++ b/drivers/gpu/drm/ci/xfails/msm-sc7180-fails.txt
@@ -0,0 +1,25 @@
+kms_cursor_legacy@cursor-vs-flip-toggle,Fail
+kms_cursor_legacy@cursor-vs-flip-varying-size,Fail
+kms_cursor_legacy@cursorA-vs-flipA-atomic-transitions,Crash
+kms_pipe_crc_basic@compare-crc-sanitycheck-nv12,Fail
+kms_plane@pixel-format,Fail
+kms_plane@pixel-format-source-clamping,Fail
+kms_plane@plane-position-covered,Fail
+kms_plane@plane-position-hole,Fail
+kms_plane@plane-position-hole-dpms,Fail
+kms_plane_alpha_blend@alpha-7efc,Fail
+kms_plane_alpha_blend@coverage-7efc,Fail
+kms_plane_alpha_blend@coverage-vs-premult-vs-constant,Fail
+kms_plane_alpha_blend@pipe-A-alpha-7efc,Fail
+kms_plane_alpha_blend@pipe-A-coverage-7efc,Fail
+kms_plane_alpha_blend@pipe-A-coverage-vs-premult-vs-constant,Fail
+kms_plane_alpha_blend@pipe-B-alpha-7efc,Fail
+kms_plane_alpha_blend@pipe-B-alpha-basic,Fail
+kms_plane_alpha_blend@pipe-B-alpha-opaque-fb,Fail
+kms_plane_alpha_blend@pipe-B-constant-alpha-max,Fail
+kms_plane_alpha_blend@pipe-B-constant-alpha-mid,Fail
+kms_plane_alpha_blend@pipe-B-coverage-7efc,Fail
+kms_plane_alpha_blend@pipe-B-coverage-vs-premult-vs-constant,Fail
+kms_rmfb@close-fd,Fail
+kms_universal_plane@disable-primary-vs-flip-pipe-b,Fail
+kms_universal_plane@universal-plane-pipe-B-sanity,Fail
diff --git a/drivers/gpu/drm/ci/xfails/msm-sc7180-flakes.txt b/drivers/gpu/drm/ci/xfails/msm-sc7180-flakes.txt
new file mode 100644
index 000000000000..636563d3e59a
--- /dev/null
+++ b/drivers/gpu/drm/ci/xfails/msm-sc7180-flakes.txt
@@ -0,0 +1,7 @@
+
+# Test ends up reading CRC from frame before cursor update
+# bug
+# sometimes.. tbd if this is a kernel CRC bug or a test
+kms_cursor_crc@.*
+kms_plane_multiple@atomic-pipe-A-tiling-none
+kms_atomic_transition@modeset-transition-nonblocking-fencing,Fail
\ No newline at end of file
diff --git a/drivers/gpu/drm/ci/xfails/msm-sc7180-skips.txt b/drivers/gpu/drm/ci/xfails/msm-sc7180-skips.txt
new file mode 100644
index 000000000000..410e0eeb3161
--- /dev/null
+++ b/drivers/gpu/drm/ci/xfails/msm-sc7180-skips.txt
@@ -0,0 +1,23 @@
+# Suspend to RAM seems to be broken on this machine
+.*suspend.*
+
+# Test incorrectly assumes that CTM support implies gamma/degamma
+# LUT support.  None of the subtests handle the case of only having
+# CTM support
+kms_color.*
+
+# 4k@60 is not supported on this hw, but driver doesn't handle it
+# too gracefully.. https://gitlab.freedesktop.org/drm/msm/-/issues/15
+kms_bw@linear-tiling-.*-displays-3840x2160p
+
+# Until igt fix lands: https://patchwork.freedesktop.org/patch/493175/
+kms_bw@linear-tiling-2.*
+kms_bw@linear-tiling-3.*
+kms_bw@linear-tiling-4.*
+kms_bw@linear-tiling-5.*
+kms_bw@linear-tiling-6.*
+
+# igt fix posted: https://patchwork.freedesktop.org/patch/499926/
+# failure mode is flakey due to randomization but fails frequently
+# enough to be detected as a Crash or occasionally UnexpectedPass.
+kms_plane_multiple@atomic-pipe-A-tiling-none
diff --git a/drivers/gpu/drm/ci/xfails/msm-sdm845-fails.txt b/drivers/gpu/drm/ci/xfails/msm-sdm845-fails.txt
new file mode 100644
index 000000000000..09c0c623cd75
--- /dev/null
+++ b/drivers/gpu/drm/ci/xfails/msm-sdm845-fails.txt
@@ -0,0 +1,68 @@
+kms_color@ctm-0-25,Fail
+kms_color@ctm-0-50,Fail
+kms_color@ctm-0-75,Fail
+kms_color@ctm-blue-to-red,Fail
+kms_color@ctm-green-to-red,Fail
+kms_color@ctm-negative,Fail
+kms_color@ctm-red-to-blue,Fail
+kms_color@ctm-signed,Fail
+kms_color@pipe-A-ctm-0-25,Fail
+kms_color@pipe-A-ctm-0-5,Fail
+kms_color@pipe-A-ctm-0-75,Fail
+kms_color@pipe-A-ctm-blue-to-red,Fail
+kms_color@pipe-A-ctm-green-to-red,Fail
+kms_color@pipe-A-ctm-max,Fail
+kms_color@pipe-A-ctm-negative,Fail
+kms_color@pipe-A-ctm-red-to-blue,Fail
+kms_color@pipe-A-legacy-gamma,Fail
+kms_cursor_legacy@basic-flip-after-cursor-atomic,Fail
+kms_cursor_legacy@basic-flip-after-cursor-legacy,Fail
+kms_cursor_legacy@basic-flip-after-cursor-varying-size,Fail
+kms_cursor_legacy@basic-flip-before-cursor-atomic,Fail
+kms_cursor_legacy@basic-flip-before-cursor-legacy,Fail
+kms_cursor_legacy@basic-flip-before-cursor-varying-size,Fail
+kms_cursor_legacy@cursor-vs-flip-atomic,Fail
+kms_cursor_legacy@cursor-vs-flip-atomic-transitions,Fail
+kms_cursor_legacy@cursor-vs-flip-atomic-transitions-varying-size,Fail
+kms_cursor_legacy@cursor-vs-flip-legacy,Fail
+kms_cursor_legacy@cursor-vs-flip-toggle,Fail
+kms_cursor_legacy@cursor-vs-flip-varying-size,Fail
+kms_cursor_legacy@cursorA-vs-flipA-toggle,Fail
+kms_cursor_legacy@flip-vs-cursor-atomic,Fail
+kms_cursor_legacy@flip-vs-cursor-crc-atomic,Fail
+kms_cursor_legacy@flip-vs-cursor-crc-legacy,Fail
+kms_cursor_legacy@flip-vs-cursor-legacy,Fail
+kms_cursor_legacy@short-flip-after-cursor-atomic-transitions,Fail
+kms_cursor_legacy@short-flip-after-cursor-atomic-transitions-varying-size,Fail
+kms_cursor_legacy@short-flip-after-cursor-toggle,Fail
+kms_cursor_legacy@short-flip-before-cursor-atomic-transitions,Fail
+kms_cursor_legacy@short-flip-before-cursor-atomic-transitions-varying-size,Fail
+kms_pipe_crc_basic@compare-crc-sanitycheck-nv12,Fail
+kms_plane@pixel-format,Fail
+kms_plane@pixel-format-source-clamping,Fail
+kms_plane_alpha_blend@alpha-7efc,Fail
+kms_plane_alpha_blend@coverage-7efc,Fail
+kms_plane_alpha_blend@coverage-vs-premult-vs-constant,Fail
+kms_plane_alpha_blend@pipe-A-alpha-7efc,Fail
+kms_plane_alpha_blend@pipe-A-coverage-7efc,Fail
+kms_plane_alpha_blend@pipe-A-coverage-vs-premult-vs-constant,Fail
+kms_plane_cursor@overlay,Fail
+kms_plane_cursor@pipe-A-overlay-size-128,Fail
+kms_plane_cursor@pipe-A-overlay-size-256,Fail
+kms_plane_cursor@pipe-A-overlay-size-64,Fail
+kms_plane_cursor@pipe-A-viewport-size-128,Fail
+kms_plane_cursor@pipe-A-viewport-size-256,Fail
+kms_plane_cursor@pipe-A-viewport-size-64,Fail
+kms_plane_cursor@viewport,Fail
+kms_plane_scaling@downscale-with-pixel-format-factor-0-25,Timeout
+kms_plane_scaling@downscale-with-pixel-format-factor-0-5,Timeout
+kms_plane_scaling@downscale-with-pixel-format-factor-0-75,Timeout
+kms_plane_scaling@plane-downscale-with-pixel-format-factor-0-25,Timeout
+kms_plane_scaling@plane-downscale-with-pixel-format-factor-0-5,Timeout
+kms_plane_scaling@plane-downscale-with-pixel-format-factor-0-75,Timeout
+kms_plane_scaling@plane-scaler-with-clipping-clamping-pixel-formats,Timeout
+kms_plane_scaling@plane-scaler-with-pixel-format-unity-scaling,Timeout
+kms_plane_scaling@planes-downscale-factor-0-25,Fail
+kms_plane_scaling@scaler-with-clipping-clamping,Timeout
+kms_plane_scaling@scaler-with-pixel-format-unity-scaling,Timeout
+kms_rmfb@close-fd,Fail
diff --git a/drivers/gpu/drm/ci/xfails/msm-sdm845-flakes.txt b/drivers/gpu/drm/ci/xfails/msm-sdm845-flakes.txt
new file mode 100644
index 000000000000..5b3aaab7ac3f
--- /dev/null
+++ b/drivers/gpu/drm/ci/xfails/msm-sdm845-flakes.txt
@@ -0,0 +1,11 @@
+
+
+# Test ends up reading CRC from frame before cursor update
+# bug
+# sometimes.. tbd if this is a kernel CRC bug or a test
+kms_cursor_crc@.*
+kms_cursor_legacy@flip-vs-cursor-toggle
+kms_cursor_legacy@pipe-A-forked-bo
+kms_cursor_legacy@pipe-A-forked-move
+kms_cursor_legacy@short-flip-before-cursor-toggle
+kms_flip@dpms-vs-vblank-race-interruptible
diff --git a/drivers/gpu/drm/ci/xfails/msm-sdm845-skips.txt b/drivers/gpu/drm/ci/xfails/msm-sdm845-skips.txt
new file mode 100644
index 000000000000..42675f1c6d76
--- /dev/null
+++ b/drivers/gpu/drm/ci/xfails/msm-sdm845-skips.txt
@@ -0,0 +1,2 @@
+# Hangs machine
+kms_bw.*
\ No newline at end of file
diff --git a/drivers/gpu/drm/ci/xfails/rockchip-rk3288-fails.txt b/drivers/gpu/drm/ci/xfails/rockchip-rk3288-fails.txt
new file mode 100644
index 000000000000..2a1baa948e12
--- /dev/null
+++ b/drivers/gpu/drm/ci/xfails/rockchip-rk3288-fails.txt
@@ -0,0 +1,48 @@
+kms_3d,Crash
+kms_bw@linear-tiling-2-displays-1920x1080p,Fail
+kms_bw@linear-tiling-2-displays-2560x1440p,Fail
+kms_bw@linear-tiling-2-displays-3840x2160p,Fail
+kms_bw@linear-tiling-3-displays-1920x1080p,Fail
+kms_bw@linear-tiling-3-displays-2560x1440p,Fail
+kms_bw@linear-tiling-3-displays-3840x2160p,Fail
+kms_force_connector_basic@force-load-detect,Fail
+kms_invalid_mode@int-max-clock,Crash
+kms_plane@pixel-format,Crash
+kms_plane@pixel-format-source-clamping,Crash
+kms_plane@plane-position-hole,Crash
+kms_plane@plane-position-hole-dpms,Crash
+kms_plane_cursor@overlay,Crash
+kms_plane_cursor@pipe-A-overlay-size-128,Fail
+kms_plane_cursor@pipe-A-overlay-size-256,Fail
+kms_plane_cursor@pipe-A-overlay-size-64,Fail
+kms_plane_cursor@pipe-A-primary-size-128,Fail
+kms_plane_cursor@pipe-A-primary-size-256,Fail
+kms_plane_cursor@pipe-A-primary-size-64,Fail
+kms_plane_cursor@pipe-A-viewport-size-128,Fail
+kms_plane_cursor@pipe-A-viewport-size-256,Fail
+kms_plane_cursor@pipe-A-viewport-size-64,Fail
+kms_plane_cursor@pipe-B-overlay-size-128,Fail
+kms_plane_cursor@pipe-B-overlay-size-256,Fail
+kms_plane_cursor@pipe-B-overlay-size-64,Fail
+kms_plane_cursor@pipe-B-primary-size-128,Fail
+kms_plane_cursor@pipe-B-primary-size-256,Fail
+kms_plane_cursor@pipe-B-primary-size-64,Fail
+kms_plane_cursor@pipe-B-viewport-size-128,Fail
+kms_plane_cursor@pipe-B-viewport-size-256,Fail
+kms_plane_cursor@pipe-B-viewport-size-64,Fail
+kms_plane_cursor@primary,Crash
+kms_plane_cursor@viewport,Crash
+kms_plane_lowres@tiling-none,Fail
+kms_plane_scaling@downscale-with-modifier-factor-0-25,Fail
+kms_plane_scaling@downscale-with-rotation-factor-0-25,Fail
+kms_plane_scaling@upscale-with-modifier-20x20,Fail
+kms_plane_scaling@upscale-with-modifier-factor-0-25,Fail
+kms_plane_scaling@upscale-with-pixel-format-20x20,Fail
+kms_plane_scaling@upscale-with-pixel-format-factor-0-25,Fail
+kms_plane_scaling@upscale-with-rotation-20x20,Fail
+kms_prime@basic-crc,Fail
+kms_properties@connector-properties-atomic,Crash
+kms_properties@connector-properties-legacy,Crash
+kms_properties@get_properties-sanity-atomic,Crash
+kms_properties@get_properties-sanity-non-atomic,Crash
+kms_setmode@invalid-clone-single-crtc,Crash
diff --git a/drivers/gpu/drm/ci/xfails/rockchip-rk3288-flakes.txt b/drivers/gpu/drm/ci/xfails/rockchip-rk3288-flakes.txt
new file mode 100644
index 000000000000..45c54c75c899
--- /dev/null
+++ b/drivers/gpu/drm/ci/xfails/rockchip-rk3288-flakes.txt
@@ -0,0 +1,9 @@
+kms_addfb_basic@addfb25-bad-modifier
+kms_cursor_crc@.*
+kms_flip@basic-flip-vs-wf_vblank
+kms_invalid_mode@int-max-clock,Crash
+kms_pipe_crc_basic@.*
+kms_properties@connector-properties-atomic,Crash
+kms_properties@get_properties-sanity-atomic,Crash
+kms_properties@get_properties-sanity-non-atomic,Crash
+kms_rmfb@close-fd
diff --git a/drivers/gpu/drm/ci/xfails/rockchip-rk3288-skips.txt b/drivers/gpu/drm/ci/xfails/rockchip-rk3288-skips.txt
new file mode 100644
index 000000000000..f20c3574b75a
--- /dev/null
+++ b/drivers/gpu/drm/ci/xfails/rockchip-rk3288-skips.txt
@@ -0,0 +1,52 @@
+# Suspend to RAM seems to be broken on this machine
+.*suspend.*
+
+# Too unstable, machine ends up hanging after lots of Oopses
+kms_cursor_legacy.*
+
+# Started hanging the machine on Linux 5.19-rc2:
+#
+# [IGT] kms_plane_lowres: executing
+# [IGT] kms_plane_lowres: starting subtest pipe-F-tiling-y
+# [IGT] kms_plane_lowres: exiting, ret=77
+# Console: switching to colour frame buffer device 170x48
+# rockchip-drm display-subsystem: [drm] *ERROR* flip_done timed out
+# rockchip-drm display-subsystem: [drm] *ERROR* [CRTC:35:crtc-0] commit wait timed out
+# BUG: spinlock bad magic on CPU#3, kms_plane_lowre/482
+# 8<--- cut here ---
+# Unable to handle kernel paging request at virtual address 7812078e
+# [7812078e] *pgd=00000000
+# Internal error: Oops: 5 [#1] SMP ARM
+# Modules linked in:
+# CPU: 3 PID: 482 Comm: kms_plane_lowre Tainted: G        W         5.19.0-rc2-323596-g00535de92171 #1
+# Hardware name: Rockchip (Device Tree)
+# Process kms_plane_lowre (pid: 482, stack limit = 0x1193ac2b)
+#  spin_dump from do_raw_spin_lock+0xa4/0xe8
+#  do_raw_spin_lock from wait_for_completion_timeout+0x2c/0x120
+#  wait_for_completion_timeout from drm_crtc_commit_wait+0x18/0x7c
+#  drm_crtc_commit_wait from drm_atomic_helper_wait_for_dependencies+0x44/0x168
+#  drm_atomic_helper_wait_for_dependencies from commit_tail+0x34/0x180
+#  commit_tail from drm_atomic_helper_commit+0x164/0x18c
+#  drm_atomic_helper_commit from drm_atomic_commit+0xac/0xe4
+#  drm_atomic_commit from drm_client_modeset_commit_atomic+0x23c/0x284
+#  drm_client_modeset_commit_atomic from drm_client_modeset_commit_locked+0x60/0x1c8
+#  drm_client_modeset_commit_locked from drm_client_modeset_commit+0x24/0x40
+#  drm_client_modeset_commit from drm_fbdev_client_restore+0x58/0x94
+#  drm_fbdev_client_restore from drm_client_dev_restore+0x70/0xbc
+#  drm_client_dev_restore from drm_release+0xf4/0x114
+#  drm_release from __fput+0x74/0x240
+#  __fput from task_work_run+0x84/0xb4
+#  task_work_run from do_exit+0x34c/0xa20
+#  do_exit from do_group_exit+0x34/0x98
+#  do_group_exit from __wake_up_parent+0x0/0x18
+# Code: e595c008 12843d19 03e00000 03093168 (15940508)
+# ---[ end trace 0000000000000000 ]---
+# note: kms_plane_lowre[482] exited with preempt_count 1
+# Fixing recursive fault but reboot is needed!
+kms_plane_lowres@pipe-F-tiling-y
+
+# Take too long, we have only two machines, and these are very flaky
+kms_cursor_crc.*
+
+# Machine is hanging in this test, so skip it
+kms_pipe_crc_basic@disable-crc-after-crtc
\ No newline at end of file
diff --git a/drivers/gpu/drm/ci/xfails/rockchip-rk3399-fails.txt b/drivers/gpu/drm/ci/xfails/rockchip-rk3399-fails.txt
new file mode 100644
index 000000000000..6db08ba6b008
--- /dev/null
+++ b/drivers/gpu/drm/ci/xfails/rockchip-rk3399-fails.txt
@@ -0,0 +1,37 @@
+kms_color@legacy-gamma,Fail
+kms_color@pipe-A-legacy-gamma,Fail
+kms_color@pipe-B-legacy-gamma,Fail
+kms_flip@basic-flip-vs-wf_vblank,Fail
+kms_flip@blocking-wf_vblank,Fail
+kms_flip@dpms-vs-vblank-race,Fail
+kms_flip@flip-vs-absolute-wf_vblank,Fail
+kms_flip@flip-vs-absolute-wf_vblank-interruptible,Fail
+kms_flip@flip-vs-blocking-wf-vblank,Fail
+kms_flip@flip-vs-panning,Fail
+kms_flip@flip-vs-panning-interruptible,Fail
+kms_flip@flip-vs-wf_vblank-interruptible,Fail
+kms_flip@plain-flip-fb-recreate,Fail
+kms_flip@plain-flip-fb-recreate-interruptible,Fail
+kms_flip@plain-flip-ts-check,Fail
+kms_flip@plain-flip-ts-check-interruptible,Fail
+kms_flip@wf_vblank-ts-check,Fail
+kms_flip@wf_vblank-ts-check-interruptible,Fail
+kms_invalid_mode@int-max-clock,Fail
+kms_plane@pixel-format,Fail
+kms_plane@pixel-format-source-clamping,Fail
+kms_plane@plane-panning-bottom-right,Fail
+kms_plane@plane-panning-top-left,Fail
+kms_plane@plane-position-covered,Fail
+kms_plane_cursor@pipe-B-overlay-size-128,Fail
+kms_plane_cursor@pipe-B-overlay-size-256,Fail
+kms_plane_cursor@pipe-B-overlay-size-64,Fail
+kms_plane_cursor@pipe-B-primary-size-128,Fail
+kms_plane_cursor@pipe-B-primary-size-256,Fail
+kms_plane_cursor@pipe-B-primary-size-64,Fail
+kms_plane_cursor@pipe-B-viewport-size-128,Fail
+kms_plane_cursor@pipe-B-viewport-size-256,Fail
+kms_plane_cursor@pipe-B-viewport-size-64,Fail
+kms_plane_multiple@atomic-pipe-B-tiling-none,Fail
+kms_plane_multiple@tiling-none,Fail
+kms_prime@basic-crc,Fail
+kms_rmfb@close-fd,Fail
diff --git a/drivers/gpu/drm/ci/xfails/rockchip-rk3399-flakes.txt b/drivers/gpu/drm/ci/xfails/rockchip-rk3399-flakes.txt
new file mode 100644
index 000000000000..4c0539b4beaf
--- /dev/null
+++ b/drivers/gpu/drm/ci/xfails/rockchip-rk3399-flakes.txt
@@ -0,0 +1,23 @@
+
+kms_cursor_crc@.*
+kms_flip@dpms-vs-vblank-race-interruptible
+kms_flip@flip-vs-expired-vblank
+kms_flip@modeset-vs-vblank-race-interruptible
+kms_pipe_crc_basic@.*
+kms_pipe_crc_basic@compare-crc-sanitycheck-pipe-A
+kms_pipe_crc_basic@compare-crc-sanitycheck-pipe-B
+kms_plane@plane-position-hole
+kms_plane_multiple@atomic-pipe-A-tiling-none
+kms_plane_multiple@atomic-pipe-B-tiling-none
+kms_sequence@get-forked
+kms_sequence@get-forked-busy
+kms_setmode@basic
+kms_universal_plane@universal-plane-pipe-B-functional,UnexpectedPass
+kms_vblank@pipe-A-accuracy-idle
+kms_vblank@pipe-A-query-busy
+kms_vblank@pipe-A-query-forked-busy
+kms_vblank@pipe-A-wait-idle
+kms_vblank@pipe-B-accuracy-idle
+kms_vblank@pipe-B-query-busy
+kms_vblank@pipe-B-query-forked-busy
+kms_vblank@pipe-B-wait-idle
diff --git a/drivers/gpu/drm/ci/xfails/rockchip-rk3399-skips.txt b/drivers/gpu/drm/ci/xfails/rockchip-rk3399-skips.txt
new file mode 100644
index 000000000000..10c3d81a919a
--- /dev/null
+++ b/drivers/gpu/drm/ci/xfails/rockchip-rk3399-skips.txt
@@ -0,0 +1,5 @@
+# Suspend to RAM seems to be broken on this machine
+.*suspend.*
+
+# Too unstable, machine ends up hanging after lots of Oopses
+kms_cursor_legacy.*
diff --git a/drivers/gpu/drm/ci/xfails/virtio_gpu-none-fails.txt b/drivers/gpu/drm/ci/xfails/virtio_gpu-none-fails.txt
new file mode 100644
index 000000000000..9586b2339f6f
--- /dev/null
+++ b/drivers/gpu/drm/ci/xfails/virtio_gpu-none-fails.txt
@@ -0,0 +1,38 @@
+kms_addfb_basic@addfb25-bad-modifier,Fail
+kms_addfb_basic@bad-pitch-65536,Fail
+kms_addfb_basic@bo-too-small,Fail
+kms_addfb_basic@size-max,Fail
+kms_addfb_basic@too-high,Fail
+kms_atomic_transition@plane-primary-toggle-with-vblank-wait,Fail
+kms_bw@linear-tiling-1-displays-1920x1080p,Fail
+kms_bw@linear-tiling-1-displays-2560x1440p,Fail
+kms_bw@linear-tiling-1-displays-3840x2160p,Fail
+kms_bw@linear-tiling-2-displays-1920x1080p,Fail
+kms_bw@linear-tiling-2-displays-2560x1440p,Fail
+kms_bw@linear-tiling-2-displays-3840x2160p,Fail
+kms_invalid_mode@int-max-clock,Fail
+kms_plane_scaling@downscale-with-modifier-factor-0-25,Fail
+kms_plane_scaling@downscale-with-rotation-factor-0-25,Fail
+kms_plane_scaling@planes-upscale-20x20,Fail
+kms_plane_scaling@planes-upscale-20x20-downscale-factor-0-25,Fail
+kms_plane_scaling@planes-upscale-20x20-downscale-factor-0-5,Fail
+kms_plane_scaling@planes-upscale-20x20-downscale-factor-0-75,Fail
+kms_plane_scaling@upscale-with-modifier-20x20,Fail
+kms_plane_scaling@upscale-with-modifier-factor-0-25,Fail
+kms_plane_scaling@upscale-with-pixel-format-20x20,Fail
+kms_plane_scaling@upscale-with-pixel-format-factor-0-25,Fail
+kms_plane_scaling@upscale-with-rotation-20x20,Fail
+kms_vblank@crtc-id,Fail
+kms_vblank@invalid,Fail
+kms_vblank@pipe-A-accuracy-idle,Fail
+kms_vblank@pipe-A-query-busy,Fail
+kms_vblank@pipe-A-query-forked,Fail
+kms_vblank@pipe-A-query-forked-busy,Fail
+kms_vblank@pipe-A-query-idle,Fail
+kms_vblank@pipe-A-ts-continuation-idle,Fail
+kms_vblank@pipe-A-ts-continuation-modeset,Fail
+kms_vblank@pipe-A-ts-continuation-suspend,Fail
+kms_vblank@pipe-A-wait-busy,Fail
+kms_vblank@pipe-A-wait-forked,Fail
+kms_vblank@pipe-A-wait-forked-busy,Fail
+kms_vblank@pipe-A-wait-idle,Fail
diff --git a/drivers/gpu/drm/ci/xfails/virtio_gpu-none-flakes.txt b/drivers/gpu/drm/ci/xfails/virtio_gpu-none-flakes.txt
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/drivers/gpu/drm/ci/xfails/virtio_gpu-none-skips.txt b/drivers/gpu/drm/ci/xfails/virtio_gpu-none-skips.txt
new file mode 100644
index 000000000000..78be18174012
--- /dev/null
+++ b/drivers/gpu/drm/ci/xfails/virtio_gpu-none-skips.txt
@@ -0,0 +1,6 @@
+# Hits a "refcount_t: underflow; use-after-free" in virtio_gpu_fence_event_process
+# When run in a particular order with other tests
+kms_cursor_legacy.*
+
+# Job just hangs without any output
+kms_flip@flip-vs-suspend.*
\ No newline at end of file
diff --git a/test b/test
new file mode 100644
index 000000000000..e69de29bb2d1
-- 
2.34.1

On Fri, Aug 11, 2023 at 02:19:53PM -0300, Helen Koike wrote:
> From: Tomeu Vizoso <tomeu.vizoso@collabora.com>
> 
> Developers can easily execute several tests on different devices
> by just pushing their branch to their fork in a repository hosted
> on gitlab.freedesktop.org which has an infrastructure to run jobs
> in several runners and farms with different devices.
> 
> There are also other automated tools that uprev dependencies,
> monitor the infra, and so on that are already used by the Mesa
> project, and we can reuse them too.
> 
> Also, store expectations about what the DRM drivers are supposed
> to pass in the IGT test suite. By storing the test expectations
> along with the code, we can make sure both stay in sync with each
> other so we can know when a code change breaks those expectations.
> 
> Also, include a configuration file that points to the out-of-tree
> CI scripts.
> 
> This will allow all contributors to drm to reuse the infrastructure
> already in gitlab.freedesktop.org to test the driver on several
> generations of the hardware.
> 
> Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
> Signed-off-by: Helen Koike <helen.koike@collabora.com>
> Acked-by: Daniel Stone <daniels@collabora.com>
> Acked-by: Rob Clark <robdclark@gmail.com>
> Tested-by: Rob Clark <robdclark@gmail.com>

Ok I pushed this into a topic/drm-ci branch in drm.git and asked sfr to
include that branch in linux-next.

But also I'd like to see a lot more acks here, we should be able to at
least pile up a bunch of (driver) maintainers from drm-misc in support of
this. Also maybe media, at least I've heard noises that they're maybe
interested too? Plus anyone else, the more the better.

Cheers, Sima

> ---
> 
> You can also see this patch on
> https://gitlab.freedesktop.org/helen.fornazier/linux/-/tree/drm-ci-tests
> 
> Thanks!
> 
> v2:
>   - Fix names of result expectation files to match SoC
>   - Don't execute tests that are going to skip on all boards
> 
> v3:
>   - Remove tracking of dmesg output during test execution
> 
> v4:
>   - Move up to drivers/gpu/drm
>   - Add support for a bunch of other drivers
>   - Explain how to incorporate fixes for CI from a
>     ${TARGET_BRANCH}-external-fixes branch
>   - Remove tests that pass from expected results file, to reduce the
>     size of in-tree files
>   - Add docs about how to deal with outages in automated testing labs
>   - Specify the exact SHA of the CI scripts to be used
> 
> v5:
>   - Remove unneeded skips from Meson expectations file
>   - Use a more advanced runner that detects flakes automatically
>   - Use a more succint format for the expectations
>   - Run many more tests (and use sharding to finish in time)
>   - Use skip lists to avoid hanging machines
>   - Add some build testing
>   - Build IGT in each pipeline for faster uprevs
>   - List failures in the GitLab UI
> 
> v6:
>   - Rebase on top of latest drm-next
>   - Lower priority of LAVA jobs to not impact Mesa CI as much
>   - Update docs
> 
> v7:
>   - Rebase on top of latest drm-next
> 
> v8:
>   - Move all files specific to testing the kernel into the kernel tree
>     (thus I have dropped the r-bs I had collected so far)
>   - Uprev Gitlab CI infrastructure scripts to the latest from Mesa
>   - Add MAINTAINERS entry
>   - Fix boot on MT8173 by adding some Kconfigs that are now needed
>   - Link to the docs from index.rst and hard-wrap the file
> 
> v9:
>   - Only automatically run the pipelines for merge requests
>   - Switch to zstd for the build artifacts to align with Mesa
>   - Add Qcom USB PHYs to config as they are now =m in the defconfig
> 
> v10:
>   - Include ci yml files from mesa/mesa (where the development is
>     current active) instead of a spin off project.
>   - Uprev Gitlab CI infrastructure scripts to the latest from Mesa
>   - Update MAINTAINERS entry
>   - Uprev igt tool
>   - add LAVA_JOB_PRIORITY: 30
>   - pipeline example:
>   https://gitlab.freedesktop.org/helen.fornazier/linux/-/pipelines/940506
> 
> v11:
>   - Fix dts paths
>   - Update xfails
>   - Remove meanless comment
>   - Disable clang-format job
> 
> ---
>  Documentation/gpu/automated_testing.rst       |  144 +
>  Documentation/gpu/index.rst                   |    1 +
>  MAINTAINERS                                   |    8 +
>  drivers/gpu/drm/ci/arm.config                 |   69 +
>  drivers/gpu/drm/ci/arm64.config               |  199 ++
>  drivers/gpu/drm/ci/build-igt.sh               |   35 +
>  drivers/gpu/drm/ci/build.sh                   |  157 +
>  drivers/gpu/drm/ci/build.yml                  |  110 +
>  drivers/gpu/drm/ci/check-patch.py             |   57 +
>  drivers/gpu/drm/ci/container.yml              |   65 +
>  drivers/gpu/drm/ci/gitlab-ci.yml              |  251 ++
>  drivers/gpu/drm/ci/igt_runner.sh              |   77 +
>  drivers/gpu/drm/ci/image-tags.yml             |   15 +
>  drivers/gpu/drm/ci/lava-submit.sh             |   57 +
>  drivers/gpu/drm/ci/static-checks.yml          |   12 +
>  drivers/gpu/drm/ci/test.yml                   |  335 ++
>  drivers/gpu/drm/ci/testlist.txt               | 2912 +++++++++++++++++
>  drivers/gpu/drm/ci/x86_64.config              |  111 +
>  .../gpu/drm/ci/xfails/amdgpu-stoney-fails.txt |   19 +
>  .../drm/ci/xfails/amdgpu-stoney-flakes.txt    |   21 +
>  .../gpu/drm/ci/xfails/amdgpu-stoney-skips.txt |    2 +
>  drivers/gpu/drm/ci/xfails/i915-amly-fails.txt |   17 +
>  .../gpu/drm/ci/xfails/i915-amly-flakes.txt    |   32 +
>  drivers/gpu/drm/ci/xfails/i915-amly-skips.txt |    4 +
>  drivers/gpu/drm/ci/xfails/i915-apl-fails.txt  |   58 +
>  drivers/gpu/drm/ci/xfails/i915-apl-flakes.txt |    1 +
>  drivers/gpu/drm/ci/xfails/i915-apl-skips.txt  |    6 +
>  drivers/gpu/drm/ci/xfails/i915-cml-fails.txt  |   18 +
>  drivers/gpu/drm/ci/xfails/i915-cml-flakes.txt |   38 +
>  drivers/gpu/drm/ci/xfails/i915-cml-skips.txt  |    2 +
>  drivers/gpu/drm/ci/xfails/i915-glk-fails.txt  |   19 +
>  drivers/gpu/drm/ci/xfails/i915-glk-flakes.txt |   41 +
>  drivers/gpu/drm/ci/xfails/i915-glk-skips.txt  |    5 +
>  drivers/gpu/drm/ci/xfails/i915-kbl-fails.txt  |   25 +
>  drivers/gpu/drm/ci/xfails/i915-kbl-flakes.txt |   26 +
>  drivers/gpu/drm/ci/xfails/i915-kbl-skips.txt  |    5 +
>  drivers/gpu/drm/ci/xfails/i915-tgl-fails.txt  |   37 +
>  drivers/gpu/drm/ci/xfails/i915-tgl-flakes.txt |    5 +
>  drivers/gpu/drm/ci/xfails/i915-tgl-skips.txt  |   11 +
>  drivers/gpu/drm/ci/xfails/i915-whl-fails.txt  |   48 +
>  drivers/gpu/drm/ci/xfails/i915-whl-flakes.txt |    1 +
>  drivers/gpu/drm/ci/xfails/i915-whl-skips.txt  |    2 +
>  .../drm/ci/xfails/mediatek-mt8173-fails.txt   |   29 +
>  .../drm/ci/xfails/mediatek-mt8173-flakes.txt  |    0
>  .../drm/ci/xfails/mediatek-mt8183-fails.txt   |   10 +
>  .../drm/ci/xfails/mediatek-mt8183-flakes.txt  |   14 +
>  .../gpu/drm/ci/xfails/meson-g12b-fails.txt    |   12 +
>  .../gpu/drm/ci/xfails/meson-g12b-flakes.txt   |    4 +
>  .../gpu/drm/ci/xfails/msm-apq8016-fails.txt   |   15 +
>  .../gpu/drm/ci/xfails/msm-apq8016-flakes.txt  |    4 +
>  .../gpu/drm/ci/xfails/msm-apq8096-fails.txt   |    2 +
>  .../gpu/drm/ci/xfails/msm-apq8096-flakes.txt  |    4 +
>  .../gpu/drm/ci/xfails/msm-apq8096-skips.txt   |    2 +
>  .../gpu/drm/ci/xfails/msm-sc7180-fails.txt    |   25 +
>  .../gpu/drm/ci/xfails/msm-sc7180-flakes.txt   |    7 +
>  .../gpu/drm/ci/xfails/msm-sc7180-skips.txt    |   23 +
>  .../gpu/drm/ci/xfails/msm-sdm845-fails.txt    |   68 +
>  .../gpu/drm/ci/xfails/msm-sdm845-flakes.txt   |   11 +
>  .../gpu/drm/ci/xfails/msm-sdm845-skips.txt    |    2 +
>  .../drm/ci/xfails/rockchip-rk3288-fails.txt   |   48 +
>  .../drm/ci/xfails/rockchip-rk3288-flakes.txt  |    9 +
>  .../drm/ci/xfails/rockchip-rk3288-skips.txt   |   52 +
>  .../drm/ci/xfails/rockchip-rk3399-fails.txt   |   37 +
>  .../drm/ci/xfails/rockchip-rk3399-flakes.txt  |   23 +
>  .../drm/ci/xfails/rockchip-rk3399-skips.txt   |    5 +
>  .../drm/ci/xfails/virtio_gpu-none-fails.txt   |   38 +
>  .../drm/ci/xfails/virtio_gpu-none-flakes.txt  |    0
>  .../drm/ci/xfails/virtio_gpu-none-skips.txt   |    6 +
>  test                                          |    0
>  69 files changed, 5508 insertions(+)
>  create mode 100644 Documentation/gpu/automated_testing.rst
>  create mode 100644 drivers/gpu/drm/ci/arm.config
>  create mode 100644 drivers/gpu/drm/ci/arm64.config
>  create mode 100644 drivers/gpu/drm/ci/build-igt.sh
>  create mode 100644 drivers/gpu/drm/ci/build.sh
>  create mode 100644 drivers/gpu/drm/ci/build.yml
>  create mode 100755 drivers/gpu/drm/ci/check-patch.py
>  create mode 100644 drivers/gpu/drm/ci/container.yml
>  create mode 100644 drivers/gpu/drm/ci/gitlab-ci.yml
>  create mode 100755 drivers/gpu/drm/ci/igt_runner.sh
>  create mode 100644 drivers/gpu/drm/ci/image-tags.yml
>  create mode 100755 drivers/gpu/drm/ci/lava-submit.sh
>  create mode 100644 drivers/gpu/drm/ci/static-checks.yml
>  create mode 100644 drivers/gpu/drm/ci/test.yml
>  create mode 100644 drivers/gpu/drm/ci/testlist.txt
>  create mode 100644 drivers/gpu/drm/ci/x86_64.config
>  create mode 100644 drivers/gpu/drm/ci/xfails/amdgpu-stoney-fails.txt
>  create mode 100644 drivers/gpu/drm/ci/xfails/amdgpu-stoney-flakes.txt
>  create mode 100644 drivers/gpu/drm/ci/xfails/amdgpu-stoney-skips.txt
>  create mode 100644 drivers/gpu/drm/ci/xfails/i915-amly-fails.txt
>  create mode 100644 drivers/gpu/drm/ci/xfails/i915-amly-flakes.txt
>  create mode 100644 drivers/gpu/drm/ci/xfails/i915-amly-skips.txt
>  create mode 100644 drivers/gpu/drm/ci/xfails/i915-apl-fails.txt
>  create mode 100644 drivers/gpu/drm/ci/xfails/i915-apl-flakes.txt
>  create mode 100644 drivers/gpu/drm/ci/xfails/i915-apl-skips.txt
>  create mode 100644 drivers/gpu/drm/ci/xfails/i915-cml-fails.txt
>  create mode 100644 drivers/gpu/drm/ci/xfails/i915-cml-flakes.txt
>  create mode 100644 drivers/gpu/drm/ci/xfails/i915-cml-skips.txt
>  create mode 100644 drivers/gpu/drm/ci/xfails/i915-glk-fails.txt
>  create mode 100644 drivers/gpu/drm/ci/xfails/i915-glk-flakes.txt
>  create mode 100644 drivers/gpu/drm/ci/xfails/i915-glk-skips.txt
>  create mode 100644 drivers/gpu/drm/ci/xfails/i915-kbl-fails.txt
>  create mode 100644 drivers/gpu/drm/ci/xfails/i915-kbl-flakes.txt
>  create mode 100644 drivers/gpu/drm/ci/xfails/i915-kbl-skips.txt
>  create mode 100644 drivers/gpu/drm/ci/xfails/i915-tgl-fails.txt
>  create mode 100644 drivers/gpu/drm/ci/xfails/i915-tgl-flakes.txt
>  create mode 100644 drivers/gpu/drm/ci/xfails/i915-tgl-skips.txt
>  create mode 100644 drivers/gpu/drm/ci/xfails/i915-whl-fails.txt
>  create mode 100644 drivers/gpu/drm/ci/xfails/i915-whl-flakes.txt
>  create mode 100644 drivers/gpu/drm/ci/xfails/i915-whl-skips.txt
>  create mode 100644 drivers/gpu/drm/ci/xfails/mediatek-mt8173-fails.txt
>  create mode 100644 drivers/gpu/drm/ci/xfails/mediatek-mt8173-flakes.txt
>  create mode 100644 drivers/gpu/drm/ci/xfails/mediatek-mt8183-fails.txt
>  create mode 100644 drivers/gpu/drm/ci/xfails/mediatek-mt8183-flakes.txt
>  create mode 100644 drivers/gpu/drm/ci/xfails/meson-g12b-fails.txt
>  create mode 100644 drivers/gpu/drm/ci/xfails/meson-g12b-flakes.txt
>  create mode 100644 drivers/gpu/drm/ci/xfails/msm-apq8016-fails.txt
>  create mode 100644 drivers/gpu/drm/ci/xfails/msm-apq8016-flakes.txt
>  create mode 100644 drivers/gpu/drm/ci/xfails/msm-apq8096-fails.txt
>  create mode 100644 drivers/gpu/drm/ci/xfails/msm-apq8096-flakes.txt
>  create mode 100644 drivers/gpu/drm/ci/xfails/msm-apq8096-skips.txt
>  create mode 100644 drivers/gpu/drm/ci/xfails/msm-sc7180-fails.txt
>  create mode 100644 drivers/gpu/drm/ci/xfails/msm-sc7180-flakes.txt
>  create mode 100644 drivers/gpu/drm/ci/xfails/msm-sc7180-skips.txt
>  create mode 100644 drivers/gpu/drm/ci/xfails/msm-sdm845-fails.txt
>  create mode 100644 drivers/gpu/drm/ci/xfails/msm-sdm845-flakes.txt
>  create mode 100644 drivers/gpu/drm/ci/xfails/msm-sdm845-skips.txt
>  create mode 100644 drivers/gpu/drm/ci/xfails/rockchip-rk3288-fails.txt
>  create mode 100644 drivers/gpu/drm/ci/xfails/rockchip-rk3288-flakes.txt
>  create mode 100644 drivers/gpu/drm/ci/xfails/rockchip-rk3288-skips.txt
>  create mode 100644 drivers/gpu/drm/ci/xfails/rockchip-rk3399-fails.txt
>  create mode 100644 drivers/gpu/drm/ci/xfails/rockchip-rk3399-flakes.txt
>  create mode 100644 drivers/gpu/drm/ci/xfails/rockchip-rk3399-skips.txt
>  create mode 100644 drivers/gpu/drm/ci/xfails/virtio_gpu-none-fails.txt
>  create mode 100644 drivers/gpu/drm/ci/xfails/virtio_gpu-none-flakes.txt
>  create mode 100644 drivers/gpu/drm/ci/xfails/virtio_gpu-none-skips.txt
>  create mode 100644 test
> 
> diff --git a/Documentation/gpu/automated_testing.rst b/Documentation/gpu/automated_testing.rst
> new file mode 100644
> index 000000000000..1b87b802ac7f
> --- /dev/null
> +++ b/Documentation/gpu/automated_testing.rst
> @@ -0,0 +1,144 @@
> +.. SPDX-License-Identifier: GPL-2.0+
> +
> +=========================================
> +Automated testing of the DRM subsystem
> +=========================================
> +
> +Introduction
> +============
> +
> +Making sure that changes to the core or drivers don't introduce regressions can
> +be very time-consuming when lots of different hardware configurations need to
> +be tested. Moreover, it isn't practical for each person interested in this
> +testing to have to acquire and maintain what can be a considerable amount of
> +hardware.
> +
> +Also, it is desirable for developers to check for regressions in their code by
> +themselves, instead of relying on the maintainers to find them and then
> +reporting back.
> +
> +There are facilities in gitlab.freedesktop.org to automatically test Mesa that
> +can be used as well for testing the DRM subsystem. This document explains how
> +people interested in testing it can use this shared infrastructure to save
> +quite some time and effort.
> +
> +
> +Relevant files
> +==============
> +
> +drivers/gpu/drm/ci/gitlab-ci.yml
> +--------------------------------
> +
> +This is the root configuration file for GitLab CI. Among other less interesting
> +bits, it specifies the specific version of the scripts to be used. There are
> +some variables that can be modified to change the behavior of the pipeline:
> +
> +DRM_CI_PROJECT_PATH
> +    Repository that contains the Mesa software infrastructure for CI
> +
> +DRM_CI_COMMIT_SHA
> +    A particular revision to use from that repository
> +
> +UPSTREAM_REPO
> +    URL to git repository containing the target branch
> +
> +TARGET_BRANCH
> +    Branch to which this branch is to be merged into
> +
> +IGT_VERSION
> +    Revision of igt-gpu-tools being used, from
> +    https://gitlab.freedesktop.org/drm/igt-gpu-tools
> +
> +drivers/gpu/drm/ci/testlist.txt
> +-------------------------------
> +
> +IGT tests to be run on all drivers (unless mentioned in a driver's *-skips.txt
> +file, see below).
> +
> +drivers/gpu/drm/ci/${DRIVER_NAME}-${HW_REVISION}-fails.txt
> +----------------------------------------------------------
> +
> +Lists the known failures for a given driver on a specific hardware revision.
> +
> +drivers/gpu/drm/ci/${DRIVER_NAME}-${HW_REVISION}-flakes.txt
> +-----------------------------------------------------------
> +
> +Lists the tests that for a given driver on a specific hardware revision are
> +known to behave unreliably. These tests won't cause a job to fail regardless of
> +the result. They will still be run.
> +
> +drivers/gpu/drm/ci/${DRIVER_NAME}-${HW_REVISION}-skips.txt
> +-----------------------------------------------------------
> +
> +Lists the tests that won't be run for a given driver on a specific hardware
> +revision. These are usually tests that interfere with the running of the test
> +list due to hanging the machine, causing OOM, taking too long, etc.
> +
> +
> +How to enable automated testing on your tree
> +============================================
> +
> +1. Create a Linux tree in https://gitlab.freedesktop.org/ if you don't have one
> +yet
> +
> +2. In your kernel repo's configuration (eg.
> +https://gitlab.freedesktop.org/janedoe/linux/-/settings/ci_cd), change the
> +CI/CD configuration file from .gitlab-ci.yml to
> +drivers/gpu/drm/ci/gitlab-ci.yml.
> +
> +3. Next time you push to this repository, you will see a CI pipeline being
> +created (eg. https://gitlab.freedesktop.org/janedoe/linux/-/pipelines)
> +
> +4. The various jobs will be run and when the pipeline is finished, all jobs
> +should be green unless a regression has been found.
> +
> +
> +How to update test expectations
> +===============================
> +
> +If your changes to the code fix any tests, you will have to remove one or more
> +lines from one or more of the files in
> +drivers/gpu/drm/ci/${DRIVER_NAME}_*_fails.txt, for each of the test platforms
> +affected by the change.
> +
> +
> +How to expand coverage
> +======================
> +
> +If your code changes make it possible to run more tests (by solving reliability
> +issues, for example), you can remove tests from the flakes and/or skips lists,
> +and then the expected results if there are any known failures.
> +
> +If there is a need for updating the version of IGT being used (maybe you have
> +added more tests to it), update the IGT_VERSION variable at the top of the
> +gitlab-ci.yml file.
> +
> +
> +How to test your changes to the scripts
> +=======================================
> +
> +For testing changes to the scripts in the drm-ci repo, change the
> +DRM_CI_PROJECT_PATH and DRM_CI_COMMIT_SHA variables in
> +drivers/gpu/drm/ci/gitlab-ci.yml to match your fork of the project (eg.
> +janedoe/drm-ci). This fork needs to be in https://gitlab.freedesktop.org/.
> +
> +
> +How to incorporate external fixes in your testing
> +=================================================
> +
> +Often, regressions in other trees will prevent testing changes local to the
> +tree under test. These fixes will be automatically merged in during the build
> +jobs from a branch in the target tree that is named as
> +${TARGET_BRANCH}-external-fixes.
> +
> +If the pipeline is not in a merge request and a branch with the same name
> +exists in the local tree, commits from that branch will be merged in as well.
> +
> +
> +How to deal with automated testing labs that may be down
> +========================================================
> +
> +If a hardware farm is down and thus causing pipelines to fail that would
> +otherwise pass, one can disable all jobs that would be submitted to that farm
> +by editing the file at
> +https://gitlab.freedesktop.org/gfx-ci/lab-status/-/blob/main/lab-status.yml.
> diff --git a/Documentation/gpu/index.rst b/Documentation/gpu/index.rst
> index eee5996acf2c..e45ff0915246 100644
> --- a/Documentation/gpu/index.rst
> +++ b/Documentation/gpu/index.rst
> @@ -17,6 +17,7 @@ GPU Driver Developer's Guide
>     backlight
>     vga-switcheroo
>     vgaarbiter
> +   automated_testing
>     todo
>     rfc/index
>  
> diff --git a/MAINTAINERS b/MAINTAINERS
> index cd882b87a3c6..d6f163ec7b31 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -7131,6 +7131,14 @@ T:	git git://anongit.freedesktop.org/drm/drm-misc
>  F:	drivers/gpu/drm/ttm/
>  F:	include/drm/ttm/
>  
> +DRM AUTOMATED TESTING
> +M:	Helen Koike <helen.koike@collabora.com>
> +L:	dri-devel@lists.freedesktop.org
> +S:	Maintained
> +T:	git git://anongit.freedesktop.org/drm/drm-misc
> +F:	Documentation/gpu/automated_testing.rst
> +F:	drivers/gpu/drm/ci/
> +
>  DSBR100 USB FM RADIO DRIVER
>  M:	Alexey Klimov <klimov.linux@gmail.com>
>  L:	linux-media@vger.kernel.org
> diff --git a/drivers/gpu/drm/ci/arm.config b/drivers/gpu/drm/ci/arm.config
> new file mode 100644
> index 000000000000..871f4de063ad
> --- /dev/null
> +++ b/drivers/gpu/drm/ci/arm.config
> @@ -0,0 +1,69 @@
> +CONFIG_LOCALVERSION_AUTO=y
> +CONFIG_DEBUG_KERNEL=y
> +
> +CONFIG_CRYPTO_ZSTD=y
> +CONFIG_ZRAM_MEMORY_TRACKING=y
> +CONFIG_ZRAM_WRITEBACK=y
> +CONFIG_ZRAM=y
> +CONFIG_ZSMALLOC_STAT=y
> +
> +# abootimg with a 'dummy' rootfs fails with root=/dev/nfs
> +CONFIG_BLK_DEV_INITRD=n
> +
> +CONFIG_DEVFREQ_GOV_PERFORMANCE=y
> +CONFIG_DEVFREQ_GOV_POWERSAVE=y
> +CONFIG_DEVFREQ_GOV_USERSPACE=y
> +CONFIG_DEVFREQ_GOV_PASSIVE=y
> +CONFIG_DEVFREQ_GOV_SIMPLE_ONDEMAND=y
> +
> +CONFIG_DRM=y
> +CONFIG_DRM_ETNAVIV=y
> +CONFIG_DRM_ROCKCHIP=y
> +CONFIG_DRM_PANFROST=y
> +CONFIG_DRM_LIMA=y
> +CONFIG_DRM_PANEL_SIMPLE=y
> +CONFIG_PWM_CROS_EC=y
> +CONFIG_BACKLIGHT_PWM=y
> +
> +CONFIG_ROCKCHIP_CDN_DP=n
> +
> +CONFIG_SPI_ROCKCHIP=y
> +CONFIG_PWM_ROCKCHIP=y
> +CONFIG_PHY_ROCKCHIP_DP=y
> +CONFIG_DWMAC_ROCKCHIP=y
> +
> +CONFIG_MFD_RK808=y
> +CONFIG_REGULATOR_RK808=y
> +CONFIG_RTC_DRV_RK808=y
> +CONFIG_COMMON_CLK_RK808=y
> +
> +CONFIG_REGULATOR_FAN53555=y
> +CONFIG_REGULATOR=y
> +
> +CONFIG_REGULATOR_VCTRL=y
> +
> +CONFIG_KASAN=n
> +CONFIG_KASAN_INLINE=n
> +CONFIG_STACKTRACE=n
> +
> +CONFIG_TMPFS=y
> +
> +CONFIG_PROVE_LOCKING=n
> +CONFIG_DEBUG_LOCKDEP=n
> +CONFIG_SOFTLOCKUP_DETECTOR=n
> +CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC=n
> +
> +CONFIG_FW_LOADER_COMPRESS=y
> +
> +CONFIG_USB_USBNET=y
> +CONFIG_NETDEVICES=y
> +CONFIG_USB_NET_DRIVERS=y
> +CONFIG_USB_RTL8152=y
> +CONFIG_USB_NET_AX8817X=y
> +CONFIG_USB_NET_SMSC95XX=y
> +
> +# TK1
> +CONFIG_ARM_TEGRA_DEVFREQ=y
> +
> +# 32-bit build failure
> +CONFIG_DRM_MSM=n
> diff --git a/drivers/gpu/drm/ci/arm64.config b/drivers/gpu/drm/ci/arm64.config
> new file mode 100644
> index 000000000000..817e18ddfd4f
> --- /dev/null
> +++ b/drivers/gpu/drm/ci/arm64.config
> @@ -0,0 +1,199 @@
> +CONFIG_LOCALVERSION_AUTO=y
> +CONFIG_DEBUG_KERNEL=y
> +
> +CONFIG_CRYPTO_ZSTD=y
> +CONFIG_ZRAM_MEMORY_TRACKING=y
> +CONFIG_ZRAM_WRITEBACK=y
> +CONFIG_ZRAM=y
> +CONFIG_ZSMALLOC_STAT=y
> +
> +# abootimg with a 'dummy' rootfs fails with root=/dev/nfs
> +CONFIG_BLK_DEV_INITRD=n
> +
> +CONFIG_DEVFREQ_GOV_PERFORMANCE=y
> +CONFIG_DEVFREQ_GOV_POWERSAVE=y
> +CONFIG_DEVFREQ_GOV_USERSPACE=y
> +CONFIG_DEVFREQ_GOV_PASSIVE=y
> +
> +CONFIG_DRM=y
> +CONFIG_DRM_ROCKCHIP=y
> +CONFIG_DRM_PANFROST=y
> +CONFIG_DRM_LIMA=y
> +CONFIG_DRM_PANEL_SIMPLE=y
> +CONFIG_DRM_PANEL_EDP=y
> +CONFIG_DRM_MSM=y
> +CONFIG_DRM_ETNAVIV=y
> +CONFIG_DRM_I2C_ADV7511=y
> +CONFIG_PWM_CROS_EC=y
> +CONFIG_BACKLIGHT_PWM=y
> +
> +CONFIG_ROCKCHIP_CDN_DP=n
> +
> +CONFIG_SPI_ROCKCHIP=y
> +CONFIG_PWM_ROCKCHIP=y
> +CONFIG_PHY_ROCKCHIP_DP=y
> +CONFIG_DWMAC_ROCKCHIP=y
> +CONFIG_STMMAC_ETH=y
> +CONFIG_TYPEC_FUSB302=y
> +CONFIG_TYPEC=y
> +CONFIG_TYPEC_TCPM=y
> +
> +# MSM platform bits
> +
> +# For CONFIG_QCOM_LMH
> +CONFIG_OF=y
> +
> +CONFIG_ARM_SMMU_QCOM=y
> +CONFIG_QCOM_COMMAND_DB=y
> +CONFIG_QCOM_RPMHPD=y
> +CONFIG_QCOM_RPMPD=y
> +CONFIG_QCOM_OCMEM=y
> +CONFIG_SDM_GPUCC_845=y
> +CONFIG_SDM_VIDEOCC_845=y
> +CONFIG_SDM_DISPCC_845=y
> +CONFIG_SDM_LPASSCC_845=y
> +CONFIG_SDM_CAMCC_845=y
> +CONFIG_RESET_QCOM_PDC=y
> +CONFIG_DRM_TI_SN65DSI86=y
> +CONFIG_I2C_QCOM_GENI=y
> +CONFIG_SPI_QCOM_GENI=y
> +CONFIG_PHY_QCOM_QUSB2=y
> +CONFIG_PHY_QCOM_QMP=y
> +CONFIG_MSM_GCC_8996=y
> +CONFIG_QCOM_CLK_APCC_MSM8996=y
> +CONFIG_QCOM_LLCC=y
> +CONFIG_QCOM_LMH=y
> +CONFIG_QCOM_SPMI_TEMP_ALARM=y
> +CONFIG_QCOM_WDT=y
> +CONFIG_POWER_RESET_QCOM_PON=y
> +CONFIG_RTC_DRV_PM8XXX=y
> +CONFIG_INTERCONNECT=y
> +CONFIG_INTERCONNECT_QCOM=y
> +CONFIG_INTERCONNECT_QCOM_MSM8996=y
> +CONFIG_INTERCONNECT_QCOM_SDM845=y
> +CONFIG_INTERCONNECT_QCOM_MSM8916=y
> +CONFIG_INTERCONNECT_QCOM_MSM8996=y
> +CONFIG_INTERCONNECT_QCOM_OSM_L3=y
> +CONFIG_INTERCONNECT_QCOM_SC7180=y
> +CONFIG_INTERCONNECT_QCOM_SM8350=y
> +CONFIG_CRYPTO_DEV_QCOM_RNG=y
> +CONFIG_SC_DISPCC_7180=y
> +CONFIG_SC_GPUCC_7180=y
> +CONFIG_SM_GPUCC_8350=y
> +CONFIG_QCOM_SPMI_ADC5=y
> +CONFIG_DRM_PARADE_PS8640=y
> +CONFIG_DRM_LONTIUM_LT9611UXC=y
> +CONFIG_PHY_QCOM_USB_HS=y
> +CONFIG_QCOM_GPI_DMA=y
> +CONFIG_USB_ONBOARD_HUB=y
> +CONFIG_NVMEM_QCOM_QFPROM=y
> +CONFIG_PHY_QCOM_USB_SNPS_FEMTO_V2=y
> +
> +
> +# db410c ethernet
> +CONFIG_USB_RTL8152=y
> +# db820c ethernet
> +CONFIG_ATL1C=y
> +# Chromebooks ethernet
> +CONFIG_USB_ONBOARD_HUB=y
> +# 888 HDK ethernet
> +CONFIG_USB_LAN78XX=y
> +
> +CONFIG_ARCH_ALPINE=n
> +CONFIG_ARCH_BCM2835=y
> +CONFIG_ARCH_BCM_IPROC=n
> +CONFIG_ARCH_BERLIN=n
> +CONFIG_ARCH_BRCMSTB=n
> +CONFIG_ARCH_EXYNOS=n
> +CONFIG_ARCH_K3=n
> +CONFIG_ARCH_LAYERSCAPE=n
> +CONFIG_ARCH_LG1K=n
> +CONFIG_ARCH_HISI=n
> +CONFIG_ARCH_MVEBU=n
> +CONFIG_ARCH_SEATTLE=n
> +CONFIG_ARCH_SYNQUACER=n
> +CONFIG_ARCH_RENESAS=n
> +CONFIG_ARCH_R8A774A1=n
> +CONFIG_ARCH_R8A774C0=n
> +CONFIG_ARCH_R8A7795=n
> +CONFIG_ARCH_R8A7796=n
> +CONFIG_ARCH_R8A77965=n
> +CONFIG_ARCH_R8A77970=n
> +CONFIG_ARCH_R8A77980=n
> +CONFIG_ARCH_R8A77990=n
> +CONFIG_ARCH_R8A77995=n
> +CONFIG_ARCH_STRATIX10=n
> +CONFIG_ARCH_TEGRA=n
> +CONFIG_ARCH_SPRD=n
> +CONFIG_ARCH_THUNDER=n
> +CONFIG_ARCH_THUNDER2=n
> +CONFIG_ARCH_UNIPHIER=n
> +CONFIG_ARCH_VEXPRESS=n
> +CONFIG_ARCH_XGENE=n
> +CONFIG_ARCH_ZX=n
> +CONFIG_ARCH_ZYNQMP=n
> +
> +# Strip out some stuff we don't need for graphics testing, to reduce
> +# the build.
> +CONFIG_CAN=n
> +CONFIG_WIRELESS=n
> +CONFIG_RFKILL=n
> +CONFIG_WLAN=n
> +
> +CONFIG_REGULATOR_FAN53555=y
> +CONFIG_REGULATOR=y
> +
> +CONFIG_REGULATOR_VCTRL=y
> +
> +CONFIG_KASAN=n
> +CONFIG_KASAN_INLINE=n
> +CONFIG_STACKTRACE=n
> +
> +CONFIG_TMPFS=y
> +
> +CONFIG_PROVE_LOCKING=n
> +CONFIG_DEBUG_LOCKDEP=n
> +CONFIG_SOFTLOCKUP_DETECTOR=y
> +CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC=y
> +
> +CONFIG_DETECT_HUNG_TASK=y
> +
> +CONFIG_FW_LOADER_COMPRESS=y
> +CONFIG_FW_LOADER_USER_HELPER=n
> +
> +CONFIG_USB_USBNET=y
> +CONFIG_NETDEVICES=y
> +CONFIG_USB_NET_DRIVERS=y
> +CONFIG_USB_RTL8152=y
> +CONFIG_USB_NET_AX8817X=y
> +CONFIG_USB_NET_SMSC95XX=y
> +
> +# For amlogic
> +CONFIG_MESON_GXL_PHY=y
> +CONFIG_MDIO_BUS_MUX_MESON_G12A=y
> +CONFIG_DRM_MESON=y
> +
> +# For Mediatek
> +CONFIG_DRM_MEDIATEK=y
> +CONFIG_PWM_MEDIATEK=y
> +CONFIG_DRM_MEDIATEK_HDMI=y
> +CONFIG_GNSS=y
> +CONFIG_GNSS_MTK_SERIAL=y
> +CONFIG_HW_RANDOM=y
> +CONFIG_HW_RANDOM_MTK=y
> +CONFIG_MTK_DEVAPC=y
> +CONFIG_PWM_MTK_DISP=y
> +CONFIG_MTK_CMDQ=y
> +
> +# For nouveau.  Note that DRM must be a module so that it's loaded after NFS is up to provide the firmware.
> +CONFIG_ARCH_TEGRA=y
> +CONFIG_DRM_NOUVEAU=m
> +CONFIG_DRM_TEGRA=m
> +CONFIG_R8169=y
> +CONFIG_STAGING=y
> +CONFIG_DRM_TEGRA_STAGING=y
> +CONFIG_TEGRA_HOST1X=y
> +CONFIG_ARM_TEGRA_DEVFREQ=y
> +CONFIG_TEGRA_SOCTHERM=y
> +CONFIG_DRM_TEGRA_DEBUG=y
> +CONFIG_PWM_TEGRA=y
> diff --git a/drivers/gpu/drm/ci/build-igt.sh b/drivers/gpu/drm/ci/build-igt.sh
> new file mode 100644
> index 000000000000..500fa4f5c30a
> --- /dev/null
> +++ b/drivers/gpu/drm/ci/build-igt.sh
> @@ -0,0 +1,35 @@
> +#!/bin/bash
> +# SPDX-License-Identifier: MIT
> +
> +set -ex
> +
> +git clone https://gitlab.freedesktop.org/drm/igt-gpu-tools.git --single-branch --no-checkout
> +cd igt-gpu-tools
> +git checkout $IGT_VERSION
> +
> +if [[ "$KERNEL_ARCH" = "arm" ]]; then
> +    . ../.gitlab-ci/container/create-cross-file.sh armhf
> +    EXTRA_MESON_ARGS="--cross-file /cross_file-armhf.txt"
> +fi
> +
> +MESON_OPTIONS="-Doverlay=disabled                    \
> +               -Dchamelium=disabled                  \
> +               -Dvalgrind=disabled                   \
> +               -Dman=enabled                         \
> +               -Dtests=enabled                       \
> +               -Drunner=enabled                      \
> +               -Dlibunwind=enabled                   \
> +               -Dprefix=/igt"
> +
> +mkdir -p /igt
> +meson build $MESON_OPTIONS $EXTRA_MESON_ARGS
> +ninja -C build -j${FDO_CI_CONCURRENT:-4} || ninja -C build -j 1
> +ninja -C build install
> +
> +mkdir -p artifacts/
> +tar -cf artifacts/igt.tar /igt
> +
> +# Pass needed files to the test stage
> +S3_ARTIFACT_NAME="igt.tar.gz"
> +gzip -c artifacts/igt.tar > ${S3_ARTIFACT_NAME}
> +ci-fairy s3cp --token-file "${CI_JOB_JWT_FILE}" ${S3_ARTIFACT_NAME} https://${PIPELINE_ARTIFACTS_BASE}/${KERNEL_ARCH}/${S3_ARTIFACT_NAME}
> diff --git a/drivers/gpu/drm/ci/build.sh b/drivers/gpu/drm/ci/build.sh
> new file mode 100644
> index 000000000000..7b014287a041
> --- /dev/null
> +++ b/drivers/gpu/drm/ci/build.sh
> @@ -0,0 +1,157 @@
> +#!/bin/bash
> +# SPDX-License-Identifier: MIT
> +
> +set -ex
> +
> +# Clean up stale rebases that GitLab might not have removed when reusing a checkout dir
> +rm -rf .git/rebase-apply
> +
> +. .gitlab-ci/container/container_pre_build.sh
> +
> +# libssl-dev was uninstalled because it was considered an ephemeral package
> +apt-get update
> +apt-get install -y libssl-dev
> +
> +if [[ "$KERNEL_ARCH" = "arm64" ]]; then
> +    GCC_ARCH="aarch64-linux-gnu"
> +    DEBIAN_ARCH="arm64"
> +    DEVICE_TREES="arch/arm64/boot/dts/rockchip/rk3399-gru-kevin.dtb"
> +    DEVICE_TREES+=" arch/arm64/boot/dts/amlogic/meson-gxl-s805x-libretech-ac.dtb"
> +    DEVICE_TREES+=" arch/arm64/boot/dts/allwinner/sun50i-h6-pine-h64.dtb"
> +    DEVICE_TREES+=" arch/arm64/boot/dts/amlogic/meson-gxm-khadas-vim2.dtb"
> +    DEVICE_TREES+=" arch/arm64/boot/dts/qcom/apq8016-sbc.dtb"
> +    DEVICE_TREES+=" arch/arm64/boot/dts/qcom/apq8096-db820c.dtb"
> +    DEVICE_TREES+=" arch/arm64/boot/dts/amlogic/meson-g12b-a311d-khadas-vim3.dtb"
> +    DEVICE_TREES+=" arch/arm64/boot/dts/mediatek/mt8173-elm-hana.dtb"
> +    DEVICE_TREES+=" arch/arm64/boot/dts/mediatek/mt8183-kukui-jacuzzi-juniper-sku16.dtb"
> +    DEVICE_TREES+=" arch/arm64/boot/dts/mediatek/mt8192-asurada-spherion-r0.dtb"
> +    DEVICE_TREES+=" arch/arm64/boot/dts/qcom/sc7180-trogdor-lazor-limozeen-nots-r5.dtb"
> +elif [[ "$KERNEL_ARCH" = "arm" ]]; then
> +    GCC_ARCH="arm-linux-gnueabihf"
> +    DEBIAN_ARCH="armhf"
> +    DEVICE_TREES="arch/arm/boot/dts/rockchip/rk3288-veyron-jaq.dtb"
> +    DEVICE_TREES+=" arch/arm/boot/dts/allwinner/sun8i-h3-libretech-all-h3-cc.dtb"
> +    DEVICE_TREES+=" arch/arm/boot/dts/nxp/imx/imx6q-cubox-i.dtb"
> +    apt-get install -y libssl-dev:armhf
> +else
> +    GCC_ARCH="x86_64-linux-gnu"
> +    DEBIAN_ARCH="x86_64"
> +    DEVICE_TREES=""
> +fi
> +
> +export ARCH=${KERNEL_ARCH}
> +export CROSS_COMPILE="${GCC_ARCH}-"
> +
> +# The kernel doesn't like the gold linker (or the old lld in our debians).
> +# Sneak in some override symlinks during kernel build until we can update
> +# debian.
> +mkdir -p ld-links
> +for i in /usr/bin/*-ld /usr/bin/ld; do
> +    i=$(basename $i)
> +    ln -sf /usr/bin/$i.bfd ld-links/$i
> +done
> +
> +NEWPATH=$(pwd)/ld-links
> +export PATH=$NEWPATH:$PATH
> +
> +git config --global user.email "fdo@example.com"
> +git config --global user.name "freedesktop.org CI"
> +git config --global pull.rebase true
> +
> +# Try to merge fixes from target repo
> +if [ "$(git ls-remote --exit-code --heads ${UPSTREAM_REPO} ${TARGET_BRANCH}-external-fixes)" ]; then
> +    git pull ${UPSTREAM_REPO} ${TARGET_BRANCH}-external-fixes
> +fi
> +
> +# Try to merge fixes from local repo if this isn't a merge request
> +if [ -z "$CI_MERGE_REQUEST_PROJECT_PATH" ]; then
> +    if [ "$(git ls-remote --exit-code --heads origin ${TARGET_BRANCH}-external-fixes)" ]; then
> +        git pull origin ${TARGET_BRANCH}-external-fixes
> +    fi
> +fi
> +
> +for opt in $ENABLE_KCONFIGS; do
> +  echo CONFIG_$opt=y >> drivers/gpu/drm/ci/${KERNEL_ARCH}.config
> +done
> +for opt in $DISABLE_KCONFIGS; do
> +  echo CONFIG_$opt=n >> drivers/gpu/drm/ci/${KERNEL_ARCH}.config
> +done
> +
> +if [[ -n "${MERGE_FRAGMENT}" ]]; then
> +    ./scripts/kconfig/merge_config.sh ${DEFCONFIG} drivers/gpu/drm/ci/${MERGE_FRAGMENT}
> +else
> +    make `basename ${DEFCONFIG}`
> +fi
> +
> +make ${KERNEL_IMAGE_NAME}
> +
> +mkdir -p /lava-files/
> +for image in ${KERNEL_IMAGE_NAME}; do
> +    cp arch/${KERNEL_ARCH}/boot/${image} /lava-files/.
> +done
> +
> +if [[ -n ${DEVICE_TREES} ]]; then
> +    make dtbs
> +    cp ${DEVICE_TREES} /lava-files/.
> +fi
> +
> +make modules
> +mkdir -p install/modules/
> +INSTALL_MOD_PATH=install/modules/ make modules_install
> +
> +if [[ ${DEBIAN_ARCH} = "arm64" ]]; then
> +    make Image.lzma
> +    mkimage \
> +        -f auto \
> +        -A arm \
> +        -O linux \
> +        -d arch/arm64/boot/Image.lzma \
> +        -C lzma\
> +        -b arch/arm64/boot/dts/qcom/sdm845-cheza-r3.dtb \
> +        /lava-files/cheza-kernel
> +    KERNEL_IMAGE_NAME+=" cheza-kernel"
> +
> +    # Make a gzipped copy of the Image for db410c.
> +    gzip -k /lava-files/Image
> +    KERNEL_IMAGE_NAME+=" Image.gz"
> +fi
> +
> +# Pass needed files to the test stage
> +mkdir -p install
> +cp -rfv .gitlab-ci/* install/.
> +cp -rfv install/common install/ci-common
> +cp -rfv drivers/gpu/drm/ci/* install/.
> +
> +. .gitlab-ci/container/container_post_build.sh
> +
> +if [[ "$UPLOAD_TO_MINIO" = "1" ]]; then
> +    xz -7 -c -T${FDO_CI_CONCURRENT:-4} vmlinux > /lava-files/vmlinux.xz
> +    FILES_TO_UPLOAD="$KERNEL_IMAGE_NAME vmlinux.xz"
> +
> +    if [[ -n $DEVICE_TREES ]]; then
> +        FILES_TO_UPLOAD="$FILES_TO_UPLOAD $(basename -a $DEVICE_TREES)"
> +    fi
> +
> +    for f in $FILES_TO_UPLOAD; do
> +        ci-fairy s3cp --token-file "${CI_JOB_JWT_FILE}" /lava-files/$f \
> +                https://${PIPELINE_ARTIFACTS_BASE}/${DEBIAN_ARCH}/$f
> +    done
> +
> +    S3_ARTIFACT_NAME="kernel-files.tar.zst"
> +    tar --zstd -cf $S3_ARTIFACT_NAME install
> +    ci-fairy s3cp --token-file "${CI_JOB_JWT_FILE}" ${S3_ARTIFACT_NAME} https://${PIPELINE_ARTIFACTS_BASE}/${DEBIAN_ARCH}/${S3_ARTIFACT_NAME}
> +
> +    echo "Download vmlinux.xz from https://${PIPELINE_ARTIFACTS_BASE}/${DEBIAN_ARCH}/vmlinux.xz"
> +fi
> +
> +mkdir -p artifacts/install/lib
> +mv install/* artifacts/install/.
> +rm -rf artifacts/install/modules
> +ln -s common artifacts/install/ci-common
> +
> +for image in ${KERNEL_IMAGE_NAME}; do
> +    cp /lava-files/$image artifacts/install/.
> +done
> +
> +tar -C artifacts -cf artifacts/install.tar install
> +rm -rf artifacts/install
> diff --git a/drivers/gpu/drm/ci/build.yml b/drivers/gpu/drm/ci/build.yml
> new file mode 100644
> index 000000000000..e6503f1c5927
> --- /dev/null
> +++ b/drivers/gpu/drm/ci/build.yml
> @@ -0,0 +1,110 @@
> +.build:
> +  extends:
> +    - .build-rules
> +  stage: build
> +  artifacts:
> +    paths:
> +      - artifacts
> +  script:
> +    - FDO_CI_CONCURRENT=${FDO_CI_CONCURRENT} bash drivers/gpu/drm/ci/build.sh
> +
> +.build:arm32:
> +  extends:
> +    - .build
> +    - .use-debian/arm64_build
> +  tags:
> +    - aarch64
> +  variables:
> +    DEFCONFIG: "arch/arm/configs/multi_v7_defconfig"
> +    KERNEL_IMAGE_NAME: "zImage"
> +    KERNEL_ARCH: "arm"
> +
> +.build:arm64:
> +  extends:
> +    - .build
> +    - .use-debian/arm64_build
> +  tags:
> +    - aarch64
> +  variables:
> +    DEFCONFIG: "arch/arm64/configs/defconfig"
> +    KERNEL_IMAGE_NAME: "Image"
> +    KERNEL_ARCH: "arm64"
> +
> +.build:x86_64:
> +  extends:
> +    - .build
> +    - .use-debian/x86_64_build
> +  variables:
> +    DEFCONFIG: "arch/x86/configs/x86_64_defconfig"
> +    KERNEL_IMAGE_NAME: "bzImage"
> +    KERNEL_ARCH: "x86_64"
> +
> +
> +# Build IGT for testing on devices
> +
> +igt:arm32:
> +  extends: .build:arm32
> +  script:
> +    - FDO_CI_CONCURRENT=${FDO_CI_CONCURRENT} bash drivers/gpu/drm/ci/build-igt.sh
> +
> +igt:arm64:
> +  extends: .build:arm64
> +  script:
> +    - FDO_CI_CONCURRENT=${FDO_CI_CONCURRENT} bash drivers/gpu/drm/ci/build-igt.sh
> +
> +igt:x86_64:
> +  extends: .build:x86_64
> +  script:
> +    - FDO_CI_CONCURRENT=${FDO_CI_CONCURRENT} bash drivers/gpu/drm/ci/build-igt.sh
> +
> +# Build kernels for testing on devices
> +
> +testing:arm32:
> +  extends: .build:arm32
> +  variables:
> +    # Would be good to have DEBUG_KMEMLEAK, but it doesn't work well with any of
> +    # PROVE_LOCKING and KASAN as of 5.17.
> +    #
> +    # db410c and db820c don't boot with KASAN_INLINE, probably due to the kernel
> +    # becoming too big for their bootloaders.
> +    ENABLE_KCONFIGS: "PROVE_LOCKING DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT"
> +    UPLOAD_TO_MINIO: 1
> +    MERGE_FRAGMENT: arm.config
> +
> +testing:arm64:
> +  extends: .build:arm64
> +  variables:
> +    # Would be good to have DEBUG_KMEMLEAK, but it doesn't work well with any of
> +    # PROVE_LOCKING and KASAN as of 5.17.
> +    #
> +    # db410c and db820c don't boot with KASAN_INLINE, probably due to the kernel
> +    # becoming too big for their bootloaders.
> +    ENABLE_KCONFIGS: "PROVE_LOCKING DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT"
> +    UPLOAD_TO_MINIO: 1
> +    MERGE_FRAGMENT: arm64.config
> +
> +testing:x86_64:
> +  extends: .build:x86_64
> +  variables:
> +    # Would be good to have DEBUG_KMEMLEAK, but it doesn't work well with any of
> +    # PROVE_LOCKING and KASAN as of 5.17.
> +    #
> +    # db410c and db820c don't boot with KASAN_INLINE, probably due to the kernel
> +    # becoming too big for their bootloaders.
> +    ENABLE_KCONFIGS: "PROVE_LOCKING DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT"
> +    UPLOAD_TO_MINIO: 1
> +    MERGE_FRAGMENT: x86_64.config
> +
> +
> +# Jobs for build-testing different configurations
> +
> +build:arm32:
> +  extends: .build:arm32
> +
> +build-nodebugfs:arm64:
> +  extends: .build:arm64
> +  variables:
> +    DISABLE_KCONFIGS: "DEBUG_FS"
> +
> +build:x86_64:
> +  extends: .build:x86_64
> diff --git a/drivers/gpu/drm/ci/check-patch.py b/drivers/gpu/drm/ci/check-patch.py
> new file mode 100755
> index 000000000000..a5f399a20e25
> --- /dev/null
> +++ b/drivers/gpu/drm/ci/check-patch.py
> @@ -0,0 +1,57 @@
> +#!/usr/bin/env python3
> +# SPDX-License-Identifier: GPL-2.0-or-later
> +#
> +# check-patch.py: run checkpatch.pl across all commits in a branch
> +#
> +# Based on qemu/.gitlab-ci.d/check-patch.py
> +#
> +# Copyright (C) 2020 Red Hat, Inc.
> +# Copyright (C) 2022 Collabora Ltd.
> +
> +import os
> +import os.path
> +import sys
> +import subprocess
> +
> +repourl = "https://gitlab.freedesktop.org/%s.git" % os.environ["CI_MERGE_REQUEST_PROJECT_PATH"]
> +
> +# GitLab CI environment does not give us any direct info about the
> +# base for the user's branch. We thus need to figure out a common
> +# ancestor between the user's branch and current git master.
> +os.environ["GIT_DEPTH"] = "1000"
> +subprocess.call(["git", "remote", "remove", "check-patch"], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
> +subprocess.check_call(["git", "remote", "add", "check-patch", repourl])
> +subprocess.check_call(["git", "fetch", "check-patch", os.environ["CI_MERGE_REQUEST_TARGET_BRANCH_NAME"]],
> +                      stdout=subprocess.DEVNULL,
> +                      stderr=subprocess.DEVNULL)
> +
> +ancestor = subprocess.check_output(["git", "merge-base",
> +                                    "check-patch/%s" % os.environ["CI_MERGE_REQUEST_TARGET_BRANCH_NAME"], "HEAD"],
> +                                   universal_newlines=True)
> +
> +ancestor = ancestor.strip()
> +
> +log = subprocess.check_output(["git", "log", "--format=%H %s",
> +                               ancestor + "..."],
> +                              universal_newlines=True)
> +
> +subprocess.check_call(["git", "remote", "rm", "check-patch"])
> +
> +if log == "":
> +    print("\nNo commits since %s, skipping checks\n" % ancestor)
> +    sys.exit(0)
> +
> +errors = False
> +
> +print("\nChecking all commits since %s...\n" % ancestor, flush=True)
> +
> +ret = subprocess.run(["scripts/checkpatch.pl",
> +                      "--terse",
> +                      "--types", os.environ["CHECKPATCH_TYPES"],
> +                      "--git", ancestor + "..."])
> +
> +if ret.returncode != 0:
> +    print("    ❌ FAIL one or more commits failed scripts/checkpatch.pl")
> +    sys.exit(1)
> +
> +sys.exit(0)
> diff --git a/drivers/gpu/drm/ci/container.yml b/drivers/gpu/drm/ci/container.yml
> new file mode 100644
> index 000000000000..9764e7921a4f
> --- /dev/null
> +++ b/drivers/gpu/drm/ci/container.yml
> @@ -0,0 +1,65 @@
> +.container:
> +  variables:
> +    CI_REPOSITORY_URL: ${DRM_CI_PROJECT_URL}.git # So ci-templates clones drm-ci instead of the repo to test
> +    CI_COMMIT_SHA: ${DRM_CI_COMMIT_SHA}
> +
> +debian/x86_64_build-base:
> +  variables:
> +    EXTRA_LOCAL_PACKAGES: "libcairo-dev libdw-dev libjson-c-dev libkmod2 libkmod-dev libpciaccess-dev libproc2-dev libudev-dev libunwind-dev python3-docutils bc python3-ply libssl-dev bc"
> +
> +debian/x86_64_test-gl:
> +  variables:
> +    EXTRA_LOCAL_PACKAGES: "jq libasound2 libcairo2 libdw1 libglib2.0-0 libjson-c5 libkmod-dev libkmod2 libgles2 libproc2-dev"
> +
> +debian/arm64_build:
> +  variables:
> +    EXTRA_LOCAL_PACKAGES: "libcairo-dev libdw-dev libjson-c-dev libproc2-dev libkmod2 libkmod-dev libpciaccess-dev libudev-dev libunwind-dev python3-docutils libssl-dev crossbuild-essential-armhf libkmod-dev:armhf libproc2-dev:armhf libunwind-dev:armhf libdw-dev:armhf libpixman-1-dev:armhf libcairo-dev:armhf libudev-dev:armhf libjson-c-dev:armhf"
> +
> +.kernel+rootfs:
> +  variables:
> +    EXTRA_LOCAL_PACKAGES: "jq libasound2 libcairo2 libdw1 libglib2.0-0 libjson-c5"
> +
> +# Disable container jobs that we won't use
> +alpine/x86_64_build:
> +  rules:
> +    - when: never
> +
> +debian/x86_64_test-vk:
> +  rules:
> +    - when: never
> +
> +fedora/x86_64_build:
> +  rules:
> +    - when: never
> +
> +debian/android_build:
> +  rules:
> +    - when: never
> +
> +debian/x86_64_test-android:
> +  rules:
> +    - when: never
> +
> +windows_build_vs2019:
> +  rules:
> +    - when: never
> +
> +windows_test_vs2019:
> +  rules:
> +    - when: never
> +
> +.debian/x86_64_build-mingw:
> +   rules:
> +    - when: never
> +
> +rustfmt:
> +   rules:
> +    - when: never
> +
> +windows_vs2019:
> +   rules:
> +    - when: never
> +
> +clang-format:
> +   rules:
> +    - when: never
> \ No newline at end of file
> diff --git a/drivers/gpu/drm/ci/gitlab-ci.yml b/drivers/gpu/drm/ci/gitlab-ci.yml
> new file mode 100644
> index 000000000000..2c4df53f5dfe
> --- /dev/null
> +++ b/drivers/gpu/drm/ci/gitlab-ci.yml
> @@ -0,0 +1,251 @@
> +variables:
> +  DRM_CI_PROJECT_PATH: &drm-ci-project-path mesa/mesa
> +  DRM_CI_COMMIT_SHA: &drm-ci-commit-sha 0dc961645c4f0241f8512cb0ec3ad59635842072
> +
> +  UPSTREAM_REPO: git://anongit.freedesktop.org/drm/drm
> +  TARGET_BRANCH: drm-next
> +
> +  IGT_VERSION: 471bfababd070e1dac0ebb87470ac4f2ae85e663
> +
> +  DEQP_RUNNER_GIT_URL: https://gitlab.freedesktop.org/anholt/deqp-runner.git
> +  DEQP_RUNNER_GIT_TAG: v0.15.0
> +
> +  FDO_UPSTREAM_REPO: helen.fornazier/linux   # The repo where the git-archive daily runs
> +  MESA_TEMPLATES_COMMIT: &ci-templates-commit d5aa3941aa03c2f716595116354fb81eb8012acb
> +  DRM_CI_PROJECT_URL: https://gitlab.freedesktop.org/${DRM_CI_PROJECT_PATH}
> +  CI_PRE_CLONE_SCRIPT: |-
> +          set -o xtrace
> +          curl -L --retry 4 -f --retry-all-errors --retry-delay 60 -s ${DRM_CI_PROJECT_URL}/-/raw/${DRM_CI_COMMIT_SHA}/.gitlab-ci/download-git-cache.sh -o download-git-cache.sh
> +          bash download-git-cache.sh
> +          rm download-git-cache.sh
> +          set +o xtrace
> +  S3_HOST: s3.freedesktop.org
> +  # per-pipeline artifact storage on MinIO
> +  PIPELINE_ARTIFACTS_BASE: ${S3_HOST}/artifacts/${CI_PROJECT_PATH}/${CI_PIPELINE_ID}
> +  # per-job artifact storage on MinIO
> +  JOB_ARTIFACTS_BASE: ${PIPELINE_ARTIFACTS_BASE}/${CI_JOB_ID}
> +
> +  LAVA_JOB_PRIORITY: 30
> +
> +default:
> +  before_script:
> +    - export SCRIPTS_DIR=$(mktemp -d)
> +    - curl -L -s --retry 4 -f --retry-all-errors --retry-delay 60 -O --output-dir "${SCRIPTS_DIR}" "${DRM_CI_PROJECT_URL}/-/raw/${DRM_CI_COMMIT_SHA}/.gitlab-ci/setup-test-env.sh"
> +    - source ${SCRIPTS_DIR}/setup-test-env.sh
> +    - echo -e "\e[0Ksection_start:$(date +%s):unset_env_vars_section[collapsed=true]\r\e[0KUnsetting vulnerable environment variables"
> +    - export CI_JOB_JWT_FILE="${CI_JOB_JWT_FILE:-$(mktemp)}"
> +    - echo -n "${CI_JOB_JWT}" > "${CI_JOB_JWT_FILE}"
> +    - unset CI_JOB_JWT
> +    - echo -e "\e[0Ksection_end:$(date +%s):unset_env_vars_section\r\e[0K"
> +
> +    - echo -e "\e[0Ksection_start:$(date +%s):drm_ci_download_section[collapsed=true]\r\e[0KDownloading mesa from $DRM_CI_PROJECT_URL/-/archive/$DRM_CI_COMMIT_SHA/mesa-$DRM_CI_COMMIT_SHA.tar.gz"
> +    - cd $CI_PROJECT_DIR
> +    - curl --output - $DRM_CI_PROJECT_URL/-/archive/$DRM_CI_COMMIT_SHA/mesa-$DRM_CI_COMMIT_SHA.tar.gz | tar -xz
> +    - mv mesa-$DRM_CI_COMMIT_SHA/.gitlab-ci* .
> +    - rm -rf mesa-$DRM_CI_COMMIT_SHA/
> +    - echo -e "\e[0Ksection_end:$(date +%s):drm_ci_download_section\r\e[0K"
> +
> +  after_script:
> +    - >
> +      set +x
> +
> +      test -e "${CI_JOB_JWT_FILE}" &&
> +      export CI_JOB_JWT="$(<${CI_JOB_JWT_FILE})" &&
> +      rm "${CI_JOB_JWT_FILE}"
> +
> +  # Retry when job fails.
> +  retry:
> +    max: 1
> +    # Ignore runner_unsupported, stale_schedule, archived_failure, or
> +    # unmet_prerequisites
> +    when:
> +      - api_failure
> +      - runner_system_failure
> +      - script_failure
> +      - job_execution_timeout
> +      - scheduler_failure
> +      - data_integrity_failure
> +      - unknown_failure
> +
> +include:
> +  - project: 'freedesktop/ci-templates'
> +    ref: 16bc29078de5e0a067ff84a1a199a3760d3b3811
> +    file:
> +      - '/templates/ci-fairy.yml'
> +  - project: 'freedesktop/ci-templates'
> +    ref: *ci-templates-commit
> +    file:
> +      - '/templates/alpine.yml'
> +      - '/templates/debian.yml'
> +      - '/templates/fedora.yml'
> +  - project: *drm-ci-project-path
> +    ref: *drm-ci-commit-sha
> +    file:
> +      - '/.gitlab-ci/farm-rules.yml'
> +      - '/.gitlab-ci/test-source-dep.yml'
> +      - '/.gitlab-ci/container/gitlab-ci.yml'
> +      - '/.gitlab-ci/test/gitlab-ci.yml'
> +      - '/.gitlab-ci/lava/lava-gitlab-ci.yml'
> +  - drivers/gpu/drm/ci/image-tags.yml
> +  - drivers/gpu/drm/ci/container.yml
> +  - drivers/gpu/drm/ci/static-checks.yml
> +  - drivers/gpu/drm/ci/build.yml
> +  - drivers/gpu/drm/ci/test.yml
> +  - 'https://gitlab.freedesktop.org/gfx-ci/lab-status/-/raw/main/lab-status.yml'
> +
> +
> +stages:
> +  - sanity
> +  - container
> +  - git-archive
> +  - build
> +  - amdgpu
> +  - i915
> +  - mediatek
> +  - meson
> +  - msm
> +  - rockchip
> +  - virtio-gpu
> +  - lint
> +
> +# YAML anchors for rule conditions
> +# --------------------------------
> +.rules-anchors:
> +  rules:
> +    # Pipeline for forked project branch
> +    - if: &is-forked-branch '$CI_COMMIT_BRANCH && $CI_PROJECT_NAMESPACE != "mesa"'
> +      when: manual
> +    # Forked project branch / pre-merge pipeline not for Marge bot
> +    - if: &is-forked-branch-or-pre-merge-not-for-marge '$CI_PROJECT_NAMESPACE != "mesa" || ($GITLAB_USER_LOGIN != "marge-bot" && $CI_PIPELINE_SOURCE == "merge_request_event")'
> +      when: manual
> +    # Pipeline runs for the main branch of the upstream Mesa project
> +    - if: &is-mesa-main '$CI_PROJECT_NAMESPACE == "mesa" && $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH && $CI_COMMIT_BRANCH'
> +      when: always
> +    # Post-merge pipeline
> +    - if: &is-post-merge '$CI_PROJECT_NAMESPACE == "mesa" && $CI_COMMIT_BRANCH'
> +      when: on_success
> +    # Post-merge pipeline, not for Marge Bot
> +    - if: &is-post-merge-not-for-marge '$CI_PROJECT_NAMESPACE == "mesa" && $GITLAB_USER_LOGIN != "marge-bot" && $CI_COMMIT_BRANCH'
> +      when: on_success
> +    # Pre-merge pipeline
> +    - if: &is-pre-merge '$CI_PIPELINE_SOURCE == "merge_request_event"'
> +      when: on_success
> +    # Pre-merge pipeline for Marge Bot
> +    - if: &is-pre-merge-for-marge '$GITLAB_USER_LOGIN == "marge-bot" && $CI_PIPELINE_SOURCE == "merge_request_event"'
> +      when: on_success
> +
> +# Rule to filter for only scheduled pipelines.
> +.scheduled_pipeline-rules:
> +  rules:
> +    - if: &is-scheduled-pipeline '$CI_PIPELINE_SOURCE == "schedule"'
> +      when: on_success
> +
> +# Generic rule to not run the job during scheduled pipelines. Jobs that aren't
> +# something like a nightly run should include this rule.
> +.no_scheduled_pipelines-rules:
> +  rules:
> +    - if: *is-scheduled-pipeline
> +      when: never
> +
> +# When to automatically run the CI for build jobs
> +.build-rules:
> +  rules:
> +    - !reference [.no_scheduled_pipelines-rules, rules]
> +    # Run automatically once all dependency jobs have passed
> +    - when: on_success
> +
> +
> +.ci-deqp-artifacts:
> +  artifacts:
> +    name: "mesa_${CI_JOB_NAME}"
> +    when: always
> +    untracked: false
> +    paths:
> +      # Watch out!  Artifacts are relative to the build dir.
> +      # https://gitlab.com/gitlab-org/gitlab-ce/commit/8788fb925706cad594adf6917a6c5f6587dd1521
> +      - artifacts
> +      - _build/meson-logs/*.txt
> +      - _build/meson-logs/strace
> +
> +
> +.container-rules:
> +  rules:
> +    - !reference [.no_scheduled_pipelines-rules, rules]
> +    # Run pipeline by default in the main project if any CI pipeline
> +    # configuration files were changed, to ensure docker images are up to date
> +    - if: *is-post-merge
> +      changes:
> +      - drivers/gpu/drm/ci/**/*
> +      when: on_success
> +    # Run pipeline by default if it was triggered by Marge Bot, is for a
> +    # merge request, and any files affecting the pipeline were changed
> +    - if: *is-pre-merge-for-marge
> +      when: on_success
> +    # Run pipeline by default in the main project if it was not triggered by
> +    # Marge Bot, and any files affecting the pipeline were changed
> +    - if: *is-post-merge-not-for-marge
> +      when: on_success
> +    # Allow triggering jobs manually in other cases
> +    - when: manual
> +
> +
> +
> +# Git archive
> +
> +make git archive:
> +  extends:
> +    - .fdo.ci-fairy
> +  stage: git-archive
> +  rules:
> +    - !reference [.scheduled_pipeline-rules, rules]
> +  # ensure we are running on packet
> +  tags:
> +    - packet.net
> +  script:
> +    # Remove drm-ci files we just added
> +    - rm -rf .gitlab-ci.*
> +
> +    # Compactify the .git directory
> +    - git gc --aggressive
> +    # compress the current folder
> +    - tar -cvzf ../$CI_PROJECT_NAME.tar.gz .
> +
> +    # login with the JWT token file
> +    - ci-fairy s3cp --token-file "${CI_JOB_JWT_FILE}" ../$CI_PROJECT_NAME.tar.gz https://$S3_HOST/git-cache/$CI_PROJECT_NAMESPACE/$CI_PROJECT_NAME/$CI_PROJECT_NAME.tar.gz
> +
> +
> +# Sanity checks of MR settings and commit logs
> +sanity:
> +  extends:
> +    - .fdo.ci-fairy
> +  stage: sanity
> +  rules:
> +    - if: *is-pre-merge
> +      when: on_success
> +    # Other cases default to never
> +  variables:
> +    GIT_STRATEGY: none
> +  script:
> +    # ci-fairy check-commits --junit-xml=check-commits.xml
> +    - ci-fairy check-merge-request --require-allow-collaboration --junit-xml=check-merge-request.xml
> +  artifacts:
> +    when: on_failure
> +    reports:
> +      junit: check-*.xml
> +
> +# Rules for tests that should not block merging, but should be available to
> +# optionally run with the "play" button in the UI in pre-merge non-marge
> +# pipelines.  This should appear in "extends:" after any includes of
> +# test-source-dep.yml rules, so that these rules replace those.
> +.test-manual-mr:
> +  rules:
> +    - !reference [.no_scheduled_pipelines-rules, rules]
> +    - if: *is-forked-branch-or-pre-merge-not-for-marge
> +      when: manual
> +  variables:
> +    JOB_TIMEOUT: 80
> +
> +
> +# Jobs that need to pass before spending hardware resources on further testing
> +.required-for-hardware-jobs:
> +  needs: []
> \ No newline at end of file
> diff --git a/drivers/gpu/drm/ci/igt_runner.sh b/drivers/gpu/drm/ci/igt_runner.sh
> new file mode 100755
> index 000000000000..2bb759165063
> --- /dev/null
> +++ b/drivers/gpu/drm/ci/igt_runner.sh
> @@ -0,0 +1,77 @@
> +#!/bin/sh
> +# SPDX-License-Identifier: MIT
> +
> +set -ex
> +
> +export IGT_FORCE_DRIVER=${DRIVER_NAME}
> +export PATH=$PATH:/igt/bin/
> +export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/igt/lib/aarch64-linux-gnu/:/igt/lib/x86_64-linux-gnu:/igt/lib:/igt/lib64
> +
> +# Uncomment the below to debug problems with driver probing
> +: '
> +ls -l /dev/dri/
> +cat /sys/kernel/debug/devices_deferred
> +cat /sys/kernel/debug/device_component/*
> +'
> +
> +# Dump drm state to confirm that kernel was able to find a connected display:
> +# TODO this path might not exist for all drivers.. maybe run modetest instead?
> +set +e
> +cat /sys/kernel/debug/dri/*/state
> +set -e
> +
> +# Cannot use HWCI_KERNEL_MODULES as at that point we don't have the module in /lib
> +if [ "$IGT_FORCE_DRIVER" = "amdgpu" ]; then
> +    mv /install/modules/lib/modules/* /lib/modules/.
> +    modprobe amdgpu
> +fi
> +
> +if [ -e "/install/xfails/$DRIVER_NAME-$GPU_VERSION-skips.txt" ]; then
> +    IGT_SKIPS="--skips /install/xfails/$DRIVER_NAME-$GPU_VERSION-skips.txt"
> +fi
> +
> +if [ -e "/install/xfails/$DRIVER_NAME-$GPU_VERSION-flakes.txt" ]; then
> +    IGT_FLAKES="--flakes /install/xfails/$DRIVER_NAME-$GPU_VERSION-flakes.txt"
> +fi
> +
> +if [ -e "/install/xfails/$DRIVER_NAME-$GPU_VERSION-fails.txt" ]; then
> +    IGT_FAILS="--baseline /install/xfails/$DRIVER_NAME-$GPU_VERSION-fails.txt"
> +fi
> +
> +if [ "`uname -m`" = "aarch64" ]; then
> +    ARCH="arm64"
> +elif [ "`uname -m`" = "armv7l" ]; then
> +    ARCH="arm"
> +else
> +    ARCH="x86_64"
> +fi
> +
> +curl -L --retry 4 -f --retry-all-errors --retry-delay 60 -s ${FDO_HTTP_CACHE_URI:-}$PIPELINE_ARTIFACTS_BASE/$ARCH/igt.tar.gz | tar --zstd -v -x -C /
> +
> +set +e
> +igt-runner \
> +    run \
> +    --igt-folder /igt/libexec/igt-gpu-tools \
> +    --caselist /install/testlist.txt \
> +    --output /results \
> +    $IGT_SKIPS \
> +    $IGT_FLAKES \
> +    $IGT_FAILS \
> +    --fraction-start $CI_NODE_INDEX \
> +    --fraction $CI_NODE_TOTAL \
> +    --jobs 1
> +ret=$?
> +set -e
> +
> +deqp-runner junit \
> +   --testsuite IGT \
> +   --results /results/failures.csv \
> +   --output /results/junit.xml \
> +   --limit 50 \
> +   --template "See https://$CI_PROJECT_ROOT_NAMESPACE.pages.freedesktop.org/-/$CI_PROJECT_NAME/-/jobs/$CI_JOB_ID/artifacts/results/{{testcase}}.xml"
> +
> +# Store the results also in the simpler format used by the runner in ChromeOS CI
> +#sed -r 's/(dmesg-warn|pass)/success/g' /results/results.txt > /results/results_simple.txt
> +
> +cd $oldpath
> +exit $ret
> diff --git a/drivers/gpu/drm/ci/image-tags.yml b/drivers/gpu/drm/ci/image-tags.yml
> new file mode 100644
> index 000000000000..f051b6c547c5
> --- /dev/null
> +++ b/drivers/gpu/drm/ci/image-tags.yml
> @@ -0,0 +1,15 @@
> +variables:
> +   CONTAINER_TAG: "2023-08-10-mesa-uprev"
> +   DEBIAN_X86_64_BUILD_BASE_IMAGE: "debian/x86_64_build-base"
> +   DEBIAN_BASE_TAG: "${CONTAINER_TAG}"
> +
> +   DEBIAN_X86_64_BUILD_IMAGE_PATH: "debian/x86_64_build"
> +   DEBIAN_BUILD_TAG: "${CONTAINER_TAG}"
> +
> +   KERNEL_ROOTFS_TAG: "${CONTAINER_TAG}"
> +
> +   DEBIAN_X86_64_TEST_BASE_IMAGE: "debian/x86_64_test-base"
> +   DEBIAN_X86_64_TEST_IMAGE_GL_PATH: "debian/x86_64_test-gl"
> +   DEBIAN_X86_64_TEST_GL_TAG: "${CONTAINER_TAG}"
> +
> +   ALPINE_X86_64_LAVA_SSH_TAG: "${CONTAINER_TAG}"
> \ No newline at end of file
> diff --git a/drivers/gpu/drm/ci/lava-submit.sh b/drivers/gpu/drm/ci/lava-submit.sh
> new file mode 100755
> index 000000000000..0c4456b21b0f
> --- /dev/null
> +++ b/drivers/gpu/drm/ci/lava-submit.sh
> @@ -0,0 +1,57 @@
> +#!/bin/bash
> +# SPDX-License-Identifier: MIT
> +
> +set -e
> +set -x
> +
> +# Try to use the kernel and rootfs built in mainline first, so we're more
> +# likely to hit cache
> +if curl -L --retry 4 -f --retry-all-errors --retry-delay 60 -s "https://${BASE_SYSTEM_MAINLINE_HOST_PATH}/done"; then
> +	BASE_SYSTEM_HOST_PATH="${BASE_SYSTEM_MAINLINE_HOST_PATH}"
> +else
> +	BASE_SYSTEM_HOST_PATH="${BASE_SYSTEM_FORK_HOST_PATH}"
> +fi
> +
> +rm -rf results
> +mkdir -p results/job-rootfs-overlay/
> +
> +cp artifacts/ci-common/capture-devcoredump.sh results/job-rootfs-overlay/
> +cp artifacts/ci-common/init-*.sh results/job-rootfs-overlay/
> +cp artifacts/ci-common/intel-gpu-freq.sh results/job-rootfs-overlay/
> +cp "$SCRIPTS_DIR"/setup-test-env.sh results/job-rootfs-overlay/
> +
> +# Prepare env vars for upload.
> +section_start variables "Variables passed through:"
> +KERNEL_IMAGE_BASE_URL="https://${BASE_SYSTEM_HOST_PATH}" \
> +	artifacts/ci-common/generate-env.sh | tee results/job-rootfs-overlay/set-job-env-vars.sh
> +section_end variables
> +
> +tar zcf job-rootfs-overlay.tar.gz -C results/job-rootfs-overlay/ .
> +ci-fairy s3cp --token-file "${CI_JOB_JWT_FILE}" job-rootfs-overlay.tar.gz "https://${JOB_ROOTFS_OVERLAY_PATH}"
> +
> +touch results/lava.log
> +tail -f results/lava.log &
> +
> +PYTHONPATH=artifacts/ artifacts/lava/lava_job_submitter.py \
> +	submit \
> +	--dump-yaml \
> +	--pipeline-info "$CI_JOB_NAME: $CI_PIPELINE_URL on $CI_COMMIT_REF_NAME ${CI_NODE_INDEX}/${CI_NODE_TOTAL}" \
> +	--rootfs-url-prefix "https://${BASE_SYSTEM_HOST_PATH}" \
> +	--kernel-url-prefix "https://${PIPELINE_ARTIFACTS_BASE}/${ARCH}" \
> +	--build-url "${FDO_HTTP_CACHE_URI:-}https://${PIPELINE_ARTIFACTS_BASE}/${ARCH}/kernel-files.tar.zst" \
> +	--job-rootfs-overlay-url "${FDO_HTTP_CACHE_URI:-}https://${JOB_ROOTFS_OVERLAY_PATH}" \
> +	--job-timeout-min ${JOB_TIMEOUT:-80} \
> +	--first-stage-init artifacts/ci-common/init-stage1.sh \
> +	--ci-project-dir "${CI_PROJECT_DIR}" \
> +	--device-type "${DEVICE_TYPE}" \
> +	--dtb-filename "${DTB}" \
> +	--jwt-file "${CI_JOB_JWT_FILE}" \
> +	--kernel-image-name "${KERNEL_IMAGE_NAME}" \
> +	--kernel-image-type "${KERNEL_IMAGE_TYPE}" \
> +	--boot-method "${BOOT_METHOD}" \
> +	--visibility-group "${VISIBILITY_GROUP}" \
> +	--lava-tags "${LAVA_TAGS}" \
> +	--mesa-job-name "$CI_JOB_NAME" \
> +	--structured-log-file "results/lava_job_detail.json" \
> +	--ssh-client-image "${LAVA_SSH_CLIENT_IMAGE}" \
> +	>> results/lava.log
> diff --git a/drivers/gpu/drm/ci/static-checks.yml b/drivers/gpu/drm/ci/static-checks.yml
> new file mode 100644
> index 000000000000..13ffa827b7fa
> --- /dev/null
> +++ b/drivers/gpu/drm/ci/static-checks.yml
> @@ -0,0 +1,12 @@
> +check-patch:
> +  extends:
> +    - .build
> +    - .use-debian/x86_64_build
> +  script:
> +    - drivers/gpu/drm/ci/check-patch.py
> +  variables:
> +    CHECKPATCH_TYPES: "BAD_SIGN_OFF,BAD_STABLE_ADDRESS_STYLE,COMMIT_COMMENT_SYMBOL,COMMIT_MESSAGE,EMAIL_SUBJECT,FROM_SIGN_OFF_MISMATCH,MISSING_SIGN_OFF,NO_AUTHOR_SIGN_OFF,DIFF_IN_COMMIT_MSG,GERRIT_CHANGE_ID,GIT_COMMIT_ID,UNKNOWN_COMMIT_ID,CODE_INDENT,BIT_MACRO,DOS_LINE_ENDINGS"
> +  rules:
> +    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
> +      when: on_success
> +    # Other cases default to never
> diff --git a/drivers/gpu/drm/ci/test.yml b/drivers/gpu/drm/ci/test.yml
> new file mode 100644
> index 000000000000..6473cddaa7a9
> --- /dev/null
> +++ b/drivers/gpu/drm/ci/test.yml
> @@ -0,0 +1,335 @@
> +.test-rules:
> +  rules:
> +    - if: '$FD_FARM == "offline" && $RUNNER_TAG =~ /^google-freedreno-/'
> +      when: never
> +    - if: '$COLLABORA_FARM == "offline" && $RUNNER_TAG =~ /^mesa-ci-x86-64-lava-/'
> +      when: never
> +    - !reference [.no_scheduled_pipelines-rules, rules]
> +    - when: on_success
> +
> +.lava-test:
> +  extends:
> +    - .test-rules
> +  script:
> +    # Note: Build dir (and thus install) may be dirty due to GIT_STRATEGY
> +    - rm -rf install
> +    - tar -xf artifacts/install.tar
> +    - mv install/* artifacts/.
> +    # Override it with our lava-submit.sh script
> +    - ./artifacts/lava-submit.sh
> +
> +.lava-igt:arm32:
> +  extends:
> +    - .lava-test:arm32
> +  variables:
> +    HWCI_TEST_SCRIPT: "/install/igt_runner.sh"
> +    ARCH: "armhf"
> +  dependencies:
> +    - testing:arm32
> +  needs:
> +    - alpine/x86_64_lava_ssh_client
> +    - kernel+rootfs_arm32
> +    - debian/x86_64_build
> +    - testing:arm32
> +    - igt:arm32
> +
> +.lava-igt:arm64:
> +  extends:
> +    - .lava-test:arm64
> +  variables:
> +    HWCI_TEST_SCRIPT: "/install/igt_runner.sh"
> +    ARCH: "arm64"
> +  dependencies:
> +    - testing:arm64
> +  needs:
> +    - alpine/x86_64_lava_ssh_client
> +    - kernel+rootfs_arm64
> +    - debian/x86_64_build
> +    - testing:arm64
> +    - igt:arm64
> +
> +.lava-igt:x86_64:
> +  extends:
> +    - .lava-test:x86_64
> +  variables:
> +    HWCI_TEST_SCRIPT: "/install/igt_runner.sh"
> +    ARCH: "x86_64"
> +  dependencies:
> +    - testing:x86_64
> +  needs:
> +    - alpine/x86_64_lava_ssh_client
> +    - kernel+rootfs_x86_64
> +    - debian/x86_64_build
> +    - testing:x86_64
> +    - igt:x86_64
> +
> +.baremetal-igt-arm64:
> +  extends:
> +    - .baremetal-test-arm64
> +    - .use-debian/arm64_test
> +    - .test-rules
> +  variables:
> +    FDO_CI_CONCURRENT: 10
> +    HWCI_TEST_SCRIPT: "/install/igt_runner.sh"
> +    S3_ARTIFACT_NAME: "arm64/kernel-files"
> +    BM_KERNEL: https://${PIPELINE_ARTIFACTS_BASE}/arm64/Image.gz
> +    BM_CMDLINE: "ip=dhcp console=ttyMSM0,115200n8 $BM_KERNEL_EXTRA_ARGS root=/dev/nfs rw nfsrootdebug nfsroot=,tcp,nfsvers=4.2 init=/init $BM_KERNELARGS"
> +  needs:
> +    - debian/arm64_test
> +    - job: testing:arm64
> +      artifacts: false
> +    - igt:arm64
> +  tags:
> +    - $RUNNER_TAG
> +
> +msm:sc7180:
> +  extends:
> +    - .lava-igt:arm64
> +  stage: msm
> +  parallel: 2
> +  variables:
> +    DRIVER_NAME: msm
> +    DEVICE_TYPE: sc7180-trogdor-lazor-limozeen
> +    DTB: sc7180-trogdor-lazor-limozeen-nots-r5
> +    BOOT_METHOD: depthcharge
> +    KERNEL_IMAGE_TYPE: ""
> +    GPU_VERSION: sc7180
> +    RUNNER_TAG: mesa-ci-x86-64-lava-sc7180-trogdor-lazor-limozeen
> +
> +msm:apq8016:
> +  extends:
> +    - .baremetal-igt-arm64
> +  stage: msm
> +  variables:
> +    DRIVER_NAME: msm
> +    BM_DTB: https://${PIPELINE_ARTIFACTS_BASE}/arm64/apq8016-sbc.dtb
> +    GPU_VERSION: apq8016
> +    BM_CMDLINE: "ip=dhcp console=ttyMSM0,115200n8 $BM_KERNEL_EXTRA_ARGS root=/dev/nfs rw nfsrootdebug nfsroot=,tcp,nfsvers=4.2 init=/init $BM_KERNELARGS"
> +    RUNNER_TAG: google-freedreno-db410c
> +  script:
> +    - ./install/bare-metal/fastboot.sh
> +  rules:
> +    # TODO: current issue: it is not fiding the NFS root. Fix and remove this rule.
> +    - when: never
> +
> +msm:apq8096:
> +  extends:
> +    - .baremetal-igt-arm64
> +  stage: msm
> +  variables:
> +    DRIVER_NAME: msm
> +    BM_KERNEL_EXTRA_ARGS: maxcpus=2
> +    BM_DTB: https://${PIPELINE_ARTIFACTS_BASE}/arm64/apq8096-db820c.dtb
> +    GPU_VERSION: apq8096
> +    RUNNER_TAG: google-freedreno-db820c
> +  script:
> +    - ./install/bare-metal/fastboot.sh
> +
> +msm:sdm845:
> +  extends:
> +    - .baremetal-igt-arm64
> +  stage: msm
> +  parallel: 6
> +  variables:
> +    DRIVER_NAME: msm
> +    BM_KERNEL: https://${PIPELINE_ARTIFACTS_BASE}/arm64/cheza-kernel
> +    GPU_VERSION: sdm845
> +    RUNNER_TAG: google-freedreno-cheza
> +  script:
> +    - ./install/bare-metal/cros-servo.sh
> +
> +rockchip:rk3288:
> +  extends:
> +    - .lava-igt:arm32
> +  stage: rockchip
> +  variables:
> +    DRIVER_NAME: rockchip
> +    DEVICE_TYPE: rk3288-veyron-jaq
> +    DTB: ${DEVICE_TYPE}
> +    BOOT_METHOD: depthcharge
> +    KERNEL_IMAGE_TYPE: "zimage"
> +    GPU_VERSION: rk3288
> +    RUNNER_TAG: mesa-ci-x86-64-lava-rk3288-veyron-jaq
> +
> +rockchip:rk3399:
> +  extends:
> +    - .lava-igt:arm64
> +  stage: rockchip
> +  parallel: 3
> +  variables:
> +    DRIVER_NAME: rockchip
> +    DEVICE_TYPE: rk3399-gru-kevin
> +    DTB: ${DEVICE_TYPE}
> +    BOOT_METHOD: depthcharge
> +    KERNEL_IMAGE_TYPE: ""
> +    GPU_VERSION: rk3399
> +    RUNNER_TAG: mesa-ci-x86-64-lava-rk3399-gru-kevin
> +
> +.i915:
> +  extends:
> +    - .lava-igt:x86_64
> +  stage: i915
> +  variables:
> +    DRIVER_NAME: i915
> +    DTB: ""
> +    BOOT_METHOD: depthcharge
> +    KERNEL_IMAGE_TYPE: ""
> +
> +i915:apl:
> +  extends:
> +    - .i915
> +  parallel: 12
> +  variables:
> +    DEVICE_TYPE: asus-C523NA-A20057-coral
> +    GPU_VERSION: apl
> +    RUNNER_TAG: mesa-ci-x86-64-lava-asus-C523NA-A20057-coral
> +
> +i915:glk:
> +  extends:
> +    - .i915
> +  parallel: 5
> +  variables:
> +    DEVICE_TYPE: hp-x360-12b-ca0010nr-n4020-octopus
> +    GPU_VERSION: glk
> +    RUNNER_TAG: mesa-ci-x86-64-lava-hp-x360-12b-ca0010nr-n4020-octopus
> +
> +i915:amly:
> +  extends:
> +    - .i915
> +  parallel: 8
> +  variables:
> +    DEVICE_TYPE: asus-C433TA-AJ0005-rammus
> +    GPU_VERSION: amly
> +    RUNNER_TAG: mesa-ci-x86-64-lava-asus-C433TA-AJ0005-rammus
> +
> +i915:kbl:
> +  extends:
> +    - .i915
> +  parallel: 5
> +  variables:
> +    DEVICE_TYPE: hp-x360-14-G1-sona
> +    GPU_VERSION: kbl
> +    RUNNER_TAG: mesa-ci-x86-64-lava-hp-x360-14-G1-sona
> +
> +i915:whl:
> +  extends:
> +    - .i915
> +  parallel: 8
> +  variables:
> +    DEVICE_TYPE: dell-latitude-5400-8665U-sarien
> +    GPU_VERSION: whl
> +    RUNNER_TAG: mesa-ci-x86-64-lava-dell-latitude-5400-8665U-sarien
> +
> +i915:cml:
> +  extends:
> +    - .i915
> +  parallel: 6
> +  variables:
> +    DEVICE_TYPE: asus-C436FA-Flip-hatch
> +    GPU_VERSION: cml
> +    RUNNER_TAG: mesa-ci-x86-64-lava-asus-C436FA-flip-hatch
> +
> +i915:tgl:
> +  extends:
> +    - .i915
> +  parallel: 6
> +  variables:
> +    DEVICE_TYPE: asus-cx9400-volteer
> +    GPU_VERSION: tgl
> +    RUNNER_TAG: mesa-ci-x86-64-lava-asus-cx9400-volteer
> +
> +.amdgpu:
> +  extends:
> +    - .lava-igt:x86_64
> +  stage: amdgpu
> +  variables:
> +    DRIVER_NAME: amdgpu
> +    DTB: ""
> +    BOOT_METHOD: depthcharge
> +    KERNEL_IMAGE_TYPE: ""
> +
> +amdgpu:stoney:
> +  extends:
> +    - .amdgpu
> +  variables:
> +    DEVICE_TYPE: hp-11A-G6-EE-grunt
> +    GPU_VERSION: stoney
> +    RUNNER_TAG: mesa-ci-x86-64-lava-hp-11A-G6-EE-grunt
> +
> +.mediatek:
> +  extends:
> +    - .lava-igt:arm64
> +  stage: mediatek
> +  variables:
> +    DRIVER_NAME: mediatek
> +    DTB: ${DEVICE_TYPE}
> +    BOOT_METHOD: depthcharge
> +    KERNEL_IMAGE_TYPE: ""
> +
> +mediatek:mt8173:
> +  extends:
> +    - .mediatek
> +  variables:
> +    DEVICE_TYPE: mt8173-elm-hana
> +    GPU_VERSION: mt8173
> +    RUNNER_TAG: mesa-ci-x86-64-lava-mt8173-elm-hana
> +  rules:
> +    # TODO: current issue: device is hanging. Fix and remove this rule.
> +    - when: never
> +
> +mediatek:mt8183:
> +  extends:
> +    - .mediatek
> +  variables:
> +    DEVICE_TYPE: mt8183-kukui-jacuzzi-juniper-sku16
> +    GPU_VERSION: mt8183
> +    RUNNER_TAG: mesa-ci-x86-64-lava-mt8183-kukui-jacuzzi-juniper-sku16
> +
> +# drm-mtk doesn't even probe yet in mainline for mt8192
> +.mediatek:mt8192:
> +  extends:
> +    - .mediatek
> +  variables:
> +    DEVICE_TYPE: mt8192-asurada-spherion-r0
> +    GPU_VERSION: mt8192
> +    RUNNER_TAG: mesa-ci-x86-64-lava-mt8192-asurada-spherion-r0
> +
> +.meson:
> +  extends:
> +    - .lava-igt:arm64
> +  stage: meson
> +  variables:
> +    DRIVER_NAME: meson
> +    DTB: ${DEVICE_TYPE}
> +    BOOT_METHOD: u-boot
> +    KERNEL_IMAGE_TYPE: "image"
> +
> +meson:g12b:
> +  extends:
> +    - .meson
> +  variables:
> +    DEVICE_TYPE: meson-g12b-a311d-khadas-vim3
> +    GPU_VERSION: g12b
> +    RUNNER_TAG: mesa-ci-x86-64-lava-meson-g12b-a311d-khadas-vim3
> +
> +virtio_gpu:none:
> +  stage: virtio-gpu
> +  variables:
> +    CROSVM_GALLIUM_DRIVER: llvmpipe
> +    DRIVER_NAME: virtio_gpu
> +    GPU_VERSION: none
> +  extends:
> +    - .test-gl
> +  tags:
> +    - kvm
> +  script:
> +    - ln -sf $CI_PROJECT_DIR/install /install
> +    - mv install/bzImage /lava-files/bzImage
> +    - install/crosvm-runner.sh install/igt_runner.sh
> +  needs:
> +    - debian/x86_64_test-gl
> +    - testing:x86_64
> +    - igt:x86_64
> +  rules:
> +    # TODO: current issue: malloc(): corrupted top size. Fix and remove this rule.
> +    - when: never
> \ No newline at end of file
> diff --git a/drivers/gpu/drm/ci/testlist.txt b/drivers/gpu/drm/ci/testlist.txt
> new file mode 100644
> index 000000000000..f82cd90372f4
> --- /dev/null
> +++ b/drivers/gpu/drm/ci/testlist.txt
> @@ -0,0 +1,2912 @@
> +core_auth@getclient-simple
> +core_auth@getclient-master-drop
> +core_auth@basic-auth
> +core_auth@many-magics
> +core_getclient
> +core_getstats
> +core_getversion
> +core_setmaster_vs_auth
> +drm_read@invalid-buffer
> +drm_read@fault-buffer
> +drm_read@empty-block
> +drm_read@empty-nonblock
> +drm_read@short-buffer-block
> +drm_read@short-buffer-nonblock
> +drm_read@short-buffer-wakeup
> +gem_eio@throttle
> +gem_eio@create
> +gem_eio@create-ext
> +gem_eio@context-create
> +gem_eio@execbuf
> +gem_eio@banned
> +gem_eio@suspend
> +gem_eio@hibernate
> +gem_eio@in-flight-external
> +gem_eio@in-flight-suspend
> +gem_eio@reset-stress
> +gem_eio@unwedge-stress
> +gem_eio@wait-immediate
> +gem_eio@wait-wedge-immediate
> +gem_eio@in-flight-immediate
> +gem_eio@in-flight-contexts-immediate
> +gem_eio@in-flight-internal-immediate
> +gem_eio@wait-1us
> +gem_eio@wait-wedge-1us
> +gem_eio@in-flight-1us
> +gem_eio@in-flight-contexts-1us
> +gem_eio@in-flight-internal-1us
> +gem_eio@wait-10ms
> +gem_eio@wait-wedge-10ms
> +gem_eio@in-flight-10ms
> +gem_eio@in-flight-contexts-10ms
> +gem_eio@in-flight-internal-10ms
> +gem_eio@kms
> +kms_3d
> +kms_addfb_basic@unused-handle
> +kms_addfb_basic@unused-pitches
> +kms_addfb_basic@unused-offsets
> +kms_addfb_basic@unused-modifier
> +kms_addfb_basic@clobberred-modifier
> +kms_addfb_basic@invalid-smem-bo-on-discrete
> +kms_addfb_basic@legacy-format
> +kms_addfb_basic@no-handle
> +kms_addfb_basic@basic
> +kms_addfb_basic@bad-pitch-0
> +kms_addfb_basic@bad-pitch-32
> +kms_addfb_basic@bad-pitch-63
> +kms_addfb_basic@bad-pitch-128
> +kms_addfb_basic@bad-pitch-256
> +kms_addfb_basic@bad-pitch-1024
> +kms_addfb_basic@bad-pitch-999
> +kms_addfb_basic@bad-pitch-65536
> +kms_addfb_basic@invalid-get-prop-any
> +kms_addfb_basic@invalid-get-prop
> +kms_addfb_basic@invalid-set-prop-any
> +kms_addfb_basic@invalid-set-prop
> +kms_addfb_basic@master-rmfb
> +kms_addfb_basic@addfb25-modifier-no-flag
> +kms_addfb_basic@addfb25-bad-modifier
> +kms_addfb_basic@addfb25-x-tiled-mismatch-legacy
> +kms_addfb_basic@addfb25-x-tiled-legacy
> +kms_addfb_basic@addfb25-framebuffer-vs-set-tiling
> +kms_addfb_basic@basic-x-tiled-legacy
> +kms_addfb_basic@framebuffer-vs-set-tiling
> +kms_addfb_basic@tile-pitch-mismatch
> +kms_addfb_basic@basic-y-tiled-legacy
> +kms_addfb_basic@size-max
> +kms_addfb_basic@too-wide
> +kms_addfb_basic@too-high
> +kms_addfb_basic@bo-too-small
> +kms_addfb_basic@small-bo
> +kms_addfb_basic@bo-too-small-due-to-tiling
> +kms_addfb_basic@addfb25-y-tiled-legacy
> +kms_addfb_basic@addfb25-yf-tiled-legacy
> +kms_addfb_basic@addfb25-y-tiled-small-legacy
> +kms_addfb_basic@addfb25-4-tiled
> +kms_async_flips@async-flip-with-page-flip-events
> +kms_async_flips@alternate-sync-async-flip
> +kms_async_flips@test-time-stamp
> +kms_async_flips@test-cursor
> +kms_async_flips@invalid-async-flip
> +kms_async_flips@crc
> +kms_atomic@plane-overlay-legacy
> +kms_atomic@plane-primary-legacy
> +kms_atomic@plane-primary-overlay-mutable-zpos
> +kms_atomic@plane-immutable-zpos
> +kms_atomic@test-only
> +kms_atomic@plane-cursor-legacy
> +kms_atomic@plane-invalid-params
> +kms_atomic@plane-invalid-params-fence
> +kms_atomic@crtc-invalid-params
> +kms_atomic@crtc-invalid-params-fence
> +kms_atomic@atomic-invalid-params
> +kms_atomic@atomic_plane_damage
> +kms_atomic_interruptible@legacy-setmode
> +kms_atomic_interruptible@atomic-setmode
> +kms_atomic_interruptible@legacy-dpms
> +kms_atomic_interruptible@legacy-pageflip
> +kms_atomic_interruptible@legacy-cursor
> +kms_atomic_interruptible@universal-setplane-primary
> +kms_atomic_interruptible@universal-setplane-cursor
> +kms_atomic_transition@plane-primary-toggle-with-vblank-wait
> +kms_atomic_transition@plane-all-transition
> +kms_atomic_transition@plane-all-transition-fencing
> +kms_atomic_transition@plane-all-transition-nonblocking
> +kms_atomic_transition@plane-all-transition-nonblocking-fencing
> +kms_atomic_transition@plane-use-after-nonblocking-unbind
> +kms_atomic_transition@plane-use-after-nonblocking-unbind-fencing
> +kms_atomic_transition@plane-all-modeset-transition
> +kms_atomic_transition@plane-all-modeset-transition-fencing
> +kms_atomic_transition@plane-all-modeset-transition-internal-panels
> +kms_atomic_transition@plane-all-modeset-transition-fencing-internal-panels
> +kms_atomic_transition@plane-toggle-modeset-transition
> +kms_atomic_transition@modeset-transition
> +kms_atomic_transition@modeset-transition-fencing
> +kms_atomic_transition@modeset-transition-nonblocking
> +kms_atomic_transition@modeset-transition-nonblocking-fencing
> +kms_big_fb@x-tiled-addfb-size-overflow
> +kms_big_fb@y-tiled-addfb-size-overflow
> +kms_big_fb@yf-tiled-addfb-size-overflow
> +kms_big_fb@4-tiled-addfb-size-overflow
> +kms_big_fb@x-tiled-addfb-size-offset-overflow
> +kms_big_fb@y-tiled-addfb-size-offset-overflow
> +kms_big_fb@yf-tiled-addfb-size-offset-overflow
> +kms_big_fb@4-tiled-addfb-size-offset-overflow
> +kms_big_fb@linear-addfb
> +kms_big_fb@x-tiled-addfb
> +kms_big_fb@y-tiled-addfb
> +kms_big_fb@yf-tiled-addfb
> +kms_big_fb@4-tiled-addfb
> +kms_big_fb@linear-8bpp-rotate-0
> +kms_big_fb@linear-8bpp-rotate-90
> +kms_big_fb@linear-8bpp-rotate-180
> +kms_big_fb@linear-8bpp-rotate-270
> +kms_big_fb@linear-16bpp-rotate-0
> +kms_big_fb@linear-16bpp-rotate-90
> +kms_big_fb@linear-16bpp-rotate-180
> +kms_big_fb@linear-16bpp-rotate-270
> +kms_big_fb@linear-32bpp-rotate-0
> +kms_big_fb@linear-32bpp-rotate-90
> +kms_big_fb@linear-32bpp-rotate-180
> +kms_big_fb@linear-32bpp-rotate-270
> +kms_big_fb@linear-64bpp-rotate-0
> +kms_big_fb@linear-64bpp-rotate-90
> +kms_big_fb@linear-64bpp-rotate-180
> +kms_big_fb@linear-64bpp-rotate-270
> +kms_big_fb@x-tiled-8bpp-rotate-0
> +kms_big_fb@x-tiled-8bpp-rotate-90
> +kms_big_fb@x-tiled-8bpp-rotate-180
> +kms_big_fb@x-tiled-8bpp-rotate-270
> +kms_big_fb@x-tiled-16bpp-rotate-0
> +kms_big_fb@x-tiled-16bpp-rotate-90
> +kms_big_fb@x-tiled-16bpp-rotate-180
> +kms_big_fb@x-tiled-16bpp-rotate-270
> +kms_big_fb@x-tiled-32bpp-rotate-0
> +kms_big_fb@x-tiled-32bpp-rotate-90
> +kms_big_fb@x-tiled-32bpp-rotate-180
> +kms_big_fb@x-tiled-32bpp-rotate-270
> +kms_big_fb@x-tiled-64bpp-rotate-0
> +kms_big_fb@x-tiled-64bpp-rotate-90
> +kms_big_fb@x-tiled-64bpp-rotate-180
> +kms_big_fb@x-tiled-64bpp-rotate-270
> +kms_big_fb@y-tiled-8bpp-rotate-0
> +kms_big_fb@y-tiled-8bpp-rotate-90
> +kms_big_fb@y-tiled-8bpp-rotate-180
> +kms_big_fb@y-tiled-8bpp-rotate-270
> +kms_big_fb@y-tiled-16bpp-rotate-0
> +kms_big_fb@y-tiled-16bpp-rotate-90
> +kms_big_fb@y-tiled-16bpp-rotate-180
> +kms_big_fb@y-tiled-16bpp-rotate-270
> +kms_big_fb@y-tiled-32bpp-rotate-0
> +kms_big_fb@y-tiled-32bpp-rotate-90
> +kms_big_fb@y-tiled-32bpp-rotate-180
> +kms_big_fb@y-tiled-32bpp-rotate-270
> +kms_big_fb@y-tiled-64bpp-rotate-0
> +kms_big_fb@y-tiled-64bpp-rotate-90
> +kms_big_fb@y-tiled-64bpp-rotate-180
> +kms_big_fb@y-tiled-64bpp-rotate-270
> +kms_big_fb@yf-tiled-8bpp-rotate-0
> +kms_big_fb@yf-tiled-8bpp-rotate-90
> +kms_big_fb@yf-tiled-8bpp-rotate-180
> +kms_big_fb@yf-tiled-8bpp-rotate-270
> +kms_big_fb@yf-tiled-16bpp-rotate-0
> +kms_big_fb@yf-tiled-16bpp-rotate-90
> +kms_big_fb@yf-tiled-16bpp-rotate-180
> +kms_big_fb@yf-tiled-16bpp-rotate-270
> +kms_big_fb@yf-tiled-32bpp-rotate-0
> +kms_big_fb@yf-tiled-32bpp-rotate-90
> +kms_big_fb@yf-tiled-32bpp-rotate-180
> +kms_big_fb@yf-tiled-32bpp-rotate-270
> +kms_big_fb@yf-tiled-64bpp-rotate-0
> +kms_big_fb@yf-tiled-64bpp-rotate-90
> +kms_big_fb@yf-tiled-64bpp-rotate-180
> +kms_big_fb@yf-tiled-64bpp-rotate-270
> +kms_big_fb@4-tiled-8bpp-rotate-0
> +kms_big_fb@4-tiled-8bpp-rotate-90
> +kms_big_fb@4-tiled-8bpp-rotate-180
> +kms_big_fb@4-tiled-8bpp-rotate-270
> +kms_big_fb@4-tiled-16bpp-rotate-0
> +kms_big_fb@4-tiled-16bpp-rotate-90
> +kms_big_fb@4-tiled-16bpp-rotate-180
> +kms_big_fb@4-tiled-16bpp-rotate-270
> +kms_big_fb@4-tiled-32bpp-rotate-0
> +kms_big_fb@4-tiled-32bpp-rotate-90
> +kms_big_fb@4-tiled-32bpp-rotate-180
> +kms_big_fb@4-tiled-32bpp-rotate-270
> +kms_big_fb@4-tiled-64bpp-rotate-0
> +kms_big_fb@4-tiled-64bpp-rotate-90
> +kms_big_fb@4-tiled-64bpp-rotate-180
> +kms_big_fb@4-tiled-64bpp-rotate-270
> +kms_big_fb@linear-max-hw-stride-32bpp-rotate-0
> +kms_big_fb@linear-max-hw-stride-32bpp-rotate-180
> +kms_big_fb@linear-max-hw-stride-64bpp-rotate-0
> +kms_big_fb@linear-max-hw-stride-64bpp-rotate-180
> +kms_big_fb@x-tiled-max-hw-stride-32bpp-rotate-0
> +kms_big_fb@x-tiled-max-hw-stride-32bpp-rotate-0-async-flip
> +kms_big_fb@x-tiled-max-hw-stride-32bpp-rotate-180
> +kms_big_fb@x-tiled-max-hw-stride-32bpp-rotate-180-async-flip
> +kms_big_fb@x-tiled-max-hw-stride-64bpp-rotate-0
> +kms_big_fb@x-tiled-max-hw-stride-64bpp-rotate-0-async-flip
> +kms_big_fb@x-tiled-max-hw-stride-64bpp-rotate-180
> +kms_big_fb@x-tiled-max-hw-stride-64bpp-rotate-180-async-flip
> +kms_big_fb@x-tiled-max-hw-stride-32bpp-rotate-0-hflip
> +kms_big_fb@x-tiled-max-hw-stride-32bpp-rotate-0-hflip-async-flip
> +kms_big_fb@x-tiled-max-hw-stride-32bpp-rotate-180-hflip
> +kms_big_fb@x-tiled-max-hw-stride-32bpp-rotate-180-hflip-async-flip
> +kms_big_fb@x-tiled-max-hw-stride-64bpp-rotate-0-hflip
> +kms_big_fb@x-tiled-max-hw-stride-64bpp-rotate-0-hflip-async-flip
> +kms_big_fb@x-tiled-max-hw-stride-64bpp-rotate-180-hflip
> +kms_big_fb@x-tiled-max-hw-stride-64bpp-rotate-180-hflip-async-flip
> +kms_big_fb@y-tiled-max-hw-stride-32bpp-rotate-0
> +kms_big_fb@y-tiled-max-hw-stride-32bpp-rotate-0-async-flip
> +kms_big_fb@y-tiled-max-hw-stride-32bpp-rotate-180
> +kms_big_fb@y-tiled-max-hw-stride-32bpp-rotate-180-async-flip
> +kms_big_fb@y-tiled-max-hw-stride-64bpp-rotate-0
> +kms_big_fb@y-tiled-max-hw-stride-64bpp-rotate-0-async-flip
> +kms_big_fb@y-tiled-max-hw-stride-64bpp-rotate-180
> +kms_big_fb@y-tiled-max-hw-stride-64bpp-rotate-180-async-flip
> +kms_big_fb@y-tiled-max-hw-stride-32bpp-rotate-0-hflip
> +kms_big_fb@y-tiled-max-hw-stride-32bpp-rotate-0-hflip-async-flip
> +kms_big_fb@y-tiled-max-hw-stride-32bpp-rotate-180-hflip
> +kms_big_fb@y-tiled-max-hw-stride-32bpp-rotate-180-hflip-async-flip
> +kms_big_fb@y-tiled-max-hw-stride-64bpp-rotate-0-hflip
> +kms_big_fb@y-tiled-max-hw-stride-64bpp-rotate-0-hflip-async-flip
> +kms_big_fb@y-tiled-max-hw-stride-64bpp-rotate-180-hflip
> +kms_big_fb@y-tiled-max-hw-stride-64bpp-rotate-180-hflip-async-flip
> +kms_big_fb@yf-tiled-max-hw-stride-32bpp-rotate-0
> +kms_big_fb@yf-tiled-max-hw-stride-32bpp-rotate-0-async-flip
> +kms_big_fb@yf-tiled-max-hw-stride-32bpp-rotate-180
> +kms_big_fb@yf-tiled-max-hw-stride-32bpp-rotate-180-async-flip
> +kms_big_fb@yf-tiled-max-hw-stride-64bpp-rotate-0
> +kms_big_fb@yf-tiled-max-hw-stride-64bpp-rotate-0-async-flip
> +kms_big_fb@yf-tiled-max-hw-stride-64bpp-rotate-180
> +kms_big_fb@yf-tiled-max-hw-stride-64bpp-rotate-180-async-flip
> +kms_big_fb@yf-tiled-max-hw-stride-32bpp-rotate-0-hflip
> +kms_big_fb@yf-tiled-max-hw-stride-32bpp-rotate-0-hflip-async-flip
> +kms_big_fb@yf-tiled-max-hw-stride-32bpp-rotate-180-hflip
> +kms_big_fb@yf-tiled-max-hw-stride-32bpp-rotate-180-hflip-async-flip
> +kms_big_fb@yf-tiled-max-hw-stride-64bpp-rotate-0-hflip
> +kms_big_fb@yf-tiled-max-hw-stride-64bpp-rotate-0-hflip-async-flip
> +kms_big_fb@yf-tiled-max-hw-stride-64bpp-rotate-180-hflip
> +kms_big_fb@yf-tiled-max-hw-stride-64bpp-rotate-180-hflip-async-flip
> +kms_big_fb@4-tiled-max-hw-stride-32bpp-rotate-0
> +kms_big_fb@4-tiled-max-hw-stride-32bpp-rotate-0-async-flip
> +kms_big_fb@4-tiled-max-hw-stride-32bpp-rotate-180
> +kms_big_fb@4-tiled-max-hw-stride-32bpp-rotate-180-async-flip
> +kms_big_fb@4-tiled-max-hw-stride-64bpp-rotate-0
> +kms_big_fb@4-tiled-max-hw-stride-64bpp-rotate-0-async-flip
> +kms_big_fb@4-tiled-max-hw-stride-64bpp-rotate-180
> +kms_big_fb@4-tiled-max-hw-stride-64bpp-rotate-180-async-flip
> +kms_big_fb@4-tiled-max-hw-stride-32bpp-rotate-0-hflip
> +kms_big_fb@4-tiled-max-hw-stride-32bpp-rotate-0-hflip-async-flip
> +kms_big_fb@4-tiled-max-hw-stride-32bpp-rotate-180-hflip
> +kms_big_fb@4-tiled-max-hw-stride-32bpp-rotate-180-hflip-async-flip
> +kms_big_fb@4-tiled-max-hw-stride-64bpp-rotate-0-hflip
> +kms_big_fb@4-tiled-max-hw-stride-64bpp-rotate-0-hflip-async-flip
> +kms_big_fb@4-tiled-max-hw-stride-64bpp-rotate-180-hflip
> +kms_big_fb@4-tiled-max-hw-stride-64bpp-rotate-180-hflip-async-flip
> +kms_big_joiner@basic
> +kms_big_joiner@invalid-modeset
> +kms_big_joiner@2x-modeset
> +kms_busy@basic
> +kms_busy@basic-hang
> +kms_busy@extended-pageflip-modeset-hang-oldfb
> +kms_busy@extended-pageflip-hang-oldfb
> +kms_busy@extended-pageflip-hang-newfb
> +kms_busy@extended-modeset-hang-oldfb
> +kms_busy@extended-modeset-hang-newfb
> +kms_busy@extended-modeset-hang-oldfb-with-reset
> +kms_busy@extended-modeset-hang-newfb-with-reset
> +kms_bw@linear-tiling-1-displays-1920x1080p
> +kms_bw@linear-tiling-1-displays-2560x1440p
> +kms_bw@linear-tiling-1-displays-3840x2160p
> +kms_bw@linear-tiling-2-displays-1920x1080p
> +kms_bw@linear-tiling-2-displays-2560x1440p
> +kms_bw@linear-tiling-2-displays-3840x2160p
> +kms_bw@linear-tiling-3-displays-1920x1080p
> +kms_bw@linear-tiling-3-displays-2560x1440p
> +kms_bw@linear-tiling-3-displays-3840x2160p
> +kms_bw@linear-tiling-4-displays-1920x1080p
> +kms_bw@linear-tiling-4-displays-2560x1440p
> +kms_bw@linear-tiling-4-displays-3840x2160p
> +kms_bw@linear-tiling-5-displays-1920x1080p
> +kms_bw@linear-tiling-5-displays-2560x1440p
> +kms_bw@linear-tiling-5-displays-3840x2160p
> +kms_bw@linear-tiling-6-displays-1920x1080p
> +kms_bw@linear-tiling-6-displays-2560x1440p
> +kms_bw@linear-tiling-6-displays-3840x2160p
> +kms_bw@linear-tiling-7-displays-1920x1080p
> +kms_bw@linear-tiling-7-displays-2560x1440p
> +kms_bw@linear-tiling-7-displays-3840x2160p
> +kms_bw@linear-tiling-8-displays-1920x1080p
> +kms_bw@linear-tiling-8-displays-2560x1440p
> +kms_bw@linear-tiling-8-displays-3840x2160p
> +kms_ccs@pipe-A-bad-pixel-format-y_tiled_ccs
> +kms_ccs@pipe-A-bad-pixel-format-yf_tiled_ccs
> +kms_ccs@pipe-A-bad-pixel-format-y_tiled_gen12_rc_ccs
> +kms_ccs@pipe-A-bad-pixel-format-y_tiled_gen12_rc_ccs_cc
> +kms_ccs@pipe-A-bad-pixel-format-y_tiled_gen12_mc_ccs
> +kms_ccs@pipe-A-bad-pixel-format-4_tiled_dg2_rc_ccs
> +kms_ccs@pipe-A-bad-pixel-format-4_tiled_dg2_mc_ccs
> +kms_ccs@pipe-A-bad-pixel-format-4_tiled_dg2_rc_ccs_cc
> +kms_ccs@pipe-A-bad-pixel-format-4_tiled_mtl_rc_ccs
> +kms_ccs@pipe-A-bad-pixel-format-4_tiled_mtl_mc_ccs
> +kms_ccs@pipe-A-bad-pixel-format-4_tiled_mtl_rc_ccs_cc
> +kms_ccs@pipe-A-bad-rotation-90-y_tiled_ccs
> +kms_ccs@pipe-A-bad-rotation-90-yf_tiled_ccs
> +kms_ccs@pipe-A-bad-rotation-90-y_tiled_gen12_rc_ccs
> +kms_ccs@pipe-A-bad-rotation-90-y_tiled_gen12_rc_ccs_cc
> +kms_ccs@pipe-A-bad-rotation-90-y_tiled_gen12_mc_ccs
> +kms_ccs@pipe-A-bad-rotation-90-4_tiled_dg2_rc_ccs
> +kms_ccs@pipe-A-bad-rotation-90-4_tiled_dg2_mc_ccs
> +kms_ccs@pipe-A-bad-rotation-90-4_tiled_dg2_rc_ccs_cc
> +kms_ccs@pipe-A-bad-rotation-90-4_tiled_mtl_rc_ccs
> +kms_ccs@pipe-A-bad-rotation-90-4_tiled_mtl_mc_ccs
> +kms_ccs@pipe-A-bad-rotation-90-4_tiled_mtl_rc_ccs_cc
> +kms_ccs@pipe-A-crc-primary-basic-y_tiled_ccs
> +kms_ccs@pipe-A-crc-primary-basic-yf_tiled_ccs
> +kms_ccs@pipe-A-crc-primary-basic-y_tiled_gen12_rc_ccs
> +kms_ccs@pipe-A-crc-primary-basic-y_tiled_gen12_rc_ccs_cc
> +kms_ccs@pipe-A-crc-primary-basic-y_tiled_gen12_mc_ccs
> +kms_ccs@pipe-A-crc-primary-basic-4_tiled_dg2_rc_ccs
> +kms_ccs@pipe-A-crc-primary-basic-4_tiled_dg2_mc_ccs
> +kms_ccs@pipe-A-crc-primary-basic-4_tiled_dg2_rc_ccs_cc
> +kms_ccs@pipe-A-crc-primary-basic-4_tiled_mtl_rc_ccs
> +kms_ccs@pipe-A-crc-primary-basic-4_tiled_mtl_mc_ccs
> +kms_ccs@pipe-A-crc-primary-basic-4_tiled_mtl_rc_ccs_cc
> +kms_ccs@pipe-A-crc-primary-rotation-180-y_tiled_ccs
> +kms_ccs@pipe-A-crc-primary-rotation-180-yf_tiled_ccs
> +kms_ccs@pipe-A-crc-primary-rotation-180-y_tiled_gen12_rc_ccs
> +kms_ccs@pipe-A-crc-primary-rotation-180-y_tiled_gen12_rc_ccs_cc
> +kms_ccs@pipe-A-crc-primary-rotation-180-y_tiled_gen12_mc_ccs
> +kms_ccs@pipe-A-crc-primary-rotation-180-4_tiled_dg2_rc_ccs
> +kms_ccs@pipe-A-crc-primary-rotation-180-4_tiled_dg2_mc_ccs
> +kms_ccs@pipe-A-crc-primary-rotation-180-4_tiled_dg2_rc_ccs_cc
> +kms_ccs@pipe-A-crc-primary-rotation-180-4_tiled_mtl_rc_ccs
> +kms_ccs@pipe-A-crc-primary-rotation-180-4_tiled_mtl_mc_ccs
> +kms_ccs@pipe-A-crc-primary-rotation-180-4_tiled_mtl_rc_ccs_cc
> +kms_ccs@pipe-A-random-ccs-data-y_tiled_ccs
> +kms_ccs@pipe-A-random-ccs-data-yf_tiled_ccs
> +kms_ccs@pipe-A-random-ccs-data-y_tiled_gen12_rc_ccs
> +kms_ccs@pipe-A-random-ccs-data-y_tiled_gen12_rc_ccs_cc
> +kms_ccs@pipe-A-random-ccs-data-y_tiled_gen12_mc_ccs
> +kms_ccs@pipe-A-random-ccs-data-4_tiled_dg2_rc_ccs
> +kms_ccs@pipe-A-random-ccs-data-4_tiled_dg2_mc_ccs
> +kms_ccs@pipe-A-random-ccs-data-4_tiled_dg2_rc_ccs_cc
> +kms_ccs@pipe-A-random-ccs-data-4_tiled_mtl_rc_ccs
> +kms_ccs@pipe-A-random-ccs-data-4_tiled_mtl_mc_ccs
> +kms_ccs@pipe-A-random-ccs-data-4_tiled_mtl_rc_ccs_cc
> +kms_ccs@pipe-A-missing-ccs-buffer-y_tiled_ccs
> +kms_ccs@pipe-A-missing-ccs-buffer-yf_tiled_ccs
> +kms_ccs@pipe-A-missing-ccs-buffer-y_tiled_gen12_rc_ccs
> +kms_ccs@pipe-A-missing-ccs-buffer-y_tiled_gen12_rc_ccs_cc
> +kms_ccs@pipe-A-missing-ccs-buffer-y_tiled_gen12_mc_ccs
> +kms_ccs@pipe-A-missing-ccs-buffer-4_tiled_mtl_rc_ccs
> +kms_ccs@pipe-A-missing-ccs-buffer-4_tiled_mtl_mc_ccs
> +kms_ccs@pipe-A-missing-ccs-buffer-4_tiled_mtl_rc_ccs_cc
> +kms_ccs@pipe-A-ccs-on-another-bo-y_tiled_ccs
> +kms_ccs@pipe-A-ccs-on-another-bo-yf_tiled_ccs
> +kms_ccs@pipe-A-ccs-on-another-bo-y_tiled_gen12_rc_ccs
> +kms_ccs@pipe-A-ccs-on-another-bo-y_tiled_gen12_rc_ccs_cc
> +kms_ccs@pipe-A-ccs-on-another-bo-y_tiled_gen12_mc_ccs
> +kms_ccs@pipe-A-ccs-on-another-bo-4_tiled_mtl_rc_ccs
> +kms_ccs@pipe-A-ccs-on-another-bo-4_tiled_mtl_mc_ccs
> +kms_ccs@pipe-A-ccs-on-another-bo-4_tiled_mtl_rc_ccs_cc
> +kms_ccs@pipe-A-bad-aux-stride-y_tiled_ccs
> +kms_ccs@pipe-A-bad-aux-stride-yf_tiled_ccs
> +kms_ccs@pipe-A-bad-aux-stride-y_tiled_gen12_rc_ccs
> +kms_ccs@pipe-A-bad-aux-stride-y_tiled_gen12_rc_ccs_cc
> +kms_ccs@pipe-A-bad-aux-stride-y_tiled_gen12_mc_ccs
> +kms_ccs@pipe-A-bad-aux-stride-4_tiled_mtl_rc_ccs
> +kms_ccs@pipe-A-bad-aux-stride-4_tiled_mtl_mc_ccs
> +kms_ccs@pipe-A-bad-aux-stride-4_tiled_mtl_rc_ccs_cc
> +kms_ccs@pipe-A-crc-sprite-planes-basic-y_tiled_ccs
> +kms_ccs@pipe-A-crc-sprite-planes-basic-yf_tiled_ccs
> +kms_ccs@pipe-A-crc-sprite-planes-basic-y_tiled_gen12_rc_ccs
> +kms_ccs@pipe-A-crc-sprite-planes-basic-y_tiled_gen12_rc_ccs_cc
> +kms_ccs@pipe-A-crc-sprite-planes-basic-y_tiled_gen12_mc_ccs
> +kms_ccs@pipe-A-crc-sprite-planes-basic-4_tiled_dg2_rc_ccs
> +kms_ccs@pipe-A-crc-sprite-planes-basic-4_tiled_dg2_mc_ccs
> +kms_ccs@pipe-A-crc-sprite-planes-basic-4_tiled_dg2_rc_ccs_cc
> +kms_ccs@pipe-A-crc-sprite-planes-basic-4_tiled_mtl_rc_ccs
> +kms_ccs@pipe-A-crc-sprite-planes-basic-4_tiled_mtl_mc_ccs
> +kms_ccs@pipe-A-crc-sprite-planes-basic-4_tiled_mtl_rc_ccs_cc
> +kms_ccs@pipe-B-bad-pixel-format-y_tiled_ccs
> +kms_ccs@pipe-B-bad-pixel-format-yf_tiled_ccs
> +kms_ccs@pipe-B-bad-pixel-format-y_tiled_gen12_rc_ccs
> +kms_ccs@pipe-B-bad-pixel-format-y_tiled_gen12_rc_ccs_cc
> +kms_ccs@pipe-B-bad-pixel-format-y_tiled_gen12_mc_ccs
> +kms_ccs@pipe-B-bad-pixel-format-4_tiled_dg2_rc_ccs
> +kms_ccs@pipe-B-bad-pixel-format-4_tiled_dg2_mc_ccs
> +kms_ccs@pipe-B-bad-pixel-format-4_tiled_dg2_rc_ccs_cc
> +kms_ccs@pipe-B-bad-pixel-format-4_tiled_mtl_rc_ccs
> +kms_ccs@pipe-B-bad-pixel-format-4_tiled_mtl_mc_ccs
> +kms_ccs@pipe-B-bad-pixel-format-4_tiled_mtl_rc_ccs_cc
> +kms_ccs@pipe-B-bad-rotation-90-y_tiled_ccs
> +kms_ccs@pipe-B-bad-rotation-90-yf_tiled_ccs
> +kms_ccs@pipe-B-bad-rotation-90-y_tiled_gen12_rc_ccs
> +kms_ccs@pipe-B-bad-rotation-90-y_tiled_gen12_rc_ccs_cc
> +kms_ccs@pipe-B-bad-rotation-90-y_tiled_gen12_mc_ccs
> +kms_ccs@pipe-B-bad-rotation-90-4_tiled_dg2_rc_ccs
> +kms_ccs@pipe-B-bad-rotation-90-4_tiled_dg2_mc_ccs
> +kms_ccs@pipe-B-bad-rotation-90-4_tiled_dg2_rc_ccs_cc
> +kms_ccs@pipe-B-bad-rotation-90-4_tiled_mtl_rc_ccs
> +kms_ccs@pipe-B-bad-rotation-90-4_tiled_mtl_mc_ccs
> +kms_ccs@pipe-B-bad-rotation-90-4_tiled_mtl_rc_ccs_cc
> +kms_ccs@pipe-B-crc-primary-basic-y_tiled_ccs
> +kms_ccs@pipe-B-crc-primary-basic-yf_tiled_ccs
> +kms_ccs@pipe-B-crc-primary-basic-y_tiled_gen12_rc_ccs
> +kms_ccs@pipe-B-crc-primary-basic-y_tiled_gen12_rc_ccs_cc
> +kms_ccs@pipe-B-crc-primary-basic-y_tiled_gen12_mc_ccs
> +kms_ccs@pipe-B-crc-primary-basic-4_tiled_dg2_rc_ccs
> +kms_ccs@pipe-B-crc-primary-basic-4_tiled_dg2_mc_ccs
> +kms_ccs@pipe-B-crc-primary-basic-4_tiled_dg2_rc_ccs_cc
> +kms_ccs@pipe-B-crc-primary-basic-4_tiled_mtl_rc_ccs
> +kms_ccs@pipe-B-crc-primary-basic-4_tiled_mtl_mc_ccs
> +kms_ccs@pipe-B-crc-primary-basic-4_tiled_mtl_rc_ccs_cc
> +kms_ccs@pipe-B-crc-primary-rotation-180-y_tiled_ccs
> +kms_ccs@pipe-B-crc-primary-rotation-180-yf_tiled_ccs
> +kms_ccs@pipe-B-crc-primary-rotation-180-y_tiled_gen12_rc_ccs
> +kms_ccs@pipe-B-crc-primary-rotation-180-y_tiled_gen12_rc_ccs_cc
> +kms_ccs@pipe-B-crc-primary-rotation-180-y_tiled_gen12_mc_ccs
> +kms_ccs@pipe-B-crc-primary-rotation-180-4_tiled_dg2_rc_ccs
> +kms_ccs@pipe-B-crc-primary-rotation-180-4_tiled_dg2_mc_ccs
> +kms_ccs@pipe-B-crc-primary-rotation-180-4_tiled_dg2_rc_ccs_cc
> +kms_ccs@pipe-B-crc-primary-rotation-180-4_tiled_mtl_rc_ccs
> +kms_ccs@pipe-B-crc-primary-rotation-180-4_tiled_mtl_mc_ccs
> +kms_ccs@pipe-B-crc-primary-rotation-180-4_tiled_mtl_rc_ccs_cc
> +kms_ccs@pipe-B-random-ccs-data-y_tiled_ccs
> +kms_ccs@pipe-B-random-ccs-data-yf_tiled_ccs
> +kms_ccs@pipe-B-random-ccs-data-y_tiled_gen12_rc_ccs
> +kms_ccs@pipe-B-random-ccs-data-y_tiled_gen12_rc_ccs_cc
> +kms_ccs@pipe-B-random-ccs-data-y_tiled_gen12_mc_ccs
> +kms_ccs@pipe-B-random-ccs-data-4_tiled_dg2_rc_ccs
> +kms_ccs@pipe-B-random-ccs-data-4_tiled_dg2_mc_ccs
> +kms_ccs@pipe-B-random-ccs-data-4_tiled_dg2_rc_ccs_cc
> +kms_ccs@pipe-B-random-ccs-data-4_tiled_mtl_rc_ccs
> +kms_ccs@pipe-B-random-ccs-data-4_tiled_mtl_mc_ccs
> +kms_ccs@pipe-B-random-ccs-data-4_tiled_mtl_rc_ccs_cc
> +kms_ccs@pipe-B-missing-ccs-buffer-y_tiled_ccs
> +kms_ccs@pipe-B-missing-ccs-buffer-yf_tiled_ccs
> +kms_ccs@pipe-B-missing-ccs-buffer-y_tiled_gen12_rc_ccs
> +kms_ccs@pipe-B-missing-ccs-buffer-y_tiled_gen12_rc_ccs_cc
> +kms_ccs@pipe-B-missing-ccs-buffer-y_tiled_gen12_mc_ccs
> +kms_ccs@pipe-B-missing-ccs-buffer-4_tiled_mtl_rc_ccs
> +kms_ccs@pipe-B-missing-ccs-buffer-4_tiled_mtl_mc_ccs
> +kms_ccs@pipe-B-missing-ccs-buffer-4_tiled_mtl_rc_ccs_cc
> +kms_ccs@pipe-B-ccs-on-another-bo-y_tiled_ccs
> +kms_ccs@pipe-B-ccs-on-another-bo-yf_tiled_ccs
> +kms_ccs@pipe-B-ccs-on-another-bo-y_tiled_gen12_rc_ccs
> +kms_ccs@pipe-B-ccs-on-another-bo-y_tiled_gen12_rc_ccs_cc
> +kms_ccs@pipe-B-ccs-on-another-bo-y_tiled_gen12_mc_ccs
> +kms_ccs@pipe-B-ccs-on-another-bo-4_tiled_mtl_rc_ccs
> +kms_ccs@pipe-B-ccs-on-another-bo-4_tiled_mtl_mc_ccs
> +kms_ccs@pipe-B-ccs-on-another-bo-4_tiled_mtl_rc_ccs_cc
> +kms_ccs@pipe-B-bad-aux-stride-y_tiled_ccs
> +kms_ccs@pipe-B-bad-aux-stride-yf_tiled_ccs
> +kms_ccs@pipe-B-bad-aux-stride-y_tiled_gen12_rc_ccs
> +kms_ccs@pipe-B-bad-aux-stride-y_tiled_gen12_rc_ccs_cc
> +kms_ccs@pipe-B-bad-aux-stride-y_tiled_gen12_mc_ccs
> +kms_ccs@pipe-B-bad-aux-stride-4_tiled_mtl_rc_ccs
> +kms_ccs@pipe-B-bad-aux-stride-4_tiled_mtl_mc_ccs
> +kms_ccs@pipe-B-bad-aux-stride-4_tiled_mtl_rc_ccs_cc
> +kms_ccs@pipe-B-crc-sprite-planes-basic-y_tiled_ccs
> +kms_ccs@pipe-B-crc-sprite-planes-basic-yf_tiled_ccs
> +kms_ccs@pipe-B-crc-sprite-planes-basic-y_tiled_gen12_rc_ccs
> +kms_ccs@pipe-B-crc-sprite-planes-basic-y_tiled_gen12_rc_ccs_cc
> +kms_ccs@pipe-B-crc-sprite-planes-basic-y_tiled_gen12_mc_ccs
> +kms_ccs@pipe-B-crc-sprite-planes-basic-4_tiled_dg2_rc_ccs
> +kms_ccs@pipe-B-crc-sprite-planes-basic-4_tiled_dg2_mc_ccs
> +kms_ccs@pipe-B-crc-sprite-planes-basic-4_tiled_dg2_rc_ccs_cc
> +kms_ccs@pipe-B-crc-sprite-planes-basic-4_tiled_mtl_rc_ccs
> +kms_ccs@pipe-B-crc-sprite-planes-basic-4_tiled_mtl_mc_ccs
> +kms_ccs@pipe-B-crc-sprite-planes-basic-4_tiled_mtl_rc_ccs_cc
> +kms_ccs@pipe-C-bad-pixel-format-y_tiled_ccs
> +kms_ccs@pipe-C-bad-pixel-format-yf_tiled_ccs
> +kms_ccs@pipe-C-bad-pixel-format-y_tiled_gen12_rc_ccs
> +kms_ccs@pipe-C-bad-pixel-format-y_tiled_gen12_rc_ccs_cc
> +kms_ccs@pipe-C-bad-pixel-format-y_tiled_gen12_mc_ccs
> +kms_ccs@pipe-C-bad-pixel-format-4_tiled_dg2_rc_ccs
> +kms_ccs@pipe-C-bad-pixel-format-4_tiled_dg2_mc_ccs
> +kms_ccs@pipe-C-bad-pixel-format-4_tiled_dg2_rc_ccs_cc
> +kms_ccs@pipe-C-bad-pixel-format-4_tiled_mtl_rc_ccs
> +kms_ccs@pipe-C-bad-pixel-format-4_tiled_mtl_mc_ccs
> +kms_ccs@pipe-C-bad-pixel-format-4_tiled_mtl_rc_ccs_cc
> +kms_ccs@pipe-C-bad-rotation-90-y_tiled_ccs
> +kms_ccs@pipe-C-bad-rotation-90-yf_tiled_ccs
> +kms_ccs@pipe-C-bad-rotation-90-y_tiled_gen12_rc_ccs
> +kms_ccs@pipe-C-bad-rotation-90-y_tiled_gen12_rc_ccs_cc
> +kms_ccs@pipe-C-bad-rotation-90-y_tiled_gen12_mc_ccs
> +kms_ccs@pipe-C-bad-rotation-90-4_tiled_dg2_rc_ccs
> +kms_ccs@pipe-C-bad-rotation-90-4_tiled_dg2_mc_ccs
> +kms_ccs@pipe-C-bad-rotation-90-4_tiled_dg2_rc_ccs_cc
> +kms_ccs@pipe-C-bad-rotation-90-4_tiled_mtl_rc_ccs
> +kms_ccs@pipe-C-bad-rotation-90-4_tiled_mtl_mc_ccs
> +kms_ccs@pipe-C-bad-rotation-90-4_tiled_mtl_rc_ccs_cc
> +kms_ccs@pipe-C-crc-primary-basic-y_tiled_ccs
> +kms_ccs@pipe-C-crc-primary-basic-yf_tiled_ccs
> +kms_ccs@pipe-C-crc-primary-basic-y_tiled_gen12_rc_ccs
> +kms_ccs@pipe-C-crc-primary-basic-y_tiled_gen12_rc_ccs_cc
> +kms_ccs@pipe-C-crc-primary-basic-y_tiled_gen12_mc_ccs
> +kms_ccs@pipe-C-crc-primary-basic-4_tiled_dg2_rc_ccs
> +kms_ccs@pipe-C-crc-primary-basic-4_tiled_dg2_mc_ccs
> +kms_ccs@pipe-C-crc-primary-basic-4_tiled_dg2_rc_ccs_cc
> +kms_ccs@pipe-C-crc-primary-basic-4_tiled_mtl_rc_ccs
> +kms_ccs@pipe-C-crc-primary-basic-4_tiled_mtl_mc_ccs
> +kms_ccs@pipe-C-crc-primary-basic-4_tiled_mtl_rc_ccs_cc
> +kms_ccs@pipe-C-crc-primary-rotation-180-y_tiled_ccs
> +kms_ccs@pipe-C-crc-primary-rotation-180-yf_tiled_ccs
> +kms_ccs@pipe-C-crc-primary-rotation-180-y_tiled_gen12_rc_ccs
> +kms_ccs@pipe-C-crc-primary-rotation-180-y_tiled_gen12_rc_ccs_cc
> +kms_ccs@pipe-C-crc-primary-rotation-180-y_tiled_gen12_mc_ccs
> +kms_ccs@pipe-C-crc-primary-rotation-180-4_tiled_dg2_rc_ccs
> +kms_ccs@pipe-C-crc-primary-rotation-180-4_tiled_dg2_mc_ccs
> +kms_ccs@pipe-C-crc-primary-rotation-180-4_tiled_dg2_rc_ccs_cc
> +kms_ccs@pipe-C-crc-primary-rotation-180-4_tiled_mtl_rc_ccs
> +kms_ccs@pipe-C-crc-primary-rotation-180-4_tiled_mtl_mc_ccs
> +kms_ccs@pipe-C-crc-primary-rotation-180-4_tiled_mtl_rc_ccs_cc
> +kms_ccs@pipe-C-random-ccs-data-y_tiled_ccs
> +kms_ccs@pipe-C-random-ccs-data-yf_tiled_ccs
> +kms_ccs@pipe-C-random-ccs-data-y_tiled_gen12_rc_ccs
> +kms_ccs@pipe-C-random-ccs-data-y_tiled_gen12_rc_ccs_cc
> +kms_ccs@pipe-C-random-ccs-data-y_tiled_gen12_mc_ccs
> +kms_ccs@pipe-C-random-ccs-data-4_tiled_dg2_rc_ccs
> +kms_ccs@pipe-C-random-ccs-data-4_tiled_dg2_mc_ccs
> +kms_ccs@pipe-C-random-ccs-data-4_tiled_dg2_rc_ccs_cc
> +kms_ccs@pipe-C-random-ccs-data-4_tiled_mtl_rc_ccs
> +kms_ccs@pipe-C-random-ccs-data-4_tiled_mtl_mc_ccs
> +kms_ccs@pipe-C-random-ccs-data-4_tiled_mtl_rc_ccs_cc
> +kms_ccs@pipe-C-missing-ccs-buffer-y_tiled_ccs
> +kms_ccs@pipe-C-missing-ccs-buffer-yf_tiled_ccs
> +kms_ccs@pipe-C-missing-ccs-buffer-y_tiled_gen12_rc_ccs
> +kms_ccs@pipe-C-missing-ccs-buffer-y_tiled_gen12_rc_ccs_cc
> +kms_ccs@pipe-C-missing-ccs-buffer-y_tiled_gen12_mc_ccs
> +kms_ccs@pipe-C-missing-ccs-buffer-4_tiled_mtl_rc_ccs
> +kms_ccs@pipe-C-missing-ccs-buffer-4_tiled_mtl_mc_ccs
> +kms_ccs@pipe-C-missing-ccs-buffer-4_tiled_mtl_rc_ccs_cc
> +kms_ccs@pipe-C-ccs-on-another-bo-y_tiled_ccs
> +kms_ccs@pipe-C-ccs-on-another-bo-yf_tiled_ccs
> +kms_ccs@pipe-C-ccs-on-another-bo-y_tiled_gen12_rc_ccs
> +kms_ccs@pipe-C-ccs-on-another-bo-y_tiled_gen12_rc_ccs_cc
> +kms_ccs@pipe-C-ccs-on-another-bo-y_tiled_gen12_mc_ccs
> +kms_ccs@pipe-C-ccs-on-another-bo-4_tiled_mtl_rc_ccs
> +kms_ccs@pipe-C-ccs-on-another-bo-4_tiled_mtl_mc_ccs
> +kms_ccs@pipe-C-ccs-on-another-bo-4_tiled_mtl_rc_ccs_cc
> +kms_ccs@pipe-C-bad-aux-stride-y_tiled_ccs
> +kms_ccs@pipe-C-bad-aux-stride-yf_tiled_ccs
> +kms_ccs@pipe-C-bad-aux-stride-y_tiled_gen12_rc_ccs
> +kms_ccs@pipe-C-bad-aux-stride-y_tiled_gen12_rc_ccs_cc
> +kms_ccs@pipe-C-bad-aux-stride-y_tiled_gen12_mc_ccs
> +kms_ccs@pipe-C-bad-aux-stride-4_tiled_mtl_rc_ccs
> +kms_ccs@pipe-C-bad-aux-stride-4_tiled_mtl_mc_ccs
> +kms_ccs@pipe-C-bad-aux-stride-4_tiled_mtl_rc_ccs_cc
> +kms_ccs@pipe-C-crc-sprite-planes-basic-y_tiled_ccs
> +kms_ccs@pipe-C-crc-sprite-planes-basic-yf_tiled_ccs
> +kms_ccs@pipe-C-crc-sprite-planes-basic-y_tiled_gen12_rc_ccs
> +kms_ccs@pipe-C-crc-sprite-planes-basic-y_tiled_gen12_rc_ccs_cc
> +kms_ccs@pipe-C-crc-sprite-planes-basic-y_tiled_gen12_mc_ccs
> +kms_ccs@pipe-C-crc-sprite-planes-basic-4_tiled_dg2_rc_ccs
> +kms_ccs@pipe-C-crc-sprite-planes-basic-4_tiled_dg2_mc_ccs
> +kms_ccs@pipe-C-crc-sprite-planes-basic-4_tiled_dg2_rc_ccs_cc
> +kms_ccs@pipe-C-crc-sprite-planes-basic-4_tiled_mtl_rc_ccs
> +kms_ccs@pipe-C-crc-sprite-planes-basic-4_tiled_mtl_mc_ccs
> +kms_ccs@pipe-C-crc-sprite-planes-basic-4_tiled_mtl_rc_ccs_cc
> +kms_ccs@pipe-D-bad-pixel-format-y_tiled_ccs
> +kms_ccs@pipe-D-bad-pixel-format-yf_tiled_ccs
> +kms_ccs@pipe-D-bad-pixel-format-y_tiled_gen12_rc_ccs
> +kms_ccs@pipe-D-bad-pixel-format-y_tiled_gen12_rc_ccs_cc
> +kms_ccs@pipe-D-bad-pixel-format-y_tiled_gen12_mc_ccs
> +kms_ccs@pipe-D-bad-pixel-format-4_tiled_dg2_rc_ccs
> +kms_ccs@pipe-D-bad-pixel-format-4_tiled_dg2_mc_ccs
> +kms_ccs@pipe-D-bad-pixel-format-4_tiled_dg2_rc_ccs_cc
> +kms_ccs@pipe-D-bad-pixel-format-4_tiled_mtl_rc_ccs
> +kms_ccs@pipe-D-bad-pixel-format-4_tiled_mtl_mc_ccs
> +kms_ccs@pipe-D-bad-pixel-format-4_tiled_mtl_rc_ccs_cc
> +kms_ccs@pipe-D-bad-rotation-90-y_tiled_ccs
> +kms_ccs@pipe-D-bad-rotation-90-yf_tiled_ccs
> +kms_ccs@pipe-D-bad-rotation-90-y_tiled_gen12_rc_ccs
> +kms_ccs@pipe-D-bad-rotation-90-y_tiled_gen12_rc_ccs_cc
> +kms_ccs@pipe-D-bad-rotation-90-y_tiled_gen12_mc_ccs
> +kms_ccs@pipe-D-bad-rotation-90-4_tiled_dg2_rc_ccs
> +kms_ccs@pipe-D-bad-rotation-90-4_tiled_dg2_mc_ccs
> +kms_ccs@pipe-D-bad-rotation-90-4_tiled_dg2_rc_ccs_cc
> +kms_ccs@pipe-D-bad-rotation-90-4_tiled_mtl_rc_ccs
> +kms_ccs@pipe-D-bad-rotation-90-4_tiled_mtl_mc_ccs
> +kms_ccs@pipe-D-bad-rotation-90-4_tiled_mtl_rc_ccs_cc
> +kms_ccs@pipe-D-crc-primary-basic-y_tiled_ccs
> +kms_ccs@pipe-D-crc-primary-basic-yf_tiled_ccs
> +kms_ccs@pipe-D-crc-primary-basic-y_tiled_gen12_rc_ccs
> +kms_ccs@pipe-D-crc-primary-basic-y_tiled_gen12_rc_ccs_cc
> +kms_ccs@pipe-D-crc-primary-basic-y_tiled_gen12_mc_ccs
> +kms_ccs@pipe-D-crc-primary-basic-4_tiled_dg2_rc_ccs
> +kms_ccs@pipe-D-crc-primary-basic-4_tiled_dg2_mc_ccs
> +kms_ccs@pipe-D-crc-primary-basic-4_tiled_dg2_rc_ccs_cc
> +kms_ccs@pipe-D-crc-primary-basic-4_tiled_mtl_rc_ccs
> +kms_ccs@pipe-D-crc-primary-basic-4_tiled_mtl_mc_ccs
> +kms_ccs@pipe-D-crc-primary-basic-4_tiled_mtl_rc_ccs_cc
> +kms_ccs@pipe-D-crc-primary-rotation-180-y_tiled_ccs
> +kms_ccs@pipe-D-crc-primary-rotation-180-yf_tiled_ccs
> +kms_ccs@pipe-D-crc-primary-rotation-180-y_tiled_gen12_rc_ccs
> +kms_ccs@pipe-D-crc-primary-rotation-180-y_tiled_gen12_rc_ccs_cc
> +kms_ccs@pipe-D-crc-primary-rotation-180-y_tiled_gen12_mc_ccs
> +kms_ccs@pipe-D-crc-primary-rotation-180-4_tiled_dg2_rc_ccs
> +kms_ccs@pipe-D-crc-primary-rotation-180-4_tiled_dg2_mc_ccs
> +kms_ccs@pipe-D-crc-primary-rotation-180-4_tiled_dg2_rc_ccs_cc
> +kms_ccs@pipe-D-crc-primary-rotation-180-4_tiled_mtl_rc_ccs
> +kms_ccs@pipe-D-crc-primary-rotation-180-4_tiled_mtl_mc_ccs
> +kms_ccs@pipe-D-crc-primary-rotation-180-4_tiled_mtl_rc_ccs_cc
> +kms_ccs@pipe-D-random-ccs-data-y_tiled_ccs
> +kms_ccs@pipe-D-random-ccs-data-yf_tiled_ccs
> +kms_ccs@pipe-D-random-ccs-data-y_tiled_gen12_rc_ccs
> +kms_ccs@pipe-D-random-ccs-data-y_tiled_gen12_rc_ccs_cc
> +kms_ccs@pipe-D-random-ccs-data-y_tiled_gen12_mc_ccs
> +kms_ccs@pipe-D-random-ccs-data-4_tiled_dg2_rc_ccs
> +kms_ccs@pipe-D-random-ccs-data-4_tiled_dg2_mc_ccs
> +kms_ccs@pipe-D-random-ccs-data-4_tiled_dg2_rc_ccs_cc
> +kms_ccs@pipe-D-random-ccs-data-4_tiled_mtl_rc_ccs
> +kms_ccs@pipe-D-random-ccs-data-4_tiled_mtl_mc_ccs
> +kms_ccs@pipe-D-random-ccs-data-4_tiled_mtl_rc_ccs_cc
> +kms_ccs@pipe-D-missing-ccs-buffer-y_tiled_ccs
> +kms_ccs@pipe-D-missing-ccs-buffer-yf_tiled_ccs
> +kms_ccs@pipe-D-missing-ccs-buffer-y_tiled_gen12_rc_ccs
> +kms_ccs@pipe-D-missing-ccs-buffer-y_tiled_gen12_rc_ccs_cc
> +kms_ccs@pipe-D-missing-ccs-buffer-y_tiled_gen12_mc_ccs
> +kms_ccs@pipe-D-missing-ccs-buffer-4_tiled_mtl_rc_ccs
> +kms_ccs@pipe-D-missing-ccs-buffer-4_tiled_mtl_mc_ccs
> +kms_ccs@pipe-D-missing-ccs-buffer-4_tiled_mtl_rc_ccs_cc
> +kms_ccs@pipe-D-ccs-on-another-bo-y_tiled_ccs
> +kms_ccs@pipe-D-ccs-on-another-bo-yf_tiled_ccs
> +kms_ccs@pipe-D-ccs-on-another-bo-y_tiled_gen12_rc_ccs
> +kms_ccs@pipe-D-ccs-on-another-bo-y_tiled_gen12_rc_ccs_cc
> +kms_ccs@pipe-D-ccs-on-another-bo-y_tiled_gen12_mc_ccs
> +kms_ccs@pipe-D-ccs-on-another-bo-4_tiled_mtl_rc_ccs
> +kms_ccs@pipe-D-ccs-on-another-bo-4_tiled_mtl_mc_ccs
> +kms_ccs@pipe-D-ccs-on-another-bo-4_tiled_mtl_rc_ccs_cc
> +kms_ccs@pipe-D-bad-aux-stride-y_tiled_ccs
> +kms_ccs@pipe-D-bad-aux-stride-yf_tiled_ccs
> +kms_ccs@pipe-D-bad-aux-stride-y_tiled_gen12_rc_ccs
> +kms_ccs@pipe-D-bad-aux-stride-y_tiled_gen12_rc_ccs_cc
> +kms_ccs@pipe-D-bad-aux-stride-y_tiled_gen12_mc_ccs
> +kms_ccs@pipe-D-bad-aux-stride-4_tiled_mtl_rc_ccs
> +kms_ccs@pipe-D-bad-aux-stride-4_tiled_mtl_mc_ccs
> +kms_ccs@pipe-D-bad-aux-stride-4_tiled_mtl_rc_ccs_cc
> +kms_ccs@pipe-D-crc-sprite-planes-basic-y_tiled_ccs
> +kms_ccs@pipe-D-crc-sprite-planes-basic-yf_tiled_ccs
> +kms_ccs@pipe-D-crc-sprite-planes-basic-y_tiled_gen12_rc_ccs
> +kms_ccs@pipe-D-crc-sprite-planes-basic-y_tiled_gen12_rc_ccs_cc
> +kms_ccs@pipe-D-crc-sprite-planes-basic-y_tiled_gen12_mc_ccs
> +kms_ccs@pipe-D-crc-sprite-planes-basic-4_tiled_dg2_rc_ccs
> +kms_ccs@pipe-D-crc-sprite-planes-basic-4_tiled_dg2_mc_ccs
> +kms_ccs@pipe-D-crc-sprite-planes-basic-4_tiled_dg2_rc_ccs_cc
> +kms_ccs@pipe-D-crc-sprite-planes-basic-4_tiled_mtl_rc_ccs
> +kms_ccs@pipe-D-crc-sprite-planes-basic-4_tiled_mtl_mc_ccs
> +kms_ccs@pipe-D-crc-sprite-planes-basic-4_tiled_mtl_rc_ccs_cc
> +kms_ccs@pipe-E-bad-pixel-format-y_tiled_ccs
> +kms_ccs@pipe-E-bad-pixel-format-yf_tiled_ccs
> +kms_ccs@pipe-E-bad-pixel-format-y_tiled_gen12_rc_ccs
> +kms_ccs@pipe-E-bad-pixel-format-y_tiled_gen12_rc_ccs_cc
> +kms_ccs@pipe-E-bad-pixel-format-y_tiled_gen12_mc_ccs
> +kms_ccs@pipe-E-bad-pixel-format-4_tiled_dg2_rc_ccs
> +kms_ccs@pipe-E-bad-pixel-format-4_tiled_dg2_mc_ccs
> +kms_ccs@pipe-E-bad-pixel-format-4_tiled_dg2_rc_ccs_cc
> +kms_ccs@pipe-E-bad-pixel-format-4_tiled_mtl_rc_ccs
> +kms_ccs@pipe-E-bad-pixel-format-4_tiled_mtl_mc_ccs
> +kms_ccs@pipe-E-bad-pixel-format-4_tiled_mtl_rc_ccs_cc
> +kms_ccs@pipe-E-bad-rotation-90-y_tiled_ccs
> +kms_ccs@pipe-E-bad-rotation-90-yf_tiled_ccs
> +kms_ccs@pipe-E-bad-rotation-90-y_tiled_gen12_rc_ccs
> +kms_ccs@pipe-E-bad-rotation-90-y_tiled_gen12_rc_ccs_cc
> +kms_ccs@pipe-E-bad-rotation-90-y_tiled_gen12_mc_ccs
> +kms_ccs@pipe-E-bad-rotation-90-4_tiled_dg2_rc_ccs
> +kms_ccs@pipe-E-bad-rotation-90-4_tiled_dg2_mc_ccs
> +kms_ccs@pipe-E-bad-rotation-90-4_tiled_dg2_rc_ccs_cc
> +kms_ccs@pipe-E-bad-rotation-90-4_tiled_mtl_rc_ccs
> +kms_ccs@pipe-E-bad-rotation-90-4_tiled_mtl_mc_ccs
> +kms_ccs@pipe-E-bad-rotation-90-4_tiled_mtl_rc_ccs_cc
> +kms_ccs@pipe-E-crc-primary-basic-y_tiled_ccs
> +kms_ccs@pipe-E-crc-primary-basic-yf_tiled_ccs
> +kms_ccs@pipe-E-crc-primary-basic-y_tiled_gen12_rc_ccs
> +kms_ccs@pipe-E-crc-primary-basic-y_tiled_gen12_rc_ccs_cc
> +kms_ccs@pipe-E-crc-primary-basic-y_tiled_gen12_mc_ccs
> +kms_ccs@pipe-E-crc-primary-basic-4_tiled_dg2_rc_ccs
> +kms_ccs@pipe-E-crc-primary-basic-4_tiled_dg2_mc_ccs
> +kms_ccs@pipe-E-crc-primary-basic-4_tiled_dg2_rc_ccs_cc
> +kms_ccs@pipe-E-crc-primary-basic-4_tiled_mtl_rc_ccs
> +kms_ccs@pipe-E-crc-primary-basic-4_tiled_mtl_mc_ccs
> +kms_ccs@pipe-E-crc-primary-basic-4_tiled_mtl_rc_ccs_cc
> +kms_ccs@pipe-E-crc-primary-rotation-180-y_tiled_ccs
> +kms_ccs@pipe-E-crc-primary-rotation-180-yf_tiled_ccs
> +kms_ccs@pipe-E-crc-primary-rotation-180-y_tiled_gen12_rc_ccs
> +kms_ccs@pipe-E-crc-primary-rotation-180-y_tiled_gen12_rc_ccs_cc
> +kms_ccs@pipe-E-crc-primary-rotation-180-y_tiled_gen12_mc_ccs
> +kms_ccs@pipe-E-crc-primary-rotation-180-4_tiled_dg2_rc_ccs
> +kms_ccs@pipe-E-crc-primary-rotation-180-4_tiled_dg2_mc_ccs
> +kms_ccs@pipe-E-crc-primary-rotation-180-4_tiled_dg2_rc_ccs_cc
> +kms_ccs@pipe-E-crc-primary-rotation-180-4_tiled_mtl_rc_ccs
> +kms_ccs@pipe-E-crc-primary-rotation-180-4_tiled_mtl_mc_ccs
> +kms_ccs@pipe-E-crc-primary-rotation-180-4_tiled_mtl_rc_ccs_cc
> +kms_ccs@pipe-E-random-ccs-data-y_tiled_ccs
> +kms_ccs@pipe-E-random-ccs-data-yf_tiled_ccs
> +kms_ccs@pipe-E-random-ccs-data-y_tiled_gen12_rc_ccs
> +kms_ccs@pipe-E-random-ccs-data-y_tiled_gen12_rc_ccs_cc
> +kms_ccs@pipe-E-random-ccs-data-y_tiled_gen12_mc_ccs
> +kms_ccs@pipe-E-random-ccs-data-4_tiled_dg2_rc_ccs
> +kms_ccs@pipe-E-random-ccs-data-4_tiled_dg2_mc_ccs
> +kms_ccs@pipe-E-random-ccs-data-4_tiled_dg2_rc_ccs_cc
> +kms_ccs@pipe-E-random-ccs-data-4_tiled_mtl_rc_ccs
> +kms_ccs@pipe-E-random-ccs-data-4_tiled_mtl_mc_ccs
> +kms_ccs@pipe-E-random-ccs-data-4_tiled_mtl_rc_ccs_cc
> +kms_ccs@pipe-E-missing-ccs-buffer-y_tiled_ccs
> +kms_ccs@pipe-E-missing-ccs-buffer-yf_tiled_ccs
> +kms_ccs@pipe-E-missing-ccs-buffer-y_tiled_gen12_rc_ccs
> +kms_ccs@pipe-E-missing-ccs-buffer-y_tiled_gen12_rc_ccs_cc
> +kms_ccs@pipe-E-missing-ccs-buffer-y_tiled_gen12_mc_ccs
> +kms_ccs@pipe-E-missing-ccs-buffer-4_tiled_mtl_rc_ccs
> +kms_ccs@pipe-E-missing-ccs-buffer-4_tiled_mtl_mc_ccs
> +kms_ccs@pipe-E-missing-ccs-buffer-4_tiled_mtl_rc_ccs_cc
> +kms_ccs@pipe-E-ccs-on-another-bo-y_tiled_ccs
> +kms_ccs@pipe-E-ccs-on-another-bo-yf_tiled_ccs
> +kms_ccs@pipe-E-ccs-on-another-bo-y_tiled_gen12_rc_ccs
> +kms_ccs@pipe-E-ccs-on-another-bo-y_tiled_gen12_rc_ccs_cc
> +kms_ccs@pipe-E-ccs-on-another-bo-y_tiled_gen12_mc_ccs
> +kms_ccs@pipe-E-ccs-on-another-bo-4_tiled_mtl_rc_ccs
> +kms_ccs@pipe-E-ccs-on-another-bo-4_tiled_mtl_mc_ccs
> +kms_ccs@pipe-E-ccs-on-another-bo-4_tiled_mtl_rc_ccs_cc
> +kms_ccs@pipe-E-bad-aux-stride-y_tiled_ccs
> +kms_ccs@pipe-E-bad-aux-stride-yf_tiled_ccs
> +kms_ccs@pipe-E-bad-aux-stride-y_tiled_gen12_rc_ccs
> +kms_ccs@pipe-E-bad-aux-stride-y_tiled_gen12_rc_ccs_cc
> +kms_ccs@pipe-E-bad-aux-stride-y_tiled_gen12_mc_ccs
> +kms_ccs@pipe-E-bad-aux-stride-4_tiled_mtl_rc_ccs
> +kms_ccs@pipe-E-bad-aux-stride-4_tiled_mtl_mc_ccs
> +kms_ccs@pipe-E-bad-aux-stride-4_tiled_mtl_rc_ccs_cc
> +kms_ccs@pipe-E-crc-sprite-planes-basic-y_tiled_ccs
> +kms_ccs@pipe-E-crc-sprite-planes-basic-yf_tiled_ccs
> +kms_ccs@pipe-E-crc-sprite-planes-basic-y_tiled_gen12_rc_ccs
> +kms_ccs@pipe-E-crc-sprite-planes-basic-y_tiled_gen12_rc_ccs_cc
> +kms_ccs@pipe-E-crc-sprite-planes-basic-y_tiled_gen12_mc_ccs
> +kms_ccs@pipe-E-crc-sprite-planes-basic-4_tiled_dg2_rc_ccs
> +kms_ccs@pipe-E-crc-sprite-planes-basic-4_tiled_dg2_mc_ccs
> +kms_ccs@pipe-E-crc-sprite-planes-basic-4_tiled_dg2_rc_ccs_cc
> +kms_ccs@pipe-E-crc-sprite-planes-basic-4_tiled_mtl_rc_ccs
> +kms_ccs@pipe-E-crc-sprite-planes-basic-4_tiled_mtl_mc_ccs
> +kms_ccs@pipe-E-crc-sprite-planes-basic-4_tiled_mtl_rc_ccs_cc
> +kms_ccs@pipe-F-bad-pixel-format-y_tiled_ccs
> +kms_ccs@pipe-F-bad-pixel-format-yf_tiled_ccs
> +kms_ccs@pipe-F-bad-pixel-format-y_tiled_gen12_rc_ccs
> +kms_ccs@pipe-F-bad-pixel-format-y_tiled_gen12_rc_ccs_cc
> +kms_ccs@pipe-F-bad-pixel-format-y_tiled_gen12_mc_ccs
> +kms_ccs@pipe-F-bad-pixel-format-4_tiled_dg2_rc_ccs
> +kms_ccs@pipe-F-bad-pixel-format-4_tiled_dg2_mc_ccs
> +kms_ccs@pipe-F-bad-pixel-format-4_tiled_dg2_rc_ccs_cc
> +kms_ccs@pipe-F-bad-pixel-format-4_tiled_mtl_rc_ccs
> +kms_ccs@pipe-F-bad-pixel-format-4_tiled_mtl_mc_ccs
> +kms_ccs@pipe-F-bad-pixel-format-4_tiled_mtl_rc_ccs_cc
> +kms_ccs@pipe-F-bad-rotation-90-y_tiled_ccs
> +kms_ccs@pipe-F-bad-rotation-90-yf_tiled_ccs
> +kms_ccs@pipe-F-bad-rotation-90-y_tiled_gen12_rc_ccs
> +kms_ccs@pipe-F-bad-rotation-90-y_tiled_gen12_rc_ccs_cc
> +kms_ccs@pipe-F-bad-rotation-90-y_tiled_gen12_mc_ccs
> +kms_ccs@pipe-F-bad-rotation-90-4_tiled_dg2_rc_ccs
> +kms_ccs@pipe-F-bad-rotation-90-4_tiled_dg2_mc_ccs
> +kms_ccs@pipe-F-bad-rotation-90-4_tiled_dg2_rc_ccs_cc
> +kms_ccs@pipe-F-bad-rotation-90-4_tiled_mtl_rc_ccs
> +kms_ccs@pipe-F-bad-rotation-90-4_tiled_mtl_mc_ccs
> +kms_ccs@pipe-F-bad-rotation-90-4_tiled_mtl_rc_ccs_cc
> +kms_ccs@pipe-F-crc-primary-basic-y_tiled_ccs
> +kms_ccs@pipe-F-crc-primary-basic-yf_tiled_ccs
> +kms_ccs@pipe-F-crc-primary-basic-y_tiled_gen12_rc_ccs
> +kms_ccs@pipe-F-crc-primary-basic-y_tiled_gen12_rc_ccs_cc
> +kms_ccs@pipe-F-crc-primary-basic-y_tiled_gen12_mc_ccs
> +kms_ccs@pipe-F-crc-primary-basic-4_tiled_dg2_rc_ccs
> +kms_ccs@pipe-F-crc-primary-basic-4_tiled_dg2_mc_ccs
> +kms_ccs@pipe-F-crc-primary-basic-4_tiled_dg2_rc_ccs_cc
> +kms_ccs@pipe-F-crc-primary-basic-4_tiled_mtl_rc_ccs
> +kms_ccs@pipe-F-crc-primary-basic-4_tiled_mtl_mc_ccs
> +kms_ccs@pipe-F-crc-primary-basic-4_tiled_mtl_rc_ccs_cc
> +kms_ccs@pipe-F-crc-primary-rotation-180-y_tiled_ccs
> +kms_ccs@pipe-F-crc-primary-rotation-180-yf_tiled_ccs
> +kms_ccs@pipe-F-crc-primary-rotation-180-y_tiled_gen12_rc_ccs
> +kms_ccs@pipe-F-crc-primary-rotation-180-y_tiled_gen12_rc_ccs_cc
> +kms_ccs@pipe-F-crc-primary-rotation-180-y_tiled_gen12_mc_ccs
> +kms_ccs@pipe-F-crc-primary-rotation-180-4_tiled_dg2_rc_ccs
> +kms_ccs@pipe-F-crc-primary-rotation-180-4_tiled_dg2_mc_ccs
> +kms_ccs@pipe-F-crc-primary-rotation-180-4_tiled_dg2_rc_ccs_cc
> +kms_ccs@pipe-F-crc-primary-rotation-180-4_tiled_mtl_rc_ccs
> +kms_ccs@pipe-F-crc-primary-rotation-180-4_tiled_mtl_mc_ccs
> +kms_ccs@pipe-F-crc-primary-rotation-180-4_tiled_mtl_rc_ccs_cc
> +kms_ccs@pipe-F-random-ccs-data-y_tiled_ccs
> +kms_ccs@pipe-F-random-ccs-data-yf_tiled_ccs
> +kms_ccs@pipe-F-random-ccs-data-y_tiled_gen12_rc_ccs
> +kms_ccs@pipe-F-random-ccs-data-y_tiled_gen12_rc_ccs_cc
> +kms_ccs@pipe-F-random-ccs-data-y_tiled_gen12_mc_ccs
> +kms_ccs@pipe-F-random-ccs-data-4_tiled_dg2_rc_ccs
> +kms_ccs@pipe-F-random-ccs-data-4_tiled_dg2_mc_ccs
> +kms_ccs@pipe-F-random-ccs-data-4_tiled_dg2_rc_ccs_cc
> +kms_ccs@pipe-F-random-ccs-data-4_tiled_mtl_rc_ccs
> +kms_ccs@pipe-F-random-ccs-data-4_tiled_mtl_mc_ccs
> +kms_ccs@pipe-F-random-ccs-data-4_tiled_mtl_rc_ccs_cc
> +kms_ccs@pipe-F-missing-ccs-buffer-y_tiled_ccs
> +kms_ccs@pipe-F-missing-ccs-buffer-yf_tiled_ccs
> +kms_ccs@pipe-F-missing-ccs-buffer-y_tiled_gen12_rc_ccs
> +kms_ccs@pipe-F-missing-ccs-buffer-y_tiled_gen12_rc_ccs_cc
> +kms_ccs@pipe-F-missing-ccs-buffer-y_tiled_gen12_mc_ccs
> +kms_ccs@pipe-F-missing-ccs-buffer-4_tiled_mtl_rc_ccs
> +kms_ccs@pipe-F-missing-ccs-buffer-4_tiled_mtl_mc_ccs
> +kms_ccs@pipe-F-missing-ccs-buffer-4_tiled_mtl_rc_ccs_cc
> +kms_ccs@pipe-F-ccs-on-another-bo-y_tiled_ccs
> +kms_ccs@pipe-F-ccs-on-another-bo-yf_tiled_ccs
> +kms_ccs@pipe-F-ccs-on-another-bo-y_tiled_gen12_rc_ccs
> +kms_ccs@pipe-F-ccs-on-another-bo-y_tiled_gen12_rc_ccs_cc
> +kms_ccs@pipe-F-ccs-on-another-bo-y_tiled_gen12_mc_ccs
> +kms_ccs@pipe-F-ccs-on-another-bo-4_tiled_mtl_rc_ccs
> +kms_ccs@pipe-F-ccs-on-another-bo-4_tiled_mtl_mc_ccs
> +kms_ccs@pipe-F-ccs-on-another-bo-4_tiled_mtl_rc_ccs_cc
> +kms_ccs@pipe-F-bad-aux-stride-y_tiled_ccs
> +kms_ccs@pipe-F-bad-aux-stride-yf_tiled_ccs
> +kms_ccs@pipe-F-bad-aux-stride-y_tiled_gen12_rc_ccs
> +kms_ccs@pipe-F-bad-aux-stride-y_tiled_gen12_rc_ccs_cc
> +kms_ccs@pipe-F-bad-aux-stride-y_tiled_gen12_mc_ccs
> +kms_ccs@pipe-F-bad-aux-stride-4_tiled_mtl_rc_ccs
> +kms_ccs@pipe-F-bad-aux-stride-4_tiled_mtl_mc_ccs
> +kms_ccs@pipe-F-bad-aux-stride-4_tiled_mtl_rc_ccs_cc
> +kms_ccs@pipe-F-crc-sprite-planes-basic-y_tiled_ccs
> +kms_ccs@pipe-F-crc-sprite-planes-basic-yf_tiled_ccs
> +kms_ccs@pipe-F-crc-sprite-planes-basic-y_tiled_gen12_rc_ccs
> +kms_ccs@pipe-F-crc-sprite-planes-basic-y_tiled_gen12_rc_ccs_cc
> +kms_ccs@pipe-F-crc-sprite-planes-basic-y_tiled_gen12_mc_ccs
> +kms_ccs@pipe-F-crc-sprite-planes-basic-4_tiled_dg2_rc_ccs
> +kms_ccs@pipe-F-crc-sprite-planes-basic-4_tiled_dg2_mc_ccs
> +kms_ccs@pipe-F-crc-sprite-planes-basic-4_tiled_dg2_rc_ccs_cc
> +kms_ccs@pipe-F-crc-sprite-planes-basic-4_tiled_mtl_rc_ccs
> +kms_ccs@pipe-F-crc-sprite-planes-basic-4_tiled_mtl_mc_ccs
> +kms_ccs@pipe-F-crc-sprite-planes-basic-4_tiled_mtl_rc_ccs_cc
> +kms_ccs@pipe-G-bad-pixel-format-y_tiled_ccs
> +kms_ccs@pipe-G-bad-pixel-format-yf_tiled_ccs
> +kms_ccs@pipe-G-bad-pixel-format-y_tiled_gen12_rc_ccs
> +kms_ccs@pipe-G-bad-pixel-format-y_tiled_gen12_rc_ccs_cc
> +kms_ccs@pipe-G-bad-pixel-format-y_tiled_gen12_mc_ccs
> +kms_ccs@pipe-G-bad-pixel-format-4_tiled_dg2_rc_ccs
> +kms_ccs@pipe-G-bad-pixel-format-4_tiled_dg2_mc_ccs
> +kms_ccs@pipe-G-bad-pixel-format-4_tiled_dg2_rc_ccs_cc
> +kms_ccs@pipe-G-bad-pixel-format-4_tiled_mtl_rc_ccs
> +kms_ccs@pipe-G-bad-pixel-format-4_tiled_mtl_mc_ccs
> +kms_ccs@pipe-G-bad-pixel-format-4_tiled_mtl_rc_ccs_cc
> +kms_ccs@pipe-G-bad-rotation-90-y_tiled_ccs
> +kms_ccs@pipe-G-bad-rotation-90-yf_tiled_ccs
> +kms_ccs@pipe-G-bad-rotation-90-y_tiled_gen12_rc_ccs
> +kms_ccs@pipe-G-bad-rotation-90-y_tiled_gen12_rc_ccs_cc
> +kms_ccs@pipe-G-bad-rotation-90-y_tiled_gen12_mc_ccs
> +kms_ccs@pipe-G-bad-rotation-90-4_tiled_dg2_rc_ccs
> +kms_ccs@pipe-G-bad-rotation-90-4_tiled_dg2_mc_ccs
> +kms_ccs@pipe-G-bad-rotation-90-4_tiled_dg2_rc_ccs_cc
> +kms_ccs@pipe-G-bad-rotation-90-4_tiled_mtl_rc_ccs
> +kms_ccs@pipe-G-bad-rotation-90-4_tiled_mtl_mc_ccs
> +kms_ccs@pipe-G-bad-rotation-90-4_tiled_mtl_rc_ccs_cc
> +kms_ccs@pipe-G-crc-primary-basic-y_tiled_ccs
> +kms_ccs@pipe-G-crc-primary-basic-yf_tiled_ccs
> +kms_ccs@pipe-G-crc-primary-basic-y_tiled_gen12_rc_ccs
> +kms_ccs@pipe-G-crc-primary-basic-y_tiled_gen12_rc_ccs_cc
> +kms_ccs@pipe-G-crc-primary-basic-y_tiled_gen12_mc_ccs
> +kms_ccs@pipe-G-crc-primary-basic-4_tiled_dg2_rc_ccs
> +kms_ccs@pipe-G-crc-primary-basic-4_tiled_dg2_mc_ccs
> +kms_ccs@pipe-G-crc-primary-basic-4_tiled_dg2_rc_ccs_cc
> +kms_ccs@pipe-G-crc-primary-basic-4_tiled_mtl_rc_ccs
> +kms_ccs@pipe-G-crc-primary-basic-4_tiled_mtl_mc_ccs
> +kms_ccs@pipe-G-crc-primary-basic-4_tiled_mtl_rc_ccs_cc
> +kms_ccs@pipe-G-crc-primary-rotation-180-y_tiled_ccs
> +kms_ccs@pipe-G-crc-primary-rotation-180-yf_tiled_ccs
> +kms_ccs@pipe-G-crc-primary-rotation-180-y_tiled_gen12_rc_ccs
> +kms_ccs@pipe-G-crc-primary-rotation-180-y_tiled_gen12_rc_ccs_cc
> +kms_ccs@pipe-G-crc-primary-rotation-180-y_tiled_gen12_mc_ccs
> +kms_ccs@pipe-G-crc-primary-rotation-180-4_tiled_dg2_rc_ccs
> +kms_ccs@pipe-G-crc-primary-rotation-180-4_tiled_dg2_mc_ccs
> +kms_ccs@pipe-G-crc-primary-rotation-180-4_tiled_dg2_rc_ccs_cc
> +kms_ccs@pipe-G-crc-primary-rotation-180-4_tiled_mtl_rc_ccs
> +kms_ccs@pipe-G-crc-primary-rotation-180-4_tiled_mtl_mc_ccs
> +kms_ccs@pipe-G-crc-primary-rotation-180-4_tiled_mtl_rc_ccs_cc
> +kms_ccs@pipe-G-random-ccs-data-y_tiled_ccs
> +kms_ccs@pipe-G-random-ccs-data-yf_tiled_ccs
> +kms_ccs@pipe-G-random-ccs-data-y_tiled_gen12_rc_ccs
> +kms_ccs@pipe-G-random-ccs-data-y_tiled_gen12_rc_ccs_cc
> +kms_ccs@pipe-G-random-ccs-data-y_tiled_gen12_mc_ccs
> +kms_ccs@pipe-G-random-ccs-data-4_tiled_dg2_rc_ccs
> +kms_ccs@pipe-G-random-ccs-data-4_tiled_dg2_mc_ccs
> +kms_ccs@pipe-G-random-ccs-data-4_tiled_dg2_rc_ccs_cc
> +kms_ccs@pipe-G-random-ccs-data-4_tiled_mtl_rc_ccs
> +kms_ccs@pipe-G-random-ccs-data-4_tiled_mtl_mc_ccs
> +kms_ccs@pipe-G-random-ccs-data-4_tiled_mtl_rc_ccs_cc
> +kms_ccs@pipe-G-missing-ccs-buffer-y_tiled_ccs
> +kms_ccs@pipe-G-missing-ccs-buffer-yf_tiled_ccs
> +kms_ccs@pipe-G-missing-ccs-buffer-y_tiled_gen12_rc_ccs
> +kms_ccs@pipe-G-missing-ccs-buffer-y_tiled_gen12_rc_ccs_cc
> +kms_ccs@pipe-G-missing-ccs-buffer-y_tiled_gen12_mc_ccs
> +kms_ccs@pipe-G-missing-ccs-buffer-4_tiled_mtl_rc_ccs
> +kms_ccs@pipe-G-missing-ccs-buffer-4_tiled_mtl_mc_ccs
> +kms_ccs@pipe-G-missing-ccs-buffer-4_tiled_mtl_rc_ccs_cc
> +kms_ccs@pipe-G-ccs-on-another-bo-y_tiled_ccs
> +kms_ccs@pipe-G-ccs-on-another-bo-yf_tiled_ccs
> +kms_ccs@pipe-G-ccs-on-another-bo-y_tiled_gen12_rc_ccs
> +kms_ccs@pipe-G-ccs-on-another-bo-y_tiled_gen12_rc_ccs_cc
> +kms_ccs@pipe-G-ccs-on-another-bo-y_tiled_gen12_mc_ccs
> +kms_ccs@pipe-G-ccs-on-another-bo-4_tiled_mtl_rc_ccs
> +kms_ccs@pipe-G-ccs-on-another-bo-4_tiled_mtl_mc_ccs
> +kms_ccs@pipe-G-ccs-on-another-bo-4_tiled_mtl_rc_ccs_cc
> +kms_ccs@pipe-G-bad-aux-stride-y_tiled_ccs
> +kms_ccs@pipe-G-bad-aux-stride-yf_tiled_ccs
> +kms_ccs@pipe-G-bad-aux-stride-y_tiled_gen12_rc_ccs
> +kms_ccs@pipe-G-bad-aux-stride-y_tiled_gen12_rc_ccs_cc
> +kms_ccs@pipe-G-bad-aux-stride-y_tiled_gen12_mc_ccs
> +kms_ccs@pipe-G-bad-aux-stride-4_tiled_mtl_rc_ccs
> +kms_ccs@pipe-G-bad-aux-stride-4_tiled_mtl_mc_ccs
> +kms_ccs@pipe-G-bad-aux-stride-4_tiled_mtl_rc_ccs_cc
> +kms_ccs@pipe-G-crc-sprite-planes-basic-y_tiled_ccs
> +kms_ccs@pipe-G-crc-sprite-planes-basic-yf_tiled_ccs
> +kms_ccs@pipe-G-crc-sprite-planes-basic-y_tiled_gen12_rc_ccs
> +kms_ccs@pipe-G-crc-sprite-planes-basic-y_tiled_gen12_rc_ccs_cc
> +kms_ccs@pipe-G-crc-sprite-planes-basic-y_tiled_gen12_mc_ccs
> +kms_ccs@pipe-G-crc-sprite-planes-basic-4_tiled_dg2_rc_ccs
> +kms_ccs@pipe-G-crc-sprite-planes-basic-4_tiled_dg2_mc_ccs
> +kms_ccs@pipe-G-crc-sprite-planes-basic-4_tiled_dg2_rc_ccs_cc
> +kms_ccs@pipe-G-crc-sprite-planes-basic-4_tiled_mtl_rc_ccs
> +kms_ccs@pipe-G-crc-sprite-planes-basic-4_tiled_mtl_mc_ccs
> +kms_ccs@pipe-G-crc-sprite-planes-basic-4_tiled_mtl_rc_ccs_cc
> +kms_ccs@pipe-H-bad-pixel-format-y_tiled_ccs
> +kms_ccs@pipe-H-bad-pixel-format-yf_tiled_ccs
> +kms_ccs@pipe-H-bad-pixel-format-y_tiled_gen12_rc_ccs
> +kms_ccs@pipe-H-bad-pixel-format-y_tiled_gen12_rc_ccs_cc
> +kms_ccs@pipe-H-bad-pixel-format-y_tiled_gen12_mc_ccs
> +kms_ccs@pipe-H-bad-pixel-format-4_tiled_dg2_rc_ccs
> +kms_ccs@pipe-H-bad-pixel-format-4_tiled_dg2_mc_ccs
> +kms_ccs@pipe-H-bad-pixel-format-4_tiled_dg2_rc_ccs_cc
> +kms_ccs@pipe-H-bad-pixel-format-4_tiled_mtl_rc_ccs
> +kms_ccs@pipe-H-bad-pixel-format-4_tiled_mtl_mc_ccs
> +kms_ccs@pipe-H-bad-pixel-format-4_tiled_mtl_rc_ccs_cc
> +kms_ccs@pipe-H-bad-rotation-90-y_tiled_ccs
> +kms_ccs@pipe-H-bad-rotation-90-yf_tiled_ccs
> +kms_ccs@pipe-H-bad-rotation-90-y_tiled_gen12_rc_ccs
> +kms_ccs@pipe-H-bad-rotation-90-y_tiled_gen12_rc_ccs_cc
> +kms_ccs@pipe-H-bad-rotation-90-y_tiled_gen12_mc_ccs
> +kms_ccs@pipe-H-bad-rotation-90-4_tiled_dg2_rc_ccs
> +kms_ccs@pipe-H-bad-rotation-90-4_tiled_dg2_mc_ccs
> +kms_ccs@pipe-H-bad-rotation-90-4_tiled_dg2_rc_ccs_cc
> +kms_ccs@pipe-H-bad-rotation-90-4_tiled_mtl_rc_ccs
> +kms_ccs@pipe-H-bad-rotation-90-4_tiled_mtl_mc_ccs
> +kms_ccs@pipe-H-bad-rotation-90-4_tiled_mtl_rc_ccs_cc
> +kms_ccs@pipe-H-crc-primary-basic-y_tiled_ccs
> +kms_ccs@pipe-H-crc-primary-basic-yf_tiled_ccs
> +kms_ccs@pipe-H-crc-primary-basic-y_tiled_gen12_rc_ccs
> +kms_ccs@pipe-H-crc-primary-basic-y_tiled_gen12_rc_ccs_cc
> +kms_ccs@pipe-H-crc-primary-basic-y_tiled_gen12_mc_ccs
> +kms_ccs@pipe-H-crc-primary-basic-4_tiled_dg2_rc_ccs
> +kms_ccs@pipe-H-crc-primary-basic-4_tiled_dg2_mc_ccs
> +kms_ccs@pipe-H-crc-primary-basic-4_tiled_dg2_rc_ccs_cc
> +kms_ccs@pipe-H-crc-primary-basic-4_tiled_mtl_rc_ccs
> +kms_ccs@pipe-H-crc-primary-basic-4_tiled_mtl_mc_ccs
> +kms_ccs@pipe-H-crc-primary-basic-4_tiled_mtl_rc_ccs_cc
> +kms_ccs@pipe-H-crc-primary-rotation-180-y_tiled_ccs
> +kms_ccs@pipe-H-crc-primary-rotation-180-yf_tiled_ccs
> +kms_ccs@pipe-H-crc-primary-rotation-180-y_tiled_gen12_rc_ccs
> +kms_ccs@pipe-H-crc-primary-rotation-180-y_tiled_gen12_rc_ccs_cc
> +kms_ccs@pipe-H-crc-primary-rotation-180-y_tiled_gen12_mc_ccs
> +kms_ccs@pipe-H-crc-primary-rotation-180-4_tiled_dg2_rc_ccs
> +kms_ccs@pipe-H-crc-primary-rotation-180-4_tiled_dg2_mc_ccs
> +kms_ccs@pipe-H-crc-primary-rotation-180-4_tiled_dg2_rc_ccs_cc
> +kms_ccs@pipe-H-crc-primary-rotation-180-4_tiled_mtl_rc_ccs
> +kms_ccs@pipe-H-crc-primary-rotation-180-4_tiled_mtl_mc_ccs
> +kms_ccs@pipe-H-crc-primary-rotation-180-4_tiled_mtl_rc_ccs_cc
> +kms_ccs@pipe-H-random-ccs-data-y_tiled_ccs
> +kms_ccs@pipe-H-random-ccs-data-yf_tiled_ccs
> +kms_ccs@pipe-H-random-ccs-data-y_tiled_gen12_rc_ccs
> +kms_ccs@pipe-H-random-ccs-data-y_tiled_gen12_rc_ccs_cc
> +kms_ccs@pipe-H-random-ccs-data-y_tiled_gen12_mc_ccs
> +kms_ccs@pipe-H-random-ccs-data-4_tiled_dg2_rc_ccs
> +kms_ccs@pipe-H-random-ccs-data-4_tiled_dg2_mc_ccs
> +kms_ccs@pipe-H-random-ccs-data-4_tiled_dg2_rc_ccs_cc
> +kms_ccs@pipe-H-random-ccs-data-4_tiled_mtl_rc_ccs
> +kms_ccs@pipe-H-random-ccs-data-4_tiled_mtl_mc_ccs
> +kms_ccs@pipe-H-random-ccs-data-4_tiled_mtl_rc_ccs_cc
> +kms_ccs@pipe-H-missing-ccs-buffer-y_tiled_ccs
> +kms_ccs@pipe-H-missing-ccs-buffer-yf_tiled_ccs
> +kms_ccs@pipe-H-missing-ccs-buffer-y_tiled_gen12_rc_ccs
> +kms_ccs@pipe-H-missing-ccs-buffer-y_tiled_gen12_rc_ccs_cc
> +kms_ccs@pipe-H-missing-ccs-buffer-y_tiled_gen12_mc_ccs
> +kms_ccs@pipe-H-missing-ccs-buffer-4_tiled_mtl_rc_ccs
> +kms_ccs@pipe-H-missing-ccs-buffer-4_tiled_mtl_mc_ccs
> +kms_ccs@pipe-H-missing-ccs-buffer-4_tiled_mtl_rc_ccs_cc
> +kms_ccs@pipe-H-ccs-on-another-bo-y_tiled_ccs
> +kms_ccs@pipe-H-ccs-on-another-bo-yf_tiled_ccs
> +kms_ccs@pipe-H-ccs-on-another-bo-y_tiled_gen12_rc_ccs
> +kms_ccs@pipe-H-ccs-on-another-bo-y_tiled_gen12_rc_ccs_cc
> +kms_ccs@pipe-H-ccs-on-another-bo-y_tiled_gen12_mc_ccs
> +kms_ccs@pipe-H-ccs-on-another-bo-4_tiled_mtl_rc_ccs
> +kms_ccs@pipe-H-ccs-on-another-bo-4_tiled_mtl_mc_ccs
> +kms_ccs@pipe-H-ccs-on-another-bo-4_tiled_mtl_rc_ccs_cc
> +kms_ccs@pipe-H-bad-aux-stride-y_tiled_ccs
> +kms_ccs@pipe-H-bad-aux-stride-yf_tiled_ccs
> +kms_ccs@pipe-H-bad-aux-stride-y_tiled_gen12_rc_ccs
> +kms_ccs@pipe-H-bad-aux-stride-y_tiled_gen12_rc_ccs_cc
> +kms_ccs@pipe-H-bad-aux-stride-y_tiled_gen12_mc_ccs
> +kms_ccs@pipe-H-bad-aux-stride-4_tiled_mtl_rc_ccs
> +kms_ccs@pipe-H-bad-aux-stride-4_tiled_mtl_mc_ccs
> +kms_ccs@pipe-H-bad-aux-stride-4_tiled_mtl_rc_ccs_cc
> +kms_ccs@pipe-H-crc-sprite-planes-basic-y_tiled_ccs
> +kms_ccs@pipe-H-crc-sprite-planes-basic-yf_tiled_ccs
> +kms_ccs@pipe-H-crc-sprite-planes-basic-y_tiled_gen12_rc_ccs
> +kms_ccs@pipe-H-crc-sprite-planes-basic-y_tiled_gen12_rc_ccs_cc
> +kms_ccs@pipe-H-crc-sprite-planes-basic-y_tiled_gen12_mc_ccs
> +kms_ccs@pipe-H-crc-sprite-planes-basic-4_tiled_dg2_rc_ccs
> +kms_ccs@pipe-H-crc-sprite-planes-basic-4_tiled_dg2_mc_ccs
> +kms_ccs@pipe-H-crc-sprite-planes-basic-4_tiled_dg2_rc_ccs_cc
> +kms_ccs@pipe-H-crc-sprite-planes-basic-4_tiled_mtl_rc_ccs
> +kms_ccs@pipe-H-crc-sprite-planes-basic-4_tiled_mtl_mc_ccs
> +kms_ccs@pipe-H-crc-sprite-planes-basic-4_tiled_mtl_rc_ccs_cc
> +kms_cdclk@plane-scaling
> +kms_cdclk@mode-transition
> +kms_cdclk@mode-transition-all-outputs
> +kms_color@degamma
> +kms_color@gamma
> +kms_color@legacy-gamma
> +kms_color@legacy-gamma-reset
> +kms_color@ctm-red-to-blue
> +kms_color@ctm-green-to-red
> +kms_color@ctm-blue-to-red
> +kms_color@ctm-max
> +kms_color@ctm-negative
> +kms_color@ctm-0-25
> +kms_color@ctm-0-50
> +kms_color@ctm-0-75
> +kms_color@ctm-signed
> +kms_color@deep-color
> +kms_color@invalid-gamma-lut-sizes
> +kms_color@invalid-degamma-lut-sizes
> +kms_color@invalid-ctm-matrix-sizes
> +kms_concurrent@pipe-A
> +kms_concurrent@pipe-B
> +kms_concurrent@pipe-C
> +kms_concurrent@pipe-D
> +kms_concurrent@pipe-E
> +kms_concurrent@pipe-F
> +kms_concurrent@pipe-G
> +kms_concurrent@pipe-H
> +kms_content_protection@legacy
> +kms_content_protection@atomic
> +kms_content_protection@atomic-dpms
> +kms_content_protection@LIC
> +kms_content_protection@type1
> +kms_content_protection@mei_interface
> +kms_content_protection@content_type_change
> +kms_content_protection@uevent
> +kms_content_protection@srm
> +kms_content_protection@dp-mst-type-0
> +kms_content_protection@dp-mst-lic-type-0
> +kms_content_protection@dp-mst-type-1
> +kms_content_protection@dp-mst-lic-type-1
> +kms_cursor_crc@cursor-size-change
> +kms_cursor_crc@cursor-alpha-opaque
> +kms_cursor_crc@cursor-alpha-transparent
> +kms_cursor_crc@cursor-dpms
> +kms_cursor_crc@cursor-suspend
> +kms_cursor_crc@cursor-onscreen-32x32
> +kms_cursor_crc@cursor-offscreen-32x32
> +kms_cursor_crc@cursor-sliding-32x32
> +kms_cursor_crc@cursor-random-32x32
> +kms_cursor_crc@cursor-rapid-movement-32x32
> +kms_cursor_crc@cursor-onscreen-32x10
> +kms_cursor_crc@cursor-offscreen-32x10
> +kms_cursor_crc@cursor-sliding-32x10
> +kms_cursor_crc@cursor-random-32x10
> +kms_cursor_crc@cursor-rapid-movement-32x10
> +kms_cursor_crc@cursor-onscreen-64x64
> +kms_cursor_crc@cursor-offscreen-64x64
> +kms_cursor_crc@cursor-sliding-64x64
> +kms_cursor_crc@cursor-random-64x64
> +kms_cursor_crc@cursor-rapid-movement-64x64
> +kms_cursor_crc@cursor-onscreen-64x21
> +kms_cursor_crc@cursor-offscreen-64x21
> +kms_cursor_crc@cursor-sliding-64x21
> +kms_cursor_crc@cursor-random-64x21
> +kms_cursor_crc@cursor-rapid-movement-64x21
> +kms_cursor_crc@cursor-onscreen-128x128
> +kms_cursor_crc@cursor-offscreen-128x128
> +kms_cursor_crc@cursor-sliding-128x128
> +kms_cursor_crc@cursor-random-128x128
> +kms_cursor_crc@cursor-rapid-movement-128x128
> +kms_cursor_crc@cursor-onscreen-128x42
> +kms_cursor_crc@cursor-offscreen-128x42
> +kms_cursor_crc@cursor-sliding-128x42
> +kms_cursor_crc@cursor-random-128x42
> +kms_cursor_crc@cursor-rapid-movement-128x42
> +kms_cursor_crc@cursor-onscreen-256x256
> +kms_cursor_crc@cursor-offscreen-256x256
> +kms_cursor_crc@cursor-sliding-256x256
> +kms_cursor_crc@cursor-random-256x256
> +kms_cursor_crc@cursor-rapid-movement-256x256
> +kms_cursor_crc@cursor-onscreen-256x85
> +kms_cursor_crc@cursor-offscreen-256x85
> +kms_cursor_crc@cursor-sliding-256x85
> +kms_cursor_crc@cursor-random-256x85
> +kms_cursor_crc@cursor-rapid-movement-256x85
> +kms_cursor_crc@cursor-onscreen-512x512
> +kms_cursor_crc@cursor-offscreen-512x512
> +kms_cursor_crc@cursor-sliding-512x512
> +kms_cursor_crc@cursor-random-512x512
> +kms_cursor_crc@cursor-rapid-movement-512x512
> +kms_cursor_crc@cursor-onscreen-512x170
> +kms_cursor_crc@cursor-offscreen-512x170
> +kms_cursor_crc@cursor-sliding-512x170
> +kms_cursor_crc@cursor-random-512x170
> +kms_cursor_crc@cursor-rapid-movement-512x170
> +kms_cursor_crc@cursor-onscreen-max-size
> +kms_cursor_crc@cursor-offscreen-max-size
> +kms_cursor_crc@cursor-sliding-max-size
> +kms_cursor_crc@cursor-random-max-size
> +kms_cursor_crc@cursor-rapid-movement-max-size
> +kms_cursor_legacy@single-bo
> +kms_cursor_legacy@single-move
> +kms_cursor_legacy@forked-bo
> +kms_cursor_legacy@forked-move
> +kms_cursor_legacy@torture-bo
> +kms_cursor_legacy@torture-move
> +kms_cursor_legacy@nonblocking-modeset-vs-cursor-atomic
> +kms_cursor_legacy@long-nonblocking-modeset-vs-cursor-atomic
> +kms_cursor_legacy@2x-flip-vs-cursor-legacy
> +kms_cursor_legacy@2x-flip-vs-cursor-atomic
> +kms_cursor_legacy@2x-long-flip-vs-cursor-legacy
> +kms_cursor_legacy@2x-long-flip-vs-cursor-atomic
> +kms_cursor_legacy@2x-nonblocking-modeset-vs-cursor-atomic
> +kms_cursor_legacy@2x-long-nonblocking-modeset-vs-cursor-atomic
> +kms_cursor_legacy@2x-cursor-vs-flip-legacy
> +kms_cursor_legacy@2x-long-cursor-vs-flip-legacy
> +kms_cursor_legacy@2x-cursor-vs-flip-atomic
> +kms_cursor_legacy@2x-long-cursor-vs-flip-atomic
> +kms_cursor_legacy@flip-vs-cursor-crc-legacy
> +kms_cursor_legacy@flip-vs-cursor-crc-atomic
> +kms_cursor_legacy@flip-vs-cursor-busy-crc-legacy
> +kms_cursor_legacy@flip-vs-cursor-busy-crc-atomic
> +kms_cursor_legacy@basic-flip-before-cursor-legacy
> +kms_cursor_legacy@basic-busy-flip-before-cursor-legacy
> +kms_cursor_legacy@basic-flip-after-cursor-legacy
> +kms_cursor_legacy@basic-flip-before-cursor-varying-size
> +kms_cursor_legacy@basic-busy-flip-before-cursor-varying-size
> +kms_cursor_legacy@basic-flip-after-cursor-varying-size
> +kms_cursor_legacy@short-flip-before-cursor-toggle
> +kms_cursor_legacy@short-busy-flip-before-cursor-toggle
> +kms_cursor_legacy@short-flip-after-cursor-toggle
> +kms_cursor_legacy@basic-flip-before-cursor-atomic
> +kms_cursor_legacy@basic-busy-flip-before-cursor-atomic
> +kms_cursor_legacy@basic-flip-after-cursor-atomic
> +kms_cursor_legacy@short-flip-before-cursor-atomic-transitions
> +kms_cursor_legacy@short-busy-flip-before-cursor-atomic-transitions
> +kms_cursor_legacy@short-flip-after-cursor-atomic-transitions
> +kms_cursor_legacy@short-flip-before-cursor-atomic-transitions-varying-size
> +kms_cursor_legacy@short-busy-flip-before-cursor-atomic-transitions-varying-size
> +kms_cursor_legacy@short-flip-after-cursor-atomic-transitions-varying-size
> +kms_cursor_legacy@cursor-vs-flip-legacy
> +kms_cursor_legacy@flip-vs-cursor-legacy
> +kms_cursor_legacy@cursorA-vs-flipA-legacy
> +kms_cursor_legacy@cursorA-vs-flipB-legacy
> +kms_cursor_legacy@cursorB-vs-flipA-legacy
> +kms_cursor_legacy@cursorB-vs-flipB-legacy
> +kms_cursor_legacy@cursor-vs-flip-varying-size
> +kms_cursor_legacy@flip-vs-cursor-varying-size
> +kms_cursor_legacy@cursorA-vs-flipA-varying-size
> +kms_cursor_legacy@cursorA-vs-flipB-varying-size
> +kms_cursor_legacy@cursorB-vs-flipA-varying-size
> +kms_cursor_legacy@cursorB-vs-flipB-varying-size
> +kms_cursor_legacy@cursor-vs-flip-toggle
> +kms_cursor_legacy@flip-vs-cursor-toggle
> +kms_cursor_legacy@cursorA-vs-flipA-toggle
> +kms_cursor_legacy@cursorA-vs-flipB-toggle
> +kms_cursor_legacy@cursorB-vs-flipA-toggle
> +kms_cursor_legacy@cursorB-vs-flipB-toggle
> +kms_cursor_legacy@cursor-vs-flip-atomic
> +kms_cursor_legacy@flip-vs-cursor-atomic
> +kms_cursor_legacy@cursorA-vs-flipA-atomic
> +kms_cursor_legacy@cursorA-vs-flipB-atomic
> +kms_cursor_legacy@cursorB-vs-flipA-atomic
> +kms_cursor_legacy@cursorB-vs-flipB-atomic
> +kms_cursor_legacy@cursor-vs-flip-atomic-transitions
> +kms_cursor_legacy@flip-vs-cursor-atomic-transitions
> +kms_cursor_legacy@cursorA-vs-flipA-atomic-transitions
> +kms_cursor_legacy@cursorA-vs-flipB-atomic-transitions
> +kms_cursor_legacy@cursorB-vs-flipA-atomic-transitions
> +kms_cursor_legacy@cursorB-vs-flipB-atomic-transitions
> +kms_cursor_legacy@cursor-vs-flip-atomic-transitions-varying-size
> +kms_cursor_legacy@flip-vs-cursor-atomic-transitions-varying-size
> +kms_cursor_legacy@cursorA-vs-flipA-atomic-transitions-varying-size
> +kms_cursor_legacy@cursorA-vs-flipB-atomic-transitions-varying-size
> +kms_cursor_legacy@cursorB-vs-flipA-atomic-transitions-varying-size
> +kms_cursor_legacy@cursorB-vs-flipB-atomic-transitions-varying-size
> +kms_dither@FB-8BPC-Vs-Panel-6BPC
> +kms_dither@FB-8BPC-Vs-Panel-8BPC
> +kms_dp_aux_dev
> +kms_tiled_display@basic-test-pattern
> +kms_tiled_display@basic-test-pattern-with-chamelium
> +kms_draw_crc@draw-method-mmap-cpu
> +kms_draw_crc@draw-method-mmap-gtt
> +kms_draw_crc@draw-method-mmap-wc
> +kms_draw_crc@draw-method-pwrite
> +kms_draw_crc@draw-method-blt
> +kms_draw_crc@draw-method-render
> +kms_draw_crc@fill-fb
> +kms_dsc@dsc-basic
> +kms_dsc@dsc-with-formats
> +kms_dsc@dsc-with-bpc
> +kms_dsc@dsc-with-bpc-formats
> +kms_dsc@dsc-with-output-formats
> +kms_fbcon_fbt@fbc
> +kms_fbcon_fbt@psr
> +kms_fbcon_fbt@fbc-suspend
> +kms_fbcon_fbt@psr-suspend
> +kms_fence_pin_leak
> +kms_flip@nonblocking-read
> +kms_flip@wf_vblank-ts-check
> +kms_flip@2x-wf_vblank-ts-check
> +kms_flip@blocking-wf_vblank
> +kms_flip@2x-blocking-wf_vblank
> +kms_flip@absolute-wf_vblank
> +kms_flip@2x-absolute-wf_vblank
> +kms_flip@blocking-absolute-wf_vblank
> +kms_flip@2x-blocking-absolute-wf_vblank
> +kms_flip@basic-plain-flip
> +kms_flip@2x-plain-flip
> +kms_flip@busy-flip
> +kms_flip@2x-busy-flip
> +kms_flip@flip-vs-fences
> +kms_flip@2x-flip-vs-fences
> +kms_flip@plain-flip-ts-check
> +kms_flip@2x-plain-flip-ts-check
> +kms_flip@plain-flip-fb-recreate
> +kms_flip@2x-plain-flip-fb-recreate
> +kms_flip@flip-vs-rmfb
> +kms_flip@2x-flip-vs-rmfb
> +kms_flip@basic-flip-vs-dpms
> +kms_flip@2x-flip-vs-dpms
> +kms_flip@flip-vs-panning
> +kms_flip@2x-flip-vs-panning
> +kms_flip@basic-flip-vs-modeset
> +kms_flip@2x-flip-vs-modeset
> +kms_flip@flip-vs-expired-vblank
> +kms_flip@2x-flip-vs-expired-vblank
> +kms_flip@flip-vs-absolute-wf_vblank
> +kms_flip@2x-flip-vs-absolute-wf_vblank
> +kms_flip@basic-flip-vs-wf_vblank
> +kms_flip@2x-flip-vs-wf_vblank
> +kms_flip@flip-vs-blocking-wf-vblank
> +kms_flip@2x-flip-vs-blocking-wf-vblank
> +kms_flip@flip-vs-modeset-vs-hang
> +kms_flip@2x-flip-vs-modeset-vs-hang
> +kms_flip@flip-vs-panning-vs-hang
> +kms_flip@2x-flip-vs-panning-vs-hang
> +kms_flip@flip-vs-dpms-off-vs-modeset
> +kms_flip@2x-flip-vs-dpms-off-vs-modeset
> +kms_flip@single-buffer-flip-vs-dpms-off-vs-modeset
> +kms_flip@2x-single-buffer-flip-vs-dpms-off-vs-modeset
> +kms_flip@dpms-off-confusion
> +kms_flip@nonexisting-fb
> +kms_flip@2x-nonexisting-fb
> +kms_flip@dpms-vs-vblank-race
> +kms_flip@2x-dpms-vs-vblank-race
> +kms_flip@modeset-vs-vblank-race
> +kms_flip@2x-modeset-vs-vblank-race
> +kms_flip@bo-too-big
> +kms_flip@flip-vs-suspend
> +kms_flip@2x-flip-vs-suspend
> +kms_flip@wf_vblank-ts-check-interruptible
> +kms_flip@2x-wf_vblank-ts-check-interruptible
> +kms_flip@absolute-wf_vblank-interruptible
> +kms_flip@2x-absolute-wf_vblank-interruptible
> +kms_flip@blocking-absolute-wf_vblank-interruptible
> +kms_flip@2x-blocking-absolute-wf_vblank-interruptible
> +kms_flip@plain-flip-interruptible
> +kms_flip@2x-plain-flip-interruptible
> +kms_flip@flip-vs-fences-interruptible
> +kms_flip@2x-flip-vs-fences-interruptible
> +kms_flip@plain-flip-ts-check-interruptible
> +kms_flip@2x-plain-flip-ts-check-interruptible
> +kms_flip@plain-flip-fb-recreate-interruptible
> +kms_flip@2x-plain-flip-fb-recreate-interruptible
> +kms_flip@flip-vs-rmfb-interruptible
> +kms_flip@2x-flip-vs-rmfb-interruptible
> +kms_flip@flip-vs-panning-interruptible
> +kms_flip@2x-flip-vs-panning-interruptible
> +kms_flip@flip-vs-expired-vblank-interruptible
> +kms_flip@2x-flip-vs-expired-vblank-interruptible
> +kms_flip@flip-vs-absolute-wf_vblank-interruptible
> +kms_flip@2x-flip-vs-absolute-wf_vblank-interruptible
> +kms_flip@flip-vs-wf_vblank-interruptible
> +kms_flip@2x-flip-vs-wf_vblank-interruptible
> +kms_flip@flip-vs-dpms-off-vs-modeset-interruptible
> +kms_flip@2x-flip-vs-dpms-off-vs-modeset-interruptible
> +kms_flip@single-buffer-flip-vs-dpms-off-vs-modeset-interruptible
> +kms_flip@2x-single-buffer-flip-vs-dpms-off-vs-modeset-interruptible
> +kms_flip@dpms-off-confusion-interruptible
> +kms_flip@nonexisting-fb-interruptible
> +kms_flip@2x-nonexisting-fb-interruptible
> +kms_flip@dpms-vs-vblank-race-interruptible
> +kms_flip@2x-dpms-vs-vblank-race-interruptible
> +kms_flip@modeset-vs-vblank-race-interruptible
> +kms_flip@2x-modeset-vs-vblank-race-interruptible
> +kms_flip@bo-too-big-interruptible
> +kms_flip@flip-vs-suspend-interruptible
> +kms_flip@2x-flip-vs-suspend-interruptible
> +kms_flip_event_leak@basic
> +kms_flip_scaled_crc@flip-32bpp-ytile-to-64bpp-ytile-downscaling
> +kms_flip_scaled_crc@flip-32bpp-yftile-to-64bpp-yftile-downscaling
> +kms_flip_scaled_crc@flip-32bpp-xtile-to-64bpp-xtile-downscaling
> +kms_flip_scaled_crc@flip-32bpp-4tile-to-64bpp-4tile-downscaling
> +kms_flip_scaled_crc@flip-32bpp-linear-to-64bpp-linear-downscaling
> +kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytile-downscaling
> +kms_flip_scaled_crc@flip-64bpp-yftile-to-32bpp-yftile-downscaling
> +kms_flip_scaled_crc@flip-64bpp-xtile-to-32bpp-xtile-downscaling
> +kms_flip_scaled_crc@flip-64bpp-4tile-to-32bpp-4tile-downscaling
> +kms_flip_scaled_crc@flip-64bpp-linear-to-32bpp-linear-downscaling
> +kms_flip_scaled_crc@flip-64bpp-ytile-to-16bpp-ytile-downscaling
> +kms_flip_scaled_crc@flip-64bpp-yftile-to-16bpp-yftile-downscaling
> +kms_flip_scaled_crc@flip-64bpp-xtile-to-16bpp-xtile-downscaling
> +kms_flip_scaled_crc@flip-64bpp-4tile-to-16bpp-4tile-downscaling
> +kms_flip_scaled_crc@flip-64bpp-linear-to-16bpp-linear-downscaling
> +kms_flip_scaled_crc@flip-32bpp-ytileccs-to-64bpp-ytile-downscaling
> +kms_flip_scaled_crc@flip-32bpp-yftileccs-to-64bpp-yftile-downscaling
> +kms_flip_scaled_crc@flip-32bpp-ytile-to-32bpp-ytilegen12rcccs-downscaling
> +kms_flip_scaled_crc@flip-32bpp-4tile-to-32bpp-4tiledg2rcccs-downscaling
> +kms_flip_scaled_crc@flip-32bpp-ytile-to-32bpp-ytileccs-downscaling
> +kms_flip_scaled_crc@flip-32bpp-yftile-to-32bpp-yftileccs-downscaling
> +kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytilercccs-downscaling
> +kms_flip_scaled_crc@flip-64bpp-4tile-to-32bpp-4tiledg2rcccs-downscaling
> +kms_flip_scaled_crc@flip-32bpp-ytile-to-64bpp-ytile-upscaling
> +kms_flip_scaled_crc@flip-32bpp-yftile-to-64bpp-yftile-upscaling
> +kms_flip_scaled_crc@flip-32bpp-xtile-to-64bpp-xtile-upscaling
> +kms_flip_scaled_crc@flip-32bpp-4tile-to-64bpp-4tile-upscaling
> +kms_flip_scaled_crc@flip-32bpp-linear-to-64bpp-linear-upscaling
> +kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytile-upscaling
> +kms_flip_scaled_crc@flip-64bpp-yftile-to-32bpp-yftile-upscaling
> +kms_flip_scaled_crc@flip-64bpp-xtile-to-32bpp-xtile-upscaling
> +kms_flip_scaled_crc@flip-64bpp-4tile-to-32bpp-4tile-upscaling
> +kms_flip_scaled_crc@flip-64bpp-linear-to-32bpp-linear-upscaling
> +kms_flip_scaled_crc@flip-64bpp-ytile-to-16bpp-ytile-upscaling
> +kms_flip_scaled_crc@flip-64bpp-yftile-to-16bpp-yftile-upscaling
> +kms_flip_scaled_crc@flip-64bpp-xtile-to-16bpp-xtile-upscaling
> +kms_flip_scaled_crc@flip-64bpp-4tile-to-16bpp-4tile-upscaling
> +kms_flip_scaled_crc@flip-64bpp-linear-to-16bpp-linear-upscaling
> +kms_flip_scaled_crc@flip-32bpp-ytileccs-to-64bpp-ytile-upscaling
> +kms_flip_scaled_crc@flip-32bpp-yftileccs-to-64bpp-yftile-upscaling
> +kms_flip_scaled_crc@flip-32bpp-ytile-to-32bpp-ytilegen12rcccs-upscaling
> +kms_flip_scaled_crc@flip-32bpp-4tile-to-32bpp-4tiledg2rcccs-upscaling
> +kms_flip_scaled_crc@flip-32bpp-ytile-to-32bpp-ytileccs-upscaling
> +kms_flip_scaled_crc@flip-32bpp-yftile-to-32bpp-yftileccs-upscaling
> +kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytilegen12rcccs-upscaling
> +kms_flip_scaled_crc@flip-64bpp-4tile-to-32bpp-4tiledg2rcccs-upscaling
> +kms_force_connector_basic@force-load-detect
> +kms_force_connector_basic@force-connector-state
> +kms_force_connector_basic@force-edid
> +kms_force_connector_basic@prune-stale-modes
> +kms_frontbuffer_tracking@fbc-1p-rte
> +kms_frontbuffer_tracking@fbc-2p-rte
> +kms_frontbuffer_tracking@psr-1p-rte
> +kms_frontbuffer_tracking@psr-2p-rte
> +kms_frontbuffer_tracking@fbcpsr-1p-rte
> +kms_frontbuffer_tracking@fbcpsr-2p-rte
> +kms_frontbuffer_tracking@drrs-1p-rte
> +kms_frontbuffer_tracking@drrs-2p-rte
> +kms_frontbuffer_tracking@fbcdrrs-1p-rte
> +kms_frontbuffer_tracking@fbcdrrs-2p-rte
> +kms_frontbuffer_tracking@psrdrrs-1p-rte
> +kms_frontbuffer_tracking@psrdrrs-2p-rte
> +kms_frontbuffer_tracking@fbcpsrdrrs-1p-rte
> +kms_frontbuffer_tracking@fbcpsrdrrs-2p-rte
> +kms_frontbuffer_tracking@fbc-1p-primscrn-pri-indfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@fbc-1p-primscrn-pri-indfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@fbc-1p-primscrn-pri-indfb-draw-mmap-wc
> +kms_frontbuffer_tracking@fbc-1p-primscrn-pri-indfb-draw-pwrite
> +kms_frontbuffer_tracking@fbc-1p-primscrn-pri-indfb-draw-blt
> +kms_frontbuffer_tracking@fbc-1p-primscrn-pri-indfb-draw-render
> +kms_frontbuffer_tracking@fbc-1p-primscrn-pri-shrfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@fbc-1p-primscrn-pri-shrfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@fbc-1p-primscrn-pri-shrfb-draw-mmap-wc
> +kms_frontbuffer_tracking@fbc-1p-primscrn-pri-shrfb-draw-pwrite
> +kms_frontbuffer_tracking@fbc-1p-primscrn-pri-shrfb-draw-blt
> +kms_frontbuffer_tracking@fbc-1p-primscrn-pri-shrfb-draw-render
> +kms_frontbuffer_tracking@fbc-1p-primscrn-cur-indfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@fbc-1p-primscrn-cur-indfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@fbc-1p-primscrn-cur-indfb-draw-mmap-wc
> +kms_frontbuffer_tracking@fbc-1p-primscrn-cur-indfb-draw-pwrite
> +kms_frontbuffer_tracking@fbc-1p-primscrn-cur-indfb-draw-blt
> +kms_frontbuffer_tracking@fbc-1p-primscrn-cur-indfb-draw-render
> +kms_frontbuffer_tracking@fbc-1p-primscrn-spr-indfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@fbc-1p-primscrn-spr-indfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@fbc-1p-primscrn-spr-indfb-draw-mmap-wc
> +kms_frontbuffer_tracking@fbc-1p-primscrn-spr-indfb-draw-pwrite
> +kms_frontbuffer_tracking@fbc-1p-primscrn-spr-indfb-draw-blt
> +kms_frontbuffer_tracking@fbc-1p-primscrn-spr-indfb-draw-render
> +kms_frontbuffer_tracking@fbc-1p-offscren-pri-indfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@fbc-1p-offscren-pri-indfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@fbc-1p-offscren-pri-indfb-draw-mmap-wc
> +kms_frontbuffer_tracking@fbc-1p-offscren-pri-indfb-draw-pwrite
> +kms_frontbuffer_tracking@fbc-1p-offscren-pri-indfb-draw-blt
> +kms_frontbuffer_tracking@fbc-1p-offscren-pri-indfb-draw-render
> +kms_frontbuffer_tracking@fbc-1p-offscren-pri-shrfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@fbc-1p-offscren-pri-shrfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@fbc-1p-offscren-pri-shrfb-draw-mmap-wc
> +kms_frontbuffer_tracking@fbc-1p-offscren-pri-shrfb-draw-pwrite
> +kms_frontbuffer_tracking@fbc-1p-offscren-pri-shrfb-draw-blt
> +kms_frontbuffer_tracking@fbc-1p-offscren-pri-shrfb-draw-render
> +kms_frontbuffer_tracking@fbc-2p-primscrn-pri-indfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@fbc-2p-primscrn-pri-indfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@fbc-2p-primscrn-pri-indfb-draw-mmap-wc
> +kms_frontbuffer_tracking@fbc-2p-primscrn-pri-indfb-draw-pwrite
> +kms_frontbuffer_tracking@fbc-2p-primscrn-pri-indfb-draw-blt
> +kms_frontbuffer_tracking@fbc-2p-primscrn-pri-indfb-draw-render
> +kms_frontbuffer_tracking@fbc-2p-primscrn-pri-shrfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@fbc-2p-primscrn-pri-shrfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@fbc-2p-primscrn-pri-shrfb-draw-mmap-wc
> +kms_frontbuffer_tracking@fbc-2p-primscrn-pri-shrfb-draw-pwrite
> +kms_frontbuffer_tracking@fbc-2p-primscrn-pri-shrfb-draw-blt
> +kms_frontbuffer_tracking@fbc-2p-primscrn-pri-shrfb-draw-render
> +kms_frontbuffer_tracking@fbc-2p-primscrn-cur-indfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@fbc-2p-primscrn-cur-indfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@fbc-2p-primscrn-cur-indfb-draw-mmap-wc
> +kms_frontbuffer_tracking@fbc-2p-primscrn-cur-indfb-draw-pwrite
> +kms_frontbuffer_tracking@fbc-2p-primscrn-cur-indfb-draw-blt
> +kms_frontbuffer_tracking@fbc-2p-primscrn-cur-indfb-draw-render
> +kms_frontbuffer_tracking@fbc-2p-primscrn-spr-indfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@fbc-2p-primscrn-spr-indfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@fbc-2p-primscrn-spr-indfb-draw-mmap-wc
> +kms_frontbuffer_tracking@fbc-2p-primscrn-spr-indfb-draw-pwrite
> +kms_frontbuffer_tracking@fbc-2p-primscrn-spr-indfb-draw-blt
> +kms_frontbuffer_tracking@fbc-2p-primscrn-spr-indfb-draw-render
> +kms_frontbuffer_tracking@fbc-2p-scndscrn-pri-indfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@fbc-2p-scndscrn-pri-indfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@fbc-2p-scndscrn-pri-indfb-draw-mmap-wc
> +kms_frontbuffer_tracking@fbc-2p-scndscrn-pri-indfb-draw-pwrite
> +kms_frontbuffer_tracking@fbc-2p-scndscrn-pri-indfb-draw-blt
> +kms_frontbuffer_tracking@fbc-2p-scndscrn-pri-indfb-draw-render
> +kms_frontbuffer_tracking@fbc-2p-scndscrn-pri-shrfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@fbc-2p-scndscrn-pri-shrfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@fbc-2p-scndscrn-pri-shrfb-draw-mmap-wc
> +kms_frontbuffer_tracking@fbc-2p-scndscrn-pri-shrfb-draw-pwrite
> +kms_frontbuffer_tracking@fbc-2p-scndscrn-pri-shrfb-draw-blt
> +kms_frontbuffer_tracking@fbc-2p-scndscrn-pri-shrfb-draw-render
> +kms_frontbuffer_tracking@fbc-2p-scndscrn-cur-indfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@fbc-2p-scndscrn-cur-indfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@fbc-2p-scndscrn-cur-indfb-draw-mmap-wc
> +kms_frontbuffer_tracking@fbc-2p-scndscrn-cur-indfb-draw-pwrite
> +kms_frontbuffer_tracking@fbc-2p-scndscrn-cur-indfb-draw-blt
> +kms_frontbuffer_tracking@fbc-2p-scndscrn-cur-indfb-draw-render
> +kms_frontbuffer_tracking@fbc-2p-scndscrn-spr-indfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@fbc-2p-scndscrn-spr-indfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@fbc-2p-scndscrn-spr-indfb-draw-mmap-wc
> +kms_frontbuffer_tracking@fbc-2p-scndscrn-spr-indfb-draw-pwrite
> +kms_frontbuffer_tracking@fbc-2p-scndscrn-spr-indfb-draw-blt
> +kms_frontbuffer_tracking@fbc-2p-scndscrn-spr-indfb-draw-render
> +kms_frontbuffer_tracking@psr-1p-primscrn-pri-indfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@psr-1p-primscrn-pri-indfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@psr-1p-primscrn-pri-indfb-draw-mmap-wc
> +kms_frontbuffer_tracking@psr-1p-primscrn-pri-indfb-draw-pwrite
> +kms_frontbuffer_tracking@psr-1p-primscrn-pri-indfb-draw-blt
> +kms_frontbuffer_tracking@psr-1p-primscrn-pri-indfb-draw-render
> +kms_frontbuffer_tracking@psr-1p-primscrn-pri-shrfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@psr-1p-primscrn-pri-shrfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@psr-1p-primscrn-pri-shrfb-draw-mmap-wc
> +kms_frontbuffer_tracking@psr-1p-primscrn-pri-shrfb-draw-pwrite
> +kms_frontbuffer_tracking@psr-1p-primscrn-pri-shrfb-draw-blt
> +kms_frontbuffer_tracking@psr-1p-primscrn-pri-shrfb-draw-render
> +kms_frontbuffer_tracking@psr-1p-primscrn-cur-indfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@psr-1p-primscrn-cur-indfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@psr-1p-primscrn-cur-indfb-draw-mmap-wc
> +kms_frontbuffer_tracking@psr-1p-primscrn-cur-indfb-draw-pwrite
> +kms_frontbuffer_tracking@psr-1p-primscrn-cur-indfb-draw-blt
> +kms_frontbuffer_tracking@psr-1p-primscrn-cur-indfb-draw-render
> +kms_frontbuffer_tracking@psr-1p-primscrn-spr-indfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@psr-1p-primscrn-spr-indfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@psr-1p-primscrn-spr-indfb-draw-mmap-wc
> +kms_frontbuffer_tracking@psr-1p-primscrn-spr-indfb-draw-pwrite
> +kms_frontbuffer_tracking@psr-1p-primscrn-spr-indfb-draw-blt
> +kms_frontbuffer_tracking@psr-1p-primscrn-spr-indfb-draw-render
> +kms_frontbuffer_tracking@psr-1p-offscren-pri-indfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@psr-1p-offscren-pri-indfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@psr-1p-offscren-pri-indfb-draw-mmap-wc
> +kms_frontbuffer_tracking@psr-1p-offscren-pri-indfb-draw-pwrite
> +kms_frontbuffer_tracking@psr-1p-offscren-pri-indfb-draw-blt
> +kms_frontbuffer_tracking@psr-1p-offscren-pri-indfb-draw-render
> +kms_frontbuffer_tracking@psr-1p-offscren-pri-shrfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@psr-1p-offscren-pri-shrfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@psr-1p-offscren-pri-shrfb-draw-mmap-wc
> +kms_frontbuffer_tracking@psr-1p-offscren-pri-shrfb-draw-pwrite
> +kms_frontbuffer_tracking@psr-1p-offscren-pri-shrfb-draw-blt
> +kms_frontbuffer_tracking@psr-1p-offscren-pri-shrfb-draw-render
> +kms_frontbuffer_tracking@psr-2p-primscrn-pri-indfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@psr-2p-primscrn-pri-indfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@psr-2p-primscrn-pri-indfb-draw-mmap-wc
> +kms_frontbuffer_tracking@psr-2p-primscrn-pri-indfb-draw-pwrite
> +kms_frontbuffer_tracking@psr-2p-primscrn-pri-indfb-draw-blt
> +kms_frontbuffer_tracking@psr-2p-primscrn-pri-indfb-draw-render
> +kms_frontbuffer_tracking@psr-2p-primscrn-pri-shrfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@psr-2p-primscrn-pri-shrfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@psr-2p-primscrn-pri-shrfb-draw-mmap-wc
> +kms_frontbuffer_tracking@psr-2p-primscrn-pri-shrfb-draw-pwrite
> +kms_frontbuffer_tracking@psr-2p-primscrn-pri-shrfb-draw-blt
> +kms_frontbuffer_tracking@psr-2p-primscrn-pri-shrfb-draw-render
> +kms_frontbuffer_tracking@psr-2p-primscrn-cur-indfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@psr-2p-primscrn-cur-indfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@psr-2p-primscrn-cur-indfb-draw-mmap-wc
> +kms_frontbuffer_tracking@psr-2p-primscrn-cur-indfb-draw-pwrite
> +kms_frontbuffer_tracking@psr-2p-primscrn-cur-indfb-draw-blt
> +kms_frontbuffer_tracking@psr-2p-primscrn-cur-indfb-draw-render
> +kms_frontbuffer_tracking@psr-2p-primscrn-spr-indfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@psr-2p-primscrn-spr-indfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@psr-2p-primscrn-spr-indfb-draw-mmap-wc
> +kms_frontbuffer_tracking@psr-2p-primscrn-spr-indfb-draw-pwrite
> +kms_frontbuffer_tracking@psr-2p-primscrn-spr-indfb-draw-blt
> +kms_frontbuffer_tracking@psr-2p-primscrn-spr-indfb-draw-render
> +kms_frontbuffer_tracking@psr-2p-scndscrn-pri-indfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@psr-2p-scndscrn-pri-indfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@psr-2p-scndscrn-pri-indfb-draw-mmap-wc
> +kms_frontbuffer_tracking@psr-2p-scndscrn-pri-indfb-draw-pwrite
> +kms_frontbuffer_tracking@psr-2p-scndscrn-pri-indfb-draw-blt
> +kms_frontbuffer_tracking@psr-2p-scndscrn-pri-indfb-draw-render
> +kms_frontbuffer_tracking@psr-2p-scndscrn-pri-shrfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@psr-2p-scndscrn-pri-shrfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@psr-2p-scndscrn-pri-shrfb-draw-mmap-wc
> +kms_frontbuffer_tracking@psr-2p-scndscrn-pri-shrfb-draw-pwrite
> +kms_frontbuffer_tracking@psr-2p-scndscrn-pri-shrfb-draw-blt
> +kms_frontbuffer_tracking@psr-2p-scndscrn-pri-shrfb-draw-render
> +kms_frontbuffer_tracking@psr-2p-scndscrn-cur-indfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@psr-2p-scndscrn-cur-indfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@psr-2p-scndscrn-cur-indfb-draw-mmap-wc
> +kms_frontbuffer_tracking@psr-2p-scndscrn-cur-indfb-draw-pwrite
> +kms_frontbuffer_tracking@psr-2p-scndscrn-cur-indfb-draw-blt
> +kms_frontbuffer_tracking@psr-2p-scndscrn-cur-indfb-draw-render
> +kms_frontbuffer_tracking@psr-2p-scndscrn-spr-indfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@psr-2p-scndscrn-spr-indfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@psr-2p-scndscrn-spr-indfb-draw-mmap-wc
> +kms_frontbuffer_tracking@psr-2p-scndscrn-spr-indfb-draw-pwrite
> +kms_frontbuffer_tracking@psr-2p-scndscrn-spr-indfb-draw-blt
> +kms_frontbuffer_tracking@psr-2p-scndscrn-spr-indfb-draw-render
> +kms_frontbuffer_tracking@fbcpsr-1p-primscrn-pri-indfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@fbcpsr-1p-primscrn-pri-indfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@fbcpsr-1p-primscrn-pri-indfb-draw-mmap-wc
> +kms_frontbuffer_tracking@fbcpsr-1p-primscrn-pri-indfb-draw-pwrite
> +kms_frontbuffer_tracking@fbcpsr-1p-primscrn-pri-indfb-draw-blt
> +kms_frontbuffer_tracking@fbcpsr-1p-primscrn-pri-indfb-draw-render
> +kms_frontbuffer_tracking@fbcpsr-1p-primscrn-pri-shrfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@fbcpsr-1p-primscrn-pri-shrfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@fbcpsr-1p-primscrn-pri-shrfb-draw-mmap-wc
> +kms_frontbuffer_tracking@fbcpsr-1p-primscrn-pri-shrfb-draw-pwrite
> +kms_frontbuffer_tracking@fbcpsr-1p-primscrn-pri-shrfb-draw-blt
> +kms_frontbuffer_tracking@fbcpsr-1p-primscrn-pri-shrfb-draw-render
> +kms_frontbuffer_tracking@fbcpsr-1p-primscrn-cur-indfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@fbcpsr-1p-primscrn-cur-indfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@fbcpsr-1p-primscrn-cur-indfb-draw-mmap-wc
> +kms_frontbuffer_tracking@fbcpsr-1p-primscrn-cur-indfb-draw-pwrite
> +kms_frontbuffer_tracking@fbcpsr-1p-primscrn-cur-indfb-draw-blt
> +kms_frontbuffer_tracking@fbcpsr-1p-primscrn-cur-indfb-draw-render
> +kms_frontbuffer_tracking@fbcpsr-1p-primscrn-spr-indfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@fbcpsr-1p-primscrn-spr-indfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@fbcpsr-1p-primscrn-spr-indfb-draw-mmap-wc
> +kms_frontbuffer_tracking@fbcpsr-1p-primscrn-spr-indfb-draw-pwrite
> +kms_frontbuffer_tracking@fbcpsr-1p-primscrn-spr-indfb-draw-blt
> +kms_frontbuffer_tracking@fbcpsr-1p-primscrn-spr-indfb-draw-render
> +kms_frontbuffer_tracking@fbcpsr-1p-offscren-pri-indfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@fbcpsr-1p-offscren-pri-indfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@fbcpsr-1p-offscren-pri-indfb-draw-mmap-wc
> +kms_frontbuffer_tracking@fbcpsr-1p-offscren-pri-indfb-draw-pwrite
> +kms_frontbuffer_tracking@fbcpsr-1p-offscren-pri-indfb-draw-blt
> +kms_frontbuffer_tracking@fbcpsr-1p-offscren-pri-indfb-draw-render
> +kms_frontbuffer_tracking@fbcpsr-1p-offscren-pri-shrfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@fbcpsr-1p-offscren-pri-shrfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@fbcpsr-1p-offscren-pri-shrfb-draw-mmap-wc
> +kms_frontbuffer_tracking@fbcpsr-1p-offscren-pri-shrfb-draw-pwrite
> +kms_frontbuffer_tracking@fbcpsr-1p-offscren-pri-shrfb-draw-blt
> +kms_frontbuffer_tracking@fbcpsr-1p-offscren-pri-shrfb-draw-render
> +kms_frontbuffer_tracking@fbcpsr-2p-primscrn-pri-indfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@fbcpsr-2p-primscrn-pri-indfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@fbcpsr-2p-primscrn-pri-indfb-draw-mmap-wc
> +kms_frontbuffer_tracking@fbcpsr-2p-primscrn-pri-indfb-draw-pwrite
> +kms_frontbuffer_tracking@fbcpsr-2p-primscrn-pri-indfb-draw-blt
> +kms_frontbuffer_tracking@fbcpsr-2p-primscrn-pri-indfb-draw-render
> +kms_frontbuffer_tracking@fbcpsr-2p-primscrn-pri-shrfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@fbcpsr-2p-primscrn-pri-shrfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@fbcpsr-2p-primscrn-pri-shrfb-draw-mmap-wc
> +kms_frontbuffer_tracking@fbcpsr-2p-primscrn-pri-shrfb-draw-pwrite
> +kms_frontbuffer_tracking@fbcpsr-2p-primscrn-pri-shrfb-draw-blt
> +kms_frontbuffer_tracking@fbcpsr-2p-primscrn-pri-shrfb-draw-render
> +kms_frontbuffer_tracking@fbcpsr-2p-primscrn-cur-indfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@fbcpsr-2p-primscrn-cur-indfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@fbcpsr-2p-primscrn-cur-indfb-draw-mmap-wc
> +kms_frontbuffer_tracking@fbcpsr-2p-primscrn-cur-indfb-draw-pwrite
> +kms_frontbuffer_tracking@fbcpsr-2p-primscrn-cur-indfb-draw-blt
> +kms_frontbuffer_tracking@fbcpsr-2p-primscrn-cur-indfb-draw-render
> +kms_frontbuffer_tracking@fbcpsr-2p-primscrn-spr-indfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@fbcpsr-2p-primscrn-spr-indfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@fbcpsr-2p-primscrn-spr-indfb-draw-mmap-wc
> +kms_frontbuffer_tracking@fbcpsr-2p-primscrn-spr-indfb-draw-pwrite
> +kms_frontbuffer_tracking@fbcpsr-2p-primscrn-spr-indfb-draw-blt
> +kms_frontbuffer_tracking@fbcpsr-2p-primscrn-spr-indfb-draw-render
> +kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-pri-indfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-pri-indfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-pri-indfb-draw-mmap-wc
> +kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-pri-indfb-draw-pwrite
> +kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-pri-indfb-draw-blt
> +kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-pri-indfb-draw-render
> +kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-pri-shrfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-pri-shrfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-pri-shrfb-draw-mmap-wc
> +kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-pri-shrfb-draw-pwrite
> +kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-pri-shrfb-draw-blt
> +kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-pri-shrfb-draw-render
> +kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-cur-indfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-cur-indfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-cur-indfb-draw-mmap-wc
> +kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-cur-indfb-draw-pwrite
> +kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-cur-indfb-draw-blt
> +kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-cur-indfb-draw-render
> +kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-spr-indfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-spr-indfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-spr-indfb-draw-mmap-wc
> +kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-spr-indfb-draw-pwrite
> +kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-spr-indfb-draw-blt
> +kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-spr-indfb-draw-render
> +kms_frontbuffer_tracking@drrs-1p-primscrn-pri-indfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@drrs-1p-primscrn-pri-indfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@drrs-1p-primscrn-pri-indfb-draw-mmap-wc
> +kms_frontbuffer_tracking@drrs-1p-primscrn-pri-indfb-draw-pwrite
> +kms_frontbuffer_tracking@drrs-1p-primscrn-pri-indfb-draw-blt
> +kms_frontbuffer_tracking@drrs-1p-primscrn-pri-indfb-draw-render
> +kms_frontbuffer_tracking@drrs-1p-primscrn-pri-shrfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@drrs-1p-primscrn-pri-shrfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@drrs-1p-primscrn-pri-shrfb-draw-mmap-wc
> +kms_frontbuffer_tracking@drrs-1p-primscrn-pri-shrfb-draw-pwrite
> +kms_frontbuffer_tracking@drrs-1p-primscrn-pri-shrfb-draw-blt
> +kms_frontbuffer_tracking@drrs-1p-primscrn-pri-shrfb-draw-render
> +kms_frontbuffer_tracking@drrs-1p-primscrn-cur-indfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@drrs-1p-primscrn-cur-indfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@drrs-1p-primscrn-cur-indfb-draw-mmap-wc
> +kms_frontbuffer_tracking@drrs-1p-primscrn-cur-indfb-draw-pwrite
> +kms_frontbuffer_tracking@drrs-1p-primscrn-cur-indfb-draw-blt
> +kms_frontbuffer_tracking@drrs-1p-primscrn-cur-indfb-draw-render
> +kms_frontbuffer_tracking@drrs-1p-primscrn-spr-indfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@drrs-1p-primscrn-spr-indfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@drrs-1p-primscrn-spr-indfb-draw-mmap-wc
> +kms_frontbuffer_tracking@drrs-1p-primscrn-spr-indfb-draw-pwrite
> +kms_frontbuffer_tracking@drrs-1p-primscrn-spr-indfb-draw-blt
> +kms_frontbuffer_tracking@drrs-1p-primscrn-spr-indfb-draw-render
> +kms_frontbuffer_tracking@drrs-1p-offscren-pri-indfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@drrs-1p-offscren-pri-indfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@drrs-1p-offscren-pri-indfb-draw-mmap-wc
> +kms_frontbuffer_tracking@drrs-1p-offscren-pri-indfb-draw-pwrite
> +kms_frontbuffer_tracking@drrs-1p-offscren-pri-indfb-draw-blt
> +kms_frontbuffer_tracking@drrs-1p-offscren-pri-indfb-draw-render
> +kms_frontbuffer_tracking@drrs-1p-offscren-pri-shrfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@drrs-1p-offscren-pri-shrfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@drrs-1p-offscren-pri-shrfb-draw-mmap-wc
> +kms_frontbuffer_tracking@drrs-1p-offscren-pri-shrfb-draw-pwrite
> +kms_frontbuffer_tracking@drrs-1p-offscren-pri-shrfb-draw-blt
> +kms_frontbuffer_tracking@drrs-1p-offscren-pri-shrfb-draw-render
> +kms_frontbuffer_tracking@drrs-2p-primscrn-pri-indfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@drrs-2p-primscrn-pri-indfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@drrs-2p-primscrn-pri-indfb-draw-mmap-wc
> +kms_frontbuffer_tracking@drrs-2p-primscrn-pri-indfb-draw-pwrite
> +kms_frontbuffer_tracking@drrs-2p-primscrn-pri-indfb-draw-blt
> +kms_frontbuffer_tracking@drrs-2p-primscrn-pri-indfb-draw-render
> +kms_frontbuffer_tracking@drrs-2p-primscrn-pri-shrfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@drrs-2p-primscrn-pri-shrfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@drrs-2p-primscrn-pri-shrfb-draw-mmap-wc
> +kms_frontbuffer_tracking@drrs-2p-primscrn-pri-shrfb-draw-pwrite
> +kms_frontbuffer_tracking@drrs-2p-primscrn-pri-shrfb-draw-blt
> +kms_frontbuffer_tracking@drrs-2p-primscrn-pri-shrfb-draw-render
> +kms_frontbuffer_tracking@drrs-2p-primscrn-cur-indfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@drrs-2p-primscrn-cur-indfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@drrs-2p-primscrn-cur-indfb-draw-mmap-wc
> +kms_frontbuffer_tracking@drrs-2p-primscrn-cur-indfb-draw-pwrite
> +kms_frontbuffer_tracking@drrs-2p-primscrn-cur-indfb-draw-blt
> +kms_frontbuffer_tracking@drrs-2p-primscrn-cur-indfb-draw-render
> +kms_frontbuffer_tracking@drrs-2p-primscrn-spr-indfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@drrs-2p-primscrn-spr-indfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@drrs-2p-primscrn-spr-indfb-draw-mmap-wc
> +kms_frontbuffer_tracking@drrs-2p-primscrn-spr-indfb-draw-pwrite
> +kms_frontbuffer_tracking@drrs-2p-primscrn-spr-indfb-draw-blt
> +kms_frontbuffer_tracking@drrs-2p-primscrn-spr-indfb-draw-render
> +kms_frontbuffer_tracking@drrs-2p-scndscrn-pri-indfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@drrs-2p-scndscrn-pri-indfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@drrs-2p-scndscrn-pri-indfb-draw-mmap-wc
> +kms_frontbuffer_tracking@drrs-2p-scndscrn-pri-indfb-draw-pwrite
> +kms_frontbuffer_tracking@drrs-2p-scndscrn-pri-indfb-draw-blt
> +kms_frontbuffer_tracking@drrs-2p-scndscrn-pri-indfb-draw-render
> +kms_frontbuffer_tracking@drrs-2p-scndscrn-pri-shrfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@drrs-2p-scndscrn-pri-shrfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@drrs-2p-scndscrn-pri-shrfb-draw-mmap-wc
> +kms_frontbuffer_tracking@drrs-2p-scndscrn-pri-shrfb-draw-pwrite
> +kms_frontbuffer_tracking@drrs-2p-scndscrn-pri-shrfb-draw-blt
> +kms_frontbuffer_tracking@drrs-2p-scndscrn-pri-shrfb-draw-render
> +kms_frontbuffer_tracking@drrs-2p-scndscrn-cur-indfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@drrs-2p-scndscrn-cur-indfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@drrs-2p-scndscrn-cur-indfb-draw-mmap-wc
> +kms_frontbuffer_tracking@drrs-2p-scndscrn-cur-indfb-draw-pwrite
> +kms_frontbuffer_tracking@drrs-2p-scndscrn-cur-indfb-draw-blt
> +kms_frontbuffer_tracking@drrs-2p-scndscrn-cur-indfb-draw-render
> +kms_frontbuffer_tracking@drrs-2p-scndscrn-spr-indfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@drrs-2p-scndscrn-spr-indfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@drrs-2p-scndscrn-spr-indfb-draw-mmap-wc
> +kms_frontbuffer_tracking@drrs-2p-scndscrn-spr-indfb-draw-pwrite
> +kms_frontbuffer_tracking@drrs-2p-scndscrn-spr-indfb-draw-blt
> +kms_frontbuffer_tracking@drrs-2p-scndscrn-spr-indfb-draw-render
> +kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-pri-indfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-pri-indfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-pri-indfb-draw-mmap-wc
> +kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-pri-indfb-draw-pwrite
> +kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-pri-indfb-draw-blt
> +kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-pri-indfb-draw-render
> +kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-pri-shrfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-pri-shrfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-pri-shrfb-draw-mmap-wc
> +kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-pri-shrfb-draw-pwrite
> +kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-pri-shrfb-draw-blt
> +kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-pri-shrfb-draw-render
> +kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-cur-indfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-cur-indfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-cur-indfb-draw-mmap-wc
> +kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-cur-indfb-draw-pwrite
> +kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-cur-indfb-draw-blt
> +kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-cur-indfb-draw-render
> +kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-spr-indfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-spr-indfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-spr-indfb-draw-mmap-wc
> +kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-spr-indfb-draw-pwrite
> +kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-spr-indfb-draw-blt
> +kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-spr-indfb-draw-render
> +kms_frontbuffer_tracking@fbcdrrs-1p-offscren-pri-indfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@fbcdrrs-1p-offscren-pri-indfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@fbcdrrs-1p-offscren-pri-indfb-draw-mmap-wc
> +kms_frontbuffer_tracking@fbcdrrs-1p-offscren-pri-indfb-draw-pwrite
> +kms_frontbuffer_tracking@fbcdrrs-1p-offscren-pri-indfb-draw-blt
> +kms_frontbuffer_tracking@fbcdrrs-1p-offscren-pri-indfb-draw-render
> +kms_frontbuffer_tracking@fbcdrrs-1p-offscren-pri-shrfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@fbcdrrs-1p-offscren-pri-shrfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@fbcdrrs-1p-offscren-pri-shrfb-draw-mmap-wc
> +kms_frontbuffer_tracking@fbcdrrs-1p-offscren-pri-shrfb-draw-pwrite
> +kms_frontbuffer_tracking@fbcdrrs-1p-offscren-pri-shrfb-draw-blt
> +kms_frontbuffer_tracking@fbcdrrs-1p-offscren-pri-shrfb-draw-render
> +kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-pri-indfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-pri-indfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-pri-indfb-draw-mmap-wc
> +kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-pri-indfb-draw-pwrite
> +kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-pri-indfb-draw-blt
> +kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-pri-indfb-draw-render
> +kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-pri-shrfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-pri-shrfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-pri-shrfb-draw-mmap-wc
> +kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-pri-shrfb-draw-pwrite
> +kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-pri-shrfb-draw-blt
> +kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-pri-shrfb-draw-render
> +kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-cur-indfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-cur-indfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-cur-indfb-draw-mmap-wc
> +kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-cur-indfb-draw-pwrite
> +kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-cur-indfb-draw-blt
> +kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-cur-indfb-draw-render
> +kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-spr-indfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-spr-indfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-spr-indfb-draw-mmap-wc
> +kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-spr-indfb-draw-pwrite
> +kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-spr-indfb-draw-blt
> +kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-spr-indfb-draw-render
> +kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-pri-indfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-pri-indfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-pri-indfb-draw-mmap-wc
> +kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-pri-indfb-draw-pwrite
> +kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-pri-indfb-draw-blt
> +kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-pri-indfb-draw-render
> +kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-pri-shrfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-pri-shrfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-pri-shrfb-draw-mmap-wc
> +kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-pri-shrfb-draw-pwrite
> +kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-pri-shrfb-draw-blt
> +kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-pri-shrfb-draw-render
> +kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-cur-indfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-cur-indfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-cur-indfb-draw-mmap-wc
> +kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-cur-indfb-draw-pwrite
> +kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-cur-indfb-draw-blt
> +kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-cur-indfb-draw-render
> +kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-spr-indfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-spr-indfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-spr-indfb-draw-mmap-wc
> +kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-spr-indfb-draw-pwrite
> +kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-spr-indfb-draw-blt
> +kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-spr-indfb-draw-render
> +kms_frontbuffer_tracking@psrdrrs-1p-primscrn-pri-indfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@psrdrrs-1p-primscrn-pri-indfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@psrdrrs-1p-primscrn-pri-indfb-draw-mmap-wc
> +kms_frontbuffer_tracking@psrdrrs-1p-primscrn-pri-indfb-draw-pwrite
> +kms_frontbuffer_tracking@psrdrrs-1p-primscrn-pri-indfb-draw-blt
> +kms_frontbuffer_tracking@psrdrrs-1p-primscrn-pri-indfb-draw-render
> +kms_frontbuffer_tracking@psrdrrs-1p-primscrn-pri-shrfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@psrdrrs-1p-primscrn-pri-shrfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@psrdrrs-1p-primscrn-pri-shrfb-draw-mmap-wc
> +kms_frontbuffer_tracking@psrdrrs-1p-primscrn-pri-shrfb-draw-pwrite
> +kms_frontbuffer_tracking@psrdrrs-1p-primscrn-pri-shrfb-draw-blt
> +kms_frontbuffer_tracking@psrdrrs-1p-primscrn-pri-shrfb-draw-render
> +kms_frontbuffer_tracking@psrdrrs-1p-primscrn-cur-indfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@psrdrrs-1p-primscrn-cur-indfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@psrdrrs-1p-primscrn-cur-indfb-draw-mmap-wc
> +kms_frontbuffer_tracking@psrdrrs-1p-primscrn-cur-indfb-draw-pwrite
> +kms_frontbuffer_tracking@psrdrrs-1p-primscrn-cur-indfb-draw-blt
> +kms_frontbuffer_tracking@psrdrrs-1p-primscrn-cur-indfb-draw-render
> +kms_frontbuffer_tracking@psrdrrs-1p-primscrn-spr-indfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@psrdrrs-1p-primscrn-spr-indfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@psrdrrs-1p-primscrn-spr-indfb-draw-mmap-wc
> +kms_frontbuffer_tracking@psrdrrs-1p-primscrn-spr-indfb-draw-pwrite
> +kms_frontbuffer_tracking@psrdrrs-1p-primscrn-spr-indfb-draw-blt
> +kms_frontbuffer_tracking@psrdrrs-1p-primscrn-spr-indfb-draw-render
> +kms_frontbuffer_tracking@psrdrrs-1p-offscren-pri-indfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@psrdrrs-1p-offscren-pri-indfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@psrdrrs-1p-offscren-pri-indfb-draw-mmap-wc
> +kms_frontbuffer_tracking@psrdrrs-1p-offscren-pri-indfb-draw-pwrite
> +kms_frontbuffer_tracking@psrdrrs-1p-offscren-pri-indfb-draw-blt
> +kms_frontbuffer_tracking@psrdrrs-1p-offscren-pri-indfb-draw-render
> +kms_frontbuffer_tracking@psrdrrs-1p-offscren-pri-shrfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@psrdrrs-1p-offscren-pri-shrfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@psrdrrs-1p-offscren-pri-shrfb-draw-mmap-wc
> +kms_frontbuffer_tracking@psrdrrs-1p-offscren-pri-shrfb-draw-pwrite
> +kms_frontbuffer_tracking@psrdrrs-1p-offscren-pri-shrfb-draw-blt
> +kms_frontbuffer_tracking@psrdrrs-1p-offscren-pri-shrfb-draw-render
> +kms_frontbuffer_tracking@psrdrrs-2p-primscrn-pri-indfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@psrdrrs-2p-primscrn-pri-indfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@psrdrrs-2p-primscrn-pri-indfb-draw-mmap-wc
> +kms_frontbuffer_tracking@psrdrrs-2p-primscrn-pri-indfb-draw-pwrite
> +kms_frontbuffer_tracking@psrdrrs-2p-primscrn-pri-indfb-draw-blt
> +kms_frontbuffer_tracking@psrdrrs-2p-primscrn-pri-indfb-draw-render
> +kms_frontbuffer_tracking@psrdrrs-2p-primscrn-pri-shrfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@psrdrrs-2p-primscrn-pri-shrfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@psrdrrs-2p-primscrn-pri-shrfb-draw-mmap-wc
> +kms_frontbuffer_tracking@psrdrrs-2p-primscrn-pri-shrfb-draw-pwrite
> +kms_frontbuffer_tracking@psrdrrs-2p-primscrn-pri-shrfb-draw-blt
> +kms_frontbuffer_tracking@psrdrrs-2p-primscrn-pri-shrfb-draw-render
> +kms_frontbuffer_tracking@psrdrrs-2p-primscrn-cur-indfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@psrdrrs-2p-primscrn-cur-indfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@psrdrrs-2p-primscrn-cur-indfb-draw-mmap-wc
> +kms_frontbuffer_tracking@psrdrrs-2p-primscrn-cur-indfb-draw-pwrite
> +kms_frontbuffer_tracking@psrdrrs-2p-primscrn-cur-indfb-draw-blt
> +kms_frontbuffer_tracking@psrdrrs-2p-primscrn-cur-indfb-draw-render
> +kms_frontbuffer_tracking@psrdrrs-2p-primscrn-spr-indfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@psrdrrs-2p-primscrn-spr-indfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@psrdrrs-2p-primscrn-spr-indfb-draw-mmap-wc
> +kms_frontbuffer_tracking@psrdrrs-2p-primscrn-spr-indfb-draw-pwrite
> +kms_frontbuffer_tracking@psrdrrs-2p-primscrn-spr-indfb-draw-blt
> +kms_frontbuffer_tracking@psrdrrs-2p-primscrn-spr-indfb-draw-render
> +kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-pri-indfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-pri-indfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-pri-indfb-draw-mmap-wc
> +kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-pri-indfb-draw-pwrite
> +kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-pri-indfb-draw-blt
> +kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-pri-indfb-draw-render
> +kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-pri-shrfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-pri-shrfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-pri-shrfb-draw-mmap-wc
> +kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-pri-shrfb-draw-pwrite
> +kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-pri-shrfb-draw-blt
> +kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-pri-shrfb-draw-render
> +kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-cur-indfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-cur-indfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-cur-indfb-draw-mmap-wc
> +kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-cur-indfb-draw-pwrite
> +kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-cur-indfb-draw-blt
> +kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-cur-indfb-draw-render
> +kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-spr-indfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-spr-indfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-spr-indfb-draw-mmap-wc
> +kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-spr-indfb-draw-pwrite
> +kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-spr-indfb-draw-blt
> +kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-spr-indfb-draw-render
> +kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-pri-indfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-pri-indfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-pri-indfb-draw-mmap-wc
> +kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-pri-indfb-draw-pwrite
> +kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-pri-indfb-draw-blt
> +kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-pri-indfb-draw-render
> +kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-pri-shrfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-pri-shrfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-pri-shrfb-draw-mmap-wc
> +kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-pri-shrfb-draw-pwrite
> +kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-pri-shrfb-draw-blt
> +kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-pri-shrfb-draw-render
> +kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-cur-indfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-cur-indfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-cur-indfb-draw-mmap-wc
> +kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-cur-indfb-draw-pwrite
> +kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-cur-indfb-draw-blt
> +kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-cur-indfb-draw-render
> +kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-spr-indfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-spr-indfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-spr-indfb-draw-mmap-wc
> +kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-spr-indfb-draw-pwrite
> +kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-spr-indfb-draw-blt
> +kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-spr-indfb-draw-render
> +kms_frontbuffer_tracking@fbcpsrdrrs-1p-offscren-pri-indfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@fbcpsrdrrs-1p-offscren-pri-indfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@fbcpsrdrrs-1p-offscren-pri-indfb-draw-mmap-wc
> +kms_frontbuffer_tracking@fbcpsrdrrs-1p-offscren-pri-indfb-draw-pwrite
> +kms_frontbuffer_tracking@fbcpsrdrrs-1p-offscren-pri-indfb-draw-blt
> +kms_frontbuffer_tracking@fbcpsrdrrs-1p-offscren-pri-indfb-draw-render
> +kms_frontbuffer_tracking@fbcpsrdrrs-1p-offscren-pri-shrfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@fbcpsrdrrs-1p-offscren-pri-shrfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@fbcpsrdrrs-1p-offscren-pri-shrfb-draw-mmap-wc
> +kms_frontbuffer_tracking@fbcpsrdrrs-1p-offscren-pri-shrfb-draw-pwrite
> +kms_frontbuffer_tracking@fbcpsrdrrs-1p-offscren-pri-shrfb-draw-blt
> +kms_frontbuffer_tracking@fbcpsrdrrs-1p-offscren-pri-shrfb-draw-render
> +kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-pri-indfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-pri-indfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-pri-indfb-draw-mmap-wc
> +kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-pri-indfb-draw-pwrite
> +kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-pri-indfb-draw-blt
> +kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-pri-indfb-draw-render
> +kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-pri-shrfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-pri-shrfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-pri-shrfb-draw-mmap-wc
> +kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-pri-shrfb-draw-pwrite
> +kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-pri-shrfb-draw-blt
> +kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-pri-shrfb-draw-render
> +kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-cur-indfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-cur-indfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-cur-indfb-draw-mmap-wc
> +kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-cur-indfb-draw-pwrite
> +kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-cur-indfb-draw-blt
> +kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-cur-indfb-draw-render
> +kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-spr-indfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-spr-indfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-spr-indfb-draw-mmap-wc
> +kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-spr-indfb-draw-pwrite
> +kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-spr-indfb-draw-blt
> +kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-spr-indfb-draw-render
> +kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-pri-indfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-pri-indfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-pri-indfb-draw-mmap-wc
> +kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-pri-indfb-draw-pwrite
> +kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-pri-indfb-draw-blt
> +kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-pri-indfb-draw-render
> +kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-pri-shrfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-pri-shrfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-pri-shrfb-draw-mmap-wc
> +kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-pri-shrfb-draw-pwrite
> +kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-pri-shrfb-draw-blt
> +kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-pri-shrfb-draw-render
> +kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-cur-indfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-cur-indfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-cur-indfb-draw-mmap-wc
> +kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-cur-indfb-draw-pwrite
> +kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-cur-indfb-draw-blt
> +kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-cur-indfb-draw-render
> +kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-spr-indfb-draw-mmap-cpu
> +kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-spr-indfb-draw-mmap-gtt
> +kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-spr-indfb-draw-mmap-wc
> +kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-spr-indfb-draw-pwrite
> +kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-spr-indfb-draw-blt
> +kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-spr-indfb-draw-render
> +kms_frontbuffer_tracking@fbc-1p-primscrn-indfb-pgflip-blt
> +kms_frontbuffer_tracking@fbc-1p-primscrn-indfb-msflip-blt
> +kms_frontbuffer_tracking@fbc-1p-primscrn-indfb-plflip-blt
> +kms_frontbuffer_tracking@fbc-1p-primscrn-shrfb-pgflip-blt
> +kms_frontbuffer_tracking@fbc-1p-primscrn-shrfb-msflip-blt
> +kms_frontbuffer_tracking@fbc-1p-primscrn-shrfb-plflip-blt
> +kms_frontbuffer_tracking@fbc-2p-primscrn-indfb-pgflip-blt
> +kms_frontbuffer_tracking@fbc-2p-primscrn-indfb-msflip-blt
> +kms_frontbuffer_tracking@fbc-2p-primscrn-indfb-plflip-blt
> +kms_frontbuffer_tracking@fbc-2p-primscrn-shrfb-pgflip-blt
> +kms_frontbuffer_tracking@fbc-2p-primscrn-shrfb-msflip-blt
> +kms_frontbuffer_tracking@fbc-2p-primscrn-shrfb-plflip-blt
> +kms_frontbuffer_tracking@fbc-2p-scndscrn-indfb-pgflip-blt
> +kms_frontbuffer_tracking@fbc-2p-scndscrn-indfb-msflip-blt
> +kms_frontbuffer_tracking@fbc-2p-scndscrn-indfb-plflip-blt
> +kms_frontbuffer_tracking@fbc-2p-scndscrn-shrfb-pgflip-blt
> +kms_frontbuffer_tracking@fbc-2p-scndscrn-shrfb-msflip-blt
> +kms_frontbuffer_tracking@fbc-2p-scndscrn-shrfb-plflip-blt
> +kms_frontbuffer_tracking@psr-1p-primscrn-indfb-pgflip-blt
> +kms_frontbuffer_tracking@psr-1p-primscrn-indfb-msflip-blt
> +kms_frontbuffer_tracking@psr-1p-primscrn-indfb-plflip-blt
> +kms_frontbuffer_tracking@psr-1p-primscrn-shrfb-pgflip-blt
> +kms_frontbuffer_tracking@psr-1p-primscrn-shrfb-msflip-blt
> +kms_frontbuffer_tracking@psr-1p-primscrn-shrfb-plflip-blt
> +kms_frontbuffer_tracking@psr-2p-primscrn-indfb-pgflip-blt
> +kms_frontbuffer_tracking@psr-2p-primscrn-indfb-msflip-blt
> +kms_frontbuffer_tracking@psr-2p-primscrn-indfb-plflip-blt
> +kms_frontbuffer_tracking@psr-2p-primscrn-shrfb-pgflip-blt
> +kms_frontbuffer_tracking@psr-2p-primscrn-shrfb-msflip-blt
> +kms_frontbuffer_tracking@psr-2p-primscrn-shrfb-plflip-blt
> +kms_frontbuffer_tracking@psr-2p-scndscrn-indfb-pgflip-blt
> +kms_frontbuffer_tracking@psr-2p-scndscrn-indfb-msflip-blt
> +kms_frontbuffer_tracking@psr-2p-scndscrn-indfb-plflip-blt
> +kms_frontbuffer_tracking@psr-2p-scndscrn-shrfb-pgflip-blt
> +kms_frontbuffer_tracking@psr-2p-scndscrn-shrfb-msflip-blt
> +kms_frontbuffer_tracking@psr-2p-scndscrn-shrfb-plflip-blt
> +kms_frontbuffer_tracking@fbcpsr-1p-primscrn-indfb-pgflip-blt
> +kms_frontbuffer_tracking@fbcpsr-1p-primscrn-indfb-msflip-blt
> +kms_frontbuffer_tracking@fbcpsr-1p-primscrn-indfb-plflip-blt
> +kms_frontbuffer_tracking@fbcpsr-1p-primscrn-shrfb-pgflip-blt
> +kms_frontbuffer_tracking@fbcpsr-1p-primscrn-shrfb-msflip-blt
> +kms_frontbuffer_tracking@fbcpsr-1p-primscrn-shrfb-plflip-blt
> +kms_frontbuffer_tracking@fbcpsr-2p-primscrn-indfb-pgflip-blt
> +kms_frontbuffer_tracking@fbcpsr-2p-primscrn-indfb-msflip-blt
> +kms_frontbuffer_tracking@fbcpsr-2p-primscrn-indfb-plflip-blt
> +kms_frontbuffer_tracking@fbcpsr-2p-primscrn-shrfb-pgflip-blt
> +kms_frontbuffer_tracking@fbcpsr-2p-primscrn-shrfb-msflip-blt
> +kms_frontbuffer_tracking@fbcpsr-2p-primscrn-shrfb-plflip-blt
> +kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-indfb-pgflip-blt
> +kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-indfb-msflip-blt
> +kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-indfb-plflip-blt
> +kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-shrfb-pgflip-blt
> +kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-shrfb-msflip-blt
> +kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-shrfb-plflip-blt
> +kms_frontbuffer_tracking@drrs-1p-primscrn-indfb-pgflip-blt
> +kms_frontbuffer_tracking@drrs-1p-primscrn-indfb-msflip-blt
> +kms_frontbuffer_tracking@drrs-1p-primscrn-indfb-plflip-blt
> +kms_frontbuffer_tracking@drrs-1p-primscrn-shrfb-pgflip-blt
> +kms_frontbuffer_tracking@drrs-1p-primscrn-shrfb-msflip-blt
> +kms_frontbuffer_tracking@drrs-1p-primscrn-shrfb-plflip-blt
> +kms_frontbuffer_tracking@drrs-2p-primscrn-indfb-pgflip-blt
> +kms_frontbuffer_tracking@drrs-2p-primscrn-indfb-msflip-blt
> +kms_frontbuffer_tracking@drrs-2p-primscrn-indfb-plflip-blt
> +kms_frontbuffer_tracking@drrs-2p-primscrn-shrfb-pgflip-blt
> +kms_frontbuffer_tracking@drrs-2p-primscrn-shrfb-msflip-blt
> +kms_frontbuffer_tracking@drrs-2p-primscrn-shrfb-plflip-blt
> +kms_frontbuffer_tracking@drrs-2p-scndscrn-indfb-pgflip-blt
> +kms_frontbuffer_tracking@drrs-2p-scndscrn-indfb-msflip-blt
> +kms_frontbuffer_tracking@drrs-2p-scndscrn-indfb-plflip-blt
> +kms_frontbuffer_tracking@drrs-2p-scndscrn-shrfb-pgflip-blt
> +kms_frontbuffer_tracking@drrs-2p-scndscrn-shrfb-msflip-blt
> +kms_frontbuffer_tracking@drrs-2p-scndscrn-shrfb-plflip-blt
> +kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-indfb-pgflip-blt
> +kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-indfb-msflip-blt
> +kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-indfb-plflip-blt
> +kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-shrfb-pgflip-blt
> +kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-shrfb-msflip-blt
> +kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-shrfb-plflip-blt
> +kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-indfb-pgflip-blt
> +kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-indfb-msflip-blt
> +kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-indfb-plflip-blt
> +kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-shrfb-pgflip-blt
> +kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-shrfb-msflip-blt
> +kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-shrfb-plflip-blt
> +kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-indfb-pgflip-blt
> +kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-indfb-msflip-blt
> +kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-indfb-plflip-blt
> +kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-shrfb-pgflip-blt
> +kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-shrfb-msflip-blt
> +kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-shrfb-plflip-blt
> +kms_frontbuffer_tracking@psrdrrs-1p-primscrn-indfb-pgflip-blt
> +kms_frontbuffer_tracking@psrdrrs-1p-primscrn-indfb-msflip-blt
> +kms_frontbuffer_tracking@psrdrrs-1p-primscrn-indfb-plflip-blt
> +kms_frontbuffer_tracking@psrdrrs-1p-primscrn-shrfb-pgflip-blt
> +kms_frontbuffer_tracking@psrdrrs-1p-primscrn-shrfb-msflip-blt
> +kms_frontbuffer_tracking@psrdrrs-1p-primscrn-shrfb-plflip-blt
> +kms_frontbuffer_tracking@psrdrrs-2p-primscrn-indfb-pgflip-blt
> +kms_frontbuffer_tracking@psrdrrs-2p-primscrn-indfb-msflip-blt
> +kms_frontbuffer_tracking@psrdrrs-2p-primscrn-indfb-plflip-blt
> +kms_frontbuffer_tracking@psrdrrs-2p-primscrn-shrfb-pgflip-blt
> +kms_frontbuffer_tracking@psrdrrs-2p-primscrn-shrfb-msflip-blt
> +kms_frontbuffer_tracking@psrdrrs-2p-primscrn-shrfb-plflip-blt
> +kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-indfb-pgflip-blt
> +kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-indfb-msflip-blt
> +kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-indfb-plflip-blt
> +kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-shrfb-pgflip-blt
> +kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-shrfb-msflip-blt
> +kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-shrfb-plflip-blt
> +kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-indfb-pgflip-blt
> +kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-indfb-msflip-blt
> +kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-indfb-plflip-blt
> +kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-shrfb-pgflip-blt
> +kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-shrfb-msflip-blt
> +kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-shrfb-plflip-blt
> +kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-indfb-pgflip-blt
> +kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-indfb-msflip-blt
> +kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-indfb-plflip-blt
> +kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-shrfb-pgflip-blt
> +kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-shrfb-msflip-blt
> +kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-shrfb-plflip-blt
> +kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-indfb-pgflip-blt
> +kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-indfb-msflip-blt
> +kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-indfb-plflip-blt
> +kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-shrfb-pgflip-blt
> +kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-shrfb-msflip-blt
> +kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-shrfb-plflip-blt
> +kms_frontbuffer_tracking@fbc-1p-indfb-fliptrack-mmap-gtt
> +kms_frontbuffer_tracking@fbc-1p-shrfb-fliptrack-mmap-gtt
> +kms_frontbuffer_tracking@fbc-2p-indfb-fliptrack-mmap-gtt
> +kms_frontbuffer_tracking@fbc-2p-shrfb-fliptrack-mmap-gtt
> +kms_frontbuffer_tracking@fbcpsr-1p-indfb-fliptrack-mmap-gtt
> +kms_frontbuffer_tracking@fbcpsr-1p-shrfb-fliptrack-mmap-gtt
> +kms_frontbuffer_tracking@fbcpsr-2p-indfb-fliptrack-mmap-gtt
> +kms_frontbuffer_tracking@fbcpsr-2p-shrfb-fliptrack-mmap-gtt
> +kms_frontbuffer_tracking@fbcdrrs-1p-indfb-fliptrack-mmap-gtt
> +kms_frontbuffer_tracking@fbcdrrs-1p-shrfb-fliptrack-mmap-gtt
> +kms_frontbuffer_tracking@fbcdrrs-2p-indfb-fliptrack-mmap-gtt
> +kms_frontbuffer_tracking@fbcdrrs-2p-shrfb-fliptrack-mmap-gtt
> +kms_frontbuffer_tracking@fbcpsrdrrs-1p-indfb-fliptrack-mmap-gtt
> +kms_frontbuffer_tracking@fbcpsrdrrs-1p-shrfb-fliptrack-mmap-gtt
> +kms_frontbuffer_tracking@fbcpsrdrrs-2p-indfb-fliptrack-mmap-gtt
> +kms_frontbuffer_tracking@fbcpsrdrrs-2p-shrfb-fliptrack-mmap-gtt
> +kms_frontbuffer_tracking@fbc-1p-primscrn-cur-indfb-move
> +kms_frontbuffer_tracking@fbc-1p-primscrn-cur-indfb-onoff
> +kms_frontbuffer_tracking@fbc-1p-primscrn-spr-indfb-move
> +kms_frontbuffer_tracking@fbc-1p-primscrn-spr-indfb-onoff
> +kms_frontbuffer_tracking@fbc-2p-primscrn-cur-indfb-move
> +kms_frontbuffer_tracking@fbc-2p-primscrn-cur-indfb-onoff
> +kms_frontbuffer_tracking@fbc-2p-primscrn-spr-indfb-move
> +kms_frontbuffer_tracking@fbc-2p-primscrn-spr-indfb-onoff
> +kms_frontbuffer_tracking@fbc-2p-scndscrn-cur-indfb-move
> +kms_frontbuffer_tracking@fbc-2p-scndscrn-cur-indfb-onoff
> +kms_frontbuffer_tracking@fbc-2p-scndscrn-spr-indfb-move
> +kms_frontbuffer_tracking@fbc-2p-scndscrn-spr-indfb-onoff
> +kms_frontbuffer_tracking@psr-1p-primscrn-cur-indfb-move
> +kms_frontbuffer_tracking@psr-1p-primscrn-cur-indfb-onoff
> +kms_frontbuffer_tracking@psr-1p-primscrn-spr-indfb-move
> +kms_frontbuffer_tracking@psr-1p-primscrn-spr-indfb-onoff
> +kms_frontbuffer_tracking@psr-2p-primscrn-cur-indfb-move
> +kms_frontbuffer_tracking@psr-2p-primscrn-cur-indfb-onoff
> +kms_frontbuffer_tracking@psr-2p-primscrn-spr-indfb-move
> +kms_frontbuffer_tracking@psr-2p-primscrn-spr-indfb-onoff
> +kms_frontbuffer_tracking@psr-2p-scndscrn-cur-indfb-move
> +kms_frontbuffer_tracking@psr-2p-scndscrn-cur-indfb-onoff
> +kms_frontbuffer_tracking@psr-2p-scndscrn-spr-indfb-move
> +kms_frontbuffer_tracking@psr-2p-scndscrn-spr-indfb-onoff
> +kms_frontbuffer_tracking@fbcpsr-1p-primscrn-cur-indfb-move
> +kms_frontbuffer_tracking@fbcpsr-1p-primscrn-cur-indfb-onoff
> +kms_frontbuffer_tracking@fbcpsr-1p-primscrn-spr-indfb-move
> +kms_frontbuffer_tracking@fbcpsr-1p-primscrn-spr-indfb-onoff
> +kms_frontbuffer_tracking@fbcpsr-2p-primscrn-cur-indfb-move
> +kms_frontbuffer_tracking@fbcpsr-2p-primscrn-cur-indfb-onoff
> +kms_frontbuffer_tracking@fbcpsr-2p-primscrn-spr-indfb-move
> +kms_frontbuffer_tracking@fbcpsr-2p-primscrn-spr-indfb-onoff
> +kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-cur-indfb-move
> +kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-cur-indfb-onoff
> +kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-spr-indfb-move
> +kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-spr-indfb-onoff
> +kms_frontbuffer_tracking@drrs-1p-primscrn-cur-indfb-move
> +kms_frontbuffer_tracking@drrs-1p-primscrn-cur-indfb-onoff
> +kms_frontbuffer_tracking@drrs-1p-primscrn-spr-indfb-move
> +kms_frontbuffer_tracking@drrs-1p-primscrn-spr-indfb-onoff
> +kms_frontbuffer_tracking@drrs-2p-primscrn-cur-indfb-move
> +kms_frontbuffer_tracking@drrs-2p-primscrn-cur-indfb-onoff
> +kms_frontbuffer_tracking@drrs-2p-primscrn-spr-indfb-move
> +kms_frontbuffer_tracking@drrs-2p-primscrn-spr-indfb-onoff
> +kms_frontbuffer_tracking@drrs-2p-scndscrn-cur-indfb-move
> +kms_frontbuffer_tracking@drrs-2p-scndscrn-cur-indfb-onoff
> +kms_frontbuffer_tracking@drrs-2p-scndscrn-spr-indfb-move
> +kms_frontbuffer_tracking@drrs-2p-scndscrn-spr-indfb-onoff
> +kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-cur-indfb-move
> +kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-cur-indfb-onoff
> +kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-spr-indfb-move
> +kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-spr-indfb-onoff
> +kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-cur-indfb-move
> +kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-cur-indfb-onoff
> +kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-spr-indfb-move
> +kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-spr-indfb-onoff
> +kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-cur-indfb-move
> +kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-cur-indfb-onoff
> +kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-spr-indfb-move
> +kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-spr-indfb-onoff
> +kms_frontbuffer_tracking@psrdrrs-1p-primscrn-cur-indfb-move
> +kms_frontbuffer_tracking@psrdrrs-1p-primscrn-cur-indfb-onoff
> +kms_frontbuffer_tracking@psrdrrs-1p-primscrn-spr-indfb-move
> +kms_frontbuffer_tracking@psrdrrs-1p-primscrn-spr-indfb-onoff
> +kms_frontbuffer_tracking@psrdrrs-2p-primscrn-cur-indfb-move
> +kms_frontbuffer_tracking@psrdrrs-2p-primscrn-cur-indfb-onoff
> +kms_frontbuffer_tracking@psrdrrs-2p-primscrn-spr-indfb-move
> +kms_frontbuffer_tracking@psrdrrs-2p-primscrn-spr-indfb-onoff
> +kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-cur-indfb-move
> +kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-cur-indfb-onoff
> +kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-spr-indfb-move
> +kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-spr-indfb-onoff
> +kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-cur-indfb-move
> +kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-cur-indfb-onoff
> +kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-spr-indfb-move
> +kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-spr-indfb-onoff
> +kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-cur-indfb-move
> +kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-cur-indfb-onoff
> +kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-spr-indfb-move
> +kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-spr-indfb-onoff
> +kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-cur-indfb-move
> +kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-cur-indfb-onoff
> +kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-spr-indfb-move
> +kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-spr-indfb-onoff
> +kms_frontbuffer_tracking@fbc-1p-primscrn-spr-indfb-fullscreen
> +kms_frontbuffer_tracking@fbc-2p-primscrn-spr-indfb-fullscreen
> +kms_frontbuffer_tracking@fbc-2p-scndscrn-spr-indfb-fullscreen
> +kms_frontbuffer_tracking@psr-1p-primscrn-spr-indfb-fullscreen
> +kms_frontbuffer_tracking@psr-2p-primscrn-spr-indfb-fullscreen
> +kms_frontbuffer_tracking@psr-2p-scndscrn-spr-indfb-fullscreen
> +kms_frontbuffer_tracking@fbcpsr-1p-primscrn-spr-indfb-fullscreen
> +kms_frontbuffer_tracking@fbcpsr-2p-primscrn-spr-indfb-fullscreen
> +kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-spr-indfb-fullscreen
> +kms_frontbuffer_tracking@drrs-1p-primscrn-spr-indfb-fullscreen
> +kms_frontbuffer_tracking@drrs-2p-primscrn-spr-indfb-fullscreen
> +kms_frontbuffer_tracking@drrs-2p-scndscrn-spr-indfb-fullscreen
> +kms_frontbuffer_tracking@fbcdrrs-1p-primscrn-spr-indfb-fullscreen
> +kms_frontbuffer_tracking@fbcdrrs-2p-primscrn-spr-indfb-fullscreen
> +kms_frontbuffer_tracking@fbcdrrs-2p-scndscrn-spr-indfb-fullscreen
> +kms_frontbuffer_tracking@psrdrrs-1p-primscrn-spr-indfb-fullscreen
> +kms_frontbuffer_tracking@psrdrrs-2p-primscrn-spr-indfb-fullscreen
> +kms_frontbuffer_tracking@psrdrrs-2p-scndscrn-spr-indfb-fullscreen
> +kms_frontbuffer_tracking@fbcpsrdrrs-1p-primscrn-spr-indfb-fullscreen
> +kms_frontbuffer_tracking@fbcpsrdrrs-2p-primscrn-spr-indfb-fullscreen
> +kms_frontbuffer_tracking@fbcpsrdrrs-2p-scndscrn-spr-indfb-fullscreen
> +kms_frontbuffer_tracking@fbc-1p-pri-indfb-multidraw
> +kms_frontbuffer_tracking@fbc-2p-pri-indfb-multidraw
> +kms_frontbuffer_tracking@psr-1p-pri-indfb-multidraw
> +kms_frontbuffer_tracking@psr-2p-pri-indfb-multidraw
> +kms_frontbuffer_tracking@fbcpsr-1p-pri-indfb-multidraw
> +kms_frontbuffer_tracking@fbcpsr-2p-pri-indfb-multidraw
> +kms_frontbuffer_tracking@drrs-1p-pri-indfb-multidraw
> +kms_frontbuffer_tracking@drrs-2p-pri-indfb-multidraw
> +kms_frontbuffer_tracking@fbcdrrs-1p-pri-indfb-multidraw
> +kms_frontbuffer_tracking@fbcdrrs-2p-pri-indfb-multidraw
> +kms_frontbuffer_tracking@psrdrrs-1p-pri-indfb-multidraw
> +kms_frontbuffer_tracking@psrdrrs-2p-pri-indfb-multidraw
> +kms_frontbuffer_tracking@fbcpsrdrrs-1p-pri-indfb-multidraw
> +kms_frontbuffer_tracking@fbcpsrdrrs-2p-pri-indfb-multidraw
> +kms_frontbuffer_tracking@fbc-farfromfence-mmap-gtt
> +kms_frontbuffer_tracking@psr-farfromfence-mmap-gtt
> +kms_frontbuffer_tracking@fbcpsr-farfromfence-mmap-gtt
> +kms_frontbuffer_tracking@drrs-farfromfence-mmap-gtt
> +kms_frontbuffer_tracking@fbcdrrs-farfromfence-mmap-gtt
> +kms_frontbuffer_tracking@psrdrrs-farfromfence-mmap-gtt
> +kms_frontbuffer_tracking@fbcpsrdrrs-farfromfence-mmap-gtt
> +kms_frontbuffer_tracking@fbc-rgb565-draw-mmap-cpu
> +kms_frontbuffer_tracking@fbc-rgb101010-draw-mmap-cpu
> +kms_frontbuffer_tracking@fbc-rgb565-draw-mmap-gtt
> +kms_frontbuffer_tracking@fbc-rgb101010-draw-mmap-gtt
> +kms_frontbuffer_tracking@fbc-rgb565-draw-mmap-wc
> +kms_frontbuffer_tracking@fbc-rgb101010-draw-mmap-wc
> +kms_frontbuffer_tracking@fbc-rgb565-draw-pwrite
> +kms_frontbuffer_tracking@fbc-rgb101010-draw-pwrite
> +kms_frontbuffer_tracking@fbc-rgb565-draw-blt
> +kms_frontbuffer_tracking@fbc-rgb101010-draw-blt
> +kms_frontbuffer_tracking@fbc-rgb565-draw-render
> +kms_frontbuffer_tracking@fbc-rgb101010-draw-render
> +kms_frontbuffer_tracking@psr-rgb565-draw-mmap-cpu
> +kms_frontbuffer_tracking@psr-rgb101010-draw-mmap-cpu
> +kms_frontbuffer_tracking@psr-rgb565-draw-mmap-gtt
> +kms_frontbuffer_tracking@psr-rgb101010-draw-mmap-gtt
> +kms_frontbuffer_tracking@psr-rgb565-draw-mmap-wc
> +kms_frontbuffer_tracking@psr-rgb101010-draw-mmap-wc
> +kms_frontbuffer_tracking@psr-rgb565-draw-pwrite
> +kms_frontbuffer_tracking@psr-rgb101010-draw-pwrite
> +kms_frontbuffer_tracking@psr-rgb565-draw-blt
> +kms_frontbuffer_tracking@psr-rgb101010-draw-blt
> +kms_frontbuffer_tracking@psr-rgb565-draw-render
> +kms_frontbuffer_tracking@psr-rgb101010-draw-render
> +kms_frontbuffer_tracking@fbcpsr-rgb565-draw-mmap-cpu
> +kms_frontbuffer_tracking@fbcpsr-rgb101010-draw-mmap-cpu
> +kms_frontbuffer_tracking@fbcpsr-rgb565-draw-mmap-gtt
> +kms_frontbuffer_tracking@fbcpsr-rgb101010-draw-mmap-gtt
> +kms_frontbuffer_tracking@fbcpsr-rgb565-draw-mmap-wc
> +kms_frontbuffer_tracking@fbcpsr-rgb101010-draw-mmap-wc
> +kms_frontbuffer_tracking@fbcpsr-rgb565-draw-pwrite
> +kms_frontbuffer_tracking@fbcpsr-rgb101010-draw-pwrite
> +kms_frontbuffer_tracking@fbcpsr-rgb565-draw-blt
> +kms_frontbuffer_tracking@fbcpsr-rgb101010-draw-blt
> +kms_frontbuffer_tracking@fbcpsr-rgb565-draw-render
> +kms_frontbuffer_tracking@fbcpsr-rgb101010-draw-render
> +kms_frontbuffer_tracking@drrs-rgb565-draw-mmap-cpu
> +kms_frontbuffer_tracking@drrs-rgb101010-draw-mmap-cpu
> +kms_frontbuffer_tracking@drrs-rgb565-draw-mmap-gtt
> +kms_frontbuffer_tracking@drrs-rgb101010-draw-mmap-gtt
> +kms_frontbuffer_tracking@drrs-rgb565-draw-mmap-wc
> +kms_frontbuffer_tracking@drrs-rgb101010-draw-mmap-wc
> +kms_frontbuffer_tracking@drrs-rgb565-draw-pwrite
> +kms_frontbuffer_tracking@drrs-rgb101010-draw-pwrite
> +kms_frontbuffer_tracking@drrs-rgb565-draw-blt
> +kms_frontbuffer_tracking@drrs-rgb101010-draw-blt
> +kms_frontbuffer_tracking@drrs-rgb565-draw-render
> +kms_frontbuffer_tracking@drrs-rgb101010-draw-render
> +kms_frontbuffer_tracking@fbcdrrs-rgb565-draw-mmap-cpu
> +kms_frontbuffer_tracking@fbcdrrs-rgb101010-draw-mmap-cpu
> +kms_frontbuffer_tracking@fbcdrrs-rgb565-draw-mmap-gtt
> +kms_frontbuffer_tracking@fbcdrrs-rgb101010-draw-mmap-gtt
> +kms_frontbuffer_tracking@fbcdrrs-rgb565-draw-mmap-wc
> +kms_frontbuffer_tracking@fbcdrrs-rgb101010-draw-mmap-wc
> +kms_frontbuffer_tracking@fbcdrrs-rgb565-draw-pwrite
> +kms_frontbuffer_tracking@fbcdrrs-rgb101010-draw-pwrite
> +kms_frontbuffer_tracking@fbcdrrs-rgb565-draw-blt
> +kms_frontbuffer_tracking@fbcdrrs-rgb101010-draw-blt
> +kms_frontbuffer_tracking@fbcdrrs-rgb565-draw-render
> +kms_frontbuffer_tracking@fbcdrrs-rgb101010-draw-render
> +kms_frontbuffer_tracking@psrdrrs-rgb565-draw-mmap-cpu
> +kms_frontbuffer_tracking@psrdrrs-rgb101010-draw-mmap-cpu
> +kms_frontbuffer_tracking@psrdrrs-rgb565-draw-mmap-gtt
> +kms_frontbuffer_tracking@psrdrrs-rgb101010-draw-mmap-gtt
> +kms_frontbuffer_tracking@psrdrrs-rgb565-draw-mmap-wc
> +kms_frontbuffer_tracking@psrdrrs-rgb101010-draw-mmap-wc
> +kms_frontbuffer_tracking@psrdrrs-rgb565-draw-pwrite
> +kms_frontbuffer_tracking@psrdrrs-rgb101010-draw-pwrite
> +kms_frontbuffer_tracking@psrdrrs-rgb565-draw-blt
> +kms_frontbuffer_tracking@psrdrrs-rgb101010-draw-blt
> +kms_frontbuffer_tracking@psrdrrs-rgb565-draw-render
> +kms_frontbuffer_tracking@psrdrrs-rgb101010-draw-render
> +kms_frontbuffer_tracking@fbcpsrdrrs-rgb565-draw-mmap-cpu
> +kms_frontbuffer_tracking@fbcpsrdrrs-rgb101010-draw-mmap-cpu
> +kms_frontbuffer_tracking@fbcpsrdrrs-rgb565-draw-mmap-gtt
> +kms_frontbuffer_tracking@fbcpsrdrrs-rgb101010-draw-mmap-gtt
> +kms_frontbuffer_tracking@fbcpsrdrrs-rgb565-draw-mmap-wc
> +kms_frontbuffer_tracking@fbcpsrdrrs-rgb101010-draw-mmap-wc
> +kms_frontbuffer_tracking@fbcpsrdrrs-rgb565-draw-pwrite
> +kms_frontbuffer_tracking@fbcpsrdrrs-rgb101010-draw-pwrite
> +kms_frontbuffer_tracking@fbcpsrdrrs-rgb565-draw-blt
> +kms_frontbuffer_tracking@fbcpsrdrrs-rgb101010-draw-blt
> +kms_frontbuffer_tracking@fbcpsrdrrs-rgb565-draw-render
> +kms_frontbuffer_tracking@fbcpsrdrrs-rgb101010-draw-render
> +kms_frontbuffer_tracking@fbc-indfb-scaledprimary
> +kms_frontbuffer_tracking@fbc-shrfb-scaledprimary
> +kms_frontbuffer_tracking@psr-indfb-scaledprimary
> +kms_frontbuffer_tracking@psr-shrfb-scaledprimary
> +kms_frontbuffer_tracking@fbcpsr-indfb-scaledprimary
> +kms_frontbuffer_tracking@fbcpsr-shrfb-scaledprimary
> +kms_frontbuffer_tracking@drrs-indfb-scaledprimary
> +kms_frontbuffer_tracking@drrs-shrfb-scaledprimary
> +kms_frontbuffer_tracking@fbcdrrs-indfb-scaledprimary
> +kms_frontbuffer_tracking@fbcdrrs-shrfb-scaledprimary
> +kms_frontbuffer_tracking@psrdrrs-indfb-scaledprimary
> +kms_frontbuffer_tracking@psrdrrs-shrfb-scaledprimary
> +kms_frontbuffer_tracking@fbcpsrdrrs-indfb-scaledprimary
> +kms_frontbuffer_tracking@fbcpsrdrrs-shrfb-scaledprimary
> +kms_frontbuffer_tracking@fbc-modesetfrombusy
> +kms_frontbuffer_tracking@fbc-badstride
> +kms_frontbuffer_tracking@fbc-stridechange
> +kms_frontbuffer_tracking@fbc-tiling-linear
> +kms_frontbuffer_tracking@fbc-tiling-y
> +kms_frontbuffer_tracking@fbc-tiling-4
> +kms_frontbuffer_tracking@fbc-suspend
> +kms_frontbuffer_tracking@psr-modesetfrombusy
> +kms_frontbuffer_tracking@psr-slowdraw
> +kms_frontbuffer_tracking@psr-suspend
> +kms_frontbuffer_tracking@fbcpsr-modesetfrombusy
> +kms_frontbuffer_tracking@fbcpsr-badstride
> +kms_frontbuffer_tracking@fbcpsr-stridechange
> +kms_frontbuffer_tracking@fbcpsr-tiling-linear
> +kms_frontbuffer_tracking@fbcpsr-tiling-y
> +kms_frontbuffer_tracking@fbcpsr-tiling-4
> +kms_frontbuffer_tracking@fbcpsr-slowdraw
> +kms_frontbuffer_tracking@fbcpsr-suspend
> +kms_frontbuffer_tracking@drrs-modesetfrombusy
> +kms_frontbuffer_tracking@drrs-slowdraw
> +kms_frontbuffer_tracking@drrs-suspend
> +kms_frontbuffer_tracking@fbcdrrs-modesetfrombusy
> +kms_frontbuffer_tracking@fbcdrrs-badstride
> +kms_frontbuffer_tracking@fbcdrrs-stridechange
> +kms_frontbuffer_tracking@fbcdrrs-tiling-linear
> +kms_frontbuffer_tracking@fbcdrrs-tiling-y
> +kms_frontbuffer_tracking@fbcdrrs-tiling-4
> +kms_frontbuffer_tracking@fbcdrrs-slowdraw
> +kms_frontbuffer_tracking@fbcdrrs-suspend
> +kms_frontbuffer_tracking@psrdrrs-modesetfrombusy
> +kms_frontbuffer_tracking@psrdrrs-slowdraw
> +kms_frontbuffer_tracking@psrdrrs-suspend
> +kms_frontbuffer_tracking@fbcpsrdrrs-modesetfrombusy
> +kms_frontbuffer_tracking@fbcpsrdrrs-badstride
> +kms_frontbuffer_tracking@fbcpsrdrrs-stridechange
> +kms_frontbuffer_tracking@fbcpsrdrrs-tiling-linear
> +kms_frontbuffer_tracking@fbcpsrdrrs-tiling-y
> +kms_frontbuffer_tracking@fbcpsrdrrs-tiling-4
> +kms_frontbuffer_tracking@fbcpsrdrrs-slowdraw
> +kms_frontbuffer_tracking@fbcpsrdrrs-suspend
> +kms_frontbuffer_tracking@basic
> +kms_getfb@getfb-handle-zero
> +kms_getfb@getfb-handle-valid
> +kms_getfb@getfb-handle-closed
> +kms_getfb@getfb-handle-not-fb
> +kms_getfb@getfb-addfb-different-handles
> +kms_getfb@getfb-repeated-different-handles
> +kms_getfb@getfb-reject-ccs
> +kms_getfb@getfb2-handle-zero
> +kms_getfb@getfb2-handle-closed
> +kms_getfb@getfb2-handle-not-fb
> +kms_getfb@getfb2-accept-ccs
> +kms_getfb@getfb2-into-addfb2
> +kms_getfb@getfb-handle-protection
> +kms_getfb@getfb2-handle-protection
> +kms_hdmi_inject@inject-4k
> +kms_hdmi_inject@inject-audio
> +kms_hdr@bpc-switch
> +kms_hdr@bpc-switch-dpms
> +kms_hdr@bpc-switch-suspend
> +kms_hdr@static-toggle
> +kms_hdr@static-toggle-dpms
> +kms_hdr@static-toggle-suspend
> +kms_hdr@static-swap
> +kms_hdr@invalid-metadata-sizes
> +kms_hdr@invalid-hdr
> +kms_invalid_mode@clock-too-high
> +kms_invalid_mode@zero-clock
> +kms_invalid_mode@int-max-clock
> +kms_invalid_mode@uint-max-clock
> +kms_invalid_mode@zero-hdisplay
> +kms_invalid_mode@zero-vdisplay
> +kms_invalid_mode@bad-hsync-start
> +kms_invalid_mode@bad-vsync-start
> +kms_invalid_mode@bad-hsync-end
> +kms_invalid_mode@bad-vsync-end
> +kms_invalid_mode@bad-htotal
> +kms_invalid_mode@bad-vtotal
> +kms_legacy_colorkey@basic
> +kms_legacy_colorkey@invalid-plane
> +kms_multipipe_modeset@basic-max-pipe-crc-check
> +kms_panel_fitting@legacy
> +kms_panel_fitting@atomic-fastset
> +kms_pipe_b_c_ivb@pipe-B-dpms-off-modeset-pipe-C
> +kms_pipe_b_c_ivb@pipe-B-double-modeset-then-modeset-pipe-C
> +kms_pipe_b_c_ivb@disable-pipe-B-enable-pipe-C
> +kms_pipe_b_c_ivb@from-pipe-C-to-B-with-3-lanes
> +kms_pipe_b_c_ivb@enable-pipe-C-while-B-has-3-lanes
> +kms_pipe_crc_basic@bad-source
> +kms_pipe_crc_basic@read-crc
> +kms_pipe_crc_basic@read-crc-frame-sequence
> +kms_pipe_crc_basic@nonblocking-crc
> +kms_pipe_crc_basic@nonblocking-crc-frame-sequence
> +kms_pipe_crc_basic@suspend-read-crc
> +kms_pipe_crc_basic@hang-read-crc
> +kms_pipe_crc_basic@disable-crc-after-crtc
> +kms_pipe_crc_basic@compare-crc-sanitycheck-xr24
> +kms_pipe_crc_basic@compare-crc-sanitycheck-nv12
> +kms_plane@pixel-format
> +kms_plane@pixel-format-source-clamping
> +kms_plane@plane-position-covered
> +kms_plane@plane-position-hole
> +kms_plane@plane-position-hole-dpms
> +kms_plane@plane-panning-top-left
> +kms_plane@plane-panning-bottom-right
> +kms_plane@plane-panning-bottom-right-suspend
> +kms_plane@invalid-pixel-format-settings
> +kms_plane_alpha_blend@alpha-basic
> +kms_plane_alpha_blend@alpha-7efc
> +kms_plane_alpha_blend@coverage-7efc
> +kms_plane_alpha_blend@coverage-vs-premult-vs-constant
> +kms_plane_alpha_blend@alpha-transparent-fb
> +kms_plane_alpha_blend@alpha-opaque-fb
> +kms_plane_alpha_blend@constant-alpha-min
> +kms_plane_alpha_blend@constant-alpha-mid
> +kms_plane_alpha_blend@constant-alpha-max
> +kms_plane_cursor@primary
> +kms_plane_cursor@overlay
> +kms_plane_cursor@viewport
> +kms_plane_lowres@tiling-none
> +kms_plane_lowres@tiling-x
> +kms_plane_lowres@tiling-y
> +kms_plane_lowres@tiling-yf
> +kms_plane_lowres@tiling-4
> +kms_plane_multiple@tiling-none
> +kms_plane_multiple@tiling-x
> +kms_plane_multiple@tiling-y
> +kms_plane_multiple@tiling-yf
> +kms_plane_multiple@tiling-4
> +kms_plane_scaling@plane-upscale-with-pixel-format-20x20
> +kms_plane_scaling@plane-upscale-with-pixel-format-factor-0-25
> +kms_plane_scaling@plane-downscale-with-pixel-format-factor-0-25
> +kms_plane_scaling@plane-downscale-with-pixel-format-factor-0-5
> +kms_plane_scaling@plane-downscale-with-pixel-format-factor-0-75
> +kms_plane_scaling@plane-scaler-with-pixel-format-unity-scaling
> +kms_plane_scaling@plane-upscale-with-rotation-20x20
> +kms_plane_scaling@plane-upscale-with-rotation-factor-0-25
> +kms_plane_scaling@plane-downscale-with-rotation-factor-0-25
> +kms_plane_scaling@plane-downscale-with-rotation-factor-0-5
> +kms_plane_scaling@plane-downscale-with-rotation-factor-0-75
> +kms_plane_scaling@plane-scaler-with-rotation-unity-scaling
> +kms_plane_scaling@plane-upscale-with-modifiers-20x20
> +kms_plane_scaling@plane-upscale-with-modifiers-factor-0-25
> +kms_plane_scaling@plane-downscale-with-modifiers-factor-0-25
> +kms_plane_scaling@plane-downscale-with-modifiers-factor-0-5
> +kms_plane_scaling@plane-downscale-with-modifiers-factor-0-75
> +kms_plane_scaling@plane-scaler-with-modifiers-unity-scaling
> +kms_plane_scaling@plane-scaler-with-clipping-clamping-pixel-formats
> +kms_plane_scaling@plane-scaler-with-clipping-clamping-rotation
> +kms_plane_scaling@plane-scaler-with-clipping-clamping-modifiers
> +kms_plane_scaling@planes-upscale-20x20
> +kms_plane_scaling@planes-upscale-factor-0-25
> +kms_plane_scaling@planes-scaler-unity-scaling
> +kms_plane_scaling@planes-downscale-factor-0-25
> +kms_plane_scaling@planes-downscale-factor-0-5
> +kms_plane_scaling@planes-downscale-factor-0-75
> +kms_plane_scaling@planes-upscale-20x20-downscale-factor-0-25
> +kms_plane_scaling@planes-upscale-20x20-downscale-factor-0-5
> +kms_plane_scaling@planes-upscale-20x20-downscale-factor-0-75
> +kms_plane_scaling@planes-upscale-factor-0-25-downscale-factor-0-25
> +kms_plane_scaling@planes-upscale-factor-0-25-downscale-factor-0-5
> +kms_plane_scaling@planes-upscale-factor-0-25-downscale-factor-0-75
> +kms_plane_scaling@planes-unity-scaling-downscale-factor-0-25
> +kms_plane_scaling@planes-unity-scaling-downscale-factor-0-5
> +kms_plane_scaling@planes-unity-scaling-downscale-factor-0-75
> +kms_plane_scaling@planes-downscale-factor-0-25-upscale-20x20
> +kms_plane_scaling@planes-downscale-factor-0-25-upscale-factor-0-25
> +kms_plane_scaling@planes-downscale-factor-0-25-unity-scaling
> +kms_plane_scaling@planes-downscale-factor-0-5-upscale-20x20
> +kms_plane_scaling@planes-downscale-factor-0-5-upscale-factor-0-25
> +kms_plane_scaling@planes-downscale-factor-0-5-unity-scaling
> +kms_plane_scaling@planes-downscale-factor-0-75-upscale-20x20
> +kms_plane_scaling@planes-downscale-factor-0-75-upscale-factor-0-25
> +kms_plane_scaling@planes-downscale-factor-0-75-unity-scaling
> +kms_plane_scaling@intel-max-src-size
> +kms_plane_scaling@invalid-num-scalers
> +kms_plane_scaling@invalid-parameters
> +kms_plane_scaling@2x-scaler-multi-pipe
> +kms_prime@basic-crc-hybrid
> +kms_prime@basic-modeset-hybrid
> +kms_prime@D3hot
> +kms_prime@basic-crc-vgem
> +kms_prop_blob@basic
> +kms_prop_blob@blob-prop-core
> +kms_prop_blob@blob-prop-validate
> +kms_prop_blob@blob-prop-lifetime
> +kms_prop_blob@blob-multiple
> +kms_prop_blob@invalid-get-prop-any
> +kms_prop_blob@invalid-get-prop
> +kms_prop_blob@invalid-set-prop-any
> +kms_prop_blob@invalid-set-prop
> +kms_properties@plane-properties-legacy
> +kms_properties@plane-properties-atomic
> +kms_properties@crtc-properties-legacy
> +kms_properties@crtc-properties-atomic
> +kms_properties@connector-properties-legacy
> +kms_properties@connector-properties-atomic
> +kms_properties@invalid-properties-legacy
> +kms_properties@invalid-properties-atomic
> +kms_properties@get_properties-sanity-atomic
> +kms_properties@get_properties-sanity-non-atomic
> +kms_psr@basic
> +kms_psr@no_drrs
> +kms_psr@primary_page_flip
> +kms_psr@primary_mmap_gtt
> +kms_psr@primary_mmap_cpu
> +kms_psr@primary_blt
> +kms_psr@primary_render
> +kms_psr@sprite_mmap_gtt
> +kms_psr@cursor_mmap_gtt
> +kms_psr@sprite_mmap_cpu
> +kms_psr@cursor_mmap_cpu
> +kms_psr@sprite_blt
> +kms_psr@cursor_blt
> +kms_psr@sprite_render
> +kms_psr@cursor_render
> +kms_psr@sprite_plane_move
> +kms_psr@cursor_plane_move
> +kms_psr@sprite_plane_onoff
> +kms_psr@cursor_plane_onoff
> +kms_psr@dpms
> +kms_psr@suspend
> +kms_psr@psr2_basic
> +kms_psr@psr2_no_drrs
> +kms_psr@psr2_primary_page_flip
> +kms_psr@psr2_primary_mmap_gtt
> +kms_psr@psr2_primary_mmap_cpu
> +kms_psr@psr2_primary_blt
> +kms_psr@psr2_primary_render
> +kms_psr@psr2_sprite_mmap_gtt
> +kms_psr@psr2_cursor_mmap_gtt
> +kms_psr@psr2_sprite_mmap_cpu
> +kms_psr@psr2_cursor_mmap_cpu
> +kms_psr@psr2_sprite_blt
> +kms_psr@psr2_cursor_blt
> +kms_psr@psr2_sprite_render
> +kms_psr@psr2_cursor_render
> +kms_psr@psr2_sprite_plane_move
> +kms_psr@psr2_cursor_plane_move
> +kms_psr@psr2_sprite_plane_onoff
> +kms_psr@psr2_cursor_plane_onoff
> +kms_psr@psr2_dpms
> +kms_psr@psr2_suspend
> +kms_psr2_sf@primary-plane-update-sf-dmg-area
> +kms_psr2_sf@primary-plane-update-sf-dmg-area-big-fb
> +kms_psr2_sf@overlay-plane-update-sf-dmg-area
> +kms_psr2_sf@cursor-plane-update-sf
> +kms_psr2_sf@cursor-plane-move-continuous-sf
> +kms_psr2_sf@cursor-plane-move-continuous-exceed-sf
> +kms_psr2_sf@cursor-plane-move-continuous-exceed-fully-sf
> +kms_psr2_sf@plane-move-sf-dmg-area
> +kms_psr2_sf@overlay-plane-move-continuous-sf
> +kms_psr2_sf@overlay-plane-move-continuous-exceed-sf
> +kms_psr2_sf@overlay-plane-move-continuous-exceed-fully-sf
> +kms_psr2_sf@overlay-primary-update-sf-dmg-area
> +kms_psr2_sf@overlay-plane-update-continuous-sf
> +kms_psr2_su@page_flip-XRGB8888
> +kms_psr2_su@page_flip-NV12
> +kms_psr2_su@page_flip-P010
> +kms_psr2_su@frontbuffer-XRGB8888
> +kms_pwrite_crc
> +kms_rmfb@rmfb-ioctl
> +kms_rmfb@close-fd
> +kms_rotation_crc@primary-rotation-90
> +kms_rotation_crc@primary-rotation-180
> +kms_rotation_crc@primary-rotation-270
> +kms_rotation_crc@sprite-rotation-90
> +kms_rotation_crc@sprite-rotation-180
> +kms_rotation_crc@sprite-rotation-270
> +kms_rotation_crc@cursor-rotation-180
> +kms_rotation_crc@sprite-rotation-90-pos-100-0
> +kms_rotation_crc@bad-pixel-format
> +kms_rotation_crc@bad-tiling
> +kms_rotation_crc@primary-x-tiled-reflect-x-0
> +kms_rotation_crc@primary-x-tiled-reflect-x-180
> +kms_rotation_crc@primary-y-tiled-reflect-x-0
> +kms_rotation_crc@primary-y-tiled-reflect-x-90
> +kms_rotation_crc@primary-y-tiled-reflect-x-180
> +kms_rotation_crc@primary-y-tiled-reflect-x-270
> +kms_rotation_crc@primary-yf-tiled-reflect-x-0
> +kms_rotation_crc@primary-yf-tiled-reflect-x-90
> +kms_rotation_crc@primary-yf-tiled-reflect-x-180
> +kms_rotation_crc@primary-yf-tiled-reflect-x-270
> +kms_rotation_crc@primary-4-tiled-reflect-x-0
> +kms_rotation_crc@primary-4-tiled-reflect-x-180
> +kms_rotation_crc@multiplane-rotation
> +kms_rotation_crc@multiplane-rotation-cropping-top
> +kms_rotation_crc@multiplane-rotation-cropping-bottom
> +kms_rotation_crc@exhaust-fences
> +kms_scaling_modes@scaling-mode-full
> +kms_scaling_modes@scaling-mode-center
> +kms_scaling_modes@scaling-mode-full-aspect
> +kms_scaling_modes@scaling-mode-none
> +kms_selftest@drm_cmdline
> +kms_selftest@drm_damage
> +kms_selftest@drm_dp_mst
> +kms_selftest@drm_format_helper
> +kms_selftest@drm_format
> +kms_selftest@framebuffer
> +kms_selftest@drm_plane
> +kms_setmode@basic
> +kms_setmode@basic-clone-single-crtc
> +kms_setmode@invalid-clone-single-crtc
> +kms_setmode@invalid-clone-exclusive-crtc
> +kms_setmode@clone-exclusive-crtc
> +kms_setmode@invalid-clone-single-crtc-stealing
> +kms_sysfs_edid_timing
> +kms_tv_load_detect@load-detect
> +kms_universal_plane@universal-plane-pipe-A-functional
> +kms_universal_plane@universal-plane-pipe-A-sanity
> +kms_universal_plane@disable-primary-vs-flip-pipe-A
> +kms_universal_plane@cursor-fb-leak-pipe-A
> +kms_universal_plane@universal-plane-pageflip-windowed-pipe-A
> +kms_universal_plane@universal-plane-pipe-B-functional
> +kms_universal_plane@universal-plane-pipe-B-sanity
> +kms_universal_plane@disable-primary-vs-flip-pipe-B
> +kms_universal_plane@cursor-fb-leak-pipe-B
> +kms_universal_plane@universal-plane-pageflip-windowed-pipe-B
> +kms_universal_plane@universal-plane-pipe-C-functional
> +kms_universal_plane@universal-plane-pipe-C-sanity
> +kms_universal_plane@disable-primary-vs-flip-pipe-C
> +kms_universal_plane@cursor-fb-leak-pipe-C
> +kms_universal_plane@universal-plane-pageflip-windowed-pipe-C
> +kms_universal_plane@universal-plane-pipe-D-functional
> +kms_universal_plane@universal-plane-pipe-D-sanity
> +kms_universal_plane@disable-primary-vs-flip-pipe-D
> +kms_universal_plane@cursor-fb-leak-pipe-D
> +kms_universal_plane@universal-plane-pageflip-windowed-pipe-D
> +kms_universal_plane@universal-plane-pipe-E-functional
> +kms_universal_plane@universal-plane-pipe-E-sanity
> +kms_universal_plane@disable-primary-vs-flip-pipe-E
> +kms_universal_plane@cursor-fb-leak-pipe-E
> +kms_universal_plane@universal-plane-pageflip-windowed-pipe-E
> +kms_universal_plane@universal-plane-pipe-F-functional
> +kms_universal_plane@universal-plane-pipe-F-sanity
> +kms_universal_plane@disable-primary-vs-flip-pipe-F
> +kms_universal_plane@cursor-fb-leak-pipe-F
> +kms_universal_plane@universal-plane-pageflip-windowed-pipe-F
> +kms_universal_plane@universal-plane-pipe-G-functional
> +kms_universal_plane@universal-plane-pipe-G-sanity
> +kms_universal_plane@disable-primary-vs-flip-pipe-G
> +kms_universal_plane@cursor-fb-leak-pipe-G
> +kms_universal_plane@universal-plane-pageflip-windowed-pipe-G
> +kms_universal_plane@universal-plane-pipe-H-functional
> +kms_universal_plane@universal-plane-pipe-H-sanity
> +kms_universal_plane@disable-primary-vs-flip-pipe-H
> +kms_universal_plane@cursor-fb-leak-pipe-H
> +kms_universal_plane@universal-plane-pageflip-windowed-pipe-H
> +kms_vblank@invalid
> +kms_vblank@crtc-id
> +kms_vblank@pipe-A-accuracy-idle
> +kms_vblank@pipe-A-query-idle
> +kms_vblank@pipe-A-query-idle-hang
> +kms_vblank@pipe-A-query-forked
> +kms_vblank@pipe-A-query-forked-hang
> +kms_vblank@pipe-A-query-busy
> +kms_vblank@pipe-A-query-busy-hang
> +kms_vblank@pipe-A-query-forked-busy
> +kms_vblank@pipe-A-query-forked-busy-hang
> +kms_vblank@pipe-A-wait-idle
> +kms_vblank@pipe-A-wait-idle-hang
> +kms_vblank@pipe-A-wait-forked
> +kms_vblank@pipe-A-wait-forked-hang
> +kms_vblank@pipe-A-wait-busy
> +kms_vblank@pipe-A-wait-busy-hang
> +kms_vblank@pipe-A-wait-forked-busy
> +kms_vblank@pipe-A-wait-forked-busy-hang
> +kms_vblank@pipe-A-ts-continuation-idle
> +kms_vblank@pipe-A-ts-continuation-idle-hang
> +kms_vblank@pipe-A-ts-continuation-dpms-rpm
> +kms_vblank@pipe-A-ts-continuation-dpms-suspend
> +kms_vblank@pipe-A-ts-continuation-suspend
> +kms_vblank@pipe-A-ts-continuation-modeset
> +kms_vblank@pipe-A-ts-continuation-modeset-hang
> +kms_vblank@pipe-A-ts-continuation-modeset-rpm
> +kms_vblank@pipe-B-accuracy-idle
> +kms_vblank@pipe-B-query-idle
> +kms_vblank@pipe-B-query-idle-hang
> +kms_vblank@pipe-B-query-forked
> +kms_vblank@pipe-B-query-forked-hang
> +kms_vblank@pipe-B-query-busy
> +kms_vblank@pipe-B-query-busy-hang
> +kms_vblank@pipe-B-query-forked-busy
> +kms_vblank@pipe-B-query-forked-busy-hang
> +kms_vblank@pipe-B-wait-idle
> +kms_vblank@pipe-B-wait-idle-hang
> +kms_vblank@pipe-B-wait-forked
> +kms_vblank@pipe-B-wait-forked-hang
> +kms_vblank@pipe-B-wait-busy
> +kms_vblank@pipe-B-wait-busy-hang
> +kms_vblank@pipe-B-wait-forked-busy
> +kms_vblank@pipe-B-wait-forked-busy-hang
> +kms_vblank@pipe-B-ts-continuation-idle
> +kms_vblank@pipe-B-ts-continuation-idle-hang
> +kms_vblank@pipe-B-ts-continuation-dpms-rpm
> +kms_vblank@pipe-B-ts-continuation-dpms-suspend
> +kms_vblank@pipe-B-ts-continuation-suspend
> +kms_vblank@pipe-B-ts-continuation-modeset
> +kms_vblank@pipe-B-ts-continuation-modeset-hang
> +kms_vblank@pipe-B-ts-continuation-modeset-rpm
> +kms_vblank@pipe-C-accuracy-idle
> +kms_vblank@pipe-C-query-idle
> +kms_vblank@pipe-C-query-idle-hang
> +kms_vblank@pipe-C-query-forked
> +kms_vblank@pipe-C-query-forked-hang
> +kms_vblank@pipe-C-query-busy
> +kms_vblank@pipe-C-query-busy-hang
> +kms_vblank@pipe-C-query-forked-busy
> +kms_vblank@pipe-C-query-forked-busy-hang
> +kms_vblank@pipe-C-wait-idle
> +kms_vblank@pipe-C-wait-idle-hang
> +kms_vblank@pipe-C-wait-forked
> +kms_vblank@pipe-C-wait-forked-hang
> +kms_vblank@pipe-C-wait-busy
> +kms_vblank@pipe-C-wait-busy-hang
> +kms_vblank@pipe-C-wait-forked-busy
> +kms_vblank@pipe-C-wait-forked-busy-hang
> +kms_vblank@pipe-C-ts-continuation-idle
> +kms_vblank@pipe-C-ts-continuation-idle-hang
> +kms_vblank@pipe-C-ts-continuation-dpms-rpm
> +kms_vblank@pipe-C-ts-continuation-dpms-suspend
> +kms_vblank@pipe-C-ts-continuation-suspend
> +kms_vblank@pipe-C-ts-continuation-modeset
> +kms_vblank@pipe-C-ts-continuation-modeset-hang
> +kms_vblank@pipe-C-ts-continuation-modeset-rpm
> +kms_vblank@pipe-D-accuracy-idle
> +kms_vblank@pipe-D-query-idle
> +kms_vblank@pipe-D-query-idle-hang
> +kms_vblank@pipe-D-query-forked
> +kms_vblank@pipe-D-query-forked-hang
> +kms_vblank@pipe-D-query-busy
> +kms_vblank@pipe-D-query-busy-hang
> +kms_vblank@pipe-D-query-forked-busy
> +kms_vblank@pipe-D-query-forked-busy-hang
> +kms_vblank@pipe-D-wait-idle
> +kms_vblank@pipe-D-wait-idle-hang
> +kms_vblank@pipe-D-wait-forked
> +kms_vblank@pipe-D-wait-forked-hang
> +kms_vblank@pipe-D-wait-busy
> +kms_vblank@pipe-D-wait-busy-hang
> +kms_vblank@pipe-D-wait-forked-busy
> +kms_vblank@pipe-D-wait-forked-busy-hang
> +kms_vblank@pipe-D-ts-continuation-idle
> +kms_vblank@pipe-D-ts-continuation-idle-hang
> +kms_vblank@pipe-D-ts-continuation-dpms-rpm
> +kms_vblank@pipe-D-ts-continuation-dpms-suspend
> +kms_vblank@pipe-D-ts-continuation-suspend
> +kms_vblank@pipe-D-ts-continuation-modeset
> +kms_vblank@pipe-D-ts-continuation-modeset-hang
> +kms_vblank@pipe-D-ts-continuation-modeset-rpm
> +kms_vblank@pipe-E-accuracy-idle
> +kms_vblank@pipe-E-query-idle
> +kms_vblank@pipe-E-query-idle-hang
> +kms_vblank@pipe-E-query-forked
> +kms_vblank@pipe-E-query-forked-hang
> +kms_vblank@pipe-E-query-busy
> +kms_vblank@pipe-E-query-busy-hang
> +kms_vblank@pipe-E-query-forked-busy
> +kms_vblank@pipe-E-query-forked-busy-hang
> +kms_vblank@pipe-E-wait-idle
> +kms_vblank@pipe-E-wait-idle-hang
> +kms_vblank@pipe-E-wait-forked
> +kms_vblank@pipe-E-wait-forked-hang
> +kms_vblank@pipe-E-wait-busy
> +kms_vblank@pipe-E-wait-busy-hang
> +kms_vblank@pipe-E-wait-forked-busy
> +kms_vblank@pipe-E-wait-forked-busy-hang
> +kms_vblank@pipe-E-ts-continuation-idle
> +kms_vblank@pipe-E-ts-continuation-idle-hang
> +kms_vblank@pipe-E-ts-continuation-dpms-rpm
> +kms_vblank@pipe-E-ts-continuation-dpms-suspend
> +kms_vblank@pipe-E-ts-continuation-suspend
> +kms_vblank@pipe-E-ts-continuation-modeset
> +kms_vblank@pipe-E-ts-continuation-modeset-hang
> +kms_vblank@pipe-E-ts-continuation-modeset-rpm
> +kms_vblank@pipe-F-accuracy-idle
> +kms_vblank@pipe-F-query-idle
> +kms_vblank@pipe-F-query-idle-hang
> +kms_vblank@pipe-F-query-forked
> +kms_vblank@pipe-F-query-forked-hang
> +kms_vblank@pipe-F-query-busy
> +kms_vblank@pipe-F-query-busy-hang
> +kms_vblank@pipe-F-query-forked-busy
> +kms_vblank@pipe-F-query-forked-busy-hang
> +kms_vblank@pipe-F-wait-idle
> +kms_vblank@pipe-F-wait-idle-hang
> +kms_vblank@pipe-F-wait-forked
> +kms_vblank@pipe-F-wait-forked-hang
> +kms_vblank@pipe-F-wait-busy
> +kms_vblank@pipe-F-wait-busy-hang
> +kms_vblank@pipe-F-wait-forked-busy
> +kms_vblank@pipe-F-wait-forked-busy-hang
> +kms_vblank@pipe-F-ts-continuation-idle
> +kms_vblank@pipe-F-ts-continuation-idle-hang
> +kms_vblank@pipe-F-ts-continuation-dpms-rpm
> +kms_vblank@pipe-F-ts-continuation-dpms-suspend
> +kms_vblank@pipe-F-ts-continuation-suspend
> +kms_vblank@pipe-F-ts-continuation-modeset
> +kms_vblank@pipe-F-ts-continuation-modeset-hang
> +kms_vblank@pipe-F-ts-continuation-modeset-rpm
> +kms_vblank@pipe-G-accuracy-idle
> +kms_vblank@pipe-G-query-idle
> +kms_vblank@pipe-G-query-idle-hang
> +kms_vblank@pipe-G-query-forked
> +kms_vblank@pipe-G-query-forked-hang
> +kms_vblank@pipe-G-query-busy
> +kms_vblank@pipe-G-query-busy-hang
> +kms_vblank@pipe-G-query-forked-busy
> +kms_vblank@pipe-G-query-forked-busy-hang
> +kms_vblank@pipe-G-wait-idle
> +kms_vblank@pipe-G-wait-idle-hang
> +kms_vblank@pipe-G-wait-forked
> +kms_vblank@pipe-G-wait-forked-hang
> +kms_vblank@pipe-G-wait-busy
> +kms_vblank@pipe-G-wait-busy-hang
> +kms_vblank@pipe-G-wait-forked-busy
> +kms_vblank@pipe-G-wait-forked-busy-hang
> +kms_vblank@pipe-G-ts-continuation-idle
> +kms_vblank@pipe-G-ts-continuation-idle-hang
> +kms_vblank@pipe-G-ts-continuation-dpms-rpm
> +kms_vblank@pipe-G-ts-continuation-dpms-suspend
> +kms_vblank@pipe-G-ts-continuation-suspend
> +kms_vblank@pipe-G-ts-continuation-modeset
> +kms_vblank@pipe-G-ts-continuation-modeset-hang
> +kms_vblank@pipe-G-ts-continuation-modeset-rpm
> +kms_vblank@pipe-H-accuracy-idle
> +kms_vblank@pipe-H-query-idle
> +kms_vblank@pipe-H-query-idle-hang
> +kms_vblank@pipe-H-query-forked
> +kms_vblank@pipe-H-query-forked-hang
> +kms_vblank@pipe-H-query-busy
> +kms_vblank@pipe-H-query-busy-hang
> +kms_vblank@pipe-H-query-forked-busy
> +kms_vblank@pipe-H-query-forked-busy-hang
> +kms_vblank@pipe-H-wait-idle
> +kms_vblank@pipe-H-wait-idle-hang
> +kms_vblank@pipe-H-wait-forked
> +kms_vblank@pipe-H-wait-forked-hang
> +kms_vblank@pipe-H-wait-busy
> +kms_vblank@pipe-H-wait-busy-hang
> +kms_vblank@pipe-H-wait-forked-busy
> +kms_vblank@pipe-H-wait-forked-busy-hang
> +kms_vblank@pipe-H-ts-continuation-idle
> +kms_vblank@pipe-H-ts-continuation-idle-hang
> +kms_vblank@pipe-H-ts-continuation-dpms-rpm
> +kms_vblank@pipe-H-ts-continuation-dpms-suspend
> +kms_vblank@pipe-H-ts-continuation-suspend
> +kms_vblank@pipe-H-ts-continuation-modeset
> +kms_vblank@pipe-H-ts-continuation-modeset-hang
> +kms_vblank@pipe-H-ts-continuation-modeset-rpm
> +kms_vrr@flip-basic
> +kms_vrr@flip-dpms
> +kms_vrr@flip-suspend
> +kms_vrr@flipline
> +kms_vrr@negative-basic
> +kms_writeback@writeback-pixel-formats
> +kms_writeback@writeback-invalid-parameters
> +kms_writeback@writeback-fb-id
> +kms_writeback@writeback-check-output
> +prime_mmap_kms@buffer-sharing
> diff --git a/drivers/gpu/drm/ci/x86_64.config b/drivers/gpu/drm/ci/x86_64.config
> new file mode 100644
> index 000000000000..1cbd49a5b23a
> --- /dev/null
> +++ b/drivers/gpu/drm/ci/x86_64.config
> @@ -0,0 +1,111 @@
> +CONFIG_LOCALVERSION_AUTO=y
> +CONFIG_DEBUG_KERNEL=y
> +
> +CONFIG_CRYPTO_ZSTD=y
> +CONFIG_ZRAM_MEMORY_TRACKING=y
> +CONFIG_ZRAM_WRITEBACK=y
> +CONFIG_ZRAM=y
> +CONFIG_ZSMALLOC_STAT=y
> +
> +CONFIG_PWM=y
> +CONFIG_PM_DEVFREQ=y
> +CONFIG_OF=y
> +CONFIG_CROS_EC=y
> +
> +# abootimg with a 'dummy' rootfs fails with root=/dev/nfs
> +CONFIG_BLK_DEV_INITRD=n
> +
> +CONFIG_DEVFREQ_GOV_PERFORMANCE=y
> +CONFIG_DEVFREQ_GOV_POWERSAVE=y
> +CONFIG_DEVFREQ_GOV_USERSPACE=y
> +CONFIG_DEVFREQ_GOV_PASSIVE=y
> +
> +CONFIG_DRM=y
> +CONFIG_DRM_PANEL_SIMPLE=y
> +CONFIG_PWM_CROS_EC=y
> +CONFIG_BACKLIGHT_PWM=y
> +
> +# Strip out some stuff we don't need for graphics testing, to reduce
> +# the build.
> +CONFIG_CAN=n
> +CONFIG_WIRELESS=n
> +CONFIG_RFKILL=n
> +CONFIG_WLAN=n
> +
> +CONFIG_REGULATOR_FAN53555=y
> +CONFIG_REGULATOR=y
> +
> +CONFIG_REGULATOR_VCTRL=y
> +
> +CONFIG_KASAN=n
> +CONFIG_KASAN_INLINE=n
> +CONFIG_STACKTRACE=n
> +
> +CONFIG_TMPFS=y
> +
> +CONFIG_PROVE_LOCKING=n
> +CONFIG_DEBUG_LOCKDEP=n
> +CONFIG_SOFTLOCKUP_DETECTOR=y
> +CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC=y
> +
> +CONFIG_DETECT_HUNG_TASK=y
> +
> +CONFIG_USB_USBNET=y
> +CONFIG_NETDEVICES=y
> +CONFIG_USB_NET_DRIVERS=y
> +CONFIG_USB_RTL8152=y
> +CONFIG_USB_NET_AX8817X=y
> +CONFIG_USB_NET_SMSC95XX=y
> +CONFIG_USB_GADGET=y
> +CONFIG_USB_ETH=y
> +
> +CONFIG_FW_LOADER_COMPRESS=y
> +
> +# options for AMD devices
> +CONFIG_X86_AMD_PLATFORM_DEVICE=y
> +CONFIG_ACPI_VIDEO=y
> +CONFIG_X86_AMD_FREQ_SENSITIVITY=y
> +CONFIG_PINCTRL=y
> +CONFIG_PINCTRL_AMD=y
> +CONFIG_DRM_AMDGPU=m
> +CONFIG_DRM_AMDGPU_SI=y
> +CONFIG_DRM_AMDGPU_USERPTR=y
> +CONFIG_DRM_AMD_ACP=n
> +CONFIG_ACPI_WMI=y
> +CONFIG_MXM_WMI=y
> +CONFIG_PARPORT=y
> +CONFIG_PARPORT_PC=y
> +CONFIG_PARPORT_SERIAL=y
> +CONFIG_SERIAL_8250_DW=y
> +CONFIG_CHROME_PLATFORMS=y
> +CONFIG_KVM_AMD=m
> +
> +#options for Intel devices
> +CONFIG_MFD_INTEL_LPSS_PCI=y
> +CONFIG_KVM_INTEL=m
> +
> +#options for KVM guests
> +CONFIG_FUSE_FS=y
> +CONFIG_HYPERVISOR_GUEST=y
> +CONFIG_KVM=y
> +CONFIG_KVM_GUEST=y
> +CONFIG_VIRT_DRIVERS=y
> +CONFIG_VIRTIO_FS=y
> +CONFIG_DRM_VIRTIO_GPU=y
> +CONFIG_SERIAL_8250_CONSOLE=y
> +CONFIG_VIRTIO_NET=y
> +CONFIG_VIRTIO_CONSOLE=y
> +CONFIG_PARAVIRT=y
> +CONFIG_VIRTIO_BLK=y
> +CONFIG_VIRTUALIZATION=y
> +CONFIG_VIRTIO=y
> +CONFIG_VIRTIO_PCI=y
> +CONFIG_VIRTIO_MMIO=y
> +CONFIG_VIRTIO_MMIO_CMDLINE_DEVICES=y
> +CONFIG_CRYPTO_DEV_VIRTIO=y
> +CONFIG_HW_RANDOM_VIRTIO=y
> +CONFIG_BLK_MQ_VIRTIO=y
> +CONFIG_TUN=y
> +CONFIG_VSOCKETS=y
> +CONFIG_VIRTIO_VSOCKETS=y
> +CONFIG_VHOST_VSOCK=m
> diff --git a/drivers/gpu/drm/ci/xfails/amdgpu-stoney-fails.txt b/drivers/gpu/drm/ci/xfails/amdgpu-stoney-fails.txt
> new file mode 100644
> index 000000000000..bd9392536e7c
> --- /dev/null
> +++ b/drivers/gpu/drm/ci/xfails/amdgpu-stoney-fails.txt
> @@ -0,0 +1,19 @@
> +kms_addfb_basic@bad-pitch-65536,Fail
> +kms_addfb_basic@bo-too-small,Fail
> +kms_async_flips@invalid-async-flip,Fail
> +kms_atomic@plane-immutable-zpos,Fail
> +kms_atomic_transition@plane-toggle-modeset-transition,Fail
> +kms_bw@linear-tiling-1-displays-2560x1440p,Fail
> +kms_bw@linear-tiling-1-displays-3840x2160p,Fail
> +kms_bw@linear-tiling-2-displays-3840x2160p,Fail
> +kms_bw@linear-tiling-3-displays-1920x1080p,Fail
> +kms_color@degamma,Fail
> +kms_cursor_crc@cursor-size-change,Fail
> +kms_cursor_crc@pipe-A-cursor-size-change,Fail
> +kms_cursor_crc@pipe-B-cursor-size-change,Fail
> +kms_cursor_legacy@forked-move,Fail
> +kms_hdr@bpc-switch,Fail
> +kms_hdr@bpc-switch-dpms,Fail
> +kms_plane_multiple@atomic-pipe-A-tiling-none,Fail
> +kms_rmfb@close-fd,Fail
> +kms_rotation_crc@primary-rotation-180,Fail
> diff --git a/drivers/gpu/drm/ci/xfails/amdgpu-stoney-flakes.txt b/drivers/gpu/drm/ci/xfails/amdgpu-stoney-flakes.txt
> new file mode 100644
> index 000000000000..f8defa0f9e67
> --- /dev/null
> +++ b/drivers/gpu/drm/ci/xfails/amdgpu-stoney-flakes.txt
> @@ -0,0 +1,21 @@
> +kms_addfb_basic@too-high
> +kms_async_flips@alternate-sync-async-flip
> +kms_async_flips@async-flip-with-page-flip-events
> +kms_async_flips@crc
> +kms_async_flips@test-cursor
> +kms_async_flips@test-time-stamp
> +kms_atomic_transition@plane-all-modeset-transition-internal-panels
> +kms_atomic_transition@plane-all-transition
> +kms_atomic_transition@plane-use-after-nonblocking-unbind
> +kms_bw@linear-tiling-1-displays-1920x1080p
> +kms_bw@linear-tiling-2-displays-1920x1080p
> +kms_bw@linear-tiling-2-displays-2560x1440p
> +kms_bw@linear-tiling-3-displays-2560x1440p
> +kms_bw@linear-tiling-3-displays-3840x2160p
> +kms_cursor_crc@pipe-A-cursor-alpha-opaque
> +kms_cursor_crc@pipe-B-cursor-alpha-opaque
> +kms_plane@pixel-format
> +kms_plane_multiple@atomic-pipe-B-tiling-none
> +kms_plane_scaling@downscale-with-rotation-factor-0-5
> +kms_universal_plane@disable-primary-vs-flip-pipe-A
> +kms_universal_plane@disable-primary-vs-flip-pipe-B
> diff --git a/drivers/gpu/drm/ci/xfails/amdgpu-stoney-skips.txt b/drivers/gpu/drm/ci/xfails/amdgpu-stoney-skips.txt
> new file mode 100644
> index 000000000000..e2c538a0f954
> --- /dev/null
> +++ b/drivers/gpu/drm/ci/xfails/amdgpu-stoney-skips.txt
> @@ -0,0 +1,2 @@
> +# Suspend to RAM seems to be broken on this machine
> +.*suspend.*
> \ No newline at end of file
> diff --git a/drivers/gpu/drm/ci/xfails/i915-amly-fails.txt b/drivers/gpu/drm/ci/xfails/i915-amly-fails.txt
> new file mode 100644
> index 000000000000..5f513c638beb
> --- /dev/null
> +++ b/drivers/gpu/drm/ci/xfails/i915-amly-fails.txt
> @@ -0,0 +1,17 @@
> +kms_flip_scaled_crc@flip-32bpp-linear-to-64bpp-linear-downscaling,Fail
> +kms_flip_scaled_crc@flip-32bpp-linear-to-64bpp-linear-upscaling,Fail
> +kms_flip_scaled_crc@flip-32bpp-xtile-to-64bpp-xtile-downscaling,Fail
> +kms_flip_scaled_crc@flip-32bpp-xtile-to-64bpp-xtile-upscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-linear-to-16bpp-linear-downscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-linear-to-16bpp-linear-upscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-linear-to-32bpp-linear-downscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-linear-to-32bpp-linear-upscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-xtile-to-16bpp-xtile-downscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-xtile-to-16bpp-xtile-upscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-xtile-to-32bpp-xtile-downscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-xtile-to-32bpp-xtile-upscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytilegen12rcccs-upscaling,Fail
> +kms_plane_alpha_blend@alpha-basic,Fail
> +kms_plane_alpha_blend@alpha-opaque-fb,Fail
> +kms_plane_alpha_blend@alpha-transparent-fb,Fail
> +kms_plane_alpha_blend@constant-alpha-max,Fail
> diff --git a/drivers/gpu/drm/ci/xfails/i915-amly-flakes.txt b/drivers/gpu/drm/ci/xfails/i915-amly-flakes.txt
> new file mode 100644
> index 000000000000..d5000515a315
> --- /dev/null
> +++ b/drivers/gpu/drm/ci/xfails/i915-amly-flakes.txt
> @@ -0,0 +1,32 @@
> +kms_bw@linear-tiling-2-displays-1920x1080p
> +kms_bw@linear-tiling-2-displays-2560x1440p
> +kms_bw@linear-tiling-2-displays-3840x2160p
> +kms_bw@linear-tiling-3-displays-1920x1080p
> +kms_bw@linear-tiling-3-displays-2560x1440p
> +kms_bw@linear-tiling-3-displays-3840x2160p
> +kms_bw@linear-tiling-4-displays-1920x1080p
> +kms_bw@linear-tiling-4-displays-2560x1440p
> +kms_bw@linear-tiling-4-displays-3840x2160p
> +kms_flip_scaled_crc@flip-32bpp-ytile-to-64bpp-ytile-downscaling
> +kms_flip_scaled_crc@flip-32bpp-ytile-to-64bpp-ytile-upscaling
> +kms_flip_scaled_crc@flip-32bpp-ytileccs-to-64bpp-ytile-downscaling
> +kms_flip_scaled_crc@flip-32bpp-ytileccs-to-64bpp-ytile-upscaling
> +kms_flip_scaled_crc@flip-64bpp-ytile-to-16bpp-ytile-downscaling
> +kms_flip_scaled_crc@flip-64bpp-ytile-to-16bpp-ytile-upscaling
> +kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytile-downscaling
> +kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytile-upscaling
> +kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytilercccs-downscaling
> +kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytilercccs-upscaling
> +kms_plane_alpha_blend@pipe-A-alpha-basic
> +kms_plane_alpha_blend@pipe-A-alpha-opaque-fb
> +kms_plane_alpha_blend@pipe-A-alpha-transparent-fb
> +kms_plane_alpha_blend@pipe-A-constant-alpha-max
> +kms_plane_alpha_blend@pipe-B-alpha-basic
> +kms_plane_alpha_blend@pipe-B-alpha-opaque-fb
> +kms_plane_alpha_blend@pipe-B-alpha-transparent-fb
> +kms_plane_alpha_blend@pipe-B-constant-alpha-max
> +kms_plane_alpha_blend@pipe-C-alpha-basic
> +kms_plane_alpha_blend@pipe-C-alpha-opaque-fb
> +kms_plane_alpha_blend@pipe-C-alpha-transparent-fb
> +kms_plane_alpha_blend@pipe-C-constant-alpha-max
> +kms_sysfs_edid_timing
> diff --git a/drivers/gpu/drm/ci/xfails/i915-amly-skips.txt b/drivers/gpu/drm/ci/xfails/i915-amly-skips.txt
> new file mode 100644
> index 000000000000..fe55540a3f9a
> --- /dev/null
> +++ b/drivers/gpu/drm/ci/xfails/i915-amly-skips.txt
> @@ -0,0 +1,4 @@
> +# Suspend to RAM seems to be broken on this machine
> +.*suspend.*
> +# This is generating kernel oops with divide error
> +kms_plane_scaling@invalid-parameters
> \ No newline at end of file
> diff --git a/drivers/gpu/drm/ci/xfails/i915-apl-fails.txt b/drivers/gpu/drm/ci/xfails/i915-apl-fails.txt
> new file mode 100644
> index 000000000000..46397ce38d5a
> --- /dev/null
> +++ b/drivers/gpu/drm/ci/xfails/i915-apl-fails.txt
> @@ -0,0 +1,58 @@
> +kms_3d,Timeout
> +kms_bw@linear-tiling-2-displays-1920x1080p,Fail
> +kms_bw@linear-tiling-2-displays-2560x1440p,Fail
> +kms_bw@linear-tiling-2-displays-3840x2160p,Fail
> +kms_bw@linear-tiling-3-displays-1920x1080p,Fail
> +kms_bw@linear-tiling-3-displays-2560x1440p,Fail
> +kms_bw@linear-tiling-3-displays-3840x2160p,Fail
> +kms_bw@linear-tiling-4-displays-1920x1080p,Fail
> +kms_bw@linear-tiling-4-displays-2560x1440p,Fail
> +kms_bw@linear-tiling-4-displays-3840x2160p,Fail
> +kms_color@ctm-0-25,Fail
> +kms_color@ctm-0-50,Fail
> +kms_color@ctm-0-75,Fail
> +kms_color@ctm-max,Fail
> +kms_color@ctm-negative,Fail
> +kms_color@ctm-red-to-blue,Fail
> +kms_color@ctm-signed,Fail
> +kms_flip_scaled_crc@flip-32bpp-linear-to-64bpp-linear-downscaling,Fail
> +kms_flip_scaled_crc@flip-32bpp-linear-to-64bpp-linear-upscaling,Fail
> +kms_flip_scaled_crc@flip-32bpp-xtile-to-64bpp-xtile-downscaling,Fail
> +kms_flip_scaled_crc@flip-32bpp-xtile-to-64bpp-xtile-upscaling,Fail
> +kms_flip_scaled_crc@flip-32bpp-ytile-to-64bpp-ytile-downscaling,Fail
> +kms_flip_scaled_crc@flip-32bpp-ytile-to-64bpp-ytile-upscaling,Fail
> +kms_flip_scaled_crc@flip-32bpp-ytileccs-to-64bpp-ytile-downscaling,Fail
> +kms_flip_scaled_crc@flip-32bpp-ytileccs-to-64bpp-ytile-upscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-linear-to-16bpp-linear-downscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-linear-to-16bpp-linear-upscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-linear-to-32bpp-linear-downscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-linear-to-32bpp-linear-upscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-xtile-to-16bpp-xtile-downscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-xtile-to-16bpp-xtile-upscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-xtile-to-32bpp-xtile-downscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-xtile-to-32bpp-xtile-upscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-ytile-to-16bpp-ytile-downscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-ytile-to-16bpp-ytile-upscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytile-downscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytile-upscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytilegen12rcccs-upscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytilercccs-downscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytilercccs-upscaling,Fail
> +kms_hdmi_inject@inject-4k,Timeout
> +kms_plane@plane-position-hole,Timeout
> +kms_plane_alpha_blend@alpha-basic,Fail
> +kms_plane_alpha_blend@alpha-opaque-fb,Fail
> +kms_plane_alpha_blend@alpha-transparent-fb,Fail
> +kms_plane_alpha_blend@constant-alpha-max,Fail
> +kms_plane_alpha_blend@pipe-A-alpha-opaque-fb,Fail
> +kms_plane_alpha_blend@pipe-A-alpha-transparent-fb,Fail
> +kms_plane_alpha_blend@pipe-A-constant-alpha-max,Fail
> +kms_plane_alpha_blend@pipe-B-alpha-opaque-fb,Fail
> +kms_plane_alpha_blend@pipe-B-alpha-transparent-fb,Fail
> +kms_plane_alpha_blend@pipe-B-constant-alpha-max,Fail
> +kms_plane_alpha_blend@pipe-C-alpha-opaque-fb,Fail
> +kms_plane_alpha_blend@pipe-C-alpha-transparent-fb,Fail
> +kms_plane_alpha_blend@pipe-C-constant-alpha-max,Fail
> +kms_plane_multiple@tiling-y,Timeout
> +kms_pwrite_crc,Timeout
> +kms_sysfs_edid_timing,Fail
> diff --git a/drivers/gpu/drm/ci/xfails/i915-apl-flakes.txt b/drivers/gpu/drm/ci/xfails/i915-apl-flakes.txt
> new file mode 100644
> index 000000000000..331c5841bb41
> --- /dev/null
> +++ b/drivers/gpu/drm/ci/xfails/i915-apl-flakes.txt
> @@ -0,0 +1 @@
> +kms_frontbuffer_tracking@fbc-tiling-linear
> diff --git a/drivers/gpu/drm/ci/xfails/i915-apl-skips.txt b/drivers/gpu/drm/ci/xfails/i915-apl-skips.txt
> new file mode 100644
> index 000000000000..3430b215c06e
> --- /dev/null
> +++ b/drivers/gpu/drm/ci/xfails/i915-apl-skips.txt
> @@ -0,0 +1,6 @@
> +# Suspend to RAM seems to be broken on this machine
> +.*suspend.*
> +# This is generating kernel oops with divide error
> +kms_plane_scaling@invalid-parameters
> +# This is cascading issues
> +kms_3d
> \ No newline at end of file
> diff --git a/drivers/gpu/drm/ci/xfails/i915-cml-fails.txt b/drivers/gpu/drm/ci/xfails/i915-cml-fails.txt
> new file mode 100644
> index 000000000000..6139b410e767
> --- /dev/null
> +++ b/drivers/gpu/drm/ci/xfails/i915-cml-fails.txt
> @@ -0,0 +1,18 @@
> +kms_color@ctm-0-25,Fail
> +kms_flip_scaled_crc@flip-32bpp-linear-to-64bpp-linear-downscaling,Fail
> +kms_flip_scaled_crc@flip-32bpp-linear-to-64bpp-linear-upscaling,Fail
> +kms_flip_scaled_crc@flip-32bpp-xtile-to-64bpp-xtile-downscaling,Fail
> +kms_flip_scaled_crc@flip-32bpp-xtile-to-64bpp-xtile-upscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-linear-to-16bpp-linear-downscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-linear-to-16bpp-linear-upscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-linear-to-32bpp-linear-downscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-linear-to-32bpp-linear-upscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-xtile-to-16bpp-xtile-downscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-xtile-to-16bpp-xtile-upscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-xtile-to-32bpp-xtile-downscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-xtile-to-32bpp-xtile-upscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytilegen12rcccs-upscaling,Fail
> +kms_plane_alpha_blend@alpha-basic,Fail
> +kms_plane_alpha_blend@alpha-opaque-fb,Fail
> +kms_plane_alpha_blend@alpha-transparent-fb,Fail
> +kms_plane_alpha_blend@constant-alpha-max,Fail
> diff --git a/drivers/gpu/drm/ci/xfails/i915-cml-flakes.txt b/drivers/gpu/drm/ci/xfails/i915-cml-flakes.txt
> new file mode 100644
> index 000000000000..0514a7b3fdb0
> --- /dev/null
> +++ b/drivers/gpu/drm/ci/xfails/i915-cml-flakes.txt
> @@ -0,0 +1,38 @@
> +kms_bw@linear-tiling-2-displays-1920x1080p
> +kms_bw@linear-tiling-2-displays-2560x1440p
> +kms_bw@linear-tiling-2-displays-3840x2160p
> +kms_bw@linear-tiling-3-displays-1920x1080p
> +kms_bw@linear-tiling-3-displays-2560x1440p
> +kms_bw@linear-tiling-3-displays-3840x2160p
> +kms_bw@linear-tiling-4-displays-1920x1080p
> +kms_bw@linear-tiling-4-displays-2560x1440p
> +kms_bw@linear-tiling-4-displays-3840x2160p
> +kms_draw_crc@draw-method-xrgb8888-render-xtiled
> +kms_flip@flip-vs-suspend
> +kms_flip_scaled_crc@flip-32bpp-ytile-to-64bpp-ytile-downscaling
> +kms_flip_scaled_crc@flip-32bpp-ytile-to-64bpp-ytile-upscaling
> +kms_flip_scaled_crc@flip-32bpp-ytileccs-to-64bpp-ytile-downscaling
> +kms_flip_scaled_crc@flip-32bpp-ytileccs-to-64bpp-ytile-upscaling
> +kms_flip_scaled_crc@flip-64bpp-ytile-to-16bpp-ytile-downscaling
> +kms_flip_scaled_crc@flip-64bpp-ytile-to-16bpp-ytile-upscaling
> +kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytile-downscaling
> +kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytile-upscaling
> +kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytilercccs-downscaling
> +kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytilercccs-upscaling
> +kms_hdr@bpc-switch-suspend
> +kms_plane_alpha_blend@constant-alpha-min
> +kms_plane_alpha_blend@pipe-A-alpha-basic
> +kms_plane_alpha_blend@pipe-A-alpha-opaque-fb
> +kms_plane_alpha_blend@pipe-A-alpha-transparent-fb
> +kms_plane_alpha_blend@pipe-A-constant-alpha-max
> +kms_plane_alpha_blend@pipe-B-alpha-basic
> +kms_plane_alpha_blend@pipe-B-alpha-opaque-fb
> +kms_plane_alpha_blend@pipe-B-alpha-transparent-fb
> +kms_plane_alpha_blend@pipe-B-constant-alpha-max
> +kms_plane_alpha_blend@pipe-C-alpha-basic
> +kms_plane_alpha_blend@pipe-C-alpha-opaque-fb
> +kms_plane_alpha_blend@pipe-C-alpha-transparent-fb
> +kms_plane_alpha_blend@pipe-C-constant-alpha-max
> +kms_psr2_su@page_flip-NV12
> +kms_psr2_su@page_flip-P010
> +kms_setmode@basic
> diff --git a/drivers/gpu/drm/ci/xfails/i915-cml-skips.txt b/drivers/gpu/drm/ci/xfails/i915-cml-skips.txt
> new file mode 100644
> index 000000000000..6d3d7ddc377f
> --- /dev/null
> +++ b/drivers/gpu/drm/ci/xfails/i915-cml-skips.txt
> @@ -0,0 +1,2 @@
> +# This is generating kernel oops with divide error
> +kms_plane_scaling@invalid-parameters
> diff --git a/drivers/gpu/drm/ci/xfails/i915-glk-fails.txt b/drivers/gpu/drm/ci/xfails/i915-glk-fails.txt
> new file mode 100644
> index 000000000000..5bd432e78129
> --- /dev/null
> +++ b/drivers/gpu/drm/ci/xfails/i915-glk-fails.txt
> @@ -0,0 +1,19 @@
> +kms_fbcon_fbt@fbc,Fail
> +kms_flip_scaled_crc@flip-32bpp-linear-to-64bpp-linear-downscaling,Fail
> +kms_flip_scaled_crc@flip-32bpp-linear-to-64bpp-linear-upscaling,Fail
> +kms_flip_scaled_crc@flip-32bpp-xtile-to-64bpp-xtile-downscaling,Fail
> +kms_flip_scaled_crc@flip-32bpp-xtile-to-64bpp-xtile-upscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-linear-to-16bpp-linear-downscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-linear-to-16bpp-linear-upscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-linear-to-32bpp-linear-downscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-linear-to-32bpp-linear-upscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-xtile-to-16bpp-xtile-downscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-xtile-to-16bpp-xtile-upscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-xtile-to-32bpp-xtile-downscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-xtile-to-32bpp-xtile-upscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytilegen12rcccs-upscaling,Fail
> +kms_frontbuffer_tracking@fbcdrrs-tiling-linear,Fail
> +kms_plane_alpha_blend@alpha-basic,Fail
> +kms_plane_alpha_blend@alpha-opaque-fb,Fail
> +kms_plane_alpha_blend@alpha-transparent-fb,Fail
> +kms_plane_alpha_blend@constant-alpha-max,Fail
> diff --git a/drivers/gpu/drm/ci/xfails/i915-glk-flakes.txt b/drivers/gpu/drm/ci/xfails/i915-glk-flakes.txt
> new file mode 100644
> index 000000000000..fc41d13a2d56
> --- /dev/null
> +++ b/drivers/gpu/drm/ci/xfails/i915-glk-flakes.txt
> @@ -0,0 +1,41 @@
> +kms_bw@linear-tiling-1-displays-3840x2160p
> +kms_bw@linear-tiling-2-displays-1920x1080p
> +kms_bw@linear-tiling-2-displays-2560x1440p
> +kms_bw@linear-tiling-2-displays-3840x2160p
> +kms_bw@linear-tiling-3-displays-1920x1080p
> +kms_bw@linear-tiling-3-displays-2560x1440p
> +kms_bw@linear-tiling-3-displays-3840x2160p
> +kms_bw@linear-tiling-4-displays-1920x1080p
> +kms_bw@linear-tiling-4-displays-2560x1440p
> +kms_bw@linear-tiling-4-displays-3840x2160p
> +kms_flip@blocking-wf_vblank
> +kms_flip@wf_vblank-ts-check
> +kms_flip@wf_vblank-ts-check-interruptible
> +kms_flip_scaled_crc@flip-32bpp-ytile-to-64bpp-ytile-downscaling
> +kms_flip_scaled_crc@flip-32bpp-ytile-to-64bpp-ytile-upscaling
> +kms_flip_scaled_crc@flip-32bpp-ytileccs-to-64bpp-ytile-downscaling
> +kms_flip_scaled_crc@flip-32bpp-ytileccs-to-64bpp-ytile-upscaling
> +kms_flip_scaled_crc@flip-64bpp-ytile-to-16bpp-ytile-downscaling
> +kms_flip_scaled_crc@flip-64bpp-ytile-to-16bpp-ytile-upscaling
> +kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytile-downscaling
> +kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytile-upscaling
> +kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytilercccs-downscaling
> +kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytilercccs-upscaling
> +kms_frontbuffer_tracking@fbc-tiling-linear
> +kms_plane_alpha_blend@pipe-A-alpha-basic
> +kms_plane_alpha_blend@pipe-A-alpha-opaque-fb
> +kms_plane_alpha_blend@pipe-A-alpha-transparent-fb
> +kms_plane_alpha_blend@pipe-A-constant-alpha-max
> +kms_plane_alpha_blend@pipe-B-alpha-basic
> +kms_plane_alpha_blend@pipe-B-alpha-opaque-fb
> +kms_plane_alpha_blend@pipe-B-alpha-transparent-fb
> +kms_plane_alpha_blend@pipe-B-constant-alpha-max
> +kms_plane_alpha_blend@pipe-C-alpha-basic
> +kms_plane_alpha_blend@pipe-C-alpha-opaque-fb
> +kms_plane_alpha_blend@pipe-C-alpha-transparent-fb
> +kms_plane_alpha_blend@pipe-C-constant-alpha-max
> +kms_prop_blob@invalid-set-prop-any
> +kms_rotation_crc@multiplane-rotation
> +kms_rotation_crc@multiplane-rotation-cropping-bottom
> +kms_rotation_crc@multiplane-rotation-cropping-top
> +kms_setmode@basic
> diff --git a/drivers/gpu/drm/ci/xfails/i915-glk-skips.txt b/drivers/gpu/drm/ci/xfails/i915-glk-skips.txt
> new file mode 100644
> index 000000000000..4c7d00ce14bc
> --- /dev/null
> +++ b/drivers/gpu/drm/ci/xfails/i915-glk-skips.txt
> @@ -0,0 +1,5 @@
> +# Suspend to RAM seems to be broken on this machine
> +.*suspend.*
> +
> +# This is generating kernel oops with divide error
> +kms_plane_scaling@invalid-parameters
> \ No newline at end of file
> diff --git a/drivers/gpu/drm/ci/xfails/i915-kbl-fails.txt b/drivers/gpu/drm/ci/xfails/i915-kbl-fails.txt
> new file mode 100644
> index 000000000000..56ec021a7679
> --- /dev/null
> +++ b/drivers/gpu/drm/ci/xfails/i915-kbl-fails.txt
> @@ -0,0 +1,25 @@
> +kms_bw@linear-tiling-2-displays-2560x1440p,Fail
> +kms_bw@linear-tiling-4-displays-2560x1440p,Fail
> +kms_flip_scaled_crc@flip-32bpp-linear-to-64bpp-linear-downscaling,Fail
> +kms_flip_scaled_crc@flip-32bpp-linear-to-64bpp-linear-upscaling,Fail
> +kms_flip_scaled_crc@flip-32bpp-xtile-to-64bpp-xtile-downscaling,Fail
> +kms_flip_scaled_crc@flip-32bpp-xtile-to-64bpp-xtile-upscaling,Fail
> +kms_flip_scaled_crc@flip-32bpp-ytile-to-64bpp-ytile-upscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-linear-to-16bpp-linear-downscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-linear-to-16bpp-linear-upscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-linear-to-32bpp-linear-downscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-linear-to-32bpp-linear-upscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-xtile-to-16bpp-xtile-downscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-xtile-to-16bpp-xtile-upscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-xtile-to-32bpp-xtile-downscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-xtile-to-32bpp-xtile-upscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytile-downscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytilegen12rcccs-upscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytilercccs-upscaling,Fail
> +kms_plane_alpha_blend@alpha-basic,Fail
> +kms_plane_alpha_blend@alpha-opaque-fb,Fail
> +kms_plane_alpha_blend@alpha-transparent-fb,Fail
> +kms_plane_alpha_blend@constant-alpha-max,Fail
> +kms_plane_alpha_blend@pipe-A-constant-alpha-max,Fail
> +kms_plane_alpha_blend@pipe-B-alpha-opaque-fb,Fail
> +kms_plane_alpha_blend@pipe-C-constant-alpha-max,Fail
> diff --git a/drivers/gpu/drm/ci/xfails/i915-kbl-flakes.txt b/drivers/gpu/drm/ci/xfails/i915-kbl-flakes.txt
> new file mode 100644
> index 000000000000..f3ba1c4c5d46
> --- /dev/null
> +++ b/drivers/gpu/drm/ci/xfails/i915-kbl-flakes.txt
> @@ -0,0 +1,26 @@
> +kms_async_flips@crc
> +kms_bw@linear-tiling-2-displays-1920x1080p
> +kms_bw@linear-tiling-2-displays-3840x2160p
> +kms_bw@linear-tiling-3-displays-1920x1080p
> +kms_bw@linear-tiling-3-displays-2560x1440p
> +kms_bw@linear-tiling-3-displays-3840x2160p
> +kms_bw@linear-tiling-4-displays-1920x1080p
> +kms_bw@linear-tiling-4-displays-3840x2160p
> +kms_color@ctm-0-25
> +kms_flip_scaled_crc@flip-32bpp-ytile-to-64bpp-ytile-downscaling
> +kms_flip_scaled_crc@flip-32bpp-ytileccs-to-64bpp-ytile-downscaling
> +kms_flip_scaled_crc@flip-32bpp-ytileccs-to-64bpp-ytile-upscaling
> +kms_flip_scaled_crc@flip-64bpp-ytile-to-16bpp-ytile-downscaling
> +kms_flip_scaled_crc@flip-64bpp-ytile-to-16bpp-ytile-upscaling
> +kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytile-upscaling
> +kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytilercccs-downscaling
> +kms_plane_alpha_blend@pipe-A-alpha-basic
> +kms_plane_alpha_blend@pipe-A-alpha-opaque-fb
> +kms_plane_alpha_blend@pipe-A-alpha-transparent-fb
> +kms_plane_alpha_blend@pipe-B-alpha-basic
> +kms_plane_alpha_blend@pipe-B-alpha-transparent-fb
> +kms_plane_alpha_blend@pipe-B-constant-alpha-max
> +kms_plane_alpha_blend@pipe-C-alpha-basic
> +kms_plane_alpha_blend@pipe-C-alpha-opaque-fb
> +kms_plane_alpha_blend@pipe-C-alpha-transparent-fb
> +kms_sysfs_edid_timing
> diff --git a/drivers/gpu/drm/ci/xfails/i915-kbl-skips.txt b/drivers/gpu/drm/ci/xfails/i915-kbl-skips.txt
> new file mode 100644
> index 000000000000..4c7d00ce14bc
> --- /dev/null
> +++ b/drivers/gpu/drm/ci/xfails/i915-kbl-skips.txt
> @@ -0,0 +1,5 @@
> +# Suspend to RAM seems to be broken on this machine
> +.*suspend.*
> +
> +# This is generating kernel oops with divide error
> +kms_plane_scaling@invalid-parameters
> \ No newline at end of file
> diff --git a/drivers/gpu/drm/ci/xfails/i915-tgl-fails.txt b/drivers/gpu/drm/ci/xfails/i915-tgl-fails.txt
> new file mode 100644
> index 000000000000..a6da5544e198
> --- /dev/null
> +++ b/drivers/gpu/drm/ci/xfails/i915-tgl-fails.txt
> @@ -0,0 +1,37 @@
> +kms_bw@linear-tiling-2-displays-3840x2160p,Fail
> +kms_bw@linear-tiling-3-displays-1920x1080p,Fail
> +kms_bw@linear-tiling-3-displays-2560x1440p,Fail
> +kms_bw@linear-tiling-3-displays-3840x2160p,Fail
> +kms_bw@linear-tiling-4-displays-1920x1080p,Fail
> +kms_bw@linear-tiling-4-displays-2560x1440p,Fail
> +kms_bw@linear-tiling-4-displays-3840x2160p,Fail
> +kms_bw@linear-tiling-5-displays-1920x1080p,Fail
> +kms_bw@linear-tiling-5-displays-2560x1440p,Fail
> +kms_bw@linear-tiling-5-displays-3840x2160p,Fail
> +kms_color@ctm-0-25,Fail
> +kms_flip@flip-vs-panning-vs-hang,Timeout
> +kms_flip_scaled_crc@flip-32bpp-linear-to-64bpp-linear-downscaling,Fail
> +kms_flip_scaled_crc@flip-32bpp-linear-to-64bpp-linear-upscaling,Fail
> +kms_flip_scaled_crc@flip-32bpp-xtile-to-64bpp-xtile-downscaling,Fail
> +kms_flip_scaled_crc@flip-32bpp-xtile-to-64bpp-xtile-upscaling,Fail
> +kms_flip_scaled_crc@flip-32bpp-ytile-to-64bpp-ytile-downscaling,Fail
> +kms_flip_scaled_crc@flip-32bpp-ytile-to-64bpp-ytile-upscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-linear-to-16bpp-linear-downscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-linear-to-16bpp-linear-upscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-linear-to-32bpp-linear-downscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-linear-to-32bpp-linear-upscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-xtile-to-16bpp-xtile-downscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-xtile-to-16bpp-xtile-upscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-xtile-to-32bpp-xtile-downscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-xtile-to-32bpp-xtile-upscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-ytile-to-16bpp-ytile-downscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-ytile-to-16bpp-ytile-upscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytile-downscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytile-upscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytilegen12rcccs-upscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytilercccs-downscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytilercccs-upscaling,Fail
> +kms_rotation_crc@bad-pixel-format,Fail
> +kms_rotation_crc@multiplane-rotation,Fail
> +kms_rotation_crc@multiplane-rotation-cropping-bottom,Fail
> +kms_rotation_crc@multiplane-rotation-cropping-top,Fail
> diff --git a/drivers/gpu/drm/ci/xfails/i915-tgl-flakes.txt b/drivers/gpu/drm/ci/xfails/i915-tgl-flakes.txt
> new file mode 100644
> index 000000000000..1cd910ee06df
> --- /dev/null
> +++ b/drivers/gpu/drm/ci/xfails/i915-tgl-flakes.txt
> @@ -0,0 +1,5 @@
> +kms_draw_crc@.*
> +kms_flip@blocking-absolute-wf_vblank
> +kms_flip@bo-too-big-interruptible
> +kms_flip@busy-flip
> +kms_flip@flip-vs-rmfb-interruptible
> diff --git a/drivers/gpu/drm/ci/xfails/i915-tgl-skips.txt b/drivers/gpu/drm/ci/xfails/i915-tgl-skips.txt
> new file mode 100644
> index 000000000000..1d0621750b14
> --- /dev/null
> +++ b/drivers/gpu/drm/ci/xfails/i915-tgl-skips.txt
> @@ -0,0 +1,11 @@
> +# Suspend to RAM seems to be broken on this machine
> +.*suspend.*
> +
> +# GPU hangs, then the whole machine
> +gem_eio.*
> +
> +# Whole machine hangs
> +kms_flip@absolute-wf_vblank@a-edp1
> +
> +# This is generating kernel oops with divide error
> +kms_plane_scaling@invalid-parameters
> \ No newline at end of file
> diff --git a/drivers/gpu/drm/ci/xfails/i915-whl-fails.txt b/drivers/gpu/drm/ci/xfails/i915-whl-fails.txt
> new file mode 100644
> index 000000000000..967327ddc1ac
> --- /dev/null
> +++ b/drivers/gpu/drm/ci/xfails/i915-whl-fails.txt
> @@ -0,0 +1,48 @@
> +kms_bw@linear-tiling-2-displays-1920x1080p,Fail
> +kms_bw@linear-tiling-2-displays-2560x1440p,Fail
> +kms_bw@linear-tiling-2-displays-3840x2160p,Fail
> +kms_bw@linear-tiling-3-displays-1920x1080p,Fail
> +kms_bw@linear-tiling-3-displays-2560x1440p,Fail
> +kms_bw@linear-tiling-3-displays-3840x2160p,Fail
> +kms_bw@linear-tiling-4-displays-1920x1080p,Fail
> +kms_bw@linear-tiling-4-displays-2560x1440p,Fail
> +kms_bw@linear-tiling-4-displays-3840x2160p,Fail
> +kms_fbcon_fbt@fbc,Fail
> +kms_fbcon_fbt@fbc-suspend,Fail
> +kms_flip_scaled_crc@flip-32bpp-linear-to-64bpp-linear-downscaling,Fail
> +kms_flip_scaled_crc@flip-32bpp-linear-to-64bpp-linear-upscaling,Fail
> +kms_flip_scaled_crc@flip-32bpp-xtile-to-64bpp-xtile-downscaling,Fail
> +kms_flip_scaled_crc@flip-32bpp-xtile-to-64bpp-xtile-upscaling,Fail
> +kms_flip_scaled_crc@flip-32bpp-ytile-to-64bpp-ytile-downscaling,Fail
> +kms_flip_scaled_crc@flip-32bpp-ytile-to-64bpp-ytile-upscaling,Fail
> +kms_flip_scaled_crc@flip-32bpp-ytileccs-to-64bpp-ytile-downscaling,Fail
> +kms_flip_scaled_crc@flip-32bpp-ytileccs-to-64bpp-ytile-upscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-linear-to-16bpp-linear-downscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-linear-to-16bpp-linear-upscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-linear-to-32bpp-linear-downscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-linear-to-32bpp-linear-upscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-xtile-to-16bpp-xtile-downscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-xtile-to-16bpp-xtile-upscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-xtile-to-32bpp-xtile-downscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-xtile-to-32bpp-xtile-upscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-ytile-to-16bpp-ytile-downscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-ytile-to-16bpp-ytile-upscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytile-downscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytile-upscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytilegen12rcccs-upscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytilercccs-downscaling,Fail
> +kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytilercccs-upscaling,Fail
> +kms_frontbuffer_tracking@fbc-tiling-linear,Fail
> +kms_plane_alpha_blend@alpha-basic,Fail
> +kms_plane_alpha_blend@alpha-opaque-fb,Fail
> +kms_plane_alpha_blend@alpha-transparent-fb,Fail
> +kms_plane_alpha_blend@constant-alpha-max,Fail
> +kms_plane_alpha_blend@pipe-A-alpha-opaque-fb,Fail
> +kms_plane_alpha_blend@pipe-A-alpha-transparent-fb,Fail
> +kms_plane_alpha_blend@pipe-A-constant-alpha-max,Fail
> +kms_plane_alpha_blend@pipe-B-alpha-opaque-fb,Fail
> +kms_plane_alpha_blend@pipe-B-alpha-transparent-fb,Fail
> +kms_plane_alpha_blend@pipe-B-constant-alpha-max,Fail
> +kms_plane_alpha_blend@pipe-C-alpha-opaque-fb,Fail
> +kms_plane_alpha_blend@pipe-C-alpha-transparent-fb,Fail
> +kms_plane_alpha_blend@pipe-C-constant-alpha-max,Fail
> diff --git a/drivers/gpu/drm/ci/xfails/i915-whl-flakes.txt b/drivers/gpu/drm/ci/xfails/i915-whl-flakes.txt
> new file mode 100644
> index 000000000000..c33202e7e2a1
> --- /dev/null
> +++ b/drivers/gpu/drm/ci/xfails/i915-whl-flakes.txt
> @@ -0,0 +1 @@
> +kms_flip@flip-vs-suspend
> diff --git a/drivers/gpu/drm/ci/xfails/i915-whl-skips.txt b/drivers/gpu/drm/ci/xfails/i915-whl-skips.txt
> new file mode 100644
> index 000000000000..f3be0888a214
> --- /dev/null
> +++ b/drivers/gpu/drm/ci/xfails/i915-whl-skips.txt
> @@ -0,0 +1,2 @@
> +# This is generating kernel oops with divide error
> +kms_plane_scaling@invalid-parameters
> \ No newline at end of file
> diff --git a/drivers/gpu/drm/ci/xfails/mediatek-mt8173-fails.txt b/drivers/gpu/drm/ci/xfails/mediatek-mt8173-fails.txt
> new file mode 100644
> index 000000000000..671916067dba
> --- /dev/null
> +++ b/drivers/gpu/drm/ci/xfails/mediatek-mt8173-fails.txt
> @@ -0,0 +1,29 @@
> +kms_3d,Fail
> +kms_addfb_basic@addfb25-bad-modifier,Fail
> +kms_bw@linear-tiling-1-displays-1920x1080p,Fail
> +kms_bw@linear-tiling-1-displays-2560x1440p,Fail
> +kms_bw@linear-tiling-1-displays-3840x2160p,Fail
> +kms_bw@linear-tiling-2-displays-1920x1080p,Fail
> +kms_bw@linear-tiling-2-displays-2560x1440p,Fail
> +kms_bw@linear-tiling-2-displays-3840x2160p,Fail
> +kms_bw@linear-tiling-3-displays-1920x1080p,Fail
> +kms_bw@linear-tiling-3-displays-2560x1440p,Fail
> +kms_bw@linear-tiling-3-displays-3840x2160p,Fail
> +kms_color@pipe-A-invalid-gamma-lut-sizes,Fail
> +kms_color@pipe-B-invalid-gamma-lut-sizes,Fail
> +kms_force_connector_basic@force-connector-state,Fail
> +kms_force_connector_basic@force-edid,Fail
> +kms_force_connector_basic@force-load-detect,Fail
> +kms_force_connector_basic@prune-stale-modes,Fail
> +kms_invalid_mode@int-max-clock,Fail
> +kms_plane_scaling@planes-upscale-20x20,Fail
> +kms_plane_scaling@planes-upscale-20x20-downscale-factor-0-25,Fail
> +kms_plane_scaling@planes-upscale-20x20-downscale-factor-0-5,Fail
> +kms_plane_scaling@planes-upscale-20x20-downscale-factor-0-75,Fail
> +kms_plane_scaling@upscale-with-modifier-20x20,Fail
> +kms_plane_scaling@upscale-with-pixel-format-20x20,Fail
> +kms_plane_scaling@upscale-with-rotation-20x20,Fail
> +kms_properties@get_properties-sanity-atomic,Fail
> +kms_properties@plane-properties-atomic,Fail
> +kms_properties@plane-properties-legacy,Fail
> +kms_rmfb@close-fd,Fail
> diff --git a/drivers/gpu/drm/ci/xfails/mediatek-mt8173-flakes.txt b/drivers/gpu/drm/ci/xfails/mediatek-mt8173-flakes.txt
> new file mode 100644
> index 000000000000..e69de29bb2d1
> diff --git a/drivers/gpu/drm/ci/xfails/mediatek-mt8183-fails.txt b/drivers/gpu/drm/ci/xfails/mediatek-mt8183-fails.txt
> new file mode 100644
> index 000000000000..6ff81d00e84e
> --- /dev/null
> +++ b/drivers/gpu/drm/ci/xfails/mediatek-mt8183-fails.txt
> @@ -0,0 +1,10 @@
> +kms_addfb_basic@addfb25-bad-modifier,Fail
> +kms_bw@linear-tiling-1-displays-2560x1440p,Fail
> +kms_bw@linear-tiling-2-displays-1920x1080p,Fail
> +kms_bw@linear-tiling-2-displays-2560x1440p,Fail
> +kms_bw@linear-tiling-2-displays-3840x2160p,Fail
> +kms_bw@linear-tiling-3-displays-2560x1440p,Fail
> +kms_bw@linear-tiling-3-displays-3840x2160p,Fail
> +kms_color@pipe-A-invalid-gamma-lut-sizes,Fail
> +kms_plane_scaling@upscale-with-rotation-20x20,Fail
> +kms_rmfb@close-fd,Fail
> \ No newline at end of file
> diff --git a/drivers/gpu/drm/ci/xfails/mediatek-mt8183-flakes.txt b/drivers/gpu/drm/ci/xfails/mediatek-mt8183-flakes.txt
> new file mode 100644
> index 000000000000..208890b79eb0
> --- /dev/null
> +++ b/drivers/gpu/drm/ci/xfails/mediatek-mt8183-flakes.txt
> @@ -0,0 +1,14 @@
> +core_setmaster_vs_auth
> +kms_bw@linear-tiling-1-displays-1920x1080p
> +kms_bw@linear-tiling-1-displays-3840x2160p
> +kms_bw@linear-tiling-3-displays-1920x1080p
> +kms_cursor_legacy@cursor-vs-flip-atomic
> +kms_plane_scaling@invalid-num-scalers
> +kms_plane_scaling@planes-upscale-20x20
> +kms_plane_scaling@planes-upscale-20x20-downscale-factor-0-5
> +kms_plane_scaling@upscale-with-modifier-20x20
> +kms_plane_scaling@upscale-with-pixel-format-20x20
> +kms_prop_blob@invalid-set-prop-any
> +kms_properties@get_properties-sanity-atomic
> +kms_properties@plane-properties-atomic
> +kms_properties@plane-properties-legacy
> \ No newline at end of file
> diff --git a/drivers/gpu/drm/ci/xfails/meson-g12b-fails.txt b/drivers/gpu/drm/ci/xfails/meson-g12b-fails.txt
> new file mode 100644
> index 000000000000..860e702091e2
> --- /dev/null
> +++ b/drivers/gpu/drm/ci/xfails/meson-g12b-fails.txt
> @@ -0,0 +1,12 @@
> +kms_3d,Fail
> +kms_properties@connector-properties-atomic,Fail
> +kms_properties@get_properties-sanity-atomic,Fail
> +kms_properties@get_properties-sanity-non-atomic,Fail
> +kms_properties@connector-properties-legacy,Fail
> +kms_cursor_legacy@forked-bo,Fail
> +kms_cursor_legacy@forked-move,Fail
> +kms_cursor_legacy@single-bo,Fail
> +kms_cursor_legacy@single-move,Fail
> +kms_cursor_legacy@torture-bo,Fail
> +kms_cursor_legacy@torture-move,Fail
> +kms_hdmi_inject@inject-4k,Fail
> \ No newline at end of file
> diff --git a/drivers/gpu/drm/ci/xfails/meson-g12b-flakes.txt b/drivers/gpu/drm/ci/xfails/meson-g12b-flakes.txt
> new file mode 100644
> index 000000000000..b63329d06767
> --- /dev/null
> +++ b/drivers/gpu/drm/ci/xfails/meson-g12b-flakes.txt
> @@ -0,0 +1,4 @@
> +kms_force_connector_basic@force-connector-state
> +kms_force_connector_basic@force-edid
> +kms_force_connector_basic@force-load-detect
> +kms_force_connector_basic@prune-stale-modes
> \ No newline at end of file
> diff --git a/drivers/gpu/drm/ci/xfails/msm-apq8016-fails.txt b/drivers/gpu/drm/ci/xfails/msm-apq8016-fails.txt
> new file mode 100644
> index 000000000000..9981682feab2
> --- /dev/null
> +++ b/drivers/gpu/drm/ci/xfails/msm-apq8016-fails.txt
> @@ -0,0 +1,15 @@
> +kms_3d,Fail
> +kms_addfb_basic@addfb25-bad-modifier,Fail
> +kms_cursor_legacy@all-pipes-forked-bo,Fail
> +kms_cursor_legacy@all-pipes-forked-move,Fail
> +kms_cursor_legacy@all-pipes-single-bo,Fail
> +kms_cursor_legacy@all-pipes-single-move,Fail
> +kms_cursor_legacy@all-pipes-torture-bo,Fail
> +kms_cursor_legacy@all-pipes-torture-move,Fail
> +kms_cursor_legacy@pipe-A-forked-bo,Fail
> +kms_cursor_legacy@pipe-A-forked-move,Fail
> +kms_cursor_legacy@pipe-A-single-bo,Fail
> +kms_cursor_legacy@pipe-A-single-move,Fail
> +kms_cursor_legacy@pipe-A-torture-bo,Fail
> +kms_cursor_legacy@pipe-A-torture-move,Fail
> +kms_hdmi_inject@inject-4k,Fail
> diff --git a/drivers/gpu/drm/ci/xfails/msm-apq8016-flakes.txt b/drivers/gpu/drm/ci/xfails/msm-apq8016-flakes.txt
> new file mode 100644
> index 000000000000..0e3b60d3fade
> --- /dev/null
> +++ b/drivers/gpu/drm/ci/xfails/msm-apq8016-flakes.txt
> @@ -0,0 +1,4 @@
> +kms_force_connector_basic@force-connector-state
> +kms_force_connector_basic@force-edid
> +kms_force_connector_basic@force-load-detect
> +kms_force_connector_basic@prune-stale-modes
> diff --git a/drivers/gpu/drm/ci/xfails/msm-apq8096-fails.txt b/drivers/gpu/drm/ci/xfails/msm-apq8096-fails.txt
> new file mode 100644
> index 000000000000..88a1fc0a3b0d
> --- /dev/null
> +++ b/drivers/gpu/drm/ci/xfails/msm-apq8096-fails.txt
> @@ -0,0 +1,2 @@
> +kms_3d,Fail
> +kms_addfb_basic@addfb25-bad-modifier,Fail
> diff --git a/drivers/gpu/drm/ci/xfails/msm-apq8096-flakes.txt b/drivers/gpu/drm/ci/xfails/msm-apq8096-flakes.txt
> new file mode 100644
> index 000000000000..0e3b60d3fade
> --- /dev/null
> +++ b/drivers/gpu/drm/ci/xfails/msm-apq8096-flakes.txt
> @@ -0,0 +1,4 @@
> +kms_force_connector_basic@force-connector-state
> +kms_force_connector_basic@force-edid
> +kms_force_connector_basic@force-load-detect
> +kms_force_connector_basic@prune-stale-modes
> diff --git a/drivers/gpu/drm/ci/xfails/msm-apq8096-skips.txt b/drivers/gpu/drm/ci/xfails/msm-apq8096-skips.txt
> new file mode 100644
> index 000000000000..cd49c8ce2059
> --- /dev/null
> +++ b/drivers/gpu/drm/ci/xfails/msm-apq8096-skips.txt
> @@ -0,0 +1,2 @@
> +# Whole machine hangs
> +kms_cursor_legacy@all-pipes-torture-move
> \ No newline at end of file
> diff --git a/drivers/gpu/drm/ci/xfails/msm-sc7180-fails.txt b/drivers/gpu/drm/ci/xfails/msm-sc7180-fails.txt
> new file mode 100644
> index 000000000000..14adeba3b62d
> --- /dev/null
> +++ b/drivers/gpu/drm/ci/xfails/msm-sc7180-fails.txt
> @@ -0,0 +1,25 @@
> +kms_cursor_legacy@cursor-vs-flip-toggle,Fail
> +kms_cursor_legacy@cursor-vs-flip-varying-size,Fail
> +kms_cursor_legacy@cursorA-vs-flipA-atomic-transitions,Crash
> +kms_pipe_crc_basic@compare-crc-sanitycheck-nv12,Fail
> +kms_plane@pixel-format,Fail
> +kms_plane@pixel-format-source-clamping,Fail
> +kms_plane@plane-position-covered,Fail
> +kms_plane@plane-position-hole,Fail
> +kms_plane@plane-position-hole-dpms,Fail
> +kms_plane_alpha_blend@alpha-7efc,Fail
> +kms_plane_alpha_blend@coverage-7efc,Fail
> +kms_plane_alpha_blend@coverage-vs-premult-vs-constant,Fail
> +kms_plane_alpha_blend@pipe-A-alpha-7efc,Fail
> +kms_plane_alpha_blend@pipe-A-coverage-7efc,Fail
> +kms_plane_alpha_blend@pipe-A-coverage-vs-premult-vs-constant,Fail
> +kms_plane_alpha_blend@pipe-B-alpha-7efc,Fail
> +kms_plane_alpha_blend@pipe-B-alpha-basic,Fail
> +kms_plane_alpha_blend@pipe-B-alpha-opaque-fb,Fail
> +kms_plane_alpha_blend@pipe-B-constant-alpha-max,Fail
> +kms_plane_alpha_blend@pipe-B-constant-alpha-mid,Fail
> +kms_plane_alpha_blend@pipe-B-coverage-7efc,Fail
> +kms_plane_alpha_blend@pipe-B-coverage-vs-premult-vs-constant,Fail
> +kms_rmfb@close-fd,Fail
> +kms_universal_plane@disable-primary-vs-flip-pipe-b,Fail
> +kms_universal_plane@universal-plane-pipe-B-sanity,Fail
> diff --git a/drivers/gpu/drm/ci/xfails/msm-sc7180-flakes.txt b/drivers/gpu/drm/ci/xfails/msm-sc7180-flakes.txt
> new file mode 100644
> index 000000000000..636563d3e59a
> --- /dev/null
> +++ b/drivers/gpu/drm/ci/xfails/msm-sc7180-flakes.txt
> @@ -0,0 +1,7 @@
> +
> +# Test ends up reading CRC from frame before cursor update
> +# bug
> +# sometimes.. tbd if this is a kernel CRC bug or a test
> +kms_cursor_crc@.*
> +kms_plane_multiple@atomic-pipe-A-tiling-none
> +kms_atomic_transition@modeset-transition-nonblocking-fencing,Fail
> \ No newline at end of file
> diff --git a/drivers/gpu/drm/ci/xfails/msm-sc7180-skips.txt b/drivers/gpu/drm/ci/xfails/msm-sc7180-skips.txt
> new file mode 100644
> index 000000000000..410e0eeb3161
> --- /dev/null
> +++ b/drivers/gpu/drm/ci/xfails/msm-sc7180-skips.txt
> @@ -0,0 +1,23 @@
> +# Suspend to RAM seems to be broken on this machine
> +.*suspend.*
> +
> +# Test incorrectly assumes that CTM support implies gamma/degamma
> +# LUT support.  None of the subtests handle the case of only having
> +# CTM support
> +kms_color.*
> +
> +# 4k@60 is not supported on this hw, but driver doesn't handle it
> +# too gracefully.. https://gitlab.freedesktop.org/drm/msm/-/issues/15
> +kms_bw@linear-tiling-.*-displays-3840x2160p
> +
> +# Until igt fix lands: https://patchwork.freedesktop.org/patch/493175/
> +kms_bw@linear-tiling-2.*
> +kms_bw@linear-tiling-3.*
> +kms_bw@linear-tiling-4.*
> +kms_bw@linear-tiling-5.*
> +kms_bw@linear-tiling-6.*
> +
> +# igt fix posted: https://patchwork.freedesktop.org/patch/499926/
> +# failure mode is flakey due to randomization but fails frequently
> +# enough to be detected as a Crash or occasionally UnexpectedPass.
> +kms_plane_multiple@atomic-pipe-A-tiling-none
> diff --git a/drivers/gpu/drm/ci/xfails/msm-sdm845-fails.txt b/drivers/gpu/drm/ci/xfails/msm-sdm845-fails.txt
> new file mode 100644
> index 000000000000..09c0c623cd75
> --- /dev/null
> +++ b/drivers/gpu/drm/ci/xfails/msm-sdm845-fails.txt
> @@ -0,0 +1,68 @@
> +kms_color@ctm-0-25,Fail
> +kms_color@ctm-0-50,Fail
> +kms_color@ctm-0-75,Fail
> +kms_color@ctm-blue-to-red,Fail
> +kms_color@ctm-green-to-red,Fail
> +kms_color@ctm-negative,Fail
> +kms_color@ctm-red-to-blue,Fail
> +kms_color@ctm-signed,Fail
> +kms_color@pipe-A-ctm-0-25,Fail
> +kms_color@pipe-A-ctm-0-5,Fail
> +kms_color@pipe-A-ctm-0-75,Fail
> +kms_color@pipe-A-ctm-blue-to-red,Fail
> +kms_color@pipe-A-ctm-green-to-red,Fail
> +kms_color@pipe-A-ctm-max,Fail
> +kms_color@pipe-A-ctm-negative,Fail
> +kms_color@pipe-A-ctm-red-to-blue,Fail
> +kms_color@pipe-A-legacy-gamma,Fail
> +kms_cursor_legacy@basic-flip-after-cursor-atomic,Fail
> +kms_cursor_legacy@basic-flip-after-cursor-legacy,Fail
> +kms_cursor_legacy@basic-flip-after-cursor-varying-size,Fail
> +kms_cursor_legacy@basic-flip-before-cursor-atomic,Fail
> +kms_cursor_legacy@basic-flip-before-cursor-legacy,Fail
> +kms_cursor_legacy@basic-flip-before-cursor-varying-size,Fail
> +kms_cursor_legacy@cursor-vs-flip-atomic,Fail
> +kms_cursor_legacy@cursor-vs-flip-atomic-transitions,Fail
> +kms_cursor_legacy@cursor-vs-flip-atomic-transitions-varying-size,Fail
> +kms_cursor_legacy@cursor-vs-flip-legacy,Fail
> +kms_cursor_legacy@cursor-vs-flip-toggle,Fail
> +kms_cursor_legacy@cursor-vs-flip-varying-size,Fail
> +kms_cursor_legacy@cursorA-vs-flipA-toggle,Fail
> +kms_cursor_legacy@flip-vs-cursor-atomic,Fail
> +kms_cursor_legacy@flip-vs-cursor-crc-atomic,Fail
> +kms_cursor_legacy@flip-vs-cursor-crc-legacy,Fail
> +kms_cursor_legacy@flip-vs-cursor-legacy,Fail
> +kms_cursor_legacy@short-flip-after-cursor-atomic-transitions,Fail
> +kms_cursor_legacy@short-flip-after-cursor-atomic-transitions-varying-size,Fail
> +kms_cursor_legacy@short-flip-after-cursor-toggle,Fail
> +kms_cursor_legacy@short-flip-before-cursor-atomic-transitions,Fail
> +kms_cursor_legacy@short-flip-before-cursor-atomic-transitions-varying-size,Fail
> +kms_pipe_crc_basic@compare-crc-sanitycheck-nv12,Fail
> +kms_plane@pixel-format,Fail
> +kms_plane@pixel-format-source-clamping,Fail
> +kms_plane_alpha_blend@alpha-7efc,Fail
> +kms_plane_alpha_blend@coverage-7efc,Fail
> +kms_plane_alpha_blend@coverage-vs-premult-vs-constant,Fail
> +kms_plane_alpha_blend@pipe-A-alpha-7efc,Fail
> +kms_plane_alpha_blend@pipe-A-coverage-7efc,Fail
> +kms_plane_alpha_blend@pipe-A-coverage-vs-premult-vs-constant,Fail
> +kms_plane_cursor@overlay,Fail
> +kms_plane_cursor@pipe-A-overlay-size-128,Fail
> +kms_plane_cursor@pipe-A-overlay-size-256,Fail
> +kms_plane_cursor@pipe-A-overlay-size-64,Fail
> +kms_plane_cursor@pipe-A-viewport-size-128,Fail
> +kms_plane_cursor@pipe-A-viewport-size-256,Fail
> +kms_plane_cursor@pipe-A-viewport-size-64,Fail
> +kms_plane_cursor@viewport,Fail
> +kms_plane_scaling@downscale-with-pixel-format-factor-0-25,Timeout
> +kms_plane_scaling@downscale-with-pixel-format-factor-0-5,Timeout
> +kms_plane_scaling@downscale-with-pixel-format-factor-0-75,Timeout
> +kms_plane_scaling@plane-downscale-with-pixel-format-factor-0-25,Timeout
> +kms_plane_scaling@plane-downscale-with-pixel-format-factor-0-5,Timeout
> +kms_plane_scaling@plane-downscale-with-pixel-format-factor-0-75,Timeout
> +kms_plane_scaling@plane-scaler-with-clipping-clamping-pixel-formats,Timeout
> +kms_plane_scaling@plane-scaler-with-pixel-format-unity-scaling,Timeout
> +kms_plane_scaling@planes-downscale-factor-0-25,Fail
> +kms_plane_scaling@scaler-with-clipping-clamping,Timeout
> +kms_plane_scaling@scaler-with-pixel-format-unity-scaling,Timeout
> +kms_rmfb@close-fd,Fail
> diff --git a/drivers/gpu/drm/ci/xfails/msm-sdm845-flakes.txt b/drivers/gpu/drm/ci/xfails/msm-sdm845-flakes.txt
> new file mode 100644
> index 000000000000..5b3aaab7ac3f
> --- /dev/null
> +++ b/drivers/gpu/drm/ci/xfails/msm-sdm845-flakes.txt
> @@ -0,0 +1,11 @@
> +
> +
> +# Test ends up reading CRC from frame before cursor update
> +# bug
> +# sometimes.. tbd if this is a kernel CRC bug or a test
> +kms_cursor_crc@.*
> +kms_cursor_legacy@flip-vs-cursor-toggle
> +kms_cursor_legacy@pipe-A-forked-bo
> +kms_cursor_legacy@pipe-A-forked-move
> +kms_cursor_legacy@short-flip-before-cursor-toggle
> +kms_flip@dpms-vs-vblank-race-interruptible
> diff --git a/drivers/gpu/drm/ci/xfails/msm-sdm845-skips.txt b/drivers/gpu/drm/ci/xfails/msm-sdm845-skips.txt
> new file mode 100644
> index 000000000000..42675f1c6d76
> --- /dev/null
> +++ b/drivers/gpu/drm/ci/xfails/msm-sdm845-skips.txt
> @@ -0,0 +1,2 @@
> +# Hangs machine
> +kms_bw.*
> \ No newline at end of file
> diff --git a/drivers/gpu/drm/ci/xfails/rockchip-rk3288-fails.txt b/drivers/gpu/drm/ci/xfails/rockchip-rk3288-fails.txt
> new file mode 100644
> index 000000000000..2a1baa948e12
> --- /dev/null
> +++ b/drivers/gpu/drm/ci/xfails/rockchip-rk3288-fails.txt
> @@ -0,0 +1,48 @@
> +kms_3d,Crash
> +kms_bw@linear-tiling-2-displays-1920x1080p,Fail
> +kms_bw@linear-tiling-2-displays-2560x1440p,Fail
> +kms_bw@linear-tiling-2-displays-3840x2160p,Fail
> +kms_bw@linear-tiling-3-displays-1920x1080p,Fail
> +kms_bw@linear-tiling-3-displays-2560x1440p,Fail
> +kms_bw@linear-tiling-3-displays-3840x2160p,Fail
> +kms_force_connector_basic@force-load-detect,Fail
> +kms_invalid_mode@int-max-clock,Crash
> +kms_plane@pixel-format,Crash
> +kms_plane@pixel-format-source-clamping,Crash
> +kms_plane@plane-position-hole,Crash
> +kms_plane@plane-position-hole-dpms,Crash
> +kms_plane_cursor@overlay,Crash
> +kms_plane_cursor@pipe-A-overlay-size-128,Fail
> +kms_plane_cursor@pipe-A-overlay-size-256,Fail
> +kms_plane_cursor@pipe-A-overlay-size-64,Fail
> +kms_plane_cursor@pipe-A-primary-size-128,Fail
> +kms_plane_cursor@pipe-A-primary-size-256,Fail
> +kms_plane_cursor@pipe-A-primary-size-64,Fail
> +kms_plane_cursor@pipe-A-viewport-size-128,Fail
> +kms_plane_cursor@pipe-A-viewport-size-256,Fail
> +kms_plane_cursor@pipe-A-viewport-size-64,Fail
> +kms_plane_cursor@pipe-B-overlay-size-128,Fail
> +kms_plane_cursor@pipe-B-overlay-size-256,Fail
> +kms_plane_cursor@pipe-B-overlay-size-64,Fail
> +kms_plane_cursor@pipe-B-primary-size-128,Fail
> +kms_plane_cursor@pipe-B-primary-size-256,Fail
> +kms_plane_cursor@pipe-B-primary-size-64,Fail
> +kms_plane_cursor@pipe-B-viewport-size-128,Fail
> +kms_plane_cursor@pipe-B-viewport-size-256,Fail
> +kms_plane_cursor@pipe-B-viewport-size-64,Fail
> +kms_plane_cursor@primary,Crash
> +kms_plane_cursor@viewport,Crash
> +kms_plane_lowres@tiling-none,Fail
> +kms_plane_scaling@downscale-with-modifier-factor-0-25,Fail
> +kms_plane_scaling@downscale-with-rotation-factor-0-25,Fail
> +kms_plane_scaling@upscale-with-modifier-20x20,Fail
> +kms_plane_scaling@upscale-with-modifier-factor-0-25,Fail
> +kms_plane_scaling@upscale-with-pixel-format-20x20,Fail
> +kms_plane_scaling@upscale-with-pixel-format-factor-0-25,Fail
> +kms_plane_scaling@upscale-with-rotation-20x20,Fail
> +kms_prime@basic-crc,Fail
> +kms_properties@connector-properties-atomic,Crash
> +kms_properties@connector-properties-legacy,Crash
> +kms_properties@get_properties-sanity-atomic,Crash
> +kms_properties@get_properties-sanity-non-atomic,Crash
> +kms_setmode@invalid-clone-single-crtc,Crash
> diff --git a/drivers/gpu/drm/ci/xfails/rockchip-rk3288-flakes.txt b/drivers/gpu/drm/ci/xfails/rockchip-rk3288-flakes.txt
> new file mode 100644
> index 000000000000..45c54c75c899
> --- /dev/null
> +++ b/drivers/gpu/drm/ci/xfails/rockchip-rk3288-flakes.txt
> @@ -0,0 +1,9 @@
> +kms_addfb_basic@addfb25-bad-modifier
> +kms_cursor_crc@.*
> +kms_flip@basic-flip-vs-wf_vblank
> +kms_invalid_mode@int-max-clock,Crash
> +kms_pipe_crc_basic@.*
> +kms_properties@connector-properties-atomic,Crash
> +kms_properties@get_properties-sanity-atomic,Crash
> +kms_properties@get_properties-sanity-non-atomic,Crash
> +kms_rmfb@close-fd
> diff --git a/drivers/gpu/drm/ci/xfails/rockchip-rk3288-skips.txt b/drivers/gpu/drm/ci/xfails/rockchip-rk3288-skips.txt
> new file mode 100644
> index 000000000000..f20c3574b75a
> --- /dev/null
> +++ b/drivers/gpu/drm/ci/xfails/rockchip-rk3288-skips.txt
> @@ -0,0 +1,52 @@
> +# Suspend to RAM seems to be broken on this machine
> +.*suspend.*
> +
> +# Too unstable, machine ends up hanging after lots of Oopses
> +kms_cursor_legacy.*
> +
> +# Started hanging the machine on Linux 5.19-rc2:
> +#
> +# [IGT] kms_plane_lowres: executing
> +# [IGT] kms_plane_lowres: starting subtest pipe-F-tiling-y
> +# [IGT] kms_plane_lowres: exiting, ret=77
> +# Console: switching to colour frame buffer device 170x48
> +# rockchip-drm display-subsystem: [drm] *ERROR* flip_done timed out
> +# rockchip-drm display-subsystem: [drm] *ERROR* [CRTC:35:crtc-0] commit wait timed out
> +# BUG: spinlock bad magic on CPU#3, kms_plane_lowre/482
> +# 8<--- cut here ---
> +# Unable to handle kernel paging request at virtual address 7812078e
> +# [7812078e] *pgd=00000000
> +# Internal error: Oops: 5 [#1] SMP ARM
> +# Modules linked in:
> +# CPU: 3 PID: 482 Comm: kms_plane_lowre Tainted: G        W         5.19.0-rc2-323596-g00535de92171 #1
> +# Hardware name: Rockchip (Device Tree)
> +# Process kms_plane_lowre (pid: 482, stack limit = 0x1193ac2b)
> +#  spin_dump from do_raw_spin_lock+0xa4/0xe8
> +#  do_raw_spin_lock from wait_for_completion_timeout+0x2c/0x120
> +#  wait_for_completion_timeout from drm_crtc_commit_wait+0x18/0x7c
> +#  drm_crtc_commit_wait from drm_atomic_helper_wait_for_dependencies+0x44/0x168
> +#  drm_atomic_helper_wait_for_dependencies from commit_tail+0x34/0x180
> +#  commit_tail from drm_atomic_helper_commit+0x164/0x18c
> +#  drm_atomic_helper_commit from drm_atomic_commit+0xac/0xe4
> +#  drm_atomic_commit from drm_client_modeset_commit_atomic+0x23c/0x284
> +#  drm_client_modeset_commit_atomic from drm_client_modeset_commit_locked+0x60/0x1c8
> +#  drm_client_modeset_commit_locked from drm_client_modeset_commit+0x24/0x40
> +#  drm_client_modeset_commit from drm_fbdev_client_restore+0x58/0x94
> +#  drm_fbdev_client_restore from drm_client_dev_restore+0x70/0xbc
> +#  drm_client_dev_restore from drm_release+0xf4/0x114
> +#  drm_release from __fput+0x74/0x240
> +#  __fput from task_work_run+0x84/0xb4
> +#  task_work_run from do_exit+0x34c/0xa20
> +#  do_exit from do_group_exit+0x34/0x98
> +#  do_group_exit from __wake_up_parent+0x0/0x18
> +# Code: e595c008 12843d19 03e00000 03093168 (15940508)
> +# ---[ end trace 0000000000000000 ]---
> +# note: kms_plane_lowre[482] exited with preempt_count 1
> +# Fixing recursive fault but reboot is needed!
> +kms_plane_lowres@pipe-F-tiling-y
> +
> +# Take too long, we have only two machines, and these are very flaky
> +kms_cursor_crc.*
> +
> +# Machine is hanging in this test, so skip it
> +kms_pipe_crc_basic@disable-crc-after-crtc
> \ No newline at end of file
> diff --git a/drivers/gpu/drm/ci/xfails/rockchip-rk3399-fails.txt b/drivers/gpu/drm/ci/xfails/rockchip-rk3399-fails.txt
> new file mode 100644
> index 000000000000..6db08ba6b008
> --- /dev/null
> +++ b/drivers/gpu/drm/ci/xfails/rockchip-rk3399-fails.txt
> @@ -0,0 +1,37 @@
> +kms_color@legacy-gamma,Fail
> +kms_color@pipe-A-legacy-gamma,Fail
> +kms_color@pipe-B-legacy-gamma,Fail
> +kms_flip@basic-flip-vs-wf_vblank,Fail
> +kms_flip@blocking-wf_vblank,Fail
> +kms_flip@dpms-vs-vblank-race,Fail
> +kms_flip@flip-vs-absolute-wf_vblank,Fail
> +kms_flip@flip-vs-absolute-wf_vblank-interruptible,Fail
> +kms_flip@flip-vs-blocking-wf-vblank,Fail
> +kms_flip@flip-vs-panning,Fail
> +kms_flip@flip-vs-panning-interruptible,Fail
> +kms_flip@flip-vs-wf_vblank-interruptible,Fail
> +kms_flip@plain-flip-fb-recreate,Fail
> +kms_flip@plain-flip-fb-recreate-interruptible,Fail
> +kms_flip@plain-flip-ts-check,Fail
> +kms_flip@plain-flip-ts-check-interruptible,Fail
> +kms_flip@wf_vblank-ts-check,Fail
> +kms_flip@wf_vblank-ts-check-interruptible,Fail
> +kms_invalid_mode@int-max-clock,Fail
> +kms_plane@pixel-format,Fail
> +kms_plane@pixel-format-source-clamping,Fail
> +kms_plane@plane-panning-bottom-right,Fail
> +kms_plane@plane-panning-top-left,Fail
> +kms_plane@plane-position-covered,Fail
> +kms_plane_cursor@pipe-B-overlay-size-128,Fail
> +kms_plane_cursor@pipe-B-overlay-size-256,Fail
> +kms_plane_cursor@pipe-B-overlay-size-64,Fail
> +kms_plane_cursor@pipe-B-primary-size-128,Fail
> +kms_plane_cursor@pipe-B-primary-size-256,Fail
> +kms_plane_cursor@pipe-B-primary-size-64,Fail
> +kms_plane_cursor@pipe-B-viewport-size-128,Fail
> +kms_plane_cursor@pipe-B-viewport-size-256,Fail
> +kms_plane_cursor@pipe-B-viewport-size-64,Fail
> +kms_plane_multiple@atomic-pipe-B-tiling-none,Fail
> +kms_plane_multiple@tiling-none,Fail
> +kms_prime@basic-crc,Fail
> +kms_rmfb@close-fd,Fail
> diff --git a/drivers/gpu/drm/ci/xfails/rockchip-rk3399-flakes.txt b/drivers/gpu/drm/ci/xfails/rockchip-rk3399-flakes.txt
> new file mode 100644
> index 000000000000..4c0539b4beaf
> --- /dev/null
> +++ b/drivers/gpu/drm/ci/xfails/rockchip-rk3399-flakes.txt
> @@ -0,0 +1,23 @@
> +
> +kms_cursor_crc@.*
> +kms_flip@dpms-vs-vblank-race-interruptible
> +kms_flip@flip-vs-expired-vblank
> +kms_flip@modeset-vs-vblank-race-interruptible
> +kms_pipe_crc_basic@.*
> +kms_pipe_crc_basic@compare-crc-sanitycheck-pipe-A
> +kms_pipe_crc_basic@compare-crc-sanitycheck-pipe-B
> +kms_plane@plane-position-hole
> +kms_plane_multiple@atomic-pipe-A-tiling-none
> +kms_plane_multiple@atomic-pipe-B-tiling-none
> +kms_sequence@get-forked
> +kms_sequence@get-forked-busy
> +kms_setmode@basic
> +kms_universal_plane@universal-plane-pipe-B-functional,UnexpectedPass
> +kms_vblank@pipe-A-accuracy-idle
> +kms_vblank@pipe-A-query-busy
> +kms_vblank@pipe-A-query-forked-busy
> +kms_vblank@pipe-A-wait-idle
> +kms_vblank@pipe-B-accuracy-idle
> +kms_vblank@pipe-B-query-busy
> +kms_vblank@pipe-B-query-forked-busy
> +kms_vblank@pipe-B-wait-idle
> diff --git a/drivers/gpu/drm/ci/xfails/rockchip-rk3399-skips.txt b/drivers/gpu/drm/ci/xfails/rockchip-rk3399-skips.txt
> new file mode 100644
> index 000000000000..10c3d81a919a
> --- /dev/null
> +++ b/drivers/gpu/drm/ci/xfails/rockchip-rk3399-skips.txt
> @@ -0,0 +1,5 @@
> +# Suspend to RAM seems to be broken on this machine
> +.*suspend.*
> +
> +# Too unstable, machine ends up hanging after lots of Oopses
> +kms_cursor_legacy.*
> diff --git a/drivers/gpu/drm/ci/xfails/virtio_gpu-none-fails.txt b/drivers/gpu/drm/ci/xfails/virtio_gpu-none-fails.txt
> new file mode 100644
> index 000000000000..9586b2339f6f
> --- /dev/null
> +++ b/drivers/gpu/drm/ci/xfails/virtio_gpu-none-fails.txt
> @@ -0,0 +1,38 @@
> +kms_addfb_basic@addfb25-bad-modifier,Fail
> +kms_addfb_basic@bad-pitch-65536,Fail
> +kms_addfb_basic@bo-too-small,Fail
> +kms_addfb_basic@size-max,Fail
> +kms_addfb_basic@too-high,Fail
> +kms_atomic_transition@plane-primary-toggle-with-vblank-wait,Fail
> +kms_bw@linear-tiling-1-displays-1920x1080p,Fail
> +kms_bw@linear-tiling-1-displays-2560x1440p,Fail
> +kms_bw@linear-tiling-1-displays-3840x2160p,Fail
> +kms_bw@linear-tiling-2-displays-1920x1080p,Fail
> +kms_bw@linear-tiling-2-displays-2560x1440p,Fail
> +kms_bw@linear-tiling-2-displays-3840x2160p,Fail
> +kms_invalid_mode@int-max-clock,Fail
> +kms_plane_scaling@downscale-with-modifier-factor-0-25,Fail
> +kms_plane_scaling@downscale-with-rotation-factor-0-25,Fail
> +kms_plane_scaling@planes-upscale-20x20,Fail
> +kms_plane_scaling@planes-upscale-20x20-downscale-factor-0-25,Fail
> +kms_plane_scaling@planes-upscale-20x20-downscale-factor-0-5,Fail
> +kms_plane_scaling@planes-upscale-20x20-downscale-factor-0-75,Fail
> +kms_plane_scaling@upscale-with-modifier-20x20,Fail
> +kms_plane_scaling@upscale-with-modifier-factor-0-25,Fail
> +kms_plane_scaling@upscale-with-pixel-format-20x20,Fail
> +kms_plane_scaling@upscale-with-pixel-format-factor-0-25,Fail
> +kms_plane_scaling@upscale-with-rotation-20x20,Fail
> +kms_vblank@crtc-id,Fail
> +kms_vblank@invalid,Fail
> +kms_vblank@pipe-A-accuracy-idle,Fail
> +kms_vblank@pipe-A-query-busy,Fail
> +kms_vblank@pipe-A-query-forked,Fail
> +kms_vblank@pipe-A-query-forked-busy,Fail
> +kms_vblank@pipe-A-query-idle,Fail
> +kms_vblank@pipe-A-ts-continuation-idle,Fail
> +kms_vblank@pipe-A-ts-continuation-modeset,Fail
> +kms_vblank@pipe-A-ts-continuation-suspend,Fail
> +kms_vblank@pipe-A-wait-busy,Fail
> +kms_vblank@pipe-A-wait-forked,Fail
> +kms_vblank@pipe-A-wait-forked-busy,Fail
> +kms_vblank@pipe-A-wait-idle,Fail
> diff --git a/drivers/gpu/drm/ci/xfails/virtio_gpu-none-flakes.txt b/drivers/gpu/drm/ci/xfails/virtio_gpu-none-flakes.txt
> new file mode 100644
> index 000000000000..e69de29bb2d1
> diff --git a/drivers/gpu/drm/ci/xfails/virtio_gpu-none-skips.txt b/drivers/gpu/drm/ci/xfails/virtio_gpu-none-skips.txt
> new file mode 100644
> index 000000000000..78be18174012
> --- /dev/null
> +++ b/drivers/gpu/drm/ci/xfails/virtio_gpu-none-skips.txt
> @@ -0,0 +1,6 @@
> +# Hits a "refcount_t: underflow; use-after-free" in virtio_gpu_fence_event_process
> +# When run in a particular order with other tests
> +kms_cursor_legacy.*
> +
> +# Job just hangs without any output
> +kms_flip@flip-vs-suspend.*
> \ No newline at end of file
> diff --git a/test b/test
> new file mode 100644
> index 000000000000..e69de29bb2d1
> -- 
> 2.34.1
> 

-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch

On Tue, Aug 22, 2023 at 04:26:06PM +0200, Daniel Vetter wrote:
> On Fri, Aug 11, 2023 at 02:19:53PM -0300, Helen Koike wrote:
> > From: Tomeu Vizoso <tomeu.vizoso@collabora.com>
> > 
> > Developers can easily execute several tests on different devices
> > by just pushing their branch to their fork in a repository hosted
> > on gitlab.freedesktop.org which has an infrastructure to run jobs
> > in several runners and farms with different devices.
> > 
> > There are also other automated tools that uprev dependencies,
> > monitor the infra, and so on that are already used by the Mesa
> > project, and we can reuse them too.
> > 
> > Also, store expectations about what the DRM drivers are supposed
> > to pass in the IGT test suite. By storing the test expectations
> > along with the code, we can make sure both stay in sync with each
> > other so we can know when a code change breaks those expectations.
> > 
> > Also, include a configuration file that points to the out-of-tree
> > CI scripts.
> > 
> > This will allow all contributors to drm to reuse the infrastructure
> > already in gitlab.freedesktop.org to test the driver on several
> > generations of the hardware.
> > 
> > Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
> > Signed-off-by: Helen Koike <helen.koike@collabora.com>
> > Acked-by: Daniel Stone <daniels@collabora.com>
> > Acked-by: Rob Clark <robdclark@gmail.com>
> > Tested-by: Rob Clark <robdclark@gmail.com>
> 
> Ok I pushed this into a topic/drm-ci branch in drm.git and asked sfr to
> include that branch in linux-next.
> 
> But also I'd like to see a lot more acks here, we should be able to at
> least pile up a bunch of (driver) maintainers from drm-misc in support of
> this. Also maybe media, at least I've heard noises that they're maybe
> interested too? Plus anyone else, the more the better.

I'm not really convinced by that approach at all, and most of the issues
I see are shown by the follow-up series here:

https://lore.kernel.org/dri-devel/20230825122435.316272-1-vignesh.raman@collabora.com/

  * We hardcode a CI farm setup into the kernel

  * We cannot trust that the code being run is actually the one being
    pushed into gitlab

  * IMO, and I know we disagree here, any IGT test we enable for a given
    platform should work, period. Allowing failures and flaky tests just
    sweeps whatever issue is there under the rug. If the test is at
    fault, we should fix the test, if the driver / kernel is at fault,
    then I certainly want to know about it.

  * This then leads to patches like this one:
    https://lore.kernel.org/dri-devel/20230825122435.316272-6-vignesh.raman@collabora.com/

    Which (and it's definitely not the author's fault) are just plain
    unreadable, reproducable or auditable by anyone not heavily involved
    in the CI farm operations and the platforms being tested.

That being said, I don't have anything better to suggest than what I
already did, and it looks like I'm alone in thinking that those are
problems, so feel free to add my ack if you want to.

Maxime

On Wed, 30 Aug 2023, Maxime Ripard <mripard@kernel.org> wrote:
> On Tue, Aug 22, 2023 at 04:26:06PM +0200, Daniel Vetter wrote:
>> On Fri, Aug 11, 2023 at 02:19:53PM -0300, Helen Koike wrote:
>> > From: Tomeu Vizoso <tomeu.vizoso@collabora.com>
>> > 
>> > Developers can easily execute several tests on different devices
>> > by just pushing their branch to their fork in a repository hosted
>> > on gitlab.freedesktop.org which has an infrastructure to run jobs
>> > in several runners and farms with different devices.
>> > 
>> > There are also other automated tools that uprev dependencies,
>> > monitor the infra, and so on that are already used by the Mesa
>> > project, and we can reuse them too.
>> > 
>> > Also, store expectations about what the DRM drivers are supposed
>> > to pass in the IGT test suite. By storing the test expectations
>> > along with the code, we can make sure both stay in sync with each
>> > other so we can know when a code change breaks those expectations.
>> > 
>> > Also, include a configuration file that points to the out-of-tree
>> > CI scripts.
>> > 
>> > This will allow all contributors to drm to reuse the infrastructure
>> > already in gitlab.freedesktop.org to test the driver on several
>> > generations of the hardware.
>> > 
>> > Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
>> > Signed-off-by: Helen Koike <helen.koike@collabora.com>
>> > Acked-by: Daniel Stone <daniels@collabora.com>
>> > Acked-by: Rob Clark <robdclark@gmail.com>
>> > Tested-by: Rob Clark <robdclark@gmail.com>
>> 
>> Ok I pushed this into a topic/drm-ci branch in drm.git and asked sfr to
>> include that branch in linux-next.
>> 
>> But also I'd like to see a lot more acks here, we should be able to at
>> least pile up a bunch of (driver) maintainers from drm-misc in support of
>> this. Also maybe media, at least I've heard noises that they're maybe
>> interested too? Plus anyone else, the more the better.
>
> I'm not really convinced by that approach at all, and most of the issues
> I see are shown by the follow-up series here:

I'm not fully convinced either, more like "let's see". In that narrow
sense, ack. I don't see harm in trying, if you're also open to backing
off in case it does not pan out.

> https://lore.kernel.org/dri-devel/20230825122435.316272-1-vignesh.raman@collabora.com/
>
>   * We hardcode a CI farm setup into the kernel
>
>   * We cannot trust that the code being run is actually the one being
>     pushed into gitlab
>
>   * IMO, and I know we disagree here, any IGT test we enable for a given
>     platform should work, period. Allowing failures and flaky tests just
>     sweeps whatever issue is there under the rug. If the test is at
>     fault, we should fix the test, if the driver / kernel is at fault,
>     then I certainly want to know about it.

At least for display, where this also depends on peripheral hardware,
it's not an easy problem, really. How reliable do you need it to be?
How many nines? Who is going to debug the issues that need hundreds or
thousands of runs to reproduce? If a commit makes some test less
reliable, how long is it going to take to even see that or pinpoint
that?

It's a kind of cop out, but this is not filesystems. In many cases I
think we might be able to make things more robust by failing faster and
failing more, but the users probably want us to plunge forward despite
some errors to try to get something on screen.

(Come to think of it, perhaps we should classify tests based on whether
external hardware plays a role.)

So I'm not so concerned about the filter lists per se, but rather about
having them in kernel.

BR,
Jani.

>
>   * This then leads to patches like this one:
>     https://lore.kernel.org/dri-devel/20230825122435.316272-6-vignesh.raman@collabora.com/
>
>     Which (and it's definitely not the author's fault) are just plain
>     unreadable, reproducable or auditable by anyone not heavily involved
>     in the CI farm operations and the platforms being tested.
>
> That being said, I don't have anything better to suggest than what I
> already did, and it looks like I'm alone in thinking that those are
> problems, so feel free to add my ack if you want to.
>
> Maxime

-- 
Jani Nikula, Intel Open Source Graphics Center

On Wed, Aug 30, 2023 at 01:58:31PM +0300, Jani Nikula wrote:
> On Wed, 30 Aug 2023, Maxime Ripard <mripard@kernel.org> wrote:
> > On Tue, Aug 22, 2023 at 04:26:06PM +0200, Daniel Vetter wrote:
> >> On Fri, Aug 11, 2023 at 02:19:53PM -0300, Helen Koike wrote:
> >> > From: Tomeu Vizoso <tomeu.vizoso@collabora.com>
> >> > 
> >> > Developers can easily execute several tests on different devices
> >> > by just pushing their branch to their fork in a repository hosted
> >> > on gitlab.freedesktop.org which has an infrastructure to run jobs
> >> > in several runners and farms with different devices.
> >> > 
> >> > There are also other automated tools that uprev dependencies,
> >> > monitor the infra, and so on that are already used by the Mesa
> >> > project, and we can reuse them too.
> >> > 
> >> > Also, store expectations about what the DRM drivers are supposed
> >> > to pass in the IGT test suite. By storing the test expectations
> >> > along with the code, we can make sure both stay in sync with each
> >> > other so we can know when a code change breaks those expectations.
> >> > 
> >> > Also, include a configuration file that points to the out-of-tree
> >> > CI scripts.
> >> > 
> >> > This will allow all contributors to drm to reuse the infrastructure
> >> > already in gitlab.freedesktop.org to test the driver on several
> >> > generations of the hardware.
> >> > 
> >> > Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
> >> > Signed-off-by: Helen Koike <helen.koike@collabora.com>
> >> > Acked-by: Daniel Stone <daniels@collabora.com>
> >> > Acked-by: Rob Clark <robdclark@gmail.com>
> >> > Tested-by: Rob Clark <robdclark@gmail.com>
> >> 
> >> Ok I pushed this into a topic/drm-ci branch in drm.git and asked sfr to
> >> include that branch in linux-next.
> >> 
> >> But also I'd like to see a lot more acks here, we should be able to at
> >> least pile up a bunch of (driver) maintainers from drm-misc in support of
> >> this. Also maybe media, at least I've heard noises that they're maybe
> >> interested too? Plus anyone else, the more the better.
> >
> > I'm not really convinced by that approach at all, and most of the issues
> > I see are shown by the follow-up series here:
> 
> I'm not fully convinced either, more like "let's see". In that narrow
> sense, ack. I don't see harm in trying, if you're also open to backing
> off in case it does not pan out.
> 
> > https://lore.kernel.org/dri-devel/20230825122435.316272-1-vignesh.raman@collabora.com/
> >
> >   * We hardcode a CI farm setup into the kernel
> >
> >   * We cannot trust that the code being run is actually the one being
> >     pushed into gitlab
> >
> >   * IMO, and I know we disagree here, any IGT test we enable for a given
> >     platform should work, period. Allowing failures and flaky tests just
> >     sweeps whatever issue is there under the rug. If the test is at
> >     fault, we should fix the test, if the driver / kernel is at fault,
> >     then I certainly want to know about it.
> 
> At least for display, where this also depends on peripheral hardware,
> it's not an easy problem, really.

Aside from the Chamelium tests, which tests actually rely on peripheral
hardware? On EDID and hotplug, sure, but that can easily be set up from
the userspace, or something like

https://www.lindy-international.com/HDMI-2-0-EDID-Emulator.htm?websale8=ld0101.ld021102&pi=32115

> How reliable do you need it to be? How many nines? Who is going to
> debug the issues that need hundreds or thousands of runs to reproduce?
> If a commit makes some test less reliable, how long is it going to
> take to even see that or pinpoint that?

I mean, that's also true for failures or success then. How many times do
you need a test to run properly to qualify it as a meaningful test? How
do you know that it's not a flaky test?

Ultimately, it's about trust. If, for a given test that just failed, I
can't be certain that it's because of the branch I just submitted, I
will just ignore the tests results after a while.

This is already what plagues kernelci, and we should do better.

And I'm sorry, but if some part of the kernel or driver just isn't
reliable, then we shouldn't claim it is (except for all the times it
isn't). If no-one has the time to look into it, fine, but flagging it
under a flaky test doesn't help anyone.

Like, from that patch, how can I know what is the issue with
kms_hdmi_inject@inject-4k or kms_addfb_basic@addfb25-bad-modifier on
mt8173. I certainly can't. And neither of those have anything to do with
peripheral hardware.

Maxime

Hi all,

Thanks for you comments.

On 30/08/2023 08:37, Maxime Ripard wrote:
> On Wed, Aug 30, 2023 at 01:58:31PM +0300, Jani Nikula wrote:
>> On Wed, 30 Aug 2023, Maxime Ripard <mripard@kernel.org> wrote:
>>> On Tue, Aug 22, 2023 at 04:26:06PM +0200, Daniel Vetter wrote:
>>>> On Fri, Aug 11, 2023 at 02:19:53PM -0300, Helen Koike wrote:
>>>>> From: Tomeu Vizoso <tomeu.vizoso@collabora.com>
>>>>>
>>>>> Developers can easily execute several tests on different devices
>>>>> by just pushing their branch to their fork in a repository hosted
>>>>> on gitlab.freedesktop.org which has an infrastructure to run jobs
>>>>> in several runners and farms with different devices.
>>>>>
>>>>> There are also other automated tools that uprev dependencies,
>>>>> monitor the infra, and so on that are already used by the Mesa
>>>>> project, and we can reuse them too.
>>>>>
>>>>> Also, store expectations about what the DRM drivers are supposed
>>>>> to pass in the IGT test suite. By storing the test expectations
>>>>> along with the code, we can make sure both stay in sync with each
>>>>> other so we can know when a code change breaks those expectations.
>>>>>
>>>>> Also, include a configuration file that points to the out-of-tree
>>>>> CI scripts.
>>>>>
>>>>> This will allow all contributors to drm to reuse the infrastructure
>>>>> already in gitlab.freedesktop.org to test the driver on several
>>>>> generations of the hardware.
>>>>>
>>>>> Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
>>>>> Signed-off-by: Helen Koike <helen.koike@collabora.com>
>>>>> Acked-by: Daniel Stone <daniels@collabora.com>
>>>>> Acked-by: Rob Clark <robdclark@gmail.com>
>>>>> Tested-by: Rob Clark <robdclark@gmail.com>
>>>>
>>>> Ok I pushed this into a topic/drm-ci branch in drm.git and asked sfr to
>>>> include that branch in linux-next.
>>>>
>>>> But also I'd like to see a lot more acks here, we should be able to at
>>>> least pile up a bunch of (driver) maintainers from drm-misc in support of
>>>> this. Also maybe media, at least I've heard noises that they're maybe
>>>> interested too? Plus anyone else, the more the better.
>>>
>>> I'm not really convinced by that approach at all, and most of the issues
>>> I see are shown by the follow-up series here:
>>
>> I'm not fully convinced either, more like "let's see". In that narrow
>> sense, ack. I don't see harm in trying, if you're also open to backing
>> off in case it does not pan out.
>>
>>> https://lore.kernel.org/dri-devel/20230825122435.316272-1-vignesh.raman@collabora.com/
>>>
>>>    * We hardcode a CI farm setup into the kernel


These could be out of tree.

There is a version outside the kernel tree where you just point the CI
configuration to a url:
https://gitlab.freedesktop.org/gfx-ci/drm-ci/-/merge_requests/1

We were discussing it here 
https://www.linuxtv.org/cgi-bin/mailman/private/linuxtv-ci/2023-August/000027.html

(I guess Sima's reply didn't got into the mailing list) but the argument 
of not having out of tree repo is due to historical bad experience of 
having to sync the kernel with the code and it can become messy.


>>>
>>>    * We cannot trust that the code being run is actually the one being
>>>      pushed into gitlab


We can improve this if this is a requirement.
For DTS configuration we can work with overlays (which is the current 
modification on that patchset). For other changes that are not suitable 
to upstream (and should be rare) we can see if we work with the 
`-external-fixes` approach or another approach, we can check it case by 
case to understand why it is not suitable for upstream.


>>>
>>>    * IMO, and I know we disagree here, any IGT test we enable for a given
>>>      platform should work, period. Allowing failures and flaky tests just
>>>      sweeps whatever issue is there under the rug. If the test is at
>>>      fault, we should fix the test, if the driver / kernel is at fault,
>>>      then I certainly want to know about it.


I believe we need a baseline and understand the current status of tests. 
If you check the xfails folder in the patch you can see that I had to 
add a few tests on *-skips.txt since those tests crashes the system and 
other on *-fails.txt that are consistently not passing.

Since the "any IGT test we enable for a given platform should work" is 
not a reality atm, we need to have a clear view about which tests are 
not corresponding to it, so we can start fixing. First we need to be 
aware of the issues so we can start fixing them, otherwise we will stay 
in the "no tests no failures" ground :)


>>
>> At least for display, where this also depends on peripheral hardware,
>> it's not an easy problem, really.
> 
> Aside from the Chamelium tests, which tests actually rely on peripheral
> hardware? On EDID and hotplug, sure, but that can easily be set up from
> the userspace, or something like
> 
> https://www.lindy-international.com/HDMI-2-0-EDID-Emulator.htm?websale8=ld0101.ld021102&pi=32115
> 
>> How reliable do you need it to be? How many nines? Who is going to
>> debug the issues that need hundreds or thousands of runs to reproduce?
>> If a commit makes some test less reliable, how long is it going to
>> take to even see that or pinpoint that?
> 
> I mean, that's also true for failures or success then. How many times do
> you need a test to run properly to qualify it as a meaningful test? How
> do you know that it's not a flaky test?
> 
> Ultimately, it's about trust. If, for a given test that just failed, I
> can't be certain that it's because of the branch I just submitted, I
> will just ignore the tests results after a while.
> 
> This is already what plagues kernelci, and we should do better.


This is something that is really nice on Mesa3D, a patch only gets 
merged if tests passes, which forces people to not ignore it, which 
forces the code to be fixed and the CI to be constantly maintained.

Of course there are bad days there, but there is real value. Nice thread 
to check: https://gitlab.freedesktop.org/mesa/mesa/-/issues/8635 (thanks 
Alyssa for the feedback).


> 
> And I'm sorry, but if some part of the kernel or driver just isn't
> reliable, then we shouldn't claim it is (except for all the times it
> isn't). If no-one has the time to look into it, fine, but flagging it
> under a flaky test doesn't help anyone.


At least we would know what is there that isn't reliable.
We could also define policies like: failing one out of 10 is fine. And 
we can have scripts that do a stress test and report back.
It is really nice for users to know this kind of status of drivers.


Thanks for your feedback, please keep them coming :)

Regards,
Helen


> 
> Like, from that patch, how can I know what is the issue with
> kms_hdmi_inject@inject-4k or kms_addfb_basic@addfb25-bad-modifier on
> mt8173. I certainly can't. And neither of those have anything to do with
> peripheral hardware.
> 
> Maxime

On Wed, Aug 30, 2023 at 10:24:49AM -0300, Helen Koike wrote:
> Hi all,
> 
> Thanks for you comments.
> 
> On 30/08/2023 08:37, Maxime Ripard wrote:
> > On Wed, Aug 30, 2023 at 01:58:31PM +0300, Jani Nikula wrote:
> > > On Wed, 30 Aug 2023, Maxime Ripard <mripard@kernel.org> wrote:
> > > > On Tue, Aug 22, 2023 at 04:26:06PM +0200, Daniel Vetter wrote:
> > > > > On Fri, Aug 11, 2023 at 02:19:53PM -0300, Helen Koike wrote:
> > > > > > From: Tomeu Vizoso <tomeu.vizoso@collabora.com>
> > > > > > 
> > > > > > Developers can easily execute several tests on different devices
> > > > > > by just pushing their branch to their fork in a repository hosted
> > > > > > on gitlab.freedesktop.org which has an infrastructure to run jobs
> > > > > > in several runners and farms with different devices.
> > > > > > 
> > > > > > There are also other automated tools that uprev dependencies,
> > > > > > monitor the infra, and so on that are already used by the Mesa
> > > > > > project, and we can reuse them too.
> > > > > > 
> > > > > > Also, store expectations about what the DRM drivers are supposed
> > > > > > to pass in the IGT test suite. By storing the test expectations
> > > > > > along with the code, we can make sure both stay in sync with each
> > > > > > other so we can know when a code change breaks those expectations.
> > > > > > 
> > > > > > Also, include a configuration file that points to the out-of-tree
> > > > > > CI scripts.
> > > > > > 
> > > > > > This will allow all contributors to drm to reuse the infrastructure
> > > > > > already in gitlab.freedesktop.org to test the driver on several
> > > > > > generations of the hardware.
> > > > > > 
> > > > > > Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
> > > > > > Signed-off-by: Helen Koike <helen.koike@collabora.com>
> > > > > > Acked-by: Daniel Stone <daniels@collabora.com>
> > > > > > Acked-by: Rob Clark <robdclark@gmail.com>
> > > > > > Tested-by: Rob Clark <robdclark@gmail.com>
> > > > > 
> > > > > Ok I pushed this into a topic/drm-ci branch in drm.git and asked sfr to
> > > > > include that branch in linux-next.
> > > > > 
> > > > > But also I'd like to see a lot more acks here, we should be able to at
> > > > > least pile up a bunch of (driver) maintainers from drm-misc in support of
> > > > > this. Also maybe media, at least I've heard noises that they're maybe
> > > > > interested too? Plus anyone else, the more the better.
> > > > 
> > > > I'm not really convinced by that approach at all, and most of the issues
> > > > I see are shown by the follow-up series here:
> > > 
> > > I'm not fully convinced either, more like "let's see". In that narrow
> > > sense, ack. I don't see harm in trying, if you're also open to backing
> > > off in case it does not pan out.
> > > 
> > > > https://lore.kernel.org/dri-devel/20230825122435.316272-1-vignesh.raman@collabora.com/
> > > > 
> > > >    * We hardcode a CI farm setup into the kernel
> 
> 
> These could be out of tree.
> 
> There is a version outside the kernel tree where you just point the CI
> configuration to a url:
> https://gitlab.freedesktop.org/gfx-ci/drm-ci/-/merge_requests/1
> 
> We were discussing it here https://www.linuxtv.org/cgi-bin/mailman/private/linuxtv-ci/2023-August/000027.html

It looks like it's private

> (I guess Sima's reply didn't got into the mailing list) but the argument of
> not having out of tree repo is due to historical bad experience of having to
> sync the kernel with the code and it can become messy.

My point is that even though the test strategy might be considered a
"property" of the kernel, how you execute it is definitely not and you
will have as many setups as you have CI farms. You can't put that into
the kernel, just like we don't put the kernel command line in it for
example.

> > > > 
> > > >    * We cannot trust that the code being run is actually the one being
> > > >      pushed into gitlab
> 
> 
> We can improve this if this is a requirement.
> For DTS configuration we can work with overlays (which is the current
> modification on that patchset). For other changes that are not suitable to
> upstream (and should be rare) we can see if we work with the
> `-external-fixes` approach or another approach, we can check it case by case
> to understand why it is not suitable for upstream.

The existence of that branch in itself is an issue to me. Again, it's a
matter of trust. How can I trust a branch I barely know about, of which
the development is not clear and isn't reviewed by any of the
maintainers of the code that might affect the test outcomes.

Or put another way, if I run the tests on my machine, it won't work. Why
should it work on the CI farm? The branch itself is broken. It might not
be due to any of the work I did, but it's broken still.

> > > >
> > > >    * IMO, and I know we disagree here, any IGT test we enable for a given
> > > >      platform should work, period. Allowing failures and flaky tests just
> > > >      sweeps whatever issue is there under the rug. If the test is at
> > > >      fault, we should fix the test, if the driver / kernel is at fault,
> > > >      then I certainly want to know about it.
>
> I believe we need a baseline and understand the current status of tests. If
> you check the xfails folder in the patch you can see that I had to add a few
> tests on *-skips.txt since those tests crashes the system and other on
> *-fails.txt that are consistently not passing.

I agree that we need a baseline, but that baseline should be defined by
the tests own merits, not their outcome on a particular platform.

In other words, I want all drivers to follow that baseline, and if they
don't it's a bug we should fix, and we should be vocal about it. We
shouldn't ignore the test because it's broken.

Going back to the example I used previously, kms_hdmi_inject@inject-4k
shouldn't fail on mt8173, ever. That's a bug. Ignoring it and reporting
that "all tests are good" isn't ok. There's something wrong with that
driver and we should fix it.

Or at the very least, explain in much details what is the breakage, how
we noticed it, why we can't fix it, and how to reproduce it.

Because in its current state, there's no chance we'll ever go over that
test list and remove some of them. Or even know if, if we ever fix a bug
somewhere, we should remove a flaky or failing test.

> Since the "any IGT test we enable for a given platform should work" is not a
> reality atm,

Thanks for the reality check, but it's very much doable: we're in
control of the test suite.

> we need to have a clear view about which tests are not corresponding
> to it, so we can start fixing. First we need to be aware of the issues
> so we can start fixing them, otherwise we will stay in the "no tests
> no failures" ground :)

I think we have somewhat contradicting goals. You want to make
regression testing, so whatever test used to work in the past should
keep working. That's fine, but it's different from "expectations about
what the DRM drivers are supposed to pass in the IGT test suite" which
is about validation, ie "all KMS drivers must behave this way".

I guess for regression you very much would like that all-green
dashboard, and it's understandable. For validation, we don't care and we
should be as vocal as possible to report broken drivers.

Eventually, we should have regression testing over the validation test
suite.

It's not about reality. We should be clear what we expect from those
test suites, and not claim that it's something it's not.

> > > At least for display, where this also depends on peripheral hardware,
> > > it's not an easy problem, really.
> > 
> > Aside from the Chamelium tests, which tests actually rely on peripheral
> > hardware? On EDID and hotplug, sure, but that can easily be set up from
> > the userspace, or something like
> > 
> > https://www.lindy-international.com/HDMI-2-0-EDID-Emulator.htm?websale8=ld0101.ld021102&pi=32115
> > 
> > > How reliable do you need it to be? How many nines? Who is going to
> > > debug the issues that need hundreds or thousands of runs to reproduce?
> > > If a commit makes some test less reliable, how long is it going to
> > > take to even see that or pinpoint that?
> > 
> > I mean, that's also true for failures or success then. How many times do
> > you need a test to run properly to qualify it as a meaningful test? How
> > do you know that it's not a flaky test?
> > 
> > Ultimately, it's about trust. If, for a given test that just failed, I
> > can't be certain that it's because of the branch I just submitted, I
> > will just ignore the tests results after a while.
> > 
> > This is already what plagues kernelci, and we should do better.
>
> This is something that is really nice on Mesa3D, a patch only gets merged if
> tests passes, which forces people to not ignore it, which forces the code to
> be fixed and the CI to be constantly maintained.
> 
> Of course there are bad days there, but there is real value. Nice thread to
> check: https://gitlab.freedesktop.org/mesa/mesa/-/issues/8635 (thanks Alyssa
> for the feedback).

I'm sure it works great for Mesa, but I'm also sure it doesn't ignore
CTS reports that a particular device isn't a valid OpenGL or Vulkan
implementation anymore.

> > And I'm sorry, but if some part of the kernel or driver just isn't
> > reliable, then we shouldn't claim it is (except for all the times it
> > isn't). If no-one has the time to look into it, fine, but flagging it
> > under a flaky test doesn't help anyone.
>
> At least we would know what is there that isn't reliable.

We would too if the test was reported as failed. But our preferred
approach to do so diverge.

Maxime

On 30/08/2023 11:57, Maxime Ripard wrote:
> On Wed, Aug 30, 2023 at 10:24:49AM -0300, Helen Koike wrote:
>> Hi all,
>>
>> Thanks for you comments.
>>
>> On 30/08/2023 08:37, Maxime Ripard wrote:
>>> On Wed, Aug 30, 2023 at 01:58:31PM +0300, Jani Nikula wrote:
>>>> On Wed, 30 Aug 2023, Maxime Ripard <mripard@kernel.org> wrote:
>>>>> On Tue, Aug 22, 2023 at 04:26:06PM +0200, Daniel Vetter wrote:
>>>>>> On Fri, Aug 11, 2023 at 02:19:53PM -0300, Helen Koike wrote:
>>>>>>> From: Tomeu Vizoso <tomeu.vizoso@collabora.com>
>>>>>>>
>>>>>>> Developers can easily execute several tests on different devices
>>>>>>> by just pushing their branch to their fork in a repository hosted
>>>>>>> on gitlab.freedesktop.org which has an infrastructure to run jobs
>>>>>>> in several runners and farms with different devices.
>>>>>>>
>>>>>>> There are also other automated tools that uprev dependencies,
>>>>>>> monitor the infra, and so on that are already used by the Mesa
>>>>>>> project, and we can reuse them too.
>>>>>>>
>>>>>>> Also, store expectations about what the DRM drivers are supposed
>>>>>>> to pass in the IGT test suite. By storing the test expectations
>>>>>>> along with the code, we can make sure both stay in sync with each
>>>>>>> other so we can know when a code change breaks those expectations.
>>>>>>>
>>>>>>> Also, include a configuration file that points to the out-of-tree
>>>>>>> CI scripts.
>>>>>>>
>>>>>>> This will allow all contributors to drm to reuse the infrastructure
>>>>>>> already in gitlab.freedesktop.org to test the driver on several
>>>>>>> generations of the hardware.
>>>>>>>
>>>>>>> Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
>>>>>>> Signed-off-by: Helen Koike <helen.koike@collabora.com>
>>>>>>> Acked-by: Daniel Stone <daniels@collabora.com>
>>>>>>> Acked-by: Rob Clark <robdclark@gmail.com>
>>>>>>> Tested-by: Rob Clark <robdclark@gmail.com>
>>>>>>
>>>>>> Ok I pushed this into a topic/drm-ci branch in drm.git and asked sfr to
>>>>>> include that branch in linux-next.
>>>>>>
>>>>>> But also I'd like to see a lot more acks here, we should be able to at
>>>>>> least pile up a bunch of (driver) maintainers from drm-misc in support of
>>>>>> this. Also maybe media, at least I've heard noises that they're maybe
>>>>>> interested too? Plus anyone else, the more the better.
>>>>>
>>>>> I'm not really convinced by that approach at all, and most of the issues
>>>>> I see are shown by the follow-up series here:
>>>>
>>>> I'm not fully convinced either, more like "let's see". In that narrow
>>>> sense, ack. I don't see harm in trying, if you're also open to backing
>>>> off in case it does not pan out.
>>>>
>>>>> https://lore.kernel.org/dri-devel/20230825122435.316272-1-vignesh.raman@collabora.com/
>>>>>
>>>>>     * We hardcode a CI farm setup into the kernel
>>
>>
>> These could be out of tree.
>>
>> There is a version outside the kernel tree where you just point the CI
>> configuration to a url:
>> https://gitlab.freedesktop.org/gfx-ci/drm-ci/-/merge_requests/1
>>
>> We were discussing it here https://www.linuxtv.org/cgi-bin/mailman/private/linuxtv-ci/2023-August/000027.html
> 
> It looks like it's private
> 
>> (I guess Sima's reply didn't got into the mailing list) but the argument of
>> not having out of tree repo is due to historical bad experience of having to
>> sync the kernel with the code and it can become messy.
> 
> My point is that even though the test strategy might be considered a
> "property" of the kernel, how you execute it is definitely not and you
> will have as many setups as you have CI farms. You can't put that into
> the kernel, just like we don't put the kernel command line in it for
> example. >
>>>>>
>>>>>     * We cannot trust that the code being run is actually the one being
>>>>>       pushed into gitlab
>>
>>
>> We can improve this if this is a requirement.
>> For DTS configuration we can work with overlays (which is the current
>> modification on that patchset). For other changes that are not suitable to
>> upstream (and should be rare) we can see if we work with the
>> `-external-fixes` approach or another approach, we can check it case by case
>> to understand why it is not suitable for upstream.
> 
> The existence of that branch in itself is an issue to me. Again, it's a
> matter of trust. How can I trust a branch I barely know about, of which
> the development is not clear and isn't reviewed by any of the
> maintainers of the code that might affect the test outcomes.
> 
> Or put another way, if I run the tests on my machine, it won't work. Why
> should it work on the CI farm? The branch itself is broken. It might not
> be due to any of the work I did, but it's broken still.
> 
>>>>>
>>>>>     * IMO, and I know we disagree here, any IGT test we enable for a given
>>>>>       platform should work, period. Allowing failures and flaky tests just
>>>>>       sweeps whatever issue is there under the rug. If the test is at
>>>>>       fault, we should fix the test, if the driver / kernel is at fault,
>>>>>       then I certainly want to know about it.
>>
>> I believe we need a baseline and understand the current status of tests. If
>> you check the xfails folder in the patch you can see that I had to add a few
>> tests on *-skips.txt since those tests crashes the system and other on
>> *-fails.txt that are consistently not passing.
> 
> I agree that we need a baseline, but that baseline should be defined by
> the tests own merits, not their outcome on a particular platform.
> 
> In other words, I want all drivers to follow that baseline, and if they
> don't it's a bug we should fix, and we should be vocal about it. We
> shouldn't ignore the test because it's broken.
> 
> Going back to the example I used previously, kms_hdmi_inject@inject-4k
> shouldn't fail on mt8173, ever. That's a bug. Ignoring it and reporting
> that "all tests are good" isn't ok. There's something wrong with that
> driver and we should fix it.
> 
> Or at the very least, explain in much details what is the breakage, how
> we noticed it, why we can't fix it, and how to reproduce it.
> 
> Because in its current state, there's no chance we'll ever go over that
> test list and remove some of them. Or even know if, if we ever fix a bug
> somewhere, we should remove a flaky or failing test.
> 
>> Since the "any IGT test we enable for a given platform should work" is not a
>> reality atm,
> 
> Thanks for the reality check, but it's very much doable: we're in
> control of the test suite.
> 
>> we need to have a clear view about which tests are not corresponding
>> to it, so we can start fixing. First we need to be aware of the issues
>> so we can start fixing them, otherwise we will stay in the "no tests
>> no failures" ground :)
> 
> I think we have somewhat contradicting goals. You want to make
> regression testing, so whatever test used to work in the past should
> keep working. That's fine, but it's different from "expectations about
> what the DRM drivers are supposed to pass in the IGT test suite" which
> is about validation, ie "all KMS drivers must behave this way".

I see. Indeed, for me it is more about regression testing.

We could have a configuration where developers could choose to run 
regression tests or overall validation (but I understand this is not the 
point, but just saying we could have both somehow).

We could have some policy: if you want to enable a certain device in the 
CI, you need to make sure it passes all tests first to force people to 
go fix the issues, but maybe it would be a big barrier.

I'm afraid that, if a test fail (and it is a clear bug), people would 
just say "work for most of the cases, this is not a priority to fix" and 
just start ignoring the CI, this is why I think regression tests is a 
good way to start with.

Anyway, just brain storming :)
I really hope we can find a nice useful solution for the community.

Regards,
Helen

> 
> I guess for regression you very much would like that all-green
> dashboard, and it's understandable. For validation, we don't care and we
> should be as vocal as possible to report broken drivers.
> 
> Eventually, we should have regression testing over the validation test
> suite.
> 
> It's not about reality. We should be clear what we expect from those
> test suites, and not claim that it's something it's not.
> 
>>>> At least for display, where this also depends on peripheral hardware,
>>>> it's not an easy problem, really.
>>>
>>> Aside from the Chamelium tests, which tests actually rely on peripheral
>>> hardware? On EDID and hotplug, sure, but that can easily be set up from
>>> the userspace, or something like
>>>
>>> https://www.lindy-international.com/HDMI-2-0-EDID-Emulator.htm?websale8=ld0101.ld021102&pi=32115
>>>
>>>> How reliable do you need it to be? How many nines? Who is going to
>>>> debug the issues that need hundreds or thousands of runs to reproduce?
>>>> If a commit makes some test less reliable, how long is it going to
>>>> take to even see that or pinpoint that?
>>>
>>> I mean, that's also true for failures or success then. How many times do
>>> you need a test to run properly to qualify it as a meaningful test? How
>>> do you know that it's not a flaky test?
>>>
>>> Ultimately, it's about trust. If, for a given test that just failed, I
>>> can't be certain that it's because of the branch I just submitted, I
>>> will just ignore the tests results after a while.
>>>
>>> This is already what plagues kernelci, and we should do better.
>>
>> This is something that is really nice on Mesa3D, a patch only gets merged if
>> tests passes, which forces people to not ignore it, which forces the code to
>> be fixed and the CI to be constantly maintained.
>>
>> Of course there are bad days there, but there is real value. Nice thread to
>> check: https://gitlab.freedesktop.org/mesa/mesa/-/issues/8635 (thanks Alyssa
>> for the feedback).
> 
> I'm sure it works great for Mesa, but I'm also sure it doesn't ignore
> CTS reports that a particular device isn't a valid OpenGL or Vulkan
> implementation anymore.
> 
>>> And I'm sorry, but if some part of the kernel or driver just isn't
>>> reliable, then we shouldn't claim it is (except for all the times it
>>> isn't). If no-one has the time to look into it, fine, but flagging it
>>> under a flaky test doesn't help anyone.
>>
>> At least we would know what is there that isn't reliable.
> 
> We would too if the test was reported as failed. But our preferred
> approach to do so diverge.
> 
> Maxime

On Wed, 30 Aug 2023 at 17:14, Helen Koike <helen.koike@collabora.com> wrote:
>
>
>
> On 30/08/2023 11:57, Maxime Ripard wrote:
> > On Wed, Aug 30, 2023 at 10:24:49AM -0300, Helen Koike wrote:
> >> Hi all,
> >>
> >> Thanks for you comments.
> >>
> >> On 30/08/2023 08:37, Maxime Ripard wrote:
> >>> On Wed, Aug 30, 2023 at 01:58:31PM +0300, Jani Nikula wrote:
> >>>> On Wed, 30 Aug 2023, Maxime Ripard <mripard@kernel.org> wrote:
> >>>>> On Tue, Aug 22, 2023 at 04:26:06PM +0200, Daniel Vetter wrote:
> >>>>>> On Fri, Aug 11, 2023 at 02:19:53PM -0300, Helen Koike wrote:
> >>>>>>> From: Tomeu Vizoso <tomeu.vizoso@collabora.com>
> >>>>>>>
> >>>>>>> Developers can easily execute several tests on different devices
> >>>>>>> by just pushing their branch to their fork in a repository hosted
> >>>>>>> on gitlab.freedesktop.org which has an infrastructure to run jobs
> >>>>>>> in several runners and farms with different devices.
> >>>>>>>
> >>>>>>> There are also other automated tools that uprev dependencies,
> >>>>>>> monitor the infra, and so on that are already used by the Mesa
> >>>>>>> project, and we can reuse them too.
> >>>>>>>
> >>>>>>> Also, store expectations about what the DRM drivers are supposed
> >>>>>>> to pass in the IGT test suite. By storing the test expectations
> >>>>>>> along with the code, we can make sure both stay in sync with each
> >>>>>>> other so we can know when a code change breaks those expectations.
> >>>>>>>
> >>>>>>> Also, include a configuration file that points to the out-of-tree
> >>>>>>> CI scripts.
> >>>>>>>
> >>>>>>> This will allow all contributors to drm to reuse the infrastructure
> >>>>>>> already in gitlab.freedesktop.org to test the driver on several
> >>>>>>> generations of the hardware.
> >>>>>>>
> >>>>>>> Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
> >>>>>>> Signed-off-by: Helen Koike <helen.koike@collabora.com>
> >>>>>>> Acked-by: Daniel Stone <daniels@collabora.com>
> >>>>>>> Acked-by: Rob Clark <robdclark@gmail.com>
> >>>>>>> Tested-by: Rob Clark <robdclark@gmail.com>
> >>>>>>
> >>>>>> Ok I pushed this into a topic/drm-ci branch in drm.git and asked sfr to
> >>>>>> include that branch in linux-next.
> >>>>>>
> >>>>>> But also I'd like to see a lot more acks here, we should be able to at
> >>>>>> least pile up a bunch of (driver) maintainers from drm-misc in support of
> >>>>>> this. Also maybe media, at least I've heard noises that they're maybe
> >>>>>> interested too? Plus anyone else, the more the better.
> >>>>>
> >>>>> I'm not really convinced by that approach at all, and most of the issues
> >>>>> I see are shown by the follow-up series here:
> >>>>
> >>>> I'm not fully convinced either, more like "let's see". In that narrow
> >>>> sense, ack. I don't see harm in trying, if you're also open to backing
> >>>> off in case it does not pan out.
> >>>>
> >>>>> https://lore.kernel.org/dri-devel/20230825122435.316272-1-vignesh.raman@collabora.com/
> >>>>>
> >>>>>     * We hardcode a CI farm setup into the kernel
> >>
> >>
> >> These could be out of tree.
> >>
> >> There is a version outside the kernel tree where you just point the CI
> >> configuration to a url:
> >> https://gitlab.freedesktop.org/gfx-ci/drm-ci/-/merge_requests/1
> >>
> >> We were discussing it here https://www.linuxtv.org/cgi-bin/mailman/private/linuxtv-ci/2023-August/000027.html
> >
> > It looks like it's private
> >
> >> (I guess Sima's reply didn't got into the mailing list) but the argument of
> >> not having out of tree repo is due to historical bad experience of having to
> >> sync the kernel with the code and it can become messy.
> >
> > My point is that even though the test strategy might be considered a
> > "property" of the kernel, how you execute it is definitely not and you
> > will have as many setups as you have CI farms. You can't put that into
> > the kernel, just like we don't put the kernel command line in it for
> > example. >
> >>>>>
> >>>>>     * We cannot trust that the code being run is actually the one being
> >>>>>       pushed into gitlab
> >>
> >>
> >> We can improve this if this is a requirement.
> >> For DTS configuration we can work with overlays (which is the current
> >> modification on that patchset). For other changes that are not suitable to
> >> upstream (and should be rare) we can see if we work with the
> >> `-external-fixes` approach or another approach, we can check it case by case
> >> to understand why it is not suitable for upstream.
> >
> > The existence of that branch in itself is an issue to me. Again, it's a
> > matter of trust. How can I trust a branch I barely know about, of which
> > the development is not clear and isn't reviewed by any of the
> > maintainers of the code that might affect the test outcomes.
> >
> > Or put another way, if I run the tests on my machine, it won't work. Why
> > should it work on the CI farm? The branch itself is broken. It might not
> > be due to any of the work I did, but it's broken still.
> >
> >>>>>
> >>>>>     * IMO, and I know we disagree here, any IGT test we enable for a given
> >>>>>       platform should work, period. Allowing failures and flaky tests just
> >>>>>       sweeps whatever issue is there under the rug. If the test is at
> >>>>>       fault, we should fix the test, if the driver / kernel is at fault,
> >>>>>       then I certainly want to know about it.
> >>
> >> I believe we need a baseline and understand the current status of tests. If
> >> you check the xfails folder in the patch you can see that I had to add a few
> >> tests on *-skips.txt since those tests crashes the system and other on
> >> *-fails.txt that are consistently not passing.
> >
> > I agree that we need a baseline, but that baseline should be defined by
> > the tests own merits, not their outcome on a particular platform.
> >
> > In other words, I want all drivers to follow that baseline, and if they
> > don't it's a bug we should fix, and we should be vocal about it. We
> > shouldn't ignore the test because it's broken.
> >
> > Going back to the example I used previously, kms_hdmi_inject@inject-4k
> > shouldn't fail on mt8173, ever. That's a bug. Ignoring it and reporting
> > that "all tests are good" isn't ok. There's something wrong with that
> > driver and we should fix it.
> >
> > Or at the very least, explain in much details what is the breakage, how
> > we noticed it, why we can't fix it, and how to reproduce it.
> >
> > Because in its current state, there's no chance we'll ever go over that
> > test list and remove some of them. Or even know if, if we ever fix a bug
> > somewhere, we should remove a flaky or failing test.
> >
> >> Since the "any IGT test we enable for a given platform should work" is not a
> >> reality atm,
> >
> > Thanks for the reality check, but it's very much doable: we're in
> > control of the test suite.
> >
> >> we need to have a clear view about which tests are not corresponding
> >> to it, so we can start fixing. First we need to be aware of the issues
> >> so we can start fixing them, otherwise we will stay in the "no tests
> >> no failures" ground :)
> >
> > I think we have somewhat contradicting goals. You want to make
> > regression testing, so whatever test used to work in the past should
> > keep working. That's fine, but it's different from "expectations about
> > what the DRM drivers are supposed to pass in the IGT test suite" which
> > is about validation, ie "all KMS drivers must behave this way".
>
> I see. Indeed, for me it is more about regression testing.
>
> We could have a configuration where developers could choose to run
> regression tests or overall validation (but I understand this is not the
> point, but just saying we could have both somehow).
>
> We could have some policy: if you want to enable a certain device in the
> CI, you need to make sure it passes all tests first to force people to
> go fix the issues, but maybe it would be a big barrier.
>
> I'm afraid that, if a test fail (and it is a clear bug), people would
> just say "work for most of the cases, this is not a priority to fix" and
> just start ignoring the CI, this is why I think regression tests is a
> good way to start with.
>
> Anyway, just brain storming :)
> I really hope we can find a nice useful solution for the community.

I think eventually we need to get to both goals, but currently driver
and test quality just isn't remotely there.

I think a good approach would be if CI work focuses on the pure sw
tests first, so kunit and running igt against vgem/vkms. And then we
could use that to polish a set of must-pass igt testcases, which also
drivers in general are supposed to pass. Plus ideally weed out the bad
igts that aren't reliable enough or have bad assumptions.

For hardware I think it will take a very long time until we get to a
point where CI can work without a test result list, we're nowhere
close to that. But for virtual driver this really should be
achievable, albeit with a huge amount of effort required to get there
I think.

Cheers, Sima

> Regards,
> Helen
>
> >
> > I guess for regression you very much would like that all-green
> > dashboard, and it's understandable. For validation, we don't care and we
> > should be as vocal as possible to report broken drivers.
> >
> > Eventually, we should have regression testing over the validation test
> > suite.
> >
> > It's not about reality. We should be clear what we expect from those
> > test suites, and not claim that it's something it's not.
> >
> >>>> At least for display, where this also depends on peripheral hardware,
> >>>> it's not an easy problem, really.
> >>>
> >>> Aside from the Chamelium tests, which tests actually rely on peripheral
> >>> hardware? On EDID and hotplug, sure, but that can easily be set up from
> >>> the userspace, or something like
> >>>
> >>> https://www.lindy-international.com/HDMI-2-0-EDID-Emulator.htm?websale8=ld0101.ld021102&pi=32115
> >>>
> >>>> How reliable do you need it to be? How many nines? Who is going to
> >>>> debug the issues that need hundreds or thousands of runs to reproduce?
> >>>> If a commit makes some test less reliable, how long is it going to
> >>>> take to even see that or pinpoint that?
> >>>
> >>> I mean, that's also true for failures or success then. How many times do
> >>> you need a test to run properly to qualify it as a meaningful test? How
> >>> do you know that it's not a flaky test?
> >>>
> >>> Ultimately, it's about trust. If, for a given test that just failed, I
> >>> can't be certain that it's because of the branch I just submitted, I
> >>> will just ignore the tests results after a while.
> >>>
> >>> This is already what plagues kernelci, and we should do better.
> >>
> >> This is something that is really nice on Mesa3D, a patch only gets merged if
> >> tests passes, which forces people to not ignore it, which forces the code to
> >> be fixed and the CI to be constantly maintained.
> >>
> >> Of course there are bad days there, but there is real value. Nice thread to
> >> check: https://gitlab.freedesktop.org/mesa/mesa/-/issues/8635 (thanks Alyssa
> >> for the feedback).
> >
> > I'm sure it works great for Mesa, but I'm also sure it doesn't ignore
> > CTS reports that a particular device isn't a valid OpenGL or Vulkan
> > implementation anymore.
> >
> >>> And I'm sorry, but if some part of the kernel or driver just isn't
> >>> reliable, then we shouldn't claim it is (except for all the times it
> >>> isn't). If no-one has the time to look into it, fine, but flagging it
> >>> under a flaky test doesn't help anyone.
> >>
> >> At least we would know what is there that isn't reliable.
> >
> > We would too if the test was reported as failed. But our preferred
> > approach to do so diverge.
> >
> > Maxime



-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch

Hi,

On 04/09/2023 09:54, Daniel Vetter wrote:
> On Wed, 30 Aug 2023 at 17:14, Helen Koike <helen.koike@collabora.com>  > wrote: >> >> On 30/08/2023 11:57, Maxime Ripard wrote: >>> >>> I 
agree that we need a baseline, but that baseline should be >>> defined 
by the tests own merits, not their outcome on a >>> particular platform. 
 >>> >>> In other words, I want all drivers to follow that baseline, and 
 >>> if they don't it's a bug we should fix, and we should be vocal >>> 
about it. We shouldn't ignore the test because it's broken. >>> >>> 
Going back to the example I used previously, >>> 
kms_hdmi_inject@inject-4k shouldn't fail on mt8173, ever. That's >>> a 
bug. Ignoring it and reporting that "all tests are good" isn't >>> ok. 
There's something wrong with that driver and we should fix >>> it. >>> 
 >>> Or at the very least, explain in much details what is the >>> 
breakage, how we noticed it, why we can't fix it, and how to >>> 
reproduce it. >>> >>> Because in its current state, there's no chance 
we'll ever go >>> over that test list and remove some of them. Or even 
know if, if >>> we ever fix a bug somewhere, we should remove a flaky or 
failing >>> test. >>> >>> [...] >>> >>>> we need to have a clear view 
about which tests are not >>>> corresponding to it, so we can start 
fixing. First we need to >>>> be aware of the issues so we can start 
fixing them, otherwise >>>> we will stay in the "no tests no failures" 
ground :) >>> >>> I think we have somewhat contradicting goals. You want 
to make >>> regression testing, so whatever test used to work in the 
past >>> should keep working. That's fine, but it's different from >>> 
"expectations about what the DRM drivers are supposed to pass in >>> the 
IGT test suite" which is about validation, ie "all KMS >>> drivers must 
behave this way". >> >> [...] >> >> >> We could have some policy: if you 
want to enable a certain device >> in the CI, you need to make sure it 
passes all tests first to force >> people to go fix the issues, but 
maybe it would be a big barrier. >> >> I'm afraid that, if a test fail 
(and it is a clear bug), people >> would just say "work for most of the 
cases, this is not a priority >> to fix" and just start ignoring the CI, 
this is why I think >> regression tests is a good way to start with. > > 
I think eventually we need to get to both goals, but currently > driver 
and test quality just isn't remotely there. > > I think a good approach 
would be if CI work focuses on the pure sw > tests first, so kunit and 
running igt against vgem/vkms. And then we > could use that to polish a 
set of must-pass igt testcases, which > also drivers in general are 
supposed to pass. Plus ideally weed out > the bad igts that aren't 
reliable enough or have bad assumptions. > > For hardware I think it 
will take a very long time until we get to a > point where CI can work 
without a test result list, we're nowhere > close to that. But for 
virtual driver this really should be > achievable, albeit with a huge 
amount of effort required to get > there I think.
Yeah, this is what our experience with Mesa (in particular) has taught us.

Having 100% of the tests pass 100% of the time on 100% of the platforms 
is a great goal that everyone should aim for. But it will also never happen.

Firstly, we're just not there yet today. Every single GPU-side DRM 
driver has userspace-triggerable faults which cause occasional errors in 
GL/Vulkan tests. Every single one. We deal with these in Mesa by 
retrying; if we didn't retry, across the breadth of hardware we test, 
I'd expect 99% of should-succeed merges to fail because of these 
intermittent bugs in the DRM drivers. We don't have the same figure for 
KMS - because we don't test it - but I'd be willing to bet no driver is 
100% if you run tests often enough.

Secondly, we will never be there. If we could pause for five years and 
sit down making all the current usecases for all the current hardware on 
the current kernel run perfectly, we'd probably get there. But we can't: 
there's new hardware, new userspace, and hundreds of new kernel trees. 
Even without the first two, what happens when the Arm SMMU maintainers 
(choosing a random target to pick on, sorry Robin) introduce subtle 
breakage which makes a lot of tests fail some of the time? Do we refuse 
to backmerge Linus into DRM until it's fixed, or do we disable all 
testing on Arm until it's fixed? When we've done that, what happens when 
we re-enable testing, and discover that a bunch of tests get broken 
because we haven't been testing?

Thirdly, hardware is capricious. 'This board doesn't make it to u-boot' 
is a clear infrastructure error, but if you test at sufficient scale, 
cold solder or failing caps surface way more often than you might think. 
And you can't really pick those out by any other means than running at 
scale, dealing with non-binary results, and looking at the trends over 
time. (Again this is something we do in Mesa - we graph test failures 
per DUT, look for outliers, and pull DUTs out of the rotation when 
they're clearly defective. But that only works if you actually run 
enough tests on them in the first place to discover trends - if you stop 
at the first failed test, it's impossible to tell the difference between 
'infuriatingly infrequent kernel/test bug?' and 'cracked main board 
maybe?'.)

What we do know is that we _can_ classify tests four ways in 
expectations. Always-passing tests should always pass. Always-failing 
tests should always fail (and update the expectations if you make them 
pass). Flaking tests work often enough that they'll always pass if you 
run them a couple/few times, but fail often enough that you can't rely 
on them. Then you just skip tests which exhibit catastrophic failure 
i.e. local DoS which affects the whole test suite.

By keeping those sets of expectations, we've been able to keep Mesa 
pretty clear of regressions, whilst having a very clear set of things 
that should be fixed to point to. It would be great if those set of 
things were zero, but it just isn't. Having that is far better than the 
two alternatives: either not testing at all (obviously bad), or having 
the test always be red so it's always ignored (might as well just not test).


Cheers,

Daniel

Hi

(Removing most of the context that got scrambled)

On Thu, Sep 07, 2023 at 01:40:02PM +0200, Daniel Stone wrote:
> Yeah, this is what our experience with Mesa (in particular) has taught us.
> 
> Having 100% of the tests pass 100% of the time on 100% of the platforms is a
> great goal that everyone should aim for. But it will also never happen.
> 
> Firstly, we're just not there yet today. Every single GPU-side DRM driver
> has userspace-triggerable faults which cause occasional errors in GL/Vulkan
> tests. Every single one. We deal with these in Mesa by retrying; if we
> didn't retry, across the breadth of hardware we test, I'd expect 99% of
> should-succeed merges to fail because of these intermittent bugs in the DRM
> drivers.

So the plan is only to ever test rendering devices? It should have been
made clearer then.

> We don't have the same figure for KMS - because we don't test it - but
> I'd be willing to bet no driver is 100% if you run tests often enough.

And I would still consider that a bug that we ought to fix, and
certainly not something we should sweep under the rug. If half the tests
are not running on a driver, then fine, they aren't. I'm not really
against having failing tests, I'm against not flagging unreliable tests
on a given hardware as failing tests.

> Secondly, we will never be there. If we could pause for five years and sit
> down making all the current usecases for all the current hardware on the
> current kernel run perfectly, we'd probably get there. But we can't: there's
> new hardware, new userspace, and hundreds of new kernel trees.

Not with that attitude :)

I'm not sure it's actually an argument, really. 10 years ago, we would
never have been at "every GPU on the market has an open-source driver"
here. 5 years ago, we would never have been at this-series-here. That
didn't stop anyone making progress, everyone involved in that thread
included.

> Even without the first two, what happens when the Arm SMMU maintainers
> (choosing a random target to pick on, sorry Robin) introduce subtle
> breakage which makes a lot of tests fail some of the time? Do we
> refuse to backmerge Linus into DRM until it's fixed, or do we disable
> all testing on Arm until it's fixed? When we've done that, what
> happens when we re-enable testing, and discover that a bunch of tests
> get broken because we haven't been testing?

I guess that's another thing that needs to be made clearer then. Do you
want to test Mesa, or the kernel?

For Mesa, I'd very much expect to rely on a stable kernel, and for the
kernel on a stable Mesa.

And if we're testing the kernel, then let's turn it the other way
around. How are we even supposed to detect those failures in the first
place if tests are flagged as unreliable?

No matter what we do here, what you describe will always happen. Like,
if we do flag those tests as unreliable, what exactly prevents another
issue to come on top undetected, and what will happen when we re-enable
testing?

On top of that, you kind of hinted at that yourself, but what set of
tests will pass is a property linked to a single commit. Having that
list within the kernel already alters that: you'll need to merge a new
branch, add a bunch of fixes and then change the test list state. You
won't have the same tree you originally tested (and defined the test
state list for).

It might or might not be an issue for Linus' release, but I can
definitely see the trouble already for stable releases where fixes will
be backported, but the test state list certainly won't be updated.

> Thirdly, hardware is capricious. 'This board doesn't make it to u-boot' is a
> clear infrastructure error, but if you test at sufficient scale, cold solder
> or failing caps surface way more often than you might think. And you can't
> really pick those out by any other means than running at scale, dealing with
> non-binary results, and looking at the trends over time. (Again this is
> something we do in Mesa - we graph test failures per DUT, look for outliers,
> and pull DUTs out of the rotation when they're clearly defective. But that
> only works if you actually run enough tests on them in the first place to
> discover trends - if you stop at the first failed test, it's impossible to
> tell the difference between 'infuriatingly infrequent kernel/test bug?' and
> 'cracked main board maybe?'.)
> 
> What we do know is that we _can_ classify tests four ways in expectations.
> Always-passing tests should always pass. Always-failing tests should always
> fail (and update the expectations if you make them pass). Flaking tests work
> often enough that they'll always pass if you run them a couple/few times,
> but fail often enough that you can't rely on them. Then you just skip tests
> which exhibit catastrophic failure i.e. local DoS which affects the whole
> test suite.
> 
> By keeping those sets of expectations, we've been able to keep Mesa pretty
> clear of regressions, whilst having a very clear set of things that should
> be fixed to point to. It would be great if those set of things were zero,
> but it just isn't. Having that is far better than the two alternatives:
> either not testing at all (obviously bad), or having the test always be red
> so it's always ignored (might as well just not test).

Isn't that what happens with flaky tests anyway? Even more so since we
have 0 context when updating that list.

I've asked a couple of times, I'll ask again. In that other series, on
the MT8173, kms_hdmi_inject@inject-4k is setup as flaky (which is a KMS
test btw).

I'm a maintainer for that part of the kernel, I'd like to look into it,
because it's seriously something that shouldn't fail, ever, the hardware
isn't involved.

How can I figure out now (or worse, let's say in a year) how to
reproduce it? What kernel version was affected? With what board? After
how many occurences?

Basically, how can I see that the bug is indeed there (or got fixed
since), and how to start fixing it?

And then repeat for any other test listed in there.

I got no other reply before because I very well know the answer: nobody
knows. And that's a serious issue to me, because that effectively means
that the flaky test list will only ever increase (since we can't even
check that it's fixed, and the CI infrastructure won't check that it got
fixed either), and we won't be able to address any of the bugs listed
there.

Maxime

On 9/11/23 11:34, Maxime Ripard wrote:
> On Thu, Sep 07, 2023 at 01:40:02PM +0200, Daniel Stone wrote:
>> Yeah, this is what our experience with Mesa (in particular) has taught us.
>>
>> Having 100% of the tests pass 100% of the time on 100% of the platforms is a
>> great goal that everyone should aim for. But it will also never happen.
>>
>> Firstly, we're just not there yet today. Every single GPU-side DRM driver
>> has userspace-triggerable faults which cause occasional errors in GL/Vulkan
>> tests. Every single one. We deal with these in Mesa by retrying; if we
>> didn't retry, across the breadth of hardware we test, I'd expect 99% of
>> should-succeed merges to fail because of these intermittent bugs in the DRM
>> drivers.
> 
> So the plan is only to ever test rendering devices? It should have been
> made clearer then.
> 
>> We don't have the same figure for KMS - because we don't test it - but
>> I'd be willing to bet no driver is 100% if you run tests often enough.
> 
> And I would still consider that a bug that we ought to fix, and
> certainly not something we should sweep under the rug. If half the tests
> are not running on a driver, then fine, they aren't. I'm not really
> against having failing tests, I'm against not flagging unreliable tests
> on a given hardware as failing tests.

A flaky test will by definition give a "pass" result at least some of the time, which would be considered a failure by the CI if the test is marked as failing.


>> Secondly, we will never be there. If we could pause for five years and sit
>> down making all the current usecases for all the current hardware on the
>> current kernel run perfectly, we'd probably get there. But we can't: there's
>> new hardware, new userspace, and hundreds of new kernel trees.
> 
> Not with that attitude :)

Attitude is not the issue, the complexity of the multiple systems involved is.


> I'm not sure it's actually an argument, really. 10 years ago, we would
> never have been at "every GPU on the market has an open-source driver"
> here. 5 years ago, we would never have been at this-series-here. That
> didn't stop anyone making progress, everyone involved in that thread
> included.

Even assuming perfection is achievable at all (which is very doubtful, given the experience from the last few years of CI in Mesa and other projects), if you demand perfection before even taking the first step, it will never get off the ground.


> How are we even supposed to detect those failures in the first
> place if tests are flagged as unreliable?

Based on experience with Mesa, only a relatively small minority of tests should need to be marked as flaky / not run at all. The majority of tests are reliable and can catch regressions even while some tests are not yet.


> No matter what we do here, what you describe will always happen. Like,
> if we do flag those tests as unreliable, what exactly prevents another
> issue to come on top undetected, and what will happen when we re-enable
> testing?

Any issues affecting a test will need to be fixed before (re-)enabling the test for CI.


> On top of that, you kind of hinted at that yourself, but what set of
> tests will pass is a property linked to a single commit. Having that
> list within the kernel already alters that: you'll need to merge a new
> branch, add a bunch of fixes and then change the test list state. You
> won't have the same tree you originally tested (and defined the test
> state list for).

Ideally, the test state lists should be changed in the same commits which affect the test results. It'll probably take a while yet to get there for the kernel.

> It might or might not be an issue for Linus' release, but I can
> definitely see the trouble already for stable releases where fixes will
> be backported, but the test state list certainly won't be updated.

If the stable branch maintainers want to take advantage of CI for the stable branches, they may need to hunt for corresponding state list commits sometimes. They'll need to take that into account for their decision.


>> By keeping those sets of expectations, we've been able to keep Mesa pretty
>> clear of regressions, whilst having a very clear set of things that should
>> be fixed to point to. It would be great if those set of things were zero,
>> but it just isn't. Having that is far better than the two alternatives:
>> either not testing at all (obviously bad), or having the test always be red
>> so it's always ignored (might as well just not test).
> 
> Isn't that what happens with flaky tests anyway?

For a small minority of tests. Daniel was referring to whole test suites.

> Even more so since we have 0 context when updating that list.

The commit log can provide whatever context is needed.


> I've asked a couple of times, I'll ask again. In that other series, on
> the MT8173, kms_hdmi_inject@inject-4k is setup as flaky (which is a KMS
> test btw).
> 
> I'm a maintainer for that part of the kernel, I'd like to look into it,
> because it's seriously something that shouldn't fail, ever, the hardware
> isn't involved.
> 
> How can I figure out now (or worse, let's say in a year) how to
> reproduce it? What kernel version was affected? With what board? After
> how many occurences?
> 
> Basically, how can I see that the bug is indeed there (or got fixed
> since), and how to start fixing it?

Many of those things should be documented in the commit log of the state list change.

How the CI works in general should be documented in some appropriate place in tree.


-- 
Earthling Michel Dänzer            |                  https://redhat.com
Libre software enthusiast          |         Mesa and Xwayland developer

On Mon, Sep 11, 2023 at 02:13:43PM +0200, Michel Dänzer wrote:
> On 9/11/23 11:34, Maxime Ripard wrote:
> > On Thu, Sep 07, 2023 at 01:40:02PM +0200, Daniel Stone wrote:
> >> Yeah, this is what our experience with Mesa (in particular) has taught us.
> >>
> >> Having 100% of the tests pass 100% of the time on 100% of the platforms is a
> >> great goal that everyone should aim for. But it will also never happen.
> >>
> >> Firstly, we're just not there yet today. Every single GPU-side DRM driver
> >> has userspace-triggerable faults which cause occasional errors in GL/Vulkan
> >> tests. Every single one. We deal with these in Mesa by retrying; if we
> >> didn't retry, across the breadth of hardware we test, I'd expect 99% of
> >> should-succeed merges to fail because of these intermittent bugs in the DRM
> >> drivers.
> > 
> > So the plan is only to ever test rendering devices? It should have been
> > made clearer then.
> > 
> >> We don't have the same figure for KMS - because we don't test it - but
> >> I'd be willing to bet no driver is 100% if you run tests often enough.
> > 
> > And I would still consider that a bug that we ought to fix, and
> > certainly not something we should sweep under the rug. If half the tests
> > are not running on a driver, then fine, they aren't. I'm not really
> > against having failing tests, I'm against not flagging unreliable tests
> > on a given hardware as failing tests.
> 
> A flaky test will by definition give a "pass" result at least some of the time, which would be considered a failure by the CI if the test is marked as failing.
> 
> 
> >> Secondly, we will never be there. If we could pause for five years and sit
> >> down making all the current usecases for all the current hardware on the
> >> current kernel run perfectly, we'd probably get there. But we can't: there's
> >> new hardware, new userspace, and hundreds of new kernel trees.
> > 
> > Not with that attitude :)
> 
> Attitude is not the issue, the complexity of the multiple systems
> involved is.

FTR, that was a meme/joke.

> > I'm not sure it's actually an argument, really. 10 years ago, we would
> > never have been at "every GPU on the market has an open-source driver"
> > here. 5 years ago, we would never have been at this-series-here. That
> > didn't stop anyone making progress, everyone involved in that thread
> > included.
> 
> Even assuming perfection is achievable at all (which is very doubtful,
> given the experience from the last few years of CI in Mesa and other
> projects), if you demand perfection before even taking the first step,
> it will never get off the ground.

Perfection and scale from the get-go isn't reasonable, yes. Building a
small, "perfect" (your words, not mine) system that you can later expand
is doable. And that's very much a design choice.

> > How are we even supposed to detect those failures in the first
> > place if tests are flagged as unreliable?
> 
> Based on experience with Mesa, only a relatively small minority of
> tests should need to be marked as flaky / not run at all. The majority
> of tests are reliable and can catch regressions even while some tests
> are not yet.

I understand and acknowledge that it worked with Mesa. That's great for
Mesa. That still doesn't mean that it's the panacea and is for every
project.

> > No matter what we do here, what you describe will always happen. Like,
> > if we do flag those tests as unreliable, what exactly prevents another
> > issue to come on top undetected, and what will happen when we re-enable
> > testing?
> 
> Any issues affecting a test will need to be fixed before (re-)enabling
> the test for CI.

If that underlying issue is never fixed, at which point do we consider
that it's a failure and should never be re-enabled? Who has that role?

> > On top of that, you kind of hinted at that yourself, but what set of
> > tests will pass is a property linked to a single commit. Having that
> > list within the kernel already alters that: you'll need to merge a new
> > branch, add a bunch of fixes and then change the test list state. You
> > won't have the same tree you originally tested (and defined the test
> > state list for).
> 
> Ideally, the test state lists should be changed in the same commits
> which affect the test results. It'll probably take a while yet to get
> there for the kernel.
> 
> > It might or might not be an issue for Linus' release, but I can
> > definitely see the trouble already for stable releases where fixes will
> > be backported, but the test state list certainly won't be updated.
> 
> If the stable branch maintainers want to take advantage of CI for the
> stable branches, they may need to hunt for corresponding state list
> commits sometimes. They'll need to take that into account for their
> decision.

So we just expect the stable maintainers to track each and every patches
involved in a test run, make sure that they are in a stable tree, and
then update the test list? Without having consulted them at all?

> >> By keeping those sets of expectations, we've been able to keep Mesa pretty
> >> clear of regressions, whilst having a very clear set of things that should
> >> be fixed to point to. It would be great if those set of things were zero,
> >> but it just isn't. Having that is far better than the two alternatives:
> >> either not testing at all (obviously bad), or having the test always be red
> >> so it's always ignored (might as well just not test).
> > 
> > Isn't that what happens with flaky tests anyway?
> 
> For a small minority of tests. Daniel was referring to whole test suites.
> 
> > Even more so since we have 0 context when updating that list.
> 
> The commit log can provide whatever context is needed.

Sure, I've yet to see that though.

There's in 6.6-rc1 around 240 reported flaky tests. None of them have
any context. That new series hads a few dozens too, without any context
either. And there's no mention about that being a plan, or a patch
adding a new policy for all tests going forward.

So I'm still fairly doubtful it will ever happen.

> > I've asked a couple of times, I'll ask again. In that other series, on
> > the MT8173, kms_hdmi_inject@inject-4k is setup as flaky (which is a KMS
> > test btw).
> > 
> > I'm a maintainer for that part of the kernel, I'd like to look into it,
> > because it's seriously something that shouldn't fail, ever, the hardware
> > isn't involved.
> > 
> > How can I figure out now (or worse, let's say in a year) how to
> > reproduce it? What kernel version was affected? With what board? After
> > how many occurences?
> > 
> > Basically, how can I see that the bug is indeed there (or got fixed
> > since), and how to start fixing it?
> 
> Many of those things should be documented in the commit log of the
> state list change.
> 
> How the CI works in general should be documented in some appropriate
> place in tree.

I think I'll stop the discussion there. It was merged anyway so I'm not
quite sure why I was asked to give my feedback on this. Any concern I
raised were met with a giant "it worked on Mesa" handwave or "someone
will probably work on it at some point".

And fine, guess I'm wrong.

Thanks
Maxime

On 9/11/23 14:51, Maxime Ripard wrote:
> On Mon, Sep 11, 2023 at 02:13:43PM +0200, Michel Dänzer wrote:
>> On 9/11/23 11:34, Maxime Ripard wrote:
>>> On Thu, Sep 07, 2023 at 01:40:02PM +0200, Daniel Stone wrote:
>>>>
>>>> Secondly, we will never be there. If we could pause for five years and sit
>>>> down making all the current usecases for all the current hardware on the
>>>> current kernel run perfectly, we'd probably get there. But we can't: there's
>>>> new hardware, new userspace, and hundreds of new kernel trees.
>>>
>>> [...]
>>> 
>>> I'm not sure it's actually an argument, really. 10 years ago, we would
>>> never have been at "every GPU on the market has an open-source driver"
>>> here. 5 years ago, we would never have been at this-series-here. That
>>> didn't stop anyone making progress, everyone involved in that thread
>>> included.
>>
>> Even assuming perfection is achievable at all (which is very doubtful,
>> given the experience from the last few years of CI in Mesa and other
>> projects), if you demand perfection before even taking the first step,
>> it will never get off the ground.
> 
> Perfection and scale from the get-go isn't reasonable, yes. Building a
> small, "perfect" (your words, not mine) system that you can later expand
> is doable.

I mean "perfect" as in every single available test runs, is reliable and gates CI. Which seems to be what you're asking for. The only possible expansion of such a system would be adding new 100% reliable tests.

What is being proposed here is an "imperfect" system which takes into account the reality that some tests are not 100% reliable, and can be improved gradually while already preventing some regressions from getting merged.


>>> How are we even supposed to detect those failures in the first
>>> place if tests are flagged as unreliable?
>>
>> Based on experience with Mesa, only a relatively small minority of
>> tests should need to be marked as flaky / not run at all. The majority
>> of tests are reliable and can catch regressions even while some tests
>> are not yet.
> 
> I understand and acknowledge that it worked with Mesa. That's great for
> Mesa. That still doesn't mean that it's the panacea and is for every
> project.

Not sure what you're referring to by panacea, or how it relates to "some tests can be useful even while others aren't yet".


>>> No matter what we do here, what you describe will always happen. Like,
>>> if we do flag those tests as unreliable, what exactly prevents another
>>> issue to come on top undetected, and what will happen when we re-enable
>>> testing?
>>
>> Any issues affecting a test will need to be fixed before (re-)enabling
>> the test for CI.
> 
> If that underlying issue is never fixed, at which point do we consider
> that it's a failure and should never be re-enabled? Who has that role?

Not sure what you're asking. Anybody can (re-)enable a test in CI, they just need to make sure first that it is reliable. Until somebody does that work, it'll stay disabled in CI.


>>> It might or might not be an issue for Linus' release, but I can
>>> definitely see the trouble already for stable releases where fixes will
>>> be backported, but the test state list certainly won't be updated.
>>
>> If the stable branch maintainers want to take advantage of CI for the
>> stable branches, they may need to hunt for corresponding state list
>> commits sometimes. They'll need to take that into account for their
>> decision.
> 
> So we just expect the stable maintainers to track each and every patches
> involved in a test run, make sure that they are in a stable tree, and
> then update the test list? Without having consulted them at all?

I don't expect them to do anything. See the If at the start of what I wrote.


>>>> By keeping those sets of expectations, we've been able to keep Mesa pretty
>>>> clear of regressions, whilst having a very clear set of things that should
>>>> be fixed to point to. It would be great if those set of things were zero,
>>>> but it just isn't. Having that is far better than the two alternatives:
>>>> either not testing at all (obviously bad), or having the test always be red
>>>> so it's always ignored (might as well just not test).
>>>
>>> Isn't that what happens with flaky tests anyway?
>>
>> For a small minority of tests. Daniel was referring to whole test suites.
>>
>>> Even more so since we have 0 context when updating that list.
>>
>> The commit log can provide whatever context is needed.
> 
> Sure, I've yet to see that though.
> 
> There's in 6.6-rc1 around 240 reported flaky tests. None of them have
> any context. That new series hads a few dozens too, without any context
> either. And there's no mention about that being a plan, or a patch
> adding a new policy for all tests going forward.

That does sound bad, would need to be raised in review.


> Any concern I raised were met with a giant "it worked on Mesa" handwave

Lessons learned from years of experience with big real-world CI systems like this are hardly "handwaving".


-- 
Earthling Michel Dänzer            |                  https://redhat.com
Libre software enthusiast          |         Mesa and Xwayland developer

Replying one more time, because I certainly don't want to create any
hard feeling here.

On Mon, Sep 11, 2023 at 03:30:55PM +0200, Michel Dänzer wrote:
> >>>> By keeping those sets of expectations, we've been able to keep Mesa pretty
> >>>> clear of regressions, whilst having a very clear set of things that should
> >>>> be fixed to point to. It would be great if those set of things were zero,
> >>>> but it just isn't. Having that is far better than the two alternatives:
> >>>> either not testing at all (obviously bad), or having the test always be red
> >>>> so it's always ignored (might as well just not test).
> >>>
> >>> Isn't that what happens with flaky tests anyway?
> >>
> >> For a small minority of tests. Daniel was referring to whole test suites.
> >>
> >>> Even more so since we have 0 context when updating that list.
> >>
> >> The commit log can provide whatever context is needed.
> > 
> > Sure, I've yet to see that though.
> > 
> > There's in 6.6-rc1 around 240 reported flaky tests. None of them have
> > any context. That new series hads a few dozens too, without any context
> > either. And there's no mention about that being a plan, or a patch
> > adding a new policy for all tests going forward.
> 
> That does sound bad, would need to be raised in review.
>
> > Any concern I raised were met with a giant "it worked on Mesa" handwave
> 
> Lessons learned from years of experience with big real-world CI
> systems like this are hardly "handwaving".

Your (and others) experience certainly isn't. It is valuable, welcome,
and very much appreciated.

However, my questions and concerns being ignored time and time again
about things like what is the process is going to be like, what is going
to be tested, who is going to be maintaining that test list, how that
interacts with stable, how we can possibly audit the flaky tests list,
etc. have felt like they were being handwaived away.

I'm not saying that because I disagree, I still do on some, but that's
fine to some extent. However, most of these issues are not so much an
infrastructure issue, but a community issue. And I don't even expect a
perfect solution right now, unlike what you seem to think. But I do
expect some kind of plan instead of just ignoring that problem.

Like, I had to ask the MT8173 question 3 times in order to get an
answer, and I'm still not sure what is going to be done to address that
particular issue.

So, I'm sorry, but I certainly feel like it here.

Maxime

Hi Maxime,
Hopefully less mangled formatting this time: turns out Thunderbird +
plain text is utterly unreadable, so that's one less MUA that is
actually usable to send email to kernel lists without getting shouted
at.

On Mon, 11 Sept 2023 at 15:46, Maxime Ripard <mripard@kernel.org> wrote:
> On Mon, Sep 11, 2023 at 03:30:55PM +0200, Michel Dänzer wrote:
> > > There's in 6.6-rc1 around 240 reported flaky tests. None of them have
> > > any context. That new series hads a few dozens too, without any context
> > > either. And there's no mention about that being a plan, or a patch
> > > adding a new policy for all tests going forward.
> >
> > That does sound bad, would need to be raised in review.
> >
> > > Any concern I raised were met with a giant "it worked on Mesa" handwave
> >
> > Lessons learned from years of experience with big real-world CI
> > systems like this are hardly "handwaving".
>
> Your (and others) experience certainly isn't. It is valuable, welcome,
> and very much appreciated.
>
> However, my questions and concerns being ignored time and time again
> about things like what is the process is going to be like, what is going
> to be tested, who is going to be maintaining that test list, how that
> interacts with stable, how we can possibly audit the flaky tests list,
> etc. have felt like they were being handwaived away.

Sorry it ended up coming across like that. It wasn't the intent.

> I'm not saying that because I disagree, I still do on some, but that's
> fine to some extent. However, most of these issues are not so much an
> infrastructure issue, but a community issue. And I don't even expect a
> perfect solution right now, unlike what you seem to think. But I do
> expect some kind of plan instead of just ignoring that problem.
>
> Like, I had to ask the MT8173 question 3 times in order to get an
> answer, and I'm still not sure what is going to be done to address that
> particular issue.
>
> So, I'm sorry, but I certainly feel like it here.

I don't quite see the same picture from your side though. For example,
my reading of what you've said is that flaky tests are utterly
unacceptable, as are partial runs, and we shouldn't pretend otherwise.
With your concrete example (which is really helpful, so thanks), what
happens to the MT8173 hdmi-inject test? Do we skip all MT8173 testing
until it's perfect, or does MT8173 testing always fail because that
test does?

Both have their downsides. Not doing any testing has the obvious
downside, and means that the driver can get worse until it gets
perfect. Always marking the test as failed means that the test results
are useless: if failure is expected, then red is good. I mean, say
you're contributing a patch to fix some documentation or add a helper
to common code which only v3d uses. The test results come back, and
your branch is failing tests on MT8173, specifically the
hdmi-inject@4k test. What then? Either as a senior contributor you
'know' that's the case, or as a casual contributor you get told 'oh
yeah, don't worry about the test results, they always fail'. Both lead
to the same outcome, which is that no-one pays any attention to the
results, and they get worse.

What we do agree on is that yes, those tests should absolutely be
fixed, and not just swept under the rug. Part of this is having
maintainers actually meaningfully own their test results. For example,
I'm looking at the expectation lists for the Intel gen in my laptop,
and I'm seeing a lot of breakage in blending tests, as well as
dual-display fails which include the resolution of my external
display. I'd expect the Intel driver maintainers to look at them, get
them fixed, and gradually prune those xfails/flakes down towards zero.

If the maintainers don't own it though, then it's not going to get
fixed. And we are exactly where we are today: broken plane blending
and 1440p on KBL, broken EDID injection on MT8173, and broken atomic
commits on stoney. Without stronger action from the maintainers (e.g.
throwing i915 out of the tree until it has 100% pass 100% of the
time), adding testing isn't making the situation better or worse in
and of itself. What it _is_ doing though, is giving really clear
documentation of the status of each driver, and backing that up by
verifying it.

Only maintainers can actually fix the drivers (or the tests tbf). But
doing the testing does let us be really clear to everyone what the
actual state is, and that way people can make informed decisions too.
And the only way we're going to drive the test rate down is by the
subsystem maintainers enforcing it.

Does that make sense on where I'm (and I think a lot of others are) coming from?

To answer the other question about 'where are the logs?': some of them
have the failure data in them, others don't. They all should going
forward at least though.

Cheers,
Daniel

Hi,

On Tue, Sep 12, 2023 at 02:16:41PM +0100, Daniel Stone wrote:
> Hopefully less mangled formatting this time: turns out Thunderbird +
> plain text is utterly unreadable, so that's one less MUA that is
> actually usable to send email to kernel lists without getting shouted
> at.

Sorry if it felt that way, it definitely wasn't my intention to shout at
you. Email is indeed kind of a pain to deal with, and I wanted to keep
the discussion going.

> On Mon, 11 Sept 2023 at 15:46, Maxime Ripard <mripard@kernel.org> wrote:
> > On Mon, Sep 11, 2023 at 03:30:55PM +0200, Michel Dänzer wrote:
> > > > There's in 6.6-rc1 around 240 reported flaky tests. None of them have
> > > > any context. That new series hads a few dozens too, without any context
> > > > either. And there's no mention about that being a plan, or a patch
> > > > adding a new policy for all tests going forward.
> > >
> > > That does sound bad, would need to be raised in review.
> > >
> > > > Any concern I raised were met with a giant "it worked on Mesa" handwave
> > >
> > > Lessons learned from years of experience with big real-world CI
> > > systems like this are hardly "handwaving".
> >
> > Your (and others) experience certainly isn't. It is valuable, welcome,
> > and very much appreciated.
> >
> > However, my questions and concerns being ignored time and time again
> > about things like what is the process is going to be like, what is going
> > to be tested, who is going to be maintaining that test list, how that
> > interacts with stable, how we can possibly audit the flaky tests list,
> > etc. have felt like they were being handwaived away.
> 
> Sorry it ended up coming across like that. It wasn't the intent.
> 
> > I'm not saying that because I disagree, I still do on some, but that's
> > fine to some extent. However, most of these issues are not so much an
> > infrastructure issue, but a community issue. And I don't even expect a
> > perfect solution right now, unlike what you seem to think. But I do
> > expect some kind of plan instead of just ignoring that problem.
> >
> > Like, I had to ask the MT8173 question 3 times in order to get an
> > answer, and I'm still not sure what is going to be done to address that
> > particular issue.
> >
> > So, I'm sorry, but I certainly feel like it here.
> 
> I don't quite see the same picture from your side though. For example,
> my reading of what you've said is that flaky tests are utterly
> unacceptable, as are partial runs, and we shouldn't pretend otherwise.
> With your concrete example (which is really helpful, so thanks), what
> happens to the MT8173 hdmi-inject test? Do we skip all MT8173 testing
> until it's perfect, or does MT8173 testing always fail because that
> test does?

It's not clear to me why that test is even running in the first place?
There's been some confusion on my side here about what we're going to
test with this. You've mentioned Mesa and GPUs before, but that's a KMS
test so there must be more to it.

Either way, it's a relevant test so I guess why not. It turns out that
the test is indeed flaky, I guess we could add it to the flaky tests
list.

BUT

I want to have every opportunity to fix whatever that failure is.

So:

  - Is the test broken? If so, we should report it to IGT dev and remove
    it from the test suite.
  - If not, is that test failure have been reported to the driver author?
  - If no answer/fix, we can add it to the flaky tests list, but do we
    have some way to reproduce the test failure?

The last part is especially critical. Looking at the list itself, I have
no idea what board, kernel version, configuration, or what the failure
rate was. Assuming I spend some time looking at the infra to find the
board and configuration, how many times do I have to run the tests to
expect to reproduce the failure (and thus consider it fixed if it
doesn't occur anymore).

Like, with that board and test, if my first 100 runs of the test work
fine, is it reasonable for me to consider it fixed, or is it only
supposed to happen once every 1000 runs?

So, ideally, having some (mandatory) metadata in the test lists with a
link to the bug report, the board (DT name?) it happened with, the
version and configuration it was first seen with, and an approximation
of the failure rate for every flaky test list.

I understand that it's probably difficult to get that after the fact on
the tests that were already merged, but I'd really like to get that
enforced for every new test going forward.

That should hopefully get us in a much better position to fix some of
those tests issues. And failing that, I can't see how that's
sustainable.

And Mesa does show what I'm talking about:

$ find -name *-flakes.txt | xargs git diff --stat  e58a10af640ba58b6001f5c5ad750b782547da76
 src/amd/ci/deqp-radv-stoney-aco-flakes.txt                        |  18 +++++
 src/broadcom/ci/deqp-v3d-rpi4-flakes.txt                          |   2 +
 src/broadcom/ci/deqp-v3dv-rpi4-flakes.txt                         |  15 ++++
 src/broadcom/ci/deqp-vc4-rpi3-flakes.txt                          |  30 ++++++++
 src/broadcom/ci/piglit-v3d-rpi4-flakes.txt                        |  14 ++++
 src/broadcom/ci/piglit-vc4-rpi3-flakes.txt                        |   5 ++
 src/freedreno/ci/deqp-freedreno-a307-flakes.txt                   |   6 ++
 src/freedreno/ci/deqp-freedreno-a530-flakes.txt                   |  31 +++++++++
 src/freedreno/ci/deqp-freedreno-a630-bypass-flakes.txt            |  11 +++
 src/freedreno/ci/deqp-freedreno-a630-flakes.txt                   | 109 ++++++++++++++++++++++++++++++
 src/gallium/drivers/freedreno/ci/piglit-freedreno-a530-flakes.txt |  21 ++++++
 src/gallium/drivers/freedreno/ci/piglit-freedreno-a630-flakes.txt |  31 +++++++++
 src/gallium/drivers/llvmpipe/ci/deqp-llvmpipe-flakes.txt          |   4 ++
 src/gallium/drivers/panfrost/ci/deqp-panfrost-g52-flakes.txt      |   6 ++
 src/gallium/drivers/panfrost/ci/deqp-panfrost-t720-flakes.txt     |   4 ++
 src/gallium/drivers/panfrost/ci/deqp-panfrost-t760-flakes.txt     |   1 +
 src/gallium/drivers/panfrost/ci/deqp-panfrost-t860-flakes.txt     |   1 +
 src/gallium/drivers/radeonsi/ci/piglit-radeonsi-stoney-flakes.txt |   1 +
 src/gallium/drivers/softpipe/ci/deqp-softpipe-flakes.txt          |   2 +
 src/gallium/drivers/virgl/ci/deqp-virgl-gl-flakes.txt             |   3 +
 src/gallium/drivers/zink/ci/deqp-zink-lvp-flakes.txt              |   0
 src/gallium/drivers/zink/ci/piglit-zink-lvp-flakes.txt            |   0
 src/gallium/frontends/lavapipe/ci/deqp-lvp-flakes.txt             |   1 +
 23 files changed, 316 insertions(+)

In the history of Mesa, there's never been a single test removed from a
flaky test list.

> Both have their downsides. Not doing any testing has the obvious
> downside, and means that the driver can get worse until it gets
> perfect. Always marking the test as failed means that the test results
> are useless: if failure is expected, then red is good. I mean, say
> you're contributing a patch to fix some documentation or add a helper
> to common code which only v3d uses. The test results come back, and
> your branch is failing tests on MT8173, specifically the
> hdmi-inject@4k test. What then? Either as a senior contributor you
> 'know' that's the case, or as a casual contributor you get told 'oh
> yeah, don't worry about the test results, they always fail'. Both lead
> to the same outcome, which is that no-one pays any attention to the
> results, and they get worse.

I believe the above would achieve a better result for us kernel folks,
while still providing enough flexibility for you.

> What we do agree on is that yes, those tests should absolutely be
> fixed, and not just swept under the rug. Part of this is having
> maintainers actually meaningfully own their test results. For example,
> I'm looking at the expectation lists for the Intel gen in my laptop,
> and I'm seeing a lot of breakage in blending tests, as well as
> dual-display fails which include the resolution of my external
> display.

I just bought a 7900XT and I'm pretty much in the same situation, so I
definitely feel what you're saying :)

However...

> I'd expect the Intel driver maintainers to look at them, get them
> fixed, and gradually prune those xfails/flakes down towards zero.
> 
> If the maintainers don't own it though, then it's not going to get
> fixed. And we are exactly where we are today: broken plane blending
> and 1440p on KBL, broken EDID injection on MT8173, and broken atomic
> commits on stoney. Without stronger action from the maintainers (e.g.
> throwing i915 out of the tree until it has 100% pass 100% of the
> time), adding testing isn't making the situation better or worse in
> and of itself. What it _is_ doing though, is giving really clear
> documentation of the status of each driver, and backing that up by
> verifying it.

... I haven't reported any of the failures yet so I don't really expect
them to be fixed by themselves. That's on me.

> Only maintainers can actually fix the drivers (or the tests tbf). But
> doing the testing does let us be really clear to everyone what the
> actual state is, and that way people can make informed decisions too.
> And the only way we're going to drive the test rate down is by the
> subsystem maintainers enforcing it.

Just FYI, I'm not on the other side of the fence there, I'd really like
to have some kind of validation. I talked about it at XDC some years
ago, and discussed it several people at length over the years. So I'm
definitely not in the CI-is-bad camp.

> Does that make sense on where I'm (and I think a lot of others are)
> coming from?

That makes sense from your perspective, but it's not clear to me how you
can expect maintainers to own the tests if they were never involved in
the process.

They are not in Cc of the flaky tests patches, they are not reported
that the bug is failing, how can they own that process if we never
reached out and involved them?

We're all overworked, you can't expect them to just look at the flaky
test list every now and then and figure it out.

Maxime

Hey,

On Thu, 14 Sept 2023 at 10:54, Maxime Ripard <mripard@kernel.org> wrote:
> On Tue, Sep 12, 2023 at 02:16:41PM +0100, Daniel Stone wrote:
> > Hopefully less mangled formatting this time: turns out Thunderbird +
> > plain text is utterly unreadable, so that's one less MUA that is
> > actually usable to send email to kernel lists without getting shouted
> > at.
>
> Sorry if it felt that way, it definitely wasn't my intention to shout at
> you. Email is indeed kind of a pain to deal with, and I wanted to keep
> the discussion going.

My bad - I didn't mean you _at all_. I was thinking of other, much
less pleasant, kernel maintainers, and the ongoing struggle of
attempting to actually send well-formatted email to kernel lists in
2023.

> > I don't quite see the same picture from your side though. For example,
> > my reading of what you've said is that flaky tests are utterly
> > unacceptable, as are partial runs, and we shouldn't pretend otherwise.
> > With your concrete example (which is really helpful, so thanks), what
> > happens to the MT8173 hdmi-inject test? Do we skip all MT8173 testing
> > until it's perfect, or does MT8173 testing always fail because that
> > test does?
>
> It's not clear to me why that test is even running in the first place?
> There's been some confusion on my side here about what we're going to
> test with this. You've mentioned Mesa and GPUs before, but that's a KMS
> test so there must be more to it.
>
> Either way, it's a relevant test so I guess why not. It turns out that
> the test is indeed flaky, I guess we could add it to the flaky tests
> list.
>
> BUT
>
> I want to have every opportunity to fix whatever that failure is.

Agreed so far!

> So:
>
>   - Is the test broken? If so, we should report it to IGT dev and remove
>     it from the test suite.
>   - If not, is that test failure have been reported to the driver author?
>   - If no answer/fix, we can add it to the flaky tests list, but do we
>     have some way to reproduce the test failure?
>
> The last part is especially critical. Looking at the list itself, I have
> no idea what board, kernel version, configuration, or what the failure
> rate was. Assuming I spend some time looking at the infra to find the
> board and configuration, how many times do I have to run the tests to
> expect to reproduce the failure (and thus consider it fixed if it
> doesn't occur anymore).
>
> Like, with that board and test, if my first 100 runs of the test work
> fine, is it reasonable for me to consider it fixed, or is it only
> supposed to happen once every 1000 runs?
>
> So, ideally, having some (mandatory) metadata in the test lists with a
> link to the bug report, the board (DT name?) it happened with, the
> version and configuration it was first seen with, and an approximation
> of the failure rate for every flaky test list.
>
> I understand that it's probably difficult to get that after the fact on
> the tests that were already merged, but I'd really like to get that
> enforced for every new test going forward.
>
> That should hopefully get us in a much better position to fix some of
> those tests issues. And failing that, I can't see how that's
> sustainable.

OK yeah, and we're still agreed here. That is definitely the standard
we should be aiming for.  It is there for some - see
drivers/gpu/drm/ci/xfails/rockchip-rk3288-skips.txt, but should be
there for the rest, it's true. (The specific board/DT it was observed
on can be easily retconned because we only run on one specific board
type per driver, again to make things more predictable; we could go
back and retrospectively add those in a header comment?)

For flakes, it can be hard to pin them down, because, well, they're
flaky. Usually when we add things in Mesa (sorry to keep coming back
to Mesa - it's not to say that it's the objective best thing that
everything should follow, only that it's the thing we have the most
experience with that we know works well), we do a manual bisect and
try to pin the blame on a specific merge request which looks like the
most likely culprit. If nothing obvious jumps out, we just note when
it was first observed and provide some sample job logs. But yeah, it
should be more verbose.

FWIW, the reason it wasn't done here - not to say that it shouldn't
have been done better, but here we are - is that we just hammered a
load of test runs, vacuumed up the results with a script, and that's
what generated those files. Given the number of tests and devices, it
was hard to narrow each down individually, but yeah, it is something
which really wants further analysis and drilling into. It's a good
to-do, and I agree it should be the standard going forward.

> And Mesa does show what I'm talking about:
>
> $ find -name *-flakes.txt | xargs git diff --stat  e58a10af640ba58b6001f5c5ad750b782547da76
> [...]
>
> In the history of Mesa, there's never been a single test removed from a
> flaky test list.

As Rob says, that's definitely wrong. But there is a good point in
there: how do you know a test isn't flaky anymore? 100 runs is a
reasonable benchmark, but 1000 is ideal. At a 1% failure rate, with 20
devices, that's just too many spurious false-fails to have a usable
workflow.

We do have some tools to make stress testing easier, but those need to
be better documented. We'll fix that. The tools we have which also
pull out the metadata etc also need documenting - right now they
aren't because they're under _extremely_ heavy development, but they
can be further enhanced to e.g. pull out the igt results automatically
and point very clearly to the cause. Also on the to-do.

> > Only maintainers can actually fix the drivers (or the tests tbf). But
> > doing the testing does let us be really clear to everyone what the
> > actual state is, and that way people can make informed decisions too.
> > And the only way we're going to drive the test rate down is by the
> > subsystem maintainers enforcing it.
>
> Just FYI, I'm not on the other side of the fence there, I'd really like
> to have some kind of validation. I talked about it at XDC some years
> ago, and discussed it several people at length over the years. So I'm
> definitely not in the CI-is-bad camp.
>
> > Does that make sense on where I'm (and I think a lot of others are)
> > coming from?
>
> That makes sense from your perspective, but it's not clear to me how you
> can expect maintainers to own the tests if they were never involved in
> the process.
>
> They are not in Cc of the flaky tests patches, they are not reported
> that the bug is failing, how can they own that process if we never
> reached out and involved them?
>
> We're all overworked, you can't expect them to just look at the flaky
> test list every now and then and figure it out.

Absolutely. We got acks (or at least not-nacks) from the driver
developers, but yeah, they should absolutely be part of the loop for
those updates. I don't think we can necessarily block on them though.
Say we add vc4 KMS tests, then after a backmerge we start to see a
bunch of flakes on it, but you're sitting on a beach for a couple of
weeks. If we wait for you to get back, see it, and merge it, then
that's two weeks of people submitting Rockchip driver changes and
getting told that their changes failed CI. That's exactly what we want
to avoid, because it erodes confidence and usefulness of CI when
people expect failures and ignore them by default.

So I would say that it's reasonable for expectations to be updated
according to what actually happens in practice, but also to make sure
that the maintainers are explicitly informed and kept in the loop, and
not just surprised when they look at the lists and see a bunch of
stuff happened without their knowledge.

Again there's much more to be done on the tooling here. Part of it is
CLI tools and automation, part of it is dashboards and
easily-digestible reporting, and then there's integration with things
like KernelCI. KCI(DB) is actually quite high up on the list, but
we're mostly waiting until a lot of the KCI rework happens so we can
actually properly integrate with the new system.

Right now a lot of the tooling we have is pretty involved - for
example, we do have ci-collate as a Python library which can inspect a
number of pipelines, pull out detailed status and logs, etc, but it
mostly needs to be used as a library with bespoke scripts, rather than
a ready-made tool. Work on that is ongoing to make it way more clear
and accessible though.

So I think it sounds like we're on the same page and going exactly in
the same direction, just that this is a starting point rather than the
desired end state. And the main point is that having a set of
xfails/flakes parachuted in with little to no context is trying to get
an MVP bootstrapped, rather than how we expect things to go in future.
Does that sound about right?

Cheers,
Daniel

On Fri, Sep 15, 2023 at 04:08:42PM +0100, Daniel Stone wrote:
> > > I don't quite see the same picture from your side though. For example,
> > > my reading of what you've said is that flaky tests are utterly
> > > unacceptable, as are partial runs, and we shouldn't pretend otherwise.
> > > With your concrete example (which is really helpful, so thanks), what
> > > happens to the MT8173 hdmi-inject test? Do we skip all MT8173 testing
> > > until it's perfect, or does MT8173 testing always fail because that
> > > test does?
> >
> > It's not clear to me why that test is even running in the first place?
> > There's been some confusion on my side here about what we're going to
> > test with this. You've mentioned Mesa and GPUs before, but that's a KMS
> > test so there must be more to it.
> >
> > Either way, it's a relevant test so I guess why not. It turns out that
> > the test is indeed flaky, I guess we could add it to the flaky tests
> > list.
> >
> > BUT
> >
> > I want to have every opportunity to fix whatever that failure is.
> 
> Agreed so far!
> 
> > So:
> >
> >   - Is the test broken? If so, we should report it to IGT dev and remove
> >     it from the test suite.
> >   - If not, is that test failure have been reported to the driver author?
> >   - If no answer/fix, we can add it to the flaky tests list, but do we
> >     have some way to reproduce the test failure?
> >
> > The last part is especially critical. Looking at the list itself, I have
> > no idea what board, kernel version, configuration, or what the failure
> > rate was. Assuming I spend some time looking at the infra to find the
> > board and configuration, how many times do I have to run the tests to
> > expect to reproduce the failure (and thus consider it fixed if it
> > doesn't occur anymore).
> >
> > Like, with that board and test, if my first 100 runs of the test work
> > fine, is it reasonable for me to consider it fixed, or is it only
> > supposed to happen once every 1000 runs?
> >
> > So, ideally, having some (mandatory) metadata in the test lists with a
> > link to the bug report, the board (DT name?) it happened with, the
> > version and configuration it was first seen with, and an approximation
> > of the failure rate for every flaky test list.
> >
> > I understand that it's probably difficult to get that after the fact on
> > the tests that were already merged, but I'd really like to get that
> > enforced for every new test going forward.
> >
> > That should hopefully get us in a much better position to fix some of
> > those tests issues. And failing that, I can't see how that's
> > sustainable.
> 
> OK yeah, and we're still agreed here. That is definitely the standard
> we should be aiming for.  It is there for some - see
> drivers/gpu/drm/ci/xfails/rockchip-rk3288-skips.txt, but should be
> there for the rest, it's true. (The specific board/DT it was observed
> on can be easily retconned because we only run on one specific board
> type per driver, again to make things more predictable; we could go
> back and retrospectively add those in a header comment?)

Yeah, metadata was meant to sound professional, but putting all that in
a somewhat parseable comment is good enough for me :)

> For flakes, it can be hard to pin them down, because, well, they're
> flaky. Usually when we add things in Mesa (sorry to keep coming back
> to Mesa - it's not to say that it's the objective best thing that
> everything should follow, only that it's the thing we have the most
> experience with that we know works well), we do a manual bisect and
> try to pin the blame on a specific merge request which looks like the
> most likely culprit. If nothing obvious jumps out, we just note when
> it was first observed and provide some sample job logs. But yeah, it
> should be more verbose.

Sure, I guess we should document that for flakes, it's not meant as a
precise indication. Still it provides some kind of "time anchor". If it
was first observed 10 years ago and I can't reproduce it, chances are
it's been fixed one way or the other since. If it was observed last
month, it's more likely something's up with my setup.

It's not definitive, but it provides some hints.

> FWIW, the reason it wasn't done here - not to say that it shouldn't
> have been done better, but here we are - is that we just hammered a
> load of test runs, vacuumed up the results with a script, and that's
> what generated those files. Given the number of tests and devices, it
> was hard to narrow each down individually, but yeah, it is something
> which really wants further analysis and drilling into. It's a good
> to-do, and I agree it should be the standard going forward.

Awesome :)

> > And Mesa does show what I'm talking about:
> >
> > $ find -name *-flakes.txt | xargs git diff --stat  e58a10af640ba58b6001f5c5ad750b782547da76
> > [...]
> >
> > In the history of Mesa, there's never been a single test removed from a
> > flaky test list.
> 
> As Rob says, that's definitely wrong.

Yeah, that was a dumb metric, sorry.

> But there is a good point in there: how do you know a test isn't flaky
> anymore? 100 runs is a reasonable benchmark, but 1000 is ideal. At a
> 1% failure rate, with 20 devices, that's just too many spurious
> false-fails to have a usable workflow.
> 
> We do have some tools to make stress testing easier, but those need to
> be better documented. We'll fix that. The tools we have which also
> pull out the metadata etc also need documenting - right now they
> aren't because they're under _extremely_ heavy development, but they
> can be further enhanced to e.g. pull out the igt results automatically
> and point very clearly to the cause. Also on the to-do.

Thanks :)

> > > Only maintainers can actually fix the drivers (or the tests tbf). But
> > > doing the testing does let us be really clear to everyone what the
> > > actual state is, and that way people can make informed decisions too.
> > > And the only way we're going to drive the test rate down is by the
> > > subsystem maintainers enforcing it.
> >
> > Just FYI, I'm not on the other side of the fence there, I'd really like
> > to have some kind of validation. I talked about it at XDC some years
> > ago, and discussed it several people at length over the years. So I'm
> > definitely not in the CI-is-bad camp.
> >
> > > Does that make sense on where I'm (and I think a lot of others are)
> > > coming from?
> >
> > That makes sense from your perspective, but it's not clear to me how you
> > can expect maintainers to own the tests if they were never involved in
> > the process.
> >
> > They are not in Cc of the flaky tests patches, they are not reported
> > that the bug is failing, how can they own that process if we never
> > reached out and involved them?
> >
> > We're all overworked, you can't expect them to just look at the flaky
> > test list every now and then and figure it out.
> 
> Absolutely. We got acks (or at least not-nacks) from the driver
> developers, but yeah, they should absolutely be part of the loop for
> those updates. I don't think we can necessarily block on them though.
> Say we add vc4 KMS tests, then after a backmerge we start to see a
> bunch of flakes on it, but you're sitting on a beach for a couple of
> weeks.

I really like where this is going :)

> If we wait for you to get back

... Oh :(

> see it, and merge it, then that's two weeks of people submitting
> Rockchip driver changes and getting told that their changes failed CI.
> That's exactly what we want to avoid, because it erodes confidence and
> usefulness of CI when people expect failures and ignore them by
> default.

Yeah, sure. That's why I mentioned earlier that we should merge it if
there wasn't any fix or answer. Of course I don't expect the entire CI
system to wait for my return, but when I get back I still want to have
some trace that it happened and it still could be a good idea to look
into it.

> So I would say that it's reasonable for expectations to be updated
> according to what actually happens in practice, but also to make sure
> that the maintainers are explicitly informed and kept in the loop, and
> not just surprised when they look at the lists and see a bunch of
> stuff happened without their knowledge.
> 
> Again there's much more to be done on the tooling here. Part of it is
> CLI tools and automation, part of it is dashboards and
> easily-digestible reporting, and then there's integration with things
> like KernelCI. KCI(DB) is actually quite high up on the list, but
> we're mostly waiting until a lot of the KCI rework happens so we can
> actually properly integrate with the new system.
> 
> Right now a lot of the tooling we have is pretty involved - for
> example, we do have ci-collate as a Python library which can inspect a
> number of pipelines, pull out detailed status and logs, etc, but it
> mostly needs to be used as a library with bespoke scripts, rather than
> a ready-made tool. Work on that is ongoing to make it way more clear
> and accessible though.

I think at the moment, putting the flake list under the driver
MAINTAINERS entry would be enough. We can probably improve it further,
but that should give a reasonably reliable heads-up which can work
pretty much today.

> So I think it sounds like we're on the same page and going exactly in
> the same direction, just that this is a starting point rather than the
> desired end state. And the main point is that having a set of
> xfails/flakes parachuted in with little to no context is trying to get
> an MVP bootstrapped, rather than how we expect things to go in future.
> Does that sound about right?

That sounds perfect :)

Thanks a lot
Maxime

On 15/09/2023 12:08, Daniel Stone wrote:
> Hey,
> 
> On Thu, 14 Sept 2023 at 10:54, Maxime Ripard <mripard@kernel.org> wrote:
>> On Tue, Sep 12, 2023 at 02:16:41PM +0100, Daniel Stone wrote:
>>> Hopefully less mangled formatting this time: turns out Thunderbird +
>>> plain text is utterly unreadable, so that's one less MUA that is
>>> actually usable to send email to kernel lists without getting shouted
>>> at.
>>
>> Sorry if it felt that way, it definitely wasn't my intention to shout at
>> you. Email is indeed kind of a pain to deal with, and I wanted to keep
>> the discussion going.
> 
> My bad - I didn't mean you _at all_. I was thinking of other, much
> less pleasant, kernel maintainers, and the ongoing struggle of
> attempting to actually send well-formatted email to kernel lists in
> 2023.
> 
>>> I don't quite see the same picture from your side though. For example,
>>> my reading of what you've said is that flaky tests are utterly
>>> unacceptable, as are partial runs, and we shouldn't pretend otherwise.
>>> With your concrete example (which is really helpful, so thanks), what
>>> happens to the MT8173 hdmi-inject test? Do we skip all MT8173 testing
>>> until it's perfect, or does MT8173 testing always fail because that
>>> test does?
>>
>> It's not clear to me why that test is even running in the first place?
>> There's been some confusion on my side here about what we're going to
>> test with this. You've mentioned Mesa and GPUs before, but that's a KMS
>> test so there must be more to it.
>>
>> Either way, it's a relevant test so I guess why not. It turns out that
>> the test is indeed flaky, I guess we could add it to the flaky tests
>> list.
>>
>> BUT
>>
>> I want to have every opportunity to fix whatever that failure is.
> 
> Agreed so far!
> 
>> So:
>>
>>    - Is the test broken? If so, we should report it to IGT dev and remove
>>      it from the test suite.
>>    - If not, is that test failure have been reported to the driver author?
>>    - If no answer/fix, we can add it to the flaky tests list, but do we
>>      have some way to reproduce the test failure?
>>
>> The last part is especially critical. Looking at the list itself, I have
>> no idea what board, kernel version, configuration, or what the failure
>> rate was. Assuming I spend some time looking at the infra to find the
>> board and configuration, how many times do I have to run the tests to
>> expect to reproduce the failure (and thus consider it fixed if it
>> doesn't occur anymore).
>>
>> Like, with that board and test, if my first 100 runs of the test work
>> fine, is it reasonable for me to consider it fixed, or is it only
>> supposed to happen once every 1000 runs?

I wonder if this should be an overall policy or just let the maintainer 
to decide.

In any case these stress tests must be run from time to time to verify 
if flakes are still flakes. We could do it automatically but we need to 
evaluate how to do it properly so it doesn't overload the infra.

>>
>> So, ideally, having some (mandatory) metadata in the test lists with a
>> link to the bug report, the board (DT name?) it happened with, the
>> version and configuration it was first seen with, and an approximation
>> of the failure rate for every flaky test list.
>>
>> I understand that it's probably difficult to get that after the fact on
>> the tests that were already merged, but I'd really like to get that
>> enforced for every new test going forward.
>>
>> That should hopefully get us in a much better position to fix some of
>> those tests issues. And failing that, I can't see how that's
>> sustainable.
> 
> OK yeah, and we're still agreed here. That is definitely the standard
> we should be aiming for.  It is there for some - see
> drivers/gpu/drm/ci/xfails/rockchip-rk3288-skips.txt, but should be
> there for the rest, it's true. (The specific board/DT it was observed
> on can be easily retconned because we only run on one specific board
> type per driver, again to make things more predictable; we could go
> back and retrospectively add those in a header comment?)
> 
> For flakes, it can be hard to pin them down, because, well, they're
> flaky. Usually when we add things in Mesa (sorry to keep coming back
> to Mesa - it's not to say that it's the objective best thing that
> everything should follow, only that it's the thing we have the most
> experience with that we know works well), we do a manual bisect and
> try to pin the blame on a specific merge request which looks like the
> most likely culprit. If nothing obvious jumps out, we just note when
> it was first observed and provide some sample job logs. But yeah, it
> should be more verbose.
> 
> FWIW, the reason it wasn't done here - not to say that it shouldn't
> have been done better, but here we are - is that we just hammered a
> load of test runs, vacuumed up the results with a script, and that's
> what generated those files. Given the number of tests and devices, it
> was hard to narrow each down individually, but yeah, it is something
> which really wants further analysis and drilling into. It's a good
> to-do, and I agree it should be the standard going forward.

Yes, during development I was just worried about to get a pipeline that 
would succeed most of the time (otherwise people would start ignoring 
when it fails), so they just got run a couple of times and a script 
filled the flakes list.
For me the idea was "let's get a starting point" first, but yeah, we 
need to improve how we deal with it from now on.

> 
>> And Mesa does show what I'm talking about:
>>
>> $ find -name *-flakes.txt | xargs git diff --stat  e58a10af640ba58b6001f5c5ad750b782547da76
>> [...]
>>
>> In the history of Mesa, there's never been a single test removed from a
>> flaky test list.
> 
> As Rob says, that's definitely wrong. But there is a good point in
> there: how do you know a test isn't flaky anymore? 100 runs is a
> reasonable benchmark, but 1000 is ideal. At a 1% failure rate, with 20
> devices, that's just too many spurious false-fails to have a usable
> workflow.
> 
> We do have some tools to make stress testing easier, but those need to
> be better documented. We'll fix that. The tools we have which also
> pull out the metadata etc also need documenting - right now they
> aren't because they're under _extremely_ heavy development, but they
> can be further enhanced to e.g. pull out the igt results automatically
> and point very clearly to the cause. Also on the to-do.
> 
>>> Only maintainers can actually fix the drivers (or the tests tbf). But
>>> doing the testing does let us be really clear to everyone what the
>>> actual state is, and that way people can make informed decisions too.
>>> And the only way we're going to drive the test rate down is by the
>>> subsystem maintainers enforcing it.
>>
>> Just FYI, I'm not on the other side of the fence there, I'd really like
>> to have some kind of validation. I talked about it at XDC some years
>> ago, and discussed it several people at length over the years. So I'm
>> definitely not in the CI-is-bad camp.
>>
>>> Does that make sense on where I'm (and I think a lot of others are)
>>> coming from?
>>
>> That makes sense from your perspective, but it's not clear to me how you
>> can expect maintainers to own the tests if they were never involved in
>> the process.
>>
>> They are not in Cc of the flaky tests patches, they are not reported
>> that the bug is failing, how can they own that process if we never
>> reached out and involved them?
>>
>> We're all overworked, you can't expect them to just look at the flaky
>> test list every now and then and figure it out.
> 
> Absolutely. We got acks (or at least not-nacks) from the driver
> developers, but yeah, they should absolutely be part of the loop for
> those updates. I don't think we can necessarily block on them though.
> Say we add vc4 KMS tests, then after a backmerge we start to see a
> bunch of flakes on it, but you're sitting on a beach for a couple of
> weeks. If we wait for you to get back, see it, and merge it, then
> that's two weeks of people submitting Rockchip driver changes and
> getting told that their changes failed CI. That's exactly what we want
> to avoid, because it erodes confidence and usefulness of CI when
> people expect failures and ignore them by default.
> 
> So I would say that it's reasonable for expectations to be updated
> according to what actually happens in practice, but also to make sure
> that the maintainers are explicitly informed and kept in the loop, and
> not just surprised when they look at the lists and see a bunch of
> stuff happened without their knowledge.

I was thinking in adding entries in MAINTAINERS file pointing to each 
flake/skip/fails list file to their maintainers, so get_maintainers.pl 
can return the right thing.

> 
> Again there's much more to be done on the tooling here. Part of it is
> CLI tools and automation, part of it is dashboards and
> easily-digestible reporting, and then there's integration with things
> like KernelCI. KCI(DB) is actually quite high up on the list, but
> we're mostly waiting until a lot of the KCI rework happens so we can
> actually properly integrate with the new system.
> 
> Right now a lot of the tooling we have is pretty involved - for
> example, we do have ci-collate as a Python library which can inspect a
> number of pipelines, pull out detailed status and logs, etc, but it
> mostly needs to be used as a library with bespoke scripts, rather than
> a ready-made tool. Work on that is ongoing to make it way more clear
> and accessible though.
> 
> So I think it sounds like we're on the same page and going exactly in
> the same direction, just that this is a starting point rather than the
> desired end state. And the main point is that having a set of
> xfails/flakes parachuted in with little to no context is trying to get
> an MVP bootstrapped, rather than how we expect things to go in future.
> Does that sound about right?
> 
> Cheers,
> Daniel

Regards,
Helen

On Mon, Sep 18, 2023 at 06:35:13PM -0300, Helen Koike wrote:
> > > > I don't quite see the same picture from your side though. For example,
> > > > my reading of what you've said is that flaky tests are utterly
> > > > unacceptable, as are partial runs, and we shouldn't pretend otherwise.
> > > > With your concrete example (which is really helpful, so thanks), what
> > > > happens to the MT8173 hdmi-inject test? Do we skip all MT8173 testing
> > > > until it's perfect, or does MT8173 testing always fail because that
> > > > test does?
> > > 
> > > It's not clear to me why that test is even running in the first place?
> > > There's been some confusion on my side here about what we're going to
> > > test with this. You've mentioned Mesa and GPUs before, but that's a KMS
> > > test so there must be more to it.
> > > 
> > > Either way, it's a relevant test so I guess why not. It turns out that
> > > the test is indeed flaky, I guess we could add it to the flaky tests
> > > list.
> > > 
> > > BUT
> > > 
> > > I want to have every opportunity to fix whatever that failure is.
> > 
> > Agreed so far!
> > 
> > > So:
> > > 
> > >    - Is the test broken? If so, we should report it to IGT dev and remove
> > >      it from the test suite.
> > >    - If not, is that test failure have been reported to the driver author?
> > >    - If no answer/fix, we can add it to the flaky tests list, but do we
> > >      have some way to reproduce the test failure?
> > > 
> > > The last part is especially critical. Looking at the list itself, I have
> > > no idea what board, kernel version, configuration, or what the failure
> > > rate was. Assuming I spend some time looking at the infra to find the
> > > board and configuration, how many times do I have to run the tests to
> > > expect to reproduce the failure (and thus consider it fixed if it
> > > doesn't occur anymore).
> > > 
> > > Like, with that board and test, if my first 100 runs of the test work
> > > fine, is it reasonable for me to consider it fixed, or is it only
> > > supposed to happen once every 1000 runs?
> 
> I wonder if this should be an overall policy or just let the maintainer to
> decide.
> 
> In any case these stress tests must be run from time to time to verify if
> flakes are still flakes. We could do it automatically but we need to
> evaluate how to do it properly so it doesn't overload the infra.

That would be a great thing to do, but we can also reasonably expect
that we will have other farms that may not run those tests on a regular
basis, and we will have some manual testing too so I think it's still
valuable.

> > > So, ideally, having some (mandatory) metadata in the test lists with a
> > > link to the bug report, the board (DT name?) it happened with, the
> > > version and configuration it was first seen with, and an approximation
> > > of the failure rate for every flaky test list.
> > > 
> > > I understand that it's probably difficult to get that after the fact on
> > > the tests that were already merged, but I'd really like to get that
> > > enforced for every new test going forward.
> > > 
> > > That should hopefully get us in a much better position to fix some of
> > > those tests issues. And failing that, I can't see how that's
> > > sustainable.
> > 
> > OK yeah, and we're still agreed here. That is definitely the standard
> > we should be aiming for.  It is there for some - see
> > drivers/gpu/drm/ci/xfails/rockchip-rk3288-skips.txt, but should be
> > there for the rest, it's true. (The specific board/DT it was observed
> > on can be easily retconned because we only run on one specific board
> > type per driver, again to make things more predictable; we could go
> > back and retrospectively add those in a header comment?)
> > 
> > For flakes, it can be hard to pin them down, because, well, they're
> > flaky. Usually when we add things in Mesa (sorry to keep coming back
> > to Mesa - it's not to say that it's the objective best thing that
> > everything should follow, only that it's the thing we have the most
> > experience with that we know works well), we do a manual bisect and
> > try to pin the blame on a specific merge request which looks like the
> > most likely culprit. If nothing obvious jumps out, we just note when
> > it was first observed and provide some sample job logs. But yeah, it
> > should be more verbose.
> > 
> > FWIW, the reason it wasn't done here - not to say that it shouldn't
> > have been done better, but here we are - is that we just hammered a
> > load of test runs, vacuumed up the results with a script, and that's
> > what generated those files. Given the number of tests and devices, it
> > was hard to narrow each down individually, but yeah, it is something
> > which really wants further analysis and drilling into. It's a good
> > to-do, and I agree it should be the standard going forward.
> 
> Yes, during development I was just worried about to get a pipeline that
> would succeed most of the time (otherwise people would start ignoring when
> it fails), so they just got run a couple of times and a script filled the
> flakes list.
> For me the idea was "let's get a starting point" first, but yeah, we need to
> improve how we deal with it from now on.

Yeah, like I said, there's not much we can do for those 250-ish flakes
we currently have in tree at the moment. I'd prefer to stay at 250 tests
with not enough context than keep expanding that list :)

> > > And Mesa does show what I'm talking about:
> > > 
> > > $ find -name *-flakes.txt | xargs git diff --stat  e58a10af640ba58b6001f5c5ad750b782547da76
> > > [...]
> > > 
> > > In the history of Mesa, there's never been a single test removed from a
> > > flaky test list.
> > 
> > As Rob says, that's definitely wrong. But there is a good point in
> > there: how do you know a test isn't flaky anymore? 100 runs is a
> > reasonable benchmark, but 1000 is ideal. At a 1% failure rate, with 20
> > devices, that's just too many spurious false-fails to have a usable
> > workflow.
> > 
> > We do have some tools to make stress testing easier, but those need to
> > be better documented. We'll fix that. The tools we have which also
> > pull out the metadata etc also need documenting - right now they
> > aren't because they're under _extremely_ heavy development, but they
> > can be further enhanced to e.g. pull out the igt results automatically
> > and point very clearly to the cause. Also on the to-do.
> > 
> > > > Only maintainers can actually fix the drivers (or the tests tbf). But
> > > > doing the testing does let us be really clear to everyone what the
> > > > actual state is, and that way people can make informed decisions too.
> > > > And the only way we're going to drive the test rate down is by the
> > > > subsystem maintainers enforcing it.
> > > 
> > > Just FYI, I'm not on the other side of the fence there, I'd really like
> > > to have some kind of validation. I talked about it at XDC some years
> > > ago, and discussed it several people at length over the years. So I'm
> > > definitely not in the CI-is-bad camp.
> > > 
> > > > Does that make sense on where I'm (and I think a lot of others are)
> > > > coming from?
> > > 
> > > That makes sense from your perspective, but it's not clear to me how you
> > > can expect maintainers to own the tests if they were never involved in
> > > the process.
> > > 
> > > They are not in Cc of the flaky tests patches, they are not reported
> > > that the bug is failing, how can they own that process if we never
> > > reached out and involved them?
> > > 
> > > We're all overworked, you can't expect them to just look at the flaky
> > > test list every now and then and figure it out.
> > 
> > Absolutely. We got acks (or at least not-nacks) from the driver
> > developers, but yeah, they should absolutely be part of the loop for
> > those updates. I don't think we can necessarily block on them though.
> > Say we add vc4 KMS tests, then after a backmerge we start to see a
> > bunch of flakes on it, but you're sitting on a beach for a couple of
> > weeks. If we wait for you to get back, see it, and merge it, then
> > that's two weeks of people submitting Rockchip driver changes and
> > getting told that their changes failed CI. That's exactly what we want
> > to avoid, because it erodes confidence and usefulness of CI when
> > people expect failures and ignore them by default.
> > 
> > So I would say that it's reasonable for expectations to be updated
> > according to what actually happens in practice, but also to make sure
> > that the maintainers are explicitly informed and kept in the loop, and
> > not just surprised when they look at the lists and see a bunch of
> > stuff happened without their knowledge.
> 
> I was thinking in adding entries in MAINTAINERS file pointing to each
> flake/skip/fails list file to their maintainers, so get_maintainers.pl can
> return the right thing.

Yeah, I think it's the best thing to do at the moment. It's cheap and
will work ok.

Maxime

On Thu, Sep 14, 2023 at 2:54 AM Maxime Ripard <mripard@kernel.org> wrote:
>
> Hi,
>
> On Tue, Sep 12, 2023 at 02:16:41PM +0100, Daniel Stone wrote:
> > Hopefully less mangled formatting this time: turns out Thunderbird +
> > plain text is utterly unreadable, so that's one less MUA that is
> > actually usable to send email to kernel lists without getting shouted
> > at.
>
> Sorry if it felt that way, it definitely wasn't my intention to shout at
> you. Email is indeed kind of a pain to deal with, and I wanted to keep
> the discussion going.
>
> > On Mon, 11 Sept 2023 at 15:46, Maxime Ripard <mripard@kernel.org> wrote:
> > > On Mon, Sep 11, 2023 at 03:30:55PM +0200, Michel Dänzer wrote:
> > > > > There's in 6.6-rc1 around 240 reported flaky tests. None of them have
> > > > > any context. That new series hads a few dozens too, without any context
> > > > > either. And there's no mention about that being a plan, or a patch
> > > > > adding a new policy for all tests going forward.
> > > >
> > > > That does sound bad, would need to be raised in review.
> > > >
> > > > > Any concern I raised were met with a giant "it worked on Mesa" handwave
> > > >
> > > > Lessons learned from years of experience with big real-world CI
> > > > systems like this are hardly "handwaving".
> > >
> > > Your (and others) experience certainly isn't. It is valuable, welcome,
> > > and very much appreciated.
> > >
> > > However, my questions and concerns being ignored time and time again
> > > about things like what is the process is going to be like, what is going
> > > to be tested, who is going to be maintaining that test list, how that
> > > interacts with stable, how we can possibly audit the flaky tests list,
> > > etc. have felt like they were being handwaived away.
> >
> > Sorry it ended up coming across like that. It wasn't the intent.
> >
> > > I'm not saying that because I disagree, I still do on some, but that's
> > > fine to some extent. However, most of these issues are not so much an
> > > infrastructure issue, but a community issue. And I don't even expect a
> > > perfect solution right now, unlike what you seem to think. But I do
> > > expect some kind of plan instead of just ignoring that problem.
> > >
> > > Like, I had to ask the MT8173 question 3 times in order to get an
> > > answer, and I'm still not sure what is going to be done to address that
> > > particular issue.
> > >
> > > So, I'm sorry, but I certainly feel like it here.
> >
> > I don't quite see the same picture from your side though. For example,
> > my reading of what you've said is that flaky tests are utterly
> > unacceptable, as are partial runs, and we shouldn't pretend otherwise.
> > With your concrete example (which is really helpful, so thanks), what
> > happens to the MT8173 hdmi-inject test? Do we skip all MT8173 testing
> > until it's perfect, or does MT8173 testing always fail because that
> > test does?
>
> It's not clear to me why that test is even running in the first place?
> There's been some confusion on my side here about what we're going to
> test with this. You've mentioned Mesa and GPUs before, but that's a KMS
> test so there must be more to it.
>
> Either way, it's a relevant test so I guess why not. It turns out that
> the test is indeed flaky, I guess we could add it to the flaky tests
> list.
>
> BUT
>
> I want to have every opportunity to fix whatever that failure is.
>
> So:
>
>   - Is the test broken? If so, we should report it to IGT dev and remove
>     it from the test suite.
>   - If not, is that test failure have been reported to the driver author?
>   - If no answer/fix, we can add it to the flaky tests list, but do we
>     have some way to reproduce the test failure?
>
> The last part is especially critical. Looking at the list itself, I have
> no idea what board, kernel version, configuration, or what the failure
> rate was. Assuming I spend some time looking at the infra to find the
> board and configuration, how many times do I have to run the tests to
> expect to reproduce the failure (and thus consider it fixed if it
> doesn't occur anymore).

Are you looking for a link to the pipeline, where you can get job
output from the various stages (which should also include igt output)?

Perhaps someone can point you to a pipeline with mtk (I don't have
that in the drm/msm tree yet)

>
> Like, with that board and test, if my first 100 runs of the test work
> fine, is it reasonable for me to consider it fixed, or is it only
> supposed to happen once every 1000 runs?
>
> So, ideally, having some (mandatory) metadata in the test lists with a
> link to the bug report, the board (DT name?) it happened with, the
> version and configuration it was first seen with, and an approximation
> of the failure rate for every flaky test list.
>
> I understand that it's probably difficult to get that after the fact on
> the tests that were already merged, but I'd really like to get that
> enforced for every new test going forward.
>
> That should hopefully get us in a much better position to fix some of
> those tests issues. And failing that, I can't see how that's
> sustainable.
>
> And Mesa does show what I'm talking about:
>
> $ find -name *-flakes.txt | xargs git diff --stat  e58a10af640ba58b6001f5c5ad750b782547da76
>  src/amd/ci/deqp-radv-stoney-aco-flakes.txt                        |  18 +++++
>  src/broadcom/ci/deqp-v3d-rpi4-flakes.txt                          |   2 +
>  src/broadcom/ci/deqp-v3dv-rpi4-flakes.txt                         |  15 ++++
>  src/broadcom/ci/deqp-vc4-rpi3-flakes.txt                          |  30 ++++++++
>  src/broadcom/ci/piglit-v3d-rpi4-flakes.txt                        |  14 ++++
>  src/broadcom/ci/piglit-vc4-rpi3-flakes.txt                        |   5 ++
>  src/freedreno/ci/deqp-freedreno-a307-flakes.txt                   |   6 ++
>  src/freedreno/ci/deqp-freedreno-a530-flakes.txt                   |  31 +++++++++
>  src/freedreno/ci/deqp-freedreno-a630-bypass-flakes.txt            |  11 +++
>  src/freedreno/ci/deqp-freedreno-a630-flakes.txt                   | 109 ++++++++++++++++++++++++++++++
>  src/gallium/drivers/freedreno/ci/piglit-freedreno-a530-flakes.txt |  21 ++++++
>  src/gallium/drivers/freedreno/ci/piglit-freedreno-a630-flakes.txt |  31 +++++++++
>  src/gallium/drivers/llvmpipe/ci/deqp-llvmpipe-flakes.txt          |   4 ++
>  src/gallium/drivers/panfrost/ci/deqp-panfrost-g52-flakes.txt      |   6 ++
>  src/gallium/drivers/panfrost/ci/deqp-panfrost-t720-flakes.txt     |   4 ++
>  src/gallium/drivers/panfrost/ci/deqp-panfrost-t760-flakes.txt     |   1 +
>  src/gallium/drivers/panfrost/ci/deqp-panfrost-t860-flakes.txt     |   1 +
>  src/gallium/drivers/radeonsi/ci/piglit-radeonsi-stoney-flakes.txt |   1 +
>  src/gallium/drivers/softpipe/ci/deqp-softpipe-flakes.txt          |   2 +
>  src/gallium/drivers/virgl/ci/deqp-virgl-gl-flakes.txt             |   3 +
>  src/gallium/drivers/zink/ci/deqp-zink-lvp-flakes.txt              |   0
>  src/gallium/drivers/zink/ci/piglit-zink-lvp-flakes.txt            |   0
>  src/gallium/frontends/lavapipe/ci/deqp-lvp-flakes.txt             |   1 +
>  23 files changed, 316 insertions(+)
>
> In the history of Mesa, there's never been a single test removed from a
> flaky test list.

That is very definitely not true, see for ex
a297624182218b24d5accb975101448b0bf97077

This command is just showing you the diffstat from the start of time,
which ofc is going to be all + and no -

BR,
-R

>
> > Both have their downsides. Not doing any testing has the obvious
> > downside, and means that the driver can get worse until it gets
> > perfect. Always marking the test as failed means that the test results
> > are useless: if failure is expected, then red is good. I mean, say
> > you're contributing a patch to fix some documentation or add a helper
> > to common code which only v3d uses. The test results come back, and
> > your branch is failing tests on MT8173, specifically the
> > hdmi-inject@4k test. What then? Either as a senior contributor you
> > 'know' that's the case, or as a casual contributor you get told 'oh
> > yeah, don't worry about the test results, they always fail'. Both lead
> > to the same outcome, which is that no-one pays any attention to the
> > results, and they get worse.
>
> I believe the above would achieve a better result for us kernel folks,
> while still providing enough flexibility for you.
>
> > What we do agree on is that yes, those tests should absolutely be
> > fixed, and not just swept under the rug. Part of this is having
> > maintainers actually meaningfully own their test results. For example,
> > I'm looking at the expectation lists for the Intel gen in my laptop,
> > and I'm seeing a lot of breakage in blending tests, as well as
> > dual-display fails which include the resolution of my external
> > display.
>
> I just bought a 7900XT and I'm pretty much in the same situation, so I
> definitely feel what you're saying :)
>
> However...
>
> > I'd expect the Intel driver maintainers to look at them, get them
> > fixed, and gradually prune those xfails/flakes down towards zero.
> >
> > If the maintainers don't own it though, then it's not going to get
> > fixed. And we are exactly where we are today: broken plane blending
> > and 1440p on KBL, broken EDID injection on MT8173, and broken atomic
> > commits on stoney. Without stronger action from the maintainers (e.g.
> > throwing i915 out of the tree until it has 100% pass 100% of the
> > time), adding testing isn't making the situation better or worse in
> > and of itself. What it _is_ doing though, is giving really clear
> > documentation of the status of each driver, and backing that up by
> > verifying it.
>
> ... I haven't reported any of the failures yet so I don't really expect
> them to be fixed by themselves. That's on me.
>
> > Only maintainers can actually fix the drivers (or the tests tbf). But
> > doing the testing does let us be really clear to everyone what the
> > actual state is, and that way people can make informed decisions too.
> > And the only way we're going to drive the test rate down is by the
> > subsystem maintainers enforcing it.
>
> Just FYI, I'm not on the other side of the fence there, I'd really like
> to have some kind of validation. I talked about it at XDC some years
> ago, and discussed it several people at length over the years. So I'm
> definitely not in the CI-is-bad camp.
>
> > Does that make sense on where I'm (and I think a lot of others are)
> > coming from?
>
> That makes sense from your perspective, but it's not clear to me how you
> can expect maintainers to own the tests if they were never involved in
> the process.
>
> They are not in Cc of the flaky tests patches, they are not reported
> that the bug is failing, how can they own that process if we never
> reached out and involved them?
>
> We're all overworked, you can't expect them to just look at the flaky
> test list every now and then and figure it out.
>
> Maxime

replying to a couple points on this thread

On Wed, Aug 30, 2023 at 6:25 AM Helen Koike <helen.koike@collabora.com> wrote:
>
> Hi all,
>
> Thanks for you comments.
>
> On 30/08/2023 08:37, Maxime Ripard wrote:
> > On Wed, Aug 30, 2023 at 01:58:31PM +0300, Jani Nikula wrote:
> >> On Wed, 30 Aug 2023, Maxime Ripard <mripard@kernel.org> wrote:
> >>> On Tue, Aug 22, 2023 at 04:26:06PM +0200, Daniel Vetter wrote:
> >>>> On Fri, Aug 11, 2023 at 02:19:53PM -0300, Helen Koike wrote:
> >>>>> From: Tomeu Vizoso <tomeu.vizoso@collabora.com>
> >>>>>
> >>>>> Developers can easily execute several tests on different devices
> >>>>> by just pushing their branch to their fork in a repository hosted
> >>>>> on gitlab.freedesktop.org which has an infrastructure to run jobs
> >>>>> in several runners and farms with different devices.
> >>>>>
> >>>>> There are also other automated tools that uprev dependencies,
> >>>>> monitor the infra, and so on that are already used by the Mesa
> >>>>> project, and we can reuse them too.
> >>>>>
> >>>>> Also, store expectations about what the DRM drivers are supposed
> >>>>> to pass in the IGT test suite. By storing the test expectations
> >>>>> along with the code, we can make sure both stay in sync with each
> >>>>> other so we can know when a code change breaks those expectations.
> >>>>>
> >>>>> Also, include a configuration file that points to the out-of-tree
> >>>>> CI scripts.
> >>>>>
> >>>>> This will allow all contributors to drm to reuse the infrastructure
> >>>>> already in gitlab.freedesktop.org to test the driver on several
> >>>>> generations of the hardware.
> >>>>>
> >>>>> Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
> >>>>> Signed-off-by: Helen Koike <helen.koike@collabora.com>
> >>>>> Acked-by: Daniel Stone <daniels@collabora.com>
> >>>>> Acked-by: Rob Clark <robdclark@gmail.com>
> >>>>> Tested-by: Rob Clark <robdclark@gmail.com>
> >>>>
> >>>> Ok I pushed this into a topic/drm-ci branch in drm.git and asked sfr to
> >>>> include that branch in linux-next.
> >>>>
> >>>> But also I'd like to see a lot more acks here, we should be able to at
> >>>> least pile up a bunch of (driver) maintainers from drm-misc in support of
> >>>> this. Also maybe media, at least I've heard noises that they're maybe
> >>>> interested too? Plus anyone else, the more the better.
> >>>
> >>> I'm not really convinced by that approach at all, and most of the issues
> >>> I see are shown by the follow-up series here:
> >>
> >> I'm not fully convinced either, more like "let's see". In that narrow
> >> sense, ack. I don't see harm in trying, if you're also open to backing
> >> off in case it does not pan out.
> >>
> >>> https://lore.kernel.org/dri-devel/20230825122435.316272-1-vignesh.raman@collabora.com/
> >>>
> >>>    * We hardcode a CI farm setup into the kernel

We do have farm status out of tree so we don't need to push a kernel
patch for outages.

Other than that, I can go either way, with the scripts and related yml
in tree or out of tree.  But the expectation files (ie. the patch you
are complaining about) absolutely need to be in-tree.  It must be
possible to update them in sync with driver changes that fix tests.
There may be a bit of related churn initially ie. based on our lengthy
experience at this point with mesa CI, we can't know about all the
flaky tests until people start using CI in anger ;-)

>
>
> These could be out of tree.
>
> There is a version outside the kernel tree where you just point the CI
> configuration to a url:
> https://gitlab.freedesktop.org/gfx-ci/drm-ci/-/merge_requests/1
>
> We were discussing it here
> https://www.linuxtv.org/cgi-bin/mailman/private/linuxtv-ci/2023-August/000027.html
>
> (I guess Sima's reply didn't got into the mailing list) but the argument
> of not having out of tree repo is due to historical bad experience of
> having to sync the kernel with the code and it can become messy.
>
>
> >>>
> >>>    * We cannot trust that the code being run is actually the one being
> >>>      pushed into gitlab
>
>
> We can improve this if this is a requirement.
> For DTS configuration we can work with overlays (which is the current
> modification on that patchset). For other changes that are not suitable
> to upstream (and should be rare) we can see if we work with the
> `-external-fixes` approach or another approach, we can check it case by
> case to understand why it is not suitable for upstream.
>

IMHO the occasional need for extra patches is a fact of life with the
kernel development process, and we'll have to live with this until
_all_ kernel patches run thru pre-merge CI.  (I'm not holding my
breath, but who knows..)

If some particular board doesn't boot because of some early-rc
breakage elsewhere in the kernel, then we can't run CI.

>
> >>>
> >>>    * IMO, and I know we disagree here, any IGT test we enable for a given
> >>>      platform should work, period. Allowing failures and flaky tests just
> >>>      sweeps whatever issue is there under the rug. If the test is at
> >>>      fault, we should fix the test, if the driver / kernel is at fault,
> >>>      then I certainly want to know about it.
>
>
> I believe we need a baseline and understand the current status of tests.
> If you check the xfails folder in the patch you can see that I had to
> add a few tests on *-skips.txt since those tests crashes the system and
> other on *-fails.txt that are consistently not passing.
>
> Since the "any IGT test we enable for a given platform should work" is
> not a reality atm, we need to have a clear view about which tests are
> not corresponding to it, so we can start fixing. First we need to be
> aware of the issues so we can start fixing them, otherwise we will stay
> in the "no tests no failures" ground :)
>

I agree with Helen here.  We need a baseline.  And sometimes someone
might fix one test without realizing the same fix fixed another.  That
is the whole purpose of xfails.  We have a long history with this
process on the mesa side, and it works.

>
> >>
> >> At least for display, where this also depends on peripheral hardware,
> >> it's not an easy problem, really.
> >
> > Aside from the Chamelium tests, which tests actually rely on peripheral
> > hardware? On EDID and hotplug, sure, but that can easily be set up from
> > the userspace, or something like
> >
> > https://www.lindy-international.com/HDMI-2-0-EDID-Emulator.htm?websale8=ld0101.ld021102&pi=32115
> >
> >> How reliable do you need it to be? How many nines? Who is going to
> >> debug the issues that need hundreds or thousands of runs to reproduce?
> >> If a commit makes some test less reliable, how long is it going to
> >> take to even see that or pinpoint that?

As far as flakes, at this point we have some infrastructure for
tracking those on the mesa side.. we could probably extend drm/ci over
time to do similarly.

> > I mean, that's also true for failures or success then. How many times do
> > you need a test to run properly to qualify it as a meaningful test? How
> > do you know that it's not a flaky test?
> >
> > Ultimately, it's about trust. If, for a given test that just failed, I
> > can't be certain that it's because of the branch I just submitted, I
> > will just ignore the tests results after a while.

Until we have contributors directly submitting MRs (which is a place
I'd like to get to eventually) it will be the various maintainers
dealing with flakes/fails rather than individual contributors.  Ie.
the expected usage pattern would be maintainers pushing patches to a
staging branch which is then submitted as a gitlab MR where CI is run.
I would hope that maintainers have sufficient experience with the code
they are maintaining to evaluate the failure.

Basically, I think now is the time to get CI in place and start fine
tuning, before we have individual contributors submitting MRs
directly.

BR,
-R

> > This is already what plagues kernelci, and we should do better.
>
>
> This is something that is really nice on Mesa3D, a patch only gets
> merged if tests passes, which forces people to not ignore it, which
> forces the code to be fixed and the CI to be constantly maintained.
>
> Of course there are bad days there, but there is real value. Nice thread
> to check: https://gitlab.freedesktop.org/mesa/mesa/-/issues/8635 (thanks
> Alyssa for the feedback).
>
>
> >
> > And I'm sorry, but if some part of the kernel or driver just isn't
> > reliable, then we shouldn't claim it is (except for all the times it
> > isn't). If no-one has the time to look into it, fine, but flagging it
> > under a flaky test doesn't help anyone.
>
>
> At least we would know what is there that isn't reliable.
> We could also define policies like: failing one out of 10 is fine. And
> we can have scripts that do a stress test and report back.
> It is really nice for users to know this kind of status of drivers.
>
>
> Thanks for your feedback, please keep them coming :)
>
> Regards,
> Helen
>
>
> >
> > Like, from that patch, how can I know what is the issue with
> > kms_hdmi_inject@inject-4k or kms_addfb_basic@addfb25-bad-modifier on
> > mt8173. I certainly can't. And neither of those have anything to do with
> > peripheral hardware.
> >
> > Maxime