add Intel Query Processing Library (QPL) compression method
introduction
Signed-off-by: Yuan Liu <yuan1.liu@intel.com>
Reviewed-by: Nanhai Zou <nanhai.zou@intel.com>
---
docs/devel/migration/features.rst | 1 +
docs/devel/migration/qpl-compression.rst | 231 +++++++++++++++++++++++
2 files changed, 232 insertions(+)
create mode 100644 docs/devel/migration/qpl-compression.rst
diff --git a/docs/devel/migration/features.rst b/docs/devel/migration/features.rst
index d5ca7b86d5..bc98b65075 100644
--- a/docs/devel/migration/features.rst
+++ b/docs/devel/migration/features.rst
@@ -12,3 +12,4 @@ Migration has plenty of features to support different use cases.
virtio
mapped-ram
CPR
+ qpl-compression
diff --git a/docs/devel/migration/qpl-compression.rst b/docs/devel/migration/qpl-compression.rst
new file mode 100644
index 0000000000..42c7969d30
--- /dev/null
+++ b/docs/devel/migration/qpl-compression.rst
@@ -0,0 +1,231 @@
+===============
+QPL Compression
+===============
+The Intel Query Processing Library (Intel ``QPL``) is an open-source library to
+provide compression and decompression features and it is based on deflate
+compression algorithm (RFC 1951).
+
+The ``QPL`` compression relies on Intel In-Memory Analytics Accelerator(``IAA``)
+and Shared Virtual Memory(``SVM``) technology, they are new features supported
+from Intel 4th Gen Intel Xeon Scalable processors, codenamed Sapphire Rapids
+processor(``SPR``).
+
+For more ``QPL`` introduction, please refer to:
+
+https://intel.github.io/qpl/documentation/introduction_docs/introduction.html
+
+QPL Compression Framework
+=========================
+
+::
+
+ +----------------+ +------------------+
+ | MultiFD Service| |accel-config tool |
+ +-------+--------+ +--------+---------+
+ | |
+ | |
+ +-------+--------+ | Setup IAA
+ | QPL library | | Resources
+ +-------+---+----+ |
+ | | |
+ | +-------------+-------+
+ | Open IAA |
+ | Devices +-----+-----+
+ | |idxd driver|
+ | +-----+-----+
+ | |
+ | |
+ | +-----+-----+
+ +-----------+IAA Devices|
+ Submit jobs +-----------+
+ via enqcmd
+
+
+Intel In-Memory Analytics Accelerator (Intel IAA) Introduction
+================================================================
+
+Intel ``IAA`` is an accelerator that has been designed to help benefit
+in-memory databases and analytic workloads. There are three main areas
+that Intel ``IAA`` can assist with analytics primitives (scan, filter, etc.),
+sparse data compression and memory tiering.
+
+``IAA`` Manual Documentation:
+
+https://www.intel.com/content/www/us/en/content-details/721858/intel-in-memory-analytics-accelerator-architecture-specification
+
+IAA Device Enabling
+-------------------
+
+- Enabling ``IAA`` devices for platform configuration, please refer to:
+
+https://www.intel.com/content/www/us/en/content-details/780887/intel-in-memory-analytics-accelerator-intel-iaa.html
+
+- ``IAA`` device driver is ``Intel Data Accelerator Driver (idxd)``, it is
+ recommended that the minimum version of Linux kernel is 5.18.
+
+- Add ``"intel_iommu=on,sm_on"`` parameter to kernel command line
+ for ``SVM`` feature enabling.
+
+Here is an easy way to verify ``IAA`` device driver and ``SVM``, refer to:
+
+https://github.com/intel/idxd-config/tree/stable/test
+
+IAA Device Management
+---------------------
+
+The number of ``IAA`` devices will vary depending on the Xeon product model.
+On a ``SPR`` server, there can be a maximum of 8 ``IAA`` devices, with up to
+4 devices per socket.
+
+By default, all ``IAA`` devices are disabled and need to be configured and
+enabled by users manually.
+
+Check the number of devices through the following command
+
+.. code-block:: shell
+
+ # lspci -d 8086:0cfe
+ # 6a:02.0 System peripheral: Intel Corporation Device 0cfe
+ # 6f:02.0 System peripheral: Intel Corporation Device 0cfe
+ # 74:02.0 System peripheral: Intel Corporation Device 0cfe
+ # 79:02.0 System peripheral: Intel Corporation Device 0cfe
+ # e7:02.0 System peripheral: Intel Corporation Device 0cfe
+ # ec:02.0 System peripheral: Intel Corporation Device 0cfe
+ # f1:02.0 System peripheral: Intel Corporation Device 0cfe
+ # f6:02.0 System peripheral: Intel Corporation Device 0cfe
+
+IAA Device Configuration
+------------------------
+
+The ``accel-config`` tool is used to enable ``IAA`` devices and configure
+``IAA`` hardware resources(work queues and engines). One ``IAA`` device
+has 8 work queues and 8 processing engines, multiple engines can be assigned
+to a work queue via ``group`` attribute.
+
+One example of configuring and enabling an ``IAA`` device.
+
+.. code-block:: shell
+
+ # accel-config config-engine iax1/engine1.0 -g 0
+ # accel-config config-engine iax1/engine1.1 -g 0
+ # accel-config config-engine iax1/engine1.2 -g 0
+ # accel-config config-engine iax1/engine1.3 -g 0
+ # accel-config config-engine iax1/engine1.4 -g 0
+ # accel-config config-engine iax1/engine1.5 -g 0
+ # accel-config config-engine iax1/engine1.6 -g 0
+ # accel-config config-engine iax1/engine1.7 -g 0
+ # accel-config config-wq iax1/wq1.0 -g 0 -s 128 -p 10 -b 1 -t 128 -m shared -y user -n app1 -d user
+ # accel-config enable-device iax1
+ # accel-config enable-wq iax1/wq1.0
+
+.. note::
+ IAX is an early name for IAA
+
+- The ``IAA`` device index is 1, use ``ls -lh /sys/bus/dsa/devices/iax*``
+ command to query the ``IAA`` device index.
+
+- 8 engines and 1 work queue are configured in group 0, so all compression jobs
+ submitted to this work queue can be processed by all engines at the same time.
+
+- Set work queue attributes including the work mode, work queue size and so on.
+
+- Enable the ``IAA1`` device and work queue 1.0
+
+.. note::
+ Set work queue mode to shared mode, since ``QPL`` library only supports
+ shared mode
+
+For more detailed configuration, please refer to:
+
+https://github.com/intel/idxd-config/tree/stable/Documentation/accfg
+
+IAA Resources Allocation For Migration
+--------------------------------------
+
+There is no ``IAA`` resource configuration parameters for migration and
+``accel-config`` tool configuration cannot directly specify the ``IAA``
+resources used for migration.
+
+``QPL`` will use all work queues that are enabled and set to shared mode,
+and use all engines assigned to the work queues with shared mode.
+
+By default, ``QPL`` will only use the local ``IAA`` device for compression
+job processing. The local ``IAA`` device means that the CPU of the job
+submission and the ``IAA`` device are on the same socket, so one CPU
+can submit the jobs to up to 4 ``IAA`` devices.
+
+Shared Virtual Memory(SVM) Introduction
+=======================================
+
+An ability for an accelerator I/O device to operate in the same virtual
+memory space of applications on host processors. It also implies the
+ability to operate from pageable memory, avoiding functional requirements
+to pin memory for DMA operations.
+
+When using ``SVM`` technology, users do not need to reserve memory for the
+``IAA`` device and perform pin memory operation. The ``IAA`` device can
+directly access data using the virtual address of the process.
+
+For more ``SVM`` technology, please refer to:
+
+https://docs.kernel.org/next/x86/sva.html
+
+
+How To Use QPL Compression In Migration
+=======================================
+
+1 - Installation of ``accel-config`` tool and ``QPL`` library
+
+ - Install ``accel-config`` tool from https://github.com/intel/idxd-config
+ - Install ``QPL`` library from https://github.com/intel/qpl
+
+2 - Configure and enable ``IAA`` devices and work queues via ``accel-config``
+
+3 - Build ``Qemu`` with ``--enable-qpl`` parameter
+
+ E.g. configure --target-list=x86_64-softmmu --enable-kvm ``--enable-qpl``
+
+4 - Start VMs with ``sudo`` command or ``root`` permission
+
+ Use the ``sudo`` command or ``root`` privilege to start the source and
+ destination virtual machines, since migration service needs permission
+ to access ``IAA`` hardware resources.
+
+5 - Enable ``QPL`` compression during migration
+
+ Set ``migrate_set_parameter multifd-compression qpl`` when migrating, the
+ ``QPL`` compression does not support configuring the compression level, it
+ only supports one compression level.
+
+The Difference Between QPL And ZLIB
+===================================
+
+Although both ``QPL`` and ``ZLIB`` are based on the deflate compression
+algorithm, and ``QPL`` can support the header and tail of ``ZLIB``, ``QPL``
+is still not fully compatible with the ``ZLIB`` compression in the migration.
+
+``QPL`` only supports 4K history buffer, and ``ZLIB`` is 32K by default. The
+``ZLIB`` compressed data that ``QPL`` may not decompress correctly and
+vice versa.
+
+``QPL`` does not support the ``Z_SYNC_FLUSH`` operation in ``ZLIB`` streaming
+compression, current ``ZLIB`` implementation uses ``Z_SYNC_FLUSH``, so each
+``multifd`` thread has a ``ZLIB`` streaming context, and all page compression
+and decompression are based on this stream. ``QPL`` cannot decompress such data
+and vice versa.
+
+The introduction for ``Z_SYNC_FLUSH``, please refer to:
+
+https://www.zlib.net/manual.html
+
+The Best Practices
+==================
+
+When the virtual machine's pages are not populated and the ``IAA`` device is
+used, I/O page faults occur, which can impact performance due to a large number
+of flush ``IOTLB`` operations.
+
+Since the normal pages on the source side are all populated, ``IOTLB`` caused
+by I/O page fault will not occur. On the destination side, a large number
+of normal pages need to be loaded, so it is recommended to add ``-mem-prealloc``
+parameter on the destination side.
--
2.39.3On Wed, Mar 20, 2024 at 12:45:21AM +0800, Yuan Liu wrote:
> add Intel Query Processing Library (QPL) compression method
> introduction
>
> Signed-off-by: Yuan Liu <yuan1.liu@intel.com>
> Reviewed-by: Nanhai Zou <nanhai.zou@intel.com>
> ---
> docs/devel/migration/features.rst | 1 +
> docs/devel/migration/qpl-compression.rst | 231 +++++++++++++++++++++++
> 2 files changed, 232 insertions(+)
> create mode 100644 docs/devel/migration/qpl-compression.rst
>
> diff --git a/docs/devel/migration/features.rst b/docs/devel/migration/features.rst
> index d5ca7b86d5..bc98b65075 100644
> --- a/docs/devel/migration/features.rst
> +++ b/docs/devel/migration/features.rst
> @@ -12,3 +12,4 @@ Migration has plenty of features to support different use cases.
> virtio
> mapped-ram
> CPR
> + qpl-compression
> diff --git a/docs/devel/migration/qpl-compression.rst b/docs/devel/migration/qpl-compression.rst
> new file mode 100644
> index 0000000000..42c7969d30
> --- /dev/null
> +++ b/docs/devel/migration/qpl-compression.rst
> @@ -0,0 +1,231 @@
> +===============
> +QPL Compression
> +===============
> +The Intel Query Processing Library (Intel ``QPL``) is an open-source library to
> +provide compression and decompression features and it is based on deflate
> +compression algorithm (RFC 1951).
> +
> +The ``QPL`` compression relies on Intel In-Memory Analytics Accelerator(``IAA``)
> +and Shared Virtual Memory(``SVM``) technology, they are new features supported
> +from Intel 4th Gen Intel Xeon Scalable processors, codenamed Sapphire Rapids
> +processor(``SPR``).
> +
> +For more ``QPL`` introduction, please refer to:
> +
> +https://intel.github.io/qpl/documentation/introduction_docs/introduction.html
There're a bunch of links in this page, please consider switching all of
them to use the link formats of .rST:
Please refer to `QPL introduction page <https://...>`_.
> +
> +QPL Compression Framework
> +=========================
> +
> +::
> +
> + +----------------+ +------------------+
> + | MultiFD Service| |accel-config tool |
> + +-------+--------+ +--------+---------+
> + | |
> + | |
> + +-------+--------+ | Setup IAA
> + | QPL library | | Resources
> + +-------+---+----+ |
> + | | |
> + | +-------------+-------+
> + | Open IAA |
> + | Devices +-----+-----+
> + | |idxd driver|
> + | +-----+-----+
> + | |
> + | |
> + | +-----+-----+
> + +-----------+IAA Devices|
> + Submit jobs +-----------+
> + via enqcmd
> +
> +
> +Intel In-Memory Analytics Accelerator (Intel IAA) Introduction
> +================================================================
> +
> +Intel ``IAA`` is an accelerator that has been designed to help benefit
> +in-memory databases and analytic workloads. There are three main areas
> +that Intel ``IAA`` can assist with analytics primitives (scan, filter, etc.),
> +sparse data compression and memory tiering.
> +
> +``IAA`` Manual Documentation:
> +
> +https://www.intel.com/content/www/us/en/content-details/721858/intel-in-memory-analytics-accelerator-architecture-specification
> +
> +IAA Device Enabling
> +-------------------
> +
> +- Enabling ``IAA`` devices for platform configuration, please refer to:
> +
> +https://www.intel.com/content/www/us/en/content-details/780887/intel-in-memory-analytics-accelerator-intel-iaa.html
> +
> +- ``IAA`` device driver is ``Intel Data Accelerator Driver (idxd)``, it is
> + recommended that the minimum version of Linux kernel is 5.18.
> +
> +- Add ``"intel_iommu=on,sm_on"`` parameter to kernel command line
> + for ``SVM`` feature enabling.
> +
> +Here is an easy way to verify ``IAA`` device driver and ``SVM``, refer to:
> +
> +https://github.com/intel/idxd-config/tree/stable/test
> +
> +IAA Device Management
> +---------------------
> +
> +The number of ``IAA`` devices will vary depending on the Xeon product model.
> +On a ``SPR`` server, there can be a maximum of 8 ``IAA`` devices, with up to
> +4 devices per socket.
> +
> +By default, all ``IAA`` devices are disabled and need to be configured and
> +enabled by users manually.
> +
> +Check the number of devices through the following command
> +
> +.. code-block:: shell
> +
> + # lspci -d 8086:0cfe
> + # 6a:02.0 System peripheral: Intel Corporation Device 0cfe
> + # 6f:02.0 System peripheral: Intel Corporation Device 0cfe
> + # 74:02.0 System peripheral: Intel Corporation Device 0cfe
> + # 79:02.0 System peripheral: Intel Corporation Device 0cfe
> + # e7:02.0 System peripheral: Intel Corporation Device 0cfe
> + # ec:02.0 System peripheral: Intel Corporation Device 0cfe
> + # f1:02.0 System peripheral: Intel Corporation Device 0cfe
> + # f6:02.0 System peripheral: Intel Corporation Device 0cfe
> +
> +IAA Device Configuration
> +------------------------
> +
> +The ``accel-config`` tool is used to enable ``IAA`` devices and configure
> +``IAA`` hardware resources(work queues and engines). One ``IAA`` device
> +has 8 work queues and 8 processing engines, multiple engines can be assigned
> +to a work queue via ``group`` attribute.
> +
> +One example of configuring and enabling an ``IAA`` device.
> +
> +.. code-block:: shell
> +
> + # accel-config config-engine iax1/engine1.0 -g 0
> + # accel-config config-engine iax1/engine1.1 -g 0
> + # accel-config config-engine iax1/engine1.2 -g 0
> + # accel-config config-engine iax1/engine1.3 -g 0
> + # accel-config config-engine iax1/engine1.4 -g 0
> + # accel-config config-engine iax1/engine1.5 -g 0
> + # accel-config config-engine iax1/engine1.6 -g 0
> + # accel-config config-engine iax1/engine1.7 -g 0
> + # accel-config config-wq iax1/wq1.0 -g 0 -s 128 -p 10 -b 1 -t 128 -m shared -y user -n app1 -d user
> + # accel-config enable-device iax1
> + # accel-config enable-wq iax1/wq1.0
> +
> +.. note::
> + IAX is an early name for IAA
> +
> +- The ``IAA`` device index is 1, use ``ls -lh /sys/bus/dsa/devices/iax*``
> + command to query the ``IAA`` device index.
> +
> +- 8 engines and 1 work queue are configured in group 0, so all compression jobs
> + submitted to this work queue can be processed by all engines at the same time.
> +
> +- Set work queue attributes including the work mode, work queue size and so on.
> +
> +- Enable the ``IAA1`` device and work queue 1.0
> +
> +.. note::
> + Set work queue mode to shared mode, since ``QPL`` library only supports
> + shared mode
> +
> +For more detailed configuration, please refer to:
> +
> +https://github.com/intel/idxd-config/tree/stable/Documentation/accfg
> +
> +IAA Resources Allocation For Migration
> +--------------------------------------
> +
> +There is no ``IAA`` resource configuration parameters for migration and
> +``accel-config`` tool configuration cannot directly specify the ``IAA``
> +resources used for migration.
> +
> +``QPL`` will use all work queues that are enabled and set to shared mode,
> +and use all engines assigned to the work queues with shared mode.
> +
> +By default, ``QPL`` will only use the local ``IAA`` device for compression
> +job processing. The local ``IAA`` device means that the CPU of the job
> +submission and the ``IAA`` device are on the same socket, so one CPU
> +can submit the jobs to up to 4 ``IAA`` devices.
> +
> +Shared Virtual Memory(SVM) Introduction
> +=======================================
> +
> +An ability for an accelerator I/O device to operate in the same virtual
> +memory space of applications on host processors. It also implies the
> +ability to operate from pageable memory, avoiding functional requirements
> +to pin memory for DMA operations.
> +
> +When using ``SVM`` technology, users do not need to reserve memory for the
> +``IAA`` device and perform pin memory operation. The ``IAA`` device can
> +directly access data using the virtual address of the process.
> +
> +For more ``SVM`` technology, please refer to:
> +
> +https://docs.kernel.org/next/x86/sva.html
> +
> +
> +How To Use QPL Compression In Migration
> +=======================================
> +
> +1 - Installation of ``accel-config`` tool and ``QPL`` library
We can drop "1 " and stick with:
- item1
- item1.1
...
- item2
> +
> + - Install ``accel-config`` tool from https://github.com/intel/idxd-config
> + - Install ``QPL`` library from https://github.com/intel/qpl
> +
> +2 - Configure and enable ``IAA`` devices and work queues via ``accel-config``
> +
> +3 - Build ``Qemu`` with ``--enable-qpl`` parameter
> +
> + E.g. configure --target-list=x86_64-softmmu --enable-kvm ``--enable-qpl``
> +
> +4 - Start VMs with ``sudo`` command or ``root`` permission
> +
> + Use the ``sudo`` command or ``root`` privilege to start the source and
> + destination virtual machines, since migration service needs permission
> + to access ``IAA`` hardware resources.
> +
> +5 - Enable ``QPL`` compression during migration
> +
> + Set ``migrate_set_parameter multifd-compression qpl`` when migrating, the
> + ``QPL`` compression does not support configuring the compression level, it
> + only supports one compression level.
> +
> +The Difference Between QPL And ZLIB
> +===================================
> +
> +Although both ``QPL`` and ``ZLIB`` are based on the deflate compression
> +algorithm, and ``QPL`` can support the header and tail of ``ZLIB``, ``QPL``
> +is still not fully compatible with the ``ZLIB`` compression in the migration.
> +
> +``QPL`` only supports 4K history buffer, and ``ZLIB`` is 32K by default. The
> +``ZLIB`` compressed data that ``QPL`` may not decompress correctly and
> +vice versa.
> +
> +``QPL`` does not support the ``Z_SYNC_FLUSH`` operation in ``ZLIB`` streaming
> +compression, current ``ZLIB`` implementation uses ``Z_SYNC_FLUSH``, so each
> +``multifd`` thread has a ``ZLIB`` streaming context, and all page compression
> +and decompression are based on this stream. ``QPL`` cannot decompress such data
> +and vice versa.
> +
> +The introduction for ``Z_SYNC_FLUSH``, please refer to:
> +
> +https://www.zlib.net/manual.html
> +
> +The Best Practices
> +==================
> +
> +When the virtual machine's pages are not populated and the ``IAA`` device is
> +used, I/O page faults occur, which can impact performance due to a large number
> +of flush ``IOTLB`` operations.
AFAIU the IOTLB issue is not expected and can be fixed, while per our
discussion in the other thread, the DMA fault latency cannot.
I think we can mention the possibility of IOTLB flush issue but that
shouldn't be the major cause of such suggestion. Again, I think it'll be
great to mention how slow a DMA page fault can be, it can be compared in
migration performance difference, or just describe an average DMA page
fault latency, v.s. a processor fault. That might explain why we suggest
prefault the pages (comparing to a generic setup where the pages are always
faulted by processors).
> +
> +Since the normal pages on the source side are all populated, ``IOTLB`` caused
> +by I/O page fault will not occur. On the destination side, a large number
> +of normal pages need to be loaded, so it is recommended to add ``-mem-prealloc``
> +parameter on the destination side.
> --
> 2.39.3
>
--
Peter Xu
> -----Original Message----- > From: Peter Xu <peterx@redhat.com> > Sent: Wednesday, March 27, 2024 1:58 AM > To: Liu, Yuan1 <yuan1.liu@intel.com> > Cc: farosas@suse.de; qemu-devel@nongnu.org; hao.xiang@bytedance.com; > bryan.zhang@bytedance.com; Zou, Nanhai <nanhai.zou@intel.com> > Subject: Re: [PATCH v5 1/7] docs/migration: add qpl compression feature > > On Wed, Mar 20, 2024 at 12:45:21AM +0800, Yuan Liu wrote: > > add Intel Query Processing Library (QPL) compression method > > introduction > > > > Signed-off-by: Yuan Liu <yuan1.liu@intel.com> > > Reviewed-by: Nanhai Zou <nanhai.zou@intel.com> > > --- > > docs/devel/migration/features.rst | 1 + > > docs/devel/migration/qpl-compression.rst | 231 +++++++++++++++++++++++ > > 2 files changed, 232 insertions(+) > > create mode 100644 docs/devel/migration/qpl-compression.rst > > > > diff --git a/docs/devel/migration/features.rst > b/docs/devel/migration/features.rst > > index d5ca7b86d5..bc98b65075 100644 > > --- a/docs/devel/migration/features.rst > > +++ b/docs/devel/migration/features.rst > > @@ -12,3 +12,4 @@ Migration has plenty of features to support different > use cases. > > virtio > > mapped-ram > > CPR > > + qpl-compression > > diff --git a/docs/devel/migration/qpl-compression.rst > b/docs/devel/migration/qpl-compression.rst > > new file mode 100644 > > index 0000000000..42c7969d30 > > --- /dev/null > > +++ b/docs/devel/migration/qpl-compression.rst > > @@ -0,0 +1,231 @@ > > +=============== > > +QPL Compression > > +=============== > > +The Intel Query Processing Library (Intel ``QPL``) is an open-source > library to > > +provide compression and decompression features and it is based on > deflate > > +compression algorithm (RFC 1951). > > + > > +The ``QPL`` compression relies on Intel In-Memory Analytics > Accelerator(``IAA``) > > +and Shared Virtual Memory(``SVM``) technology, they are new features > supported > > +from Intel 4th Gen Intel Xeon Scalable processors, codenamed Sapphire > Rapids > > +processor(``SPR``). > > + > > +For more ``QPL`` introduction, please refer to: > > + > > > +https://intel.github.io/qpl/documentation/introduction_docs/introduction. > html > > There're a bunch of links in this page, please consider switching all of > them to use the link formats of .rST: > > Please refer to `QPL introduction page <https://...>`_. Sure, thanks for the suggestion > > + > > +QPL Compression Framework > > +========================= > > + > > +:: > > + > > + +----------------+ +------------------+ > > + | MultiFD Service| |accel-config tool | > > + +-------+--------+ +--------+---------+ > > + | | > > + | | > > + +-------+--------+ | Setup IAA > > + | QPL library | | Resources > > + +-------+---+----+ | > > + | | | > > + | +-------------+-------+ > > + | Open IAA | > > + | Devices +-----+-----+ > > + | |idxd driver| > > + | +-----+-----+ > > + | | > > + | | > > + | +-----+-----+ > > + +-----------+IAA Devices| > > + Submit jobs +-----------+ > > + via enqcmd > > + > > + > > +Intel In-Memory Analytics Accelerator (Intel IAA) Introduction > > +================================================================ > > + > > +Intel ``IAA`` is an accelerator that has been designed to help benefit > > +in-memory databases and analytic workloads. There are three main areas > > +that Intel ``IAA`` can assist with analytics primitives (scan, filter, > etc.), > > +sparse data compression and memory tiering. > > + > > +``IAA`` Manual Documentation: > > + > > +https://www.intel.com/content/www/us/en/content-details/721858/intel- > in-memory-analytics-accelerator-architecture-specification > > + > > +IAA Device Enabling > > +------------------- > > + > > +- Enabling ``IAA`` devices for platform configuration, please refer to: > > + > > +https://www.intel.com/content/www/us/en/content-details/780887/intel- > in-memory-analytics-accelerator-intel-iaa.html > > + > > +- ``IAA`` device driver is ``Intel Data Accelerator Driver (idxd)``, it > is > > + recommended that the minimum version of Linux kernel is 5.18. > > + > > +- Add ``"intel_iommu=on,sm_on"`` parameter to kernel command line > > + for ``SVM`` feature enabling. > > + > > +Here is an easy way to verify ``IAA`` device driver and ``SVM``, refer > to: > > + > > +https://github.com/intel/idxd-config/tree/stable/test > > + > > +IAA Device Management > > +--------------------- > > + > > +The number of ``IAA`` devices will vary depending on the Xeon product > model. > > +On a ``SPR`` server, there can be a maximum of 8 ``IAA`` devices, with > up to > > +4 devices per socket. > > + > > +By default, all ``IAA`` devices are disabled and need to be configured > and > > +enabled by users manually. > > + > > +Check the number of devices through the following command > > + > > +.. code-block:: shell > > + > > + # lspci -d 8086:0cfe > > + # 6a:02.0 System peripheral: Intel Corporation Device 0cfe > > + # 6f:02.0 System peripheral: Intel Corporation Device 0cfe > > + # 74:02.0 System peripheral: Intel Corporation Device 0cfe > > + # 79:02.0 System peripheral: Intel Corporation Device 0cfe > > + # e7:02.0 System peripheral: Intel Corporation Device 0cfe > > + # ec:02.0 System peripheral: Intel Corporation Device 0cfe > > + # f1:02.0 System peripheral: Intel Corporation Device 0cfe > > + # f6:02.0 System peripheral: Intel Corporation Device 0cfe > > + > > +IAA Device Configuration > > +------------------------ > > + > > +The ``accel-config`` tool is used to enable ``IAA`` devices and > configure > > +``IAA`` hardware resources(work queues and engines). One ``IAA`` device > > +has 8 work queues and 8 processing engines, multiple engines can be > assigned > > +to a work queue via ``group`` attribute. > > + > > +One example of configuring and enabling an ``IAA`` device. > > + > > +.. code-block:: shell > > + > > + # accel-config config-engine iax1/engine1.0 -g 0 > > + # accel-config config-engine iax1/engine1.1 -g 0 > > + # accel-config config-engine iax1/engine1.2 -g 0 > > + # accel-config config-engine iax1/engine1.3 -g 0 > > + # accel-config config-engine iax1/engine1.4 -g 0 > > + # accel-config config-engine iax1/engine1.5 -g 0 > > + # accel-config config-engine iax1/engine1.6 -g 0 > > + # accel-config config-engine iax1/engine1.7 -g 0 > > + # accel-config config-wq iax1/wq1.0 -g 0 -s 128 -p 10 -b 1 -t 128 -m > shared -y user -n app1 -d user > > + # accel-config enable-device iax1 > > + # accel-config enable-wq iax1/wq1.0 > > + > > +.. note:: > > + IAX is an early name for IAA > > + > > +- The ``IAA`` device index is 1, use ``ls -lh > /sys/bus/dsa/devices/iax*`` > > + command to query the ``IAA`` device index. > > + > > +- 8 engines and 1 work queue are configured in group 0, so all > compression jobs > > + submitted to this work queue can be processed by all engines at the > same time. > > + > > +- Set work queue attributes including the work mode, work queue size > and so on. > > + > > +- Enable the ``IAA1`` device and work queue 1.0 > > + > > +.. note:: > > + Set work queue mode to shared mode, since ``QPL`` library only > supports > > + shared mode > > + > > +For more detailed configuration, please refer to: > > + > > +https://github.com/intel/idxd-config/tree/stable/Documentation/accfg > > + > > +IAA Resources Allocation For Migration > > +-------------------------------------- > > + > > +There is no ``IAA`` resource configuration parameters for migration and > > +``accel-config`` tool configuration cannot directly specify the ``IAA`` > > +resources used for migration. > > + > > +``QPL`` will use all work queues that are enabled and set to shared > mode, > > +and use all engines assigned to the work queues with shared mode. > > + > > +By default, ``QPL`` will only use the local ``IAA`` device for > compression > > +job processing. The local ``IAA`` device means that the CPU of the job > > +submission and the ``IAA`` device are on the same socket, so one CPU > > +can submit the jobs to up to 4 ``IAA`` devices. > > + > > +Shared Virtual Memory(SVM) Introduction > > +======================================= > > + > > +An ability for an accelerator I/O device to operate in the same virtual > > +memory space of applications on host processors. It also implies the > > +ability to operate from pageable memory, avoiding functional > requirements > > +to pin memory for DMA operations. > > + > > +When using ``SVM`` technology, users do not need to reserve memory for > the > > +``IAA`` device and perform pin memory operation. The ``IAA`` device can > > +directly access data using the virtual address of the process. > > + > > +For more ``SVM`` technology, please refer to: > > + > > +https://docs.kernel.org/next/x86/sva.html > > + > > + > > +How To Use QPL Compression In Migration > > +======================================= > > + > > +1 - Installation of ``accel-config`` tool and ``QPL`` library > > We can drop "1 " and stick with: > > - item1 > - item1.1 > ... > - item2 > Yes, it is better. > > + > > + - Install ``accel-config`` tool from https://github.com/intel/idxd- > config > > + - Install ``QPL`` library from https://github.com/intel/qpl > > + > > +2 - Configure and enable ``IAA`` devices and work queues via ``accel- > config`` > > + > > +3 - Build ``Qemu`` with ``--enable-qpl`` parameter > > + > > + E.g. configure --target-list=x86_64-softmmu --enable-kvm ``--enable- > qpl`` > > + > > +4 - Start VMs with ``sudo`` command or ``root`` permission > > + > > + Use the ``sudo`` command or ``root`` privilege to start the source > and > > + destination virtual machines, since migration service needs > permission > > + to access ``IAA`` hardware resources. > > + > > +5 - Enable ``QPL`` compression during migration > > + > > + Set ``migrate_set_parameter multifd-compression qpl`` when migrating, > the > > + ``QPL`` compression does not support configuring the compression > level, it > > + only supports one compression level. > > + > > +The Difference Between QPL And ZLIB > > +=================================== > > + > > +Although both ``QPL`` and ``ZLIB`` are based on the deflate compression > > +algorithm, and ``QPL`` can support the header and tail of ``ZLIB``, > ``QPL`` > > +is still not fully compatible with the ``ZLIB`` compression in the > migration. > > + > > +``QPL`` only supports 4K history buffer, and ``ZLIB`` is 32K by > default. The > > +``ZLIB`` compressed data that ``QPL`` may not decompress correctly and > > +vice versa. > > + > > +``QPL`` does not support the ``Z_SYNC_FLUSH`` operation in ``ZLIB`` > streaming > > +compression, current ``ZLIB`` implementation uses ``Z_SYNC_FLUSH``, so > each > > +``multifd`` thread has a ``ZLIB`` streaming context, and all page > compression > > +and decompression are based on this stream. ``QPL`` cannot decompress > such data > > +and vice versa. > > + > > +The introduction for ``Z_SYNC_FLUSH``, please refer to: > > + > > +https://www.zlib.net/manual.html > > + > > +The Best Practices > > +================== > > + > > +When the virtual machine's pages are not populated and the ``IAA`` > device is > > +used, I/O page faults occur, which can impact performance due to a > large number > > +of flush ``IOTLB`` operations. > > AFAIU the IOTLB issue is not expected and can be fixed, while per our > discussion in the other thread, the DMA fault latency cannot. > > I think we can mention the possibility of IOTLB flush issue but that > shouldn't be the major cause of such suggestion. Again, I think it'll be > great to mention how slow a DMA page fault can be, it can be compared in > migration performance difference, or just describe an average DMA page > fault latency, v.s. a processor fault. That might explain why we suggest > prefault the pages (comparing to a generic setup where the pages are > always > faulted by processors). Get it, I will explain this issue in depth and add some comparative data, including I/O page fault Vs. process fault performance comparison and the performance impact of I/O page fault on the entire live migration. Help developers and users better understand why -mem- prealloc needs to be added currently. > > + > > +Since the normal pages on the source side are all populated, ``IOTLB`` > caused > > +by I/O page fault will not occur. On the destination side, a large > number > > +of normal pages need to be loaded, so it is recommended to add ``-mem- > prealloc`` > > +parameter on the destination side. > > -- > > 2.39.3 > > > > -- > Peter Xu
© 2016 - 2026 Red Hat, Inc.