From nobody Mon May 13 03:14:43 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass header.i=@intel.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=intel.com ARC-Seal: i=1; a=rsa-sha256; t=1710923698; cv=none; d=zohomail.com; s=zohoarc; b=KKjJ6eM1C8iqQjNUjqOQCMoShS77DItaxBFL7nSoLY7VJTx0Bw1T7oe73ZK9AJj3mvIVQmxeBYQ3tY/knif5Jai8aM9HNNKbz2G+LQqhc18TvuTHey1DLaUr6Re015bvJbsHf07P9w3AYHThQbtYKjPf+O90XgXC0UyQU11CDbY= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1710923698; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=vt/NlI4Sc0r2U9uF2XXhbTyLPE2Nw+M0CbRSYhO5TC0=; b=IRqVRqEdoaUPfSFY08cAmwqFJ015ldo9pYizInyEit14oS3790sKHueF45VKTINQLSEi5aczeKwYOaBF94uf9h4ZJBI6N+xbD7c37Ek3oMwzdm62PaOb+rxxp++Qwg7aRtQuo0tea4padbmESSk71ZSvS+RG696fwUZdJgpST50= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass header.i=@intel.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1710923698496977.0362011260203; Wed, 20 Mar 2024 01:34:58 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rmrNw-0004WZ-L8; Wed, 20 Mar 2024 04:33:20 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rmrNp-0004VO-Qj for qemu-devel@nongnu.org; Wed, 20 Mar 2024 04:33:14 -0400 Received: from mgamail.intel.com ([198.175.65.18]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rmrNk-0005U3-HP for qemu-devel@nongnu.org; Wed, 20 Mar 2024 04:33:13 -0400 Received: from orviesa010.jf.intel.com ([10.64.159.150]) by orvoesa110.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 20 Mar 2024 01:33:03 -0700 Received: from sae-gw02.sh.intel.com (HELO localhost) ([10.239.45.110]) by orviesa010.jf.intel.com with ESMTP; 20 Mar 2024 01:33:00 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1710923589; x=1742459589; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=jHoiX8d2IXGLQEQeJbBznT8raIIAGzjHe1GxhmOwODU=; b=JnzXLBnz5YfIH6Glow9UGMO3nL/oigIRvuGt8XJ5hqIT9tPpyNRRVpKV AJei3zYeKqJaMmVXhRbGyEArCT308KU2kUlVD/IivRuMoVNU0MDdxBg/n yVy6TDHJjHT4dgI0wj7BJBvA6IT++khWrTqtC4et5D5yh7XkpB7rVCoIM cAwUm4Q4515zq8KW4bpqqZkRso2GWO5AwZ+HdwBXCt+8rClADfV7bGlHC VlvnuRmk2qczJ7xSitjs0xgNpVyKMUIHHADW9a9kkNP+pz9wcJABsNKIC 41iAz4Ld+TJwmldmgtcZDyArykJXoLtfv2qRjeSlwll6IOlNZ3FPzdBqz w==; X-IronPort-AV: E=McAfee;i="6600,9927,11018"; a="5952907" X-IronPort-AV: E=Sophos;i="6.07,139,1708416000"; d="scan'208";a="5952907" X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,139,1708416000"; d="scan'208";a="13986696" From: Yuan Liu To: peterx@redhat.com, farosas@suse.de Cc: qemu-devel@nongnu.org, hao.xiang@bytedance.com, bryan.zhang@bytedance.com, yuan1.liu@intel.com, nanhai.zou@intel.com Subject: [PATCH v5 1/7] docs/migration: add qpl compression feature Date: Wed, 20 Mar 2024 00:45:21 +0800 Message-Id: <20240319164527.1873891-2-yuan1.liu@intel.com> X-Mailer: git-send-email 2.39.3 In-Reply-To: <20240319164527.1873891-1-yuan1.liu@intel.com> References: <20240319164527.1873891-1-yuan1.liu@intel.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=198.175.65.18; envelope-from=yuan1.liu@intel.com; helo=mgamail.intel.com X-Spam_score_int: -14 X-Spam_score: -1.5 X-Spam_bar: - X-Spam_report: (-1.5 / 5.0 requ) BAYES_00=-1.9, DATE_IN_PAST_12_24=1.049, DKIMWL_WL_HIGH=-0.422, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @intel.com) X-ZM-MESSAGEID: 1710923700732100007 Content-Type: text/plain; charset="utf-8" add Intel Query Processing Library (QPL) compression method introduction Signed-off-by: Yuan Liu Reviewed-by: Nanhai Zou --- docs/devel/migration/features.rst | 1 + docs/devel/migration/qpl-compression.rst | 231 +++++++++++++++++++++++ 2 files changed, 232 insertions(+) create mode 100644 docs/devel/migration/qpl-compression.rst diff --git a/docs/devel/migration/features.rst b/docs/devel/migration/featu= res.rst index d5ca7b86d5..bc98b65075 100644 --- a/docs/devel/migration/features.rst +++ b/docs/devel/migration/features.rst @@ -12,3 +12,4 @@ Migration has plenty of features to support different use= cases. virtio mapped-ram CPR + qpl-compression diff --git a/docs/devel/migration/qpl-compression.rst b/docs/devel/migratio= n/qpl-compression.rst new file mode 100644 index 0000000000..42c7969d30 --- /dev/null +++ b/docs/devel/migration/qpl-compression.rst @@ -0,0 +1,231 @@ +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +QPL Compression +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +The Intel Query Processing Library (Intel ``QPL``) is an open-source libra= ry to +provide compression and decompression features and it is based on deflate +compression algorithm (RFC 1951). + +The ``QPL`` compression relies on Intel In-Memory Analytics Accelerator(``= IAA``) +and Shared Virtual Memory(``SVM``) technology, they are new features suppo= rted +from Intel 4th Gen Intel Xeon Scalable processors, codenamed Sapphire Rapi= ds +processor(``SPR``). + +For more ``QPL`` introduction, please refer to: + +https://intel.github.io/qpl/documentation/introduction_docs/introduction.h= tml + +QPL Compression Framework +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D + +:: + + +----------------+ +------------------+ + | MultiFD Service| |accel-config tool | + +-------+--------+ +--------+---------+ + | | + | | + +-------+--------+ | Setup IAA + | QPL library | | Resources + +-------+---+----+ | + | | | + | +-------------+-------+ + | Open IAA | + | Devices +-----+-----+ + | |idxd driver| + | +-----+-----+ + | | + | | + | +-----+-----+ + +-----------+IAA Devices| + Submit jobs +-----------+ + via enqcmd + + +Intel In-Memory Analytics Accelerator (Intel IAA) Introduction +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +Intel ``IAA`` is an accelerator that has been designed to help benefit +in-memory databases and analytic workloads. There are three main areas +that Intel ``IAA`` can assist with analytics primitives (scan, filter, etc= .), +sparse data compression and memory tiering. + +``IAA`` Manual Documentation: + +https://www.intel.com/content/www/us/en/content-details/721858/intel-in-me= mory-analytics-accelerator-architecture-specification + +IAA Device Enabling +------------------- + +- Enabling ``IAA`` devices for platform configuration, please refer to: + +https://www.intel.com/content/www/us/en/content-details/780887/intel-in-me= mory-analytics-accelerator-intel-iaa.html + +- ``IAA`` device driver is ``Intel Data Accelerator Driver (idxd)``, it is + recommended that the minimum version of Linux kernel is 5.18. + +- Add ``"intel_iommu=3Don,sm_on"`` parameter to kernel command line + for ``SVM`` feature enabling. + +Here is an easy way to verify ``IAA`` device driver and ``SVM``, refer to: + +https://github.com/intel/idxd-config/tree/stable/test + +IAA Device Management +--------------------- + +The number of ``IAA`` devices will vary depending on the Xeon product mode= l. +On a ``SPR`` server, there can be a maximum of 8 ``IAA`` devices, with up = to +4 devices per socket. + +By default, all ``IAA`` devices are disabled and need to be configured and +enabled by users manually. + +Check the number of devices through the following command + +.. code-block:: shell + + # lspci -d 8086:0cfe + # 6a:02.0 System peripheral: Intel Corporation Device 0cfe + # 6f:02.0 System peripheral: Intel Corporation Device 0cfe + # 74:02.0 System peripheral: Intel Corporation Device 0cfe + # 79:02.0 System peripheral: Intel Corporation Device 0cfe + # e7:02.0 System peripheral: Intel Corporation Device 0cfe + # ec:02.0 System peripheral: Intel Corporation Device 0cfe + # f1:02.0 System peripheral: Intel Corporation Device 0cfe + # f6:02.0 System peripheral: Intel Corporation Device 0cfe + +IAA Device Configuration +------------------------ + +The ``accel-config`` tool is used to enable ``IAA`` devices and configure +``IAA`` hardware resources(work queues and engines). One ``IAA`` device +has 8 work queues and 8 processing engines, multiple engines can be assign= ed +to a work queue via ``group`` attribute. + +One example of configuring and enabling an ``IAA`` device. + +.. code-block:: shell + + # accel-config config-engine iax1/engine1.0 -g 0 + # accel-config config-engine iax1/engine1.1 -g 0 + # accel-config config-engine iax1/engine1.2 -g 0 + # accel-config config-engine iax1/engine1.3 -g 0 + # accel-config config-engine iax1/engine1.4 -g 0 + # accel-config config-engine iax1/engine1.5 -g 0 + # accel-config config-engine iax1/engine1.6 -g 0 + # accel-config config-engine iax1/engine1.7 -g 0 + # accel-config config-wq iax1/wq1.0 -g 0 -s 128 -p 10 -b 1 -t 128 -m sha= red -y user -n app1 -d user + # accel-config enable-device iax1 + # accel-config enable-wq iax1/wq1.0 + +.. note:: + IAX is an early name for IAA + +- The ``IAA`` device index is 1, use ``ls -lh /sys/bus/dsa/devices/iax*`` + command to query the ``IAA`` device index. + +- 8 engines and 1 work queue are configured in group 0, so all compression= jobs + submitted to this work queue can be processed by all engines at the same= time. + +- Set work queue attributes including the work mode, work queue size and s= o on. + +- Enable the ``IAA1`` device and work queue 1.0 + +.. note:: + Set work queue mode to shared mode, since ``QPL`` library only supports + shared mode + +For more detailed configuration, please refer to: + +https://github.com/intel/idxd-config/tree/stable/Documentation/accfg + +IAA Resources Allocation For Migration +-------------------------------------- + +There is no ``IAA`` resource configuration parameters for migration and +``accel-config`` tool configuration cannot directly specify the ``IAA`` +resources used for migration. + +``QPL`` will use all work queues that are enabled and set to shared mode, +and use all engines assigned to the work queues with shared mode. + +By default, ``QPL`` will only use the local ``IAA`` device for compression +job processing. The local ``IAA`` device means that the CPU of the job +submission and the ``IAA`` device are on the same socket, so one CPU +can submit the jobs to up to 4 ``IAA`` devices. + +Shared Virtual Memory(SVM) Introduction +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +An ability for an accelerator I/O device to operate in the same virtual +memory space of applications on host processors. It also implies the +ability to operate from pageable memory, avoiding functional requirements +to pin memory for DMA operations. + +When using ``SVM`` technology, users do not need to reserve memory for the +``IAA`` device and perform pin memory operation. The ``IAA`` device can +directly access data using the virtual address of the process. + +For more ``SVM`` technology, please refer to: + +https://docs.kernel.org/next/x86/sva.html + + +How To Use QPL Compression In Migration +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +1 - Installation of ``accel-config`` tool and ``QPL`` library + + - Install ``accel-config`` tool from https://github.com/intel/idxd-config + - Install ``QPL`` library from https://github.com/intel/qpl + +2 - Configure and enable ``IAA`` devices and work queues via ``accel-confi= g`` + +3 - Build ``Qemu`` with ``--enable-qpl`` parameter + + E.g. configure --target-list=3Dx86_64-softmmu --enable-kvm ``--enable-qp= l`` + +4 - Start VMs with ``sudo`` command or ``root`` permission + + Use the ``sudo`` command or ``root`` privilege to start the source and + destination virtual machines, since migration service needs permission + to access ``IAA`` hardware resources. + +5 - Enable ``QPL`` compression during migration + + Set ``migrate_set_parameter multifd-compression qpl`` when migrating, the + ``QPL`` compression does not support configuring the compression level, = it + only supports one compression level. + +The Difference Between QPL And ZLIB +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +Although both ``QPL`` and ``ZLIB`` are based on the deflate compression +algorithm, and ``QPL`` can support the header and tail of ``ZLIB``, ``QPL`` +is still not fully compatible with the ``ZLIB`` compression in the migrati= on. + +``QPL`` only supports 4K history buffer, and ``ZLIB`` is 32K by default. T= he +``ZLIB`` compressed data that ``QPL`` may not decompress correctly and +vice versa. + +``QPL`` does not support the ``Z_SYNC_FLUSH`` operation in ``ZLIB`` stream= ing +compression, current ``ZLIB`` implementation uses ``Z_SYNC_FLUSH``, so each +``multifd`` thread has a ``ZLIB`` streaming context, and all page compress= ion +and decompression are based on this stream. ``QPL`` cannot decompress such= data +and vice versa. + +The introduction for ``Z_SYNC_FLUSH``, please refer to: + +https://www.zlib.net/manual.html + +The Best Practices +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +When the virtual machine's pages are not populated and the ``IAA`` device = is +used, I/O page faults occur, which can impact performance due to a large n= umber +of flush ``IOTLB`` operations. + +Since the normal pages on the source side are all populated, ``IOTLB`` cau= sed +by I/O page fault will not occur. On the destination side, a large number +of normal pages need to be loaded, so it is recommended to add ``-mem-prea= lloc`` +parameter on the destination side. --=20 2.39.3 From nobody Mon May 13 03:14:43 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass header.i=@intel.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=intel.com ARC-Seal: i=1; a=rsa-sha256; t=1710923657; cv=none; d=zohomail.com; s=zohoarc; b=OnrZWqtvOlx0m/DHXpLl0MGZDlRuNFNb5aqzPxAIU1c/Sgpq0+D9LzrTP6PDMtuX0pG97PQ8MF5vUGxTPhAYhPSdOzgGCzMTBZg9VY2c7p+sI5sQbMNI1L5saeetPiBhlTgvIPhSYVmvduMM36pnDD++Z2roLn11JuWBkK3O218= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1710923657; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=5xiU90KSgBYucJABDyQQpfyC0TR/wauKfigtkPR5OEM=; b=T5AU/SjiVdjp3io+9l8eBQSUrYg2ggSlYL+mIHYeWCloA5UkV9znVHlHztfnqnOR2ydYdSrH8qVoACfVTttX7QzfQAUUldNXQlB5+WmaVnOvPnBDaDLuExp7e2pxoe2dqwdBeCTV4M5uTMjqPVXLQE5BMTRZy1YJa2eE1EJ7Tyk= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass header.i=@intel.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1710923657728301.6022157770043; Wed, 20 Mar 2024 01:34:17 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rmrNz-0004Ww-0U; Wed, 20 Mar 2024 04:33:23 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rmrNr-0004VY-1S for qemu-devel@nongnu.org; Wed, 20 Mar 2024 04:33:15 -0400 Received: from mgamail.intel.com ([198.175.65.18]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rmrNn-0005UI-Bm for qemu-devel@nongnu.org; Wed, 20 Mar 2024 04:33:14 -0400 Received: from orviesa010.jf.intel.com ([10.64.159.150]) by orvoesa110.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 20 Mar 2024 01:33:06 -0700 Received: from sae-gw02.sh.intel.com (HELO localhost) ([10.239.45.110]) by orviesa010.jf.intel.com with ESMTP; 20 Mar 2024 01:33:04 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1710923592; x=1742459592; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=6D6zFZg6n7LP7yFNxtj2KPov5L9k6CTzYGh4pC8iT8s=; b=bNgFWiOrc1SQOk+MKxv73FVoa7FE3K5wDZNtFNHTBjaEdvRLuU122vKr 1ol5jsZW03YYpq/rdhjXVi4p5XE6HXZDj2PNEKLxmT5XOxN7OpdGrWgfP 0fmxsK2QT5kdTzc8P7MKpBVOJ57RSP2rVfqIa1tNk7hNlz/972OVF8G/8 fKkCQLg95suEVZMdpEAKV3UEAsDjxQUcDUyOKL3uvYGZrzIF/7d901YVz rO3G3y1CldOnIO89gVz76oSgAb7zc+wb487BtOAWCAM+MSqSW9qs+V8Ez fdkSybkJhUMKnayUfcz+S6cbyffFoJ8UvJoVAxHxmn8pDgeiXSrf4y4Em w==; X-IronPort-AV: E=McAfee;i="6600,9927,11018"; a="5952917" X-IronPort-AV: E=Sophos;i="6.07,139,1708416000"; d="scan'208";a="5952917" X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,139,1708416000"; d="scan'208";a="13986709" From: Yuan Liu To: peterx@redhat.com, farosas@suse.de Cc: qemu-devel@nongnu.org, hao.xiang@bytedance.com, bryan.zhang@bytedance.com, yuan1.liu@intel.com, nanhai.zou@intel.com Subject: [PATCH v5 2/7] migration/multifd: put IOV initialization into compression method Date: Wed, 20 Mar 2024 00:45:22 +0800 Message-Id: <20240319164527.1873891-3-yuan1.liu@intel.com> X-Mailer: git-send-email 2.39.3 In-Reply-To: <20240319164527.1873891-1-yuan1.liu@intel.com> References: <20240319164527.1873891-1-yuan1.liu@intel.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=198.175.65.18; envelope-from=yuan1.liu@intel.com; helo=mgamail.intel.com X-Spam_score_int: -14 X-Spam_score: -1.5 X-Spam_bar: - X-Spam_report: (-1.5 / 5.0 requ) BAYES_00=-1.9, DATE_IN_PAST_12_24=1.049, DKIMWL_WL_HIGH=-0.422, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @intel.com) X-ZM-MESSAGEID: 1710923658725100001 Content-Type: text/plain; charset="utf-8" Different compression methods may require different numbers of IOVs. Based on streaming compression of zlib and zstd, all pages will be compressed to a data block, so two IOVs are needed for packet header and compressed data block. Signed-off-by: Yuan Liu Reviewed-by: Nanhai Zou --- migration/multifd-zlib.c | 4 ++++ migration/multifd-zstd.c | 6 +++++- migration/multifd.c | 8 +++++--- 3 files changed, 14 insertions(+), 4 deletions(-) diff --git a/migration/multifd-zlib.c b/migration/multifd-zlib.c index 99821cd4d5..8095ef8e28 100644 --- a/migration/multifd-zlib.c +++ b/migration/multifd-zlib.c @@ -70,6 +70,10 @@ static int zlib_send_setup(MultiFDSendParams *p, Error *= *errp) goto err_free_zbuff; } p->compress_data =3D z; + + assert(p->iov =3D=3D NULL); + /* For packet header and zlib streaming compression block */ + p->iov =3D g_new0(struct iovec, 2); return 0; =20 err_free_zbuff: diff --git a/migration/multifd-zstd.c b/migration/multifd-zstd.c index 02112255ad..9c9217794e 100644 --- a/migration/multifd-zstd.c +++ b/migration/multifd-zstd.c @@ -52,7 +52,6 @@ static int zstd_send_setup(MultiFDSendParams *p, Error **= errp) struct zstd_data *z =3D g_new0(struct zstd_data, 1); int res; =20 - p->compress_data =3D z; z->zcs =3D ZSTD_createCStream(); if (!z->zcs) { g_free(z); @@ -77,6 +76,11 @@ static int zstd_send_setup(MultiFDSendParams *p, Error *= *errp) error_setg(errp, "multifd %u: out of memory for zbuff", p->id); return -1; } + p->compress_data =3D z; + + assert(p->iov =3D=3D NULL); + /* For packet header and zstd streaming compression block */ + p->iov =3D g_new0(struct iovec, 2); return 0; } =20 diff --git a/migration/multifd.c b/migration/multifd.c index 0179422f6d..5155e02ae3 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -1181,9 +1181,11 @@ bool multifd_send_setup(void) p->packet =3D g_malloc0(p->packet_len); p->packet->magic =3D cpu_to_be32(MULTIFD_MAGIC); p->packet->version =3D cpu_to_be32(MULTIFD_VERSION); - - /* We need one extra place for the packet header */ - p->iov =3D g_new0(struct iovec, page_count + 1); + /* IOVs are initialized in send_setup of compression method */ + if (!migrate_multifd_compression()) { + /* We need one extra place for the packet header */ + p->iov =3D g_new0(struct iovec, page_count + 1); + } } else { p->iov =3D g_new0(struct iovec, page_count); } --=20 2.39.3 From nobody Mon May 13 03:14:43 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass header.i=@intel.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=intel.com ARC-Seal: i=1; a=rsa-sha256; t=1710923651; cv=none; d=zohomail.com; s=zohoarc; b=Vsv927rPqjfWYc0YnjmnOaC3RtAxlbuD/lTQk6iYdrcqNGBuCmmkwA7uJO5ZvI75xVn3GJRWc0BgCexeoi1F5whDZvOkIw1Bg7UZLZAf2TmHM9xh0amNZ4WGJP6QODFRINkeRpn+yu41KDSO/cZBTJUlRNa8V9ipWtSWZveW81M= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1710923651; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=SdBjytB5gDCc5WayI6teC+tZwbXvCxzjJrQ/zmSmtlo=; b=EzXLlxouiomuh14eQWUnVwfQLKgnU7ud71k794vNLFxB0S7mKGNssTHVb0FR4mWzyanhCp4Uv5xI0qC9AspxYsofTlB1aGqh0jvKaAlB9sGESqP3RsnviupiBsE2XsKJGCFsB9d9DDgG9XseSRGrT50KWX5lVjg00vxIg6XIw74= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass header.i=@intel.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1710923651748402.38520499659296; Wed, 20 Mar 2024 01:34:11 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rmrNz-0004XG-Gp; Wed, 20 Mar 2024 04:33:24 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rmrNs-0004W6-J5 for qemu-devel@nongnu.org; Wed, 20 Mar 2024 04:33:18 -0400 Received: from mgamail.intel.com ([198.175.65.18]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rmrNq-0005U3-4n for qemu-devel@nongnu.org; Wed, 20 Mar 2024 04:33:16 -0400 Received: from orviesa010.jf.intel.com ([10.64.159.150]) by orvoesa110.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 20 Mar 2024 01:33:10 -0700 Received: from sae-gw02.sh.intel.com (HELO localhost) ([10.239.45.110]) by orviesa010.jf.intel.com with ESMTP; 20 Mar 2024 01:33:07 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1710923595; x=1742459595; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=CIWnqF9EoTF2zMvqZ/+tNdqB3NlIMG2yD3I2MMICmII=; b=PxxUPcM30cdkSDuWfzDPRl6CKRXYB7TxMBjSba1yfS+9Wa6MQ3dVDDgv vBedKcd6Hgtu/7+kMqHoE8OjA529N7BwmWFyR60q9IvjyGtnh6PBxGm2O a/WRvHpyogAOMq8zPcPvyTWI3VwjjexzFw/uxxbVkFSlkWvH3XBE1vgd/ /VN2ClIRjvTzUCm0ot6CnpTfsVvjlbifv3VCxV/fxfgKQ4ZcXViuSayED g1frDTzyqYcoaJBWsbkW24AOAQgi0BOuK6yl1cAJn3ty0gujxRUIHK3im Y6iHB0xi+AlvdBSj79eX/HqqEfb4vuh/1BD/AG09jTqfOo6FzrTXhMhfQ A==; X-IronPort-AV: E=McAfee;i="6600,9927,11018"; a="5952927" X-IronPort-AV: E=Sophos;i="6.07,139,1708416000"; d="scan'208";a="5952927" X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,139,1708416000"; d="scan'208";a="13986718" From: Yuan Liu To: peterx@redhat.com, farosas@suse.de Cc: qemu-devel@nongnu.org, hao.xiang@bytedance.com, bryan.zhang@bytedance.com, yuan1.liu@intel.com, nanhai.zou@intel.com Subject: [PATCH v5 3/7] configure: add --enable-qpl build option Date: Wed, 20 Mar 2024 00:45:23 +0800 Message-Id: <20240319164527.1873891-4-yuan1.liu@intel.com> X-Mailer: git-send-email 2.39.3 In-Reply-To: <20240319164527.1873891-1-yuan1.liu@intel.com> References: <20240319164527.1873891-1-yuan1.liu@intel.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=198.175.65.18; envelope-from=yuan1.liu@intel.com; helo=mgamail.intel.com X-Spam_score_int: -14 X-Spam_score: -1.5 X-Spam_bar: - X-Spam_report: (-1.5 / 5.0 requ) BAYES_00=-1.9, DATE_IN_PAST_12_24=1.049, DKIMWL_WL_HIGH=-0.422, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @intel.com) X-ZM-MESSAGEID: 1710923653032100001 Content-Type: text/plain; charset="utf-8" add --enable-qpl and --disable-qpl options to enable and disable the QPL compression method for multifd migration. the Query Processing Library (QPL) is an open-source library that supports data compression and decompression features. The QPL compression is based on the deflate compression algorithm and use Intel In-Memory Analytics Accelerator(IAA) hardware for compression and decompression acceleration. Please refer to the following for more information about QPL https://intel.github.io/qpl/documentation/introduction_docs/introduction.ht= ml Signed-off-by: Yuan Liu Reviewed-by: Nanhai Zou --- meson.build | 16 ++++++++++++++++ meson_options.txt | 2 ++ scripts/meson-buildoptions.sh | 3 +++ 3 files changed, 21 insertions(+) diff --git a/meson.build b/meson.build index b375248a76..bee7dcd53b 100644 --- a/meson.build +++ b/meson.build @@ -1200,6 +1200,20 @@ if not get_option('zstd').auto() or have_block required: get_option('zstd'), method: 'pkg-config') endif +qpl =3D not_found +if not get_option('qpl').auto() + libqpl =3D cc.find_library('qpl', required: false) + if not libqpl.found() + error('libqpl not found, please install it from ' + + 'https://intel.github.io/qpl/documentation/get_started_docs/installati= on.html') + endif + libaccel =3D dependency('libaccel-config', version: '>=3D4.0.0', + required: true, + method: 'pkg-config') + qpl =3D declare_dependency(dependencies: [libqpl, libaccel, + cc.find_library('dl', required: get_option('qpl'))], + link_args: ['-lstdc++']) +endif virgl =3D not_found =20 have_vhost_user_gpu =3D have_tools and host_os =3D=3D 'linux' and pixman.f= ound() @@ -2305,6 +2319,7 @@ config_host_data.set('CONFIG_MALLOC_TRIM', has_malloc= _trim) config_host_data.set('CONFIG_STATX', has_statx) config_host_data.set('CONFIG_STATX_MNT_ID', has_statx_mnt_id) config_host_data.set('CONFIG_ZSTD', zstd.found()) +config_host_data.set('CONFIG_QPL', qpl.found()) config_host_data.set('CONFIG_FUSE', fuse.found()) config_host_data.set('CONFIG_FUSE_LSEEK', fuse_lseek.found()) config_host_data.set('CONFIG_SPICE_PROTOCOL', spice_protocol.found()) @@ -4462,6 +4477,7 @@ summary_info +=3D {'snappy support': snappy} summary_info +=3D {'bzip2 support': libbzip2} summary_info +=3D {'lzfse support': liblzfse} summary_info +=3D {'zstd support': zstd} +summary_info +=3D {'Query Processing Library support': qpl} summary_info +=3D {'NUMA host support': numa} summary_info +=3D {'capstone': capstone} summary_info +=3D {'libpmem support': libpmem} diff --git a/meson_options.txt b/meson_options.txt index 0a99a059ec..06cd675572 100644 --- a/meson_options.txt +++ b/meson_options.txt @@ -259,6 +259,8 @@ option('xkbcommon', type : 'feature', value : 'auto', description: 'xkbcommon support') option('zstd', type : 'feature', value : 'auto', description: 'zstd compression support') +option('qpl', type : 'feature', value : 'auto', + description: 'Query Processing Library support') option('fuse', type: 'feature', value: 'auto', description: 'FUSE block device export') option('fuse_lseek', type : 'feature', value : 'auto', diff --git a/scripts/meson-buildoptions.sh b/scripts/meson-buildoptions.sh index 680fa3f581..784f74fde9 100644 --- a/scripts/meson-buildoptions.sh +++ b/scripts/meson-buildoptions.sh @@ -222,6 +222,7 @@ meson_options_help() { printf "%s\n" ' Xen PCI passthrough support' printf "%s\n" ' xkbcommon xkbcommon support' printf "%s\n" ' zstd zstd compression support' + printf "%s\n" ' qpl Query Processing Library support' } _meson_option_parse() { case $1 in @@ -562,6 +563,8 @@ _meson_option_parse() { --disable-xkbcommon) printf "%s" -Dxkbcommon=3Ddisabled ;; --enable-zstd) printf "%s" -Dzstd=3Denabled ;; --disable-zstd) printf "%s" -Dzstd=3Ddisabled ;; + --enable-qpl) printf "%s" -Dqpl=3Denabled ;; + --disable-qpl) printf "%s" -Dqpl=3Ddisabled ;; *) return 1 ;; esac } --=20 2.39.3 From nobody Mon May 13 03:14:43 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass header.i=@intel.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=intel.com ARC-Seal: i=1; a=rsa-sha256; t=1710923651; cv=none; d=zohomail.com; s=zohoarc; b=DS9kPJlJ+j37Ll8Wbkr8IVytKMBxQoFAVDEddfEznOJDwKOgJKp6tHoniBO1vnKldHV3JbvDKSqPmOvf0ECrKy1ozmXa9teQ37i+S/APQQvX5iu+M0JlzzQU3o2RSkOD/96ExPPe3B03RRQGutaGuyxCET3SA5WeKwJAKvLl76o= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1710923651; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=njodWOMeAP2ki7SOuUrkFXu7xfC/kmi4ZRWLwpbAbFk=; b=WfN4cAD0fu6OQXOGVeuURijbF5D8zjHieGf2VjKxPrQs6ZUyVkTZQdgHjVoqwUUG7gu12jrWxfiPxUimQpjPwej+Yaevn4wb78QiQvaQ4O3svuETmJqx3QgvtIYT2G9d2imTY/bZFiyESWkzguZPGLeXyZOTlyQjc7lzt0iiM2w= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass header.i=@intel.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1710923651739716.1040530640907; Wed, 20 Mar 2024 01:34:11 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rmrO7-0004Xh-DU; Wed, 20 Mar 2024 04:33:32 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rmrNt-0004WA-0R for qemu-devel@nongnu.org; Wed, 20 Mar 2024 04:33:18 -0400 Received: from mgamail.intel.com ([198.175.65.18]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rmrNr-0005UI-C3 for qemu-devel@nongnu.org; Wed, 20 Mar 2024 04:33:16 -0400 Received: from orviesa010.jf.intel.com ([10.64.159.150]) by orvoesa110.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 20 Mar 2024 01:33:13 -0700 Received: from sae-gw02.sh.intel.com (HELO localhost) ([10.239.45.110]) by orviesa010.jf.intel.com with ESMTP; 20 Mar 2024 01:33:10 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1710923596; x=1742459596; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=mz9Bf/tG16d1JBIuhsfID5eXSf48Pqg0iXFjNioFLYQ=; b=XSsF0yMzWgEIaZG4BABfuAtEtiqBUrOJKjLSV4rpoTloCg6KCPpiDIiR wZ9EDAYm9C5ZD9JuG6W/bkC2Iq1Q2wfvpaRso8EUMP+Px2cgfdGHMreyR JWnt4m8djUVz7imofCTXomsNbv8RLk5gKXEl3racsO16DMwNnAh7CM0gZ 9wHiI5wIdiGPg8MZE9L7oNIPh95QP+qam0w3+rgcvvs6j44+DdAPjK9YP oCD3RXoJak3VDUEtaHwyFrWUlxEm9a9P6nGZ/zAlJCKFtEl6anzXK2hL/ inpYQ3M9BZfVqa1rhyL/dAkeuKnyMGE3gwjpvu1LLrImuz9mL0U+iVHTj Q==; X-IronPort-AV: E=McAfee;i="6600,9927,11018"; a="5952939" X-IronPort-AV: E=Sophos;i="6.07,139,1708416000"; d="scan'208";a="5952939" X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,139,1708416000"; d="scan'208";a="13986727" From: Yuan Liu To: peterx@redhat.com, farosas@suse.de Cc: qemu-devel@nongnu.org, hao.xiang@bytedance.com, bryan.zhang@bytedance.com, yuan1.liu@intel.com, nanhai.zou@intel.com Subject: [PATCH v5 4/7] migration/multifd: add qpl compression method Date: Wed, 20 Mar 2024 00:45:24 +0800 Message-Id: <20240319164527.1873891-5-yuan1.liu@intel.com> X-Mailer: git-send-email 2.39.3 In-Reply-To: <20240319164527.1873891-1-yuan1.liu@intel.com> References: <20240319164527.1873891-1-yuan1.liu@intel.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=198.175.65.18; envelope-from=yuan1.liu@intel.com; helo=mgamail.intel.com X-Spam_score_int: -14 X-Spam_score: -1.5 X-Spam_bar: - X-Spam_report: (-1.5 / 5.0 requ) BAYES_00=-1.9, DATE_IN_PAST_12_24=1.049, DKIMWL_WL_HIGH=-0.422, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @intel.com) X-ZM-MESSAGEID: 1710923653146100003 Content-Type: text/plain; charset="utf-8" add the Query Processing Library (QPL) compression method Although both qpl and zlib support deflate compression, qpl will only use the In-Memory Analytics Accelerator(IAA) for compression and decompression, and IAA is not compatible with the Zlib in migration, so qpl is used as a new compression method for migration. How to enable qpl compression during migration: migrate_set_parameter multifd-compression qpl The qpl only supports one compression level, there is no qpl compression level parameter added, users do not need to specify the qpl compression level. Signed-off-by: Yuan Liu Reviewed-by: Nanhai Zou Reviewed-by: Peter Xu --- hw/core/qdev-properties-system.c | 2 +- migration/meson.build | 1 + migration/multifd-qpl.c | 20 ++++++++++++++++++++ migration/multifd.h | 1 + qapi/migration.json | 7 ++++++- 5 files changed, 29 insertions(+), 2 deletions(-) create mode 100644 migration/multifd-qpl.c diff --git a/hw/core/qdev-properties-system.c b/hw/core/qdev-properties-sys= tem.c index d79d6f4b53..6ccd7224f6 100644 --- a/hw/core/qdev-properties-system.c +++ b/hw/core/qdev-properties-system.c @@ -659,7 +659,7 @@ const PropertyInfo qdev_prop_fdc_drive_type =3D { const PropertyInfo qdev_prop_multifd_compression =3D { .name =3D "MultiFDCompression", .description =3D "multifd_compression values, " - "none/zlib/zstd", + "none/zlib/zstd/qpl", .enum_table =3D &MultiFDCompression_lookup, .get =3D qdev_propinfo_get_enum, .set =3D qdev_propinfo_set_enum, diff --git a/migration/meson.build b/migration/meson.build index 1eeb915ff6..cb177de1d2 100644 --- a/migration/meson.build +++ b/migration/meson.build @@ -41,6 +41,7 @@ if get_option('live_block_migration').allowed() system_ss.add(files('block.c')) endif system_ss.add(when: zstd, if_true: files('multifd-zstd.c')) +system_ss.add(when: qpl, if_true: files('multifd-qpl.c')) =20 specific_ss.add(when: 'CONFIG_SYSTEM_ONLY', if_true: files('ram.c', diff --git a/migration/multifd-qpl.c b/migration/multifd-qpl.c new file mode 100644 index 0000000000..056a68a060 --- /dev/null +++ b/migration/multifd-qpl.c @@ -0,0 +1,20 @@ +/* + * Multifd qpl compression accelerator implementation + * + * Copyright (c) 2023 Intel Corporation + * + * Authors: + * Yuan Liu + * + * This work is licensed under the terms of the GNU GPL, version 2 or late= r. + * See the COPYING file in the top-level directory. + */ +#include "qemu/osdep.h" +#include "qemu/module.h" + +static void multifd_qpl_register(void) +{ + /* noop */ +} + +migration_init(multifd_qpl_register); diff --git a/migration/multifd.h b/migration/multifd.h index c9d9b09239..5b7d9b15f8 100644 --- a/migration/multifd.h +++ b/migration/multifd.h @@ -40,6 +40,7 @@ MultiFDRecvData *multifd_get_recv_data(void); #define MULTIFD_FLAG_NOCOMP (0 << 1) #define MULTIFD_FLAG_ZLIB (1 << 1) #define MULTIFD_FLAG_ZSTD (2 << 1) +#define MULTIFD_FLAG_QPL (4 << 1) =20 /* This value needs to be a multiple of qemu_target_page_size() */ #define MULTIFD_PACKET_SIZE (512 * 1024) diff --git a/qapi/migration.json b/qapi/migration.json index aa1b39bce1..dceb35db5b 100644 --- a/qapi/migration.json +++ b/qapi/migration.json @@ -629,11 +629,16 @@ # # @zstd: use zstd compression method. # +# @qpl: use qpl compression method. Query Processing Library(qpl) is based= on +# the deflate compression algorithm and use the Intel In-Memory Anal= ytics +# Accelerator(IAA) accelerated compression and decompression. (Since= 9.0) +# # Since: 5.0 ## { 'enum': 'MultiFDCompression', 'data': [ 'none', 'zlib', - { 'name': 'zstd', 'if': 'CONFIG_ZSTD' } ] } + { 'name': 'zstd', 'if': 'CONFIG_ZSTD' }, + { 'name': 'qpl', 'if': 'CONFIG_QPL' } ] } =20 ## # @MigMode: --=20 2.39.3 From nobody Mon May 13 03:14:43 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass header.i=@intel.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=intel.com ARC-Seal: i=1; a=rsa-sha256; t=1710923709; cv=none; d=zohomail.com; s=zohoarc; b=f7J/estHt7F2Yqc2mVUctE4oT8pRsNisL+PiEQOlpETcRT3WNuYhvKeTeGKoVDB70TaQSj1dAY49PMA/fQiZ5CmmyyY8tRbkPcMntStQYyBaGHvoe8+S2yAO/s13ouY29HsknmK98hlTe/4GwrKxfuCwhacUPaO7PN5e5XQeHE0= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1710923709; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=C+t0MeO6bIcsJ36G7s5Q2c4Gqta4ZtZMxc7SIsjoK4I=; b=dWh9/EXMmy6nXMD4tCQ4GHAueb1trnidtzlH90EC+I2SPKV0B96XOkZAXafjqiaV8o7ewU8olyIb5Dg8x6PZUCwoUQYc1QfYc/UhI5oGqUen2HgwuKMcb82FuPzfpa1X8ZT2YvLZLr3JR47BkRBqlrmQX+bm6Z+Ud1u5ctJLmMQ= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass header.i=@intel.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1710923709811234.53221238391768; Wed, 20 Mar 2024 01:35:09 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rmrO3-0004XN-ST; Wed, 20 Mar 2024 04:33:27 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rmrNv-0004Wa-KR for qemu-devel@nongnu.org; Wed, 20 Mar 2024 04:33:20 -0400 Received: from mgamail.intel.com ([198.175.65.18]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rmrNs-0005Th-GD for qemu-devel@nongnu.org; Wed, 20 Mar 2024 04:33:18 -0400 Received: from orviesa010.jf.intel.com ([10.64.159.150]) by orvoesa110.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 20 Mar 2024 01:33:16 -0700 Received: from sae-gw02.sh.intel.com (HELO localhost) ([10.239.45.110]) by orviesa010.jf.intel.com with ESMTP; 20 Mar 2024 01:33:14 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1710923597; x=1742459597; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Ejrg1rOtZ3/ZvVXzfZ2it1XhOgKXcEWIjQNpiJPefHc=; b=DQt/zLS3CIiTikCs4iy/DGRjSPKgs4NyQDux2QVBayIgmsVlsiQEeyYl XvElZ2j+X++i8K/eVUGq6PJO7tv+S+GR6RcoKNSzfZXrv5ZZI8LSi2sxV xm/1JdrOxU7k/5Go9FBFKdew/o/GZwBG+SR33wZHY1Cz6V/B03mTf4Hu3 OYjeAEF3BtnKcOnlW/hC4Sum6Ms98bjXRl+8A9cvfjsule9ZTsNABQb7F nafKxN2/oWtBot4vd4Zk/u8XHRzy3O2NsaMDcuGb1cn8/CJjYLhY7ATQJ 8rtH3NaR6ZqJ7iKxMuFd3rPm0hwzfWOCsKK+mVpfbUsskkOdmS5vUuJJX g==; X-IronPort-AV: E=McAfee;i="6600,9927,11018"; a="5952946" X-IronPort-AV: E=Sophos;i="6.07,139,1708416000"; d="scan'208";a="5952946" X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,139,1708416000"; d="scan'208";a="13986732" From: Yuan Liu To: peterx@redhat.com, farosas@suse.de Cc: qemu-devel@nongnu.org, hao.xiang@bytedance.com, bryan.zhang@bytedance.com, yuan1.liu@intel.com, nanhai.zou@intel.com Subject: [PATCH v5 5/7] migration/multifd: implement initialization of qpl compression Date: Wed, 20 Mar 2024 00:45:25 +0800 Message-Id: <20240319164527.1873891-6-yuan1.liu@intel.com> X-Mailer: git-send-email 2.39.3 In-Reply-To: <20240319164527.1873891-1-yuan1.liu@intel.com> References: <20240319164527.1873891-1-yuan1.liu@intel.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=198.175.65.18; envelope-from=yuan1.liu@intel.com; helo=mgamail.intel.com X-Spam_score_int: -14 X-Spam_score: -1.5 X-Spam_bar: - X-Spam_report: (-1.5 / 5.0 requ) BAYES_00=-1.9, DATE_IN_PAST_12_24=1.049, DKIMWL_WL_HIGH=-0.422, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @intel.com) X-ZM-MESSAGEID: 1710923710724100001 Content-Type: text/plain; charset="utf-8" the qpl initialization includes memory allocation for compressed data and the qpl job initialization. the qpl initialization will check whether the In-Memory Analytics Accelerator(IAA) hardware is available, if the platform does not have IAA hardware or the IAA hardware is not available, the QPL compression initialization will fail. Signed-off-by: Yuan Liu Reviewed-by: Nanhai Zou --- migration/multifd-qpl.c | 243 +++++++++++++++++++++++++++++++++++++++- 1 file changed, 242 insertions(+), 1 deletion(-) diff --git a/migration/multifd-qpl.c b/migration/multifd-qpl.c index 056a68a060..6de65e9da7 100644 --- a/migration/multifd-qpl.c +++ b/migration/multifd-qpl.c @@ -9,12 +9,253 @@ * This work is licensed under the terms of the GNU GPL, version 2 or late= r. * See the COPYING file in the top-level directory. */ + #include "qemu/osdep.h" #include "qemu/module.h" +#include "qapi/error.h" +#include "migration.h" +#include "multifd.h" +#include "qpl/qpl.h" + +typedef struct { + qpl_job **job_array; + /* the number of allocated jobs */ + uint32_t job_num; + /* the size of data processed by a qpl job */ + uint32_t data_size; + /* compressed data buffer */ + uint8_t *zbuf; + /* the length of compressed data */ + uint32_t *zbuf_hdr; +} QplData; + +static void free_zbuf(QplData *qpl) +{ + if (qpl->zbuf !=3D NULL) { + munmap(qpl->zbuf, qpl->job_num * qpl->data_size); + qpl->zbuf =3D NULL; + } + if (qpl->zbuf_hdr !=3D NULL) { + g_free(qpl->zbuf_hdr); + qpl->zbuf_hdr =3D NULL; + } +} + +static int alloc_zbuf(QplData *qpl, uint8_t chan_id, Error **errp) +{ + int flags =3D MAP_PRIVATE | MAP_POPULATE | MAP_ANONYMOUS; + uint32_t size =3D qpl->job_num * qpl->data_size; + uint8_t *buf; + + buf =3D (uint8_t *) mmap(NULL, size, PROT_READ | PROT_WRITE, flags, -1= , 0); + if (buf =3D=3D MAP_FAILED) { + error_setg(errp, "multifd: %u: alloc_zbuf failed, job num %u, size= %u", + chan_id, qpl->job_num, qpl->data_size); + return -1; + } + qpl->zbuf =3D buf; + qpl->zbuf_hdr =3D g_new0(uint32_t, qpl->job_num); + return 0; +} + +static void free_jobs(QplData *qpl) +{ + for (int i =3D 0; i < qpl->job_num; i++) { + qpl_fini_job(qpl->job_array[i]); + g_free(qpl->job_array[i]); + qpl->job_array[i] =3D NULL; + } + g_free(qpl->job_array); + qpl->job_array =3D NULL; +} + +static int alloc_jobs(QplData *qpl, uint8_t chan_id, Error **errp) +{ + qpl_status status; + uint32_t job_size =3D 0; + qpl_job *job =3D NULL; + /* always use IAA hardware accelerator */ + qpl_path_t path =3D qpl_path_hardware; + + status =3D qpl_get_job_size(path, &job_size); + if (status !=3D QPL_STS_OK) { + error_setg(errp, "multifd: %u: qpl_get_job_size failed with error = %d", + chan_id, status); + return -1; + } + qpl->job_array =3D g_new0(qpl_job *, qpl->job_num); + for (int i =3D 0; i < qpl->job_num; i++) { + job =3D g_malloc0(job_size); + status =3D qpl_init_job(path, job); + if (status !=3D QPL_STS_OK) { + error_setg(errp, "multifd: %u: qpl_init_job failed with error = %d", + chan_id, status); + free_jobs(qpl); + return -1; + } + qpl->job_array[i] =3D job; + } + return 0; +} + +static int init_qpl(QplData *qpl, uint32_t job_num, uint32_t data_size, + uint8_t chan_id, Error **errp) +{ + qpl->job_num =3D job_num; + qpl->data_size =3D data_size; + if (alloc_zbuf(qpl, chan_id, errp) !=3D 0) { + return -1; + } + if (alloc_jobs(qpl, chan_id, errp) !=3D 0) { + free_zbuf(qpl); + return -1; + } + return 0; +} + +static void deinit_qpl(QplData *qpl) +{ + if (qpl !=3D NULL) { + free_jobs(qpl); + free_zbuf(qpl); + qpl->job_num =3D 0; + qpl->data_size =3D 0; + } +} + +/** + * qpl_send_setup: setup send side + * + * Setup each channel with QPL compression. + * + * Returns 0 for success or -1 for error + * + * @p: Params for the channel that we are using + * @errp: pointer to an error + */ +static int qpl_send_setup(MultiFDSendParams *p, Error **errp) +{ + QplData *qpl; + + qpl =3D g_new0(QplData, 1); + if (init_qpl(qpl, p->page_count, p->page_size, p->id, errp) !=3D 0) { + g_free(qpl); + return -1; + } + p->compress_data =3D qpl; + + assert(p->iov =3D=3D NULL); + /* + * Each page will be compressed independently and sent using an IOV. T= he + * additional two IOVs are used to store packet header and compressed = data + * length + */ + p->iov =3D g_new0(struct iovec, p->page_count + 2); + return 0; +} + +/** + * qpl_send_cleanup: cleanup send side + * + * Close the channel and return memory. + * + * @p: Params for the channel that we are using + * @errp: pointer to an error + */ +static void qpl_send_cleanup(MultiFDSendParams *p, Error **errp) +{ + QplData *qpl =3D p->compress_data; + + deinit_qpl(qpl); + g_free(p->compress_data); + p->compress_data =3D NULL; +} + +/** + * qpl_send_prepare: prepare data to be able to send + * + * Create a compressed buffer with all the pages that we are going to + * send. + * + * Returns 0 for success or -1 for error + * + * @p: Params for the channel that we are using + * @errp: pointer to an error + */ +static int qpl_send_prepare(MultiFDSendParams *p, Error **errp) +{ + /* Implement in next patch */ + return -1; +} + +/** + * qpl_recv_setup: setup receive side + * + * Create the compressed channel and buffer. + * + * Returns 0 for success or -1 for error + * + * @p: Params for the channel that we are using + * @errp: pointer to an error + */ +static int qpl_recv_setup(MultiFDRecvParams *p, Error **errp) +{ + QplData *qpl; + + qpl =3D g_new0(QplData, 1); + if (init_qpl(qpl, p->page_count, p->page_size, p->id, errp) !=3D 0) { + g_free(qpl); + return -1; + } + p->compress_data =3D qpl; + return 0; +} + +/** + * qpl_recv_cleanup: setup receive side + * + * Close the channel and return memory. + * + * @p: Params for the channel that we are using + */ +static void qpl_recv_cleanup(MultiFDRecvParams *p) +{ + QplData *qpl =3D p->compress_data; + + deinit_qpl(qpl); + g_free(p->compress_data); + p->compress_data =3D NULL; +} + +/** + * qpl_recv: read the data from the channel into actual pages + * + * Read the compressed buffer, and uncompress it into the actual + * pages. + * + * Returns 0 for success or -1 for error + * + * @p: Params for the channel that we are using + * @errp: pointer to an error + */ +static int qpl_recv(MultiFDRecvParams *p, Error **errp) +{ + /* Implement in next patch */ + return -1; +} + +static MultiFDMethods multifd_qpl_ops =3D { + .send_setup =3D qpl_send_setup, + .send_cleanup =3D qpl_send_cleanup, + .send_prepare =3D qpl_send_prepare, + .recv_setup =3D qpl_recv_setup, + .recv_cleanup =3D qpl_recv_cleanup, + .recv =3D qpl_recv, +}; =20 static void multifd_qpl_register(void) { - /* noop */ + multifd_register_ops(MULTIFD_COMPRESSION_QPL, &multifd_qpl_ops); } =20 migration_init(multifd_qpl_register); --=20 2.39.3 From nobody Mon May 13 03:14:43 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass header.i=@intel.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=intel.com ARC-Seal: i=1; a=rsa-sha256; t=1710923696; cv=none; d=zohomail.com; s=zohoarc; b=AE9c6pC3wzLKBAu3i48SwFad1Kw60ji10D23HcfT9YKr3fAAzFvjGtV0AM+k65OyA2+hldw+pxJdC4w75ITOQhpzJeuyd231yw4j5evts4VJP0tdO4xJT2yEVIMGWCRbQ+Y+OWJtvehKhObkyOUeJgfhXombQF1f8bk8gR3CyFc= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1710923696; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=P09f2zaJZGDn80Uie3UMM1pv5W/K621I213qkHr/x4U=; b=k9qKKnn8FBZlnqvNQAXxVBkLbn1voMio3IVM3lzlGmrMNlB3btcXJr7ouextDaGf6HDNF6d+XWLl0Kn4Fum60o4B9pfiVSRw85IBI6CcOFSwFtkNRZx0lcVflpuP5h/OfF4L8VzppZK+VE1tgWfidnLa8OwwZtiG2DtxYeMKQxY= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass header.i=@intel.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1710923696595660.6846898962666; Wed, 20 Mar 2024 01:34:56 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rmrOC-0004b4-7k; Wed, 20 Mar 2024 04:33:38 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rmrO5-0004Xk-Ao for qemu-devel@nongnu.org; Wed, 20 Mar 2024 04:33:31 -0400 Received: from mgamail.intel.com ([198.175.65.18]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rmrO3-0005X7-5L for qemu-devel@nongnu.org; Wed, 20 Mar 2024 04:33:28 -0400 Received: from orviesa010.jf.intel.com ([10.64.159.150]) by orvoesa110.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 20 Mar 2024 01:33:25 -0700 Received: from sae-gw02.sh.intel.com (HELO localhost) ([10.239.45.110]) by orviesa010.jf.intel.com with ESMTP; 20 Mar 2024 01:33:22 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1710923608; x=1742459608; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=bpzLqW2ZegY22d41gn9ZtAMrAQ7kfnyo5nCZBjoB/PQ=; b=B0fCZgin+69YQ9x9pCSUbVcJMiLHQvEdQy4I1T5JnSqIrmgcLmjQYN0a 2RfrqRISoYURGyi+0mw3uH8aAHwjbQ6HuRhI4j08NZet9sFdmlvhBZ3bq nZSllZpdxmyi9n5xqhtdDLH/dfw29ZOtxtH+3oGoQHz9mmisVeTTFvFS3 vA4325RONH0NCc5g9/PMTcQH8j4nDJUO1pV7twZs+hW8tmRatdECxohb1 6Da1KPz3PIHh2my0ntXfvSvxHMBZbfURc/SWyVcPNdStx0SdLkbnKyc3S bBbKR3irAVUUE+58mJ55m41ogRGldkAQBFjNGzrNbDlhJk5RwhdsNu0om Q==; X-IronPort-AV: E=McAfee;i="6600,9927,11018"; a="5952967" X-IronPort-AV: E=Sophos;i="6.07,139,1708416000"; d="scan'208";a="5952967" X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,139,1708416000"; d="scan'208";a="13986747" From: Yuan Liu To: peterx@redhat.com, farosas@suse.de Cc: qemu-devel@nongnu.org, hao.xiang@bytedance.com, bryan.zhang@bytedance.com, yuan1.liu@intel.com, nanhai.zou@intel.com Subject: [PATCH v5 6/7] migration/multifd: implement qpl compression and decompression Date: Wed, 20 Mar 2024 00:45:26 +0800 Message-Id: <20240319164527.1873891-7-yuan1.liu@intel.com> X-Mailer: git-send-email 2.39.3 In-Reply-To: <20240319164527.1873891-1-yuan1.liu@intel.com> References: <20240319164527.1873891-1-yuan1.liu@intel.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=198.175.65.18; envelope-from=yuan1.liu@intel.com; helo=mgamail.intel.com X-Spam_score_int: -14 X-Spam_score: -1.5 X-Spam_bar: - X-Spam_report: (-1.5 / 5.0 requ) BAYES_00=-1.9, DATE_IN_PAST_12_24=1.049, DKIMWL_WL_HIGH=-0.422, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @intel.com) X-ZM-MESSAGEID: 1710923698646100003 Content-Type: text/plain; charset="utf-8" each qpl job is used to (de)compress a normal page and it can be processed independently by the IAA hardware. All qpl jobs are submitted to the hardware at once, and wait for all jobs completion. Signed-off-by: Yuan Liu Reviewed-by: Nanhai Zou --- migration/multifd-qpl.c | 229 +++++++++++++++++++++++++++++++++++++++- 1 file changed, 225 insertions(+), 4 deletions(-) diff --git a/migration/multifd-qpl.c b/migration/multifd-qpl.c index 6de65e9da7..479b051b24 100644 --- a/migration/multifd-qpl.c +++ b/migration/multifd-qpl.c @@ -13,6 +13,7 @@ #include "qemu/osdep.h" #include "qemu/module.h" #include "qapi/error.h" +#include "exec/ramblock.h" #include "migration.h" #include "multifd.h" #include "qpl/qpl.h" @@ -171,6 +172,112 @@ static void qpl_send_cleanup(MultiFDSendParams *p, Er= ror **errp) p->compress_data =3D NULL; } =20 +static inline void prepare_job(qpl_job *job, uint8_t *input, uint32_t inpu= t_len, + uint8_t *output, uint32_t output_len, + bool is_compression) +{ + job->op =3D is_compression ? qpl_op_compress : qpl_op_decompress; + job->next_in_ptr =3D input; + job->next_out_ptr =3D output; + job->available_in =3D input_len; + job->available_out =3D output_len; + job->flags =3D QPL_FLAG_FIRST | QPL_FLAG_LAST | QPL_FLAG_OMIT_VERIFY; + /* only supports one compression level */ + job->level =3D 1; +} + +/** + * set_raw_data_hdr: set the length of raw data + * + * If the length of the compressed output data is greater than or equal to + * the page size, then set the compressed data length to the data size and + * send raw data directly. + * + * @qpl: pointer to the QplData structure + * @index: the index of the compression job header + */ +static inline void set_raw_data_hdr(QplData *qpl, uint32_t index) +{ + assert(index < qpl->job_num); + qpl->zbuf_hdr[index] =3D cpu_to_be32(qpl->data_size); +} + +/** + * is_raw_data: check if the data is raw data + * + * The raw data length is always equal to data size, which is the + * size of one page. + * + * Returns true if the data is raw data, otherwise false + * + * @qpl: pointer to the QplData structure + * @index: the index of the decompressed job header + */ +static inline bool is_raw_data(QplData *qpl, uint32_t index) +{ + assert(index < qpl->job_num); + return qpl->zbuf_hdr[index] =3D=3D qpl->data_size; +} + +static int run_comp_jobs(MultiFDSendParams *p, Error **errp) +{ + qpl_status status; + QplData *qpl =3D p->compress_data; + MultiFDPages_t *pages =3D p->pages; + uint32_t job_num =3D pages->normal_num; + qpl_job *job =3D NULL; + uint32_t off =3D 0; + + assert(job_num <=3D qpl->job_num); + /* submit all compression jobs */ + for (int i =3D 0; i < job_num; i++) { + job =3D qpl->job_array[i]; + /* the compressed data size should be less than one page */ + prepare_job(job, pages->block->host + pages->offset[i], qpl->data_= size, + qpl->zbuf + off, qpl->data_size - 1, true); +retry: + status =3D qpl_submit_job(job); + if (status =3D=3D QPL_STS_OK) { + off +=3D qpl->data_size; + } else if (status =3D=3D QPL_STS_QUEUES_ARE_BUSY_ERR) { + goto retry; + } else { + error_setg(errp, "multifd %u: qpl_submit_job failed with error= %d", + p->id, status); + return -1; + } + } + + /* wait all jobs to complete */ + for (int i =3D 0; i < job_num; i++) { + job =3D qpl->job_array[i]; + status =3D qpl_wait_job(job); + if (status =3D=3D QPL_STS_OK) { + qpl->zbuf_hdr[i] =3D cpu_to_be32(job->total_out); + p->iov[p->iovs_num].iov_len =3D job->total_out; + p->iov[p->iovs_num].iov_base =3D qpl->zbuf + (qpl->data_size *= i); + p->next_packet_size +=3D job->total_out; + } else if (status =3D=3D QPL_STS_MORE_OUTPUT_NEEDED) { + /* + * the compression job does not fail, the output data + * size is larger than the provided memory size. In this + * case, raw data is sent directly to the destination. + */ + set_raw_data_hdr(qpl, i); + p->iov[p->iovs_num].iov_len =3D qpl->data_size; + p->iov[p->iovs_num].iov_base =3D pages->block->host + + pages->offset[i]; + p->next_packet_size +=3D qpl->data_size; + } else { + error_setg(errp, "multifd %u: qpl_wait_job failed with error %= d", + p->id, status); + return -1; + } + p->iovs_num++; + } + return 0; +} + /** * qpl_send_prepare: prepare data to be able to send * @@ -184,8 +291,28 @@ static void qpl_send_cleanup(MultiFDSendParams *p, Err= or **errp) */ static int qpl_send_prepare(MultiFDSendParams *p, Error **errp) { - /* Implement in next patch */ - return -1; + QplData *qpl =3D p->compress_data; + uint32_t hdr_size; + + if (!multifd_send_prepare_common(p)) { + goto out; + } + + assert(p->pages->normal_num <=3D qpl->job_num); + hdr_size =3D p->pages->normal_num * sizeof(uint32_t); + /* prepare the header that stores the lengths of all compressed data */ + p->iov[1].iov_base =3D (uint8_t *) qpl->zbuf_hdr; + p->iov[1].iov_len =3D hdr_size; + p->iovs_num++; + p->next_packet_size +=3D hdr_size; + if (run_comp_jobs(p, errp) !=3D 0) { + return -1; + } + +out: + p->flags |=3D MULTIFD_FLAG_QPL; + multifd_send_fill_packet(p); + return 0; } =20 /** @@ -227,6 +354,60 @@ static void qpl_recv_cleanup(MultiFDRecvParams *p) p->compress_data =3D NULL; } =20 +static int run_decomp_jobs(MultiFDRecvParams *p, Error **errp) +{ + qpl_status status; + qpl_job *job; + QplData *qpl =3D p->compress_data; + uint32_t off =3D 0; + uint32_t job_num =3D p->normal_num; + + assert(job_num <=3D qpl->job_num); + /* submit all decompression jobs */ + for (int i =3D 0; i < job_num; i++) { + /* for the raw data, load it directly */ + if (is_raw_data(qpl, i)) { + memcpy(p->host + p->normal[i], qpl->zbuf + off, qpl->data_size= ); + off +=3D qpl->data_size; + continue; + } + job =3D qpl->job_array[i]; + prepare_job(job, qpl->zbuf + off, qpl->zbuf_hdr[i], + p->host + p->normal[i], qpl->data_size, false); +retry: + status =3D qpl_submit_job(job); + if (status =3D=3D QPL_STS_OK) { + off +=3D qpl->zbuf_hdr[i]; + } else if (status =3D=3D QPL_STS_QUEUES_ARE_BUSY_ERR) { + goto retry; + } else { + error_setg(errp, "multifd %u: qpl_submit_job failed with error= %d", + p->id, status); + return -1; + } + } + + /* wait all jobs to complete */ + for (int i =3D 0; i < job_num; i++) { + if (is_raw_data(qpl, i)) { + continue; + } + job =3D qpl->job_array[i]; + status =3D qpl_wait_job(job); + if (status !=3D QPL_STS_OK) { + error_setg(errp, "multifd %u: qpl_wait_job failed with error %= d", + p->id, status); + return -1; + } + if (job->total_out !=3D qpl->data_size) { + error_setg(errp, "multifd %u: decompressed len %u, expected le= n %u", + p->id, job->total_out, qpl->data_size); + return -1; + } + } + return 0; +} + /** * qpl_recv: read the data from the channel into actual pages * @@ -240,8 +421,48 @@ static void qpl_recv_cleanup(MultiFDRecvParams *p) */ static int qpl_recv(MultiFDRecvParams *p, Error **errp) { - /* Implement in next patch */ - return -1; + QplData *qpl =3D p->compress_data; + uint32_t in_size =3D p->next_packet_size; + uint32_t flags =3D p->flags & MULTIFD_FLAG_COMPRESSION_MASK; + uint32_t hdr_len =3D p->normal_num * sizeof(uint32_t); + uint32_t data_len =3D 0; + int ret; + + if (flags !=3D MULTIFD_FLAG_QPL) { + error_setg(errp, "multifd %u: flags received %x flags expected %x", + p->id, flags, MULTIFD_FLAG_QPL); + return -1; + } + multifd_recv_zero_page_process(p); + if (!p->normal_num) { + assert(in_size =3D=3D 0); + return 0; + } + + /* read compressed data lengths */ + assert(hdr_len < in_size); + ret =3D qio_channel_read_all(p->c, (void *) qpl->zbuf_hdr, hdr_len, er= rp); + if (ret !=3D 0) { + return ret; + } + assert(p->normal_num <=3D qpl->job_num); + for (int i =3D 0; i < p->normal_num; i++) { + qpl->zbuf_hdr[i] =3D be32_to_cpu(qpl->zbuf_hdr[i]); + data_len +=3D qpl->zbuf_hdr[i]; + assert(qpl->zbuf_hdr[i] <=3D qpl->data_size); + } + + /* read compressed data */ + assert(in_size =3D=3D hdr_len + data_len); + ret =3D qio_channel_read_all(p->c, (void *) qpl->zbuf, data_len, errp); + if (ret !=3D 0) { + return ret; + } + + if (run_decomp_jobs(p, errp) !=3D 0) { + return -1; + } + return 0; } =20 static MultiFDMethods multifd_qpl_ops =3D { --=20 2.39.3 From nobody Mon May 13 03:14:43 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass header.i=@intel.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=intel.com ARC-Seal: i=1; a=rsa-sha256; t=1710923652; cv=none; d=zohomail.com; s=zohoarc; b=jA4pwjEwrA6H+gmsz6qMVd2FoybUn4zwjg6qXtQZAcCOfYvwP16HqG96JEEbGDwWit+RuhybTYjpDFje1HdRmLFexZjehDZOY7RD6fSfdtNq0Zqx7PsBjzrf+l6bQlhxf1yU7uCbd0xj9P+IzEwKVmwthX+1IvPZl+VV54cau84= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1710923652; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=Qb9BrJJHJGUFRHFKaZfB6FvkF4MMJdcIH8uU+Ju254k=; b=PCfv8O0/8NIX3wUP7WhPR1XKr/0akQ8FZhjau/rFaeKvtLntD+zwV1bwUU4rz2vqc5aBV2jbJCkVAUxEFT28/XUcENXmTLrAiL5PHbvxMomnkUKPHUzrB396Gcia3bpxPOowszYJ8XPtVBzCDOdaqBZQGnTGgfG+sTHEpJp4RPw= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass header.i=@intel.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1710923652518850.2755148357215; Wed, 20 Mar 2024 01:34:12 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rmrOF-0004c7-MU; Wed, 20 Mar 2024 04:33:39 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rmrOA-0004am-Hx for qemu-devel@nongnu.org; Wed, 20 Mar 2024 04:33:34 -0400 Received: from mgamail.intel.com ([198.175.65.18]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rmrO5-0005XU-04 for qemu-devel@nongnu.org; Wed, 20 Mar 2024 04:33:30 -0400 Received: from orviesa010.jf.intel.com ([10.64.159.150]) by orvoesa110.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 20 Mar 2024 01:33:28 -0700 Received: from sae-gw02.sh.intel.com (HELO localhost) ([10.239.45.110]) by orviesa010.jf.intel.com with ESMTP; 20 Mar 2024 01:33:26 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1710923609; x=1742459609; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=ojS7//kmXedbnaPprHSWdUIukwbzAZV70hqMHOhV4NQ=; b=RTHq2iA41H94oAS4yqD1qq4xKRUsRLQC3F2tdLHBFrXrMnrhMcmdZXAh WEsHlgpzwt1//WTUyRICF8Baf5EJCqPavE5aZsI4BHSDzFXBmchR9K3bA rxoaHDC3jCd+U75dwn77Vmb0kJldreztb955BujFIzaGJcNezpRtK2Eqt y8rJbM18C56PnGnxFmZutgbQgM7ah3c9ViifVBq6RT3iK2FOBd7FnFfJD DpvNWMXS2jiuQAGfBJ3/VFfxQn2xZ/y9bA3WKQUObz7V0QyUDG3UYPOVg +eanlXZ3ay3HvtHlRIyBWUYmJKu/VBSGl8+R+Ge1Tin90VoKgrf71gmrB w==; X-IronPort-AV: E=McAfee;i="6600,9927,11018"; a="5952974" X-IronPort-AV: E=Sophos;i="6.07,139,1708416000"; d="scan'208";a="5952974" X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,139,1708416000"; d="scan'208";a="13986755" From: Yuan Liu To: peterx@redhat.com, farosas@suse.de Cc: qemu-devel@nongnu.org, hao.xiang@bytedance.com, bryan.zhang@bytedance.com, yuan1.liu@intel.com, nanhai.zou@intel.com Subject: [PATCH v5 7/7] tests/migration-test: add qpl compression test Date: Wed, 20 Mar 2024 00:45:27 +0800 Message-Id: <20240319164527.1873891-8-yuan1.liu@intel.com> X-Mailer: git-send-email 2.39.3 In-Reply-To: <20240319164527.1873891-1-yuan1.liu@intel.com> References: <20240319164527.1873891-1-yuan1.liu@intel.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=198.175.65.18; envelope-from=yuan1.liu@intel.com; helo=mgamail.intel.com X-Spam_score_int: -14 X-Spam_score: -1.5 X-Spam_bar: - X-Spam_report: (-1.5 / 5.0 requ) BAYES_00=-1.9, DATE_IN_PAST_12_24=1.049, DKIMWL_WL_HIGH=-0.422, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @intel.com) X-ZM-MESSAGEID: 1710923654599100011 Content-Type: text/plain; charset="utf-8" add qpl to compression method test for multifd migration the migration with qpl compression needs to access IAA hardware resource, please run "check-qtest" with sudo or root permission, otherwise migration test will fail Signed-off-by: Yuan Liu Reviewed-by: Nanhai Zou --- tests/qtest/migration-test.c | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/tests/qtest/migration-test.c b/tests/qtest/migration-test.c index 71895abb7f..052d0d60fd 100644 --- a/tests/qtest/migration-test.c +++ b/tests/qtest/migration-test.c @@ -2815,6 +2815,15 @@ test_migrate_precopy_tcp_multifd_zstd_start(QTestSta= te *from, } #endif /* CONFIG_ZSTD */ =20 +#ifdef CONFIG_QPL +static void * +test_migrate_precopy_tcp_multifd_qpl_start(QTestState *from, + QTestState *to) +{ + return test_migrate_precopy_tcp_multifd_start_common(from, to, "qpl"); +} +#endif /* CONFIG_QPL */ + static void test_multifd_tcp_none(void) { MigrateCommon args =3D { @@ -2880,6 +2889,17 @@ static void test_multifd_tcp_zstd(void) } #endif =20 +#ifdef CONFIG_QPL +static void test_multifd_tcp_qpl(void) +{ + MigrateCommon args =3D { + .listen_uri =3D "defer", + .start_hook =3D test_migrate_precopy_tcp_multifd_qpl_start, + }; + test_precopy_common(&args); +} +#endif + #ifdef CONFIG_GNUTLS static void * test_migrate_multifd_tcp_tls_psk_start_match(QTestState *from, @@ -3789,6 +3809,10 @@ int main(int argc, char **argv) migration_test_add("/migration/multifd/tcp/plain/zstd", test_multifd_tcp_zstd); #endif +#ifdef CONFIG_QPL + migration_test_add("/migration/multifd/tcp/plain/qpl", + test_multifd_tcp_qpl); +#endif #ifdef CONFIG_GNUTLS migration_test_add("/migration/multifd/tcp/tls/psk/match", test_multifd_tcp_tls_psk_match); --=20 2.39.3