From nobody Tue Feb 10 04:17:34 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1668515364; cv=none; d=zohomail.com; s=zohoarc; b=H7wYccAzxc/HILav3K+SHBmC2vZWRCkASIFuJ9Uo4o3XxudhxBDMa980u/XRfbBRATm5AQFbBhW/IWUZt13YUWLz/PiksCjM9tjQjxRQ/jZJ2oJbsxGuRELporUNnF8ihplFb8uB/0budADkdyYI9e9OaLlnWh5OmjGjaN/rXZs= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1668515364; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=qvaUxDudVCAwGQRW2EUntGvs2VEXPayNHnuUyocEiNk=; b=USN2HKjKz4RH0zsP/KjsC0w39VLcI10PBnLNVXoACT24tNcuASeh+qZt9VDFw/oi7UNMtiyNnBQTOZ+ZfnwCaKKSU4TTQXj/5ZXyLJzGrcNrBFWNX2U62M5ptbAzOfyNszI8j4Au7qp2ep9Na5qrXuESNhxg4fIxbmuQdrDFzpo= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1668515364797112.09929821191224; Tue, 15 Nov 2022 04:29:24 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ouuoh-0004Qe-Mf; Tue, 15 Nov 2022 07:13:27 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ouuof-0004L5-Py for qemu-devel@nongnu.org; Tue, 15 Nov 2022 07:13:25 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ouuob-0004fJ-PQ for qemu-devel@nongnu.org; Tue, 15 Nov 2022 07:13:25 -0500 Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-417-4Ii4rg6sMn6mz6t25Lazkg-1; Tue, 15 Nov 2022 07:13:19 -0500 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 55C6C1C09064; Tue, 15 Nov 2022 12:13:02 +0000 (UTC) Received: from secure.mitica (unknown [10.39.195.11]) by smtp.corp.redhat.com (Postfix) with ESMTP id 46FD8492B05; Tue, 15 Nov 2022 12:12:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1668514401; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=qvaUxDudVCAwGQRW2EUntGvs2VEXPayNHnuUyocEiNk=; b=RRAHgyeBw2OIOE+XFHA+9Vrgg3U7Gw0it3p4QUdnVU0MbIXP3zcmN9f2+Sa2IK8feBBVcR byecuY3Z4u0denfedApHM5yp6E6A1+cIO+FUa5cOxaX95XDGTD9F/xhFEvy7rA34nODw/W qBCmOSuNzwj4pl/kWeHy6MJngI2rbWc= X-MC-Unique: 4Ii4rg6sMn6mz6t25Lazkg-1 From: Juan Quintela To: qemu-devel@nongnu.org Cc: Paolo Bonzini , David Hildenbrand , Laurent Vivier , Stefan Hajnoczi , Fam Zheng , qemu-block@nongnu.org, "Dr. David Alan Gilbert" , Thomas Huth , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , qemu-trivial@nongnu.org, Michael Tokarev , =?UTF-8?q?Daniel=20P=2E=20Berrang=C3=A9?= , =?UTF-8?q?Marc-Andr=C3=A9=20Lureau?= , Peter Xu , Juan Quintela , ling xu , Zhou Zhao , Jun Jin Subject: [PATCH 08/30] Update AVX512 support for xbzrle_encode_buffer Date: Tue, 15 Nov 2022 13:12:04 +0100 Message-Id: <20221115121226.26609-9-quintela@redhat.com> In-Reply-To: <20221115121226.26609-1-quintela@redhat.com> References: <20221115121226.26609-1-quintela@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.1 on 10.11.54.10 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.129.124; envelope-from=quintela@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1668515365436100001 Content-Type: text/plain; charset="utf-8" From: ling xu This commit updates code of avx512 support for xbzrle_encode_buffer function to accelerate xbzrle encoding speed. Runtime check of avx512 support and benchmark for this feature are added. Compared with C version of xbzrle_encode_buffer function, avx512 version can achieve 50%-70% performance improvement on benchmarking. In addition, if dirty data is randomly located in 4K page, the avx512 version can achieve almost 140% performance gain. Signed-off-by: ling xu Co-authored-by: Zhou Zhao Co-authored-by: Jun Jin Reviewed-by: Juan Quintela Signed-off-by: Juan Quintela --- meson.build | 16 +++++ migration/xbzrle.h | 4 ++ migration/ram.c | 34 +++++++++- migration/xbzrle.c | 124 ++++++++++++++++++++++++++++++++++ meson_options.txt | 2 + scripts/meson-buildoptions.sh | 14 ++-- 6 files changed, 186 insertions(+), 8 deletions(-) diff --git a/meson.build b/meson.build index cf3e517e56..d0d28f5c9e 100644 --- a/meson.build +++ b/meson.build @@ -2344,6 +2344,22 @@ config_host_data.set('CONFIG_AVX512F_OPT', get_optio= n('avx512f') \ int main(int argc, char *argv[]) { return bar(argv[argc - 1]); } '''), error_message: 'AVX512F not available').allowed()) =20 +config_host_data.set('CONFIG_AVX512BW_OPT', get_option('avx512bw') \ + .require(have_cpuid_h, error_message: 'cpuid.h not available, cannot ena= ble AVX512BW') \ + .require(cc.links(''' + #pragma GCC push_options + #pragma GCC target("avx512bw") + #include + #include + static int bar(void *a) { + + __m512i *x =3D a; + __m512i res=3D _mm512_abs_epi8(*x); + return res[1]; + } + int main(int argc, char *argv[]) { return bar(argv[0]); } + '''), error_message: 'AVX512BW not available').allowed()) + have_pvrdma =3D get_option('pvrdma') \ .require(rdma.found(), error_message: 'PVRDMA requires OpenFabrics libra= ries') \ .require(cc.compiles(gnu_source_prefix + ''' diff --git a/migration/xbzrle.h b/migration/xbzrle.h index a0db507b9c..6feb49160a 100644 --- a/migration/xbzrle.h +++ b/migration/xbzrle.h @@ -18,4 +18,8 @@ int xbzrle_encode_buffer(uint8_t *old_buf, uint8_t *new_b= uf, int slen, uint8_t *dst, int dlen); =20 int xbzrle_decode_buffer(uint8_t *src, int slen, uint8_t *dst, int dlen); +#if defined(CONFIG_AVX512BW_OPT) +int xbzrle_encode_buffer_avx512(uint8_t *old_buf, uint8_t *new_buf, int sl= en, + uint8_t *dst, int dlen); +#endif #endif diff --git a/migration/ram.c b/migration/ram.c index 67e41dd2c0..bb4f08bfed 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -83,6 +83,34 @@ /* 0x80 is reserved in migration.h start with 0x100 next */ #define RAM_SAVE_FLAG_COMPRESS_PAGE 0x100 =20 +int (*xbzrle_encode_buffer_func)(uint8_t *, uint8_t *, int, + uint8_t *, int) =3D xbzrle_encode_buffer; +#if defined(CONFIG_AVX512BW_OPT) +#include "qemu/cpuid.h" +static void __attribute__((constructor)) init_cpu_flag(void) +{ + unsigned max =3D __get_cpuid_max(0, NULL); + int a, b, c, d; + if (max >=3D 1) { + __cpuid(1, a, b, c, d); + /* We must check that AVX is not just available, but usable. */ + if ((c & bit_OSXSAVE) && (c & bit_AVX) && max >=3D 7) { + int bv; + __asm("xgetbv" : "=3Da"(bv), "=3Dd"(d) : "c"(0)); + __cpuid_count(7, 0, a, b, c, d); + /* 0xe6: + * XCR0[7:5] =3D 111b (OPMASK state, upper 256-bit of ZMM0-ZMM= 15 + * and ZMM16-ZMM31 state are enabled by OS) + * XCR0[2:1] =3D 11b (XMM state and YMM state are enabled by O= S) + */ + if ((bv & 0xe6) =3D=3D 0xe6 && (b & bit_AVX512BW)) { + xbzrle_encode_buffer_func =3D xbzrle_encode_buffer_avx512; + } + } + } +} +#endif + XBZRLECacheStats xbzrle_counters; =20 /* struct contains XBZRLE cache and a static page @@ -802,9 +830,9 @@ static int save_xbzrle_page(RAMState *rs, uint8_t **cur= rent_data, memcpy(XBZRLE.current_buf, *current_data, TARGET_PAGE_SIZE); =20 /* XBZRLE encoding (if there is no overflow) */ - encoded_len =3D xbzrle_encode_buffer(prev_cached_page, XBZRLE.current_= buf, - TARGET_PAGE_SIZE, XBZRLE.encoded_bu= f, - TARGET_PAGE_SIZE); + encoded_len =3D xbzrle_encode_buffer_func(prev_cached_page, XBZRLE.cur= rent_buf, + TARGET_PAGE_SIZE, XBZRLE.encod= ed_buf, + TARGET_PAGE_SIZE); =20 /* * Update the cache contents, so that it corresponds to the data diff --git a/migration/xbzrle.c b/migration/xbzrle.c index 1ba482ded9..05366e86c0 100644 --- a/migration/xbzrle.c +++ b/migration/xbzrle.c @@ -174,3 +174,127 @@ int xbzrle_decode_buffer(uint8_t *src, int slen, uint= 8_t *dst, int dlen) =20 return d; } + +#if defined(CONFIG_AVX512BW_OPT) +#pragma GCC push_options +#pragma GCC target("avx512bw") +#include +int xbzrle_encode_buffer_avx512(uint8_t *old_buf, uint8_t *new_buf, int sl= en, + uint8_t *dst, int dlen) +{ + uint32_t zrun_len =3D 0, nzrun_len =3D 0; + int d =3D 0, i =3D 0, num =3D 0; + uint8_t *nzrun_start =3D NULL; + /* add 1 to include residual part in main loop */ + uint32_t count512s =3D (slen >> 6) + 1; + /* countResidual is tail of data, i.e., countResidual =3D slen % 64 */ + uint32_t count_residual =3D slen & 0b111111; + bool never_same =3D true; + uint64_t mask_residual =3D 1; + mask_residual <<=3D count_residual; + mask_residual -=3D 1; + __m512i r =3D _mm512_set1_epi32(0); + + while (count512s) { + if (d + 2 > dlen) { + return -1; + } + + int bytes_to_check =3D 64; + uint64_t mask =3D 0xffffffffffffffff; + if (count512s =3D=3D 1) { + bytes_to_check =3D count_residual; + mask =3D mask_residual; + } + __m512i old_data =3D _mm512_mask_loadu_epi8(r, + mask, old_buf + i); + __m512i new_data =3D _mm512_mask_loadu_epi8(r, + mask, new_buf + i); + uint64_t comp =3D _mm512_cmpeq_epi8_mask(old_data, new_data); + count512s--; + + bool is_same =3D (comp & 0x1); + while (bytes_to_check) { + if (is_same) { + if (nzrun_len) { + d +=3D uleb128_encode_small(dst + d, nzrun_len); + if (d + nzrun_len > dlen) { + return -1; + } + nzrun_start =3D new_buf + i - nzrun_len; + memcpy(dst + d, nzrun_start, nzrun_len); + d +=3D nzrun_len; + nzrun_len =3D 0; + } + /* 64 data at a time for speed */ + if (count512s && (comp =3D=3D 0xffffffffffffffff)) { + i +=3D 64; + zrun_len +=3D 64; + break; + } + never_same =3D false; + num =3D __builtin_ctzll(~comp); + num =3D (num < bytes_to_check) ? num : bytes_to_check; + zrun_len +=3D num; + bytes_to_check -=3D num; + comp >>=3D num; + i +=3D num; + if (bytes_to_check) { + /* still has different data after same data */ + d +=3D uleb128_encode_small(dst + d, zrun_len); + zrun_len =3D 0; + } else { + break; + } + } + if (never_same || zrun_len) { + /* + * never_same only acts if + * data begins with diff in first count512s + */ + d +=3D uleb128_encode_small(dst + d, zrun_len); + zrun_len =3D 0; + never_same =3D false; + } + /* has diff, 64 data at a time for speed */ + if ((bytes_to_check =3D=3D 64) && (comp =3D=3D 0x0)) { + i +=3D 64; + nzrun_len +=3D 64; + break; + } + num =3D __builtin_ctzll(comp); + num =3D (num < bytes_to_check) ? num : bytes_to_check; + nzrun_len +=3D num; + bytes_to_check -=3D num; + comp >>=3D num; + i +=3D num; + if (bytes_to_check) { + /* mask like 111000 */ + d +=3D uleb128_encode_small(dst + d, nzrun_len); + /* overflow */ + if (d + nzrun_len > dlen) { + return -1; + } + nzrun_start =3D new_buf + i - nzrun_len; + memcpy(dst + d, nzrun_start, nzrun_len); + d +=3D nzrun_len; + nzrun_len =3D 0; + is_same =3D true; + } + } + } + + if (nzrun_len !=3D 0) { + d +=3D uleb128_encode_small(dst + d, nzrun_len); + /* overflow */ + if (d + nzrun_len > dlen) { + return -1; + } + nzrun_start =3D new_buf + i - nzrun_len; + memcpy(dst + d, nzrun_start, nzrun_len); + d +=3D nzrun_len; + } + return d; +} +#pragma GCC pop_options +#endif diff --git a/meson_options.txt b/meson_options.txt index 66128178bf..96814dd211 100644 --- a/meson_options.txt +++ b/meson_options.txt @@ -104,6 +104,8 @@ option('avx2', type: 'feature', value: 'auto', description: 'AVX2 optimizations') option('avx512f', type: 'feature', value: 'disabled', description: 'AVX512F optimizations') +option('avx512bw', type: 'feature', value: 'auto', + description: 'AVX512BW optimizations') option('keyring', type: 'feature', value: 'auto', description: 'Linux keyring support') =20 diff --git a/scripts/meson-buildoptions.sh b/scripts/meson-buildoptions.sh index 2cb0de5601..bcb5d854a5 100644 --- a/scripts/meson-buildoptions.sh +++ b/scripts/meson-buildoptions.sh @@ -40,7 +40,8 @@ meson_options_help() { printf "%s\n" ' --enable-trace-backends=3DCHOICES' printf "%s\n" ' Set available tracing backends= [log] (choices:' printf "%s\n" ' dtrace/ftrace/log/nop/simple/s= yslog/ust)' - printf "%s\n" ' --firmwarepath=3DVALUES search PATH for firmware fil= es [share/qemu-firmware]' + printf "%s\n" ' --firmwarepath=3DVALUES search PATH for firmware fil= es [share/qemu-' + printf "%s\n" ' firmware]' printf "%s\n" ' --iasl=3DVALUE Path to ACPI disassembler' printf "%s\n" ' --includedir=3DVALUE Header file directory [inclu= de]' printf "%s\n" ' --interp-prefix=3DVALUE where to find shared librari= es etc., use %M for' @@ -66,6 +67,7 @@ meson_options_help() { printf "%s\n" ' attr attr/xattr support' printf "%s\n" ' auth-pam PAM access control' printf "%s\n" ' avx2 AVX2 optimizations' + printf "%s\n" ' avx512bw AVX512BW optimizations' printf "%s\n" ' avx512f AVX512F optimizations' printf "%s\n" ' blkio libblkio block device driver' printf "%s\n" ' bochs bochs image format support' @@ -155,6 +157,8 @@ meson_options_help() { printf "%s\n" ' usb-redir libusbredir support' printf "%s\n" ' vde vde network backend support' printf "%s\n" ' vdi vdi image format support' + printf "%s\n" ' vduse-blk-export' + printf "%s\n" ' VDUSE block export support' printf "%s\n" ' vfio-user-server' printf "%s\n" ' vfio-user server support' printf "%s\n" ' vhost-crypto vhost-user crypto backend support' @@ -163,8 +167,6 @@ meson_options_help() { printf "%s\n" ' vhost-user vhost-user backend support' printf "%s\n" ' vhost-user-blk-server' printf "%s\n" ' build vhost-user-blk server' - printf "%s\n" ' vduse-blk-export' - printf "%s\n" ' VDUSE block export support' printf "%s\n" ' vhost-vdpa vhost-vdpa kernel backend support' printf "%s\n" ' virglrenderer virgl rendering support' printf "%s\n" ' virtfs virtio-9p support' @@ -193,6 +195,8 @@ _meson_option_parse() { --disable-auth-pam) printf "%s" -Dauth_pam=3Ddisabled ;; --enable-avx2) printf "%s" -Davx2=3Denabled ;; --disable-avx2) printf "%s" -Davx2=3Ddisabled ;; + --enable-avx512bw) printf "%s" -Davx512bw=3Denabled ;; + --disable-avx512bw) printf "%s" -Davx512bw=3Ddisabled ;; --enable-avx512f) printf "%s" -Davx512f=3Denabled ;; --disable-avx512f) printf "%s" -Davx512f=3Ddisabled ;; --enable-gcov) printf "%s" -Db_coverage=3Dtrue ;; @@ -426,6 +430,8 @@ _meson_option_parse() { --disable-vde) printf "%s" -Dvde=3Ddisabled ;; --enable-vdi) printf "%s" -Dvdi=3Denabled ;; --disable-vdi) printf "%s" -Dvdi=3Ddisabled ;; + --enable-vduse-blk-export) printf "%s" -Dvduse_blk_export=3Denabled ;; + --disable-vduse-blk-export) printf "%s" -Dvduse_blk_export=3Ddisabled = ;; --enable-vfio-user-server) printf "%s" -Dvfio_user_server=3Denabled ;; --disable-vfio-user-server) printf "%s" -Dvfio_user_server=3Ddisabled = ;; --enable-vhost-crypto) printf "%s" -Dvhost_crypto=3Denabled ;; @@ -438,8 +444,6 @@ _meson_option_parse() { --disable-vhost-user) printf "%s" -Dvhost_user=3Ddisabled ;; --enable-vhost-user-blk-server) printf "%s" -Dvhost_user_blk_server=3D= enabled ;; --disable-vhost-user-blk-server) printf "%s" -Dvhost_user_blk_server= =3Ddisabled ;; - --enable-vduse-blk-export) printf "%s" -Dvduse_blk_export=3Denabled ;; - --disable-vduse-blk-export) printf "%s" -Dvduse_blk_export=3Ddisabled = ;; --enable-vhost-vdpa) printf "%s" -Dvhost_vdpa=3Denabled ;; --disable-vhost-vdpa) printf "%s" -Dvhost_vdpa=3Ddisabled ;; --enable-virglrenderer) printf "%s" -Dvirglrenderer=3Denabled ;; --=20 2.38.1