From nobody Wed Apr 9 23:22:12 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=linaro.org ARC-Seal: i=1; a=rsa-sha256; t=1684886418; cv=none; d=zohomail.com; s=zohoarc; b=aia34OHeBRX273V3c/wG3TQeRdAS/j2dwsnIBszqd5KFi+ZmvV6jPgC8uzaqG4l2wpsrAEBJCcSiWllLTkLNG9+MIxaVzXMM6sU2ulO1Rb5v0w/bilbDBibCLeOyF3ZX2BFEy7JaObSk3xqIHOPfvvm3s1xULxqlSzD12KBORWg= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1684886418; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=7ye0/RXUVZWBrvEwXDswgbk/0qm/ligDVrCm5lhuZP0=; b=ZWDWO0FjDQ7ZVWoebiq1Bmhqz3w+uVnlNHCuu9Np55YaFdnx2zuk4b8w5nh3kZBgABs78KTRmX/vUxoAot5c8i3PB5zHCY04Ozlq7Okky/MPdMDG7bgHZTHmh/wUMngsZaUF6QaMw+7D60CjxO4gmWT4yXBlZyqcZhugoRrDVjM= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1684886418164311.06142935780895; Tue, 23 May 2023 17:00:18 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1q1btU-0004BK-OW; Tue, 23 May 2023 19:58:20 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q1btO-00049r-EK for qemu-devel@nongnu.org; Tue, 23 May 2023 19:58:15 -0400 Received: from mail-pf1-x42b.google.com ([2607:f8b0:4864:20::42b]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1q1btM-0001mh-HL for qemu-devel@nongnu.org; Tue, 23 May 2023 19:58:14 -0400 Received: by mail-pf1-x42b.google.com with SMTP id d2e1a72fcca58-64d3491609fso102340b3a.3 for ; Tue, 23 May 2023 16:58:12 -0700 (PDT) Received: from stoup.. ([2602:ae:1598:4c01:c13a:d73:4f88:3654]) by smtp.gmail.com with ESMTPSA id p18-20020aa78612000000b0063b7c42a070sm6285041pfn.68.2023.05.23.16.58.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 23 May 2023 16:58:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1684886291; x=1687478291; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=7ye0/RXUVZWBrvEwXDswgbk/0qm/ligDVrCm5lhuZP0=; b=iEdq+DMv47db3W/6eb6wJDsYOz+GsrBJzgUBX6t/3H0Z8BdUkOqfrdhWaeVzFpRiWe +qrxn1YbIkStcws9d7S5jAh0c0uPA+xz7eSmXvvz7KHZQYA2xzDPaWkKiW7Rst5kivpc Ze9bV1uyLLFw0bp1HYLaf3D2bkshXf07aH4VypfV1LbpIFPf0yQX4vuxaS7jq0QOYoH0 I9ohpZyQkowp2dt7IeMQi0Lws9m0i97O8jWuktxJAVUTgQyDNwdXtYv8w7mBUJiF0B7m zMwZ4krQjtXKL5lM+c0gsC/Bfr64uCYpiCQAhoT/xMxGpz58fT8bk8bWQTMWV+lKhekm MDbg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684886291; x=1687478291; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7ye0/RXUVZWBrvEwXDswgbk/0qm/ligDVrCm5lhuZP0=; b=Clx3jhT/JSGYu4LebTmbF5WAfXWgG5DM00g7RBv56GPDOSVsvt/JWkRvRudhLqhtQB FhnvhgvCwO6+rJ+xNoAwWoKo+BDOAr8oukebrcAtZmN+jZiEujXeUVTMXarMKstRa4jy Y6eBBP9DphO/WQvJoeLBVZoHsdVwYhAbe6yCpn+lXvFKKqe/basACp9vcE2d+cbPfT2w bQd4xVvIHIYcsflFmGWc/gTNzGUMusmrTCoeyahy5ANaCsCP7bTWjoGV2CW1sGAiRelB 1T3sNaYPeH7L/OZPmcbFIQ/5Yar8m5ZrTWguPBIETyWMyOjsgtfC8U82dQOgRIgYe7yf Yd5A== X-Gm-Message-State: AC+VfDy45Uu35x7kccN4ISSEki9Z42QJAtffdoBqiy2N+OqMpnw3FdiN e9oSMwstDQK0VBCtj4r9lvaGzzDOw+lSt5R8F+4= X-Google-Smtp-Source: ACHHUZ5l8z10pfzFMfFyXZMqJ5zDTu4Ugw63P3UyNkmRxspvGKCySle5QTowCoE0GdoUZrIaUbDPtQ== X-Received: by 2002:a05:6a00:22c2:b0:64c:c5f9:1533 with SMTP id f2-20020a056a0022c200b0064cc5f91533mr1048118pfj.33.1684886291261; Tue, 23 May 2023 16:58:11 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Cc: =?UTF-8?q?Alex=20Benn=C3=A9e?= , Juan Quintela Subject: [PULL 06/28] migration/xbzrle: Shuffle function order Date: Tue, 23 May 2023 16:57:42 -0700 Message-Id: <20230523235804.747803-7-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230523235804.747803-1-richard.henderson@linaro.org> References: <20230523235804.747803-1-richard.henderson@linaro.org> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2607:f8b0:4864:20::42b; envelope-from=richard.henderson@linaro.org; helo=mail-pf1-x42b.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @linaro.org) X-ZM-MESSAGEID: 1684886419842100003 Place the CONFIG_AVX512BW_OPT block at the top, which will aid function selection in the next patch. Reviewed-by: Alex Benn=C3=A9e Reviewed-by: Juan Quintela Signed-off-by: Richard Henderson --- migration/xbzrle.c | 244 ++++++++++++++++++++++----------------------- 1 file changed, 122 insertions(+), 122 deletions(-) diff --git a/migration/xbzrle.c b/migration/xbzrle.c index 258e4959c9..751b5428f7 100644 --- a/migration/xbzrle.c +++ b/migration/xbzrle.c @@ -15,6 +15,128 @@ #include "qemu/host-utils.h" #include "xbzrle.h" =20 +#if defined(CONFIG_AVX512BW_OPT) +#include + +int __attribute__((target("avx512bw"))) +xbzrle_encode_buffer_avx512(uint8_t *old_buf, uint8_t *new_buf, int slen, + uint8_t *dst, int dlen) +{ + uint32_t zrun_len =3D 0, nzrun_len =3D 0; + int d =3D 0, i =3D 0, num =3D 0; + uint8_t *nzrun_start =3D NULL; + /* add 1 to include residual part in main loop */ + uint32_t count512s =3D (slen >> 6) + 1; + /* countResidual is tail of data, i.e., countResidual =3D slen % 64 */ + uint32_t count_residual =3D slen & 0b111111; + bool never_same =3D true; + uint64_t mask_residual =3D 1; + mask_residual <<=3D count_residual; + mask_residual -=3D 1; + __m512i r =3D _mm512_set1_epi32(0); + + while (count512s) { + int bytes_to_check =3D 64; + uint64_t mask =3D 0xffffffffffffffff; + if (count512s =3D=3D 1) { + bytes_to_check =3D count_residual; + mask =3D mask_residual; + } + __m512i old_data =3D _mm512_mask_loadu_epi8(r, + mask, old_buf + i); + __m512i new_data =3D _mm512_mask_loadu_epi8(r, + mask, new_buf + i); + uint64_t comp =3D _mm512_cmpeq_epi8_mask(old_data, new_data); + count512s--; + + bool is_same =3D (comp & 0x1); + while (bytes_to_check) { + if (d + 2 > dlen) { + return -1; + } + if (is_same) { + if (nzrun_len) { + d +=3D uleb128_encode_small(dst + d, nzrun_len); + if (d + nzrun_len > dlen) { + return -1; + } + nzrun_start =3D new_buf + i - nzrun_len; + memcpy(dst + d, nzrun_start, nzrun_len); + d +=3D nzrun_len; + nzrun_len =3D 0; + } + /* 64 data at a time for speed */ + if (count512s && (comp =3D=3D 0xffffffffffffffff)) { + i +=3D 64; + zrun_len +=3D 64; + break; + } + never_same =3D false; + num =3D ctz64(~comp); + num =3D (num < bytes_to_check) ? num : bytes_to_check; + zrun_len +=3D num; + bytes_to_check -=3D num; + comp >>=3D num; + i +=3D num; + if (bytes_to_check) { + /* still has different data after same data */ + d +=3D uleb128_encode_small(dst + d, zrun_len); + zrun_len =3D 0; + } else { + break; + } + } + if (never_same || zrun_len) { + /* + * never_same only acts if + * data begins with diff in first count512s + */ + d +=3D uleb128_encode_small(dst + d, zrun_len); + zrun_len =3D 0; + never_same =3D false; + } + /* has diff, 64 data at a time for speed */ + if ((bytes_to_check =3D=3D 64) && (comp =3D=3D 0x0)) { + i +=3D 64; + nzrun_len +=3D 64; + break; + } + num =3D ctz64(comp); + num =3D (num < bytes_to_check) ? num : bytes_to_check; + nzrun_len +=3D num; + bytes_to_check -=3D num; + comp >>=3D num; + i +=3D num; + if (bytes_to_check) { + /* mask like 111000 */ + d +=3D uleb128_encode_small(dst + d, nzrun_len); + /* overflow */ + if (d + nzrun_len > dlen) { + return -1; + } + nzrun_start =3D new_buf + i - nzrun_len; + memcpy(dst + d, nzrun_start, nzrun_len); + d +=3D nzrun_len; + nzrun_len =3D 0; + is_same =3D true; + } + } + } + + if (nzrun_len !=3D 0) { + d +=3D uleb128_encode_small(dst + d, nzrun_len); + /* overflow */ + if (d + nzrun_len > dlen) { + return -1; + } + nzrun_start =3D new_buf + i - nzrun_len; + memcpy(dst + d, nzrun_start, nzrun_len); + d +=3D nzrun_len; + } + return d; +} +#endif + /* page =3D zrun nzrun | zrun nzrun page @@ -175,125 +297,3 @@ int xbzrle_decode_buffer(uint8_t *src, int slen, uint= 8_t *dst, int dlen) =20 return d; } - -#if defined(CONFIG_AVX512BW_OPT) -#include - -int __attribute__((target("avx512bw"))) -xbzrle_encode_buffer_avx512(uint8_t *old_buf, uint8_t *new_buf, int slen, - uint8_t *dst, int dlen) -{ - uint32_t zrun_len =3D 0, nzrun_len =3D 0; - int d =3D 0, i =3D 0, num =3D 0; - uint8_t *nzrun_start =3D NULL; - /* add 1 to include residual part in main loop */ - uint32_t count512s =3D (slen >> 6) + 1; - /* countResidual is tail of data, i.e., countResidual =3D slen % 64 */ - uint32_t count_residual =3D slen & 0b111111; - bool never_same =3D true; - uint64_t mask_residual =3D 1; - mask_residual <<=3D count_residual; - mask_residual -=3D 1; - __m512i r =3D _mm512_set1_epi32(0); - - while (count512s) { - int bytes_to_check =3D 64; - uint64_t mask =3D 0xffffffffffffffff; - if (count512s =3D=3D 1) { - bytes_to_check =3D count_residual; - mask =3D mask_residual; - } - __m512i old_data =3D _mm512_mask_loadu_epi8(r, - mask, old_buf + i); - __m512i new_data =3D _mm512_mask_loadu_epi8(r, - mask, new_buf + i); - uint64_t comp =3D _mm512_cmpeq_epi8_mask(old_data, new_data); - count512s--; - - bool is_same =3D (comp & 0x1); - while (bytes_to_check) { - if (d + 2 > dlen) { - return -1; - } - if (is_same) { - if (nzrun_len) { - d +=3D uleb128_encode_small(dst + d, nzrun_len); - if (d + nzrun_len > dlen) { - return -1; - } - nzrun_start =3D new_buf + i - nzrun_len; - memcpy(dst + d, nzrun_start, nzrun_len); - d +=3D nzrun_len; - nzrun_len =3D 0; - } - /* 64 data at a time for speed */ - if (count512s && (comp =3D=3D 0xffffffffffffffff)) { - i +=3D 64; - zrun_len +=3D 64; - break; - } - never_same =3D false; - num =3D ctz64(~comp); - num =3D (num < bytes_to_check) ? num : bytes_to_check; - zrun_len +=3D num; - bytes_to_check -=3D num; - comp >>=3D num; - i +=3D num; - if (bytes_to_check) { - /* still has different data after same data */ - d +=3D uleb128_encode_small(dst + d, zrun_len); - zrun_len =3D 0; - } else { - break; - } - } - if (never_same || zrun_len) { - /* - * never_same only acts if - * data begins with diff in first count512s - */ - d +=3D uleb128_encode_small(dst + d, zrun_len); - zrun_len =3D 0; - never_same =3D false; - } - /* has diff, 64 data at a time for speed */ - if ((bytes_to_check =3D=3D 64) && (comp =3D=3D 0x0)) { - i +=3D 64; - nzrun_len +=3D 64; - break; - } - num =3D ctz64(comp); - num =3D (num < bytes_to_check) ? num : bytes_to_check; - nzrun_len +=3D num; - bytes_to_check -=3D num; - comp >>=3D num; - i +=3D num; - if (bytes_to_check) { - /* mask like 111000 */ - d +=3D uleb128_encode_small(dst + d, nzrun_len); - /* overflow */ - if (d + nzrun_len > dlen) { - return -1; - } - nzrun_start =3D new_buf + i - nzrun_len; - memcpy(dst + d, nzrun_start, nzrun_len); - d +=3D nzrun_len; - nzrun_len =3D 0; - is_same =3D true; - } - } - } - - if (nzrun_len !=3D 0) { - d +=3D uleb128_encode_small(dst + d, nzrun_len); - /* overflow */ - if (d + nzrun_len > dlen) { - return -1; - } - nzrun_start =3D new_buf + i - nzrun_len; - memcpy(dst + d, nzrun_start, nzrun_len); - d +=3D nzrun_len; - } - return d; -} -#endif --=20 2.34.1