From nobody Tue Feb 10 07:44:13 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=linaro.org ARC-Seal: i=1; a=rsa-sha256; t=1621908916; cv=none; d=zohomail.com; s=zohoarc; b=fXCXv/YjV2sF+4qwjMZbQrZaTZlLCs5/1Fr9YmZ4PhzCM55lfuXJlO7FaNGIRGtnmRXB0XQoFJk5a01/5nndxkT+Ur9T9nMTdTQv/YxE5EUPcyZQsVrcZTYLZkcTGh9SYXWhp1lDWWPiVPkOesDgKsO3wQZmBzrKff7YW0OYo24= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1621908916; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=TSJhIg/oEj78OAJlzsUKkaERruGiMdx8U7Tjdyc3i70=; b=h8TTMltm2WoAGbxZCf2FVYTiQHsJBEyWcPLTCbKmNzOcVQvlwVoiTTkZOldHxHwbX39pLX9TetcOR8U4IkC2LaFUgQb3KQ7NGHuS33c/AlTC5u4nKhKVIKqshitVzSXo2FHSLWPjy5fe+w3YL+FkriQ/EVnG1Xk9+JINeO6bpgE= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1621908916030607.5327407764782; Mon, 24 May 2021 19:15:16 -0700 (PDT) Received: from localhost ([::1]:57616 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1llMbC-0005zH-S9 for importer@patchew.org; Mon, 24 May 2021 22:15:14 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:55600) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1llLYT-0000pO-7E for qemu-devel@nongnu.org; Mon, 24 May 2021 21:08:21 -0400 Received: from mail-pg1-x52c.google.com ([2607:f8b0:4864:20::52c]:33448) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1llLXs-000460-8d for qemu-devel@nongnu.org; Mon, 24 May 2021 21:08:20 -0400 Received: by mail-pg1-x52c.google.com with SMTP id i5so21439354pgm.0 for ; Mon, 24 May 2021 18:07:40 -0700 (PDT) Received: from localhost.localdomain (174-21-70-228.tukw.qwest.net. [174.21.70.228]) by smtp.gmail.com with ESMTPSA id b16sm11748176pju.35.2021.05.24.18.07.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 24 May 2021 18:07:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=TSJhIg/oEj78OAJlzsUKkaERruGiMdx8U7Tjdyc3i70=; b=UpeH7HGx3DEoqpbxxdl1btOB4yS91c6cBOZqmH9T67vPVoRYtEG2lk0stVvr4eHB4R A3RUnM8VcuCwo0Y4M+C0aBSr7cvIWWjPBiPsyCYHjexbcFmoNJvKC0tJAkIVAxAxGKk7 Mh6jTMflxELLGyMjqr4GqWso3tJTIpABwhvbVRRmXXD9FkiQY/GPJP7ZVpt3l+BWKo2l 3/Prs35f38DYZlFWHKliSxB+FKtXNmOWDb6zKTR9aJGIiH4N3L5UHPtGSFYau8IZDher jUSkLQa4GbIopGRbhPx63BgAIDrRiqd39J8O9ntisURDsnwhmVYbH7SYJixgOdd2gTcU /dwg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=TSJhIg/oEj78OAJlzsUKkaERruGiMdx8U7Tjdyc3i70=; b=WpCc/GJNq9MFaqRZHy3vZuCHrclC+CjZL6JYwsQk30YwoHZMX5O7O1R00HrxfT/I9r 2IpfKe/tvdf7f5WWMFTdLaXj7bn3LPcuQnceMZWfa58THsiYwQtrkT+4DLlZCIdw7I9t aDmuR0CNuhPmVRV2Aj8QKe6CjGhd3VY36uCJB2fXFlqhwDywR0rFcrCnuc6ahVhEdrw/ J1tewg6AEURLUl82p+oiJWLwOmfL2sJ+fLU0ux2zL8/i1Ubd2bCl2N0MlhwUe+kJolWL LKIp61ZWReTAvFOBWsdWbmtoKqMtH9DUVO/74MLZcgBODrzP875X4wsqMwh7LY0xVb/M M5/Q== X-Gm-Message-State: AOAM532gZAaU49/4Ky6EukhJSvnf8J3yBi5VQLKFcgHm0V3Tb9xuaHNn mIA2wQMzk/XNDanX2vT46Vkj9Coj54juQg== X-Google-Smtp-Source: ABdhPJwz8Q1tz2CWh8KBmfj/csEz1MEWcsiN2bzrXQh3ice4lRZKDKX0YbzbrJUXHQDmhy+CPp0c2Q== X-Received: by 2002:a63:5d04:: with SMTP id r4mr16266249pgb.178.1621904859504; Mon, 24 May 2021 18:07:39 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH v7 65/92] target/arm: Macroize helper_gvec_{s,u}dot_idx_{b,h} Date: Mon, 24 May 2021 18:03:31 -0700 Message-Id: <20210525010358.152808-66-richard.henderson@linaro.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210525010358.152808-1-richard.henderson@linaro.org> References: <20210525010358.152808-1-richard.henderson@linaro.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2607:f8b0:4864:20::52c; envelope-from=richard.henderson@linaro.org; helo=mail-pg1-x52c.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: qemu-arm@nongnu.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @linaro.org) Content-Type: text/plain; charset="utf-8" We're about to add more variations on this theme. Accept the inner loop for the _h variants, rather than keep it unrolled. Signed-off-by: Richard Henderson --- target/arm/vec_helper.c | 160 ++++++++-------------------------------- 1 file changed, 29 insertions(+), 131 deletions(-) diff --git a/target/arm/vec_helper.c b/target/arm/vec_helper.c index cddf095c74..1c4266a9c0 100644 --- a/target/arm/vec_helper.c +++ b/target/arm/vec_helper.c @@ -569,139 +569,37 @@ DO_DOT(gvec_udot_b, uint32_t, uint8_t, uint8_t) DO_DOT(gvec_sdot_h, int64_t, int16_t, int16_t) DO_DOT(gvec_udot_h, uint64_t, uint16_t, uint16_t) =20 -void HELPER(gvec_sdot_idx_b)(void *vd, void *vn, void *vm, - void *va, uint32_t desc) -{ - intptr_t i, segend, opr_sz =3D simd_oprsz(desc), opr_sz_4 =3D opr_sz /= 4; - intptr_t index =3D simd_data(desc); - int32_t *d =3D vd, *a =3D va; - int8_t *n =3D vn; - int8_t *m_indexed =3D (int8_t *)vm + H4(index) * 4; - - /* Notice the special case of opr_sz =3D=3D 8, from aa64/aa32 advsimd. - * Otherwise opr_sz is a multiple of 16. - */ - segend =3D MIN(4, opr_sz_4); - i =3D 0; - do { - int8_t m0 =3D m_indexed[i * 4 + 0]; - int8_t m1 =3D m_indexed[i * 4 + 1]; - int8_t m2 =3D m_indexed[i * 4 + 2]; - int8_t m3 =3D m_indexed[i * 4 + 3]; - - do { - d[i] =3D (a[i] + - n[i * 4 + 0] * m0 + - n[i * 4 + 1] * m1 + - n[i * 4 + 2] * m2 + - n[i * 4 + 3] * m3); - } while (++i < segend); - segend =3D i + 4; - } while (i < opr_sz_4); - - clear_tail(d, opr_sz, simd_maxsz(desc)); +#define DO_DOT_IDX(NAME, TYPED, TYPEN, TYPEM, HD) \ +void HELPER(NAME)(void *vd, void *vn, void *vm, void *va, uint32_t desc) \ +{ \ + intptr_t i =3D 0, opr_sz =3D simd_oprsz(desc); = \ + intptr_t opr_sz_n =3D opr_sz / sizeof(TYPED); = \ + intptr_t segend =3D MIN(16 / sizeof(TYPED), opr_sz_n); = \ + intptr_t index =3D simd_data(desc); = \ + TYPED *d =3D vd, *a =3D va; = \ + TYPEN *n =3D vn; = \ + TYPEM *m_indexed =3D (TYPEM *)vm + HD(index) * 4; = \ + do { \ + TYPED m0 =3D m_indexed[i * 4 + 0]; = \ + TYPED m1 =3D m_indexed[i * 4 + 1]; = \ + TYPED m2 =3D m_indexed[i * 4 + 2]; = \ + TYPED m3 =3D m_indexed[i * 4 + 3]; = \ + do { \ + d[i] =3D (a[i] + = \ + n[i * 4 + 0] * m0 + \ + n[i * 4 + 1] * m1 + \ + n[i * 4 + 2] * m2 + \ + n[i * 4 + 3] * m3); \ + } while (++i < segend); \ + segend =3D i + 4; = \ + } while (i < opr_sz_n); \ + clear_tail(d, opr_sz, simd_maxsz(desc)); \ } =20 -void HELPER(gvec_udot_idx_b)(void *vd, void *vn, void *vm, - void *va, uint32_t desc) -{ - intptr_t i, segend, opr_sz =3D simd_oprsz(desc), opr_sz_4 =3D opr_sz /= 4; - intptr_t index =3D simd_data(desc); - uint32_t *d =3D vd, *a =3D va; - uint8_t *n =3D vn; - uint8_t *m_indexed =3D (uint8_t *)vm + H4(index) * 4; - - /* Notice the special case of opr_sz =3D=3D 8, from aa64/aa32 advsimd. - * Otherwise opr_sz is a multiple of 16. - */ - segend =3D MIN(4, opr_sz_4); - i =3D 0; - do { - uint8_t m0 =3D m_indexed[i * 4 + 0]; - uint8_t m1 =3D m_indexed[i * 4 + 1]; - uint8_t m2 =3D m_indexed[i * 4 + 2]; - uint8_t m3 =3D m_indexed[i * 4 + 3]; - - do { - d[i] =3D (a[i] + - n[i * 4 + 0] * m0 + - n[i * 4 + 1] * m1 + - n[i * 4 + 2] * m2 + - n[i * 4 + 3] * m3); - } while (++i < segend); - segend =3D i + 4; - } while (i < opr_sz_4); - - clear_tail(d, opr_sz, simd_maxsz(desc)); -} - -void HELPER(gvec_sdot_idx_h)(void *vd, void *vn, void *vm, - void *va, uint32_t desc) -{ - intptr_t i, opr_sz =3D simd_oprsz(desc), opr_sz_8 =3D opr_sz / 8; - intptr_t index =3D simd_data(desc); - int64_t *d =3D vd, *a =3D va; - int16_t *n =3D vn; - int16_t *m_indexed =3D (int16_t *)vm + index * 4; - - /* This is supported by SVE only, so opr_sz is always a multiple of 16. - * Process the entire segment all at once, writing back the results - * only after we've consumed all of the inputs. - */ - for (i =3D 0; i < opr_sz_8; i +=3D 2) { - int64_t d0, d1; - - d0 =3D a[i + 0]; - d0 +=3D n[i * 4 + 0] * (int64_t)m_indexed[i * 4 + 0]; - d0 +=3D n[i * 4 + 1] * (int64_t)m_indexed[i * 4 + 1]; - d0 +=3D n[i * 4 + 2] * (int64_t)m_indexed[i * 4 + 2]; - d0 +=3D n[i * 4 + 3] * (int64_t)m_indexed[i * 4 + 3]; - - d1 =3D a[i + 1]; - d1 +=3D n[i * 4 + 4] * (int64_t)m_indexed[i * 4 + 0]; - d1 +=3D n[i * 4 + 5] * (int64_t)m_indexed[i * 4 + 1]; - d1 +=3D n[i * 4 + 6] * (int64_t)m_indexed[i * 4 + 2]; - d1 +=3D n[i * 4 + 7] * (int64_t)m_indexed[i * 4 + 3]; - - d[i + 0] =3D d0; - d[i + 1] =3D d1; - } - clear_tail(d, opr_sz, simd_maxsz(desc)); -} - -void HELPER(gvec_udot_idx_h)(void *vd, void *vn, void *vm, - void *va, uint32_t desc) -{ - intptr_t i, opr_sz =3D simd_oprsz(desc), opr_sz_8 =3D opr_sz / 8; - intptr_t index =3D simd_data(desc); - uint64_t *d =3D vd, *a =3D va; - uint16_t *n =3D vn; - uint16_t *m_indexed =3D (uint16_t *)vm + index * 4; - - /* This is supported by SVE only, so opr_sz is always a multiple of 16. - * Process the entire segment all at once, writing back the results - * only after we've consumed all of the inputs. - */ - for (i =3D 0; i < opr_sz_8; i +=3D 2) { - uint64_t d0, d1; - - d0 =3D a[i + 0]; - d0 +=3D n[i * 4 + 0] * (uint64_t)m_indexed[i * 4 + 0]; - d0 +=3D n[i * 4 + 1] * (uint64_t)m_indexed[i * 4 + 1]; - d0 +=3D n[i * 4 + 2] * (uint64_t)m_indexed[i * 4 + 2]; - d0 +=3D n[i * 4 + 3] * (uint64_t)m_indexed[i * 4 + 3]; - - d1 =3D a[i + 1]; - d1 +=3D n[i * 4 + 4] * (uint64_t)m_indexed[i * 4 + 0]; - d1 +=3D n[i * 4 + 5] * (uint64_t)m_indexed[i * 4 + 1]; - d1 +=3D n[i * 4 + 6] * (uint64_t)m_indexed[i * 4 + 2]; - d1 +=3D n[i * 4 + 7] * (uint64_t)m_indexed[i * 4 + 3]; - - d[i + 0] =3D d0; - d[i + 1] =3D d1; - } - clear_tail(d, opr_sz, simd_maxsz(desc)); -} +DO_DOT_IDX(gvec_sdot_idx_b, int32_t, int8_t, int8_t, H4) +DO_DOT_IDX(gvec_udot_idx_b, uint32_t, uint8_t, uint8_t, H4) +DO_DOT_IDX(gvec_sdot_idx_h, int64_t, int16_t, int16_t, ) +DO_DOT_IDX(gvec_udot_idx_h, uint64_t, uint16_t, uint16_t, ) =20 void HELPER(gvec_fcaddh)(void *vd, void *vn, void *vm, void *vfpst, uint32_t desc) --=20 2.25.1