From nobody Wed Feb 11 03:08:08 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1529546575154272.2140548191461; Wed, 20 Jun 2018 19:02:55 -0700 (PDT) Received: from localhost ([::1]:52521 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fVovx-0001pG-32 for importer@patchew.org; Wed, 20 Jun 2018 22:02:49 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:39184) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fVonl-00045Z-Kh for qemu-devel@nongnu.org; Wed, 20 Jun 2018 21:54:23 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fVonk-000321-4W for qemu-devel@nongnu.org; Wed, 20 Jun 2018 21:54:21 -0400 Received: from mail-pf0-x234.google.com ([2607:f8b0:400e:c00::234]:38383) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fVonj-00031i-RW for qemu-devel@nongnu.org; Wed, 20 Jun 2018 21:54:20 -0400 Received: by mail-pf0-x234.google.com with SMTP id b74-v6so688247pfl.5 for ; Wed, 20 Jun 2018 18:54:19 -0700 (PDT) Received: from cloudburst.twiddle.net (mta-98-147-121-51.hawaii.rr.com. [98.147.121.51]) by smtp.gmail.com with ESMTPSA id a27-v6sm6187946pfc.18.2018.06.20.18.54.16 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 20 Jun 2018 18:54:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=jINFLC02lMv+InyN1iespI0Zzx3CuWqfLW8uGBHW3LI=; b=Iu7CDaR5A0qYg9rvkRicFbrbTGtKK6l0qfEy4W4Wjx8XmZwj4BJr2tdbB/t5sst889 vtd/5KvWw6ae3eDb4rZqM4qG/hQboQDijbyQZOcH8itVczk8NidmA+XXHU5uCdIUJWxS JmEyy6qTc0Up6v9X39STi0HEzeKWciaucAbAE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=jINFLC02lMv+InyN1iespI0Zzx3CuWqfLW8uGBHW3LI=; b=TXeSHKRHkzjxGV8dcaqEjriYD6UZKaUqdC/GlV7aNp7BNyw2JxXVOn1Xl3cSBLG243 NkMm9tbZJUP37VKAiWSlE1z9TFm8Uwsz5jABcm5g2pWRVyaAomr/PghXT4w9+y9i95Tw n8+IkfmkuFk2fVk903whWlAyNUK/hisXPVuTj+EM2kr8Pqi8yQ99X+wSQwORY5Z8CPBE 4c+hRlCSvD4fsyv/ya9slbgkoHnyyL1l+Fbbave+rUqpOd0JNPxo+MK8bLf/sjiYHYrP 0QJRfdevGTaNKnzmeYx9pHd2dj5PyNnbw46wrT9v5PA9FGxZxfCMOe0kzRQ/E8fJJNmx uFbA== X-Gm-Message-State: APt69E3a0t5yjyBRrIKou+xoCytecylOrh7zlwAUEBLEV6rSxVgKKQ2P ju5Hgs8U4O3H8t23mSgw8gq+V6MXeQk= X-Google-Smtp-Source: ADUXVKKFcaeoXv2QjNj/AHOslMMhZ6KjImvlfNYgGeC5Ja7cZaiW6RdquCXw2kyLyHS9how8Iukvdg== X-Received: by 2002:a65:48c9:: with SMTP id o9-v6mr21085398pgs.262.1529546058526; Wed, 20 Jun 2018 18:54:18 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Date: Wed, 20 Jun 2018 15:53:31 -1000 Message-Id: <20180621015359.12018-8-richard.henderson@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180621015359.12018-1-richard.henderson@linaro.org> References: <20180621015359.12018-1-richard.henderson@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c00::234 Subject: [Qemu-devel] [PATCH v5 07/35] target/arm: Implement SVE FP Multiply-Add Group X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Signed-off-by: Richard Henderson --- target/arm/helper-sve.h | 16 ++++ target/arm/sve_helper.c | 158 +++++++++++++++++++++++++++++++++++++ target/arm/translate-sve.c | 49 ++++++++++++ target/arm/sve.decode | 17 ++++ 4 files changed, 240 insertions(+) diff --git a/target/arm/helper-sve.h b/target/arm/helper-sve.h index 4097b55f0e..eb0645dd43 100644 --- a/target/arm/helper-sve.h +++ b/target/arm/helper-sve.h @@ -827,6 +827,22 @@ DEF_HELPER_FLAGS_5(sve_ucvt_ds, TCG_CALL_NO_RWG, DEF_HELPER_FLAGS_5(sve_ucvt_dd, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, ptr, i32) =20 +DEF_HELPER_FLAGS_3(sve_fmla_zpzzz_h, TCG_CALL_NO_RWG, void, env, ptr, i32) +DEF_HELPER_FLAGS_3(sve_fmla_zpzzz_s, TCG_CALL_NO_RWG, void, env, ptr, i32) +DEF_HELPER_FLAGS_3(sve_fmla_zpzzz_d, TCG_CALL_NO_RWG, void, env, ptr, i32) + +DEF_HELPER_FLAGS_3(sve_fmls_zpzzz_h, TCG_CALL_NO_RWG, void, env, ptr, i32) +DEF_HELPER_FLAGS_3(sve_fmls_zpzzz_s, TCG_CALL_NO_RWG, void, env, ptr, i32) +DEF_HELPER_FLAGS_3(sve_fmls_zpzzz_d, TCG_CALL_NO_RWG, void, env, ptr, i32) + +DEF_HELPER_FLAGS_3(sve_fnmla_zpzzz_h, TCG_CALL_NO_RWG, void, env, ptr, i32) +DEF_HELPER_FLAGS_3(sve_fnmla_zpzzz_s, TCG_CALL_NO_RWG, void, env, ptr, i32) +DEF_HELPER_FLAGS_3(sve_fnmla_zpzzz_d, TCG_CALL_NO_RWG, void, env, ptr, i32) + +DEF_HELPER_FLAGS_3(sve_fnmls_zpzzz_h, TCG_CALL_NO_RWG, void, env, ptr, i32) +DEF_HELPER_FLAGS_3(sve_fnmls_zpzzz_s, TCG_CALL_NO_RWG, void, env, ptr, i32) +DEF_HELPER_FLAGS_3(sve_fnmls_zpzzz_d, TCG_CALL_NO_RWG, void, env, ptr, i32) + DEF_HELPER_FLAGS_4(sve_ld1bb_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) DEF_HELPER_FLAGS_4(sve_ld2bb_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) DEF_HELPER_FLAGS_4(sve_ld3bb_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index 6bf30a3e66..aeb4ccadd9 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -2938,6 +2938,164 @@ DO_ZPZ_FP(sve_ucvt_dd, uint64_t, , uint64_to_fl= oat64) =20 #undef DO_ZPZ_FP =20 +/* 4-operand predicated multiply-add. This requires 7 operands to pass + * "properly", so we need to encode some of the registers into DESC. + */ +QEMU_BUILD_BUG_ON(SIMD_DATA_SHIFT + 20 > 32); + +static void do_fmla_zpzzz_h(CPUARMState *env, void *vg, uint32_t desc, + uint16_t neg1, uint16_t neg3) +{ + intptr_t i =3D simd_oprsz(desc); + unsigned rd =3D extract32(desc, SIMD_DATA_SHIFT, 5); + unsigned rn =3D extract32(desc, SIMD_DATA_SHIFT + 5, 5); + unsigned rm =3D extract32(desc, SIMD_DATA_SHIFT + 10, 5); + unsigned ra =3D extract32(desc, SIMD_DATA_SHIFT + 15, 5); + void *vd =3D &env->vfp.zregs[rd]; + void *vn =3D &env->vfp.zregs[rn]; + void *vm =3D &env->vfp.zregs[rm]; + void *va =3D &env->vfp.zregs[ra]; + uint64_t *g =3D vg; + + do { + uint64_t pg =3D g[(i - 1) >> 6]; + do { + i -=3D 2; + if (likely((pg >> (i & 63)) & 1)) { + float16 e1, e2, e3, r; + + e1 =3D *(uint16_t *)(vn + H1_2(i)) ^ neg1; + e2 =3D *(uint16_t *)(vm + H1_2(i)); + e3 =3D *(uint16_t *)(va + H1_2(i)) ^ neg3; + r =3D float16_muladd(e1, e2, e3, 0, &env->vfp.fp_status); + *(uint16_t *)(vd + H1_2(i)) =3D r; + } + } while (i & 63); + } while (i !=3D 0); +} + +void HELPER(sve_fmla_zpzzz_h)(CPUARMState *env, void *vg, uint32_t desc) +{ + do_fmla_zpzzz_h(env, vg, desc, 0, 0); +} + +void HELPER(sve_fmls_zpzzz_h)(CPUARMState *env, void *vg, uint32_t desc) +{ + do_fmla_zpzzz_h(env, vg, desc, 0x8000, 0); +} + +void HELPER(sve_fnmla_zpzzz_h)(CPUARMState *env, void *vg, uint32_t desc) +{ + do_fmla_zpzzz_h(env, vg, desc, 0x8000, 0x8000); +} + +void HELPER(sve_fnmls_zpzzz_h)(CPUARMState *env, void *vg, uint32_t desc) +{ + do_fmla_zpzzz_h(env, vg, desc, 0, 0x8000); +} + +static void do_fmla_zpzzz_s(CPUARMState *env, void *vg, uint32_t desc, + uint32_t neg1, uint32_t neg3) +{ + intptr_t i =3D simd_oprsz(desc); + unsigned rd =3D extract32(desc, SIMD_DATA_SHIFT, 5); + unsigned rn =3D extract32(desc, SIMD_DATA_SHIFT + 5, 5); + unsigned rm =3D extract32(desc, SIMD_DATA_SHIFT + 10, 5); + unsigned ra =3D extract32(desc, SIMD_DATA_SHIFT + 15, 5); + void *vd =3D &env->vfp.zregs[rd]; + void *vn =3D &env->vfp.zregs[rn]; + void *vm =3D &env->vfp.zregs[rm]; + void *va =3D &env->vfp.zregs[ra]; + uint64_t *g =3D vg; + + do { + uint64_t pg =3D g[(i - 1) >> 6]; + do { + i -=3D 4; + if (likely((pg >> (i & 63)) & 1)) { + float32 e1, e2, e3, r; + + e1 =3D *(uint32_t *)(vn + H1_4(i)) ^ neg1; + e2 =3D *(uint32_t *)(vm + H1_4(i)); + e3 =3D *(uint32_t *)(va + H1_4(i)) ^ neg3; + r =3D float32_muladd(e1, e2, e3, 0, &env->vfp.fp_status); + *(uint32_t *)(vd + H1_4(i)) =3D r; + } + } while (i & 63); + } while (i !=3D 0); +} + +void HELPER(sve_fmla_zpzzz_s)(CPUARMState *env, void *vg, uint32_t desc) +{ + do_fmla_zpzzz_s(env, vg, desc, 0, 0); +} + +void HELPER(sve_fmls_zpzzz_s)(CPUARMState *env, void *vg, uint32_t desc) +{ + do_fmla_zpzzz_s(env, vg, desc, 0x80000000, 0); +} + +void HELPER(sve_fnmla_zpzzz_s)(CPUARMState *env, void *vg, uint32_t desc) +{ + do_fmla_zpzzz_s(env, vg, desc, 0x80000000, 0x80000000); +} + +void HELPER(sve_fnmls_zpzzz_s)(CPUARMState *env, void *vg, uint32_t desc) +{ + do_fmla_zpzzz_s(env, vg, desc, 0, 0x80000000); +} + +static void do_fmla_zpzzz_d(CPUARMState *env, void *vg, uint32_t desc, + uint64_t neg1, uint64_t neg3) +{ + intptr_t i =3D simd_oprsz(desc); + unsigned rd =3D extract32(desc, SIMD_DATA_SHIFT, 5); + unsigned rn =3D extract32(desc, SIMD_DATA_SHIFT + 5, 5); + unsigned rm =3D extract32(desc, SIMD_DATA_SHIFT + 10, 5); + unsigned ra =3D extract32(desc, SIMD_DATA_SHIFT + 15, 5); + void *vd =3D &env->vfp.zregs[rd]; + void *vn =3D &env->vfp.zregs[rn]; + void *vm =3D &env->vfp.zregs[rm]; + void *va =3D &env->vfp.zregs[ra]; + uint64_t *g =3D vg; + + do { + uint64_t pg =3D g[(i - 1) >> 6]; + do { + i -=3D 8; + if (likely((pg >> (i & 63)) & 1)) { + float64 e1, e2, e3, r; + + e1 =3D *(uint64_t *)(vn + i) ^ neg1; + e2 =3D *(uint64_t *)(vm + i); + e3 =3D *(uint64_t *)(va + i) ^ neg3; + r =3D float64_muladd(e1, e2, e3, 0, &env->vfp.fp_status); + *(uint64_t *)(vd + i) =3D r; + } + } while (i & 63); + } while (i !=3D 0); +} + +void HELPER(sve_fmla_zpzzz_d)(CPUARMState *env, void *vg, uint32_t desc) +{ + do_fmla_zpzzz_d(env, vg, desc, 0, 0); +} + +void HELPER(sve_fmls_zpzzz_d)(CPUARMState *env, void *vg, uint32_t desc) +{ + do_fmla_zpzzz_d(env, vg, desc, INT64_MIN, 0); +} + +void HELPER(sve_fnmla_zpzzz_d)(CPUARMState *env, void *vg, uint32_t desc) +{ + do_fmla_zpzzz_d(env, vg, desc, INT64_MIN, INT64_MIN); +} + +void HELPER(sve_fnmls_zpzzz_d)(CPUARMState *env, void *vg, uint32_t desc) +{ + do_fmla_zpzzz_d(env, vg, desc, 0, INT64_MIN); +} + /* * Load contiguous data, protected by a governing predicate. */ diff --git a/target/arm/translate-sve.c b/target/arm/translate-sve.c index 4df5360da9..acad6374ef 100644 --- a/target/arm/translate-sve.c +++ b/target/arm/translate-sve.c @@ -3472,6 +3472,55 @@ DO_FP3(FMULX, fmulx) =20 #undef DO_FP3 =20 +typedef void gen_helper_sve_fmla(TCGv_env, TCGv_ptr, TCGv_i32); + +static bool do_fmla(DisasContext *s, arg_rprrr_esz *a, gen_helper_sve_fmla= *fn) +{ + if (fn =3D=3D NULL) { + return false; + } + if (!sve_access_check(s)) { + return true; + } + + unsigned vsz =3D vec_full_reg_size(s); + unsigned desc; + TCGv_i32 t_desc; + TCGv_ptr pg =3D tcg_temp_new_ptr(); + + /* We would need 7 operands to pass these arguments "properly". + * So we encode all the register numbers into the descriptor. + */ + desc =3D deposit32(a->rd, 5, 5, a->rn); + desc =3D deposit32(desc, 10, 5, a->rm); + desc =3D deposit32(desc, 15, 5, a->ra); + desc =3D simd_desc(vsz, vsz, desc); + + t_desc =3D tcg_const_i32(desc); + tcg_gen_addi_ptr(pg, cpu_env, pred_full_reg_offset(s, a->pg)); + fn(cpu_env, pg, t_desc); + tcg_temp_free_i32(t_desc); + tcg_temp_free_ptr(pg); + return true; +} + +#define DO_FMLA(NAME, name) \ +static bool trans_##NAME(DisasContext *s, arg_rprrr_esz *a, uint32_t insn)= \ +{ \ + static gen_helper_sve_fmla * const fns[4] =3D { \ + NULL, gen_helper_sve_##name##_h, \ + gen_helper_sve_##name##_s, gen_helper_sve_##name##_d \ + }; \ + return do_fmla(s, a, fns[a->esz]); \ +} + +DO_FMLA(FMLA_zpzzz, fmla_zpzzz) +DO_FMLA(FMLS_zpzzz, fmls_zpzzz) +DO_FMLA(FNMLA_zpzzz, fnmla_zpzzz) +DO_FMLA(FNMLS_zpzzz, fnmls_zpzzz) + +#undef DO_FMLA + /* *** SVE Floating Point Unary Operations Prediated Group */ diff --git a/target/arm/sve.decode b/target/arm/sve.decode index 636212a638..70e5a3aeb5 100644 --- a/target/arm/sve.decode +++ b/target/arm/sve.decode @@ -128,6 +128,8 @@ &rprrr_esz ra=3D%reg_movprfx @rdn_pg_ra_rm ........ esz:2 . rm:5 ... pg:3 ra:5 rd:5 \ &rprrr_esz rn=3D%reg_movprfx +@rdn_pg_rm_ra ........ esz:2 . ra:5 ... pg:3 rm:5 rd:5 \ + &rprrr_esz rn=3D%reg_movprfx =20 # One register operand, with governing predicate, vector element size @rd_pg_rn ........ esz:2 ... ... ... pg:3 rn:5 rd:5 &rpr_esz @@ -701,6 +703,21 @@ FMULX 01100101 .. 00 1010 100 ... ..... ....= . @rdn_pg_rm FDIV 01100101 .. 00 1100 100 ... ..... ..... @rdm_pg_rn # FD= IVR FDIV 01100101 .. 00 1101 100 ... ..... ..... @rdn_pg_rm =20 +### SVE FP Multiply-Add Group + +# SVE floating-point multiply-accumulate writing addend +FMLA_zpzzz 01100101 .. 1 ..... 000 ... ..... ..... @rda_pg_rn= _rm +FMLS_zpzzz 01100101 .. 1 ..... 001 ... ..... ..... @rda_pg_rn= _rm +FNMLA_zpzzz 01100101 .. 1 ..... 010 ... ..... ..... @rda_pg_rn= _rm +FNMLS_zpzzz 01100101 .. 1 ..... 011 ... ..... ..... @rda_pg_rn= _rm + +# SVE floating-point multiply-accumulate writing multiplicand +# FMAD, FMSB, FNMAD, FNMS +FMLA_zpzzz 01100101 .. 1 ..... 100 ... ..... ..... @rdn_pg_rm= _ra +FMLS_zpzzz 01100101 .. 1 ..... 101 ... ..... ..... @rdn_pg_rm= _ra +FNMLA_zpzzz 01100101 .. 1 ..... 110 ... ..... ..... @rdn_pg_rm= _ra +FNMLS_zpzzz 01100101 .. 1 ..... 111 ... ..... ..... @rdn_pg_rm= _ra + ### SVE FP Unary Operations Predicated Group =20 # SVE integer convert to floating-point --=20 2.17.1