From nobody Wed Oct 22 06:35:00 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1519403136442335.4020073526241; Fri, 23 Feb 2018 08:25:36 -0800 (PST) Received: from localhost ([::1]:45595 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1epGAB-0004iv-EX for importer@patchew.org; Fri, 23 Feb 2018 11:25:35 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:55413) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1epFxc-0002CC-6S for qemu-devel@nongnu.org; Fri, 23 Feb 2018 11:12:38 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1epFxW-0000EX-PG for qemu-devel@nongnu.org; Fri, 23 Feb 2018 11:12:36 -0500 Received: from mail-pf0-x241.google.com ([2607:f8b0:400e:c00::241]:38880) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1epFxW-0000Dh-FM for qemu-devel@nongnu.org; Fri, 23 Feb 2018 11:12:30 -0500 Received: by mail-pf0-x241.google.com with SMTP id d26so3692624pfn.5 for ; Fri, 23 Feb 2018 08:12:30 -0800 (PST) Received: from cloudburst.twiddle.net (97-113-169-147.tukw.qwest.net. [97.113.169.147]) by smtp.gmail.com with ESMTPSA id q14sm4712643pgq.2.2018.02.23.08.12.27 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Fri, 23 Feb 2018 08:12:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id; bh=ejb3pyZwj1z7lRj8D2FE2a+09QIB8zBRI5m6AynqfZo=; b=WD+V0+hPq35CKA1+VKXvosfPXn0mSsQx/BLcYbLqrzO6kjHUQEU7DLFNZaFyCZRz82 IasEPpkDgnHvdDMvQkqgnw8DmeYg/f79dWaT4WcVzuAbQzChhIOaDrllTHcXmG0AdNjY gTBQHMhLSoY/IYc4xLizGGbK0mr5TM3wy2M1c= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=ejb3pyZwj1z7lRj8D2FE2a+09QIB8zBRI5m6AynqfZo=; b=JGJCP4iUDC/WyGVq46FwNNZFvc+7oCqW+WOcX6xmAKPaJylFOTmCX4L+sG2qom9IWW GSdpyRON2m1XVbjYgluVyeX/fCBukS9QPb1HJPnKXmuSVQor0hEHE805b+0yr5NGX1pg ecA+E/nNDof9Up3dSVB6v5JjG9e7+h+ksZBGEKdyeLSJT+gOpgAg0gcF8DWQn0PT+e7I L17SlD2JvlQSKighBhS+Acx900iv/V+ec+9QTKx+l1UD3qkr9DyvgR6RhUX+CsKWZO4U hDTgD/LEnJ98jH+PIZYBXNOG7tUPekxOcnYZ/z2knwlLYGBymqJnsaFxGRuCTOROHFoM WRZg== X-Gm-Message-State: APf1xPAHyeY+BRHWeUDZ2Qz/sD7BQbgXohzWCeCdtJ7jNNyBKdcIwYf1 obgf0nroqq/FSlTFmVtpx5y89C06cOc= X-Google-Smtp-Source: AH8x224Eup+Vrlot4b0F7DXxneSCs5lCc0hH/HyWBWHZDOL1C5JlPi76bF0CO6HFFkT4abipWoWdyg== X-Received: by 10.99.116.69 with SMTP id e5mr1861216pgn.437.1519402348453; Fri, 23 Feb 2018 08:12:28 -0800 (PST) From: Richard Henderson To: qemu-devel@nongnu.org Date: Fri, 23 Feb 2018 08:12:26 -0800 Message-Id: <20180223161226.11640-1-richard.henderson@linaro.org> X-Mailer: git-send-email 2.14.3 X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c00::241 Subject: [Qemu-devel] [PATCH v2] tcg: Add choose_vector_size X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZohoMail: RDKM_2 RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" This unifies 5 copies of checks for supported vector size, and in the process fixes a missing check in tcg_gen_gvec_2s. This lead to an assertion failure for 64-bit vector multiply, which is not available in the AVX instruction set. Suggested-by: Peter Maydell Signed-off-by: Richard Henderson --- tcg/tcg-op-gvec.c | 438 ++++++++++++++++++++++++++++++++------------------= ---- 1 file changed, 259 insertions(+), 179 deletions(-) This is a replacement for two previous patches. I'm keeping the OP =3D=3D 0 test local, where it makes sense. But since all of those tests are now isolated in one function, it's cleaner than before. r~ --- diff --git a/tcg/tcg-op-gvec.c b/tcg/tcg-op-gvec.c index bfe44bba81..22db1590d5 100644 --- a/tcg/tcg-op-gvec.c +++ b/tcg/tcg-op-gvec.c @@ -351,6 +351,42 @@ static void gen_dup_i64(unsigned vece, TCGv_i64 out, T= CGv_i64 in) } } =20 +/* Select a supported vector type for implementing an operation on SIZE + * bytes. If OP is 0, assume that the real operation to be performed is + * required by all backends. Otherwise, make sure than OP can be performed + * on elements of size VECE in the selected type. Do not select V64 if + * PREFER_I64 is true. Return 0 if no vector type is selected. + */ +static TCGType choose_vector_type(TCGOpcode op, unsigned vece, uint32_t si= ze, + bool prefer_i64) +{ + if (TCG_TARGET_HAS_v256 && check_size_impl(size, 32)) { + if (op =3D=3D 0) { + return TCG_TYPE_V256; + } + /* Recall that ARM SVE allows vector sizes that are not a + * power of 2, but always a multiple of 16. The intent is + * that e.g. size =3D=3D 80 would be expanded with 2x32 + 1x16. + * It is hard to imagine a case in which v256 is supported + * but v128 is not, but check anyway. + */ + if (tcg_can_emit_vec_op(op, TCG_TYPE_V256, vece) + && (size % 32 =3D=3D 0 + || tcg_can_emit_vec_op(op, TCG_TYPE_V128, vece))) { + return TCG_TYPE_V256; + } + } + if (TCG_TARGET_HAS_v128 && check_size_impl(size, 16) + && (op =3D=3D 0 || tcg_can_emit_vec_op(op, TCG_TYPE_V128, vece))) { + return TCG_TYPE_V128; + } + if (TCG_TARGET_HAS_v64 && !prefer_i64 && check_size_impl(size, 8) + && (op =3D=3D 0 || tcg_can_emit_vec_op(op, TCG_TYPE_V64, vece))) { + return TCG_TYPE_V64; + } + return 0; +} + /* Set OPRSZ bytes at DOFS to replications of IN_32, IN_64 or IN_C. * Only one of IN_32 or IN_64 may be set; * IN_C is used if IN_32 and IN_64 are unset. @@ -376,19 +412,12 @@ static void do_dup(unsigned vece, uint32_t dofs, uint= 32_t oprsz, } } =20 - type =3D 0; - if (TCG_TARGET_HAS_v256 && check_size_impl(oprsz, 32)) { - type =3D TCG_TYPE_V256; - } else if (TCG_TARGET_HAS_v128 && check_size_impl(oprsz, 16)) { - type =3D TCG_TYPE_V128; - } else if (TCG_TARGET_HAS_v64 && check_size_impl(oprsz, 8) - /* Prefer integer when 64-bit host and no variable dup. */ - && !(TCG_TARGET_REG_BITS =3D=3D 64 && in_32 =3D=3D NULL - && (in_64 =3D=3D NULL || vece =3D=3D MO_64))) { - type =3D TCG_TYPE_V64; - } - - /* Implement inline with a vector type, if possible. */ + /* Implement inline with a vector type, if possible. + * Prefer integer when 64-bit host and no variable dup. + */ + type =3D choose_vector_type(0, vece, oprsz, + (TCG_TARGET_REG_BITS =3D=3D 64 && in_32 =3D= =3D NULL + && (in_64 =3D=3D NULL || vece =3D=3D MO_64)= )); if (type !=3D 0) { TCGv_vec t_vec =3D tcg_temp_new_vec(type); =20 @@ -414,21 +443,30 @@ static void do_dup(unsigned vece, uint32_t dofs, uint= 32_t oprsz, } =20 i =3D 0; - if (TCG_TARGET_HAS_v256) { + switch (type) { + case TCG_TYPE_V256: + /* Recall that ARM SVE allows vector sizes that are not a + * power of 2, but always a multiple of 16. The intent is + * that e.g. size =3D=3D 80 would be expanded with 2x32 + 1x16. + */ for (; i + 32 <=3D oprsz; i +=3D 32) { tcg_gen_stl_vec(t_vec, cpu_env, dofs + i, TCG_TYPE_V256); } - } - if (TCG_TARGET_HAS_v128) { + /* fallthru */ + case TCG_TYPE_V128: for (; i + 16 <=3D oprsz; i +=3D 16) { tcg_gen_stl_vec(t_vec, cpu_env, dofs + i, TCG_TYPE_V128); } - } - if (TCG_TARGET_HAS_v64) { + break; + case TCG_TYPE_V64: for (; i < oprsz; i +=3D 8) { tcg_gen_stl_vec(t_vec, cpu_env, dofs + i, TCG_TYPE_V64); } + break; + default: + g_assert_not_reached(); } + tcg_temp_free_vec(t_vec); goto done; } @@ -484,7 +522,7 @@ static void do_dup(unsigned vece, uint32_t dofs, uint32= _t oprsz, } tcg_temp_free_i64(t_64); goto done; - }=20 + } } =20 /* Otherwise implement out of line. */ @@ -866,49 +904,55 @@ static void expand_4_vec(unsigned vece, uint32_t dofs= , uint32_t aofs, void tcg_gen_gvec_2(uint32_t dofs, uint32_t aofs, uint32_t oprsz, uint32_t maxsz, const GVecGen2 *g) { + TCGType type; + uint32_t some; + check_size_align(oprsz, maxsz, dofs | aofs); check_overlap_2(dofs, aofs, maxsz); =20 - /* Recall that ARM SVE allows vector sizes that are not a power of 2. - Expand with successively smaller host vector sizes. The intent is - that e.g. oprsz =3D=3D 80 would be expanded with 2x32 + 1x16. */ - /* ??? For maxsz > oprsz, the host may be able to use an opr-sized - operation, zeroing the balance of the register. We can then - use a max-sized store to implement the clearing without an extra - store operation. This is true for aarch64 and x86_64 hosts. */ - - if (TCG_TARGET_HAS_v256 && g->fniv && check_size_impl(oprsz, 32) - && (!g->opc || tcg_can_emit_vec_op(g->opc, TCG_TYPE_V256, g->vece)= )) { - uint32_t some =3D QEMU_ALIGN_DOWN(oprsz, 32); + type =3D 0; + if (g->fniv) { + type =3D choose_vector_type(g->opc, g->vece, oprsz, g->prefer_i64); + } + switch (type) { + case TCG_TYPE_V256: + /* Recall that ARM SVE allows vector sizes that are not a + * power of 2, but always a multiple of 16. The intent is + * that e.g. size =3D=3D 80 would be expanded with 2x32 + 1x16. + */ + some =3D QEMU_ALIGN_DOWN(oprsz, 32); expand_2_vec(g->vece, dofs, aofs, some, 32, TCG_TYPE_V256, g->fniv= ); if (some =3D=3D oprsz) { - goto done; + break; } dofs +=3D some; aofs +=3D some; oprsz -=3D some; maxsz -=3D some; - } - - if (TCG_TARGET_HAS_v128 && g->fniv && check_size_impl(oprsz, 16) - && (!g->opc || tcg_can_emit_vec_op(g->opc, TCG_TYPE_V128, g->vece)= )) { + /* fallthru */ + case TCG_TYPE_V128: expand_2_vec(g->vece, dofs, aofs, oprsz, 16, TCG_TYPE_V128, g->fni= v); - } else if (TCG_TARGET_HAS_v64 && !g->prefer_i64 - && g->fniv && check_size_impl(oprsz, 8) - && (!g->opc - || tcg_can_emit_vec_op(g->opc, TCG_TYPE_V64, g->vece)))= { + break; + case TCG_TYPE_V64: expand_2_vec(g->vece, dofs, aofs, oprsz, 8, TCG_TYPE_V64, g->fniv); - } else if (g->fni8 && check_size_impl(oprsz, 8)) { - expand_2_i64(dofs, aofs, oprsz, g->fni8); - } else if (g->fni4 && check_size_impl(oprsz, 4)) { - expand_2_i32(dofs, aofs, oprsz, g->fni4); - } else { - assert(g->fno !=3D NULL); - tcg_gen_gvec_2_ool(dofs, aofs, oprsz, maxsz, g->data, g->fno); - return; + break; + + case 0: + if (g->fni8 && check_size_impl(oprsz, 8)) { + expand_2_i64(dofs, aofs, oprsz, g->fni8); + } else if (g->fni4 && check_size_impl(oprsz, 4)) { + expand_2_i32(dofs, aofs, oprsz, g->fni4); + } else { + assert(g->fno !=3D NULL); + tcg_gen_gvec_2_ool(dofs, aofs, oprsz, maxsz, g->data, g->fno); + return; + } + break; + + default: + g_assert_not_reached(); } =20 - done: if (oprsz < maxsz) { expand_clr(dofs + oprsz, maxsz - oprsz); } @@ -918,53 +962,64 @@ void tcg_gen_gvec_2(uint32_t dofs, uint32_t aofs, void tcg_gen_gvec_2i(uint32_t dofs, uint32_t aofs, uint32_t oprsz, uint32_t maxsz, int64_t c, const GVecGen2i *g) { + TCGType type; + uint32_t some; + check_size_align(oprsz, maxsz, dofs | aofs); check_overlap_2(dofs, aofs, maxsz); =20 - /* Recall that ARM SVE allows vector sizes that are not a power of 2. - Expand with successively smaller host vector sizes. The intent is - that e.g. oprsz =3D=3D 80 would be expanded with 2x32 + 1x16. */ - - if (TCG_TARGET_HAS_v256 && g->fniv && check_size_impl(oprsz, 32) - && (!g->opc || tcg_can_emit_vec_op(g->opc, TCG_TYPE_V256, g->vece)= )) { - uint32_t some =3D QEMU_ALIGN_DOWN(oprsz, 32); + type =3D 0; + if (g->fniv) { + type =3D choose_vector_type(g->opc, g->vece, oprsz, g->prefer_i64); + } + switch (type) { + case TCG_TYPE_V256: + /* Recall that ARM SVE allows vector sizes that are not a + * power of 2, but always a multiple of 16. The intent is + * that e.g. size =3D=3D 80 would be expanded with 2x32 + 1x16. + */ + some =3D QEMU_ALIGN_DOWN(oprsz, 32); expand_2i_vec(g->vece, dofs, aofs, some, 32, TCG_TYPE_V256, c, g->load_dest, g->fniv); if (some =3D=3D oprsz) { - goto done; + break; } dofs +=3D some; aofs +=3D some; oprsz -=3D some; maxsz -=3D some; - } - - if (TCG_TARGET_HAS_v128 && g->fniv && check_size_impl(oprsz, 16) - && (!g->opc || tcg_can_emit_vec_op(g->opc, TCG_TYPE_V128, g->vece)= )) { + /* fallthru */ + case TCG_TYPE_V128: expand_2i_vec(g->vece, dofs, aofs, oprsz, 16, TCG_TYPE_V128, c, g->load_dest, g->fniv); - } else if (TCG_TARGET_HAS_v64 && !g->prefer_i64 - && g->fniv && check_size_impl(oprsz, 8) - && (!g->opc - || tcg_can_emit_vec_op(g->opc, TCG_TYPE_V64, g->vece)))= { + break; + case TCG_TYPE_V64: expand_2i_vec(g->vece, dofs, aofs, oprsz, 8, TCG_TYPE_V64, c, g->load_dest, g->fniv); - } else if (g->fni8 && check_size_impl(oprsz, 8)) { - expand_2i_i64(dofs, aofs, oprsz, c, g->load_dest, g->fni8); - } else if (g->fni4 && check_size_impl(oprsz, 4)) { - expand_2i_i32(dofs, aofs, oprsz, c, g->load_dest, g->fni4); - } else { - if (g->fno) { - tcg_gen_gvec_2_ool(dofs, aofs, oprsz, maxsz, c, g->fno); + break; + + case 0: + if (g->fni8 && check_size_impl(oprsz, 8)) { + expand_2i_i64(dofs, aofs, oprsz, c, g->load_dest, g->fni8); + } else if (g->fni4 && check_size_impl(oprsz, 4)) { + expand_2i_i32(dofs, aofs, oprsz, c, g->load_dest, g->fni4); } else { - TCGv_i64 tcg_c =3D tcg_const_i64(c); - tcg_gen_gvec_2i_ool(dofs, aofs, tcg_c, oprsz, maxsz, c, g->fno= i); - tcg_temp_free_i64(tcg_c); + if (g->fno) { + tcg_gen_gvec_2_ool(dofs, aofs, oprsz, maxsz, c, g->fno); + } else { + TCGv_i64 tcg_c =3D tcg_const_i64(c); + tcg_gen_gvec_2i_ool(dofs, aofs, tcg_c, oprsz, + maxsz, c, g->fnoi); + tcg_temp_free_i64(tcg_c); + } + return; } - return; + break; + + default: + g_assert_not_reached(); } =20 - done: if (oprsz < maxsz) { expand_clr(dofs + oprsz, maxsz - oprsz); } @@ -981,37 +1036,30 @@ void tcg_gen_gvec_2s(uint32_t dofs, uint32_t aofs, u= int32_t oprsz, =20 type =3D 0; if (g->fniv) { - if (TCG_TARGET_HAS_v256 && check_size_impl(oprsz, 32)) { - type =3D TCG_TYPE_V256; - } else if (TCG_TARGET_HAS_v128 && check_size_impl(oprsz, 16)) { - type =3D TCG_TYPE_V128; - } else if (TCG_TARGET_HAS_v64 && !g->prefer_i64 - && check_size_impl(oprsz, 8)) { - type =3D TCG_TYPE_V64; - } + type =3D choose_vector_type(g->opc, g->vece, oprsz, g->prefer_i64); } if (type !=3D 0) { TCGv_vec t_vec =3D tcg_temp_new_vec(type); + uint32_t some; =20 tcg_gen_dup_i64_vec(g->vece, t_vec, c); =20 - /* Recall that ARM SVE allows vector sizes that are not a power of= 2. - Expand with successively smaller host vector sizes. The intent= is - that e.g. oprsz =3D=3D 80 would be expanded with 2x32 + 1x16. = */ switch (type) { case TCG_TYPE_V256: - { - uint32_t some =3D QEMU_ALIGN_DOWN(oprsz, 32); - expand_2s_vec(g->vece, dofs, aofs, some, 32, TCG_TYPE_V256, - t_vec, g->scalar_first, g->fniv); - if (some =3D=3D oprsz) { - break; - } - dofs +=3D some; - aofs +=3D some; - oprsz -=3D some; - maxsz -=3D some; + /* Recall that ARM SVE allows vector sizes that are not a + * power of 2, but always a multiple of 16. The intent is + * that e.g. size =3D=3D 80 would be expanded with 2x32 + 1x16. + */ + some =3D QEMU_ALIGN_DOWN(oprsz, 32); + expand_2s_vec(g->vece, dofs, aofs, some, 32, TCG_TYPE_V256, + t_vec, g->scalar_first, g->fniv); + if (some =3D=3D oprsz) { + break; } + dofs +=3D some; + aofs +=3D some; + oprsz -=3D some; + maxsz -=3D some; /* fallthru */ =20 case TCG_TYPE_V128: @@ -1055,48 +1103,60 @@ void tcg_gen_gvec_2s(uint32_t dofs, uint32_t aofs, = uint32_t oprsz, void tcg_gen_gvec_3(uint32_t dofs, uint32_t aofs, uint32_t bofs, uint32_t oprsz, uint32_t maxsz, const GVecGen3 *g) { + TCGType type; + uint32_t some; + check_size_align(oprsz, maxsz, dofs | aofs | bofs); check_overlap_3(dofs, aofs, bofs, maxsz); =20 - /* Recall that ARM SVE allows vector sizes that are not a power of 2. - Expand with successively smaller host vector sizes. The intent is - that e.g. oprsz =3D=3D 80 would be expanded with 2x32 + 1x16. */ - - if (TCG_TARGET_HAS_v256 && g->fniv && check_size_impl(oprsz, 32) - && (!g->opc || tcg_can_emit_vec_op(g->opc, TCG_TYPE_V256, g->vece)= )) { - uint32_t some =3D QEMU_ALIGN_DOWN(oprsz, 32); + type =3D 0; + if (g->fniv) { + type =3D choose_vector_type(g->opc, g->vece, oprsz, g->prefer_i64); + } + switch (type) { + case TCG_TYPE_V256: + /* Recall that ARM SVE allows vector sizes that are not a + * power of 2, but always a multiple of 16. The intent is + * that e.g. size =3D=3D 80 would be expanded with 2x32 + 1x16. + */ + some =3D QEMU_ALIGN_DOWN(oprsz, 32); expand_3_vec(g->vece, dofs, aofs, bofs, some, 32, TCG_TYPE_V256, g->load_dest, g->fniv); if (some =3D=3D oprsz) { - goto done; + break; } dofs +=3D some; aofs +=3D some; bofs +=3D some; oprsz -=3D some; maxsz -=3D some; - } - - if (TCG_TARGET_HAS_v128 && g->fniv && check_size_impl(oprsz, 16) - && (!g->opc || tcg_can_emit_vec_op(g->opc, TCG_TYPE_V128, g->vece)= )) { + /* fallthru */ + case TCG_TYPE_V128: expand_3_vec(g->vece, dofs, aofs, bofs, oprsz, 16, TCG_TYPE_V128, g->load_dest, g->fniv); - } else if (TCG_TARGET_HAS_v64 && !g->prefer_i64 - && g->fniv && check_size_impl(oprsz, 8) - && (!g->opc - || tcg_can_emit_vec_op(g->opc, TCG_TYPE_V64, g->vece)))= { + break; + case TCG_TYPE_V64: expand_3_vec(g->vece, dofs, aofs, bofs, oprsz, 8, TCG_TYPE_V64, g->load_dest, g->fniv); - } else if (g->fni8 && check_size_impl(oprsz, 8)) { - expand_3_i64(dofs, aofs, bofs, oprsz, g->load_dest, g->fni8); - } else if (g->fni4 && check_size_impl(oprsz, 4)) { - expand_3_i32(dofs, aofs, bofs, oprsz, g->load_dest, g->fni4); - } else { - assert(g->fno !=3D NULL); - tcg_gen_gvec_3_ool(dofs, aofs, bofs, oprsz, maxsz, g->data, g->fno= ); + break; + + case 0: + if (g->fni8 && check_size_impl(oprsz, 8)) { + expand_3_i64(dofs, aofs, bofs, oprsz, g->load_dest, g->fni8); + } else if (g->fni4 && check_size_impl(oprsz, 4)) { + expand_3_i32(dofs, aofs, bofs, oprsz, g->load_dest, g->fni4); + } else { + assert(g->fno !=3D NULL); + tcg_gen_gvec_3_ool(dofs, aofs, bofs, oprsz, + maxsz, g->data, g->fno); + return; + } + break; + + default: + g_assert_not_reached(); } =20 - done: if (oprsz < maxsz) { expand_clr(dofs + oprsz, maxsz - oprsz); } @@ -1106,20 +1166,27 @@ void tcg_gen_gvec_3(uint32_t dofs, uint32_t aofs, u= int32_t bofs, void tcg_gen_gvec_4(uint32_t dofs, uint32_t aofs, uint32_t bofs, uint32_t = cofs, uint32_t oprsz, uint32_t maxsz, const GVecGen4 *g) { + TCGType type; + uint32_t some; + check_size_align(oprsz, maxsz, dofs | aofs | bofs | cofs); check_overlap_4(dofs, aofs, bofs, cofs, maxsz); =20 - /* Recall that ARM SVE allows vector sizes that are not a power of 2. - Expand with successively smaller host vector sizes. The intent is - that e.g. oprsz =3D=3D 80 would be expanded with 2x32 + 1x16. */ - - if (TCG_TARGET_HAS_v256 && g->fniv && check_size_impl(oprsz, 32) - && (!g->opc || tcg_can_emit_vec_op(g->opc, TCG_TYPE_V256, g->vece)= )) { - uint32_t some =3D QEMU_ALIGN_DOWN(oprsz, 32); + type =3D 0; + if (g->fniv) { + type =3D choose_vector_type(g->opc, g->vece, oprsz, g->prefer_i64); + } + switch (type) { + case TCG_TYPE_V256: + /* Recall that ARM SVE allows vector sizes that are not a + * power of 2, but always a multiple of 16. The intent is + * that e.g. size =3D=3D 80 would be expanded with 2x32 + 1x16. + */ + some =3D QEMU_ALIGN_DOWN(oprsz, 32); expand_4_vec(g->vece, dofs, aofs, bofs, cofs, some, 32, TCG_TYPE_V256, g->fniv); if (some =3D=3D oprsz) { - goto done; + break; } dofs +=3D some; aofs +=3D some; @@ -1127,30 +1194,33 @@ void tcg_gen_gvec_4(uint32_t dofs, uint32_t aofs, u= int32_t bofs, uint32_t cofs, cofs +=3D some; oprsz -=3D some; maxsz -=3D some; - } - - if (TCG_TARGET_HAS_v128 && g->fniv && check_size_impl(oprsz, 16) - && (!g->opc || tcg_can_emit_vec_op(g->opc, TCG_TYPE_V128, g->vece)= )) { + /* fallthru */ + case TCG_TYPE_V128: expand_4_vec(g->vece, dofs, aofs, bofs, cofs, oprsz, 16, TCG_TYPE_V128, g->fniv); - } else if (TCG_TARGET_HAS_v64 && !g->prefer_i64 - && g->fniv && check_size_impl(oprsz, 8) - && (!g->opc - || tcg_can_emit_vec_op(g->opc, TCG_TYPE_V64, g->vece))= ) { + break; + case TCG_TYPE_V64: expand_4_vec(g->vece, dofs, aofs, bofs, cofs, oprsz, 8, TCG_TYPE_V64, g->fniv); - } else if (g->fni8 && check_size_impl(oprsz, 8)) { - expand_4_i64(dofs, aofs, bofs, cofs, oprsz, g->fni8); - } else if (g->fni4 && check_size_impl(oprsz, 4)) { - expand_4_i32(dofs, aofs, bofs, cofs, oprsz, g->fni4); - } else { - assert(g->fno !=3D NULL); - tcg_gen_gvec_4_ool(dofs, aofs, bofs, cofs, - oprsz, maxsz, g->data, g->fno); - return; + break; + + case 0: + if (g->fni8 && check_size_impl(oprsz, 8)) { + expand_4_i64(dofs, aofs, bofs, cofs, oprsz, g->fni8); + } else if (g->fni4 && check_size_impl(oprsz, 4)) { + expand_4_i32(dofs, aofs, bofs, cofs, oprsz, g->fni4); + } else { + assert(g->fno !=3D NULL); + tcg_gen_gvec_4_ool(dofs, aofs, bofs, cofs, + oprsz, maxsz, g->data, g->fno); + return; + } + break; + + default: + g_assert_not_reached(); } =20 - done: if (oprsz < maxsz) { expand_clr(dofs + oprsz, maxsz - oprsz); } @@ -2155,6 +2225,8 @@ void tcg_gen_gvec_cmp(TCGCond cond, unsigned vece, ui= nt32_t dofs, [TCG_COND_LTU] =3D ltu_fn, [TCG_COND_LEU] =3D leu_fn, }; + TCGType type; + uint32_t some; =20 check_size_align(oprsz, maxsz, dofs | aofs | bofs); check_overlap_3(dofs, aofs, bofs, maxsz); @@ -2165,51 +2237,59 @@ void tcg_gen_gvec_cmp(TCGCond cond, unsigned vece, = uint32_t dofs, return; } =20 - /* Recall that ARM SVE allows vector sizes that are not a power of 2. - Expand with successively smaller host vector sizes. The intent is - that e.g. oprsz =3D=3D 80 would be expanded with 2x32 + 1x16. */ - - if (TCG_TARGET_HAS_v256 && check_size_impl(oprsz, 32) - && tcg_can_emit_vec_op(INDEX_op_cmp_vec, TCG_TYPE_V256, vece)) { - uint32_t some =3D QEMU_ALIGN_DOWN(oprsz, 32); + /* Implement inline with a vector type, if possible. + * Prefer integer when 64-bit host and 64-bit comparison. + */ + type =3D choose_vector_type(INDEX_op_cmp_vec, vece, oprsz, + TCG_TARGET_REG_BITS =3D=3D 64 && vece =3D=3D= MO_64); + switch (type) { + case TCG_TYPE_V256: + /* Recall that ARM SVE allows vector sizes that are not a + * power of 2, but always a multiple of 16. The intent is + * that e.g. size =3D=3D 80 would be expanded with 2x32 + 1x16. + */ + some =3D QEMU_ALIGN_DOWN(oprsz, 32); expand_cmp_vec(vece, dofs, aofs, bofs, some, 32, TCG_TYPE_V256, co= nd); if (some =3D=3D oprsz) { - goto done; + break; } dofs +=3D some; aofs +=3D some; bofs +=3D some; oprsz -=3D some; maxsz -=3D some; - } - - if (TCG_TARGET_HAS_v128 && check_size_impl(oprsz, 16) - && tcg_can_emit_vec_op(INDEX_op_cmp_vec, TCG_TYPE_V128, vece)) { + /* fallthru */ + case TCG_TYPE_V128: expand_cmp_vec(vece, dofs, aofs, bofs, oprsz, 16, TCG_TYPE_V128, c= ond); - } else if (TCG_TARGET_HAS_v64 - && check_size_impl(oprsz, 8) - && (TCG_TARGET_REG_BITS =3D=3D 32 || vece !=3D MO_64) - && tcg_can_emit_vec_op(INDEX_op_cmp_vec, TCG_TYPE_V64, vece= )) { + break; + case TCG_TYPE_V64: expand_cmp_vec(vece, dofs, aofs, bofs, oprsz, 8, TCG_TYPE_V64, con= d); - } else if (vece =3D=3D MO_64 && check_size_impl(oprsz, 8)) { - expand_cmp_i64(dofs, aofs, bofs, oprsz, cond); - } else if (vece =3D=3D MO_32 && check_size_impl(oprsz, 4)) { - expand_cmp_i32(dofs, aofs, bofs, oprsz, cond); - } else { - gen_helper_gvec_3 * const *fn =3D fns[cond]; + break; =20 - if (fn =3D=3D NULL) { - uint32_t tmp; - tmp =3D aofs, aofs =3D bofs, bofs =3D tmp; - cond =3D tcg_swap_cond(cond); - fn =3D fns[cond]; - assert(fn !=3D NULL); + case 0: + if (vece =3D=3D MO_64 && check_size_impl(oprsz, 8)) { + expand_cmp_i64(dofs, aofs, bofs, oprsz, cond); + } else if (vece =3D=3D MO_32 && check_size_impl(oprsz, 4)) { + expand_cmp_i32(dofs, aofs, bofs, oprsz, cond); + } else { + gen_helper_gvec_3 * const *fn =3D fns[cond]; + + if (fn =3D=3D NULL) { + uint32_t tmp; + tmp =3D aofs, aofs =3D bofs, bofs =3D tmp; + cond =3D tcg_swap_cond(cond); + fn =3D fns[cond]; + assert(fn !=3D NULL); + } + tcg_gen_gvec_3_ool(dofs, aofs, bofs, oprsz, maxsz, 0, fn[vece]= ); + return; } - tcg_gen_gvec_3_ool(dofs, aofs, bofs, oprsz, maxsz, 0, fn[vece]); - return; + break; + + default: + g_assert_not_reached(); } =20 - done: if (oprsz < maxsz) { expand_clr(dofs + oprsz, maxsz - oprsz); } --=20 2.14.3