The following changes since commit 005ad32358f12fe9313a4a01918a55e60d4f39e5: Merge tag 'pull-tpm-2023-09-12-3' of https://github.com/stefanberger/qemu-tpm into staging (2023-09-13 13:41:57 -0400) are available in the Git repository at: https://gitlab.com/rth7680/qemu.git tags/pull-tcg-20230915 for you to fetch changes up to e0d9f49c143359b4a34cb80737af57228c62a008: accel/tcg: Restrict tcg_exec_[un]realizefn() to TCG (2023-09-15 19:06:29 -0700) ---------------------------------------------------------------- *: Delete checks for old host definitions tcg/loongarch64: Generate LSX instructions fpu: Add conversions between bfloat16 and [u]int8 fpu: Handle m68k extended precision denormals properly accel/tcg: Improve cputlb i/o organization accel/tcg: Simplify tlb_plugin_lookup accel/tcg: Remove false-negative halted assertion tcg: Add gvec compare with immediate and scalar operand tcg/aarch64: Emit BTI insns at jump landing pads ---------------------------------------------------------------- Akihiko Odaki (3): util: Delete checks for old host definitions softmmu: Delete checks for old host definitions thunk: Delete checks for old host definitions Anton Johansson (9): target/arm: Replace TARGET_PAGE_ENTRY_EXTRA accel/tcg: Modify tlb_*() to use CPUState accel/tcg: Modify probe_access_internal() to use CPUState accel/tcg: Modify memory access functions to use CPUState accel/tcg: Modify atomic_mmu_lookup() to use CPUState accel/tcg: Use CPUState in atomicity helpers accel/tcg: Remove env_tlb() accel/tcg: Unify user and softmmu do_[st|ld]*_mmu() accel/tcg: move ld/st helpers to ldst_common.c.inc Jiajie Chen (16): tcg/loongarch64: Import LSX instructions tcg/loongarch64: Lower basic tcg vec ops to LSX tcg: pass vece to tcg_target_const_match() tcg/loongarch64: Lower cmp_vec to vseq/vsle/vslt tcg/loongarch64: Lower add/sub_vec to vadd/vsub tcg/loongarch64: Lower vector bitwise operations tcg/loongarch64: Lower neg_vec to vneg tcg/loongarch64: Lower mul_vec to vmul tcg/loongarch64: Lower vector min max ops tcg/loongarch64: Lower vector saturated ops tcg/loongarch64: Lower vector shift vector ops tcg/loongarch64: Lower bitsel_vec to vbitsel tcg/loongarch64: Lower vector shift integer ops tcg/loongarch64: Lower rotv_vec ops to LSX tcg/loongarch64: Lower rotli_vec to vrotri tcg/loongarch64: Implement 128-bit load & store LIU Zhiwei (2): accel/tcg: Fix the comment for CPUTLBEntryFull fpu: Add conversions between bfloat16 and [u]int8 Nicholas Piggin (1): accel/tcg: mttcg remove false-negative halted assertion Philippe Mathieu-Daudé (16): exec: Make EXCP_FOO definitions target agnostic exec: Move cpu_loop_foo() target agnostic functions to 'cpu-common.h' accel/tcg: Restrict dump_exec_info() declaration accel: Make accel-blocker.o target agnostic accel: Rename accel-common.c -> accel-target.c exec: Rename cpu.c -> cpu-target.c exec: Rename target specific page-vary.c -> page-vary-target.c accel/tcg: Rename target-specific 'internal.h' -> 'internal-target.h' accel/tcg: Make monitor.c a target-agnostic unit accel/tcg: Make icount.o a target agnostic unit accel/tcg: Make cpu-exec-common.c a target agnostic unit accel: Rename accel_cpu_realizefn() -> accel_cpu_realize() accel: Introduce accel_cpu_unrealize() stub accel: Declare AccelClass::[un]realize_cpu() handlers accel/tcg: Have tcg_exec_realizefn() return a boolean accel/tcg: Restrict tcg_exec_[un]realizefn() to TCG Richard Henderson (31): tcg: Add gvec compare with immediate and scalar operand target/arm: Use tcg_gen_gvec_cmpi for compare vs 0 accel/tcg: Simplify tlb_plugin_lookup accel/tcg: Split out io_prepare and io_failed accel/tcg: Use CPUTLBEntryFull.phys_addr in io_failed plugin: Simplify struct qemu_plugin_hwaddr accel/tcg: Merge cpu_transaction_failed into io_failed accel/tcg: Replace direct use of io_readx/io_writex in do_{ld,st}_1 accel/tcg: Merge io_readx into do_ld_mmio_beN accel/tcg: Merge io_writex into do_st_mmio_leN accel/tcg: Introduce do_ld16_mmio_beN accel/tcg: Introduce do_st16_mmio_leN fpu: Handle m68k extended precision denormals properly tcg: Add tcg_out_tb_start backend hook util/cpuinfo-aarch64: Add CPUINFO_BTI tcg/aarch64: Emit BTI insns at jump landing pads tcg: Map code_gen_buffer with PROT_BTI accel/tcg: Move CPUTLB definitions from cpu-defs.h qom: Propagate alignment through type system target/arm: Remove size and alignment for cpu subclasses target/*: Add instance_align to all cpu base classes accel/tcg: Validate placement of CPUNegativeOffsetState accel/tcg: Move CPUNegativeOffsetState into CPUState accel/tcg: Remove CPUState.icount_decr_ptr accel/tcg: Move can_do_io to CPUNegativeOffsetState accel/tcg: Remove cpu_neg() tcg: Rename cpu_env to tcg_env accel/tcg: Replace CPUState.env_ptr with cpu_env() accel/tcg: Remove cpu_set_cpustate_pointers accel/tcg: Remove env_neg() tcg: Remove TCGContext.tlb_fast_offset MAINTAINERS | 7 +- meson.build | 6 +- accel/tcg/atomic_template.h | 20 +- accel/tcg/internal-common.h | 28 + accel/tcg/{internal.h => internal-target.h} | 21 +- accel/tcg/tcg-runtime.h | 25 + host/include/aarch64/host/cpuinfo.h | 1 + include/exec/cpu-all.h | 67 +- include/exec/cpu-common.h | 39 + include/exec/cpu-defs.h | 138 - include/exec/cpu_ldst.h | 8 +- include/exec/exec-all.h | 32 +- include/exec/user/thunk.h | 3 +- include/fpu/softfloat.h | 12 + include/hw/core/cpu.h | 180 +- include/qemu/accel.h | 12 +- include/qemu/plugin-memory.h | 11 +- include/qemu/typedefs.h | 1 - include/tcg/tcg-op-gvec-common.h | 6 + include/tcg/tcg.h | 3 +- target/alpha/cpu.h | 1 - target/arm/common-semi-target.h | 2 +- target/arm/cpu-param.h | 12 - target/arm/cpu.h | 1 - target/arm/tcg/translate-a32.h | 2 +- target/arm/tcg/translate-a64.h | 4 +- target/arm/tcg/translate.h | 16 +- target/avr/cpu.h | 1 - target/cris/cpu.h | 1 - target/hexagon/cpu.h | 2 +- target/hexagon/gen_tcg.h | 120 +- target/hexagon/gen_tcg_hvx.h | 20 +- target/hexagon/macros.h | 8 +- target/hppa/cpu.h | 1 - target/i386/cpu.h | 1 - target/loongarch/cpu.h | 1 - target/m68k/cpu.h | 1 - target/microblaze/cpu.h | 6 +- target/mips/cpu.h | 4 +- target/mips/tcg/translate.h | 6 +- target/nios2/cpu.h | 1 - target/openrisc/cpu.h | 1 - target/ppc/cpu.h | 1 - target/riscv/cpu.h | 2 +- target/rx/cpu.h | 1 - target/s390x/cpu.h | 1 - target/sh4/cpu.h | 1 - target/sparc/cpu.h | 1 - target/tricore/cpu.h | 1 - target/xtensa/cpu.h | 3 +- tcg/loongarch64/tcg-target-con-set.h | 9 + tcg/loongarch64/tcg-target-con-str.h | 3 + tcg/loongarch64/tcg-target.h | 40 +- tcg/loongarch64/tcg-target.opc.h | 12 + accel/{accel-common.c => accel-target.c} | 27 +- accel/dummy-cpus.c | 2 +- accel/kvm/kvm-accel-ops.c | 2 +- accel/tcg/cpu-exec-common.c | 5 +- accel/tcg/cpu-exec.c | 31 +- accel/tcg/cputlb.c | 1156 ++-- softmmu/icount.c => accel/tcg/icount-common.c | 7 +- accel/tcg/monitor.c | 2 +- accel/tcg/plugin-gen.c | 10 +- accel/tcg/tb-maint.c | 3 +- accel/tcg/tcg-accel-ops-icount.c | 8 +- accel/tcg/tcg-accel-ops-mttcg.c | 11 +- accel/tcg/tcg-accel-ops-rr.c | 4 +- accel/tcg/tcg-accel-ops.c | 2 +- accel/tcg/tcg-all.c | 4 +- accel/tcg/tcg-runtime-gvec.c | 26 + accel/tcg/translate-all.c | 15 +- accel/tcg/translator.c | 22 +- accel/tcg/user-exec.c | 279 +- bsd-user/main.c | 2 +- bsd-user/signal.c | 10 +- cpus-common.c => cpu-common.c | 0 cpu.c => cpu-target.c | 13 +- fpu/softfloat.c | 67 +- gdbstub/gdbstub.c | 4 +- gdbstub/user-target.c | 2 +- hw/core/cpu-common.c | 6 +- hw/i386/kvm/clock.c | 2 +- hw/intc/mips_gic.c | 2 +- hw/intc/riscv_aclint.c | 12 +- hw/intc/riscv_imsic.c | 2 +- hw/ppc/e500.c | 4 +- hw/ppc/spapr.c | 2 +- linux-user/elfload.c | 4 +- linux-user/i386/cpu_loop.c | 2 +- linux-user/main.c | 4 +- linux-user/signal.c | 15 +- monitor/hmp-cmds-target.c | 2 +- page-vary.c => page-vary-target.c | 0 plugins/api.c | 27 +- qom/object.c | 14 + semihosting/arm-compat-semi.c | 6 +- semihosting/syscalls.c | 28 +- softmmu/async-teardown.c | 3 - softmmu/watchpoint.c | 2 +- target/alpha/cpu.c | 3 +- target/alpha/translate.c | 146 +- target/arm/cpu.c | 12 +- target/arm/cpu64.c | 4 - target/arm/helper.c | 2 +- target/arm/ptw.c | 4 +- target/arm/tcg/mte_helper.c | 2 +- target/arm/tcg/sve_helper.c | 2 +- target/arm/tcg/tlb_helper.c | 4 +- target/arm/tcg/translate-a64.c | 380 +- target/arm/tcg/translate-m-nocp.c | 24 +- target/arm/tcg/translate-mve.c | 52 +- target/arm/tcg/translate-neon.c | 78 +- target/arm/tcg/translate-sme.c | 8 +- target/arm/tcg/translate-sve.c | 172 +- target/arm/tcg/translate-vfp.c | 56 +- target/arm/tcg/translate.c | 290 +- target/avr/cpu.c | 3 +- target/avr/translate.c | 66 +- target/cris/cpu.c | 3 +- target/cris/translate.c | 72 +- target/hexagon/cpu.c | 4 +- target/hexagon/genptr.c | 36 +- target/hexagon/idef-parser/parser-helpers.c | 2 +- target/hexagon/translate.c | 52 +- target/hppa/cpu.c | 2 +- target/hppa/mem_helper.c | 2 +- target/hppa/translate.c | 148 +- target/i386/cpu.c | 2 +- target/i386/kvm/kvm-cpu.c | 2 +- target/i386/nvmm/nvmm-all.c | 14 +- target/i386/tcg/sysemu/excp_helper.c | 2 +- target/i386/tcg/tcg-cpu.c | 2 +- target/i386/tcg/translate.c | 584 +- target/i386/whpx/whpx-all.c | 26 +- target/loongarch/cpu.c | 9 +- target/loongarch/translate.c | 22 +- target/m68k/cpu.c | 9 +- target/m68k/translate.c | 306 +- target/microblaze/cpu.c | 2 +- target/microblaze/translate.c | 52 +- target/mips/cpu.c | 2 +- target/mips/tcg/lcsr_translate.c | 6 +- target/mips/tcg/msa_translate.c | 34 +- target/mips/tcg/mxu_translate.c | 4 +- target/mips/tcg/sysemu/mips-semi.c | 4 +- target/mips/tcg/translate.c | 1288 ++--- target/mips/tcg/vr54xx_translate.c | 2 +- target/nios2/cpu.c | 5 +- target/nios2/translate.c | 52 +- target/openrisc/cpu.c | 7 +- target/openrisc/translate.c | 86 +- target/ppc/cpu_init.c | 1 - target/ppc/excp_helper.c | 10 +- target/ppc/translate.c | 366 +- target/riscv/cpu.c | 8 +- target/riscv/translate.c | 56 +- target/rx/cpu.c | 5 +- target/rx/translate.c | 58 +- target/s390x/cpu.c | 2 - target/s390x/tcg/translate.c | 426 +- target/sh4/cpu.c | 3 +- target/sh4/op_helper.c | 2 +- target/sh4/translate.c | 128 +- target/sparc/cpu.c | 3 +- target/sparc/translate.c | 332 +- target/tricore/cpu.c | 10 +- target/tricore/translate.c | 224 +- target/xtensa/cpu.c | 2 +- target/xtensa/translate.c | 192 +- tcg/region.c | 41 +- tcg/tcg-op-gvec.c | 437 +- tcg/tcg-op-ldst.c | 22 +- tcg/tcg-op.c | 2 +- tcg/tcg.c | 26 +- tests/tcg/m68k/denormal.c | 53 + util/cpuinfo-aarch64.c | 7 + util/oslib-posix.c | 15 +- accel/tcg/ldst_atomicity.c.inc | 88 +- accel/tcg/ldst_common.c.inc | 225 + fpu/softfloat-parts.c.inc | 7 +- target/cris/translate_v10.c.inc | 28 +- target/i386/tcg/decode-new.c.inc | 4 +- target/i386/tcg/emit.c.inc | 262 +- target/loongarch/insn_trans/trans_atomic.c.inc | 4 +- target/loongarch/insn_trans/trans_branch.c.inc | 2 +- target/loongarch/insn_trans/trans_extra.c.inc | 10 +- target/loongarch/insn_trans/trans_farith.c.inc | 6 +- target/loongarch/insn_trans/trans_fcmp.c.inc | 8 +- target/loongarch/insn_trans/trans_fmemory.c.inc | 8 +- target/loongarch/insn_trans/trans_fmov.c.inc | 20 +- target/loongarch/insn_trans/trans_lsx.c.inc | 44 +- target/loongarch/insn_trans/trans_memory.c.inc | 8 +- target/loongarch/insn_trans/trans_privileged.c.inc | 52 +- target/mips/tcg/micromips_translate.c.inc | 12 +- target/mips/tcg/nanomips_translate.c.inc | 200 +- target/ppc/power8-pmu-regs.c.inc | 8 +- target/ppc/translate/branch-impl.c.inc | 2 +- target/ppc/translate/dfp-impl.c.inc | 22 +- target/ppc/translate/fixedpoint-impl.c.inc | 2 +- target/ppc/translate/fp-impl.c.inc | 50 +- target/ppc/translate/processor-ctrl-impl.c.inc | 8 +- target/ppc/translate/spe-impl.c.inc | 30 +- target/ppc/translate/storage-ctrl-impl.c.inc | 26 +- target/ppc/translate/vmx-impl.c.inc | 34 +- target/ppc/translate/vsx-impl.c.inc | 54 +- target/riscv/insn_trans/trans_privileged.c.inc | 8 +- target/riscv/insn_trans/trans_rvbf16.c.inc | 10 +- target/riscv/insn_trans/trans_rvd.c.inc | 48 +- target/riscv/insn_trans/trans_rvf.c.inc | 46 +- target/riscv/insn_trans/trans_rvh.c.inc | 8 +- target/riscv/insn_trans/trans_rvi.c.inc | 16 +- target/riscv/insn_trans/trans_rvm.c.inc | 16 +- target/riscv/insn_trans/trans_rvv.c.inc | 130 +- target/riscv/insn_trans/trans_rvvk.c.inc | 30 +- target/riscv/insn_trans/trans_rvzce.c.inc | 2 +- target/riscv/insn_trans/trans_rvzfa.c.inc | 38 +- target/riscv/insn_trans/trans_rvzfh.c.inc | 54 +- target/riscv/insn_trans/trans_rvzicbo.c.inc | 8 +- target/riscv/insn_trans/trans_svinval.c.inc | 6 +- target/riscv/insn_trans/trans_xthead.c.inc | 2 +- target/s390x/tcg/translate_vx.c.inc | 104 +- tcg/aarch64/tcg-target.c.inc | 61 +- tcg/arm/tcg-target.c.inc | 9 +- tcg/i386/tcg-target.c.inc | 7 +- tcg/loongarch64/tcg-insn-defs.c.inc | 6019 +++++++++++++++++++- tcg/loongarch64/tcg-target.c.inc | 628 +- tcg/mips/tcg-target.c.inc | 7 +- tcg/ppc/tcg-target.c.inc | 7 +- tcg/riscv/tcg-target.c.inc | 7 +- tcg/s390x/tcg-target.c.inc | 7 +- tcg/sparc64/tcg-target.c.inc | 7 +- tcg/tci/tcg-target.c.inc | 7 +- accel/meson.build | 4 +- accel/tcg/meson.build | 8 +- softmmu/meson.build | 4 - target/hexagon/README | 10 +- target/hexagon/gen_tcg_funcs.py | 16 +- tests/tcg/m68k/Makefile.target | 2 +- 238 files changed, 12363 insertions(+), 5537 deletions(-) create mode 100644 accel/tcg/internal-common.h rename accel/tcg/{internal.h => internal-target.h} (89%) create mode 100644 tcg/loongarch64/tcg-target.opc.h rename accel/{accel-common.c => accel-target.c} (87%) rename softmmu/icount.c => accel/tcg/icount-common.c (99%) rename cpus-common.c => cpu-common.c (100%) rename cpu.c => cpu-target.c (97%) rename page-vary.c => page-vary-target.c (100%) create mode 100644 tests/tcg/m68k/denormal.c From: Nicholas Piggin <npiggin@gmail.com> mttcg asserts that an execution ending with EXCP_HALTED must have cpu->halted. However between the event or instruction that sets cpu->halted and requests exit and the assertion here, an asynchronous event could clear cpu->halted. This leads to crashes running AIX on ppc/pseries because it uses H_CEDE/H_PROD hcalls, where H_CEDE sets self->halted = 1 and H_PROD sets other cpu->halted = 0 and kicks it. H_PROD could be turned into an interrupt to wake, but several other places in ppc, sparc, and semihosting follow what looks like a similar pattern setting halted = 0 directly. So remove this assertion. Reported-by: Ivan Warren <ivan@vmfacility.fr> Signed-off-by: Nicholas Piggin <npiggin@gmail.com> Message-Id: <20230829010658.8252-1-npiggin@gmail.com> [rth: Keep the case label and adjust the comment.] Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- accel/tcg/tcg-accel-ops-mttcg.c | 9 ++------- 1 file changed, 2 insertions(+), 7 deletions(-) diff --git a/accel/tcg/tcg-accel-ops-mttcg.c b/accel/tcg/tcg-accel-ops-mttcg.c index XXXXXXX..XXXXXXX 100644 --- a/accel/tcg/tcg-accel-ops-mttcg.c +++ b/accel/tcg/tcg-accel-ops-mttcg.c @@ -XXX,XX +XXX,XX @@ static void *mttcg_cpu_thread_fn(void *arg) break; case EXCP_HALTED: /* - * during start-up the vCPU is reset and the thread is - * kicked several times. If we don't ensure we go back - * to sleep in the halted state we won't cleanly - * start-up when the vCPU is enabled. - * - * cpu->halted should ensure we sleep in wait_io_event + * Usually cpu->halted is set, but may have already been + * reset by another thread by the time we arrive here. */ - g_assert(cpu->halted); break; case EXCP_ATOMIC: qemu_mutex_unlock_iothread(); -- 2.34.1 From: LIU Zhiwei <zhiwei_liu@linux.alibaba.com> When memory region is ram, the lower TARGET_PAGE_BITS is not the physical section number. Instead, its value is always 0. Add comment and assert to make it clear. Signed-off-by: LIU Zhiwei <zhiwei_liu@linux.alibaba.com> Message-Id: <20230901060118.379-1-zhiwei_liu@linux.alibaba.com> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- include/exec/cpu-defs.h | 12 ++++++------ accel/tcg/cputlb.c | 11 +++++++---- 2 files changed, 13 insertions(+), 10 deletions(-) diff --git a/include/exec/cpu-defs.h b/include/exec/cpu-defs.h index XXXXXXX..XXXXXXX 100644 --- a/include/exec/cpu-defs.h +++ b/include/exec/cpu-defs.h @@ -XXX,XX +XXX,XX @@ typedef struct CPUTLBEntryFull { /* * @xlat_section contains: - * - in the lower TARGET_PAGE_BITS, a physical section number - * - with the lower TARGET_PAGE_BITS masked off, an offset which - * must be added to the virtual address to obtain: - * + the ram_addr_t of the target RAM (if the physical section - * number is PHYS_SECTION_NOTDIRTY or PHYS_SECTION_ROM) - * + the offset within the target MemoryRegion (otherwise) + * - For ram, an offset which must be added to the virtual address + * to obtain the ram_addr_t of the target RAM + * - For other memory regions, + * + in the lower TARGET_PAGE_BITS, the physical section number + * + with the TARGET_PAGE_BITS masked off, the offset within + * the target MemoryRegion */ hwaddr xlat_section; diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c index XXXXXXX..XXXXXXX 100644 --- a/accel/tcg/cputlb.c +++ b/accel/tcg/cputlb.c @@ -XXX,XX +XXX,XX @@ void tlb_set_page_full(CPUState *cpu, int mmu_idx, write_flags = read_flags; if (is_ram) { iotlb = memory_region_get_ram_addr(section->mr) + xlat; + assert(!(iotlb & ~TARGET_PAGE_MASK)); /* * Computing is_clean is expensive; avoid all that unless * the page is actually writable. @@ -XXX,XX +XXX,XX @@ void tlb_set_page_full(CPUState *cpu, int mmu_idx, /* refill the tlb */ /* - * At this point iotlb contains a physical section number in the lower - * TARGET_PAGE_BITS, and either - * + the ram_addr_t of the page base of the target RAM (RAM) - * + the offset within section->mr of the page base (I/O, ROMD) + * When memory region is ram, iotlb contains a TARGET_PAGE_BITS + * aligned ram_addr_t of the page base of the target RAM. + * Otherwise, iotlb contains + * - a physical section number in the lower TARGET_PAGE_BITS + * - the offset within section->mr of the page base (I/O, ROMD) with the + * TARGET_PAGE_BITS masked off. * We subtract addr_page (which is page aligned and thus won't * disturb the low bits) to give an offset which can be added to the * (non-page-aligned) vaddr of the eventual memory access to get -- 2.34.1 From: Akihiko Odaki <akihiko.odaki@daynix.com> IA-64 and PA-RISC host support is already removed with commit b1cef6d02f ("Drop remaining bits of ia64 host support"). Signed-off-by: Akihiko Odaki <akihiko.odaki@daynix.com> Message-Id: <20230810225922.21600-1-akihiko.odaki@daynix.com> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- util/oslib-posix.c | 15 +++------------ 1 file changed, 3 insertions(+), 12 deletions(-) diff --git a/util/oslib-posix.c b/util/oslib-posix.c index XXXXXXX..XXXXXXX 100644 --- a/util/oslib-posix.c +++ b/util/oslib-posix.c @@ -XXX,XX +XXX,XX @@ char *qemu_get_pid_name(pid_t pid) void *qemu_alloc_stack(size_t *sz) { - void *ptr, *guardpage; + void *ptr; int flags; #ifdef CONFIG_DEBUG_STACK_USAGE void *ptr2; @@ -XXX,XX +XXX,XX @@ void *qemu_alloc_stack(size_t *sz) abort(); } -#if defined(HOST_IA64) - /* separate register stack */ - guardpage = ptr + (((*sz - pagesz) / 2) & ~pagesz); -#elif defined(HOST_HPPA) - /* stack grows up */ - guardpage = ptr + *sz - pagesz; -#else - /* stack grows down */ - guardpage = ptr; -#endif - if (mprotect(guardpage, pagesz, PROT_NONE) != 0) { + /* Stack grows down -- guard page at the bottom. */ + if (mprotect(ptr, pagesz, PROT_NONE) != 0) { perror("failed to set up stack guard page"); abort(); } -- 2.34.1 From: Akihiko Odaki <akihiko.odaki@daynix.com> PA-RISC host support is already removed with commit b1cef6d02f ("Drop remaining bits of ia64 host support"). Signed-off-by: Akihiko Odaki <akihiko.odaki@daynix.com> Message-Id: <20230810225922.21600-1-akihiko.odaki@daynix.com> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- softmmu/async-teardown.c | 3 --- 1 file changed, 3 deletions(-) diff --git a/softmmu/async-teardown.c b/softmmu/async-teardown.c index XXXXXXX..XXXXXXX 100644 --- a/softmmu/async-teardown.c +++ b/softmmu/async-teardown.c @@ -XXX,XX +XXX,XX @@ static void *new_stack_for_clone(void) /* Allocate a new stack and get a pointer to its top. */ stack_ptr = qemu_alloc_stack(&stack_size); -#if !defined(HOST_HPPA) - /* The top is at the end of the area, except on HPPA. */ stack_ptr += stack_size; -#endif return stack_ptr; } -- 2.34.1 From: Jiajie Chen <c@jia.je> Add opcodes and encoder functions for LSX. Generated from https://github.com/jiegec/loongarch-opcodes/tree/qemu-lsx. Signed-off-by: Jiajie Chen <c@jia.je> Acked-by: Richard Henderson <richard.henderson@linaro.org> Message-Id: <20230908022302.180442-2-c@jia.je> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- tcg/loongarch64/tcg-insn-defs.c.inc | 6019 ++++++++++++++++++++++++++- 1 file changed, 6018 insertions(+), 1 deletion(-) diff --git a/tcg/loongarch64/tcg-insn-defs.c.inc b/tcg/loongarch64/tcg-insn-defs.c.inc index XXXXXXX..XXXXXXX 100644 --- a/tcg/loongarch64/tcg-insn-defs.c.inc +++ b/tcg/loongarch64/tcg-insn-defs.c.inc @@ -XXX,XX +XXX,XX @@ * * This file is auto-generated by genqemutcgdefs from * https://github.com/loongson-community/loongarch-opcodes, - * from commit 25ca7effe9d88101c1cf96c4005423643386d81f. + * from commit 8027da9a8157a8b47fc48ff1def292e09c5668bd. * DO NOT EDIT. */ @@ -XXX,XX +XXX,XX @@ typedef enum { OPC_ANDI = 0x03400000, OPC_ORI = 0x03800000, OPC_XORI = 0x03c00000, + OPC_VFMADD_S = 0x09100000, + OPC_VFMADD_D = 0x09200000, + OPC_VFMSUB_S = 0x09500000, + OPC_VFMSUB_D = 0x09600000, + OPC_VFNMADD_S = 0x09900000, + OPC_VFNMADD_D = 0x09a00000, + OPC_VFNMSUB_S = 0x09d00000, + OPC_VFNMSUB_D = 0x09e00000, + OPC_VFCMP_CAF_S = 0x0c500000, + OPC_VFCMP_SAF_S = 0x0c508000, + OPC_VFCMP_CLT_S = 0x0c510000, + OPC_VFCMP_SLT_S = 0x0c518000, + OPC_VFCMP_CEQ_S = 0x0c520000, + OPC_VFCMP_SEQ_S = 0x0c528000, + OPC_VFCMP_CLE_S = 0x0c530000, + OPC_VFCMP_SLE_S = 0x0c538000, + OPC_VFCMP_CUN_S = 0x0c540000, + OPC_VFCMP_SUN_S = 0x0c548000, + OPC_VFCMP_CULT_S = 0x0c550000, + OPC_VFCMP_SULT_S = 0x0c558000, + OPC_VFCMP_CUEQ_S = 0x0c560000, + OPC_VFCMP_SUEQ_S = 0x0c568000, + OPC_VFCMP_CULE_S = 0x0c570000, + OPC_VFCMP_SULE_S = 0x0c578000, + OPC_VFCMP_CNE_S = 0x0c580000, + OPC_VFCMP_SNE_S = 0x0c588000, + OPC_VFCMP_COR_S = 0x0c5a0000, + OPC_VFCMP_SOR_S = 0x0c5a8000, + OPC_VFCMP_CUNE_S = 0x0c5c0000, + OPC_VFCMP_SUNE_S = 0x0c5c8000, + OPC_VFCMP_CAF_D = 0x0c600000, + OPC_VFCMP_SAF_D = 0x0c608000, + OPC_VFCMP_CLT_D = 0x0c610000, + OPC_VFCMP_SLT_D = 0x0c618000, + OPC_VFCMP_CEQ_D = 0x0c620000, + OPC_VFCMP_SEQ_D = 0x0c628000, + OPC_VFCMP_CLE_D = 0x0c630000, + OPC_VFCMP_SLE_D = 0x0c638000, + OPC_VFCMP_CUN_D = 0x0c640000, + OPC_VFCMP_SUN_D = 0x0c648000, + OPC_VFCMP_CULT_D = 0x0c650000, + OPC_VFCMP_SULT_D = 0x0c658000, + OPC_VFCMP_CUEQ_D = 0x0c660000, + OPC_VFCMP_SUEQ_D = 0x0c668000, + OPC_VFCMP_CULE_D = 0x0c670000, + OPC_VFCMP_SULE_D = 0x0c678000, + OPC_VFCMP_CNE_D = 0x0c680000, + OPC_VFCMP_SNE_D = 0x0c688000, + OPC_VFCMP_COR_D = 0x0c6a0000, + OPC_VFCMP_SOR_D = 0x0c6a8000, + OPC_VFCMP_CUNE_D = 0x0c6c0000, + OPC_VFCMP_SUNE_D = 0x0c6c8000, + OPC_VBITSEL_V = 0x0d100000, + OPC_VSHUF_B = 0x0d500000, OPC_ADDU16I_D = 0x10000000, OPC_LU12I_W = 0x14000000, OPC_CU32I_D = 0x16000000, @@ -XXX,XX +XXX,XX @@ typedef enum { OPC_LD_BU = 0x2a000000, OPC_LD_HU = 0x2a400000, OPC_LD_WU = 0x2a800000, + OPC_VLD = 0x2c000000, + OPC_VST = 0x2c400000, + OPC_VLDREPL_D = 0x30100000, + OPC_VLDREPL_W = 0x30200000, + OPC_VLDREPL_H = 0x30400000, + OPC_VLDREPL_B = 0x30800000, + OPC_VSTELM_D = 0x31100000, + OPC_VSTELM_W = 0x31200000, + OPC_VSTELM_H = 0x31400000, + OPC_VSTELM_B = 0x31800000, OPC_LDX_B = 0x38000000, OPC_LDX_H = 0x38040000, OPC_LDX_W = 0x38080000, @@ -XXX,XX +XXX,XX @@ typedef enum { OPC_LDX_BU = 0x38200000, OPC_LDX_HU = 0x38240000, OPC_LDX_WU = 0x38280000, + OPC_VLDX = 0x38400000, + OPC_VSTX = 0x38440000, OPC_DBAR = 0x38720000, OPC_JIRL = 0x4c000000, OPC_B = 0x50000000, @@ -XXX,XX +XXX,XX @@ typedef enum { OPC_BLE = 0x64000000, OPC_BGTU = 0x68000000, OPC_BLEU = 0x6c000000, + OPC_VSEQ_B = 0x70000000, + OPC_VSEQ_H = 0x70008000, + OPC_VSEQ_W = 0x70010000, + OPC_VSEQ_D = 0x70018000, + OPC_VSLE_B = 0x70020000, + OPC_VSLE_H = 0x70028000, + OPC_VSLE_W = 0x70030000, + OPC_VSLE_D = 0x70038000, + OPC_VSLE_BU = 0x70040000, + OPC_VSLE_HU = 0x70048000, + OPC_VSLE_WU = 0x70050000, + OPC_VSLE_DU = 0x70058000, + OPC_VSLT_B = 0x70060000, + OPC_VSLT_H = 0x70068000, + OPC_VSLT_W = 0x70070000, + OPC_VSLT_D = 0x70078000, + OPC_VSLT_BU = 0x70080000, + OPC_VSLT_HU = 0x70088000, + OPC_VSLT_WU = 0x70090000, + OPC_VSLT_DU = 0x70098000, + OPC_VADD_B = 0x700a0000, + OPC_VADD_H = 0x700a8000, + OPC_VADD_W = 0x700b0000, + OPC_VADD_D = 0x700b8000, + OPC_VSUB_B = 0x700c0000, + OPC_VSUB_H = 0x700c8000, + OPC_VSUB_W = 0x700d0000, + OPC_VSUB_D = 0x700d8000, + OPC_VADDWEV_H_B = 0x701e0000, + OPC_VADDWEV_W_H = 0x701e8000, + OPC_VADDWEV_D_W = 0x701f0000, + OPC_VADDWEV_Q_D = 0x701f8000, + OPC_VSUBWEV_H_B = 0x70200000, + OPC_VSUBWEV_W_H = 0x70208000, + OPC_VSUBWEV_D_W = 0x70210000, + OPC_VSUBWEV_Q_D = 0x70218000, + OPC_VADDWOD_H_B = 0x70220000, + OPC_VADDWOD_W_H = 0x70228000, + OPC_VADDWOD_D_W = 0x70230000, + OPC_VADDWOD_Q_D = 0x70238000, + OPC_VSUBWOD_H_B = 0x70240000, + OPC_VSUBWOD_W_H = 0x70248000, + OPC_VSUBWOD_D_W = 0x70250000, + OPC_VSUBWOD_Q_D = 0x70258000, + OPC_VADDWEV_H_BU = 0x702e0000, + OPC_VADDWEV_W_HU = 0x702e8000, + OPC_VADDWEV_D_WU = 0x702f0000, + OPC_VADDWEV_Q_DU = 0x702f8000, + OPC_VSUBWEV_H_BU = 0x70300000, + OPC_VSUBWEV_W_HU = 0x70308000, + OPC_VSUBWEV_D_WU = 0x70310000, + OPC_VSUBWEV_Q_DU = 0x70318000, + OPC_VADDWOD_H_BU = 0x70320000, + OPC_VADDWOD_W_HU = 0x70328000, + OPC_VADDWOD_D_WU = 0x70330000, + OPC_VADDWOD_Q_DU = 0x70338000, + OPC_VSUBWOD_H_BU = 0x70340000, + OPC_VSUBWOD_W_HU = 0x70348000, + OPC_VSUBWOD_D_WU = 0x70350000, + OPC_VSUBWOD_Q_DU = 0x70358000, + OPC_VADDWEV_H_BU_B = 0x703e0000, + OPC_VADDWEV_W_HU_H = 0x703e8000, + OPC_VADDWEV_D_WU_W = 0x703f0000, + OPC_VADDWEV_Q_DU_D = 0x703f8000, + OPC_VADDWOD_H_BU_B = 0x70400000, + OPC_VADDWOD_W_HU_H = 0x70408000, + OPC_VADDWOD_D_WU_W = 0x70410000, + OPC_VADDWOD_Q_DU_D = 0x70418000, + OPC_VSADD_B = 0x70460000, + OPC_VSADD_H = 0x70468000, + OPC_VSADD_W = 0x70470000, + OPC_VSADD_D = 0x70478000, + OPC_VSSUB_B = 0x70480000, + OPC_VSSUB_H = 0x70488000, + OPC_VSSUB_W = 0x70490000, + OPC_VSSUB_D = 0x70498000, + OPC_VSADD_BU = 0x704a0000, + OPC_VSADD_HU = 0x704a8000, + OPC_VSADD_WU = 0x704b0000, + OPC_VSADD_DU = 0x704b8000, + OPC_VSSUB_BU = 0x704c0000, + OPC_VSSUB_HU = 0x704c8000, + OPC_VSSUB_WU = 0x704d0000, + OPC_VSSUB_DU = 0x704d8000, + OPC_VHADDW_H_B = 0x70540000, + OPC_VHADDW_W_H = 0x70548000, + OPC_VHADDW_D_W = 0x70550000, + OPC_VHADDW_Q_D = 0x70558000, + OPC_VHSUBW_H_B = 0x70560000, + OPC_VHSUBW_W_H = 0x70568000, + OPC_VHSUBW_D_W = 0x70570000, + OPC_VHSUBW_Q_D = 0x70578000, + OPC_VHADDW_HU_BU = 0x70580000, + OPC_VHADDW_WU_HU = 0x70588000, + OPC_VHADDW_DU_WU = 0x70590000, + OPC_VHADDW_QU_DU = 0x70598000, + OPC_VHSUBW_HU_BU = 0x705a0000, + OPC_VHSUBW_WU_HU = 0x705a8000, + OPC_VHSUBW_DU_WU = 0x705b0000, + OPC_VHSUBW_QU_DU = 0x705b8000, + OPC_VADDA_B = 0x705c0000, + OPC_VADDA_H = 0x705c8000, + OPC_VADDA_W = 0x705d0000, + OPC_VADDA_D = 0x705d8000, + OPC_VABSD_B = 0x70600000, + OPC_VABSD_H = 0x70608000, + OPC_VABSD_W = 0x70610000, + OPC_VABSD_D = 0x70618000, + OPC_VABSD_BU = 0x70620000, + OPC_VABSD_HU = 0x70628000, + OPC_VABSD_WU = 0x70630000, + OPC_VABSD_DU = 0x70638000, + OPC_VAVG_B = 0x70640000, + OPC_VAVG_H = 0x70648000, + OPC_VAVG_W = 0x70650000, + OPC_VAVG_D = 0x70658000, + OPC_VAVG_BU = 0x70660000, + OPC_VAVG_HU = 0x70668000, + OPC_VAVG_WU = 0x70670000, + OPC_VAVG_DU = 0x70678000, + OPC_VAVGR_B = 0x70680000, + OPC_VAVGR_H = 0x70688000, + OPC_VAVGR_W = 0x70690000, + OPC_VAVGR_D = 0x70698000, + OPC_VAVGR_BU = 0x706a0000, + OPC_VAVGR_HU = 0x706a8000, + OPC_VAVGR_WU = 0x706b0000, + OPC_VAVGR_DU = 0x706b8000, + OPC_VMAX_B = 0x70700000, + OPC_VMAX_H = 0x70708000, + OPC_VMAX_W = 0x70710000, + OPC_VMAX_D = 0x70718000, + OPC_VMIN_B = 0x70720000, + OPC_VMIN_H = 0x70728000, + OPC_VMIN_W = 0x70730000, + OPC_VMIN_D = 0x70738000, + OPC_VMAX_BU = 0x70740000, + OPC_VMAX_HU = 0x70748000, + OPC_VMAX_WU = 0x70750000, + OPC_VMAX_DU = 0x70758000, + OPC_VMIN_BU = 0x70760000, + OPC_VMIN_HU = 0x70768000, + OPC_VMIN_WU = 0x70770000, + OPC_VMIN_DU = 0x70778000, + OPC_VMUL_B = 0x70840000, + OPC_VMUL_H = 0x70848000, + OPC_VMUL_W = 0x70850000, + OPC_VMUL_D = 0x70858000, + OPC_VMUH_B = 0x70860000, + OPC_VMUH_H = 0x70868000, + OPC_VMUH_W = 0x70870000, + OPC_VMUH_D = 0x70878000, + OPC_VMUH_BU = 0x70880000, + OPC_VMUH_HU = 0x70888000, + OPC_VMUH_WU = 0x70890000, + OPC_VMUH_DU = 0x70898000, + OPC_VMULWEV_H_B = 0x70900000, + OPC_VMULWEV_W_H = 0x70908000, + OPC_VMULWEV_D_W = 0x70910000, + OPC_VMULWEV_Q_D = 0x70918000, + OPC_VMULWOD_H_B = 0x70920000, + OPC_VMULWOD_W_H = 0x70928000, + OPC_VMULWOD_D_W = 0x70930000, + OPC_VMULWOD_Q_D = 0x70938000, + OPC_VMULWEV_H_BU = 0x70980000, + OPC_VMULWEV_W_HU = 0x70988000, + OPC_VMULWEV_D_WU = 0x70990000, + OPC_VMULWEV_Q_DU = 0x70998000, + OPC_VMULWOD_H_BU = 0x709a0000, + OPC_VMULWOD_W_HU = 0x709a8000, + OPC_VMULWOD_D_WU = 0x709b0000, + OPC_VMULWOD_Q_DU = 0x709b8000, + OPC_VMULWEV_H_BU_B = 0x70a00000, + OPC_VMULWEV_W_HU_H = 0x70a08000, + OPC_VMULWEV_D_WU_W = 0x70a10000, + OPC_VMULWEV_Q_DU_D = 0x70a18000, + OPC_VMULWOD_H_BU_B = 0x70a20000, + OPC_VMULWOD_W_HU_H = 0x70a28000, + OPC_VMULWOD_D_WU_W = 0x70a30000, + OPC_VMULWOD_Q_DU_D = 0x70a38000, + OPC_VMADD_B = 0x70a80000, + OPC_VMADD_H = 0x70a88000, + OPC_VMADD_W = 0x70a90000, + OPC_VMADD_D = 0x70a98000, + OPC_VMSUB_B = 0x70aa0000, + OPC_VMSUB_H = 0x70aa8000, + OPC_VMSUB_W = 0x70ab0000, + OPC_VMSUB_D = 0x70ab8000, + OPC_VMADDWEV_H_B = 0x70ac0000, + OPC_VMADDWEV_W_H = 0x70ac8000, + OPC_VMADDWEV_D_W = 0x70ad0000, + OPC_VMADDWEV_Q_D = 0x70ad8000, + OPC_VMADDWOD_H_B = 0x70ae0000, + OPC_VMADDWOD_W_H = 0x70ae8000, + OPC_VMADDWOD_D_W = 0x70af0000, + OPC_VMADDWOD_Q_D = 0x70af8000, + OPC_VMADDWEV_H_BU = 0x70b40000, + OPC_VMADDWEV_W_HU = 0x70b48000, + OPC_VMADDWEV_D_WU = 0x70b50000, + OPC_VMADDWEV_Q_DU = 0x70b58000, + OPC_VMADDWOD_H_BU = 0x70b60000, + OPC_VMADDWOD_W_HU = 0x70b68000, + OPC_VMADDWOD_D_WU = 0x70b70000, + OPC_VMADDWOD_Q_DU = 0x70b78000, + OPC_VMADDWEV_H_BU_B = 0x70bc0000, + OPC_VMADDWEV_W_HU_H = 0x70bc8000, + OPC_VMADDWEV_D_WU_W = 0x70bd0000, + OPC_VMADDWEV_Q_DU_D = 0x70bd8000, + OPC_VMADDWOD_H_BU_B = 0x70be0000, + OPC_VMADDWOD_W_HU_H = 0x70be8000, + OPC_VMADDWOD_D_WU_W = 0x70bf0000, + OPC_VMADDWOD_Q_DU_D = 0x70bf8000, + OPC_VDIV_B = 0x70e00000, + OPC_VDIV_H = 0x70e08000, + OPC_VDIV_W = 0x70e10000, + OPC_VDIV_D = 0x70e18000, + OPC_VMOD_B = 0x70e20000, + OPC_VMOD_H = 0x70e28000, + OPC_VMOD_W = 0x70e30000, + OPC_VMOD_D = 0x70e38000, + OPC_VDIV_BU = 0x70e40000, + OPC_VDIV_HU = 0x70e48000, + OPC_VDIV_WU = 0x70e50000, + OPC_VDIV_DU = 0x70e58000, + OPC_VMOD_BU = 0x70e60000, + OPC_VMOD_HU = 0x70e68000, + OPC_VMOD_WU = 0x70e70000, + OPC_VMOD_DU = 0x70e78000, + OPC_VSLL_B = 0x70e80000, + OPC_VSLL_H = 0x70e88000, + OPC_VSLL_W = 0x70e90000, + OPC_VSLL_D = 0x70e98000, + OPC_VSRL_B = 0x70ea0000, + OPC_VSRL_H = 0x70ea8000, + OPC_VSRL_W = 0x70eb0000, + OPC_VSRL_D = 0x70eb8000, + OPC_VSRA_B = 0x70ec0000, + OPC_VSRA_H = 0x70ec8000, + OPC_VSRA_W = 0x70ed0000, + OPC_VSRA_D = 0x70ed8000, + OPC_VROTR_B = 0x70ee0000, + OPC_VROTR_H = 0x70ee8000, + OPC_VROTR_W = 0x70ef0000, + OPC_VROTR_D = 0x70ef8000, + OPC_VSRLR_B = 0x70f00000, + OPC_VSRLR_H = 0x70f08000, + OPC_VSRLR_W = 0x70f10000, + OPC_VSRLR_D = 0x70f18000, + OPC_VSRAR_B = 0x70f20000, + OPC_VSRAR_H = 0x70f28000, + OPC_VSRAR_W = 0x70f30000, + OPC_VSRAR_D = 0x70f38000, + OPC_VSRLN_B_H = 0x70f48000, + OPC_VSRLN_H_W = 0x70f50000, + OPC_VSRLN_W_D = 0x70f58000, + OPC_VSRAN_B_H = 0x70f68000, + OPC_VSRAN_H_W = 0x70f70000, + OPC_VSRAN_W_D = 0x70f78000, + OPC_VSRLRN_B_H = 0x70f88000, + OPC_VSRLRN_H_W = 0x70f90000, + OPC_VSRLRN_W_D = 0x70f98000, + OPC_VSRARN_B_H = 0x70fa8000, + OPC_VSRARN_H_W = 0x70fb0000, + OPC_VSRARN_W_D = 0x70fb8000, + OPC_VSSRLN_B_H = 0x70fc8000, + OPC_VSSRLN_H_W = 0x70fd0000, + OPC_VSSRLN_W_D = 0x70fd8000, + OPC_VSSRAN_B_H = 0x70fe8000, + OPC_VSSRAN_H_W = 0x70ff0000, + OPC_VSSRAN_W_D = 0x70ff8000, + OPC_VSSRLRN_B_H = 0x71008000, + OPC_VSSRLRN_H_W = 0x71010000, + OPC_VSSRLRN_W_D = 0x71018000, + OPC_VSSRARN_B_H = 0x71028000, + OPC_VSSRARN_H_W = 0x71030000, + OPC_VSSRARN_W_D = 0x71038000, + OPC_VSSRLN_BU_H = 0x71048000, + OPC_VSSRLN_HU_W = 0x71050000, + OPC_VSSRLN_WU_D = 0x71058000, + OPC_VSSRAN_BU_H = 0x71068000, + OPC_VSSRAN_HU_W = 0x71070000, + OPC_VSSRAN_WU_D = 0x71078000, + OPC_VSSRLRN_BU_H = 0x71088000, + OPC_VSSRLRN_HU_W = 0x71090000, + OPC_VSSRLRN_WU_D = 0x71098000, + OPC_VSSRARN_BU_H = 0x710a8000, + OPC_VSSRARN_HU_W = 0x710b0000, + OPC_VSSRARN_WU_D = 0x710b8000, + OPC_VBITCLR_B = 0x710c0000, + OPC_VBITCLR_H = 0x710c8000, + OPC_VBITCLR_W = 0x710d0000, + OPC_VBITCLR_D = 0x710d8000, + OPC_VBITSET_B = 0x710e0000, + OPC_VBITSET_H = 0x710e8000, + OPC_VBITSET_W = 0x710f0000, + OPC_VBITSET_D = 0x710f8000, + OPC_VBITREV_B = 0x71100000, + OPC_VBITREV_H = 0x71108000, + OPC_VBITREV_W = 0x71110000, + OPC_VBITREV_D = 0x71118000, + OPC_VPACKEV_B = 0x71160000, + OPC_VPACKEV_H = 0x71168000, + OPC_VPACKEV_W = 0x71170000, + OPC_VPACKEV_D = 0x71178000, + OPC_VPACKOD_B = 0x71180000, + OPC_VPACKOD_H = 0x71188000, + OPC_VPACKOD_W = 0x71190000, + OPC_VPACKOD_D = 0x71198000, + OPC_VILVL_B = 0x711a0000, + OPC_VILVL_H = 0x711a8000, + OPC_VILVL_W = 0x711b0000, + OPC_VILVL_D = 0x711b8000, + OPC_VILVH_B = 0x711c0000, + OPC_VILVH_H = 0x711c8000, + OPC_VILVH_W = 0x711d0000, + OPC_VILVH_D = 0x711d8000, + OPC_VPICKEV_B = 0x711e0000, + OPC_VPICKEV_H = 0x711e8000, + OPC_VPICKEV_W = 0x711f0000, + OPC_VPICKEV_D = 0x711f8000, + OPC_VPICKOD_B = 0x71200000, + OPC_VPICKOD_H = 0x71208000, + OPC_VPICKOD_W = 0x71210000, + OPC_VPICKOD_D = 0x71218000, + OPC_VREPLVE_B = 0x71220000, + OPC_VREPLVE_H = 0x71228000, + OPC_VREPLVE_W = 0x71230000, + OPC_VREPLVE_D = 0x71238000, + OPC_VAND_V = 0x71260000, + OPC_VOR_V = 0x71268000, + OPC_VXOR_V = 0x71270000, + OPC_VNOR_V = 0x71278000, + OPC_VANDN_V = 0x71280000, + OPC_VORN_V = 0x71288000, + OPC_VFRSTP_B = 0x712b0000, + OPC_VFRSTP_H = 0x712b8000, + OPC_VADD_Q = 0x712d0000, + OPC_VSUB_Q = 0x712d8000, + OPC_VSIGNCOV_B = 0x712e0000, + OPC_VSIGNCOV_H = 0x712e8000, + OPC_VSIGNCOV_W = 0x712f0000, + OPC_VSIGNCOV_D = 0x712f8000, + OPC_VFADD_S = 0x71308000, + OPC_VFADD_D = 0x71310000, + OPC_VFSUB_S = 0x71328000, + OPC_VFSUB_D = 0x71330000, + OPC_VFMUL_S = 0x71388000, + OPC_VFMUL_D = 0x71390000, + OPC_VFDIV_S = 0x713a8000, + OPC_VFDIV_D = 0x713b0000, + OPC_VFMAX_S = 0x713c8000, + OPC_VFMAX_D = 0x713d0000, + OPC_VFMIN_S = 0x713e8000, + OPC_VFMIN_D = 0x713f0000, + OPC_VFMAXA_S = 0x71408000, + OPC_VFMAXA_D = 0x71410000, + OPC_VFMINA_S = 0x71428000, + OPC_VFMINA_D = 0x71430000, + OPC_VFCVT_H_S = 0x71460000, + OPC_VFCVT_S_D = 0x71468000, + OPC_VFFINT_S_L = 0x71480000, + OPC_VFTINT_W_D = 0x71498000, + OPC_VFTINTRM_W_D = 0x714a0000, + OPC_VFTINTRP_W_D = 0x714a8000, + OPC_VFTINTRZ_W_D = 0x714b0000, + OPC_VFTINTRNE_W_D = 0x714b8000, + OPC_VSHUF_H = 0x717a8000, + OPC_VSHUF_W = 0x717b0000, + OPC_VSHUF_D = 0x717b8000, + OPC_VSEQI_B = 0x72800000, + OPC_VSEQI_H = 0x72808000, + OPC_VSEQI_W = 0x72810000, + OPC_VSEQI_D = 0x72818000, + OPC_VSLEI_B = 0x72820000, + OPC_VSLEI_H = 0x72828000, + OPC_VSLEI_W = 0x72830000, + OPC_VSLEI_D = 0x72838000, + OPC_VSLEI_BU = 0x72840000, + OPC_VSLEI_HU = 0x72848000, + OPC_VSLEI_WU = 0x72850000, + OPC_VSLEI_DU = 0x72858000, + OPC_VSLTI_B = 0x72860000, + OPC_VSLTI_H = 0x72868000, + OPC_VSLTI_W = 0x72870000, + OPC_VSLTI_D = 0x72878000, + OPC_VSLTI_BU = 0x72880000, + OPC_VSLTI_HU = 0x72888000, + OPC_VSLTI_WU = 0x72890000, + OPC_VSLTI_DU = 0x72898000, + OPC_VADDI_BU = 0x728a0000, + OPC_VADDI_HU = 0x728a8000, + OPC_VADDI_WU = 0x728b0000, + OPC_VADDI_DU = 0x728b8000, + OPC_VSUBI_BU = 0x728c0000, + OPC_VSUBI_HU = 0x728c8000, + OPC_VSUBI_WU = 0x728d0000, + OPC_VSUBI_DU = 0x728d8000, + OPC_VBSLL_V = 0x728e0000, + OPC_VBSRL_V = 0x728e8000, + OPC_VMAXI_B = 0x72900000, + OPC_VMAXI_H = 0x72908000, + OPC_VMAXI_W = 0x72910000, + OPC_VMAXI_D = 0x72918000, + OPC_VMINI_B = 0x72920000, + OPC_VMINI_H = 0x72928000, + OPC_VMINI_W = 0x72930000, + OPC_VMINI_D = 0x72938000, + OPC_VMAXI_BU = 0x72940000, + OPC_VMAXI_HU = 0x72948000, + OPC_VMAXI_WU = 0x72950000, + OPC_VMAXI_DU = 0x72958000, + OPC_VMINI_BU = 0x72960000, + OPC_VMINI_HU = 0x72968000, + OPC_VMINI_WU = 0x72970000, + OPC_VMINI_DU = 0x72978000, + OPC_VFRSTPI_B = 0x729a0000, + OPC_VFRSTPI_H = 0x729a8000, + OPC_VCLO_B = 0x729c0000, + OPC_VCLO_H = 0x729c0400, + OPC_VCLO_W = 0x729c0800, + OPC_VCLO_D = 0x729c0c00, + OPC_VCLZ_B = 0x729c1000, + OPC_VCLZ_H = 0x729c1400, + OPC_VCLZ_W = 0x729c1800, + OPC_VCLZ_D = 0x729c1c00, + OPC_VPCNT_B = 0x729c2000, + OPC_VPCNT_H = 0x729c2400, + OPC_VPCNT_W = 0x729c2800, + OPC_VPCNT_D = 0x729c2c00, + OPC_VNEG_B = 0x729c3000, + OPC_VNEG_H = 0x729c3400, + OPC_VNEG_W = 0x729c3800, + OPC_VNEG_D = 0x729c3c00, + OPC_VMSKLTZ_B = 0x729c4000, + OPC_VMSKLTZ_H = 0x729c4400, + OPC_VMSKLTZ_W = 0x729c4800, + OPC_VMSKLTZ_D = 0x729c4c00, + OPC_VMSKGEZ_B = 0x729c5000, + OPC_VMSKNZ_B = 0x729c6000, + OPC_VSETEQZ_V = 0x729c9800, + OPC_VSETNEZ_V = 0x729c9c00, + OPC_VSETANYEQZ_B = 0x729ca000, + OPC_VSETANYEQZ_H = 0x729ca400, + OPC_VSETANYEQZ_W = 0x729ca800, + OPC_VSETANYEQZ_D = 0x729cac00, + OPC_VSETALLNEZ_B = 0x729cb000, + OPC_VSETALLNEZ_H = 0x729cb400, + OPC_VSETALLNEZ_W = 0x729cb800, + OPC_VSETALLNEZ_D = 0x729cbc00, + OPC_VFLOGB_S = 0x729cc400, + OPC_VFLOGB_D = 0x729cc800, + OPC_VFCLASS_S = 0x729cd400, + OPC_VFCLASS_D = 0x729cd800, + OPC_VFSQRT_S = 0x729ce400, + OPC_VFSQRT_D = 0x729ce800, + OPC_VFRECIP_S = 0x729cf400, + OPC_VFRECIP_D = 0x729cf800, + OPC_VFRSQRT_S = 0x729d0400, + OPC_VFRSQRT_D = 0x729d0800, + OPC_VFRINT_S = 0x729d3400, + OPC_VFRINT_D = 0x729d3800, + OPC_VFRINTRM_S = 0x729d4400, + OPC_VFRINTRM_D = 0x729d4800, + OPC_VFRINTRP_S = 0x729d5400, + OPC_VFRINTRP_D = 0x729d5800, + OPC_VFRINTRZ_S = 0x729d6400, + OPC_VFRINTRZ_D = 0x729d6800, + OPC_VFRINTRNE_S = 0x729d7400, + OPC_VFRINTRNE_D = 0x729d7800, + OPC_VFCVTL_S_H = 0x729de800, + OPC_VFCVTH_S_H = 0x729dec00, + OPC_VFCVTL_D_S = 0x729df000, + OPC_VFCVTH_D_S = 0x729df400, + OPC_VFFINT_S_W = 0x729e0000, + OPC_VFFINT_S_WU = 0x729e0400, + OPC_VFFINT_D_L = 0x729e0800, + OPC_VFFINT_D_LU = 0x729e0c00, + OPC_VFFINTL_D_W = 0x729e1000, + OPC_VFFINTH_D_W = 0x729e1400, + OPC_VFTINT_W_S = 0x729e3000, + OPC_VFTINT_L_D = 0x729e3400, + OPC_VFTINTRM_W_S = 0x729e3800, + OPC_VFTINTRM_L_D = 0x729e3c00, + OPC_VFTINTRP_W_S = 0x729e4000, + OPC_VFTINTRP_L_D = 0x729e4400, + OPC_VFTINTRZ_W_S = 0x729e4800, + OPC_VFTINTRZ_L_D = 0x729e4c00, + OPC_VFTINTRNE_W_S = 0x729e5000, + OPC_VFTINTRNE_L_D = 0x729e5400, + OPC_VFTINT_WU_S = 0x729e5800, + OPC_VFTINT_LU_D = 0x729e5c00, + OPC_VFTINTRZ_WU_S = 0x729e7000, + OPC_VFTINTRZ_LU_D = 0x729e7400, + OPC_VFTINTL_L_S = 0x729e8000, + OPC_VFTINTH_L_S = 0x729e8400, + OPC_VFTINTRML_L_S = 0x729e8800, + OPC_VFTINTRMH_L_S = 0x729e8c00, + OPC_VFTINTRPL_L_S = 0x729e9000, + OPC_VFTINTRPH_L_S = 0x729e9400, + OPC_VFTINTRZL_L_S = 0x729e9800, + OPC_VFTINTRZH_L_S = 0x729e9c00, + OPC_VFTINTRNEL_L_S = 0x729ea000, + OPC_VFTINTRNEH_L_S = 0x729ea400, + OPC_VEXTH_H_B = 0x729ee000, + OPC_VEXTH_W_H = 0x729ee400, + OPC_VEXTH_D_W = 0x729ee800, + OPC_VEXTH_Q_D = 0x729eec00, + OPC_VEXTH_HU_BU = 0x729ef000, + OPC_VEXTH_WU_HU = 0x729ef400, + OPC_VEXTH_DU_WU = 0x729ef800, + OPC_VEXTH_QU_DU = 0x729efc00, + OPC_VREPLGR2VR_B = 0x729f0000, + OPC_VREPLGR2VR_H = 0x729f0400, + OPC_VREPLGR2VR_W = 0x729f0800, + OPC_VREPLGR2VR_D = 0x729f0c00, + OPC_VROTRI_B = 0x72a02000, + OPC_VROTRI_H = 0x72a04000, + OPC_VROTRI_W = 0x72a08000, + OPC_VROTRI_D = 0x72a10000, + OPC_VSRLRI_B = 0x72a42000, + OPC_VSRLRI_H = 0x72a44000, + OPC_VSRLRI_W = 0x72a48000, + OPC_VSRLRI_D = 0x72a50000, + OPC_VSRARI_B = 0x72a82000, + OPC_VSRARI_H = 0x72a84000, + OPC_VSRARI_W = 0x72a88000, + OPC_VSRARI_D = 0x72a90000, + OPC_VINSGR2VR_B = 0x72eb8000, + OPC_VINSGR2VR_H = 0x72ebc000, + OPC_VINSGR2VR_W = 0x72ebe000, + OPC_VINSGR2VR_D = 0x72ebf000, + OPC_VPICKVE2GR_B = 0x72ef8000, + OPC_VPICKVE2GR_H = 0x72efc000, + OPC_VPICKVE2GR_W = 0x72efe000, + OPC_VPICKVE2GR_D = 0x72eff000, + OPC_VPICKVE2GR_BU = 0x72f38000, + OPC_VPICKVE2GR_HU = 0x72f3c000, + OPC_VPICKVE2GR_WU = 0x72f3e000, + OPC_VPICKVE2GR_DU = 0x72f3f000, + OPC_VREPLVEI_B = 0x72f78000, + OPC_VREPLVEI_H = 0x72f7c000, + OPC_VREPLVEI_W = 0x72f7e000, + OPC_VREPLVEI_D = 0x72f7f000, + OPC_VSLLWIL_H_B = 0x73082000, + OPC_VSLLWIL_W_H = 0x73084000, + OPC_VSLLWIL_D_W = 0x73088000, + OPC_VEXTL_Q_D = 0x73090000, + OPC_VSLLWIL_HU_BU = 0x730c2000, + OPC_VSLLWIL_WU_HU = 0x730c4000, + OPC_VSLLWIL_DU_WU = 0x730c8000, + OPC_VEXTL_QU_DU = 0x730d0000, + OPC_VBITCLRI_B = 0x73102000, + OPC_VBITCLRI_H = 0x73104000, + OPC_VBITCLRI_W = 0x73108000, + OPC_VBITCLRI_D = 0x73110000, + OPC_VBITSETI_B = 0x73142000, + OPC_VBITSETI_H = 0x73144000, + OPC_VBITSETI_W = 0x73148000, + OPC_VBITSETI_D = 0x73150000, + OPC_VBITREVI_B = 0x73182000, + OPC_VBITREVI_H = 0x73184000, + OPC_VBITREVI_W = 0x73188000, + OPC_VBITREVI_D = 0x73190000, + OPC_VSAT_B = 0x73242000, + OPC_VSAT_H = 0x73244000, + OPC_VSAT_W = 0x73248000, + OPC_VSAT_D = 0x73250000, + OPC_VSAT_BU = 0x73282000, + OPC_VSAT_HU = 0x73284000, + OPC_VSAT_WU = 0x73288000, + OPC_VSAT_DU = 0x73290000, + OPC_VSLLI_B = 0x732c2000, + OPC_VSLLI_H = 0x732c4000, + OPC_VSLLI_W = 0x732c8000, + OPC_VSLLI_D = 0x732d0000, + OPC_VSRLI_B = 0x73302000, + OPC_VSRLI_H = 0x73304000, + OPC_VSRLI_W = 0x73308000, + OPC_VSRLI_D = 0x73310000, + OPC_VSRAI_B = 0x73342000, + OPC_VSRAI_H = 0x73344000, + OPC_VSRAI_W = 0x73348000, + OPC_VSRAI_D = 0x73350000, + OPC_VSRLNI_B_H = 0x73404000, + OPC_VSRLNI_H_W = 0x73408000, + OPC_VSRLNI_W_D = 0x73410000, + OPC_VSRLNI_D_Q = 0x73420000, + OPC_VSRLRNI_B_H = 0x73444000, + OPC_VSRLRNI_H_W = 0x73448000, + OPC_VSRLRNI_W_D = 0x73450000, + OPC_VSRLRNI_D_Q = 0x73460000, + OPC_VSSRLNI_B_H = 0x73484000, + OPC_VSSRLNI_H_W = 0x73488000, + OPC_VSSRLNI_W_D = 0x73490000, + OPC_VSSRLNI_D_Q = 0x734a0000, + OPC_VSSRLNI_BU_H = 0x734c4000, + OPC_VSSRLNI_HU_W = 0x734c8000, + OPC_VSSRLNI_WU_D = 0x734d0000, + OPC_VSSRLNI_DU_Q = 0x734e0000, + OPC_VSSRLRNI_B_H = 0x73504000, + OPC_VSSRLRNI_H_W = 0x73508000, + OPC_VSSRLRNI_W_D = 0x73510000, + OPC_VSSRLRNI_D_Q = 0x73520000, + OPC_VSSRLRNI_BU_H = 0x73544000, + OPC_VSSRLRNI_HU_W = 0x73548000, + OPC_VSSRLRNI_WU_D = 0x73550000, + OPC_VSSRLRNI_DU_Q = 0x73560000, + OPC_VSRANI_B_H = 0x73584000, + OPC_VSRANI_H_W = 0x73588000, + OPC_VSRANI_W_D = 0x73590000, + OPC_VSRANI_D_Q = 0x735a0000, + OPC_VSRARNI_B_H = 0x735c4000, + OPC_VSRARNI_H_W = 0x735c8000, + OPC_VSRARNI_W_D = 0x735d0000, + OPC_VSRARNI_D_Q = 0x735e0000, + OPC_VSSRANI_B_H = 0x73604000, + OPC_VSSRANI_H_W = 0x73608000, + OPC_VSSRANI_W_D = 0x73610000, + OPC_VSSRANI_D_Q = 0x73620000, + OPC_VSSRANI_BU_H = 0x73644000, + OPC_VSSRANI_HU_W = 0x73648000, + OPC_VSSRANI_WU_D = 0x73650000, + OPC_VSSRANI_DU_Q = 0x73660000, + OPC_VSSRARNI_B_H = 0x73684000, + OPC_VSSRARNI_H_W = 0x73688000, + OPC_VSSRARNI_W_D = 0x73690000, + OPC_VSSRARNI_D_Q = 0x736a0000, + OPC_VSSRARNI_BU_H = 0x736c4000, + OPC_VSSRARNI_HU_W = 0x736c8000, + OPC_VSSRARNI_WU_D = 0x736d0000, + OPC_VSSRARNI_DU_Q = 0x736e0000, + OPC_VEXTRINS_D = 0x73800000, + OPC_VEXTRINS_W = 0x73840000, + OPC_VEXTRINS_H = 0x73880000, + OPC_VEXTRINS_B = 0x738c0000, + OPC_VSHUF4I_B = 0x73900000, + OPC_VSHUF4I_H = 0x73940000, + OPC_VSHUF4I_W = 0x73980000, + OPC_VSHUF4I_D = 0x739c0000, + OPC_VBITSELI_B = 0x73c40000, + OPC_VANDI_B = 0x73d00000, + OPC_VORI_B = 0x73d40000, + OPC_VXORI_B = 0x73d80000, + OPC_VNORI_B = 0x73dc0000, + OPC_VLDI = 0x73e00000, + OPC_VPERMI_W = 0x73e40000, } LoongArchInsn; static int32_t __attribute__((unused)) @@ -XXX,XX +XXX,XX @@ encode_djk_slots(LoongArchInsn opc, uint32_t d, uint32_t j, uint32_t k) return opc | d | j << 5 | k << 10; } +static int32_t __attribute__((unused)) +encode_djka_slots(LoongArchInsn opc, uint32_t d, uint32_t j, uint32_t k, + uint32_t a) +{ + return opc | d | j << 5 | k << 10 | a << 15; +} + static int32_t __attribute__((unused)) encode_djkm_slots(LoongArchInsn opc, uint32_t d, uint32_t j, uint32_t k, uint32_t m) @@ -XXX,XX +XXX,XX @@ encode_djkm_slots(LoongArchInsn opc, uint32_t d, uint32_t j, uint32_t k, return opc | d | j << 5 | k << 10 | m << 16; } +static int32_t __attribute__((unused)) +encode_djkn_slots(LoongArchInsn opc, uint32_t d, uint32_t j, uint32_t k, + uint32_t n) +{ + return opc | d | j << 5 | k << 10 | n << 18; +} + static int32_t __attribute__((unused)) encode_dk_slots(LoongArchInsn opc, uint32_t d, uint32_t k) { return opc | d | k << 10; } +static int32_t __attribute__((unused)) +encode_cdvj_insn(LoongArchInsn opc, TCGReg cd, TCGReg vj) +{ + tcg_debug_assert(cd >= 0 && cd <= 0x7); + tcg_debug_assert(vj >= 0x20 && vj <= 0x3f); + return encode_dj_slots(opc, cd, vj & 0x1f); +} + static int32_t __attribute__((unused)) encode_dj_insn(LoongArchInsn opc, TCGReg d, TCGReg j) { @@ -XXX,XX +XXX,XX @@ encode_dsj20_insn(LoongArchInsn opc, TCGReg d, int32_t sj20) return encode_dj_slots(opc, d, sj20 & 0xfffff); } +static int32_t __attribute__((unused)) +encode_dvjuk1_insn(LoongArchInsn opc, TCGReg d, TCGReg vj, uint32_t uk1) +{ + tcg_debug_assert(d >= 0 && d <= 0x1f); + tcg_debug_assert(vj >= 0x20 && vj <= 0x3f); + tcg_debug_assert(uk1 <= 0x1); + return encode_djk_slots(opc, d, vj & 0x1f, uk1); +} + +static int32_t __attribute__((unused)) +encode_dvjuk2_insn(LoongArchInsn opc, TCGReg d, TCGReg vj, uint32_t uk2) +{ + tcg_debug_assert(d >= 0 && d <= 0x1f); + tcg_debug_assert(vj >= 0x20 && vj <= 0x3f); + tcg_debug_assert(uk2 <= 0x3); + return encode_djk_slots(opc, d, vj & 0x1f, uk2); +} + +static int32_t __attribute__((unused)) +encode_dvjuk3_insn(LoongArchInsn opc, TCGReg d, TCGReg vj, uint32_t uk3) +{ + tcg_debug_assert(d >= 0 && d <= 0x1f); + tcg_debug_assert(vj >= 0x20 && vj <= 0x3f); + tcg_debug_assert(uk3 <= 0x7); + return encode_djk_slots(opc, d, vj & 0x1f, uk3); +} + +static int32_t __attribute__((unused)) +encode_dvjuk4_insn(LoongArchInsn opc, TCGReg d, TCGReg vj, uint32_t uk4) +{ + tcg_debug_assert(d >= 0 && d <= 0x1f); + tcg_debug_assert(vj >= 0x20 && vj <= 0x3f); + tcg_debug_assert(uk4 <= 0xf); + return encode_djk_slots(opc, d, vj & 0x1f, uk4); +} + static int32_t __attribute__((unused)) encode_sd10k16_insn(LoongArchInsn opc, int32_t sd10k16) { @@ -XXX,XX +XXX,XX @@ encode_ud15_insn(LoongArchInsn opc, uint32_t ud15) return encode_d_slot(opc, ud15); } +static int32_t __attribute__((unused)) +encode_vdj_insn(LoongArchInsn opc, TCGReg vd, TCGReg j) +{ + tcg_debug_assert(vd >= 0x20 && vd <= 0x3f); + tcg_debug_assert(j >= 0 && j <= 0x1f); + return encode_dj_slots(opc, vd & 0x1f, j); +} + +static int32_t __attribute__((unused)) +encode_vdjk_insn(LoongArchInsn opc, TCGReg vd, TCGReg j, TCGReg k) +{ + tcg_debug_assert(vd >= 0x20 && vd <= 0x3f); + tcg_debug_assert(j >= 0 && j <= 0x1f); + tcg_debug_assert(k >= 0 && k <= 0x1f); + return encode_djk_slots(opc, vd & 0x1f, j, k); +} + +static int32_t __attribute__((unused)) +encode_vdjsk10_insn(LoongArchInsn opc, TCGReg vd, TCGReg j, int32_t sk10) +{ + tcg_debug_assert(vd >= 0x20 && vd <= 0x3f); + tcg_debug_assert(j >= 0 && j <= 0x1f); + tcg_debug_assert(sk10 >= -0x200 && sk10 <= 0x1ff); + return encode_djk_slots(opc, vd & 0x1f, j, sk10 & 0x3ff); +} + +static int32_t __attribute__((unused)) +encode_vdjsk11_insn(LoongArchInsn opc, TCGReg vd, TCGReg j, int32_t sk11) +{ + tcg_debug_assert(vd >= 0x20 && vd <= 0x3f); + tcg_debug_assert(j >= 0 && j <= 0x1f); + tcg_debug_assert(sk11 >= -0x400 && sk11 <= 0x3ff); + return encode_djk_slots(opc, vd & 0x1f, j, sk11 & 0x7ff); +} + +static int32_t __attribute__((unused)) +encode_vdjsk12_insn(LoongArchInsn opc, TCGReg vd, TCGReg j, int32_t sk12) +{ + tcg_debug_assert(vd >= 0x20 && vd <= 0x3f); + tcg_debug_assert(j >= 0 && j <= 0x1f); + tcg_debug_assert(sk12 >= -0x800 && sk12 <= 0x7ff); + return encode_djk_slots(opc, vd & 0x1f, j, sk12 & 0xfff); +} + +static int32_t __attribute__((unused)) +encode_vdjsk8un1_insn(LoongArchInsn opc, TCGReg vd, TCGReg j, int32_t sk8, + uint32_t un1) +{ + tcg_debug_assert(vd >= 0x20 && vd <= 0x3f); + tcg_debug_assert(j >= 0 && j <= 0x1f); + tcg_debug_assert(sk8 >= -0x80 && sk8 <= 0x7f); + tcg_debug_assert(un1 <= 0x1); + return encode_djkn_slots(opc, vd & 0x1f, j, sk8 & 0xff, un1); +} + +static int32_t __attribute__((unused)) +encode_vdjsk8un2_insn(LoongArchInsn opc, TCGReg vd, TCGReg j, int32_t sk8, + uint32_t un2) +{ + tcg_debug_assert(vd >= 0x20 && vd <= 0x3f); + tcg_debug_assert(j >= 0 && j <= 0x1f); + tcg_debug_assert(sk8 >= -0x80 && sk8 <= 0x7f); + tcg_debug_assert(un2 <= 0x3); + return encode_djkn_slots(opc, vd & 0x1f, j, sk8 & 0xff, un2); +} + +static int32_t __attribute__((unused)) +encode_vdjsk8un3_insn(LoongArchInsn opc, TCGReg vd, TCGReg j, int32_t sk8, + uint32_t un3) +{ + tcg_debug_assert(vd >= 0x20 && vd <= 0x3f); + tcg_debug_assert(j >= 0 && j <= 0x1f); + tcg_debug_assert(sk8 >= -0x80 && sk8 <= 0x7f); + tcg_debug_assert(un3 <= 0x7); + return encode_djkn_slots(opc, vd & 0x1f, j, sk8 & 0xff, un3); +} + +static int32_t __attribute__((unused)) +encode_vdjsk8un4_insn(LoongArchInsn opc, TCGReg vd, TCGReg j, int32_t sk8, + uint32_t un4) +{ + tcg_debug_assert(vd >= 0x20 && vd <= 0x3f); + tcg_debug_assert(j >= 0 && j <= 0x1f); + tcg_debug_assert(sk8 >= -0x80 && sk8 <= 0x7f); + tcg_debug_assert(un4 <= 0xf); + return encode_djkn_slots(opc, vd & 0x1f, j, sk8 & 0xff, un4); +} + +static int32_t __attribute__((unused)) +encode_vdjsk9_insn(LoongArchInsn opc, TCGReg vd, TCGReg j, int32_t sk9) +{ + tcg_debug_assert(vd >= 0x20 && vd <= 0x3f); + tcg_debug_assert(j >= 0 && j <= 0x1f); + tcg_debug_assert(sk9 >= -0x100 && sk9 <= 0xff); + return encode_djk_slots(opc, vd & 0x1f, j, sk9 & 0x1ff); +} + +static int32_t __attribute__((unused)) +encode_vdjuk1_insn(LoongArchInsn opc, TCGReg vd, TCGReg j, uint32_t uk1) +{ + tcg_debug_assert(vd >= 0x20 && vd <= 0x3f); + tcg_debug_assert(j >= 0 && j <= 0x1f); + tcg_debug_assert(uk1 <= 0x1); + return encode_djk_slots(opc, vd & 0x1f, j, uk1); +} + +static int32_t __attribute__((unused)) +encode_vdjuk2_insn(LoongArchInsn opc, TCGReg vd, TCGReg j, uint32_t uk2) +{ + tcg_debug_assert(vd >= 0x20 && vd <= 0x3f); + tcg_debug_assert(j >= 0 && j <= 0x1f); + tcg_debug_assert(uk2 <= 0x3); + return encode_djk_slots(opc, vd & 0x1f, j, uk2); +} + +static int32_t __attribute__((unused)) +encode_vdjuk3_insn(LoongArchInsn opc, TCGReg vd, TCGReg j, uint32_t uk3) +{ + tcg_debug_assert(vd >= 0x20 && vd <= 0x3f); + tcg_debug_assert(j >= 0 && j <= 0x1f); + tcg_debug_assert(uk3 <= 0x7); + return encode_djk_slots(opc, vd & 0x1f, j, uk3); +} + +static int32_t __attribute__((unused)) +encode_vdjuk4_insn(LoongArchInsn opc, TCGReg vd, TCGReg j, uint32_t uk4) +{ + tcg_debug_assert(vd >= 0x20 && vd <= 0x3f); + tcg_debug_assert(j >= 0 && j <= 0x1f); + tcg_debug_assert(uk4 <= 0xf); + return encode_djk_slots(opc, vd & 0x1f, j, uk4); +} + +static int32_t __attribute__((unused)) +encode_vdsj13_insn(LoongArchInsn opc, TCGReg vd, int32_t sj13) +{ + tcg_debug_assert(vd >= 0x20 && vd <= 0x3f); + tcg_debug_assert(sj13 >= -0x1000 && sj13 <= 0xfff); + return encode_dj_slots(opc, vd & 0x1f, sj13 & 0x1fff); +} + +static int32_t __attribute__((unused)) +encode_vdvj_insn(LoongArchInsn opc, TCGReg vd, TCGReg vj) +{ + tcg_debug_assert(vd >= 0x20 && vd <= 0x3f); + tcg_debug_assert(vj >= 0x20 && vj <= 0x3f); + return encode_dj_slots(opc, vd & 0x1f, vj & 0x1f); +} + +static int32_t __attribute__((unused)) +encode_vdvjk_insn(LoongArchInsn opc, TCGReg vd, TCGReg vj, TCGReg k) +{ + tcg_debug_assert(vd >= 0x20 && vd <= 0x3f); + tcg_debug_assert(vj >= 0x20 && vj <= 0x3f); + tcg_debug_assert(k >= 0 && k <= 0x1f); + return encode_djk_slots(opc, vd & 0x1f, vj & 0x1f, k); +} + +static int32_t __attribute__((unused)) +encode_vdvjsk5_insn(LoongArchInsn opc, TCGReg vd, TCGReg vj, int32_t sk5) +{ + tcg_debug_assert(vd >= 0x20 && vd <= 0x3f); + tcg_debug_assert(vj >= 0x20 && vj <= 0x3f); + tcg_debug_assert(sk5 >= -0x10 && sk5 <= 0xf); + return encode_djk_slots(opc, vd & 0x1f, vj & 0x1f, sk5 & 0x1f); +} + +static int32_t __attribute__((unused)) +encode_vdvjuk1_insn(LoongArchInsn opc, TCGReg vd, TCGReg vj, uint32_t uk1) +{ + tcg_debug_assert(vd >= 0x20 && vd <= 0x3f); + tcg_debug_assert(vj >= 0x20 && vj <= 0x3f); + tcg_debug_assert(uk1 <= 0x1); + return encode_djk_slots(opc, vd & 0x1f, vj & 0x1f, uk1); +} + +static int32_t __attribute__((unused)) +encode_vdvjuk2_insn(LoongArchInsn opc, TCGReg vd, TCGReg vj, uint32_t uk2) +{ + tcg_debug_assert(vd >= 0x20 && vd <= 0x3f); + tcg_debug_assert(vj >= 0x20 && vj <= 0x3f); + tcg_debug_assert(uk2 <= 0x3); + return encode_djk_slots(opc, vd & 0x1f, vj & 0x1f, uk2); +} + +static int32_t __attribute__((unused)) +encode_vdvjuk3_insn(LoongArchInsn opc, TCGReg vd, TCGReg vj, uint32_t uk3) +{ + tcg_debug_assert(vd >= 0x20 && vd <= 0x3f); + tcg_debug_assert(vj >= 0x20 && vj <= 0x3f); + tcg_debug_assert(uk3 <= 0x7); + return encode_djk_slots(opc, vd & 0x1f, vj & 0x1f, uk3); +} + +static int32_t __attribute__((unused)) +encode_vdvjuk4_insn(LoongArchInsn opc, TCGReg vd, TCGReg vj, uint32_t uk4) +{ + tcg_debug_assert(vd >= 0x20 && vd <= 0x3f); + tcg_debug_assert(vj >= 0x20 && vj <= 0x3f); + tcg_debug_assert(uk4 <= 0xf); + return encode_djk_slots(opc, vd & 0x1f, vj & 0x1f, uk4); +} + +static int32_t __attribute__((unused)) +encode_vdvjuk5_insn(LoongArchInsn opc, TCGReg vd, TCGReg vj, uint32_t uk5) +{ + tcg_debug_assert(vd >= 0x20 && vd <= 0x3f); + tcg_debug_assert(vj >= 0x20 && vj <= 0x3f); + tcg_debug_assert(uk5 <= 0x1f); + return encode_djk_slots(opc, vd & 0x1f, vj & 0x1f, uk5); +} + +static int32_t __attribute__((unused)) +encode_vdvjuk6_insn(LoongArchInsn opc, TCGReg vd, TCGReg vj, uint32_t uk6) +{ + tcg_debug_assert(vd >= 0x20 && vd <= 0x3f); + tcg_debug_assert(vj >= 0x20 && vj <= 0x3f); + tcg_debug_assert(uk6 <= 0x3f); + return encode_djk_slots(opc, vd & 0x1f, vj & 0x1f, uk6); +} + +static int32_t __attribute__((unused)) +encode_vdvjuk7_insn(LoongArchInsn opc, TCGReg vd, TCGReg vj, uint32_t uk7) +{ + tcg_debug_assert(vd >= 0x20 && vd <= 0x3f); + tcg_debug_assert(vj >= 0x20 && vj <= 0x3f); + tcg_debug_assert(uk7 <= 0x7f); + return encode_djk_slots(opc, vd & 0x1f, vj & 0x1f, uk7); +} + +static int32_t __attribute__((unused)) +encode_vdvjuk8_insn(LoongArchInsn opc, TCGReg vd, TCGReg vj, uint32_t uk8) +{ + tcg_debug_assert(vd >= 0x20 && vd <= 0x3f); + tcg_debug_assert(vj >= 0x20 && vj <= 0x3f); + tcg_debug_assert(uk8 <= 0xff); + return encode_djk_slots(opc, vd & 0x1f, vj & 0x1f, uk8); +} + +static int32_t __attribute__((unused)) +encode_vdvjvk_insn(LoongArchInsn opc, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_debug_assert(vd >= 0x20 && vd <= 0x3f); + tcg_debug_assert(vj >= 0x20 && vj <= 0x3f); + tcg_debug_assert(vk >= 0x20 && vk <= 0x3f); + return encode_djk_slots(opc, vd & 0x1f, vj & 0x1f, vk & 0x1f); +} + +static int32_t __attribute__((unused)) +encode_vdvjvkva_insn(LoongArchInsn opc, TCGReg vd, TCGReg vj, TCGReg vk, + TCGReg va) +{ + tcg_debug_assert(vd >= 0x20 && vd <= 0x3f); + tcg_debug_assert(vj >= 0x20 && vj <= 0x3f); + tcg_debug_assert(vk >= 0x20 && vk <= 0x3f); + tcg_debug_assert(va >= 0x20 && va <= 0x3f); + return encode_djka_slots(opc, vd & 0x1f, vj & 0x1f, vk & 0x1f, va & 0x1f); +} + /* Emits the `clz.w d, j` instruction. */ static void __attribute__((unused)) tcg_out_opc_clz_w(TCGContext *s, TCGReg d, TCGReg j) @@ -XXX,XX +XXX,XX @@ tcg_out_opc_xori(TCGContext *s, TCGReg d, TCGReg j, uint32_t uk12) tcg_out32(s, encode_djuk12_insn(OPC_XORI, d, j, uk12)); } +/* Emits the `vfmadd.s vd, vj, vk, va` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfmadd_s(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk, TCGReg va) +{ + tcg_out32(s, encode_vdvjvkva_insn(OPC_VFMADD_S, vd, vj, vk, va)); +} + +/* Emits the `vfmadd.d vd, vj, vk, va` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfmadd_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk, TCGReg va) +{ + tcg_out32(s, encode_vdvjvkva_insn(OPC_VFMADD_D, vd, vj, vk, va)); +} + +/* Emits the `vfmsub.s vd, vj, vk, va` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfmsub_s(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk, TCGReg va) +{ + tcg_out32(s, encode_vdvjvkva_insn(OPC_VFMSUB_S, vd, vj, vk, va)); +} + +/* Emits the `vfmsub.d vd, vj, vk, va` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfmsub_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk, TCGReg va) +{ + tcg_out32(s, encode_vdvjvkva_insn(OPC_VFMSUB_D, vd, vj, vk, va)); +} + +/* Emits the `vfnmadd.s vd, vj, vk, va` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfnmadd_s(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk, TCGReg va) +{ + tcg_out32(s, encode_vdvjvkva_insn(OPC_VFNMADD_S, vd, vj, vk, va)); +} + +/* Emits the `vfnmadd.d vd, vj, vk, va` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfnmadd_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk, TCGReg va) +{ + tcg_out32(s, encode_vdvjvkva_insn(OPC_VFNMADD_D, vd, vj, vk, va)); +} + +/* Emits the `vfnmsub.s vd, vj, vk, va` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfnmsub_s(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk, TCGReg va) +{ + tcg_out32(s, encode_vdvjvkva_insn(OPC_VFNMSUB_S, vd, vj, vk, va)); +} + +/* Emits the `vfnmsub.d vd, vj, vk, va` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfnmsub_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk, TCGReg va) +{ + tcg_out32(s, encode_vdvjvkva_insn(OPC_VFNMSUB_D, vd, vj, vk, va)); +} + +/* Emits the `vfcmp.caf.s vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfcmp_caf_s(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VFCMP_CAF_S, vd, vj, vk)); +} + +/* Emits the `vfcmp.saf.s vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfcmp_saf_s(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VFCMP_SAF_S, vd, vj, vk)); +} + +/* Emits the `vfcmp.clt.s vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfcmp_clt_s(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VFCMP_CLT_S, vd, vj, vk)); +} + +/* Emits the `vfcmp.slt.s vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfcmp_slt_s(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VFCMP_SLT_S, vd, vj, vk)); +} + +/* Emits the `vfcmp.ceq.s vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfcmp_ceq_s(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VFCMP_CEQ_S, vd, vj, vk)); +} + +/* Emits the `vfcmp.seq.s vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfcmp_seq_s(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VFCMP_SEQ_S, vd, vj, vk)); +} + +/* Emits the `vfcmp.cle.s vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfcmp_cle_s(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VFCMP_CLE_S, vd, vj, vk)); +} + +/* Emits the `vfcmp.sle.s vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfcmp_sle_s(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VFCMP_SLE_S, vd, vj, vk)); +} + +/* Emits the `vfcmp.cun.s vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfcmp_cun_s(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VFCMP_CUN_S, vd, vj, vk)); +} + +/* Emits the `vfcmp.sun.s vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfcmp_sun_s(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VFCMP_SUN_S, vd, vj, vk)); +} + +/* Emits the `vfcmp.cult.s vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfcmp_cult_s(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VFCMP_CULT_S, vd, vj, vk)); +} + +/* Emits the `vfcmp.sult.s vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfcmp_sult_s(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VFCMP_SULT_S, vd, vj, vk)); +} + +/* Emits the `vfcmp.cueq.s vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfcmp_cueq_s(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VFCMP_CUEQ_S, vd, vj, vk)); +} + +/* Emits the `vfcmp.sueq.s vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfcmp_sueq_s(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VFCMP_SUEQ_S, vd, vj, vk)); +} + +/* Emits the `vfcmp.cule.s vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfcmp_cule_s(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VFCMP_CULE_S, vd, vj, vk)); +} + +/* Emits the `vfcmp.sule.s vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfcmp_sule_s(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VFCMP_SULE_S, vd, vj, vk)); +} + +/* Emits the `vfcmp.cne.s vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfcmp_cne_s(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VFCMP_CNE_S, vd, vj, vk)); +} + +/* Emits the `vfcmp.sne.s vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfcmp_sne_s(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VFCMP_SNE_S, vd, vj, vk)); +} + +/* Emits the `vfcmp.cor.s vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfcmp_cor_s(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VFCMP_COR_S, vd, vj, vk)); +} + +/* Emits the `vfcmp.sor.s vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfcmp_sor_s(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VFCMP_SOR_S, vd, vj, vk)); +} + +/* Emits the `vfcmp.cune.s vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfcmp_cune_s(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VFCMP_CUNE_S, vd, vj, vk)); +} + +/* Emits the `vfcmp.sune.s vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfcmp_sune_s(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VFCMP_SUNE_S, vd, vj, vk)); +} + +/* Emits the `vfcmp.caf.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfcmp_caf_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VFCMP_CAF_D, vd, vj, vk)); +} + +/* Emits the `vfcmp.saf.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfcmp_saf_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VFCMP_SAF_D, vd, vj, vk)); +} + +/* Emits the `vfcmp.clt.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfcmp_clt_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VFCMP_CLT_D, vd, vj, vk)); +} + +/* Emits the `vfcmp.slt.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfcmp_slt_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VFCMP_SLT_D, vd, vj, vk)); +} + +/* Emits the `vfcmp.ceq.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfcmp_ceq_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VFCMP_CEQ_D, vd, vj, vk)); +} + +/* Emits the `vfcmp.seq.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfcmp_seq_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VFCMP_SEQ_D, vd, vj, vk)); +} + +/* Emits the `vfcmp.cle.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfcmp_cle_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VFCMP_CLE_D, vd, vj, vk)); +} + +/* Emits the `vfcmp.sle.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfcmp_sle_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VFCMP_SLE_D, vd, vj, vk)); +} + +/* Emits the `vfcmp.cun.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfcmp_cun_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VFCMP_CUN_D, vd, vj, vk)); +} + +/* Emits the `vfcmp.sun.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfcmp_sun_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VFCMP_SUN_D, vd, vj, vk)); +} + +/* Emits the `vfcmp.cult.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfcmp_cult_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VFCMP_CULT_D, vd, vj, vk)); +} + +/* Emits the `vfcmp.sult.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfcmp_sult_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VFCMP_SULT_D, vd, vj, vk)); +} + +/* Emits the `vfcmp.cueq.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfcmp_cueq_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VFCMP_CUEQ_D, vd, vj, vk)); +} + +/* Emits the `vfcmp.sueq.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfcmp_sueq_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VFCMP_SUEQ_D, vd, vj, vk)); +} + +/* Emits the `vfcmp.cule.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfcmp_cule_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VFCMP_CULE_D, vd, vj, vk)); +} + +/* Emits the `vfcmp.sule.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfcmp_sule_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VFCMP_SULE_D, vd, vj, vk)); +} + +/* Emits the `vfcmp.cne.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfcmp_cne_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VFCMP_CNE_D, vd, vj, vk)); +} + +/* Emits the `vfcmp.sne.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfcmp_sne_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VFCMP_SNE_D, vd, vj, vk)); +} + +/* Emits the `vfcmp.cor.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfcmp_cor_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VFCMP_COR_D, vd, vj, vk)); +} + +/* Emits the `vfcmp.sor.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfcmp_sor_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VFCMP_SOR_D, vd, vj, vk)); +} + +/* Emits the `vfcmp.cune.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfcmp_cune_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VFCMP_CUNE_D, vd, vj, vk)); +} + +/* Emits the `vfcmp.sune.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfcmp_sune_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VFCMP_SUNE_D, vd, vj, vk)); +} + +/* Emits the `vbitsel.v vd, vj, vk, va` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vbitsel_v(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk, TCGReg va) +{ + tcg_out32(s, encode_vdvjvkva_insn(OPC_VBITSEL_V, vd, vj, vk, va)); +} + +/* Emits the `vshuf.b vd, vj, vk, va` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vshuf_b(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk, TCGReg va) +{ + tcg_out32(s, encode_vdvjvkva_insn(OPC_VSHUF_B, vd, vj, vk, va)); +} + /* Emits the `addu16i.d d, j, sk16` instruction. */ static void __attribute__((unused)) tcg_out_opc_addu16i_d(TCGContext *s, TCGReg d, TCGReg j, int32_t sk16) @@ -XXX,XX +XXX,XX @@ tcg_out_opc_ld_wu(TCGContext *s, TCGReg d, TCGReg j, int32_t sk12) tcg_out32(s, encode_djsk12_insn(OPC_LD_WU, d, j, sk12)); } +/* Emits the `vld vd, j, sk12` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vld(TCGContext *s, TCGReg vd, TCGReg j, int32_t sk12) +{ + tcg_out32(s, encode_vdjsk12_insn(OPC_VLD, vd, j, sk12)); +} + +/* Emits the `vst vd, j, sk12` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vst(TCGContext *s, TCGReg vd, TCGReg j, int32_t sk12) +{ + tcg_out32(s, encode_vdjsk12_insn(OPC_VST, vd, j, sk12)); +} + +/* Emits the `vldrepl.d vd, j, sk9` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vldrepl_d(TCGContext *s, TCGReg vd, TCGReg j, int32_t sk9) +{ + tcg_out32(s, encode_vdjsk9_insn(OPC_VLDREPL_D, vd, j, sk9)); +} + +/* Emits the `vldrepl.w vd, j, sk10` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vldrepl_w(TCGContext *s, TCGReg vd, TCGReg j, int32_t sk10) +{ + tcg_out32(s, encode_vdjsk10_insn(OPC_VLDREPL_W, vd, j, sk10)); +} + +/* Emits the `vldrepl.h vd, j, sk11` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vldrepl_h(TCGContext *s, TCGReg vd, TCGReg j, int32_t sk11) +{ + tcg_out32(s, encode_vdjsk11_insn(OPC_VLDREPL_H, vd, j, sk11)); +} + +/* Emits the `vldrepl.b vd, j, sk12` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vldrepl_b(TCGContext *s, TCGReg vd, TCGReg j, int32_t sk12) +{ + tcg_out32(s, encode_vdjsk12_insn(OPC_VLDREPL_B, vd, j, sk12)); +} + +/* Emits the `vstelm.d vd, j, sk8, un1` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vstelm_d(TCGContext *s, TCGReg vd, TCGReg j, int32_t sk8, + uint32_t un1) +{ + tcg_out32(s, encode_vdjsk8un1_insn(OPC_VSTELM_D, vd, j, sk8, un1)); +} + +/* Emits the `vstelm.w vd, j, sk8, un2` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vstelm_w(TCGContext *s, TCGReg vd, TCGReg j, int32_t sk8, + uint32_t un2) +{ + tcg_out32(s, encode_vdjsk8un2_insn(OPC_VSTELM_W, vd, j, sk8, un2)); +} + +/* Emits the `vstelm.h vd, j, sk8, un3` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vstelm_h(TCGContext *s, TCGReg vd, TCGReg j, int32_t sk8, + uint32_t un3) +{ + tcg_out32(s, encode_vdjsk8un3_insn(OPC_VSTELM_H, vd, j, sk8, un3)); +} + +/* Emits the `vstelm.b vd, j, sk8, un4` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vstelm_b(TCGContext *s, TCGReg vd, TCGReg j, int32_t sk8, + uint32_t un4) +{ + tcg_out32(s, encode_vdjsk8un4_insn(OPC_VSTELM_B, vd, j, sk8, un4)); +} + /* Emits the `ldx.b d, j, k` instruction. */ static void __attribute__((unused)) tcg_out_opc_ldx_b(TCGContext *s, TCGReg d, TCGReg j, TCGReg k) @@ -XXX,XX +XXX,XX @@ tcg_out_opc_ldx_wu(TCGContext *s, TCGReg d, TCGReg j, TCGReg k) tcg_out32(s, encode_djk_insn(OPC_LDX_WU, d, j, k)); } +/* Emits the `vldx vd, j, k` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vldx(TCGContext *s, TCGReg vd, TCGReg j, TCGReg k) +{ + tcg_out32(s, encode_vdjk_insn(OPC_VLDX, vd, j, k)); +} + +/* Emits the `vstx vd, j, k` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vstx(TCGContext *s, TCGReg vd, TCGReg j, TCGReg k) +{ + tcg_out32(s, encode_vdjk_insn(OPC_VSTX, vd, j, k)); +} + /* Emits the `dbar ud15` instruction. */ static void __attribute__((unused)) tcg_out_opc_dbar(TCGContext *s, uint32_t ud15) @@ -XXX,XX +XXX,XX @@ tcg_out_opc_bleu(TCGContext *s, TCGReg d, TCGReg j, int32_t sk16) tcg_out32(s, encode_djsk16_insn(OPC_BLEU, d, j, sk16)); } +/* Emits the `vseq.b vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vseq_b(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSEQ_B, vd, vj, vk)); +} + +/* Emits the `vseq.h vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vseq_h(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSEQ_H, vd, vj, vk)); +} + +/* Emits the `vseq.w vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vseq_w(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSEQ_W, vd, vj, vk)); +} + +/* Emits the `vseq.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vseq_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSEQ_D, vd, vj, vk)); +} + +/* Emits the `vsle.b vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsle_b(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSLE_B, vd, vj, vk)); +} + +/* Emits the `vsle.h vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsle_h(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSLE_H, vd, vj, vk)); +} + +/* Emits the `vsle.w vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsle_w(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSLE_W, vd, vj, vk)); +} + +/* Emits the `vsle.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsle_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSLE_D, vd, vj, vk)); +} + +/* Emits the `vsle.bu vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsle_bu(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSLE_BU, vd, vj, vk)); +} + +/* Emits the `vsle.hu vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsle_hu(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSLE_HU, vd, vj, vk)); +} + +/* Emits the `vsle.wu vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsle_wu(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSLE_WU, vd, vj, vk)); +} + +/* Emits the `vsle.du vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsle_du(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSLE_DU, vd, vj, vk)); +} + +/* Emits the `vslt.b vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vslt_b(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSLT_B, vd, vj, vk)); +} + +/* Emits the `vslt.h vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vslt_h(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSLT_H, vd, vj, vk)); +} + +/* Emits the `vslt.w vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vslt_w(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSLT_W, vd, vj, vk)); +} + +/* Emits the `vslt.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vslt_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSLT_D, vd, vj, vk)); +} + +/* Emits the `vslt.bu vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vslt_bu(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSLT_BU, vd, vj, vk)); +} + +/* Emits the `vslt.hu vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vslt_hu(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSLT_HU, vd, vj, vk)); +} + +/* Emits the `vslt.wu vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vslt_wu(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSLT_WU, vd, vj, vk)); +} + +/* Emits the `vslt.du vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vslt_du(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSLT_DU, vd, vj, vk)); +} + +/* Emits the `vadd.b vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vadd_b(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VADD_B, vd, vj, vk)); +} + +/* Emits the `vadd.h vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vadd_h(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VADD_H, vd, vj, vk)); +} + +/* Emits the `vadd.w vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vadd_w(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VADD_W, vd, vj, vk)); +} + +/* Emits the `vadd.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vadd_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VADD_D, vd, vj, vk)); +} + +/* Emits the `vsub.b vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsub_b(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSUB_B, vd, vj, vk)); +} + +/* Emits the `vsub.h vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsub_h(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSUB_H, vd, vj, vk)); +} + +/* Emits the `vsub.w vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsub_w(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSUB_W, vd, vj, vk)); +} + +/* Emits the `vsub.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsub_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSUB_D, vd, vj, vk)); +} + +/* Emits the `vaddwev.h.b vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vaddwev_h_b(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VADDWEV_H_B, vd, vj, vk)); +} + +/* Emits the `vaddwev.w.h vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vaddwev_w_h(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VADDWEV_W_H, vd, vj, vk)); +} + +/* Emits the `vaddwev.d.w vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vaddwev_d_w(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VADDWEV_D_W, vd, vj, vk)); +} + +/* Emits the `vaddwev.q.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vaddwev_q_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VADDWEV_Q_D, vd, vj, vk)); +} + +/* Emits the `vsubwev.h.b vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsubwev_h_b(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSUBWEV_H_B, vd, vj, vk)); +} + +/* Emits the `vsubwev.w.h vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsubwev_w_h(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSUBWEV_W_H, vd, vj, vk)); +} + +/* Emits the `vsubwev.d.w vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsubwev_d_w(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSUBWEV_D_W, vd, vj, vk)); +} + +/* Emits the `vsubwev.q.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsubwev_q_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSUBWEV_Q_D, vd, vj, vk)); +} + +/* Emits the `vaddwod.h.b vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vaddwod_h_b(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VADDWOD_H_B, vd, vj, vk)); +} + +/* Emits the `vaddwod.w.h vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vaddwod_w_h(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VADDWOD_W_H, vd, vj, vk)); +} + +/* Emits the `vaddwod.d.w vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vaddwod_d_w(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VADDWOD_D_W, vd, vj, vk)); +} + +/* Emits the `vaddwod.q.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vaddwod_q_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VADDWOD_Q_D, vd, vj, vk)); +} + +/* Emits the `vsubwod.h.b vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsubwod_h_b(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSUBWOD_H_B, vd, vj, vk)); +} + +/* Emits the `vsubwod.w.h vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsubwod_w_h(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSUBWOD_W_H, vd, vj, vk)); +} + +/* Emits the `vsubwod.d.w vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsubwod_d_w(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSUBWOD_D_W, vd, vj, vk)); +} + +/* Emits the `vsubwod.q.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsubwod_q_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSUBWOD_Q_D, vd, vj, vk)); +} + +/* Emits the `vaddwev.h.bu vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vaddwev_h_bu(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VADDWEV_H_BU, vd, vj, vk)); +} + +/* Emits the `vaddwev.w.hu vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vaddwev_w_hu(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VADDWEV_W_HU, vd, vj, vk)); +} + +/* Emits the `vaddwev.d.wu vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vaddwev_d_wu(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VADDWEV_D_WU, vd, vj, vk)); +} + +/* Emits the `vaddwev.q.du vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vaddwev_q_du(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VADDWEV_Q_DU, vd, vj, vk)); +} + +/* Emits the `vsubwev.h.bu vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsubwev_h_bu(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSUBWEV_H_BU, vd, vj, vk)); +} + +/* Emits the `vsubwev.w.hu vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsubwev_w_hu(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSUBWEV_W_HU, vd, vj, vk)); +} + +/* Emits the `vsubwev.d.wu vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsubwev_d_wu(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSUBWEV_D_WU, vd, vj, vk)); +} + +/* Emits the `vsubwev.q.du vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsubwev_q_du(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSUBWEV_Q_DU, vd, vj, vk)); +} + +/* Emits the `vaddwod.h.bu vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vaddwod_h_bu(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VADDWOD_H_BU, vd, vj, vk)); +} + +/* Emits the `vaddwod.w.hu vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vaddwod_w_hu(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VADDWOD_W_HU, vd, vj, vk)); +} + +/* Emits the `vaddwod.d.wu vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vaddwod_d_wu(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VADDWOD_D_WU, vd, vj, vk)); +} + +/* Emits the `vaddwod.q.du vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vaddwod_q_du(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VADDWOD_Q_DU, vd, vj, vk)); +} + +/* Emits the `vsubwod.h.bu vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsubwod_h_bu(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSUBWOD_H_BU, vd, vj, vk)); +} + +/* Emits the `vsubwod.w.hu vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsubwod_w_hu(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSUBWOD_W_HU, vd, vj, vk)); +} + +/* Emits the `vsubwod.d.wu vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsubwod_d_wu(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSUBWOD_D_WU, vd, vj, vk)); +} + +/* Emits the `vsubwod.q.du vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsubwod_q_du(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSUBWOD_Q_DU, vd, vj, vk)); +} + +/* Emits the `vaddwev.h.bu.b vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vaddwev_h_bu_b(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VADDWEV_H_BU_B, vd, vj, vk)); +} + +/* Emits the `vaddwev.w.hu.h vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vaddwev_w_hu_h(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VADDWEV_W_HU_H, vd, vj, vk)); +} + +/* Emits the `vaddwev.d.wu.w vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vaddwev_d_wu_w(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VADDWEV_D_WU_W, vd, vj, vk)); +} + +/* Emits the `vaddwev.q.du.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vaddwev_q_du_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VADDWEV_Q_DU_D, vd, vj, vk)); +} + +/* Emits the `vaddwod.h.bu.b vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vaddwod_h_bu_b(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VADDWOD_H_BU_B, vd, vj, vk)); +} + +/* Emits the `vaddwod.w.hu.h vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vaddwod_w_hu_h(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VADDWOD_W_HU_H, vd, vj, vk)); +} + +/* Emits the `vaddwod.d.wu.w vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vaddwod_d_wu_w(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VADDWOD_D_WU_W, vd, vj, vk)); +} + +/* Emits the `vaddwod.q.du.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vaddwod_q_du_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VADDWOD_Q_DU_D, vd, vj, vk)); +} + +/* Emits the `vsadd.b vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsadd_b(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSADD_B, vd, vj, vk)); +} + +/* Emits the `vsadd.h vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsadd_h(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSADD_H, vd, vj, vk)); +} + +/* Emits the `vsadd.w vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsadd_w(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSADD_W, vd, vj, vk)); +} + +/* Emits the `vsadd.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsadd_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSADD_D, vd, vj, vk)); +} + +/* Emits the `vssub.b vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vssub_b(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSSUB_B, vd, vj, vk)); +} + +/* Emits the `vssub.h vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vssub_h(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSSUB_H, vd, vj, vk)); +} + +/* Emits the `vssub.w vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vssub_w(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSSUB_W, vd, vj, vk)); +} + +/* Emits the `vssub.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vssub_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSSUB_D, vd, vj, vk)); +} + +/* Emits the `vsadd.bu vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsadd_bu(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSADD_BU, vd, vj, vk)); +} + +/* Emits the `vsadd.hu vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsadd_hu(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSADD_HU, vd, vj, vk)); +} + +/* Emits the `vsadd.wu vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsadd_wu(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSADD_WU, vd, vj, vk)); +} + +/* Emits the `vsadd.du vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsadd_du(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSADD_DU, vd, vj, vk)); +} + +/* Emits the `vssub.bu vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vssub_bu(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSSUB_BU, vd, vj, vk)); +} + +/* Emits the `vssub.hu vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vssub_hu(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSSUB_HU, vd, vj, vk)); +} + +/* Emits the `vssub.wu vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vssub_wu(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSSUB_WU, vd, vj, vk)); +} + +/* Emits the `vssub.du vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vssub_du(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSSUB_DU, vd, vj, vk)); +} + +/* Emits the `vhaddw.h.b vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vhaddw_h_b(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VHADDW_H_B, vd, vj, vk)); +} + +/* Emits the `vhaddw.w.h vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vhaddw_w_h(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VHADDW_W_H, vd, vj, vk)); +} + +/* Emits the `vhaddw.d.w vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vhaddw_d_w(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VHADDW_D_W, vd, vj, vk)); +} + +/* Emits the `vhaddw.q.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vhaddw_q_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VHADDW_Q_D, vd, vj, vk)); +} + +/* Emits the `vhsubw.h.b vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vhsubw_h_b(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VHSUBW_H_B, vd, vj, vk)); +} + +/* Emits the `vhsubw.w.h vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vhsubw_w_h(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VHSUBW_W_H, vd, vj, vk)); +} + +/* Emits the `vhsubw.d.w vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vhsubw_d_w(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VHSUBW_D_W, vd, vj, vk)); +} + +/* Emits the `vhsubw.q.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vhsubw_q_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VHSUBW_Q_D, vd, vj, vk)); +} + +/* Emits the `vhaddw.hu.bu vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vhaddw_hu_bu(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VHADDW_HU_BU, vd, vj, vk)); +} + +/* Emits the `vhaddw.wu.hu vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vhaddw_wu_hu(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VHADDW_WU_HU, vd, vj, vk)); +} + +/* Emits the `vhaddw.du.wu vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vhaddw_du_wu(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VHADDW_DU_WU, vd, vj, vk)); +} + +/* Emits the `vhaddw.qu.du vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vhaddw_qu_du(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VHADDW_QU_DU, vd, vj, vk)); +} + +/* Emits the `vhsubw.hu.bu vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vhsubw_hu_bu(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VHSUBW_HU_BU, vd, vj, vk)); +} + +/* Emits the `vhsubw.wu.hu vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vhsubw_wu_hu(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VHSUBW_WU_HU, vd, vj, vk)); +} + +/* Emits the `vhsubw.du.wu vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vhsubw_du_wu(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VHSUBW_DU_WU, vd, vj, vk)); +} + +/* Emits the `vhsubw.qu.du vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vhsubw_qu_du(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VHSUBW_QU_DU, vd, vj, vk)); +} + +/* Emits the `vadda.b vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vadda_b(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VADDA_B, vd, vj, vk)); +} + +/* Emits the `vadda.h vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vadda_h(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VADDA_H, vd, vj, vk)); +} + +/* Emits the `vadda.w vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vadda_w(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VADDA_W, vd, vj, vk)); +} + +/* Emits the `vadda.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vadda_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VADDA_D, vd, vj, vk)); +} + +/* Emits the `vabsd.b vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vabsd_b(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VABSD_B, vd, vj, vk)); +} + +/* Emits the `vabsd.h vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vabsd_h(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VABSD_H, vd, vj, vk)); +} + +/* Emits the `vabsd.w vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vabsd_w(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VABSD_W, vd, vj, vk)); +} + +/* Emits the `vabsd.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vabsd_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VABSD_D, vd, vj, vk)); +} + +/* Emits the `vabsd.bu vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vabsd_bu(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VABSD_BU, vd, vj, vk)); +} + +/* Emits the `vabsd.hu vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vabsd_hu(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VABSD_HU, vd, vj, vk)); +} + +/* Emits the `vabsd.wu vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vabsd_wu(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VABSD_WU, vd, vj, vk)); +} + +/* Emits the `vabsd.du vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vabsd_du(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VABSD_DU, vd, vj, vk)); +} + +/* Emits the `vavg.b vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vavg_b(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VAVG_B, vd, vj, vk)); +} + +/* Emits the `vavg.h vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vavg_h(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VAVG_H, vd, vj, vk)); +} + +/* Emits the `vavg.w vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vavg_w(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VAVG_W, vd, vj, vk)); +} + +/* Emits the `vavg.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vavg_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VAVG_D, vd, vj, vk)); +} + +/* Emits the `vavg.bu vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vavg_bu(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VAVG_BU, vd, vj, vk)); +} + +/* Emits the `vavg.hu vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vavg_hu(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VAVG_HU, vd, vj, vk)); +} + +/* Emits the `vavg.wu vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vavg_wu(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VAVG_WU, vd, vj, vk)); +} + +/* Emits the `vavg.du vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vavg_du(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VAVG_DU, vd, vj, vk)); +} + +/* Emits the `vavgr.b vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vavgr_b(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VAVGR_B, vd, vj, vk)); +} + +/* Emits the `vavgr.h vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vavgr_h(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VAVGR_H, vd, vj, vk)); +} + +/* Emits the `vavgr.w vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vavgr_w(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VAVGR_W, vd, vj, vk)); +} + +/* Emits the `vavgr.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vavgr_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VAVGR_D, vd, vj, vk)); +} + +/* Emits the `vavgr.bu vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vavgr_bu(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VAVGR_BU, vd, vj, vk)); +} + +/* Emits the `vavgr.hu vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vavgr_hu(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VAVGR_HU, vd, vj, vk)); +} + +/* Emits the `vavgr.wu vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vavgr_wu(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VAVGR_WU, vd, vj, vk)); +} + +/* Emits the `vavgr.du vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vavgr_du(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VAVGR_DU, vd, vj, vk)); +} + +/* Emits the `vmax.b vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmax_b(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMAX_B, vd, vj, vk)); +} + +/* Emits the `vmax.h vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmax_h(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMAX_H, vd, vj, vk)); +} + +/* Emits the `vmax.w vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmax_w(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMAX_W, vd, vj, vk)); +} + +/* Emits the `vmax.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmax_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMAX_D, vd, vj, vk)); +} + +/* Emits the `vmin.b vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmin_b(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMIN_B, vd, vj, vk)); +} + +/* Emits the `vmin.h vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmin_h(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMIN_H, vd, vj, vk)); +} + +/* Emits the `vmin.w vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmin_w(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMIN_W, vd, vj, vk)); +} + +/* Emits the `vmin.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmin_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMIN_D, vd, vj, vk)); +} + +/* Emits the `vmax.bu vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmax_bu(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMAX_BU, vd, vj, vk)); +} + +/* Emits the `vmax.hu vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmax_hu(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMAX_HU, vd, vj, vk)); +} + +/* Emits the `vmax.wu vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmax_wu(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMAX_WU, vd, vj, vk)); +} + +/* Emits the `vmax.du vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmax_du(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMAX_DU, vd, vj, vk)); +} + +/* Emits the `vmin.bu vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmin_bu(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMIN_BU, vd, vj, vk)); +} + +/* Emits the `vmin.hu vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmin_hu(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMIN_HU, vd, vj, vk)); +} + +/* Emits the `vmin.wu vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmin_wu(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMIN_WU, vd, vj, vk)); +} + +/* Emits the `vmin.du vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmin_du(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMIN_DU, vd, vj, vk)); +} + +/* Emits the `vmul.b vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmul_b(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMUL_B, vd, vj, vk)); +} + +/* Emits the `vmul.h vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmul_h(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMUL_H, vd, vj, vk)); +} + +/* Emits the `vmul.w vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmul_w(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMUL_W, vd, vj, vk)); +} + +/* Emits the `vmul.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmul_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMUL_D, vd, vj, vk)); +} + +/* Emits the `vmuh.b vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmuh_b(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMUH_B, vd, vj, vk)); +} + +/* Emits the `vmuh.h vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmuh_h(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMUH_H, vd, vj, vk)); +} + +/* Emits the `vmuh.w vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmuh_w(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMUH_W, vd, vj, vk)); +} + +/* Emits the `vmuh.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmuh_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMUH_D, vd, vj, vk)); +} + +/* Emits the `vmuh.bu vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmuh_bu(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMUH_BU, vd, vj, vk)); +} + +/* Emits the `vmuh.hu vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmuh_hu(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMUH_HU, vd, vj, vk)); +} + +/* Emits the `vmuh.wu vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmuh_wu(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMUH_WU, vd, vj, vk)); +} + +/* Emits the `vmuh.du vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmuh_du(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMUH_DU, vd, vj, vk)); +} + +/* Emits the `vmulwev.h.b vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmulwev_h_b(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMULWEV_H_B, vd, vj, vk)); +} + +/* Emits the `vmulwev.w.h vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmulwev_w_h(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMULWEV_W_H, vd, vj, vk)); +} + +/* Emits the `vmulwev.d.w vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmulwev_d_w(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMULWEV_D_W, vd, vj, vk)); +} + +/* Emits the `vmulwev.q.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmulwev_q_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMULWEV_Q_D, vd, vj, vk)); +} + +/* Emits the `vmulwod.h.b vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmulwod_h_b(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMULWOD_H_B, vd, vj, vk)); +} + +/* Emits the `vmulwod.w.h vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmulwod_w_h(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMULWOD_W_H, vd, vj, vk)); +} + +/* Emits the `vmulwod.d.w vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmulwod_d_w(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMULWOD_D_W, vd, vj, vk)); +} + +/* Emits the `vmulwod.q.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmulwod_q_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMULWOD_Q_D, vd, vj, vk)); +} + +/* Emits the `vmulwev.h.bu vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmulwev_h_bu(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMULWEV_H_BU, vd, vj, vk)); +} + +/* Emits the `vmulwev.w.hu vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmulwev_w_hu(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMULWEV_W_HU, vd, vj, vk)); +} + +/* Emits the `vmulwev.d.wu vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmulwev_d_wu(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMULWEV_D_WU, vd, vj, vk)); +} + +/* Emits the `vmulwev.q.du vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmulwev_q_du(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMULWEV_Q_DU, vd, vj, vk)); +} + +/* Emits the `vmulwod.h.bu vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmulwod_h_bu(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMULWOD_H_BU, vd, vj, vk)); +} + +/* Emits the `vmulwod.w.hu vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmulwod_w_hu(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMULWOD_W_HU, vd, vj, vk)); +} + +/* Emits the `vmulwod.d.wu vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmulwod_d_wu(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMULWOD_D_WU, vd, vj, vk)); +} + +/* Emits the `vmulwod.q.du vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmulwod_q_du(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMULWOD_Q_DU, vd, vj, vk)); +} + +/* Emits the `vmulwev.h.bu.b vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmulwev_h_bu_b(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMULWEV_H_BU_B, vd, vj, vk)); +} + +/* Emits the `vmulwev.w.hu.h vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmulwev_w_hu_h(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMULWEV_W_HU_H, vd, vj, vk)); +} + +/* Emits the `vmulwev.d.wu.w vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmulwev_d_wu_w(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMULWEV_D_WU_W, vd, vj, vk)); +} + +/* Emits the `vmulwev.q.du.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmulwev_q_du_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMULWEV_Q_DU_D, vd, vj, vk)); +} + +/* Emits the `vmulwod.h.bu.b vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmulwod_h_bu_b(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMULWOD_H_BU_B, vd, vj, vk)); +} + +/* Emits the `vmulwod.w.hu.h vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmulwod_w_hu_h(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMULWOD_W_HU_H, vd, vj, vk)); +} + +/* Emits the `vmulwod.d.wu.w vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmulwod_d_wu_w(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMULWOD_D_WU_W, vd, vj, vk)); +} + +/* Emits the `vmulwod.q.du.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmulwod_q_du_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMULWOD_Q_DU_D, vd, vj, vk)); +} + +/* Emits the `vmadd.b vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmadd_b(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMADD_B, vd, vj, vk)); +} + +/* Emits the `vmadd.h vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmadd_h(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMADD_H, vd, vj, vk)); +} + +/* Emits the `vmadd.w vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmadd_w(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMADD_W, vd, vj, vk)); +} + +/* Emits the `vmadd.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmadd_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMADD_D, vd, vj, vk)); +} + +/* Emits the `vmsub.b vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmsub_b(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMSUB_B, vd, vj, vk)); +} + +/* Emits the `vmsub.h vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmsub_h(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMSUB_H, vd, vj, vk)); +} + +/* Emits the `vmsub.w vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmsub_w(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMSUB_W, vd, vj, vk)); +} + +/* Emits the `vmsub.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmsub_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMSUB_D, vd, vj, vk)); +} + +/* Emits the `vmaddwev.h.b vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmaddwev_h_b(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMADDWEV_H_B, vd, vj, vk)); +} + +/* Emits the `vmaddwev.w.h vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmaddwev_w_h(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMADDWEV_W_H, vd, vj, vk)); +} + +/* Emits the `vmaddwev.d.w vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmaddwev_d_w(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMADDWEV_D_W, vd, vj, vk)); +} + +/* Emits the `vmaddwev.q.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmaddwev_q_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMADDWEV_Q_D, vd, vj, vk)); +} + +/* Emits the `vmaddwod.h.b vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmaddwod_h_b(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMADDWOD_H_B, vd, vj, vk)); +} + +/* Emits the `vmaddwod.w.h vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmaddwod_w_h(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMADDWOD_W_H, vd, vj, vk)); +} + +/* Emits the `vmaddwod.d.w vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmaddwod_d_w(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMADDWOD_D_W, vd, vj, vk)); +} + +/* Emits the `vmaddwod.q.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmaddwod_q_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMADDWOD_Q_D, vd, vj, vk)); +} + +/* Emits the `vmaddwev.h.bu vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmaddwev_h_bu(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMADDWEV_H_BU, vd, vj, vk)); +} + +/* Emits the `vmaddwev.w.hu vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmaddwev_w_hu(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMADDWEV_W_HU, vd, vj, vk)); +} + +/* Emits the `vmaddwev.d.wu vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmaddwev_d_wu(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMADDWEV_D_WU, vd, vj, vk)); +} + +/* Emits the `vmaddwev.q.du vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmaddwev_q_du(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMADDWEV_Q_DU, vd, vj, vk)); +} + +/* Emits the `vmaddwod.h.bu vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmaddwod_h_bu(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMADDWOD_H_BU, vd, vj, vk)); +} + +/* Emits the `vmaddwod.w.hu vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmaddwod_w_hu(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMADDWOD_W_HU, vd, vj, vk)); +} + +/* Emits the `vmaddwod.d.wu vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmaddwod_d_wu(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMADDWOD_D_WU, vd, vj, vk)); +} + +/* Emits the `vmaddwod.q.du vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmaddwod_q_du(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMADDWOD_Q_DU, vd, vj, vk)); +} + +/* Emits the `vmaddwev.h.bu.b vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmaddwev_h_bu_b(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMADDWEV_H_BU_B, vd, vj, vk)); +} + +/* Emits the `vmaddwev.w.hu.h vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmaddwev_w_hu_h(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMADDWEV_W_HU_H, vd, vj, vk)); +} + +/* Emits the `vmaddwev.d.wu.w vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmaddwev_d_wu_w(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMADDWEV_D_WU_W, vd, vj, vk)); +} + +/* Emits the `vmaddwev.q.du.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmaddwev_q_du_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMADDWEV_Q_DU_D, vd, vj, vk)); +} + +/* Emits the `vmaddwod.h.bu.b vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmaddwod_h_bu_b(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMADDWOD_H_BU_B, vd, vj, vk)); +} + +/* Emits the `vmaddwod.w.hu.h vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmaddwod_w_hu_h(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMADDWOD_W_HU_H, vd, vj, vk)); +} + +/* Emits the `vmaddwod.d.wu.w vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmaddwod_d_wu_w(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMADDWOD_D_WU_W, vd, vj, vk)); +} + +/* Emits the `vmaddwod.q.du.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmaddwod_q_du_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMADDWOD_Q_DU_D, vd, vj, vk)); +} + +/* Emits the `vdiv.b vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vdiv_b(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VDIV_B, vd, vj, vk)); +} + +/* Emits the `vdiv.h vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vdiv_h(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VDIV_H, vd, vj, vk)); +} + +/* Emits the `vdiv.w vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vdiv_w(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VDIV_W, vd, vj, vk)); +} + +/* Emits the `vdiv.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vdiv_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VDIV_D, vd, vj, vk)); +} + +/* Emits the `vmod.b vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmod_b(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMOD_B, vd, vj, vk)); +} + +/* Emits the `vmod.h vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmod_h(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMOD_H, vd, vj, vk)); +} + +/* Emits the `vmod.w vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmod_w(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMOD_W, vd, vj, vk)); +} + +/* Emits the `vmod.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmod_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMOD_D, vd, vj, vk)); +} + +/* Emits the `vdiv.bu vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vdiv_bu(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VDIV_BU, vd, vj, vk)); +} + +/* Emits the `vdiv.hu vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vdiv_hu(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VDIV_HU, vd, vj, vk)); +} + +/* Emits the `vdiv.wu vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vdiv_wu(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VDIV_WU, vd, vj, vk)); +} + +/* Emits the `vdiv.du vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vdiv_du(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VDIV_DU, vd, vj, vk)); +} + +/* Emits the `vmod.bu vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmod_bu(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMOD_BU, vd, vj, vk)); +} + +/* Emits the `vmod.hu vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmod_hu(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMOD_HU, vd, vj, vk)); +} + +/* Emits the `vmod.wu vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmod_wu(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMOD_WU, vd, vj, vk)); +} + +/* Emits the `vmod.du vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmod_du(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VMOD_DU, vd, vj, vk)); +} + +/* Emits the `vsll.b vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsll_b(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSLL_B, vd, vj, vk)); +} + +/* Emits the `vsll.h vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsll_h(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSLL_H, vd, vj, vk)); +} + +/* Emits the `vsll.w vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsll_w(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSLL_W, vd, vj, vk)); +} + +/* Emits the `vsll.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsll_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSLL_D, vd, vj, vk)); +} + +/* Emits the `vsrl.b vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsrl_b(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSRL_B, vd, vj, vk)); +} + +/* Emits the `vsrl.h vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsrl_h(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSRL_H, vd, vj, vk)); +} + +/* Emits the `vsrl.w vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsrl_w(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSRL_W, vd, vj, vk)); +} + +/* Emits the `vsrl.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsrl_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSRL_D, vd, vj, vk)); +} + +/* Emits the `vsra.b vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsra_b(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSRA_B, vd, vj, vk)); +} + +/* Emits the `vsra.h vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsra_h(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSRA_H, vd, vj, vk)); +} + +/* Emits the `vsra.w vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsra_w(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSRA_W, vd, vj, vk)); +} + +/* Emits the `vsra.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsra_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSRA_D, vd, vj, vk)); +} + +/* Emits the `vrotr.b vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vrotr_b(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VROTR_B, vd, vj, vk)); +} + +/* Emits the `vrotr.h vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vrotr_h(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VROTR_H, vd, vj, vk)); +} + +/* Emits the `vrotr.w vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vrotr_w(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VROTR_W, vd, vj, vk)); +} + +/* Emits the `vrotr.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vrotr_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VROTR_D, vd, vj, vk)); +} + +/* Emits the `vsrlr.b vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsrlr_b(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSRLR_B, vd, vj, vk)); +} + +/* Emits the `vsrlr.h vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsrlr_h(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSRLR_H, vd, vj, vk)); +} + +/* Emits the `vsrlr.w vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsrlr_w(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSRLR_W, vd, vj, vk)); +} + +/* Emits the `vsrlr.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsrlr_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSRLR_D, vd, vj, vk)); +} + +/* Emits the `vsrar.b vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsrar_b(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSRAR_B, vd, vj, vk)); +} + +/* Emits the `vsrar.h vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsrar_h(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSRAR_H, vd, vj, vk)); +} + +/* Emits the `vsrar.w vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsrar_w(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSRAR_W, vd, vj, vk)); +} + +/* Emits the `vsrar.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsrar_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSRAR_D, vd, vj, vk)); +} + +/* Emits the `vsrln.b.h vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsrln_b_h(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSRLN_B_H, vd, vj, vk)); +} + +/* Emits the `vsrln.h.w vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsrln_h_w(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSRLN_H_W, vd, vj, vk)); +} + +/* Emits the `vsrln.w.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsrln_w_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSRLN_W_D, vd, vj, vk)); +} + +/* Emits the `vsran.b.h vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsran_b_h(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSRAN_B_H, vd, vj, vk)); +} + +/* Emits the `vsran.h.w vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsran_h_w(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSRAN_H_W, vd, vj, vk)); +} + +/* Emits the `vsran.w.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsran_w_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSRAN_W_D, vd, vj, vk)); +} + +/* Emits the `vsrlrn.b.h vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsrlrn_b_h(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSRLRN_B_H, vd, vj, vk)); +} + +/* Emits the `vsrlrn.h.w vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsrlrn_h_w(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSRLRN_H_W, vd, vj, vk)); +} + +/* Emits the `vsrlrn.w.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsrlrn_w_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSRLRN_W_D, vd, vj, vk)); +} + +/* Emits the `vsrarn.b.h vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsrarn_b_h(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSRARN_B_H, vd, vj, vk)); +} + +/* Emits the `vsrarn.h.w vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsrarn_h_w(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSRARN_H_W, vd, vj, vk)); +} + +/* Emits the `vsrarn.w.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsrarn_w_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSRARN_W_D, vd, vj, vk)); +} + +/* Emits the `vssrln.b.h vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vssrln_b_h(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSSRLN_B_H, vd, vj, vk)); +} + +/* Emits the `vssrln.h.w vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vssrln_h_w(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSSRLN_H_W, vd, vj, vk)); +} + +/* Emits the `vssrln.w.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vssrln_w_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSSRLN_W_D, vd, vj, vk)); +} + +/* Emits the `vssran.b.h vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vssran_b_h(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSSRAN_B_H, vd, vj, vk)); +} + +/* Emits the `vssran.h.w vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vssran_h_w(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSSRAN_H_W, vd, vj, vk)); +} + +/* Emits the `vssran.w.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vssran_w_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSSRAN_W_D, vd, vj, vk)); +} + +/* Emits the `vssrlrn.b.h vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vssrlrn_b_h(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSSRLRN_B_H, vd, vj, vk)); +} + +/* Emits the `vssrlrn.h.w vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vssrlrn_h_w(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSSRLRN_H_W, vd, vj, vk)); +} + +/* Emits the `vssrlrn.w.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vssrlrn_w_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSSRLRN_W_D, vd, vj, vk)); +} + +/* Emits the `vssrarn.b.h vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vssrarn_b_h(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSSRARN_B_H, vd, vj, vk)); +} + +/* Emits the `vssrarn.h.w vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vssrarn_h_w(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSSRARN_H_W, vd, vj, vk)); +} + +/* Emits the `vssrarn.w.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vssrarn_w_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSSRARN_W_D, vd, vj, vk)); +} + +/* Emits the `vssrln.bu.h vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vssrln_bu_h(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSSRLN_BU_H, vd, vj, vk)); +} + +/* Emits the `vssrln.hu.w vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vssrln_hu_w(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSSRLN_HU_W, vd, vj, vk)); +} + +/* Emits the `vssrln.wu.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vssrln_wu_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSSRLN_WU_D, vd, vj, vk)); +} + +/* Emits the `vssran.bu.h vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vssran_bu_h(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSSRAN_BU_H, vd, vj, vk)); +} + +/* Emits the `vssran.hu.w vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vssran_hu_w(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSSRAN_HU_W, vd, vj, vk)); +} + +/* Emits the `vssran.wu.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vssran_wu_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSSRAN_WU_D, vd, vj, vk)); +} + +/* Emits the `vssrlrn.bu.h vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vssrlrn_bu_h(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSSRLRN_BU_H, vd, vj, vk)); +} + +/* Emits the `vssrlrn.hu.w vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vssrlrn_hu_w(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSSRLRN_HU_W, vd, vj, vk)); +} + +/* Emits the `vssrlrn.wu.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vssrlrn_wu_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSSRLRN_WU_D, vd, vj, vk)); +} + +/* Emits the `vssrarn.bu.h vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vssrarn_bu_h(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSSRARN_BU_H, vd, vj, vk)); +} + +/* Emits the `vssrarn.hu.w vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vssrarn_hu_w(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSSRARN_HU_W, vd, vj, vk)); +} + +/* Emits the `vssrarn.wu.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vssrarn_wu_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSSRARN_WU_D, vd, vj, vk)); +} + +/* Emits the `vbitclr.b vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vbitclr_b(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VBITCLR_B, vd, vj, vk)); +} + +/* Emits the `vbitclr.h vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vbitclr_h(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VBITCLR_H, vd, vj, vk)); +} + +/* Emits the `vbitclr.w vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vbitclr_w(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VBITCLR_W, vd, vj, vk)); +} + +/* Emits the `vbitclr.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vbitclr_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VBITCLR_D, vd, vj, vk)); +} + +/* Emits the `vbitset.b vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vbitset_b(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VBITSET_B, vd, vj, vk)); +} + +/* Emits the `vbitset.h vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vbitset_h(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VBITSET_H, vd, vj, vk)); +} + +/* Emits the `vbitset.w vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vbitset_w(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VBITSET_W, vd, vj, vk)); +} + +/* Emits the `vbitset.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vbitset_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VBITSET_D, vd, vj, vk)); +} + +/* Emits the `vbitrev.b vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vbitrev_b(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VBITREV_B, vd, vj, vk)); +} + +/* Emits the `vbitrev.h vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vbitrev_h(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VBITREV_H, vd, vj, vk)); +} + +/* Emits the `vbitrev.w vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vbitrev_w(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VBITREV_W, vd, vj, vk)); +} + +/* Emits the `vbitrev.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vbitrev_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VBITREV_D, vd, vj, vk)); +} + +/* Emits the `vpackev.b vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vpackev_b(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VPACKEV_B, vd, vj, vk)); +} + +/* Emits the `vpackev.h vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vpackev_h(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VPACKEV_H, vd, vj, vk)); +} + +/* Emits the `vpackev.w vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vpackev_w(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VPACKEV_W, vd, vj, vk)); +} + +/* Emits the `vpackev.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vpackev_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VPACKEV_D, vd, vj, vk)); +} + +/* Emits the `vpackod.b vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vpackod_b(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VPACKOD_B, vd, vj, vk)); +} + +/* Emits the `vpackod.h vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vpackod_h(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VPACKOD_H, vd, vj, vk)); +} + +/* Emits the `vpackod.w vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vpackod_w(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VPACKOD_W, vd, vj, vk)); +} + +/* Emits the `vpackod.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vpackod_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VPACKOD_D, vd, vj, vk)); +} + +/* Emits the `vilvl.b vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vilvl_b(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VILVL_B, vd, vj, vk)); +} + +/* Emits the `vilvl.h vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vilvl_h(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VILVL_H, vd, vj, vk)); +} + +/* Emits the `vilvl.w vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vilvl_w(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VILVL_W, vd, vj, vk)); +} + +/* Emits the `vilvl.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vilvl_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VILVL_D, vd, vj, vk)); +} + +/* Emits the `vilvh.b vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vilvh_b(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VILVH_B, vd, vj, vk)); +} + +/* Emits the `vilvh.h vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vilvh_h(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VILVH_H, vd, vj, vk)); +} + +/* Emits the `vilvh.w vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vilvh_w(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VILVH_W, vd, vj, vk)); +} + +/* Emits the `vilvh.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vilvh_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VILVH_D, vd, vj, vk)); +} + +/* Emits the `vpickev.b vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vpickev_b(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VPICKEV_B, vd, vj, vk)); +} + +/* Emits the `vpickev.h vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vpickev_h(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VPICKEV_H, vd, vj, vk)); +} + +/* Emits the `vpickev.w vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vpickev_w(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VPICKEV_W, vd, vj, vk)); +} + +/* Emits the `vpickev.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vpickev_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VPICKEV_D, vd, vj, vk)); +} + +/* Emits the `vpickod.b vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vpickod_b(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VPICKOD_B, vd, vj, vk)); +} + +/* Emits the `vpickod.h vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vpickod_h(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VPICKOD_H, vd, vj, vk)); +} + +/* Emits the `vpickod.w vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vpickod_w(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VPICKOD_W, vd, vj, vk)); +} + +/* Emits the `vpickod.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vpickod_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VPICKOD_D, vd, vj, vk)); +} + +/* Emits the `vreplve.b vd, vj, k` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vreplve_b(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg k) +{ + tcg_out32(s, encode_vdvjk_insn(OPC_VREPLVE_B, vd, vj, k)); +} + +/* Emits the `vreplve.h vd, vj, k` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vreplve_h(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg k) +{ + tcg_out32(s, encode_vdvjk_insn(OPC_VREPLVE_H, vd, vj, k)); +} + +/* Emits the `vreplve.w vd, vj, k` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vreplve_w(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg k) +{ + tcg_out32(s, encode_vdvjk_insn(OPC_VREPLVE_W, vd, vj, k)); +} + +/* Emits the `vreplve.d vd, vj, k` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vreplve_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg k) +{ + tcg_out32(s, encode_vdvjk_insn(OPC_VREPLVE_D, vd, vj, k)); +} + +/* Emits the `vand.v vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vand_v(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VAND_V, vd, vj, vk)); +} + +/* Emits the `vor.v vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vor_v(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VOR_V, vd, vj, vk)); +} + +/* Emits the `vxor.v vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vxor_v(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VXOR_V, vd, vj, vk)); +} + +/* Emits the `vnor.v vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vnor_v(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VNOR_V, vd, vj, vk)); +} + +/* Emits the `vandn.v vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vandn_v(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VANDN_V, vd, vj, vk)); +} + +/* Emits the `vorn.v vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vorn_v(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VORN_V, vd, vj, vk)); +} + +/* Emits the `vfrstp.b vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfrstp_b(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VFRSTP_B, vd, vj, vk)); +} + +/* Emits the `vfrstp.h vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfrstp_h(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VFRSTP_H, vd, vj, vk)); +} + +/* Emits the `vadd.q vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vadd_q(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VADD_Q, vd, vj, vk)); +} + +/* Emits the `vsub.q vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsub_q(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSUB_Q, vd, vj, vk)); +} + +/* Emits the `vsigncov.b vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsigncov_b(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSIGNCOV_B, vd, vj, vk)); +} + +/* Emits the `vsigncov.h vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsigncov_h(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSIGNCOV_H, vd, vj, vk)); +} + +/* Emits the `vsigncov.w vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsigncov_w(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSIGNCOV_W, vd, vj, vk)); +} + +/* Emits the `vsigncov.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsigncov_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSIGNCOV_D, vd, vj, vk)); +} + +/* Emits the `vfadd.s vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfadd_s(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VFADD_S, vd, vj, vk)); +} + +/* Emits the `vfadd.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfadd_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VFADD_D, vd, vj, vk)); +} + +/* Emits the `vfsub.s vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfsub_s(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VFSUB_S, vd, vj, vk)); +} + +/* Emits the `vfsub.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfsub_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VFSUB_D, vd, vj, vk)); +} + +/* Emits the `vfmul.s vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfmul_s(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VFMUL_S, vd, vj, vk)); +} + +/* Emits the `vfmul.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfmul_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VFMUL_D, vd, vj, vk)); +} + +/* Emits the `vfdiv.s vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfdiv_s(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VFDIV_S, vd, vj, vk)); +} + +/* Emits the `vfdiv.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfdiv_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VFDIV_D, vd, vj, vk)); +} + +/* Emits the `vfmax.s vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfmax_s(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VFMAX_S, vd, vj, vk)); +} + +/* Emits the `vfmax.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfmax_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VFMAX_D, vd, vj, vk)); +} + +/* Emits the `vfmin.s vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfmin_s(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VFMIN_S, vd, vj, vk)); +} + +/* Emits the `vfmin.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfmin_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VFMIN_D, vd, vj, vk)); +} + +/* Emits the `vfmaxa.s vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfmaxa_s(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VFMAXA_S, vd, vj, vk)); +} + +/* Emits the `vfmaxa.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfmaxa_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VFMAXA_D, vd, vj, vk)); +} + +/* Emits the `vfmina.s vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfmina_s(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VFMINA_S, vd, vj, vk)); +} + +/* Emits the `vfmina.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfmina_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VFMINA_D, vd, vj, vk)); +} + +/* Emits the `vfcvt.h.s vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfcvt_h_s(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VFCVT_H_S, vd, vj, vk)); +} + +/* Emits the `vfcvt.s.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfcvt_s_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VFCVT_S_D, vd, vj, vk)); +} + +/* Emits the `vffint.s.l vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vffint_s_l(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VFFINT_S_L, vd, vj, vk)); +} + +/* Emits the `vftint.w.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vftint_w_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VFTINT_W_D, vd, vj, vk)); +} + +/* Emits the `vftintrm.w.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vftintrm_w_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VFTINTRM_W_D, vd, vj, vk)); +} + +/* Emits the `vftintrp.w.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vftintrp_w_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VFTINTRP_W_D, vd, vj, vk)); +} + +/* Emits the `vftintrz.w.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vftintrz_w_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VFTINTRZ_W_D, vd, vj, vk)); +} + +/* Emits the `vftintrne.w.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vftintrne_w_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VFTINTRNE_W_D, vd, vj, vk)); +} + +/* Emits the `vshuf.h vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vshuf_h(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSHUF_H, vd, vj, vk)); +} + +/* Emits the `vshuf.w vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vshuf_w(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSHUF_W, vd, vj, vk)); +} + +/* Emits the `vshuf.d vd, vj, vk` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vshuf_d(TCGContext *s, TCGReg vd, TCGReg vj, TCGReg vk) +{ + tcg_out32(s, encode_vdvjvk_insn(OPC_VSHUF_D, vd, vj, vk)); +} + +/* Emits the `vseqi.b vd, vj, sk5` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vseqi_b(TCGContext *s, TCGReg vd, TCGReg vj, int32_t sk5) +{ + tcg_out32(s, encode_vdvjsk5_insn(OPC_VSEQI_B, vd, vj, sk5)); +} + +/* Emits the `vseqi.h vd, vj, sk5` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vseqi_h(TCGContext *s, TCGReg vd, TCGReg vj, int32_t sk5) +{ + tcg_out32(s, encode_vdvjsk5_insn(OPC_VSEQI_H, vd, vj, sk5)); +} + +/* Emits the `vseqi.w vd, vj, sk5` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vseqi_w(TCGContext *s, TCGReg vd, TCGReg vj, int32_t sk5) +{ + tcg_out32(s, encode_vdvjsk5_insn(OPC_VSEQI_W, vd, vj, sk5)); +} + +/* Emits the `vseqi.d vd, vj, sk5` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vseqi_d(TCGContext *s, TCGReg vd, TCGReg vj, int32_t sk5) +{ + tcg_out32(s, encode_vdvjsk5_insn(OPC_VSEQI_D, vd, vj, sk5)); +} + +/* Emits the `vslei.b vd, vj, sk5` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vslei_b(TCGContext *s, TCGReg vd, TCGReg vj, int32_t sk5) +{ + tcg_out32(s, encode_vdvjsk5_insn(OPC_VSLEI_B, vd, vj, sk5)); +} + +/* Emits the `vslei.h vd, vj, sk5` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vslei_h(TCGContext *s, TCGReg vd, TCGReg vj, int32_t sk5) +{ + tcg_out32(s, encode_vdvjsk5_insn(OPC_VSLEI_H, vd, vj, sk5)); +} + +/* Emits the `vslei.w vd, vj, sk5` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vslei_w(TCGContext *s, TCGReg vd, TCGReg vj, int32_t sk5) +{ + tcg_out32(s, encode_vdvjsk5_insn(OPC_VSLEI_W, vd, vj, sk5)); +} + +/* Emits the `vslei.d vd, vj, sk5` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vslei_d(TCGContext *s, TCGReg vd, TCGReg vj, int32_t sk5) +{ + tcg_out32(s, encode_vdvjsk5_insn(OPC_VSLEI_D, vd, vj, sk5)); +} + +/* Emits the `vslei.bu vd, vj, uk5` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vslei_bu(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk5) +{ + tcg_out32(s, encode_vdvjuk5_insn(OPC_VSLEI_BU, vd, vj, uk5)); +} + +/* Emits the `vslei.hu vd, vj, uk5` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vslei_hu(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk5) +{ + tcg_out32(s, encode_vdvjuk5_insn(OPC_VSLEI_HU, vd, vj, uk5)); +} + +/* Emits the `vslei.wu vd, vj, uk5` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vslei_wu(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk5) +{ + tcg_out32(s, encode_vdvjuk5_insn(OPC_VSLEI_WU, vd, vj, uk5)); +} + +/* Emits the `vslei.du vd, vj, uk5` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vslei_du(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk5) +{ + tcg_out32(s, encode_vdvjuk5_insn(OPC_VSLEI_DU, vd, vj, uk5)); +} + +/* Emits the `vslti.b vd, vj, sk5` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vslti_b(TCGContext *s, TCGReg vd, TCGReg vj, int32_t sk5) +{ + tcg_out32(s, encode_vdvjsk5_insn(OPC_VSLTI_B, vd, vj, sk5)); +} + +/* Emits the `vslti.h vd, vj, sk5` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vslti_h(TCGContext *s, TCGReg vd, TCGReg vj, int32_t sk5) +{ + tcg_out32(s, encode_vdvjsk5_insn(OPC_VSLTI_H, vd, vj, sk5)); +} + +/* Emits the `vslti.w vd, vj, sk5` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vslti_w(TCGContext *s, TCGReg vd, TCGReg vj, int32_t sk5) +{ + tcg_out32(s, encode_vdvjsk5_insn(OPC_VSLTI_W, vd, vj, sk5)); +} + +/* Emits the `vslti.d vd, vj, sk5` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vslti_d(TCGContext *s, TCGReg vd, TCGReg vj, int32_t sk5) +{ + tcg_out32(s, encode_vdvjsk5_insn(OPC_VSLTI_D, vd, vj, sk5)); +} + +/* Emits the `vslti.bu vd, vj, uk5` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vslti_bu(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk5) +{ + tcg_out32(s, encode_vdvjuk5_insn(OPC_VSLTI_BU, vd, vj, uk5)); +} + +/* Emits the `vslti.hu vd, vj, uk5` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vslti_hu(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk5) +{ + tcg_out32(s, encode_vdvjuk5_insn(OPC_VSLTI_HU, vd, vj, uk5)); +} + +/* Emits the `vslti.wu vd, vj, uk5` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vslti_wu(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk5) +{ + tcg_out32(s, encode_vdvjuk5_insn(OPC_VSLTI_WU, vd, vj, uk5)); +} + +/* Emits the `vslti.du vd, vj, uk5` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vslti_du(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk5) +{ + tcg_out32(s, encode_vdvjuk5_insn(OPC_VSLTI_DU, vd, vj, uk5)); +} + +/* Emits the `vaddi.bu vd, vj, uk5` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vaddi_bu(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk5) +{ + tcg_out32(s, encode_vdvjuk5_insn(OPC_VADDI_BU, vd, vj, uk5)); +} + +/* Emits the `vaddi.hu vd, vj, uk5` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vaddi_hu(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk5) +{ + tcg_out32(s, encode_vdvjuk5_insn(OPC_VADDI_HU, vd, vj, uk5)); +} + +/* Emits the `vaddi.wu vd, vj, uk5` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vaddi_wu(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk5) +{ + tcg_out32(s, encode_vdvjuk5_insn(OPC_VADDI_WU, vd, vj, uk5)); +} + +/* Emits the `vaddi.du vd, vj, uk5` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vaddi_du(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk5) +{ + tcg_out32(s, encode_vdvjuk5_insn(OPC_VADDI_DU, vd, vj, uk5)); +} + +/* Emits the `vsubi.bu vd, vj, uk5` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsubi_bu(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk5) +{ + tcg_out32(s, encode_vdvjuk5_insn(OPC_VSUBI_BU, vd, vj, uk5)); +} + +/* Emits the `vsubi.hu vd, vj, uk5` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsubi_hu(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk5) +{ + tcg_out32(s, encode_vdvjuk5_insn(OPC_VSUBI_HU, vd, vj, uk5)); +} + +/* Emits the `vsubi.wu vd, vj, uk5` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsubi_wu(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk5) +{ + tcg_out32(s, encode_vdvjuk5_insn(OPC_VSUBI_WU, vd, vj, uk5)); +} + +/* Emits the `vsubi.du vd, vj, uk5` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsubi_du(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk5) +{ + tcg_out32(s, encode_vdvjuk5_insn(OPC_VSUBI_DU, vd, vj, uk5)); +} + +/* Emits the `vbsll.v vd, vj, uk5` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vbsll_v(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk5) +{ + tcg_out32(s, encode_vdvjuk5_insn(OPC_VBSLL_V, vd, vj, uk5)); +} + +/* Emits the `vbsrl.v vd, vj, uk5` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vbsrl_v(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk5) +{ + tcg_out32(s, encode_vdvjuk5_insn(OPC_VBSRL_V, vd, vj, uk5)); +} + +/* Emits the `vmaxi.b vd, vj, sk5` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmaxi_b(TCGContext *s, TCGReg vd, TCGReg vj, int32_t sk5) +{ + tcg_out32(s, encode_vdvjsk5_insn(OPC_VMAXI_B, vd, vj, sk5)); +} + +/* Emits the `vmaxi.h vd, vj, sk5` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmaxi_h(TCGContext *s, TCGReg vd, TCGReg vj, int32_t sk5) +{ + tcg_out32(s, encode_vdvjsk5_insn(OPC_VMAXI_H, vd, vj, sk5)); +} + +/* Emits the `vmaxi.w vd, vj, sk5` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmaxi_w(TCGContext *s, TCGReg vd, TCGReg vj, int32_t sk5) +{ + tcg_out32(s, encode_vdvjsk5_insn(OPC_VMAXI_W, vd, vj, sk5)); +} + +/* Emits the `vmaxi.d vd, vj, sk5` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmaxi_d(TCGContext *s, TCGReg vd, TCGReg vj, int32_t sk5) +{ + tcg_out32(s, encode_vdvjsk5_insn(OPC_VMAXI_D, vd, vj, sk5)); +} + +/* Emits the `vmini.b vd, vj, sk5` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmini_b(TCGContext *s, TCGReg vd, TCGReg vj, int32_t sk5) +{ + tcg_out32(s, encode_vdvjsk5_insn(OPC_VMINI_B, vd, vj, sk5)); +} + +/* Emits the `vmini.h vd, vj, sk5` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmini_h(TCGContext *s, TCGReg vd, TCGReg vj, int32_t sk5) +{ + tcg_out32(s, encode_vdvjsk5_insn(OPC_VMINI_H, vd, vj, sk5)); +} + +/* Emits the `vmini.w vd, vj, sk5` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmini_w(TCGContext *s, TCGReg vd, TCGReg vj, int32_t sk5) +{ + tcg_out32(s, encode_vdvjsk5_insn(OPC_VMINI_W, vd, vj, sk5)); +} + +/* Emits the `vmini.d vd, vj, sk5` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmini_d(TCGContext *s, TCGReg vd, TCGReg vj, int32_t sk5) +{ + tcg_out32(s, encode_vdvjsk5_insn(OPC_VMINI_D, vd, vj, sk5)); +} + +/* Emits the `vmaxi.bu vd, vj, uk5` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmaxi_bu(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk5) +{ + tcg_out32(s, encode_vdvjuk5_insn(OPC_VMAXI_BU, vd, vj, uk5)); +} + +/* Emits the `vmaxi.hu vd, vj, uk5` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmaxi_hu(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk5) +{ + tcg_out32(s, encode_vdvjuk5_insn(OPC_VMAXI_HU, vd, vj, uk5)); +} + +/* Emits the `vmaxi.wu vd, vj, uk5` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmaxi_wu(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk5) +{ + tcg_out32(s, encode_vdvjuk5_insn(OPC_VMAXI_WU, vd, vj, uk5)); +} + +/* Emits the `vmaxi.du vd, vj, uk5` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmaxi_du(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk5) +{ + tcg_out32(s, encode_vdvjuk5_insn(OPC_VMAXI_DU, vd, vj, uk5)); +} + +/* Emits the `vmini.bu vd, vj, uk5` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmini_bu(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk5) +{ + tcg_out32(s, encode_vdvjuk5_insn(OPC_VMINI_BU, vd, vj, uk5)); +} + +/* Emits the `vmini.hu vd, vj, uk5` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmini_hu(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk5) +{ + tcg_out32(s, encode_vdvjuk5_insn(OPC_VMINI_HU, vd, vj, uk5)); +} + +/* Emits the `vmini.wu vd, vj, uk5` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmini_wu(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk5) +{ + tcg_out32(s, encode_vdvjuk5_insn(OPC_VMINI_WU, vd, vj, uk5)); +} + +/* Emits the `vmini.du vd, vj, uk5` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmini_du(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk5) +{ + tcg_out32(s, encode_vdvjuk5_insn(OPC_VMINI_DU, vd, vj, uk5)); +} + +/* Emits the `vfrstpi.b vd, vj, uk5` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfrstpi_b(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk5) +{ + tcg_out32(s, encode_vdvjuk5_insn(OPC_VFRSTPI_B, vd, vj, uk5)); +} + +/* Emits the `vfrstpi.h vd, vj, uk5` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfrstpi_h(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk5) +{ + tcg_out32(s, encode_vdvjuk5_insn(OPC_VFRSTPI_H, vd, vj, uk5)); +} + +/* Emits the `vclo.b vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vclo_b(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VCLO_B, vd, vj)); +} + +/* Emits the `vclo.h vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vclo_h(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VCLO_H, vd, vj)); +} + +/* Emits the `vclo.w vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vclo_w(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VCLO_W, vd, vj)); +} + +/* Emits the `vclo.d vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vclo_d(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VCLO_D, vd, vj)); +} + +/* Emits the `vclz.b vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vclz_b(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VCLZ_B, vd, vj)); +} + +/* Emits the `vclz.h vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vclz_h(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VCLZ_H, vd, vj)); +} + +/* Emits the `vclz.w vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vclz_w(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VCLZ_W, vd, vj)); +} + +/* Emits the `vclz.d vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vclz_d(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VCLZ_D, vd, vj)); +} + +/* Emits the `vpcnt.b vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vpcnt_b(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VPCNT_B, vd, vj)); +} + +/* Emits the `vpcnt.h vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vpcnt_h(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VPCNT_H, vd, vj)); +} + +/* Emits the `vpcnt.w vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vpcnt_w(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VPCNT_W, vd, vj)); +} + +/* Emits the `vpcnt.d vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vpcnt_d(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VPCNT_D, vd, vj)); +} + +/* Emits the `vneg.b vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vneg_b(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VNEG_B, vd, vj)); +} + +/* Emits the `vneg.h vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vneg_h(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VNEG_H, vd, vj)); +} + +/* Emits the `vneg.w vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vneg_w(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VNEG_W, vd, vj)); +} + +/* Emits the `vneg.d vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vneg_d(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VNEG_D, vd, vj)); +} + +/* Emits the `vmskltz.b vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmskltz_b(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VMSKLTZ_B, vd, vj)); +} + +/* Emits the `vmskltz.h vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmskltz_h(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VMSKLTZ_H, vd, vj)); +} + +/* Emits the `vmskltz.w vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmskltz_w(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VMSKLTZ_W, vd, vj)); +} + +/* Emits the `vmskltz.d vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmskltz_d(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VMSKLTZ_D, vd, vj)); +} + +/* Emits the `vmskgez.b vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmskgez_b(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VMSKGEZ_B, vd, vj)); +} + +/* Emits the `vmsknz.b vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vmsknz_b(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VMSKNZ_B, vd, vj)); +} + +/* Emits the `vseteqz.v cd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vseteqz_v(TCGContext *s, TCGReg cd, TCGReg vj) +{ + tcg_out32(s, encode_cdvj_insn(OPC_VSETEQZ_V, cd, vj)); +} + +/* Emits the `vsetnez.v cd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsetnez_v(TCGContext *s, TCGReg cd, TCGReg vj) +{ + tcg_out32(s, encode_cdvj_insn(OPC_VSETNEZ_V, cd, vj)); +} + +/* Emits the `vsetanyeqz.b cd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsetanyeqz_b(TCGContext *s, TCGReg cd, TCGReg vj) +{ + tcg_out32(s, encode_cdvj_insn(OPC_VSETANYEQZ_B, cd, vj)); +} + +/* Emits the `vsetanyeqz.h cd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsetanyeqz_h(TCGContext *s, TCGReg cd, TCGReg vj) +{ + tcg_out32(s, encode_cdvj_insn(OPC_VSETANYEQZ_H, cd, vj)); +} + +/* Emits the `vsetanyeqz.w cd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsetanyeqz_w(TCGContext *s, TCGReg cd, TCGReg vj) +{ + tcg_out32(s, encode_cdvj_insn(OPC_VSETANYEQZ_W, cd, vj)); +} + +/* Emits the `vsetanyeqz.d cd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsetanyeqz_d(TCGContext *s, TCGReg cd, TCGReg vj) +{ + tcg_out32(s, encode_cdvj_insn(OPC_VSETANYEQZ_D, cd, vj)); +} + +/* Emits the `vsetallnez.b cd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsetallnez_b(TCGContext *s, TCGReg cd, TCGReg vj) +{ + tcg_out32(s, encode_cdvj_insn(OPC_VSETALLNEZ_B, cd, vj)); +} + +/* Emits the `vsetallnez.h cd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsetallnez_h(TCGContext *s, TCGReg cd, TCGReg vj) +{ + tcg_out32(s, encode_cdvj_insn(OPC_VSETALLNEZ_H, cd, vj)); +} + +/* Emits the `vsetallnez.w cd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsetallnez_w(TCGContext *s, TCGReg cd, TCGReg vj) +{ + tcg_out32(s, encode_cdvj_insn(OPC_VSETALLNEZ_W, cd, vj)); +} + +/* Emits the `vsetallnez.d cd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsetallnez_d(TCGContext *s, TCGReg cd, TCGReg vj) +{ + tcg_out32(s, encode_cdvj_insn(OPC_VSETALLNEZ_D, cd, vj)); +} + +/* Emits the `vflogb.s vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vflogb_s(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VFLOGB_S, vd, vj)); +} + +/* Emits the `vflogb.d vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vflogb_d(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VFLOGB_D, vd, vj)); +} + +/* Emits the `vfclass.s vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfclass_s(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VFCLASS_S, vd, vj)); +} + +/* Emits the `vfclass.d vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfclass_d(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VFCLASS_D, vd, vj)); +} + +/* Emits the `vfsqrt.s vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfsqrt_s(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VFSQRT_S, vd, vj)); +} + +/* Emits the `vfsqrt.d vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfsqrt_d(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VFSQRT_D, vd, vj)); +} + +/* Emits the `vfrecip.s vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfrecip_s(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VFRECIP_S, vd, vj)); +} + +/* Emits the `vfrecip.d vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfrecip_d(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VFRECIP_D, vd, vj)); +} + +/* Emits the `vfrsqrt.s vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfrsqrt_s(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VFRSQRT_S, vd, vj)); +} + +/* Emits the `vfrsqrt.d vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfrsqrt_d(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VFRSQRT_D, vd, vj)); +} + +/* Emits the `vfrint.s vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfrint_s(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VFRINT_S, vd, vj)); +} + +/* Emits the `vfrint.d vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfrint_d(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VFRINT_D, vd, vj)); +} + +/* Emits the `vfrintrm.s vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfrintrm_s(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VFRINTRM_S, vd, vj)); +} + +/* Emits the `vfrintrm.d vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfrintrm_d(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VFRINTRM_D, vd, vj)); +} + +/* Emits the `vfrintrp.s vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfrintrp_s(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VFRINTRP_S, vd, vj)); +} + +/* Emits the `vfrintrp.d vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfrintrp_d(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VFRINTRP_D, vd, vj)); +} + +/* Emits the `vfrintrz.s vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfrintrz_s(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VFRINTRZ_S, vd, vj)); +} + +/* Emits the `vfrintrz.d vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfrintrz_d(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VFRINTRZ_D, vd, vj)); +} + +/* Emits the `vfrintrne.s vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfrintrne_s(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VFRINTRNE_S, vd, vj)); +} + +/* Emits the `vfrintrne.d vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfrintrne_d(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VFRINTRNE_D, vd, vj)); +} + +/* Emits the `vfcvtl.s.h vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfcvtl_s_h(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VFCVTL_S_H, vd, vj)); +} + +/* Emits the `vfcvth.s.h vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfcvth_s_h(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VFCVTH_S_H, vd, vj)); +} + +/* Emits the `vfcvtl.d.s vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfcvtl_d_s(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VFCVTL_D_S, vd, vj)); +} + +/* Emits the `vfcvth.d.s vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vfcvth_d_s(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VFCVTH_D_S, vd, vj)); +} + +/* Emits the `vffint.s.w vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vffint_s_w(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VFFINT_S_W, vd, vj)); +} + +/* Emits the `vffint.s.wu vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vffint_s_wu(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VFFINT_S_WU, vd, vj)); +} + +/* Emits the `vffint.d.l vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vffint_d_l(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VFFINT_D_L, vd, vj)); +} + +/* Emits the `vffint.d.lu vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vffint_d_lu(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VFFINT_D_LU, vd, vj)); +} + +/* Emits the `vffintl.d.w vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vffintl_d_w(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VFFINTL_D_W, vd, vj)); +} + +/* Emits the `vffinth.d.w vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vffinth_d_w(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VFFINTH_D_W, vd, vj)); +} + +/* Emits the `vftint.w.s vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vftint_w_s(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VFTINT_W_S, vd, vj)); +} + +/* Emits the `vftint.l.d vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vftint_l_d(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VFTINT_L_D, vd, vj)); +} + +/* Emits the `vftintrm.w.s vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vftintrm_w_s(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VFTINTRM_W_S, vd, vj)); +} + +/* Emits the `vftintrm.l.d vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vftintrm_l_d(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VFTINTRM_L_D, vd, vj)); +} + +/* Emits the `vftintrp.w.s vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vftintrp_w_s(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VFTINTRP_W_S, vd, vj)); +} + +/* Emits the `vftintrp.l.d vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vftintrp_l_d(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VFTINTRP_L_D, vd, vj)); +} + +/* Emits the `vftintrz.w.s vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vftintrz_w_s(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VFTINTRZ_W_S, vd, vj)); +} + +/* Emits the `vftintrz.l.d vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vftintrz_l_d(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VFTINTRZ_L_D, vd, vj)); +} + +/* Emits the `vftintrne.w.s vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vftintrne_w_s(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VFTINTRNE_W_S, vd, vj)); +} + +/* Emits the `vftintrne.l.d vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vftintrne_l_d(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VFTINTRNE_L_D, vd, vj)); +} + +/* Emits the `vftint.wu.s vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vftint_wu_s(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VFTINT_WU_S, vd, vj)); +} + +/* Emits the `vftint.lu.d vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vftint_lu_d(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VFTINT_LU_D, vd, vj)); +} + +/* Emits the `vftintrz.wu.s vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vftintrz_wu_s(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VFTINTRZ_WU_S, vd, vj)); +} + +/* Emits the `vftintrz.lu.d vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vftintrz_lu_d(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VFTINTRZ_LU_D, vd, vj)); +} + +/* Emits the `vftintl.l.s vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vftintl_l_s(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VFTINTL_L_S, vd, vj)); +} + +/* Emits the `vftinth.l.s vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vftinth_l_s(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VFTINTH_L_S, vd, vj)); +} + +/* Emits the `vftintrml.l.s vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vftintrml_l_s(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VFTINTRML_L_S, vd, vj)); +} + +/* Emits the `vftintrmh.l.s vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vftintrmh_l_s(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VFTINTRMH_L_S, vd, vj)); +} + +/* Emits the `vftintrpl.l.s vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vftintrpl_l_s(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VFTINTRPL_L_S, vd, vj)); +} + +/* Emits the `vftintrph.l.s vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vftintrph_l_s(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VFTINTRPH_L_S, vd, vj)); +} + +/* Emits the `vftintrzl.l.s vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vftintrzl_l_s(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VFTINTRZL_L_S, vd, vj)); +} + +/* Emits the `vftintrzh.l.s vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vftintrzh_l_s(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VFTINTRZH_L_S, vd, vj)); +} + +/* Emits the `vftintrnel.l.s vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vftintrnel_l_s(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VFTINTRNEL_L_S, vd, vj)); +} + +/* Emits the `vftintrneh.l.s vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vftintrneh_l_s(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VFTINTRNEH_L_S, vd, vj)); +} + +/* Emits the `vexth.h.b vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vexth_h_b(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VEXTH_H_B, vd, vj)); +} + +/* Emits the `vexth.w.h vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vexth_w_h(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VEXTH_W_H, vd, vj)); +} + +/* Emits the `vexth.d.w vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vexth_d_w(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VEXTH_D_W, vd, vj)); +} + +/* Emits the `vexth.q.d vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vexth_q_d(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VEXTH_Q_D, vd, vj)); +} + +/* Emits the `vexth.hu.bu vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vexth_hu_bu(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VEXTH_HU_BU, vd, vj)); +} + +/* Emits the `vexth.wu.hu vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vexth_wu_hu(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VEXTH_WU_HU, vd, vj)); +} + +/* Emits the `vexth.du.wu vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vexth_du_wu(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VEXTH_DU_WU, vd, vj)); +} + +/* Emits the `vexth.qu.du vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vexth_qu_du(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VEXTH_QU_DU, vd, vj)); +} + +/* Emits the `vreplgr2vr.b vd, j` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vreplgr2vr_b(TCGContext *s, TCGReg vd, TCGReg j) +{ + tcg_out32(s, encode_vdj_insn(OPC_VREPLGR2VR_B, vd, j)); +} + +/* Emits the `vreplgr2vr.h vd, j` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vreplgr2vr_h(TCGContext *s, TCGReg vd, TCGReg j) +{ + tcg_out32(s, encode_vdj_insn(OPC_VREPLGR2VR_H, vd, j)); +} + +/* Emits the `vreplgr2vr.w vd, j` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vreplgr2vr_w(TCGContext *s, TCGReg vd, TCGReg j) +{ + tcg_out32(s, encode_vdj_insn(OPC_VREPLGR2VR_W, vd, j)); +} + +/* Emits the `vreplgr2vr.d vd, j` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vreplgr2vr_d(TCGContext *s, TCGReg vd, TCGReg j) +{ + tcg_out32(s, encode_vdj_insn(OPC_VREPLGR2VR_D, vd, j)); +} + +/* Emits the `vrotri.b vd, vj, uk3` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vrotri_b(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk3) +{ + tcg_out32(s, encode_vdvjuk3_insn(OPC_VROTRI_B, vd, vj, uk3)); +} + +/* Emits the `vrotri.h vd, vj, uk4` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vrotri_h(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk4) +{ + tcg_out32(s, encode_vdvjuk4_insn(OPC_VROTRI_H, vd, vj, uk4)); +} + +/* Emits the `vrotri.w vd, vj, uk5` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vrotri_w(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk5) +{ + tcg_out32(s, encode_vdvjuk5_insn(OPC_VROTRI_W, vd, vj, uk5)); +} + +/* Emits the `vrotri.d vd, vj, uk6` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vrotri_d(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk6) +{ + tcg_out32(s, encode_vdvjuk6_insn(OPC_VROTRI_D, vd, vj, uk6)); +} + +/* Emits the `vsrlri.b vd, vj, uk3` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsrlri_b(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk3) +{ + tcg_out32(s, encode_vdvjuk3_insn(OPC_VSRLRI_B, vd, vj, uk3)); +} + +/* Emits the `vsrlri.h vd, vj, uk4` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsrlri_h(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk4) +{ + tcg_out32(s, encode_vdvjuk4_insn(OPC_VSRLRI_H, vd, vj, uk4)); +} + +/* Emits the `vsrlri.w vd, vj, uk5` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsrlri_w(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk5) +{ + tcg_out32(s, encode_vdvjuk5_insn(OPC_VSRLRI_W, vd, vj, uk5)); +} + +/* Emits the `vsrlri.d vd, vj, uk6` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsrlri_d(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk6) +{ + tcg_out32(s, encode_vdvjuk6_insn(OPC_VSRLRI_D, vd, vj, uk6)); +} + +/* Emits the `vsrari.b vd, vj, uk3` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsrari_b(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk3) +{ + tcg_out32(s, encode_vdvjuk3_insn(OPC_VSRARI_B, vd, vj, uk3)); +} + +/* Emits the `vsrari.h vd, vj, uk4` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsrari_h(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk4) +{ + tcg_out32(s, encode_vdvjuk4_insn(OPC_VSRARI_H, vd, vj, uk4)); +} + +/* Emits the `vsrari.w vd, vj, uk5` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsrari_w(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk5) +{ + tcg_out32(s, encode_vdvjuk5_insn(OPC_VSRARI_W, vd, vj, uk5)); +} + +/* Emits the `vsrari.d vd, vj, uk6` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsrari_d(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk6) +{ + tcg_out32(s, encode_vdvjuk6_insn(OPC_VSRARI_D, vd, vj, uk6)); +} + +/* Emits the `vinsgr2vr.b vd, j, uk4` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vinsgr2vr_b(TCGContext *s, TCGReg vd, TCGReg j, uint32_t uk4) +{ + tcg_out32(s, encode_vdjuk4_insn(OPC_VINSGR2VR_B, vd, j, uk4)); +} + +/* Emits the `vinsgr2vr.h vd, j, uk3` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vinsgr2vr_h(TCGContext *s, TCGReg vd, TCGReg j, uint32_t uk3) +{ + tcg_out32(s, encode_vdjuk3_insn(OPC_VINSGR2VR_H, vd, j, uk3)); +} + +/* Emits the `vinsgr2vr.w vd, j, uk2` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vinsgr2vr_w(TCGContext *s, TCGReg vd, TCGReg j, uint32_t uk2) +{ + tcg_out32(s, encode_vdjuk2_insn(OPC_VINSGR2VR_W, vd, j, uk2)); +} + +/* Emits the `vinsgr2vr.d vd, j, uk1` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vinsgr2vr_d(TCGContext *s, TCGReg vd, TCGReg j, uint32_t uk1) +{ + tcg_out32(s, encode_vdjuk1_insn(OPC_VINSGR2VR_D, vd, j, uk1)); +} + +/* Emits the `vpickve2gr.b d, vj, uk4` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vpickve2gr_b(TCGContext *s, TCGReg d, TCGReg vj, uint32_t uk4) +{ + tcg_out32(s, encode_dvjuk4_insn(OPC_VPICKVE2GR_B, d, vj, uk4)); +} + +/* Emits the `vpickve2gr.h d, vj, uk3` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vpickve2gr_h(TCGContext *s, TCGReg d, TCGReg vj, uint32_t uk3) +{ + tcg_out32(s, encode_dvjuk3_insn(OPC_VPICKVE2GR_H, d, vj, uk3)); +} + +/* Emits the `vpickve2gr.w d, vj, uk2` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vpickve2gr_w(TCGContext *s, TCGReg d, TCGReg vj, uint32_t uk2) +{ + tcg_out32(s, encode_dvjuk2_insn(OPC_VPICKVE2GR_W, d, vj, uk2)); +} + +/* Emits the `vpickve2gr.d d, vj, uk1` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vpickve2gr_d(TCGContext *s, TCGReg d, TCGReg vj, uint32_t uk1) +{ + tcg_out32(s, encode_dvjuk1_insn(OPC_VPICKVE2GR_D, d, vj, uk1)); +} + +/* Emits the `vpickve2gr.bu d, vj, uk4` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vpickve2gr_bu(TCGContext *s, TCGReg d, TCGReg vj, uint32_t uk4) +{ + tcg_out32(s, encode_dvjuk4_insn(OPC_VPICKVE2GR_BU, d, vj, uk4)); +} + +/* Emits the `vpickve2gr.hu d, vj, uk3` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vpickve2gr_hu(TCGContext *s, TCGReg d, TCGReg vj, uint32_t uk3) +{ + tcg_out32(s, encode_dvjuk3_insn(OPC_VPICKVE2GR_HU, d, vj, uk3)); +} + +/* Emits the `vpickve2gr.wu d, vj, uk2` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vpickve2gr_wu(TCGContext *s, TCGReg d, TCGReg vj, uint32_t uk2) +{ + tcg_out32(s, encode_dvjuk2_insn(OPC_VPICKVE2GR_WU, d, vj, uk2)); +} + +/* Emits the `vpickve2gr.du d, vj, uk1` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vpickve2gr_du(TCGContext *s, TCGReg d, TCGReg vj, uint32_t uk1) +{ + tcg_out32(s, encode_dvjuk1_insn(OPC_VPICKVE2GR_DU, d, vj, uk1)); +} + +/* Emits the `vreplvei.b vd, vj, uk4` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vreplvei_b(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk4) +{ + tcg_out32(s, encode_vdvjuk4_insn(OPC_VREPLVEI_B, vd, vj, uk4)); +} + +/* Emits the `vreplvei.h vd, vj, uk3` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vreplvei_h(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk3) +{ + tcg_out32(s, encode_vdvjuk3_insn(OPC_VREPLVEI_H, vd, vj, uk3)); +} + +/* Emits the `vreplvei.w vd, vj, uk2` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vreplvei_w(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk2) +{ + tcg_out32(s, encode_vdvjuk2_insn(OPC_VREPLVEI_W, vd, vj, uk2)); +} + +/* Emits the `vreplvei.d vd, vj, uk1` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vreplvei_d(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk1) +{ + tcg_out32(s, encode_vdvjuk1_insn(OPC_VREPLVEI_D, vd, vj, uk1)); +} + +/* Emits the `vsllwil.h.b vd, vj, uk3` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsllwil_h_b(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk3) +{ + tcg_out32(s, encode_vdvjuk3_insn(OPC_VSLLWIL_H_B, vd, vj, uk3)); +} + +/* Emits the `vsllwil.w.h vd, vj, uk4` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsllwil_w_h(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk4) +{ + tcg_out32(s, encode_vdvjuk4_insn(OPC_VSLLWIL_W_H, vd, vj, uk4)); +} + +/* Emits the `vsllwil.d.w vd, vj, uk5` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsllwil_d_w(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk5) +{ + tcg_out32(s, encode_vdvjuk5_insn(OPC_VSLLWIL_D_W, vd, vj, uk5)); +} + +/* Emits the `vextl.q.d vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vextl_q_d(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VEXTL_Q_D, vd, vj)); +} + +/* Emits the `vsllwil.hu.bu vd, vj, uk3` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsllwil_hu_bu(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk3) +{ + tcg_out32(s, encode_vdvjuk3_insn(OPC_VSLLWIL_HU_BU, vd, vj, uk3)); +} + +/* Emits the `vsllwil.wu.hu vd, vj, uk4` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsllwil_wu_hu(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk4) +{ + tcg_out32(s, encode_vdvjuk4_insn(OPC_VSLLWIL_WU_HU, vd, vj, uk4)); +} + +/* Emits the `vsllwil.du.wu vd, vj, uk5` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsllwil_du_wu(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk5) +{ + tcg_out32(s, encode_vdvjuk5_insn(OPC_VSLLWIL_DU_WU, vd, vj, uk5)); +} + +/* Emits the `vextl.qu.du vd, vj` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vextl_qu_du(TCGContext *s, TCGReg vd, TCGReg vj) +{ + tcg_out32(s, encode_vdvj_insn(OPC_VEXTL_QU_DU, vd, vj)); +} + +/* Emits the `vbitclri.b vd, vj, uk3` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vbitclri_b(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk3) +{ + tcg_out32(s, encode_vdvjuk3_insn(OPC_VBITCLRI_B, vd, vj, uk3)); +} + +/* Emits the `vbitclri.h vd, vj, uk4` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vbitclri_h(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk4) +{ + tcg_out32(s, encode_vdvjuk4_insn(OPC_VBITCLRI_H, vd, vj, uk4)); +} + +/* Emits the `vbitclri.w vd, vj, uk5` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vbitclri_w(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk5) +{ + tcg_out32(s, encode_vdvjuk5_insn(OPC_VBITCLRI_W, vd, vj, uk5)); +} + +/* Emits the `vbitclri.d vd, vj, uk6` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vbitclri_d(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk6) +{ + tcg_out32(s, encode_vdvjuk6_insn(OPC_VBITCLRI_D, vd, vj, uk6)); +} + +/* Emits the `vbitseti.b vd, vj, uk3` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vbitseti_b(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk3) +{ + tcg_out32(s, encode_vdvjuk3_insn(OPC_VBITSETI_B, vd, vj, uk3)); +} + +/* Emits the `vbitseti.h vd, vj, uk4` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vbitseti_h(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk4) +{ + tcg_out32(s, encode_vdvjuk4_insn(OPC_VBITSETI_H, vd, vj, uk4)); +} + +/* Emits the `vbitseti.w vd, vj, uk5` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vbitseti_w(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk5) +{ + tcg_out32(s, encode_vdvjuk5_insn(OPC_VBITSETI_W, vd, vj, uk5)); +} + +/* Emits the `vbitseti.d vd, vj, uk6` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vbitseti_d(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk6) +{ + tcg_out32(s, encode_vdvjuk6_insn(OPC_VBITSETI_D, vd, vj, uk6)); +} + +/* Emits the `vbitrevi.b vd, vj, uk3` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vbitrevi_b(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk3) +{ + tcg_out32(s, encode_vdvjuk3_insn(OPC_VBITREVI_B, vd, vj, uk3)); +} + +/* Emits the `vbitrevi.h vd, vj, uk4` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vbitrevi_h(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk4) +{ + tcg_out32(s, encode_vdvjuk4_insn(OPC_VBITREVI_H, vd, vj, uk4)); +} + +/* Emits the `vbitrevi.w vd, vj, uk5` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vbitrevi_w(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk5) +{ + tcg_out32(s, encode_vdvjuk5_insn(OPC_VBITREVI_W, vd, vj, uk5)); +} + +/* Emits the `vbitrevi.d vd, vj, uk6` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vbitrevi_d(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk6) +{ + tcg_out32(s, encode_vdvjuk6_insn(OPC_VBITREVI_D, vd, vj, uk6)); +} + +/* Emits the `vsat.b vd, vj, uk3` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsat_b(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk3) +{ + tcg_out32(s, encode_vdvjuk3_insn(OPC_VSAT_B, vd, vj, uk3)); +} + +/* Emits the `vsat.h vd, vj, uk4` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsat_h(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk4) +{ + tcg_out32(s, encode_vdvjuk4_insn(OPC_VSAT_H, vd, vj, uk4)); +} + +/* Emits the `vsat.w vd, vj, uk5` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsat_w(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk5) +{ + tcg_out32(s, encode_vdvjuk5_insn(OPC_VSAT_W, vd, vj, uk5)); +} + +/* Emits the `vsat.d vd, vj, uk6` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsat_d(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk6) +{ + tcg_out32(s, encode_vdvjuk6_insn(OPC_VSAT_D, vd, vj, uk6)); +} + +/* Emits the `vsat.bu vd, vj, uk3` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsat_bu(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk3) +{ + tcg_out32(s, encode_vdvjuk3_insn(OPC_VSAT_BU, vd, vj, uk3)); +} + +/* Emits the `vsat.hu vd, vj, uk4` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsat_hu(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk4) +{ + tcg_out32(s, encode_vdvjuk4_insn(OPC_VSAT_HU, vd, vj, uk4)); +} + +/* Emits the `vsat.wu vd, vj, uk5` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsat_wu(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk5) +{ + tcg_out32(s, encode_vdvjuk5_insn(OPC_VSAT_WU, vd, vj, uk5)); +} + +/* Emits the `vsat.du vd, vj, uk6` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsat_du(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk6) +{ + tcg_out32(s, encode_vdvjuk6_insn(OPC_VSAT_DU, vd, vj, uk6)); +} + +/* Emits the `vslli.b vd, vj, uk3` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vslli_b(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk3) +{ + tcg_out32(s, encode_vdvjuk3_insn(OPC_VSLLI_B, vd, vj, uk3)); +} + +/* Emits the `vslli.h vd, vj, uk4` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vslli_h(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk4) +{ + tcg_out32(s, encode_vdvjuk4_insn(OPC_VSLLI_H, vd, vj, uk4)); +} + +/* Emits the `vslli.w vd, vj, uk5` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vslli_w(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk5) +{ + tcg_out32(s, encode_vdvjuk5_insn(OPC_VSLLI_W, vd, vj, uk5)); +} + +/* Emits the `vslli.d vd, vj, uk6` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vslli_d(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk6) +{ + tcg_out32(s, encode_vdvjuk6_insn(OPC_VSLLI_D, vd, vj, uk6)); +} + +/* Emits the `vsrli.b vd, vj, uk3` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsrli_b(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk3) +{ + tcg_out32(s, encode_vdvjuk3_insn(OPC_VSRLI_B, vd, vj, uk3)); +} + +/* Emits the `vsrli.h vd, vj, uk4` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsrli_h(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk4) +{ + tcg_out32(s, encode_vdvjuk4_insn(OPC_VSRLI_H, vd, vj, uk4)); +} + +/* Emits the `vsrli.w vd, vj, uk5` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsrli_w(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk5) +{ + tcg_out32(s, encode_vdvjuk5_insn(OPC_VSRLI_W, vd, vj, uk5)); +} + +/* Emits the `vsrli.d vd, vj, uk6` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsrli_d(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk6) +{ + tcg_out32(s, encode_vdvjuk6_insn(OPC_VSRLI_D, vd, vj, uk6)); +} + +/* Emits the `vsrai.b vd, vj, uk3` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsrai_b(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk3) +{ + tcg_out32(s, encode_vdvjuk3_insn(OPC_VSRAI_B, vd, vj, uk3)); +} + +/* Emits the `vsrai.h vd, vj, uk4` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsrai_h(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk4) +{ + tcg_out32(s, encode_vdvjuk4_insn(OPC_VSRAI_H, vd, vj, uk4)); +} + +/* Emits the `vsrai.w vd, vj, uk5` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsrai_w(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk5) +{ + tcg_out32(s, encode_vdvjuk5_insn(OPC_VSRAI_W, vd, vj, uk5)); +} + +/* Emits the `vsrai.d vd, vj, uk6` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsrai_d(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk6) +{ + tcg_out32(s, encode_vdvjuk6_insn(OPC_VSRAI_D, vd, vj, uk6)); +} + +/* Emits the `vsrlni.b.h vd, vj, uk4` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsrlni_b_h(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk4) +{ + tcg_out32(s, encode_vdvjuk4_insn(OPC_VSRLNI_B_H, vd, vj, uk4)); +} + +/* Emits the `vsrlni.h.w vd, vj, uk5` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsrlni_h_w(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk5) +{ + tcg_out32(s, encode_vdvjuk5_insn(OPC_VSRLNI_H_W, vd, vj, uk5)); +} + +/* Emits the `vsrlni.w.d vd, vj, uk6` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsrlni_w_d(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk6) +{ + tcg_out32(s, encode_vdvjuk6_insn(OPC_VSRLNI_W_D, vd, vj, uk6)); +} + +/* Emits the `vsrlni.d.q vd, vj, uk7` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsrlni_d_q(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk7) +{ + tcg_out32(s, encode_vdvjuk7_insn(OPC_VSRLNI_D_Q, vd, vj, uk7)); +} + +/* Emits the `vsrlrni.b.h vd, vj, uk4` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsrlrni_b_h(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk4) +{ + tcg_out32(s, encode_vdvjuk4_insn(OPC_VSRLRNI_B_H, vd, vj, uk4)); +} + +/* Emits the `vsrlrni.h.w vd, vj, uk5` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsrlrni_h_w(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk5) +{ + tcg_out32(s, encode_vdvjuk5_insn(OPC_VSRLRNI_H_W, vd, vj, uk5)); +} + +/* Emits the `vsrlrni.w.d vd, vj, uk6` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsrlrni_w_d(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk6) +{ + tcg_out32(s, encode_vdvjuk6_insn(OPC_VSRLRNI_W_D, vd, vj, uk6)); +} + +/* Emits the `vsrlrni.d.q vd, vj, uk7` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsrlrni_d_q(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk7) +{ + tcg_out32(s, encode_vdvjuk7_insn(OPC_VSRLRNI_D_Q, vd, vj, uk7)); +} + +/* Emits the `vssrlni.b.h vd, vj, uk4` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vssrlni_b_h(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk4) +{ + tcg_out32(s, encode_vdvjuk4_insn(OPC_VSSRLNI_B_H, vd, vj, uk4)); +} + +/* Emits the `vssrlni.h.w vd, vj, uk5` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vssrlni_h_w(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk5) +{ + tcg_out32(s, encode_vdvjuk5_insn(OPC_VSSRLNI_H_W, vd, vj, uk5)); +} + +/* Emits the `vssrlni.w.d vd, vj, uk6` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vssrlni_w_d(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk6) +{ + tcg_out32(s, encode_vdvjuk6_insn(OPC_VSSRLNI_W_D, vd, vj, uk6)); +} + +/* Emits the `vssrlni.d.q vd, vj, uk7` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vssrlni_d_q(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk7) +{ + tcg_out32(s, encode_vdvjuk7_insn(OPC_VSSRLNI_D_Q, vd, vj, uk7)); +} + +/* Emits the `vssrlni.bu.h vd, vj, uk4` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vssrlni_bu_h(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk4) +{ + tcg_out32(s, encode_vdvjuk4_insn(OPC_VSSRLNI_BU_H, vd, vj, uk4)); +} + +/* Emits the `vssrlni.hu.w vd, vj, uk5` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vssrlni_hu_w(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk5) +{ + tcg_out32(s, encode_vdvjuk5_insn(OPC_VSSRLNI_HU_W, vd, vj, uk5)); +} + +/* Emits the `vssrlni.wu.d vd, vj, uk6` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vssrlni_wu_d(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk6) +{ + tcg_out32(s, encode_vdvjuk6_insn(OPC_VSSRLNI_WU_D, vd, vj, uk6)); +} + +/* Emits the `vssrlni.du.q vd, vj, uk7` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vssrlni_du_q(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk7) +{ + tcg_out32(s, encode_vdvjuk7_insn(OPC_VSSRLNI_DU_Q, vd, vj, uk7)); +} + +/* Emits the `vssrlrni.b.h vd, vj, uk4` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vssrlrni_b_h(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk4) +{ + tcg_out32(s, encode_vdvjuk4_insn(OPC_VSSRLRNI_B_H, vd, vj, uk4)); +} + +/* Emits the `vssrlrni.h.w vd, vj, uk5` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vssrlrni_h_w(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk5) +{ + tcg_out32(s, encode_vdvjuk5_insn(OPC_VSSRLRNI_H_W, vd, vj, uk5)); +} + +/* Emits the `vssrlrni.w.d vd, vj, uk6` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vssrlrni_w_d(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk6) +{ + tcg_out32(s, encode_vdvjuk6_insn(OPC_VSSRLRNI_W_D, vd, vj, uk6)); +} + +/* Emits the `vssrlrni.d.q vd, vj, uk7` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vssrlrni_d_q(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk7) +{ + tcg_out32(s, encode_vdvjuk7_insn(OPC_VSSRLRNI_D_Q, vd, vj, uk7)); +} + +/* Emits the `vssrlrni.bu.h vd, vj, uk4` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vssrlrni_bu_h(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk4) +{ + tcg_out32(s, encode_vdvjuk4_insn(OPC_VSSRLRNI_BU_H, vd, vj, uk4)); +} + +/* Emits the `vssrlrni.hu.w vd, vj, uk5` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vssrlrni_hu_w(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk5) +{ + tcg_out32(s, encode_vdvjuk5_insn(OPC_VSSRLRNI_HU_W, vd, vj, uk5)); +} + +/* Emits the `vssrlrni.wu.d vd, vj, uk6` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vssrlrni_wu_d(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk6) +{ + tcg_out32(s, encode_vdvjuk6_insn(OPC_VSSRLRNI_WU_D, vd, vj, uk6)); +} + +/* Emits the `vssrlrni.du.q vd, vj, uk7` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vssrlrni_du_q(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk7) +{ + tcg_out32(s, encode_vdvjuk7_insn(OPC_VSSRLRNI_DU_Q, vd, vj, uk7)); +} + +/* Emits the `vsrani.b.h vd, vj, uk4` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsrani_b_h(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk4) +{ + tcg_out32(s, encode_vdvjuk4_insn(OPC_VSRANI_B_H, vd, vj, uk4)); +} + +/* Emits the `vsrani.h.w vd, vj, uk5` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsrani_h_w(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk5) +{ + tcg_out32(s, encode_vdvjuk5_insn(OPC_VSRANI_H_W, vd, vj, uk5)); +} + +/* Emits the `vsrani.w.d vd, vj, uk6` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsrani_w_d(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk6) +{ + tcg_out32(s, encode_vdvjuk6_insn(OPC_VSRANI_W_D, vd, vj, uk6)); +} + +/* Emits the `vsrani.d.q vd, vj, uk7` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsrani_d_q(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk7) +{ + tcg_out32(s, encode_vdvjuk7_insn(OPC_VSRANI_D_Q, vd, vj, uk7)); +} + +/* Emits the `vsrarni.b.h vd, vj, uk4` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsrarni_b_h(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk4) +{ + tcg_out32(s, encode_vdvjuk4_insn(OPC_VSRARNI_B_H, vd, vj, uk4)); +} + +/* Emits the `vsrarni.h.w vd, vj, uk5` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsrarni_h_w(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk5) +{ + tcg_out32(s, encode_vdvjuk5_insn(OPC_VSRARNI_H_W, vd, vj, uk5)); +} + +/* Emits the `vsrarni.w.d vd, vj, uk6` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsrarni_w_d(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk6) +{ + tcg_out32(s, encode_vdvjuk6_insn(OPC_VSRARNI_W_D, vd, vj, uk6)); +} + +/* Emits the `vsrarni.d.q vd, vj, uk7` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vsrarni_d_q(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk7) +{ + tcg_out32(s, encode_vdvjuk7_insn(OPC_VSRARNI_D_Q, vd, vj, uk7)); +} + +/* Emits the `vssrani.b.h vd, vj, uk4` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vssrani_b_h(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk4) +{ + tcg_out32(s, encode_vdvjuk4_insn(OPC_VSSRANI_B_H, vd, vj, uk4)); +} + +/* Emits the `vssrani.h.w vd, vj, uk5` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vssrani_h_w(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk5) +{ + tcg_out32(s, encode_vdvjuk5_insn(OPC_VSSRANI_H_W, vd, vj, uk5)); +} + +/* Emits the `vssrani.w.d vd, vj, uk6` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vssrani_w_d(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk6) +{ + tcg_out32(s, encode_vdvjuk6_insn(OPC_VSSRANI_W_D, vd, vj, uk6)); +} + +/* Emits the `vssrani.d.q vd, vj, uk7` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vssrani_d_q(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk7) +{ + tcg_out32(s, encode_vdvjuk7_insn(OPC_VSSRANI_D_Q, vd, vj, uk7)); +} + +/* Emits the `vssrani.bu.h vd, vj, uk4` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vssrani_bu_h(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk4) +{ + tcg_out32(s, encode_vdvjuk4_insn(OPC_VSSRANI_BU_H, vd, vj, uk4)); +} + +/* Emits the `vssrani.hu.w vd, vj, uk5` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vssrani_hu_w(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk5) +{ + tcg_out32(s, encode_vdvjuk5_insn(OPC_VSSRANI_HU_W, vd, vj, uk5)); +} + +/* Emits the `vssrani.wu.d vd, vj, uk6` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vssrani_wu_d(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk6) +{ + tcg_out32(s, encode_vdvjuk6_insn(OPC_VSSRANI_WU_D, vd, vj, uk6)); +} + +/* Emits the `vssrani.du.q vd, vj, uk7` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vssrani_du_q(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk7) +{ + tcg_out32(s, encode_vdvjuk7_insn(OPC_VSSRANI_DU_Q, vd, vj, uk7)); +} + +/* Emits the `vssrarni.b.h vd, vj, uk4` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vssrarni_b_h(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk4) +{ + tcg_out32(s, encode_vdvjuk4_insn(OPC_VSSRARNI_B_H, vd, vj, uk4)); +} + +/* Emits the `vssrarni.h.w vd, vj, uk5` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vssrarni_h_w(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk5) +{ + tcg_out32(s, encode_vdvjuk5_insn(OPC_VSSRARNI_H_W, vd, vj, uk5)); +} + +/* Emits the `vssrarni.w.d vd, vj, uk6` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vssrarni_w_d(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk6) +{ + tcg_out32(s, encode_vdvjuk6_insn(OPC_VSSRARNI_W_D, vd, vj, uk6)); +} + +/* Emits the `vssrarni.d.q vd, vj, uk7` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vssrarni_d_q(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk7) +{ + tcg_out32(s, encode_vdvjuk7_insn(OPC_VSSRARNI_D_Q, vd, vj, uk7)); +} + +/* Emits the `vssrarni.bu.h vd, vj, uk4` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vssrarni_bu_h(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk4) +{ + tcg_out32(s, encode_vdvjuk4_insn(OPC_VSSRARNI_BU_H, vd, vj, uk4)); +} + +/* Emits the `vssrarni.hu.w vd, vj, uk5` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vssrarni_hu_w(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk5) +{ + tcg_out32(s, encode_vdvjuk5_insn(OPC_VSSRARNI_HU_W, vd, vj, uk5)); +} + +/* Emits the `vssrarni.wu.d vd, vj, uk6` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vssrarni_wu_d(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk6) +{ + tcg_out32(s, encode_vdvjuk6_insn(OPC_VSSRARNI_WU_D, vd, vj, uk6)); +} + +/* Emits the `vssrarni.du.q vd, vj, uk7` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vssrarni_du_q(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk7) +{ + tcg_out32(s, encode_vdvjuk7_insn(OPC_VSSRARNI_DU_Q, vd, vj, uk7)); +} + +/* Emits the `vextrins.d vd, vj, uk8` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vextrins_d(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk8) +{ + tcg_out32(s, encode_vdvjuk8_insn(OPC_VEXTRINS_D, vd, vj, uk8)); +} + +/* Emits the `vextrins.w vd, vj, uk8` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vextrins_w(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk8) +{ + tcg_out32(s, encode_vdvjuk8_insn(OPC_VEXTRINS_W, vd, vj, uk8)); +} + +/* Emits the `vextrins.h vd, vj, uk8` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vextrins_h(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk8) +{ + tcg_out32(s, encode_vdvjuk8_insn(OPC_VEXTRINS_H, vd, vj, uk8)); +} + +/* Emits the `vextrins.b vd, vj, uk8` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vextrins_b(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk8) +{ + tcg_out32(s, encode_vdvjuk8_insn(OPC_VEXTRINS_B, vd, vj, uk8)); +} + +/* Emits the `vshuf4i.b vd, vj, uk8` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vshuf4i_b(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk8) +{ + tcg_out32(s, encode_vdvjuk8_insn(OPC_VSHUF4I_B, vd, vj, uk8)); +} + +/* Emits the `vshuf4i.h vd, vj, uk8` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vshuf4i_h(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk8) +{ + tcg_out32(s, encode_vdvjuk8_insn(OPC_VSHUF4I_H, vd, vj, uk8)); +} + +/* Emits the `vshuf4i.w vd, vj, uk8` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vshuf4i_w(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk8) +{ + tcg_out32(s, encode_vdvjuk8_insn(OPC_VSHUF4I_W, vd, vj, uk8)); +} + +/* Emits the `vshuf4i.d vd, vj, uk8` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vshuf4i_d(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk8) +{ + tcg_out32(s, encode_vdvjuk8_insn(OPC_VSHUF4I_D, vd, vj, uk8)); +} + +/* Emits the `vbitseli.b vd, vj, uk8` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vbitseli_b(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk8) +{ + tcg_out32(s, encode_vdvjuk8_insn(OPC_VBITSELI_B, vd, vj, uk8)); +} + +/* Emits the `vandi.b vd, vj, uk8` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vandi_b(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk8) +{ + tcg_out32(s, encode_vdvjuk8_insn(OPC_VANDI_B, vd, vj, uk8)); +} + +/* Emits the `vori.b vd, vj, uk8` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vori_b(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk8) +{ + tcg_out32(s, encode_vdvjuk8_insn(OPC_VORI_B, vd, vj, uk8)); +} + +/* Emits the `vxori.b vd, vj, uk8` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vxori_b(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk8) +{ + tcg_out32(s, encode_vdvjuk8_insn(OPC_VXORI_B, vd, vj, uk8)); +} + +/* Emits the `vnori.b vd, vj, uk8` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vnori_b(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk8) +{ + tcg_out32(s, encode_vdvjuk8_insn(OPC_VNORI_B, vd, vj, uk8)); +} + +/* Emits the `vldi vd, sj13` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vldi(TCGContext *s, TCGReg vd, int32_t sj13) +{ + tcg_out32(s, encode_vdsj13_insn(OPC_VLDI, vd, sj13)); +} + +/* Emits the `vpermi.w vd, vj, uk8` instruction. */ +static void __attribute__((unused)) +tcg_out_opc_vpermi_w(TCGContext *s, TCGReg vd, TCGReg vj, uint32_t uk8) +{ + tcg_out32(s, encode_vdvjuk8_insn(OPC_VPERMI_W, vd, vj, uk8)); +} + /* End of generated code. */ -- 2.34.1 From: Jiajie Chen <c@jia.je> LSX support on host cpu is detected via hwcap. Lower the following ops to LSX: - dup_vec - dupi_vec - dupm_vec - ld_vec - st_vec Signed-off-by: Jiajie Chen <c@jia.je> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-Id: <20230908022302.180442-3-c@jia.je> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- tcg/loongarch64/tcg-target-con-set.h | 2 + tcg/loongarch64/tcg-target-con-str.h | 1 + tcg/loongarch64/tcg-target.h | 38 ++++- tcg/loongarch64/tcg-target.opc.h | 12 ++ tcg/loongarch64/tcg-target.c.inc | 219 ++++++++++++++++++++++++++- 5 files changed, 270 insertions(+), 2 deletions(-) create mode 100644 tcg/loongarch64/tcg-target.opc.h diff --git a/tcg/loongarch64/tcg-target-con-set.h b/tcg/loongarch64/tcg-target-con-set.h index XXXXXXX..XXXXXXX 100644 --- a/tcg/loongarch64/tcg-target-con-set.h +++ b/tcg/loongarch64/tcg-target-con-set.h @@ -XXX,XX +XXX,XX @@ C_O0_I1(r) C_O0_I2(rZ, r) C_O0_I2(rZ, rZ) +C_O0_I2(w, r) C_O1_I1(r, r) +C_O1_I1(w, r) C_O1_I2(r, r, rC) C_O1_I2(r, r, ri) C_O1_I2(r, r, rI) diff --git a/tcg/loongarch64/tcg-target-con-str.h b/tcg/loongarch64/tcg-target-con-str.h index XXXXXXX..XXXXXXX 100644 --- a/tcg/loongarch64/tcg-target-con-str.h +++ b/tcg/loongarch64/tcg-target-con-str.h @@ -XXX,XX +XXX,XX @@ * REGS(letter, register_mask) */ REGS('r', ALL_GENERAL_REGS) +REGS('w', ALL_VECTOR_REGS) /* * Define constraint letters for constants: diff --git a/tcg/loongarch64/tcg-target.h b/tcg/loongarch64/tcg-target.h index XXXXXXX..XXXXXXX 100644 --- a/tcg/loongarch64/tcg-target.h +++ b/tcg/loongarch64/tcg-target.h @@ -XXX,XX +XXX,XX @@ #define LOONGARCH_TCG_TARGET_H #define TCG_TARGET_INSN_UNIT_SIZE 4 -#define TCG_TARGET_NB_REGS 32 +#define TCG_TARGET_NB_REGS 64 #define MAX_CODE_GEN_BUFFER_SIZE ((size_t)-1) @@ -XXX,XX +XXX,XX @@ typedef enum { TCG_REG_S7, TCG_REG_S8, + TCG_REG_V0 = 32, TCG_REG_V1, TCG_REG_V2, TCG_REG_V3, + TCG_REG_V4, TCG_REG_V5, TCG_REG_V6, TCG_REG_V7, + TCG_REG_V8, TCG_REG_V9, TCG_REG_V10, TCG_REG_V11, + TCG_REG_V12, TCG_REG_V13, TCG_REG_V14, TCG_REG_V15, + TCG_REG_V16, TCG_REG_V17, TCG_REG_V18, TCG_REG_V19, + TCG_REG_V20, TCG_REG_V21, TCG_REG_V22, TCG_REG_V23, + TCG_REG_V24, TCG_REG_V25, TCG_REG_V26, TCG_REG_V27, + TCG_REG_V28, TCG_REG_V29, TCG_REG_V30, TCG_REG_V31, + /* aliases */ TCG_AREG0 = TCG_REG_S0, TCG_REG_TMP0 = TCG_REG_T8, TCG_REG_TMP1 = TCG_REG_T7, TCG_REG_TMP2 = TCG_REG_T6, + TCG_VEC_TMP0 = TCG_REG_V23, } TCGReg; +extern bool use_lsx_instructions; + /* used for function call generation */ #define TCG_REG_CALL_STACK TCG_REG_SP #define TCG_TARGET_STACK_ALIGN 16 @@ -XXX,XX +XXX,XX @@ typedef enum { #define TCG_TARGET_HAS_qemu_ldst_i128 0 +#define TCG_TARGET_HAS_v64 0 +#define TCG_TARGET_HAS_v128 use_lsx_instructions +#define TCG_TARGET_HAS_v256 0 + +#define TCG_TARGET_HAS_not_vec 0 +#define TCG_TARGET_HAS_neg_vec 0 +#define TCG_TARGET_HAS_abs_vec 0 +#define TCG_TARGET_HAS_andc_vec 0 +#define TCG_TARGET_HAS_orc_vec 0 +#define TCG_TARGET_HAS_nand_vec 0 +#define TCG_TARGET_HAS_nor_vec 0 +#define TCG_TARGET_HAS_eqv_vec 0 +#define TCG_TARGET_HAS_mul_vec 0 +#define TCG_TARGET_HAS_shi_vec 0 +#define TCG_TARGET_HAS_shs_vec 0 +#define TCG_TARGET_HAS_shv_vec 0 +#define TCG_TARGET_HAS_roti_vec 0 +#define TCG_TARGET_HAS_rots_vec 0 +#define TCG_TARGET_HAS_rotv_vec 0 +#define TCG_TARGET_HAS_sat_vec 0 +#define TCG_TARGET_HAS_minmax_vec 0 +#define TCG_TARGET_HAS_bitsel_vec 0 +#define TCG_TARGET_HAS_cmpsel_vec 0 + #define TCG_TARGET_DEFAULT_MO (0) #define TCG_TARGET_NEED_LDST_LABELS diff --git a/tcg/loongarch64/tcg-target.opc.h b/tcg/loongarch64/tcg-target.opc.h new file mode 100644 index XXXXXXX..XXXXXXX --- /dev/null +++ b/tcg/loongarch64/tcg-target.opc.h @@ -XXX,XX +XXX,XX @@ +/* + * Copyright (c) 2023 Jiajie Chen + * + * This work is licensed under the terms of the GNU GPL, version 2 or + * (at your option) any later version. + * + * See the COPYING file in the top-level directory for details. + * + * Target-specific opcodes for host vector expansion. These will be + * emitted by tcg_expand_vec_op. For those familiar with GCC internals, + * consider these to be UNSPEC with names. + */ diff --git a/tcg/loongarch64/tcg-target.c.inc b/tcg/loongarch64/tcg-target.c.inc index XXXXXXX..XXXXXXX 100644 --- a/tcg/loongarch64/tcg-target.c.inc +++ b/tcg/loongarch64/tcg-target.c.inc @@ -XXX,XX +XXX,XX @@ #include "../tcg-ldst.c.inc" #include <asm/hwcap.h> +bool use_lsx_instructions; + #ifdef CONFIG_DEBUG_TCG static const char * const tcg_target_reg_names[TCG_TARGET_NB_REGS] = { "zero", @@ -XXX,XX +XXX,XX @@ static const char * const tcg_target_reg_names[TCG_TARGET_NB_REGS] = { "s5", "s6", "s7", - "s8" + "s8", + "vr0", + "vr1", + "vr2", + "vr3", + "vr4", + "vr5", + "vr6", + "vr7", + "vr8", + "vr9", + "vr10", + "vr11", + "vr12", + "vr13", + "vr14", + "vr15", + "vr16", + "vr17", + "vr18", + "vr19", + "vr20", + "vr21", + "vr22", + "vr23", + "vr24", + "vr25", + "vr26", + "vr27", + "vr28", + "vr29", + "vr30", + "vr31", }; #endif @@ -XXX,XX +XXX,XX @@ static const int tcg_target_reg_alloc_order[] = { TCG_REG_A2, TCG_REG_A1, TCG_REG_A0, + + /* Vector registers */ + TCG_REG_V0, TCG_REG_V1, TCG_REG_V2, TCG_REG_V3, + TCG_REG_V4, TCG_REG_V5, TCG_REG_V6, TCG_REG_V7, + TCG_REG_V8, TCG_REG_V9, TCG_REG_V10, TCG_REG_V11, + TCG_REG_V12, TCG_REG_V13, TCG_REG_V14, TCG_REG_V15, + TCG_REG_V16, TCG_REG_V17, TCG_REG_V18, TCG_REG_V19, + TCG_REG_V20, TCG_REG_V21, TCG_REG_V22, TCG_REG_V23, + /* V24 - V31 are caller-saved, and skipped. */ }; static const int tcg_target_call_iarg_regs[] = { @@ -XXX,XX +XXX,XX @@ static TCGReg tcg_target_call_oarg_reg(TCGCallReturnKind kind, int slot) #define TCG_CT_CONST_WSZ 0x2000 #define ALL_GENERAL_REGS MAKE_64BIT_MASK(0, 32) +#define ALL_VECTOR_REGS MAKE_64BIT_MASK(32, 32) static inline tcg_target_long sextreg(tcg_target_long val, int pos, int len) { @@ -XXX,XX +XXX,XX @@ static void tcg_out_op(TCGContext *s, TCGOpcode opc, } } +static bool tcg_out_dup_vec(TCGContext *s, TCGType type, unsigned vece, + TCGReg rd, TCGReg rs) +{ + switch (vece) { + case MO_8: + tcg_out_opc_vreplgr2vr_b(s, rd, rs); + break; + case MO_16: + tcg_out_opc_vreplgr2vr_h(s, rd, rs); + break; + case MO_32: + tcg_out_opc_vreplgr2vr_w(s, rd, rs); + break; + case MO_64: + tcg_out_opc_vreplgr2vr_d(s, rd, rs); + break; + default: + g_assert_not_reached(); + } + return true; +} + +static bool tcg_out_dupm_vec(TCGContext *s, TCGType type, unsigned vece, + TCGReg r, TCGReg base, intptr_t offset) +{ + /* Handle imm overflow and division (vldrepl.d imm is divided by 8) */ + if (offset < -0x800 || offset > 0x7ff || \ + (offset & ((1 << vece) - 1)) != 0) { + tcg_out_addi(s, TCG_TYPE_I64, TCG_REG_TMP0, base, offset); + base = TCG_REG_TMP0; + offset = 0; + } + offset >>= vece; + + switch (vece) { + case MO_8: + tcg_out_opc_vldrepl_b(s, r, base, offset); + break; + case MO_16: + tcg_out_opc_vldrepl_h(s, r, base, offset); + break; + case MO_32: + tcg_out_opc_vldrepl_w(s, r, base, offset); + break; + case MO_64: + tcg_out_opc_vldrepl_d(s, r, base, offset); + break; + default: + g_assert_not_reached(); + } + return true; +} + +static void tcg_out_dupi_vec(TCGContext *s, TCGType type, unsigned vece, + TCGReg rd, int64_t v64) +{ + /* Try vldi if imm can fit */ + int64_t value = sextract64(v64, 0, 8 << vece); + if (-0x200 <= value && value <= 0x1FF) { + uint32_t imm = (vece << 10) | ((uint32_t)v64 & 0x3FF); + tcg_out_opc_vldi(s, rd, imm); + return; + } + + /* TODO: vldi patterns when imm 12 is set */ + + /* Fallback to vreplgr2vr */ + tcg_out_movi(s, TCG_TYPE_I64, TCG_REG_TMP0, value); + switch (vece) { + case MO_8: + tcg_out_opc_vreplgr2vr_b(s, rd, TCG_REG_TMP0); + break; + case MO_16: + tcg_out_opc_vreplgr2vr_h(s, rd, TCG_REG_TMP0); + break; + case MO_32: + tcg_out_opc_vreplgr2vr_w(s, rd, TCG_REG_TMP0); + break; + case MO_64: + tcg_out_opc_vreplgr2vr_d(s, rd, TCG_REG_TMP0); + break; + default: + g_assert_not_reached(); + } +} + +static void tcg_out_vec_op(TCGContext *s, TCGOpcode opc, + unsigned vecl, unsigned vece, + const TCGArg args[TCG_MAX_OP_ARGS], + const int const_args[TCG_MAX_OP_ARGS]) +{ + TCGType type = vecl + TCG_TYPE_V64; + TCGArg a0, a1, a2; + TCGReg temp = TCG_REG_TMP0; + + a0 = args[0]; + a1 = args[1]; + a2 = args[2]; + + /* Currently only supports V128 */ + tcg_debug_assert(type == TCG_TYPE_V128); + + switch (opc) { + case INDEX_op_st_vec: + /* Try to fit vst imm */ + if (-0x800 <= a2 && a2 <= 0x7ff) { + tcg_out_opc_vst(s, a0, a1, a2); + } else { + tcg_out_movi(s, TCG_TYPE_I64, temp, a2); + tcg_out_opc_vstx(s, a0, a1, temp); + } + break; + case INDEX_op_ld_vec: + /* Try to fit vld imm */ + if (-0x800 <= a2 && a2 <= 0x7ff) { + tcg_out_opc_vld(s, a0, a1, a2); + } else { + tcg_out_movi(s, TCG_TYPE_I64, temp, a2); + tcg_out_opc_vldx(s, a0, a1, temp); + } + break; + case INDEX_op_dupm_vec: + tcg_out_dupm_vec(s, type, vece, a0, a1, a2); + break; + default: + g_assert_not_reached(); + } +} + +int tcg_can_emit_vec_op(TCGOpcode opc, TCGType type, unsigned vece) +{ + switch (opc) { + case INDEX_op_ld_vec: + case INDEX_op_st_vec: + case INDEX_op_dup_vec: + case INDEX_op_dupm_vec: + return 1; + default: + return 0; + } +} + +void tcg_expand_vec_op(TCGOpcode opc, TCGType type, unsigned vece, + TCGArg a0, ...) +{ + g_assert_not_reached(); +} + static TCGConstraintSetIndex tcg_target_op_def(TCGOpcode op) { switch (op) { @@ -XXX,XX +XXX,XX @@ static TCGConstraintSetIndex tcg_target_op_def(TCGOpcode op) case INDEX_op_movcond_i64: return C_O1_I4(r, rZ, rJ, rZ, rZ); + case INDEX_op_ld_vec: + case INDEX_op_dupm_vec: + case INDEX_op_dup_vec: + return C_O1_I1(w, r); + + case INDEX_op_st_vec: + return C_O0_I2(w, r); + default: g_assert_not_reached(); } @@ -XXX,XX +XXX,XX @@ static void tcg_target_init(TCGContext *s) exit(EXIT_FAILURE); } + if (hwcap & HWCAP_LOONGARCH_LSX) { + use_lsx_instructions = 1; + } + tcg_target_available_regs[TCG_TYPE_I32] = ALL_GENERAL_REGS; tcg_target_available_regs[TCG_TYPE_I64] = ALL_GENERAL_REGS; @@ -XXX,XX +XXX,XX @@ static void tcg_target_init(TCGContext *s) tcg_regset_reset_reg(tcg_target_call_clobber_regs, TCG_REG_S8); tcg_regset_reset_reg(tcg_target_call_clobber_regs, TCG_REG_S9); + if (use_lsx_instructions) { + tcg_target_available_regs[TCG_TYPE_V128] = ALL_VECTOR_REGS; + tcg_regset_reset_reg(tcg_target_call_clobber_regs, TCG_REG_V24); + tcg_regset_reset_reg(tcg_target_call_clobber_regs, TCG_REG_V25); + tcg_regset_reset_reg(tcg_target_call_clobber_regs, TCG_REG_V26); + tcg_regset_reset_reg(tcg_target_call_clobber_regs, TCG_REG_V27); + tcg_regset_reset_reg(tcg_target_call_clobber_regs, TCG_REG_V28); + tcg_regset_reset_reg(tcg_target_call_clobber_regs, TCG_REG_V29); + tcg_regset_reset_reg(tcg_target_call_clobber_regs, TCG_REG_V30); + tcg_regset_reset_reg(tcg_target_call_clobber_regs, TCG_REG_V31); + } + s->reserved_regs = 0; tcg_regset_set_reg(s->reserved_regs, TCG_REG_ZERO); tcg_regset_set_reg(s->reserved_regs, TCG_REG_TMP0); @@ -XXX,XX +XXX,XX @@ static void tcg_target_init(TCGContext *s) tcg_regset_set_reg(s->reserved_regs, TCG_REG_SP); tcg_regset_set_reg(s->reserved_regs, TCG_REG_TP); tcg_regset_set_reg(s->reserved_regs, TCG_REG_RESERVED); + tcg_regset_set_reg(s->reserved_regs, TCG_VEC_TMP0); } typedef struct { -- 2.34.1 From: Jiajie Chen <c@jia.je> Pass vece to tcg_target_const_match() to allow correct interpretation of const args of vector ops. Signed-off-by: Jiajie Chen <c@jia.je> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Message-Id: <20230908022302.180442-4-c@jia.je> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- tcg/tcg.c | 4 ++-- tcg/aarch64/tcg-target.c.inc | 2 +- tcg/arm/tcg-target.c.inc | 2 +- tcg/i386/tcg-target.c.inc | 2 +- tcg/loongarch64/tcg-target.c.inc | 2 +- tcg/mips/tcg-target.c.inc | 2 +- tcg/ppc/tcg-target.c.inc | 2 +- tcg/riscv/tcg-target.c.inc | 2 +- tcg/s390x/tcg-target.c.inc | 2 +- tcg/sparc64/tcg-target.c.inc | 2 +- tcg/tci/tcg-target.c.inc | 2 +- 11 files changed, 12 insertions(+), 12 deletions(-) diff --git a/tcg/tcg.c b/tcg/tcg.c index XXXXXXX..XXXXXXX 100644 --- a/tcg/tcg.c +++ b/tcg/tcg.c @@ -XXX,XX +XXX,XX @@ static bool tcg_out_sti(TCGContext *s, TCGType type, TCGArg val, static void tcg_out_call(TCGContext *s, const tcg_insn_unit *target, const TCGHelperInfo *info); static TCGReg tcg_target_call_oarg_reg(TCGCallReturnKind kind, int slot); -static bool tcg_target_const_match(int64_t val, TCGType type, int ct); +static bool tcg_target_const_match(int64_t val, TCGType type, int ct, int vece); #ifdef TCG_TARGET_NEED_LDST_LABELS static int tcg_out_ldst_finalize(TCGContext *s); #endif @@ -XXX,XX +XXX,XX @@ static void tcg_reg_alloc_op(TCGContext *s, const TCGOp *op) ts = arg_temp(arg); if (ts->val_type == TEMP_VAL_CONST - && tcg_target_const_match(ts->val, ts->type, arg_ct->ct)) { + && tcg_target_const_match(ts->val, ts->type, arg_ct->ct, TCGOP_VECE(op))) { /* constant is OK for instruction */ const_args[i] = 1; new_args[i] = ts->val; diff --git a/tcg/aarch64/tcg-target.c.inc b/tcg/aarch64/tcg-target.c.inc index XXXXXXX..XXXXXXX 100644 --- a/tcg/aarch64/tcg-target.c.inc +++ b/tcg/aarch64/tcg-target.c.inc @@ -XXX,XX +XXX,XX @@ static bool is_shimm1632(uint32_t v32, int *cmode, int *imm8) } } -static bool tcg_target_const_match(int64_t val, TCGType type, int ct) +static bool tcg_target_const_match(int64_t val, TCGType type, int ct, int vece) { if (ct & TCG_CT_CONST) { return 1; diff --git a/tcg/arm/tcg-target.c.inc b/tcg/arm/tcg-target.c.inc index XXXXXXX..XXXXXXX 100644 --- a/tcg/arm/tcg-target.c.inc +++ b/tcg/arm/tcg-target.c.inc @@ -XXX,XX +XXX,XX @@ static bool is_shimm1632(uint32_t v32, int *cmode, int *imm8) * mov operand2: values represented with x << (2 * y), x < 0x100 * add, sub, eor...: ditto */ -static bool tcg_target_const_match(int64_t val, TCGType type, int ct) +static bool tcg_target_const_match(int64_t val, TCGType type, int ct, int vece) { if (ct & TCG_CT_CONST) { return 1; diff --git a/tcg/i386/tcg-target.c.inc b/tcg/i386/tcg-target.c.inc index XXXXXXX..XXXXXXX 100644 --- a/tcg/i386/tcg-target.c.inc +++ b/tcg/i386/tcg-target.c.inc @@ -XXX,XX +XXX,XX @@ static bool patch_reloc(tcg_insn_unit *code_ptr, int type, } /* test if a constant matches the constraint */ -static bool tcg_target_const_match(int64_t val, TCGType type, int ct) +static bool tcg_target_const_match(int64_t val, TCGType type, int ct, int vece) { if (ct & TCG_CT_CONST) { return 1; diff --git a/tcg/loongarch64/tcg-target.c.inc b/tcg/loongarch64/tcg-target.c.inc index XXXXXXX..XXXXXXX 100644 --- a/tcg/loongarch64/tcg-target.c.inc +++ b/tcg/loongarch64/tcg-target.c.inc @@ -XXX,XX +XXX,XX @@ static inline tcg_target_long sextreg(tcg_target_long val, int pos, int len) } /* test if a constant matches the constraint */ -static bool tcg_target_const_match(int64_t val, TCGType type, int ct) +static bool tcg_target_const_match(int64_t val, TCGType type, int ct, int vece) { if (ct & TCG_CT_CONST) { return true; diff --git a/tcg/mips/tcg-target.c.inc b/tcg/mips/tcg-target.c.inc index XXXXXXX..XXXXXXX 100644 --- a/tcg/mips/tcg-target.c.inc +++ b/tcg/mips/tcg-target.c.inc @@ -XXX,XX +XXX,XX @@ static bool is_p2m1(tcg_target_long val) } /* test if a constant matches the constraint */ -static bool tcg_target_const_match(int64_t val, TCGType type, int ct) +static bool tcg_target_const_match(int64_t val, TCGType type, int ct, int vece) { if (ct & TCG_CT_CONST) { return 1; diff --git a/tcg/ppc/tcg-target.c.inc b/tcg/ppc/tcg-target.c.inc index XXXXXXX..XXXXXXX 100644 --- a/tcg/ppc/tcg-target.c.inc +++ b/tcg/ppc/tcg-target.c.inc @@ -XXX,XX +XXX,XX @@ static bool reloc_pc14(tcg_insn_unit *src_rw, const tcg_insn_unit *target) } /* test if a constant matches the constraint */ -static bool tcg_target_const_match(int64_t val, TCGType type, int ct) +static bool tcg_target_const_match(int64_t val, TCGType type, int ct, int vece) { if (ct & TCG_CT_CONST) { return 1; diff --git a/tcg/riscv/tcg-target.c.inc b/tcg/riscv/tcg-target.c.inc index XXXXXXX..XXXXXXX 100644 --- a/tcg/riscv/tcg-target.c.inc +++ b/tcg/riscv/tcg-target.c.inc @@ -XXX,XX +XXX,XX @@ static TCGReg tcg_target_call_oarg_reg(TCGCallReturnKind kind, int slot) #define sextreg sextract64 /* test if a constant matches the constraint */ -static bool tcg_target_const_match(int64_t val, TCGType type, int ct) +static bool tcg_target_const_match(int64_t val, TCGType type, int ct, int vece) { if (ct & TCG_CT_CONST) { return 1; diff --git a/tcg/s390x/tcg-target.c.inc b/tcg/s390x/tcg-target.c.inc index XXXXXXX..XXXXXXX 100644 --- a/tcg/s390x/tcg-target.c.inc +++ b/tcg/s390x/tcg-target.c.inc @@ -XXX,XX +XXX,XX @@ static bool risbg_mask(uint64_t c) } /* Test if a constant matches the constraint. */ -static bool tcg_target_const_match(int64_t val, TCGType type, int ct) +static bool tcg_target_const_match(int64_t val, TCGType type, int ct, int vece) { if (ct & TCG_CT_CONST) { return 1; diff --git a/tcg/sparc64/tcg-target.c.inc b/tcg/sparc64/tcg-target.c.inc index XXXXXXX..XXXXXXX 100644 --- a/tcg/sparc64/tcg-target.c.inc +++ b/tcg/sparc64/tcg-target.c.inc @@ -XXX,XX +XXX,XX @@ static bool patch_reloc(tcg_insn_unit *src_rw, int type, } /* test if a constant matches the constraint */ -static bool tcg_target_const_match(int64_t val, TCGType type, int ct) +static bool tcg_target_const_match(int64_t val, TCGType type, int ct, int vece) { if (ct & TCG_CT_CONST) { return 1; diff --git a/tcg/tci/tcg-target.c.inc b/tcg/tci/tcg-target.c.inc index XXXXXXX..XXXXXXX 100644 --- a/tcg/tci/tcg-target.c.inc +++ b/tcg/tci/tcg-target.c.inc @@ -XXX,XX +XXX,XX @@ static inline bool tcg_out_sti(TCGContext *s, TCGType type, TCGArg val, } /* Test if a constant matches the constraint. */ -static bool tcg_target_const_match(int64_t val, TCGType type, int ct) +static bool tcg_target_const_match(int64_t val, TCGType type, int ct, int vece) { return ct & TCG_CT_CONST; } -- 2.34.1 From: Jiajie Chen <c@jia.je> Signed-off-by: Jiajie Chen <c@jia.je> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-Id: <20230908022302.180442-5-c@jia.je> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- tcg/loongarch64/tcg-target-con-set.h | 1 + tcg/loongarch64/tcg-target-con-str.h | 1 + tcg/loongarch64/tcg-target.c.inc | 65 ++++++++++++++++++++++++++++ 3 files changed, 67 insertions(+) diff --git a/tcg/loongarch64/tcg-target-con-set.h b/tcg/loongarch64/tcg-target-con-set.h index XXXXXXX..XXXXXXX 100644 --- a/tcg/loongarch64/tcg-target-con-set.h +++ b/tcg/loongarch64/tcg-target-con-set.h @@ -XXX,XX +XXX,XX @@ C_O1_I2(r, 0, rZ) C_O1_I2(r, rZ, ri) C_O1_I2(r, rZ, rJ) C_O1_I2(r, rZ, rZ) +C_O1_I2(w, w, wM) C_O1_I4(r, rZ, rJ, rZ, rZ) diff --git a/tcg/loongarch64/tcg-target-con-str.h b/tcg/loongarch64/tcg-target-con-str.h index XXXXXXX..XXXXXXX 100644 --- a/tcg/loongarch64/tcg-target-con-str.h +++ b/tcg/loongarch64/tcg-target-con-str.h @@ -XXX,XX +XXX,XX @@ CONST('U', TCG_CT_CONST_U12) CONST('Z', TCG_CT_CONST_ZERO) CONST('C', TCG_CT_CONST_C12) CONST('W', TCG_CT_CONST_WSZ) +CONST('M', TCG_CT_CONST_VCMP) diff --git a/tcg/loongarch64/tcg-target.c.inc b/tcg/loongarch64/tcg-target.c.inc index XXXXXXX..XXXXXXX 100644 --- a/tcg/loongarch64/tcg-target.c.inc +++ b/tcg/loongarch64/tcg-target.c.inc @@ -XXX,XX +XXX,XX @@ static TCGReg tcg_target_call_oarg_reg(TCGCallReturnKind kind, int slot) #define TCG_CT_CONST_U12 0x800 #define TCG_CT_CONST_C12 0x1000 #define TCG_CT_CONST_WSZ 0x2000 +#define TCG_CT_CONST_VCMP 0x4000 #define ALL_GENERAL_REGS MAKE_64BIT_MASK(0, 32) #define ALL_VECTOR_REGS MAKE_64BIT_MASK(32, 32) @@ -XXX,XX +XXX,XX @@ static bool tcg_target_const_match(int64_t val, TCGType type, int ct, int vece) if ((ct & TCG_CT_CONST_WSZ) && val == (type == TCG_TYPE_I32 ? 32 : 64)) { return true; } + int64_t vec_val = sextract64(val, 0, 8 << vece); + if ((ct & TCG_CT_CONST_VCMP) && -0x10 <= vec_val && vec_val <= 0x1f) { + return true; + } return false; } @@ -XXX,XX +XXX,XX @@ static void tcg_out_vec_op(TCGContext *s, TCGOpcode opc, TCGType type = vecl + TCG_TYPE_V64; TCGArg a0, a1, a2; TCGReg temp = TCG_REG_TMP0; + TCGReg temp_vec = TCG_VEC_TMP0; + + static const LoongArchInsn cmp_vec_insn[16][4] = { + [TCG_COND_EQ] = {OPC_VSEQ_B, OPC_VSEQ_H, OPC_VSEQ_W, OPC_VSEQ_D}, + [TCG_COND_LE] = {OPC_VSLE_B, OPC_VSLE_H, OPC_VSLE_W, OPC_VSLE_D}, + [TCG_COND_LEU] = {OPC_VSLE_BU, OPC_VSLE_HU, OPC_VSLE_WU, OPC_VSLE_DU}, + [TCG_COND_LT] = {OPC_VSLT_B, OPC_VSLT_H, OPC_VSLT_W, OPC_VSLT_D}, + [TCG_COND_LTU] = {OPC_VSLT_BU, OPC_VSLT_HU, OPC_VSLT_WU, OPC_VSLT_DU}, + }; + static const LoongArchInsn cmp_vec_imm_insn[16][4] = { + [TCG_COND_EQ] = {OPC_VSEQI_B, OPC_VSEQI_H, OPC_VSEQI_W, OPC_VSEQI_D}, + [TCG_COND_LE] = {OPC_VSLEI_B, OPC_VSLEI_H, OPC_VSLEI_W, OPC_VSLEI_D}, + [TCG_COND_LEU] = {OPC_VSLEI_BU, OPC_VSLEI_HU, OPC_VSLEI_WU, OPC_VSLEI_DU}, + [TCG_COND_LT] = {OPC_VSLTI_B, OPC_VSLTI_H, OPC_VSLTI_W, OPC_VSLTI_D}, + [TCG_COND_LTU] = {OPC_VSLTI_BU, OPC_VSLTI_HU, OPC_VSLTI_WU, OPC_VSLTI_DU}, + }; + LoongArchInsn insn; a0 = args[0]; a1 = args[1]; @@ -XXX,XX +XXX,XX @@ static void tcg_out_vec_op(TCGContext *s, TCGOpcode opc, tcg_out_opc_vldx(s, a0, a1, temp); } break; + case INDEX_op_cmp_vec: + TCGCond cond = args[3]; + if (const_args[2]) { + /* + * cmp_vec dest, src, value + * Try vseqi/vslei/vslti + */ + int64_t value = sextract64(a2, 0, 8 << vece); + if ((cond == TCG_COND_EQ || cond == TCG_COND_LE || \ + cond == TCG_COND_LT) && (-0x10 <= value && value <= 0x0f)) { + tcg_out32(s, encode_vdvjsk5_insn(cmp_vec_imm_insn[cond][vece], \ + a0, a1, value)); + break; + } else if ((cond == TCG_COND_LEU || cond == TCG_COND_LTU) && + (0x00 <= value && value <= 0x1f)) { + tcg_out32(s, encode_vdvjuk5_insn(cmp_vec_imm_insn[cond][vece], \ + a0, a1, value)); + break; + } + + /* + * Fallback to: + * dupi_vec temp, a2 + * cmp_vec a0, a1, temp, cond + */ + tcg_out_dupi_vec(s, type, vece, temp_vec, a2); + a2 = temp_vec; + } + + insn = cmp_vec_insn[cond][vece]; + if (insn == 0) { + TCGArg t; + t = a1, a1 = a2, a2 = t; + cond = tcg_swap_cond(cond); + insn = cmp_vec_insn[cond][vece]; + tcg_debug_assert(insn != 0); + } + tcg_out32(s, encode_vdvjvk_insn(insn, a0, a1, a2)); + break; case INDEX_op_dupm_vec: tcg_out_dupm_vec(s, type, vece, a0, a1, a2); break; @@ -XXX,XX +XXX,XX @@ int tcg_can_emit_vec_op(TCGOpcode opc, TCGType type, unsigned vece) case INDEX_op_st_vec: case INDEX_op_dup_vec: case INDEX_op_dupm_vec: + case INDEX_op_cmp_vec: return 1; default: return 0; @@ -XXX,XX +XXX,XX @@ static TCGConstraintSetIndex tcg_target_op_def(TCGOpcode op) case INDEX_op_st_vec: return C_O0_I2(w, r); + case INDEX_op_cmp_vec: + return C_O1_I2(w, w, wM); + default: g_assert_not_reached(); } -- 2.34.1 From: Jiajie Chen <c@jia.je> Lower the following ops: - add_vec - sub_vec Signed-off-by: Jiajie Chen <c@jia.je> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-Id: <20230908022302.180442-6-c@jia.je> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- tcg/loongarch64/tcg-target-con-set.h | 1 + tcg/loongarch64/tcg-target-con-str.h | 1 + tcg/loongarch64/tcg-target.c.inc | 61 ++++++++++++++++++++++++++++ 3 files changed, 63 insertions(+) diff --git a/tcg/loongarch64/tcg-target-con-set.h b/tcg/loongarch64/tcg-target-con-set.h index XXXXXXX..XXXXXXX 100644 --- a/tcg/loongarch64/tcg-target-con-set.h +++ b/tcg/loongarch64/tcg-target-con-set.h @@ -XXX,XX +XXX,XX @@ C_O1_I2(r, rZ, ri) C_O1_I2(r, rZ, rJ) C_O1_I2(r, rZ, rZ) C_O1_I2(w, w, wM) +C_O1_I2(w, w, wA) C_O1_I4(r, rZ, rJ, rZ, rZ) diff --git a/tcg/loongarch64/tcg-target-con-str.h b/tcg/loongarch64/tcg-target-con-str.h index XXXXXXX..XXXXXXX 100644 --- a/tcg/loongarch64/tcg-target-con-str.h +++ b/tcg/loongarch64/tcg-target-con-str.h @@ -XXX,XX +XXX,XX @@ CONST('Z', TCG_CT_CONST_ZERO) CONST('C', TCG_CT_CONST_C12) CONST('W', TCG_CT_CONST_WSZ) CONST('M', TCG_CT_CONST_VCMP) +CONST('A', TCG_CT_CONST_VADD) diff --git a/tcg/loongarch64/tcg-target.c.inc b/tcg/loongarch64/tcg-target.c.inc index XXXXXXX..XXXXXXX 100644 --- a/tcg/loongarch64/tcg-target.c.inc +++ b/tcg/loongarch64/tcg-target.c.inc @@ -XXX,XX +XXX,XX @@ static TCGReg tcg_target_call_oarg_reg(TCGCallReturnKind kind, int slot) #define TCG_CT_CONST_C12 0x1000 #define TCG_CT_CONST_WSZ 0x2000 #define TCG_CT_CONST_VCMP 0x4000 +#define TCG_CT_CONST_VADD 0x8000 #define ALL_GENERAL_REGS MAKE_64BIT_MASK(0, 32) #define ALL_VECTOR_REGS MAKE_64BIT_MASK(32, 32) @@ -XXX,XX +XXX,XX @@ static bool tcg_target_const_match(int64_t val, TCGType type, int ct, int vece) if ((ct & TCG_CT_CONST_VCMP) && -0x10 <= vec_val && vec_val <= 0x1f) { return true; } + if ((ct & TCG_CT_CONST_VADD) && -0x1f <= vec_val && vec_val <= 0x1f) { + return true; + } return false; } @@ -XXX,XX +XXX,XX @@ static void tcg_out_dupi_vec(TCGContext *s, TCGType type, unsigned vece, } } +static void tcg_out_addsub_vec(TCGContext *s, unsigned vece, const TCGArg a0, + const TCGArg a1, const TCGArg a2, + bool a2_is_const, bool is_add) +{ + static const LoongArchInsn add_vec_insn[4] = { + OPC_VADD_B, OPC_VADD_H, OPC_VADD_W, OPC_VADD_D + }; + static const LoongArchInsn add_vec_imm_insn[4] = { + OPC_VADDI_BU, OPC_VADDI_HU, OPC_VADDI_WU, OPC_VADDI_DU + }; + static const LoongArchInsn sub_vec_insn[4] = { + OPC_VSUB_B, OPC_VSUB_H, OPC_VSUB_W, OPC_VSUB_D + }; + static const LoongArchInsn sub_vec_imm_insn[4] = { + OPC_VSUBI_BU, OPC_VSUBI_HU, OPC_VSUBI_WU, OPC_VSUBI_DU + }; + + if (a2_is_const) { + int64_t value = sextract64(a2, 0, 8 << vece); + if (!is_add) { + value = -value; + } + + /* Try vaddi/vsubi */ + if (0 <= value && value <= 0x1f) { + tcg_out32(s, encode_vdvjuk5_insn(add_vec_imm_insn[vece], a0, \ + a1, value)); + return; + } else if (-0x1f <= value && value < 0) { + tcg_out32(s, encode_vdvjuk5_insn(sub_vec_imm_insn[vece], a0, \ + a1, -value)); + return; + } + + /* constraint TCG_CT_CONST_VADD ensures unreachable */ + g_assert_not_reached(); + } + + if (is_add) { + tcg_out32(s, encode_vdvjvk_insn(add_vec_insn[vece], a0, a1, a2)); + } else { + tcg_out32(s, encode_vdvjvk_insn(sub_vec_insn[vece], a0, a1, a2)); + } +} + static void tcg_out_vec_op(TCGContext *s, TCGOpcode opc, unsigned vecl, unsigned vece, const TCGArg args[TCG_MAX_OP_ARGS], @@ -XXX,XX +XXX,XX @@ static void tcg_out_vec_op(TCGContext *s, TCGOpcode opc, } tcg_out32(s, encode_vdvjvk_insn(insn, a0, a1, a2)); break; + case INDEX_op_add_vec: + tcg_out_addsub_vec(s, vece, a0, a1, a2, const_args[2], true); + break; + case INDEX_op_sub_vec: + tcg_out_addsub_vec(s, vece, a0, a1, a2, const_args[2], false); + break; case INDEX_op_dupm_vec: tcg_out_dupm_vec(s, type, vece, a0, a1, a2); break; @@ -XXX,XX +XXX,XX @@ int tcg_can_emit_vec_op(TCGOpcode opc, TCGType type, unsigned vece) case INDEX_op_dup_vec: case INDEX_op_dupm_vec: case INDEX_op_cmp_vec: + case INDEX_op_add_vec: + case INDEX_op_sub_vec: return 1; default: return 0; @@ -XXX,XX +XXX,XX @@ static TCGConstraintSetIndex tcg_target_op_def(TCGOpcode op) case INDEX_op_cmp_vec: return C_O1_I2(w, w, wM); + case INDEX_op_add_vec: + case INDEX_op_sub_vec: + return C_O1_I2(w, w, wA); + default: g_assert_not_reached(); } -- 2.34.1 From: Jiajie Chen <c@jia.je> Lower the following ops: - and_vec - andc_vec - or_vec - orc_vec - xor_vec - nor_vec - not_vec Signed-off-by: Jiajie Chen <c@jia.je> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-Id: <20230908022302.180442-7-c@jia.je> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- tcg/loongarch64/tcg-target-con-set.h | 2 ++ tcg/loongarch64/tcg-target.h | 8 ++--- tcg/loongarch64/tcg-target.c.inc | 44 ++++++++++++++++++++++++++++ 3 files changed, 50 insertions(+), 4 deletions(-) diff --git a/tcg/loongarch64/tcg-target-con-set.h b/tcg/loongarch64/tcg-target-con-set.h index XXXXXXX..XXXXXXX 100644 --- a/tcg/loongarch64/tcg-target-con-set.h +++ b/tcg/loongarch64/tcg-target-con-set.h @@ -XXX,XX +XXX,XX @@ C_O0_I2(rZ, rZ) C_O0_I2(w, r) C_O1_I1(r, r) C_O1_I1(w, r) +C_O1_I1(w, w) C_O1_I2(r, r, rC) C_O1_I2(r, r, ri) C_O1_I2(r, r, rI) @@ -XXX,XX +XXX,XX @@ C_O1_I2(r, 0, rZ) C_O1_I2(r, rZ, ri) C_O1_I2(r, rZ, rJ) C_O1_I2(r, rZ, rZ) +C_O1_I2(w, w, w) C_O1_I2(w, w, wM) C_O1_I2(w, w, wA) C_O1_I4(r, rZ, rJ, rZ, rZ) diff --git a/tcg/loongarch64/tcg-target.h b/tcg/loongarch64/tcg-target.h index XXXXXXX..XXXXXXX 100644 --- a/tcg/loongarch64/tcg-target.h +++ b/tcg/loongarch64/tcg-target.h @@ -XXX,XX +XXX,XX @@ extern bool use_lsx_instructions; #define TCG_TARGET_HAS_v128 use_lsx_instructions #define TCG_TARGET_HAS_v256 0 -#define TCG_TARGET_HAS_not_vec 0 +#define TCG_TARGET_HAS_not_vec 1 #define TCG_TARGET_HAS_neg_vec 0 #define TCG_TARGET_HAS_abs_vec 0 -#define TCG_TARGET_HAS_andc_vec 0 -#define TCG_TARGET_HAS_orc_vec 0 +#define TCG_TARGET_HAS_andc_vec 1 +#define TCG_TARGET_HAS_orc_vec 1 #define TCG_TARGET_HAS_nand_vec 0 -#define TCG_TARGET_HAS_nor_vec 0 +#define TCG_TARGET_HAS_nor_vec 1 #define TCG_TARGET_HAS_eqv_vec 0 #define TCG_TARGET_HAS_mul_vec 0 #define TCG_TARGET_HAS_shi_vec 0 diff --git a/tcg/loongarch64/tcg-target.c.inc b/tcg/loongarch64/tcg-target.c.inc index XXXXXXX..XXXXXXX 100644 --- a/tcg/loongarch64/tcg-target.c.inc +++ b/tcg/loongarch64/tcg-target.c.inc @@ -XXX,XX +XXX,XX @@ static void tcg_out_vec_op(TCGContext *s, TCGOpcode opc, tcg_out_opc_vldx(s, a0, a1, temp); } break; + case INDEX_op_and_vec: + tcg_out_opc_vand_v(s, a0, a1, a2); + break; + case INDEX_op_andc_vec: + /* + * vandn vd, vj, vk: vd = vk & ~vj + * andc_vec vd, vj, vk: vd = vj & ~vk + * vk and vk are swapped + */ + tcg_out_opc_vandn_v(s, a0, a2, a1); + break; + case INDEX_op_or_vec: + tcg_out_opc_vor_v(s, a0, a1, a2); + break; + case INDEX_op_orc_vec: + tcg_out_opc_vorn_v(s, a0, a1, a2); + break; + case INDEX_op_xor_vec: + tcg_out_opc_vxor_v(s, a0, a1, a2); + break; + case INDEX_op_nor_vec: + tcg_out_opc_vnor_v(s, a0, a1, a2); + break; + case INDEX_op_not_vec: + tcg_out_opc_vnor_v(s, a0, a1, a1); + break; case INDEX_op_cmp_vec: TCGCond cond = args[3]; if (const_args[2]) { @@ -XXX,XX +XXX,XX @@ int tcg_can_emit_vec_op(TCGOpcode opc, TCGType type, unsigned vece) case INDEX_op_cmp_vec: case INDEX_op_add_vec: case INDEX_op_sub_vec: + case INDEX_op_and_vec: + case INDEX_op_andc_vec: + case INDEX_op_or_vec: + case INDEX_op_orc_vec: + case INDEX_op_xor_vec: + case INDEX_op_nor_vec: + case INDEX_op_not_vec: return 1; default: return 0; @@ -XXX,XX +XXX,XX @@ static TCGConstraintSetIndex tcg_target_op_def(TCGOpcode op) case INDEX_op_sub_vec: return C_O1_I2(w, w, wA); + case INDEX_op_and_vec: + case INDEX_op_andc_vec: + case INDEX_op_or_vec: + case INDEX_op_orc_vec: + case INDEX_op_xor_vec: + case INDEX_op_nor_vec: + return C_O1_I2(w, w, w); + + case INDEX_op_not_vec: + return C_O1_I1(w, w); + default: g_assert_not_reached(); } -- 2.34.1 From: Jiajie Chen <c@jia.je> Signed-off-by: Jiajie Chen <c@jia.je> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-Id: <20230908022302.180442-8-c@jia.je> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- tcg/loongarch64/tcg-target.h | 2 +- tcg/loongarch64/tcg-target.c.inc | 8 ++++++++ 2 files changed, 9 insertions(+), 1 deletion(-) diff --git a/tcg/loongarch64/tcg-target.h b/tcg/loongarch64/tcg-target.h index XXXXXXX..XXXXXXX 100644 --- a/tcg/loongarch64/tcg-target.h +++ b/tcg/loongarch64/tcg-target.h @@ -XXX,XX +XXX,XX @@ extern bool use_lsx_instructions; #define TCG_TARGET_HAS_v256 0 #define TCG_TARGET_HAS_not_vec 1 -#define TCG_TARGET_HAS_neg_vec 0 +#define TCG_TARGET_HAS_neg_vec 1 #define TCG_TARGET_HAS_abs_vec 0 #define TCG_TARGET_HAS_andc_vec 1 #define TCG_TARGET_HAS_orc_vec 1 diff --git a/tcg/loongarch64/tcg-target.c.inc b/tcg/loongarch64/tcg-target.c.inc index XXXXXXX..XXXXXXX 100644 --- a/tcg/loongarch64/tcg-target.c.inc +++ b/tcg/loongarch64/tcg-target.c.inc @@ -XXX,XX +XXX,XX @@ static void tcg_out_vec_op(TCGContext *s, TCGOpcode opc, [TCG_COND_LTU] = {OPC_VSLTI_BU, OPC_VSLTI_HU, OPC_VSLTI_WU, OPC_VSLTI_DU}, }; LoongArchInsn insn; + static const LoongArchInsn neg_vec_insn[4] = { + OPC_VNEG_B, OPC_VNEG_H, OPC_VNEG_W, OPC_VNEG_D + }; a0 = args[0]; a1 = args[1]; @@ -XXX,XX +XXX,XX @@ static void tcg_out_vec_op(TCGContext *s, TCGOpcode opc, case INDEX_op_sub_vec: tcg_out_addsub_vec(s, vece, a0, a1, a2, const_args[2], false); break; + case INDEX_op_neg_vec: + tcg_out32(s, encode_vdvj_insn(neg_vec_insn[vece], a0, a1)); + break; case INDEX_op_dupm_vec: tcg_out_dupm_vec(s, type, vece, a0, a1, a2); break; @@ -XXX,XX +XXX,XX @@ int tcg_can_emit_vec_op(TCGOpcode opc, TCGType type, unsigned vece) case INDEX_op_xor_vec: case INDEX_op_nor_vec: case INDEX_op_not_vec: + case INDEX_op_neg_vec: return 1; default: return 0; @@ -XXX,XX +XXX,XX @@ static TCGConstraintSetIndex tcg_target_op_def(TCGOpcode op) return C_O1_I2(w, w, w); case INDEX_op_not_vec: + case INDEX_op_neg_vec: return C_O1_I1(w, w); default: -- 2.34.1 From: Jiajie Chen <c@jia.je> Signed-off-by: Jiajie Chen <c@jia.je> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-Id: <20230908022302.180442-9-c@jia.je> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- tcg/loongarch64/tcg-target.h | 2 +- tcg/loongarch64/tcg-target.c.inc | 8 ++++++++ 2 files changed, 9 insertions(+), 1 deletion(-) diff --git a/tcg/loongarch64/tcg-target.h b/tcg/loongarch64/tcg-target.h index XXXXXXX..XXXXXXX 100644 --- a/tcg/loongarch64/tcg-target.h +++ b/tcg/loongarch64/tcg-target.h @@ -XXX,XX +XXX,XX @@ extern bool use_lsx_instructions; #define TCG_TARGET_HAS_nand_vec 0 #define TCG_TARGET_HAS_nor_vec 1 #define TCG_TARGET_HAS_eqv_vec 0 -#define TCG_TARGET_HAS_mul_vec 0 +#define TCG_TARGET_HAS_mul_vec 1 #define TCG_TARGET_HAS_shi_vec 0 #define TCG_TARGET_HAS_shs_vec 0 #define TCG_TARGET_HAS_shv_vec 0 diff --git a/tcg/loongarch64/tcg-target.c.inc b/tcg/loongarch64/tcg-target.c.inc index XXXXXXX..XXXXXXX 100644 --- a/tcg/loongarch64/tcg-target.c.inc +++ b/tcg/loongarch64/tcg-target.c.inc @@ -XXX,XX +XXX,XX @@ static void tcg_out_vec_op(TCGContext *s, TCGOpcode opc, static const LoongArchInsn neg_vec_insn[4] = { OPC_VNEG_B, OPC_VNEG_H, OPC_VNEG_W, OPC_VNEG_D }; + static const LoongArchInsn mul_vec_insn[4] = { + OPC_VMUL_B, OPC_VMUL_H, OPC_VMUL_W, OPC_VMUL_D + }; a0 = args[0]; a1 = args[1]; @@ -XXX,XX +XXX,XX @@ static void tcg_out_vec_op(TCGContext *s, TCGOpcode opc, case INDEX_op_neg_vec: tcg_out32(s, encode_vdvj_insn(neg_vec_insn[vece], a0, a1)); break; + case INDEX_op_mul_vec: + tcg_out32(s, encode_vdvjvk_insn(mul_vec_insn[vece], a0, a1, a2)); + break; case INDEX_op_dupm_vec: tcg_out_dupm_vec(s, type, vece, a0, a1, a2); break; @@ -XXX,XX +XXX,XX @@ int tcg_can_emit_vec_op(TCGOpcode opc, TCGType type, unsigned vece) case INDEX_op_nor_vec: case INDEX_op_not_vec: case INDEX_op_neg_vec: + case INDEX_op_mul_vec: return 1; default: return 0; @@ -XXX,XX +XXX,XX @@ static TCGConstraintSetIndex tcg_target_op_def(TCGOpcode op) case INDEX_op_orc_vec: case INDEX_op_xor_vec: case INDEX_op_nor_vec: + case INDEX_op_mul_vec: return C_O1_I2(w, w, w); case INDEX_op_not_vec: -- 2.34.1 From: Jiajie Chen <c@jia.je> Lower the following ops: - smin_vec - smax_vec - umin_vec - umax_vec Signed-off-by: Jiajie Chen <c@jia.je> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-Id: <20230908022302.180442-10-c@jia.je> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- tcg/loongarch64/tcg-target.h | 2 +- tcg/loongarch64/tcg-target.c.inc | 32 ++++++++++++++++++++++++++++++++ 2 files changed, 33 insertions(+), 1 deletion(-) diff --git a/tcg/loongarch64/tcg-target.h b/tcg/loongarch64/tcg-target.h index XXXXXXX..XXXXXXX 100644 --- a/tcg/loongarch64/tcg-target.h +++ b/tcg/loongarch64/tcg-target.h @@ -XXX,XX +XXX,XX @@ extern bool use_lsx_instructions; #define TCG_TARGET_HAS_rots_vec 0 #define TCG_TARGET_HAS_rotv_vec 0 #define TCG_TARGET_HAS_sat_vec 0 -#define TCG_TARGET_HAS_minmax_vec 0 +#define TCG_TARGET_HAS_minmax_vec 1 #define TCG_TARGET_HAS_bitsel_vec 0 #define TCG_TARGET_HAS_cmpsel_vec 0 diff --git a/tcg/loongarch64/tcg-target.c.inc b/tcg/loongarch64/tcg-target.c.inc index XXXXXXX..XXXXXXX 100644 --- a/tcg/loongarch64/tcg-target.c.inc +++ b/tcg/loongarch64/tcg-target.c.inc @@ -XXX,XX +XXX,XX @@ static void tcg_out_vec_op(TCGContext *s, TCGOpcode opc, static const LoongArchInsn mul_vec_insn[4] = { OPC_VMUL_B, OPC_VMUL_H, OPC_VMUL_W, OPC_VMUL_D }; + static const LoongArchInsn smin_vec_insn[4] = { + OPC_VMIN_B, OPC_VMIN_H, OPC_VMIN_W, OPC_VMIN_D + }; + static const LoongArchInsn umin_vec_insn[4] = { + OPC_VMIN_BU, OPC_VMIN_HU, OPC_VMIN_WU, OPC_VMIN_DU + }; + static const LoongArchInsn smax_vec_insn[4] = { + OPC_VMAX_B, OPC_VMAX_H, OPC_VMAX_W, OPC_VMAX_D + }; + static const LoongArchInsn umax_vec_insn[4] = { + OPC_VMAX_BU, OPC_VMAX_HU, OPC_VMAX_WU, OPC_VMAX_DU + }; a0 = args[0]; a1 = args[1]; @@ -XXX,XX +XXX,XX @@ static void tcg_out_vec_op(TCGContext *s, TCGOpcode opc, case INDEX_op_mul_vec: tcg_out32(s, encode_vdvjvk_insn(mul_vec_insn[vece], a0, a1, a2)); break; + case INDEX_op_smin_vec: + tcg_out32(s, encode_vdvjvk_insn(smin_vec_insn[vece], a0, a1, a2)); + break; + case INDEX_op_smax_vec: + tcg_out32(s, encode_vdvjvk_insn(smax_vec_insn[vece], a0, a1, a2)); + break; + case INDEX_op_umin_vec: + tcg_out32(s, encode_vdvjvk_insn(umin_vec_insn[vece], a0, a1, a2)); + break; + case INDEX_op_umax_vec: + tcg_out32(s, encode_vdvjvk_insn(umax_vec_insn[vece], a0, a1, a2)); + break; case INDEX_op_dupm_vec: tcg_out_dupm_vec(s, type, vece, a0, a1, a2); break; @@ -XXX,XX +XXX,XX @@ int tcg_can_emit_vec_op(TCGOpcode opc, TCGType type, unsigned vece) case INDEX_op_not_vec: case INDEX_op_neg_vec: case INDEX_op_mul_vec: + case INDEX_op_smin_vec: + case INDEX_op_smax_vec: + case INDEX_op_umin_vec: + case INDEX_op_umax_vec: return 1; default: return 0; @@ -XXX,XX +XXX,XX @@ static TCGConstraintSetIndex tcg_target_op_def(TCGOpcode op) case INDEX_op_xor_vec: case INDEX_op_nor_vec: case INDEX_op_mul_vec: + case INDEX_op_smin_vec: + case INDEX_op_smax_vec: + case INDEX_op_umin_vec: + case INDEX_op_umax_vec: return C_O1_I2(w, w, w); case INDEX_op_not_vec: -- 2.34.1 From: Jiajie Chen <c@jia.je> Lower the following ops: - ssadd_vec - usadd_vec - sssub_vec - ussub_vec Signed-off-by: Jiajie Chen <c@jia.je> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-Id: <20230908022302.180442-11-c@jia.je> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- tcg/loongarch64/tcg-target.h | 2 +- tcg/loongarch64/tcg-target.c.inc | 32 ++++++++++++++++++++++++++++++++ 2 files changed, 33 insertions(+), 1 deletion(-) diff --git a/tcg/loongarch64/tcg-target.h b/tcg/loongarch64/tcg-target.h index XXXXXXX..XXXXXXX 100644 --- a/tcg/loongarch64/tcg-target.h +++ b/tcg/loongarch64/tcg-target.h @@ -XXX,XX +XXX,XX @@ extern bool use_lsx_instructions; #define TCG_TARGET_HAS_roti_vec 0 #define TCG_TARGET_HAS_rots_vec 0 #define TCG_TARGET_HAS_rotv_vec 0 -#define TCG_TARGET_HAS_sat_vec 0 +#define TCG_TARGET_HAS_sat_vec 1 #define TCG_TARGET_HAS_minmax_vec 1 #define TCG_TARGET_HAS_bitsel_vec 0 #define TCG_TARGET_HAS_cmpsel_vec 0 diff --git a/tcg/loongarch64/tcg-target.c.inc b/tcg/loongarch64/tcg-target.c.inc index XXXXXXX..XXXXXXX 100644 --- a/tcg/loongarch64/tcg-target.c.inc +++ b/tcg/loongarch64/tcg-target.c.inc @@ -XXX,XX +XXX,XX @@ static void tcg_out_vec_op(TCGContext *s, TCGOpcode opc, static const LoongArchInsn umax_vec_insn[4] = { OPC_VMAX_BU, OPC_VMAX_HU, OPC_VMAX_WU, OPC_VMAX_DU }; + static const LoongArchInsn ssadd_vec_insn[4] = { + OPC_VSADD_B, OPC_VSADD_H, OPC_VSADD_W, OPC_VSADD_D + }; + static const LoongArchInsn usadd_vec_insn[4] = { + OPC_VSADD_BU, OPC_VSADD_HU, OPC_VSADD_WU, OPC_VSADD_DU + }; + static const LoongArchInsn sssub_vec_insn[4] = { + OPC_VSSUB_B, OPC_VSSUB_H, OPC_VSSUB_W, OPC_VSSUB_D + }; + static const LoongArchInsn ussub_vec_insn[4] = { + OPC_VSSUB_BU, OPC_VSSUB_HU, OPC_VSSUB_WU, OPC_VSSUB_DU + }; a0 = args[0]; a1 = args[1]; @@ -XXX,XX +XXX,XX @@ static void tcg_out_vec_op(TCGContext *s, TCGOpcode opc, case INDEX_op_umax_vec: tcg_out32(s, encode_vdvjvk_insn(umax_vec_insn[vece], a0, a1, a2)); break; + case INDEX_op_ssadd_vec: + tcg_out32(s, encode_vdvjvk_insn(ssadd_vec_insn[vece], a0, a1, a2)); + break; + case INDEX_op_usadd_vec: + tcg_out32(s, encode_vdvjvk_insn(usadd_vec_insn[vece], a0, a1, a2)); + break; + case INDEX_op_sssub_vec: + tcg_out32(s, encode_vdvjvk_insn(sssub_vec_insn[vece], a0, a1, a2)); + break; + case INDEX_op_ussub_vec: + tcg_out32(s, encode_vdvjvk_insn(ussub_vec_insn[vece], a0, a1, a2)); + break; case INDEX_op_dupm_vec: tcg_out_dupm_vec(s, type, vece, a0, a1, a2); break; @@ -XXX,XX +XXX,XX @@ int tcg_can_emit_vec_op(TCGOpcode opc, TCGType type, unsigned vece) case INDEX_op_smax_vec: case INDEX_op_umin_vec: case INDEX_op_umax_vec: + case INDEX_op_ssadd_vec: + case INDEX_op_usadd_vec: + case INDEX_op_sssub_vec: + case INDEX_op_ussub_vec: return 1; default: return 0; @@ -XXX,XX +XXX,XX @@ static TCGConstraintSetIndex tcg_target_op_def(TCGOpcode op) case INDEX_op_smax_vec: case INDEX_op_umin_vec: case INDEX_op_umax_vec: + case INDEX_op_ssadd_vec: + case INDEX_op_usadd_vec: + case INDEX_op_sssub_vec: + case INDEX_op_ussub_vec: return C_O1_I2(w, w, w); case INDEX_op_not_vec: -- 2.34.1 From: Jiajie Chen <c@jia.je> Lower the following ops: - shlv_vec - shrv_vec - sarv_vec Signed-off-by: Jiajie Chen <c@jia.je> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-Id: <20230908022302.180442-12-c@jia.je> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- tcg/loongarch64/tcg-target.h | 2 +- tcg/loongarch64/tcg-target.c.inc | 24 ++++++++++++++++++++++++ 2 files changed, 25 insertions(+), 1 deletion(-) diff --git a/tcg/loongarch64/tcg-target.h b/tcg/loongarch64/tcg-target.h index XXXXXXX..XXXXXXX 100644 --- a/tcg/loongarch64/tcg-target.h +++ b/tcg/loongarch64/tcg-target.h @@ -XXX,XX +XXX,XX @@ extern bool use_lsx_instructions; #define TCG_TARGET_HAS_mul_vec 1 #define TCG_TARGET_HAS_shi_vec 0 #define TCG_TARGET_HAS_shs_vec 0 -#define TCG_TARGET_HAS_shv_vec 0 +#define TCG_TARGET_HAS_shv_vec 1 #define TCG_TARGET_HAS_roti_vec 0 #define TCG_TARGET_HAS_rots_vec 0 #define TCG_TARGET_HAS_rotv_vec 0 diff --git a/tcg/loongarch64/tcg-target.c.inc b/tcg/loongarch64/tcg-target.c.inc index XXXXXXX..XXXXXXX 100644 --- a/tcg/loongarch64/tcg-target.c.inc +++ b/tcg/loongarch64/tcg-target.c.inc @@ -XXX,XX +XXX,XX @@ static void tcg_out_vec_op(TCGContext *s, TCGOpcode opc, static const LoongArchInsn ussub_vec_insn[4] = { OPC_VSSUB_BU, OPC_VSSUB_HU, OPC_VSSUB_WU, OPC_VSSUB_DU }; + static const LoongArchInsn shlv_vec_insn[4] = { + OPC_VSLL_B, OPC_VSLL_H, OPC_VSLL_W, OPC_VSLL_D + }; + static const LoongArchInsn shrv_vec_insn[4] = { + OPC_VSRL_B, OPC_VSRL_H, OPC_VSRL_W, OPC_VSRL_D + }; + static const LoongArchInsn sarv_vec_insn[4] = { + OPC_VSRA_B, OPC_VSRA_H, OPC_VSRA_W, OPC_VSRA_D + }; a0 = args[0]; a1 = args[1]; @@ -XXX,XX +XXX,XX @@ static void tcg_out_vec_op(TCGContext *s, TCGOpcode opc, case INDEX_op_ussub_vec: tcg_out32(s, encode_vdvjvk_insn(ussub_vec_insn[vece], a0, a1, a2)); break; + case INDEX_op_shlv_vec: + tcg_out32(s, encode_vdvjvk_insn(shlv_vec_insn[vece], a0, a1, a2)); + break; + case INDEX_op_shrv_vec: + tcg_out32(s, encode_vdvjvk_insn(shrv_vec_insn[vece], a0, a1, a2)); + break; + case INDEX_op_sarv_vec: + tcg_out32(s, encode_vdvjvk_insn(sarv_vec_insn[vece], a0, a1, a2)); + break; case INDEX_op_dupm_vec: tcg_out_dupm_vec(s, type, vece, a0, a1, a2); break; @@ -XXX,XX +XXX,XX @@ int tcg_can_emit_vec_op(TCGOpcode opc, TCGType type, unsigned vece) case INDEX_op_usadd_vec: case INDEX_op_sssub_vec: case INDEX_op_ussub_vec: + case INDEX_op_shlv_vec: + case INDEX_op_shrv_vec: + case INDEX_op_sarv_vec: return 1; default: return 0; @@ -XXX,XX +XXX,XX @@ static TCGConstraintSetIndex tcg_target_op_def(TCGOpcode op) case INDEX_op_usadd_vec: case INDEX_op_sssub_vec: case INDEX_op_ussub_vec: + case INDEX_op_shlv_vec: + case INDEX_op_shrv_vec: + case INDEX_op_sarv_vec: return C_O1_I2(w, w, w); case INDEX_op_not_vec: -- 2.34.1 From: Jiajie Chen <c@jia.je> Signed-off-by: Jiajie Chen <c@jia.je> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-Id: <20230908022302.180442-13-c@jia.je> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- tcg/loongarch64/tcg-target-con-set.h | 1 + tcg/loongarch64/tcg-target.h | 2 +- tcg/loongarch64/tcg-target.c.inc | 11 ++++++++++- 3 files changed, 12 insertions(+), 2 deletions(-) diff --git a/tcg/loongarch64/tcg-target-con-set.h b/tcg/loongarch64/tcg-target-con-set.h index XXXXXXX..XXXXXXX 100644 --- a/tcg/loongarch64/tcg-target-con-set.h +++ b/tcg/loongarch64/tcg-target-con-set.h @@ -XXX,XX +XXX,XX @@ C_O1_I2(r, rZ, rZ) C_O1_I2(w, w, w) C_O1_I2(w, w, wM) C_O1_I2(w, w, wA) +C_O1_I3(w, w, w, w) C_O1_I4(r, rZ, rJ, rZ, rZ) diff --git a/tcg/loongarch64/tcg-target.h b/tcg/loongarch64/tcg-target.h index XXXXXXX..XXXXXXX 100644 --- a/tcg/loongarch64/tcg-target.h +++ b/tcg/loongarch64/tcg-target.h @@ -XXX,XX +XXX,XX @@ extern bool use_lsx_instructions; #define TCG_TARGET_HAS_rotv_vec 0 #define TCG_TARGET_HAS_sat_vec 1 #define TCG_TARGET_HAS_minmax_vec 1 -#define TCG_TARGET_HAS_bitsel_vec 0 +#define TCG_TARGET_HAS_bitsel_vec 1 #define TCG_TARGET_HAS_cmpsel_vec 0 #define TCG_TARGET_DEFAULT_MO (0) diff --git a/tcg/loongarch64/tcg-target.c.inc b/tcg/loongarch64/tcg-target.c.inc index XXXXXXX..XXXXXXX 100644 --- a/tcg/loongarch64/tcg-target.c.inc +++ b/tcg/loongarch64/tcg-target.c.inc @@ -XXX,XX +XXX,XX @@ static void tcg_out_vec_op(TCGContext *s, TCGOpcode opc, const int const_args[TCG_MAX_OP_ARGS]) { TCGType type = vecl + TCG_TYPE_V64; - TCGArg a0, a1, a2; + TCGArg a0, a1, a2, a3; TCGReg temp = TCG_REG_TMP0; TCGReg temp_vec = TCG_VEC_TMP0; @@ -XXX,XX +XXX,XX @@ static void tcg_out_vec_op(TCGContext *s, TCGOpcode opc, a0 = args[0]; a1 = args[1]; a2 = args[2]; + a3 = args[3]; /* Currently only supports V128 */ tcg_debug_assert(type == TCG_TYPE_V128); @@ -XXX,XX +XXX,XX @@ static void tcg_out_vec_op(TCGContext *s, TCGOpcode opc, case INDEX_op_sarv_vec: tcg_out32(s, encode_vdvjvk_insn(sarv_vec_insn[vece], a0, a1, a2)); break; + case INDEX_op_bitsel_vec: + /* vbitsel vd, vj, vk, va = bitsel_vec vd, va, vk, vj */ + tcg_out_opc_vbitsel_v(s, a0, a3, a2, a1); + break; case INDEX_op_dupm_vec: tcg_out_dupm_vec(s, type, vece, a0, a1, a2); break; @@ -XXX,XX +XXX,XX @@ int tcg_can_emit_vec_op(TCGOpcode opc, TCGType type, unsigned vece) case INDEX_op_shlv_vec: case INDEX_op_shrv_vec: case INDEX_op_sarv_vec: + case INDEX_op_bitsel_vec: return 1; default: return 0; @@ -XXX,XX +XXX,XX @@ static TCGConstraintSetIndex tcg_target_op_def(TCGOpcode op) case INDEX_op_neg_vec: return C_O1_I1(w, w); + case INDEX_op_bitsel_vec: + return C_O1_I3(w, w, w, w); + default: g_assert_not_reached(); } -- 2.34.1 From: Jiajie Chen <c@jia.je> Lower the following ops: - shli_vec - shrv_vec - sarv_vec Signed-off-by: Jiajie Chen <c@jia.je> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-Id: <20230908022302.180442-14-c@jia.je> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- tcg/loongarch64/tcg-target.h | 2 +- tcg/loongarch64/tcg-target.c.inc | 21 +++++++++++++++++++++ 2 files changed, 22 insertions(+), 1 deletion(-) diff --git a/tcg/loongarch64/tcg-target.h b/tcg/loongarch64/tcg-target.h index XXXXXXX..XXXXXXX 100644 --- a/tcg/loongarch64/tcg-target.h +++ b/tcg/loongarch64/tcg-target.h @@ -XXX,XX +XXX,XX @@ extern bool use_lsx_instructions; #define TCG_TARGET_HAS_nor_vec 1 #define TCG_TARGET_HAS_eqv_vec 0 #define TCG_TARGET_HAS_mul_vec 1 -#define TCG_TARGET_HAS_shi_vec 0 +#define TCG_TARGET_HAS_shi_vec 1 #define TCG_TARGET_HAS_shs_vec 0 #define TCG_TARGET_HAS_shv_vec 1 #define TCG_TARGET_HAS_roti_vec 0 diff --git a/tcg/loongarch64/tcg-target.c.inc b/tcg/loongarch64/tcg-target.c.inc index XXXXXXX..XXXXXXX 100644 --- a/tcg/loongarch64/tcg-target.c.inc +++ b/tcg/loongarch64/tcg-target.c.inc @@ -XXX,XX +XXX,XX @@ static void tcg_out_vec_op(TCGContext *s, TCGOpcode opc, static const LoongArchInsn sarv_vec_insn[4] = { OPC_VSRA_B, OPC_VSRA_H, OPC_VSRA_W, OPC_VSRA_D }; + static const LoongArchInsn shli_vec_insn[4] = { + OPC_VSLLI_B, OPC_VSLLI_H, OPC_VSLLI_W, OPC_VSLLI_D + }; + static const LoongArchInsn shri_vec_insn[4] = { + OPC_VSRLI_B, OPC_VSRLI_H, OPC_VSRLI_W, OPC_VSRLI_D + }; + static const LoongArchInsn sari_vec_insn[4] = { + OPC_VSRAI_B, OPC_VSRAI_H, OPC_VSRAI_W, OPC_VSRAI_D + }; a0 = args[0]; a1 = args[1]; @@ -XXX,XX +XXX,XX @@ static void tcg_out_vec_op(TCGContext *s, TCGOpcode opc, case INDEX_op_sarv_vec: tcg_out32(s, encode_vdvjvk_insn(sarv_vec_insn[vece], a0, a1, a2)); break; + case INDEX_op_shli_vec: + tcg_out32(s, encode_vdvjuk3_insn(shli_vec_insn[vece], a0, a1, a2)); + break; + case INDEX_op_shri_vec: + tcg_out32(s, encode_vdvjuk3_insn(shri_vec_insn[vece], a0, a1, a2)); + break; + case INDEX_op_sari_vec: + tcg_out32(s, encode_vdvjuk3_insn(sari_vec_insn[vece], a0, a1, a2)); + break; case INDEX_op_bitsel_vec: /* vbitsel vd, vj, vk, va = bitsel_vec vd, va, vk, vj */ tcg_out_opc_vbitsel_v(s, a0, a3, a2, a1); @@ -XXX,XX +XXX,XX @@ static TCGConstraintSetIndex tcg_target_op_def(TCGOpcode op) case INDEX_op_not_vec: case INDEX_op_neg_vec: + case INDEX_op_shli_vec: + case INDEX_op_shri_vec: + case INDEX_op_sari_vec: return C_O1_I1(w, w); case INDEX_op_bitsel_vec: -- 2.34.1 From: Jiajie Chen <c@jia.je> Lower the following ops: - rotrv_vec - rotlv_vec Signed-off-by: Jiajie Chen <c@jia.je> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-Id: <20230908022302.180442-15-c@jia.je> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- tcg/loongarch64/tcg-target.h | 2 +- tcg/loongarch64/tcg-target.c.inc | 14 ++++++++++++++ 2 files changed, 15 insertions(+), 1 deletion(-) diff --git a/tcg/loongarch64/tcg-target.h b/tcg/loongarch64/tcg-target.h index XXXXXXX..XXXXXXX 100644 --- a/tcg/loongarch64/tcg-target.h +++ b/tcg/loongarch64/tcg-target.h @@ -XXX,XX +XXX,XX @@ extern bool use_lsx_instructions; #define TCG_TARGET_HAS_shv_vec 1 #define TCG_TARGET_HAS_roti_vec 0 #define TCG_TARGET_HAS_rots_vec 0 -#define TCG_TARGET_HAS_rotv_vec 0 +#define TCG_TARGET_HAS_rotv_vec 1 #define TCG_TARGET_HAS_sat_vec 1 #define TCG_TARGET_HAS_minmax_vec 1 #define TCG_TARGET_HAS_bitsel_vec 1 diff --git a/tcg/loongarch64/tcg-target.c.inc b/tcg/loongarch64/tcg-target.c.inc index XXXXXXX..XXXXXXX 100644 --- a/tcg/loongarch64/tcg-target.c.inc +++ b/tcg/loongarch64/tcg-target.c.inc @@ -XXX,XX +XXX,XX @@ static void tcg_out_vec_op(TCGContext *s, TCGOpcode opc, static const LoongArchInsn sari_vec_insn[4] = { OPC_VSRAI_B, OPC_VSRAI_H, OPC_VSRAI_W, OPC_VSRAI_D }; + static const LoongArchInsn rotrv_vec_insn[4] = { + OPC_VROTR_B, OPC_VROTR_H, OPC_VROTR_W, OPC_VROTR_D + }; a0 = args[0]; a1 = args[1]; @@ -XXX,XX +XXX,XX @@ static void tcg_out_vec_op(TCGContext *s, TCGOpcode opc, case INDEX_op_sari_vec: tcg_out32(s, encode_vdvjuk3_insn(sari_vec_insn[vece], a0, a1, a2)); break; + case INDEX_op_rotrv_vec: + tcg_out32(s, encode_vdvjvk_insn(rotrv_vec_insn[vece], a0, a1, a2)); + break; + case INDEX_op_rotlv_vec: + /* rotlv_vec a1, a2 = rotrv_vec a1, -a2 */ + tcg_out32(s, encode_vdvj_insn(neg_vec_insn[vece], temp_vec, a2)); + tcg_out32(s, encode_vdvjvk_insn(rotrv_vec_insn[vece], a0, a1, + temp_vec)); + break; case INDEX_op_bitsel_vec: /* vbitsel vd, vj, vk, va = bitsel_vec vd, va, vk, vj */ tcg_out_opc_vbitsel_v(s, a0, a3, a2, a1); @@ -XXX,XX +XXX,XX @@ static TCGConstraintSetIndex tcg_target_op_def(TCGOpcode op) case INDEX_op_shlv_vec: case INDEX_op_shrv_vec: case INDEX_op_sarv_vec: + case INDEX_op_rotrv_vec: + case INDEX_op_rotlv_vec: return C_O1_I2(w, w, w); case INDEX_op_not_vec: -- 2.34.1 From: Jiajie Chen <c@jia.je> Signed-off-by: Jiajie Chen <c@jia.je> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-Id: <20230908022302.180442-16-c@jia.je> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- tcg/loongarch64/tcg-target.h | 2 +- tcg/loongarch64/tcg-target.c.inc | 21 +++++++++++++++++++++ 2 files changed, 22 insertions(+), 1 deletion(-) diff --git a/tcg/loongarch64/tcg-target.h b/tcg/loongarch64/tcg-target.h index XXXXXXX..XXXXXXX 100644 --- a/tcg/loongarch64/tcg-target.h +++ b/tcg/loongarch64/tcg-target.h @@ -XXX,XX +XXX,XX @@ extern bool use_lsx_instructions; #define TCG_TARGET_HAS_shi_vec 1 #define TCG_TARGET_HAS_shs_vec 0 #define TCG_TARGET_HAS_shv_vec 1 -#define TCG_TARGET_HAS_roti_vec 0 +#define TCG_TARGET_HAS_roti_vec 1 #define TCG_TARGET_HAS_rots_vec 0 #define TCG_TARGET_HAS_rotv_vec 1 #define TCG_TARGET_HAS_sat_vec 1 diff --git a/tcg/loongarch64/tcg-target.c.inc b/tcg/loongarch64/tcg-target.c.inc index XXXXXXX..XXXXXXX 100644 --- a/tcg/loongarch64/tcg-target.c.inc +++ b/tcg/loongarch64/tcg-target.c.inc @@ -XXX,XX +XXX,XX @@ static void tcg_out_vec_op(TCGContext *s, TCGOpcode opc, tcg_out32(s, encode_vdvjvk_insn(rotrv_vec_insn[vece], a0, a1, temp_vec)); break; + case INDEX_op_rotli_vec: + /* rotli_vec a1, a2 = rotri_vec a1, -a2 */ + a2 = extract32(-a2, 0, 3 + vece); + switch (vece) { + case MO_8: + tcg_out_opc_vrotri_b(s, a0, a1, a2); + break; + case MO_16: + tcg_out_opc_vrotri_h(s, a0, a1, a2); + break; + case MO_32: + tcg_out_opc_vrotri_w(s, a0, a1, a2); + break; + case MO_64: + tcg_out_opc_vrotri_d(s, a0, a1, a2); + break; + default: + g_assert_not_reached(); + } + break; case INDEX_op_bitsel_vec: /* vbitsel vd, vj, vk, va = bitsel_vec vd, va, vk, vj */ tcg_out_opc_vbitsel_v(s, a0, a3, a2, a1); @@ -XXX,XX +XXX,XX @@ static TCGConstraintSetIndex tcg_target_op_def(TCGOpcode op) case INDEX_op_shli_vec: case INDEX_op_shri_vec: case INDEX_op_sari_vec: + case INDEX_op_rotli_vec: return C_O1_I1(w, w); case INDEX_op_bitsel_vec: -- 2.34.1 From: Jiajie Chen <c@jia.je> If LSX is available, use LSX instructions to implement 128-bit load & store when MO_128 is required, otherwise use two 64-bit loads & stores. Signed-off-by: Jiajie Chen <c@jia.je> Message-Id: <20230908022302.180442-17-c@jia.je> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- tcg/loongarch64/tcg-target-con-set.h | 2 + tcg/loongarch64/tcg-target.h | 2 +- tcg/loongarch64/tcg-target.c.inc | 63 ++++++++++++++++++++++++++++ 3 files changed, 66 insertions(+), 1 deletion(-) diff --git a/tcg/loongarch64/tcg-target-con-set.h b/tcg/loongarch64/tcg-target-con-set.h index XXXXXXX..XXXXXXX 100644 --- a/tcg/loongarch64/tcg-target-con-set.h +++ b/tcg/loongarch64/tcg-target-con-set.h @@ -XXX,XX +XXX,XX @@ C_O0_I1(r) C_O0_I2(rZ, r) C_O0_I2(rZ, rZ) C_O0_I2(w, r) +C_O0_I3(r, r, r) C_O1_I1(r, r) C_O1_I1(w, r) C_O1_I1(w, w) @@ -XXX,XX +XXX,XX @@ C_O1_I2(w, w, wM) C_O1_I2(w, w, wA) C_O1_I3(w, w, w, w) C_O1_I4(r, rZ, rJ, rZ, rZ) +C_O2_I1(r, r, r) diff --git a/tcg/loongarch64/tcg-target.h b/tcg/loongarch64/tcg-target.h index XXXXXXX..XXXXXXX 100644 --- a/tcg/loongarch64/tcg-target.h +++ b/tcg/loongarch64/tcg-target.h @@ -XXX,XX +XXX,XX @@ extern bool use_lsx_instructions; #define TCG_TARGET_HAS_muluh_i64 1 #define TCG_TARGET_HAS_mulsh_i64 1 -#define TCG_TARGET_HAS_qemu_ldst_i128 0 +#define TCG_TARGET_HAS_qemu_ldst_i128 use_lsx_instructions #define TCG_TARGET_HAS_v64 0 #define TCG_TARGET_HAS_v128 use_lsx_instructions diff --git a/tcg/loongarch64/tcg-target.c.inc b/tcg/loongarch64/tcg-target.c.inc index XXXXXXX..XXXXXXX 100644 --- a/tcg/loongarch64/tcg-target.c.inc +++ b/tcg/loongarch64/tcg-target.c.inc @@ -XXX,XX +XXX,XX @@ static void tcg_out_qemu_st(TCGContext *s, TCGReg data_reg, TCGReg addr_reg, } } +static void tcg_out_qemu_ldst_i128(TCGContext *s, TCGReg data_lo, TCGReg data_hi, + TCGReg addr_reg, MemOpIdx oi, bool is_ld) +{ + TCGLabelQemuLdst *ldst; + HostAddress h; + + ldst = prepare_host_addr(s, &h, addr_reg, oi, is_ld); + + if (h.aa.atom == MO_128) { + /* + * Use VLDX/VSTX when 128-bit atomicity is required. + * If address is aligned to 16-bytes, the 128-bit load/store is atomic. + */ + if (is_ld) { + tcg_out_opc_vldx(s, TCG_VEC_TMP0, h.base, h.index); + tcg_out_opc_vpickve2gr_d(s, data_lo, TCG_VEC_TMP0, 0); + tcg_out_opc_vpickve2gr_d(s, data_hi, TCG_VEC_TMP0, 1); + } else { + tcg_out_opc_vinsgr2vr_d(s, TCG_VEC_TMP0, data_lo, 0); + tcg_out_opc_vinsgr2vr_d(s, TCG_VEC_TMP0, data_hi, 1); + tcg_out_opc_vstx(s, TCG_VEC_TMP0, h.base, h.index); + } + } else { + /* Otherwise use a pair of LD/ST. */ + TCGReg base = h.base; + if (h.index != TCG_REG_ZERO) { + base = TCG_REG_TMP0; + tcg_out_opc_add_d(s, base, h.base, h.index); + } + if (is_ld) { + tcg_out_opc_ld_d(s, data_lo, base, 0); + tcg_out_opc_ld_d(s, data_hi, base, 8); + } else { + tcg_out_opc_st_d(s, data_lo, base, 0); + tcg_out_opc_st_d(s, data_hi, base, 8); + } + } + + if (ldst) { + ldst->type = TCG_TYPE_I128; + ldst->datalo_reg = data_lo; + ldst->datahi_reg = data_hi; + ldst->raddr = tcg_splitwx_to_rx(s->code_ptr); + } +} + /* * Entry-points */ @@ -XXX,XX +XXX,XX @@ static void tcg_out_op(TCGContext *s, TCGOpcode opc, TCGArg a0 = args[0]; TCGArg a1 = args[1]; TCGArg a2 = args[2]; + TCGArg a3 = args[3]; int c2 = const_args[2]; switch (opc) { @@ -XXX,XX +XXX,XX @@ static void tcg_out_op(TCGContext *s, TCGOpcode opc, case INDEX_op_qemu_ld_a64_i64: tcg_out_qemu_ld(s, a0, a1, a2, TCG_TYPE_I64); break; + case INDEX_op_qemu_ld_a32_i128: + case INDEX_op_qemu_ld_a64_i128: + tcg_out_qemu_ldst_i128(s, a0, a1, a2, a3, true); + break; case INDEX_op_qemu_st_a32_i32: case INDEX_op_qemu_st_a64_i32: tcg_out_qemu_st(s, a0, a1, a2, TCG_TYPE_I32); @@ -XXX,XX +XXX,XX @@ static void tcg_out_op(TCGContext *s, TCGOpcode opc, case INDEX_op_qemu_st_a64_i64: tcg_out_qemu_st(s, a0, a1, a2, TCG_TYPE_I64); break; + case INDEX_op_qemu_st_a32_i128: + case INDEX_op_qemu_st_a64_i128: + tcg_out_qemu_ldst_i128(s, a0, a1, a2, a3, false); + break; case INDEX_op_mov_i32: /* Always emitted via tcg_out_mov. */ case INDEX_op_mov_i64: @@ -XXX,XX +XXX,XX @@ static TCGConstraintSetIndex tcg_target_op_def(TCGOpcode op) case INDEX_op_qemu_st_a64_i64: return C_O0_I2(rZ, r); + case INDEX_op_qemu_ld_a32_i128: + case INDEX_op_qemu_ld_a64_i128: + return C_O2_I1(r, r, r); + + case INDEX_op_qemu_st_a32_i128: + case INDEX_op_qemu_st_a64_i128: + return C_O0_I3(r, r, r); + case INDEX_op_brcond_i32: case INDEX_op_brcond_i64: return C_O0_I2(rZ, rZ); -- 2.34.1 Signed-off-by: Richard Henderson <richard.henderson@linaro.org> Tested-by: Song Gao <gaosong@loongson.cn> Reviewed-by: Song Gao <gaosong@loongson.cn> Message-Id: <20230831030904.1194667-2-richard.henderson@linaro.org> --- accel/tcg/tcg-runtime.h | 25 ++++++ include/tcg/tcg-op-gvec-common.h | 6 ++ accel/tcg/tcg-runtime-gvec.c | 26 ++++++ tcg/tcg-op-gvec.c | 149 +++++++++++++++++++++++++++++++ 4 files changed, 206 insertions(+) diff --git a/accel/tcg/tcg-runtime.h b/accel/tcg/tcg-runtime.h index XXXXXXX..XXXXXXX 100644 --- a/accel/tcg/tcg-runtime.h +++ b/accel/tcg/tcg-runtime.h @@ -XXX,XX +XXX,XX @@ DEF_HELPER_FLAGS_4(gvec_leu16, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) DEF_HELPER_FLAGS_4(gvec_leu32, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) DEF_HELPER_FLAGS_4(gvec_leu64, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_4(gvec_eqs8, TCG_CALL_NO_RWG, void, ptr, ptr, i64, i32) +DEF_HELPER_FLAGS_4(gvec_eqs16, TCG_CALL_NO_RWG, void, ptr, ptr, i64, i32) +DEF_HELPER_FLAGS_4(gvec_eqs32, TCG_CALL_NO_RWG, void, ptr, ptr, i64, i32) +DEF_HELPER_FLAGS_4(gvec_eqs64, TCG_CALL_NO_RWG, void, ptr, ptr, i64, i32) + +DEF_HELPER_FLAGS_4(gvec_lts8, TCG_CALL_NO_RWG, void, ptr, ptr, i64, i32) +DEF_HELPER_FLAGS_4(gvec_lts16, TCG_CALL_NO_RWG, void, ptr, ptr, i64, i32) +DEF_HELPER_FLAGS_4(gvec_lts32, TCG_CALL_NO_RWG, void, ptr, ptr, i64, i32) +DEF_HELPER_FLAGS_4(gvec_lts64, TCG_CALL_NO_RWG, void, ptr, ptr, i64, i32) + +DEF_HELPER_FLAGS_4(gvec_les8, TCG_CALL_NO_RWG, void, ptr, ptr, i64, i32) +DEF_HELPER_FLAGS_4(gvec_les16, TCG_CALL_NO_RWG, void, ptr, ptr, i64, i32) +DEF_HELPER_FLAGS_4(gvec_les32, TCG_CALL_NO_RWG, void, ptr, ptr, i64, i32) +DEF_HELPER_FLAGS_4(gvec_les64, TCG_CALL_NO_RWG, void, ptr, ptr, i64, i32) + +DEF_HELPER_FLAGS_4(gvec_ltus8, TCG_CALL_NO_RWG, void, ptr, ptr, i64, i32) +DEF_HELPER_FLAGS_4(gvec_ltus16, TCG_CALL_NO_RWG, void, ptr, ptr, i64, i32) +DEF_HELPER_FLAGS_4(gvec_ltus32, TCG_CALL_NO_RWG, void, ptr, ptr, i64, i32) +DEF_HELPER_FLAGS_4(gvec_ltus64, TCG_CALL_NO_RWG, void, ptr, ptr, i64, i32) + +DEF_HELPER_FLAGS_4(gvec_leus8, TCG_CALL_NO_RWG, void, ptr, ptr, i64, i32) +DEF_HELPER_FLAGS_4(gvec_leus16, TCG_CALL_NO_RWG, void, ptr, ptr, i64, i32) +DEF_HELPER_FLAGS_4(gvec_leus32, TCG_CALL_NO_RWG, void, ptr, ptr, i64, i32) +DEF_HELPER_FLAGS_4(gvec_leus64, TCG_CALL_NO_RWG, void, ptr, ptr, i64, i32) + DEF_HELPER_FLAGS_5(gvec_bitsel, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, ptr, i32) diff --git a/include/tcg/tcg-op-gvec-common.h b/include/tcg/tcg-op-gvec-common.h index XXXXXXX..XXXXXXX 100644 --- a/include/tcg/tcg-op-gvec-common.h +++ b/include/tcg/tcg-op-gvec-common.h @@ -XXX,XX +XXX,XX @@ void tcg_gen_gvec_rotrv(unsigned vece, uint32_t dofs, uint32_t aofs, void tcg_gen_gvec_cmp(TCGCond cond, unsigned vece, uint32_t dofs, uint32_t aofs, uint32_t bofs, uint32_t oprsz, uint32_t maxsz); +void tcg_gen_gvec_cmpi(TCGCond cond, unsigned vece, uint32_t dofs, + uint32_t aofs, int64_t c, + uint32_t oprsz, uint32_t maxsz); +void tcg_gen_gvec_cmps(TCGCond cond, unsigned vece, uint32_t dofs, + uint32_t aofs, TCGv_i64 c, + uint32_t oprsz, uint32_t maxsz); /* * Perform vector bit select: d = (b & a) | (c & ~a). diff --git a/accel/tcg/tcg-runtime-gvec.c b/accel/tcg/tcg-runtime-gvec.c index XXXXXXX..XXXXXXX 100644 --- a/accel/tcg/tcg-runtime-gvec.c +++ b/accel/tcg/tcg-runtime-gvec.c @@ -XXX,XX +XXX,XX @@ DO_CMP2(64) #undef DO_CMP1 #undef DO_CMP2 +#define DO_CMP1(NAME, TYPE, OP) \ +void HELPER(NAME)(void *d, void *a, uint64_t b64, uint32_t desc) \ +{ \ + intptr_t oprsz = simd_oprsz(desc); \ + TYPE inv = simd_data(desc), b = b64; \ + for (intptr_t i = 0; i < oprsz; i += sizeof(TYPE)) { \ + *(TYPE *)(d + i) = -((*(TYPE *)(a + i) OP b) ^ inv); \ + } \ + clear_high(d, oprsz, desc); \ +} + +#define DO_CMP2(SZ) \ + DO_CMP1(gvec_eqs##SZ, uint##SZ##_t, ==) \ + DO_CMP1(gvec_lts##SZ, int##SZ##_t, <) \ + DO_CMP1(gvec_les##SZ, int##SZ##_t, <=) \ + DO_CMP1(gvec_ltus##SZ, uint##SZ##_t, <) \ + DO_CMP1(gvec_leus##SZ, uint##SZ##_t, <=) + +DO_CMP2(8) +DO_CMP2(16) +DO_CMP2(32) +DO_CMP2(64) + +#undef DO_CMP1 +#undef DO_CMP2 + void HELPER(gvec_ssadd8)(void *d, void *a, void *b, uint32_t desc) { intptr_t oprsz = simd_oprsz(desc); diff --git a/tcg/tcg-op-gvec.c b/tcg/tcg-op-gvec.c index XXXXXXX..XXXXXXX 100644 --- a/tcg/tcg-op-gvec.c +++ b/tcg/tcg-op-gvec.c @@ -XXX,XX +XXX,XX @@ void tcg_gen_gvec_cmp(TCGCond cond, unsigned vece, uint32_t dofs, } } +static void expand_cmps_vec(unsigned vece, uint32_t dofs, uint32_t aofs, + uint32_t oprsz, uint32_t tysz, TCGType type, + TCGCond cond, TCGv_vec c) +{ + TCGv_vec t0 = tcg_temp_new_vec(type); + TCGv_vec t1 = tcg_temp_new_vec(type); + uint32_t i; + + for (i = 0; i < oprsz; i += tysz) { + tcg_gen_ld_vec(t1, cpu_env, aofs + i); + tcg_gen_cmp_vec(cond, vece, t0, t1, c); + tcg_gen_st_vec(t0, cpu_env, dofs + i); + } +} + +void tcg_gen_gvec_cmps(TCGCond cond, unsigned vece, uint32_t dofs, + uint32_t aofs, TCGv_i64 c, + uint32_t oprsz, uint32_t maxsz) +{ + static const TCGOpcode cmp_list[] = { INDEX_op_cmp_vec, 0 }; + static gen_helper_gvec_2i * const eq_fn[4] = { + gen_helper_gvec_eqs8, gen_helper_gvec_eqs16, + gen_helper_gvec_eqs32, gen_helper_gvec_eqs64 + }; + static gen_helper_gvec_2i * const lt_fn[4] = { + gen_helper_gvec_lts8, gen_helper_gvec_lts16, + gen_helper_gvec_lts32, gen_helper_gvec_lts64 + }; + static gen_helper_gvec_2i * const le_fn[4] = { + gen_helper_gvec_les8, gen_helper_gvec_les16, + gen_helper_gvec_les32, gen_helper_gvec_les64 + }; + static gen_helper_gvec_2i * const ltu_fn[4] = { + gen_helper_gvec_ltus8, gen_helper_gvec_ltus16, + gen_helper_gvec_ltus32, gen_helper_gvec_ltus64 + }; + static gen_helper_gvec_2i * const leu_fn[4] = { + gen_helper_gvec_leus8, gen_helper_gvec_leus16, + gen_helper_gvec_leus32, gen_helper_gvec_leus64 + }; + static gen_helper_gvec_2i * const * const fns[16] = { + [TCG_COND_EQ] = eq_fn, + [TCG_COND_LT] = lt_fn, + [TCG_COND_LE] = le_fn, + [TCG_COND_LTU] = ltu_fn, + [TCG_COND_LEU] = leu_fn, + }; + + TCGType type; + + check_size_align(oprsz, maxsz, dofs | aofs); + check_overlap_2(dofs, aofs, maxsz); + + if (cond == TCG_COND_NEVER || cond == TCG_COND_ALWAYS) { + do_dup(MO_8, dofs, oprsz, maxsz, + NULL, NULL, -(cond == TCG_COND_ALWAYS)); + return; + } + + /* + * Implement inline with a vector type, if possible. + * Prefer integer when 64-bit host and 64-bit comparison. + */ + type = choose_vector_type(cmp_list, vece, oprsz, + TCG_TARGET_REG_BITS == 64 && vece == MO_64); + if (type != 0) { + const TCGOpcode *hold_list = tcg_swap_vecop_list(cmp_list); + TCGv_vec t_vec = tcg_temp_new_vec(type); + uint32_t some; + + tcg_gen_dup_i64_vec(vece, t_vec, c); + switch (type) { + case TCG_TYPE_V256: + some = QEMU_ALIGN_DOWN(oprsz, 32); + expand_cmps_vec(vece, dofs, aofs, some, 32, + TCG_TYPE_V256, cond, t_vec); + aofs += some; + dofs += some; + oprsz -= some; + maxsz -= some; + /* fallthru */ + + case TCG_TYPE_V128: + some = QEMU_ALIGN_DOWN(oprsz, 16); + expand_cmps_vec(vece, dofs, aofs, some, 16, + TCG_TYPE_V128, cond, t_vec); + break; + + case TCG_TYPE_V64: + some = QEMU_ALIGN_DOWN(oprsz, 8); + expand_cmps_vec(vece, dofs, aofs, some, 8, + TCG_TYPE_V64, cond, t_vec); + break; + + default: + g_assert_not_reached(); + } + tcg_temp_free_vec(t_vec); + tcg_swap_vecop_list(hold_list); + } else if (vece == MO_64 && check_size_impl(oprsz, 8)) { + TCGv_i64 t0 = tcg_temp_ebb_new_i64(); + uint32_t i; + + for (i = 0; i < oprsz; i += 8) { + tcg_gen_ld_i64(t0, cpu_env, aofs + i); + tcg_gen_negsetcond_i64(cond, t0, t0, c); + tcg_gen_st_i64(t0, cpu_env, dofs + i); + } + tcg_temp_free_i64(t0); + } else if (vece == MO_32 && check_size_impl(oprsz, 4)) { + TCGv_i32 t0 = tcg_temp_ebb_new_i32(); + TCGv_i32 t1 = tcg_temp_ebb_new_i32(); + uint32_t i; + + tcg_gen_extrl_i64_i32(t1, c); + for (i = 0; i < oprsz; i += 8) { + tcg_gen_ld_i32(t0, cpu_env, aofs + i); + tcg_gen_negsetcond_i32(cond, t0, t0, t1); + tcg_gen_st_i32(t0, cpu_env, dofs + i); + } + tcg_temp_free_i32(t0); + tcg_temp_free_i32(t1); + } else { + gen_helper_gvec_2i * const *fn = fns[cond]; + bool inv = false; + + if (fn == NULL) { + cond = tcg_invert_cond(cond); + fn = fns[cond]; + assert(fn != NULL); + inv = true; + } + tcg_gen_gvec_2i_ool(dofs, aofs, c, oprsz, maxsz, inv, fn[vece]); + return; + } + + if (oprsz < maxsz) { + expand_clr(dofs + oprsz, maxsz - oprsz); + } +} + +void tcg_gen_gvec_cmpi(TCGCond cond, unsigned vece, uint32_t dofs, + uint32_t aofs, int64_t c, + uint32_t oprsz, uint32_t maxsz) +{ + TCGv_i64 tmp = tcg_constant_i64(c); + tcg_gen_gvec_cmps(cond, vece, dofs, aofs, tmp, oprsz, maxsz); +} + static void tcg_gen_bitsel_i64(TCGv_i64 d, TCGv_i64 a, TCGv_i64 b, TCGv_i64 c) { TCGv_i64 t = tcg_temp_ebb_new_i64(); -- 2.34.1 Signed-off-by: Richard Henderson <richard.henderson@linaro.org> Reviewed-by: Song Gao <gaosong@loongson.cn> Message-Id: <20230831030904.1194667-3-richard.henderson@linaro.org> --- target/arm/tcg/translate.c | 56 ++++++-------------------------------- 1 file changed, 9 insertions(+), 47 deletions(-) diff --git a/target/arm/tcg/translate.c b/target/arm/tcg/translate.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/tcg/translate.c +++ b/target/arm/tcg/translate.c @@ -XXX,XX +XXX,XX @@ void gen_gvec_sqrdmlsh_qc(unsigned vece, uint32_t rd_ofs, uint32_t rn_ofs, gen_gvec_fn3_qc(rd_ofs, rn_ofs, rm_ofs, opr_sz, max_sz, fns[vece - 1]); } -#define GEN_CMP0(NAME, COND) \ - static void gen_##NAME##0_i32(TCGv_i32 d, TCGv_i32 a) \ - { \ - tcg_gen_negsetcond_i32(COND, d, a, tcg_constant_i32(0)); \ - } \ - static void gen_##NAME##0_i64(TCGv_i64 d, TCGv_i64 a) \ - { \ - tcg_gen_negsetcond_i64(COND, d, a, tcg_constant_i64(0)); \ - } \ - static void gen_##NAME##0_vec(unsigned vece, TCGv_vec d, TCGv_vec a) \ - { \ - TCGv_vec zero = tcg_constant_vec_matching(d, vece, 0); \ - tcg_gen_cmp_vec(COND, vece, d, a, zero); \ - } \ - void gen_gvec_##NAME##0(unsigned vece, uint32_t d, uint32_t m, \ - uint32_t opr_sz, uint32_t max_sz) \ - { \ - const GVecGen2 op[4] = { \ - { .fno = gen_helper_gvec_##NAME##0_b, \ - .fniv = gen_##NAME##0_vec, \ - .opt_opc = vecop_list_cmp, \ - .vece = MO_8 }, \ - { .fno = gen_helper_gvec_##NAME##0_h, \ - .fniv = gen_##NAME##0_vec, \ - .opt_opc = vecop_list_cmp, \ - .vece = MO_16 }, \ - { .fni4 = gen_##NAME##0_i32, \ - .fniv = gen_##NAME##0_vec, \ - .opt_opc = vecop_list_cmp, \ - .vece = MO_32 }, \ - { .fni8 = gen_##NAME##0_i64, \ - .fniv = gen_##NAME##0_vec, \ - .opt_opc = vecop_list_cmp, \ - .prefer_i64 = TCG_TARGET_REG_BITS == 64, \ - .vece = MO_64 }, \ - }; \ - tcg_gen_gvec_2(d, m, opr_sz, max_sz, &op[vece]); \ - } +#define GEN_CMP0(NAME, COND) \ + void NAME(unsigned vece, uint32_t d, uint32_t m, \ + uint32_t opr_sz, uint32_t max_sz) \ + { tcg_gen_gvec_cmpi(COND, vece, d, m, 0, opr_sz, max_sz); } -static const TCGOpcode vecop_list_cmp[] = { - INDEX_op_cmp_vec, 0 -}; - -GEN_CMP0(ceq, TCG_COND_EQ) -GEN_CMP0(cle, TCG_COND_LE) -GEN_CMP0(cge, TCG_COND_GE) -GEN_CMP0(clt, TCG_COND_LT) -GEN_CMP0(cgt, TCG_COND_GT) +GEN_CMP0(gen_gvec_ceq0, TCG_COND_EQ) +GEN_CMP0(gen_gvec_cle0, TCG_COND_LE) +GEN_CMP0(gen_gvec_cge0, TCG_COND_GE) +GEN_CMP0(gen_gvec_clt0, TCG_COND_LT) +GEN_CMP0(gen_gvec_cgt0, TCG_COND_GT) #undef GEN_CMP0 -- 2.34.1 Now that we defer address space update and tlb_flush until the next async_run_on_cpu, the plugin run at the end of the instruction no longer has to contend with a flushed tlb. Therefore, delete SavedIOTLB entirely. Properly return false from tlb_plugin_lookup when we do not have a tlb match. Fixes a bug in which SavedIOTLB had stale data, because there were multiple i/o accesses within a single insn. Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- include/hw/core/cpu.h | 13 ------- include/qemu/typedefs.h | 1 - accel/tcg/cputlb.c | 79 ++++++++++++----------------------------- 3 files changed, 23 insertions(+), 70 deletions(-) diff --git a/include/hw/core/cpu.h b/include/hw/core/cpu.h index XXXXXXX..XXXXXXX 100644 --- a/include/hw/core/cpu.h +++ b/include/hw/core/cpu.h @@ -XXX,XX +XXX,XX @@ struct CPUWatchpoint { QTAILQ_ENTRY(CPUWatchpoint) entry; }; -#ifdef CONFIG_PLUGIN -/* - * For plugins we sometime need to save the resolved iotlb data before - * the memory regions get moved around by io_writex. - */ -typedef struct SavedIOTLB { - MemoryRegionSection *section; - hwaddr mr_offset; -} SavedIOTLB; -#endif - struct KVMState; struct kvm_run; @@ -XXX,XX +XXX,XX @@ struct CPUState { #ifdef CONFIG_PLUGIN GArray *plugin_mem_cbs; - /* saved iotlb data from io_writex */ - SavedIOTLB saved_iotlb; #endif /* TODO Move common fields from CPUArchState here. */ diff --git a/include/qemu/typedefs.h b/include/qemu/typedefs.h index XXXXXXX..XXXXXXX 100644 --- a/include/qemu/typedefs.h +++ b/include/qemu/typedefs.h @@ -XXX,XX +XXX,XX @@ typedef struct QString QString; typedef struct RAMBlock RAMBlock; typedef struct Range Range; typedef struct ReservedRegion ReservedRegion; -typedef struct SavedIOTLB SavedIOTLB; typedef struct SHPCDevice SHPCDevice; typedef struct SSIBus SSIBus; typedef struct TCGHelperInfo TCGHelperInfo; diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c index XXXXXXX..XXXXXXX 100644 --- a/accel/tcg/cputlb.c +++ b/accel/tcg/cputlb.c @@ -XXX,XX +XXX,XX @@ static inline void cpu_transaction_failed(CPUState *cpu, hwaddr physaddr, } } -/* - * Save a potentially trashed CPUTLBEntryFull for later lookup by plugin. - * This is read by tlb_plugin_lookup if the fulltlb entry doesn't match - * because of the side effect of io_writex changing memory layout. - */ -static void save_iotlb_data(CPUState *cs, MemoryRegionSection *section, - hwaddr mr_offset) -{ -#ifdef CONFIG_PLUGIN - SavedIOTLB *saved = &cs->saved_iotlb; - saved->section = section; - saved->mr_offset = mr_offset; -#endif -} - static uint64_t io_readx(CPUArchState *env, CPUTLBEntryFull *full, int mmu_idx, vaddr addr, uintptr_t retaddr, MMUAccessType access_type, MemOp op) @@ -XXX,XX +XXX,XX @@ static uint64_t io_readx(CPUArchState *env, CPUTLBEntryFull *full, cpu_io_recompile(cpu, retaddr); } - /* - * The memory_region_dispatch may trigger a flush/resize - * so for plugins we save the iotlb_data just in case. - */ - save_iotlb_data(cpu, section, mr_offset); - { QEMU_IOTHREAD_LOCK_GUARD(); r = memory_region_dispatch_read(mr, mr_offset, &val, op, full->attrs); @@ -XXX,XX +XXX,XX @@ static void io_writex(CPUArchState *env, CPUTLBEntryFull *full, } cpu->mem_io_pc = retaddr; - /* - * The memory_region_dispatch may trigger a flush/resize - * so for plugins we save the iotlb_data just in case. - */ - save_iotlb_data(cpu, section, mr_offset); - { QEMU_IOTHREAD_LOCK_GUARD(); r = memory_region_dispatch_write(mr, mr_offset, val, op, full->attrs); @@ -XXX,XX +XXX,XX @@ tb_page_addr_t get_page_addr_code_hostp(CPUArchState *env, vaddr addr, * in the softmmu lookup code (or helper). We don't handle re-fills or * checking the victim table. This is purely informational. * - * This almost never fails as the memory access being instrumented - * should have just filled the TLB. The one corner case is io_writex - * which can cause TLB flushes and potential resizing of the TLBs - * losing the information we need. In those cases we need to recover - * data from a copy of the CPUTLBEntryFull. As long as this always occurs - * from the same thread (which a mem callback will be) this is safe. + * The one corner case is i/o write, which can cause changes to the + * address space. Those changes, and the corresponding tlb flush, + * should be delayed until the next TB, so even then this ought not fail. + * But check, Just in Case. */ - bool tlb_plugin_lookup(CPUState *cpu, vaddr addr, int mmu_idx, bool is_store, struct qemu_plugin_hwaddr *data) { CPUArchState *env = cpu->env_ptr; CPUTLBEntry *tlbe = tlb_entry(env, mmu_idx, addr); uintptr_t index = tlb_index(env, mmu_idx, addr); - uint64_t tlb_addr = is_store ? tlb_addr_write(tlbe) : tlbe->addr_read; + MMUAccessType access_type = is_store ? MMU_DATA_STORE : MMU_DATA_LOAD; + uint64_t tlb_addr = tlb_read_idx(tlbe, access_type); - if (likely(tlb_hit(tlb_addr, addr))) { - /* We must have an iotlb entry for MMIO */ - if (tlb_addr & TLB_MMIO) { - CPUTLBEntryFull *full; - full = &env_tlb(env)->d[mmu_idx].fulltlb[index]; - data->is_io = true; - data->v.io.section = - iotlb_to_section(cpu, full->xlat_section, full->attrs); - data->v.io.offset = (full->xlat_section & TARGET_PAGE_MASK) + addr; - } else { - data->is_io = false; - data->v.ram.hostaddr = (void *)((uintptr_t)addr + tlbe->addend); - } - return true; - } else { - SavedIOTLB *saved = &cpu->saved_iotlb; - data->is_io = true; - data->v.io.section = saved->section; - data->v.io.offset = saved->mr_offset; - return true; + if (unlikely(!tlb_hit(tlb_addr, addr))) { + return false; } -} + /* We must have an iotlb entry for MMIO */ + if (tlb_addr & TLB_MMIO) { + CPUTLBEntryFull *full = &env_tlb(env)->d[mmu_idx].fulltlb[index]; + hwaddr xlat = full->xlat_section; + + data->is_io = true; + data->v.io.offset = (xlat & TARGET_PAGE_MASK) + addr; + data->v.io.section = + iotlb_to_section(cpu, xlat & ~TARGET_PAGE_MASK, full->attrs); + } else { + data->is_io = false; + data->v.ram.hostaddr = (void *)((uintptr_t)addr + tlbe->addend); + } + return true; +} #endif /* -- 2.34.1 These are common code from io_readx and io_writex. Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- accel/tcg/cputlb.c | 77 +++++++++++++++++++++++++++------------------- 1 file changed, 45 insertions(+), 32 deletions(-) diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c index XXXXXXX..XXXXXXX 100644 --- a/accel/tcg/cputlb.c +++ b/accel/tcg/cputlb.c @@ -XXX,XX +XXX,XX @@ void tlb_set_page_full(CPUState *cpu, int mmu_idx, * (non-page-aligned) vaddr of the eventual memory access to get * the MemoryRegion offset for the access. Note that the vaddr we * subtract here is that of the page base, and not the same as the - * vaddr we add back in io_readx()/io_writex()/get_page_addr_code(). + * vaddr we add back in io_prepare()/get_page_addr_code(). */ desc->fulltlb[index] = *full; full = &desc->fulltlb[index]; @@ -XXX,XX +XXX,XX @@ static inline void cpu_transaction_failed(CPUState *cpu, hwaddr physaddr, } } -static uint64_t io_readx(CPUArchState *env, CPUTLBEntryFull *full, - int mmu_idx, vaddr addr, uintptr_t retaddr, - MMUAccessType access_type, MemOp op) +static MemoryRegionSection * +io_prepare(hwaddr *out_offset, CPUArchState *env, hwaddr xlat, + MemTxAttrs attrs, vaddr addr, uintptr_t retaddr) { CPUState *cpu = env_cpu(env); - hwaddr mr_offset; MemoryRegionSection *section; - MemoryRegion *mr; - uint64_t val; - MemTxResult r; + hwaddr mr_offset; - section = iotlb_to_section(cpu, full->xlat_section, full->attrs); - mr = section->mr; - mr_offset = (full->xlat_section & TARGET_PAGE_MASK) + addr; + section = iotlb_to_section(cpu, xlat, attrs); + mr_offset = (xlat & TARGET_PAGE_MASK) + addr; cpu->mem_io_pc = retaddr; if (!cpu->can_do_io) { cpu_io_recompile(cpu, retaddr); } + *out_offset = mr_offset; + return section; +} + +static void io_failed(CPUArchState *env, CPUTLBEntryFull *full, vaddr addr, + unsigned size, MMUAccessType access_type, int mmu_idx, + MemTxResult response, uintptr_t retaddr, + MemoryRegionSection *section, hwaddr mr_offset) +{ + hwaddr physaddr = (mr_offset + + section->offset_within_address_space - + section->offset_within_region); + + cpu_transaction_failed(env_cpu(env), physaddr, addr, size, access_type, + mmu_idx, full->attrs, response, retaddr); +} + +static uint64_t io_readx(CPUArchState *env, CPUTLBEntryFull *full, + int mmu_idx, vaddr addr, uintptr_t retaddr, + MMUAccessType access_type, MemOp op) +{ + MemoryRegionSection *section; + hwaddr mr_offset; + MemoryRegion *mr; + MemTxResult r; + uint64_t val; + + section = io_prepare(&mr_offset, env, full->xlat_section, + full->attrs, addr, retaddr); + mr = section->mr; + { QEMU_IOTHREAD_LOCK_GUARD(); r = memory_region_dispatch_read(mr, mr_offset, &val, op, full->attrs); } if (r != MEMTX_OK) { - hwaddr physaddr = mr_offset + - section->offset_within_address_space - - section->offset_within_region; - - cpu_transaction_failed(cpu, physaddr, addr, memop_size(op), access_type, - mmu_idx, full->attrs, r, retaddr); + io_failed(env, full, addr, memop_size(op), access_type, mmu_idx, + r, retaddr, section, mr_offset); } return val; } @@ -XXX,XX +XXX,XX @@ static void io_writex(CPUArchState *env, CPUTLBEntryFull *full, int mmu_idx, uint64_t val, vaddr addr, uintptr_t retaddr, MemOp op) { - CPUState *cpu = env_cpu(env); - hwaddr mr_offset; MemoryRegionSection *section; + hwaddr mr_offset; MemoryRegion *mr; MemTxResult r; - section = iotlb_to_section(cpu, full->xlat_section, full->attrs); + section = io_prepare(&mr_offset, env, full->xlat_section, + full->attrs, addr, retaddr); mr = section->mr; - mr_offset = (full->xlat_section & TARGET_PAGE_MASK) + addr; - if (!cpu->can_do_io) { - cpu_io_recompile(cpu, retaddr); - } - cpu->mem_io_pc = retaddr; { QEMU_IOTHREAD_LOCK_GUARD(); @@ -XXX,XX +XXX,XX @@ static void io_writex(CPUArchState *env, CPUTLBEntryFull *full, } if (r != MEMTX_OK) { - hwaddr physaddr = mr_offset + - section->offset_within_address_space - - section->offset_within_region; - - cpu_transaction_failed(cpu, physaddr, addr, memop_size(op), - MMU_DATA_STORE, mmu_idx, full->attrs, r, - retaddr); + io_failed(env, full, addr, memop_size(op), MMU_DATA_STORE, mmu_idx, + r, retaddr, section, mr_offset); } } -- 2.34.1 Since the introduction of CPUTLBEntryFull, we can recover the full cpu address space physical address without having to examine the MemoryRegionSection. Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- accel/tcg/cputlb.c | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-) diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c index XXXXXXX..XXXXXXX 100644 --- a/accel/tcg/cputlb.c +++ b/accel/tcg/cputlb.c @@ -XXX,XX +XXX,XX @@ io_prepare(hwaddr *out_offset, CPUArchState *env, hwaddr xlat, static void io_failed(CPUArchState *env, CPUTLBEntryFull *full, vaddr addr, unsigned size, MMUAccessType access_type, int mmu_idx, - MemTxResult response, uintptr_t retaddr, - MemoryRegionSection *section, hwaddr mr_offset) + MemTxResult response, uintptr_t retaddr) { - hwaddr physaddr = (mr_offset + - section->offset_within_address_space - - section->offset_within_region); - + hwaddr physaddr = full->phys_addr | (addr & ~TARGET_PAGE_MASK); cpu_transaction_failed(env_cpu(env), physaddr, addr, size, access_type, mmu_idx, full->attrs, response, retaddr); } @@ -XXX,XX +XXX,XX @@ static uint64_t io_readx(CPUArchState *env, CPUTLBEntryFull *full, if (r != MEMTX_OK) { io_failed(env, full, addr, memop_size(op), access_type, mmu_idx, - r, retaddr, section, mr_offset); + r, retaddr); } return val; } @@ -XXX,XX +XXX,XX @@ static void io_writex(CPUArchState *env, CPUTLBEntryFull *full, if (r != MEMTX_OK) { io_failed(env, full, addr, memop_size(op), MMU_DATA_STORE, mmu_idx, - r, retaddr, section, mr_offset); + r, retaddr); } } -- 2.34.1 Rather than saving MemoryRegionSection and offset, save phys_addr and MemoryRegion. This matches up much closer with the plugin api. Reviewed-by: Alex Bennée <alex.bennee@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- include/qemu/plugin-memory.h | 11 ++--------- accel/tcg/cputlb.c | 16 +++++++++------- plugins/api.c | 27 ++++++--------------------- 3 files changed, 17 insertions(+), 37 deletions(-) diff --git a/include/qemu/plugin-memory.h b/include/qemu/plugin-memory.h index XXXXXXX..XXXXXXX 100644 --- a/include/qemu/plugin-memory.h +++ b/include/qemu/plugin-memory.h @@ -XXX,XX +XXX,XX @@ struct qemu_plugin_hwaddr { bool is_io; bool is_store; - union { - struct { - MemoryRegionSection *section; - hwaddr offset; - } io; - struct { - void *hostaddr; - } ram; - } v; + hwaddr phys_addr; + MemoryRegion *mr; }; /** diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c index XXXXXXX..XXXXXXX 100644 --- a/accel/tcg/cputlb.c +++ b/accel/tcg/cputlb.c @@ -XXX,XX +XXX,XX @@ bool tlb_plugin_lookup(CPUState *cpu, vaddr addr, int mmu_idx, uintptr_t index = tlb_index(env, mmu_idx, addr); MMUAccessType access_type = is_store ? MMU_DATA_STORE : MMU_DATA_LOAD; uint64_t tlb_addr = tlb_read_idx(tlbe, access_type); + CPUTLBEntryFull *full; if (unlikely(!tlb_hit(tlb_addr, addr))) { return false; } + full = &env_tlb(env)->d[mmu_idx].fulltlb[index]; + data->phys_addr = full->phys_addr | (addr & ~TARGET_PAGE_MASK); + /* We must have an iotlb entry for MMIO */ if (tlb_addr & TLB_MMIO) { - CPUTLBEntryFull *full = &env_tlb(env)->d[mmu_idx].fulltlb[index]; - hwaddr xlat = full->xlat_section; - + MemoryRegionSection *section = + iotlb_to_section(cpu, full->xlat_section & ~TARGET_PAGE_MASK, + full->attrs); data->is_io = true; - data->v.io.offset = (xlat & TARGET_PAGE_MASK) + addr; - data->v.io.section = - iotlb_to_section(cpu, xlat & ~TARGET_PAGE_MASK, full->attrs); + data->mr = section->mr; } else { data->is_io = false; - data->v.ram.hostaddr = (void *)((uintptr_t)addr + tlbe->addend); + data->mr = NULL; } return true; } diff --git a/plugins/api.c b/plugins/api.c index XXXXXXX..XXXXXXX 100644 --- a/plugins/api.c +++ b/plugins/api.c @@ -XXX,XX +XXX,XX @@ uint64_t qemu_plugin_hwaddr_phys_addr(const struct qemu_plugin_hwaddr *haddr) { #ifdef CONFIG_SOFTMMU if (haddr) { - if (!haddr->is_io) { - RAMBlock *block; - ram_addr_t offset; - void *hostaddr = haddr->v.ram.hostaddr; - - block = qemu_ram_block_from_host(hostaddr, false, &offset); - if (!block) { - error_report("Bad host ram pointer %p", haddr->v.ram.hostaddr); - abort(); - } - - return block->offset + offset + block->mr->addr; - } else { - MemoryRegionSection *mrs = haddr->v.io.section; - return mrs->offset_within_address_space + haddr->v.io.offset; - } + return haddr->phys_addr; } #endif return 0; @@ -XXX,XX +XXX,XX @@ const char *qemu_plugin_hwaddr_device_name(const struct qemu_plugin_hwaddr *h) { #ifdef CONFIG_SOFTMMU if (h && h->is_io) { - MemoryRegionSection *mrs = h->v.io.section; - if (!mrs->mr->name) { - unsigned long maddr = 0xffffffff & (uintptr_t) mrs->mr; - g_autofree char *temp = g_strdup_printf("anon%08lx", maddr); + MemoryRegion *mr = h->mr; + if (!mr->name) { + unsigned maddr = (uintptr_t)mr; + g_autofree char *temp = g_strdup_printf("anon%08x", maddr); return g_intern_string(temp); } else { - return g_intern_string(mrs->mr->name); + return g_intern_string(mr->name); } } else { return g_intern_static_string("RAM"); -- 2.34.1 Push computation down into the if statements to the point the data is used. Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- accel/tcg/cputlb.c | 33 +++++++++++++-------------------- 1 file changed, 13 insertions(+), 20 deletions(-) diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c index XXXXXXX..XXXXXXX 100644 --- a/accel/tcg/cputlb.c +++ b/accel/tcg/cputlb.c @@ -XXX,XX +XXX,XX @@ static inline void cpu_unaligned_access(CPUState *cpu, vaddr addr, mmu_idx, retaddr); } -static inline void cpu_transaction_failed(CPUState *cpu, hwaddr physaddr, - vaddr addr, unsigned size, - MMUAccessType access_type, - int mmu_idx, MemTxAttrs attrs, - MemTxResult response, - uintptr_t retaddr) -{ - CPUClass *cc = CPU_GET_CLASS(cpu); - - if (!cpu->ignore_memory_transaction_failures && - cc->tcg_ops->do_transaction_failed) { - cc->tcg_ops->do_transaction_failed(cpu, physaddr, addr, size, - access_type, mmu_idx, attrs, - response, retaddr); - } -} - static MemoryRegionSection * io_prepare(hwaddr *out_offset, CPUArchState *env, hwaddr xlat, MemTxAttrs attrs, vaddr addr, uintptr_t retaddr) @@ -XXX,XX +XXX,XX @@ static void io_failed(CPUArchState *env, CPUTLBEntryFull *full, vaddr addr, unsigned size, MMUAccessType access_type, int mmu_idx, MemTxResult response, uintptr_t retaddr) { - hwaddr physaddr = full->phys_addr | (addr & ~TARGET_PAGE_MASK); - cpu_transaction_failed(env_cpu(env), physaddr, addr, size, access_type, - mmu_idx, full->attrs, response, retaddr); + CPUState *cpu = env_cpu(env); + + if (!cpu->ignore_memory_transaction_failures) { + CPUClass *cc = CPU_GET_CLASS(cpu); + + if (cc->tcg_ops->do_transaction_failed) { + hwaddr physaddr = full->phys_addr | (addr & ~TARGET_PAGE_MASK); + + cc->tcg_ops->do_transaction_failed(cpu, physaddr, addr, size, + access_type, mmu_idx, + full->attrs, response, retaddr); + } + } } static uint64_t io_readx(CPUArchState *env, CPUTLBEntryFull *full, -- 2.34.1 Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- accel/tcg/cputlb.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c index XXXXXXX..XXXXXXX 100644 --- a/accel/tcg/cputlb.c +++ b/accel/tcg/cputlb.c @@ -XXX,XX +XXX,XX @@ static uint8_t do_ld_1(CPUArchState *env, MMULookupPageData *p, int mmu_idx, MMUAccessType type, uintptr_t ra) { if (unlikely(p->flags & TLB_MMIO)) { - return io_readx(env, p->full, mmu_idx, p->addr, ra, type, MO_UB); + QEMU_IOTHREAD_LOCK_GUARD(); + return do_ld_mmio_beN(env, p->full, 0, p->addr, 1, mmu_idx, type, ra); } else { return *(uint8_t *)p->haddr; } @@ -XXX,XX +XXX,XX @@ static void do_st_1(CPUArchState *env, MMULookupPageData *p, uint8_t val, int mmu_idx, uintptr_t ra) { if (unlikely(p->flags & TLB_MMIO)) { - io_writex(env, p->full, mmu_idx, val, p->addr, ra, MO_UB); + QEMU_IOTHREAD_LOCK_GUARD(); + do_st_mmio_leN(env, p->full, val, p->addr, 1, mmu_idx, ra); } else if (unlikely(p->flags & TLB_DISCARD_WRITE)) { /* nothing */ } else { -- 2.34.1 Avoid multiple calls to io_prepare for unaligned acceses. One call to do_ld_mmio_beN will never cross pages. Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- accel/tcg/cputlb.c | 84 +++++++++++++++++----------------------------- 1 file changed, 30 insertions(+), 54 deletions(-) diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c index XXXXXXX..XXXXXXX 100644 --- a/accel/tcg/cputlb.c +++ b/accel/tcg/cputlb.c @@ -XXX,XX +XXX,XX @@ static void io_failed(CPUArchState *env, CPUTLBEntryFull *full, vaddr addr, } } -static uint64_t io_readx(CPUArchState *env, CPUTLBEntryFull *full, - int mmu_idx, vaddr addr, uintptr_t retaddr, - MMUAccessType access_type, MemOp op) -{ - MemoryRegionSection *section; - hwaddr mr_offset; - MemoryRegion *mr; - MemTxResult r; - uint64_t val; - - section = io_prepare(&mr_offset, env, full->xlat_section, - full->attrs, addr, retaddr); - mr = section->mr; - - { - QEMU_IOTHREAD_LOCK_GUARD(); - r = memory_region_dispatch_read(mr, mr_offset, &val, op, full->attrs); - } - - if (r != MEMTX_OK) { - io_failed(env, full, addr, memop_size(op), access_type, mmu_idx, - r, retaddr); - } - return val; -} - static void io_writex(CPUArchState *env, CPUTLBEntryFull *full, int mmu_idx, uint64_t val, vaddr addr, uintptr_t retaddr, MemOp op) @@ -XXX,XX +XXX,XX @@ static uint64_t do_ld_mmio_beN(CPUArchState *env, CPUTLBEntryFull *full, uint64_t ret_be, vaddr addr, int size, int mmu_idx, MMUAccessType type, uintptr_t ra) { - uint64_t t; + MemoryRegionSection *section; + hwaddr mr_offset; + MemoryRegion *mr; + MemTxAttrs attrs; tcg_debug_assert(size > 0 && size <= 8); + + attrs = full->attrs; + section = io_prepare(&mr_offset, env, full->xlat_section, attrs, addr, ra); + mr = section->mr; + do { + MemOp this_mop; + unsigned this_size; + uint64_t val; + MemTxResult r; + /* Read aligned pieces up to 8 bytes. */ - switch ((size | (int)addr) & 7) { - case 1: - case 3: - case 5: - case 7: - t = io_readx(env, full, mmu_idx, addr, ra, type, MO_UB); - ret_be = (ret_be << 8) | t; - size -= 1; - addr += 1; - break; - case 2: - case 6: - t = io_readx(env, full, mmu_idx, addr, ra, type, MO_BEUW); - ret_be = (ret_be << 16) | t; - size -= 2; - addr += 2; - break; - case 4: - t = io_readx(env, full, mmu_idx, addr, ra, type, MO_BEUL); - ret_be = (ret_be << 32) | t; - size -= 4; - addr += 4; - break; - case 0: - return io_readx(env, full, mmu_idx, addr, ra, type, MO_BEUQ); - default: - qemu_build_not_reached(); + this_mop = ctz32(size | (int)addr | 8); + this_size = 1 << this_mop; + this_mop |= MO_BE; + + r = memory_region_dispatch_read(mr, mr_offset, &val, this_mop, attrs); + if (unlikely(r != MEMTX_OK)) { + io_failed(env, full, addr, this_size, type, mmu_idx, r, ra); } + if (this_size == 8) { + return val; + } + + ret_be = (ret_be << (this_size * 8)) | val; + addr += this_size; + mr_offset += this_size; + size -= this_size; } while (size); + return ret_be; } -- 2.34.1 Avoid multiple calls to io_prepare for unaligned acceses. One call to do_st_mmio_leN will never cross pages. Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- accel/tcg/cputlb.c | 82 +++++++++++++++++----------------------------- 1 file changed, 30 insertions(+), 52 deletions(-) diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c index XXXXXXX..XXXXXXX 100644 --- a/accel/tcg/cputlb.c +++ b/accel/tcg/cputlb.c @@ -XXX,XX +XXX,XX @@ static void io_failed(CPUArchState *env, CPUTLBEntryFull *full, vaddr addr, } } -static void io_writex(CPUArchState *env, CPUTLBEntryFull *full, - int mmu_idx, uint64_t val, vaddr addr, - uintptr_t retaddr, MemOp op) -{ - MemoryRegionSection *section; - hwaddr mr_offset; - MemoryRegion *mr; - MemTxResult r; - - section = io_prepare(&mr_offset, env, full->xlat_section, - full->attrs, addr, retaddr); - mr = section->mr; - - { - QEMU_IOTHREAD_LOCK_GUARD(); - r = memory_region_dispatch_write(mr, mr_offset, val, op, full->attrs); - } - - if (r != MEMTX_OK) { - io_failed(env, full, addr, memop_size(op), MMU_DATA_STORE, mmu_idx, - r, retaddr); - } -} - /* Return true if ADDR is present in the victim tlb, and has been copied back to the main tlb. */ static bool victim_tlb_hit(CPUArchState *env, size_t mmu_idx, size_t index, @@ -XXX,XX +XXX,XX @@ static uint64_t do_st_mmio_leN(CPUArchState *env, CPUTLBEntryFull *full, uint64_t val_le, vaddr addr, int size, int mmu_idx, uintptr_t ra) { + MemoryRegionSection *section; + hwaddr mr_offset; + MemoryRegion *mr; + MemTxAttrs attrs; + tcg_debug_assert(size > 0 && size <= 8); + attrs = full->attrs; + section = io_prepare(&mr_offset, env, full->xlat_section, attrs, addr, ra); + mr = section->mr; + do { + MemOp this_mop; + unsigned this_size; + MemTxResult r; + /* Store aligned pieces up to 8 bytes. */ - switch ((size | (int)addr) & 7) { - case 1: - case 3: - case 5: - case 7: - io_writex(env, full, mmu_idx, val_le, addr, ra, MO_UB); - val_le >>= 8; - size -= 1; - addr += 1; - break; - case 2: - case 6: - io_writex(env, full, mmu_idx, val_le, addr, ra, MO_LEUW); - val_le >>= 16; - size -= 2; - addr += 2; - break; - case 4: - io_writex(env, full, mmu_idx, val_le, addr, ra, MO_LEUL); - val_le >>= 32; - size -= 4; - addr += 4; - break; - case 0: - io_writex(env, full, mmu_idx, val_le, addr, ra, MO_LEUQ); - return 0; - default: - qemu_build_not_reached(); + this_mop = ctz32(size | (int)addr | 8); + this_size = 1 << this_mop; + this_mop |= MO_LE; + + r = memory_region_dispatch_write(mr, mr_offset, val_le, + this_mop, attrs); + if (unlikely(r != MEMTX_OK)) { + io_failed(env, full, addr, this_size, MMU_DATA_STORE, + mmu_idx, r, ra); } + if (this_size == 8) { + return 0; + } + + val_le >>= this_size * 8; + addr += this_size; + mr_offset += this_size; + size -= this_size; } while (size); return val_le; -- 2.34.1 Split out int_ld_mmio_beN, to be used by both do_ld_mmio_beN and do_ld16_mmio_beN. Move the locks down into the two functions, since each one now covers all accesses to once page. Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- accel/tcg/cputlb.c | 91 ++++++++++++++++++++++++++++++---------------- 1 file changed, 59 insertions(+), 32 deletions(-) diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c index XXXXXXX..XXXXXXX 100644 --- a/accel/tcg/cputlb.c +++ b/accel/tcg/cputlb.c @@ -XXX,XX +XXX,XX @@ static void *atomic_mmu_lookup(CPUArchState *env, vaddr addr, MemOpIdx oi, * Load @size bytes from @addr, which is memory-mapped i/o. * The bytes are concatenated in big-endian order with @ret_be. */ -static uint64_t do_ld_mmio_beN(CPUArchState *env, CPUTLBEntryFull *full, - uint64_t ret_be, vaddr addr, int size, - int mmu_idx, MMUAccessType type, uintptr_t ra) +static uint64_t int_ld_mmio_beN(CPUArchState *env, CPUTLBEntryFull *full, + uint64_t ret_be, vaddr addr, int size, + int mmu_idx, MMUAccessType type, uintptr_t ra, + MemoryRegion *mr, hwaddr mr_offset) { - MemoryRegionSection *section; - hwaddr mr_offset; - MemoryRegion *mr; - MemTxAttrs attrs; - - tcg_debug_assert(size > 0 && size <= 8); - - attrs = full->attrs; - section = io_prepare(&mr_offset, env, full->xlat_section, attrs, addr, ra); - mr = section->mr; - do { MemOp this_mop; unsigned this_size; @@ -XXX,XX +XXX,XX @@ static uint64_t do_ld_mmio_beN(CPUArchState *env, CPUTLBEntryFull *full, this_size = 1 << this_mop; this_mop |= MO_BE; - r = memory_region_dispatch_read(mr, mr_offset, &val, this_mop, attrs); + r = memory_region_dispatch_read(mr, mr_offset, &val, + this_mop, full->attrs); if (unlikely(r != MEMTX_OK)) { io_failed(env, full, addr, this_size, type, mmu_idx, r, ra); } @@ -XXX,XX +XXX,XX @@ static uint64_t do_ld_mmio_beN(CPUArchState *env, CPUTLBEntryFull *full, return ret_be; } +static uint64_t do_ld_mmio_beN(CPUArchState *env, CPUTLBEntryFull *full, + uint64_t ret_be, vaddr addr, int size, + int mmu_idx, MMUAccessType type, uintptr_t ra) +{ + MemoryRegionSection *section; + MemoryRegion *mr; + hwaddr mr_offset; + MemTxAttrs attrs; + uint64_t ret; + + tcg_debug_assert(size > 0 && size <= 8); + + attrs = full->attrs; + section = io_prepare(&mr_offset, env, full->xlat_section, attrs, addr, ra); + mr = section->mr; + + qemu_mutex_lock_iothread(); + ret = int_ld_mmio_beN(env, full, ret_be, addr, size, mmu_idx, + type, ra, mr, mr_offset); + qemu_mutex_unlock_iothread(); + + return ret; +} + +static Int128 do_ld16_mmio_beN(CPUArchState *env, CPUTLBEntryFull *full, + uint64_t ret_be, vaddr addr, int size, + int mmu_idx, uintptr_t ra) +{ + MemoryRegionSection *section; + MemoryRegion *mr; + hwaddr mr_offset; + MemTxAttrs attrs; + uint64_t a, b; + + tcg_debug_assert(size > 8 && size <= 16); + + attrs = full->attrs; + section = io_prepare(&mr_offset, env, full->xlat_section, attrs, addr, ra); + mr = section->mr; + + qemu_mutex_lock_iothread(); + a = int_ld_mmio_beN(env, full, ret_be, addr, size - 8, mmu_idx, + MMU_DATA_LOAD, ra, mr, mr_offset); + b = int_ld_mmio_beN(env, full, ret_be, addr + size - 8, 8, mmu_idx, + MMU_DATA_LOAD, ra, mr, mr_offset + size - 8); + qemu_mutex_unlock_iothread(); + + return int128_make128(b, a); +} + /** * do_ld_bytes_beN * @p: translation parameters @@ -XXX,XX +XXX,XX @@ static uint64_t do_ld_beN(CPUArchState *env, MMULookupPageData *p, unsigned tmp, half_size; if (unlikely(p->flags & TLB_MMIO)) { - QEMU_IOTHREAD_LOCK_GUARD(); return do_ld_mmio_beN(env, p->full, ret_be, p->addr, p->size, mmu_idx, type, ra); } @@ -XXX,XX +XXX,XX @@ static Int128 do_ld16_beN(CPUArchState *env, MMULookupPageData *p, MemOp atom; if (unlikely(p->flags & TLB_MMIO)) { - QEMU_IOTHREAD_LOCK_GUARD(); - a = do_ld_mmio_beN(env, p->full, a, p->addr, size - 8, - mmu_idx, MMU_DATA_LOAD, ra); - b = do_ld_mmio_beN(env, p->full, 0, p->addr + 8, 8, - mmu_idx, MMU_DATA_LOAD, ra); - return int128_make128(b, a); + return do_ld16_mmio_beN(env, p->full, a, p->addr, size, mmu_idx, ra); } /* @@ -XXX,XX +XXX,XX @@ static uint8_t do_ld_1(CPUArchState *env, MMULookupPageData *p, int mmu_idx, MMUAccessType type, uintptr_t ra) { if (unlikely(p->flags & TLB_MMIO)) { - QEMU_IOTHREAD_LOCK_GUARD(); return do_ld_mmio_beN(env, p->full, 0, p->addr, 1, mmu_idx, type, ra); } else { return *(uint8_t *)p->haddr; @@ -XXX,XX +XXX,XX @@ static uint16_t do_ld_2(CPUArchState *env, MMULookupPageData *p, int mmu_idx, uint16_t ret; if (unlikely(p->flags & TLB_MMIO)) { - QEMU_IOTHREAD_LOCK_GUARD(); ret = do_ld_mmio_beN(env, p->full, 0, p->addr, 2, mmu_idx, type, ra); if ((memop & MO_BSWAP) == MO_LE) { ret = bswap16(ret); @@ -XXX,XX +XXX,XX @@ static uint32_t do_ld_4(CPUArchState *env, MMULookupPageData *p, int mmu_idx, uint32_t ret; if (unlikely(p->flags & TLB_MMIO)) { - QEMU_IOTHREAD_LOCK_GUARD(); ret = do_ld_mmio_beN(env, p->full, 0, p->addr, 4, mmu_idx, type, ra); if ((memop & MO_BSWAP) == MO_LE) { ret = bswap32(ret); @@ -XXX,XX +XXX,XX @@ static uint64_t do_ld_8(CPUArchState *env, MMULookupPageData *p, int mmu_idx, uint64_t ret; if (unlikely(p->flags & TLB_MMIO)) { - QEMU_IOTHREAD_LOCK_GUARD(); ret = do_ld_mmio_beN(env, p->full, 0, p->addr, 8, mmu_idx, type, ra); if ((memop & MO_BSWAP) == MO_LE) { ret = bswap64(ret); @@ -XXX,XX +XXX,XX @@ static Int128 do_ld16_mmu(CPUArchState *env, vaddr addr, crosspage = mmu_lookup(env, addr, oi, ra, MMU_DATA_LOAD, &l); if (likely(!crosspage)) { if (unlikely(l.page[0].flags & TLB_MMIO)) { - QEMU_IOTHREAD_LOCK_GUARD(); - a = do_ld_mmio_beN(env, l.page[0].full, 0, addr, 8, - l.mmu_idx, MMU_DATA_LOAD, ra); - b = do_ld_mmio_beN(env, l.page[0].full, 0, addr + 8, 8, - l.mmu_idx, MMU_DATA_LOAD, ra); - ret = int128_make128(b, a); + ret = do_ld16_mmio_beN(env, l.page[0].full, 0, addr, 16, + l.mmu_idx, ra); if ((l.memop & MO_BSWAP) == MO_LE) { ret = bswap128(ret); } -- 2.34.1 Split out int_st_mmio_leN, to be used by both do_st_mmio_leN and do_st16_mmio_leN. Move the locks down into the two functions, since each one now covers all accesses to once page. Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- accel/tcg/cputlb.c | 88 ++++++++++++++++++++++++++++++---------------- 1 file changed, 58 insertions(+), 30 deletions(-) diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c index XXXXXXX..XXXXXXX 100644 --- a/accel/tcg/cputlb.c +++ b/accel/tcg/cputlb.c @@ -XXX,XX +XXX,XX @@ Int128 cpu_ld16_mmu(CPUArchState *env, abi_ptr addr, * The bytes to store are extracted in little-endian order from @val_le; * return the bytes of @val_le beyond @p->size that have not been stored. */ -static uint64_t do_st_mmio_leN(CPUArchState *env, CPUTLBEntryFull *full, - uint64_t val_le, vaddr addr, int size, - int mmu_idx, uintptr_t ra) +static uint64_t int_st_mmio_leN(CPUArchState *env, CPUTLBEntryFull *full, + uint64_t val_le, vaddr addr, int size, + int mmu_idx, uintptr_t ra, + MemoryRegion *mr, hwaddr mr_offset) { - MemoryRegionSection *section; - hwaddr mr_offset; - MemoryRegion *mr; - MemTxAttrs attrs; - - tcg_debug_assert(size > 0 && size <= 8); - - attrs = full->attrs; - section = io_prepare(&mr_offset, env, full->xlat_section, attrs, addr, ra); - mr = section->mr; - do { MemOp this_mop; unsigned this_size; @@ -XXX,XX +XXX,XX @@ static uint64_t do_st_mmio_leN(CPUArchState *env, CPUTLBEntryFull *full, this_mop |= MO_LE; r = memory_region_dispatch_write(mr, mr_offset, val_le, - this_mop, attrs); + this_mop, full->attrs); if (unlikely(r != MEMTX_OK)) { io_failed(env, full, addr, this_size, MMU_DATA_STORE, mmu_idx, r, ra); @@ -XXX,XX +XXX,XX @@ static uint64_t do_st_mmio_leN(CPUArchState *env, CPUTLBEntryFull *full, return val_le; } +static uint64_t do_st_mmio_leN(CPUArchState *env, CPUTLBEntryFull *full, + uint64_t val_le, vaddr addr, int size, + int mmu_idx, uintptr_t ra) +{ + MemoryRegionSection *section; + hwaddr mr_offset; + MemoryRegion *mr; + MemTxAttrs attrs; + uint64_t ret; + + tcg_debug_assert(size > 0 && size <= 8); + + attrs = full->attrs; + section = io_prepare(&mr_offset, env, full->xlat_section, attrs, addr, ra); + mr = section->mr; + + qemu_mutex_lock_iothread(); + ret = int_st_mmio_leN(env, full, val_le, addr, size, mmu_idx, + ra, mr, mr_offset); + qemu_mutex_unlock_iothread(); + + return ret; +} + +static uint64_t do_st16_mmio_leN(CPUArchState *env, CPUTLBEntryFull *full, + Int128 val_le, vaddr addr, int size, + int mmu_idx, uintptr_t ra) +{ + MemoryRegionSection *section; + MemoryRegion *mr; + hwaddr mr_offset; + MemTxAttrs attrs; + uint64_t ret; + + tcg_debug_assert(size > 8 && size <= 16); + + attrs = full->attrs; + section = io_prepare(&mr_offset, env, full->xlat_section, attrs, addr, ra); + mr = section->mr; + + qemu_mutex_lock_iothread(); + int_st_mmio_leN(env, full, int128_getlo(val_le), addr, 8, + mmu_idx, ra, mr, mr_offset); + ret = int_st_mmio_leN(env, full, int128_gethi(val_le), addr + 8, + size - 8, mmu_idx, ra, mr, mr_offset + 8); + qemu_mutex_unlock_iothread(); + + return ret; +} + /* * Wrapper for the above. */ @@ -XXX,XX +XXX,XX @@ static uint64_t do_st_leN(CPUArchState *env, MMULookupPageData *p, unsigned tmp, half_size; if (unlikely(p->flags & TLB_MMIO)) { - QEMU_IOTHREAD_LOCK_GUARD(); return do_st_mmio_leN(env, p->full, val_le, p->addr, p->size, mmu_idx, ra); } else if (unlikely(p->flags & TLB_DISCARD_WRITE)) { @@ -XXX,XX +XXX,XX @@ static uint64_t do_st16_leN(CPUArchState *env, MMULookupPageData *p, MemOp atom; if (unlikely(p->flags & TLB_MMIO)) { - QEMU_IOTHREAD_LOCK_GUARD(); - do_st_mmio_leN(env, p->full, int128_getlo(val_le), - p->addr, 8, mmu_idx, ra); - return do_st_mmio_leN(env, p->full, int128_gethi(val_le), - p->addr + 8, size - 8, mmu_idx, ra); + return do_st16_mmio_leN(env, p->full, val_le, p->addr, + size, mmu_idx, ra); } else if (unlikely(p->flags & TLB_DISCARD_WRITE)) { return int128_gethi(val_le) >> ((size - 8) * 8); } @@ -XXX,XX +XXX,XX @@ static void do_st_1(CPUArchState *env, MMULookupPageData *p, uint8_t val, int mmu_idx, uintptr_t ra) { if (unlikely(p->flags & TLB_MMIO)) { - QEMU_IOTHREAD_LOCK_GUARD(); do_st_mmio_leN(env, p->full, val, p->addr, 1, mmu_idx, ra); } else if (unlikely(p->flags & TLB_DISCARD_WRITE)) { /* nothing */ @@ -XXX,XX +XXX,XX @@ static void do_st_2(CPUArchState *env, MMULookupPageData *p, uint16_t val, if ((memop & MO_BSWAP) != MO_LE) { val = bswap16(val); } - QEMU_IOTHREAD_LOCK_GUARD(); do_st_mmio_leN(env, p->full, val, p->addr, 2, mmu_idx, ra); } else if (unlikely(p->flags & TLB_DISCARD_WRITE)) { /* nothing */ @@ -XXX,XX +XXX,XX @@ static void do_st_4(CPUArchState *env, MMULookupPageData *p, uint32_t val, if ((memop & MO_BSWAP) != MO_LE) { val = bswap32(val); } - QEMU_IOTHREAD_LOCK_GUARD(); do_st_mmio_leN(env, p->full, val, p->addr, 4, mmu_idx, ra); } else if (unlikely(p->flags & TLB_DISCARD_WRITE)) { /* nothing */ @@ -XXX,XX +XXX,XX @@ static void do_st_8(CPUArchState *env, MMULookupPageData *p, uint64_t val, if ((memop & MO_BSWAP) != MO_LE) { val = bswap64(val); } - QEMU_IOTHREAD_LOCK_GUARD(); do_st_mmio_leN(env, p->full, val, p->addr, 8, mmu_idx, ra); } else if (unlikely(p->flags & TLB_DISCARD_WRITE)) { /* nothing */ @@ -XXX,XX +XXX,XX @@ static void do_st16_mmu(CPUArchState *env, vaddr addr, Int128 val, if ((l.memop & MO_BSWAP) != MO_LE) { val = bswap128(val); } - a = int128_getlo(val); - b = int128_gethi(val); - QEMU_IOTHREAD_LOCK_GUARD(); - do_st_mmio_leN(env, l.page[0].full, a, addr, 8, l.mmu_idx, ra); - do_st_mmio_leN(env, l.page[0].full, b, addr + 8, 8, l.mmu_idx, ra); + do_st16_mmio_leN(env, l.page[0].full, val, addr, 16, l.mmu_idx, ra); } else if (unlikely(l.page[0].flags & TLB_DISCARD_WRITE)) { /* nothing */ } else { -- 2.34.1 From: LIU Zhiwei <zhiwei_liu@linux.alibaba.com> We missed these functions when upstreaming the bfloat16 support. Signed-off-by: LIU Zhiwei <zhiwei_liu@linux.alibaba.com> Message-Id: <20230531065458.2082-1-zhiwei_liu@linux.alibaba.com> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- include/fpu/softfloat.h | 12 +++++++++ fpu/softfloat.c | 58 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 70 insertions(+) diff --git a/include/fpu/softfloat.h b/include/fpu/softfloat.h index XXXXXXX..XXXXXXX 100644 --- a/include/fpu/softfloat.h +++ b/include/fpu/softfloat.h @@ -XXX,XX +XXX,XX @@ float32 bfloat16_to_float32(bfloat16, float_status *status); bfloat16 float64_to_bfloat16(float64 a, float_status *status); float64 bfloat16_to_float64(bfloat16 a, float_status *status); +int8_t bfloat16_to_int8_scalbn(bfloat16, FloatRoundMode, + int, float_status *status); int16_t bfloat16_to_int16_scalbn(bfloat16, FloatRoundMode, int, float_status *status); int32_t bfloat16_to_int32_scalbn(bfloat16, FloatRoundMode, @@ -XXX,XX +XXX,XX @@ int32_t bfloat16_to_int32_scalbn(bfloat16, FloatRoundMode, int64_t bfloat16_to_int64_scalbn(bfloat16, FloatRoundMode, int, float_status *status); +int8_t bfloat16_to_int8(bfloat16, float_status *status); int16_t bfloat16_to_int16(bfloat16, float_status *status); int32_t bfloat16_to_int32(bfloat16, float_status *status); int64_t bfloat16_to_int64(bfloat16, float_status *status); +int8_t bfloat16_to_int8_round_to_zero(bfloat16, float_status *status); int16_t bfloat16_to_int16_round_to_zero(bfloat16, float_status *status); int32_t bfloat16_to_int32_round_to_zero(bfloat16, float_status *status); int64_t bfloat16_to_int64_round_to_zero(bfloat16, float_status *status); +uint8_t bfloat16_to_uint8_scalbn(bfloat16 a, FloatRoundMode, + int, float_status *status); uint16_t bfloat16_to_uint16_scalbn(bfloat16 a, FloatRoundMode, int, float_status *status); uint32_t bfloat16_to_uint32_scalbn(bfloat16 a, FloatRoundMode, @@ -XXX,XX +XXX,XX @@ uint32_t bfloat16_to_uint32_scalbn(bfloat16 a, FloatRoundMode, uint64_t bfloat16_to_uint64_scalbn(bfloat16 a, FloatRoundMode, int, float_status *status); +uint8_t bfloat16_to_uint8(bfloat16 a, float_status *status); uint16_t bfloat16_to_uint16(bfloat16 a, float_status *status); uint32_t bfloat16_to_uint32(bfloat16 a, float_status *status); uint64_t bfloat16_to_uint64(bfloat16 a, float_status *status); +uint8_t bfloat16_to_uint8_round_to_zero(bfloat16 a, float_status *status); uint16_t bfloat16_to_uint16_round_to_zero(bfloat16 a, float_status *status); uint32_t bfloat16_to_uint32_round_to_zero(bfloat16 a, float_status *status); uint64_t bfloat16_to_uint64_round_to_zero(bfloat16 a, float_status *status); +bfloat16 int8_to_bfloat16_scalbn(int8_t a, int, float_status *status); bfloat16 int16_to_bfloat16_scalbn(int16_t a, int, float_status *status); bfloat16 int32_to_bfloat16_scalbn(int32_t a, int, float_status *status); bfloat16 int64_to_bfloat16_scalbn(int64_t a, int, float_status *status); +bfloat16 uint8_to_bfloat16_scalbn(uint8_t a, int, float_status *status); bfloat16 uint16_to_bfloat16_scalbn(uint16_t a, int, float_status *status); bfloat16 uint32_to_bfloat16_scalbn(uint32_t a, int, float_status *status); bfloat16 uint64_to_bfloat16_scalbn(uint64_t a, int, float_status *status); +bfloat16 int8_to_bfloat16(int8_t a, float_status *status); bfloat16 int16_to_bfloat16(int16_t a, float_status *status); bfloat16 int32_to_bfloat16(int32_t a, float_status *status); bfloat16 int64_to_bfloat16(int64_t a, float_status *status); +bfloat16 uint8_to_bfloat16(uint8_t a, float_status *status); bfloat16 uint16_to_bfloat16(uint16_t a, float_status *status); bfloat16 uint32_to_bfloat16(uint32_t a, float_status *status); bfloat16 uint64_to_bfloat16(uint64_t a, float_status *status); diff --git a/fpu/softfloat.c b/fpu/softfloat.c index XXXXXXX..XXXXXXX 100644 --- a/fpu/softfloat.c +++ b/fpu/softfloat.c @@ -XXX,XX +XXX,XX @@ int64_t float64_to_int64_scalbn(float64 a, FloatRoundMode rmode, int scale, return parts_float_to_sint(&p, rmode, scale, INT64_MIN, INT64_MAX, s); } +int8_t bfloat16_to_int8_scalbn(bfloat16 a, FloatRoundMode rmode, int scale, + float_status *s) +{ + FloatParts64 p; + + bfloat16_unpack_canonical(&p, a, s); + return parts_float_to_sint(&p, rmode, scale, INT8_MIN, INT8_MAX, s); +} + int16_t bfloat16_to_int16_scalbn(bfloat16 a, FloatRoundMode rmode, int scale, float_status *s) { @@ -XXX,XX +XXX,XX @@ int64_t floatx80_to_int64_round_to_zero(floatx80 a, float_status *s) return floatx80_to_int64_scalbn(a, float_round_to_zero, 0, s); } +int8_t bfloat16_to_int8(bfloat16 a, float_status *s) +{ + return bfloat16_to_int8_scalbn(a, s->float_rounding_mode, 0, s); +} + int16_t bfloat16_to_int16(bfloat16 a, float_status *s) { return bfloat16_to_int16_scalbn(a, s->float_rounding_mode, 0, s); @@ -XXX,XX +XXX,XX @@ int64_t bfloat16_to_int64(bfloat16 a, float_status *s) return bfloat16_to_int64_scalbn(a, s->float_rounding_mode, 0, s); } +int8_t bfloat16_to_int8_round_to_zero(bfloat16 a, float_status *s) +{ + return bfloat16_to_int8_scalbn(a, float_round_to_zero, 0, s); +} + int16_t bfloat16_to_int16_round_to_zero(bfloat16 a, float_status *s) { return bfloat16_to_int16_scalbn(a, float_round_to_zero, 0, s); @@ -XXX,XX +XXX,XX @@ uint64_t float64_to_uint64_scalbn(float64 a, FloatRoundMode rmode, int scale, return parts_float_to_uint(&p, rmode, scale, UINT64_MAX, s); } +uint8_t bfloat16_to_uint8_scalbn(bfloat16 a, FloatRoundMode rmode, + int scale, float_status *s) +{ + FloatParts64 p; + + bfloat16_unpack_canonical(&p, a, s); + return parts_float_to_uint(&p, rmode, scale, UINT8_MAX, s); +} + uint16_t bfloat16_to_uint16_scalbn(bfloat16 a, FloatRoundMode rmode, int scale, float_status *s) { @@ -XXX,XX +XXX,XX @@ Int128 float128_to_uint128_round_to_zero(float128 a, float_status *s) return float128_to_uint128_scalbn(a, float_round_to_zero, 0, s); } +uint8_t bfloat16_to_uint8(bfloat16 a, float_status *s) +{ + return bfloat16_to_uint8_scalbn(a, s->float_rounding_mode, 0, s); +} + uint16_t bfloat16_to_uint16(bfloat16 a, float_status *s) { return bfloat16_to_uint16_scalbn(a, s->float_rounding_mode, 0, s); @@ -XXX,XX +XXX,XX @@ uint64_t bfloat16_to_uint64(bfloat16 a, float_status *s) return bfloat16_to_uint64_scalbn(a, s->float_rounding_mode, 0, s); } +uint8_t bfloat16_to_uint8_round_to_zero(bfloat16 a, float_status *s) +{ + return bfloat16_to_uint8_scalbn(a, float_round_to_zero, 0, s); +} + uint16_t bfloat16_to_uint16_round_to_zero(bfloat16 a, float_status *s) { return bfloat16_to_uint16_scalbn(a, float_round_to_zero, 0, s); @@ -XXX,XX +XXX,XX @@ bfloat16 int16_to_bfloat16_scalbn(int16_t a, int scale, float_status *status) return int64_to_bfloat16_scalbn(a, scale, status); } +bfloat16 int8_to_bfloat16_scalbn(int8_t a, int scale, float_status *status) +{ + return int64_to_bfloat16_scalbn(a, scale, status); +} + bfloat16 int64_to_bfloat16(int64_t a, float_status *status) { return int64_to_bfloat16_scalbn(a, 0, status); @@ -XXX,XX +XXX,XX @@ bfloat16 int16_to_bfloat16(int16_t a, float_status *status) return int64_to_bfloat16_scalbn(a, 0, status); } +bfloat16 int8_to_bfloat16(int8_t a, float_status *status) +{ + return int64_to_bfloat16_scalbn(a, 0, status); +} + float128 int128_to_float128(Int128 a, float_status *status) { FloatParts128 p = { }; @@ -XXX,XX +XXX,XX @@ bfloat16 uint16_to_bfloat16_scalbn(uint16_t a, int scale, float_status *status) return uint64_to_bfloat16_scalbn(a, scale, status); } +bfloat16 uint8_to_bfloat16_scalbn(uint8_t a, int scale, float_status *status) +{ + return uint64_to_bfloat16_scalbn(a, scale, status); +} + bfloat16 uint64_to_bfloat16(uint64_t a, float_status *status) { return uint64_to_bfloat16_scalbn(a, 0, status); @@ -XXX,XX +XXX,XX @@ bfloat16 uint16_to_bfloat16(uint16_t a, float_status *status) return uint64_to_bfloat16_scalbn(a, 0, status); } +bfloat16 uint8_to_bfloat16(uint8_t a, float_status *status) +{ + return uint64_to_bfloat16_scalbn(a, 0, status); +} + float128 uint64_to_float128(uint64_t a, float_status *status) { FloatParts128 p; -- 2.34.1 Motorola treats denormals with explicit integer bit set as having unbiased exponent 0, unlike Intel which treats it as having unbiased exponent 1 (more like all other IEEE formats that have no explicit integer bit). Add a flag on FloatFmt to differentiate the behaviour. Reported-by: Keith Packard <keithp@keithp.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- fpu/softfloat.c | 9 +++++- tests/tcg/m68k/denormal.c | 53 ++++++++++++++++++++++++++++++++++ fpu/softfloat-parts.c.inc | 7 +++-- tests/tcg/m68k/Makefile.target | 2 +- 4 files changed, 66 insertions(+), 5 deletions(-) create mode 100644 tests/tcg/m68k/denormal.c diff --git a/fpu/softfloat.c b/fpu/softfloat.c index XXXXXXX..XXXXXXX 100644 --- a/fpu/softfloat.c +++ b/fpu/softfloat.c @@ -XXX,XX +XXX,XX @@ typedef struct { * round_mask: bits below lsb which must be rounded * The following optional modifiers are available: * arm_althp: handle ARM Alternative Half Precision + * m68k_denormal: explicit integer bit for extended precision may be 1 */ typedef struct { int exp_size; @@ -XXX,XX +XXX,XX @@ typedef struct { int frac_size; int frac_shift; bool arm_althp; + bool m68k_denormal; uint64_t round_mask; } FloatFmt; @@ -XXX,XX +XXX,XX @@ static const FloatFmt float128_params = { static const FloatFmt floatx80_params[3] = { [floatx80_precision_s] = { FLOATX80_PARAMS(23) }, [floatx80_precision_d] = { FLOATX80_PARAMS(52) }, - [floatx80_precision_x] = { FLOATX80_PARAMS(64) }, + [floatx80_precision_x] = { + FLOATX80_PARAMS(64), +#ifdef TARGET_M68K + .m68k_denormal = true, +#endif + }, }; /* Unpack a float to parts, but do not canonicalize. */ diff --git a/tests/tcg/m68k/denormal.c b/tests/tcg/m68k/denormal.c new file mode 100644 index XXXXXXX..XXXXXXX --- /dev/null +++ b/tests/tcg/m68k/denormal.c @@ -XXX,XX +XXX,XX @@ +/* + * Test m68k extended double denormals. + */ + +#include <stdio.h> +#include <stdint.h> + +#define TEST(X, Y) { X, Y, X * Y } + +static volatile long double test[][3] = { + TEST(0x1p+16383l, 0x1p-16446l), + TEST(0x1.1p-8223l, 0x1.1p-8224l), + TEST(1.0l, 0x1p-16383l), +}; + +#undef TEST + +static void dump_ld(const char *label, long double ld) +{ + union { + long double d; + struct { + uint32_t exp:16; + uint32_t space:16; + uint32_t h; + uint32_t l; + }; + } u; + + u.d = ld; + printf("%12s: % -27La 0x%04x 0x%08x 0x%08x\n", label, u.d, u.exp, u.h, u.l); +} + +int main(void) +{ + int i, n = sizeof(test) / sizeof(test[0]), err = 0; + + for (i = 0; i < n; ++i) { + long double x = test[i][0]; + long double y = test[i][1]; + long double build_mul = test[i][2]; + long double runtime_mul = x * y; + + if (runtime_mul != build_mul) { + dump_ld("x", x); + dump_ld("y", y); + dump_ld("build_mul", build_mul); + dump_ld("runtime_mul", runtime_mul); + err = 1; + } + } + return err; +} diff --git a/fpu/softfloat-parts.c.inc b/fpu/softfloat-parts.c.inc index XXXXXXX..XXXXXXX 100644 --- a/fpu/softfloat-parts.c.inc +++ b/fpu/softfloat-parts.c.inc @@ -XXX,XX +XXX,XX @@ static void partsN(canonicalize)(FloatPartsN *p, float_status *status, } else { int shift = frac_normalize(p); p->cls = float_class_normal; - p->exp = fmt->frac_shift - fmt->exp_bias - shift + 1; + p->exp = fmt->frac_shift - fmt->exp_bias + - shift + !fmt->m68k_denormal; } } else if (likely(p->exp < fmt->exp_max) || fmt->arm_althp) { p->cls = float_class_normal; @@ -XXX,XX +XXX,XX @@ static void partsN(uncanon_normal)(FloatPartsN *p, float_status *s, is_tiny = !frac_addi(&discard, p, inc); } - frac_shrjam(p, 1 - exp); + frac_shrjam(p, !fmt->m68k_denormal - exp); if (p->frac_lo & round_mask) { /* Need to recompute round-to-even/round-to-odd. */ @@ -XXX,XX +XXX,XX @@ static void partsN(uncanon_normal)(FloatPartsN *p, float_status *s, p->frac_lo &= ~round_mask; } - exp = (p->frac_hi & DECOMPOSED_IMPLICIT_BIT) != 0; + exp = (p->frac_hi & DECOMPOSED_IMPLICIT_BIT) && !fmt->m68k_denormal; frac_shr(p, frac_shift); if (is_tiny && (flags & float_flag_inexact)) { diff --git a/tests/tcg/m68k/Makefile.target b/tests/tcg/m68k/Makefile.target index XXXXXXX..XXXXXXX 100644 --- a/tests/tcg/m68k/Makefile.target +++ b/tests/tcg/m68k/Makefile.target @@ -XXX,XX +XXX,XX @@ # VPATH += $(SRC_PATH)/tests/tcg/m68k -TESTS += trap +TESTS += trap denormal # On m68k Linux supports 4k and 8k pages (but 8k is currently broken) EXTRA_RUNS+=run-test-mmap-4096 # run-test-mmap-8192 -- 2.34.1 This hook may emit code at the beginning of the TB. Suggested-by: Jordan Niethe <jniethe5@gmail.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- tcg/tcg.c | 3 +++ tcg/aarch64/tcg-target.c.inc | 5 +++++ tcg/arm/tcg-target.c.inc | 5 +++++ tcg/i386/tcg-target.c.inc | 5 +++++ tcg/loongarch64/tcg-target.c.inc | 5 +++++ tcg/mips/tcg-target.c.inc | 5 +++++ tcg/ppc/tcg-target.c.inc | 5 +++++ tcg/riscv/tcg-target.c.inc | 5 +++++ tcg/s390x/tcg-target.c.inc | 5 +++++ tcg/sparc64/tcg-target.c.inc | 5 +++++ tcg/tci/tcg-target.c.inc | 5 +++++ 11 files changed, 53 insertions(+) diff --git a/tcg/tcg.c b/tcg/tcg.c index XXXXXXX..XXXXXXX 100644 --- a/tcg/tcg.c +++ b/tcg/tcg.c @@ -XXX,XX +XXX,XX @@ static void tcg_register_jit_int(const void *buf, size_t size, __attribute__((unused)); /* Forward declarations for functions declared and used in tcg-target.c.inc. */ +static void tcg_out_tb_start(TCGContext *s); static void tcg_out_ld(TCGContext *s, TCGType type, TCGReg ret, TCGReg arg1, intptr_t arg2); static bool tcg_out_mov(TCGContext *s, TCGType type, TCGReg ret, TCGReg arg); @@ -XXX,XX +XXX,XX @@ int tcg_gen_code(TCGContext *s, TranslationBlock *tb, uint64_t pc_start) s->gen_insn_data = tcg_malloc(sizeof(uint64_t) * s->gen_tb->icount * start_words); + tcg_out_tb_start(s); + num_insns = -1; QTAILQ_FOREACH(op, &s->ops, link) { TCGOpcode opc = op->opc; diff --git a/tcg/aarch64/tcg-target.c.inc b/tcg/aarch64/tcg-target.c.inc index XXXXXXX..XXXXXXX 100644 --- a/tcg/aarch64/tcg-target.c.inc +++ b/tcg/aarch64/tcg-target.c.inc @@ -XXX,XX +XXX,XX @@ static void tcg_target_qemu_prologue(TCGContext *s) tcg_out_insn(s, 3207, RET, TCG_REG_LR); } +static void tcg_out_tb_start(TCGContext *s) +{ + /* nothing to do */ +} + static void tcg_out_nop_fill(tcg_insn_unit *p, int count) { int i; diff --git a/tcg/arm/tcg-target.c.inc b/tcg/arm/tcg-target.c.inc index XXXXXXX..XXXXXXX 100644 --- a/tcg/arm/tcg-target.c.inc +++ b/tcg/arm/tcg-target.c.inc @@ -XXX,XX +XXX,XX @@ static void tcg_out_epilogue(TCGContext *s) (1 << TCG_REG_R10) | (1 << TCG_REG_R11) | (1 << TCG_REG_PC)); } +static void tcg_out_tb_start(TCGContext *s) +{ + /* nothing to do */ +} + typedef struct { DebugFrameHeader h; uint8_t fde_def_cfa[4]; diff --git a/tcg/i386/tcg-target.c.inc b/tcg/i386/tcg-target.c.inc index XXXXXXX..XXXXXXX 100644 --- a/tcg/i386/tcg-target.c.inc +++ b/tcg/i386/tcg-target.c.inc @@ -XXX,XX +XXX,XX @@ static void tcg_target_qemu_prologue(TCGContext *s) tcg_out_opc(s, OPC_RET, 0, 0, 0); } +static void tcg_out_tb_start(TCGContext *s) +{ + /* nothing to do */ +} + static void tcg_out_nop_fill(tcg_insn_unit *p, int count) { memset(p, 0x90, count); diff --git a/tcg/loongarch64/tcg-target.c.inc b/tcg/loongarch64/tcg-target.c.inc index XXXXXXX..XXXXXXX 100644 --- a/tcg/loongarch64/tcg-target.c.inc +++ b/tcg/loongarch64/tcg-target.c.inc @@ -XXX,XX +XXX,XX @@ static void tcg_target_qemu_prologue(TCGContext *s) tcg_out_opc_jirl(s, TCG_REG_ZERO, TCG_REG_RA, 0); } +static void tcg_out_tb_start(TCGContext *s) +{ + /* nothing to do */ +} + static void tcg_target_init(TCGContext *s) { unsigned long hwcap = qemu_getauxval(AT_HWCAP); diff --git a/tcg/mips/tcg-target.c.inc b/tcg/mips/tcg-target.c.inc index XXXXXXX..XXXXXXX 100644 --- a/tcg/mips/tcg-target.c.inc +++ b/tcg/mips/tcg-target.c.inc @@ -XXX,XX +XXX,XX @@ static void tcg_target_qemu_prologue(TCGContext *s) tcg_out_opc_reg(s, OPC_OR, TCG_TMP3, TCG_TMP3, TCG_TMP1); } +static void tcg_out_tb_start(TCGContext *s) +{ + /* nothing to do */ +} + static void tcg_target_init(TCGContext *s) { tcg_target_detect_isa(); diff --git a/tcg/ppc/tcg-target.c.inc b/tcg/ppc/tcg-target.c.inc index XXXXXXX..XXXXXXX 100644 --- a/tcg/ppc/tcg-target.c.inc +++ b/tcg/ppc/tcg-target.c.inc @@ -XXX,XX +XXX,XX @@ static void tcg_target_qemu_prologue(TCGContext *s) tcg_out32(s, BCLR | BO_ALWAYS); } +static void tcg_out_tb_start(TCGContext *s) +{ + /* nothing to do */ +} + static void tcg_out_exit_tb(TCGContext *s, uintptr_t arg) { tcg_out_movi(s, TCG_TYPE_PTR, TCG_REG_R3, arg); diff --git a/tcg/riscv/tcg-target.c.inc b/tcg/riscv/tcg-target.c.inc index XXXXXXX..XXXXXXX 100644 --- a/tcg/riscv/tcg-target.c.inc +++ b/tcg/riscv/tcg-target.c.inc @@ -XXX,XX +XXX,XX @@ static void tcg_target_qemu_prologue(TCGContext *s) tcg_out_opc_imm(s, OPC_JALR, TCG_REG_ZERO, TCG_REG_RA, 0); } +static void tcg_out_tb_start(TCGContext *s) +{ + /* nothing to do */ +} + static volatile sig_atomic_t got_sigill; static void sigill_handler(int signo, siginfo_t *si, void *data) diff --git a/tcg/s390x/tcg-target.c.inc b/tcg/s390x/tcg-target.c.inc index XXXXXXX..XXXXXXX 100644 --- a/tcg/s390x/tcg-target.c.inc +++ b/tcg/s390x/tcg-target.c.inc @@ -XXX,XX +XXX,XX @@ static void tcg_target_qemu_prologue(TCGContext *s) tcg_out_insn(s, RR, BCR, S390_CC_ALWAYS, TCG_REG_R14); } +static void tcg_out_tb_start(TCGContext *s) +{ + /* nothing to do */ +} + static void tcg_out_nop_fill(tcg_insn_unit *p, int count) { memset(p, 0x07, count * sizeof(tcg_insn_unit)); diff --git a/tcg/sparc64/tcg-target.c.inc b/tcg/sparc64/tcg-target.c.inc index XXXXXXX..XXXXXXX 100644 --- a/tcg/sparc64/tcg-target.c.inc +++ b/tcg/sparc64/tcg-target.c.inc @@ -XXX,XX +XXX,XX @@ static void tcg_target_qemu_prologue(TCGContext *s) tcg_out_movi_s13(s, TCG_REG_O0, 0); } +static void tcg_out_tb_start(TCGContext *s) +{ + /* nothing to do */ +} + static void tcg_out_nop_fill(tcg_insn_unit *p, int count) { int i; diff --git a/tcg/tci/tcg-target.c.inc b/tcg/tci/tcg-target.c.inc index XXXXXXX..XXXXXXX 100644 --- a/tcg/tci/tcg-target.c.inc +++ b/tcg/tci/tcg-target.c.inc @@ -XXX,XX +XXX,XX @@ static inline void tcg_target_qemu_prologue(TCGContext *s) { } +static void tcg_out_tb_start(TCGContext *s) +{ + /* nothing to do */ +} + bool tcg_target_has_memory_bswap(MemOp memop) { return true; -- 2.34.1 Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- host/include/aarch64/host/cpuinfo.h | 1 + util/cpuinfo-aarch64.c | 7 +++++++ 2 files changed, 8 insertions(+) diff --git a/host/include/aarch64/host/cpuinfo.h b/host/include/aarch64/host/cpuinfo.h index XXXXXXX..XXXXXXX 100644 --- a/host/include/aarch64/host/cpuinfo.h +++ b/host/include/aarch64/host/cpuinfo.h @@ -XXX,XX +XXX,XX @@ #define CPUINFO_LSE (1u << 1) #define CPUINFO_LSE2 (1u << 2) #define CPUINFO_AES (1u << 3) +#define CPUINFO_BTI (1u << 4) /* Initialized with a constructor. */ extern unsigned cpuinfo; diff --git a/util/cpuinfo-aarch64.c b/util/cpuinfo-aarch64.c index XXXXXXX..XXXXXXX 100644 --- a/util/cpuinfo-aarch64.c +++ b/util/cpuinfo-aarch64.c @@ -XXX,XX +XXX,XX @@ # include <asm/hwcap.h> # include "elf.h" # endif +# ifndef HWCAP2_BTI +# define HWCAP2_BTI 0 /* added in glibc 2.32 */ +# endif #endif #ifdef CONFIG_DARWIN # include <sys/sysctl.h> @@ -XXX,XX +XXX,XX @@ unsigned __attribute__((constructor)) cpuinfo_init(void) info |= (hwcap & HWCAP_ATOMICS ? CPUINFO_LSE : 0); info |= (hwcap & HWCAP_USCAT ? CPUINFO_LSE2 : 0); info |= (hwcap & HWCAP_AES ? CPUINFO_AES: 0); + + unsigned long hwcap2 = qemu_getauxval(AT_HWCAP2); + info |= (hwcap2 & HWCAP2_BTI ? CPUINFO_BTI : 0); #endif #ifdef CONFIG_DARWIN info |= sysctl_for_bool("hw.optional.arm.FEAT_LSE") * CPUINFO_LSE; info |= sysctl_for_bool("hw.optional.arm.FEAT_LSE2") * CPUINFO_LSE2; info |= sysctl_for_bool("hw.optional.arm.FEAT_AES") * CPUINFO_AES; + info |= sysctl_for_bool("hw.optional.arm.FEAT_BTI") * CPUINFO_BTI; #endif cpuinfo = info; -- 2.34.1 The prologue is entered via "call"; the epilogue, each tb, and each goto_tb continuation point are all reached via "jump". As tcg_out_goto_long is only used by tcg_out_exit_tb, merge the two functions. Change the indirect register used to TCG_REG_TMP1, aka X17, so that the BTI condition created is "jump" instead of "jump or call". Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- tcg/aarch64/tcg-target.c.inc | 54 ++++++++++++++++++++++++++---------- 1 file changed, 39 insertions(+), 15 deletions(-) diff --git a/tcg/aarch64/tcg-target.c.inc b/tcg/aarch64/tcg-target.c.inc index XXXXXXX..XXXXXXX 100644 --- a/tcg/aarch64/tcg-target.c.inc +++ b/tcg/aarch64/tcg-target.c.inc @@ -XXX,XX +XXX,XX @@ typedef enum { DMB_ISH = 0xd50338bf, DMB_LD = 0x00000100, DMB_ST = 0x00000200, + + BTI_C = 0xd503245f, + BTI_J = 0xd503249f, + BTI_JC = 0xd50324df, } AArch64Insn; static inline uint32_t tcg_in32(TCGContext *s) @@ -XXX,XX +XXX,XX @@ static void tcg_out_insn_3313(TCGContext *s, AArch64Insn insn, | rn << 5 | (rd & 0x1f)); } +static void tcg_out_bti(TCGContext *s, AArch64Insn insn) +{ + /* + * While BTI insns are nops on hosts without FEAT_BTI, + * there is no point in emitting them in that case either. + */ + if (cpuinfo & CPUINFO_BTI) { + tcg_out32(s, insn); + } +} + /* Register to register move using ORR (shifted register with no shift). */ static void tcg_out_movr(TCGContext *s, TCGType ext, TCGReg rd, TCGReg rm) { @@ -XXX,XX +XXX,XX @@ static void tcg_out_goto(TCGContext *s, const tcg_insn_unit *target) tcg_out_insn(s, 3206, B, offset); } -static void tcg_out_goto_long(TCGContext *s, const tcg_insn_unit *target) -{ - ptrdiff_t offset = tcg_pcrel_diff(s, target) >> 2; - if (offset == sextract64(offset, 0, 26)) { - tcg_out_insn(s, 3206, B, offset); - } else { - /* Choose X9 as a call-clobbered non-LR temporary. */ - tcg_out_movi(s, TCG_TYPE_I64, TCG_REG_X9, (intptr_t)target); - tcg_out_insn(s, 3207, BR, TCG_REG_X9); - } -} - static void tcg_out_call_int(TCGContext *s, const tcg_insn_unit *target) { ptrdiff_t offset = tcg_pcrel_diff(s, target) >> 2; @@ -XXX,XX +XXX,XX @@ static const tcg_insn_unit *tb_ret_addr; static void tcg_out_exit_tb(TCGContext *s, uintptr_t a0) { + const tcg_insn_unit *target; + ptrdiff_t offset; + /* Reuse the zeroing that exists for goto_ptr. */ if (a0 == 0) { - tcg_out_goto_long(s, tcg_code_gen_epilogue); + target = tcg_code_gen_epilogue; } else { tcg_out_movi(s, TCG_TYPE_I64, TCG_REG_X0, a0); - tcg_out_goto_long(s, tb_ret_addr); + target = tb_ret_addr; + } + + offset = tcg_pcrel_diff(s, target) >> 2; + if (offset == sextract64(offset, 0, 26)) { + tcg_out_insn(s, 3206, B, offset); + } else { + /* + * Only x16/x17 generate BTI type Jump (2), + * other registers generate BTI type Jump|Call (3). + */ + QEMU_BUILD_BUG_ON(TCG_REG_TMP0 != TCG_REG_X16); + tcg_out_movi(s, TCG_TYPE_I64, TCG_REG_TMP0, (intptr_t)target); + tcg_out_insn(s, 3207, BR, TCG_REG_TMP0); } } @@ -XXX,XX +XXX,XX @@ static void tcg_out_goto_tb(TCGContext *s, int which) tcg_out32(s, I3206_B); tcg_out_insn(s, 3207, BR, TCG_REG_TMP0); set_jmp_reset_offset(s, which); + tcg_out_bti(s, BTI_J); } void tb_target_set_jmp_target(const TranslationBlock *tb, int n, @@ -XXX,XX +XXX,XX @@ static void tcg_target_qemu_prologue(TCGContext *s) { TCGReg r; + tcg_out_bti(s, BTI_C); + /* Push (FP, LR) and allocate space for all saved registers. */ tcg_out_insn(s, 3314, STP, TCG_REG_FP, TCG_REG_LR, TCG_REG_SP, -PUSH_SIZE, 1, 1); @@ -XXX,XX +XXX,XX @@ static void tcg_target_qemu_prologue(TCGContext *s) * and fall through to the rest of the epilogue. */ tcg_code_gen_epilogue = tcg_splitwx_to_rx(s->code_ptr); + tcg_out_bti(s, BTI_J); tcg_out_movi(s, TCG_TYPE_REG, TCG_REG_X0, 0); /* TB epilogue */ tb_ret_addr = tcg_splitwx_to_rx(s->code_ptr); + tcg_out_bti(s, BTI_J); /* Remove TCG locals stack space. */ tcg_out_insn(s, 3401, ADDI, TCG_TYPE_I64, TCG_REG_SP, TCG_REG_SP, @@ -XXX,XX +XXX,XX @@ static void tcg_target_qemu_prologue(TCGContext *s) static void tcg_out_tb_start(TCGContext *s) { - /* nothing to do */ + tcg_out_bti(s, BTI_J); } static void tcg_out_nop_fill(tcg_insn_unit *p, int count) -- 2.34.1 For linux aarch64 host supporting BTI, map the buffer to require BTI instructions at branch landing pads. Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- tcg/region.c | 41 ++++++++++++++++++++++++++++++----------- 1 file changed, 30 insertions(+), 11 deletions(-) diff --git a/tcg/region.c b/tcg/region.c index XXXXXXX..XXXXXXX 100644 --- a/tcg/region.c +++ b/tcg/region.c @@ -XXX,XX +XXX,XX @@ #include "tcg/tcg.h" #include "exec/translation-block.h" #include "tcg-internal.h" +#include "host/cpuinfo.h" +/* + * Local source-level compatibility with Unix. + * Used by tcg_region_init below. + */ +#if defined(_WIN32) +#define PROT_READ 1 +#define PROT_WRITE 2 +#define PROT_EXEC 4 +#endif + struct tcg_region_tree { QemuMutex lock; QTree *tree; @@ -XXX,XX +XXX,XX @@ bool in_code_gen_buffer(const void *p) return (size_t)(p - region.start_aligned) <= region.total_size; } +#ifndef CONFIG_TCG_INTERPRETER +static int host_prot_read_exec(void) +{ +#if defined(CONFIG_LINUX) && defined(HOST_AARCH64) && defined(PROT_BTI) + if (cpuinfo & CPUINFO_BTI) { + return PROT_READ | PROT_EXEC | PROT_BTI; + } +#endif + return PROT_READ | PROT_EXEC; +} +#endif + #ifdef CONFIG_DEBUG_TCG const void *tcg_splitwx_to_rx(void *rw) { @@ -XXX,XX +XXX,XX @@ static int alloc_code_gen_buffer(size_t tb_size, int splitwx, Error **errp) return PROT_READ | PROT_WRITE; } #elif defined(_WIN32) -/* - * Local source-level compatibility with Unix. - * Used by tcg_region_init below. - */ -#define PROT_READ 1 -#define PROT_WRITE 2 -#define PROT_EXEC 4 - static int alloc_code_gen_buffer(size_t size, int splitwx, Error **errp) { void *buf; @@ -XXX,XX +XXX,XX @@ static int alloc_code_gen_buffer_splitwx_memfd(size_t size, Error **errp) goto fail; } - buf_rx = mmap(NULL, size, PROT_READ | PROT_EXEC, MAP_SHARED, fd, 0); + buf_rx = mmap(NULL, size, host_prot_read_exec(), MAP_SHARED, fd, 0); if (buf_rx == MAP_FAILED) { goto fail_rx; } @@ -XXX,XX +XXX,XX @@ static int alloc_code_gen_buffer_splitwx_vmremap(size_t size, Error **errp) return -1; } - if (mprotect((void *)buf_rx, size, PROT_READ | PROT_EXEC) != 0) { + if (mprotect((void *)buf_rx, size, host_prot_read_exec()) != 0) { error_setg_errno(errp, errno, "mprotect for jit splitwx"); munmap((void *)buf_rx, size); munmap((void *)buf_rw, size); @@ -XXX,XX +XXX,XX @@ void tcg_region_init(size_t tb_size, int splitwx, unsigned max_cpus) need_prot = PROT_READ | PROT_WRITE; #ifndef CONFIG_TCG_INTERPRETER if (tcg_splitwx_diff == 0) { - need_prot |= PROT_EXEC; + need_prot |= host_prot_read_exec(); } #endif for (size_t i = 0, n = region.n; i < n; i++) { @@ -XXX,XX +XXX,XX @@ void tcg_region_init(size_t tb_size, int splitwx, unsigned max_cpus) } else if (need_prot == (PROT_READ | PROT_WRITE)) { rc = qemu_mprotect_rw(start, end - start); } else { +#ifdef CONFIG_POSIX + rc = mprotect(start, end - start, need_prot); +#else g_assert_not_reached(); +#endif } if (rc) { error_setg_errno(&error_fatal, errno, -- 2.34.1 The following changes since commit aa3a285b5bc56a4208b3b57d4a55291e9c260107: Merge tag 'mem-2024-12-21' of https://github.com/davidhildenbrand/qemu into staging (2024-12-22 14:33:27 -0500) are available in the Git repository at: https://gitlab.com/rth7680/qemu.git tags/pull-tcg-20241224 for you to fetch changes up to e4a8e093dc74be049f4829831dce76e5edab0003: accel/tcg: Move gen_intermediate_code to TCGCPUOps.translate_core (2024-12-24 08:32:15 -0800) ---------------------------------------------------------------- tcg/optimize: Remove in-flight mask data from OptContext fpu: Add float*_muladd_scalbn fpu: Remove float_muladd_halve_result fpu: Add float_round_nearest_even_max fpu: Add float_muladd_suppress_add_product_zero target/hexagon: Use float32_muladd accel/tcg: Move gen_intermediate_code to TCGCPUOps.translate_core ---------------------------------------------------------------- Ilya Leoshkevich (1): tests/tcg: Do not use inttypes.h in multiarch/system/memory.c Pierrick Bouvier (1): plugins: optimize cpu_index code generation Richard Henderson (70): tcg/optimize: Split out finish_bb, finish_ebb tcg/optimize: Split out fold_affected_mask tcg/optimize: Copy mask writeback to fold_masks tcg/optimize: Split out fold_masks_zs tcg/optimize: Augment s_mask from z_mask in fold_masks_zs tcg/optimize: Change representation of s_mask tcg/optimize: Use finish_folding in fold_add, fold_add_vec, fold_addsub2 tcg/optimize: Introduce const value accessors for TempOptInfo tcg/optimize: Use fold_masks_zs in fold_and tcg/optimize: Use fold_masks_zs in fold_andc tcg/optimize: Use fold_masks_zs in fold_bswap tcg/optimize: Use fold_masks_zs in fold_count_zeros tcg/optimize: Use fold_masks_z in fold_ctpop tcg/optimize: Use fold_and and fold_masks_z in fold_deposit tcg/optimize: Compute sign mask in fold_deposit tcg/optimize: Use finish_folding in fold_divide tcg/optimize: Use finish_folding in fold_dup, fold_dup2 tcg/optimize: Use fold_masks_s in fold_eqv tcg/optimize: Use fold_masks_z in fold_extract tcg/optimize: Use finish_folding in fold_extract2 tcg/optimize: Use fold_masks_zs in fold_exts tcg/optimize: Use fold_masks_z in fold_extu tcg/optimize: Use fold_masks_zs in fold_movcond tcg/optimize: Use finish_folding in fold_mul* tcg/optimize: Use fold_masks_s in fold_nand tcg/optimize: Use fold_masks_z in fold_neg_no_const tcg/optimize: Use fold_masks_s in fold_nor tcg/optimize: Use fold_masks_s in fold_not tcg/optimize: Use fold_masks_zs in fold_or tcg/optimize: Use fold_masks_zs in fold_orc tcg/optimize: Use fold_masks_zs in fold_qemu_ld tcg/optimize: Return true from fold_qemu_st, fold_tcg_st tcg/optimize: Use finish_folding in fold_remainder tcg/optimize: Distinguish simplification in fold_setcond_zmask tcg/optimize: Use fold_masks_z in fold_setcond tcg/optimize: Use fold_masks_s in fold_negsetcond tcg/optimize: Use fold_masks_z in fold_setcond2 tcg/optimize: Use finish_folding in fold_cmp_vec tcg/optimize: Use finish_folding in fold_cmpsel_vec tcg/optimize: Use fold_masks_zs in fold_sextract tcg/optimize: Use fold_masks_zs, fold_masks_s in fold_shift tcg/optimize: Simplify sign bit test in fold_shift tcg/optimize: Use finish_folding in fold_sub, fold_sub_vec tcg/optimize: Use fold_masks_zs in fold_tcg_ld tcg/optimize: Use finish_folding in fold_tcg_ld_memcopy tcg/optimize: Use fold_masks_zs in fold_xor tcg/optimize: Use finish_folding in fold_bitsel_vec tcg/optimize: Use finish_folding as default in tcg_optimize tcg/optimize: Remove z_mask, s_mask from OptContext tcg/optimize: Re-enable sign-mask optimizations tcg/optimize: Move fold_bitsel_vec into alphabetic sort tcg/optimize: Move fold_cmp_vec, fold_cmpsel_vec into alphabetic sort softfloat: Add float{16,32,64}_muladd_scalbn target/arm: Use float*_muladd_scalbn target/sparc: Use float*_muladd_scalbn softfloat: Remove float_muladd_halve_result softfloat: Add float_round_nearest_even_max softfloat: Add float_muladd_suppress_add_product_zero target/hexagon: Use float32_mul in helper_sfmpy target/hexagon: Use float32_muladd for helper_sffma target/hexagon: Use float32_muladd for helper_sffms target/hexagon: Use float32_muladd_scalbn for helper_sffma_sc target/hexagon: Use float32_muladd for helper_sffm[as]_lib target/hexagon: Remove internal_fmafx target/hexagon: Expand GEN_XF_ROUND target/hexagon: Remove Float target/hexagon: Remove Double target/hexagon: Use mulu64 for int128_mul_6464 target/hexagon: Simplify internal_mpyhh setup accel/tcg: Move gen_intermediate_code to TCGCPUOps.translate_core include/exec/translator.h | 14 - include/fpu/softfloat-types.h | 2 + include/fpu/softfloat.h | 14 +- include/hw/core/tcg-cpu-ops.h | 13 + target/alpha/cpu.h | 2 + target/arm/internals.h | 2 + target/avr/cpu.h | 2 + target/hexagon/cpu.h | 2 + target/hexagon/fma_emu.h | 3 - target/hppa/cpu.h | 2 + target/i386/tcg/helper-tcg.h | 2 + target/loongarch/internals.h | 2 + target/m68k/cpu.h | 2 + target/microblaze/cpu.h | 2 + target/mips/tcg/tcg-internal.h | 2 + target/openrisc/cpu.h | 2 + target/ppc/cpu.h | 2 + target/riscv/cpu.h | 3 + target/rx/cpu.h | 2 + target/s390x/s390x-internal.h | 2 + target/sh4/cpu.h | 2 + target/sparc/cpu.h | 2 + target/sparc/helper.h | 4 +- target/tricore/cpu.h | 2 + target/xtensa/cpu.h | 2 + accel/tcg/cpu-exec.c | 8 +- accel/tcg/plugin-gen.c | 9 + accel/tcg/translate-all.c | 8 +- fpu/softfloat.c | 63 +-- target/alpha/cpu.c | 1 + target/alpha/translate.c | 4 +- target/arm/cpu.c | 1 + target/arm/tcg/cpu-v7m.c | 1 + target/arm/tcg/helper-a64.c | 6 +- target/arm/tcg/translate.c | 5 +- target/avr/cpu.c | 1 + target/avr/translate.c | 6 +- target/hexagon/cpu.c | 1 + target/hexagon/fma_emu.c | 496 ++++++--------------- target/hexagon/op_helper.c | 125 ++---- target/hexagon/translate.c | 4 +- target/hppa/cpu.c | 1 + target/hppa/translate.c | 4 +- target/i386/tcg/tcg-cpu.c | 1 + target/i386/tcg/translate.c | 5 +- target/loongarch/cpu.c | 1 + target/loongarch/tcg/translate.c | 4 +- target/m68k/cpu.c | 1 + target/m68k/translate.c | 4 +- target/microblaze/cpu.c | 1 + target/microblaze/translate.c | 4 +- target/mips/cpu.c | 1 + target/mips/tcg/translate.c | 4 +- target/openrisc/cpu.c | 1 + target/openrisc/translate.c | 4 +- target/ppc/cpu_init.c | 1 + target/ppc/translate.c | 4 +- target/riscv/tcg/tcg-cpu.c | 1 + target/riscv/translate.c | 4 +- target/rx/cpu.c | 1 + target/rx/translate.c | 4 +- target/s390x/cpu.c | 1 + target/s390x/tcg/translate.c | 4 +- target/sh4/cpu.c | 1 + target/sh4/translate.c | 4 +- target/sparc/cpu.c | 1 + target/sparc/fop_helper.c | 8 +- target/sparc/translate.c | 84 ++-- target/tricore/cpu.c | 1 + target/tricore/translate.c | 5 +- target/xtensa/cpu.c | 1 + target/xtensa/translate.c | 4 +- tcg/optimize.c | 857 +++++++++++++++++++----------------- tests/tcg/multiarch/system/memory.c | 9 +- fpu/softfloat-parts.c.inc | 16 +- 75 files changed, 866 insertions(+), 1009 deletions(-) From: Ilya Leoshkevich <iii@linux.ibm.com> make check-tcg fails on Fedora with the following error message: alpha-linux-gnu-gcc [...] qemu/tests/tcg/multiarch/system/memory.c -o memory [...] qemu/tests/tcg/multiarch/system/memory.c:17:10: fatal error: inttypes.h: No such file or directory 17 | #include <inttypes.h> | ^~~~~~~~~~~~ compilation terminated. The reason is that Fedora has cross-compilers, but no cross-glibc headers. Fix by hardcoding the format specifiers and dropping the include. An alternative fix would be to introduce a configure check for inttypes.h. But this would make it impossible to use Fedora cross-compilers for softmmu tests, which used to work so far. Fixes: ecbcc9ead2f8 ("tests/tcg: add a system test to check memory instrumentation") Signed-off-by: Ilya Leoshkevich <iii@linux.ibm.com> Reviewed-by: Paolo Bonzini <pbonzini@redhat.com> Message-ID: <20241010085906.226249-1-iii@linux.ibm.com> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- tests/tcg/multiarch/system/memory.c | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/tests/tcg/multiarch/system/memory.c b/tests/tcg/multiarch/system/memory.c index XXXXXXX..XXXXXXX 100644 --- a/tests/tcg/multiarch/system/memory.c +++ b/tests/tcg/multiarch/system/memory.c @@ -XXX,XX +XXX,XX @@ #include <stdint.h> #include <stdbool.h> -#include <inttypes.h> #include <minilib.h> #ifndef CHECK_UNALIGNED @@ -XXX,XX +XXX,XX @@ int main(void) int i; bool ok = true; - ml_printf("Test data start: 0x%"PRIxPTR"\n", &test_data[0]); - ml_printf("Test data end: 0x%"PRIxPTR"\n", &test_data[TEST_SIZE]); + ml_printf("Test data start: 0x%lx\n", (unsigned long)&test_data[0]); + ml_printf("Test data end: 0x%lx\n", (unsigned long)&test_data[TEST_SIZE]); /* Run through the unsigned tests first */ for (i = 0; i < ARRAY_SIZE(init_ufns) && ok; i++) { @@ -XXX,XX +XXX,XX @@ int main(void) ok = do_signed_reads(true); } - ml_printf("Test data read: %"PRId32"\n", test_read_count); - ml_printf("Test data write: %"PRId32"\n", test_write_count); + ml_printf("Test data read: %lu\n", (unsigned long)test_read_count); + ml_printf("Test data write: %lu\n", (unsigned long)test_write_count); ml_printf("Test complete: %s\n", ok ? "PASSED" : "FAILED"); return ok ? 0 : -1; } -- 2.43.0 From: Pierrick Bouvier <pierrick.bouvier@linaro.org> When running with a single vcpu, we can return a constant instead of a load when accessing cpu_index. A side effect is that all tcg operations using it are optimized, most notably scoreboard access. When running a simple loop in user-mode, the speedup is around 20%. Signed-off-by: Pierrick Bouvier <pierrick.bouvier@linaro.org> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> Message-ID: <20241128213843.1023080-1-pierrick.bouvier@linaro.org> --- accel/tcg/plugin-gen.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/accel/tcg/plugin-gen.c b/accel/tcg/plugin-gen.c index XXXXXXX..XXXXXXX 100644 --- a/accel/tcg/plugin-gen.c +++ b/accel/tcg/plugin-gen.c @@ -XXX,XX +XXX,XX @@ static void gen_disable_mem_helper(void) static TCGv_i32 gen_cpu_index(void) { + /* + * Optimize when we run with a single vcpu. All values using cpu_index, + * including scoreboard index, will be optimized out. + * User-mode calls tb_flush when setting this flag. In system-mode, all + * vcpus are created before generating code. + */ + if (!tcg_cflags_has(current_cpu, CF_PARALLEL)) { + return tcg_constant_i32(current_cpu->cpu_index); + } TCGv_i32 cpu_index = tcg_temp_ebb_new_i32(); tcg_gen_ld_i32(cpu_index, tcg_env, -offsetof(ArchCPU, env) + offsetof(CPUState, cpu_index)); -- 2.43.0 Call them directly from the opcode switch statement in tcg_optimize, rather than in finish_folding based on opcode flags. Adjust folding of conditional branches to match. Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- tcg/optimize.c | 47 +++++++++++++++++++++++++++++++---------------- 1 file changed, 31 insertions(+), 16 deletions(-) diff --git a/tcg/optimize.c b/tcg/optimize.c index XXXXXXX..XXXXXXX 100644 --- a/tcg/optimize.c +++ b/tcg/optimize.c @@ -XXX,XX +XXX,XX @@ static void copy_propagate(OptContext *ctx, TCGOp *op, } } +static void finish_bb(OptContext *ctx) +{ + /* We only optimize memory barriers across basic blocks. */ + ctx->prev_mb = NULL; +} + +static void finish_ebb(OptContext *ctx) +{ + finish_bb(ctx); + /* We only optimize across extended basic blocks. */ + memset(&ctx->temps_used, 0, sizeof(ctx->temps_used)); + remove_mem_copy_all(ctx); +} + static void finish_folding(OptContext *ctx, TCGOp *op) { const TCGOpDef *def = &tcg_op_defs[op->opc]; int i, nb_oargs; - /* - * We only optimize extended basic blocks. If the opcode ends a BB - * and is not a conditional branch, reset all temp data. - */ - if (def->flags & TCG_OPF_BB_END) { - ctx->prev_mb = NULL; - if (!(def->flags & TCG_OPF_COND_BRANCH)) { - memset(&ctx->temps_used, 0, sizeof(ctx->temps_used)); - remove_mem_copy_all(ctx); - } - return; - } - nb_oargs = def->nb_oargs; for (i = 0; i < nb_oargs; i++) { TCGTemp *ts = arg_temp(op->args[i]); @@ -XXX,XX +XXX,XX @@ static bool fold_brcond(OptContext *ctx, TCGOp *op) if (i > 0) { op->opc = INDEX_op_br; op->args[0] = op->args[3]; + finish_ebb(ctx); + } else { + finish_bb(ctx); } - return false; + return true; } static bool fold_brcond2(OptContext *ctx, TCGOp *op) @@ -XXX,XX +XXX,XX @@ static bool fold_brcond2(OptContext *ctx, TCGOp *op) } op->opc = INDEX_op_br; op->args[0] = label; - break; + finish_ebb(ctx); + return true; } - return false; + + finish_bb(ctx); + return true; } static bool fold_bswap(OptContext *ctx, TCGOp *op) @@ -XXX,XX +XXX,XX @@ void tcg_optimize(TCGContext *s) CASE_OP_32_64_VEC(xor): done = fold_xor(&ctx, op); break; + case INDEX_op_set_label: + case INDEX_op_br: + case INDEX_op_exit_tb: + case INDEX_op_goto_tb: + case INDEX_op_goto_ptr: + finish_ebb(&ctx); + done = true; + break; default: break; } -- 2.43.0 There are only a few logical operations which can compute an "affected" mask. Split out handling of this optimization to a separate function, only to be called when applicable. Remove the a_mask field from OptContext, as the mask is no longer stored anywhere. Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- tcg/optimize.c | 42 +++++++++++++++++++++++++++--------------- 1 file changed, 27 insertions(+), 15 deletions(-) diff --git a/tcg/optimize.c b/tcg/optimize.c index XXXXXXX..XXXXXXX 100644 --- a/tcg/optimize.c +++ b/tcg/optimize.c @@ -XXX,XX +XXX,XX @@ typedef struct OptContext { QSIMPLEQ_HEAD(, MemCopyInfo) mem_free; /* In flight values from optimization. */ - uint64_t a_mask; /* mask bit is 0 iff value identical to first input */ uint64_t z_mask; /* mask bit is 0 iff value bit is 0 */ uint64_t s_mask; /* mask of clrsb(value) bits */ TCGType type; @@ -XXX,XX +XXX,XX @@ static bool fold_const2_commutative(OptContext *ctx, TCGOp *op) static bool fold_masks(OptContext *ctx, TCGOp *op) { - uint64_t a_mask = ctx->a_mask; uint64_t z_mask = ctx->z_mask; uint64_t s_mask = ctx->s_mask; @@ -XXX,XX +XXX,XX @@ static bool fold_masks(OptContext *ctx, TCGOp *op) * type changing opcodes. */ if (ctx->type == TCG_TYPE_I32) { - a_mask = (int32_t)a_mask; z_mask = (int32_t)z_mask; s_mask |= MAKE_64BIT_MASK(32, 32); ctx->z_mask = z_mask; @@ -XXX,XX +XXX,XX @@ static bool fold_masks(OptContext *ctx, TCGOp *op) if (z_mask == 0) { return tcg_opt_gen_movi(ctx, op, op->args[0], 0); } + return false; +} + +/* + * An "affected" mask bit is 0 if and only if the result is identical + * to the first input. Thus if the entire mask is 0, the operation + * is equivalent to a copy. + */ +static bool fold_affected_mask(OptContext *ctx, TCGOp *op, uint64_t a_mask) +{ + if (ctx->type == TCG_TYPE_I32) { + a_mask = (uint32_t)a_mask; + } if (a_mask == 0) { return tcg_opt_gen_mov(ctx, op, op->args[0], op->args[1]); } @@ -XXX,XX +XXX,XX @@ static bool fold_and(OptContext *ctx, TCGOp *op) * Known-zeros does not imply known-ones. Therefore unless * arg2 is constant, we can't infer affected bits from it. */ - if (arg_is_const(op->args[2])) { - ctx->a_mask = z1 & ~z2; + if (arg_is_const(op->args[2]) && + fold_affected_mask(ctx, op, z1 & ~z2)) { + return true; } return fold_masks(ctx, op); @@ -XXX,XX +XXX,XX @@ static bool fold_andc(OptContext *ctx, TCGOp *op) */ if (arg_is_const(op->args[2])) { uint64_t z2 = ~arg_info(op->args[2])->z_mask; - ctx->a_mask = z1 & ~z2; + if (fold_affected_mask(ctx, op, z1 & ~z2)) { + return true; + } z1 &= z2; } ctx->z_mask = z1; @@ -XXX,XX +XXX,XX @@ static bool fold_extract(OptContext *ctx, TCGOp *op) z_mask_old = arg_info(op->args[1])->z_mask; z_mask = extract64(z_mask_old, pos, len); - if (pos == 0) { - ctx->a_mask = z_mask_old ^ z_mask; + if (pos == 0 && fold_affected_mask(ctx, op, z_mask_old ^ z_mask)) { + return true; } ctx->z_mask = z_mask; ctx->s_mask = smask_from_zmask(z_mask); @@ -XXX,XX +XXX,XX @@ static bool fold_exts(OptContext *ctx, TCGOp *op) ctx->z_mask = z_mask; ctx->s_mask = s_mask; - if (!type_change) { - ctx->a_mask = s_mask & ~s_mask_old; + if (!type_change && fold_affected_mask(ctx, op, s_mask & ~s_mask_old)) { + return true; } return fold_masks(ctx, op); @@ -XXX,XX +XXX,XX @@ static bool fold_extu(OptContext *ctx, TCGOp *op) ctx->z_mask = z_mask; ctx->s_mask = smask_from_zmask(z_mask); - if (!type_change) { - ctx->a_mask = z_mask_old ^ z_mask; + if (!type_change && fold_affected_mask(ctx, op, z_mask_old ^ z_mask)) { + return true; } return fold_masks(ctx, op); } @@ -XXX,XX +XXX,XX @@ static bool fold_sextract(OptContext *ctx, TCGOp *op) s_mask |= MAKE_64BIT_MASK(len, 64 - len); ctx->s_mask = s_mask; - if (pos == 0) { - ctx->a_mask = s_mask & ~s_mask_old; + if (pos == 0 && fold_affected_mask(ctx, op, s_mask & ~s_mask_old)) { + return true; } return fold_masks(ctx, op); @@ -XXX,XX +XXX,XX @@ void tcg_optimize(TCGContext *s) } /* Assume all bits affected, no bits known zero, no sign reps. */ - ctx.a_mask = -1; ctx.z_mask = -1; ctx.s_mask = 0; -- 2.43.0 Use of fold_masks should be restricted to those opcodes that can reliably make use of it -- those with a single output, and from higher-level folders that set up the masks. Prepare for conversion of each folder in turn. Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- tcg/optimize.c | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-) diff --git a/tcg/optimize.c b/tcg/optimize.c index XXXXXXX..XXXXXXX 100644 --- a/tcg/optimize.c +++ b/tcg/optimize.c @@ -XXX,XX +XXX,XX @@ static bool fold_masks(OptContext *ctx, TCGOp *op) { uint64_t z_mask = ctx->z_mask; uint64_t s_mask = ctx->s_mask; + const TCGOpDef *def = &tcg_op_defs[op->opc]; + TCGTemp *ts; + TempOptInfo *ti; + + /* Only single-output opcodes are supported here. */ + tcg_debug_assert(def->nb_oargs == 1); /* * 32-bit ops generate 32-bit results, which for the purpose of @@ -XXX,XX +XXX,XX @@ static bool fold_masks(OptContext *ctx, TCGOp *op) if (ctx->type == TCG_TYPE_I32) { z_mask = (int32_t)z_mask; s_mask |= MAKE_64BIT_MASK(32, 32); - ctx->z_mask = z_mask; - ctx->s_mask = s_mask; } if (z_mask == 0) { return tcg_opt_gen_movi(ctx, op, op->args[0], 0); } - return false; + + ts = arg_temp(op->args[0]); + reset_ts(ctx, ts); + + ti = ts_info(ts); + ti->z_mask = z_mask; + ti->s_mask = s_mask; + return true; } /* -- 2.43.0 Add a routine to which masks can be passed directly, rather than storing them into OptContext. To be used in upcoming patches. Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- tcg/optimize.c | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/tcg/optimize.c b/tcg/optimize.c index XXXXXXX..XXXXXXX 100644 --- a/tcg/optimize.c +++ b/tcg/optimize.c @@ -XXX,XX +XXX,XX @@ static bool fold_const2_commutative(OptContext *ctx, TCGOp *op) return fold_const2(ctx, op); } -static bool fold_masks(OptContext *ctx, TCGOp *op) +/* + * Record "zero" and "sign" masks for the single output of @op. + * See TempOptInfo definition of z_mask and s_mask. + * If z_mask allows, fold the output to constant zero. + */ +static bool fold_masks_zs(OptContext *ctx, TCGOp *op, + uint64_t z_mask, uint64_t s_mask) { - uint64_t z_mask = ctx->z_mask; - uint64_t s_mask = ctx->s_mask; const TCGOpDef *def = &tcg_op_defs[op->opc]; TCGTemp *ts; TempOptInfo *ti; @@ -XXX,XX +XXX,XX @@ static bool fold_masks(OptContext *ctx, TCGOp *op) return true; } +static bool fold_masks(OptContext *ctx, TCGOp *op) +{ + return fold_masks_zs(ctx, op, ctx->z_mask, ctx->s_mask); +} + /* * An "affected" mask bit is 0 if and only if the result is identical * to the first input. Thus if the entire mask is 0, the operation -- 2.43.0 Consider the passed s_mask to be a minimum deduced from either existing s_mask or from a sign-extension operation. We may be able to deduce more from the set of known zeros. Remove identical logic from several opcode folders. Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- tcg/optimize.c | 21 ++++++--------------- 1 file changed, 6 insertions(+), 15 deletions(-) diff --git a/tcg/optimize.c b/tcg/optimize.c index XXXXXXX..XXXXXXX 100644 --- a/tcg/optimize.c +++ b/tcg/optimize.c @@ -XXX,XX +XXX,XX @@ static bool fold_const2_commutative(OptContext *ctx, TCGOp *op) * Record "zero" and "sign" masks for the single output of @op. * See TempOptInfo definition of z_mask and s_mask. * If z_mask allows, fold the output to constant zero. + * The passed s_mask may be augmented by z_mask. */ static bool fold_masks_zs(OptContext *ctx, TCGOp *op, uint64_t z_mask, uint64_t s_mask) @@ -XXX,XX +XXX,XX @@ static bool fold_masks_zs(OptContext *ctx, TCGOp *op, ti = ts_info(ts); ti->z_mask = z_mask; - ti->s_mask = s_mask; + ti->s_mask = s_mask | smask_from_zmask(z_mask); return true; } @@ -XXX,XX +XXX,XX @@ static bool fold_bswap(OptContext *ctx, TCGOp *op) default: g_assert_not_reached(); } - s_mask = smask_from_zmask(z_mask); + s_mask = 0; switch (op->args[2] & (TCG_BSWAP_OZ | TCG_BSWAP_OS)) { case TCG_BSWAP_OZ: break; @@ -XXX,XX +XXX,XX @@ static bool fold_bswap(OptContext *ctx, TCGOp *op) default: /* The high bits are undefined: force all bits above the sign to 1. */ z_mask |= sign << 1; - s_mask = 0; break; } ctx->z_mask = z_mask; @@ -XXX,XX +XXX,XX @@ static bool fold_count_zeros(OptContext *ctx, TCGOp *op) g_assert_not_reached(); } ctx->z_mask = arg_info(op->args[2])->z_mask | z_mask; - ctx->s_mask = smask_from_zmask(ctx->z_mask); return false; } @@ -XXX,XX +XXX,XX @@ static bool fold_ctpop(OptContext *ctx, TCGOp *op) default: g_assert_not_reached(); } - ctx->s_mask = smask_from_zmask(ctx->z_mask); return false; } @@ -XXX,XX +XXX,XX @@ static bool fold_extract(OptContext *ctx, TCGOp *op) return true; } ctx->z_mask = z_mask; - ctx->s_mask = smask_from_zmask(z_mask); return fold_masks(ctx, op); } @@ -XXX,XX +XXX,XX @@ static bool fold_extu(OptContext *ctx, TCGOp *op) } ctx->z_mask = z_mask; - ctx->s_mask = smask_from_zmask(z_mask); if (!type_change && fold_affected_mask(ctx, op, z_mask_old ^ z_mask)) { return true; } @@ -XXX,XX +XXX,XX @@ static bool fold_qemu_ld(OptContext *ctx, TCGOp *op) int width = 8 * memop_size(mop); if (width < 64) { - ctx->s_mask = MAKE_64BIT_MASK(width, 64 - width); - if (!(mop & MO_SIGN)) { + if (mop & MO_SIGN) { + ctx->s_mask = MAKE_64BIT_MASK(width, 64 - width); + } else { ctx->z_mask = MAKE_64BIT_MASK(0, width); - ctx->s_mask <<= 1; } } @@ -XXX,XX +XXX,XX @@ static bool fold_setcond(OptContext *ctx, TCGOp *op) fold_setcond_tst_pow2(ctx, op, false); ctx->z_mask = 1; - ctx->s_mask = smask_from_zmask(1); return false; } @@ -XXX,XX +XXX,XX @@ static bool fold_setcond2(OptContext *ctx, TCGOp *op) } ctx->z_mask = 1; - ctx->s_mask = smask_from_zmask(1); return false; do_setcond_const: @@ -XXX,XX +XXX,XX @@ static bool fold_tcg_ld(OptContext *ctx, TCGOp *op) break; CASE_OP_32_64(ld8u): ctx->z_mask = MAKE_64BIT_MASK(0, 8); - ctx->s_mask = MAKE_64BIT_MASK(9, 55); break; CASE_OP_32_64(ld16s): ctx->s_mask = MAKE_64BIT_MASK(16, 48); break; CASE_OP_32_64(ld16u): ctx->z_mask = MAKE_64BIT_MASK(0, 16); - ctx->s_mask = MAKE_64BIT_MASK(17, 47); break; case INDEX_op_ld32s_i64: ctx->s_mask = MAKE_64BIT_MASK(32, 32); break; case INDEX_op_ld32u_i64: ctx->z_mask = MAKE_64BIT_MASK(0, 32); - ctx->s_mask = MAKE_64BIT_MASK(33, 31); break; default: g_assert_not_reached(); -- 2.43.0 Change the representation from sign bit repetitions to all bits equal to the sign bit, including the sign bit itself. The previous format has a problem in that it is difficult to recreate a valid sign mask after a shift operation: the "repetitions" part of the previous format meant that applying the same shift as for the value lead to an off-by-one value. The new format, including the sign bit itself, means that the sign mask can be manipulated in exactly the same way as the value, canonicalization is easier. Canonicalize the s_mask in fold_masks_zs, rather than requiring callers to do so. Treat 0 as a non-canonical but typeless input for no sign information, which will be reset as appropriate for the data type. We can easily fold in the data from z_mask while canonicalizing. Temporarily disable optimizations using s_mask while each operation is converted to use fold_masks_zs and to the new form. Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- tcg/optimize.c | 64 ++++++++++++-------------------------------------- 1 file changed, 15 insertions(+), 49 deletions(-) diff --git a/tcg/optimize.c b/tcg/optimize.c index XXXXXXX..XXXXXXX 100644 --- a/tcg/optimize.c +++ b/tcg/optimize.c @@ -XXX,XX +XXX,XX @@ typedef struct TempOptInfo { QSIMPLEQ_HEAD(, MemCopyInfo) mem_copy; uint64_t val; uint64_t z_mask; /* mask bit is 0 if and only if value bit is 0 */ - uint64_t s_mask; /* a left-aligned mask of clrsb(value) bits. */ + uint64_t s_mask; /* mask bit is 1 if value bit matches msb */ } TempOptInfo; typedef struct OptContext { @@ -XXX,XX +XXX,XX @@ typedef struct OptContext { /* In flight values from optimization. */ uint64_t z_mask; /* mask bit is 0 iff value bit is 0 */ - uint64_t s_mask; /* mask of clrsb(value) bits */ + uint64_t s_mask; /* mask bit is 1 if value bit matches msb */ TCGType type; } OptContext; -/* Calculate the smask for a specific value. */ -static uint64_t smask_from_value(uint64_t value) -{ - int rep = clrsb64(value); - return ~(~0ull >> rep); -} - -/* - * Calculate the smask for a given set of known-zeros. - * If there are lots of zeros on the left, we can consider the remainder - * an unsigned field, and thus the corresponding signed field is one bit - * larger. - */ -static uint64_t smask_from_zmask(uint64_t zmask) -{ - /* - * Only the 0 bits are significant for zmask, thus the msb itself - * must be zero, else we have no sign information. - */ - int rep = clz64(zmask); - if (rep == 0) { - return 0; - } - rep -= 1; - return ~(~0ull >> rep); -} - -/* - * Recreate a properly left-aligned smask after manipulation. - * Some bit-shuffling, particularly shifts and rotates, may - * retain sign bits on the left, but may scatter disconnected - * sign bits on the right. Retain only what remains to the left. - */ -static uint64_t smask_from_smask(int64_t smask) -{ - /* Only the 1 bits are significant for smask */ - return smask_from_zmask(~smask); -} - static inline TempOptInfo *ts_info(TCGTemp *ts) { return ts->state_ptr; @@ -XXX,XX +XXX,XX @@ static void init_ts_info(OptContext *ctx, TCGTemp *ts) ti->is_const = true; ti->val = ts->val; ti->z_mask = ts->val; - ti->s_mask = smask_from_value(ts->val); + ti->s_mask = INT64_MIN >> clrsb64(ts->val); } else { ti->is_const = false; ti->z_mask = -1; @@ -XXX,XX +XXX,XX @@ static void finish_folding(OptContext *ctx, TCGOp *op) */ if (i == 0) { ts_info(ts)->z_mask = ctx->z_mask; - ts_info(ts)->s_mask = ctx->s_mask; } } } @@ -XXX,XX +XXX,XX @@ static bool fold_const2_commutative(OptContext *ctx, TCGOp *op) * The passed s_mask may be augmented by z_mask. */ static bool fold_masks_zs(OptContext *ctx, TCGOp *op, - uint64_t z_mask, uint64_t s_mask) + uint64_t z_mask, int64_t s_mask) { const TCGOpDef *def = &tcg_op_defs[op->opc]; TCGTemp *ts; TempOptInfo *ti; + int rep; /* Only single-output opcodes are supported here. */ tcg_debug_assert(def->nb_oargs == 1); @@ -XXX,XX +XXX,XX @@ static bool fold_masks_zs(OptContext *ctx, TCGOp *op, */ if (ctx->type == TCG_TYPE_I32) { z_mask = (int32_t)z_mask; - s_mask |= MAKE_64BIT_MASK(32, 32); + s_mask |= INT32_MIN; } if (z_mask == 0) { @@ -XXX,XX +XXX,XX @@ static bool fold_masks_zs(OptContext *ctx, TCGOp *op, ti = ts_info(ts); ti->z_mask = z_mask; - ti->s_mask = s_mask | smask_from_zmask(z_mask); + + /* Canonicalize s_mask and incorporate data from z_mask. */ + rep = clz64(~s_mask); + rep = MAX(rep, clz64(z_mask)); + rep = MAX(rep - 1, 0); + ti->s_mask = INT64_MIN >> rep; + return true; } @@ -XXX,XX +XXX,XX @@ static bool fold_exts(OptContext *ctx, TCGOp *op) ctx->z_mask = z_mask; ctx->s_mask = s_mask; - if (!type_change && fold_affected_mask(ctx, op, s_mask & ~s_mask_old)) { + if (0 && !type_change && fold_affected_mask(ctx, op, s_mask & ~s_mask_old)) { return true; } @@ -XXX,XX +XXX,XX @@ static bool fold_sextract(OptContext *ctx, TCGOp *op) s_mask |= MAKE_64BIT_MASK(len, 64 - len); ctx->s_mask = s_mask; - if (pos == 0 && fold_affected_mask(ctx, op, s_mask & ~s_mask_old)) { + if (0 && pos == 0 && fold_affected_mask(ctx, op, s_mask & ~s_mask_old)) { return true; } @@ -XXX,XX +XXX,XX @@ static bool fold_shift(OptContext *ctx, TCGOp *op) ctx->z_mask = do_constant_folding(op->opc, ctx->type, z_mask, sh); s_mask = do_constant_folding(op->opc, ctx->type, s_mask, sh); - ctx->s_mask = smask_from_smask(s_mask); return fold_masks(ctx, op); } -- 2.43.0 Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- tcg/optimize.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/tcg/optimize.c b/tcg/optimize.c index XXXXXXX..XXXXXXX 100644 --- a/tcg/optimize.c +++ b/tcg/optimize.c @@ -XXX,XX +XXX,XX @@ static void finish_ebb(OptContext *ctx) remove_mem_copy_all(ctx); } -static void finish_folding(OptContext *ctx, TCGOp *op) +static bool finish_folding(OptContext *ctx, TCGOp *op) { const TCGOpDef *def = &tcg_op_defs[op->opc]; int i, nb_oargs; @@ -XXX,XX +XXX,XX @@ static void finish_folding(OptContext *ctx, TCGOp *op) ts_info(ts)->z_mask = ctx->z_mask; } } + return true; } /* @@ -XXX,XX +XXX,XX @@ static bool fold_add(OptContext *ctx, TCGOp *op) fold_xi_to_x(ctx, op, 0)) { return true; } - return false; + return finish_folding(ctx, op); } /* We cannot as yet do_constant_folding with vectors. */ @@ -XXX,XX +XXX,XX @@ static bool fold_add_vec(OptContext *ctx, TCGOp *op) fold_xi_to_x(ctx, op, 0)) { return true; } - return false; + return finish_folding(ctx, op); } static bool fold_addsub2(OptContext *ctx, TCGOp *op, bool add) @@ -XXX,XX +XXX,XX @@ static bool fold_addsub2(OptContext *ctx, TCGOp *op, bool add) op->args[4] = arg_new_constant(ctx, bl); op->args[5] = arg_new_constant(ctx, bh); } - return false; + return finish_folding(ctx, op); } static bool fold_add2(OptContext *ctx, TCGOp *op) -- 2.43.0 Introduce ti_is_const, ti_const_val, ti_is_const_val. Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- tcg/optimize.c | 20 +++++++++++++++++--- 1 file changed, 17 insertions(+), 3 deletions(-) diff --git a/tcg/optimize.c b/tcg/optimize.c index XXXXXXX..XXXXXXX 100644 --- a/tcg/optimize.c +++ b/tcg/optimize.c @@ -XXX,XX +XXX,XX @@ static inline TempOptInfo *arg_info(TCGArg arg) return ts_info(arg_temp(arg)); } +static inline bool ti_is_const(TempOptInfo *ti) +{ + return ti->is_const; +} + +static inline uint64_t ti_const_val(TempOptInfo *ti) +{ + return ti->val; +} + +static inline bool ti_is_const_val(TempOptInfo *ti, uint64_t val) +{ + return ti_is_const(ti) && ti_const_val(ti) == val; +} + static inline bool ts_is_const(TCGTemp *ts) { - return ts_info(ts)->is_const; + return ti_is_const(ts_info(ts)); } static inline bool ts_is_const_val(TCGTemp *ts, uint64_t val) { - TempOptInfo *ti = ts_info(ts); - return ti->is_const && ti->val == val; + return ti_is_const_val(ts_info(ts), val); } static inline bool arg_is_const(TCGArg arg) -- 2.43.0 Avoid the use of the OptContext slots. Find TempOptInfo once. Sink mask computation below fold_affected_mask early exit. Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- tcg/optimize.c | 30 ++++++++++++++++-------------- 1 file changed, 16 insertions(+), 14 deletions(-) diff --git a/tcg/optimize.c b/tcg/optimize.c index XXXXXXX..XXXXXXX 100644 --- a/tcg/optimize.c +++ b/tcg/optimize.c @@ -XXX,XX +XXX,XX @@ static bool fold_add2(OptContext *ctx, TCGOp *op) static bool fold_and(OptContext *ctx, TCGOp *op) { - uint64_t z1, z2; + uint64_t z1, z2, z_mask, s_mask; + TempOptInfo *t1, *t2; if (fold_const2_commutative(ctx, op) || fold_xi_to_i(ctx, op, 0) || @@ -XXX,XX +XXX,XX @@ static bool fold_and(OptContext *ctx, TCGOp *op) return true; } - z1 = arg_info(op->args[1])->z_mask; - z2 = arg_info(op->args[2])->z_mask; - ctx->z_mask = z1 & z2; - - /* - * Sign repetitions are perforce all identical, whether they are 1 or 0. - * Bitwise operations preserve the relative quantity of the repetitions. - */ - ctx->s_mask = arg_info(op->args[1])->s_mask - & arg_info(op->args[2])->s_mask; + t1 = arg_info(op->args[1]); + t2 = arg_info(op->args[2]); + z1 = t1->z_mask; + z2 = t2->z_mask; /* * Known-zeros does not imply known-ones. Therefore unless * arg2 is constant, we can't infer affected bits from it. */ - if (arg_is_const(op->args[2]) && - fold_affected_mask(ctx, op, z1 & ~z2)) { + if (ti_is_const(t2) && fold_affected_mask(ctx, op, z1 & ~z2)) { return true; } - return fold_masks(ctx, op); + z_mask = z1 & z2; + + /* + * Sign repetitions are perforce all identical, whether they are 1 or 0. + * Bitwise operations preserve the relative quantity of the repetitions. + */ + s_mask = t1->s_mask & t2->s_mask; + + return fold_masks_zs(ctx, op, z_mask, s_mask); } static bool fold_andc(OptContext *ctx, TCGOp *op) -- 2.43.0 Avoid the use of the OptContext slots. Find TempOptInfo once. Avoid double inversion of the value of second const operand. Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- tcg/optimize.c | 21 +++++++++++---------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/tcg/optimize.c b/tcg/optimize.c index XXXXXXX..XXXXXXX 100644 --- a/tcg/optimize.c +++ b/tcg/optimize.c @@ -XXX,XX +XXX,XX @@ static bool fold_and(OptContext *ctx, TCGOp *op) static bool fold_andc(OptContext *ctx, TCGOp *op) { - uint64_t z1; + uint64_t z_mask, s_mask; + TempOptInfo *t1, *t2; if (fold_const2(ctx, op) || fold_xx_to_i(ctx, op, 0) || @@ -XXX,XX +XXX,XX @@ static bool fold_andc(OptContext *ctx, TCGOp *op) return true; } - z1 = arg_info(op->args[1])->z_mask; + t1 = arg_info(op->args[1]); + t2 = arg_info(op->args[2]); + z_mask = t1->z_mask; /* * Known-zeros does not imply known-ones. Therefore unless * arg2 is constant, we can't infer anything from it. */ - if (arg_is_const(op->args[2])) { - uint64_t z2 = ~arg_info(op->args[2])->z_mask; - if (fold_affected_mask(ctx, op, z1 & ~z2)) { + if (ti_is_const(t2)) { + uint64_t v2 = ti_const_val(t2); + if (fold_affected_mask(ctx, op, z_mask & v2)) { return true; } - z1 &= z2; + z_mask &= ~v2; } - ctx->z_mask = z1; - ctx->s_mask = arg_info(op->args[1])->s_mask - & arg_info(op->args[2])->s_mask; - return fold_masks(ctx, op); + s_mask = t1->s_mask & t2->s_mask; + return fold_masks_zs(ctx, op, z_mask, s_mask); } static bool fold_brcond(OptContext *ctx, TCGOp *op) -- 2.43.0 Avoid the use of the OptContext slots. Find TempOptInfo once. Always set s_mask along the BSWAP_OS path, since the result is being explicitly sign-extended. Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- tcg/optimize.c | 21 ++++++++++----------- 1 file changed, 10 insertions(+), 11 deletions(-) diff --git a/tcg/optimize.c b/tcg/optimize.c index XXXXXXX..XXXXXXX 100644 --- a/tcg/optimize.c +++ b/tcg/optimize.c @@ -XXX,XX +XXX,XX @@ static bool fold_brcond2(OptContext *ctx, TCGOp *op) static bool fold_bswap(OptContext *ctx, TCGOp *op) { uint64_t z_mask, s_mask, sign; + TempOptInfo *t1 = arg_info(op->args[1]); - if (arg_is_const(op->args[1])) { - uint64_t t = arg_info(op->args[1])->val; - - t = do_constant_folding(op->opc, ctx->type, t, op->args[2]); - return tcg_opt_gen_movi(ctx, op, op->args[0], t); + if (ti_is_const(t1)) { + return tcg_opt_gen_movi(ctx, op, op->args[0], + do_constant_folding(op->opc, ctx->type, + ti_const_val(t1), + op->args[2])); } - z_mask = arg_info(op->args[1])->z_mask; - + z_mask = t1->z_mask; switch (op->opc) { case INDEX_op_bswap16_i32: case INDEX_op_bswap16_i64: @@ -XXX,XX +XXX,XX @@ static bool fold_bswap(OptContext *ctx, TCGOp *op) /* If the sign bit may be 1, force all the bits above to 1. */ if (z_mask & sign) { z_mask |= sign; - s_mask = sign << 1; } + /* The value and therefore s_mask is explicitly sign-extended. */ + s_mask = sign; break; default: /* The high bits are undefined: force all bits above the sign to 1. */ z_mask |= sign << 1; break; } - ctx->z_mask = z_mask; - ctx->s_mask = s_mask; - return fold_masks(ctx, op); + return fold_masks_zs(ctx, op, z_mask, s_mask); } static bool fold_call(OptContext *ctx, TCGOp *op) -- 2.43.0 Avoid the use of the OptContext slots. Find TempOptInfo once. Compute s_mask from the union of the maximum count and the op2 fallback for op1 being zero. Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- tcg/optimize.c | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/tcg/optimize.c b/tcg/optimize.c index XXXXXXX..XXXXXXX 100644 --- a/tcg/optimize.c +++ b/tcg/optimize.c @@ -XXX,XX +XXX,XX @@ static bool fold_call(OptContext *ctx, TCGOp *op) static bool fold_count_zeros(OptContext *ctx, TCGOp *op) { - uint64_t z_mask; + uint64_t z_mask, s_mask; + TempOptInfo *t1 = arg_info(op->args[1]); + TempOptInfo *t2 = arg_info(op->args[2]); - if (arg_is_const(op->args[1])) { - uint64_t t = arg_info(op->args[1])->val; + if (ti_is_const(t1)) { + uint64_t t = ti_const_val(t1); if (t != 0) { t = do_constant_folding(op->opc, ctx->type, t, 0); @@ -XXX,XX +XXX,XX @@ static bool fold_count_zeros(OptContext *ctx, TCGOp *op) default: g_assert_not_reached(); } - ctx->z_mask = arg_info(op->args[2])->z_mask | z_mask; - return false; + s_mask = ~z_mask; + z_mask |= t2->z_mask; + s_mask &= t2->s_mask; + + return fold_masks_zs(ctx, op, z_mask, s_mask); } static bool fold_ctpop(OptContext *ctx, TCGOp *op) -- 2.43.0 Add fold_masks_z as a trivial wrapper around fold_masks_zs. Avoid the use of the OptContext slots. Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- tcg/optimize.c | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/tcg/optimize.c b/tcg/optimize.c index XXXXXXX..XXXXXXX 100644 --- a/tcg/optimize.c +++ b/tcg/optimize.c @@ -XXX,XX +XXX,XX @@ static bool fold_masks_zs(OptContext *ctx, TCGOp *op, return true; } +static bool fold_masks_z(OptContext *ctx, TCGOp *op, uint64_t z_mask) +{ + return fold_masks_zs(ctx, op, z_mask, 0); +} + static bool fold_masks(OptContext *ctx, TCGOp *op) { return fold_masks_zs(ctx, op, ctx->z_mask, ctx->s_mask); @@ -XXX,XX +XXX,XX @@ static bool fold_count_zeros(OptContext *ctx, TCGOp *op) static bool fold_ctpop(OptContext *ctx, TCGOp *op) { + uint64_t z_mask; + if (fold_const1(ctx, op)) { return true; } switch (ctx->type) { case TCG_TYPE_I32: - ctx->z_mask = 32 | 31; + z_mask = 32 | 31; break; case TCG_TYPE_I64: - ctx->z_mask = 64 | 63; + z_mask = 64 | 63; break; default: g_assert_not_reached(); } - return false; + return fold_masks_z(ctx, op, z_mask); } static bool fold_deposit(OptContext *ctx, TCGOp *op) -- 2.43.0 Avoid the use of the OptContext slots. Find TempOptInfo once. When we fold to and, use fold_and. Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- tcg/optimize.c | 35 +++++++++++++++++------------------ 1 file changed, 17 insertions(+), 18 deletions(-) diff --git a/tcg/optimize.c b/tcg/optimize.c index XXXXXXX..XXXXXXX 100644 --- a/tcg/optimize.c +++ b/tcg/optimize.c @@ -XXX,XX +XXX,XX @@ static bool fold_ctpop(OptContext *ctx, TCGOp *op) static bool fold_deposit(OptContext *ctx, TCGOp *op) { + TempOptInfo *t1 = arg_info(op->args[1]); + TempOptInfo *t2 = arg_info(op->args[2]); + int ofs = op->args[3]; + int len = op->args[4]; TCGOpcode and_opc; + uint64_t z_mask; - if (arg_is_const(op->args[1]) && arg_is_const(op->args[2])) { - uint64_t t1 = arg_info(op->args[1])->val; - uint64_t t2 = arg_info(op->args[2])->val; - - t1 = deposit64(t1, op->args[3], op->args[4], t2); - return tcg_opt_gen_movi(ctx, op, op->args[0], t1); + if (ti_is_const(t1) && ti_is_const(t2)) { + return tcg_opt_gen_movi(ctx, op, op->args[0], + deposit64(ti_const_val(t1), ofs, len, + ti_const_val(t2))); } switch (ctx->type) { @@ -XXX,XX +XXX,XX @@ static bool fold_deposit(OptContext *ctx, TCGOp *op) } /* Inserting a value into zero at offset 0. */ - if (arg_is_const_val(op->args[1], 0) && op->args[3] == 0) { - uint64_t mask = MAKE_64BIT_MASK(0, op->args[4]); + if (ti_is_const_val(t1, 0) && ofs == 0) { + uint64_t mask = MAKE_64BIT_MASK(0, len); op->opc = and_opc; op->args[1] = op->args[2]; op->args[2] = arg_new_constant(ctx, mask); - ctx->z_mask = mask & arg_info(op->args[1])->z_mask; - return false; + return fold_and(ctx, op); } /* Inserting zero into a value. */ - if (arg_is_const_val(op->args[2], 0)) { - uint64_t mask = deposit64(-1, op->args[3], op->args[4], 0); + if (ti_is_const_val(t2, 0)) { + uint64_t mask = deposit64(-1, ofs, len, 0); op->opc = and_opc; op->args[2] = arg_new_constant(ctx, mask); - ctx->z_mask = mask & arg_info(op->args[1])->z_mask; - return false; + return fold_and(ctx, op); } - ctx->z_mask = deposit64(arg_info(op->args[1])->z_mask, - op->args[3], op->args[4], - arg_info(op->args[2])->z_mask); - return false; + z_mask = deposit64(t1->z_mask, ofs, len, t2->z_mask); + return fold_masks_z(ctx, op, z_mask); } static bool fold_divide(OptContext *ctx, TCGOp *op) -- 2.43.0 The input which overlaps the sign bit of the output can have its input s_mask propagated to the output s_mask. Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- tcg/optimize.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/tcg/optimize.c b/tcg/optimize.c index XXXXXXX..XXXXXXX 100644 --- a/tcg/optimize.c +++ b/tcg/optimize.c @@ -XXX,XX +XXX,XX @@ static bool fold_deposit(OptContext *ctx, TCGOp *op) TempOptInfo *t2 = arg_info(op->args[2]); int ofs = op->args[3]; int len = op->args[4]; + int width; TCGOpcode and_opc; - uint64_t z_mask; + uint64_t z_mask, s_mask; if (ti_is_const(t1) && ti_is_const(t2)) { return tcg_opt_gen_movi(ctx, op, op->args[0], @@ -XXX,XX +XXX,XX @@ static bool fold_deposit(OptContext *ctx, TCGOp *op) switch (ctx->type) { case TCG_TYPE_I32: and_opc = INDEX_op_and_i32; + width = 32; break; case TCG_TYPE_I64: and_opc = INDEX_op_and_i64; + width = 64; break; default: g_assert_not_reached(); @@ -XXX,XX +XXX,XX @@ static bool fold_deposit(OptContext *ctx, TCGOp *op) return fold_and(ctx, op); } + /* The s_mask from the top portion of the deposit is still valid. */ + if (ofs + len == width) { + s_mask = t2->s_mask << ofs; + } else { + s_mask = t1->s_mask & ~MAKE_64BIT_MASK(0, ofs + len); + } + z_mask = deposit64(t1->z_mask, ofs, len, t2->z_mask); - return fold_masks_z(ctx, op, z_mask); + return fold_masks_zs(ctx, op, z_mask, s_mask); } static bool fold_divide(OptContext *ctx, TCGOp *op) -- 2.43.0 Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- tcg/optimize.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tcg/optimize.c b/tcg/optimize.c index XXXXXXX..XXXXXXX 100644 --- a/tcg/optimize.c +++ b/tcg/optimize.c @@ -XXX,XX +XXX,XX @@ static bool fold_divide(OptContext *ctx, TCGOp *op) fold_xi_to_x(ctx, op, 1)) { return true; } - return false; + return finish_folding(ctx, op); } static bool fold_dup(OptContext *ctx, TCGOp *op) -- 2.43.0 Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- tcg/optimize.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tcg/optimize.c b/tcg/optimize.c index XXXXXXX..XXXXXXX 100644 --- a/tcg/optimize.c +++ b/tcg/optimize.c @@ -XXX,XX +XXX,XX @@ static bool fold_dup(OptContext *ctx, TCGOp *op) t = dup_const(TCGOP_VECE(op), t); return tcg_opt_gen_movi(ctx, op, op->args[0], t); } - return false; + return finish_folding(ctx, op); } static bool fold_dup2(OptContext *ctx, TCGOp *op) @@ -XXX,XX +XXX,XX @@ static bool fold_dup2(OptContext *ctx, TCGOp *op) op->opc = INDEX_op_dup_vec; TCGOP_VECE(op) = MO_32; } - return false; + return finish_folding(ctx, op); } static bool fold_eqv(OptContext *ctx, TCGOp *op) -- 2.43.0 Add fold_masks_s as a trivial wrapper around fold_masks_zs. Avoid the use of the OptContext slots. Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- tcg/optimize.c | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/tcg/optimize.c b/tcg/optimize.c index XXXXXXX..XXXXXXX 100644 --- a/tcg/optimize.c +++ b/tcg/optimize.c @@ -XXX,XX +XXX,XX @@ static bool fold_masks_z(OptContext *ctx, TCGOp *op, uint64_t z_mask) return fold_masks_zs(ctx, op, z_mask, 0); } +static bool fold_masks_s(OptContext *ctx, TCGOp *op, uint64_t s_mask) +{ + return fold_masks_zs(ctx, op, -1, s_mask); +} + static bool fold_masks(OptContext *ctx, TCGOp *op) { return fold_masks_zs(ctx, op, ctx->z_mask, ctx->s_mask); @@ -XXX,XX +XXX,XX @@ static bool fold_dup2(OptContext *ctx, TCGOp *op) static bool fold_eqv(OptContext *ctx, TCGOp *op) { + uint64_t s_mask; + if (fold_const2_commutative(ctx, op) || fold_xi_to_x(ctx, op, -1) || fold_xi_to_not(ctx, op, 0)) { return true; } - ctx->s_mask = arg_info(op->args[1])->s_mask - & arg_info(op->args[2])->s_mask; - return false; + s_mask = arg_info(op->args[1])->s_mask + & arg_info(op->args[2])->s_mask; + return fold_masks_s(ctx, op, s_mask); } static bool fold_extract(OptContext *ctx, TCGOp *op) -- 2.43.0 Avoid the use of the OptContext slots. Find TempOptInfo once. Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- tcg/optimize.c | 15 ++++++--------- 1 file changed, 6 insertions(+), 9 deletions(-) diff --git a/tcg/optimize.c b/tcg/optimize.c index XXXXXXX..XXXXXXX 100644 --- a/tcg/optimize.c +++ b/tcg/optimize.c @@ -XXX,XX +XXX,XX @@ static bool fold_eqv(OptContext *ctx, TCGOp *op) static bool fold_extract(OptContext *ctx, TCGOp *op) { uint64_t z_mask_old, z_mask; + TempOptInfo *t1 = arg_info(op->args[1]); int pos = op->args[2]; int len = op->args[3]; - if (arg_is_const(op->args[1])) { - uint64_t t; - - t = arg_info(op->args[1])->val; - t = extract64(t, pos, len); - return tcg_opt_gen_movi(ctx, op, op->args[0], t); + if (ti_is_const(t1)) { + return tcg_opt_gen_movi(ctx, op, op->args[0], + extract64(ti_const_val(t1), pos, len)); } - z_mask_old = arg_info(op->args[1])->z_mask; + z_mask_old = t1->z_mask; z_mask = extract64(z_mask_old, pos, len); if (pos == 0 && fold_affected_mask(ctx, op, z_mask_old ^ z_mask)) { return true; } - ctx->z_mask = z_mask; - return fold_masks(ctx, op); + return fold_masks_z(ctx, op, z_mask); } static bool fold_extract2(OptContext *ctx, TCGOp *op) -- 2.43.0 Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- tcg/optimize.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tcg/optimize.c b/tcg/optimize.c index XXXXXXX..XXXXXXX 100644 --- a/tcg/optimize.c +++ b/tcg/optimize.c @@ -XXX,XX +XXX,XX @@ static bool fold_extract2(OptContext *ctx, TCGOp *op) } return tcg_opt_gen_movi(ctx, op, op->args[0], v1 | v2); } - return false; + return finish_folding(ctx, op); } static bool fold_exts(OptContext *ctx, TCGOp *op) -- 2.43.0 Avoid the use of the OptContext slots. Find TempOptInfo once. Explicitly sign-extend z_mask instead of doing that manually. Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- tcg/optimize.c | 29 ++++++++++++----------------- 1 file changed, 12 insertions(+), 17 deletions(-) diff --git a/tcg/optimize.c b/tcg/optimize.c index XXXXXXX..XXXXXXX 100644 --- a/tcg/optimize.c +++ b/tcg/optimize.c @@ -XXX,XX +XXX,XX @@ static bool fold_extract2(OptContext *ctx, TCGOp *op) static bool fold_exts(OptContext *ctx, TCGOp *op) { - uint64_t s_mask_old, s_mask, z_mask, sign; + uint64_t s_mask_old, s_mask, z_mask; bool type_change = false; + TempOptInfo *t1; if (fold_const1(ctx, op)) { return true; } - z_mask = arg_info(op->args[1])->z_mask; - s_mask = arg_info(op->args[1])->s_mask; + t1 = arg_info(op->args[1]); + z_mask = t1->z_mask; + s_mask = t1->s_mask; s_mask_old = s_mask; switch (op->opc) { CASE_OP_32_64(ext8s): - sign = INT8_MIN; - z_mask = (uint8_t)z_mask; + s_mask |= INT8_MIN; + z_mask = (int8_t)z_mask; break; CASE_OP_32_64(ext16s): - sign = INT16_MIN; - z_mask = (uint16_t)z_mask; + s_mask |= INT16_MIN; + z_mask = (int16_t)z_mask; break; case INDEX_op_ext_i32_i64: type_change = true; QEMU_FALLTHROUGH; case INDEX_op_ext32s_i64: - sign = INT32_MIN; - z_mask = (uint32_t)z_mask; + s_mask |= INT32_MIN; + z_mask = (int32_t)z_mask; break; default: g_assert_not_reached(); } - if (z_mask & sign) { - z_mask |= sign; - } - s_mask |= sign << 1; - - ctx->z_mask = z_mask; - ctx->s_mask = s_mask; if (0 && !type_change && fold_affected_mask(ctx, op, s_mask & ~s_mask_old)) { return true; } - return fold_masks(ctx, op); + return fold_masks_zs(ctx, op, z_mask, s_mask); } static bool fold_extu(OptContext *ctx, TCGOp *op) -- 2.43.0 Avoid the use of the OptContext slots. Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- tcg/optimize.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tcg/optimize.c b/tcg/optimize.c index XXXXXXX..XXXXXXX 100644 --- a/tcg/optimize.c +++ b/tcg/optimize.c @@ -XXX,XX +XXX,XX @@ static bool fold_extu(OptContext *ctx, TCGOp *op) g_assert_not_reached(); } - ctx->z_mask = z_mask; if (!type_change && fold_affected_mask(ctx, op, z_mask_old ^ z_mask)) { return true; } - return fold_masks(ctx, op); + + return fold_masks_z(ctx, op, z_mask); } static bool fold_mb(OptContext *ctx, TCGOp *op) -- 2.43.0 Avoid the use of the OptContext slots. Find TempOptInfo once. Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- tcg/optimize.c | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-) diff --git a/tcg/optimize.c b/tcg/optimize.c index XXXXXXX..XXXXXXX 100644 --- a/tcg/optimize.c +++ b/tcg/optimize.c @@ -XXX,XX +XXX,XX @@ static bool fold_mov(OptContext *ctx, TCGOp *op) static bool fold_movcond(OptContext *ctx, TCGOp *op) { + uint64_t z_mask, s_mask; + TempOptInfo *tt, *ft; int i; /* If true and false values are the same, eliminate the cmp. */ @@ -XXX,XX +XXX,XX @@ static bool fold_movcond(OptContext *ctx, TCGOp *op) return tcg_opt_gen_mov(ctx, op, op->args[0], op->args[4 - i]); } - ctx->z_mask = arg_info(op->args[3])->z_mask - | arg_info(op->args[4])->z_mask; - ctx->s_mask = arg_info(op->args[3])->s_mask - & arg_info(op->args[4])->s_mask; + tt = arg_info(op->args[3]); + ft = arg_info(op->args[4]); + z_mask = tt->z_mask | ft->z_mask; + s_mask = tt->s_mask & ft->s_mask; - if (arg_is_const(op->args[3]) && arg_is_const(op->args[4])) { - uint64_t tv = arg_info(op->args[3])->val; - uint64_t fv = arg_info(op->args[4])->val; + if (ti_is_const(tt) && ti_is_const(ft)) { + uint64_t tv = ti_const_val(tt); + uint64_t fv = ti_const_val(ft); TCGOpcode opc, negopc = 0; TCGCond cond = op->args[5]; @@ -XXX,XX +XXX,XX @@ static bool fold_movcond(OptContext *ctx, TCGOp *op) } } } - return false; + + return fold_masks_zs(ctx, op, z_mask, s_mask); } static bool fold_mul(OptContext *ctx, TCGOp *op) -- 2.43.0 Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- tcg/optimize.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/tcg/optimize.c b/tcg/optimize.c index XXXXXXX..XXXXXXX 100644 --- a/tcg/optimize.c +++ b/tcg/optimize.c @@ -XXX,XX +XXX,XX @@ static bool fold_mul(OptContext *ctx, TCGOp *op) fold_xi_to_x(ctx, op, 1)) { return true; } - return false; + return finish_folding(ctx, op); } static bool fold_mul_highpart(OptContext *ctx, TCGOp *op) @@ -XXX,XX +XXX,XX @@ static bool fold_mul_highpart(OptContext *ctx, TCGOp *op) fold_xi_to_i(ctx, op, 0)) { return true; } - return false; + return finish_folding(ctx, op); } static bool fold_multiply2(OptContext *ctx, TCGOp *op) @@ -XXX,XX +XXX,XX @@ static bool fold_multiply2(OptContext *ctx, TCGOp *op) tcg_opt_gen_movi(ctx, op2, rh, h); return true; } - return false; + return finish_folding(ctx, op); } static bool fold_nand(OptContext *ctx, TCGOp *op) -- 2.43.0 Avoid the use of the OptContext slots. Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- tcg/optimize.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/tcg/optimize.c b/tcg/optimize.c index XXXXXXX..XXXXXXX 100644 --- a/tcg/optimize.c +++ b/tcg/optimize.c @@ -XXX,XX +XXX,XX @@ static bool fold_multiply2(OptContext *ctx, TCGOp *op) static bool fold_nand(OptContext *ctx, TCGOp *op) { + uint64_t s_mask; + if (fold_const2_commutative(ctx, op) || fold_xi_to_not(ctx, op, -1)) { return true; } - ctx->s_mask = arg_info(op->args[1])->s_mask - & arg_info(op->args[2])->s_mask; - return false; + s_mask = arg_info(op->args[1])->s_mask + & arg_info(op->args[2])->s_mask; + return fold_masks_s(ctx, op, s_mask); } static bool fold_neg_no_const(OptContext *ctx, TCGOp *op) -- 2.43.0 Avoid the use of the OptContext slots. Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- tcg/optimize.c | 9 ++------- 1 file changed, 2 insertions(+), 7 deletions(-) diff --git a/tcg/optimize.c b/tcg/optimize.c index XXXXXXX..XXXXXXX 100644 --- a/tcg/optimize.c +++ b/tcg/optimize.c @@ -XXX,XX +XXX,XX @@ static bool fold_neg_no_const(OptContext *ctx, TCGOp *op) { /* Set to 1 all bits to the left of the rightmost. */ uint64_t z_mask = arg_info(op->args[1])->z_mask; - ctx->z_mask = -(z_mask & -z_mask); + z_mask = -(z_mask & -z_mask); - /* - * Because of fold_sub_to_neg, we want to always return true, - * via finish_folding. - */ - finish_folding(ctx, op); - return true; + return fold_masks_z(ctx, op, z_mask); } static bool fold_neg(OptContext *ctx, TCGOp *op) -- 2.43.0 Avoid the use of the OptContext slots. Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- tcg/optimize.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/tcg/optimize.c b/tcg/optimize.c index XXXXXXX..XXXXXXX 100644 --- a/tcg/optimize.c +++ b/tcg/optimize.c @@ -XXX,XX +XXX,XX @@ static bool fold_neg(OptContext *ctx, TCGOp *op) static bool fold_nor(OptContext *ctx, TCGOp *op) { + uint64_t s_mask; + if (fold_const2_commutative(ctx, op) || fold_xi_to_not(ctx, op, 0)) { return true; } - ctx->s_mask = arg_info(op->args[1])->s_mask - & arg_info(op->args[2])->s_mask; - return false; + s_mask = arg_info(op->args[1])->s_mask + & arg_info(op->args[2])->s_mask; + return fold_masks_s(ctx, op, s_mask); } static bool fold_not(OptContext *ctx, TCGOp *op) -- 2.43.0 Avoid the use of the OptContext slots. Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- tcg/optimize.c | 7 +------ 1 file changed, 1 insertion(+), 6 deletions(-) diff --git a/tcg/optimize.c b/tcg/optimize.c index XXXXXXX..XXXXXXX 100644 --- a/tcg/optimize.c +++ b/tcg/optimize.c @@ -XXX,XX +XXX,XX @@ static bool fold_not(OptContext *ctx, TCGOp *op) if (fold_const1(ctx, op)) { return true; } - - ctx->s_mask = arg_info(op->args[1])->s_mask; - - /* Because of fold_to_not, we want to always return true, via finish. */ - finish_folding(ctx, op); - return true; + return fold_masks_s(ctx, op, arg_info(op->args[1])->s_mask); } static bool fold_or(OptContext *ctx, TCGOp *op) -- 2.43.0 Avoid the use of the OptContext slots. Find TempOptInfo once. Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- tcg/optimize.c | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/tcg/optimize.c b/tcg/optimize.c index XXXXXXX..XXXXXXX 100644 --- a/tcg/optimize.c +++ b/tcg/optimize.c @@ -XXX,XX +XXX,XX @@ static bool fold_not(OptContext *ctx, TCGOp *op) static bool fold_or(OptContext *ctx, TCGOp *op) { + uint64_t z_mask, s_mask; + TempOptInfo *t1, *t2; + if (fold_const2_commutative(ctx, op) || fold_xi_to_x(ctx, op, 0) || fold_xx_to_x(ctx, op)) { return true; } - ctx->z_mask = arg_info(op->args[1])->z_mask - | arg_info(op->args[2])->z_mask; - ctx->s_mask = arg_info(op->args[1])->s_mask - & arg_info(op->args[2])->s_mask; - return fold_masks(ctx, op); + t1 = arg_info(op->args[1]); + t2 = arg_info(op->args[2]); + z_mask = t1->z_mask | t2->z_mask; + s_mask = t1->s_mask & t2->s_mask; + return fold_masks_zs(ctx, op, z_mask, s_mask); } static bool fold_orc(OptContext *ctx, TCGOp *op) -- 2.43.0 Avoid the use of the OptContext slots. Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- tcg/optimize.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/tcg/optimize.c b/tcg/optimize.c index XXXXXXX..XXXXXXX 100644 --- a/tcg/optimize.c +++ b/tcg/optimize.c @@ -XXX,XX +XXX,XX @@ static bool fold_or(OptContext *ctx, TCGOp *op) static bool fold_orc(OptContext *ctx, TCGOp *op) { + uint64_t s_mask; + if (fold_const2(ctx, op) || fold_xx_to_i(ctx, op, -1) || fold_xi_to_x(ctx, op, -1) || @@ -XXX,XX +XXX,XX @@ static bool fold_orc(OptContext *ctx, TCGOp *op) return true; } - ctx->s_mask = arg_info(op->args[1])->s_mask - & arg_info(op->args[2])->s_mask; - return false; + s_mask = arg_info(op->args[1])->s_mask + & arg_info(op->args[2])->s_mask; + return fold_masks_s(ctx, op, s_mask); } static bool fold_qemu_ld(OptContext *ctx, TCGOp *op) -- 2.43.0 Avoid the use of the OptContext slots. Be careful not to call fold_masks_zs when the memory operation is wide enough to require multiple outputs, so split into two functions: fold_qemu_ld_1reg and fold_qemu_ld_2reg. Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- tcg/optimize.c | 26 +++++++++++++++++++++----- 1 file changed, 21 insertions(+), 5 deletions(-) diff --git a/tcg/optimize.c b/tcg/optimize.c index XXXXXXX..XXXXXXX 100644 --- a/tcg/optimize.c +++ b/tcg/optimize.c @@ -XXX,XX +XXX,XX @@ static bool fold_orc(OptContext *ctx, TCGOp *op) return fold_masks_s(ctx, op, s_mask); } -static bool fold_qemu_ld(OptContext *ctx, TCGOp *op) +static bool fold_qemu_ld_1reg(OptContext *ctx, TCGOp *op) { const TCGOpDef *def = &tcg_op_defs[op->opc]; MemOpIdx oi = op->args[def->nb_oargs + def->nb_iargs]; MemOp mop = get_memop(oi); int width = 8 * memop_size(mop); + uint64_t z_mask = -1, s_mask = 0; if (width < 64) { if (mop & MO_SIGN) { - ctx->s_mask = MAKE_64BIT_MASK(width, 64 - width); + s_mask = MAKE_64BIT_MASK(width - 1, 64 - (width - 1)); } else { - ctx->z_mask = MAKE_64BIT_MASK(0, width); + z_mask = MAKE_64BIT_MASK(0, width); } } /* Opcodes that touch guest memory stop the mb optimization. */ ctx->prev_mb = NULL; - return false; + + return fold_masks_zs(ctx, op, z_mask, s_mask); +} + +static bool fold_qemu_ld_2reg(OptContext *ctx, TCGOp *op) +{ + /* Opcodes that touch guest memory stop the mb optimization. */ + ctx->prev_mb = NULL; + return finish_folding(ctx, op); } static bool fold_qemu_st(OptContext *ctx, TCGOp *op) @@ -XXX,XX +XXX,XX @@ void tcg_optimize(TCGContext *s) break; case INDEX_op_qemu_ld_a32_i32: case INDEX_op_qemu_ld_a64_i32: + done = fold_qemu_ld_1reg(&ctx, op); + break; case INDEX_op_qemu_ld_a32_i64: case INDEX_op_qemu_ld_a64_i64: + if (TCG_TARGET_REG_BITS == 64) { + done = fold_qemu_ld_1reg(&ctx, op); + break; + } + QEMU_FALLTHROUGH; case INDEX_op_qemu_ld_a32_i128: case INDEX_op_qemu_ld_a64_i128: - done = fold_qemu_ld(&ctx, op); + done = fold_qemu_ld_2reg(&ctx, op); break; case INDEX_op_qemu_st8_a32_i32: case INDEX_op_qemu_st8_a64_i32: -- 2.43.0 Stores have no output operands, and so need no further work. Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- tcg/optimize.c | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/tcg/optimize.c b/tcg/optimize.c index XXXXXXX..XXXXXXX 100644 --- a/tcg/optimize.c +++ b/tcg/optimize.c @@ -XXX,XX +XXX,XX @@ static bool fold_qemu_st(OptContext *ctx, TCGOp *op) { /* Opcodes that touch guest memory stop the mb optimization. */ ctx->prev_mb = NULL; - return false; + return true; } static bool fold_remainder(OptContext *ctx, TCGOp *op) @@ -XXX,XX +XXX,XX @@ static bool fold_tcg_st(OptContext *ctx, TCGOp *op) if (op->args[1] != tcgv_ptr_arg(tcg_env)) { remove_mem_copy_all(ctx); - return false; + return true; } switch (op->opc) { @@ -XXX,XX +XXX,XX @@ static bool fold_tcg_st(OptContext *ctx, TCGOp *op) g_assert_not_reached(); } remove_mem_copy_in(ctx, ofs, ofs + lm1); - return false; + return true; } static bool fold_tcg_st_memcopy(OptContext *ctx, TCGOp *op) @@ -XXX,XX +XXX,XX @@ static bool fold_tcg_st_memcopy(OptContext *ctx, TCGOp *op) TCGType type; if (op->args[1] != tcgv_ptr_arg(tcg_env)) { - fold_tcg_st(ctx, op); - return false; + return fold_tcg_st(ctx, op); } src = arg_temp(op->args[0]); @@ -XXX,XX +XXX,XX @@ static bool fold_tcg_st_memcopy(OptContext *ctx, TCGOp *op) last = ofs + tcg_type_size(type) - 1; remove_mem_copy_in(ctx, ofs, last); record_mem_copy(ctx, type, src, ofs, last); - return false; + return true; } static bool fold_xor(OptContext *ctx, TCGOp *op) -- 2.43.0 Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- tcg/optimize.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tcg/optimize.c b/tcg/optimize.c index XXXXXXX..XXXXXXX 100644 --- a/tcg/optimize.c +++ b/tcg/optimize.c @@ -XXX,XX +XXX,XX @@ static bool fold_remainder(OptContext *ctx, TCGOp *op) fold_xx_to_i(ctx, op, 0)) { return true; } - return false; + return finish_folding(ctx, op); } static bool fold_setcond_zmask(OptContext *ctx, TCGOp *op, bool neg) -- 2.43.0 Change return from bool to int; distinguish between complete folding, simplification, and no change. Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- tcg/optimize.c | 22 ++++++++++++++-------- 1 file changed, 14 insertions(+), 8 deletions(-) diff --git a/tcg/optimize.c b/tcg/optimize.c index XXXXXXX..XXXXXXX 100644 --- a/tcg/optimize.c +++ b/tcg/optimize.c @@ -XXX,XX +XXX,XX @@ static bool fold_remainder(OptContext *ctx, TCGOp *op) return finish_folding(ctx, op); } -static bool fold_setcond_zmask(OptContext *ctx, TCGOp *op, bool neg) +/* Return 1 if finished, -1 if simplified, 0 if unchanged. */ +static int fold_setcond_zmask(OptContext *ctx, TCGOp *op, bool neg) { uint64_t a_zmask, b_val; TCGCond cond; @@ -XXX,XX +XXX,XX @@ static bool fold_setcond_zmask(OptContext *ctx, TCGOp *op, bool neg) op->opc = xor_opc; op->args[2] = arg_new_constant(ctx, 1); } - return false; + return -1; } } - - return false; + return 0; } static void fold_setcond_tst_pow2(OptContext *ctx, TCGOp *op, bool neg) @@ -XXX,XX +XXX,XX @@ static bool fold_setcond(OptContext *ctx, TCGOp *op) return tcg_opt_gen_movi(ctx, op, op->args[0], i); } - if (fold_setcond_zmask(ctx, op, false)) { + i = fold_setcond_zmask(ctx, op, false); + if (i > 0) { return true; } - fold_setcond_tst_pow2(ctx, op, false); + if (i == 0) { + fold_setcond_tst_pow2(ctx, op, false); + } ctx->z_mask = 1; return false; @@ -XXX,XX +XXX,XX @@ static bool fold_negsetcond(OptContext *ctx, TCGOp *op) return tcg_opt_gen_movi(ctx, op, op->args[0], -i); } - if (fold_setcond_zmask(ctx, op, true)) { + i = fold_setcond_zmask(ctx, op, true); + if (i > 0) { return true; } - fold_setcond_tst_pow2(ctx, op, true); + if (i == 0) { + fold_setcond_tst_pow2(ctx, op, true); + } /* Value is {0,-1} so all bits are repetitions of the sign. */ ctx->s_mask = -1; -- 2.43.0 Avoid the use of the OptContext slots. Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- tcg/optimize.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/tcg/optimize.c b/tcg/optimize.c index XXXXXXX..XXXXXXX 100644 --- a/tcg/optimize.c +++ b/tcg/optimize.c @@ -XXX,XX +XXX,XX @@ static bool fold_setcond(OptContext *ctx, TCGOp *op) fold_setcond_tst_pow2(ctx, op, false); } - ctx->z_mask = 1; - return false; + return fold_masks_z(ctx, op, 1); } static bool fold_negsetcond(OptContext *ctx, TCGOp *op) -- 2.43.0 Avoid the use of the OptContext slots. Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- tcg/optimize.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/tcg/optimize.c b/tcg/optimize.c index XXXXXXX..XXXXXXX 100644 --- a/tcg/optimize.c +++ b/tcg/optimize.c @@ -XXX,XX +XXX,XX @@ static bool fold_negsetcond(OptContext *ctx, TCGOp *op) } /* Value is {0,-1} so all bits are repetitions of the sign. */ - ctx->s_mask = -1; - return false; + return fold_masks_s(ctx, op, -1); } static bool fold_setcond2(OptContext *ctx, TCGOp *op) -- 2.43.0 Avoid the use of the OptContext slots. Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- tcg/optimize.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/tcg/optimize.c b/tcg/optimize.c index XXXXXXX..XXXXXXX 100644 --- a/tcg/optimize.c +++ b/tcg/optimize.c @@ -XXX,XX +XXX,XX @@ static bool fold_setcond2(OptContext *ctx, TCGOp *op) return fold_setcond(ctx, op); } - ctx->z_mask = 1; - return false; + return fold_masks_z(ctx, op, 1); do_setcond_const: return tcg_opt_gen_movi(ctx, op, op->args[0], i); -- 2.43.0 Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- tcg/optimize.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tcg/optimize.c b/tcg/optimize.c index XXXXXXX..XXXXXXX 100644 --- a/tcg/optimize.c +++ b/tcg/optimize.c @@ -XXX,XX +XXX,XX @@ static bool fold_cmp_vec(OptContext *ctx, TCGOp *op) if (swap_commutative(NO_DEST, &op->args[1], &op->args[2])) { op->args[3] = tcg_swap_cond(op->args[3]); } - return false; + return finish_folding(ctx, op); } static bool fold_cmpsel_vec(OptContext *ctx, TCGOp *op) -- 2.43.0 Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- tcg/optimize.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tcg/optimize.c b/tcg/optimize.c index XXXXXXX..XXXXXXX 100644 --- a/tcg/optimize.c +++ b/tcg/optimize.c @@ -XXX,XX +XXX,XX @@ static bool fold_cmpsel_vec(OptContext *ctx, TCGOp *op) if (swap_commutative(op->args[0], &op->args[4], &op->args[3])) { op->args[5] = tcg_invert_cond(op->args[5]); } - return false; + return finish_folding(ctx, op); } static bool fold_sextract(OptContext *ctx, TCGOp *op) -- 2.43.0 Avoid the use of the OptContext slots. Find TempOptInfo once. Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- tcg/optimize.c | 24 +++++++++--------------- 1 file changed, 9 insertions(+), 15 deletions(-) diff --git a/tcg/optimize.c b/tcg/optimize.c index XXXXXXX..XXXXXXX 100644 --- a/tcg/optimize.c +++ b/tcg/optimize.c @@ -XXX,XX +XXX,XX @@ static bool fold_cmpsel_vec(OptContext *ctx, TCGOp *op) static bool fold_sextract(OptContext *ctx, TCGOp *op) { uint64_t z_mask, s_mask, s_mask_old; + TempOptInfo *t1 = arg_info(op->args[1]); int pos = op->args[2]; int len = op->args[3]; - if (arg_is_const(op->args[1])) { - uint64_t t; - - t = arg_info(op->args[1])->val; - t = sextract64(t, pos, len); - return tcg_opt_gen_movi(ctx, op, op->args[0], t); + if (ti_is_const(t1)) { + return tcg_opt_gen_movi(ctx, op, op->args[0], + sextract64(ti_const_val(t1), pos, len)); } - z_mask = arg_info(op->args[1])->z_mask; - z_mask = sextract64(z_mask, pos, len); - ctx->z_mask = z_mask; - - s_mask_old = arg_info(op->args[1])->s_mask; - s_mask = sextract64(s_mask_old, pos, len); - s_mask |= MAKE_64BIT_MASK(len, 64 - len); - ctx->s_mask = s_mask; + s_mask_old = t1->s_mask; + s_mask = s_mask_old >> pos; + s_mask |= -1ull << (len - 1); if (0 && pos == 0 && fold_affected_mask(ctx, op, s_mask & ~s_mask_old)) { return true; } - return fold_masks(ctx, op); + z_mask = sextract64(t1->z_mask, pos, len); + return fold_masks_zs(ctx, op, z_mask, s_mask); } static bool fold_shift(OptContext *ctx, TCGOp *op) -- 2.43.0 Avoid the use of the OptContext slots. Find TempOptInfo once. Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- tcg/optimize.c | 27 ++++++++++++++------------- 1 file changed, 14 insertions(+), 13 deletions(-) diff --git a/tcg/optimize.c b/tcg/optimize.c index XXXXXXX..XXXXXXX 100644 --- a/tcg/optimize.c +++ b/tcg/optimize.c @@ -XXX,XX +XXX,XX @@ static bool fold_sextract(OptContext *ctx, TCGOp *op) static bool fold_shift(OptContext *ctx, TCGOp *op) { uint64_t s_mask, z_mask, sign; + TempOptInfo *t1, *t2; if (fold_const2(ctx, op) || fold_ix_to_i(ctx, op, 0) || @@ -XXX,XX +XXX,XX @@ static bool fold_shift(OptContext *ctx, TCGOp *op) return true; } - s_mask = arg_info(op->args[1])->s_mask; - z_mask = arg_info(op->args[1])->z_mask; + t1 = arg_info(op->args[1]); + t2 = arg_info(op->args[2]); + s_mask = t1->s_mask; + z_mask = t1->z_mask; - if (arg_is_const(op->args[2])) { - int sh = arg_info(op->args[2])->val; - - ctx->z_mask = do_constant_folding(op->opc, ctx->type, z_mask, sh); + if (ti_is_const(t2)) { + int sh = ti_const_val(t2); + z_mask = do_constant_folding(op->opc, ctx->type, z_mask, sh); s_mask = do_constant_folding(op->opc, ctx->type, s_mask, sh); - return fold_masks(ctx, op); + return fold_masks_zs(ctx, op, z_mask, s_mask); } switch (op->opc) { @@ -XXX,XX +XXX,XX @@ static bool fold_shift(OptContext *ctx, TCGOp *op) * Arithmetic right shift will not reduce the number of * input sign repetitions. */ - ctx->s_mask = s_mask; - break; + return fold_masks_s(ctx, op, s_mask); CASE_OP_32_64(shr): /* * If the sign bit is known zero, then logical right shift - * will not reduced the number of input sign repetitions. + * will not reduce the number of input sign repetitions. */ - sign = (s_mask & -s_mask) >> 1; + sign = -s_mask; if (sign && !(z_mask & sign)) { - ctx->s_mask = s_mask; + return fold_masks_s(ctx, op, s_mask); } break; default: break; } - return false; + return finish_folding(ctx, op); } static bool fold_sub_to_neg(OptContext *ctx, TCGOp *op) -- 2.43.0 Merge the two conditions, sign != 0 && !(z_mask & sign), by testing ~z_mask & sign. If sign == 0, the logical and will produce false. Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- tcg/optimize.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/tcg/optimize.c b/tcg/optimize.c index XXXXXXX..XXXXXXX 100644 --- a/tcg/optimize.c +++ b/tcg/optimize.c @@ -XXX,XX +XXX,XX @@ static bool fold_sextract(OptContext *ctx, TCGOp *op) static bool fold_shift(OptContext *ctx, TCGOp *op) { - uint64_t s_mask, z_mask, sign; + uint64_t s_mask, z_mask; TempOptInfo *t1, *t2; if (fold_const2(ctx, op) || @@ -XXX,XX +XXX,XX @@ static bool fold_shift(OptContext *ctx, TCGOp *op) * If the sign bit is known zero, then logical right shift * will not reduce the number of input sign repetitions. */ - sign = -s_mask; - if (sign && !(z_mask & sign)) { + if (~z_mask & -s_mask) { return fold_masks_s(ctx, op, s_mask); } break; -- 2.43.0 Duplicate fold_sub_vec into fold_sub instead of calling it, now that fold_sub_vec always returns true. Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- tcg/optimize.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/tcg/optimize.c b/tcg/optimize.c index XXXXXXX..XXXXXXX 100644 --- a/tcg/optimize.c +++ b/tcg/optimize.c @@ -XXX,XX +XXX,XX @@ static bool fold_sub_vec(OptContext *ctx, TCGOp *op) fold_sub_to_neg(ctx, op)) { return true; } - return false; + return finish_folding(ctx, op); } static bool fold_sub(OptContext *ctx, TCGOp *op) { - if (fold_const2(ctx, op) || fold_sub_vec(ctx, op)) { + if (fold_const2(ctx, op) || + fold_xx_to_i(ctx, op, 0) || + fold_xi_to_x(ctx, op, 0) || + fold_sub_to_neg(ctx, op)) { return true; } @@ -XXX,XX +XXX,XX @@ static bool fold_sub(OptContext *ctx, TCGOp *op) ? INDEX_op_add_i32 : INDEX_op_add_i64); op->args[2] = arg_new_constant(ctx, -val); } - return false; + return finish_folding(ctx, op); } static bool fold_sub2(OptContext *ctx, TCGOp *op) -- 2.43.0 Avoid the use of the OptContext slots. Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- tcg/optimize.c | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/tcg/optimize.c b/tcg/optimize.c index XXXXXXX..XXXXXXX 100644 --- a/tcg/optimize.c +++ b/tcg/optimize.c @@ -XXX,XX +XXX,XX @@ static bool fold_sub2(OptContext *ctx, TCGOp *op) static bool fold_tcg_ld(OptContext *ctx, TCGOp *op) { + uint64_t z_mask = -1, s_mask = 0; + /* We can't do any folding with a load, but we can record bits. */ switch (op->opc) { CASE_OP_32_64(ld8s): - ctx->s_mask = MAKE_64BIT_MASK(8, 56); + s_mask = INT8_MIN; break; CASE_OP_32_64(ld8u): - ctx->z_mask = MAKE_64BIT_MASK(0, 8); + z_mask = MAKE_64BIT_MASK(0, 8); break; CASE_OP_32_64(ld16s): - ctx->s_mask = MAKE_64BIT_MASK(16, 48); + s_mask = INT16_MIN; break; CASE_OP_32_64(ld16u): - ctx->z_mask = MAKE_64BIT_MASK(0, 16); + z_mask = MAKE_64BIT_MASK(0, 16); break; case INDEX_op_ld32s_i64: - ctx->s_mask = MAKE_64BIT_MASK(32, 32); + s_mask = INT32_MIN; break; case INDEX_op_ld32u_i64: - ctx->z_mask = MAKE_64BIT_MASK(0, 32); + z_mask = MAKE_64BIT_MASK(0, 32); break; default: g_assert_not_reached(); } - return false; + return fold_masks_zs(ctx, op, z_mask, s_mask); } static bool fold_tcg_ld_memcopy(OptContext *ctx, TCGOp *op) -- 2.43.0 Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- tcg/optimize.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tcg/optimize.c b/tcg/optimize.c index XXXXXXX..XXXXXXX 100644 --- a/tcg/optimize.c +++ b/tcg/optimize.c @@ -XXX,XX +XXX,XX @@ static bool fold_tcg_ld_memcopy(OptContext *ctx, TCGOp *op) TCGType type; if (op->args[1] != tcgv_ptr_arg(tcg_env)) { - return false; + return finish_folding(ctx, op); } type = ctx->type; -- 2.43.0 Avoid the use of the OptContext slots. Find TempOptInfo once. Remove fold_masks as the function becomes unused. Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- tcg/optimize.c | 18 ++++++++---------- 1 file changed, 8 insertions(+), 10 deletions(-) diff --git a/tcg/optimize.c b/tcg/optimize.c index XXXXXXX..XXXXXXX 100644 --- a/tcg/optimize.c +++ b/tcg/optimize.c @@ -XXX,XX +XXX,XX @@ static bool fold_masks_s(OptContext *ctx, TCGOp *op, uint64_t s_mask) return fold_masks_zs(ctx, op, -1, s_mask); } -static bool fold_masks(OptContext *ctx, TCGOp *op) -{ - return fold_masks_zs(ctx, op, ctx->z_mask, ctx->s_mask); -} - /* * An "affected" mask bit is 0 if and only if the result is identical * to the first input. Thus if the entire mask is 0, the operation @@ -XXX,XX +XXX,XX @@ static bool fold_tcg_st_memcopy(OptContext *ctx, TCGOp *op) static bool fold_xor(OptContext *ctx, TCGOp *op) { + uint64_t z_mask, s_mask; + TempOptInfo *t1, *t2; + if (fold_const2_commutative(ctx, op) || fold_xx_to_i(ctx, op, 0) || fold_xi_to_x(ctx, op, 0) || @@ -XXX,XX +XXX,XX @@ static bool fold_xor(OptContext *ctx, TCGOp *op) return true; } - ctx->z_mask = arg_info(op->args[1])->z_mask - | arg_info(op->args[2])->z_mask; - ctx->s_mask = arg_info(op->args[1])->s_mask - & arg_info(op->args[2])->s_mask; - return fold_masks(ctx, op); + t1 = arg_info(op->args[1]); + t2 = arg_info(op->args[2]); + z_mask = t1->z_mask | t2->z_mask; + s_mask = t1->s_mask & t2->s_mask; + return fold_masks_zs(ctx, op, z_mask, s_mask); } static bool fold_bitsel_vec(OptContext *ctx, TCGOp *op) -- 2.43.0 Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- tcg/optimize.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tcg/optimize.c b/tcg/optimize.c index XXXXXXX..XXXXXXX 100644 --- a/tcg/optimize.c +++ b/tcg/optimize.c @@ -XXX,XX +XXX,XX @@ static bool fold_bitsel_vec(OptContext *ctx, TCGOp *op) return fold_orc(ctx, op); } } - return false; + return finish_folding(ctx, op); } /* Propagate constants and copies, fold constant expressions. */ -- 2.43.0 All non-default cases now finish folding within each function. Do the same with the default case and assert it is done after. Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- tcg/optimize.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/tcg/optimize.c b/tcg/optimize.c index XXXXXXX..XXXXXXX 100644 --- a/tcg/optimize.c +++ b/tcg/optimize.c @@ -XXX,XX +XXX,XX @@ void tcg_optimize(TCGContext *s) done = true; break; default: + done = finish_folding(&ctx, op); break; } - - if (!done) { - finish_folding(&ctx, op); - } + tcg_debug_assert(done); } } -- 2.43.0 All mask setting is now done with parameters via fold_masks_*. Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- tcg/optimize.c | 13 ------------- 1 file changed, 13 deletions(-) diff --git a/tcg/optimize.c b/tcg/optimize.c index XXXXXXX..XXXXXXX 100644 --- a/tcg/optimize.c +++ b/tcg/optimize.c @@ -XXX,XX +XXX,XX @@ typedef struct OptContext { QSIMPLEQ_HEAD(, MemCopyInfo) mem_free; /* In flight values from optimization. */ - uint64_t z_mask; /* mask bit is 0 iff value bit is 0 */ - uint64_t s_mask; /* mask bit is 1 if value bit matches msb */ TCGType type; } OptContext; @@ -XXX,XX +XXX,XX @@ static bool finish_folding(OptContext *ctx, TCGOp *op) for (i = 0; i < nb_oargs; i++) { TCGTemp *ts = arg_temp(op->args[i]); reset_ts(ctx, ts); - /* - * Save the corresponding known-zero/sign bits mask for the - * first output argument (only one supported so far). - */ - if (i == 0) { - ts_info(ts)->z_mask = ctx->z_mask; - } } return true; } @@ -XXX,XX +XXX,XX @@ void tcg_optimize(TCGContext *s) ctx.type = TCG_TYPE_I32; } - /* Assume all bits affected, no bits known zero, no sign reps. */ - ctx.z_mask = -1; - ctx.s_mask = 0; - /* * Process each opcode. * Sorted alphabetically by opcode as much as possible. -- 2.43.0 All instances of s_mask have been converted to the new representation. We can now re-enable usage. Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- tcg/optimize.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tcg/optimize.c b/tcg/optimize.c index XXXXXXX..XXXXXXX 100644 --- a/tcg/optimize.c +++ b/tcg/optimize.c @@ -XXX,XX +XXX,XX @@ static bool fold_exts(OptContext *ctx, TCGOp *op) g_assert_not_reached(); } - if (0 && !type_change && fold_affected_mask(ctx, op, s_mask & ~s_mask_old)) { + if (!type_change && fold_affected_mask(ctx, op, s_mask & ~s_mask_old)) { return true; } @@ -XXX,XX +XXX,XX @@ static bool fold_sextract(OptContext *ctx, TCGOp *op) s_mask = s_mask_old >> pos; s_mask |= -1ull << (len - 1); - if (0 && pos == 0 && fold_affected_mask(ctx, op, s_mask & ~s_mask_old)) { + if (pos == 0 && fold_affected_mask(ctx, op, s_mask & ~s_mask_old)) { return true; } -- 2.43.0 The big comment just above says functions should be sorted. Add forward declarations as needed. Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- tcg/optimize.c | 114 +++++++++++++++++++++++++------------------------ 1 file changed, 59 insertions(+), 55 deletions(-) diff --git a/tcg/optimize.c b/tcg/optimize.c index XXXXXXX..XXXXXXX 100644 --- a/tcg/optimize.c +++ b/tcg/optimize.c @@ -XXX,XX +XXX,XX @@ static bool fold_xx_to_x(OptContext *ctx, TCGOp *op) * 3) those that produce information about the result value. */ +static bool fold_or(OptContext *ctx, TCGOp *op); +static bool fold_orc(OptContext *ctx, TCGOp *op); +static bool fold_xor(OptContext *ctx, TCGOp *op); + static bool fold_add(OptContext *ctx, TCGOp *op) { if (fold_const2_commutative(ctx, op) || @@ -XXX,XX +XXX,XX @@ static bool fold_andc(OptContext *ctx, TCGOp *op) return fold_masks_zs(ctx, op, z_mask, s_mask); } +static bool fold_bitsel_vec(OptContext *ctx, TCGOp *op) +{ + /* If true and false values are the same, eliminate the cmp. */ + if (args_are_copies(op->args[2], op->args[3])) { + return tcg_opt_gen_mov(ctx, op, op->args[0], op->args[2]); + } + + if (arg_is_const(op->args[2]) && arg_is_const(op->args[3])) { + uint64_t tv = arg_info(op->args[2])->val; + uint64_t fv = arg_info(op->args[3])->val; + + if (tv == -1 && fv == 0) { + return tcg_opt_gen_mov(ctx, op, op->args[0], op->args[1]); + } + if (tv == 0 && fv == -1) { + if (TCG_TARGET_HAS_not_vec) { + op->opc = INDEX_op_not_vec; + return fold_not(ctx, op); + } else { + op->opc = INDEX_op_xor_vec; + op->args[2] = arg_new_constant(ctx, -1); + return fold_xor(ctx, op); + } + } + } + if (arg_is_const(op->args[2])) { + uint64_t tv = arg_info(op->args[2])->val; + if (tv == -1) { + op->opc = INDEX_op_or_vec; + op->args[2] = op->args[3]; + return fold_or(ctx, op); + } + if (tv == 0 && TCG_TARGET_HAS_andc_vec) { + op->opc = INDEX_op_andc_vec; + op->args[2] = op->args[1]; + op->args[1] = op->args[3]; + return fold_andc(ctx, op); + } + } + if (arg_is_const(op->args[3])) { + uint64_t fv = arg_info(op->args[3])->val; + if (fv == 0) { + op->opc = INDEX_op_and_vec; + return fold_and(ctx, op); + } + if (fv == -1 && TCG_TARGET_HAS_orc_vec) { + op->opc = INDEX_op_orc_vec; + op->args[2] = op->args[1]; + op->args[1] = op->args[3]; + return fold_orc(ctx, op); + } + } + return finish_folding(ctx, op); +} + static bool fold_brcond(OptContext *ctx, TCGOp *op) { int i = do_constant_folding_cond1(ctx, op, NO_DEST, &op->args[0], @@ -XXX,XX +XXX,XX @@ static bool fold_xor(OptContext *ctx, TCGOp *op) return fold_masks_zs(ctx, op, z_mask, s_mask); } -static bool fold_bitsel_vec(OptContext *ctx, TCGOp *op) -{ - /* If true and false values are the same, eliminate the cmp. */ - if (args_are_copies(op->args[2], op->args[3])) { - return tcg_opt_gen_mov(ctx, op, op->args[0], op->args[2]); - } - - if (arg_is_const(op->args[2]) && arg_is_const(op->args[3])) { - uint64_t tv = arg_info(op->args[2])->val; - uint64_t fv = arg_info(op->args[3])->val; - - if (tv == -1 && fv == 0) { - return tcg_opt_gen_mov(ctx, op, op->args[0], op->args[1]); - } - if (tv == 0 && fv == -1) { - if (TCG_TARGET_HAS_not_vec) { - op->opc = INDEX_op_not_vec; - return fold_not(ctx, op); - } else { - op->opc = INDEX_op_xor_vec; - op->args[2] = arg_new_constant(ctx, -1); - return fold_xor(ctx, op); - } - } - } - if (arg_is_const(op->args[2])) { - uint64_t tv = arg_info(op->args[2])->val; - if (tv == -1) { - op->opc = INDEX_op_or_vec; - op->args[2] = op->args[3]; - return fold_or(ctx, op); - } - if (tv == 0 && TCG_TARGET_HAS_andc_vec) { - op->opc = INDEX_op_andc_vec; - op->args[2] = op->args[1]; - op->args[1] = op->args[3]; - return fold_andc(ctx, op); - } - } - if (arg_is_const(op->args[3])) { - uint64_t fv = arg_info(op->args[3])->val; - if (fv == 0) { - op->opc = INDEX_op_and_vec; - return fold_and(ctx, op); - } - if (fv == -1 && TCG_TARGET_HAS_orc_vec) { - op->opc = INDEX_op_orc_vec; - op->args[2] = op->args[1]; - op->args[1] = op->args[3]; - return fold_orc(ctx, op); - } - } - return finish_folding(ctx, op); -} - /* Propagate constants and copies, fold constant expressions. */ void tcg_optimize(TCGContext *s) { -- 2.43.0 The big comment just above says functions should be sorted. Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- tcg/optimize.c | 60 +++++++++++++++++++++++++------------------------- 1 file changed, 30 insertions(+), 30 deletions(-) diff --git a/tcg/optimize.c b/tcg/optimize.c index XXXXXXX..XXXXXXX 100644 --- a/tcg/optimize.c +++ b/tcg/optimize.c @@ -XXX,XX +XXX,XX @@ static bool fold_call(OptContext *ctx, TCGOp *op) return true; } +static bool fold_cmp_vec(OptContext *ctx, TCGOp *op) +{ + /* Canonicalize the comparison to put immediate second. */ + if (swap_commutative(NO_DEST, &op->args[1], &op->args[2])) { + op->args[3] = tcg_swap_cond(op->args[3]); + } + return finish_folding(ctx, op); +} + +static bool fold_cmpsel_vec(OptContext *ctx, TCGOp *op) +{ + /* If true and false values are the same, eliminate the cmp. */ + if (args_are_copies(op->args[3], op->args[4])) { + return tcg_opt_gen_mov(ctx, op, op->args[0], op->args[3]); + } + + /* Canonicalize the comparison to put immediate second. */ + if (swap_commutative(NO_DEST, &op->args[1], &op->args[2])) { + op->args[5] = tcg_swap_cond(op->args[5]); + } + /* + * Canonicalize the "false" input reg to match the destination, + * so that the tcg backend can implement "move if true". + */ + if (swap_commutative(op->args[0], &op->args[4], &op->args[3])) { + op->args[5] = tcg_invert_cond(op->args[5]); + } + return finish_folding(ctx, op); +} + static bool fold_count_zeros(OptContext *ctx, TCGOp *op) { uint64_t z_mask, s_mask; @@ -XXX,XX +XXX,XX @@ static bool fold_setcond2(OptContext *ctx, TCGOp *op) return tcg_opt_gen_movi(ctx, op, op->args[0], i); } -static bool fold_cmp_vec(OptContext *ctx, TCGOp *op) -{ - /* Canonicalize the comparison to put immediate second. */ - if (swap_commutative(NO_DEST, &op->args[1], &op->args[2])) { - op->args[3] = tcg_swap_cond(op->args[3]); - } - return finish_folding(ctx, op); -} - -static bool fold_cmpsel_vec(OptContext *ctx, TCGOp *op) -{ - /* If true and false values are the same, eliminate the cmp. */ - if (args_are_copies(op->args[3], op->args[4])) { - return tcg_opt_gen_mov(ctx, op, op->args[0], op->args[3]); - } - - /* Canonicalize the comparison to put immediate second. */ - if (swap_commutative(NO_DEST, &op->args[1], &op->args[2])) { - op->args[5] = tcg_swap_cond(op->args[5]); - } - /* - * Canonicalize the "false" input reg to match the destination, - * so that the tcg backend can implement "move if true". - */ - if (swap_commutative(op->args[0], &op->args[4], &op->args[3])) { - op->args[5] = tcg_invert_cond(op->args[5]); - } - return finish_folding(ctx, op); -} - static bool fold_sextract(OptContext *ctx, TCGOp *op) { uint64_t z_mask, s_mask, s_mask_old; -- 2.43.0 We currently have a flag, float_muladd_halve_result, to scale the result by 2**-1. Extend this to handle arbitrary scaling. Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- include/fpu/softfloat.h | 6 ++++ fpu/softfloat.c | 58 ++++++++++++++++++++++----------------- fpu/softfloat-parts.c.inc | 7 +++-- 3 files changed, 44 insertions(+), 27 deletions(-) diff --git a/include/fpu/softfloat.h b/include/fpu/softfloat.h index XXXXXXX..XXXXXXX 100644 --- a/include/fpu/softfloat.h +++ b/include/fpu/softfloat.h @@ -XXX,XX +XXX,XX @@ float16 float16_add(float16, float16, float_status *status); float16 float16_sub(float16, float16, float_status *status); float16 float16_mul(float16, float16, float_status *status); float16 float16_muladd(float16, float16, float16, int, float_status *status); +float16 float16_muladd_scalbn(float16, float16, float16, + int, int, float_status *status); float16 float16_div(float16, float16, float_status *status); float16 float16_scalbn(float16, int, float_status *status); float16 float16_min(float16, float16, float_status *status); @@ -XXX,XX +XXX,XX @@ float32 float32_mul(float32, float32, float_status *status); float32 float32_div(float32, float32, float_status *status); float32 float32_rem(float32, float32, float_status *status); float32 float32_muladd(float32, float32, float32, int, float_status *status); +float32 float32_muladd_scalbn(float32, float32, float32, + int, int, float_status *status); float32 float32_sqrt(float32, float_status *status); float32 float32_exp2(float32, float_status *status); float32 float32_log2(float32, float_status *status); @@ -XXX,XX +XXX,XX @@ float64 float64_mul(float64, float64, float_status *status); float64 float64_div(float64, float64, float_status *status); float64 float64_rem(float64, float64, float_status *status); float64 float64_muladd(float64, float64, float64, int, float_status *status); +float64 float64_muladd_scalbn(float64, float64, float64, + int, int, float_status *status); float64 float64_sqrt(float64, float_status *status); float64 float64_log2(float64, float_status *status); FloatRelation float64_compare(float64, float64, float_status *status); diff --git a/fpu/softfloat.c b/fpu/softfloat.c index XXXXXXX..XXXXXXX 100644 --- a/fpu/softfloat.c +++ b/fpu/softfloat.c @@ -XXX,XX +XXX,XX @@ static FloatParts128 *parts128_mul(FloatParts128 *a, FloatParts128 *b, #define parts_mul(A, B, S) \ PARTS_GENERIC_64_128(mul, A)(A, B, S) -static FloatParts64 *parts64_muladd(FloatParts64 *a, FloatParts64 *b, - FloatParts64 *c, int flags, - float_status *s); -static FloatParts128 *parts128_muladd(FloatParts128 *a, FloatParts128 *b, - FloatParts128 *c, int flags, - float_status *s); +static FloatParts64 *parts64_muladd_scalbn(FloatParts64 *a, FloatParts64 *b, + FloatParts64 *c, int scale, + int flags, float_status *s); +static FloatParts128 *parts128_muladd_scalbn(FloatParts128 *a, FloatParts128 *b, + FloatParts128 *c, int scale, + int flags, float_status *s); -#define parts_muladd(A, B, C, Z, S) \ - PARTS_GENERIC_64_128(muladd, A)(A, B, C, Z, S) +#define parts_muladd_scalbn(A, B, C, Z, Y, S) \ + PARTS_GENERIC_64_128(muladd_scalbn, A)(A, B, C, Z, Y, S) static FloatParts64 *parts64_div(FloatParts64 *a, FloatParts64 *b, float_status *s); @@ -XXX,XX +XXX,XX @@ floatx80_mul(floatx80 a, floatx80 b, float_status *status) * Fused multiply-add */ -float16 QEMU_FLATTEN float16_muladd(float16 a, float16 b, float16 c, - int flags, float_status *status) +float16 QEMU_FLATTEN +float16_muladd_scalbn(float16 a, float16 b, float16 c, + int scale, int flags, float_status *status) { FloatParts64 pa, pb, pc, *pr; float16_unpack_canonical(&pa, a, status); float16_unpack_canonical(&pb, b, status); float16_unpack_canonical(&pc, c, status); - pr = parts_muladd(&pa, &pb, &pc, flags, status); + pr = parts_muladd_scalbn(&pa, &pb, &pc, scale, flags, status); return float16_round_pack_canonical(pr, status); } -static float32 QEMU_SOFTFLOAT_ATTR -soft_f32_muladd(float32 a, float32 b, float32 c, int flags, - float_status *status) +float16 float16_muladd(float16 a, float16 b, float16 c, + int flags, float_status *status) +{ + return float16_muladd_scalbn(a, b, c, 0, flags, status); +} + +float32 QEMU_SOFTFLOAT_ATTR +float32_muladd_scalbn(float32 a, float32 b, float32 c, + int scale, int flags, float_status *status) { FloatParts64 pa, pb, pc, *pr; float32_unpack_canonical(&pa, a, status); float32_unpack_canonical(&pb, b, status); float32_unpack_canonical(&pc, c, status); - pr = parts_muladd(&pa, &pb, &pc, flags, status); + pr = parts_muladd_scalbn(&pa, &pb, &pc, scale, flags, status); return float32_round_pack_canonical(pr, status); } -static float64 QEMU_SOFTFLOAT_ATTR -soft_f64_muladd(float64 a, float64 b, float64 c, int flags, - float_status *status) +float64 QEMU_SOFTFLOAT_ATTR +float64_muladd_scalbn(float64 a, float64 b, float64 c, + int scale, int flags, float_status *status) { FloatParts64 pa, pb, pc, *pr; float64_unpack_canonical(&pa, a, status); float64_unpack_canonical(&pb, b, status); float64_unpack_canonical(&pc, c, status); - pr = parts_muladd(&pa, &pb, &pc, flags, status); + pr = parts_muladd_scalbn(&pa, &pb, &pc, scale, flags, status); return float64_round_pack_canonical(pr, status); } @@ -XXX,XX +XXX,XX @@ float32_muladd(float32 xa, float32 xb, float32 xc, int flags, float_status *s) return ur.s; soft: - return soft_f32_muladd(ua.s, ub.s, uc.s, flags, s); + return float32_muladd_scalbn(ua.s, ub.s, uc.s, 0, flags, s); } float64 QEMU_FLATTEN @@ -XXX,XX +XXX,XX @@ float64_muladd(float64 xa, float64 xb, float64 xc, int flags, float_status *s) return ur.s; soft: - return soft_f64_muladd(ua.s, ub.s, uc.s, flags, s); + return float64_muladd_scalbn(ua.s, ub.s, uc.s, 0, flags, s); } float64 float64r32_muladd(float64 a, float64 b, float64 c, @@ -XXX,XX +XXX,XX @@ float64 float64r32_muladd(float64 a, float64 b, float64 c, float64_unpack_canonical(&pa, a, status); float64_unpack_canonical(&pb, b, status); float64_unpack_canonical(&pc, c, status); - pr = parts_muladd(&pa, &pb, &pc, flags, status); + pr = parts_muladd_scalbn(&pa, &pb, &pc, 0, flags, status); return float64r32_round_pack_canonical(pr, status); } @@ -XXX,XX +XXX,XX @@ bfloat16 QEMU_FLATTEN bfloat16_muladd(bfloat16 a, bfloat16 b, bfloat16 c, bfloat16_unpack_canonical(&pa, a, status); bfloat16_unpack_canonical(&pb, b, status); bfloat16_unpack_canonical(&pc, c, status); - pr = parts_muladd(&pa, &pb, &pc, flags, status); + pr = parts_muladd_scalbn(&pa, &pb, &pc, 0, flags, status); return bfloat16_round_pack_canonical(pr, status); } @@ -XXX,XX +XXX,XX @@ float128 QEMU_FLATTEN float128_muladd(float128 a, float128 b, float128 c, float128_unpack_canonical(&pa, a, status); float128_unpack_canonical(&pb, b, status); float128_unpack_canonical(&pc, c, status); - pr = parts_muladd(&pa, &pb, &pc, flags, status); + pr = parts_muladd_scalbn(&pa, &pb, &pc, 0, flags, status); return float128_round_pack_canonical(pr, status); } @@ -XXX,XX +XXX,XX @@ float32 float32_exp2(float32 a, float_status *status) float64_unpack_canonical(&rp, float64_one, status); for (i = 0 ; i < 15 ; i++) { + float64_unpack_canonical(&tp, float32_exp2_coefficients[i], status); - rp = *parts_muladd(&tp, &xnp, &rp, 0, status); + rp = *parts_muladd_scalbn(&tp, &xnp, &rp, 0, 0, status); xnp = *parts_mul(&xnp, &xp, status); } diff --git a/fpu/softfloat-parts.c.inc b/fpu/softfloat-parts.c.inc index XXXXXXX..XXXXXXX 100644 --- a/fpu/softfloat-parts.c.inc +++ b/fpu/softfloat-parts.c.inc @@ -XXX,XX +XXX,XX @@ static FloatPartsN *partsN(mul)(FloatPartsN *a, FloatPartsN *b, * Requires A and C extracted into a double-sized structure to provide the * extra space for the widening multiply. */ -static FloatPartsN *partsN(muladd)(FloatPartsN *a, FloatPartsN *b, - FloatPartsN *c, int flags, float_status *s) +static FloatPartsN *partsN(muladd_scalbn)(FloatPartsN *a, FloatPartsN *b, + FloatPartsN *c, int scale, + int flags, float_status *s) { int ab_mask, abc_mask; FloatPartsW p_widen, c_widen; @@ -XXX,XX +XXX,XX @@ static FloatPartsN *partsN(muladd)(FloatPartsN *a, FloatPartsN *b, a->exp = p_widen.exp; return_normal: + /* TODO: Replace all use of float_muladd_halve_result with scale. */ if (flags & float_muladd_halve_result) { a->exp -= 1; } + a->exp += scale; finish_sign: if (flags & float_muladd_negate_result) { a->sign ^= 1; -- 2.43.0 Use the scalbn interface instead of float_muladd_halve_result. Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- target/arm/tcg/helper-a64.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/target/arm/tcg/helper-a64.c b/target/arm/tcg/helper-a64.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/tcg/helper-a64.c +++ b/target/arm/tcg/helper-a64.c @@ -XXX,XX +XXX,XX @@ uint32_t HELPER(rsqrtsf_f16)(uint32_t a, uint32_t b, float_status *fpst) (float16_is_infinity(b) && float16_is_zero(a))) { return float16_one_point_five; } - return float16_muladd(a, b, float16_three, float_muladd_halve_result, fpst); + return float16_muladd_scalbn(a, b, float16_three, -1, 0, fpst); } float32 HELPER(rsqrtsf_f32)(float32 a, float32 b, float_status *fpst) @@ -XXX,XX +XXX,XX @@ float32 HELPER(rsqrtsf_f32)(float32 a, float32 b, float_status *fpst) (float32_is_infinity(b) && float32_is_zero(a))) { return float32_one_point_five; } - return float32_muladd(a, b, float32_three, float_muladd_halve_result, fpst); + return float32_muladd_scalbn(a, b, float32_three, -1, 0, fpst); } float64 HELPER(rsqrtsf_f64)(float64 a, float64 b, float_status *fpst) @@ -XXX,XX +XXX,XX @@ float64 HELPER(rsqrtsf_f64)(float64 a, float64 b, float_status *fpst) (float64_is_infinity(b) && float64_is_zero(a))) { return float64_one_point_five; } - return float64_muladd(a, b, float64_three, float_muladd_halve_result, fpst); + return float64_muladd_scalbn(a, b, float64_three, -1, 0, fpst); } /* Floating-point reciprocal exponent - see FPRecpX in ARM ARM */ -- 2.43.0 Use the scalbn interface instead of float_muladd_halve_result. Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- target/sparc/helper.h | 4 +- target/sparc/fop_helper.c | 8 ++-- target/sparc/translate.c | 80 +++++++++++++++++++++++---------------- 3 files changed, 54 insertions(+), 38 deletions(-) diff --git a/target/sparc/helper.h b/target/sparc/helper.h index XXXXXXX..XXXXXXX 100644 --- a/target/sparc/helper.h +++ b/target/sparc/helper.h @@ -XXX,XX +XXX,XX @@ DEF_HELPER_FLAGS_3(faddd, TCG_CALL_NO_WG, f64, env, f64, f64) DEF_HELPER_FLAGS_3(fsubd, TCG_CALL_NO_WG, f64, env, f64, f64) DEF_HELPER_FLAGS_3(fmuld, TCG_CALL_NO_WG, f64, env, f64, f64) DEF_HELPER_FLAGS_3(fdivd, TCG_CALL_NO_WG, f64, env, f64, f64) -DEF_HELPER_FLAGS_5(fmaddd, TCG_CALL_NO_WG, f64, env, f64, f64, f64, i32) +DEF_HELPER_FLAGS_6(fmaddd, TCG_CALL_NO_WG, f64, env, f64, f64, f64, s32, i32) DEF_HELPER_FLAGS_3(fnaddd, TCG_CALL_NO_WG, f64, env, f64, f64) DEF_HELPER_FLAGS_3(fnmuld, TCG_CALL_NO_WG, f64, env, f64, f64) @@ -XXX,XX +XXX,XX @@ DEF_HELPER_FLAGS_3(fadds, TCG_CALL_NO_WG, f32, env, f32, f32) DEF_HELPER_FLAGS_3(fsubs, TCG_CALL_NO_WG, f32, env, f32, f32) DEF_HELPER_FLAGS_3(fmuls, TCG_CALL_NO_WG, f32, env, f32, f32) DEF_HELPER_FLAGS_3(fdivs, TCG_CALL_NO_WG, f32, env, f32, f32) -DEF_HELPER_FLAGS_5(fmadds, TCG_CALL_NO_WG, f32, env, f32, f32, f32, i32) +DEF_HELPER_FLAGS_6(fmadds, TCG_CALL_NO_WG, f32, env, f32, f32, f32, s32, i32) DEF_HELPER_FLAGS_3(fnadds, TCG_CALL_NO_WG, f32, env, f32, f32) DEF_HELPER_FLAGS_3(fnmuls, TCG_CALL_NO_WG, f32, env, f32, f32) diff --git a/target/sparc/fop_helper.c b/target/sparc/fop_helper.c index XXXXXXX..XXXXXXX 100644 --- a/target/sparc/fop_helper.c +++ b/target/sparc/fop_helper.c @@ -XXX,XX +XXX,XX @@ Int128 helper_fsqrtq(CPUSPARCState *env, Int128 src) } float32 helper_fmadds(CPUSPARCState *env, float32 s1, - float32 s2, float32 s3, uint32_t op) + float32 s2, float32 s3, int32_t sc, uint32_t op) { - float32 ret = float32_muladd(s1, s2, s3, op, &env->fp_status); + float32 ret = float32_muladd_scalbn(s1, s2, s3, sc, op, &env->fp_status); check_ieee_exceptions(env, GETPC()); return ret; } float64 helper_fmaddd(CPUSPARCState *env, float64 s1, - float64 s2, float64 s3, uint32_t op) + float64 s2, float64 s3, int32_t sc, uint32_t op) { - float64 ret = float64_muladd(s1, s2, s3, op, &env->fp_status); + float64 ret = float64_muladd_scalbn(s1, s2, s3, sc, op, &env->fp_status); check_ieee_exceptions(env, GETPC()); return ret; } diff --git a/target/sparc/translate.c b/target/sparc/translate.c index XXXXXXX..XXXXXXX 100644 --- a/target/sparc/translate.c +++ b/target/sparc/translate.c @@ -XXX,XX +XXX,XX @@ static void gen_op_fabsq(TCGv_i128 dst, TCGv_i128 src) static void gen_op_fmadds(TCGv_i32 d, TCGv_i32 s1, TCGv_i32 s2, TCGv_i32 s3) { - gen_helper_fmadds(d, tcg_env, s1, s2, s3, tcg_constant_i32(0)); + TCGv_i32 z = tcg_constant_i32(0); + gen_helper_fmadds(d, tcg_env, s1, s2, s3, z, z); } static void gen_op_fmaddd(TCGv_i64 d, TCGv_i64 s1, TCGv_i64 s2, TCGv_i64 s3) { - gen_helper_fmaddd(d, tcg_env, s1, s2, s3, tcg_constant_i32(0)); + TCGv_i32 z = tcg_constant_i32(0); + gen_helper_fmaddd(d, tcg_env, s1, s2, s3, z, z); } static void gen_op_fmsubs(TCGv_i32 d, TCGv_i32 s1, TCGv_i32 s2, TCGv_i32 s3) { - int op = float_muladd_negate_c; - gen_helper_fmadds(d, tcg_env, s1, s2, s3, tcg_constant_i32(op)); + TCGv_i32 z = tcg_constant_i32(0); + TCGv_i32 op = tcg_constant_i32(float_muladd_negate_c); + gen_helper_fmadds(d, tcg_env, s1, s2, s3, z, op); } static void gen_op_fmsubd(TCGv_i64 d, TCGv_i64 s1, TCGv_i64 s2, TCGv_i64 s3) { - int op = float_muladd_negate_c; - gen_helper_fmaddd(d, tcg_env, s1, s2, s3, tcg_constant_i32(op)); + TCGv_i32 z = tcg_constant_i32(0); + TCGv_i32 op = tcg_constant_i32(float_muladd_negate_c); + gen_helper_fmaddd(d, tcg_env, s1, s2, s3, z, op); } static void gen_op_fnmsubs(TCGv_i32 d, TCGv_i32 s1, TCGv_i32 s2, TCGv_i32 s3) { - int op = float_muladd_negate_c | float_muladd_negate_result; - gen_helper_fmadds(d, tcg_env, s1, s2, s3, tcg_constant_i32(op)); + TCGv_i32 z = tcg_constant_i32(0); + TCGv_i32 op = tcg_constant_i32(float_muladd_negate_c | + float_muladd_negate_result); + gen_helper_fmadds(d, tcg_env, s1, s2, s3, z, op); } static void gen_op_fnmsubd(TCGv_i64 d, TCGv_i64 s1, TCGv_i64 s2, TCGv_i64 s3) { - int op = float_muladd_negate_c | float_muladd_negate_result; - gen_helper_fmaddd(d, tcg_env, s1, s2, s3, tcg_constant_i32(op)); + TCGv_i32 z = tcg_constant_i32(0); + TCGv_i32 op = tcg_constant_i32(float_muladd_negate_c | + float_muladd_negate_result); + gen_helper_fmaddd(d, tcg_env, s1, s2, s3, z, op); } static void gen_op_fnmadds(TCGv_i32 d, TCGv_i32 s1, TCGv_i32 s2, TCGv_i32 s3) { - int op = float_muladd_negate_result; - gen_helper_fmadds(d, tcg_env, s1, s2, s3, tcg_constant_i32(op)); + TCGv_i32 z = tcg_constant_i32(0); + TCGv_i32 op = tcg_constant_i32(float_muladd_negate_result); + gen_helper_fmadds(d, tcg_env, s1, s2, s3, z, op); } static void gen_op_fnmaddd(TCGv_i64 d, TCGv_i64 s1, TCGv_i64 s2, TCGv_i64 s3) { - int op = float_muladd_negate_result; - gen_helper_fmaddd(d, tcg_env, s1, s2, s3, tcg_constant_i32(op)); + TCGv_i32 z = tcg_constant_i32(0); + TCGv_i32 op = tcg_constant_i32(float_muladd_negate_result); + gen_helper_fmaddd(d, tcg_env, s1, s2, s3, z, op); } /* Use muladd to compute (1 * src1) + src2 / 2 with one rounding. */ static void gen_op_fhadds(TCGv_i32 d, TCGv_i32 s1, TCGv_i32 s2) { - TCGv_i32 one = tcg_constant_i32(float32_one); - int op = float_muladd_halve_result; - gen_helper_fmadds(d, tcg_env, one, s1, s2, tcg_constant_i32(op)); + TCGv_i32 fone = tcg_constant_i32(float32_one); + TCGv_i32 mone = tcg_constant_i32(-1); + TCGv_i32 op = tcg_constant_i32(0); + gen_helper_fmadds(d, tcg_env, fone, s1, s2, mone, op); } static void gen_op_fhaddd(TCGv_i64 d, TCGv_i64 s1, TCGv_i64 s2) { - TCGv_i64 one = tcg_constant_i64(float64_one); - int op = float_muladd_halve_result; - gen_helper_fmaddd(d, tcg_env, one, s1, s2, tcg_constant_i32(op)); + TCGv_i64 fone = tcg_constant_i64(float64_one); + TCGv_i32 mone = tcg_constant_i32(-1); + TCGv_i32 op = tcg_constant_i32(0); + gen_helper_fmaddd(d, tcg_env, fone, s1, s2, mone, op); } /* Use muladd to compute (1 * src1) - src2 / 2 with one rounding. */ static void gen_op_fhsubs(TCGv_i32 d, TCGv_i32 s1, TCGv_i32 s2) { - TCGv_i32 one = tcg_constant_i32(float32_one); - int op = float_muladd_negate_c | float_muladd_halve_result; - gen_helper_fmadds(d, tcg_env, one, s1, s2, tcg_constant_i32(op)); + TCGv_i32 fone = tcg_constant_i32(float32_one); + TCGv_i32 mone = tcg_constant_i32(-1); + TCGv_i32 op = tcg_constant_i32(float_muladd_negate_c); + gen_helper_fmadds(d, tcg_env, fone, s1, s2, mone, op); } static void gen_op_fhsubd(TCGv_i64 d, TCGv_i64 s1, TCGv_i64 s2) { - TCGv_i64 one = tcg_constant_i64(float64_one); - int op = float_muladd_negate_c | float_muladd_halve_result; - gen_helper_fmaddd(d, tcg_env, one, s1, s2, tcg_constant_i32(op)); + TCGv_i64 fone = tcg_constant_i64(float64_one); + TCGv_i32 mone = tcg_constant_i32(-1); + TCGv_i32 op = tcg_constant_i32(float_muladd_negate_c); + gen_helper_fmaddd(d, tcg_env, fone, s1, s2, mone, op); } /* Use muladd to compute -((1 * src1) + src2 / 2) with one rounding. */ static void gen_op_fnhadds(TCGv_i32 d, TCGv_i32 s1, TCGv_i32 s2) { - TCGv_i32 one = tcg_constant_i32(float32_one); - int op = float_muladd_negate_result | float_muladd_halve_result; - gen_helper_fmadds(d, tcg_env, one, s1, s2, tcg_constant_i32(op)); + TCGv_i32 fone = tcg_constant_i32(float32_one); + TCGv_i32 mone = tcg_constant_i32(-1); + TCGv_i32 op = tcg_constant_i32(float_muladd_negate_result); + gen_helper_fmadds(d, tcg_env, fone, s1, s2, mone, op); } static void gen_op_fnhaddd(TCGv_i64 d, TCGv_i64 s1, TCGv_i64 s2) { - TCGv_i64 one = tcg_constant_i64(float64_one); - int op = float_muladd_negate_result | float_muladd_halve_result; - gen_helper_fmaddd(d, tcg_env, one, s1, s2, tcg_constant_i32(op)); + TCGv_i64 fone = tcg_constant_i64(float64_one); + TCGv_i32 mone = tcg_constant_i32(-1); + TCGv_i32 op = tcg_constant_i32(float_muladd_negate_result); + gen_helper_fmaddd(d, tcg_env, fone, s1, s2, mone, op); } static void gen_op_fpexception_im(DisasContext *dc, int ftt) -- 2.43.0 All uses have been convered to float*_muladd_scalbn. Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- include/fpu/softfloat.h | 3 --- fpu/softfloat.c | 6 ------ fpu/softfloat-parts.c.inc | 4 ---- 3 files changed, 13 deletions(-) diff --git a/include/fpu/softfloat.h b/include/fpu/softfloat.h index XXXXXXX..XXXXXXX 100644 --- a/include/fpu/softfloat.h +++ b/include/fpu/softfloat.h @@ -XXX,XX +XXX,XX @@ bfloat16 bfloat16_squash_input_denormal(bfloat16 a, float_status *status); | Using these differs from negating an input or output before calling | the muladd function in that this means that a NaN doesn't have its | sign bit inverted before it is propagated. -| We also support halving the result before rounding, as a special -| case to support the ARM fused-sqrt-step instruction FRSQRTS. *----------------------------------------------------------------------------*/ enum { float_muladd_negate_c = 1, float_muladd_negate_product = 2, float_muladd_negate_result = 4, - float_muladd_halve_result = 8, }; /*---------------------------------------------------------------------------- diff --git a/fpu/softfloat.c b/fpu/softfloat.c index XXXXXXX..XXXXXXX 100644 --- a/fpu/softfloat.c +++ b/fpu/softfloat.c @@ -XXX,XX +XXX,XX @@ float32_muladd(float32 xa, float32 xb, float32 xc, int flags, float_status *s) if (unlikely(!can_use_fpu(s))) { goto soft; } - if (unlikely(flags & float_muladd_halve_result)) { - goto soft; - } float32_input_flush3(&ua.s, &ub.s, &uc.s, s); if (unlikely(!f32_is_zon3(ua, ub, uc))) { @@ -XXX,XX +XXX,XX @@ float64_muladd(float64 xa, float64 xb, float64 xc, int flags, float_status *s) if (unlikely(!can_use_fpu(s))) { goto soft; } - if (unlikely(flags & float_muladd_halve_result)) { - goto soft; - } float64_input_flush3(&ua.s, &ub.s, &uc.s, s); if (unlikely(!f64_is_zon3(ua, ub, uc))) { diff --git a/fpu/softfloat-parts.c.inc b/fpu/softfloat-parts.c.inc index XXXXXXX..XXXXXXX 100644 --- a/fpu/softfloat-parts.c.inc +++ b/fpu/softfloat-parts.c.inc @@ -XXX,XX +XXX,XX @@ static FloatPartsN *partsN(muladd_scalbn)(FloatPartsN *a, FloatPartsN *b, a->exp = p_widen.exp; return_normal: - /* TODO: Replace all use of float_muladd_halve_result with scale. */ - if (flags & float_muladd_halve_result) { - a->exp -= 1; - } a->exp += scale; finish_sign: if (flags & float_muladd_negate_result) { -- 2.43.0 This rounding mode is used by Hexagon. Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- include/fpu/softfloat-types.h | 2 ++ fpu/softfloat-parts.c.inc | 3 +++ 2 files changed, 5 insertions(+) diff --git a/include/fpu/softfloat-types.h b/include/fpu/softfloat-types.h index XXXXXXX..XXXXXXX 100644 --- a/include/fpu/softfloat-types.h +++ b/include/fpu/softfloat-types.h @@ -XXX,XX +XXX,XX @@ typedef enum __attribute__((__packed__)) { float_round_to_odd = 5, /* Not an IEEE rounding mode: round to closest odd, overflow to inf */ float_round_to_odd_inf = 6, + /* Not an IEEE rounding mode: round to nearest even, overflow to max */ + float_round_nearest_even_max = 7, } FloatRoundMode; /* diff --git a/fpu/softfloat-parts.c.inc b/fpu/softfloat-parts.c.inc index XXXXXXX..XXXXXXX 100644 --- a/fpu/softfloat-parts.c.inc +++ b/fpu/softfloat-parts.c.inc @@ -XXX,XX +XXX,XX @@ static void partsN(uncanon_normal)(FloatPartsN *p, float_status *s, int exp, flags = 0; switch (s->float_rounding_mode) { + case float_round_nearest_even_max: + overflow_norm = true; + /* fall through */ case float_round_nearest_even: if (N > 64 && frac_lsb == 0) { inc = ((p->frac_hi & 1) || (p->frac_lo & round_mask) != frac_lsbm1 -- 2.43.0 Certain Hexagon instructions suppress changes to the result when the product of fma() is a true zero. Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- include/fpu/softfloat.h | 5 +++++ fpu/softfloat.c | 3 +++ fpu/softfloat-parts.c.inc | 4 +++- 3 files changed, 11 insertions(+), 1 deletion(-) diff --git a/include/fpu/softfloat.h b/include/fpu/softfloat.h index XXXXXXX..XXXXXXX 100644 --- a/include/fpu/softfloat.h +++ b/include/fpu/softfloat.h @@ -XXX,XX +XXX,XX @@ bfloat16 bfloat16_squash_input_denormal(bfloat16 a, float_status *status); | Using these differs from negating an input or output before calling | the muladd function in that this means that a NaN doesn't have its | sign bit inverted before it is propagated. +| +| With float_muladd_suppress_add_product_zero, if A or B is zero +| such that the product is a true zero, then return C without addition. +| This preserves the sign of C when C is +/- 0. Used for Hexagon. *----------------------------------------------------------------------------*/ enum { float_muladd_negate_c = 1, float_muladd_negate_product = 2, float_muladd_negate_result = 4, + float_muladd_suppress_add_product_zero = 8, }; /*---------------------------------------------------------------------------- diff --git a/fpu/softfloat.c b/fpu/softfloat.c index XXXXXXX..XXXXXXX 100644 --- a/fpu/softfloat.c +++ b/fpu/softfloat.c @@ -XXX,XX +XXX,XX @@ float32_muladd(float32 xa, float32 xb, float32 xc, int flags, float_status *s) if (unlikely(!can_use_fpu(s))) { goto soft; } + if (unlikely(flags & float_muladd_suppress_add_product_zero)) { + goto soft; + } float32_input_flush3(&ua.s, &ub.s, &uc.s, s); if (unlikely(!f32_is_zon3(ua, ub, uc))) { diff --git a/fpu/softfloat-parts.c.inc b/fpu/softfloat-parts.c.inc index XXXXXXX..XXXXXXX 100644 --- a/fpu/softfloat-parts.c.inc +++ b/fpu/softfloat-parts.c.inc @@ -XXX,XX +XXX,XX @@ static FloatPartsN *partsN(muladd_scalbn)(FloatPartsN *a, FloatPartsN *b, goto return_normal; } if (c->cls == float_class_zero) { - if (a->sign != c->sign) { + if (flags & float_muladd_suppress_add_product_zero) { + a->sign = c->sign; + } else if (a->sign != c->sign) { goto return_sub_zero; } goto return_zero; -- 2.43.0 There are no special cases for this instruction. Remove internal_mpyf as unused. Reviewed-by: Brian Cain <brian.cain@oss.qualcomm.com> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- target/hexagon/fma_emu.h | 1 - target/hexagon/fma_emu.c | 8 -------- target/hexagon/op_helper.c | 2 +- 3 files changed, 1 insertion(+), 10 deletions(-) diff --git a/target/hexagon/fma_emu.h b/target/hexagon/fma_emu.h index XXXXXXX..XXXXXXX 100644 --- a/target/hexagon/fma_emu.h +++ b/target/hexagon/fma_emu.h @@ -XXX,XX +XXX,XX @@ int32_t float32_getexp(float32 f32); float32 infinite_float32(uint8_t sign); float32 internal_fmafx(float32 a, float32 b, float32 c, int scale, float_status *fp_status); -float32 internal_mpyf(float32 a, float32 b, float_status *fp_status); float64 internal_mpyhh(float64 a, float64 b, unsigned long long int accumulated, float_status *fp_status); diff --git a/target/hexagon/fma_emu.c b/target/hexagon/fma_emu.c index XXXXXXX..XXXXXXX 100644 --- a/target/hexagon/fma_emu.c +++ b/target/hexagon/fma_emu.c @@ -XXX,XX +XXX,XX @@ float32 internal_fmafx(float32 a, float32 b, float32 c, int scale, return accum_round_float32(result, fp_status); } -float32 internal_mpyf(float32 a, float32 b, float_status *fp_status) -{ - if (float32_is_zero(a) || float32_is_zero(b)) { - return float32_mul(a, b, fp_status); - } - return internal_fmafx(a, b, float32_zero, 0, fp_status); -} - float64 internal_mpyhh(float64 a, float64 b, unsigned long long int accumulated, float_status *fp_status) diff --git a/target/hexagon/op_helper.c b/target/hexagon/op_helper.c index XXXXXXX..XXXXXXX 100644 --- a/target/hexagon/op_helper.c +++ b/target/hexagon/op_helper.c @@ -XXX,XX +XXX,XX @@ float32 HELPER(sfmpy)(CPUHexagonState *env, float32 RsV, float32 RtV) { float32 RdV; arch_fpop_start(env); - RdV = internal_mpyf(RsV, RtV, &env->fp_status); + RdV = float32_mul(RsV, RtV, &env->fp_status); arch_fpop_end(env); return RdV; } -- 2.43.0 There are no special cases for this instruction. Reviewed-by: Brian Cain <brian.cain@oss.qualcomm.com> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- target/hexagon/op_helper.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/target/hexagon/op_helper.c b/target/hexagon/op_helper.c index XXXXXXX..XXXXXXX 100644 --- a/target/hexagon/op_helper.c +++ b/target/hexagon/op_helper.c @@ -XXX,XX +XXX,XX @@ float32 HELPER(sffma)(CPUHexagonState *env, float32 RxV, float32 RsV, float32 RtV) { arch_fpop_start(env); - RxV = internal_fmafx(RsV, RtV, RxV, 0, &env->fp_status); + RxV = float32_muladd(RsV, RtV, RxV, 0, &env->fp_status); arch_fpop_end(env); return RxV; } -- 2.43.0 There are no special cases for this instruction. Since hexagon always uses default-nan mode, explicitly negating the first input is unnecessary. Use float_muladd_negate_product instead. Reviewed-by: Brian Cain <brian.cain@oss.qualcomm.com> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- target/hexagon/op_helper.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/target/hexagon/op_helper.c b/target/hexagon/op_helper.c index XXXXXXX..XXXXXXX 100644 --- a/target/hexagon/op_helper.c +++ b/target/hexagon/op_helper.c @@ -XXX,XX +XXX,XX @@ float32 HELPER(sffma_sc)(CPUHexagonState *env, float32 RxV, float32 HELPER(sffms)(CPUHexagonState *env, float32 RxV, float32 RsV, float32 RtV) { - float32 neg_RsV; arch_fpop_start(env); - neg_RsV = float32_set_sign(RsV, float32_is_neg(RsV) ? 0 : 1); - RxV = internal_fmafx(neg_RsV, RtV, RxV, 0, &env->fp_status); + RxV = float32_muladd(RsV, RtV, RxV, float_muladd_negate_product, + &env->fp_status); arch_fpop_end(env); return RxV; } -- 2.43.0 This instruction has a special case that 0 * x + c returns c without the normal sign folding that comes with 0 + -0. Use the new float_muladd_suppress_add_product_zero to describe this. Reviewed-by: Brian Cain <brian.cain@oss.qualcomm.com> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- target/hexagon/op_helper.c | 11 +++-------- 1 file changed, 3 insertions(+), 8 deletions(-) diff --git a/target/hexagon/op_helper.c b/target/hexagon/op_helper.c index XXXXXXX..XXXXXXX 100644 --- a/target/hexagon/op_helper.c +++ b/target/hexagon/op_helper.c @@ -XXX,XX +XXX,XX @@ static float32 check_nan(float32 dst, float32 x, float_status *fp_status) float32 HELPER(sffma_sc)(CPUHexagonState *env, float32 RxV, float32 RsV, float32 RtV, float32 PuV) { - size4s_t tmp; arch_fpop_start(env); - RxV = check_nan(RxV, RxV, &env->fp_status); - RxV = check_nan(RxV, RsV, &env->fp_status); - RxV = check_nan(RxV, RtV, &env->fp_status); - tmp = internal_fmafx(RsV, RtV, RxV, fSXTN(8, 64, PuV), &env->fp_status); - if (!(float32_is_zero(RxV) && is_zero_prod(RsV, RtV))) { - RxV = tmp; - } + RxV = float32_muladd_scalbn(RsV, RtV, RxV, fSXTN(8, 64, PuV), + float_muladd_suppress_add_product_zero, + &env->fp_status); arch_fpop_end(env); return RxV; } -- 2.43.0 There are multiple special cases for this instruction. (1) The saturate to normal maximum instead of overflow to infinity is handled by the new float_round_nearest_even_max rounding mode. (2) The 0 * n + c special case is handled by the new float_muladd_suppress_add_product_zero flag. (3) The Inf - Inf -> 0 special case can be detected after the fact by examining float_flag_invalid_isi. Reviewed-by: Brian Cain <brian.cain@oss.qualcomm.com> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- target/hexagon/op_helper.c | 105 +++++++++---------------------------- 1 file changed, 26 insertions(+), 79 deletions(-) diff --git a/target/hexagon/op_helper.c b/target/hexagon/op_helper.c index XXXXXXX..XXXXXXX 100644 --- a/target/hexagon/op_helper.c +++ b/target/hexagon/op_helper.c @@ -XXX,XX +XXX,XX @@ float32 HELPER(sffma)(CPUHexagonState *env, float32 RxV, return RxV; } -static bool is_zero_prod(float32 a, float32 b) -{ - return ((float32_is_zero(a) && is_finite(b)) || - (float32_is_zero(b) && is_finite(a))); -} - -static float32 check_nan(float32 dst, float32 x, float_status *fp_status) -{ - float32 ret = dst; - if (float32_is_any_nan(x)) { - if (extract32(x, 22, 1) == 0) { - float_raise(float_flag_invalid, fp_status); - } - ret = make_float32(0xffffffff); /* nan */ - } - return ret; -} - float32 HELPER(sffma_sc)(CPUHexagonState *env, float32 RxV, float32 RsV, float32 RtV, float32 PuV) { @@ -XXX,XX +XXX,XX @@ float32 HELPER(sffms)(CPUHexagonState *env, float32 RxV, return RxV; } -static bool is_inf_prod(int32_t a, int32_t b) +static float32 do_sffma_lib(CPUHexagonState *env, float32 RxV, + float32 RsV, float32 RtV, int negate) { - return (float32_is_infinity(a) && float32_is_infinity(b)) || - (float32_is_infinity(a) && is_finite(b) && !float32_is_zero(b)) || - (float32_is_infinity(b) && is_finite(a) && !float32_is_zero(a)); + int flags; + + arch_fpop_start(env); + + set_float_rounding_mode(float_round_nearest_even_max, &env->fp_status); + RxV = float32_muladd(RsV, RtV, RxV, + negate | float_muladd_suppress_add_product_zero, + &env->fp_status); + + flags = get_float_exception_flags(&env->fp_status); + if (flags) { + /* Flags are suppressed by this instruction. */ + set_float_exception_flags(0, &env->fp_status); + + /* Return 0 for Inf - Inf. */ + if (flags & float_flag_invalid_isi) { + RxV = 0; + } + } + + arch_fpop_end(env); + return RxV; } float32 HELPER(sffma_lib)(CPUHexagonState *env, float32 RxV, float32 RsV, float32 RtV) { - bool infinp; - bool infminusinf; - float32 tmp; - - arch_fpop_start(env); - set_float_rounding_mode(float_round_nearest_even, &env->fp_status); - infminusinf = float32_is_infinity(RxV) && - is_inf_prod(RsV, RtV) && - (fGETBIT(31, RsV ^ RxV ^ RtV) != 0); - infinp = float32_is_infinity(RxV) || - float32_is_infinity(RtV) || - float32_is_infinity(RsV); - RxV = check_nan(RxV, RxV, &env->fp_status); - RxV = check_nan(RxV, RsV, &env->fp_status); - RxV = check_nan(RxV, RtV, &env->fp_status); - tmp = internal_fmafx(RsV, RtV, RxV, 0, &env->fp_status); - if (!(float32_is_zero(RxV) && is_zero_prod(RsV, RtV))) { - RxV = tmp; - } - set_float_exception_flags(0, &env->fp_status); - if (float32_is_infinity(RxV) && !infinp) { - RxV = RxV - 1; - } - if (infminusinf) { - RxV = 0; - } - arch_fpop_end(env); - return RxV; + return do_sffma_lib(env, RxV, RsV, RtV, 0); } float32 HELPER(sffms_lib)(CPUHexagonState *env, float32 RxV, float32 RsV, float32 RtV) { - bool infinp; - bool infminusinf; - float32 tmp; - - arch_fpop_start(env); - set_float_rounding_mode(float_round_nearest_even, &env->fp_status); - infminusinf = float32_is_infinity(RxV) && - is_inf_prod(RsV, RtV) && - (fGETBIT(31, RsV ^ RxV ^ RtV) == 0); - infinp = float32_is_infinity(RxV) || - float32_is_infinity(RtV) || - float32_is_infinity(RsV); - RxV = check_nan(RxV, RxV, &env->fp_status); - RxV = check_nan(RxV, RsV, &env->fp_status); - RxV = check_nan(RxV, RtV, &env->fp_status); - float32 minus_RsV = float32_sub(float32_zero, RsV, &env->fp_status); - tmp = internal_fmafx(minus_RsV, RtV, RxV, 0, &env->fp_status); - if (!(float32_is_zero(RxV) && is_zero_prod(RsV, RtV))) { - RxV = tmp; - } - set_float_exception_flags(0, &env->fp_status); - if (float32_is_infinity(RxV) && !infinp) { - RxV = RxV - 1; - } - if (infminusinf) { - RxV = 0; - } - arch_fpop_end(env); - return RxV; + return do_sffma_lib(env, RxV, RsV, RtV, float_muladd_negate_product); } float64 HELPER(dfmpyfix)(CPUHexagonState *env, float64 RssV, float64 RttV) -- 2.43.0 The function is now unused. Reviewed-by: Brian Cain <brian.cain@oss.qualcomm.com> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- target/hexagon/fma_emu.h | 2 - target/hexagon/fma_emu.c | 171 --------------------------------------- 2 files changed, 173 deletions(-) diff --git a/target/hexagon/fma_emu.h b/target/hexagon/fma_emu.h index XXXXXXX..XXXXXXX 100644 --- a/target/hexagon/fma_emu.h +++ b/target/hexagon/fma_emu.h @@ -XXX,XX +XXX,XX @@ static inline uint32_t float32_getexp_raw(float32 f32) } int32_t float32_getexp(float32 f32); float32 infinite_float32(uint8_t sign); -float32 internal_fmafx(float32 a, float32 b, float32 c, - int scale, float_status *fp_status); float64 internal_mpyhh(float64 a, float64 b, unsigned long long int accumulated, float_status *fp_status); diff --git a/target/hexagon/fma_emu.c b/target/hexagon/fma_emu.c index XXXXXXX..XXXXXXX 100644 --- a/target/hexagon/fma_emu.c +++ b/target/hexagon/fma_emu.c @@ -XXX,XX +XXX,XX @@ int32_t float64_getexp(float64 f64) return -1; } -static uint64_t float32_getmant(float32 f32) -{ - Float a = { .i = f32 }; - if (float32_is_normal(f32)) { - return a.mant | 1ULL << 23; - } - if (float32_is_zero(f32)) { - return 0; - } - if (float32_is_denormal(f32)) { - return a.mant; - } - return ~0ULL; -} - int32_t float32_getexp(float32 f32) { Float a = { .i = f32 }; @@ -XXX,XX +XXX,XX @@ float32 infinite_float32(uint8_t sign) } /* Return a maximum finite value with the requested sign */ -static float32 maxfinite_float32(uint8_t sign) -{ - if (sign) { - return make_float32(SF_MINUS_MAXF); - } else { - return make_float32(SF_MAXF); - } -} - -/* Return a zero value with requested sign */ -static float32 zero_float32(uint8_t sign) -{ - if (sign) { - return make_float32(0x80000000); - } else { - return float32_zero; - } -} - #define GEN_XF_ROUND(SUFFIX, MANTBITS, INF_EXP, INTERNAL_TYPE) \ static SUFFIX accum_round_##SUFFIX(Accum a, float_status * fp_status) \ { \ @@ -XXX,XX +XXX,XX @@ static SUFFIX accum_round_##SUFFIX(Accum a, float_status * fp_status) \ } GEN_XF_ROUND(float64, DF_MANTBITS, DF_INF_EXP, Double) -GEN_XF_ROUND(float32, SF_MANTBITS, SF_INF_EXP, Float) - -static bool is_inf_prod(float64 a, float64 b) -{ - return ((float64_is_infinity(a) && float64_is_infinity(b)) || - (float64_is_infinity(a) && is_finite(b) && (!float64_is_zero(b))) || - (float64_is_infinity(b) && is_finite(a) && (!float64_is_zero(a)))); -} - -static float64 special_fma(float64 a, float64 b, float64 c, - float_status *fp_status) -{ - float64 ret = make_float64(0); - - /* - * If A multiplied by B is an exact infinity and C is also an infinity - * but with the opposite sign, FMA returns NaN and raises invalid. - */ - uint8_t a_sign = float64_is_neg(a); - uint8_t b_sign = float64_is_neg(b); - uint8_t c_sign = float64_is_neg(c); - if (is_inf_prod(a, b) && float64_is_infinity(c)) { - if ((a_sign ^ b_sign) != c_sign) { - ret = make_float64(DF_NAN); - float_raise(float_flag_invalid, fp_status); - return ret; - } - } - if ((float64_is_infinity(a) && float64_is_zero(b)) || - (float64_is_zero(a) && float64_is_infinity(b))) { - ret = make_float64(DF_NAN); - float_raise(float_flag_invalid, fp_status); - return ret; - } - /* - * If none of the above checks are true and C is a NaN, - * a NaN shall be returned - * If A or B are NaN, a NAN shall be returned. - */ - if (float64_is_any_nan(a) || - float64_is_any_nan(b) || - float64_is_any_nan(c)) { - if (float64_is_any_nan(a) && (fGETBIT(51, a) == 0)) { - float_raise(float_flag_invalid, fp_status); - } - if (float64_is_any_nan(b) && (fGETBIT(51, b) == 0)) { - float_raise(float_flag_invalid, fp_status); - } - if (float64_is_any_nan(c) && (fGETBIT(51, c) == 0)) { - float_raise(float_flag_invalid, fp_status); - } - ret = make_float64(DF_NAN); - return ret; - } - /* - * We have checked for adding opposite-signed infinities. - * Other infinities return infinity with the correct sign - */ - if (float64_is_infinity(c)) { - ret = infinite_float64(c_sign); - return ret; - } - if (float64_is_infinity(a) || float64_is_infinity(b)) { - ret = infinite_float64(a_sign ^ b_sign); - return ret; - } - g_assert_not_reached(); -} - -static float32 special_fmaf(float32 a, float32 b, float32 c, - float_status *fp_status) -{ - float64 aa, bb, cc; - aa = float32_to_float64(a, fp_status); - bb = float32_to_float64(b, fp_status); - cc = float32_to_float64(c, fp_status); - return float64_to_float32(special_fma(aa, bb, cc, fp_status), fp_status); -} - -float32 internal_fmafx(float32 a, float32 b, float32 c, int scale, - float_status *fp_status) -{ - Accum prod; - Accum acc; - Accum result; - accum_init(&prod); - accum_init(&acc); - accum_init(&result); - - uint8_t a_sign = float32_is_neg(a); - uint8_t b_sign = float32_is_neg(b); - uint8_t c_sign = float32_is_neg(c); - if (float32_is_infinity(a) || - float32_is_infinity(b) || - float32_is_infinity(c)) { - return special_fmaf(a, b, c, fp_status); - } - if (float32_is_any_nan(a) || - float32_is_any_nan(b) || - float32_is_any_nan(c)) { - return special_fmaf(a, b, c, fp_status); - } - if ((scale == 0) && (float32_is_zero(a) || float32_is_zero(b))) { - float32 tmp = float32_mul(a, b, fp_status); - tmp = float32_add(tmp, c, fp_status); - return tmp; - } - - /* (a * 2**b) * (c * 2**d) == a*c * 2**(b+d) */ - prod.mant = int128_mul_6464(float32_getmant(a), float32_getmant(b)); - - /* - * Note: extracting the mantissa into an int is multiplying by - * 2**23, so adjust here - */ - prod.exp = float32_getexp(a) + float32_getexp(b) - SF_BIAS - 23; - prod.sign = a_sign ^ b_sign; - if (float32_is_zero(a) || float32_is_zero(b)) { - prod.exp = -2 * WAY_BIG_EXP; - } - if ((scale > 0) && float32_is_denormal(c)) { - acc.mant = int128_mul_6464(0, 0); - acc.exp = -WAY_BIG_EXP; - acc.sign = c_sign; - acc.sticky = 1; - result = accum_add(prod, acc); - } else if (!float32_is_zero(c)) { - acc.mant = int128_mul_6464(float32_getmant(c), 1); - acc.exp = float32_getexp(c); - acc.sign = c_sign; - result = accum_add(prod, acc); - } else { - result = prod; - } - result.exp += scale; - return accum_round_float32(result, fp_status); -} float64 internal_mpyhh(float64 a, float64 b, unsigned long long int accumulated, -- 2.43.0 This massive macro is now only used once. Expand it for use only by float64. Reviewed-by: Brian Cain <brian.cain@oss.qualcomm.com> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- target/hexagon/fma_emu.c | 255 +++++++++++++++++++-------------------- 1 file changed, 127 insertions(+), 128 deletions(-) diff --git a/target/hexagon/fma_emu.c b/target/hexagon/fma_emu.c index XXXXXXX..XXXXXXX 100644 --- a/target/hexagon/fma_emu.c +++ b/target/hexagon/fma_emu.c @@ -XXX,XX +XXX,XX @@ float32 infinite_float32(uint8_t sign) } /* Return a maximum finite value with the requested sign */ -#define GEN_XF_ROUND(SUFFIX, MANTBITS, INF_EXP, INTERNAL_TYPE) \ -static SUFFIX accum_round_##SUFFIX(Accum a, float_status * fp_status) \ -{ \ - if ((int128_gethi(a.mant) == 0) && (int128_getlo(a.mant) == 0) \ - && ((a.guard | a.round | a.sticky) == 0)) { \ - /* result zero */ \ - switch (fp_status->float_rounding_mode) { \ - case float_round_down: \ - return zero_##SUFFIX(1); \ - default: \ - return zero_##SUFFIX(0); \ - } \ - } \ - /* Normalize right */ \ - /* We want MANTBITS bits of mantissa plus the leading one. */ \ - /* That means that we want MANTBITS+1 bits, or 0x000000000000FF_FFFF */ \ - /* So we need to normalize right while the high word is non-zero and \ - * while the low word is nonzero when masked with 0xffe0_0000_0000_0000 */ \ - while ((int128_gethi(a.mant) != 0) || \ - ((int128_getlo(a.mant) >> (MANTBITS + 1)) != 0)) { \ - a = accum_norm_right(a, 1); \ - } \ - /* \ - * OK, now normalize left \ - * We want to normalize left until we have a leading one in bit 24 \ - * Theoretically, we only need to shift a maximum of one to the left if we \ - * shifted out lots of bits from B, or if we had no shift / 1 shift sticky \ - * should be 0 \ - */ \ - while ((int128_getlo(a.mant) & (1ULL << MANTBITS)) == 0) { \ - a = accum_norm_left(a); \ - } \ - /* \ - * OK, now we might need to denormalize because of potential underflow. \ - * We need to do this before rounding, and rounding might make us normal \ - * again \ - */ \ - while (a.exp <= 0) { \ - a = accum_norm_right(a, 1 - a.exp); \ - /* \ - * Do we have underflow? \ - * That's when we get an inexact answer because we ran out of bits \ - * in a denormal. \ - */ \ - if (a.guard || a.round || a.sticky) { \ - float_raise(float_flag_underflow, fp_status); \ - } \ - } \ - /* OK, we're relatively canonical... now we need to round */ \ - if (a.guard || a.round || a.sticky) { \ - float_raise(float_flag_inexact, fp_status); \ - switch (fp_status->float_rounding_mode) { \ - case float_round_to_zero: \ - /* Chop and we're done */ \ - break; \ - case float_round_up: \ - if (a.sign == 0) { \ - a.mant = int128_add(a.mant, int128_one()); \ - } \ - break; \ - case float_round_down: \ - if (a.sign != 0) { \ - a.mant = int128_add(a.mant, int128_one()); \ - } \ - break; \ - default: \ - if (a.round || a.sticky) { \ - /* round up if guard is 1, down if guard is zero */ \ - a.mant = int128_add(a.mant, int128_make64(a.guard)); \ - } else if (a.guard) { \ - /* exactly .5, round up if odd */ \ - a.mant = int128_add(a.mant, int128_and(a.mant, int128_one())); \ - } \ - break; \ - } \ - } \ - /* \ - * OK, now we might have carried all the way up. \ - * So we might need to shr once \ - * at least we know that the lsb should be zero if we rounded and \ - * got a carry out... \ - */ \ - if ((int128_getlo(a.mant) >> (MANTBITS + 1)) != 0) { \ - a = accum_norm_right(a, 1); \ - } \ - /* Overflow? */ \ - if (a.exp >= INF_EXP) { \ - /* Yep, inf result */ \ - float_raise(float_flag_overflow, fp_status); \ - float_raise(float_flag_inexact, fp_status); \ - switch (fp_status->float_rounding_mode) { \ - case float_round_to_zero: \ - return maxfinite_##SUFFIX(a.sign); \ - case float_round_up: \ - if (a.sign == 0) { \ - return infinite_##SUFFIX(a.sign); \ - } else { \ - return maxfinite_##SUFFIX(a.sign); \ - } \ - case float_round_down: \ - if (a.sign != 0) { \ - return infinite_##SUFFIX(a.sign); \ - } else { \ - return maxfinite_##SUFFIX(a.sign); \ - } \ - default: \ - return infinite_##SUFFIX(a.sign); \ - } \ - } \ - /* Underflow? */ \ - if (int128_getlo(a.mant) & (1ULL << MANTBITS)) { \ - /* Leading one means: No, we're normal. So, we should be done... */ \ - INTERNAL_TYPE ret; \ - ret.i = 0; \ - ret.sign = a.sign; \ - ret.exp = a.exp; \ - ret.mant = int128_getlo(a.mant); \ - return ret.i; \ - } \ - assert(a.exp == 1); \ - INTERNAL_TYPE ret; \ - ret.i = 0; \ - ret.sign = a.sign; \ - ret.exp = 0; \ - ret.mant = int128_getlo(a.mant); \ - return ret.i; \ +static float64 accum_round_float64(Accum a, float_status *fp_status) +{ + if ((int128_gethi(a.mant) == 0) && (int128_getlo(a.mant) == 0) + && ((a.guard | a.round | a.sticky) == 0)) { + /* result zero */ + switch (fp_status->float_rounding_mode) { + case float_round_down: + return zero_float64(1); + default: + return zero_float64(0); + } + } + /* + * Normalize right + * We want DF_MANTBITS bits of mantissa plus the leading one. + * That means that we want DF_MANTBITS+1 bits, or 0x000000000000FF_FFFF + * So we need to normalize right while the high word is non-zero and + * while the low word is nonzero when masked with 0xffe0_0000_0000_0000 + */ + while ((int128_gethi(a.mant) != 0) || + ((int128_getlo(a.mant) >> (DF_MANTBITS + 1)) != 0)) { + a = accum_norm_right(a, 1); + } + /* + * OK, now normalize left + * We want to normalize left until we have a leading one in bit 24 + * Theoretically, we only need to shift a maximum of one to the left if we + * shifted out lots of bits from B, or if we had no shift / 1 shift sticky + * should be 0 + */ + while ((int128_getlo(a.mant) & (1ULL << DF_MANTBITS)) == 0) { + a = accum_norm_left(a); + } + /* + * OK, now we might need to denormalize because of potential underflow. + * We need to do this before rounding, and rounding might make us normal + * again + */ + while (a.exp <= 0) { + a = accum_norm_right(a, 1 - a.exp); + /* + * Do we have underflow? + * That's when we get an inexact answer because we ran out of bits + * in a denormal. + */ + if (a.guard || a.round || a.sticky) { + float_raise(float_flag_underflow, fp_status); + } + } + /* OK, we're relatively canonical... now we need to round */ + if (a.guard || a.round || a.sticky) { + float_raise(float_flag_inexact, fp_status); + switch (fp_status->float_rounding_mode) { + case float_round_to_zero: + /* Chop and we're done */ + break; + case float_round_up: + if (a.sign == 0) { + a.mant = int128_add(a.mant, int128_one()); + } + break; + case float_round_down: + if (a.sign != 0) { + a.mant = int128_add(a.mant, int128_one()); + } + break; + default: + if (a.round || a.sticky) { + /* round up if guard is 1, down if guard is zero */ + a.mant = int128_add(a.mant, int128_make64(a.guard)); + } else if (a.guard) { + /* exactly .5, round up if odd */ + a.mant = int128_add(a.mant, int128_and(a.mant, int128_one())); + } + break; + } + } + /* + * OK, now we might have carried all the way up. + * So we might need to shr once + * at least we know that the lsb should be zero if we rounded and + * got a carry out... + */ + if ((int128_getlo(a.mant) >> (DF_MANTBITS + 1)) != 0) { + a = accum_norm_right(a, 1); + } + /* Overflow? */ + if (a.exp >= DF_INF_EXP) { + /* Yep, inf result */ + float_raise(float_flag_overflow, fp_status); + float_raise(float_flag_inexact, fp_status); + switch (fp_status->float_rounding_mode) { + case float_round_to_zero: + return maxfinite_float64(a.sign); + case float_round_up: + if (a.sign == 0) { + return infinite_float64(a.sign); + } else { + return maxfinite_float64(a.sign); + } + case float_round_down: + if (a.sign != 0) { + return infinite_float64(a.sign); + } else { + return maxfinite_float64(a.sign); + } + default: + return infinite_float64(a.sign); + } + } + /* Underflow? */ + if (int128_getlo(a.mant) & (1ULL << DF_MANTBITS)) { + /* Leading one means: No, we're normal. So, we should be done... */ + Double ret; + ret.i = 0; + ret.sign = a.sign; + ret.exp = a.exp; + ret.mant = int128_getlo(a.mant); + return ret.i; + } + assert(a.exp == 1); + Double ret; + ret.i = 0; + ret.sign = a.sign; + ret.exp = 0; + ret.mant = int128_getlo(a.mant); + return ret.i; } -GEN_XF_ROUND(float64, DF_MANTBITS, DF_INF_EXP, Double) - float64 internal_mpyhh(float64 a, float64 b, unsigned long long int accumulated, float_status *fp_status) -- 2.43.0 This structure, with bitfields, is incorrect for big-endian. Use the existing float32_getexp_raw which uses extract32. Reviewed-by: Brian Cain <brian.cain@oss.qualcomm.com> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- target/hexagon/fma_emu.c | 16 +++------------- 1 file changed, 3 insertions(+), 13 deletions(-) diff --git a/target/hexagon/fma_emu.c b/target/hexagon/fma_emu.c index XXXXXXX..XXXXXXX 100644 --- a/target/hexagon/fma_emu.c +++ b/target/hexagon/fma_emu.c @@ -XXX,XX +XXX,XX @@ typedef union { }; } Double; -typedef union { - float f; - uint32_t i; - struct { - uint32_t mant:23; - uint32_t exp:8; - uint32_t sign:1; - }; -} Float; - static uint64_t float64_getmant(float64 f64) { Double a = { .i = f64 }; @@ -XXX,XX +XXX,XX @@ int32_t float64_getexp(float64 f64) int32_t float32_getexp(float32 f32) { - Float a = { .i = f32 }; + int exp = float32_getexp_raw(f32); if (float32_is_normal(f32)) { - return a.exp; + return exp; } if (float32_is_denormal(f32)) { - return a.exp + 1; + return exp + 1; } return -1; } -- 2.43.0 This structure, with bitfields, is incorrect for big-endian. Use extract64 and deposit64 instead. Reviewed-by: Brian Cain <brian.cain@oss.qualcomm.com> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- target/hexagon/fma_emu.c | 46 ++++++++++++++-------------------------- 1 file changed, 16 insertions(+), 30 deletions(-) diff --git a/target/hexagon/fma_emu.c b/target/hexagon/fma_emu.c index XXXXXXX..XXXXXXX 100644 --- a/target/hexagon/fma_emu.c +++ b/target/hexagon/fma_emu.c @@ -XXX,XX +XXX,XX @@ #define WAY_BIG_EXP 4096 -typedef union { - double f; - uint64_t i; - struct { - uint64_t mant:52; - uint64_t exp:11; - uint64_t sign:1; - }; -} Double; - static uint64_t float64_getmant(float64 f64) { - Double a = { .i = f64 }; + uint64_t mant = extract64(f64, 0, 52); if (float64_is_normal(f64)) { - return a.mant | 1ULL << 52; + return mant | 1ULL << 52; } if (float64_is_zero(f64)) { return 0; } if (float64_is_denormal(f64)) { - return a.mant; + return mant; } return ~0ULL; } int32_t float64_getexp(float64 f64) { - Double a = { .i = f64 }; + int exp = extract64(f64, 52, 11); if (float64_is_normal(f64)) { - return a.exp; + return exp; } if (float64_is_denormal(f64)) { - return a.exp + 1; + return exp + 1; } return -1; } @@ -XXX,XX +XXX,XX @@ float32 infinite_float32(uint8_t sign) /* Return a maximum finite value with the requested sign */ static float64 accum_round_float64(Accum a, float_status *fp_status) { + uint64_t ret; + if ((int128_gethi(a.mant) == 0) && (int128_getlo(a.mant) == 0) && ((a.guard | a.round | a.sticky) == 0)) { /* result zero */ @@ -XXX,XX +XXX,XX @@ static float64 accum_round_float64(Accum a, float_status *fp_status) } } /* Underflow? */ - if (int128_getlo(a.mant) & (1ULL << DF_MANTBITS)) { + ret = int128_getlo(a.mant); + if (ret & (1ULL << DF_MANTBITS)) { /* Leading one means: No, we're normal. So, we should be done... */ - Double ret; - ret.i = 0; - ret.sign = a.sign; - ret.exp = a.exp; - ret.mant = int128_getlo(a.mant); - return ret.i; + ret = deposit64(ret, 52, 11, a.exp); + } else { + assert(a.exp == 1); + ret = deposit64(ret, 52, 11, 0); } - assert(a.exp == 1); - Double ret; - ret.i = 0; - ret.sign = a.sign; - ret.exp = 0; - ret.mant = int128_getlo(a.mant); - return ret.i; + ret = deposit64(ret, 63, 1, a.sign); + return ret; } float64 internal_mpyhh(float64 a, float64 b, -- 2.43.0 No need to open-code 64x64->128-bit multiplication. Reviewed-by: Brian Cain <brian.cain@oss.qualcomm.com> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- target/hexagon/fma_emu.c | 32 +++----------------------------- 1 file changed, 3 insertions(+), 29 deletions(-) diff --git a/target/hexagon/fma_emu.c b/target/hexagon/fma_emu.c index XXXXXXX..XXXXXXX 100644 --- a/target/hexagon/fma_emu.c +++ b/target/hexagon/fma_emu.c @@ -XXX,XX +XXX,XX @@ int32_t float32_getexp(float32 f32) return -1; } -static uint32_t int128_getw0(Int128 x) -{ - return int128_getlo(x); -} - -static uint32_t int128_getw1(Int128 x) -{ - return int128_getlo(x) >> 32; -} - static Int128 int128_mul_6464(uint64_t ai, uint64_t bi) { - Int128 a, b; - uint64_t pp0, pp1a, pp1b, pp1s, pp2; + uint64_t l, h; - a = int128_make64(ai); - b = int128_make64(bi); - pp0 = (uint64_t)int128_getw0(a) * (uint64_t)int128_getw0(b); - pp1a = (uint64_t)int128_getw1(a) * (uint64_t)int128_getw0(b); - pp1b = (uint64_t)int128_getw1(b) * (uint64_t)int128_getw0(a); - pp2 = (uint64_t)int128_getw1(a) * (uint64_t)int128_getw1(b); - - pp1s = pp1a + pp1b; - if ((pp1s < pp1a) || (pp1s < pp1b)) { - pp2 += (1ULL << 32); - } - uint64_t ret_low = pp0 + (pp1s << 32); - if ((ret_low < pp0) || (ret_low < (pp1s << 32))) { - pp2 += 1; - } - - return int128_make128(ret_low, pp2 + (pp1s >> 32)); + mulu64(&l, &h, ai, bi); + return int128_make128(l, h); } static Int128 int128_sub_borrow(Int128 a, Int128 b, int borrow) -- 2.43.0 Initialize x with accumulated via direct assignment, rather than multiplying by 1. Reviewed-by: Brian Cain <brian.cain@oss.qualcomm.com> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- target/hexagon/fma_emu.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/target/hexagon/fma_emu.c b/target/hexagon/fma_emu.c index XXXXXXX..XXXXXXX 100644 --- a/target/hexagon/fma_emu.c +++ b/target/hexagon/fma_emu.c @@ -XXX,XX +XXX,XX @@ float64 internal_mpyhh(float64 a, float64 b, float64_is_infinity(b)) { return float64_mul(a, b, fp_status); } - x.mant = int128_mul_6464(accumulated, 1); + x.mant = int128_make64(accumulated); x.sticky = sticky; prod = fGETUWORD(1, float64_getmant(a)) * fGETUWORD(1, float64_getmant(b)); x.mant = int128_add(x.mant, int128_mul_6464(prod, 0x100000000ULL)); -- 2.43.0 Convert all targets simultaneously, as the gen_intermediate_code function disappears from the target. While there are possible workarounds, they're larger than simply performing the conversion. Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> --- include/exec/translator.h | 14 -------------- include/hw/core/tcg-cpu-ops.h | 13 +++++++++++++ target/alpha/cpu.h | 2 ++ target/arm/internals.h | 2 ++ target/avr/cpu.h | 2 ++ target/hexagon/cpu.h | 2 ++ target/hppa/cpu.h | 2 ++ target/i386/tcg/helper-tcg.h | 2 ++ target/loongarch/internals.h | 2 ++ target/m68k/cpu.h | 2 ++ target/microblaze/cpu.h | 2 ++ target/mips/tcg/tcg-internal.h | 2 ++ target/openrisc/cpu.h | 2 ++ target/ppc/cpu.h | 2 ++ target/riscv/cpu.h | 3 +++ target/rx/cpu.h | 2 ++ target/s390x/s390x-internal.h | 2 ++ target/sh4/cpu.h | 2 ++ target/sparc/cpu.h | 2 ++ target/tricore/cpu.h | 2 ++ target/xtensa/cpu.h | 2 ++ accel/tcg/cpu-exec.c | 8 +++++--- accel/tcg/translate-all.c | 8 +++++--- target/alpha/cpu.c | 1 + target/alpha/translate.c | 4 ++-- target/arm/cpu.c | 1 + target/arm/tcg/cpu-v7m.c | 1 + target/arm/tcg/translate.c | 5 ++--- target/avr/cpu.c | 1 + target/avr/translate.c | 6 +++--- target/hexagon/cpu.c | 1 + target/hexagon/translate.c | 4 ++-- target/hppa/cpu.c | 1 + target/hppa/translate.c | 4 ++-- target/i386/tcg/tcg-cpu.c | 1 + target/i386/tcg/translate.c | 5 ++--- target/loongarch/cpu.c | 1 + target/loongarch/tcg/translate.c | 4 ++-- target/m68k/cpu.c | 1 + target/m68k/translate.c | 4 ++-- target/microblaze/cpu.c | 1 + target/microblaze/translate.c | 4 ++-- target/mips/cpu.c | 1 + target/mips/tcg/translate.c | 4 ++-- target/openrisc/cpu.c | 1 + target/openrisc/translate.c | 4 ++-- target/ppc/cpu_init.c | 1 + target/ppc/translate.c | 4 ++-- target/riscv/tcg/tcg-cpu.c | 1 + target/riscv/translate.c | 4 ++-- target/rx/cpu.c | 1 + target/rx/translate.c | 4 ++-- target/s390x/cpu.c | 1 + target/s390x/tcg/translate.c | 4 ++-- target/sh4/cpu.c | 1 + target/sh4/translate.c | 4 ++-- target/sparc/cpu.c | 1 + target/sparc/translate.c | 4 ++-- target/tricore/cpu.c | 1 + target/tricore/translate.c | 5 ++--- target/xtensa/cpu.c | 1 + target/xtensa/translate.c | 4 ++-- 62 files changed, 121 insertions(+), 62 deletions(-) diff --git a/include/exec/translator.h b/include/exec/translator.h index XXXXXXX..XXXXXXX 100644 --- a/include/exec/translator.h +++ b/include/exec/translator.h @@ -XXX,XX +XXX,XX @@ #include "qemu/bswap.h" #include "exec/vaddr.h" -/** - * gen_intermediate_code - * @cpu: cpu context - * @tb: translation block - * @max_insns: max number of instructions to translate - * @pc: guest virtual program counter address - * @host_pc: host physical program counter address - * - * This function must be provided by the target, which should create - * the target-specific DisasContext, and then invoke translator_loop. - */ -void gen_intermediate_code(CPUState *cpu, TranslationBlock *tb, int *max_insns, - vaddr pc, void *host_pc); - /** * DisasJumpType: * @DISAS_NEXT: Next instruction in program order. diff --git a/include/hw/core/tcg-cpu-ops.h b/include/hw/core/tcg-cpu-ops.h index XXXXXXX..XXXXXXX 100644 --- a/include/hw/core/tcg-cpu-ops.h +++ b/include/hw/core/tcg-cpu-ops.h @@ -XXX,XX +XXX,XX @@ struct TCGCPUOps { * Called when the first CPU is realized. */ void (*initialize)(void); + /** + * @translate_code: Translate guest instructions to TCGOps + * @cpu: cpu context + * @tb: translation block + * @max_insns: max number of instructions to translate + * @pc: guest virtual program counter address + * @host_pc: host physical program counter address + * + * This function must be provided by the target, which should create + * the target-specific DisasContext, and then invoke translator_loop. + */ + void (*translate_code)(CPUState *cpu, TranslationBlock *tb, + int *max_insns, vaddr pc, void *host_pc); /** * @synchronize_from_tb: Synchronize state from a TCG #TranslationBlock * diff --git a/target/alpha/cpu.h b/target/alpha/cpu.h index XXXXXXX..XXXXXXX 100644 --- a/target/alpha/cpu.h +++ b/target/alpha/cpu.h @@ -XXX,XX +XXX,XX @@ enum { }; void alpha_translate_init(void); +void alpha_translate_code(CPUState *cs, TranslationBlock *tb, + int *max_insns, vaddr pc, void *host_pc); #define CPU_RESOLVING_TYPE TYPE_ALPHA_CPU diff --git a/target/arm/internals.h b/target/arm/internals.h index XXXXXXX..XXXXXXX 100644 --- a/target/arm/internals.h +++ b/target/arm/internals.h @@ -XXX,XX +XXX,XX @@ void init_cpreg_list(ARMCPU *cpu); void arm_cpu_register_gdb_regs_for_features(ARMCPU *cpu); void arm_translate_init(void); +void arm_translate_code(CPUState *cs, TranslationBlock *tb, + int *max_insns, vaddr pc, void *host_pc); void arm_cpu_register_gdb_commands(ARMCPU *cpu); void aarch64_cpu_register_gdb_commands(ARMCPU *cpu, GString *, diff --git a/target/avr/cpu.h b/target/avr/cpu.h index XXXXXXX..XXXXXXX 100644 --- a/target/avr/cpu.h +++ b/target/avr/cpu.h @@ -XXX,XX +XXX,XX @@ static inline void set_avr_feature(CPUAVRState *env, int feature) } void avr_cpu_tcg_init(void); +void avr_cpu_translate_code(CPUState *cs, TranslationBlock *tb, + int *max_insns, vaddr pc, void *host_pc); int cpu_avr_exec(CPUState *cpu); diff --git a/target/hexagon/cpu.h b/target/hexagon/cpu.h index XXXXXXX..XXXXXXX 100644 --- a/target/hexagon/cpu.h +++ b/target/hexagon/cpu.h @@ -XXX,XX +XXX,XX @@ static inline void cpu_get_tb_cpu_state(CPUHexagonState *env, vaddr *pc, typedef HexagonCPU ArchCPU; void hexagon_translate_init(void); +void hexagon_translate_code(CPUState *cs, TranslationBlock *tb, + int *max_insns, vaddr pc, void *host_pc); #include "exec/cpu-all.h" diff --git a/target/hppa/cpu.h b/target/hppa/cpu.h index XXXXXXX..XXXXXXX 100644 --- a/target/hppa/cpu.h +++ b/target/hppa/cpu.h @@ -XXX,XX +XXX,XX @@ static inline int HPPA_BTLB_ENTRIES(CPUHPPAState *env) } void hppa_translate_init(void); +void hppa_translate_code(CPUState *cs, TranslationBlock *tb, + int *max_insns, vaddr pc, void *host_pc); #define CPU_RESOLVING_TYPE TYPE_HPPA_CPU diff --git a/target/i386/tcg/helper-tcg.h b/target/i386/tcg/helper-tcg.h index XXXXXXX..XXXXXXX 100644 --- a/target/i386/tcg/helper-tcg.h +++ b/target/i386/tcg/helper-tcg.h @@ -XXX,XX +XXX,XX @@ static inline target_long lshift(target_long x, int n) /* translate.c */ void tcg_x86_init(void); +void x86_translate_code(CPUState *cs, TranslationBlock *tb, + int *max_insns, vaddr pc, void *host_pc); /* excp_helper.c */ G_NORETURN void raise_exception(CPUX86State *env, int exception_index); diff --git a/target/loongarch/internals.h b/target/loongarch/internals.h index XXXXXXX..XXXXXXX 100644 --- a/target/loongarch/internals.h +++ b/target/loongarch/internals.h @@ -XXX,XX +XXX,XX @@ #define TARGET_VIRT_MASK MAKE_64BIT_MASK(0, TARGET_VIRT_ADDR_SPACE_BITS) void loongarch_translate_init(void); +void loongarch_translate_code(CPUState *cs, TranslationBlock *tb, + int *max_insns, vaddr pc, void *host_pc); void G_NORETURN do_raise_exception(CPULoongArchState *env, uint32_t exception, diff --git a/target/m68k/cpu.h b/target/m68k/cpu.h index XXXXXXX..XXXXXXX 100644 --- a/target/m68k/cpu.h +++ b/target/m68k/cpu.h @@ -XXX,XX +XXX,XX @@ int m68k_cpu_gdb_read_register(CPUState *cpu, GByteArray *buf, int reg); int m68k_cpu_gdb_write_register(CPUState *cpu, uint8_t *buf, int reg); void m68k_tcg_init(void); +void m68k_translate_code(CPUState *cs, TranslationBlock *tb, + int *max_insns, vaddr pc, void *host_pc); void m68k_cpu_init_gdb(M68kCPU *cpu); uint32_t cpu_m68k_get_ccr(CPUM68KState *env); void cpu_m68k_set_ccr(CPUM68KState *env, uint32_t); diff --git a/target/microblaze/cpu.h b/target/microblaze/cpu.h index XXXXXXX..XXXXXXX 100644 --- a/target/microblaze/cpu.h +++ b/target/microblaze/cpu.h @@ -XXX,XX +XXX,XX @@ static inline void mb_cpu_write_msr(CPUMBState *env, uint32_t val) } void mb_tcg_init(void); +void mb_translate_code(CPUState *cs, TranslationBlock *tb, + int *max_insns, vaddr pc, void *host_pc); #define CPU_RESOLVING_TYPE TYPE_MICROBLAZE_CPU diff --git a/target/mips/tcg/tcg-internal.h b/target/mips/tcg/tcg-internal.h index XXXXXXX..XXXXXXX 100644 --- a/target/mips/tcg/tcg-internal.h +++ b/target/mips/tcg/tcg-internal.h @@ -XXX,XX +XXX,XX @@ #include "cpu.h" void mips_tcg_init(void); +void mips_translate_code(CPUState *cs, TranslationBlock *tb, + int *max_insns, vaddr pc, void *host_pc); void mips_cpu_synchronize_from_tb(CPUState *cs, const TranslationBlock *tb); G_NORETURN void mips_cpu_do_unaligned_access(CPUState *cpu, vaddr addr, diff --git a/target/openrisc/cpu.h b/target/openrisc/cpu.h index XXXXXXX..XXXXXXX 100644 --- a/target/openrisc/cpu.h +++ b/target/openrisc/cpu.h @@ -XXX,XX +XXX,XX @@ void openrisc_cpu_dump_state(CPUState *cpu, FILE *f, int flags); int openrisc_cpu_gdb_read_register(CPUState *cpu, GByteArray *buf, int reg); int openrisc_cpu_gdb_write_register(CPUState *cpu, uint8_t *buf, int reg); void openrisc_translate_init(void); +void openrisc_translate_code(CPUState *cs, TranslationBlock *tb, + int *max_insns, vaddr pc, void *host_pc); int print_insn_or1k(bfd_vma addr, disassemble_info *info); #ifndef CONFIG_USER_ONLY diff --git a/target/ppc/cpu.h b/target/ppc/cpu.h index XXXXXXX..XXXXXXX 100644 --- a/target/ppc/cpu.h +++ b/target/ppc/cpu.h @@ -XXX,XX +XXX,XX @@ extern const VMStateDescription vmstate_ppc_cpu; /*****************************************************************************/ void ppc_translate_init(void); +void ppc_translate_code(CPUState *cs, TranslationBlock *tb, + int *max_insns, vaddr pc, void *host_pc); #if !defined(CONFIG_USER_ONLY) void ppc_store_sdr1(CPUPPCState *env, target_ulong value); diff --git a/target/riscv/cpu.h b/target/riscv/cpu.h index XXXXXXX..XXXXXXX 100644 --- a/target/riscv/cpu.h +++ b/target/riscv/cpu.h @@ -XXX,XX +XXX,XX @@ RISCVException smstateen_acc_ok(CPURISCVState *env, int index, uint64_t bit); void riscv_cpu_set_mode(CPURISCVState *env, target_ulong newpriv, bool virt_en); void riscv_translate_init(void); +void riscv_translate_code(CPUState *cs, TranslationBlock *tb, + int *max_insns, vaddr pc, void *host_pc); + G_NORETURN void riscv_raise_exception(CPURISCVState *env, uint32_t exception, uintptr_t pc); diff --git a/target/rx/cpu.h b/target/rx/cpu.h index XXXXXXX..XXXXXXX 100644 --- a/target/rx/cpu.h +++ b/target/rx/cpu.h @@ -XXX,XX +XXX,XX @@ int rx_cpu_gdb_read_register(CPUState *cpu, GByteArray *buf, int reg); int rx_cpu_gdb_write_register(CPUState *cpu, uint8_t *buf, int reg); void rx_translate_init(void); +void rx_translate_code(CPUState *cs, TranslationBlock *tb, + int *max_insns, vaddr pc, void *host_pc); void rx_cpu_unpack_psw(CPURXState *env, uint32_t psw, int rte); #include "exec/cpu-all.h" diff --git a/target/s390x/s390x-internal.h b/target/s390x/s390x-internal.h index XXXXXXX..XXXXXXX 100644 --- a/target/s390x/s390x-internal.h +++ b/target/s390x/s390x-internal.h @@ -XXX,XX +XXX,XX @@ void handle_diag_308(CPUS390XState *env, uint64_t r1, uint64_t r3, /* translate.c */ void s390x_translate_init(void); +void s390x_translate_code(CPUState *cs, TranslationBlock *tb, + int *max_insns, vaddr pc, void *host_pc); void s390x_restore_state_to_opc(CPUState *cs, const TranslationBlock *tb, const uint64_t *data); diff --git a/target/sh4/cpu.h b/target/sh4/cpu.h index XXXXXXX..XXXXXXX 100644 --- a/target/sh4/cpu.h +++ b/target/sh4/cpu.h @@ -XXX,XX +XXX,XX @@ G_NORETURN void superh_cpu_do_unaligned_access(CPUState *cpu, vaddr addr, uintptr_t retaddr); void sh4_translate_init(void); +void sh4_translate_code(CPUState *cs, TranslationBlock *tb, + int *max_insns, vaddr pc, void *host_pc); #if !defined(CONFIG_USER_ONLY) hwaddr superh_cpu_get_phys_page_debug(CPUState *cpu, vaddr addr); diff --git a/target/sparc/cpu.h b/target/sparc/cpu.h index XXXXXXX..XXXXXXX 100644 --- a/target/sparc/cpu.h +++ b/target/sparc/cpu.h @@ -XXX,XX +XXX,XX @@ int sparc_cpu_memory_rw_debug(CPUState *cpu, vaddr addr, /* translate.c */ void sparc_tcg_init(void); +void sparc_translate_code(CPUState *cs, TranslationBlock *tb, + int *max_insns, vaddr pc, void *host_pc); /* fop_helper.c */ target_ulong cpu_get_fsr(CPUSPARCState *); diff --git a/target/tricore/cpu.h b/target/tricore/cpu.h index XXXXXXX..XXXXXXX 100644 --- a/target/tricore/cpu.h +++ b/target/tricore/cpu.h @@ -XXX,XX +XXX,XX @@ FIELD(TB_FLAGS, PRIV, 0, 2) void cpu_state_reset(CPUTriCoreState *s); void tricore_tcg_init(void); +void tricore_translate_code(CPUState *cs, TranslationBlock *tb, + int *max_insns, vaddr pc, void *host_pc); static inline void cpu_get_tb_cpu_state(CPUTriCoreState *env, vaddr *pc, uint64_t *cs_base, uint32_t *flags) diff --git a/target/xtensa/cpu.h b/target/xtensa/cpu.h index XXXXXXX..XXXXXXX 100644 --- a/target/xtensa/cpu.h +++ b/target/xtensa/cpu.h @@ -XXX,XX +XXX,XX @@ G_NORETURN void xtensa_cpu_do_unaligned_access(CPUState *cpu, vaddr addr, void xtensa_collect_sr_names(const XtensaConfig *config); void xtensa_translate_init(void); +void xtensa_translate_code(CPUState *cs, TranslationBlock *tb, + int *max_insns, vaddr pc, void *host_pc); void **xtensa_get_regfile_by_name(const char *name, int entries, int bits); void xtensa_breakpoint_handler(CPUState *cs); void xtensa_register_core(XtensaConfigList *node); diff --git a/accel/tcg/cpu-exec.c b/accel/tcg/cpu-exec.c index XXXXXXX..XXXXXXX 100644 --- a/accel/tcg/cpu-exec.c +++ b/accel/tcg/cpu-exec.c @@ -XXX,XX +XXX,XX @@ bool tcg_exec_realizefn(CPUState *cpu, Error **errp) if (!tcg_target_initialized) { /* Check mandatory TCGCPUOps handlers */ + const TCGCPUOps *tcg_ops = cpu->cc->tcg_ops; #ifndef CONFIG_USER_ONLY - assert(cpu->cc->tcg_ops->cpu_exec_halt); - assert(cpu->cc->tcg_ops->cpu_exec_interrupt); + assert(tcg_ops->cpu_exec_halt); + assert(tcg_ops->cpu_exec_interrupt); #endif /* !CONFIG_USER_ONLY */ - cpu->cc->tcg_ops->initialize(); + assert(tcg_ops->translate_code); + tcg_ops->initialize(); tcg_target_initialized = true; } diff --git a/accel/tcg/translate-all.c b/accel/tcg/translate-all.c index XXXXXXX..XXXXXXX 100644 --- a/accel/tcg/translate-all.c +++ b/accel/tcg/translate-all.c @@ -XXX,XX +XXX,XX @@ static int setjmp_gen_code(CPUArchState *env, TranslationBlock *tb, tcg_func_start(tcg_ctx); - tcg_ctx->cpu = env_cpu(env); - gen_intermediate_code(env_cpu(env), tb, max_insns, pc, host_pc); + CPUState *cs = env_cpu(env); + tcg_ctx->cpu = cs; + cs->cc->tcg_ops->translate_code(cs, tb, max_insns, pc, host_pc); + assert(tb->size != 0); tcg_ctx->cpu = NULL; *max_insns = tb->icount; @@ -XXX,XX +XXX,XX @@ TranslationBlock *tb_gen_code(CPUState *cpu, /* * Overflow of code_gen_buffer, or the current slice of it. * - * TODO: We don't need to re-do gen_intermediate_code, nor + * TODO: We don't need to re-do tcg_ops->translate_code, nor * should we re-do the tcg optimization currently hidden * inside tcg_gen_code. All that should be required is to * flush the TBs, allocate a new TB, re-initialize it per diff --git a/target/alpha/cpu.c b/target/alpha/cpu.c index XXXXXXX..XXXXXXX 100644 --- a/target/alpha/cpu.c +++ b/target/alpha/cpu.c @@ -XXX,XX +XXX,XX @@ static const struct SysemuCPUOps alpha_sysemu_ops = { static const TCGCPUOps alpha_tcg_ops = { .initialize = alpha_translate_init, + .translate_code = alpha_translate_code, .synchronize_from_tb = alpha_cpu_synchronize_from_tb, .restore_state_to_opc = alpha_restore_state_to_opc, diff --git a/target/alpha/translate.c b/target/alpha/translate.c index XXXXXXX..XXXXXXX 100644 --- a/target/alpha/translate.c +++ b/target/alpha/translate.c @@ -XXX,XX +XXX,XX @@ static const TranslatorOps alpha_tr_ops = { .tb_stop = alpha_tr_tb_stop, }; -void gen_intermediate_code(CPUState *cpu, TranslationBlock *tb, int *max_insns, - vaddr pc, void *host_pc) +void alpha_translate_code(CPUState *cpu, TranslationBlock *tb, + int *max_insns, vaddr pc, void *host_pc) { DisasContext dc; translator_loop(cpu, tb, max_insns, pc, host_pc, &alpha_tr_ops, &dc.base); diff --git a/target/arm/cpu.c b/target/arm/cpu.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/cpu.c +++ b/target/arm/cpu.c @@ -XXX,XX +XXX,XX @@ static const struct SysemuCPUOps arm_sysemu_ops = { #ifdef CONFIG_TCG static const TCGCPUOps arm_tcg_ops = { .initialize = arm_translate_init, + .translate_code = arm_translate_code, .synchronize_from_tb = arm_cpu_synchronize_from_tb, .debug_excp_handler = arm_debug_excp_handler, .restore_state_to_opc = arm_restore_state_to_opc, diff --git a/target/arm/tcg/cpu-v7m.c b/target/arm/tcg/cpu-v7m.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/tcg/cpu-v7m.c +++ b/target/arm/tcg/cpu-v7m.c @@ -XXX,XX +XXX,XX @@ static void cortex_m55_initfn(Object *obj) static const TCGCPUOps arm_v7m_tcg_ops = { .initialize = arm_translate_init, + .translate_code = arm_translate_code, .synchronize_from_tb = arm_cpu_synchronize_from_tb, .debug_excp_handler = arm_debug_excp_handler, .restore_state_to_opc = arm_restore_state_to_opc, diff --git a/target/arm/tcg/translate.c b/target/arm/tcg/translate.c index XXXXXXX..XXXXXXX 100644 --- a/target/arm/tcg/translate.c +++ b/target/arm/tcg/translate.c @@ -XXX,XX +XXX,XX @@ static const TranslatorOps thumb_translator_ops = { .tb_stop = arm_tr_tb_stop, }; -/* generate intermediate code for basic block 'tb'. */ -void gen_intermediate_code(CPUState *cpu, TranslationBlock *tb, int *max_insns, - vaddr pc, void *host_pc) +void arm_translate_code(CPUState *cpu, TranslationBlock *tb, + int *max_insns, vaddr pc, void *host_pc) { DisasContext dc = { }; const TranslatorOps *ops = &arm_translator_ops; diff --git a/target/avr/cpu.c b/target/avr/cpu.c index XXXXXXX..XXXXXXX 100644 --- a/target/avr/cpu.c +++ b/target/avr/cpu.c @@ -XXX,XX +XXX,XX @@ static const struct SysemuCPUOps avr_sysemu_ops = { static const TCGCPUOps avr_tcg_ops = { .initialize = avr_cpu_tcg_init, + .translate_code = avr_cpu_translate_code, .synchronize_from_tb = avr_cpu_synchronize_from_tb, .restore_state_to_opc = avr_restore_state_to_opc, .cpu_exec_interrupt = avr_cpu_exec_interrupt, diff --git a/target/avr/translate.c b/target/avr/translate.c index XXXXXXX..XXXXXXX 100644 --- a/target/avr/translate.c +++ b/target/avr/translate.c @@ -XXX,XX +XXX,XX @@ static bool trans_WDR(DisasContext *ctx, arg_WDR *a) * * - translate() * - canonicalize_skip() - * - gen_intermediate_code() + * - translate_code() * - restore_state_to_opc() * */ @@ -XXX,XX +XXX,XX @@ static const TranslatorOps avr_tr_ops = { .tb_stop = avr_tr_tb_stop, }; -void gen_intermediate_code(CPUState *cs, TranslationBlock *tb, int *max_insns, - vaddr pc, void *host_pc) +void avr_cpu_translate_code(CPUState *cs, TranslationBlock *tb, + int *max_insns, vaddr pc, void *host_pc) { DisasContext dc = { }; translator_loop(cs, tb, max_insns, pc, host_pc, &avr_tr_ops, &dc.base); diff --git a/target/hexagon/cpu.c b/target/hexagon/cpu.c index XXXXXXX..XXXXXXX 100644 --- a/target/hexagon/cpu.c +++ b/target/hexagon/cpu.c @@ -XXX,XX +XXX,XX @@ static void hexagon_cpu_init(Object *obj) static const TCGCPUOps hexagon_tcg_ops = { .initialize = hexagon_translate_init, + .translate_code = hexagon_translate_code, .synchronize_from_tb = hexagon_cpu_synchronize_from_tb, .restore_state_to_opc = hexagon_restore_state_to_opc, }; diff --git a/target/hexagon/translate.c b/target/hexagon/translate.c index XXXXXXX..XXXXXXX 100644 --- a/target/hexagon/translate.c +++ b/target/hexagon/translate.c @@ -XXX,XX +XXX,XX @@ static const TranslatorOps hexagon_tr_ops = { .tb_stop = hexagon_tr_tb_stop, }; -void gen_intermediate_code(CPUState *cs, TranslationBlock *tb, int *max_insns, - vaddr pc, void *host_pc) +void hexagon_translate_code(CPUState *cs, TranslationBlock *tb, + int *max_insns, vaddr pc, void *host_pc) { DisasContext ctx; diff --git a/target/hppa/cpu.c b/target/hppa/cpu.c index XXXXXXX..XXXXXXX 100644 --- a/target/hppa/cpu.c +++ b/target/hppa/cpu.c @@ -XXX,XX +XXX,XX @@ static const struct SysemuCPUOps hppa_sysemu_ops = { static const TCGCPUOps hppa_tcg_ops = { .initialize = hppa_translate_init, + .translate_code = hppa_translate_code, .synchronize_from_tb = hppa_cpu_synchronize_from_tb, .restore_state_to_opc = hppa_restore_state_to_opc, diff --git a/target/hppa/translate.c b/target/hppa/translate.c index XXXXXXX..XXXXXXX 100644 --- a/target/hppa/translate.c +++ b/target/hppa/translate.c @@ -XXX,XX +XXX,XX @@ static const TranslatorOps hppa_tr_ops = { #endif }; -void gen_intermediate_code(CPUState *cs, TranslationBlock *tb, int *max_insns, - vaddr pc, void *host_pc) +void hppa_translate_code(CPUState *cs, TranslationBlock *tb, + int *max_insns, vaddr pc, void *host_pc) { DisasContext ctx = { }; translator_loop(cs, tb, max_insns, pc, host_pc, &hppa_tr_ops, &ctx.base); diff --git a/target/i386/tcg/tcg-cpu.c b/target/i386/tcg/tcg-cpu.c index XXXXXXX..XXXXXXX 100644 --- a/target/i386/tcg/tcg-cpu.c +++ b/target/i386/tcg/tcg-cpu.c @@ -XXX,XX +XXX,XX @@ static bool x86_debug_check_breakpoint(CPUState *cs) static const TCGCPUOps x86_tcg_ops = { .initialize = tcg_x86_init, + .translate_code = x86_translate_code, .synchronize_from_tb = x86_cpu_synchronize_from_tb, .restore_state_to_opc = x86_restore_state_to_opc, .cpu_exec_enter = x86_cpu_exec_enter, diff --git a/target/i386/tcg/translate.c b/target/i386/tcg/translate.c index XXXXXXX..XXXXXXX 100644 --- a/target/i386/tcg/translate.c +++ b/target/i386/tcg/translate.c @@ -XXX,XX +XXX,XX @@ static const TranslatorOps i386_tr_ops = { .tb_stop = i386_tr_tb_stop, }; -/* generate intermediate code for basic block 'tb'. */ -void gen_intermediate_code(CPUState *cpu, TranslationBlock *tb, int *max_insns, - vaddr pc, void *host_pc) +void x86_translate_code(CPUState *cpu, TranslationBlock *tb, + int *max_insns, vaddr pc, void *host_pc) { DisasContext dc; diff --git a/target/loongarch/cpu.c b/target/loongarch/cpu.c index XXXXXXX..XXXXXXX 100644 --- a/target/loongarch/cpu.c +++ b/target/loongarch/cpu.c @@ -XXX,XX +XXX,XX @@ static void loongarch_cpu_dump_state(CPUState *cs, FILE *f, int flags) static const TCGCPUOps loongarch_tcg_ops = { .initialize = loongarch_translate_init, + .translate_code = loongarch_translate_code, .synchronize_from_tb = loongarch_cpu_synchronize_from_tb, .restore_state_to_opc = loongarch_restore_state_to_opc, diff --git a/target/loongarch/tcg/translate.c b/target/loongarch/tcg/translate.c index XXXXXXX..XXXXXXX 100644 --- a/target/loongarch/tcg/translate.c +++ b/target/loongarch/tcg/translate.c @@ -XXX,XX +XXX,XX @@ static const TranslatorOps loongarch_tr_ops = { .tb_stop = loongarch_tr_tb_stop, }; -void gen_intermediate_code(CPUState *cs, TranslationBlock *tb, int *max_insns, - vaddr pc, void *host_pc) +void loongarch_translate_code(CPUState *cs, TranslationBlock *tb, + int *max_insns, vaddr pc, void *host_pc) { DisasContext ctx; diff --git a/target/m68k/cpu.c b/target/m68k/cpu.c index XXXXXXX..XXXXXXX 100644 --- a/target/m68k/cpu.c +++ b/target/m68k/cpu.c @@ -XXX,XX +XXX,XX @@ static const struct SysemuCPUOps m68k_sysemu_ops = { static const TCGCPUOps m68k_tcg_ops = { .initialize = m68k_tcg_init, + .translate_code = m68k_translate_code, .restore_state_to_opc = m68k_restore_state_to_opc, #ifndef CONFIG_USER_ONLY diff --git a/target/m68k/translate.c b/target/m68k/translate.c index XXXXXXX..XXXXXXX 100644 --- a/target/m68k/translate.c +++ b/target/m68k/translate.c @@ -XXX,XX +XXX,XX @@ static const TranslatorOps m68k_tr_ops = { .tb_stop = m68k_tr_tb_stop, }; -void gen_intermediate_code(CPUState *cpu, TranslationBlock *tb, int *max_insns, - vaddr pc, void *host_pc) +void m68k_translate_code(CPUState *cpu, TranslationBlock *tb, + int *max_insns, vaddr pc, void *host_pc) { DisasContext dc; translator_loop(cpu, tb, max_insns, pc, host_pc, &m68k_tr_ops, &dc.base); diff --git a/target/microblaze/cpu.c b/target/microblaze/cpu.c index XXXXXXX..XXXXXXX 100644 --- a/target/microblaze/cpu.c +++ b/target/microblaze/cpu.c @@ -XXX,XX +XXX,XX @@ static const struct SysemuCPUOps mb_sysemu_ops = { static const TCGCPUOps mb_tcg_ops = { .initialize = mb_tcg_init, + .translate_code = mb_translate_code, .synchronize_from_tb = mb_cpu_synchronize_from_tb, .restore_state_to_opc = mb_restore_state_to_opc, diff --git a/target/microblaze/translate.c b/target/microblaze/translate.c index XXXXXXX..XXXXXXX 100644 --- a/target/microblaze/translate.c +++ b/target/microblaze/translate.c @@ -XXX,XX +XXX,XX @@ static const TranslatorOps mb_tr_ops = { .tb_stop = mb_tr_tb_stop, }; -void gen_intermediate_code(CPUState *cpu, TranslationBlock *tb, int *max_insns, - vaddr pc, void *host_pc) +void mb_translate_code(CPUState *cpu, TranslationBlock *tb, + int *max_insns, vaddr pc, void *host_pc) { DisasContext dc; translator_loop(cpu, tb, max_insns, pc, host_pc, &mb_tr_ops, &dc.base); diff --git a/target/mips/cpu.c b/target/mips/cpu.c index XXXXXXX..XXXXXXX 100644 --- a/target/mips/cpu.c +++ b/target/mips/cpu.c @@ -XXX,XX +XXX,XX @@ static const Property mips_cpu_properties[] = { #include "hw/core/tcg-cpu-ops.h" static const TCGCPUOps mips_tcg_ops = { .initialize = mips_tcg_init, + .translate_code = mips_translate_code, .synchronize_from_tb = mips_cpu_synchronize_from_tb, .restore_state_to_opc = mips_restore_state_to_opc, diff --git a/target/mips/tcg/translate.c b/target/mips/tcg/translate.c index XXXXXXX..XXXXXXX 100644 --- a/target/mips/tcg/translate.c +++ b/target/mips/tcg/translate.c @@ -XXX,XX +XXX,XX @@ static const TranslatorOps mips_tr_ops = { .tb_stop = mips_tr_tb_stop, }; -void gen_intermediate_code(CPUState *cs, TranslationBlock *tb, int *max_insns, - vaddr pc, void *host_pc) +void mips_translate_code(CPUState *cs, TranslationBlock *tb, + int *max_insns, vaddr pc, void *host_pc) { DisasContext ctx; diff --git a/target/openrisc/cpu.c b/target/openrisc/cpu.c index XXXXXXX..XXXXXXX 100644 --- a/target/openrisc/cpu.c +++ b/target/openrisc/cpu.c @@ -XXX,XX +XXX,XX @@ static const struct SysemuCPUOps openrisc_sysemu_ops = { static const TCGCPUOps openrisc_tcg_ops = { .initialize = openrisc_translate_init, + .translate_code = openrisc_translate_code, .synchronize_from_tb = openrisc_cpu_synchronize_from_tb, .restore_state_to_opc = openrisc_restore_state_to_opc, diff --git a/target/openrisc/translate.c b/target/openrisc/translate.c index XXXXXXX..XXXXXXX 100644 --- a/target/openrisc/translate.c +++ b/target/openrisc/translate.c @@ -XXX,XX +XXX,XX @@ static const TranslatorOps openrisc_tr_ops = { .tb_stop = openrisc_tr_tb_stop, }; -void gen_intermediate_code(CPUState *cs, TranslationBlock *tb, int *max_insns, - vaddr pc, void *host_pc) +void openrisc_translate_code(CPUState *cs, TranslationBlock *tb, + int *max_insns, vaddr pc, void *host_pc) { DisasContext ctx; diff --git a/target/ppc/cpu_init.c b/target/ppc/cpu_init.c index XXXXXXX..XXXXXXX 100644 --- a/target/ppc/cpu_init.c +++ b/target/ppc/cpu_init.c @@ -XXX,XX +XXX,XX @@ static const struct SysemuCPUOps ppc_sysemu_ops = { static const TCGCPUOps ppc_tcg_ops = { .initialize = ppc_translate_init, + .translate_code = ppc_translate_code, .restore_state_to_opc = ppc_restore_state_to_opc, #ifdef CONFIG_USER_ONLY diff --git a/target/ppc/translate.c b/target/ppc/translate.c index XXXXXXX..XXXXXXX 100644 --- a/target/ppc/translate.c +++ b/target/ppc/translate.c @@ -XXX,XX +XXX,XX @@ static const TranslatorOps ppc_tr_ops = { .tb_stop = ppc_tr_tb_stop, }; -void gen_intermediate_code(CPUState *cs, TranslationBlock *tb, int *max_insns, - vaddr pc, void *host_pc) +void ppc_translate_code(CPUState *cs, TranslationBlock *tb, + int *max_insns, vaddr pc, void *host_pc) { DisasContext ctx; diff --git a/target/riscv/tcg/tcg-cpu.c b/target/riscv/tcg/tcg-cpu.c index XXXXXXX..XXXXXXX 100644 --- a/target/riscv/tcg/tcg-cpu.c +++ b/target/riscv/tcg/tcg-cpu.c @@ -XXX,XX +XXX,XX @@ static void riscv_restore_state_to_opc(CPUState *cs, static const TCGCPUOps riscv_tcg_ops = { .initialize = riscv_translate_init, + .translate_code = riscv_translate_code, .synchronize_from_tb = riscv_cpu_synchronize_from_tb, .restore_state_to_opc = riscv_restore_state_to_opc, diff --git a/target/riscv/translate.c b/target/riscv/translate.c index XXXXXXX..XXXXXXX 100644 --- a/target/riscv/translate.c +++ b/target/riscv/translate.c @@ -XXX,XX +XXX,XX @@ static const TranslatorOps riscv_tr_ops = { .tb_stop = riscv_tr_tb_stop, }; -void gen_intermediate_code(CPUState *cs, TranslationBlock *tb, int *max_insns, - vaddr pc, void *host_pc) +void riscv_translate_code(CPUState *cs, TranslationBlock *tb, + int *max_insns, vaddr pc, void *host_pc) { DisasContext ctx; diff --git a/target/rx/cpu.c b/target/rx/cpu.c index XXXXXXX..XXXXXXX 100644 --- a/target/rx/cpu.c +++ b/target/rx/cpu.c @@ -XXX,XX +XXX,XX @@ static const struct SysemuCPUOps rx_sysemu_ops = { static const TCGCPUOps rx_tcg_ops = { .initialize = rx_translate_init, + .translate_code = rx_translate_code, .synchronize_from_tb = rx_cpu_synchronize_from_tb, .restore_state_to_opc = rx_restore_state_to_opc, .tlb_fill = rx_cpu_tlb_fill, diff --git a/target/rx/translate.c b/target/rx/translate.c index XXXXXXX..XXXXXXX 100644 --- a/target/rx/translate.c +++ b/target/rx/translate.c @@ -XXX,XX +XXX,XX @@ static const TranslatorOps rx_tr_ops = { .tb_stop = rx_tr_tb_stop, }; -void gen_intermediate_code(CPUState *cs, TranslationBlock *tb, int *max_insns, - vaddr pc, void *host_pc) +void rx_translate_code(CPUState *cs, TranslationBlock *tb, + int *max_insns, vaddr pc, void *host_pc) { DisasContext dc; diff --git a/target/s390x/cpu.c b/target/s390x/cpu.c index XXXXXXX..XXXXXXX 100644 --- a/target/s390x/cpu.c +++ b/target/s390x/cpu.c @@ -XXX,XX +XXX,XX @@ void cpu_get_tb_cpu_state(CPUS390XState *env, vaddr *pc, static const TCGCPUOps s390_tcg_ops = { .initialize = s390x_translate_init, + .translate_code = s390x_translate_code, .restore_state_to_opc = s390x_restore_state_to_opc, #ifdef CONFIG_USER_ONLY diff --git a/target/s390x/tcg/translate.c b/target/s390x/tcg/translate.c index XXXXXXX..XXXXXXX 100644 --- a/target/s390x/tcg/translate.c +++ b/target/s390x/tcg/translate.c @@ -XXX,XX +XXX,XX @@ static const TranslatorOps s390x_tr_ops = { .disas_log = s390x_tr_disas_log, }; -void gen_intermediate_code(CPUState *cs, TranslationBlock *tb, int *max_insns, - vaddr pc, void *host_pc) +void s390x_translate_code(CPUState *cs, TranslationBlock *tb, + int *max_insns, vaddr pc, void *host_pc) { DisasContext dc; diff --git a/target/sh4/cpu.c b/target/sh4/cpu.c index XXXXXXX..XXXXXXX 100644 --- a/target/sh4/cpu.c +++ b/target/sh4/cpu.c @@ -XXX,XX +XXX,XX @@ static const struct SysemuCPUOps sh4_sysemu_ops = { static const TCGCPUOps superh_tcg_ops = { .initialize = sh4_translate_init, + .translate_code = sh4_translate_code, .synchronize_from_tb = superh_cpu_synchronize_from_tb, .restore_state_to_opc = superh_restore_state_to_opc, diff --git a/target/sh4/translate.c b/target/sh4/translate.c index XXXXXXX..XXXXXXX 100644 --- a/target/sh4/translate.c +++ b/target/sh4/translate.c @@ -XXX,XX +XXX,XX @@ static const TranslatorOps sh4_tr_ops = { .tb_stop = sh4_tr_tb_stop, }; -void gen_intermediate_code(CPUState *cs, TranslationBlock *tb, int *max_insns, - vaddr pc, void *host_pc) +void sh4_translate_code(CPUState *cs, TranslationBlock *tb, + int *max_insns, vaddr pc, void *host_pc) { DisasContext ctx; diff --git a/target/sparc/cpu.c b/target/sparc/cpu.c index XXXXXXX..XXXXXXX 100644 --- a/target/sparc/cpu.c +++ b/target/sparc/cpu.c @@ -XXX,XX +XXX,XX @@ static const struct SysemuCPUOps sparc_sysemu_ops = { static const TCGCPUOps sparc_tcg_ops = { .initialize = sparc_tcg_init, + .translate_code = sparc_translate_code, .synchronize_from_tb = sparc_cpu_synchronize_from_tb, .restore_state_to_opc = sparc_restore_state_to_opc, diff --git a/target/sparc/translate.c b/target/sparc/translate.c index XXXXXXX..XXXXXXX 100644 --- a/target/sparc/translate.c +++ b/target/sparc/translate.c @@ -XXX,XX +XXX,XX @@ static const TranslatorOps sparc_tr_ops = { .tb_stop = sparc_tr_tb_stop, }; -void gen_intermediate_code(CPUState *cs, TranslationBlock *tb, int *max_insns, - vaddr pc, void *host_pc) +void sparc_translate_code(CPUState *cs, TranslationBlock *tb, + int *max_insns, vaddr pc, void *host_pc) { DisasContext dc = {}; diff --git a/target/tricore/cpu.c b/target/tricore/cpu.c index XXXXXXX..XXXXXXX 100644 --- a/target/tricore/cpu.c +++ b/target/tricore/cpu.c @@ -XXX,XX +XXX,XX @@ static const struct SysemuCPUOps tricore_sysemu_ops = { static const TCGCPUOps tricore_tcg_ops = { .initialize = tricore_tcg_init, + .translate_code = tricore_translate_code, .synchronize_from_tb = tricore_cpu_synchronize_from_tb, .restore_state_to_opc = tricore_restore_state_to_opc, .tlb_fill = tricore_cpu_tlb_fill, diff --git a/target/tricore/translate.c b/target/tricore/translate.c index XXXXXXX..XXXXXXX 100644 --- a/target/tricore/translate.c +++ b/target/tricore/translate.c @@ -XXX,XX +XXX,XX @@ static const TranslatorOps tricore_tr_ops = { .tb_stop = tricore_tr_tb_stop, }; - -void gen_intermediate_code(CPUState *cs, TranslationBlock *tb, int *max_insns, - vaddr pc, void *host_pc) +void tricore_translate_code(CPUState *cs, TranslationBlock *tb, + int *max_insns, vaddr pc, void *host_pc) { DisasContext ctx; translator_loop(cs, tb, max_insns, pc, host_pc, diff --git a/target/xtensa/cpu.c b/target/xtensa/cpu.c index XXXXXXX..XXXXXXX 100644 --- a/target/xtensa/cpu.c +++ b/target/xtensa/cpu.c @@ -XXX,XX +XXX,XX @@ static const struct SysemuCPUOps xtensa_sysemu_ops = { static const TCGCPUOps xtensa_tcg_ops = { .initialize = xtensa_translate_init, + .translate_code = xtensa_translate_code, .debug_excp_handler = xtensa_breakpoint_handler, .restore_state_to_opc = xtensa_restore_state_to_opc, diff --git a/target/xtensa/translate.c b/target/xtensa/translate.c index XXXXXXX..XXXXXXX 100644 --- a/target/xtensa/translate.c +++ b/target/xtensa/translate.c @@ -XXX,XX +XXX,XX @@ static const TranslatorOps xtensa_translator_ops = { .tb_stop = xtensa_tr_tb_stop, }; -void gen_intermediate_code(CPUState *cpu, TranslationBlock *tb, int *max_insns, - vaddr pc, void *host_pc) +void xtensa_translate_code(CPUState *cpu, TranslationBlock *tb, + int *max_insns, vaddr pc, void *host_pc) { DisasContext dc = {}; translator_loop(cpu, tb, max_insns, pc, host_pc, -- 2.43.0