1 | target-arm queue: the big things in here are SVE in system | 1 | A few last patches to go in for rc3... |
---|---|---|---|
2 | emulation mode, and v8M stack limit checking; there are | ||
3 | also a handful of smaller fixes. | ||
4 | 2 | ||
5 | thanks | 3 | The following changes since commit c1e90def01bdb8fcbdbebd9d1eaa8e4827ece620: |
6 | -- PMM | ||
7 | 4 | ||
8 | The following changes since commit 079911cb6e26898e16f5bb56ef4f9d33cf92d32d: | 5 | Merge remote-tracking branch 'remotes/pmaydell/tags/pull-target-arm-20210412' into staging (2021-04-12 12:12:09 +0100) |
9 | |||
10 | Merge remote-tracking branch 'remotes/rth/tags/pull-fpu-20181005' into staging (2018-10-08 12:44:35 +0100) | ||
11 | 6 | ||
12 | are available in the Git repository at: | 7 | are available in the Git repository at: |
13 | 8 | ||
14 | https://git.linaro.org/people/pmaydell/qemu-arm.git tags/pull-target-arm-20181008 | 9 | https://git.linaro.org/people/pmaydell/qemu-arm.git tags/pull-target-arm-20210413 |
15 | 10 | ||
16 | for you to fetch changes up to 74e2e59b8d0a68be0956310fc349179c89fd7be0: | 11 | for you to fetch changes up to 2d18b4ca023ca1a3aee18064251d6e6e1084f3eb: |
17 | 12 | ||
18 | hw/display/bcm2835_fb: Silence Coverity warning about multiply overflow (2018-10-08 14:55:05 +0100) | 13 | sphinx: qapidoc: Wrap "If" section body in a paragraph node (2021-04-13 10:14:58 +0100) |
19 | 14 | ||
20 | ---------------------------------------------------------------- | 15 | ---------------------------------------------------------------- |
21 | target-arm queue: | 16 | target-arm queue: |
22 | * target/arm: fix error in a code comment | 17 | * Fix MPC setting for AN524 SRAM block |
23 | * virt: Suppress external aborts on virt-2.10 and earlier | 18 | * sphinx: qapidoc: Wrap "If" section body in a paragraph node |
24 | * target/arm: Correct condition for v8M callee stack push | ||
25 | * target/arm: Don't read r4 from v8M exception stackframe twice | ||
26 | * target/arm: Support SVE in system emulation mode | ||
27 | * target/arm: Implement v8M hardware stack limit checking | ||
28 | * hw/display/bcm2835_fb: Silence Coverity warning about multiply overflow | ||
29 | 19 | ||
30 | ---------------------------------------------------------------- | 20 | ---------------------------------------------------------------- |
31 | Dongjiu Geng (1): | 21 | John Snow (1): |
32 | target/arm: fix code comments error | 22 | sphinx: qapidoc: Wrap "If" section body in a paragraph node |
33 | 23 | ||
34 | Peter Maydell (17): | 24 | Peter Maydell (2): |
35 | virt: Suppress external aborts on virt-2.10 and earlier | 25 | hw/arm/mps2-tz: Fix MPC setting for AN524 SRAM block |
36 | target/arm: Correct condition for v8M callee stack push | 26 | hw/arm/mps2-tz: Assert if more than one RAM is attached to an MPC |
37 | target/arm: Don't read r4 from v8M exception stackframe twice | ||
38 | target/arm: Define new TBFLAG for v8M stack checking | ||
39 | target/arm: Define new EXCP type for v8M stack overflows | ||
40 | target/arm: Move v7m_using_psp() to internals.h | ||
41 | target/arm: Add v8M stack checks on ADD/SUB/MOV of SP | ||
42 | target/arm: Add some comments in Thumb decode | ||
43 | target/arm: Add v8M stack checks on exception entry | ||
44 | target/arm: Add v8M stack limit checks on NS function calls | ||
45 | target/arm: Add v8M stack checks for LDRD/STRD (imm) | ||
46 | target/arm: Add v8M stack checks for Thumb2 LDM/STM | ||
47 | target/arm: Add v8M stack checks for T32 load/store single | ||
48 | target/arm: Add v8M stack checks for Thumb push/pop | ||
49 | target/arm: Add v8M stack checks for VLDM/VSTM | ||
50 | target/arm: Add v8M stack checks for MSR to SP_NS | ||
51 | hw/display/bcm2835_fb: Silence Coverity warning about multiply overflow | ||
52 | 27 | ||
53 | Richard Henderson (15): | 28 | docs/sphinx/qapidoc.py | 4 +++- |
54 | target/arm: Define ID_AA64ZFR0_EL1 | 29 | hw/arm/mps2-tz.c | 10 +++++++--- |
55 | target/arm: Adjust sve_exception_el | 30 | 2 files changed, 10 insertions(+), 4 deletions(-) |
56 | target/arm: Pass in current_el to fp and sve_exception_el | ||
57 | target/arm: Handle SVE vector length changes in system mode | ||
58 | target/arm: Adjust aarch64_cpu_dump_state for system mode SVE | ||
59 | target/arm: Clear unused predicate bits for LD1RQ | ||
60 | target/arm: Rewrite helper_sve_ld1*_r using pages | ||
61 | target/arm: Rewrite helper_sve_ld[234]*_r | ||
62 | target/arm: Rewrite helper_sve_st[1234]*_r | ||
63 | target/arm: Split contiguous loads for endianness | ||
64 | target/arm: Split contiguous stores for endianness | ||
65 | target/arm: Rewrite vector gather loads | ||
66 | target/arm: Rewrite vector gather stores | ||
67 | target/arm: Rewrite vector gather first-fault loads | ||
68 | target/arm: Pass TCGMemOpIdx to sve memory helpers | ||
69 | 31 | ||
70 | target/arm/cpu.h | 17 + | ||
71 | target/arm/helper-sve.h | 385 ++++++--- | ||
72 | target/arm/helper.h | 2 + | ||
73 | target/arm/internals.h | 44 + | ||
74 | target/arm/kvm_arm.h | 4 +- | ||
75 | target/arm/translate.h | 1 + | ||
76 | hw/arm/virt.c | 2 + | ||
77 | hw/display/bcm2835_fb.c | 2 +- | ||
78 | target/arm/cpu64.c | 42 - | ||
79 | target/arm/helper.c | 345 +++++--- | ||
80 | target/arm/kvm.c | 2 +- | ||
81 | target/arm/op_helper.c | 24 +- | ||
82 | target/arm/sve_helper.c | 1961 ++++++++++++++++++++++++++++++-------------- | ||
83 | target/arm/translate-a64.c | 8 +- | ||
84 | target/arm/translate-sve.c | 670 ++++++++++----- | ||
85 | target/arm/translate.c | 198 ++++- | ||
86 | 16 files changed, 2611 insertions(+), 1096 deletions(-) | ||
87 | diff view generated by jsdifflib |
Deleted patch | |||
---|---|---|---|
1 | From: Dongjiu Geng <gengdongjiu@huawei.com> | ||
2 | 1 | ||
3 | The parameter of kvm_arm_init_cpreg_list() is ARMCPU instead of | ||
4 | CPUState, so correct the note to make it match the code. | ||
5 | |||
6 | Signed-off-by: Dongjiu Geng <gengdongjiu@huawei.com> | ||
7 | Message-id: 1538069046-5757-1-git-send-email-gengdongjiu@huawei.com | ||
8 | Reviewed-by: Peter Maydell <peter.maydell@linaro.org> | ||
9 | Signed-off-by: Peter Maydell <peter.maydell@linaro.org> | ||
10 | --- | ||
11 | target/arm/kvm_arm.h | 4 ++-- | ||
12 | target/arm/kvm.c | 2 +- | ||
13 | 2 files changed, 3 insertions(+), 3 deletions(-) | ||
14 | |||
15 | diff --git a/target/arm/kvm_arm.h b/target/arm/kvm_arm.h | ||
16 | index XXXXXXX..XXXXXXX 100644 | ||
17 | --- a/target/arm/kvm_arm.h | ||
18 | +++ b/target/arm/kvm_arm.h | ||
19 | @@ -XXX,XX +XXX,XX @@ void kvm_arm_register_device(MemoryRegion *mr, uint64_t devid, uint64_t group, | ||
20 | |||
21 | /** | ||
22 | * kvm_arm_init_cpreg_list: | ||
23 | - * @cs: CPUState | ||
24 | + * @cpu: ARMCPU | ||
25 | * | ||
26 | - * Initialize the CPUState's cpreg list according to the kernel's | ||
27 | + * Initialize the ARMCPU cpreg list according to the kernel's | ||
28 | * definition of what CPU registers it knows about (and throw away | ||
29 | * the previous TCG-created cpreg list). | ||
30 | * | ||
31 | diff --git a/target/arm/kvm.c b/target/arm/kvm.c | ||
32 | index XXXXXXX..XXXXXXX 100644 | ||
33 | --- a/target/arm/kvm.c | ||
34 | +++ b/target/arm/kvm.c | ||
35 | @@ -XXX,XX +XXX,XX @@ static int compare_u64(const void *a, const void *b) | ||
36 | return 0; | ||
37 | } | ||
38 | |||
39 | -/* Initialize the CPUState's cpreg list according to the kernel's | ||
40 | +/* Initialize the ARMCPU cpreg list according to the kernel's | ||
41 | * definition of what CPU registers it knows about (and throw away | ||
42 | * the previous TCG-created cpreg list). | ||
43 | */ | ||
44 | -- | ||
45 | 2.19.0 | ||
46 | |||
47 | diff view generated by jsdifflib |
1 | In v7m_exception_taken() we were incorrectly using a | 1 | The AN524 has three MPCs: one for the BRAM, one for the QSPI flash, |
---|---|---|---|
2 | "LR bit EXCRET.ES is 1" check when it should be 0 | 2 | and one for the DDR. We incorrectly set the .mpc field in the |
3 | (compare the pseudocode ExceptionTaken() function). | 3 | RAMInfo struct for the SRAM block to 1, giving it the same MPC we are |
4 | This meant we didn't stack the callee-saved registers | 4 | using for the QSPI. The effect of this was that the QSPI didn't get |
5 | when tailchaining from a NonSecure to a Secure exception. | 5 | mapped into the system address space at all, via an MPC or otherwise, |
6 | and guest programs which tried to read from the QSPI would get a bus | ||
7 | error. Correct the SRAM RAMInfo to indicate that it does not have an | ||
8 | associated MPC. | ||
6 | 9 | ||
7 | Cc: qemu-stable@nongnu.org | 10 | Fixes: 25ff112a8cc ("hw/arm/mps2-tz: Add new mps3-an524 board") |
8 | Signed-off-by: Peter Maydell <peter.maydell@linaro.org> | 11 | Signed-off-by: Peter Maydell <peter.maydell@linaro.org> |
9 | Reviewed-by: Richard Henderson <richard.henderson@linaro.org> | 12 | Reviewed-by: Richard Henderson <richard.henderson@linaro.org> |
10 | Message-id: 20181002145940.30931-1-peter.maydell@linaro.org | 13 | Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org> |
14 | Message-id: 20210409150527.15053-2-peter.maydell@linaro.org | ||
11 | --- | 15 | --- |
12 | target/arm/helper.c | 2 +- | 16 | hw/arm/mps2-tz.c | 2 +- |
13 | 1 file changed, 1 insertion(+), 1 deletion(-) | 17 | 1 file changed, 1 insertion(+), 1 deletion(-) |
14 | 18 | ||
15 | diff --git a/target/arm/helper.c b/target/arm/helper.c | 19 | diff --git a/hw/arm/mps2-tz.c b/hw/arm/mps2-tz.c |
16 | index XXXXXXX..XXXXXXX 100644 | 20 | index XXXXXXX..XXXXXXX 100644 |
17 | --- a/target/arm/helper.c | 21 | --- a/hw/arm/mps2-tz.c |
18 | +++ b/target/arm/helper.c | 22 | +++ b/hw/arm/mps2-tz.c |
19 | @@ -XXX,XX +XXX,XX @@ static void v7m_exception_taken(ARMCPU *cpu, uint32_t lr, bool dotailchain, | 23 | @@ -XXX,XX +XXX,XX @@ static const RAMInfo an524_raminfo[] = { { |
20 | * not already saved. | 24 | .name = "sram", |
21 | */ | 25 | .base = 0x20000000, |
22 | if (lr & R_V7M_EXCRET_DCRS_MASK && | 26 | .size = 32 * 4 * KiB, |
23 | - !(dotailchain && (lr & R_V7M_EXCRET_ES_MASK))) { | 27 | - .mpc = 1, |
24 | + !(dotailchain && !(lr & R_V7M_EXCRET_ES_MASK))) { | 28 | + .mpc = -1, |
25 | push_failed = v7m_push_callee_stack(cpu, lr, dotailchain, | 29 | .mrindex = 1, |
26 | ignore_stackfaults); | 30 | }, { |
27 | } | 31 | /* We don't model QSPI flash yet; for now expose it as simple ROM */ |
28 | -- | 32 | -- |
29 | 2.19.0 | 33 | 2.20.1 |
30 | 34 | ||
31 | 35 | diff view generated by jsdifflib |
1 | Add code to insert calls to a helper function to do the stack | 1 | Each board in mps2-tz.c specifies a RAMInfo[] array providing |
---|---|---|---|
2 | limit checking when we handle these forms of instruction | 2 | information about each RAM in the board. The .mpc field of the |
3 | that write to SP: | 3 | RAMInfo struct specifies which MPC, if any, the RAM is attached to. |
4 | * ADD (SP plus immediate) | 4 | We already assert if the array doesn't have any entry for an MPC, but |
5 | * ADD (SP plus register) | 5 | we don't diagnose the error of using the same MPC number twice (which |
6 | * SUB (SP minus immediate) | 6 | is quite easy to do by accident if copy-and-pasting structure |
7 | * SUB (SP minus register) | 7 | entries). |
8 | * MOV (register) | 8 | |
9 | Enhance find_raminfo_for_mpc() so that it detects multiple entries | ||
10 | for the MPC as well as missing entries. | ||
9 | 11 | ||
10 | Signed-off-by: Peter Maydell <peter.maydell@linaro.org> | 12 | Signed-off-by: Peter Maydell <peter.maydell@linaro.org> |
11 | Reviewed-by: Richard Henderson <richard.henderson@linaro.org> | 13 | Reviewed-by: Richard Henderson <richard.henderson@linaro.org> |
12 | Message-id: 20181002163556.10279-5-peter.maydell@linaro.org | 14 | Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org> |
15 | Message-id: 20210409150527.15053-3-peter.maydell@linaro.org | ||
13 | --- | 16 | --- |
14 | target/arm/helper.h | 2 ++ | 17 | hw/arm/mps2-tz.c | 8 ++++++-- |
15 | target/arm/internals.h | 14 ++++++++ | 18 | 1 file changed, 6 insertions(+), 2 deletions(-) |
16 | target/arm/op_helper.c | 19 ++++++++++ | ||
17 | target/arm/translate.c | 80 +++++++++++++++++++++++++++++++++++++----- | ||
18 | 4 files changed, 106 insertions(+), 9 deletions(-) | ||
19 | 19 | ||
20 | diff --git a/target/arm/helper.h b/target/arm/helper.h | 20 | diff --git a/hw/arm/mps2-tz.c b/hw/arm/mps2-tz.c |
21 | index XXXXXXX..XXXXXXX 100644 | 21 | index XXXXXXX..XXXXXXX 100644 |
22 | --- a/target/arm/helper.h | 22 | --- a/hw/arm/mps2-tz.c |
23 | +++ b/target/arm/helper.h | 23 | +++ b/hw/arm/mps2-tz.c |
24 | @@ -XXX,XX +XXX,XX @@ DEF_HELPER_2(v7m_blxns, void, env, i32) | 24 | @@ -XXX,XX +XXX,XX @@ static const RAMInfo *find_raminfo_for_mpc(MPS2TZMachineState *mms, int mpc) |
25 | 25 | { | |
26 | DEF_HELPER_3(v7m_tt, i32, env, i32, i32) | 26 | MPS2TZMachineClass *mmc = MPS2TZ_MACHINE_GET_CLASS(mms); |
27 | 27 | const RAMInfo *p; | |
28 | +DEF_HELPER_2(v8m_stackcheck, void, env, i32) | 28 | + const RAMInfo *found = NULL; |
29 | + | 29 | |
30 | DEF_HELPER_4(access_check_cp_reg, void, env, ptr, i32, i32) | 30 | for (p = mmc->raminfo; p->name; p++) { |
31 | DEF_HELPER_3(set_cp_reg, void, env, ptr, i32) | 31 | if (p->mpc == mpc && !(p->flags & IS_ALIAS)) { |
32 | DEF_HELPER_2(get_cp_reg, i32, env, ptr) | 32 | - return p; |
33 | diff --git a/target/arm/internals.h b/target/arm/internals.h | 33 | + /* There should only be one entry in the array for this MPC */ |
34 | index XXXXXXX..XXXXXXX 100644 | 34 | + g_assert(!found); |
35 | --- a/target/arm/internals.h | 35 | + found = p; |
36 | +++ b/target/arm/internals.h | 36 | } |
37 | @@ -XXX,XX +XXX,XX @@ static inline bool v7m_using_psp(CPUARMState *env) | 37 | } |
38 | env->v7m.control[env->v7m.secure] & R_V7M_CONTROL_SPSEL_MASK; | 38 | /* if raminfo array doesn't have an entry for each MPC this is a bug */ |
39 | - g_assert_not_reached(); | ||
40 | + assert(found); | ||
41 | + return found; | ||
39 | } | 42 | } |
40 | 43 | ||
41 | +/** | 44 | static MemoryRegion *mr_for_raminfo(MPS2TZMachineState *mms, |
42 | + * v7m_sp_limit: Return SP limit for current CPU state | ||
43 | + * Return the SP limit value for the current CPU security state | ||
44 | + * and stack pointer. | ||
45 | + */ | ||
46 | +static inline uint32_t v7m_sp_limit(CPUARMState *env) | ||
47 | +{ | ||
48 | + if (v7m_using_psp(env)) { | ||
49 | + return env->v7m.psplim[env->v7m.secure]; | ||
50 | + } else { | ||
51 | + return env->v7m.msplim[env->v7m.secure]; | ||
52 | + } | ||
53 | +} | ||
54 | + | ||
55 | #endif | ||
56 | diff --git a/target/arm/op_helper.c b/target/arm/op_helper.c | ||
57 | index XXXXXXX..XXXXXXX 100644 | ||
58 | --- a/target/arm/op_helper.c | ||
59 | +++ b/target/arm/op_helper.c | ||
60 | @@ -XXX,XX +XXX,XX @@ void arm_cpu_do_transaction_failed(CPUState *cs, hwaddr physaddr, | ||
61 | |||
62 | #endif /* !defined(CONFIG_USER_ONLY) */ | ||
63 | |||
64 | +void HELPER(v8m_stackcheck)(CPUARMState *env, uint32_t newvalue) | ||
65 | +{ | ||
66 | + /* | ||
67 | + * Perform the v8M stack limit check for SP updates from translated code, | ||
68 | + * raising an exception if the limit is breached. | ||
69 | + */ | ||
70 | + if (newvalue < v7m_sp_limit(env)) { | ||
71 | + CPUState *cs = CPU(arm_env_get_cpu(env)); | ||
72 | + | ||
73 | + /* | ||
74 | + * Stack limit exceptions are a rare case, so rather than syncing | ||
75 | + * PC/condbits before the call, we use cpu_restore_state() to | ||
76 | + * get them right before raising the exception. | ||
77 | + */ | ||
78 | + cpu_restore_state(cs, GETPC(), true); | ||
79 | + raise_exception(env, EXCP_STKOF, 0, 1); | ||
80 | + } | ||
81 | +} | ||
82 | + | ||
83 | uint32_t HELPER(add_setq)(CPUARMState *env, uint32_t a, uint32_t b) | ||
84 | { | ||
85 | uint32_t res = a + b; | ||
86 | diff --git a/target/arm/translate.c b/target/arm/translate.c | ||
87 | index XXXXXXX..XXXXXXX 100644 | ||
88 | --- a/target/arm/translate.c | ||
89 | +++ b/target/arm/translate.c | ||
90 | @@ -XXX,XX +XXX,XX @@ static void store_reg(DisasContext *s, int reg, TCGv_i32 var) | ||
91 | tcg_temp_free_i32(var); | ||
92 | } | ||
93 | |||
94 | +/* | ||
95 | + * Variant of store_reg which applies v8M stack-limit checks before updating | ||
96 | + * SP. If the check fails this will result in an exception being taken. | ||
97 | + * We disable the stack checks for CONFIG_USER_ONLY because we have | ||
98 | + * no idea what the stack limits should be in that case. | ||
99 | + * If stack checking is not being done this just acts like store_reg(). | ||
100 | + */ | ||
101 | +static void store_sp_checked(DisasContext *s, TCGv_i32 var) | ||
102 | +{ | ||
103 | +#ifndef CONFIG_USER_ONLY | ||
104 | + if (s->v8m_stackcheck) { | ||
105 | + gen_helper_v8m_stackcheck(cpu_env, var); | ||
106 | + } | ||
107 | +#endif | ||
108 | + store_reg(s, 13, var); | ||
109 | +} | ||
110 | + | ||
111 | /* Value extensions. */ | ||
112 | #define gen_uxtb(var) tcg_gen_ext8u_i32(var, var) | ||
113 | #define gen_uxth(var) tcg_gen_ext16u_i32(var, var) | ||
114 | @@ -XXX,XX +XXX,XX @@ static void disas_thumb2_insn(DisasContext *s, uint32_t insn) | ||
115 | if (gen_thumb2_data_op(s, op, conds, 0, tmp, tmp2)) | ||
116 | goto illegal_op; | ||
117 | tcg_temp_free_i32(tmp2); | ||
118 | - if (rd != 15) { | ||
119 | + if (rd == 13 && | ||
120 | + ((op == 2 && rn == 15) || | ||
121 | + (op == 8 && rn == 13) || | ||
122 | + (op == 13 && rn == 13))) { | ||
123 | + /* MOV SP, ... or ADD SP, SP, ... or SUB SP, SP, ... */ | ||
124 | + store_sp_checked(s, tmp); | ||
125 | + } else if (rd != 15) { | ||
126 | store_reg(s, rd, tmp); | ||
127 | } else { | ||
128 | tcg_temp_free_i32(tmp); | ||
129 | @@ -XXX,XX +XXX,XX @@ static void disas_thumb2_insn(DisasContext *s, uint32_t insn) | ||
130 | gen_jmp(s, s->pc + offset); | ||
131 | } | ||
132 | } else { | ||
133 | - /* Data processing immediate. */ | ||
134 | + /* | ||
135 | + * 0b1111_0xxx_xxxx_0xxx_xxxx_xxxx | ||
136 | + * - Data-processing (modified immediate, plain binary immediate) | ||
137 | + */ | ||
138 | if (insn & (1 << 25)) { | ||
139 | + /* | ||
140 | + * 0b1111_0x1x_xxxx_0xxx_xxxx_xxxx | ||
141 | + * - Data-processing (plain binary immediate) | ||
142 | + */ | ||
143 | if (insn & (1 << 24)) { | ||
144 | if (insn & (1 << 20)) | ||
145 | goto illegal_op; | ||
146 | @@ -XXX,XX +XXX,XX @@ static void disas_thumb2_insn(DisasContext *s, uint32_t insn) | ||
147 | tmp = tcg_temp_new_i32(); | ||
148 | tcg_gen_movi_i32(tmp, imm); | ||
149 | } | ||
150 | + store_reg(s, rd, tmp); | ||
151 | } else { | ||
152 | /* Add/sub 12-bit immediate. */ | ||
153 | if (rn == 15) { | ||
154 | @@ -XXX,XX +XXX,XX @@ static void disas_thumb2_insn(DisasContext *s, uint32_t insn) | ||
155 | offset += imm; | ||
156 | tmp = tcg_temp_new_i32(); | ||
157 | tcg_gen_movi_i32(tmp, offset); | ||
158 | + store_reg(s, rd, tmp); | ||
159 | } else { | ||
160 | tmp = load_reg(s, rn); | ||
161 | if (insn & (1 << 23)) | ||
162 | tcg_gen_subi_i32(tmp, tmp, imm); | ||
163 | else | ||
164 | tcg_gen_addi_i32(tmp, tmp, imm); | ||
165 | + if (rn == 13 && rd == 13) { | ||
166 | + /* ADD SP, SP, imm or SUB SP, SP, imm */ | ||
167 | + store_sp_checked(s, tmp); | ||
168 | + } else { | ||
169 | + store_reg(s, rd, tmp); | ||
170 | + } | ||
171 | } | ||
172 | } | ||
173 | - store_reg(s, rd, tmp); | ||
174 | } | ||
175 | } else { | ||
176 | + /* | ||
177 | + * 0b1111_0x0x_xxxx_0xxx_xxxx_xxxx | ||
178 | + * - Data-processing (modified immediate) | ||
179 | + */ | ||
180 | int shifter_out = 0; | ||
181 | /* modified 12-bit immediate. */ | ||
182 | shift = ((insn & 0x04000000) >> 23) | ((insn & 0x7000) >> 12); | ||
183 | @@ -XXX,XX +XXX,XX @@ static void disas_thumb2_insn(DisasContext *s, uint32_t insn) | ||
184 | goto illegal_op; | ||
185 | tcg_temp_free_i32(tmp2); | ||
186 | rd = (insn >> 8) & 0xf; | ||
187 | - if (rd != 15) { | ||
188 | + if (rd == 13 && rn == 13 | ||
189 | + && (op == 8 || op == 13)) { | ||
190 | + /* ADD(S) SP, SP, imm or SUB(S) SP, SP, imm */ | ||
191 | + store_sp_checked(s, tmp); | ||
192 | + } else if (rd != 15) { | ||
193 | store_reg(s, rd, tmp); | ||
194 | } else { | ||
195 | tcg_temp_free_i32(tmp); | ||
196 | @@ -XXX,XX +XXX,XX @@ static void disas_thumb_insn(DisasContext *s, uint32_t insn) | ||
197 | tmp2 = load_reg(s, rm); | ||
198 | tcg_gen_add_i32(tmp, tmp, tmp2); | ||
199 | tcg_temp_free_i32(tmp2); | ||
200 | - store_reg(s, rd, tmp); | ||
201 | + if (rd == 13) { | ||
202 | + /* ADD SP, SP, reg */ | ||
203 | + store_sp_checked(s, tmp); | ||
204 | + } else { | ||
205 | + store_reg(s, rd, tmp); | ||
206 | + } | ||
207 | break; | ||
208 | case 1: /* cmp */ | ||
209 | tmp = load_reg(s, rd); | ||
210 | @@ -XXX,XX +XXX,XX @@ static void disas_thumb_insn(DisasContext *s, uint32_t insn) | ||
211 | break; | ||
212 | case 2: /* mov/cpy */ | ||
213 | tmp = load_reg(s, rm); | ||
214 | - store_reg(s, rd, tmp); | ||
215 | + if (rd == 13) { | ||
216 | + /* MOV SP, reg */ | ||
217 | + store_sp_checked(s, tmp); | ||
218 | + } else { | ||
219 | + store_reg(s, rd, tmp); | ||
220 | + } | ||
221 | break; | ||
222 | case 3: | ||
223 | { | ||
224 | @@ -XXX,XX +XXX,XX @@ static void disas_thumb_insn(DisasContext *s, uint32_t insn) | ||
225 | break; | ||
226 | |||
227 | case 10: | ||
228 | - /* add to high reg */ | ||
229 | + /* | ||
230 | + * 0b1010_xxxx_xxxx_xxxx | ||
231 | + * - Add PC/SP (immediate) | ||
232 | + */ | ||
233 | rd = (insn >> 8) & 7; | ||
234 | if (insn & (1 << 11)) { | ||
235 | /* SP */ | ||
236 | @@ -XXX,XX +XXX,XX @@ static void disas_thumb_insn(DisasContext *s, uint32_t insn) | ||
237 | op = (insn >> 8) & 0xf; | ||
238 | switch (op) { | ||
239 | case 0: | ||
240 | - /* adjust stack pointer */ | ||
241 | + /* | ||
242 | + * 0b1011_0000_xxxx_xxxx | ||
243 | + * - ADD (SP plus immediate) | ||
244 | + * - SUB (SP minus immediate) | ||
245 | + */ | ||
246 | tmp = load_reg(s, 13); | ||
247 | val = (insn & 0x7f) * 4; | ||
248 | if (insn & (1 << 7)) | ||
249 | val = -(int32_t)val; | ||
250 | tcg_gen_addi_i32(tmp, tmp, val); | ||
251 | - store_reg(s, 13, tmp); | ||
252 | + store_sp_checked(s, tmp); | ||
253 | break; | ||
254 | |||
255 | case 2: /* sign/zero extend. */ | ||
256 | -- | 45 | -- |
257 | 2.19.0 | 46 | 2.20.1 |
258 | 47 | ||
259 | 48 | diff view generated by jsdifflib |
1 | In commit c79c0a314c43b78 we enabled emulation of external aborts | 1 | From: John Snow <jsnow@redhat.com> |
---|---|---|---|
2 | when the guest attempts to access a physical address with no | ||
3 | mapped device. In commit 4672cbd7bed88dc6 we suppress this for | ||
4 | most legacy boards to prevent breakage of previously working | ||
5 | guests, but we didn't suppress it in the 'virt' board, with | ||
6 | the rationale "we know that guests won't try to prod devices | ||
7 | that we don't describe in the device tree or ACPI tables". This | ||
8 | is mostly true, but we've had a report of a Linux guest image | ||
9 | that this did break. The problem seems to be that the guest | ||
10 | is (incorrectly) configured with a DEBUG_UART_PHYS value that | ||
11 | tells it there is a uart at 0x10009000 (which is true for | ||
12 | vexpress but not for virt), so in early bootup the kernel | ||
13 | probes this bogus address. | ||
14 | 2 | ||
15 | This is a misconfigured guest, so we don't need to worry | 3 | These sections need to be wrapped in a block-level element, such as |
16 | about it too much, but we can arrange that guests that ran | 4 | Paragraph in order for them to be rendered into Texinfo correctly. |
17 | on QEMU v2.10 (before c79c0a314c43b78) will still run on | ||
18 | the "virt-2.10" board model, by suppressing external aborts | ||
19 | only for that version and earlier. This seems a reasonable | ||
20 | compromise: "virt-2.10" is supposed to behave the same way | ||
21 | that "virt" did in the 2.10 release, and making it do that | ||
22 | provides a usable workaround for guests with bugs like this. | ||
23 | 5 | ||
24 | Cc: qemu-stable@nongnu.org | 6 | Before (e.g.): |
7 | |||
8 | <section ids="qapidoc-713"> | ||
9 | <title>If</title> | ||
10 | <literal>defined(CONFIG_REPLICATION)</literal> | ||
11 | </section> | ||
12 | |||
13 | became: | ||
14 | |||
15 | .SS If | ||
16 | \fBdefined(CONFIG_REPLICATION)\fP.SS \fBBlockdevOptionsReplication\fP (Object) | ||
17 | ... | ||
18 | |||
19 | After: | ||
20 | |||
21 | <section ids="qapidoc-713"> | ||
22 | <title>If</title> | ||
23 | <paragraph> | ||
24 | <literal>defined(CONFIG_REPLICATION)</literal> | ||
25 | </paragraph> | ||
26 | </section> | ||
27 | |||
28 | becomes: | ||
29 | |||
30 | .SS If | ||
31 | .sp | ||
32 | \fBdefined(CONFIG_REPLICATION)\fP | ||
33 | .SS \fBBlockdevOptionsReplication\fP (Object) | ||
34 | ... | ||
35 | |||
36 | Reported-by: Markus Armbruster <armbru@redhat.com> | ||
37 | Tested-by: Markus Armbruster <armbru@redhat.com> | ||
38 | Signed-off-by: John Snow <jsnow@redhat.com> | ||
39 | Reviewed-by: Peter Maydell <peter.maydell@linaro.org> | ||
40 | Message-id: 20210406141909.1992225-2-jsnow@redhat.com | ||
25 | Signed-off-by: Peter Maydell <peter.maydell@linaro.org> | 41 | Signed-off-by: Peter Maydell <peter.maydell@linaro.org> |
26 | Message-id: 20180925144127.31965-1-peter.maydell@linaro.org | ||
27 | Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> | ||
28 | --- | 42 | --- |
29 | hw/arm/virt.c | 2 ++ | 43 | docs/sphinx/qapidoc.py | 4 +++- |
30 | 1 file changed, 2 insertions(+) | 44 | 1 file changed, 3 insertions(+), 1 deletion(-) |
31 | 45 | ||
32 | diff --git a/hw/arm/virt.c b/hw/arm/virt.c | 46 | diff --git a/docs/sphinx/qapidoc.py b/docs/sphinx/qapidoc.py |
33 | index XXXXXXX..XXXXXXX 100644 | 47 | index XXXXXXX..XXXXXXX 100644 |
34 | --- a/hw/arm/virt.c | 48 | --- a/docs/sphinx/qapidoc.py |
35 | +++ b/hw/arm/virt.c | 49 | +++ b/docs/sphinx/qapidoc.py |
36 | @@ -XXX,XX +XXX,XX @@ static void virt_machine_2_10_options(MachineClass *mc) | 50 | @@ -XXX,XX +XXX,XX @@ def _nodes_for_if_section(self, ifcond): |
37 | { | 51 | nodelist = [] |
38 | virt_machine_2_11_options(mc); | 52 | if ifcond: |
39 | SET_MACHINE_COMPAT(mc, VIRT_COMPAT_2_10); | 53 | snode = self._make_section('If') |
40 | + /* before 2.11 we never faulted accesses to bad addresses */ | 54 | - snode += self._nodes_for_ifcond(ifcond, with_if=False) |
41 | + mc->ignore_memory_transaction_failures = true; | 55 | + snode += nodes.paragraph( |
42 | } | 56 | + '', '', *self._nodes_for_ifcond(ifcond, with_if=False) |
43 | DEFINE_VIRT_MACHINE(2, 10) | 57 | + ) |
58 | nodelist.append(snode) | ||
59 | return nodelist | ||
44 | 60 | ||
45 | -- | 61 | -- |
46 | 2.19.0 | 62 | 2.20.1 |
47 | 63 | ||
48 | 64 | diff view generated by jsdifflib |
Deleted patch | |||
---|---|---|---|
1 | A cut-and-paste error meant we were reading r4 from the v8M | ||
2 | callee-saves exception stack frame twice. This is harmless | ||
3 | since it just meant we did two memory accesses to the same | ||
4 | location, but it's unnecessary. Delete it. | ||
5 | 1 | ||
6 | Signed-off-by: Peter Maydell <peter.maydell@linaro.org> | ||
7 | Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> | ||
8 | Reviewed-by: Richard Henderson <richard.henderson@linaro.org> | ||
9 | Message-id: 20181002150304.2287-1-peter.maydell@linaro.org | ||
10 | --- | ||
11 | target/arm/helper.c | 1 - | ||
12 | 1 file changed, 1 deletion(-) | ||
13 | |||
14 | diff --git a/target/arm/helper.c b/target/arm/helper.c | ||
15 | index XXXXXXX..XXXXXXX 100644 | ||
16 | --- a/target/arm/helper.c | ||
17 | +++ b/target/arm/helper.c | ||
18 | @@ -XXX,XX +XXX,XX @@ static void do_v7m_exception_exit(ARMCPU *cpu) | ||
19 | } | ||
20 | |||
21 | pop_ok = pop_ok && | ||
22 | - v7m_stack_read(cpu, &env->regs[4], frameptr + 0x8, mmu_idx) && | ||
23 | v7m_stack_read(cpu, &env->regs[4], frameptr + 0x8, mmu_idx) && | ||
24 | v7m_stack_read(cpu, &env->regs[5], frameptr + 0xc, mmu_idx) && | ||
25 | v7m_stack_read(cpu, &env->regs[6], frameptr + 0x10, mmu_idx) && | ||
26 | -- | ||
27 | 2.19.0 | ||
28 | |||
29 | diff view generated by jsdifflib |
Deleted patch | |||
---|---|---|---|
1 | From: Richard Henderson <richard.henderson@linaro.org> | ||
2 | 1 | ||
3 | Given that the only field defined for this new register may only | ||
4 | be 0, we don't actually need to change anything except the name. | ||
5 | |||
6 | Reviewed-by: Peter Maydell <peter.maydell@linaro.org> | ||
7 | Tested-by: Laurent Desnogues <laurent.desnogues@gmail.com> | ||
8 | Signed-off-by: Richard Henderson <richard.henderson@linaro.org> | ||
9 | Message-id: 20181005175350.30752-2-richard.henderson@linaro.org | ||
10 | Signed-off-by: Peter Maydell <peter.maydell@linaro.org> | ||
11 | --- | ||
12 | target/arm/helper.c | 3 ++- | ||
13 | 1 file changed, 2 insertions(+), 1 deletion(-) | ||
14 | |||
15 | diff --git a/target/arm/helper.c b/target/arm/helper.c | ||
16 | index XXXXXXX..XXXXXXX 100644 | ||
17 | --- a/target/arm/helper.c | ||
18 | +++ b/target/arm/helper.c | ||
19 | @@ -XXX,XX +XXX,XX @@ void register_cp_regs_for_features(ARMCPU *cpu) | ||
20 | .opc0 = 3, .opc1 = 0, .crn = 0, .crm = 4, .opc2 = 3, | ||
21 | .access = PL1_R, .type = ARM_CP_CONST, | ||
22 | .resetvalue = 0 }, | ||
23 | - { .name = "ID_AA64PFR4_EL1_RESERVED", .state = ARM_CP_STATE_AA64, | ||
24 | + { .name = "ID_AA64ZFR0_EL1", .state = ARM_CP_STATE_AA64, | ||
25 | .opc0 = 3, .opc1 = 0, .crn = 0, .crm = 4, .opc2 = 4, | ||
26 | .access = PL1_R, .type = ARM_CP_CONST, | ||
27 | + /* At present, only SVEver == 0 is defined anyway. */ | ||
28 | .resetvalue = 0 }, | ||
29 | { .name = "ID_AA64PFR5_EL1_RESERVED", .state = ARM_CP_STATE_AA64, | ||
30 | .opc0 = 3, .opc1 = 0, .crn = 0, .crm = 4, .opc2 = 5, | ||
31 | -- | ||
32 | 2.19.0 | ||
33 | |||
34 | diff view generated by jsdifflib |
Deleted patch | |||
---|---|---|---|
1 | From: Richard Henderson <richard.henderson@linaro.org> | ||
2 | 1 | ||
3 | Check for EL3 before testing CPTR_EL3.EZ. Return 0 when the exception | ||
4 | should be routed via AdvSIMDFPAccessTrap. Mirror the structure of | ||
5 | CheckSVEEnabled more closely. | ||
6 | |||
7 | Fixes: 5be5e8eda78 | ||
8 | Reviewed-by: Peter Maydell <peter.maydell@linaro.org> | ||
9 | Tested-by: Laurent Desnogues <laurent.desnogues@gmail.com> | ||
10 | Signed-off-by: Richard Henderson <richard.henderson@linaro.org> | ||
11 | Message-id: 20181005175350.30752-3-richard.henderson@linaro.org | ||
12 | Signed-off-by: Peter Maydell <peter.maydell@linaro.org> | ||
13 | --- | ||
14 | target/arm/helper.c | 96 ++++++++++++++++++++++----------------------- | ||
15 | 1 file changed, 46 insertions(+), 50 deletions(-) | ||
16 | |||
17 | diff --git a/target/arm/helper.c b/target/arm/helper.c | ||
18 | index XXXXXXX..XXXXXXX 100644 | ||
19 | --- a/target/arm/helper.c | ||
20 | +++ b/target/arm/helper.c | ||
21 | @@ -XXX,XX +XXX,XX @@ static const ARMCPRegInfo debug_lpae_cp_reginfo[] = { | ||
22 | REGINFO_SENTINEL | ||
23 | }; | ||
24 | |||
25 | -/* Return the exception level to which SVE-disabled exceptions should | ||
26 | - * be taken, or 0 if SVE is enabled. | ||
27 | +/* Return the exception level to which exceptions should be taken | ||
28 | + * via SVEAccessTrap. If an exception should be routed through | ||
29 | + * AArch64.AdvSIMDFPAccessTrap, return 0; fp_exception_el should | ||
30 | + * take care of raising that exception. | ||
31 | + * C.f. the ARM pseudocode function CheckSVEEnabled. | ||
32 | */ | ||
33 | static int sve_exception_el(CPUARMState *env) | ||
34 | { | ||
35 | #ifndef CONFIG_USER_ONLY | ||
36 | unsigned current_el = arm_current_el(env); | ||
37 | |||
38 | - /* The CPACR.ZEN controls traps to EL1: | ||
39 | - * 0, 2 : trap EL0 and EL1 accesses | ||
40 | - * 1 : trap only EL0 accesses | ||
41 | - * 3 : trap no accesses | ||
42 | + if (current_el <= 1) { | ||
43 | + bool disabled = false; | ||
44 | + | ||
45 | + /* The CPACR.ZEN controls traps to EL1: | ||
46 | + * 0, 2 : trap EL0 and EL1 accesses | ||
47 | + * 1 : trap only EL0 accesses | ||
48 | + * 3 : trap no accesses | ||
49 | + */ | ||
50 | + if (!extract32(env->cp15.cpacr_el1, 16, 1)) { | ||
51 | + disabled = true; | ||
52 | + } else if (!extract32(env->cp15.cpacr_el1, 17, 1)) { | ||
53 | + disabled = current_el == 0; | ||
54 | + } | ||
55 | + if (disabled) { | ||
56 | + /* route_to_el2 */ | ||
57 | + return (arm_feature(env, ARM_FEATURE_EL2) | ||
58 | + && !arm_is_secure(env) | ||
59 | + && (env->cp15.hcr_el2 & HCR_TGE) ? 2 : 1); | ||
60 | + } | ||
61 | + | ||
62 | + /* Check CPACR.FPEN. */ | ||
63 | + if (!extract32(env->cp15.cpacr_el1, 20, 1)) { | ||
64 | + disabled = true; | ||
65 | + } else if (!extract32(env->cp15.cpacr_el1, 21, 1)) { | ||
66 | + disabled = current_el == 0; | ||
67 | + } | ||
68 | + if (disabled) { | ||
69 | + return 0; | ||
70 | + } | ||
71 | + } | ||
72 | + | ||
73 | + /* CPTR_EL2. Since TZ and TFP are positive, | ||
74 | + * they will be zero when EL2 is not present. | ||
75 | */ | ||
76 | - switch (extract32(env->cp15.cpacr_el1, 16, 2)) { | ||
77 | - default: | ||
78 | - if (current_el <= 1) { | ||
79 | - /* Trap to PL1, which might be EL1 or EL3 */ | ||
80 | - if (arm_is_secure(env) && !arm_el_is_aa64(env, 3)) { | ||
81 | - return 3; | ||
82 | - } | ||
83 | - return 1; | ||
84 | + if (current_el <= 2 && !arm_is_secure_below_el3(env)) { | ||
85 | + if (env->cp15.cptr_el[2] & CPTR_TZ) { | ||
86 | + return 2; | ||
87 | } | ||
88 | - break; | ||
89 | - case 1: | ||
90 | - if (current_el == 0) { | ||
91 | - return 1; | ||
92 | + if (env->cp15.cptr_el[2] & CPTR_TFP) { | ||
93 | + return 0; | ||
94 | } | ||
95 | - break; | ||
96 | - case 3: | ||
97 | - break; | ||
98 | } | ||
99 | |||
100 | - /* Similarly for CPACR.FPEN, after having checked ZEN. */ | ||
101 | - switch (extract32(env->cp15.cpacr_el1, 20, 2)) { | ||
102 | - default: | ||
103 | - if (current_el <= 1) { | ||
104 | - if (arm_is_secure(env) && !arm_el_is_aa64(env, 3)) { | ||
105 | - return 3; | ||
106 | - } | ||
107 | - return 1; | ||
108 | - } | ||
109 | - break; | ||
110 | - case 1: | ||
111 | - if (current_el == 0) { | ||
112 | - return 1; | ||
113 | - } | ||
114 | - break; | ||
115 | - case 3: | ||
116 | - break; | ||
117 | - } | ||
118 | - | ||
119 | - /* CPTR_EL2. Check both TZ and TFP. */ | ||
120 | - if (current_el <= 2 | ||
121 | - && (env->cp15.cptr_el[2] & (CPTR_TFP | CPTR_TZ)) | ||
122 | - && !arm_is_secure_below_el3(env)) { | ||
123 | - return 2; | ||
124 | - } | ||
125 | - | ||
126 | - /* CPTR_EL3. Check both EZ and TFP. */ | ||
127 | - if (!(env->cp15.cptr_el[3] & CPTR_EZ) | ||
128 | - || (env->cp15.cptr_el[3] & CPTR_TFP)) { | ||
129 | + /* CPTR_EL3. Since EZ is negative we must check for EL3. */ | ||
130 | + if (arm_feature(env, ARM_FEATURE_EL3) | ||
131 | + && !(env->cp15.cptr_el[3] & CPTR_EZ)) { | ||
132 | return 3; | ||
133 | } | ||
134 | #endif | ||
135 | -- | ||
136 | 2.19.0 | ||
137 | |||
138 | diff view generated by jsdifflib |
Deleted patch | |||
---|---|---|---|
1 | From: Richard Henderson <richard.henderson@linaro.org> | ||
2 | 1 | ||
3 | We are going to want to determine whether sve is enabled | ||
4 | for EL other than current. | ||
5 | |||
6 | Tested-by: Laurent Desnogues <laurent.desnogues@gmail.com> | ||
7 | Reviewed-by: Peter Maydell <peter.maydell@linaro.org> | ||
8 | Signed-off-by: Richard Henderson <richard.henderson@linaro.org> | ||
9 | Message-id: 20181005175350.30752-4-richard.henderson@linaro.org | ||
10 | Signed-off-by: Peter Maydell <peter.maydell@linaro.org> | ||
11 | --- | ||
12 | target/arm/helper.c | 21 +++++++++------------ | ||
13 | 1 file changed, 9 insertions(+), 12 deletions(-) | ||
14 | |||
15 | diff --git a/target/arm/helper.c b/target/arm/helper.c | ||
16 | index XXXXXXX..XXXXXXX 100644 | ||
17 | --- a/target/arm/helper.c | ||
18 | +++ b/target/arm/helper.c | ||
19 | @@ -XXX,XX +XXX,XX @@ static const ARMCPRegInfo debug_lpae_cp_reginfo[] = { | ||
20 | * take care of raising that exception. | ||
21 | * C.f. the ARM pseudocode function CheckSVEEnabled. | ||
22 | */ | ||
23 | -static int sve_exception_el(CPUARMState *env) | ||
24 | +static int sve_exception_el(CPUARMState *env, int el) | ||
25 | { | ||
26 | #ifndef CONFIG_USER_ONLY | ||
27 | - unsigned current_el = arm_current_el(env); | ||
28 | - | ||
29 | - if (current_el <= 1) { | ||
30 | + if (el <= 1) { | ||
31 | bool disabled = false; | ||
32 | |||
33 | /* The CPACR.ZEN controls traps to EL1: | ||
34 | @@ -XXX,XX +XXX,XX @@ static int sve_exception_el(CPUARMState *env) | ||
35 | if (!extract32(env->cp15.cpacr_el1, 16, 1)) { | ||
36 | disabled = true; | ||
37 | } else if (!extract32(env->cp15.cpacr_el1, 17, 1)) { | ||
38 | - disabled = current_el == 0; | ||
39 | + disabled = el == 0; | ||
40 | } | ||
41 | if (disabled) { | ||
42 | /* route_to_el2 */ | ||
43 | @@ -XXX,XX +XXX,XX @@ static int sve_exception_el(CPUARMState *env) | ||
44 | if (!extract32(env->cp15.cpacr_el1, 20, 1)) { | ||
45 | disabled = true; | ||
46 | } else if (!extract32(env->cp15.cpacr_el1, 21, 1)) { | ||
47 | - disabled = current_el == 0; | ||
48 | + disabled = el == 0; | ||
49 | } | ||
50 | if (disabled) { | ||
51 | return 0; | ||
52 | @@ -XXX,XX +XXX,XX @@ static int sve_exception_el(CPUARMState *env) | ||
53 | /* CPTR_EL2. Since TZ and TFP are positive, | ||
54 | * they will be zero when EL2 is not present. | ||
55 | */ | ||
56 | - if (current_el <= 2 && !arm_is_secure_below_el3(env)) { | ||
57 | + if (el <= 2 && !arm_is_secure_below_el3(env)) { | ||
58 | if (env->cp15.cptr_el[2] & CPTR_TZ) { | ||
59 | return 2; | ||
60 | } | ||
61 | @@ -XXX,XX +XXX,XX @@ uint32_t HELPER(crc32c)(uint32_t acc, uint32_t val, uint32_t bytes) | ||
62 | /* Return the exception level to which FP-disabled exceptions should | ||
63 | * be taken, or 0 if FP is enabled. | ||
64 | */ | ||
65 | -static inline int fp_exception_el(CPUARMState *env) | ||
66 | +static int fp_exception_el(CPUARMState *env, int cur_el) | ||
67 | { | ||
68 | #ifndef CONFIG_USER_ONLY | ||
69 | int fpen; | ||
70 | - int cur_el = arm_current_el(env); | ||
71 | |||
72 | /* CPACR and the CPTR registers don't exist before v6, so FP is | ||
73 | * always accessible | ||
74 | @@ -XXX,XX +XXX,XX @@ void cpu_get_tb_cpu_state(CPUARMState *env, target_ulong *pc, | ||
75 | target_ulong *cs_base, uint32_t *pflags) | ||
76 | { | ||
77 | ARMMMUIdx mmu_idx = core_to_arm_mmu_idx(env, cpu_mmu_index(env, false)); | ||
78 | - int fp_el = fp_exception_el(env); | ||
79 | + int current_el = arm_current_el(env); | ||
80 | + int fp_el = fp_exception_el(env, current_el); | ||
81 | uint32_t flags; | ||
82 | |||
83 | if (is_a64(env)) { | ||
84 | @@ -XXX,XX +XXX,XX @@ void cpu_get_tb_cpu_state(CPUARMState *env, target_ulong *pc, | ||
85 | flags |= (arm_regime_tbi1(env, mmu_idx) << ARM_TBFLAG_TBI1_SHIFT); | ||
86 | |||
87 | if (arm_feature(env, ARM_FEATURE_SVE)) { | ||
88 | - int sve_el = sve_exception_el(env); | ||
89 | + int sve_el = sve_exception_el(env, current_el); | ||
90 | uint32_t zcr_len; | ||
91 | |||
92 | /* If SVE is disabled, but FP is enabled, | ||
93 | @@ -XXX,XX +XXX,XX @@ void cpu_get_tb_cpu_state(CPUARMState *env, target_ulong *pc, | ||
94 | if (sve_el != 0 && fp_el == 0) { | ||
95 | zcr_len = 0; | ||
96 | } else { | ||
97 | - int current_el = arm_current_el(env); | ||
98 | ARMCPU *cpu = arm_env_get_cpu(env); | ||
99 | |||
100 | zcr_len = cpu->sve_max_vq - 1; | ||
101 | -- | ||
102 | 2.19.0 | ||
103 | |||
104 | diff view generated by jsdifflib |
Deleted patch | |||
---|---|---|---|
1 | From: Richard Henderson <richard.henderson@linaro.org> | ||
2 | 1 | ||
3 | SVE vector length can change when changing EL, or when writing | ||
4 | to one of the ZCR_ELn registers. | ||
5 | |||
6 | For correctness, our implementation requires that predicate bits | ||
7 | that are inaccessible are never set. Which means noticing length | ||
8 | changes and zeroing the appropriate register bits. | ||
9 | |||
10 | Tested-by: Laurent Desnogues <laurent.desnogues@gmail.com> | ||
11 | Signed-off-by: Richard Henderson <richard.henderson@linaro.org> | ||
12 | Message-id: 20181005175350.30752-5-richard.henderson@linaro.org | ||
13 | Reviewed-by: Peter Maydell <peter.maydell@linaro.org> | ||
14 | Signed-off-by: Peter Maydell <peter.maydell@linaro.org> | ||
15 | --- | ||
16 | target/arm/cpu.h | 4 ++ | ||
17 | target/arm/cpu64.c | 42 ------------- | ||
18 | target/arm/helper.c | 133 +++++++++++++++++++++++++++++++++++++---- | ||
19 | target/arm/op_helper.c | 1 + | ||
20 | 4 files changed, 125 insertions(+), 55 deletions(-) | ||
21 | |||
22 | diff --git a/target/arm/cpu.h b/target/arm/cpu.h | ||
23 | index XXXXXXX..XXXXXXX 100644 | ||
24 | --- a/target/arm/cpu.h | ||
25 | +++ b/target/arm/cpu.h | ||
26 | @@ -XXX,XX +XXX,XX @@ int arm_cpu_write_elf32_note(WriteCoreDumpFunction f, CPUState *cs, | ||
27 | int aarch64_cpu_gdb_read_register(CPUState *cpu, uint8_t *buf, int reg); | ||
28 | int aarch64_cpu_gdb_write_register(CPUState *cpu, uint8_t *buf, int reg); | ||
29 | void aarch64_sve_narrow_vq(CPUARMState *env, unsigned vq); | ||
30 | +void aarch64_sve_change_el(CPUARMState *env, int old_el, int new_el); | ||
31 | +#else | ||
32 | +static inline void aarch64_sve_narrow_vq(CPUARMState *env, unsigned vq) { } | ||
33 | +static inline void aarch64_sve_change_el(CPUARMState *env, int o, int n) { } | ||
34 | #endif | ||
35 | |||
36 | target_ulong do_arm_semihosting(CPUARMState *env); | ||
37 | diff --git a/target/arm/cpu64.c b/target/arm/cpu64.c | ||
38 | index XXXXXXX..XXXXXXX 100644 | ||
39 | --- a/target/arm/cpu64.c | ||
40 | +++ b/target/arm/cpu64.c | ||
41 | @@ -XXX,XX +XXX,XX @@ static void aarch64_cpu_register_types(void) | ||
42 | } | ||
43 | |||
44 | type_init(aarch64_cpu_register_types) | ||
45 | - | ||
46 | -/* The manual says that when SVE is enabled and VQ is widened the | ||
47 | - * implementation is allowed to zero the previously inaccessible | ||
48 | - * portion of the registers. The corollary to that is that when | ||
49 | - * SVE is enabled and VQ is narrowed we are also allowed to zero | ||
50 | - * the now inaccessible portion of the registers. | ||
51 | - * | ||
52 | - * The intent of this is that no predicate bit beyond VQ is ever set. | ||
53 | - * Which means that some operations on predicate registers themselves | ||
54 | - * may operate on full uint64_t or even unrolled across the maximum | ||
55 | - * uint64_t[4]. Performing 4 bits of host arithmetic unconditionally | ||
56 | - * may well be cheaper than conditionals to restrict the operation | ||
57 | - * to the relevant portion of a uint16_t[16]. | ||
58 | - * | ||
59 | - * TODO: Need to call this for changes to the real system registers | ||
60 | - * and EL state changes. | ||
61 | - */ | ||
62 | -void aarch64_sve_narrow_vq(CPUARMState *env, unsigned vq) | ||
63 | -{ | ||
64 | - int i, j; | ||
65 | - uint64_t pmask; | ||
66 | - | ||
67 | - assert(vq >= 1 && vq <= ARM_MAX_VQ); | ||
68 | - assert(vq <= arm_env_get_cpu(env)->sve_max_vq); | ||
69 | - | ||
70 | - /* Zap the high bits of the zregs. */ | ||
71 | - for (i = 0; i < 32; i++) { | ||
72 | - memset(&env->vfp.zregs[i].d[2 * vq], 0, 16 * (ARM_MAX_VQ - vq)); | ||
73 | - } | ||
74 | - | ||
75 | - /* Zap the high bits of the pregs and ffr. */ | ||
76 | - pmask = 0; | ||
77 | - if (vq & 3) { | ||
78 | - pmask = ~(-1ULL << (16 * (vq & 3))); | ||
79 | - } | ||
80 | - for (j = vq / 4; j < ARM_MAX_VQ / 4; j++) { | ||
81 | - for (i = 0; i < 17; ++i) { | ||
82 | - env->vfp.pregs[i].p[j] &= pmask; | ||
83 | - } | ||
84 | - pmask = 0; | ||
85 | - } | ||
86 | -} | ||
87 | diff --git a/target/arm/helper.c b/target/arm/helper.c | ||
88 | index XXXXXXX..XXXXXXX 100644 | ||
89 | --- a/target/arm/helper.c | ||
90 | +++ b/target/arm/helper.c | ||
91 | @@ -XXX,XX +XXX,XX @@ static int sve_exception_el(CPUARMState *env, int el) | ||
92 | return 0; | ||
93 | } | ||
94 | |||
95 | +/* | ||
96 | + * Given that SVE is enabled, return the vector length for EL. | ||
97 | + */ | ||
98 | +static uint32_t sve_zcr_len_for_el(CPUARMState *env, int el) | ||
99 | +{ | ||
100 | + ARMCPU *cpu = arm_env_get_cpu(env); | ||
101 | + uint32_t zcr_len = cpu->sve_max_vq - 1; | ||
102 | + | ||
103 | + if (el <= 1) { | ||
104 | + zcr_len = MIN(zcr_len, 0xf & (uint32_t)env->vfp.zcr_el[1]); | ||
105 | + } | ||
106 | + if (el < 2 && arm_feature(env, ARM_FEATURE_EL2)) { | ||
107 | + zcr_len = MIN(zcr_len, 0xf & (uint32_t)env->vfp.zcr_el[2]); | ||
108 | + } | ||
109 | + if (el < 3 && arm_feature(env, ARM_FEATURE_EL3)) { | ||
110 | + zcr_len = MIN(zcr_len, 0xf & (uint32_t)env->vfp.zcr_el[3]); | ||
111 | + } | ||
112 | + return zcr_len; | ||
113 | +} | ||
114 | + | ||
115 | static void zcr_write(CPUARMState *env, const ARMCPRegInfo *ri, | ||
116 | uint64_t value) | ||
117 | { | ||
118 | + int cur_el = arm_current_el(env); | ||
119 | + int old_len = sve_zcr_len_for_el(env, cur_el); | ||
120 | + int new_len; | ||
121 | + | ||
122 | /* Bits other than [3:0] are RAZ/WI. */ | ||
123 | raw_write(env, ri, value & 0xf); | ||
124 | + | ||
125 | + /* | ||
126 | + * Because we arrived here, we know both FP and SVE are enabled; | ||
127 | + * otherwise we would have trapped access to the ZCR_ELn register. | ||
128 | + */ | ||
129 | + new_len = sve_zcr_len_for_el(env, cur_el); | ||
130 | + if (new_len < old_len) { | ||
131 | + aarch64_sve_narrow_vq(env, new_len + 1); | ||
132 | + } | ||
133 | } | ||
134 | |||
135 | static const ARMCPRegInfo zcr_el1_reginfo = { | ||
136 | @@ -XXX,XX +XXX,XX @@ static void arm_cpu_do_interrupt_aarch64(CPUState *cs) | ||
137 | unsigned int new_el = env->exception.target_el; | ||
138 | target_ulong addr = env->cp15.vbar_el[new_el]; | ||
139 | unsigned int new_mode = aarch64_pstate_mode(new_el, true); | ||
140 | + unsigned int cur_el = arm_current_el(env); | ||
141 | |||
142 | - if (arm_current_el(env) < new_el) { | ||
143 | + aarch64_sve_change_el(env, cur_el, new_el); | ||
144 | + | ||
145 | + if (cur_el < new_el) { | ||
146 | /* Entry vector offset depends on whether the implemented EL | ||
147 | * immediately lower than the target level is using AArch32 or AArch64 | ||
148 | */ | ||
149 | @@ -XXX,XX +XXX,XX @@ void cpu_get_tb_cpu_state(CPUARMState *env, target_ulong *pc, | ||
150 | if (sve_el != 0 && fp_el == 0) { | ||
151 | zcr_len = 0; | ||
152 | } else { | ||
153 | - ARMCPU *cpu = arm_env_get_cpu(env); | ||
154 | - | ||
155 | - zcr_len = cpu->sve_max_vq - 1; | ||
156 | - if (current_el <= 1) { | ||
157 | - zcr_len = MIN(zcr_len, 0xf & (uint32_t)env->vfp.zcr_el[1]); | ||
158 | - } | ||
159 | - if (current_el < 2 && arm_feature(env, ARM_FEATURE_EL2)) { | ||
160 | - zcr_len = MIN(zcr_len, 0xf & (uint32_t)env->vfp.zcr_el[2]); | ||
161 | - } | ||
162 | - if (current_el < 3 && arm_feature(env, ARM_FEATURE_EL3)) { | ||
163 | - zcr_len = MIN(zcr_len, 0xf & (uint32_t)env->vfp.zcr_el[3]); | ||
164 | - } | ||
165 | + zcr_len = sve_zcr_len_for_el(env, current_el); | ||
166 | } | ||
167 | flags |= sve_el << ARM_TBFLAG_SVEEXC_EL_SHIFT; | ||
168 | flags |= zcr_len << ARM_TBFLAG_ZCR_LEN_SHIFT; | ||
169 | @@ -XXX,XX +XXX,XX @@ void cpu_get_tb_cpu_state(CPUARMState *env, target_ulong *pc, | ||
170 | *pflags = flags; | ||
171 | *cs_base = 0; | ||
172 | } | ||
173 | + | ||
174 | +#ifdef TARGET_AARCH64 | ||
175 | +/* | ||
176 | + * The manual says that when SVE is enabled and VQ is widened the | ||
177 | + * implementation is allowed to zero the previously inaccessible | ||
178 | + * portion of the registers. The corollary to that is that when | ||
179 | + * SVE is enabled and VQ is narrowed we are also allowed to zero | ||
180 | + * the now inaccessible portion of the registers. | ||
181 | + * | ||
182 | + * The intent of this is that no predicate bit beyond VQ is ever set. | ||
183 | + * Which means that some operations on predicate registers themselves | ||
184 | + * may operate on full uint64_t or even unrolled across the maximum | ||
185 | + * uint64_t[4]. Performing 4 bits of host arithmetic unconditionally | ||
186 | + * may well be cheaper than conditionals to restrict the operation | ||
187 | + * to the relevant portion of a uint16_t[16]. | ||
188 | + */ | ||
189 | +void aarch64_sve_narrow_vq(CPUARMState *env, unsigned vq) | ||
190 | +{ | ||
191 | + int i, j; | ||
192 | + uint64_t pmask; | ||
193 | + | ||
194 | + assert(vq >= 1 && vq <= ARM_MAX_VQ); | ||
195 | + assert(vq <= arm_env_get_cpu(env)->sve_max_vq); | ||
196 | + | ||
197 | + /* Zap the high bits of the zregs. */ | ||
198 | + for (i = 0; i < 32; i++) { | ||
199 | + memset(&env->vfp.zregs[i].d[2 * vq], 0, 16 * (ARM_MAX_VQ - vq)); | ||
200 | + } | ||
201 | + | ||
202 | + /* Zap the high bits of the pregs and ffr. */ | ||
203 | + pmask = 0; | ||
204 | + if (vq & 3) { | ||
205 | + pmask = ~(-1ULL << (16 * (vq & 3))); | ||
206 | + } | ||
207 | + for (j = vq / 4; j < ARM_MAX_VQ / 4; j++) { | ||
208 | + for (i = 0; i < 17; ++i) { | ||
209 | + env->vfp.pregs[i].p[j] &= pmask; | ||
210 | + } | ||
211 | + pmask = 0; | ||
212 | + } | ||
213 | +} | ||
214 | + | ||
215 | +/* | ||
216 | + * Notice a change in SVE vector size when changing EL. | ||
217 | + */ | ||
218 | +void aarch64_sve_change_el(CPUARMState *env, int old_el, int new_el) | ||
219 | +{ | ||
220 | + int old_len, new_len; | ||
221 | + | ||
222 | + /* Nothing to do if no SVE. */ | ||
223 | + if (!arm_feature(env, ARM_FEATURE_SVE)) { | ||
224 | + return; | ||
225 | + } | ||
226 | + | ||
227 | + /* Nothing to do if FP is disabled in either EL. */ | ||
228 | + if (fp_exception_el(env, old_el) || fp_exception_el(env, new_el)) { | ||
229 | + return; | ||
230 | + } | ||
231 | + | ||
232 | + /* | ||
233 | + * DDI0584A.d sec 3.2: "If SVE instructions are disabled or trapped | ||
234 | + * at ELx, or not available because the EL is in AArch32 state, then | ||
235 | + * for all purposes other than a direct read, the ZCR_ELx.LEN field | ||
236 | + * has an effective value of 0". | ||
237 | + * | ||
238 | + * Consider EL2 (aa64, vq=4) -> EL0 (aa32) -> EL1 (aa64, vq=0). | ||
239 | + * If we ignore aa32 state, we would fail to see the vq4->vq0 transition | ||
240 | + * from EL2->EL1. Thus we go ahead and narrow when entering aa32 so that | ||
241 | + * we already have the correct register contents when encountering the | ||
242 | + * vq0->vq0 transition between EL0->EL1. | ||
243 | + */ | ||
244 | + old_len = (arm_el_is_aa64(env, old_el) && !sve_exception_el(env, old_el) | ||
245 | + ? sve_zcr_len_for_el(env, old_el) : 0); | ||
246 | + new_len = (arm_el_is_aa64(env, new_el) && !sve_exception_el(env, new_el) | ||
247 | + ? sve_zcr_len_for_el(env, new_el) : 0); | ||
248 | + | ||
249 | + /* When changing vector length, clear inaccessible state. */ | ||
250 | + if (new_len < old_len) { | ||
251 | + aarch64_sve_narrow_vq(env, new_len + 1); | ||
252 | + } | ||
253 | +} | ||
254 | +#endif | ||
255 | diff --git a/target/arm/op_helper.c b/target/arm/op_helper.c | ||
256 | index XXXXXXX..XXXXXXX 100644 | ||
257 | --- a/target/arm/op_helper.c | ||
258 | +++ b/target/arm/op_helper.c | ||
259 | @@ -XXX,XX +XXX,XX @@ void HELPER(exception_return)(CPUARMState *env) | ||
260 | "AArch64 EL%d PC 0x%" PRIx64 "\n", | ||
261 | cur_el, new_el, env->pc); | ||
262 | } | ||
263 | + aarch64_sve_change_el(env, cur_el, new_el); | ||
264 | |||
265 | qemu_mutex_lock_iothread(); | ||
266 | arm_call_el_change_hook(arm_env_get_cpu(env)); | ||
267 | -- | ||
268 | 2.19.0 | ||
269 | |||
270 | diff view generated by jsdifflib |
Deleted patch | |||
---|---|---|---|
1 | From: Richard Henderson <richard.henderson@linaro.org> | ||
2 | 1 | ||
3 | Use the existing helpers to determine if (1) the fpu is enabled, | ||
4 | (2) sve state is enabled, and (3) the current sve vector length. | ||
5 | |||
6 | Tested-by: Laurent Desnogues <laurent.desnogues@gmail.com> | ||
7 | Reviewed-by: Peter Maydell <peter.maydell@linaro.org> | ||
8 | Signed-off-by: Richard Henderson <richard.henderson@linaro.org> | ||
9 | Message-id: 20181005175350.30752-6-richard.henderson@linaro.org | ||
10 | Signed-off-by: Peter Maydell <peter.maydell@linaro.org> | ||
11 | --- | ||
12 | target/arm/cpu.h | 4 ++++ | ||
13 | target/arm/helper.c | 6 +++--- | ||
14 | target/arm/translate-a64.c | 8 ++++++-- | ||
15 | 3 files changed, 13 insertions(+), 5 deletions(-) | ||
16 | |||
17 | diff --git a/target/arm/cpu.h b/target/arm/cpu.h | ||
18 | index XXXXXXX..XXXXXXX 100644 | ||
19 | --- a/target/arm/cpu.h | ||
20 | +++ b/target/arm/cpu.h | ||
21 | @@ -XXX,XX +XXX,XX @@ target_ulong do_arm_semihosting(CPUARMState *env); | ||
22 | void aarch64_sync_32_to_64(CPUARMState *env); | ||
23 | void aarch64_sync_64_to_32(CPUARMState *env); | ||
24 | |||
25 | +int fp_exception_el(CPUARMState *env, int cur_el); | ||
26 | +int sve_exception_el(CPUARMState *env, int cur_el); | ||
27 | +uint32_t sve_zcr_len_for_el(CPUARMState *env, int el); | ||
28 | + | ||
29 | static inline bool is_a64(CPUARMState *env) | ||
30 | { | ||
31 | return env->aarch64; | ||
32 | diff --git a/target/arm/helper.c b/target/arm/helper.c | ||
33 | index XXXXXXX..XXXXXXX 100644 | ||
34 | --- a/target/arm/helper.c | ||
35 | +++ b/target/arm/helper.c | ||
36 | @@ -XXX,XX +XXX,XX @@ static const ARMCPRegInfo debug_lpae_cp_reginfo[] = { | ||
37 | * take care of raising that exception. | ||
38 | * C.f. the ARM pseudocode function CheckSVEEnabled. | ||
39 | */ | ||
40 | -static int sve_exception_el(CPUARMState *env, int el) | ||
41 | +int sve_exception_el(CPUARMState *env, int el) | ||
42 | { | ||
43 | #ifndef CONFIG_USER_ONLY | ||
44 | if (el <= 1) { | ||
45 | @@ -XXX,XX +XXX,XX @@ static int sve_exception_el(CPUARMState *env, int el) | ||
46 | /* | ||
47 | * Given that SVE is enabled, return the vector length for EL. | ||
48 | */ | ||
49 | -static uint32_t sve_zcr_len_for_el(CPUARMState *env, int el) | ||
50 | +uint32_t sve_zcr_len_for_el(CPUARMState *env, int el) | ||
51 | { | ||
52 | ARMCPU *cpu = arm_env_get_cpu(env); | ||
53 | uint32_t zcr_len = cpu->sve_max_vq - 1; | ||
54 | @@ -XXX,XX +XXX,XX @@ uint32_t HELPER(crc32c)(uint32_t acc, uint32_t val, uint32_t bytes) | ||
55 | /* Return the exception level to which FP-disabled exceptions should | ||
56 | * be taken, or 0 if FP is enabled. | ||
57 | */ | ||
58 | -static int fp_exception_el(CPUARMState *env, int cur_el) | ||
59 | +int fp_exception_el(CPUARMState *env, int cur_el) | ||
60 | { | ||
61 | #ifndef CONFIG_USER_ONLY | ||
62 | int fpen; | ||
63 | diff --git a/target/arm/translate-a64.c b/target/arm/translate-a64.c | ||
64 | index XXXXXXX..XXXXXXX 100644 | ||
65 | --- a/target/arm/translate-a64.c | ||
66 | +++ b/target/arm/translate-a64.c | ||
67 | @@ -XXX,XX +XXX,XX @@ void aarch64_cpu_dump_state(CPUState *cs, FILE *f, | ||
68 | cpu_fprintf(f, "\n"); | ||
69 | return; | ||
70 | } | ||
71 | + if (fp_exception_el(env, el) != 0) { | ||
72 | + cpu_fprintf(f, " FPU disabled\n"); | ||
73 | + return; | ||
74 | + } | ||
75 | cpu_fprintf(f, " FPCR=%08x FPSR=%08x\n", | ||
76 | vfp_get_fpcr(env), vfp_get_fpsr(env)); | ||
77 | |||
78 | - if (arm_feature(env, ARM_FEATURE_SVE)) { | ||
79 | - int j, zcr_len = env->vfp.zcr_el[1] & 0xf; /* fix for system mode */ | ||
80 | + if (arm_feature(env, ARM_FEATURE_SVE) && sve_exception_el(env, el) == 0) { | ||
81 | + int j, zcr_len = sve_zcr_len_for_el(env, el); | ||
82 | |||
83 | for (i = 0; i <= FFR_PRED_NUM; i++) { | ||
84 | bool eol; | ||
85 | -- | ||
86 | 2.19.0 | ||
87 | |||
88 | diff view generated by jsdifflib |
Deleted patch | |||
---|---|---|---|
1 | From: Richard Henderson <richard.henderson@linaro.org> | ||
2 | 1 | ||
3 | The 16-byte load only uses 16 predicate bits. But while | ||
4 | reusing the other load infrastructure, we find other bits | ||
5 | that are set and trigger an assert. To avoid this and | ||
6 | retain the assert, zero-extend the predicate that we pass | ||
7 | to the LD1 helper. | ||
8 | |||
9 | Tested-by: Laurent Desnogues <laurent.desnogues@gmail.com> | ||
10 | Reported-by: Laurent Desnogues <laurent.desnogues@gmail.com> | ||
11 | Reviewed-by: Peter Maydell <peter.maydell@linaro.org> | ||
12 | Signed-off-by: Richard Henderson <richard.henderson@linaro.org> | ||
13 | Message-id: 20181005175350.30752-7-richard.henderson@linaro.org | ||
14 | Signed-off-by: Peter Maydell <peter.maydell@linaro.org> | ||
15 | --- | ||
16 | target/arm/translate-sve.c | 25 +++++++++++++++++++++++-- | ||
17 | 1 file changed, 23 insertions(+), 2 deletions(-) | ||
18 | |||
19 | diff --git a/target/arm/translate-sve.c b/target/arm/translate-sve.c | ||
20 | index XXXXXXX..XXXXXXX 100644 | ||
21 | --- a/target/arm/translate-sve.c | ||
22 | +++ b/target/arm/translate-sve.c | ||
23 | @@ -XXX,XX +XXX,XX @@ static void do_ldrq(DisasContext *s, int zt, int pg, TCGv_i64 addr, int msz) | ||
24 | unsigned vsz = vec_full_reg_size(s); | ||
25 | TCGv_ptr t_pg; | ||
26 | TCGv_i32 desc; | ||
27 | + int poff; | ||
28 | |||
29 | /* Load the first quadword using the normal predicated load helpers. */ | ||
30 | desc = tcg_const_i32(simd_desc(16, 16, zt)); | ||
31 | - t_pg = tcg_temp_new_ptr(); | ||
32 | |||
33 | - tcg_gen_addi_ptr(t_pg, cpu_env, pred_full_reg_offset(s, pg)); | ||
34 | + poff = pred_full_reg_offset(s, pg); | ||
35 | + if (vsz > 16) { | ||
36 | + /* | ||
37 | + * Zero-extend the first 16 bits of the predicate into a temporary. | ||
38 | + * This avoids triggering an assert making sure we don't have bits | ||
39 | + * set within a predicate beyond VQ, but we have lowered VQ to 1 | ||
40 | + * for this load operation. | ||
41 | + */ | ||
42 | + TCGv_i64 tmp = tcg_temp_new_i64(); | ||
43 | +#ifdef HOST_WORDS_BIGENDIAN | ||
44 | + poff += 6; | ||
45 | +#endif | ||
46 | + tcg_gen_ld16u_i64(tmp, cpu_env, poff); | ||
47 | + | ||
48 | + poff = offsetof(CPUARMState, vfp.preg_tmp); | ||
49 | + tcg_gen_st_i64(tmp, cpu_env, poff); | ||
50 | + tcg_temp_free_i64(tmp); | ||
51 | + } | ||
52 | + | ||
53 | + t_pg = tcg_temp_new_ptr(); | ||
54 | + tcg_gen_addi_ptr(t_pg, cpu_env, poff); | ||
55 | + | ||
56 | fns[msz](cpu_env, t_pg, addr, desc); | ||
57 | |||
58 | tcg_temp_free_ptr(t_pg); | ||
59 | -- | ||
60 | 2.19.0 | ||
61 | |||
62 | diff view generated by jsdifflib |
Deleted patch | |||
---|---|---|---|
1 | From: Richard Henderson <richard.henderson@linaro.org> | ||
2 | 1 | ||
3 | Uses tlb_vaddr_to_host for correct operation with softmmu. | ||
4 | Optimize for accesses within a single page or pair of pages. | ||
5 | |||
6 | Reviewed-by: Peter Maydell <peter.maydell@linaro.org> | ||
7 | Signed-off-by: Richard Henderson <richard.henderson@linaro.org> | ||
8 | Message-id: 20181005175350.30752-8-richard.henderson@linaro.org | ||
9 | Signed-off-by: Peter Maydell <peter.maydell@linaro.org> | ||
10 | --- | ||
11 | target/arm/sve_helper.c | 731 +++++++++++++++++++++++++++++++--------- | ||
12 | 1 file changed, 569 insertions(+), 162 deletions(-) | ||
13 | |||
14 | diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c | ||
15 | index XXXXXXX..XXXXXXX 100644 | ||
16 | --- a/target/arm/sve_helper.c | ||
17 | +++ b/target/arm/sve_helper.c | ||
18 | @@ -XXX,XX +XXX,XX @@ static void swap_memmove(void *vd, void *vs, size_t n) | ||
19 | } | ||
20 | } | ||
21 | |||
22 | +/* Similarly for memset of 0. */ | ||
23 | +static void swap_memzero(void *vd, size_t n) | ||
24 | +{ | ||
25 | + uintptr_t d = (uintptr_t)vd; | ||
26 | + uintptr_t o = (d | n) & 7; | ||
27 | + size_t i; | ||
28 | + | ||
29 | + /* Usually, the first bit of a predicate is set, so N is 0. */ | ||
30 | + if (likely(n == 0)) { | ||
31 | + return; | ||
32 | + } | ||
33 | + | ||
34 | +#ifndef HOST_WORDS_BIGENDIAN | ||
35 | + o = 0; | ||
36 | +#endif | ||
37 | + switch (o) { | ||
38 | + case 0: | ||
39 | + memset(vd, 0, n); | ||
40 | + break; | ||
41 | + | ||
42 | + case 4: | ||
43 | + for (i = 0; i < n; i += 4) { | ||
44 | + *(uint32_t *)H1_4(d + i) = 0; | ||
45 | + } | ||
46 | + break; | ||
47 | + | ||
48 | + case 2: | ||
49 | + case 6: | ||
50 | + for (i = 0; i < n; i += 2) { | ||
51 | + *(uint16_t *)H1_2(d + i) = 0; | ||
52 | + } | ||
53 | + break; | ||
54 | + | ||
55 | + default: | ||
56 | + for (i = 0; i < n; i++) { | ||
57 | + *(uint8_t *)H1(d + i) = 0; | ||
58 | + } | ||
59 | + break; | ||
60 | + } | ||
61 | +} | ||
62 | + | ||
63 | void HELPER(sve_ext)(void *vd, void *vn, void *vm, uint32_t desc) | ||
64 | { | ||
65 | intptr_t opr_sz = simd_oprsz(desc); | ||
66 | @@ -XXX,XX +XXX,XX @@ void HELPER(sve_fcmla_zpzzz_d)(CPUARMState *env, void *vg, uint32_t desc) | ||
67 | /* | ||
68 | * Load contiguous data, protected by a governing predicate. | ||
69 | */ | ||
70 | -#define DO_LD1(NAME, FN, TYPEE, TYPEM, H) \ | ||
71 | -static void do_##NAME(CPUARMState *env, void *vd, void *vg, \ | ||
72 | - target_ulong addr, intptr_t oprsz, \ | ||
73 | - uintptr_t ra) \ | ||
74 | -{ \ | ||
75 | - intptr_t i = 0; \ | ||
76 | - do { \ | ||
77 | - uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); \ | ||
78 | - do { \ | ||
79 | - TYPEM m = 0; \ | ||
80 | - if (pg & 1) { \ | ||
81 | - m = FN(env, addr, ra); \ | ||
82 | - } \ | ||
83 | - *(TYPEE *)(vd + H(i)) = m; \ | ||
84 | - i += sizeof(TYPEE), pg >>= sizeof(TYPEE); \ | ||
85 | - addr += sizeof(TYPEM); \ | ||
86 | - } while (i & 15); \ | ||
87 | - } while (i < oprsz); \ | ||
88 | -} \ | ||
89 | -void HELPER(NAME)(CPUARMState *env, void *vg, \ | ||
90 | - target_ulong addr, uint32_t desc) \ | ||
91 | -{ \ | ||
92 | - do_##NAME(env, &env->vfp.zregs[simd_data(desc)], vg, \ | ||
93 | - addr, simd_oprsz(desc), GETPC()); \ | ||
94 | + | ||
95 | +/* | ||
96 | + * Load elements into @vd, controlled by @vg, from @host + @mem_ofs. | ||
97 | + * Memory is valid through @host + @mem_max. The register element | ||
98 | + * indicies are inferred from @mem_ofs, as modified by the types for | ||
99 | + * which the helper is built. Return the @mem_ofs of the first element | ||
100 | + * not loaded (which is @mem_max if they are all loaded). | ||
101 | + * | ||
102 | + * For softmmu, we have fully validated the guest page. For user-only, | ||
103 | + * we cannot fully validate without taking the mmap lock, but since we | ||
104 | + * know the access is within one host page, if any access is valid they | ||
105 | + * all must be valid. However, when @vg is all false, it may be that | ||
106 | + * no access is valid. | ||
107 | + */ | ||
108 | +typedef intptr_t sve_ld1_host_fn(void *vd, void *vg, void *host, | ||
109 | + intptr_t mem_ofs, intptr_t mem_max); | ||
110 | + | ||
111 | +/* | ||
112 | + * Load one element into @vd + @reg_off from (@env, @vaddr, @ra). | ||
113 | + * The controlling predicate is known to be true. | ||
114 | + */ | ||
115 | +typedef void sve_ld1_tlb_fn(CPUARMState *env, void *vd, intptr_t reg_off, | ||
116 | + target_ulong vaddr, int mmu_idx, uintptr_t ra); | ||
117 | + | ||
118 | +/* | ||
119 | + * Generate the above primitives. | ||
120 | + */ | ||
121 | + | ||
122 | +#define DO_LD_HOST(NAME, H, TYPEE, TYPEM, HOST) \ | ||
123 | +static intptr_t sve_##NAME##_host(void *vd, void *vg, void *host, \ | ||
124 | + intptr_t mem_off, const intptr_t mem_max) \ | ||
125 | +{ \ | ||
126 | + intptr_t reg_off = mem_off * (sizeof(TYPEE) / sizeof(TYPEM)); \ | ||
127 | + uint64_t *pg = vg; \ | ||
128 | + while (mem_off + sizeof(TYPEM) <= mem_max) { \ | ||
129 | + TYPEM val = 0; \ | ||
130 | + if (likely((pg[reg_off >> 6] >> (reg_off & 63)) & 1)) { \ | ||
131 | + val = HOST(host + mem_off); \ | ||
132 | + } \ | ||
133 | + *(TYPEE *)(vd + H(reg_off)) = val; \ | ||
134 | + mem_off += sizeof(TYPEM), reg_off += sizeof(TYPEE); \ | ||
135 | + } \ | ||
136 | + return mem_off; \ | ||
137 | } | ||
138 | |||
139 | +#ifdef CONFIG_SOFTMMU | ||
140 | +#define DO_LD_TLB(NAME, H, TYPEE, TYPEM, HOST, MOEND, TLB) \ | ||
141 | +static void sve_##NAME##_tlb(CPUARMState *env, void *vd, intptr_t reg_off, \ | ||
142 | + target_ulong addr, int mmu_idx, uintptr_t ra) \ | ||
143 | +{ \ | ||
144 | + TCGMemOpIdx oi = make_memop_idx(ctz32(sizeof(TYPEM)) | MOEND, mmu_idx); \ | ||
145 | + TYPEM val = TLB(env, addr, oi, ra); \ | ||
146 | + *(TYPEE *)(vd + H(reg_off)) = val; \ | ||
147 | +} | ||
148 | +#else | ||
149 | +#define DO_LD_TLB(NAME, H, TYPEE, TYPEM, HOST, MOEND, TLB) \ | ||
150 | +static void sve_##NAME##_tlb(CPUARMState *env, void *vd, intptr_t reg_off, \ | ||
151 | + target_ulong addr, int mmu_idx, uintptr_t ra) \ | ||
152 | +{ \ | ||
153 | + TYPEM val = HOST(g2h(addr)); \ | ||
154 | + *(TYPEE *)(vd + H(reg_off)) = val; \ | ||
155 | +} | ||
156 | +#endif | ||
157 | + | ||
158 | +#define DO_LD_PRIM_1(NAME, H, TE, TM) \ | ||
159 | + DO_LD_HOST(NAME, H, TE, TM, ldub_p) \ | ||
160 | + DO_LD_TLB(NAME, H, TE, TM, ldub_p, 0, helper_ret_ldub_mmu) | ||
161 | + | ||
162 | +DO_LD_PRIM_1(ld1bb, H1, uint8_t, uint8_t) | ||
163 | +DO_LD_PRIM_1(ld1bhu, H1_2, uint16_t, uint8_t) | ||
164 | +DO_LD_PRIM_1(ld1bhs, H1_2, uint16_t, int8_t) | ||
165 | +DO_LD_PRIM_1(ld1bsu, H1_4, uint32_t, uint8_t) | ||
166 | +DO_LD_PRIM_1(ld1bss, H1_4, uint32_t, int8_t) | ||
167 | +DO_LD_PRIM_1(ld1bdu, , uint64_t, uint8_t) | ||
168 | +DO_LD_PRIM_1(ld1bds, , uint64_t, int8_t) | ||
169 | + | ||
170 | +#define DO_LD_PRIM_2(NAME, end, MOEND, H, TE, TM, PH, PT) \ | ||
171 | + DO_LD_HOST(NAME##_##end, H, TE, TM, PH##_##end##_p) \ | ||
172 | + DO_LD_TLB(NAME##_##end, H, TE, TM, PH##_##end##_p, \ | ||
173 | + MOEND, helper_##end##_##PT##_mmu) | ||
174 | + | ||
175 | +DO_LD_PRIM_2(ld1hh, le, MO_LE, H1_2, uint16_t, uint16_t, lduw, lduw) | ||
176 | +DO_LD_PRIM_2(ld1hsu, le, MO_LE, H1_4, uint32_t, uint16_t, lduw, lduw) | ||
177 | +DO_LD_PRIM_2(ld1hss, le, MO_LE, H1_4, uint32_t, int16_t, lduw, lduw) | ||
178 | +DO_LD_PRIM_2(ld1hdu, le, MO_LE, , uint64_t, uint16_t, lduw, lduw) | ||
179 | +DO_LD_PRIM_2(ld1hds, le, MO_LE, , uint64_t, int16_t, lduw, lduw) | ||
180 | + | ||
181 | +DO_LD_PRIM_2(ld1ss, le, MO_LE, H1_4, uint32_t, uint32_t, ldl, ldul) | ||
182 | +DO_LD_PRIM_2(ld1sdu, le, MO_LE, , uint64_t, uint32_t, ldl, ldul) | ||
183 | +DO_LD_PRIM_2(ld1sds, le, MO_LE, , uint64_t, int32_t, ldl, ldul) | ||
184 | + | ||
185 | +DO_LD_PRIM_2(ld1dd, le, MO_LE, , uint64_t, uint64_t, ldq, ldq) | ||
186 | + | ||
187 | +DO_LD_PRIM_2(ld1hh, be, MO_BE, H1_2, uint16_t, uint16_t, lduw, lduw) | ||
188 | +DO_LD_PRIM_2(ld1hsu, be, MO_BE, H1_4, uint32_t, uint16_t, lduw, lduw) | ||
189 | +DO_LD_PRIM_2(ld1hss, be, MO_BE, H1_4, uint32_t, int16_t, lduw, lduw) | ||
190 | +DO_LD_PRIM_2(ld1hdu, be, MO_BE, , uint64_t, uint16_t, lduw, lduw) | ||
191 | +DO_LD_PRIM_2(ld1hds, be, MO_BE, , uint64_t, int16_t, lduw, lduw) | ||
192 | + | ||
193 | +DO_LD_PRIM_2(ld1ss, be, MO_BE, H1_4, uint32_t, uint32_t, ldl, ldul) | ||
194 | +DO_LD_PRIM_2(ld1sdu, be, MO_BE, , uint64_t, uint32_t, ldl, ldul) | ||
195 | +DO_LD_PRIM_2(ld1sds, be, MO_BE, , uint64_t, int32_t, ldl, ldul) | ||
196 | + | ||
197 | +DO_LD_PRIM_2(ld1dd, be, MO_BE, , uint64_t, uint64_t, ldq, ldq) | ||
198 | + | ||
199 | +#undef DO_LD_TLB | ||
200 | +#undef DO_LD_HOST | ||
201 | +#undef DO_LD_PRIM_1 | ||
202 | +#undef DO_LD_PRIM_2 | ||
203 | + | ||
204 | +/* | ||
205 | + * Skip through a sequence of inactive elements in the guarding predicate @vg, | ||
206 | + * beginning at @reg_off bounded by @reg_max. Return the offset of the active | ||
207 | + * element >= @reg_off, or @reg_max if there were no active elements at all. | ||
208 | + */ | ||
209 | +static intptr_t find_next_active(uint64_t *vg, intptr_t reg_off, | ||
210 | + intptr_t reg_max, int esz) | ||
211 | +{ | ||
212 | + uint64_t pg_mask = pred_esz_masks[esz]; | ||
213 | + uint64_t pg = (vg[reg_off >> 6] & pg_mask) >> (reg_off & 63); | ||
214 | + | ||
215 | + /* In normal usage, the first element is active. */ | ||
216 | + if (likely(pg & 1)) { | ||
217 | + return reg_off; | ||
218 | + } | ||
219 | + | ||
220 | + if (pg == 0) { | ||
221 | + reg_off &= -64; | ||
222 | + do { | ||
223 | + reg_off += 64; | ||
224 | + if (unlikely(reg_off >= reg_max)) { | ||
225 | + /* The entire predicate was false. */ | ||
226 | + return reg_max; | ||
227 | + } | ||
228 | + pg = vg[reg_off >> 6] & pg_mask; | ||
229 | + } while (pg == 0); | ||
230 | + } | ||
231 | + reg_off += ctz64(pg); | ||
232 | + | ||
233 | + /* We should never see an out of range predicate bit set. */ | ||
234 | + tcg_debug_assert(reg_off < reg_max); | ||
235 | + return reg_off; | ||
236 | +} | ||
237 | + | ||
238 | +/* | ||
239 | + * Return the maximum offset <= @mem_max which is still within the page | ||
240 | + * referenced by @base + @mem_off. | ||
241 | + */ | ||
242 | +static intptr_t max_for_page(target_ulong base, intptr_t mem_off, | ||
243 | + intptr_t mem_max) | ||
244 | +{ | ||
245 | + target_ulong addr = base + mem_off; | ||
246 | + intptr_t split = -(intptr_t)(addr | TARGET_PAGE_MASK); | ||
247 | + return MIN(split, mem_max - mem_off) + mem_off; | ||
248 | +} | ||
249 | + | ||
250 | +static inline void set_helper_retaddr(uintptr_t ra) | ||
251 | +{ | ||
252 | +#ifdef CONFIG_USER_ONLY | ||
253 | + helper_retaddr = ra; | ||
254 | +#endif | ||
255 | +} | ||
256 | + | ||
257 | +/* | ||
258 | + * The result of tlb_vaddr_to_host for user-only is just g2h(x), | ||
259 | + * which is always non-null. Elide the useless test. | ||
260 | + */ | ||
261 | +static inline bool test_host_page(void *host) | ||
262 | +{ | ||
263 | +#ifdef CONFIG_USER_ONLY | ||
264 | + return true; | ||
265 | +#else | ||
266 | + return likely(host != NULL); | ||
267 | +#endif | ||
268 | +} | ||
269 | + | ||
270 | +/* | ||
271 | + * Common helper for all contiguous one-register predicated loads. | ||
272 | + */ | ||
273 | +static void sve_ld1_r(CPUARMState *env, void *vg, const target_ulong addr, | ||
274 | + uint32_t desc, const uintptr_t retaddr, | ||
275 | + const int esz, const int msz, | ||
276 | + sve_ld1_host_fn *host_fn, | ||
277 | + sve_ld1_tlb_fn *tlb_fn) | ||
278 | +{ | ||
279 | + void *vd = &env->vfp.zregs[simd_data(desc)]; | ||
280 | + const int diffsz = esz - msz; | ||
281 | + const intptr_t reg_max = simd_oprsz(desc); | ||
282 | + const intptr_t mem_max = reg_max >> diffsz; | ||
283 | + const int mmu_idx = cpu_mmu_index(env, false); | ||
284 | + ARMVectorReg scratch; | ||
285 | + void *host; | ||
286 | + intptr_t split, reg_off, mem_off; | ||
287 | + | ||
288 | + /* Find the first active element. */ | ||
289 | + reg_off = find_next_active(vg, 0, reg_max, esz); | ||
290 | + if (unlikely(reg_off == reg_max)) { | ||
291 | + /* The entire predicate was false; no load occurs. */ | ||
292 | + memset(vd, 0, reg_max); | ||
293 | + return; | ||
294 | + } | ||
295 | + mem_off = reg_off >> diffsz; | ||
296 | + set_helper_retaddr(retaddr); | ||
297 | + | ||
298 | + /* | ||
299 | + * If the (remaining) load is entirely within a single page, then: | ||
300 | + * For softmmu, and the tlb hits, then no faults will occur; | ||
301 | + * For user-only, either the first load will fault or none will. | ||
302 | + * We can thus perform the load directly to the destination and | ||
303 | + * Vd will be unmodified on any exception path. | ||
304 | + */ | ||
305 | + split = max_for_page(addr, mem_off, mem_max); | ||
306 | + if (likely(split == mem_max)) { | ||
307 | + host = tlb_vaddr_to_host(env, addr + mem_off, MMU_DATA_LOAD, mmu_idx); | ||
308 | + if (test_host_page(host)) { | ||
309 | + mem_off = host_fn(vd, vg, host - mem_off, mem_off, mem_max); | ||
310 | + tcg_debug_assert(mem_off == mem_max); | ||
311 | + set_helper_retaddr(0); | ||
312 | + /* After having taken any fault, zero leading inactive elements. */ | ||
313 | + swap_memzero(vd, reg_off); | ||
314 | + return; | ||
315 | + } | ||
316 | + } | ||
317 | + | ||
318 | + /* | ||
319 | + * Perform the predicated read into a temporary, thus ensuring | ||
320 | + * if the load of the last element faults, Vd is not modified. | ||
321 | + */ | ||
322 | +#ifdef CONFIG_USER_ONLY | ||
323 | + swap_memzero(&scratch, reg_off); | ||
324 | + host_fn(&scratch, vg, g2h(addr), mem_off, mem_max); | ||
325 | +#else | ||
326 | + memset(&scratch, 0, reg_max); | ||
327 | + goto start; | ||
328 | + while (1) { | ||
329 | + reg_off = find_next_active(vg, reg_off, reg_max, esz); | ||
330 | + if (reg_off >= reg_max) { | ||
331 | + break; | ||
332 | + } | ||
333 | + mem_off = reg_off >> diffsz; | ||
334 | + split = max_for_page(addr, mem_off, mem_max); | ||
335 | + | ||
336 | + start: | ||
337 | + if (split - mem_off >= (1 << msz)) { | ||
338 | + /* At least one whole element on this page. */ | ||
339 | + host = tlb_vaddr_to_host(env, addr + mem_off, | ||
340 | + MMU_DATA_LOAD, mmu_idx); | ||
341 | + if (host) { | ||
342 | + mem_off = host_fn(&scratch, vg, host - mem_off, | ||
343 | + mem_off, split); | ||
344 | + reg_off = mem_off << diffsz; | ||
345 | + continue; | ||
346 | + } | ||
347 | + } | ||
348 | + | ||
349 | + /* | ||
350 | + * Perform one normal read. This may fault, longjmping out to the | ||
351 | + * main loop in order to raise an exception. It may succeed, and | ||
352 | + * as a side-effect load the TLB entry for the next round. Finally, | ||
353 | + * in the extremely unlikely case we're performing this operation | ||
354 | + * on I/O memory, it may succeed but not bring in the TLB entry. | ||
355 | + * But even then we have still made forward progress. | ||
356 | + */ | ||
357 | + tlb_fn(env, &scratch, reg_off, addr + mem_off, mmu_idx, retaddr); | ||
358 | + reg_off += 1 << esz; | ||
359 | + } | ||
360 | +#endif | ||
361 | + | ||
362 | + set_helper_retaddr(0); | ||
363 | + memcpy(vd, &scratch, reg_max); | ||
364 | +} | ||
365 | + | ||
366 | +#define DO_LD1_1(NAME, ESZ) \ | ||
367 | +void HELPER(sve_##NAME##_r)(CPUARMState *env, void *vg, \ | ||
368 | + target_ulong addr, uint32_t desc) \ | ||
369 | +{ \ | ||
370 | + sve_ld1_r(env, vg, addr, desc, GETPC(), ESZ, 0, \ | ||
371 | + sve_##NAME##_host, sve_##NAME##_tlb); \ | ||
372 | +} | ||
373 | + | ||
374 | +/* TODO: Propagate the endian check back to the translator. */ | ||
375 | +#define DO_LD1_2(NAME, ESZ, MSZ) \ | ||
376 | +void HELPER(sve_##NAME##_r)(CPUARMState *env, void *vg, \ | ||
377 | + target_ulong addr, uint32_t desc) \ | ||
378 | +{ \ | ||
379 | + if (arm_cpu_data_is_big_endian(env)) { \ | ||
380 | + sve_ld1_r(env, vg, addr, desc, GETPC(), ESZ, MSZ, \ | ||
381 | + sve_##NAME##_be_host, sve_##NAME##_be_tlb); \ | ||
382 | + } else { \ | ||
383 | + sve_ld1_r(env, vg, addr, desc, GETPC(), ESZ, MSZ, \ | ||
384 | + sve_##NAME##_le_host, sve_##NAME##_le_tlb); \ | ||
385 | + } \ | ||
386 | +} | ||
387 | + | ||
388 | +DO_LD1_1(ld1bb, 0) | ||
389 | +DO_LD1_1(ld1bhu, 1) | ||
390 | +DO_LD1_1(ld1bhs, 1) | ||
391 | +DO_LD1_1(ld1bsu, 2) | ||
392 | +DO_LD1_1(ld1bss, 2) | ||
393 | +DO_LD1_1(ld1bdu, 3) | ||
394 | +DO_LD1_1(ld1bds, 3) | ||
395 | + | ||
396 | +DO_LD1_2(ld1hh, 1, 1) | ||
397 | +DO_LD1_2(ld1hsu, 2, 1) | ||
398 | +DO_LD1_2(ld1hss, 2, 1) | ||
399 | +DO_LD1_2(ld1hdu, 3, 1) | ||
400 | +DO_LD1_2(ld1hds, 3, 1) | ||
401 | + | ||
402 | +DO_LD1_2(ld1ss, 2, 2) | ||
403 | +DO_LD1_2(ld1sdu, 3, 2) | ||
404 | +DO_LD1_2(ld1sds, 3, 2) | ||
405 | + | ||
406 | +DO_LD1_2(ld1dd, 3, 3) | ||
407 | + | ||
408 | +#undef DO_LD1_1 | ||
409 | +#undef DO_LD1_2 | ||
410 | + | ||
411 | #define DO_LD2(NAME, FN, TYPEE, TYPEM, H) \ | ||
412 | void HELPER(NAME)(CPUARMState *env, void *vg, \ | ||
413 | target_ulong addr, uint32_t desc) \ | ||
414 | @@ -XXX,XX +XXX,XX @@ void HELPER(NAME)(CPUARMState *env, void *vg, \ | ||
415 | } \ | ||
416 | } | ||
417 | |||
418 | -DO_LD1(sve_ld1bhu_r, cpu_ldub_data_ra, uint16_t, uint8_t, H1_2) | ||
419 | -DO_LD1(sve_ld1bhs_r, cpu_ldsb_data_ra, uint16_t, int8_t, H1_2) | ||
420 | -DO_LD1(sve_ld1bsu_r, cpu_ldub_data_ra, uint32_t, uint8_t, H1_4) | ||
421 | -DO_LD1(sve_ld1bss_r, cpu_ldsb_data_ra, uint32_t, int8_t, H1_4) | ||
422 | -DO_LD1(sve_ld1bdu_r, cpu_ldub_data_ra, uint64_t, uint8_t, ) | ||
423 | -DO_LD1(sve_ld1bds_r, cpu_ldsb_data_ra, uint64_t, int8_t, ) | ||
424 | - | ||
425 | -DO_LD1(sve_ld1hsu_r, cpu_lduw_data_ra, uint32_t, uint16_t, H1_4) | ||
426 | -DO_LD1(sve_ld1hss_r, cpu_ldsw_data_ra, uint32_t, int16_t, H1_4) | ||
427 | -DO_LD1(sve_ld1hdu_r, cpu_lduw_data_ra, uint64_t, uint16_t, ) | ||
428 | -DO_LD1(sve_ld1hds_r, cpu_ldsw_data_ra, uint64_t, int16_t, ) | ||
429 | - | ||
430 | -DO_LD1(sve_ld1sdu_r, cpu_ldl_data_ra, uint64_t, uint32_t, ) | ||
431 | -DO_LD1(sve_ld1sds_r, cpu_ldl_data_ra, uint64_t, int32_t, ) | ||
432 | - | ||
433 | -DO_LD1(sve_ld1bb_r, cpu_ldub_data_ra, uint8_t, uint8_t, H1) | ||
434 | DO_LD2(sve_ld2bb_r, cpu_ldub_data_ra, uint8_t, uint8_t, H1) | ||
435 | DO_LD3(sve_ld3bb_r, cpu_ldub_data_ra, uint8_t, uint8_t, H1) | ||
436 | DO_LD4(sve_ld4bb_r, cpu_ldub_data_ra, uint8_t, uint8_t, H1) | ||
437 | |||
438 | -DO_LD1(sve_ld1hh_r, cpu_lduw_data_ra, uint16_t, uint16_t, H1_2) | ||
439 | DO_LD2(sve_ld2hh_r, cpu_lduw_data_ra, uint16_t, uint16_t, H1_2) | ||
440 | DO_LD3(sve_ld3hh_r, cpu_lduw_data_ra, uint16_t, uint16_t, H1_2) | ||
441 | DO_LD4(sve_ld4hh_r, cpu_lduw_data_ra, uint16_t, uint16_t, H1_2) | ||
442 | |||
443 | -DO_LD1(sve_ld1ss_r, cpu_ldl_data_ra, uint32_t, uint32_t, H1_4) | ||
444 | DO_LD2(sve_ld2ss_r, cpu_ldl_data_ra, uint32_t, uint32_t, H1_4) | ||
445 | DO_LD3(sve_ld3ss_r, cpu_ldl_data_ra, uint32_t, uint32_t, H1_4) | ||
446 | DO_LD4(sve_ld4ss_r, cpu_ldl_data_ra, uint32_t, uint32_t, H1_4) | ||
447 | |||
448 | -DO_LD1(sve_ld1dd_r, cpu_ldq_data_ra, uint64_t, uint64_t, ) | ||
449 | DO_LD2(sve_ld2dd_r, cpu_ldq_data_ra, uint64_t, uint64_t, ) | ||
450 | DO_LD3(sve_ld3dd_r, cpu_ldq_data_ra, uint64_t, uint64_t, ) | ||
451 | DO_LD4(sve_ld4dd_r, cpu_ldq_data_ra, uint64_t, uint64_t, ) | ||
452 | |||
453 | -#undef DO_LD1 | ||
454 | #undef DO_LD2 | ||
455 | #undef DO_LD3 | ||
456 | #undef DO_LD4 | ||
457 | |||
458 | /* | ||
459 | * Load contiguous data, first-fault and no-fault. | ||
460 | + * | ||
461 | + * For user-only, one could argue that we should hold the mmap_lock during | ||
462 | + * the operation so that there is no race between page_check_range and the | ||
463 | + * load operation. However, unmapping pages out from under a running thread | ||
464 | + * is extraordinarily unlikely. This theoretical race condition also affects | ||
465 | + * linux-user/ in its get_user/put_user macros. | ||
466 | + * | ||
467 | + * TODO: Construct some helpers, written in assembly, that interact with | ||
468 | + * handle_cpu_signal to produce memory ops which can properly report errors | ||
469 | + * without racing. | ||
470 | */ | ||
471 | |||
472 | -#ifdef CONFIG_USER_ONLY | ||
473 | - | ||
474 | /* Fault on byte I. All bits in FFR from I are cleared. The vector | ||
475 | * result from I is CONSTRAINED UNPREDICTABLE; we choose the MERGE | ||
476 | * option, which leaves subsequent data unchanged. | ||
477 | @@ -XXX,XX +XXX,XX @@ static void record_fault(CPUARMState *env, uintptr_t i, uintptr_t oprsz) | ||
478 | } | ||
479 | } | ||
480 | |||
481 | -/* Hold the mmap lock during the operation so that there is no race | ||
482 | - * between page_check_range and the load operation. We expect the | ||
483 | - * usual case to have no faults at all, so we check the whole range | ||
484 | - * first and if successful defer to the normal load operation. | ||
485 | - * | ||
486 | - * TODO: Change mmap_lock to a rwlock so that multiple readers | ||
487 | - * can run simultaneously. This will probably help other uses | ||
488 | - * within QEMU as well. | ||
489 | +/* | ||
490 | + * Common helper for all contiguous first-fault loads. | ||
491 | */ | ||
492 | -#define DO_LDFF1(PART, FN, TYPEE, TYPEM, H) \ | ||
493 | -static void do_sve_ldff1##PART(CPUARMState *env, void *vd, void *vg, \ | ||
494 | - target_ulong addr, intptr_t oprsz, \ | ||
495 | - bool first, uintptr_t ra) \ | ||
496 | -{ \ | ||
497 | - intptr_t i = 0; \ | ||
498 | - do { \ | ||
499 | - uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); \ | ||
500 | - do { \ | ||
501 | - TYPEM m = 0; \ | ||
502 | - if (pg & 1) { \ | ||
503 | - if (!first && \ | ||
504 | - unlikely(page_check_range(addr, sizeof(TYPEM), \ | ||
505 | - PAGE_READ))) { \ | ||
506 | - record_fault(env, i, oprsz); \ | ||
507 | - return; \ | ||
508 | - } \ | ||
509 | - m = FN(env, addr, ra); \ | ||
510 | - first = false; \ | ||
511 | - } \ | ||
512 | - *(TYPEE *)(vd + H(i)) = m; \ | ||
513 | - i += sizeof(TYPEE), pg >>= sizeof(TYPEE); \ | ||
514 | - addr += sizeof(TYPEM); \ | ||
515 | - } while (i & 15); \ | ||
516 | - } while (i < oprsz); \ | ||
517 | -} \ | ||
518 | -void HELPER(sve_ldff1##PART)(CPUARMState *env, void *vg, \ | ||
519 | - target_ulong addr, uint32_t desc) \ | ||
520 | -{ \ | ||
521 | - intptr_t oprsz = simd_oprsz(desc); \ | ||
522 | - unsigned rd = simd_data(desc); \ | ||
523 | - void *vd = &env->vfp.zregs[rd]; \ | ||
524 | - mmap_lock(); \ | ||
525 | - if (likely(page_check_range(addr, oprsz, PAGE_READ) == 0)) { \ | ||
526 | - do_sve_ld1##PART(env, vd, vg, addr, oprsz, GETPC()); \ | ||
527 | - } else { \ | ||
528 | - do_sve_ldff1##PART(env, vd, vg, addr, oprsz, true, GETPC()); \ | ||
529 | - } \ | ||
530 | - mmap_unlock(); \ | ||
531 | -} | ||
532 | +static void sve_ldff1_r(CPUARMState *env, void *vg, const target_ulong addr, | ||
533 | + uint32_t desc, const uintptr_t retaddr, | ||
534 | + const int esz, const int msz, | ||
535 | + sve_ld1_host_fn *host_fn, | ||
536 | + sve_ld1_tlb_fn *tlb_fn) | ||
537 | +{ | ||
538 | + void *vd = &env->vfp.zregs[simd_data(desc)]; | ||
539 | + const int diffsz = esz - msz; | ||
540 | + const intptr_t reg_max = simd_oprsz(desc); | ||
541 | + const intptr_t mem_max = reg_max >> diffsz; | ||
542 | + const int mmu_idx = cpu_mmu_index(env, false); | ||
543 | + intptr_t split, reg_off, mem_off; | ||
544 | + void *host; | ||
545 | |||
546 | -/* No-fault loads are like first-fault loads without the | ||
547 | - * first faulting special case. | ||
548 | - */ | ||
549 | -#define DO_LDNF1(PART) \ | ||
550 | -void HELPER(sve_ldnf1##PART)(CPUARMState *env, void *vg, \ | ||
551 | - target_ulong addr, uint32_t desc) \ | ||
552 | -{ \ | ||
553 | - intptr_t oprsz = simd_oprsz(desc); \ | ||
554 | - unsigned rd = simd_data(desc); \ | ||
555 | - void *vd = &env->vfp.zregs[rd]; \ | ||
556 | - mmap_lock(); \ | ||
557 | - if (likely(page_check_range(addr, oprsz, PAGE_READ) == 0)) { \ | ||
558 | - do_sve_ld1##PART(env, vd, vg, addr, oprsz, GETPC()); \ | ||
559 | - } else { \ | ||
560 | - do_sve_ldff1##PART(env, vd, vg, addr, oprsz, false, GETPC()); \ | ||
561 | - } \ | ||
562 | - mmap_unlock(); \ | ||
563 | -} | ||
564 | + /* Skip to the first active element. */ | ||
565 | + reg_off = find_next_active(vg, 0, reg_max, esz); | ||
566 | + if (unlikely(reg_off == reg_max)) { | ||
567 | + /* The entire predicate was false; no load occurs. */ | ||
568 | + memset(vd, 0, reg_max); | ||
569 | + return; | ||
570 | + } | ||
571 | + mem_off = reg_off >> diffsz; | ||
572 | + set_helper_retaddr(retaddr); | ||
573 | |||
574 | + /* | ||
575 | + * If the (remaining) load is entirely within a single page, then: | ||
576 | + * For softmmu, and the tlb hits, then no faults will occur; | ||
577 | + * For user-only, either the first load will fault or none will. | ||
578 | + * We can thus perform the load directly to the destination and | ||
579 | + * Vd will be unmodified on any exception path. | ||
580 | + */ | ||
581 | + split = max_for_page(addr, mem_off, mem_max); | ||
582 | + if (likely(split == mem_max)) { | ||
583 | + host = tlb_vaddr_to_host(env, addr + mem_off, MMU_DATA_LOAD, mmu_idx); | ||
584 | + if (test_host_page(host)) { | ||
585 | + mem_off = host_fn(vd, vg, host - mem_off, mem_off, mem_max); | ||
586 | + tcg_debug_assert(mem_off == mem_max); | ||
587 | + set_helper_retaddr(0); | ||
588 | + /* After any fault, zero any leading inactive elements. */ | ||
589 | + swap_memzero(vd, reg_off); | ||
590 | + return; | ||
591 | + } | ||
592 | + } | ||
593 | + | ||
594 | +#ifdef CONFIG_USER_ONLY | ||
595 | + /* | ||
596 | + * The page(s) containing this first element at ADDR+MEM_OFF must | ||
597 | + * be valid. Considering that this first element may be misaligned | ||
598 | + * and cross a page boundary itself, take the rest of the page from | ||
599 | + * the last byte of the element. | ||
600 | + */ | ||
601 | + split = max_for_page(addr, mem_off + (1 << msz) - 1, mem_max); | ||
602 | + mem_off = host_fn(vd, vg, g2h(addr), mem_off, split); | ||
603 | + | ||
604 | + /* After any fault, zero any leading inactive elements. */ | ||
605 | + swap_memzero(vd, reg_off); | ||
606 | + reg_off = mem_off << diffsz; | ||
607 | #else | ||
608 | + /* | ||
609 | + * Perform one normal read, which will fault or not. | ||
610 | + * But it is likely to bring the page into the tlb. | ||
611 | + */ | ||
612 | + tlb_fn(env, vd, reg_off, addr + mem_off, mmu_idx, retaddr); | ||
613 | |||
614 | -/* TODO: System mode is not yet supported. | ||
615 | - * This would probably use tlb_vaddr_to_host. | ||
616 | - */ | ||
617 | -#define DO_LDFF1(PART, FN, TYPEE, TYPEM, H) \ | ||
618 | -void HELPER(sve_ldff1##PART)(CPUARMState *env, void *vg, \ | ||
619 | - target_ulong addr, uint32_t desc) \ | ||
620 | -{ \ | ||
621 | - g_assert_not_reached(); \ | ||
622 | -} | ||
623 | - | ||
624 | -#define DO_LDNF1(PART) \ | ||
625 | -void HELPER(sve_ldnf1##PART)(CPUARMState *env, void *vg, \ | ||
626 | - target_ulong addr, uint32_t desc) \ | ||
627 | -{ \ | ||
628 | - g_assert_not_reached(); \ | ||
629 | -} | ||
630 | + /* After any fault, zero any leading predicated false elts. */ | ||
631 | + swap_memzero(vd, reg_off); | ||
632 | + mem_off += 1 << msz; | ||
633 | + reg_off += 1 << esz; | ||
634 | |||
635 | + /* Try again to read the balance of the page. */ | ||
636 | + split = max_for_page(addr, mem_off - 1, mem_max); | ||
637 | + if (split >= (1 << msz)) { | ||
638 | + host = tlb_vaddr_to_host(env, addr + mem_off, MMU_DATA_LOAD, mmu_idx); | ||
639 | + if (host) { | ||
640 | + mem_off = host_fn(vd, vg, host - mem_off, mem_off, split); | ||
641 | + reg_off = mem_off << diffsz; | ||
642 | + } | ||
643 | + } | ||
644 | #endif | ||
645 | |||
646 | -DO_LDFF1(bb_r, cpu_ldub_data_ra, uint8_t, uint8_t, H1) | ||
647 | -DO_LDFF1(bhu_r, cpu_ldub_data_ra, uint16_t, uint8_t, H1_2) | ||
648 | -DO_LDFF1(bhs_r, cpu_ldsb_data_ra, uint16_t, int8_t, H1_2) | ||
649 | -DO_LDFF1(bsu_r, cpu_ldub_data_ra, uint32_t, uint8_t, H1_4) | ||
650 | -DO_LDFF1(bss_r, cpu_ldsb_data_ra, uint32_t, int8_t, H1_4) | ||
651 | -DO_LDFF1(bdu_r, cpu_ldub_data_ra, uint64_t, uint8_t, ) | ||
652 | -DO_LDFF1(bds_r, cpu_ldsb_data_ra, uint64_t, int8_t, ) | ||
653 | + set_helper_retaddr(0); | ||
654 | + record_fault(env, reg_off, reg_max); | ||
655 | +} | ||
656 | |||
657 | -DO_LDFF1(hh_r, cpu_lduw_data_ra, uint16_t, uint16_t, H1_2) | ||
658 | -DO_LDFF1(hsu_r, cpu_lduw_data_ra, uint32_t, uint16_t, H1_4) | ||
659 | -DO_LDFF1(hss_r, cpu_ldsw_data_ra, uint32_t, int8_t, H1_4) | ||
660 | -DO_LDFF1(hdu_r, cpu_lduw_data_ra, uint64_t, uint16_t, ) | ||
661 | -DO_LDFF1(hds_r, cpu_ldsw_data_ra, uint64_t, int16_t, ) | ||
662 | +/* | ||
663 | + * Common helper for all contiguous no-fault loads. | ||
664 | + */ | ||
665 | +static void sve_ldnf1_r(CPUARMState *env, void *vg, const target_ulong addr, | ||
666 | + uint32_t desc, const int esz, const int msz, | ||
667 | + sve_ld1_host_fn *host_fn) | ||
668 | +{ | ||
669 | + void *vd = &env->vfp.zregs[simd_data(desc)]; | ||
670 | + const int diffsz = esz - msz; | ||
671 | + const intptr_t reg_max = simd_oprsz(desc); | ||
672 | + const intptr_t mem_max = reg_max >> diffsz; | ||
673 | + const int mmu_idx = cpu_mmu_index(env, false); | ||
674 | + intptr_t split, reg_off, mem_off; | ||
675 | + void *host; | ||
676 | |||
677 | -DO_LDFF1(ss_r, cpu_ldl_data_ra, uint32_t, uint32_t, H1_4) | ||
678 | -DO_LDFF1(sdu_r, cpu_ldl_data_ra, uint64_t, uint32_t, ) | ||
679 | -DO_LDFF1(sds_r, cpu_ldl_data_ra, uint64_t, int32_t, ) | ||
680 | +#ifdef CONFIG_USER_ONLY | ||
681 | + host = tlb_vaddr_to_host(env, addr, MMU_DATA_LOAD, mmu_idx); | ||
682 | + if (likely(page_check_range(addr, mem_max, PAGE_READ) == 0)) { | ||
683 | + /* The entire operation is valid and will not fault. */ | ||
684 | + host_fn(vd, vg, host, 0, mem_max); | ||
685 | + return; | ||
686 | + } | ||
687 | +#endif | ||
688 | |||
689 | -DO_LDFF1(dd_r, cpu_ldq_data_ra, uint64_t, uint64_t, ) | ||
690 | + /* There will be no fault, so we may modify in advance. */ | ||
691 | + memset(vd, 0, reg_max); | ||
692 | |||
693 | -#undef DO_LDFF1 | ||
694 | + /* Skip to the first active element. */ | ||
695 | + reg_off = find_next_active(vg, 0, reg_max, esz); | ||
696 | + if (unlikely(reg_off == reg_max)) { | ||
697 | + /* The entire predicate was false; no load occurs. */ | ||
698 | + return; | ||
699 | + } | ||
700 | + mem_off = reg_off >> diffsz; | ||
701 | |||
702 | -DO_LDNF1(bb_r) | ||
703 | -DO_LDNF1(bhu_r) | ||
704 | -DO_LDNF1(bhs_r) | ||
705 | -DO_LDNF1(bsu_r) | ||
706 | -DO_LDNF1(bss_r) | ||
707 | -DO_LDNF1(bdu_r) | ||
708 | -DO_LDNF1(bds_r) | ||
709 | +#ifdef CONFIG_USER_ONLY | ||
710 | + if (page_check_range(addr + mem_off, 1 << msz, PAGE_READ) == 0) { | ||
711 | + /* At least one load is valid; take the rest of the page. */ | ||
712 | + split = max_for_page(addr, mem_off + (1 << msz) - 1, mem_max); | ||
713 | + mem_off = host_fn(vd, vg, host, mem_off, split); | ||
714 | + reg_off = mem_off << diffsz; | ||
715 | + } | ||
716 | +#else | ||
717 | + /* | ||
718 | + * If the address is not in the TLB, we have no way to bring the | ||
719 | + * entry into the TLB without also risking a fault. Note that | ||
720 | + * the corollary is that we never load from an address not in RAM. | ||
721 | + * | ||
722 | + * This last is out of spec, in a weird corner case. | ||
723 | + * Per the MemNF/MemSingleNF pseudocode, a NF load from Device memory | ||
724 | + * must not actually hit the bus -- it returns UNKNOWN data instead. | ||
725 | + * But if you map non-RAM with Normal memory attributes and do a NF | ||
726 | + * load then it should access the bus. (Nobody ought actually do this | ||
727 | + * in the real world, obviously.) | ||
728 | + * | ||
729 | + * Then there are the annoying special cases with watchpoints... | ||
730 | + * | ||
731 | + * TODO: Add a form of tlb_fill that does not raise an exception, | ||
732 | + * with a form of tlb_vaddr_to_host and a set of loads to match. | ||
733 | + * The non_fault_vaddr_to_host would handle everything, usually, | ||
734 | + * and the loads would handle the iomem path for watchpoints. | ||
735 | + */ | ||
736 | + host = tlb_vaddr_to_host(env, addr + mem_off, MMU_DATA_LOAD, mmu_idx); | ||
737 | + split = max_for_page(addr, mem_off, mem_max); | ||
738 | + if (host && split >= (1 << msz)) { | ||
739 | + mem_off = host_fn(vd, vg, host - mem_off, mem_off, split); | ||
740 | + reg_off = mem_off << diffsz; | ||
741 | + } | ||
742 | +#endif | ||
743 | |||
744 | -DO_LDNF1(hh_r) | ||
745 | -DO_LDNF1(hsu_r) | ||
746 | -DO_LDNF1(hss_r) | ||
747 | -DO_LDNF1(hdu_r) | ||
748 | -DO_LDNF1(hds_r) | ||
749 | + record_fault(env, reg_off, reg_max); | ||
750 | +} | ||
751 | |||
752 | -DO_LDNF1(ss_r) | ||
753 | -DO_LDNF1(sdu_r) | ||
754 | -DO_LDNF1(sds_r) | ||
755 | +#define DO_LDFF1_LDNF1_1(PART, ESZ) \ | ||
756 | +void HELPER(sve_ldff1##PART##_r)(CPUARMState *env, void *vg, \ | ||
757 | + target_ulong addr, uint32_t desc) \ | ||
758 | +{ \ | ||
759 | + sve_ldff1_r(env, vg, addr, desc, GETPC(), ESZ, 0, \ | ||
760 | + sve_ld1##PART##_host, sve_ld1##PART##_tlb); \ | ||
761 | +} \ | ||
762 | +void HELPER(sve_ldnf1##PART##_r)(CPUARMState *env, void *vg, \ | ||
763 | + target_ulong addr, uint32_t desc) \ | ||
764 | +{ \ | ||
765 | + sve_ldnf1_r(env, vg, addr, desc, ESZ, 0, sve_ld1##PART##_host); \ | ||
766 | +} | ||
767 | |||
768 | -DO_LDNF1(dd_r) | ||
769 | +/* TODO: Propagate the endian check back to the translator. */ | ||
770 | +#define DO_LDFF1_LDNF1_2(PART, ESZ, MSZ) \ | ||
771 | +void HELPER(sve_ldff1##PART##_r)(CPUARMState *env, void *vg, \ | ||
772 | + target_ulong addr, uint32_t desc) \ | ||
773 | +{ \ | ||
774 | + if (arm_cpu_data_is_big_endian(env)) { \ | ||
775 | + sve_ldff1_r(env, vg, addr, desc, GETPC(), ESZ, MSZ, \ | ||
776 | + sve_ld1##PART##_be_host, sve_ld1##PART##_be_tlb); \ | ||
777 | + } else { \ | ||
778 | + sve_ldff1_r(env, vg, addr, desc, GETPC(), ESZ, MSZ, \ | ||
779 | + sve_ld1##PART##_le_host, sve_ld1##PART##_le_tlb); \ | ||
780 | + } \ | ||
781 | +} \ | ||
782 | +void HELPER(sve_ldnf1##PART##_r)(CPUARMState *env, void *vg, \ | ||
783 | + target_ulong addr, uint32_t desc) \ | ||
784 | +{ \ | ||
785 | + if (arm_cpu_data_is_big_endian(env)) { \ | ||
786 | + sve_ldnf1_r(env, vg, addr, desc, ESZ, MSZ, \ | ||
787 | + sve_ld1##PART##_be_host); \ | ||
788 | + } else { \ | ||
789 | + sve_ldnf1_r(env, vg, addr, desc, ESZ, MSZ, \ | ||
790 | + sve_ld1##PART##_le_host); \ | ||
791 | + } \ | ||
792 | +} | ||
793 | |||
794 | -#undef DO_LDNF1 | ||
795 | +DO_LDFF1_LDNF1_1(bb, 0) | ||
796 | +DO_LDFF1_LDNF1_1(bhu, 1) | ||
797 | +DO_LDFF1_LDNF1_1(bhs, 1) | ||
798 | +DO_LDFF1_LDNF1_1(bsu, 2) | ||
799 | +DO_LDFF1_LDNF1_1(bss, 2) | ||
800 | +DO_LDFF1_LDNF1_1(bdu, 3) | ||
801 | +DO_LDFF1_LDNF1_1(bds, 3) | ||
802 | + | ||
803 | +DO_LDFF1_LDNF1_2(hh, 1, 1) | ||
804 | +DO_LDFF1_LDNF1_2(hsu, 2, 1) | ||
805 | +DO_LDFF1_LDNF1_2(hss, 2, 1) | ||
806 | +DO_LDFF1_LDNF1_2(hdu, 3, 1) | ||
807 | +DO_LDFF1_LDNF1_2(hds, 3, 1) | ||
808 | + | ||
809 | +DO_LDFF1_LDNF1_2(ss, 2, 2) | ||
810 | +DO_LDFF1_LDNF1_2(sdu, 3, 2) | ||
811 | +DO_LDFF1_LDNF1_2(sds, 3, 2) | ||
812 | + | ||
813 | +DO_LDFF1_LDNF1_2(dd, 3, 3) | ||
814 | + | ||
815 | +#undef DO_LDFF1_LDNF1_1 | ||
816 | +#undef DO_LDFF1_LDNF1_2 | ||
817 | |||
818 | /* | ||
819 | * Store contiguous data, protected by a governing predicate. | ||
820 | -- | ||
821 | 2.19.0 | ||
822 | |||
823 | diff view generated by jsdifflib |
Deleted patch | |||
---|---|---|---|
1 | From: Richard Henderson <richard.henderson@linaro.org> | ||
2 | 1 | ||
3 | Use the same *_tlb primitives as we use for ld1. | ||
4 | |||
5 | For linux-user, this hoists the set of helper_retaddr. For softmmu, | ||
6 | hoists the computation of the current mmu_idx outside the loop, | ||
7 | fixes the endianness problem, and moves the main loop out of a | ||
8 | macro and into an inlined function. | ||
9 | |||
10 | Reviewed-by: Peter Maydell <peter.maydell@linaro.org> | ||
11 | Tested-by: Laurent Desnogues <laurent.desnogues@gmail.com> | ||
12 | Signed-off-by: Richard Henderson <richard.henderson@linaro.org> | ||
13 | Message-id: 20181005175350.30752-9-richard.henderson@linaro.org | ||
14 | Signed-off-by: Peter Maydell <peter.maydell@linaro.org> | ||
15 | --- | ||
16 | target/arm/sve_helper.c | 210 ++++++++++++++++++++++------------------ | ||
17 | 1 file changed, 117 insertions(+), 93 deletions(-) | ||
18 | |||
19 | diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c | ||
20 | index XXXXXXX..XXXXXXX 100644 | ||
21 | --- a/target/arm/sve_helper.c | ||
22 | +++ b/target/arm/sve_helper.c | ||
23 | @@ -XXX,XX +XXX,XX @@ DO_LD1_2(ld1dd, 3, 3) | ||
24 | #undef DO_LD1_1 | ||
25 | #undef DO_LD1_2 | ||
26 | |||
27 | -#define DO_LD2(NAME, FN, TYPEE, TYPEM, H) \ | ||
28 | -void HELPER(NAME)(CPUARMState *env, void *vg, \ | ||
29 | - target_ulong addr, uint32_t desc) \ | ||
30 | -{ \ | ||
31 | - intptr_t i, oprsz = simd_oprsz(desc); \ | ||
32 | - intptr_t ra = GETPC(); \ | ||
33 | - unsigned rd = simd_data(desc); \ | ||
34 | - void *d1 = &env->vfp.zregs[rd]; \ | ||
35 | - void *d2 = &env->vfp.zregs[(rd + 1) & 31]; \ | ||
36 | - for (i = 0; i < oprsz; ) { \ | ||
37 | - uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); \ | ||
38 | - do { \ | ||
39 | - TYPEM m1 = 0, m2 = 0; \ | ||
40 | - if (pg & 1) { \ | ||
41 | - m1 = FN(env, addr, ra); \ | ||
42 | - m2 = FN(env, addr + sizeof(TYPEM), ra); \ | ||
43 | - } \ | ||
44 | - *(TYPEE *)(d1 + H(i)) = m1; \ | ||
45 | - *(TYPEE *)(d2 + H(i)) = m2; \ | ||
46 | - i += sizeof(TYPEE), pg >>= sizeof(TYPEE); \ | ||
47 | - addr += 2 * sizeof(TYPEM); \ | ||
48 | - } while (i & 15); \ | ||
49 | - } \ | ||
50 | +/* | ||
51 | + * Common helpers for all contiguous 2,3,4-register predicated loads. | ||
52 | + */ | ||
53 | +static void sve_ld2_r(CPUARMState *env, void *vg, target_ulong addr, | ||
54 | + uint32_t desc, int size, uintptr_t ra, | ||
55 | + sve_ld1_tlb_fn *tlb_fn) | ||
56 | +{ | ||
57 | + const int mmu_idx = cpu_mmu_index(env, false); | ||
58 | + intptr_t i, oprsz = simd_oprsz(desc); | ||
59 | + unsigned rd = simd_data(desc); | ||
60 | + ARMVectorReg scratch[2] = { }; | ||
61 | + | ||
62 | + set_helper_retaddr(ra); | ||
63 | + for (i = 0; i < oprsz; ) { | ||
64 | + uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); | ||
65 | + do { | ||
66 | + if (pg & 1) { | ||
67 | + tlb_fn(env, &scratch[0], i, addr, mmu_idx, ra); | ||
68 | + tlb_fn(env, &scratch[1], i, addr + size, mmu_idx, ra); | ||
69 | + } | ||
70 | + i += size, pg >>= size; | ||
71 | + addr += 2 * size; | ||
72 | + } while (i & 15); | ||
73 | + } | ||
74 | + set_helper_retaddr(0); | ||
75 | + | ||
76 | + /* Wait until all exceptions have been raised to write back. */ | ||
77 | + memcpy(&env->vfp.zregs[rd], &scratch[0], oprsz); | ||
78 | + memcpy(&env->vfp.zregs[(rd + 1) & 31], &scratch[1], oprsz); | ||
79 | } | ||
80 | |||
81 | -#define DO_LD3(NAME, FN, TYPEE, TYPEM, H) \ | ||
82 | -void HELPER(NAME)(CPUARMState *env, void *vg, \ | ||
83 | - target_ulong addr, uint32_t desc) \ | ||
84 | -{ \ | ||
85 | - intptr_t i, oprsz = simd_oprsz(desc); \ | ||
86 | - intptr_t ra = GETPC(); \ | ||
87 | - unsigned rd = simd_data(desc); \ | ||
88 | - void *d1 = &env->vfp.zregs[rd]; \ | ||
89 | - void *d2 = &env->vfp.zregs[(rd + 1) & 31]; \ | ||
90 | - void *d3 = &env->vfp.zregs[(rd + 2) & 31]; \ | ||
91 | - for (i = 0; i < oprsz; ) { \ | ||
92 | - uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); \ | ||
93 | - do { \ | ||
94 | - TYPEM m1 = 0, m2 = 0, m3 = 0; \ | ||
95 | - if (pg & 1) { \ | ||
96 | - m1 = FN(env, addr, ra); \ | ||
97 | - m2 = FN(env, addr + sizeof(TYPEM), ra); \ | ||
98 | - m3 = FN(env, addr + 2 * sizeof(TYPEM), ra); \ | ||
99 | - } \ | ||
100 | - *(TYPEE *)(d1 + H(i)) = m1; \ | ||
101 | - *(TYPEE *)(d2 + H(i)) = m2; \ | ||
102 | - *(TYPEE *)(d3 + H(i)) = m3; \ | ||
103 | - i += sizeof(TYPEE), pg >>= sizeof(TYPEE); \ | ||
104 | - addr += 3 * sizeof(TYPEM); \ | ||
105 | - } while (i & 15); \ | ||
106 | - } \ | ||
107 | +static void sve_ld3_r(CPUARMState *env, void *vg, target_ulong addr, | ||
108 | + uint32_t desc, int size, uintptr_t ra, | ||
109 | + sve_ld1_tlb_fn *tlb_fn) | ||
110 | +{ | ||
111 | + const int mmu_idx = cpu_mmu_index(env, false); | ||
112 | + intptr_t i, oprsz = simd_oprsz(desc); | ||
113 | + unsigned rd = simd_data(desc); | ||
114 | + ARMVectorReg scratch[3] = { }; | ||
115 | + | ||
116 | + set_helper_retaddr(ra); | ||
117 | + for (i = 0; i < oprsz; ) { | ||
118 | + uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); | ||
119 | + do { | ||
120 | + if (pg & 1) { | ||
121 | + tlb_fn(env, &scratch[0], i, addr, mmu_idx, ra); | ||
122 | + tlb_fn(env, &scratch[1], i, addr + size, mmu_idx, ra); | ||
123 | + tlb_fn(env, &scratch[2], i, addr + 2 * size, mmu_idx, ra); | ||
124 | + } | ||
125 | + i += size, pg >>= size; | ||
126 | + addr += 3 * size; | ||
127 | + } while (i & 15); | ||
128 | + } | ||
129 | + set_helper_retaddr(0); | ||
130 | + | ||
131 | + /* Wait until all exceptions have been raised to write back. */ | ||
132 | + memcpy(&env->vfp.zregs[rd], &scratch[0], oprsz); | ||
133 | + memcpy(&env->vfp.zregs[(rd + 1) & 31], &scratch[1], oprsz); | ||
134 | + memcpy(&env->vfp.zregs[(rd + 2) & 31], &scratch[2], oprsz); | ||
135 | } | ||
136 | |||
137 | -#define DO_LD4(NAME, FN, TYPEE, TYPEM, H) \ | ||
138 | -void HELPER(NAME)(CPUARMState *env, void *vg, \ | ||
139 | - target_ulong addr, uint32_t desc) \ | ||
140 | -{ \ | ||
141 | - intptr_t i, oprsz = simd_oprsz(desc); \ | ||
142 | - intptr_t ra = GETPC(); \ | ||
143 | - unsigned rd = simd_data(desc); \ | ||
144 | - void *d1 = &env->vfp.zregs[rd]; \ | ||
145 | - void *d2 = &env->vfp.zregs[(rd + 1) & 31]; \ | ||
146 | - void *d3 = &env->vfp.zregs[(rd + 2) & 31]; \ | ||
147 | - void *d4 = &env->vfp.zregs[(rd + 3) & 31]; \ | ||
148 | - for (i = 0; i < oprsz; ) { \ | ||
149 | - uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); \ | ||
150 | - do { \ | ||
151 | - TYPEM m1 = 0, m2 = 0, m3 = 0, m4 = 0; \ | ||
152 | - if (pg & 1) { \ | ||
153 | - m1 = FN(env, addr, ra); \ | ||
154 | - m2 = FN(env, addr + sizeof(TYPEM), ra); \ | ||
155 | - m3 = FN(env, addr + 2 * sizeof(TYPEM), ra); \ | ||
156 | - m4 = FN(env, addr + 3 * sizeof(TYPEM), ra); \ | ||
157 | - } \ | ||
158 | - *(TYPEE *)(d1 + H(i)) = m1; \ | ||
159 | - *(TYPEE *)(d2 + H(i)) = m2; \ | ||
160 | - *(TYPEE *)(d3 + H(i)) = m3; \ | ||
161 | - *(TYPEE *)(d4 + H(i)) = m4; \ | ||
162 | - i += sizeof(TYPEE), pg >>= sizeof(TYPEE); \ | ||
163 | - addr += 4 * sizeof(TYPEM); \ | ||
164 | - } while (i & 15); \ | ||
165 | - } \ | ||
166 | +static void sve_ld4_r(CPUARMState *env, void *vg, target_ulong addr, | ||
167 | + uint32_t desc, int size, uintptr_t ra, | ||
168 | + sve_ld1_tlb_fn *tlb_fn) | ||
169 | +{ | ||
170 | + const int mmu_idx = cpu_mmu_index(env, false); | ||
171 | + intptr_t i, oprsz = simd_oprsz(desc); | ||
172 | + unsigned rd = simd_data(desc); | ||
173 | + ARMVectorReg scratch[4] = { }; | ||
174 | + | ||
175 | + set_helper_retaddr(ra); | ||
176 | + for (i = 0; i < oprsz; ) { | ||
177 | + uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); | ||
178 | + do { | ||
179 | + if (pg & 1) { | ||
180 | + tlb_fn(env, &scratch[0], i, addr, mmu_idx, ra); | ||
181 | + tlb_fn(env, &scratch[1], i, addr + size, mmu_idx, ra); | ||
182 | + tlb_fn(env, &scratch[2], i, addr + 2 * size, mmu_idx, ra); | ||
183 | + tlb_fn(env, &scratch[3], i, addr + 3 * size, mmu_idx, ra); | ||
184 | + } | ||
185 | + i += size, pg >>= size; | ||
186 | + addr += 4 * size; | ||
187 | + } while (i & 15); | ||
188 | + } | ||
189 | + set_helper_retaddr(0); | ||
190 | + | ||
191 | + /* Wait until all exceptions have been raised to write back. */ | ||
192 | + memcpy(&env->vfp.zregs[rd], &scratch[0], oprsz); | ||
193 | + memcpy(&env->vfp.zregs[(rd + 1) & 31], &scratch[1], oprsz); | ||
194 | + memcpy(&env->vfp.zregs[(rd + 2) & 31], &scratch[2], oprsz); | ||
195 | + memcpy(&env->vfp.zregs[(rd + 3) & 31], &scratch[3], oprsz); | ||
196 | } | ||
197 | |||
198 | -DO_LD2(sve_ld2bb_r, cpu_ldub_data_ra, uint8_t, uint8_t, H1) | ||
199 | -DO_LD3(sve_ld3bb_r, cpu_ldub_data_ra, uint8_t, uint8_t, H1) | ||
200 | -DO_LD4(sve_ld4bb_r, cpu_ldub_data_ra, uint8_t, uint8_t, H1) | ||
201 | +#define DO_LDN_1(N) \ | ||
202 | +void __attribute__((flatten)) HELPER(sve_ld##N##bb_r) \ | ||
203 | + (CPUARMState *env, void *vg, target_ulong addr, uint32_t desc) \ | ||
204 | +{ \ | ||
205 | + sve_ld##N##_r(env, vg, addr, desc, 1, GETPC(), sve_ld1bb_tlb); \ | ||
206 | +} | ||
207 | |||
208 | -DO_LD2(sve_ld2hh_r, cpu_lduw_data_ra, uint16_t, uint16_t, H1_2) | ||
209 | -DO_LD3(sve_ld3hh_r, cpu_lduw_data_ra, uint16_t, uint16_t, H1_2) | ||
210 | -DO_LD4(sve_ld4hh_r, cpu_lduw_data_ra, uint16_t, uint16_t, H1_2) | ||
211 | +#define DO_LDN_2(N, SUFF, SIZE) \ | ||
212 | +void __attribute__((flatten)) HELPER(sve_ld##N##SUFF##_r) \ | ||
213 | + (CPUARMState *env, void *vg, target_ulong addr, uint32_t desc) \ | ||
214 | +{ \ | ||
215 | + sve_ld##N##_r(env, vg, addr, desc, SIZE, GETPC(), \ | ||
216 | + arm_cpu_data_is_big_endian(env) \ | ||
217 | + ? sve_ld1##SUFF##_be_tlb : sve_ld1##SUFF##_le_tlb); \ | ||
218 | +} | ||
219 | |||
220 | -DO_LD2(sve_ld2ss_r, cpu_ldl_data_ra, uint32_t, uint32_t, H1_4) | ||
221 | -DO_LD3(sve_ld3ss_r, cpu_ldl_data_ra, uint32_t, uint32_t, H1_4) | ||
222 | -DO_LD4(sve_ld4ss_r, cpu_ldl_data_ra, uint32_t, uint32_t, H1_4) | ||
223 | +DO_LDN_1(2) | ||
224 | +DO_LDN_1(3) | ||
225 | +DO_LDN_1(4) | ||
226 | |||
227 | -DO_LD2(sve_ld2dd_r, cpu_ldq_data_ra, uint64_t, uint64_t, ) | ||
228 | -DO_LD3(sve_ld3dd_r, cpu_ldq_data_ra, uint64_t, uint64_t, ) | ||
229 | -DO_LD4(sve_ld4dd_r, cpu_ldq_data_ra, uint64_t, uint64_t, ) | ||
230 | +DO_LDN_2(2, hh, 2) | ||
231 | +DO_LDN_2(3, hh, 2) | ||
232 | +DO_LDN_2(4, hh, 2) | ||
233 | |||
234 | -#undef DO_LD2 | ||
235 | -#undef DO_LD3 | ||
236 | -#undef DO_LD4 | ||
237 | +DO_LDN_2(2, ss, 4) | ||
238 | +DO_LDN_2(3, ss, 4) | ||
239 | +DO_LDN_2(4, ss, 4) | ||
240 | + | ||
241 | +DO_LDN_2(2, dd, 8) | ||
242 | +DO_LDN_2(3, dd, 8) | ||
243 | +DO_LDN_2(4, dd, 8) | ||
244 | + | ||
245 | +#undef DO_LDN_1 | ||
246 | +#undef DO_LDN_2 | ||
247 | |||
248 | /* | ||
249 | * Load contiguous data, first-fault and no-fault. | ||
250 | -- | ||
251 | 2.19.0 | ||
252 | |||
253 | diff view generated by jsdifflib |
Deleted patch | |||
---|---|---|---|
1 | From: Richard Henderson <richard.henderson@linaro.org> | ||
2 | 1 | ||
3 | This fixes the endianness problem for softmmu, and moves the | ||
4 | main loop out of a macro and into an inlined function. | ||
5 | |||
6 | Reviewed-by: Peter Maydell <peter.maydell@linaro.org> | ||
7 | Tested-by: Laurent Desnogues <laurent.desnogues@gmail.com> | ||
8 | Signed-off-by: Richard Henderson <richard.henderson@linaro.org> | ||
9 | Message-id: 20181005175350.30752-10-richard.henderson@linaro.org | ||
10 | Signed-off-by: Peter Maydell <peter.maydell@linaro.org> | ||
11 | --- | ||
12 | target/arm/sve_helper.c | 351 ++++++++++++++++++++-------------------- | ||
13 | 1 file changed, 172 insertions(+), 179 deletions(-) | ||
14 | |||
15 | diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c | ||
16 | index XXXXXXX..XXXXXXX 100644 | ||
17 | --- a/target/arm/sve_helper.c | ||
18 | +++ b/target/arm/sve_helper.c | ||
19 | @@ -XXX,XX +XXX,XX @@ typedef intptr_t sve_ld1_host_fn(void *vd, void *vg, void *host, | ||
20 | */ | ||
21 | typedef void sve_ld1_tlb_fn(CPUARMState *env, void *vd, intptr_t reg_off, | ||
22 | target_ulong vaddr, int mmu_idx, uintptr_t ra); | ||
23 | +typedef sve_ld1_tlb_fn sve_st1_tlb_fn; | ||
24 | |||
25 | /* | ||
26 | * Generate the above primitives. | ||
27 | @@ -XXX,XX +XXX,XX @@ DO_LDFF1_LDNF1_2(dd, 3, 3) | ||
28 | /* | ||
29 | * Store contiguous data, protected by a governing predicate. | ||
30 | */ | ||
31 | -#define DO_ST1(NAME, FN, TYPEE, TYPEM, H) \ | ||
32 | -void HELPER(NAME)(CPUARMState *env, void *vg, \ | ||
33 | - target_ulong addr, uint32_t desc) \ | ||
34 | -{ \ | ||
35 | - intptr_t i, oprsz = simd_oprsz(desc); \ | ||
36 | - intptr_t ra = GETPC(); \ | ||
37 | - unsigned rd = simd_data(desc); \ | ||
38 | - void *vd = &env->vfp.zregs[rd]; \ | ||
39 | - for (i = 0; i < oprsz; ) { \ | ||
40 | - uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); \ | ||
41 | - do { \ | ||
42 | - if (pg & 1) { \ | ||
43 | - TYPEM m = *(TYPEE *)(vd + H(i)); \ | ||
44 | - FN(env, addr, m, ra); \ | ||
45 | - } \ | ||
46 | - i += sizeof(TYPEE), pg >>= sizeof(TYPEE); \ | ||
47 | - addr += sizeof(TYPEM); \ | ||
48 | - } while (i & 15); \ | ||
49 | - } \ | ||
50 | + | ||
51 | +#ifdef CONFIG_SOFTMMU | ||
52 | +#define DO_ST_TLB(NAME, H, TYPEM, HOST, MOEND, TLB) \ | ||
53 | +static void sve_##NAME##_tlb(CPUARMState *env, void *vd, intptr_t reg_off, \ | ||
54 | + target_ulong addr, int mmu_idx, uintptr_t ra) \ | ||
55 | +{ \ | ||
56 | + TCGMemOpIdx oi = make_memop_idx(ctz32(sizeof(TYPEM)) | MOEND, mmu_idx); \ | ||
57 | + TLB(env, addr, *(TYPEM *)(vd + H(reg_off)), oi, ra); \ | ||
58 | } | ||
59 | - | ||
60 | -#define DO_ST1_D(NAME, FN, TYPEM) \ | ||
61 | -void HELPER(NAME)(CPUARMState *env, void *vg, \ | ||
62 | - target_ulong addr, uint32_t desc) \ | ||
63 | -{ \ | ||
64 | - intptr_t i, oprsz = simd_oprsz(desc) / 8; \ | ||
65 | - intptr_t ra = GETPC(); \ | ||
66 | - unsigned rd = simd_data(desc); \ | ||
67 | - uint64_t *d = &env->vfp.zregs[rd].d[0]; \ | ||
68 | - uint8_t *pg = vg; \ | ||
69 | - for (i = 0; i < oprsz; i += 1) { \ | ||
70 | - if (pg[H1(i)] & 1) { \ | ||
71 | - FN(env, addr, d[i], ra); \ | ||
72 | - } \ | ||
73 | - addr += sizeof(TYPEM); \ | ||
74 | - } \ | ||
75 | +#else | ||
76 | +#define DO_ST_TLB(NAME, H, TYPEM, HOST, MOEND, TLB) \ | ||
77 | +static void sve_##NAME##_tlb(CPUARMState *env, void *vd, intptr_t reg_off, \ | ||
78 | + target_ulong addr, int mmu_idx, uintptr_t ra) \ | ||
79 | +{ \ | ||
80 | + HOST(g2h(addr), *(TYPEM *)(vd + H(reg_off))); \ | ||
81 | } | ||
82 | +#endif | ||
83 | |||
84 | -#define DO_ST2(NAME, FN, TYPEE, TYPEM, H) \ | ||
85 | -void HELPER(NAME)(CPUARMState *env, void *vg, \ | ||
86 | - target_ulong addr, uint32_t desc) \ | ||
87 | -{ \ | ||
88 | - intptr_t i, oprsz = simd_oprsz(desc); \ | ||
89 | - intptr_t ra = GETPC(); \ | ||
90 | - unsigned rd = simd_data(desc); \ | ||
91 | - void *d1 = &env->vfp.zregs[rd]; \ | ||
92 | - void *d2 = &env->vfp.zregs[(rd + 1) & 31]; \ | ||
93 | - for (i = 0; i < oprsz; ) { \ | ||
94 | - uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); \ | ||
95 | - do { \ | ||
96 | - if (pg & 1) { \ | ||
97 | - TYPEM m1 = *(TYPEE *)(d1 + H(i)); \ | ||
98 | - TYPEM m2 = *(TYPEE *)(d2 + H(i)); \ | ||
99 | - FN(env, addr, m1, ra); \ | ||
100 | - FN(env, addr + sizeof(TYPEM), m2, ra); \ | ||
101 | - } \ | ||
102 | - i += sizeof(TYPEE), pg >>= sizeof(TYPEE); \ | ||
103 | - addr += 2 * sizeof(TYPEM); \ | ||
104 | - } while (i & 15); \ | ||
105 | - } \ | ||
106 | -} | ||
107 | +DO_ST_TLB(st1bb, H1, uint8_t, stb_p, 0, helper_ret_stb_mmu) | ||
108 | +DO_ST_TLB(st1bh, H1_2, uint16_t, stb_p, 0, helper_ret_stb_mmu) | ||
109 | +DO_ST_TLB(st1bs, H1_4, uint32_t, stb_p, 0, helper_ret_stb_mmu) | ||
110 | +DO_ST_TLB(st1bd, , uint64_t, stb_p, 0, helper_ret_stb_mmu) | ||
111 | |||
112 | -#define DO_ST3(NAME, FN, TYPEE, TYPEM, H) \ | ||
113 | -void HELPER(NAME)(CPUARMState *env, void *vg, \ | ||
114 | - target_ulong addr, uint32_t desc) \ | ||
115 | -{ \ | ||
116 | - intptr_t i, oprsz = simd_oprsz(desc); \ | ||
117 | - intptr_t ra = GETPC(); \ | ||
118 | - unsigned rd = simd_data(desc); \ | ||
119 | - void *d1 = &env->vfp.zregs[rd]; \ | ||
120 | - void *d2 = &env->vfp.zregs[(rd + 1) & 31]; \ | ||
121 | - void *d3 = &env->vfp.zregs[(rd + 2) & 31]; \ | ||
122 | - for (i = 0; i < oprsz; ) { \ | ||
123 | - uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); \ | ||
124 | - do { \ | ||
125 | - if (pg & 1) { \ | ||
126 | - TYPEM m1 = *(TYPEE *)(d1 + H(i)); \ | ||
127 | - TYPEM m2 = *(TYPEE *)(d2 + H(i)); \ | ||
128 | - TYPEM m3 = *(TYPEE *)(d3 + H(i)); \ | ||
129 | - FN(env, addr, m1, ra); \ | ||
130 | - FN(env, addr + sizeof(TYPEM), m2, ra); \ | ||
131 | - FN(env, addr + 2 * sizeof(TYPEM), m3, ra); \ | ||
132 | - } \ | ||
133 | - i += sizeof(TYPEE), pg >>= sizeof(TYPEE); \ | ||
134 | - addr += 3 * sizeof(TYPEM); \ | ||
135 | - } while (i & 15); \ | ||
136 | - } \ | ||
137 | -} | ||
138 | +DO_ST_TLB(st1hh_le, H1_2, uint16_t, stw_le_p, MO_LE, helper_le_stw_mmu) | ||
139 | +DO_ST_TLB(st1hs_le, H1_4, uint32_t, stw_le_p, MO_LE, helper_le_stw_mmu) | ||
140 | +DO_ST_TLB(st1hd_le, , uint64_t, stw_le_p, MO_LE, helper_le_stw_mmu) | ||
141 | |||
142 | -#define DO_ST4(NAME, FN, TYPEE, TYPEM, H) \ | ||
143 | -void HELPER(NAME)(CPUARMState *env, void *vg, \ | ||
144 | - target_ulong addr, uint32_t desc) \ | ||
145 | -{ \ | ||
146 | - intptr_t i, oprsz = simd_oprsz(desc); \ | ||
147 | - intptr_t ra = GETPC(); \ | ||
148 | - unsigned rd = simd_data(desc); \ | ||
149 | - void *d1 = &env->vfp.zregs[rd]; \ | ||
150 | - void *d2 = &env->vfp.zregs[(rd + 1) & 31]; \ | ||
151 | - void *d3 = &env->vfp.zregs[(rd + 2) & 31]; \ | ||
152 | - void *d4 = &env->vfp.zregs[(rd + 3) & 31]; \ | ||
153 | - for (i = 0; i < oprsz; ) { \ | ||
154 | - uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); \ | ||
155 | - do { \ | ||
156 | - if (pg & 1) { \ | ||
157 | - TYPEM m1 = *(TYPEE *)(d1 + H(i)); \ | ||
158 | - TYPEM m2 = *(TYPEE *)(d2 + H(i)); \ | ||
159 | - TYPEM m3 = *(TYPEE *)(d3 + H(i)); \ | ||
160 | - TYPEM m4 = *(TYPEE *)(d4 + H(i)); \ | ||
161 | - FN(env, addr, m1, ra); \ | ||
162 | - FN(env, addr + sizeof(TYPEM), m2, ra); \ | ||
163 | - FN(env, addr + 2 * sizeof(TYPEM), m3, ra); \ | ||
164 | - FN(env, addr + 3 * sizeof(TYPEM), m4, ra); \ | ||
165 | - } \ | ||
166 | - i += sizeof(TYPEE), pg >>= sizeof(TYPEE); \ | ||
167 | - addr += 4 * sizeof(TYPEM); \ | ||
168 | - } while (i & 15); \ | ||
169 | - } \ | ||
170 | -} | ||
171 | +DO_ST_TLB(st1ss_le, H1_4, uint32_t, stl_le_p, MO_LE, helper_le_stl_mmu) | ||
172 | +DO_ST_TLB(st1sd_le, , uint64_t, stl_le_p, MO_LE, helper_le_stl_mmu) | ||
173 | |||
174 | -DO_ST1(sve_st1bh_r, cpu_stb_data_ra, uint16_t, uint8_t, H1_2) | ||
175 | -DO_ST1(sve_st1bs_r, cpu_stb_data_ra, uint32_t, uint8_t, H1_4) | ||
176 | -DO_ST1_D(sve_st1bd_r, cpu_stb_data_ra, uint8_t) | ||
177 | +DO_ST_TLB(st1dd_le, , uint64_t, stq_le_p, MO_LE, helper_le_stq_mmu) | ||
178 | |||
179 | -DO_ST1(sve_st1hs_r, cpu_stw_data_ra, uint32_t, uint16_t, H1_4) | ||
180 | -DO_ST1_D(sve_st1hd_r, cpu_stw_data_ra, uint16_t) | ||
181 | +DO_ST_TLB(st1hh_be, H1_2, uint16_t, stw_be_p, MO_BE, helper_be_stw_mmu) | ||
182 | +DO_ST_TLB(st1hs_be, H1_4, uint32_t, stw_be_p, MO_BE, helper_be_stw_mmu) | ||
183 | +DO_ST_TLB(st1hd_be, , uint64_t, stw_be_p, MO_BE, helper_be_stw_mmu) | ||
184 | |||
185 | -DO_ST1_D(sve_st1sd_r, cpu_stl_data_ra, uint32_t) | ||
186 | +DO_ST_TLB(st1ss_be, H1_4, uint32_t, stl_be_p, MO_BE, helper_be_stl_mmu) | ||
187 | +DO_ST_TLB(st1sd_be, , uint64_t, stl_be_p, MO_BE, helper_be_stl_mmu) | ||
188 | |||
189 | -DO_ST1(sve_st1bb_r, cpu_stb_data_ra, uint8_t, uint8_t, H1) | ||
190 | -DO_ST2(sve_st2bb_r, cpu_stb_data_ra, uint8_t, uint8_t, H1) | ||
191 | -DO_ST3(sve_st3bb_r, cpu_stb_data_ra, uint8_t, uint8_t, H1) | ||
192 | -DO_ST4(sve_st4bb_r, cpu_stb_data_ra, uint8_t, uint8_t, H1) | ||
193 | +DO_ST_TLB(st1dd_be, , uint64_t, stq_be_p, MO_BE, helper_be_stq_mmu) | ||
194 | |||
195 | -DO_ST1(sve_st1hh_r, cpu_stw_data_ra, uint16_t, uint16_t, H1_2) | ||
196 | -DO_ST2(sve_st2hh_r, cpu_stw_data_ra, uint16_t, uint16_t, H1_2) | ||
197 | -DO_ST3(sve_st3hh_r, cpu_stw_data_ra, uint16_t, uint16_t, H1_2) | ||
198 | -DO_ST4(sve_st4hh_r, cpu_stw_data_ra, uint16_t, uint16_t, H1_2) | ||
199 | +#undef DO_ST_TLB | ||
200 | |||
201 | -DO_ST1(sve_st1ss_r, cpu_stl_data_ra, uint32_t, uint32_t, H1_4) | ||
202 | -DO_ST2(sve_st2ss_r, cpu_stl_data_ra, uint32_t, uint32_t, H1_4) | ||
203 | -DO_ST3(sve_st3ss_r, cpu_stl_data_ra, uint32_t, uint32_t, H1_4) | ||
204 | -DO_ST4(sve_st4ss_r, cpu_stl_data_ra, uint32_t, uint32_t, H1_4) | ||
205 | - | ||
206 | -DO_ST1_D(sve_st1dd_r, cpu_stq_data_ra, uint64_t) | ||
207 | - | ||
208 | -void HELPER(sve_st2dd_r)(CPUARMState *env, void *vg, | ||
209 | - target_ulong addr, uint32_t desc) | ||
210 | +/* | ||
211 | + * Common helpers for all contiguous 1,2,3,4-register predicated stores. | ||
212 | + */ | ||
213 | +static void sve_st1_r(CPUARMState *env, void *vg, target_ulong addr, | ||
214 | + uint32_t desc, const uintptr_t ra, | ||
215 | + const int esize, const int msize, | ||
216 | + sve_st1_tlb_fn *tlb_fn) | ||
217 | { | ||
218 | - intptr_t i, oprsz = simd_oprsz(desc) / 8; | ||
219 | - intptr_t ra = GETPC(); | ||
220 | + const int mmu_idx = cpu_mmu_index(env, false); | ||
221 | + intptr_t i, oprsz = simd_oprsz(desc); | ||
222 | unsigned rd = simd_data(desc); | ||
223 | - uint64_t *d1 = &env->vfp.zregs[rd].d[0]; | ||
224 | - uint64_t *d2 = &env->vfp.zregs[(rd + 1) & 31].d[0]; | ||
225 | - uint8_t *pg = vg; | ||
226 | + void *vd = &env->vfp.zregs[rd]; | ||
227 | |||
228 | - for (i = 0; i < oprsz; i += 1) { | ||
229 | - if (pg[H1(i)] & 1) { | ||
230 | - cpu_stq_data_ra(env, addr, d1[i], ra); | ||
231 | - cpu_stq_data_ra(env, addr + 8, d2[i], ra); | ||
232 | - } | ||
233 | - addr += 2 * 8; | ||
234 | + set_helper_retaddr(ra); | ||
235 | + for (i = 0; i < oprsz; ) { | ||
236 | + uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); | ||
237 | + do { | ||
238 | + if (pg & 1) { | ||
239 | + tlb_fn(env, vd, i, addr, mmu_idx, ra); | ||
240 | + } | ||
241 | + i += esize, pg >>= esize; | ||
242 | + addr += msize; | ||
243 | + } while (i & 15); | ||
244 | } | ||
245 | + set_helper_retaddr(0); | ||
246 | } | ||
247 | |||
248 | -void HELPER(sve_st3dd_r)(CPUARMState *env, void *vg, | ||
249 | - target_ulong addr, uint32_t desc) | ||
250 | +static void sve_st2_r(CPUARMState *env, void *vg, target_ulong addr, | ||
251 | + uint32_t desc, const uintptr_t ra, | ||
252 | + const int esize, const int msize, | ||
253 | + sve_st1_tlb_fn *tlb_fn) | ||
254 | { | ||
255 | - intptr_t i, oprsz = simd_oprsz(desc) / 8; | ||
256 | - intptr_t ra = GETPC(); | ||
257 | + const int mmu_idx = cpu_mmu_index(env, false); | ||
258 | + intptr_t i, oprsz = simd_oprsz(desc); | ||
259 | unsigned rd = simd_data(desc); | ||
260 | - uint64_t *d1 = &env->vfp.zregs[rd].d[0]; | ||
261 | - uint64_t *d2 = &env->vfp.zregs[(rd + 1) & 31].d[0]; | ||
262 | - uint64_t *d3 = &env->vfp.zregs[(rd + 2) & 31].d[0]; | ||
263 | - uint8_t *pg = vg; | ||
264 | + void *d1 = &env->vfp.zregs[rd]; | ||
265 | + void *d2 = &env->vfp.zregs[(rd + 1) & 31]; | ||
266 | |||
267 | - for (i = 0; i < oprsz; i += 1) { | ||
268 | - if (pg[H1(i)] & 1) { | ||
269 | - cpu_stq_data_ra(env, addr, d1[i], ra); | ||
270 | - cpu_stq_data_ra(env, addr + 8, d2[i], ra); | ||
271 | - cpu_stq_data_ra(env, addr + 16, d3[i], ra); | ||
272 | - } | ||
273 | - addr += 3 * 8; | ||
274 | + set_helper_retaddr(ra); | ||
275 | + for (i = 0; i < oprsz; ) { | ||
276 | + uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); | ||
277 | + do { | ||
278 | + if (pg & 1) { | ||
279 | + tlb_fn(env, d1, i, addr, mmu_idx, ra); | ||
280 | + tlb_fn(env, d2, i, addr + msize, mmu_idx, ra); | ||
281 | + } | ||
282 | + i += esize, pg >>= esize; | ||
283 | + addr += 2 * msize; | ||
284 | + } while (i & 15); | ||
285 | } | ||
286 | + set_helper_retaddr(0); | ||
287 | } | ||
288 | |||
289 | -void HELPER(sve_st4dd_r)(CPUARMState *env, void *vg, | ||
290 | - target_ulong addr, uint32_t desc) | ||
291 | +static void sve_st3_r(CPUARMState *env, void *vg, target_ulong addr, | ||
292 | + uint32_t desc, const uintptr_t ra, | ||
293 | + const int esize, const int msize, | ||
294 | + sve_st1_tlb_fn *tlb_fn) | ||
295 | { | ||
296 | - intptr_t i, oprsz = simd_oprsz(desc) / 8; | ||
297 | - intptr_t ra = GETPC(); | ||
298 | + const int mmu_idx = cpu_mmu_index(env, false); | ||
299 | + intptr_t i, oprsz = simd_oprsz(desc); | ||
300 | unsigned rd = simd_data(desc); | ||
301 | - uint64_t *d1 = &env->vfp.zregs[rd].d[0]; | ||
302 | - uint64_t *d2 = &env->vfp.zregs[(rd + 1) & 31].d[0]; | ||
303 | - uint64_t *d3 = &env->vfp.zregs[(rd + 2) & 31].d[0]; | ||
304 | - uint64_t *d4 = &env->vfp.zregs[(rd + 3) & 31].d[0]; | ||
305 | - uint8_t *pg = vg; | ||
306 | + void *d1 = &env->vfp.zregs[rd]; | ||
307 | + void *d2 = &env->vfp.zregs[(rd + 1) & 31]; | ||
308 | + void *d3 = &env->vfp.zregs[(rd + 2) & 31]; | ||
309 | |||
310 | - for (i = 0; i < oprsz; i += 1) { | ||
311 | - if (pg[H1(i)] & 1) { | ||
312 | - cpu_stq_data_ra(env, addr, d1[i], ra); | ||
313 | - cpu_stq_data_ra(env, addr + 8, d2[i], ra); | ||
314 | - cpu_stq_data_ra(env, addr + 16, d3[i], ra); | ||
315 | - cpu_stq_data_ra(env, addr + 24, d4[i], ra); | ||
316 | - } | ||
317 | - addr += 4 * 8; | ||
318 | + set_helper_retaddr(ra); | ||
319 | + for (i = 0; i < oprsz; ) { | ||
320 | + uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); | ||
321 | + do { | ||
322 | + if (pg & 1) { | ||
323 | + tlb_fn(env, d1, i, addr, mmu_idx, ra); | ||
324 | + tlb_fn(env, d2, i, addr + msize, mmu_idx, ra); | ||
325 | + tlb_fn(env, d3, i, addr + 2 * msize, mmu_idx, ra); | ||
326 | + } | ||
327 | + i += esize, pg >>= esize; | ||
328 | + addr += 3 * msize; | ||
329 | + } while (i & 15); | ||
330 | } | ||
331 | + set_helper_retaddr(0); | ||
332 | } | ||
333 | |||
334 | +static void sve_st4_r(CPUARMState *env, void *vg, target_ulong addr, | ||
335 | + uint32_t desc, const uintptr_t ra, | ||
336 | + const int esize, const int msize, | ||
337 | + sve_st1_tlb_fn *tlb_fn) | ||
338 | +{ | ||
339 | + const int mmu_idx = cpu_mmu_index(env, false); | ||
340 | + intptr_t i, oprsz = simd_oprsz(desc); | ||
341 | + unsigned rd = simd_data(desc); | ||
342 | + void *d1 = &env->vfp.zregs[rd]; | ||
343 | + void *d2 = &env->vfp.zregs[(rd + 1) & 31]; | ||
344 | + void *d3 = &env->vfp.zregs[(rd + 2) & 31]; | ||
345 | + void *d4 = &env->vfp.zregs[(rd + 3) & 31]; | ||
346 | + | ||
347 | + set_helper_retaddr(ra); | ||
348 | + for (i = 0; i < oprsz; ) { | ||
349 | + uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); | ||
350 | + do { | ||
351 | + if (pg & 1) { | ||
352 | + tlb_fn(env, d1, i, addr, mmu_idx, ra); | ||
353 | + tlb_fn(env, d2, i, addr + msize, mmu_idx, ra); | ||
354 | + tlb_fn(env, d3, i, addr + 2 * msize, mmu_idx, ra); | ||
355 | + tlb_fn(env, d4, i, addr + 3 * msize, mmu_idx, ra); | ||
356 | + } | ||
357 | + i += esize, pg >>= esize; | ||
358 | + addr += 4 * msize; | ||
359 | + } while (i & 15); | ||
360 | + } | ||
361 | + set_helper_retaddr(0); | ||
362 | +} | ||
363 | + | ||
364 | +#define DO_STN_1(N, NAME, ESIZE) \ | ||
365 | +void __attribute__((flatten)) HELPER(sve_st##N##NAME##_r) \ | ||
366 | + (CPUARMState *env, void *vg, target_ulong addr, uint32_t desc) \ | ||
367 | +{ \ | ||
368 | + sve_st##N##_r(env, vg, addr, desc, GETPC(), ESIZE, 1, \ | ||
369 | + sve_st1##NAME##_tlb); \ | ||
370 | +} | ||
371 | + | ||
372 | +#define DO_STN_2(N, NAME, ESIZE, MSIZE) \ | ||
373 | +void __attribute__((flatten)) HELPER(sve_st##N##NAME##_r) \ | ||
374 | + (CPUARMState *env, void *vg, target_ulong addr, uint32_t desc) \ | ||
375 | +{ \ | ||
376 | + sve_st##N##_r(env, vg, addr, desc, GETPC(), ESIZE, MSIZE, \ | ||
377 | + arm_cpu_data_is_big_endian(env) \ | ||
378 | + ? sve_st1##NAME##_be_tlb : sve_st1##NAME##_le_tlb); \ | ||
379 | +} | ||
380 | + | ||
381 | +DO_STN_1(1, bb, 1) | ||
382 | +DO_STN_1(1, bh, 2) | ||
383 | +DO_STN_1(1, bs, 4) | ||
384 | +DO_STN_1(1, bd, 8) | ||
385 | +DO_STN_1(2, bb, 1) | ||
386 | +DO_STN_1(3, bb, 1) | ||
387 | +DO_STN_1(4, bb, 1) | ||
388 | + | ||
389 | +DO_STN_2(1, hh, 2, 2) | ||
390 | +DO_STN_2(1, hs, 4, 2) | ||
391 | +DO_STN_2(1, hd, 8, 2) | ||
392 | +DO_STN_2(2, hh, 2, 2) | ||
393 | +DO_STN_2(3, hh, 2, 2) | ||
394 | +DO_STN_2(4, hh, 2, 2) | ||
395 | + | ||
396 | +DO_STN_2(1, ss, 4, 4) | ||
397 | +DO_STN_2(1, sd, 8, 4) | ||
398 | +DO_STN_2(2, ss, 4, 4) | ||
399 | +DO_STN_2(3, ss, 4, 4) | ||
400 | +DO_STN_2(4, ss, 4, 4) | ||
401 | + | ||
402 | +DO_STN_2(1, dd, 8, 8) | ||
403 | +DO_STN_2(2, dd, 8, 8) | ||
404 | +DO_STN_2(3, dd, 8, 8) | ||
405 | +DO_STN_2(4, dd, 8, 8) | ||
406 | + | ||
407 | +#undef DO_STN_1 | ||
408 | +#undef DO_STN_2 | ||
409 | + | ||
410 | /* Loads with a vector index. */ | ||
411 | |||
412 | #define DO_LD1_ZPZ_S(NAME, TYPEI, TYPEM, FN) \ | ||
413 | -- | ||
414 | 2.19.0 | ||
415 | |||
416 | diff view generated by jsdifflib |
Deleted patch | |||
---|---|---|---|
1 | From: Richard Henderson <richard.henderson@linaro.org> | ||
2 | 1 | ||
3 | We can choose the endianness at translation time, rather than | ||
4 | re-computing it at execution time. | ||
5 | |||
6 | Tested-by: Laurent Desnogues <laurent.desnogues@gmail.com> | ||
7 | Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org> | ||
8 | Signed-off-by: Richard Henderson <richard.henderson@linaro.org> | ||
9 | Message-id: 20181005175350.30752-11-richard.henderson@linaro.org | ||
10 | Signed-off-by: Peter Maydell <peter.maydell@linaro.org> | ||
11 | --- | ||
12 | target/arm/helper-sve.h | 117 +++++++++++++++------- | ||
13 | target/arm/sve_helper.c | 70 ++++++------- | ||
14 | target/arm/translate-sve.c | 196 +++++++++++++++++++++++++------------ | ||
15 | 3 files changed, 252 insertions(+), 131 deletions(-) | ||
16 | |||
17 | diff --git a/target/arm/helper-sve.h b/target/arm/helper-sve.h | ||
18 | index XXXXXXX..XXXXXXX 100644 | ||
19 | --- a/target/arm/helper-sve.h | ||
20 | +++ b/target/arm/helper-sve.h | ||
21 | @@ -XXX,XX +XXX,XX @@ DEF_HELPER_FLAGS_4(sve_ld2bb_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
22 | DEF_HELPER_FLAGS_4(sve_ld3bb_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
23 | DEF_HELPER_FLAGS_4(sve_ld4bb_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
24 | |||
25 | -DEF_HELPER_FLAGS_4(sve_ld1hh_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
26 | -DEF_HELPER_FLAGS_4(sve_ld2hh_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
27 | -DEF_HELPER_FLAGS_4(sve_ld3hh_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
28 | -DEF_HELPER_FLAGS_4(sve_ld4hh_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
29 | +DEF_HELPER_FLAGS_4(sve_ld1hh_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
30 | +DEF_HELPER_FLAGS_4(sve_ld2hh_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
31 | +DEF_HELPER_FLAGS_4(sve_ld3hh_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
32 | +DEF_HELPER_FLAGS_4(sve_ld4hh_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
33 | |||
34 | -DEF_HELPER_FLAGS_4(sve_ld1ss_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
35 | -DEF_HELPER_FLAGS_4(sve_ld2ss_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
36 | -DEF_HELPER_FLAGS_4(sve_ld3ss_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
37 | -DEF_HELPER_FLAGS_4(sve_ld4ss_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
38 | +DEF_HELPER_FLAGS_4(sve_ld1hh_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
39 | +DEF_HELPER_FLAGS_4(sve_ld2hh_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
40 | +DEF_HELPER_FLAGS_4(sve_ld3hh_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
41 | +DEF_HELPER_FLAGS_4(sve_ld4hh_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
42 | |||
43 | -DEF_HELPER_FLAGS_4(sve_ld1dd_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
44 | -DEF_HELPER_FLAGS_4(sve_ld2dd_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
45 | -DEF_HELPER_FLAGS_4(sve_ld3dd_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
46 | -DEF_HELPER_FLAGS_4(sve_ld4dd_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
47 | +DEF_HELPER_FLAGS_4(sve_ld1ss_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
48 | +DEF_HELPER_FLAGS_4(sve_ld2ss_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
49 | +DEF_HELPER_FLAGS_4(sve_ld3ss_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
50 | +DEF_HELPER_FLAGS_4(sve_ld4ss_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
51 | + | ||
52 | +DEF_HELPER_FLAGS_4(sve_ld1ss_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
53 | +DEF_HELPER_FLAGS_4(sve_ld2ss_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
54 | +DEF_HELPER_FLAGS_4(sve_ld3ss_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
55 | +DEF_HELPER_FLAGS_4(sve_ld4ss_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
56 | + | ||
57 | +DEF_HELPER_FLAGS_4(sve_ld1dd_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
58 | +DEF_HELPER_FLAGS_4(sve_ld2dd_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
59 | +DEF_HELPER_FLAGS_4(sve_ld3dd_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
60 | +DEF_HELPER_FLAGS_4(sve_ld4dd_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
61 | + | ||
62 | +DEF_HELPER_FLAGS_4(sve_ld1dd_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
63 | +DEF_HELPER_FLAGS_4(sve_ld2dd_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
64 | +DEF_HELPER_FLAGS_4(sve_ld3dd_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
65 | +DEF_HELPER_FLAGS_4(sve_ld4dd_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
66 | |||
67 | DEF_HELPER_FLAGS_4(sve_ld1bhu_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
68 | DEF_HELPER_FLAGS_4(sve_ld1bsu_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
69 | @@ -XXX,XX +XXX,XX @@ DEF_HELPER_FLAGS_4(sve_ld1bhs_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
70 | DEF_HELPER_FLAGS_4(sve_ld1bss_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
71 | DEF_HELPER_FLAGS_4(sve_ld1bds_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
72 | |||
73 | -DEF_HELPER_FLAGS_4(sve_ld1hsu_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
74 | -DEF_HELPER_FLAGS_4(sve_ld1hdu_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
75 | -DEF_HELPER_FLAGS_4(sve_ld1hss_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
76 | -DEF_HELPER_FLAGS_4(sve_ld1hds_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
77 | +DEF_HELPER_FLAGS_4(sve_ld1hsu_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
78 | +DEF_HELPER_FLAGS_4(sve_ld1hdu_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
79 | +DEF_HELPER_FLAGS_4(sve_ld1hss_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
80 | +DEF_HELPER_FLAGS_4(sve_ld1hds_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
81 | |||
82 | -DEF_HELPER_FLAGS_4(sve_ld1sdu_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
83 | -DEF_HELPER_FLAGS_4(sve_ld1sds_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
84 | +DEF_HELPER_FLAGS_4(sve_ld1hsu_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
85 | +DEF_HELPER_FLAGS_4(sve_ld1hdu_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
86 | +DEF_HELPER_FLAGS_4(sve_ld1hss_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
87 | +DEF_HELPER_FLAGS_4(sve_ld1hds_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
88 | + | ||
89 | +DEF_HELPER_FLAGS_4(sve_ld1sdu_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
90 | +DEF_HELPER_FLAGS_4(sve_ld1sds_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
91 | + | ||
92 | +DEF_HELPER_FLAGS_4(sve_ld1sdu_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
93 | +DEF_HELPER_FLAGS_4(sve_ld1sds_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
94 | |||
95 | DEF_HELPER_FLAGS_4(sve_ldff1bb_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
96 | DEF_HELPER_FLAGS_4(sve_ldff1bhu_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
97 | @@ -XXX,XX +XXX,XX @@ DEF_HELPER_FLAGS_4(sve_ldff1bhs_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
98 | DEF_HELPER_FLAGS_4(sve_ldff1bss_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
99 | DEF_HELPER_FLAGS_4(sve_ldff1bds_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
100 | |||
101 | -DEF_HELPER_FLAGS_4(sve_ldff1hh_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
102 | -DEF_HELPER_FLAGS_4(sve_ldff1hsu_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
103 | -DEF_HELPER_FLAGS_4(sve_ldff1hdu_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
104 | -DEF_HELPER_FLAGS_4(sve_ldff1hss_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
105 | -DEF_HELPER_FLAGS_4(sve_ldff1hds_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
106 | +DEF_HELPER_FLAGS_4(sve_ldff1hh_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
107 | +DEF_HELPER_FLAGS_4(sve_ldff1hsu_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
108 | +DEF_HELPER_FLAGS_4(sve_ldff1hdu_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
109 | +DEF_HELPER_FLAGS_4(sve_ldff1hss_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
110 | +DEF_HELPER_FLAGS_4(sve_ldff1hds_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
111 | |||
112 | -DEF_HELPER_FLAGS_4(sve_ldff1ss_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
113 | -DEF_HELPER_FLAGS_4(sve_ldff1sdu_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
114 | -DEF_HELPER_FLAGS_4(sve_ldff1sds_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
115 | +DEF_HELPER_FLAGS_4(sve_ldff1hh_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
116 | +DEF_HELPER_FLAGS_4(sve_ldff1hsu_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
117 | +DEF_HELPER_FLAGS_4(sve_ldff1hdu_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
118 | +DEF_HELPER_FLAGS_4(sve_ldff1hss_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
119 | +DEF_HELPER_FLAGS_4(sve_ldff1hds_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
120 | |||
121 | -DEF_HELPER_FLAGS_4(sve_ldff1dd_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
122 | +DEF_HELPER_FLAGS_4(sve_ldff1ss_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
123 | +DEF_HELPER_FLAGS_4(sve_ldff1sdu_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
124 | +DEF_HELPER_FLAGS_4(sve_ldff1sds_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
125 | + | ||
126 | +DEF_HELPER_FLAGS_4(sve_ldff1ss_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
127 | +DEF_HELPER_FLAGS_4(sve_ldff1sdu_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
128 | +DEF_HELPER_FLAGS_4(sve_ldff1sds_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
129 | + | ||
130 | +DEF_HELPER_FLAGS_4(sve_ldff1dd_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
131 | +DEF_HELPER_FLAGS_4(sve_ldff1dd_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
132 | |||
133 | DEF_HELPER_FLAGS_4(sve_ldnf1bb_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
134 | DEF_HELPER_FLAGS_4(sve_ldnf1bhu_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
135 | @@ -XXX,XX +XXX,XX @@ DEF_HELPER_FLAGS_4(sve_ldnf1bhs_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
136 | DEF_HELPER_FLAGS_4(sve_ldnf1bss_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
137 | DEF_HELPER_FLAGS_4(sve_ldnf1bds_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
138 | |||
139 | -DEF_HELPER_FLAGS_4(sve_ldnf1hh_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
140 | -DEF_HELPER_FLAGS_4(sve_ldnf1hsu_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
141 | -DEF_HELPER_FLAGS_4(sve_ldnf1hdu_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
142 | -DEF_HELPER_FLAGS_4(sve_ldnf1hss_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
143 | -DEF_HELPER_FLAGS_4(sve_ldnf1hds_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
144 | +DEF_HELPER_FLAGS_4(sve_ldnf1hh_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
145 | +DEF_HELPER_FLAGS_4(sve_ldnf1hsu_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
146 | +DEF_HELPER_FLAGS_4(sve_ldnf1hdu_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
147 | +DEF_HELPER_FLAGS_4(sve_ldnf1hss_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
148 | +DEF_HELPER_FLAGS_4(sve_ldnf1hds_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
149 | |||
150 | -DEF_HELPER_FLAGS_4(sve_ldnf1ss_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
151 | -DEF_HELPER_FLAGS_4(sve_ldnf1sdu_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
152 | -DEF_HELPER_FLAGS_4(sve_ldnf1sds_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
153 | +DEF_HELPER_FLAGS_4(sve_ldnf1hh_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
154 | +DEF_HELPER_FLAGS_4(sve_ldnf1hsu_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
155 | +DEF_HELPER_FLAGS_4(sve_ldnf1hdu_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
156 | +DEF_HELPER_FLAGS_4(sve_ldnf1hss_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
157 | +DEF_HELPER_FLAGS_4(sve_ldnf1hds_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
158 | |||
159 | -DEF_HELPER_FLAGS_4(sve_ldnf1dd_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
160 | +DEF_HELPER_FLAGS_4(sve_ldnf1ss_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
161 | +DEF_HELPER_FLAGS_4(sve_ldnf1sdu_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
162 | +DEF_HELPER_FLAGS_4(sve_ldnf1sds_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
163 | + | ||
164 | +DEF_HELPER_FLAGS_4(sve_ldnf1ss_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
165 | +DEF_HELPER_FLAGS_4(sve_ldnf1sdu_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
166 | +DEF_HELPER_FLAGS_4(sve_ldnf1sds_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
167 | + | ||
168 | +DEF_HELPER_FLAGS_4(sve_ldnf1dd_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
169 | +DEF_HELPER_FLAGS_4(sve_ldnf1dd_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
170 | |||
171 | DEF_HELPER_FLAGS_4(sve_st1bb_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
172 | DEF_HELPER_FLAGS_4(sve_st2bb_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
173 | diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c | ||
174 | index XXXXXXX..XXXXXXX 100644 | ||
175 | --- a/target/arm/sve_helper.c | ||
176 | +++ b/target/arm/sve_helper.c | ||
177 | @@ -XXX,XX +XXX,XX @@ void HELPER(sve_##NAME##_r)(CPUARMState *env, void *vg, \ | ||
178 | sve_##NAME##_host, sve_##NAME##_tlb); \ | ||
179 | } | ||
180 | |||
181 | -/* TODO: Propagate the endian check back to the translator. */ | ||
182 | #define DO_LD1_2(NAME, ESZ, MSZ) \ | ||
183 | -void HELPER(sve_##NAME##_r)(CPUARMState *env, void *vg, \ | ||
184 | - target_ulong addr, uint32_t desc) \ | ||
185 | -{ \ | ||
186 | - if (arm_cpu_data_is_big_endian(env)) { \ | ||
187 | - sve_ld1_r(env, vg, addr, desc, GETPC(), ESZ, MSZ, \ | ||
188 | - sve_##NAME##_be_host, sve_##NAME##_be_tlb); \ | ||
189 | - } else { \ | ||
190 | - sve_ld1_r(env, vg, addr, desc, GETPC(), ESZ, MSZ, \ | ||
191 | - sve_##NAME##_le_host, sve_##NAME##_le_tlb); \ | ||
192 | - } \ | ||
193 | +void HELPER(sve_##NAME##_le_r)(CPUARMState *env, void *vg, \ | ||
194 | + target_ulong addr, uint32_t desc) \ | ||
195 | +{ \ | ||
196 | + sve_ld1_r(env, vg, addr, desc, GETPC(), ESZ, MSZ, \ | ||
197 | + sve_##NAME##_le_host, sve_##NAME##_le_tlb); \ | ||
198 | +} \ | ||
199 | +void HELPER(sve_##NAME##_be_r)(CPUARMState *env, void *vg, \ | ||
200 | + target_ulong addr, uint32_t desc) \ | ||
201 | +{ \ | ||
202 | + sve_ld1_r(env, vg, addr, desc, GETPC(), ESZ, MSZ, \ | ||
203 | + sve_##NAME##_be_host, sve_##NAME##_be_tlb); \ | ||
204 | } | ||
205 | |||
206 | DO_LD1_1(ld1bb, 0) | ||
207 | @@ -XXX,XX +XXX,XX @@ void __attribute__((flatten)) HELPER(sve_ld##N##bb_r) \ | ||
208 | } | ||
209 | |||
210 | #define DO_LDN_2(N, SUFF, SIZE) \ | ||
211 | -void __attribute__((flatten)) HELPER(sve_ld##N##SUFF##_r) \ | ||
212 | +void __attribute__((flatten)) HELPER(sve_ld##N##SUFF##_le_r) \ | ||
213 | (CPUARMState *env, void *vg, target_ulong addr, uint32_t desc) \ | ||
214 | { \ | ||
215 | sve_ld##N##_r(env, vg, addr, desc, SIZE, GETPC(), \ | ||
216 | - arm_cpu_data_is_big_endian(env) \ | ||
217 | - ? sve_ld1##SUFF##_be_tlb : sve_ld1##SUFF##_le_tlb); \ | ||
218 | + sve_ld1##SUFF##_le_tlb); \ | ||
219 | +} \ | ||
220 | +void __attribute__((flatten)) HELPER(sve_ld##N##SUFF##_be_r) \ | ||
221 | + (CPUARMState *env, void *vg, target_ulong addr, uint32_t desc) \ | ||
222 | +{ \ | ||
223 | + sve_ld##N##_r(env, vg, addr, desc, SIZE, GETPC(), \ | ||
224 | + sve_ld1##SUFF##_be_tlb); \ | ||
225 | } | ||
226 | |||
227 | DO_LDN_1(2) | ||
228 | @@ -XXX,XX +XXX,XX @@ void HELPER(sve_ldnf1##PART##_r)(CPUARMState *env, void *vg, \ | ||
229 | sve_ldnf1_r(env, vg, addr, desc, ESZ, 0, sve_ld1##PART##_host); \ | ||
230 | } | ||
231 | |||
232 | -/* TODO: Propagate the endian check back to the translator. */ | ||
233 | #define DO_LDFF1_LDNF1_2(PART, ESZ, MSZ) \ | ||
234 | -void HELPER(sve_ldff1##PART##_r)(CPUARMState *env, void *vg, \ | ||
235 | - target_ulong addr, uint32_t desc) \ | ||
236 | +void HELPER(sve_ldff1##PART##_le_r)(CPUARMState *env, void *vg, \ | ||
237 | + target_ulong addr, uint32_t desc) \ | ||
238 | { \ | ||
239 | - if (arm_cpu_data_is_big_endian(env)) { \ | ||
240 | - sve_ldff1_r(env, vg, addr, desc, GETPC(), ESZ, MSZ, \ | ||
241 | - sve_ld1##PART##_be_host, sve_ld1##PART##_be_tlb); \ | ||
242 | - } else { \ | ||
243 | - sve_ldff1_r(env, vg, addr, desc, GETPC(), ESZ, MSZ, \ | ||
244 | - sve_ld1##PART##_le_host, sve_ld1##PART##_le_tlb); \ | ||
245 | - } \ | ||
246 | + sve_ldff1_r(env, vg, addr, desc, GETPC(), ESZ, MSZ, \ | ||
247 | + sve_ld1##PART##_le_host, sve_ld1##PART##_le_tlb); \ | ||
248 | } \ | ||
249 | -void HELPER(sve_ldnf1##PART##_r)(CPUARMState *env, void *vg, \ | ||
250 | - target_ulong addr, uint32_t desc) \ | ||
251 | +void HELPER(sve_ldnf1##PART##_le_r)(CPUARMState *env, void *vg, \ | ||
252 | + target_ulong addr, uint32_t desc) \ | ||
253 | { \ | ||
254 | - if (arm_cpu_data_is_big_endian(env)) { \ | ||
255 | - sve_ldnf1_r(env, vg, addr, desc, ESZ, MSZ, \ | ||
256 | - sve_ld1##PART##_be_host); \ | ||
257 | - } else { \ | ||
258 | - sve_ldnf1_r(env, vg, addr, desc, ESZ, MSZ, \ | ||
259 | - sve_ld1##PART##_le_host); \ | ||
260 | - } \ | ||
261 | + sve_ldnf1_r(env, vg, addr, desc, ESZ, MSZ, sve_ld1##PART##_le_host); \ | ||
262 | +} \ | ||
263 | +void HELPER(sve_ldff1##PART##_be_r)(CPUARMState *env, void *vg, \ | ||
264 | + target_ulong addr, uint32_t desc) \ | ||
265 | +{ \ | ||
266 | + sve_ldff1_r(env, vg, addr, desc, GETPC(), ESZ, MSZ, \ | ||
267 | + sve_ld1##PART##_be_host, sve_ld1##PART##_be_tlb); \ | ||
268 | +} \ | ||
269 | +void HELPER(sve_ldnf1##PART##_be_r)(CPUARMState *env, void *vg, \ | ||
270 | + target_ulong addr, uint32_t desc) \ | ||
271 | +{ \ | ||
272 | + sve_ldnf1_r(env, vg, addr, desc, ESZ, MSZ, sve_ld1##PART##_be_host); \ | ||
273 | } | ||
274 | |||
275 | DO_LDFF1_LDNF1_1(bb, 0) | ||
276 | diff --git a/target/arm/translate-sve.c b/target/arm/translate-sve.c | ||
277 | index XXXXXXX..XXXXXXX 100644 | ||
278 | --- a/target/arm/translate-sve.c | ||
279 | +++ b/target/arm/translate-sve.c | ||
280 | @@ -XXX,XX +XXX,XX @@ static void do_mem_zpa(DisasContext *s, int zt, int pg, TCGv_i64 addr, | ||
281 | static void do_ld_zpa(DisasContext *s, int zt, int pg, | ||
282 | TCGv_i64 addr, int dtype, int nreg) | ||
283 | { | ||
284 | - static gen_helper_gvec_mem * const fns[16][4] = { | ||
285 | - { gen_helper_sve_ld1bb_r, gen_helper_sve_ld2bb_r, | ||
286 | - gen_helper_sve_ld3bb_r, gen_helper_sve_ld4bb_r }, | ||
287 | - { gen_helper_sve_ld1bhu_r, NULL, NULL, NULL }, | ||
288 | - { gen_helper_sve_ld1bsu_r, NULL, NULL, NULL }, | ||
289 | - { gen_helper_sve_ld1bdu_r, NULL, NULL, NULL }, | ||
290 | + static gen_helper_gvec_mem * const fns[2][16][4] = { | ||
291 | + /* Little-endian */ | ||
292 | + { { gen_helper_sve_ld1bb_r, gen_helper_sve_ld2bb_r, | ||
293 | + gen_helper_sve_ld3bb_r, gen_helper_sve_ld4bb_r }, | ||
294 | + { gen_helper_sve_ld1bhu_r, NULL, NULL, NULL }, | ||
295 | + { gen_helper_sve_ld1bsu_r, NULL, NULL, NULL }, | ||
296 | + { gen_helper_sve_ld1bdu_r, NULL, NULL, NULL }, | ||
297 | |||
298 | - { gen_helper_sve_ld1sds_r, NULL, NULL, NULL }, | ||
299 | - { gen_helper_sve_ld1hh_r, gen_helper_sve_ld2hh_r, | ||
300 | - gen_helper_sve_ld3hh_r, gen_helper_sve_ld4hh_r }, | ||
301 | - { gen_helper_sve_ld1hsu_r, NULL, NULL, NULL }, | ||
302 | - { gen_helper_sve_ld1hdu_r, NULL, NULL, NULL }, | ||
303 | + { gen_helper_sve_ld1sds_le_r, NULL, NULL, NULL }, | ||
304 | + { gen_helper_sve_ld1hh_le_r, gen_helper_sve_ld2hh_le_r, | ||
305 | + gen_helper_sve_ld3hh_le_r, gen_helper_sve_ld4hh_le_r }, | ||
306 | + { gen_helper_sve_ld1hsu_le_r, NULL, NULL, NULL }, | ||
307 | + { gen_helper_sve_ld1hdu_le_r, NULL, NULL, NULL }, | ||
308 | |||
309 | - { gen_helper_sve_ld1hds_r, NULL, NULL, NULL }, | ||
310 | - { gen_helper_sve_ld1hss_r, NULL, NULL, NULL }, | ||
311 | - { gen_helper_sve_ld1ss_r, gen_helper_sve_ld2ss_r, | ||
312 | - gen_helper_sve_ld3ss_r, gen_helper_sve_ld4ss_r }, | ||
313 | - { gen_helper_sve_ld1sdu_r, NULL, NULL, NULL }, | ||
314 | + { gen_helper_sve_ld1hds_le_r, NULL, NULL, NULL }, | ||
315 | + { gen_helper_sve_ld1hss_le_r, NULL, NULL, NULL }, | ||
316 | + { gen_helper_sve_ld1ss_le_r, gen_helper_sve_ld2ss_le_r, | ||
317 | + gen_helper_sve_ld3ss_le_r, gen_helper_sve_ld4ss_le_r }, | ||
318 | + { gen_helper_sve_ld1sdu_le_r, NULL, NULL, NULL }, | ||
319 | |||
320 | - { gen_helper_sve_ld1bds_r, NULL, NULL, NULL }, | ||
321 | - { gen_helper_sve_ld1bss_r, NULL, NULL, NULL }, | ||
322 | - { gen_helper_sve_ld1bhs_r, NULL, NULL, NULL }, | ||
323 | - { gen_helper_sve_ld1dd_r, gen_helper_sve_ld2dd_r, | ||
324 | - gen_helper_sve_ld3dd_r, gen_helper_sve_ld4dd_r }, | ||
325 | + { gen_helper_sve_ld1bds_r, NULL, NULL, NULL }, | ||
326 | + { gen_helper_sve_ld1bss_r, NULL, NULL, NULL }, | ||
327 | + { gen_helper_sve_ld1bhs_r, NULL, NULL, NULL }, | ||
328 | + { gen_helper_sve_ld1dd_le_r, gen_helper_sve_ld2dd_le_r, | ||
329 | + gen_helper_sve_ld3dd_le_r, gen_helper_sve_ld4dd_le_r } }, | ||
330 | + | ||
331 | + /* Big-endian */ | ||
332 | + { { gen_helper_sve_ld1bb_r, gen_helper_sve_ld2bb_r, | ||
333 | + gen_helper_sve_ld3bb_r, gen_helper_sve_ld4bb_r }, | ||
334 | + { gen_helper_sve_ld1bhu_r, NULL, NULL, NULL }, | ||
335 | + { gen_helper_sve_ld1bsu_r, NULL, NULL, NULL }, | ||
336 | + { gen_helper_sve_ld1bdu_r, NULL, NULL, NULL }, | ||
337 | + | ||
338 | + { gen_helper_sve_ld1sds_be_r, NULL, NULL, NULL }, | ||
339 | + { gen_helper_sve_ld1hh_be_r, gen_helper_sve_ld2hh_be_r, | ||
340 | + gen_helper_sve_ld3hh_be_r, gen_helper_sve_ld4hh_be_r }, | ||
341 | + { gen_helper_sve_ld1hsu_be_r, NULL, NULL, NULL }, | ||
342 | + { gen_helper_sve_ld1hdu_be_r, NULL, NULL, NULL }, | ||
343 | + | ||
344 | + { gen_helper_sve_ld1hds_be_r, NULL, NULL, NULL }, | ||
345 | + { gen_helper_sve_ld1hss_be_r, NULL, NULL, NULL }, | ||
346 | + { gen_helper_sve_ld1ss_be_r, gen_helper_sve_ld2ss_be_r, | ||
347 | + gen_helper_sve_ld3ss_be_r, gen_helper_sve_ld4ss_be_r }, | ||
348 | + { gen_helper_sve_ld1sdu_be_r, NULL, NULL, NULL }, | ||
349 | + | ||
350 | + { gen_helper_sve_ld1bds_r, NULL, NULL, NULL }, | ||
351 | + { gen_helper_sve_ld1bss_r, NULL, NULL, NULL }, | ||
352 | + { gen_helper_sve_ld1bhs_r, NULL, NULL, NULL }, | ||
353 | + { gen_helper_sve_ld1dd_be_r, gen_helper_sve_ld2dd_be_r, | ||
354 | + gen_helper_sve_ld3dd_be_r, gen_helper_sve_ld4dd_be_r } } | ||
355 | }; | ||
356 | - gen_helper_gvec_mem *fn = fns[dtype][nreg]; | ||
357 | + gen_helper_gvec_mem *fn = fns[s->be_data == MO_BE][dtype][nreg]; | ||
358 | |||
359 | /* While there are holes in the table, they are not | ||
360 | * accessible via the instruction encoding. | ||
361 | @@ -XXX,XX +XXX,XX @@ static bool trans_LD_zpri(DisasContext *s, arg_rpri_load *a, uint32_t insn) | ||
362 | |||
363 | static bool trans_LDFF1_zprr(DisasContext *s, arg_rprr_load *a, uint32_t insn) | ||
364 | { | ||
365 | - static gen_helper_gvec_mem * const fns[16] = { | ||
366 | - gen_helper_sve_ldff1bb_r, | ||
367 | - gen_helper_sve_ldff1bhu_r, | ||
368 | - gen_helper_sve_ldff1bsu_r, | ||
369 | - gen_helper_sve_ldff1bdu_r, | ||
370 | + static gen_helper_gvec_mem * const fns[2][16] = { | ||
371 | + /* Little-endian */ | ||
372 | + { gen_helper_sve_ldff1bb_r, | ||
373 | + gen_helper_sve_ldff1bhu_r, | ||
374 | + gen_helper_sve_ldff1bsu_r, | ||
375 | + gen_helper_sve_ldff1bdu_r, | ||
376 | |||
377 | - gen_helper_sve_ldff1sds_r, | ||
378 | - gen_helper_sve_ldff1hh_r, | ||
379 | - gen_helper_sve_ldff1hsu_r, | ||
380 | - gen_helper_sve_ldff1hdu_r, | ||
381 | + gen_helper_sve_ldff1sds_le_r, | ||
382 | + gen_helper_sve_ldff1hh_le_r, | ||
383 | + gen_helper_sve_ldff1hsu_le_r, | ||
384 | + gen_helper_sve_ldff1hdu_le_r, | ||
385 | |||
386 | - gen_helper_sve_ldff1hds_r, | ||
387 | - gen_helper_sve_ldff1hss_r, | ||
388 | - gen_helper_sve_ldff1ss_r, | ||
389 | - gen_helper_sve_ldff1sdu_r, | ||
390 | + gen_helper_sve_ldff1hds_le_r, | ||
391 | + gen_helper_sve_ldff1hss_le_r, | ||
392 | + gen_helper_sve_ldff1ss_le_r, | ||
393 | + gen_helper_sve_ldff1sdu_le_r, | ||
394 | |||
395 | - gen_helper_sve_ldff1bds_r, | ||
396 | - gen_helper_sve_ldff1bss_r, | ||
397 | - gen_helper_sve_ldff1bhs_r, | ||
398 | - gen_helper_sve_ldff1dd_r, | ||
399 | + gen_helper_sve_ldff1bds_r, | ||
400 | + gen_helper_sve_ldff1bss_r, | ||
401 | + gen_helper_sve_ldff1bhs_r, | ||
402 | + gen_helper_sve_ldff1dd_le_r }, | ||
403 | + | ||
404 | + /* Big-endian */ | ||
405 | + { gen_helper_sve_ldff1bb_r, | ||
406 | + gen_helper_sve_ldff1bhu_r, | ||
407 | + gen_helper_sve_ldff1bsu_r, | ||
408 | + gen_helper_sve_ldff1bdu_r, | ||
409 | + | ||
410 | + gen_helper_sve_ldff1sds_be_r, | ||
411 | + gen_helper_sve_ldff1hh_be_r, | ||
412 | + gen_helper_sve_ldff1hsu_be_r, | ||
413 | + gen_helper_sve_ldff1hdu_be_r, | ||
414 | + | ||
415 | + gen_helper_sve_ldff1hds_be_r, | ||
416 | + gen_helper_sve_ldff1hss_be_r, | ||
417 | + gen_helper_sve_ldff1ss_be_r, | ||
418 | + gen_helper_sve_ldff1sdu_be_r, | ||
419 | + | ||
420 | + gen_helper_sve_ldff1bds_r, | ||
421 | + gen_helper_sve_ldff1bss_r, | ||
422 | + gen_helper_sve_ldff1bhs_r, | ||
423 | + gen_helper_sve_ldff1dd_be_r }, | ||
424 | }; | ||
425 | |||
426 | if (sve_access_check(s)) { | ||
427 | TCGv_i64 addr = new_tmp_a64(s); | ||
428 | tcg_gen_shli_i64(addr, cpu_reg(s, a->rm), dtype_msz(a->dtype)); | ||
429 | tcg_gen_add_i64(addr, addr, cpu_reg_sp(s, a->rn)); | ||
430 | - do_mem_zpa(s, a->rd, a->pg, addr, fns[a->dtype]); | ||
431 | + do_mem_zpa(s, a->rd, a->pg, addr, fns[s->be_data == MO_BE][a->dtype]); | ||
432 | } | ||
433 | return true; | ||
434 | } | ||
435 | |||
436 | static bool trans_LDNF1_zpri(DisasContext *s, arg_rpri_load *a, uint32_t insn) | ||
437 | { | ||
438 | - static gen_helper_gvec_mem * const fns[16] = { | ||
439 | - gen_helper_sve_ldnf1bb_r, | ||
440 | - gen_helper_sve_ldnf1bhu_r, | ||
441 | - gen_helper_sve_ldnf1bsu_r, | ||
442 | - gen_helper_sve_ldnf1bdu_r, | ||
443 | + static gen_helper_gvec_mem * const fns[2][16] = { | ||
444 | + /* Little-endian */ | ||
445 | + { gen_helper_sve_ldnf1bb_r, | ||
446 | + gen_helper_sve_ldnf1bhu_r, | ||
447 | + gen_helper_sve_ldnf1bsu_r, | ||
448 | + gen_helper_sve_ldnf1bdu_r, | ||
449 | |||
450 | - gen_helper_sve_ldnf1sds_r, | ||
451 | - gen_helper_sve_ldnf1hh_r, | ||
452 | - gen_helper_sve_ldnf1hsu_r, | ||
453 | - gen_helper_sve_ldnf1hdu_r, | ||
454 | + gen_helper_sve_ldnf1sds_le_r, | ||
455 | + gen_helper_sve_ldnf1hh_le_r, | ||
456 | + gen_helper_sve_ldnf1hsu_le_r, | ||
457 | + gen_helper_sve_ldnf1hdu_le_r, | ||
458 | |||
459 | - gen_helper_sve_ldnf1hds_r, | ||
460 | - gen_helper_sve_ldnf1hss_r, | ||
461 | - gen_helper_sve_ldnf1ss_r, | ||
462 | - gen_helper_sve_ldnf1sdu_r, | ||
463 | + gen_helper_sve_ldnf1hds_le_r, | ||
464 | + gen_helper_sve_ldnf1hss_le_r, | ||
465 | + gen_helper_sve_ldnf1ss_le_r, | ||
466 | + gen_helper_sve_ldnf1sdu_le_r, | ||
467 | |||
468 | - gen_helper_sve_ldnf1bds_r, | ||
469 | - gen_helper_sve_ldnf1bss_r, | ||
470 | - gen_helper_sve_ldnf1bhs_r, | ||
471 | - gen_helper_sve_ldnf1dd_r, | ||
472 | + gen_helper_sve_ldnf1bds_r, | ||
473 | + gen_helper_sve_ldnf1bss_r, | ||
474 | + gen_helper_sve_ldnf1bhs_r, | ||
475 | + gen_helper_sve_ldnf1dd_le_r }, | ||
476 | + | ||
477 | + /* Big-endian */ | ||
478 | + { gen_helper_sve_ldnf1bb_r, | ||
479 | + gen_helper_sve_ldnf1bhu_r, | ||
480 | + gen_helper_sve_ldnf1bsu_r, | ||
481 | + gen_helper_sve_ldnf1bdu_r, | ||
482 | + | ||
483 | + gen_helper_sve_ldnf1sds_be_r, | ||
484 | + gen_helper_sve_ldnf1hh_be_r, | ||
485 | + gen_helper_sve_ldnf1hsu_be_r, | ||
486 | + gen_helper_sve_ldnf1hdu_be_r, | ||
487 | + | ||
488 | + gen_helper_sve_ldnf1hds_be_r, | ||
489 | + gen_helper_sve_ldnf1hss_be_r, | ||
490 | + gen_helper_sve_ldnf1ss_be_r, | ||
491 | + gen_helper_sve_ldnf1sdu_be_r, | ||
492 | + | ||
493 | + gen_helper_sve_ldnf1bds_r, | ||
494 | + gen_helper_sve_ldnf1bss_r, | ||
495 | + gen_helper_sve_ldnf1bhs_r, | ||
496 | + gen_helper_sve_ldnf1dd_be_r }, | ||
497 | }; | ||
498 | |||
499 | if (sve_access_check(s)) { | ||
500 | @@ -XXX,XX +XXX,XX @@ static bool trans_LDNF1_zpri(DisasContext *s, arg_rpri_load *a, uint32_t insn) | ||
501 | TCGv_i64 addr = new_tmp_a64(s); | ||
502 | |||
503 | tcg_gen_addi_i64(addr, cpu_reg_sp(s, a->rn), off); | ||
504 | - do_mem_zpa(s, a->rd, a->pg, addr, fns[a->dtype]); | ||
505 | + do_mem_zpa(s, a->rd, a->pg, addr, fns[s->be_data == MO_BE][a->dtype]); | ||
506 | } | ||
507 | return true; | ||
508 | } | ||
509 | |||
510 | static void do_ldrq(DisasContext *s, int zt, int pg, TCGv_i64 addr, int msz) | ||
511 | { | ||
512 | - static gen_helper_gvec_mem * const fns[4] = { | ||
513 | - gen_helper_sve_ld1bb_r, gen_helper_sve_ld1hh_r, | ||
514 | - gen_helper_sve_ld1ss_r, gen_helper_sve_ld1dd_r, | ||
515 | + static gen_helper_gvec_mem * const fns[2][4] = { | ||
516 | + { gen_helper_sve_ld1bb_r, gen_helper_sve_ld1hh_le_r, | ||
517 | + gen_helper_sve_ld1ss_le_r, gen_helper_sve_ld1dd_le_r }, | ||
518 | + { gen_helper_sve_ld1bb_r, gen_helper_sve_ld1hh_be_r, | ||
519 | + gen_helper_sve_ld1ss_be_r, gen_helper_sve_ld1dd_be_r }, | ||
520 | }; | ||
521 | unsigned vsz = vec_full_reg_size(s); | ||
522 | TCGv_ptr t_pg; | ||
523 | @@ -XXX,XX +XXX,XX @@ static void do_ldrq(DisasContext *s, int zt, int pg, TCGv_i64 addr, int msz) | ||
524 | t_pg = tcg_temp_new_ptr(); | ||
525 | tcg_gen_addi_ptr(t_pg, cpu_env, poff); | ||
526 | |||
527 | - fns[msz](cpu_env, t_pg, addr, desc); | ||
528 | + fns[s->be_data == MO_BE][msz](cpu_env, t_pg, addr, desc); | ||
529 | |||
530 | tcg_temp_free_ptr(t_pg); | ||
531 | tcg_temp_free_i32(desc); | ||
532 | -- | ||
533 | 2.19.0 | ||
534 | |||
535 | diff view generated by jsdifflib |
Deleted patch | |||
---|---|---|---|
1 | From: Richard Henderson <richard.henderson@linaro.org> | ||
2 | 1 | ||
3 | We can choose the endianness at translation time, rather than | ||
4 | re-computing it at execution time. | ||
5 | |||
6 | Tested-by: Laurent Desnogues <laurent.desnogues@gmail.com> | ||
7 | Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org> | ||
8 | Signed-off-by: Richard Henderson <richard.henderson@linaro.org> | ||
9 | Message-id: 20181005175350.30752-12-richard.henderson@linaro.org | ||
10 | Signed-off-by: Peter Maydell <peter.maydell@linaro.org> | ||
11 | --- | ||
12 | target/arm/helper-sve.h | 48 +++++++++++++++++-------- | ||
13 | target/arm/sve_helper.c | 11 ++++-- | ||
14 | target/arm/translate-sve.c | 72 +++++++++++++++++++++++++++++--------- | ||
15 | 3 files changed, 96 insertions(+), 35 deletions(-) | ||
16 | |||
17 | diff --git a/target/arm/helper-sve.h b/target/arm/helper-sve.h | ||
18 | index XXXXXXX..XXXXXXX 100644 | ||
19 | --- a/target/arm/helper-sve.h | ||
20 | +++ b/target/arm/helper-sve.h | ||
21 | @@ -XXX,XX +XXX,XX @@ DEF_HELPER_FLAGS_4(sve_st2bb_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
22 | DEF_HELPER_FLAGS_4(sve_st3bb_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
23 | DEF_HELPER_FLAGS_4(sve_st4bb_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
24 | |||
25 | -DEF_HELPER_FLAGS_4(sve_st1hh_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
26 | -DEF_HELPER_FLAGS_4(sve_st2hh_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
27 | -DEF_HELPER_FLAGS_4(sve_st3hh_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
28 | -DEF_HELPER_FLAGS_4(sve_st4hh_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
29 | +DEF_HELPER_FLAGS_4(sve_st1hh_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
30 | +DEF_HELPER_FLAGS_4(sve_st2hh_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
31 | +DEF_HELPER_FLAGS_4(sve_st3hh_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
32 | +DEF_HELPER_FLAGS_4(sve_st4hh_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
33 | |||
34 | -DEF_HELPER_FLAGS_4(sve_st1ss_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
35 | -DEF_HELPER_FLAGS_4(sve_st2ss_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
36 | -DEF_HELPER_FLAGS_4(sve_st3ss_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
37 | -DEF_HELPER_FLAGS_4(sve_st4ss_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
38 | +DEF_HELPER_FLAGS_4(sve_st1hh_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
39 | +DEF_HELPER_FLAGS_4(sve_st2hh_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
40 | +DEF_HELPER_FLAGS_4(sve_st3hh_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
41 | +DEF_HELPER_FLAGS_4(sve_st4hh_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
42 | |||
43 | -DEF_HELPER_FLAGS_4(sve_st1dd_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
44 | -DEF_HELPER_FLAGS_4(sve_st2dd_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
45 | -DEF_HELPER_FLAGS_4(sve_st3dd_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
46 | -DEF_HELPER_FLAGS_4(sve_st4dd_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
47 | +DEF_HELPER_FLAGS_4(sve_st1ss_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
48 | +DEF_HELPER_FLAGS_4(sve_st2ss_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
49 | +DEF_HELPER_FLAGS_4(sve_st3ss_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
50 | +DEF_HELPER_FLAGS_4(sve_st4ss_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
51 | + | ||
52 | +DEF_HELPER_FLAGS_4(sve_st1ss_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
53 | +DEF_HELPER_FLAGS_4(sve_st2ss_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
54 | +DEF_HELPER_FLAGS_4(sve_st3ss_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
55 | +DEF_HELPER_FLAGS_4(sve_st4ss_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
56 | + | ||
57 | +DEF_HELPER_FLAGS_4(sve_st1dd_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
58 | +DEF_HELPER_FLAGS_4(sve_st2dd_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
59 | +DEF_HELPER_FLAGS_4(sve_st3dd_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
60 | +DEF_HELPER_FLAGS_4(sve_st4dd_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
61 | + | ||
62 | +DEF_HELPER_FLAGS_4(sve_st1dd_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
63 | +DEF_HELPER_FLAGS_4(sve_st2dd_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
64 | +DEF_HELPER_FLAGS_4(sve_st3dd_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
65 | +DEF_HELPER_FLAGS_4(sve_st4dd_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
66 | |||
67 | DEF_HELPER_FLAGS_4(sve_st1bh_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
68 | DEF_HELPER_FLAGS_4(sve_st1bs_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
69 | DEF_HELPER_FLAGS_4(sve_st1bd_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
70 | |||
71 | -DEF_HELPER_FLAGS_4(sve_st1hs_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
72 | -DEF_HELPER_FLAGS_4(sve_st1hd_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
73 | +DEF_HELPER_FLAGS_4(sve_st1hs_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
74 | +DEF_HELPER_FLAGS_4(sve_st1hd_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
75 | +DEF_HELPER_FLAGS_4(sve_st1hs_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
76 | +DEF_HELPER_FLAGS_4(sve_st1hd_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
77 | |||
78 | -DEF_HELPER_FLAGS_4(sve_st1sd_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
79 | +DEF_HELPER_FLAGS_4(sve_st1sd_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
80 | +DEF_HELPER_FLAGS_4(sve_st1sd_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
81 | |||
82 | DEF_HELPER_FLAGS_6(sve_ldbsu_zsu, TCG_CALL_NO_WG, | ||
83 | void, env, ptr, ptr, ptr, tl, i32) | ||
84 | diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c | ||
85 | index XXXXXXX..XXXXXXX 100644 | ||
86 | --- a/target/arm/sve_helper.c | ||
87 | +++ b/target/arm/sve_helper.c | ||
88 | @@ -XXX,XX +XXX,XX @@ void __attribute__((flatten)) HELPER(sve_st##N##NAME##_r) \ | ||
89 | } | ||
90 | |||
91 | #define DO_STN_2(N, NAME, ESIZE, MSIZE) \ | ||
92 | -void __attribute__((flatten)) HELPER(sve_st##N##NAME##_r) \ | ||
93 | +void __attribute__((flatten)) HELPER(sve_st##N##NAME##_le_r) \ | ||
94 | (CPUARMState *env, void *vg, target_ulong addr, uint32_t desc) \ | ||
95 | { \ | ||
96 | sve_st##N##_r(env, vg, addr, desc, GETPC(), ESIZE, MSIZE, \ | ||
97 | - arm_cpu_data_is_big_endian(env) \ | ||
98 | - ? sve_st1##NAME##_be_tlb : sve_st1##NAME##_le_tlb); \ | ||
99 | + sve_st1##NAME##_le_tlb); \ | ||
100 | +} \ | ||
101 | +void __attribute__((flatten)) HELPER(sve_st##N##NAME##_be_r) \ | ||
102 | + (CPUARMState *env, void *vg, target_ulong addr, uint32_t desc) \ | ||
103 | +{ \ | ||
104 | + sve_st##N##_r(env, vg, addr, desc, GETPC(), ESIZE, MSIZE, \ | ||
105 | + sve_st1##NAME##_be_tlb); \ | ||
106 | } | ||
107 | |||
108 | DO_STN_1(1, bb, 1) | ||
109 | diff --git a/target/arm/translate-sve.c b/target/arm/translate-sve.c | ||
110 | index XXXXXXX..XXXXXXX 100644 | ||
111 | --- a/target/arm/translate-sve.c | ||
112 | +++ b/target/arm/translate-sve.c | ||
113 | @@ -XXX,XX +XXX,XX @@ static bool trans_LD1R_zpri(DisasContext *s, arg_rpri_load *a, uint32_t insn) | ||
114 | static void do_st_zpa(DisasContext *s, int zt, int pg, TCGv_i64 addr, | ||
115 | int msz, int esz, int nreg) | ||
116 | { | ||
117 | - static gen_helper_gvec_mem * const fn_single[4][4] = { | ||
118 | - { gen_helper_sve_st1bb_r, gen_helper_sve_st1bh_r, | ||
119 | - gen_helper_sve_st1bs_r, gen_helper_sve_st1bd_r }, | ||
120 | - { NULL, gen_helper_sve_st1hh_r, | ||
121 | - gen_helper_sve_st1hs_r, gen_helper_sve_st1hd_r }, | ||
122 | - { NULL, NULL, | ||
123 | - gen_helper_sve_st1ss_r, gen_helper_sve_st1sd_r }, | ||
124 | - { NULL, NULL, NULL, gen_helper_sve_st1dd_r }, | ||
125 | + static gen_helper_gvec_mem * const fn_single[2][4][4] = { | ||
126 | + { { gen_helper_sve_st1bb_r, | ||
127 | + gen_helper_sve_st1bh_r, | ||
128 | + gen_helper_sve_st1bs_r, | ||
129 | + gen_helper_sve_st1bd_r }, | ||
130 | + { NULL, | ||
131 | + gen_helper_sve_st1hh_le_r, | ||
132 | + gen_helper_sve_st1hs_le_r, | ||
133 | + gen_helper_sve_st1hd_le_r }, | ||
134 | + { NULL, NULL, | ||
135 | + gen_helper_sve_st1ss_le_r, | ||
136 | + gen_helper_sve_st1sd_le_r }, | ||
137 | + { NULL, NULL, NULL, | ||
138 | + gen_helper_sve_st1dd_le_r } }, | ||
139 | + { { gen_helper_sve_st1bb_r, | ||
140 | + gen_helper_sve_st1bh_r, | ||
141 | + gen_helper_sve_st1bs_r, | ||
142 | + gen_helper_sve_st1bd_r }, | ||
143 | + { NULL, | ||
144 | + gen_helper_sve_st1hh_be_r, | ||
145 | + gen_helper_sve_st1hs_be_r, | ||
146 | + gen_helper_sve_st1hd_be_r }, | ||
147 | + { NULL, NULL, | ||
148 | + gen_helper_sve_st1ss_be_r, | ||
149 | + gen_helper_sve_st1sd_be_r }, | ||
150 | + { NULL, NULL, NULL, | ||
151 | + gen_helper_sve_st1dd_be_r } }, | ||
152 | }; | ||
153 | - static gen_helper_gvec_mem * const fn_multiple[3][4] = { | ||
154 | - { gen_helper_sve_st2bb_r, gen_helper_sve_st2hh_r, | ||
155 | - gen_helper_sve_st2ss_r, gen_helper_sve_st2dd_r }, | ||
156 | - { gen_helper_sve_st3bb_r, gen_helper_sve_st3hh_r, | ||
157 | - gen_helper_sve_st3ss_r, gen_helper_sve_st3dd_r }, | ||
158 | - { gen_helper_sve_st4bb_r, gen_helper_sve_st4hh_r, | ||
159 | - gen_helper_sve_st4ss_r, gen_helper_sve_st4dd_r }, | ||
160 | + static gen_helper_gvec_mem * const fn_multiple[2][3][4] = { | ||
161 | + { { gen_helper_sve_st2bb_r, | ||
162 | + gen_helper_sve_st2hh_le_r, | ||
163 | + gen_helper_sve_st2ss_le_r, | ||
164 | + gen_helper_sve_st2dd_le_r }, | ||
165 | + { gen_helper_sve_st3bb_r, | ||
166 | + gen_helper_sve_st3hh_le_r, | ||
167 | + gen_helper_sve_st3ss_le_r, | ||
168 | + gen_helper_sve_st3dd_le_r }, | ||
169 | + { gen_helper_sve_st4bb_r, | ||
170 | + gen_helper_sve_st4hh_le_r, | ||
171 | + gen_helper_sve_st4ss_le_r, | ||
172 | + gen_helper_sve_st4dd_le_r } }, | ||
173 | + { { gen_helper_sve_st2bb_r, | ||
174 | + gen_helper_sve_st2hh_be_r, | ||
175 | + gen_helper_sve_st2ss_be_r, | ||
176 | + gen_helper_sve_st2dd_be_r }, | ||
177 | + { gen_helper_sve_st3bb_r, | ||
178 | + gen_helper_sve_st3hh_be_r, | ||
179 | + gen_helper_sve_st3ss_be_r, | ||
180 | + gen_helper_sve_st3dd_be_r }, | ||
181 | + { gen_helper_sve_st4bb_r, | ||
182 | + gen_helper_sve_st4hh_be_r, | ||
183 | + gen_helper_sve_st4ss_be_r, | ||
184 | + gen_helper_sve_st4dd_be_r } }, | ||
185 | }; | ||
186 | gen_helper_gvec_mem *fn; | ||
187 | + int be = s->be_data == MO_BE; | ||
188 | |||
189 | if (nreg == 0) { | ||
190 | /* ST1 */ | ||
191 | - fn = fn_single[msz][esz]; | ||
192 | + fn = fn_single[be][msz][esz]; | ||
193 | } else { | ||
194 | /* ST2, ST3, ST4 -- msz == esz, enforced by encoding */ | ||
195 | assert(msz == esz); | ||
196 | - fn = fn_multiple[nreg - 1][msz]; | ||
197 | + fn = fn_multiple[be][nreg - 1][msz]; | ||
198 | } | ||
199 | assert(fn != NULL); | ||
200 | do_mem_zpa(s, zt, pg, addr, fn); | ||
201 | -- | ||
202 | 2.19.0 | ||
203 | |||
204 | diff view generated by jsdifflib |
Deleted patch | |||
---|---|---|---|
1 | From: Richard Henderson <richard.henderson@linaro.org> | ||
2 | 1 | ||
3 | This fixes the endianness problem for softmmu, and moves | ||
4 | the main loop out of a macro and into an inlined function. | ||
5 | |||
6 | Reviewed-by: Peter Maydell <peter.maydell@linaro.org> | ||
7 | Tested-by: Laurent Desnogues <laurent.desnogues@gmail.com> | ||
8 | Signed-off-by: Richard Henderson <richard.henderson@linaro.org> | ||
9 | Message-id: 20181005175350.30752-13-richard.henderson@linaro.org | ||
10 | Signed-off-by: Peter Maydell <peter.maydell@linaro.org> | ||
11 | --- | ||
12 | target/arm/helper-sve.h | 84 +++++++++---- | ||
13 | target/arm/sve_helper.c | 225 ++++++++++++++++++++++++---------- | ||
14 | target/arm/translate-sve.c | 244 +++++++++++++++++++++++++------------ | ||
15 | 3 files changed, 386 insertions(+), 167 deletions(-) | ||
16 | |||
17 | diff --git a/target/arm/helper-sve.h b/target/arm/helper-sve.h | ||
18 | index XXXXXXX..XXXXXXX 100644 | ||
19 | --- a/target/arm/helper-sve.h | ||
20 | +++ b/target/arm/helper-sve.h | ||
21 | @@ -XXX,XX +XXX,XX @@ DEF_HELPER_FLAGS_4(sve_st1sd_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) | ||
22 | |||
23 | DEF_HELPER_FLAGS_6(sve_ldbsu_zsu, TCG_CALL_NO_WG, | ||
24 | void, env, ptr, ptr, ptr, tl, i32) | ||
25 | -DEF_HELPER_FLAGS_6(sve_ldhsu_zsu, TCG_CALL_NO_WG, | ||
26 | +DEF_HELPER_FLAGS_6(sve_ldhsu_le_zsu, TCG_CALL_NO_WG, | ||
27 | void, env, ptr, ptr, ptr, tl, i32) | ||
28 | -DEF_HELPER_FLAGS_6(sve_ldssu_zsu, TCG_CALL_NO_WG, | ||
29 | +DEF_HELPER_FLAGS_6(sve_ldhsu_be_zsu, TCG_CALL_NO_WG, | ||
30 | + void, env, ptr, ptr, ptr, tl, i32) | ||
31 | +DEF_HELPER_FLAGS_6(sve_ldss_le_zsu, TCG_CALL_NO_WG, | ||
32 | + void, env, ptr, ptr, ptr, tl, i32) | ||
33 | +DEF_HELPER_FLAGS_6(sve_ldss_be_zsu, TCG_CALL_NO_WG, | ||
34 | void, env, ptr, ptr, ptr, tl, i32) | ||
35 | DEF_HELPER_FLAGS_6(sve_ldbss_zsu, TCG_CALL_NO_WG, | ||
36 | void, env, ptr, ptr, ptr, tl, i32) | ||
37 | -DEF_HELPER_FLAGS_6(sve_ldhss_zsu, TCG_CALL_NO_WG, | ||
38 | +DEF_HELPER_FLAGS_6(sve_ldhss_le_zsu, TCG_CALL_NO_WG, | ||
39 | + void, env, ptr, ptr, ptr, tl, i32) | ||
40 | +DEF_HELPER_FLAGS_6(sve_ldhss_be_zsu, TCG_CALL_NO_WG, | ||
41 | void, env, ptr, ptr, ptr, tl, i32) | ||
42 | |||
43 | DEF_HELPER_FLAGS_6(sve_ldbsu_zss, TCG_CALL_NO_WG, | ||
44 | void, env, ptr, ptr, ptr, tl, i32) | ||
45 | -DEF_HELPER_FLAGS_6(sve_ldhsu_zss, TCG_CALL_NO_WG, | ||
46 | +DEF_HELPER_FLAGS_6(sve_ldhsu_le_zss, TCG_CALL_NO_WG, | ||
47 | void, env, ptr, ptr, ptr, tl, i32) | ||
48 | -DEF_HELPER_FLAGS_6(sve_ldssu_zss, TCG_CALL_NO_WG, | ||
49 | +DEF_HELPER_FLAGS_6(sve_ldhsu_be_zss, TCG_CALL_NO_WG, | ||
50 | + void, env, ptr, ptr, ptr, tl, i32) | ||
51 | +DEF_HELPER_FLAGS_6(sve_ldss_le_zss, TCG_CALL_NO_WG, | ||
52 | + void, env, ptr, ptr, ptr, tl, i32) | ||
53 | +DEF_HELPER_FLAGS_6(sve_ldss_be_zss, TCG_CALL_NO_WG, | ||
54 | void, env, ptr, ptr, ptr, tl, i32) | ||
55 | DEF_HELPER_FLAGS_6(sve_ldbss_zss, TCG_CALL_NO_WG, | ||
56 | void, env, ptr, ptr, ptr, tl, i32) | ||
57 | -DEF_HELPER_FLAGS_6(sve_ldhss_zss, TCG_CALL_NO_WG, | ||
58 | +DEF_HELPER_FLAGS_6(sve_ldhss_le_zss, TCG_CALL_NO_WG, | ||
59 | + void, env, ptr, ptr, ptr, tl, i32) | ||
60 | +DEF_HELPER_FLAGS_6(sve_ldhss_be_zss, TCG_CALL_NO_WG, | ||
61 | void, env, ptr, ptr, ptr, tl, i32) | ||
62 | |||
63 | DEF_HELPER_FLAGS_6(sve_ldbdu_zsu, TCG_CALL_NO_WG, | ||
64 | void, env, ptr, ptr, ptr, tl, i32) | ||
65 | -DEF_HELPER_FLAGS_6(sve_ldhdu_zsu, TCG_CALL_NO_WG, | ||
66 | +DEF_HELPER_FLAGS_6(sve_ldhdu_le_zsu, TCG_CALL_NO_WG, | ||
67 | void, env, ptr, ptr, ptr, tl, i32) | ||
68 | -DEF_HELPER_FLAGS_6(sve_ldsdu_zsu, TCG_CALL_NO_WG, | ||
69 | +DEF_HELPER_FLAGS_6(sve_ldhdu_be_zsu, TCG_CALL_NO_WG, | ||
70 | void, env, ptr, ptr, ptr, tl, i32) | ||
71 | -DEF_HELPER_FLAGS_6(sve_ldddu_zsu, TCG_CALL_NO_WG, | ||
72 | +DEF_HELPER_FLAGS_6(sve_ldsdu_le_zsu, TCG_CALL_NO_WG, | ||
73 | + void, env, ptr, ptr, ptr, tl, i32) | ||
74 | +DEF_HELPER_FLAGS_6(sve_ldsdu_be_zsu, TCG_CALL_NO_WG, | ||
75 | + void, env, ptr, ptr, ptr, tl, i32) | ||
76 | +DEF_HELPER_FLAGS_6(sve_lddd_le_zsu, TCG_CALL_NO_WG, | ||
77 | + void, env, ptr, ptr, ptr, tl, i32) | ||
78 | +DEF_HELPER_FLAGS_6(sve_lddd_be_zsu, TCG_CALL_NO_WG, | ||
79 | void, env, ptr, ptr, ptr, tl, i32) | ||
80 | DEF_HELPER_FLAGS_6(sve_ldbds_zsu, TCG_CALL_NO_WG, | ||
81 | void, env, ptr, ptr, ptr, tl, i32) | ||
82 | -DEF_HELPER_FLAGS_6(sve_ldhds_zsu, TCG_CALL_NO_WG, | ||
83 | +DEF_HELPER_FLAGS_6(sve_ldhds_le_zsu, TCG_CALL_NO_WG, | ||
84 | void, env, ptr, ptr, ptr, tl, i32) | ||
85 | -DEF_HELPER_FLAGS_6(sve_ldsds_zsu, TCG_CALL_NO_WG, | ||
86 | +DEF_HELPER_FLAGS_6(sve_ldhds_be_zsu, TCG_CALL_NO_WG, | ||
87 | + void, env, ptr, ptr, ptr, tl, i32) | ||
88 | +DEF_HELPER_FLAGS_6(sve_ldsds_le_zsu, TCG_CALL_NO_WG, | ||
89 | + void, env, ptr, ptr, ptr, tl, i32) | ||
90 | +DEF_HELPER_FLAGS_6(sve_ldsds_be_zsu, TCG_CALL_NO_WG, | ||
91 | void, env, ptr, ptr, ptr, tl, i32) | ||
92 | |||
93 | DEF_HELPER_FLAGS_6(sve_ldbdu_zss, TCG_CALL_NO_WG, | ||
94 | void, env, ptr, ptr, ptr, tl, i32) | ||
95 | -DEF_HELPER_FLAGS_6(sve_ldhdu_zss, TCG_CALL_NO_WG, | ||
96 | +DEF_HELPER_FLAGS_6(sve_ldhdu_le_zss, TCG_CALL_NO_WG, | ||
97 | void, env, ptr, ptr, ptr, tl, i32) | ||
98 | -DEF_HELPER_FLAGS_6(sve_ldsdu_zss, TCG_CALL_NO_WG, | ||
99 | +DEF_HELPER_FLAGS_6(sve_ldhdu_be_zss, TCG_CALL_NO_WG, | ||
100 | void, env, ptr, ptr, ptr, tl, i32) | ||
101 | -DEF_HELPER_FLAGS_6(sve_ldddu_zss, TCG_CALL_NO_WG, | ||
102 | +DEF_HELPER_FLAGS_6(sve_ldsdu_le_zss, TCG_CALL_NO_WG, | ||
103 | + void, env, ptr, ptr, ptr, tl, i32) | ||
104 | +DEF_HELPER_FLAGS_6(sve_ldsdu_be_zss, TCG_CALL_NO_WG, | ||
105 | + void, env, ptr, ptr, ptr, tl, i32) | ||
106 | +DEF_HELPER_FLAGS_6(sve_lddd_le_zss, TCG_CALL_NO_WG, | ||
107 | + void, env, ptr, ptr, ptr, tl, i32) | ||
108 | +DEF_HELPER_FLAGS_6(sve_lddd_be_zss, TCG_CALL_NO_WG, | ||
109 | void, env, ptr, ptr, ptr, tl, i32) | ||
110 | DEF_HELPER_FLAGS_6(sve_ldbds_zss, TCG_CALL_NO_WG, | ||
111 | void, env, ptr, ptr, ptr, tl, i32) | ||
112 | -DEF_HELPER_FLAGS_6(sve_ldhds_zss, TCG_CALL_NO_WG, | ||
113 | +DEF_HELPER_FLAGS_6(sve_ldhds_le_zss, TCG_CALL_NO_WG, | ||
114 | void, env, ptr, ptr, ptr, tl, i32) | ||
115 | -DEF_HELPER_FLAGS_6(sve_ldsds_zss, TCG_CALL_NO_WG, | ||
116 | +DEF_HELPER_FLAGS_6(sve_ldhds_be_zss, TCG_CALL_NO_WG, | ||
117 | + void, env, ptr, ptr, ptr, tl, i32) | ||
118 | +DEF_HELPER_FLAGS_6(sve_ldsds_le_zss, TCG_CALL_NO_WG, | ||
119 | + void, env, ptr, ptr, ptr, tl, i32) | ||
120 | +DEF_HELPER_FLAGS_6(sve_ldsds_be_zss, TCG_CALL_NO_WG, | ||
121 | void, env, ptr, ptr, ptr, tl, i32) | ||
122 | |||
123 | DEF_HELPER_FLAGS_6(sve_ldbdu_zd, TCG_CALL_NO_WG, | ||
124 | void, env, ptr, ptr, ptr, tl, i32) | ||
125 | -DEF_HELPER_FLAGS_6(sve_ldhdu_zd, TCG_CALL_NO_WG, | ||
126 | +DEF_HELPER_FLAGS_6(sve_ldhdu_le_zd, TCG_CALL_NO_WG, | ||
127 | void, env, ptr, ptr, ptr, tl, i32) | ||
128 | -DEF_HELPER_FLAGS_6(sve_ldsdu_zd, TCG_CALL_NO_WG, | ||
129 | +DEF_HELPER_FLAGS_6(sve_ldhdu_be_zd, TCG_CALL_NO_WG, | ||
130 | void, env, ptr, ptr, ptr, tl, i32) | ||
131 | -DEF_HELPER_FLAGS_6(sve_ldddu_zd, TCG_CALL_NO_WG, | ||
132 | +DEF_HELPER_FLAGS_6(sve_ldsdu_le_zd, TCG_CALL_NO_WG, | ||
133 | + void, env, ptr, ptr, ptr, tl, i32) | ||
134 | +DEF_HELPER_FLAGS_6(sve_ldsdu_be_zd, TCG_CALL_NO_WG, | ||
135 | + void, env, ptr, ptr, ptr, tl, i32) | ||
136 | +DEF_HELPER_FLAGS_6(sve_lddd_le_zd, TCG_CALL_NO_WG, | ||
137 | + void, env, ptr, ptr, ptr, tl, i32) | ||
138 | +DEF_HELPER_FLAGS_6(sve_lddd_be_zd, TCG_CALL_NO_WG, | ||
139 | void, env, ptr, ptr, ptr, tl, i32) | ||
140 | DEF_HELPER_FLAGS_6(sve_ldbds_zd, TCG_CALL_NO_WG, | ||
141 | void, env, ptr, ptr, ptr, tl, i32) | ||
142 | -DEF_HELPER_FLAGS_6(sve_ldhds_zd, TCG_CALL_NO_WG, | ||
143 | +DEF_HELPER_FLAGS_6(sve_ldhds_le_zd, TCG_CALL_NO_WG, | ||
144 | void, env, ptr, ptr, ptr, tl, i32) | ||
145 | -DEF_HELPER_FLAGS_6(sve_ldsds_zd, TCG_CALL_NO_WG, | ||
146 | +DEF_HELPER_FLAGS_6(sve_ldhds_be_zd, TCG_CALL_NO_WG, | ||
147 | + void, env, ptr, ptr, ptr, tl, i32) | ||
148 | +DEF_HELPER_FLAGS_6(sve_ldsds_le_zd, TCG_CALL_NO_WG, | ||
149 | + void, env, ptr, ptr, ptr, tl, i32) | ||
150 | +DEF_HELPER_FLAGS_6(sve_ldsds_be_zd, TCG_CALL_NO_WG, | ||
151 | void, env, ptr, ptr, ptr, tl, i32) | ||
152 | |||
153 | DEF_HELPER_FLAGS_6(sve_ldffbsu_zsu, TCG_CALL_NO_WG, | ||
154 | diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c | ||
155 | index XXXXXXX..XXXXXXX 100644 | ||
156 | --- a/target/arm/sve_helper.c | ||
157 | +++ b/target/arm/sve_helper.c | ||
158 | @@ -XXX,XX +XXX,XX @@ DO_STN_2(4, dd, 8, 8) | ||
159 | #undef DO_STN_1 | ||
160 | #undef DO_STN_2 | ||
161 | |||
162 | -/* Loads with a vector index. */ | ||
163 | +/* | ||
164 | + * Loads with a vector index. | ||
165 | + */ | ||
166 | |||
167 | -#define DO_LD1_ZPZ_S(NAME, TYPEI, TYPEM, FN) \ | ||
168 | -void HELPER(NAME)(CPUARMState *env, void *vd, void *vg, void *vm, \ | ||
169 | - target_ulong base, uint32_t desc) \ | ||
170 | -{ \ | ||
171 | - intptr_t i, oprsz = simd_oprsz(desc); \ | ||
172 | - unsigned scale = simd_data(desc); \ | ||
173 | - uintptr_t ra = GETPC(); \ | ||
174 | - for (i = 0; i < oprsz; ) { \ | ||
175 | - uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); \ | ||
176 | - do { \ | ||
177 | - TYPEM m = 0; \ | ||
178 | - if (pg & 1) { \ | ||
179 | - target_ulong off = *(TYPEI *)(vm + H1_4(i)); \ | ||
180 | - m = FN(env, base + (off << scale), ra); \ | ||
181 | - } \ | ||
182 | - *(uint32_t *)(vd + H1_4(i)) = m; \ | ||
183 | - i += 4, pg >>= 4; \ | ||
184 | - } while (i & 15); \ | ||
185 | - } \ | ||
186 | +/* | ||
187 | + * Load the element at @reg + @reg_ofs, sign or zero-extend as needed. | ||
188 | + */ | ||
189 | +typedef target_ulong zreg_off_fn(void *reg, intptr_t reg_ofs); | ||
190 | + | ||
191 | +static target_ulong off_zsu_s(void *reg, intptr_t reg_ofs) | ||
192 | +{ | ||
193 | + return *(uint32_t *)(reg + H1_4(reg_ofs)); | ||
194 | } | ||
195 | |||
196 | -#define DO_LD1_ZPZ_D(NAME, TYPEI, TYPEM, FN) \ | ||
197 | -void HELPER(NAME)(CPUARMState *env, void *vd, void *vg, void *vm, \ | ||
198 | - target_ulong base, uint32_t desc) \ | ||
199 | -{ \ | ||
200 | - intptr_t i, oprsz = simd_oprsz(desc) / 8; \ | ||
201 | - unsigned scale = simd_data(desc); \ | ||
202 | - uintptr_t ra = GETPC(); \ | ||
203 | - uint64_t *d = vd, *m = vm; uint8_t *pg = vg; \ | ||
204 | - for (i = 0; i < oprsz; i++) { \ | ||
205 | - TYPEM mm = 0; \ | ||
206 | - if (pg[H1(i)] & 1) { \ | ||
207 | - target_ulong off = (TYPEI)m[i]; \ | ||
208 | - mm = FN(env, base + (off << scale), ra); \ | ||
209 | - } \ | ||
210 | - d[i] = mm; \ | ||
211 | - } \ | ||
212 | +static target_ulong off_zss_s(void *reg, intptr_t reg_ofs) | ||
213 | +{ | ||
214 | + return *(int32_t *)(reg + H1_4(reg_ofs)); | ||
215 | } | ||
216 | |||
217 | -DO_LD1_ZPZ_S(sve_ldbsu_zsu, uint32_t, uint8_t, cpu_ldub_data_ra) | ||
218 | -DO_LD1_ZPZ_S(sve_ldhsu_zsu, uint32_t, uint16_t, cpu_lduw_data_ra) | ||
219 | -DO_LD1_ZPZ_S(sve_ldssu_zsu, uint32_t, uint32_t, cpu_ldl_data_ra) | ||
220 | -DO_LD1_ZPZ_S(sve_ldbss_zsu, uint32_t, int8_t, cpu_ldub_data_ra) | ||
221 | -DO_LD1_ZPZ_S(sve_ldhss_zsu, uint32_t, int16_t, cpu_lduw_data_ra) | ||
222 | +static target_ulong off_zsu_d(void *reg, intptr_t reg_ofs) | ||
223 | +{ | ||
224 | + return (uint32_t)*(uint64_t *)(reg + reg_ofs); | ||
225 | +} | ||
226 | |||
227 | -DO_LD1_ZPZ_S(sve_ldbsu_zss, int32_t, uint8_t, cpu_ldub_data_ra) | ||
228 | -DO_LD1_ZPZ_S(sve_ldhsu_zss, int32_t, uint16_t, cpu_lduw_data_ra) | ||
229 | -DO_LD1_ZPZ_S(sve_ldssu_zss, int32_t, uint32_t, cpu_ldl_data_ra) | ||
230 | -DO_LD1_ZPZ_S(sve_ldbss_zss, int32_t, int8_t, cpu_ldub_data_ra) | ||
231 | -DO_LD1_ZPZ_S(sve_ldhss_zss, int32_t, int16_t, cpu_lduw_data_ra) | ||
232 | +static target_ulong off_zss_d(void *reg, intptr_t reg_ofs) | ||
233 | +{ | ||
234 | + return (int32_t)*(uint64_t *)(reg + reg_ofs); | ||
235 | +} | ||
236 | |||
237 | -DO_LD1_ZPZ_D(sve_ldbdu_zsu, uint32_t, uint8_t, cpu_ldub_data_ra) | ||
238 | -DO_LD1_ZPZ_D(sve_ldhdu_zsu, uint32_t, uint16_t, cpu_lduw_data_ra) | ||
239 | -DO_LD1_ZPZ_D(sve_ldsdu_zsu, uint32_t, uint32_t, cpu_ldl_data_ra) | ||
240 | -DO_LD1_ZPZ_D(sve_ldddu_zsu, uint32_t, uint64_t, cpu_ldq_data_ra) | ||
241 | -DO_LD1_ZPZ_D(sve_ldbds_zsu, uint32_t, int8_t, cpu_ldub_data_ra) | ||
242 | -DO_LD1_ZPZ_D(sve_ldhds_zsu, uint32_t, int16_t, cpu_lduw_data_ra) | ||
243 | -DO_LD1_ZPZ_D(sve_ldsds_zsu, uint32_t, int32_t, cpu_ldl_data_ra) | ||
244 | +static target_ulong off_zd_d(void *reg, intptr_t reg_ofs) | ||
245 | +{ | ||
246 | + return *(uint64_t *)(reg + reg_ofs); | ||
247 | +} | ||
248 | |||
249 | -DO_LD1_ZPZ_D(sve_ldbdu_zss, int32_t, uint8_t, cpu_ldub_data_ra) | ||
250 | -DO_LD1_ZPZ_D(sve_ldhdu_zss, int32_t, uint16_t, cpu_lduw_data_ra) | ||
251 | -DO_LD1_ZPZ_D(sve_ldsdu_zss, int32_t, uint32_t, cpu_ldl_data_ra) | ||
252 | -DO_LD1_ZPZ_D(sve_ldddu_zss, int32_t, uint64_t, cpu_ldq_data_ra) | ||
253 | -DO_LD1_ZPZ_D(sve_ldbds_zss, int32_t, int8_t, cpu_ldub_data_ra) | ||
254 | -DO_LD1_ZPZ_D(sve_ldhds_zss, int32_t, int16_t, cpu_lduw_data_ra) | ||
255 | -DO_LD1_ZPZ_D(sve_ldsds_zss, int32_t, int32_t, cpu_ldl_data_ra) | ||
256 | +static void sve_ld1_zs(CPUARMState *env, void *vd, void *vg, void *vm, | ||
257 | + target_ulong base, uint32_t desc, uintptr_t ra, | ||
258 | + zreg_off_fn *off_fn, sve_ld1_tlb_fn *tlb_fn) | ||
259 | +{ | ||
260 | + const int mmu_idx = cpu_mmu_index(env, false); | ||
261 | + intptr_t i, oprsz = simd_oprsz(desc); | ||
262 | + unsigned scale = simd_data(desc); | ||
263 | + ARMVectorReg scratch = { }; | ||
264 | |||
265 | -DO_LD1_ZPZ_D(sve_ldbdu_zd, uint64_t, uint8_t, cpu_ldub_data_ra) | ||
266 | -DO_LD1_ZPZ_D(sve_ldhdu_zd, uint64_t, uint16_t, cpu_lduw_data_ra) | ||
267 | -DO_LD1_ZPZ_D(sve_ldsdu_zd, uint64_t, uint32_t, cpu_ldl_data_ra) | ||
268 | -DO_LD1_ZPZ_D(sve_ldddu_zd, uint64_t, uint64_t, cpu_ldq_data_ra) | ||
269 | -DO_LD1_ZPZ_D(sve_ldbds_zd, uint64_t, int8_t, cpu_ldub_data_ra) | ||
270 | -DO_LD1_ZPZ_D(sve_ldhds_zd, uint64_t, int16_t, cpu_lduw_data_ra) | ||
271 | -DO_LD1_ZPZ_D(sve_ldsds_zd, uint64_t, int32_t, cpu_ldl_data_ra) | ||
272 | + set_helper_retaddr(ra); | ||
273 | + for (i = 0; i < oprsz; ) { | ||
274 | + uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); | ||
275 | + do { | ||
276 | + if (likely(pg & 1)) { | ||
277 | + target_ulong off = off_fn(vm, i); | ||
278 | + tlb_fn(env, &scratch, i, base + (off << scale), mmu_idx, ra); | ||
279 | + } | ||
280 | + i += 4, pg >>= 4; | ||
281 | + } while (i & 15); | ||
282 | + } | ||
283 | + set_helper_retaddr(0); | ||
284 | + | ||
285 | + /* Wait until all exceptions have been raised to write back. */ | ||
286 | + memcpy(vd, &scratch, oprsz); | ||
287 | +} | ||
288 | + | ||
289 | +static void sve_ld1_zd(CPUARMState *env, void *vd, void *vg, void *vm, | ||
290 | + target_ulong base, uint32_t desc, uintptr_t ra, | ||
291 | + zreg_off_fn *off_fn, sve_ld1_tlb_fn *tlb_fn) | ||
292 | +{ | ||
293 | + const int mmu_idx = cpu_mmu_index(env, false); | ||
294 | + intptr_t i, oprsz = simd_oprsz(desc) / 8; | ||
295 | + unsigned scale = simd_data(desc); | ||
296 | + ARMVectorReg scratch = { }; | ||
297 | + | ||
298 | + set_helper_retaddr(ra); | ||
299 | + for (i = 0; i < oprsz; i++) { | ||
300 | + uint8_t pg = *(uint8_t *)(vg + H1(i)); | ||
301 | + if (likely(pg & 1)) { | ||
302 | + target_ulong off = off_fn(vm, i * 8); | ||
303 | + tlb_fn(env, &scratch, i * 8, base + (off << scale), mmu_idx, ra); | ||
304 | + } | ||
305 | + } | ||
306 | + set_helper_retaddr(0); | ||
307 | + | ||
308 | + /* Wait until all exceptions have been raised to write back. */ | ||
309 | + memcpy(vd, &scratch, oprsz * 8); | ||
310 | +} | ||
311 | + | ||
312 | +#define DO_LD1_ZPZ_S(MEM, OFS) \ | ||
313 | +void __attribute__((flatten)) HELPER(sve_ld##MEM##_##OFS) \ | ||
314 | + (CPUARMState *env, void *vd, void *vg, void *vm, \ | ||
315 | + target_ulong base, uint32_t desc) \ | ||
316 | +{ \ | ||
317 | + sve_ld1_zs(env, vd, vg, vm, base, desc, GETPC(), \ | ||
318 | + off_##OFS##_s, sve_ld1##MEM##_tlb); \ | ||
319 | +} | ||
320 | + | ||
321 | +#define DO_LD1_ZPZ_D(MEM, OFS) \ | ||
322 | +void __attribute__((flatten)) HELPER(sve_ld##MEM##_##OFS) \ | ||
323 | + (CPUARMState *env, void *vd, void *vg, void *vm, \ | ||
324 | + target_ulong base, uint32_t desc) \ | ||
325 | +{ \ | ||
326 | + sve_ld1_zd(env, vd, vg, vm, base, desc, GETPC(), \ | ||
327 | + off_##OFS##_d, sve_ld1##MEM##_tlb); \ | ||
328 | +} | ||
329 | + | ||
330 | +DO_LD1_ZPZ_S(bsu, zsu) | ||
331 | +DO_LD1_ZPZ_S(bsu, zss) | ||
332 | +DO_LD1_ZPZ_D(bdu, zsu) | ||
333 | +DO_LD1_ZPZ_D(bdu, zss) | ||
334 | +DO_LD1_ZPZ_D(bdu, zd) | ||
335 | + | ||
336 | +DO_LD1_ZPZ_S(bss, zsu) | ||
337 | +DO_LD1_ZPZ_S(bss, zss) | ||
338 | +DO_LD1_ZPZ_D(bds, zsu) | ||
339 | +DO_LD1_ZPZ_D(bds, zss) | ||
340 | +DO_LD1_ZPZ_D(bds, zd) | ||
341 | + | ||
342 | +DO_LD1_ZPZ_S(hsu_le, zsu) | ||
343 | +DO_LD1_ZPZ_S(hsu_le, zss) | ||
344 | +DO_LD1_ZPZ_D(hdu_le, zsu) | ||
345 | +DO_LD1_ZPZ_D(hdu_le, zss) | ||
346 | +DO_LD1_ZPZ_D(hdu_le, zd) | ||
347 | + | ||
348 | +DO_LD1_ZPZ_S(hsu_be, zsu) | ||
349 | +DO_LD1_ZPZ_S(hsu_be, zss) | ||
350 | +DO_LD1_ZPZ_D(hdu_be, zsu) | ||
351 | +DO_LD1_ZPZ_D(hdu_be, zss) | ||
352 | +DO_LD1_ZPZ_D(hdu_be, zd) | ||
353 | + | ||
354 | +DO_LD1_ZPZ_S(hss_le, zsu) | ||
355 | +DO_LD1_ZPZ_S(hss_le, zss) | ||
356 | +DO_LD1_ZPZ_D(hds_le, zsu) | ||
357 | +DO_LD1_ZPZ_D(hds_le, zss) | ||
358 | +DO_LD1_ZPZ_D(hds_le, zd) | ||
359 | + | ||
360 | +DO_LD1_ZPZ_S(hss_be, zsu) | ||
361 | +DO_LD1_ZPZ_S(hss_be, zss) | ||
362 | +DO_LD1_ZPZ_D(hds_be, zsu) | ||
363 | +DO_LD1_ZPZ_D(hds_be, zss) | ||
364 | +DO_LD1_ZPZ_D(hds_be, zd) | ||
365 | + | ||
366 | +DO_LD1_ZPZ_S(ss_le, zsu) | ||
367 | +DO_LD1_ZPZ_S(ss_le, zss) | ||
368 | +DO_LD1_ZPZ_D(sdu_le, zsu) | ||
369 | +DO_LD1_ZPZ_D(sdu_le, zss) | ||
370 | +DO_LD1_ZPZ_D(sdu_le, zd) | ||
371 | + | ||
372 | +DO_LD1_ZPZ_S(ss_be, zsu) | ||
373 | +DO_LD1_ZPZ_S(ss_be, zss) | ||
374 | +DO_LD1_ZPZ_D(sdu_be, zsu) | ||
375 | +DO_LD1_ZPZ_D(sdu_be, zss) | ||
376 | +DO_LD1_ZPZ_D(sdu_be, zd) | ||
377 | + | ||
378 | +DO_LD1_ZPZ_D(sds_le, zsu) | ||
379 | +DO_LD1_ZPZ_D(sds_le, zss) | ||
380 | +DO_LD1_ZPZ_D(sds_le, zd) | ||
381 | + | ||
382 | +DO_LD1_ZPZ_D(sds_be, zsu) | ||
383 | +DO_LD1_ZPZ_D(sds_be, zss) | ||
384 | +DO_LD1_ZPZ_D(sds_be, zd) | ||
385 | + | ||
386 | +DO_LD1_ZPZ_D(dd_le, zsu) | ||
387 | +DO_LD1_ZPZ_D(dd_le, zss) | ||
388 | +DO_LD1_ZPZ_D(dd_le, zd) | ||
389 | + | ||
390 | +DO_LD1_ZPZ_D(dd_be, zsu) | ||
391 | +DO_LD1_ZPZ_D(dd_be, zss) | ||
392 | +DO_LD1_ZPZ_D(dd_be, zd) | ||
393 | + | ||
394 | +#undef DO_LD1_ZPZ_S | ||
395 | +#undef DO_LD1_ZPZ_D | ||
396 | |||
397 | /* First fault loads with a vector index. */ | ||
398 | |||
399 | diff --git a/target/arm/translate-sve.c b/target/arm/translate-sve.c | ||
400 | index XXXXXXX..XXXXXXX 100644 | ||
401 | --- a/target/arm/translate-sve.c | ||
402 | +++ b/target/arm/translate-sve.c | ||
403 | @@ -XXX,XX +XXX,XX @@ static void do_mem_zpz(DisasContext *s, int zt, int pg, int zm, int scale, | ||
404 | tcg_temp_free_i32(desc); | ||
405 | } | ||
406 | |||
407 | -/* Indexed by [ff][xs][u][msz]. */ | ||
408 | -static gen_helper_gvec_mem_scatter * const gather_load_fn32[2][2][2][3] = { | ||
409 | - { { { gen_helper_sve_ldbss_zsu, | ||
410 | - gen_helper_sve_ldhss_zsu, | ||
411 | - NULL, }, | ||
412 | - { gen_helper_sve_ldbsu_zsu, | ||
413 | - gen_helper_sve_ldhsu_zsu, | ||
414 | - gen_helper_sve_ldssu_zsu, } }, | ||
415 | - { { gen_helper_sve_ldbss_zss, | ||
416 | - gen_helper_sve_ldhss_zss, | ||
417 | - NULL, }, | ||
418 | - { gen_helper_sve_ldbsu_zss, | ||
419 | - gen_helper_sve_ldhsu_zss, | ||
420 | - gen_helper_sve_ldssu_zss, } } }, | ||
421 | +/* Indexed by [be][ff][xs][u][msz]. */ | ||
422 | +static gen_helper_gvec_mem_scatter * const gather_load_fn32[2][2][2][2][3] = { | ||
423 | + /* Little-endian */ | ||
424 | + { { { { gen_helper_sve_ldbss_zsu, | ||
425 | + gen_helper_sve_ldhss_le_zsu, | ||
426 | + NULL, }, | ||
427 | + { gen_helper_sve_ldbsu_zsu, | ||
428 | + gen_helper_sve_ldhsu_le_zsu, | ||
429 | + gen_helper_sve_ldss_le_zsu, } }, | ||
430 | + { { gen_helper_sve_ldbss_zss, | ||
431 | + gen_helper_sve_ldhss_le_zss, | ||
432 | + NULL, }, | ||
433 | + { gen_helper_sve_ldbsu_zss, | ||
434 | + gen_helper_sve_ldhsu_le_zss, | ||
435 | + gen_helper_sve_ldss_le_zss, } } }, | ||
436 | |||
437 | - { { { gen_helper_sve_ldffbss_zsu, | ||
438 | - gen_helper_sve_ldffhss_zsu, | ||
439 | - NULL, }, | ||
440 | - { gen_helper_sve_ldffbsu_zsu, | ||
441 | - gen_helper_sve_ldffhsu_zsu, | ||
442 | - gen_helper_sve_ldffssu_zsu, } }, | ||
443 | - { { gen_helper_sve_ldffbss_zss, | ||
444 | - gen_helper_sve_ldffhss_zss, | ||
445 | - NULL, }, | ||
446 | - { gen_helper_sve_ldffbsu_zss, | ||
447 | - gen_helper_sve_ldffhsu_zss, | ||
448 | - gen_helper_sve_ldffssu_zss, } } } | ||
449 | + /* First-fault */ | ||
450 | + { { { gen_helper_sve_ldffbss_zsu, | ||
451 | + gen_helper_sve_ldffhss_zsu, | ||
452 | + NULL, }, | ||
453 | + { gen_helper_sve_ldffbsu_zsu, | ||
454 | + gen_helper_sve_ldffhsu_zsu, | ||
455 | + gen_helper_sve_ldffssu_zsu, } }, | ||
456 | + { { gen_helper_sve_ldffbss_zss, | ||
457 | + gen_helper_sve_ldffhss_zss, | ||
458 | + NULL, }, | ||
459 | + { gen_helper_sve_ldffbsu_zss, | ||
460 | + gen_helper_sve_ldffhsu_zss, | ||
461 | + gen_helper_sve_ldffssu_zss, } } } }, | ||
462 | + | ||
463 | + /* Big-endian */ | ||
464 | + { { { { gen_helper_sve_ldbss_zsu, | ||
465 | + gen_helper_sve_ldhss_be_zsu, | ||
466 | + NULL, }, | ||
467 | + { gen_helper_sve_ldbsu_zsu, | ||
468 | + gen_helper_sve_ldhsu_be_zsu, | ||
469 | + gen_helper_sve_ldss_be_zsu, } }, | ||
470 | + { { gen_helper_sve_ldbss_zss, | ||
471 | + gen_helper_sve_ldhss_be_zss, | ||
472 | + NULL, }, | ||
473 | + { gen_helper_sve_ldbsu_zss, | ||
474 | + gen_helper_sve_ldhsu_be_zss, | ||
475 | + gen_helper_sve_ldss_be_zss, } } }, | ||
476 | + | ||
477 | + /* First-fault */ | ||
478 | + { { { gen_helper_sve_ldffbss_zsu, | ||
479 | + gen_helper_sve_ldffhss_zsu, | ||
480 | + NULL, }, | ||
481 | + { gen_helper_sve_ldffbsu_zsu, | ||
482 | + gen_helper_sve_ldffhsu_zsu, | ||
483 | + gen_helper_sve_ldffssu_zsu, } }, | ||
484 | + { { gen_helper_sve_ldffbss_zss, | ||
485 | + gen_helper_sve_ldffhss_zss, | ||
486 | + NULL, }, | ||
487 | + { gen_helper_sve_ldffbsu_zss, | ||
488 | + gen_helper_sve_ldffhsu_zss, | ||
489 | + gen_helper_sve_ldffssu_zss, } } } }, | ||
490 | }; | ||
491 | |||
492 | /* Note that we overload xs=2 to indicate 64-bit offset. */ | ||
493 | -static gen_helper_gvec_mem_scatter * const gather_load_fn64[2][3][2][4] = { | ||
494 | - { { { gen_helper_sve_ldbds_zsu, | ||
495 | - gen_helper_sve_ldhds_zsu, | ||
496 | - gen_helper_sve_ldsds_zsu, | ||
497 | - NULL, }, | ||
498 | - { gen_helper_sve_ldbdu_zsu, | ||
499 | - gen_helper_sve_ldhdu_zsu, | ||
500 | - gen_helper_sve_ldsdu_zsu, | ||
501 | - gen_helper_sve_ldddu_zsu, } }, | ||
502 | - { { gen_helper_sve_ldbds_zss, | ||
503 | - gen_helper_sve_ldhds_zss, | ||
504 | - gen_helper_sve_ldsds_zss, | ||
505 | - NULL, }, | ||
506 | - { gen_helper_sve_ldbdu_zss, | ||
507 | - gen_helper_sve_ldhdu_zss, | ||
508 | - gen_helper_sve_ldsdu_zss, | ||
509 | - gen_helper_sve_ldddu_zss, } }, | ||
510 | - { { gen_helper_sve_ldbds_zd, | ||
511 | - gen_helper_sve_ldhds_zd, | ||
512 | - gen_helper_sve_ldsds_zd, | ||
513 | - NULL, }, | ||
514 | - { gen_helper_sve_ldbdu_zd, | ||
515 | - gen_helper_sve_ldhdu_zd, | ||
516 | - gen_helper_sve_ldsdu_zd, | ||
517 | - gen_helper_sve_ldddu_zd, } } }, | ||
518 | +static gen_helper_gvec_mem_scatter * const gather_load_fn64[2][2][3][2][4] = { | ||
519 | + /* Little-endian */ | ||
520 | + { { { { gen_helper_sve_ldbds_zsu, | ||
521 | + gen_helper_sve_ldhds_le_zsu, | ||
522 | + gen_helper_sve_ldsds_le_zsu, | ||
523 | + NULL, }, | ||
524 | + { gen_helper_sve_ldbdu_zsu, | ||
525 | + gen_helper_sve_ldhdu_le_zsu, | ||
526 | + gen_helper_sve_ldsdu_le_zsu, | ||
527 | + gen_helper_sve_lddd_le_zsu, } }, | ||
528 | + { { gen_helper_sve_ldbds_zss, | ||
529 | + gen_helper_sve_ldhds_le_zss, | ||
530 | + gen_helper_sve_ldsds_le_zss, | ||
531 | + NULL, }, | ||
532 | + { gen_helper_sve_ldbdu_zss, | ||
533 | + gen_helper_sve_ldhdu_le_zss, | ||
534 | + gen_helper_sve_ldsdu_le_zss, | ||
535 | + gen_helper_sve_lddd_le_zss, } }, | ||
536 | + { { gen_helper_sve_ldbds_zd, | ||
537 | + gen_helper_sve_ldhds_le_zd, | ||
538 | + gen_helper_sve_ldsds_le_zd, | ||
539 | + NULL, }, | ||
540 | + { gen_helper_sve_ldbdu_zd, | ||
541 | + gen_helper_sve_ldhdu_le_zd, | ||
542 | + gen_helper_sve_ldsdu_le_zd, | ||
543 | + gen_helper_sve_lddd_le_zd, } } }, | ||
544 | |||
545 | - { { { gen_helper_sve_ldffbds_zsu, | ||
546 | - gen_helper_sve_ldffhds_zsu, | ||
547 | - gen_helper_sve_ldffsds_zsu, | ||
548 | - NULL, }, | ||
549 | - { gen_helper_sve_ldffbdu_zsu, | ||
550 | - gen_helper_sve_ldffhdu_zsu, | ||
551 | - gen_helper_sve_ldffsdu_zsu, | ||
552 | - gen_helper_sve_ldffddu_zsu, } }, | ||
553 | - { { gen_helper_sve_ldffbds_zss, | ||
554 | - gen_helper_sve_ldffhds_zss, | ||
555 | - gen_helper_sve_ldffsds_zss, | ||
556 | - NULL, }, | ||
557 | - { gen_helper_sve_ldffbdu_zss, | ||
558 | - gen_helper_sve_ldffhdu_zss, | ||
559 | - gen_helper_sve_ldffsdu_zss, | ||
560 | - gen_helper_sve_ldffddu_zss, } }, | ||
561 | - { { gen_helper_sve_ldffbds_zd, | ||
562 | - gen_helper_sve_ldffhds_zd, | ||
563 | - gen_helper_sve_ldffsds_zd, | ||
564 | - NULL, }, | ||
565 | - { gen_helper_sve_ldffbdu_zd, | ||
566 | - gen_helper_sve_ldffhdu_zd, | ||
567 | - gen_helper_sve_ldffsdu_zd, | ||
568 | - gen_helper_sve_ldffddu_zd, } } } | ||
569 | + /* First-fault */ | ||
570 | + { { { gen_helper_sve_ldffbds_zsu, | ||
571 | + gen_helper_sve_ldffhds_zsu, | ||
572 | + gen_helper_sve_ldffsds_zsu, | ||
573 | + NULL, }, | ||
574 | + { gen_helper_sve_ldffbdu_zsu, | ||
575 | + gen_helper_sve_ldffhdu_zsu, | ||
576 | + gen_helper_sve_ldffsdu_zsu, | ||
577 | + gen_helper_sve_ldffddu_zsu, } }, | ||
578 | + { { gen_helper_sve_ldffbds_zss, | ||
579 | + gen_helper_sve_ldffhds_zss, | ||
580 | + gen_helper_sve_ldffsds_zss, | ||
581 | + NULL, }, | ||
582 | + { gen_helper_sve_ldffbdu_zss, | ||
583 | + gen_helper_sve_ldffhdu_zss, | ||
584 | + gen_helper_sve_ldffsdu_zss, | ||
585 | + gen_helper_sve_ldffddu_zss, } }, | ||
586 | + { { gen_helper_sve_ldffbds_zd, | ||
587 | + gen_helper_sve_ldffhds_zd, | ||
588 | + gen_helper_sve_ldffsds_zd, | ||
589 | + NULL, }, | ||
590 | + { gen_helper_sve_ldffbdu_zd, | ||
591 | + gen_helper_sve_ldffhdu_zd, | ||
592 | + gen_helper_sve_ldffsdu_zd, | ||
593 | + gen_helper_sve_ldffddu_zd, } } } }, | ||
594 | + | ||
595 | + /* Big-endian */ | ||
596 | + { { { { gen_helper_sve_ldbds_zsu, | ||
597 | + gen_helper_sve_ldhds_be_zsu, | ||
598 | + gen_helper_sve_ldsds_be_zsu, | ||
599 | + NULL, }, | ||
600 | + { gen_helper_sve_ldbdu_zsu, | ||
601 | + gen_helper_sve_ldhdu_be_zsu, | ||
602 | + gen_helper_sve_ldsdu_be_zsu, | ||
603 | + gen_helper_sve_lddd_be_zsu, } }, | ||
604 | + { { gen_helper_sve_ldbds_zss, | ||
605 | + gen_helper_sve_ldhds_be_zss, | ||
606 | + gen_helper_sve_ldsds_be_zss, | ||
607 | + NULL, }, | ||
608 | + { gen_helper_sve_ldbdu_zss, | ||
609 | + gen_helper_sve_ldhdu_be_zss, | ||
610 | + gen_helper_sve_ldsdu_be_zss, | ||
611 | + gen_helper_sve_lddd_be_zss, } }, | ||
612 | + { { gen_helper_sve_ldbds_zd, | ||
613 | + gen_helper_sve_ldhds_be_zd, | ||
614 | + gen_helper_sve_ldsds_be_zd, | ||
615 | + NULL, }, | ||
616 | + { gen_helper_sve_ldbdu_zd, | ||
617 | + gen_helper_sve_ldhdu_be_zd, | ||
618 | + gen_helper_sve_ldsdu_be_zd, | ||
619 | + gen_helper_sve_lddd_be_zd, } } }, | ||
620 | + | ||
621 | + /* First-fault */ | ||
622 | + { { { gen_helper_sve_ldffbds_zsu, | ||
623 | + gen_helper_sve_ldffhds_zsu, | ||
624 | + gen_helper_sve_ldffsds_zsu, | ||
625 | + NULL, }, | ||
626 | + { gen_helper_sve_ldffbdu_zsu, | ||
627 | + gen_helper_sve_ldffhdu_zsu, | ||
628 | + gen_helper_sve_ldffsdu_zsu, | ||
629 | + gen_helper_sve_ldffddu_zsu, } }, | ||
630 | + { { gen_helper_sve_ldffbds_zss, | ||
631 | + gen_helper_sve_ldffhds_zss, | ||
632 | + gen_helper_sve_ldffsds_zss, | ||
633 | + NULL, }, | ||
634 | + { gen_helper_sve_ldffbdu_zss, | ||
635 | + gen_helper_sve_ldffhdu_zss, | ||
636 | + gen_helper_sve_ldffsdu_zss, | ||
637 | + gen_helper_sve_ldffddu_zss, } }, | ||
638 | + { { gen_helper_sve_ldffbds_zd, | ||
639 | + gen_helper_sve_ldffhds_zd, | ||
640 | + gen_helper_sve_ldffsds_zd, | ||
641 | + NULL, }, | ||
642 | + { gen_helper_sve_ldffbdu_zd, | ||
643 | + gen_helper_sve_ldffhdu_zd, | ||
644 | + gen_helper_sve_ldffsdu_zd, | ||
645 | + gen_helper_sve_ldffddu_zd, } } } }, | ||
646 | }; | ||
647 | |||
648 | static bool trans_LD1_zprz(DisasContext *s, arg_LD1_zprz *a, uint32_t insn) | ||
649 | { | ||
650 | gen_helper_gvec_mem_scatter *fn = NULL; | ||
651 | + int be = s->be_data == MO_BE; | ||
652 | |||
653 | if (!sve_access_check(s)) { | ||
654 | return true; | ||
655 | @@ -XXX,XX +XXX,XX @@ static bool trans_LD1_zprz(DisasContext *s, arg_LD1_zprz *a, uint32_t insn) | ||
656 | |||
657 | switch (a->esz) { | ||
658 | case MO_32: | ||
659 | - fn = gather_load_fn32[a->ff][a->xs][a->u][a->msz]; | ||
660 | + fn = gather_load_fn32[be][a->ff][a->xs][a->u][a->msz]; | ||
661 | break; | ||
662 | case MO_64: | ||
663 | - fn = gather_load_fn64[a->ff][a->xs][a->u][a->msz]; | ||
664 | + fn = gather_load_fn64[be][a->ff][a->xs][a->u][a->msz]; | ||
665 | break; | ||
666 | } | ||
667 | assert(fn != NULL); | ||
668 | @@ -XXX,XX +XXX,XX @@ static bool trans_LD1_zprz(DisasContext *s, arg_LD1_zprz *a, uint32_t insn) | ||
669 | static bool trans_LD1_zpiz(DisasContext *s, arg_LD1_zpiz *a, uint32_t insn) | ||
670 | { | ||
671 | gen_helper_gvec_mem_scatter *fn = NULL; | ||
672 | + int be = s->be_data == MO_BE; | ||
673 | TCGv_i64 imm; | ||
674 | |||
675 | if (a->esz < a->msz || (a->esz == a->msz && !a->u)) { | ||
676 | @@ -XXX,XX +XXX,XX @@ static bool trans_LD1_zpiz(DisasContext *s, arg_LD1_zpiz *a, uint32_t insn) | ||
677 | |||
678 | switch (a->esz) { | ||
679 | case MO_32: | ||
680 | - fn = gather_load_fn32[a->ff][0][a->u][a->msz]; | ||
681 | + fn = gather_load_fn32[be][a->ff][0][a->u][a->msz]; | ||
682 | break; | ||
683 | case MO_64: | ||
684 | - fn = gather_load_fn64[a->ff][2][a->u][a->msz]; | ||
685 | + fn = gather_load_fn64[be][a->ff][2][a->u][a->msz]; | ||
686 | break; | ||
687 | } | ||
688 | assert(fn != NULL); | ||
689 | -- | ||
690 | 2.19.0 | ||
691 | |||
692 | diff view generated by jsdifflib |
Deleted patch | |||
---|---|---|---|
1 | From: Richard Henderson <richard.henderson@linaro.org> | ||
2 | 1 | ||
3 | This fixes the endianness problem for softmmu, and moves | ||
4 | the main loop out of a macro and into an inlined function. | ||
5 | |||
6 | Reviewed-by: Peter Maydell <peter.maydell@linaro.org> | ||
7 | Tested-by: Laurent Desnogues <laurent.desnogues@gmail.com> | ||
8 | Signed-off-by: Richard Henderson <richard.henderson@linaro.org> | ||
9 | Message-id: 20181005175350.30752-14-richard.henderson@linaro.org | ||
10 | Signed-off-by: Peter Maydell <peter.maydell@linaro.org> | ||
11 | --- | ||
12 | target/arm/helper-sve.h | 52 ++++++++++---- | ||
13 | target/arm/sve_helper.c | 139 ++++++++++++++++++++++++------------- | ||
14 | target/arm/translate-sve.c | 74 +++++++++++++------- | ||
15 | 3 files changed, 177 insertions(+), 88 deletions(-) | ||
16 | |||
17 | diff --git a/target/arm/helper-sve.h b/target/arm/helper-sve.h | ||
18 | index XXXXXXX..XXXXXXX 100644 | ||
19 | --- a/target/arm/helper-sve.h | ||
20 | +++ b/target/arm/helper-sve.h | ||
21 | @@ -XXX,XX +XXX,XX @@ DEF_HELPER_FLAGS_6(sve_ldffsds_zd, TCG_CALL_NO_WG, | ||
22 | |||
23 | DEF_HELPER_FLAGS_6(sve_stbs_zsu, TCG_CALL_NO_WG, | ||
24 | void, env, ptr, ptr, ptr, tl, i32) | ||
25 | -DEF_HELPER_FLAGS_6(sve_sths_zsu, TCG_CALL_NO_WG, | ||
26 | +DEF_HELPER_FLAGS_6(sve_sths_le_zsu, TCG_CALL_NO_WG, | ||
27 | void, env, ptr, ptr, ptr, tl, i32) | ||
28 | -DEF_HELPER_FLAGS_6(sve_stss_zsu, TCG_CALL_NO_WG, | ||
29 | +DEF_HELPER_FLAGS_6(sve_sths_be_zsu, TCG_CALL_NO_WG, | ||
30 | + void, env, ptr, ptr, ptr, tl, i32) | ||
31 | +DEF_HELPER_FLAGS_6(sve_stss_le_zsu, TCG_CALL_NO_WG, | ||
32 | + void, env, ptr, ptr, ptr, tl, i32) | ||
33 | +DEF_HELPER_FLAGS_6(sve_stss_be_zsu, TCG_CALL_NO_WG, | ||
34 | void, env, ptr, ptr, ptr, tl, i32) | ||
35 | |||
36 | DEF_HELPER_FLAGS_6(sve_stbs_zss, TCG_CALL_NO_WG, | ||
37 | void, env, ptr, ptr, ptr, tl, i32) | ||
38 | -DEF_HELPER_FLAGS_6(sve_sths_zss, TCG_CALL_NO_WG, | ||
39 | +DEF_HELPER_FLAGS_6(sve_sths_le_zss, TCG_CALL_NO_WG, | ||
40 | void, env, ptr, ptr, ptr, tl, i32) | ||
41 | -DEF_HELPER_FLAGS_6(sve_stss_zss, TCG_CALL_NO_WG, | ||
42 | +DEF_HELPER_FLAGS_6(sve_sths_be_zss, TCG_CALL_NO_WG, | ||
43 | + void, env, ptr, ptr, ptr, tl, i32) | ||
44 | +DEF_HELPER_FLAGS_6(sve_stss_le_zss, TCG_CALL_NO_WG, | ||
45 | + void, env, ptr, ptr, ptr, tl, i32) | ||
46 | +DEF_HELPER_FLAGS_6(sve_stss_be_zss, TCG_CALL_NO_WG, | ||
47 | void, env, ptr, ptr, ptr, tl, i32) | ||
48 | |||
49 | DEF_HELPER_FLAGS_6(sve_stbd_zsu, TCG_CALL_NO_WG, | ||
50 | void, env, ptr, ptr, ptr, tl, i32) | ||
51 | -DEF_HELPER_FLAGS_6(sve_sthd_zsu, TCG_CALL_NO_WG, | ||
52 | +DEF_HELPER_FLAGS_6(sve_sthd_le_zsu, TCG_CALL_NO_WG, | ||
53 | void, env, ptr, ptr, ptr, tl, i32) | ||
54 | -DEF_HELPER_FLAGS_6(sve_stsd_zsu, TCG_CALL_NO_WG, | ||
55 | +DEF_HELPER_FLAGS_6(sve_sthd_be_zsu, TCG_CALL_NO_WG, | ||
56 | void, env, ptr, ptr, ptr, tl, i32) | ||
57 | -DEF_HELPER_FLAGS_6(sve_stdd_zsu, TCG_CALL_NO_WG, | ||
58 | +DEF_HELPER_FLAGS_6(sve_stsd_le_zsu, TCG_CALL_NO_WG, | ||
59 | + void, env, ptr, ptr, ptr, tl, i32) | ||
60 | +DEF_HELPER_FLAGS_6(sve_stsd_be_zsu, TCG_CALL_NO_WG, | ||
61 | + void, env, ptr, ptr, ptr, tl, i32) | ||
62 | +DEF_HELPER_FLAGS_6(sve_stdd_le_zsu, TCG_CALL_NO_WG, | ||
63 | + void, env, ptr, ptr, ptr, tl, i32) | ||
64 | +DEF_HELPER_FLAGS_6(sve_stdd_be_zsu, TCG_CALL_NO_WG, | ||
65 | void, env, ptr, ptr, ptr, tl, i32) | ||
66 | |||
67 | DEF_HELPER_FLAGS_6(sve_stbd_zss, TCG_CALL_NO_WG, | ||
68 | void, env, ptr, ptr, ptr, tl, i32) | ||
69 | -DEF_HELPER_FLAGS_6(sve_sthd_zss, TCG_CALL_NO_WG, | ||
70 | +DEF_HELPER_FLAGS_6(sve_sthd_le_zss, TCG_CALL_NO_WG, | ||
71 | void, env, ptr, ptr, ptr, tl, i32) | ||
72 | -DEF_HELPER_FLAGS_6(sve_stsd_zss, TCG_CALL_NO_WG, | ||
73 | +DEF_HELPER_FLAGS_6(sve_sthd_be_zss, TCG_CALL_NO_WG, | ||
74 | void, env, ptr, ptr, ptr, tl, i32) | ||
75 | -DEF_HELPER_FLAGS_6(sve_stdd_zss, TCG_CALL_NO_WG, | ||
76 | +DEF_HELPER_FLAGS_6(sve_stsd_le_zss, TCG_CALL_NO_WG, | ||
77 | + void, env, ptr, ptr, ptr, tl, i32) | ||
78 | +DEF_HELPER_FLAGS_6(sve_stsd_be_zss, TCG_CALL_NO_WG, | ||
79 | + void, env, ptr, ptr, ptr, tl, i32) | ||
80 | +DEF_HELPER_FLAGS_6(sve_stdd_le_zss, TCG_CALL_NO_WG, | ||
81 | + void, env, ptr, ptr, ptr, tl, i32) | ||
82 | +DEF_HELPER_FLAGS_6(sve_stdd_be_zss, TCG_CALL_NO_WG, | ||
83 | void, env, ptr, ptr, ptr, tl, i32) | ||
84 | |||
85 | DEF_HELPER_FLAGS_6(sve_stbd_zd, TCG_CALL_NO_WG, | ||
86 | void, env, ptr, ptr, ptr, tl, i32) | ||
87 | -DEF_HELPER_FLAGS_6(sve_sthd_zd, TCG_CALL_NO_WG, | ||
88 | +DEF_HELPER_FLAGS_6(sve_sthd_le_zd, TCG_CALL_NO_WG, | ||
89 | void, env, ptr, ptr, ptr, tl, i32) | ||
90 | -DEF_HELPER_FLAGS_6(sve_stsd_zd, TCG_CALL_NO_WG, | ||
91 | +DEF_HELPER_FLAGS_6(sve_sthd_be_zd, TCG_CALL_NO_WG, | ||
92 | void, env, ptr, ptr, ptr, tl, i32) | ||
93 | -DEF_HELPER_FLAGS_6(sve_stdd_zd, TCG_CALL_NO_WG, | ||
94 | +DEF_HELPER_FLAGS_6(sve_stsd_le_zd, TCG_CALL_NO_WG, | ||
95 | + void, env, ptr, ptr, ptr, tl, i32) | ||
96 | +DEF_HELPER_FLAGS_6(sve_stsd_be_zd, TCG_CALL_NO_WG, | ||
97 | + void, env, ptr, ptr, ptr, tl, i32) | ||
98 | +DEF_HELPER_FLAGS_6(sve_stdd_le_zd, TCG_CALL_NO_WG, | ||
99 | + void, env, ptr, ptr, ptr, tl, i32) | ||
100 | +DEF_HELPER_FLAGS_6(sve_stdd_be_zd, TCG_CALL_NO_WG, | ||
101 | void, env, ptr, ptr, ptr, tl, i32) | ||
102 | diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c | ||
103 | index XXXXXXX..XXXXXXX 100644 | ||
104 | --- a/target/arm/sve_helper.c | ||
105 | +++ b/target/arm/sve_helper.c | ||
106 | @@ -XXX,XX +XXX,XX @@ DO_LDFF1_ZPZ_D(sve_ldffsds_zd, uint64_t, int32_t, cpu_ldl_data_ra) | ||
107 | |||
108 | /* Stores with a vector index. */ | ||
109 | |||
110 | -#define DO_ST1_ZPZ_S(NAME, TYPEI, FN) \ | ||
111 | -void HELPER(NAME)(CPUARMState *env, void *vd, void *vg, void *vm, \ | ||
112 | - target_ulong base, uint32_t desc) \ | ||
113 | -{ \ | ||
114 | - intptr_t i, oprsz = simd_oprsz(desc); \ | ||
115 | - unsigned scale = simd_data(desc); \ | ||
116 | - uintptr_t ra = GETPC(); \ | ||
117 | - for (i = 0; i < oprsz; ) { \ | ||
118 | - uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); \ | ||
119 | - do { \ | ||
120 | - if (likely(pg & 1)) { \ | ||
121 | - target_ulong off = *(TYPEI *)(vm + H1_4(i)); \ | ||
122 | - uint32_t d = *(uint32_t *)(vd + H1_4(i)); \ | ||
123 | - FN(env, base + (off << scale), d, ra); \ | ||
124 | - } \ | ||
125 | - i += sizeof(uint32_t), pg >>= sizeof(uint32_t); \ | ||
126 | - } while (i & 15); \ | ||
127 | - } \ | ||
128 | +static void sve_st1_zs(CPUARMState *env, void *vd, void *vg, void *vm, | ||
129 | + target_ulong base, uint32_t desc, uintptr_t ra, | ||
130 | + zreg_off_fn *off_fn, sve_ld1_tlb_fn *tlb_fn) | ||
131 | +{ | ||
132 | + const int mmu_idx = cpu_mmu_index(env, false); | ||
133 | + intptr_t i, oprsz = simd_oprsz(desc); | ||
134 | + unsigned scale = simd_data(desc); | ||
135 | + | ||
136 | + set_helper_retaddr(ra); | ||
137 | + for (i = 0; i < oprsz; ) { | ||
138 | + uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); | ||
139 | + do { | ||
140 | + if (likely(pg & 1)) { | ||
141 | + target_ulong off = off_fn(vm, i); | ||
142 | + tlb_fn(env, vd, i, base + (off << scale), mmu_idx, ra); | ||
143 | + } | ||
144 | + i += 4, pg >>= 4; | ||
145 | + } while (i & 15); | ||
146 | + } | ||
147 | + set_helper_retaddr(0); | ||
148 | } | ||
149 | |||
150 | -#define DO_ST1_ZPZ_D(NAME, TYPEI, FN) \ | ||
151 | -void HELPER(NAME)(CPUARMState *env, void *vd, void *vg, void *vm, \ | ||
152 | - target_ulong base, uint32_t desc) \ | ||
153 | -{ \ | ||
154 | - intptr_t i, oprsz = simd_oprsz(desc) / 8; \ | ||
155 | - unsigned scale = simd_data(desc); \ | ||
156 | - uintptr_t ra = GETPC(); \ | ||
157 | - uint64_t *d = vd, *m = vm; uint8_t *pg = vg; \ | ||
158 | - for (i = 0; i < oprsz; i++) { \ | ||
159 | - if (likely(pg[H1(i)] & 1)) { \ | ||
160 | - target_ulong off = (target_ulong)(TYPEI)m[i] << scale; \ | ||
161 | - FN(env, base + off, d[i], ra); \ | ||
162 | - } \ | ||
163 | - } \ | ||
164 | +static void sve_st1_zd(CPUARMState *env, void *vd, void *vg, void *vm, | ||
165 | + target_ulong base, uint32_t desc, uintptr_t ra, | ||
166 | + zreg_off_fn *off_fn, sve_ld1_tlb_fn *tlb_fn) | ||
167 | +{ | ||
168 | + const int mmu_idx = cpu_mmu_index(env, false); | ||
169 | + intptr_t i, oprsz = simd_oprsz(desc) / 8; | ||
170 | + unsigned scale = simd_data(desc); | ||
171 | + | ||
172 | + set_helper_retaddr(ra); | ||
173 | + for (i = 0; i < oprsz; i++) { | ||
174 | + uint8_t pg = *(uint8_t *)(vg + H1(i)); | ||
175 | + if (likely(pg & 1)) { | ||
176 | + target_ulong off = off_fn(vm, i * 8); | ||
177 | + tlb_fn(env, vd, i * 8, base + (off << scale), mmu_idx, ra); | ||
178 | + } | ||
179 | + } | ||
180 | + set_helper_retaddr(0); | ||
181 | } | ||
182 | |||
183 | -DO_ST1_ZPZ_S(sve_stbs_zsu, uint32_t, cpu_stb_data_ra) | ||
184 | -DO_ST1_ZPZ_S(sve_sths_zsu, uint32_t, cpu_stw_data_ra) | ||
185 | -DO_ST1_ZPZ_S(sve_stss_zsu, uint32_t, cpu_stl_data_ra) | ||
186 | +#define DO_ST1_ZPZ_S(MEM, OFS) \ | ||
187 | +void __attribute__((flatten)) HELPER(sve_st##MEM##_##OFS) \ | ||
188 | + (CPUARMState *env, void *vd, void *vg, void *vm, \ | ||
189 | + target_ulong base, uint32_t desc) \ | ||
190 | +{ \ | ||
191 | + sve_st1_zs(env, vd, vg, vm, base, desc, GETPC(), \ | ||
192 | + off_##OFS##_s, sve_st1##MEM##_tlb); \ | ||
193 | +} | ||
194 | |||
195 | -DO_ST1_ZPZ_S(sve_stbs_zss, int32_t, cpu_stb_data_ra) | ||
196 | -DO_ST1_ZPZ_S(sve_sths_zss, int32_t, cpu_stw_data_ra) | ||
197 | -DO_ST1_ZPZ_S(sve_stss_zss, int32_t, cpu_stl_data_ra) | ||
198 | +#define DO_ST1_ZPZ_D(MEM, OFS) \ | ||
199 | +void __attribute__((flatten)) HELPER(sve_st##MEM##_##OFS) \ | ||
200 | + (CPUARMState *env, void *vd, void *vg, void *vm, \ | ||
201 | + target_ulong base, uint32_t desc) \ | ||
202 | +{ \ | ||
203 | + sve_st1_zd(env, vd, vg, vm, base, desc, GETPC(), \ | ||
204 | + off_##OFS##_d, sve_st1##MEM##_tlb); \ | ||
205 | +} | ||
206 | |||
207 | -DO_ST1_ZPZ_D(sve_stbd_zsu, uint32_t, cpu_stb_data_ra) | ||
208 | -DO_ST1_ZPZ_D(sve_sthd_zsu, uint32_t, cpu_stw_data_ra) | ||
209 | -DO_ST1_ZPZ_D(sve_stsd_zsu, uint32_t, cpu_stl_data_ra) | ||
210 | -DO_ST1_ZPZ_D(sve_stdd_zsu, uint32_t, cpu_stq_data_ra) | ||
211 | +DO_ST1_ZPZ_S(bs, zsu) | ||
212 | +DO_ST1_ZPZ_S(hs_le, zsu) | ||
213 | +DO_ST1_ZPZ_S(hs_be, zsu) | ||
214 | +DO_ST1_ZPZ_S(ss_le, zsu) | ||
215 | +DO_ST1_ZPZ_S(ss_be, zsu) | ||
216 | |||
217 | -DO_ST1_ZPZ_D(sve_stbd_zss, int32_t, cpu_stb_data_ra) | ||
218 | -DO_ST1_ZPZ_D(sve_sthd_zss, int32_t, cpu_stw_data_ra) | ||
219 | -DO_ST1_ZPZ_D(sve_stsd_zss, int32_t, cpu_stl_data_ra) | ||
220 | -DO_ST1_ZPZ_D(sve_stdd_zss, int32_t, cpu_stq_data_ra) | ||
221 | +DO_ST1_ZPZ_S(bs, zss) | ||
222 | +DO_ST1_ZPZ_S(hs_le, zss) | ||
223 | +DO_ST1_ZPZ_S(hs_be, zss) | ||
224 | +DO_ST1_ZPZ_S(ss_le, zss) | ||
225 | +DO_ST1_ZPZ_S(ss_be, zss) | ||
226 | |||
227 | -DO_ST1_ZPZ_D(sve_stbd_zd, uint64_t, cpu_stb_data_ra) | ||
228 | -DO_ST1_ZPZ_D(sve_sthd_zd, uint64_t, cpu_stw_data_ra) | ||
229 | -DO_ST1_ZPZ_D(sve_stsd_zd, uint64_t, cpu_stl_data_ra) | ||
230 | -DO_ST1_ZPZ_D(sve_stdd_zd, uint64_t, cpu_stq_data_ra) | ||
231 | +DO_ST1_ZPZ_D(bd, zsu) | ||
232 | +DO_ST1_ZPZ_D(hd_le, zsu) | ||
233 | +DO_ST1_ZPZ_D(hd_be, zsu) | ||
234 | +DO_ST1_ZPZ_D(sd_le, zsu) | ||
235 | +DO_ST1_ZPZ_D(sd_be, zsu) | ||
236 | +DO_ST1_ZPZ_D(dd_le, zsu) | ||
237 | +DO_ST1_ZPZ_D(dd_be, zsu) | ||
238 | + | ||
239 | +DO_ST1_ZPZ_D(bd, zss) | ||
240 | +DO_ST1_ZPZ_D(hd_le, zss) | ||
241 | +DO_ST1_ZPZ_D(hd_be, zss) | ||
242 | +DO_ST1_ZPZ_D(sd_le, zss) | ||
243 | +DO_ST1_ZPZ_D(sd_be, zss) | ||
244 | +DO_ST1_ZPZ_D(dd_le, zss) | ||
245 | +DO_ST1_ZPZ_D(dd_be, zss) | ||
246 | + | ||
247 | +DO_ST1_ZPZ_D(bd, zd) | ||
248 | +DO_ST1_ZPZ_D(hd_le, zd) | ||
249 | +DO_ST1_ZPZ_D(hd_be, zd) | ||
250 | +DO_ST1_ZPZ_D(sd_le, zd) | ||
251 | +DO_ST1_ZPZ_D(sd_be, zd) | ||
252 | +DO_ST1_ZPZ_D(dd_le, zd) | ||
253 | +DO_ST1_ZPZ_D(dd_be, zd) | ||
254 | + | ||
255 | +#undef DO_ST1_ZPZ_S | ||
256 | +#undef DO_ST1_ZPZ_D | ||
257 | diff --git a/target/arm/translate-sve.c b/target/arm/translate-sve.c | ||
258 | index XXXXXXX..XXXXXXX 100644 | ||
259 | --- a/target/arm/translate-sve.c | ||
260 | +++ b/target/arm/translate-sve.c | ||
261 | @@ -XXX,XX +XXX,XX @@ static bool trans_LD1_zpiz(DisasContext *s, arg_LD1_zpiz *a, uint32_t insn) | ||
262 | return true; | ||
263 | } | ||
264 | |||
265 | -/* Indexed by [xs][msz]. */ | ||
266 | -static gen_helper_gvec_mem_scatter * const scatter_store_fn32[2][3] = { | ||
267 | - { gen_helper_sve_stbs_zsu, | ||
268 | - gen_helper_sve_sths_zsu, | ||
269 | - gen_helper_sve_stss_zsu, }, | ||
270 | - { gen_helper_sve_stbs_zss, | ||
271 | - gen_helper_sve_sths_zss, | ||
272 | - gen_helper_sve_stss_zss, }, | ||
273 | +/* Indexed by [be][xs][msz]. */ | ||
274 | +static gen_helper_gvec_mem_scatter * const scatter_store_fn32[2][2][3] = { | ||
275 | + /* Little-endian */ | ||
276 | + { { gen_helper_sve_stbs_zsu, | ||
277 | + gen_helper_sve_sths_le_zsu, | ||
278 | + gen_helper_sve_stss_le_zsu, }, | ||
279 | + { gen_helper_sve_stbs_zss, | ||
280 | + gen_helper_sve_sths_le_zss, | ||
281 | + gen_helper_sve_stss_le_zss, } }, | ||
282 | + /* Big-endian */ | ||
283 | + { { gen_helper_sve_stbs_zsu, | ||
284 | + gen_helper_sve_sths_be_zsu, | ||
285 | + gen_helper_sve_stss_be_zsu, }, | ||
286 | + { gen_helper_sve_stbs_zss, | ||
287 | + gen_helper_sve_sths_be_zss, | ||
288 | + gen_helper_sve_stss_be_zss, } }, | ||
289 | }; | ||
290 | |||
291 | /* Note that we overload xs=2 to indicate 64-bit offset. */ | ||
292 | -static gen_helper_gvec_mem_scatter * const scatter_store_fn64[3][4] = { | ||
293 | - { gen_helper_sve_stbd_zsu, | ||
294 | - gen_helper_sve_sthd_zsu, | ||
295 | - gen_helper_sve_stsd_zsu, | ||
296 | - gen_helper_sve_stdd_zsu, }, | ||
297 | - { gen_helper_sve_stbd_zss, | ||
298 | - gen_helper_sve_sthd_zss, | ||
299 | - gen_helper_sve_stsd_zss, | ||
300 | - gen_helper_sve_stdd_zss, }, | ||
301 | - { gen_helper_sve_stbd_zd, | ||
302 | - gen_helper_sve_sthd_zd, | ||
303 | - gen_helper_sve_stsd_zd, | ||
304 | - gen_helper_sve_stdd_zd, }, | ||
305 | +static gen_helper_gvec_mem_scatter * const scatter_store_fn64[2][3][4] = { | ||
306 | + /* Little-endian */ | ||
307 | + { { gen_helper_sve_stbd_zsu, | ||
308 | + gen_helper_sve_sthd_le_zsu, | ||
309 | + gen_helper_sve_stsd_le_zsu, | ||
310 | + gen_helper_sve_stdd_le_zsu, }, | ||
311 | + { gen_helper_sve_stbd_zss, | ||
312 | + gen_helper_sve_sthd_le_zss, | ||
313 | + gen_helper_sve_stsd_le_zss, | ||
314 | + gen_helper_sve_stdd_le_zss, }, | ||
315 | + { gen_helper_sve_stbd_zd, | ||
316 | + gen_helper_sve_sthd_le_zd, | ||
317 | + gen_helper_sve_stsd_le_zd, | ||
318 | + gen_helper_sve_stdd_le_zd, } }, | ||
319 | + /* Big-endian */ | ||
320 | + { { gen_helper_sve_stbd_zsu, | ||
321 | + gen_helper_sve_sthd_be_zsu, | ||
322 | + gen_helper_sve_stsd_be_zsu, | ||
323 | + gen_helper_sve_stdd_be_zsu, }, | ||
324 | + { gen_helper_sve_stbd_zss, | ||
325 | + gen_helper_sve_sthd_be_zss, | ||
326 | + gen_helper_sve_stsd_be_zss, | ||
327 | + gen_helper_sve_stdd_be_zss, }, | ||
328 | + { gen_helper_sve_stbd_zd, | ||
329 | + gen_helper_sve_sthd_be_zd, | ||
330 | + gen_helper_sve_stsd_be_zd, | ||
331 | + gen_helper_sve_stdd_be_zd, } }, | ||
332 | }; | ||
333 | |||
334 | static bool trans_ST1_zprz(DisasContext *s, arg_ST1_zprz *a, uint32_t insn) | ||
335 | { | ||
336 | gen_helper_gvec_mem_scatter *fn; | ||
337 | + int be = s->be_data == MO_BE; | ||
338 | |||
339 | if (a->esz < a->msz || (a->msz == 0 && a->scale)) { | ||
340 | return false; | ||
341 | @@ -XXX,XX +XXX,XX @@ static bool trans_ST1_zprz(DisasContext *s, arg_ST1_zprz *a, uint32_t insn) | ||
342 | } | ||
343 | switch (a->esz) { | ||
344 | case MO_32: | ||
345 | - fn = scatter_store_fn32[a->xs][a->msz]; | ||
346 | + fn = scatter_store_fn32[be][a->xs][a->msz]; | ||
347 | break; | ||
348 | case MO_64: | ||
349 | - fn = scatter_store_fn64[a->xs][a->msz]; | ||
350 | + fn = scatter_store_fn64[be][a->xs][a->msz]; | ||
351 | break; | ||
352 | default: | ||
353 | g_assert_not_reached(); | ||
354 | @@ -XXX,XX +XXX,XX @@ static bool trans_ST1_zprz(DisasContext *s, arg_ST1_zprz *a, uint32_t insn) | ||
355 | static bool trans_ST1_zpiz(DisasContext *s, arg_ST1_zpiz *a, uint32_t insn) | ||
356 | { | ||
357 | gen_helper_gvec_mem_scatter *fn = NULL; | ||
358 | + int be = s->be_data == MO_BE; | ||
359 | TCGv_i64 imm; | ||
360 | |||
361 | if (a->esz < a->msz) { | ||
362 | @@ -XXX,XX +XXX,XX @@ static bool trans_ST1_zpiz(DisasContext *s, arg_ST1_zpiz *a, uint32_t insn) | ||
363 | |||
364 | switch (a->esz) { | ||
365 | case MO_32: | ||
366 | - fn = scatter_store_fn32[0][a->msz]; | ||
367 | + fn = scatter_store_fn32[be][0][a->msz]; | ||
368 | break; | ||
369 | case MO_64: | ||
370 | - fn = scatter_store_fn64[2][a->msz]; | ||
371 | + fn = scatter_store_fn64[be][2][a->msz]; | ||
372 | break; | ||
373 | } | ||
374 | assert(fn != NULL); | ||
375 | -- | ||
376 | 2.19.0 | ||
377 | |||
378 | diff view generated by jsdifflib |
Deleted patch | |||
---|---|---|---|
1 | From: Richard Henderson <richard.henderson@linaro.org> | ||
2 | 1 | ||
3 | This implements the feature for softmmu, and moves the | ||
4 | main loop out of a macro and into a function. | ||
5 | |||
6 | Reviewed-by: Peter Maydell <peter.maydell@linaro.org> | ||
7 | Tested-by: Laurent Desnogues <laurent.desnogues@gmail.com> | ||
8 | Signed-off-by: Richard Henderson <richard.henderson@linaro.org> | ||
9 | Message-id: 20181005175350.30752-15-richard.henderson@linaro.org | ||
10 | Signed-off-by: Peter Maydell <peter.maydell@linaro.org> | ||
11 | --- | ||
12 | target/arm/helper-sve.h | 84 ++++++++--- | ||
13 | target/arm/sve_helper.c | 290 +++++++++++++++++++++++++++---------- | ||
14 | target/arm/translate-sve.c | 84 +++++------ | ||
15 | 3 files changed, 321 insertions(+), 137 deletions(-) | ||
16 | |||
17 | diff --git a/target/arm/helper-sve.h b/target/arm/helper-sve.h | ||
18 | index XXXXXXX..XXXXXXX 100644 | ||
19 | --- a/target/arm/helper-sve.h | ||
20 | +++ b/target/arm/helper-sve.h | ||
21 | @@ -XXX,XX +XXX,XX @@ DEF_HELPER_FLAGS_6(sve_ldsds_be_zd, TCG_CALL_NO_WG, | ||
22 | |||
23 | DEF_HELPER_FLAGS_6(sve_ldffbsu_zsu, TCG_CALL_NO_WG, | ||
24 | void, env, ptr, ptr, ptr, tl, i32) | ||
25 | -DEF_HELPER_FLAGS_6(sve_ldffhsu_zsu, TCG_CALL_NO_WG, | ||
26 | +DEF_HELPER_FLAGS_6(sve_ldffhsu_le_zsu, TCG_CALL_NO_WG, | ||
27 | void, env, ptr, ptr, ptr, tl, i32) | ||
28 | -DEF_HELPER_FLAGS_6(sve_ldffssu_zsu, TCG_CALL_NO_WG, | ||
29 | +DEF_HELPER_FLAGS_6(sve_ldffhsu_be_zsu, TCG_CALL_NO_WG, | ||
30 | + void, env, ptr, ptr, ptr, tl, i32) | ||
31 | +DEF_HELPER_FLAGS_6(sve_ldffss_le_zsu, TCG_CALL_NO_WG, | ||
32 | + void, env, ptr, ptr, ptr, tl, i32) | ||
33 | +DEF_HELPER_FLAGS_6(sve_ldffss_be_zsu, TCG_CALL_NO_WG, | ||
34 | void, env, ptr, ptr, ptr, tl, i32) | ||
35 | DEF_HELPER_FLAGS_6(sve_ldffbss_zsu, TCG_CALL_NO_WG, | ||
36 | void, env, ptr, ptr, ptr, tl, i32) | ||
37 | -DEF_HELPER_FLAGS_6(sve_ldffhss_zsu, TCG_CALL_NO_WG, | ||
38 | +DEF_HELPER_FLAGS_6(sve_ldffhss_le_zsu, TCG_CALL_NO_WG, | ||
39 | + void, env, ptr, ptr, ptr, tl, i32) | ||
40 | +DEF_HELPER_FLAGS_6(sve_ldffhss_be_zsu, TCG_CALL_NO_WG, | ||
41 | void, env, ptr, ptr, ptr, tl, i32) | ||
42 | |||
43 | DEF_HELPER_FLAGS_6(sve_ldffbsu_zss, TCG_CALL_NO_WG, | ||
44 | void, env, ptr, ptr, ptr, tl, i32) | ||
45 | -DEF_HELPER_FLAGS_6(sve_ldffhsu_zss, TCG_CALL_NO_WG, | ||
46 | +DEF_HELPER_FLAGS_6(sve_ldffhsu_le_zss, TCG_CALL_NO_WG, | ||
47 | void, env, ptr, ptr, ptr, tl, i32) | ||
48 | -DEF_HELPER_FLAGS_6(sve_ldffssu_zss, TCG_CALL_NO_WG, | ||
49 | +DEF_HELPER_FLAGS_6(sve_ldffhsu_be_zss, TCG_CALL_NO_WG, | ||
50 | + void, env, ptr, ptr, ptr, tl, i32) | ||
51 | +DEF_HELPER_FLAGS_6(sve_ldffss_le_zss, TCG_CALL_NO_WG, | ||
52 | + void, env, ptr, ptr, ptr, tl, i32) | ||
53 | +DEF_HELPER_FLAGS_6(sve_ldffss_be_zss, TCG_CALL_NO_WG, | ||
54 | void, env, ptr, ptr, ptr, tl, i32) | ||
55 | DEF_HELPER_FLAGS_6(sve_ldffbss_zss, TCG_CALL_NO_WG, | ||
56 | void, env, ptr, ptr, ptr, tl, i32) | ||
57 | -DEF_HELPER_FLAGS_6(sve_ldffhss_zss, TCG_CALL_NO_WG, | ||
58 | +DEF_HELPER_FLAGS_6(sve_ldffhss_le_zss, TCG_CALL_NO_WG, | ||
59 | + void, env, ptr, ptr, ptr, tl, i32) | ||
60 | +DEF_HELPER_FLAGS_6(sve_ldffhss_be_zss, TCG_CALL_NO_WG, | ||
61 | void, env, ptr, ptr, ptr, tl, i32) | ||
62 | |||
63 | DEF_HELPER_FLAGS_6(sve_ldffbdu_zsu, TCG_CALL_NO_WG, | ||
64 | void, env, ptr, ptr, ptr, tl, i32) | ||
65 | -DEF_HELPER_FLAGS_6(sve_ldffhdu_zsu, TCG_CALL_NO_WG, | ||
66 | +DEF_HELPER_FLAGS_6(sve_ldffhdu_le_zsu, TCG_CALL_NO_WG, | ||
67 | void, env, ptr, ptr, ptr, tl, i32) | ||
68 | -DEF_HELPER_FLAGS_6(sve_ldffsdu_zsu, TCG_CALL_NO_WG, | ||
69 | +DEF_HELPER_FLAGS_6(sve_ldffhdu_be_zsu, TCG_CALL_NO_WG, | ||
70 | void, env, ptr, ptr, ptr, tl, i32) | ||
71 | -DEF_HELPER_FLAGS_6(sve_ldffddu_zsu, TCG_CALL_NO_WG, | ||
72 | +DEF_HELPER_FLAGS_6(sve_ldffsdu_le_zsu, TCG_CALL_NO_WG, | ||
73 | + void, env, ptr, ptr, ptr, tl, i32) | ||
74 | +DEF_HELPER_FLAGS_6(sve_ldffsdu_be_zsu, TCG_CALL_NO_WG, | ||
75 | + void, env, ptr, ptr, ptr, tl, i32) | ||
76 | +DEF_HELPER_FLAGS_6(sve_ldffdd_le_zsu, TCG_CALL_NO_WG, | ||
77 | + void, env, ptr, ptr, ptr, tl, i32) | ||
78 | +DEF_HELPER_FLAGS_6(sve_ldffdd_be_zsu, TCG_CALL_NO_WG, | ||
79 | void, env, ptr, ptr, ptr, tl, i32) | ||
80 | DEF_HELPER_FLAGS_6(sve_ldffbds_zsu, TCG_CALL_NO_WG, | ||
81 | void, env, ptr, ptr, ptr, tl, i32) | ||
82 | -DEF_HELPER_FLAGS_6(sve_ldffhds_zsu, TCG_CALL_NO_WG, | ||
83 | +DEF_HELPER_FLAGS_6(sve_ldffhds_le_zsu, TCG_CALL_NO_WG, | ||
84 | void, env, ptr, ptr, ptr, tl, i32) | ||
85 | -DEF_HELPER_FLAGS_6(sve_ldffsds_zsu, TCG_CALL_NO_WG, | ||
86 | +DEF_HELPER_FLAGS_6(sve_ldffhds_be_zsu, TCG_CALL_NO_WG, | ||
87 | + void, env, ptr, ptr, ptr, tl, i32) | ||
88 | +DEF_HELPER_FLAGS_6(sve_ldffsds_le_zsu, TCG_CALL_NO_WG, | ||
89 | + void, env, ptr, ptr, ptr, tl, i32) | ||
90 | +DEF_HELPER_FLAGS_6(sve_ldffsds_be_zsu, TCG_CALL_NO_WG, | ||
91 | void, env, ptr, ptr, ptr, tl, i32) | ||
92 | |||
93 | DEF_HELPER_FLAGS_6(sve_ldffbdu_zss, TCG_CALL_NO_WG, | ||
94 | void, env, ptr, ptr, ptr, tl, i32) | ||
95 | -DEF_HELPER_FLAGS_6(sve_ldffhdu_zss, TCG_CALL_NO_WG, | ||
96 | +DEF_HELPER_FLAGS_6(sve_ldffhdu_le_zss, TCG_CALL_NO_WG, | ||
97 | void, env, ptr, ptr, ptr, tl, i32) | ||
98 | -DEF_HELPER_FLAGS_6(sve_ldffsdu_zss, TCG_CALL_NO_WG, | ||
99 | +DEF_HELPER_FLAGS_6(sve_ldffhdu_be_zss, TCG_CALL_NO_WG, | ||
100 | void, env, ptr, ptr, ptr, tl, i32) | ||
101 | -DEF_HELPER_FLAGS_6(sve_ldffddu_zss, TCG_CALL_NO_WG, | ||
102 | +DEF_HELPER_FLAGS_6(sve_ldffsdu_le_zss, TCG_CALL_NO_WG, | ||
103 | + void, env, ptr, ptr, ptr, tl, i32) | ||
104 | +DEF_HELPER_FLAGS_6(sve_ldffsdu_be_zss, TCG_CALL_NO_WG, | ||
105 | + void, env, ptr, ptr, ptr, tl, i32) | ||
106 | +DEF_HELPER_FLAGS_6(sve_ldffdd_le_zss, TCG_CALL_NO_WG, | ||
107 | + void, env, ptr, ptr, ptr, tl, i32) | ||
108 | +DEF_HELPER_FLAGS_6(sve_ldffdd_be_zss, TCG_CALL_NO_WG, | ||
109 | void, env, ptr, ptr, ptr, tl, i32) | ||
110 | DEF_HELPER_FLAGS_6(sve_ldffbds_zss, TCG_CALL_NO_WG, | ||
111 | void, env, ptr, ptr, ptr, tl, i32) | ||
112 | -DEF_HELPER_FLAGS_6(sve_ldffhds_zss, TCG_CALL_NO_WG, | ||
113 | +DEF_HELPER_FLAGS_6(sve_ldffhds_le_zss, TCG_CALL_NO_WG, | ||
114 | void, env, ptr, ptr, ptr, tl, i32) | ||
115 | -DEF_HELPER_FLAGS_6(sve_ldffsds_zss, TCG_CALL_NO_WG, | ||
116 | +DEF_HELPER_FLAGS_6(sve_ldffhds_be_zss, TCG_CALL_NO_WG, | ||
117 | + void, env, ptr, ptr, ptr, tl, i32) | ||
118 | +DEF_HELPER_FLAGS_6(sve_ldffsds_le_zss, TCG_CALL_NO_WG, | ||
119 | + void, env, ptr, ptr, ptr, tl, i32) | ||
120 | +DEF_HELPER_FLAGS_6(sve_ldffsds_be_zss, TCG_CALL_NO_WG, | ||
121 | void, env, ptr, ptr, ptr, tl, i32) | ||
122 | |||
123 | DEF_HELPER_FLAGS_6(sve_ldffbdu_zd, TCG_CALL_NO_WG, | ||
124 | void, env, ptr, ptr, ptr, tl, i32) | ||
125 | -DEF_HELPER_FLAGS_6(sve_ldffhdu_zd, TCG_CALL_NO_WG, | ||
126 | +DEF_HELPER_FLAGS_6(sve_ldffhdu_le_zd, TCG_CALL_NO_WG, | ||
127 | void, env, ptr, ptr, ptr, tl, i32) | ||
128 | -DEF_HELPER_FLAGS_6(sve_ldffsdu_zd, TCG_CALL_NO_WG, | ||
129 | +DEF_HELPER_FLAGS_6(sve_ldffhdu_be_zd, TCG_CALL_NO_WG, | ||
130 | void, env, ptr, ptr, ptr, tl, i32) | ||
131 | -DEF_HELPER_FLAGS_6(sve_ldffddu_zd, TCG_CALL_NO_WG, | ||
132 | +DEF_HELPER_FLAGS_6(sve_ldffsdu_le_zd, TCG_CALL_NO_WG, | ||
133 | + void, env, ptr, ptr, ptr, tl, i32) | ||
134 | +DEF_HELPER_FLAGS_6(sve_ldffsdu_be_zd, TCG_CALL_NO_WG, | ||
135 | + void, env, ptr, ptr, ptr, tl, i32) | ||
136 | +DEF_HELPER_FLAGS_6(sve_ldffdd_le_zd, TCG_CALL_NO_WG, | ||
137 | + void, env, ptr, ptr, ptr, tl, i32) | ||
138 | +DEF_HELPER_FLAGS_6(sve_ldffdd_be_zd, TCG_CALL_NO_WG, | ||
139 | void, env, ptr, ptr, ptr, tl, i32) | ||
140 | DEF_HELPER_FLAGS_6(sve_ldffbds_zd, TCG_CALL_NO_WG, | ||
141 | void, env, ptr, ptr, ptr, tl, i32) | ||
142 | -DEF_HELPER_FLAGS_6(sve_ldffhds_zd, TCG_CALL_NO_WG, | ||
143 | +DEF_HELPER_FLAGS_6(sve_ldffhds_le_zd, TCG_CALL_NO_WG, | ||
144 | void, env, ptr, ptr, ptr, tl, i32) | ||
145 | -DEF_HELPER_FLAGS_6(sve_ldffsds_zd, TCG_CALL_NO_WG, | ||
146 | +DEF_HELPER_FLAGS_6(sve_ldffhds_be_zd, TCG_CALL_NO_WG, | ||
147 | + void, env, ptr, ptr, ptr, tl, i32) | ||
148 | +DEF_HELPER_FLAGS_6(sve_ldffsds_le_zd, TCG_CALL_NO_WG, | ||
149 | + void, env, ptr, ptr, ptr, tl, i32) | ||
150 | +DEF_HELPER_FLAGS_6(sve_ldffsds_be_zd, TCG_CALL_NO_WG, | ||
151 | void, env, ptr, ptr, ptr, tl, i32) | ||
152 | |||
153 | DEF_HELPER_FLAGS_6(sve_stbs_zsu, TCG_CALL_NO_WG, | ||
154 | diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c | ||
155 | index XXXXXXX..XXXXXXX 100644 | ||
156 | --- a/target/arm/sve_helper.c | ||
157 | +++ b/target/arm/sve_helper.c | ||
158 | @@ -XXX,XX +XXX,XX @@ DO_LD1_ZPZ_D(dd_be, zd) | ||
159 | |||
160 | /* First fault loads with a vector index. */ | ||
161 | |||
162 | -#ifdef CONFIG_USER_ONLY | ||
163 | +/* Load one element into VD+REG_OFF from (ENV,VADDR) without faulting. | ||
164 | + * The controlling predicate is known to be true. Return true if the | ||
165 | + * load was successful. | ||
166 | + */ | ||
167 | +typedef bool sve_ld1_nf_fn(CPUARMState *env, void *vd, intptr_t reg_off, | ||
168 | + target_ulong vaddr, int mmu_idx); | ||
169 | |||
170 | -#define DO_LDFF1_ZPZ(NAME, TYPEE, TYPEI, TYPEM, FN, H) \ | ||
171 | -void HELPER(NAME)(CPUARMState *env, void *vd, void *vg, void *vm, \ | ||
172 | - target_ulong base, uint32_t desc) \ | ||
173 | -{ \ | ||
174 | - intptr_t i, oprsz = simd_oprsz(desc); \ | ||
175 | - unsigned scale = simd_data(desc); \ | ||
176 | - uintptr_t ra = GETPC(); \ | ||
177 | - bool first = true; \ | ||
178 | - mmap_lock(); \ | ||
179 | - for (i = 0; i < oprsz; ) { \ | ||
180 | - uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); \ | ||
181 | - do { \ | ||
182 | - TYPEM m = 0; \ | ||
183 | - if (pg & 1) { \ | ||
184 | - target_ulong off = *(TYPEI *)(vm + H(i)); \ | ||
185 | - target_ulong addr = base + (off << scale); \ | ||
186 | - if (!first && \ | ||
187 | - page_check_range(addr, sizeof(TYPEM), PAGE_READ)) { \ | ||
188 | - record_fault(env, i, oprsz); \ | ||
189 | - goto exit; \ | ||
190 | - } \ | ||
191 | - m = FN(env, addr, ra); \ | ||
192 | - first = false; \ | ||
193 | - } \ | ||
194 | - *(TYPEE *)(vd + H(i)) = m; \ | ||
195 | - i += sizeof(TYPEE), pg >>= sizeof(TYPEE); \ | ||
196 | - } while (i & 15); \ | ||
197 | - } \ | ||
198 | - exit: \ | ||
199 | - mmap_unlock(); \ | ||
200 | +#ifdef CONFIG_SOFTMMU | ||
201 | +#define DO_LD_NF(NAME, H, TYPEE, TYPEM, HOST) \ | ||
202 | +static bool sve_ld##NAME##_nf(CPUARMState *env, void *vd, intptr_t reg_off, \ | ||
203 | + target_ulong addr, int mmu_idx) \ | ||
204 | +{ \ | ||
205 | + target_ulong next_page = -(addr | TARGET_PAGE_MASK); \ | ||
206 | + if (likely(next_page - addr >= sizeof(TYPEM))) { \ | ||
207 | + void *host = tlb_vaddr_to_host(env, addr, MMU_DATA_LOAD, mmu_idx); \ | ||
208 | + if (likely(host)) { \ | ||
209 | + TYPEM val = HOST(host); \ | ||
210 | + *(TYPEE *)(vd + H(reg_off)) = val; \ | ||
211 | + return true; \ | ||
212 | + } \ | ||
213 | + } \ | ||
214 | + return false; \ | ||
215 | } | ||
216 | - | ||
217 | #else | ||
218 | - | ||
219 | -#define DO_LDFF1_ZPZ(NAME, TYPEE, TYPEI, TYPEM, FN, H) \ | ||
220 | -void HELPER(NAME)(CPUARMState *env, void *vd, void *vg, void *vm, \ | ||
221 | - target_ulong base, uint32_t desc) \ | ||
222 | -{ \ | ||
223 | - g_assert_not_reached(); \ | ||
224 | +#define DO_LD_NF(NAME, H, TYPEE, TYPEM, HOST) \ | ||
225 | +static bool sve_ld##NAME##_nf(CPUARMState *env, void *vd, intptr_t reg_off, \ | ||
226 | + target_ulong addr, int mmu_idx) \ | ||
227 | +{ \ | ||
228 | + if (likely(page_check_range(addr, sizeof(TYPEM), PAGE_READ))) { \ | ||
229 | + TYPEM val = HOST(g2h(addr)); \ | ||
230 | + *(TYPEE *)(vd + H(reg_off)) = val; \ | ||
231 | + return true; \ | ||
232 | + } \ | ||
233 | + return false; \ | ||
234 | } | ||
235 | - | ||
236 | #endif | ||
237 | |||
238 | -#define DO_LDFF1_ZPZ_S(NAME, TYPEI, TYPEM, FN) \ | ||
239 | - DO_LDFF1_ZPZ(NAME, uint32_t, TYPEI, TYPEM, FN, H1_4) | ||
240 | -#define DO_LDFF1_ZPZ_D(NAME, TYPEI, TYPEM, FN) \ | ||
241 | - DO_LDFF1_ZPZ(NAME, uint64_t, TYPEI, TYPEM, FN, ) | ||
242 | +DO_LD_NF(bsu, H1_4, uint32_t, uint8_t, ldub_p) | ||
243 | +DO_LD_NF(bss, H1_4, uint32_t, int8_t, ldsb_p) | ||
244 | +DO_LD_NF(bdu, , uint64_t, uint8_t, ldub_p) | ||
245 | +DO_LD_NF(bds, , uint64_t, int8_t, ldsb_p) | ||
246 | |||
247 | -DO_LDFF1_ZPZ_S(sve_ldffbsu_zsu, uint32_t, uint8_t, cpu_ldub_data_ra) | ||
248 | -DO_LDFF1_ZPZ_S(sve_ldffhsu_zsu, uint32_t, uint16_t, cpu_lduw_data_ra) | ||
249 | -DO_LDFF1_ZPZ_S(sve_ldffssu_zsu, uint32_t, uint32_t, cpu_ldl_data_ra) | ||
250 | -DO_LDFF1_ZPZ_S(sve_ldffbss_zsu, uint32_t, int8_t, cpu_ldub_data_ra) | ||
251 | -DO_LDFF1_ZPZ_S(sve_ldffhss_zsu, uint32_t, int16_t, cpu_lduw_data_ra) | ||
252 | +DO_LD_NF(hsu_le, H1_4, uint32_t, uint16_t, lduw_le_p) | ||
253 | +DO_LD_NF(hss_le, H1_4, uint32_t, int16_t, ldsw_le_p) | ||
254 | +DO_LD_NF(hsu_be, H1_4, uint32_t, uint16_t, lduw_be_p) | ||
255 | +DO_LD_NF(hss_be, H1_4, uint32_t, int16_t, ldsw_be_p) | ||
256 | +DO_LD_NF(hdu_le, , uint64_t, uint16_t, lduw_le_p) | ||
257 | +DO_LD_NF(hds_le, , uint64_t, int16_t, ldsw_le_p) | ||
258 | +DO_LD_NF(hdu_be, , uint64_t, uint16_t, lduw_be_p) | ||
259 | +DO_LD_NF(hds_be, , uint64_t, int16_t, ldsw_be_p) | ||
260 | |||
261 | -DO_LDFF1_ZPZ_S(sve_ldffbsu_zss, int32_t, uint8_t, cpu_ldub_data_ra) | ||
262 | -DO_LDFF1_ZPZ_S(sve_ldffhsu_zss, int32_t, uint16_t, cpu_lduw_data_ra) | ||
263 | -DO_LDFF1_ZPZ_S(sve_ldffssu_zss, int32_t, uint32_t, cpu_ldl_data_ra) | ||
264 | -DO_LDFF1_ZPZ_S(sve_ldffbss_zss, int32_t, int8_t, cpu_ldub_data_ra) | ||
265 | -DO_LDFF1_ZPZ_S(sve_ldffhss_zss, int32_t, int16_t, cpu_lduw_data_ra) | ||
266 | +DO_LD_NF(ss_le, H1_4, uint32_t, uint32_t, ldl_le_p) | ||
267 | +DO_LD_NF(ss_be, H1_4, uint32_t, uint32_t, ldl_be_p) | ||
268 | +DO_LD_NF(sdu_le, , uint64_t, uint32_t, ldl_le_p) | ||
269 | +DO_LD_NF(sds_le, , uint64_t, int32_t, ldl_le_p) | ||
270 | +DO_LD_NF(sdu_be, , uint64_t, uint32_t, ldl_be_p) | ||
271 | +DO_LD_NF(sds_be, , uint64_t, int32_t, ldl_be_p) | ||
272 | |||
273 | -DO_LDFF1_ZPZ_D(sve_ldffbdu_zsu, uint32_t, uint8_t, cpu_ldub_data_ra) | ||
274 | -DO_LDFF1_ZPZ_D(sve_ldffhdu_zsu, uint32_t, uint16_t, cpu_lduw_data_ra) | ||
275 | -DO_LDFF1_ZPZ_D(sve_ldffsdu_zsu, uint32_t, uint32_t, cpu_ldl_data_ra) | ||
276 | -DO_LDFF1_ZPZ_D(sve_ldffddu_zsu, uint32_t, uint64_t, cpu_ldq_data_ra) | ||
277 | -DO_LDFF1_ZPZ_D(sve_ldffbds_zsu, uint32_t, int8_t, cpu_ldub_data_ra) | ||
278 | -DO_LDFF1_ZPZ_D(sve_ldffhds_zsu, uint32_t, int16_t, cpu_lduw_data_ra) | ||
279 | -DO_LDFF1_ZPZ_D(sve_ldffsds_zsu, uint32_t, int32_t, cpu_ldl_data_ra) | ||
280 | +DO_LD_NF(dd_le, , uint64_t, uint64_t, ldq_le_p) | ||
281 | +DO_LD_NF(dd_be, , uint64_t, uint64_t, ldq_be_p) | ||
282 | |||
283 | -DO_LDFF1_ZPZ_D(sve_ldffbdu_zss, int32_t, uint8_t, cpu_ldub_data_ra) | ||
284 | -DO_LDFF1_ZPZ_D(sve_ldffhdu_zss, int32_t, uint16_t, cpu_lduw_data_ra) | ||
285 | -DO_LDFF1_ZPZ_D(sve_ldffsdu_zss, int32_t, uint32_t, cpu_ldl_data_ra) | ||
286 | -DO_LDFF1_ZPZ_D(sve_ldffddu_zss, int32_t, uint64_t, cpu_ldq_data_ra) | ||
287 | -DO_LDFF1_ZPZ_D(sve_ldffbds_zss, int32_t, int8_t, cpu_ldub_data_ra) | ||
288 | -DO_LDFF1_ZPZ_D(sve_ldffhds_zss, int32_t, int16_t, cpu_lduw_data_ra) | ||
289 | -DO_LDFF1_ZPZ_D(sve_ldffsds_zss, int32_t, int32_t, cpu_ldl_data_ra) | ||
290 | +/* | ||
291 | + * Common helper for all gather first-faulting loads. | ||
292 | + */ | ||
293 | +static inline void sve_ldff1_zs(CPUARMState *env, void *vd, void *vg, void *vm, | ||
294 | + target_ulong base, uint32_t desc, uintptr_t ra, | ||
295 | + zreg_off_fn *off_fn, sve_ld1_tlb_fn *tlb_fn, | ||
296 | + sve_ld1_nf_fn *nonfault_fn) | ||
297 | +{ | ||
298 | + const int mmu_idx = cpu_mmu_index(env, false); | ||
299 | + intptr_t reg_off, reg_max = simd_oprsz(desc); | ||
300 | + unsigned scale = simd_data(desc); | ||
301 | + target_ulong addr; | ||
302 | |||
303 | -DO_LDFF1_ZPZ_D(sve_ldffbdu_zd, uint64_t, uint8_t, cpu_ldub_data_ra) | ||
304 | -DO_LDFF1_ZPZ_D(sve_ldffhdu_zd, uint64_t, uint16_t, cpu_lduw_data_ra) | ||
305 | -DO_LDFF1_ZPZ_D(sve_ldffsdu_zd, uint64_t, uint32_t, cpu_ldl_data_ra) | ||
306 | -DO_LDFF1_ZPZ_D(sve_ldffddu_zd, uint64_t, uint64_t, cpu_ldq_data_ra) | ||
307 | -DO_LDFF1_ZPZ_D(sve_ldffbds_zd, uint64_t, int8_t, cpu_ldub_data_ra) | ||
308 | -DO_LDFF1_ZPZ_D(sve_ldffhds_zd, uint64_t, int16_t, cpu_lduw_data_ra) | ||
309 | -DO_LDFF1_ZPZ_D(sve_ldffsds_zd, uint64_t, int32_t, cpu_ldl_data_ra) | ||
310 | + /* Skip to the first true predicate. */ | ||
311 | + reg_off = find_next_active(vg, 0, reg_max, MO_32); | ||
312 | + if (likely(reg_off < reg_max)) { | ||
313 | + /* Perform one normal read, which will fault or not. */ | ||
314 | + set_helper_retaddr(ra); | ||
315 | + addr = off_fn(vm, reg_off); | ||
316 | + addr = base + (addr << scale); | ||
317 | + tlb_fn(env, vd, reg_off, addr, mmu_idx, ra); | ||
318 | + | ||
319 | + /* The rest of the reads will be non-faulting. */ | ||
320 | + set_helper_retaddr(0); | ||
321 | + } | ||
322 | + | ||
323 | + /* After any fault, zero the leading predicated false elements. */ | ||
324 | + swap_memzero(vd, reg_off); | ||
325 | + | ||
326 | + while (likely((reg_off += 4) < reg_max)) { | ||
327 | + uint64_t pg = *(uint64_t *)(vg + (reg_off >> 6) * 8); | ||
328 | + if (likely((pg >> (reg_off & 63)) & 1)) { | ||
329 | + addr = off_fn(vm, reg_off); | ||
330 | + addr = base + (addr << scale); | ||
331 | + if (!nonfault_fn(env, vd, reg_off, addr, mmu_idx)) { | ||
332 | + record_fault(env, reg_off, reg_max); | ||
333 | + break; | ||
334 | + } | ||
335 | + } else { | ||
336 | + *(uint32_t *)(vd + H1_4(reg_off)) = 0; | ||
337 | + } | ||
338 | + } | ||
339 | +} | ||
340 | + | ||
341 | +static inline void sve_ldff1_zd(CPUARMState *env, void *vd, void *vg, void *vm, | ||
342 | + target_ulong base, uint32_t desc, uintptr_t ra, | ||
343 | + zreg_off_fn *off_fn, sve_ld1_tlb_fn *tlb_fn, | ||
344 | + sve_ld1_nf_fn *nonfault_fn) | ||
345 | +{ | ||
346 | + const int mmu_idx = cpu_mmu_index(env, false); | ||
347 | + intptr_t reg_off, reg_max = simd_oprsz(desc); | ||
348 | + unsigned scale = simd_data(desc); | ||
349 | + target_ulong addr; | ||
350 | + | ||
351 | + /* Skip to the first true predicate. */ | ||
352 | + reg_off = find_next_active(vg, 0, reg_max, MO_64); | ||
353 | + if (likely(reg_off < reg_max)) { | ||
354 | + /* Perform one normal read, which will fault or not. */ | ||
355 | + set_helper_retaddr(ra); | ||
356 | + addr = off_fn(vm, reg_off); | ||
357 | + addr = base + (addr << scale); | ||
358 | + tlb_fn(env, vd, reg_off, addr, mmu_idx, ra); | ||
359 | + | ||
360 | + /* The rest of the reads will be non-faulting. */ | ||
361 | + set_helper_retaddr(0); | ||
362 | + } | ||
363 | + | ||
364 | + /* After any fault, zero the leading predicated false elements. */ | ||
365 | + swap_memzero(vd, reg_off); | ||
366 | + | ||
367 | + while (likely((reg_off += 8) < reg_max)) { | ||
368 | + uint8_t pg = *(uint8_t *)(vg + H1(reg_off >> 3)); | ||
369 | + if (likely(pg & 1)) { | ||
370 | + addr = off_fn(vm, reg_off); | ||
371 | + addr = base + (addr << scale); | ||
372 | + if (!nonfault_fn(env, vd, reg_off, addr, mmu_idx)) { | ||
373 | + record_fault(env, reg_off, reg_max); | ||
374 | + break; | ||
375 | + } | ||
376 | + } else { | ||
377 | + *(uint64_t *)(vd + reg_off) = 0; | ||
378 | + } | ||
379 | + } | ||
380 | +} | ||
381 | + | ||
382 | +#define DO_LDFF1_ZPZ_S(MEM, OFS) \ | ||
383 | +void HELPER(sve_ldff##MEM##_##OFS) \ | ||
384 | + (CPUARMState *env, void *vd, void *vg, void *vm, \ | ||
385 | + target_ulong base, uint32_t desc) \ | ||
386 | +{ \ | ||
387 | + sve_ldff1_zs(env, vd, vg, vm, base, desc, GETPC(), \ | ||
388 | + off_##OFS##_s, sve_ld1##MEM##_tlb, sve_ld##MEM##_nf); \ | ||
389 | +} | ||
390 | + | ||
391 | +#define DO_LDFF1_ZPZ_D(MEM, OFS) \ | ||
392 | +void HELPER(sve_ldff##MEM##_##OFS) \ | ||
393 | + (CPUARMState *env, void *vd, void *vg, void *vm, \ | ||
394 | + target_ulong base, uint32_t desc) \ | ||
395 | +{ \ | ||
396 | + sve_ldff1_zd(env, vd, vg, vm, base, desc, GETPC(), \ | ||
397 | + off_##OFS##_d, sve_ld1##MEM##_tlb, sve_ld##MEM##_nf); \ | ||
398 | +} | ||
399 | + | ||
400 | +DO_LDFF1_ZPZ_S(bsu, zsu) | ||
401 | +DO_LDFF1_ZPZ_S(bsu, zss) | ||
402 | +DO_LDFF1_ZPZ_D(bdu, zsu) | ||
403 | +DO_LDFF1_ZPZ_D(bdu, zss) | ||
404 | +DO_LDFF1_ZPZ_D(bdu, zd) | ||
405 | + | ||
406 | +DO_LDFF1_ZPZ_S(bss, zsu) | ||
407 | +DO_LDFF1_ZPZ_S(bss, zss) | ||
408 | +DO_LDFF1_ZPZ_D(bds, zsu) | ||
409 | +DO_LDFF1_ZPZ_D(bds, zss) | ||
410 | +DO_LDFF1_ZPZ_D(bds, zd) | ||
411 | + | ||
412 | +DO_LDFF1_ZPZ_S(hsu_le, zsu) | ||
413 | +DO_LDFF1_ZPZ_S(hsu_le, zss) | ||
414 | +DO_LDFF1_ZPZ_D(hdu_le, zsu) | ||
415 | +DO_LDFF1_ZPZ_D(hdu_le, zss) | ||
416 | +DO_LDFF1_ZPZ_D(hdu_le, zd) | ||
417 | + | ||
418 | +DO_LDFF1_ZPZ_S(hsu_be, zsu) | ||
419 | +DO_LDFF1_ZPZ_S(hsu_be, zss) | ||
420 | +DO_LDFF1_ZPZ_D(hdu_be, zsu) | ||
421 | +DO_LDFF1_ZPZ_D(hdu_be, zss) | ||
422 | +DO_LDFF1_ZPZ_D(hdu_be, zd) | ||
423 | + | ||
424 | +DO_LDFF1_ZPZ_S(hss_le, zsu) | ||
425 | +DO_LDFF1_ZPZ_S(hss_le, zss) | ||
426 | +DO_LDFF1_ZPZ_D(hds_le, zsu) | ||
427 | +DO_LDFF1_ZPZ_D(hds_le, zss) | ||
428 | +DO_LDFF1_ZPZ_D(hds_le, zd) | ||
429 | + | ||
430 | +DO_LDFF1_ZPZ_S(hss_be, zsu) | ||
431 | +DO_LDFF1_ZPZ_S(hss_be, zss) | ||
432 | +DO_LDFF1_ZPZ_D(hds_be, zsu) | ||
433 | +DO_LDFF1_ZPZ_D(hds_be, zss) | ||
434 | +DO_LDFF1_ZPZ_D(hds_be, zd) | ||
435 | + | ||
436 | +DO_LDFF1_ZPZ_S(ss_le, zsu) | ||
437 | +DO_LDFF1_ZPZ_S(ss_le, zss) | ||
438 | +DO_LDFF1_ZPZ_D(sdu_le, zsu) | ||
439 | +DO_LDFF1_ZPZ_D(sdu_le, zss) | ||
440 | +DO_LDFF1_ZPZ_D(sdu_le, zd) | ||
441 | + | ||
442 | +DO_LDFF1_ZPZ_S(ss_be, zsu) | ||
443 | +DO_LDFF1_ZPZ_S(ss_be, zss) | ||
444 | +DO_LDFF1_ZPZ_D(sdu_be, zsu) | ||
445 | +DO_LDFF1_ZPZ_D(sdu_be, zss) | ||
446 | +DO_LDFF1_ZPZ_D(sdu_be, zd) | ||
447 | + | ||
448 | +DO_LDFF1_ZPZ_D(sds_le, zsu) | ||
449 | +DO_LDFF1_ZPZ_D(sds_le, zss) | ||
450 | +DO_LDFF1_ZPZ_D(sds_le, zd) | ||
451 | + | ||
452 | +DO_LDFF1_ZPZ_D(sds_be, zsu) | ||
453 | +DO_LDFF1_ZPZ_D(sds_be, zss) | ||
454 | +DO_LDFF1_ZPZ_D(sds_be, zd) | ||
455 | + | ||
456 | +DO_LDFF1_ZPZ_D(dd_le, zsu) | ||
457 | +DO_LDFF1_ZPZ_D(dd_le, zss) | ||
458 | +DO_LDFF1_ZPZ_D(dd_le, zd) | ||
459 | + | ||
460 | +DO_LDFF1_ZPZ_D(dd_be, zsu) | ||
461 | +DO_LDFF1_ZPZ_D(dd_be, zss) | ||
462 | +DO_LDFF1_ZPZ_D(dd_be, zd) | ||
463 | |||
464 | /* Stores with a vector index. */ | ||
465 | |||
466 | diff --git a/target/arm/translate-sve.c b/target/arm/translate-sve.c | ||
467 | index XXXXXXX..XXXXXXX 100644 | ||
468 | --- a/target/arm/translate-sve.c | ||
469 | +++ b/target/arm/translate-sve.c | ||
470 | @@ -XXX,XX +XXX,XX @@ static gen_helper_gvec_mem_scatter * const gather_load_fn32[2][2][2][2][3] = { | ||
471 | |||
472 | /* First-fault */ | ||
473 | { { { gen_helper_sve_ldffbss_zsu, | ||
474 | - gen_helper_sve_ldffhss_zsu, | ||
475 | + gen_helper_sve_ldffhss_le_zsu, | ||
476 | NULL, }, | ||
477 | { gen_helper_sve_ldffbsu_zsu, | ||
478 | - gen_helper_sve_ldffhsu_zsu, | ||
479 | - gen_helper_sve_ldffssu_zsu, } }, | ||
480 | + gen_helper_sve_ldffhsu_le_zsu, | ||
481 | + gen_helper_sve_ldffss_le_zsu, } }, | ||
482 | { { gen_helper_sve_ldffbss_zss, | ||
483 | - gen_helper_sve_ldffhss_zss, | ||
484 | + gen_helper_sve_ldffhss_le_zss, | ||
485 | NULL, }, | ||
486 | { gen_helper_sve_ldffbsu_zss, | ||
487 | - gen_helper_sve_ldffhsu_zss, | ||
488 | - gen_helper_sve_ldffssu_zss, } } } }, | ||
489 | + gen_helper_sve_ldffhsu_le_zss, | ||
490 | + gen_helper_sve_ldffss_le_zss, } } } }, | ||
491 | |||
492 | /* Big-endian */ | ||
493 | { { { { gen_helper_sve_ldbss_zsu, | ||
494 | @@ -XXX,XX +XXX,XX @@ static gen_helper_gvec_mem_scatter * const gather_load_fn32[2][2][2][2][3] = { | ||
495 | |||
496 | /* First-fault */ | ||
497 | { { { gen_helper_sve_ldffbss_zsu, | ||
498 | - gen_helper_sve_ldffhss_zsu, | ||
499 | + gen_helper_sve_ldffhss_be_zsu, | ||
500 | NULL, }, | ||
501 | { gen_helper_sve_ldffbsu_zsu, | ||
502 | - gen_helper_sve_ldffhsu_zsu, | ||
503 | - gen_helper_sve_ldffssu_zsu, } }, | ||
504 | + gen_helper_sve_ldffhsu_be_zsu, | ||
505 | + gen_helper_sve_ldffss_be_zsu, } }, | ||
506 | { { gen_helper_sve_ldffbss_zss, | ||
507 | - gen_helper_sve_ldffhss_zss, | ||
508 | + gen_helper_sve_ldffhss_be_zss, | ||
509 | NULL, }, | ||
510 | { gen_helper_sve_ldffbsu_zss, | ||
511 | - gen_helper_sve_ldffhsu_zss, | ||
512 | - gen_helper_sve_ldffssu_zss, } } } }, | ||
513 | + gen_helper_sve_ldffhsu_be_zss, | ||
514 | + gen_helper_sve_ldffss_be_zss, } } } }, | ||
515 | }; | ||
516 | |||
517 | /* Note that we overload xs=2 to indicate 64-bit offset. */ | ||
518 | @@ -XXX,XX +XXX,XX @@ static gen_helper_gvec_mem_scatter * const gather_load_fn64[2][2][3][2][4] = { | ||
519 | |||
520 | /* First-fault */ | ||
521 | { { { gen_helper_sve_ldffbds_zsu, | ||
522 | - gen_helper_sve_ldffhds_zsu, | ||
523 | - gen_helper_sve_ldffsds_zsu, | ||
524 | + gen_helper_sve_ldffhds_le_zsu, | ||
525 | + gen_helper_sve_ldffsds_le_zsu, | ||
526 | NULL, }, | ||
527 | { gen_helper_sve_ldffbdu_zsu, | ||
528 | - gen_helper_sve_ldffhdu_zsu, | ||
529 | - gen_helper_sve_ldffsdu_zsu, | ||
530 | - gen_helper_sve_ldffddu_zsu, } }, | ||
531 | + gen_helper_sve_ldffhdu_le_zsu, | ||
532 | + gen_helper_sve_ldffsdu_le_zsu, | ||
533 | + gen_helper_sve_ldffdd_le_zsu, } }, | ||
534 | { { gen_helper_sve_ldffbds_zss, | ||
535 | - gen_helper_sve_ldffhds_zss, | ||
536 | - gen_helper_sve_ldffsds_zss, | ||
537 | + gen_helper_sve_ldffhds_le_zss, | ||
538 | + gen_helper_sve_ldffsds_le_zss, | ||
539 | NULL, }, | ||
540 | { gen_helper_sve_ldffbdu_zss, | ||
541 | - gen_helper_sve_ldffhdu_zss, | ||
542 | - gen_helper_sve_ldffsdu_zss, | ||
543 | - gen_helper_sve_ldffddu_zss, } }, | ||
544 | + gen_helper_sve_ldffhdu_le_zss, | ||
545 | + gen_helper_sve_ldffsdu_le_zss, | ||
546 | + gen_helper_sve_ldffdd_le_zss, } }, | ||
547 | { { gen_helper_sve_ldffbds_zd, | ||
548 | - gen_helper_sve_ldffhds_zd, | ||
549 | - gen_helper_sve_ldffsds_zd, | ||
550 | + gen_helper_sve_ldffhds_le_zd, | ||
551 | + gen_helper_sve_ldffsds_le_zd, | ||
552 | NULL, }, | ||
553 | { gen_helper_sve_ldffbdu_zd, | ||
554 | - gen_helper_sve_ldffhdu_zd, | ||
555 | - gen_helper_sve_ldffsdu_zd, | ||
556 | - gen_helper_sve_ldffddu_zd, } } } }, | ||
557 | + gen_helper_sve_ldffhdu_le_zd, | ||
558 | + gen_helper_sve_ldffsdu_le_zd, | ||
559 | + gen_helper_sve_ldffdd_le_zd, } } } }, | ||
560 | |||
561 | /* Big-endian */ | ||
562 | { { { { gen_helper_sve_ldbds_zsu, | ||
563 | @@ -XXX,XX +XXX,XX @@ static gen_helper_gvec_mem_scatter * const gather_load_fn64[2][2][3][2][4] = { | ||
564 | |||
565 | /* First-fault */ | ||
566 | { { { gen_helper_sve_ldffbds_zsu, | ||
567 | - gen_helper_sve_ldffhds_zsu, | ||
568 | - gen_helper_sve_ldffsds_zsu, | ||
569 | + gen_helper_sve_ldffhds_be_zsu, | ||
570 | + gen_helper_sve_ldffsds_be_zsu, | ||
571 | NULL, }, | ||
572 | { gen_helper_sve_ldffbdu_zsu, | ||
573 | - gen_helper_sve_ldffhdu_zsu, | ||
574 | - gen_helper_sve_ldffsdu_zsu, | ||
575 | - gen_helper_sve_ldffddu_zsu, } }, | ||
576 | + gen_helper_sve_ldffhdu_be_zsu, | ||
577 | + gen_helper_sve_ldffsdu_be_zsu, | ||
578 | + gen_helper_sve_ldffdd_be_zsu, } }, | ||
579 | { { gen_helper_sve_ldffbds_zss, | ||
580 | - gen_helper_sve_ldffhds_zss, | ||
581 | - gen_helper_sve_ldffsds_zss, | ||
582 | + gen_helper_sve_ldffhds_be_zss, | ||
583 | + gen_helper_sve_ldffsds_be_zss, | ||
584 | NULL, }, | ||
585 | { gen_helper_sve_ldffbdu_zss, | ||
586 | - gen_helper_sve_ldffhdu_zss, | ||
587 | - gen_helper_sve_ldffsdu_zss, | ||
588 | - gen_helper_sve_ldffddu_zss, } }, | ||
589 | + gen_helper_sve_ldffhdu_be_zss, | ||
590 | + gen_helper_sve_ldffsdu_be_zss, | ||
591 | + gen_helper_sve_ldffdd_be_zss, } }, | ||
592 | { { gen_helper_sve_ldffbds_zd, | ||
593 | - gen_helper_sve_ldffhds_zd, | ||
594 | - gen_helper_sve_ldffsds_zd, | ||
595 | + gen_helper_sve_ldffhds_be_zd, | ||
596 | + gen_helper_sve_ldffsds_be_zd, | ||
597 | NULL, }, | ||
598 | { gen_helper_sve_ldffbdu_zd, | ||
599 | - gen_helper_sve_ldffhdu_zd, | ||
600 | - gen_helper_sve_ldffsdu_zd, | ||
601 | - gen_helper_sve_ldffddu_zd, } } } }, | ||
602 | + gen_helper_sve_ldffhdu_be_zd, | ||
603 | + gen_helper_sve_ldffsdu_be_zd, | ||
604 | + gen_helper_sve_ldffdd_be_zd, } } } }, | ||
605 | }; | ||
606 | |||
607 | static bool trans_LD1_zprz(DisasContext *s, arg_LD1_zprz *a, uint32_t insn) | ||
608 | -- | ||
609 | 2.19.0 | ||
610 | |||
611 | diff view generated by jsdifflib |
Deleted patch | |||
---|---|---|---|
1 | From: Richard Henderson <richard.henderson@linaro.org> | ||
2 | 1 | ||
3 | There is quite a lot of code required to compute cpu_mem_index, | ||
4 | or even put together the full TCGMemOpIdx. This can easily be | ||
5 | done at translation time. | ||
6 | |||
7 | Reviewed-by: Peter Maydell <peter.maydell@linaro.org> | ||
8 | Tested-by: Laurent Desnogues <laurent.desnogues@gmail.com> | ||
9 | Signed-off-by: Richard Henderson <richard.henderson@linaro.org> | ||
10 | Message-id: 20181005175350.30752-16-richard.henderson@linaro.org | ||
11 | Signed-off-by: Peter Maydell <peter.maydell@linaro.org> | ||
12 | --- | ||
13 | target/arm/internals.h | 5 ++ | ||
14 | target/arm/sve_helper.c | 138 +++++++++++++++++++------------------ | ||
15 | target/arm/translate-sve.c | 67 +++++++++++------- | ||
16 | 3 files changed, 121 insertions(+), 89 deletions(-) | ||
17 | |||
18 | diff --git a/target/arm/internals.h b/target/arm/internals.h | ||
19 | index XXXXXXX..XXXXXXX 100644 | ||
20 | --- a/target/arm/internals.h | ||
21 | +++ b/target/arm/internals.h | ||
22 | @@ -XXX,XX +XXX,XX @@ static inline uint32_t arm_debug_exception_fsr(CPUARMState *env) | ||
23 | } | ||
24 | } | ||
25 | |||
26 | +/* Note make_memop_idx reserves 4 bits for mmu_idx, and MO_BSWAP is bit 3. | ||
27 | + * Thus a TCGMemOpIdx, without any MO_ALIGN bits, fits in 8 bits. | ||
28 | + */ | ||
29 | +#define MEMOPIDX_SHIFT 8 | ||
30 | + | ||
31 | #endif | ||
32 | diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c | ||
33 | index XXXXXXX..XXXXXXX 100644 | ||
34 | --- a/target/arm/sve_helper.c | ||
35 | +++ b/target/arm/sve_helper.c | ||
36 | @@ -XXX,XX +XXX,XX @@ | ||
37 | |||
38 | #include "qemu/osdep.h" | ||
39 | #include "cpu.h" | ||
40 | +#include "internals.h" | ||
41 | #include "exec/exec-all.h" | ||
42 | #include "exec/cpu_ldst.h" | ||
43 | #include "exec/helper-proto.h" | ||
44 | @@ -XXX,XX +XXX,XX @@ typedef intptr_t sve_ld1_host_fn(void *vd, void *vg, void *host, | ||
45 | * The controlling predicate is known to be true. | ||
46 | */ | ||
47 | typedef void sve_ld1_tlb_fn(CPUARMState *env, void *vd, intptr_t reg_off, | ||
48 | - target_ulong vaddr, int mmu_idx, uintptr_t ra); | ||
49 | + target_ulong vaddr, TCGMemOpIdx oi, uintptr_t ra); | ||
50 | typedef sve_ld1_tlb_fn sve_st1_tlb_fn; | ||
51 | |||
52 | /* | ||
53 | @@ -XXX,XX +XXX,XX @@ static intptr_t sve_##NAME##_host(void *vd, void *vg, void *host, \ | ||
54 | #ifdef CONFIG_SOFTMMU | ||
55 | #define DO_LD_TLB(NAME, H, TYPEE, TYPEM, HOST, MOEND, TLB) \ | ||
56 | static void sve_##NAME##_tlb(CPUARMState *env, void *vd, intptr_t reg_off, \ | ||
57 | - target_ulong addr, int mmu_idx, uintptr_t ra) \ | ||
58 | + target_ulong addr, TCGMemOpIdx oi, uintptr_t ra) \ | ||
59 | { \ | ||
60 | - TCGMemOpIdx oi = make_memop_idx(ctz32(sizeof(TYPEM)) | MOEND, mmu_idx); \ | ||
61 | TYPEM val = TLB(env, addr, oi, ra); \ | ||
62 | *(TYPEE *)(vd + H(reg_off)) = val; \ | ||
63 | } | ||
64 | #else | ||
65 | #define DO_LD_TLB(NAME, H, TYPEE, TYPEM, HOST, MOEND, TLB) \ | ||
66 | static void sve_##NAME##_tlb(CPUARMState *env, void *vd, intptr_t reg_off, \ | ||
67 | - target_ulong addr, int mmu_idx, uintptr_t ra) \ | ||
68 | + target_ulong addr, TCGMemOpIdx oi, uintptr_t ra) \ | ||
69 | { \ | ||
70 | TYPEM val = HOST(g2h(addr)); \ | ||
71 | *(TYPEE *)(vd + H(reg_off)) = val; \ | ||
72 | @@ -XXX,XX +XXX,XX @@ static void sve_ld1_r(CPUARMState *env, void *vg, const target_ulong addr, | ||
73 | sve_ld1_host_fn *host_fn, | ||
74 | sve_ld1_tlb_fn *tlb_fn) | ||
75 | { | ||
76 | - void *vd = &env->vfp.zregs[simd_data(desc)]; | ||
77 | + const TCGMemOpIdx oi = extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHIFT); | ||
78 | + const int mmu_idx = get_mmuidx(oi); | ||
79 | + const unsigned rd = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 5); | ||
80 | + void *vd = &env->vfp.zregs[rd]; | ||
81 | const int diffsz = esz - msz; | ||
82 | const intptr_t reg_max = simd_oprsz(desc); | ||
83 | const intptr_t mem_max = reg_max >> diffsz; | ||
84 | - const int mmu_idx = cpu_mmu_index(env, false); | ||
85 | ARMVectorReg scratch; | ||
86 | void *host; | ||
87 | intptr_t split, reg_off, mem_off; | ||
88 | @@ -XXX,XX +XXX,XX @@ static void sve_ld1_r(CPUARMState *env, void *vg, const target_ulong addr, | ||
89 | * on I/O memory, it may succeed but not bring in the TLB entry. | ||
90 | * But even then we have still made forward progress. | ||
91 | */ | ||
92 | - tlb_fn(env, &scratch, reg_off, addr + mem_off, mmu_idx, retaddr); | ||
93 | + tlb_fn(env, &scratch, reg_off, addr + mem_off, oi, retaddr); | ||
94 | reg_off += 1 << esz; | ||
95 | } | ||
96 | #endif | ||
97 | @@ -XXX,XX +XXX,XX @@ static void sve_ld2_r(CPUARMState *env, void *vg, target_ulong addr, | ||
98 | uint32_t desc, int size, uintptr_t ra, | ||
99 | sve_ld1_tlb_fn *tlb_fn) | ||
100 | { | ||
101 | - const int mmu_idx = cpu_mmu_index(env, false); | ||
102 | + const TCGMemOpIdx oi = extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHIFT); | ||
103 | + const unsigned rd = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 5); | ||
104 | intptr_t i, oprsz = simd_oprsz(desc); | ||
105 | - unsigned rd = simd_data(desc); | ||
106 | ARMVectorReg scratch[2] = { }; | ||
107 | |||
108 | set_helper_retaddr(ra); | ||
109 | @@ -XXX,XX +XXX,XX @@ static void sve_ld2_r(CPUARMState *env, void *vg, target_ulong addr, | ||
110 | uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); | ||
111 | do { | ||
112 | if (pg & 1) { | ||
113 | - tlb_fn(env, &scratch[0], i, addr, mmu_idx, ra); | ||
114 | - tlb_fn(env, &scratch[1], i, addr + size, mmu_idx, ra); | ||
115 | + tlb_fn(env, &scratch[0], i, addr, oi, ra); | ||
116 | + tlb_fn(env, &scratch[1], i, addr + size, oi, ra); | ||
117 | } | ||
118 | i += size, pg >>= size; | ||
119 | addr += 2 * size; | ||
120 | @@ -XXX,XX +XXX,XX @@ static void sve_ld3_r(CPUARMState *env, void *vg, target_ulong addr, | ||
121 | uint32_t desc, int size, uintptr_t ra, | ||
122 | sve_ld1_tlb_fn *tlb_fn) | ||
123 | { | ||
124 | - const int mmu_idx = cpu_mmu_index(env, false); | ||
125 | + const TCGMemOpIdx oi = extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHIFT); | ||
126 | + const unsigned rd = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 5); | ||
127 | intptr_t i, oprsz = simd_oprsz(desc); | ||
128 | - unsigned rd = simd_data(desc); | ||
129 | ARMVectorReg scratch[3] = { }; | ||
130 | |||
131 | set_helper_retaddr(ra); | ||
132 | @@ -XXX,XX +XXX,XX @@ static void sve_ld3_r(CPUARMState *env, void *vg, target_ulong addr, | ||
133 | uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); | ||
134 | do { | ||
135 | if (pg & 1) { | ||
136 | - tlb_fn(env, &scratch[0], i, addr, mmu_idx, ra); | ||
137 | - tlb_fn(env, &scratch[1], i, addr + size, mmu_idx, ra); | ||
138 | - tlb_fn(env, &scratch[2], i, addr + 2 * size, mmu_idx, ra); | ||
139 | + tlb_fn(env, &scratch[0], i, addr, oi, ra); | ||
140 | + tlb_fn(env, &scratch[1], i, addr + size, oi, ra); | ||
141 | + tlb_fn(env, &scratch[2], i, addr + 2 * size, oi, ra); | ||
142 | } | ||
143 | i += size, pg >>= size; | ||
144 | addr += 3 * size; | ||
145 | @@ -XXX,XX +XXX,XX @@ static void sve_ld4_r(CPUARMState *env, void *vg, target_ulong addr, | ||
146 | uint32_t desc, int size, uintptr_t ra, | ||
147 | sve_ld1_tlb_fn *tlb_fn) | ||
148 | { | ||
149 | - const int mmu_idx = cpu_mmu_index(env, false); | ||
150 | + const TCGMemOpIdx oi = extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHIFT); | ||
151 | + const unsigned rd = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 5); | ||
152 | intptr_t i, oprsz = simd_oprsz(desc); | ||
153 | - unsigned rd = simd_data(desc); | ||
154 | ARMVectorReg scratch[4] = { }; | ||
155 | |||
156 | set_helper_retaddr(ra); | ||
157 | @@ -XXX,XX +XXX,XX @@ static void sve_ld4_r(CPUARMState *env, void *vg, target_ulong addr, | ||
158 | uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); | ||
159 | do { | ||
160 | if (pg & 1) { | ||
161 | - tlb_fn(env, &scratch[0], i, addr, mmu_idx, ra); | ||
162 | - tlb_fn(env, &scratch[1], i, addr + size, mmu_idx, ra); | ||
163 | - tlb_fn(env, &scratch[2], i, addr + 2 * size, mmu_idx, ra); | ||
164 | - tlb_fn(env, &scratch[3], i, addr + 3 * size, mmu_idx, ra); | ||
165 | + tlb_fn(env, &scratch[0], i, addr, oi, ra); | ||
166 | + tlb_fn(env, &scratch[1], i, addr + size, oi, ra); | ||
167 | + tlb_fn(env, &scratch[2], i, addr + 2 * size, oi, ra); | ||
168 | + tlb_fn(env, &scratch[3], i, addr + 3 * size, oi, ra); | ||
169 | } | ||
170 | i += size, pg >>= size; | ||
171 | addr += 4 * size; | ||
172 | @@ -XXX,XX +XXX,XX @@ static void sve_ldff1_r(CPUARMState *env, void *vg, const target_ulong addr, | ||
173 | sve_ld1_host_fn *host_fn, | ||
174 | sve_ld1_tlb_fn *tlb_fn) | ||
175 | { | ||
176 | - void *vd = &env->vfp.zregs[simd_data(desc)]; | ||
177 | + const TCGMemOpIdx oi = extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHIFT); | ||
178 | + const int mmu_idx = get_mmuidx(oi); | ||
179 | + const unsigned rd = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 5); | ||
180 | + void *vd = &env->vfp.zregs[rd]; | ||
181 | const int diffsz = esz - msz; | ||
182 | const intptr_t reg_max = simd_oprsz(desc); | ||
183 | const intptr_t mem_max = reg_max >> diffsz; | ||
184 | - const int mmu_idx = cpu_mmu_index(env, false); | ||
185 | intptr_t split, reg_off, mem_off; | ||
186 | void *host; | ||
187 | |||
188 | @@ -XXX,XX +XXX,XX @@ static void sve_ldff1_r(CPUARMState *env, void *vg, const target_ulong addr, | ||
189 | * Perform one normal read, which will fault or not. | ||
190 | * But it is likely to bring the page into the tlb. | ||
191 | */ | ||
192 | - tlb_fn(env, vd, reg_off, addr + mem_off, mmu_idx, retaddr); | ||
193 | + tlb_fn(env, vd, reg_off, addr + mem_off, oi, retaddr); | ||
194 | |||
195 | /* After any fault, zero any leading predicated false elts. */ | ||
196 | swap_memzero(vd, reg_off); | ||
197 | @@ -XXX,XX +XXX,XX @@ static void sve_ldnf1_r(CPUARMState *env, void *vg, const target_ulong addr, | ||
198 | uint32_t desc, const int esz, const int msz, | ||
199 | sve_ld1_host_fn *host_fn) | ||
200 | { | ||
201 | - void *vd = &env->vfp.zregs[simd_data(desc)]; | ||
202 | + const unsigned rd = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 5); | ||
203 | + void *vd = &env->vfp.zregs[rd]; | ||
204 | const int diffsz = esz - msz; | ||
205 | const intptr_t reg_max = simd_oprsz(desc); | ||
206 | const intptr_t mem_max = reg_max >> diffsz; | ||
207 | @@ -XXX,XX +XXX,XX @@ DO_LDFF1_LDNF1_2(dd, 3, 3) | ||
208 | #ifdef CONFIG_SOFTMMU | ||
209 | #define DO_ST_TLB(NAME, H, TYPEM, HOST, MOEND, TLB) \ | ||
210 | static void sve_##NAME##_tlb(CPUARMState *env, void *vd, intptr_t reg_off, \ | ||
211 | - target_ulong addr, int mmu_idx, uintptr_t ra) \ | ||
212 | + target_ulong addr, TCGMemOpIdx oi, uintptr_t ra) \ | ||
213 | { \ | ||
214 | - TCGMemOpIdx oi = make_memop_idx(ctz32(sizeof(TYPEM)) | MOEND, mmu_idx); \ | ||
215 | TLB(env, addr, *(TYPEM *)(vd + H(reg_off)), oi, ra); \ | ||
216 | } | ||
217 | #else | ||
218 | #define DO_ST_TLB(NAME, H, TYPEM, HOST, MOEND, TLB) \ | ||
219 | static void sve_##NAME##_tlb(CPUARMState *env, void *vd, intptr_t reg_off, \ | ||
220 | - target_ulong addr, int mmu_idx, uintptr_t ra) \ | ||
221 | + target_ulong addr, TCGMemOpIdx oi, uintptr_t ra) \ | ||
222 | { \ | ||
223 | HOST(g2h(addr), *(TYPEM *)(vd + H(reg_off))); \ | ||
224 | } | ||
225 | @@ -XXX,XX +XXX,XX @@ static void sve_st1_r(CPUARMState *env, void *vg, target_ulong addr, | ||
226 | const int esize, const int msize, | ||
227 | sve_st1_tlb_fn *tlb_fn) | ||
228 | { | ||
229 | - const int mmu_idx = cpu_mmu_index(env, false); | ||
230 | + const TCGMemOpIdx oi = extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHIFT); | ||
231 | + const unsigned rd = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 5); | ||
232 | intptr_t i, oprsz = simd_oprsz(desc); | ||
233 | - unsigned rd = simd_data(desc); | ||
234 | void *vd = &env->vfp.zregs[rd]; | ||
235 | |||
236 | set_helper_retaddr(ra); | ||
237 | @@ -XXX,XX +XXX,XX @@ static void sve_st1_r(CPUARMState *env, void *vg, target_ulong addr, | ||
238 | uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); | ||
239 | do { | ||
240 | if (pg & 1) { | ||
241 | - tlb_fn(env, vd, i, addr, mmu_idx, ra); | ||
242 | + tlb_fn(env, vd, i, addr, oi, ra); | ||
243 | } | ||
244 | i += esize, pg >>= esize; | ||
245 | addr += msize; | ||
246 | @@ -XXX,XX +XXX,XX @@ static void sve_st2_r(CPUARMState *env, void *vg, target_ulong addr, | ||
247 | const int esize, const int msize, | ||
248 | sve_st1_tlb_fn *tlb_fn) | ||
249 | { | ||
250 | - const int mmu_idx = cpu_mmu_index(env, false); | ||
251 | + const TCGMemOpIdx oi = extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHIFT); | ||
252 | + const unsigned rd = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 5); | ||
253 | intptr_t i, oprsz = simd_oprsz(desc); | ||
254 | - unsigned rd = simd_data(desc); | ||
255 | void *d1 = &env->vfp.zregs[rd]; | ||
256 | void *d2 = &env->vfp.zregs[(rd + 1) & 31]; | ||
257 | |||
258 | @@ -XXX,XX +XXX,XX @@ static void sve_st2_r(CPUARMState *env, void *vg, target_ulong addr, | ||
259 | uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); | ||
260 | do { | ||
261 | if (pg & 1) { | ||
262 | - tlb_fn(env, d1, i, addr, mmu_idx, ra); | ||
263 | - tlb_fn(env, d2, i, addr + msize, mmu_idx, ra); | ||
264 | + tlb_fn(env, d1, i, addr, oi, ra); | ||
265 | + tlb_fn(env, d2, i, addr + msize, oi, ra); | ||
266 | } | ||
267 | i += esize, pg >>= esize; | ||
268 | addr += 2 * msize; | ||
269 | @@ -XXX,XX +XXX,XX @@ static void sve_st3_r(CPUARMState *env, void *vg, target_ulong addr, | ||
270 | const int esize, const int msize, | ||
271 | sve_st1_tlb_fn *tlb_fn) | ||
272 | { | ||
273 | - const int mmu_idx = cpu_mmu_index(env, false); | ||
274 | + const TCGMemOpIdx oi = extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHIFT); | ||
275 | + const unsigned rd = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 5); | ||
276 | intptr_t i, oprsz = simd_oprsz(desc); | ||
277 | - unsigned rd = simd_data(desc); | ||
278 | void *d1 = &env->vfp.zregs[rd]; | ||
279 | void *d2 = &env->vfp.zregs[(rd + 1) & 31]; | ||
280 | void *d3 = &env->vfp.zregs[(rd + 2) & 31]; | ||
281 | @@ -XXX,XX +XXX,XX @@ static void sve_st3_r(CPUARMState *env, void *vg, target_ulong addr, | ||
282 | uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); | ||
283 | do { | ||
284 | if (pg & 1) { | ||
285 | - tlb_fn(env, d1, i, addr, mmu_idx, ra); | ||
286 | - tlb_fn(env, d2, i, addr + msize, mmu_idx, ra); | ||
287 | - tlb_fn(env, d3, i, addr + 2 * msize, mmu_idx, ra); | ||
288 | + tlb_fn(env, d1, i, addr, oi, ra); | ||
289 | + tlb_fn(env, d2, i, addr + msize, oi, ra); | ||
290 | + tlb_fn(env, d3, i, addr + 2 * msize, oi, ra); | ||
291 | } | ||
292 | i += esize, pg >>= esize; | ||
293 | addr += 3 * msize; | ||
294 | @@ -XXX,XX +XXX,XX @@ static void sve_st4_r(CPUARMState *env, void *vg, target_ulong addr, | ||
295 | const int esize, const int msize, | ||
296 | sve_st1_tlb_fn *tlb_fn) | ||
297 | { | ||
298 | - const int mmu_idx = cpu_mmu_index(env, false); | ||
299 | + const TCGMemOpIdx oi = extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHIFT); | ||
300 | + const unsigned rd = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 5); | ||
301 | intptr_t i, oprsz = simd_oprsz(desc); | ||
302 | - unsigned rd = simd_data(desc); | ||
303 | void *d1 = &env->vfp.zregs[rd]; | ||
304 | void *d2 = &env->vfp.zregs[(rd + 1) & 31]; | ||
305 | void *d3 = &env->vfp.zregs[(rd + 2) & 31]; | ||
306 | @@ -XXX,XX +XXX,XX @@ static void sve_st4_r(CPUARMState *env, void *vg, target_ulong addr, | ||
307 | uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); | ||
308 | do { | ||
309 | if (pg & 1) { | ||
310 | - tlb_fn(env, d1, i, addr, mmu_idx, ra); | ||
311 | - tlb_fn(env, d2, i, addr + msize, mmu_idx, ra); | ||
312 | - tlb_fn(env, d3, i, addr + 2 * msize, mmu_idx, ra); | ||
313 | - tlb_fn(env, d4, i, addr + 3 * msize, mmu_idx, ra); | ||
314 | + tlb_fn(env, d1, i, addr, oi, ra); | ||
315 | + tlb_fn(env, d2, i, addr + msize, oi, ra); | ||
316 | + tlb_fn(env, d3, i, addr + 2 * msize, oi, ra); | ||
317 | + tlb_fn(env, d4, i, addr + 3 * msize, oi, ra); | ||
318 | } | ||
319 | i += esize, pg >>= esize; | ||
320 | addr += 4 * msize; | ||
321 | @@ -XXX,XX +XXX,XX @@ static void sve_ld1_zs(CPUARMState *env, void *vd, void *vg, void *vm, | ||
322 | target_ulong base, uint32_t desc, uintptr_t ra, | ||
323 | zreg_off_fn *off_fn, sve_ld1_tlb_fn *tlb_fn) | ||
324 | { | ||
325 | - const int mmu_idx = cpu_mmu_index(env, false); | ||
326 | + const TCGMemOpIdx oi = extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHIFT); | ||
327 | + const int scale = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 2); | ||
328 | intptr_t i, oprsz = simd_oprsz(desc); | ||
329 | - unsigned scale = simd_data(desc); | ||
330 | ARMVectorReg scratch = { }; | ||
331 | |||
332 | set_helper_retaddr(ra); | ||
333 | @@ -XXX,XX +XXX,XX @@ static void sve_ld1_zs(CPUARMState *env, void *vd, void *vg, void *vm, | ||
334 | do { | ||
335 | if (likely(pg & 1)) { | ||
336 | target_ulong off = off_fn(vm, i); | ||
337 | - tlb_fn(env, &scratch, i, base + (off << scale), mmu_idx, ra); | ||
338 | + tlb_fn(env, &scratch, i, base + (off << scale), oi, ra); | ||
339 | } | ||
340 | i += 4, pg >>= 4; | ||
341 | } while (i & 15); | ||
342 | @@ -XXX,XX +XXX,XX @@ static void sve_ld1_zd(CPUARMState *env, void *vd, void *vg, void *vm, | ||
343 | target_ulong base, uint32_t desc, uintptr_t ra, | ||
344 | zreg_off_fn *off_fn, sve_ld1_tlb_fn *tlb_fn) | ||
345 | { | ||
346 | - const int mmu_idx = cpu_mmu_index(env, false); | ||
347 | + const TCGMemOpIdx oi = extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHIFT); | ||
348 | + const int scale = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 2); | ||
349 | intptr_t i, oprsz = simd_oprsz(desc) / 8; | ||
350 | - unsigned scale = simd_data(desc); | ||
351 | ARMVectorReg scratch = { }; | ||
352 | |||
353 | set_helper_retaddr(ra); | ||
354 | @@ -XXX,XX +XXX,XX @@ static void sve_ld1_zd(CPUARMState *env, void *vd, void *vg, void *vm, | ||
355 | uint8_t pg = *(uint8_t *)(vg + H1(i)); | ||
356 | if (likely(pg & 1)) { | ||
357 | target_ulong off = off_fn(vm, i * 8); | ||
358 | - tlb_fn(env, &scratch, i * 8, base + (off << scale), mmu_idx, ra); | ||
359 | + tlb_fn(env, &scratch, i * 8, base + (off << scale), oi, ra); | ||
360 | } | ||
361 | } | ||
362 | set_helper_retaddr(0); | ||
363 | @@ -XXX,XX +XXX,XX @@ typedef bool sve_ld1_nf_fn(CPUARMState *env, void *vd, intptr_t reg_off, | ||
364 | #ifdef CONFIG_SOFTMMU | ||
365 | #define DO_LD_NF(NAME, H, TYPEE, TYPEM, HOST) \ | ||
366 | static bool sve_ld##NAME##_nf(CPUARMState *env, void *vd, intptr_t reg_off, \ | ||
367 | - target_ulong addr, int mmu_idx) \ | ||
368 | + target_ulong addr, int mmu_idx) \ | ||
369 | { \ | ||
370 | target_ulong next_page = -(addr | TARGET_PAGE_MASK); \ | ||
371 | if (likely(next_page - addr >= sizeof(TYPEM))) { \ | ||
372 | @@ -XXX,XX +XXX,XX @@ static inline void sve_ldff1_zs(CPUARMState *env, void *vd, void *vg, void *vm, | ||
373 | zreg_off_fn *off_fn, sve_ld1_tlb_fn *tlb_fn, | ||
374 | sve_ld1_nf_fn *nonfault_fn) | ||
375 | { | ||
376 | - const int mmu_idx = cpu_mmu_index(env, false); | ||
377 | + const TCGMemOpIdx oi = extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHIFT); | ||
378 | + const int mmu_idx = get_mmuidx(oi); | ||
379 | + const int scale = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 2); | ||
380 | intptr_t reg_off, reg_max = simd_oprsz(desc); | ||
381 | - unsigned scale = simd_data(desc); | ||
382 | target_ulong addr; | ||
383 | |||
384 | /* Skip to the first true predicate. */ | ||
385 | @@ -XXX,XX +XXX,XX @@ static inline void sve_ldff1_zs(CPUARMState *env, void *vd, void *vg, void *vm, | ||
386 | set_helper_retaddr(ra); | ||
387 | addr = off_fn(vm, reg_off); | ||
388 | addr = base + (addr << scale); | ||
389 | - tlb_fn(env, vd, reg_off, addr, mmu_idx, ra); | ||
390 | + tlb_fn(env, vd, reg_off, addr, oi, ra); | ||
391 | |||
392 | /* The rest of the reads will be non-faulting. */ | ||
393 | set_helper_retaddr(0); | ||
394 | @@ -XXX,XX +XXX,XX @@ static inline void sve_ldff1_zd(CPUARMState *env, void *vd, void *vg, void *vm, | ||
395 | zreg_off_fn *off_fn, sve_ld1_tlb_fn *tlb_fn, | ||
396 | sve_ld1_nf_fn *nonfault_fn) | ||
397 | { | ||
398 | - const int mmu_idx = cpu_mmu_index(env, false); | ||
399 | + const TCGMemOpIdx oi = extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHIFT); | ||
400 | + const int mmu_idx = get_mmuidx(oi); | ||
401 | + const int scale = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 2); | ||
402 | intptr_t reg_off, reg_max = simd_oprsz(desc); | ||
403 | - unsigned scale = simd_data(desc); | ||
404 | target_ulong addr; | ||
405 | |||
406 | /* Skip to the first true predicate. */ | ||
407 | @@ -XXX,XX +XXX,XX @@ static inline void sve_ldff1_zd(CPUARMState *env, void *vd, void *vg, void *vm, | ||
408 | set_helper_retaddr(ra); | ||
409 | addr = off_fn(vm, reg_off); | ||
410 | addr = base + (addr << scale); | ||
411 | - tlb_fn(env, vd, reg_off, addr, mmu_idx, ra); | ||
412 | + tlb_fn(env, vd, reg_off, addr, oi, ra); | ||
413 | |||
414 | /* The rest of the reads will be non-faulting. */ | ||
415 | set_helper_retaddr(0); | ||
416 | @@ -XXX,XX +XXX,XX @@ static void sve_st1_zs(CPUARMState *env, void *vd, void *vg, void *vm, | ||
417 | target_ulong base, uint32_t desc, uintptr_t ra, | ||
418 | zreg_off_fn *off_fn, sve_ld1_tlb_fn *tlb_fn) | ||
419 | { | ||
420 | - const int mmu_idx = cpu_mmu_index(env, false); | ||
421 | + const TCGMemOpIdx oi = extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHIFT); | ||
422 | + const int scale = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 2); | ||
423 | intptr_t i, oprsz = simd_oprsz(desc); | ||
424 | - unsigned scale = simd_data(desc); | ||
425 | |||
426 | set_helper_retaddr(ra); | ||
427 | for (i = 0; i < oprsz; ) { | ||
428 | @@ -XXX,XX +XXX,XX @@ static void sve_st1_zs(CPUARMState *env, void *vd, void *vg, void *vm, | ||
429 | do { | ||
430 | if (likely(pg & 1)) { | ||
431 | target_ulong off = off_fn(vm, i); | ||
432 | - tlb_fn(env, vd, i, base + (off << scale), mmu_idx, ra); | ||
433 | + tlb_fn(env, vd, i, base + (off << scale), oi, ra); | ||
434 | } | ||
435 | i += 4, pg >>= 4; | ||
436 | } while (i & 15); | ||
437 | @@ -XXX,XX +XXX,XX @@ static void sve_st1_zd(CPUARMState *env, void *vd, void *vg, void *vm, | ||
438 | target_ulong base, uint32_t desc, uintptr_t ra, | ||
439 | zreg_off_fn *off_fn, sve_ld1_tlb_fn *tlb_fn) | ||
440 | { | ||
441 | - const int mmu_idx = cpu_mmu_index(env, false); | ||
442 | + const TCGMemOpIdx oi = extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHIFT); | ||
443 | + const int scale = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 2); | ||
444 | intptr_t i, oprsz = simd_oprsz(desc) / 8; | ||
445 | - unsigned scale = simd_data(desc); | ||
446 | |||
447 | set_helper_retaddr(ra); | ||
448 | for (i = 0; i < oprsz; i++) { | ||
449 | uint8_t pg = *(uint8_t *)(vg + H1(i)); | ||
450 | if (likely(pg & 1)) { | ||
451 | target_ulong off = off_fn(vm, i * 8); | ||
452 | - tlb_fn(env, vd, i * 8, base + (off << scale), mmu_idx, ra); | ||
453 | + tlb_fn(env, vd, i * 8, base + (off << scale), oi, ra); | ||
454 | } | ||
455 | } | ||
456 | set_helper_retaddr(0); | ||
457 | diff --git a/target/arm/translate-sve.c b/target/arm/translate-sve.c | ||
458 | index XXXXXXX..XXXXXXX 100644 | ||
459 | --- a/target/arm/translate-sve.c | ||
460 | +++ b/target/arm/translate-sve.c | ||
461 | @@ -XXX,XX +XXX,XX @@ static const uint8_t dtype_esz[16] = { | ||
462 | 3, 2, 1, 3 | ||
463 | }; | ||
464 | |||
465 | +static TCGMemOpIdx sve_memopidx(DisasContext *s, int dtype) | ||
466 | +{ | ||
467 | + return make_memop_idx(s->be_data | dtype_mop[dtype], get_mem_index(s)); | ||
468 | +} | ||
469 | + | ||
470 | static void do_mem_zpa(DisasContext *s, int zt, int pg, TCGv_i64 addr, | ||
471 | - gen_helper_gvec_mem *fn) | ||
472 | + int dtype, gen_helper_gvec_mem *fn) | ||
473 | { | ||
474 | unsigned vsz = vec_full_reg_size(s); | ||
475 | TCGv_ptr t_pg; | ||
476 | - TCGv_i32 desc; | ||
477 | + TCGv_i32 t_desc; | ||
478 | + int desc; | ||
479 | |||
480 | /* For e.g. LD4, there are not enough arguments to pass all 4 | ||
481 | * registers as pointers, so encode the regno into the data field. | ||
482 | * For consistency, do this even for LD1. | ||
483 | */ | ||
484 | - desc = tcg_const_i32(simd_desc(vsz, vsz, zt)); | ||
485 | + desc = sve_memopidx(s, dtype); | ||
486 | + desc |= zt << MEMOPIDX_SHIFT; | ||
487 | + desc = simd_desc(vsz, vsz, desc); | ||
488 | + t_desc = tcg_const_i32(desc); | ||
489 | t_pg = tcg_temp_new_ptr(); | ||
490 | |||
491 | tcg_gen_addi_ptr(t_pg, cpu_env, pred_full_reg_offset(s, pg)); | ||
492 | - fn(cpu_env, t_pg, addr, desc); | ||
493 | + fn(cpu_env, t_pg, addr, t_desc); | ||
494 | |||
495 | tcg_temp_free_ptr(t_pg); | ||
496 | - tcg_temp_free_i32(desc); | ||
497 | + tcg_temp_free_i32(t_desc); | ||
498 | } | ||
499 | |||
500 | static void do_ld_zpa(DisasContext *s, int zt, int pg, | ||
501 | @@ -XXX,XX +XXX,XX @@ static void do_ld_zpa(DisasContext *s, int zt, int pg, | ||
502 | * accessible via the instruction encoding. | ||
503 | */ | ||
504 | assert(fn != NULL); | ||
505 | - do_mem_zpa(s, zt, pg, addr, fn); | ||
506 | + do_mem_zpa(s, zt, pg, addr, dtype, fn); | ||
507 | } | ||
508 | |||
509 | static bool trans_LD_zprr(DisasContext *s, arg_rprr_load *a, uint32_t insn) | ||
510 | @@ -XXX,XX +XXX,XX @@ static bool trans_LDFF1_zprr(DisasContext *s, arg_rprr_load *a, uint32_t insn) | ||
511 | TCGv_i64 addr = new_tmp_a64(s); | ||
512 | tcg_gen_shli_i64(addr, cpu_reg(s, a->rm), dtype_msz(a->dtype)); | ||
513 | tcg_gen_add_i64(addr, addr, cpu_reg_sp(s, a->rn)); | ||
514 | - do_mem_zpa(s, a->rd, a->pg, addr, fns[s->be_data == MO_BE][a->dtype]); | ||
515 | + do_mem_zpa(s, a->rd, a->pg, addr, a->dtype, | ||
516 | + fns[s->be_data == MO_BE][a->dtype]); | ||
517 | } | ||
518 | return true; | ||
519 | } | ||
520 | @@ -XXX,XX +XXX,XX @@ static bool trans_LDNF1_zpri(DisasContext *s, arg_rpri_load *a, uint32_t insn) | ||
521 | TCGv_i64 addr = new_tmp_a64(s); | ||
522 | |||
523 | tcg_gen_addi_i64(addr, cpu_reg_sp(s, a->rn), off); | ||
524 | - do_mem_zpa(s, a->rd, a->pg, addr, fns[s->be_data == MO_BE][a->dtype]); | ||
525 | + do_mem_zpa(s, a->rd, a->pg, addr, a->dtype, | ||
526 | + fns[s->be_data == MO_BE][a->dtype]); | ||
527 | } | ||
528 | return true; | ||
529 | } | ||
530 | @@ -XXX,XX +XXX,XX @@ static void do_ldrq(DisasContext *s, int zt, int pg, TCGv_i64 addr, int msz) | ||
531 | }; | ||
532 | unsigned vsz = vec_full_reg_size(s); | ||
533 | TCGv_ptr t_pg; | ||
534 | - TCGv_i32 desc; | ||
535 | - int poff; | ||
536 | + TCGv_i32 t_desc; | ||
537 | + int desc, poff; | ||
538 | |||
539 | /* Load the first quadword using the normal predicated load helpers. */ | ||
540 | - desc = tcg_const_i32(simd_desc(16, 16, zt)); | ||
541 | + desc = sve_memopidx(s, msz_dtype(msz)); | ||
542 | + desc |= zt << MEMOPIDX_SHIFT; | ||
543 | + desc = simd_desc(16, 16, desc); | ||
544 | + t_desc = tcg_const_i32(desc); | ||
545 | |||
546 | poff = pred_full_reg_offset(s, pg); | ||
547 | if (vsz > 16) { | ||
548 | @@ -XXX,XX +XXX,XX @@ static void do_ldrq(DisasContext *s, int zt, int pg, TCGv_i64 addr, int msz) | ||
549 | t_pg = tcg_temp_new_ptr(); | ||
550 | tcg_gen_addi_ptr(t_pg, cpu_env, poff); | ||
551 | |||
552 | - fns[s->be_data == MO_BE][msz](cpu_env, t_pg, addr, desc); | ||
553 | + fns[s->be_data == MO_BE][msz](cpu_env, t_pg, addr, t_desc); | ||
554 | |||
555 | tcg_temp_free_ptr(t_pg); | ||
556 | - tcg_temp_free_i32(desc); | ||
557 | + tcg_temp_free_i32(t_desc); | ||
558 | |||
559 | /* Replicate that first quadword. */ | ||
560 | if (vsz > 16) { | ||
561 | @@ -XXX,XX +XXX,XX @@ static void do_st_zpa(DisasContext *s, int zt, int pg, TCGv_i64 addr, | ||
562 | fn = fn_multiple[be][nreg - 1][msz]; | ||
563 | } | ||
564 | assert(fn != NULL); | ||
565 | - do_mem_zpa(s, zt, pg, addr, fn); | ||
566 | + do_mem_zpa(s, zt, pg, addr, msz_dtype(msz), fn); | ||
567 | } | ||
568 | |||
569 | static bool trans_ST_zprr(DisasContext *s, arg_rprr_store *a, uint32_t insn) | ||
570 | @@ -XXX,XX +XXX,XX @@ static bool trans_ST_zpri(DisasContext *s, arg_rpri_store *a, uint32_t insn) | ||
571 | *** SVE gather loads / scatter stores | ||
572 | */ | ||
573 | |||
574 | -static void do_mem_zpz(DisasContext *s, int zt, int pg, int zm, int scale, | ||
575 | - TCGv_i64 scalar, gen_helper_gvec_mem_scatter *fn) | ||
576 | +static void do_mem_zpz(DisasContext *s, int zt, int pg, int zm, | ||
577 | + int scale, TCGv_i64 scalar, int msz, | ||
578 | + gen_helper_gvec_mem_scatter *fn) | ||
579 | { | ||
580 | unsigned vsz = vec_full_reg_size(s); | ||
581 | - TCGv_i32 desc = tcg_const_i32(simd_desc(vsz, vsz, scale)); | ||
582 | TCGv_ptr t_zm = tcg_temp_new_ptr(); | ||
583 | TCGv_ptr t_pg = tcg_temp_new_ptr(); | ||
584 | TCGv_ptr t_zt = tcg_temp_new_ptr(); | ||
585 | + TCGv_i32 t_desc; | ||
586 | + int desc; | ||
587 | + | ||
588 | + desc = sve_memopidx(s, msz_dtype(msz)); | ||
589 | + desc |= scale << MEMOPIDX_SHIFT; | ||
590 | + desc = simd_desc(vsz, vsz, desc); | ||
591 | + t_desc = tcg_const_i32(desc); | ||
592 | |||
593 | tcg_gen_addi_ptr(t_pg, cpu_env, pred_full_reg_offset(s, pg)); | ||
594 | tcg_gen_addi_ptr(t_zm, cpu_env, vec_full_reg_offset(s, zm)); | ||
595 | tcg_gen_addi_ptr(t_zt, cpu_env, vec_full_reg_offset(s, zt)); | ||
596 | - fn(cpu_env, t_zt, t_pg, t_zm, scalar, desc); | ||
597 | + fn(cpu_env, t_zt, t_pg, t_zm, scalar, t_desc); | ||
598 | |||
599 | tcg_temp_free_ptr(t_zt); | ||
600 | tcg_temp_free_ptr(t_zm); | ||
601 | tcg_temp_free_ptr(t_pg); | ||
602 | - tcg_temp_free_i32(desc); | ||
603 | + tcg_temp_free_i32(t_desc); | ||
604 | } | ||
605 | |||
606 | /* Indexed by [be][ff][xs][u][msz]. */ | ||
607 | @@ -XXX,XX +XXX,XX @@ static bool trans_LD1_zprz(DisasContext *s, arg_LD1_zprz *a, uint32_t insn) | ||
608 | assert(fn != NULL); | ||
609 | |||
610 | do_mem_zpz(s, a->rd, a->pg, a->rm, a->scale * a->msz, | ||
611 | - cpu_reg_sp(s, a->rn), fn); | ||
612 | + cpu_reg_sp(s, a->rn), a->msz, fn); | ||
613 | return true; | ||
614 | } | ||
615 | |||
616 | @@ -XXX,XX +XXX,XX @@ static bool trans_LD1_zpiz(DisasContext *s, arg_LD1_zpiz *a, uint32_t insn) | ||
617 | * by loading the immediate into the scalar parameter. | ||
618 | */ | ||
619 | imm = tcg_const_i64(a->imm << a->msz); | ||
620 | - do_mem_zpz(s, a->rd, a->pg, a->rn, 0, imm, fn); | ||
621 | + do_mem_zpz(s, a->rd, a->pg, a->rn, 0, imm, a->msz, fn); | ||
622 | tcg_temp_free_i64(imm); | ||
623 | return true; | ||
624 | } | ||
625 | @@ -XXX,XX +XXX,XX @@ static bool trans_ST1_zprz(DisasContext *s, arg_ST1_zprz *a, uint32_t insn) | ||
626 | g_assert_not_reached(); | ||
627 | } | ||
628 | do_mem_zpz(s, a->rd, a->pg, a->rm, a->scale * a->msz, | ||
629 | - cpu_reg_sp(s, a->rn), fn); | ||
630 | + cpu_reg_sp(s, a->rn), a->msz, fn); | ||
631 | return true; | ||
632 | } | ||
633 | |||
634 | @@ -XXX,XX +XXX,XX @@ static bool trans_ST1_zpiz(DisasContext *s, arg_ST1_zpiz *a, uint32_t insn) | ||
635 | * by loading the immediate into the scalar parameter. | ||
636 | */ | ||
637 | imm = tcg_const_i64(a->imm << a->msz); | ||
638 | - do_mem_zpz(s, a->rd, a->pg, a->rn, 0, imm, fn); | ||
639 | + do_mem_zpz(s, a->rd, a->pg, a->rn, 0, imm, a->msz, fn); | ||
640 | tcg_temp_free_i64(imm); | ||
641 | return true; | ||
642 | } | ||
643 | -- | ||
644 | 2.19.0 | ||
645 | |||
646 | diff view generated by jsdifflib |
Deleted patch | |||
---|---|---|---|
1 | The Arm v8M architecture includes hardware stack limit checking. | ||
2 | When certain instructions update the stack pointer, if the new | ||
3 | value of SP is below the limit set in the associated limit register | ||
4 | then an exception is taken. Add a TB flag that tracks whether | ||
5 | the limit-checking code needs to be emitted. | ||
6 | 1 | ||
7 | Signed-off-by: Peter Maydell <peter.maydell@linaro.org> | ||
8 | Reviewed-by: Richard Henderson <richard.henderson@linaro.org> | ||
9 | Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> | ||
10 | Message-id: 20181002163556.10279-2-peter.maydell@linaro.org | ||
11 | --- | ||
12 | target/arm/cpu.h | 7 +++++++ | ||
13 | target/arm/translate.h | 1 + | ||
14 | target/arm/helper.c | 10 ++++++++++ | ||
15 | target/arm/translate.c | 1 + | ||
16 | 4 files changed, 19 insertions(+) | ||
17 | |||
18 | diff --git a/target/arm/cpu.h b/target/arm/cpu.h | ||
19 | index XXXXXXX..XXXXXXX 100644 | ||
20 | --- a/target/arm/cpu.h | ||
21 | +++ b/target/arm/cpu.h | ||
22 | @@ -XXX,XX +XXX,XX @@ FIELD(V7M_CCR, UNALIGN_TRP, 3, 1) | ||
23 | FIELD(V7M_CCR, DIV_0_TRP, 4, 1) | ||
24 | FIELD(V7M_CCR, BFHFNMIGN, 8, 1) | ||
25 | FIELD(V7M_CCR, STKALIGN, 9, 1) | ||
26 | +FIELD(V7M_CCR, STKOFHFNMIGN, 10, 1) | ||
27 | FIELD(V7M_CCR, DC, 16, 1) | ||
28 | FIELD(V7M_CCR, IC, 17, 1) | ||
29 | +FIELD(V7M_CCR, BP, 18, 1) | ||
30 | |||
31 | /* V7M SCR bits */ | ||
32 | FIELD(V7M_SCR, SLEEPONEXIT, 1, 1) | ||
33 | @@ -XXX,XX +XXX,XX @@ static inline bool arm_cpu_data_is_big_endian(CPUARMState *env) | ||
34 | /* For M profile only, Handler (ie not Thread) mode */ | ||
35 | #define ARM_TBFLAG_HANDLER_SHIFT 21 | ||
36 | #define ARM_TBFLAG_HANDLER_MASK (1 << ARM_TBFLAG_HANDLER_SHIFT) | ||
37 | +/* For M profile only, whether we should generate stack-limit checks */ | ||
38 | +#define ARM_TBFLAG_STACKCHECK_SHIFT 22 | ||
39 | +#define ARM_TBFLAG_STACKCHECK_MASK (1 << ARM_TBFLAG_STACKCHECK_SHIFT) | ||
40 | |||
41 | /* Bit usage when in AArch64 state */ | ||
42 | #define ARM_TBFLAG_TBI0_SHIFT 0 /* TBI0 for EL0/1 or TBI for EL2/3 */ | ||
43 | @@ -XXX,XX +XXX,XX @@ static inline bool arm_cpu_data_is_big_endian(CPUARMState *env) | ||
44 | (((F) & ARM_TBFLAG_BE_DATA_MASK) >> ARM_TBFLAG_BE_DATA_SHIFT) | ||
45 | #define ARM_TBFLAG_HANDLER(F) \ | ||
46 | (((F) & ARM_TBFLAG_HANDLER_MASK) >> ARM_TBFLAG_HANDLER_SHIFT) | ||
47 | +#define ARM_TBFLAG_STACKCHECK(F) \ | ||
48 | + (((F) & ARM_TBFLAG_STACKCHECK_MASK) >> ARM_TBFLAG_STACKCHECK_SHIFT) | ||
49 | #define ARM_TBFLAG_TBI0(F) \ | ||
50 | (((F) & ARM_TBFLAG_TBI0_MASK) >> ARM_TBFLAG_TBI0_SHIFT) | ||
51 | #define ARM_TBFLAG_TBI1(F) \ | ||
52 | diff --git a/target/arm/translate.h b/target/arm/translate.h | ||
53 | index XXXXXXX..XXXXXXX 100644 | ||
54 | --- a/target/arm/translate.h | ||
55 | +++ b/target/arm/translate.h | ||
56 | @@ -XXX,XX +XXX,XX @@ typedef struct DisasContext { | ||
57 | int vec_stride; | ||
58 | bool v7m_handler_mode; | ||
59 | bool v8m_secure; /* true if v8M and we're in Secure mode */ | ||
60 | + bool v8m_stackcheck; /* true if we need to perform v8M stack limit checks */ | ||
61 | /* Immediate value in AArch32 SVC insn; must be set if is_jmp == DISAS_SWI | ||
62 | * so that top level loop can generate correct syndrome information. | ||
63 | */ | ||
64 | diff --git a/target/arm/helper.c b/target/arm/helper.c | ||
65 | index XXXXXXX..XXXXXXX 100644 | ||
66 | --- a/target/arm/helper.c | ||
67 | +++ b/target/arm/helper.c | ||
68 | @@ -XXX,XX +XXX,XX @@ void cpu_get_tb_cpu_state(CPUARMState *env, target_ulong *pc, | ||
69 | flags |= ARM_TBFLAG_HANDLER_MASK; | ||
70 | } | ||
71 | |||
72 | + /* v8M always applies stack limit checks unless CCR.STKOFHFNMIGN is | ||
73 | + * suppressing them because the requested execution priority is less than 0. | ||
74 | + */ | ||
75 | + if (arm_feature(env, ARM_FEATURE_V8) && | ||
76 | + arm_feature(env, ARM_FEATURE_M) && | ||
77 | + !((mmu_idx & ARM_MMU_IDX_M_NEGPRI) && | ||
78 | + (env->v7m.ccr[env->v7m.secure] & R_V7M_CCR_STKOFHFNMIGN_MASK))) { | ||
79 | + flags |= ARM_TBFLAG_STACKCHECK_MASK; | ||
80 | + } | ||
81 | + | ||
82 | *pflags = flags; | ||
83 | *cs_base = 0; | ||
84 | } | ||
85 | diff --git a/target/arm/translate.c b/target/arm/translate.c | ||
86 | index XXXXXXX..XXXXXXX 100644 | ||
87 | --- a/target/arm/translate.c | ||
88 | +++ b/target/arm/translate.c | ||
89 | @@ -XXX,XX +XXX,XX @@ static void arm_tr_init_disas_context(DisasContextBase *dcbase, CPUState *cs) | ||
90 | dc->v7m_handler_mode = ARM_TBFLAG_HANDLER(dc->base.tb->flags); | ||
91 | dc->v8m_secure = arm_feature(env, ARM_FEATURE_M_SECURITY) && | ||
92 | regime_is_secure(env, dc->mmu_idx); | ||
93 | + dc->v8m_stackcheck = ARM_TBFLAG_STACKCHECK(dc->base.tb->flags); | ||
94 | dc->cp_regs = cpu->cp_regs; | ||
95 | dc->features = env->features; | ||
96 | |||
97 | -- | ||
98 | 2.19.0 | ||
99 | |||
100 | diff view generated by jsdifflib |
Deleted patch | |||
---|---|---|---|
1 | Define EXCP_STKOF, and arrange for it to cause us to take | ||
2 | a UsageFault with CFSR.STKOF set. | ||
3 | 1 | ||
4 | Signed-off-by: Peter Maydell <peter.maydell@linaro.org> | ||
5 | Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> | ||
6 | Reviewed-by: Richard Henderson <richard.henderson@linaro.org> | ||
7 | Message-id: 20181002163556.10279-3-peter.maydell@linaro.org | ||
8 | --- | ||
9 | target/arm/cpu.h | 2 ++ | ||
10 | target/arm/helper.c | 5 +++++ | ||
11 | 2 files changed, 7 insertions(+) | ||
12 | |||
13 | diff --git a/target/arm/cpu.h b/target/arm/cpu.h | ||
14 | index XXXXXXX..XXXXXXX 100644 | ||
15 | --- a/target/arm/cpu.h | ||
16 | +++ b/target/arm/cpu.h | ||
17 | @@ -XXX,XX +XXX,XX @@ | ||
18 | #define EXCP_SEMIHOST 16 /* semihosting call */ | ||
19 | #define EXCP_NOCP 17 /* v7M NOCP UsageFault */ | ||
20 | #define EXCP_INVSTATE 18 /* v7M INVSTATE UsageFault */ | ||
21 | +#define EXCP_STKOF 19 /* v8M STKOF UsageFault */ | ||
22 | /* NB: add new EXCP_ defines to the array in arm_log_exception() too */ | ||
23 | |||
24 | #define ARMV7M_EXCP_RESET 1 | ||
25 | @@ -XXX,XX +XXX,XX @@ FIELD(V7M_CFSR, UNDEFINSTR, 16 + 0, 1) | ||
26 | FIELD(V7M_CFSR, INVSTATE, 16 + 1, 1) | ||
27 | FIELD(V7M_CFSR, INVPC, 16 + 2, 1) | ||
28 | FIELD(V7M_CFSR, NOCP, 16 + 3, 1) | ||
29 | +FIELD(V7M_CFSR, STKOF, 16 + 4, 1) | ||
30 | FIELD(V7M_CFSR, UNALIGNED, 16 + 8, 1) | ||
31 | FIELD(V7M_CFSR, DIVBYZERO, 16 + 9, 1) | ||
32 | |||
33 | diff --git a/target/arm/helper.c b/target/arm/helper.c | ||
34 | index XXXXXXX..XXXXXXX 100644 | ||
35 | --- a/target/arm/helper.c | ||
36 | +++ b/target/arm/helper.c | ||
37 | @@ -XXX,XX +XXX,XX @@ static void arm_log_exception(int idx) | ||
38 | [EXCP_SEMIHOST] = "Semihosting call", | ||
39 | [EXCP_NOCP] = "v7M NOCP UsageFault", | ||
40 | [EXCP_INVSTATE] = "v7M INVSTATE UsageFault", | ||
41 | + [EXCP_STKOF] = "v8M STKOF UsageFault", | ||
42 | }; | ||
43 | |||
44 | if (idx >= 0 && idx < ARRAY_SIZE(excnames)) { | ||
45 | @@ -XXX,XX +XXX,XX @@ void arm_v7m_cpu_do_interrupt(CPUState *cs) | ||
46 | armv7m_nvic_set_pending(env->nvic, ARMV7M_EXCP_USAGE, env->v7m.secure); | ||
47 | env->v7m.cfsr[env->v7m.secure] |= R_V7M_CFSR_INVSTATE_MASK; | ||
48 | break; | ||
49 | + case EXCP_STKOF: | ||
50 | + armv7m_nvic_set_pending(env->nvic, ARMV7M_EXCP_USAGE, env->v7m.secure); | ||
51 | + env->v7m.cfsr[env->v7m.secure] |= R_V7M_CFSR_STKOF_MASK; | ||
52 | + break; | ||
53 | case EXCP_SWI: | ||
54 | /* The PC already points to the next instruction. */ | ||
55 | armv7m_nvic_set_pending(env->nvic, ARMV7M_EXCP_SVC, env->v7m.secure); | ||
56 | -- | ||
57 | 2.19.0 | ||
58 | |||
59 | diff view generated by jsdifflib |
Deleted patch | |||
---|---|---|---|
1 | We're going to want v7m_using_psp() in op_helper.c in the | ||
2 | next patch, so move it from helper.c to internals.h. | ||
3 | 1 | ||
4 | Signed-off-by: Peter Maydell <peter.maydell@linaro.org> | ||
5 | Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> | ||
6 | Reviewed-by: Richard Henderson <richard.henderson@linaro.org> | ||
7 | Message-id: 20181002163556.10279-4-peter.maydell@linaro.org | ||
8 | --- | ||
9 | target/arm/internals.h | 16 ++++++++++++++++ | ||
10 | target/arm/helper.c | 12 ------------ | ||
11 | 2 files changed, 16 insertions(+), 12 deletions(-) | ||
12 | |||
13 | diff --git a/target/arm/internals.h b/target/arm/internals.h | ||
14 | index XXXXXXX..XXXXXXX 100644 | ||
15 | --- a/target/arm/internals.h | ||
16 | +++ b/target/arm/internals.h | ||
17 | @@ -XXX,XX +XXX,XX @@ static inline uint32_t arm_debug_exception_fsr(CPUARMState *env) | ||
18 | */ | ||
19 | #define MEMOPIDX_SHIFT 8 | ||
20 | |||
21 | +/** | ||
22 | + * v7m_using_psp: Return true if using process stack pointer | ||
23 | + * Return true if the CPU is currently using the process stack | ||
24 | + * pointer, or false if it is using the main stack pointer. | ||
25 | + */ | ||
26 | +static inline bool v7m_using_psp(CPUARMState *env) | ||
27 | +{ | ||
28 | + /* Handler mode always uses the main stack; for thread mode | ||
29 | + * the CONTROL.SPSEL bit determines the answer. | ||
30 | + * Note that in v7M it is not possible to be in Handler mode with | ||
31 | + * CONTROL.SPSEL non-zero, but in v8M it is, so we must check both. | ||
32 | + */ | ||
33 | + return !arm_v7m_is_handler_mode(env) && | ||
34 | + env->v7m.control[env->v7m.secure] & R_V7M_CONTROL_SPSEL_MASK; | ||
35 | +} | ||
36 | + | ||
37 | #endif | ||
38 | diff --git a/target/arm/helper.c b/target/arm/helper.c | ||
39 | index XXXXXXX..XXXXXXX 100644 | ||
40 | --- a/target/arm/helper.c | ||
41 | +++ b/target/arm/helper.c | ||
42 | @@ -XXX,XX +XXX,XX @@ pend_fault: | ||
43 | return false; | ||
44 | } | ||
45 | |||
46 | -/* Return true if we're using the process stack pointer (not the MSP) */ | ||
47 | -static bool v7m_using_psp(CPUARMState *env) | ||
48 | -{ | ||
49 | - /* Handler mode always uses the main stack; for thread mode | ||
50 | - * the CONTROL.SPSEL bit determines the answer. | ||
51 | - * Note that in v7M it is not possible to be in Handler mode with | ||
52 | - * CONTROL.SPSEL non-zero, but in v8M it is, so we must check both. | ||
53 | - */ | ||
54 | - return !arm_v7m_is_handler_mode(env) && | ||
55 | - env->v7m.control[env->v7m.secure] & R_V7M_CONTROL_SPSEL_MASK; | ||
56 | -} | ||
57 | - | ||
58 | /* Write to v7M CONTROL.SPSEL bit for the specified security bank. | ||
59 | * This may change the current stack pointer between Main and Process | ||
60 | * stack pointers if it is done for the CONTROL register for the current | ||
61 | -- | ||
62 | 2.19.0 | ||
63 | |||
64 | diff view generated by jsdifflib |
Deleted patch | |||
---|---|---|---|
1 | Add some comments to the Thumb decoder indicating what bits | ||
2 | of the instruction have been decoded at various points in | ||
3 | the code. | ||
4 | 1 | ||
5 | This is not an exhaustive set of comments; we're gradually | ||
6 | adding comments as we work with particular bits of the code. | ||
7 | |||
8 | Signed-off-by: Peter Maydell <peter.maydell@linaro.org> | ||
9 | Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> | ||
10 | Reviewed-by: Richard Henderson <richard.henderson@linaro.org> | ||
11 | Message-id: 20181002163556.10279-6-peter.maydell@linaro.org | ||
12 | --- | ||
13 | target/arm/translate.c | 20 +++++++++++++++++--- | ||
14 | 1 file changed, 17 insertions(+), 3 deletions(-) | ||
15 | |||
16 | diff --git a/target/arm/translate.c b/target/arm/translate.c | ||
17 | index XXXXXXX..XXXXXXX 100644 | ||
18 | --- a/target/arm/translate.c | ||
19 | +++ b/target/arm/translate.c | ||
20 | @@ -XXX,XX +XXX,XX @@ static void disas_thumb2_insn(DisasContext *s, uint32_t insn) | ||
21 | tmp2 = load_reg(s, rm); | ||
22 | if ((insn & 0x70) != 0) | ||
23 | goto illegal_op; | ||
24 | + /* | ||
25 | + * 0b1111_1010_0xxx_xxxx_1111_xxxx_0000_xxxx: | ||
26 | + * - MOV, MOVS (register-shifted register), flagsetting | ||
27 | + */ | ||
28 | op = (insn >> 21) & 3; | ||
29 | logic_cc = (insn & (1 << 20)) != 0; | ||
30 | gen_arm_shift_reg(tmp, op, tmp2, logic_cc); | ||
31 | @@ -XXX,XX +XXX,XX @@ static void disas_thumb_insn(DisasContext *s, uint32_t insn) | ||
32 | rd = insn & 7; | ||
33 | op = (insn >> 11) & 3; | ||
34 | if (op == 3) { | ||
35 | - /* add/subtract */ | ||
36 | + /* | ||
37 | + * 0b0001_1xxx_xxxx_xxxx | ||
38 | + * - Add, subtract (three low registers) | ||
39 | + * - Add, subtract (two low registers and immediate) | ||
40 | + */ | ||
41 | rn = (insn >> 3) & 7; | ||
42 | tmp = load_reg(s, rn); | ||
43 | if (insn & (1 << 10)) { | ||
44 | @@ -XXX,XX +XXX,XX @@ static void disas_thumb_insn(DisasContext *s, uint32_t insn) | ||
45 | } | ||
46 | break; | ||
47 | case 2: case 3: | ||
48 | - /* arithmetic large immediate */ | ||
49 | + /* | ||
50 | + * 0b001x_xxxx_xxxx_xxxx | ||
51 | + * - Add, subtract, compare, move (one low register and immediate) | ||
52 | + */ | ||
53 | op = (insn >> 11) & 3; | ||
54 | rd = (insn >> 8) & 0x7; | ||
55 | if (op == 0) { /* mov */ | ||
56 | @@ -XXX,XX +XXX,XX @@ static void disas_thumb_insn(DisasContext *s, uint32_t insn) | ||
57 | break; | ||
58 | } | ||
59 | |||
60 | - /* data processing register */ | ||
61 | + /* | ||
62 | + * 0b0100_00xx_xxxx_xxxx | ||
63 | + * - Data-processing (two low registers) | ||
64 | + */ | ||
65 | rd = insn & 7; | ||
66 | rm = (insn >> 3) & 7; | ||
67 | op = (insn >> 6) & 0xf; | ||
68 | -- | ||
69 | 2.19.0 | ||
70 | |||
71 | diff view generated by jsdifflib |
Deleted patch | |||
---|---|---|---|
1 | Add checks for breaches of the v8M stack limit when the | ||
2 | stack pointer is decremented to push the exception frame | ||
3 | for exception entry. | ||
4 | 1 | ||
5 | Note that the exception-entry case is unique in that the | ||
6 | stack pointer is updated to be the limit value if the limit | ||
7 | is hit (per rule R_ZLZG). | ||
8 | |||
9 | Signed-off-by: Peter Maydell <peter.maydell@linaro.org> | ||
10 | Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> | ||
11 | Reviewed-by: Richard Henderson <richard.henderson@linaro.org> | ||
12 | Message-id: 20181002163556.10279-7-peter.maydell@linaro.org | ||
13 | --- | ||
14 | target/arm/helper.c | 54 ++++++++++++++++++++++++++++++++++++++------- | ||
15 | 1 file changed, 46 insertions(+), 8 deletions(-) | ||
16 | |||
17 | diff --git a/target/arm/helper.c b/target/arm/helper.c | ||
18 | index XXXXXXX..XXXXXXX 100644 | ||
19 | --- a/target/arm/helper.c | ||
20 | +++ b/target/arm/helper.c | ||
21 | @@ -XXX,XX +XXX,XX @@ static bool v7m_push_callee_stack(ARMCPU *cpu, uint32_t lr, bool dotailchain, | ||
22 | uint32_t frameptr; | ||
23 | ARMMMUIdx mmu_idx; | ||
24 | bool stacked_ok; | ||
25 | + uint32_t limit; | ||
26 | + bool want_psp; | ||
27 | |||
28 | if (dotailchain) { | ||
29 | bool mode = lr & R_V7M_EXCRET_MODE_MASK; | ||
30 | @@ -XXX,XX +XXX,XX @@ static bool v7m_push_callee_stack(ARMCPU *cpu, uint32_t lr, bool dotailchain, | ||
31 | mmu_idx = arm_v7m_mmu_idx_for_secstate_and_priv(env, M_REG_S, priv); | ||
32 | frame_sp_p = get_v7m_sp_ptr(env, M_REG_S, mode, | ||
33 | lr & R_V7M_EXCRET_SPSEL_MASK); | ||
34 | + want_psp = mode && (lr & R_V7M_EXCRET_SPSEL_MASK); | ||
35 | + if (want_psp) { | ||
36 | + limit = env->v7m.psplim[M_REG_S]; | ||
37 | + } else { | ||
38 | + limit = env->v7m.msplim[M_REG_S]; | ||
39 | + } | ||
40 | } else { | ||
41 | mmu_idx = core_to_arm_mmu_idx(env, cpu_mmu_index(env, false)); | ||
42 | frame_sp_p = &env->regs[13]; | ||
43 | + limit = v7m_sp_limit(env); | ||
44 | } | ||
45 | |||
46 | frameptr = *frame_sp_p - 0x28; | ||
47 | + if (frameptr < limit) { | ||
48 | + /* | ||
49 | + * Stack limit failure: set SP to the limit value, and generate | ||
50 | + * STKOF UsageFault. Stack pushes below the limit must not be | ||
51 | + * performed. It is IMPDEF whether pushes above the limit are | ||
52 | + * performed; we choose not to. | ||
53 | + */ | ||
54 | + qemu_log_mask(CPU_LOG_INT, | ||
55 | + "...STKOF during callee-saves register stacking\n"); | ||
56 | + env->v7m.cfsr[env->v7m.secure] |= R_V7M_CFSR_STKOF_MASK; | ||
57 | + armv7m_nvic_set_pending(env->nvic, ARMV7M_EXCP_USAGE, | ||
58 | + env->v7m.secure); | ||
59 | + *frame_sp_p = limit; | ||
60 | + return true; | ||
61 | + } | ||
62 | |||
63 | /* Write as much of the stack frame as we can. A write failure may | ||
64 | * cause us to pend a derived exception. | ||
65 | @@ -XXX,XX +XXX,XX @@ static bool v7m_push_callee_stack(ARMCPU *cpu, uint32_t lr, bool dotailchain, | ||
66 | v7m_stack_write(cpu, frameptr + 0x24, env->regs[11], mmu_idx, | ||
67 | ignore_faults); | ||
68 | |||
69 | - /* Update SP regardless of whether any of the stack accesses failed. | ||
70 | - * When we implement v8M stack limit checking then this attempt to | ||
71 | - * update SP might also fail and result in a derived exception. | ||
72 | - */ | ||
73 | + /* Update SP regardless of whether any of the stack accesses failed. */ | ||
74 | *frame_sp_p = frameptr; | ||
75 | |||
76 | return !stacked_ok; | ||
77 | @@ -XXX,XX +XXX,XX @@ static bool v7m_push_stack(ARMCPU *cpu) | ||
78 | |||
79 | frameptr -= 0x20; | ||
80 | |||
81 | + if (arm_feature(env, ARM_FEATURE_V8)) { | ||
82 | + uint32_t limit = v7m_sp_limit(env); | ||
83 | + | ||
84 | + if (frameptr < limit) { | ||
85 | + /* | ||
86 | + * Stack limit failure: set SP to the limit value, and generate | ||
87 | + * STKOF UsageFault. Stack pushes below the limit must not be | ||
88 | + * performed. It is IMPDEF whether pushes above the limit are | ||
89 | + * performed; we choose not to. | ||
90 | + */ | ||
91 | + qemu_log_mask(CPU_LOG_INT, | ||
92 | + "...STKOF during stacking\n"); | ||
93 | + env->v7m.cfsr[env->v7m.secure] |= R_V7M_CFSR_STKOF_MASK; | ||
94 | + armv7m_nvic_set_pending(env->nvic, ARMV7M_EXCP_USAGE, | ||
95 | + env->v7m.secure); | ||
96 | + env->regs[13] = limit; | ||
97 | + return true; | ||
98 | + } | ||
99 | + } | ||
100 | + | ||
101 | /* Write as much of the stack frame as we can. If we fail a stack | ||
102 | * write this will result in a derived exception being pended | ||
103 | * (which may be taken in preference to the one we started with | ||
104 | @@ -XXX,XX +XXX,XX @@ static bool v7m_push_stack(ARMCPU *cpu) | ||
105 | v7m_stack_write(cpu, frameptr + 24, env->regs[15], mmu_idx, false) && | ||
106 | v7m_stack_write(cpu, frameptr + 28, xpsr, mmu_idx, false); | ||
107 | |||
108 | - /* Update SP regardless of whether any of the stack accesses failed. | ||
109 | - * When we implement v8M stack limit checking then this attempt to | ||
110 | - * update SP might also fail and result in a derived exception. | ||
111 | - */ | ||
112 | + /* Update SP regardless of whether any of the stack accesses failed. */ | ||
113 | env->regs[13] = frameptr; | ||
114 | |||
115 | return !stacked_ok; | ||
116 | -- | ||
117 | 2.19.0 | ||
118 | |||
119 | diff view generated by jsdifflib |
Deleted patch | |||
---|---|---|---|
1 | Check the v8M stack limits when pushing the frame for a | ||
2 | non-secure function call via BLXNS. | ||
3 | 1 | ||
4 | In order to be able to generate the exception we need to | ||
5 | promote raise_exception() from being local to op_helper.c | ||
6 | so we can call it from helper.c. | ||
7 | |||
8 | Signed-off-by: Peter Maydell <peter.maydell@linaro.org> | ||
9 | Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> | ||
10 | Reviewed-by: Richard Henderson <richard.henderson@linaro.org> | ||
11 | Message-id: 20181002163556.10279-8-peter.maydell@linaro.org | ||
12 | --- | ||
13 | target/arm/internals.h | 9 +++++++++ | ||
14 | target/arm/helper.c | 4 ++++ | ||
15 | target/arm/op_helper.c | 4 ++-- | ||
16 | 3 files changed, 15 insertions(+), 2 deletions(-) | ||
17 | |||
18 | diff --git a/target/arm/internals.h b/target/arm/internals.h | ||
19 | index XXXXXXX..XXXXXXX 100644 | ||
20 | --- a/target/arm/internals.h | ||
21 | +++ b/target/arm/internals.h | ||
22 | @@ -XXX,XX +XXX,XX @@ FIELD(V7M_EXCRET, RES1, 7, 25) /* including the must-be-1 prefix */ | ||
23 | #define M_FAKE_FSR_NSC_EXEC 0xf /* NS executing in S&NSC memory */ | ||
24 | #define M_FAKE_FSR_SFAULT 0xe /* SecureFault INVTRAN, INVEP or AUVIOL */ | ||
25 | |||
26 | +/** | ||
27 | + * raise_exception: Raise the specified exception. | ||
28 | + * Raise a guest exception with the specified value, syndrome register | ||
29 | + * and target exception level. This should be called from helper functions, | ||
30 | + * and never returns because we will longjump back up to the CPU main loop. | ||
31 | + */ | ||
32 | +void QEMU_NORETURN raise_exception(CPUARMState *env, uint32_t excp, | ||
33 | + uint32_t syndrome, uint32_t target_el); | ||
34 | + | ||
35 | /* | ||
36 | * For AArch64, map a given EL to an index in the banked_spsr array. | ||
37 | * Note that this mapping and the AArch32 mapping defined in bank_number() | ||
38 | diff --git a/target/arm/helper.c b/target/arm/helper.c | ||
39 | index XXXXXXX..XXXXXXX 100644 | ||
40 | --- a/target/arm/helper.c | ||
41 | +++ b/target/arm/helper.c | ||
42 | @@ -XXX,XX +XXX,XX @@ void HELPER(v7m_blxns)(CPUARMState *env, uint32_t dest) | ||
43 | "BLXNS with misaligned SP is UNPREDICTABLE\n"); | ||
44 | } | ||
45 | |||
46 | + if (sp < v7m_sp_limit(env)) { | ||
47 | + raise_exception(env, EXCP_STKOF, 0, 1); | ||
48 | + } | ||
49 | + | ||
50 | saved_psr = env->v7m.exception; | ||
51 | if (env->v7m.control[M_REG_S] & R_V7M_CONTROL_SFPA_MASK) { | ||
52 | saved_psr |= XPSR_SFPA; | ||
53 | diff --git a/target/arm/op_helper.c b/target/arm/op_helper.c | ||
54 | index XXXXXXX..XXXXXXX 100644 | ||
55 | --- a/target/arm/op_helper.c | ||
56 | +++ b/target/arm/op_helper.c | ||
57 | @@ -XXX,XX +XXX,XX @@ | ||
58 | #define SIGNBIT (uint32_t)0x80000000 | ||
59 | #define SIGNBIT64 ((uint64_t)1 << 63) | ||
60 | |||
61 | -static void raise_exception(CPUARMState *env, uint32_t excp, | ||
62 | - uint32_t syndrome, uint32_t target_el) | ||
63 | +void raise_exception(CPUARMState *env, uint32_t excp, | ||
64 | + uint32_t syndrome, uint32_t target_el) | ||
65 | { | ||
66 | CPUState *cs = CPU(arm_env_get_cpu(env)); | ||
67 | |||
68 | -- | ||
69 | 2.19.0 | ||
70 | |||
71 | diff view generated by jsdifflib |
Deleted patch | |||
---|---|---|---|
1 | Add the v8M stack checks for: | ||
2 | * LDRD (immediate) | ||
3 | * STRD (immediate) | ||
4 | 1 | ||
5 | Loads and stores are more complicated than ADD/SUB/MOV, because we | ||
6 | must ensure that memory accesses below the stack limit are not | ||
7 | performed, so we can't simply do the check when we actually update | ||
8 | SP. | ||
9 | |||
10 | For these instructions, if the stack limit check triggers | ||
11 | we must not: | ||
12 | * perform any memory access below the SP limit | ||
13 | * update PC, SP or the load/store base register | ||
14 | but it is IMPDEF whether we: | ||
15 | * perform any accesses above or equal to the SP limit | ||
16 | * update destination registers for loads | ||
17 | |||
18 | For QEMU we choose to always check the limit before doing any other | ||
19 | part of the load or store, so we won't update any registers or | ||
20 | perform any memory accesses. | ||
21 | |||
22 | It is UNKNOWN whether the limit check triggers for a load or store | ||
23 | where the initial SP value is below the limit and one of the stores | ||
24 | would be below the limit, but the writeback moves SP to above the | ||
25 | limit. For QEMU we choose to trigger the check in this situation. | ||
26 | |||
27 | Note that limit checks happen only for loads and stores which update | ||
28 | SP via writeback; they do not happen for loads and stores which | ||
29 | simply use SP as a base register. | ||
30 | |||
31 | Signed-off-by: Peter Maydell <peter.maydell@linaro.org> | ||
32 | Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> | ||
33 | Reviewed-by: Richard Henderson <richard.henderson@linaro.org> | ||
34 | Message-id: 20181002163556.10279-9-peter.maydell@linaro.org | ||
35 | --- | ||
36 | target/arm/translate.c | 27 +++++++++++++++++++++++++-- | ||
37 | 1 file changed, 25 insertions(+), 2 deletions(-) | ||
38 | |||
39 | diff --git a/target/arm/translate.c b/target/arm/translate.c | ||
40 | index XXXXXXX..XXXXXXX 100644 | ||
41 | --- a/target/arm/translate.c | ||
42 | +++ b/target/arm/translate.c | ||
43 | @@ -XXX,XX +XXX,XX @@ static void disas_thumb2_insn(DisasContext *s, uint32_t insn) | ||
44 | * 0b1111_1001_x11x_xxxx_xxxx_xxxx_xxxx_xxxx | ||
45 | * - load/store dual (pre-indexed) | ||
46 | */ | ||
47 | + bool wback = extract32(insn, 21, 1); | ||
48 | + | ||
49 | if (rn == 15) { | ||
50 | if (insn & (1 << 21)) { | ||
51 | /* UNPREDICTABLE */ | ||
52 | @@ -XXX,XX +XXX,XX @@ static void disas_thumb2_insn(DisasContext *s, uint32_t insn) | ||
53 | addr = load_reg(s, rn); | ||
54 | } | ||
55 | offset = (insn & 0xff) * 4; | ||
56 | - if ((insn & (1 << 23)) == 0) | ||
57 | + if ((insn & (1 << 23)) == 0) { | ||
58 | offset = -offset; | ||
59 | + } | ||
60 | + | ||
61 | + if (s->v8m_stackcheck && rn == 13 && wback) { | ||
62 | + /* | ||
63 | + * Here 'addr' is the current SP; if offset is +ve we're | ||
64 | + * moving SP up, else down. It is UNKNOWN whether the limit | ||
65 | + * check triggers when SP starts below the limit and ends | ||
66 | + * up above it; check whichever of the current and final | ||
67 | + * SP is lower, so QEMU will trigger in that situation. | ||
68 | + */ | ||
69 | + if ((int32_t)offset < 0) { | ||
70 | + TCGv_i32 newsp = tcg_temp_new_i32(); | ||
71 | + | ||
72 | + tcg_gen_addi_i32(newsp, addr, offset); | ||
73 | + gen_helper_v8m_stackcheck(cpu_env, newsp); | ||
74 | + tcg_temp_free_i32(newsp); | ||
75 | + } else { | ||
76 | + gen_helper_v8m_stackcheck(cpu_env, addr); | ||
77 | + } | ||
78 | + } | ||
79 | + | ||
80 | if (insn & (1 << 24)) { | ||
81 | tcg_gen_addi_i32(addr, addr, offset); | ||
82 | offset = 0; | ||
83 | @@ -XXX,XX +XXX,XX @@ static void disas_thumb2_insn(DisasContext *s, uint32_t insn) | ||
84 | gen_aa32_st32(s, tmp, addr, get_mem_index(s)); | ||
85 | tcg_temp_free_i32(tmp); | ||
86 | } | ||
87 | - if (insn & (1 << 21)) { | ||
88 | + if (wback) { | ||
89 | /* Base writeback. */ | ||
90 | tcg_gen_addi_i32(addr, addr, offset - 4); | ||
91 | store_reg(s, rn, addr); | ||
92 | -- | ||
93 | 2.19.0 | ||
94 | |||
95 | diff view generated by jsdifflib |
Deleted patch | |||
---|---|---|---|
1 | Add the v8M stack checks for: | ||
2 | * LDM (T2 encoding) | ||
3 | * STM (T2 encoding) | ||
4 | 1 | ||
5 | This includes the 32-bit encodings of the instructions listed | ||
6 | in v8M ARM ARM rule R_YVWT as | ||
7 | * LDM, LDMIA, LDMFD | ||
8 | * LDMDB, LDMEA | ||
9 | * POP (multiple registers) | ||
10 | * PUSH (muliple registers) | ||
11 | * STM, STMIA, STMEA | ||
12 | * STMDB, STMFD | ||
13 | |||
14 | We perform the stack limit before doing any other part | ||
15 | of the load or store. | ||
16 | |||
17 | Signed-off-by: Peter Maydell <peter.maydell@linaro.org> | ||
18 | Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> | ||
19 | Reviewed-by: Richard Henderson <richard.henderson@linaro.org> | ||
20 | Message-id: 20181002163556.10279-10-peter.maydell@linaro.org | ||
21 | --- | ||
22 | target/arm/translate.c | 19 ++++++++++++++++++- | ||
23 | 1 file changed, 18 insertions(+), 1 deletion(-) | ||
24 | |||
25 | diff --git a/target/arm/translate.c b/target/arm/translate.c | ||
26 | index XXXXXXX..XXXXXXX 100644 | ||
27 | --- a/target/arm/translate.c | ||
28 | +++ b/target/arm/translate.c | ||
29 | @@ -XXX,XX +XXX,XX @@ static void disas_thumb2_insn(DisasContext *s, uint32_t insn) | ||
30 | } else { | ||
31 | int i, loaded_base = 0; | ||
32 | TCGv_i32 loaded_var; | ||
33 | + bool wback = extract32(insn, 21, 1); | ||
34 | /* Load/store multiple. */ | ||
35 | addr = load_reg(s, rn); | ||
36 | offset = 0; | ||
37 | @@ -XXX,XX +XXX,XX @@ static void disas_thumb2_insn(DisasContext *s, uint32_t insn) | ||
38 | if (insn & (1 << i)) | ||
39 | offset += 4; | ||
40 | } | ||
41 | + | ||
42 | if (insn & (1 << 24)) { | ||
43 | tcg_gen_addi_i32(addr, addr, -offset); | ||
44 | } | ||
45 | |||
46 | + if (s->v8m_stackcheck && rn == 13 && wback) { | ||
47 | + /* | ||
48 | + * If the writeback is incrementing SP rather than | ||
49 | + * decrementing it, and the initial SP is below the | ||
50 | + * stack limit but the final written-back SP would | ||
51 | + * be above, then then we must not perform any memory | ||
52 | + * accesses, but it is IMPDEF whether we generate | ||
53 | + * an exception. We choose to do so in this case. | ||
54 | + * At this point 'addr' is the lowest address, so | ||
55 | + * either the original SP (if incrementing) or our | ||
56 | + * final SP (if decrementing), so that's what we check. | ||
57 | + */ | ||
58 | + gen_helper_v8m_stackcheck(cpu_env, addr); | ||
59 | + } | ||
60 | + | ||
61 | loaded_var = NULL; | ||
62 | for (i = 0; i < 16; i++) { | ||
63 | if ((insn & (1 << i)) == 0) | ||
64 | @@ -XXX,XX +XXX,XX @@ static void disas_thumb2_insn(DisasContext *s, uint32_t insn) | ||
65 | if (loaded_base) { | ||
66 | store_reg(s, rn, loaded_var); | ||
67 | } | ||
68 | - if (insn & (1 << 21)) { | ||
69 | + if (wback) { | ||
70 | /* Base register writeback. */ | ||
71 | if (insn & (1 << 24)) { | ||
72 | tcg_gen_addi_i32(addr, addr, -offset); | ||
73 | -- | ||
74 | 2.19.0 | ||
75 | |||
76 | diff view generated by jsdifflib |
Deleted patch | |||
---|---|---|---|
1 | Add v8M stack checks for the instructions in the T32 | ||
2 | "load/store single" encoding class: these are the | ||
3 | "immediate pre-indexed" and "immediate, post-indexed" | ||
4 | LDR and STR instructions. | ||
5 | 1 | ||
6 | Signed-off-by: Peter Maydell <peter.maydell@linaro.org> | ||
7 | Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> | ||
8 | Reviewed-by: Richard Henderson <richard.henderson@linaro.org> | ||
9 | Message-id: 20181002163556.10279-11-peter.maydell@linaro.org | ||
10 | --- | ||
11 | target/arm/translate.c | 23 ++++++++++++++++++++++- | ||
12 | 1 file changed, 22 insertions(+), 1 deletion(-) | ||
13 | |||
14 | diff --git a/target/arm/translate.c b/target/arm/translate.c | ||
15 | index XXXXXXX..XXXXXXX 100644 | ||
16 | --- a/target/arm/translate.c | ||
17 | +++ b/target/arm/translate.c | ||
18 | @@ -XXX,XX +XXX,XX @@ static void disas_thumb2_insn(DisasContext *s, uint32_t insn) | ||
19 | imm = -imm; | ||
20 | /* Fall through. */ | ||
21 | case 0xf: /* Pre-increment. */ | ||
22 | - tcg_gen_addi_i32(addr, addr, imm); | ||
23 | writeback = 1; | ||
24 | break; | ||
25 | default: | ||
26 | @@ -XXX,XX +XXX,XX @@ static void disas_thumb2_insn(DisasContext *s, uint32_t insn) | ||
27 | |||
28 | issinfo = writeback ? ISSInvalid : rs; | ||
29 | |||
30 | + if (s->v8m_stackcheck && rn == 13 && writeback) { | ||
31 | + /* | ||
32 | + * Stackcheck. Here we know 'addr' is the current SP; | ||
33 | + * if imm is +ve we're moving SP up, else down. It is | ||
34 | + * UNKNOWN whether the limit check triggers when SP starts | ||
35 | + * below the limit and ends up above it; we chose to do so. | ||
36 | + */ | ||
37 | + if ((int32_t)imm < 0) { | ||
38 | + TCGv_i32 newsp = tcg_temp_new_i32(); | ||
39 | + | ||
40 | + tcg_gen_addi_i32(newsp, addr, imm); | ||
41 | + gen_helper_v8m_stackcheck(cpu_env, newsp); | ||
42 | + tcg_temp_free_i32(newsp); | ||
43 | + } else { | ||
44 | + gen_helper_v8m_stackcheck(cpu_env, addr); | ||
45 | + } | ||
46 | + } | ||
47 | + | ||
48 | + if (writeback && !postinc) { | ||
49 | + tcg_gen_addi_i32(addr, addr, imm); | ||
50 | + } | ||
51 | + | ||
52 | if (insn & (1 << 20)) { | ||
53 | /* Load. */ | ||
54 | tmp = tcg_temp_new_i32(); | ||
55 | -- | ||
56 | 2.19.0 | ||
57 | |||
58 | diff view generated by jsdifflib |
Deleted patch | |||
---|---|---|---|
1 | Add v8M stack checks for the 16-bit Thumb push/pop | ||
2 | encodings: STMDB, STMFD, LDM, LDMIA, LDMFD. | ||
3 | 1 | ||
4 | Signed-off-by: Peter Maydell <peter.maydell@linaro.org> | ||
5 | Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> | ||
6 | Reviewed-by: Richard Henderson <richard.henderson@linaro.org> | ||
7 | Message-id: 20181002163556.10279-12-peter.maydell@linaro.org | ||
8 | --- | ||
9 | target/arm/translate.c | 16 +++++++++++++++- | ||
10 | 1 file changed, 15 insertions(+), 1 deletion(-) | ||
11 | |||
12 | diff --git a/target/arm/translate.c b/target/arm/translate.c | ||
13 | index XXXXXXX..XXXXXXX 100644 | ||
14 | --- a/target/arm/translate.c | ||
15 | +++ b/target/arm/translate.c | ||
16 | @@ -XXX,XX +XXX,XX @@ static void disas_thumb_insn(DisasContext *s, uint32_t insn) | ||
17 | store_reg(s, rd, tmp); | ||
18 | break; | ||
19 | case 4: case 5: case 0xc: case 0xd: | ||
20 | - /* push/pop */ | ||
21 | + /* | ||
22 | + * 0b1011_x10x_xxxx_xxxx | ||
23 | + * - push/pop | ||
24 | + */ | ||
25 | addr = load_reg(s, 13); | ||
26 | if (insn & (1 << 8)) | ||
27 | offset = 4; | ||
28 | @@ -XXX,XX +XXX,XX @@ static void disas_thumb_insn(DisasContext *s, uint32_t insn) | ||
29 | if ((insn & (1 << 11)) == 0) { | ||
30 | tcg_gen_addi_i32(addr, addr, -offset); | ||
31 | } | ||
32 | + | ||
33 | + if (s->v8m_stackcheck) { | ||
34 | + /* | ||
35 | + * Here 'addr' is the lower of "old SP" and "new SP"; | ||
36 | + * if this is a pop that starts below the limit and ends | ||
37 | + * above it, it is UNKNOWN whether the limit check triggers; | ||
38 | + * we choose to trigger. | ||
39 | + */ | ||
40 | + gen_helper_v8m_stackcheck(cpu_env, addr); | ||
41 | + } | ||
42 | + | ||
43 | for (i = 0; i < 8; i++) { | ||
44 | if (insn & (1 << i)) { | ||
45 | if (insn & (1 << 11)) { | ||
46 | -- | ||
47 | 2.19.0 | ||
48 | |||
49 | diff view generated by jsdifflib |
Deleted patch | |||
---|---|---|---|
1 | Add the v8M stack checks for the VLDM/VSTM | ||
2 | (aka VPUSH/VPOP) instructions. This code is currently | ||
3 | unreachable because we haven't yet implemented M profile | ||
4 | floating point support, but since the change is simple, | ||
5 | we add it now because otherwise we're likely to forget to | ||
6 | do it later. | ||
7 | 1 | ||
8 | Signed-off-by: Peter Maydell <peter.maydell@linaro.org> | ||
9 | Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> | ||
10 | Reviewed-by: Richard Henderson <richard.henderson@linaro.org> | ||
11 | Message-id: 20181002163556.10279-13-peter.maydell@linaro.org | ||
12 | --- | ||
13 | target/arm/translate.c | 12 ++++++++++++ | ||
14 | 1 file changed, 12 insertions(+) | ||
15 | |||
16 | diff --git a/target/arm/translate.c b/target/arm/translate.c | ||
17 | index XXXXXXX..XXXXXXX 100644 | ||
18 | --- a/target/arm/translate.c | ||
19 | +++ b/target/arm/translate.c | ||
20 | @@ -XXX,XX +XXX,XX @@ static int disas_vfp_insn(DisasContext *s, uint32_t insn) | ||
21 | if (insn & (1 << 24)) /* pre-decrement */ | ||
22 | tcg_gen_addi_i32(addr, addr, -((insn & 0xff) << 2)); | ||
23 | |||
24 | + if (s->v8m_stackcheck && rn == 13 && w) { | ||
25 | + /* | ||
26 | + * Here 'addr' is the lowest address we will store to, | ||
27 | + * and is either the old SP (if post-increment) or | ||
28 | + * the new SP (if pre-decrement). For post-increment | ||
29 | + * where the old value is below the limit and the new | ||
30 | + * value is above, it is UNKNOWN whether the limit check | ||
31 | + * triggers; we choose to trigger. | ||
32 | + */ | ||
33 | + gen_helper_v8m_stackcheck(cpu_env, addr); | ||
34 | + } | ||
35 | + | ||
36 | if (dp) | ||
37 | offset = 8; | ||
38 | else | ||
39 | -- | ||
40 | 2.19.0 | ||
41 | |||
42 | diff view generated by jsdifflib |
Deleted patch | |||
---|---|---|---|
1 | Updating the NS stack pointer via MSR to SP_NS should include | ||
2 | a check whether the new SP value is below the stack limit. | ||
3 | No other kinds of update to the various stack pointer and | ||
4 | limit registers via MSR should perform a check. | ||
5 | 1 | ||
6 | Signed-off-by: Peter Maydell <peter.maydell@linaro.org> | ||
7 | Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> | ||
8 | Reviewed-by: Richard Henderson <richard.henderson@linaro.org> | ||
9 | Message-id: 20181002163556.10279-14-peter.maydell@linaro.org | ||
10 | --- | ||
11 | target/arm/helper.c | 14 +++++++++++++- | ||
12 | 1 file changed, 13 insertions(+), 1 deletion(-) | ||
13 | |||
14 | diff --git a/target/arm/helper.c b/target/arm/helper.c | ||
15 | index XXXXXXX..XXXXXXX 100644 | ||
16 | --- a/target/arm/helper.c | ||
17 | +++ b/target/arm/helper.c | ||
18 | @@ -XXX,XX +XXX,XX @@ void HELPER(v7m_msr)(CPUARMState *env, uint32_t maskreg, uint32_t val) | ||
19 | * currently in handler mode or not, using the NS CONTROL.SPSEL. | ||
20 | */ | ||
21 | bool spsel = env->v7m.control[M_REG_NS] & R_V7M_CONTROL_SPSEL_MASK; | ||
22 | + bool is_psp = !arm_v7m_is_handler_mode(env) && spsel; | ||
23 | + uint32_t limit; | ||
24 | |||
25 | if (!env->v7m.secure) { | ||
26 | return; | ||
27 | } | ||
28 | - if (!arm_v7m_is_handler_mode(env) && spsel) { | ||
29 | + | ||
30 | + limit = is_psp ? env->v7m.psplim[false] : env->v7m.msplim[false]; | ||
31 | + | ||
32 | + if (val < limit) { | ||
33 | + CPUState *cs = CPU(arm_env_get_cpu(env)); | ||
34 | + | ||
35 | + cpu_restore_state(cs, GETPC(), true); | ||
36 | + raise_exception(env, EXCP_STKOF, 0, 1); | ||
37 | + } | ||
38 | + | ||
39 | + if (is_psp) { | ||
40 | env->v7m.other_ss_psp = val; | ||
41 | } else { | ||
42 | env->v7m.other_ss_msp = val; | ||
43 | -- | ||
44 | 2.19.0 | ||
45 | |||
46 | diff view generated by jsdifflib |
Deleted patch | |||
---|---|---|---|
1 | Coverity complains (CID 1395628) that the multiply in the calculation | ||
2 | of the framebuffer base is performed as 32x32 but then used in a | ||
3 | context that takes a 64-bit hwaddr. This can't actually ever | ||
4 | overflow the 32-bit result, because of the constraints placed on | ||
5 | the s->config values in bcm2835_fb_validate_config(). But we | ||
6 | can placate Coverity anyway, by explicitly casting one of the | ||
7 | inputs to a hwaddr, so the whole expression is calculated with | ||
8 | 64-bit arithmetic. | ||
9 | 1 | ||
10 | Signed-off-by: Peter Maydell <peter.maydell@linaro.org> | ||
11 | Reviewed-by: Paolo Bonzini <pbonzini@redhat.com> | ||
12 | Message-id: 20181005133012.26490-1-peter.maydell@linaro.org | ||
13 | --- | ||
14 | hw/display/bcm2835_fb.c | 2 +- | ||
15 | 1 file changed, 1 insertion(+), 1 deletion(-) | ||
16 | |||
17 | diff --git a/hw/display/bcm2835_fb.c b/hw/display/bcm2835_fb.c | ||
18 | index XXXXXXX..XXXXXXX 100644 | ||
19 | --- a/hw/display/bcm2835_fb.c | ||
20 | +++ b/hw/display/bcm2835_fb.c | ||
21 | @@ -XXX,XX +XXX,XX @@ static void fb_update_display(void *opaque) | ||
22 | } | ||
23 | |||
24 | if (s->invalidate) { | ||
25 | - hwaddr base = s->config.base + xoff + yoff * src_width; | ||
26 | + hwaddr base = s->config.base + xoff + (hwaddr)yoff * src_width; | ||
27 | framebuffer_update_memory_section(&s->fbsection, s->dma_mr, | ||
28 | base, | ||
29 | s->config.yres, src_width); | ||
30 | -- | ||
31 | 2.19.0 | ||
32 | |||
33 | diff view generated by jsdifflib |