Splitting Matt's larger patch set:
> 4 tcg: add tcg_gen_goto_jc_{i32,i64,tl}()
>
> Preparation. A second dispatch interface alongside
> tcg_gen_lookup_and_goto_ptr(), which is unchanged. The caller passes
> the destination PC and thereby states that the CPU state is already
> the destination's, which is what an inline lookup needs and what the
> existing interface cannot promise. --enable-debug-tcg checks that
> claim at runtime. Five targets are migrated; every other target and
> every unmigrated call site is untouched.
>
> 5 accel/tcg: add CF_NO_GOTO_JC, set while a breakpoint is present
>
> Preparation. The one thing an inline jump cache probe cannot check is
> breakpoints, and it does not have to: the probe compares cflags, so a
> cflag set while cpu->breakpoints is non-empty keeps such blocks both
> from dispatching inline and from being reached by a block that does.
> Nothing reads it yet.
>
> 6 RFC: tcg: probe the TB jump cache inline instead of calling a helper
>
> 95.8% of those 8.4 billion helper_lookup_tb_ptr() calls hit the jump
> cache. Emit the probe inline (hash, four guarded loads, goto_ptr) and
> call the helper only on a miss. -34.67%, -25.94% wall
Changes for v6:
* Split target patches from infrastructure patches.
* Use tcg-op.c rewrite infrastructure.
* Add tcg_gen_goto_jc3_*, for the guest to pass all 3 values.
* Fix bugs in target/
- alpha: Missed several system-mode points at which
we cannot use goto_jc (mtpr ps, mtpr fen, halt), and one
where we cannot use goto_jc, but can use goto_jc3 (call_pal).
- loongarch: Only use goto_jc for branches. There are several
other corner cases lurking in the priviledged insns, and
they can't be as important as branches.
- mips: Droped everything. I expect that this could be made
to work using goto_jc3, but all of the tb->flags frobbing
related to delayed branches needs some thought.
- s390x: Only use jc for branches. There are several other
corner cases with SACF and SAM insns. EX emulation requires
special treatment: always use goto_jc3 so that we only look
for TBs with ex_value == 0.
* Use gen_lea in gen_jc_probe.
* Add goto_jc opcode.
- Implement for x86_64. Code sample before:
00: 4d 8b 6c 24 18 movq 0x18(%r12), %r13
05: 4c 3b eb cmpq %rbx, %r13
08: 0f 85 32 00 00 00 jne 40f
0c: 49 8b 5c 24 10 movq 0x10(%r12), %rbx
11: 48 85 db testq %rbx, %rbx
14: 0f 84 24 00 00 00 je 40f
1a: 4c 8b 63 10 movq 0x10(%rbx), %r12
20: 49 81 fc 02 08 00 01 cmpq $0x1000802, %r12
27: 0f 85 13 00 00 00 jne 40f
2d: 4c 8b 63 08 movq 8(%rbx), %r12
31: 4d 85 e4 testq %r12, %r12
34: 0f 85 06 00 00 00 jne 40f
3a: 48 8b 5b 18 movq 0x18(%rbx), %rbx
3c: ff e3 jmpq *%rbx
40:
Code sample after:
00: 49 3b 5c 24 18 cmpq 0x18(%r12), %rbx
05: 75 22 jne 29f
07: 4d 8b 64 24 10 movq 0x10(%r12), %r12
0c: 4d 85 e4 testq %r12, %r12
0f: 74 18 je 29f
11: 49 83 7c 24 08 00 cmpq $0, 8(%r12)
17: 75 10 jne 29f
19: 49 81 7c 24 10 02 08 00 01 cmpq $0x1000802, 0x10(%r12)
22: 75 05 jne 29f
24: 41 ff 64 24 18 jmpq *0x18(%r12)
29:
- Implement for aarch64. Code sample before:
00: f9400eb6 ldr x22, [x21, #0x18]
04: eb1402df cmp x22, x20
08: 54000181 b.ne 38f
0c: f9400ab4 ldr x20, [x21, #0x10]
10: b4000154 cbz x20, 38f
14: f9400a95 ldr x21, [x20, #0x10]
18: 52810056 movz w22, #0x802
1c: 72a02016 movk w22, #0x100, lsl #16
20: eb1602bf cmp x21, x22
24: 540000a1 b.ne 38f
28: f9400695 ldr x21, [x20, #8]
2c: b5000075 cbnz x21, 38f
30: f9400e94 ldr x20, [x20, #0x18]
34: d61f0280 br x20
38:
Code sample after:
00: 52810056 movz w22, #0x802
04: 72a02016 movk w22, #0x100, lsl #16
08: a94142b5 ldp x21, x16, [x21, #0x10]
0c: f10002bf cmp x21, #0
10: fa501280 ccmp x20, x16, #0, ne
14: 540000e1 b.ne 34f
18: a940c6b0 ldp x16, x17, [x21, #8]
1c: f9400eb5 ldr x21, [x21, #0x18]
20: eb1003ff cmp xzr, x16
24: fa5102c0 ccmp x22, x17, #0, eq
2c: 54000041 b.ne 34f
30: d61f02a0 br x21
34:
- Implement for s390x. Code sample before:
00: e3b0 c018 0004 lg %r11, 0x18(%r12)
06: ecbd 001c 6064 cgrjlh %r11, %r13, 0x3e
0c: e3d0 c010 0004 lg %r13, 0x10(%r12)
12: ecd8 0016 007c cgije %r13, 0, 0x3e
18: e3c0 d010 0004 lg %r12, 0x10(%r13)
1e: c0be 0100 0802 llihf %r11, 0x1000802
24: eccb 000d 6064 cgrjlh %r12, %r11, 0x3e
2a: e3c0 d008 0004 lg %r12, 8(%r13)
30: ecc6 0007 007c cgijlh %r12, 0, 0x3e
36: e3d0 d018 0004 lg %r13, 0x18(%r13)
3c: 07fd br %r13
3e:
Code sample after:
00: c0be 0100 0802 llihf %r11, 0x1000802
06: eb12 c010 0004 lmg %r1, %r2, 0x10(%r12)
0c: ec2d 0010 6064 cgrjlh %r2, %r13, 0x2c
12: ec18 000d 007c cgije %r1, 0, 0x2c
18: eb02 1008 0004 lmg %r0, %r2, 8(%r1)
1e: ec06 0007 007c cgijlh %r0, 0, 0x2c
24: ec1b 0004 6064 cgrjlh %r1, %r11, 0x2c
2a: 07f2 br %r2
2c:
Lots of prerequisites. Full tree at
https://gitlab.com/rth7680/qemu/-/tree/tcg-matt
r~
Matt Turner (5):
tcg: add tcg_gen_goto_jc{3}_{i32,i64,tl}
target/loongarch: Use tcg_gen_goto_jc_tl for JIRL
target/ppc: Use tcg_gen_goto_jc_tl
accel/tcg: add CF_NO_GOTO_JC, set while a breakpoint is present
tcg: Probe the TB jump cache inline instead of calling a helper
Richard Henderson (8):
target/alpha: Use tcg_gen_goto_jc{3}_i64
target/s390x/tcg: Remove DISAS_PC_UPDATED
target/s390x/tcg: Use tcg_gen_goto_jc3_i64 for branches
include/exec: Move tc within TranslationBlock
tcg: Add INDEX_op_goto_jc
tcg/x86_64: Implement INDEX_op_goto_jc
tcg/aarch64: Implement INDEX_op_goto_jc
tcg/s390x: Implement INDEX_op_goto_jc
docs/devel/tcg-ops.rst | 11 ++
accel/tcg/tcg-runtime.h | 4 +
include/exec/translation-block.h | 9 +-
include/tcg/tcg-op-common.h | 29 +++
include/tcg/tcg-op.h | 4 +
include/tcg/tcg-opc.h | 1 +
tcg/aarch64/tcg-target-con-set.h | 1 +
tcg/s390x/tcg-target-con-set.h | 1 +
tcg/s390x/tcg-target-con-str.h | 1 +
tcg/x86_64/tcg-target-con-set.h | 1 +
include/tcg/tcg-op-def.h.inc | 2 +
accel/tcg/cpu-exec-common.c | 16 +-
accel/tcg/cpu-exec.c | 27 +++
cpu-common.c | 25 ++-
target/alpha/translate.c | 60 +++---
target/loongarch/tcg/translate.c | 2 +-
target/ppc/translate.c | 4 +-
target/s390x/tcg/translate.c | 32 ++--
tcg/tcg-op.c | 178 +++++++++++++++++-
tcg/tcg.c | 23 +++
.../tcg/insn_trans/trans_branch.c.inc | 2 +-
tcg/aarch64/tcg-target.c.inc | 60 ++++++
tcg/loongarch64/tcg-target.c.inc | 4 +
tcg/ppc64/tcg-target.c.inc | 4 +
tcg/riscv64/tcg-target.c.inc | 4 +
tcg/s390x/tcg-target.c.inc | 42 +++++
tcg/sparc64/tcg-target.c.inc | 4 +
tcg/tci/tcg-target.c.inc | 4 +
tcg/x86_64/tcg-target.c.inc | 65 +++++++
29 files changed, 561 insertions(+), 59 deletions(-)
--
2.53.0