From nobody Fri May 9 03:02:41 2025
Delivered-To: importer@patchew.org
Authentication-Results: mx.zohomail.com;
dkim=pass;
spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as
permitted sender)
smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org;
dmarc=pass(p=none dis=none) header.from=linaro.org
ARC-Seal: i=1; a=rsa-sha256; t=1662017942; cv=none;
d=zohomail.com; s=zohoarc;
b=Q1sfbh2QYfjVQeC2OoCbnObIeaf9ZwXz15EIKi6ilqOo57+f3cNctQV83zj1ayPVl/obpFnahivuZeD24AGVr8knXfE/kwcd98ZPPdYKDwz9Jy5t/1Z4CXW2DE7QcbHWRLNrRfuMyc9drtDA8woZeeQnGj8ToGN8bR04s8Mdla8=
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com;
s=zohoarc;
t=1662017942;
h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To;
bh=qpt+vP7ZY28XWgIAuEmGW0ut/2H0xouGRGTfyPOQPGA=;
b=ApQbJ9nvmu3btlgStwZBJ4tay7bIYddHAEgy/KUz7LRuLa8ya4d4W5fY1kWVs7TyDaj+sH1Rs08/uERKtA3JNqFvyLlpwQKnn8dT57AVh9bylq8SIhy6qSHWlh90uPgxC8tVB3hmZ3a6f1ns/fPPliBQwebu5yLyj1CW99POr/I=
ARC-Authentication-Results: i=1; mx.zohomail.com;
dkim=pass;
spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as
permitted sender)
smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org;
dmarc=pass header.from=<richard.henderson@linaro.org> (p=none dis=none)
Return-Path: <qemu-devel-bounces+importer=patchew.org@nongnu.org>
Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by
mx.zohomail.com
with SMTPS id 1662017942109847.8423581922877;
Thu, 1 Sep 2022 00:39:02 -0700 (PDT)
Received: from localhost ([::1]:35368 helo=lists1p.gnu.org)
by lists.gnu.org with esmtp (Exim 4.90_1)
(envelope-from <qemu-devel-bounces+importer=patchew.org@nongnu.org>)
id 1oTemy-0005Yd-IA
for importer@patchew.org; Thu, 01 Sep 2022 03:39:00 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:52462)
by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
(Exim 4.90_1) (envelope-from <richard.henderson@linaro.org>)
id 1oTe4j-0006Q4-4k
for qemu-devel@nongnu.org; Thu, 01 Sep 2022 02:53:17 -0400
Received: from mail-wr1-x429.google.com ([2a00:1450:4864:20::429]:41762)
by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128)
(Exim 4.90_1) (envelope-from <richard.henderson@linaro.org>)
id 1oTe4g-0003Qc-T1
for qemu-devel@nongnu.org; Thu, 01 Sep 2022 02:53:16 -0400
Received: by mail-wr1-x429.google.com with SMTP id v16so18280055wrm.8
for <qemu-devel@nongnu.org>; Wed, 31 Aug 2022 23:53:14 -0700 (PDT)
Received: from stoup.. ([87.192.221.83]) by smtp.gmail.com with ESMTPSA id
a6-20020a5d4d46000000b00226dedf1ab7sm8308153wru.76.2022.08.31.23.53.10
(version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
Wed, 31 Aug 2022 23:53:11 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google;
h=content-transfer-encoding:mime-version:references:in-reply-to
:message-id:date:subject:cc:to:from:from:to:cc;
bh=qpt+vP7ZY28XWgIAuEmGW0ut/2H0xouGRGTfyPOQPGA=;
b=qsv0jAFmfbikXIFDy7Wf1ri4KDXrwD1Dc+iSxeOwNCyRY4roa+dqG5XIfbJlDupJ7X
xyO7h3LQWkxgiwhwrfuKo/gq3CPLpsedrBTLjA2rBMKRJ3JY4IIF+aOTMnUjukBuERHU
i2jmoFD5HBEQWtSMk0OLnxdHz0wP+9gI5siyqTMmUzNzSMD9dPb9AeCgyh+wiz31uU2Z
n53jlNDORd4OQ3wYbjGuf1eY3xVjmzFqv37UjGDVTvh6R/5Xp8gwcO38HT9WxYDdoZ1+
LVL76iB14OcvcTEfzl7UFZao2qisnowVxrA6vouSvGU6xDBODVEH6Qgj0JaQz9yMRBOX
TTiw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20210112;
h=content-transfer-encoding:mime-version:references:in-reply-to
:message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc;
bh=qpt+vP7ZY28XWgIAuEmGW0ut/2H0xouGRGTfyPOQPGA=;
b=I3auZoQysPu4MJ8yx01b4nxuwAqpm5U6+yxCkVu6leGRFmh2BSNH8jJ1ifzRKFTaU0
XSPRRVONMP/feKyJP3ZttQ7tXd7npkRx2SNfooCht81T1vcFx9aStecSUyDB/hDt47o/
aUyRtX3Z9D6aP2QcCJdUZfow4sluWaR9LmNX/H6621C815k8DwI54LXUeZV6RQxY0YgN
q02CR8ws9juTNPtesBFHs/Nbht5awzX5jT650fcXL9o5jciLXz8DPE1TmVTF6tgY7TQw
RO4SJ8qy1GDNTQR2lCP1oEHAHjhUWQKsB+FRNPjcdEaLeayc2zJf9KSnQOmkkay3vmu6
s4EQ==
X-Gm-Message-State: ACgBeo22pMc5tbD0OYCxSK0jgTjfrBmKZp/0K2kPuxam4jnG236vrA0u
4OyIEQEdqzqg5/HwAMCzYhnnwID6hRKWPovx
X-Google-Smtp-Source:
AA6agR489PA+QgRo36Gpk+aO59lpYT1T8X/yuwOJiZeopV2PjPnBHLUS58ynoBchu/E3nTwL8jdlEw==
X-Received: by 2002:a5d:64e9:0:b0:220:7dd7:63eb with SMTP id
g9-20020a5d64e9000000b002207dd763ebmr13504173wri.590.1662015192279;
Wed, 31 Aug 2022 23:53:12 -0700 (PDT)
From: Richard Henderson <richard.henderson@linaro.org>
To: qemu-devel@nongnu.org
Cc: Ilya Leoshkevich <iii@linux.ibm.com>
Subject: [PULL 16/20] accel/tcg: Add fast path for translator_ld*
Date: Thu, 1 Sep 2022 07:52:06 +0100
Message-Id: <20220901065210.117081-21-richard.henderson@linaro.org>
X-Mailer: git-send-email 2.34.1
In-Reply-To: <20220901065210.117081-1-richard.henderson@linaro.org>
References: <20220901065210.117081-1-richard.henderson@linaro.org>
MIME-Version: 1.0
Content-Transfer-Encoding: quoted-printable
Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17
as permitted sender) client-ip=209.51.188.17;
envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org;
helo=lists.gnu.org;
Received-SPF: pass client-ip=2a00:1450:4864:20::429;
envelope-from=richard.henderson@linaro.org; helo=mail-wr1-x429.google.com
X-Spam_score_int: -20
X-Spam_score: -2.1
X-Spam_bar: --
X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1,
DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1,
RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001,
T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no
X-Spam_action: no action
X-BeenThere: qemu-devel@nongnu.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: <qemu-devel.nongnu.org>
List-Unsubscribe: <https://lists.nongnu.org/mailman/options/qemu-devel>,
<mailto:qemu-devel-request@nongnu.org?subject=unsubscribe>
List-Archive: <https://lists.nongnu.org/archive/html/qemu-devel>
List-Post: <mailto:qemu-devel@nongnu.org>
List-Help: <mailto:qemu-devel-request@nongnu.org?subject=help>
List-Subscribe: <https://lists.nongnu.org/mailman/listinfo/qemu-devel>,
<mailto:qemu-devel-request@nongnu.org?subject=subscribe>
Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org
Sender: "Qemu-devel" <qemu-devel-bounces+importer=patchew.org@nongnu.org>
X-ZohoMail-DKIM: pass (identity @linaro.org)
X-ZM-MESSAGEID: 1662017943171100001
Content-Type: text/plain; charset="utf-8"
Cache the translation from guest to host address, so we may
use direct loads when we hit on the primary translation page.
Look up the second translation page only once, during translation.
This obviates another lookup of the second page within tb_gen_code
after translation.
Fixes a bug in that plugin_insn_append should be passed the bytes
in the original memory order, not bswapped by pieces.
Acked-by: Ilya Leoshkevich <iii@linux.ibm.com>
Tested-by: Ilya Leoshkevich <iii@linux.ibm.com>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
include/exec/translator.h | 63 +++++++++++--------
accel/tcg/translate-all.c | 23 +++----
accel/tcg/translator.c | 126 +++++++++++++++++++++++++++++---------
3 files changed, 141 insertions(+), 71 deletions(-)
diff --git a/include/exec/translator.h b/include/exec/translator.h
index 69db0f5c21..329a42fe46 100644
--- a/include/exec/translator.h
+++ b/include/exec/translator.h
@@ -81,24 +81,14 @@ typedef enum DisasJumpType {
* Architecture-agnostic disassembly context.
*/
typedef struct DisasContextBase {
- const TranslationBlock *tb;
+ TranslationBlock *tb;
target_ulong pc_first;
target_ulong pc_next;
DisasJumpType is_jmp;
int num_insns;
int max_insns;
bool singlestep_enabled;
-#ifdef CONFIG_USER_ONLY
- /*
- * Guest address of the last byte of the last protected page.
- *
- * Pages containing the translated instructions are made non-writable =
in
- * order to achieve consistency in case another thread is modifying the
- * code while translate_insn() fetches the instruction bytes piecemeal.
- * Such writer threads are blocked on mmap_lock() in page_unprotect().
- */
- target_ulong page_protect_end;
-#endif
+ void *host_addr[2];
} DisasContextBase;
=20
/**
@@ -183,24 +173,43 @@ bool translator_use_goto_tb(DisasContextBase *db, tar=
get_ulong dest);
* the relevant information at translation time.
*/
=20
-#define GEN_TRANSLATOR_LD(fullname, type, load_fn, swap_fn) \
- type fullname ## _swap(CPUArchState *env, DisasContextBase *dcbase, \
- abi_ptr pc, bool do_swap); \
- static inline type fullname(CPUArchState *env, \
- DisasContextBase *dcbase, abi_ptr pc) \
- { \
- return fullname ## _swap(env, dcbase, pc, false); \
+uint8_t translator_ldub(CPUArchState *env, DisasContextBase *db, abi_ptr p=
c);
+uint16_t translator_lduw(CPUArchState *env, DisasContextBase *db, abi_ptr =
pc);
+uint32_t translator_ldl(CPUArchState *env, DisasContextBase *db, abi_ptr p=
c);
+uint64_t translator_ldq(CPUArchState *env, DisasContextBase *db, abi_ptr p=
c);
+
+static inline uint16_t
+translator_lduw_swap(CPUArchState *env, DisasContextBase *db,
+ abi_ptr pc, bool do_swap)
+{
+ uint16_t ret =3D translator_lduw(env, db, pc);
+ if (do_swap) {
+ ret =3D bswap16(ret);
}
+ return ret;
+}
=20
-#define FOR_EACH_TRANSLATOR_LD(F) \
- F(translator_ldub, uint8_t, cpu_ldub_code, /* no swap */) \
- F(translator_lduw, uint16_t, cpu_lduw_code, bswap16) \
- F(translator_ldl, uint32_t, cpu_ldl_code, bswap32) \
- F(translator_ldq, uint64_t, cpu_ldq_code, bswap64)
+static inline uint32_t
+translator_ldl_swap(CPUArchState *env, DisasContextBase *db,
+ abi_ptr pc, bool do_swap)
+{
+ uint32_t ret =3D translator_ldl(env, db, pc);
+ if (do_swap) {
+ ret =3D bswap32(ret);
+ }
+ return ret;
+}
=20
-FOR_EACH_TRANSLATOR_LD(GEN_TRANSLATOR_LD)
-
-#undef GEN_TRANSLATOR_LD
+static inline uint64_t
+translator_ldq_swap(CPUArchState *env, DisasContextBase *db,
+ abi_ptr pc, bool do_swap)
+{
+ uint64_t ret =3D translator_ldq_swap(env, db, pc, false);
+ if (do_swap) {
+ ret =3D bswap64(ret);
+ }
+ return ret;
+}
=20
/*
* Return whether addr is on the same page as where disassembly started.
diff --git a/accel/tcg/translate-all.c b/accel/tcg/translate-all.c
index 587886aa4e..f5e8592d4a 100644
--- a/accel/tcg/translate-all.c
+++ b/accel/tcg/translate-all.c
@@ -1385,8 +1385,7 @@ TranslationBlock *tb_gen_code(CPUState *cpu,
{
CPUArchState *env =3D cpu->env_ptr;
TranslationBlock *tb, *existing_tb;
- tb_page_addr_t phys_pc, phys_page2;
- target_ulong virt_page2;
+ tb_page_addr_t phys_pc;
tcg_insn_unit *gen_code_buf;
int gen_code_size, search_size, max_insns;
#ifdef CONFIG_PROFILER
@@ -1429,6 +1428,8 @@ TranslationBlock *tb_gen_code(CPUState *cpu,
tb->flags =3D flags;
tb->cflags =3D cflags;
tb->trace_vcpu_dstate =3D *cpu->trace_dstate;
+ tb->page_addr[0] =3D phys_pc;
+ tb->page_addr[1] =3D -1;
tcg_ctx->tb_cflags =3D cflags;
tb_overflow:
=20
@@ -1622,13 +1623,11 @@ TranslationBlock *tb_gen_code(CPUState *cpu,
}
=20
/*
- * If the TB is not associated with a physical RAM page then
- * it must be a temporary one-insn TB, and we have nothing to do
- * except fill in the page_addr[] fields. Return early before
- * attempting to link to other TBs or add to the lookup table.
+ * If the TB is not associated with a physical RAM page then it must be
+ * a temporary one-insn TB, and we have nothing left to do. Return ear=
ly
+ * before attempting to link to other TBs or add to the lookup table.
*/
- if (phys_pc =3D=3D -1) {
- tb->page_addr[0] =3D tb->page_addr[1] =3D -1;
+ if (tb->page_addr[0] =3D=3D -1) {
return tb;
}
=20
@@ -1639,17 +1638,11 @@ TranslationBlock *tb_gen_code(CPUState *cpu,
*/
tcg_tb_insert(tb);
=20
- /* check next page if needed */
- virt_page2 =3D (pc + tb->size - 1) & TARGET_PAGE_MASK;
- phys_page2 =3D -1;
- if ((pc & TARGET_PAGE_MASK) !=3D virt_page2) {
- phys_page2 =3D get_page_addr_code(env, virt_page2);
- }
/*
* No explicit memory barrier is required -- tb_link_page() makes the
* TB visible in a consistent state.
*/
- existing_tb =3D tb_link_page(tb, phys_pc, phys_page2);
+ existing_tb =3D tb_link_page(tb, tb->page_addr[0], tb->page_addr[1]);
/* if the TB already exists, discard what we just translated */
if (unlikely(existing_tb !=3D tb)) {
uintptr_t orig_aligned =3D (uintptr_t)gen_code_buf;
diff --git a/accel/tcg/translator.c b/accel/tcg/translator.c
index 3eef30d93a..ca8a5f2d83 100644
--- a/accel/tcg/translator.c
+++ b/accel/tcg/translator.c
@@ -42,15 +42,6 @@ bool translator_use_goto_tb(DisasContextBase *db, target=
_ulong dest)
return ((db->pc_first ^ dest) & TARGET_PAGE_MASK) =3D=3D 0;
}
=20
-static inline void translator_page_protect(DisasContextBase *dcbase,
- target_ulong pc)
-{
-#ifdef CONFIG_USER_ONLY
- dcbase->page_protect_end =3D pc | ~TARGET_PAGE_MASK;
- page_protect(pc);
-#endif
-}
-
void translator_loop(CPUState *cpu, TranslationBlock *tb, int max_insns,
target_ulong pc, void *host_pc,
const TranslatorOps *ops, DisasContextBase *db)
@@ -66,7 +57,12 @@ void translator_loop(CPUState *cpu, TranslationBlock *tb=
, int max_insns,
db->num_insns =3D 0;
db->max_insns =3D max_insns;
db->singlestep_enabled =3D cflags & CF_SINGLE_STEP;
- translator_page_protect(db, db->pc_next);
+ db->host_addr[0] =3D host_pc;
+ db->host_addr[1] =3D NULL;
+
+#ifdef CONFIG_USER_ONLY
+ page_protect(pc);
+#endif
=20
ops->init_disas_context(db, cpu);
tcg_debug_assert(db->is_jmp =3D=3D DISAS_NEXT); /* no early exit */
@@ -151,31 +147,103 @@ void translator_loop(CPUState *cpu, TranslationBlock=
*tb, int max_insns,
#endif
}
=20
-static inline void translator_maybe_page_protect(DisasContextBase *dcbase,
- target_ulong pc, size_t l=
en)
+static void *translator_access(CPUArchState *env, DisasContextBase *db,
+ target_ulong pc, size_t len)
{
-#ifdef CONFIG_USER_ONLY
- target_ulong end =3D pc + len - 1;
+ void *host;
+ target_ulong base, end;
+ TranslationBlock *tb;
=20
- if (end > dcbase->page_protect_end) {
- translator_page_protect(dcbase, end);
+ tb =3D db->tb;
+
+ /* Use slow path if first page is MMIO. */
+ if (unlikely(tb->page_addr[0] =3D=3D -1)) {
+ return NULL;
}
+
+ end =3D pc + len - 1;
+ if (likely(is_same_page(db, end))) {
+ host =3D db->host_addr[0];
+ base =3D db->pc_first;
+ } else {
+ host =3D db->host_addr[1];
+ base =3D TARGET_PAGE_ALIGN(db->pc_first);
+ if (host =3D=3D NULL) {
+ tb->page_addr[1] =3D
+ get_page_addr_code_hostp(env, base, &db->host_addr[1]);
+#ifdef CONFIG_USER_ONLY
+ page_protect(end);
#endif
+ /* We cannot handle MMIO as second page. */
+ assert(tb->page_addr[1] !=3D -1);
+ host =3D db->host_addr[1];
+ }
+
+ /* Use slow path when crossing pages. */
+ if (is_same_page(db, pc)) {
+ return NULL;
+ }
+ }
+
+ tcg_debug_assert(pc >=3D base);
+ return host + (pc - base);
}
=20
-#define GEN_TRANSLATOR_LD(fullname, type, load_fn, swap_fn) \
- type fullname ## _swap(CPUArchState *env, DisasContextBase *dcbase, \
- abi_ptr pc, bool do_swap) \
- { \
- translator_maybe_page_protect(dcbase, pc, sizeof(type)); \
- type ret =3D load_fn(env, pc); \
- if (do_swap) { \
- ret =3D swap_fn(ret); \
- } \
- plugin_insn_append(pc, &ret, sizeof(ret)); \
- return ret; \
+uint8_t translator_ldub(CPUArchState *env, DisasContextBase *db, abi_ptr p=
c)
+{
+ uint8_t ret;
+ void *p =3D translator_access(env, db, pc, sizeof(ret));
+
+ if (p) {
+ plugin_insn_append(pc, p, sizeof(ret));
+ return ldub_p(p);
}
+ ret =3D cpu_ldub_code(env, pc);
+ plugin_insn_append(pc, &ret, sizeof(ret));
+ return ret;
+}
=20
-FOR_EACH_TRANSLATOR_LD(GEN_TRANSLATOR_LD)
+uint16_t translator_lduw(CPUArchState *env, DisasContextBase *db, abi_ptr =
pc)
+{
+ uint16_t ret, plug;
+ void *p =3D translator_access(env, db, pc, sizeof(ret));
=20
-#undef GEN_TRANSLATOR_LD
+ if (p) {
+ plugin_insn_append(pc, p, sizeof(ret));
+ return lduw_p(p);
+ }
+ ret =3D cpu_lduw_code(env, pc);
+ plug =3D tswap16(ret);
+ plugin_insn_append(pc, &plug, sizeof(ret));
+ return ret;
+}
+
+uint32_t translator_ldl(CPUArchState *env, DisasContextBase *db, abi_ptr p=
c)
+{
+ uint32_t ret, plug;
+ void *p =3D translator_access(env, db, pc, sizeof(ret));
+
+ if (p) {
+ plugin_insn_append(pc, p, sizeof(ret));
+ return ldl_p(p);
+ }
+ ret =3D cpu_ldl_code(env, pc);
+ plug =3D tswap32(ret);
+ plugin_insn_append(pc, &plug, sizeof(ret));
+ return ret;
+}
+
+uint64_t translator_ldq(CPUArchState *env, DisasContextBase *db, abi_ptr p=
c)
+{
+ uint64_t ret, plug;
+ void *p =3D translator_access(env, db, pc, sizeof(ret));
+
+ if (p) {
+ plugin_insn_append(pc, p, sizeof(ret));
+ return ldq_p(p);
+ }
+ ret =3D cpu_ldq_code(env, pc);
+ plug =3D tswap64(ret);
+ plugin_insn_append(pc, &plug, sizeof(ret));
+ return ret;
+}
--=20
2.34.1