From nobody Wed Feb 11 00:37:03 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=linaro.org ARC-Seal: i=1; a=rsa-sha256; t=1660683681; cv=none; d=zohomail.com; s=zohoarc; b=aPrhiBQKzsJHj7Is9HdDDwjOm+heKIEi7Zzzm1fCT+dZtPAWgLlalchx5cVXpD04Vatr7vyvFN5wLE0A8Y9AAov1SZdjABmzRKrJxgSoCmsm3RQDIPgCuQPiz28mROY+4KcL/+QseL/j3frDKnDzKzOhJulpP2qF797XsPQJlmo= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1660683681; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=0wEozwiMMGON8j83eoeWLlqUaXUdXZtfgTyD9icvNrg=; b=HfLqNJy677ONX+MA9Hgb7nmL/myZoyceMoLwXN/swHX35kOBFscBCj5WgTDk6dev/96eR9rIYfArkQ+sdxCnz60ra6ZrihOr69DChu3E2qtFpShcULi0B7/BhMXtwCWExG1GuVEl/pFpRGKDIMYyJJw9guKFapwU0BNt4k19IGA= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1660683681342223.96660623489822; Tue, 16 Aug 2022 14:01:21 -0700 (PDT) Received: from localhost ([::1]:41444 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1oO3gd-0004GI-ME for importer@patchew.org; Tue, 16 Aug 2022 17:01:19 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:40038) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1oO3H8-00059a-WF for qemu-devel@nongnu.org; Tue, 16 Aug 2022 16:34:59 -0400 Received: from mail-ot1-x334.google.com ([2607:f8b0:4864:20::334]:45690) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1oO3H1-0004bR-7y for qemu-devel@nongnu.org; Tue, 16 Aug 2022 16:34:58 -0400 Received: by mail-ot1-x334.google.com with SMTP id l5-20020a05683004a500b0063707ff8244so8167038otd.12 for ; Tue, 16 Aug 2022 13:34:47 -0700 (PDT) Received: from stoup.. ([2605:ef80:80f6:61fa:9bc2:5095:d612:5e22]) by smtp.gmail.com with ESMTPSA id y23-20020a056870b01700b0010ef8ccde67sm2285220oae.13.2022.08.16.13.34.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Aug 2022 13:34:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=0wEozwiMMGON8j83eoeWLlqUaXUdXZtfgTyD9icvNrg=; b=nZlrGs7j0Pyx9eyNjRq5q83zXLIhsty9SfCOK3mtJbwRzuofKAcvINcGzM2zA9bARc 8+8JhLloISHL9Uawoi685+yW1HRf5rztbcXsCgJ5PsyrWaXFrXqEYCd+62WzTfhRC9RK oa2Rd64FBSbwMfiiAHRrcY8SBs6/oVXMZ3wmiwfU8e5WRFVVLKYqAzSY9IfQJ1rNbVNc VA+9Zw+LwJQZq/KgP53ozQq+SRO1uPXwrZfQ/BKOPY06ImRbB6KFRveB7bxQeA7Ms51I nUO/g3IzhbjMrspJ1VGze0KVg57QgSnCRN6HtfVxL0/l/J6RnsAZZG53KE/fd1B6KUph l1EA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=0wEozwiMMGON8j83eoeWLlqUaXUdXZtfgTyD9icvNrg=; b=txHF/sqoibcpx/mvWJPSzvPYvnfZmsXSZsp8kVPe1//UP2skb76W82+CjOYU729Cgc Mg/S4IwfPwNxMPp/j90WgVs2VfuEFl1RyT7IMoK51UDkTh0E34oP4EGKB8tboYpoptNf Eoj4+2ovMYh2K3orgM7YlsRJHkz6QdTEwSQEF/zM8TICkjR/xhH9kpE9Ofu4IZX34rG0 aIlemC3ShpfkDpcokumxsbvwIz/LQAzc8PfsugdO4xxFAzAS22Rt2qcXWOCzF40JRQAT vGMVA7xqccElD9NUQ65fnqpe59/dJWmKMt+JNVlalquoQ5fJBrI50LvS1hhQHuRhDK1+ Y5kg== X-Gm-Message-State: ACgBeo0Mj0v7zh/4PKkwFUpF+d6E+1I001Mskp6j6L/nHQP+85rSza0m uRdJhVrAokBZfmuwYERE6ERZO5+P4G/mfQ== X-Google-Smtp-Source: AA6agR7X94qf3OtUDJGLzkj4cWsjYibg6YY4IV1xy6XLdAl9FD6j61Bdm2MS33CLMF48SwPwYFdsoQ== X-Received: by 2002:a9d:6ad7:0:b0:636:f76b:638a with SMTP id m23-20020a9d6ad7000000b00636f76b638amr8583857otq.233.1660682086486; Tue, 16 Aug 2022 13:34:46 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Cc: qemu-arm@nongnu.org, alex.bennee@linaro.org, iii@linux.ibm.com Subject: [PATCH v2 18/33] accel/tcg: Add fast path for translator_ld* Date: Tue, 16 Aug 2022 15:33:45 -0500 Message-Id: <20220816203400.161187-19-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20220816203400.161187-1-richard.henderson@linaro.org> References: <20220816203400.161187-1-richard.henderson@linaro.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2607:f8b0:4864:20::334; envelope-from=richard.henderson@linaro.org; helo=mail-ot1-x334.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @linaro.org) X-ZM-MESSAGEID: 1660683683620100001 Content-Type: text/plain; charset="utf-8" Cache the translation from guest to host address, so we may use direct loads when we hit on the primary translation page. Look up the second translation page only once, during translation. This obviates another lookup of the second page within tb_gen_code after translation. Fixes a bug in that plugin_insn_append should be passed the bytes in the original memory order, not bswapped by pieces. Signed-off-by: Richard Henderson --- include/exec/translator.h | 52 ++++++++++++------ accel/tcg/translate-all.c | 26 ++++----- accel/tcg/translator.c | 111 +++++++++++++++++++++++++++++++------- 3 files changed, 138 insertions(+), 51 deletions(-) diff --git a/include/exec/translator.h b/include/exec/translator.h index 69db0f5c21..177a001698 100644 --- a/include/exec/translator.h +++ b/include/exec/translator.h @@ -81,13 +81,14 @@ typedef enum DisasJumpType { * Architecture-agnostic disassembly context. */ typedef struct DisasContextBase { - const TranslationBlock *tb; + TranslationBlock *tb; target_ulong pc_first; target_ulong pc_next; DisasJumpType is_jmp; int num_insns; int max_insns; bool singlestep_enabled; + void *host_addr[2]; #ifdef CONFIG_USER_ONLY /* * Guest address of the last byte of the last protected page. @@ -183,24 +184,43 @@ bool translator_use_goto_tb(DisasContextBase *db, tar= get_ulong dest); * the relevant information at translation time. */ =20 -#define GEN_TRANSLATOR_LD(fullname, type, load_fn, swap_fn) \ - type fullname ## _swap(CPUArchState *env, DisasContextBase *dcbase, \ - abi_ptr pc, bool do_swap); \ - static inline type fullname(CPUArchState *env, \ - DisasContextBase *dcbase, abi_ptr pc) \ - { \ - return fullname ## _swap(env, dcbase, pc, false); \ +uint8_t translator_ldub(CPUArchState *env, DisasContextBase *db, abi_ptr p= c); +uint16_t translator_lduw(CPUArchState *env, DisasContextBase *db, abi_ptr = pc); +uint32_t translator_ldl(CPUArchState *env, DisasContextBase *db, abi_ptr p= c); +uint64_t translator_ldq(CPUArchState *env, DisasContextBase *db, abi_ptr p= c); + +static inline uint16_t +translator_lduw_swap(CPUArchState *env, DisasContextBase *db, + abi_ptr pc, bool do_swap) +{ + uint16_t ret =3D translator_lduw(env, db, pc); + if (do_swap) { + ret =3D bswap16(ret); } + return ret; +} =20 -#define FOR_EACH_TRANSLATOR_LD(F) \ - F(translator_ldub, uint8_t, cpu_ldub_code, /* no swap */) \ - F(translator_lduw, uint16_t, cpu_lduw_code, bswap16) \ - F(translator_ldl, uint32_t, cpu_ldl_code, bswap32) \ - F(translator_ldq, uint64_t, cpu_ldq_code, bswap64) +static inline uint32_t +translator_ldl_swap(CPUArchState *env, DisasContextBase *db, + abi_ptr pc, bool do_swap) +{ + uint32_t ret =3D translator_ldl(env, db, pc); + if (do_swap) { + ret =3D bswap32(ret); + } + return ret; +} =20 -FOR_EACH_TRANSLATOR_LD(GEN_TRANSLATOR_LD) - -#undef GEN_TRANSLATOR_LD +static inline uint64_t +translator_ldq_swap(CPUArchState *env, DisasContextBase *db, + abi_ptr pc, bool do_swap) +{ + uint64_t ret =3D translator_ldq_swap(env, db, pc, false); + if (do_swap) { + ret =3D bswap64(ret); + } + return ret; +} =20 /* * Return whether addr is on the same page as where disassembly started. diff --git a/accel/tcg/translate-all.c b/accel/tcg/translate-all.c index 7360ecdb38..a8f1c34c4e 100644 --- a/accel/tcg/translate-all.c +++ b/accel/tcg/translate-all.c @@ -1332,10 +1332,10 @@ TranslationBlock *tb_gen_code(CPUState *cpu, { CPUArchState *env =3D cpu->env_ptr; TranslationBlock *tb, *existing_tb; - tb_page_addr_t phys_pc, phys_page2; - target_ulong virt_page2; + tb_page_addr_t phys_pc; tcg_insn_unit *gen_code_buf; int gen_code_size, search_size, max_insns; + void *host_pc; #ifdef CONFIG_PROFILER TCGProfile *prof =3D &tcg_ctx->prof; int64_t ti; @@ -1344,7 +1344,7 @@ TranslationBlock *tb_gen_code(CPUState *cpu, assert_memory_lock(); qemu_thread_jit_write(); =20 - phys_pc =3D get_page_addr_code_hostp(env, pc, false, NULL); + phys_pc =3D get_page_addr_code_hostp(env, pc, false, &host_pc); =20 if (phys_pc =3D=3D -1) { /* Generate a one-shot TB with 1 insn in it */ @@ -1375,6 +1375,8 @@ TranslationBlock *tb_gen_code(CPUState *cpu, tb->flags =3D flags; tb->cflags =3D cflags; tb->trace_vcpu_dstate =3D *cpu->trace_dstate; + tb->page_addr[0] =3D phys_pc; + tb->page_addr[1] =3D -1; tcg_ctx->tb_cflags =3D cflags; tb_overflow: =20 @@ -1568,13 +1570,11 @@ TranslationBlock *tb_gen_code(CPUState *cpu, } =20 /* - * If the TB is not associated with a physical RAM page then - * it must be a temporary one-insn TB, and we have nothing to do - * except fill in the page_addr[] fields. Return early before - * attempting to link to other TBs or add to the lookup table. + * If the TB is not associated with a physical RAM page then it must be + * a temporary one-insn TB, and we have nothing left to do. Return ear= ly + * before attempting to link to other TBs or add to the lookup table. */ - if (phys_pc =3D=3D -1) { - tb->page_addr[0] =3D tb->page_addr[1] =3D -1; + if (tb->page_addr[0] =3D=3D -1) { return tb; } =20 @@ -1585,17 +1585,11 @@ TranslationBlock *tb_gen_code(CPUState *cpu, */ tcg_tb_insert(tb); =20 - /* check next page if needed */ - virt_page2 =3D (pc + tb->size - 1) & TARGET_PAGE_MASK; - phys_page2 =3D -1; - if ((pc & TARGET_PAGE_MASK) !=3D virt_page2) { - phys_page2 =3D get_page_addr_code(env, virt_page2); - } /* * No explicit memory barrier is required -- tb_link_page() makes the * TB visible in a consistent state. */ - existing_tb =3D tb_link_page(tb, phys_pc, phys_page2); + existing_tb =3D tb_link_page(tb, tb->page_addr[0], tb->page_addr[1]); /* if the TB already exists, discard what we just translated */ if (unlikely(existing_tb !=3D tb)) { uintptr_t orig_aligned =3D (uintptr_t)gen_code_buf; diff --git a/accel/tcg/translator.c b/accel/tcg/translator.c index 3eef30d93a..a693c17259 100644 --- a/accel/tcg/translator.c +++ b/accel/tcg/translator.c @@ -66,6 +66,8 @@ void translator_loop(CPUState *cpu, TranslationBlock *tb,= int max_insns, db->num_insns =3D 0; db->max_insns =3D max_insns; db->singlestep_enabled =3D cflags & CF_SINGLE_STEP; + db->host_addr[0] =3D host_pc; + db->host_addr[1] =3D NULL; translator_page_protect(db, db->pc_next); =20 ops->init_disas_context(db, cpu); @@ -151,31 +153,102 @@ void translator_loop(CPUState *cpu, TranslationBlock= *tb, int max_insns, #endif } =20 -static inline void translator_maybe_page_protect(DisasContextBase *dcbase, - target_ulong pc, size_t l= en) +static void *translator_access(CPUArchState *env, DisasContextBase *db, + target_ulong pc, size_t len) { + void *host; + target_ulong base; + TranslationBlock *tb; + #ifdef CONFIG_USER_ONLY target_ulong end =3D pc + len - 1; - - if (end > dcbase->page_protect_end) { - translator_page_protect(dcbase, end); + if (end > db->page_protect_end) { + translator_page_protect(db, end); } #endif -} =20 -#define GEN_TRANSLATOR_LD(fullname, type, load_fn, swap_fn) \ - type fullname ## _swap(CPUArchState *env, DisasContextBase *dcbase, \ - abi_ptr pc, bool do_swap) \ - { \ - translator_maybe_page_protect(dcbase, pc, sizeof(type)); \ - type ret =3D load_fn(env, pc); \ - if (do_swap) { \ - ret =3D swap_fn(ret); \ - } \ - plugin_insn_append(pc, &ret, sizeof(ret)); \ - return ret; \ + tb =3D db->tb; + if (unlikely(tb->page_addr[0] =3D=3D -1)) { + /* Use slow path if first page is MMIO. */ + return NULL; + } else if (likely(is_same_page(db, pc + len - 1))) { + host =3D db->host_addr[0]; + base =3D db->pc_first; + } else if (is_same_page(db, pc)) { + /* Use slow path when crossing pages. */ + return NULL; + } else { + host =3D db->host_addr[1]; + base =3D TARGET_PAGE_ALIGN(db->pc_first); + if (host =3D=3D NULL) { + tb->page_addr[1] =3D + get_page_addr_code_hostp(env, base, false, + &db->host_addr[1]); + /* We cannot handle MMIO as second page. */ + assert(tb->page_addr[1] !=3D -1); + host =3D db->host_addr[1]; + } } =20 -FOR_EACH_TRANSLATOR_LD(GEN_TRANSLATOR_LD) + tcg_debug_assert(pc >=3D base); + return host + (pc - base); +} =20 -#undef GEN_TRANSLATOR_LD +uint8_t translator_ldub(CPUArchState *env, DisasContextBase *db, abi_ptr p= c) +{ + uint8_t ret; + void *p =3D translator_access(env, db, pc, sizeof(ret)); + + if (p) { + plugin_insn_append(pc, p, sizeof(ret)); + return ldub_p(p); + } + ret =3D cpu_ldub_code(env, pc); + plugin_insn_append(pc, &ret, sizeof(ret)); + return ret; +} + +uint16_t translator_lduw(CPUArchState *env, DisasContextBase *db, abi_ptr = pc) +{ + uint16_t ret, plug; + void *p =3D translator_access(env, db, pc, sizeof(ret)); + + if (p) { + plugin_insn_append(pc, p, sizeof(ret)); + return lduw_p(p); + } + ret =3D cpu_lduw_code(env, pc); + plug =3D tswap16(ret); + plugin_insn_append(pc, &plug, sizeof(ret)); + return ret; +} + +uint32_t translator_ldl(CPUArchState *env, DisasContextBase *db, abi_ptr p= c) +{ + uint32_t ret, plug; + void *p =3D translator_access(env, db, pc, sizeof(ret)); + + if (p) { + plugin_insn_append(pc, p, sizeof(ret)); + return ldl_p(p); + } + ret =3D cpu_ldl_code(env, pc); + plug =3D tswap32(ret); + plugin_insn_append(pc, &plug, sizeof(ret)); + return ret; +} + +uint64_t translator_ldq(CPUArchState *env, DisasContextBase *db, abi_ptr p= c) +{ + uint64_t ret, plug; + void *p =3D translator_access(env, db, pc, sizeof(ret)); + + if (p) { + plugin_insn_append(pc, p, sizeof(ret)); + return ldq_p(p); + } + ret =3D cpu_ldq_code(env, pc); + plug =3D tswap64(ret); + plugin_insn_append(pc, &plug, sizeof(ret)); + return ret; +} --=20 2.34.1