From nobody Mon Sep 28 00:10:01 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=gmail.com ARC-Seal: i=1; a=rsa-sha256; t=1786993354; cv=none; d=zohomail.com; s=zohoarc; b=YNFdlbKgKGFpkk82kWdnlHX8BoPDAIBHu0281ebvq3w2dqEHS4PUsGClt7/6v3OysVK5eDo+32EQx5ROKyyiKTuvChZqIFdtKpzSbOi2tS1wobv/QWEuhaMPHrtJVOAg2SNqdtVvbb+pDLM4UdNR9BP9TA5s0kV22CSXAQutjoo= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1786993354; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=36OsLB5ufwRDhdUK7CGcFELdtjM2z4D2GMUmwLDmDN0=; b=aHOiX4cwRUAdxX5MTEADpQVu6BqIorkeBpNBfwjxzN/eygoqkMqHa9+GaytOWoMloDYpuynHdMj6gmOFqw5px2A323uwC8EKaF6IcKR65iM6GFQ1K88lVfMBb8hZDXWOw3YX454dl0wEgoJCF2WikQI1qGqrRfm9tiptwRujDNQ= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists1p.gnu.org (lists1p.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1786993354460676.0180198999365; Mon, 17 Aug 2026 12:02:34 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists1p.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ww2ak-0000AP-RG; Mon, 17 Aug 2026 15:01:50 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists1p.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ww2Zm-0006XV-4F for qemu-devel@nongnu.org; Mon, 17 Aug 2026 15:00:50 -0400 Received: from mail-yw1-x1129.google.com ([2607:f8b0:4864:20::1129]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1ww2Zj-0006k8-LX for qemu-devel@nongnu.org; Mon, 17 Aug 2026 15:00:49 -0400 Received: by mail-yw1-x1129.google.com with SMTP id 00721157ae682-81ea0b7d137so31285737b3.2 for ; Mon, 17 Aug 2026 12:00:47 -0700 (PDT) Received: from localhost ([2600:1702:7a90:6f9f:8bc4:8aec:108d:7a04]) by smtp.gmail.com with ESMTPSA id 00721157ae682-84068d1a5a1sm11090507b3.14.2026.08.17.12.00.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 17 Aug 2026 12:00:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1786993246; x=1787598046; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=36OsLB5ufwRDhdUK7CGcFELdtjM2z4D2GMUmwLDmDN0=; b=NDCCkm/qgXpStnhDMu9ftD13+vrW9Gm5q4/5zf2z4mBbe3xr6nG5zdIHY5nadMGkeK ZH1ZPn9XxJm+xMZX4MBJ3CSYFeRBLVsi0n4XdyhyE44qkoXMcockCL+7xfZfvkTqa0XS s8M7BcM9NQ1Ag3G2/HQhpHQP2WgMGUADSL04a0f0uYV5MkFddmLDPqXMPtpMEUCxbJ2J 7J+blnwwWoJGEhE/F7ehBp7mr4cdjGXaT4xOqx/nqwSsAZfgRHQVo6tgEpyWJo3ISsmx ImarAGjYyCWw/diCgkbQPzhwMfR6g2GR2bC7msXW3nHPLc9xkGYoYoseZHkkM1w85xSf qK4g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1786993246; x=1787598046; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=36OsLB5ufwRDhdUK7CGcFELdtjM2z4D2GMUmwLDmDN0=; b=stmMvcAl0k2TizVn8HlVbHAbxnPW849eRc6+7C8zO85l6e5mGGslQDg8itOgGmunLy uNphj3rfXVmVw7pxFHd53V8DtrlWBl0epIWPEicUTWGNSGxSSTp8vleRSQlzQtnCAuxy kRv9rGJ4BxXtaMoTbLiCmtO1xPSOM5LMWhm6kGEC6eeBQC1EnNNG06Xa92UTeAx31UYR uT8sOdmuWNvD8S37G7+Bbon4z9CljZFepA8LZ0W6Jsa0Fly+vwcrs3Gsc5PvHjAVhomE uaqoskJ8QWc13QFajaI2p38N7Bs5xVjha99ZpCiwqwqB26yAB07LIH5SM0F9grfPAqrB baWg== X-Gm-Message-State: AOJu0YxNRCURIQ2TwhbAAIBQonfqTo9eiLTezGKJkw6hnyyPBmIXDiTs HyRlLATzHMwqlkME2smDA5rXk3OLj5F/RbEiG8t6VTk6PqsDHaneTT7WVWasZ0ZSovc= X-Gm-Gg: AR+sD12tRd/xnPf8YM/MYDRlWBK73Rwv6ZC3Osr9mATW1oIzQFdTdZ10GcAqZRvMCyL ZHvH9xvTnVDR1ip4fjZrGpZGP0GfU4uteWjG7+jVO+I/0UykNRcOQqNIooZ2cg/j4ILrFl8hdSW 6mntgbHYxR4LkqFB8hIIgmZQz3mNihkoTdBjSuH0KQ4jS8UDLLHd2Kn1e7ymGbtSphKVa9Iu4ie OjkS94g9TGkg8LAjREaf4q4dL+Z2NUGGTL7AXyf5FplWqJpk2r32eiY5qDlvnive7pkzS7ew8Vz 2bRj0KjFLFkAFI0KKiv+Cn2/DqXe7YA77yfUO10DSixwkFNdCPOsfuTbnCdNj599/qIX3RcXcKY 2cGY/G8VonvA6Q98fgvljUtlABnjHS7hkAv/jn6z1CvL1sPdvJx2KFLc6nffy4CMV1nJJMT59pa OwTdoYslTUagvsSz985UMpiEXyU0tnTjWcrbqOtrOBl8uYU4D3TGJbBbn0l5KT X-Received: by 2002:a05:690c:4027:b0:81f:a7b5:7f0e with SMTP id 00721157ae682-83711ffe649mr79221277b3.26.1786993246004; Mon, 17 Aug 2026 12:00:46 -0700 (PDT) From: Matt Turner To: qemu-devel@nongnu.org Cc: richard.henderson@linaro.org, pbonzini@redhat.com, philmd@mailo.com, zhao1.liu@intel.com, laurent@vivier.eu, deller@gmx.de, pierrick.bouvier@oss.qualcomm.com, Matt Turner Subject: [RFC PATCH 1/8] accel/tcg: cache the result of curr_cflags() Date: Mon, 17 Aug 2026 15:00:31 -0400 Message-ID: <20260817190038.580257-2-mattst88@gmail.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260817190038.580257-1-mattst88@gmail.com> References: <20260817190038.580257-1-mattst88@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists1p.gnu.org; Received-SPF: pass client-ip=2607:f8b0:4864:20::1129; envelope-from=mattst88@gmail.com; helo=mail-yw1-x1129.google.com X-Spam_score_int: -17 X-Spam_score: -1.8 X-Spam_bar: - X-Spam_report: (-1.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @gmail.com) X-ZM-MESSAGEID: 1786993355688158500 Content-Type: text/plain; charset="utf-8" curr_cflags() is called once per TB dispatch, from helper_lookup_tb_ptr() and from the cpu_exec() loop. It recomputes the same value every time: uint32_t cflags =3D cpu->tcg_cflags; if (unlikely(cpu_single_stepping(cpu))) { ... } else if (qatomic_read(&one_insn_per_tb)) { ... } else if (qemu_loglevel_mask(CPU_LOG_TB_NOCHAIN)) { ... } That is three loads and three branches on the hottest path in the interpreter, for state that changes only when gdb enables single-step, when one-insn-per-tb is toggled, or when the log mask changes. Compute the value once into CPUState::tcg_curr_cflags and recompute it from the four places that can change an input: tcg_cflags_set(), cpu_single_step(), tcg_set_one_insn_per_tb() and qemu_set_log_internal(). curr_cflags() becomes a single load. Measured with qemu-alpha running an emulated alpha gcc 16.2.0 compiling the SQLite 3.45.1 amalgamation (255k lines, -O2) on an x86-64 host, in a build configured with --enable-lto: before: 1,647,901,588,726 instructions after: 1,563,829,403,943 instructions -5.10% That workload issues 8.4 billion dispatches, so the per-call saving is small but the aggregate is not. The emulated compiler produces byte-identical output before and after. Wall clock does not move: 133.57s to 133.13s, a 0.33% difference against a run-to-run spread of the same size. The removed work is a few predictable loads and branches that the host executes largely in parallel with the surrounding dispatch, so this patch is worth taking for the instruction count and for what it enables, not for a time saving that can be measured on its own. Note that tcg_set_one_insn_per_tb() does not tb_flush(), so the cache cannot piggyback on TB flushing and needs its own update call. Caching makes the pre-computed cflags one half of a pair that has to be kept in step, and nothing in C enforces that. The cost of getting it wrong is not a crash but silently wrong code generation: a stale cache that is missing CF_PARALLEL makes TCG emit the non-atomic form of guest atomics, and the guest then corrupts its own mutexes. linux-user's cpu_copy() is exactly such a trap. do_fork() calls begin_parallel_context() on the parent before cpu_copy(), so the child inherits CF_PARALLEL and never calls tcg_cflags_set() itself; assigning the field directly would leave every cloned thread dispatching with a cflags of zero. Close the hole from both ends. Name the field tcg_cflags_priv and add tcg_cflags_get(), so that tcg_cflags_has()/get()/set() are the only ways to reach it. C cannot really make a struct member private, but an open-coded access now fails to compile rather than quietly going stale, which is enough to force a rebased or newly written user to look at the accessors. target/alpha's CF_PCREL setup is converted along with it: it would be benign either way today, because tcg_cpu_init_cflags() refreshes the cache afterwards in system mode, but it is the same pattern and only ordering saved it. Then have curr_cflags() recompute the value and compare, under CONFIG_DEBUG_TCG. That is the check that catches a missed update on the first dispatch, rather than days later by way of corrupted guest mutexes. It cannot be unconditional, as recomputing on every dispatch is the very cost the cache exists to avoid. Verified by dropping the tcg_cflags_set() call from cpu_copy() against a debug-tcg build: the assert fires immediately, reporting the cached and recomputed values and the bits that differ. Signed-off-by: Matt Turner --- accel/tcg/cpu-exec-common.c | 48 +++++++++++++++++++++++++++++--- accel/tcg/internal-common.h | 19 ++++++++++++- accel/tcg/tcg-all.c | 1 + cpu-target.c | 3 ++ include/exec/translation-block.h | 6 ++++ include/hw/core/cpu.h | 13 +++++++-- include/system/tcg.h | 9 ++++++ linux-user/main.c | 2 +- stubs/meson.build | 1 + stubs/tcg-cflags.c | 16 +++++++++++ target/alpha/cpu.c | 2 +- util/log.c | 4 +++ 12 files changed, 114 insertions(+), 10 deletions(-) create mode 100644 stubs/tcg-cflags.c diff --git ./accel/tcg/cpu-exec-common.c ./accel/tcg/cpu-exec-common.c index 44e84344f3..c75fbd344d 100644 --- ./accel/tcg/cpu-exec-common.c +++ ./accel/tcg/cpu-exec-common.c @@ -28,17 +28,23 @@ bool tcg_allowed; =20 bool tcg_cflags_has(CPUState *cpu, uint32_t flags) { - return cpu->tcg_cflags & flags; + return cpu->tcg_cflags_priv & flags; +} + +uint32_t tcg_cflags_get(CPUState *cpu) +{ + return cpu->tcg_cflags_priv; } =20 void tcg_cflags_set(CPUState *cpu, uint32_t flags) { - cpu->tcg_cflags |=3D flags; + cpu->tcg_cflags_priv |=3D flags; + tcg_update_curr_cflags(cpu); } =20 -uint32_t curr_cflags(CPUState *cpu) +static uint32_t compute_curr_cflags(CPUState *cpu) { - uint32_t cflags =3D cpu->tcg_cflags; + uint32_t cflags =3D cpu->tcg_cflags_priv; =20 /* * Record gdb single-step. We should be exiting the TB by raising @@ -58,6 +64,40 @@ uint32_t curr_cflags(CPUState *cpu) return cflags; } =20 +void tcg_update_curr_cflags(CPUState *cpu) +{ + cpu->tcg_curr_cflags =3D compute_curr_cflags(cpu); +} + +void tcg_update_all_curr_cflags(void) +{ + CPUState *cpu; + + CPU_FOREACH(cpu) { + tcg_update_curr_cflags(cpu); + } +} + +#ifdef CONFIG_DEBUG_TCG +/* + * Catch a cached value that has gone stale because an input changed witho= ut + * a matching tcg_update_curr_cflags(). Called from curr_cflags() on the + * dispatch path, so it exists only in debug-tcg builds. + */ +void tcg_assert_curr_cflags(CPUState *cpu) +{ + uint32_t cached =3D cpu->tcg_curr_cflags; + uint32_t fresh =3D compute_curr_cflags(cpu); + + if (unlikely(cached !=3D fresh)) { + fprintf(stderr, "stale tcg_curr_cflags on CPU %d: " + "cached 0x%08x, recomputed 0x%08x (differ in 0x%08x)\n", + cpu->cpu_index, cached, fresh, cached ^ fresh); + g_assert_not_reached(); + } +} +#endif + /* exit the current TB, but without causing any exception to be raised */ void cpu_loop_exit_noexc(CPUState *cpu) { diff --git ./accel/tcg/internal-common.h ./accel/tcg/internal-common.h index 9e7be2d78d..dc713a6e1a 100644 --- ./accel/tcg/internal-common.h +++ ./accel/tcg/internal-common.h @@ -70,7 +70,24 @@ bool tcg_exec_realizefn(CPUState *cpu, Error **errp); void tcg_exec_unrealizefn(CPUState *cpu); =20 /* current cflags for hashing/comparison */ -uint32_t curr_cflags(CPUState *cpu); +/* + * Cached by tcg_update_curr_cflags(). This is on the hot TB dispatch + * path, so it must stay a single load; see commit message. A debug-tcg + * build pays for a recompute here to prove the cache is still in step, + * which turns a missed update into a loud failure rather than subtly + * wrong code generation. + */ +#ifdef CONFIG_DEBUG_TCG +void tcg_assert_curr_cflags(CPUState *cpu); +#endif + +static inline uint32_t curr_cflags(CPUState *cpu) +{ +#ifdef CONFIG_DEBUG_TCG + tcg_assert_curr_cflags(cpu); +#endif + return cpu->tcg_curr_cflags; +} =20 void tb_check_watchpoint(CPUState *cpu, uintptr_t retaddr); =20 diff --git ./accel/tcg/tcg-all.c ./accel/tcg/tcg-all.c index 7186c10cf0..8f892f580f 100644 --- ./accel/tcg/tcg-all.c +++ ./accel/tcg/tcg-all.c @@ -254,6 +254,7 @@ static void tcg_set_one_insn_per_tb(Object *obj, bool v= alue, Error **errp) s->one_insn_per_tb =3D value; /* Set the global also: this changes the behaviour */ qatomic_set(&one_insn_per_tb, value); + tcg_update_all_curr_cflags(); } =20 static void tcg_accel_class_init(ObjectClass *oc, const void *data) diff --git ./cpu-target.c ./cpu-target.c index 4783845c9b..9affbcd9c5 100644 --- ./cpu-target.c +++ ./cpu-target.c @@ -24,6 +24,7 @@ #include "exec/replay-core.h" #include "exec/log.h" #include "hw/core/cpu.h" +#include "system/tcg.h" #include "trace/trace-root.h" =20 /* enable or disable single step mode. EXCP_DEBUG is returned by the @@ -35,6 +36,8 @@ void cpu_single_step(CPUState *cpu, unsigned flags) cpu->singlestep_flags, flags); cpu->singlestep_flags =3D flags; =20 + tcg_update_curr_cflags(cpu); + #if !defined(CONFIG_USER_ONLY) const AccelOpsClass *ops =3D cpus_get_accel(); if (ops->update_guest_debug) { diff --git ./include/exec/translation-block.h ./include/exec/translation-bl= ock.h index 40cc699031..ed2ce87503 100644 --- ./include/exec/translation-block.h +++ ./include/exec/translation-block.h @@ -158,7 +158,13 @@ static inline uint32_t tb_cflags(const TranslationBloc= k *tb) return qatomic_read(&tb->cflags); } =20 +/* + * CPUState::tcg_cflags_priv is reached only through these. The setter ke= eps + * the derived CPUState::tcg_curr_cflags in step, and assigning the field + * directly would silently leave that cache stale. + */ bool tcg_cflags_has(CPUState *cpu, uint32_t flags); +uint32_t tcg_cflags_get(CPUState *cpu); void tcg_cflags_set(CPUState *cpu, uint32_t flags); =20 static inline tb_page_addr_t tb_page_addr0(const TranslationBlock *tb) diff --git ./include/hw/core/cpu.h ./include/hw/core/cpu.h index b54035fb13..172872d005 100644 --- ./include/hw/core/cpu.h +++ ./include/hw/core/cpu.h @@ -411,10 +411,16 @@ struct qemu_work_item; * to a cluster this will be UNASSIGNED_CLUSTER_INDEX; otherwise it will * be the same as the cluster-id property of the CPU object's TYPE_CPU_C= LUSTER * QOM parent. - * Under TCG this value is propagated to @tcg_cflags. + * Under TCG this value is propagated to @tcg_cflags_priv. * See TranslationBlock::TCG CF_CLUSTER_MASK. * @start_powered_off: Indicates whether the CPU starts in powered-off sta= te. - * @tcg_cflags: Pre-computed cflags for this cpu. + * @tcg_cflags_priv: Pre-computed cflags for this cpu. Private to + * tcg_cflags_has() and tcg_cflags_set(): @tcg_curr_cflags is derived fr= om + * it and is refreshed by the setter, so a direct assignment here would + * leave the two out of step. The name is deliberately awkward to make = an + * open-coded access fail to compile rather than silently go stale. + * @tcg_curr_cflags: Cached result of curr_cflags(), recomputed by + * tcg_update_curr_cflags() whenever any of its inputs change. * @nr_threads: Number of threads within this CPU core. * @thread: Host thread details, only live once @created is #true * @sem: WIN32 only semaphore used only for qtest @@ -557,7 +563,8 @@ struct CPUState { /* TODO Move common fields from CPUArchState here. */ int cpu_index; int cluster_index; - uint32_t tcg_cflags; + uint32_t tcg_cflags_priv; + uint32_t tcg_curr_cflags; uint32_t halted; int32_t exception_index; =20 diff --git ./include/system/tcg.h ./include/system/tcg.h index 7622dcea30..f41e6b3219 100644 --- ./include/system/tcg.h +++ ./include/system/tcg.h @@ -17,6 +17,15 @@ extern bool tcg_allowed; #define tcg_enabled() 0 #endif =20 +/* + * Recompute CPUState::tcg_curr_cflags. Must be called whenever any input + * to the computation changes: CPUState::tcg_cflags_priv, gdb single-step + * state, one-insn-per-tb, or the CPU_LOG_TB_NOCHAIN log flag. The first = of + * those is covered already, tcg_cflags_set() being the only way to change= it. + */ +void tcg_update_curr_cflags(CPUState *cpu); +void tcg_update_all_curr_cflags(void); + /** * qemu_tcg_mttcg_enabled: * Check whether we are running MultiThread TCG or not. diff --git ./linux-user/main.c ./linux-user/main.c index 60a695b7ca..ba04773398 100644 --- ./linux-user/main.c +++ ./linux-user/main.c @@ -242,7 +242,7 @@ CPUArchState *cpu_copy(CPUArchState *env) /* Reset non arch specific state */ cpu_reset(new_cpu); =20 - new_cpu->tcg_cflags =3D cpu->tcg_cflags; + tcg_cflags_set(new_cpu, tcg_cflags_get(cpu)); memcpy(new_env, env, sizeof(CPUArchState)); #if defined(TARGET_I386) || defined(TARGET_X86_64) new_env->gdt.base =3D target_mmap(0, sizeof(uint64_t) * TARGET_GDT_ENT= RIES, diff --git ./stubs/meson.build ./stubs/meson.build index 3b2f2680b1..0025e79226 100644 --- ./stubs/meson.build +++ ./stubs/meson.build @@ -3,6 +3,7 @@ # below, so that it is clear who needs the stubbed functionality. =20 stub_ss.add(files('cpu-get-clock.c')) +stub_ss.add(files('tcg-cflags.c')) stub_ss.add(files('fdset.c')) stub_ss.add(files('iothread-lock.c')) stub_ss.add(files('is-daemonized.c')) diff --git ./stubs/tcg-cflags.c ./stubs/tcg-cflags.c new file mode 100644 index 0000000000..bd74fabf0e --- /dev/null +++ ./stubs/tcg-cflags.c @@ -0,0 +1,16 @@ +/* + * Stubs for the cached cflags update hooks, for binaries that link + * util/log.c or cpu-target.c without linking TCG. + * + * SPDX-License-Identifier: GPL-2.0-or-later + */ +#include "qemu/osdep.h" +#include "system/tcg.h" + +void tcg_update_curr_cflags(CPUState *cpu) +{ +} + +void tcg_update_all_curr_cflags(void) +{ +} diff --git ./target/alpha/cpu.c ./target/alpha/cpu.c index 0c35067b20..fcc676c7fb 100644 --- ./target/alpha/cpu.c +++ ./target/alpha/cpu.c @@ -114,7 +114,7 @@ static void alpha_cpu_realizefn(DeviceState *dev, Error= **errp) =20 #ifndef CONFIG_USER_ONLY /* Use pc-relative instructions in system-mode */ - cs->tcg_cflags |=3D CF_PCREL; + tcg_cflags_set(cs, CF_PCREL); #endif =20 cpu_exec_realizefn(cs, &local_err); diff --git ./util/log.c ./util/log.c index 7cffbc1bf8..62c7f09609 100644 --- ./util/log.c +++ ./util/log.c @@ -27,6 +27,7 @@ #include "qemu/thread.h" #include "qemu/lockable.h" #include "qemu/rcu.h" +#include "system/tcg.h" #ifdef CONFIG_LINUX #include #endif @@ -301,6 +302,9 @@ static bool qemu_set_log_internal(const char *filename,= bool changed_name, #endif qemu_loglevel =3D log_flags; =20 + /* CPU_LOG_TB_NOCHAIN feeds into the per-CPU cached cflags. */ + tcg_update_all_curr_cflags(); + daemonized =3D is_daemonized(); need_to_open_file =3D false; if (!daemonized) { --=20 2.54.0 From nobody Mon Sep 28 00:10:01 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=gmail.com ARC-Seal: i=1; a=rsa-sha256; t=1786993410; cv=none; d=zohomail.com; s=zohoarc; b=BXca+QNIVRLv3majwIRODAy8uxBXKV3g07w85iwLd3gKGgLHkdwJ+hsxF0cxG6q6C7fi0ILXRG0nXVHhdshFlIJgTBCjQD6eE7scthH6TYMG7WOaqbM/pvJcoIWDBW66BmSo08NpB6x9Hg+J7zGggjLrWFFfgSNN/tih63Jj2Oc= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1786993410; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=WM409keD7J89yEAgYdhBWa8aMMqXlnNUvLTfJiZti+8=; b=KLI9Jj0m9QstshxpCMoLY9Ey5oer9Vl0IH5u5Qieo4yduyBgaF4K36GBVQ3hkrJOSzwn3yKzMbKAWciKqMG1K0m1m/dSnfEQLYTj1jP4mrRB9Ksll/5I2AKDIqeIpoUJDZn5qduVIybVwLzBejGPyHhed+6R1OpiUCkdQzmgv1k= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists1p.gnu.org (lists1p.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1786993410863643.7665519778162; Mon, 17 Aug 2026 12:03:30 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists1p.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ww2as-0001I2-Si; Mon, 17 Aug 2026 15:01:58 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists1p.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ww2Zo-0006bz-VD for qemu-devel@nongnu.org; Mon, 17 Aug 2026 15:00:58 -0400 Received: from mail-yx1-xb136.google.com ([2607:f8b0:4864:20::b136]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1ww2Zm-0006kR-HL for qemu-devel@nongnu.org; Mon, 17 Aug 2026 15:00:51 -0400 Received: by mail-yx1-xb136.google.com with SMTP id 956f58d0204a3-66b35d2c4edso5393532d50.2 for ; Mon, 17 Aug 2026 12:00:50 -0700 (PDT) Received: from localhost ([2600:1702:7a90:6f9f:8bc4:8aec:108d:7a04]) by smtp.gmail.com with ESMTPSA id 956f58d0204a3-66cb41e5d05sm1230965d50.7.2026.08.17.12.00.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 17 Aug 2026 12:00:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1786993249; x=1787598049; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=WM409keD7J89yEAgYdhBWa8aMMqXlnNUvLTfJiZti+8=; b=MX4bvP1zTni9Frr4PI+8nAPIwbOsU9E2AEb8dQMNFwjDf3qw1tiMVGNoNnVxoS1bLl werseKDFYDmX8A6uSzDD4bH81O0Hl2bB8OMrkr11Qn5hK+FkhGYgbGrNEN7vEYWjJgMg bXTyu6AVK6a8F+Lhtzl3/0jQNy+EprYAOfqhyZIVoGQ2z4n4iWbVJcHPvZmjPNvB2rLj zLliS3MlVkFls1Pp48Fb0W3C7tp22S8wAIoFbf5RqD18J/Lt0wDAiL8Jm+I3AeYPBwx5 mxBXb4OyQ8jTWqyY0sRxZDlI70shg1fO3cvJ6EZPUbG0YJgAkR27+jMigza2ZCklNebw OdrA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1786993249; x=1787598049; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=WM409keD7J89yEAgYdhBWa8aMMqXlnNUvLTfJiZti+8=; b=OsrxnB6UmSy5GCgdfkk0m3iGdLMETXTY7LP1hLuxlZgemfnNqLcz/OKpmCr33t6cHW qFWTQ2M4eo+nGWrfw1BbLQFD56uDGIZ4W2Fievjsb2fCvwIXq8hVPYQsmB8ICq9gjKTF ajq/GSOkNETadrI9JmcMdAtkF5gLsguSETpHF2x5g0Zl+0FE0IBJZWqLhOv4uubUr3JH TWmHHCAstmnp7TgCnpNy0CiT0bzwkiNmQ3k/AWJYc0rjRwuefmMMZT3f0DHjOwHaHZHN AcgIeNR+DXGyA/FgB0HQCNMq8hg0ZYW/swhvMedCkUTUS6MH2xGg7VLbZSp+4sFOoTIu 09yA== X-Gm-Message-State: AOJu0Yxt8Ck2yXIdtKO/zjKbYc187R9AA/6CXP9ejq8LfyBIh6wUkarq 6DpZT+5abPANBhQYAhdES6lY7LwNOzEXp+6pUP2dIAQkotoQVW0wX3j+RW7zHaO9kKs= X-Gm-Gg: AR+sD13R3dC1jG1jVqWfuOWwHVMWx8kd3Yb4A9eLs4ti23XGwSuRRsHtlB9MOiaGnNL oZ3AAxiKT4H4VoXeeS/QbzcxNbpMlJinP883DZUBA/FtUOmxiFjhv3Hpc9QSMm4zaDPrOR9klQ/ NgUadptld78udD72+4nXioJpmEVLgndRWDWS+4t5uUeQTReWy6LSs5ZstWnAy7aRuqPQaT9e4vv Br4gPu5J4FrMhtsJcKEeypMkHWVsXtVMlzxxppIq3Om5TkIgRLxQ9fZu71JaOcai2HSKvo4rBdS 0wuY7aLmN8AjLgPu3UeR0EibkE0aan+67RolgJiLed9qmHeFc8G6m6yxwLPS7Ll885nai3JRIyc wMGOTqq2G/Gc9r/NwxN0UlgGMJDv6GQcL61D3xyr9RcxRX1dYMkxZl9BSZH6HvBILIl83FzCf8S RwjNf4UJ5I62VjFE728aSMdjgJ/dMzPnVFKNLPw72F7pu4FH4OgwMmOBiSC0xljlkq7ULAhkw= X-Received: by 2002:a05:690e:4145:b0:669:3fb5:d084 with SMTP id 956f58d0204a3-66c72be8f18mr9639481d50.24.1786993248797; Mon, 17 Aug 2026 12:00:48 -0700 (PDT) From: Matt Turner To: qemu-devel@nongnu.org Cc: richard.henderson@linaro.org, pbonzini@redhat.com, philmd@mailo.com, zhao1.liu@intel.com, laurent@vivier.eu, deller@gmx.de, pierrick.bouvier@oss.qualcomm.com, Matt Turner Subject: [RFC PATCH 2/8] accel/tcg: enlarge the TB jump cache to 64K entries Date: Mon, 17 Aug 2026 15:00:32 -0400 Message-ID: <20260817190038.580257-3-mattst88@gmail.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260817190038.580257-1-mattst88@gmail.com> References: <20260817190038.580257-1-mattst88@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists1p.gnu.org; Received-SPF: pass client-ip=2607:f8b0:4864:20::b136; envelope-from=mattst88@gmail.com; helo=mail-yx1-xb136.google.com X-Spam_score_int: -17 X-Spam_score: -1.8 X-Spam_bar: - X-Spam_report: (-1.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @gmail.com) X-ZM-MESSAGEID: 1786993412359158500 Content-Type: text/plain; charset="utf-8" The per-CPU TB jump cache has held 4096 entries since it was introduced. That is too small for guests running large programs: an emulated compiler misses often enough that the fallback qht lookup shows up prominently in a profile. Measured with qemu-alpha running an emulated alpha gcc 16.2.0 compiling the SQLite 3.45.1 amalgamation (255k lines, -O2) on an x86-64 host. The compile performs 34.2 billion TB executions, of which 8.4 billion take the indirect dispatch path. Sizing curve, on top of the preceding patch, instructions retired and wall clock: 12 bits ( 64 KiB): 1,563,829,403,943 133.13s 14 bits ( 256 KiB): 1,493,865,985,972 -4.47% 124.89s -6.19% 16 bits ( 1 MiB): 1,469,729,281,442 -6.02% 120.97s -9.13% 18 bits ( 4 MiB): 1,462,262,363,257 -6.49% 120.16s -9.74% 16 bits is the knee. 18 buys another 0.47% of instructions for four times the memory, and since instructions retired does not account for the data cache pressure of a 4 MiB table, that 0.47% is probably not real: the wall clock difference between 16 and 18 bits is 0.67%, against a run-to-run spread of the same order. In a perf profile the mechanism is visible directly: tb_htable_lookup(), which is where qht_lookup_custom() lands once it is inlined in an LTO build, falls from 5.73% of samples to 1.52%. The cost is memory: the cache grows from 64 KiB to 1 MiB per vCPU. That is easy to justify for a single-vCPU linux-user process and less obvious for system emulation with many vCPUs, so this may want to be sized by target or made tunable rather than raised unconditionally. Signed-off-by: Matt Turner --- accel/tcg/tb-jmp-cache.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git ./accel/tcg/tb-jmp-cache.h ./accel/tcg/tb-jmp-cache.h index c3a505e394..268dacd7ba 100644 --- ./accel/tcg/tb-jmp-cache.h +++ ./accel/tcg/tb-jmp-cache.h @@ -12,7 +12,7 @@ #include "qemu/rcu.h" #include "exec/cpu-common.h" =20 -#define TB_JMP_CACHE_BITS 12 +#define TB_JMP_CACHE_BITS 16 #define TB_JMP_CACHE_SIZE (1 << TB_JMP_CACHE_BITS) =20 /* --=20 2.54.0 From nobody Mon Sep 28 00:10:01 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=gmail.com ARC-Seal: i=1; a=rsa-sha256; t=1786993391; cv=none; d=zohomail.com; s=zohoarc; b=Ap1w01/A744B9c7RkXjbujZTyRjKu+sK7B0yZHH6FfBVmkdrPZMJFlPOYKxwIDROJbfUhTB1Fezsu1/8F12Kirg4YIW6/yPDyoVhTlZbsaHimpS+PSaCc2vSxE5FaOU7qY2rVf5WxCs2eEbtWD4V/6cd+WdPyfLer+CE+xNvHP4= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1786993391; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=KpvUL1chzZkJ6/YikyruA6kEN75GfHYu2PKgo1YZbyA=; b=CU3cAqGtkN6LsSd7t9xEcy+V2JErv5Gf80dxPsyvz+VCtXARKCN/pylKCd7h3szYUAnhXqLRJ0NSbV+Em7AXobobRZIdL6qLEJ9rhkuBiJ571MAZTVTvlf4oo3lCFq6tyR0Z7SZh9VJuFI4CH8ua7dM6YFY4oEirPT0ePckNaWk= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists1p.gnu.org (lists1p.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1786993391758785.4110551242686; Mon, 17 Aug 2026 12:03:11 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists1p.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ww2b1-0002QZ-TZ; Mon, 17 Aug 2026 15:02:08 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists1p.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ww2Zs-0006h8-VB for qemu-devel@nongnu.org; Mon, 17 Aug 2026 15:01:11 -0400 Received: from mail-yw1-x112e.google.com ([2607:f8b0:4864:20::112e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1ww2Zp-0006kh-Te for qemu-devel@nongnu.org; Mon, 17 Aug 2026 15:00:56 -0400 Received: by mail-yw1-x112e.google.com with SMTP id 00721157ae682-81ea0b7d137so31286837b3.2 for ; Mon, 17 Aug 2026 12:00:52 -0700 (PDT) Received: from localhost ([2600:1702:7a90:6f9f:8bc4:8aec:108d:7a04]) by smtp.gmail.com with ESMTPSA id 00721157ae682-840f3264142sm8242527b3.38.2026.08.17.12.00.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 17 Aug 2026 12:00:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1786993252; x=1787598052; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=KpvUL1chzZkJ6/YikyruA6kEN75GfHYu2PKgo1YZbyA=; b=PEetXr7lP03kZgsqxOYxiHWmce38/5yCva8IDHp7PuL4JFjLs4+6ZJVHzTb7zZg4ww 2obSoOXoqDw6JsL1Oa4WgbC0H/huCPlIlULz8TzAXsGYQ4yapE1NAEQSsFk4RFMVyafd 2zxxq5+/dvaYD+WjYF/y8n8ck/0xjEeubZTKKhdTZo7M8FahKrGZrG0mLBFGaNBYcTOm LxV+ezfnXfC93go1WHZQtt5XXO5cAJFl88axet/AOiGo74te5INfR2za2IkUgR5gW+J6 dsF4/A7Bp8onbtFkRst8N2ZmxdRfkzJfgyp9gup1zzagXmT/HYEZRsduYPclYf9CisRL rslQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1786993252; x=1787598052; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=KpvUL1chzZkJ6/YikyruA6kEN75GfHYu2PKgo1YZbyA=; b=DpGbxV90o9mHLfzt4gndqxIXFryBOVYsv/1NK+aieBaY0Spv9FvSFkZQ3TJKkio+q8 FGuknRbGfGK1L3ntVjVAIORSzZTpEli8mijtBX2L69R8dTMFig6p+Sp9hiSBNHbb86Sy PER+N35RA1/aHbaTxtbT1PyIPwJN9rFsrQ1I86mv+FIDuVcIkD2XbVuCg8+UvtUo+l6d UO64AV/txQ9A7m558hBID0FFwCM2AX3s9EJStN0K38W13fFArdaTRQmcN+GaeNTRAy5Q zgP274ruIUvz4eXJBX4RHmtYmNbeyhSn0smmO+X0KtQwe8WCzyKKb1wcPIRuTiOsMtN/ G3Ew== X-Gm-Message-State: AOJu0YxG7ZqpTZTN5fHrfGrghOZZyzH2PbeyqljYWjbPt6EIKT/wongw XnP+CjzTL2wt94x23PSZqMj2NgeUQi4145XesdbOiR4Xt4dRaPS+xVpJhE2VzMNTwus= X-Gm-Gg: AR+sD11G7PVAQ61+PSbzfPpnhaJ22tXLQfmh02g3589ZWlwmKMSuLhFIf3uyy0IR5vz +bFY2Brpofk6v+L0uYJoxII3Qu/BpmFkIufO4SdNr3ZOv7fq4PnbP0EFHi1B5TtV6j17sGdSUd3 92NcrQuuDO+LNpbBFvD3/KhLkjkgyIR6Gev6RDIVmDTaFLvegW4hEw/nSe0ZcfXPp4klzZ6F/JX 4zw+FILT2mJzZVOobbpVsRAzWrKdPHeuX9tI454fogN4Gjqx9bfzjdHQXMkIIsbk4XbRiJX8Vck Wg5ehB0FmmNq5kvnk6zSWaolMbz0aSWulTenXIhRGDVnrrJRzJbgWdjuEn8jeaJkPEgbuelxJfF jajhrGJ3JHhVuQlncTYb4aAWU/ObdCZwByw3+pjWlctcF672JwRntnnKAAKSDuoEAyJCTYTkPGf rrFSghQ06YsB942DiRIWXekWBL1+LYpImmcTQeG1klXyh2psJbBq+Y/fmTAeGw X-Received: by 2002:a05:690c:c511:b0:81e:6cdf:b2f5 with SMTP id 00721157ae682-837077121e3mr103551117b3.0.1786993251718; Mon, 17 Aug 2026 12:00:51 -0700 (PDT) From: Matt Turner To: qemu-devel@nongnu.org Cc: richard.henderson@linaro.org, pbonzini@redhat.com, philmd@mailo.com, zhao1.liu@intel.com, laurent@vivier.eu, deller@gmx.de, pierrick.bouvier@oss.qualcomm.com, Matt Turner Subject: [RFC PATCH 3/8] accel/tcg: skip the can_do_io stores in user-only builds Date: Mon, 17 Aug 2026 15:00:33 -0400 Message-ID: <20260817190038.580257-4-mattst88@gmail.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260817190038.580257-1-mattst88@gmail.com> References: <20260817190038.580257-1-mattst88@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists1p.gnu.org; Received-SPF: pass client-ip=2607:f8b0:4864:20::112e; envelope-from=mattst88@gmail.com; helo=mail-yw1-x112e.google.com X-Spam_score_int: -17 X-Spam_score: -1.8 X-Spam_bar: - X-Spam_report: (-1.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @gmail.com) X-ZM-MESSAGEID: 1786993393780158501 Content-Type: text/plain; charset="utf-8" Every translation block stores to cpu->neg.can_do_io twice: false before the first instruction, true before the last one. Nothing reads it in a user-only build. There is no memory-mapped I/O in linux-user, and every reader is in system_ss: cputlb.c, watchpoint.c, icount-common.c and tcg-accel-ops-icount.c. Two stores per TB is not much on its own, but TBs are short. An emulated alpha gcc 16.2.0 compiling the SQLite 3.45.1 amalgamation (255k lines, -O2) executes 34.2 billion TBs at 6.04 guest instructions each, so this is 68 billion stores for nothing. Measured on an x86-64 host, LTO build, on top of the preceding two patches: before: 1,469,729,281,442 instructions after: 1,402,816,253,499 instructions -4.55% before: 120.97s wall clock after: 115.75s wall clock -4.32% The emulated compiler produces byte-identical output. Signed-off-by: Matt Turner --- accel/tcg/translator.c | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git ./accel/tcg/translator.c ./accel/tcg/translator.c index cd7d079fe0..29e609b2ec 100644 --- ./accel/tcg/translator.c +++ ./accel/tcg/translator.c @@ -21,12 +21,14 @@ #include "disas/disas.h" #include "tb-internal.h" =20 +#ifndef CONFIG_USER_ONLY static void set_can_do_io(DisasContextBase *db, bool val) { QEMU_BUILD_BUG_ON(sizeof_field(CPUState, neg.can_do_io) !=3D 1); tcg_gen_st8_i32(tcg_constant_i32(val), tcg_env, offsetof(CPUState, neg.can_do_io) - sizeof(CPUState)); } +#endif =20 bool translator_io_start(DisasContextBase *db) { @@ -210,17 +212,25 @@ void translator_loop(CPUState *cpu, TranslationBlock = *tb, int *max_insns, /* * Manage can_do_io for the translation block: set to false before * the first insn and set to true before the last insn. + * + * Nothing reads can_do_io in user-only builds. There is no MMIO + * there, and every reader (cputlb.c, watchpoint.c, icount) is in + * system_ss, so skip the two stores per TB entirely. */ if (db->num_insns =3D=3D 1) { tcg_debug_assert(first_insn_start =3D=3D db->insn_start); } else { tcg_debug_assert(first_insn_start !=3D db->insn_start); +#ifndef CONFIG_USER_ONLY tcg_ctx->emit_before_op =3D first_insn_start; set_can_do_io(db, false); +#endif } +#ifndef CONFIG_USER_ONLY tcg_ctx->emit_before_op =3D db->insn_start; set_can_do_io(db, true); tcg_ctx->emit_before_op =3D NULL; +#endif =20 /* May be used by disas_log or plugin callbacks. */ tb->size =3D db->pc_next - db->pc_first; --=20 2.54.0 From nobody Mon Sep 28 00:10:01 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=gmail.com ARC-Seal: i=1; a=rsa-sha256; t=1786993409; cv=none; d=zohomail.com; s=zohoarc; b=epsMPmD3AzQxfstp55rm1+Pn3JuTnIb+vwQiuyrU1yyzXT2SR4U7iZSwD/hm/Y4zpCjdji9BhxgRVB2F5kWYR8q2oqc2U/jtXq/h7pUo01G/ynll/wW3TANUqxfGyM4rGCcQSw5jBL3m/wlU92Bm15IXPxYZwDb7cz5fyGJaIOA= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1786993409; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=aftNfT0+u3BFLzEUlvezRaDsOyAyZSmnNqYupCof0dM=; b=j3vZNUpLf7r3Xi8qIYtqXJGoxJIrY46L0nlrDJGVG1lolZ/ZwcV7wl2SAtJXOnJlpVV7HSaO9Iuj6uFagn0YUICLmwzItt84ZlFAUCZ3kpekxZecx+kxZYyLDLaefr0B7kokrR6am5wNkYlFus2hk4ociJwlzDvuTqOuMhtymU0= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists1p.gnu.org (lists1p.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1786993409082528.4246468836194; Mon, 17 Aug 2026 12:03:29 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists1p.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ww2am-0000IQ-9H; Mon, 17 Aug 2026 15:01:52 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists1p.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ww2Zu-0006hG-OI for qemu-devel@nongnu.org; Mon, 17 Aug 2026 15:01:10 -0400 Received: from mail-yx1-xb12a.google.com ([2607:f8b0:4864:20::b12a]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1ww2Zs-0006kw-Mf for qemu-devel@nongnu.org; Mon, 17 Aug 2026 15:00:58 -0400 Received: by mail-yx1-xb12a.google.com with SMTP id 956f58d0204a3-66c7e3a2332so2066434d50.2 for ; Mon, 17 Aug 2026 12:00:55 -0700 (PDT) Received: from localhost ([2600:1702:7a90:6f9f:8bc4:8aec:108d:7a04]) by smtp.gmail.com with ESMTPSA id 956f58d0204a3-66cb47a6f08sm1259656d50.17.2026.08.17.12.00.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 17 Aug 2026 12:00:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1786993254; x=1787598054; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=aftNfT0+u3BFLzEUlvezRaDsOyAyZSmnNqYupCof0dM=; b=BGaClJjyZYtK9GCt8P5LxmJGRalhUNFOYh4HgXb+I5d0jGaVVYTE/9Eab3ynlxTEK2 i7UXpiyuxGHVrVfaMdLL2wS+D07Ll5qzcdyVWtV+RwnzMYL5JcwLOzCD6ZpiWl3ydPxH 2Yc5+pcIYQNv7ydfISjKS99s6vrX5g3P+fd9O6xZ65itLJ8hczbzg/juXCVFX88YuoZV WQDu9AOosQEHDS/My9X7I3M9Bn7X6RMXyPAq+Nhg6FFnrciZHquu7x9ZhLD1j9FNMBXr ur2ahnMFNsv23cOySCN2Xjmfpsu8vomgno/sISoLc6OApYfA5XmNSEEWChvD0rufdZ+9 E6/A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1786993254; x=1787598054; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=aftNfT0+u3BFLzEUlvezRaDsOyAyZSmnNqYupCof0dM=; b=tTMDMJZPxLeVcI+JA9kKFO/elFreNf5/jffJdTMynSjJuqgTJgKAr62JLqyBHMiTR4 pwf591TphHw2Qut/IQrlTgN4adgz5MYa/M+6lFrH8nfwgtecftUOcJKbcgClKsNAwMxZ 5Djl8sIhRotPMFln1HX4W4W7fhRuci9Vav+cySh1vMntCXQuI7FTaPLu/ZBHs00tvOwe nO5SnOmD9TAFOsiYErByLgKoDOuCencyjBt4QEmPdB7mojxlrhUgEQxbrEvQ5hiJP5yx JK2DJRqQ2qvzy54Lo9kIHbhSRy85WAxLZBUEo2IUmJ0ip83qtc0pyCxysStE0oNQtxG4 lRhw== X-Gm-Message-State: AOJu0YzrLZb4xl4oFj7qd43gQfny16DkF7KkHO5uMjiG82fKAabWPUaq 7w9/45x/CDgqKAnqT0qCEPBSDXaPsjFNCYfmEc30vSGNESSzhrxcv0MsuJwTmUwKeE8= X-Gm-Gg: AR+sD12QoTZ7jThWJ+sMrHW260WpV75ICSFolCiNmwZBlWNJAgqEyWzYE4Mt9VYDY18 l8h/MMDgIIoCupVxrzuJHFbyOgQbpg3AkwjihBV4vSpa5M5mzNhNoBMf3hCsFXNyIq28iYv7Xmr nsrwR7Em73EO/si0hXqDtWjODlrR7R9NX/5MEL9C7RRltbgDbJQZjyJ/m9rlGSXv/zXqfCQImFa NLuNkBbHn3lWNA6RlGDMs1N+l1vaMhEo4lGbTytNpiHjtJcjHvE9evTHbUKfRt5BwLlN8JNYQhz noCsFzS40ERIP9LHGkX+o7eSpuql9vH4AV+wSFZXa9KHXtIrbIPq20e6eP40PDWVGcTSqyWMQNG 4uf+f5i4MGo7zIzsNSax3c20+ElbHq0lVp0HTgPtnn4Vv7Z3yhasAiMFii7H04Rmwb7yje2d8/p uLRsv7mMxUMl+0yeNumAtp+26B0qcRWkZR/NrNmwAEbhwmM9inQyplDW7BW1Fd X-Received: by 2002:a53:ac85:0:b0:667:bb94:18d9 with SMTP id 956f58d0204a3-66c72b844d1mr10326287d50.13.1786993254083; Mon, 17 Aug 2026 12:00:54 -0700 (PDT) From: Matt Turner To: qemu-devel@nongnu.org Cc: richard.henderson@linaro.org, pbonzini@redhat.com, philmd@mailo.com, zhao1.liu@intel.com, laurent@vivier.eu, deller@gmx.de, pierrick.bouvier@oss.qualcomm.com, Matt Turner Subject: [RFC PATCH 4/8] RFC: tcg: probe the TB jump cache inline instead of calling a helper Date: Mon, 17 Aug 2026 15:00:34 -0400 Message-ID: <20260817190038.580257-5-mattst88@gmail.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260817190038.580257-1-mattst88@gmail.com> References: <20260817190038.580257-1-mattst88@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists1p.gnu.org; Received-SPF: pass client-ip=2607:f8b0:4864:20::b12a; envelope-from=mattst88@gmail.com; helo=mail-yx1-xb12a.google.com X-Spam_score_int: -17 X-Spam_score: -1.8 X-Spam_bar: - X-Spam_report: (-1.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @gmail.com) X-ZM-MESSAGEID: 1786993409924158500 Content-Type: text/plain; charset="utf-8" Every indirect branch that cannot use goto_tb ends in tcg_gen_lookup_and_goto_ptr(), which calls helper_lookup_tb_ptr(). For an emulated compiler that is 8.4 billion helper calls in a single translation unit: 24.6% of all TB exits take this path, because jsr/ret/jmp have a register destination and because goto_tb is restricted to same-page targets. The helper itself is already tight, but each call pays for a call frame, the can_do_io store, the get_tb_cpu_state() indirect call through TCGCPUOps, curr_cflags(), and a breakpoint check, before it gets to the jump cache probe that almost always hits (95.8% for this workload). Emit the probe inline instead. The destination PC is already in a TCG temp, and the flags and cflags the destination must match are constants at translation time, so the fast path is a hash, three guarded loads and a goto_ptr. Only a miss calls the helper, which still owns filling the cache. Two details matter for the generated code. The flags and cflags guards are folded into a single aligned 64-bit load and compare, since the fields are adjacent. And each path emits its own goto_ptr rather than branching to a shared one: a temp live across the label is spilled and reloaded on every dispatch, which cost 6.3% on its own. Measured with qemu-alpha running an emulated alpha gcc 16.2.0 compiling the SQLite 3.45.1 amalgamation (255k lines, -O2) on an x86-64 host, LTO build, on top of the preceding three patches: before: 1,402,816,253,499 instructions after: 890,713,633,237 instructions -36.51% before: 115.75s wall clock after: 84.44s wall clock -27.05% The gap between the two is the point at which this stops being a straight-line win: the helper call was highly predictable work that the host pipelined well, so removing it retires far fewer instructions than it saves time. IPC falls from 2.48 to 2.15 across this patch for that reason. Despite emitting more code, this also reduces instruction cache pressure, because a dispatch no longer jumps into qemu's .text and evicts translated code: before: 11,476,318,964 L1-icache-load-misses after: 6,990,186,701 L1-icache-load-misses -39.1% The mechanism is visible directly in a profile: helper_lookup_tb_ptr() falls from 30.97% of samples to 0.42%, and qemu's own .text falls from 38.9% to 5.4%, with the balance moving into generated code. Combined with the three preceding patches, against an unmodified LTO build, 1,647,901,588,726 instructions fall to 890,713,633,237, or -45.95%. The emulated compiler produces byte-identical output throughout. Open issues, hence RFC: - The flags/cflags guards use the *current* TB's values as constants. That assumes the CPU flags feeding get_tb_cpu_state() cannot change within a TB, and that curr_cflags() cannot change under a running TB (gdb attaching to enable single-step would). Both need to be established or the values need to be loaded at runtime. - tcg/tcg-op.c has no business including accel/tcg/tb-jmp-cache.h or knowing the CPUJumpCache layout. The probe likely belongs in accel/tcg with a small emit helper exported from tcg/. - The jump cache entry is read without qatomic_read(); entries are invalidated concurrently by setting tb to NULL. - Only wired up for alpha so far, and only for 64-bit guest PCs. Signed-off-by: Matt Turner --- include/tcg/tcg-op-common.h | 2 + target/alpha/translate.c | 6 ++- tcg/tcg-op.c | 77 +++++++++++++++++++++++++++++++++++++ 3 files changed, 83 insertions(+), 2 deletions(-) diff --git ./include/tcg/tcg-op-common.h ./include/tcg/tcg-op-common.h index 1fe342db0d..52cd0d3eab 100644 --- ./include/tcg/tcg-op-common.h +++ ./include/tcg/tcg-op-common.h @@ -84,6 +84,8 @@ void tcg_gen_goto_tb(unsigned idx); * this op is equivalent to calling tcg_gen_exit_tb() with 0 as the argume= nt. */ void tcg_gen_lookup_and_goto_ptr(void); +void tcg_gen_lookup_and_goto_ptr_inline(TCGv_i64 pc, uint32_t flags, + uint32_t cflags); =20 void tcg_gen_plugin_cb(unsigned from); void tcg_gen_plugin_mem_cb(TCGv_i64 addr, unsigned meminfo); diff --git ./target/alpha/translate.c ./target/alpha/translate.c index c66e3f9c14..10a4ec0c11 100644 --- ./target/alpha/translate.c +++ ./target/alpha/translate.c @@ -449,7 +449,8 @@ static void gen_goto_tb(DisasContext *ctx, unsigned tb_= slot_idx, int32_t disp) tcg_gen_exit_tb(ctx->base.tb, tb_slot_idx); } else { gen_pc_disp(ctx, cpu_pc, disp); - tcg_gen_lookup_and_goto_ptr(); + tcg_gen_lookup_and_goto_ptr_inline(cpu_pc, ctx->base.tb->flags, + ctx->base.tb->cflags); } } =20 @@ -2917,7 +2918,8 @@ static void alpha_tr_tb_stop(DisasContextBase *dcbase= , CPUState *cpu) gen_pc_disp(ctx, cpu_pc, 0); /* FALLTHRU */ case DISAS_PC_UPDATED: - tcg_gen_lookup_and_goto_ptr(); + tcg_gen_lookup_and_goto_ptr_inline(cpu_pc, ctx->base.tb->flags, + ctx->base.tb->cflags); break; case DISAS_PC_UPDATED_NOCHAIN: tcg_gen_exit_tb(NULL, 0); diff --git ./tcg/tcg-op.c ./tcg/tcg-op.c index bbcb510c76..a3efc56a9a 100644 --- ./tcg/tcg-op.c +++ ./tcg/tcg-op.c @@ -28,6 +28,8 @@ #include "tcg/tcg-op-common.h" #include "exec/translation-block.h" #include "exec/plugin-gen.h" +#include "hw/core/cpu.h" +#include "../accel/tcg/tb-jmp-cache.h" #include "tcg-internal.h" #include "tcg-has.h" =20 @@ -2620,3 +2622,78 @@ void tcg_gen_lookup_and_goto_ptr(void) tcg_gen_op1i(INDEX_op_goto_ptr, TCG_TYPE_PTR, tcgv_ptr_arg(ptr)); tcg_temp_free_ptr(ptr); } + +/* + * As tcg_gen_lookup_and_goto_ptr(), but probe the TB jump cache inline + * instead of calling helper_lookup_tb_ptr() unconditionally. @pc must + * hold the destination guest PC; @flags and @cflags are the values the + * destination TB must have been translated with. + */ +void tcg_gen_lookup_and_goto_ptr_inline(TCGv_i64 pc, uint32_t flags, + uint32_t cflags) +{ + TCGv_ptr jc, ent, tbp, ptr; + TCGv_i64 h, tmp; + TCGLabel *slow; + uint64_t fpair; + + if (tcg_ctx->gen_tb->cflags & CF_NO_GOTO_PTR) { + tcg_gen_exit_tb(NULL, 0); + return; + } + + plugin_gen_disable_mem_helpers(); + + QEMU_BUILD_BUG_ON(sizeof(((CPUJumpCache *)0)->array[0]) !=3D 16); + QEMU_BUILD_BUG_ON(offsetof(TranslationBlock, cflags) !=3D + offsetof(TranslationBlock, flags) + 4); + + jc =3D tcg_temp_ebb_new_ptr(); + ent =3D tcg_temp_ebb_new_ptr(); + tbp =3D tcg_temp_ebb_new_ptr(); + ptr =3D tcg_temp_ebb_new_ptr(); + h =3D tcg_temp_ebb_new_i64(); + tmp =3D tcg_temp_ebb_new_i64(); + slow =3D gen_new_label(); + + /* h =3D tb_jmp_cache_hash_func(pc) * sizeof(array[0]) */ + tcg_gen_shri_i64(h, pc, TB_JMP_CACHE_BITS); + tcg_gen_xor_i64(h, h, pc); + tcg_gen_andi_i64(h, h, TB_JMP_CACHE_SIZE - 1); + tcg_gen_shli_i64(h, h, 4); + + tcg_gen_ld_ptr(jc, tcg_env, + offsetof(CPUState, tb_jmp_cache) - sizeof(CPUState)); + tcg_gen_trunc_i64_ptr(ent, h); + tcg_gen_add_ptr(ent, jc, ent); + + tcg_gen_ld_ptr(tbp, ent, offsetof(CPUJumpCache, array[0].tb)); + tcg_gen_brcondi_ptr(TCG_COND_EQ, tbp, 0, slow); + + tcg_gen_ld_i64(tmp, ent, offsetof(CPUJumpCache, array[0].pc)); + tcg_gen_brcond_i64(TCG_COND_NE, tmp, pc, slow); + + /* + * flags and cflags are adjacent uint32_t, so one aligned 64-bit load + * and compare covers both. + */ +#if HOST_BIG_ENDIAN + fpair =3D ((uint64_t)flags << 32) | cflags; +#else + fpair =3D ((uint64_t)cflags << 32) | flags; +#endif + tcg_gen_ld_i64(tmp, tbp, offsetof(TranslationBlock, flags)); + tcg_gen_brcondi_i64(TCG_COND_NE, tmp, fpair, slow); + + tcg_gen_ld_ptr(ptr, tbp, offsetof(TranslationBlock, tc.ptr)); + tcg_gen_op1i(INDEX_op_goto_ptr, TCG_TYPE_PTR, tcgv_ptr_arg(ptr)); + + /* + * Emit a second goto_ptr rather than branching to a shared one: a temp + * live across the label would be spilled and reloaded on every dispat= ch. + */ + gen_set_label(slow); + ptr =3D tcg_temp_ebb_new_ptr(); + gen_helper_lookup_tb_ptr(ptr, tcg_env); + tcg_gen_op1i(INDEX_op_goto_ptr, TCG_TYPE_PTR, tcgv_ptr_arg(ptr)); +} --=20 2.54.0 From nobody Mon Sep 28 00:10:01 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=gmail.com ARC-Seal: i=1; a=rsa-sha256; t=1786993361; cv=none; d=zohomail.com; s=zohoarc; b=VMDem36DWWGIN2LkNQ6btwJhDq9gFIPNIGCBlQgnyB0puvTOIAox75otuFb7xZrOpCnG28ICdD4kyG+OdnvECxsEt4VMWWtO1DRg4sjCDjSDZoJoEJfjVXpKgJKfCuI9kA+jYOIBQD0lPHdcPK9cBeBqYYu8DZCmQ8s0OPVlJx8= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1786993361; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=Tc0VjzEQDHo8/Dm/bwwN45pmK61wlymDtBgaGkTeofs=; b=U9DEao35ngBxm89nHo6oaIPvP4HG29OuFtDgbVpnPmpA14pgEGhCFYJZKU87+ihGCeBEcqC6MgUZySP3Ygh+CcfE3gmUObrjv7MEuwgSMRDmRQ5tGVvk8zxzgI6+4CEQ4ddYCnCamlxkY3RFZA8NjtUukwZZcFuDI+x0SLiOu50= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists1p.gnu.org (lists1p.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1786993361193728.2883653121407; Mon, 17 Aug 2026 12:02:41 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists1p.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ww2bJ-0002rP-CN; Mon, 17 Aug 2026 15:02:28 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists1p.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ww2Zw-0006mB-Ob for qemu-devel@nongnu.org; Mon, 17 Aug 2026 15:01:10 -0400 Received: from mail-yx1-xb12a.google.com ([2607:f8b0:4864:20::b12a]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1ww2Zu-0006lB-A8 for qemu-devel@nongnu.org; Mon, 17 Aug 2026 15:01:00 -0400 Received: by mail-yx1-xb12a.google.com with SMTP id 956f58d0204a3-66c70e2904dso4085464d50.1 for ; Mon, 17 Aug 2026 12:00:57 -0700 (PDT) Received: from localhost ([2600:1702:7a90:6f9f:8bc4:8aec:108d:7a04]) by smtp.gmail.com with ESMTPSA id 956f58d0204a3-66cb4780b7asm1261880d50.12.2026.08.17.12.00.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 17 Aug 2026 12:00:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1786993257; x=1787598057; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=Tc0VjzEQDHo8/Dm/bwwN45pmK61wlymDtBgaGkTeofs=; b=MWMvpT2HMRpSqo7CR6cUcsdq1lzhvJ3xCCC7YKSJpl31epID+LPI8VzDmd5is6lsLp aVpjZG/s8oVV8+tR3nrefK1uL2EaWThK11pJeu/xPEh2Z+F/UXmab3BSY2X9vVpH31c+ z6Hs4gdnw4CE0pExF6M13h4sGO2PV0+j4e5gr/pPdcZ/aoFnT/+NVze7Z1vhoemCpTN0 K8Eg2OIVN5stCzftx0vKZe4/tCc0WVzto9VGdRr7UXmVKbsHzlpSvON0MEVKlJQerQr5 sRyvPV6WM/CPN8Uxv6eaRetv7SMSsS2RWU9kt9Y1egt4q7x8y/8ef49QastYGcZhoQ0J YSFg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1786993257; x=1787598057; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=Tc0VjzEQDHo8/Dm/bwwN45pmK61wlymDtBgaGkTeofs=; b=A/UQYMy6k1lodNzGnnR2B7NnlyoSt4F5dLjDViWtAIliARlq5EzaeY8ZgtdqgJLIDf b4SbZLkPZs8JR/uwtBs6sGiknQ0Nrmrw45D6HOZrMV7XVPtBJ2lZbZKGMrtvFDtsfohm uxFZEUQnjKP4i1yk49rNbmrefK8k2ixbvCmuutKI4YJv/e5OY73WS40zykdt10bcmUNS LhLVlt421ZyEdB8thDdrYdzICuOddguvARix+wdcL3WnWRn2bi5hV7e1kO+9g4cSIBIv LI8zWHp9WHeYsRqQJM7Ri8WeiRSoo/10Hl57OVcPWD1KICAL13lVp84YCnNqABrbJZAR do4w== X-Gm-Message-State: AOJu0YzPD7eunYeXKHJ/43gmA+JTdrkITzE4tpf41mrzObJfkgnU/Ior CiBGnmphAuDE6G8msQx4iLberQvflZb6R4SsBvAC26Fv2yxEXIGVBWi8Y2UYvvWiWIs= X-Gm-Gg: AR+sD10KFuXXDfbYP7qTkhHF6EwkJXoCou/YL9PoScN2x5Tx6DIiHs7aRUA06MgnoY+ S2bwVNfu/TrAzJzj0gJLLN60mJFx12sxQ+bcmk261h3LARYMmTdeM1vKrTY0ggIiu4PC8uzP6bB YMxrsXDnTWluh596NiEZqgqGyjLSso29JayLUXQvuxeAxfegrPGMzf8sUWKbeLFT5KdoXw9fuLQ Dxm7jEikqzfd3GMedWU632DwUnGxZJLQQYVevQbLOjHZMlL1GP3AscrbkvwIrs4ezosi/qJgVjn VfYe4lwPYZCpLSWnjLE8XpY6bg6+iTAz3mMn6NTqq1hqdFLJRWX6BmtZLrsYedNI2UNaF9HYJvQ qJ05aLDJpOzZ9IuBbNSp3qUdoqRFjOJb4rGCYx1gEfWva5yqIyqzSfXw4zhVjKMdkxanTJxYmBT SFs6sYq6gGuEdgItJpJw/L25LX1UmZJuRcLpwQviGXPOTkXVR+iyGDgDv1SJkd X-Received: by 2002:a05:690e:b89:b0:66b:354e:5202 with SMTP id 956f58d0204a3-66c72b18b52mr8631640d50.7.1786993256866; Mon, 17 Aug 2026 12:00:56 -0700 (PDT) From: Matt Turner To: qemu-devel@nongnu.org Cc: richard.henderson@linaro.org, pbonzini@redhat.com, philmd@mailo.com, zhao1.liu@intel.com, laurent@vivier.eu, deller@gmx.de, pierrick.bouvier@oss.qualcomm.com, Matt Turner Subject: [RFC PATCH 5/8] RFC: accel/tcg: allow cross-page goto_tb chaining in user-only builds Date: Mon, 17 Aug 2026 15:00:35 -0400 Message-ID: <20260817190038.580257-6-mattst88@gmail.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260817190038.580257-1-mattst88@gmail.com> References: <20260817190038.580257-1-mattst88@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists1p.gnu.org; Received-SPF: pass client-ip=2607:f8b0:4864:20::b12a; envelope-from=mattst88@gmail.com; helo=mail-yx1-xb12a.google.com X-Spam_score_int: -17 X-Spam_score: -1.8 X-Spam_bar: - X-Spam_report: (-1.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @gmail.com) X-ZM-MESSAGEID: 1786993363726158501 Content-Type: text/plain; charset="utf-8" translator_use_goto_tb() refuses to chain unless the destination is on the same page as the start of the TB. For guests whose text is much larger than a page this is expensive: an emulated alpha gcc compiling a 255k line translation unit takes the indirect dispatch path for 8.4 billion of its 34.2 billion TB exits, and a large share of those are ordinary direct branches that simply crossed an 8 KiB page boundary. The restriction was made unconditional by d3a2a1d803 ("accel/tcg: Introduce translator_use_goto_tb"), whose rationale was: Various targets avoid the page crossing test for CONFIG_USER_ONLY, but that is wrong: mmap and mprotect can change page permissions. That is true, but in user-only builds the invalidation path already covers it. There are no page tables: every mmap, mprotect and munmap reaches page_set_flags(), which calls tb_invalidate_phys_range() whenever the flags actually change, and tb_phys_invalidate() calls tb_jmp_unlink() to reset incoming jumps. A chained cross-page jump is therefore broken whenever the destination page's permissions change. This is not true in system mode, where TBs are keyed by physical address and a page table change invalidates nothing, so the restriction is kept there. Add tests/tcg/alpha/test-xpage-chain.c to cover the hazard directly. It places a direct branch near the end of one page targeting the next page, runs it 200000 times so the chain is established, then checks that mprotect(PROT_NONE) makes the next call fault, and that remapping the page with different code runs the new code rather than a stale translation. The test detects the hazard it is meant to detect: with the tb_invalidate_phys_range() call in page_set_flags() commented out, it fails both phases, executing page B after PROT_NONE and returning the stale result. Measured with qemu-alpha running an emulated alpha gcc 16.2.0 compiling the SQLite 3.45.1 amalgamation on an x86-64 host, LTO build, on top of the preceding patches: before: 890,713,633,237 instructions after: 869,178,598,378 instructions -2.42% before: 84.44s wall clock after: 80.49s wall clock -4.68% Note that this is worth more in time than in instructions, the reverse of the preceding patch: a chained jump replaces a cache probe whose loads can miss, so the instructions it removes are more expensive than average. Measured before the inline jump cache probe, when a missed chain cost a helper call rather than an inline probe, the same change was worth -7.9%. RFC because this reverses a deliberate decision and the reasoning above wants review from someone who knows the invalidation paths better than I do. Signed-off-by: Matt Turner --- accel/tcg/translator.c | 12 ++++ tests/tcg/alpha/Makefile.target | 2 +- tests/tcg/alpha/test-xpage-chain.c | 111 +++++++++++++++++++++++++++++ 3 files changed, 124 insertions(+), 1 deletion(-) create mode 100644 tests/tcg/alpha/test-xpage-chain.c diff --git ./accel/tcg/translator.c ./accel/tcg/translator.c index 29e609b2ec..4921bf978c 100644 --- ./accel/tcg/translator.c +++ ./accel/tcg/translator.c @@ -117,8 +117,20 @@ bool translator_use_goto_tb(DisasContextBase *db, vadd= r dest) return false; } =20 +#ifdef CONFIG_USER_ONLY + /* + * There are no page tables in user-only mode. Every mmap, mprotect a= nd + * munmap goes through page_set_flags(), which calls + * tb_invalidate_phys_range() whenever the flags actually change, and + * tb_phys_invalidate() unlinks incoming jumps. A chained cross-page + * jump is therefore broken whenever the destination page's permissions + * change, so the same-page restriction is not needed here. + */ + return true; +#else /* Check for the dest on the same page as the start of the TB. */ return translator_is_same_page(db, dest); +#endif } =20 void translator_loop(CPUState *cpu, TranslationBlock *tb, int *max_insns, diff --git ./tests/tcg/alpha/Makefile.target ./tests/tcg/alpha/Makefile.tar= get index 36d8ed1eae..eee986bab6 100644 --- ./tests/tcg/alpha/Makefile.target +++ ./tests/tcg/alpha/Makefile.target @@ -5,7 +5,7 @@ ALPHA_SRC=3D$(SRC_PATH)/tests/tcg/alpha VPATH+=3D$(ALPHA_SRC) =20 -ALPHA_TESTS=3Dhello-alpha test-cond test-cmov test-ovf test-cvttq +ALPHA_TESTS=3Dhello-alpha test-cond test-cmov test-ovf test-cvttq test-xpa= ge-chain TESTS+=3D$(ALPHA_TESTS) =20 test-cmov: EXTRA_CFLAGS=3D-DTEST_CMOV diff --git ./tests/tcg/alpha/test-xpage-chain.c ./tests/tcg/alpha/test-xpag= e-chain.c new file mode 100644 index 0000000000..7916d544af --- /dev/null +++ ./tests/tcg/alpha/test-xpage-chain.c @@ -0,0 +1,111 @@ +/* + * Cross-page TB chaining hazard test. + * + * Phase 1: a direct branch (br) near the end of page A targets page B. + * Run it enough times that QEMU chains TB_A -> TB_B. + * Phase 2: mprotect page B away. Re-running must fault. + * Phase 3: remap page B with different code. Re-running must execute the + * NEW code, not a stale chained translation of the old code. + * + * SPDX-License-Identifier: GPL-2.0-or-later + */ +#include +#include +#include +#include +#include +#include +#include + +#define PS 8192 + +static sigjmp_buf jb; +/* + * Written by the SIGSEGV handler and read by main(), so it must not be + * cached in a register across the faulting call. + */ +static volatile sig_atomic_t caught; + +static void segv(int sig) +{ + caught =3D 1; + siglongjmp(jb, 1); +} + +/* lda $0, imm($31) -> v0 =3D imm */ +static unsigned int lda_v0(int imm) +{ + return 0x201F0000u | (unsigned short)imm; +} + +int main(void) +{ + struct sigaction sa; + int rc =3D 0; + unsigned char *m =3D mmap(NULL, 2 * PS, PROT_READ | PROT_WRITE | PROT_= EXEC, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + if (m =3D=3D MAP_FAILED) { + perror("mmap"); + return 2; + } + + unsigned char *pa =3D m, *pb =3D m + PS; + unsigned int *entry =3D (unsigned int *)(pa + PS - 64); + unsigned int *tgt =3D (unsigned int *)(pb + 16); + + entry[0] =3D lda_v0(1); + long disp =3D ((long)tgt - ((long)&entry[1] + 4)) / 4; + entry[1] =3D 0xC3E00000u | (unsigned int)(disp & 0x1FFFFF); /* br $31= ,tgt */ + tgt[0] =3D 0x6BFA8001u; /* ret = */ + __builtin___clear_cache((char *)m, (char *)m + 2 * PS); + + long (*fn)(void) =3D (long (*)(void))entry; + + for (int i =3D 0; i < 200000; i++) { + if (fn() !=3D 1) { + printf("FAIL: phase 1 wrong result\n"); + return 1; + } + } + printf("phase 1 ok (chained)\n"); + + memset(&sa, 0, sizeof(sa)); + sa.sa_handler =3D segv; + sigemptyset(&sa.sa_mask); + if (sigaction(SIGSEGV, &sa, NULL) !=3D 0) { + perror("sigaction"); + return 2; + } + if (mprotect(pb, PS, PROT_NONE) !=3D 0) { + perror("mprotect"); + return 2; + } + if (sigsetjmp(jb, 1) =3D=3D 0) { + fn(); + printf("FAIL: phase 2 executed page B after mprotect(PROT_NONE)\n"= ); + rc =3D 1; + } else if (!caught) { + printf("FAIL: phase 2 longjmp without entering the handler\n"); + rc =3D 1; + } else { + printf("phase 2 ok (faulted)\n"); + } + + /* Phase 3: remap with different code, expect the new code to run. */ + if (mprotect(pb, PS, PROT_READ | PROT_WRITE | PROT_EXEC) !=3D 0) { + perror("mprotect back"); + return 2; + } + tgt[0] =3D lda_v0(2); + tgt[1] =3D 0x6BFA8001u; + __builtin___clear_cache((char *)pb, (char *)pb + PS); + + long r =3D fn(); + if (r !=3D 2) { + printf("FAIL: phase 3 returned %ld, expected 2 (stale chain)\n", r= ); + rc =3D 1; + } else { + printf("phase 3 ok (new code ran)\n"); + } + return rc; +} --=20 2.54.0 From nobody Mon Sep 28 00:10:01 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=gmail.com ARC-Seal: i=1; a=rsa-sha256; t=1786993417; cv=none; d=zohomail.com; s=zohoarc; b=JMUJQIqK6VSEGddtT1eYP4LiruAlB1lVJe60mWIR69/peznwld/2d4rGq23uNVNaGno3wCZCfyT62HRBdex1yzR/Rx4s8/i/1rRNUeyCOsBzR7dRgcTVsCBQoeGFSz4fb1+VtwApQYOSM0t4SpVdQWR9WIeoAnHa9lohueYXoEI= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1786993417; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=N2baksM9JrMxPQxDR9n0GKs5u4RQLJmCtvwi2YKEvp4=; b=ITK61gI3k7Wv+FG6nC8TmfaUTrl2HGjFFEYezigXjZrh9Xw5ZOcDAtOfFTig3dUGT5jSKMv9Cw3QY1kqRmiG/G/TfFaJoNJJdGoI8SG0UVyGcZiCTN3XaSlOMSZCIS7CL++ta4IQWs6RajWm/C+VYseuL5adDqRYPwwQWs5mmlM= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists1p.gnu.org (lists1p.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1786993417186335.7729209396216; Mon, 17 Aug 2026 12:03:37 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists1p.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ww2ap-0000wJ-Q5; Mon, 17 Aug 2026 15:01:55 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists1p.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ww2a0-0006mT-6c for qemu-devel@nongnu.org; Mon, 17 Aug 2026 15:01:11 -0400 Received: from mail-yw1-x1135.google.com ([2607:f8b0:4864:20::1135]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1ww2Zx-0006ll-Ok for qemu-devel@nongnu.org; Mon, 17 Aug 2026 15:01:03 -0400 Received: by mail-yw1-x1135.google.com with SMTP id 00721157ae682-836ce6cbe1eso48592197b3.1 for ; Mon, 17 Aug 2026 12:01:01 -0700 (PDT) Received: from localhost ([2600:1702:7a90:6f9f:8bc4:8aec:108d:7a04]) by smtp.gmail.com with ESMTPSA id 00721157ae682-840f3f092c4sm8661397b3.47.2026.08.17.12.00.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 17 Aug 2026 12:00:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1786993260; x=1787598060; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=N2baksM9JrMxPQxDR9n0GKs5u4RQLJmCtvwi2YKEvp4=; b=PFDP45iUJigqdByHCWL661AvMw/OmI5ffKRVHAkaPZBExLHpo6e/pLI7QsaU5FZyR0 ubX0URrsOmh2cqME+57iF+BJcAiuS7QXP0C82Lk09sQQkVZ4qrQ5VmVof+o5YgfyMR78 lAefp3svBZTIIF0o7MeFpsgJ+zBDlq6c6eQcO0XZOQ5OXSEpiG1FzjjEIKbR2WW+PHT/ mqHXEyJJ1ndkdZvQdfGi7zyuhDwDKm8YJVROaOC0YPJ3xebhNoa9/MA7TbedulwqN/cX mtTm+k/x67fTeNEtrj7Vqa/VjTVpKUU/lPF44PPWK4POGFf5Tg7MS7CQyFdmn7nTzTBP Q4/A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1786993260; x=1787598060; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=N2baksM9JrMxPQxDR9n0GKs5u4RQLJmCtvwi2YKEvp4=; b=BKtE83vDVKtewyzV5BQJzF2pmAZ/Bk1Ofkesa0uOuoVRGQgjTPWJUr0jIA1Vakc7R2 AY7Sl8k/+92Z713OSbKYoUWetjhsC9ECa4KL0jqEzlwoEmXk1CjsPAbmjuphBmzJTKbD Euwp/WxfTPSb7m1wiWkeRUZY58aysccp75uQvsrVXK1iGEZlUWW5YPbrRgn36dCRvJIU oeIn++rlxxZOAQuzV4gAyA/ni1onhNIAO6TNqG1+7vKXrt4j7kVYBLU/3umeF1h5eWJG vrGbxD7/h90HDthPh75obBoiZZcuLdQGPBa8DMD9DA92iC2B3d0BTwz1aU1gDrOlPxL5 na1Q== X-Gm-Message-State: AOJu0YyQ2ZIOCD5sA4uZjmc/KaCrPR36L+OQ2gwEMB3cYuWBcFdiV3HD tjWFt0H5TnX+IBHw1ukpvdznXoRg4eTF7vC8jXCJnwYv8E5mU19Win81bi7oMZMHJSI= X-Gm-Gg: AR+sD12QwvzdxYxyn58KIC5oIqg5cQevLDWLaaqClOX0+X6VPBleImYI/mDds8GEb0v +SAOPiaLxngdTqNf1sI50Fe3XMJNeXQNvCuOjRdMNS1Iif+JCFAN2FQZ5yJEdGk8/inCzbEaJWf QCbOcqFIP4ky3zGfNx2SiSg/5UkEpjzjimndVf9QRssTDww006rhBh2FB4nHXRai87luUbykT39 tsfw3zx4jMhykIJIPThh3rTHOAAS8FpinCP+S1sQnRlPUTROVVPWbqjiW21u0rtr9dR86bVVrvh 6Q4lhnOmHmh503Z1777sSg/Dqh0sADFcTURfE9V/iR6qqM9g5wcY4MUzvRYsrkLNaNbgkL6s/FM 3hbWxGTVZn358bEMVcsRoxrkfLRc+scOnOVyYfz0rKZj4KRvI7LyLHYmgurP2zCIXUabq8eCYlQ z2XsSaZwknLSbGk7LIBN1xMoHdD+tq+uiD0YudQqHZ00l1PkicPce3u2N0gfud X-Received: by 2002:a05:690c:4b10:b0:81c:8005:9104 with SMTP id 00721157ae682-841325f8c28mr13314137b3.30.1786993260222; Mon, 17 Aug 2026 12:01:00 -0700 (PDT) From: Matt Turner To: qemu-devel@nongnu.org Cc: richard.henderson@linaro.org, pbonzini@redhat.com, philmd@mailo.com, zhao1.liu@intel.com, laurent@vivier.eu, deller@gmx.de, pierrick.bouvier@oss.qualcomm.com, Matt Turner Subject: [RFC PATCH 6/8] RFC: accel/tcg: only poll for interrupts in blocks that can close a cycle Date: Mon, 17 Aug 2026 15:00:36 -0400 Message-ID: <20260817190038.580257-7-mattst88@gmail.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260817190038.580257-1-mattst88@gmail.com> References: <20260817190038.580257-1-mattst88@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists1p.gnu.org; Received-SPF: pass client-ip=2607:f8b0:4864:20::1135; envelope-from=mattst88@gmail.com; helo=mail-yw1-x1135.google.com X-Spam_score_int: -17 X-Spam_score: -1.8 X-Spam_bar: - X-Spam_report: (-1.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @gmail.com) X-ZM-MESSAGEID: 1786993418153158500 Content-Type: text/plain; charset="utf-8" Every translation block begins by loading cpu->neg.icount_decr.u32, testing it and branching to the exit path. That is three host instructions at the top of every TB. Blocks are short, so this is expensive: an emulated alpha gcc 16.2.0 compiling a 255k line translation unit executes 34.2 billion TBs at 6.04 guest instructions each. Forcing CF_NOIRQ on for the whole run, which is not correct but bounds the prize, is worth 12.0% of all instructions retired. The check does not have to be in every block. Interrupt latency is bounded as long as every cycle in the guest control flow graph passes through at least one block that polls. Any such cycle must contain either an edge whose destination is at or below the start of the block it leaves from, or an edge whose destination is not known at translation time: take the block with the lowest start address in the cycle, and the edge entering it comes from a block at or above it. So record, during translation, whether this TB has such an edge. translator_use_goto_tb() already sees every statically known destination, and every target that emits goto_tb reaches it, so a backward edge sets DisasContextBase::needs_exit_check there. Indirect destinations are flagged by tcg_gen_lookup_and_goto_ptr(). Blocks with neither cannot close a cycle on their own and can skip the poll. The check is therefore emitted retroactively in gen_tb_end(), using the same emit_before_op mechanism the can_do_io stores use, and only when one of the two flags is set. icount opts out and keeps the unconditional counter. Measured on an x86-64 host, LTO build, on top of the preceding patches: before: 869,178,598,378 instructions after: 809,988,851,304 instructions -6.81% before: 80.49s wall clock after: 78.00s wall clock -3.10% That is 57% of the 12.0% ceiling, which is about right: roughly a quarter of TB exits are indirect and are still polled, plus every loop back edge. For the series as a whole, against an unmodified LTO build, instructions retired fall from 1,647,901,588,726 to 809,988,851,304 (-50.85%) and wall clock from 133.57s to 78.00s (-41.61%). The two do not match because what the series removes is mostly cheap, well-predicted dispatch overhead: IPC falls from 2.53 to 2.12 as the remaining work gets less regular. tests/tcg/alpha/test-xpage-chain.c still passes, the emulated compiler still produces byte-identical output, and a tight loop under alarm(1) is still interrupted, after 897 million iterations. RFC because: - The soundness argument depends on every goto_tb destination passing through translator_use_goto_tb(). No target in the tree bypasses it today, but nothing enforces that. - System mode interrupt latency now depends on guest control flow rather than on block count. The bound is one straight-line run between cycles, which should be fine, but timer-driven guests deserve a closer look than I can give them. Signed-off-by: Matt Turner --- accel/tcg/translator.c | 57 ++++++++++++++++++++++++++++++++++++--- include/exec/translator.h | 2 ++ include/tcg/tcg.h | 2 ++ tcg/tcg-op.c | 2 ++ 4 files changed, 60 insertions(+), 3 deletions(-) diff --git ./accel/tcg/translator.c ./accel/tcg/translator.c index 4921bf978c..ee61dec1c6 100644 --- ./accel/tcg/translator.c +++ ./accel/tcg/translator.c @@ -42,12 +42,29 @@ bool translator_io_start(DisasContextBase *db) return true; } =20 +/* + * Any cycle in the guest control flow graph must contain an edge whose + * destination is at or below the start of the block it leaves from, or an + * edge whose destination is not known at translation time. Only blocks wi= th + * such an edge need the interrupt check, so defer the decision until the = end + * of translation, when we know which edges this TB has. + * + * icount needs the counter unconditionally, so it opts out. + */ +static bool defer_exit_check(uint32_t cflags) +{ + return !(cflags & CF_USE_ICOUNT); +} + static TCGOp *gen_tb_start(DisasContextBase *db, uint32_t cflags) { TCGv_i32 count =3D NULL; TCGOp *icount_start_insn =3D NULL; =20 - if ((cflags & CF_USE_ICOUNT) || !(cflags & CF_NOIRQ)) { + tcg_ctx->exit_check_needed =3D false; + + if ((cflags & CF_USE_ICOUNT) || + (!(cflags & CF_NOIRQ) && !defer_exit_check(cflags))) { count =3D tcg_temp_new_i32(); tcg_gen_ld_i32(count, tcg_env, offsetof(CPUState, neg.icount_decr.u32) - @@ -73,6 +90,12 @@ static TCGOp *gen_tb_start(DisasContextBase *db, uint32_= t cflags) */ if (cflags & CF_NOIRQ) { tcg_ctx->exitreq_label =3D NULL; + } else if (defer_exit_check(cflags)) { + /* + * Emitted retroactively by gen_tb_end(), but only if this TB can = be + * part of a control flow cycle. + */ + tcg_ctx->exitreq_label =3D gen_new_label(); } else { tcg_ctx->exitreq_label =3D gen_new_label(); tcg_gen_brcondi_i32(TCG_COND_LT, count, 0, tcg_ctx->exitreq_label); @@ -88,7 +111,8 @@ static TCGOp *gen_tb_start(DisasContextBase *db, uint32_= t cflags) } =20 static void gen_tb_end(const TranslationBlock *tb, uint32_t cflags, - TCGOp *icount_start_insn, int num_insns) + TCGOp *icount_start_insn, int num_insns, + DisasContextBase *db, TCGOp *first_insn_start) { if (cflags & CF_USE_ICOUNT) { /* @@ -99,6 +123,23 @@ static void gen_tb_end(const TranslationBlock *tb, uint= 32_t cflags, tcgv_i32_arg(tcg_constant_i32(num_insns))); } =20 + if (tcg_ctx->exitreq_label && defer_exit_check(cflags) && + !(cflags & CF_NOIRQ)) { + if (db->needs_exit_check || tcg_ctx->exit_check_needed) { + TCGv_i32 count =3D tcg_temp_new_i32(); + TCGOp *save =3D tcg_ctx->emit_before_op; + + tcg_ctx->emit_before_op =3D first_insn_start; + tcg_gen_ld_i32(count, tcg_env, + offsetof(CPUState, neg.icount_decr.u32) - + sizeof(CPUState)); + tcg_gen_brcondi_i32(TCG_COND_LT, count, 0, tcg_ctx->exitreq_la= bel); + tcg_ctx->emit_before_op =3D save; + } else { + tcg_ctx->exitreq_label =3D NULL; + } + } + if (tcg_ctx->exitreq_label) { gen_set_label(tcg_ctx->exitreq_label); tcg_gen_exit_tb(tb, TB_EXIT_REQUESTED); @@ -117,6 +158,14 @@ bool translator_use_goto_tb(DisasContextBase *db, vadd= r dest) return false; } =20 + /* + * A destination at or below the start of this TB can close a cycle, so + * this TB must poll for interrupts. See defer_exit_check(). + */ + if (dest <=3D db->pc_first) { + db->needs_exit_check =3D true; + } + #ifdef CONFIG_USER_ONLY /* * There are no page tables in user-only mode. Every mmap, mprotect a= nd @@ -153,6 +202,7 @@ void translator_loop(CPUState *cpu, TranslationBlock *t= b, int *max_insns, db->max_insns =3D *max_insns; db->insn_start =3D NULL; db->fake_insn =3D false; + db->needs_exit_check =3D false; db->host_addr[0] =3D host_pc; db->host_addr[1] =3D NULL; db->record_start =3D 0; @@ -219,7 +269,8 @@ void translator_loop(CPUState *cpu, TranslationBlock *t= b, int *max_insns, =20 /* Emit code to exit the TB, as indicated by db->is_jmp. */ ops->tb_stop(db, cpu); - gen_tb_end(tb, cflags, icount_start_insn, db->num_insns); + gen_tb_end(tb, cflags, icount_start_insn, db->num_insns, db, + first_insn_start); =20 /* * Manage can_do_io for the translation block: set to false before diff --git ./include/exec/translator.h ./include/exec/translator.h index 978dee25ad..003926c7f0 100644 --- ./include/exec/translator.h +++ ./include/exec/translator.h @@ -74,6 +74,8 @@ struct DisasContextBase { int max_insns; bool plugin_enabled; bool fake_insn; + /* Set when this TB can be part of a control flow cycle. */ + bool needs_exit_check; uint8_t code_mmuidx; struct TCGOp *insn_start; void *host_addr[2]; diff --git ./include/tcg/tcg.h ./include/tcg/tcg.h index 7669dc1c2d..be9ce7a0e2 100644 --- ./include/tcg/tcg.h +++ ./include/tcg/tcg.h @@ -389,6 +389,8 @@ struct TCGContext { struct TCGLabelPoolData *pool_labels; =20 TCGLabel *exitreq_label; + /* Set by goto_ptr emission: destination is not known statically. */ + bool exit_check_needed; =20 #ifdef CONFIG_PLUGIN /* diff --git ./tcg/tcg-op.c ./tcg/tcg-op.c index a3efc56a9a..367e96627c 100644 --- ./tcg/tcg-op.c +++ ./tcg/tcg-op.c @@ -2616,6 +2616,7 @@ void tcg_gen_lookup_and_goto_ptr(void) return; } =20 + tcg_ctx->exit_check_needed =3D true; plugin_gen_disable_mem_helpers(); ptr =3D tcg_temp_ebb_new_ptr(); gen_helper_lookup_tb_ptr(ptr, tcg_env); @@ -2642,6 +2643,7 @@ void tcg_gen_lookup_and_goto_ptr_inline(TCGv_i64 pc, = uint32_t flags, return; } =20 + tcg_ctx->exit_check_needed =3D true; plugin_gen_disable_mem_helpers(); =20 QEMU_BUILD_BUG_ON(sizeof(((CPUJumpCache *)0)->array[0]) !=3D 16); --=20 2.54.0 From nobody Mon Sep 28 00:10:01 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=gmail.com ARC-Seal: i=1; a=rsa-sha256; t=1786993399; cv=none; d=zohomail.com; s=zohoarc; b=gD3Jyix0g0b441ibqliZr2JyHBTNr0YFiDpQDn2GllPTio+kylkWf23rI5Xry4nET3R5WX85hF3VRqUd/N0UqjeDC2CKO0U3wK+7AEW62gRMhq+ZobI0CEMJpQZ3vAeMbibVfc9KEytg+bvQooCodvTeEgoQZzDBA2QWbL9DnWk= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1786993399; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=IrAKrcFvyVNCybtfLdnOXlgFLKcEEo0o2+JIi5u2YZ8=; b=fdEc71/f/JDJNjFSNqXeFym06q1aB0zQ5AzqtrsfXDAIra0sRZ8nmhJReuSJ172Cr9FMeYj3a/K/QjBYzwiGfDaSQ6C9/ih0uML4u6dXzhlfovFxy/fNjqbWK4K8ilYvFW8Xs9beWM0L9KCvKkLhEzaVkKkVXmW1RhaBWeX6wtQ= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists1p.gnu.org (lists1p.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1786993399797561.9834785165016; Mon, 17 Aug 2026 12:03:19 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists1p.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ww2an-0000dX-Q0; Mon, 17 Aug 2026 15:01:53 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists1p.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ww2a2-0006mq-WB for qemu-devel@nongnu.org; Mon, 17 Aug 2026 15:01:15 -0400 Received: from mail-yw1-x1136.google.com ([2607:f8b0:4864:20::1136]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1ww2a0-0006m1-D9 for qemu-devel@nongnu.org; Mon, 17 Aug 2026 15:01:06 -0400 Received: by mail-yw1-x1136.google.com with SMTP id 00721157ae682-836c43641baso59944047b3.0 for ; Mon, 17 Aug 2026 12:01:03 -0700 (PDT) Received: from localhost ([2600:1702:7a90:6f9f:8bc4:8aec:108d:7a04]) by smtp.gmail.com with ESMTPSA id 00721157ae682-840f306cbdesm8186087b3.34.2026.08.17.12.01.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 17 Aug 2026 12:01:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1786993263; x=1787598063; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=IrAKrcFvyVNCybtfLdnOXlgFLKcEEo0o2+JIi5u2YZ8=; b=VvnPp96xbxMym2260URqWgOzpW8w5HYVwSufAUN686mI4WopAL4Og84wHTHvcgCRqB TOwBIMUPCWsMXjmCvFT7oCLultUQuWRY77wjQkV/Z2qWcNC4IBzQmwPG39OWYbxg3MAN OZuOtOvjHH99VwIxrh2CW9wNieg80mWjE0FaSOpNN1QN7pAaJJquLqIsuDdWfOgwtrAM ka2hf2CIceGJH+Q6LuvGh5d9lPfw/kvDMdUfRr44/OFmJWfz7BXcLPCNo9u8/YyY6yER lizA2+HrdNi3pKn1iQtYGWMg4zJtCKV15H2ZPRp0qCnR2kexWYRFFwKK8WXxlXSorsav nsfQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1786993263; x=1787598063; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=IrAKrcFvyVNCybtfLdnOXlgFLKcEEo0o2+JIi5u2YZ8=; b=kI1BT+1toZ2ZPJBKZk1Yq6CXOH7BayGM3DXJMrECeHtfl7dlGGUdilZQ+3M321nwcA 33o4FXGK+gFGVPARNVgBD1SCodF2Uvs+kz3BDxVM8FPdEKPZHbsTMBO0uFpYmEJ/DYAd zC3ZNyu2F6/11tB2WJqBCaXhSwfgEmbcgoRz2r+XKjLeibjPIbX4gx/kuUwE6twUv4YI 9NkpEtgGmASsDw/8ree//U1scLZ9VQlyp8L9hDLdcTpNCLciNauXxjvfMM5uvi2sM70Y rsYyBiUGSwOJWW/KDWyssRQ9BGmpHVFvkq6GXEyP4M//pLz2Dxkt3BoGvjYckMfwJIFi ovlw== X-Gm-Message-State: AOJu0YzDi/a80saQxpm12PUU0wzCiXDLdCnkxXM+eOVi+TLQ0XX+W64o ks9sCemer6OS0JEwcWGbO9jv5ntqOT1xyGJdiEiNx5xS9Sb/MvMTwSpTmUFp8hJgCMU= X-Gm-Gg: AR+sD13rqlEtqbqpBUvbT+2FTRrZbJG2XKKUNF+cdaY++Xa3JmT234KBdXB/FuJ5oS/ MjsUlhC2NUlEpOAdet9LMssMvHY+RkZ4L88wA3SBoQZh1G9zrojJM3TFxskhGCqpgFaPH9hAvZE B0scbAOcxf/nqcQ85W3++Sp7OxzoahJebbic0xBzZ8/kTi/uhjO/wzX/FVU+EPd0jSz0nEzB3WN QmDpSp1vAs8sM5kAPSgVv72ZHQWjbrqYJWyp/VQYGWz3edtT0+o0RvHHcis5DQ1KWuqcAFkEasM WYVqxL25GHa46m/RAGbC1seH4uNYuXnyHWU8LrfjgZh5emXg8RUzbJS9u8H1bI6psfAAoREgo2d wx0QUCXTIHfdecTez6afHPzR2lgWYjMGAxJWnJHB8hX2s5v9OLeHLV6Ii4a8G0BToT4+PUOoqNr eIn6R2paZZiqMWjFNeJJbVf0GVmzScGlzV+kh4NL8CczBbzAwYo2smtlsho+YE X-Received: by 2002:a05:690c:1b:b0:81e:8ccc:cd59 with SMTP id 00721157ae682-841304da1f5mr12547737b3.22.1786993262697; Mon, 17 Aug 2026 12:01:02 -0700 (PDT) From: Matt Turner To: qemu-devel@nongnu.org Cc: richard.henderson@linaro.org, pbonzini@redhat.com, philmd@mailo.com, zhao1.liu@intel.com, laurent@vivier.eu, deller@gmx.de, pierrick.bouvier@oss.qualcomm.com, Matt Turner Subject: [RFC PATCH 7/8] RFC: accel/tcg: poison the jump cache instead of polling for indirect exits Date: Mon, 17 Aug 2026 15:00:37 -0400 Message-ID: <20260817190038.580257-8-mattst88@gmail.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260817190038.580257-1-mattst88@gmail.com> References: <20260817190038.580257-1-mattst88@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists1p.gnu.org; Received-SPF: pass client-ip=2607:f8b0:4864:20::1136; envelope-from=mattst88@gmail.com; helo=mail-yw1-x1136.google.com X-Spam_score_int: -17 X-Spam_score: -1.8 X-Spam_bar: - X-Spam_report: (-1.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @gmail.com) X-ZM-MESSAGEID: 1786993401902158500 Content-Type: text/plain; charset="utf-8" f799aeecff dropped the icount_decr poll from blocks that cannot close a control flow cycle. What is left is dominated by blocks that end in an indirect branch: tcg_gen_lookup_and_goto_ptr{,_inline}() set exit_check_needed, because the destination is unknown at translation time and so the block might be part of a cycle. For the emulated compiler that is still 55.5% of translated blocks and 51.2% of generated-code cycles, and the two leading instructions of those blocks mov -0x10(%rbp),%ebx test %ebx,%ebx carry 5.01% of all cycles spent in generated code, measured with cycles:pp so that the figure is not just skid from the dispatch that jumped there. A block dispatching indirectly does not need to poll, because the dispatch itself can be made to notice. The out-of-line path already calls helper_lookup_tb_ptr() every time, so it only needs the helper to return the epilogue while an exit is pending. The inline probe added by 14c3e5a1f3 is the interesting case: it already loads the jump cache base from CPUState and already branches to the slow path when the entry it finds has a NULL tb. Pointing that base at a page of zeroes therefore turns every indirect dispatch into a miss, and a miss lands in the same helper. The poll becomes a pointer swap on the exit request path and costs the fast path nothing. So give the probe its own base pointer, tb_jmp_cache_probe, that nothing else reads. The two places that set icount_decr.u16.high poison it; the place that clears the flag restores it. The real tb_jmp_cache is untouched throughout, so no cache contents are lost and the recovery is one store. Blocks with a backward goto_tb edge still poll. Interrupt latency is unchanged in kind: an exit is noticed at the next cycle-closing edge, which is now either a poll or a dispatch, rather than only a poll. Measured on an x86-64 host, LTO build, on top of the preceding patches. The control was measured in the same session, because the host's all-core turbo varies by ~3% between sessions and swamps the effect otherwise: before: 810,079,619,264 instructions, 79.563s after: 787,483,360,681 instructions, 78.019s -2.79% instructions, -1.94% wall Forcing the check off entirely, which is incorrect but is the ceiling, gives -5.64% instructions and -2.55% wall. So this takes half the instructions and three quarters of the time: what it removes sits directly after an indirect branch, where the poll's dependent load was the most expensive place a poll could be. In a jitdump profile of the same workload, blocks opening with the poll fall from 55.5% to 25.9%, cycles in blocks that poll from 51.2% to 18.5%, and cycles on the two poll instructions from 5.01% to 1.50%. tests/tcg/alpha/test-indirect-irq.c is added for this: a loop whose only ba= ck edge is an indirect branch, under alarm(1). It passes before and after, and hangs if the check is simply deleted, which is what makes it a test of the new mechanism rather than of the old poll. The other alpha tests still pass and the emulated compiler still produces byte-identical output. RFC because: - The restore in cpu_handle_interrupt() races a concurrent poison from another thread. The existing barrier around icount_decr.u16.high covers it -- a poison that lands after the restore also re-set the flag, and exit_request was stored before it -- but this deserves more eyes than the single-threaded user-mode testing I have given it. - Only the inline probe needs the poison, and only alpha uses the inline probe today. Targets on the out-of-line path are covered by the helper check alone, but that has not been measured. - The shared zero-filled CPUJumpCache is a 1MB allocation that is never written. A read-only mapping would express that better. Signed-off-by: Matt Turner --- accel/tcg/cpu-exec.c | 54 +++++++++++++++++++++++++++++ accel/tcg/internal-common.h | 3 ++ accel/tcg/tcg-accel-ops.c | 2 ++ accel/tcg/translator.c | 4 +-- include/hw/core/cpu.h | 10 ++++++ include/tcg/tcg.h | 2 -- tcg/tcg-op.c | 13 +++++-- tests/tcg/alpha/Makefile.target | 3 +- tests/tcg/alpha/test-indirect-irq.c | 53 ++++++++++++++++++++++++++++ 9 files changed, 135 insertions(+), 9 deletions(-) create mode 100644 tests/tcg/alpha/test-indirect-irq.c diff --git ./accel/tcg/cpu-exec.c ./accel/tcg/cpu-exec.c index 257211235d..af466ca14e 100644 --- ./accel/tcg/cpu-exec.c +++ ./accel/tcg/cpu-exec.c @@ -388,6 +388,16 @@ const void *HELPER(lookup_tb_ptr)(CPUArchState *env) */ cpu->neg.can_do_io =3D true; =20 + /* + * A block that dispatches indirectly does not emit the icount_decr po= ll, + * so this is where a pending exit is noticed for that path: either the + * probe was poisoned and every dispatch arrives here, or the target u= ses + * the out-of-line lookup and always did. + */ + if (unlikely(cpu_loop_exit_requested(cpu))) { + return tcg_code_gen_epilogue; + } + TCGTBCPUState s =3D cpu->cc->tcg_ops->get_tb_cpu_state(cpu); s.cflags =3D curr_cflags(cpu); =20 @@ -752,6 +762,44 @@ static inline bool cpu_handle_exception(CPUState *cpu,= int *ret) return false; } =20 +/* + * The inline jump cache probe reads cpu->tb_jmp_cache_probe and takes the + * slow path when the entry it finds has a NULL tb. Pointing the probe at a + * region that is all zeroes therefore forces every indirect dispatch into + * helper_lookup_tb_ptr(), which returns to the main loop while an exit is + * pending. That is what lets a block ending in an indirect branch skip the + * icount_decr poll: the poll's job is done by a pointer swap that costs t= he + * fast path nothing. + * + * Only ever read from, and only the tb field of one entry per dispatch, so + * one shared zero-filled cache is enough for every CPU. + */ +static const CPUJumpCache *tb_jmp_cache_poison(void) +{ + static CPUJumpCache *poison; + + if (unlikely(poison =3D=3D NULL)) { + /* Raced allocations are harmless: both are all zeroes. */ + qatomic_cmpxchg(&poison, NULL, g_new0(CPUJumpCache, 1)); + } + return poison; +} + +void tcg_cpu_poison_jmp_cache(CPUState *cpu) +{ + if (qatomic_read(&cpu->tb_jmp_cache_probe) !=3D NULL) { + qatomic_set(&cpu->tb_jmp_cache_probe, + (CPUJumpCache *)tb_jmp_cache_poison()); + } +} + +void tcg_cpu_restore_jmp_cache(CPUState *cpu) +{ + if (qatomic_read(&cpu->tb_jmp_cache_probe) !=3D NULL) { + qatomic_set(&cpu->tb_jmp_cache_probe, cpu->tb_jmp_cache); + } +} + void tcg_kick_vcpu_thread(CPUState *cpu) { /* @@ -764,6 +812,9 @@ void tcg_kick_vcpu_thread(CPUState *cpu) =20 /* Ensure cpu_exec will see the exit request after TCG has exited. */ qatomic_store_release(&cpu->neg.icount_decr.u16.high, -1); + + /* Blocks that only dispatch indirectly do not poll; stop them chainin= g. */ + tcg_cpu_poison_jmp_cache(cpu); } =20 static inline bool icount_exit_request(CPUState *cpu) @@ -796,6 +847,7 @@ static inline bool cpu_handle_interrupt(CPUState *cpu, * tcg_kick_vcpu_thread()) */ qatomic_set_mb(&cpu->neg.icount_decr.u16.high, 0); + tcg_cpu_restore_jmp_cache(cpu); =20 #ifdef CONFIG_USER_ONLY assert(!cpu_test_interrupt(cpu, ~0)); @@ -1069,6 +1121,7 @@ bool tcg_exec_realizefn(CPUState *cpu, Error **errp) } =20 cpu->tb_jmp_cache =3D g_new0(CPUJumpCache, 1); + qatomic_set(&cpu->tb_jmp_cache_probe, cpu->tb_jmp_cache); tlb_init(cpu); #ifndef CONFIG_USER_ONLY tcg_iommu_init_notifier_list(cpu); @@ -1086,5 +1139,6 @@ void tcg_exec_unrealizefn(CPUState *cpu) #endif /* !CONFIG_USER_ONLY */ =20 tlb_destroy(cpu); + qatomic_set(&cpu->tb_jmp_cache_probe, NULL); g_free_rcu(cpu->tb_jmp_cache, rcu); } diff --git ./accel/tcg/internal-common.h ./accel/tcg/internal-common.h index dc713a6e1a..6007223285 100644 --- ./accel/tcg/internal-common.h +++ ./accel/tcg/internal-common.h @@ -154,6 +154,9 @@ void page_table_config_init(void); G_NORETURN void cpu_io_recompile(CPUState *cpu, uintptr_t retaddr); #endif /* CONFIG_USER_ONLY */ =20 +void tcg_cpu_poison_jmp_cache(CPUState *cpu); +void tcg_cpu_restore_jmp_cache(CPUState *cpu); + void tb_phys_invalidate(TranslationBlock *tb, tb_page_addr_t page_addr); void tb_set_jmp_target(TranslationBlock *tb, int n, uintptr_t addr); =20 diff --git ./accel/tcg/tcg-accel-ops.c ./accel/tcg/tcg-accel-ops.c index 560fe2554b..fc134c48d9 100644 --- ./accel/tcg/tcg-accel-ops.c +++ ./accel/tcg/tcg-accel-ops.c @@ -44,6 +44,7 @@ =20 #include "hw/core/cpu.h" =20 +#include "internal-common.h" #include "tcg-accel-ops.h" #include "tcg-accel-ops-mttcg.h" #include "tcg-accel-ops-rr.h" @@ -106,6 +107,7 @@ void tcg_handle_interrupt(CPUState *cpu, int mask) qemu_cpu_kick(cpu); } else { qatomic_set(&cpu->neg.icount_decr.u16.high, -1); + tcg_cpu_poison_jmp_cache(cpu); } } =20 diff --git ./accel/tcg/translator.c ./accel/tcg/translator.c index ee61dec1c6..aab2b2b1a5 100644 --- ./accel/tcg/translator.c +++ ./accel/tcg/translator.c @@ -61,8 +61,6 @@ static TCGOp *gen_tb_start(DisasContextBase *db, uint32_t= cflags) TCGv_i32 count =3D NULL; TCGOp *icount_start_insn =3D NULL; =20 - tcg_ctx->exit_check_needed =3D false; - if ((cflags & CF_USE_ICOUNT) || (!(cflags & CF_NOIRQ) && !defer_exit_check(cflags))) { count =3D tcg_temp_new_i32(); @@ -125,7 +123,7 @@ static void gen_tb_end(const TranslationBlock *tb, uint= 32_t cflags, =20 if (tcg_ctx->exitreq_label && defer_exit_check(cflags) && !(cflags & CF_NOIRQ)) { - if (db->needs_exit_check || tcg_ctx->exit_check_needed) { + if (db->needs_exit_check) { TCGv_i32 count =3D tcg_temp_new_i32(); TCGOp *save =3D tcg_ctx->emit_before_op; =20 diff --git ./include/hw/core/cpu.h ./include/hw/core/cpu.h index 172872d005..0aacd35f64 100644 --- ./include/hw/core/cpu.h +++ ./include/hw/core/cpu.h @@ -526,6 +526,16 @@ struct CPUState { =20 struct CPUJumpCache *tb_jmp_cache; =20 + /* + * What the inline jump cache probe emitted by + * tcg_gen_lookup_and_goto_ptr_inline() reads. Normally equal to + * tb_jmp_cache; pointed at a page of zeroes while an exit is pending,= so + * that every indirect dispatch misses and lands in the helper, which + * returns to the main loop. Only generated code and the two accessors= in + * cpu-exec.c may touch it. + */ + struct CPUJumpCache *tb_jmp_cache_probe; + GArray *gdb_regs; int gdb_num_regs; int gdb_num_g_regs; diff --git ./include/tcg/tcg.h ./include/tcg/tcg.h index be9ce7a0e2..7669dc1c2d 100644 --- ./include/tcg/tcg.h +++ ./include/tcg/tcg.h @@ -389,8 +389,6 @@ struct TCGContext { struct TCGLabelPoolData *pool_labels; =20 TCGLabel *exitreq_label; - /* Set by goto_ptr emission: destination is not known statically. */ - bool exit_check_needed; =20 #ifdef CONFIG_PLUGIN /* diff --git ./tcg/tcg-op.c ./tcg/tcg-op.c index 367e96627c..446a34d5ed 100644 --- ./tcg/tcg-op.c +++ ./tcg/tcg-op.c @@ -2616,7 +2616,10 @@ void tcg_gen_lookup_and_goto_ptr(void) return; } =20 - tcg_ctx->exit_check_needed =3D true; + /* + * No icount_decr poll is needed for this exit: the helper is called on + * every dispatch and returns to the main loop while an exit is pendin= g. + */ plugin_gen_disable_mem_helpers(); ptr =3D tcg_temp_ebb_new_ptr(); gen_helper_lookup_tb_ptr(ptr, tcg_env); @@ -2643,7 +2646,11 @@ void tcg_gen_lookup_and_goto_ptr_inline(TCGv_i64 pc,= uint32_t flags, return; } =20 - tcg_ctx->exit_check_needed =3D true; + /* + * No icount_decr poll is needed for this exit either. A pending exit + * poisons tb_jmp_cache_probe, so the guarded load below finds a NULL = tb, + * takes the slow path, and the helper returns to the main loop. + */ plugin_gen_disable_mem_helpers(); =20 QEMU_BUILD_BUG_ON(sizeof(((CPUJumpCache *)0)->array[0]) !=3D 16); @@ -2665,7 +2672,7 @@ void tcg_gen_lookup_and_goto_ptr_inline(TCGv_i64 pc, = uint32_t flags, tcg_gen_shli_i64(h, h, 4); =20 tcg_gen_ld_ptr(jc, tcg_env, - offsetof(CPUState, tb_jmp_cache) - sizeof(CPUState)); + offsetof(CPUState, tb_jmp_cache_probe) - sizeof(CPUStat= e)); tcg_gen_trunc_i64_ptr(ent, h); tcg_gen_add_ptr(ent, jc, ent); =20 diff --git ./tests/tcg/alpha/Makefile.target ./tests/tcg/alpha/Makefile.tar= get index eee986bab6..f9f135fc2f 100644 --- ./tests/tcg/alpha/Makefile.target +++ ./tests/tcg/alpha/Makefile.target @@ -5,7 +5,8 @@ ALPHA_SRC=3D$(SRC_PATH)/tests/tcg/alpha VPATH+=3D$(ALPHA_SRC) =20 -ALPHA_TESTS=3Dhello-alpha test-cond test-cmov test-ovf test-cvttq test-xpa= ge-chain +ALPHA_TESTS=3Dhello-alpha test-cond test-cmov test-ovf test-cvttq test-xpa= ge-chain \ + test-indirect-irq TESTS+=3D$(ALPHA_TESTS) =20 test-cmov: EXTRA_CFLAGS=3D-DTEST_CMOV diff --git ./tests/tcg/alpha/test-indirect-irq.c ./tests/tcg/alpha/test-ind= irect-irq.c new file mode 100644 index 0000000000..bef2844fd9 --- /dev/null +++ ./tests/tcg/alpha/test-indirect-irq.c @@ -0,0 +1,53 @@ +/* + * A loop whose only back edge is an indirect branch must still be + * interruptible. + * + * Blocks that dispatch indirectly do not emit the icount_decr poll; a pen= ding + * exit instead poisons the inline jump cache probe so that the dispatch f= alls + * into helper_lookup_tb_ptr(), which returns to the main loop. If that + * mechanism breaks, this program never leaves the loop and the test times + * out rather than failing an assertion. + * + * A computed goto is used deliberately: a plain while(1) closes the cycle + * with a direct backward branch, which is still polled, and so would not + * exercise the path under test. + */ +#include +#include +#include +#include +#include + +static volatile sig_atomic_t fired; +static volatile unsigned long iterations; + +static void handler(int sig) +{ + fired =3D 1; +} + +int main(void) +{ + /* + * Indexing a table with a volatile index, rather than jumping through= a + * volatile pointer: gcc happily proves a single-valued pointer consta= nt + * and emits a direct branch, which is the case this test is not about. + */ + void *target[2]; + volatile int idx =3D 0; + + assert(signal(SIGALRM, handler) !=3D SIG_ERR); + alarm(1); + + target[0] =3D &&spin; + target[1] =3D &&out; +spin: + iterations++; + if (!fired) { + goto *target[idx]; + } +out: + + printf("interrupted after %lu iterations\n", iterations); + return 0; +} --=20 2.54.0 From nobody Mon Sep 28 00:10:01 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=gmail.com ARC-Seal: i=1; a=rsa-sha256; t=1786993378; cv=none; d=zohomail.com; s=zohoarc; b=FNUCy9je4ty3Jkj6dMSCrIAk8qQ/CVdAC25Tjkc3gTqHZG1s1cSHYAloODIeAGryuGB6fdOl53XvtkWgCVYcw6l/SE7UeFfahxAb2KPQh4JutQ1L8P3ATnMJBDDY+eX5NxrWFlOH4+NNs56SU9SL3d+Tao0G285IVbeSnerfP6g= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1786993378; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=BjxyEeK8365VbqkIEoqsYg/DoU9ewvIYcbSIMSbaWL4=; b=n/Yx3Mm8a4yHVTDOas8fgyw35MzZwwYNq5zCRjJqoKsa5NVXNKmQnyn+ocy9NsxJOUgyO/9D8IAre0R96zhABP+/wWPmE7Ko5nNXHupcEX/J2sUtyEkedEEcCrbG3svD5nJBqAZResIkd+ivddJ3qRqyPs+LnWASjRGADHFqtCA= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists1p.gnu.org (lists1p.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1786993378365607.7921781782388; Mon, 17 Aug 2026 12:02:58 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists1p.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ww2bN-00031J-Sq; Mon, 17 Aug 2026 15:02:29 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists1p.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ww2a6-0006oU-3u for qemu-devel@nongnu.org; Mon, 17 Aug 2026 15:01:15 -0400 Received: from mail-yw1-x112c.google.com ([2607:f8b0:4864:20::112c]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1ww2a2-0006mL-OM for qemu-devel@nongnu.org; Mon, 17 Aug 2026 15:01:08 -0400 Received: by mail-yw1-x112c.google.com with SMTP id 00721157ae682-81f3b227a4aso55321577b3.1 for ; Mon, 17 Aug 2026 12:01:06 -0700 (PDT) Received: from localhost ([2600:1702:7a90:6f9f:8bc4:8aec:108d:7a04]) by smtp.gmail.com with ESMTPSA id 00721157ae682-840f325cce8sm8265967b3.37.2026.08.17.12.01.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 17 Aug 2026 12:01:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1786993265; x=1787598065; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=BjxyEeK8365VbqkIEoqsYg/DoU9ewvIYcbSIMSbaWL4=; b=mcFORbx64v9MNiluuHU/U/VINQwjvjU+B3NhWFS2iLiuM+p/Ok4nZNAHb5ZUgUPbCw 5vT0+QE9mWDWvEhswAn7GLswVbI6ontcXItBmjbUuJjsEiOfd8Lq21wBYwebwRywG/g6 FVmeEOLFuhzSjarI7YPr4hbj8UeMO7oALPb4rCPL7lP23SbCfOIJJvMNJRHj8iLLLkg5 rqQyv46oKl79tV8aRATyPqrGGXD/PRr7YW5WplU0TD1RiI706o1ZSitHiJ9pTshO3fyJ /7Nu3y1EphucmMo+10s2JuKx2m9A+D8ShKCKWATbdlvufPJfVTo0hg4SL2Ma1/e56Z4K 3gMA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1786993265; x=1787598065; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=BjxyEeK8365VbqkIEoqsYg/DoU9ewvIYcbSIMSbaWL4=; b=lfb/2XxAgM0wHe5FUFvZHudLRxTkBK9cfoFgo6CydxlyXBBmfLSYn5F0UZqWD7O5bE Pepq68y8ra83AsNltPRDgY1J7bdJpNItfgpnAukfByWOIcH3Of0LktMgeLsKEa82zqrS LyF6ZmWz9Le96RoYLITftrHaSVLzlnhITyLawyuJHNLcxDItfVaSbSEFgBBAzjWfr2qL 5tUGj2pzR98RMZI+PvAaGEFNiYSaB8JungFKTjKgDKJgCBqCvdjmNUdKot1lx0gID95S 1MlURee2drXSKoI03p0d/ULGehsd2JjOQ6WRImoqDVI2PkZz1lE3RYfnpovRTtdluchK 51+g== X-Gm-Message-State: AOJu0YzyZbAuYXk20YQmTr58rl5y5XRipsbBd62R8C9u25n3ZMZqnCRw rHpneDXcB/3FYPsOitpdhhFkQiVjgYgHvLUnBaP28LJ0bMa00QNUBMdiQCNnIrnyHWQ= X-Gm-Gg: AR+sD11+cihCg5IpmoMpCDUJi/JOUWAxWIz16CUrfto7pW52xqo6AZrzidYd+VmZHw5 lkJp8AUksKElStKTwsbfepLnAgi1pOjbQ6BNWss8B5tuSVCxQHGTYvsTYxdaUaqmHFYcDrw3RBl T7MLOFGCEjIG0YOsNxv0eMJsZiBmMdgde3MtelAmLjj27aWGjl80AhUU6yvyG1fzMKEWcAVtqnZ oS1wJpVs9uUW6ba8mSSvbHVy1hLL8nk3UKS8SZ++Ux6a0DCfjgTz0ojLBjehT0SZgho6Zm2gQSE QctOd4cWP0uu/xrYHiQi3TsWNQPoOuUeJGBsp+6gBq7nSEIp4uHRgQaY6bc4NKGhcTywFRX7Bp6 yMLnIQvDZ2viz9dlos9/GmdvATJEHliRigfcqNl2iIHQm+8DMKlaEuuu6UCsX8WOmw83CyI9vaj VQ0ZA9WI+15D5WQajbrguCBz4nMHrRUCWo2313/5n1rREn1/mpWlUZ7oq/3IYz X-Received: by 2002:a05:690c:c383:b0:820:dd2:3c84 with SMTP id 00721157ae682-837118357bemr82731077b3.22.1786993265354; Mon, 17 Aug 2026 12:01:05 -0700 (PDT) From: Matt Turner To: qemu-devel@nongnu.org Cc: richard.henderson@linaro.org, pbonzini@redhat.com, philmd@mailo.com, zhao1.liu@intel.com, laurent@vivier.eu, deller@gmx.de, pierrick.bouvier@oss.qualcomm.com, Matt Turner Subject: [RFC PATCH 8/8] RFC: tcg: fold a guest displacement into the host addressing mode Date: Mon, 17 Aug 2026 15:00:38 -0400 Message-ID: <20260817190038.580257-9-mattst88@gmail.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260817190038.580257-1-mattst88@gmail.com> References: <20260817190038.580257-1-mattst88@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists1p.gnu.org; Received-SPF: pass client-ip=2607:f8b0:4864:20::112c; envelope-from=mattst88@gmail.com; helo=mail-yw1-x112c.google.com X-Spam_score_int: -17 X-Spam_score: -1.8 X-Spam_bar: - X-Spam_report: (-1.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @gmail.com) X-ZM-MESSAGEID: 1786993379896158500 Content-Type: text/plain; charset="utf-8" Nothing in the TCG frontend interface can express a based memory access. tcg_gen_qemu_ld/st take an address and nothing else, so a target with a displacement in its load and store encodings -- which is most of them -- has to materialise the address first: ldq a1,8(a0) -> mov 0x80(%rbp),%rbx reload a0 lea 0x8(%rbx),%r12 address mov (%r12),%r12 the load mov %r12,0x88(%rbp) spill a1 The lea is pure loss on a host whose addressing mode has a displacement field sitting empty. It also needs a register, at the point in a block where pressure is highest. Fold it. After optimisation, look for an add of a constant immediately before a guest access, defining that access's address operand, and move the constant into a new second constant argument on the op. The add is left for liveness to remove, so nothing breaks if its result has another use. Only the immediately preceding op is examined: that is what the frontends emit, and a window of one op means the pass does not have to reason about what could have happened in between. The one thing it does check is that the add did not clobber the base it read, since the access now reads that base directly. Targets opt in with TCG_TARGET_HAS_ldst_disp and an out_disp member on TCGOutOpQemuLdSt. Without it the pass does not run, the displacement stays zero and the existing out member is called exactly as before, so no other backend changes behaviour or needs touching. For x86_64 the displacement goes in the disp32 that prepare_host_addr() already fills in for guest_base. The fold is refused unless there is no slow path at all, which means user-only -- softmmu compares the unadjusted address against the TLB -- and an access needing no alignment test, since the slow path hands addr_reg to the helper and that register no longer holds the full guest address. It is also refused if guest_base plus the displacement leaves disp32. Measured with qemu-alpha running an emulated alpha gcc 16.2.0 compiling the SQLite 3.45.1 amalgamation (255k lines, -O2) on an x86-64 host, LTO build, on top of the preceding patches, against a control measured in the same session: before: 787,483,360,681 instructions, 78.250s after: 738,020,161,466 instructions, 75.679s -6.29% instructions, -3.29% wall Emitted code shrinks from 46.29MB to 44.53MB over the run, 153.3 to 147.5 bytes per block. Per Alpha opcode, the host bytes emitted for an access fall as expected and nothing else moves: ldq 17.2 -> 14.3 ldah 16.9 -> 16.9 ldl 14.1 -> 11.6 lda 12.6 -> 12.6 stq 12.7 -> 9.7 mov 9.7 -> 9.7 The emulated compiler produces byte-identical output and the alpha tests still pass, including with a non-zero guest_base forced via -B. RFC because: - Only wired up for x86_64, and only for qemu_ld and qemu_st; the i128 qemu_ld2 and qemu_st2 pairs are left alone. - Requiring that no slow path exists is stricter than necessary. Recording the displacement in TCGLabelQemuLdst and emitting one lea on the slow path would cover alignment-checked accesses too, at no fast path cost. - Softmmu wants the displacement folded into the TLB comparison as well, which is a bigger change than this one. - A one op window catches everything the frontends emit today but is trivially defeated by anything scheduled in between. Signed-off-by: Matt Turner --- include/tcg/tcg-opc.h | 9 +++- tcg/tcg-op-ldst.c | 3 +- tcg/tcg.c | 86 ++++++++++++++++++++++++++++++++++++- tcg/x86_64/tcg-target.c.inc | 61 ++++++++++++++++++++++++++ tcg/x86_64/tcg-target.h | 3 ++ 5 files changed, 158 insertions(+), 4 deletions(-) diff --git ./include/tcg/tcg-opc.h ./include/tcg/tcg-opc.h index 61f1c28858..0a3b4330f1 100644 --- ./include/tcg/tcg-opc.h +++ ./include/tcg/tcg-opc.h @@ -118,8 +118,13 @@ DEF(goto_ptr, 0, 1, 0, TCG_OPF_BB_EXIT | TCG_OPF_BB_EN= D) DEF(plugin_cb, 0, 0, 1, TCG_OPF_NOT_PRESENT) DEF(plugin_mem_cb, 0, 1, 1, TCG_OPF_NOT_PRESENT) =20 -DEF(qemu_ld, 1, 1, 1, TCG_OPF_CALL_CLOBBER | TCG_OPF_SIDE_EFFECTS | TCG_OP= F_INT) -DEF(qemu_st, 0, 2, 1, TCG_OPF_CALL_CLOBBER | TCG_OPF_SIDE_EFFECTS | TCG_OP= F_INT) +/* + * The second constant argument is a displacement to add to the address, + * zero unless a target advertises TCG_TARGET_HAS_ldst_disp and the fold in + * fold_ldst_disp() applied. + */ +DEF(qemu_ld, 1, 1, 2, TCG_OPF_CALL_CLOBBER | TCG_OPF_SIDE_EFFECTS | TCG_OP= F_INT) +DEF(qemu_st, 0, 2, 2, TCG_OPF_CALL_CLOBBER | TCG_OPF_SIDE_EFFECTS | TCG_OP= F_INT) DEF(qemu_ld2, 2, 1, 1, TCG_OPF_CALL_CLOBBER | TCG_OPF_SIDE_EFFECTS | TCG_O= PF_INT) DEF(qemu_st2, 0, 3, 1, TCG_OPF_CALL_CLOBBER | TCG_OPF_SIDE_EFFECTS | TCG_O= PF_INT) =20 diff --git ./tcg/tcg-op-ldst.c ./tcg/tcg-op-ldst.c index 22211ccb45..ffc5e651a6 100644 --- ./tcg/tcg-op-ldst.c +++ ./tcg/tcg-op-ldst.c @@ -92,7 +92,8 @@ static MemOp tcg_canonicalize_memop(MemOp op, bool is64, = bool st) static void gen_ldst1(TCGOpcode opc, TCGType type, TCGTemp *v, TCGTemp *addr, MemOpIdx oi) { - TCGOp *op =3D tcg_gen_op3(opc, type, temp_arg(v), temp_arg(addr), oi); + /* The trailing zero is the address displacement; see fold_ldst_disp()= . */ + TCGOp *op =3D tcg_gen_op4(opc, type, temp_arg(v), temp_arg(addr), oi, = 0); TCGOP_FLAGS(op) =3D get_memop(oi) & MO_SIZE; } =20 diff --git ./tcg/tcg.c ./tcg/tcg.c index 1e77f2365a..bdd95304ea 100644 --- ./tcg/tcg.c +++ ./tcg/tcg.c @@ -1058,6 +1058,13 @@ typedef struct TCGOutOpQemuLdSt { TCGOutOp base; void (*out)(TCGContext *s, TCGType type, TCGReg dest, TCGReg addr, MemOpIdx oi); + /* + * As out(), for an access at addr + disp. Only required of targets th= at + * define TCG_TARGET_HAS_ldst_disp; for everyone else fold_ldst_disp() + * never runs and the displacement is always zero. + */ + void (*out_disp)(TCGContext *s, TCGType type, TCGReg dest, + TCGReg addr, MemOpIdx oi, int32_t disp); } TCGOutOpQemuLdSt; =20 typedef struct TCGOutOpQemuLdSt2 { @@ -3560,6 +3567,77 @@ static void move_label_uses(TCGLabel *to, TCGLabel *= from) QSIMPLEQ_CONCAT(&to->branches, &from->branches); } =20 +#ifndef TCG_TARGET_HAS_ldst_disp +#define TCG_TARGET_HAS_ldst_disp 0 +static bool tcg_target_ldst_disp_ok(TCGContext *s, MemOpIdx oi, int64_t di= sp) +{ + return false; +} +#endif + +/* + * Fold "add addr, base, $disp" into the guest access that follows it, so + * that the displacement becomes part of the host addressing mode instead = of + * a separate instruction. Frontends have no way to express this: there is + * no displacement operand on tcg_gen_qemu_ld/st, so a based access always + * costs an extra add, and an extra register to hold its result. + * + * Only an add in the op immediately before the access is recognised. That + * is what the frontends emit, and a window of one op means no analysis is + * needed of what might have happened in between. The add is left in place; + * liveness removes it if its result has no other use. + */ +static void __attribute__((noinline)) +fold_ldst_disp(TCGContext *s) +{ + TCGOp *op; + + if (!TCG_TARGET_HAS_ldst_disp) { + return; + } + + QTAILQ_FOREACH(op, &s->ops, link) { + TCGOp *prev; + TCGTemp *cts; + int64_t disp; + + switch (op->opc) { + case INDEX_op_qemu_ld: + case INDEX_op_qemu_st: + break; + default: + continue; + } + + prev =3D QTAILQ_PREV(op, link); + if (prev =3D=3D NULL || prev->opc !=3D INDEX_op_add || + TCGOP_TYPE(prev) !=3D s->addr_type) { + continue; + } + + /* + * The add must define the address operand, and must not have + * clobbered the base it read: after the fold the access reads the + * base directly, so the base has to still hold its original value. + */ + if (prev->args[0] !=3D op->args[1] || prev->args[0] =3D=3D prev->a= rgs[1]) { + continue; + } + + cts =3D arg_temp(prev->args[2]); + if (cts->kind !=3D TEMP_CONST) { + continue; + } + disp =3D cts->val; + if (disp =3D=3D 0 || !tcg_target_ldst_disp_ok(s, op->args[2], disp= )) { + continue; + } + + op->args[1] =3D prev->args[1]; + op->args[3] =3D disp; + } +} + /* Reachable analysis : remove unreachable code. */ static void __attribute__((noinline)) reachable_code_pass(TCGContext *s) @@ -5707,7 +5785,12 @@ static void tcg_reg_alloc_op(TCGContext *s, const TC= GOp *op) const TCGOutOpQemuLdSt *out =3D container_of(all_outop[op->opc], TCGOutOpQemuLdSt, base); =20 - out->out(s, type, new_args[0], new_args[1], new_args[2]); + if (new_args[3]) { + out->out_disp(s, type, new_args[0], new_args[1], + new_args[2], new_args[3]); + } else { + out->out(s, type, new_args[0], new_args[1], new_args[2]); + } } break; =20 @@ -6590,6 +6673,7 @@ int tcg_gen_code(TCGContext *s, TranslationBlock *tb,= uint64_t pc_start) tcg_temp_ebb_reset_freed(s); =20 tcg_optimize(s); + fold_ldst_disp(s); =20 reachable_code_pass(s); liveness_pass_0(s); diff --git ./tcg/x86_64/tcg-target.c.inc ./tcg/x86_64/tcg-target.c.inc index 1fc45e4ec6..d3bde91816 100644 --- ./tcg/x86_64/tcg-target.c.inc +++ ./tcg/x86_64/tcg-target.c.inc @@ -2015,6 +2015,39 @@ static TCGLabelQemuLdst *prepare_host_addr(TCGContex= t *s, HostAddress *h, return ldst; } =20 +/* + * Whether the displacement of a guest access can be folded into the host + * addressing mode rather than materialised by a separate lea. + * + * Folding rewrites the access to use base + disp, so nothing may need a + * register holding the complete guest address. The softmmu TLB comparison + * does, and so does any slow path, which hands addr_reg to the helper. In + * user-only mode prepare_host_addr() creates a slow path only for an + * alignment test, so requiring that none is needed rules it out. What is + * left to check is guest_base, which shares the disp32 field. + */ +static bool tcg_target_ldst_disp_ok(TCGContext *s, MemOpIdx oi, int64_t di= sp) +{ +#ifdef CONFIG_USER_ONLY + MemOp opc =3D get_memop(oi); + TCGAtomAlign aa; + int64_t ofs; + + if (tcg_use_softmmu || s->addr_type !=3D TCG_TYPE_I64) { + return false; + } + aa =3D atom_and_align_for_opc(s, opc, MO_ATOM_WITHIN16, + (opc & MO_SIZE) =3D=3D MO_128); + if (aa.align) { + return false; + } + ofs =3D (int64_t)x86_guest_base.ofs + disp; + return ofs =3D=3D (int32_t)ofs; +#else + return false; +#endif +} + static void tcg_out_qemu_ld_direct(TCGContext *s, TCGReg datalo, TCGReg da= tahi, HostAddress h, TCGType type, MemOp memo= p) { @@ -2171,9 +2204,23 @@ static void tgen_qemu_ld(TCGContext *s, TCGType type= , TCGReg data, } } =20 +static void tgen_qemu_ld_disp(TCGContext *s, TCGType type, TCGReg data, + TCGReg addr, MemOpIdx oi, int32_t disp) +{ + TCGLabelQemuLdst *ldst; + HostAddress h; + + ldst =3D prepare_host_addr(s, &h, addr, oi, true); + /* tcg_target_ldst_disp_ok() has ruled out every slow path. */ + tcg_debug_assert(ldst =3D=3D NULL); + h.ofs +=3D disp; + tcg_out_qemu_ld_direct(s, data, -1, h, type, get_memop(oi)); +} + static const TCGOutOpQemuLdSt outop_qemu_ld =3D { .base.static_constraint =3D C_O1_I1(r, L), .out =3D tgen_qemu_ld, + .out_disp =3D tgen_qemu_ld_disp, }; =20 static void tgen_qemu_ld2(TCGContext *s, TCGType type, TCGReg datalo, @@ -2309,9 +2356,23 @@ static void tgen_qemu_st(TCGContext *s, TCGType type= , TCGReg data, } } =20 +static void tgen_qemu_st_disp(TCGContext *s, TCGType type, TCGReg data, + TCGReg addr, MemOpIdx oi, int32_t disp) +{ + TCGLabelQemuLdst *ldst; + HostAddress h; + + ldst =3D prepare_host_addr(s, &h, addr, oi, false); + /* tcg_target_ldst_disp_ok() has ruled out every slow path. */ + tcg_debug_assert(ldst =3D=3D NULL); + h.ofs +=3D disp; + tcg_out_qemu_st_direct(s, data, -1, h, get_memop(oi)); +} + static const TCGOutOpQemuLdSt outop_qemu_st =3D { .base.static_constraint =3D C_O0_I2(L, L), .out =3D tgen_qemu_st, + .out_disp =3D tgen_qemu_st_disp, }; =20 static void tgen_qemu_st2(TCGContext *s, TCGType type, TCGReg datalo, diff --git ./tcg/x86_64/tcg-target.h ./tcg/x86_64/tcg-target.h index 7ebae56a7d..8f2315c15e 100644 --- ./tcg/x86_64/tcg-target.h +++ ./tcg/x86_64/tcg-target.h @@ -30,6 +30,9 @@ #define TCG_TARGET_NB_REGS 32 #define MAX_CODE_GEN_BUFFER_SIZE (2 * GiB) =20 +/* A guest displacement can go in the disp32 of the addressing mode. */ +#define TCG_TARGET_HAS_ldst_disp 1 + typedef enum { TCG_REG_EAX =3D 0, TCG_REG_ECX, --=20 2.54.0