From nobody Mon Sep 28 00:51:17 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=gmail.com ARC-Seal: i=1; a=rsa-sha256; t=1787075044; cv=none; d=zohomail.com; s=zohoarc; b=II+hPHIDaq2P7hGb4pWygjGf0crwEtajSDcUrPVHAescuVwpqQwXtPo9sXVwHC+O8qWRA/qvCaP7Q2btG/e+v4MUfiYXfcr/iRc7HcCIIWKzqBMHc9MRMdju/9JntiH9c3Pswz5G0izwI6Hv6sqxHfrlctBdLuENMvB8MWtzBrI= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1787075044; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=KmUKXBRtZXE0gks8E242wAGgapbeBjl/3Us4eEXqmQI=; b=Wu/xLwT18MF4w7peOQydiREqhUojrQRXhcgZimEEpD3cK2/v9FpDaAkxDPPUPqVaFLKFJ//G6sdeyNrnltQLnrFpfo0ENw2m4QFAC2LE+A+t+ZNgdO2Vy5gtq94VlvOS3tspalag3VCGASFjXHn3Z4j9lVGDXn1vr/z+U+AFRtI= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists1p.gnu.org (lists1p.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1787075044706575.3956158201697; Tue, 18 Aug 2026 10:44:04 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists1p.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1wwNq4-0002sw-4L; Tue, 18 Aug 2026 13:43:04 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists1p.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1wwNq2-0002sN-Ju for qemu-devel@nongnu.org; Tue, 18 Aug 2026 13:43:02 -0400 Received: from mail-yx1-xb12c.google.com ([2607:f8b0:4864:20::b12c]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1wwNq0-0007vW-2v for qemu-devel@nongnu.org; Tue, 18 Aug 2026 13:43:02 -0400 Received: by mail-yx1-xb12c.google.com with SMTP id 956f58d0204a3-66c4e9769bdso299717d50.0 for ; Tue, 18 Aug 2026 10:42:59 -0700 (PDT) Received: from localhost ([2600:1702:7a90:6f9f:8bc4:8aec:108d:7a04]) by smtp.gmail.com with ESMTPSA id 00721157ae682-84068c23cb2sm25249267b3.13.2026.08.18.10.42.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Aug 2026 10:42:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1787074979; x=1787679779; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=KmUKXBRtZXE0gks8E242wAGgapbeBjl/3Us4eEXqmQI=; b=WFE/jkM2dRXRVliyBDezJmF9bw1aOb8JlZv8MuraKkHfyTvQNaHYMZLCo9K28ravSl Ypsgd8ObFV0+3uUMfeeElnaFCePsobQrx0vqC0zk2FEs5vPUoEO7fXGPD6PwdNiGqrHr fTqi/c1Y82R7Ajc5R/GQ/lEcsYFrEmb6slAb6/YeGAEhh+G6csOABWmNEzvqRQmALjGd ptbtT4VZMrHuxhj8oSvcfgJr7haGcVLiUkQMHYJVf/GHL/g57zRVSB8YVi0wo1MVLOli RhCetyTDIrsjj1JdhLVRAbGHbpCTF+0Q6vsYoCM4Fm+aE9aPLn4C1h6zLcMDsqbCqaJ5 pSHw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787074979; x=1787679779; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=KmUKXBRtZXE0gks8E242wAGgapbeBjl/3Us4eEXqmQI=; b=Iay8TSVWk698ZtLmnDMt7lz4T3N0KeSvzblhLH3QwbvDyVdzEnDCU04u8iMDtKfG/Z RfHSGrAxDhOqhXF80dSqCgMf6WjO4nQMsHXGwFpC+5w0yALwZxKdKMtOCE0fuTRQvDtm eTM2t+NR2AuarSq8iqfxIPB7Ai2Z0vk9Kpg0pXf7rtcR24R3sxnYCqVRV0X3qMsQnedX F4OICyyJnR6Hb0r3onPcsWEvhRaUfWeuGrrSbTUi5lGW/vcjFTx7r2/3+Ru1lRkaJzbJ APu+FCMd8xZciy9eX8v1aUjwqqezZHU3vod13GmT48906Tl0W/UMgdf/U52tSad/sww5 wlyA== X-Gm-Message-State: AOJu0Yyj1VRXjGIllZo+e7ZL7sG3u5wlvRF4Fjc85WXqwEEVHW2YmRup SO1aICyXn3eHpyUzswhFSC1+Brqp4xRLeAEyFX4C0kfyzc7Ropuzbk9Owb3wIenLlrM= X-Gm-Gg: AR+sD122TYJfrOGWcqZxZd79p4FXC4yUJcoEYF2Q/rGp3ddlhvrpWw13BZryCRpIjCo VBApcefHUKt1yo+1JRj9rs+PgTZ8S+Wi6WqFHGpjsUAN7RC+0+l4z6cYNBU0izGRp3nCpoIjg6Y QJAxtemO6Re4SzB/pHRPGkFA/6OuxQi9ErhNlNxIzQx54ySBs7pevN2KfVwjOTFZHPlv1wwaaPJ //7uSsR7b+WuwniahwC5Fo42NcAVjGblSnyf2XgJ7XM3JjTmxOtkf9jaNMSIOH6DaSZifyRqxWq DTlISqXvLTdXL4fzaNw12pjaF+STCxPLlkgLbf8wRgU4tNmZq3mN7RDJZI5TlmT/H6ixDRo+Vv/ YZBHZ32dz1xs4qLKamUdsQRGtoIDgGWcI0OsoRtvKZ6ajO8cNFtB1r3iKU4C9jyXfdrHBOss4Ys p/+u1ATEaR5STIoMFA9LlP19Ej9bm+mcHBE9fSCLcp05kLattTDApyEEVndlph X-Received: by 2002:a05:690e:408a:b0:667:770c:f88e with SMTP id 956f58d0204a3-66c72d6ab0dmr12764902d50.45.1787074978764; Tue, 18 Aug 2026 10:42:58 -0700 (PDT) From: Matt Turner To: qemu-devel@nongnu.org Cc: richard.henderson@linaro.org, pbonzini@redhat.com, philmd@mailo.com, zhao1.liu@intel.com, laurent@vivier.eu, deller@gmx.de, pierrick.bouvier@oss.qualcomm.com, Matt Turner Subject: [PATCH 1/8] accel/tcg: cache the result of curr_cflags() Date: Tue, 18 Aug 2026 13:42:40 -0400 Message-ID: <20260818174247.649526-2-mattst88@gmail.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260818174247.649526-1-mattst88@gmail.com> References: <20260818174247.649526-1-mattst88@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists1p.gnu.org; Received-SPF: pass client-ip=2607:f8b0:4864:20::b12c; envelope-from=mattst88@gmail.com; helo=mail-yx1-xb12c.google.com X-Spam_score_int: -17 X-Spam_score: -1.8 X-Spam_bar: - X-Spam_report: (-1.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @gmail.com) X-ZM-MESSAGEID: 1787075046051158500 Content-Type: text/plain; charset="utf-8" curr_cflags() is called once per TB dispatch, from helper_lookup_tb_ptr() and from the cpu_exec() loop. It recomputes the same value every time: uint32_t cflags =3D cpu->tcg_cflags; if (unlikely(cpu_single_stepping(cpu))) { ... } else if (qatomic_read(&one_insn_per_tb)) { ... } else if (qemu_loglevel_mask(CPU_LOG_TB_NOCHAIN)) { ... } That is three loads and three branches on the hottest path in the interpreter, for state that changes only when gdb enables single-step, when one-insn-per-tb is toggled, or when the log mask changes. Compute the value once into CPUState::tcg_curr_cflags and recompute it from the four places that can change an input: tcg_cflags_set(), cpu_single_step(), tcg_set_one_insn_per_tb() and qemu_set_log_internal(). curr_cflags() becomes a single load. Measured with qemu-alpha running an emulated alpha gcc 16.2.0 compiling the SQLite 3.45.1 amalgamation (255k lines, -O2) on an x86-64 host, in a build configured with --enable-lto: before: 1,647,901,588,726 instructions after: 1,563,829,403,943 instructions -5.10% That workload issues 8.4 billion dispatches, so the per-call saving is small but the aggregate is not. The emulated compiler produces byte-identical output before and after. Wall clock does not move: 133.57s to 133.13s, a 0.33% difference against a run-to-run spread of the same size. The removed work is a few predictable loads and branches that the host executes largely in parallel with the surrounding dispatch, so this patch is worth taking for the instruction count and for what it enables, not for a time saving that can be measured on its own. Note that tcg_set_one_insn_per_tb() does not tb_flush(), so the cache cannot piggyback on TB flushing and needs its own update call. Caching makes the pre-computed cflags one half of a pair that has to be kept in step, and nothing in C enforces that. The cost of getting it wrong is not a crash but silently wrong code generation: a stale cache that is missing CF_PARALLEL makes TCG emit the non-atomic form of guest atomics, and the guest then corrupts its own mutexes. linux-user's cpu_copy() is exactly such a trap. do_fork() calls begin_parallel_context() on the parent before cpu_copy(), so the child inherits CF_PARALLEL and never calls tcg_cflags_set() itself; assigning the field directly would leave every cloned thread dispatching with a cflags of zero. Close the hole from both ends. Name the field tcg_cflags_priv and add tcg_cflags_get(), so that tcg_cflags_has()/get()/set() are the only ways to reach it. C cannot really make a struct member private, but an open-coded access now fails to compile rather than quietly going stale, which is enough to force a rebased or newly written user to look at the accessors. target/alpha's CF_PCREL setup is converted along with it: it would be benign either way today, because tcg_cpu_init_cflags() refreshes the cache afterwards in system mode, but it is the same pattern and only ordering saved it. Then have curr_cflags() recompute the value and compare, under CONFIG_DEBUG_TCG. That is the check that catches a missed update on the first dispatch, rather than days later by way of corrupted guest mutexes. It cannot be unconditional, as recomputing on every dispatch is the very cost the cache exists to avoid. Verified by dropping the tcg_cflags_set() call from cpu_copy() against a debug-tcg build: the assert fires immediately, reporting the cached and recomputed values and the bits that differ. Signed-off-by: Matt Turner --- accel/tcg/cpu-exec-common.c | 48 +++++++++++++++++++++++++++++--- accel/tcg/internal-common.h | 19 ++++++++++++- accel/tcg/tcg-all.c | 1 + cpu-target.c | 3 ++ include/exec/translation-block.h | 6 ++++ include/hw/core/cpu.h | 13 +++++++-- include/system/tcg.h | 9 ++++++ linux-user/main.c | 2 +- stubs/meson.build | 1 + stubs/tcg-cflags.c | 16 +++++++++++ target/alpha/cpu.c | 2 +- util/log.c | 4 +++ 12 files changed, 114 insertions(+), 10 deletions(-) create mode 100644 stubs/tcg-cflags.c diff --git ./accel/tcg/cpu-exec-common.c ./accel/tcg/cpu-exec-common.c index 44e84344f3..c75fbd344d 100644 --- ./accel/tcg/cpu-exec-common.c +++ ./accel/tcg/cpu-exec-common.c @@ -28,17 +28,23 @@ bool tcg_allowed; =20 bool tcg_cflags_has(CPUState *cpu, uint32_t flags) { - return cpu->tcg_cflags & flags; + return cpu->tcg_cflags_priv & flags; +} + +uint32_t tcg_cflags_get(CPUState *cpu) +{ + return cpu->tcg_cflags_priv; } =20 void tcg_cflags_set(CPUState *cpu, uint32_t flags) { - cpu->tcg_cflags |=3D flags; + cpu->tcg_cflags_priv |=3D flags; + tcg_update_curr_cflags(cpu); } =20 -uint32_t curr_cflags(CPUState *cpu) +static uint32_t compute_curr_cflags(CPUState *cpu) { - uint32_t cflags =3D cpu->tcg_cflags; + uint32_t cflags =3D cpu->tcg_cflags_priv; =20 /* * Record gdb single-step. We should be exiting the TB by raising @@ -58,6 +64,40 @@ uint32_t curr_cflags(CPUState *cpu) return cflags; } =20 +void tcg_update_curr_cflags(CPUState *cpu) +{ + cpu->tcg_curr_cflags =3D compute_curr_cflags(cpu); +} + +void tcg_update_all_curr_cflags(void) +{ + CPUState *cpu; + + CPU_FOREACH(cpu) { + tcg_update_curr_cflags(cpu); + } +} + +#ifdef CONFIG_DEBUG_TCG +/* + * Catch a cached value that has gone stale because an input changed witho= ut + * a matching tcg_update_curr_cflags(). Called from curr_cflags() on the + * dispatch path, so it exists only in debug-tcg builds. + */ +void tcg_assert_curr_cflags(CPUState *cpu) +{ + uint32_t cached =3D cpu->tcg_curr_cflags; + uint32_t fresh =3D compute_curr_cflags(cpu); + + if (unlikely(cached !=3D fresh)) { + fprintf(stderr, "stale tcg_curr_cflags on CPU %d: " + "cached 0x%08x, recomputed 0x%08x (differ in 0x%08x)\n", + cpu->cpu_index, cached, fresh, cached ^ fresh); + g_assert_not_reached(); + } +} +#endif + /* exit the current TB, but without causing any exception to be raised */ void cpu_loop_exit_noexc(CPUState *cpu) { diff --git ./accel/tcg/internal-common.h ./accel/tcg/internal-common.h index 9e7be2d78d..dc713a6e1a 100644 --- ./accel/tcg/internal-common.h +++ ./accel/tcg/internal-common.h @@ -70,7 +70,24 @@ bool tcg_exec_realizefn(CPUState *cpu, Error **errp); void tcg_exec_unrealizefn(CPUState *cpu); =20 /* current cflags for hashing/comparison */ -uint32_t curr_cflags(CPUState *cpu); +/* + * Cached by tcg_update_curr_cflags(). This is on the hot TB dispatch + * path, so it must stay a single load; see commit message. A debug-tcg + * build pays for a recompute here to prove the cache is still in step, + * which turns a missed update into a loud failure rather than subtly + * wrong code generation. + */ +#ifdef CONFIG_DEBUG_TCG +void tcg_assert_curr_cflags(CPUState *cpu); +#endif + +static inline uint32_t curr_cflags(CPUState *cpu) +{ +#ifdef CONFIG_DEBUG_TCG + tcg_assert_curr_cflags(cpu); +#endif + return cpu->tcg_curr_cflags; +} =20 void tb_check_watchpoint(CPUState *cpu, uintptr_t retaddr); =20 diff --git ./accel/tcg/tcg-all.c ./accel/tcg/tcg-all.c index 7186c10cf0..8f892f580f 100644 --- ./accel/tcg/tcg-all.c +++ ./accel/tcg/tcg-all.c @@ -254,6 +254,7 @@ static void tcg_set_one_insn_per_tb(Object *obj, bool v= alue, Error **errp) s->one_insn_per_tb =3D value; /* Set the global also: this changes the behaviour */ qatomic_set(&one_insn_per_tb, value); + tcg_update_all_curr_cflags(); } =20 static void tcg_accel_class_init(ObjectClass *oc, const void *data) diff --git ./cpu-target.c ./cpu-target.c index 4783845c9b..9affbcd9c5 100644 --- ./cpu-target.c +++ ./cpu-target.c @@ -24,6 +24,7 @@ #include "exec/replay-core.h" #include "exec/log.h" #include "hw/core/cpu.h" +#include "system/tcg.h" #include "trace/trace-root.h" =20 /* enable or disable single step mode. EXCP_DEBUG is returned by the @@ -35,6 +36,8 @@ void cpu_single_step(CPUState *cpu, unsigned flags) cpu->singlestep_flags, flags); cpu->singlestep_flags =3D flags; =20 + tcg_update_curr_cflags(cpu); + #if !defined(CONFIG_USER_ONLY) const AccelOpsClass *ops =3D cpus_get_accel(); if (ops->update_guest_debug) { diff --git ./include/exec/translation-block.h ./include/exec/translation-bl= ock.h index 40cc699031..ed2ce87503 100644 --- ./include/exec/translation-block.h +++ ./include/exec/translation-block.h @@ -158,7 +158,13 @@ static inline uint32_t tb_cflags(const TranslationBloc= k *tb) return qatomic_read(&tb->cflags); } =20 +/* + * CPUState::tcg_cflags_priv is reached only through these. The setter ke= eps + * the derived CPUState::tcg_curr_cflags in step, and assigning the field + * directly would silently leave that cache stale. + */ bool tcg_cflags_has(CPUState *cpu, uint32_t flags); +uint32_t tcg_cflags_get(CPUState *cpu); void tcg_cflags_set(CPUState *cpu, uint32_t flags); =20 static inline tb_page_addr_t tb_page_addr0(const TranslationBlock *tb) diff --git ./include/hw/core/cpu.h ./include/hw/core/cpu.h index 81af7b9ee1..399ac7bd57 100644 --- ./include/hw/core/cpu.h +++ ./include/hw/core/cpu.h @@ -411,10 +411,16 @@ struct qemu_work_item; * to a cluster this will be UNASSIGNED_CLUSTER_INDEX; otherwise it will * be the same as the cluster-id property of the CPU object's TYPE_CPU_C= LUSTER * QOM parent. - * Under TCG this value is propagated to @tcg_cflags. + * Under TCG this value is propagated to @tcg_cflags_priv. * See TranslationBlock::TCG CF_CLUSTER_MASK. * @start_powered_off: Indicates whether the CPU starts in powered-off sta= te. - * @tcg_cflags: Pre-computed cflags for this cpu. + * @tcg_cflags_priv: Pre-computed cflags for this cpu. Private to + * tcg_cflags_has() and tcg_cflags_set(): @tcg_curr_cflags is derived fr= om + * it and is refreshed by the setter, so a direct assignment here would + * leave the two out of step. The name is deliberately awkward to make = an + * open-coded access fail to compile rather than silently go stale. + * @tcg_curr_cflags: Cached result of curr_cflags(), recomputed by + * tcg_update_curr_cflags() whenever any of its inputs change. * @nr_threads: Number of threads within this CPU core. * @thread: Host thread details, only live once @created is #true * @sem: WIN32 only semaphore used only for qtest @@ -557,7 +563,8 @@ struct CPUState { /* TODO Move common fields from CPUArchState here. */ int cpu_index; int cluster_index; - uint32_t tcg_cflags; + uint32_t tcg_cflags_priv; + uint32_t tcg_curr_cflags; uint32_t halted; int32_t exception_index; =20 diff --git ./include/system/tcg.h ./include/system/tcg.h index 7622dcea30..f41e6b3219 100644 --- ./include/system/tcg.h +++ ./include/system/tcg.h @@ -17,6 +17,15 @@ extern bool tcg_allowed; #define tcg_enabled() 0 #endif =20 +/* + * Recompute CPUState::tcg_curr_cflags. Must be called whenever any input + * to the computation changes: CPUState::tcg_cflags_priv, gdb single-step + * state, one-insn-per-tb, or the CPU_LOG_TB_NOCHAIN log flag. The first = of + * those is covered already, tcg_cflags_set() being the only way to change= it. + */ +void tcg_update_curr_cflags(CPUState *cpu); +void tcg_update_all_curr_cflags(void); + /** * qemu_tcg_mttcg_enabled: * Check whether we are running MultiThread TCG or not. diff --git ./linux-user/main.c ./linux-user/main.c index 60a695b7ca..ba04773398 100644 --- ./linux-user/main.c +++ ./linux-user/main.c @@ -242,7 +242,7 @@ CPUArchState *cpu_copy(CPUArchState *env) /* Reset non arch specific state */ cpu_reset(new_cpu); =20 - new_cpu->tcg_cflags =3D cpu->tcg_cflags; + tcg_cflags_set(new_cpu, tcg_cflags_get(cpu)); memcpy(new_env, env, sizeof(CPUArchState)); #if defined(TARGET_I386) || defined(TARGET_X86_64) new_env->gdt.base =3D target_mmap(0, sizeof(uint64_t) * TARGET_GDT_ENT= RIES, diff --git ./stubs/meson.build ./stubs/meson.build index 3b2f2680b1..0025e79226 100644 --- ./stubs/meson.build +++ ./stubs/meson.build @@ -3,6 +3,7 @@ # below, so that it is clear who needs the stubbed functionality. =20 stub_ss.add(files('cpu-get-clock.c')) +stub_ss.add(files('tcg-cflags.c')) stub_ss.add(files('fdset.c')) stub_ss.add(files('iothread-lock.c')) stub_ss.add(files('is-daemonized.c')) diff --git ./stubs/tcg-cflags.c ./stubs/tcg-cflags.c new file mode 100644 index 0000000000..bd74fabf0e --- /dev/null +++ ./stubs/tcg-cflags.c @@ -0,0 +1,16 @@ +/* + * Stubs for the cached cflags update hooks, for binaries that link + * util/log.c or cpu-target.c without linking TCG. + * + * SPDX-License-Identifier: GPL-2.0-or-later + */ +#include "qemu/osdep.h" +#include "system/tcg.h" + +void tcg_update_curr_cflags(CPUState *cpu) +{ +} + +void tcg_update_all_curr_cflags(void) +{ +} diff --git ./target/alpha/cpu.c ./target/alpha/cpu.c index 12e8602166..7a768eae9f 100644 --- ./target/alpha/cpu.c +++ ./target/alpha/cpu.c @@ -114,7 +114,7 @@ static void alpha_cpu_realizefn(DeviceState *dev, Error= **errp) =20 #ifndef CONFIG_USER_ONLY /* Use pc-relative instructions in system-mode */ - cs->tcg_cflags |=3D CF_PCREL; + tcg_cflags_set(cs, CF_PCREL); #endif =20 cpu_common_realize(cs, &local_err); diff --git ./util/log.c ./util/log.c index 7cffbc1bf8..62c7f09609 100644 --- ./util/log.c +++ ./util/log.c @@ -27,6 +27,7 @@ #include "qemu/thread.h" #include "qemu/lockable.h" #include "qemu/rcu.h" +#include "system/tcg.h" #ifdef CONFIG_LINUX #include #endif @@ -301,6 +302,9 @@ static bool qemu_set_log_internal(const char *filename,= bool changed_name, #endif qemu_loglevel =3D log_flags; =20 + /* CPU_LOG_TB_NOCHAIN feeds into the per-CPU cached cflags. */ + tcg_update_all_curr_cflags(); + daemonized =3D is_daemonized(); need_to_open_file =3D false; if (!daemonized) { --=20 2.54.0 From nobody Mon Sep 28 00:51:17 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=gmail.com ARC-Seal: i=1; a=rsa-sha256; t=1787075066; cv=none; d=zohomail.com; s=zohoarc; b=RdAQ7vyCXpj8Xq3rNYAaM08qlHyvixrQlegtVSNpzDZRZKQDeFV1lAnwxS0Eu0LZGs44FVLxN4SnbcBBoMChSq3zVokXVESrUtX04t+kBDHOLiUublgLxYsCBYyd+h+lDHvfLKodoxwbA7D63Adp93GHDlhs8JKSe44YGzbZDZg= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1787075066; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=WM409keD7J89yEAgYdhBWa8aMMqXlnNUvLTfJiZti+8=; b=PSRB2GZdHLOrS6tiuit6bCr1I8Zarxbn1nzxpR9F8ieLGd4XS+QEa1GoxJlyGHPN8xFNQm4atRvJpfl5nFsm6GuRbKBB9EMmcoqYeZ4NapjWwCLirT7Iop6Aj5BCOPYg4D74WTLdqc+UhjH+6INqppB+Lrp4wvCLQkygRM/KKJk= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists1p.gnu.org (lists1p.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1787075066238469.4473921886697; Tue, 18 Aug 2026 10:44:26 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists1p.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1wwNq5-0002tS-74; Tue, 18 Aug 2026 13:43:05 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists1p.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1wwNq3-0002sn-JE for qemu-devel@nongnu.org; Tue, 18 Aug 2026 13:43:03 -0400 Received: from mail-yw1-x112e.google.com ([2607:f8b0:4864:20::112e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1wwNq2-0007vv-3B for qemu-devel@nongnu.org; Tue, 18 Aug 2026 13:43:03 -0400 Received: by mail-yw1-x112e.google.com with SMTP id 00721157ae682-836c436a6b3so3001627b3.0 for ; Tue, 18 Aug 2026 10:43:01 -0700 (PDT) Received: from localhost ([2600:1702:7a90:6f9f:8bc4:8aec:108d:7a04]) by smtp.gmail.com with ESMTPSA id 00721157ae682-84068a3b800sm24907847b3.12.2026.08.18.10.42.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Aug 2026 10:42:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1787074980; x=1787679780; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=WM409keD7J89yEAgYdhBWa8aMMqXlnNUvLTfJiZti+8=; b=ftxhQEbFYU6nUPnMbqhb/s1yUFg0S+7mR2yvFkATzcsFCUtNVEn/HJM4ce8gjchaAh hZxIJ4ZkHlvGnS5qCS2+w0JZEuyd/j2TmN1WtVcd07BEhUC+cBLlKYos2JUj3bkduy/t ITbOagJLeRoFW/FdatU+7VDkpYkSZCNp33m2gA59OygPkKsZgSjCr6hScv65K86nkIqU a1Jui5pVCu/KnphHTiqSQFUQl+Z08NjlXliF2HebiV24sGvYO97Be14QOkpGzCTALxC/ 2edsSlyZZACGNk4cc19zNYvCXtdE14dJKrxVItoYBzSlmtY9/WAfQ2k9E9LsYlSWXkEC hF5Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787074980; x=1787679780; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=WM409keD7J89yEAgYdhBWa8aMMqXlnNUvLTfJiZti+8=; b=Eoe4sMzVaEjRrKMn1PZ45qVdx0FT19USkwwv3r5BzKh6mkzty7lJ2NsHmegj4lDXcH 3iiFsn5Cb7tz4nfswnjsRqC3ENNXAXyBl4j6md8/kbn+JcAFBV+Kl4yrU5T7t/PtH0TA E7Lca+VDbvDpZzmF1IcP/b9ASqLE0NrKWt0qNzEu/AY1gCEKWXmiGpbsx+vwvJWT5At+ uhLWe+3w0MruMYIRdhjxHanXT1Ar5flaYwblSdEu6z6RvMtTcIxqDbyLU60P1JzfeoRH CPcsFz2DOuPiX9jnL4zIUe6HFBfzpLeRmzdD1vYHW6iiFQhPTdiBae8bKNk66o06yOgE fV6A== X-Gm-Message-State: AOJu0Yxttm4d6JjS0+U6w2PjnJUKW/A4P/0XV20gnydj0zH0ljYTlsvm nJwH0FyybU87hJi/jPrgZCXQF58qAvTurZtv7hnS6yOsBPRaHinWdRCCHryxFy0X3oU= X-Gm-Gg: AR+sD11p3gGIoTAs7b5Gg81jzi/Fun2gTc7jKzlnFeWnjIU8hrU6hFmNTvNfW45RqKS kHDAn31+JccHHjKI9wIFKwicNnhRO65vvauximK9JM4vm/dCQDotR5ZVOewdhWxEoGZgo6dTZFZ Avgl848kuPxHe/CmZGYQmqn0ZT7xxlPjp0CANs/VXenEnOA08KUaCAWORtcuBAHaaXJi6AQtAo+ wqSfSMwng+UfD5HirM+9q0+U5zClY8wZPfx+P9sVKoa3aDWXQzHbR/9n43x85pOGLdNkq/JZrjz P2h5CccJygAye7cCu3N0eNXLH4XXoCTihY4Rb6v5CZHzgyD9OHFRB0G79oTxoXV/KxVV0vOM6df 9YdGn/xGKULGq3Lda0V30E/tSsphedEyTA+CubM/RpUJJBJuJkWHfACO/3Ui12em741Roy6TCw9 2AIFuIbJ1ulh8CiRUPLiU8l6VxPCWi3KpctnCigv1HYe/iu1KSa85oGbCT1jM3 X-Received: by 2002:a05:690c:26c2:b0:814:5f5b:6378 with SMTP id 00721157ae682-8370dfb3aedmr146926227b3.12.1787074980454; Tue, 18 Aug 2026 10:43:00 -0700 (PDT) From: Matt Turner To: qemu-devel@nongnu.org Cc: richard.henderson@linaro.org, pbonzini@redhat.com, philmd@mailo.com, zhao1.liu@intel.com, laurent@vivier.eu, deller@gmx.de, pierrick.bouvier@oss.qualcomm.com, Matt Turner Subject: [PATCH 2/8] accel/tcg: enlarge the TB jump cache to 64K entries Date: Tue, 18 Aug 2026 13:42:41 -0400 Message-ID: <20260818174247.649526-3-mattst88@gmail.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260818174247.649526-1-mattst88@gmail.com> References: <20260818174247.649526-1-mattst88@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists1p.gnu.org; Received-SPF: pass client-ip=2607:f8b0:4864:20::112e; envelope-from=mattst88@gmail.com; helo=mail-yw1-x112e.google.com X-Spam_score_int: -17 X-Spam_score: -1.8 X-Spam_bar: - X-Spam_report: (-1.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @gmail.com) X-ZM-MESSAGEID: 1787075067859158500 Content-Type: text/plain; charset="utf-8" The per-CPU TB jump cache has held 4096 entries since it was introduced. That is too small for guests running large programs: an emulated compiler misses often enough that the fallback qht lookup shows up prominently in a profile. Measured with qemu-alpha running an emulated alpha gcc 16.2.0 compiling the SQLite 3.45.1 amalgamation (255k lines, -O2) on an x86-64 host. The compile performs 34.2 billion TB executions, of which 8.4 billion take the indirect dispatch path. Sizing curve, on top of the preceding patch, instructions retired and wall clock: 12 bits ( 64 KiB): 1,563,829,403,943 133.13s 14 bits ( 256 KiB): 1,493,865,985,972 -4.47% 124.89s -6.19% 16 bits ( 1 MiB): 1,469,729,281,442 -6.02% 120.97s -9.13% 18 bits ( 4 MiB): 1,462,262,363,257 -6.49% 120.16s -9.74% 16 bits is the knee. 18 buys another 0.47% of instructions for four times the memory, and since instructions retired does not account for the data cache pressure of a 4 MiB table, that 0.47% is probably not real: the wall clock difference between 16 and 18 bits is 0.67%, against a run-to-run spread of the same order. In a perf profile the mechanism is visible directly: tb_htable_lookup(), which is where qht_lookup_custom() lands once it is inlined in an LTO build, falls from 5.73% of samples to 1.52%. The cost is memory: the cache grows from 64 KiB to 1 MiB per vCPU. That is easy to justify for a single-vCPU linux-user process and less obvious for system emulation with many vCPUs, so this may want to be sized by target or made tunable rather than raised unconditionally. Signed-off-by: Matt Turner --- accel/tcg/tb-jmp-cache.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git ./accel/tcg/tb-jmp-cache.h ./accel/tcg/tb-jmp-cache.h index c3a505e394..268dacd7ba 100644 --- ./accel/tcg/tb-jmp-cache.h +++ ./accel/tcg/tb-jmp-cache.h @@ -12,7 +12,7 @@ #include "qemu/rcu.h" #include "exec/cpu-common.h" =20 -#define TB_JMP_CACHE_BITS 12 +#define TB_JMP_CACHE_BITS 16 #define TB_JMP_CACHE_SIZE (1 << TB_JMP_CACHE_BITS) =20 /* --=20 2.54.0 From nobody Mon Sep 28 00:51:17 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=gmail.com ARC-Seal: i=1; a=rsa-sha256; t=1787075050; cv=none; d=zohomail.com; s=zohoarc; b=bPRXg2YYzbwIfP3LGlQlsk42r7jSXXOAi3FRQBVw9yi/OAfXtAEnVgedvhbTKETrnifhpQ7ZbT8YflTjJXHKbPblpt6alelJ0cG1wE3zOWWPOhapM6qvq46s/ghw+DxwAOGT15heeZfCwEqtmf2XeiEc/g7bRZQKbZOelKgImTo= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1787075050; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=0rVJQgX3N6MU4TQ/rSCCOxj3fDdMcQCXO6e891ulN60=; b=eCXk/bZzMlDc7m3OeFF7xnxPoELNpRo48wc2aICFRX1uO3ljAChoQlr+xGg66QZJ24LOKBZlIvR1mowLK1OznC2w+Pe2OoTfpRXn2hLQfqjN7iysVPGjd8/4dCJwPL0eoE0bbm7wEI94oSWTpFX343OSsG1NxZChJu1HWi8yNnQ= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists1p.gnu.org (lists1p.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1787075050877569.7587456867764; Tue, 18 Aug 2026 10:44:10 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists1p.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1wwNq7-0002u8-3r; Tue, 18 Aug 2026 13:43:07 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists1p.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1wwNq4-0002sx-Ol for qemu-devel@nongnu.org; Tue, 18 Aug 2026 13:43:04 -0400 Received: from mail-yw1-x112f.google.com ([2607:f8b0:4864:20::112f]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1wwNq3-0007wI-9c for qemu-devel@nongnu.org; Tue, 18 Aug 2026 13:43:04 -0400 Received: by mail-yw1-x112f.google.com with SMTP id 00721157ae682-841f1dfc30fso14888937b3.1 for ; Tue, 18 Aug 2026 10:43:02 -0700 (PDT) Received: from localhost ([2600:1702:7a90:6f9f:8bc4:8aec:108d:7a04]) by smtp.gmail.com with ESMTPSA id 956f58d0204a3-66cb47ee907sm2659066d50.21.2026.08.18.10.43.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Aug 2026 10:43:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1787074982; x=1787679782; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=0rVJQgX3N6MU4TQ/rSCCOxj3fDdMcQCXO6e891ulN60=; b=hO7/E+JlwPHQOsQTO19WObemiA3DBQo9vBFD7VBTpLs7lZSzR1jEsGbV4EESjP8G0H HqZX5CaIpSKsZghmtR/hGF+Nl4bvaryI7eVUG5WI1ljvtT7SgLxVo3IHbVBx+aTggYoq pglDx3IfBKOY0TPr6DVjPPoE9wEVxxByqjSqIYKeHjri0vaHCoXWA/1xBjvO0GK64xDq ZTiERoDGPa1dKXAHcGMxnxNacWHQgpgYbfdWDoUspxbfhe3V6W5Cs7T2Iy2pWEwfRQXI tDuWUkGXDZ9VUC+bg+Fk4wl0kpsbPpkmTO1uQ0RNxxVpgD5Rq9wVqWxszJ9ToHi75FT0 kJQg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787074982; x=1787679782; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=0rVJQgX3N6MU4TQ/rSCCOxj3fDdMcQCXO6e891ulN60=; b=srOGhR4NHZsCjXWB/BfETsNGxNf3A5msELptBNnzpWFJkLJLpsVJuWOZEXHjKSbpq9 /l8kjVunQ1gKpMBTIPdvMZTmJWcd/WTDAnuNrmku1WYLKMfacn/BUH+tiMQUpvt/qnHr BiTXZY4KEfjvj1N8YNafde136e+8NJVNTDrqLMjUen9x8OiLdFFpKDWbt4VEork6aTwl HkP0eegpeUFRoLbKQdOHLUKBb8HpgiYLWJGZXYF33RIiAe0iu6VU3Otyr6BQcVoD//Go hCpGiV9gLoSMGVWg6eHPb4MIGDxC8DZjNFnO+5bwY1RjMVqRyFIJuzPiydEYKAyMrTNf 60Mg== X-Gm-Message-State: AOJu0YxmMv9wetzRqRsBntTCevhWHk3pSREifOuhU1vrjmLwQISo3tqo D61jESEVcE7rpZt02T1tFuBTp9s56z/fmZeURKi2EwCyMfR17hZFZ0RLS8mc+yC9ES8= X-Gm-Gg: AR+sD10AGnOADU3924U+a0dA33bi45iE/I10bkfUndT4BuTZhkrprQLQBziXE+PlHng eboNNboQd2NFA4hBNJURkit8XJxIVS6aGcJ1Vo/Ip2za6soHhp4IRwvX2eBoTKWSMueDpRDbgoM dZvDomn2ODrrO6OtIx4znJDa2M6TOpkh4D4WnXt+VRu7NdY3ju4YlkrPN0+BTOhzPfYTguMHXWc p20+tdTazlEfHDSylJ8QmDOFwcEn5z/cFXGE8NjsLdW9+ROK8wNdeMUV7djsd0Lo6/gAsMkaG1a O07zF2Vj5tCb1jO/99fw8hqr+jQSV38K0SWqBkgL/3pqd6hd/HpM6VTOy0xTNKjxJQsFicGO7fi O/+8hc1qjVGdhxQxkP5u1a1h6XFaZTQ/TGYXIYQwr6J3ky0WXiWra31heo/CjFu+TIxEs6Mlg4y VY2WtHAvrerYinGgb/k1iGPsYhi2VQvR+KISqw7K5UsqkBGl+MM2/p3eI07qeM X-Received: by 2002:a05:690e:4403:b0:668:14a7:d00d with SMTP id 956f58d0204a3-66cbc0da051mr2190665d50.2.1787074982026; Tue, 18 Aug 2026 10:43:02 -0700 (PDT) From: Matt Turner To: qemu-devel@nongnu.org Cc: richard.henderson@linaro.org, pbonzini@redhat.com, philmd@mailo.com, zhao1.liu@intel.com, laurent@vivier.eu, deller@gmx.de, pierrick.bouvier@oss.qualcomm.com, Matt Turner Subject: [PATCH 3/8] accel/tcg: skip the can_do_io stores in user-only builds Date: Tue, 18 Aug 2026 13:42:42 -0400 Message-ID: <20260818174247.649526-4-mattst88@gmail.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260818174247.649526-1-mattst88@gmail.com> References: <20260818174247.649526-1-mattst88@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists1p.gnu.org; Received-SPF: pass client-ip=2607:f8b0:4864:20::112f; envelope-from=mattst88@gmail.com; helo=mail-yw1-x112f.google.com X-Spam_score_int: -17 X-Spam_score: -1.8 X-Spam_bar: - X-Spam_report: (-1.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @gmail.com) X-ZM-MESSAGEID: 1787075051892158500 Content-Type: text/plain; charset="utf-8" Every translation block stores to cpu->neg.can_do_io twice: false before the first instruction, true before the last one. Nothing reads it in a user-only build. There is no memory-mapped I/O in linux-user, and every reader is in system_ss: cputlb.c, watchpoint.c, icount-common.c and tcg-accel-ops-icount.c. Two stores per TB is not much on its own, but TBs are short. An emulated alpha gcc 16.2.0 compiling the SQLite 3.45.1 amalgamation (255k lines, -O2) executes 34.2 billion TBs at 6.04 guest instructions each, so this is 68 billion stores for nothing. Measured on an x86-64 host, LTO build, on top of the preceding two patches: before: 1,469,729,281,442 instructions after: 1,402,816,253,499 instructions -4.55% before: 120.97s wall clock after: 115.75s wall clock -4.32% The emulated compiler produces byte-identical output. Signed-off-by: Matt Turner Reviewed-by: Philippe Mathieu-Daud=C3=A9 Reviewed-by: Richard Henderson --- accel/tcg/translator.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git ./accel/tcg/translator.c ./accel/tcg/translator.c index cd7d079fe0..85bb21e911 100644 --- ./accel/tcg/translator.c +++ ./accel/tcg/translator.c @@ -23,6 +23,9 @@ =20 static void set_can_do_io(DisasContextBase *db, bool val) { + if (IS_ENABLED(CONFIG_USER_ONLY)) { + return; + } QEMU_BUILD_BUG_ON(sizeof_field(CPUState, neg.can_do_io) !=3D 1); tcg_gen_st8_i32(tcg_constant_i32(val), tcg_env, offsetof(CPUState, neg.can_do_io) - sizeof(CPUState)); @@ -210,6 +213,10 @@ void translator_loop(CPUState *cpu, TranslationBlock *= tb, int *max_insns, /* * Manage can_do_io for the translation block: set to false before * the first insn and set to true before the last insn. + * + * Nothing reads can_do_io in user-only builds. There is no MMIO + * there, and every reader (cputlb.c, watchpoint.c, icount) is in + * system_ss, so skip the two stores per TB entirely. */ if (db->num_insns =3D=3D 1) { tcg_debug_assert(first_insn_start =3D=3D db->insn_start); --=20 2.54.0 From nobody Mon Sep 28 00:51:17 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=gmail.com ARC-Seal: i=1; a=rsa-sha256; t=1787075086; cv=none; d=zohomail.com; s=zohoarc; b=L2iF7z++Bo+Tbw+IE31fbvnrz1M9D99U789fjD9RhEgaOamGA2J2iJd6jbDNEkYwBZjA8DqWXe3dI8vg2UXtuVGDJ0YSUw8QjKlH3nCY7Z7abvzLFldaXmHtRhbTLODYO7D0p9t/wekmhvLJNMVXb5aHIYNWdgNs0kyJDFbNb3U= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1787075086; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=8MikWi2ZkuQ48w4BbQpRaAViV4sVN9NaKK78tQU/xZ8=; b=H2w50grm4PjEcXn+kspFPnarHO8dkGFkt0G8BsTcSm2+cDwhgkATGwso9VdCIB4dl1S9XBhGaYSPyzGrwc1I0xZETBBnx/fO/V76fgSzU41lO2dN0LBwxlVRvuCkSny0Ysl2t7DDqFComvkJiXuRgcX7nVCHwWflJJHp61d6cF0= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists1p.gnu.org (lists1p.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1787075086968336.44297721082035; Tue, 18 Aug 2026 10:44:46 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists1p.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1wwNqH-0002vE-Ks; Tue, 18 Aug 2026 13:43:17 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists1p.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1wwNqE-0002uc-TQ for qemu-devel@nongnu.org; Tue, 18 Aug 2026 13:43:15 -0400 Received: from mail-yw1-x112c.google.com ([2607:f8b0:4864:20::112c]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1wwNq6-0007xR-9P for qemu-devel@nongnu.org; Tue, 18 Aug 2026 13:43:09 -0400 Received: by mail-yw1-x112c.google.com with SMTP id 00721157ae682-836c718715fso2777087b3.0 for ; Tue, 18 Aug 2026 10:43:05 -0700 (PDT) Received: from localhost ([2600:1702:7a90:6f9f:8bc4:8aec:108d:7a04]) by smtp.gmail.com with ESMTPSA id 00721157ae682-84068d1eafasm25220817b3.19.2026.08.18.10.43.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Aug 2026 10:43:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1787074985; x=1787679785; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=8MikWi2ZkuQ48w4BbQpRaAViV4sVN9NaKK78tQU/xZ8=; b=ZrVkZwidaHvKC2vDo+GiMtwkMDcTcWyEjFXB4voP7D1E61pGnmouikk+WnX8xEefct /NR8oVncMXORnSObL4eeSzMGPE3D+ddFCQQC7EjKMBR+d5jwn8e36q/ZlkzT3DhOXqEU 2Vs6QIRSlSUaSxlox4/w6/V82drUBJxV/XfissD58/1/O2IJn8R7VJE9eEFgaSLKlARQ S0Wh8IZgKF73FxZ9kvKP2X6AxuhxDHgn+c2CQggcMe2SSnTu26jDqqaqYstv4xqpsQkv 5kk/boRscRZLMTotYAKy2pihXX7sCMDWoRdvWsUVLMUo3EizIXEzVuP5uxdmZ3xkmrWx BbHw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787074985; x=1787679785; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=8MikWi2ZkuQ48w4BbQpRaAViV4sVN9NaKK78tQU/xZ8=; b=n1lsWw9rN65GTZ6uUWfC3gXYssaesKcR4+mWp4ktqjEP0OJTU3f8ual97sHskWwYgX +PrwKJXbFASjDqRvGNLubjeWqrN1k4mn5303JG5CSaF8OvFQcxOJAR55723pR2jOWykN 3S+/vcbW2Qd+Y4v1lkc0dyJGlMhGCh+KBfxa6AsKzEsLWagrMCCid6pUgH/Fn5MwoXxu 9v7qunWBWmScNWlkHmo8ViD0rKNa3dRkBmVp4XW19KmtTM+0jxJftQxnboRZjTtH7hkB L2XFcU1ghl+zjCnTp5VxDN5/CU7YebLDWmQOl05atBGZnlYC2fQ69YVW3E7IsJ4FAekj TH2Q== X-Gm-Message-State: AOJu0YzXAB2HyOtge7CuL70VZdraGhNYKUFkzBjpq9aMNKp+CJxNJmt2 jQ/Qmho/4f0Rcv2Mpj5D+FebwN72JVQmYUlS4K8rskyWCCFjZoOyyQbeoxeL4B/yTb4= X-Gm-Gg: AR+sD10adPuBQxzlvUC7cFE/bPUojUEwOebfiiG4mjjWsTmFvW56jF7dIdy+pNEFCDP UbOe1X3459yKAWR1DQ/wttFJuIN5fr+qsKBw/5hpyjZys8tOSUmpGL/nOmlYRpLCnUBM9EVvb34 P+QGP+1FqE0aXv5gTx1aJthWeKciQEh6TQMsys+p0I7gsxjykdnawu4t4uot7UWaV76xGn1ug9a 5tonSay361MqMx8Rzhhi3mJridNCEiIZTd2M2CzRT709aNat1TDIrCU0S74rgEDtvF/N4sNRnX0 DGdjQGiC6FQmSAZQk9/PKERQNkDM4e/MkWVVsWmGwX1nPHhmKfgYP+yL1JGwqOH05d74yvCHj6z PxelmlIWOnBAaBOmdwW0840oRwnz++Q2eWCEC29oJzjVNidlHd+YY6wErQAaghO1jX2lHWRgDqL Dl2YlzdpHOSAJY7XP8FjM5JqPuYvCZuBfjO0778NPmQW3sRcdF2x5GN1jaX637CIO5YY5KtC7i X-Received: by 2002:a05:690c:103:b0:81d:bc5:4624 with SMTP id 00721157ae682-83711e15d36mr125378377b3.25.1787074984961; Tue, 18 Aug 2026 10:43:04 -0700 (PDT) From: Matt Turner To: qemu-devel@nongnu.org Cc: richard.henderson@linaro.org, pbonzini@redhat.com, philmd@mailo.com, zhao1.liu@intel.com, laurent@vivier.eu, deller@gmx.de, pierrick.bouvier@oss.qualcomm.com, Matt Turner Subject: [PATCH 4/8] RFC: tcg: probe the TB jump cache inline instead of calling a helper Date: Tue, 18 Aug 2026 13:42:43 -0400 Message-ID: <20260818174247.649526-5-mattst88@gmail.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260818174247.649526-1-mattst88@gmail.com> References: <20260818174247.649526-1-mattst88@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists1p.gnu.org; Received-SPF: pass client-ip=2607:f8b0:4864:20::112c; envelope-from=mattst88@gmail.com; helo=mail-yw1-x112c.google.com X-Spam_score_int: -17 X-Spam_score: -1.8 X-Spam_bar: - X-Spam_report: (-1.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @gmail.com) X-ZM-MESSAGEID: 1787075088271158500 Content-Type: text/plain; charset="utf-8" Every indirect branch that cannot use goto_tb ends in tcg_gen_lookup_and_goto_ptr(), which calls helper_lookup_tb_ptr(). For an emulated compiler that is 8.4 billion helper calls in a single translation unit: 24.6% of all TB exits take this path, because jsr/ret/jmp have a register destination and because goto_tb is restricted to same-page targets. The helper itself is already tight, but each call pays for a call frame, the can_do_io store, the get_tb_cpu_state() indirect call through TCGCPUOps, curr_cflags(), and a breakpoint check, before it gets to the jump cache probe that almost always hits (95.8% for this workload). Emit the probe inline instead. The destination PC is already in a TCG temp, and the flags and cflags the destination must match are constants at translation time, so the fast path is a hash, three guarded loads and a goto_ptr. Only a miss calls the helper, which still owns filling the cache. Two details matter for the generated code. The flags and cflags guards are folded into a single aligned 64-bit load and compare, since the fields are adjacent. And each path emits its own goto_ptr rather than branching to a shared one: a temp live across the label is spilled and reloaded on every dispatch, which cost 6.3% on its own. Measured with qemu-alpha running an emulated alpha gcc 16.2.0 compiling the SQLite 3.45.1 amalgamation (255k lines, -O2) on an x86-64 host, LTO build, on top of the preceding three patches: before: 1,402,816,253,499 instructions after: 890,713,633,237 instructions -36.51% before: 115.75s wall clock after: 84.44s wall clock -27.05% The gap between the two is the point at which this stops being a straight-line win: the helper call was highly predictable work that the host pipelined well, so removing it retires far fewer instructions than it saves time. IPC falls from 2.48 to 2.15 across this patch for that reason. Despite emitting more code, this also reduces instruction cache pressure, because a dispatch no longer jumps into qemu's .text and evicts translated code: before: 11,476,318,964 L1-icache-load-misses after: 6,990,186,701 L1-icache-load-misses -39.1% The mechanism is visible directly in a profile: helper_lookup_tb_ptr() falls from 30.97% of samples to 0.42%, and qemu's own .text falls from 38.9% to 5.4%, with the balance moving into generated code. Combined with the three preceding patches, against an unmodified LTO build, 1,647,901,588,726 instructions fall to 890,713,633,237, or -45.95%. The emulated compiler produces byte-identical output throughout. Open issues, hence RFC: - The flags/cflags guards use the *current* TB's values as constants. That assumes the CPU flags feeding get_tb_cpu_state() cannot change within a TB, and that curr_cflags() cannot change under a running TB (gdb attaching to enable single-step would). Both need to be established or the values need to be loaded at runtime. - tcg/tcg-op.c has no business including accel/tcg/tb-jmp-cache.h or knowing the CPUJumpCache layout. The probe likely belongs in accel/tcg with a small emit helper exported from tcg/. - The jump cache entry is read without qatomic_read(); entries are invalidated concurrently by setting tb to NULL. - Only wired up for alpha so far, and only for 64-bit guest PCs. Signed-off-by: Matt Turner --- include/tcg/tcg-op-common.h | 2 + target/alpha/translate.c | 4 +- tcg/tcg-op.c | 79 +++++++++++++++++++++++++++++++++++++ 3 files changed, 83 insertions(+), 2 deletions(-) diff --git ./include/tcg/tcg-op-common.h ./include/tcg/tcg-op-common.h index 1fe342db0d..65084c38b9 100644 --- ./include/tcg/tcg-op-common.h +++ ./include/tcg/tcg-op-common.h @@ -84,6 +84,8 @@ void tcg_gen_goto_tb(unsigned idx); * this op is equivalent to calling tcg_gen_exit_tb() with 0 as the argume= nt. */ void tcg_gen_lookup_and_goto_ptr(void); +void tcg_gen_lookup_and_goto_ptr_inline(TCGv_i64 pc, + const TranslationBlock *tb); =20 void tcg_gen_plugin_cb(unsigned from); void tcg_gen_plugin_mem_cb(TCGv_i64 addr, unsigned meminfo); diff --git ./target/alpha/translate.c ./target/alpha/translate.c index c66e3f9c14..939170a28b 100644 --- ./target/alpha/translate.c +++ ./target/alpha/translate.c @@ -449,7 +449,7 @@ static void gen_goto_tb(DisasContext *ctx, unsigned tb_= slot_idx, int32_t disp) tcg_gen_exit_tb(ctx->base.tb, tb_slot_idx); } else { gen_pc_disp(ctx, cpu_pc, disp); - tcg_gen_lookup_and_goto_ptr(); + tcg_gen_lookup_and_goto_ptr_inline(cpu_pc, ctx->base.tb); } } =20 @@ -2917,7 +2917,7 @@ static void alpha_tr_tb_stop(DisasContextBase *dcbase= , CPUState *cpu) gen_pc_disp(ctx, cpu_pc, 0); /* FALLTHRU */ case DISAS_PC_UPDATED: - tcg_gen_lookup_and_goto_ptr(); + tcg_gen_lookup_and_goto_ptr_inline(cpu_pc, ctx->base.tb); break; case DISAS_PC_UPDATED_NOCHAIN: tcg_gen_exit_tb(NULL, 0); diff --git ./tcg/tcg-op.c ./tcg/tcg-op.c index bbcb510c76..ab8d101871 100644 --- ./tcg/tcg-op.c +++ ./tcg/tcg-op.c @@ -28,6 +28,8 @@ #include "tcg/tcg-op-common.h" #include "exec/translation-block.h" #include "exec/plugin-gen.h" +#include "hw/core/cpu.h" +#include "../accel/tcg/tb-jmp-cache.h" #include "tcg-internal.h" #include "tcg-has.h" =20 @@ -2620,3 +2622,80 @@ void tcg_gen_lookup_and_goto_ptr(void) tcg_gen_op1i(INDEX_op_goto_ptr, TCG_TYPE_PTR, tcgv_ptr_arg(ptr)); tcg_temp_free_ptr(ptr); } + +/* + * As tcg_gen_lookup_and_goto_ptr(), but probe the TB jump cache inline + * instead of calling helper_lookup_tb_ptr() unconditionally. @pc must + * hold the destination guest PC; @flags and @cflags are the values the + * destination TB must have been translated with. + */ +void tcg_gen_lookup_and_goto_ptr_inline(TCGv_i64 pc, + const TranslationBlock *tb) +{ + uint32_t flags =3D tb->flags; + uint32_t cflags =3D tb->cflags; + TCGv_ptr jc, ent, tbp, ptr; + TCGv_i64 h, tmp; + TCGLabel *slow; + uint64_t fpair; + + if (tcg_ctx->gen_tb->cflags & CF_NO_GOTO_PTR) { + tcg_gen_exit_tb(NULL, 0); + return; + } + + plugin_gen_disable_mem_helpers(); + + QEMU_BUILD_BUG_ON(sizeof(((CPUJumpCache *)0)->array[0]) !=3D 16); + QEMU_BUILD_BUG_ON(offsetof(TranslationBlock, cflags) !=3D + offsetof(TranslationBlock, flags) + 4); + + jc =3D tcg_temp_ebb_new_ptr(); + ent =3D tcg_temp_ebb_new_ptr(); + tbp =3D tcg_temp_ebb_new_ptr(); + ptr =3D tcg_temp_ebb_new_ptr(); + h =3D tcg_temp_ebb_new_i64(); + tmp =3D tcg_temp_ebb_new_i64(); + slow =3D gen_new_label(); + + /* h =3D tb_jmp_cache_hash_func(pc) * sizeof(array[0]) */ + tcg_gen_shri_i64(h, pc, TB_JMP_CACHE_BITS); + tcg_gen_xor_i64(h, h, pc); + tcg_gen_andi_i64(h, h, TB_JMP_CACHE_SIZE - 1); + tcg_gen_shli_i64(h, h, 4); + + tcg_gen_ld_ptr(jc, tcg_env, + offsetof(CPUState, tb_jmp_cache) - sizeof(CPUState)); + tcg_gen_trunc_i64_ptr(ent, h); + tcg_gen_add_ptr(ent, jc, ent); + + tcg_gen_ld_ptr(tbp, ent, offsetof(CPUJumpCache, array[0].tb)); + tcg_gen_brcondi_ptr(TCG_COND_EQ, tbp, 0, slow); + + tcg_gen_ld_i64(tmp, ent, offsetof(CPUJumpCache, array[0].pc)); + tcg_gen_brcond_i64(TCG_COND_NE, tmp, pc, slow); + + /* + * flags and cflags are adjacent uint32_t, so one aligned 64-bit load + * and compare covers both. + */ +#if HOST_BIG_ENDIAN + fpair =3D ((uint64_t)flags << 32) | cflags; +#else + fpair =3D ((uint64_t)cflags << 32) | flags; +#endif + tcg_gen_ld_i64(tmp, tbp, offsetof(TranslationBlock, flags)); + tcg_gen_brcondi_i64(TCG_COND_NE, tmp, fpair, slow); + + tcg_gen_ld_ptr(ptr, tbp, offsetof(TranslationBlock, tc.ptr)); + tcg_gen_op1i(INDEX_op_goto_ptr, TCG_TYPE_PTR, tcgv_ptr_arg(ptr)); + + /* + * Emit a second goto_ptr rather than branching to a shared one: a temp + * live across the label would be spilled and reloaded on every dispat= ch. + */ + gen_set_label(slow); + ptr =3D tcg_temp_ebb_new_ptr(); + gen_helper_lookup_tb_ptr(ptr, tcg_env); + tcg_gen_op1i(INDEX_op_goto_ptr, TCG_TYPE_PTR, tcgv_ptr_arg(ptr)); +} --=20 2.54.0 From nobody Mon Sep 28 00:51:17 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=gmail.com ARC-Seal: i=1; a=rsa-sha256; t=1787075039; cv=none; d=zohomail.com; s=zohoarc; b=iXC2sHO1Xxg0Tvk9PYG7KjRAKQmyz8SqTjxQZeCuWwbp0REpj9AZwGTyOHVpxrFXQkTAe3QTSCW5UIitSLeh4Uwz1ckl+rKplZD/WwzWONOMJECodErAT9nKF724kwIsGvraRGINgkQ/Z3Vj5Kkdsal/v016bW71zmP0CjLNzCM= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1787075039; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=FD6Ur+eg4ui02rsiunbrWHJ0I/7Mgp+cJLNtmLWUAGs=; b=BYxozqK5DGiRri26ORT2P8J6QpCMpURBWZTxJuENKU4CTZnHW72g70Fp65BZXWhDQgnxU82sAWioVmAeMLMUgkww9bEQtZoaRdw/t/yy/8/lgI5HrdB0WJfkdePuKgDTaN2A2m/n6f26FWZeFeRvEIg88Q62OW34Dr9/p2wopsw= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists1p.gnu.org (lists1p.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1787075039252639.829432103061; Tue, 18 Aug 2026 10:43:59 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists1p.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1wwNqN-0002vd-KK; Tue, 18 Aug 2026 13:43:23 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists1p.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1wwNqE-0002ub-T0 for qemu-devel@nongnu.org; Tue, 18 Aug 2026 13:43:15 -0400 Received: from mail-yx1-xb12d.google.com ([2607:f8b0:4864:20::b12d]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1wwNq7-0007yk-TH for qemu-devel@nongnu.org; Tue, 18 Aug 2026 13:43:09 -0400 Received: by mail-yx1-xb12d.google.com with SMTP id 956f58d0204a3-66807ba2f0fso312532d50.3 for ; Tue, 18 Aug 2026 10:43:07 -0700 (PDT) Received: from localhost ([2600:1702:7a90:6f9f:8bc4:8aec:108d:7a04]) by smtp.gmail.com with ESMTPSA id 956f58d0204a3-66cb47c8cc5sm2682003d50.18.2026.08.18.10.43.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Aug 2026 10:43:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1787074986; x=1787679786; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=FD6Ur+eg4ui02rsiunbrWHJ0I/7Mgp+cJLNtmLWUAGs=; b=IXoy0YGTYsi45pJ++J46BeM0RcKT4fgIdG66obP86Umkl9TomV80qQaehuTp4EJS4C N5nZwmXhh4a46JcS+F4XjoFT0rG5j7dsm4qe5b+NILM/2P99FQMckBHuYme12pOeFRyz j7ymJ/ay4I4LN3fCwMYIpymniS+PpM+vKwCpxhB45sTL7Rj23V4sI13njej5CLkVkG6t ftHJmD5C/abKDhysAWGowXwbBY4zsd415UszTp4cqi41a3xmfBHbdcioe2JTNb3nLoaE gLb2fjwenO5XBb4erMQYDkTeCQclFIyHrqdl6cDUwWY9ohrIf1hQBq5p2YNvcBiiWKxf W5iQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787074986; x=1787679786; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=FD6Ur+eg4ui02rsiunbrWHJ0I/7Mgp+cJLNtmLWUAGs=; b=mCK1f38CQNVEyvmpGiPynIYdUjAIclzft9lxAClb8E+B+nzSJKT6p7uV+8GS5fOIN4 +Ke6EZBJzmh1yAzMeJlac3mm+mKrcFeQy/n/WTmId/mCs7gPzPJ26B9vBb7X3gI68Zpx MUZjSS1TcpMLXBjyNKeDkph064Bkp7Z6niZxFMzut+8Jjym2nz3jt2s5fZu5YZNxwxik Qulzz7vjifjl3A15nPkpuwxZhXFVa/tzKT1lmQVb/nuw8o3GUzhuBTXbpAopqmo0bUw+ QcU8RbI0nBgf//oPDXgZ0t9a3/Gg6qupxJub9NyegQ4t5XR/eGIZl3oJpqEQUd2khIGC AqAA== X-Gm-Message-State: AOJu0YwCYwTEtpwfP9JEkTvBPFUErJShJ7+7jAqxK9cUmB2WfewwkuZR X2v3x8eSis2fLac78xdNw1cN/qaynJUuBsWrYkgHNNXnenYLS9SDD4+cQwK07w== X-Gm-Gg: AR+sD10QA5xYfviv+xGQMOaH6O64YczuOE3f/2AKw3Pexy7iVBKVb3FF+nnt0henvgE 9jJRKXbeXaOwOXxfE64doC+RCSodVhRND+dX1n/pZEgdxHhVFNmXsrcCGYieXK+wbSWlPfkE9LW JwqG1YRGNxVJVU7Xky9IIDCxZaWlxhEEHzIO+ftRvCmBHM63zC1H7pgJ/5TBXFPPCWTfc+/b2qD pV86EafJ3WZmXaVwX6+gk9YLZyuzQqhOfKzFFbCKgJ5Dwz2aN59YvOMD3RHWXHrL08sC6aE6X2t cwihjWVpCQr/Hwvk+27wLrqP2S8ajV5l9c2F/Iyt0Y9nV5vZoffkTg4BiZIxJkMXHI3yFMLDjbZ L0IiaUL3kAu/Y0YmKv7Aepq+TxJId75F/rAudEio36CjcH+0yq8t/JOeud8+QYSSOh0gwVE6sVg WShgfUO3JicMabTXvXNWv/3iD4j4YdYEYGi6ileoqfRw7916PEPROGzdW58WY9 X-Received: by 2002:a05:690e:1441:b0:66b:2fe9:cd2 with SMTP id 956f58d0204a3-66c72d2a100mr12487608d50.41.1787074986470; Tue, 18 Aug 2026 10:43:06 -0700 (PDT) From: Matt Turner To: qemu-devel@nongnu.org Cc: richard.henderson@linaro.org, pbonzini@redhat.com, philmd@mailo.com, zhao1.liu@intel.com, laurent@vivier.eu, deller@gmx.de, pierrick.bouvier@oss.qualcomm.com, Matt Turner Subject: [PATCH 5/8] RFC: accel/tcg: allow cross-page goto_tb chaining in user-only builds Date: Tue, 18 Aug 2026 13:42:44 -0400 Message-ID: <20260818174247.649526-6-mattst88@gmail.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260818174247.649526-1-mattst88@gmail.com> References: <20260818174247.649526-1-mattst88@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists1p.gnu.org; Received-SPF: pass client-ip=2607:f8b0:4864:20::b12d; envelope-from=mattst88@gmail.com; helo=mail-yx1-xb12d.google.com X-Spam_score_int: -17 X-Spam_score: -1.8 X-Spam_bar: - X-Spam_report: (-1.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @gmail.com) X-ZM-MESSAGEID: 1787075040207158500 Content-Type: text/plain; charset="utf-8" translator_use_goto_tb() refuses to chain unless the destination is on the same page as the start of the TB. For guests whose text is much larger than a page this is expensive: an emulated alpha gcc compiling a 255k line translation unit takes the indirect dispatch path for 8.4 billion of its 34.2 billion TB exits, and a large share of those are ordinary direct branches that simply crossed an 8 KiB page boundary. The restriction was made unconditional by d3a2a1d803 ("accel/tcg: Introduce translator_use_goto_tb"), whose rationale was: Various targets avoid the page crossing test for CONFIG_USER_ONLY, but that is wrong: mmap and mprotect can change page permissions. That is true, but in user-only builds the invalidation path already covers it. There are no page tables: every mmap, mprotect and munmap reaches page_set_flags(), which calls tb_invalidate_phys_range() whenever the flags actually change, and tb_phys_invalidate() calls tb_jmp_unlink() to reset incoming jumps. A chained cross-page jump is therefore broken whenever the destination page's permissions change. This is not true in system mode, where TBs are keyed by physical address and a page table change invalidates nothing, so the restriction is kept there. Add tests/tcg/alpha/test-xpage-chain.c to cover the hazard directly. It places a direct branch near the end of one page targeting the next page, runs it 200000 times so the chain is established, then checks that mprotect(PROT_NONE) makes the next call fault, and that remapping the page with different code runs the new code rather than a stale translation. The test detects the hazard it is meant to detect: with the tb_invalidate_phys_range() call in page_set_flags() commented out, it fails both phases, executing page B after PROT_NONE and returning the stale result. Measured with qemu-alpha running an emulated alpha gcc 16.2.0 compiling the SQLite 3.45.1 amalgamation on an x86-64 host, LTO build, on top of the preceding patches: before: 890,713,633,237 instructions after: 869,178,598,378 instructions -2.42% before: 84.44s wall clock after: 80.49s wall clock -4.68% Note that this is worth more in time than in instructions, the reverse of the preceding patch: a chained jump replaces a cache probe whose loads can miss, so the instructions it removes are more expensive than average. Measured before the inline jump cache probe, when a missed chain cost a helper call rather than an inline probe, the same change was worth -7.9%. RFC because this reverses a deliberate decision and the reasoning above wants review from someone who knows the invalidation paths better than I do. Signed-off-by: Matt Turner --- accel/tcg/translator.c | 12 +++- tests/tcg/alpha/Makefile.target | 2 +- tests/tcg/alpha/test-xpage-chain.c | 111 +++++++++++++++++++++++++++++ 3 files changed, 123 insertions(+), 2 deletions(-) create mode 100644 tests/tcg/alpha/test-xpage-chain.c diff --git ./accel/tcg/translator.c ./accel/tcg/translator.c index 85bb21e911..3eab9f570d 100644 --- ./accel/tcg/translator.c +++ ./accel/tcg/translator.c @@ -108,6 +108,17 @@ static void gen_tb_end(const TranslationBlock *tb, uin= t32_t cflags, =20 bool translator_is_same_page(const DisasContextBase *db, vaddr addr) { + /* + * In user-only mode there are no page tables. Every mmap, mprotect a= nd + * munmap goes through page_set_flags(), which calls + * tb_invalidate_phys_range() whenever the flags actually change, and + * tb_phys_invalidate() unlinks incoming jumps. A cross-page link is + * therefore broken whenever the destination page's permissions change, + * so the same-page restriction is not needed. + */ + if (IS_ENABLED(CONFIG_USER_ONLY)) { + return true; + } return ((addr ^ db->pc_first) & TARGET_PAGE_MASK) =3D=3D 0; } =20 @@ -118,7 +129,6 @@ bool translator_use_goto_tb(DisasContextBase *db, vaddr= dest) return false; } =20 - /* Check for the dest on the same page as the start of the TB. */ return translator_is_same_page(db, dest); } =20 diff --git ./tests/tcg/alpha/Makefile.target ./tests/tcg/alpha/Makefile.tar= get index 36d8ed1eae..eee986bab6 100644 --- ./tests/tcg/alpha/Makefile.target +++ ./tests/tcg/alpha/Makefile.target @@ -5,7 +5,7 @@ ALPHA_SRC=3D$(SRC_PATH)/tests/tcg/alpha VPATH+=3D$(ALPHA_SRC) =20 -ALPHA_TESTS=3Dhello-alpha test-cond test-cmov test-ovf test-cvttq +ALPHA_TESTS=3Dhello-alpha test-cond test-cmov test-ovf test-cvttq test-xpa= ge-chain TESTS+=3D$(ALPHA_TESTS) =20 test-cmov: EXTRA_CFLAGS=3D-DTEST_CMOV diff --git ./tests/tcg/alpha/test-xpage-chain.c ./tests/tcg/alpha/test-xpag= e-chain.c new file mode 100644 index 0000000000..7916d544af --- /dev/null +++ ./tests/tcg/alpha/test-xpage-chain.c @@ -0,0 +1,111 @@ +/* + * Cross-page TB chaining hazard test. + * + * Phase 1: a direct branch (br) near the end of page A targets page B. + * Run it enough times that QEMU chains TB_A -> TB_B. + * Phase 2: mprotect page B away. Re-running must fault. + * Phase 3: remap page B with different code. Re-running must execute the + * NEW code, not a stale chained translation of the old code. + * + * SPDX-License-Identifier: GPL-2.0-or-later + */ +#include +#include +#include +#include +#include +#include +#include + +#define PS 8192 + +static sigjmp_buf jb; +/* + * Written by the SIGSEGV handler and read by main(), so it must not be + * cached in a register across the faulting call. + */ +static volatile sig_atomic_t caught; + +static void segv(int sig) +{ + caught =3D 1; + siglongjmp(jb, 1); +} + +/* lda $0, imm($31) -> v0 =3D imm */ +static unsigned int lda_v0(int imm) +{ + return 0x201F0000u | (unsigned short)imm; +} + +int main(void) +{ + struct sigaction sa; + int rc =3D 0; + unsigned char *m =3D mmap(NULL, 2 * PS, PROT_READ | PROT_WRITE | PROT_= EXEC, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + if (m =3D=3D MAP_FAILED) { + perror("mmap"); + return 2; + } + + unsigned char *pa =3D m, *pb =3D m + PS; + unsigned int *entry =3D (unsigned int *)(pa + PS - 64); + unsigned int *tgt =3D (unsigned int *)(pb + 16); + + entry[0] =3D lda_v0(1); + long disp =3D ((long)tgt - ((long)&entry[1] + 4)) / 4; + entry[1] =3D 0xC3E00000u | (unsigned int)(disp & 0x1FFFFF); /* br $31= ,tgt */ + tgt[0] =3D 0x6BFA8001u; /* ret = */ + __builtin___clear_cache((char *)m, (char *)m + 2 * PS); + + long (*fn)(void) =3D (long (*)(void))entry; + + for (int i =3D 0; i < 200000; i++) { + if (fn() !=3D 1) { + printf("FAIL: phase 1 wrong result\n"); + return 1; + } + } + printf("phase 1 ok (chained)\n"); + + memset(&sa, 0, sizeof(sa)); + sa.sa_handler =3D segv; + sigemptyset(&sa.sa_mask); + if (sigaction(SIGSEGV, &sa, NULL) !=3D 0) { + perror("sigaction"); + return 2; + } + if (mprotect(pb, PS, PROT_NONE) !=3D 0) { + perror("mprotect"); + return 2; + } + if (sigsetjmp(jb, 1) =3D=3D 0) { + fn(); + printf("FAIL: phase 2 executed page B after mprotect(PROT_NONE)\n"= ); + rc =3D 1; + } else if (!caught) { + printf("FAIL: phase 2 longjmp without entering the handler\n"); + rc =3D 1; + } else { + printf("phase 2 ok (faulted)\n"); + } + + /* Phase 3: remap with different code, expect the new code to run. */ + if (mprotect(pb, PS, PROT_READ | PROT_WRITE | PROT_EXEC) !=3D 0) { + perror("mprotect back"); + return 2; + } + tgt[0] =3D lda_v0(2); + tgt[1] =3D 0x6BFA8001u; + __builtin___clear_cache((char *)pb, (char *)pb + PS); + + long r =3D fn(); + if (r !=3D 2) { + printf("FAIL: phase 3 returned %ld, expected 2 (stale chain)\n", r= ); + rc =3D 1; + } else { + printf("phase 3 ok (new code ran)\n"); + } + return rc; +} --=20 2.54.0 From nobody Mon Sep 28 00:51:17 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=gmail.com ARC-Seal: i=1; a=rsa-sha256; t=1787075038; cv=none; d=zohomail.com; s=zohoarc; b=CKrMZrBMGI7MJyGejGD3kZNwm4m5CMJ0AxfAAxoRSamE9n3DjDpfvChABTcV9eEHTNni5JMpdp4iBIO6LOPJuzZ6K0ijwY6sPvzf5GamuvVKXv9bmOPRBscebjjV2bhbzCYdSR56h6K4BSxGtdnkIWUrBYcm4ZMrBkCBEC5wMmA= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1787075038; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=Yl8NWho8R38WPE5uL/Y10j/MjLRNAzzDEp+FlNiwof8=; b=TWKfvwW1ZD7/efKFZXbndJvYckeQ8yelnVcInzJbzajPakrQo8+AkAFiTNFkSzdcnA3C4XAXxi3gih0QOGysOGQ/YI6LVp8EWZOa+KYm4SO1CaKpq0qRDz3kcXAF/tneMfMR/BuoW6qKQE3w0i5i/4o9Zrd0757sQIuQ8y98bhY= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists1p.gnu.org (lists1p.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1787075038856320.3784867570314; Tue, 18 Aug 2026 10:43:58 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists1p.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1wwNqS-0002x1-1m; Tue, 18 Aug 2026 13:43:28 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists1p.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1wwNqI-0002vP-H7 for qemu-devel@nongnu.org; Tue, 18 Aug 2026 13:43:18 -0400 Received: from mail-yw1-x1129.google.com ([2607:f8b0:4864:20::1129]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1wwNqE-0007zg-Nv for qemu-devel@nongnu.org; Tue, 18 Aug 2026 13:43:18 -0400 Received: by mail-yw1-x1129.google.com with SMTP id 00721157ae682-836c4474028so2360547b3.0 for ; Tue, 18 Aug 2026 10:43:09 -0700 (PDT) Received: from localhost ([2600:1702:7a90:6f9f:8bc4:8aec:108d:7a04]) by smtp.gmail.com with ESMTPSA id 00721157ae682-84068c23cb2sm25251817b3.13.2026.08.18.10.43.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Aug 2026 10:43:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1787074988; x=1787679788; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=Yl8NWho8R38WPE5uL/Y10j/MjLRNAzzDEp+FlNiwof8=; b=EiAVlgFrwpBLiQv8isBoRs7Yft0bGILmn0Lg4Dvzm6wOKaqZ64SMo0ErYLXAmRGxeV UYbgX9EqzaETjp0HytbHwSA13DX8NvxtHm47v80D9zFwOzsS53yBuGVf5Xxrxsx3AG5+ yADm/DMRTCnmSBKqKLxi/SP4FgK6bY6uYnbL196gRlCe5TzM0s+JDwRAHD40kujRTHA0 dKPWGU7pH0M+IF4rzckBH50AI7nJ6X1r39tIQSBojIm9mujOMExy7AU6fw1824Rh0+St v4Jkq2CjK7HuEYtjHInSPMmD+BoBbjIV+KoNhKi0RPaDf6xP7p4QZt/UyjvXZ2WS4hcV AI1g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787074988; x=1787679788; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=Yl8NWho8R38WPE5uL/Y10j/MjLRNAzzDEp+FlNiwof8=; b=ojLDBIc+tW1Rny8guaEsIdLHCbyaa72oKDnjCV2ZiAcey+O48u81O6A9tlQZkHw+Pk mjrM4MD4WBxjpgx2fgCmLGFiD+TuMgcQWbuds3cbK4lBko0YcRW80yXkotYoMpXOu9lX 9o3lljGE1qys3KkoO9yjD6E7vBQV2S6FqSSBW1UVtZr9KX5ErqB0jqZbYVWDpSIXQsqR Sr0kWNS1S2t+jbI66ICChgzMZPovOlix1D5rqaILj8xapmOiGVZaVB+c+6k28AOtYvmH KOVW0D2y7DY9N/EwBfy8TfGZmLdY6ViEP3xfRXIhCNVUGJDL9ZNQJ9s3UirGDjM2j5yT eCTw== X-Gm-Message-State: AOJu0YyvODLPcJKfKbddwMQvWtOD0kuP2VzXvHE8IZg5qVUeWU/uuxhX 39bZTupKpkl8gJYBN7Gn9fmLASSDR7Jp+kuT9mgZbRQfkA8DH842/rsYeBCMtB3AFsg= X-Gm-Gg: AR+sD13k9bE7htIpaifduaFIiL3eAvPQd2x246S8dfCLv617FPahuuQ8Yx8AAqY4eZ8 GDnmWFxEnY2kYJEmBh9QpT9yLI0NRT0WE1j8Ntp1R61A6FMCqzbPlX0QIgI1L2mHk5KwhiFWvcc KBjjb6JQ1mT41LGJ7KkMGE/t4q2tPYztkj+PDCxyPqm4DdEGiUjeAnTimwlPU8C+h7Y78Ctbps2 9btWMevGMVQSGa1EQWO00jznE1YGEIbHA1pE6D4RO5L3SQKejsOOMm8Mo39g0qLk7SfwA1VyVxh uJ4FxLIdo7uBrPbXstNjsPTXzfcKhgl5HZvTdxdkyaXyAqgewhOJ717yWgmnwgbJZbMkt9HQ4hj HAwq1ckw/ugfzBivoc3gWrmFcjLhnyHjZTKXSQlg7kVowZpOeKbrBk50Lf/I8VZwYZShxjmTbjR IDDc+Zl7U3QbmQO9bXjjq+d2Wo4HLB5Whyd6bO9P5ox9C1IYX8ofmK/JvnaSkU X-Received: by 2002:a05:690c:b0b:b0:80d:f9bb:3d3c with SMTP id 00721157ae682-837149e3da4mr116585987b3.36.1787074988059; Tue, 18 Aug 2026 10:43:08 -0700 (PDT) From: Matt Turner To: qemu-devel@nongnu.org Cc: richard.henderson@linaro.org, pbonzini@redhat.com, philmd@mailo.com, zhao1.liu@intel.com, laurent@vivier.eu, deller@gmx.de, pierrick.bouvier@oss.qualcomm.com, Matt Turner Subject: [PATCH 6/8] RFC: accel/tcg: only poll for interrupts in blocks that can close a cycle Date: Tue, 18 Aug 2026 13:42:45 -0400 Message-ID: <20260818174247.649526-7-mattst88@gmail.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260818174247.649526-1-mattst88@gmail.com> References: <20260818174247.649526-1-mattst88@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists1p.gnu.org; Received-SPF: pass client-ip=2607:f8b0:4864:20::1129; envelope-from=mattst88@gmail.com; helo=mail-yw1-x1129.google.com X-Spam_score_int: -17 X-Spam_score: -1.8 X-Spam_bar: - X-Spam_report: (-1.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @gmail.com) X-ZM-MESSAGEID: 1787075040208158500 Content-Type: text/plain; charset="utf-8" Every translation block begins by loading cpu->neg.icount_decr.u32, testing it and branching to the exit path. That is three host instructions at the top of every TB. Blocks are short, so this is expensive: an emulated alpha gcc 16.2.0 compiling a 255k line translation unit executes 34.2 billion TBs at 6.04 guest instructions each. Forcing CF_NOIRQ on for the whole run, which is not correct but bounds the prize, is worth 12.0% of all instructions retired. The check does not have to be in every block. Interrupt latency is bounded as long as every cycle in the guest control flow graph passes through at least one block that polls. Any such cycle must contain either an edge whose destination is at or below the start of the block it leaves from, or an edge whose destination is not known at translation time: take the block with the lowest start address in the cycle, and the edge entering it comes from a block at or above it. So record, during translation, whether this TB has such an edge. translator_use_goto_tb() already sees every statically known destination, and every target that emits goto_tb reaches it, so a backward edge sets DisasContextBase::needs_exit_check there. Indirect destinations are flagged by tcg_gen_lookup_and_goto_ptr(). Blocks with neither cannot close a cycle on their own and can skip the poll. The check is therefore emitted retroactively in gen_tb_end(), using the same emit_before_op mechanism the can_do_io stores use, and only when one of the two flags is set. icount opts out and keeps the unconditional counter. Measured on an x86-64 host, LTO build, on top of the preceding patches: before: 869,178,598,378 instructions after: 809,988,851,304 instructions -6.81% before: 80.49s wall clock after: 78.00s wall clock -3.10% That is 57% of the 12.0% ceiling, which is about right: roughly a quarter of TB exits are indirect and are still polled, plus every loop back edge. For the series as a whole, against an unmodified LTO build, instructions retired fall from 1,647,901,588,726 to 809,988,851,304 (-50.85%) and wall clock from 133.57s to 78.00s (-41.61%). The two do not match because what the series removes is mostly cheap, well-predicted dispatch overhead: IPC falls from 2.53 to 2.12 as the remaining work gets less regular. tests/tcg/alpha/test-xpage-chain.c still passes, the emulated compiler still produces byte-identical output, and a tight loop under alarm(1) is still interrupted, after 897 million iterations. RFC because: - The soundness argument depends on every goto_tb destination passing through translator_use_goto_tb(). No target in the tree bypasses it today, but nothing enforces that. - System mode interrupt latency now depends on guest control flow rather than on block count. The bound is one straight-line run between cycles, which should be fine, but timer-driven guests deserve a closer look than I can give them. Signed-off-by: Matt Turner --- accel/tcg/translator.c | 57 ++++++++++++++++++++++++++++++++++++--- include/exec/translator.h | 2 ++ include/tcg/tcg.h | 2 ++ tcg/tcg-op.c | 2 ++ 4 files changed, 60 insertions(+), 3 deletions(-) diff --git ./accel/tcg/translator.c ./accel/tcg/translator.c index 3eab9f570d..048ad8bad2 100644 --- ./accel/tcg/translator.c +++ ./accel/tcg/translator.c @@ -43,12 +43,29 @@ bool translator_io_start(DisasContextBase *db) return true; } =20 +/* + * Any cycle in the guest control flow graph must contain an edge whose + * destination is at or below the start of the block it leaves from, or an + * edge whose destination is not known at translation time. Only blocks wi= th + * such an edge need the interrupt check, so defer the decision until the = end + * of translation, when we know which edges this TB has. + * + * icount needs the counter unconditionally, so it opts out. + */ +static bool defer_exit_check(uint32_t cflags) +{ + return !(cflags & CF_USE_ICOUNT); +} + static TCGOp *gen_tb_start(DisasContextBase *db, uint32_t cflags) { TCGv_i32 count =3D NULL; TCGOp *icount_start_insn =3D NULL; =20 - if ((cflags & CF_USE_ICOUNT) || !(cflags & CF_NOIRQ)) { + tcg_ctx->exit_check_needed =3D false; + + if ((cflags & CF_USE_ICOUNT) || + (!(cflags & CF_NOIRQ) && !defer_exit_check(cflags))) { count =3D tcg_temp_new_i32(); tcg_gen_ld_i32(count, tcg_env, offsetof(CPUState, neg.icount_decr.u32) - @@ -74,6 +91,12 @@ static TCGOp *gen_tb_start(DisasContextBase *db, uint32_= t cflags) */ if (cflags & CF_NOIRQ) { tcg_ctx->exitreq_label =3D NULL; + } else if (defer_exit_check(cflags)) { + /* + * Emitted retroactively by gen_tb_end(), but only if this TB can = be + * part of a control flow cycle. + */ + tcg_ctx->exitreq_label =3D gen_new_label(); } else { tcg_ctx->exitreq_label =3D gen_new_label(); tcg_gen_brcondi_i32(TCG_COND_LT, count, 0, tcg_ctx->exitreq_label); @@ -89,7 +112,8 @@ static TCGOp *gen_tb_start(DisasContextBase *db, uint32_= t cflags) } =20 static void gen_tb_end(const TranslationBlock *tb, uint32_t cflags, - TCGOp *icount_start_insn, int num_insns) + TCGOp *icount_start_insn, int num_insns, + DisasContextBase *db, TCGOp *first_insn_start) { if (cflags & CF_USE_ICOUNT) { /* @@ -100,6 +124,23 @@ static void gen_tb_end(const TranslationBlock *tb, uin= t32_t cflags, tcgv_i32_arg(tcg_constant_i32(num_insns))); } =20 + if (tcg_ctx->exitreq_label && defer_exit_check(cflags) && + !(cflags & CF_NOIRQ)) { + if (db->needs_exit_check || tcg_ctx->exit_check_needed) { + TCGv_i32 count =3D tcg_temp_new_i32(); + TCGOp *save =3D tcg_ctx->emit_before_op; + + tcg_ctx->emit_before_op =3D first_insn_start; + tcg_gen_ld_i32(count, tcg_env, + offsetof(CPUState, neg.icount_decr.u32) - + sizeof(CPUState)); + tcg_gen_brcondi_i32(TCG_COND_LT, count, 0, tcg_ctx->exitreq_la= bel); + tcg_ctx->emit_before_op =3D save; + } else { + tcg_ctx->exitreq_label =3D NULL; + } + } + if (tcg_ctx->exitreq_label) { gen_set_label(tcg_ctx->exitreq_label); tcg_gen_exit_tb(tb, TB_EXIT_REQUESTED); @@ -129,6 +170,14 @@ bool translator_use_goto_tb(DisasContextBase *db, vadd= r dest) return false; } =20 + /* + * A destination at or below the start of this TB can close a cycle, so + * this TB must poll for interrupts. See defer_exit_check(). + */ + if (dest <=3D db->pc_first) { + db->needs_exit_check =3D true; + } + return translator_is_same_page(db, dest); } =20 @@ -152,6 +201,7 @@ void translator_loop(CPUState *cpu, TranslationBlock *t= b, int *max_insns, db->max_insns =3D *max_insns; db->insn_start =3D NULL; db->fake_insn =3D false; + db->needs_exit_check =3D false; db->host_addr[0] =3D host_pc; db->host_addr[1] =3D NULL; db->record_start =3D 0; @@ -218,7 +268,8 @@ void translator_loop(CPUState *cpu, TranslationBlock *t= b, int *max_insns, =20 /* Emit code to exit the TB, as indicated by db->is_jmp. */ ops->tb_stop(db, cpu); - gen_tb_end(tb, cflags, icount_start_insn, db->num_insns); + gen_tb_end(tb, cflags, icount_start_insn, db->num_insns, db, + first_insn_start); =20 /* * Manage can_do_io for the translation block: set to false before diff --git ./include/exec/translator.h ./include/exec/translator.h index 978dee25ad..003926c7f0 100644 --- ./include/exec/translator.h +++ ./include/exec/translator.h @@ -74,6 +74,8 @@ struct DisasContextBase { int max_insns; bool plugin_enabled; bool fake_insn; + /* Set when this TB can be part of a control flow cycle. */ + bool needs_exit_check; uint8_t code_mmuidx; struct TCGOp *insn_start; void *host_addr[2]; diff --git ./include/tcg/tcg.h ./include/tcg/tcg.h index 7669dc1c2d..be9ce7a0e2 100644 --- ./include/tcg/tcg.h +++ ./include/tcg/tcg.h @@ -389,6 +389,8 @@ struct TCGContext { struct TCGLabelPoolData *pool_labels; =20 TCGLabel *exitreq_label; + /* Set by goto_ptr emission: destination is not known statically. */ + bool exit_check_needed; =20 #ifdef CONFIG_PLUGIN /* diff --git ./tcg/tcg-op.c ./tcg/tcg-op.c index ab8d101871..8282afa61a 100644 --- ./tcg/tcg-op.c +++ ./tcg/tcg-op.c @@ -2616,6 +2616,7 @@ void tcg_gen_lookup_and_goto_ptr(void) return; } =20 + tcg_ctx->exit_check_needed =3D true; plugin_gen_disable_mem_helpers(); ptr =3D tcg_temp_ebb_new_ptr(); gen_helper_lookup_tb_ptr(ptr, tcg_env); @@ -2644,6 +2645,7 @@ void tcg_gen_lookup_and_goto_ptr_inline(TCGv_i64 pc, return; } =20 + tcg_ctx->exit_check_needed =3D true; plugin_gen_disable_mem_helpers(); =20 QEMU_BUILD_BUG_ON(sizeof(((CPUJumpCache *)0)->array[0]) !=3D 16); --=20 2.54.0 From nobody Mon Sep 28 00:51:17 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=gmail.com ARC-Seal: i=1; a=rsa-sha256; t=1787075108; cv=none; d=zohomail.com; s=zohoarc; b=LjYnxkv+5Kq7QuIKj1bvke31Z3cIKDyuGkuOB1IRFFcM5YQwyBug0r+ODyrtVhNHFw/Cjm6PIW2Ar3yz4pEJdxYJJpCLJYoPA0HSunR7f+P5fzUJy4r1ywm6wr0OkKI8xsu6XvfixVhzsi6M2OH9+hiKugYiep/7dgmQn+l46vY= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1787075108; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=/Q8w9k9lV2JrwiLBsWz2gBgu1b79v9MQACncuAAjwkE=; b=G6ZcJrZo18xmuorghuYAY0R8IDnG093F7/qoo+RtRG73cY3f9LOPYrgauW8ukMiKKCsT5nB3SvLGWqAxNPl02khHCbOapR4ye3an5kVP1EhpV2/njIwjEANnTTELXv8MnYQWyo+KzeWELk93ScFBtKEoDeeRDkzbDdsjDOq316I= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists1p.gnu.org (lists1p.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 17870751079919.227617678035813; Tue, 18 Aug 2026 10:45:07 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists1p.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1wwNqS-0002xB-5g; Tue, 18 Aug 2026 13:43:28 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists1p.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1wwNqK-0002vv-Ho for qemu-devel@nongnu.org; Tue, 18 Aug 2026 13:43:23 -0400 Received: from mail-yw1-x112f.google.com ([2607:f8b0:4864:20::112f]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1wwNqE-000811-Po for qemu-devel@nongnu.org; Tue, 18 Aug 2026 13:43:18 -0400 Received: by mail-yw1-x112f.google.com with SMTP id 00721157ae682-836c5b01e82so3409217b3.0 for ; Tue, 18 Aug 2026 10:43:10 -0700 (PDT) Received: from localhost ([2600:1702:7a90:6f9f:8bc4:8aec:108d:7a04]) by smtp.gmail.com with ESMTPSA id 00721157ae682-840f25bed4csm22573147b3.29.2026.08.18.10.43.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Aug 2026 10:43:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1787074990; x=1787679790; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=/Q8w9k9lV2JrwiLBsWz2gBgu1b79v9MQACncuAAjwkE=; b=Q3Q20pYCNapVgwKFmR9y9amjm21b30E0X2dxULspDDFQ6fjJ/Tz9GJdtG2GWlCY6wX UkN0ZHGEf4k8A0o1ViRq4pPqn+I5q8L/PtMhF8l5PLNc4pZ+/fR7IVHIFBJcuF41Li6H Q12nP5TKexURpC2dTAn07WpdONkt9ibetgLNCaOCY0YiBW6fYr6q1OOMt/+Mi9I80TIG PxAAFDffCbZtRuNAj6+p85h/N2NB6mKpFnMg7ff6ZyqbjUZpiKCbqWeBptNLW4eMEZ0u AnEyEnJUCy6SOB+T4A5ScEj7twUuz7pg9HvU3E1bRNvWG71ilFMqyiSwEIu882svqMn3 PhBA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787074990; x=1787679790; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=/Q8w9k9lV2JrwiLBsWz2gBgu1b79v9MQACncuAAjwkE=; b=sPW5O3WuwQigOQzCN7WgvxocwzbfbQOklnldH28M6unKa/dI0WiqgbmKfzgK3E4enc PvzV1fQeIr436wNRWqnwHpYNsxpk43bV0fUJ4gbRcdk5m6jPuOBEvzG0CAFLOljyWP9a r4kri+teGMtsXuQIjNQqml5XD1s4zYZb3lVpZFlnhF/9hCGgd2gfzTefGdJJO0YTRlEd SdOw0cQb9Ugfg8n1KoqVJYv9I4yO15ZZOdr8ASLBiEzv4RKKU4ojf9RDVzfsOCPbH1kI Wi+ZVMqOZaDpbGrjAogdrPoxf4NK7PHlTscWzEPhUf1ZCfVT335e/sEgYnDTlkpclXeG B+pg== X-Gm-Message-State: AOJu0Yyih9eTl6mG39AFstsTUMtO1ZczqCOrt9ap7tYBCsNrJHnJJxdP bI6IzUr/kYT/f9CQzuTGHGy0wFJqBR7yaPpZ/d8sj6Y+863S4QD+fIzvulPtLYKWY44= X-Gm-Gg: AR+sD11ia5jaLcZrIzw1QneUAmUndcXmWUeCguN5gS5nxWmSAD8fYWGvwWSMrjLujGb LHMESUllkzSaUi9fgmsNh88UeUdtFnjCZ3GyHTOZBNmfBX0B8DPCia/RYCQOpIDVI8QPC/ZH47v IxSG8huS3AbzaRVd1mU2FAf74NQqP9gjBAN9MnvlyS9wCowK8HPhsK9HeSFoHQXkQerZ3uAaqSa ePYBL9+jWSuQcFc+BBmrWr9PrfJy2s0yN7X4hWShCSutqNJjvDyfYzdKJ4bkOo6uD2w3+njMxl/ Xm6kyBTY3uOwOgRtSg9hD049AQ4Sn8QiXPhzk6fPuAPpPYzBOK9XZQCOSX8QHI2k2CtMIKLk3eR dcWN13VJUDh6K0UkBB63jWU93PQ8205cB/iNRhkgzSh4QBt9T1z2c54dY/CJe0usIUjSzSdxfva hqwrQnEBw97Vo/boL85ZE3wfv5BZejtr7SM5kvy7Tuvr/44zZVSX/H889zIW8WYEI9MQOX5zQC X-Received: by 2002:a05:690c:e28b:20b0:81d:a225:ecc0 with SMTP id 00721157ae682-841316581e9mr32565097b3.27.1787074989676; Tue, 18 Aug 2026 10:43:09 -0700 (PDT) From: Matt Turner To: qemu-devel@nongnu.org Cc: richard.henderson@linaro.org, pbonzini@redhat.com, philmd@mailo.com, zhao1.liu@intel.com, laurent@vivier.eu, deller@gmx.de, pierrick.bouvier@oss.qualcomm.com, Matt Turner Subject: [PATCH 7/8] RFC: accel/tcg: poison the jump cache instead of polling for indirect exits Date: Tue, 18 Aug 2026 13:42:46 -0400 Message-ID: <20260818174247.649526-8-mattst88@gmail.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260818174247.649526-1-mattst88@gmail.com> References: <20260818174247.649526-1-mattst88@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists1p.gnu.org; Received-SPF: pass client-ip=2607:f8b0:4864:20::112f; envelope-from=mattst88@gmail.com; helo=mail-yw1-x112f.google.com X-Spam_score_int: -17 X-Spam_score: -1.8 X-Spam_bar: - X-Spam_report: (-1.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @gmail.com) X-ZM-MESSAGEID: 1787075110195158500 Content-Type: text/plain; charset="utf-8" f799aeecff dropped the icount_decr poll from blocks that cannot close a control flow cycle. What is left is dominated by blocks that end in an indirect branch: tcg_gen_lookup_and_goto_ptr{,_inline}() set exit_check_needed, because the destination is unknown at translation time and so the block might be part of a cycle. For the emulated compiler that is still 55.5% of translated blocks and 51.2% of generated-code cycles, and the two leading instructions of those blocks mov -0x10(%rbp),%ebx test %ebx,%ebx carry 5.01% of all cycles spent in generated code, measured with cycles:pp so that the figure is not just skid from the dispatch that jumped there. A block dispatching indirectly does not need to poll, because the dispatch itself can be made to notice. The out-of-line path already calls helper_lookup_tb_ptr() every time, so it only needs the helper to return the epilogue while an exit is pending. The inline probe added by 14c3e5a1f3 is the interesting case: it already loads the jump cache base from CPUState and already branches to the slow path when the entry it finds has a NULL tb. Pointing that base at a page of zeroes therefore turns every indirect dispatch into a miss, and a miss lands in the same helper. The poll becomes a pointer swap on the exit request path and costs the fast path nothing. So give the probe its own base pointer, tb_jmp_cache_probe, that nothing else reads. The two places that set icount_decr.u16.high poison it; the place that clears the flag restores it. The real tb_jmp_cache is untouched throughout, so no cache contents are lost and the recovery is one store. Blocks with a backward goto_tb edge still poll. Interrupt latency is unchanged in kind: an exit is noticed at the next cycle-closing edge, which is now either a poll or a dispatch, rather than only a poll. Measured on an x86-64 host, LTO build, on top of the preceding patches. The control was measured in the same session, because the host's all-core turbo varies by ~3% between sessions and swamps the effect otherwise: before: 810,079,619,264 instructions, 79.563s after: 787,483,360,681 instructions, 78.019s -2.79% instructions, -1.94% wall Forcing the check off entirely, which is incorrect but is the ceiling, gives -5.64% instructions and -2.55% wall. So this takes half the instructions and three quarters of the time: what it removes sits directly after an indirect branch, where the poll's dependent load was the most expensive place a poll could be. In a jitdump profile of the same workload, blocks opening with the poll fall from 55.5% to 25.9%, cycles in blocks that poll from 51.2% to 18.5%, and cycles on the two poll instructions from 5.01% to 1.50%. tests/tcg/alpha/test-indirect-irq.c is added for this: a loop whose only ba= ck edge is an indirect branch, under alarm(1). It passes before and after, and hangs if the check is simply deleted, which is what makes it a test of the new mechanism rather than of the old poll. The other alpha tests still pass and the emulated compiler still produces byte-identical output. RFC because: - The restore in cpu_handle_interrupt() races a concurrent poison from another thread. The existing barrier around icount_decr.u16.high covers it -- a poison that lands after the restore also re-set the flag, and exit_request was stored before it -- but this deserves more eyes than the single-threaded user-mode testing I have given it. - Only the inline probe needs the poison, and only alpha uses the inline probe today. Targets on the out-of-line path are covered by the helper check alone, but that has not been measured. - The shared zero-filled CPUJumpCache is a 1MB allocation that is never written. A read-only mapping would express that better. Signed-off-by: Matt Turner --- accel/tcg/cpu-exec.c | 54 +++++++++++++++++++++++++++++ accel/tcg/internal-common.h | 3 ++ accel/tcg/tcg-accel-ops.c | 2 ++ accel/tcg/translator.c | 4 +-- include/hw/core/cpu.h | 10 ++++++ include/tcg/tcg.h | 2 -- tcg/tcg-op.c | 13 +++++-- tests/tcg/alpha/Makefile.target | 3 +- tests/tcg/alpha/test-indirect-irq.c | 53 ++++++++++++++++++++++++++++ 9 files changed, 135 insertions(+), 9 deletions(-) create mode 100644 tests/tcg/alpha/test-indirect-irq.c diff --git ./accel/tcg/cpu-exec.c ./accel/tcg/cpu-exec.c index 257211235d..af466ca14e 100644 --- ./accel/tcg/cpu-exec.c +++ ./accel/tcg/cpu-exec.c @@ -388,6 +388,16 @@ const void *HELPER(lookup_tb_ptr)(CPUArchState *env) */ cpu->neg.can_do_io =3D true; =20 + /* + * A block that dispatches indirectly does not emit the icount_decr po= ll, + * so this is where a pending exit is noticed for that path: either the + * probe was poisoned and every dispatch arrives here, or the target u= ses + * the out-of-line lookup and always did. + */ + if (unlikely(cpu_loop_exit_requested(cpu))) { + return tcg_code_gen_epilogue; + } + TCGTBCPUState s =3D cpu->cc->tcg_ops->get_tb_cpu_state(cpu); s.cflags =3D curr_cflags(cpu); =20 @@ -752,6 +762,44 @@ static inline bool cpu_handle_exception(CPUState *cpu,= int *ret) return false; } =20 +/* + * The inline jump cache probe reads cpu->tb_jmp_cache_probe and takes the + * slow path when the entry it finds has a NULL tb. Pointing the probe at a + * region that is all zeroes therefore forces every indirect dispatch into + * helper_lookup_tb_ptr(), which returns to the main loop while an exit is + * pending. That is what lets a block ending in an indirect branch skip the + * icount_decr poll: the poll's job is done by a pointer swap that costs t= he + * fast path nothing. + * + * Only ever read from, and only the tb field of one entry per dispatch, so + * one shared zero-filled cache is enough for every CPU. + */ +static const CPUJumpCache *tb_jmp_cache_poison(void) +{ + static CPUJumpCache *poison; + + if (unlikely(poison =3D=3D NULL)) { + /* Raced allocations are harmless: both are all zeroes. */ + qatomic_cmpxchg(&poison, NULL, g_new0(CPUJumpCache, 1)); + } + return poison; +} + +void tcg_cpu_poison_jmp_cache(CPUState *cpu) +{ + if (qatomic_read(&cpu->tb_jmp_cache_probe) !=3D NULL) { + qatomic_set(&cpu->tb_jmp_cache_probe, + (CPUJumpCache *)tb_jmp_cache_poison()); + } +} + +void tcg_cpu_restore_jmp_cache(CPUState *cpu) +{ + if (qatomic_read(&cpu->tb_jmp_cache_probe) !=3D NULL) { + qatomic_set(&cpu->tb_jmp_cache_probe, cpu->tb_jmp_cache); + } +} + void tcg_kick_vcpu_thread(CPUState *cpu) { /* @@ -764,6 +812,9 @@ void tcg_kick_vcpu_thread(CPUState *cpu) =20 /* Ensure cpu_exec will see the exit request after TCG has exited. */ qatomic_store_release(&cpu->neg.icount_decr.u16.high, -1); + + /* Blocks that only dispatch indirectly do not poll; stop them chainin= g. */ + tcg_cpu_poison_jmp_cache(cpu); } =20 static inline bool icount_exit_request(CPUState *cpu) @@ -796,6 +847,7 @@ static inline bool cpu_handle_interrupt(CPUState *cpu, * tcg_kick_vcpu_thread()) */ qatomic_set_mb(&cpu->neg.icount_decr.u16.high, 0); + tcg_cpu_restore_jmp_cache(cpu); =20 #ifdef CONFIG_USER_ONLY assert(!cpu_test_interrupt(cpu, ~0)); @@ -1069,6 +1121,7 @@ bool tcg_exec_realizefn(CPUState *cpu, Error **errp) } =20 cpu->tb_jmp_cache =3D g_new0(CPUJumpCache, 1); + qatomic_set(&cpu->tb_jmp_cache_probe, cpu->tb_jmp_cache); tlb_init(cpu); #ifndef CONFIG_USER_ONLY tcg_iommu_init_notifier_list(cpu); @@ -1086,5 +1139,6 @@ void tcg_exec_unrealizefn(CPUState *cpu) #endif /* !CONFIG_USER_ONLY */ =20 tlb_destroy(cpu); + qatomic_set(&cpu->tb_jmp_cache_probe, NULL); g_free_rcu(cpu->tb_jmp_cache, rcu); } diff --git ./accel/tcg/internal-common.h ./accel/tcg/internal-common.h index dc713a6e1a..6007223285 100644 --- ./accel/tcg/internal-common.h +++ ./accel/tcg/internal-common.h @@ -154,6 +154,9 @@ void page_table_config_init(void); G_NORETURN void cpu_io_recompile(CPUState *cpu, uintptr_t retaddr); #endif /* CONFIG_USER_ONLY */ =20 +void tcg_cpu_poison_jmp_cache(CPUState *cpu); +void tcg_cpu_restore_jmp_cache(CPUState *cpu); + void tb_phys_invalidate(TranslationBlock *tb, tb_page_addr_t page_addr); void tb_set_jmp_target(TranslationBlock *tb, int n, uintptr_t addr); =20 diff --git ./accel/tcg/tcg-accel-ops.c ./accel/tcg/tcg-accel-ops.c index 560fe2554b..fc134c48d9 100644 --- ./accel/tcg/tcg-accel-ops.c +++ ./accel/tcg/tcg-accel-ops.c @@ -44,6 +44,7 @@ =20 #include "hw/core/cpu.h" =20 +#include "internal-common.h" #include "tcg-accel-ops.h" #include "tcg-accel-ops-mttcg.h" #include "tcg-accel-ops-rr.h" @@ -106,6 +107,7 @@ void tcg_handle_interrupt(CPUState *cpu, int mask) qemu_cpu_kick(cpu); } else { qatomic_set(&cpu->neg.icount_decr.u16.high, -1); + tcg_cpu_poison_jmp_cache(cpu); } } =20 diff --git ./accel/tcg/translator.c ./accel/tcg/translator.c index 048ad8bad2..5f98d6446b 100644 --- ./accel/tcg/translator.c +++ ./accel/tcg/translator.c @@ -62,8 +62,6 @@ static TCGOp *gen_tb_start(DisasContextBase *db, uint32_t= cflags) TCGv_i32 count =3D NULL; TCGOp *icount_start_insn =3D NULL; =20 - tcg_ctx->exit_check_needed =3D false; - if ((cflags & CF_USE_ICOUNT) || (!(cflags & CF_NOIRQ) && !defer_exit_check(cflags))) { count =3D tcg_temp_new_i32(); @@ -126,7 +124,7 @@ static void gen_tb_end(const TranslationBlock *tb, uint= 32_t cflags, =20 if (tcg_ctx->exitreq_label && defer_exit_check(cflags) && !(cflags & CF_NOIRQ)) { - if (db->needs_exit_check || tcg_ctx->exit_check_needed) { + if (db->needs_exit_check) { TCGv_i32 count =3D tcg_temp_new_i32(); TCGOp *save =3D tcg_ctx->emit_before_op; =20 diff --git ./include/hw/core/cpu.h ./include/hw/core/cpu.h index 399ac7bd57..cecf9d0583 100644 --- ./include/hw/core/cpu.h +++ ./include/hw/core/cpu.h @@ -526,6 +526,16 @@ struct CPUState { =20 struct CPUJumpCache *tb_jmp_cache; =20 + /* + * What the inline jump cache probe emitted by + * tcg_gen_lookup_and_goto_ptr_inline() reads. Normally equal to + * tb_jmp_cache; pointed at a page of zeroes while an exit is pending,= so + * that every indirect dispatch misses and lands in the helper, which + * returns to the main loop. Only generated code and the two accessors= in + * cpu-exec.c may touch it. + */ + struct CPUJumpCache *tb_jmp_cache_probe; + GArray *gdb_regs; int gdb_num_regs; int gdb_num_g_regs; diff --git ./include/tcg/tcg.h ./include/tcg/tcg.h index be9ce7a0e2..7669dc1c2d 100644 --- ./include/tcg/tcg.h +++ ./include/tcg/tcg.h @@ -389,8 +389,6 @@ struct TCGContext { struct TCGLabelPoolData *pool_labels; =20 TCGLabel *exitreq_label; - /* Set by goto_ptr emission: destination is not known statically. */ - bool exit_check_needed; =20 #ifdef CONFIG_PLUGIN /* diff --git ./tcg/tcg-op.c ./tcg/tcg-op.c index 8282afa61a..ff63f16ccd 100644 --- ./tcg/tcg-op.c +++ ./tcg/tcg-op.c @@ -2616,7 +2616,10 @@ void tcg_gen_lookup_and_goto_ptr(void) return; } =20 - tcg_ctx->exit_check_needed =3D true; + /* + * No icount_decr poll is needed for this exit: the helper is called on + * every dispatch and returns to the main loop while an exit is pendin= g. + */ plugin_gen_disable_mem_helpers(); ptr =3D tcg_temp_ebb_new_ptr(); gen_helper_lookup_tb_ptr(ptr, tcg_env); @@ -2645,7 +2648,11 @@ void tcg_gen_lookup_and_goto_ptr_inline(TCGv_i64 pc, return; } =20 - tcg_ctx->exit_check_needed =3D true; + /* + * No icount_decr poll is needed for this exit either. A pending exit + * poisons tb_jmp_cache_probe, so the guarded load below finds a NULL = tb, + * takes the slow path, and the helper returns to the main loop. + */ plugin_gen_disable_mem_helpers(); =20 QEMU_BUILD_BUG_ON(sizeof(((CPUJumpCache *)0)->array[0]) !=3D 16); @@ -2667,7 +2674,7 @@ void tcg_gen_lookup_and_goto_ptr_inline(TCGv_i64 pc, tcg_gen_shli_i64(h, h, 4); =20 tcg_gen_ld_ptr(jc, tcg_env, - offsetof(CPUState, tb_jmp_cache) - sizeof(CPUState)); + offsetof(CPUState, tb_jmp_cache_probe) - sizeof(CPUStat= e)); tcg_gen_trunc_i64_ptr(ent, h); tcg_gen_add_ptr(ent, jc, ent); =20 diff --git ./tests/tcg/alpha/Makefile.target ./tests/tcg/alpha/Makefile.tar= get index eee986bab6..f9f135fc2f 100644 --- ./tests/tcg/alpha/Makefile.target +++ ./tests/tcg/alpha/Makefile.target @@ -5,7 +5,8 @@ ALPHA_SRC=3D$(SRC_PATH)/tests/tcg/alpha VPATH+=3D$(ALPHA_SRC) =20 -ALPHA_TESTS=3Dhello-alpha test-cond test-cmov test-ovf test-cvttq test-xpa= ge-chain +ALPHA_TESTS=3Dhello-alpha test-cond test-cmov test-ovf test-cvttq test-xpa= ge-chain \ + test-indirect-irq TESTS+=3D$(ALPHA_TESTS) =20 test-cmov: EXTRA_CFLAGS=3D-DTEST_CMOV diff --git ./tests/tcg/alpha/test-indirect-irq.c ./tests/tcg/alpha/test-ind= irect-irq.c new file mode 100644 index 0000000000..bef2844fd9 --- /dev/null +++ ./tests/tcg/alpha/test-indirect-irq.c @@ -0,0 +1,53 @@ +/* + * A loop whose only back edge is an indirect branch must still be + * interruptible. + * + * Blocks that dispatch indirectly do not emit the icount_decr poll; a pen= ding + * exit instead poisons the inline jump cache probe so that the dispatch f= alls + * into helper_lookup_tb_ptr(), which returns to the main loop. If that + * mechanism breaks, this program never leaves the loop and the test times + * out rather than failing an assertion. + * + * A computed goto is used deliberately: a plain while(1) closes the cycle + * with a direct backward branch, which is still polled, and so would not + * exercise the path under test. + */ +#include +#include +#include +#include +#include + +static volatile sig_atomic_t fired; +static volatile unsigned long iterations; + +static void handler(int sig) +{ + fired =3D 1; +} + +int main(void) +{ + /* + * Indexing a table with a volatile index, rather than jumping through= a + * volatile pointer: gcc happily proves a single-valued pointer consta= nt + * and emits a direct branch, which is the case this test is not about. + */ + void *target[2]; + volatile int idx =3D 0; + + assert(signal(SIGALRM, handler) !=3D SIG_ERR); + alarm(1); + + target[0] =3D &&spin; + target[1] =3D &&out; +spin: + iterations++; + if (!fired) { + goto *target[idx]; + } +out: + + printf("interrupted after %lu iterations\n", iterations); + return 0; +} --=20 2.54.0 From nobody Mon Sep 28 00:51:17 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=gmail.com ARC-Seal: i=1; a=rsa-sha256; t=1787075039; cv=none; d=zohomail.com; s=zohoarc; b=azQv95ZVGT2ilW6DUXnwsVe3fENGJgJmjP3HrUbXCZVG3bvooPYUOQSxLI83zljzppno37DlxPi6zHuy5iesr/O9JfnV6iKXYHM/LcOAVVez6oVJ4b9SraY0JN6QY7rOeba6RcJEp2Q1R+MJh64y/iFeFcsg4QibjefT8VwUxv0= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1787075039; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=BjxyEeK8365VbqkIEoqsYg/DoU9ewvIYcbSIMSbaWL4=; b=MNSe4M6vfAxRpGwAhekUEiXdhTMPkfRxTrcSz7ivWQeRVI5sgLTluwPtkDNYDCbqsqepSmo5FGa+rm7D0JenFuscQMHxzDrOm1DX266I/DF4RZYZuPMFGDNjWAMv+OS7H96UI3rOcy5cO8joQOcmMruizmrzWYYQRrx8MNv1cBQ= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists1p.gnu.org (lists1p.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1787075039879212.0403463126337; Tue, 18 Aug 2026 10:43:59 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists1p.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1wwNqQ-0002wb-3w; Tue, 18 Aug 2026 13:43:27 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists1p.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1wwNqI-0002vO-E6 for qemu-devel@nongnu.org; Tue, 18 Aug 2026 13:43:18 -0400 Received: from mail-yw1-x112c.google.com ([2607:f8b0:4864:20::112c]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1wwNqE-00081i-Pk for qemu-devel@nongnu.org; Tue, 18 Aug 2026 13:43:18 -0400 Received: by mail-yw1-x112c.google.com with SMTP id 00721157ae682-8200ca16f26so2472367b3.3 for ; Tue, 18 Aug 2026 10:43:12 -0700 (PDT) Received: from localhost ([2600:1702:7a90:6f9f:8bc4:8aec:108d:7a04]) by smtp.gmail.com with ESMTPSA id 00721157ae682-84068077d9csm24909897b3.8.2026.08.18.10.43.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Aug 2026 10:43:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1787074991; x=1787679791; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=BjxyEeK8365VbqkIEoqsYg/DoU9ewvIYcbSIMSbaWL4=; b=DDP4WzN5DbzaUtvr2rFtOmeGCs9uKhzFkJLziRg+bCiMdcUUQRBcfXiwd3KicJa5rm 1n0YjHOGaj7BdJPT0BrkUN/jUF9/mjv8VB25Sw0Vr4tf9pwS5LGKNxuVaQpnDYuzTEgZ FeiGwjmLiBaoePQ9PWqIDVcuaOflucy23yGCYtCclTCDW+Dv9+qepEyk+HZYhfPG44B4 MpTLxpQ89yKeqy1RsWUrFsu1ikjGOh7H0g9I7AvExckxAmPF+QW+WzBmJ6MvEhNFnt8q JT42dHWEDerI71Cggs6iqAyAreZ4MUNq+FFA4uNkwv2EcoWmZ1QWQFLjwX/+3vlr9eh4 wOYQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787074991; x=1787679791; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=BjxyEeK8365VbqkIEoqsYg/DoU9ewvIYcbSIMSbaWL4=; b=g6YUWv1I+TUHhve3WV9k8tuv4kEeC87KBM12RcwwJBjYDs8gOraCd47Yr+vMs8xeYB VeTY1cLycnrpkmNG2WZkVbfXYQfvNmiQVnvAwfj0tRsjKYw/ur5yTvVyVLU/HdznaUSw sJGiCVamBLfkvlMDjazgAqmuj1pmDNJfXr+xcHM+X6EEjxu81cFexvIJjQNRIRybm1vc iQRnzaRqQ/5IRCaItqPSBEjq14naqvx3364C730Mt0q26DNGmMZqdtwEHRx/q489ODJe 6wDKS0wFPawu7nmd4hQIdF9wb+wHRef2d0HInZcJwXL1qYdR4/qxnL4HQgbc2cw9oXRg eh2w== X-Gm-Message-State: AOJu0Yz+UuKHFTZqR8qSmfSmDRyedWkCvCdEiBpuqE+4VPAn+hxSrM5E ikA2CM0qanAqBzu2KNRsiXwZoETlBV3e2UVE4R4uoVqQJAGRkhtKo5Y2pceISsxwZos= X-Gm-Gg: AR+sD12uVvPmVWDIGEu2dTkm+Lh0PGvWaKU31euoO8+wABM23bAyQTPnELE6fn3AkkD wjyu7IGqTu4KQsl/ax/zoPKONJA57lmU3g3U9MiDpXWlNJ7hsb20OCVn/Sf5wduzEs9dW/VxQSw WtppvraiMlY9Z9ElOs/r3568TKvMM1jIRqFSerKgsCrQiSdTbPHCCABww1h+A0krJLVZYQpruZT 9hTrBRR40RGUsmbAKZ2ck1ybLfN7+dRarC6p6XLbeXHgAI34MWC+ntldZ/oMQ68eX/ZGw1dLciV hJDrEZrcW/klaO8GufQZigxbStgh33PhnPbdrItrp8Mc5nylZoPLS0cr57VSGVLDj24QMhf/X9t P0B3MiACxe8kD54NH35L7uEb8UKubAmn+YGspu2e0ZIE+z2ubnskynDrY05+QPTokidYkrkXWU/ EtqTqx+EOPGpIKpP5tdGa1JbA233O2A0q2AcVQrbjQ+uzdWkA0b5jvbVPfMDtlzrlhvg== X-Received: by 2002:a05:690c:18:b0:7b2:1bf1:8016 with SMTP id 00721157ae682-8412cd494f2mr39819657b3.2.1787074991302; Tue, 18 Aug 2026 10:43:11 -0700 (PDT) From: Matt Turner To: qemu-devel@nongnu.org Cc: richard.henderson@linaro.org, pbonzini@redhat.com, philmd@mailo.com, zhao1.liu@intel.com, laurent@vivier.eu, deller@gmx.de, pierrick.bouvier@oss.qualcomm.com, Matt Turner Subject: [PATCH 8/8] RFC: tcg: fold a guest displacement into the host addressing mode Date: Tue, 18 Aug 2026 13:42:47 -0400 Message-ID: <20260818174247.649526-9-mattst88@gmail.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260818174247.649526-1-mattst88@gmail.com> References: <20260818174247.649526-1-mattst88@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists1p.gnu.org; Received-SPF: pass client-ip=2607:f8b0:4864:20::112c; envelope-from=mattst88@gmail.com; helo=mail-yw1-x112c.google.com X-Spam_score_int: -17 X-Spam_score: -1.8 X-Spam_bar: - X-Spam_report: (-1.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @gmail.com) X-ZM-MESSAGEID: 1787075041782158500 Content-Type: text/plain; charset="utf-8" Nothing in the TCG frontend interface can express a based memory access. tcg_gen_qemu_ld/st take an address and nothing else, so a target with a displacement in its load and store encodings -- which is most of them -- has to materialise the address first: ldq a1,8(a0) -> mov 0x80(%rbp),%rbx reload a0 lea 0x8(%rbx),%r12 address mov (%r12),%r12 the load mov %r12,0x88(%rbp) spill a1 The lea is pure loss on a host whose addressing mode has a displacement field sitting empty. It also needs a register, at the point in a block where pressure is highest. Fold it. After optimisation, look for an add of a constant immediately before a guest access, defining that access's address operand, and move the constant into a new second constant argument on the op. The add is left for liveness to remove, so nothing breaks if its result has another use. Only the immediately preceding op is examined: that is what the frontends emit, and a window of one op means the pass does not have to reason about what could have happened in between. The one thing it does check is that the add did not clobber the base it read, since the access now reads that base directly. Targets opt in with TCG_TARGET_HAS_ldst_disp and an out_disp member on TCGOutOpQemuLdSt. Without it the pass does not run, the displacement stays zero and the existing out member is called exactly as before, so no other backend changes behaviour or needs touching. For x86_64 the displacement goes in the disp32 that prepare_host_addr() already fills in for guest_base. The fold is refused unless there is no slow path at all, which means user-only -- softmmu compares the unadjusted address against the TLB -- and an access needing no alignment test, since the slow path hands addr_reg to the helper and that register no longer holds the full guest address. It is also refused if guest_base plus the displacement leaves disp32. Measured with qemu-alpha running an emulated alpha gcc 16.2.0 compiling the SQLite 3.45.1 amalgamation (255k lines, -O2) on an x86-64 host, LTO build, on top of the preceding patches, against a control measured in the same session: before: 787,483,360,681 instructions, 78.250s after: 738,020,161,466 instructions, 75.679s -6.29% instructions, -3.29% wall Emitted code shrinks from 46.29MB to 44.53MB over the run, 153.3 to 147.5 bytes per block. Per Alpha opcode, the host bytes emitted for an access fall as expected and nothing else moves: ldq 17.2 -> 14.3 ldah 16.9 -> 16.9 ldl 14.1 -> 11.6 lda 12.6 -> 12.6 stq 12.7 -> 9.7 mov 9.7 -> 9.7 The emulated compiler produces byte-identical output and the alpha tests still pass, including with a non-zero guest_base forced via -B. RFC because: - Only wired up for x86_64, and only for qemu_ld and qemu_st; the i128 qemu_ld2 and qemu_st2 pairs are left alone. - Requiring that no slow path exists is stricter than necessary. Recording the displacement in TCGLabelQemuLdst and emitting one lea on the slow path would cover alignment-checked accesses too, at no fast path cost. - Softmmu wants the displacement folded into the TLB comparison as well, which is a bigger change than this one. - A one op window catches everything the frontends emit today but is trivially defeated by anything scheduled in between. Signed-off-by: Matt Turner --- include/tcg/tcg-opc.h | 9 +++- tcg/tcg-op-ldst.c | 3 +- tcg/tcg.c | 86 ++++++++++++++++++++++++++++++++++++- tcg/x86_64/tcg-target.c.inc | 61 ++++++++++++++++++++++++++ tcg/x86_64/tcg-target.h | 3 ++ 5 files changed, 158 insertions(+), 4 deletions(-) diff --git ./include/tcg/tcg-opc.h ./include/tcg/tcg-opc.h index 61f1c28858..0a3b4330f1 100644 --- ./include/tcg/tcg-opc.h +++ ./include/tcg/tcg-opc.h @@ -118,8 +118,13 @@ DEF(goto_ptr, 0, 1, 0, TCG_OPF_BB_EXIT | TCG_OPF_BB_EN= D) DEF(plugin_cb, 0, 0, 1, TCG_OPF_NOT_PRESENT) DEF(plugin_mem_cb, 0, 1, 1, TCG_OPF_NOT_PRESENT) =20 -DEF(qemu_ld, 1, 1, 1, TCG_OPF_CALL_CLOBBER | TCG_OPF_SIDE_EFFECTS | TCG_OP= F_INT) -DEF(qemu_st, 0, 2, 1, TCG_OPF_CALL_CLOBBER | TCG_OPF_SIDE_EFFECTS | TCG_OP= F_INT) +/* + * The second constant argument is a displacement to add to the address, + * zero unless a target advertises TCG_TARGET_HAS_ldst_disp and the fold in + * fold_ldst_disp() applied. + */ +DEF(qemu_ld, 1, 1, 2, TCG_OPF_CALL_CLOBBER | TCG_OPF_SIDE_EFFECTS | TCG_OP= F_INT) +DEF(qemu_st, 0, 2, 2, TCG_OPF_CALL_CLOBBER | TCG_OPF_SIDE_EFFECTS | TCG_OP= F_INT) DEF(qemu_ld2, 2, 1, 1, TCG_OPF_CALL_CLOBBER | TCG_OPF_SIDE_EFFECTS | TCG_O= PF_INT) DEF(qemu_st2, 0, 3, 1, TCG_OPF_CALL_CLOBBER | TCG_OPF_SIDE_EFFECTS | TCG_O= PF_INT) =20 diff --git ./tcg/tcg-op-ldst.c ./tcg/tcg-op-ldst.c index 22211ccb45..ffc5e651a6 100644 --- ./tcg/tcg-op-ldst.c +++ ./tcg/tcg-op-ldst.c @@ -92,7 +92,8 @@ static MemOp tcg_canonicalize_memop(MemOp op, bool is64, = bool st) static void gen_ldst1(TCGOpcode opc, TCGType type, TCGTemp *v, TCGTemp *addr, MemOpIdx oi) { - TCGOp *op =3D tcg_gen_op3(opc, type, temp_arg(v), temp_arg(addr), oi); + /* The trailing zero is the address displacement; see fold_ldst_disp()= . */ + TCGOp *op =3D tcg_gen_op4(opc, type, temp_arg(v), temp_arg(addr), oi, = 0); TCGOP_FLAGS(op) =3D get_memop(oi) & MO_SIZE; } =20 diff --git ./tcg/tcg.c ./tcg/tcg.c index 1e77f2365a..bdd95304ea 100644 --- ./tcg/tcg.c +++ ./tcg/tcg.c @@ -1058,6 +1058,13 @@ typedef struct TCGOutOpQemuLdSt { TCGOutOp base; void (*out)(TCGContext *s, TCGType type, TCGReg dest, TCGReg addr, MemOpIdx oi); + /* + * As out(), for an access at addr + disp. Only required of targets th= at + * define TCG_TARGET_HAS_ldst_disp; for everyone else fold_ldst_disp() + * never runs and the displacement is always zero. + */ + void (*out_disp)(TCGContext *s, TCGType type, TCGReg dest, + TCGReg addr, MemOpIdx oi, int32_t disp); } TCGOutOpQemuLdSt; =20 typedef struct TCGOutOpQemuLdSt2 { @@ -3560,6 +3567,77 @@ static void move_label_uses(TCGLabel *to, TCGLabel *= from) QSIMPLEQ_CONCAT(&to->branches, &from->branches); } =20 +#ifndef TCG_TARGET_HAS_ldst_disp +#define TCG_TARGET_HAS_ldst_disp 0 +static bool tcg_target_ldst_disp_ok(TCGContext *s, MemOpIdx oi, int64_t di= sp) +{ + return false; +} +#endif + +/* + * Fold "add addr, base, $disp" into the guest access that follows it, so + * that the displacement becomes part of the host addressing mode instead = of + * a separate instruction. Frontends have no way to express this: there is + * no displacement operand on tcg_gen_qemu_ld/st, so a based access always + * costs an extra add, and an extra register to hold its result. + * + * Only an add in the op immediately before the access is recognised. That + * is what the frontends emit, and a window of one op means no analysis is + * needed of what might have happened in between. The add is left in place; + * liveness removes it if its result has no other use. + */ +static void __attribute__((noinline)) +fold_ldst_disp(TCGContext *s) +{ + TCGOp *op; + + if (!TCG_TARGET_HAS_ldst_disp) { + return; + } + + QTAILQ_FOREACH(op, &s->ops, link) { + TCGOp *prev; + TCGTemp *cts; + int64_t disp; + + switch (op->opc) { + case INDEX_op_qemu_ld: + case INDEX_op_qemu_st: + break; + default: + continue; + } + + prev =3D QTAILQ_PREV(op, link); + if (prev =3D=3D NULL || prev->opc !=3D INDEX_op_add || + TCGOP_TYPE(prev) !=3D s->addr_type) { + continue; + } + + /* + * The add must define the address operand, and must not have + * clobbered the base it read: after the fold the access reads the + * base directly, so the base has to still hold its original value. + */ + if (prev->args[0] !=3D op->args[1] || prev->args[0] =3D=3D prev->a= rgs[1]) { + continue; + } + + cts =3D arg_temp(prev->args[2]); + if (cts->kind !=3D TEMP_CONST) { + continue; + } + disp =3D cts->val; + if (disp =3D=3D 0 || !tcg_target_ldst_disp_ok(s, op->args[2], disp= )) { + continue; + } + + op->args[1] =3D prev->args[1]; + op->args[3] =3D disp; + } +} + /* Reachable analysis : remove unreachable code. */ static void __attribute__((noinline)) reachable_code_pass(TCGContext *s) @@ -5707,7 +5785,12 @@ static void tcg_reg_alloc_op(TCGContext *s, const TC= GOp *op) const TCGOutOpQemuLdSt *out =3D container_of(all_outop[op->opc], TCGOutOpQemuLdSt, base); =20 - out->out(s, type, new_args[0], new_args[1], new_args[2]); + if (new_args[3]) { + out->out_disp(s, type, new_args[0], new_args[1], + new_args[2], new_args[3]); + } else { + out->out(s, type, new_args[0], new_args[1], new_args[2]); + } } break; =20 @@ -6590,6 +6673,7 @@ int tcg_gen_code(TCGContext *s, TranslationBlock *tb,= uint64_t pc_start) tcg_temp_ebb_reset_freed(s); =20 tcg_optimize(s); + fold_ldst_disp(s); =20 reachable_code_pass(s); liveness_pass_0(s); diff --git ./tcg/x86_64/tcg-target.c.inc ./tcg/x86_64/tcg-target.c.inc index 1fc45e4ec6..d3bde91816 100644 --- ./tcg/x86_64/tcg-target.c.inc +++ ./tcg/x86_64/tcg-target.c.inc @@ -2015,6 +2015,39 @@ static TCGLabelQemuLdst *prepare_host_addr(TCGContex= t *s, HostAddress *h, return ldst; } =20 +/* + * Whether the displacement of a guest access can be folded into the host + * addressing mode rather than materialised by a separate lea. + * + * Folding rewrites the access to use base + disp, so nothing may need a + * register holding the complete guest address. The softmmu TLB comparison + * does, and so does any slow path, which hands addr_reg to the helper. In + * user-only mode prepare_host_addr() creates a slow path only for an + * alignment test, so requiring that none is needed rules it out. What is + * left to check is guest_base, which shares the disp32 field. + */ +static bool tcg_target_ldst_disp_ok(TCGContext *s, MemOpIdx oi, int64_t di= sp) +{ +#ifdef CONFIG_USER_ONLY + MemOp opc =3D get_memop(oi); + TCGAtomAlign aa; + int64_t ofs; + + if (tcg_use_softmmu || s->addr_type !=3D TCG_TYPE_I64) { + return false; + } + aa =3D atom_and_align_for_opc(s, opc, MO_ATOM_WITHIN16, + (opc & MO_SIZE) =3D=3D MO_128); + if (aa.align) { + return false; + } + ofs =3D (int64_t)x86_guest_base.ofs + disp; + return ofs =3D=3D (int32_t)ofs; +#else + return false; +#endif +} + static void tcg_out_qemu_ld_direct(TCGContext *s, TCGReg datalo, TCGReg da= tahi, HostAddress h, TCGType type, MemOp memo= p) { @@ -2171,9 +2204,23 @@ static void tgen_qemu_ld(TCGContext *s, TCGType type= , TCGReg data, } } =20 +static void tgen_qemu_ld_disp(TCGContext *s, TCGType type, TCGReg data, + TCGReg addr, MemOpIdx oi, int32_t disp) +{ + TCGLabelQemuLdst *ldst; + HostAddress h; + + ldst =3D prepare_host_addr(s, &h, addr, oi, true); + /* tcg_target_ldst_disp_ok() has ruled out every slow path. */ + tcg_debug_assert(ldst =3D=3D NULL); + h.ofs +=3D disp; + tcg_out_qemu_ld_direct(s, data, -1, h, type, get_memop(oi)); +} + static const TCGOutOpQemuLdSt outop_qemu_ld =3D { .base.static_constraint =3D C_O1_I1(r, L), .out =3D tgen_qemu_ld, + .out_disp =3D tgen_qemu_ld_disp, }; =20 static void tgen_qemu_ld2(TCGContext *s, TCGType type, TCGReg datalo, @@ -2309,9 +2356,23 @@ static void tgen_qemu_st(TCGContext *s, TCGType type= , TCGReg data, } } =20 +static void tgen_qemu_st_disp(TCGContext *s, TCGType type, TCGReg data, + TCGReg addr, MemOpIdx oi, int32_t disp) +{ + TCGLabelQemuLdst *ldst; + HostAddress h; + + ldst =3D prepare_host_addr(s, &h, addr, oi, false); + /* tcg_target_ldst_disp_ok() has ruled out every slow path. */ + tcg_debug_assert(ldst =3D=3D NULL); + h.ofs +=3D disp; + tcg_out_qemu_st_direct(s, data, -1, h, get_memop(oi)); +} + static const TCGOutOpQemuLdSt outop_qemu_st =3D { .base.static_constraint =3D C_O0_I2(L, L), .out =3D tgen_qemu_st, + .out_disp =3D tgen_qemu_st_disp, }; =20 static void tgen_qemu_st2(TCGContext *s, TCGType type, TCGReg datalo, diff --git ./tcg/x86_64/tcg-target.h ./tcg/x86_64/tcg-target.h index 7ebae56a7d..8f2315c15e 100644 --- ./tcg/x86_64/tcg-target.h +++ ./tcg/x86_64/tcg-target.h @@ -30,6 +30,9 @@ #define TCG_TARGET_NB_REGS 32 #define MAX_CODE_GEN_BUFFER_SIZE (2 * GiB) =20 +/* A guest displacement can go in the disp32 of the addressing mode. */ +#define TCG_TARGET_HAS_ldst_disp 1 + typedef enum { TCG_REG_EAX =3D 0, TCG_REG_ECX, --=20 2.54.0