From nobody Thu Nov 6 01:17:26 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1539107786093335.2446639715405; Tue, 9 Oct 2018 10:56:26 -0700 (PDT) Received: from localhost ([::1]:53185 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1g9wF6-0001z0-PA for importer@patchew.org; Tue, 09 Oct 2018 13:56:24 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:54276) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1g9wAU-0006lR-TL for qemu-devel@nongnu.org; Tue, 09 Oct 2018 13:51:41 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1g9wAR-0002Ex-LF for qemu-devel@nongnu.org; Tue, 09 Oct 2018 13:51:38 -0400 Received: from wout1-smtp.messagingengine.com ([64.147.123.24]:56923) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1g9wAR-0002DS-8E for qemu-devel@nongnu.org; Tue, 09 Oct 2018 13:51:35 -0400 Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.west.internal (Postfix) with ESMTP id CE40D7FE; Tue, 9 Oct 2018 13:51:33 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute4.internal (MEProxy); Tue, 09 Oct 2018 13:51:34 -0400 Received: from localhost (flamenco.cs.columbia.edu [128.59.20.216]) by mail.messagingengine.com (Postfix) with ESMTPA id 01BFFE4240; Tue, 9 Oct 2018 13:51:33 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=braap.org; h= from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-type:content-transfer-encoding; s=mesmtp; bh=BH1mGAByPHWA0orRjeXvDMWBRWHILp45prcZeGanebM=; b=esoclBQTvL+G 1LuywvR4b7mAqoGhVva7J+bSJRi/V0wshYlpitjQiwBAY+5fY6tZLCSBQx6pgqfK fAzpL3kfctU1aDAnFXx0dFswjI4F1yQfe1I8LK0gAziuRHIyUk2Nv0vrFELr8BJf hsjfh/Yy9t9ul/b7sGmYaFPcBAkppKU= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm1; bh=BH1mGAByPHWA0orRjeXvDMWBRWHILp45prcZeGane bM=; b=MoRdZZFoMvgrwPzg1mkc6IUbyhMxN1LlsJsyyS7J5rhBT4Xsmi7IQ8aYA tBtrhxirNiMpq35i+BwDhh8pfoRr8DHQcCMHHVF87opvmysthg1TpvLeQI15v9ix AYJApWNLf4gjwyN9NVWqBVFF2OlKm6855UcPIUdIvpyBTUOBSQZNcnwfGbVls0Oj dsNK2q86xke7j/zsKNAzTSw97Fxhqqr6GAsRNNc8zw0Ry+uDCwgUebh0FQw4gvg4 xdwOVN4kBMtmj7vomRg+o7erI+dzPjf9XgxPzKBUxj3B7eEXVg5nB5ik3dGNDa/c GaZD6UNFqHmya+r8PAvXF0a8yWJ2g== X-ME-Sender: X-ME-Proxy: From: "Emilio G. Cota" To: qemu-devel@nongnu.org Date: Tue, 9 Oct 2018 13:51:25 -0400 Message-Id: <20181009175129.17888-2-cota@braap.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181009175129.17888-1-cota@braap.org> References: <20181009175129.17888-1-cota@braap.org> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 64.147.123.24 Subject: [Qemu-devel] [RFC v3 1/5] tcg: Add tlb_index and tlb_entry helpers X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: =?UTF-8?q?Alex=20Benn=C3=A9e?= , Richard Henderson Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 From: Richard Henderson Isolate the computation of an index from an address into a helper before we change that function. Reviewed-by: Alex Benn=C3=A9e Signed-off-by: Richard Henderson [ cota: convert tlb_vaddr_to_host; use atomic_read on addr_write ] Signed-off-by: Emilio G. Cota --- accel/tcg/softmmu_template.h | 68 ++++++++++++++++---------------- include/exec/cpu_ldst.h | 19 +++++++-- include/exec/cpu_ldst_template.h | 25 ++++++------ accel/tcg/cputlb.c | 61 +++++++++++++--------------- 4 files changed, 90 insertions(+), 83 deletions(-) diff --git a/accel/tcg/softmmu_template.h b/accel/tcg/softmmu_template.h index 1e50263871..3657c1f2e1 100644 --- a/accel/tcg/softmmu_template.h +++ b/accel/tcg/softmmu_template.h @@ -111,9 +111,10 @@ static inline DATA_TYPE glue(io_read, SUFFIX)(CPUArchS= tate *env, WORD_TYPE helper_le_ld_name(CPUArchState *env, target_ulong addr, TCGMemOpIdx oi, uintptr_t retaddr) { - unsigned mmu_idx =3D get_mmuidx(oi); - int index =3D (addr >> TARGET_PAGE_BITS) & (CPU_TLB_SIZE - 1); - target_ulong tlb_addr =3D env->tlb_table[mmu_idx][index].ADDR_READ; + uintptr_t mmu_idx =3D get_mmuidx(oi); + uintptr_t index =3D tlb_index(env, mmu_idx, addr); + CPUTLBEntry *entry =3D tlb_entry(env, mmu_idx, addr); + target_ulong tlb_addr =3D entry->ADDR_READ; unsigned a_bits =3D get_alignment_bits(get_memop(oi)); uintptr_t haddr; DATA_TYPE res; @@ -129,7 +130,7 @@ WORD_TYPE helper_le_ld_name(CPUArchState *env, target_u= long addr, tlb_fill(ENV_GET_CPU(env), addr, DATA_SIZE, READ_ACCESS_TYPE, mmu_idx, retaddr); } - tlb_addr =3D env->tlb_table[mmu_idx][index].ADDR_READ; + tlb_addr =3D entry->ADDR_READ; } =20 /* Handle an IO access. */ @@ -166,7 +167,7 @@ WORD_TYPE helper_le_ld_name(CPUArchState *env, target_u= long addr, return res; } =20 - haddr =3D addr + env->tlb_table[mmu_idx][index].addend; + haddr =3D addr + entry->addend; #if DATA_SIZE =3D=3D 1 res =3D glue(glue(ld, LSUFFIX), _p)((uint8_t *)haddr); #else @@ -179,9 +180,10 @@ WORD_TYPE helper_le_ld_name(CPUArchState *env, target_= ulong addr, WORD_TYPE helper_be_ld_name(CPUArchState *env, target_ulong addr, TCGMemOpIdx oi, uintptr_t retaddr) { - unsigned mmu_idx =3D get_mmuidx(oi); - int index =3D (addr >> TARGET_PAGE_BITS) & (CPU_TLB_SIZE - 1); - target_ulong tlb_addr =3D env->tlb_table[mmu_idx][index].ADDR_READ; + uintptr_t mmu_idx =3D get_mmuidx(oi); + uintptr_t index =3D tlb_index(env, mmu_idx, addr); + CPUTLBEntry *entry =3D tlb_entry(env, mmu_idx, addr); + target_ulong tlb_addr =3D entry->ADDR_READ; unsigned a_bits =3D get_alignment_bits(get_memop(oi)); uintptr_t haddr; DATA_TYPE res; @@ -197,7 +199,7 @@ WORD_TYPE helper_be_ld_name(CPUArchState *env, target_u= long addr, tlb_fill(ENV_GET_CPU(env), addr, DATA_SIZE, READ_ACCESS_TYPE, mmu_idx, retaddr); } - tlb_addr =3D env->tlb_table[mmu_idx][index].ADDR_READ; + tlb_addr =3D entry->ADDR_READ; } =20 /* Handle an IO access. */ @@ -234,7 +236,7 @@ WORD_TYPE helper_be_ld_name(CPUArchState *env, target_u= long addr, return res; } =20 - haddr =3D addr + env->tlb_table[mmu_idx][index].addend; + haddr =3D addr + entry->addend; res =3D glue(glue(ld, LSUFFIX), _be_p)((uint8_t *)haddr); return res; } @@ -275,10 +277,10 @@ static inline void glue(io_write, SUFFIX)(CPUArchStat= e *env, void helper_le_st_name(CPUArchState *env, target_ulong addr, DATA_TYPE val, TCGMemOpIdx oi, uintptr_t retaddr) { - unsigned mmu_idx =3D get_mmuidx(oi); - int index =3D (addr >> TARGET_PAGE_BITS) & (CPU_TLB_SIZE - 1); - target_ulong tlb_addr =3D - atomic_read(&env->tlb_table[mmu_idx][index].addr_write); + uintptr_t mmu_idx =3D get_mmuidx(oi); + uintptr_t index =3D tlb_index(env, mmu_idx, addr); + CPUTLBEntry *entry =3D tlb_entry(env, mmu_idx, addr); + target_ulong tlb_addr =3D atomic_read(&entry->addr_write); unsigned a_bits =3D get_alignment_bits(get_memop(oi)); uintptr_t haddr; =20 @@ -293,8 +295,7 @@ void helper_le_st_name(CPUArchState *env, target_ulong = addr, DATA_TYPE val, tlb_fill(ENV_GET_CPU(env), addr, DATA_SIZE, MMU_DATA_STORE, mmu_idx, retaddr); } - tlb_addr =3D atomic_read(&env->tlb_table[mmu_idx][index].addr_writ= e) & - ~TLB_INVALID_MASK; + tlb_addr =3D atomic_read(&entry->addr_write) & ~TLB_INVALID_MASK; } =20 /* Handle an IO access. */ @@ -315,16 +316,16 @@ void helper_le_st_name(CPUArchState *env, target_ulon= g addr, DATA_TYPE val, if (DATA_SIZE > 1 && unlikely((addr & ~TARGET_PAGE_MASK) + DATA_SIZE - 1 >=3D TARGET_PAGE_SIZE)) { - int i, index2; - target_ulong page2, tlb_addr2; + int i; + target_ulong page2; + CPUTLBEntry *entry2; do_unaligned_access: /* Ensure the second page is in the TLB. Note that the first page is already guaranteed to be filled, and that the second page cannot evict the first. */ page2 =3D (addr + DATA_SIZE) & TARGET_PAGE_MASK; - index2 =3D (page2 >> TARGET_PAGE_BITS) & (CPU_TLB_SIZE - 1); - tlb_addr2 =3D atomic_read(&env->tlb_table[mmu_idx][index2].addr_wr= ite); - if (!tlb_hit_page(tlb_addr2, page2) + entry2 =3D tlb_entry(env, mmu_idx, page2); + if (!tlb_hit_page(atomic_read(&entry2->addr_write), page2) && !VICTIM_TLB_HIT(addr_write, page2)) { tlb_fill(ENV_GET_CPU(env), page2, DATA_SIZE, MMU_DATA_STORE, mmu_idx, retaddr); @@ -342,7 +343,7 @@ void helper_le_st_name(CPUArchState *env, target_ulong = addr, DATA_TYPE val, return; } =20 - haddr =3D addr + env->tlb_table[mmu_idx][index].addend; + haddr =3D addr + entry->addend; #if DATA_SIZE =3D=3D 1 glue(glue(st, SUFFIX), _p)((uint8_t *)haddr, val); #else @@ -354,10 +355,10 @@ void helper_le_st_name(CPUArchState *env, target_ulon= g addr, DATA_TYPE val, void helper_be_st_name(CPUArchState *env, target_ulong addr, DATA_TYPE val, TCGMemOpIdx oi, uintptr_t retaddr) { - unsigned mmu_idx =3D get_mmuidx(oi); - int index =3D (addr >> TARGET_PAGE_BITS) & (CPU_TLB_SIZE - 1); - target_ulong tlb_addr =3D - atomic_read(&env->tlb_table[mmu_idx][index].addr_write); + uintptr_t mmu_idx =3D get_mmuidx(oi); + uintptr_t index =3D tlb_index(env, mmu_idx, addr); + CPUTLBEntry *entry =3D tlb_entry(env, mmu_idx, addr); + target_ulong tlb_addr =3D atomic_read(&entry->addr_write); unsigned a_bits =3D get_alignment_bits(get_memop(oi)); uintptr_t haddr; =20 @@ -372,8 +373,7 @@ void helper_be_st_name(CPUArchState *env, target_ulong = addr, DATA_TYPE val, tlb_fill(ENV_GET_CPU(env), addr, DATA_SIZE, MMU_DATA_STORE, mmu_idx, retaddr); } - tlb_addr =3D atomic_read(&env->tlb_table[mmu_idx][index].addr_writ= e) & - ~TLB_INVALID_MASK; + tlb_addr =3D atomic_read(&entry->addr_write) & ~TLB_INVALID_MASK; } =20 /* Handle an IO access. */ @@ -394,16 +394,16 @@ void helper_be_st_name(CPUArchState *env, target_ulon= g addr, DATA_TYPE val, if (DATA_SIZE > 1 && unlikely((addr & ~TARGET_PAGE_MASK) + DATA_SIZE - 1 >=3D TARGET_PAGE_SIZE)) { - int i, index2; - target_ulong page2, tlb_addr2; + int i; + target_ulong page2; + CPUTLBEntry *entry2; do_unaligned_access: /* Ensure the second page is in the TLB. Note that the first page is already guaranteed to be filled, and that the second page cannot evict the first. */ page2 =3D (addr + DATA_SIZE) & TARGET_PAGE_MASK; - index2 =3D (page2 >> TARGET_PAGE_BITS) & (CPU_TLB_SIZE - 1); - tlb_addr2 =3D atomic_read(&env->tlb_table[mmu_idx][index2].addr_wr= ite); - if (!tlb_hit_page(tlb_addr2, page2) + entry2 =3D tlb_entry(env, mmu_idx, page2); + if (!tlb_hit_page(atomic_read(&entry2->addr_write), page2) && !VICTIM_TLB_HIT(addr_write, page2)) { tlb_fill(ENV_GET_CPU(env), page2, DATA_SIZE, MMU_DATA_STORE, mmu_idx, retaddr); @@ -421,7 +421,7 @@ void helper_be_st_name(CPUArchState *env, target_ulong = addr, DATA_TYPE val, return; } =20 - haddr =3D addr + env->tlb_table[mmu_idx][index].addend; + haddr =3D addr + entry->addend; glue(glue(st, SUFFIX), _be_p)((uint8_t *)haddr, val); } #endif /* DATA_SIZE > 1 */ diff --git a/include/exec/cpu_ldst.h b/include/exec/cpu_ldst.h index 9581587ce1..e3d8d738aa 100644 --- a/include/exec/cpu_ldst.h +++ b/include/exec/cpu_ldst.h @@ -126,6 +126,20 @@ extern __thread uintptr_t helper_retaddr; /* The memory helpers for tcg-generated code need tcg_target_long etc. */ #include "tcg.h" =20 +/* Find the TLB index corresponding to the mmu_idx + address pair. */ +static inline uintptr_t tlb_index(CPUArchState *env, uintptr_t mmu_idx, + target_ulong addr) +{ + return (addr >> TARGET_PAGE_BITS) & (CPU_TLB_SIZE - 1); +} + +/* Find the TLB entry corresponding to the mmu_idx + address pair. */ +static inline CPUTLBEntry *tlb_entry(CPUArchState *env, uintptr_t mmu_idx, + target_ulong addr) +{ + return &env->tlb_table[mmu_idx][tlb_index(env, mmu_idx, addr)]; +} + #ifdef MMU_MODE0_SUFFIX #define CPU_MMU_INDEX 0 #define MEMSUFFIX MMU_MODE0_SUFFIX @@ -416,8 +430,7 @@ static inline void *tlb_vaddr_to_host(CPUArchState *env= , abi_ptr addr, #if defined(CONFIG_USER_ONLY) return g2h(addr); #else - int index =3D (addr >> TARGET_PAGE_BITS) & (CPU_TLB_SIZE - 1); - CPUTLBEntry *tlbentry =3D &env->tlb_table[mmu_idx][index]; + CPUTLBEntry *tlbentry =3D tlb_entry(env, mmu_idx, addr); abi_ptr tlb_addr; uintptr_t haddr; =20 @@ -445,7 +458,7 @@ static inline void *tlb_vaddr_to_host(CPUArchState *env= , abi_ptr addr, return NULL; } =20 - haddr =3D addr + env->tlb_table[mmu_idx][index].addend; + haddr =3D addr + tlbentry->addend; return (void *)haddr; #endif /* defined(CONFIG_USER_ONLY) */ } diff --git a/include/exec/cpu_ldst_template.h b/include/exec/cpu_ldst_templ= ate.h index ba7a11123c..924713eeed 100644 --- a/include/exec/cpu_ldst_template.h +++ b/include/exec/cpu_ldst_template.h @@ -81,7 +81,7 @@ glue(glue(glue(cpu_ld, USUFFIX), MEMSUFFIX), _ra)(CPUArch= State *env, target_ulong ptr, uintptr_t retaddr) { - int page_index; + CPUTLBEntry *entry; RES_TYPE res; target_ulong addr; int mmu_idx; @@ -94,15 +94,15 @@ glue(glue(glue(cpu_ld, USUFFIX), MEMSUFFIX), _ra)(CPUAr= chState *env, #endif =20 addr =3D ptr; - page_index =3D (addr >> TARGET_PAGE_BITS) & (CPU_TLB_SIZE - 1); mmu_idx =3D CPU_MMU_INDEX; - if (unlikely(env->tlb_table[mmu_idx][page_index].ADDR_READ !=3D + entry =3D tlb_entry(env, mmu_idx, addr); + if (unlikely(entry->ADDR_READ !=3D (addr & (TARGET_PAGE_MASK | (DATA_SIZE - 1))))) { oi =3D make_memop_idx(SHIFT, mmu_idx); res =3D glue(glue(helper_ret_ld, URETSUFFIX), MMUSUFFIX)(env, addr, oi, retaddr); } else { - uintptr_t hostaddr =3D addr + env->tlb_table[mmu_idx][page_index].= addend; + uintptr_t hostaddr =3D addr + entry->addend; res =3D glue(glue(ld, USUFFIX), _p)((uint8_t *)hostaddr); } return res; @@ -120,7 +120,8 @@ glue(glue(glue(cpu_lds, SUFFIX), MEMSUFFIX), _ra)(CPUAr= chState *env, target_ulong ptr, uintptr_t retaddr) { - int res, page_index; + CPUTLBEntry *entry; + int res; target_ulong addr; int mmu_idx; TCGMemOpIdx oi; @@ -132,15 +133,15 @@ glue(glue(glue(cpu_lds, SUFFIX), MEMSUFFIX), _ra)(CPU= ArchState *env, #endif =20 addr =3D ptr; - page_index =3D (addr >> TARGET_PAGE_BITS) & (CPU_TLB_SIZE - 1); mmu_idx =3D CPU_MMU_INDEX; - if (unlikely(env->tlb_table[mmu_idx][page_index].ADDR_READ !=3D + entry =3D tlb_entry(env, mmu_idx, addr); + if (unlikely(entry->ADDR_READ !=3D (addr & (TARGET_PAGE_MASK | (DATA_SIZE - 1))))) { oi =3D make_memop_idx(SHIFT, mmu_idx); res =3D (DATA_STYPE)glue(glue(helper_ret_ld, SRETSUFFIX), MMUSUFFIX)(env, addr, oi, retaddr); } else { - uintptr_t hostaddr =3D addr + env->tlb_table[mmu_idx][page_index].= addend; + uintptr_t hostaddr =3D addr + entry->addend; res =3D glue(glue(lds, SUFFIX), _p)((uint8_t *)hostaddr); } return res; @@ -162,7 +163,7 @@ glue(glue(glue(cpu_st, SUFFIX), MEMSUFFIX), _ra)(CPUArc= hState *env, target_ulong ptr, RES_TYPE v, uintptr_t ret= addr) { - int page_index; + CPUTLBEntry *entry; target_ulong addr; int mmu_idx; TCGMemOpIdx oi; @@ -174,15 +175,15 @@ glue(glue(glue(cpu_st, SUFFIX), MEMSUFFIX), _ra)(CPUA= rchState *env, #endif =20 addr =3D ptr; - page_index =3D (addr >> TARGET_PAGE_BITS) & (CPU_TLB_SIZE - 1); mmu_idx =3D CPU_MMU_INDEX; - if (unlikely(atomic_read(&env->tlb_table[mmu_idx][page_index].addr_wri= te) !=3D + entry =3D tlb_entry(env, mmu_idx, addr); + if (unlikely(atomic_read(&entry->addr_write) !=3D (addr & (TARGET_PAGE_MASK | (DATA_SIZE - 1))))) { oi =3D make_memop_idx(SHIFT, mmu_idx); glue(glue(helper_ret_st, SUFFIX), MMUSUFFIX)(env, addr, v, oi, retaddr); } else { - uintptr_t hostaddr =3D addr + env->tlb_table[mmu_idx][page_index].= addend; + uintptr_t hostaddr =3D addr + entry->addend; glue(glue(st, SUFFIX), _p)((uint8_t *)hostaddr, v); } } diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c index 200444142a..a5972773de 100644 --- a/accel/tcg/cputlb.c +++ b/accel/tcg/cputlb.c @@ -286,7 +286,6 @@ static void tlb_flush_page_async_work(CPUState *cpu, ru= n_on_cpu_data data) { CPUArchState *env =3D cpu->env_ptr; target_ulong addr =3D (target_ulong) data.target_ptr; - int i; int mmu_idx; =20 assert_cpu_is_self(cpu); @@ -304,10 +303,9 @@ static void tlb_flush_page_async_work(CPUState *cpu, r= un_on_cpu_data data) } =20 addr &=3D TARGET_PAGE_MASK; - i =3D (addr >> TARGET_PAGE_BITS) & (CPU_TLB_SIZE - 1); qemu_spin_lock(&env->tlb_lock); for (mmu_idx =3D 0; mmu_idx < NB_MMU_MODES; mmu_idx++) { - tlb_flush_entry_locked(&env->tlb_table[mmu_idx][i], addr); + tlb_flush_entry_locked(tlb_entry(env, mmu_idx, addr), addr); tlb_flush_vtlb_page_locked(env, mmu_idx, addr); } qemu_spin_unlock(&env->tlb_lock); @@ -339,18 +337,17 @@ static void tlb_flush_page_by_mmuidx_async_work(CPUSt= ate *cpu, target_ulong addr_and_mmuidx =3D (target_ulong) data.target_ptr; target_ulong addr =3D addr_and_mmuidx & TARGET_PAGE_MASK; unsigned long mmu_idx_bitmap =3D addr_and_mmuidx & ALL_MMUIDX_BITS; - int page =3D (addr >> TARGET_PAGE_BITS) & (CPU_TLB_SIZE - 1); int mmu_idx; =20 assert_cpu_is_self(cpu); =20 - tlb_debug("page:%d addr:"TARGET_FMT_lx" mmu_idx:0x%lx\n", - page, addr, mmu_idx_bitmap); + tlb_debug("flush page addr:"TARGET_FMT_lx" mmu_idx:0x%lx\n", + addr, mmu_idx_bitmap); =20 qemu_spin_lock(&env->tlb_lock); for (mmu_idx =3D 0; mmu_idx < NB_MMU_MODES; mmu_idx++) { if (test_bit(mmu_idx, &mmu_idx_bitmap)) { - tlb_flush_entry_locked(&env->tlb_table[mmu_idx][page], addr); + tlb_flush_entry_locked(tlb_entry(env, mmu_idx, addr), addr); tlb_flush_vtlb_page_locked(env, mmu_idx, addr); } } @@ -554,16 +551,14 @@ static inline void tlb_set_dirty1_locked(CPUTLBEntry = *tlb_entry, void tlb_set_dirty(CPUState *cpu, target_ulong vaddr) { CPUArchState *env =3D cpu->env_ptr; - int i; int mmu_idx; =20 assert_cpu_is_self(cpu); =20 vaddr &=3D TARGET_PAGE_MASK; - i =3D (vaddr >> TARGET_PAGE_BITS) & (CPU_TLB_SIZE - 1); qemu_spin_lock(&env->tlb_lock); for (mmu_idx =3D 0; mmu_idx < NB_MMU_MODES; mmu_idx++) { - tlb_set_dirty1_locked(&env->tlb_table[mmu_idx][i], vaddr); + tlb_set_dirty1_locked(tlb_entry(env, mmu_idx, vaddr), vaddr); } =20 for (mmu_idx =3D 0; mmu_idx < NB_MMU_MODES; mmu_idx++) { @@ -663,8 +658,8 @@ void tlb_set_page_with_attrs(CPUState *cpu, target_ulon= g vaddr, iotlb =3D memory_region_section_get_iotlb(cpu, section, vaddr_page, paddr_page, xlat, prot, &addre= ss); =20 - index =3D (vaddr_page >> TARGET_PAGE_BITS) & (CPU_TLB_SIZE - 1); - te =3D &env->tlb_table[mmu_idx][index]; + index =3D tlb_index(env, mmu_idx, vaddr_page); + te =3D tlb_entry(env, mmu_idx, vaddr_page); =20 /* * Hold the TLB lock for the rest of the function. We could acquire/re= lease @@ -786,16 +781,16 @@ static uint64_t io_readx(CPUArchState *env, CPUIOTLBE= ntry *iotlbentry, * repeat the MMU check here. This tlb_fill() call might * longjump out if this access should cause a guest exception. */ - int index; + CPUTLBEntry *entry; target_ulong tlb_addr; =20 tlb_fill(cpu, addr, size, MMU_DATA_LOAD, mmu_idx, retaddr); =20 - index =3D (addr >> TARGET_PAGE_BITS) & (CPU_TLB_SIZE - 1); - tlb_addr =3D env->tlb_table[mmu_idx][index].addr_read; + entry =3D tlb_entry(env, mmu_idx, addr); + tlb_addr =3D entry->addr_read; if (!(tlb_addr & ~(TARGET_PAGE_MASK | TLB_RECHECK))) { /* RAM access */ - uintptr_t haddr =3D addr + env->tlb_table[mmu_idx][index].adde= nd; + uintptr_t haddr =3D addr + entry->addend; =20 return ldn_p((void *)haddr, size); } @@ -853,16 +848,16 @@ static void io_writex(CPUArchState *env, CPUIOTLBEntr= y *iotlbentry, * repeat the MMU check here. This tlb_fill() call might * longjump out if this access should cause a guest exception. */ - int index; + CPUTLBEntry *entry; target_ulong tlb_addr; =20 tlb_fill(cpu, addr, size, MMU_DATA_STORE, mmu_idx, retaddr); =20 - index =3D (addr >> TARGET_PAGE_BITS) & (CPU_TLB_SIZE - 1); - tlb_addr =3D atomic_read(&env->tlb_table[mmu_idx][index].addr_writ= e); + entry =3D tlb_entry(env, mmu_idx, addr); + tlb_addr =3D atomic_read(&entry->addr_write); if (!(tlb_addr & ~(TARGET_PAGE_MASK | TLB_RECHECK))) { /* RAM access */ - uintptr_t haddr =3D addr + env->tlb_table[mmu_idx][index].adde= nd; + uintptr_t haddr =3D addr + entry->addend; =20 stn_p((void *)haddr, size, val); return; @@ -943,20 +938,19 @@ static bool victim_tlb_hit(CPUArchState *env, size_t = mmu_idx, size_t index, */ tb_page_addr_t get_page_addr_code(CPUArchState *env, target_ulong addr) { - int mmu_idx, index; + uintptr_t mmu_idx =3D cpu_mmu_index(env, true); + uintptr_t index =3D tlb_index(env, mmu_idx, addr); + CPUTLBEntry *entry =3D tlb_entry(env, mmu_idx, addr); void *p; =20 - index =3D (addr >> TARGET_PAGE_BITS) & (CPU_TLB_SIZE - 1); - mmu_idx =3D cpu_mmu_index(env, true); - if (unlikely(!tlb_hit(env->tlb_table[mmu_idx][index].addr_code, addr))= ) { + if (unlikely(!tlb_hit(entry->addr_code, addr))) { if (!VICTIM_TLB_HIT(addr_code, addr)) { tlb_fill(ENV_GET_CPU(env), addr, 0, MMU_INST_FETCH, mmu_idx, 0= ); } - assert(tlb_hit(env->tlb_table[mmu_idx][index].addr_code, addr)); + assert(tlb_hit(entry->addr_code, addr)); } =20 - if (unlikely(env->tlb_table[mmu_idx][index].addr_code & - (TLB_RECHECK | TLB_MMIO))) { + if (unlikely(entry->addr_code & (TLB_RECHECK | TLB_MMIO))) { /* * Return -1 if we can't translate and execute from an entire * page of RAM here, which will cause us to execute by loading @@ -968,7 +962,7 @@ tb_page_addr_t get_page_addr_code(CPUArchState *env, ta= rget_ulong addr) return -1; } =20 - p =3D (void *)((uintptr_t)addr + env->tlb_table[mmu_idx][index].addend= ); + p =3D (void *)((uintptr_t)addr + entry->addend); return qemu_ram_addr_from_host_nofail(p); } =20 @@ -981,11 +975,10 @@ tb_page_addr_t get_page_addr_code(CPUArchState *env, = target_ulong addr) void probe_write(CPUArchState *env, target_ulong addr, int size, int mmu_i= dx, uintptr_t retaddr) { - int index =3D (addr >> TARGET_PAGE_BITS) & (CPU_TLB_SIZE - 1); - target_ulong tlb_addr =3D - atomic_read(&env->tlb_table[mmu_idx][index].addr_write); + uintptr_t index =3D tlb_index(env, mmu_idx, addr); + CPUTLBEntry *entry =3D tlb_entry(env, mmu_idx, addr); =20 - if (!tlb_hit(tlb_addr, addr)) { + if (!tlb_hit(atomic_read(&entry->addr_write), addr)) { /* TLB entry is for a different page */ if (!VICTIM_TLB_HIT(addr_write, addr)) { tlb_fill(ENV_GET_CPU(env), addr, size, MMU_DATA_STORE, @@ -1001,8 +994,8 @@ static void *atomic_mmu_lookup(CPUArchState *env, targ= et_ulong addr, NotDirtyInfo *ndi) { size_t mmu_idx =3D get_mmuidx(oi); - size_t index =3D (addr >> TARGET_PAGE_BITS) & (CPU_TLB_SIZE - 1); - CPUTLBEntry *tlbe =3D &env->tlb_table[mmu_idx][index]; + uintptr_t index =3D tlb_index(env, mmu_idx, addr); + CPUTLBEntry *tlbe =3D tlb_entry(env, mmu_idx, addr); target_ulong tlb_addr =3D atomic_read(&tlbe->addr_write); TCGMemOp mop =3D get_memop(oi); int a_bits =3D get_alignment_bits(mop); --=20 2.17.1 From nobody Thu Nov 6 01:17:26 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 153910813556096.22380977043508; Tue, 9 Oct 2018 11:02:15 -0700 (PDT) Received: from localhost ([::1]:53223 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1g9wKk-0006dh-GC for importer@patchew.org; Tue, 09 Oct 2018 14:02:14 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:54272) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1g9wAU-0006lO-Sx for qemu-devel@nongnu.org; Tue, 09 Oct 2018 13:51:40 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1g9wAR-0002F5-NA for qemu-devel@nongnu.org; Tue, 09 Oct 2018 13:51:38 -0400 Received: from wout1-smtp.messagingengine.com ([64.147.123.24]:35603) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1g9wAR-0002DW-96 for qemu-devel@nongnu.org; Tue, 09 Oct 2018 13:51:35 -0400 Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.west.internal (Postfix) with ESMTP id E2ADC8DC; Tue, 9 Oct 2018 13:51:33 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute4.internal (MEProxy); Tue, 09 Oct 2018 13:51:34 -0400 Received: from localhost (flamenco.cs.columbia.edu [128.59.20.216]) by mail.messagingengine.com (Postfix) with ESMTPA id 392B2E461F; Tue, 9 Oct 2018 13:51:33 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=braap.org; h= from:to:cc:subject:date:message-id:in-reply-to:references; s= mesmtp; bh=lRVR2jlWZ4ZEB6lHBjOQ7Vz/GqSwBz43R7s/6BEmuqI=; b=ro0z/ xU3l8+/CD4WSmbUGDi7yGX3pzOJCE2Sa7nnuNwoYAkN7QsgIv7J2IM8eElOL0anh bV1h930PhkKr9F+PKpXNEPywJ6cEt6Z3JYKqlmjSeEtUdKMty8Pqng6hXOZmv6wq ErvIuxl9lPxDcI7SBSB1AN9K32GB04IBqAuisQ= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:date:from:in-reply-to:message-id :references:subject:to:x-me-proxy:x-me-proxy:x-me-sender :x-me-sender:x-sasl-enc; s=fm1; bh=lRVR2jlWZ4ZEB6lHBjOQ7Vz/GqSwB z43R7s/6BEmuqI=; b=D6eJrZEeVQRUm6M9iFek+k+v7lk/sjS3QL6MvpmgeOoGR EakC+HQJfh+5U1e4SJI4hWRhRlkeIhEfXqHMQd/Z8sLdAMlPJJfYGhbCvrogWkNu ds0ZnpeOk+1S9lsHGM2IHSf5ziGYjyBr6XgNgT76rWX2fSqjZdLIYH01MQ5i04iO J4zlStdggblKKGoVPTvLt6Ug1Vpl6vLq+fjIbMwNWyIQhDqeaZuLDwiF+0vMOWAT 45b+AveTwtRIhzEG1fgVyS5oTpdp8R+k2Qp5q+ZI6voA0AMDS07ob9JdSYcDABPW hvstTP9cMfrzXkZyCRBvVoWkdlBnmqqy4ebsmvrPA== X-ME-Sender: X-ME-Proxy: From: "Emilio G. Cota" To: qemu-devel@nongnu.org Date: Tue, 9 Oct 2018 13:51:26 -0400 Message-Id: <20181009175129.17888-3-cota@braap.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181009175129.17888-1-cota@braap.org> References: <20181009175129.17888-1-cota@braap.org> X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 64.147.123.24 Subject: [Qemu-devel] [RFC v3 2/5] (XXX) cputlb: introduce indirection for TLB size X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: =?UTF-8?q?Alex=20Benn=C3=A9e?= , Richard Henderson Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" This paves the way for implementing dynamic TLB resizing. XXX: convert other TCG backends Signed-off-by: Emilio G. Cota --- include/exec/cpu-defs.h | 10 ++++++---- include/exec/cpu_ldst.h | 14 +++++++++++++- accel/tcg/cputlb.c | 18 +++++++++++++++--- tcg/i386/tcg-target.inc.c | 26 +++++++++++++------------- 4 files changed, 47 insertions(+), 21 deletions(-) diff --git a/include/exec/cpu-defs.h b/include/exec/cpu-defs.h index 4ff62f32bf..87cd015f60 100644 --- a/include/exec/cpu-defs.h +++ b/include/exec/cpu-defs.h @@ -141,13 +141,15 @@ typedef struct CPUIOTLBEntry { MemTxAttrs attrs; } CPUIOTLBEntry; =20 -#define CPU_COMMON_TLB \ +#define CPU_COMMON_TLB \ /* The meaning of the MMU modes is defined in the target code. */ \ - /* tlb_lock serializes updates to tlb_table and tlb_v_table */ \ + /* tlb_lock serializes updates to tlb_mask, tlb_table and tlb_v_table = */ \ QemuSpin tlb_lock; \ - CPUTLBEntry tlb_table[NB_MMU_MODES][CPU_TLB_SIZE]; \ + /* tlb_mask[i] contains (n_entries - 1) << CPU_TLB_ENTRY_BITS */ \ + uintptr_t tlb_mask[NB_MMU_MODES]; \ + CPUTLBEntry *tlb_table[NB_MMU_MODES]; \ CPUTLBEntry tlb_v_table[NB_MMU_MODES][CPU_VTLB_SIZE]; \ - CPUIOTLBEntry iotlb[NB_MMU_MODES][CPU_TLB_SIZE]; \ + CPUIOTLBEntry *iotlb[NB_MMU_MODES]; \ CPUIOTLBEntry iotlb_v[NB_MMU_MODES][CPU_VTLB_SIZE]; \ size_t tlb_flush_count; \ target_ulong tlb_flush_addr; \ diff --git a/include/exec/cpu_ldst.h b/include/exec/cpu_ldst.h index e3d8d738aa..3ded1df9b7 100644 --- a/include/exec/cpu_ldst.h +++ b/include/exec/cpu_ldst.h @@ -130,7 +130,9 @@ extern __thread uintptr_t helper_retaddr; static inline uintptr_t tlb_index(CPUArchState *env, uintptr_t mmu_idx, target_ulong addr) { - return (addr >> TARGET_PAGE_BITS) & (CPU_TLB_SIZE - 1); + uintptr_t size_mask =3D env->tlb_mask[mmu_idx] >> CPU_TLB_ENTRY_BITS; + + return (addr >> TARGET_PAGE_BITS) & size_mask; } =20 /* Find the TLB entry corresponding to the mmu_idx + address pair. */ @@ -140,6 +142,16 @@ static inline CPUTLBEntry *tlb_entry(CPUArchState *env= , uintptr_t mmu_idx, return &env->tlb_table[mmu_idx][tlb_index(env, mmu_idx, addr)]; } =20 +static inline size_t sizeof_tlb(CPUArchState *env, uintptr_t mmu_idx) +{ + return env->tlb_mask[mmu_idx] + (1 << CPU_TLB_ENTRY_BITS); +} + +static inline size_t tlb_n_entries(CPUArchState *env, uintptr_t mmu_idx) +{ + return (env->tlb_mask[mmu_idx] >> CPU_TLB_ENTRY_BITS) + 1; +} + #ifdef MMU_MODE0_SUFFIX #define CPU_MMU_INDEX 0 #define MEMSUFFIX MMU_MODE0_SUFFIX diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c index a5972773de..80406f1033 100644 --- a/accel/tcg/cputlb.c +++ b/accel/tcg/cputlb.c @@ -76,8 +76,16 @@ QEMU_BUILD_BUG_ON(NB_MMU_MODES > 16); void tlb_init(CPUState *cpu) { CPUArchState *env =3D cpu->env_ptr; + int i; =20 qemu_spin_init(&env->tlb_lock); + for (i =3D 0; i < NB_MMU_MODES; i++) { + size_t n_entries =3D CPU_TLB_SIZE; + + env->tlb_mask[i] =3D (n_entries - 1) << CPU_TLB_ENTRY_BITS; + env->tlb_table[i] =3D g_new(CPUTLBEntry, n_entries); + env->iotlb[i] =3D g_new0(CPUIOTLBEntry, n_entries); + } } =20 /* flush_all_helper: run fn across all cpus @@ -120,6 +128,7 @@ size_t tlb_flush_count(void) static void tlb_flush_nocheck(CPUState *cpu) { CPUArchState *env =3D cpu->env_ptr; + int i; =20 /* The QOM tests will trigger tlb_flushes without setting up TCG * so we bug out here in that case. @@ -139,7 +148,9 @@ static void tlb_flush_nocheck(CPUState *cpu) * that do not hold the lock are performed by the same owner thread. */ qemu_spin_lock(&env->tlb_lock); - memset(env->tlb_table, -1, sizeof(env->tlb_table)); + for (i =3D 0; i < NB_MMU_MODES; i++) { + memset(env->tlb_table[i], -1, sizeof_tlb(env, i)); + } memset(env->tlb_v_table, -1, sizeof(env->tlb_v_table)); qemu_spin_unlock(&env->tlb_lock); =20 @@ -200,7 +211,7 @@ static void tlb_flush_by_mmuidx_async_work(CPUState *cp= u, run_on_cpu_data data) if (test_bit(mmu_idx, &mmu_idx_bitmask)) { tlb_debug("%d\n", mmu_idx); =20 - memset(env->tlb_table[mmu_idx], -1, sizeof(env->tlb_table[0])); + memset(env->tlb_table[mmu_idx], -1, sizeof_tlb(env, mmu_idx)); memset(env->tlb_v_table[mmu_idx], -1, sizeof(env->tlb_v_table[= 0])); } } @@ -523,8 +534,9 @@ void tlb_reset_dirty(CPUState *cpu, ram_addr_t start1, = ram_addr_t length) qemu_spin_lock(&env->tlb_lock); for (mmu_idx =3D 0; mmu_idx < NB_MMU_MODES; mmu_idx++) { unsigned int i; + unsigned int n =3D tlb_n_entries(env, mmu_idx); =20 - for (i =3D 0; i < CPU_TLB_SIZE; i++) { + for (i =3D 0; i < n; i++) { tlb_reset_dirty_range_locked(&env->tlb_table[mmu_idx][i], star= t1, length); } diff --git a/tcg/i386/tcg-target.inc.c b/tcg/i386/tcg-target.inc.c index 436195894b..91bbabd6e8 100644 --- a/tcg/i386/tcg-target.inc.c +++ b/tcg/i386/tcg-target.inc.c @@ -330,6 +330,7 @@ static inline int tcg_target_const_match(tcg_target_lon= g val, TCGType type, #define OPC_ARITH_GvEv (0x03) /* ... plus (ARITH_FOO << 3) */ #define OPC_ANDN (0xf2 | P_EXT38) #define OPC_ADD_GvEv (OPC_ARITH_GvEv | (ARITH_ADD << 3)) +#define OPC_AND_GvEv (OPC_ARITH_GvEv | (ARITH_AND << 3)) #define OPC_BLENDPS (0x0c | P_EXT3A | P_DATA16) #define OPC_BSF (0xbc | P_EXT) #define OPC_BSR (0xbd | P_EXT) @@ -1633,6 +1634,15 @@ static inline void tcg_out_tlb_load(TCGContext *s, T= CGReg addrlo, TCGReg addrhi, } =20 tcg_out_mov(s, tlbtype, r0, addrlo); + tcg_out_shifti(s, SHIFT_SHR + tlbrexw, r0, + TARGET_PAGE_BITS - CPU_TLB_ENTRY_BITS); + + tcg_out_modrm_offset(s, OPC_AND_GvEv + trexw, r0, TCG_AREG0, + offsetof(CPUArchState, tlb_mask[mem_index])); + + tcg_out_modrm_offset(s, OPC_ADD_GvEv + hrexw, r0, TCG_AREG0, + offsetof(CPUArchState, tlb_table[mem_index])); + /* If the required alignment is at least as large as the access, simply copy the address and mask. For lesser alignments, check that we do= n't cross pages for the complete access. */ @@ -1642,20 +1652,10 @@ static inline void tcg_out_tlb_load(TCGContext *s, = TCGReg addrlo, TCGReg addrhi, tcg_out_modrm_offset(s, OPC_LEA + trexw, r1, addrlo, s_mask - a_ma= sk); } tlb_mask =3D (target_ulong)TARGET_PAGE_MASK | a_mask; - - tcg_out_shifti(s, SHIFT_SHR + tlbrexw, r0, - TARGET_PAGE_BITS - CPU_TLB_ENTRY_BITS); - tgen_arithi(s, ARITH_AND + trexw, r1, tlb_mask, 0); - tgen_arithi(s, ARITH_AND + tlbrexw, r0, - (CPU_TLB_SIZE - 1) << CPU_TLB_ENTRY_BITS, 0); - - tcg_out_modrm_sib_offset(s, OPC_LEA + hrexw, r0, TCG_AREG0, r0, 0, - offsetof(CPUArchState, tlb_table[mem_index][0= ]) - + which); =20 /* cmp 0(r0), r1 */ - tcg_out_modrm_offset(s, OPC_CMP_GvEv + trexw, r1, r0, 0); + tcg_out_modrm_offset(s, OPC_CMP_GvEv + trexw, r1, r0, which); =20 /* Prepare for both the fast path add of the tlb addend, and the slow path function argument setup. There are two cases worth note: @@ -1672,7 +1672,7 @@ static inline void tcg_out_tlb_load(TCGContext *s, TC= GReg addrlo, TCGReg addrhi, =20 if (TARGET_LONG_BITS > TCG_TARGET_REG_BITS) { /* cmp 4(r0), addrhi */ - tcg_out_modrm_offset(s, OPC_CMP_GvEv, addrhi, r0, 4); + tcg_out_modrm_offset(s, OPC_CMP_GvEv, addrhi, r0, which + 4); =20 /* jne slow_path */ tcg_out_opc(s, OPC_JCC_long + JCC_JNE, 0, 0, 0); @@ -1684,7 +1684,7 @@ static inline void tcg_out_tlb_load(TCGContext *s, TC= GReg addrlo, TCGReg addrhi, =20 /* add addend(r0), r1 */ tcg_out_modrm_offset(s, OPC_ADD_GvEv + hrexw, r1, r0, - offsetof(CPUTLBEntry, addend) - which); + offsetof(CPUTLBEntry, addend)); } =20 /* --=20 2.17.1 From nobody Thu Nov 6 01:17:26 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1539107618990410.45175005354554; Tue, 9 Oct 2018 10:53:38 -0700 (PDT) Received: from localhost ([::1]:53160 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1g9wCQ-00082G-1M for importer@patchew.org; Tue, 09 Oct 2018 13:53:38 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:54271) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1g9wAU-0006lN-Sp for qemu-devel@nongnu.org; Tue, 09 Oct 2018 13:51:39 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1g9wAR-0002FI-PP for qemu-devel@nongnu.org; Tue, 09 Oct 2018 13:51:38 -0400 Received: from wout1-smtp.messagingengine.com ([64.147.123.24]:36783) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1g9wAR-0002Dl-Bj for qemu-devel@nongnu.org; Tue, 09 Oct 2018 13:51:35 -0400 Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.west.internal (Postfix) with ESMTP id 254F68E0; Tue, 9 Oct 2018 13:51:34 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute4.internal (MEProxy); Tue, 09 Oct 2018 13:51:34 -0400 Received: from localhost (flamenco.cs.columbia.edu [128.59.20.216]) by mail.messagingengine.com (Postfix) with ESMTPA id 6FC46E461E; Tue, 9 Oct 2018 13:51:33 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=braap.org; h= from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-type:content-transfer-encoding; s=mesmtp; bh=kA9Y1GxErYNkPAF631IPcLOwQrgadlucUVhZPQas3PY=; b=yQc/b9AZ4OAD z0ZBwYBh+0XqBedmWftZEww/Im2D0XrS7j951NJLGqoJ/X2aK2ek+cxk16UwM6tI 5ShcUpFyaeoEmxhZAvEclZqONr8J0HOx7P3DxxxiM1d/8/Lfnea1HTXhp8pdoBog n9CyQam1Lj2Q7MvmgNNwwM5T9a8IC+M= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm1; bh=kA9Y1GxErYNkPAF631IPcLOwQrgadlucUVhZPQas3 PY=; b=rtWR7tMRIXW8eddNpot4pkH0Lf0enyn3pScuIiM68UdOV3gZAeALs/kFO o55g0KW62JKC6NMQbRBhO+Z/iUmzxNPpyaeAEoEGsIgU1iJb4vyspW5wLTLUba0H zKmPhrAOOxXWIIZmwxQ8CfqMaijcdSCggN0vSrhemBLYD8aZJ89tLWN48tBccf99 EiyqjpjdRblgq1tthtuZJrtx/OV4pD3S6Y7i6TytXzbF6VsPUy2mSTKQOp0QxiOC iQ7CmYIjsCh/kyP3oROpdX915IJjPB379lKHe4zOBldbOvh0Cqtf0e6hlNy7FejS H1oE42yH5N7qRqifLBe86PIUP4elg== X-ME-Sender: X-ME-Proxy: From: "Emilio G. Cota" To: qemu-devel@nongnu.org Date: Tue, 9 Oct 2018 13:51:27 -0400 Message-Id: <20181009175129.17888-4-cota@braap.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181009175129.17888-1-cota@braap.org> References: <20181009175129.17888-1-cota@braap.org> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 64.147.123.24 Subject: [Qemu-devel] [RFC v3 3/5] cputlb: do not evict empty entries to the vtlb X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: =?UTF-8?q?Alex=20Benn=C3=A9e?= , Richard Henderson Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Currently we evict an entry to the victim TLB when it doesn't match the current address. But it could be that there's no match because the current entry is empty (i.e. all -1's, for instance via tlb_flush). Do not evict the entry to the vtlb in that case. This change will help us keep track of the TLB's use rate. Reviewed-by: Alex Benn=C3=A9e Reviewed-by: Richard Henderson Signed-off-by: Emilio G. Cota --- include/exec/cpu-all.h | 9 +++++++++ accel/tcg/cputlb.c | 2 +- 2 files changed, 10 insertions(+), 1 deletion(-) diff --git a/include/exec/cpu-all.h b/include/exec/cpu-all.h index 117d2fbbca..e21140049b 100644 --- a/include/exec/cpu-all.h +++ b/include/exec/cpu-all.h @@ -362,6 +362,15 @@ static inline bool tlb_hit(target_ulong tlb_addr, targ= et_ulong addr) return tlb_hit_page(tlb_addr, addr & TARGET_PAGE_MASK); } =20 +/** + * tlb_entry_is_empty - return true if the entry is not in use + * @te: pointer to CPUTLBEntry + */ +static inline bool tlb_entry_is_empty(const CPUTLBEntry *te) +{ + return te->addr_read =3D=3D -1 && te->addr_write =3D=3D -1 && te->addr= _code =3D=3D -1; +} + void dump_exec_info(FILE *f, fprintf_function cpu_fprintf); void dump_opcount_info(FILE *f, fprintf_function cpu_fprintf); #endif /* !CONFIG_USER_ONLY */ diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c index 80406f1033..4dc47e603c 100644 --- a/accel/tcg/cputlb.c +++ b/accel/tcg/cputlb.c @@ -689,7 +689,7 @@ void tlb_set_page_with_attrs(CPUState *cpu, target_ulon= g vaddr, * Only evict the old entry to the victim tlb if it's for a * different page; otherwise just overwrite the stale data. */ - if (!tlb_hit_page_anyprot(te, vaddr_page)) { + if (!tlb_hit_page_anyprot(te, vaddr_page) && !tlb_entry_is_empty(te)) { unsigned vidx =3D env->vtlb_index++ % CPU_VTLB_SIZE; CPUTLBEntry *tv =3D &env->tlb_v_table[mmu_idx][vidx]; =20 --=20 2.17.1 From nobody Thu Nov 6 01:17:26 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1539107968628757.0346187941477; Tue, 9 Oct 2018 10:59:28 -0700 (PDT) Received: from localhost ([::1]:53201 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1g9wI2-0004UO-6j for importer@patchew.org; Tue, 09 Oct 2018 13:59:26 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:54274) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1g9wAU-0006lP-TJ for qemu-devel@nongnu.org; Tue, 09 Oct 2018 13:51:40 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1g9wAS-0002FW-39 for qemu-devel@nongnu.org; Tue, 09 Oct 2018 13:51:38 -0400 Received: from wout1-smtp.messagingengine.com ([64.147.123.24]:55387) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1g9wAR-0002EB-P3 for qemu-devel@nongnu.org; Tue, 09 Oct 2018 13:51:35 -0400 Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.west.internal (Postfix) with ESMTP id 681C78E2; Tue, 9 Oct 2018 13:51:34 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute4.internal (MEProxy); Tue, 09 Oct 2018 13:51:34 -0400 Received: from localhost (flamenco.cs.columbia.edu [128.59.20.216]) by mail.messagingengine.com (Postfix) with ESMTPA id A2E7FE4799; Tue, 9 Oct 2018 13:51:33 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=braap.org; h= from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-type:content-transfer-encoding; s=mesmtp; bh=EyJP6DVd6xofEFskD+JrmABpv7irbRFumHGjByJcWnE=; b=X5QqpgHYlyUX vyWBE+QPMpEzQPeTAi4YkKU8uq5GiFOYNmjq6x9JgOwGqteba1KhMOE2sc00fzb5 tY3ZjudwnESTASik7eFaTEeZoatd9RLQ4NKdDNNGhoE7uA0JSGV/QjIL0UPeiOk5 k0VjCDgmgANNW5V5nnOfip2NFHQODNA= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm1; bh=EyJP6DVd6xofEFskD+JrmABpv7irbRFumHGjByJcW nE=; b=L851rdkgeJzrvitwdpVUQYiFZ33X5sOUhpdQYP3Wd/7LN+dcY1MYEgfNc KWW5P7qLd7hQyfQ4v4noRRevmIoitGfAAspZCQPvYTdy9urULAhifrUTAW62CXR1 KeR3sAFtF+EcJGiTC4bSNxkU8Hrg2ljggM3q66SW7wlI4kNxfYeRLbOuH4ZtTTWO NaEB1J+uh56H0Yw9uuxKVSMs9NybpGhQElDZmjjpPJ8wskf/1KJC4vslZ9WUZDbY t2Vy7Dz5ZbWQ2BirHlWVJwjp9kOuD65Xj9HV55vVmsfFoiQ3oeY0R794z/WcIp0O O6qYY8LB+vE8A1LliIRJp2rYJfjHQ== X-ME-Sender: X-ME-Proxy: From: "Emilio G. Cota" To: qemu-devel@nongnu.org Date: Tue, 9 Oct 2018 13:51:28 -0400 Message-Id: <20181009175129.17888-5-cota@braap.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181009175129.17888-1-cota@braap.org> References: <20181009175129.17888-1-cota@braap.org> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 64.147.123.24 Subject: [Qemu-devel] [RFC v3 4/5] cputlb: track TLB use rate X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: =?UTF-8?q?Alex=20Benn=C3=A9e?= , Richard Henderson Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 This paves the way for implementing a dynamically-sized softmmu. Reviewed-by: Alex Benn=C3=A9e Signed-off-by: Emilio G. Cota --- include/exec/cpu-defs.h | 5 +++++ accel/tcg/cputlb.c | 17 ++++++++++++++--- 2 files changed, 19 insertions(+), 3 deletions(-) diff --git a/include/exec/cpu-defs.h b/include/exec/cpu-defs.h index 87cd015f60..56f1887c7f 100644 --- a/include/exec/cpu-defs.h +++ b/include/exec/cpu-defs.h @@ -141,10 +141,15 @@ typedef struct CPUIOTLBEntry { MemTxAttrs attrs; } CPUIOTLBEntry; =20 +typedef struct CPUTLBDesc { + size_t n_used_entries; +} CPUTLBDesc; + #define CPU_COMMON_TLB \ /* The meaning of the MMU modes is defined in the target code. */ \ /* tlb_lock serializes updates to tlb_mask, tlb_table and tlb_v_table = */ \ QemuSpin tlb_lock; \ + CPUTLBDesc tlb_desc[NB_MMU_MODES]; \ /* tlb_mask[i] contains (n_entries - 1) << CPU_TLB_ENTRY_BITS */ \ uintptr_t tlb_mask[NB_MMU_MODES]; \ CPUTLBEntry *tlb_table[NB_MMU_MODES]; \ diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c index 4dc47e603c..11d6060eb0 100644 --- a/accel/tcg/cputlb.c +++ b/accel/tcg/cputlb.c @@ -82,6 +82,7 @@ void tlb_init(CPUState *cpu) for (i =3D 0; i < NB_MMU_MODES; i++) { size_t n_entries =3D CPU_TLB_SIZE; =20 + env->tlb_desc[i].n_used_entries =3D 0; env->tlb_mask[i] =3D (n_entries - 1) << CPU_TLB_ENTRY_BITS; env->tlb_table[i] =3D g_new(CPUTLBEntry, n_entries); env->iotlb[i] =3D g_new0(CPUIOTLBEntry, n_entries); @@ -150,6 +151,7 @@ static void tlb_flush_nocheck(CPUState *cpu) qemu_spin_lock(&env->tlb_lock); for (i =3D 0; i < NB_MMU_MODES; i++) { memset(env->tlb_table[i], -1, sizeof_tlb(env, i)); + env->tlb_desc[i].n_used_entries =3D 0; } memset(env->tlb_v_table, -1, sizeof(env->tlb_v_table)); qemu_spin_unlock(&env->tlb_lock); @@ -213,6 +215,7 @@ static void tlb_flush_by_mmuidx_async_work(CPUState *cp= u, run_on_cpu_data data) =20 memset(env->tlb_table[mmu_idx], -1, sizeof_tlb(env, mmu_idx)); memset(env->tlb_v_table[mmu_idx], -1, sizeof(env->tlb_v_table[= 0])); + env->tlb_desc[mmu_idx].n_used_entries =3D 0; } } qemu_spin_unlock(&env->tlb_lock); @@ -273,12 +276,14 @@ static inline bool tlb_hit_page_anyprot(CPUTLBEntry *= tlb_entry, } =20 /* Called with tlb_lock held */ -static inline void tlb_flush_entry_locked(CPUTLBEntry *tlb_entry, +static inline bool tlb_flush_entry_locked(CPUTLBEntry *tlb_entry, target_ulong page) { if (tlb_hit_page_anyprot(tlb_entry, page)) { memset(tlb_entry, -1, sizeof(*tlb_entry)); + return true; } + return false; } =20 /* Called with tlb_lock held */ @@ -316,7 +321,9 @@ static void tlb_flush_page_async_work(CPUState *cpu, ru= n_on_cpu_data data) addr &=3D TARGET_PAGE_MASK; qemu_spin_lock(&env->tlb_lock); for (mmu_idx =3D 0; mmu_idx < NB_MMU_MODES; mmu_idx++) { - tlb_flush_entry_locked(tlb_entry(env, mmu_idx, addr), addr); + if (tlb_flush_entry_locked(tlb_entry(env, mmu_idx, addr), addr)) { + env->tlb_desc[mmu_idx].n_used_entries--; + } tlb_flush_vtlb_page_locked(env, mmu_idx, addr); } qemu_spin_unlock(&env->tlb_lock); @@ -358,7 +365,9 @@ static void tlb_flush_page_by_mmuidx_async_work(CPUStat= e *cpu, qemu_spin_lock(&env->tlb_lock); for (mmu_idx =3D 0; mmu_idx < NB_MMU_MODES; mmu_idx++) { if (test_bit(mmu_idx, &mmu_idx_bitmap)) { - tlb_flush_entry_locked(tlb_entry(env, mmu_idx, addr), addr); + if (tlb_flush_entry_locked(tlb_entry(env, mmu_idx, addr), addr= )) { + env->tlb_desc[mmu_idx].n_used_entries--; + } tlb_flush_vtlb_page_locked(env, mmu_idx, addr); } } @@ -696,6 +705,7 @@ void tlb_set_page_with_attrs(CPUState *cpu, target_ulon= g vaddr, /* Evict the old entry into the victim tlb. */ copy_tlb_helper_locked(tv, te); env->iotlb_v[mmu_idx][vidx] =3D env->iotlb[mmu_idx][index]; + env->tlb_desc[mmu_idx].n_used_entries--; } =20 /* refill the tlb */ @@ -747,6 +757,7 @@ void tlb_set_page_with_attrs(CPUState *cpu, target_ulon= g vaddr, } =20 copy_tlb_helper_locked(te, &tn); + env->tlb_desc[mmu_idx].n_used_entries++; qemu_spin_unlock(&env->tlb_lock); } =20 --=20 2.17.1 From nobody Thu Nov 6 01:17:26 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1539107813435239.78574054806109; Tue, 9 Oct 2018 10:56:53 -0700 (PDT) Received: from localhost ([::1]:53187 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1g9wFY-0002O4-9C for importer@patchew.org; Tue, 09 Oct 2018 13:56:52 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:54269) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1g9wAU-0006lM-SH for qemu-devel@nongnu.org; Tue, 09 Oct 2018 13:51:40 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1g9wAS-0002GG-Ml for qemu-devel@nongnu.org; Tue, 09 Oct 2018 13:51:38 -0400 Received: from wout1-smtp.messagingengine.com ([64.147.123.24]:38227) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1g9wAS-0002F9-CA for qemu-devel@nongnu.org; Tue, 09 Oct 2018 13:51:36 -0400 Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.west.internal (Postfix) with ESMTP id 8FB26926; Tue, 9 Oct 2018 13:51:34 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute4.internal (MEProxy); Tue, 09 Oct 2018 13:51:34 -0400 Received: from localhost (flamenco.cs.columbia.edu [128.59.20.216]) by mail.messagingengine.com (Postfix) with ESMTPA id D8714E47CB; Tue, 9 Oct 2018 13:51:33 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=braap.org; h= from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-type:content-transfer-encoding; s=mesmtp; bh=X28E+fGSdGBEu191+vNPKPNtkcpWzqJUM5luQ+4enxA=; b=2fivhf94LKMH b+uELmYztIHqM5ccHLbrwY4a0FDD7w6X89BPOSyLOEaMXO0om6+C/KyvmRztqRvK i8U6CGbsJet8QP+LWlY7UJ28JNBZkwcz+4XmbiLU/xyTdkSBy3Wgifw2Lacz9zwW 5nNmu9pP3pxMsAdBeay0a1chn7QT5Wg= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm1; bh=X28E+fGSdGBEu191+vNPKPNtkcpWzqJUM5luQ+4en xA=; b=XRCs0Cng3AV1Jo9c75bosthYZxvWf4eWbQAU2LMsPwT6jiXaO37n8LF3W 05yEki51WnTZD7tzx+ZhqwW8vgUJ1O5SmsRr3zviUam7sTFbC0zCuLZinhVFQkeP MWgV14jbBH1QT7IKCzXWcZfNjob+ZHIfo/QNxFrCL8SfwNWCycfgvdm7fCXOePoD RCPSS40C8zK8VMt0htqcNcDxlBQlbky6W3u/UzCS4Auy5IjPtGrVLTLg1k9uWsCs m+k/dVD3x7eFDX0PKmJ3u4+QfNKK+e1YkhikP9U90NYL1FTuLswKjIbTS7vwcP+q u5hArJQrnWa5AHLFjzFO7exwEZ1cQ== X-ME-Sender: X-ME-Proxy: From: "Emilio G. Cota" To: qemu-devel@nongnu.org Date: Tue, 9 Oct 2018 13:51:29 -0400 Message-Id: <20181009175129.17888-6-cota@braap.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181009175129.17888-1-cota@braap.org> References: <20181009175129.17888-1-cota@braap.org> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 64.147.123.24 Subject: [Qemu-devel] [RFC v3 5/5] (XXX) cputlb: dynamically resize TLBs based on use rate X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: =?UTF-8?q?Alex=20Benn=C3=A9e?= , Richard Henderson Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 XXX: convert CPU_TLB_BITS/CPU_TLB_SIZE users in non-i386 TCG backends Perform the resizing only on flushes, otherwise we'd have to take a perf hit by either rehashing the array or unnecessarily flushing it. We grow the array aggressively, and reduce the size more slowly. This accommodates mixed workloads, where some processes might be memory-heavy while others are not. As the following experiments show, this a net perf gain, particularly for memory-heavy workloads. Experiments are run on an Intel i7-6700K CPU @ 4.00GHz. 1. System boot + shudown, debian aarch64: - Before (tb-lock-v3): Performance counter stats for 'taskset -c 0 ../img/aarch64/die.sh' (10 run= s): 7469.363393 task-clock (msec) # 0.998 CPUs utilized = ( +- 0.07% ) 31,507,707,190 cycles # 4.218 GHz = ( +- 0.07% ) 57,101,577,452 instructions # 1.81 insns per cycl= e ( +- 0.08% ) 10,265,531,804 branches # 1374.352 M/sec = ( +- 0.07% ) 173,020,681 branch-misses # 1.69% of all branche= s ( +- 0.10% ) 7.483359063 seconds time elapsed = ( +- 0.08% ) - After: Performance counter stats for 'taskset -c 0 ../img/aarch64/die.sh' (10 run= s): 7185.036730 task-clock (msec) # 0.999 CPUs utilized = ( +- 0.11% ) 30,303,501,143 cycles # 4.218 GHz = ( +- 0.11% ) 54,198,386,487 instructions # 1.79 insns per cycl= e ( +- 0.08% ) 9,726,518,945 branches # 1353.719 M/sec = ( +- 0.08% ) 167,082,307 branch-misses # 1.72% of all branche= s ( +- 0.08% ) 7.195597842 seconds time elapsed = ( +- 0.11% ) That is, a 3.8% improvement. 2. System boot + shutdown, ubuntu 18.04 x86_64: - Before (tb-lock-v3): Performance counter stats for 'taskset -c 0 ../img/x86_64/ubuntu-die.sh -no= graphic' (2 runs): 49971.036482 task-clock (msec) # 0.999 CPUs utilized = ( +- 1.62% ) 210,766,077,140 cycles # 4.218 GHz = ( +- 1.63% ) 428,829,830,790 instructions # 2.03 insns per cycl= e ( +- 0.75% ) 77,313,384,038 branches # 1547.164 M/sec = ( +- 0.54% ) 835,610,706 branch-misses # 1.08% of all branche= s ( +- 2.97% ) 50.003855102 seconds time elapsed = ( +- 1.61% ) - After: Performance counter stats for 'taskset -c 0 ../img/x86_64/ubuntu-die.sh -n= ographic' (2 runs): 50118.124477 task-clock (msec) # 0.999 CPUs utilized = ( +- 4.30% ) 132,396 context-switches # 0.003 M/sec = ( +- 1.20% ) 0 cpu-migrations # 0.000 K/sec = ( +-100.00% ) 167,754 page-faults # 0.003 M/sec = ( +- 0.06% ) 211,414,701,601 cycles # 4.218 GHz = ( +- 4.30% ) stalled-cycles-frontend stalled-cycles-backend 431,618,818,597 instructions # 2.04 insns per cycl= e ( +- 6.40% ) 80,197,256,524 branches # 1600.165 M/sec = ( +- 8.59% ) 794,830,352 branch-misses # 0.99% of all branche= s ( +- 2.05% ) 50.177077175 seconds time elapsed = ( +- 4.23% ) No improvement (within noise range). 3. x86_64 SPEC06int: SPEC06int (test set) [ Y axis: speedup over master ] 8 +-+--+----+----+-----+----+----+----+----+----+----+-----+----+----+--+= -+ | = | | tlb-lock-v3 = | 7 +-+..................$$$...........................+indirection += -+ | $ $ +resizing = | | $ $ = | 6 +-+..................$.$..............................................+= -+ | $ $ = | | $ $ = | 5 +-+..................$.$..............................................+= -+ | $ $ = | | $ $ = | 4 +-+..................$.$..............................................+= -+ | $ $ = | | +++ $ $ = | 3 +-+........$$+.......$.$..............................................+= -+ | $$ $ $ = | | $$ $ $ $$$ = | 2 +-+........$$........$.$.................................$.$..........+= -+ | $$ $ $ $ $ +$$ = | | $$ $$+ $ $ $$$ +$$ $ $ $$$ $$ = | 1 +-+***#$***#$+**#$+**#+$**#+$**##$**##$***#$***#$+**#$+**#+$**#+$**##$+= -+ | * *#$* *#$ **#$ **# $**# $** #$** #$* *#$* *#$ **#$ **# $**# $** #$ = | | * *#$* *#$ **#$ **# $**# $** #$** #$* *#$* *#$ **#$ **# $**# $** #$ = | 0 +-+***#$***#$-**#$-**#$$**#$$**##$**##$***#$***#$-**#$-**#$$**#$$**##$+= -+ 401.bzi403.gc429445.g456.h462.libq464.h471.omne4483.xalancbgeomean png: https://imgur.com/a/b1wn3wc That is, a 1.53x average speedup over master, with a max speedup of 7.13x. Note that "indirection" (i.e. the "cputlb: introduce indirection for TLB si= ze" patch in this series) incurs no overhead, on average. To conclude, here is a different look at the SPEC06int results, using linux-user as the baseline and comparing master and this series ("tlb-dyn"): Softmmu slowdown vs. linux-user for SPEC06int (test set) [ Y axis: slowdown over linux-user ] 14 +-+--+----+----+----+----+----+-----+----+----+----+----+----+----+--+= -+ | = | | master = | 12 +-+...............+**..................................tlb-dyn.......+= -+ | ** = | | ** = | | ** = | 10 +-+................**................................................+= -+ | ** = | | ** = | 8 +-+................**................................................+= -+ | ** = | | ** = | | ** = | 6 +-+................**................................................+= -+ | *** ** = | | * * ** = | 4 +-+.....*.*........**.................................***............+= -+ | * * ** * * = | | * * +++ ** *** *** * * *** *** = | | * * +**++ ** **## *+*# *** * *#+* * * *##* * = | 2 +-+.....*.*##.**##.**##.**.#.**##.*+*#.***#.*+*#.*.*#.*.*#+*.*.#*.*##+= -+ |++***##*+*+#+**+#+**+#+**+#+**+#+*+*#+*+*#+*+*#+*+*#+*+*#+*+*+#*+*+#+= +| | * * #* * # ** # ** # ** # ** # * *# * *# * *# * *# * *# * * #* * # = | 0 +-+***##***##-**##-**##-**##-**##-***#-***#-***#-***#-***#-***##***##+= -+ 401.bzi403.g429445.g456.hm462.libq464.h471.omn4483.xalancbgeomean png: https://imgur.com/a/eXkjMCE After this series, we bring down the average softmmu overhead from 2.77x to 1.80x, with a maximum slowdown of 2.48x (omnetpp). Reviewed-by: Alex Benn=C3=A9e Signed-off-by: Emilio G. Cota --- include/exec/cpu-defs.h | 39 +++++++++---------------------------- accel/tcg/cputlb.c | 41 +++++++++++++++++++++++++++++++++++++-- tcg/i386/tcg-target.inc.c | 2 +- 3 files changed, 49 insertions(+), 33 deletions(-) diff --git a/include/exec/cpu-defs.h b/include/exec/cpu-defs.h index 56f1887c7f..d4af0b2a2d 100644 --- a/include/exec/cpu-defs.h +++ b/include/exec/cpu-defs.h @@ -67,37 +67,15 @@ typedef uint64_t target_ulong; #define CPU_TLB_ENTRY_BITS 5 #endif =20 -/* TCG_TARGET_TLB_DISPLACEMENT_BITS is used in CPU_TLB_BITS to ensure that - * the TLB is not unnecessarily small, but still small enough for the - * TLB lookup instruction sequence used by the TCG target. - * - * TCG will have to generate an operand as large as the distance between - * env and the tlb_table[NB_MMU_MODES - 1][0].addend. For simplicity, - * the TCG targets just round everything up to the next power of two, and - * count bits. This works because: 1) the size of each TLB is a largish - * power of two, 2) and because the limit of the displacement is really cl= ose - * to a power of two, 3) the offset of tlb_table[0][0] inside env is small= er - * than the size of a TLB. - * - * For example, the maximum displacement 0xFFF0 on PPC and MIPS, but TCG - * just says "the displacement is 16 bits". TCG_TARGET_TLB_DISPLACEMENT_B= ITS - * then ensures that tlb_table at least 0x8000 bytes large ("not unnecessa= rily - * small": 2^15). The operand then will come up smaller than 0xFFF0 witho= ut - * any particular care, because the TLB for a single MMU mode is larger th= an - * 0x10000-0xFFF0=3D16 bytes. In the end, the maximum value of the operand - * could be something like 0xC000 (the offset of the last TLB table) plus - * 0x18 (the offset of the addend field in each TLB entry) plus the offset - * of tlb_table inside env (which is non-trivial but not huge). +#define MIN_CPU_TLB_BITS 6 +#define DEFAULT_CPU_TLB_BITS 8 +/* + * Assuming TARGET_PAGE_BITS=3D=3D12, with 2**22 entries we can cover 2**(= 22+12) =3D=3D + * 2**34 =3D=3D 16G of address space. This is roughly what one would expec= t a + * TLB to cover in a modern (as of 2018) x86_64 CPU. For instance, Intel + * Skylake's Level-2 STLB has 16 1G entries. */ -#define CPU_TLB_BITS \ - MIN(8, \ - TCG_TARGET_TLB_DISPLACEMENT_BITS - CPU_TLB_ENTRY_BITS - \ - (NB_MMU_MODES <=3D 1 ? 0 : \ - NB_MMU_MODES <=3D 2 ? 1 : \ - NB_MMU_MODES <=3D 4 ? 2 : \ - NB_MMU_MODES <=3D 8 ? 3 : 4)) - -#define CPU_TLB_SIZE (1 << CPU_TLB_BITS) +#define MAX_CPU_TLB_BITS 22 =20 typedef struct CPUTLBEntry { /* bit TARGET_LONG_BITS to TARGET_PAGE_BITS : virtual address @@ -143,6 +121,7 @@ typedef struct CPUIOTLBEntry { =20 typedef struct CPUTLBDesc { size_t n_used_entries; + size_t n_flushes_low_rate; } CPUTLBDesc; =20 #define CPU_COMMON_TLB \ diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c index 11d6060eb0..ebd5e43155 100644 --- a/accel/tcg/cputlb.c +++ b/accel/tcg/cputlb.c @@ -80,12 +80,13 @@ void tlb_init(CPUState *cpu) =20 qemu_spin_init(&env->tlb_lock); for (i =3D 0; i < NB_MMU_MODES; i++) { - size_t n_entries =3D CPU_TLB_SIZE; + size_t n_entries =3D 1 << DEFAULT_CPU_TLB_BITS; =20 env->tlb_desc[i].n_used_entries =3D 0; + env->tlb_desc[i].n_flushes_low_rate =3D 0; env->tlb_mask[i] =3D (n_entries - 1) << CPU_TLB_ENTRY_BITS; env->tlb_table[i] =3D g_new(CPUTLBEntry, n_entries); - env->iotlb[i] =3D g_new0(CPUIOTLBEntry, n_entries); + env->iotlb[i] =3D g_new(CPUIOTLBEntry, n_entries); } } =20 @@ -121,6 +122,40 @@ size_t tlb_flush_count(void) return count; } =20 +/* Call with tlb_lock held */ +static void tlb_mmu_resize_locked(CPUArchState *env, int mmu_idx) +{ + CPUTLBDesc *desc =3D &env->tlb_desc[mmu_idx]; + size_t old_size =3D tlb_n_entries(env, mmu_idx); + size_t rate =3D desc->n_used_entries * 100 / old_size; + size_t new_size =3D old_size; + + if (rate =3D=3D 100) { + new_size =3D MIN(old_size << 2, 1 << MAX_CPU_TLB_BITS); + } else if (rate > 70) { + new_size =3D MIN(old_size << 1, 1 << MAX_CPU_TLB_BITS); + } else if (rate < 30) { + desc->n_flushes_low_rate++; + if (desc->n_flushes_low_rate =3D=3D 100) { + new_size =3D MAX(old_size >> 1, 1 << MIN_CPU_TLB_BITS); + desc->n_flushes_low_rate =3D 0; + } + } + + if (new_size =3D=3D old_size) { + return; + } + + g_free(env->tlb_table[mmu_idx]); + g_free(env->iotlb[mmu_idx]); + + /* desc->n_used_entries is cleared by the caller */ + desc->n_flushes_low_rate =3D 0; + env->tlb_mask[mmu_idx] =3D (new_size - 1) << CPU_TLB_ENTRY_BITS; + env->tlb_table[mmu_idx] =3D g_new(CPUTLBEntry, new_size); + env->iotlb[mmu_idx] =3D g_new(CPUIOTLBEntry, new_size); +} + /* This is OK because CPU architectures generally permit an * implementation to drop entries from the TLB at any time, so * flushing more entries than required is only an efficiency issue, @@ -150,6 +185,7 @@ static void tlb_flush_nocheck(CPUState *cpu) */ qemu_spin_lock(&env->tlb_lock); for (i =3D 0; i < NB_MMU_MODES; i++) { + tlb_mmu_resize_locked(env, i); memset(env->tlb_table[i], -1, sizeof_tlb(env, i)); env->tlb_desc[i].n_used_entries =3D 0; } @@ -213,6 +249,7 @@ static void tlb_flush_by_mmuidx_async_work(CPUState *cp= u, run_on_cpu_data data) if (test_bit(mmu_idx, &mmu_idx_bitmask)) { tlb_debug("%d\n", mmu_idx); =20 + tlb_mmu_resize_locked(env, mmu_idx); memset(env->tlb_table[mmu_idx], -1, sizeof_tlb(env, mmu_idx)); memset(env->tlb_v_table[mmu_idx], -1, sizeof(env->tlb_v_table[= 0])); env->tlb_desc[mmu_idx].n_used_entries =3D 0; diff --git a/tcg/i386/tcg-target.inc.c b/tcg/i386/tcg-target.inc.c index 91bbabd6e8..55f2d2b71a 100644 --- a/tcg/i386/tcg-target.inc.c +++ b/tcg/i386/tcg-target.inc.c @@ -1626,7 +1626,7 @@ static inline void tcg_out_tlb_load(TCGContext *s, TC= GReg addrlo, TCGReg addrhi, } if (TCG_TYPE_PTR =3D=3D TCG_TYPE_I64) { hrexw =3D P_REXW; - if (TARGET_PAGE_BITS + CPU_TLB_BITS > 32) { + if (TARGET_PAGE_BITS + MAX_CPU_TLB_BITS > 32) { tlbtype =3D TCG_TYPE_I64; tlbrexw =3D P_REXW; } --=20 2.17.1