From nobody Mon Sep 16 19:31:22 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=linaro.org ARC-Seal: i=1; a=rsa-sha256; t=1694900612; cv=none; d=zohomail.com; s=zohoarc; b=V9GKCl5kTRI083BD3TVNvQ4ydcyQ8V4/fquTSrYHIJA+kDRZUtDcO+rTFQjnwQIVVS+VBGBSt4WXzdH3WfprTv7+ZfMls5JVQ2hmCP+F8jzt5B34BqPQr7Lc2z0TbX/I6eFZxiW/Yl8UHsnvc87pwf8oNccoNErf/RlM73OImTk= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1694900612; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=Eqdx5KYfe7+Tgpre/r7adTy9jlck+D44aSymYCDoQAc=; b=CnfF6AOLaQUTLmDJb5rb03Ds6JzxQ2mnPatKCENUDj+ml4pN+QsRkkkbyEnFxh/x5PRJSIHNnxyMR0U6+7Uc40n0lOc/SQKf6IWW6YQrMzh3LZxQ3LO2ugJ0aYFR6qNu196nb6mmsVdfK03C7hn2i1tQMJE7m8m29a2IsSZii8w= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1694900611976810.2966081747443; Sat, 16 Sep 2023 14:43:31 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qhd32-0005CU-DT; Sat, 16 Sep 2023 17:41:52 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qhd2y-0005AT-LA for qemu-devel@nongnu.org; Sat, 16 Sep 2023 17:41:48 -0400 Received: from mail-oa1-x32.google.com ([2001:4860:4864:20::32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1qhd2t-0000Sg-BS for qemu-devel@nongnu.org; Sat, 16 Sep 2023 17:41:48 -0400 Received: by mail-oa1-x32.google.com with SMTP id 586e51a60fabf-1c504386370so2017818fac.1 for ; Sat, 16 Sep 2023 14:41:43 -0700 (PDT) Received: from stoup.. ([71.212.131.115]) by smtp.gmail.com with ESMTPSA id n21-20020aa79055000000b0068fde95aa93sm4871708pfo.135.2023.09.16.14.41.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 16 Sep 2023 14:41:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1694900502; x=1695505302; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Eqdx5KYfe7+Tgpre/r7adTy9jlck+D44aSymYCDoQAc=; b=RfcDki/Y13BUGq6pMNr8KJ5uN6DBfKutzo5IG7fE9s28WsBje0azxhZr35bwBoTGZv 8jnaBQ8ErzDJZcRpC5YcADxpjtNUeG3Havvvne4o5kbJ7jWhj+Nb9FEGpqtIJ8yt0sWy NC16YrE1hepNpyJFHveeOBeP2ydcayhZVapoI3nlLIZQb1rmAy7muZPi7b2GhOlfBknx YWpmuf5bmLZX2MdxvyS3XF/1dYHvHjr3mgddDB4hUBvTrteMnw3ugqG41hfs8IGtyBm8 b4BTcRV3ToVo4ykSSfPv8XJ4HDABOSnNbuPctdTQjfQQeZjwBwu9Y9JQZbut0KhLHHq7 qEBw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694900502; x=1695505302; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Eqdx5KYfe7+Tgpre/r7adTy9jlck+D44aSymYCDoQAc=; b=A6FsvKynMFpRuA0anofIMlG4Qngga/y66B2m625/Qvkp3DLtafFeZJu4pB+mzhbg8R 8vgVxx3vAPFlIEcsn0Xr+TSNngb5GC0hBhEt+5Vq+IPVUn+djQY5xM3wbJrbWmzFuxsQ pFkXRqGG42oNaCzjAYqokVXejzDhkUMlfDL8thONiC3Z6U2EIVCvdwk0rPomsi3fwrIJ J2pk9SmaHM8PHFMFw7fspX8Z1iIwAiTkUCZYkESpPic+CrGulwP51eUJFsO6SKfo2ldb HZ9WILWnBwUvUEJ2gIYDGbJnuk6WXRZm6jCc2VcIkm3/rSr5Dl4jmbnE3yD2eVkVqQeP FIRw== X-Gm-Message-State: AOJu0YygYLaSyrVND7qiWo2d9rC0le9kInYKyVG3fVMxTZBuzUPpdq1T pa/VIC6ynqh/gtRKX0J0V1TpFrWT6YWfRPe6BiU= X-Google-Smtp-Source: AGHT+IGxVeDOACbxEqDjClh/WuibLYGpmClwa/oHQjfpST76CBS/GSc0kn4Wp2mdPFGn0sW6yF3z/g== X-Received: by 2002:a05:6871:687:b0:1ba:caf2:acc3 with SMTP id l7-20020a056871068700b001bacaf2acc3mr6488145oao.5.1694900501758; Sat, 16 Sep 2023 14:41:41 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Cc: philmd@linaro.org, anjo@rev.ng Subject: [PATCH v3 16/39] accel/tcg: Modify tlb_*() to use CPUState Date: Sat, 16 Sep 2023 14:41:00 -0700 Message-Id: <20230916214123.525796-17-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230916214123.525796-1-richard.henderson@linaro.org> References: <20230916214123.525796-1-richard.henderson@linaro.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2001:4860:4864:20::32; envelope-from=richard.henderson@linaro.org; helo=mail-oa1-x32.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @linaro.org) X-ZM-MESSAGEID: 1694900612769100001 Content-Type: text/plain; charset="utf-8" From: Anton Johansson Changes tlb_*() functions to take CPUState instead of CPUArchState, as they don't require the full CPUArchState. This makes it easier to decouple target-(in)dependent code. Signed-off-by: Anton Johansson Message-Id: <20230912153428.17816-4-anjo@rev.ng> Reviewed-by: Richard Henderson [rth: Use cpu->neg.tlb instead of cpu_tlb()] Signed-off-by: Richard Henderson Reviewed-by: Philippe Mathieu-Daud=C3=A9 --- include/exec/cpu_ldst.h | 8 +- accel/tcg/cputlb.c | 220 +++++++++++++++++++--------------------- 2 files changed, 108 insertions(+), 120 deletions(-) diff --git a/include/exec/cpu_ldst.h b/include/exec/cpu_ldst.h index da10ba1433..6061e33ac9 100644 --- a/include/exec/cpu_ldst.h +++ b/include/exec/cpu_ldst.h @@ -361,19 +361,19 @@ static inline uint64_t tlb_addr_write(const CPUTLBEnt= ry *entry) } =20 /* Find the TLB index corresponding to the mmu_idx + address pair. */ -static inline uintptr_t tlb_index(CPUArchState *env, uintptr_t mmu_idx, +static inline uintptr_t tlb_index(CPUState *cpu, uintptr_t mmu_idx, vaddr addr) { - uintptr_t size_mask =3D env_tlb(env)->f[mmu_idx].mask >> CPU_TLB_ENTRY= _BITS; + uintptr_t size_mask =3D cpu->neg.tlb.f[mmu_idx].mask >> CPU_TLB_ENTRY_= BITS; =20 return (addr >> TARGET_PAGE_BITS) & size_mask; } =20 /* Find the TLB entry corresponding to the mmu_idx + address pair. */ -static inline CPUTLBEntry *tlb_entry(CPUArchState *env, uintptr_t mmu_idx, +static inline CPUTLBEntry *tlb_entry(CPUState *cpu, uintptr_t mmu_idx, vaddr addr) { - return &env_tlb(env)->f[mmu_idx].table[tlb_index(env, mmu_idx, addr)]; + return &cpu->neg.tlb.f[mmu_idx].table[tlb_index(cpu, mmu_idx, addr)]; } =20 #endif /* defined(CONFIG_USER_ONLY) */ diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c index f790be5b6e..f88c394594 100644 --- a/accel/tcg/cputlb.c +++ b/accel/tcg/cputlb.c @@ -240,11 +240,11 @@ static void tlb_mmu_flush_locked(CPUTLBDesc *desc, CP= UTLBDescFast *fast) memset(desc->vtable, -1, sizeof(desc->vtable)); } =20 -static void tlb_flush_one_mmuidx_locked(CPUArchState *env, int mmu_idx, +static void tlb_flush_one_mmuidx_locked(CPUState *cpu, int mmu_idx, int64_t now) { - CPUTLBDesc *desc =3D &env_tlb(env)->d[mmu_idx]; - CPUTLBDescFast *fast =3D &env_tlb(env)->f[mmu_idx]; + CPUTLBDesc *desc =3D &cpu->neg.tlb.d[mmu_idx]; + CPUTLBDescFast *fast =3D &cpu->neg.tlb.f[mmu_idx]; =20 tlb_mmu_resize_locked(desc, fast, now); tlb_mmu_flush_locked(desc, fast); @@ -262,41 +262,39 @@ static void tlb_mmu_init(CPUTLBDesc *desc, CPUTLBDesc= Fast *fast, int64_t now) tlb_mmu_flush_locked(desc, fast); } =20 -static inline void tlb_n_used_entries_inc(CPUArchState *env, uintptr_t mmu= _idx) +static inline void tlb_n_used_entries_inc(CPUState *cpu, uintptr_t mmu_idx) { - env_tlb(env)->d[mmu_idx].n_used_entries++; + cpu->neg.tlb.d[mmu_idx].n_used_entries++; } =20 -static inline void tlb_n_used_entries_dec(CPUArchState *env, uintptr_t mmu= _idx) +static inline void tlb_n_used_entries_dec(CPUState *cpu, uintptr_t mmu_idx) { - env_tlb(env)->d[mmu_idx].n_used_entries--; + cpu->neg.tlb.d[mmu_idx].n_used_entries--; } =20 void tlb_init(CPUState *cpu) { - CPUArchState *env =3D cpu_env(cpu); int64_t now =3D get_clock_realtime(); int i; =20 - qemu_spin_init(&env_tlb(env)->c.lock); + qemu_spin_init(&cpu->neg.tlb.c.lock); =20 /* All tlbs are initialized flushed. */ - env_tlb(env)->c.dirty =3D 0; + cpu->neg.tlb.c.dirty =3D 0; =20 for (i =3D 0; i < NB_MMU_MODES; i++) { - tlb_mmu_init(&env_tlb(env)->d[i], &env_tlb(env)->f[i], now); + tlb_mmu_init(&cpu->neg.tlb.d[i], &cpu->neg.tlb.f[i], now); } } =20 void tlb_destroy(CPUState *cpu) { - CPUArchState *env =3D cpu_env(cpu); int i; =20 - qemu_spin_destroy(&env_tlb(env)->c.lock); + qemu_spin_destroy(&cpu->neg.tlb.c.lock); for (i =3D 0; i < NB_MMU_MODES; i++) { - CPUTLBDesc *desc =3D &env_tlb(env)->d[i]; - CPUTLBDescFast *fast =3D &env_tlb(env)->f[i]; + CPUTLBDesc *desc =3D &cpu->neg.tlb.d[i]; + CPUTLBDescFast *fast =3D &cpu->neg.tlb.f[i]; =20 g_free(fast->table); g_free(desc->fulltlb); @@ -328,11 +326,9 @@ void tlb_flush_counts(size_t *pfull, size_t *ppart, si= ze_t *pelide) size_t full =3D 0, part =3D 0, elide =3D 0; =20 CPU_FOREACH(cpu) { - CPUArchState *env =3D cpu_env(cpu); - - full +=3D qatomic_read(&env_tlb(env)->c.full_flush_count); - part +=3D qatomic_read(&env_tlb(env)->c.part_flush_count); - elide +=3D qatomic_read(&env_tlb(env)->c.elide_flush_count); + full +=3D qatomic_read(&cpu->neg.tlb.c.full_flush_count); + part +=3D qatomic_read(&cpu->neg.tlb.c.part_flush_count); + elide +=3D qatomic_read(&cpu->neg.tlb.c.elide_flush_count); } *pfull =3D full; *ppart =3D part; @@ -341,7 +337,6 @@ void tlb_flush_counts(size_t *pfull, size_t *ppart, siz= e_t *pelide) =20 static void tlb_flush_by_mmuidx_async_work(CPUState *cpu, run_on_cpu_data = data) { - CPUArchState *env =3D cpu_env(cpu); uint16_t asked =3D data.host_int; uint16_t all_dirty, work, to_clean; int64_t now =3D get_clock_realtime(); @@ -350,32 +345,32 @@ static void tlb_flush_by_mmuidx_async_work(CPUState *= cpu, run_on_cpu_data data) =20 tlb_debug("mmu_idx:0x%04" PRIx16 "\n", asked); =20 - qemu_spin_lock(&env_tlb(env)->c.lock); + qemu_spin_lock(&cpu->neg.tlb.c.lock); =20 - all_dirty =3D env_tlb(env)->c.dirty; + all_dirty =3D cpu->neg.tlb.c.dirty; to_clean =3D asked & all_dirty; all_dirty &=3D ~to_clean; - env_tlb(env)->c.dirty =3D all_dirty; + cpu->neg.tlb.c.dirty =3D all_dirty; =20 for (work =3D to_clean; work !=3D 0; work &=3D work - 1) { int mmu_idx =3D ctz32(work); - tlb_flush_one_mmuidx_locked(env, mmu_idx, now); + tlb_flush_one_mmuidx_locked(cpu, mmu_idx, now); } =20 - qemu_spin_unlock(&env_tlb(env)->c.lock); + qemu_spin_unlock(&cpu->neg.tlb.c.lock); =20 tcg_flush_jmp_cache(cpu); =20 if (to_clean =3D=3D ALL_MMUIDX_BITS) { - qatomic_set(&env_tlb(env)->c.full_flush_count, - env_tlb(env)->c.full_flush_count + 1); + qatomic_set(&cpu->neg.tlb.c.full_flush_count, + cpu->neg.tlb.c.full_flush_count + 1); } else { - qatomic_set(&env_tlb(env)->c.part_flush_count, - env_tlb(env)->c.part_flush_count + ctpop16(to_clean)); + qatomic_set(&cpu->neg.tlb.c.part_flush_count, + cpu->neg.tlb.c.part_flush_count + ctpop16(to_clean)); if (to_clean !=3D asked) { - qatomic_set(&env_tlb(env)->c.elide_flush_count, - env_tlb(env)->c.elide_flush_count + - ctpop16(asked & ~to_clean)); + qatomic_set(&cpu->neg.tlb.c.elide_flush_count, + cpu->neg.tlb.c.elide_flush_count + + ctpop16(asked & ~to_clean)); } } } @@ -470,43 +465,43 @@ static inline bool tlb_flush_entry_locked(CPUTLBEntry= *tlb_entry, vaddr page) } =20 /* Called with tlb_c.lock held */ -static void tlb_flush_vtlb_page_mask_locked(CPUArchState *env, int mmu_idx, +static void tlb_flush_vtlb_page_mask_locked(CPUState *cpu, int mmu_idx, vaddr page, vaddr mask) { - CPUTLBDesc *d =3D &env_tlb(env)->d[mmu_idx]; + CPUTLBDesc *d =3D &cpu->neg.tlb.d[mmu_idx]; int k; =20 - assert_cpu_is_self(env_cpu(env)); + assert_cpu_is_self(cpu); for (k =3D 0; k < CPU_VTLB_SIZE; k++) { if (tlb_flush_entry_mask_locked(&d->vtable[k], page, mask)) { - tlb_n_used_entries_dec(env, mmu_idx); + tlb_n_used_entries_dec(cpu, mmu_idx); } } } =20 -static inline void tlb_flush_vtlb_page_locked(CPUArchState *env, int mmu_i= dx, +static inline void tlb_flush_vtlb_page_locked(CPUState *cpu, int mmu_idx, vaddr page) { - tlb_flush_vtlb_page_mask_locked(env, mmu_idx, page, -1); + tlb_flush_vtlb_page_mask_locked(cpu, mmu_idx, page, -1); } =20 -static void tlb_flush_page_locked(CPUArchState *env, int midx, vaddr page) +static void tlb_flush_page_locked(CPUState *cpu, int midx, vaddr page) { - vaddr lp_addr =3D env_tlb(env)->d[midx].large_page_addr; - vaddr lp_mask =3D env_tlb(env)->d[midx].large_page_mask; + vaddr lp_addr =3D cpu->neg.tlb.d[midx].large_page_addr; + vaddr lp_mask =3D cpu->neg.tlb.d[midx].large_page_mask; =20 /* Check if we need to flush due to large pages. */ if ((page & lp_mask) =3D=3D lp_addr) { tlb_debug("forcing full flush midx %d (%016" VADDR_PRIx "/%016" VADDR_PRIx ")\n", midx, lp_addr, lp_mask); - tlb_flush_one_mmuidx_locked(env, midx, get_clock_realtime()); + tlb_flush_one_mmuidx_locked(cpu, midx, get_clock_realtime()); } else { - if (tlb_flush_entry_locked(tlb_entry(env, midx, page), page)) { - tlb_n_used_entries_dec(env, midx); + if (tlb_flush_entry_locked(tlb_entry(cpu, midx, page), page)) { + tlb_n_used_entries_dec(cpu, midx); } - tlb_flush_vtlb_page_locked(env, midx, page); + tlb_flush_vtlb_page_locked(cpu, midx, page); } } =20 @@ -523,20 +518,19 @@ static void tlb_flush_page_by_mmuidx_async_0(CPUState= *cpu, vaddr addr, uint16_t idxmap) { - CPUArchState *env =3D cpu_env(cpu); int mmu_idx; =20 assert_cpu_is_self(cpu); =20 tlb_debug("page addr: %016" VADDR_PRIx " mmu_map:0x%x\n", addr, idxmap= ); =20 - qemu_spin_lock(&env_tlb(env)->c.lock); + qemu_spin_lock(&cpu->neg.tlb.c.lock); for (mmu_idx =3D 0; mmu_idx < NB_MMU_MODES; mmu_idx++) { if ((idxmap >> mmu_idx) & 1) { - tlb_flush_page_locked(env, mmu_idx, addr); + tlb_flush_page_locked(cpu, mmu_idx, addr); } } - qemu_spin_unlock(&env_tlb(env)->c.lock); + qemu_spin_unlock(&cpu->neg.tlb.c.lock); =20 /* * Discard jump cache entries for any tb which might potentially @@ -709,12 +703,12 @@ void tlb_flush_page_all_cpus_synced(CPUState *src, va= ddr addr) tlb_flush_page_by_mmuidx_all_cpus_synced(src, addr, ALL_MMUIDX_BITS); } =20 -static void tlb_flush_range_locked(CPUArchState *env, int midx, +static void tlb_flush_range_locked(CPUState *cpu, int midx, vaddr addr, vaddr len, unsigned bits) { - CPUTLBDesc *d =3D &env_tlb(env)->d[midx]; - CPUTLBDescFast *f =3D &env_tlb(env)->f[midx]; + CPUTLBDesc *d =3D &cpu->neg.tlb.d[midx]; + CPUTLBDescFast *f =3D &cpu->neg.tlb.f[midx]; vaddr mask =3D MAKE_64BIT_MASK(0, bits); =20 /* @@ -731,7 +725,7 @@ static void tlb_flush_range_locked(CPUArchState *env, i= nt midx, tlb_debug("forcing full flush midx %d (" "%016" VADDR_PRIx "/%016" VADDR_PRIx "+%016" VADDR_PRIx = ")\n", midx, addr, mask, len); - tlb_flush_one_mmuidx_locked(env, midx, get_clock_realtime()); + tlb_flush_one_mmuidx_locked(cpu, midx, get_clock_realtime()); return; } =20 @@ -744,18 +738,18 @@ static void tlb_flush_range_locked(CPUArchState *env,= int midx, tlb_debug("forcing full flush midx %d (" "%016" VADDR_PRIx "/%016" VADDR_PRIx ")\n", midx, d->large_page_addr, d->large_page_mask); - tlb_flush_one_mmuidx_locked(env, midx, get_clock_realtime()); + tlb_flush_one_mmuidx_locked(cpu, midx, get_clock_realtime()); return; } =20 for (vaddr i =3D 0; i < len; i +=3D TARGET_PAGE_SIZE) { vaddr page =3D addr + i; - CPUTLBEntry *entry =3D tlb_entry(env, midx, page); + CPUTLBEntry *entry =3D tlb_entry(cpu, midx, page); =20 if (tlb_flush_entry_mask_locked(entry, page, mask)) { - tlb_n_used_entries_dec(env, midx); + tlb_n_used_entries_dec(cpu, midx); } - tlb_flush_vtlb_page_mask_locked(env, midx, page, mask); + tlb_flush_vtlb_page_mask_locked(cpu, midx, page, mask); } } =20 @@ -769,7 +763,6 @@ typedef struct { static void tlb_flush_range_by_mmuidx_async_0(CPUState *cpu, TLBFlushRangeData d) { - CPUArchState *env =3D cpu_env(cpu); int mmu_idx; =20 assert_cpu_is_self(cpu); @@ -777,13 +770,13 @@ static void tlb_flush_range_by_mmuidx_async_0(CPUStat= e *cpu, tlb_debug("range: %016" VADDR_PRIx "/%u+%016" VADDR_PRIx " mmu_map:0x%= x\n", d.addr, d.bits, d.len, d.idxmap); =20 - qemu_spin_lock(&env_tlb(env)->c.lock); + qemu_spin_lock(&cpu->neg.tlb.c.lock); for (mmu_idx =3D 0; mmu_idx < NB_MMU_MODES; mmu_idx++) { if ((d.idxmap >> mmu_idx) & 1) { - tlb_flush_range_locked(env, mmu_idx, d.addr, d.len, d.bits); + tlb_flush_range_locked(cpu, mmu_idx, d.addr, d.len, d.bits); } } - qemu_spin_unlock(&env_tlb(env)->c.lock); + qemu_spin_unlock(&cpu->neg.tlb.c.lock); =20 /* * If the length is larger than the jump cache size, then it will take @@ -1028,27 +1021,24 @@ static inline void copy_tlb_helper_locked(CPUTLBEnt= ry *d, const CPUTLBEntry *s) */ void tlb_reset_dirty(CPUState *cpu, ram_addr_t start1, ram_addr_t length) { - CPUArchState *env; - int mmu_idx; =20 - env =3D cpu_env(cpu); - qemu_spin_lock(&env_tlb(env)->c.lock); + qemu_spin_lock(&cpu->neg.tlb.c.lock); for (mmu_idx =3D 0; mmu_idx < NB_MMU_MODES; mmu_idx++) { unsigned int i; - unsigned int n =3D tlb_n_entries(&env_tlb(env)->f[mmu_idx]); + unsigned int n =3D tlb_n_entries(&cpu->neg.tlb.f[mmu_idx]); =20 for (i =3D 0; i < n; i++) { - tlb_reset_dirty_range_locked(&env_tlb(env)->f[mmu_idx].table[i= ], + tlb_reset_dirty_range_locked(&cpu->neg.tlb.f[mmu_idx].table[i], start1, length); } =20 for (i =3D 0; i < CPU_VTLB_SIZE; i++) { - tlb_reset_dirty_range_locked(&env_tlb(env)->d[mmu_idx].vtable[= i], + tlb_reset_dirty_range_locked(&cpu->neg.tlb.d[mmu_idx].vtable[i= ], start1, length); } } - qemu_spin_unlock(&env_tlb(env)->c.lock); + qemu_spin_unlock(&cpu->neg.tlb.c.lock); } =20 /* Called with tlb_c.lock held */ @@ -1064,32 +1054,31 @@ static inline void tlb_set_dirty1_locked(CPUTLBEntr= y *tlb_entry, so that it is no longer dirty */ void tlb_set_dirty(CPUState *cpu, vaddr addr) { - CPUArchState *env =3D cpu_env(cpu); int mmu_idx; =20 assert_cpu_is_self(cpu); =20 addr &=3D TARGET_PAGE_MASK; - qemu_spin_lock(&env_tlb(env)->c.lock); + qemu_spin_lock(&cpu->neg.tlb.c.lock); for (mmu_idx =3D 0; mmu_idx < NB_MMU_MODES; mmu_idx++) { - tlb_set_dirty1_locked(tlb_entry(env, mmu_idx, addr), addr); + tlb_set_dirty1_locked(tlb_entry(cpu, mmu_idx, addr), addr); } =20 for (mmu_idx =3D 0; mmu_idx < NB_MMU_MODES; mmu_idx++) { int k; for (k =3D 0; k < CPU_VTLB_SIZE; k++) { - tlb_set_dirty1_locked(&env_tlb(env)->d[mmu_idx].vtable[k], add= r); + tlb_set_dirty1_locked(&cpu->neg.tlb.d[mmu_idx].vtable[k], addr= ); } } - qemu_spin_unlock(&env_tlb(env)->c.lock); + qemu_spin_unlock(&cpu->neg.tlb.c.lock); } =20 /* Our TLB does not support large pages, so remember the area covered by large pages and trigger a full TLB flush if these are invalidated. */ -static void tlb_add_large_page(CPUArchState *env, int mmu_idx, +static void tlb_add_large_page(CPUState *cpu, int mmu_idx, vaddr addr, uint64_t size) { - vaddr lp_addr =3D env_tlb(env)->d[mmu_idx].large_page_addr; + vaddr lp_addr =3D cpu->neg.tlb.d[mmu_idx].large_page_addr; vaddr lp_mask =3D ~(size - 1); =20 if (lp_addr =3D=3D (vaddr)-1) { @@ -1099,13 +1088,13 @@ static void tlb_add_large_page(CPUArchState *env, i= nt mmu_idx, /* Extend the existing region to include the new page. This is a compromise between unnecessary flushes and the cost of maintaining a full variable size TLB. */ - lp_mask &=3D env_tlb(env)->d[mmu_idx].large_page_mask; + lp_mask &=3D cpu->neg.tlb.d[mmu_idx].large_page_mask; while (((lp_addr ^ addr) & lp_mask) !=3D 0) { lp_mask <<=3D 1; } } - env_tlb(env)->d[mmu_idx].large_page_addr =3D lp_addr & lp_mask; - env_tlb(env)->d[mmu_idx].large_page_mask =3D lp_mask; + cpu->neg.tlb.d[mmu_idx].large_page_addr =3D lp_addr & lp_mask; + cpu->neg.tlb.d[mmu_idx].large_page_mask =3D lp_mask; } =20 static inline void tlb_set_compare(CPUTLBEntryFull *full, CPUTLBEntry *ent, @@ -1137,8 +1126,7 @@ static inline void tlb_set_compare(CPUTLBEntryFull *f= ull, CPUTLBEntry *ent, void tlb_set_page_full(CPUState *cpu, int mmu_idx, vaddr addr, CPUTLBEntryFull *full) { - CPUArchState *env =3D cpu_env(cpu); - CPUTLB *tlb =3D env_tlb(env); + CPUTLB *tlb =3D &cpu->neg.tlb; CPUTLBDesc *desc =3D &tlb->d[mmu_idx]; MemoryRegionSection *section; unsigned int index, read_flags, write_flags; @@ -1155,7 +1143,7 @@ void tlb_set_page_full(CPUState *cpu, int mmu_idx, sz =3D TARGET_PAGE_SIZE; } else { sz =3D (hwaddr)1 << full->lg_page_size; - tlb_add_large_page(env, mmu_idx, addr, sz); + tlb_add_large_page(cpu, mmu_idx, addr, sz); } addr_page =3D addr & TARGET_PAGE_MASK; paddr_page =3D full->phys_addr & TARGET_PAGE_MASK; @@ -1222,8 +1210,8 @@ void tlb_set_page_full(CPUState *cpu, int mmu_idx, wp_flags =3D cpu_watchpoint_address_matches(cpu, addr_page, TARGET_PAGE_SIZE); =20 - index =3D tlb_index(env, mmu_idx, addr_page); - te =3D tlb_entry(env, mmu_idx, addr_page); + index =3D tlb_index(cpu, mmu_idx, addr_page); + te =3D tlb_entry(cpu, mmu_idx, addr_page); =20 /* * Hold the TLB lock for the rest of the function. We could acquire/re= lease @@ -1238,7 +1226,7 @@ void tlb_set_page_full(CPUState *cpu, int mmu_idx, tlb->c.dirty |=3D 1 << mmu_idx; =20 /* Make sure there's no cached translation for the new page. */ - tlb_flush_vtlb_page_locked(env, mmu_idx, addr_page); + tlb_flush_vtlb_page_locked(cpu, mmu_idx, addr_page); =20 /* * Only evict the old entry to the victim tlb if it's for a @@ -1251,7 +1239,7 @@ void tlb_set_page_full(CPUState *cpu, int mmu_idx, /* Evict the old entry into the victim tlb. */ copy_tlb_helper_locked(tv, te); desc->vfulltlb[vidx] =3D desc->fulltlb[index]; - tlb_n_used_entries_dec(env, mmu_idx); + tlb_n_used_entries_dec(cpu, mmu_idx); } =20 /* refill the tlb */ @@ -1296,7 +1284,7 @@ void tlb_set_page_full(CPUState *cpu, int mmu_idx, MMU_DATA_STORE, prot & PAGE_WRITE); =20 copy_tlb_helper_locked(te, &tn); - tlb_n_used_entries_inc(env, mmu_idx); + tlb_n_used_entries_inc(cpu, mmu_idx); qemu_spin_unlock(&tlb->c.lock); } =20 @@ -1390,28 +1378,28 @@ static void io_failed(CPUArchState *env, CPUTLBEntr= yFull *full, vaddr addr, =20 /* Return true if ADDR is present in the victim tlb, and has been copied back to the main tlb. */ -static bool victim_tlb_hit(CPUArchState *env, size_t mmu_idx, size_t index, +static bool victim_tlb_hit(CPUState *cpu, size_t mmu_idx, size_t index, MMUAccessType access_type, vaddr page) { size_t vidx; =20 - assert_cpu_is_self(env_cpu(env)); + assert_cpu_is_self(cpu); for (vidx =3D 0; vidx < CPU_VTLB_SIZE; ++vidx) { - CPUTLBEntry *vtlb =3D &env_tlb(env)->d[mmu_idx].vtable[vidx]; + CPUTLBEntry *vtlb =3D &cpu->neg.tlb.d[mmu_idx].vtable[vidx]; uint64_t cmp =3D tlb_read_idx(vtlb, access_type); =20 if (cmp =3D=3D page) { /* Found entry in victim tlb, swap tlb and iotlb. */ - CPUTLBEntry tmptlb, *tlb =3D &env_tlb(env)->f[mmu_idx].table[i= ndex]; + CPUTLBEntry tmptlb, *tlb =3D &cpu->neg.tlb.f[mmu_idx].table[in= dex]; =20 - qemu_spin_lock(&env_tlb(env)->c.lock); + qemu_spin_lock(&cpu->neg.tlb.c.lock); copy_tlb_helper_locked(&tmptlb, tlb); copy_tlb_helper_locked(tlb, vtlb); copy_tlb_helper_locked(vtlb, &tmptlb); - qemu_spin_unlock(&env_tlb(env)->c.lock); + qemu_spin_unlock(&cpu->neg.tlb.c.lock); =20 - CPUTLBEntryFull *f1 =3D &env_tlb(env)->d[mmu_idx].fulltlb[inde= x]; - CPUTLBEntryFull *f2 =3D &env_tlb(env)->d[mmu_idx].vfulltlb[vid= x]; + CPUTLBEntryFull *f1 =3D &cpu->neg.tlb.d[mmu_idx].fulltlb[index= ]; + CPUTLBEntryFull *f2 =3D &cpu->neg.tlb.d[mmu_idx].vfulltlb[vidx= ]; CPUTLBEntryFull tmpf; tmpf =3D *f1; *f1 =3D *f2; *f2 =3D tmpf; return true; @@ -1450,8 +1438,8 @@ static int probe_access_internal(CPUArchState *env, v= addr addr, void **phost, CPUTLBEntryFull **pfull, uintptr_t retaddr, bool check_mem_cbs) { - uintptr_t index =3D tlb_index(env, mmu_idx, addr); - CPUTLBEntry *entry =3D tlb_entry(env, mmu_idx, addr); + uintptr_t index =3D tlb_index(env_cpu(env), mmu_idx, addr); + CPUTLBEntry *entry =3D tlb_entry(env_cpu(env), mmu_idx, addr); uint64_t tlb_addr =3D tlb_read_idx(entry, access_type); vaddr page_addr =3D addr & TARGET_PAGE_MASK; int flags =3D TLB_FLAGS_MASK & ~TLB_FORCE_SLOW; @@ -1459,7 +1447,8 @@ static int probe_access_internal(CPUArchState *env, v= addr addr, CPUTLBEntryFull *full; =20 if (!tlb_hit_page(tlb_addr, page_addr)) { - if (!victim_tlb_hit(env, mmu_idx, index, access_type, page_addr)) { + if (!victim_tlb_hit(env_cpu(env), mmu_idx, index, + access_type, page_addr)) { CPUState *cs =3D env_cpu(env); =20 if (!cs->cc->tcg_ops->tlb_fill(cs, addr, fault_size, access_ty= pe, @@ -1471,8 +1460,8 @@ static int probe_access_internal(CPUArchState *env, v= addr addr, } =20 /* TLB resize via tlb_fill may have moved the entry. */ - index =3D tlb_index(env, mmu_idx, addr); - entry =3D tlb_entry(env, mmu_idx, addr); + index =3D tlb_index(env_cpu(env), mmu_idx, addr); + entry =3D tlb_entry(env_cpu(env), mmu_idx, addr); =20 /* * With PAGE_WRITE_INV, we set TLB_INVALID_MASK immediately, @@ -1662,9 +1651,8 @@ tb_page_addr_t get_page_addr_code_hostp(CPUArchState = *env, vaddr addr, bool tlb_plugin_lookup(CPUState *cpu, vaddr addr, int mmu_idx, bool is_store, struct qemu_plugin_hwaddr *data) { - CPUArchState *env =3D cpu_env(cpu); - CPUTLBEntry *tlbe =3D tlb_entry(env, mmu_idx, addr); - uintptr_t index =3D tlb_index(env, mmu_idx, addr); + CPUTLBEntry *tlbe =3D tlb_entry(cpu, mmu_idx, addr); + uintptr_t index =3D tlb_index(cpu, mmu_idx, addr); MMUAccessType access_type =3D is_store ? MMU_DATA_STORE : MMU_DATA_LOA= D; uint64_t tlb_addr =3D tlb_read_idx(tlbe, access_type); CPUTLBEntryFull *full; @@ -1673,7 +1661,7 @@ bool tlb_plugin_lookup(CPUState *cpu, vaddr addr, int= mmu_idx, return false; } =20 - full =3D &env_tlb(env)->d[mmu_idx].fulltlb[index]; + full =3D &cpu->neg.tlb.d[mmu_idx].fulltlb[index]; data->phys_addr =3D full->phys_addr | (addr & ~TARGET_PAGE_MASK); =20 /* We must have an iotlb entry for MMIO */ @@ -1727,8 +1715,8 @@ static bool mmu_lookup1(CPUArchState *env, MMULookupP= ageData *data, int mmu_idx, MMUAccessType access_type, uintptr_t = ra) { vaddr addr =3D data->addr; - uintptr_t index =3D tlb_index(env, mmu_idx, addr); - CPUTLBEntry *entry =3D tlb_entry(env, mmu_idx, addr); + uintptr_t index =3D tlb_index(env_cpu(env), mmu_idx, addr); + CPUTLBEntry *entry =3D tlb_entry(env_cpu(env), mmu_idx, addr); uint64_t tlb_addr =3D tlb_read_idx(entry, access_type); bool maybe_resized =3D false; CPUTLBEntryFull *full; @@ -1736,12 +1724,12 @@ static bool mmu_lookup1(CPUArchState *env, MMULooku= pPageData *data, =20 /* If the TLB entry is for a different page, reload and try again. */ if (!tlb_hit(tlb_addr, addr)) { - if (!victim_tlb_hit(env, mmu_idx, index, access_type, + if (!victim_tlb_hit(env_cpu(env), mmu_idx, index, access_type, addr & TARGET_PAGE_MASK)) { tlb_fill(env_cpu(env), addr, data->size, access_type, mmu_idx,= ra); maybe_resized =3D true; - index =3D tlb_index(env, mmu_idx, addr); - entry =3D tlb_entry(env, mmu_idx, addr); + index =3D tlb_index(env_cpu(env), mmu_idx, addr); + entry =3D tlb_entry(env_cpu(env), mmu_idx, addr); } tlb_addr =3D tlb_read_idx(entry, access_type) & ~TLB_INVALID_MASK; } @@ -1849,7 +1837,7 @@ static bool mmu_lookup(CPUArchState *env, vaddr addr,= MemOpIdx oi, */ mmu_lookup1(env, &l->page[0], l->mmu_idx, type, ra); if (mmu_lookup1(env, &l->page[1], l->mmu_idx, type, ra)) { - uintptr_t index =3D tlb_index(env, l->mmu_idx, addr); + uintptr_t index =3D tlb_index(env_cpu(env), l->mmu_idx, addr); l->page[0].full =3D &env_tlb(env)->d[l->mmu_idx].fulltlb[index= ]; } =20 @@ -1907,18 +1895,18 @@ static void *atomic_mmu_lookup(CPUArchState *env, v= addr addr, MemOpIdx oi, goto stop_the_world; } =20 - index =3D tlb_index(env, mmu_idx, addr); - tlbe =3D tlb_entry(env, mmu_idx, addr); + index =3D tlb_index(env_cpu(env), mmu_idx, addr); + tlbe =3D tlb_entry(env_cpu(env), mmu_idx, addr); =20 /* Check TLB entry and enforce page permissions. */ tlb_addr =3D tlb_addr_write(tlbe); if (!tlb_hit(tlb_addr, addr)) { - if (!victim_tlb_hit(env, mmu_idx, index, MMU_DATA_STORE, + if (!victim_tlb_hit(env_cpu(env), mmu_idx, index, MMU_DATA_STORE, addr & TARGET_PAGE_MASK)) { tlb_fill(env_cpu(env), addr, size, MMU_DATA_STORE, mmu_idx, retaddr); - index =3D tlb_index(env, mmu_idx, addr); - tlbe =3D tlb_entry(env, mmu_idx, addr); + index =3D tlb_index(env_cpu(env), mmu_idx, addr); + tlbe =3D tlb_entry(env_cpu(env), mmu_idx, addr); } tlb_addr =3D tlb_addr_write(tlbe) & ~TLB_INVALID_MASK; } --=20 2.34.1