From nobody Sun May 5 05:19:29 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1538516255399453.6296569401095; Tue, 2 Oct 2018 14:37:35 -0700 (PDT) Received: from localhost ([::1]:45789 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1g7SMI-0005OL-5i for importer@patchew.org; Tue, 02 Oct 2018 17:37:34 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:44070) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1g7SEo-0007fu-UI for qemu-devel@nongnu.org; Tue, 02 Oct 2018 17:29:51 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1g7SEj-0001Mn-Qr for qemu-devel@nongnu.org; Tue, 02 Oct 2018 17:29:50 -0400 Received: from out3-smtp.messagingengine.com ([66.111.4.27]:51401) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1g7SEj-0001LS-Ky for qemu-devel@nongnu.org; Tue, 02 Oct 2018 17:29:45 -0400 Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.nyi.internal (Postfix) with ESMTP id 66D4A21E1D; Tue, 2 Oct 2018 17:29:43 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute4.internal (MEProxy); Tue, 02 Oct 2018 17:29:43 -0400 Received: from localhost (flamenco.cs.columbia.edu [128.59.20.216]) by mail.messagingengine.com (Postfix) with ESMTPA id 78E4CE4118; Tue, 2 Oct 2018 17:29:42 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=braap.org; h=cc :date:from:in-reply-to:message-id:references:subject:to :x-me-sender:x-me-sender:x-sasl-enc; s=mesmtp; bh=IuTddT7FiFfxXv wQpcfQ3nJxG1K0NrdQP+f/2ExFaok=; b=ij4jY6WsvWIDK0v8CJ85i3FOUUCR54 ZEy5ejXzQ43JlBjAZMDVY2onKL0PDu1gcy7rtWuJg8KrIeFe07ldvkGS9Iqt/3D4 x+J8zAfYGZcQ3Wrj9IWSdFPAO5U2rhzhHlpPAfN2HSv0MssX+orBzghGg9LoMWBe VHuyR79YQ2rZU= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:date:from:in-reply-to:message-id :references:subject:to:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; bh=IuTddT7FiFfxXvwQpcfQ3nJxG1K0NrdQP+f/2ExFaok=; b=R2Og282D jNEy03e39+MzymB8KoFpRHUvdgCnI0+7DLOxmLAGe/eXjyy4roW/XM2bpaMKrPgt joI0sspPriS5+tIj+rdli/aVXDov9lWpfbIvqwPgFUNl+REXdybXMOBpv8rrgPWZ LiXynEFEkNcV2qkxGV/5/MBvb8d2ztPUNxA8xh3cvqL8PSWaVNswJypjRxxugNsm UfyOk4uV3NR1NuCk7casah3C4wsHvMybUF94apJ/eYnkw3E5xfUK+nJZojwEciXZ 2348QiQJBz9TJniifJEKyIKZm083nxKwJuzPeaEwjD0YATTXcT9msA/uYLDpUCNB wEoI5edella/UA== X-ME-Sender: X-ME-Proxy: From: "Emilio G. Cota" To: qemu-devel@nongnu.org Date: Tue, 2 Oct 2018 17:29:19 -0400 Message-Id: <20181002212921.30982-2-cota@braap.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181002212921.30982-1-cota@braap.org> References: <20181002212921.30982-1-cota@braap.org> X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 66.111.4.27 Subject: [Qemu-devel] [PATCH 1/3] exec: introduce tlb_init X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Paolo Bonzini , =?UTF-8?q?Alex=20Benn=C3=A9e?= , Richard Henderson Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Paves the way for the addition of a per-TLB lock. Signed-off-by: Emilio G. Cota --- include/exec/exec-all.h | 8 ++++++++ accel/tcg/cputlb.c | 4 ++++ exec.c | 1 + 3 files changed, 13 insertions(+) diff --git a/include/exec/exec-all.h b/include/exec/exec-all.h index 5f78125582..815e5b1e83 100644 --- a/include/exec/exec-all.h +++ b/include/exec/exec-all.h @@ -99,6 +99,11 @@ void cpu_address_space_init(CPUState *cpu, int asidx, =20 #if !defined(CONFIG_USER_ONLY) && defined(CONFIG_TCG) /* cputlb.c */ +/** + * tlb_init - initialize a CPU's TLB + * @cpu: CPU whose TLB should be initialized + */ +void tlb_init(CPUState *cpu); /** * tlb_flush_page: * @cpu: CPU whose TLB should be flushed @@ -258,6 +263,9 @@ void tlb_set_page(CPUState *cpu, target_ulong vaddr, void probe_write(CPUArchState *env, target_ulong addr, int size, int mmu_i= dx, uintptr_t retaddr); #else +static inline void tlb_init(CPUState *cpu) +{ +} static inline void tlb_flush_page(CPUState *cpu, target_ulong addr) { } diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c index f4702ce91f..502eea2850 100644 --- a/accel/tcg/cputlb.c +++ b/accel/tcg/cputlb.c @@ -73,6 +73,10 @@ QEMU_BUILD_BUG_ON(sizeof(target_ulong) > sizeof(run_on_c= pu_data)); QEMU_BUILD_BUG_ON(NB_MMU_MODES > 16); #define ALL_MMUIDX_BITS ((1 << NB_MMU_MODES) - 1) =20 +void tlb_init(CPUState *cpu) +{ +} + /* flush_all_helper: run fn across all cpus * * If the wait flag is set then the src cpu's helper will be queued as diff --git a/exec.c b/exec.c index 6826c8337d..595deae346 100644 --- a/exec.c +++ b/exec.c @@ -965,6 +965,7 @@ void cpu_exec_realizefn(CPUState *cpu, Error **errp) tcg_target_initialized =3D true; cc->tcg_initialize(); } + tlb_init(cpu); =20 #ifndef CONFIG_USER_ONLY if (qdev_get_vmsd(DEVICE(cpu)) =3D=3D NULL) { --=20 2.17.1 From nobody Sun May 5 05:19:29 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1538516601339467.50997982721844; Tue, 2 Oct 2018 14:43:21 -0700 (PDT) Received: from localhost ([::1]:45824 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1g7SRs-0002hO-6B for importer@patchew.org; Tue, 02 Oct 2018 17:43:20 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:44073) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1g7SEo-0007fx-Ux for qemu-devel@nongnu.org; Tue, 02 Oct 2018 17:29:52 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1g7SEj-0001Mx-RQ for qemu-devel@nongnu.org; Tue, 02 Oct 2018 17:29:50 -0400 Received: from out3-smtp.messagingengine.com ([66.111.4.27]:58639) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1g7SEj-0001LR-Kw for qemu-devel@nongnu.org; Tue, 02 Oct 2018 17:29:45 -0400 Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.nyi.internal (Postfix) with ESMTP id 6CC9321E33; Tue, 2 Oct 2018 17:29:43 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute4.internal (MEProxy); Tue, 02 Oct 2018 17:29:43 -0400 Received: from localhost (flamenco.cs.columbia.edu [128.59.20.216]) by mail.messagingengine.com (Postfix) with ESMTPA id B5F78E49CB; Tue, 2 Oct 2018 17:29:42 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=braap.org; h=cc :date:from:in-reply-to:message-id:references:subject:to :x-me-sender:x-me-sender:x-sasl-enc; s=mesmtp; bh=jWd/ueBukeYKf/ HYsd0n3gINJfKVyK+oskBNl369UoE=; b=x1d20ZPHLICXu7kZA6Wwz7LrM0aGVd OZKYk2beGNVJJLujDskMRxqX1knY2TjKrkZmMVjB1jdbGgDnBdOHKzRPrwekpaQo 0CVFEcPkG9E/NC+iJdGauE/T6fo16y0eGLrbADzMjNCPLqb9XYOTV+dODJEc/ixA rxbCm+sXRnAHk= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:date:from:in-reply-to:message-id :references:subject:to:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; bh=jWd/ueBukeYKf/HYsd0n3gINJfKVyK+oskBNl369UoE=; b=wjjkitKm eKmJ7ZpQvulX2k04i7b8ehCZlShkMc+uEobtDcWpWw2PzgT9rz0gFnuIo3xKmvP0 zxXahqVShd2DnS9YaIvi3/kOQifNfRywPHFNmHc+DTiQgc+Mo2NsEUW3anGoTLAy mUjP64aL2geiarnvuLxbMcNGQtwiUpmjbnqlfEqopgSCQ4C9ffyL7O0YOvz9CmQJ 4XZORI+CU5nzKsPrLcBucnnUioPHjqNauGqV3lycrk6pbaNoEC5uDMCeLSgvbl7n 1/qMobhVNNs0w6LSmeyYcOuHtjdW/iNJn4gtDUurLnYjXTS9+LQtcC66RzQvo5S7 26aH1ePOd9GOWQ== X-ME-Sender: X-ME-Proxy: From: "Emilio G. Cota" To: qemu-devel@nongnu.org Date: Tue, 2 Oct 2018 17:29:20 -0400 Message-Id: <20181002212921.30982-3-cota@braap.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181002212921.30982-1-cota@braap.org> References: <20181002212921.30982-1-cota@braap.org> X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 66.111.4.27 Subject: [Qemu-devel] [PATCH 2/3] cputlb: serialize tlb updates with env->tlb_lock X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Paolo Bonzini , =?UTF-8?q?Alex=20Benn=C3=A9e?= , Richard Henderson Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Currently we rely on atomic operations for cross-CPU invalidations. There are two cases that these atomics miss: cross-CPU invalidations can race with either (1) vCPU threads flushing their TLB, which happens via memset, or (2) vCPUs calling tlb_reset_dirty on their TLB, which updates .addr_write with a regular store. This results in undefined behaviour, since we're mixing regular and atomic ops on concurrent accesses. Fix it by using tlb_lock, a per-vCPU lock. All updaters of tlb_table and the corresponding victim cache now hold the lock. The readers that do not hold tlb_lock must use atomic reads when reading .addr_write, since this field can be updated by other threads; the conversion to atomic reads is done in the next patch. Note that an alternative fix would be to expand the use of atomic ops. However, in the case of TLB flushes this would have a huge performance impact, since (1) TLB flushes can happen very frequently and (2) we currently use a full memory barrier to flush each TLB entry, and a TLB has many entries. Instead, acquiring the lock is barely slower than a full memory barrier since it is uncontended, and with a single lock acquisition we can flush the entire TLB. Signed-off-by: Emilio G. Cota --- include/exec/cpu-defs.h | 2 + accel/tcg/cputlb.c | 108 ++++++++++++++++++---------------------- 2 files changed, 50 insertions(+), 60 deletions(-) diff --git a/include/exec/cpu-defs.h b/include/exec/cpu-defs.h index a171ffc1a4..bcc40c8ef5 100644 --- a/include/exec/cpu-defs.h +++ b/include/exec/cpu-defs.h @@ -142,6 +142,8 @@ typedef struct CPUIOTLBEntry { =20 #define CPU_COMMON_TLB \ /* The meaning of the MMU modes is defined in the target code. */ \ + /* tlb_lock serializes updates to tlb_table and tlb_v_table */ \ + QemuMutex tlb_lock; \ CPUTLBEntry tlb_table[NB_MMU_MODES][CPU_TLB_SIZE]; \ CPUTLBEntry tlb_v_table[NB_MMU_MODES][CPU_VTLB_SIZE]; \ CPUIOTLBEntry iotlb[NB_MMU_MODES][CPU_TLB_SIZE]; \ diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c index 502eea2850..01b33f9d28 100644 --- a/accel/tcg/cputlb.c +++ b/accel/tcg/cputlb.c @@ -75,6 +75,9 @@ QEMU_BUILD_BUG_ON(NB_MMU_MODES > 16); =20 void tlb_init(CPUState *cpu) { + CPUArchState *env =3D cpu->env_ptr; + + qemu_mutex_init(&env->tlb_lock); } =20 /* flush_all_helper: run fn across all cpus @@ -129,8 +132,11 @@ static void tlb_flush_nocheck(CPUState *cpu) atomic_set(&env->tlb_flush_count, env->tlb_flush_count + 1); tlb_debug("(count: %zu)\n", tlb_flush_count()); =20 + qemu_mutex_lock(&env->tlb_lock); memset(env->tlb_table, -1, sizeof(env->tlb_table)); memset(env->tlb_v_table, -1, sizeof(env->tlb_v_table)); + qemu_mutex_unlock(&env->tlb_lock); + cpu_tb_jmp_cache_clear(cpu); =20 env->vtlb_index =3D 0; @@ -454,72 +460,48 @@ void tlb_unprotect_code(ram_addr_t ram_addr) * most usual is detecting writes to code regions which may invalidate * generated code. * - * Because we want other vCPUs to respond to changes straight away we - * update the te->addr_write field atomically. If the TLB entry has - * been changed by the vCPU in the mean time we skip the update. + * Other vCPUs might be reading their TLBs during guest execution, so we u= pdate + * te->addr_write with atomic_set. We don't need to worry about this for + * oversized guests as MTTCG is disabled for them. * - * As this function uses atomic accesses we also need to ensure - * updates to tlb_entries follow the same access rules. We don't need - * to worry about this for oversized guests as MTTCG is disabled for - * them. + * Called with tlb_lock held. */ - -static void tlb_reset_dirty_range(CPUTLBEntry *tlb_entry, uintptr_t start, - uintptr_t length) +static void tlb_reset_dirty_range_locked(CPUTLBEntry *tlb_entry, + uintptr_t start, uintptr_t length) { -#if TCG_OVERSIZED_GUEST uintptr_t addr =3D tlb_entry->addr_write; =20 if ((addr & (TLB_INVALID_MASK | TLB_MMIO | TLB_NOTDIRTY)) =3D=3D 0) { addr &=3D TARGET_PAGE_MASK; addr +=3D tlb_entry->addend; if ((addr - start) < length) { +#if TCG_OVERSIZED_GUEST tlb_entry->addr_write |=3D TLB_NOTDIRTY; - } - } #else - /* paired with atomic_mb_set in tlb_set_page_with_attrs */ - uintptr_t orig_addr =3D atomic_mb_read(&tlb_entry->addr_write); - uintptr_t addr =3D orig_addr; - - if ((addr & (TLB_INVALID_MASK | TLB_MMIO | TLB_NOTDIRTY)) =3D=3D 0) { - addr &=3D TARGET_PAGE_MASK; - addr +=3D atomic_read(&tlb_entry->addend); - if ((addr - start) < length) { - uintptr_t notdirty_addr =3D orig_addr | TLB_NOTDIRTY; - atomic_cmpxchg(&tlb_entry->addr_write, orig_addr, notdirty_add= r); + atomic_set(&tlb_entry->addr_write, + tlb_entry->addr_write | TLB_NOTDIRTY); +#endif } } -#endif } =20 -/* For atomic correctness when running MTTCG we need to use the right - * primitives when copying entries */ -static inline void copy_tlb_helper(CPUTLBEntry *d, CPUTLBEntry *s, - bool atomic_set) +/* Called with tlb_lock held */ +static void copy_tlb_helper_locked(CPUTLBEntry *d, const CPUTLBEntry *s) { -#if TCG_OVERSIZED_GUEST *d =3D *s; -#else - if (atomic_set) { - d->addr_read =3D s->addr_read; - d->addr_code =3D s->addr_code; - atomic_set(&d->addend, atomic_read(&s->addend)); - /* Pairs with flag setting in tlb_reset_dirty_range */ - atomic_mb_set(&d->addr_write, atomic_read(&s->addr_write)); - } else { - d->addr_read =3D s->addr_read; - d->addr_write =3D atomic_read(&s->addr_write); - d->addr_code =3D s->addr_code; - d->addend =3D atomic_read(&s->addend); - } -#endif +} + +static void copy_tlb_helper(CPUArchState *env, CPUTLBEntry *d, CPUTLBEntry= *s) +{ + qemu_mutex_lock(&env->tlb_lock); + copy_tlb_helper_locked(d, s); + qemu_mutex_unlock(&env->tlb_lock); } =20 /* This is a cross vCPU call (i.e. another vCPU resetting the flags of - * the target vCPU). As such care needs to be taken that we don't - * dangerously race with another vCPU update. The only thing actually - * updated is the target TLB entry ->addr_write flags. + * the target vCPU). + * We must take tlb_lock to avoid racing with another vCPU update. The only + * thing actually updated is the target TLB entry ->addr_write flags. */ void tlb_reset_dirty(CPUState *cpu, ram_addr_t start1, ram_addr_t length) { @@ -528,22 +510,26 @@ void tlb_reset_dirty(CPUState *cpu, ram_addr_t start1= , ram_addr_t length) int mmu_idx; =20 env =3D cpu->env_ptr; + qemu_mutex_lock(&env->tlb_lock); for (mmu_idx =3D 0; mmu_idx < NB_MMU_MODES; mmu_idx++) { unsigned int i; =20 for (i =3D 0; i < CPU_TLB_SIZE; i++) { - tlb_reset_dirty_range(&env->tlb_table[mmu_idx][i], - start1, length); + tlb_reset_dirty_range_locked(&env->tlb_table[mmu_idx][i], star= t1, + length); } =20 for (i =3D 0; i < CPU_VTLB_SIZE; i++) { - tlb_reset_dirty_range(&env->tlb_v_table[mmu_idx][i], - start1, length); + tlb_reset_dirty_range_locked(&env->tlb_v_table[mmu_idx][i], st= art1, + length); } } + qemu_mutex_unlock(&env->tlb_lock); } =20 -static inline void tlb_set_dirty1(CPUTLBEntry *tlb_entry, target_ulong vad= dr) +/* Called with tlb_lock held */ +static inline void tlb_set_dirty1_locked(CPUTLBEntry *tlb_entry, + target_ulong vaddr) { if (tlb_entry->addr_write =3D=3D (vaddr | TLB_NOTDIRTY)) { tlb_entry->addr_write =3D vaddr; @@ -562,16 +548,18 @@ void tlb_set_dirty(CPUState *cpu, target_ulong vaddr) =20 vaddr &=3D TARGET_PAGE_MASK; i =3D (vaddr >> TARGET_PAGE_BITS) & (CPU_TLB_SIZE - 1); + qemu_mutex_lock(&env->tlb_lock); for (mmu_idx =3D 0; mmu_idx < NB_MMU_MODES; mmu_idx++) { - tlb_set_dirty1(&env->tlb_table[mmu_idx][i], vaddr); + tlb_set_dirty1_locked(&env->tlb_table[mmu_idx][i], vaddr); } =20 for (mmu_idx =3D 0; mmu_idx < NB_MMU_MODES; mmu_idx++) { int k; for (k =3D 0; k < CPU_VTLB_SIZE; k++) { - tlb_set_dirty1(&env->tlb_v_table[mmu_idx][k], vaddr); + tlb_set_dirty1_locked(&env->tlb_v_table[mmu_idx][k], vaddr); } } + qemu_mutex_unlock(&env->tlb_lock); } =20 /* Our TLB does not support large pages, so remember the area covered by @@ -677,7 +665,7 @@ void tlb_set_page_with_attrs(CPUState *cpu, target_ulon= g vaddr, CPUTLBEntry *tv =3D &env->tlb_v_table[mmu_idx][vidx]; =20 /* Evict the old entry into the victim tlb. */ - copy_tlb_helper(tv, te, true); + copy_tlb_helper(env, tv, te); env->iotlb_v[mmu_idx][vidx] =3D env->iotlb[mmu_idx][index]; } =20 @@ -729,9 +717,7 @@ void tlb_set_page_with_attrs(CPUState *cpu, target_ulon= g vaddr, } } =20 - /* Pairs with flag setting in tlb_reset_dirty_range */ - copy_tlb_helper(te, &tn, true); - /* atomic_mb_set(&te->addr_write, write_address); */ + copy_tlb_helper(env, te, &tn); } =20 /* Add a new TLB entry, but without specifying the memory @@ -903,9 +889,11 @@ static bool victim_tlb_hit(CPUArchState *env, size_t m= mu_idx, size_t index, /* Found entry in victim tlb, swap tlb and iotlb. */ CPUTLBEntry tmptlb, *tlb =3D &env->tlb_table[mmu_idx][index]; =20 - copy_tlb_helper(&tmptlb, tlb, false); - copy_tlb_helper(tlb, vtlb, true); - copy_tlb_helper(vtlb, &tmptlb, true); + qemu_mutex_lock(&env->tlb_lock); + copy_tlb_helper_locked(&tmptlb, tlb); + copy_tlb_helper_locked(tlb, vtlb); + copy_tlb_helper_locked(vtlb, &tmptlb); + qemu_mutex_unlock(&env->tlb_lock); =20 CPUIOTLBEntry tmpio, *io =3D &env->iotlb[mmu_idx][index]; CPUIOTLBEntry *vio =3D &env->iotlb_v[mmu_idx][vidx]; --=20 2.17.1 From nobody Sun May 5 05:19:29 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1538516470784654.3416126626068; Tue, 2 Oct 2018 14:41:10 -0700 (PDT) Received: from localhost ([::1]:45809 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1g7SPl-0000X2-FA for importer@patchew.org; Tue, 02 Oct 2018 17:41:09 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:44071) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1g7SEo-0007fw-Uv for qemu-devel@nongnu.org; Tue, 02 Oct 2018 17:29:52 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1g7SEk-0001Nb-RI for qemu-devel@nongnu.org; Tue, 02 Oct 2018 17:29:50 -0400 Received: from out3-smtp.messagingengine.com ([66.111.4.27]:51247) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1g7SEk-0001NG-Lw for qemu-devel@nongnu.org; Tue, 02 Oct 2018 17:29:46 -0400 Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.nyi.internal (Postfix) with ESMTP id 750A321E00; Tue, 2 Oct 2018 17:29:46 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute4.internal (MEProxy); Tue, 02 Oct 2018 17:29:46 -0400 Received: from localhost (flamenco.cs.columbia.edu [128.59.20.216]) by mail.messagingengine.com (Postfix) with ESMTPA id 21181E4118; Tue, 2 Oct 2018 17:29:46 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=braap.org; h=cc :date:from:in-reply-to:message-id:references:subject:to :x-me-sender:x-me-sender:x-sasl-enc; s=mesmtp; bh=yYm+mMPpOKobpj rcI0VnV8pP7RT0XUJXxLG3j3p9D+w=; b=h1MF2wZwtZiDap5OxtDwFafW7ND8WU ztYeTuBLa1PoLYzSIIapAnlPEUVVWidPqW6g/7bHD5uZkEh65qQLGgfgujoooGAm O+WN+wlivw4OwCVRnw7zbO9FHXBBHekW03h6nnppKoMrrAf3dqp9OG0C5T80db5K WvfRDVm2myj2k= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:date:from:in-reply-to:message-id :references:subject:to:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; bh=yYm+mMPpOKobpjrcI0VnV8pP7RT0XUJXxLG3j3p9D+w=; b=tZ6uecrQ bErwl3alWtXTsYXSu5/hmnDZeCkbbB7UWXsMQnA25Hum+B0zLBhERSvXxl8uXgvB W4hXcWKdlJJypBlzMtNIFGCsuFHGnMAzveaPzEiU5PbosneIflWRdHTZkou7Wb5N 8MaVyz5b8+fMF9iw7ahKUFVwhPNPta/FWFz8anL/GGIFqtsre48VHHK3rnnc0Mxb MHyAAIF0PWcZEjsPJrkJsOYfu3ACFun67pCA877ZPKNbg+rdjDkwrjEsnuiK77+u mBM8lcmoxhDLsOdirmFz6jLzphWcKO9tikIhp+fTSiH8G+XbTMoFm6796CrfDIpV 3sE23rixYz+nIw== X-ME-Sender: X-ME-Proxy: From: "Emilio G. Cota" To: qemu-devel@nongnu.org Date: Tue, 2 Oct 2018 17:29:21 -0400 Message-Id: <20181002212921.30982-4-cota@braap.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181002212921.30982-1-cota@braap.org> References: <20181002212921.30982-1-cota@braap.org> X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 66.111.4.27 Subject: [Qemu-devel] [PATCH 3/3] cputlb: read CPUTLBEntry.addr_write atomically X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Paolo Bonzini , =?UTF-8?q?Alex=20Benn=C3=A9e?= , Richard Henderson Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Updates can come from other threads, so readers that do not take tlb_lock must use atomic_read to avoid undefined behaviour. Signed-off-by: Emilio G. Cota --- accel/tcg/softmmu_template.h | 16 ++++++++++------ include/exec/cpu_ldst.h | 2 +- include/exec/cpu_ldst_template.h | 2 +- accel/tcg/cputlb.c | 15 +++++++++------ 4 files changed, 21 insertions(+), 14 deletions(-) diff --git a/accel/tcg/softmmu_template.h b/accel/tcg/softmmu_template.h index f060a693d4..1e50263871 100644 --- a/accel/tcg/softmmu_template.h +++ b/accel/tcg/softmmu_template.h @@ -277,7 +277,8 @@ void helper_le_st_name(CPUArchState *env, target_ulong = addr, DATA_TYPE val, { unsigned mmu_idx =3D get_mmuidx(oi); int index =3D (addr >> TARGET_PAGE_BITS) & (CPU_TLB_SIZE - 1); - target_ulong tlb_addr =3D env->tlb_table[mmu_idx][index].addr_write; + target_ulong tlb_addr =3D + atomic_read(&env->tlb_table[mmu_idx][index].addr_write); unsigned a_bits =3D get_alignment_bits(get_memop(oi)); uintptr_t haddr; =20 @@ -292,7 +293,8 @@ void helper_le_st_name(CPUArchState *env, target_ulong = addr, DATA_TYPE val, tlb_fill(ENV_GET_CPU(env), addr, DATA_SIZE, MMU_DATA_STORE, mmu_idx, retaddr); } - tlb_addr =3D env->tlb_table[mmu_idx][index].addr_write & ~TLB_INVA= LID_MASK; + tlb_addr =3D atomic_read(&env->tlb_table[mmu_idx][index].addr_writ= e) & + ~TLB_INVALID_MASK; } =20 /* Handle an IO access. */ @@ -321,7 +323,7 @@ void helper_le_st_name(CPUArchState *env, target_ulong = addr, DATA_TYPE val, cannot evict the first. */ page2 =3D (addr + DATA_SIZE) & TARGET_PAGE_MASK; index2 =3D (page2 >> TARGET_PAGE_BITS) & (CPU_TLB_SIZE - 1); - tlb_addr2 =3D env->tlb_table[mmu_idx][index2].addr_write; + tlb_addr2 =3D atomic_read(&env->tlb_table[mmu_idx][index2].addr_wr= ite); if (!tlb_hit_page(tlb_addr2, page2) && !VICTIM_TLB_HIT(addr_write, page2)) { tlb_fill(ENV_GET_CPU(env), page2, DATA_SIZE, MMU_DATA_STORE, @@ -354,7 +356,8 @@ void helper_be_st_name(CPUArchState *env, target_ulong = addr, DATA_TYPE val, { unsigned mmu_idx =3D get_mmuidx(oi); int index =3D (addr >> TARGET_PAGE_BITS) & (CPU_TLB_SIZE - 1); - target_ulong tlb_addr =3D env->tlb_table[mmu_idx][index].addr_write; + target_ulong tlb_addr =3D + atomic_read(&env->tlb_table[mmu_idx][index].addr_write); unsigned a_bits =3D get_alignment_bits(get_memop(oi)); uintptr_t haddr; =20 @@ -369,7 +372,8 @@ void helper_be_st_name(CPUArchState *env, target_ulong = addr, DATA_TYPE val, tlb_fill(ENV_GET_CPU(env), addr, DATA_SIZE, MMU_DATA_STORE, mmu_idx, retaddr); } - tlb_addr =3D env->tlb_table[mmu_idx][index].addr_write & ~TLB_INVA= LID_MASK; + tlb_addr =3D atomic_read(&env->tlb_table[mmu_idx][index].addr_writ= e) & + ~TLB_INVALID_MASK; } =20 /* Handle an IO access. */ @@ -398,7 +402,7 @@ void helper_be_st_name(CPUArchState *env, target_ulong = addr, DATA_TYPE val, cannot evict the first. */ page2 =3D (addr + DATA_SIZE) & TARGET_PAGE_MASK; index2 =3D (page2 >> TARGET_PAGE_BITS) & (CPU_TLB_SIZE - 1); - tlb_addr2 =3D env->tlb_table[mmu_idx][index2].addr_write; + tlb_addr2 =3D atomic_read(&env->tlb_table[mmu_idx][index2].addr_wr= ite); if (!tlb_hit_page(tlb_addr2, page2) && !VICTIM_TLB_HIT(addr_write, page2)) { tlb_fill(ENV_GET_CPU(env), page2, DATA_SIZE, MMU_DATA_STORE, diff --git a/include/exec/cpu_ldst.h b/include/exec/cpu_ldst.h index 41ed0526e2..9581587ce1 100644 --- a/include/exec/cpu_ldst.h +++ b/include/exec/cpu_ldst.h @@ -426,7 +426,7 @@ static inline void *tlb_vaddr_to_host(CPUArchState *env= , abi_ptr addr, tlb_addr =3D tlbentry->addr_read; break; case 1: - tlb_addr =3D tlbentry->addr_write; + tlb_addr =3D atomic_read(&tlbentry->addr_write); break; case 2: tlb_addr =3D tlbentry->addr_code; diff --git a/include/exec/cpu_ldst_template.h b/include/exec/cpu_ldst_templ= ate.h index 4db2302962..ba7a11123c 100644 --- a/include/exec/cpu_ldst_template.h +++ b/include/exec/cpu_ldst_template.h @@ -176,7 +176,7 @@ glue(glue(glue(cpu_st, SUFFIX), MEMSUFFIX), _ra)(CPUArc= hState *env, addr =3D ptr; page_index =3D (addr >> TARGET_PAGE_BITS) & (CPU_TLB_SIZE - 1); mmu_idx =3D CPU_MMU_INDEX; - if (unlikely(env->tlb_table[mmu_idx][page_index].addr_write !=3D + if (unlikely(atomic_read(&env->tlb_table[mmu_idx][page_index].addr_wri= te) !=3D (addr & (TARGET_PAGE_MASK | (DATA_SIZE - 1))))) { oi =3D make_memop_idx(SHIFT, mmu_idx); glue(glue(helper_ret_st, SUFFIX), MMUSUFFIX)(env, addr, v, oi, diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c index 01b33f9d28..77091474ad 100644 --- a/accel/tcg/cputlb.c +++ b/accel/tcg/cputlb.c @@ -249,7 +249,7 @@ static inline bool tlb_hit_page_anyprot(CPUTLBEntry *tl= b_entry, target_ulong page) { return tlb_hit_page(tlb_entry->addr_read, page) || - tlb_hit_page(tlb_entry->addr_write, page) || + tlb_hit_page(atomic_read(&tlb_entry->addr_write), page) || tlb_hit_page(tlb_entry->addr_code, page); } =20 @@ -836,7 +836,7 @@ static void io_writex(CPUArchState *env, CPUIOTLBEntry = *iotlbentry, tlb_fill(cpu, addr, size, MMU_DATA_STORE, mmu_idx, retaddr); =20 index =3D (addr >> TARGET_PAGE_BITS) & (CPU_TLB_SIZE - 1); - tlb_addr =3D env->tlb_table[mmu_idx][index].addr_write; + tlb_addr =3D atomic_read(&env->tlb_table[mmu_idx][index].addr_writ= e); if (!(tlb_addr & ~(TARGET_PAGE_MASK | TLB_RECHECK))) { /* RAM access */ uintptr_t haddr =3D addr + env->tlb_table[mmu_idx][index].adde= nd; @@ -883,7 +883,9 @@ static bool victim_tlb_hit(CPUArchState *env, size_t mm= u_idx, size_t index, size_t vidx; for (vidx =3D 0; vidx < CPU_VTLB_SIZE; ++vidx) { CPUTLBEntry *vtlb =3D &env->tlb_v_table[mmu_idx][vidx]; - target_ulong cmp =3D *(target_ulong *)((uintptr_t)vtlb + elt_ofs); + /* elt_ofs might correspond to .addr_write, so use atomic_read */ + target_ulong cmp =3D + atomic_read((target_ulong *)((uintptr_t)vtlb + elt_ofs)); =20 if (cmp =3D=3D page) { /* Found entry in victim tlb, swap tlb and iotlb. */ @@ -955,7 +957,8 @@ void probe_write(CPUArchState *env, target_ulong addr, = int size, int mmu_idx, uintptr_t retaddr) { int index =3D (addr >> TARGET_PAGE_BITS) & (CPU_TLB_SIZE - 1); - target_ulong tlb_addr =3D env->tlb_table[mmu_idx][index].addr_write; + target_ulong tlb_addr =3D + atomic_read(&env->tlb_table[mmu_idx][index].addr_write); =20 if (!tlb_hit(tlb_addr, addr)) { /* TLB entry is for a different page */ @@ -975,7 +978,7 @@ static void *atomic_mmu_lookup(CPUArchState *env, targe= t_ulong addr, size_t mmu_idx =3D get_mmuidx(oi); size_t index =3D (addr >> TARGET_PAGE_BITS) & (CPU_TLB_SIZE - 1); CPUTLBEntry *tlbe =3D &env->tlb_table[mmu_idx][index]; - target_ulong tlb_addr =3D tlbe->addr_write; + target_ulong tlb_addr =3D atomic_read(&tlbe->addr_write); TCGMemOp mop =3D get_memop(oi); int a_bits =3D get_alignment_bits(mop); int s_bits =3D mop & MO_SIZE; @@ -1006,7 +1009,7 @@ static void *atomic_mmu_lookup(CPUArchState *env, tar= get_ulong addr, tlb_fill(ENV_GET_CPU(env), addr, 1 << s_bits, MMU_DATA_STORE, mmu_idx, retaddr); } - tlb_addr =3D tlbe->addr_write & ~TLB_INVALID_MASK; + tlb_addr =3D atomic_read(&tlbe->addr_write) & ~TLB_INVALID_MASK; } =20 /* Notice an IO access or a needs-MMU-lookup access */ --=20 2.17.1