From nobody Sat Sep 21 05:41:35 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=linaro.org ARC-Seal: i=1; a=rsa-sha256; t=1671599047; cv=none; d=zohomail.com; s=zohoarc; b=P6xOVdpntT7oLqZuIsD0nwE3e+MP9pEmclO1qXgQKiOPJfuCPRQvj5U9lwGAo31Rz1rNmQDo4eR3elVfZe99AwPTEEQ4AyVPpedsPaC/tb8FHiD9ZLf0otptV3oYAjhvfbLQFZkLSKR9ogWUoVNf+SjNCbMYS1msBnI+XgJ/01U= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1671599047; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=J/h0b2L1icbN8detkr/ytUwOyb8luYEYfEhmAMSx4bk=; b=UVnudbir0hBsypkDv+YTOY7OdBKbjj5Rr/OmkyZpsekjwek09pXPHVGxfX7dJIzboR44fcIWdnnzaxNooo0TAIMx9IG6nJDGVBkiUVoZvXczs8iv2vf2JXegUlEopmDoRlzbUJqPllnFpYzrE0Y7LXKiJR9Z/pZi42NIcv5oxBo= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1671599047790786.8046082111445; Tue, 20 Dec 2022 21:04:07 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1p7rGI-0002GC-GJ; Wed, 21 Dec 2022 00:03:26 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1p7rGF-0002DB-Ro for qemu-devel@nongnu.org; Wed, 21 Dec 2022 00:03:23 -0500 Received: from mail-pj1-x102b.google.com ([2607:f8b0:4864:20::102b]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1p7rGC-0003N9-PD for qemu-devel@nongnu.org; Wed, 21 Dec 2022 00:03:23 -0500 Received: by mail-pj1-x102b.google.com with SMTP id k88-20020a17090a4ce100b00219d0b857bcso973386pjh.1 for ; Tue, 20 Dec 2022 21:03:20 -0800 (PST) Received: from stoup.. ([2602:47:d48c:8101:3efa:624c:5fb:32c0]) by smtp.gmail.com with ESMTPSA id a8-20020a17090a688800b002135e8074b1sm390645pjd.55.2022.12.20.21.03.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 20 Dec 2022 21:03:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=J/h0b2L1icbN8detkr/ytUwOyb8luYEYfEhmAMSx4bk=; b=OAwnQBw6OsU5Cf+Ez1dyjmoT9BuktyrQQENAmJHCVOVzLZpTtiQ66fPME++BzwhxPz AG9DBru+5fBMbi+ewuvZt2dgZeKPnDCG+qvQOzm2+6zY8Aw4RcXbCaB59TMo1BQwdlXk ptLe9CuykOw4jRToTilWgIkvb16aRxdrJjEeZoDobHLJa4dJBZzXa/07sJ+B4QNr5GfH uCTKGnF8fk0wR/Af52lU1odHRkDxG2Mcw8AFgPaWCx88kxh4lZg3zWkqsWzjiORxDH/B WcEVy6yB6EWWl3lOctkLDqbWZRYOvL6ELLoLDaj0LFnp8k7lTnnpFkwulIKKpS1ZhEMq u33Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=J/h0b2L1icbN8detkr/ytUwOyb8luYEYfEhmAMSx4bk=; b=WOJPUw50UI3o2iIT6YLjSI8F+vWtkAoRh5cZHxi3ad4Y9iS42iDVFHzehuE7urQpFR cs1hgtfgE1yLO+1HaWP1dE3w56XNdV2q0nosgUy78EepDYzsBn3l5TwrKkXPT41Hoj76 bwKk6clQ+Zv87WZOlHwEnTAbZKZB/lRhZxPTQ+S/DxuH06jcic/i/0Fu+1TR2C1oB97y LOPjepSUDT9EMiJJj2NCPeIY884ceTETAepOBvwTEDxXhPswo+GetlymwbZpAQPYbnvM SdJMOsJ/ARc6vTsdo7hRjCLFA9999V+xXtHFoV8b321mMg/9MY7AXa+7ir4ewNIIVed5 /fHw== X-Gm-Message-State: AFqh2kq5SDn1pmBRYbitX34GJvLMeqBsXo8OZUnvtN4S9lzkNA+wEYp6 QYdI26YWw4ETJxQGXPpGF+605CHWptlFBMh+ X-Google-Smtp-Source: AMrXdXsnb+epzLWjraLIL/Dio+o4kDCwe8q3RPZFxELPPvKeeYby5y0wKwEw8zfbSZRl54HqhvUsiQ== X-Received: by 2002:a17:90a:66ce:b0:219:fe32:5dc with SMTP id z14-20020a17090a66ce00b00219fe3205dcmr715500pjl.18.1671598998853; Tue, 20 Dec 2022 21:03:18 -0800 (PST) From: Richard Henderson To: qemu-devel@nongnu.org Cc: peter.maydell@linaro.org, =?UTF-8?q?Alex=20Benn=C3=A9e?= Subject: [PULL v2 03/14] accel/tcg: Use interval tree for TBs in user-only mode Date: Tue, 20 Dec 2022 21:03:02 -0800 Message-Id: <20221221050313.2950701-4-richard.henderson@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20221221050313.2950701-1-richard.henderson@linaro.org> References: <20221221050313.2950701-1-richard.henderson@linaro.org> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2607:f8b0:4864:20::102b; envelope-from=richard.henderson@linaro.org; helo=mail-pj1-x102b.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @linaro.org) X-ZM-MESSAGEID: 1671599048529100001 Begin weaning user-only away from PageDesc. Since, for user-only, all TB (and page) manipulation is done with a single mutex, and there is no virtual/physical discontinuity to split a TB across discontinuous pages, place all of the TBs into a single IntervalTree. This makes it trivial to find all of the TBs intersecting a range. Retain the existing PageDesc + linked list implementation for system mode. Move the portion of the implementation that overlaps the new user-only code behind the common ifdef. Reviewed-by: Alex Benn=C3=A9e Signed-off-by: Richard Henderson --- accel/tcg/internal.h | 16 +- include/exec/exec-all.h | 43 ++++- accel/tcg/tb-maint.c | 387 ++++++++++++++++++++++---------------- accel/tcg/translate-all.c | 4 +- 4 files changed, 279 insertions(+), 171 deletions(-) diff --git a/accel/tcg/internal.h b/accel/tcg/internal.h index cb13bade4f..bf1bf62e2a 100644 --- a/accel/tcg/internal.h +++ b/accel/tcg/internal.h @@ -24,14 +24,13 @@ #endif =20 typedef struct PageDesc { - /* list of TBs intersecting this ram page */ - uintptr_t first_tb; #ifdef CONFIG_USER_ONLY unsigned long flags; void *target_data; -#endif -#ifdef CONFIG_SOFTMMU +#else QemuSpin lock; + /* list of TBs intersecting this ram page */ + uintptr_t first_tb; #endif } PageDesc; =20 @@ -69,9 +68,6 @@ static inline PageDesc *page_find(tb_page_addr_t index) tb; tb =3D (TranslationBlock *)tb->field[n], n =3D (uintptr_t)tb = & 1, \ tb =3D (TranslationBlock *)((uintptr_t)tb & ~1)) =20 -#define PAGE_FOR_EACH_TB(pagedesc, tb, n) \ - TB_FOR_EACH_TAGGED((pagedesc)->first_tb, tb, n, page_next) - #define TB_FOR_EACH_JMP(head_tb, tb, n) \ TB_FOR_EACH_TAGGED((head_tb)->jmp_list_head, tb, n, jmp_list_next) =20 @@ -89,6 +85,12 @@ void do_assert_page_locked(const PageDesc *pd, const cha= r *file, int line); #endif void page_lock(PageDesc *pd); void page_unlock(PageDesc *pd); + +/* TODO: For now, still shared with translate-all.c for system mode. */ +typedef int PageForEachNext; +#define PAGE_FOR_EACH_TB(start, end, pagedesc, tb, n) \ + TB_FOR_EACH_TAGGED((pagedesc)->first_tb, tb, n, page_next) + #endif #if !defined(CONFIG_USER_ONLY) && defined(CONFIG_DEBUG_TCG) void assert_no_pages_locked(void); diff --git a/include/exec/exec-all.h b/include/exec/exec-all.h index 9b7bfbf09a..25e11b0a8d 100644 --- a/include/exec/exec-all.h +++ b/include/exec/exec-all.h @@ -24,6 +24,7 @@ #ifdef CONFIG_TCG #include "exec/cpu_ldst.h" #endif +#include "qemu/interval-tree.h" =20 /* allow to see translation results - the slowdown should be negligible, s= o we leave it */ #define DEBUG_DISAS @@ -559,11 +560,20 @@ struct TranslationBlock { =20 struct tb_tc tc; =20 - /* first and second physical page containing code. The lower bit - of the pointer tells the index in page_next[]. - The list is protected by the TB's page('s) lock(s) */ + /* + * Track tb_page_addr_t intervals that intersect this TB. + * For user-only, the virtual addresses are always contiguous, + * and we use a unified interval tree. For system, we use a + * linked list headed in each PageDesc. Within the list, the lsb + * of the previous pointer tells the index of page_next[], and the + * list is protected by the PageDesc lock(s). + */ +#ifdef CONFIG_USER_ONLY + IntervalTreeNode itree; +#else uintptr_t page_next[2]; tb_page_addr_t page_addr[2]; +#endif =20 /* jmp_lock placed here to fill a 4-byte hole. Its documentation is be= low */ QemuSpin jmp_lock; @@ -619,24 +629,51 @@ static inline uint32_t tb_cflags(const TranslationBlo= ck *tb) =20 static inline tb_page_addr_t tb_page_addr0(const TranslationBlock *tb) { +#ifdef CONFIG_USER_ONLY + return tb->itree.start; +#else return tb->page_addr[0]; +#endif } =20 static inline tb_page_addr_t tb_page_addr1(const TranslationBlock *tb) { +#ifdef CONFIG_USER_ONLY + tb_page_addr_t next =3D tb->itree.last & TARGET_PAGE_MASK; + return next =3D=3D (tb->itree.start & TARGET_PAGE_MASK) ? -1 : next; +#else return tb->page_addr[1]; +#endif } =20 static inline void tb_set_page_addr0(TranslationBlock *tb, tb_page_addr_t addr) { +#ifdef CONFIG_USER_ONLY + tb->itree.start =3D addr; + /* + * To begin, we record an interval of one byte. When the translation + * loop encounters a second page, the interval will be extended to + * include the first byte of the second page, which is sufficient to + * allow tb_page_addr1() above to work properly. The final corrected + * interval will be set by tb_page_add() from tb->size before the + * node is added to the interval tree. + */ + tb->itree.last =3D addr; +#else tb->page_addr[0] =3D addr; +#endif } =20 static inline void tb_set_page_addr1(TranslationBlock *tb, tb_page_addr_t addr) { +#ifdef CONFIG_USER_ONLY + /* Extend the interval to the first byte of the second page. See abov= e. */ + tb->itree.last =3D addr; +#else tb->page_addr[1] =3D addr; +#endif } =20 /* current cflags for hashing/comparison */ diff --git a/accel/tcg/tb-maint.c b/accel/tcg/tb-maint.c index b5b90347ae..8da2c64d87 100644 --- a/accel/tcg/tb-maint.c +++ b/accel/tcg/tb-maint.c @@ -18,6 +18,7 @@ */ =20 #include "qemu/osdep.h" +#include "qemu/interval-tree.h" #include "exec/cputlb.h" #include "exec/log.h" #include "exec/exec-all.h" @@ -50,6 +51,75 @@ void tb_htable_init(void) qht_init(&tb_ctx.htable, tb_cmp, CODE_GEN_HTABLE_SIZE, mode); } =20 +#ifdef CONFIG_USER_ONLY +/* + * For user-only, since we are protecting all of memory with a single lock, + * and because the two pages of a TranslationBlock are always contiguous, + * use a single data structure to record all TranslationBlocks. + */ +static IntervalTreeRoot tb_root; + +static void tb_remove_all(void) +{ + assert_memory_lock(); + memset(&tb_root, 0, sizeof(tb_root)); +} + +/* Call with mmap_lock held. */ +static void tb_record(TranslationBlock *tb, PageDesc *p1, PageDesc *p2) +{ + /* translator_loop() must have made all TB pages non-writable */ + assert(!(p1->flags & PAGE_WRITE)); + if (p2) { + assert(!(p2->flags & PAGE_WRITE)); + } + + assert_memory_lock(); + + tb->itree.last =3D tb->itree.start + tb->size - 1; + interval_tree_insert(&tb->itree, &tb_root); +} + +/* Call with mmap_lock held. */ +static void tb_remove(TranslationBlock *tb) +{ + assert_memory_lock(); + interval_tree_remove(&tb->itree, &tb_root); +} + +/* TODO: For now, still shared with translate-all.c for system mode. */ +#define PAGE_FOR_EACH_TB(start, end, pagedesc, T, N) \ + for (T =3D foreach_tb_first(start, end), \ + N =3D foreach_tb_next(T, start, end); \ + T !=3D NULL; \ + T =3D N, N =3D foreach_tb_next(N, start, end)) + +typedef TranslationBlock *PageForEachNext; + +static PageForEachNext foreach_tb_first(tb_page_addr_t start, + tb_page_addr_t end) +{ + IntervalTreeNode *n =3D interval_tree_iter_first(&tb_root, start, end = - 1); + return n ? container_of(n, TranslationBlock, itree) : NULL; +} + +static PageForEachNext foreach_tb_next(PageForEachNext tb, + tb_page_addr_t start, + tb_page_addr_t end) +{ + IntervalTreeNode *n; + + if (tb) { + n =3D interval_tree_iter_next(&tb->itree, start, end - 1); + if (n) { + return container_of(n, TranslationBlock, itree); + } + } + return NULL; +} + +#else + /* Set to NULL all the 'first_tb' fields in all PageDescs. */ static void tb_remove_all_1(int level, void **lp) { @@ -84,6 +154,70 @@ static void tb_remove_all(void) } } =20 +/* + * Add the tb in the target page and protect it if necessary. + * Called with @p->lock held. + */ +static inline void tb_page_add(PageDesc *p, TranslationBlock *tb, + unsigned int n) +{ + bool page_already_protected; + + assert_page_locked(p); + + tb->page_next[n] =3D p->first_tb; + page_already_protected =3D p->first_tb !=3D 0; + p->first_tb =3D (uintptr_t)tb | n; + + /* + * If some code is already present, then the pages are already + * protected. So we handle the case where only the first TB is + * allocated in a physical page. + */ + if (!page_already_protected) { + tlb_protect_code(tb->page_addr[n] & TARGET_PAGE_MASK); + } +} + +static void tb_record(TranslationBlock *tb, PageDesc *p1, PageDesc *p2) +{ + tb_page_add(p1, tb, 0); + if (unlikely(p2)) { + tb_page_add(p2, tb, 1); + } +} + +static inline void tb_page_remove(PageDesc *pd, TranslationBlock *tb) +{ + TranslationBlock *tb1; + uintptr_t *pprev; + PageForEachNext n1; + + assert_page_locked(pd); + pprev =3D &pd->first_tb; + PAGE_FOR_EACH_TB(unused, unused, pd, tb1, n1) { + if (tb1 =3D=3D tb) { + *pprev =3D tb1->page_next[n1]; + return; + } + pprev =3D &tb1->page_next[n1]; + } + g_assert_not_reached(); +} + +static void tb_remove(TranslationBlock *tb) +{ + PageDesc *pd; + + pd =3D page_find(tb->page_addr[0] >> TARGET_PAGE_BITS); + tb_page_remove(pd, tb); + if (unlikely(tb->page_addr[1] !=3D -1)) { + pd =3D page_find(tb->page_addr[1] >> TARGET_PAGE_BITS); + tb_page_remove(pd, tb); + } +} +#endif /* CONFIG_USER_ONLY */ + /* flush all the translation blocks */ static void do_tb_flush(CPUState *cpu, run_on_cpu_data tb_flush_count) { @@ -128,28 +262,6 @@ void tb_flush(CPUState *cpu) } } =20 -/* - * user-mode: call with mmap_lock held - * !user-mode: call with @pd->lock held - */ -static inline void tb_page_remove(PageDesc *pd, TranslationBlock *tb) -{ - TranslationBlock *tb1; - uintptr_t *pprev; - unsigned int n1; - - assert_page_locked(pd); - pprev =3D &pd->first_tb; - PAGE_FOR_EACH_TB(pd, tb1, n1) { - if (tb1 =3D=3D tb) { - *pprev =3D tb1->page_next[n1]; - return; - } - pprev =3D &tb1->page_next[n1]; - } - g_assert_not_reached(); -} - /* remove @orig from its @n_orig-th jump list */ static inline void tb_remove_from_jmp_list(TranslationBlock *orig, int n_o= rig) { @@ -255,7 +367,6 @@ static void tb_jmp_cache_inval_tb(TranslationBlock *tb) */ static void do_tb_phys_invalidate(TranslationBlock *tb, bool rm_from_page_= list) { - PageDesc *p; uint32_t h; tb_page_addr_t phys_pc; uint32_t orig_cflags =3D tb_cflags(tb); @@ -277,13 +388,7 @@ static void do_tb_phys_invalidate(TranslationBlock *tb= , bool rm_from_page_list) =20 /* remove the TB from the page list */ if (rm_from_page_list) { - p =3D page_find(phys_pc >> TARGET_PAGE_BITS); - tb_page_remove(p, tb); - phys_pc =3D tb_page_addr1(tb); - if (phys_pc !=3D -1) { - p =3D page_find(phys_pc >> TARGET_PAGE_BITS); - tb_page_remove(p, tb); - } + tb_remove(tb); } =20 /* remove the TB from the hash list */ @@ -387,41 +492,6 @@ void tb_phys_invalidate(TranslationBlock *tb, tb_page_= addr_t page_addr) } } =20 -/* - * Add the tb in the target page and protect it if necessary. - * Called with mmap_lock held for user-mode emulation. - * Called with @p->lock held in !user-mode. - */ -static inline void tb_page_add(PageDesc *p, TranslationBlock *tb, - unsigned int n, tb_page_addr_t page_addr) -{ -#ifndef CONFIG_USER_ONLY - bool page_already_protected; -#endif - - assert_page_locked(p); - - tb->page_next[n] =3D p->first_tb; -#ifndef CONFIG_USER_ONLY - page_already_protected =3D p->first_tb !=3D (uintptr_t)NULL; -#endif - p->first_tb =3D (uintptr_t)tb | n; - -#if defined(CONFIG_USER_ONLY) - /* translator_loop() must have made all TB pages non-writable */ - assert(!(p->flags & PAGE_WRITE)); -#else - /* - * If some code is already present, then the pages are already - * protected. So we handle the case where only the first TB is - * allocated in a physical page. - */ - if (!page_already_protected) { - tlb_protect_code(page_addr); - } -#endif -} - /* * Add a new TB and link it to the physical page tables. phys_page2 is * (-1) to indicate that only one page contains the TB. @@ -453,10 +523,7 @@ TranslationBlock *tb_link_page(TranslationBlock *tb, t= b_page_addr_t phys_pc, * we can only insert TBs that are fully initialized. */ page_lock_pair(&p, phys_pc, &p2, phys_page2, true); - tb_page_add(p, tb, 0, phys_pc); - if (p2) { - tb_page_add(p2, tb, 1, phys_page2); - } + tb_record(tb, p, p2); =20 /* add in the hash table */ h =3D tb_hash_func(phys_pc, (TARGET_TB_PCREL ? 0 : tb_pc(tb)), @@ -465,10 +532,7 @@ TranslationBlock *tb_link_page(TranslationBlock *tb, t= b_page_addr_t phys_pc, =20 /* remove TB from the page(s) if we couldn't insert it */ if (unlikely(existing_tb)) { - tb_page_remove(p, tb); - if (p2) { - tb_page_remove(p2, tb); - } + tb_remove(tb); tb =3D existing_tb; } =20 @@ -479,6 +543,87 @@ TranslationBlock *tb_link_page(TranslationBlock *tb, t= b_page_addr_t phys_pc, return tb; } =20 +#ifdef CONFIG_USER_ONLY +/* + * Invalidate all TBs which intersect with the target address range. + * Called with mmap_lock held for user-mode emulation. + * NOTE: this function must not be called while a TB is running. + */ +void tb_invalidate_phys_range(tb_page_addr_t start, tb_page_addr_t end) +{ + TranslationBlock *tb; + PageForEachNext n; + + assert_memory_lock(); + + PAGE_FOR_EACH_TB(start, end, unused, tb, n) { + tb_phys_invalidate__locked(tb); + } +} + +/* + * Invalidate all TBs which intersect with the target address page @addr. + * Called with mmap_lock held for user-mode emulation + * NOTE: this function must not be called while a TB is running. + */ +void tb_invalidate_phys_page(tb_page_addr_t addr) +{ + tb_page_addr_t start, end; + + start =3D addr & TARGET_PAGE_MASK; + end =3D start + TARGET_PAGE_SIZE; + tb_invalidate_phys_range(start, end); +} + +/* + * Called with mmap_lock held. If pc is not 0 then it indicates the + * host PC of the faulting store instruction that caused this invalidate. + * Returns true if the caller needs to abort execution of the current + * TB (because it was modified by this store and the guest CPU has + * precise-SMC semantics). + */ +bool tb_invalidate_phys_page_unwind(tb_page_addr_t addr, uintptr_t pc) +{ + assert(pc !=3D 0); +#ifdef TARGET_HAS_PRECISE_SMC + assert_memory_lock(); + { + TranslationBlock *current_tb =3D tcg_tb_lookup(pc); + bool current_tb_modified =3D false; + TranslationBlock *tb; + PageForEachNext n; + + addr &=3D TARGET_PAGE_MASK; + + PAGE_FOR_EACH_TB(addr, addr + TARGET_PAGE_SIZE, unused, tb, n) { + if (current_tb =3D=3D tb && + (tb_cflags(current_tb) & CF_COUNT_MASK) !=3D 1) { + /* + * If we are modifying the current TB, we must stop its + * execution. We could be more precise by checking that + * the modification is after the current PC, but it would + * require a specialized function to partially restore + * the CPU state. + */ + current_tb_modified =3D true; + cpu_restore_state_from_tb(current_cpu, current_tb, pc); + } + tb_phys_invalidate__locked(tb); + } + + if (current_tb_modified) { + /* Force execution of one insn next time. */ + CPUState *cpu =3D current_cpu; + cpu->cflags_next_tb =3D 1 | CF_NOIRQ | curr_cflags(current_cpu= ); + return true; + } + } +#else + tb_invalidate_phys_page(addr); +#endif /* TARGET_HAS_PRECISE_SMC */ + return false; +} +#else /* * @p must be non-NULL. * user-mode: call with mmap_lock held. @@ -492,22 +637,17 @@ tb_invalidate_phys_page_range__locked(struct page_col= lection *pages, { TranslationBlock *tb; tb_page_addr_t tb_start, tb_end; - int n; + PageForEachNext n; #ifdef TARGET_HAS_PRECISE_SMC - CPUState *cpu =3D current_cpu; - bool current_tb_not_found =3D retaddr !=3D 0; bool current_tb_modified =3D false; - TranslationBlock *current_tb =3D NULL; + TranslationBlock *current_tb =3D retaddr ? tcg_tb_lookup(retaddr) : NU= LL; #endif /* TARGET_HAS_PRECISE_SMC */ =20 - assert_page_locked(p); - /* * We remove all the TBs in the range [start, end[. * XXX: see if in some cases it could be faster to invalidate all the = code */ - PAGE_FOR_EACH_TB(p, tb, n) { - assert_page_locked(p); + PAGE_FOR_EACH_TB(start, end, p, tb, n) { /* NOTE: this is subtle as a TB may span two physical pages */ if (n =3D=3D 0) { /* NOTE: tb_end may be after the end of the page, but @@ -521,11 +661,6 @@ tb_invalidate_phys_page_range__locked(struct page_coll= ection *pages, } if (!(tb_end <=3D start || tb_start >=3D end)) { #ifdef TARGET_HAS_PRECISE_SMC - if (current_tb_not_found) { - current_tb_not_found =3D false; - /* now we have a real cpu fault */ - current_tb =3D tcg_tb_lookup(retaddr); - } if (current_tb =3D=3D tb && (tb_cflags(current_tb) & CF_COUNT_MASK) !=3D 1) { /* @@ -536,25 +671,25 @@ tb_invalidate_phys_page_range__locked(struct page_col= lection *pages, * restore the CPU state. */ current_tb_modified =3D true; - cpu_restore_state_from_tb(cpu, current_tb, retaddr); + cpu_restore_state_from_tb(current_cpu, current_tb, retaddr= ); } #endif /* TARGET_HAS_PRECISE_SMC */ tb_phys_invalidate__locked(tb); } } -#if !defined(CONFIG_USER_ONLY) + /* if no code remaining, no need to continue to use slow writes */ if (!p->first_tb) { tlb_unprotect_code(start); } -#endif + #ifdef TARGET_HAS_PRECISE_SMC if (current_tb_modified) { page_collection_unlock(pages); /* Force execution of one insn next time. */ - cpu->cflags_next_tb =3D 1 | CF_NOIRQ | curr_cflags(cpu); + current_cpu->cflags_next_tb =3D 1 | CF_NOIRQ | curr_cflags(current= _cpu); mmap_unlock(); - cpu_loop_exit_noexc(cpu); + cpu_loop_exit_noexc(current_cpu); } #endif } @@ -571,8 +706,6 @@ void tb_invalidate_phys_page(tb_page_addr_t addr) tb_page_addr_t start, end; PageDesc *p; =20 - assert_memory_lock(); - p =3D page_find(addr >> TARGET_PAGE_BITS); if (p =3D=3D NULL) { return; @@ -599,8 +732,6 @@ void tb_invalidate_phys_range(tb_page_addr_t start, tb_= page_addr_t end) struct page_collection *pages; tb_page_addr_t next; =20 - assert_memory_lock(); - pages =3D page_collection_lock(start, end); for (next =3D (start & TARGET_PAGE_MASK) + TARGET_PAGE_SIZE; start < end; @@ -611,12 +742,12 @@ void tb_invalidate_phys_range(tb_page_addr_t start, t= b_page_addr_t end) if (pd =3D=3D NULL) { continue; } + assert_page_locked(pd); tb_invalidate_phys_page_range__locked(pages, pd, start, bound, 0); } page_collection_unlock(pages); } =20 -#ifdef CONFIG_SOFTMMU /* * len must be <=3D 8 and start must be a multiple of len. * Called via softmmu_template.h when code areas are written to with @@ -630,8 +761,6 @@ void tb_invalidate_phys_page_fast(struct page_collectio= n *pages, { PageDesc *p; =20 - assert_memory_lock(); - p =3D page_find(start >> TARGET_PAGE_BITS); if (!p) { return; @@ -641,64 +770,4 @@ void tb_invalidate_phys_page_fast(struct page_collecti= on *pages, tb_invalidate_phys_page_range__locked(pages, p, start, start + len, retaddr); } -#else -/* - * Called with mmap_lock held. If pc is not 0 then it indicates the - * host PC of the faulting store instruction that caused this invalidate. - * Returns true if the caller needs to abort execution of the current - * TB (because it was modified by this store and the guest CPU has - * precise-SMC semantics). - */ -bool tb_invalidate_phys_page_unwind(tb_page_addr_t addr, uintptr_t pc) -{ - TranslationBlock *tb; - PageDesc *p; - int n; -#ifdef TARGET_HAS_PRECISE_SMC - TranslationBlock *current_tb =3D NULL; - CPUState *cpu =3D current_cpu; - bool current_tb_modified =3D false; -#endif - - assert_memory_lock(); - - addr &=3D TARGET_PAGE_MASK; - p =3D page_find(addr >> TARGET_PAGE_BITS); - if (!p) { - return false; - } - -#ifdef TARGET_HAS_PRECISE_SMC - if (p->first_tb && pc !=3D 0) { - current_tb =3D tcg_tb_lookup(pc); - } -#endif - assert_page_locked(p); - PAGE_FOR_EACH_TB(p, tb, n) { -#ifdef TARGET_HAS_PRECISE_SMC - if (current_tb =3D=3D tb && - (tb_cflags(current_tb) & CF_COUNT_MASK) !=3D 1) { - /* - * If we are modifying the current TB, we must stop its execut= ion. - * We could be more precise by checking that the modification = is - * after the current PC, but it would require a specialized - * function to partially restore the CPU state. - */ - current_tb_modified =3D true; - cpu_restore_state_from_tb(cpu, current_tb, pc); - } -#endif /* TARGET_HAS_PRECISE_SMC */ - tb_phys_invalidate(tb, addr); - } - p->first_tb =3D (uintptr_t)NULL; -#ifdef TARGET_HAS_PRECISE_SMC - if (current_tb_modified) { - /* Force execution of one insn next time. */ - cpu->cflags_next_tb =3D 1 | CF_NOIRQ | curr_cflags(cpu); - return true; - } -#endif - - return false; -} -#endif +#endif /* CONFIG_USER_ONLY */ diff --git a/accel/tcg/translate-all.c b/accel/tcg/translate-all.c index ac3ee3740c..b964ea44d7 100644 --- a/accel/tcg/translate-all.c +++ b/accel/tcg/translate-all.c @@ -709,7 +709,7 @@ page_collection_lock(tb_page_addr_t start, tb_page_addr= _t end) =20 for (index =3D start; index <=3D end; index++) { TranslationBlock *tb; - int n; + PageForEachNext n; =20 pd =3D page_find(index); if (pd =3D=3D NULL) { @@ -720,7 +720,7 @@ page_collection_lock(tb_page_addr_t start, tb_page_addr= _t end) goto retry; } assert_page_locked(pd); - PAGE_FOR_EACH_TB(pd, tb, n) { + PAGE_FOR_EACH_TB(unused, unused, pd, tb, n) { if (page_trylock_add(set, tb_page_addr0(tb)) || (tb_page_addr1(tb) !=3D -1 && page_trylock_add(set, tb_page_addr1(tb)))) { --=20 2.34.1