From nobody Mon Feb 9 21:08:28 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=linaro.org ARC-Seal: i=1; a=rsa-sha256; t=1583909169; cv=none; d=zohomail.com; s=zohoarc; b=eJd7KhyOARvBlOsiDDNVv+RBCVHTO7PbuZRJGXiY3KLs3iwgXwcNVYu8xPiINIh4r8qCy7Jm4Nsfg/xtIakynU1XRoErs9ppLVWw+L8xOW7uEwiDKi20ebWJiTWHoMFgfud4IoJPEILIDzz2HgQKZq1IuMGY6mfN2/mPZy90ll4= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1583909169; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=ME4wdyObWWzU36JOWOP068GfXM/kHUGKiy3+FE+Caek=; b=jjuuFiPw96xzDV5Trb2kWHkeWM/YquNtGb9g+uSmvrSCvkOdAv3PUoUzacmWjAedBGqvh8CJ1cyWiQgQkk6tl2jlHzzYLT8RM2gtCvuf2pIiQ+RoArUra+l7hD30IVV5cKVaIFKibk44Y0LzsB0Lc3IDlnLDjfBbcSDvfTat5io= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1583909169737583.3147542576406; Tue, 10 Mar 2020 23:46:09 -0700 (PDT) Received: from localhost ([::1]:46374 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jBv84-0007oS-M1 for importer@patchew.org; Wed, 11 Mar 2020 02:46:08 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:59301) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jBv6Q-000665-Es for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:27 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jBv6O-0001f1-Hm for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:26 -0400 Received: from mail-pg1-x541.google.com ([2607:f8b0:4864:20::541]:38245) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1jBv6O-0001cs-Cw for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:24 -0400 Received: by mail-pg1-x541.google.com with SMTP id x7so639014pgh.5 for ; Tue, 10 Mar 2020 23:44:24 -0700 (PDT) Received: from localhost.localdomain (97-126-123-70.tukw.qwest.net. [97.126.123.70]) by smtp.gmail.com with ESMTPSA id z3sm50137807pfz.155.2020.03.10.23.44.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 10 Mar 2020 23:44:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ME4wdyObWWzU36JOWOP068GfXM/kHUGKiy3+FE+Caek=; b=nAqcHY80e4ejxIJMyR8NQ/oenp6q+JKoqoSjzsnDo3V0irsukHBJXuIFSZMeH2FA91 nqThih0v1cRFxjbr0+jn2U6EUvJt+351gAGkWdsQAusWjD+k7ZaTRrMfVcBzFUqudBDs 22boIFp7rItiG50EB5sHmTOm72nPPhIJS06g+UEHGhfUS73ZT/FP6OTlWitNLXQlBEzr 7TVgPHIA4hzZx87mHNTDbqn3DTtgSD9DTLzNLd578xOMPPNqmAivDyftR8KhKM84zPxh OVkWZsWp3ZHMK9Ce/V6QQEVWr58hC6Lnr3t+DW6KHbHRF0DdT54SXtBgkpMQSNF8FK7z LIfw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ME4wdyObWWzU36JOWOP068GfXM/kHUGKiy3+FE+Caek=; b=nU5XCeWwXbqhto6JeU5oamSSXxTywmXgW6mm5lolLlmn26B1EiR7DOqggSL5AMxzVd VeFi/l2VU+9eCpMlD50VHsizCC/ZGbB2tIJ6kRp7lrWHJSDsnITp0m86q2W8CRiPiSdI /4ZOw7Z630bQ0sbqSj3+XmZW7RHwkpkqJ3UV0sHzCCDlzKGZiJ8fJwFdtbZ8JZBH8QzN jt5ucceGpM5Ngu8oOA221MKBTZlra12d2ivv/EfJ//t2yZMV6/GCTQQpH1kuBcnxj2RX Immfv2yJ6TyCD+nIqXX+f5WOIelmrwkYnL2okQWLnMBluTZpN2Vjc+Q30/BOgnOuK31E maJw== X-Gm-Message-State: ANhLgQ26GrGurwQBKKAYmJF1gs+xc7ZpaBFw65gdeS+BsAXTabOLp9cx 3tubebmVIU3TfBmQye0euEnr+4TlD7w= X-Google-Smtp-Source: ADFU+vuGqUilr1XU/ok4yOfa9+P48hgRWQbcNbgdR/RVEKSyJehZgAE/NEzq/VEolA//OnSo/40W9A== X-Received: by 2002:a62:7cc9:: with SMTP id x192mr767250pfc.176.1583909062988; Tue, 10 Mar 2020 23:44:22 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH 01/16] accel/tcg: Add block comment for probe_access Date: Tue, 10 Mar 2020 23:44:05 -0700 Message-Id: <20200311064420.30606-2-richard.henderson@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200311064420.30606-1-richard.henderson@linaro.org> References: <20200311064420.30606-1-richard.henderson@linaro.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::541 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, qemu-arm@nongnu.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @linaro.org) Content-Type: text/plain; charset="utf-8" Signed-off-by: Richard Henderson Reviewed-by: Peter Maydell --- include/exec/exec-all.h | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/include/exec/exec-all.h b/include/exec/exec-all.h index 350c4b451b..d656a1f05c 100644 --- a/include/exec/exec-all.h +++ b/include/exec/exec-all.h @@ -330,6 +330,23 @@ static inline void tlb_flush_by_mmuidx_all_cpus_synced= (CPUState *cpu, { } #endif +/** + * probe_access: + * @env: CPUArchState + * @addr: guest virtual address to look up + * @size: size of the access + * @access_type: read, write or execute permission + * @mmu_idx: MMU index to use for lookup + * @retaddr: return address for unwinding + * + * Look up the guest virtual address @addr. Raise an exception if the + * page does not satisfy @access_type. Raise an exception if the + * access (@addr, @size) hits a watchpoint. For writes, mark a clean + * page as dirty. + * + * Finally, return the host address for a page that is backed by RAM, + * or NULL if the page requires I/O. + */ void *probe_access(CPUArchState *env, target_ulong addr, int size, MMUAccessType access_type, int mmu_idx, uintptr_t retad= dr); =20 --=20 2.20.1 From nobody Mon Feb 9 21:08:28 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=linaro.org ARC-Seal: i=1; a=rsa-sha256; t=1583909176; cv=none; d=zohomail.com; s=zohoarc; b=ChW15lrvAIiRk0FRVgVbMgXchX+2mWB8OS3ZARPKghYIqpdB5qH8w03pLqsJQLKt+EmNoxqWk1iYcN43/aKXAVTAMJyr9Q4XGHy3AFd1ZoRRDrR1UDuCXmpPjSp+zRatiMwYTERMB0I6yqWzQh+cCLvo7JLqRYvBSgZTyc27RuM= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1583909176; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=WFkeQZK5wV/2b9hgN0NoiBdZ/Pf2pozUeegzyTZXJXc=; b=fanfHpUaSRWw2jyCBSWM6fxL5McEyu2KOBcTO0Tttlom2caUYqe+X8sHmvQyFvsQAbujibU7b3kVAyO1n1VccUX5mOzmEZKky7CxzmZs+FM9JF3nSyKuiCIz8cZlzYSmXeew8i+CsuQdTZRw5dUFVTggwopRqfwyjyX0rtSwcZk= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1583909176209329.62947636734555; Tue, 10 Mar 2020 23:46:16 -0700 (PDT) Received: from localhost ([::1]:46378 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jBv8B-00083H-2z for importer@patchew.org; Wed, 11 Mar 2020 02:46:15 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:59385) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jBv6V-00069K-US for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:34 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jBv6S-0001mW-5h for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:30 -0400 Received: from mail-pg1-x544.google.com ([2607:f8b0:4864:20::544]:39444) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1jBv6Q-0001hB-D0 for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:27 -0400 Received: by mail-pg1-x544.google.com with SMTP id s2so637509pgv.6 for ; Tue, 10 Mar 2020 23:44:26 -0700 (PDT) Received: from localhost.localdomain (97-126-123-70.tukw.qwest.net. [97.126.123.70]) by smtp.gmail.com with ESMTPSA id z3sm50137807pfz.155.2020.03.10.23.44.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 10 Mar 2020 23:44:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=WFkeQZK5wV/2b9hgN0NoiBdZ/Pf2pozUeegzyTZXJXc=; b=faxQT0/TZ2lP8LdufRbHEAMgV5SqtHpRoMMBYOjXzi8/oIgSE3khGexXHAGUuNG0TG HBFx4yEsGDAuJfnSqyKCTqX3yy0UoII0WkRISbXRIvE1r5FuUy9PrM/cMnn7z5BD+ZoR Sm/7bl3KmX+FCEFDGgjYlJwxMRzggqNhG4pbEjEyH0aRzMhXhNnE75kJ3lv8fHbm7VNx gkOmIqWNgU1ALaPxbOXA/XouNF/Ej3x7j+H02mErdIAckpsh8fhVcNPDUlTBbQSyk+BU tqlm0WdPXZ0rXAXQsxHs4dVKNUX76eREDFyqqAzVgXlSMu0Rg7S4Xq2tI64LGIpxVXO5 9FXg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=WFkeQZK5wV/2b9hgN0NoiBdZ/Pf2pozUeegzyTZXJXc=; b=K31lWmGYEx0M7T/8X0rwbiiOfC/rQykIM7ng0gzcQyri5UwLgjV3pZpjxN6QpyDgs2 Cu9VX2kjqaS4++1ou4gX40nGfQnJqLcFV+Ypb+5/Ybmo6UMUNK2TeWrRo2r+Ufx9jTSP m+uLucfW6WQJ8L3yK/AK0Gr+uK3MzjJnCtodooZeUQr36WjMx5xJbWTqJjh7bkMwOhM1 m9dgTjK1K5Xe+fxQ0t0qNyLv0nYojG4/W+cmwE/pw7gSaq19Nin23N3pnUuOEAD8o6M9 NN9Oq84Jr+Eva3e5/r78iov6ixA1g3JG3lmF9bsPKaZif2tcC+6YRLywkCUpwyIBXwrR 2IJw== X-Gm-Message-State: ANhLgQ1w+RbMvjbhvItVqAJPmik5giBmKTfAgQBZEVhpS5f6yLnrS5JC hefIslMFGGDC1JEV7VJR7lesvzZlEEM= X-Google-Smtp-Source: ADFU+vscHFYOUsqUCYSm/DcBOf/RkUoHO+NY8/Ep+wTZXuiNzAk0nLaVUiILr/VG477fYH+5xaNpgg== X-Received: by 2002:a63:a65:: with SMTP id z37mr1405166pgk.31.1583909064289; Tue, 10 Mar 2020 23:44:24 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH 02/16] accel/tcg: Add probe_access_flags Date: Tue, 10 Mar 2020 23:44:06 -0700 Message-Id: <20200311064420.30606-3-richard.henderson@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200311064420.30606-1-richard.henderson@linaro.org> References: <20200311064420.30606-1-richard.henderson@linaro.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::544 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, qemu-arm@nongnu.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @linaro.org) Content-Type: text/plain; charset="utf-8" This new interface will allow targets to probe for a page and then handle watchpoints themselves. This will be most useful for vector predicated memory operations, where one page lookup can be used for many operations, and one test can avoid many watchpoint checks. Signed-off-by: Richard Henderson --- include/exec/cpu-all.h | 13 ++- include/exec/exec-all.h | 22 +++++ accel/tcg/cputlb.c | 178 ++++++++++++++++++++-------------------- accel/tcg/user-exec.c | 36 +++++--- 4 files changed, 150 insertions(+), 99 deletions(-) diff --git a/include/exec/cpu-all.h b/include/exec/cpu-all.h index 49384bb66a..43ddcf024c 100644 --- a/include/exec/cpu-all.h +++ b/include/exec/cpu-all.h @@ -328,7 +328,18 @@ CPUArchState *cpu_copy(CPUArchState *env); | CPU_INTERRUPT_TGT_EXT_3 \ | CPU_INTERRUPT_TGT_EXT_4) =20 -#if !defined(CONFIG_USER_ONLY) +#ifdef CONFIG_USER_ONLY + +/* + * Allow some level of source compatibility with softmmu. We do not + * support any of the more exotic features, so only invalid pages may + * be signaled by probe_access_flags(). + */ +#define TLB_INVALID_MASK (1 << (TARGET_PAGE_BITS_MIN - 1)) +#define TLB_MMIO 0 +#define TLB_WATCHPOINT 0 + +#else =20 /* * Flags stored in the low bits of the TLB virtual address. diff --git a/include/exec/exec-all.h b/include/exec/exec-all.h index d656a1f05c..8792bea07a 100644 --- a/include/exec/exec-all.h +++ b/include/exec/exec-all.h @@ -362,6 +362,28 @@ static inline void *probe_read(CPUArchState *env, targ= et_ulong addr, int size, return probe_access(env, addr, size, MMU_DATA_LOAD, mmu_idx, retaddr); } =20 +/** + * probe_access_flags: + * @env: CPUArchState + * @addr: guest virtual address to look up + * @access_type: read, write or execute permission + * @mmu_idx: MMU index to use for lookup + * @nonfault: suppress the fault + * @phost: return value for host address + * @retaddr: return address for unwinding + * + * Similar to probe_access, loosely returning the TLB_FLAGS_MASK for + * the page, and storing the host address for RAM in @phost. + * + * If @nonfault is set, do not raise an exception but return TLB_INVALID_M= ASK. + * Do not handle watchpoints, but include TLB_WATCHPOINT in the returned f= lags. + * Do handle clean pages, so exclude TLB_NOTDIRY from the returned flags. + * For simplicity, all "mmio-like" flags are folded to TLB_MMIO. + */ +int probe_access_flags(CPUArchState *env, target_ulong addr, + MMUAccessType access_type, int mmu_idx, + bool nonfault, void **phost, uintptr_t retaddr); + #define CODE_GEN_ALIGN 16 /* must be >=3D of the size of a icach= e line */ =20 /* Estimated block size for TB allocation. */ diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c index e3b5750c3b..687474ccea 100644 --- a/accel/tcg/cputlb.c +++ b/accel/tcg/cputlb.c @@ -1231,86 +1231,16 @@ static void notdirty_write(CPUState *cpu, vaddr mem= _vaddr, unsigned size, } } =20 -/* - * Probe for whether the specified guest access is permitted. If it is not - * permitted then an exception will be taken in the same way as if this - * were a real access (and we will not return). - * If the size is 0 or the page requires I/O access, returns NULL; otherwi= se, - * returns the address of the host page similar to tlb_vaddr_to_host(). - */ -void *probe_access(CPUArchState *env, target_ulong addr, int size, - MMUAccessType access_type, int mmu_idx, uintptr_t retad= dr) +static int probe_access_internal(CPUArchState *env, target_ulong addr, + int fault_size, MMUAccessType access_type, + int mmu_idx, bool nonfault, + void **phost, uintptr_t retaddr) { uintptr_t index =3D tlb_index(env, mmu_idx, addr); CPUTLBEntry *entry =3D tlb_entry(env, mmu_idx, addr); - target_ulong tlb_addr; - size_t elt_ofs; - int wp_access; - - g_assert(-(addr | TARGET_PAGE_MASK) >=3D size); - - switch (access_type) { - case MMU_DATA_LOAD: - elt_ofs =3D offsetof(CPUTLBEntry, addr_read); - wp_access =3D BP_MEM_READ; - break; - case MMU_DATA_STORE: - elt_ofs =3D offsetof(CPUTLBEntry, addr_write); - wp_access =3D BP_MEM_WRITE; - break; - case MMU_INST_FETCH: - elt_ofs =3D offsetof(CPUTLBEntry, addr_code); - wp_access =3D BP_MEM_READ; - break; - default: - g_assert_not_reached(); - } - tlb_addr =3D tlb_read_ofs(entry, elt_ofs); - - if (unlikely(!tlb_hit(tlb_addr, addr))) { - if (!victim_tlb_hit(env, mmu_idx, index, elt_ofs, - addr & TARGET_PAGE_MASK)) { - tlb_fill(env_cpu(env), addr, size, access_type, mmu_idx, retad= dr); - /* TLB resize via tlb_fill may have moved the entry. */ - index =3D tlb_index(env, mmu_idx, addr); - entry =3D tlb_entry(env, mmu_idx, addr); - } - tlb_addr =3D tlb_read_ofs(entry, elt_ofs); - } - - if (!size) { - return NULL; - } - - if (unlikely(tlb_addr & TLB_FLAGS_MASK)) { - CPUIOTLBEntry *iotlbentry =3D &env_tlb(env)->d[mmu_idx].iotlb[inde= x]; - - /* Reject I/O access, or other required slow-path. */ - if (tlb_addr & (TLB_MMIO | TLB_BSWAP | TLB_DISCARD_WRITE)) { - return NULL; - } - - /* Handle watchpoints. */ - if (tlb_addr & TLB_WATCHPOINT) { - cpu_check_watchpoint(env_cpu(env), addr, size, - iotlbentry->attrs, wp_access, retaddr); - } - - /* Handle clean RAM pages. */ - if (tlb_addr & TLB_NOTDIRTY) { - notdirty_write(env_cpu(env), addr, size, iotlbentry, retaddr); - } - } - - return (void *)((uintptr_t)addr + entry->addend); -} - -void *tlb_vaddr_to_host(CPUArchState *env, abi_ptr addr, - MMUAccessType access_type, int mmu_idx) -{ - CPUTLBEntry *entry =3D tlb_entry(env, mmu_idx, addr); - target_ulong tlb_addr, page; + target_ulong tlb_addr, page_addr; size_t elt_ofs; + int flags; =20 switch (access_type) { case MMU_DATA_LOAD: @@ -1325,20 +1255,19 @@ void *tlb_vaddr_to_host(CPUArchState *env, abi_ptr = addr, default: g_assert_not_reached(); } - - page =3D addr & TARGET_PAGE_MASK; tlb_addr =3D tlb_read_ofs(entry, elt_ofs); =20 - if (!tlb_hit_page(tlb_addr, page)) { - uintptr_t index =3D tlb_index(env, mmu_idx, addr); - - if (!victim_tlb_hit(env, mmu_idx, index, elt_ofs, page)) { + page_addr =3D addr & TARGET_PAGE_MASK; + if (!tlb_hit_page(tlb_addr, page_addr)) { + if (!victim_tlb_hit(env, mmu_idx, index, elt_ofs, page_addr)) { CPUState *cs =3D env_cpu(env); CPUClass *cc =3D CPU_GET_CLASS(cs); =20 - if (!cc->tlb_fill(cs, addr, 0, access_type, mmu_idx, true, 0))= { + if (!cc->tlb_fill(cs, addr, fault_size, access_type, + mmu_idx, nonfault, retaddr)) { /* Non-faulting page table read failed. */ - return NULL; + *phost =3D NULL; + return TLB_INVALID_MASK; } =20 /* TLB resize via tlb_fill may have moved the entry. */ @@ -1346,15 +1275,90 @@ void *tlb_vaddr_to_host(CPUArchState *env, abi_ptr = addr, } tlb_addr =3D tlb_read_ofs(entry, elt_ofs); } + flags =3D tlb_addr & TLB_FLAGS_MASK; =20 - if (tlb_addr & ~TARGET_PAGE_MASK) { - /* IO access */ + /* Fold all "mmio-like" bits into TLB_MMIO. This is not RAM. */ + if (unlikely(flags & ~(TLB_WATCHPOINT | TLB_NOTDIRTY))) { + *phost =3D NULL; + return TLB_MMIO; + } + + /* Everything else is RAM. */ + *phost =3D (void *)((uintptr_t)addr + entry->addend); + return flags; +} + +int probe_access_flags(CPUArchState *env, target_ulong addr, + MMUAccessType access_type, int mmu_idx, + bool nonfault, void **phost, uintptr_t retaddr) +{ + void *host; + int flags; + + flags =3D probe_access_internal(env, addr, 0, access_type, mmu_idx, + nonfault, &host, retaddr); + + /* Handle clean RAM pages. */ + if (unlikely(flags & TLB_NOTDIRTY)) { + uintptr_t index =3D tlb_index(env, mmu_idx, addr); + CPUIOTLBEntry *iotlbentry =3D &env_tlb(env)->d[mmu_idx].iotlb[inde= x]; + + notdirty_write(env_cpu(env), addr, 1, iotlbentry, retaddr); + flags &=3D ~TLB_NOTDIRTY; + } + + return flags; +} + +void *probe_access(CPUArchState *env, target_ulong addr, int size, + MMUAccessType access_type, int mmu_idx, uintptr_t retad= dr) +{ + void *host; + int flags; + + g_assert(-(addr | TARGET_PAGE_MASK) >=3D size); + + flags =3D probe_access_internal(env, addr, size, access_type, mmu_idx, + false, &host, retaddr); + + /* Per the interface, size =3D=3D 0 merely faults the access. */ + if (size =3D=3D 0) { return NULL; } =20 - return (void *)((uintptr_t)addr + entry->addend); + if (unlikely(flags & (TLB_NOTDIRTY | TLB_WATCHPOINT))) { + uintptr_t index =3D tlb_index(env, mmu_idx, addr); + CPUIOTLBEntry *iotlbentry =3D &env_tlb(env)->d[mmu_idx].iotlb[inde= x]; + + /* Handle clean RAM pages. */ + if (flags & TLB_NOTDIRTY) { + notdirty_write(env_cpu(env), addr, 1, iotlbentry, retaddr); + } + + /* Handle watchpoints. */ + if (flags & TLB_WATCHPOINT) { + int wp_access =3D (access_type =3D=3D MMU_DATA_STORE + ? BP_MEM_WRITE : BP_MEM_READ); + cpu_check_watchpoint(env_cpu(env), addr, size, + iotlbentry->attrs, wp_access, retaddr); + } + } + + return host; } =20 +void *tlb_vaddr_to_host(CPUArchState *env, abi_ptr addr, + MMUAccessType access_type, int mmu_idx) +{ + void *host; + int flags; + + flags =3D probe_access_internal(env, addr, 0, access_type, + mmu_idx, true, &host, 0); + + /* No combination of flags are expected by the caller. */ + return flags ? NULL : host; +} =20 #ifdef CONFIG_PLUGIN /* diff --git a/accel/tcg/user-exec.c b/accel/tcg/user-exec.c index 4be78eb9b3..c52dd8a95a 100644 --- a/accel/tcg/user-exec.c +++ b/accel/tcg/user-exec.c @@ -190,13 +190,12 @@ static inline int handle_cpu_signal(uintptr_t pc, sig= info_t *info, g_assert_not_reached(); } =20 -void *probe_access(CPUArchState *env, target_ulong addr, int size, - MMUAccessType access_type, int mmu_idx, uintptr_t retad= dr) +int probe_access_flags(CPUArchState *env, target_ulong addr, + MMUAccessType access_type, int mmu_idx, + bool nonfault, void **phost, uintptr_t retaddr) { int flags; =20 - g_assert(-(addr | TARGET_PAGE_MASK) >=3D size); - switch (access_type) { case MMU_DATA_STORE: flags =3D PAGE_WRITE; @@ -211,15 +210,30 @@ void *probe_access(CPUArchState *env, target_ulong ad= dr, int size, g_assert_not_reached(); } =20 - if (!guest_addr_valid(addr) || page_check_range(addr, size, flags) < 0= ) { - CPUState *cpu =3D env_cpu(env); - CPUClass *cc =3D CPU_GET_CLASS(cpu); - cc->tlb_fill(cpu, addr, size, access_type, MMU_USER_IDX, false, - retaddr); - g_assert_not_reached(); + if (!guest_addr_valid(addr) || page_check_range(addr, 1, flags) < 0) { + if (nonfault) { + *phost =3D NULL; + return TLB_INVALID_MASK; + } else { + CPUState *cpu =3D env_cpu(env); + CPUClass *cc =3D CPU_GET_CLASS(cpu); + cc->tlb_fill(cpu, addr, 0, access_type, MMU_USER_IDX, false, r= etaddr); + g_assert_not_reached(); + } } =20 - return size ? g2h(addr) : NULL; + *phost =3D g2h(addr); + return 0; +} + +void *probe_access(CPUArchState *env, target_ulong addr, int size, + MMUAccessType access_type, int mmu_idx, uintptr_t retad= dr) +{ + void *host; + + g_assert(-(addr | TARGET_PAGE_MASK) >=3D size); + probe_access_flags(env, addr, access_type, mmu_idx, false, &host, reta= ddr); + return host; } =20 #if defined(__i386__) --=20 2.20.1 From nobody Mon Feb 9 21:08:28 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=linaro.org ARC-Seal: i=1; a=rsa-sha256; t=1583909370; cv=none; d=zohomail.com; s=zohoarc; b=VxcC503hBB1q0Hbq/9enWPvrWsuH+ENZS3rTLTMOcTGv9JyZ2/1cDybrQg5RuD+iRT4J7FPUNpgnIAhROdD63QKJFQFAPZVPXs9abdtIjzPAgIqe5ItZVBKZVqdfPN9txRA5sAS8XrLZ4PHnlIT0qf+aO4PXYQX9j/lcN5lSbio= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1583909370; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=B3XDGQrOIHo2Z/Fxm4hqpcuhDa9MTHMM7/GJHSmNf48=; b=JSZX18VpWIl0s3vWCv7C8muv2GC9FpncppKomxRDZtg7Sd5tuYBN0re+FEbjPNBBgAPGiOjFfRk6MYfixN52zzVdLI6GFNYwv+HvVs/OOAt2V1s0J8E2ZtR+n9S/1AZeas3WQUk0gEwmJNmbPOK4qsHGnEtARzWZN5tI5QS76Wk= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1583909370149806.9479286908545; Tue, 10 Mar 2020 23:49:30 -0700 (PDT) Received: from localhost ([::1]:46502 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jBvBH-0004eL-Uj for importer@patchew.org; Wed, 11 Mar 2020 02:49:27 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:59349) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jBv6U-00067p-0u for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:31 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jBv6S-0001nR-9i for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:29 -0400 Received: from mail-pg1-x542.google.com ([2607:f8b0:4864:20::542]:36474) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1jBv6S-0001hI-4P for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:28 -0400 Received: by mail-pg1-x542.google.com with SMTP id c7so645489pgw.3 for ; Tue, 10 Mar 2020 23:44:26 -0700 (PDT) Received: from localhost.localdomain (97-126-123-70.tukw.qwest.net. [97.126.123.70]) by smtp.gmail.com with ESMTPSA id z3sm50137807pfz.155.2020.03.10.23.44.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 10 Mar 2020 23:44:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=B3XDGQrOIHo2Z/Fxm4hqpcuhDa9MTHMM7/GJHSmNf48=; b=GshumYDGLFLuvesKBTQC9q0ilBA4N4T54w6gyQPr9gXCq4SD15BazgVA4Dze02YJEj RdGGW+qMy4eYhuAM0HTSXQYYOVfGekhn8MoWFP3NgI0+Zy0pi69L6o9X5xJJ+qruIMw4 dYH1Q8KSF1a6SwDOgg8y3js+Ib15hBL3oSa3V4M5tqp5C6pD5AIRJuxNY0GSDRpKkukt 4dDaPwC5j+or5ACIrzO0WHc0CQ6kE3Iojl84TVuDMy9u9ffvwHxcipzjndCplSySkJ6u dkGCMiaURON6IW0WxGhs1uTyMSL4kQhHmqUsoLARvngm3bTwVE4Cnn/CGh7RMRP1Zkmw DskA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=B3XDGQrOIHo2Z/Fxm4hqpcuhDa9MTHMM7/GJHSmNf48=; b=p/S70XvQ0F9fNM120i7gJOynfNH79XUsrFMZawi0+3M/htGclLPl6r80S33aR5vOMR W65/7pfSFsyUOrvTuwU3XHOfcm3+wn9/qojXsBUi9P6ZpFrzkslGrSmwJGZnv0Etrqkx HZAZZDYEiXSADCdCsBKxqB1kNGZvqUXAC3uBVEQO7uYIfdx6FNXcuMc+ZxgwFf5UGRww SUidxA4ica+ucIzL1bmoYNX1rLAZAuROcqU/WHTHTXjiy/hWTPCmO8lx8McAwq32pZ+j S9eMyoPDDpKCsdsthbdo8UocyCTe0OrTL5gJunwdDPTATnw1NBIWMhSYz75/CngZG8cg Q3Rg== X-Gm-Message-State: ANhLgQ0V8zPOtg0/cgOnwnRewbK2eOB5RNEFGM4ReSwwhNbjawbaats0 mBTcEUJ78dRqBuCeeJ/aQHcF6txKsXA= X-Google-Smtp-Source: ADFU+vsDE5YtRcI3StH6dTp1gXyvB9ugFozopyK1HGRiWTX0lDd2N6a4UY92uZlvFbLbCQ0wK4CZPA== X-Received: by 2002:aa7:9732:: with SMTP id k18mr1368276pfg.231.1583909065441; Tue, 10 Mar 2020 23:44:25 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH 03/16] exec: Add cpu_probe_watchpoint Date: Tue, 10 Mar 2020 23:44:07 -0700 Message-Id: <20200311064420.30606-4-richard.henderson@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200311064420.30606-1-richard.henderson@linaro.org> References: <20200311064420.30606-1-richard.henderson@linaro.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::542 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, qemu-arm@nongnu.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @linaro.org) Content-Type: text/plain; charset="utf-8" Allow probing of a watchpoint *without* raising an exception. This is of most use for no-fault loads, which should indicate via some architectural means that the load did not occur. Signed-off-by: Richard Henderson --- include/hw/core/cpu.h | 7 +++++++ exec.c | 19 +++++++++++++++++++ 2 files changed, 26 insertions(+) diff --git a/include/hw/core/cpu.h b/include/hw/core/cpu.h index 73e9a869a4..8ec44267a4 100644 --- a/include/hw/core/cpu.h +++ b/include/hw/core/cpu.h @@ -1090,6 +1090,12 @@ static inline void cpu_check_watchpoint(CPUState *cp= u, vaddr addr, vaddr len, { } =20 +static inline bool cpu_probe_watchpoint(CPUState *cpu, vaddr addr, vaddr l= en, + int flags) +{ + return false; +} + static inline int cpu_watchpoint_address_matches(CPUState *cpu, vaddr addr, vaddr len) { @@ -1104,6 +1110,7 @@ void cpu_watchpoint_remove_by_ref(CPUState *cpu, CPUW= atchpoint *watchpoint); void cpu_watchpoint_remove_all(CPUState *cpu, int mask); void cpu_check_watchpoint(CPUState *cpu, vaddr addr, vaddr len, MemTxAttrs attrs, int flags, uintptr_t ra); +bool cpu_probe_watchpoint(CPUState *cpu, vaddr addr, vaddr len, int flags); int cpu_watchpoint_address_matches(CPUState *cpu, vaddr addr, vaddr len); #endif =20 diff --git a/exec.c b/exec.c index 0cc500d53a..2b8f601b9e 100644 --- a/exec.c +++ b/exec.c @@ -2735,6 +2735,25 @@ void cpu_check_watchpoint(CPUState *cpu, vaddr addr,= vaddr len, } } =20 +bool cpu_probe_watchpoint(CPUState *cpu, vaddr addr, vaddr len, int flags) +{ + CPUClass *cc =3D CPU_GET_CLASS(cpu); + CPUWatchpoint *wp; + + assert(tcg_enabled()); + + addr =3D cc->adjust_watchpoint_address(cpu, addr, len); + QTAILQ_FOREACH(wp, &cpu->watchpoints, entry) { + if (watchpoint_address_matches(wp, addr, len) &&=20 + (wp->flags & flags) && + (!(wp->flags & BP_CPU) || + !cc->debug_check_watchpoint(cpu, wp))) { + return true; + } + } + return false; +} + static MemTxResult flatview_read(FlatView *fv, hwaddr addr, MemTxAttrs attrs, void *buf, hwaddr len); static MemTxResult flatview_write(FlatView *fv, hwaddr addr, MemTxAttrs at= trs, --=20 2.20.1 From nobody Mon Feb 9 21:08:28 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=linaro.org ARC-Seal: i=1; a=rsa-sha256; t=1583909310; cv=none; d=zohomail.com; s=zohoarc; b=kUrO1pUr8U6J9Hykg2uzqWPUQ6UhFycB/73yUUDYwX+WBSrmRJQiAENM8QnDcBlmc/i1qVebH5ZjSNW3qdyuXU1Vq2vlF6euHGrVCvyA1ldNaSfVFULHMTSFpovgPVT+aEDbIcgjUvY07UMT9Y5F3QtG8T7rPTAlQxXI7/s2oi0= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1583909310; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=GCUC3ck5APqYH1cRrCKzJ3VHOu3Vf4yYuFQi3uqihJ0=; b=X53hWjVtXdBjKFTwyuNvs3AtGbjIFY6IfjUzv6bw5hAn2cderTm6hHAs4UcAO/Hm/aWt6BpjLSyP//jfgIi7jWQWyqtWHwWoEek7LOWFN+/ZRBRHcrw49AVjNUyMnRICworNI3LzbEXhNuFp1B7iEM6dmGYd+51O3ON+i+mBw0A= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1583909310802592.3287743607971; Tue, 10 Mar 2020 23:48:30 -0700 (PDT) Received: from localhost ([::1]:46444 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jBvAL-0002y2-Lb for importer@patchew.org; Wed, 11 Mar 2020 02:48:29 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:59390) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jBv6W-00069U-0I for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:34 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jBv6T-0001oQ-0j for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:31 -0400 Received: from mail-pf1-x442.google.com ([2607:f8b0:4864:20::442]:44859) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1jBv6S-0001n1-NS for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:28 -0400 Received: by mail-pf1-x442.google.com with SMTP id b72so732418pfb.11 for ; Tue, 10 Mar 2020 23:44:28 -0700 (PDT) Received: from localhost.localdomain (97-126-123-70.tukw.qwest.net. [97.126.123.70]) by smtp.gmail.com with ESMTPSA id z3sm50137807pfz.155.2020.03.10.23.44.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 10 Mar 2020 23:44:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=GCUC3ck5APqYH1cRrCKzJ3VHOu3Vf4yYuFQi3uqihJ0=; b=p/xn6S31kRYlmK7AzUGp7ETctjPY7TgDPfSOK+iyfTxoaQBIUUimHloq9CJHNt7dWS ewo2luyE9ARy8RHIPLT1LUtaATfFnHpbvgwBC3fBpvzPAaOkFPIa90RPShzUPOyBwg0A 9VS9e9M1ImaR74B670ygBT8O8o2IjoX0jtRwvkPzyE+NVSYno4/UnUJpV7mrDAzQh/fa C4tGxtbRraf3BBZ4mSzN69ENcjFxALq3YbKwtY6lU8kysJdtL8he/+lMa6NZtt4E7lBT YHxZsikiiaiTP5fMbHRePhvKt1ThHc5rRkgmBLME0kLrXMozYGv78XD38jI1kfEbUxle yHCw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=GCUC3ck5APqYH1cRrCKzJ3VHOu3Vf4yYuFQi3uqihJ0=; b=PW7RdhGKg6y9AfIptFBZBfMdjotvDEUEQfuaOYHjwDnKeRXgvY9bE1Z2zPbPAQcvoK MO6HPiZEzM3L97Cs8dkPIFqTLV0jlxCdzDlIeshgwlSSuoZg596KfXrQ3MMEQx96YLhs HbPJSHRMVYurJIiMDaXqBwaRoIIFnP1pLG85g+NUKt1KO0KA1ZMf+uUZKGa6DeGH/WQc TSisAHCWdqQrLIHXXD8+bEV9nJF8Fb1FxTEZdx8oX9H34ypN9CHxJmjxNy/r84mtsT75 RC4TDLhU3twpOvMsD3kVxYZnGuHKqVbfgOR1JAx02UKKFR5hMwL3EpQw1hFsjck+SIu0 QWAQ== X-Gm-Message-State: ANhLgQ38CN8sKJ704Z4GnvI9cn7IXGkXKNCvDAECejzUZi77rW4lw/x8 G2hpgmS4vh9AsurSP6HorIIsoa6tykU= X-Google-Smtp-Source: ADFU+vsI5qUmVfDSKLxF+k3tZheoza7khF8b2arlNHmhdf05++Pu4LdVGYDeZSdYJniJKQg90MyGlQ== X-Received: by 2002:a62:2587:: with SMTP id l129mr474534pfl.180.1583909066713; Tue, 10 Mar 2020 23:44:26 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH 04/16] target/arm: Use cpu_*_data_ra for sve_ldst_tlb_fn Date: Tue, 10 Mar 2020 23:44:08 -0700 Message-Id: <20200311064420.30606-5-richard.henderson@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200311064420.30606-1-richard.henderson@linaro.org> References: <20200311064420.30606-1-richard.henderson@linaro.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::442 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, qemu-arm@nongnu.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @linaro.org) Content-Type: text/plain; charset="utf-8" Use the "normal" memory access functions, rather than the softmmu internal helper functions directly. Since fb901c905dc3, cpu_mem_index is now a simple extract from env->hflags and not a large computation. Which means that it's now more work to pass around this value than it is to recompute it. This only adjusts the primitives, and does not clean up all of the uses within sve_helper.c. Signed-off-by: Richard Henderson --- target/arm/sve_helper.c | 241 ++++++++++++++++++---------------------- 1 file changed, 107 insertions(+), 134 deletions(-) diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index fdfa652094..f8b310a16b 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -3991,12 +3991,15 @@ typedef intptr_t sve_ld1_host_fn(void *vd, void *vg= , void *host, * Load one element into @vd + @reg_off from (@env, @vaddr, @ra). * The controlling predicate is known to be true. */ -typedef void sve_ld1_tlb_fn(CPUARMState *env, void *vd, intptr_t reg_off, - target_ulong vaddr, TCGMemOpIdx oi, uintptr_t = ra); -typedef sve_ld1_tlb_fn sve_st1_tlb_fn; +typedef void sve_ldst1_tlb_fn(CPUARMState *env, void *vd, intptr_t reg_off, + target_ulong vaddr, uintptr_t retaddr); =20 /* * Generate the above primitives. + * + * For *_tlb, this uses the cpu_*_data_ra helpers. There are not + * endian-specific versions of these, so we must handle endianness + * locally. */ =20 #define DO_LD_HOST(NAME, H, TYPEE, TYPEM, HOST) \ @@ -4016,27 +4019,25 @@ static intptr_t sve_##NAME##_host(void *vd, void *v= g, void *host, \ return mem_off; = \ } =20 -#ifdef CONFIG_SOFTMMU -#define DO_LD_TLB(NAME, H, TYPEE, TYPEM, HOST, MOEND, TLB) \ +#define DO_LD_TLB(NAME, H, TYPEE, TYPEM, BSWAP, TLB) \ static void sve_##NAME##_tlb(CPUARMState *env, void *vd, intptr_t reg_off,= \ - target_ulong addr, TCGMemOpIdx oi, uintptr_t = ra) \ + target_ulong addr, uintptr_t ra) = \ { = \ - TYPEM val =3D TLB(env, addr, oi, ra); = \ + TYPEM val =3D BSWAP(TLB(env, addr, ra)); = \ *(TYPEE *)(vd + H(reg_off)) =3D val; = \ } -#else -#define DO_LD_TLB(NAME, H, TYPEE, TYPEM, HOST, MOEND, TLB) = \ + +#define DO_ST_TLB(NAME, H, TYPEE, TYPEM, BSWAP, TLB) \ static void sve_##NAME##_tlb(CPUARMState *env, void *vd, intptr_t reg_off,= \ - target_ulong addr, TCGMemOpIdx oi, uintptr_t = ra) \ + target_ulong addr, uintptr_t ra) = \ { = \ - TYPEM val =3D HOST(g2h(addr)); = \ - *(TYPEE *)(vd + H(reg_off)) =3D val; = \ + TYPEM val =3D *(TYPEE *)(vd + H(reg_off)); = \ + TLB(env, addr, BSWAP(val), ra); = \ } -#endif =20 #define DO_LD_PRIM_1(NAME, H, TE, TM) \ DO_LD_HOST(NAME, H, TE, TM, ldub_p) \ - DO_LD_TLB(NAME, H, TE, TM, ldub_p, 0, helper_ret_ldub_mmu) + DO_LD_TLB(NAME, H, TE, TM, , cpu_ldub_data_ra) =20 DO_LD_PRIM_1(ld1bb, H1, uint8_t, uint8_t) DO_LD_PRIM_1(ld1bhu, H1_2, uint16_t, uint8_t) @@ -4046,39 +4047,67 @@ DO_LD_PRIM_1(ld1bss, H1_4, uint32_t, int8_t) DO_LD_PRIM_1(ld1bdu, , uint64_t, uint8_t) DO_LD_PRIM_1(ld1bds, , uint64_t, int8_t) =20 -#define DO_LD_PRIM_2(NAME, end, MOEND, H, TE, TM, PH, PT) \ - DO_LD_HOST(NAME##_##end, H, TE, TM, PH##_##end##_p) \ - DO_LD_TLB(NAME##_##end, H, TE, TM, PH##_##end##_p, \ - MOEND, helper_##end##_##PT##_mmu) +#define DO_ST_PRIM_1(NAME, H, TE, TM) \ + DO_ST_TLB(st1##NAME, H, TE, TM, , cpu_stb_data_ra) =20 -DO_LD_PRIM_2(ld1hh, le, MO_LE, H1_2, uint16_t, uint16_t, lduw, lduw) -DO_LD_PRIM_2(ld1hsu, le, MO_LE, H1_4, uint32_t, uint16_t, lduw, lduw) -DO_LD_PRIM_2(ld1hss, le, MO_LE, H1_4, uint32_t, int16_t, lduw, lduw) -DO_LD_PRIM_2(ld1hdu, le, MO_LE, , uint64_t, uint16_t, lduw, lduw) -DO_LD_PRIM_2(ld1hds, le, MO_LE, , uint64_t, int16_t, lduw, lduw) +DO_ST_PRIM_1(bb, H1, uint8_t, uint8_t) +DO_ST_PRIM_1(bh, H1_2, uint16_t, uint8_t) +DO_ST_PRIM_1(bs, H1_4, uint32_t, uint8_t) +DO_ST_PRIM_1(bd, , uint64_t, uint8_t) =20 -DO_LD_PRIM_2(ld1ss, le, MO_LE, H1_4, uint32_t, uint32_t, ldl, ldul) -DO_LD_PRIM_2(ld1sdu, le, MO_LE, , uint64_t, uint32_t, ldl, ldul) -DO_LD_PRIM_2(ld1sds, le, MO_LE, , uint64_t, int32_t, ldl, ldul) +#define DO_LD_PRIM_2(NAME, H, TE, TM, BSWAP, LD) \ + DO_LD_HOST(ld1##NAME##_be, H, TE, TM, LD##_be_p) \ + DO_LD_HOST(ld1##NAME##_le, H, TE, TM, LD##_le_p) \ + DO_LD_TLB(ld1##NAME##_be, H, TE, TM, be_##BSWAP, cpu_##LD##_data_ra) \ + DO_LD_TLB(ld1##NAME##_le, H, TE, TM, le_##BSWAP, cpu_##LD##_data_ra) =20 -DO_LD_PRIM_2(ld1dd, le, MO_LE, , uint64_t, uint64_t, ldq, ldq) +#define DO_ST_PRIM_2(NAME, H, TE, TM, BSWAP, ST) \ + DO_ST_TLB(st1##NAME##_be, H, TE, TM, be_##BSWAP, cpu_##ST##_data_ra) \ + DO_ST_TLB(st1##NAME##_le, H, TE, TM, le_##BSWAP, cpu_##ST##_data_ra) =20 -DO_LD_PRIM_2(ld1hh, be, MO_BE, H1_2, uint16_t, uint16_t, lduw, lduw) -DO_LD_PRIM_2(ld1hsu, be, MO_BE, H1_4, uint32_t, uint16_t, lduw, lduw) -DO_LD_PRIM_2(ld1hss, be, MO_BE, H1_4, uint32_t, int16_t, lduw, lduw) -DO_LD_PRIM_2(ld1hdu, be, MO_BE, , uint64_t, uint16_t, lduw, lduw) -DO_LD_PRIM_2(ld1hds, be, MO_BE, , uint64_t, int16_t, lduw, lduw) +#ifdef TARGET_WORDS_BIGENDIAN +# define be_bswap16(x) ((uint16_t)(x)) +# define be_bswap32(x) ((uint32_t)(x)) +# define be_bswap64(x) ((uint64_t)(x)) +# define le_bswap16(x) bswap16(x) +# define le_bswap32(x) bswap32(x) +# define le_bswap64(x) bswap64(x) +#else +# define be_bswap16(x) bswap16(x) +# define be_bswap32(x) bswap32(x) +# define be_bswap64(x) bswap64(x) +# define le_bswap16(x) ((uint16_t)(x)) +# define le_bswap32(x) ((uint32_t)(x)) +# define le_bswap64(x) ((uint64_t)(x)) +#endif =20 -DO_LD_PRIM_2(ld1ss, be, MO_BE, H1_4, uint32_t, uint32_t, ldl, ldul) -DO_LD_PRIM_2(ld1sdu, be, MO_BE, , uint64_t, uint32_t, ldl, ldul) -DO_LD_PRIM_2(ld1sds, be, MO_BE, , uint64_t, int32_t, ldl, ldul) +DO_LD_PRIM_2(hh, H1_2, uint16_t, uint16_t, bswap16, lduw) +DO_LD_PRIM_2(hsu, H1_4, uint32_t, uint16_t, bswap16, lduw) +DO_LD_PRIM_2(hss, H1_4, uint32_t, int16_t, bswap16, lduw) +DO_LD_PRIM_2(hdu, , uint64_t, uint16_t, bswap16, lduw) +DO_LD_PRIM_2(hds, , uint64_t, int16_t, bswap16, lduw) =20 -DO_LD_PRIM_2(ld1dd, be, MO_BE, , uint64_t, uint64_t, ldq, ldq) +DO_ST_PRIM_2(hh, H1_2, uint16_t, uint16_t, bswap16, stw) +DO_ST_PRIM_2(hs, H1_4, uint32_t, uint16_t, bswap16, stw) +DO_ST_PRIM_2(hd, , uint64_t, uint16_t, bswap16, stw) + +DO_LD_PRIM_2(ss, H1_4, uint32_t, uint32_t, bswap32, ldl) +DO_LD_PRIM_2(sdu, , uint64_t, uint32_t, bswap32, ldl) +DO_LD_PRIM_2(sds, , uint64_t, int32_t, bswap32, ldl) + +DO_ST_PRIM_2(ss, H1_4, uint32_t, uint32_t, bswap32, stl) +DO_ST_PRIM_2(sd, , uint64_t, uint32_t, bswap32, stl) + +DO_LD_PRIM_2(dd, , uint64_t, uint64_t, bswap64, ldq) +DO_ST_PRIM_2(dd, , uint64_t, uint64_t, bswap64, stq) =20 #undef DO_LD_TLB +#undef DO_ST_TLB #undef DO_LD_HOST #undef DO_LD_PRIM_1 +#undef DO_ST_PRIM_1 #undef DO_LD_PRIM_2 +#undef DO_ST_PRIM_2 =20 /* * Skip through a sequence of inactive elements in the guarding predicate = @vg, @@ -4152,7 +4181,7 @@ static void sve_ld1_r(CPUARMState *env, void *vg, con= st target_ulong addr, uint32_t desc, const uintptr_t retaddr, const int esz, const int msz, sve_ld1_host_fn *host_fn, - sve_ld1_tlb_fn *tlb_fn) + sve_ldst1_tlb_fn *tlb_fn) { const TCGMemOpIdx oi =3D extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHI= FT); const int mmu_idx =3D get_mmuidx(oi); @@ -4234,7 +4263,7 @@ static void sve_ld1_r(CPUARMState *env, void *vg, con= st target_ulong addr, * on I/O memory, it may succeed but not bring in the TLB entry. * But even then we have still made forward progress. */ - tlb_fn(env, &scratch, reg_off, addr + mem_off, oi, retaddr); + tlb_fn(env, &scratch, reg_off, addr + mem_off, retaddr); reg_off +=3D 1 << esz; } #endif @@ -4293,9 +4322,8 @@ DO_LD1_2(ld1dd, 3, 3) */ static void sve_ld2_r(CPUARMState *env, void *vg, target_ulong addr, uint32_t desc, int size, uintptr_t ra, - sve_ld1_tlb_fn *tlb_fn) + sve_ldst1_tlb_fn *tlb_fn) { - const TCGMemOpIdx oi =3D extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHI= FT); const unsigned rd =3D extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT= , 5); intptr_t i, oprsz =3D simd_oprsz(desc); ARMVectorReg scratch[2] =3D { }; @@ -4305,8 +4333,8 @@ static void sve_ld2_r(CPUARMState *env, void *vg, tar= get_ulong addr, uint16_t pg =3D *(uint16_t *)(vg + H1_2(i >> 3)); do { if (pg & 1) { - tlb_fn(env, &scratch[0], i, addr, oi, ra); - tlb_fn(env, &scratch[1], i, addr + size, oi, ra); + tlb_fn(env, &scratch[0], i, addr, ra); + tlb_fn(env, &scratch[1], i, addr + size, ra); } i +=3D size, pg >>=3D size; addr +=3D 2 * size; @@ -4321,9 +4349,8 @@ static void sve_ld2_r(CPUARMState *env, void *vg, tar= get_ulong addr, =20 static void sve_ld3_r(CPUARMState *env, void *vg, target_ulong addr, uint32_t desc, int size, uintptr_t ra, - sve_ld1_tlb_fn *tlb_fn) + sve_ldst1_tlb_fn *tlb_fn) { - const TCGMemOpIdx oi =3D extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHI= FT); const unsigned rd =3D extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT= , 5); intptr_t i, oprsz =3D simd_oprsz(desc); ARMVectorReg scratch[3] =3D { }; @@ -4333,9 +4360,9 @@ static void sve_ld3_r(CPUARMState *env, void *vg, tar= get_ulong addr, uint16_t pg =3D *(uint16_t *)(vg + H1_2(i >> 3)); do { if (pg & 1) { - tlb_fn(env, &scratch[0], i, addr, oi, ra); - tlb_fn(env, &scratch[1], i, addr + size, oi, ra); - tlb_fn(env, &scratch[2], i, addr + 2 * size, oi, ra); + tlb_fn(env, &scratch[0], i, addr, ra); + tlb_fn(env, &scratch[1], i, addr + size, ra); + tlb_fn(env, &scratch[2], i, addr + 2 * size, ra); } i +=3D size, pg >>=3D size; addr +=3D 3 * size; @@ -4351,9 +4378,8 @@ static void sve_ld3_r(CPUARMState *env, void *vg, tar= get_ulong addr, =20 static void sve_ld4_r(CPUARMState *env, void *vg, target_ulong addr, uint32_t desc, int size, uintptr_t ra, - sve_ld1_tlb_fn *tlb_fn) + sve_ldst1_tlb_fn *tlb_fn) { - const TCGMemOpIdx oi =3D extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHI= FT); const unsigned rd =3D extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT= , 5); intptr_t i, oprsz =3D simd_oprsz(desc); ARMVectorReg scratch[4] =3D { }; @@ -4363,10 +4389,10 @@ static void sve_ld4_r(CPUARMState *env, void *vg, t= arget_ulong addr, uint16_t pg =3D *(uint16_t *)(vg + H1_2(i >> 3)); do { if (pg & 1) { - tlb_fn(env, &scratch[0], i, addr, oi, ra); - tlb_fn(env, &scratch[1], i, addr + size, oi, ra); - tlb_fn(env, &scratch[2], i, addr + 2 * size, oi, ra); - tlb_fn(env, &scratch[3], i, addr + 3 * size, oi, ra); + tlb_fn(env, &scratch[0], i, addr, ra); + tlb_fn(env, &scratch[1], i, addr + size, ra); + tlb_fn(env, &scratch[2], i, addr + 2 * size, ra); + tlb_fn(env, &scratch[3], i, addr + 3 * size, ra); } i +=3D size, pg >>=3D size; addr +=3D 4 * size; @@ -4459,7 +4485,7 @@ static void sve_ldff1_r(CPUARMState *env, void *vg, c= onst target_ulong addr, uint32_t desc, const uintptr_t retaddr, const int esz, const int msz, sve_ld1_host_fn *host_fn, - sve_ld1_tlb_fn *tlb_fn) + sve_ldst1_tlb_fn *tlb_fn) { const TCGMemOpIdx oi =3D extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHI= FT); const int mmu_idx =3D get_mmuidx(oi); @@ -4519,7 +4545,7 @@ static void sve_ldff1_r(CPUARMState *env, void *vg, c= onst target_ulong addr, * Perform one normal read, which will fault or not. * But it is likely to bring the page into the tlb. */ - tlb_fn(env, vd, reg_off, addr + mem_off, oi, retaddr); + tlb_fn(env, vd, reg_off, addr + mem_off, retaddr); =20 /* After any fault, zero any leading predicated false elts. */ swap_memzero(vd, reg_off); @@ -4671,60 +4697,14 @@ DO_LDFF1_LDNF1_2(dd, 3, 3) #undef DO_LDFF1_LDNF1_1 #undef DO_LDFF1_LDNF1_2 =20 -/* - * Store contiguous data, protected by a governing predicate. - */ - -#ifdef CONFIG_SOFTMMU -#define DO_ST_TLB(NAME, H, TYPEM, HOST, MOEND, TLB) \ -static void sve_##NAME##_tlb(CPUARMState *env, void *vd, intptr_t reg_off,= \ - target_ulong addr, TCGMemOpIdx oi, uintptr_t = ra) \ -{ = \ - TLB(env, addr, *(TYPEM *)(vd + H(reg_off)), oi, ra); = \ -} -#else -#define DO_ST_TLB(NAME, H, TYPEM, HOST, MOEND, TLB) \ -static void sve_##NAME##_tlb(CPUARMState *env, void *vd, intptr_t reg_off,= \ - target_ulong addr, TCGMemOpIdx oi, uintptr_t = ra) \ -{ = \ - HOST(g2h(addr), *(TYPEM *)(vd + H(reg_off))); = \ -} -#endif - -DO_ST_TLB(st1bb, H1, uint8_t, stb_p, 0, helper_ret_stb_mmu) -DO_ST_TLB(st1bh, H1_2, uint16_t, stb_p, 0, helper_ret_stb_mmu) -DO_ST_TLB(st1bs, H1_4, uint32_t, stb_p, 0, helper_ret_stb_mmu) -DO_ST_TLB(st1bd, , uint64_t, stb_p, 0, helper_ret_stb_mmu) - -DO_ST_TLB(st1hh_le, H1_2, uint16_t, stw_le_p, MO_LE, helper_le_stw_mmu) -DO_ST_TLB(st1hs_le, H1_4, uint32_t, stw_le_p, MO_LE, helper_le_stw_mmu) -DO_ST_TLB(st1hd_le, , uint64_t, stw_le_p, MO_LE, helper_le_stw_mmu) - -DO_ST_TLB(st1ss_le, H1_4, uint32_t, stl_le_p, MO_LE, helper_le_stl_mmu) -DO_ST_TLB(st1sd_le, , uint64_t, stl_le_p, MO_LE, helper_le_stl_mmu) - -DO_ST_TLB(st1dd_le, , uint64_t, stq_le_p, MO_LE, helper_le_stq_mmu) - -DO_ST_TLB(st1hh_be, H1_2, uint16_t, stw_be_p, MO_BE, helper_be_stw_mmu) -DO_ST_TLB(st1hs_be, H1_4, uint32_t, stw_be_p, MO_BE, helper_be_stw_mmu) -DO_ST_TLB(st1hd_be, , uint64_t, stw_be_p, MO_BE, helper_be_stw_mmu) - -DO_ST_TLB(st1ss_be, H1_4, uint32_t, stl_be_p, MO_BE, helper_be_stl_mmu) -DO_ST_TLB(st1sd_be, , uint64_t, stl_be_p, MO_BE, helper_be_stl_mmu) - -DO_ST_TLB(st1dd_be, , uint64_t, stq_be_p, MO_BE, helper_be_stq_mmu) - -#undef DO_ST_TLB - /* * Common helpers for all contiguous 1,2,3,4-register predicated stores. */ static void sve_st1_r(CPUARMState *env, void *vg, target_ulong addr, uint32_t desc, const uintptr_t ra, const int esize, const int msize, - sve_st1_tlb_fn *tlb_fn) + sve_ldst1_tlb_fn *tlb_fn) { - const TCGMemOpIdx oi =3D extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHI= FT); const unsigned rd =3D extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT= , 5); intptr_t i, oprsz =3D simd_oprsz(desc); void *vd =3D &env->vfp.zregs[rd]; @@ -4734,7 +4714,7 @@ static void sve_st1_r(CPUARMState *env, void *vg, tar= get_ulong addr, uint16_t pg =3D *(uint16_t *)(vg + H1_2(i >> 3)); do { if (pg & 1) { - tlb_fn(env, vd, i, addr, oi, ra); + tlb_fn(env, vd, i, addr, ra); } i +=3D esize, pg >>=3D esize; addr +=3D msize; @@ -4746,9 +4726,8 @@ static void sve_st1_r(CPUARMState *env, void *vg, tar= get_ulong addr, static void sve_st2_r(CPUARMState *env, void *vg, target_ulong addr, uint32_t desc, const uintptr_t ra, const int esize, const int msize, - sve_st1_tlb_fn *tlb_fn) + sve_ldst1_tlb_fn *tlb_fn) { - const TCGMemOpIdx oi =3D extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHI= FT); const unsigned rd =3D extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT= , 5); intptr_t i, oprsz =3D simd_oprsz(desc); void *d1 =3D &env->vfp.zregs[rd]; @@ -4759,8 +4738,8 @@ static void sve_st2_r(CPUARMState *env, void *vg, tar= get_ulong addr, uint16_t pg =3D *(uint16_t *)(vg + H1_2(i >> 3)); do { if (pg & 1) { - tlb_fn(env, d1, i, addr, oi, ra); - tlb_fn(env, d2, i, addr + msize, oi, ra); + tlb_fn(env, d1, i, addr, ra); + tlb_fn(env, d2, i, addr + msize, ra); } i +=3D esize, pg >>=3D esize; addr +=3D 2 * msize; @@ -4772,9 +4751,8 @@ static void sve_st2_r(CPUARMState *env, void *vg, tar= get_ulong addr, static void sve_st3_r(CPUARMState *env, void *vg, target_ulong addr, uint32_t desc, const uintptr_t ra, const int esize, const int msize, - sve_st1_tlb_fn *tlb_fn) + sve_ldst1_tlb_fn *tlb_fn) { - const TCGMemOpIdx oi =3D extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHI= FT); const unsigned rd =3D extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT= , 5); intptr_t i, oprsz =3D simd_oprsz(desc); void *d1 =3D &env->vfp.zregs[rd]; @@ -4786,9 +4764,9 @@ static void sve_st3_r(CPUARMState *env, void *vg, tar= get_ulong addr, uint16_t pg =3D *(uint16_t *)(vg + H1_2(i >> 3)); do { if (pg & 1) { - tlb_fn(env, d1, i, addr, oi, ra); - tlb_fn(env, d2, i, addr + msize, oi, ra); - tlb_fn(env, d3, i, addr + 2 * msize, oi, ra); + tlb_fn(env, d1, i, addr, ra); + tlb_fn(env, d2, i, addr + msize, ra); + tlb_fn(env, d3, i, addr + 2 * msize, ra); } i +=3D esize, pg >>=3D esize; addr +=3D 3 * msize; @@ -4800,9 +4778,8 @@ static void sve_st3_r(CPUARMState *env, void *vg, tar= get_ulong addr, static void sve_st4_r(CPUARMState *env, void *vg, target_ulong addr, uint32_t desc, const uintptr_t ra, const int esize, const int msize, - sve_st1_tlb_fn *tlb_fn) + sve_ldst1_tlb_fn *tlb_fn) { - const TCGMemOpIdx oi =3D extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHI= FT); const unsigned rd =3D extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT= , 5); intptr_t i, oprsz =3D simd_oprsz(desc); void *d1 =3D &env->vfp.zregs[rd]; @@ -4815,10 +4792,10 @@ static void sve_st4_r(CPUARMState *env, void *vg, t= arget_ulong addr, uint16_t pg =3D *(uint16_t *)(vg + H1_2(i >> 3)); do { if (pg & 1) { - tlb_fn(env, d1, i, addr, oi, ra); - tlb_fn(env, d2, i, addr + msize, oi, ra); - tlb_fn(env, d3, i, addr + 2 * msize, oi, ra); - tlb_fn(env, d4, i, addr + 3 * msize, oi, ra); + tlb_fn(env, d1, i, addr, ra); + tlb_fn(env, d2, i, addr + msize, ra); + tlb_fn(env, d3, i, addr + 2 * msize, ra); + tlb_fn(env, d4, i, addr + 3 * msize, ra); } i +=3D esize, pg >>=3D esize; addr +=3D 4 * msize; @@ -4914,9 +4891,8 @@ static target_ulong off_zd_d(void *reg, intptr_t reg_= ofs) =20 static void sve_ld1_zs(CPUARMState *env, void *vd, void *vg, void *vm, target_ulong base, uint32_t desc, uintptr_t ra, - zreg_off_fn *off_fn, sve_ld1_tlb_fn *tlb_fn) + zreg_off_fn *off_fn, sve_ldst1_tlb_fn *tlb_fn) { - const TCGMemOpIdx oi =3D extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHI= FT); const int scale =3D extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, = 2); intptr_t i, oprsz =3D simd_oprsz(desc); ARMVectorReg scratch =3D { }; @@ -4927,7 +4903,7 @@ static void sve_ld1_zs(CPUARMState *env, void *vd, vo= id *vg, void *vm, do { if (likely(pg & 1)) { target_ulong off =3D off_fn(vm, i); - tlb_fn(env, &scratch, i, base + (off << scale), oi, ra); + tlb_fn(env, &scratch, i, base + (off << scale), ra); } i +=3D 4, pg >>=3D 4; } while (i & 15); @@ -4940,9 +4916,8 @@ static void sve_ld1_zs(CPUARMState *env, void *vd, vo= id *vg, void *vm, =20 static void sve_ld1_zd(CPUARMState *env, void *vd, void *vg, void *vm, target_ulong base, uint32_t desc, uintptr_t ra, - zreg_off_fn *off_fn, sve_ld1_tlb_fn *tlb_fn) + zreg_off_fn *off_fn, sve_ldst1_tlb_fn *tlb_fn) { - const TCGMemOpIdx oi =3D extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHI= FT); const int scale =3D extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, = 2); intptr_t i, oprsz =3D simd_oprsz(desc) / 8; ARMVectorReg scratch =3D { }; @@ -4952,7 +4927,7 @@ static void sve_ld1_zd(CPUARMState *env, void *vd, vo= id *vg, void *vm, uint8_t pg =3D *(uint8_t *)(vg + H1(i)); if (likely(pg & 1)) { target_ulong off =3D off_fn(vm, i * 8); - tlb_fn(env, &scratch, i * 8, base + (off << scale), oi, ra); + tlb_fn(env, &scratch, i * 8, base + (off << scale), ra); } } clear_helper_retaddr(); @@ -5114,7 +5089,7 @@ DO_LD_NF(dd_be, , uint64_t, uint64_t, ldq_be_p) */ static inline void sve_ldff1_zs(CPUARMState *env, void *vd, void *vg, void= *vm, target_ulong base, uint32_t desc, uintptr_= t ra, - zreg_off_fn *off_fn, sve_ld1_tlb_fn *tlb_f= n, + zreg_off_fn *off_fn, sve_ldst1_tlb_fn *tlb= _fn, sve_ld1_nf_fn *nonfault_fn) { const TCGMemOpIdx oi =3D extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHI= FT); @@ -5130,7 +5105,7 @@ static inline void sve_ldff1_zs(CPUARMState *env, voi= d *vd, void *vg, void *vm, set_helper_retaddr(ra); addr =3D off_fn(vm, reg_off); addr =3D base + (addr << scale); - tlb_fn(env, vd, reg_off, addr, oi, ra); + tlb_fn(env, vd, reg_off, addr, ra); =20 /* The rest of the reads will be non-faulting. */ clear_helper_retaddr(); @@ -5156,7 +5131,7 @@ static inline void sve_ldff1_zs(CPUARMState *env, voi= d *vd, void *vg, void *vm, =20 static inline void sve_ldff1_zd(CPUARMState *env, void *vd, void *vg, void= *vm, target_ulong base, uint32_t desc, uintptr_= t ra, - zreg_off_fn *off_fn, sve_ld1_tlb_fn *tlb_f= n, + zreg_off_fn *off_fn, sve_ldst1_tlb_fn *tlb= _fn, sve_ld1_nf_fn *nonfault_fn) { const TCGMemOpIdx oi =3D extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHI= FT); @@ -5172,7 +5147,7 @@ static inline void sve_ldff1_zd(CPUARMState *env, voi= d *vd, void *vg, void *vm, set_helper_retaddr(ra); addr =3D off_fn(vm, reg_off); addr =3D base + (addr << scale); - tlb_fn(env, vd, reg_off, addr, oi, ra); + tlb_fn(env, vd, reg_off, addr, ra); =20 /* The rest of the reads will be non-faulting. */ clear_helper_retaddr(); @@ -5282,9 +5257,8 @@ DO_LDFF1_ZPZ_D(dd_be, zd) =20 static void sve_st1_zs(CPUARMState *env, void *vd, void *vg, void *vm, target_ulong base, uint32_t desc, uintptr_t ra, - zreg_off_fn *off_fn, sve_ld1_tlb_fn *tlb_fn) + zreg_off_fn *off_fn, sve_ldst1_tlb_fn *tlb_fn) { - const TCGMemOpIdx oi =3D extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHI= FT); const int scale =3D extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, = 2); intptr_t i, oprsz =3D simd_oprsz(desc); =20 @@ -5294,7 +5268,7 @@ static void sve_st1_zs(CPUARMState *env, void *vd, vo= id *vg, void *vm, do { if (likely(pg & 1)) { target_ulong off =3D off_fn(vm, i); - tlb_fn(env, vd, i, base + (off << scale), oi, ra); + tlb_fn(env, vd, i, base + (off << scale), ra); } i +=3D 4, pg >>=3D 4; } while (i & 15); @@ -5304,9 +5278,8 @@ static void sve_st1_zs(CPUARMState *env, void *vd, vo= id *vg, void *vm, =20 static void sve_st1_zd(CPUARMState *env, void *vd, void *vg, void *vm, target_ulong base, uint32_t desc, uintptr_t ra, - zreg_off_fn *off_fn, sve_ld1_tlb_fn *tlb_fn) + zreg_off_fn *off_fn, sve_ldst1_tlb_fn *tlb_fn) { - const TCGMemOpIdx oi =3D extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHI= FT); const int scale =3D extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, = 2); intptr_t i, oprsz =3D simd_oprsz(desc) / 8; =20 @@ -5315,7 +5288,7 @@ static void sve_st1_zd(CPUARMState *env, void *vd, vo= id *vg, void *vm, uint8_t pg =3D *(uint8_t *)(vg + H1(i)); if (likely(pg & 1)) { target_ulong off =3D off_fn(vm, i * 8); - tlb_fn(env, vd, i * 8, base + (off << scale), oi, ra); + tlb_fn(env, vd, i * 8, base + (off << scale), ra); } } clear_helper_retaddr(); --=20 2.20.1 From nobody Mon Feb 9 21:08:28 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=linaro.org ARC-Seal: i=1; a=rsa-sha256; t=1583909316; cv=none; d=zohomail.com; s=zohoarc; b=iiDgxO06kTQstZsFl8Sjy1eV4HauVRyFEDpkhq4Q6fIYv5uJboqMUg2Q3qw/R6l5jV0rKu7X7YH3BfzavPSnnz7AnBEbsW7ThxCpoDBKv+6vggQRZKlT5r0Oz96jrvEZpH6KZnC2H4kdCyUHXQoY/QBHpYsV+efkDDeGaItV0SE= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1583909316; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=x2sQUQOTqOZ29kmZWF8+Fedva6/mV+pCTu/MNQTzpjU=; b=PpXJYnAThS6uW9lXYQP7JLPaHSqFzOivX/sfit8EmrtugfY2bQUJU7t6rdZlVWsxQMOYtbQl6Ovf2IP2rVqWVvXFYZacBrbrsbUexjcS+Simon4vupYR1b2/OKrnnCf1GtWq5bMoF35grGyg9WgkgllqnXcQAxB6eYwJKbwCgdA= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1583909316475520.5927349181193; Tue, 10 Mar 2020 23:48:36 -0700 (PDT) Received: from localhost ([::1]:46452 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jBvAR-00036q-Co for importer@patchew.org; Wed, 11 Mar 2020 02:48:35 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:59417) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jBv6X-0006Az-S2 for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:35 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jBv6V-0001tI-UR for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:33 -0400 Received: from mail-pf1-x444.google.com ([2607:f8b0:4864:20::444]:34535) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1jBv6T-0001oH-UW for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:31 -0400 Received: by mail-pf1-x444.google.com with SMTP id 23so755088pfj.1 for ; Tue, 10 Mar 2020 23:44:29 -0700 (PDT) Received: from localhost.localdomain (97-126-123-70.tukw.qwest.net. [97.126.123.70]) by smtp.gmail.com with ESMTPSA id z3sm50137807pfz.155.2020.03.10.23.44.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 10 Mar 2020 23:44:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=x2sQUQOTqOZ29kmZWF8+Fedva6/mV+pCTu/MNQTzpjU=; b=jUs5ATgiOY2goWQDP8wHHTO4JnkBdWW94XVQdx35YRib2w1wDldTvX3S19jJvkQm8M Mp6lGiODavUH6VzuzQ8Ceme2SSze9je3T9syMM6QDcYpUMiYnCkI6Xw/xAg608PXky7N /6maAxk4PCIsgC/PfiqPCSai43ih5aqKJjJNI6fyZjfHlub0Y0XwXDVhEUoiRVhWs4Oe JmIaJggZlD4ceq8Un7bWOMrI1yJGTWuSBaOTcD5DSqhuyqWdfaeaxchUsDP4kG2xn55f TD65DC4mdajR/VmPc+s3sxUeVL+CMCwxX+lM+q0k5A0rOI8OKwNjsHRCWdNG9iHciogm 2beQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=x2sQUQOTqOZ29kmZWF8+Fedva6/mV+pCTu/MNQTzpjU=; b=UYqOFK41atvtWmrWHlSkVD0BpvrsPcp/DsGjeqEkoWufk/H1hOZY+r98d0danQKHCC eaQHX9b2ZnYJJ57r1B+FKpvnBTLqcuZmM/5fYSERp4W7a8C9t93I/fM7fmvqRjnSM8Ac eGj9KMMUK29j/RC29rRDlA023q3KW32kcCcfeaPCWIwa9WNVghaBCw3ECENrfdQVxy8x 1+l7VWxgBUexUxj1AbIFiBWKhU5Wn39BtM4moGnu6Nrj4p2Ez/npPeBp++ErTcEtUGjw or/3FESa1EbT+i8RoEigufZ+mw5H2Axf7Z04if3Pj0H4ujrJlJTL79OXqt4ghvW5dJek mUpA== X-Gm-Message-State: ANhLgQ3e2tWsGG4eFEkZCFzLJzePOuZTm3/7/e5VZWKZpO6ShsSEy1RQ jtMkJz8JfnHuDQvmklkhhRW+rLKzI/8= X-Google-Smtp-Source: ADFU+vv0LXby0UachfMLs3C7sOaWckcZTnpCj+7kFVYM8BDCWH11JT1ql9Q1UGCCcqZkDmvxJmhB7Q== X-Received: by 2002:a63:257:: with SMTP id 84mr1436504pgc.304.1583909067912; Tue, 10 Mar 2020 23:44:27 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH 05/16] target/arm: Drop manual handling of set/clear_helper_retaddr Date: Tue, 10 Mar 2020 23:44:09 -0700 Message-Id: <20200311064420.30606-6-richard.henderson@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200311064420.30606-1-richard.henderson@linaro.org> References: <20200311064420.30606-1-richard.henderson@linaro.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::444 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, qemu-arm@nongnu.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @linaro.org) Content-Type: text/plain; charset="utf-8" Since we converted back to cpu_*_data_ra, we do not need to do this ourselves. Signed-off-by: Richard Henderson Reviewed-by: Peter Maydell --- target/arm/sve_helper.c | 38 -------------------------------------- 1 file changed, 38 deletions(-) diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index f8b310a16b..8b470991db 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -4155,12 +4155,6 @@ static intptr_t max_for_page(target_ulong base, intp= tr_t mem_off, return MIN(split, mem_max - mem_off) + mem_off; } =20 -#ifndef CONFIG_USER_ONLY -/* These are normally defined only for CONFIG_USER_ONLY in */ -static inline void set_helper_retaddr(uintptr_t ra) { } -static inline void clear_helper_retaddr(void) { } -#endif - /* * The result of tlb_vaddr_to_host for user-only is just g2h(x), * which is always non-null. Elide the useless test. @@ -4202,7 +4196,6 @@ static void sve_ld1_r(CPUARMState *env, void *vg, con= st target_ulong addr, return; } mem_off =3D reg_off >> diffsz; - set_helper_retaddr(retaddr); =20 /* * If the (remaining) load is entirely within a single page, then: @@ -4217,7 +4210,6 @@ static void sve_ld1_r(CPUARMState *env, void *vg, con= st target_ulong addr, if (test_host_page(host)) { mem_off =3D host_fn(vd, vg, host - mem_off, mem_off, mem_max); tcg_debug_assert(mem_off =3D=3D mem_max); - clear_helper_retaddr(); /* After having taken any fault, zero leading inactive element= s. */ swap_memzero(vd, reg_off); return; @@ -4268,7 +4260,6 @@ static void sve_ld1_r(CPUARMState *env, void *vg, con= st target_ulong addr, } #endif =20 - clear_helper_retaddr(); memcpy(vd, &scratch, reg_max); } =20 @@ -4328,7 +4319,6 @@ static void sve_ld2_r(CPUARMState *env, void *vg, tar= get_ulong addr, intptr_t i, oprsz =3D simd_oprsz(desc); ARMVectorReg scratch[2] =3D { }; =20 - set_helper_retaddr(ra); for (i =3D 0; i < oprsz; ) { uint16_t pg =3D *(uint16_t *)(vg + H1_2(i >> 3)); do { @@ -4340,7 +4330,6 @@ static void sve_ld2_r(CPUARMState *env, void *vg, tar= get_ulong addr, addr +=3D 2 * size; } while (i & 15); } - clear_helper_retaddr(); =20 /* Wait until all exceptions have been raised to write back. */ memcpy(&env->vfp.zregs[rd], &scratch[0], oprsz); @@ -4355,7 +4344,6 @@ static void sve_ld3_r(CPUARMState *env, void *vg, tar= get_ulong addr, intptr_t i, oprsz =3D simd_oprsz(desc); ARMVectorReg scratch[3] =3D { }; =20 - set_helper_retaddr(ra); for (i =3D 0; i < oprsz; ) { uint16_t pg =3D *(uint16_t *)(vg + H1_2(i >> 3)); do { @@ -4368,7 +4356,6 @@ static void sve_ld3_r(CPUARMState *env, void *vg, tar= get_ulong addr, addr +=3D 3 * size; } while (i & 15); } - clear_helper_retaddr(); =20 /* Wait until all exceptions have been raised to write back. */ memcpy(&env->vfp.zregs[rd], &scratch[0], oprsz); @@ -4384,7 +4371,6 @@ static void sve_ld4_r(CPUARMState *env, void *vg, tar= get_ulong addr, intptr_t i, oprsz =3D simd_oprsz(desc); ARMVectorReg scratch[4] =3D { }; =20 - set_helper_retaddr(ra); for (i =3D 0; i < oprsz; ) { uint16_t pg =3D *(uint16_t *)(vg + H1_2(i >> 3)); do { @@ -4398,7 +4384,6 @@ static void sve_ld4_r(CPUARMState *env, void *vg, tar= get_ulong addr, addr +=3D 4 * size; } while (i & 15); } - clear_helper_retaddr(); =20 /* Wait until all exceptions have been raised to write back. */ memcpy(&env->vfp.zregs[rd], &scratch[0], oprsz); @@ -4505,7 +4490,6 @@ static void sve_ldff1_r(CPUARMState *env, void *vg, c= onst target_ulong addr, return; } mem_off =3D reg_off >> diffsz; - set_helper_retaddr(retaddr); =20 /* * If the (remaining) load is entirely within a single page, then: @@ -4520,7 +4504,6 @@ static void sve_ldff1_r(CPUARMState *env, void *vg, c= onst target_ulong addr, if (test_host_page(host)) { mem_off =3D host_fn(vd, vg, host - mem_off, mem_off, mem_max); tcg_debug_assert(mem_off =3D=3D mem_max); - clear_helper_retaddr(); /* After any fault, zero any leading inactive elements. */ swap_memzero(vd, reg_off); return; @@ -4563,7 +4546,6 @@ static void sve_ldff1_r(CPUARMState *env, void *vg, c= onst target_ulong addr, } #endif =20 - clear_helper_retaddr(); record_fault(env, reg_off, reg_max); } =20 @@ -4709,7 +4691,6 @@ static void sve_st1_r(CPUARMState *env, void *vg, tar= get_ulong addr, intptr_t i, oprsz =3D simd_oprsz(desc); void *vd =3D &env->vfp.zregs[rd]; =20 - set_helper_retaddr(ra); for (i =3D 0; i < oprsz; ) { uint16_t pg =3D *(uint16_t *)(vg + H1_2(i >> 3)); do { @@ -4720,7 +4701,6 @@ static void sve_st1_r(CPUARMState *env, void *vg, tar= get_ulong addr, addr +=3D msize; } while (i & 15); } - clear_helper_retaddr(); } =20 static void sve_st2_r(CPUARMState *env, void *vg, target_ulong addr, @@ -4733,7 +4713,6 @@ static void sve_st2_r(CPUARMState *env, void *vg, tar= get_ulong addr, void *d1 =3D &env->vfp.zregs[rd]; void *d2 =3D &env->vfp.zregs[(rd + 1) & 31]; =20 - set_helper_retaddr(ra); for (i =3D 0; i < oprsz; ) { uint16_t pg =3D *(uint16_t *)(vg + H1_2(i >> 3)); do { @@ -4745,7 +4724,6 @@ static void sve_st2_r(CPUARMState *env, void *vg, tar= get_ulong addr, addr +=3D 2 * msize; } while (i & 15); } - clear_helper_retaddr(); } =20 static void sve_st3_r(CPUARMState *env, void *vg, target_ulong addr, @@ -4759,7 +4737,6 @@ static void sve_st3_r(CPUARMState *env, void *vg, tar= get_ulong addr, void *d2 =3D &env->vfp.zregs[(rd + 1) & 31]; void *d3 =3D &env->vfp.zregs[(rd + 2) & 31]; =20 - set_helper_retaddr(ra); for (i =3D 0; i < oprsz; ) { uint16_t pg =3D *(uint16_t *)(vg + H1_2(i >> 3)); do { @@ -4772,7 +4749,6 @@ static void sve_st3_r(CPUARMState *env, void *vg, tar= get_ulong addr, addr +=3D 3 * msize; } while (i & 15); } - clear_helper_retaddr(); } =20 static void sve_st4_r(CPUARMState *env, void *vg, target_ulong addr, @@ -4787,7 +4763,6 @@ static void sve_st4_r(CPUARMState *env, void *vg, tar= get_ulong addr, void *d3 =3D &env->vfp.zregs[(rd + 2) & 31]; void *d4 =3D &env->vfp.zregs[(rd + 3) & 31]; =20 - set_helper_retaddr(ra); for (i =3D 0; i < oprsz; ) { uint16_t pg =3D *(uint16_t *)(vg + H1_2(i >> 3)); do { @@ -4801,7 +4776,6 @@ static void sve_st4_r(CPUARMState *env, void *vg, tar= get_ulong addr, addr +=3D 4 * msize; } while (i & 15); } - clear_helper_retaddr(); } =20 #define DO_STN_1(N, NAME, ESIZE) \ @@ -4897,7 +4871,6 @@ static void sve_ld1_zs(CPUARMState *env, void *vd, vo= id *vg, void *vm, intptr_t i, oprsz =3D simd_oprsz(desc); ARMVectorReg scratch =3D { }; =20 - set_helper_retaddr(ra); for (i =3D 0; i < oprsz; ) { uint16_t pg =3D *(uint16_t *)(vg + H1_2(i >> 3)); do { @@ -4908,7 +4881,6 @@ static void sve_ld1_zs(CPUARMState *env, void *vd, vo= id *vg, void *vm, i +=3D 4, pg >>=3D 4; } while (i & 15); } - clear_helper_retaddr(); =20 /* Wait until all exceptions have been raised to write back. */ memcpy(vd, &scratch, oprsz); @@ -4922,7 +4894,6 @@ static void sve_ld1_zd(CPUARMState *env, void *vd, vo= id *vg, void *vm, intptr_t i, oprsz =3D simd_oprsz(desc) / 8; ARMVectorReg scratch =3D { }; =20 - set_helper_retaddr(ra); for (i =3D 0; i < oprsz; i++) { uint8_t pg =3D *(uint8_t *)(vg + H1(i)); if (likely(pg & 1)) { @@ -4930,7 +4901,6 @@ static void sve_ld1_zd(CPUARMState *env, void *vd, vo= id *vg, void *vm, tlb_fn(env, &scratch, i * 8, base + (off << scale), ra); } } - clear_helper_retaddr(); =20 /* Wait until all exceptions have been raised to write back. */ memcpy(vd, &scratch, oprsz * 8); @@ -5102,13 +5072,11 @@ static inline void sve_ldff1_zs(CPUARMState *env, v= oid *vd, void *vg, void *vm, reg_off =3D find_next_active(vg, 0, reg_max, MO_32); if (likely(reg_off < reg_max)) { /* Perform one normal read, which will fault or not. */ - set_helper_retaddr(ra); addr =3D off_fn(vm, reg_off); addr =3D base + (addr << scale); tlb_fn(env, vd, reg_off, addr, ra); =20 /* The rest of the reads will be non-faulting. */ - clear_helper_retaddr(); } =20 /* After any fault, zero the leading predicated false elements. */ @@ -5144,13 +5112,11 @@ static inline void sve_ldff1_zd(CPUARMState *env, v= oid *vd, void *vg, void *vm, reg_off =3D find_next_active(vg, 0, reg_max, MO_64); if (likely(reg_off < reg_max)) { /* Perform one normal read, which will fault or not. */ - set_helper_retaddr(ra); addr =3D off_fn(vm, reg_off); addr =3D base + (addr << scale); tlb_fn(env, vd, reg_off, addr, ra); =20 /* The rest of the reads will be non-faulting. */ - clear_helper_retaddr(); } =20 /* After any fault, zero the leading predicated false elements. */ @@ -5262,7 +5228,6 @@ static void sve_st1_zs(CPUARMState *env, void *vd, vo= id *vg, void *vm, const int scale =3D extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, = 2); intptr_t i, oprsz =3D simd_oprsz(desc); =20 - set_helper_retaddr(ra); for (i =3D 0; i < oprsz; ) { uint16_t pg =3D *(uint16_t *)(vg + H1_2(i >> 3)); do { @@ -5273,7 +5238,6 @@ static void sve_st1_zs(CPUARMState *env, void *vd, vo= id *vg, void *vm, i +=3D 4, pg >>=3D 4; } while (i & 15); } - clear_helper_retaddr(); } =20 static void sve_st1_zd(CPUARMState *env, void *vd, void *vg, void *vm, @@ -5283,7 +5247,6 @@ static void sve_st1_zd(CPUARMState *env, void *vd, vo= id *vg, void *vm, const int scale =3D extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, = 2); intptr_t i, oprsz =3D simd_oprsz(desc) / 8; =20 - set_helper_retaddr(ra); for (i =3D 0; i < oprsz; i++) { uint8_t pg =3D *(uint8_t *)(vg + H1(i)); if (likely(pg & 1)) { @@ -5291,7 +5254,6 @@ static void sve_st1_zd(CPUARMState *env, void *vd, vo= id *vg, void *vm, tlb_fn(env, vd, i * 8, base + (off << scale), ra); } } - clear_helper_retaddr(); } =20 #define DO_ST1_ZPZ_S(MEM, OFS) \ --=20 2.20.1 From nobody Mon Feb 9 21:08:28 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=linaro.org ARC-Seal: i=1; a=rsa-sha256; t=1583909408; cv=none; d=zohomail.com; s=zohoarc; b=fXfT78urqpPB7Fs1j0oHypBLOeK1OWeok63EE8hF8qHjbRtK/aR9J5veZcpRWVTl9F9d+VcUgz7PgASSWy/tsMwehF3cjqwUSKk8bNws9tWw4YY4G7TjO+g2Jn7Rz1syLTb551U9egUYrUa3773v0ppsZJJGQI8TBUDYNoIsmEI= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1583909408; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=WeaNzwzC03UA9+8arqHbMRnYm8SNdcLfajSQ8cx3zrw=; b=IHwJb67DM9XPDMkk9uZVjcGBSHxetaOinoO6jZXwB0gEyIjL3Xm+KUrAjhO+b8Qn99CgIeZ/3EWvyPHBkYrwfwDfRTKL74NQtbUNlPw4fzsFfnQkFGIBbShXhXKcUS0x+VPbBC4/zsE7BKkZo2XDIo7DZfrgkJBV8Tyx4uv8dNs= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1583909408561949.281320404941; Tue, 10 Mar 2020 23:50:08 -0700 (PDT) Received: from localhost ([::1]:46528 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jBvBv-00062r-Eo for importer@patchew.org; Wed, 11 Mar 2020 02:50:07 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:59454) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jBv6Z-0006CR-Mi for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:37 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jBv6X-0001vU-PY for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:35 -0400 Received: from mail-pl1-x642.google.com ([2607:f8b0:4864:20::642]:41528) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1jBv6V-0001rw-TO for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:33 -0400 Received: by mail-pl1-x642.google.com with SMTP id t14so614380plr.8 for ; Tue, 10 Mar 2020 23:44:30 -0700 (PDT) Received: from localhost.localdomain (97-126-123-70.tukw.qwest.net. [97.126.123.70]) by smtp.gmail.com with ESMTPSA id z3sm50137807pfz.155.2020.03.10.23.44.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 10 Mar 2020 23:44:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=WeaNzwzC03UA9+8arqHbMRnYm8SNdcLfajSQ8cx3zrw=; b=EL2RWt2DGrX6kJg5C4+cQ18aibmZetpuMBLgbQxG11B+rOZfBhV54BmfC2uh+j8+Z6 SVWdFT4QIl7aQvjuGBj2T6f1Z3O8zNKAwSNEsYY8lY7nR+WnmC7yk1AWZhBOf4grB87Y 4q2WOi/C3HO4mG2QSTyVvDm/eaAG4eMJnZMwb7ZUiLuVlC3+tpFiBCFwEwpVlSwUcsXS TeBkoJsy6vpa84dCFdTK5TmBtCEA2d1Rqpvwo50X+HZXQZg/CBLHQ7GxlhoE07Z2ECnE nuSwpZ5gsSBImNhjcbEAakSOR28LDROD041GlDZiCJXCY7LeUPWi7efTC41AKXZKLVaz 21uA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=WeaNzwzC03UA9+8arqHbMRnYm8SNdcLfajSQ8cx3zrw=; b=kEVQQuq8ye0JoE8LvcimlFow18shbpF9y8uyUatcXIwi0GDl7Q48c0lueFEfQLhH0h pImPzng/LwQdYUrIsZCtRfRxvJZfKKrOSDWFD2WB3esZKMhJbc4KrtsfKtJPVm7Qq4Nw 0z9Zm2GXemvjsTskmsKFJvVIkxeUpIUNwO7xBzQavBTbI/EXUhx9ni/ANNI1W0F78Wkb y2QMWd9G6SUHuw1OsX5Ugmlm7GAGhHF7GDSsPqiUvtCmniUJ8kIH9JYJGtl/APgpY0LY at3RE75s42RUJr7kgBC1kWGzLFdmx400TdGPK3lNcelGxjMhXpTxuRDBKl/BkzZsccSo 99jg== X-Gm-Message-State: ANhLgQ12Nw/0L3BrEsCnFvBeWbTQpINQi+NEm4J8GRmQb5kyT3Fdzg0W 7BzMQuSPyECQJ/6vifGc9SpCSn/2j0w= X-Google-Smtp-Source: ADFU+vspqwWUbwrKnd7zrvYKWblSz69ADlPtluClng+gC0u+7AW07LyUTe9wlUObKCiEjD18SZq+OA== X-Received: by 2002:a17:902:a408:: with SMTP id p8mr1782284plq.132.1583909069346; Tue, 10 Mar 2020 23:44:29 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH 06/16] target/arm: Add sve infrastructure for page lookup Date: Tue, 10 Mar 2020 23:44:10 -0700 Message-Id: <20200311064420.30606-7-richard.henderson@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200311064420.30606-1-richard.henderson@linaro.org> References: <20200311064420.30606-1-richard.henderson@linaro.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::642 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, qemu-arm@nongnu.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @linaro.org) Content-Type: text/plain; charset="utf-8" For contiguous predicated memory operations, we want to minimize the number of tlb lookups performed. We have open-coded this for sve_ld1_r, but for correctness with MTE we will need this for all of the memory operations. Create a structure that holds the bounds of active elements, and metadata for two pages. Add routines to find those active elements, lookup the pages, and run watchpoints for those pages. Temporarily mark the functions unused to avoid Werror. Signed-off-by: Richard Henderson --- target/arm/sve_helper.c | 240 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 240 insertions(+) diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index 8b470991db..3f653e46a0 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -4155,6 +4155,246 @@ static intptr_t max_for_page(target_ulong base, int= ptr_t mem_off, return MIN(split, mem_max - mem_off) + mem_off; } =20 +/* + * Resolve the guest virtual address to info->host and info->flags. + * If @nofault, return false if the page is invalid, otherwise + * exit via page fault exception. + */ + +typedef struct { + void *host; + int flags; + MemTxAttrs attrs; +} SVEHostPage; + +static bool sve_probe_page(SVEHostPage *info, bool nofault, + CPUARMState *env, target_ulong addr, + int mem_off, MMUAccessType access_type, + int mmu_idx, uintptr_t retaddr) +{ + int flags; + + addr +=3D mem_off; + flags =3D probe_access_flags(env, addr, access_type, mmu_idx, nofault, + &info->host, retaddr); + info->flags =3D flags; + + if (flags & TLB_INVALID_MASK) { + g_assert(nofault); + return false; + } + + /* Ensure that info->host[] is relative to addr, not addr + mem_off. */ + info->host -=3D mem_off; + +#ifdef CONFIG_USER_ONLY + memset(&info->attrs, 0, sizeof(info->attrs)); +#else + /* + * Find the iotlbentry for addr and return the transaction attributes. + * This *must* be present in the TLB because we just found the mapping. + */ + { + uintptr_t index =3D tlb_index(env, mmu_idx, addr); + +# ifdef CONFIG_DEBUG_TCG + CPUTLBEntry *entry =3D tlb_entry(env, mmu_idx, addr); + target_ulong comparator =3D (access_type =3D=3D MMU_DATA_LOAD + ? entry->addr_read + : tlb_addr_write(entry)); + g_assert(tlb_hit(comparator, addr)); +# endif + + CPUIOTLBEntry *iotlbentry =3D &env_tlb(env)->d[mmu_idx].iotlb[inde= x]; + info->attrs =3D iotlbentry->attrs; + } +#endif + + return true; +} + + +/* + * Analyse contiguous data, protected by a governing predicate. + */ + +typedef enum { + FAULT_NO, + FAULT_FIRST, + FAULT_ALL, +} SVEContFault; + +typedef struct { + /* First and last element wholy contained within the two pages. */ + int16_t mem_off_first[2]; + int16_t reg_off_first[2]; + int16_t reg_off_last[2]; + + /* One element that is misaligned and spans both pages. */ + int16_t mem_off_split; + int16_t reg_off_split; + int16_t page_split; + + /* TLB data for the two pages. */ + SVEHostPage page[2]; +} SVEContLdSt; + +/* + * Find first active element on each page, and a loose bound for the + * final element on each page. Identify any single element that spans + * the page boundary. Return true if there are any active elements. + */ +static bool __attribute__((unused)) +sve_cont_ldst_elements(SVEContLdSt *info, target_ulong addr, uint64_t *vg, + intptr_t reg_max, int esz, int msize) +{ + const int esize =3D 1 << esz; + const uint64_t pg_mask =3D pred_esz_masks[esz]; + intptr_t reg_off_first =3D -1, reg_off_last =3D -1, reg_off_split; + intptr_t mem_off_last, mem_off_split; + intptr_t page_split, elt_split; + intptr_t i; + + /* Set all of the element indicies to -1, and the TLB data to 0. */ + memset(info, -1, offsetof(SVEContLdSt, page)); + memset(info->page, 0, sizeof(info->page)); + + /* Gross scan over the entire predicate to find bounds. */ + i =3D 0; + do { + uint64_t pg =3D vg[i] & pg_mask; + if (pg) { + reg_off_last =3D i * 64 + 63 - clz64(pg); + if (reg_off_first < 0) { + reg_off_first =3D i * 64 + ctz64(pg); + } + } + } while (++i * 64 < reg_max); + + if (unlikely(reg_off_first < 0)) { + /* No active elements, no pages touched. */ + return false; + } + tcg_debug_assert(reg_off_last >=3D 0 && reg_off_last < reg_max); + + info->reg_off_first[0] =3D reg_off_first; + info->mem_off_first[0] =3D (reg_off_first >> esz) * msize; + mem_off_last =3D (reg_off_last >> esz) * msize; + + page_split =3D -(addr | TARGET_PAGE_MASK); + if (likely(mem_off_last + msize <=3D page_split)) { + /* The entire operation fits within a single page. */ + info->reg_off_last[0] =3D reg_off_last; + return true; + } + + info->page_split =3D page_split; + elt_split =3D page_split / msize; + reg_off_split =3D elt_split << esz; + mem_off_split =3D elt_split * msize; + + /* + * This is the last full element on the first page, but it is not + * necessarily active. If there is no full element, i.e. the first + * active element is the one that's split, this value remains -1. + * It is useful as iteration bounds. + */ + if (elt_split !=3D 0) { + info->reg_off_last[0] =3D reg_off_split - esize; + } + + /* Determine if an unaligned element spans the pages. */ + if (page_split % msize !=3D 0) { + /* It is helpful to know if the split element is active. */ + if ((vg[reg_off_split >> 6] >> (reg_off_split & 63)) & 1) { + info->reg_off_split =3D reg_off_split; + info->mem_off_split =3D mem_off_split; + + if (reg_off_split =3D=3D reg_off_last) { + /* The page crossing element is last. */ + return true; + } + } + reg_off_split +=3D esize; + mem_off_split +=3D msize; + } + + /* + * We do want the first active element on the second page, because + * this may affect the address reported in an exception. + */ + reg_off_split =3D find_next_active(vg, reg_off_split, reg_max, esz); + tcg_debug_assert(reg_off_split <=3D reg_off_last); + info->reg_off_first[1] =3D reg_off_split; + info->mem_off_first[1] =3D (reg_off_split >> esz) * msize; + info->reg_off_last[1] =3D reg_off_last; + return true; +} + +/* + * Resolve the guest virtual addresses to info->page[]. + * Control the generation of page faults with @fault. Return false if + * there is no work to do, which can only happen with @fault =3D=3D FAULT_= NO. + */ +static bool __attribute__((unused)) +sve_cont_ldst_pages(SVEContLdSt *info, SVEContFault fault, CPUARMState *en= v, + target_ulong addr, MMUAccessType access_type, + uintptr_t retaddr) +{ + int mmu_idx =3D cpu_mmu_index(env, false); + int mem_off =3D info->mem_off_first[0]; + bool nofault =3D fault =3D=3D FAULT_NO; + bool have_work =3D true; + + if (!sve_probe_page(&info->page[0], nofault, env, addr, mem_off, + access_type, mmu_idx, retaddr)) { + /* No work to be done. */ + return false; + } + + if (likely(info->page_split < 0)) { + /* The entire operation was on the one page. */ + return true; + } + + /* + * If the second page is invalid, then we want the fault address to be + * the first byte on that page which is accessed. + */ + if (info->mem_off_split >=3D 0) { + /* + * There is an element split across the pages. The fault address + * should be the first byte of the second page. + */ + mem_off =3D info->page_split; + /* + * If the split element is also the first active element + * of the vector, then: For first-fault we should continue + * to generate faults for the second page. For no-fault, + * we have work only if the second page is valid. + */ + if (info->mem_off_first[0] < info->mem_off_split) { + nofault =3D FAULT_FIRST; + have_work =3D false; + } + } else { + /* + * There is no element split across the pages. The fault address + * should be the first active element on the second page. + */ + mem_off =3D info->mem_off_first[1]; + /* + * There must have been one active element on the first page, + * so we're out of first-fault territory. + */ + nofault =3D fault !=3D FAULT_ALL; + } + + have_work |=3D sve_probe_page(&info->page[1], nofault, env, addr, mem_= off, + access_type, mmu_idx, retaddr); + return have_work; +} + /* * The result of tlb_vaddr_to_host for user-only is just g2h(x), * which is always non-null. Elide the useless test. --=20 2.20.1 From nobody Mon Feb 9 21:08:28 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=linaro.org ARC-Seal: i=1; a=rsa-sha256; t=1583909589; cv=none; d=zohomail.com; s=zohoarc; b=QfGAPs27tJD5t75zSL/N70RKL+FIN5+uwtHcTKsLFRJbshZt3TrAiS2Wd9qXOD3Yogbvy+O9/TiuyvtFVqs4f2mLVce4YIwCzp0f1XfXwlnDYiCqrAPdAVFOPatrc7l2N9ESiK9SZl1Iu6cRgZaDI2BxeFXoC/Qh01YftLty9XI= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1583909589; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=3qRBHeH49omwqgoxg8Z+ZrefAQ7kYCLg+akntsfuZdA=; b=hNkGcYG70IAnWFmA1/FjTQugiK4wkadx7rPfC2UEEyKBEl0qky5Nnktm7pK8rSGTNr3XGmaWrVH1GsmL1KKfm57FCgrkAQ/kVdF6x49veJHEXZnSX4yjdSIWdQPTMIUSF5XrsWJOodtvzRygYkeUpCBjXg0/96JdJpcm8geMQqc= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1583909589362793.9211141522449; Tue, 10 Mar 2020 23:53:09 -0700 (PDT) Received: from localhost ([::1]:46650 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jBvEq-0002AQ-6S for importer@patchew.org; Wed, 11 Mar 2020 02:53:08 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:59515) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jBv6b-0006FL-PO for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:40 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jBv6Z-0001ws-Kd for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:37 -0400 Received: from mail-pl1-x62f.google.com ([2607:f8b0:4864:20::62f]:34341) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1jBv6Y-0001tB-7v for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:34 -0400 Received: by mail-pl1-x62f.google.com with SMTP id a23so630448plm.1 for ; Tue, 10 Mar 2020 23:44:32 -0700 (PDT) Received: from localhost.localdomain (97-126-123-70.tukw.qwest.net. [97.126.123.70]) by smtp.gmail.com with ESMTPSA id z3sm50137807pfz.155.2020.03.10.23.44.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 10 Mar 2020 23:44:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=3qRBHeH49omwqgoxg8Z+ZrefAQ7kYCLg+akntsfuZdA=; b=QR0vyAfXuvl8umdBF3mzlxvOCfvxsRghnY2q01fR0IaVMAMSiFnoy9QsH7BNhA652F MFLrDpP6DPCiSXowIJfoAgQHSdz7uZeNQz2DeY50VZFkuJghMXu5FsU5zzvW+j4cVoaD bXO45GF7RwA5i+Dmsu5bPT8Nrs2XRII0sVS6dSeQhbXapVNWF/JIgV/bt7MjV7gu6BmX FyAIiUn88NWN47ByGImQFIVw2gZP/cWrTPUgX6L88NY3f+86Iy26fxNP135MFGqDTIKg /VqZ7aqTlJ3YFTPVc8xnS/qLAfcQx+0qffx71PjMhsrHEnJTyrUDAMBM+YL1v/XkU7FO kULg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=3qRBHeH49omwqgoxg8Z+ZrefAQ7kYCLg+akntsfuZdA=; b=FfkvzGM8HwnfJEqJBrunbMTqi8nwct50dsjRhuL5MvkdlO50qFZfsNSnoOveocgz6m FCUN+CAwnDkfu8Scp1UUXLfvjn2a9aSYtj2IxI/7Mu053zSHbeZiY6rUY6947U94yVPh e/lAyl8DXjJgOQVcNWhyDO/dGyKvW5O3h0tdTsItjDIDlLsRC9MzYWCAzv4edcXKeOcQ dHNBADVhJTdwzQNGfwsxT+DT1LFxIymQ8AQ8irBZJ5jIh3m24rJbPikFdyhcKYiht2GI 8uuE73/LZRNb0RpDJ9q1pr2SP+Vya9M5oFfmY81W7PTkKz2X3aKt1s3V22O1GpYh5Jon 7cdQ== X-Gm-Message-State: ANhLgQ0rovBZPqlpPqEqApRjmAfZ41ZM0qExlr4Ck763POA0Hh6atZ6L itukmB0N3wsZSMpwDUW9UoWhRdMWo/w= X-Google-Smtp-Source: ADFU+vuZuH6tHfEPvGuVxPO2i2jSrf7AW1wuRDOSvQlQVE7cyjpgrEhBFZG+P32jS1JXgkDSWByBoQ== X-Received: by 2002:a17:902:7b8d:: with SMTP id w13mr1795283pll.78.1583909070792; Tue, 10 Mar 2020 23:44:30 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH 07/16] target/arm: Adjust interface of sve_ld1_host_fn Date: Tue, 10 Mar 2020 23:44:11 -0700 Message-Id: <20200311064420.30606-8-richard.henderson@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200311064420.30606-1-richard.henderson@linaro.org> References: <20200311064420.30606-1-richard.henderson@linaro.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::62f X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, qemu-arm@nongnu.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @linaro.org) Content-Type: text/plain; charset="utf-8" The current interface includes a loop; change it to load a single element. We will then be able to use the function for ld{2,3,4} where individual vector elements are not adjacent. Replace each call with the simplest possible loop over active elements. Signed-off-by: Richard Henderson Reviewed-by: Peter Maydell --- target/arm/sve_helper.c | 127 +++++++++++++++++++++------------------- 1 file changed, 66 insertions(+), 61 deletions(-) diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index 3f653e46a0..1397c2b634 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -3972,20 +3972,10 @@ void HELPER(sve_fcmla_zpzzz_d)(CPUARMState *env, vo= id *vg, uint32_t desc) */ =20 /* - * Load elements into @vd, controlled by @vg, from @host + @mem_ofs. - * Memory is valid through @host + @mem_max. The register element - * indicies are inferred from @mem_ofs, as modified by the types for - * which the helper is built. Return the @mem_ofs of the first element - * not loaded (which is @mem_max if they are all loaded). - * - * For softmmu, we have fully validated the guest page. For user-only, - * we cannot fully validate without taking the mmap lock, but since we - * know the access is within one host page, if any access is valid they - * all must be valid. However, when @vg is all false, it may be that - * no access is valid. + * Load elements into @vd + @reg_off, from @host, + * or the reverse for stores. */ -typedef intptr_t sve_ld1_host_fn(void *vd, void *vg, void *host, - intptr_t mem_ofs, intptr_t mem_max); +typedef void sve_ldst1_host_fn(void *vd, intptr_t reg_off, void *host); =20 /* * Load one element into @vd + @reg_off from (@env, @vaddr, @ra). @@ -4000,23 +3990,16 @@ typedef void sve_ldst1_tlb_fn(CPUARMState *env, voi= d *vd, intptr_t reg_off, * For *_tlb, this uses the cpu_*_data_ra helpers. There are not * endian-specific versions of these, so we must handle endianness * locally. + * + * For *_host, this is a trivial application of the + * endian-specific access followed by a store into the vector register. */ =20 #define DO_LD_HOST(NAME, H, TYPEE, TYPEM, HOST) \ -static intptr_t sve_##NAME##_host(void *vd, void *vg, void *host, = \ - intptr_t mem_off, const intptr_t mem_max= ) \ -{ = \ - intptr_t reg_off =3D mem_off * (sizeof(TYPEE) / sizeof(TYPEM)); = \ - uint64_t *pg =3D vg; = \ - while (mem_off + sizeof(TYPEM) <=3D mem_max) { = \ - TYPEM val =3D 0; = \ - if (likely((pg[reg_off >> 6] >> (reg_off & 63)) & 1)) { = \ - val =3D HOST(host + mem_off); = \ - } = \ - *(TYPEE *)(vd + H(reg_off)) =3D val; = \ - mem_off +=3D sizeof(TYPEM), reg_off +=3D sizeof(TYPEE); = \ - } = \ - return mem_off; = \ +static void sve_##NAME##_host(void *vd, intptr_t reg_off, void *host) \ +{ \ + TYPEM val =3D HOST(host); \ + *(TYPEE *)(vd + H(reg_off)) =3D val; \ } =20 #define DO_LD_TLB(NAME, H, TYPEE, TYPEM, BSWAP, TLB) \ @@ -4414,7 +4397,7 @@ static inline bool test_host_page(void *host) static void sve_ld1_r(CPUARMState *env, void *vg, const target_ulong addr, uint32_t desc, const uintptr_t retaddr, const int esz, const int msz, - sve_ld1_host_fn *host_fn, + sve_ldst1_host_fn *host_fn, sve_ldst1_tlb_fn *tlb_fn) { const TCGMemOpIdx oi =3D extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHI= FT); @@ -4448,8 +4431,12 @@ static void sve_ld1_r(CPUARMState *env, void *vg, co= nst target_ulong addr, if (likely(split =3D=3D mem_max)) { host =3D tlb_vaddr_to_host(env, addr + mem_off, MMU_DATA_LOAD, mmu= _idx); if (test_host_page(host)) { - mem_off =3D host_fn(vd, vg, host - mem_off, mem_off, mem_max); - tcg_debug_assert(mem_off =3D=3D mem_max); + intptr_t i =3D reg_off; + host -=3D mem_off; + do { + host_fn(vd, i, host + (i >> diffsz)); + i =3D find_next_active(vg, i + (1 << esz), reg_max, esz); + } while (i < reg_max); /* After having taken any fault, zero leading inactive element= s. */ swap_memzero(vd, reg_off); return; @@ -4462,7 +4449,12 @@ static void sve_ld1_r(CPUARMState *env, void *vg, co= nst target_ulong addr, */ #ifdef CONFIG_USER_ONLY swap_memzero(&scratch, reg_off); - host_fn(&scratch, vg, g2h(addr), mem_off, mem_max); + host =3D g2h(addr); + do { + host_fn(&scratch, reg_off, host + (reg_off >> diffsz)); + reg_off +=3D 1 << esz; + reg_off =3D find_next_active(vg, reg_off, reg_max, esz); + } while (reg_off < reg_max); #else memset(&scratch, 0, reg_max); goto start; @@ -4480,9 +4472,13 @@ static void sve_ld1_r(CPUARMState *env, void *vg, co= nst target_ulong addr, host =3D tlb_vaddr_to_host(env, addr + mem_off, MMU_DATA_LOAD, mmu_idx); if (host) { - mem_off =3D host_fn(&scratch, vg, host - mem_off, - mem_off, split); - reg_off =3D mem_off << diffsz; + host -=3D mem_off; + do { + host_fn(&scratch, reg_off, host + mem_off); + reg_off +=3D 1 << esz; + reg_off =3D find_next_active(vg, reg_off, reg_max, esz= ); + mem_off =3D reg_off >> diffsz; + } while (split - mem_off >=3D (1 << msz)); continue; } } @@ -4709,7 +4705,7 @@ static void record_fault(CPUARMState *env, uintptr_t = i, uintptr_t oprsz) static void sve_ldff1_r(CPUARMState *env, void *vg, const target_ulong add= r, uint32_t desc, const uintptr_t retaddr, const int esz, const int msz, - sve_ld1_host_fn *host_fn, + sve_ldst1_host_fn *host_fn, sve_ldst1_tlb_fn *tlb_fn) { const TCGMemOpIdx oi =3D extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHI= FT); @@ -4719,7 +4715,7 @@ static void sve_ldff1_r(CPUARMState *env, void *vg, c= onst target_ulong addr, const int diffsz =3D esz - msz; const intptr_t reg_max =3D simd_oprsz(desc); const intptr_t mem_max =3D reg_max >> diffsz; - intptr_t split, reg_off, mem_off; + intptr_t split, reg_off, mem_off, i; void *host; =20 /* Skip to the first active element. */ @@ -4742,28 +4738,18 @@ static void sve_ldff1_r(CPUARMState *env, void *vg,= const target_ulong addr, if (likely(split =3D=3D mem_max)) { host =3D tlb_vaddr_to_host(env, addr + mem_off, MMU_DATA_LOAD, mmu= _idx); if (test_host_page(host)) { - mem_off =3D host_fn(vd, vg, host - mem_off, mem_off, mem_max); - tcg_debug_assert(mem_off =3D=3D mem_max); + i =3D reg_off; + host -=3D mem_off; + do { + host_fn(vd, i, host + (i >> diffsz)); + i =3D find_next_active(vg, i + (1 << esz), reg_max, esz); + } while (i < reg_max); /* After any fault, zero any leading inactive elements. */ swap_memzero(vd, reg_off); return; } } =20 -#ifdef CONFIG_USER_ONLY - /* - * The page(s) containing this first element at ADDR+MEM_OFF must - * be valid. Considering that this first element may be misaligned - * and cross a page boundary itself, take the rest of the page from - * the last byte of the element. - */ - split =3D max_for_page(addr, mem_off + (1 << msz) - 1, mem_max); - mem_off =3D host_fn(vd, vg, g2h(addr), mem_off, split); - - /* After any fault, zero any leading inactive elements. */ - swap_memzero(vd, reg_off); - reg_off =3D mem_off << diffsz; -#else /* * Perform one normal read, which will fault or not. * But it is likely to bring the page into the tlb. @@ -4780,11 +4766,15 @@ static void sve_ldff1_r(CPUARMState *env, void *vg,= const target_ulong addr, if (split >=3D (1 << msz)) { host =3D tlb_vaddr_to_host(env, addr + mem_off, MMU_DATA_LOAD, mmu= _idx); if (host) { - mem_off =3D host_fn(vd, vg, host - mem_off, mem_off, split); - reg_off =3D mem_off << diffsz; + host -=3D mem_off; + do { + host_fn(vd, reg_off, host + mem_off); + reg_off +=3D 1 << esz; + reg_off =3D find_next_active(vg, reg_off, reg_max, esz); + mem_off =3D reg_off >> diffsz; + } while (split - mem_off >=3D (1 << msz)); } } -#endif =20 record_fault(env, reg_off, reg_max); } @@ -4794,7 +4784,7 @@ static void sve_ldff1_r(CPUARMState *env, void *vg, c= onst target_ulong addr, */ static void sve_ldnf1_r(CPUARMState *env, void *vg, const target_ulong add= r, uint32_t desc, const int esz, const int msz, - sve_ld1_host_fn *host_fn) + sve_ldst1_host_fn *host_fn) { const unsigned rd =3D extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT= , 5); void *vd =3D &env->vfp.zregs[rd]; @@ -4809,7 +4799,13 @@ static void sve_ldnf1_r(CPUARMState *env, void *vg, = const target_ulong addr, host =3D tlb_vaddr_to_host(env, addr, MMU_DATA_LOAD, mmu_idx); if (likely(page_check_range(addr, mem_max, PAGE_READ) =3D=3D 0)) { /* The entire operation is valid and will not fault. */ - host_fn(vd, vg, host, 0, mem_max); + reg_off =3D 0; + do { + mem_off =3D reg_off >> diffsz; + host_fn(vd, reg_off, host + mem_off); + reg_off +=3D 1 << esz; + reg_off =3D find_next_active(vg, reg_off, reg_max, esz); + } while (reg_off < reg_max); return; } #endif @@ -4829,8 +4825,12 @@ static void sve_ldnf1_r(CPUARMState *env, void *vg, = const target_ulong addr, if (page_check_range(addr + mem_off, 1 << msz, PAGE_READ) =3D=3D 0) { /* At least one load is valid; take the rest of the page. */ split =3D max_for_page(addr, mem_off + (1 << msz) - 1, mem_max); - mem_off =3D host_fn(vd, vg, host, mem_off, split); - reg_off =3D mem_off << diffsz; + do { + host_fn(vd, reg_off, host + mem_off); + reg_off +=3D 1 << esz; + reg_off =3D find_next_active(vg, reg_off, reg_max, esz); + mem_off =3D reg_off >> diffsz; + } while (split - mem_off >=3D (1 << msz)); } #else /* @@ -4851,8 +4851,13 @@ static void sve_ldnf1_r(CPUARMState *env, void *vg, = const target_ulong addr, host =3D tlb_vaddr_to_host(env, addr + mem_off, MMU_DATA_LOAD, mmu_idx= ); split =3D max_for_page(addr, mem_off, mem_max); if (host && split >=3D (1 << msz)) { - mem_off =3D host_fn(vd, vg, host - mem_off, mem_off, split); - reg_off =3D mem_off << diffsz; + host -=3D mem_off; + do { + host_fn(vd, reg_off, host + mem_off); + reg_off +=3D 1 << esz; + reg_off =3D find_next_active(vg, reg_off, reg_max, esz); + mem_off =3D reg_off >> diffsz; + } while (split - mem_off >=3D (1 << msz)); } #endif =20 --=20 2.20.1 From nobody Mon Feb 9 21:08:28 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=linaro.org ARC-Seal: i=1; a=rsa-sha256; t=1583909471; cv=none; d=zohomail.com; s=zohoarc; b=bchbjbbBBdb2zXtkvPfeAX+WW2VZov8HDRDVN98whmvCzQC7VmW4b7ymiNuCt+nd9micY1AZeZYk1N1W/Ph6ksCby/KAnzt3UeCt7bAgvslNH89War6hBJ1AE51U8E7hgC/RIRC53AAxffToTRjHkOlQmMpY26OavyuYtxw53w4= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1583909471; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=hwv0ZdGNzk1JPF3B9TKzm+UaZXEnJM80ZJXFaFLa00w=; b=gJktt0znJFAnfyNkgyFsP/Jej7RUybtKqYIIXKVu9kVpKbCNTR7NGRfW36oaoFNtQkqAQo+BAPiPjMgHAvzHwRNz7t5HX4hwKk9Td7wH9D5IW7P/Ng+FfAVXm4reTAd3gL4JKc2PMCDsScHdqwwAC28G0HAVU/0wLCA/KqMimDg= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1583909471897747.6901788641385; Tue, 10 Mar 2020 23:51:11 -0700 (PDT) Received: from localhost ([::1]:46562 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jBvCw-0007eI-MN for importer@patchew.org; Wed, 11 Mar 2020 02:51:10 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:59513) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jBv6b-0006FI-NW for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:39 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jBv6Z-0001xD-MB for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:37 -0400 Received: from mail-pj1-x1041.google.com ([2607:f8b0:4864:20::1041]:34411) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1jBv6Z-0001v3-9A for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:35 -0400 Received: by mail-pj1-x1041.google.com with SMTP id 39so1174667pjo.1 for ; Tue, 10 Mar 2020 23:44:33 -0700 (PDT) Received: from localhost.localdomain (97-126-123-70.tukw.qwest.net. [97.126.123.70]) by smtp.gmail.com with ESMTPSA id z3sm50137807pfz.155.2020.03.10.23.44.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 10 Mar 2020 23:44:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=hwv0ZdGNzk1JPF3B9TKzm+UaZXEnJM80ZJXFaFLa00w=; b=N2/L1msjt0mGGn+cirOIcg8NYShSTV/hYDQ59P8VIQpAf5S+ShR5sh2OuX2r3JvY8g UCDGflEv8oz8HRj0mlU47215JBpcOrDUjse83MJkXZaCosGYIGj1h/IeZMvuhr7ALRau CXJHgl+tJVFn20/93wkBMBRJPWOph/beNzPN82tDgaFzpfj5aCnFumB84CmnYp7XqVir 3GhWX5ZPFC0jDd6g24cjnS7/yoL7PMmIuw90Ba/P/Ifeb0Av1J2TwnI64Pf4guEJrsT5 7sq4MKLbk7KjQIHwAM9APieWLOMXVuJV9rjBltciNYy6ulCxq79tZQisXjOUNl/nQzen nOgg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=hwv0ZdGNzk1JPF3B9TKzm+UaZXEnJM80ZJXFaFLa00w=; b=Pi9UnlpLvwHWkurJlANl4yL4DeEW8beBH9mLkcIj9VVYF1vr6LVnNDIctZY9Rm78lQ 7mRMmrh+l9Hj6/K1WWdT+9+XMNO+T3uP+3GokbT4hJSBX1x1TVCJgbwQzVbCQZXz9OfJ hxmLRGSSPbl9O7GuGSyV+S69jCIv1s8Y8g0GhfEL6cAne2fHRi4ZuMqbKB7HeZf1PG9V 4R0cMBG0GrxhuYbhKo0ZQQA9ifJpYnRvYUnP8EbIXYH3TI5nxMhZvm3OKOgLjFJVMS9g DNMFvGskuKlfG/XBjhIqr2xNrk84VSJdbWMPb8IeK/PJin8/th8A2II+wmMAYBbJz39o p7fw== X-Gm-Message-State: ANhLgQ3BPBBh4F9e5PC0QXiG2lXcbCyPARRsHTpq4TfKSH+0smCd9y/T fT6G801uXTmLAxXUG8YWcHENv6Eh6hg= X-Google-Smtp-Source: ADFU+vtBLBQXkKjkL9NKVvmD2eibpwa6DgfXdpLlYXmDhfPRXWZZCDj2/AhapkOb0vGc8m6jBO/OBA== X-Received: by 2002:a17:90b:4c47:: with SMTP id np7mr1873292pjb.140.1583909072197; Tue, 10 Mar 2020 23:44:32 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH 08/16] target/arm: Use SVEContLdSt in sve_ld1_r Date: Tue, 10 Mar 2020 23:44:12 -0700 Message-Id: <20200311064420.30606-9-richard.henderson@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200311064420.30606-1-richard.henderson@linaro.org> References: <20200311064420.30606-1-richard.henderson@linaro.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::1041 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, qemu-arm@nongnu.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @linaro.org) Content-Type: text/plain; charset="utf-8" First use of the new helper functions, so we can remove the unused markup. No longer need a scratch for user-only, as we completely probe the page set before reading; system mode still requires a scratch for MMIO. Signed-off-by: Richard Henderson Reviewed-by: Peter Maydell --- target/arm/sve_helper.c | 188 +++++++++++++++++++++------------------- 1 file changed, 97 insertions(+), 91 deletions(-) diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index 1397c2b634..b827900a4e 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -4227,9 +4227,9 @@ typedef struct { * final element on each page. Identify any single element that spans * the page boundary. Return true if there are any active elements. */ -static bool __attribute__((unused)) -sve_cont_ldst_elements(SVEContLdSt *info, target_ulong addr, uint64_t *vg, - intptr_t reg_max, int esz, int msize) +static bool sve_cont_ldst_elements(SVEContLdSt *info, target_ulong addr, + uint64_t *vg, intptr_t reg_max, + int esz, int msize) { const int esize =3D 1 << esz; const uint64_t pg_mask =3D pred_esz_masks[esz]; @@ -4319,10 +4319,9 @@ sve_cont_ldst_elements(SVEContLdSt *info, target_ulo= ng addr, uint64_t *vg, * Control the generation of page faults with @fault. Return false if * there is no work to do, which can only happen with @fault =3D=3D FAULT_= NO. */ -static bool __attribute__((unused)) -sve_cont_ldst_pages(SVEContLdSt *info, SVEContFault fault, CPUARMState *en= v, - target_ulong addr, MMUAccessType access_type, - uintptr_t retaddr) +static bool sve_cont_ldst_pages(SVEContLdSt *info, SVEContFault fault, + CPUARMState *env, target_ulong addr, + MMUAccessType access_type, uintptr_t retad= dr) { int mmu_idx =3D cpu_mmu_index(env, false); int mem_off =3D info->mem_off_first[0]; @@ -4394,109 +4393,116 @@ static inline bool test_host_page(void *host) /* * Common helper for all contiguous one-register predicated loads. */ -static void sve_ld1_r(CPUARMState *env, void *vg, const target_ulong addr, - uint32_t desc, const uintptr_t retaddr, - const int esz, const int msz, - sve_ldst1_host_fn *host_fn, - sve_ldst1_tlb_fn *tlb_fn) +static inline QEMU_ALWAYS_INLINE +void sve_ld1_r(CPUARMState *env, uint64_t *vg, const target_ulong addr, + uint32_t desc, const uintptr_t retaddr, + const int esz, const int msz, + sve_ldst1_host_fn *host_fn, + sve_ldst1_tlb_fn *tlb_fn) { - const TCGMemOpIdx oi =3D extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHI= FT); - const int mmu_idx =3D get_mmuidx(oi); const unsigned rd =3D extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT= , 5); void *vd =3D &env->vfp.zregs[rd]; - const int diffsz =3D esz - msz; const intptr_t reg_max =3D simd_oprsz(desc); - const intptr_t mem_max =3D reg_max >> diffsz; - ARMVectorReg scratch; + intptr_t reg_off, reg_last, mem_off; + SVEContLdSt info; void *host; - intptr_t split, reg_off, mem_off; + int flags; =20 - /* Find the first active element. */ - reg_off =3D find_next_active(vg, 0, reg_max, esz); - if (unlikely(reg_off =3D=3D reg_max)) { + /* Find the active elements. */ + if (!sve_cont_ldst_elements(&info, addr, vg, reg_max, esz, 1 << msz)) { /* The entire predicate was false; no load occurs. */ memset(vd, 0, reg_max); return; } - mem_off =3D reg_off >> diffsz; =20 - /* - * If the (remaining) load is entirely within a single page, then: - * For softmmu, and the tlb hits, then no faults will occur; - * For user-only, either the first load will fault or none will. - * We can thus perform the load directly to the destination and - * Vd will be unmodified on any exception path. - */ - split =3D max_for_page(addr, mem_off, mem_max); - if (likely(split =3D=3D mem_max)) { - host =3D tlb_vaddr_to_host(env, addr + mem_off, MMU_DATA_LOAD, mmu= _idx); - if (test_host_page(host)) { - intptr_t i =3D reg_off; - host -=3D mem_off; - do { - host_fn(vd, i, host + (i >> diffsz)); - i =3D find_next_active(vg, i + (1 << esz), reg_max, esz); - } while (i < reg_max); - /* After having taken any fault, zero leading inactive element= s. */ - swap_memzero(vd, reg_off); - return; - } - } + /* Probe the page(s). Exit with exception for any invalid page. */ + sve_cont_ldst_pages(&info, FAULT_ALL, env, addr, MMU_DATA_LOAD, retadd= r); =20 - /* - * Perform the predicated read into a temporary, thus ensuring - * if the load of the last element faults, Vd is not modified. - */ + flags =3D info.page[0].flags | info.page[1].flags; + if (unlikely(flags !=3D 0)) { #ifdef CONFIG_USER_ONLY - swap_memzero(&scratch, reg_off); - host =3D g2h(addr); - do { - host_fn(&scratch, reg_off, host + (reg_off >> diffsz)); - reg_off +=3D 1 << esz; - reg_off =3D find_next_active(vg, reg_off, reg_max, esz); - } while (reg_off < reg_max); + g_assert_not_reached(); #else - memset(&scratch, 0, reg_max); - goto start; - while (1) { - reg_off =3D find_next_active(vg, reg_off, reg_max, esz); - if (reg_off >=3D reg_max) { - break; - } - mem_off =3D reg_off >> diffsz; - split =3D max_for_page(addr, mem_off, mem_max); + /* + * At least one page includes MMIO (or watchpoints). + * Any bus operation can fail with cpu_transaction_failed, + * which for ARM will raise SyncExternal. Perform the load + * into scratch memory to preserve register state until the end. + */ + ARMVectorReg scratch; =20 - start: - if (split - mem_off >=3D (1 << msz)) { - /* At least one whole element on this page. */ - host =3D tlb_vaddr_to_host(env, addr + mem_off, - MMU_DATA_LOAD, mmu_idx); - if (host) { - host -=3D mem_off; - do { - host_fn(&scratch, reg_off, host + mem_off); - reg_off +=3D 1 << esz; - reg_off =3D find_next_active(vg, reg_off, reg_max, esz= ); - mem_off =3D reg_off >> diffsz; - } while (split - mem_off >=3D (1 << msz)); - continue; + memset(&scratch, 0, reg_max); + mem_off =3D info.mem_off_first[0]; + reg_off =3D info.reg_off_first[0]; + reg_last =3D info.reg_off_last[1]; + if (reg_last < 0) { + reg_last =3D info.reg_off_split; + if (reg_last < 0) { + reg_last =3D info.reg_off_last[0]; } } =20 - /* - * Perform one normal read. This may fault, longjmping out to the - * main loop in order to raise an exception. It may succeed, and - * as a side-effect load the TLB entry for the next round. Finall= y, - * in the extremely unlikely case we're performing this operation - * on I/O memory, it may succeed but not bring in the TLB entry. - * But even then we have still made forward progress. - */ - tlb_fn(env, &scratch, reg_off, addr + mem_off, retaddr); - reg_off +=3D 1 << esz; - } -#endif + do { + uint64_t pg =3D vg[reg_off >> 6]; + do { + if ((pg >> (reg_off & 63)) & 1) { + tlb_fn(env, &scratch, reg_off, addr + mem_off, retaddr= ); + } + reg_off +=3D 1 << esz; + mem_off +=3D 1 << msz; + } while (reg_off & 63); + } while (reg_off <=3D reg_last); =20 - memcpy(vd, &scratch, reg_max); + memcpy(vd, &scratch, reg_max); + return; +#endif + } + + /* The entire operation is in RAM, on valid pages. */ + + memset(vd, 0, reg_max); + mem_off =3D info.mem_off_first[0]; + reg_off =3D info.reg_off_first[0]; + reg_last =3D info.reg_off_last[0]; + host =3D info.page[0].host; + + while (reg_off <=3D reg_last) { + uint64_t pg =3D vg[reg_off >> 6]; + do { + if ((pg >> (reg_off & 63)) & 1) { + host_fn(vd, reg_off, host + mem_off); + } + reg_off +=3D 1 << esz; + mem_off +=3D 1 << msz; + } while (reg_off <=3D reg_last && (reg_off & 63)); + } + + /* + * Use the slow path to manage the cross-page misalignment. + * But we know this is RAM and cannot trap. + */ + mem_off =3D info.mem_off_split; + if (unlikely(mem_off >=3D 0)) { + tlb_fn(env, vd, info.reg_off_split, addr + mem_off, retaddr); + } + + mem_off =3D info.mem_off_first[1]; + if (unlikely(mem_off >=3D 0)) { + reg_off =3D info.reg_off_first[1]; + reg_last =3D info.reg_off_last[1]; + host =3D info.page[1].host; + + do { + uint64_t pg =3D vg[reg_off >> 6]; + do { + if ((pg >> (reg_off & 63)) & 1) { + host_fn(vd, reg_off, host + mem_off); + } + reg_off +=3D 1 << esz; + mem_off +=3D 1 << msz; + } while (reg_off & 63); + } while (reg_off <=3D reg_last); + } } =20 #define DO_LD1_1(NAME, ESZ) \ --=20 2.20.1 From nobody Mon Feb 9 21:08:28 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=linaro.org ARC-Seal: i=1; a=rsa-sha256; t=1583909414; cv=none; d=zohomail.com; s=zohoarc; b=OHivyLdgxBdBvjDm0irTSnmT6PxKpa02xydwNtUQKxLmqfBhN95BhEvB9bq2DG/ltKadz03ur6ZLS6CHj9LHdFKW3tDp7Chs0ELdoiGilfRBuAvNIYMGzpL2xhXKmhHAbwg/5MnacWqn/2Ua9LQI9m8cWxemRyJgumQshjvurq4= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1583909414; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=Tz6r3+bMnyO04HqqXq5aHLFFiU4Amy53tPfIpPhOcd0=; b=P/ab0kYxkPZwZOfZv9UuUXCK2rKB6vuEq57pcHR0UqQtNQsl9fmbDWeDyPJYhktKyguBnW85MgFyzWRj/Najo48ugsG9VCV3qj7XU7PLMNHpAsfYJpl1VCYskD/VUnCKtbpLXlj2ET64kSa92I0I4mURqVJjvhb+949r13X9gqo= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1583909414709332.9684001325155; Tue, 10 Mar 2020 23:50:14 -0700 (PDT) Received: from localhost ([::1]:46532 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jBvC1-0006DG-L4 for importer@patchew.org; Wed, 11 Mar 2020 02:50:13 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:59508) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jBv6b-0006F9-MV for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:39 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jBv6Z-0001xe-Q9 for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:37 -0400 Received: from mail-pl1-x643.google.com ([2607:f8b0:4864:20::643]:42707) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1jBv6Z-0001vs-Jf for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:35 -0400 Received: by mail-pl1-x643.google.com with SMTP id t3so613231plz.9 for ; Tue, 10 Mar 2020 23:44:34 -0700 (PDT) Received: from localhost.localdomain (97-126-123-70.tukw.qwest.net. [97.126.123.70]) by smtp.gmail.com with ESMTPSA id z3sm50137807pfz.155.2020.03.10.23.44.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 10 Mar 2020 23:44:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Tz6r3+bMnyO04HqqXq5aHLFFiU4Amy53tPfIpPhOcd0=; b=iY6j3nvVY5+1mQagFTmecA511o9b0ILObTCimvRrrl3YbKGbCuEWO9y0TAksf9STzJ Jhc005Jx+vL3VyAm9GUq08M8WxBGQX/pMAqbJeCLUlLFY5KNytNBFNqfyRehExCwvYgC 4PTaZs7w8Rxs/yonODh+eY3QRjdVBmaXaoocQgugj5bpyRKiANwoG6paPfVmEiFJbKeb mLtYEg0SLg2Kl8EL+Gz19pF1lAmdFLA/6pGW0MouAdPS0yXuyYClmLQa2Mjyl0/cgEah IML880+wuN3TmdARqkKLmKC7gbbYJvOdXuNNnUN9mRfWxOpd6sYXDcW67uqkXonBAJWQ CFgg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Tz6r3+bMnyO04HqqXq5aHLFFiU4Amy53tPfIpPhOcd0=; b=SvRecTS1Jm9+h/iSS2baDfsrki1W4DKIFMu+jDnXuP9aK+pNh9x9C1o2lgV7/Vr05c xotaku8K4rccIDgDJLWxVe+IFZq49Q2nqzsJGJRqv6j8iL+g7c+90Bm7sW4cgVjtuknv ehwD5oRZozGjG1MTPLBzCNM6oIyJ/GkBDHzwlY65PnnmhWD3KB3FsqCrPw7QG66jqIL6 iDXgkRuaSnZTwo6XtlLx5bqNgaGTlsIn/AFjnbq/ofwSINpE2p1RuxlXh4kCkVtl5vAO 8dfOE9OMcVr8GZjh1zabVx4uCy4qq/K7TZ+yI+dXgOJnPTaMwURBPkZSMF+480BmM3jf /OBw== X-Gm-Message-State: ANhLgQ2J/hf2cOnonC17a0Aw8YtGbHta0gv1rWSMdUynZKRxxFck4NI2 Hd852ffcYNYH2CQLb8ivbVau6f2+J/Q= X-Google-Smtp-Source: ADFU+vtbUuD/CUGb6wvvwD6jmbSIU78VhoSeyfEvrrkp/Qh/tGMvJCPfyW/FpF0PFwHpvpEWfjA0zg== X-Received: by 2002:a17:90b:3653:: with SMTP id nh19mr1896194pjb.41.1583909073613; Tue, 10 Mar 2020 23:44:33 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH 09/16] target/arm: Handle watchpoints in sve_ld1_r Date: Tue, 10 Mar 2020 23:44:13 -0700 Message-Id: <20200311064420.30606-10-richard.henderson@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200311064420.30606-1-richard.henderson@linaro.org> References: <20200311064420.30606-1-richard.henderson@linaro.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::643 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, qemu-arm@nongnu.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @linaro.org) Content-Type: text/plain; charset="utf-8" Handle all of the watchpoints for active elements all at once, before we've modified the vector register. This removes the TLB_WATCHPOINT bit from page[].flags, which means that we can use the normal fast path via RAM. Signed-off-by: Richard Henderson Reviewed-by: Peter Maydell --- target/arm/sve_helper.c | 70 ++++++++++++++++++++++++++++++++++++++++- 1 file changed, 69 insertions(+), 1 deletion(-) diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index b827900a4e..2c0e838ed3 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -4377,6 +4377,68 @@ static bool sve_cont_ldst_pages(SVEContLdSt *info, S= VEContFault fault, return have_work; } =20 +static void sve_cont_ldst_watchpoints(SVEContLdSt *info, CPUARMState *env, + uint64_t *vg, target_ulong addr, + int esize, int msize, int wp_access, + uintptr_t retaddr) +{ +#ifndef CONFIG_USER_ONLY + intptr_t mem_off, reg_off, reg_last; + int flags0 =3D info->page[0].flags; + int flags1 =3D info->page[1].flags; + + if (likely(!((flags0 | flags1) & TLB_WATCHPOINT))) { + return; + } + + /* Indicate that watchpoints are handled. */ + info->page[0].flags =3D flags0 & ~TLB_WATCHPOINT; + info->page[1].flags =3D flags1 & ~TLB_WATCHPOINT; + + if (flags0 & TLB_WATCHPOINT) { + mem_off =3D info->mem_off_first[0]; + reg_off =3D info->reg_off_first[0]; + reg_last =3D info->reg_off_last[0]; + + while (reg_off <=3D reg_last) { + uint64_t pg =3D vg[reg_off >> 6]; + do { + if ((pg >> (reg_off & 63)) & 1) { + cpu_check_watchpoint(env_cpu(env), addr + mem_off, msi= ze, + info->page[0].attrs, wp_access, r= etaddr); + } + reg_off +=3D esize; + mem_off +=3D msize; + } while (reg_off <=3D reg_last && (reg_off & 63)); + } + } + + mem_off =3D info->mem_off_split; + if (mem_off >=3D 0) { + cpu_check_watchpoint(env_cpu(env), addr + mem_off, msize, + info->page[0].attrs, wp_access, retaddr); + } + + mem_off =3D info->mem_off_first[1]; + if ((flags1 & TLB_WATCHPOINT) && mem_off >=3D 0) { + reg_off =3D info->reg_off_first[1]; + reg_last =3D info->reg_off_last[1]; + + do { + uint64_t pg =3D vg[reg_off >> 6]; + do { + if ((pg >> (reg_off & 63)) & 1) { + cpu_check_watchpoint(env_cpu(env), addr + mem_off, msi= ze, + info->page[1].attrs, wp_access, r= etaddr); + } + reg_off +=3D esize; + mem_off +=3D msize; + } while (reg_off & 63); + } while (reg_off <=3D reg_last); + } +#endif +} + /* * The result of tlb_vaddr_to_host for user-only is just g2h(x), * which is always non-null. Elide the useless test. @@ -4418,13 +4480,19 @@ void sve_ld1_r(CPUARMState *env, uint64_t *vg, cons= t target_ulong addr, /* Probe the page(s). Exit with exception for any invalid page. */ sve_cont_ldst_pages(&info, FAULT_ALL, env, addr, MMU_DATA_LOAD, retadd= r); =20 + /* Handle watchpoints for all active elements. */ + sve_cont_ldst_watchpoints(&info, env, vg, addr, 1 << esz, 1 << msz, + BP_MEM_READ, retaddr); + + /* TODO: MTE check. */ + flags =3D info.page[0].flags | info.page[1].flags; if (unlikely(flags !=3D 0)) { #ifdef CONFIG_USER_ONLY g_assert_not_reached(); #else /* - * At least one page includes MMIO (or watchpoints). + * At least one page includes MMIO. * Any bus operation can fail with cpu_transaction_failed, * which for ARM will raise SyncExternal. Perform the load * into scratch memory to preserve register state until the end. --=20 2.20.1 From nobody Mon Feb 9 21:08:28 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=linaro.org ARC-Seal: i=1; a=rsa-sha256; t=1583909692; cv=none; d=zohomail.com; s=zohoarc; b=HRG9WeIS61I0fCFtJ16NqhNy8515y8ZruATV6+HzDXDmpB4Kj4miO4fyUKYB4WpvYg+veU+ql1EqUNnlEp3n4he/XxhzWBqmZgFPfvPfBjFEtqNXLNu3Qr/CPfranqJLiKKL2yBqRzXTEngzsEIapk2Lbvq7Ot5NHZvHh2S/vmo= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1583909692; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=2YeIG7aIPNgxF4xn6eg+uOsdMNeYSZujzfaE66vwnTQ=; b=g5ZijdKicWG6BuoO6Po/+UdSJG+9bm4VA7Hld7zY5zUSCGT1e7UX0xf9o2CHiLAKvDyRyQ9/+iR23mD/3VJAklKZIr3kG6mSlEKuLPSh4uatS5g5I9BrzWcGtnYdoqvMJ4EUXaFhPGZLxZ3xfur/AdL+q92rdv+tfcbyi8ukO9k= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 158390969270460.97616384555511; Tue, 10 Mar 2020 23:54:52 -0700 (PDT) Received: from localhost ([::1]:46722 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jBvGV-0004vv-Gl for importer@patchew.org; Wed, 11 Mar 2020 02:54:51 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:59577) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jBv6e-0006MW-78 for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:42 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jBv6b-0001zw-Th for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:40 -0400 Received: from mail-pl1-x643.google.com ([2607:f8b0:4864:20::643]:33829) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1jBv6b-0001yF-J7 for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:37 -0400 Received: by mail-pl1-x643.google.com with SMTP id a23so630572plm.1 for ; Tue, 10 Mar 2020 23:44:36 -0700 (PDT) Received: from localhost.localdomain (97-126-123-70.tukw.qwest.net. [97.126.123.70]) by smtp.gmail.com with ESMTPSA id z3sm50137807pfz.155.2020.03.10.23.44.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 10 Mar 2020 23:44:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=2YeIG7aIPNgxF4xn6eg+uOsdMNeYSZujzfaE66vwnTQ=; b=TwxJZbePUuZD0UKRO0B1c7EH1QHO/USGqld4kccmFkAim3AJz0p7HyYReHA7N4A2LA D05AVktBBaDVZ0u46Ka52UYZCOnLeB7uElinU1MoK26iIBcL9ilGQZ9yut4iy94rEy4j dct4zCcqTlAfVz9RTOe4tiZTBZnI1NSkmPawCohA4FONJHBTemY8/Ld0nSp5jlwoOsXj iy+JeXjzXFuqqSa9m0n1dLpzNEei1hBmrkHU8ExlzuZkX2venoAM5ZfJzY0cVpPfxtCq zWPUDyBH2gWg7fjye0cZMcJOLKapsuAChlwHl4wHI90zW5vrAO5LT2SImMWqPhHYTBlk ye4Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=2YeIG7aIPNgxF4xn6eg+uOsdMNeYSZujzfaE66vwnTQ=; b=odFj4sgvoDmIG++7YeoMZwnPd3tqy67roup23MTsz7QMwqc6Uv61Os2ol3L3VcGGPq Y/MmaVFvPBpdcIHtXL/QR43vFrsbuyXPzkbmXb6fKzXaO70fT+Pt8uJ21XW4x7CCF2jA zyB/DnipfnJLd1NXZV+tbP40AlItsfigDqKpfVmtzihHk5yA6LlXrBv0uGg/tmYHchnQ 1z0fsJXnMGM7qf1KFy6xJ1EywgdqgQ3LhcE0tbUVHhy5+QYMahw/8awn8FPXnSPvurJ7 jjHsqpAonokiGY5bxHz45dnRWzWcO6P1Vhmx47yWO4rNYMfvZ9fNu7p2kKHPKVtgDiee oaEw== X-Gm-Message-State: ANhLgQ1kvryJwrLbXnUmH3higW3+/MQw67JQIrp9HO9fGSpNbRJ455lt ZRdau8EzV9uboWR3SCNIypoXsFhkoOU= X-Google-Smtp-Source: ADFU+vvk2PnRC/LFJlzyLl/r3xTgfinnc04ZRHV5T4rfYb1Hnno3vMrIcDc/hUrviPIM2D7colV+ow== X-Received: by 2002:a17:90b:1983:: with SMTP id mv3mr1910165pjb.86.1583909075176; Tue, 10 Mar 2020 23:44:35 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH 10/16] target/arm: Use SVEContLdSt for multi-register contiguous loads Date: Tue, 10 Mar 2020 23:44:14 -0700 Message-Id: <20200311064420.30606-11-richard.henderson@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200311064420.30606-1-richard.henderson@linaro.org> References: <20200311064420.30606-1-richard.henderson@linaro.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::643 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, qemu-arm@nongnu.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @linaro.org) Content-Type: text/plain; charset="utf-8" Signed-off-by: Richard Henderson Reviewed-by: Peter Maydell --- target/arm/sve_helper.c | 223 ++++++++++++++-------------------------- 1 file changed, 79 insertions(+), 144 deletions(-) diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index 2c0e838ed3..56407446eb 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -4453,27 +4453,28 @@ static inline bool test_host_page(void *host) } =20 /* - * Common helper for all contiguous one-register predicated loads. + * Common helper for all contiguous 1,2,3,4-register predicated stores. */ static inline QEMU_ALWAYS_INLINE -void sve_ld1_r(CPUARMState *env, uint64_t *vg, const target_ulong addr, +void sve_ldN_r(CPUARMState *env, uint64_t *vg, const target_ulong addr, uint32_t desc, const uintptr_t retaddr, - const int esz, const int msz, + const int esz, const int msz, const int N, sve_ldst1_host_fn *host_fn, sve_ldst1_tlb_fn *tlb_fn) { const unsigned rd =3D extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT= , 5); - void *vd =3D &env->vfp.zregs[rd]; const intptr_t reg_max =3D simd_oprsz(desc); intptr_t reg_off, reg_last, mem_off; SVEContLdSt info; void *host; - int flags; + int flags, i; =20 /* Find the active elements. */ - if (!sve_cont_ldst_elements(&info, addr, vg, reg_max, esz, 1 << msz)) { + if (!sve_cont_ldst_elements(&info, addr, vg, reg_max, esz, N << msz)) { /* The entire predicate was false; no load occurs. */ - memset(vd, 0, reg_max); + for (i =3D 0; i < N; ++i) { + memset(&env->vfp.zregs[(rd + i) & 31], 0, reg_max); + } return; } =20 @@ -4481,7 +4482,7 @@ void sve_ld1_r(CPUARMState *env, uint64_t *vg, const = target_ulong addr, sve_cont_ldst_pages(&info, FAULT_ALL, env, addr, MMU_DATA_LOAD, retadd= r); =20 /* Handle watchpoints for all active elements. */ - sve_cont_ldst_watchpoints(&info, env, vg, addr, 1 << esz, 1 << msz, + sve_cont_ldst_watchpoints(&info, env, vg, addr, 1 << esz, N << msz, BP_MEM_READ, retaddr); =20 /* TODO: MTE check. */ @@ -4497,9 +4498,8 @@ void sve_ld1_r(CPUARMState *env, uint64_t *vg, const = target_ulong addr, * which for ARM will raise SyncExternal. Perform the load * into scratch memory to preserve register state until the end. */ - ARMVectorReg scratch; + ARMVectorReg scratch[4] =3D { }; =20 - memset(&scratch, 0, reg_max); mem_off =3D info.mem_off_first[0]; reg_off =3D info.reg_off_first[0]; reg_last =3D info.reg_off_last[1]; @@ -4514,21 +4514,29 @@ void sve_ld1_r(CPUARMState *env, uint64_t *vg, cons= t target_ulong addr, uint64_t pg =3D vg[reg_off >> 6]; do { if ((pg >> (reg_off & 63)) & 1) { - tlb_fn(env, &scratch, reg_off, addr + mem_off, retaddr= ); + for (i =3D 0; i < N; ++i) { + tlb_fn(env, &scratch[i], reg_off, + addr + mem_off + (i << msz), retaddr); + } } reg_off +=3D 1 << esz; - mem_off +=3D 1 << msz; + mem_off +=3D N << msz; } while (reg_off & 63); } while (reg_off <=3D reg_last); =20 - memcpy(vd, &scratch, reg_max); + for (i =3D 0; i < N; ++i) { + memcpy(&env->vfp.zregs[(rd + i) & 31], &scratch[i], reg_max); + } return; #endif } =20 /* The entire operation is in RAM, on valid pages. */ =20 - memset(vd, 0, reg_max); + for (i =3D 0; i < N; ++i) { + memset(&env->vfp.zregs[(rd + i) & 31], 0, reg_max); + } + mem_off =3D info.mem_off_first[0]; reg_off =3D info.reg_off_first[0]; reg_last =3D info.reg_off_last[0]; @@ -4538,10 +4546,13 @@ void sve_ld1_r(CPUARMState *env, uint64_t *vg, cons= t target_ulong addr, uint64_t pg =3D vg[reg_off >> 6]; do { if ((pg >> (reg_off & 63)) & 1) { - host_fn(vd, reg_off, host + mem_off); + for (i =3D 0; i < N; ++i) { + host_fn(&env->vfp.zregs[(rd + i) & 31], reg_off, + host + mem_off + (i << msz)); + } } reg_off +=3D 1 << esz; - mem_off +=3D 1 << msz; + mem_off +=3D N << msz; } while (reg_off <=3D reg_last && (reg_off & 63)); } =20 @@ -4551,7 +4562,11 @@ void sve_ld1_r(CPUARMState *env, uint64_t *vg, const= target_ulong addr, */ mem_off =3D info.mem_off_split; if (unlikely(mem_off >=3D 0)) { - tlb_fn(env, vd, info.reg_off_split, addr + mem_off, retaddr); + reg_off =3D info.reg_off_split; + for (i =3D 0; i < N; ++i) { + tlb_fn(env, &env->vfp.zregs[(rd + i) & 31], reg_off, + addr + mem_off + (i << msz), retaddr); + } } =20 mem_off =3D info.mem_off_first[1]; @@ -4564,10 +4579,13 @@ void sve_ld1_r(CPUARMState *env, uint64_t *vg, cons= t target_ulong addr, uint64_t pg =3D vg[reg_off >> 6]; do { if ((pg >> (reg_off & 63)) & 1) { - host_fn(vd, reg_off, host + mem_off); + for (i =3D 0; i < N; ++i) { + host_fn(&env->vfp.zregs[(rd + i) & 31], reg_off, + host + mem_off + (i << msz)); + } } reg_off +=3D 1 << esz; - mem_off +=3D 1 << msz; + mem_off +=3D N << msz; } while (reg_off & 63); } while (reg_off <=3D reg_last); } @@ -4577,7 +4595,7 @@ void sve_ld1_r(CPUARMState *env, uint64_t *vg, const = target_ulong addr, void HELPER(sve_##NAME##_r)(CPUARMState *env, void *vg, \ target_ulong addr, uint32_t desc) \ { \ - sve_ld1_r(env, vg, addr, desc, GETPC(), ESZ, 0, \ + sve_ldN_r(env, vg, addr, desc, GETPC(), ESZ, MO_8, 1, \ sve_##NAME##_host, sve_##NAME##_tlb); \ } =20 @@ -4585,159 +4603,76 @@ void HELPER(sve_##NAME##_r)(CPUARMState *env, void= *vg, \ void HELPER(sve_##NAME##_le_r)(CPUARMState *env, void *vg, \ target_ulong addr, uint32_t desc) \ { \ - sve_ld1_r(env, vg, addr, desc, GETPC(), ESZ, MSZ, \ + sve_ldN_r(env, vg, addr, desc, GETPC(), ESZ, MSZ, 1, \ sve_##NAME##_le_host, sve_##NAME##_le_tlb); \ } \ void HELPER(sve_##NAME##_be_r)(CPUARMState *env, void *vg, \ target_ulong addr, uint32_t desc) \ { \ - sve_ld1_r(env, vg, addr, desc, GETPC(), ESZ, MSZ, \ + sve_ldN_r(env, vg, addr, desc, GETPC(), ESZ, MSZ, 1, \ sve_##NAME##_be_host, sve_##NAME##_be_tlb); \ } =20 -DO_LD1_1(ld1bb, 0) -DO_LD1_1(ld1bhu, 1) -DO_LD1_1(ld1bhs, 1) -DO_LD1_1(ld1bsu, 2) -DO_LD1_1(ld1bss, 2) -DO_LD1_1(ld1bdu, 3) -DO_LD1_1(ld1bds, 3) +DO_LD1_1(ld1bb, MO_8) +DO_LD1_1(ld1bhu, MO_16) +DO_LD1_1(ld1bhs, MO_16) +DO_LD1_1(ld1bsu, MO_32) +DO_LD1_1(ld1bss, MO_32) +DO_LD1_1(ld1bdu, MO_64) +DO_LD1_1(ld1bds, MO_64) =20 -DO_LD1_2(ld1hh, 1, 1) -DO_LD1_2(ld1hsu, 2, 1) -DO_LD1_2(ld1hss, 2, 1) -DO_LD1_2(ld1hdu, 3, 1) -DO_LD1_2(ld1hds, 3, 1) +DO_LD1_2(ld1hh, MO_16, MO_16) +DO_LD1_2(ld1hsu, MO_32, MO_16) +DO_LD1_2(ld1hss, MO_32, MO_16) +DO_LD1_2(ld1hdu, MO_64, MO_16) +DO_LD1_2(ld1hds, MO_64, MO_16) =20 -DO_LD1_2(ld1ss, 2, 2) -DO_LD1_2(ld1sdu, 3, 2) -DO_LD1_2(ld1sds, 3, 2) +DO_LD1_2(ld1ss, MO_32, MO_32) +DO_LD1_2(ld1sdu, MO_64, MO_32) +DO_LD1_2(ld1sds, MO_64, MO_32) =20 -DO_LD1_2(ld1dd, 3, 3) +DO_LD1_2(ld1dd, MO_64, MO_64) =20 #undef DO_LD1_1 #undef DO_LD1_2 =20 -/* - * Common helpers for all contiguous 2,3,4-register predicated loads. - */ -static void sve_ld2_r(CPUARMState *env, void *vg, target_ulong addr, - uint32_t desc, int size, uintptr_t ra, - sve_ldst1_tlb_fn *tlb_fn) -{ - const unsigned rd =3D extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT= , 5); - intptr_t i, oprsz =3D simd_oprsz(desc); - ARMVectorReg scratch[2] =3D { }; - - for (i =3D 0; i < oprsz; ) { - uint16_t pg =3D *(uint16_t *)(vg + H1_2(i >> 3)); - do { - if (pg & 1) { - tlb_fn(env, &scratch[0], i, addr, ra); - tlb_fn(env, &scratch[1], i, addr + size, ra); - } - i +=3D size, pg >>=3D size; - addr +=3D 2 * size; - } while (i & 15); - } - - /* Wait until all exceptions have been raised to write back. */ - memcpy(&env->vfp.zregs[rd], &scratch[0], oprsz); - memcpy(&env->vfp.zregs[(rd + 1) & 31], &scratch[1], oprsz); -} - -static void sve_ld3_r(CPUARMState *env, void *vg, target_ulong addr, - uint32_t desc, int size, uintptr_t ra, - sve_ldst1_tlb_fn *tlb_fn) -{ - const unsigned rd =3D extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT= , 5); - intptr_t i, oprsz =3D simd_oprsz(desc); - ARMVectorReg scratch[3] =3D { }; - - for (i =3D 0; i < oprsz; ) { - uint16_t pg =3D *(uint16_t *)(vg + H1_2(i >> 3)); - do { - if (pg & 1) { - tlb_fn(env, &scratch[0], i, addr, ra); - tlb_fn(env, &scratch[1], i, addr + size, ra); - tlb_fn(env, &scratch[2], i, addr + 2 * size, ra); - } - i +=3D size, pg >>=3D size; - addr +=3D 3 * size; - } while (i & 15); - } - - /* Wait until all exceptions have been raised to write back. */ - memcpy(&env->vfp.zregs[rd], &scratch[0], oprsz); - memcpy(&env->vfp.zregs[(rd + 1) & 31], &scratch[1], oprsz); - memcpy(&env->vfp.zregs[(rd + 2) & 31], &scratch[2], oprsz); -} - -static void sve_ld4_r(CPUARMState *env, void *vg, target_ulong addr, - uint32_t desc, int size, uintptr_t ra, - sve_ldst1_tlb_fn *tlb_fn) -{ - const unsigned rd =3D extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT= , 5); - intptr_t i, oprsz =3D simd_oprsz(desc); - ARMVectorReg scratch[4] =3D { }; - - for (i =3D 0; i < oprsz; ) { - uint16_t pg =3D *(uint16_t *)(vg + H1_2(i >> 3)); - do { - if (pg & 1) { - tlb_fn(env, &scratch[0], i, addr, ra); - tlb_fn(env, &scratch[1], i, addr + size, ra); - tlb_fn(env, &scratch[2], i, addr + 2 * size, ra); - tlb_fn(env, &scratch[3], i, addr + 3 * size, ra); - } - i +=3D size, pg >>=3D size; - addr +=3D 4 * size; - } while (i & 15); - } - - /* Wait until all exceptions have been raised to write back. */ - memcpy(&env->vfp.zregs[rd], &scratch[0], oprsz); - memcpy(&env->vfp.zregs[(rd + 1) & 31], &scratch[1], oprsz); - memcpy(&env->vfp.zregs[(rd + 2) & 31], &scratch[2], oprsz); - memcpy(&env->vfp.zregs[(rd + 3) & 31], &scratch[3], oprsz); -} - #define DO_LDN_1(N) \ -void QEMU_FLATTEN HELPER(sve_ld##N##bb_r) \ - (CPUARMState *env, void *vg, target_ulong addr, uint32_t desc) \ -{ \ - sve_ld##N##_r(env, vg, addr, desc, 1, GETPC(), sve_ld1bb_tlb); \ +void HELPER(sve_ld##N##bb_r)(CPUARMState *env, void *vg, \ + target_ulong addr, uint32_t desc) \ +{ \ + sve_ldN_r(env, vg, addr, desc, GETPC(), MO_8, MO_8, N, \ + sve_ld1bb_host, sve_ld1bb_tlb); \ } =20 -#define DO_LDN_2(N, SUFF, SIZE) \ -void QEMU_FLATTEN HELPER(sve_ld##N##SUFF##_le_r) \ - (CPUARMState *env, void *vg, target_ulong addr, uint32_t desc) \ +#define DO_LDN_2(N, SUFF, ESZ) \ +void HELPER(sve_ld##N##SUFF##_le_r)(CPUARMState *env, void *vg, \ + target_ulong addr, uint32_t desc) \ { \ - sve_ld##N##_r(env, vg, addr, desc, SIZE, GETPC(), \ - sve_ld1##SUFF##_le_tlb); \ + sve_ldN_r(env, vg, addr, desc, GETPC(), ESZ, ESZ, N, \ + sve_ld1##SUFF##_le_host, sve_ld1##SUFF##_le_tlb); \ } \ -void QEMU_FLATTEN HELPER(sve_ld##N##SUFF##_be_r) \ - (CPUARMState *env, void *vg, target_ulong addr, uint32_t desc) \ +void HELPER(sve_ld##N##SUFF##_be_r)(CPUARMState *env, void *vg, \ + target_ulong addr, uint32_t desc) \ { \ - sve_ld##N##_r(env, vg, addr, desc, SIZE, GETPC(), \ - sve_ld1##SUFF##_be_tlb); \ + sve_ldN_r(env, vg, addr, desc, GETPC(), ESZ, ESZ, N, \ + sve_ld1##SUFF##_be_host, sve_ld1##SUFF##_be_tlb); \ } =20 DO_LDN_1(2) DO_LDN_1(3) DO_LDN_1(4) =20 -DO_LDN_2(2, hh, 2) -DO_LDN_2(3, hh, 2) -DO_LDN_2(4, hh, 2) +DO_LDN_2(2, hh, MO_16) +DO_LDN_2(3, hh, MO_16) +DO_LDN_2(4, hh, MO_16) =20 -DO_LDN_2(2, ss, 4) -DO_LDN_2(3, ss, 4) -DO_LDN_2(4, ss, 4) +DO_LDN_2(2, ss, MO_32) +DO_LDN_2(3, ss, MO_32) +DO_LDN_2(4, ss, MO_32) =20 -DO_LDN_2(2, dd, 8) -DO_LDN_2(3, dd, 8) -DO_LDN_2(4, dd, 8) +DO_LDN_2(2, dd, MO_64) +DO_LDN_2(3, dd, MO_64) +DO_LDN_2(4, dd, MO_64) =20 #undef DO_LDN_1 #undef DO_LDN_2 --=20 2.20.1 From nobody Mon Feb 9 21:08:28 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=linaro.org ARC-Seal: i=1; a=rsa-sha256; t=1583909795; cv=none; d=zohomail.com; s=zohoarc; b=d+Di2qkubbnmQDncwXY6aRJnzSzhDf4NktemXuveUJBMm8OE6YuBjt+kVKp3L/b2lbJ2tbLRFM0G5WUzcAJzUL00hM8vX5lUElUuRsfNGAHqFCcPD5S8YNfCkxAsuNmQlx256He66JWDH6JiVJlVv8Mp2gI2DWU+idrwSqTrWlg= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1583909795; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=K+5/whU+/G/MKfzhK6hriD90MgA0aJH5jQ1gy3fK8ao=; b=SDHajZboIHBRzGr7BMT5FxXBGZLeLrdQB+N5Aj3qgzTATHLBQerz5VRdET8GumNaPSRZjLoL81DkEYrb9ENpFDSj7X83WykYsMQ0zIYI+avkgkMee07yQBAxJTnc0kLGrB8S5CJmgc5U4EwNNnKSQxc8qEcGjdCHPgnB46w49u0= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1583909795168176.61152462832365; Tue, 10 Mar 2020 23:56:35 -0700 (PDT) Received: from localhost ([::1]:46888 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jBvIA-0008KP-1p for importer@patchew.org; Wed, 11 Mar 2020 02:56:34 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:59606) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jBv6f-0006PR-4w for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:43 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jBv6c-00020b-H5 for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:41 -0400 Received: from mail-pl1-x644.google.com ([2607:f8b0:4864:20::644]:42708) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1jBv6c-0001ze-8l for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:38 -0400 Received: by mail-pl1-x644.google.com with SMTP id t3so613331plz.9 for ; Tue, 10 Mar 2020 23:44:38 -0700 (PDT) Received: from localhost.localdomain (97-126-123-70.tukw.qwest.net. [97.126.123.70]) by smtp.gmail.com with ESMTPSA id z3sm50137807pfz.155.2020.03.10.23.44.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 10 Mar 2020 23:44:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=K+5/whU+/G/MKfzhK6hriD90MgA0aJH5jQ1gy3fK8ao=; b=o03UG8AUyo4rZxXQeYENCuwLQNV+8PdV6giiGk8VNWEQRkWsumInVv46Ovfqj68vp/ u8suePTXGneG2SPtuEKGeuDWZo6dizN+aKwLFDUw0Lx6OiuKCx4JbWNkFJFyteQLVPEm P+JxEzz+X7z9SRziSAxAZ5M8GSSyxmL7JylZVbNtJdNVMRJ8lPok77NhnD+qaGi4f1hJ 3xT2MH7wKuCu4J0zd5/wXCU07cEo6htpyLI1RTqINVgHLmWKelp9bd6c2eHTA15978b7 1nqSnPi71sAjbmt2JJRcjymFp04gOjtx45I22ciPbweYz+zVXhmyFD28EoAWW4dTfywG kqTg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=K+5/whU+/G/MKfzhK6hriD90MgA0aJH5jQ1gy3fK8ao=; b=riCTDmU21138jWh17Qn7s9LcyZD7lp1Xnfev2W0m31KsKIMnqD8X0yuQmIav1XnWnG 1ZwXofFTHZNptTDqi3wQW3J5bjH4GejqIrRPD8fRNn4xvXZfx6V21zGeCaAv9VoFAPZJ MHIChX/9n0c/MxfewQLjO/GN56iChH0polPvGv6MOB9c8XPUfRVijTaMDSwS0HwJ9SyS 1VoPRD4IRXWdXrN80XX/x6TGZ9oul3YV8x0NkEXxZkVdKHYrRNMG4QfVczfF5WMzCPhv nsjFfH0avguf16yC9VeBNmMtHGFQSTcjrFPEFRCb7aSB7s49KWTo07PfrukB2jwfBtn0 zSew== X-Gm-Message-State: ANhLgQ0DrxtCrourco8s+4PTAklbLSXp8cA9gzycxFBrHyqLaxpVJDJ5 kVCVvHh/jpU6aQTK25hekINph4Gc+us= X-Google-Smtp-Source: ADFU+vv+z/sSoKImCWSuRXsuPtGx675qamgwSoysveznDUmpsomnKJIKAwlHAex/f3S8tPwmhTGkFw== X-Received: by 2002:a17:90b:352:: with SMTP id fh18mr1896827pjb.168.1583909076323; Tue, 10 Mar 2020 23:44:36 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH 11/16] target/arm: Update contiguous first-fault and no-fault loads Date: Tue, 10 Mar 2020 23:44:15 -0700 Message-Id: <20200311064420.30606-12-richard.henderson@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200311064420.30606-1-richard.henderson@linaro.org> References: <20200311064420.30606-1-richard.henderson@linaro.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::644 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, qemu-arm@nongnu.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @linaro.org) Content-Type: text/plain; charset="utf-8" With sve_cont_ldst_pages, the differences between first-fault and no-fault are minimal, so unify the routines. With cpu_probe_watchpoint, we are able to make progress through pages with TLB_WATCHPOINT set when the watchpoint does not actually fire. Signed-off-by: Richard Henderson --- target/arm/sve_helper.c | 340 ++++++++++++++++++---------------------- 1 file changed, 156 insertions(+), 184 deletions(-) diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index 56407446eb..9198e1612a 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -4126,18 +4126,6 @@ static intptr_t find_next_active(uint64_t *vg, intpt= r_t reg_off, return reg_off; } =20 -/* - * Return the maximum offset <=3D @mem_max which is still within the page - * referenced by @base + @mem_off. - */ -static intptr_t max_for_page(target_ulong base, intptr_t mem_off, - intptr_t mem_max) -{ - target_ulong addr =3D base + mem_off; - intptr_t split =3D -(intptr_t)(addr | TARGET_PAGE_MASK); - return MIN(split, mem_max - mem_off) + mem_off; -} - /* * Resolve the guest virtual address to info->host and info->flags. * If @nofault, return false if the page is invalid, otherwise @@ -4439,19 +4427,6 @@ static void sve_cont_ldst_watchpoints(SVEContLdSt *i= nfo, CPUARMState *env, #endif } =20 -/* - * The result of tlb_vaddr_to_host for user-only is just g2h(x), - * which is always non-null. Elide the useless test. - */ -static inline bool test_host_page(void *host) -{ -#ifdef CONFIG_USER_ONLY - return true; -#else - return likely(host !=3D NULL); -#endif -} - /* * Common helper for all contiguous 1,2,3,4-register predicated stores. */ @@ -4709,167 +4684,161 @@ static void record_fault(CPUARMState *env, uintpt= r_t i, uintptr_t oprsz) } =20 /* - * Common helper for all contiguous first-fault loads. + * Common helper for all contiguous no-fault and first-fault loads. */ -static void sve_ldff1_r(CPUARMState *env, void *vg, const target_ulong add= r, - uint32_t desc, const uintptr_t retaddr, - const int esz, const int msz, - sve_ldst1_host_fn *host_fn, - sve_ldst1_tlb_fn *tlb_fn) +static inline QEMU_ALWAYS_INLINE +void sve_ldnfff1_r(CPUARMState *env, void *vg, const target_ulong addr, + uint32_t desc, const uintptr_t retaddr, + const int esz, const int msz, const SVEContFault fault, + sve_ldst1_host_fn *host_fn, + sve_ldst1_tlb_fn *tlb_fn) { - const TCGMemOpIdx oi =3D extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHI= FT); - const int mmu_idx =3D get_mmuidx(oi); const unsigned rd =3D extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT= , 5); void *vd =3D &env->vfp.zregs[rd]; - const int diffsz =3D esz - msz; const intptr_t reg_max =3D simd_oprsz(desc); - const intptr_t mem_max =3D reg_max >> diffsz; - intptr_t split, reg_off, mem_off, i; + intptr_t reg_off, mem_off, reg_last; + SVEContLdSt info; + int flags; void *host; =20 - /* Skip to the first active element. */ - reg_off =3D find_next_active(vg, 0, reg_max, esz); - if (unlikely(reg_off =3D=3D reg_max)) { + /* Find the active elements. */ + if (!sve_cont_ldst_elements(&info, addr, vg, reg_max, esz, 1 << msz)) { /* The entire predicate was false; no load occurs. */ memset(vd, 0, reg_max); return; } - mem_off =3D reg_off >> diffsz; + reg_off =3D info.reg_off_first[0]; =20 - /* - * If the (remaining) load is entirely within a single page, then: - * For softmmu, and the tlb hits, then no faults will occur; - * For user-only, either the first load will fault or none will. - * We can thus perform the load directly to the destination and - * Vd will be unmodified on any exception path. - */ - split =3D max_for_page(addr, mem_off, mem_max); - if (likely(split =3D=3D mem_max)) { - host =3D tlb_vaddr_to_host(env, addr + mem_off, MMU_DATA_LOAD, mmu= _idx); - if (test_host_page(host)) { - i =3D reg_off; - host -=3D mem_off; - do { - host_fn(vd, i, host + (i >> diffsz)); - i =3D find_next_active(vg, i + (1 << esz), reg_max, esz); - } while (i < reg_max); - /* After any fault, zero any leading inactive elements. */ + /* Probe the page(s). */ + if (!sve_cont_ldst_pages(&info, fault, env, addr, MMU_DATA_LOAD, retad= dr)) { + /* Fault on first element. */ + tcg_debug_assert(fault =3D=3D FAULT_NO); + memset(vd, 0, reg_max); + goto do_fault; + } + + mem_off =3D info.mem_off_first[0]; + flags =3D info.page[0].flags; + + if (fault =3D=3D FAULT_FIRST) { + /* + * Special handling of the first active element, + * if it crosses a page boundary or is MMIO. + */ + bool is_split =3D mem_off =3D=3D info.mem_off_split; + /* TODO: MTE check. */ + if (unlikely(flags !=3D 0) || unlikely(is_split)) { + /* + * Use the slow path for cross-page handling. + * Might trap for MMIO or watchpoints. + */ + tlb_fn(env, vd, reg_off, addr + mem_off, retaddr); + + /* After any fault, zero the other elements. */ swap_memzero(vd, reg_off); - return; + reg_off +=3D 1 << esz; + mem_off +=3D 1 << msz; + swap_memzero(vd + reg_off, reg_max - reg_off); + + if (is_split) { + goto second_page; + } + } else { + memset(vd, 0, reg_max); + } + } else { + memset(vd, 0, reg_max); + if (unlikely(mem_off =3D=3D info.mem_off_split)) { + /* The first active element crosses a page boundary. */ + flags |=3D info.page[1].flags; + if (unlikely(flags & TLB_MMIO)) { + /* Some page is MMIO, see below. */ + goto do_fault; + } + if (unlikely(flags & TLB_WATCHPOINT) && + cpu_probe_watchpoint(env_cpu(env), addr + mem_off, + 1 << msz, BP_MEM_READ)) { + /* Watchpoint hit, see below. */ + goto do_fault; + } + /* TODO: MTE check. */ + /* + * Use the slow path for cross-page handling. + * This is RAM, without a watchpoint, and will not trap. + */ + tlb_fn(env, vd, reg_off, addr + mem_off, retaddr); + goto second_page; } } =20 /* - * Perform one normal read, which will fault or not. - * But it is likely to bring the page into the tlb. + * From this point on, all memory operations are MemSingleNF. + * + * Per the MemSingleNF pseudocode, a no-fault load from Device memory + * must not actually hit the bus -- it returns (UNKNOWN, FAULT) instea= d. + * If you map non-RAM with Normal memory attributes and do a NF + * load then it should access the bus -- but doing so is illegal. + * + * While we do not have access to the memory attributes from the PTE + * to tell Device memory from Normal memory, we can validly assume that + * non-RAM has been mapped as Device memory. Thus we indicate fault + * on all MMIO. + * + * Similarly, CPU_BP breakpoints would raise exceptions, and so + * return (UNKNOWN, FAULT). For simplicity, we consider gdb and + * architectural breakpoints the same. */ - tlb_fn(env, vd, reg_off, addr + mem_off, retaddr); + if (unlikely(flags & TLB_MMIO)) { + goto do_fault; + } =20 - /* After any fault, zero any leading predicated false elts. */ - swap_memzero(vd, reg_off); - mem_off +=3D 1 << msz; - reg_off +=3D 1 << esz; + reg_last =3D info.reg_off_last[0]; + host =3D info.page[0].host; =20 - /* Try again to read the balance of the page. */ - split =3D max_for_page(addr, mem_off - 1, mem_max); - if (split >=3D (1 << msz)) { - host =3D tlb_vaddr_to_host(env, addr + mem_off, MMU_DATA_LOAD, mmu= _idx); - if (host) { - host -=3D mem_off; - do { + do { + uint64_t pg =3D *(uint64_t *)(vg + (reg_off >> 3)); + do { + if ((pg >> (reg_off & 63)) & 1) { + if (unlikely(flags & TLB_WATCHPOINT) && + cpu_probe_watchpoint(env_cpu(env), addr + mem_off, + 1 << msz, BP_MEM_READ)) { + goto do_fault; + } + /* TODO: MTE check. */ host_fn(vd, reg_off, host + mem_off); - reg_off +=3D 1 << esz; - reg_off =3D find_next_active(vg, reg_off, reg_max, esz); - mem_off =3D reg_off >> diffsz; - } while (split - mem_off >=3D (1 << msz)); - } - } - - record_fault(env, reg_off, reg_max); -} - -/* - * Common helper for all contiguous no-fault loads. - */ -static void sve_ldnf1_r(CPUARMState *env, void *vg, const target_ulong add= r, - uint32_t desc, const int esz, const int msz, - sve_ldst1_host_fn *host_fn) -{ - const unsigned rd =3D extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT= , 5); - void *vd =3D &env->vfp.zregs[rd]; - const int diffsz =3D esz - msz; - const intptr_t reg_max =3D simd_oprsz(desc); - const intptr_t mem_max =3D reg_max >> diffsz; - const int mmu_idx =3D cpu_mmu_index(env, false); - intptr_t split, reg_off, mem_off; - void *host; - -#ifdef CONFIG_USER_ONLY - host =3D tlb_vaddr_to_host(env, addr, MMU_DATA_LOAD, mmu_idx); - if (likely(page_check_range(addr, mem_max, PAGE_READ) =3D=3D 0)) { - /* The entire operation is valid and will not fault. */ - reg_off =3D 0; - do { - mem_off =3D reg_off >> diffsz; - host_fn(vd, reg_off, host + mem_off); + } reg_off +=3D 1 << esz; - reg_off =3D find_next_active(vg, reg_off, reg_max, esz); - } while (reg_off < reg_max); - return; - } -#endif + mem_off +=3D 1 << msz; + } while (reg_off <=3D reg_last && (reg_off & 63)); + } while (reg_off <=3D reg_last); =20 - /* There will be no fault, so we may modify in advance. */ - memset(vd, 0, reg_max); - - /* Skip to the first active element. */ - reg_off =3D find_next_active(vg, 0, reg_max, esz); - if (unlikely(reg_off =3D=3D reg_max)) { - /* The entire predicate was false; no load occurs. */ - return; - } - mem_off =3D reg_off >> diffsz; - -#ifdef CONFIG_USER_ONLY - if (page_check_range(addr + mem_off, 1 << msz, PAGE_READ) =3D=3D 0) { - /* At least one load is valid; take the rest of the page. */ - split =3D max_for_page(addr, mem_off + (1 << msz) - 1, mem_max); - do { - host_fn(vd, reg_off, host + mem_off); - reg_off +=3D 1 << esz; - reg_off =3D find_next_active(vg, reg_off, reg_max, esz); - mem_off =3D reg_off >> diffsz; - } while (split - mem_off >=3D (1 << msz)); - } -#else /* - * If the address is not in the TLB, we have no way to bring the - * entry into the TLB without also risking a fault. Note that - * the corollary is that we never load from an address not in RAM. - * - * This last is out of spec, in a weird corner case. - * Per the MemNF/MemSingleNF pseudocode, a NF load from Device memory - * must not actually hit the bus -- it returns UNKNOWN data instead. - * But if you map non-RAM with Normal memory attributes and do a NF - * load then it should access the bus. (Nobody ought actually do this - * in the real world, obviously.) - * - * Then there are the annoying special cases with watchpoints... - * TODO: Add a form of non-faulting loads using cc->tlb_fill(probe=3Dt= rue). + * MemSingleNF is allowed to fail for any reason. We have special + * code above to handle the first element crossing a page boundary. + * As an implementation choice, decline to handle a cross-page element + * in any other position. */ - host =3D tlb_vaddr_to_host(env, addr + mem_off, MMU_DATA_LOAD, mmu_idx= ); - split =3D max_for_page(addr, mem_off, mem_max); - if (host && split >=3D (1 << msz)) { - host -=3D mem_off; - do { - host_fn(vd, reg_off, host + mem_off); - reg_off +=3D 1 << esz; - reg_off =3D find_next_active(vg, reg_off, reg_max, esz); - mem_off =3D reg_off >> diffsz; - } while (split - mem_off >=3D (1 << msz)); + reg_off =3D info.reg_off_split; + if (reg_off >=3D 0) { + goto do_fault; } -#endif =20 + second_page: + reg_off =3D info.reg_off_first[1]; + if (likely(reg_off < 0)) { + /* No active elements on the second page. All done. */ + return; + } + + /* + * MemSingleNF is allowed to fail for any reason. As an implementation + * choice, decline to handle elements on the second page. This should + * be low frequency as the guest walks through memory -- the next + * iteration of the guest's loop should be aligned on the page boundar= y, + * and then all following iterations will stay aligned. + */ + + do_fault: record_fault(env, reg_off, reg_max); } =20 @@ -4877,58 +4846,61 @@ static void sve_ldnf1_r(CPUARMState *env, void *vg,= const target_ulong addr, void HELPER(sve_ldff1##PART##_r)(CPUARMState *env, void *vg, \ target_ulong addr, uint32_t desc) \ { \ - sve_ldff1_r(env, vg, addr, desc, GETPC(), ESZ, 0, \ - sve_ld1##PART##_host, sve_ld1##PART##_tlb); \ + sve_ldnfff1_r(env, vg, addr, desc, GETPC(), ESZ, MO_8, FAULT_FIRST, \ + sve_ld1##PART##_host, sve_ld1##PART##_tlb); \ } \ void HELPER(sve_ldnf1##PART##_r)(CPUARMState *env, void *vg, \ target_ulong addr, uint32_t desc) \ { \ - sve_ldnf1_r(env, vg, addr, desc, ESZ, 0, sve_ld1##PART##_host); \ + sve_ldnfff1_r(env, vg, addr, desc, GETPC(), ESZ, MO_8, FAULT_NO, \ + sve_ld1##PART##_host, sve_ld1##PART##_tlb); \ } =20 #define DO_LDFF1_LDNF1_2(PART, ESZ, MSZ) \ void HELPER(sve_ldff1##PART##_le_r)(CPUARMState *env, void *vg, \ target_ulong addr, uint32_t desc) \ { \ - sve_ldff1_r(env, vg, addr, desc, GETPC(), ESZ, MSZ, \ - sve_ld1##PART##_le_host, sve_ld1##PART##_le_tlb); \ + sve_ldnfff1_r(env, vg, addr, desc, GETPC(), ESZ, MSZ, FAULT_FIRST, \ + sve_ld1##PART##_le_host, sve_ld1##PART##_le_tlb); \ } \ void HELPER(sve_ldnf1##PART##_le_r)(CPUARMState *env, void *vg, \ target_ulong addr, uint32_t desc) \ { \ - sve_ldnf1_r(env, vg, addr, desc, ESZ, MSZ, sve_ld1##PART##_le_host); \ + sve_ldnfff1_r(env, vg, addr, desc, GETPC(), ESZ, MSZ, FAULT_NO, \ + sve_ld1##PART##_le_host, sve_ld1##PART##_le_tlb); \ } \ void HELPER(sve_ldff1##PART##_be_r)(CPUARMState *env, void *vg, \ target_ulong addr, uint32_t desc) \ { \ - sve_ldff1_r(env, vg, addr, desc, GETPC(), ESZ, MSZ, \ - sve_ld1##PART##_be_host, sve_ld1##PART##_be_tlb); \ + sve_ldnfff1_r(env, vg, addr, desc, GETPC(), ESZ, MSZ, FAULT_FIRST, \ + sve_ld1##PART##_be_host, sve_ld1##PART##_be_tlb); \ } \ void HELPER(sve_ldnf1##PART##_be_r)(CPUARMState *env, void *vg, \ target_ulong addr, uint32_t desc) \ { \ - sve_ldnf1_r(env, vg, addr, desc, ESZ, MSZ, sve_ld1##PART##_be_host); \ + sve_ldnfff1_r(env, vg, addr, desc, GETPC(), ESZ, MSZ, FAULT_NO, \ + sve_ld1##PART##_be_host, sve_ld1##PART##_be_tlb); \ } =20 -DO_LDFF1_LDNF1_1(bb, 0) -DO_LDFF1_LDNF1_1(bhu, 1) -DO_LDFF1_LDNF1_1(bhs, 1) -DO_LDFF1_LDNF1_1(bsu, 2) -DO_LDFF1_LDNF1_1(bss, 2) -DO_LDFF1_LDNF1_1(bdu, 3) -DO_LDFF1_LDNF1_1(bds, 3) +DO_LDFF1_LDNF1_1(bb, MO_8) +DO_LDFF1_LDNF1_1(bhu, MO_16) +DO_LDFF1_LDNF1_1(bhs, MO_16) +DO_LDFF1_LDNF1_1(bsu, MO_32) +DO_LDFF1_LDNF1_1(bss, MO_32) +DO_LDFF1_LDNF1_1(bdu, MO_64) +DO_LDFF1_LDNF1_1(bds, MO_64) =20 -DO_LDFF1_LDNF1_2(hh, 1, 1) -DO_LDFF1_LDNF1_2(hsu, 2, 1) -DO_LDFF1_LDNF1_2(hss, 2, 1) -DO_LDFF1_LDNF1_2(hdu, 3, 1) -DO_LDFF1_LDNF1_2(hds, 3, 1) +DO_LDFF1_LDNF1_2(hh, MO_16, MO_16) +DO_LDFF1_LDNF1_2(hsu, MO_32, MO_16) +DO_LDFF1_LDNF1_2(hss, MO_32, MO_16) +DO_LDFF1_LDNF1_2(hdu, MO_64, MO_16) +DO_LDFF1_LDNF1_2(hds, MO_64, MO_16) =20 -DO_LDFF1_LDNF1_2(ss, 2, 2) -DO_LDFF1_LDNF1_2(sdu, 3, 2) -DO_LDFF1_LDNF1_2(sds, 3, 2) +DO_LDFF1_LDNF1_2(ss, MO_32, MO_32) +DO_LDFF1_LDNF1_2(sdu, MO_64, MO_32) +DO_LDFF1_LDNF1_2(sds, MO_64, MO_32) =20 -DO_LDFF1_LDNF1_2(dd, 3, 3) +DO_LDFF1_LDNF1_2(dd, MO_64, MO_64) =20 #undef DO_LDFF1_LDNF1_1 #undef DO_LDFF1_LDNF1_2 --=20 2.20.1 From nobody Mon Feb 9 21:08:28 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=linaro.org ARC-Seal: i=1; a=rsa-sha256; t=1583909205; cv=none; d=zohomail.com; s=zohoarc; b=hSYlEPjvnxGY6YuTwLGdrh1d9a26B3Y4qmq6wBW16gvcbm8asnzr0K7WByzRNcgDfW56XZwKBRAS1UrFMHK1gm7BeQtwM0W+Y1szF0IZqmQwqHKbDqn9M0UFckBBPjU+L6LtdBCFXnw97ZB/RUXOLaBVfyJVmPGZgwi3Qnk40lQ= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1583909205; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=KhxITN+NjmEXUXhdWp+bRLvbeT9VqflVpRPfCgP2sDA=; b=jsktX6JnU9mDzKupWpd7CcF8eAc0eVg4LUuKU0PaA82A9v6SoHmoy+J1fe7N/upYFqKlYtRjUmiGAUcXNuscPraPM7IsQWASQnkn9DurnquEbWBJMuTFr6jQAFqQqDxDaJxirE3t+JUwPQ/hRktrzwUmbpAHQIQ/esBAXtAF0vc= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 158390920576922.18416979493918; Tue, 10 Mar 2020 23:46:45 -0700 (PDT) Received: from localhost ([::1]:46398 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jBv8e-0000Yu-Fw for importer@patchew.org; Wed, 11 Mar 2020 02:46:44 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:59617) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jBv6f-0006R3-P1 for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:43 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jBv6d-00021V-Fl for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:41 -0400 Received: from mail-pl1-x644.google.com ([2607:f8b0:4864:20::644]:33484) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1jBv6d-00020n-7q for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:39 -0400 Received: by mail-pl1-x644.google.com with SMTP id ay11so632423plb.0 for ; Tue, 10 Mar 2020 23:44:39 -0700 (PDT) Received: from localhost.localdomain (97-126-123-70.tukw.qwest.net. [97.126.123.70]) by smtp.gmail.com with ESMTPSA id z3sm50137807pfz.155.2020.03.10.23.44.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 10 Mar 2020 23:44:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=KhxITN+NjmEXUXhdWp+bRLvbeT9VqflVpRPfCgP2sDA=; b=K01SY383f8aqADDrj0upIPIKGpQoU0zUlEzSfCwYctxykf5wgQFE9IQxNCfJvIUNHm OBByQVOUBuJtB+6VJ+HI14y2nNJPNWb6WNZVf9FFxwC7DRl8E3+3nNSi6t7kZlQ9zdcp PLGc1eEQVk6pT09At+jFxQbd3jBxo556CiGTzFKNiw2Zsm7n/uUf4gj7qfkNNfaTyLyC F4sT19Y0R7pevYh/JPYXQpJrn8NWN4ujHcBeqW/Lu83K7lIE/Lm25lUSLevv6IX+dMEq 7DIwcwj4DX0zeer+KgRew1kBfUu/HMFdZ/G0LqezpZAi6oywzgFLFCz6Ba+tZLnpTxIG MDFQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=KhxITN+NjmEXUXhdWp+bRLvbeT9VqflVpRPfCgP2sDA=; b=S//qHPpIUpZHU1mrljVodb+jGF6+NkauhGKf8Ag0JeFTPUleyFdylEr4dgrI0+EwmR YxwrhgDwSVDRvKLCGfu342NMH9TlEzLV1DtT8hwbbyl++eZ+agUTPjM1HI9M9Z6hR4Cu +6SM7Kp6z3m4Kp1pIRMl4D3O2WUY0spfKufmfoNRGXQVyqhe1JIkDtF9IoWXk9tw2hGw L7wNgkQcQvpGdZmbc4rze+dinQiZjoOuxjp5t7Sbbbcp+psYFLujfvbIWpcy6mRzKIrb Du2tjlzqGZkerVm9tDeR/wnKNEsSwsiV/e11mJb10CWAnn33OPv3rvVOvXSDjoJL1bX8 jCVw== X-Gm-Message-State: ANhLgQ2xHMaEaXX71GMfDi4vwMRkSuJN2DyvdO+PdoCjbBe3oAkM6Z0b x4y+ze9nTSxgUmCfbzmjn7Oz35hL+NU= X-Google-Smtp-Source: ADFU+vvHKL5wWPCOVIHa7QNGUf3SAildcKM4fkQKtR5Jg7gw912WM3T1/EfxgLqAfZI7PKsOhBWFaA== X-Received: by 2002:a17:90a:7309:: with SMTP id m9mr1850242pjk.52.1583909077579; Tue, 10 Mar 2020 23:44:37 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH 12/16] target/arm: Use SVEContLdSt for contiguous stores Date: Tue, 10 Mar 2020 23:44:16 -0700 Message-Id: <20200311064420.30606-13-richard.henderson@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200311064420.30606-1-richard.henderson@linaro.org> References: <20200311064420.30606-1-richard.henderson@linaro.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::644 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, qemu-arm@nongnu.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @linaro.org) Content-Type: text/plain; charset="utf-8" Follow the model set up for contiguous loads. This handles watchpoints correctly for contiguous stores, recognizing the exception before any changes to memory. Signed-off-by: Richard Henderson Reviewed-by: Peter Maydell --- target/arm/sve_helper.c | 288 ++++++++++++++++++++++------------------ 1 file changed, 162 insertions(+), 126 deletions(-) diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index 9198e1612a..c6f22928d5 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -4002,6 +4002,13 @@ static void sve_##NAME##_host(void *vd, intptr_t reg= _off, void *host) \ *(TYPEE *)(vd + H(reg_off)) =3D val; \ } =20 +#define DO_ST_HOST(NAME, H, TYPEE, TYPEM, HOST) \ +static void sve_##NAME##_host(void *vd, intptr_t reg_off, void *host) \ +{ \ + TYPEM val =3D *(TYPEE *)(vd + H(reg_off)); \ + HOST(host, val); \ +} + #define DO_LD_TLB(NAME, H, TYPEE, TYPEM, BSWAP, TLB) \ static void sve_##NAME##_tlb(CPUARMState *env, void *vd, intptr_t reg_off,= \ target_ulong addr, uintptr_t ra) = \ @@ -4031,6 +4038,7 @@ DO_LD_PRIM_1(ld1bdu, , uint64_t, uint8_t) DO_LD_PRIM_1(ld1bds, , uint64_t, int8_t) =20 #define DO_ST_PRIM_1(NAME, H, TE, TM) \ + DO_ST_HOST(st1##NAME, H, TE, TM, stb_p) \ DO_ST_TLB(st1##NAME, H, TE, TM, , cpu_stb_data_ra) =20 DO_ST_PRIM_1(bb, H1, uint8_t, uint8_t) @@ -4045,6 +4053,8 @@ DO_ST_PRIM_1(bd, , uint64_t, uint8_t) DO_LD_TLB(ld1##NAME##_le, H, TE, TM, le_##BSWAP, cpu_##LD##_data_ra) =20 #define DO_ST_PRIM_2(NAME, H, TE, TM, BSWAP, ST) \ + DO_ST_HOST(st1##NAME##_be, H, TE, TM, ST##_be_p) \ + DO_ST_HOST(st1##NAME##_le, H, TE, TM, ST##_le_p) \ DO_ST_TLB(st1##NAME##_be, H, TE, TM, be_##BSWAP, cpu_##ST##_data_ra) \ DO_ST_TLB(st1##NAME##_le, H, TE, TM, le_##BSWAP, cpu_##ST##_data_ra) =20 @@ -4906,151 +4916,177 @@ DO_LDFF1_LDNF1_2(dd, MO_64, MO_64) #undef DO_LDFF1_LDNF1_2 =20 /* - * Common helpers for all contiguous 1,2,3,4-register predicated stores. + * Common helper for all contiguous 1,2,3,4-register predicated stores. */ -static void sve_st1_r(CPUARMState *env, void *vg, target_ulong addr, - uint32_t desc, const uintptr_t ra, - const int esize, const int msize, - sve_ldst1_tlb_fn *tlb_fn) + +static inline QEMU_ALWAYS_INLINE +void sve_stN_r(CPUARMState *env, uint64_t *vg, target_ulong addr, uint32_t= desc, + const uintptr_t retaddr, const int esz, + const int msz, const int N, + sve_ldst1_host_fn *host_fn, + sve_ldst1_tlb_fn *tlb_fn) { const unsigned rd =3D extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT= , 5); - intptr_t i, oprsz =3D simd_oprsz(desc); - void *vd =3D &env->vfp.zregs[rd]; + const intptr_t reg_max =3D simd_oprsz(desc); + intptr_t reg_off, reg_last, mem_off; + SVEContLdSt info; + void *host; + int i, flags; =20 - for (i =3D 0; i < oprsz; ) { - uint16_t pg =3D *(uint16_t *)(vg + H1_2(i >> 3)); - do { - if (pg & 1) { - tlb_fn(env, vd, i, addr, ra); + /* Find the active elements. */ + if (!sve_cont_ldst_elements(&info, addr, vg, reg_max, esz, N << msz)) { + /* The entire predicate was false; no store occurs. */ + return; + } + + /* Probe the page(s). Exit with exception for any invalid page. */ + sve_cont_ldst_pages(&info, FAULT_ALL, env, addr, MMU_DATA_STORE, retad= dr); + + /* Handle watchpoints for all active elements. */ + sve_cont_ldst_watchpoints(&info, env, vg, addr, 1 << esz, N << msz, + BP_MEM_WRITE, retaddr); + + /* TODO: MTE check. */ + + flags =3D info.page[0].flags | info.page[1].flags; + if (unlikely(flags !=3D 0)) { +#ifdef CONFIG_USER_ONLY + g_assert_not_reached(); +#else + /* + * At least one page includes MMIO. + * Any bus operation can fail with cpu_transaction_failed, + * which for ARM will raise SyncExternal. We cannot avoid + * this fault and will leave with the store incomplete. + */ + mem_off =3D info.mem_off_first[0]; + reg_off =3D info.reg_off_first[0]; + reg_last =3D info.reg_off_last[1]; + if (reg_last < 0) { + reg_last =3D info.reg_off_split; + if (reg_last < 0) { + reg_last =3D info.reg_off_last[0]; } - i +=3D esize, pg >>=3D esize; - addr +=3D msize; - } while (i & 15); + } + + do { + uint64_t pg =3D vg[reg_off >> 6]; + do { + if ((pg >> (reg_off & 63)) & 1) { + for (i =3D 0; i < N; ++i) { + tlb_fn(env, &env->vfp.zregs[(rd + i) & 31], reg_of= f, + addr + mem_off + (i << msz), retaddr); + } + } + reg_off +=3D 1 << esz; + mem_off +=3D N << msz; + } while (reg_off & 63); + } while (reg_off <=3D reg_last); + return; +#endif + } + + mem_off =3D info.mem_off_first[0]; + reg_off =3D info.reg_off_first[0]; + reg_last =3D info.reg_off_last[0]; + host =3D info.page[0].host; + + while (reg_off <=3D reg_last) { + uint64_t pg =3D vg[reg_off >> 6]; + do { + if ((pg >> (reg_off & 63)) & 1) { + for (i =3D 0; i < N; ++i) { + host_fn(&env->vfp.zregs[(rd + i) & 31], reg_off, + host + mem_off + (i << msz)); + } + } + reg_off +=3D 1 << esz; + mem_off +=3D N << msz; + } while (reg_off <=3D reg_last && (reg_off & 63)); + } + + /* + * Use the slow path to manage the cross-page misalignment. + * But we know this is RAM and cannot trap. + */ + mem_off =3D info.mem_off_split; + if (unlikely(mem_off >=3D 0)) { + reg_off =3D info.reg_off_split; + for (i =3D 0; i < N; ++i) { + tlb_fn(env, &env->vfp.zregs[(rd + i) & 31], reg_off, + addr + mem_off + (i << msz), retaddr); + } + } + + mem_off =3D info.mem_off_first[1]; + if (unlikely(mem_off >=3D 0)) { + reg_off =3D info.reg_off_first[1]; + reg_last =3D info.reg_off_last[1]; + host =3D info.page[1].host; + + do { + uint64_t pg =3D vg[reg_off >> 6]; + do { + if ((pg >> (reg_off & 63)) & 1) { + for (i =3D 0; i < N; ++i) { + host_fn(&env->vfp.zregs[(rd + i) & 31], reg_off, + host + mem_off + (i << msz)); + } + } + reg_off +=3D 1 << esz; + mem_off +=3D N << msz; + } while (reg_off & 63); + } while (reg_off <=3D reg_last); } } =20 -static void sve_st2_r(CPUARMState *env, void *vg, target_ulong addr, - uint32_t desc, const uintptr_t ra, - const int esize, const int msize, - sve_ldst1_tlb_fn *tlb_fn) -{ - const unsigned rd =3D extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT= , 5); - intptr_t i, oprsz =3D simd_oprsz(desc); - void *d1 =3D &env->vfp.zregs[rd]; - void *d2 =3D &env->vfp.zregs[(rd + 1) & 31]; - - for (i =3D 0; i < oprsz; ) { - uint16_t pg =3D *(uint16_t *)(vg + H1_2(i >> 3)); - do { - if (pg & 1) { - tlb_fn(env, d1, i, addr, ra); - tlb_fn(env, d2, i, addr + msize, ra); - } - i +=3D esize, pg >>=3D esize; - addr +=3D 2 * msize; - } while (i & 15); - } -} - -static void sve_st3_r(CPUARMState *env, void *vg, target_ulong addr, - uint32_t desc, const uintptr_t ra, - const int esize, const int msize, - sve_ldst1_tlb_fn *tlb_fn) -{ - const unsigned rd =3D extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT= , 5); - intptr_t i, oprsz =3D simd_oprsz(desc); - void *d1 =3D &env->vfp.zregs[rd]; - void *d2 =3D &env->vfp.zregs[(rd + 1) & 31]; - void *d3 =3D &env->vfp.zregs[(rd + 2) & 31]; - - for (i =3D 0; i < oprsz; ) { - uint16_t pg =3D *(uint16_t *)(vg + H1_2(i >> 3)); - do { - if (pg & 1) { - tlb_fn(env, d1, i, addr, ra); - tlb_fn(env, d2, i, addr + msize, ra); - tlb_fn(env, d3, i, addr + 2 * msize, ra); - } - i +=3D esize, pg >>=3D esize; - addr +=3D 3 * msize; - } while (i & 15); - } -} - -static void sve_st4_r(CPUARMState *env, void *vg, target_ulong addr, - uint32_t desc, const uintptr_t ra, - const int esize, const int msize, - sve_ldst1_tlb_fn *tlb_fn) -{ - const unsigned rd =3D extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT= , 5); - intptr_t i, oprsz =3D simd_oprsz(desc); - void *d1 =3D &env->vfp.zregs[rd]; - void *d2 =3D &env->vfp.zregs[(rd + 1) & 31]; - void *d3 =3D &env->vfp.zregs[(rd + 2) & 31]; - void *d4 =3D &env->vfp.zregs[(rd + 3) & 31]; - - for (i =3D 0; i < oprsz; ) { - uint16_t pg =3D *(uint16_t *)(vg + H1_2(i >> 3)); - do { - if (pg & 1) { - tlb_fn(env, d1, i, addr, ra); - tlb_fn(env, d2, i, addr + msize, ra); - tlb_fn(env, d3, i, addr + 2 * msize, ra); - tlb_fn(env, d4, i, addr + 3 * msize, ra); - } - i +=3D esize, pg >>=3D esize; - addr +=3D 4 * msize; - } while (i & 15); - } -} - -#define DO_STN_1(N, NAME, ESIZE) \ -void QEMU_FLATTEN HELPER(sve_st##N##NAME##_r) \ - (CPUARMState *env, void *vg, target_ulong addr, uint32_t desc) \ +#define DO_STN_1(N, NAME, ESZ) \ +void HELPER(sve_st##N##NAME##_r)(CPUARMState *env, void *vg, \ + target_ulong addr, uint32_t desc) \ { \ - sve_st##N##_r(env, vg, addr, desc, GETPC(), ESIZE, 1, \ - sve_st1##NAME##_tlb); \ + sve_stN_r(env, vg, addr, desc, GETPC(), ESZ, MO_8, N, \ + sve_st1##NAME##_host, sve_st1##NAME##_tlb); \ } =20 -#define DO_STN_2(N, NAME, ESIZE, MSIZE) \ -void QEMU_FLATTEN HELPER(sve_st##N##NAME##_le_r) \ - (CPUARMState *env, void *vg, target_ulong addr, uint32_t desc) \ +#define DO_STN_2(N, NAME, ESZ, MSZ) \ +void HELPER(sve_st##N##NAME##_le_r)(CPUARMState *env, void *vg, \ + target_ulong addr, uint32_t desc) \ { \ - sve_st##N##_r(env, vg, addr, desc, GETPC(), ESIZE, MSIZE, \ - sve_st1##NAME##_le_tlb); \ + sve_stN_r(env, vg, addr, desc, GETPC(), ESZ, MSZ, N, \ + sve_st1##NAME##_le_host, sve_st1##NAME##_le_tlb); \ } \ -void QEMU_FLATTEN HELPER(sve_st##N##NAME##_be_r) \ - (CPUARMState *env, void *vg, target_ulong addr, uint32_t desc) \ +void HELPER(sve_st##N##NAME##_be_r)(CPUARMState *env, void *vg, \ + target_ulong addr, uint32_t desc) \ { \ - sve_st##N##_r(env, vg, addr, desc, GETPC(), ESIZE, MSIZE, \ - sve_st1##NAME##_be_tlb); \ + sve_stN_r(env, vg, addr, desc, GETPC(), ESZ, MSZ, N, \ + sve_st1##NAME##_be_host, sve_st1##NAME##_be_tlb); \ } =20 -DO_STN_1(1, bb, 1) -DO_STN_1(1, bh, 2) -DO_STN_1(1, bs, 4) -DO_STN_1(1, bd, 8) -DO_STN_1(2, bb, 1) -DO_STN_1(3, bb, 1) -DO_STN_1(4, bb, 1) +DO_STN_1(1, bb, MO_8) +DO_STN_1(1, bh, MO_16) +DO_STN_1(1, bs, MO_32) +DO_STN_1(1, bd, MO_64) +DO_STN_1(2, bb, MO_8) +DO_STN_1(3, bb, MO_8) +DO_STN_1(4, bb, MO_8) =20 -DO_STN_2(1, hh, 2, 2) -DO_STN_2(1, hs, 4, 2) -DO_STN_2(1, hd, 8, 2) -DO_STN_2(2, hh, 2, 2) -DO_STN_2(3, hh, 2, 2) -DO_STN_2(4, hh, 2, 2) +DO_STN_2(1, hh, MO_16, MO_16) +DO_STN_2(1, hs, MO_32, MO_16) +DO_STN_2(1, hd, MO_64, MO_16) +DO_STN_2(2, hh, MO_16, MO_16) +DO_STN_2(3, hh, MO_16, MO_16) +DO_STN_2(4, hh, MO_16, MO_16) =20 -DO_STN_2(1, ss, 4, 4) -DO_STN_2(1, sd, 8, 4) -DO_STN_2(2, ss, 4, 4) -DO_STN_2(3, ss, 4, 4) -DO_STN_2(4, ss, 4, 4) +DO_STN_2(1, ss, MO_32, MO_32) +DO_STN_2(1, sd, MO_64, MO_32) +DO_STN_2(2, ss, MO_32, MO_32) +DO_STN_2(3, ss, MO_32, MO_32) +DO_STN_2(4, ss, MO_32, MO_32) =20 -DO_STN_2(1, dd, 8, 8) -DO_STN_2(2, dd, 8, 8) -DO_STN_2(3, dd, 8, 8) -DO_STN_2(4, dd, 8, 8) +DO_STN_2(1, dd, MO_64, MO_64) +DO_STN_2(2, dd, MO_64, MO_64) +DO_STN_2(3, dd, MO_64, MO_64) +DO_STN_2(4, dd, MO_64, MO_64) =20 #undef DO_STN_1 #undef DO_STN_2 --=20 2.20.1 From nobody Mon Feb 9 21:08:28 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=linaro.org ARC-Seal: i=1; a=rsa-sha256; t=1583909584; cv=none; d=zohomail.com; s=zohoarc; b=JllumVZpGHaLID30ZB5kKSGiI1K1cvh2Y3tIJVj5pPB+f85Q48HDq62OV1OEdk3MKcgHH2bZDqwAar81e2ZP3X39c9qZhOHwozmu34ZPWd/R3JfzUYIZy7GFhWMRLH9ZZ6Ou6O9XtggxEnvHQfMt1Zyc+Pi0SMcyClTDPXc+y/s= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1583909584; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=x3t5PDoyjO7TC0DW2CC4BuCDJWVwgzq1cMlGkXCUhYU=; b=KPdc4/Zd9H857hXNC8fvbEkGUljaQDPrZBnAfWTye5E3uhd6DjWjHp6NBT8LTfw5AsxURDJ1MhNuReJItT1sxmLBapMa+sRQzUWB7oZolSIkYhLS7pa/sOzllSK+Lx2OirN0e+9i5v0nQhOXWoW35WZfQZ9f/pPqA7dvMr6VVGw= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1583909584111775.6184171278372; Tue, 10 Mar 2020 23:53:04 -0700 (PDT) Received: from localhost ([::1]:46636 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jBvEk-0001yu-VY for importer@patchew.org; Wed, 11 Mar 2020 02:53:03 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:59648) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jBv6h-0006SS-47 for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:45 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jBv6e-00022a-VV for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:43 -0400 Received: from mail-pl1-x644.google.com ([2607:f8b0:4864:20::644]:46643) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1jBv6e-00022B-NE for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:40 -0400 Received: by mail-pl1-x644.google.com with SMTP id w12so604769pll.13 for ; Tue, 10 Mar 2020 23:44:40 -0700 (PDT) Received: from localhost.localdomain (97-126-123-70.tukw.qwest.net. [97.126.123.70]) by smtp.gmail.com with ESMTPSA id z3sm50137807pfz.155.2020.03.10.23.44.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 10 Mar 2020 23:44:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=x3t5PDoyjO7TC0DW2CC4BuCDJWVwgzq1cMlGkXCUhYU=; b=KPCy48BoRzZwRXMbdj1MeiGeFlCbpdtdfvqD8a6aYnpYkxsEFudnzhwI1zWvZqOGhl lG2fZwjmhLVCnB2tuuZoP1iFc8FHWmE5mnfpW2jL/pj3dwv4xoiAo1ZGbH7OTZCfpfbO LFS4r+MrmmSdOQUM6fDsWxhi7F3RR15G75YBsSA3wvbgnsMeVPpW2OcG4DyxhqZPPF4L iwglr61/imuf1IzXuAif8S2dbkKyeyh4LiLzJhm9GEzAc9yuaerZvAiBmCd1RmxxYyfU Doa9p59TmJZJR8pLW/QWDkcDeySVCD5lfJel8ehw/V+Bn/0x3pNT4YgY3TBUVyVZCnZm cNEg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=x3t5PDoyjO7TC0DW2CC4BuCDJWVwgzq1cMlGkXCUhYU=; b=bw+zIG9tPJDNvddyr5zYMAFQoAcGvtf6YOfye5xrtH/6n7c8q8QSg2dakUkVIVxKT5 UYr79Rn5+uF4V3v1XBJ8g95N2Y5JfkWilWGWtINvk2Y8xROzRW8bJy/567HxrXLjnyQm kyad9vjTUaYln79oTM3zq94IZOzalDcloycWUDDQk4SG4H+Wsaa9X+N20zpDxfvSsSfz VDRQxSbKjUjEwxfPvY0+6xT59kE5Fit+gT0MgOmWZfhpEiquRgNFXczBCGaeHdvICAI7 Bp7mmOLsHYtW6BIEztMI506gVzV1K1K16OUSGdDAhpFe6RhLMq1lso+D3roPqJwDa9HH 6DOA== X-Gm-Message-State: ANhLgQ3Eeag16M2FVXI5pzTif0pZgXKp5fB16kbilR6QAUPvguzKWp9o o54VCzXKf3QIP0gEP8U2G9q3skFmcaU= X-Google-Smtp-Source: ADFU+vu3vqHLKeiffIrUCl20F6oTly59rHVJ4JKw90gUT05XXNioPijPN5yFOknUzZXMDuG+JqEFIA== X-Received: by 2002:a17:902:7c0a:: with SMTP id x10mr1641782pll.229.1583909079053; Tue, 10 Mar 2020 23:44:39 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH 13/16] target/arm: Reuse sve_probe_page for gather first-fault loads Date: Tue, 10 Mar 2020 23:44:17 -0700 Message-Id: <20200311064420.30606-14-richard.henderson@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200311064420.30606-1-richard.henderson@linaro.org> References: <20200311064420.30606-1-richard.henderson@linaro.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::644 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, qemu-arm@nongnu.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @linaro.org) Content-Type: text/plain; charset="utf-8" This avoids the need for a separate set of helpers to implement no-fault semantics, and will enable MTE in the future. Signed-off-by: Richard Henderson Reviewed-by: Peter Maydell --- target/arm/sve_helper.c | 323 ++++++++++++++++------------------------ 1 file changed, 127 insertions(+), 196 deletions(-) diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index c6f22928d5..a33699adf8 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -5255,231 +5255,162 @@ DO_LD1_ZPZ_D(dd_be, zd) =20 /* First fault loads with a vector index. */ =20 -/* Load one element into VD+REG_OFF from (ENV,VADDR) without faulting. - * The controlling predicate is known to be true. Return true if the - * load was successful. - */ -typedef bool sve_ld1_nf_fn(CPUARMState *env, void *vd, intptr_t reg_off, - target_ulong vaddr, int mmu_idx); - -#ifdef CONFIG_SOFTMMU -#define DO_LD_NF(NAME, H, TYPEE, TYPEM, HOST) \ -static bool sve_ld##NAME##_nf(CPUARMState *env, void *vd, intptr_t reg_off= , \ - target_ulong addr, int mmu_idx) = \ -{ = \ - target_ulong next_page =3D -(addr | TARGET_PAGE_MASK); = \ - if (likely(next_page - addr >=3D sizeof(TYPEM))) { = \ - void *host =3D tlb_vaddr_to_host(env, addr, MMU_DATA_LOAD, mmu_idx= ); \ - if (likely(host)) { = \ - TYPEM val =3D HOST(host); = \ - *(TYPEE *)(vd + H(reg_off)) =3D val; = \ - return true; = \ - } = \ - } = \ - return false; = \ -} -#else -#define DO_LD_NF(NAME, H, TYPEE, TYPEM, HOST) \ -static bool sve_ld##NAME##_nf(CPUARMState *env, void *vd, intptr_t reg_off= , \ - target_ulong addr, int mmu_idx) = \ -{ = \ - if (likely(page_check_range(addr, sizeof(TYPEM), PAGE_READ))) { = \ - TYPEM val =3D HOST(g2h(addr)); = \ - *(TYPEE *)(vd + H(reg_off)) =3D val; = \ - return true; = \ - } = \ - return false; = \ -} -#endif - -DO_LD_NF(bsu, H1_4, uint32_t, uint8_t, ldub_p) -DO_LD_NF(bss, H1_4, uint32_t, int8_t, ldsb_p) -DO_LD_NF(bdu, , uint64_t, uint8_t, ldub_p) -DO_LD_NF(bds, , uint64_t, int8_t, ldsb_p) - -DO_LD_NF(hsu_le, H1_4, uint32_t, uint16_t, lduw_le_p) -DO_LD_NF(hss_le, H1_4, uint32_t, int16_t, ldsw_le_p) -DO_LD_NF(hsu_be, H1_4, uint32_t, uint16_t, lduw_be_p) -DO_LD_NF(hss_be, H1_4, uint32_t, int16_t, ldsw_be_p) -DO_LD_NF(hdu_le, , uint64_t, uint16_t, lduw_le_p) -DO_LD_NF(hds_le, , uint64_t, int16_t, ldsw_le_p) -DO_LD_NF(hdu_be, , uint64_t, uint16_t, lduw_be_p) -DO_LD_NF(hds_be, , uint64_t, int16_t, ldsw_be_p) - -DO_LD_NF(ss_le, H1_4, uint32_t, uint32_t, ldl_le_p) -DO_LD_NF(ss_be, H1_4, uint32_t, uint32_t, ldl_be_p) -DO_LD_NF(sdu_le, , uint64_t, uint32_t, ldl_le_p) -DO_LD_NF(sds_le, , uint64_t, int32_t, ldl_le_p) -DO_LD_NF(sdu_be, , uint64_t, uint32_t, ldl_be_p) -DO_LD_NF(sds_be, , uint64_t, int32_t, ldl_be_p) - -DO_LD_NF(dd_le, , uint64_t, uint64_t, ldq_le_p) -DO_LD_NF(dd_be, , uint64_t, uint64_t, ldq_be_p) - /* - * Common helper for all gather first-faulting loads. + * Common helpers for all gather first-faulting loads. */ -static inline void sve_ldff1_zs(CPUARMState *env, void *vd, void *vg, void= *vm, - target_ulong base, uint32_t desc, uintptr_= t ra, - zreg_off_fn *off_fn, sve_ldst1_tlb_fn *tlb= _fn, - sve_ld1_nf_fn *nonfault_fn) + +static inline QEMU_ALWAYS_INLINE +void sve_ldff1_z(CPUARMState *env, void *vd, uint64_t *vg, void *vm, + target_ulong base, uint32_t desc, uintptr_t retaddr, + const int esz, const int msz, zreg_off_fn *off_fn, + sve_ldst1_host_fn *host_fn, + sve_ldst1_tlb_fn *tlb_fn) { - const TCGMemOpIdx oi =3D extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHI= FT); - const int mmu_idx =3D get_mmuidx(oi); + const int mmu_idx =3D cpu_mmu_index(env, false); const int scale =3D extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, = 2); - intptr_t reg_off, reg_max =3D simd_oprsz(desc); - target_ulong addr; + const int esize =3D 1 << esz; + const int msize =3D 1 << msz; + const intptr_t reg_max =3D simd_oprsz(desc); + intptr_t reg_off; + SVEHostPage info; + target_ulong addr, in_page; =20 /* Skip to the first true predicate. */ - reg_off =3D find_next_active(vg, 0, reg_max, MO_32); - if (likely(reg_off < reg_max)) { - /* Perform one normal read, which will fault or not. */ - addr =3D off_fn(vm, reg_off); - addr =3D base + (addr << scale); - tlb_fn(env, vd, reg_off, addr, ra); - - /* The rest of the reads will be non-faulting. */ + reg_off =3D find_next_active(vg, 0, reg_max, esz); + if (unlikely(reg_off >=3D reg_max)) { + /* The entire predicate was false; no load occurs. */ + memset(vd, 0, reg_max); + return; } =20 - /* After any fault, zero the leading predicated false elements. */ + /* + * Probe the first element, allowing faults. + */ + addr =3D base + (off_fn(vm, reg_off) << scale); + tlb_fn(env, vd, reg_off, addr, retaddr); + + /* After any fault, zero the other elements. */ swap_memzero(vd, reg_off); + reg_off +=3D esize; + swap_memzero(vd + reg_off, reg_max - reg_off); =20 - while (likely((reg_off +=3D 4) < reg_max)) { - uint64_t pg =3D *(uint64_t *)(vg + (reg_off >> 6) * 8); - if (likely((pg >> (reg_off & 63)) & 1)) { - addr =3D off_fn(vm, reg_off); - addr =3D base + (addr << scale); - if (!nonfault_fn(env, vd, reg_off, addr, mmu_idx)) { - record_fault(env, reg_off, reg_max); - break; + /* + * Probe the remaining elements, not allowing faults. + */ + while (reg_off < reg_max) { + uint64_t pg =3D vg[reg_off >> 6]; + do { + if (likely((pg >> (reg_off & 63)) & 1)) { + addr =3D base + (off_fn(vm, reg_off) << scale); + in_page =3D -(addr | TARGET_PAGE_MASK); + + if (unlikely(in_page < msize)) { + /* Stop if the element crosses a page boundary. */ + goto fault; + } + + sve_probe_page(&info, true, env, addr, 0, MMU_DATA_LOAD, + mmu_idx, retaddr); + if (unlikely(info.flags & (TLB_INVALID_MASK | TLB_MMIO))) { + goto fault; + } + if (unlikely(info.flags & TLB_WATCHPOINT) && + cpu_probe_watchpoint(env_cpu(env), addr, msize, + BP_MEM_READ)) { + goto fault; + } + /* TODO: MTE check. */ + + host_fn(vd, reg_off, info.host); } - } else { - *(uint32_t *)(vd + H1_4(reg_off)) =3D 0; - } + reg_off +=3D esize; + } while (reg_off & 63); } + return; + + fault: + record_fault(env, reg_off, reg_max); } =20 -static inline void sve_ldff1_zd(CPUARMState *env, void *vd, void *vg, void= *vm, - target_ulong base, uint32_t desc, uintptr_= t ra, - zreg_off_fn *off_fn, sve_ldst1_tlb_fn *tlb= _fn, - sve_ld1_nf_fn *nonfault_fn) -{ - const TCGMemOpIdx oi =3D extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHI= FT); - const int mmu_idx =3D get_mmuidx(oi); - const int scale =3D extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, = 2); - intptr_t reg_off, reg_max =3D simd_oprsz(desc); - target_ulong addr; - - /* Skip to the first true predicate. */ - reg_off =3D find_next_active(vg, 0, reg_max, MO_64); - if (likely(reg_off < reg_max)) { - /* Perform one normal read, which will fault or not. */ - addr =3D off_fn(vm, reg_off); - addr =3D base + (addr << scale); - tlb_fn(env, vd, reg_off, addr, ra); - - /* The rest of the reads will be non-faulting. */ - } - - /* After any fault, zero the leading predicated false elements. */ - swap_memzero(vd, reg_off); - - while (likely((reg_off +=3D 8) < reg_max)) { - uint8_t pg =3D *(uint8_t *)(vg + H1(reg_off >> 3)); - if (likely(pg & 1)) { - addr =3D off_fn(vm, reg_off); - addr =3D base + (addr << scale); - if (!nonfault_fn(env, vd, reg_off, addr, mmu_idx)) { - record_fault(env, reg_off, reg_max); - break; - } - } else { - *(uint64_t *)(vd + reg_off) =3D 0; - } - } +#define DO_LDFF1_ZPZ_S(MEM, OFS, MSZ) \ +void HELPER(sve_ldff##MEM##_##OFS)(CPUARMState *env, void *vd, void *vg, = \ + void *vm, target_ulong base, uint32_t d= esc) \ +{ = \ + sve_ldff1_z(env, vd, vg, vm, base, desc, GETPC(), MO_32, MSZ, = \ + off_##OFS##_s, sve_ld1##MEM##_host, sve_ld1##MEM##_tlb); = \ } =20 -#define DO_LDFF1_ZPZ_S(MEM, OFS) \ -void HELPER(sve_ldff##MEM##_##OFS) \ - (CPUARMState *env, void *vd, void *vg, void *vm, \ - target_ulong base, uint32_t desc) \ -{ \ - sve_ldff1_zs(env, vd, vg, vm, base, desc, GETPC(), \ - off_##OFS##_s, sve_ld1##MEM##_tlb, sve_ld##MEM##_nf); \ +#define DO_LDFF1_ZPZ_D(MEM, OFS, MSZ) \ +void HELPER(sve_ldff##MEM##_##OFS)(CPUARMState *env, void *vd, void *vg, = \ + void *vm, target_ulong base, uint32_t d= esc) \ +{ = \ + sve_ldff1_z(env, vd, vg, vm, base, desc, GETPC(), MO_64, MSZ, = \ + off_##OFS##_d, sve_ld1##MEM##_host, sve_ld1##MEM##_tlb); = \ } =20 -#define DO_LDFF1_ZPZ_D(MEM, OFS) \ -void HELPER(sve_ldff##MEM##_##OFS) \ - (CPUARMState *env, void *vd, void *vg, void *vm, \ - target_ulong base, uint32_t desc) \ -{ \ - sve_ldff1_zd(env, vd, vg, vm, base, desc, GETPC(), \ - off_##OFS##_d, sve_ld1##MEM##_tlb, sve_ld##MEM##_nf); \ -} +DO_LDFF1_ZPZ_S(bsu, zsu, MO_8) +DO_LDFF1_ZPZ_S(bsu, zss, MO_8) +DO_LDFF1_ZPZ_D(bdu, zsu, MO_8) +DO_LDFF1_ZPZ_D(bdu, zss, MO_8) +DO_LDFF1_ZPZ_D(bdu, zd, MO_8) =20 -DO_LDFF1_ZPZ_S(bsu, zsu) -DO_LDFF1_ZPZ_S(bsu, zss) -DO_LDFF1_ZPZ_D(bdu, zsu) -DO_LDFF1_ZPZ_D(bdu, zss) -DO_LDFF1_ZPZ_D(bdu, zd) +DO_LDFF1_ZPZ_S(bss, zsu, MO_8) +DO_LDFF1_ZPZ_S(bss, zss, MO_8) +DO_LDFF1_ZPZ_D(bds, zsu, MO_8) +DO_LDFF1_ZPZ_D(bds, zss, MO_8) +DO_LDFF1_ZPZ_D(bds, zd, MO_8) =20 -DO_LDFF1_ZPZ_S(bss, zsu) -DO_LDFF1_ZPZ_S(bss, zss) -DO_LDFF1_ZPZ_D(bds, zsu) -DO_LDFF1_ZPZ_D(bds, zss) -DO_LDFF1_ZPZ_D(bds, zd) +DO_LDFF1_ZPZ_S(hsu_le, zsu, MO_16) +DO_LDFF1_ZPZ_S(hsu_le, zss, MO_16) +DO_LDFF1_ZPZ_D(hdu_le, zsu, MO_16) +DO_LDFF1_ZPZ_D(hdu_le, zss, MO_16) +DO_LDFF1_ZPZ_D(hdu_le, zd, MO_16) =20 -DO_LDFF1_ZPZ_S(hsu_le, zsu) -DO_LDFF1_ZPZ_S(hsu_le, zss) -DO_LDFF1_ZPZ_D(hdu_le, zsu) -DO_LDFF1_ZPZ_D(hdu_le, zss) -DO_LDFF1_ZPZ_D(hdu_le, zd) +DO_LDFF1_ZPZ_S(hsu_be, zsu, MO_16) +DO_LDFF1_ZPZ_S(hsu_be, zss, MO_16) +DO_LDFF1_ZPZ_D(hdu_be, zsu, MO_16) +DO_LDFF1_ZPZ_D(hdu_be, zss, MO_16) +DO_LDFF1_ZPZ_D(hdu_be, zd, MO_16) =20 -DO_LDFF1_ZPZ_S(hsu_be, zsu) -DO_LDFF1_ZPZ_S(hsu_be, zss) -DO_LDFF1_ZPZ_D(hdu_be, zsu) -DO_LDFF1_ZPZ_D(hdu_be, zss) -DO_LDFF1_ZPZ_D(hdu_be, zd) +DO_LDFF1_ZPZ_S(hss_le, zsu, MO_16) +DO_LDFF1_ZPZ_S(hss_le, zss, MO_16) +DO_LDFF1_ZPZ_D(hds_le, zsu, MO_16) +DO_LDFF1_ZPZ_D(hds_le, zss, MO_16) +DO_LDFF1_ZPZ_D(hds_le, zd, MO_16) =20 -DO_LDFF1_ZPZ_S(hss_le, zsu) -DO_LDFF1_ZPZ_S(hss_le, zss) -DO_LDFF1_ZPZ_D(hds_le, zsu) -DO_LDFF1_ZPZ_D(hds_le, zss) -DO_LDFF1_ZPZ_D(hds_le, zd) +DO_LDFF1_ZPZ_S(hss_be, zsu, MO_16) +DO_LDFF1_ZPZ_S(hss_be, zss, MO_16) +DO_LDFF1_ZPZ_D(hds_be, zsu, MO_16) +DO_LDFF1_ZPZ_D(hds_be, zss, MO_16) +DO_LDFF1_ZPZ_D(hds_be, zd, MO_16) =20 -DO_LDFF1_ZPZ_S(hss_be, zsu) -DO_LDFF1_ZPZ_S(hss_be, zss) -DO_LDFF1_ZPZ_D(hds_be, zsu) -DO_LDFF1_ZPZ_D(hds_be, zss) -DO_LDFF1_ZPZ_D(hds_be, zd) +DO_LDFF1_ZPZ_S(ss_le, zsu, MO_32) +DO_LDFF1_ZPZ_S(ss_le, zss, MO_32) +DO_LDFF1_ZPZ_D(sdu_le, zsu, MO_32) +DO_LDFF1_ZPZ_D(sdu_le, zss, MO_32) +DO_LDFF1_ZPZ_D(sdu_le, zd, MO_32) =20 -DO_LDFF1_ZPZ_S(ss_le, zsu) -DO_LDFF1_ZPZ_S(ss_le, zss) -DO_LDFF1_ZPZ_D(sdu_le, zsu) -DO_LDFF1_ZPZ_D(sdu_le, zss) -DO_LDFF1_ZPZ_D(sdu_le, zd) +DO_LDFF1_ZPZ_S(ss_be, zsu, MO_32) +DO_LDFF1_ZPZ_S(ss_be, zss, MO_32) +DO_LDFF1_ZPZ_D(sdu_be, zsu, MO_32) +DO_LDFF1_ZPZ_D(sdu_be, zss, MO_32) +DO_LDFF1_ZPZ_D(sdu_be, zd, MO_32) =20 -DO_LDFF1_ZPZ_S(ss_be, zsu) -DO_LDFF1_ZPZ_S(ss_be, zss) -DO_LDFF1_ZPZ_D(sdu_be, zsu) -DO_LDFF1_ZPZ_D(sdu_be, zss) -DO_LDFF1_ZPZ_D(sdu_be, zd) +DO_LDFF1_ZPZ_D(sds_le, zsu, MO_32) +DO_LDFF1_ZPZ_D(sds_le, zss, MO_32) +DO_LDFF1_ZPZ_D(sds_le, zd, MO_32) =20 -DO_LDFF1_ZPZ_D(sds_le, zsu) -DO_LDFF1_ZPZ_D(sds_le, zss) -DO_LDFF1_ZPZ_D(sds_le, zd) +DO_LDFF1_ZPZ_D(sds_be, zsu, MO_32) +DO_LDFF1_ZPZ_D(sds_be, zss, MO_32) +DO_LDFF1_ZPZ_D(sds_be, zd, MO_32) =20 -DO_LDFF1_ZPZ_D(sds_be, zsu) -DO_LDFF1_ZPZ_D(sds_be, zss) -DO_LDFF1_ZPZ_D(sds_be, zd) +DO_LDFF1_ZPZ_D(dd_le, zsu, MO_64) +DO_LDFF1_ZPZ_D(dd_le, zss, MO_64) +DO_LDFF1_ZPZ_D(dd_le, zd, MO_64) =20 -DO_LDFF1_ZPZ_D(dd_le, zsu) -DO_LDFF1_ZPZ_D(dd_le, zss) -DO_LDFF1_ZPZ_D(dd_le, zd) - -DO_LDFF1_ZPZ_D(dd_be, zsu) -DO_LDFF1_ZPZ_D(dd_be, zss) -DO_LDFF1_ZPZ_D(dd_be, zd) +DO_LDFF1_ZPZ_D(dd_be, zsu, MO_64) +DO_LDFF1_ZPZ_D(dd_be, zss, MO_64) +DO_LDFF1_ZPZ_D(dd_be, zd, MO_64) =20 /* Stores with a vector index. */ =20 --=20 2.20.1 From nobody Mon Feb 9 21:08:28 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=linaro.org ARC-Seal: i=1; a=rsa-sha256; t=1583909896; cv=none; d=zohomail.com; s=zohoarc; b=IS6dMAkMq3dR+hc0/bi8lnVJ6ZCX4Zh+Ve6sRuU+6VjPzIsBAG978GB7LumZRDCj0UNG28QNKSFtzmQjfoDd4HmZ/2fFLOJt6stSigXqOC0/GnzYHUqBy8cvgrMgRSgPnp/eFPvrJCwf4HDw5T6YZaSJ6KMpp+Mk6gVq5EQI/f0= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1583909896; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=lMQ8Ls/5UNh2CeM9JrBjlxyyT05PO0oq+06rVR26ElQ=; b=NFZRP1g1WCbMsT/PXWKl1Jwb77cBCImGnRbbFxPNt3bYgq9fMSWXDsRyfg7jjdeyTMcWbpB9AqOq2H17h1tN18FTOFonJzmgFrxR1koPmyZ56AfoN4wOgRH8AU2UCRnHrgO/bpcaA8sXwFgtD3acKDrKVN1xFqAV/v9iEbfXPdM= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1583909896417127.53063303868942; Tue, 10 Mar 2020 23:58:16 -0700 (PDT) Received: from localhost ([::1]:47038 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jBvJn-0002Gl-CE for importer@patchew.org; Wed, 11 Mar 2020 02:58:15 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:59699) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jBv6j-0006Uc-LB for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:47 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jBv6h-00024i-Cs for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:45 -0400 Received: from mail-pj1-x1044.google.com ([2607:f8b0:4864:20::1044]:37668) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1jBv6h-00023q-1q for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:43 -0400 Received: by mail-pj1-x1044.google.com with SMTP id ca13so505618pjb.2 for ; Tue, 10 Mar 2020 23:44:42 -0700 (PDT) Received: from localhost.localdomain (97-126-123-70.tukw.qwest.net. [97.126.123.70]) by smtp.gmail.com with ESMTPSA id z3sm50137807pfz.155.2020.03.10.23.44.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 10 Mar 2020 23:44:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=lMQ8Ls/5UNh2CeM9JrBjlxyyT05PO0oq+06rVR26ElQ=; b=pnsSDz7uX7GhYBZZYMXEfjVFEGPwaQGv7iR7MVDt6XU0LW1Tc/eITVr8UjAcfxvK4V 2B4Fu0DhX74i8csLOBBc+UDgPV85jn5k84zP86wz9JIuJ1V5T840XSH3obuDoTn/6Fbp tYzJOBB3QsLnT643Tp5+KdKKwjHOU+cNNHu2kh7Z37ZMteBQNAf8ah68jkLbsR9PGjK/ f+A6jOioYsKWmdEAHaoK1u2yLyT+ID8FJFKXIgPhvDeTc8HpoN7smPU3NCHOolFjgmW4 la7nrqjJmb6BQiW5r4AAPyoq602MxLUej4VkFpm2Lw410Poj3NiLhVHVxMEcjF9bl1ho 3x3Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=lMQ8Ls/5UNh2CeM9JrBjlxyyT05PO0oq+06rVR26ElQ=; b=uj4P/eco1zmtYZ9uFRY6BBQJOmaFe/QFB49STpAdWHyXIHDMeC7iRt+0uej2Vpd3O0 Z0G53vq0QIWqcRFfI5k7zA7SMz01apJX8i+5RosuPCiuuRjWnOtzKjQ/Ve2Qh6vE7gUR 9p4DAj081ahyg1fb5KEVuO3FWpPfwhFMi8Am+4HMBrCRd3qAfilyvCX7TaKWGMTl5NpD CbC918Zroxb938XOf6j5fZx9DnYaEC36mp407l9wfx/26Pajq4Uc7a/MM45GFGiSl03a TtbUBNYePxh09kgCdc2TqpNxOUu0RGNjuwhdIU57oX/mVGM++BzFChqPp3Hh6wtng2Z9 s/TA== X-Gm-Message-State: ANhLgQ1XKWO5j0Y6phn+gKMsuV6PFlgbWJtWfagt52GS+gMsCkFau716 1c+2CpSVF4Nrxh9XcImo1GlZt/RIuao= X-Google-Smtp-Source: ADFU+vvjqqoWeA5l4BdfN6GnjVYhrsVIGO924tfsDbRVTQHq670SCt1jVayt476ortzERQb9HQSPrQ== X-Received: by 2002:a17:90a:e7c8:: with SMTP id kb8mr1881923pjb.113.1583909080607; Tue, 10 Mar 2020 23:44:40 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH 14/16] target/arm: Reuse sve_probe_page for scatter stores Date: Tue, 10 Mar 2020 23:44:18 -0700 Message-Id: <20200311064420.30606-15-richard.henderson@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200311064420.30606-1-richard.henderson@linaro.org> References: <20200311064420.30606-1-richard.henderson@linaro.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::1044 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, qemu-arm@nongnu.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @linaro.org) Content-Type: text/plain; charset="utf-8" Signed-off-by: Richard Henderson Reviewed-by: Peter Maydell --- target/arm/sve_helper.c | 182 ++++++++++++++++++++++++---------------- 1 file changed, 111 insertions(+), 71 deletions(-) diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index a33699adf8..e8b8f2560d 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -5414,94 +5414,134 @@ DO_LDFF1_ZPZ_D(dd_be, zd, MO_64) =20 /* Stores with a vector index. */ =20 -static void sve_st1_zs(CPUARMState *env, void *vd, void *vg, void *vm, - target_ulong base, uint32_t desc, uintptr_t ra, - zreg_off_fn *off_fn, sve_ldst1_tlb_fn *tlb_fn) +static inline QEMU_ALWAYS_INLINE +void sve_st1_z(CPUARMState *env, void *vd, uint64_t *vg, void *vm, + target_ulong base, uint32_t desc, uintptr_t retaddr, + int esize, int msize, zreg_off_fn *off_fn, + sve_ldst1_host_fn *host_fn, + sve_ldst1_tlb_fn *tlb_fn) { const int scale =3D extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, = 2); - intptr_t i, oprsz =3D simd_oprsz(desc); + const int mmu_idx =3D cpu_mmu_index(env, false); + const intptr_t reg_max =3D simd_oprsz(desc); + void *host[ARM_MAX_VQ * 4]; + intptr_t reg_off, i; + SVEHostPage info, info2; =20 - for (i =3D 0; i < oprsz; ) { - uint16_t pg =3D *(uint16_t *)(vg + H1_2(i >> 3)); + /* + * Probe all of the elements for host addresses and flags. + */ + i =3D reg_off =3D 0; + do { + uint64_t pg =3D vg[reg_off >> 6]; do { - if (likely(pg & 1)) { - target_ulong off =3D off_fn(vm, i); - tlb_fn(env, vd, i, base + (off << scale), ra); + target_ulong addr =3D base + (off_fn(vm, reg_off) << scale); + target_ulong in_page =3D -(addr | TARGET_PAGE_MASK); + + host[i] =3D NULL; + if (likely((pg >> (reg_off & 63)) & 1)) { + if (likely(in_page >=3D msize)) { + sve_probe_page(&info, false, env, addr, 0, MMU_DATA_ST= ORE, + mmu_idx, retaddr); + host[i] =3D info.host; + } else { + /* + * Element crosses the page boundary. + * Probe both pages, but do not record the host addres= s, + * so that we use the slow path. + */ + sve_probe_page(&info, false, env, addr, 0, + MMU_DATA_STORE, mmu_idx, retaddr); + sve_probe_page(&info2, false, env, addr + in_page, 0, + MMU_DATA_STORE, mmu_idx, retaddr); + info.flags |=3D info2.flags; + } + + if (unlikely(info.flags & TLB_WATCHPOINT)) { + cpu_check_watchpoint(env_cpu(env), addr, msize, + info.attrs, BP_MEM_WRITE, retaddr= ); + } + /* TODO: MTE check. */ } - i +=3D 4, pg >>=3D 4; - } while (i & 15); - } -} + i +=3D 1; + reg_off +=3D esize; + } while (reg_off & 63); + } while (reg_off < reg_max); =20 -static void sve_st1_zd(CPUARMState *env, void *vd, void *vg, void *vm, - target_ulong base, uint32_t desc, uintptr_t ra, - zreg_off_fn *off_fn, sve_ldst1_tlb_fn *tlb_fn) -{ - const int scale =3D extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, = 2); - intptr_t i, oprsz =3D simd_oprsz(desc) / 8; - - for (i =3D 0; i < oprsz; i++) { - uint8_t pg =3D *(uint8_t *)(vg + H1(i)); - if (likely(pg & 1)) { - target_ulong off =3D off_fn(vm, i * 8); - tlb_fn(env, vd, i * 8, base + (off << scale), ra); + /* + * Now that we have recognized all exceptions except SyncExternal + * (from TLB_MMIO), which we cannot avoid, perform all of the stores. + * + * Note for the common case of an element in RAM, not crossing a page + * boundary, we have stored the host address in host[]. This doubles + * as a first-level check against the predicate, since only enabled + * elements have non-null host addresses. + */ + i =3D reg_off =3D 0; + do { + void *h =3D host[i]; + if (likely(h !=3D NULL)) { + host_fn(vd, reg_off, h); + } else if ((vg[reg_off >> 6] >> (reg_off & 63)) & 1) { + target_ulong addr =3D base + (off_fn(vm, reg_off) << scale); + tlb_fn(env, vd, reg_off, addr, retaddr); } - } + i +=3D 1; + reg_off +=3D esize; + } while (reg_off < reg_max); } =20 -#define DO_ST1_ZPZ_S(MEM, OFS) \ -void QEMU_FLATTEN HELPER(sve_st##MEM##_##OFS) \ - (CPUARMState *env, void *vd, void *vg, void *vm, \ - target_ulong base, uint32_t desc) \ -{ \ - sve_st1_zs(env, vd, vg, vm, base, desc, GETPC(), \ - off_##OFS##_s, sve_st1##MEM##_tlb); \ +#define DO_ST1_ZPZ_S(MEM, OFS, MSZ) \ +void HELPER(sve_st##MEM##_##OFS)(CPUARMState *env, void *vd, void *vg, = \ + void *vm, target_ulong base, uint32_t des= c) \ +{ = \ + sve_st1_z(env, vd, vg, vm, base, desc, GETPC(), 4, 1 << MSZ, = \ + off_##OFS##_s, sve_st1##MEM##_host, sve_st1##MEM##_tlb); = \ } =20 -#define DO_ST1_ZPZ_D(MEM, OFS) \ -void QEMU_FLATTEN HELPER(sve_st##MEM##_##OFS) \ - (CPUARMState *env, void *vd, void *vg, void *vm, \ - target_ulong base, uint32_t desc) \ -{ \ - sve_st1_zd(env, vd, vg, vm, base, desc, GETPC(), \ - off_##OFS##_d, sve_st1##MEM##_tlb); \ +#define DO_ST1_ZPZ_D(MEM, OFS, MSZ) \ +void HELPER(sve_st##MEM##_##OFS)(CPUARMState *env, void *vd, void *vg, = \ + void *vm, target_ulong base, uint32_t des= c) \ +{ = \ + sve_st1_z(env, vd, vg, vm, base, desc, GETPC(), 8, 1 << MSZ, = \ + off_##OFS##_d, sve_st1##MEM##_host, sve_st1##MEM##_tlb); = \ } =20 -DO_ST1_ZPZ_S(bs, zsu) -DO_ST1_ZPZ_S(hs_le, zsu) -DO_ST1_ZPZ_S(hs_be, zsu) -DO_ST1_ZPZ_S(ss_le, zsu) -DO_ST1_ZPZ_S(ss_be, zsu) +DO_ST1_ZPZ_S(bs, zsu, MO_8) +DO_ST1_ZPZ_S(hs_le, zsu, MO_16) +DO_ST1_ZPZ_S(hs_be, zsu, MO_16) +DO_ST1_ZPZ_S(ss_le, zsu, MO_32) +DO_ST1_ZPZ_S(ss_be, zsu, MO_32) =20 -DO_ST1_ZPZ_S(bs, zss) -DO_ST1_ZPZ_S(hs_le, zss) -DO_ST1_ZPZ_S(hs_be, zss) -DO_ST1_ZPZ_S(ss_le, zss) -DO_ST1_ZPZ_S(ss_be, zss) +DO_ST1_ZPZ_S(bs, zss, MO_8) +DO_ST1_ZPZ_S(hs_le, zss, MO_16) +DO_ST1_ZPZ_S(hs_be, zss, MO_16) +DO_ST1_ZPZ_S(ss_le, zss, MO_32) +DO_ST1_ZPZ_S(ss_be, zss, MO_32) =20 -DO_ST1_ZPZ_D(bd, zsu) -DO_ST1_ZPZ_D(hd_le, zsu) -DO_ST1_ZPZ_D(hd_be, zsu) -DO_ST1_ZPZ_D(sd_le, zsu) -DO_ST1_ZPZ_D(sd_be, zsu) -DO_ST1_ZPZ_D(dd_le, zsu) -DO_ST1_ZPZ_D(dd_be, zsu) +DO_ST1_ZPZ_D(bd, zsu, MO_8) +DO_ST1_ZPZ_D(hd_le, zsu, MO_16) +DO_ST1_ZPZ_D(hd_be, zsu, MO_16) +DO_ST1_ZPZ_D(sd_le, zsu, MO_32) +DO_ST1_ZPZ_D(sd_be, zsu, MO_32) +DO_ST1_ZPZ_D(dd_le, zsu, MO_64) +DO_ST1_ZPZ_D(dd_be, zsu, MO_64) =20 -DO_ST1_ZPZ_D(bd, zss) -DO_ST1_ZPZ_D(hd_le, zss) -DO_ST1_ZPZ_D(hd_be, zss) -DO_ST1_ZPZ_D(sd_le, zss) -DO_ST1_ZPZ_D(sd_be, zss) -DO_ST1_ZPZ_D(dd_le, zss) -DO_ST1_ZPZ_D(dd_be, zss) +DO_ST1_ZPZ_D(bd, zss, MO_8) +DO_ST1_ZPZ_D(hd_le, zss, MO_16) +DO_ST1_ZPZ_D(hd_be, zss, MO_16) +DO_ST1_ZPZ_D(sd_le, zss, MO_32) +DO_ST1_ZPZ_D(sd_be, zss, MO_32) +DO_ST1_ZPZ_D(dd_le, zss, MO_64) +DO_ST1_ZPZ_D(dd_be, zss, MO_64) =20 -DO_ST1_ZPZ_D(bd, zd) -DO_ST1_ZPZ_D(hd_le, zd) -DO_ST1_ZPZ_D(hd_be, zd) -DO_ST1_ZPZ_D(sd_le, zd) -DO_ST1_ZPZ_D(sd_be, zd) -DO_ST1_ZPZ_D(dd_le, zd) -DO_ST1_ZPZ_D(dd_be, zd) +DO_ST1_ZPZ_D(bd, zd, MO_8) +DO_ST1_ZPZ_D(hd_le, zd, MO_16) +DO_ST1_ZPZ_D(hd_be, zd, MO_16) +DO_ST1_ZPZ_D(sd_le, zd, MO_32) +DO_ST1_ZPZ_D(sd_be, zd, MO_32) +DO_ST1_ZPZ_D(dd_le, zd, MO_64) +DO_ST1_ZPZ_D(dd_be, zd, MO_64) =20 #undef DO_ST1_ZPZ_S #undef DO_ST1_ZPZ_D --=20 2.20.1 From nobody Mon Feb 9 21:08:28 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=linaro.org ARC-Seal: i=1; a=rsa-sha256; t=1583909551; cv=none; d=zohomail.com; s=zohoarc; b=C771mmyiuwMpt8nh24wcOOLpepc6Gl2cVTCIEa/CMPZZunRhfd+N1Re+B5ssxhxfqpA/wMavpvKr4qv2xia3J8nNa/RfFtX6IfkA1Th58zGH/x2nF4ezkkcTtCnTC9ItBydZ3I+x5U3CgF1nxIlBfsf1Xu/XbItYvlquHRqehi0= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1583909551; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=JO9UDGNI/3FywIxMw2Y2YRQLM5xOitYFKQKOyJDpZrk=; b=FWvU7JSz3lXZZsmgiIo9mJ8ElnyiwcuUXMv8E62XnhQ5NkP23kl09m2ldvDR6BCNKfY5Kjo2Jab0PnRmbJNgDR240zG06QA0TU5+/uPIlJOy8Q4Qk1abvd4Rtp+sGEDQ9ETE0wfQ2zGGIibFOPlQjaBXjUx3Yb9+zYaW7lqHkjE= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1583909551352255.28507025556019; Tue, 10 Mar 2020 23:52:31 -0700 (PDT) Received: from localhost ([::1]:46616 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jBvEB-0000m5-9B for importer@patchew.org; Wed, 11 Mar 2020 02:52:27 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:59700) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jBv6j-0006Ud-Kv for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:47 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jBv6h-00025F-Ro for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:45 -0400 Received: from mail-pf1-x441.google.com ([2607:f8b0:4864:20::441]:36836) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1jBv6h-00024P-KX for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:43 -0400 Received: by mail-pf1-x441.google.com with SMTP id i13so751166pfe.3 for ; Tue, 10 Mar 2020 23:44:43 -0700 (PDT) Received: from localhost.localdomain (97-126-123-70.tukw.qwest.net. [97.126.123.70]) by smtp.gmail.com with ESMTPSA id z3sm50137807pfz.155.2020.03.10.23.44.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 10 Mar 2020 23:44:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=JO9UDGNI/3FywIxMw2Y2YRQLM5xOitYFKQKOyJDpZrk=; b=M+Sfwx6b6MM1KXMzEOpPGdsCRxA/iZj8DUjKXgTHEyzUTW3VwXzdgwViEcdW+c6zyP zn0xpW2rrCXV5gpIgQKUgWqybgDX3mbePqLm8GRCXbodPhO/U5A36zi0/QUcuubZQUXG c9FC/7vf1XdSoc4FN2H48kQ2EUJFPgCp7rd8gxBfrBAU+vRvQf3HBbJgEjSaJ/tej7JI acnQh5NNkh+cgiISJJssHOuypnUD6LdEgcQuG77TdILlFQIngnQBprYjkmGzZIcibTim q+vN04OuozaQLky0O3veMW2r4nY3cF75y66qgqS7wo6WiDX8Y4VA5N6GG9cdkXWJIFLx JCUw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=JO9UDGNI/3FywIxMw2Y2YRQLM5xOitYFKQKOyJDpZrk=; b=O6uyd2YdwONtorl8F8TWf9aHWT9dqH5ox1xTevtWgVk2R1gWcEFjBgqG2+3wAUnikq KQHstb27UOtdjpuwDnmxVgrphTZbJG3JBlSeI8+EbRxhVOfwSoFikPFRgVB8AQG/u63/ MDP6FHbwk3lD/995ZRSkHa5lprHq2+jL5ngvtl5icaDnCl/ZxRMyo8SaITJzUbutHRfG ECG7r+9dy+PzwZdrsEW2lHKfK3QzXvvyhgWcfd1QUPWcF0kvGDBjQ9Ihl7WzBYNLAIOe M8dG8PlAYXfjNYxatH/wF7qvER/UTSYu/NzZdxNevSEJFIditEwWWZVdA4a3iZxlEXVb Tn3Q== X-Gm-Message-State: ANhLgQ3paJZVsFxZeOtN5OOwn0bjoUty0qJJa8i00OxztbbNxnPLEt8D aukkVkQ+7NZOgZzok8YXvID3ryWyv+o= X-Google-Smtp-Source: ADFU+vv9WymiP6fnCIEYXEA+aqjxq1F1SGzfJdTOErZ1fTz3X4WTYESwMBguVaEPxwmy3g8Tp6/rHA== X-Received: by 2002:a62:be04:: with SMTP id l4mr1473040pff.234.1583909082120; Tue, 10 Mar 2020 23:44:42 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH 15/16] target/arm: Reuse sve_probe_page for gather loads Date: Tue, 10 Mar 2020 23:44:19 -0700 Message-Id: <20200311064420.30606-16-richard.henderson@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200311064420.30606-1-richard.henderson@linaro.org> References: <20200311064420.30606-1-richard.henderson@linaro.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::441 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, qemu-arm@nongnu.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @linaro.org) Content-Type: text/plain; charset="utf-8" Signed-off-by: Richard Henderson Reviewed-by: Peter Maydell --- target/arm/sve_helper.c | 208 +++++++++++++++++++++------------------- 1 file changed, 109 insertions(+), 99 deletions(-) diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index e8b8f2560d..4ad870f1ee 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -5125,130 +5125,140 @@ static target_ulong off_zd_d(void *reg, intptr_t = reg_ofs) return *(uint64_t *)(reg + reg_ofs); } =20 -static void sve_ld1_zs(CPUARMState *env, void *vd, void *vg, void *vm, - target_ulong base, uint32_t desc, uintptr_t ra, - zreg_off_fn *off_fn, sve_ldst1_tlb_fn *tlb_fn) +static inline QEMU_ALWAYS_INLINE +void sve_ld1_z(CPUARMState *env, void *vd, uint64_t *vg, void *vm, + target_ulong base, uint32_t desc, uintptr_t retaddr, + int esize, int msize, zreg_off_fn *off_fn, + sve_ldst1_host_fn *host_fn, + sve_ldst1_tlb_fn *tlb_fn) { const int scale =3D extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, = 2); - intptr_t i, oprsz =3D simd_oprsz(desc); - ARMVectorReg scratch =3D { }; + const int mmu_idx =3D cpu_mmu_index(env, false); + const intptr_t reg_max =3D simd_oprsz(desc); + ARMVectorReg scratch; + intptr_t reg_off; + SVEHostPage info, info2; =20 - for (i =3D 0; i < oprsz; ) { - uint16_t pg =3D *(uint16_t *)(vg + H1_2(i >> 3)); + memset(&scratch, 0, reg_max); + reg_off =3D 0; + do { + uint64_t pg =3D vg[reg_off >> 6]; do { if (likely(pg & 1)) { - target_ulong off =3D off_fn(vm, i); - tlb_fn(env, &scratch, i, base + (off << scale), ra); + target_ulong addr =3D base + (off_fn(vm, reg_off) << scale= ); + target_ulong in_page =3D -(addr | TARGET_PAGE_MASK); + + sve_probe_page(&info, false, env, addr, 0, MMU_DATA_LOAD, + mmu_idx, retaddr); + + if (likely(in_page >=3D msize)) { + if (unlikely(info.flags & TLB_WATCHPOINT)) { + cpu_check_watchpoint(env_cpu(env), addr, msize, + info.attrs, BP_MEM_READ, reta= ddr); + } + /* TODO: MTE check */ + host_fn(&scratch, reg_off, info.host); + } else { + /* Element crosses the page boundary. */ + sve_probe_page(&info2, false, env, addr + in_page, 0, + MMU_DATA_LOAD, mmu_idx, retaddr); + if (unlikely((info.flags | info2.flags) & TLB_WATCHPOI= NT)) { + cpu_check_watchpoint(env_cpu(env), addr, + msize, info.attrs, + BP_MEM_READ, retaddr); + } + /* TODO: MTE check */ + tlb_fn(env, &scratch, reg_off, addr, retaddr); + } } - i +=3D 4, pg >>=3D 4; - } while (i & 15); - } + reg_off +=3D esize; + pg >>=3D esize; + } while (reg_off & 63); + } while (reg_off < reg_max); =20 /* Wait until all exceptions have been raised to write back. */ - memcpy(vd, &scratch, oprsz); + memcpy(vd, &scratch, reg_max); } =20 -static void sve_ld1_zd(CPUARMState *env, void *vd, void *vg, void *vm, - target_ulong base, uint32_t desc, uintptr_t ra, - zreg_off_fn *off_fn, sve_ldst1_tlb_fn *tlb_fn) -{ - const int scale =3D extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, = 2); - intptr_t i, oprsz =3D simd_oprsz(desc) / 8; - ARMVectorReg scratch =3D { }; - - for (i =3D 0; i < oprsz; i++) { - uint8_t pg =3D *(uint8_t *)(vg + H1(i)); - if (likely(pg & 1)) { - target_ulong off =3D off_fn(vm, i * 8); - tlb_fn(env, &scratch, i * 8, base + (off << scale), ra); - } - } - - /* Wait until all exceptions have been raised to write back. */ - memcpy(vd, &scratch, oprsz * 8); +#define DO_LD1_ZPZ_S(MEM, OFS, MSZ) \ +void HELPER(sve_ld##MEM##_##OFS)(CPUARMState *env, void *vd, void *vg, = \ + void *vm, target_ulong base, uint32_t des= c) \ +{ = \ + sve_ld1_z(env, vd, vg, vm, base, desc, GETPC(), 4, 1 << MSZ, = \ + off_##OFS##_s, sve_ld1##MEM##_host, sve_ld1##MEM##_tlb); = \ } =20 -#define DO_LD1_ZPZ_S(MEM, OFS) \ -void QEMU_FLATTEN HELPER(sve_ld##MEM##_##OFS) \ - (CPUARMState *env, void *vd, void *vg, void *vm, \ - target_ulong base, uint32_t desc) \ -{ \ - sve_ld1_zs(env, vd, vg, vm, base, desc, GETPC(), \ - off_##OFS##_s, sve_ld1##MEM##_tlb); \ +#define DO_LD1_ZPZ_D(MEM, OFS, MSZ) \ +void HELPER(sve_ld##MEM##_##OFS)(CPUARMState *env, void *vd, void *vg, = \ + void *vm, target_ulong base, uint32_t des= c) \ +{ = \ + sve_ld1_z(env, vd, vg, vm, base, desc, GETPC(), 8, 1 << MSZ, = \ + off_##OFS##_d, sve_ld1##MEM##_host, sve_ld1##MEM##_tlb); = \ } =20 -#define DO_LD1_ZPZ_D(MEM, OFS) \ -void QEMU_FLATTEN HELPER(sve_ld##MEM##_##OFS) \ - (CPUARMState *env, void *vd, void *vg, void *vm, \ - target_ulong base, uint32_t desc) \ -{ \ - sve_ld1_zd(env, vd, vg, vm, base, desc, GETPC(), \ - off_##OFS##_d, sve_ld1##MEM##_tlb); \ -} +DO_LD1_ZPZ_S(bsu, zsu, MO_8) +DO_LD1_ZPZ_S(bsu, zss, MO_8) +DO_LD1_ZPZ_D(bdu, zsu, MO_8) +DO_LD1_ZPZ_D(bdu, zss, MO_8) +DO_LD1_ZPZ_D(bdu, zd, MO_8) =20 -DO_LD1_ZPZ_S(bsu, zsu) -DO_LD1_ZPZ_S(bsu, zss) -DO_LD1_ZPZ_D(bdu, zsu) -DO_LD1_ZPZ_D(bdu, zss) -DO_LD1_ZPZ_D(bdu, zd) +DO_LD1_ZPZ_S(bss, zsu, MO_8) +DO_LD1_ZPZ_S(bss, zss, MO_8) +DO_LD1_ZPZ_D(bds, zsu, MO_8) +DO_LD1_ZPZ_D(bds, zss, MO_8) +DO_LD1_ZPZ_D(bds, zd, MO_8) =20 -DO_LD1_ZPZ_S(bss, zsu) -DO_LD1_ZPZ_S(bss, zss) -DO_LD1_ZPZ_D(bds, zsu) -DO_LD1_ZPZ_D(bds, zss) -DO_LD1_ZPZ_D(bds, zd) +DO_LD1_ZPZ_S(hsu_le, zsu, MO_16) +DO_LD1_ZPZ_S(hsu_le, zss, MO_16) +DO_LD1_ZPZ_D(hdu_le, zsu, MO_16) +DO_LD1_ZPZ_D(hdu_le, zss, MO_16) +DO_LD1_ZPZ_D(hdu_le, zd, MO_16) =20 -DO_LD1_ZPZ_S(hsu_le, zsu) -DO_LD1_ZPZ_S(hsu_le, zss) -DO_LD1_ZPZ_D(hdu_le, zsu) -DO_LD1_ZPZ_D(hdu_le, zss) -DO_LD1_ZPZ_D(hdu_le, zd) +DO_LD1_ZPZ_S(hsu_be, zsu, MO_16) +DO_LD1_ZPZ_S(hsu_be, zss, MO_16) +DO_LD1_ZPZ_D(hdu_be, zsu, MO_16) +DO_LD1_ZPZ_D(hdu_be, zss, MO_16) +DO_LD1_ZPZ_D(hdu_be, zd, MO_16) =20 -DO_LD1_ZPZ_S(hsu_be, zsu) -DO_LD1_ZPZ_S(hsu_be, zss) -DO_LD1_ZPZ_D(hdu_be, zsu) -DO_LD1_ZPZ_D(hdu_be, zss) -DO_LD1_ZPZ_D(hdu_be, zd) +DO_LD1_ZPZ_S(hss_le, zsu, MO_16) +DO_LD1_ZPZ_S(hss_le, zss, MO_16) +DO_LD1_ZPZ_D(hds_le, zsu, MO_16) +DO_LD1_ZPZ_D(hds_le, zss, MO_16) +DO_LD1_ZPZ_D(hds_le, zd, MO_16) =20 -DO_LD1_ZPZ_S(hss_le, zsu) -DO_LD1_ZPZ_S(hss_le, zss) -DO_LD1_ZPZ_D(hds_le, zsu) -DO_LD1_ZPZ_D(hds_le, zss) -DO_LD1_ZPZ_D(hds_le, zd) +DO_LD1_ZPZ_S(hss_be, zsu, MO_16) +DO_LD1_ZPZ_S(hss_be, zss, MO_16) +DO_LD1_ZPZ_D(hds_be, zsu, MO_16) +DO_LD1_ZPZ_D(hds_be, zss, MO_16) +DO_LD1_ZPZ_D(hds_be, zd, MO_16) =20 -DO_LD1_ZPZ_S(hss_be, zsu) -DO_LD1_ZPZ_S(hss_be, zss) -DO_LD1_ZPZ_D(hds_be, zsu) -DO_LD1_ZPZ_D(hds_be, zss) -DO_LD1_ZPZ_D(hds_be, zd) +DO_LD1_ZPZ_S(ss_le, zsu, MO_32) +DO_LD1_ZPZ_S(ss_le, zss, MO_32) +DO_LD1_ZPZ_D(sdu_le, zsu, MO_32) +DO_LD1_ZPZ_D(sdu_le, zss, MO_32) +DO_LD1_ZPZ_D(sdu_le, zd, MO_32) =20 -DO_LD1_ZPZ_S(ss_le, zsu) -DO_LD1_ZPZ_S(ss_le, zss) -DO_LD1_ZPZ_D(sdu_le, zsu) -DO_LD1_ZPZ_D(sdu_le, zss) -DO_LD1_ZPZ_D(sdu_le, zd) +DO_LD1_ZPZ_S(ss_be, zsu, MO_32) +DO_LD1_ZPZ_S(ss_be, zss, MO_32) +DO_LD1_ZPZ_D(sdu_be, zsu, MO_32) +DO_LD1_ZPZ_D(sdu_be, zss, MO_32) +DO_LD1_ZPZ_D(sdu_be, zd, MO_32) =20 -DO_LD1_ZPZ_S(ss_be, zsu) -DO_LD1_ZPZ_S(ss_be, zss) -DO_LD1_ZPZ_D(sdu_be, zsu) -DO_LD1_ZPZ_D(sdu_be, zss) -DO_LD1_ZPZ_D(sdu_be, zd) +DO_LD1_ZPZ_D(sds_le, zsu, MO_32) +DO_LD1_ZPZ_D(sds_le, zss, MO_32) +DO_LD1_ZPZ_D(sds_le, zd, MO_32) =20 -DO_LD1_ZPZ_D(sds_le, zsu) -DO_LD1_ZPZ_D(sds_le, zss) -DO_LD1_ZPZ_D(sds_le, zd) +DO_LD1_ZPZ_D(sds_be, zsu, MO_32) +DO_LD1_ZPZ_D(sds_be, zss, MO_32) +DO_LD1_ZPZ_D(sds_be, zd, MO_32) =20 -DO_LD1_ZPZ_D(sds_be, zsu) -DO_LD1_ZPZ_D(sds_be, zss) -DO_LD1_ZPZ_D(sds_be, zd) +DO_LD1_ZPZ_D(dd_le, zsu, MO_64) +DO_LD1_ZPZ_D(dd_le, zss, MO_64) +DO_LD1_ZPZ_D(dd_le, zd, MO_64) =20 -DO_LD1_ZPZ_D(dd_le, zsu) -DO_LD1_ZPZ_D(dd_le, zss) -DO_LD1_ZPZ_D(dd_le, zd) - -DO_LD1_ZPZ_D(dd_be, zsu) -DO_LD1_ZPZ_D(dd_be, zss) -DO_LD1_ZPZ_D(dd_be, zd) +DO_LD1_ZPZ_D(dd_be, zsu, MO_64) +DO_LD1_ZPZ_D(dd_be, zss, MO_64) +DO_LD1_ZPZ_D(dd_be, zd, MO_64) =20 #undef DO_LD1_ZPZ_S #undef DO_LD1_ZPZ_D --=20 2.20.1 From nobody Mon Feb 9 21:08:28 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=linaro.org ARC-Seal: i=1; a=rsa-sha256; t=1583909643; cv=none; d=zohomail.com; s=zohoarc; b=UT2dTGPmfXGr5/+lqIPXoC0bEKlOpVbby8N8wJ23ZgJVKADlRXhTIZCefYt2mZ2QM2j3w+zR1iRvvTzlSxNOl4dHgkpfLY0ojq4G5A6KRFOqo3/OOzOtDo8VXNbHsEJJG4OeClScMilXhUdYCBgoaz+aZpfPobBrgJds3a5RqwM= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1583909643; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=j19UswlgyGkpUPAbfqRH89tG+gAVqbIVGxthtVV4itA=; b=F41agZGwXwD9H7Z0HAasFKT4eoIX2RLNYJXi3rMhQfXSZIX0TcJmXKjYK/sMakARZe4zhxFdH7lEZyrkcZbIqXLPDL2NHmQ0r6K5rzO4IfoJPUld+dd5TRsiFPhsr9FPekovO6Ghm1/+bNZk+KK37Heu/7IriBhSdk6v5+KZwts= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1583909643935435.22346917086895; Tue, 10 Mar 2020 23:54:03 -0700 (PDT) Received: from localhost ([::1]:46660 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jBvFi-0003Xh-LV for importer@patchew.org; Wed, 11 Mar 2020 02:54:02 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:59745) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jBv6l-0006Xj-TB for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:49 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jBv6j-00026Y-Kh for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:46 -0400 Received: from mail-pg1-x541.google.com ([2607:f8b0:4864:20::541]:40532) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1jBv6j-00025r-F3 for qemu-devel@nongnu.org; Wed, 11 Mar 2020 02:44:45 -0400 Received: by mail-pg1-x541.google.com with SMTP id t24so635026pgj.7 for ; Tue, 10 Mar 2020 23:44:44 -0700 (PDT) Received: from localhost.localdomain (97-126-123-70.tukw.qwest.net. [97.126.123.70]) by smtp.gmail.com with ESMTPSA id z3sm50137807pfz.155.2020.03.10.23.44.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 10 Mar 2020 23:44:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=j19UswlgyGkpUPAbfqRH89tG+gAVqbIVGxthtVV4itA=; b=CrtmWX0NIBqwTNOCpdyYTnzEErpf+EQg+JsW/Sv2a7jjjAG5F1LzZJLIq8zeni96rz z4fCuaKRU9ANW99qtYHBq5vK3OeZYUy4/crU7iXPTPFuIC1PqGKEVVKXuw1aDFQ0RGZ0 ldl6/jqFKrH4g3p8AQHi8E3fd66wA78XLLuMZkHukNdWL4EeJFD7IlbmZHSihj4DcR7X QqdQI/mwtZwBscZo68mxG0yGy1MyUIo3K4hDxjJujyPtJgAIICvgqRyLP9Q+1dyZo1dt xIgEce01eEgsqsOylmCdyivRQlJYCBxx2zTabKZzh2mB74p0zWvw3YHRZqKwjXHlnAKg vhGw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=j19UswlgyGkpUPAbfqRH89tG+gAVqbIVGxthtVV4itA=; b=j1PLcEgOu3TDjr9dhZZymrMAPEXg7736pXUO5ccpqHT+/4qFk5BZRIJ3HLisIfT3Tv Lt81BV4etKAdJZduCA+pB4moKhx+HDr5ghmvAKLo6gC7/jNOcqUnw38vqWaR4sKiChR5 BeZwcH87fGioFRiVy9TK3Zs2rZz7mG0AK6di7PCqh7GeimqxnTeicHppBelda9pOYa6U 9Tgxn3EgrpzGcGyHW5puEncfv7tkWRd+ZHHC4T7wa+9AAcHwjy+PB9tdyRWdITt6sZce BFDMq1IX6ZUqp2r59j+w8hk13CTMa0KeQdji9jMuc6jqoNvksfRTHmpFaIaGOhrarr+z b7dw== X-Gm-Message-State: ANhLgQ37KnOfmeRGt5orNST7HmyHpuqoZ8eaVNcbalrzhMGNGK6Vwoh1 2j8KuEPBaB1KOHnzvlw2Ovcq+TEMcyA= X-Google-Smtp-Source: ADFU+vuJraqm0vVdGSTHT8hV4/ij8d2+cQ6D7a/ztx5Erckk3iEOavFOZ491zm8NJQ0f8Q9VF0wiiA== X-Received: by 2002:a63:82c2:: with SMTP id w185mr1505089pgd.382.1583909083428; Tue, 10 Mar 2020 23:44:43 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH 16/16] target/arm: Remove sve_memopidx Date: Tue, 10 Mar 2020 23:44:20 -0700 Message-Id: <20200311064420.30606-17-richard.henderson@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200311064420.30606-1-richard.henderson@linaro.org> References: <20200311064420.30606-1-richard.henderson@linaro.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::541 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, qemu-arm@nongnu.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @linaro.org) Content-Type: text/plain; charset="utf-8" None of the sve helpers use TCGMemOpIdx any longer, so we can stop passing it. Signed-off-by: Richard Henderson Reviewed-by: Peter Maydell --- target/arm/internals.h | 5 ----- target/arm/sve_helper.c | 14 +++++++------- target/arm/translate-sve.c | 17 +++-------------- 3 files changed, 10 insertions(+), 26 deletions(-) diff --git a/target/arm/internals.h b/target/arm/internals.h index e633aff36e..a833e3941d 100644 --- a/target/arm/internals.h +++ b/target/arm/internals.h @@ -979,11 +979,6 @@ static inline int arm_num_ctx_cmps(ARMCPU *cpu) } } =20 -/* Note make_memop_idx reserves 4 bits for mmu_idx, and MO_BSWAP is bit 3. - * Thus a TCGMemOpIdx, without any MO_ALIGN bits, fits in 8 bits. - */ -#define MEMOPIDX_SHIFT 8 - /** * v7m_using_psp: Return true if using process stack pointer * Return true if the CPU is currently using the process stack diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index 4ad870f1ee..80453953ad 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -4447,7 +4447,7 @@ void sve_ldN_r(CPUARMState *env, uint64_t *vg, const = target_ulong addr, sve_ldst1_host_fn *host_fn, sve_ldst1_tlb_fn *tlb_fn) { - const unsigned rd =3D extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT= , 5); + const unsigned rd =3D simd_data(desc); const intptr_t reg_max =3D simd_oprsz(desc); intptr_t reg_off, reg_last, mem_off; SVEContLdSt info; @@ -4703,7 +4703,7 @@ void sve_ldnfff1_r(CPUARMState *env, void *vg, const = target_ulong addr, sve_ldst1_host_fn *host_fn, sve_ldst1_tlb_fn *tlb_fn) { - const unsigned rd =3D extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT= , 5); + const unsigned rd =3D simd_data(desc); void *vd =3D &env->vfp.zregs[rd]; const intptr_t reg_max =3D simd_oprsz(desc); intptr_t reg_off, mem_off, reg_last; @@ -4926,7 +4926,7 @@ void sve_stN_r(CPUARMState *env, uint64_t *vg, target= _ulong addr, uint32_t desc, sve_ldst1_host_fn *host_fn, sve_ldst1_tlb_fn *tlb_fn) { - const unsigned rd =3D extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT= , 5); + const unsigned rd =3D simd_data(desc); const intptr_t reg_max =3D simd_oprsz(desc); intptr_t reg_off, reg_last, mem_off; SVEContLdSt info; @@ -5132,9 +5132,9 @@ void sve_ld1_z(CPUARMState *env, void *vd, uint64_t *= vg, void *vm, sve_ldst1_host_fn *host_fn, sve_ldst1_tlb_fn *tlb_fn) { - const int scale =3D extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, = 2); const int mmu_idx =3D cpu_mmu_index(env, false); const intptr_t reg_max =3D simd_oprsz(desc); + const int scale =3D simd_data(desc); ARMVectorReg scratch; intptr_t reg_off; SVEHostPage info, info2; @@ -5277,10 +5277,10 @@ void sve_ldff1_z(CPUARMState *env, void *vd, uint64= _t *vg, void *vm, sve_ldst1_tlb_fn *tlb_fn) { const int mmu_idx =3D cpu_mmu_index(env, false); - const int scale =3D extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, = 2); + const intptr_t reg_max =3D simd_oprsz(desc); + const int scale =3D simd_data(desc); const int esize =3D 1 << esz; const int msize =3D 1 << msz; - const intptr_t reg_max =3D simd_oprsz(desc); intptr_t reg_off; SVEHostPage info; target_ulong addr, in_page; @@ -5431,9 +5431,9 @@ void sve_st1_z(CPUARMState *env, void *vd, uint64_t *= vg, void *vm, sve_ldst1_host_fn *host_fn, sve_ldst1_tlb_fn *tlb_fn) { - const int scale =3D extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, = 2); const int mmu_idx =3D cpu_mmu_index(env, false); const intptr_t reg_max =3D simd_oprsz(desc); + const int scale =3D simd_data(desc); void *host[ARM_MAX_VQ * 4]; intptr_t reg_off, i; SVEHostPage info, info2; diff --git a/target/arm/translate-sve.c b/target/arm/translate-sve.c index b35bad245e..7bd7de80e6 100644 --- a/target/arm/translate-sve.c +++ b/target/arm/translate-sve.c @@ -4584,11 +4584,6 @@ static const uint8_t dtype_esz[16] =3D { 3, 2, 1, 3 }; =20 -static TCGMemOpIdx sve_memopidx(DisasContext *s, int dtype) -{ - return make_memop_idx(s->be_data | dtype_mop[dtype], get_mem_index(s)); -} - static void do_mem_zpa(DisasContext *s, int zt, int pg, TCGv_i64 addr, int dtype, gen_helper_gvec_mem *fn) { @@ -4601,9 +4596,7 @@ static void do_mem_zpa(DisasContext *s, int zt, int p= g, TCGv_i64 addr, * registers as pointers, so encode the regno into the data field. * For consistency, do this even for LD1. */ - desc =3D sve_memopidx(s, dtype); - desc |=3D zt << MEMOPIDX_SHIFT; - desc =3D simd_desc(vsz, vsz, desc); + desc =3D simd_desc(vsz, vsz, zt); t_desc =3D tcg_const_i32(desc); t_pg =3D tcg_temp_new_ptr(); =20 @@ -4835,9 +4828,7 @@ static void do_ldrq(DisasContext *s, int zt, int pg, = TCGv_i64 addr, int msz) int desc, poff; =20 /* Load the first quadword using the normal predicated load helpers. = */ - desc =3D sve_memopidx(s, msz_dtype(s, msz)); - desc |=3D zt << MEMOPIDX_SHIFT; - desc =3D simd_desc(16, 16, desc); + desc =3D simd_desc(16, 16, zt); t_desc =3D tcg_const_i32(desc); =20 poff =3D pred_full_reg_offset(s, pg); @@ -5066,9 +5057,7 @@ static void do_mem_zpz(DisasContext *s, int zt, int p= g, int zm, TCGv_i32 t_desc; int desc; =20 - desc =3D sve_memopidx(s, msz_dtype(s, msz)); - desc |=3D scale << MEMOPIDX_SHIFT; - desc =3D simd_desc(vsz, vsz, desc); + desc =3D simd_desc(vsz, vsz, scale); t_desc =3D tcg_const_i32(desc); =20 tcg_gen_addi_ptr(t_pg, cpu_env, pred_full_reg_offset(s, pg)); --=20 2.20.1