From nobody Sun May 19 14:32:52 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=linaro.org ARC-Seal: i=1; a=rsa-sha256; t=1588264636; cv=none; d=zohomail.com; s=zohoarc; b=RfP6A7XtkbpS+/L4INo4JwG5CJFVNLUjWdPLHZQOTFOuBg9dfyIODVQXo2AO2MS0+9FHbefzkM4Ov72BS0n7giXBkLSRn6Y43DUkg+s5mqerr+m7ltdrgHhfpDHx9CpyuwUu9kqAq1m82BcxIclcNT8zzMo04iGs+lRVXUfUYPI= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1588264636; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=60NHpUUrot7w8V6Qsa37kh//ITveCZoOpAk4TR6uozc=; b=WhKGY6h7NwH/SI6zmN4LKtAROANXETi2/Wrbs6K5xM9jNYIRezHkshjlkS0mxPfV5XiPk0zlNxEQTASQqvBejN9L/yuK7FqJa6dOOBTpTrDpi4t/QMy/RFDYfRBU9KwhOfSacFFxmzNPdDN8lgleaLfFSW2+4B0Sivh5chelq1c= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1588264636885232.45064057199204; Thu, 30 Apr 2020 09:37:16 -0700 (PDT) Received: from localhost ([::1]:51942 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jUCBX-0005Iw-Hu for importer@patchew.org; Thu, 30 Apr 2020 12:37:15 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:49288) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jUC2u-0001VL-N4 for qemu-devel@nongnu.org; Thu, 30 Apr 2020 12:28:22 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.90_1) (envelope-from ) id 1jUC2t-0006Ll-02 for qemu-devel@nongnu.org; Thu, 30 Apr 2020 12:28:20 -0400 Received: from mail-pj1-x1041.google.com ([2607:f8b0:4864:20::1041]:39140) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jUC2s-0006LS-K6 for qemu-devel@nongnu.org; Thu, 30 Apr 2020 12:28:18 -0400 Received: by mail-pj1-x1041.google.com with SMTP id e6so906728pjt.4 for ; Thu, 30 Apr 2020 09:28:18 -0700 (PDT) Received: from localhost.localdomain (174-21-149-226.tukw.qwest.net. [174.21.149.226]) by smtp.gmail.com with ESMTPSA id j5sm243514pfh.58.2020.04.30.09.28.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Apr 2020 09:28:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=60NHpUUrot7w8V6Qsa37kh//ITveCZoOpAk4TR6uozc=; b=WGRsa/jb4slKUiLxKpF6qoZYYNfNvBuoETV4tG9wSE13m/jUvjchzXgxa1rSfXRT13 /CgcsPHr5LCUILO9VY7BX6SJ0SKYsIcXP/+LJp+Wnm0J7TR61kaqy9QLdomzo+4TmcbS 8NaP9oQ7Sj/L/sgx5tLQoQA9VUziIuJ/QE3IFF6TPF2hhYxagvojGDS0wLwBA4Dw6R94 AI1HRxQ5a0GgFG0uYtwZUkbSXyonU60+rLJCwmAOePsqnvCichA8WvF2yFaX5OZa8CXd Vm3AiDeIgL29yTuSip76/6rIkEF6nj5BQmDXHNGZBxDUqpeXht7GY/re6SoEQG9VkGPL cq6w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=60NHpUUrot7w8V6Qsa37kh//ITveCZoOpAk4TR6uozc=; b=U2sTDm8bu4TxoTM1mTlhQk2pzKjCLA7YVvOHOMeUPp+FP2RjHO8OMcncXwHKiLEb4N VdptbqmgcfRxPVZZ+9RmdoOdg0aK/16lzr88Wl/YvrqX9TJqBVXjcf7bVnsawMKoxb01 W/QjMM05PBIT2FLnEraqSmBrSAyTxmv2VVl3e1zYr55RTWwdq2yZLyf2R6IpeoIj3xfm pUHpZA99xqnIQlkX91uRnLCeHkyl8MVk0C9jUnCI3wpZVPjSE49luEA/YD286oF4uKmG 4DV4mgfOk0M0c22kmUpneOUEfVb4RLEWC9AqMPnaL7sMY+qf493/JcxdBv3aVJhsEmd/ +MAA== X-Gm-Message-State: AGi0PuYtS+wNALYl01ARIvlHWO0ORGOU4Vu2SNtEo/cjAWayrh3PwRBm qKXiksg00ZP0Q9MtIESlJWXVm1fgfVM= X-Google-Smtp-Source: APiQypLRep7am6BYi3UWF0K0/qL9z+NQ/MzqfEVLPSGG2uEcEuDFNSVloqUi2msTAFcB50uO3YGOEw== X-Received: by 2002:a17:90a:276a:: with SMTP id o97mr3964206pje.194.1588264096700; Thu, 30 Apr 2020 09:28:16 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH v4 01/18] exec: Add block comments for watchpoint routines Date: Thu, 30 Apr 2020 09:27:56 -0700 Message-Id: <20200430162813.17671-2-richard.henderson@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200430162813.17671-1-richard.henderson@linaro.org> References: <20200430162813.17671-1-richard.henderson@linaro.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2607:f8b0:4864:20::1041; envelope-from=richard.henderson@linaro.org; helo=mail-pj1-x1041.google.com X-detected-operating-system: by eggs.gnu.org: Error: [-] PROGRAM ABORT : Malformed IPv6 address (bad octet value). Location : parse_addr6(), p0f-client.c:67 X-Received-From: 2607:f8b0:4864:20::1041 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, alex.bennee@linaro.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @linaro.org) Content-Type: text/plain; charset="utf-8" Reviewed-by: Peter Maydell Signed-off-by: Richard Henderson --- include/hw/core/cpu.h | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/include/hw/core/cpu.h b/include/hw/core/cpu.h index 5bf94d28cf..07f7698155 100644 --- a/include/hw/core/cpu.h +++ b/include/hw/core/cpu.h @@ -1100,8 +1100,31 @@ int cpu_watchpoint_remove(CPUState *cpu, vaddr addr, vaddr len, int flags); void cpu_watchpoint_remove_by_ref(CPUState *cpu, CPUWatchpoint *watchpoint= ); void cpu_watchpoint_remove_all(CPUState *cpu, int mask); + +/** + * cpu_check_watchpoint: + * @cpu: cpu context + * @addr: guest virtual address + * @len: access length + * @attrs: memory access attributes + * @flags: watchpoint access type + * @ra: unwind return address + * + * Check for a watchpoint hit in [addr, addr+len) of the type + * specified by @flags. Exit via exception with a hit. + */ void cpu_check_watchpoint(CPUState *cpu, vaddr addr, vaddr len, MemTxAttrs attrs, int flags, uintptr_t ra); + +/** + * cpu_watchpoint_address_matches: + * @cpu: cpu context + * @addr: guest virtual address + * @len: access length + * + * Return the watchpoint flags that apply to [addr, addr+len). + * If no watchpoint is registered for the range, the result is 0. + */ int cpu_watchpoint_address_matches(CPUState *cpu, vaddr addr, vaddr len); #endif =20 --=20 2.20.1 From nobody Sun May 19 14:32:52 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=linaro.org ARC-Seal: i=1; a=rsa-sha256; t=1588264586; cv=none; d=zohomail.com; s=zohoarc; b=bxs8RIUfK22pcPJEbFXoXENg+8VAWIsP9KH4IHUDIau0iopHCIQ8crDhjeWjARokeTc3lZnP9J1IpwGnXpgJ4G1gNvQT+5jcOMQsRFAO2CLrFaiIwdGwOSOO7zCyRUYeTDb2hlFVOnbhBZaFBl3G0fXL2gT+NTEvM7RjDt/jN1M= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1588264586; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=NN/QLBosnfSE6MOjBuSmBLZSNupnPgsxwETGi0A5VuM=; b=CwHeNW2E0JTc2HchhoahSqglU7qujjFbswyg0bzS7MpIjDqlW9JlyeX89x2GICh+EUbuSG0OOmSl1EBfRxan7BPuXvIgt/MljZizUDTdlxcl2MSp4qnV9cHfLT830xWWN7GhqQyd4JOQBcAfX2QaIkspacqDuAQ5eJNFwIuzkdw= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1588264586584705.1456692011571; Thu, 30 Apr 2020 09:36:26 -0700 (PDT) Received: from localhost ([::1]:47486 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jUCAj-0003Ks-54 for importer@patchew.org; Thu, 30 Apr 2020 12:36:25 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:49292) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jUC2v-0001Vh-98 for qemu-devel@nongnu.org; Thu, 30 Apr 2020 12:28:24 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.90_1) (envelope-from ) id 1jUC2u-0006M1-Dc for qemu-devel@nongnu.org; Thu, 30 Apr 2020 12:28:21 -0400 Received: from mail-pl1-x643.google.com ([2607:f8b0:4864:20::643]:41677) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jUC2u-0006Lj-10 for qemu-devel@nongnu.org; Thu, 30 Apr 2020 12:28:20 -0400 Received: by mail-pl1-x643.google.com with SMTP id d24so2419335pll.8 for ; Thu, 30 Apr 2020 09:28:19 -0700 (PDT) Received: from localhost.localdomain (174-21-149-226.tukw.qwest.net. [174.21.149.226]) by smtp.gmail.com with ESMTPSA id j5sm243514pfh.58.2020.04.30.09.28.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Apr 2020 09:28:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=NN/QLBosnfSE6MOjBuSmBLZSNupnPgsxwETGi0A5VuM=; b=WCIbL0bHsTglQoIoaV8dyoQZjDmKocpOyvRdoC8SvDDaqqiyH3LxUHNSqbZihRxZuC q0JK8p3wVhQJPrVsX5Pefqmom8aFFecProtRAdi2QBmdB2sZP7ynniCEtXAw26pshi3C TBW8Z01g73qysRU9zW3Wo1rPUirGki9f8KtM2fCHeywQFdrku07uUENIuNPlAEBNmEdO 9VSXNkRaGYD7pxS7y8K2MUrHWcfr2zvB1xH3TmdNl2utAqy76LINo1eqp/nyYsdMg0y+ 2AsgpL6aMkVjfnVRFNf75IMDr7y6wihY4yQL2bfK6SaXyF/l7oaOsiay+uPhHa6qSS0W eisA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=NN/QLBosnfSE6MOjBuSmBLZSNupnPgsxwETGi0A5VuM=; b=XUgCryIH27SnXh0Rdijo/lW8EFQRLvqw4TCg5OCSAhb/wy42BJ4OEYqh+J8gmniARi +OMkEosDwQaFuWpUslp6TAEBV2jEEZKqLtB90/PUBI3tDOYOJoI6OE/5Igbv9zX1J5Uz YiFs9WS4zrNYbimUydJNEWPj5EzHQNDYCQ5qIWGscZq1xvunKxqpvBgnfv/NwTLb79Xl 6Q0DM6XXW/Kf2IZiERaASBIp9zUIblUBj6mDM8e1yisInJo8mWqZ+8lES5XV6tLuoMPu x4bsJ4plmnK1MaIQFwxCGVNnPBi3MajMpVUdEAFVv2n6xweSqkHkJ93iGmQ67pPVaKcW 14tw== X-Gm-Message-State: AGi0PubvG3us572knfY3Nk2U3VNtA96WPer4qXsAQfQMuBeB0Zkv1ZDG 0TWPqXraggZtdwwpSAsX6JEpzEwpLqE= X-Google-Smtp-Source: APiQypKxTlpcwir3lVQyzL3kUuPn3bbW4QxX3SQFCoNPWkTXBJneR0D8mhIelE2MlQzEpjmEG7WHUA== X-Received: by 2002:a17:90a:de8d:: with SMTP id n13mr3686027pjv.173.1588264098026; Thu, 30 Apr 2020 09:28:18 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH v4 02/18] exec: Fix cpu_watchpoint_address_matches address length Date: Thu, 30 Apr 2020 09:27:57 -0700 Message-Id: <20200430162813.17671-3-richard.henderson@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200430162813.17671-1-richard.henderson@linaro.org> References: <20200430162813.17671-1-richard.henderson@linaro.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2607:f8b0:4864:20::643; envelope-from=richard.henderson@linaro.org; helo=mail-pl1-x643.google.com X-detected-operating-system: by eggs.gnu.org: Error: [-] PROGRAM ABORT : Malformed IPv6 address (bad octet value). Location : parse_addr6(), p0f-client.c:67 X-Received-From: 2607:f8b0:4864:20::643 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, alex.bennee@linaro.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @linaro.org) Content-Type: text/plain; charset="utf-8" The only caller of cpu_watchpoint_address_matches passes TARGET_PAGE_SIZE, so the bug is not currently visible. Reviewed-by: Peter Maydell Signed-off-by: Richard Henderson --- exec.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/exec.c b/exec.c index 2874bb5088..5162f0d12f 100644 --- a/exec.c +++ b/exec.c @@ -1127,7 +1127,7 @@ int cpu_watchpoint_address_matches(CPUState *cpu, vad= dr addr, vaddr len) int ret =3D 0; =20 QTAILQ_FOREACH(wp, &cpu->watchpoints, entry) { - if (watchpoint_address_matches(wp, addr, TARGET_PAGE_SIZE)) { + if (watchpoint_address_matches(wp, addr, len)) { ret |=3D wp->flags; } } --=20 2.20.1 From nobody Sun May 19 14:32:52 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=linaro.org ARC-Seal: i=1; a=rsa-sha256; t=1588264602; cv=none; d=zohomail.com; s=zohoarc; b=nKr6kAr8KDfXcPJPo9NppoYDNTD6ciHuJZ9gSWeuef7Hfsp5rF1+vKVkYOx+p+Y6bLphTxfiiANaHAE+G7xIVUBvfu+RqMMaUrMePkqPobaY1Z/XGKTfGPhpI3bO5Z514A26SEBZzGYIsDjaTmWVS0dxcJnqz2VSsa5IGqTpFKk= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1588264602; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=Ar4ktbgi8ho4D2K0AlAA7ABuossIOEQqKwU/YvDdwbY=; b=OyrX2N4cGfvpZFELtQRBNVECZ7HuhQaNyOxjryeWne2lIEaNAeNhbGPAHXW1De+V6i3p8DIO1jdj3XWNQ4ZP+/KkQjfwNnhxpSo3JQBCxvP8PuxG6tmVon0dSWwEjx4aguSqYvepineMVxprlGZ/c75bZ7RgC+zcw/UPGERN2iQ= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1588264602243326.581081744106; Thu, 30 Apr 2020 09:36:42 -0700 (PDT) Received: from localhost ([::1]:48716 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jUCAy-0003yg-My for importer@patchew.org; Thu, 30 Apr 2020 12:36:40 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:49304) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jUC2z-0001ZA-8D for qemu-devel@nongnu.org; Thu, 30 Apr 2020 12:28:26 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.90_1) (envelope-from ) id 1jUC2v-0006MR-LE for qemu-devel@nongnu.org; Thu, 30 Apr 2020 12:28:24 -0400 Received: from mail-pl1-x641.google.com ([2607:f8b0:4864:20::641]:36074) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jUC2v-0006M6-91 for qemu-devel@nongnu.org; Thu, 30 Apr 2020 12:28:21 -0400 Received: by mail-pl1-x641.google.com with SMTP id f15so2433121plr.3 for ; Thu, 30 Apr 2020 09:28:20 -0700 (PDT) Received: from localhost.localdomain (174-21-149-226.tukw.qwest.net. [174.21.149.226]) by smtp.gmail.com with ESMTPSA id j5sm243514pfh.58.2020.04.30.09.28.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Apr 2020 09:28:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Ar4ktbgi8ho4D2K0AlAA7ABuossIOEQqKwU/YvDdwbY=; b=kPA4ilfVPF+5jUmAE++MclnZDZt6AkP+V2y4utOwmCj2wNC/OsvuUKdEKer2A/8pqi 2YzTkmSUvSwYi1FJTh7eimnc1B5y6F+IgQEGr8xbyaEJOG8n2WJfKg1X1lHWhPUuG40U RMmD5BuMVFw+o3IKCUD8kV8zlziFy9ksx2MPQ8e1stuucWZRVQRQEbs7sPc8+0iH78cg cMJQ16Zx7aKVsVMM/UCpoPTx7P+ijLzA15/Ec0qkTFd4T0blBNhNWQYJNJweNlJjgTTE E9CJLzEdVJOb28u6mcOKcC3QlCU0dO7WmbOI8bSE3Fbi7wi3GYy3QPLFUyir/uwDVJdu JiGg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Ar4ktbgi8ho4D2K0AlAA7ABuossIOEQqKwU/YvDdwbY=; b=E9D96VU1fq4ziedvCgaWuJinvg2AnDbB3v/ZGMBM6pUa7+3lzJQ+hiKAOBFOr4zNwi O2Cs4cvNaFKlORv/vLnv0q5Swgo4V42Yv1cRuRqiXs+wgogr/8ts+tR47Z6BLsX1Arm7 8Hie//7G6PBRiWCGbRZGbHyt/TOYnQ1ZFGUNQdFMDNLMZZyr9D4TocUoaB3GMWz6LfaT fTbmdgwgJrkc/qpM57jAUBPAU+l6sN7yml/Ewg2i0L4LGi1jAkXhljGSbRb67K261MjG tY8kH89pV8AXgVO6gHCMtFXB0VrnV+gusFFB4GHsmPst4FjKGBiWtfGcc+L2ADW6Zpjm FQ+w== X-Gm-Message-State: AGi0Puapfmcy/vA8bznL38tC6X/VTzBNQTJbeWlc1jQDlge3NGfXfm77 vMJkZSE4x0y8DbkZnraN7a7mEpo7S0c= X-Google-Smtp-Source: APiQypJD8rnlv/Bz6w316u37pgsauAwmlZp/smHakdnsG+Qy+5pJD1CIIGLqty8+UdOvEQtKTyb8iw== X-Received: by 2002:a17:90a:d192:: with SMTP id fu18mr3822147pjb.98.1588264099491; Thu, 30 Apr 2020 09:28:19 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH v4 03/18] accel/tcg: Add block comment for probe_access Date: Thu, 30 Apr 2020 09:27:58 -0700 Message-Id: <20200430162813.17671-4-richard.henderson@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200430162813.17671-1-richard.henderson@linaro.org> References: <20200430162813.17671-1-richard.henderson@linaro.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2607:f8b0:4864:20::641; envelope-from=richard.henderson@linaro.org; helo=mail-pl1-x641.google.com X-detected-operating-system: by eggs.gnu.org: Error: [-] PROGRAM ABORT : Malformed IPv6 address (bad octet value). Location : parse_addr6(), p0f-client.c:67 X-Received-From: 2607:f8b0:4864:20::641 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, alex.bennee@linaro.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @linaro.org) Content-Type: text/plain; charset="utf-8" Reviewed-by: Peter Maydell Signed-off-by: Richard Henderson --- include/exec/exec-all.h | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/include/exec/exec-all.h b/include/exec/exec-all.h index 350c4b451b..d656a1f05c 100644 --- a/include/exec/exec-all.h +++ b/include/exec/exec-all.h @@ -330,6 +330,23 @@ static inline void tlb_flush_by_mmuidx_all_cpus_synced= (CPUState *cpu, { } #endif +/** + * probe_access: + * @env: CPUArchState + * @addr: guest virtual address to look up + * @size: size of the access + * @access_type: read, write or execute permission + * @mmu_idx: MMU index to use for lookup + * @retaddr: return address for unwinding + * + * Look up the guest virtual address @addr. Raise an exception if the + * page does not satisfy @access_type. Raise an exception if the + * access (@addr, @size) hits a watchpoint. For writes, mark a clean + * page as dirty. + * + * Finally, return the host address for a page that is backed by RAM, + * or NULL if the page requires I/O. + */ void *probe_access(CPUArchState *env, target_ulong addr, int size, MMUAccessType access_type, int mmu_idx, uintptr_t retad= dr); =20 --=20 2.20.1 From nobody Sun May 19 14:32:52 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=linaro.org ARC-Seal: i=1; a=rsa-sha256; t=1588264768; cv=none; d=zohomail.com; s=zohoarc; b=AA3QO8QddmBZWrvln/Hq28K67w+fuqFDkgKzEyyXJMDj9lo9JE7DL9uc77bp+XstqlAEcvbeWbfgrQubgzYNGcHO9FnDRf5202lhi0dRcqiFteIcuYnxL9HQFugIRxWkPXa9zULVRj2llK8d4IHl6eLmvbYysnhWRZg4rTNcPFw= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1588264768; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=+P5jItpXz/tjAGeO3l2qriIEmglug+qlPtimhug8Qps=; b=d7EubfYV1bAv7sLraDntBW7DayxvzczV1QZFawBkLLmFzKcRie+BqVgXSnCx957FbgP2HNLN5u77MG8vEJUQbv/ctk1qNYVb/3Wx6IFQvnnpSyTwVIHoJjRKdIM/vvbwQcHwrbwjstsLekKgfwirXZVsNr//uNIzztHs0RAdKXI= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 158826476799842.406655596507335; Thu, 30 Apr 2020 09:39:27 -0700 (PDT) Received: from localhost ([::1]:59260 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jUCDe-0008FG-GD for importer@patchew.org; Thu, 30 Apr 2020 12:39:26 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:49316) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jUC30-0001c9-EI for qemu-devel@nongnu.org; Thu, 30 Apr 2020 12:28:29 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.90_1) (envelope-from ) id 1jUC2y-0006Mk-OC for qemu-devel@nongnu.org; Thu, 30 Apr 2020 12:28:26 -0400 Received: from mail-pg1-x541.google.com ([2607:f8b0:4864:20::541]:37367) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jUC2y-0006MY-6v for qemu-devel@nongnu.org; Thu, 30 Apr 2020 12:28:24 -0400 Received: by mail-pg1-x541.google.com with SMTP id r4so2982787pgg.4 for ; Thu, 30 Apr 2020 09:28:22 -0700 (PDT) Received: from localhost.localdomain (174-21-149-226.tukw.qwest.net. [174.21.149.226]) by smtp.gmail.com with ESMTPSA id j5sm243514pfh.58.2020.04.30.09.28.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Apr 2020 09:28:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=+P5jItpXz/tjAGeO3l2qriIEmglug+qlPtimhug8Qps=; b=QqLoaaVUMy2cSVLKmUb4I9iCoxjlYAtLhE5BQaqqAEDb3jbshH0LupHQCZpiRxHBmy kBKhEAi1LqFcsZ3WVzOquH2nHxdiJXDDNrDkjnJrMOW8qwWymzEK5dHzPBgnbWAaYfJK 7JBIpyv2Xg4NktqZUGxryi68R+3gOws6x/zwvoXMl6qL6nShFGLTjjRh2mVx5rj/oVod naB6tPrGCAydaZzttteIYGHaO+jrtr0Do1O2QZETETw2Cq3aRvm2fcqdS1gHgvx8SVYt /Us/dUaGcUetpvvlCDZI59ihNWAwnr4tf/LdAJfdm8WyXS/tXMSZuDSmE0mLig1gUO44 KIew== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=+P5jItpXz/tjAGeO3l2qriIEmglug+qlPtimhug8Qps=; b=Whne35I6bBj1jmYw1/dGyd6yyXl3QsMZMrC7yAfDzwq6xPzfxkm3IueU5T0xuSUnmM EDZi5rCyHlqNbeBIV/3E2IfcHjPq2Iorf4cqkBvT2yRT9xxvoHNGFzbvwXdmx5FmL/+5 vxmz6Shjb0LplEc1MSGQXyr+Va8+93L1MvJuCv1CFY2ZDq0dTBOWi9oqv6amb1VpZavV uJCXupWMNPZzQsx/zf/sAGLvJZxHrC+kK0e0D8jA0wNs8OMuVlpjr5Q9TvXi1lBLWmCX MOw9U6OCOkcKqP6qZlQj/yDGN1klPlhOnaFWrtqIvqTs/zZEDW0m6H2jDbl3R6gJPh9L keKg== X-Gm-Message-State: AGi0PuZiPfKruBKyqaycOjW0eQ/Vv+YswBSdLjBUmkb7mwzNtAZAMIat w3SEO5NBUkvr/QqRbq7Z6O8DiP8kaJI= X-Google-Smtp-Source: APiQypL0xYxEQe29BI5+j18L0eYDU/OIIAPPyOELt/EYy7DYQ9a5CQkUD1+wppwbFNmoTMMaCrkthg== X-Received: by 2002:aa7:9ab3:: with SMTP id x19mr92961pfi.141.1588264101037; Thu, 30 Apr 2020 09:28:21 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH v4 04/18] accel/tcg: Add probe_access_flags Date: Thu, 30 Apr 2020 09:27:59 -0700 Message-Id: <20200430162813.17671-5-richard.henderson@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200430162813.17671-1-richard.henderson@linaro.org> References: <20200430162813.17671-1-richard.henderson@linaro.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2607:f8b0:4864:20::541; envelope-from=richard.henderson@linaro.org; helo=mail-pg1-x541.google.com X-detected-operating-system: by eggs.gnu.org: Error: [-] PROGRAM ABORT : Malformed IPv6 address (bad octet value). Location : parse_addr6(), p0f-client.c:67 X-Received-From: 2607:f8b0:4864:20::541 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, alex.bennee@linaro.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @linaro.org) Content-Type: text/plain; charset="utf-8" This new interface will allow targets to probe for a page and then handle watchpoints themselves. This will be most useful for vector predicated memory operations, where one page lookup can be used for many operations, and one test can avoid many watchpoint checks. Signed-off-by: Richard Henderson --- v2: Fix return of host pointer in softmmu probe_access_flags. v4: Match user-only version closer to softmmu version. --- include/exec/cpu-all.h | 13 ++- include/exec/exec-all.h | 22 +++++ accel/tcg/cputlb.c | 177 ++++++++++++++++++++-------------------- accel/tcg/user-exec.c | 45 +++++++--- 4 files changed, 159 insertions(+), 98 deletions(-) diff --git a/include/exec/cpu-all.h b/include/exec/cpu-all.h index 49384bb66a..43ddcf024c 100644 --- a/include/exec/cpu-all.h +++ b/include/exec/cpu-all.h @@ -328,7 +328,18 @@ CPUArchState *cpu_copy(CPUArchState *env); | CPU_INTERRUPT_TGT_EXT_3 \ | CPU_INTERRUPT_TGT_EXT_4) =20 -#if !defined(CONFIG_USER_ONLY) +#ifdef CONFIG_USER_ONLY + +/* + * Allow some level of source compatibility with softmmu. We do not + * support any of the more exotic features, so only invalid pages may + * be signaled by probe_access_flags(). + */ +#define TLB_INVALID_MASK (1 << (TARGET_PAGE_BITS_MIN - 1)) +#define TLB_MMIO 0 +#define TLB_WATCHPOINT 0 + +#else =20 /* * Flags stored in the low bits of the TLB virtual address. diff --git a/include/exec/exec-all.h b/include/exec/exec-all.h index d656a1f05c..8792bea07a 100644 --- a/include/exec/exec-all.h +++ b/include/exec/exec-all.h @@ -362,6 +362,28 @@ static inline void *probe_read(CPUArchState *env, targ= et_ulong addr, int size, return probe_access(env, addr, size, MMU_DATA_LOAD, mmu_idx, retaddr); } =20 +/** + * probe_access_flags: + * @env: CPUArchState + * @addr: guest virtual address to look up + * @access_type: read, write or execute permission + * @mmu_idx: MMU index to use for lookup + * @nonfault: suppress the fault + * @phost: return value for host address + * @retaddr: return address for unwinding + * + * Similar to probe_access, loosely returning the TLB_FLAGS_MASK for + * the page, and storing the host address for RAM in @phost. + * + * If @nonfault is set, do not raise an exception but return TLB_INVALID_M= ASK. + * Do not handle watchpoints, but include TLB_WATCHPOINT in the returned f= lags. + * Do handle clean pages, so exclude TLB_NOTDIRY from the returned flags. + * For simplicity, all "mmio-like" flags are folded to TLB_MMIO. + */ +int probe_access_flags(CPUArchState *env, target_ulong addr, + MMUAccessType access_type, int mmu_idx, + bool nonfault, void **phost, uintptr_t retaddr); + #define CODE_GEN_ALIGN 16 /* must be >=3D of the size of a icach= e line */ =20 /* Estimated block size for TB allocation. */ diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c index e3b5750c3b..c708e9785f 100644 --- a/accel/tcg/cputlb.c +++ b/accel/tcg/cputlb.c @@ -1231,86 +1231,16 @@ static void notdirty_write(CPUState *cpu, vaddr mem= _vaddr, unsigned size, } } =20 -/* - * Probe for whether the specified guest access is permitted. If it is not - * permitted then an exception will be taken in the same way as if this - * were a real access (and we will not return). - * If the size is 0 or the page requires I/O access, returns NULL; otherwi= se, - * returns the address of the host page similar to tlb_vaddr_to_host(). - */ -void *probe_access(CPUArchState *env, target_ulong addr, int size, - MMUAccessType access_type, int mmu_idx, uintptr_t retad= dr) +static int probe_access_internal(CPUArchState *env, target_ulong addr, + int fault_size, MMUAccessType access_type, + int mmu_idx, bool nonfault, + void **phost, uintptr_t retaddr) { uintptr_t index =3D tlb_index(env, mmu_idx, addr); CPUTLBEntry *entry =3D tlb_entry(env, mmu_idx, addr); - target_ulong tlb_addr; - size_t elt_ofs; - int wp_access; - - g_assert(-(addr | TARGET_PAGE_MASK) >=3D size); - - switch (access_type) { - case MMU_DATA_LOAD: - elt_ofs =3D offsetof(CPUTLBEntry, addr_read); - wp_access =3D BP_MEM_READ; - break; - case MMU_DATA_STORE: - elt_ofs =3D offsetof(CPUTLBEntry, addr_write); - wp_access =3D BP_MEM_WRITE; - break; - case MMU_INST_FETCH: - elt_ofs =3D offsetof(CPUTLBEntry, addr_code); - wp_access =3D BP_MEM_READ; - break; - default: - g_assert_not_reached(); - } - tlb_addr =3D tlb_read_ofs(entry, elt_ofs); - - if (unlikely(!tlb_hit(tlb_addr, addr))) { - if (!victim_tlb_hit(env, mmu_idx, index, elt_ofs, - addr & TARGET_PAGE_MASK)) { - tlb_fill(env_cpu(env), addr, size, access_type, mmu_idx, retad= dr); - /* TLB resize via tlb_fill may have moved the entry. */ - index =3D tlb_index(env, mmu_idx, addr); - entry =3D tlb_entry(env, mmu_idx, addr); - } - tlb_addr =3D tlb_read_ofs(entry, elt_ofs); - } - - if (!size) { - return NULL; - } - - if (unlikely(tlb_addr & TLB_FLAGS_MASK)) { - CPUIOTLBEntry *iotlbentry =3D &env_tlb(env)->d[mmu_idx].iotlb[inde= x]; - - /* Reject I/O access, or other required slow-path. */ - if (tlb_addr & (TLB_MMIO | TLB_BSWAP | TLB_DISCARD_WRITE)) { - return NULL; - } - - /* Handle watchpoints. */ - if (tlb_addr & TLB_WATCHPOINT) { - cpu_check_watchpoint(env_cpu(env), addr, size, - iotlbentry->attrs, wp_access, retaddr); - } - - /* Handle clean RAM pages. */ - if (tlb_addr & TLB_NOTDIRTY) { - notdirty_write(env_cpu(env), addr, size, iotlbentry, retaddr); - } - } - - return (void *)((uintptr_t)addr + entry->addend); -} - -void *tlb_vaddr_to_host(CPUArchState *env, abi_ptr addr, - MMUAccessType access_type, int mmu_idx) -{ - CPUTLBEntry *entry =3D tlb_entry(env, mmu_idx, addr); - target_ulong tlb_addr, page; + target_ulong tlb_addr, page_addr; size_t elt_ofs; + int flags; =20 switch (access_type) { case MMU_DATA_LOAD: @@ -1325,20 +1255,19 @@ void *tlb_vaddr_to_host(CPUArchState *env, abi_ptr = addr, default: g_assert_not_reached(); } - - page =3D addr & TARGET_PAGE_MASK; tlb_addr =3D tlb_read_ofs(entry, elt_ofs); =20 - if (!tlb_hit_page(tlb_addr, page)) { - uintptr_t index =3D tlb_index(env, mmu_idx, addr); - - if (!victim_tlb_hit(env, mmu_idx, index, elt_ofs, page)) { + page_addr =3D addr & TARGET_PAGE_MASK; + if (!tlb_hit_page(tlb_addr, page_addr)) { + if (!victim_tlb_hit(env, mmu_idx, index, elt_ofs, page_addr)) { CPUState *cs =3D env_cpu(env); CPUClass *cc =3D CPU_GET_CLASS(cs); =20 - if (!cc->tlb_fill(cs, addr, 0, access_type, mmu_idx, true, 0))= { + if (!cc->tlb_fill(cs, addr, fault_size, access_type, + mmu_idx, nonfault, retaddr)) { /* Non-faulting page table read failed. */ - return NULL; + *phost =3D NULL; + return TLB_INVALID_MASK; } =20 /* TLB resize via tlb_fill may have moved the entry. */ @@ -1346,15 +1275,89 @@ void *tlb_vaddr_to_host(CPUArchState *env, abi_ptr = addr, } tlb_addr =3D tlb_read_ofs(entry, elt_ofs); } + flags =3D tlb_addr & TLB_FLAGS_MASK; =20 - if (tlb_addr & ~TARGET_PAGE_MASK) { - /* IO access */ + /* Fold all "mmio-like" bits into TLB_MMIO. This is not RAM. */ + if (unlikely(flags & ~(TLB_WATCHPOINT | TLB_NOTDIRTY))) { + *phost =3D NULL; + return TLB_MMIO; + } + + /* Everything else is RAM. */ + *phost =3D (void *)((uintptr_t)addr + entry->addend); + return flags; +} + +int probe_access_flags(CPUArchState *env, target_ulong addr, + MMUAccessType access_type, int mmu_idx, + bool nonfault, void **phost, uintptr_t retaddr) +{ + int flags; + + flags =3D probe_access_internal(env, addr, 0, access_type, mmu_idx, + nonfault, phost, retaddr); + + /* Handle clean RAM pages. */ + if (unlikely(flags & TLB_NOTDIRTY)) { + uintptr_t index =3D tlb_index(env, mmu_idx, addr); + CPUIOTLBEntry *iotlbentry =3D &env_tlb(env)->d[mmu_idx].iotlb[inde= x]; + + notdirty_write(env_cpu(env), addr, 1, iotlbentry, retaddr); + flags &=3D ~TLB_NOTDIRTY; + } + + return flags; +} + +void *probe_access(CPUArchState *env, target_ulong addr, int size, + MMUAccessType access_type, int mmu_idx, uintptr_t retad= dr) +{ + void *host; + int flags; + + g_assert(-(addr | TARGET_PAGE_MASK) >=3D size); + + flags =3D probe_access_internal(env, addr, size, access_type, mmu_idx, + false, &host, retaddr); + + /* Per the interface, size =3D=3D 0 merely faults the access. */ + if (size =3D=3D 0) { return NULL; } =20 - return (void *)((uintptr_t)addr + entry->addend); + if (unlikely(flags & (TLB_NOTDIRTY | TLB_WATCHPOINT))) { + uintptr_t index =3D tlb_index(env, mmu_idx, addr); + CPUIOTLBEntry *iotlbentry =3D &env_tlb(env)->d[mmu_idx].iotlb[inde= x]; + + /* Handle watchpoints. */ + if (flags & TLB_WATCHPOINT) { + int wp_access =3D (access_type =3D=3D MMU_DATA_STORE + ? BP_MEM_WRITE : BP_MEM_READ); + cpu_check_watchpoint(env_cpu(env), addr, size, + iotlbentry->attrs, wp_access, retaddr); + } + + /* Handle clean RAM pages. */ + if (flags & TLB_NOTDIRTY) { + notdirty_write(env_cpu(env), addr, 1, iotlbentry, retaddr); + } + } + + return host; } =20 +void *tlb_vaddr_to_host(CPUArchState *env, abi_ptr addr, + MMUAccessType access_type, int mmu_idx) +{ + void *host; + int flags; + + flags =3D probe_access_internal(env, addr, 0, access_type, + mmu_idx, true, &host, 0); + + /* No combination of flags are expected by the caller. */ + return flags ? NULL : host; +} =20 #ifdef CONFIG_PLUGIN /* diff --git a/accel/tcg/user-exec.c b/accel/tcg/user-exec.c index 4be78eb9b3..987342c50c 100644 --- a/accel/tcg/user-exec.c +++ b/accel/tcg/user-exec.c @@ -190,13 +190,12 @@ static inline int handle_cpu_signal(uintptr_t pc, sig= info_t *info, g_assert_not_reached(); } =20 -void *probe_access(CPUArchState *env, target_ulong addr, int size, - MMUAccessType access_type, int mmu_idx, uintptr_t retad= dr) +static int probe_access_internal(CPUArchState *env, target_ulong addr, + int fault_size, MMUAccessType access_type, + bool nonfault, uintptr_t ra) { int flags; =20 - g_assert(-(addr | TARGET_PAGE_MASK) >=3D size); - switch (access_type) { case MMU_DATA_STORE: flags =3D PAGE_WRITE; @@ -211,13 +210,39 @@ void *probe_access(CPUArchState *env, target_ulong ad= dr, int size, g_assert_not_reached(); } =20 - if (!guest_addr_valid(addr) || page_check_range(addr, size, flags) < 0= ) { - CPUState *cpu =3D env_cpu(env); - CPUClass *cc =3D CPU_GET_CLASS(cpu); - cc->tlb_fill(cpu, addr, size, access_type, MMU_USER_IDX, false, - retaddr); - g_assert_not_reached(); + if (!guest_addr_valid(addr) || page_check_range(addr, 1, flags) < 0) { + if (nonfault) { + return TLB_INVALID_MASK; + } else { + CPUState *cpu =3D env_cpu(env); + CPUClass *cc =3D CPU_GET_CLASS(cpu); + cc->tlb_fill(cpu, addr, fault_size, access_type, + MMU_USER_IDX, false, ra); + g_assert_not_reached(); + } } + return 0; +} + +int probe_access_flags(CPUArchState *env, target_ulong addr, + MMUAccessType access_type, int mmu_idx, + bool nonfault, void **phost, uintptr_t ra) +{ + int flags; + + flags =3D probe_access_internal(env, addr, 0, access_type, nonfault, r= a); + *phost =3D flags ? NULL : g2h(addr); + return flags; +} + +void *probe_access(CPUArchState *env, target_ulong addr, int size, + MMUAccessType access_type, int mmu_idx, uintptr_t ra) +{ + int flags; + + g_assert(-(addr | TARGET_PAGE_MASK) >=3D size); + flags =3D probe_access_internal(env, addr, size, access_type, false, r= a); + g_assert(flags =3D=3D 0); =20 return size ? g2h(addr) : NULL; } --=20 2.20.1 From nobody Sun May 19 14:32:52 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=linaro.org ARC-Seal: i=1; a=rsa-sha256; t=1588264721; cv=none; d=zohomail.com; s=zohoarc; b=KYweFUFgzQxUhSBe5I+Oi/OQy0Ugt6bQkQIUtwTqUKrL3Gf4HIKKN+s16tJlvL6oqbcI8PDnp1HBryQ+Qt6unrSIProgWDDtBYVWnMIbJ/Cnm/cgpRimK7KdabM05wTzUVP9+4/rcpQ7WZd2V64TRWJ8ewksIjC30ElFA6Ta7R8= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1588264721; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=J5zhPuFSG1IvgY/HZM+zfSrn5bmQLwsRBPEuzZHx2Fo=; b=CE/XdgwZjGJzWPkmu4PHr5k4vMyn3VjsKk3tUONtDrI+zrZQmPVeswXlq6ZCFi/Vd7s0Tfinvv0xAesAFkvoZKOtOIxT4aVrjemfI29u3N7yudlNd4yViIPfE68Yz6ggy10M6CkadhUi6F/fyPY+hGFfCcotJ8kkHaYpcfi4dGo= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1588264721359867.7477058696146; Thu, 30 Apr 2020 09:38:41 -0700 (PDT) Received: from localhost ([::1]:57172 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jUCCt-0007Mg-Pe for importer@patchew.org; Thu, 30 Apr 2020 12:38:39 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:49332) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jUC33-0001io-6u for qemu-devel@nongnu.org; Thu, 30 Apr 2020 12:28:32 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.90_1) (envelope-from ) id 1jUC2z-0006N4-TV for qemu-devel@nongnu.org; Thu, 30 Apr 2020 12:28:28 -0400 Received: from mail-pg1-x543.google.com ([2607:f8b0:4864:20::543]:45206) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jUC2z-0006Mi-E0 for qemu-devel@nongnu.org; Thu, 30 Apr 2020 12:28:25 -0400 Received: by mail-pg1-x543.google.com with SMTP id s18so2963330pgl.12 for ; Thu, 30 Apr 2020 09:28:25 -0700 (PDT) Received: from localhost.localdomain (174-21-149-226.tukw.qwest.net. [174.21.149.226]) by smtp.gmail.com with ESMTPSA id j5sm243514pfh.58.2020.04.30.09.28.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Apr 2020 09:28:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=J5zhPuFSG1IvgY/HZM+zfSrn5bmQLwsRBPEuzZHx2Fo=; b=IgQKV4mecw4tGcAUyMJID2PhFJS2DG98kdxJATnLHdfdlaci1MEX+1j3WEOcInJb1Y 8cI4kjvOeRAFETxjDdhg5ShhaQcqCs+9Xb9rU2uIngGeEfa85T6gPHA1ghehIA7Ga9Yi xI8PS6pkSNWSRqmlL/guSdxU+myJEnfIuf7n9pGaVvRP+QA389YFD3z3j4rQ4KZa+WLc 9tQ70ab5A74TS7a18mhRNU0zIgowesyXNQuOZZEJdGjtWl/I8TiIhm3/cwHO/C17tZiu i2gZUx1jfhcwpcFNDsQhJu1rw72cM4sikAcok3GLnhAlg0IYvPpuX0TiWqhtZ7XgW6KI FY1Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=J5zhPuFSG1IvgY/HZM+zfSrn5bmQLwsRBPEuzZHx2Fo=; b=WJafK6D3/fmXo4tamlEAJ+OnbDWQqIE5doK+6dkZCLbpj/ecpWNLkHfTtJ6SYv1Nnw YwaKp8AL38+VV0qeA+fbvYi0IW7WVsuQeO/fPNFRJfpGJE4MgoFKQpu117v2YJXlaOxF 8ePrtnToVK5T4pMaBRDfPMYRCgIjnWuo9H/k1FGujA+MR++ggBUZdx3tTHRFm7jXh9NO UgueH1cKkF4ITmUQD+ewd0t1Fqe0TqY3h4pzIgbZvS0gI7/zYz3Pu/W03LB6W2+e7hOo olJgrTfPCguYPOKmoby4S2vSHEdEZo4VGpI1wk7rj+wfrs8FxVMTcXCZd16uk8M5MKxM Oa5w== X-Gm-Message-State: AGi0PuZXXu9Oy11jMXVEWZni8w0j3rFxcPvmhXRCicTNSit0YTVl82AA Sv4IvbFT9oeJ7LN7WMq6Fc/1rDTdDJA= X-Google-Smtp-Source: APiQypKSOuUm8BusoxpVjnX4k1q6dsTGpTUQpA9fb+41xneWo9JZReIZWx7ZQR6w/cvPyMC7J4Rkfg== X-Received: by 2002:a63:575f:: with SMTP id h31mr3513128pgm.200.1588264102439; Thu, 30 Apr 2020 09:28:22 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH v4 05/18] accel/tcg: Add endian-specific cpu_{ld, st}* operations Date: Thu, 30 Apr 2020 09:28:00 -0700 Message-Id: <20200430162813.17671-6-richard.henderson@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200430162813.17671-1-richard.henderson@linaro.org> References: <20200430162813.17671-1-richard.henderson@linaro.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2607:f8b0:4864:20::543; envelope-from=richard.henderson@linaro.org; helo=mail-pg1-x543.google.com X-detected-operating-system: by eggs.gnu.org: Error: [-] PROGRAM ABORT : Malformed IPv6 address (bad octet value). Location : parse_addr6(), p0f-client.c:67 X-Received-From: 2607:f8b0:4864:20::543 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, alex.bennee@linaro.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Type: text/plain; charset="utf-8" We currently have target-endian versions of these operations, but no easy way to force a specific endianness. This can be helpful if the target has endian-specific operations, or a mode that swaps endianness. Reviewed-by: Peter Maydell Signed-off-by: Richard Henderson --- docs/devel/loads-stores.rst | 39 +++-- include/exec/cpu_ldst.h | 283 +++++++++++++++++++++++++++--------- accel/tcg/cputlb.c | 236 ++++++++++++++++++++++-------- accel/tcg/user-exec.c | 211 ++++++++++++++++++++++----- 4 files changed, 587 insertions(+), 182 deletions(-) diff --git a/docs/devel/loads-stores.rst b/docs/devel/loads-stores.rst index 0d99eb24c1..9a944ef1af 100644 --- a/docs/devel/loads-stores.rst +++ b/docs/devel/loads-stores.rst @@ -97,9 +97,9 @@ function, which is a return address into the generated co= de. =20 Function names follow the pattern: =20 -load: ``cpu_ld{sign}{size}_mmuidx_ra(env, ptr, mmuidx, retaddr)`` +load: ``cpu_ld{sign}{size}{end}_mmuidx_ra(env, ptr, mmuidx, retaddr)`` =20 -store: ``cpu_st{size}_mmuidx_ra(env, ptr, val, mmuidx, retaddr)`` +store: ``cpu_st{size}{end}_mmuidx_ra(env, ptr, val, mmuidx, retaddr)`` =20 ``sign`` - (empty) : for 32 or 64 bit sizes @@ -112,9 +112,14 @@ store: ``cpu_st{size}_mmuidx_ra(env, ptr, val, mmuidx,= retaddr)`` - ``l`` : 32 bits - ``q`` : 64 bits =20 +``end`` + - (empty) : for target endian, or 8 bit sizes + - ``_be`` : big endian + - ``_le`` : little endian + Regexes for git grep: - - ``\`` - - ``\`` + - ``\`` + - ``\`` =20 ``cpu_{ld,st}*_data_ra`` ~~~~~~~~~~~~~~~~~~~~~~~~ @@ -129,9 +134,9 @@ be performed with a context other than the default. =20 Function names follow the pattern: =20 -load: ``cpu_ld{sign}{size}_data_ra(env, ptr, ra)`` +load: ``cpu_ld{sign}{size}{end}_data_ra(env, ptr, ra)`` =20 -store: ``cpu_st{size}_data_ra(env, ptr, val, ra)`` +store: ``cpu_st{size}{end}_data_ra(env, ptr, val, ra)`` =20 ``sign`` - (empty) : for 32 or 64 bit sizes @@ -144,9 +149,14 @@ store: ``cpu_st{size}_data_ra(env, ptr, val, ra)`` - ``l`` : 32 bits - ``q`` : 64 bits =20 +``end`` + - (empty) : for target endian, or 8 bit sizes + - ``_be`` : big endian + - ``_le`` : little endian + Regexes for git grep: - - ``\`` - - ``\`` + - ``\`` + - ``\`` =20 ``cpu_{ld,st}*_data`` ~~~~~~~~~~~~~~~~~~~~~ @@ -163,9 +173,9 @@ the CPU state anyway. =20 Function names follow the pattern: =20 -load: ``cpu_ld{sign}{size}_data(env, ptr)`` +load: ``cpu_ld{sign}{size}{end}_data(env, ptr)`` =20 -store: ``cpu_st{size}_data(env, ptr, val)`` +store: ``cpu_st{size}{end}_data(env, ptr, val)`` =20 ``sign`` - (empty) : for 32 or 64 bit sizes @@ -178,9 +188,14 @@ store: ``cpu_st{size}_data(env, ptr, val)`` - ``l`` : 32 bits - ``q`` : 64 bits =20 +``end`` + - (empty) : for target endian, or 8 bit sizes + - ``_be`` : big endian + - ``_le`` : little endian + Regexes for git grep - - ``\`` - - ``\`` + - ``\`` + - ``\`` =20 ``cpu_ld*_code`` ~~~~~~~~~~~~~~~~ diff --git a/include/exec/cpu_ldst.h b/include/exec/cpu_ldst.h index 53de19753a..c14a48f65e 100644 --- a/include/exec/cpu_ldst.h +++ b/include/exec/cpu_ldst.h @@ -25,13 +25,13 @@ * * The syntax for the accessors is: * - * load: cpu_ld{sign}{size}_{mmusuffix}(env, ptr) - * cpu_ld{sign}{size}_{mmusuffix}_ra(env, ptr, retaddr) - * cpu_ld{sign}{size}_mmuidx_ra(env, ptr, mmu_idx, retaddr) + * load: cpu_ld{sign}{size}{end}_{mmusuffix}(env, ptr) + * cpu_ld{sign}{size}{end}_{mmusuffix}_ra(env, ptr, retaddr) + * cpu_ld{sign}{size}{end}_mmuidx_ra(env, ptr, mmu_idx, retaddr) * - * store: cpu_st{size}_{mmusuffix}(env, ptr, val) - * cpu_st{size}_{mmusuffix}_ra(env, ptr, val, retaddr) - * cpu_st{size}_mmuidx_ra(env, ptr, val, mmu_idx, retaddr) + * store: cpu_st{size}{end}_{mmusuffix}(env, ptr, val) + * cpu_st{size}{end}_{mmusuffix}_ra(env, ptr, val, retaddr) + * cpu_st{size}{end}_mmuidx_ra(env, ptr, val, mmu_idx, retaddr) * * sign is: * (empty): for 32 and 64 bit sizes @@ -44,6 +44,11 @@ * l: 32 bits * q: 64 bits * + * end is: + * (empty): for target native endian, or for 8 bit access + * _be: for forced big endian + * _le: for forced little endian + * * mmusuffix is one of the generic suffixes "data" or "code", or "mmuidx". * The "mmuidx" suffix carries an extra mmu_idx argument that specifies * the index to use; the "data" and "code" suffixes take the index from @@ -95,32 +100,57 @@ typedef target_ulong abi_ptr; #endif =20 uint32_t cpu_ldub_data(CPUArchState *env, abi_ptr ptr); -uint32_t cpu_lduw_data(CPUArchState *env, abi_ptr ptr); -uint32_t cpu_ldl_data(CPUArchState *env, abi_ptr ptr); -uint64_t cpu_ldq_data(CPUArchState *env, abi_ptr ptr); int cpu_ldsb_data(CPUArchState *env, abi_ptr ptr); -int cpu_ldsw_data(CPUArchState *env, abi_ptr ptr); =20 -uint32_t cpu_ldub_data_ra(CPUArchState *env, abi_ptr ptr, uintptr_t retadd= r); -uint32_t cpu_lduw_data_ra(CPUArchState *env, abi_ptr ptr, uintptr_t retadd= r); -uint32_t cpu_ldl_data_ra(CPUArchState *env, abi_ptr ptr, uintptr_t retaddr= ); -uint64_t cpu_ldq_data_ra(CPUArchState *env, abi_ptr ptr, uintptr_t retaddr= ); -int cpu_ldsb_data_ra(CPUArchState *env, abi_ptr ptr, uintptr_t retaddr); -int cpu_ldsw_data_ra(CPUArchState *env, abi_ptr ptr, uintptr_t retaddr); +uint32_t cpu_lduw_be_data(CPUArchState *env, abi_ptr ptr); +int cpu_ldsw_be_data(CPUArchState *env, abi_ptr ptr); +uint32_t cpu_ldl_be_data(CPUArchState *env, abi_ptr ptr); +uint64_t cpu_ldq_be_data(CPUArchState *env, abi_ptr ptr); + +uint32_t cpu_lduw_le_data(CPUArchState *env, abi_ptr ptr); +int cpu_ldsw_le_data(CPUArchState *env, abi_ptr ptr); +uint32_t cpu_ldl_le_data(CPUArchState *env, abi_ptr ptr); +uint64_t cpu_ldq_le_data(CPUArchState *env, abi_ptr ptr); + +uint32_t cpu_ldub_data_ra(CPUArchState *env, abi_ptr ptr, uintptr_t ra); +int cpu_ldsb_data_ra(CPUArchState *env, abi_ptr ptr, uintptr_t ra); + +uint32_t cpu_lduw_be_data_ra(CPUArchState *env, abi_ptr ptr, uintptr_t ra); +int cpu_ldsw_be_data_ra(CPUArchState *env, abi_ptr ptr, uintptr_t ra); +uint32_t cpu_ldl_be_data_ra(CPUArchState *env, abi_ptr ptr, uintptr_t ra); +uint64_t cpu_ldq_be_data_ra(CPUArchState *env, abi_ptr ptr, uintptr_t ra); + +uint32_t cpu_lduw_le_data_ra(CPUArchState *env, abi_ptr ptr, uintptr_t ra); +int cpu_ldsw_le_data_ra(CPUArchState *env, abi_ptr ptr, uintptr_t ra); +uint32_t cpu_ldl_le_data_ra(CPUArchState *env, abi_ptr ptr, uintptr_t ra); +uint64_t cpu_ldq_le_data_ra(CPUArchState *env, abi_ptr ptr, uintptr_t ra); =20 void cpu_stb_data(CPUArchState *env, abi_ptr ptr, uint32_t val); -void cpu_stw_data(CPUArchState *env, abi_ptr ptr, uint32_t val); -void cpu_stl_data(CPUArchState *env, abi_ptr ptr, uint32_t val); -void cpu_stq_data(CPUArchState *env, abi_ptr ptr, uint64_t val); + +void cpu_stw_be_data(CPUArchState *env, abi_ptr ptr, uint32_t val); +void cpu_stl_be_data(CPUArchState *env, abi_ptr ptr, uint32_t val); +void cpu_stq_be_data(CPUArchState *env, abi_ptr ptr, uint64_t val); + +void cpu_stw_le_data(CPUArchState *env, abi_ptr ptr, uint32_t val); +void cpu_stl_le_data(CPUArchState *env, abi_ptr ptr, uint32_t val); +void cpu_stq_le_data(CPUArchState *env, abi_ptr ptr, uint64_t val); =20 void cpu_stb_data_ra(CPUArchState *env, abi_ptr ptr, - uint32_t val, uintptr_t retaddr); -void cpu_stw_data_ra(CPUArchState *env, abi_ptr ptr, - uint32_t val, uintptr_t retaddr); -void cpu_stl_data_ra(CPUArchState *env, abi_ptr ptr, - uint32_t val, uintptr_t retaddr); -void cpu_stq_data_ra(CPUArchState *env, abi_ptr ptr, - uint64_t val, uintptr_t retaddr); + uint32_t val, uintptr_t ra); + +void cpu_stw_be_data_ra(CPUArchState *env, abi_ptr ptr, + uint32_t val, uintptr_t ra); +void cpu_stl_be_data_ra(CPUArchState *env, abi_ptr ptr, + uint32_t val, uintptr_t ra); +void cpu_stq_be_data_ra(CPUArchState *env, abi_ptr ptr, + uint64_t val, uintptr_t ra); + +void cpu_stw_le_data_ra(CPUArchState *env, abi_ptr ptr, + uint32_t val, uintptr_t ra); +void cpu_stl_le_data_ra(CPUArchState *env, abi_ptr ptr, + uint32_t val, uintptr_t ra); +void cpu_stq_le_data_ra(CPUArchState *env, abi_ptr ptr, + uint64_t val, uintptr_t ra); =20 #if defined(CONFIG_USER_ONLY) =20 @@ -157,34 +187,58 @@ static inline uint32_t cpu_ldub_mmuidx_ra(CPUArchStat= e *env, abi_ptr addr, return cpu_ldub_data_ra(env, addr, ra); } =20 -static inline uint32_t cpu_lduw_mmuidx_ra(CPUArchState *env, abi_ptr addr, - int mmu_idx, uintptr_t ra) -{ - return cpu_lduw_data_ra(env, addr, ra); -} - -static inline uint32_t cpu_ldl_mmuidx_ra(CPUArchState *env, abi_ptr addr, - int mmu_idx, uintptr_t ra) -{ - return cpu_ldl_data_ra(env, addr, ra); -} - -static inline uint64_t cpu_ldq_mmuidx_ra(CPUArchState *env, abi_ptr addr, - int mmu_idx, uintptr_t ra) -{ - return cpu_ldq_data_ra(env, addr, ra); -} - static inline int cpu_ldsb_mmuidx_ra(CPUArchState *env, abi_ptr addr, int mmu_idx, uintptr_t ra) { return cpu_ldsb_data_ra(env, addr, ra); } =20 -static inline int cpu_ldsw_mmuidx_ra(CPUArchState *env, abi_ptr addr, - int mmu_idx, uintptr_t ra) +static inline uint32_t cpu_lduw_be_mmuidx_ra(CPUArchState *env, abi_ptr ad= dr, + int mmu_idx, uintptr_t ra) { - return cpu_ldsw_data_ra(env, addr, ra); + return cpu_lduw_be_data_ra(env, addr, ra); +} + +static inline int cpu_ldsw_be_mmuidx_ra(CPUArchState *env, abi_ptr addr, + int mmu_idx, uintptr_t ra) +{ + return cpu_ldsw_be_data_ra(env, addr, ra); +} + +static inline uint32_t cpu_ldl_be_mmuidx_ra(CPUArchState *env, abi_ptr add= r, + int mmu_idx, uintptr_t ra) +{ + return cpu_ldl_be_data_ra(env, addr, ra); +} + +static inline uint64_t cpu_ldq_be_mmuidx_ra(CPUArchState *env, abi_ptr add= r, + int mmu_idx, uintptr_t ra) +{ + return cpu_ldq_be_data_ra(env, addr, ra); +} + +static inline uint32_t cpu_lduw_le_mmuidx_ra(CPUArchState *env, abi_ptr ad= dr, + int mmu_idx, uintptr_t ra) +{ + return cpu_lduw_le_data_ra(env, addr, ra); +} + +static inline int cpu_ldsw_le_mmuidx_ra(CPUArchState *env, abi_ptr addr, + int mmu_idx, uintptr_t ra) +{ + return cpu_ldsw_le_data_ra(env, addr, ra); +} + +static inline uint32_t cpu_ldl_le_mmuidx_ra(CPUArchState *env, abi_ptr add= r, + int mmu_idx, uintptr_t ra) +{ + return cpu_ldl_le_data_ra(env, addr, ra); +} + +static inline uint64_t cpu_ldq_le_mmuidx_ra(CPUArchState *env, abi_ptr add= r, + int mmu_idx, uintptr_t ra) +{ + return cpu_ldq_le_data_ra(env, addr, ra); } =20 static inline void cpu_stb_mmuidx_ra(CPUArchState *env, abi_ptr addr, @@ -193,22 +247,46 @@ static inline void cpu_stb_mmuidx_ra(CPUArchState *en= v, abi_ptr addr, cpu_stb_data_ra(env, addr, val, ra); } =20 -static inline void cpu_stw_mmuidx_ra(CPUArchState *env, abi_ptr addr, - uint32_t val, int mmu_idx, uintptr_t = ra) +static inline void cpu_stw_be_mmuidx_ra(CPUArchState *env, abi_ptr addr, + uint32_t val, int mmu_idx, + uintptr_t ra) { - cpu_stw_data_ra(env, addr, val, ra); + cpu_stw_be_data_ra(env, addr, val, ra); } =20 -static inline void cpu_stl_mmuidx_ra(CPUArchState *env, abi_ptr addr, - uint32_t val, int mmu_idx, uintptr_t = ra) +static inline void cpu_stl_be_mmuidx_ra(CPUArchState *env, abi_ptr addr, + uint32_t val, int mmu_idx, + uintptr_t ra) { - cpu_stl_data_ra(env, addr, val, ra); + cpu_stl_be_data_ra(env, addr, val, ra); } =20 -static inline void cpu_stq_mmuidx_ra(CPUArchState *env, abi_ptr addr, - uint64_t val, int mmu_idx, uintptr_t = ra) +static inline void cpu_stq_be_mmuidx_ra(CPUArchState *env, abi_ptr addr, + uint64_t val, int mmu_idx, + uintptr_t ra) { - cpu_stq_data_ra(env, addr, val, ra); + cpu_stq_be_data_ra(env, addr, val, ra); +} + +static inline void cpu_stw_le_mmuidx_ra(CPUArchState *env, abi_ptr addr, + uint32_t val, int mmu_idx, + uintptr_t ra) +{ + cpu_stw_le_data_ra(env, addr, val, ra); +} + +static inline void cpu_stl_le_mmuidx_ra(CPUArchState *env, abi_ptr addr, + uint32_t val, int mmu_idx, + uintptr_t ra) +{ + cpu_stl_le_data_ra(env, addr, val, ra); +} + +static inline void cpu_stq_le_mmuidx_ra(CPUArchState *env, abi_ptr addr, + uint64_t val, int mmu_idx, + uintptr_t ra) +{ + cpu_stq_le_data_ra(env, addr, val, ra); } =20 #else @@ -243,29 +321,92 @@ static inline CPUTLBEntry *tlb_entry(CPUArchState *en= v, uintptr_t mmu_idx, =20 uint32_t cpu_ldub_mmuidx_ra(CPUArchState *env, abi_ptr addr, int mmu_idx, uintptr_t ra); -uint32_t cpu_lduw_mmuidx_ra(CPUArchState *env, abi_ptr addr, - int mmu_idx, uintptr_t ra); -uint32_t cpu_ldl_mmuidx_ra(CPUArchState *env, abi_ptr addr, - int mmu_idx, uintptr_t ra); -uint64_t cpu_ldq_mmuidx_ra(CPUArchState *env, abi_ptr addr, - int mmu_idx, uintptr_t ra); - int cpu_ldsb_mmuidx_ra(CPUArchState *env, abi_ptr addr, int mmu_idx, uintptr_t ra); -int cpu_ldsw_mmuidx_ra(CPUArchState *env, abi_ptr addr, - int mmu_idx, uintptr_t ra); + +uint32_t cpu_lduw_be_mmuidx_ra(CPUArchState *env, abi_ptr addr, + int mmu_idx, uintptr_t ra); +int cpu_ldsw_be_mmuidx_ra(CPUArchState *env, abi_ptr addr, + int mmu_idx, uintptr_t ra); +uint32_t cpu_ldl_be_mmuidx_ra(CPUArchState *env, abi_ptr addr, + int mmu_idx, uintptr_t ra); +uint64_t cpu_ldq_be_mmuidx_ra(CPUArchState *env, abi_ptr addr, + int mmu_idx, uintptr_t ra); + +uint32_t cpu_lduw_le_mmuidx_ra(CPUArchState *env, abi_ptr addr, + int mmu_idx, uintptr_t ra); +int cpu_ldsw_le_mmuidx_ra(CPUArchState *env, abi_ptr addr, + int mmu_idx, uintptr_t ra); +uint32_t cpu_ldl_le_mmuidx_ra(CPUArchState *env, abi_ptr addr, + int mmu_idx, uintptr_t ra); +uint64_t cpu_ldq_le_mmuidx_ra(CPUArchState *env, abi_ptr addr, + int mmu_idx, uintptr_t ra); =20 void cpu_stb_mmuidx_ra(CPUArchState *env, abi_ptr addr, uint32_t val, int mmu_idx, uintptr_t retaddr); -void cpu_stw_mmuidx_ra(CPUArchState *env, abi_ptr addr, uint32_t val, - int mmu_idx, uintptr_t retaddr); -void cpu_stl_mmuidx_ra(CPUArchState *env, abi_ptr addr, uint32_t val, - int mmu_idx, uintptr_t retaddr); -void cpu_stq_mmuidx_ra(CPUArchState *env, abi_ptr addr, uint64_t val, - int mmu_idx, uintptr_t retaddr); + +void cpu_stw_be_mmuidx_ra(CPUArchState *env, abi_ptr addr, uint32_t val, + int mmu_idx, uintptr_t retaddr); +void cpu_stl_be_mmuidx_ra(CPUArchState *env, abi_ptr addr, uint32_t val, + int mmu_idx, uintptr_t retaddr); +void cpu_stq_be_mmuidx_ra(CPUArchState *env, abi_ptr addr, uint64_t val, + int mmu_idx, uintptr_t retaddr); + +void cpu_stw_le_mmuidx_ra(CPUArchState *env, abi_ptr addr, uint32_t val, + int mmu_idx, uintptr_t retaddr); +void cpu_stl_le_mmuidx_ra(CPUArchState *env, abi_ptr addr, uint32_t val, + int mmu_idx, uintptr_t retaddr); +void cpu_stq_le_mmuidx_ra(CPUArchState *env, abi_ptr addr, uint64_t val, + int mmu_idx, uintptr_t retaddr); =20 #endif /* defined(CONFIG_USER_ONLY) */ =20 +#ifdef TARGET_WORDS_BIGENDIAN +# define cpu_lduw_data cpu_lduw_be_data +# define cpu_ldsw_data cpu_ldsw_be_data +# define cpu_ldl_data cpu_ldl_be_data +# define cpu_ldq_data cpu_ldq_be_data +# define cpu_lduw_data_ra cpu_lduw_be_data_ra +# define cpu_ldsw_data_ra cpu_ldsw_be_data_ra +# define cpu_ldl_data_ra cpu_ldl_be_data_ra +# define cpu_ldq_data_ra cpu_ldq_be_data_ra +# define cpu_lduw_mmuidx_ra cpu_lduw_be_mmuidx_ra +# define cpu_ldsw_mmuidx_ra cpu_ldsw_be_mmuidx_ra +# define cpu_ldl_mmuidx_ra cpu_ldl_be_mmuidx_ra +# define cpu_ldq_mmuidx_ra cpu_ldq_be_mmuidx_ra +# define cpu_stw_data cpu_stw_be_data +# define cpu_stl_data cpu_stl_be_data +# define cpu_stq_data cpu_stq_be_data +# define cpu_stw_data_ra cpu_stw_be_data_ra +# define cpu_stl_data_ra cpu_stl_be_data_ra +# define cpu_stq_data_ra cpu_stq_be_data_ra +# define cpu_stw_mmuidx_ra cpu_stw_be_mmuidx_ra +# define cpu_stl_mmuidx_ra cpu_stl_be_mmuidx_ra +# define cpu_stq_mmuidx_ra cpu_stq_be_mmuidx_ra +#else +# define cpu_lduw_data cpu_lduw_le_data +# define cpu_ldsw_data cpu_ldsw_le_data +# define cpu_ldl_data cpu_ldl_le_data +# define cpu_ldq_data cpu_ldq_le_data +# define cpu_lduw_data_ra cpu_lduw_le_data_ra +# define cpu_ldsw_data_ra cpu_ldsw_le_data_ra +# define cpu_ldl_data_ra cpu_ldl_le_data_ra +# define cpu_ldq_data_ra cpu_ldq_le_data_ra +# define cpu_lduw_mmuidx_ra cpu_lduw_le_mmuidx_ra +# define cpu_ldsw_mmuidx_ra cpu_ldsw_le_mmuidx_ra +# define cpu_ldl_mmuidx_ra cpu_ldl_le_mmuidx_ra +# define cpu_ldq_mmuidx_ra cpu_ldq_le_mmuidx_ra +# define cpu_stw_data cpu_stw_le_data +# define cpu_stl_data cpu_stl_le_data +# define cpu_stq_data cpu_stq_le_data +# define cpu_stw_data_ra cpu_stw_le_data_ra +# define cpu_stl_data_ra cpu_stl_le_data_ra +# define cpu_stq_data_ra cpu_stq_le_data_ra +# define cpu_stw_mmuidx_ra cpu_stw_le_mmuidx_ra +# define cpu_stl_mmuidx_ra cpu_stl_le_mmuidx_ra +# define cpu_stq_mmuidx_ra cpu_stq_le_mmuidx_ra +#endif + uint32_t cpu_ldub_code(CPUArchState *env, abi_ptr addr); uint32_t cpu_lduw_code(CPUArchState *env, abi_ptr addr); uint32_t cpu_ldl_code(CPUArchState *env, abi_ptr addr); diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c index c708e9785f..eb2cf9de5e 100644 --- a/accel/tcg/cputlb.c +++ b/accel/tcg/cputlb.c @@ -1772,36 +1772,54 @@ int cpu_ldsb_mmuidx_ra(CPUArchState *env, abi_ptr a= ddr, full_ldub_mmu); } =20 -uint32_t cpu_lduw_mmuidx_ra(CPUArchState *env, abi_ptr addr, - int mmu_idx, uintptr_t ra) +uint32_t cpu_lduw_be_mmuidx_ra(CPUArchState *env, abi_ptr addr, + int mmu_idx, uintptr_t ra) { - return cpu_load_helper(env, addr, mmu_idx, ra, MO_TEUW, - MO_TE =3D=3D MO_LE - ? full_le_lduw_mmu : full_be_lduw_mmu); + return cpu_load_helper(env, addr, mmu_idx, ra, MO_BEUW, full_be_lduw_m= mu); } =20 -int cpu_ldsw_mmuidx_ra(CPUArchState *env, abi_ptr addr, - int mmu_idx, uintptr_t ra) +int cpu_ldsw_be_mmuidx_ra(CPUArchState *env, abi_ptr addr, + int mmu_idx, uintptr_t ra) { - return (int16_t)cpu_load_helper(env, addr, mmu_idx, ra, MO_TESW, - MO_TE =3D=3D MO_LE - ? full_le_lduw_mmu : full_be_lduw_mmu); + return (int16_t)cpu_load_helper(env, addr, mmu_idx, ra, MO_BESW, + full_be_lduw_mmu); } =20 -uint32_t cpu_ldl_mmuidx_ra(CPUArchState *env, abi_ptr addr, - int mmu_idx, uintptr_t ra) +uint32_t cpu_ldl_be_mmuidx_ra(CPUArchState *env, abi_ptr addr, + int mmu_idx, uintptr_t ra) { - return cpu_load_helper(env, addr, mmu_idx, ra, MO_TEUL, - MO_TE =3D=3D MO_LE - ? full_le_ldul_mmu : full_be_ldul_mmu); + return cpu_load_helper(env, addr, mmu_idx, ra, MO_BEUL, full_be_ldul_m= mu); } =20 -uint64_t cpu_ldq_mmuidx_ra(CPUArchState *env, abi_ptr addr, - int mmu_idx, uintptr_t ra) +uint64_t cpu_ldq_be_mmuidx_ra(CPUArchState *env, abi_ptr addr, + int mmu_idx, uintptr_t ra) { - return cpu_load_helper(env, addr, mmu_idx, ra, MO_TEQ, - MO_TE =3D=3D MO_LE - ? helper_le_ldq_mmu : helper_be_ldq_mmu); + return cpu_load_helper(env, addr, mmu_idx, ra, MO_BEQ, helper_be_ldq_m= mu); +} + +uint32_t cpu_lduw_le_mmuidx_ra(CPUArchState *env, abi_ptr addr, + int mmu_idx, uintptr_t ra) +{ + return cpu_load_helper(env, addr, mmu_idx, ra, MO_LEUW, full_le_lduw_m= mu); +} + +int cpu_ldsw_le_mmuidx_ra(CPUArchState *env, abi_ptr addr, + int mmu_idx, uintptr_t ra) +{ + return (int16_t)cpu_load_helper(env, addr, mmu_idx, ra, MO_LESW, + full_le_lduw_mmu); +} + +uint32_t cpu_ldl_le_mmuidx_ra(CPUArchState *env, abi_ptr addr, + int mmu_idx, uintptr_t ra) +{ + return cpu_load_helper(env, addr, mmu_idx, ra, MO_LEUL, full_le_ldul_m= mu); +} + +uint64_t cpu_ldq_le_mmuidx_ra(CPUArchState *env, abi_ptr addr, + int mmu_idx, uintptr_t ra) +{ + return cpu_load_helper(env, addr, mmu_idx, ra, MO_LEQ, helper_le_ldq_m= mu); } =20 uint32_t cpu_ldub_data_ra(CPUArchState *env, target_ulong ptr, @@ -1815,25 +1833,50 @@ int cpu_ldsb_data_ra(CPUArchState *env, target_ulon= g ptr, uintptr_t retaddr) return cpu_ldsb_mmuidx_ra(env, ptr, cpu_mmu_index(env, false), retaddr= ); } =20 -uint32_t cpu_lduw_data_ra(CPUArchState *env, target_ulong ptr, - uintptr_t retaddr) +uint32_t cpu_lduw_be_data_ra(CPUArchState *env, target_ulong ptr, + uintptr_t retaddr) { - return cpu_lduw_mmuidx_ra(env, ptr, cpu_mmu_index(env, false), retaddr= ); + return cpu_lduw_be_mmuidx_ra(env, ptr, cpu_mmu_index(env, false), reta= ddr); } =20 -int cpu_ldsw_data_ra(CPUArchState *env, target_ulong ptr, uintptr_t retadd= r) +int cpu_ldsw_be_data_ra(CPUArchState *env, target_ulong ptr, uintptr_t ret= addr) { - return cpu_ldsw_mmuidx_ra(env, ptr, cpu_mmu_index(env, false), retaddr= ); + return cpu_ldsw_be_mmuidx_ra(env, ptr, cpu_mmu_index(env, false), reta= ddr); } =20 -uint32_t cpu_ldl_data_ra(CPUArchState *env, target_ulong ptr, uintptr_t re= taddr) +uint32_t cpu_ldl_be_data_ra(CPUArchState *env, target_ulong ptr, + uintptr_t retaddr) { - return cpu_ldl_mmuidx_ra(env, ptr, cpu_mmu_index(env, false), retaddr); + return cpu_ldl_be_mmuidx_ra(env, ptr, cpu_mmu_index(env, false), retad= dr); } =20 -uint64_t cpu_ldq_data_ra(CPUArchState *env, target_ulong ptr, uintptr_t re= taddr) +uint64_t cpu_ldq_be_data_ra(CPUArchState *env, target_ulong ptr, + uintptr_t retaddr) { - return cpu_ldq_mmuidx_ra(env, ptr, cpu_mmu_index(env, false), retaddr); + return cpu_ldq_be_mmuidx_ra(env, ptr, cpu_mmu_index(env, false), retad= dr); +} + +uint32_t cpu_lduw_le_data_ra(CPUArchState *env, target_ulong ptr, + uintptr_t retaddr) +{ + return cpu_lduw_le_mmuidx_ra(env, ptr, cpu_mmu_index(env, false), reta= ddr); +} + +int cpu_ldsw_le_data_ra(CPUArchState *env, target_ulong ptr, uintptr_t ret= addr) +{ + return cpu_ldsw_le_mmuidx_ra(env, ptr, cpu_mmu_index(env, false), reta= ddr); +} + +uint32_t cpu_ldl_le_data_ra(CPUArchState *env, target_ulong ptr, + uintptr_t retaddr) +{ + return cpu_ldl_le_mmuidx_ra(env, ptr, cpu_mmu_index(env, false), retad= dr); +} + +uint64_t cpu_ldq_le_data_ra(CPUArchState *env, target_ulong ptr, + uintptr_t retaddr) +{ + return cpu_ldq_le_mmuidx_ra(env, ptr, cpu_mmu_index(env, false), retad= dr); } =20 uint32_t cpu_ldub_data(CPUArchState *env, target_ulong ptr) @@ -1846,24 +1889,44 @@ int cpu_ldsb_data(CPUArchState *env, target_ulong p= tr) return cpu_ldsb_data_ra(env, ptr, 0); } =20 -uint32_t cpu_lduw_data(CPUArchState *env, target_ulong ptr) +uint32_t cpu_lduw_be_data(CPUArchState *env, target_ulong ptr) { - return cpu_lduw_data_ra(env, ptr, 0); + return cpu_lduw_be_data_ra(env, ptr, 0); } =20 -int cpu_ldsw_data(CPUArchState *env, target_ulong ptr) +int cpu_ldsw_be_data(CPUArchState *env, target_ulong ptr) { - return cpu_ldsw_data_ra(env, ptr, 0); + return cpu_ldsw_be_data_ra(env, ptr, 0); } =20 -uint32_t cpu_ldl_data(CPUArchState *env, target_ulong ptr) +uint32_t cpu_ldl_be_data(CPUArchState *env, target_ulong ptr) { - return cpu_ldl_data_ra(env, ptr, 0); + return cpu_ldl_be_data_ra(env, ptr, 0); } =20 -uint64_t cpu_ldq_data(CPUArchState *env, target_ulong ptr) +uint64_t cpu_ldq_be_data(CPUArchState *env, target_ulong ptr) { - return cpu_ldq_data_ra(env, ptr, 0); + return cpu_ldq_be_data_ra(env, ptr, 0); +} + +uint32_t cpu_lduw_le_data(CPUArchState *env, target_ulong ptr) +{ + return cpu_lduw_le_data_ra(env, ptr, 0); +} + +int cpu_ldsw_le_data(CPUArchState *env, target_ulong ptr) +{ + return cpu_ldsw_le_data_ra(env, ptr, 0); +} + +uint32_t cpu_ldl_le_data(CPUArchState *env, target_ulong ptr) +{ + return cpu_ldl_le_data_ra(env, ptr, 0); +} + +uint64_t cpu_ldq_le_data(CPUArchState *env, target_ulong ptr) +{ + return cpu_ldq_le_data_ra(env, ptr, 0); } =20 /* @@ -2121,22 +2184,40 @@ void cpu_stb_mmuidx_ra(CPUArchState *env, target_ul= ong addr, uint32_t val, cpu_store_helper(env, addr, val, mmu_idx, retaddr, MO_UB); } =20 -void cpu_stw_mmuidx_ra(CPUArchState *env, target_ulong addr, uint32_t val, - int mmu_idx, uintptr_t retaddr) +void cpu_stw_be_mmuidx_ra(CPUArchState *env, target_ulong addr, uint32_t v= al, + int mmu_idx, uintptr_t retaddr) { - cpu_store_helper(env, addr, val, mmu_idx, retaddr, MO_TEUW); + cpu_store_helper(env, addr, val, mmu_idx, retaddr, MO_BEUW); } =20 -void cpu_stl_mmuidx_ra(CPUArchState *env, target_ulong addr, uint32_t val, - int mmu_idx, uintptr_t retaddr) +void cpu_stl_be_mmuidx_ra(CPUArchState *env, target_ulong addr, uint32_t v= al, + int mmu_idx, uintptr_t retaddr) { - cpu_store_helper(env, addr, val, mmu_idx, retaddr, MO_TEUL); + cpu_store_helper(env, addr, val, mmu_idx, retaddr, MO_BEUL); } =20 -void cpu_stq_mmuidx_ra(CPUArchState *env, target_ulong addr, uint64_t val, - int mmu_idx, uintptr_t retaddr) +void cpu_stq_be_mmuidx_ra(CPUArchState *env, target_ulong addr, uint64_t v= al, + int mmu_idx, uintptr_t retaddr) { - cpu_store_helper(env, addr, val, mmu_idx, retaddr, MO_TEQ); + cpu_store_helper(env, addr, val, mmu_idx, retaddr, MO_BEQ); +} + +void cpu_stw_le_mmuidx_ra(CPUArchState *env, target_ulong addr, uint32_t v= al, + int mmu_idx, uintptr_t retaddr) +{ + cpu_store_helper(env, addr, val, mmu_idx, retaddr, MO_LEUW); +} + +void cpu_stl_le_mmuidx_ra(CPUArchState *env, target_ulong addr, uint32_t v= al, + int mmu_idx, uintptr_t retaddr) +{ + cpu_store_helper(env, addr, val, mmu_idx, retaddr, MO_LEUL); +} + +void cpu_stq_le_mmuidx_ra(CPUArchState *env, target_ulong addr, uint64_t v= al, + int mmu_idx, uintptr_t retaddr) +{ + cpu_store_helper(env, addr, val, mmu_idx, retaddr, MO_LEQ); } =20 void cpu_stb_data_ra(CPUArchState *env, target_ulong ptr, @@ -2145,22 +2226,40 @@ void cpu_stb_data_ra(CPUArchState *env, target_ulon= g ptr, cpu_stb_mmuidx_ra(env, ptr, val, cpu_mmu_index(env, false), retaddr); } =20 -void cpu_stw_data_ra(CPUArchState *env, target_ulong ptr, - uint32_t val, uintptr_t retaddr) +void cpu_stw_be_data_ra(CPUArchState *env, target_ulong ptr, + uint32_t val, uintptr_t retaddr) { - cpu_stw_mmuidx_ra(env, ptr, val, cpu_mmu_index(env, false), retaddr); + cpu_stw_be_mmuidx_ra(env, ptr, val, cpu_mmu_index(env, false), retaddr= ); } =20 -void cpu_stl_data_ra(CPUArchState *env, target_ulong ptr, - uint32_t val, uintptr_t retaddr) +void cpu_stl_be_data_ra(CPUArchState *env, target_ulong ptr, + uint32_t val, uintptr_t retaddr) { - cpu_stl_mmuidx_ra(env, ptr, val, cpu_mmu_index(env, false), retaddr); + cpu_stl_be_mmuidx_ra(env, ptr, val, cpu_mmu_index(env, false), retaddr= ); } =20 -void cpu_stq_data_ra(CPUArchState *env, target_ulong ptr, - uint64_t val, uintptr_t retaddr) +void cpu_stq_be_data_ra(CPUArchState *env, target_ulong ptr, + uint64_t val, uintptr_t retaddr) { - cpu_stq_mmuidx_ra(env, ptr, val, cpu_mmu_index(env, false), retaddr); + cpu_stq_be_mmuidx_ra(env, ptr, val, cpu_mmu_index(env, false), retaddr= ); +} + +void cpu_stw_le_data_ra(CPUArchState *env, target_ulong ptr, + uint32_t val, uintptr_t retaddr) +{ + cpu_stw_le_mmuidx_ra(env, ptr, val, cpu_mmu_index(env, false), retaddr= ); +} + +void cpu_stl_le_data_ra(CPUArchState *env, target_ulong ptr, + uint32_t val, uintptr_t retaddr) +{ + cpu_stl_le_mmuidx_ra(env, ptr, val, cpu_mmu_index(env, false), retaddr= ); +} + +void cpu_stq_le_data_ra(CPUArchState *env, target_ulong ptr, + uint64_t val, uintptr_t retaddr) +{ + cpu_stq_le_mmuidx_ra(env, ptr, val, cpu_mmu_index(env, false), retaddr= ); } =20 void cpu_stb_data(CPUArchState *env, target_ulong ptr, uint32_t val) @@ -2168,19 +2267,34 @@ void cpu_stb_data(CPUArchState *env, target_ulong p= tr, uint32_t val) cpu_stb_data_ra(env, ptr, val, 0); } =20 -void cpu_stw_data(CPUArchState *env, target_ulong ptr, uint32_t val) +void cpu_stw_be_data(CPUArchState *env, target_ulong ptr, uint32_t val) { - cpu_stw_data_ra(env, ptr, val, 0); + cpu_stw_be_data_ra(env, ptr, val, 0); } =20 -void cpu_stl_data(CPUArchState *env, target_ulong ptr, uint32_t val) +void cpu_stl_be_data(CPUArchState *env, target_ulong ptr, uint32_t val) { - cpu_stl_data_ra(env, ptr, val, 0); + cpu_stl_be_data_ra(env, ptr, val, 0); } =20 -void cpu_stq_data(CPUArchState *env, target_ulong ptr, uint64_t val) +void cpu_stq_be_data(CPUArchState *env, target_ulong ptr, uint64_t val) { - cpu_stq_data_ra(env, ptr, val, 0); + cpu_stq_be_data_ra(env, ptr, val, 0); +} + +void cpu_stw_le_data(CPUArchState *env, target_ulong ptr, uint32_t val) +{ + cpu_stw_le_data_ra(env, ptr, val, 0); +} + +void cpu_stl_le_data(CPUArchState *env, target_ulong ptr, uint32_t val) +{ + cpu_stl_le_data_ra(env, ptr, val, 0); +} + +void cpu_stq_le_data(CPUArchState *env, target_ulong ptr, uint64_t val) +{ + cpu_stq_le_data_ra(env, ptr, val, 0); } =20 /* First set of helpers allows passing in of OI and RETADDR. This makes diff --git a/accel/tcg/user-exec.c b/accel/tcg/user-exec.c index 987342c50c..52359949df 100644 --- a/accel/tcg/user-exec.c +++ b/accel/tcg/user-exec.c @@ -783,46 +783,90 @@ int cpu_ldsb_data(CPUArchState *env, abi_ptr ptr) return ret; } =20 -uint32_t cpu_lduw_data(CPUArchState *env, abi_ptr ptr) +uint32_t cpu_lduw_be_data(CPUArchState *env, abi_ptr ptr) { uint32_t ret; - uint16_t meminfo =3D trace_mem_get_info(MO_TEUW, MMU_USER_IDX, false); + uint16_t meminfo =3D trace_mem_get_info(MO_BEUW, MMU_USER_IDX, false); =20 trace_guest_mem_before_exec(env_cpu(env), ptr, meminfo); - ret =3D lduw_p(g2h(ptr)); + ret =3D lduw_be_p(g2h(ptr)); qemu_plugin_vcpu_mem_cb(env_cpu(env), ptr, meminfo); return ret; } =20 -int cpu_ldsw_data(CPUArchState *env, abi_ptr ptr) +int cpu_ldsw_be_data(CPUArchState *env, abi_ptr ptr) { int ret; - uint16_t meminfo =3D trace_mem_get_info(MO_TESW, MMU_USER_IDX, false); + uint16_t meminfo =3D trace_mem_get_info(MO_BESW, MMU_USER_IDX, false); =20 trace_guest_mem_before_exec(env_cpu(env), ptr, meminfo); - ret =3D ldsw_p(g2h(ptr)); + ret =3D ldsw_be_p(g2h(ptr)); qemu_plugin_vcpu_mem_cb(env_cpu(env), ptr, meminfo); return ret; } =20 -uint32_t cpu_ldl_data(CPUArchState *env, abi_ptr ptr) +uint32_t cpu_ldl_be_data(CPUArchState *env, abi_ptr ptr) { uint32_t ret; - uint16_t meminfo =3D trace_mem_get_info(MO_TEUL, MMU_USER_IDX, false); + uint16_t meminfo =3D trace_mem_get_info(MO_BEUL, MMU_USER_IDX, false); =20 trace_guest_mem_before_exec(env_cpu(env), ptr, meminfo); - ret =3D ldl_p(g2h(ptr)); + ret =3D ldl_be_p(g2h(ptr)); qemu_plugin_vcpu_mem_cb(env_cpu(env), ptr, meminfo); return ret; } =20 -uint64_t cpu_ldq_data(CPUArchState *env, abi_ptr ptr) +uint64_t cpu_ldq_be_data(CPUArchState *env, abi_ptr ptr) { uint64_t ret; - uint16_t meminfo =3D trace_mem_get_info(MO_TEQ, MMU_USER_IDX, false); + uint16_t meminfo =3D trace_mem_get_info(MO_BEQ, MMU_USER_IDX, false); =20 trace_guest_mem_before_exec(env_cpu(env), ptr, meminfo); - ret =3D ldq_p(g2h(ptr)); + ret =3D ldq_be_p(g2h(ptr)); + qemu_plugin_vcpu_mem_cb(env_cpu(env), ptr, meminfo); + return ret; +} + +uint32_t cpu_lduw_le_data(CPUArchState *env, abi_ptr ptr) +{ + uint32_t ret; + uint16_t meminfo =3D trace_mem_get_info(MO_LEUW, MMU_USER_IDX, false); + + trace_guest_mem_before_exec(env_cpu(env), ptr, meminfo); + ret =3D lduw_le_p(g2h(ptr)); + qemu_plugin_vcpu_mem_cb(env_cpu(env), ptr, meminfo); + return ret; +} + +int cpu_ldsw_le_data(CPUArchState *env, abi_ptr ptr) +{ + int ret; + uint16_t meminfo =3D trace_mem_get_info(MO_LESW, MMU_USER_IDX, false); + + trace_guest_mem_before_exec(env_cpu(env), ptr, meminfo); + ret =3D ldsw_le_p(g2h(ptr)); + qemu_plugin_vcpu_mem_cb(env_cpu(env), ptr, meminfo); + return ret; +} + +uint32_t cpu_ldl_le_data(CPUArchState *env, abi_ptr ptr) +{ + uint32_t ret; + uint16_t meminfo =3D trace_mem_get_info(MO_LEUL, MMU_USER_IDX, false); + + trace_guest_mem_before_exec(env_cpu(env), ptr, meminfo); + ret =3D ldl_le_p(g2h(ptr)); + qemu_plugin_vcpu_mem_cb(env_cpu(env), ptr, meminfo); + return ret; +} + +uint64_t cpu_ldq_le_data(CPUArchState *env, abi_ptr ptr) +{ + uint64_t ret; + uint16_t meminfo =3D trace_mem_get_info(MO_LEQ, MMU_USER_IDX, false); + + trace_guest_mem_before_exec(env_cpu(env), ptr, meminfo); + ret =3D ldq_le_p(g2h(ptr)); qemu_plugin_vcpu_mem_cb(env_cpu(env), ptr, meminfo); return ret; } @@ -847,42 +891,82 @@ int cpu_ldsb_data_ra(CPUArchState *env, abi_ptr ptr, = uintptr_t retaddr) return ret; } =20 -uint32_t cpu_lduw_data_ra(CPUArchState *env, abi_ptr ptr, uintptr_t retadd= r) +uint32_t cpu_lduw_be_data_ra(CPUArchState *env, abi_ptr ptr, uintptr_t ret= addr) { uint32_t ret; =20 set_helper_retaddr(retaddr); - ret =3D cpu_lduw_data(env, ptr); + ret =3D cpu_lduw_be_data(env, ptr); clear_helper_retaddr(); return ret; } =20 -int cpu_ldsw_data_ra(CPUArchState *env, abi_ptr ptr, uintptr_t retaddr) +int cpu_ldsw_be_data_ra(CPUArchState *env, abi_ptr ptr, uintptr_t retaddr) { int ret; =20 set_helper_retaddr(retaddr); - ret =3D cpu_ldsw_data(env, ptr); + ret =3D cpu_ldsw_be_data(env, ptr); clear_helper_retaddr(); return ret; } =20 -uint32_t cpu_ldl_data_ra(CPUArchState *env, abi_ptr ptr, uintptr_t retaddr) +uint32_t cpu_ldl_be_data_ra(CPUArchState *env, abi_ptr ptr, uintptr_t reta= ddr) { uint32_t ret; =20 set_helper_retaddr(retaddr); - ret =3D cpu_ldl_data(env, ptr); + ret =3D cpu_ldl_be_data(env, ptr); clear_helper_retaddr(); return ret; } =20 -uint64_t cpu_ldq_data_ra(CPUArchState *env, abi_ptr ptr, uintptr_t retaddr) +uint64_t cpu_ldq_be_data_ra(CPUArchState *env, abi_ptr ptr, uintptr_t reta= ddr) { uint64_t ret; =20 set_helper_retaddr(retaddr); - ret =3D cpu_ldq_data(env, ptr); + ret =3D cpu_ldq_be_data(env, ptr); + clear_helper_retaddr(); + return ret; +} + +uint32_t cpu_lduw_le_data_ra(CPUArchState *env, abi_ptr ptr, uintptr_t ret= addr) +{ + uint32_t ret; + + set_helper_retaddr(retaddr); + ret =3D cpu_lduw_le_data(env, ptr); + clear_helper_retaddr(); + return ret; +} + +int cpu_ldsw_le_data_ra(CPUArchState *env, abi_ptr ptr, uintptr_t retaddr) +{ + int ret; + + set_helper_retaddr(retaddr); + ret =3D cpu_ldsw_le_data(env, ptr); + clear_helper_retaddr(); + return ret; +} + +uint32_t cpu_ldl_le_data_ra(CPUArchState *env, abi_ptr ptr, uintptr_t reta= ddr) +{ + uint32_t ret; + + set_helper_retaddr(retaddr); + ret =3D cpu_ldl_le_data(env, ptr); + clear_helper_retaddr(); + return ret; +} + +uint64_t cpu_ldq_le_data_ra(CPUArchState *env, abi_ptr ptr, uintptr_t reta= ddr) +{ + uint64_t ret; + + set_helper_retaddr(retaddr); + ret =3D cpu_ldq_le_data(env, ptr); clear_helper_retaddr(); return ret; } @@ -896,30 +980,57 @@ void cpu_stb_data(CPUArchState *env, abi_ptr ptr, uin= t32_t val) qemu_plugin_vcpu_mem_cb(env_cpu(env), ptr, meminfo); } =20 -void cpu_stw_data(CPUArchState *env, abi_ptr ptr, uint32_t val) +void cpu_stw_be_data(CPUArchState *env, abi_ptr ptr, uint32_t val) { - uint16_t meminfo =3D trace_mem_get_info(MO_TEUW, MMU_USER_IDX, true); + uint16_t meminfo =3D trace_mem_get_info(MO_BEUW, MMU_USER_IDX, true); =20 trace_guest_mem_before_exec(env_cpu(env), ptr, meminfo); - stw_p(g2h(ptr), val); + stw_be_p(g2h(ptr), val); qemu_plugin_vcpu_mem_cb(env_cpu(env), ptr, meminfo); } =20 -void cpu_stl_data(CPUArchState *env, abi_ptr ptr, uint32_t val) +void cpu_stl_be_data(CPUArchState *env, abi_ptr ptr, uint32_t val) { - uint16_t meminfo =3D trace_mem_get_info(MO_TEUL, MMU_USER_IDX, true); + uint16_t meminfo =3D trace_mem_get_info(MO_BEUL, MMU_USER_IDX, true); =20 trace_guest_mem_before_exec(env_cpu(env), ptr, meminfo); - stl_p(g2h(ptr), val); + stl_be_p(g2h(ptr), val); qemu_plugin_vcpu_mem_cb(env_cpu(env), ptr, meminfo); } =20 -void cpu_stq_data(CPUArchState *env, abi_ptr ptr, uint64_t val) +void cpu_stq_be_data(CPUArchState *env, abi_ptr ptr, uint64_t val) { - uint16_t meminfo =3D trace_mem_get_info(MO_TEQ, MMU_USER_IDX, true); + uint16_t meminfo =3D trace_mem_get_info(MO_BEQ, MMU_USER_IDX, true); =20 trace_guest_mem_before_exec(env_cpu(env), ptr, meminfo); - stq_p(g2h(ptr), val); + stq_be_p(g2h(ptr), val); + qemu_plugin_vcpu_mem_cb(env_cpu(env), ptr, meminfo); +} + +void cpu_stw_le_data(CPUArchState *env, abi_ptr ptr, uint32_t val) +{ + uint16_t meminfo =3D trace_mem_get_info(MO_LEUW, MMU_USER_IDX, true); + + trace_guest_mem_before_exec(env_cpu(env), ptr, meminfo); + stw_le_p(g2h(ptr), val); + qemu_plugin_vcpu_mem_cb(env_cpu(env), ptr, meminfo); +} + +void cpu_stl_le_data(CPUArchState *env, abi_ptr ptr, uint32_t val) +{ + uint16_t meminfo =3D trace_mem_get_info(MO_LEUL, MMU_USER_IDX, true); + + trace_guest_mem_before_exec(env_cpu(env), ptr, meminfo); + stl_le_p(g2h(ptr), val); + qemu_plugin_vcpu_mem_cb(env_cpu(env), ptr, meminfo); +} + +void cpu_stq_le_data(CPUArchState *env, abi_ptr ptr, uint64_t val) +{ + uint16_t meminfo =3D trace_mem_get_info(MO_LEQ, MMU_USER_IDX, true); + + trace_guest_mem_before_exec(env_cpu(env), ptr, meminfo); + stq_le_p(g2h(ptr), val); qemu_plugin_vcpu_mem_cb(env_cpu(env), ptr, meminfo); } =20 @@ -931,27 +1042,51 @@ void cpu_stb_data_ra(CPUArchState *env, abi_ptr ptr, clear_helper_retaddr(); } =20 -void cpu_stw_data_ra(CPUArchState *env, abi_ptr ptr, - uint32_t val, uintptr_t retaddr) +void cpu_stw_be_data_ra(CPUArchState *env, abi_ptr ptr, + uint32_t val, uintptr_t retaddr) { set_helper_retaddr(retaddr); - cpu_stw_data(env, ptr, val); + cpu_stw_be_data(env, ptr, val); clear_helper_retaddr(); } =20 -void cpu_stl_data_ra(CPUArchState *env, abi_ptr ptr, - uint32_t val, uintptr_t retaddr) +void cpu_stl_be_data_ra(CPUArchState *env, abi_ptr ptr, + uint32_t val, uintptr_t retaddr) { set_helper_retaddr(retaddr); - cpu_stl_data(env, ptr, val); + cpu_stl_be_data(env, ptr, val); clear_helper_retaddr(); } =20 -void cpu_stq_data_ra(CPUArchState *env, abi_ptr ptr, - uint64_t val, uintptr_t retaddr) +void cpu_stq_be_data_ra(CPUArchState *env, abi_ptr ptr, + uint64_t val, uintptr_t retaddr) { set_helper_retaddr(retaddr); - cpu_stq_data(env, ptr, val); + cpu_stq_be_data(env, ptr, val); + clear_helper_retaddr(); +} + +void cpu_stw_le_data_ra(CPUArchState *env, abi_ptr ptr, + uint32_t val, uintptr_t retaddr) +{ + set_helper_retaddr(retaddr); + cpu_stw_le_data(env, ptr, val); + clear_helper_retaddr(); +} + +void cpu_stl_le_data_ra(CPUArchState *env, abi_ptr ptr, + uint32_t val, uintptr_t retaddr) +{ + set_helper_retaddr(retaddr); + cpu_stl_le_data(env, ptr, val); + clear_helper_retaddr(); +} + +void cpu_stq_le_data_ra(CPUArchState *env, abi_ptr ptr, + uint64_t val, uintptr_t retaddr) +{ + set_helper_retaddr(retaddr); + cpu_stq_le_data(env, ptr, val); clear_helper_retaddr(); } =20 --=20 2.20.1 From nobody Sun May 19 14:32:52 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=linaro.org ARC-Seal: i=1; a=rsa-sha256; t=1588264704; cv=none; d=zohomail.com; s=zohoarc; b=ZVm+BccBCPvYZftp/MNoRyDAl9BU4tCYoEXYDb4WkkZUhynHC6gunmSLsM7AQ2y/wniGwOhNqrlcKY+7GXkagNkdTz46Pv3EoDBmQZfZpJItvsNz3/a6668nGggujyFvSgWPK4nLOyRnC5hQGGFB+fHoXBCzn2QXvkSRsGnHCDs= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1588264704; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=inLNN7uJ7qGWpr+8fOsifMmei61PsL0wTHKUUOgzn/8=; b=FHCxmqKsdlJMW/rJqxzzzQApJRPqn6Luoy5dGXVAhRTvw/0j8WeBb+hSPxiqvYiAW+JMWpyhRlbpIBf+/qFAG0FTAwK1EL32SfTbl6heW86Ur9xLFCqKmcVha4FvfG63MizlsjgBCylfH68c0HlZEzbW2vghLD0KjWYXMimW2eQ= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1588264704678222.07814295738137; Thu, 30 Apr 2020 09:38:24 -0700 (PDT) Received: from localhost ([::1]:56360 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jUCCd-000737-3Y for importer@patchew.org; Thu, 30 Apr 2020 12:38:23 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:49326) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jUC32-0001hq-PK for qemu-devel@nongnu.org; Thu, 30 Apr 2020 12:28:31 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.90_1) (envelope-from ) id 1jUC30-0006NC-A4 for qemu-devel@nongnu.org; Thu, 30 Apr 2020 12:28:28 -0400 Received: from mail-pg1-x543.google.com ([2607:f8b0:4864:20::543]:38208) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jUC2z-0006Mr-Rj for qemu-devel@nongnu.org; Thu, 30 Apr 2020 12:28:25 -0400 Received: by mail-pg1-x543.google.com with SMTP id l25so645572pgc.5 for ; Thu, 30 Apr 2020 09:28:25 -0700 (PDT) Received: from localhost.localdomain (174-21-149-226.tukw.qwest.net. [174.21.149.226]) by smtp.gmail.com with ESMTPSA id j5sm243514pfh.58.2020.04.30.09.28.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Apr 2020 09:28:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=inLNN7uJ7qGWpr+8fOsifMmei61PsL0wTHKUUOgzn/8=; b=eGvdEsseUpik6e92Km0QYRFMGJg43oz00Q8RjbuR8bMf6FuD73n/MT/xDehonVgPt5 21zR4gaOPDr+sZM5cgY4putSirAwQ3hQuL9QEvqYaKZWzKJftIc3EnTXPaKhm9ua+9S+ GH2NLbmIe3mfMJqMzO8/E+1/xMAnZC+LmDdntef8+dNo8KZVRp3fMD16m55HWPdW8OBD v6oS004qMmXcJfjk+ttVNuVYtjFQt9Yta5u6/oMaCMGx8nSvIhDQTbgIOOFPvQsrdH41 wchRJXao4MObIR1ebgngfx0XhSNzIGcalxZMeCIJBRqyyKTVsMWSgu3fJyFE2t8x2/6d gO9Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=inLNN7uJ7qGWpr+8fOsifMmei61PsL0wTHKUUOgzn/8=; b=eSRk3+koMKC4DGTmeV04etQL6q4muKmnWHcT5Z/5KtYT7kS4tHKzYw1t/+/E8I68VO Z9ZeZwVf6pn9/+g8w0KQ2VWCJhEQgEcVxb605pVUQzQQWUZ/3p592R0HVsh+hQyto1xO 390DGbxcK4RYmQZa3QgDTsN63BSjdp0cAMewi/P+ZlXmOg5r/6scDbS6DlhevBlD/1w4 tTCcoHdwtE+0RwAqrfgRq1CG4Ez1jq3mM6tbpz48tm6j1ZR6ka+gvoiz0aQfLe/DljvL BpLBonpP07gSpFln2/eNi7HnqU2rM2lq9PJzwLY+glUdsCAHK+rljeOUcvGtnQIo9Z5t 7cZA== X-Gm-Message-State: AGi0PuZpl0JQBrTk1YaqzC4exSHx90VbIkw3s0FmIai/PCOoBVfba/1S FN8eNOMSBGsyMZP5k1KPIKhA3JWJWOc= X-Google-Smtp-Source: APiQypIF7XDhLbNMPfrj5WsTTaQwxf4xgzVVQYyy6urhwyWbhcSPH2pGrzJ0wpUWOKlpve+QQW8/vA== X-Received: by 2002:a63:7742:: with SMTP id s63mr3991341pgc.133.1588264103743; Thu, 30 Apr 2020 09:28:23 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH v4 06/18] target/arm: Use cpu_*_data_ra for sve_ldst_tlb_fn Date: Thu, 30 Apr 2020 09:28:01 -0700 Message-Id: <20200430162813.17671-7-richard.henderson@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200430162813.17671-1-richard.henderson@linaro.org> References: <20200430162813.17671-1-richard.henderson@linaro.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2607:f8b0:4864:20::543; envelope-from=richard.henderson@linaro.org; helo=mail-pg1-x543.google.com X-detected-operating-system: by eggs.gnu.org: Error: [-] PROGRAM ABORT : Malformed IPv6 address (bad octet value). Location : parse_addr6(), p0f-client.c:67 X-Received-From: 2607:f8b0:4864:20::543 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, alex.bennee@linaro.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @linaro.org) Content-Type: text/plain; charset="utf-8" Use the "normal" memory access functions, rather than the softmmu internal helper functions directly. Since fb901c905dc3, cpu_mem_index is now a simple extract from env->hflags and not a large computation. Which means that it's now more work to pass around this value than it is to recompute it. This only adjusts the primitives, and does not clean up all of the uses within sve_helper.c. Reviewed-by: Peter Maydell Signed-off-by: Richard Henderson --- target/arm/sve_helper.c | 221 ++++++++++++++++------------------------ 1 file changed, 86 insertions(+), 135 deletions(-) diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index fdfa652094..655bc9476f 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -3991,9 +3991,8 @@ typedef intptr_t sve_ld1_host_fn(void *vd, void *vg, = void *host, * Load one element into @vd + @reg_off from (@env, @vaddr, @ra). * The controlling predicate is known to be true. */ -typedef void sve_ld1_tlb_fn(CPUARMState *env, void *vd, intptr_t reg_off, - target_ulong vaddr, TCGMemOpIdx oi, uintptr_t = ra); -typedef sve_ld1_tlb_fn sve_st1_tlb_fn; +typedef void sve_ldst1_tlb_fn(CPUARMState *env, void *vd, intptr_t reg_off, + target_ulong vaddr, uintptr_t retaddr); =20 /* * Generate the above primitives. @@ -4016,27 +4015,23 @@ static intptr_t sve_##NAME##_host(void *vd, void *v= g, void *host, \ return mem_off; = \ } =20 -#ifdef CONFIG_SOFTMMU -#define DO_LD_TLB(NAME, H, TYPEE, TYPEM, HOST, MOEND, TLB) \ +#define DO_LD_TLB(NAME, H, TYPEE, TYPEM, TLB) \ static void sve_##NAME##_tlb(CPUARMState *env, void *vd, intptr_t reg_off,= \ - target_ulong addr, TCGMemOpIdx oi, uintptr_t = ra) \ + target_ulong addr, uintptr_t ra) = \ { = \ - TYPEM val =3D TLB(env, addr, oi, ra); = \ - *(TYPEE *)(vd + H(reg_off)) =3D val; = \ + *(TYPEE *)(vd + H(reg_off)) =3D (TYPEM)TLB(env, addr, ra); = \ } -#else -#define DO_LD_TLB(NAME, H, TYPEE, TYPEM, HOST, MOEND, TLB) = \ + +#define DO_ST_TLB(NAME, H, TYPEE, TYPEM, TLB) \ static void sve_##NAME##_tlb(CPUARMState *env, void *vd, intptr_t reg_off,= \ - target_ulong addr, TCGMemOpIdx oi, uintptr_t = ra) \ + target_ulong addr, uintptr_t ra) = \ { = \ - TYPEM val =3D HOST(g2h(addr)); = \ - *(TYPEE *)(vd + H(reg_off)) =3D val; = \ + TLB(env, addr, (TYPEM)*(TYPEE *)(vd + H(reg_off)), ra); = \ } -#endif =20 #define DO_LD_PRIM_1(NAME, H, TE, TM) \ DO_LD_HOST(NAME, H, TE, TM, ldub_p) \ - DO_LD_TLB(NAME, H, TE, TM, ldub_p, 0, helper_ret_ldub_mmu) + DO_LD_TLB(NAME, H, TE, TM, cpu_ldub_data_ra) =20 DO_LD_PRIM_1(ld1bb, H1, uint8_t, uint8_t) DO_LD_PRIM_1(ld1bhu, H1_2, uint16_t, uint8_t) @@ -4046,39 +4041,51 @@ DO_LD_PRIM_1(ld1bss, H1_4, uint32_t, int8_t) DO_LD_PRIM_1(ld1bdu, , uint64_t, uint8_t) DO_LD_PRIM_1(ld1bds, , uint64_t, int8_t) =20 -#define DO_LD_PRIM_2(NAME, end, MOEND, H, TE, TM, PH, PT) \ - DO_LD_HOST(NAME##_##end, H, TE, TM, PH##_##end##_p) \ - DO_LD_TLB(NAME##_##end, H, TE, TM, PH##_##end##_p, \ - MOEND, helper_##end##_##PT##_mmu) +#define DO_ST_PRIM_1(NAME, H, TE, TM) \ + DO_ST_TLB(st1##NAME, H, TE, TM, cpu_stb_data_ra) =20 -DO_LD_PRIM_2(ld1hh, le, MO_LE, H1_2, uint16_t, uint16_t, lduw, lduw) -DO_LD_PRIM_2(ld1hsu, le, MO_LE, H1_4, uint32_t, uint16_t, lduw, lduw) -DO_LD_PRIM_2(ld1hss, le, MO_LE, H1_4, uint32_t, int16_t, lduw, lduw) -DO_LD_PRIM_2(ld1hdu, le, MO_LE, , uint64_t, uint16_t, lduw, lduw) -DO_LD_PRIM_2(ld1hds, le, MO_LE, , uint64_t, int16_t, lduw, lduw) +DO_ST_PRIM_1(bb, H1, uint8_t, uint8_t) +DO_ST_PRIM_1(bh, H1_2, uint16_t, uint8_t) +DO_ST_PRIM_1(bs, H1_4, uint32_t, uint8_t) +DO_ST_PRIM_1(bd, , uint64_t, uint8_t) =20 -DO_LD_PRIM_2(ld1ss, le, MO_LE, H1_4, uint32_t, uint32_t, ldl, ldul) -DO_LD_PRIM_2(ld1sdu, le, MO_LE, , uint64_t, uint32_t, ldl, ldul) -DO_LD_PRIM_2(ld1sds, le, MO_LE, , uint64_t, int32_t, ldl, ldul) +#define DO_LD_PRIM_2(NAME, H, TE, TM, LD) \ + DO_LD_HOST(ld1##NAME##_be, H, TE, TM, LD##_be_p) \ + DO_LD_HOST(ld1##NAME##_le, H, TE, TM, LD##_le_p) \ + DO_LD_TLB(ld1##NAME##_be, H, TE, TM, cpu_##LD##_be_data_ra) \ + DO_LD_TLB(ld1##NAME##_le, H, TE, TM, cpu_##LD##_le_data_ra) =20 -DO_LD_PRIM_2(ld1dd, le, MO_LE, , uint64_t, uint64_t, ldq, ldq) +#define DO_ST_PRIM_2(NAME, H, TE, TM, ST) \ + DO_ST_TLB(st1##NAME##_be, H, TE, TM, cpu_##ST##_be_data_ra) \ + DO_ST_TLB(st1##NAME##_le, H, TE, TM, cpu_##ST##_le_data_ra) =20 -DO_LD_PRIM_2(ld1hh, be, MO_BE, H1_2, uint16_t, uint16_t, lduw, lduw) -DO_LD_PRIM_2(ld1hsu, be, MO_BE, H1_4, uint32_t, uint16_t, lduw, lduw) -DO_LD_PRIM_2(ld1hss, be, MO_BE, H1_4, uint32_t, int16_t, lduw, lduw) -DO_LD_PRIM_2(ld1hdu, be, MO_BE, , uint64_t, uint16_t, lduw, lduw) -DO_LD_PRIM_2(ld1hds, be, MO_BE, , uint64_t, int16_t, lduw, lduw) +DO_LD_PRIM_2(hh, H1_2, uint16_t, uint16_t, lduw) +DO_LD_PRIM_2(hsu, H1_4, uint32_t, uint16_t, lduw) +DO_LD_PRIM_2(hss, H1_4, uint32_t, int16_t, lduw) +DO_LD_PRIM_2(hdu, , uint64_t, uint16_t, lduw) +DO_LD_PRIM_2(hds, , uint64_t, int16_t, lduw) =20 -DO_LD_PRIM_2(ld1ss, be, MO_BE, H1_4, uint32_t, uint32_t, ldl, ldul) -DO_LD_PRIM_2(ld1sdu, be, MO_BE, , uint64_t, uint32_t, ldl, ldul) -DO_LD_PRIM_2(ld1sds, be, MO_BE, , uint64_t, int32_t, ldl, ldul) +DO_ST_PRIM_2(hh, H1_2, uint16_t, uint16_t, stw) +DO_ST_PRIM_2(hs, H1_4, uint32_t, uint16_t, stw) +DO_ST_PRIM_2(hd, , uint64_t, uint16_t, stw) =20 -DO_LD_PRIM_2(ld1dd, be, MO_BE, , uint64_t, uint64_t, ldq, ldq) +DO_LD_PRIM_2(ss, H1_4, uint32_t, uint32_t, ldl) +DO_LD_PRIM_2(sdu, , uint64_t, uint32_t, ldl) +DO_LD_PRIM_2(sds, , uint64_t, int32_t, ldl) + +DO_ST_PRIM_2(ss, H1_4, uint32_t, uint32_t, stl) +DO_ST_PRIM_2(sd, , uint64_t, uint32_t, stl) + +DO_LD_PRIM_2(dd, , uint64_t, uint64_t, ldq) +DO_ST_PRIM_2(dd, , uint64_t, uint64_t, stq) =20 #undef DO_LD_TLB +#undef DO_ST_TLB #undef DO_LD_HOST #undef DO_LD_PRIM_1 +#undef DO_ST_PRIM_1 #undef DO_LD_PRIM_2 +#undef DO_ST_PRIM_2 =20 /* * Skip through a sequence of inactive elements in the guarding predicate = @vg, @@ -4152,7 +4159,7 @@ static void sve_ld1_r(CPUARMState *env, void *vg, con= st target_ulong addr, uint32_t desc, const uintptr_t retaddr, const int esz, const int msz, sve_ld1_host_fn *host_fn, - sve_ld1_tlb_fn *tlb_fn) + sve_ldst1_tlb_fn *tlb_fn) { const TCGMemOpIdx oi =3D extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHI= FT); const int mmu_idx =3D get_mmuidx(oi); @@ -4234,7 +4241,7 @@ static void sve_ld1_r(CPUARMState *env, void *vg, con= st target_ulong addr, * on I/O memory, it may succeed but not bring in the TLB entry. * But even then we have still made forward progress. */ - tlb_fn(env, &scratch, reg_off, addr + mem_off, oi, retaddr); + tlb_fn(env, &scratch, reg_off, addr + mem_off, retaddr); reg_off +=3D 1 << esz; } #endif @@ -4293,9 +4300,8 @@ DO_LD1_2(ld1dd, 3, 3) */ static void sve_ld2_r(CPUARMState *env, void *vg, target_ulong addr, uint32_t desc, int size, uintptr_t ra, - sve_ld1_tlb_fn *tlb_fn) + sve_ldst1_tlb_fn *tlb_fn) { - const TCGMemOpIdx oi =3D extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHI= FT); const unsigned rd =3D extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT= , 5); intptr_t i, oprsz =3D simd_oprsz(desc); ARMVectorReg scratch[2] =3D { }; @@ -4305,8 +4311,8 @@ static void sve_ld2_r(CPUARMState *env, void *vg, tar= get_ulong addr, uint16_t pg =3D *(uint16_t *)(vg + H1_2(i >> 3)); do { if (pg & 1) { - tlb_fn(env, &scratch[0], i, addr, oi, ra); - tlb_fn(env, &scratch[1], i, addr + size, oi, ra); + tlb_fn(env, &scratch[0], i, addr, ra); + tlb_fn(env, &scratch[1], i, addr + size, ra); } i +=3D size, pg >>=3D size; addr +=3D 2 * size; @@ -4321,9 +4327,8 @@ static void sve_ld2_r(CPUARMState *env, void *vg, tar= get_ulong addr, =20 static void sve_ld3_r(CPUARMState *env, void *vg, target_ulong addr, uint32_t desc, int size, uintptr_t ra, - sve_ld1_tlb_fn *tlb_fn) + sve_ldst1_tlb_fn *tlb_fn) { - const TCGMemOpIdx oi =3D extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHI= FT); const unsigned rd =3D extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT= , 5); intptr_t i, oprsz =3D simd_oprsz(desc); ARMVectorReg scratch[3] =3D { }; @@ -4333,9 +4338,9 @@ static void sve_ld3_r(CPUARMState *env, void *vg, tar= get_ulong addr, uint16_t pg =3D *(uint16_t *)(vg + H1_2(i >> 3)); do { if (pg & 1) { - tlb_fn(env, &scratch[0], i, addr, oi, ra); - tlb_fn(env, &scratch[1], i, addr + size, oi, ra); - tlb_fn(env, &scratch[2], i, addr + 2 * size, oi, ra); + tlb_fn(env, &scratch[0], i, addr, ra); + tlb_fn(env, &scratch[1], i, addr + size, ra); + tlb_fn(env, &scratch[2], i, addr + 2 * size, ra); } i +=3D size, pg >>=3D size; addr +=3D 3 * size; @@ -4351,9 +4356,8 @@ static void sve_ld3_r(CPUARMState *env, void *vg, tar= get_ulong addr, =20 static void sve_ld4_r(CPUARMState *env, void *vg, target_ulong addr, uint32_t desc, int size, uintptr_t ra, - sve_ld1_tlb_fn *tlb_fn) + sve_ldst1_tlb_fn *tlb_fn) { - const TCGMemOpIdx oi =3D extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHI= FT); const unsigned rd =3D extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT= , 5); intptr_t i, oprsz =3D simd_oprsz(desc); ARMVectorReg scratch[4] =3D { }; @@ -4363,10 +4367,10 @@ static void sve_ld4_r(CPUARMState *env, void *vg, t= arget_ulong addr, uint16_t pg =3D *(uint16_t *)(vg + H1_2(i >> 3)); do { if (pg & 1) { - tlb_fn(env, &scratch[0], i, addr, oi, ra); - tlb_fn(env, &scratch[1], i, addr + size, oi, ra); - tlb_fn(env, &scratch[2], i, addr + 2 * size, oi, ra); - tlb_fn(env, &scratch[3], i, addr + 3 * size, oi, ra); + tlb_fn(env, &scratch[0], i, addr, ra); + tlb_fn(env, &scratch[1], i, addr + size, ra); + tlb_fn(env, &scratch[2], i, addr + 2 * size, ra); + tlb_fn(env, &scratch[3], i, addr + 3 * size, ra); } i +=3D size, pg >>=3D size; addr +=3D 4 * size; @@ -4459,7 +4463,7 @@ static void sve_ldff1_r(CPUARMState *env, void *vg, c= onst target_ulong addr, uint32_t desc, const uintptr_t retaddr, const int esz, const int msz, sve_ld1_host_fn *host_fn, - sve_ld1_tlb_fn *tlb_fn) + sve_ldst1_tlb_fn *tlb_fn) { const TCGMemOpIdx oi =3D extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHI= FT); const int mmu_idx =3D get_mmuidx(oi); @@ -4519,7 +4523,7 @@ static void sve_ldff1_r(CPUARMState *env, void *vg, c= onst target_ulong addr, * Perform one normal read, which will fault or not. * But it is likely to bring the page into the tlb. */ - tlb_fn(env, vd, reg_off, addr + mem_off, oi, retaddr); + tlb_fn(env, vd, reg_off, addr + mem_off, retaddr); =20 /* After any fault, zero any leading predicated false elts. */ swap_memzero(vd, reg_off); @@ -4671,60 +4675,14 @@ DO_LDFF1_LDNF1_2(dd, 3, 3) #undef DO_LDFF1_LDNF1_1 #undef DO_LDFF1_LDNF1_2 =20 -/* - * Store contiguous data, protected by a governing predicate. - */ - -#ifdef CONFIG_SOFTMMU -#define DO_ST_TLB(NAME, H, TYPEM, HOST, MOEND, TLB) \ -static void sve_##NAME##_tlb(CPUARMState *env, void *vd, intptr_t reg_off,= \ - target_ulong addr, TCGMemOpIdx oi, uintptr_t = ra) \ -{ = \ - TLB(env, addr, *(TYPEM *)(vd + H(reg_off)), oi, ra); = \ -} -#else -#define DO_ST_TLB(NAME, H, TYPEM, HOST, MOEND, TLB) \ -static void sve_##NAME##_tlb(CPUARMState *env, void *vd, intptr_t reg_off,= \ - target_ulong addr, TCGMemOpIdx oi, uintptr_t = ra) \ -{ = \ - HOST(g2h(addr), *(TYPEM *)(vd + H(reg_off))); = \ -} -#endif - -DO_ST_TLB(st1bb, H1, uint8_t, stb_p, 0, helper_ret_stb_mmu) -DO_ST_TLB(st1bh, H1_2, uint16_t, stb_p, 0, helper_ret_stb_mmu) -DO_ST_TLB(st1bs, H1_4, uint32_t, stb_p, 0, helper_ret_stb_mmu) -DO_ST_TLB(st1bd, , uint64_t, stb_p, 0, helper_ret_stb_mmu) - -DO_ST_TLB(st1hh_le, H1_2, uint16_t, stw_le_p, MO_LE, helper_le_stw_mmu) -DO_ST_TLB(st1hs_le, H1_4, uint32_t, stw_le_p, MO_LE, helper_le_stw_mmu) -DO_ST_TLB(st1hd_le, , uint64_t, stw_le_p, MO_LE, helper_le_stw_mmu) - -DO_ST_TLB(st1ss_le, H1_4, uint32_t, stl_le_p, MO_LE, helper_le_stl_mmu) -DO_ST_TLB(st1sd_le, , uint64_t, stl_le_p, MO_LE, helper_le_stl_mmu) - -DO_ST_TLB(st1dd_le, , uint64_t, stq_le_p, MO_LE, helper_le_stq_mmu) - -DO_ST_TLB(st1hh_be, H1_2, uint16_t, stw_be_p, MO_BE, helper_be_stw_mmu) -DO_ST_TLB(st1hs_be, H1_4, uint32_t, stw_be_p, MO_BE, helper_be_stw_mmu) -DO_ST_TLB(st1hd_be, , uint64_t, stw_be_p, MO_BE, helper_be_stw_mmu) - -DO_ST_TLB(st1ss_be, H1_4, uint32_t, stl_be_p, MO_BE, helper_be_stl_mmu) -DO_ST_TLB(st1sd_be, , uint64_t, stl_be_p, MO_BE, helper_be_stl_mmu) - -DO_ST_TLB(st1dd_be, , uint64_t, stq_be_p, MO_BE, helper_be_stq_mmu) - -#undef DO_ST_TLB - /* * Common helpers for all contiguous 1,2,3,4-register predicated stores. */ static void sve_st1_r(CPUARMState *env, void *vg, target_ulong addr, uint32_t desc, const uintptr_t ra, const int esize, const int msize, - sve_st1_tlb_fn *tlb_fn) + sve_ldst1_tlb_fn *tlb_fn) { - const TCGMemOpIdx oi =3D extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHI= FT); const unsigned rd =3D extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT= , 5); intptr_t i, oprsz =3D simd_oprsz(desc); void *vd =3D &env->vfp.zregs[rd]; @@ -4734,7 +4692,7 @@ static void sve_st1_r(CPUARMState *env, void *vg, tar= get_ulong addr, uint16_t pg =3D *(uint16_t *)(vg + H1_2(i >> 3)); do { if (pg & 1) { - tlb_fn(env, vd, i, addr, oi, ra); + tlb_fn(env, vd, i, addr, ra); } i +=3D esize, pg >>=3D esize; addr +=3D msize; @@ -4746,9 +4704,8 @@ static void sve_st1_r(CPUARMState *env, void *vg, tar= get_ulong addr, static void sve_st2_r(CPUARMState *env, void *vg, target_ulong addr, uint32_t desc, const uintptr_t ra, const int esize, const int msize, - sve_st1_tlb_fn *tlb_fn) + sve_ldst1_tlb_fn *tlb_fn) { - const TCGMemOpIdx oi =3D extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHI= FT); const unsigned rd =3D extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT= , 5); intptr_t i, oprsz =3D simd_oprsz(desc); void *d1 =3D &env->vfp.zregs[rd]; @@ -4759,8 +4716,8 @@ static void sve_st2_r(CPUARMState *env, void *vg, tar= get_ulong addr, uint16_t pg =3D *(uint16_t *)(vg + H1_2(i >> 3)); do { if (pg & 1) { - tlb_fn(env, d1, i, addr, oi, ra); - tlb_fn(env, d2, i, addr + msize, oi, ra); + tlb_fn(env, d1, i, addr, ra); + tlb_fn(env, d2, i, addr + msize, ra); } i +=3D esize, pg >>=3D esize; addr +=3D 2 * msize; @@ -4772,9 +4729,8 @@ static void sve_st2_r(CPUARMState *env, void *vg, tar= get_ulong addr, static void sve_st3_r(CPUARMState *env, void *vg, target_ulong addr, uint32_t desc, const uintptr_t ra, const int esize, const int msize, - sve_st1_tlb_fn *tlb_fn) + sve_ldst1_tlb_fn *tlb_fn) { - const TCGMemOpIdx oi =3D extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHI= FT); const unsigned rd =3D extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT= , 5); intptr_t i, oprsz =3D simd_oprsz(desc); void *d1 =3D &env->vfp.zregs[rd]; @@ -4786,9 +4742,9 @@ static void sve_st3_r(CPUARMState *env, void *vg, tar= get_ulong addr, uint16_t pg =3D *(uint16_t *)(vg + H1_2(i >> 3)); do { if (pg & 1) { - tlb_fn(env, d1, i, addr, oi, ra); - tlb_fn(env, d2, i, addr + msize, oi, ra); - tlb_fn(env, d3, i, addr + 2 * msize, oi, ra); + tlb_fn(env, d1, i, addr, ra); + tlb_fn(env, d2, i, addr + msize, ra); + tlb_fn(env, d3, i, addr + 2 * msize, ra); } i +=3D esize, pg >>=3D esize; addr +=3D 3 * msize; @@ -4800,9 +4756,8 @@ static void sve_st3_r(CPUARMState *env, void *vg, tar= get_ulong addr, static void sve_st4_r(CPUARMState *env, void *vg, target_ulong addr, uint32_t desc, const uintptr_t ra, const int esize, const int msize, - sve_st1_tlb_fn *tlb_fn) + sve_ldst1_tlb_fn *tlb_fn) { - const TCGMemOpIdx oi =3D extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHI= FT); const unsigned rd =3D extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT= , 5); intptr_t i, oprsz =3D simd_oprsz(desc); void *d1 =3D &env->vfp.zregs[rd]; @@ -4815,10 +4770,10 @@ static void sve_st4_r(CPUARMState *env, void *vg, t= arget_ulong addr, uint16_t pg =3D *(uint16_t *)(vg + H1_2(i >> 3)); do { if (pg & 1) { - tlb_fn(env, d1, i, addr, oi, ra); - tlb_fn(env, d2, i, addr + msize, oi, ra); - tlb_fn(env, d3, i, addr + 2 * msize, oi, ra); - tlb_fn(env, d4, i, addr + 3 * msize, oi, ra); + tlb_fn(env, d1, i, addr, ra); + tlb_fn(env, d2, i, addr + msize, ra); + tlb_fn(env, d3, i, addr + 2 * msize, ra); + tlb_fn(env, d4, i, addr + 3 * msize, ra); } i +=3D esize, pg >>=3D esize; addr +=3D 4 * msize; @@ -4914,9 +4869,8 @@ static target_ulong off_zd_d(void *reg, intptr_t reg_= ofs) =20 static void sve_ld1_zs(CPUARMState *env, void *vd, void *vg, void *vm, target_ulong base, uint32_t desc, uintptr_t ra, - zreg_off_fn *off_fn, sve_ld1_tlb_fn *tlb_fn) + zreg_off_fn *off_fn, sve_ldst1_tlb_fn *tlb_fn) { - const TCGMemOpIdx oi =3D extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHI= FT); const int scale =3D extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, = 2); intptr_t i, oprsz =3D simd_oprsz(desc); ARMVectorReg scratch =3D { }; @@ -4927,7 +4881,7 @@ static void sve_ld1_zs(CPUARMState *env, void *vd, vo= id *vg, void *vm, do { if (likely(pg & 1)) { target_ulong off =3D off_fn(vm, i); - tlb_fn(env, &scratch, i, base + (off << scale), oi, ra); + tlb_fn(env, &scratch, i, base + (off << scale), ra); } i +=3D 4, pg >>=3D 4; } while (i & 15); @@ -4940,9 +4894,8 @@ static void sve_ld1_zs(CPUARMState *env, void *vd, vo= id *vg, void *vm, =20 static void sve_ld1_zd(CPUARMState *env, void *vd, void *vg, void *vm, target_ulong base, uint32_t desc, uintptr_t ra, - zreg_off_fn *off_fn, sve_ld1_tlb_fn *tlb_fn) + zreg_off_fn *off_fn, sve_ldst1_tlb_fn *tlb_fn) { - const TCGMemOpIdx oi =3D extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHI= FT); const int scale =3D extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, = 2); intptr_t i, oprsz =3D simd_oprsz(desc) / 8; ARMVectorReg scratch =3D { }; @@ -4952,7 +4905,7 @@ static void sve_ld1_zd(CPUARMState *env, void *vd, vo= id *vg, void *vm, uint8_t pg =3D *(uint8_t *)(vg + H1(i)); if (likely(pg & 1)) { target_ulong off =3D off_fn(vm, i * 8); - tlb_fn(env, &scratch, i * 8, base + (off << scale), oi, ra); + tlb_fn(env, &scratch, i * 8, base + (off << scale), ra); } } clear_helper_retaddr(); @@ -5114,7 +5067,7 @@ DO_LD_NF(dd_be, , uint64_t, uint64_t, ldq_be_p) */ static inline void sve_ldff1_zs(CPUARMState *env, void *vd, void *vg, void= *vm, target_ulong base, uint32_t desc, uintptr_= t ra, - zreg_off_fn *off_fn, sve_ld1_tlb_fn *tlb_f= n, + zreg_off_fn *off_fn, sve_ldst1_tlb_fn *tlb= _fn, sve_ld1_nf_fn *nonfault_fn) { const TCGMemOpIdx oi =3D extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHI= FT); @@ -5130,7 +5083,7 @@ static inline void sve_ldff1_zs(CPUARMState *env, voi= d *vd, void *vg, void *vm, set_helper_retaddr(ra); addr =3D off_fn(vm, reg_off); addr =3D base + (addr << scale); - tlb_fn(env, vd, reg_off, addr, oi, ra); + tlb_fn(env, vd, reg_off, addr, ra); =20 /* The rest of the reads will be non-faulting. */ clear_helper_retaddr(); @@ -5156,7 +5109,7 @@ static inline void sve_ldff1_zs(CPUARMState *env, voi= d *vd, void *vg, void *vm, =20 static inline void sve_ldff1_zd(CPUARMState *env, void *vd, void *vg, void= *vm, target_ulong base, uint32_t desc, uintptr_= t ra, - zreg_off_fn *off_fn, sve_ld1_tlb_fn *tlb_f= n, + zreg_off_fn *off_fn, sve_ldst1_tlb_fn *tlb= _fn, sve_ld1_nf_fn *nonfault_fn) { const TCGMemOpIdx oi =3D extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHI= FT); @@ -5172,7 +5125,7 @@ static inline void sve_ldff1_zd(CPUARMState *env, voi= d *vd, void *vg, void *vm, set_helper_retaddr(ra); addr =3D off_fn(vm, reg_off); addr =3D base + (addr << scale); - tlb_fn(env, vd, reg_off, addr, oi, ra); + tlb_fn(env, vd, reg_off, addr, ra); =20 /* The rest of the reads will be non-faulting. */ clear_helper_retaddr(); @@ -5282,9 +5235,8 @@ DO_LDFF1_ZPZ_D(dd_be, zd) =20 static void sve_st1_zs(CPUARMState *env, void *vd, void *vg, void *vm, target_ulong base, uint32_t desc, uintptr_t ra, - zreg_off_fn *off_fn, sve_ld1_tlb_fn *tlb_fn) + zreg_off_fn *off_fn, sve_ldst1_tlb_fn *tlb_fn) { - const TCGMemOpIdx oi =3D extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHI= FT); const int scale =3D extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, = 2); intptr_t i, oprsz =3D simd_oprsz(desc); =20 @@ -5294,7 +5246,7 @@ static void sve_st1_zs(CPUARMState *env, void *vd, vo= id *vg, void *vm, do { if (likely(pg & 1)) { target_ulong off =3D off_fn(vm, i); - tlb_fn(env, vd, i, base + (off << scale), oi, ra); + tlb_fn(env, vd, i, base + (off << scale), ra); } i +=3D 4, pg >>=3D 4; } while (i & 15); @@ -5304,9 +5256,8 @@ static void sve_st1_zs(CPUARMState *env, void *vd, vo= id *vg, void *vm, =20 static void sve_st1_zd(CPUARMState *env, void *vd, void *vg, void *vm, target_ulong base, uint32_t desc, uintptr_t ra, - zreg_off_fn *off_fn, sve_ld1_tlb_fn *tlb_fn) + zreg_off_fn *off_fn, sve_ldst1_tlb_fn *tlb_fn) { - const TCGMemOpIdx oi =3D extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHI= FT); const int scale =3D extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, = 2); intptr_t i, oprsz =3D simd_oprsz(desc) / 8; =20 @@ -5315,7 +5266,7 @@ static void sve_st1_zd(CPUARMState *env, void *vd, vo= id *vg, void *vm, uint8_t pg =3D *(uint8_t *)(vg + H1(i)); if (likely(pg & 1)) { target_ulong off =3D off_fn(vm, i * 8); - tlb_fn(env, vd, i * 8, base + (off << scale), oi, ra); + tlb_fn(env, vd, i * 8, base + (off << scale), ra); } } clear_helper_retaddr(); --=20 2.20.1 From nobody Sun May 19 14:32:52 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=linaro.org ARC-Seal: i=1; a=rsa-sha256; t=1588264898; cv=none; d=zohomail.com; s=zohoarc; b=KxuOs+npU/gwe6Xot8YPjcvY5T0NAYUHLAUTVy6nNuYG8zN6KP7xA+qMB8cysQ+CTQSecb8wORTPcHJFoZw+GmDiiqHdkjISAyFed8HB3FNV9qAaD8uA9SNwFYcQVMpIx7TYe8f8iBg//Is0QYV3Q0mKTPlGRbct5IHzJA1uF+k= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1588264898; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=fF+5wwLOen+npTtt/gQ1qVYGPHz8onoyiX2hDw1RdH0=; b=NzjspDo3eLq55tsefqDy757y6wuf9BSdHJzEfNBE2O44nYuQh8h1gpex20v0UBdzWdSeDRRfgLKUEmOtKqjwpWoxq2u/20LupaIxc9b4Us5Mn7Q0qgwbxLR8Qv1f1Qox2E9B4hkUp50HP8HMlPvJm54k19f9yplTMggYXC+IXN4= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1588264898666317.21899883858055; Thu, 30 Apr 2020 09:41:38 -0700 (PDT) Received: from localhost ([::1]:39630 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jUCFl-0003uL-As for importer@patchew.org; Thu, 30 Apr 2020 12:41:37 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:49346) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jUC34-0001lJ-Dg for qemu-devel@nongnu.org; Thu, 30 Apr 2020 12:28:37 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.90_1) (envelope-from ) id 1jUC31-0006PF-Se for qemu-devel@nongnu.org; Thu, 30 Apr 2020 12:28:30 -0400 Received: from mail-pl1-x644.google.com ([2607:f8b0:4864:20::644]:39676) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jUC31-0006NO-D3 for qemu-devel@nongnu.org; Thu, 30 Apr 2020 12:28:27 -0400 Received: by mail-pl1-x644.google.com with SMTP id s20so2423907plp.6 for ; Thu, 30 Apr 2020 09:28:26 -0700 (PDT) Received: from localhost.localdomain (174-21-149-226.tukw.qwest.net. [174.21.149.226]) by smtp.gmail.com with ESMTPSA id j5sm243514pfh.58.2020.04.30.09.28.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Apr 2020 09:28:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=fF+5wwLOen+npTtt/gQ1qVYGPHz8onoyiX2hDw1RdH0=; b=ryO0Z/9VTEorOTI4tehYyq7r8ZaTSrQ2w3uWPQT45P5LIXmk5YgmteFesGzOpBjQk3 cgVQitmw76/phs4F2d7g0zTfDuXs9vlXtbxQTIG4aMQXG4C4b0l5i770/PhbDG8ElSrK OePxduvncQOpT6cWZbsRp2A2eyDK/7Tsd2pGAS6/cvfhylNCtaVVvuYMu9p6L+iW9a4N MiJKzVKPLuaC6/N1cacmRZjPKBo5C4XSoatnCMpZPHiT+DejkcZK8vDx5mcejK+wNa8a obD7GUwS/XGv4gDPQ2PSfqpuALmA4RVP89cm6SJ0lwWhxRPTIBsUjrKbSRv0oW37AdWg stzw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=fF+5wwLOen+npTtt/gQ1qVYGPHz8onoyiX2hDw1RdH0=; b=YQP846ktlhUBlmKXD/PqdKpQvRYYEGp3noAKT0DC9kt7BC26olVLGF3OVSMR7XARCG 0UOW5w8ImSCa7Cbm+iJHeAr5PKxSMo29X9k99ofGWXPZulGQh/QymeSeT3qJSfUS3LJR NydGH3+NHb6faNTEpFDqujHF9B6e/4OjRRtCYWDNLF2boPBKH8GVFPALM1QCZiGBnCUo bdUN9P8CL21BXr402d69XGU3nJpPXh14wU8pvI6sByi8sl+siCk7XjXdw63aqJCpLJEu VVo8ip99JipCYJjJmHdfwxdSzOA4DbqP91BzXsJDYhxxdn0I1C02Yp6j8fWAUxFCcFNp xyAQ== X-Gm-Message-State: AGi0PuZ+3J3YKCOfN8+3a6JfRUF2+y6/cKDMv4AKbD4GDukmAM3kKzUJ RKlm1eLY2CmSUfe65uSdhMvSGfHpGCE= X-Google-Smtp-Source: APiQypImbVgNefblPbyGGJw14HzCaax2vPP3WrjcL+JPlgLA/E00voMOUe8bnSroSsOcmcha5+K1mw== X-Received: by 2002:a17:902:b7cc:: with SMTP id v12mr4478544plz.320.1588264105412; Thu, 30 Apr 2020 09:28:25 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH v4 07/18] target/arm: Drop manual handling of set/clear_helper_retaddr Date: Thu, 30 Apr 2020 09:28:02 -0700 Message-Id: <20200430162813.17671-8-richard.henderson@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200430162813.17671-1-richard.henderson@linaro.org> References: <20200430162813.17671-1-richard.henderson@linaro.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2607:f8b0:4864:20::644; envelope-from=richard.henderson@linaro.org; helo=mail-pl1-x644.google.com X-detected-operating-system: by eggs.gnu.org: Error: [-] PROGRAM ABORT : Malformed IPv6 address (bad octet value). Location : parse_addr6(), p0f-client.c:67 X-Received-From: 2607:f8b0:4864:20::644 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, alex.bennee@linaro.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @linaro.org) Content-Type: text/plain; charset="utf-8" Since we converted back to cpu_*_data_ra, we do not need to do this ourselves. Reviewed-by: Peter Maydell Signed-off-by: Richard Henderson --- target/arm/sve_helper.c | 38 -------------------------------------- 1 file changed, 38 deletions(-) diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index 655bc9476f..aad2c8c237 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -4133,12 +4133,6 @@ static intptr_t max_for_page(target_ulong base, intp= tr_t mem_off, return MIN(split, mem_max - mem_off) + mem_off; } =20 -#ifndef CONFIG_USER_ONLY -/* These are normally defined only for CONFIG_USER_ONLY in */ -static inline void set_helper_retaddr(uintptr_t ra) { } -static inline void clear_helper_retaddr(void) { } -#endif - /* * The result of tlb_vaddr_to_host for user-only is just g2h(x), * which is always non-null. Elide the useless test. @@ -4180,7 +4174,6 @@ static void sve_ld1_r(CPUARMState *env, void *vg, con= st target_ulong addr, return; } mem_off =3D reg_off >> diffsz; - set_helper_retaddr(retaddr); =20 /* * If the (remaining) load is entirely within a single page, then: @@ -4195,7 +4188,6 @@ static void sve_ld1_r(CPUARMState *env, void *vg, con= st target_ulong addr, if (test_host_page(host)) { mem_off =3D host_fn(vd, vg, host - mem_off, mem_off, mem_max); tcg_debug_assert(mem_off =3D=3D mem_max); - clear_helper_retaddr(); /* After having taken any fault, zero leading inactive element= s. */ swap_memzero(vd, reg_off); return; @@ -4246,7 +4238,6 @@ static void sve_ld1_r(CPUARMState *env, void *vg, con= st target_ulong addr, } #endif =20 - clear_helper_retaddr(); memcpy(vd, &scratch, reg_max); } =20 @@ -4306,7 +4297,6 @@ static void sve_ld2_r(CPUARMState *env, void *vg, tar= get_ulong addr, intptr_t i, oprsz =3D simd_oprsz(desc); ARMVectorReg scratch[2] =3D { }; =20 - set_helper_retaddr(ra); for (i =3D 0; i < oprsz; ) { uint16_t pg =3D *(uint16_t *)(vg + H1_2(i >> 3)); do { @@ -4318,7 +4308,6 @@ static void sve_ld2_r(CPUARMState *env, void *vg, tar= get_ulong addr, addr +=3D 2 * size; } while (i & 15); } - clear_helper_retaddr(); =20 /* Wait until all exceptions have been raised to write back. */ memcpy(&env->vfp.zregs[rd], &scratch[0], oprsz); @@ -4333,7 +4322,6 @@ static void sve_ld3_r(CPUARMState *env, void *vg, tar= get_ulong addr, intptr_t i, oprsz =3D simd_oprsz(desc); ARMVectorReg scratch[3] =3D { }; =20 - set_helper_retaddr(ra); for (i =3D 0; i < oprsz; ) { uint16_t pg =3D *(uint16_t *)(vg + H1_2(i >> 3)); do { @@ -4346,7 +4334,6 @@ static void sve_ld3_r(CPUARMState *env, void *vg, tar= get_ulong addr, addr +=3D 3 * size; } while (i & 15); } - clear_helper_retaddr(); =20 /* Wait until all exceptions have been raised to write back. */ memcpy(&env->vfp.zregs[rd], &scratch[0], oprsz); @@ -4362,7 +4349,6 @@ static void sve_ld4_r(CPUARMState *env, void *vg, tar= get_ulong addr, intptr_t i, oprsz =3D simd_oprsz(desc); ARMVectorReg scratch[4] =3D { }; =20 - set_helper_retaddr(ra); for (i =3D 0; i < oprsz; ) { uint16_t pg =3D *(uint16_t *)(vg + H1_2(i >> 3)); do { @@ -4376,7 +4362,6 @@ static void sve_ld4_r(CPUARMState *env, void *vg, tar= get_ulong addr, addr +=3D 4 * size; } while (i & 15); } - clear_helper_retaddr(); =20 /* Wait until all exceptions have been raised to write back. */ memcpy(&env->vfp.zregs[rd], &scratch[0], oprsz); @@ -4483,7 +4468,6 @@ static void sve_ldff1_r(CPUARMState *env, void *vg, c= onst target_ulong addr, return; } mem_off =3D reg_off >> diffsz; - set_helper_retaddr(retaddr); =20 /* * If the (remaining) load is entirely within a single page, then: @@ -4498,7 +4482,6 @@ static void sve_ldff1_r(CPUARMState *env, void *vg, c= onst target_ulong addr, if (test_host_page(host)) { mem_off =3D host_fn(vd, vg, host - mem_off, mem_off, mem_max); tcg_debug_assert(mem_off =3D=3D mem_max); - clear_helper_retaddr(); /* After any fault, zero any leading inactive elements. */ swap_memzero(vd, reg_off); return; @@ -4541,7 +4524,6 @@ static void sve_ldff1_r(CPUARMState *env, void *vg, c= onst target_ulong addr, } #endif =20 - clear_helper_retaddr(); record_fault(env, reg_off, reg_max); } =20 @@ -4687,7 +4669,6 @@ static void sve_st1_r(CPUARMState *env, void *vg, tar= get_ulong addr, intptr_t i, oprsz =3D simd_oprsz(desc); void *vd =3D &env->vfp.zregs[rd]; =20 - set_helper_retaddr(ra); for (i =3D 0; i < oprsz; ) { uint16_t pg =3D *(uint16_t *)(vg + H1_2(i >> 3)); do { @@ -4698,7 +4679,6 @@ static void sve_st1_r(CPUARMState *env, void *vg, tar= get_ulong addr, addr +=3D msize; } while (i & 15); } - clear_helper_retaddr(); } =20 static void sve_st2_r(CPUARMState *env, void *vg, target_ulong addr, @@ -4711,7 +4691,6 @@ static void sve_st2_r(CPUARMState *env, void *vg, tar= get_ulong addr, void *d1 =3D &env->vfp.zregs[rd]; void *d2 =3D &env->vfp.zregs[(rd + 1) & 31]; =20 - set_helper_retaddr(ra); for (i =3D 0; i < oprsz; ) { uint16_t pg =3D *(uint16_t *)(vg + H1_2(i >> 3)); do { @@ -4723,7 +4702,6 @@ static void sve_st2_r(CPUARMState *env, void *vg, tar= get_ulong addr, addr +=3D 2 * msize; } while (i & 15); } - clear_helper_retaddr(); } =20 static void sve_st3_r(CPUARMState *env, void *vg, target_ulong addr, @@ -4737,7 +4715,6 @@ static void sve_st3_r(CPUARMState *env, void *vg, tar= get_ulong addr, void *d2 =3D &env->vfp.zregs[(rd + 1) & 31]; void *d3 =3D &env->vfp.zregs[(rd + 2) & 31]; =20 - set_helper_retaddr(ra); for (i =3D 0; i < oprsz; ) { uint16_t pg =3D *(uint16_t *)(vg + H1_2(i >> 3)); do { @@ -4750,7 +4727,6 @@ static void sve_st3_r(CPUARMState *env, void *vg, tar= get_ulong addr, addr +=3D 3 * msize; } while (i & 15); } - clear_helper_retaddr(); } =20 static void sve_st4_r(CPUARMState *env, void *vg, target_ulong addr, @@ -4765,7 +4741,6 @@ static void sve_st4_r(CPUARMState *env, void *vg, tar= get_ulong addr, void *d3 =3D &env->vfp.zregs[(rd + 2) & 31]; void *d4 =3D &env->vfp.zregs[(rd + 3) & 31]; =20 - set_helper_retaddr(ra); for (i =3D 0; i < oprsz; ) { uint16_t pg =3D *(uint16_t *)(vg + H1_2(i >> 3)); do { @@ -4779,7 +4754,6 @@ static void sve_st4_r(CPUARMState *env, void *vg, tar= get_ulong addr, addr +=3D 4 * msize; } while (i & 15); } - clear_helper_retaddr(); } =20 #define DO_STN_1(N, NAME, ESIZE) \ @@ -4875,7 +4849,6 @@ static void sve_ld1_zs(CPUARMState *env, void *vd, vo= id *vg, void *vm, intptr_t i, oprsz =3D simd_oprsz(desc); ARMVectorReg scratch =3D { }; =20 - set_helper_retaddr(ra); for (i =3D 0; i < oprsz; ) { uint16_t pg =3D *(uint16_t *)(vg + H1_2(i >> 3)); do { @@ -4886,7 +4859,6 @@ static void sve_ld1_zs(CPUARMState *env, void *vd, vo= id *vg, void *vm, i +=3D 4, pg >>=3D 4; } while (i & 15); } - clear_helper_retaddr(); =20 /* Wait until all exceptions have been raised to write back. */ memcpy(vd, &scratch, oprsz); @@ -4900,7 +4872,6 @@ static void sve_ld1_zd(CPUARMState *env, void *vd, vo= id *vg, void *vm, intptr_t i, oprsz =3D simd_oprsz(desc) / 8; ARMVectorReg scratch =3D { }; =20 - set_helper_retaddr(ra); for (i =3D 0; i < oprsz; i++) { uint8_t pg =3D *(uint8_t *)(vg + H1(i)); if (likely(pg & 1)) { @@ -4908,7 +4879,6 @@ static void sve_ld1_zd(CPUARMState *env, void *vd, vo= id *vg, void *vm, tlb_fn(env, &scratch, i * 8, base + (off << scale), ra); } } - clear_helper_retaddr(); =20 /* Wait until all exceptions have been raised to write back. */ memcpy(vd, &scratch, oprsz * 8); @@ -5080,13 +5050,11 @@ static inline void sve_ldff1_zs(CPUARMState *env, v= oid *vd, void *vg, void *vm, reg_off =3D find_next_active(vg, 0, reg_max, MO_32); if (likely(reg_off < reg_max)) { /* Perform one normal read, which will fault or not. */ - set_helper_retaddr(ra); addr =3D off_fn(vm, reg_off); addr =3D base + (addr << scale); tlb_fn(env, vd, reg_off, addr, ra); =20 /* The rest of the reads will be non-faulting. */ - clear_helper_retaddr(); } =20 /* After any fault, zero the leading predicated false elements. */ @@ -5122,13 +5090,11 @@ static inline void sve_ldff1_zd(CPUARMState *env, v= oid *vd, void *vg, void *vm, reg_off =3D find_next_active(vg, 0, reg_max, MO_64); if (likely(reg_off < reg_max)) { /* Perform one normal read, which will fault or not. */ - set_helper_retaddr(ra); addr =3D off_fn(vm, reg_off); addr =3D base + (addr << scale); tlb_fn(env, vd, reg_off, addr, ra); =20 /* The rest of the reads will be non-faulting. */ - clear_helper_retaddr(); } =20 /* After any fault, zero the leading predicated false elements. */ @@ -5240,7 +5206,6 @@ static void sve_st1_zs(CPUARMState *env, void *vd, vo= id *vg, void *vm, const int scale =3D extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, = 2); intptr_t i, oprsz =3D simd_oprsz(desc); =20 - set_helper_retaddr(ra); for (i =3D 0; i < oprsz; ) { uint16_t pg =3D *(uint16_t *)(vg + H1_2(i >> 3)); do { @@ -5251,7 +5216,6 @@ static void sve_st1_zs(CPUARMState *env, void *vd, vo= id *vg, void *vm, i +=3D 4, pg >>=3D 4; } while (i & 15); } - clear_helper_retaddr(); } =20 static void sve_st1_zd(CPUARMState *env, void *vd, void *vg, void *vm, @@ -5261,7 +5225,6 @@ static void sve_st1_zd(CPUARMState *env, void *vd, vo= id *vg, void *vm, const int scale =3D extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, = 2); intptr_t i, oprsz =3D simd_oprsz(desc) / 8; =20 - set_helper_retaddr(ra); for (i =3D 0; i < oprsz; i++) { uint8_t pg =3D *(uint8_t *)(vg + H1(i)); if (likely(pg & 1)) { @@ -5269,7 +5232,6 @@ static void sve_st1_zd(CPUARMState *env, void *vd, vo= id *vg, void *vm, tlb_fn(env, vd, i * 8, base + (off << scale), ra); } } - clear_helper_retaddr(); } =20 #define DO_ST1_ZPZ_S(MEM, OFS) \ --=20 2.20.1 From nobody Sun May 19 14:32:52 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=linaro.org ARC-Seal: i=1; a=rsa-sha256; t=1588264853; cv=none; d=zohomail.com; s=zohoarc; b=BmGo0kf7UC4/cmhqf6s3JQaBIVwP7O+dlEoVdjReqS2+4d3jCGorfkG0pSsHTrQ2tYEt7nhbH0QhxrS3p4nJUHbYQGvfA7dZ7EX0P30HPrOhqq42WCeiUWkeEK//TdInPKN81SZzvLUSn045ratJMCfojo46el/srD7fJdatq2M= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1588264853; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=ITiSi2RLqhYUiFxJMkeBvHtUt4UbgRa64Bukhjjf9rQ=; b=ExDOSDEw2CURgC8HOCvFTpeUgP67d3+d3Wjv6Db5WwNsoKUflnVfXPSjVPYXV8NjthCfVA4yPoJgzwDJ4vDI2KNhNlDwsLJfgKHbP0GtaC3E345jNZEyNgrZRp4wwsiBFlACljhnF7h4BGQQPLAN1qPBQMsHo5XjZHo7sH9iblQ= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1588264853179680.742859260128; Thu, 30 Apr 2020 09:40:53 -0700 (PDT) Received: from localhost ([::1]:36940 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jUCF1-0002ZW-OI for importer@patchew.org; Thu, 30 Apr 2020 12:40:51 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:49354) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jUC36-0001p1-F7 for qemu-devel@nongnu.org; Thu, 30 Apr 2020 12:28:37 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.90_1) (envelope-from ) id 1jUC33-0006Tm-GL for qemu-devel@nongnu.org; Thu, 30 Apr 2020 12:28:32 -0400 Received: from mail-pf1-x441.google.com ([2607:f8b0:4864:20::441]:38284) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jUC32-0006RK-WD for qemu-devel@nongnu.org; Thu, 30 Apr 2020 12:28:29 -0400 Received: by mail-pf1-x441.google.com with SMTP id y25so35358pfn.5 for ; Thu, 30 Apr 2020 09:28:28 -0700 (PDT) Received: from localhost.localdomain (174-21-149-226.tukw.qwest.net. [174.21.149.226]) by smtp.gmail.com with ESMTPSA id j5sm243514pfh.58.2020.04.30.09.28.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Apr 2020 09:28:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ITiSi2RLqhYUiFxJMkeBvHtUt4UbgRa64Bukhjjf9rQ=; b=oUQ7OYQVr9h4DuIc26x6QTqDM8AXAg7CQT/qY/Xc850OcxrA22pUGqDfbZRkZwNLXK EYnQ+fHQQ+wvBmRB8UBDL3zovey0xO34+m3GiybhgPK34tDDHD2v4yUIuAkJ7qko42D2 3VZt1G2/vxr4uPmZ1Ezy2M5enuPCwY0mgcICHkOEHkLcWIMuhcQqrThboWudzX2E1J4r kD+kfNTwK8SrkdYCdOkXQ0bAH/+BDtPKx5j1UBI12bCOqnlT4IQ9enefGlf7FhPiVxgY nvmrwL/l7OUvbbTKoKcy0rrF9r2a8tf7Rmc6s2s3bdlwr2Lxam6cLLwWxRXzkdV5lMiF 3qWQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ITiSi2RLqhYUiFxJMkeBvHtUt4UbgRa64Bukhjjf9rQ=; b=d9zKzfLHLfTaxEYtqTqH/wntRUDcUVb8WeiFm/DqLSAwYGT59lFxjVuK0m/bzydUu/ LLSOzeyu1lPC4eCaqQfg5hTrbmIol16y/AreFbi3ewxhu1NsouJjqu4TyQt63QgS2u0A iw549OYLqLoOaqJYDLHTgd9gF7QpI76bHXw8Ar5uw1mJbdXZOEzsTwWcCZl56BmwDEdj a76wA3vU/ovKF1pYtf/VUfGhgcWvIUxYPGzRYxRszGvDssfLeIOAJzmNzAk4ET9sJggq NkHZyqoV2RvVh7dwBYQ9iuHeqXce/2qGlGcnUW+8BJNz0xckXlE8YRH1kEoW9ZOrx5u+ pZcw== X-Gm-Message-State: AGi0PubkiTQdOxZJOMM0o4Uh/RSCB5iwhHgLBJnUHlMPh9qgMxRsSz2t 7GWUmvOn+Q9EH8emiGSoVbECrpi6o+s= X-Google-Smtp-Source: APiQypKmFgQS3qFDqfImYPjNH3axnO/vetWUJZnI+ETAcmvS/Ps4Ms04DJuXOJBuxMotAbr2NHiDFA== X-Received: by 2002:a63:de49:: with SMTP id y9mr3829010pgi.435.1588264106859; Thu, 30 Apr 2020 09:28:26 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH v4 08/18] target/arm: Add sve infrastructure for page lookup Date: Thu, 30 Apr 2020 09:28:03 -0700 Message-Id: <20200430162813.17671-9-richard.henderson@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200430162813.17671-1-richard.henderson@linaro.org> References: <20200430162813.17671-1-richard.henderson@linaro.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2607:f8b0:4864:20::441; envelope-from=richard.henderson@linaro.org; helo=mail-pf1-x441.google.com X-detected-operating-system: by eggs.gnu.org: Error: [-] PROGRAM ABORT : Malformed IPv6 address (bad octet value). Location : parse_addr6(), p0f-client.c:67 X-Received-From: 2607:f8b0:4864:20::441 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, alex.bennee@linaro.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @linaro.org) Content-Type: text/plain; charset="utf-8" For contiguous predicated memory operations, we want to minimize the number of tlb lookups performed. We have open-coded this for sve_ld1_r, but for correctness with MTE we will need this for all of the memory operations. Create a structure that holds the bounds of active elements, and metadata for two pages. Add routines to find those active elements, lookup the pages, and run watchpoints for those pages. Temporarily mark the functions unused to avoid Werror. Reviewed-by: Peter Maydell Signed-off-by: Richard Henderson --- target/arm/sve_helper.c | 263 +++++++++++++++++++++++++++++++++++++++- 1 file changed, 261 insertions(+), 2 deletions(-) diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index aad2c8c237..2f053a9152 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -1630,7 +1630,7 @@ void HELPER(sve_cpy_z_d)(void *vd, void *vg, uint64_t= val, uint32_t desc) } } =20 -/* Big-endian hosts need to frob the byte indicies. If the copy +/* Big-endian hosts need to frob the byte indices. If the copy * happens to be 8-byte aligned, then no frobbing necessary. */ static void swap_memmove(void *vd, void *vs, size_t n) @@ -3974,7 +3974,7 @@ void HELPER(sve_fcmla_zpzzz_d)(CPUARMState *env, void= *vg, uint32_t desc) /* * Load elements into @vd, controlled by @vg, from @host + @mem_ofs. * Memory is valid through @host + @mem_max. The register element - * indicies are inferred from @mem_ofs, as modified by the types for + * indices are inferred from @mem_ofs, as modified by the types for * which the helper is built. Return the @mem_ofs of the first element * not loaded (which is @mem_max if they are all loaded). * @@ -4133,6 +4133,265 @@ static intptr_t max_for_page(target_ulong base, int= ptr_t mem_off, return MIN(split, mem_max - mem_off) + mem_off; } =20 +/* + * Resolve the guest virtual address to info->host and info->flags. + * If @nofault, return false if the page is invalid, otherwise + * exit via page fault exception. + */ + +typedef struct { + void *host; + int flags; + MemTxAttrs attrs; +} SVEHostPage; + +static bool sve_probe_page(SVEHostPage *info, bool nofault, + CPUARMState *env, target_ulong addr, + int mem_off, MMUAccessType access_type, + int mmu_idx, uintptr_t retaddr) +{ + int flags; + + addr +=3D mem_off; + flags =3D probe_access_flags(env, addr, access_type, mmu_idx, nofault, + &info->host, retaddr); + info->flags =3D flags; + + if (flags & TLB_INVALID_MASK) { + g_assert(nofault); + return false; + } + + /* Ensure that info->host[] is relative to addr, not addr + mem_off. */ + info->host -=3D mem_off; + +#ifdef CONFIG_USER_ONLY + memset(&info->attrs, 0, sizeof(info->attrs)); +#else + /* + * Find the iotlbentry for addr and return the transaction attributes. + * This *must* be present in the TLB because we just found the mapping. + */ + { + uintptr_t index =3D tlb_index(env, mmu_idx, addr); + +# ifdef CONFIG_DEBUG_TCG + CPUTLBEntry *entry =3D tlb_entry(env, mmu_idx, addr); + target_ulong comparator =3D (access_type =3D=3D MMU_DATA_LOAD + ? entry->addr_read + : tlb_addr_write(entry)); + g_assert(tlb_hit(comparator, addr)); +# endif + + CPUIOTLBEntry *iotlbentry =3D &env_tlb(env)->d[mmu_idx].iotlb[inde= x]; + info->attrs =3D iotlbentry->attrs; + } +#endif + + return true; +} + + +/* + * Analyse contiguous data, protected by a governing predicate. + */ + +typedef enum { + FAULT_NO, + FAULT_FIRST, + FAULT_ALL, +} SVEContFault; + +typedef struct { + /* + * First and last element wholly contained within the two pages. + * mem_off_first[0] and reg_off_first[0] are always set >=3D 0. + * reg_off_last[0] may be < 0 if the first element crosses pages. + * All of mem_off_first[1], reg_off_first[1] and reg_off_last[1] + * are set >=3D 0 only if there are complete elements on a second page. + * + * The reg_off_* offsets are relative to the internal vector register. + * The mem_off_first offset is relative to the memory address; the + * two offsets are different when a load operation extends, a store + * operation truncates, or for multi-register operations. + */ + int16_t mem_off_first[2]; + int16_t reg_off_first[2]; + int16_t reg_off_last[2]; + + /* + * One element that is misaligned and spans both pages, + * or -1 if there is no such active element. + */ + int16_t mem_off_split; + int16_t reg_off_split; + + /* + * The byte offset at which the entire operation crosses a page bounda= ry. + * Set >=3D 0 if and only if the entire operation spans two pages. + */ + int16_t page_split; + + /* TLB data for the two pages. */ + SVEHostPage page[2]; +} SVEContLdSt; + +/* + * Find first active element on each page, and a loose bound for the + * final element on each page. Identify any single element that spans + * the page boundary. Return true if there are any active elements. + */ +static bool __attribute__((unused)) +sve_cont_ldst_elements(SVEContLdSt *info, target_ulong addr, uint64_t *vg, + intptr_t reg_max, int esz, int msize) +{ + const int esize =3D 1 << esz; + const uint64_t pg_mask =3D pred_esz_masks[esz]; + intptr_t reg_off_first =3D -1, reg_off_last =3D -1, reg_off_split; + intptr_t mem_off_last, mem_off_split; + intptr_t page_split, elt_split; + intptr_t i; + + /* Set all of the element indices to -1, and the TLB data to 0. */ + memset(info, -1, offsetof(SVEContLdSt, page)); + memset(info->page, 0, sizeof(info->page)); + + /* Gross scan over the entire predicate to find bounds. */ + i =3D 0; + do { + uint64_t pg =3D vg[i] & pg_mask; + if (pg) { + reg_off_last =3D i * 64 + 63 - clz64(pg); + if (reg_off_first < 0) { + reg_off_first =3D i * 64 + ctz64(pg); + } + } + } while (++i * 64 < reg_max); + + if (unlikely(reg_off_first < 0)) { + /* No active elements, no pages touched. */ + return false; + } + tcg_debug_assert(reg_off_last >=3D 0 && reg_off_last < reg_max); + + info->reg_off_first[0] =3D reg_off_first; + info->mem_off_first[0] =3D (reg_off_first >> esz) * msize; + mem_off_last =3D (reg_off_last >> esz) * msize; + + page_split =3D -(addr | TARGET_PAGE_MASK); + if (likely(mem_off_last + msize <=3D page_split)) { + /* The entire operation fits within a single page. */ + info->reg_off_last[0] =3D reg_off_last; + return true; + } + + info->page_split =3D page_split; + elt_split =3D page_split / msize; + reg_off_split =3D elt_split << esz; + mem_off_split =3D elt_split * msize; + + /* + * This is the last full element on the first page, but it is not + * necessarily active. If there is no full element, i.e. the first + * active element is the one that's split, this value remains -1. + * It is useful as iteration bounds. + */ + if (elt_split !=3D 0) { + info->reg_off_last[0] =3D reg_off_split - esize; + } + + /* Determine if an unaligned element spans the pages. */ + if (page_split % msize !=3D 0) { + /* It is helpful to know if the split element is active. */ + if ((vg[reg_off_split >> 6] >> (reg_off_split & 63)) & 1) { + info->reg_off_split =3D reg_off_split; + info->mem_off_split =3D mem_off_split; + + if (reg_off_split =3D=3D reg_off_last) { + /* The page crossing element is last. */ + return true; + } + } + reg_off_split +=3D esize; + mem_off_split +=3D msize; + } + + /* + * We do want the first active element on the second page, because + * this may affect the address reported in an exception. + */ + reg_off_split =3D find_next_active(vg, reg_off_split, reg_max, esz); + tcg_debug_assert(reg_off_split <=3D reg_off_last); + info->reg_off_first[1] =3D reg_off_split; + info->mem_off_first[1] =3D (reg_off_split >> esz) * msize; + info->reg_off_last[1] =3D reg_off_last; + return true; +} + +/* + * Resolve the guest virtual addresses to info->page[]. + * Control the generation of page faults with @fault. Return false if + * there is no work to do, which can only happen with @fault =3D=3D FAULT_= NO. + */ +static bool __attribute__((unused)) +sve_cont_ldst_pages(SVEContLdSt *info, SVEContFault fault, CPUARMState *en= v, + target_ulong addr, MMUAccessType access_type, + uintptr_t retaddr) +{ + int mmu_idx =3D cpu_mmu_index(env, false); + int mem_off =3D info->mem_off_first[0]; + bool nofault =3D fault =3D=3D FAULT_NO; + bool have_work =3D true; + + if (!sve_probe_page(&info->page[0], nofault, env, addr, mem_off, + access_type, mmu_idx, retaddr)) { + /* No work to be done. */ + return false; + } + + if (likely(info->page_split < 0)) { + /* The entire operation was on the one page. */ + return true; + } + + /* + * If the second page is invalid, then we want the fault address to be + * the first byte on that page which is accessed. + */ + if (info->mem_off_split >=3D 0) { + /* + * There is an element split across the pages. The fault address + * should be the first byte of the second page. + */ + mem_off =3D info->page_split; + /* + * If the split element is also the first active element + * of the vector, then: For first-fault we should continue + * to generate faults for the second page. For no-fault, + * we have work only if the second page is valid. + */ + if (info->mem_off_first[0] < info->mem_off_split) { + nofault =3D FAULT_FIRST; + have_work =3D false; + } + } else { + /* + * There is no element split across the pages. The fault address + * should be the first active element on the second page. + */ + mem_off =3D info->mem_off_first[1]; + /* + * There must have been one active element on the first page, + * so we're out of first-fault territory. + */ + nofault =3D fault !=3D FAULT_ALL; + } + + have_work |=3D sve_probe_page(&info->page[1], nofault, env, addr, mem_= off, + access_type, mmu_idx, retaddr); + return have_work; +} + /* * The result of tlb_vaddr_to_host for user-only is just g2h(x), * which is always non-null. Elide the useless test. --=20 2.20.1 From nobody Sun May 19 14:32:52 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=linaro.org ARC-Seal: i=1; a=rsa-sha256; t=1588264835; cv=none; d=zohomail.com; s=zohoarc; b=NmzFQ73xRfAmUvqxXvx1V2C9e/lNwDn+T5C+8OLlT9K38oBmErpXheNhUespxO/mPCyJhllORPm8WZX3fKmO5lnYFTPRBhoY3At3qdtP7jSRhEFf8Bjcfph/Jg/Th8lHtOjIqGTUd44xwoRW5xhwIrhEK1vVJpRtWcCmukwJBTg= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1588264835; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=suwrl9RFQOuVoDr4Q1KFh+iDEpTs3nZzUnaA8dPRNoc=; b=b8+S9v6R1qnmzS+fSkniACcQuekxijL9HdI6qQl6C+m8ETxUVJd/a7HRT5TaWMGun27mcb0gxg7wK0GoyMLCw2SVzEUJ3r1GYZqSt/oDIP3Z7gUxqDE/O33IF59wAh7wpUJWm1JvyOk1cRABI4zOUEcb0E13aQswEbhVVADIpqc= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1588264835092336.12565317865085; Thu, 30 Apr 2020 09:40:35 -0700 (PDT) Received: from localhost ([::1]:35724 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jUCEj-0001ns-O2 for importer@patchew.org; Thu, 30 Apr 2020 12:40:33 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:49370) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jUC3A-0001pl-2r for qemu-devel@nongnu.org; Thu, 30 Apr 2020 12:28:40 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.90_1) (envelope-from ) id 1jUC34-0006Xj-Bn for qemu-devel@nongnu.org; Thu, 30 Apr 2020 12:28:35 -0400 Received: from mail-pg1-x541.google.com ([2607:f8b0:4864:20::541]:38207) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jUC33-0006TW-TS for qemu-devel@nongnu.org; Thu, 30 Apr 2020 12:28:29 -0400 Received: by mail-pg1-x541.google.com with SMTP id l25so645641pgc.5 for ; Thu, 30 Apr 2020 09:28:29 -0700 (PDT) Received: from localhost.localdomain (174-21-149-226.tukw.qwest.net. [174.21.149.226]) by smtp.gmail.com with ESMTPSA id j5sm243514pfh.58.2020.04.30.09.28.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Apr 2020 09:28:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=suwrl9RFQOuVoDr4Q1KFh+iDEpTs3nZzUnaA8dPRNoc=; b=JLix+IVuTMnjIs/ulxnD9IEp6TfTl7q2sFcBbtsQ6EI4sTOScilDE5oiA/PfGnXBZl 0OYCW9aBvlRPS2EZV3hYYSIotA6VdchjVUtdMOSvDi1Lz0t1QFZy2azsBx/2EORt4Btq CVBXKhjlnxHqiuGMvNQf5lDlq+P+qo8e6ZQKuDIpD8qCpZPARvXRk3noaBZ80Fp/T346 wvHo2QGf7peF9AqKZ2AgNM2Fbc2O9B+r1zsBQRhXWsF/ZtkQbswe7R77sWiKWY0QtS6j RGMKKG7zMORZbt14FYthbfFrCym97Oqqqis2Yg8cT6jPnxZOFU3jvu2sUuk2n1Oz8E6t VdjA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=suwrl9RFQOuVoDr4Q1KFh+iDEpTs3nZzUnaA8dPRNoc=; b=a71DDDcg7KRO6vqlFtQgiFLOH4Bf+FPJYOmw3eyTIhEjKKEa+YTVjDik9XfaoHB3PZ 7LI/bgmEZnIIuEhn7JwzRfehw0F63mi2CHRxLC5CAA/Jqdp7C2Im9dKmHy7kz/Pf1fxQ Z5u0SelVnSh63MRTreeHONpidJ+G1N/x3IoUcys7pSNdYK256d0hsTG6pSpiaV9aGIP0 jUNxhkRk5Lm9A5B3dm0WWEqC2S1KS6+SCph1vn/KOKJmnY6/vhMlNa7moZX5iT9h4GMi VYFqprg/apUWeIAZrWlkdXOCMAYHZsOVncTSdP3soDhuROLpS7VdIc0klzbydyXwAuOH LIfw== X-Gm-Message-State: AGi0PuaN8bsHFudFkTqF/SLBP/gSwe2P2O08vJf2aWg/w6EKEu+nrs9z Nxc8IrebzjLk2ZcNtZz1jllNgEar1RU= X-Google-Smtp-Source: APiQypI1fybAuARIEMORrUQpBtSDe4fxXSFf1LArZ2qxE4+Yx6isEnzCRhtfOVq5JiWifxItYLjXuQ== X-Received: by 2002:a63:4665:: with SMTP id v37mr3780795pgk.297.1588264107940; Thu, 30 Apr 2020 09:28:27 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH v4 09/18] target/arm: Adjust interface of sve_ld1_host_fn Date: Thu, 30 Apr 2020 09:28:04 -0700 Message-Id: <20200430162813.17671-10-richard.henderson@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200430162813.17671-1-richard.henderson@linaro.org> References: <20200430162813.17671-1-richard.henderson@linaro.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2607:f8b0:4864:20::541; envelope-from=richard.henderson@linaro.org; helo=mail-pg1-x541.google.com X-detected-operating-system: by eggs.gnu.org: Error: [-] PROGRAM ABORT : Malformed IPv6 address (bad octet value). Location : parse_addr6(), p0f-client.c:67 X-Received-From: 2607:f8b0:4864:20::541 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, alex.bennee@linaro.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @linaro.org) Content-Type: text/plain; charset="utf-8" The current interface includes a loop; change it to load a single element. We will then be able to use the function for ld{2,3,4} where individual vector elements are not adjacent. Replace each call with the simplest possible loop over active elements. Reviewed-by: Peter Maydell Signed-off-by: Richard Henderson --- target/arm/sve_helper.c | 124 ++++++++++++++++++++-------------------- 1 file changed, 63 insertions(+), 61 deletions(-) diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index 2f053a9152..d007137735 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -3972,20 +3972,10 @@ void HELPER(sve_fcmla_zpzzz_d)(CPUARMState *env, vo= id *vg, uint32_t desc) */ =20 /* - * Load elements into @vd, controlled by @vg, from @host + @mem_ofs. - * Memory is valid through @host + @mem_max. The register element - * indices are inferred from @mem_ofs, as modified by the types for - * which the helper is built. Return the @mem_ofs of the first element - * not loaded (which is @mem_max if they are all loaded). - * - * For softmmu, we have fully validated the guest page. For user-only, - * we cannot fully validate without taking the mmap lock, but since we - * know the access is within one host page, if any access is valid they - * all must be valid. However, when @vg is all false, it may be that - * no access is valid. + * Load one element into @vd + @reg_off from @host. + * The controlling predicate is known to be true. */ -typedef intptr_t sve_ld1_host_fn(void *vd, void *vg, void *host, - intptr_t mem_ofs, intptr_t mem_max); +typedef void sve_ldst1_host_fn(void *vd, intptr_t reg_off, void *host); =20 /* * Load one element into @vd + @reg_off from (@env, @vaddr, @ra). @@ -3999,20 +3989,10 @@ typedef void sve_ldst1_tlb_fn(CPUARMState *env, voi= d *vd, intptr_t reg_off, */ =20 #define DO_LD_HOST(NAME, H, TYPEE, TYPEM, HOST) \ -static intptr_t sve_##NAME##_host(void *vd, void *vg, void *host, = \ - intptr_t mem_off, const intptr_t mem_max= ) \ -{ = \ - intptr_t reg_off =3D mem_off * (sizeof(TYPEE) / sizeof(TYPEM)); = \ - uint64_t *pg =3D vg; = \ - while (mem_off + sizeof(TYPEM) <=3D mem_max) { = \ - TYPEM val =3D 0; = \ - if (likely((pg[reg_off >> 6] >> (reg_off & 63)) & 1)) { = \ - val =3D HOST(host + mem_off); = \ - } = \ - *(TYPEE *)(vd + H(reg_off)) =3D val; = \ - mem_off +=3D sizeof(TYPEM), reg_off +=3D sizeof(TYPEE); = \ - } = \ - return mem_off; = \ +static void sve_##NAME##_host(void *vd, intptr_t reg_off, void *host) \ +{ \ + TYPEM val =3D HOST(host); \ + *(TYPEE *)(vd + H(reg_off)) =3D val; \ } =20 #define DO_LD_TLB(NAME, H, TYPEE, TYPEM, TLB) \ @@ -4411,7 +4391,7 @@ static inline bool test_host_page(void *host) static void sve_ld1_r(CPUARMState *env, void *vg, const target_ulong addr, uint32_t desc, const uintptr_t retaddr, const int esz, const int msz, - sve_ld1_host_fn *host_fn, + sve_ldst1_host_fn *host_fn, sve_ldst1_tlb_fn *tlb_fn) { const TCGMemOpIdx oi =3D extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHI= FT); @@ -4445,8 +4425,12 @@ static void sve_ld1_r(CPUARMState *env, void *vg, co= nst target_ulong addr, if (likely(split =3D=3D mem_max)) { host =3D tlb_vaddr_to_host(env, addr + mem_off, MMU_DATA_LOAD, mmu= _idx); if (test_host_page(host)) { - mem_off =3D host_fn(vd, vg, host - mem_off, mem_off, mem_max); - tcg_debug_assert(mem_off =3D=3D mem_max); + intptr_t i =3D reg_off; + host -=3D mem_off; + do { + host_fn(vd, i, host + (i >> diffsz)); + i =3D find_next_active(vg, i + (1 << esz), reg_max, esz); + } while (i < reg_max); /* After having taken any fault, zero leading inactive element= s. */ swap_memzero(vd, reg_off); return; @@ -4459,7 +4443,12 @@ static void sve_ld1_r(CPUARMState *env, void *vg, co= nst target_ulong addr, */ #ifdef CONFIG_USER_ONLY swap_memzero(&scratch, reg_off); - host_fn(&scratch, vg, g2h(addr), mem_off, mem_max); + host =3D g2h(addr); + do { + host_fn(&scratch, reg_off, host + (reg_off >> diffsz)); + reg_off +=3D 1 << esz; + reg_off =3D find_next_active(vg, reg_off, reg_max, esz); + } while (reg_off < reg_max); #else memset(&scratch, 0, reg_max); goto start; @@ -4477,9 +4466,13 @@ static void sve_ld1_r(CPUARMState *env, void *vg, co= nst target_ulong addr, host =3D tlb_vaddr_to_host(env, addr + mem_off, MMU_DATA_LOAD, mmu_idx); if (host) { - mem_off =3D host_fn(&scratch, vg, host - mem_off, - mem_off, split); - reg_off =3D mem_off << diffsz; + host -=3D mem_off; + do { + host_fn(&scratch, reg_off, host + mem_off); + reg_off +=3D 1 << esz; + reg_off =3D find_next_active(vg, reg_off, reg_max, esz= ); + mem_off =3D reg_off >> diffsz; + } while (split - mem_off >=3D (1 << msz)); continue; } } @@ -4706,7 +4699,7 @@ static void record_fault(CPUARMState *env, uintptr_t = i, uintptr_t oprsz) static void sve_ldff1_r(CPUARMState *env, void *vg, const target_ulong add= r, uint32_t desc, const uintptr_t retaddr, const int esz, const int msz, - sve_ld1_host_fn *host_fn, + sve_ldst1_host_fn *host_fn, sve_ldst1_tlb_fn *tlb_fn) { const TCGMemOpIdx oi =3D extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHI= FT); @@ -4716,7 +4709,7 @@ static void sve_ldff1_r(CPUARMState *env, void *vg, c= onst target_ulong addr, const int diffsz =3D esz - msz; const intptr_t reg_max =3D simd_oprsz(desc); const intptr_t mem_max =3D reg_max >> diffsz; - intptr_t split, reg_off, mem_off; + intptr_t split, reg_off, mem_off, i; void *host; =20 /* Skip to the first active element. */ @@ -4739,28 +4732,18 @@ static void sve_ldff1_r(CPUARMState *env, void *vg,= const target_ulong addr, if (likely(split =3D=3D mem_max)) { host =3D tlb_vaddr_to_host(env, addr + mem_off, MMU_DATA_LOAD, mmu= _idx); if (test_host_page(host)) { - mem_off =3D host_fn(vd, vg, host - mem_off, mem_off, mem_max); - tcg_debug_assert(mem_off =3D=3D mem_max); + i =3D reg_off; + host -=3D mem_off; + do { + host_fn(vd, i, host + (i >> diffsz)); + i =3D find_next_active(vg, i + (1 << esz), reg_max, esz); + } while (i < reg_max); /* After any fault, zero any leading inactive elements. */ swap_memzero(vd, reg_off); return; } } =20 -#ifdef CONFIG_USER_ONLY - /* - * The page(s) containing this first element at ADDR+MEM_OFF must - * be valid. Considering that this first element may be misaligned - * and cross a page boundary itself, take the rest of the page from - * the last byte of the element. - */ - split =3D max_for_page(addr, mem_off + (1 << msz) - 1, mem_max); - mem_off =3D host_fn(vd, vg, g2h(addr), mem_off, split); - - /* After any fault, zero any leading inactive elements. */ - swap_memzero(vd, reg_off); - reg_off =3D mem_off << diffsz; -#else /* * Perform one normal read, which will fault or not. * But it is likely to bring the page into the tlb. @@ -4777,11 +4760,15 @@ static void sve_ldff1_r(CPUARMState *env, void *vg,= const target_ulong addr, if (split >=3D (1 << msz)) { host =3D tlb_vaddr_to_host(env, addr + mem_off, MMU_DATA_LOAD, mmu= _idx); if (host) { - mem_off =3D host_fn(vd, vg, host - mem_off, mem_off, split); - reg_off =3D mem_off << diffsz; + host -=3D mem_off; + do { + host_fn(vd, reg_off, host + mem_off); + reg_off +=3D 1 << esz; + reg_off =3D find_next_active(vg, reg_off, reg_max, esz); + mem_off =3D reg_off >> diffsz; + } while (split - mem_off >=3D (1 << msz)); } } -#endif =20 record_fault(env, reg_off, reg_max); } @@ -4791,7 +4778,7 @@ static void sve_ldff1_r(CPUARMState *env, void *vg, c= onst target_ulong addr, */ static void sve_ldnf1_r(CPUARMState *env, void *vg, const target_ulong add= r, uint32_t desc, const int esz, const int msz, - sve_ld1_host_fn *host_fn) + sve_ldst1_host_fn *host_fn) { const unsigned rd =3D extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT= , 5); void *vd =3D &env->vfp.zregs[rd]; @@ -4806,7 +4793,13 @@ static void sve_ldnf1_r(CPUARMState *env, void *vg, = const target_ulong addr, host =3D tlb_vaddr_to_host(env, addr, MMU_DATA_LOAD, mmu_idx); if (likely(page_check_range(addr, mem_max, PAGE_READ) =3D=3D 0)) { /* The entire operation is valid and will not fault. */ - host_fn(vd, vg, host, 0, mem_max); + reg_off =3D 0; + do { + mem_off =3D reg_off >> diffsz; + host_fn(vd, reg_off, host + mem_off); + reg_off +=3D 1 << esz; + reg_off =3D find_next_active(vg, reg_off, reg_max, esz); + } while (reg_off < reg_max); return; } #endif @@ -4826,8 +4819,12 @@ static void sve_ldnf1_r(CPUARMState *env, void *vg, = const target_ulong addr, if (page_check_range(addr + mem_off, 1 << msz, PAGE_READ) =3D=3D 0) { /* At least one load is valid; take the rest of the page. */ split =3D max_for_page(addr, mem_off + (1 << msz) - 1, mem_max); - mem_off =3D host_fn(vd, vg, host, mem_off, split); - reg_off =3D mem_off << diffsz; + do { + host_fn(vd, reg_off, host + mem_off); + reg_off +=3D 1 << esz; + reg_off =3D find_next_active(vg, reg_off, reg_max, esz); + mem_off =3D reg_off >> diffsz; + } while (split - mem_off >=3D (1 << msz)); } #else /* @@ -4848,8 +4845,13 @@ static void sve_ldnf1_r(CPUARMState *env, void *vg, = const target_ulong addr, host =3D tlb_vaddr_to_host(env, addr + mem_off, MMU_DATA_LOAD, mmu_idx= ); split =3D max_for_page(addr, mem_off, mem_max); if (host && split >=3D (1 << msz)) { - mem_off =3D host_fn(vd, vg, host - mem_off, mem_off, split); - reg_off =3D mem_off << diffsz; + host -=3D mem_off; + do { + host_fn(vd, reg_off, host + mem_off); + reg_off +=3D 1 << esz; + reg_off =3D find_next_active(vg, reg_off, reg_max, esz); + mem_off =3D reg_off >> diffsz; + } while (split - mem_off >=3D (1 << msz)); } #endif =20 --=20 2.20.1 From nobody Sun May 19 14:32:52 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=linaro.org ARC-Seal: i=1; a=rsa-sha256; t=1588265040; cv=none; d=zohomail.com; s=zohoarc; b=Dbdq0mAaGcGXOiEicY5+V7uc4ZpgoCannr/ssK29hfjPDtS5enLAw9y3vhJbv/dcsxa50G87DgiWUv5bHzHo+j2GVRDyhHA1m8ZR20NjuK81fkQjbL+j10voKRm1rMG6udG+S1yOSCc+VieGjMbiXtxSo8Ojqkypw0fdRwLUX20= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1588265040; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=7ZJAf0ODZypJUEx/2hOu0Olpz0sJ7Rm6b/UcyBKBG+Y=; b=VyIxCsJ/SxnMyfyMNRatD1euNUHEzoSyUd6MoAVL4fzCpAN92CF93X2LoMW1l7wjoF8F2r9RvbbC+Dm/2fvlulKwdIiCI87UqysZmgMkZA17bW3f52hNDbquFzbKFMv9wim9JYsA5kDkxiytt2knOxOfVqRHTA/richS0kndD9k= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1588265040130632.8420874806707; Thu, 30 Apr 2020 09:44:00 -0700 (PDT) Received: from localhost ([::1]:49670 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jUCI2-0000x3-HY for importer@patchew.org; Thu, 30 Apr 2020 12:43:58 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:49388) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jUC3B-0001qT-KR for qemu-devel@nongnu.org; Thu, 30 Apr 2020 12:28:46 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.90_1) (envelope-from ) id 1jUC35-0006dr-KS for qemu-devel@nongnu.org; Thu, 30 Apr 2020 12:28:37 -0400 Received: from mail-pl1-x643.google.com ([2607:f8b0:4864:20::643]:41679) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jUC35-0006YI-4q for qemu-devel@nongnu.org; Thu, 30 Apr 2020 12:28:31 -0400 Received: by mail-pl1-x643.google.com with SMTP id d24so2419494pll.8 for ; Thu, 30 Apr 2020 09:28:30 -0700 (PDT) Received: from localhost.localdomain (174-21-149-226.tukw.qwest.net. [174.21.149.226]) by smtp.gmail.com with ESMTPSA id j5sm243514pfh.58.2020.04.30.09.28.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Apr 2020 09:28:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=7ZJAf0ODZypJUEx/2hOu0Olpz0sJ7Rm6b/UcyBKBG+Y=; b=ZereAmgQO3hlSgIReZWndX/4U5wg9gmXv0KJO6OjOedqGmhrWYnB3STted5JkysLvH Nks+dwZxXd/g5sy1cRBnGLf47o4WQQLJU7KdMeWBFxfWO0o7I8yLUK+tVY/W5E1D7N+a P6ETwlMTyEPXZGyPR8O3EJWCrF0O/8DsdzmRE67PaKInPd3WDnPu7BfRWV0lLVOXp50Q Hl31S322UraBlxcWt3uYcBBrjpPn/+gLa4TO89skuAt/71MT5YsrpB/yhCmACkgKTMqY Wt2CPd27i4P/hXKsbxf1jNvn9O/qB8T015jbK8oJOQWLgWiIR7WINPhkc1LmBtILKvaL X5SA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=7ZJAf0ODZypJUEx/2hOu0Olpz0sJ7Rm6b/UcyBKBG+Y=; b=GyK/u1KgKKRCzUr86NwvQjiTpi+P/uk+mpbYk3zK5BKLs10TG5o0AuHZzrT8an/kUe fy4QUTFSeLTYFeUqcUd4YddzxJ4u8hjKLyvKRzjl5eJ/pSRtBuedcbxa1HlgzdIuvg6Y w0gqs4YI3Acz0X6wT16JvGwF93j73K8seGEc2usyhrfqMWFIp6xP8Wt7jnyAjP6pOBcb Eazuj9NkuriB6O1qT4lorhbl0D/9RZnOC5Vu66txztQRCBWt2pd47TN2M34LaQaS69Vn fTgV/lVxmZycrRX+AcY2HpW8iZ9ySmQJFQ0cAsJMjuSy3MFNC0+tLYx4g+J9crTidADE A2fw== X-Gm-Message-State: AGi0PuYmnasLrtm+nhmvY114+uNRJbDVhbc/E/W+R5gS8J5imVCbWUsD bBzxLargQKBBy2MGlu8LKpgZiw59lpo= X-Google-Smtp-Source: APiQypKUsBLg6bcqc1vDh0FR6yHeza7MjeYLH8+q/wJlQrjbpVPEAh4Hhd8cyT5xFoqugW13ollYbw== X-Received: by 2002:a17:902:9882:: with SMTP id s2mr4409551plp.184.1588264109355; Thu, 30 Apr 2020 09:28:29 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH v4 10/18] target/arm: Use SVEContLdSt in sve_ld1_r Date: Thu, 30 Apr 2020 09:28:05 -0700 Message-Id: <20200430162813.17671-11-richard.henderson@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200430162813.17671-1-richard.henderson@linaro.org> References: <20200430162813.17671-1-richard.henderson@linaro.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2607:f8b0:4864:20::643; envelope-from=richard.henderson@linaro.org; helo=mail-pl1-x643.google.com X-detected-operating-system: by eggs.gnu.org: Error: [-] PROGRAM ABORT : Malformed IPv6 address (bad octet value). Location : parse_addr6(), p0f-client.c:67 X-Received-From: 2607:f8b0:4864:20::643 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, alex.bennee@linaro.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @linaro.org) Content-Type: text/plain; charset="utf-8" First use of the new helper functions, so we can remove the unused markup. No longer need a scratch for user-only, as we completely probe the page set before reading; system mode still requires a scratch for MMIO. Reviewed-by: Peter Maydell Signed-off-by: Richard Henderson --- target/arm/sve_helper.c | 188 +++++++++++++++++++++------------------- 1 file changed, 97 insertions(+), 91 deletions(-) diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index d007137735..6bae342a17 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -4221,9 +4221,9 @@ typedef struct { * final element on each page. Identify any single element that spans * the page boundary. Return true if there are any active elements. */ -static bool __attribute__((unused)) -sve_cont_ldst_elements(SVEContLdSt *info, target_ulong addr, uint64_t *vg, - intptr_t reg_max, int esz, int msize) +static bool sve_cont_ldst_elements(SVEContLdSt *info, target_ulong addr, + uint64_t *vg, intptr_t reg_max, + int esz, int msize) { const int esize =3D 1 << esz; const uint64_t pg_mask =3D pred_esz_masks[esz]; @@ -4313,10 +4313,9 @@ sve_cont_ldst_elements(SVEContLdSt *info, target_ulo= ng addr, uint64_t *vg, * Control the generation of page faults with @fault. Return false if * there is no work to do, which can only happen with @fault =3D=3D FAULT_= NO. */ -static bool __attribute__((unused)) -sve_cont_ldst_pages(SVEContLdSt *info, SVEContFault fault, CPUARMState *en= v, - target_ulong addr, MMUAccessType access_type, - uintptr_t retaddr) +static bool sve_cont_ldst_pages(SVEContLdSt *info, SVEContFault fault, + CPUARMState *env, target_ulong addr, + MMUAccessType access_type, uintptr_t retad= dr) { int mmu_idx =3D cpu_mmu_index(env, false); int mem_off =3D info->mem_off_first[0]; @@ -4388,109 +4387,116 @@ static inline bool test_host_page(void *host) /* * Common helper for all contiguous one-register predicated loads. */ -static void sve_ld1_r(CPUARMState *env, void *vg, const target_ulong addr, - uint32_t desc, const uintptr_t retaddr, - const int esz, const int msz, - sve_ldst1_host_fn *host_fn, - sve_ldst1_tlb_fn *tlb_fn) +static inline QEMU_ALWAYS_INLINE +void sve_ld1_r(CPUARMState *env, uint64_t *vg, const target_ulong addr, + uint32_t desc, const uintptr_t retaddr, + const int esz, const int msz, + sve_ldst1_host_fn *host_fn, + sve_ldst1_tlb_fn *tlb_fn) { - const TCGMemOpIdx oi =3D extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHI= FT); - const int mmu_idx =3D get_mmuidx(oi); const unsigned rd =3D extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT= , 5); void *vd =3D &env->vfp.zregs[rd]; - const int diffsz =3D esz - msz; const intptr_t reg_max =3D simd_oprsz(desc); - const intptr_t mem_max =3D reg_max >> diffsz; - ARMVectorReg scratch; + intptr_t reg_off, reg_last, mem_off; + SVEContLdSt info; void *host; - intptr_t split, reg_off, mem_off; + int flags; =20 - /* Find the first active element. */ - reg_off =3D find_next_active(vg, 0, reg_max, esz); - if (unlikely(reg_off =3D=3D reg_max)) { + /* Find the active elements. */ + if (!sve_cont_ldst_elements(&info, addr, vg, reg_max, esz, 1 << msz)) { /* The entire predicate was false; no load occurs. */ memset(vd, 0, reg_max); return; } - mem_off =3D reg_off >> diffsz; =20 - /* - * If the (remaining) load is entirely within a single page, then: - * For softmmu, and the tlb hits, then no faults will occur; - * For user-only, either the first load will fault or none will. - * We can thus perform the load directly to the destination and - * Vd will be unmodified on any exception path. - */ - split =3D max_for_page(addr, mem_off, mem_max); - if (likely(split =3D=3D mem_max)) { - host =3D tlb_vaddr_to_host(env, addr + mem_off, MMU_DATA_LOAD, mmu= _idx); - if (test_host_page(host)) { - intptr_t i =3D reg_off; - host -=3D mem_off; - do { - host_fn(vd, i, host + (i >> diffsz)); - i =3D find_next_active(vg, i + (1 << esz), reg_max, esz); - } while (i < reg_max); - /* After having taken any fault, zero leading inactive element= s. */ - swap_memzero(vd, reg_off); - return; - } - } + /* Probe the page(s). Exit with exception for any invalid page. */ + sve_cont_ldst_pages(&info, FAULT_ALL, env, addr, MMU_DATA_LOAD, retadd= r); =20 - /* - * Perform the predicated read into a temporary, thus ensuring - * if the load of the last element faults, Vd is not modified. - */ + flags =3D info.page[0].flags | info.page[1].flags; + if (unlikely(flags !=3D 0)) { #ifdef CONFIG_USER_ONLY - swap_memzero(&scratch, reg_off); - host =3D g2h(addr); - do { - host_fn(&scratch, reg_off, host + (reg_off >> diffsz)); - reg_off +=3D 1 << esz; - reg_off =3D find_next_active(vg, reg_off, reg_max, esz); - } while (reg_off < reg_max); + g_assert_not_reached(); #else - memset(&scratch, 0, reg_max); - goto start; - while (1) { - reg_off =3D find_next_active(vg, reg_off, reg_max, esz); - if (reg_off >=3D reg_max) { - break; - } - mem_off =3D reg_off >> diffsz; - split =3D max_for_page(addr, mem_off, mem_max); + /* + * At least one page includes MMIO (or watchpoints). + * Any bus operation can fail with cpu_transaction_failed, + * which for ARM will raise SyncExternal. Perform the load + * into scratch memory to preserve register state until the end. + */ + ARMVectorReg scratch; =20 - start: - if (split - mem_off >=3D (1 << msz)) { - /* At least one whole element on this page. */ - host =3D tlb_vaddr_to_host(env, addr + mem_off, - MMU_DATA_LOAD, mmu_idx); - if (host) { - host -=3D mem_off; - do { - host_fn(&scratch, reg_off, host + mem_off); - reg_off +=3D 1 << esz; - reg_off =3D find_next_active(vg, reg_off, reg_max, esz= ); - mem_off =3D reg_off >> diffsz; - } while (split - mem_off >=3D (1 << msz)); - continue; + memset(&scratch, 0, reg_max); + mem_off =3D info.mem_off_first[0]; + reg_off =3D info.reg_off_first[0]; + reg_last =3D info.reg_off_last[1]; + if (reg_last < 0) { + reg_last =3D info.reg_off_split; + if (reg_last < 0) { + reg_last =3D info.reg_off_last[0]; } } =20 - /* - * Perform one normal read. This may fault, longjmping out to the - * main loop in order to raise an exception. It may succeed, and - * as a side-effect load the TLB entry for the next round. Finall= y, - * in the extremely unlikely case we're performing this operation - * on I/O memory, it may succeed but not bring in the TLB entry. - * But even then we have still made forward progress. - */ - tlb_fn(env, &scratch, reg_off, addr + mem_off, retaddr); - reg_off +=3D 1 << esz; - } -#endif + do { + uint64_t pg =3D vg[reg_off >> 6]; + do { + if ((pg >> (reg_off & 63)) & 1) { + tlb_fn(env, &scratch, reg_off, addr + mem_off, retaddr= ); + } + reg_off +=3D 1 << esz; + mem_off +=3D 1 << msz; + } while (reg_off & 63); + } while (reg_off <=3D reg_last); =20 - memcpy(vd, &scratch, reg_max); + memcpy(vd, &scratch, reg_max); + return; +#endif + } + + /* The entire operation is in RAM, on valid pages. */ + + memset(vd, 0, reg_max); + mem_off =3D info.mem_off_first[0]; + reg_off =3D info.reg_off_first[0]; + reg_last =3D info.reg_off_last[0]; + host =3D info.page[0].host; + + while (reg_off <=3D reg_last) { + uint64_t pg =3D vg[reg_off >> 6]; + do { + if ((pg >> (reg_off & 63)) & 1) { + host_fn(vd, reg_off, host + mem_off); + } + reg_off +=3D 1 << esz; + mem_off +=3D 1 << msz; + } while (reg_off <=3D reg_last && (reg_off & 63)); + } + + /* + * Use the slow path to manage the cross-page misalignment. + * But we know this is RAM and cannot trap. + */ + mem_off =3D info.mem_off_split; + if (unlikely(mem_off >=3D 0)) { + tlb_fn(env, vd, info.reg_off_split, addr + mem_off, retaddr); + } + + mem_off =3D info.mem_off_first[1]; + if (unlikely(mem_off >=3D 0)) { + reg_off =3D info.reg_off_first[1]; + reg_last =3D info.reg_off_last[1]; + host =3D info.page[1].host; + + do { + uint64_t pg =3D vg[reg_off >> 6]; + do { + if ((pg >> (reg_off & 63)) & 1) { + host_fn(vd, reg_off, host + mem_off); + } + reg_off +=3D 1 << esz; + mem_off +=3D 1 << msz; + } while (reg_off & 63); + } while (reg_off <=3D reg_last); + } } =20 #define DO_LD1_1(NAME, ESZ) \ --=20 2.20.1 From nobody Sun May 19 14:32:52 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=linaro.org ARC-Seal: i=1; a=rsa-sha256; t=1588264953; cv=none; d=zohomail.com; s=zohoarc; b=aBaR6WefZKxR7hCr3/5/6uRpNYU6wrWUjeCHM9/NP/DaLgQyNK/hRV40IX0Q3/RJ59XnIw1IQmXg1SykC/GPrbv5ZGZUbRlIkvhwf3iphqRJAdfKSKJK/YbT2vwNtTyd/oFwNDp/6Qwr1KtDXuPCPopjfteni642yMEno87DJgw= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1588264953; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=4tSyrjuk3Om51Zcagt3t6yh4FkYKfOOT3yWLv6tJ2Vs=; b=h2KfmV0ViEWoeNcuVLbOVpx42XOJPe4ueE1pmi/uWx+5UZdP5pDFp4eGbjubcuma/lV1fsO4OB8WZFKbh7uBlvyKqLES+wqCxs4WuKF5myYZrR87IjMlrrmsgBjhf2LX1esRU/3GJd/za5tkiZJWqb4aq841CH6MN7GE8tt9oFg= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1588264953279823.0610846198274; Thu, 30 Apr 2020 09:42:33 -0700 (PDT) Received: from localhost ([::1]:43202 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jUCGd-0005eR-Uf for importer@patchew.org; Thu, 30 Apr 2020 12:42:31 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:49398) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jUC3C-0001se-Es for qemu-devel@nongnu.org; Thu, 30 Apr 2020 12:28:46 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.90_1) (envelope-from ) id 1jUC38-0006fe-CR for qemu-devel@nongnu.org; Thu, 30 Apr 2020 12:28:38 -0400 Received: from mail-pl1-x643.google.com ([2607:f8b0:4864:20::643]:37049) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jUC36-0006dy-Es for qemu-devel@nongnu.org; Thu, 30 Apr 2020 12:28:32 -0400 Received: by mail-pl1-x643.google.com with SMTP id c21so2428864plz.4 for ; Thu, 30 Apr 2020 09:28:31 -0700 (PDT) Received: from localhost.localdomain (174-21-149-226.tukw.qwest.net. [174.21.149.226]) by smtp.gmail.com with ESMTPSA id j5sm243514pfh.58.2020.04.30.09.28.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Apr 2020 09:28:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=4tSyrjuk3Om51Zcagt3t6yh4FkYKfOOT3yWLv6tJ2Vs=; b=lRytBLdHoJENE73egrty/xfy33fiJkqQRxA1qlwzv4pwf/ATDMFu2uxhF7hTQqkEv6 rToeiJGDVgWuRjFteCaHjFpqQWj4ep6MTy0iaZlKrt1woLJOSoPrg1kFU0KYH5j6BZxv O3p+V6qXVPD6MwVxQXV8iv1GIT5+4Vst/bE85XiO+qVEPZ3VPmmAnuEI/rkrO3lCFEDy sXgamO25mmnOfWvU7sFZEDFS1BTaL0CROXCU/qc8DOHCmSStSoka7Kgu5hoV27bjD7Um rKz4nB9x5zO/yk7fPF58Q3XduLfvaVEwF644bmub2fsCgpogvBnPelU1t/aoKKhxqUJu T0qw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=4tSyrjuk3Om51Zcagt3t6yh4FkYKfOOT3yWLv6tJ2Vs=; b=rg7dRGxQv0SpF1T6QK0aiWw0QrGo6TGFzJ8Td1y2udbFIAKcMynE4SfG1ed3gJq3Kh DgcanVeYRHyAOmCcqZGmFhBeVCKQ/ltsZ8kwUAFUKWmZpWrVc2UTSvXmGWTqLghqNhRO FLpog1avUumctQRJGgGJWexj4NAZ5OPIxnYezbnZHvnSOb/7alteEIDhBe7vsGO+GcwI wLTLBNOIqw3p3IG4hC+BZaNNnfg3J1XDmsJjNNUhjtlMkPJAi1JgwsqRfcqxblSa1Nc2 iSTxY/vzvPILpEDqzUIbsGIC5wCfj5HCvX+D7CP0ttjPMZGcW7VfZbxuSfhjPo6egJWx ETJw== X-Gm-Message-State: AGi0PuZgDirN8W+E6nkPRX6YKSIg5XzJivzrF3uGa5xr16hz9duJtHNH nnOlIhd46qtfnXjH1TwwCW0i/2XZ7j0= X-Google-Smtp-Source: APiQypJHuIMjRTCKLhQ6XeKpzUAMJ5YWzqXJ5y9yJ4UIGpX8wH9zy9czFYPYAmHqz9aDXrAZXtunDg== X-Received: by 2002:a17:902:9697:: with SMTP id n23mr4482672plp.150.1588264110686; Thu, 30 Apr 2020 09:28:30 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH v4 11/18] target/arm: Handle watchpoints in sve_ld1_r Date: Thu, 30 Apr 2020 09:28:06 -0700 Message-Id: <20200430162813.17671-12-richard.henderson@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200430162813.17671-1-richard.henderson@linaro.org> References: <20200430162813.17671-1-richard.henderson@linaro.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2607:f8b0:4864:20::643; envelope-from=richard.henderson@linaro.org; helo=mail-pl1-x643.google.com X-detected-operating-system: by eggs.gnu.org: Error: [-] PROGRAM ABORT : Malformed IPv6 address (bad octet value). Location : parse_addr6(), p0f-client.c:67 X-Received-From: 2607:f8b0:4864:20::643 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, alex.bennee@linaro.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @linaro.org) Content-Type: text/plain; charset="utf-8" Handle all of the watchpoints for active elements all at once, before we've modified the vector register. This removes the TLB_WATCHPOINT bit from page[].flags, which means that we can use the normal fast path via RAM. Reviewed-by: Peter Maydell Signed-off-by: Richard Henderson --- target/arm/sve_helper.c | 72 ++++++++++++++++++++++++++++++++++++++++- 1 file changed, 71 insertions(+), 1 deletion(-) diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index 6bae342a17..7992a569b0 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -4371,6 +4371,70 @@ static bool sve_cont_ldst_pages(SVEContLdSt *info, S= VEContFault fault, return have_work; } =20 +static void sve_cont_ldst_watchpoints(SVEContLdSt *info, CPUARMState *env, + uint64_t *vg, target_ulong addr, + int esize, int msize, int wp_access, + uintptr_t retaddr) +{ +#ifndef CONFIG_USER_ONLY + intptr_t mem_off, reg_off, reg_last; + int flags0 =3D info->page[0].flags; + int flags1 =3D info->page[1].flags; + + if (likely(!((flags0 | flags1) & TLB_WATCHPOINT))) { + return; + } + + /* Indicate that watchpoints are handled. */ + info->page[0].flags =3D flags0 & ~TLB_WATCHPOINT; + info->page[1].flags =3D flags1 & ~TLB_WATCHPOINT; + + if (flags0 & TLB_WATCHPOINT) { + mem_off =3D info->mem_off_first[0]; + reg_off =3D info->reg_off_first[0]; + reg_last =3D info->reg_off_last[0]; + + while (reg_off <=3D reg_last) { + uint64_t pg =3D vg[reg_off >> 6]; + do { + if ((pg >> (reg_off & 63)) & 1) { + cpu_check_watchpoint(env_cpu(env), addr + mem_off, + msize, info->page[0].attrs, + wp_access, retaddr); + } + reg_off +=3D esize; + mem_off +=3D msize; + } while (reg_off <=3D reg_last && (reg_off & 63)); + } + } + + mem_off =3D info->mem_off_split; + if (mem_off >=3D 0) { + cpu_check_watchpoint(env_cpu(env), addr + mem_off, msize, + info->page[0].attrs, wp_access, retaddr); + } + + mem_off =3D info->mem_off_first[1]; + if ((flags1 & TLB_WATCHPOINT) && mem_off >=3D 0) { + reg_off =3D info->reg_off_first[1]; + reg_last =3D info->reg_off_last[1]; + + do { + uint64_t pg =3D vg[reg_off >> 6]; + do { + if ((pg >> (reg_off & 63)) & 1) { + cpu_check_watchpoint(env_cpu(env), addr + mem_off, + msize, info->page[1].attrs, + wp_access, retaddr); + } + reg_off +=3D esize; + mem_off +=3D msize; + } while (reg_off & 63); + } while (reg_off <=3D reg_last); + } +#endif +} + /* * The result of tlb_vaddr_to_host for user-only is just g2h(x), * which is always non-null. Elide the useless test. @@ -4412,13 +4476,19 @@ void sve_ld1_r(CPUARMState *env, uint64_t *vg, cons= t target_ulong addr, /* Probe the page(s). Exit with exception for any invalid page. */ sve_cont_ldst_pages(&info, FAULT_ALL, env, addr, MMU_DATA_LOAD, retadd= r); =20 + /* Handle watchpoints for all active elements. */ + sve_cont_ldst_watchpoints(&info, env, vg, addr, 1 << esz, 1 << msz, + BP_MEM_READ, retaddr); + + /* TODO: MTE check. */ + flags =3D info.page[0].flags | info.page[1].flags; if (unlikely(flags !=3D 0)) { #ifdef CONFIG_USER_ONLY g_assert_not_reached(); #else /* - * At least one page includes MMIO (or watchpoints). + * At least one page includes MMIO. * Any bus operation can fail with cpu_transaction_failed, * which for ARM will raise SyncExternal. Perform the load * into scratch memory to preserve register state until the end. --=20 2.20.1 From nobody Sun May 19 14:32:52 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=linaro.org ARC-Seal: i=1; a=rsa-sha256; t=1588265179; cv=none; d=zohomail.com; s=zohoarc; b=LQpf3470++afrcDyBJgz7lEZD/xLaOmZz+elfmxXBBkkPf7SbOGCufPG/Fg7ZUPXAoA+k7nbHg2n6z89qmFKuukeQ94hZLTZ/eYCfcINj8jBicwPOsOY05aDnQHT65BMZvq7Y8A5ya1pdEMs3A/P85qi5ASNkJgE/IR/4bOJ8pk= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1588265179; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=9amWqshE7k/NAq9a276bjEryeqReUJxYJNXk6wnm0Gs=; b=PIlmMs+TJfAckBRSvsIx8q7cpqTdUwmIWV26xVjlKI/BwYnc3KL2bjoW6v4EQSigvTKD7YMwC8RnEO6fGKbLGmOsKLjWxp4L2xkwHDV6O0Eo4gOWlYOYbJXRGTPf2p9SePlAWDgzmQT6eTRYVu90eQeBtRo4IRLFinPGV+xJKFc= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 15882651797331003.8485339467769; Thu, 30 Apr 2020 09:46:19 -0700 (PDT) Received: from localhost ([::1]:56148 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jUCKI-0007Ru-BU for importer@patchew.org; Thu, 30 Apr 2020 12:46:18 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:49410) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jUC3E-0001tS-AT for qemu-devel@nongnu.org; Thu, 30 Apr 2020 12:28:50 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.90_1) (envelope-from ) id 1jUC38-0006fo-U9 for qemu-devel@nongnu.org; Thu, 30 Apr 2020 12:28:40 -0400 Received: from mail-pg1-x542.google.com ([2607:f8b0:4864:20::542]:45206) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jUC38-0006fU-Hb for qemu-devel@nongnu.org; Thu, 30 Apr 2020 12:28:34 -0400 Received: by mail-pg1-x542.google.com with SMTP id s18so2963521pgl.12 for ; Thu, 30 Apr 2020 09:28:34 -0700 (PDT) Received: from localhost.localdomain (174-21-149-226.tukw.qwest.net. [174.21.149.226]) by smtp.gmail.com with ESMTPSA id j5sm243514pfh.58.2020.04.30.09.28.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Apr 2020 09:28:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=9amWqshE7k/NAq9a276bjEryeqReUJxYJNXk6wnm0Gs=; b=Gobkm0WNmpMHRxfsTJPPpQe1THDWEuBE4HoqM8uMSu8xFT782Am3ciUb1rl0orFLXw tU6M9p2X9ENP9h/7RYl0Czim2z/shB+nTKeEyOsvcMSd5kvBdQdDvwypt8KnPQSAEhA4 JUex41Z/8Gu1RgSnjF1ExQ7QYmkj+vfMDpRmdCHQrHNmBwEdtQYdgdCGcHSDYc3FmEKU ti0kMyY5NyCn9Z6Ylm/Ak+wDwPW6WKyi5VZu/kExqDVc4tuqe3EIt+SlveSvbCcqtJOK htKlJuO0N5Upu8+V8YUF8UZsrzlyBJ2Gy8n8k0rFEPTl5NuV44SW5Um5BrEfpbxGHZ/M Xbcg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=9amWqshE7k/NAq9a276bjEryeqReUJxYJNXk6wnm0Gs=; b=MIx2xKplMK2zcDII7hm795KHF0wGTLQ5pd/1XaPSnQLiZ1jTh452GdWAZTThfxN304 BmqYiBnbtvWigls1HWdQyPOxpc0xZZctAj/FAW5JYKzQTUKbVNpPu1VK6xomglwmX4GF guebJ3qne9SgLAM9UjKL3VU07WqFBs3hKaVftgrRYemP2MbL8vO864NFSiBN7LcrwejF ng9dUBptBnhUtTUiMXeOT2btgyHNu0vl/+e7vf2w4dfOZcXQ3ulCE8eZFrauCP/5z/8G hSH91gIP/PlJY8Hw1hOzfpEruGvBgLQii+GVKVXO0TqFWs/K8QZIfNfvPfPO1R7kRClf FEPA== X-Gm-Message-State: AGi0PubF4Tv3vG8wWEPKVJNbAFb4bkbVln4CSHGIdLB2thFLECS04S4e +X1NR/5mUUkBk4IHR/BReewentyUvXE= X-Google-Smtp-Source: APiQypJzJ/r0qnlUhPI0BCQcD/Rg7FgF3/XzS+gOPEdbBT3BEn1Dxd83Abwz77rRflDbAy6Ghx2vMA== X-Received: by 2002:a63:6285:: with SMTP id w127mr3812816pgb.449.1588264112321; Thu, 30 Apr 2020 09:28:32 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH v4 12/18] target/arm: Use SVEContLdSt for multi-register contiguous loads Date: Thu, 30 Apr 2020 09:28:07 -0700 Message-Id: <20200430162813.17671-13-richard.henderson@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200430162813.17671-1-richard.henderson@linaro.org> References: <20200430162813.17671-1-richard.henderson@linaro.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2607:f8b0:4864:20::542; envelope-from=richard.henderson@linaro.org; helo=mail-pg1-x542.google.com X-detected-operating-system: by eggs.gnu.org: Error: [-] PROGRAM ABORT : Malformed IPv6 address (bad octet value). Location : parse_addr6(), p0f-client.c:67 X-Received-From: 2607:f8b0:4864:20::542 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, alex.bennee@linaro.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @linaro.org) Content-Type: text/plain; charset="utf-8" Reviewed-by: Peter Maydell Signed-off-by: Richard Henderson --- target/arm/sve_helper.c | 223 ++++++++++++++-------------------------- 1 file changed, 79 insertions(+), 144 deletions(-) diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index 7992a569b0..9365e32646 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -4449,27 +4449,28 @@ static inline bool test_host_page(void *host) } =20 /* - * Common helper for all contiguous one-register predicated loads. + * Common helper for all contiguous 1,2,3,4-register predicated stores. */ static inline QEMU_ALWAYS_INLINE -void sve_ld1_r(CPUARMState *env, uint64_t *vg, const target_ulong addr, +void sve_ldN_r(CPUARMState *env, uint64_t *vg, const target_ulong addr, uint32_t desc, const uintptr_t retaddr, - const int esz, const int msz, + const int esz, const int msz, const int N, sve_ldst1_host_fn *host_fn, sve_ldst1_tlb_fn *tlb_fn) { const unsigned rd =3D extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT= , 5); - void *vd =3D &env->vfp.zregs[rd]; const intptr_t reg_max =3D simd_oprsz(desc); intptr_t reg_off, reg_last, mem_off; SVEContLdSt info; void *host; - int flags; + int flags, i; =20 /* Find the active elements. */ - if (!sve_cont_ldst_elements(&info, addr, vg, reg_max, esz, 1 << msz)) { + if (!sve_cont_ldst_elements(&info, addr, vg, reg_max, esz, N << msz)) { /* The entire predicate was false; no load occurs. */ - memset(vd, 0, reg_max); + for (i =3D 0; i < N; ++i) { + memset(&env->vfp.zregs[(rd + i) & 31], 0, reg_max); + } return; } =20 @@ -4477,7 +4478,7 @@ void sve_ld1_r(CPUARMState *env, uint64_t *vg, const = target_ulong addr, sve_cont_ldst_pages(&info, FAULT_ALL, env, addr, MMU_DATA_LOAD, retadd= r); =20 /* Handle watchpoints for all active elements. */ - sve_cont_ldst_watchpoints(&info, env, vg, addr, 1 << esz, 1 << msz, + sve_cont_ldst_watchpoints(&info, env, vg, addr, 1 << esz, N << msz, BP_MEM_READ, retaddr); =20 /* TODO: MTE check. */ @@ -4493,9 +4494,8 @@ void sve_ld1_r(CPUARMState *env, uint64_t *vg, const = target_ulong addr, * which for ARM will raise SyncExternal. Perform the load * into scratch memory to preserve register state until the end. */ - ARMVectorReg scratch; + ARMVectorReg scratch[4] =3D { }; =20 - memset(&scratch, 0, reg_max); mem_off =3D info.mem_off_first[0]; reg_off =3D info.reg_off_first[0]; reg_last =3D info.reg_off_last[1]; @@ -4510,21 +4510,29 @@ void sve_ld1_r(CPUARMState *env, uint64_t *vg, cons= t target_ulong addr, uint64_t pg =3D vg[reg_off >> 6]; do { if ((pg >> (reg_off & 63)) & 1) { - tlb_fn(env, &scratch, reg_off, addr + mem_off, retaddr= ); + for (i =3D 0; i < N; ++i) { + tlb_fn(env, &scratch[i], reg_off, + addr + mem_off + (i << msz), retaddr); + } } reg_off +=3D 1 << esz; - mem_off +=3D 1 << msz; + mem_off +=3D N << msz; } while (reg_off & 63); } while (reg_off <=3D reg_last); =20 - memcpy(vd, &scratch, reg_max); + for (i =3D 0; i < N; ++i) { + memcpy(&env->vfp.zregs[(rd + i) & 31], &scratch[i], reg_max); + } return; #endif } =20 /* The entire operation is in RAM, on valid pages. */ =20 - memset(vd, 0, reg_max); + for (i =3D 0; i < N; ++i) { + memset(&env->vfp.zregs[(rd + i) & 31], 0, reg_max); + } + mem_off =3D info.mem_off_first[0]; reg_off =3D info.reg_off_first[0]; reg_last =3D info.reg_off_last[0]; @@ -4534,10 +4542,13 @@ void sve_ld1_r(CPUARMState *env, uint64_t *vg, cons= t target_ulong addr, uint64_t pg =3D vg[reg_off >> 6]; do { if ((pg >> (reg_off & 63)) & 1) { - host_fn(vd, reg_off, host + mem_off); + for (i =3D 0; i < N; ++i) { + host_fn(&env->vfp.zregs[(rd + i) & 31], reg_off, + host + mem_off + (i << msz)); + } } reg_off +=3D 1 << esz; - mem_off +=3D 1 << msz; + mem_off +=3D N << msz; } while (reg_off <=3D reg_last && (reg_off & 63)); } =20 @@ -4547,7 +4558,11 @@ void sve_ld1_r(CPUARMState *env, uint64_t *vg, const= target_ulong addr, */ mem_off =3D info.mem_off_split; if (unlikely(mem_off >=3D 0)) { - tlb_fn(env, vd, info.reg_off_split, addr + mem_off, retaddr); + reg_off =3D info.reg_off_split; + for (i =3D 0; i < N; ++i) { + tlb_fn(env, &env->vfp.zregs[(rd + i) & 31], reg_off, + addr + mem_off + (i << msz), retaddr); + } } =20 mem_off =3D info.mem_off_first[1]; @@ -4560,10 +4575,13 @@ void sve_ld1_r(CPUARMState *env, uint64_t *vg, cons= t target_ulong addr, uint64_t pg =3D vg[reg_off >> 6]; do { if ((pg >> (reg_off & 63)) & 1) { - host_fn(vd, reg_off, host + mem_off); + for (i =3D 0; i < N; ++i) { + host_fn(&env->vfp.zregs[(rd + i) & 31], reg_off, + host + mem_off + (i << msz)); + } } reg_off +=3D 1 << esz; - mem_off +=3D 1 << msz; + mem_off +=3D N << msz; } while (reg_off & 63); } while (reg_off <=3D reg_last); } @@ -4573,7 +4591,7 @@ void sve_ld1_r(CPUARMState *env, uint64_t *vg, const = target_ulong addr, void HELPER(sve_##NAME##_r)(CPUARMState *env, void *vg, \ target_ulong addr, uint32_t desc) \ { \ - sve_ld1_r(env, vg, addr, desc, GETPC(), ESZ, 0, \ + sve_ldN_r(env, vg, addr, desc, GETPC(), ESZ, MO_8, 1, \ sve_##NAME##_host, sve_##NAME##_tlb); \ } =20 @@ -4581,159 +4599,76 @@ void HELPER(sve_##NAME##_r)(CPUARMState *env, void= *vg, \ void HELPER(sve_##NAME##_le_r)(CPUARMState *env, void *vg, \ target_ulong addr, uint32_t desc) \ { \ - sve_ld1_r(env, vg, addr, desc, GETPC(), ESZ, MSZ, \ + sve_ldN_r(env, vg, addr, desc, GETPC(), ESZ, MSZ, 1, \ sve_##NAME##_le_host, sve_##NAME##_le_tlb); \ } \ void HELPER(sve_##NAME##_be_r)(CPUARMState *env, void *vg, \ target_ulong addr, uint32_t desc) \ { \ - sve_ld1_r(env, vg, addr, desc, GETPC(), ESZ, MSZ, \ + sve_ldN_r(env, vg, addr, desc, GETPC(), ESZ, MSZ, 1, \ sve_##NAME##_be_host, sve_##NAME##_be_tlb); \ } =20 -DO_LD1_1(ld1bb, 0) -DO_LD1_1(ld1bhu, 1) -DO_LD1_1(ld1bhs, 1) -DO_LD1_1(ld1bsu, 2) -DO_LD1_1(ld1bss, 2) -DO_LD1_1(ld1bdu, 3) -DO_LD1_1(ld1bds, 3) +DO_LD1_1(ld1bb, MO_8) +DO_LD1_1(ld1bhu, MO_16) +DO_LD1_1(ld1bhs, MO_16) +DO_LD1_1(ld1bsu, MO_32) +DO_LD1_1(ld1bss, MO_32) +DO_LD1_1(ld1bdu, MO_64) +DO_LD1_1(ld1bds, MO_64) =20 -DO_LD1_2(ld1hh, 1, 1) -DO_LD1_2(ld1hsu, 2, 1) -DO_LD1_2(ld1hss, 2, 1) -DO_LD1_2(ld1hdu, 3, 1) -DO_LD1_2(ld1hds, 3, 1) +DO_LD1_2(ld1hh, MO_16, MO_16) +DO_LD1_2(ld1hsu, MO_32, MO_16) +DO_LD1_2(ld1hss, MO_32, MO_16) +DO_LD1_2(ld1hdu, MO_64, MO_16) +DO_LD1_2(ld1hds, MO_64, MO_16) =20 -DO_LD1_2(ld1ss, 2, 2) -DO_LD1_2(ld1sdu, 3, 2) -DO_LD1_2(ld1sds, 3, 2) +DO_LD1_2(ld1ss, MO_32, MO_32) +DO_LD1_2(ld1sdu, MO_64, MO_32) +DO_LD1_2(ld1sds, MO_64, MO_32) =20 -DO_LD1_2(ld1dd, 3, 3) +DO_LD1_2(ld1dd, MO_64, MO_64) =20 #undef DO_LD1_1 #undef DO_LD1_2 =20 -/* - * Common helpers for all contiguous 2,3,4-register predicated loads. - */ -static void sve_ld2_r(CPUARMState *env, void *vg, target_ulong addr, - uint32_t desc, int size, uintptr_t ra, - sve_ldst1_tlb_fn *tlb_fn) -{ - const unsigned rd =3D extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT= , 5); - intptr_t i, oprsz =3D simd_oprsz(desc); - ARMVectorReg scratch[2] =3D { }; - - for (i =3D 0; i < oprsz; ) { - uint16_t pg =3D *(uint16_t *)(vg + H1_2(i >> 3)); - do { - if (pg & 1) { - tlb_fn(env, &scratch[0], i, addr, ra); - tlb_fn(env, &scratch[1], i, addr + size, ra); - } - i +=3D size, pg >>=3D size; - addr +=3D 2 * size; - } while (i & 15); - } - - /* Wait until all exceptions have been raised to write back. */ - memcpy(&env->vfp.zregs[rd], &scratch[0], oprsz); - memcpy(&env->vfp.zregs[(rd + 1) & 31], &scratch[1], oprsz); -} - -static void sve_ld3_r(CPUARMState *env, void *vg, target_ulong addr, - uint32_t desc, int size, uintptr_t ra, - sve_ldst1_tlb_fn *tlb_fn) -{ - const unsigned rd =3D extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT= , 5); - intptr_t i, oprsz =3D simd_oprsz(desc); - ARMVectorReg scratch[3] =3D { }; - - for (i =3D 0; i < oprsz; ) { - uint16_t pg =3D *(uint16_t *)(vg + H1_2(i >> 3)); - do { - if (pg & 1) { - tlb_fn(env, &scratch[0], i, addr, ra); - tlb_fn(env, &scratch[1], i, addr + size, ra); - tlb_fn(env, &scratch[2], i, addr + 2 * size, ra); - } - i +=3D size, pg >>=3D size; - addr +=3D 3 * size; - } while (i & 15); - } - - /* Wait until all exceptions have been raised to write back. */ - memcpy(&env->vfp.zregs[rd], &scratch[0], oprsz); - memcpy(&env->vfp.zregs[(rd + 1) & 31], &scratch[1], oprsz); - memcpy(&env->vfp.zregs[(rd + 2) & 31], &scratch[2], oprsz); -} - -static void sve_ld4_r(CPUARMState *env, void *vg, target_ulong addr, - uint32_t desc, int size, uintptr_t ra, - sve_ldst1_tlb_fn *tlb_fn) -{ - const unsigned rd =3D extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT= , 5); - intptr_t i, oprsz =3D simd_oprsz(desc); - ARMVectorReg scratch[4] =3D { }; - - for (i =3D 0; i < oprsz; ) { - uint16_t pg =3D *(uint16_t *)(vg + H1_2(i >> 3)); - do { - if (pg & 1) { - tlb_fn(env, &scratch[0], i, addr, ra); - tlb_fn(env, &scratch[1], i, addr + size, ra); - tlb_fn(env, &scratch[2], i, addr + 2 * size, ra); - tlb_fn(env, &scratch[3], i, addr + 3 * size, ra); - } - i +=3D size, pg >>=3D size; - addr +=3D 4 * size; - } while (i & 15); - } - - /* Wait until all exceptions have been raised to write back. */ - memcpy(&env->vfp.zregs[rd], &scratch[0], oprsz); - memcpy(&env->vfp.zregs[(rd + 1) & 31], &scratch[1], oprsz); - memcpy(&env->vfp.zregs[(rd + 2) & 31], &scratch[2], oprsz); - memcpy(&env->vfp.zregs[(rd + 3) & 31], &scratch[3], oprsz); -} - #define DO_LDN_1(N) \ -void QEMU_FLATTEN HELPER(sve_ld##N##bb_r) \ - (CPUARMState *env, void *vg, target_ulong addr, uint32_t desc) \ -{ \ - sve_ld##N##_r(env, vg, addr, desc, 1, GETPC(), sve_ld1bb_tlb); \ +void HELPER(sve_ld##N##bb_r)(CPUARMState *env, void *vg, \ + target_ulong addr, uint32_t desc) \ +{ \ + sve_ldN_r(env, vg, addr, desc, GETPC(), MO_8, MO_8, N, \ + sve_ld1bb_host, sve_ld1bb_tlb); \ } =20 -#define DO_LDN_2(N, SUFF, SIZE) \ -void QEMU_FLATTEN HELPER(sve_ld##N##SUFF##_le_r) \ - (CPUARMState *env, void *vg, target_ulong addr, uint32_t desc) \ +#define DO_LDN_2(N, SUFF, ESZ) \ +void HELPER(sve_ld##N##SUFF##_le_r)(CPUARMState *env, void *vg, \ + target_ulong addr, uint32_t desc) \ { \ - sve_ld##N##_r(env, vg, addr, desc, SIZE, GETPC(), \ - sve_ld1##SUFF##_le_tlb); \ + sve_ldN_r(env, vg, addr, desc, GETPC(), ESZ, ESZ, N, \ + sve_ld1##SUFF##_le_host, sve_ld1##SUFF##_le_tlb); \ } \ -void QEMU_FLATTEN HELPER(sve_ld##N##SUFF##_be_r) \ - (CPUARMState *env, void *vg, target_ulong addr, uint32_t desc) \ +void HELPER(sve_ld##N##SUFF##_be_r)(CPUARMState *env, void *vg, \ + target_ulong addr, uint32_t desc) \ { \ - sve_ld##N##_r(env, vg, addr, desc, SIZE, GETPC(), \ - sve_ld1##SUFF##_be_tlb); \ + sve_ldN_r(env, vg, addr, desc, GETPC(), ESZ, ESZ, N, \ + sve_ld1##SUFF##_be_host, sve_ld1##SUFF##_be_tlb); \ } =20 DO_LDN_1(2) DO_LDN_1(3) DO_LDN_1(4) =20 -DO_LDN_2(2, hh, 2) -DO_LDN_2(3, hh, 2) -DO_LDN_2(4, hh, 2) +DO_LDN_2(2, hh, MO_16) +DO_LDN_2(3, hh, MO_16) +DO_LDN_2(4, hh, MO_16) =20 -DO_LDN_2(2, ss, 4) -DO_LDN_2(3, ss, 4) -DO_LDN_2(4, ss, 4) +DO_LDN_2(2, ss, MO_32) +DO_LDN_2(3, ss, MO_32) +DO_LDN_2(4, ss, MO_32) =20 -DO_LDN_2(2, dd, 8) -DO_LDN_2(3, dd, 8) -DO_LDN_2(4, dd, 8) +DO_LDN_2(2, dd, MO_64) +DO_LDN_2(3, dd, MO_64) +DO_LDN_2(4, dd, MO_64) =20 #undef DO_LDN_1 #undef DO_LDN_2 --=20 2.20.1 From nobody Sun May 19 14:32:52 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=linaro.org ARC-Seal: i=1; a=rsa-sha256; t=1588264999; cv=none; d=zohomail.com; s=zohoarc; b=Fe70d97oTPJbpVTj6IW7v+qurKS9jJI5mSML1AwEFDmtFGCeAsddTUNgNn2aSJI3HoXxNgJk62hmp2QGjXF0rmXI6XnZxJvVAcwrB/UKgK/ZmBFfwBplITZCN8izpy84iJcRNixL/cCYekRz3yTnHf+wUpr32syH8Ue1oxL2Kxo= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1588264999; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=ufJAM7+O3t9uSk4u1leOqQvbmYyjTat9N2uJ2S+dGpE=; b=mOdZ7CZJA0PBKrQqLEndsBcLIr5PvgYCwFAAGN71Nzl1RfXpAxOcXjUkm/Wl/pWweiTU4lrN8XCsFITHvIyWsgQjo6/+PWMgYGSTrwhg8+HfwNsYluOWXyLPRuQ90xijTAeijK7g7R1DzQwqQQNfWi+SxvaCrMzVlL12oTIa/00= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1588264999136351.43283017776844; Thu, 30 Apr 2020 09:43:19 -0700 (PDT) Received: from localhost ([::1]:46640 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jUCHN-0007Gg-O0 for importer@patchew.org; Thu, 30 Apr 2020 12:43:17 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:49426) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jUC3G-0001uh-Gz for qemu-devel@nongnu.org; Thu, 30 Apr 2020 12:28:51 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.90_1) (envelope-from ) id 1jUC3A-0006gS-Vf for qemu-devel@nongnu.org; Thu, 30 Apr 2020 12:28:42 -0400 Received: from mail-pj1-x1041.google.com ([2607:f8b0:4864:20::1041]:40949) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jUC39-0006fk-Vx for qemu-devel@nongnu.org; Thu, 30 Apr 2020 12:28:36 -0400 Received: by mail-pj1-x1041.google.com with SMTP id fu13so907147pjb.5 for ; Thu, 30 Apr 2020 09:28:35 -0700 (PDT) Received: from localhost.localdomain (174-21-149-226.tukw.qwest.net. [174.21.149.226]) by smtp.gmail.com with ESMTPSA id j5sm243514pfh.58.2020.04.30.09.28.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Apr 2020 09:28:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ufJAM7+O3t9uSk4u1leOqQvbmYyjTat9N2uJ2S+dGpE=; b=EkeqVueNiiR+Vr4xziYhB4FqBz3PGxVFnhFc3Lq01RX/WNCbRrDMGdvKegug+XWAZJ guCzuisYXPQbK+WAQ681aa8KPQbBJp2LziD2B6LBQabrEtCjF/uwivfgeALvfVw1qvGI 9g5dC7i7LhSX8Qud5e89NUAUzdVmFPyJfQim5fcLcWCJCnl6bQBegQHCqRHWrxJxXJ7U ACH8foR29CT3RApzZq1z6UhDAAUp0DjVAgBQLJD8XpnrRhPqMAvHqMfrjnu5FKYvJ77Y IpLuNFsGo1vMDpyEtpugVhEQG3Z2VJxLpB1pc9E+dfvzm0chZ89h+NbD5SlbYxhCrdiC OUng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ufJAM7+O3t9uSk4u1leOqQvbmYyjTat9N2uJ2S+dGpE=; b=NAbeP2N1A89W03MFdbFWmm2i5MscMR4jdrqkuJKegdneTHtuHRccbCu2Thm1KW/1Vn vukJvHwr2bWr9WuOd3htwNIjBdjuQ2W4WGg2sBKwnE4cEIb3bP2zKEehUHCoxftL+J7a 90F4kH210SK7AZ9MPjW1SLm1J8lXlMWSkwEE+KK+j0FrL5ixXxol5oElE6dNl4oWaPb/ Uw7O8ube/uLDyi+35jeqBZiC3KUVaXcguPIzr4WQD+AspsdUVnDmKjwe5IKO1qEcmW5M UTul3DPung+a0V81fNQLOtpUb2IWzoRutNLBckZqdxKMGL7gdtjtbh1+KXA5MpH+nK1D KGHg== X-Gm-Message-State: AGi0PuavY2rL9s91T6WLIBuBz0uzhFLUz0xwgS/IBMbQfpFLdBqitW3y RCTilzwr/gthWZwarFe0C/EDSA41FBE= X-Google-Smtp-Source: APiQypIln3dHuV/9nbG7oQiAmexOpQEQuRMI/87+V7BN3ExfmddlmpEd/zpd7ODGDvweP5iBCD5wrA== X-Received: by 2002:a17:902:b495:: with SMTP id y21mr4410433plr.111.1588264113299; Thu, 30 Apr 2020 09:28:33 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH v4 13/18] target/arm: Update contiguous first-fault and no-fault loads Date: Thu, 30 Apr 2020 09:28:08 -0700 Message-Id: <20200430162813.17671-14-richard.henderson@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200430162813.17671-1-richard.henderson@linaro.org> References: <20200430162813.17671-1-richard.henderson@linaro.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2607:f8b0:4864:20::1041; envelope-from=richard.henderson@linaro.org; helo=mail-pj1-x1041.google.com X-detected-operating-system: by eggs.gnu.org: Error: [-] PROGRAM ABORT : Malformed IPv6 address (bad octet value). Location : parse_addr6(), p0f-client.c:67 X-Received-From: 2607:f8b0:4864:20::1041 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, alex.bennee@linaro.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @linaro.org) Content-Type: text/plain; charset="utf-8" With sve_cont_ldst_pages, the differences between first-fault and no-fault are minimal, so unify the routines. With cpu_probe_watchpoint, we are able to make progress through pages with TLB_WATCHPOINT set when the watchpoint does not actually fire. Signed-off-by: Richard Henderson Reviewed-by: Peter Maydell --- target/arm/sve_helper.c | 346 +++++++++++++++++++--------------------- 1 file changed, 162 insertions(+), 184 deletions(-) diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index 9365e32646..f4969347d4 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -4101,18 +4101,6 @@ static intptr_t find_next_active(uint64_t *vg, intpt= r_t reg_off, return reg_off; } =20 -/* - * Return the maximum offset <=3D @mem_max which is still within the page - * referenced by @base + @mem_off. - */ -static intptr_t max_for_page(target_ulong base, intptr_t mem_off, - intptr_t mem_max) -{ - target_ulong addr =3D base + mem_off; - intptr_t split =3D -(intptr_t)(addr | TARGET_PAGE_MASK); - return MIN(split, mem_max - mem_off) + mem_off; -} - /* * Resolve the guest virtual address to info->host and info->flags. * If @nofault, return false if the page is invalid, otherwise @@ -4435,19 +4423,6 @@ static void sve_cont_ldst_watchpoints(SVEContLdSt *i= nfo, CPUARMState *env, #endif } =20 -/* - * The result of tlb_vaddr_to_host for user-only is just g2h(x), - * which is always non-null. Elide the useless test. - */ -static inline bool test_host_page(void *host) -{ -#ifdef CONFIG_USER_ONLY - return true; -#else - return likely(host !=3D NULL); -#endif -} - /* * Common helper for all contiguous 1,2,3,4-register predicated stores. */ @@ -4705,167 +4680,167 @@ static void record_fault(CPUARMState *env, uintpt= r_t i, uintptr_t oprsz) } =20 /* - * Common helper for all contiguous first-fault loads. + * Common helper for all contiguous no-fault and first-fault loads. */ -static void sve_ldff1_r(CPUARMState *env, void *vg, const target_ulong add= r, - uint32_t desc, const uintptr_t retaddr, - const int esz, const int msz, - sve_ldst1_host_fn *host_fn, - sve_ldst1_tlb_fn *tlb_fn) +static inline QEMU_ALWAYS_INLINE +void sve_ldnfff1_r(CPUARMState *env, void *vg, const target_ulong addr, + uint32_t desc, const uintptr_t retaddr, + const int esz, const int msz, const SVEContFault fault, + sve_ldst1_host_fn *host_fn, + sve_ldst1_tlb_fn *tlb_fn) { - const TCGMemOpIdx oi =3D extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHI= FT); - const int mmu_idx =3D get_mmuidx(oi); const unsigned rd =3D extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT= , 5); void *vd =3D &env->vfp.zregs[rd]; - const int diffsz =3D esz - msz; const intptr_t reg_max =3D simd_oprsz(desc); - const intptr_t mem_max =3D reg_max >> diffsz; - intptr_t split, reg_off, mem_off, i; + intptr_t reg_off, mem_off, reg_last; + SVEContLdSt info; + int flags; void *host; =20 - /* Skip to the first active element. */ - reg_off =3D find_next_active(vg, 0, reg_max, esz); - if (unlikely(reg_off =3D=3D reg_max)) { + /* Find the active elements. */ + if (!sve_cont_ldst_elements(&info, addr, vg, reg_max, esz, 1 << msz)) { /* The entire predicate was false; no load occurs. */ memset(vd, 0, reg_max); return; } - mem_off =3D reg_off >> diffsz; + reg_off =3D info.reg_off_first[0]; =20 - /* - * If the (remaining) load is entirely within a single page, then: - * For softmmu, and the tlb hits, then no faults will occur; - * For user-only, either the first load will fault or none will. - * We can thus perform the load directly to the destination and - * Vd will be unmodified on any exception path. - */ - split =3D max_for_page(addr, mem_off, mem_max); - if (likely(split =3D=3D mem_max)) { - host =3D tlb_vaddr_to_host(env, addr + mem_off, MMU_DATA_LOAD, mmu= _idx); - if (test_host_page(host)) { - i =3D reg_off; - host -=3D mem_off; - do { - host_fn(vd, i, host + (i >> diffsz)); - i =3D find_next_active(vg, i + (1 << esz), reg_max, esz); - } while (i < reg_max); - /* After any fault, zero any leading inactive elements. */ + /* Probe the page(s). */ + if (!sve_cont_ldst_pages(&info, fault, env, addr, MMU_DATA_LOAD, retad= dr)) { + /* Fault on first element. */ + tcg_debug_assert(fault =3D=3D FAULT_NO); + memset(vd, 0, reg_max); + goto do_fault; + } + + mem_off =3D info.mem_off_first[0]; + flags =3D info.page[0].flags; + + if (fault =3D=3D FAULT_FIRST) { + /* + * Special handling of the first active element, + * if it crosses a page boundary or is MMIO. + */ + bool is_split =3D mem_off =3D=3D info.mem_off_split; + /* TODO: MTE check. */ + if (unlikely(flags !=3D 0) || unlikely(is_split)) { + /* + * Use the slow path for cross-page handling. + * Might trap for MMIO or watchpoints. + */ + tlb_fn(env, vd, reg_off, addr + mem_off, retaddr); + + /* After any fault, zero the other elements. */ swap_memzero(vd, reg_off); - return; + reg_off +=3D 1 << esz; + mem_off +=3D 1 << msz; + swap_memzero(vd + reg_off, reg_max - reg_off); + + if (is_split) { + goto second_page; + } + } else { + memset(vd, 0, reg_max); + } + } else { + memset(vd, 0, reg_max); + if (unlikely(mem_off =3D=3D info.mem_off_split)) { + /* The first active element crosses a page boundary. */ + flags |=3D info.page[1].flags; + if (unlikely(flags & TLB_MMIO)) { + /* Some page is MMIO, see below. */ + goto do_fault; + } + if (unlikely(flags & TLB_WATCHPOINT) && + (cpu_watchpoint_address_matches + (env_cpu(env), addr + mem_off, 1 << msz) + & BP_MEM_READ)) { + /* Watchpoint hit, see below. */ + goto do_fault; + } + /* TODO: MTE check. */ + /* + * Use the slow path for cross-page handling. + * This is RAM, without a watchpoint, and will not trap. + */ + tlb_fn(env, vd, reg_off, addr + mem_off, retaddr); + goto second_page; } } =20 /* - * Perform one normal read, which will fault or not. - * But it is likely to bring the page into the tlb. + * From this point on, all memory operations are MemSingleNF. + * + * Per the MemSingleNF pseudocode, a no-fault load from Device memory + * must not actually hit the bus -- it returns (UNKNOWN, FAULT) instea= d. + * + * Unfortuately we do not have access to the memory attributes from the + * PTE to tell Device memory from Normal memory. So we make a mostly + * correct check, and indicate (UNKNOWN, FAULT) for any MMIO. + * This gives the right answer for the common cases of "Normal memory, + * backed by host RAM" and "Device memory, backed by MMIO". + * The architecture allows us to suppress an NF load and return + * (UNKNOWN, FAULT) for any reason, so our behaviour for the corner + * case of "Normal memory, backed by MMIO" is permitted. The case we + * get wrong is "Device memory, backed by host RAM", for which we + * should return (UNKNOWN, FAULT) for but do not. + * + * Similarly, CPU_BP breakpoints would raise exceptions, and so + * return (UNKNOWN, FAULT). For simplicity, we consider gdb and + * architectural breakpoints the same. */ - tlb_fn(env, vd, reg_off, addr + mem_off, retaddr); + if (unlikely(flags & TLB_MMIO)) { + goto do_fault; + } =20 - /* After any fault, zero any leading predicated false elts. */ - swap_memzero(vd, reg_off); - mem_off +=3D 1 << msz; - reg_off +=3D 1 << esz; + reg_last =3D info.reg_off_last[0]; + host =3D info.page[0].host; =20 - /* Try again to read the balance of the page. */ - split =3D max_for_page(addr, mem_off - 1, mem_max); - if (split >=3D (1 << msz)) { - host =3D tlb_vaddr_to_host(env, addr + mem_off, MMU_DATA_LOAD, mmu= _idx); - if (host) { - host -=3D mem_off; - do { + do { + uint64_t pg =3D *(uint64_t *)(vg + (reg_off >> 3)); + do { + if ((pg >> (reg_off & 63)) & 1) { + if (unlikely(flags & TLB_WATCHPOINT) && + (cpu_watchpoint_address_matches + (env_cpu(env), addr + mem_off, 1 << msz) + & BP_MEM_READ)) { + goto do_fault; + } + /* TODO: MTE check. */ host_fn(vd, reg_off, host + mem_off); - reg_off +=3D 1 << esz; - reg_off =3D find_next_active(vg, reg_off, reg_max, esz); - mem_off =3D reg_off >> diffsz; - } while (split - mem_off >=3D (1 << msz)); - } - } - - record_fault(env, reg_off, reg_max); -} - -/* - * Common helper for all contiguous no-fault loads. - */ -static void sve_ldnf1_r(CPUARMState *env, void *vg, const target_ulong add= r, - uint32_t desc, const int esz, const int msz, - sve_ldst1_host_fn *host_fn) -{ - const unsigned rd =3D extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT= , 5); - void *vd =3D &env->vfp.zregs[rd]; - const int diffsz =3D esz - msz; - const intptr_t reg_max =3D simd_oprsz(desc); - const intptr_t mem_max =3D reg_max >> diffsz; - const int mmu_idx =3D cpu_mmu_index(env, false); - intptr_t split, reg_off, mem_off; - void *host; - -#ifdef CONFIG_USER_ONLY - host =3D tlb_vaddr_to_host(env, addr, MMU_DATA_LOAD, mmu_idx); - if (likely(page_check_range(addr, mem_max, PAGE_READ) =3D=3D 0)) { - /* The entire operation is valid and will not fault. */ - reg_off =3D 0; - do { - mem_off =3D reg_off >> diffsz; - host_fn(vd, reg_off, host + mem_off); + } reg_off +=3D 1 << esz; - reg_off =3D find_next_active(vg, reg_off, reg_max, esz); - } while (reg_off < reg_max); - return; - } -#endif + mem_off +=3D 1 << msz; + } while (reg_off <=3D reg_last && (reg_off & 63)); + } while (reg_off <=3D reg_last); =20 - /* There will be no fault, so we may modify in advance. */ - memset(vd, 0, reg_max); - - /* Skip to the first active element. */ - reg_off =3D find_next_active(vg, 0, reg_max, esz); - if (unlikely(reg_off =3D=3D reg_max)) { - /* The entire predicate was false; no load occurs. */ - return; - } - mem_off =3D reg_off >> diffsz; - -#ifdef CONFIG_USER_ONLY - if (page_check_range(addr + mem_off, 1 << msz, PAGE_READ) =3D=3D 0) { - /* At least one load is valid; take the rest of the page. */ - split =3D max_for_page(addr, mem_off + (1 << msz) - 1, mem_max); - do { - host_fn(vd, reg_off, host + mem_off); - reg_off +=3D 1 << esz; - reg_off =3D find_next_active(vg, reg_off, reg_max, esz); - mem_off =3D reg_off >> diffsz; - } while (split - mem_off >=3D (1 << msz)); - } -#else /* - * If the address is not in the TLB, we have no way to bring the - * entry into the TLB without also risking a fault. Note that - * the corollary is that we never load from an address not in RAM. - * - * This last is out of spec, in a weird corner case. - * Per the MemNF/MemSingleNF pseudocode, a NF load from Device memory - * must not actually hit the bus -- it returns UNKNOWN data instead. - * But if you map non-RAM with Normal memory attributes and do a NF - * load then it should access the bus. (Nobody ought actually do this - * in the real world, obviously.) - * - * Then there are the annoying special cases with watchpoints... - * TODO: Add a form of non-faulting loads using cc->tlb_fill(probe=3Dt= rue). + * MemSingleNF is allowed to fail for any reason. We have special + * code above to handle the first element crossing a page boundary. + * As an implementation choice, decline to handle a cross-page element + * in any other position. */ - host =3D tlb_vaddr_to_host(env, addr + mem_off, MMU_DATA_LOAD, mmu_idx= ); - split =3D max_for_page(addr, mem_off, mem_max); - if (host && split >=3D (1 << msz)) { - host -=3D mem_off; - do { - host_fn(vd, reg_off, host + mem_off); - reg_off +=3D 1 << esz; - reg_off =3D find_next_active(vg, reg_off, reg_max, esz); - mem_off =3D reg_off >> diffsz; - } while (split - mem_off >=3D (1 << msz)); + reg_off =3D info.reg_off_split; + if (reg_off >=3D 0) { + goto do_fault; } -#endif =20 + second_page: + reg_off =3D info.reg_off_first[1]; + if (likely(reg_off < 0)) { + /* No active elements on the second page. All done. */ + return; + } + + /* + * MemSingleNF is allowed to fail for any reason. As an implementation + * choice, decline to handle elements on the second page. This should + * be low frequency as the guest walks through memory -- the next + * iteration of the guest's loop should be aligned on the page boundar= y, + * and then all following iterations will stay aligned. + */ + + do_fault: record_fault(env, reg_off, reg_max); } =20 @@ -4873,58 +4848,61 @@ static void sve_ldnf1_r(CPUARMState *env, void *vg,= const target_ulong addr, void HELPER(sve_ldff1##PART##_r)(CPUARMState *env, void *vg, \ target_ulong addr, uint32_t desc) \ { \ - sve_ldff1_r(env, vg, addr, desc, GETPC(), ESZ, 0, \ - sve_ld1##PART##_host, sve_ld1##PART##_tlb); \ + sve_ldnfff1_r(env, vg, addr, desc, GETPC(), ESZ, MO_8, FAULT_FIRST, \ + sve_ld1##PART##_host, sve_ld1##PART##_tlb); \ } \ void HELPER(sve_ldnf1##PART##_r)(CPUARMState *env, void *vg, \ target_ulong addr, uint32_t desc) \ { \ - sve_ldnf1_r(env, vg, addr, desc, ESZ, 0, sve_ld1##PART##_host); \ + sve_ldnfff1_r(env, vg, addr, desc, GETPC(), ESZ, MO_8, FAULT_NO, \ + sve_ld1##PART##_host, sve_ld1##PART##_tlb); \ } =20 #define DO_LDFF1_LDNF1_2(PART, ESZ, MSZ) \ void HELPER(sve_ldff1##PART##_le_r)(CPUARMState *env, void *vg, \ target_ulong addr, uint32_t desc) \ { \ - sve_ldff1_r(env, vg, addr, desc, GETPC(), ESZ, MSZ, \ - sve_ld1##PART##_le_host, sve_ld1##PART##_le_tlb); \ + sve_ldnfff1_r(env, vg, addr, desc, GETPC(), ESZ, MSZ, FAULT_FIRST, \ + sve_ld1##PART##_le_host, sve_ld1##PART##_le_tlb); \ } \ void HELPER(sve_ldnf1##PART##_le_r)(CPUARMState *env, void *vg, \ target_ulong addr, uint32_t desc) \ { \ - sve_ldnf1_r(env, vg, addr, desc, ESZ, MSZ, sve_ld1##PART##_le_host); \ + sve_ldnfff1_r(env, vg, addr, desc, GETPC(), ESZ, MSZ, FAULT_NO, \ + sve_ld1##PART##_le_host, sve_ld1##PART##_le_tlb); \ } \ void HELPER(sve_ldff1##PART##_be_r)(CPUARMState *env, void *vg, \ target_ulong addr, uint32_t desc) \ { \ - sve_ldff1_r(env, vg, addr, desc, GETPC(), ESZ, MSZ, \ - sve_ld1##PART##_be_host, sve_ld1##PART##_be_tlb); \ + sve_ldnfff1_r(env, vg, addr, desc, GETPC(), ESZ, MSZ, FAULT_FIRST, \ + sve_ld1##PART##_be_host, sve_ld1##PART##_be_tlb); \ } \ void HELPER(sve_ldnf1##PART##_be_r)(CPUARMState *env, void *vg, \ target_ulong addr, uint32_t desc) \ { \ - sve_ldnf1_r(env, vg, addr, desc, ESZ, MSZ, sve_ld1##PART##_be_host); \ + sve_ldnfff1_r(env, vg, addr, desc, GETPC(), ESZ, MSZ, FAULT_NO, \ + sve_ld1##PART##_be_host, sve_ld1##PART##_be_tlb); \ } =20 -DO_LDFF1_LDNF1_1(bb, 0) -DO_LDFF1_LDNF1_1(bhu, 1) -DO_LDFF1_LDNF1_1(bhs, 1) -DO_LDFF1_LDNF1_1(bsu, 2) -DO_LDFF1_LDNF1_1(bss, 2) -DO_LDFF1_LDNF1_1(bdu, 3) -DO_LDFF1_LDNF1_1(bds, 3) +DO_LDFF1_LDNF1_1(bb, MO_8) +DO_LDFF1_LDNF1_1(bhu, MO_16) +DO_LDFF1_LDNF1_1(bhs, MO_16) +DO_LDFF1_LDNF1_1(bsu, MO_32) +DO_LDFF1_LDNF1_1(bss, MO_32) +DO_LDFF1_LDNF1_1(bdu, MO_64) +DO_LDFF1_LDNF1_1(bds, MO_64) =20 -DO_LDFF1_LDNF1_2(hh, 1, 1) -DO_LDFF1_LDNF1_2(hsu, 2, 1) -DO_LDFF1_LDNF1_2(hss, 2, 1) -DO_LDFF1_LDNF1_2(hdu, 3, 1) -DO_LDFF1_LDNF1_2(hds, 3, 1) +DO_LDFF1_LDNF1_2(hh, MO_16, MO_16) +DO_LDFF1_LDNF1_2(hsu, MO_32, MO_16) +DO_LDFF1_LDNF1_2(hss, MO_32, MO_16) +DO_LDFF1_LDNF1_2(hdu, MO_64, MO_16) +DO_LDFF1_LDNF1_2(hds, MO_64, MO_16) =20 -DO_LDFF1_LDNF1_2(ss, 2, 2) -DO_LDFF1_LDNF1_2(sdu, 3, 2) -DO_LDFF1_LDNF1_2(sds, 3, 2) +DO_LDFF1_LDNF1_2(ss, MO_32, MO_32) +DO_LDFF1_LDNF1_2(sdu, MO_64, MO_32) +DO_LDFF1_LDNF1_2(sds, MO_64, MO_32) =20 -DO_LDFF1_LDNF1_2(dd, 3, 3) +DO_LDFF1_LDNF1_2(dd, MO_64, MO_64) =20 #undef DO_LDFF1_LDNF1_1 #undef DO_LDFF1_LDNF1_2 --=20 2.20.1 From nobody Sun May 19 14:32:52 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=linaro.org ARC-Seal: i=1; a=rsa-sha256; t=1588265332; cv=none; d=zohomail.com; s=zohoarc; b=GrjhZS34b2AlNVuW0umVGRql4JMgGWnKaZeeMQjnBreDpO5h2ghzK+wS9F2KEFEhqECsFpL1hHzoQVHl+a/OmLarpYcIqcEBhvPmPCQJ6PqBtH8IlZJSi7HloJatN8ZBefYagWncZs4gJE9dmDXIb5EesmltFABIAi/Yo6hru4s= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1588265332; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=JFw94RsIxJxFivVcuXkKXo6q5XbvxxZGz8yeQyMWqCM=; b=hTUsB1Jpsaq76W+cehQtTmPltmCy/mqVizz0Jc5/uHzdrXVNccGs0unXJn7YXQoWJjeuq0rMdaVFT/USUQGKRN7/R21N8w54lXjdYm+SwF6G6YZsXo8GSkEtppRMJfSObektulkol12xNRghPka0Cd8ShSzyqBpQqezUphmmgRw= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1588265332597331.69759489964633; Thu, 30 Apr 2020 09:48:52 -0700 (PDT) Received: from localhost ([::1]:32926 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jUCMl-00027P-5j for importer@patchew.org; Thu, 30 Apr 2020 12:48:51 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:49416) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jUC3F-0001u2-OO for qemu-devel@nongnu.org; Thu, 30 Apr 2020 12:28:50 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.90_1) (envelope-from ) id 1jUC3B-0006gZ-2a for qemu-devel@nongnu.org; Thu, 30 Apr 2020 12:28:41 -0400 Received: from mail-pg1-x541.google.com ([2607:f8b0:4864:20::541]:33034) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jUC3A-0006fw-LT for qemu-devel@nongnu.org; Thu, 30 Apr 2020 12:28:36 -0400 Received: by mail-pg1-x541.google.com with SMTP id d17so2989301pgo.0 for ; Thu, 30 Apr 2020 09:28:36 -0700 (PDT) Received: from localhost.localdomain (174-21-149-226.tukw.qwest.net. [174.21.149.226]) by smtp.gmail.com with ESMTPSA id j5sm243514pfh.58.2020.04.30.09.28.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Apr 2020 09:28:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=JFw94RsIxJxFivVcuXkKXo6q5XbvxxZGz8yeQyMWqCM=; b=cFmMudyCL5EI8OXGwns3X5ZE4DD6TKeFnanfayi3wjqcOP4XMi/IzjYjKW4x/bjZg4 2RaocvujBqiVo4mJRcYyXNQD/5KKZCvRxbt4LOza1u5CKGKh0kWl4lui5xVyktXtMJan FSsq/arZksdJbqydh3CrX9Xofby9GFuttXCLBUeuY9M49LoeRFQzY9XSf8nE8swYslYh u/rq+BprLSmFLYXeEqQ3f41jmTojRmQAJYeG2lDyimLQ+yDe6v5AlUJTCxp/AtTMa1Nl QxBvZLh0BgZjuJSXBmL2zTBdtNl7zwYwzwIZhiQroOE4ZoyxOKCJqm2wUFjp5IeKPjBE HOSg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=JFw94RsIxJxFivVcuXkKXo6q5XbvxxZGz8yeQyMWqCM=; b=sOUws+tff1sqWiGnTKaERT0oB8vQiwPUr/r1BSP652XZkVghsksHgBA8LqWUGHqdZd s+4M1TNl86Gp/DsD68qZjo0DC4cWmXQbvQhHs4R9dJAt9kYl9vHiqUftPwgm17K9VlSG EHxRysUI/HqSYex4aH8WPBYVR87BazKkllFyeUuIxqOov9G1S/XgNGWMLDLigHyf9IAV /SH05K57Fq1ztAQyo3ynkd0Es8S1oPy7c5O4U/c+XxojDzsW7qo2Msb4/EBcVi6mtwnn k5Hf8S9jMZpFjmP0LGcTuFOQHW+MYFawbH3PlnVufTa7fZVs69Gk/Gixcs6Zs6JKLsb2 oyTQ== X-Gm-Message-State: AGi0PuZu0F+bGe9JENgUAvQeLiMhSPgCMgsGyqLbG12IET1/yIRFtPps Q1sBtI1bxiXpZWPhYWy6vevyhhc+M9k= X-Google-Smtp-Source: APiQypIFccFxsXjvfYO7FPxUrRTbrBuIA46egB9sf0m03sOmduvWhhJBwNNZB7ca6VVXMKUSU0Qg/Q== X-Received: by 2002:a63:b604:: with SMTP id j4mr4041718pgf.124.1588264114413; Thu, 30 Apr 2020 09:28:34 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH v4 14/18] target/arm: Use SVEContLdSt for contiguous stores Date: Thu, 30 Apr 2020 09:28:09 -0700 Message-Id: <20200430162813.17671-15-richard.henderson@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200430162813.17671-1-richard.henderson@linaro.org> References: <20200430162813.17671-1-richard.henderson@linaro.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2607:f8b0:4864:20::541; envelope-from=richard.henderson@linaro.org; helo=mail-pg1-x541.google.com X-detected-operating-system: by eggs.gnu.org: Error: [-] PROGRAM ABORT : Malformed IPv6 address (bad octet value). Location : parse_addr6(), p0f-client.c:67 X-Received-From: 2607:f8b0:4864:20::541 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, alex.bennee@linaro.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @linaro.org) Content-Type: text/plain; charset="utf-8" Follow the model set up for contiguous loads. This handles watchpoints correctly for contiguous stores, recognizing the exception before any changes to memory. Reviewed-by: Peter Maydell Signed-off-by: Richard Henderson --- target/arm/sve_helper.c | 285 ++++++++++++++++++++++------------------ 1 file changed, 159 insertions(+), 126 deletions(-) diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index f4969347d4..4ed9bbe1ee 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -3995,6 +3995,10 @@ static void sve_##NAME##_host(void *vd, intptr_t reg= _off, void *host) \ *(TYPEE *)(vd + H(reg_off)) =3D val; \ } =20 +#define DO_ST_HOST(NAME, H, TYPEE, TYPEM, HOST) \ +static void sve_##NAME##_host(void *vd, intptr_t reg_off, void *host) \ +{ HOST(host, (TYPEM)*(TYPEE *)(vd + H(reg_off))); } + #define DO_LD_TLB(NAME, H, TYPEE, TYPEM, TLB) \ static void sve_##NAME##_tlb(CPUARMState *env, void *vd, intptr_t reg_off,= \ target_ulong addr, uintptr_t ra) = \ @@ -4022,6 +4026,7 @@ DO_LD_PRIM_1(ld1bdu, , uint64_t, uint8_t) DO_LD_PRIM_1(ld1bds, , uint64_t, int8_t) =20 #define DO_ST_PRIM_1(NAME, H, TE, TM) \ + DO_ST_HOST(st1##NAME, H, TE, TM, stb_p) \ DO_ST_TLB(st1##NAME, H, TE, TM, cpu_stb_data_ra) =20 DO_ST_PRIM_1(bb, H1, uint8_t, uint8_t) @@ -4036,6 +4041,8 @@ DO_ST_PRIM_1(bd, , uint64_t, uint8_t) DO_LD_TLB(ld1##NAME##_le, H, TE, TM, cpu_##LD##_le_data_ra) =20 #define DO_ST_PRIM_2(NAME, H, TE, TM, ST) \ + DO_ST_HOST(st1##NAME##_be, H, TE, TM, ST##_be_p) \ + DO_ST_HOST(st1##NAME##_le, H, TE, TM, ST##_le_p) \ DO_ST_TLB(st1##NAME##_be, H, TE, TM, cpu_##ST##_be_data_ra) \ DO_ST_TLB(st1##NAME##_le, H, TE, TM, cpu_##ST##_le_data_ra) =20 @@ -4908,151 +4915,177 @@ DO_LDFF1_LDNF1_2(dd, MO_64, MO_64) #undef DO_LDFF1_LDNF1_2 =20 /* - * Common helpers for all contiguous 1,2,3,4-register predicated stores. + * Common helper for all contiguous 1,2,3,4-register predicated stores. */ -static void sve_st1_r(CPUARMState *env, void *vg, target_ulong addr, - uint32_t desc, const uintptr_t ra, - const int esize, const int msize, - sve_ldst1_tlb_fn *tlb_fn) + +static inline QEMU_ALWAYS_INLINE +void sve_stN_r(CPUARMState *env, uint64_t *vg, target_ulong addr, uint32_t= desc, + const uintptr_t retaddr, const int esz, + const int msz, const int N, + sve_ldst1_host_fn *host_fn, + sve_ldst1_tlb_fn *tlb_fn) { const unsigned rd =3D extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT= , 5); - intptr_t i, oprsz =3D simd_oprsz(desc); - void *vd =3D &env->vfp.zregs[rd]; + const intptr_t reg_max =3D simd_oprsz(desc); + intptr_t reg_off, reg_last, mem_off; + SVEContLdSt info; + void *host; + int i, flags; =20 - for (i =3D 0; i < oprsz; ) { - uint16_t pg =3D *(uint16_t *)(vg + H1_2(i >> 3)); - do { - if (pg & 1) { - tlb_fn(env, vd, i, addr, ra); + /* Find the active elements. */ + if (!sve_cont_ldst_elements(&info, addr, vg, reg_max, esz, N << msz)) { + /* The entire predicate was false; no store occurs. */ + return; + } + + /* Probe the page(s). Exit with exception for any invalid page. */ + sve_cont_ldst_pages(&info, FAULT_ALL, env, addr, MMU_DATA_STORE, retad= dr); + + /* Handle watchpoints for all active elements. */ + sve_cont_ldst_watchpoints(&info, env, vg, addr, 1 << esz, N << msz, + BP_MEM_WRITE, retaddr); + + /* TODO: MTE check. */ + + flags =3D info.page[0].flags | info.page[1].flags; + if (unlikely(flags !=3D 0)) { +#ifdef CONFIG_USER_ONLY + g_assert_not_reached(); +#else + /* + * At least one page includes MMIO. + * Any bus operation can fail with cpu_transaction_failed, + * which for ARM will raise SyncExternal. We cannot avoid + * this fault and will leave with the store incomplete. + */ + mem_off =3D info.mem_off_first[0]; + reg_off =3D info.reg_off_first[0]; + reg_last =3D info.reg_off_last[1]; + if (reg_last < 0) { + reg_last =3D info.reg_off_split; + if (reg_last < 0) { + reg_last =3D info.reg_off_last[0]; } - i +=3D esize, pg >>=3D esize; - addr +=3D msize; - } while (i & 15); + } + + do { + uint64_t pg =3D vg[reg_off >> 6]; + do { + if ((pg >> (reg_off & 63)) & 1) { + for (i =3D 0; i < N; ++i) { + tlb_fn(env, &env->vfp.zregs[(rd + i) & 31], reg_of= f, + addr + mem_off + (i << msz), retaddr); + } + } + reg_off +=3D 1 << esz; + mem_off +=3D N << msz; + } while (reg_off & 63); + } while (reg_off <=3D reg_last); + return; +#endif + } + + mem_off =3D info.mem_off_first[0]; + reg_off =3D info.reg_off_first[0]; + reg_last =3D info.reg_off_last[0]; + host =3D info.page[0].host; + + while (reg_off <=3D reg_last) { + uint64_t pg =3D vg[reg_off >> 6]; + do { + if ((pg >> (reg_off & 63)) & 1) { + for (i =3D 0; i < N; ++i) { + host_fn(&env->vfp.zregs[(rd + i) & 31], reg_off, + host + mem_off + (i << msz)); + } + } + reg_off +=3D 1 << esz; + mem_off +=3D N << msz; + } while (reg_off <=3D reg_last && (reg_off & 63)); + } + + /* + * Use the slow path to manage the cross-page misalignment. + * But we know this is RAM and cannot trap. + */ + mem_off =3D info.mem_off_split; + if (unlikely(mem_off >=3D 0)) { + reg_off =3D info.reg_off_split; + for (i =3D 0; i < N; ++i) { + tlb_fn(env, &env->vfp.zregs[(rd + i) & 31], reg_off, + addr + mem_off + (i << msz), retaddr); + } + } + + mem_off =3D info.mem_off_first[1]; + if (unlikely(mem_off >=3D 0)) { + reg_off =3D info.reg_off_first[1]; + reg_last =3D info.reg_off_last[1]; + host =3D info.page[1].host; + + do { + uint64_t pg =3D vg[reg_off >> 6]; + do { + if ((pg >> (reg_off & 63)) & 1) { + for (i =3D 0; i < N; ++i) { + host_fn(&env->vfp.zregs[(rd + i) & 31], reg_off, + host + mem_off + (i << msz)); + } + } + reg_off +=3D 1 << esz; + mem_off +=3D N << msz; + } while (reg_off & 63); + } while (reg_off <=3D reg_last); } } =20 -static void sve_st2_r(CPUARMState *env, void *vg, target_ulong addr, - uint32_t desc, const uintptr_t ra, - const int esize, const int msize, - sve_ldst1_tlb_fn *tlb_fn) -{ - const unsigned rd =3D extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT= , 5); - intptr_t i, oprsz =3D simd_oprsz(desc); - void *d1 =3D &env->vfp.zregs[rd]; - void *d2 =3D &env->vfp.zregs[(rd + 1) & 31]; - - for (i =3D 0; i < oprsz; ) { - uint16_t pg =3D *(uint16_t *)(vg + H1_2(i >> 3)); - do { - if (pg & 1) { - tlb_fn(env, d1, i, addr, ra); - tlb_fn(env, d2, i, addr + msize, ra); - } - i +=3D esize, pg >>=3D esize; - addr +=3D 2 * msize; - } while (i & 15); - } -} - -static void sve_st3_r(CPUARMState *env, void *vg, target_ulong addr, - uint32_t desc, const uintptr_t ra, - const int esize, const int msize, - sve_ldst1_tlb_fn *tlb_fn) -{ - const unsigned rd =3D extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT= , 5); - intptr_t i, oprsz =3D simd_oprsz(desc); - void *d1 =3D &env->vfp.zregs[rd]; - void *d2 =3D &env->vfp.zregs[(rd + 1) & 31]; - void *d3 =3D &env->vfp.zregs[(rd + 2) & 31]; - - for (i =3D 0; i < oprsz; ) { - uint16_t pg =3D *(uint16_t *)(vg + H1_2(i >> 3)); - do { - if (pg & 1) { - tlb_fn(env, d1, i, addr, ra); - tlb_fn(env, d2, i, addr + msize, ra); - tlb_fn(env, d3, i, addr + 2 * msize, ra); - } - i +=3D esize, pg >>=3D esize; - addr +=3D 3 * msize; - } while (i & 15); - } -} - -static void sve_st4_r(CPUARMState *env, void *vg, target_ulong addr, - uint32_t desc, const uintptr_t ra, - const int esize, const int msize, - sve_ldst1_tlb_fn *tlb_fn) -{ - const unsigned rd =3D extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT= , 5); - intptr_t i, oprsz =3D simd_oprsz(desc); - void *d1 =3D &env->vfp.zregs[rd]; - void *d2 =3D &env->vfp.zregs[(rd + 1) & 31]; - void *d3 =3D &env->vfp.zregs[(rd + 2) & 31]; - void *d4 =3D &env->vfp.zregs[(rd + 3) & 31]; - - for (i =3D 0; i < oprsz; ) { - uint16_t pg =3D *(uint16_t *)(vg + H1_2(i >> 3)); - do { - if (pg & 1) { - tlb_fn(env, d1, i, addr, ra); - tlb_fn(env, d2, i, addr + msize, ra); - tlb_fn(env, d3, i, addr + 2 * msize, ra); - tlb_fn(env, d4, i, addr + 3 * msize, ra); - } - i +=3D esize, pg >>=3D esize; - addr +=3D 4 * msize; - } while (i & 15); - } -} - -#define DO_STN_1(N, NAME, ESIZE) \ -void QEMU_FLATTEN HELPER(sve_st##N##NAME##_r) \ - (CPUARMState *env, void *vg, target_ulong addr, uint32_t desc) \ +#define DO_STN_1(N, NAME, ESZ) \ +void HELPER(sve_st##N##NAME##_r)(CPUARMState *env, void *vg, \ + target_ulong addr, uint32_t desc) \ { \ - sve_st##N##_r(env, vg, addr, desc, GETPC(), ESIZE, 1, \ - sve_st1##NAME##_tlb); \ + sve_stN_r(env, vg, addr, desc, GETPC(), ESZ, MO_8, N, \ + sve_st1##NAME##_host, sve_st1##NAME##_tlb); \ } =20 -#define DO_STN_2(N, NAME, ESIZE, MSIZE) \ -void QEMU_FLATTEN HELPER(sve_st##N##NAME##_le_r) \ - (CPUARMState *env, void *vg, target_ulong addr, uint32_t desc) \ +#define DO_STN_2(N, NAME, ESZ, MSZ) \ +void HELPER(sve_st##N##NAME##_le_r)(CPUARMState *env, void *vg, \ + target_ulong addr, uint32_t desc) \ { \ - sve_st##N##_r(env, vg, addr, desc, GETPC(), ESIZE, MSIZE, \ - sve_st1##NAME##_le_tlb); \ + sve_stN_r(env, vg, addr, desc, GETPC(), ESZ, MSZ, N, \ + sve_st1##NAME##_le_host, sve_st1##NAME##_le_tlb); \ } \ -void QEMU_FLATTEN HELPER(sve_st##N##NAME##_be_r) \ - (CPUARMState *env, void *vg, target_ulong addr, uint32_t desc) \ +void HELPER(sve_st##N##NAME##_be_r)(CPUARMState *env, void *vg, \ + target_ulong addr, uint32_t desc) \ { \ - sve_st##N##_r(env, vg, addr, desc, GETPC(), ESIZE, MSIZE, \ - sve_st1##NAME##_be_tlb); \ + sve_stN_r(env, vg, addr, desc, GETPC(), ESZ, MSZ, N, \ + sve_st1##NAME##_be_host, sve_st1##NAME##_be_tlb); \ } =20 -DO_STN_1(1, bb, 1) -DO_STN_1(1, bh, 2) -DO_STN_1(1, bs, 4) -DO_STN_1(1, bd, 8) -DO_STN_1(2, bb, 1) -DO_STN_1(3, bb, 1) -DO_STN_1(4, bb, 1) +DO_STN_1(1, bb, MO_8) +DO_STN_1(1, bh, MO_16) +DO_STN_1(1, bs, MO_32) +DO_STN_1(1, bd, MO_64) +DO_STN_1(2, bb, MO_8) +DO_STN_1(3, bb, MO_8) +DO_STN_1(4, bb, MO_8) =20 -DO_STN_2(1, hh, 2, 2) -DO_STN_2(1, hs, 4, 2) -DO_STN_2(1, hd, 8, 2) -DO_STN_2(2, hh, 2, 2) -DO_STN_2(3, hh, 2, 2) -DO_STN_2(4, hh, 2, 2) +DO_STN_2(1, hh, MO_16, MO_16) +DO_STN_2(1, hs, MO_32, MO_16) +DO_STN_2(1, hd, MO_64, MO_16) +DO_STN_2(2, hh, MO_16, MO_16) +DO_STN_2(3, hh, MO_16, MO_16) +DO_STN_2(4, hh, MO_16, MO_16) =20 -DO_STN_2(1, ss, 4, 4) -DO_STN_2(1, sd, 8, 4) -DO_STN_2(2, ss, 4, 4) -DO_STN_2(3, ss, 4, 4) -DO_STN_2(4, ss, 4, 4) +DO_STN_2(1, ss, MO_32, MO_32) +DO_STN_2(1, sd, MO_64, MO_32) +DO_STN_2(2, ss, MO_32, MO_32) +DO_STN_2(3, ss, MO_32, MO_32) +DO_STN_2(4, ss, MO_32, MO_32) =20 -DO_STN_2(1, dd, 8, 8) -DO_STN_2(2, dd, 8, 8) -DO_STN_2(3, dd, 8, 8) -DO_STN_2(4, dd, 8, 8) +DO_STN_2(1, dd, MO_64, MO_64) +DO_STN_2(2, dd, MO_64, MO_64) +DO_STN_2(3, dd, MO_64, MO_64) +DO_STN_2(4, dd, MO_64, MO_64) =20 #undef DO_STN_1 #undef DO_STN_2 --=20 2.20.1 From nobody Sun May 19 14:32:52 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=linaro.org ARC-Seal: i=1; a=rsa-sha256; t=1588265442; cv=none; d=zohomail.com; s=zohoarc; b=cfyjTDDlDkpK7MP72SYaBvvMgjvxXGpUjvK6VRVIqkdzwtQ0N5ClnIyqyNvgl2G0/RaXcQtoRje4lltk2B/10/e1v03BYeRJPI3R38GivIbFyM0nYB/n2z32PwChSsrkNPfBjkMbbtgTjnjwptKEralOPQdlVyze2V2y5457vok= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1588265442; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=NJ7tcpWJHZVF+lqFFr3iSzkxO+gqeqH9eerAkzbMBl4=; b=NARUJTZmvFMHg5h68B+TdFg195tlYTtIHKV/4hYldjFiYP2n6iZGmd+tcvh78b4bdBJVUvy1QNtjZtifg7cAhAelI9cUBu/Mq8ElNK7DpK1JekJMLAmIuCDiob5Hfga1gPs2YdTdt7hI4mE0z4AS7B8ssNIojGoNs7u4JNWn1Fc= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1588265442456147.9577779860382; Thu, 30 Apr 2020 09:50:42 -0700 (PDT) Received: from localhost ([::1]:41504 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jUCOX-0005jy-0P for importer@patchew.org; Thu, 30 Apr 2020 12:50:41 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:49438) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jUC3K-0001x0-Fl for qemu-devel@nongnu.org; Thu, 30 Apr 2020 12:28:55 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.90_1) (envelope-from ) id 1jUC3C-0006lM-MN for qemu-devel@nongnu.org; Thu, 30 Apr 2020 12:28:46 -0400 Received: from mail-pl1-x641.google.com ([2607:f8b0:4864:20::641]:43671) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jUC3C-0006gr-7U for qemu-devel@nongnu.org; Thu, 30 Apr 2020 12:28:38 -0400 Received: by mail-pl1-x641.google.com with SMTP id z6so2416214plk.10 for ; Thu, 30 Apr 2020 09:28:37 -0700 (PDT) Received: from localhost.localdomain (174-21-149-226.tukw.qwest.net. [174.21.149.226]) by smtp.gmail.com with ESMTPSA id j5sm243514pfh.58.2020.04.30.09.28.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Apr 2020 09:28:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=NJ7tcpWJHZVF+lqFFr3iSzkxO+gqeqH9eerAkzbMBl4=; b=wGfEHdJiGfQSHZsbE7ku6KqcU4DJum1Sw2mEKLkTe8+xJT0zw4CG8YrhoI/9dC1/dM UcyUafQSe6CkP71BlbiURtQsN1+un/YGfxJcF98Fmq3nuvqI7OTNUb2L7633RLE0LFV9 IOIATteyph15TJ7/iVrRIXIvvJdCUwwV4szzkd1vt+/6flM/r07kHmj3YXp+wKajc//d JC6SJmFUJyB/k4JGVlhp5XshhIRpYwww4y/w+g9d0GWa0GvKau+dueVw5uZavTCADXK9 l8eoaTGX4q1TXSvvo22zyuBqPa/RhUoMvKZ71rMKrXBYv+4Th79R03ZpmtNnHIrvRX46 f+0g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=NJ7tcpWJHZVF+lqFFr3iSzkxO+gqeqH9eerAkzbMBl4=; b=ffXp77EonADs3QqkZ+u6zh3FwKGXnVF9z+ersvttF2v4lYknqQK4Vs8jyWynz0iVvX CHINkruYRbvBUazZPEGEJkeJkBJD9L/+mVxevlV+khMCXnl4nsJ+Zo80bFhsa//c6Nts Aenj8jyqBDRdDQjFQag3RWd1/a1bY+4ln1nkoHSDC6C9VdF4fhSsRr6jQ4xWKQsIvTws FZ9WDpikZvuRcyk3bJ7ZgfywAXgefCjtw3Qjon5og//lsLi6CDC0uzJ7VxPPWck3k9mu W7CWQ4p4zjAWTBSRR9nuUR/W9fd0FW8yVGX9MYR78CCtKhfVeW61TGCF76BU2XphrC+x vBCw== X-Gm-Message-State: AGi0PuZwKIDzHMnREzAfAUNLNKKXosSWbAlWI3GhJ5bk87ryppkpu3Kl gD84ArbQnOIYZ2lS3F3RcjLrYWzxTcs= X-Google-Smtp-Source: APiQypK6pJQLpnq1gZfqXa089P5G7d3lSESza9Ja5izyF0zZyCzddWirfnimQN1G18D+qh7pFvoHvA== X-Received: by 2002:a17:90a:1b67:: with SMTP id q94mr3993063pjq.84.1588264116092; Thu, 30 Apr 2020 09:28:36 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH v4 15/18] target/arm: Reuse sve_probe_page for gather first-fault loads Date: Thu, 30 Apr 2020 09:28:10 -0700 Message-Id: <20200430162813.17671-16-richard.henderson@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200430162813.17671-1-richard.henderson@linaro.org> References: <20200430162813.17671-1-richard.henderson@linaro.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2607:f8b0:4864:20::641; envelope-from=richard.henderson@linaro.org; helo=mail-pl1-x641.google.com X-detected-operating-system: by eggs.gnu.org: Error: [-] PROGRAM ABORT : Malformed IPv6 address (bad octet value). Location : parse_addr6(), p0f-client.c:67 X-Received-From: 2607:f8b0:4864:20::641 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, alex.bennee@linaro.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @linaro.org) Content-Type: text/plain; charset="utf-8" This avoids the need for a separate set of helpers to implement no-fault semantics, and will enable MTE in the future. Reviewed-by: Peter Maydell Signed-off-by: Richard Henderson --- target/arm/sve_helper.c | 323 ++++++++++++++++------------------------ 1 file changed, 127 insertions(+), 196 deletions(-) diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index 4ed9bbe1ee..1560129b08 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -5254,231 +5254,162 @@ DO_LD1_ZPZ_D(dd_be, zd) =20 /* First fault loads with a vector index. */ =20 -/* Load one element into VD+REG_OFF from (ENV,VADDR) without faulting. - * The controlling predicate is known to be true. Return true if the - * load was successful. - */ -typedef bool sve_ld1_nf_fn(CPUARMState *env, void *vd, intptr_t reg_off, - target_ulong vaddr, int mmu_idx); - -#ifdef CONFIG_SOFTMMU -#define DO_LD_NF(NAME, H, TYPEE, TYPEM, HOST) \ -static bool sve_ld##NAME##_nf(CPUARMState *env, void *vd, intptr_t reg_off= , \ - target_ulong addr, int mmu_idx) = \ -{ = \ - target_ulong next_page =3D -(addr | TARGET_PAGE_MASK); = \ - if (likely(next_page - addr >=3D sizeof(TYPEM))) { = \ - void *host =3D tlb_vaddr_to_host(env, addr, MMU_DATA_LOAD, mmu_idx= ); \ - if (likely(host)) { = \ - TYPEM val =3D HOST(host); = \ - *(TYPEE *)(vd + H(reg_off)) =3D val; = \ - return true; = \ - } = \ - } = \ - return false; = \ -} -#else -#define DO_LD_NF(NAME, H, TYPEE, TYPEM, HOST) \ -static bool sve_ld##NAME##_nf(CPUARMState *env, void *vd, intptr_t reg_off= , \ - target_ulong addr, int mmu_idx) = \ -{ = \ - if (likely(page_check_range(addr, sizeof(TYPEM), PAGE_READ))) { = \ - TYPEM val =3D HOST(g2h(addr)); = \ - *(TYPEE *)(vd + H(reg_off)) =3D val; = \ - return true; = \ - } = \ - return false; = \ -} -#endif - -DO_LD_NF(bsu, H1_4, uint32_t, uint8_t, ldub_p) -DO_LD_NF(bss, H1_4, uint32_t, int8_t, ldsb_p) -DO_LD_NF(bdu, , uint64_t, uint8_t, ldub_p) -DO_LD_NF(bds, , uint64_t, int8_t, ldsb_p) - -DO_LD_NF(hsu_le, H1_4, uint32_t, uint16_t, lduw_le_p) -DO_LD_NF(hss_le, H1_4, uint32_t, int16_t, ldsw_le_p) -DO_LD_NF(hsu_be, H1_4, uint32_t, uint16_t, lduw_be_p) -DO_LD_NF(hss_be, H1_4, uint32_t, int16_t, ldsw_be_p) -DO_LD_NF(hdu_le, , uint64_t, uint16_t, lduw_le_p) -DO_LD_NF(hds_le, , uint64_t, int16_t, ldsw_le_p) -DO_LD_NF(hdu_be, , uint64_t, uint16_t, lduw_be_p) -DO_LD_NF(hds_be, , uint64_t, int16_t, ldsw_be_p) - -DO_LD_NF(ss_le, H1_4, uint32_t, uint32_t, ldl_le_p) -DO_LD_NF(ss_be, H1_4, uint32_t, uint32_t, ldl_be_p) -DO_LD_NF(sdu_le, , uint64_t, uint32_t, ldl_le_p) -DO_LD_NF(sds_le, , uint64_t, int32_t, ldl_le_p) -DO_LD_NF(sdu_be, , uint64_t, uint32_t, ldl_be_p) -DO_LD_NF(sds_be, , uint64_t, int32_t, ldl_be_p) - -DO_LD_NF(dd_le, , uint64_t, uint64_t, ldq_le_p) -DO_LD_NF(dd_be, , uint64_t, uint64_t, ldq_be_p) - /* - * Common helper for all gather first-faulting loads. + * Common helpers for all gather first-faulting loads. */ -static inline void sve_ldff1_zs(CPUARMState *env, void *vd, void *vg, void= *vm, - target_ulong base, uint32_t desc, uintptr_= t ra, - zreg_off_fn *off_fn, sve_ldst1_tlb_fn *tlb= _fn, - sve_ld1_nf_fn *nonfault_fn) + +static inline QEMU_ALWAYS_INLINE +void sve_ldff1_z(CPUARMState *env, void *vd, uint64_t *vg, void *vm, + target_ulong base, uint32_t desc, uintptr_t retaddr, + const int esz, const int msz, zreg_off_fn *off_fn, + sve_ldst1_host_fn *host_fn, + sve_ldst1_tlb_fn *tlb_fn) { - const TCGMemOpIdx oi =3D extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHI= FT); - const int mmu_idx =3D get_mmuidx(oi); + const int mmu_idx =3D cpu_mmu_index(env, false); const int scale =3D extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, = 2); - intptr_t reg_off, reg_max =3D simd_oprsz(desc); - target_ulong addr; + const int esize =3D 1 << esz; + const int msize =3D 1 << msz; + const intptr_t reg_max =3D simd_oprsz(desc); + intptr_t reg_off; + SVEHostPage info; + target_ulong addr, in_page; =20 /* Skip to the first true predicate. */ - reg_off =3D find_next_active(vg, 0, reg_max, MO_32); - if (likely(reg_off < reg_max)) { - /* Perform one normal read, which will fault or not. */ - addr =3D off_fn(vm, reg_off); - addr =3D base + (addr << scale); - tlb_fn(env, vd, reg_off, addr, ra); - - /* The rest of the reads will be non-faulting. */ + reg_off =3D find_next_active(vg, 0, reg_max, esz); + if (unlikely(reg_off >=3D reg_max)) { + /* The entire predicate was false; no load occurs. */ + memset(vd, 0, reg_max); + return; } =20 - /* After any fault, zero the leading predicated false elements. */ + /* + * Probe the first element, allowing faults. + */ + addr =3D base + (off_fn(vm, reg_off) << scale); + tlb_fn(env, vd, reg_off, addr, retaddr); + + /* After any fault, zero the other elements. */ swap_memzero(vd, reg_off); + reg_off +=3D esize; + swap_memzero(vd + reg_off, reg_max - reg_off); =20 - while (likely((reg_off +=3D 4) < reg_max)) { - uint64_t pg =3D *(uint64_t *)(vg + (reg_off >> 6) * 8); - if (likely((pg >> (reg_off & 63)) & 1)) { - addr =3D off_fn(vm, reg_off); - addr =3D base + (addr << scale); - if (!nonfault_fn(env, vd, reg_off, addr, mmu_idx)) { - record_fault(env, reg_off, reg_max); - break; + /* + * Probe the remaining elements, not allowing faults. + */ + while (reg_off < reg_max) { + uint64_t pg =3D vg[reg_off >> 6]; + do { + if (likely((pg >> (reg_off & 63)) & 1)) { + addr =3D base + (off_fn(vm, reg_off) << scale); + in_page =3D -(addr | TARGET_PAGE_MASK); + + if (unlikely(in_page < msize)) { + /* Stop if the element crosses a page boundary. */ + goto fault; + } + + sve_probe_page(&info, true, env, addr, 0, MMU_DATA_LOAD, + mmu_idx, retaddr); + if (unlikely(info.flags & (TLB_INVALID_MASK | TLB_MMIO))) { + goto fault; + } + if (unlikely(info.flags & TLB_WATCHPOINT) && + (cpu_watchpoint_address_matches + (env_cpu(env), addr, msize) & BP_MEM_READ)) { + goto fault; + } + /* TODO: MTE check. */ + + host_fn(vd, reg_off, info.host); } - } else { - *(uint32_t *)(vd + H1_4(reg_off)) =3D 0; - } + reg_off +=3D esize; + } while (reg_off & 63); } + return; + + fault: + record_fault(env, reg_off, reg_max); } =20 -static inline void sve_ldff1_zd(CPUARMState *env, void *vd, void *vg, void= *vm, - target_ulong base, uint32_t desc, uintptr_= t ra, - zreg_off_fn *off_fn, sve_ldst1_tlb_fn *tlb= _fn, - sve_ld1_nf_fn *nonfault_fn) -{ - const TCGMemOpIdx oi =3D extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHI= FT); - const int mmu_idx =3D get_mmuidx(oi); - const int scale =3D extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, = 2); - intptr_t reg_off, reg_max =3D simd_oprsz(desc); - target_ulong addr; - - /* Skip to the first true predicate. */ - reg_off =3D find_next_active(vg, 0, reg_max, MO_64); - if (likely(reg_off < reg_max)) { - /* Perform one normal read, which will fault or not. */ - addr =3D off_fn(vm, reg_off); - addr =3D base + (addr << scale); - tlb_fn(env, vd, reg_off, addr, ra); - - /* The rest of the reads will be non-faulting. */ - } - - /* After any fault, zero the leading predicated false elements. */ - swap_memzero(vd, reg_off); - - while (likely((reg_off +=3D 8) < reg_max)) { - uint8_t pg =3D *(uint8_t *)(vg + H1(reg_off >> 3)); - if (likely(pg & 1)) { - addr =3D off_fn(vm, reg_off); - addr =3D base + (addr << scale); - if (!nonfault_fn(env, vd, reg_off, addr, mmu_idx)) { - record_fault(env, reg_off, reg_max); - break; - } - } else { - *(uint64_t *)(vd + reg_off) =3D 0; - } - } +#define DO_LDFF1_ZPZ_S(MEM, OFS, MSZ) \ +void HELPER(sve_ldff##MEM##_##OFS)(CPUARMState *env, void *vd, void *vg, = \ + void *vm, target_ulong base, uint32_t d= esc) \ +{ = \ + sve_ldff1_z(env, vd, vg, vm, base, desc, GETPC(), MO_32, MSZ, = \ + off_##OFS##_s, sve_ld1##MEM##_host, sve_ld1##MEM##_tlb); = \ } =20 -#define DO_LDFF1_ZPZ_S(MEM, OFS) \ -void HELPER(sve_ldff##MEM##_##OFS) \ - (CPUARMState *env, void *vd, void *vg, void *vm, \ - target_ulong base, uint32_t desc) \ -{ \ - sve_ldff1_zs(env, vd, vg, vm, base, desc, GETPC(), \ - off_##OFS##_s, sve_ld1##MEM##_tlb, sve_ld##MEM##_nf); \ +#define DO_LDFF1_ZPZ_D(MEM, OFS, MSZ) \ +void HELPER(sve_ldff##MEM##_##OFS)(CPUARMState *env, void *vd, void *vg, = \ + void *vm, target_ulong base, uint32_t d= esc) \ +{ = \ + sve_ldff1_z(env, vd, vg, vm, base, desc, GETPC(), MO_64, MSZ, = \ + off_##OFS##_d, sve_ld1##MEM##_host, sve_ld1##MEM##_tlb); = \ } =20 -#define DO_LDFF1_ZPZ_D(MEM, OFS) \ -void HELPER(sve_ldff##MEM##_##OFS) \ - (CPUARMState *env, void *vd, void *vg, void *vm, \ - target_ulong base, uint32_t desc) \ -{ \ - sve_ldff1_zd(env, vd, vg, vm, base, desc, GETPC(), \ - off_##OFS##_d, sve_ld1##MEM##_tlb, sve_ld##MEM##_nf); \ -} +DO_LDFF1_ZPZ_S(bsu, zsu, MO_8) +DO_LDFF1_ZPZ_S(bsu, zss, MO_8) +DO_LDFF1_ZPZ_D(bdu, zsu, MO_8) +DO_LDFF1_ZPZ_D(bdu, zss, MO_8) +DO_LDFF1_ZPZ_D(bdu, zd, MO_8) =20 -DO_LDFF1_ZPZ_S(bsu, zsu) -DO_LDFF1_ZPZ_S(bsu, zss) -DO_LDFF1_ZPZ_D(bdu, zsu) -DO_LDFF1_ZPZ_D(bdu, zss) -DO_LDFF1_ZPZ_D(bdu, zd) +DO_LDFF1_ZPZ_S(bss, zsu, MO_8) +DO_LDFF1_ZPZ_S(bss, zss, MO_8) +DO_LDFF1_ZPZ_D(bds, zsu, MO_8) +DO_LDFF1_ZPZ_D(bds, zss, MO_8) +DO_LDFF1_ZPZ_D(bds, zd, MO_8) =20 -DO_LDFF1_ZPZ_S(bss, zsu) -DO_LDFF1_ZPZ_S(bss, zss) -DO_LDFF1_ZPZ_D(bds, zsu) -DO_LDFF1_ZPZ_D(bds, zss) -DO_LDFF1_ZPZ_D(bds, zd) +DO_LDFF1_ZPZ_S(hsu_le, zsu, MO_16) +DO_LDFF1_ZPZ_S(hsu_le, zss, MO_16) +DO_LDFF1_ZPZ_D(hdu_le, zsu, MO_16) +DO_LDFF1_ZPZ_D(hdu_le, zss, MO_16) +DO_LDFF1_ZPZ_D(hdu_le, zd, MO_16) =20 -DO_LDFF1_ZPZ_S(hsu_le, zsu) -DO_LDFF1_ZPZ_S(hsu_le, zss) -DO_LDFF1_ZPZ_D(hdu_le, zsu) -DO_LDFF1_ZPZ_D(hdu_le, zss) -DO_LDFF1_ZPZ_D(hdu_le, zd) +DO_LDFF1_ZPZ_S(hsu_be, zsu, MO_16) +DO_LDFF1_ZPZ_S(hsu_be, zss, MO_16) +DO_LDFF1_ZPZ_D(hdu_be, zsu, MO_16) +DO_LDFF1_ZPZ_D(hdu_be, zss, MO_16) +DO_LDFF1_ZPZ_D(hdu_be, zd, MO_16) =20 -DO_LDFF1_ZPZ_S(hsu_be, zsu) -DO_LDFF1_ZPZ_S(hsu_be, zss) -DO_LDFF1_ZPZ_D(hdu_be, zsu) -DO_LDFF1_ZPZ_D(hdu_be, zss) -DO_LDFF1_ZPZ_D(hdu_be, zd) +DO_LDFF1_ZPZ_S(hss_le, zsu, MO_16) +DO_LDFF1_ZPZ_S(hss_le, zss, MO_16) +DO_LDFF1_ZPZ_D(hds_le, zsu, MO_16) +DO_LDFF1_ZPZ_D(hds_le, zss, MO_16) +DO_LDFF1_ZPZ_D(hds_le, zd, MO_16) =20 -DO_LDFF1_ZPZ_S(hss_le, zsu) -DO_LDFF1_ZPZ_S(hss_le, zss) -DO_LDFF1_ZPZ_D(hds_le, zsu) -DO_LDFF1_ZPZ_D(hds_le, zss) -DO_LDFF1_ZPZ_D(hds_le, zd) +DO_LDFF1_ZPZ_S(hss_be, zsu, MO_16) +DO_LDFF1_ZPZ_S(hss_be, zss, MO_16) +DO_LDFF1_ZPZ_D(hds_be, zsu, MO_16) +DO_LDFF1_ZPZ_D(hds_be, zss, MO_16) +DO_LDFF1_ZPZ_D(hds_be, zd, MO_16) =20 -DO_LDFF1_ZPZ_S(hss_be, zsu) -DO_LDFF1_ZPZ_S(hss_be, zss) -DO_LDFF1_ZPZ_D(hds_be, zsu) -DO_LDFF1_ZPZ_D(hds_be, zss) -DO_LDFF1_ZPZ_D(hds_be, zd) +DO_LDFF1_ZPZ_S(ss_le, zsu, MO_32) +DO_LDFF1_ZPZ_S(ss_le, zss, MO_32) +DO_LDFF1_ZPZ_D(sdu_le, zsu, MO_32) +DO_LDFF1_ZPZ_D(sdu_le, zss, MO_32) +DO_LDFF1_ZPZ_D(sdu_le, zd, MO_32) =20 -DO_LDFF1_ZPZ_S(ss_le, zsu) -DO_LDFF1_ZPZ_S(ss_le, zss) -DO_LDFF1_ZPZ_D(sdu_le, zsu) -DO_LDFF1_ZPZ_D(sdu_le, zss) -DO_LDFF1_ZPZ_D(sdu_le, zd) +DO_LDFF1_ZPZ_S(ss_be, zsu, MO_32) +DO_LDFF1_ZPZ_S(ss_be, zss, MO_32) +DO_LDFF1_ZPZ_D(sdu_be, zsu, MO_32) +DO_LDFF1_ZPZ_D(sdu_be, zss, MO_32) +DO_LDFF1_ZPZ_D(sdu_be, zd, MO_32) =20 -DO_LDFF1_ZPZ_S(ss_be, zsu) -DO_LDFF1_ZPZ_S(ss_be, zss) -DO_LDFF1_ZPZ_D(sdu_be, zsu) -DO_LDFF1_ZPZ_D(sdu_be, zss) -DO_LDFF1_ZPZ_D(sdu_be, zd) +DO_LDFF1_ZPZ_D(sds_le, zsu, MO_32) +DO_LDFF1_ZPZ_D(sds_le, zss, MO_32) +DO_LDFF1_ZPZ_D(sds_le, zd, MO_32) =20 -DO_LDFF1_ZPZ_D(sds_le, zsu) -DO_LDFF1_ZPZ_D(sds_le, zss) -DO_LDFF1_ZPZ_D(sds_le, zd) +DO_LDFF1_ZPZ_D(sds_be, zsu, MO_32) +DO_LDFF1_ZPZ_D(sds_be, zss, MO_32) +DO_LDFF1_ZPZ_D(sds_be, zd, MO_32) =20 -DO_LDFF1_ZPZ_D(sds_be, zsu) -DO_LDFF1_ZPZ_D(sds_be, zss) -DO_LDFF1_ZPZ_D(sds_be, zd) +DO_LDFF1_ZPZ_D(dd_le, zsu, MO_64) +DO_LDFF1_ZPZ_D(dd_le, zss, MO_64) +DO_LDFF1_ZPZ_D(dd_le, zd, MO_64) =20 -DO_LDFF1_ZPZ_D(dd_le, zsu) -DO_LDFF1_ZPZ_D(dd_le, zss) -DO_LDFF1_ZPZ_D(dd_le, zd) - -DO_LDFF1_ZPZ_D(dd_be, zsu) -DO_LDFF1_ZPZ_D(dd_be, zss) -DO_LDFF1_ZPZ_D(dd_be, zd) +DO_LDFF1_ZPZ_D(dd_be, zsu, MO_64) +DO_LDFF1_ZPZ_D(dd_be, zss, MO_64) +DO_LDFF1_ZPZ_D(dd_be, zd, MO_64) =20 /* Stores with a vector index. */ =20 --=20 2.20.1 From nobody Sun May 19 14:32:52 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=linaro.org ARC-Seal: i=1; a=rsa-sha256; t=1588265118; cv=none; d=zohomail.com; s=zohoarc; b=hPYMEyoa2ZI3LRhFNVVYcerl2mzw+uiEubRIyVwd8FQ9PKGjkJvOWuMiDLaTJYn+ImwoOjZUHSHyK24/NvuSrZw/YNDpq4iUKT2E9ZytpQf2rGr7KlAwCJ4YtA+U40nJPJzpOHq6UfmHi1+EVn9e2dpR4m5ihswGM6mcQ/d2cQ4= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1588265118; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=2NrE6foXFUL+vDhzqPi3FsUrqKV9Hi7b6uhKwfiWAO8=; b=nWpnFqQyhplijwjZfmmeDlBmPPx3ZhPv/gUruxfLCb87hVTTCsBz3IP4fnhrU1vtdnyf1zc9PKsVKOo1CfB4pWNkqbTjcaveeZwC6IjSAUvbW9ST6PEaYmVED5BhDzwxMxneUaQuXhiR1KLBWI6gyT+wVgG9Cbn04n+SkHnnBJY= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1588265118327592.2347364061111; Thu, 30 Apr 2020 09:45:18 -0700 (PDT) Received: from localhost ([::1]:53028 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jUCJI-0005dp-Sk for importer@patchew.org; Thu, 30 Apr 2020 12:45:16 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:49444) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jUC3M-0001yO-DM for qemu-devel@nongnu.org; Thu, 30 Apr 2020 12:28:54 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.90_1) (envelope-from ) id 1jUC3E-0006qa-DI for qemu-devel@nongnu.org; Thu, 30 Apr 2020 12:28:48 -0400 Received: from mail-pg1-x543.google.com ([2607:f8b0:4864:20::543]:39079) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jUC3C-0006k1-VS for qemu-devel@nongnu.org; Thu, 30 Apr 2020 12:28:39 -0400 Received: by mail-pg1-x543.google.com with SMTP id d3so2979860pgj.6 for ; Thu, 30 Apr 2020 09:28:38 -0700 (PDT) Received: from localhost.localdomain (174-21-149-226.tukw.qwest.net. [174.21.149.226]) by smtp.gmail.com with ESMTPSA id j5sm243514pfh.58.2020.04.30.09.28.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Apr 2020 09:28:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=2NrE6foXFUL+vDhzqPi3FsUrqKV9Hi7b6uhKwfiWAO8=; b=swyzA96hOANTc7PbvBV42AgJucDAlZr+MLeRdqk4UAKUnJDPZC0+e8CJjfbHJt4PIb QJimWcVSeTYIBug5iWNJymlR18dQP/1/R/CbPq5IdvN7OwUsQ0s3804ISRXxhQikl7pp FA28FTnoiIw1XzU4qn3tjhg8+gA9ZufIBqd/Q7N3SLRG8tZjwOEMZLIETeR+eHptVCfX ocNBf1Sek1Ba4KZjPm3lk8WCs75MRnYjRZJW59aPS9Xe+yeSxP9eMY5t9TUWWIkTRhVC ToAbkaztjY8gBf2f/vJqHLD8Om478GxSwvXmNY22zXCPaPsfccNBRvCCdoBUHGVFccOI tfyQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=2NrE6foXFUL+vDhzqPi3FsUrqKV9Hi7b6uhKwfiWAO8=; b=oWtgpjkGHY5JH0X0x24zz5tyOWyxyxTkyMm1RBUEaC4ThCXVxT8SxzEcHcRAqq81HF 8o/h6QXI+0tBbIVJF9mn5mfw7FHCyJSB0Vf59NZVktN+jSBAIQQjwmsD1l++KQC9j8VY hXx6bqoNDc6HXYbZqXOyy5ZXx8t+7IuH0g0RoguS+VENbK2k/l012JFRz2eRSCS22YZm P8hi3EYIdy9Z8qEL24VvlP7fd6HQWL6ivCtVQRyoIOVkEwBK9VL9edRUJEv1RzfBaI8N xpaOkkZcNvzKYf+PIJ2E7/NyZ3WluqdJejpB0S1L4q/a0tQ+zIpIgQ+eNvQNM1lTJWLP 9yFg== X-Gm-Message-State: AGi0PuZk/8qhvffZBm0CIutXcGfeJNMf4bY0VjAoTq8li4iPaij+SdMi ZQVd0jQKybG+gxklB2yF5WEqM7KtOMQ= X-Google-Smtp-Source: APiQypJpoHI204LilrCprb34ijBR8O3PY2G3N7mTs0jVggZe6jqtl35JsbYfJzeGLDluSlMHI4CF2Q== X-Received: by 2002:a62:5ec7:: with SMTP id s190mr140188pfb.130.1588264117181; Thu, 30 Apr 2020 09:28:37 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH v4 16/18] target/arm: Reuse sve_probe_page for scatter stores Date: Thu, 30 Apr 2020 09:28:11 -0700 Message-Id: <20200430162813.17671-17-richard.henderson@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200430162813.17671-1-richard.henderson@linaro.org> References: <20200430162813.17671-1-richard.henderson@linaro.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2607:f8b0:4864:20::543; envelope-from=richard.henderson@linaro.org; helo=mail-pg1-x543.google.com X-detected-operating-system: by eggs.gnu.org: Error: [-] PROGRAM ABORT : Malformed IPv6 address (bad octet value). Location : parse_addr6(), p0f-client.c:67 X-Received-From: 2607:f8b0:4864:20::543 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, alex.bennee@linaro.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @linaro.org) Content-Type: text/plain; charset="utf-8" Reviewed-by: Peter Maydell Signed-off-by: Richard Henderson --- target/arm/sve_helper.c | 182 ++++++++++++++++++++++++---------------- 1 file changed, 111 insertions(+), 71 deletions(-) diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index 1560129b08..ad7e10f1e7 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -5413,94 +5413,134 @@ DO_LDFF1_ZPZ_D(dd_be, zd, MO_64) =20 /* Stores with a vector index. */ =20 -static void sve_st1_zs(CPUARMState *env, void *vd, void *vg, void *vm, - target_ulong base, uint32_t desc, uintptr_t ra, - zreg_off_fn *off_fn, sve_ldst1_tlb_fn *tlb_fn) +static inline QEMU_ALWAYS_INLINE +void sve_st1_z(CPUARMState *env, void *vd, uint64_t *vg, void *vm, + target_ulong base, uint32_t desc, uintptr_t retaddr, + int esize, int msize, zreg_off_fn *off_fn, + sve_ldst1_host_fn *host_fn, + sve_ldst1_tlb_fn *tlb_fn) { const int scale =3D extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, = 2); - intptr_t i, oprsz =3D simd_oprsz(desc); + const int mmu_idx =3D cpu_mmu_index(env, false); + const intptr_t reg_max =3D simd_oprsz(desc); + void *host[ARM_MAX_VQ * 4]; + intptr_t reg_off, i; + SVEHostPage info, info2; =20 - for (i =3D 0; i < oprsz; ) { - uint16_t pg =3D *(uint16_t *)(vg + H1_2(i >> 3)); + /* + * Probe all of the elements for host addresses and flags. + */ + i =3D reg_off =3D 0; + do { + uint64_t pg =3D vg[reg_off >> 6]; do { - if (likely(pg & 1)) { - target_ulong off =3D off_fn(vm, i); - tlb_fn(env, vd, i, base + (off << scale), ra); + target_ulong addr =3D base + (off_fn(vm, reg_off) << scale); + target_ulong in_page =3D -(addr | TARGET_PAGE_MASK); + + host[i] =3D NULL; + if (likely((pg >> (reg_off & 63)) & 1)) { + if (likely(in_page >=3D msize)) { + sve_probe_page(&info, false, env, addr, 0, MMU_DATA_ST= ORE, + mmu_idx, retaddr); + host[i] =3D info.host; + } else { + /* + * Element crosses the page boundary. + * Probe both pages, but do not record the host addres= s, + * so that we use the slow path. + */ + sve_probe_page(&info, false, env, addr, 0, + MMU_DATA_STORE, mmu_idx, retaddr); + sve_probe_page(&info2, false, env, addr + in_page, 0, + MMU_DATA_STORE, mmu_idx, retaddr); + info.flags |=3D info2.flags; + } + + if (unlikely(info.flags & TLB_WATCHPOINT)) { + cpu_check_watchpoint(env_cpu(env), addr, msize, + info.attrs, BP_MEM_WRITE, retaddr= ); + } + /* TODO: MTE check. */ } - i +=3D 4, pg >>=3D 4; - } while (i & 15); - } -} + i +=3D 1; + reg_off +=3D esize; + } while (reg_off & 63); + } while (reg_off < reg_max); =20 -static void sve_st1_zd(CPUARMState *env, void *vd, void *vg, void *vm, - target_ulong base, uint32_t desc, uintptr_t ra, - zreg_off_fn *off_fn, sve_ldst1_tlb_fn *tlb_fn) -{ - const int scale =3D extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, = 2); - intptr_t i, oprsz =3D simd_oprsz(desc) / 8; - - for (i =3D 0; i < oprsz; i++) { - uint8_t pg =3D *(uint8_t *)(vg + H1(i)); - if (likely(pg & 1)) { - target_ulong off =3D off_fn(vm, i * 8); - tlb_fn(env, vd, i * 8, base + (off << scale), ra); + /* + * Now that we have recognized all exceptions except SyncExternal + * (from TLB_MMIO), which we cannot avoid, perform all of the stores. + * + * Note for the common case of an element in RAM, not crossing a page + * boundary, we have stored the host address in host[]. This doubles + * as a first-level check against the predicate, since only enabled + * elements have non-null host addresses. + */ + i =3D reg_off =3D 0; + do { + void *h =3D host[i]; + if (likely(h !=3D NULL)) { + host_fn(vd, reg_off, h); + } else if ((vg[reg_off >> 6] >> (reg_off & 63)) & 1) { + target_ulong addr =3D base + (off_fn(vm, reg_off) << scale); + tlb_fn(env, vd, reg_off, addr, retaddr); } - } + i +=3D 1; + reg_off +=3D esize; + } while (reg_off < reg_max); } =20 -#define DO_ST1_ZPZ_S(MEM, OFS) \ -void QEMU_FLATTEN HELPER(sve_st##MEM##_##OFS) \ - (CPUARMState *env, void *vd, void *vg, void *vm, \ - target_ulong base, uint32_t desc) \ -{ \ - sve_st1_zs(env, vd, vg, vm, base, desc, GETPC(), \ - off_##OFS##_s, sve_st1##MEM##_tlb); \ +#define DO_ST1_ZPZ_S(MEM, OFS, MSZ) \ +void HELPER(sve_st##MEM##_##OFS)(CPUARMState *env, void *vd, void *vg, = \ + void *vm, target_ulong base, uint32_t des= c) \ +{ = \ + sve_st1_z(env, vd, vg, vm, base, desc, GETPC(), 4, 1 << MSZ, = \ + off_##OFS##_s, sve_st1##MEM##_host, sve_st1##MEM##_tlb); = \ } =20 -#define DO_ST1_ZPZ_D(MEM, OFS) \ -void QEMU_FLATTEN HELPER(sve_st##MEM##_##OFS) \ - (CPUARMState *env, void *vd, void *vg, void *vm, \ - target_ulong base, uint32_t desc) \ -{ \ - sve_st1_zd(env, vd, vg, vm, base, desc, GETPC(), \ - off_##OFS##_d, sve_st1##MEM##_tlb); \ +#define DO_ST1_ZPZ_D(MEM, OFS, MSZ) \ +void HELPER(sve_st##MEM##_##OFS)(CPUARMState *env, void *vd, void *vg, = \ + void *vm, target_ulong base, uint32_t des= c) \ +{ = \ + sve_st1_z(env, vd, vg, vm, base, desc, GETPC(), 8, 1 << MSZ, = \ + off_##OFS##_d, sve_st1##MEM##_host, sve_st1##MEM##_tlb); = \ } =20 -DO_ST1_ZPZ_S(bs, zsu) -DO_ST1_ZPZ_S(hs_le, zsu) -DO_ST1_ZPZ_S(hs_be, zsu) -DO_ST1_ZPZ_S(ss_le, zsu) -DO_ST1_ZPZ_S(ss_be, zsu) +DO_ST1_ZPZ_S(bs, zsu, MO_8) +DO_ST1_ZPZ_S(hs_le, zsu, MO_16) +DO_ST1_ZPZ_S(hs_be, zsu, MO_16) +DO_ST1_ZPZ_S(ss_le, zsu, MO_32) +DO_ST1_ZPZ_S(ss_be, zsu, MO_32) =20 -DO_ST1_ZPZ_S(bs, zss) -DO_ST1_ZPZ_S(hs_le, zss) -DO_ST1_ZPZ_S(hs_be, zss) -DO_ST1_ZPZ_S(ss_le, zss) -DO_ST1_ZPZ_S(ss_be, zss) +DO_ST1_ZPZ_S(bs, zss, MO_8) +DO_ST1_ZPZ_S(hs_le, zss, MO_16) +DO_ST1_ZPZ_S(hs_be, zss, MO_16) +DO_ST1_ZPZ_S(ss_le, zss, MO_32) +DO_ST1_ZPZ_S(ss_be, zss, MO_32) =20 -DO_ST1_ZPZ_D(bd, zsu) -DO_ST1_ZPZ_D(hd_le, zsu) -DO_ST1_ZPZ_D(hd_be, zsu) -DO_ST1_ZPZ_D(sd_le, zsu) -DO_ST1_ZPZ_D(sd_be, zsu) -DO_ST1_ZPZ_D(dd_le, zsu) -DO_ST1_ZPZ_D(dd_be, zsu) +DO_ST1_ZPZ_D(bd, zsu, MO_8) +DO_ST1_ZPZ_D(hd_le, zsu, MO_16) +DO_ST1_ZPZ_D(hd_be, zsu, MO_16) +DO_ST1_ZPZ_D(sd_le, zsu, MO_32) +DO_ST1_ZPZ_D(sd_be, zsu, MO_32) +DO_ST1_ZPZ_D(dd_le, zsu, MO_64) +DO_ST1_ZPZ_D(dd_be, zsu, MO_64) =20 -DO_ST1_ZPZ_D(bd, zss) -DO_ST1_ZPZ_D(hd_le, zss) -DO_ST1_ZPZ_D(hd_be, zss) -DO_ST1_ZPZ_D(sd_le, zss) -DO_ST1_ZPZ_D(sd_be, zss) -DO_ST1_ZPZ_D(dd_le, zss) -DO_ST1_ZPZ_D(dd_be, zss) +DO_ST1_ZPZ_D(bd, zss, MO_8) +DO_ST1_ZPZ_D(hd_le, zss, MO_16) +DO_ST1_ZPZ_D(hd_be, zss, MO_16) +DO_ST1_ZPZ_D(sd_le, zss, MO_32) +DO_ST1_ZPZ_D(sd_be, zss, MO_32) +DO_ST1_ZPZ_D(dd_le, zss, MO_64) +DO_ST1_ZPZ_D(dd_be, zss, MO_64) =20 -DO_ST1_ZPZ_D(bd, zd) -DO_ST1_ZPZ_D(hd_le, zd) -DO_ST1_ZPZ_D(hd_be, zd) -DO_ST1_ZPZ_D(sd_le, zd) -DO_ST1_ZPZ_D(sd_be, zd) -DO_ST1_ZPZ_D(dd_le, zd) -DO_ST1_ZPZ_D(dd_be, zd) +DO_ST1_ZPZ_D(bd, zd, MO_8) +DO_ST1_ZPZ_D(hd_le, zd, MO_16) +DO_ST1_ZPZ_D(hd_be, zd, MO_16) +DO_ST1_ZPZ_D(sd_le, zd, MO_32) +DO_ST1_ZPZ_D(sd_be, zd, MO_32) +DO_ST1_ZPZ_D(dd_le, zd, MO_64) +DO_ST1_ZPZ_D(dd_be, zd, MO_64) =20 #undef DO_ST1_ZPZ_S #undef DO_ST1_ZPZ_D --=20 2.20.1 From nobody Sun May 19 14:32:52 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=linaro.org ARC-Seal: i=1; a=rsa-sha256; t=1588265231; cv=none; d=zohomail.com; s=zohoarc; b=Y2S/5VIhTfwYHkxQCehMsufNrldi8Vyn5/dkalZIuFMadpJRudbikqKpY8SxRDklSRVDjYA/mW4AYL/FEtn71XOdR2byZpAde/gLTYUhr3V9NIvdFXKnTaNZwFmc6PT1zt9mxCJYrbJ1OtQ8oa1om7tUb9Z22Uu9R9Rmwucs+9M= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1588265231; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=i+hk2OIEePTNyeNKvZxmoRr45fnk+O5bEX7jWkmNbw0=; b=DzpIa2Os5Ga1FRVrvf0QERg0ZLmYyfPmEDrINJ2Uk12UykDLZxte3QTgpcBSGsnrt/sGgBA818aPKH2Cy7NKoUyGIPZRNHxtf6QGSgPVeSE7PGWZlfc8R2wV5OUErWwQlOr0xgVSwyaZYaRNRsJsyb73oMDAGtSMsw/bV76BvWg= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1588265231211832.5121878392766; Thu, 30 Apr 2020 09:47:11 -0700 (PDT) Received: from localhost ([::1]:58950 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jUCL7-0000WV-RF for importer@patchew.org; Thu, 30 Apr 2020 12:47:09 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:49452) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jUC3O-00020W-61 for qemu-devel@nongnu.org; Thu, 30 Apr 2020 12:28:56 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.90_1) (envelope-from ) id 1jUC3E-0006qm-Mo for qemu-devel@nongnu.org; Thu, 30 Apr 2020 12:28:49 -0400 Received: from mail-pj1-x1043.google.com ([2607:f8b0:4864:20::1043]:36153) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jUC3E-0006qR-6A for qemu-devel@nongnu.org; Thu, 30 Apr 2020 12:28:40 -0400 Received: by mail-pj1-x1043.google.com with SMTP id a31so914845pje.1 for ; Thu, 30 Apr 2020 09:28:39 -0700 (PDT) Received: from localhost.localdomain (174-21-149-226.tukw.qwest.net. [174.21.149.226]) by smtp.gmail.com with ESMTPSA id j5sm243514pfh.58.2020.04.30.09.28.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Apr 2020 09:28:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=i+hk2OIEePTNyeNKvZxmoRr45fnk+O5bEX7jWkmNbw0=; b=wh3bgOw+E125XWhq+8BN71456+GH9Rbko7UJJ09fr2PTLZPwfQYzntn8f2t/f0gOjc G4P6T1EU3SfIWsHqlFdjcf9X8EtLCOVywl2GyOHtIvAX8VRrb5B/b2zeRD/JhA12sdeZ TLEd5ZbxSLFDf8ZyEZYYbAfiTev4BPYohayG0+Y3b47cQaoGtK65ENq0cGg8WE7HXbsD RCYXFKu1jIMY/EaudAdd3Ot0YrmyuAAJ8Dj5ywE68gMzX4BW+tyciMeDd/9IftzP96SB A8WtVsdSepZ/1EXJhXskSdspGAx+1btA7if4JeqV6RUsoNxn5UpcvsTVsqnd8PlKsLZE h0xA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=i+hk2OIEePTNyeNKvZxmoRr45fnk+O5bEX7jWkmNbw0=; b=Nn13s6i4qsy3uabFY61j0s1ZSMEO93ak2rkXLP0RZpgSVzP8g/gF7dKRIElJtA5/hK RO5Mk1EPUBbR1hBdEZhVZ4FVmEoS24TdgUys28TFGBoYq/3Vp/kyI4vBgs6R47itjkYN M7xKH+cWLdwJOVKSVob3/XBDrF4b3KLNBL+wt+bzf8hlMhLdclI8ahzJvGIpaVHgyyPp a2ZN/wlYFZw3ah3CIMQJl5z5kpmqZQO6xZsz2PbBvrQB+wWV5Aw/9AJawariXypr4qbk VUi2PiTKaYMLKC2z9oDDWQaFhUd9SSzkifszKXh459HwUfvcB07ckOsOyJaEf9L3kp+K 311A== X-Gm-Message-State: AGi0Pua0FndM5SB92yJaNgwiaG2klaW76FOq13M+WHdG30nsgDkJb9bC 8dSRXn92/wEAxA1aY6JS3NchVcou5dU= X-Google-Smtp-Source: APiQypLBRo9sxcrI9FWxPQS34+2RIR6bHmmLhwU8eG3FtLCLu6LBVk4ZfDIIodxGLDdtBX9EB5tVUA== X-Received: by 2002:a17:90a:9a89:: with SMTP id e9mr4037189pjp.108.1588264118519; Thu, 30 Apr 2020 09:28:38 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH v4 17/18] target/arm: Reuse sve_probe_page for gather loads Date: Thu, 30 Apr 2020 09:28:12 -0700 Message-Id: <20200430162813.17671-18-richard.henderson@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200430162813.17671-1-richard.henderson@linaro.org> References: <20200430162813.17671-1-richard.henderson@linaro.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2607:f8b0:4864:20::1043; envelope-from=richard.henderson@linaro.org; helo=mail-pj1-x1043.google.com X-detected-operating-system: by eggs.gnu.org: Error: [-] PROGRAM ABORT : Malformed IPv6 address (bad octet value). Location : parse_addr6(), p0f-client.c:67 X-Received-From: 2607:f8b0:4864:20::1043 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, alex.bennee@linaro.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @linaro.org) Content-Type: text/plain; charset="utf-8" Reviewed-by: Peter Maydell Signed-off-by: Richard Henderson --- target/arm/sve_helper.c | 208 +++++++++++++++++++++------------------- 1 file changed, 109 insertions(+), 99 deletions(-) diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index ad7e10f1e7..f1870aabc2 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -5124,130 +5124,140 @@ static target_ulong off_zd_d(void *reg, intptr_t = reg_ofs) return *(uint64_t *)(reg + reg_ofs); } =20 -static void sve_ld1_zs(CPUARMState *env, void *vd, void *vg, void *vm, - target_ulong base, uint32_t desc, uintptr_t ra, - zreg_off_fn *off_fn, sve_ldst1_tlb_fn *tlb_fn) +static inline QEMU_ALWAYS_INLINE +void sve_ld1_z(CPUARMState *env, void *vd, uint64_t *vg, void *vm, + target_ulong base, uint32_t desc, uintptr_t retaddr, + int esize, int msize, zreg_off_fn *off_fn, + sve_ldst1_host_fn *host_fn, + sve_ldst1_tlb_fn *tlb_fn) { const int scale =3D extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, = 2); - intptr_t i, oprsz =3D simd_oprsz(desc); - ARMVectorReg scratch =3D { }; + const int mmu_idx =3D cpu_mmu_index(env, false); + const intptr_t reg_max =3D simd_oprsz(desc); + ARMVectorReg scratch; + intptr_t reg_off; + SVEHostPage info, info2; =20 - for (i =3D 0; i < oprsz; ) { - uint16_t pg =3D *(uint16_t *)(vg + H1_2(i >> 3)); + memset(&scratch, 0, reg_max); + reg_off =3D 0; + do { + uint64_t pg =3D vg[reg_off >> 6]; do { if (likely(pg & 1)) { - target_ulong off =3D off_fn(vm, i); - tlb_fn(env, &scratch, i, base + (off << scale), ra); + target_ulong addr =3D base + (off_fn(vm, reg_off) << scale= ); + target_ulong in_page =3D -(addr | TARGET_PAGE_MASK); + + sve_probe_page(&info, false, env, addr, 0, MMU_DATA_LOAD, + mmu_idx, retaddr); + + if (likely(in_page >=3D msize)) { + if (unlikely(info.flags & TLB_WATCHPOINT)) { + cpu_check_watchpoint(env_cpu(env), addr, msize, + info.attrs, BP_MEM_READ, reta= ddr); + } + /* TODO: MTE check */ + host_fn(&scratch, reg_off, info.host); + } else { + /* Element crosses the page boundary. */ + sve_probe_page(&info2, false, env, addr + in_page, 0, + MMU_DATA_LOAD, mmu_idx, retaddr); + if (unlikely((info.flags | info2.flags) & TLB_WATCHPOI= NT)) { + cpu_check_watchpoint(env_cpu(env), addr, + msize, info.attrs, + BP_MEM_READ, retaddr); + } + /* TODO: MTE check */ + tlb_fn(env, &scratch, reg_off, addr, retaddr); + } } - i +=3D 4, pg >>=3D 4; - } while (i & 15); - } + reg_off +=3D esize; + pg >>=3D esize; + } while (reg_off & 63); + } while (reg_off < reg_max); =20 /* Wait until all exceptions have been raised to write back. */ - memcpy(vd, &scratch, oprsz); + memcpy(vd, &scratch, reg_max); } =20 -static void sve_ld1_zd(CPUARMState *env, void *vd, void *vg, void *vm, - target_ulong base, uint32_t desc, uintptr_t ra, - zreg_off_fn *off_fn, sve_ldst1_tlb_fn *tlb_fn) -{ - const int scale =3D extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, = 2); - intptr_t i, oprsz =3D simd_oprsz(desc) / 8; - ARMVectorReg scratch =3D { }; - - for (i =3D 0; i < oprsz; i++) { - uint8_t pg =3D *(uint8_t *)(vg + H1(i)); - if (likely(pg & 1)) { - target_ulong off =3D off_fn(vm, i * 8); - tlb_fn(env, &scratch, i * 8, base + (off << scale), ra); - } - } - - /* Wait until all exceptions have been raised to write back. */ - memcpy(vd, &scratch, oprsz * 8); +#define DO_LD1_ZPZ_S(MEM, OFS, MSZ) \ +void HELPER(sve_ld##MEM##_##OFS)(CPUARMState *env, void *vd, void *vg, = \ + void *vm, target_ulong base, uint32_t des= c) \ +{ = \ + sve_ld1_z(env, vd, vg, vm, base, desc, GETPC(), 4, 1 << MSZ, = \ + off_##OFS##_s, sve_ld1##MEM##_host, sve_ld1##MEM##_tlb); = \ } =20 -#define DO_LD1_ZPZ_S(MEM, OFS) \ -void QEMU_FLATTEN HELPER(sve_ld##MEM##_##OFS) \ - (CPUARMState *env, void *vd, void *vg, void *vm, \ - target_ulong base, uint32_t desc) \ -{ \ - sve_ld1_zs(env, vd, vg, vm, base, desc, GETPC(), \ - off_##OFS##_s, sve_ld1##MEM##_tlb); \ +#define DO_LD1_ZPZ_D(MEM, OFS, MSZ) \ +void HELPER(sve_ld##MEM##_##OFS)(CPUARMState *env, void *vd, void *vg, = \ + void *vm, target_ulong base, uint32_t des= c) \ +{ = \ + sve_ld1_z(env, vd, vg, vm, base, desc, GETPC(), 8, 1 << MSZ, = \ + off_##OFS##_d, sve_ld1##MEM##_host, sve_ld1##MEM##_tlb); = \ } =20 -#define DO_LD1_ZPZ_D(MEM, OFS) \ -void QEMU_FLATTEN HELPER(sve_ld##MEM##_##OFS) \ - (CPUARMState *env, void *vd, void *vg, void *vm, \ - target_ulong base, uint32_t desc) \ -{ \ - sve_ld1_zd(env, vd, vg, vm, base, desc, GETPC(), \ - off_##OFS##_d, sve_ld1##MEM##_tlb); \ -} +DO_LD1_ZPZ_S(bsu, zsu, MO_8) +DO_LD1_ZPZ_S(bsu, zss, MO_8) +DO_LD1_ZPZ_D(bdu, zsu, MO_8) +DO_LD1_ZPZ_D(bdu, zss, MO_8) +DO_LD1_ZPZ_D(bdu, zd, MO_8) =20 -DO_LD1_ZPZ_S(bsu, zsu) -DO_LD1_ZPZ_S(bsu, zss) -DO_LD1_ZPZ_D(bdu, zsu) -DO_LD1_ZPZ_D(bdu, zss) -DO_LD1_ZPZ_D(bdu, zd) +DO_LD1_ZPZ_S(bss, zsu, MO_8) +DO_LD1_ZPZ_S(bss, zss, MO_8) +DO_LD1_ZPZ_D(bds, zsu, MO_8) +DO_LD1_ZPZ_D(bds, zss, MO_8) +DO_LD1_ZPZ_D(bds, zd, MO_8) =20 -DO_LD1_ZPZ_S(bss, zsu) -DO_LD1_ZPZ_S(bss, zss) -DO_LD1_ZPZ_D(bds, zsu) -DO_LD1_ZPZ_D(bds, zss) -DO_LD1_ZPZ_D(bds, zd) +DO_LD1_ZPZ_S(hsu_le, zsu, MO_16) +DO_LD1_ZPZ_S(hsu_le, zss, MO_16) +DO_LD1_ZPZ_D(hdu_le, zsu, MO_16) +DO_LD1_ZPZ_D(hdu_le, zss, MO_16) +DO_LD1_ZPZ_D(hdu_le, zd, MO_16) =20 -DO_LD1_ZPZ_S(hsu_le, zsu) -DO_LD1_ZPZ_S(hsu_le, zss) -DO_LD1_ZPZ_D(hdu_le, zsu) -DO_LD1_ZPZ_D(hdu_le, zss) -DO_LD1_ZPZ_D(hdu_le, zd) +DO_LD1_ZPZ_S(hsu_be, zsu, MO_16) +DO_LD1_ZPZ_S(hsu_be, zss, MO_16) +DO_LD1_ZPZ_D(hdu_be, zsu, MO_16) +DO_LD1_ZPZ_D(hdu_be, zss, MO_16) +DO_LD1_ZPZ_D(hdu_be, zd, MO_16) =20 -DO_LD1_ZPZ_S(hsu_be, zsu) -DO_LD1_ZPZ_S(hsu_be, zss) -DO_LD1_ZPZ_D(hdu_be, zsu) -DO_LD1_ZPZ_D(hdu_be, zss) -DO_LD1_ZPZ_D(hdu_be, zd) +DO_LD1_ZPZ_S(hss_le, zsu, MO_16) +DO_LD1_ZPZ_S(hss_le, zss, MO_16) +DO_LD1_ZPZ_D(hds_le, zsu, MO_16) +DO_LD1_ZPZ_D(hds_le, zss, MO_16) +DO_LD1_ZPZ_D(hds_le, zd, MO_16) =20 -DO_LD1_ZPZ_S(hss_le, zsu) -DO_LD1_ZPZ_S(hss_le, zss) -DO_LD1_ZPZ_D(hds_le, zsu) -DO_LD1_ZPZ_D(hds_le, zss) -DO_LD1_ZPZ_D(hds_le, zd) +DO_LD1_ZPZ_S(hss_be, zsu, MO_16) +DO_LD1_ZPZ_S(hss_be, zss, MO_16) +DO_LD1_ZPZ_D(hds_be, zsu, MO_16) +DO_LD1_ZPZ_D(hds_be, zss, MO_16) +DO_LD1_ZPZ_D(hds_be, zd, MO_16) =20 -DO_LD1_ZPZ_S(hss_be, zsu) -DO_LD1_ZPZ_S(hss_be, zss) -DO_LD1_ZPZ_D(hds_be, zsu) -DO_LD1_ZPZ_D(hds_be, zss) -DO_LD1_ZPZ_D(hds_be, zd) +DO_LD1_ZPZ_S(ss_le, zsu, MO_32) +DO_LD1_ZPZ_S(ss_le, zss, MO_32) +DO_LD1_ZPZ_D(sdu_le, zsu, MO_32) +DO_LD1_ZPZ_D(sdu_le, zss, MO_32) +DO_LD1_ZPZ_D(sdu_le, zd, MO_32) =20 -DO_LD1_ZPZ_S(ss_le, zsu) -DO_LD1_ZPZ_S(ss_le, zss) -DO_LD1_ZPZ_D(sdu_le, zsu) -DO_LD1_ZPZ_D(sdu_le, zss) -DO_LD1_ZPZ_D(sdu_le, zd) +DO_LD1_ZPZ_S(ss_be, zsu, MO_32) +DO_LD1_ZPZ_S(ss_be, zss, MO_32) +DO_LD1_ZPZ_D(sdu_be, zsu, MO_32) +DO_LD1_ZPZ_D(sdu_be, zss, MO_32) +DO_LD1_ZPZ_D(sdu_be, zd, MO_32) =20 -DO_LD1_ZPZ_S(ss_be, zsu) -DO_LD1_ZPZ_S(ss_be, zss) -DO_LD1_ZPZ_D(sdu_be, zsu) -DO_LD1_ZPZ_D(sdu_be, zss) -DO_LD1_ZPZ_D(sdu_be, zd) +DO_LD1_ZPZ_D(sds_le, zsu, MO_32) +DO_LD1_ZPZ_D(sds_le, zss, MO_32) +DO_LD1_ZPZ_D(sds_le, zd, MO_32) =20 -DO_LD1_ZPZ_D(sds_le, zsu) -DO_LD1_ZPZ_D(sds_le, zss) -DO_LD1_ZPZ_D(sds_le, zd) +DO_LD1_ZPZ_D(sds_be, zsu, MO_32) +DO_LD1_ZPZ_D(sds_be, zss, MO_32) +DO_LD1_ZPZ_D(sds_be, zd, MO_32) =20 -DO_LD1_ZPZ_D(sds_be, zsu) -DO_LD1_ZPZ_D(sds_be, zss) -DO_LD1_ZPZ_D(sds_be, zd) +DO_LD1_ZPZ_D(dd_le, zsu, MO_64) +DO_LD1_ZPZ_D(dd_le, zss, MO_64) +DO_LD1_ZPZ_D(dd_le, zd, MO_64) =20 -DO_LD1_ZPZ_D(dd_le, zsu) -DO_LD1_ZPZ_D(dd_le, zss) -DO_LD1_ZPZ_D(dd_le, zd) - -DO_LD1_ZPZ_D(dd_be, zsu) -DO_LD1_ZPZ_D(dd_be, zss) -DO_LD1_ZPZ_D(dd_be, zd) +DO_LD1_ZPZ_D(dd_be, zsu, MO_64) +DO_LD1_ZPZ_D(dd_be, zss, MO_64) +DO_LD1_ZPZ_D(dd_be, zd, MO_64) =20 #undef DO_LD1_ZPZ_S #undef DO_LD1_ZPZ_D --=20 2.20.1 From nobody Sun May 19 14:32:52 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=linaro.org ARC-Seal: i=1; a=rsa-sha256; t=1588265546; cv=none; d=zohomail.com; s=zohoarc; b=ajxPpRW1ncmzBOww4uRo+NsjTlJe6Ugq3P+F15NGUM/wyUHwIGoHhCmvYKVEatkAgUH7uRL1RAc26N9dmyf0eJeC9nF7U0BWfOC78tjrdEhYDGlStiyEzcZgleWVMV2mp+Gz0DtbFCJW88mHd9ZSdUqUt6Hz0hyQKXtjGlltONE= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1588265546; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=H4Z2I014pBK6BpW2g3z5ROXLQbDXVrrB931WRgoDLnc=; b=asGo/Myz/NmSH7yyW5yIpL9FkUM4aaal7bx9EXGwlttCQmMa9hrttFIvBoQ7ZdKpvnf2Z3GkiRBO50cleVKtxzmF6mTiJO8OBUGGBWIbca26m31SGLKfL48AL7bQ1wTDRv96BSe5vv9QgMZV1grvNvA+meJJzvKuwPXfSv2cGMs= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1588265546718438.0579915460262; Thu, 30 Apr 2020 09:52:26 -0700 (PDT) Received: from localhost ([::1]:48016 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jUCQD-0008W6-DF for importer@patchew.org; Thu, 30 Apr 2020 12:52:25 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:49458) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jUC3P-00023P-Vq for qemu-devel@nongnu.org; Thu, 30 Apr 2020 12:28:56 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.90_1) (envelope-from ) id 1jUC3G-0006v8-4T for qemu-devel@nongnu.org; Thu, 30 Apr 2020 12:28:51 -0400 Received: from mail-pj1-x1042.google.com ([2607:f8b0:4864:20::1042]:52035) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jUC3F-0006qr-N4 for qemu-devel@nongnu.org; Thu, 30 Apr 2020 12:28:41 -0400 Received: by mail-pj1-x1042.google.com with SMTP id mq3so909870pjb.1 for ; Thu, 30 Apr 2020 09:28:41 -0700 (PDT) Received: from localhost.localdomain (174-21-149-226.tukw.qwest.net. [174.21.149.226]) by smtp.gmail.com with ESMTPSA id j5sm243514pfh.58.2020.04.30.09.28.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Apr 2020 09:28:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=H4Z2I014pBK6BpW2g3z5ROXLQbDXVrrB931WRgoDLnc=; b=chhd+rOxagPotU6qSMFkEH7xpp7DjttF2OnWsBs/ahR90xqoHNn1LRbRvEXz0fcHLa Nw/SfqTqp/3SVFK0MqsS+qyE44TEjLocC9RYBPlq/yCTGtITKjcmnmIFBUh1mojSCbIV mCaC5bPbwHHhftCYAE7XeKItba408eTIjojyUr1798T3sCx5xp/EDzuzL8bLkSEtNDww klFF5Irj7VJ7A+mrwPIUoOi1B49icsLIXI7QdVJpv2x3AIeNFgEHvZr2dOXNQUP7ZYwF FbRB9nuKaremc6wZXnIINzzYMo3z87T2DJJQpPyylO0yoa8LOUCgOkFcb6Hg74wE0W/v n+2A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=H4Z2I014pBK6BpW2g3z5ROXLQbDXVrrB931WRgoDLnc=; b=E7Z6IwhW6P0ELk3nXqjjbrZgwfRMsoiuAu+yzeJ3Zp7WH3s+pv/YsLkGs32BuBJshe QM/QcCY+MkTtw766IoZJBkF2A5imjXJ9XTedRRAdOnvU9PGV2Ymbpwvb684CNqAuyMEg npAE7HPJGKr7+KJXeT7qgwiUX2bEiKKDecDzTZsrNbSu7blSgT/kkP1xq5NyBdZWH+sC kvVHttaYIFuHbhVJpTXEk3SkvuNv1COVwe+71cABpL8ISwa/2sTkAdq7UfE8LCsYfr/H P/dBFFRA6ylh1aGlSDqV6NPu4DdvLrfEZY9bYziaWBNvn1ZHBXgn+w0mlSIJ4Z9scFV9 ePlw== X-Gm-Message-State: AGi0PuZn8BW1l/8QMD0VMnqwPbCkQGU52leEzqKteRuLGrbcUZYMbbqU aqMhs/Gffjj5nM/LKiYAfDhrb6yfLzM= X-Google-Smtp-Source: APiQypJ0QtOF6u1xcTwCwB6bwoJGJ0I5AJ/PbgHlUEm+W6fo7johcPK2uHaB7SqHcaa/MzUNRV0cFw== X-Received: by 2002:a17:902:bb97:: with SMTP id m23mr3966150pls.253.1588264119985; Thu, 30 Apr 2020 09:28:39 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH v4 18/18] target/arm: Remove sve_memopidx Date: Thu, 30 Apr 2020 09:28:13 -0700 Message-Id: <20200430162813.17671-19-richard.henderson@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200430162813.17671-1-richard.henderson@linaro.org> References: <20200430162813.17671-1-richard.henderson@linaro.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2607:f8b0:4864:20::1042; envelope-from=richard.henderson@linaro.org; helo=mail-pj1-x1042.google.com X-detected-operating-system: by eggs.gnu.org: Error: [-] PROGRAM ABORT : Malformed IPv6 address (bad octet value). Location : parse_addr6(), p0f-client.c:67 X-Received-From: 2607:f8b0:4864:20::1042 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, alex.bennee@linaro.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @linaro.org) Content-Type: text/plain; charset="utf-8" None of the sve helpers use TCGMemOpIdx any longer, so we can stop passing it. Reviewed-by: Peter Maydell Signed-off-by: Richard Henderson --- target/arm/internals.h | 5 ----- target/arm/sve_helper.c | 14 +++++++------- target/arm/translate-sve.c | 17 +++-------------- 3 files changed, 10 insertions(+), 26 deletions(-) diff --git a/target/arm/internals.h b/target/arm/internals.h index e633aff36e..a833e3941d 100644 --- a/target/arm/internals.h +++ b/target/arm/internals.h @@ -979,11 +979,6 @@ static inline int arm_num_ctx_cmps(ARMCPU *cpu) } } =20 -/* Note make_memop_idx reserves 4 bits for mmu_idx, and MO_BSWAP is bit 3. - * Thus a TCGMemOpIdx, without any MO_ALIGN bits, fits in 8 bits. - */ -#define MEMOPIDX_SHIFT 8 - /** * v7m_using_psp: Return true if using process stack pointer * Return true if the CPU is currently using the process stack diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index f1870aabc2..116d535fa5 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -4440,7 +4440,7 @@ void sve_ldN_r(CPUARMState *env, uint64_t *vg, const = target_ulong addr, sve_ldst1_host_fn *host_fn, sve_ldst1_tlb_fn *tlb_fn) { - const unsigned rd =3D extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT= , 5); + const unsigned rd =3D simd_data(desc); const intptr_t reg_max =3D simd_oprsz(desc); intptr_t reg_off, reg_last, mem_off; SVEContLdSt info; @@ -4696,7 +4696,7 @@ void sve_ldnfff1_r(CPUARMState *env, void *vg, const = target_ulong addr, sve_ldst1_host_fn *host_fn, sve_ldst1_tlb_fn *tlb_fn) { - const unsigned rd =3D extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT= , 5); + const unsigned rd =3D simd_data(desc); void *vd =3D &env->vfp.zregs[rd]; const intptr_t reg_max =3D simd_oprsz(desc); intptr_t reg_off, mem_off, reg_last; @@ -4925,7 +4925,7 @@ void sve_stN_r(CPUARMState *env, uint64_t *vg, target= _ulong addr, uint32_t desc, sve_ldst1_host_fn *host_fn, sve_ldst1_tlb_fn *tlb_fn) { - const unsigned rd =3D extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT= , 5); + const unsigned rd =3D simd_data(desc); const intptr_t reg_max =3D simd_oprsz(desc); intptr_t reg_off, reg_last, mem_off; SVEContLdSt info; @@ -5131,9 +5131,9 @@ void sve_ld1_z(CPUARMState *env, void *vd, uint64_t *= vg, void *vm, sve_ldst1_host_fn *host_fn, sve_ldst1_tlb_fn *tlb_fn) { - const int scale =3D extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, = 2); const int mmu_idx =3D cpu_mmu_index(env, false); const intptr_t reg_max =3D simd_oprsz(desc); + const int scale =3D simd_data(desc); ARMVectorReg scratch; intptr_t reg_off; SVEHostPage info, info2; @@ -5276,10 +5276,10 @@ void sve_ldff1_z(CPUARMState *env, void *vd, uint64= _t *vg, void *vm, sve_ldst1_tlb_fn *tlb_fn) { const int mmu_idx =3D cpu_mmu_index(env, false); - const int scale =3D extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, = 2); + const intptr_t reg_max =3D simd_oprsz(desc); + const int scale =3D simd_data(desc); const int esize =3D 1 << esz; const int msize =3D 1 << msz; - const intptr_t reg_max =3D simd_oprsz(desc); intptr_t reg_off; SVEHostPage info; target_ulong addr, in_page; @@ -5430,9 +5430,9 @@ void sve_st1_z(CPUARMState *env, void *vd, uint64_t *= vg, void *vm, sve_ldst1_host_fn *host_fn, sve_ldst1_tlb_fn *tlb_fn) { - const int scale =3D extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, = 2); const int mmu_idx =3D cpu_mmu_index(env, false); const intptr_t reg_max =3D simd_oprsz(desc); + const int scale =3D simd_data(desc); void *host[ARM_MAX_VQ * 4]; intptr_t reg_off, i; SVEHostPage info, info2; diff --git a/target/arm/translate-sve.c b/target/arm/translate-sve.c index b35bad245e..7bd7de80e6 100644 --- a/target/arm/translate-sve.c +++ b/target/arm/translate-sve.c @@ -4584,11 +4584,6 @@ static const uint8_t dtype_esz[16] =3D { 3, 2, 1, 3 }; =20 -static TCGMemOpIdx sve_memopidx(DisasContext *s, int dtype) -{ - return make_memop_idx(s->be_data | dtype_mop[dtype], get_mem_index(s)); -} - static void do_mem_zpa(DisasContext *s, int zt, int pg, TCGv_i64 addr, int dtype, gen_helper_gvec_mem *fn) { @@ -4601,9 +4596,7 @@ static void do_mem_zpa(DisasContext *s, int zt, int p= g, TCGv_i64 addr, * registers as pointers, so encode the regno into the data field. * For consistency, do this even for LD1. */ - desc =3D sve_memopidx(s, dtype); - desc |=3D zt << MEMOPIDX_SHIFT; - desc =3D simd_desc(vsz, vsz, desc); + desc =3D simd_desc(vsz, vsz, zt); t_desc =3D tcg_const_i32(desc); t_pg =3D tcg_temp_new_ptr(); =20 @@ -4835,9 +4828,7 @@ static void do_ldrq(DisasContext *s, int zt, int pg, = TCGv_i64 addr, int msz) int desc, poff; =20 /* Load the first quadword using the normal predicated load helpers. = */ - desc =3D sve_memopidx(s, msz_dtype(s, msz)); - desc |=3D zt << MEMOPIDX_SHIFT; - desc =3D simd_desc(16, 16, desc); + desc =3D simd_desc(16, 16, zt); t_desc =3D tcg_const_i32(desc); =20 poff =3D pred_full_reg_offset(s, pg); @@ -5066,9 +5057,7 @@ static void do_mem_zpz(DisasContext *s, int zt, int p= g, int zm, TCGv_i32 t_desc; int desc; =20 - desc =3D sve_memopidx(s, msz_dtype(s, msz)); - desc |=3D scale << MEMOPIDX_SHIFT; - desc =3D simd_desc(vsz, vsz, desc); + desc =3D simd_desc(vsz, vsz, scale); t_desc =3D tcg_const_i32(desc); =20 tcg_gen_addi_ptr(t_pg, cpu_env, pred_full_reg_offset(s, pg)); --=20 2.20.1