From nobody Mon Sep 16 19:08:48 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=reject dis=none) header.from=sifive.com ARC-Seal: i=1; a=rsa-sha256; t=1721223757; cv=none; d=zohomail.com; s=zohoarc; b=e0XRi227Uxqe615h45oxceItnlIhM7w7Op1k5pp0n+jxEwANrU9siQSiDNGiB6bytFxkkiJYu/94nbGjZ08ZVIs5qAGVyr8hAsVA4fSwEOuAK/c+SYCeWDH6wh2GHV6/gzinmxXGsM3RcerAO3WVRiaruJKqCg9lxM/4TBgwSbE= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1721223757; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=Nfs9IDqgQpVVigMZU11K8UdOmxP2QGM7nTxJBjJv74w=; b=XfrAXb8+dKpt91PMKRQGZucVkp+/CzdvzBTXA5Y6guR/HChitKfRct/yRWCz3D6zFXpt7JuzOkARpOCOl1Cc0lEg99cy6PSHcd5drgLICT2ngqW4KKAadtxaVu5bo7jySYzQsEt0RQF3ZapUzm4a3Zp+mAPBEqo24za9GyutSOE= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=reject dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1721223757869308.2881243681769; Wed, 17 Jul 2024 06:42:37 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sU4ss-00007a-4g; Wed, 17 Jul 2024 09:39:54 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sU4sq-0008SM-Bj for qemu-devel@nongnu.org; Wed, 17 Jul 2024 09:39:52 -0400 Received: from mail-pl1-x633.google.com ([2607:f8b0:4864:20::633]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1sU4so-0007Qu-1L for qemu-devel@nongnu.org; Wed, 17 Jul 2024 09:39:51 -0400 Received: by mail-pl1-x633.google.com with SMTP id d9443c01a7336-1fbfb8e5e0cso48540445ad.0 for ; Wed, 17 Jul 2024 06:39:49 -0700 (PDT) Received: from duncan.localdomain (114-35-142-126.hinet-ip.hinet.net. [114.35.142.126]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-1fc0bc5aa6esm74903495ad.299.2024.07.17.06.39.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 17 Jul 2024 06:39:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1721223588; x=1721828388; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Nfs9IDqgQpVVigMZU11K8UdOmxP2QGM7nTxJBjJv74w=; b=h3hZhDEm8ps3HIcANJe0SMW043NsAhDZZgDAta6dcNhO/AD9Zy8r843UTUQs8tZ8n5 GOtQjPkftk7aH0Sh/HXTofZgDJ+lXA9nXXYvu1QjEoP3B7dDkTNFUd+87YIRIaYd+gRd dHJD2aU4UJW8lJyFgmUk4Jj0TPXC+cvbLOHZXjWfoKsNoMAwMzKMb/cJjPwR7Oa5q8C0 882C6yeKw2E8OI8ZeO4mXZqqLWWSdlUhEsp7MGd8q0TBtDMCq5JXHn2Q+2beEME+egkj yAVw75p05ZhM5+UXnVf0qvzlDBalNPsyZf0N/hR7kO1Jg5qvK44dykeB0lhQY08m3C44 B6vw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721223588; x=1721828388; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Nfs9IDqgQpVVigMZU11K8UdOmxP2QGM7nTxJBjJv74w=; b=GkL3mzVOxCS1gVE/bV5Hf6PK5vEOaZjm3ZoNjzWu0lJIN6AcMc4I928nJ5DkFEdRLQ 4FnFaCtQoNINheJgKeWgEMaG4RJO94OsmPNo3HODXC49au/6mECuLVoL9qF6Mgv3by6l yQCz4r5jmeeCnEh2cXnPWzZGiAcOYFFrFC80pf5QQEpSElNJyccUT9Wlh1RvNKyVv/EZ EHX8hQJKFVaUA3knvDCoBOWw/MM8biTLPwZAvUeX8bl99Cqn7ACw4SsCfqjd03iTXCC1 J39a8/dKAWcdwnr4WxxbYtZ/LC7v9P+xiUKnNCVzyOWqlf+T2s3r/gbQCzCLzSw2u37v VHaw== X-Gm-Message-State: AOJu0YyybwOdOymUYoHdjauRI58z+9mZ/imPJlqIE5ixdOc0zG+Ci+/k fzgq6XTp4SG3kbVZNEAhcsl5Y2EXgjnGutkFpt3bY4927/jD/jvB/F6t2W9epsyfBa6UwbKW3jU TkQzDFzWhNDIKfUTtJ16mBEy4r2IPuzOEwineYF9cz1rAINeDLupa70ny31t1KIzF6nvGijgWe7 7GAXT1PzRtXaDZ2+lgraR1Uzb6GGV4xXN7VRkgUA== X-Google-Smtp-Source: AGHT+IE8pc+hJ9pD+Tt81iKqmPfRNNbZNqlasHOE/UP9cJmMOu7HUIy004Z4Qo7CKRfgVJ67kMIDCA== X-Received: by 2002:a17:902:db01:b0:1fa:d3d:a68d with SMTP id d9443c01a7336-1fc4e17c7d9mr16685155ad.22.1721223588113; Wed, 17 Jul 2024 06:39:48 -0700 (PDT) From: Max Chou To: qemu-devel@nongnu.org, qemu-riscv@nongnu.org Cc: Palmer Dabbelt , Alistair Francis , Bin Meng , Weiwei Li , Daniel Henrique Barboza , Liu Zhiwei , richard.henderson@linaro.org, Max Chou Subject: [RFC PATCH v5 1/5] target/riscv: Set vdata.vm field for vector load/store whole register instructions Date: Wed, 17 Jul 2024 21:39:32 +0800 Message-Id: <20240717133936.713642-2-max.chou@sifive.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240717133936.713642-1-max.chou@sifive.com> References: <20240717133936.713642-1-max.chou@sifive.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2607:f8b0:4864:20::633; envelope-from=max.chou@sifive.com; helo=mail-pl1-x633.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @sifive.com) X-ZM-MESSAGEID: 1721223759585116600 Content-Type: text/plain; charset="utf-8" The vm field of the vector load/store whole register instruction's encoding is 1. The helper function of the vector load/store whole register instructions may need the vdata.vm field to do some optimizations. Signed-off-by: Max Chou --- target/riscv/insn_trans/trans_rvv.c.inc | 3 +++ 1 file changed, 3 insertions(+) diff --git a/target/riscv/insn_trans/trans_rvv.c.inc b/target/riscv/insn_tr= ans/trans_rvv.c.inc index 3a3896ba06c..14e10568bd7 100644 --- a/target/riscv/insn_trans/trans_rvv.c.inc +++ b/target/riscv/insn_trans/trans_rvv.c.inc @@ -770,6 +770,7 @@ static bool ld_us_mask_op(DisasContext *s, arg_vlm_v *a= , uint8_t eew) /* Mask destination register are always tail-agnostic */ data =3D FIELD_DP32(data, VDATA, VTA, s->cfg_vta_all_1s); data =3D FIELD_DP32(data, VDATA, VMA, s->vma); + data =3D FIELD_DP32(data, VDATA, VM, 1); return ldst_us_trans(a->rd, a->rs1, data, fn, s, false); } =20 @@ -787,6 +788,7 @@ static bool st_us_mask_op(DisasContext *s, arg_vsm_v *a= , uint8_t eew) /* EMUL =3D 1, NFIELDS =3D 1 */ data =3D FIELD_DP32(data, VDATA, LMUL, 0); data =3D FIELD_DP32(data, VDATA, NF, 1); + data =3D FIELD_DP32(data, VDATA, VM, 1); return ldst_us_trans(a->rd, a->rs1, data, fn, s, true); } =20 @@ -1106,6 +1108,7 @@ static bool ldst_whole_trans(uint32_t vd, uint32_t rs= 1, uint32_t nf, TCGv_i32 desc; =20 uint32_t data =3D FIELD_DP32(0, VDATA, NF, nf); + data =3D FIELD_DP32(data, VDATA, VM, 1); dest =3D tcg_temp_new_ptr(); desc =3D tcg_constant_i32(simd_desc(s->cfg_ptr->vlenb, s->cfg_ptr->vlenb, data)); --=20 2.34.1 From nobody Mon Sep 16 19:08:48 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=reject dis=none) header.from=sifive.com ARC-Seal: i=1; a=rsa-sha256; t=1721223630; cv=none; d=zohomail.com; s=zohoarc; b=hCh+Ohgg8ApHM2D42hXaoake42Ua0ZgHawT+M2lXdx/60jqnP16uNcmiGlV3tUn0GfBQNGCEIdRdE/bA2J5NKlt90WhfJPg4V+ZEgb8TKmsSmtCnAxXDlSzYqlFmvjBkRhQjS43XcrZrTCZMRennbxUw6pWqLKvP6N6pB0jtLBQ= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1721223630; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=pWJCUGOB3yGRYzMSOqJt//vIQjqu868mkDHHMDRR26Y=; b=SsTknOLGjQTWT8r2kx7mz8eTNnvGEGyFYrj/7mAkY0wIx9QwlmQONyY+QKHMEpPNbzpd5MoUO8yx5NwmOHwarJW3atg+NI4ZK2A85E6dueG450m6OnNX2Gxc6drBD0YexbxDkyZ0Z8KdyYXBx9Lw8HOgpbm14YNIi2bm/QmuMzM= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=reject dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1721223630611242.39784010779306; Wed, 17 Jul 2024 06:40:30 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sU4sw-0000Sr-BG; Wed, 17 Jul 2024 09:39:58 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sU4su-0000Mr-RN for qemu-devel@nongnu.org; Wed, 17 Jul 2024 09:39:56 -0400 Received: from mail-pl1-x636.google.com ([2607:f8b0:4864:20::636]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1sU4sr-0007SF-U6 for qemu-devel@nongnu.org; Wed, 17 Jul 2024 09:39:56 -0400 Received: by mail-pl1-x636.google.com with SMTP id d9443c01a7336-1fbfb8e5e0cso48541275ad.0 for ; Wed, 17 Jul 2024 06:39:52 -0700 (PDT) Received: from duncan.localdomain (114-35-142-126.hinet-ip.hinet.net. [114.35.142.126]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-1fc0bc5aa6esm74903495ad.299.2024.07.17.06.39.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 17 Jul 2024 06:39:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1721223592; x=1721828392; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=pWJCUGOB3yGRYzMSOqJt//vIQjqu868mkDHHMDRR26Y=; b=j0rixpThQNcgAEOd3qHYaKTaMppFvkiRg1i2qR/cEBzHREhwZ2DvmxKMZQeFQq0XcE H85oA81+Bphcw40d9Q/sqlgGiANzq0lM1DzChLMgKX4yAbDKiD2Nyvgpbg/1Fw8IChCd MqrBMi1yDhRoX8XnsnE/A7F1DvXrTPSfaAnRp0NCqs1pyPVMbAGYgRMPG4Dk/VNXpHlv wPfhW0E9PHrYqRGaFgN7IZgcob43WoujQuNjJ8yUwfVOEhOH2KmwnuJovguvI1tlU5m1 QKOKDU/Y0QcFdoREqFb1iL40r3douCWz/hjcC4NNjiDeUwBUcYYQ/UI984RnumVJFb3x VfZQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721223592; x=1721828392; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=pWJCUGOB3yGRYzMSOqJt//vIQjqu868mkDHHMDRR26Y=; b=gVItRw226onymlcZxBD9u5dL36ZSzJO/kgY/+n9OGcV20ufcDwPYCZvU5zKc3HNVh3 RThusTVKX/2L6Yah5LOFrsHhu27C0WWt4Vgq+Pvs7kOkCqyz0qQNhEuGZju2Wx0w3GKf zuRAP2LfYTougTg431OqInbiVsfCXlUCOd0w7FapVYXnDwPboY1QZKPrwockMLDXI0RG 8oK/6kTHzlhiPn9D8MrH+pyeedAAx7HSideRMWf5NnGL+HKDRfbociUaXLPpEv4f+G98 kjc+45qOc4mbXgQFMStUblOZ08HTAFCssZkC3ft1xYTTazt4mrTYeN/bWQ2FV1NDbolS MGZg== X-Gm-Message-State: AOJu0YxpKDaoNXLvkHioo60sOG8KzAkhag2yeIgIVzSza///Xac9OCcY 36pfgr1HFgX91xssHq1drvl2c5T1fCI3mXepxuLZrbt1SL294HrGGe+b6OOIzw677bm2od9ADaZ jYyOKlzKO+aIvfUFy7J1kjBKrNsALFek53LKW87ewEzobbEd/oRuSjWEfpBvoNoqLc4Lzxkh+1p CKbvAvAZf3VBV6wx22/KuWEctAEVI1Eow7o2O+cw== X-Google-Smtp-Source: AGHT+IHpatpH5R1Icul3xDqPaJDIAep+jodvKmSRn1CWTf/oPi4uCD2kBh8p252U7OxEV/Q3y9ZXoQ== X-Received: by 2002:a17:902:d2c6:b0:1fb:8e98:4468 with SMTP id d9443c01a7336-1fc4e6b6e3emr12186685ad.50.1721223591224; Wed, 17 Jul 2024 06:39:51 -0700 (PDT) From: Max Chou To: qemu-devel@nongnu.org, qemu-riscv@nongnu.org Cc: Palmer Dabbelt , Alistair Francis , Bin Meng , Weiwei Li , Daniel Henrique Barboza , Liu Zhiwei , richard.henderson@linaro.org, Max Chou Subject: [RFC PATCH v5 2/5] target/riscv: rvv: Provide a fast path using direct access to host ram for unmasked unit-stride load/store Date: Wed, 17 Jul 2024 21:39:33 +0800 Message-Id: <20240717133936.713642-3-max.chou@sifive.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240717133936.713642-1-max.chou@sifive.com> References: <20240717133936.713642-1-max.chou@sifive.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2607:f8b0:4864:20::636; envelope-from=max.chou@sifive.com; helo=mail-pl1-x636.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @sifive.com) X-ZM-MESSAGEID: 1721223630908116600 Content-Type: text/plain; charset="utf-8" This commit references the sve_ldN_r/sve_stN_r helper functions in ARM target to optimize the vector unmasked unit-stride load/store implementation with following optimizations: * Get the page boundary * Probing pages/resolving host memory address at the beginning if possible * Provide new interface to direct access host memory * Switch to the original slow TLB access when cross page element/violate page permission/violate pmp/watchpoints in page The original element load/store interface is replaced by the new element load/store functions with _tlb & _host postfix that means doing the element load/store through the original softmmu flow and the direct access host memory flow. Signed-off-by: Max Chou --- target/riscv/vector_helper.c | 361 +++++++++++++++++++++-------------- 1 file changed, 220 insertions(+), 141 deletions(-) diff --git a/target/riscv/vector_helper.c b/target/riscv/vector_helper.c index 4d72eb74d3a..23396a1b750 100644 --- a/target/riscv/vector_helper.c +++ b/target/riscv/vector_helper.c @@ -146,34 +146,47 @@ static inline void vext_set_elem_mask(void *v0, int i= ndex, } =20 /* elements operations for load and store */ -typedef void vext_ldst_elem_fn(CPURISCVState *env, abi_ptr addr, - uint32_t idx, void *vd, uintptr_t retaddr); +typedef void vext_ldst_elem_fn_tlb(CPURISCVState *env, abi_ptr addr, + uint32_t idx, void *vd, uintptr_t retad= dr); +typedef void vext_ldst_elem_fn_host(void *vd, uint32_t idx, void *host); =20 -#define GEN_VEXT_LD_ELEM(NAME, ETYPE, H, LDSUF) \ -static void NAME(CPURISCVState *env, abi_ptr addr, \ - uint32_t idx, void *vd, uintptr_t retaddr)\ -{ \ - ETYPE *cur =3D ((ETYPE *)vd + H(idx)); \ - *cur =3D cpu_##LDSUF##_data_ra(env, addr, retaddr); \ -} \ - -GEN_VEXT_LD_ELEM(lde_b, int8_t, H1, ldsb) -GEN_VEXT_LD_ELEM(lde_h, int16_t, H2, ldsw) -GEN_VEXT_LD_ELEM(lde_w, int32_t, H4, ldl) -GEN_VEXT_LD_ELEM(lde_d, int64_t, H8, ldq) - -#define GEN_VEXT_ST_ELEM(NAME, ETYPE, H, STSUF) \ -static void NAME(CPURISCVState *env, abi_ptr addr, \ - uint32_t idx, void *vd, uintptr_t retaddr)\ -{ \ - ETYPE data =3D *((ETYPE *)vd + H(idx)); \ - cpu_##STSUF##_data_ra(env, addr, data, retaddr); \ +#define GEN_VEXT_LD_ELEM(NAME, ETYPE, H, LDSUF) \ +static void NAME##_tlb(CPURISCVState *env, abi_ptr addr, \ + uint32_t byte_off, void *vd, uintptr_t retaddr) \ +{ \ + ETYPE *cur =3D vd + byte_off; \ + *cur =3D cpu_##LDSUF##_data_ra(env, addr, retaddr); \ +} \ + \ +static void NAME##_host(void *vd, uint32_t byte_off, void *host) \ +{ \ + ETYPE val =3D LDSUF##_p(host); \ + *(ETYPE *)(vd + byte_off) =3D val; \ +} + +GEN_VEXT_LD_ELEM(lde_b, uint8_t, H1, ldub) +GEN_VEXT_LD_ELEM(lde_h, uint16_t, H2, lduw) +GEN_VEXT_LD_ELEM(lde_w, uint32_t, H4, ldl) +GEN_VEXT_LD_ELEM(lde_d, uint64_t, H8, ldq) + +#define GEN_VEXT_ST_ELEM(NAME, ETYPE, H, STSUF) \ +static void NAME##_tlb(CPURISCVState *env, abi_ptr addr, \ + uint32_t byte_off, void *vd, uintptr_t retaddr) \ +{ \ + ETYPE data =3D *(ETYPE *)(vd + byte_off); \ + cpu_##STSUF##_data_ra(env, addr, data, retaddr); \ +} \ + \ +static void NAME##_host(void *vd, uint32_t byte_off, void *host) \ +{ \ + ETYPE val =3D *(ETYPE *)(vd + byte_off); \ + STSUF##_p(host, val); \ } =20 -GEN_VEXT_ST_ELEM(ste_b, int8_t, H1, stb) -GEN_VEXT_ST_ELEM(ste_h, int16_t, H2, stw) -GEN_VEXT_ST_ELEM(ste_w, int32_t, H4, stl) -GEN_VEXT_ST_ELEM(ste_d, int64_t, H8, stq) +GEN_VEXT_ST_ELEM(ste_b, uint8_t, H1, stb) +GEN_VEXT_ST_ELEM(ste_h, uint16_t, H2, stw) +GEN_VEXT_ST_ELEM(ste_w, uint32_t, H4, stl) +GEN_VEXT_ST_ELEM(ste_d, uint64_t, H8, stq) =20 static void vext_set_tail_elems_1s(target_ulong vl, void *vd, uint32_t desc, uint32_t nf, @@ -199,7 +212,7 @@ static void vext_ldst_stride(void *vd, void *v0, target_ulong base, target_ulong stride, CPURISCVState *env, uint32_t desc, uint32_t vm, - vext_ldst_elem_fn *ldst_elem, + vext_ldst_elem_fn_tlb * ldst_elem, uint32_t log2_esz, uintptr_t ra) { uint32_t i, k; @@ -221,7 +234,8 @@ vext_ldst_stride(void *vd, void *v0, target_ulong base, continue; } target_ulong addr =3D base + stride * i + (k << log2_esz); - ldst_elem(env, adjust_addr(env, addr), i + k * max_elems, vd, = ra); + ldst_elem(env, adjust_addr(env, addr), + (i + k * max_elems) << log2_esz, vd, ra); k++; } } @@ -240,10 +254,10 @@ void HELPER(NAME)(void *vd, void * v0, target_ulong b= ase, \ ctzl(sizeof(ETYPE)), GETPC()); \ } =20 -GEN_VEXT_LD_STRIDE(vlse8_v, int8_t, lde_b) -GEN_VEXT_LD_STRIDE(vlse16_v, int16_t, lde_h) -GEN_VEXT_LD_STRIDE(vlse32_v, int32_t, lde_w) -GEN_VEXT_LD_STRIDE(vlse64_v, int64_t, lde_d) +GEN_VEXT_LD_STRIDE(vlse8_v, int8_t, lde_b_tlb) +GEN_VEXT_LD_STRIDE(vlse16_v, int16_t, lde_h_tlb) +GEN_VEXT_LD_STRIDE(vlse32_v, int32_t, lde_w_tlb) +GEN_VEXT_LD_STRIDE(vlse64_v, int64_t, lde_d_tlb) =20 #define GEN_VEXT_ST_STRIDE(NAME, ETYPE, STORE_FN) \ void HELPER(NAME)(void *vd, void *v0, target_ulong base, \ @@ -255,39 +269,100 @@ void HELPER(NAME)(void *vd, void *v0, target_ulong b= ase, \ ctzl(sizeof(ETYPE)), GETPC()); \ } =20 -GEN_VEXT_ST_STRIDE(vsse8_v, int8_t, ste_b) -GEN_VEXT_ST_STRIDE(vsse16_v, int16_t, ste_h) -GEN_VEXT_ST_STRIDE(vsse32_v, int32_t, ste_w) -GEN_VEXT_ST_STRIDE(vsse64_v, int64_t, ste_d) +GEN_VEXT_ST_STRIDE(vsse8_v, int8_t, ste_b_tlb) +GEN_VEXT_ST_STRIDE(vsse16_v, int16_t, ste_h_tlb) +GEN_VEXT_ST_STRIDE(vsse32_v, int32_t, ste_w_tlb) +GEN_VEXT_ST_STRIDE(vsse64_v, int64_t, ste_d_tlb) =20 /* * unit-stride: access elements stored contiguously in memory */ =20 /* unmasked unit-stride load and store operation */ +static void +vext_page_ldst_us(CPURISCVState *env, void *vd, target_ulong addr, + uint32_t elems, uint32_t nf, uint32_t max_elems, + uint32_t log2_esz, bool is_load, + vext_ldst_elem_fn_tlb *ldst_tlb, + vext_ldst_elem_fn_host *ldst_host, uintptr_t ra) +{ + void *host; + int i, k, flags; + uint32_t esz =3D 1 << log2_esz; + uint32_t size =3D (elems * nf) << log2_esz; + uint32_t evl =3D env->vstart + elems; + int mmu_index =3D riscv_env_mmu_index(env, false); + MMUAccessType access_type =3D is_load ? MMU_DATA_LOAD : MMU_DATA_STORE; + + /* Check page permission/pmp/watchpoint/etc. */ + flags =3D probe_access_flags(env, adjust_addr(env, addr), size, access= _type, + mmu_index, true, &host, ra); + + if (host && flags =3D=3D 0) { + for (i =3D env->vstart; i < evl; ++i) { + k =3D 0; + while (k < nf) { + ldst_host(vd, (i + k * max_elems) << log2_esz, host); + host +=3D esz; + k++; + } + } + env->vstart +=3D elems; + } else { + /* load bytes from guest memory */ + for (i =3D env->vstart; i < evl; env->vstart =3D ++i) { + k =3D 0; + while (k < nf) { + ldst_tlb(env, adjust_addr(env, addr), + (i + k * max_elems) << log2_esz, vd, ra); + addr +=3D esz; + k++; + } + } + } +} + static void vext_ldst_us(void *vd, target_ulong base, CPURISCVState *env, uint32_t des= c, - vext_ldst_elem_fn *ldst_elem, uint32_t log2_esz, uint32_t evl, - uintptr_t ra) + vext_ldst_elem_fn_tlb *ldst_tlb, + vext_ldst_elem_fn_host *ldst_host, uint32_t log2_esz, + uint32_t evl, uintptr_t ra, bool is_load) { - uint32_t i, k; + uint32_t k; + target_ulong page_split, elems, addr; uint32_t nf =3D vext_nf(desc); uint32_t max_elems =3D vext_max_elems(desc, log2_esz); uint32_t esz =3D 1 << log2_esz; + uint32_t msize =3D nf * esz; =20 VSTART_CHECK_EARLY_EXIT(env); =20 - /* load bytes from guest memory */ - for (i =3D env->vstart; i < evl; env->vstart =3D ++i) { - k =3D 0; - while (k < nf) { - target_ulong addr =3D base + ((i * nf + k) << log2_esz); - ldst_elem(env, adjust_addr(env, addr), i + k * max_elems, vd, = ra); - k++; + while (env->vstart < evl) { + /* Calculate page range */ + addr =3D base + ((env->vstart * nf) << log2_esz); + page_split =3D -(addr | TARGET_PAGE_MASK); + /* Get number of elements */ + elems =3D page_split / msize; + if (unlikely(env->vstart + elems >=3D evl)) { + elems =3D evl - env->vstart; + } + + /* Load/store elements in page */ + vext_page_ldst_us(env, vd, addr, elems, nf, max_elems, log2_esz, + is_load, ldst_tlb, ldst_host, ra); + + /* Cross page element */ + if (unlikely((page_split % msize) !=3D 0 && (env->vstart + 1) < ev= l)) { + for (k =3D 0; k < nf; k++) { + addr =3D base + ((env->vstart * nf + k) << log2_esz); + ldst_tlb(env, adjust_addr(env, addr), + (k * max_elems + env->vstart) << log2_esz, vd, ra= ); + } + env->vstart++; } } - env->vstart =3D 0; =20 + env->vstart =3D 0; vext_set_tail_elems_1s(evl, vd, desc, nf, esz, max_elems); } =20 @@ -296,47 +371,47 @@ vext_ldst_us(void *vd, target_ulong base, CPURISCVSta= te *env, uint32_t desc, * stride, stride =3D NF * sizeof (ETYPE) */ =20 -#define GEN_VEXT_LD_US(NAME, ETYPE, LOAD_FN) \ -void HELPER(NAME##_mask)(void *vd, void *v0, target_ulong base, \ - CPURISCVState *env, uint32_t desc) \ -{ \ - uint32_t stride =3D vext_nf(desc) << ctzl(sizeof(ETYPE)); \ - vext_ldst_stride(vd, v0, base, stride, env, desc, false, LOAD_FN, \ - ctzl(sizeof(ETYPE)), GETPC()); \ -} \ - \ -void HELPER(NAME)(void *vd, void *v0, target_ulong base, \ - CPURISCVState *env, uint32_t desc) \ -{ \ - vext_ldst_us(vd, base, env, desc, LOAD_FN, \ - ctzl(sizeof(ETYPE)), env->vl, GETPC()); \ +#define GEN_VEXT_LD_US(NAME, ETYPE, LOAD_FN_TLB, LOAD_FN_HOST) \ +void HELPER(NAME##_mask)(void *vd, void *v0, target_ulong base, \ + CPURISCVState *env, uint32_t desc) \ +{ \ + uint32_t stride =3D vext_nf(desc) << ctzl(sizeof(ETYPE)); \ + vext_ldst_stride(vd, v0, base, stride, env, desc, false, \ + LOAD_FN_TLB, ctzl(sizeof(ETYPE)), GETPC()); \ +} \ + \ +void HELPER(NAME)(void *vd, void *v0, target_ulong base, \ + CPURISCVState *env, uint32_t desc) \ +{ \ + vext_ldst_us(vd, base, env, desc, LOAD_FN_TLB, LOAD_FN_HOST, \ + ctzl(sizeof(ETYPE)), env->vl, GETPC(), true); \ } =20 -GEN_VEXT_LD_US(vle8_v, int8_t, lde_b) -GEN_VEXT_LD_US(vle16_v, int16_t, lde_h) -GEN_VEXT_LD_US(vle32_v, int32_t, lde_w) -GEN_VEXT_LD_US(vle64_v, int64_t, lde_d) +GEN_VEXT_LD_US(vle8_v, int8_t, lde_b_tlb, lde_b_host) +GEN_VEXT_LD_US(vle16_v, int16_t, lde_h_tlb, lde_h_host) +GEN_VEXT_LD_US(vle32_v, int32_t, lde_w_tlb, lde_w_host) +GEN_VEXT_LD_US(vle64_v, int64_t, lde_d_tlb, lde_d_host) =20 -#define GEN_VEXT_ST_US(NAME, ETYPE, STORE_FN) \ +#define GEN_VEXT_ST_US(NAME, ETYPE, STORE_FN_TLB, STORE_FN_HOST) \ void HELPER(NAME##_mask)(void *vd, void *v0, target_ulong base, \ CPURISCVState *env, uint32_t desc) \ { \ uint32_t stride =3D vext_nf(desc) << ctzl(sizeof(ETYPE)); = \ - vext_ldst_stride(vd, v0, base, stride, env, desc, false, STORE_FN, \ - ctzl(sizeof(ETYPE)), GETPC()); \ + vext_ldst_stride(vd, v0, base, stride, env, desc, false, \ + STORE_FN_TLB, ctzl(sizeof(ETYPE)), GETPC()); \ } \ \ void HELPER(NAME)(void *vd, void *v0, target_ulong base, \ CPURISCVState *env, uint32_t desc) \ { \ - vext_ldst_us(vd, base, env, desc, STORE_FN, \ - ctzl(sizeof(ETYPE)), env->vl, GETPC()); \ + vext_ldst_us(vd, base, env, desc, STORE_FN_TLB, STORE_FN_HOST, \ + ctzl(sizeof(ETYPE)), env->vl, GETPC(), false); \ } =20 -GEN_VEXT_ST_US(vse8_v, int8_t, ste_b) -GEN_VEXT_ST_US(vse16_v, int16_t, ste_h) -GEN_VEXT_ST_US(vse32_v, int32_t, ste_w) -GEN_VEXT_ST_US(vse64_v, int64_t, ste_d) +GEN_VEXT_ST_US(vse8_v, int8_t, ste_b_tlb, ste_b_host) +GEN_VEXT_ST_US(vse16_v, int16_t, ste_h_tlb, ste_h_host) +GEN_VEXT_ST_US(vse32_v, int32_t, ste_w_tlb, ste_w_host) +GEN_VEXT_ST_US(vse64_v, int64_t, ste_d_tlb, ste_d_host) =20 /* * unit stride mask load and store, EEW =3D 1 @@ -346,8 +421,8 @@ void HELPER(vlm_v)(void *vd, void *v0, target_ulong bas= e, { /* evl =3D ceil(vl/8) */ uint8_t evl =3D (env->vl + 7) >> 3; - vext_ldst_us(vd, base, env, desc, lde_b, - 0, evl, GETPC()); + vext_ldst_us(vd, base, env, desc, lde_b_tlb, lde_b_host, + 0, evl, GETPC(), true); } =20 void HELPER(vsm_v)(void *vd, void *v0, target_ulong base, @@ -355,8 +430,8 @@ void HELPER(vsm_v)(void *vd, void *v0, target_ulong bas= e, { /* evl =3D ceil(vl/8) */ uint8_t evl =3D (env->vl + 7) >> 3; - vext_ldst_us(vd, base, env, desc, ste_b, - 0, evl, GETPC()); + vext_ldst_us(vd, base, env, desc, ste_b_tlb, ste_b_host, + 0, evl, GETPC(), false); } =20 /* @@ -381,7 +456,7 @@ static inline void vext_ldst_index(void *vd, void *v0, target_ulong base, void *vs2, CPURISCVState *env, uint32_t desc, vext_get_index_addr get_index_addr, - vext_ldst_elem_fn *ldst_elem, + vext_ldst_elem_fn_tlb *ldst_elem, uint32_t log2_esz, uintptr_t ra) { uint32_t i, k; @@ -405,7 +480,8 @@ vext_ldst_index(void *vd, void *v0, target_ulong base, continue; } abi_ptr addr =3D get_index_addr(base, i, vs2) + (k << log2_esz= ); - ldst_elem(env, adjust_addr(env, addr), i + k * max_elems, vd, = ra); + ldst_elem(env, adjust_addr(env, addr), + (i + k * max_elems) << log2_esz, vd, ra); k++; } } @@ -422,22 +498,22 @@ void HELPER(NAME)(void *vd, void *v0, target_ulong ba= se, \ LOAD_FN, ctzl(sizeof(ETYPE)), GETPC()); = \ } =20 -GEN_VEXT_LD_INDEX(vlxei8_8_v, int8_t, idx_b, lde_b) -GEN_VEXT_LD_INDEX(vlxei8_16_v, int16_t, idx_b, lde_h) -GEN_VEXT_LD_INDEX(vlxei8_32_v, int32_t, idx_b, lde_w) -GEN_VEXT_LD_INDEX(vlxei8_64_v, int64_t, idx_b, lde_d) -GEN_VEXT_LD_INDEX(vlxei16_8_v, int8_t, idx_h, lde_b) -GEN_VEXT_LD_INDEX(vlxei16_16_v, int16_t, idx_h, lde_h) -GEN_VEXT_LD_INDEX(vlxei16_32_v, int32_t, idx_h, lde_w) -GEN_VEXT_LD_INDEX(vlxei16_64_v, int64_t, idx_h, lde_d) -GEN_VEXT_LD_INDEX(vlxei32_8_v, int8_t, idx_w, lde_b) -GEN_VEXT_LD_INDEX(vlxei32_16_v, int16_t, idx_w, lde_h) -GEN_VEXT_LD_INDEX(vlxei32_32_v, int32_t, idx_w, lde_w) -GEN_VEXT_LD_INDEX(vlxei32_64_v, int64_t, idx_w, lde_d) -GEN_VEXT_LD_INDEX(vlxei64_8_v, int8_t, idx_d, lde_b) -GEN_VEXT_LD_INDEX(vlxei64_16_v, int16_t, idx_d, lde_h) -GEN_VEXT_LD_INDEX(vlxei64_32_v, int32_t, idx_d, lde_w) -GEN_VEXT_LD_INDEX(vlxei64_64_v, int64_t, idx_d, lde_d) +GEN_VEXT_LD_INDEX(vlxei8_8_v, int8_t, idx_b, lde_b_tlb) +GEN_VEXT_LD_INDEX(vlxei8_16_v, int16_t, idx_b, lde_h_tlb) +GEN_VEXT_LD_INDEX(vlxei8_32_v, int32_t, idx_b, lde_w_tlb) +GEN_VEXT_LD_INDEX(vlxei8_64_v, int64_t, idx_b, lde_d_tlb) +GEN_VEXT_LD_INDEX(vlxei16_8_v, int8_t, idx_h, lde_b_tlb) +GEN_VEXT_LD_INDEX(vlxei16_16_v, int16_t, idx_h, lde_h_tlb) +GEN_VEXT_LD_INDEX(vlxei16_32_v, int32_t, idx_h, lde_w_tlb) +GEN_VEXT_LD_INDEX(vlxei16_64_v, int64_t, idx_h, lde_d_tlb) +GEN_VEXT_LD_INDEX(vlxei32_8_v, int8_t, idx_w, lde_b_tlb) +GEN_VEXT_LD_INDEX(vlxei32_16_v, int16_t, idx_w, lde_h_tlb) +GEN_VEXT_LD_INDEX(vlxei32_32_v, int32_t, idx_w, lde_w_tlb) +GEN_VEXT_LD_INDEX(vlxei32_64_v, int64_t, idx_w, lde_d_tlb) +GEN_VEXT_LD_INDEX(vlxei64_8_v, int8_t, idx_d, lde_b_tlb) +GEN_VEXT_LD_INDEX(vlxei64_16_v, int16_t, idx_d, lde_h_tlb) +GEN_VEXT_LD_INDEX(vlxei64_32_v, int32_t, idx_d, lde_w_tlb) +GEN_VEXT_LD_INDEX(vlxei64_64_v, int64_t, idx_d, lde_d_tlb) =20 #define GEN_VEXT_ST_INDEX(NAME, ETYPE, INDEX_FN, STORE_FN) \ void HELPER(NAME)(void *vd, void *v0, target_ulong base, \ @@ -448,22 +524,22 @@ void HELPER(NAME)(void *vd, void *v0, target_ulong ba= se, \ GETPC()); \ } =20 -GEN_VEXT_ST_INDEX(vsxei8_8_v, int8_t, idx_b, ste_b) -GEN_VEXT_ST_INDEX(vsxei8_16_v, int16_t, idx_b, ste_h) -GEN_VEXT_ST_INDEX(vsxei8_32_v, int32_t, idx_b, ste_w) -GEN_VEXT_ST_INDEX(vsxei8_64_v, int64_t, idx_b, ste_d) -GEN_VEXT_ST_INDEX(vsxei16_8_v, int8_t, idx_h, ste_b) -GEN_VEXT_ST_INDEX(vsxei16_16_v, int16_t, idx_h, ste_h) -GEN_VEXT_ST_INDEX(vsxei16_32_v, int32_t, idx_h, ste_w) -GEN_VEXT_ST_INDEX(vsxei16_64_v, int64_t, idx_h, ste_d) -GEN_VEXT_ST_INDEX(vsxei32_8_v, int8_t, idx_w, ste_b) -GEN_VEXT_ST_INDEX(vsxei32_16_v, int16_t, idx_w, ste_h) -GEN_VEXT_ST_INDEX(vsxei32_32_v, int32_t, idx_w, ste_w) -GEN_VEXT_ST_INDEX(vsxei32_64_v, int64_t, idx_w, ste_d) -GEN_VEXT_ST_INDEX(vsxei64_8_v, int8_t, idx_d, ste_b) -GEN_VEXT_ST_INDEX(vsxei64_16_v, int16_t, idx_d, ste_h) -GEN_VEXT_ST_INDEX(vsxei64_32_v, int32_t, idx_d, ste_w) -GEN_VEXT_ST_INDEX(vsxei64_64_v, int64_t, idx_d, ste_d) +GEN_VEXT_ST_INDEX(vsxei8_8_v, int8_t, idx_b, ste_b_tlb) +GEN_VEXT_ST_INDEX(vsxei8_16_v, int16_t, idx_b, ste_h_tlb) +GEN_VEXT_ST_INDEX(vsxei8_32_v, int32_t, idx_b, ste_w_tlb) +GEN_VEXT_ST_INDEX(vsxei8_64_v, int64_t, idx_b, ste_d_tlb) +GEN_VEXT_ST_INDEX(vsxei16_8_v, int8_t, idx_h, ste_b_tlb) +GEN_VEXT_ST_INDEX(vsxei16_16_v, int16_t, idx_h, ste_h_tlb) +GEN_VEXT_ST_INDEX(vsxei16_32_v, int32_t, idx_h, ste_w_tlb) +GEN_VEXT_ST_INDEX(vsxei16_64_v, int64_t, idx_h, ste_d_tlb) +GEN_VEXT_ST_INDEX(vsxei32_8_v, int8_t, idx_w, ste_b_tlb) +GEN_VEXT_ST_INDEX(vsxei32_16_v, int16_t, idx_w, ste_h_tlb) +GEN_VEXT_ST_INDEX(vsxei32_32_v, int32_t, idx_w, ste_w_tlb) +GEN_VEXT_ST_INDEX(vsxei32_64_v, int64_t, idx_w, ste_d_tlb) +GEN_VEXT_ST_INDEX(vsxei64_8_v, int8_t, idx_d, ste_b_tlb) +GEN_VEXT_ST_INDEX(vsxei64_16_v, int16_t, idx_d, ste_h_tlb) +GEN_VEXT_ST_INDEX(vsxei64_32_v, int32_t, idx_d, ste_w_tlb) +GEN_VEXT_ST_INDEX(vsxei64_64_v, int64_t, idx_d, ste_d_tlb) =20 /* * unit-stride fault-only-fisrt load instructions @@ -471,7 +547,7 @@ GEN_VEXT_ST_INDEX(vsxei64_64_v, int64_t, idx_d, ste_d) static inline void vext_ldff(void *vd, void *v0, target_ulong base, CPURISCVState *env, uint32_t desc, - vext_ldst_elem_fn *ldst_elem, + vext_ldst_elem_fn_tlb *ldst_elem, uint32_t log2_esz, uintptr_t ra) { uint32_t i, k, vl =3D 0; @@ -539,7 +615,8 @@ vext_ldff(void *vd, void *v0, target_ulong base, continue; } addr =3D base + ((i * nf + k) << log2_esz); - ldst_elem(env, adjust_addr(env, addr), i + k * max_elems, vd, = ra); + ldst_elem(env, adjust_addr(env, addr), + (i + k * max_elems) << log2_esz, vd, ra); k++; } } @@ -556,10 +633,10 @@ void HELPER(NAME)(void *vd, void *v0, target_ulong ba= se, \ ctzl(sizeof(ETYPE)), GETPC()); \ } =20 -GEN_VEXT_LDFF(vle8ff_v, int8_t, lde_b) -GEN_VEXT_LDFF(vle16ff_v, int16_t, lde_h) -GEN_VEXT_LDFF(vle32ff_v, int32_t, lde_w) -GEN_VEXT_LDFF(vle64ff_v, int64_t, lde_d) +GEN_VEXT_LDFF(vle8ff_v, int8_t, lde_b_tlb) +GEN_VEXT_LDFF(vle16ff_v, int16_t, lde_h_tlb) +GEN_VEXT_LDFF(vle32ff_v, int32_t, lde_w_tlb) +GEN_VEXT_LDFF(vle64ff_v, int64_t, lde_d_tlb) =20 #define DO_SWAP(N, M) (M) #define DO_AND(N, M) (N & M) @@ -576,7 +653,8 @@ GEN_VEXT_LDFF(vle64ff_v, int64_t, lde_d) */ static void vext_ldst_whole(void *vd, target_ulong base, CPURISCVState *env, uint32_t = desc, - vext_ldst_elem_fn *ldst_elem, uint32_t log2_esz, uintptr_t= ra) + vext_ldst_elem_fn_tlb *ldst_elem, uint32_t log2_esz, + uintptr_t ra) { uint32_t i, k, off, pos; uint32_t nf =3D vext_nf(desc); @@ -595,8 +673,8 @@ vext_ldst_whole(void *vd, target_ulong base, CPURISCVSt= ate *env, uint32_t desc, /* load/store rest of elements of current segment pointed by vstar= t */ for (pos =3D off; pos < max_elems; pos++, env->vstart++) { target_ulong addr =3D base + ((pos + k * max_elems) << log2_es= z); - ldst_elem(env, adjust_addr(env, addr), pos + k * max_elems, vd, - ra); + ldst_elem(env, adjust_addr(env, addr), + (pos + k * max_elems) << log2_esz, vd, ra); } k++; } @@ -605,7 +683,8 @@ vext_ldst_whole(void *vd, target_ulong base, CPURISCVSt= ate *env, uint32_t desc, for (; k < nf; k++) { for (i =3D 0; i < max_elems; i++, env->vstart++) { target_ulong addr =3D base + ((i + k * max_elems) << log2_esz); - ldst_elem(env, adjust_addr(env, addr), i + k * max_elems, vd, = ra); + ldst_elem(env, adjust_addr(env, addr), + (i + k * max_elems) << log2_esz, vd, ra); } } =20 @@ -620,22 +699,22 @@ void HELPER(NAME)(void *vd, target_ulong base, \ ctzl(sizeof(ETYPE)), GETPC()); \ } =20 -GEN_VEXT_LD_WHOLE(vl1re8_v, int8_t, lde_b) -GEN_VEXT_LD_WHOLE(vl1re16_v, int16_t, lde_h) -GEN_VEXT_LD_WHOLE(vl1re32_v, int32_t, lde_w) -GEN_VEXT_LD_WHOLE(vl1re64_v, int64_t, lde_d) -GEN_VEXT_LD_WHOLE(vl2re8_v, int8_t, lde_b) -GEN_VEXT_LD_WHOLE(vl2re16_v, int16_t, lde_h) -GEN_VEXT_LD_WHOLE(vl2re32_v, int32_t, lde_w) -GEN_VEXT_LD_WHOLE(vl2re64_v, int64_t, lde_d) -GEN_VEXT_LD_WHOLE(vl4re8_v, int8_t, lde_b) -GEN_VEXT_LD_WHOLE(vl4re16_v, int16_t, lde_h) -GEN_VEXT_LD_WHOLE(vl4re32_v, int32_t, lde_w) -GEN_VEXT_LD_WHOLE(vl4re64_v, int64_t, lde_d) -GEN_VEXT_LD_WHOLE(vl8re8_v, int8_t, lde_b) -GEN_VEXT_LD_WHOLE(vl8re16_v, int16_t, lde_h) -GEN_VEXT_LD_WHOLE(vl8re32_v, int32_t, lde_w) -GEN_VEXT_LD_WHOLE(vl8re64_v, int64_t, lde_d) +GEN_VEXT_LD_WHOLE(vl1re8_v, int8_t, lde_b_tlb) +GEN_VEXT_LD_WHOLE(vl1re16_v, int16_t, lde_h_tlb) +GEN_VEXT_LD_WHOLE(vl1re32_v, int32_t, lde_w_tlb) +GEN_VEXT_LD_WHOLE(vl1re64_v, int64_t, lde_d_tlb) +GEN_VEXT_LD_WHOLE(vl2re8_v, int8_t, lde_b_tlb) +GEN_VEXT_LD_WHOLE(vl2re16_v, int16_t, lde_h_tlb) +GEN_VEXT_LD_WHOLE(vl2re32_v, int32_t, lde_w_tlb) +GEN_VEXT_LD_WHOLE(vl2re64_v, int64_t, lde_d_tlb) +GEN_VEXT_LD_WHOLE(vl4re8_v, int8_t, lde_b_tlb) +GEN_VEXT_LD_WHOLE(vl4re16_v, int16_t, lde_h_tlb) +GEN_VEXT_LD_WHOLE(vl4re32_v, int32_t, lde_w_tlb) +GEN_VEXT_LD_WHOLE(vl4re64_v, int64_t, lde_d_tlb) +GEN_VEXT_LD_WHOLE(vl8re8_v, int8_t, lde_b_tlb) +GEN_VEXT_LD_WHOLE(vl8re16_v, int16_t, lde_h_tlb) +GEN_VEXT_LD_WHOLE(vl8re32_v, int32_t, lde_w_tlb) +GEN_VEXT_LD_WHOLE(vl8re64_v, int64_t, lde_d_tlb) =20 #define GEN_VEXT_ST_WHOLE(NAME, ETYPE, STORE_FN) \ void HELPER(NAME)(void *vd, target_ulong base, \ @@ -645,10 +724,10 @@ void HELPER(NAME)(void *vd, target_ulong base, \ ctzl(sizeof(ETYPE)), GETPC()); \ } =20 -GEN_VEXT_ST_WHOLE(vs1r_v, int8_t, ste_b) -GEN_VEXT_ST_WHOLE(vs2r_v, int8_t, ste_b) -GEN_VEXT_ST_WHOLE(vs4r_v, int8_t, ste_b) -GEN_VEXT_ST_WHOLE(vs8r_v, int8_t, ste_b) +GEN_VEXT_ST_WHOLE(vs1r_v, int8_t, ste_b_tlb) +GEN_VEXT_ST_WHOLE(vs2r_v, int8_t, ste_b_tlb) +GEN_VEXT_ST_WHOLE(vs4r_v, int8_t, ste_b_tlb) +GEN_VEXT_ST_WHOLE(vs8r_v, int8_t, ste_b_tlb) =20 /* * Vector Integer Arithmetic Instructions --=20 2.34.1 From nobody Mon Sep 16 19:08:48 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=reject dis=none) header.from=sifive.com ARC-Seal: i=1; a=rsa-sha256; t=1721223631; cv=none; d=zohomail.com; s=zohoarc; b=muZGRNfK8gcLhZ05Yq5TOAVZrm5XrLdwffCbk8sNmJOBo19hPIeW6nriFLzrSWFCbU4zRPW0GTOF5X7MK9DquRbjK6qGuYRD3KKfiF5Kmfe3kLHwTbWg04RUu4ICHigl82jYdFrPukkx7kYE8kaqCBB5ARiA72ijRlaTo44nmRI= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1721223631; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=AgdC7+ev+17fl87ruXR55XIU4+t2//DA3K4P/cKC9yA=; b=MkWnzDpVn3sLWrap2y2a8vl/ryhJTizSUEYDQbQ3pxvX6e2W3VSDYtn/VqjqIPtRgUG1TLy6pQZ6pMZ5Me6lspsP4C3j7B+iSJior7NJHIywbdoX+SgEOAcASwhP1tDbHKe8lWCFCbeDWWC9JcCaHOnPT5Ao7OOGZNeVhsPQr+o= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=reject dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1721223631442838.817685321517; Wed, 17 Jul 2024 06:40:31 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sU4t9-0001JP-Ef; Wed, 17 Jul 2024 09:40:11 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sU4t5-00015u-Qn for qemu-devel@nongnu.org; Wed, 17 Jul 2024 09:40:07 -0400 Received: from mail-pl1-x630.google.com ([2607:f8b0:4864:20::630]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1sU4st-0007Sj-Sy for qemu-devel@nongnu.org; Wed, 17 Jul 2024 09:40:06 -0400 Received: by mail-pl1-x630.google.com with SMTP id d9443c01a7336-1fb8781ef1bso54156445ad.3 for ; Wed, 17 Jul 2024 06:39:55 -0700 (PDT) Received: from duncan.localdomain (114-35-142-126.hinet-ip.hinet.net. [114.35.142.126]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-1fc0bc5aa6esm74903495ad.299.2024.07.17.06.39.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 17 Jul 2024 06:39:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1721223594; x=1721828394; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=AgdC7+ev+17fl87ruXR55XIU4+t2//DA3K4P/cKC9yA=; b=jRHPO91zyD3PeU5/jU8j7Ayy6H1L0dYV8iOO6lgc3CY7+1dg1LSEx3vpIFr9stvWTP yheZIhqGz1j+GXre+zCumh44R+TLErppYWLITxWnLYchrHeubCVEiOhcS/sXfgDBqcJQ U4n4et5OcGEt9Wkh8ZICsqxSRrzJ3VV1b7TJFMwPJ+c89MGh4YTbtwCLCZwLkcdvAxjU pcxoHHAfoiZlwT1FT8iVKORwf7ws2HPnZNZZuHQ0J3ZB+2gx73HjXNkSAjUqP1KFCEQN zOpS00Ujk8tfjzKMVtBt/di5iVC/mGlEEI3281H+p0vIpTazW6xI4Y3tUR/hkb6twzKx R4oA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721223594; x=1721828394; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=AgdC7+ev+17fl87ruXR55XIU4+t2//DA3K4P/cKC9yA=; b=jaX4YUF0zQD5ym+be1xTF5yQWQaULh6/05CzGojOjFMIeEURnVo1wwVa9RDL65ksWF kqgcGpbgTiobjU1hoF9iFQvafFKV2gmH6LQ+jIyhvvTnGlm8kFJrRnHI6xNQ94r7TUyE fB+zrrMwNc3WU+tlk+EJ6Etu1lA3ix45nJxIT5qkNyIoC1PzmL5DK+nMbaQ2/SCsMBbw zgVXCSf5WTVnLBKgwRPO+f5+odoTjiLkDwg7PCKntfYgfaD9y+SlNgXEi77nHa5htyus VLmCem06aLNbi9Ahq0QzuHne0K7/PMDYopoGEyqgFkg8mV5h3kOLdHNMFgXX3l5XiY+e lexQ== X-Gm-Message-State: AOJu0YwOkpsRkyZ9EMDgvyxmprMGlPW6pkvLdXTHrRxjG2JSOjOIKzp2 vG0HhHq8x5hQ9il9PP60VtYCv14ZAfo0FUlzsn5d36av9Ul25LnDbVA/iodSADunYHzSsHpweu3 glL/pI/TjK5pH6tLXiVOOze5Ph6mzWmybwIcuvQjoVn84W2qpKQv8Rx3QMmPB+0n/TDPaJtB0Lr b93fMyxmicqeDCHTTIkJoFXEBEL+wrz6HA+lJ39w== X-Google-Smtp-Source: AGHT+IEnRb7c1roUlDRyF+f17QD0fG9J5BjOWX4PSelCmqG85q5wJF9H7/Cz+lwm5Z9iy+IdI3/3wA== X-Received: by 2002:a17:902:d4cb:b0:1f8:44f8:a364 with SMTP id d9443c01a7336-1fc4e6886e0mr13809345ad.48.1721223594102; Wed, 17 Jul 2024 06:39:54 -0700 (PDT) From: Max Chou To: qemu-devel@nongnu.org, qemu-riscv@nongnu.org Cc: Palmer Dabbelt , Alistair Francis , Bin Meng , Weiwei Li , Daniel Henrique Barboza , Liu Zhiwei , richard.henderson@linaro.org, Max Chou Subject: [RFC PATCH v5 3/5] target/riscv: rvv: Provide a fast path using direct access to host ram for unit-stride whole register load/store Date: Wed, 17 Jul 2024 21:39:34 +0800 Message-Id: <20240717133936.713642-4-max.chou@sifive.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240717133936.713642-1-max.chou@sifive.com> References: <20240717133936.713642-1-max.chou@sifive.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2607:f8b0:4864:20::630; envelope-from=max.chou@sifive.com; helo=mail-pl1-x630.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @sifive.com) X-ZM-MESSAGEID: 1721223632847116600 Content-Type: text/plain; charset="utf-8" The vector unit-stride whole register load/store instructions are similar to unmasked unit-stride load/store instructions that is suitable to be optimized by using a direct access to host ram fast path. Because the vector whole register load/store instructions do not need to handle the tail agnostic, so remove the vstart early exit checking. Signed-off-by: Max Chou --- target/riscv/vector_helper.c | 123 +++++++++++++++++------------------ 1 file changed, 61 insertions(+), 62 deletions(-) diff --git a/target/riscv/vector_helper.c b/target/riscv/vector_helper.c index 23396a1b750..5343a08e6ad 100644 --- a/target/riscv/vector_helper.c +++ b/target/riscv/vector_helper.c @@ -653,81 +653,80 @@ GEN_VEXT_LDFF(vle64ff_v, int64_t, lde_d_tlb) */ static void vext_ldst_whole(void *vd, target_ulong base, CPURISCVState *env, uint32_t = desc, - vext_ldst_elem_fn_tlb *ldst_elem, uint32_t log2_esz, - uintptr_t ra) + vext_ldst_elem_fn_tlb *ldst_tlb, + vext_ldst_elem_fn_host *ldst_host, uint32_t log2_esz, + uintptr_t ra, bool is_load) { - uint32_t i, k, off, pos; + target_ulong page_split, elems, addr; uint32_t nf =3D vext_nf(desc); uint32_t vlenb =3D riscv_cpu_cfg(env)->vlenb; uint32_t max_elems =3D vlenb >> log2_esz; + uint32_t evl =3D nf * max_elems; + uint32_t esz =3D 1 << log2_esz; =20 - if (env->vstart >=3D ((vlenb * nf) >> log2_esz)) { - env->vstart =3D 0; - return; - } - - k =3D env->vstart / max_elems; - off =3D env->vstart % max_elems; - - if (off) { - /* load/store rest of elements of current segment pointed by vstar= t */ - for (pos =3D off; pos < max_elems; pos++, env->vstart++) { - target_ulong addr =3D base + ((pos + k * max_elems) << log2_es= z); - ldst_elem(env, adjust_addr(env, addr), - (pos + k * max_elems) << log2_esz, vd, ra); + while (env->vstart < evl) { + /* Calculate page range */ + addr =3D base + (env->vstart << log2_esz); + page_split =3D -(addr | TARGET_PAGE_MASK); + /* Get number of elements */ + elems =3D page_split / esz; + if (unlikely(env->vstart + elems >=3D evl)) { + elems =3D evl - env->vstart; } - k++; - } =20 - /* load/store elements for rest of segments */ - for (; k < nf; k++) { - for (i =3D 0; i < max_elems; i++, env->vstart++) { - target_ulong addr =3D base + ((i + k * max_elems) << log2_esz); - ldst_elem(env, adjust_addr(env, addr), - (i + k * max_elems) << log2_esz, vd, ra); + /* Load/store elements in page */ + vext_page_ldst_us(env, vd, addr, elems, 1, max_elems, log2_esz, + is_load, ldst_tlb, ldst_host, ra); + + /* Cross page element */ + if (unlikely((page_split % esz) !=3D 0 && (env->vstart + 1) < evl)= ) { + addr =3D base + (env->vstart << log2_esz); + ldst_tlb(env, adjust_addr(env, addr), (env->vstart << log2_esz= ), + vd, ra); + env->vstart++; } } =20 env->vstart =3D 0; } =20 -#define GEN_VEXT_LD_WHOLE(NAME, ETYPE, LOAD_FN) \ -void HELPER(NAME)(void *vd, target_ulong base, \ - CPURISCVState *env, uint32_t desc) \ -{ \ - vext_ldst_whole(vd, base, env, desc, LOAD_FN, \ - ctzl(sizeof(ETYPE)), GETPC()); \ -} - -GEN_VEXT_LD_WHOLE(vl1re8_v, int8_t, lde_b_tlb) -GEN_VEXT_LD_WHOLE(vl1re16_v, int16_t, lde_h_tlb) -GEN_VEXT_LD_WHOLE(vl1re32_v, int32_t, lde_w_tlb) -GEN_VEXT_LD_WHOLE(vl1re64_v, int64_t, lde_d_tlb) -GEN_VEXT_LD_WHOLE(vl2re8_v, int8_t, lde_b_tlb) -GEN_VEXT_LD_WHOLE(vl2re16_v, int16_t, lde_h_tlb) -GEN_VEXT_LD_WHOLE(vl2re32_v, int32_t, lde_w_tlb) -GEN_VEXT_LD_WHOLE(vl2re64_v, int64_t, lde_d_tlb) -GEN_VEXT_LD_WHOLE(vl4re8_v, int8_t, lde_b_tlb) -GEN_VEXT_LD_WHOLE(vl4re16_v, int16_t, lde_h_tlb) -GEN_VEXT_LD_WHOLE(vl4re32_v, int32_t, lde_w_tlb) -GEN_VEXT_LD_WHOLE(vl4re64_v, int64_t, lde_d_tlb) -GEN_VEXT_LD_WHOLE(vl8re8_v, int8_t, lde_b_tlb) -GEN_VEXT_LD_WHOLE(vl8re16_v, int16_t, lde_h_tlb) -GEN_VEXT_LD_WHOLE(vl8re32_v, int32_t, lde_w_tlb) -GEN_VEXT_LD_WHOLE(vl8re64_v, int64_t, lde_d_tlb) - -#define GEN_VEXT_ST_WHOLE(NAME, ETYPE, STORE_FN) \ -void HELPER(NAME)(void *vd, target_ulong base, \ - CPURISCVState *env, uint32_t desc) \ -{ \ - vext_ldst_whole(vd, base, env, desc, STORE_FN, \ - ctzl(sizeof(ETYPE)), GETPC()); \ -} - -GEN_VEXT_ST_WHOLE(vs1r_v, int8_t, ste_b_tlb) -GEN_VEXT_ST_WHOLE(vs2r_v, int8_t, ste_b_tlb) -GEN_VEXT_ST_WHOLE(vs4r_v, int8_t, ste_b_tlb) -GEN_VEXT_ST_WHOLE(vs8r_v, int8_t, ste_b_tlb) +#define GEN_VEXT_LD_WHOLE(NAME, ETYPE, LOAD_FN_TLB, LOAD_FN_HOST) \ +void HELPER(NAME)(void *vd, target_ulong base, CPURISCVState *env, \ + uint32_t desc) \ +{ \ + vext_ldst_whole(vd, base, env, desc, LOAD_FN_TLB, LOAD_FN_HOST, \ + ctzl(sizeof(ETYPE)), GETPC(), true); \ +} + +GEN_VEXT_LD_WHOLE(vl1re8_v, int8_t, lde_b_tlb, lde_b_host) +GEN_VEXT_LD_WHOLE(vl1re16_v, int16_t, lde_h_tlb, lde_h_host) +GEN_VEXT_LD_WHOLE(vl1re32_v, int32_t, lde_w_tlb, lde_w_host) +GEN_VEXT_LD_WHOLE(vl1re64_v, int64_t, lde_d_tlb, lde_d_host) +GEN_VEXT_LD_WHOLE(vl2re8_v, int8_t, lde_b_tlb, lde_b_host) +GEN_VEXT_LD_WHOLE(vl2re16_v, int16_t, lde_h_tlb, lde_h_host) +GEN_VEXT_LD_WHOLE(vl2re32_v, int32_t, lde_w_tlb, lde_w_host) +GEN_VEXT_LD_WHOLE(vl2re64_v, int64_t, lde_d_tlb, lde_d_host) +GEN_VEXT_LD_WHOLE(vl4re8_v, int8_t, lde_b_tlb, lde_b_host) +GEN_VEXT_LD_WHOLE(vl4re16_v, int16_t, lde_h_tlb, lde_h_host) +GEN_VEXT_LD_WHOLE(vl4re32_v, int32_t, lde_w_tlb, lde_w_host) +GEN_VEXT_LD_WHOLE(vl4re64_v, int64_t, lde_d_tlb, lde_d_host) +GEN_VEXT_LD_WHOLE(vl8re8_v, int8_t, lde_b_tlb, lde_b_host) +GEN_VEXT_LD_WHOLE(vl8re16_v, int16_t, lde_h_tlb, lde_h_host) +GEN_VEXT_LD_WHOLE(vl8re32_v, int32_t, lde_w_tlb, lde_w_host) +GEN_VEXT_LD_WHOLE(vl8re64_v, int64_t, lde_d_tlb, lde_d_host) + +#define GEN_VEXT_ST_WHOLE(NAME, ETYPE, STORE_FN_TLB, STORE_FN_HOST) \ +void HELPER(NAME)(void *vd, target_ulong base, CPURISCVState *env, \ + uint32_t desc) \ +{ \ + vext_ldst_whole(vd, base, env, desc, STORE_FN_TLB, STORE_FN_HOST, \ + ctzl(sizeof(ETYPE)), GETPC(), false); \ +} + +GEN_VEXT_ST_WHOLE(vs1r_v, int8_t, ste_b_tlb, ste_b_host) +GEN_VEXT_ST_WHOLE(vs2r_v, int8_t, ste_b_tlb, ste_b_host) +GEN_VEXT_ST_WHOLE(vs4r_v, int8_t, ste_b_tlb, ste_b_host) +GEN_VEXT_ST_WHOLE(vs8r_v, int8_t, ste_b_tlb, ste_b_host) =20 /* * Vector Integer Arithmetic Instructions --=20 2.34.1 From nobody Mon Sep 16 19:08:48 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=reject dis=none) header.from=sifive.com ARC-Seal: i=1; a=rsa-sha256; t=1721223680; cv=none; d=zohomail.com; s=zohoarc; b=cnruJf96aERAhq5gaByzKRlVsJQzY+iASSkBknJLT4MTDsgb5j1wEUVkPTggGdfiCNGa5PaVVF8qOh1Sc7m0DMCw/3lkzivQ47Sfvwt8omL0DzCpePaFd+ONB1ogkp8PQqg97oYS7Zdg39p1GoxBgMRVWFe9oMkpPzfVuVFSYqI= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1721223680; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=Z6Ehnl/PC6i+F6ruu+0LWnyhe2hlKO/B5I8PcWhvJ30=; b=DiF7GzrQqaZcROm/nsBmWDqF2tH07gIPBBn3hpzEQIenVyZO9tIHGSrjcx/wGkyE3V43SUIL2sHBX7XMCnlZbril8R1YezCyP6EPhdZtd47lHD8nOt6Nq/pcBpq8QixMggC2hERI0QmVyKJy6baUZtgOszZCiI3MzgGrKIEXcPc= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=reject dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1721223680570465.1336679661897; Wed, 17 Jul 2024 06:41:20 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sU4t2-0000pM-20; Wed, 17 Jul 2024 09:40:04 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sU4t0-0000j7-Bc for qemu-devel@nongnu.org; Wed, 17 Jul 2024 09:40:02 -0400 Received: from mail-pl1-x62c.google.com ([2607:f8b0:4864:20::62c]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1sU4sx-0007TS-Nb for qemu-devel@nongnu.org; Wed, 17 Jul 2024 09:40:01 -0400 Received: by mail-pl1-x62c.google.com with SMTP id d9443c01a7336-1fc5239faebso4441335ad.1 for ; Wed, 17 Jul 2024 06:39:58 -0700 (PDT) Received: from duncan.localdomain (114-35-142-126.hinet-ip.hinet.net. [114.35.142.126]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-1fc0bc5aa6esm74903495ad.299.2024.07.17.06.39.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 17 Jul 2024 06:39:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1721223597; x=1721828397; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Z6Ehnl/PC6i+F6ruu+0LWnyhe2hlKO/B5I8PcWhvJ30=; b=WgiDzTO3lA8bwV4C5/rbaL8uK6neHuEvI99B9Uvy2dxQ+ec6GiVdTbFgCzmYkFVpJM Kwfddj3LyozTTJyWEiZrZ5M7QQLq3BHCx65z8JrnfLR9g2fHaDoR8K1pR1WKDpALPBnv 6XAviYUcB0Yqn4jlnM4/Ce8sTteyPpGznBSvxR5aiarKR2vhLQntEV9dM9BfKqvw1F/h eEREntaWO3+f+vHx62QSiPMlg0iCbOYO5nF7ZSEotihsiThs7ZWHHGiSeUKxGyMUET9w flwdfF5lUvQS+kAkt6obUrQwpwkKhydZmy4YX2pR8aCgBJ1rKKaHgKCh5NP6fyW9yonI 7P+Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721223597; x=1721828397; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Z6Ehnl/PC6i+F6ruu+0LWnyhe2hlKO/B5I8PcWhvJ30=; b=K+hFCiW8wYYBOXB12ajZqsYetw2Wb6X4NEAOuIMtlqkZpmqLNS5uHy0gbcUl6+CBTc I1BoDRTTMPVwR4ZoDNaIJXfz1bROKqBPQ2RUr2UxIy0wEurRP7CAC7L9geHlOWEghJBq ZxHDcR3HiYQczFQXLvefTyvCeum1jdBFo6cD+XBy+CE86DPv6Y3iA8MpcG5md8BGcFca YRVFtSG9kW4tc+nZgG37hkHah/U+/bkNPp2ALG239WLTXBJHkMD6brbBVtbcZainqcdf S7P+IBy+QpJg+/Cn/lbYs4TvOM7qot5Rww+CLT7DLD7Cb2ItIxf0nRxAlGorJ2cMNyWw 7/BQ== X-Gm-Message-State: AOJu0YxZYu3OR3xp8WP+oLeDJJ23CNoq4Mma18AyusYC4vwxappLR+5h n8eVmdkw+sGXv89/bWo1kTMGWb7S55kCVOzXOJd1KWUM4mt+X7LdPrKe5Oh/PJLXeNRUSsO+10T 6d1vgCib+xIBj0u4vMDtyAItjZ3Zhj0mW0R9AMBi7ZyUgxyGhdWvKeQpkx5avhS0xeoqpvE7+Ei kwgx0IAiLsVjtCqhdyZvo5EHZkpDA/sbfXnINAEA== X-Google-Smtp-Source: AGHT+IGjRLPU/NirqcygJsnHY63Eon2tLPCacB2JY4snTGbeP50DBiB2/Q1kbCiRn6LIwIDFXYoLRw== X-Received: by 2002:a17:903:1211:b0:1f6:fcd9:5b86 with SMTP id d9443c01a7336-1fc4e121a48mr13418155ad.12.1721223597024; Wed, 17 Jul 2024 06:39:57 -0700 (PDT) From: Max Chou To: qemu-devel@nongnu.org, qemu-riscv@nongnu.org Cc: Palmer Dabbelt , Alistair Francis , Bin Meng , Weiwei Li , Daniel Henrique Barboza , Liu Zhiwei , richard.henderson@linaro.org, Max Chou Subject: [RFC PATCH v5 4/5] target/riscv: rvv: Provide group continuous ld/st flow for unit-stride ld/st instructions Date: Wed, 17 Jul 2024 21:39:35 +0800 Message-Id: <20240717133936.713642-5-max.chou@sifive.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240717133936.713642-1-max.chou@sifive.com> References: <20240717133936.713642-1-max.chou@sifive.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2607:f8b0:4864:20::62c; envelope-from=max.chou@sifive.com; helo=mail-pl1-x62c.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @sifive.com) X-ZM-MESSAGEID: 1721223681078116600 Content-Type: text/plain; charset="utf-8" The vector unmasked unit-stride and whole register load/store instructions will load/store continuous memory. If the endian of both the host and guest architecture are the same, then we can group the element load/store to load/store more data at a time. Signed-off-by: Max Chou --- target/riscv/vector_helper.c | 72 +++++++++++++++++++++++++++++------- 1 file changed, 58 insertions(+), 14 deletions(-) diff --git a/target/riscv/vector_helper.c b/target/riscv/vector_helper.c index 5343a08e6ad..2e675b4220c 100644 --- a/target/riscv/vector_helper.c +++ b/target/riscv/vector_helper.c @@ -188,6 +188,40 @@ GEN_VEXT_ST_ELEM(ste_h, uint16_t, H2, stw) GEN_VEXT_ST_ELEM(ste_w, uint32_t, H4, stl) GEN_VEXT_ST_ELEM(ste_d, uint64_t, H8, stq) =20 +static inline QEMU_ALWAYS_INLINE void +vext_continus_ldst_tlb(CPURISCVState *env, vext_ldst_elem_fn_tlb *ldst_tlb, + void *vd, uint32_t evl, target_ulong addr, + uint32_t reg_start, uintptr_t ra, uint32_t esz, + bool is_load) +{ + uint32_t i; + for (i =3D env->vstart; i < evl; env->vstart =3D ++i, addr +=3D esz) { + ldst_tlb(env, adjust_addr(env, addr), i * esz, vd, ra); + } +} + +static inline QEMU_ALWAYS_INLINE void +vext_continus_ldst_host(CPURISCVState *env, vext_ldst_elem_fn_host *ldst_h= ost, + void *vd, uint32_t evl, uint32_t reg_start, void *= host, + uint32_t esz, bool is_load) +{ +#if TARGET_BIG_ENDIAN !=3D HOST_BIG_ENDIAN + for (; reg_start < evl; reg_start++, host +=3D esz) { + uint32_t byte_off =3D reg_start * esz; + ldst_host(vd, byte_off, host); + } +#else + uint32_t byte_offset =3D reg_start * esz; + uint32_t size =3D (evl - reg_start) * esz; + + if (is_load) { + memcpy(vd + byte_offset, host, size); + } else { + memcpy(host, vd + byte_offset, size); + } +#endif +} + static void vext_set_tail_elems_1s(target_ulong vl, void *vd, uint32_t desc, uint32_t nf, uint32_t esz, uint32_t max_elems) @@ -299,24 +333,34 @@ vext_page_ldst_us(CPURISCVState *env, void *vd, targe= t_ulong addr, mmu_index, true, &host, ra); =20 if (host && flags =3D=3D 0) { - for (i =3D env->vstart; i < evl; ++i) { - k =3D 0; - while (k < nf) { - ldst_host(vd, (i + k * max_elems) << log2_esz, host); - host +=3D esz; - k++; + if (nf =3D=3D 1) { + vext_continus_ldst_host(env, ldst_host, vd, evl, env->vstart, = host, + esz, is_load); + } else { + for (i =3D env->vstart; i < evl; ++i) { + k =3D 0; + while (k < nf) { + ldst_host(vd, (i + k * max_elems) << log2_esz, host); + host +=3D esz; + k++; + } } } env->vstart +=3D elems; } else { - /* load bytes from guest memory */ - for (i =3D env->vstart; i < evl; env->vstart =3D ++i) { - k =3D 0; - while (k < nf) { - ldst_tlb(env, adjust_addr(env, addr), - (i + k * max_elems) << log2_esz, vd, ra); - addr +=3D esz; - k++; + if (nf =3D=3D 1) { + vext_continus_ldst_tlb(env, ldst_tlb, vd, evl, addr, env->vsta= rt, + ra, esz, is_load); + } else { + /* load bytes from guest memory */ + for (i =3D env->vstart; i < evl; env->vstart =3D ++i) { + k =3D 0; + while (k < nf) { + ldst_tlb(env, adjust_addr(env, addr), + (i + k * max_elems) << log2_esz, vd, ra); + addr +=3D esz; + k++; + } } } } --=20 2.34.1 From nobody Mon Sep 16 19:08:48 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=reject dis=none) header.from=sifive.com ARC-Seal: i=1; a=rsa-sha256; t=1721223792; cv=none; d=zohomail.com; s=zohoarc; b=PY9qtM+pLYHSDC7IdiXgk99MpFlbXhVVHXhqgQby+6QF8pABSXbOMMZSlV0X2ET8dx6l5iAlYLaKzRKpGgKUIRByswyxSch200UaPTmVa8b3RRboz94ZLlkYnSf542QTXPkU6rwaE+TnbxVq3tJUZqDoDCj9vaBbpgMk6jQwq/8= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1721223792; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=YXRnQrR7vltoXSYXHi7IhAgNjpxq9maTne8vi5kmaO0=; b=dcOGEdvLDy6iu+1VGqlvt0Aehg/THtEfv6y0yQlVRySOAn9VNIbRV22kYY0248tZ0N/2c/GgrHDhgZfWV0mzYut7kxrEBdHB3P1vg252TY2OaP26pRH1wIsjUXum9cjBrLwbImYviKhZHVLyCwTkXwXwWUMCPftGDs4ezFAShOU= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=reject dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 17212237922181011.4562046808525; Wed, 17 Jul 2024 06:43:12 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sU4t5-00012a-5f; Wed, 17 Jul 2024 09:40:07 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sU4t2-0000sw-UD for qemu-devel@nongnu.org; Wed, 17 Jul 2024 09:40:05 -0400 Received: from mail-pg1-x530.google.com ([2607:f8b0:4864:20::530]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1sU4t1-0007Tv-0G for qemu-devel@nongnu.org; Wed, 17 Jul 2024 09:40:04 -0400 Received: by mail-pg1-x530.google.com with SMTP id 41be03b00d2f7-793a3a79a83so2210438a12.3 for ; Wed, 17 Jul 2024 06:40:01 -0700 (PDT) Received: from duncan.localdomain (114-35-142-126.hinet-ip.hinet.net. [114.35.142.126]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-1fc0bc5aa6esm74903495ad.299.2024.07.17.06.39.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 17 Jul 2024 06:39:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1721223600; x=1721828400; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=YXRnQrR7vltoXSYXHi7IhAgNjpxq9maTne8vi5kmaO0=; b=TobRxArKJEkOB3Lwmfv4UXNmkO92uBIZqpXywTZcjWz0iscPFK2liWi0jQpPy3IEiD n5eJVwsMYreTdVi9RVHfq2llzTkAHfT9TIYeADWV+VEAXZFhtpYakBh4cwsEFoXCF8HR vMcsJLfeJi+IjlGmWJtGhMLQ4rdT69iVknW6HPUNuEKuS8uzV9q7zo0DjC1He5CE5FwM 2IifLDb3qyL/ORGcK5QjsbhdfGQdrCj/SvqRl0RIkr5P83pONRphsA/JIMAJd6J1tJBD ed9glj/W189Ipu/OPVwsGcY6liXvELIofNTscHYufyHp6ipqpkPLlgmqXwQ/DcjMp4+J eooQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721223600; x=1721828400; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=YXRnQrR7vltoXSYXHi7IhAgNjpxq9maTne8vi5kmaO0=; b=bB+b4bztB8GP2wABTHoQFqQSHf4qYZzp9V9WtGmOJf/1bDequlzuavI6Rq+vWS177I soS8GiMGy2RulHzaFJsx7nNSyZ2e80FYviVDw8jqavbta56VOMYBhddAXfT20MHbUJGD Slf+Y9akjJQX6IErmcYXYeFhsp3x4nfL8yrIDWMDQ9we8WBg930V5EQ/qqYacREEzheg mPdEpAHZ+FYEbclO1788jYI3uKfz+Dj8XaxYciFjSvo4GdTqxFU3dNS4NWPTDvoDjWq/ MRgZkPrzx66CY2f+jxnClmfEHGRlAb1Oxs13eQUOzZY/m+F1Biq684q4QXZk0PspxxnY kViw== X-Gm-Message-State: AOJu0YwDmCTxS6P8aPNmLZkpe77yU8I9eom1aBbhZ4j7sLxOcby1E8R/ tFyCIqvlIfYC9napnTbTtVzllzMZStdMJ+sVCT5X9IuTFhVnEt22iLpoHAp2wBhrIBWpvTf3Vd2 j+pYrUk85Y6HY1zGGPCarV5SApKFIoDhrm6N2Hpalof/mMMRkFBcKR68uvDZJCIOzi+tIR0Rg+V h1Xy41139b4yYVZ6ku9i7owtDlZpAIpn7ZgNiwag== X-Google-Smtp-Source: AGHT+IEhFYiPKAJcmMHCQiAv/RxC9Qf0bgVvf4GhEV0jMHuHpRo9DU2lJ6A2ptelZafr7Wr+zeOkaA== X-Received: by 2002:a05:6a21:6e4b:b0:1c0:dd3d:ef3a with SMTP id adf61e73a8af0-1c3fdcd0ac1mr1943388637.29.1721223599905; Wed, 17 Jul 2024 06:39:59 -0700 (PDT) From: Max Chou To: qemu-devel@nongnu.org, qemu-riscv@nongnu.org Cc: Palmer Dabbelt , Alistair Francis , Bin Meng , Weiwei Li , Daniel Henrique Barboza , Liu Zhiwei , richard.henderson@linaro.org, Max Chou Subject: [RFC PATCH v5 5/5] target/riscv: Inline unit-stride ld/st and corresponding functions for performance Date: Wed, 17 Jul 2024 21:39:36 +0800 Message-Id: <20240717133936.713642-6-max.chou@sifive.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240717133936.713642-1-max.chou@sifive.com> References: <20240717133936.713642-1-max.chou@sifive.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2607:f8b0:4864:20::530; envelope-from=max.chou@sifive.com; helo=mail-pg1-x530.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @sifive.com) X-ZM-MESSAGEID: 1721223793594116600 Content-Type: text/plain; charset="utf-8" In the vector unit-stride load/store helper functions. the vext_ldst_us & vext_ldst_whole functions corresponding most of the execution time. Inline the functions can avoid the function call overhead to improve the helper function performance. Signed-off-by: Max Chou Reviewed-by: Richard Henderson --- target/riscv/vector_helper.c | 56 +++++++++++++++++++----------------- 1 file changed, 30 insertions(+), 26 deletions(-) diff --git a/target/riscv/vector_helper.c b/target/riscv/vector_helper.c index 2e675b4220c..95394c425ed 100644 --- a/target/riscv/vector_helper.c +++ b/target/riscv/vector_helper.c @@ -150,18 +150,20 @@ typedef void vext_ldst_elem_fn_tlb(CPURISCVState *env= , abi_ptr addr, uint32_t idx, void *vd, uintptr_t retad= dr); typedef void vext_ldst_elem_fn_host(void *vd, uint32_t idx, void *host); =20 -#define GEN_VEXT_LD_ELEM(NAME, ETYPE, H, LDSUF) \ -static void NAME##_tlb(CPURISCVState *env, abi_ptr addr, \ - uint32_t byte_off, void *vd, uintptr_t retaddr) \ -{ \ - ETYPE *cur =3D vd + byte_off; \ - *cur =3D cpu_##LDSUF##_data_ra(env, addr, retaddr); \ -} \ - \ -static void NAME##_host(void *vd, uint32_t byte_off, void *host) \ -{ \ - ETYPE val =3D LDSUF##_p(host); \ - *(ETYPE *)(vd + byte_off) =3D val; \ +#define GEN_VEXT_LD_ELEM(NAME, ETYPE, H, LDSUF) \ +static inline QEMU_ALWAYS_INLINE \ +void NAME##_tlb(CPURISCVState *env, abi_ptr addr, \ + uint32_t byte_off, void *vd, uintptr_t retaddr) \ +{ \ + ETYPE *cur =3D vd + byte_off; \ + *cur =3D cpu_##LDSUF##_data_ra(env, addr, retaddr); \ +} \ + \ +static inline QEMU_ALWAYS_INLINE \ +void NAME##_host(void *vd, uint32_t byte_off, void *host) \ +{ \ + ETYPE val =3D LDSUF##_p(host); \ + *(ETYPE *)(vd + byte_off) =3D val; \ } =20 GEN_VEXT_LD_ELEM(lde_b, uint8_t, H1, ldub) @@ -169,18 +171,20 @@ GEN_VEXT_LD_ELEM(lde_h, uint16_t, H2, lduw) GEN_VEXT_LD_ELEM(lde_w, uint32_t, H4, ldl) GEN_VEXT_LD_ELEM(lde_d, uint64_t, H8, ldq) =20 -#define GEN_VEXT_ST_ELEM(NAME, ETYPE, H, STSUF) \ -static void NAME##_tlb(CPURISCVState *env, abi_ptr addr, \ - uint32_t byte_off, void *vd, uintptr_t retaddr) \ -{ \ - ETYPE data =3D *(ETYPE *)(vd + byte_off); \ - cpu_##STSUF##_data_ra(env, addr, data, retaddr); \ -} \ - \ -static void NAME##_host(void *vd, uint32_t byte_off, void *host) \ -{ \ - ETYPE val =3D *(ETYPE *)(vd + byte_off); \ - STSUF##_p(host, val); \ +#define GEN_VEXT_ST_ELEM(NAME, ETYPE, H, STSUF) \ +static inline QEMU_ALWAYS_INLINE \ +void NAME##_tlb(CPURISCVState *env, abi_ptr addr, \ + uint32_t byte_off, void *vd, uintptr_t retaddr) \ +{ \ + ETYPE data =3D *(ETYPE *)(vd + byte_off); \ + cpu_##STSUF##_data_ra(env, addr, data, retaddr); \ +} \ + \ +static inline QEMU_ALWAYS_INLINE \ +void NAME##_host(void *vd, uint32_t byte_off, void *host) \ +{ \ + ETYPE val =3D *(ETYPE *)(vd + byte_off); \ + STSUF##_p(host, val); \ } =20 GEN_VEXT_ST_ELEM(ste_b, uint8_t, H1, stb) @@ -366,7 +370,7 @@ vext_page_ldst_us(CPURISCVState *env, void *vd, target_= ulong addr, } } =20 -static void +static inline QEMU_ALWAYS_INLINE void vext_ldst_us(void *vd, target_ulong base, CPURISCVState *env, uint32_t des= c, vext_ldst_elem_fn_tlb *ldst_tlb, vext_ldst_elem_fn_host *ldst_host, uint32_t log2_esz, @@ -695,7 +699,7 @@ GEN_VEXT_LDFF(vle64ff_v, int64_t, lde_d_tlb) /* * load and store whole register instructions */ -static void +static inline QEMU_ALWAYS_INLINE void vext_ldst_whole(void *vd, target_ulong base, CPURISCVState *env, uint32_t = desc, vext_ldst_elem_fn_tlb *ldst_tlb, vext_ldst_elem_fn_host *ldst_host, uint32_t log2_esz, --=20 2.34.1