From nobody Wed Dec 17 03:07:16 2025 Received: from mail-wm1-f53.google.com (mail-wm1-f53.google.com [209.85.128.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 62F7419CD1D for ; Mon, 21 Apr 2025 14:26:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.53 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745245589; cv=none; b=llyA/6VzKxky7KERMpw469/sbj98B3Nj+AOJ3Pths8TO0yeGiRekpBvlYXYKq6kYpIydh0TyrWaoTrSflOcKLvRt+m0pOq6v9GcAqXyD13z+ZzMrWa0i7j/2OVGpXeZcSvKL10brAz006aaeJWDqu31hXxTGpqKmF8bMB63jaZ0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745245589; c=relaxed/simple; bh=Utwokb7er4q5JknyfiZW2i0dXjwbYWTz3mwbW9uIn3o=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=d1w3Uu3IXBb6ora9FHPflgeOgD/sgyZT3ds73wJ/94o4o7QaSYR98xpHkoiNG7zbACannX33/lfs7ze1+eu2BSXrLrLu0IyJep/vvE+1ErtQBOOku2Xx822SWM2sxrTaDOxI8NwWfCcUzWfRtNRfNfVJwzEx8dNUaKh6Su8ueB8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com; spf=pass smtp.mailfrom=rivosinc.com; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b=hzrHaTIQ; arc=none smtp.client-ip=209.85.128.53 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b="hzrHaTIQ" Received: by mail-wm1-f53.google.com with SMTP id 5b1f17b1804b1-43cebe06e9eso33658835e9.3 for ; Mon, 21 Apr 2025 07:26:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1745245585; x=1745850385; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=r8EzXGFQbwSP2O8EyG3ctzvPoUfJEY6y+SbPIPc5xJE=; b=hzrHaTIQvtTYKqt0N6zcmzNroJcC3zz0wH3fo/vYL9bxk8MVj/2NiMYwJhEzd4wIei mfUV1xQ5MCpyvYssCRNZEu9G7BTL9+ksE2vDjZI+U7iLfTOMy0+25FqtAon0DbHCtmIK pOiOfFfY76yYaHajyL4w71IGfs4QpWT3nhHlZAJNENl5qGm5piDtAATY3/48dEPolA/l +CHxYTOc2C/WHwrxwqjO7zSlDxAmGBBaxTVU5wIIOQEdjiiTs3cpJo8xXQKV6MlJzA0p RXm0VySbnY9kjzTOnpgbTV0iIORX1QmQJ9qRGexnhT7ue7lc3icH5I5AuKvJOGKyubwx vwZg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1745245585; x=1745850385; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=r8EzXGFQbwSP2O8EyG3ctzvPoUfJEY6y+SbPIPc5xJE=; b=b/iYpkLFTf7/fH6PsVWpxkxZBYTjAwzlmbcH3vzYBSv8Wbt0DITD1CvjNRnnrLfsu/ 6Bw9M0Dfliiw5bzyWNc2fgLVTkvuFSsHAGWr1OAawKXE+1IZ3bdvvw5Avo+48cQ9KVUj 7kYhHYSNUmwtULwLL0qgMqVOLqVvUm0786t70G/wZNyBIeBHo+gbFldfavCHHr0k28uH c1lngwnJeFrAhptNu5qEPsbTDc5BOsXnSzGUhaRDsdP1fCa4V0dO5xj3l/OoTtcAmJ2w 0u8xe+imfA6fdrIZi1yjXvd3ZUAkBCKFQjRV5lMkAEgK5AyfTbEGHkCNiX2o4v1h1ZrT qadQ== X-Forwarded-Encrypted: i=1; AJvYcCV7B+DkUwBNTpy4P+mQcuHbWWhdlHyiZP0iQRAVwO9HpsJ35mTJyIMTKsFuw2OoHGWIV5gODP8na5AXk5Q=@vger.kernel.org X-Gm-Message-State: AOJu0Yx0HCSf+4UoHGCxNSGoQgp1EZyApU+cDBpkgplpqL+0l8S19bNH 8u4DzxGpGxWi/EGAebfIH8rm6y4vnksjtkQVsYktM+J3srCTuVR37EctcVAwDlg= X-Gm-Gg: ASbGncsKFbi6jnoGpyn1XUhnwMbBua1vb9j/KD6xCdZaCg8oEud1tuxXpQTB/vXdRWo qFpjyar2AU9iHqlcsocooKF7XxRY72U22pTkLbZAqAmTEYoozYymxWsm6MRdH2N9lod2FoStCa9 nvzjayatcAeloLAbv2v6HgcBjmYiC3k9l0g8O/16WVyd88ma2ZLDnQ3cIay9Nu3QC5bt4eosdep bruoKuYwqxLuit5fD8va34EqIUlS7tivhSD38ycm2agCBfCn0icsHrV+tUphTE7KazcvGXz7Fqk DDqCbGlN1EGpA/PBsXh/UfZdA939U5OinPnRxYqr7e+8EoN9r8DIwNZ9MhE= X-Google-Smtp-Source: AGHT+IGbR8cH86R+FPRiB//fxh3ANX3cYxfoHYfa1pDXh1hmpUYRDkZBnNT8HxNX2o8g7qT41lDP+Q== X-Received: by 2002:a05:600c:1e14:b0:43d:abd:ad0e with SMTP id 5b1f17b1804b1-4406aba617dmr115364515e9.18.1745245585543; Mon, 21 Apr 2025 07:26:25 -0700 (PDT) Received: from localhost.localdomain ([2001:861:3382:ef90:cd58:9e76:e88c:5cd]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-4406d5a9ed9sm139610375e9.3.2025.04.21.07.26.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 21 Apr 2025 07:26:25 -0700 (PDT) From: Alexandre Ghiti To: Paul Walmsley , Palmer Dabbelt , Alexandre Ghiti , linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org Cc: Alexandre Ghiti , Guo Ren Subject: [PATCH v3 1/4] riscv: Introduce Zicbop instructions Date: Mon, 21 Apr 2025 16:24:38 +0200 Message-Id: <20250421142441.395849-2-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20250421142441.395849-1-alexghiti@rivosinc.com> References: <20250421142441.395849-1-alexghiti@rivosinc.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The S-type instructions are first introduced and then used to define the encoding of the Zicbop prefetching instructions. Co-developed-by: Guo Ren Signed-off-by: Guo Ren Signed-off-by: Alexandre Ghiti Tested-by: Andrea Parri --- arch/riscv/include/asm/insn-def.h | 60 +++++++++++++++++++++++++++++++ 1 file changed, 60 insertions(+) diff --git a/arch/riscv/include/asm/insn-def.h b/arch/riscv/include/asm/ins= n-def.h index 71060a2f838e..02c92c1657d2 100644 --- a/arch/riscv/include/asm/insn-def.h +++ b/arch/riscv/include/asm/insn-def.h @@ -18,6 +18,13 @@ #define INSN_I_RD_SHIFT 7 #define INSN_I_OPCODE_SHIFT 0 =20 +#define INSN_S_SIMM7_SHIFT 25 +#define INSN_S_RS2_SHIFT 20 +#define INSN_S_RS1_SHIFT 15 +#define INSN_S_FUNC3_SHIFT 12 +#define INSN_S_SIMM5_SHIFT 7 +#define INSN_S_OPCODE_SHIFT 0 + #ifdef __ASSEMBLY__ =20 #ifdef CONFIG_AS_HAS_INSN @@ -30,6 +37,10 @@ .insn i \opcode, \func3, \rd, \rs1, \simm12 .endm =20 + .macro insn_s, opcode, func3, rs2, simm12, rs1 + .insn s \opcode, \func3, \rs2, \simm12(\rs1) + .endm + #else =20 #include @@ -51,10 +62,20 @@ (\simm12 << INSN_I_SIMM12_SHIFT)) .endm =20 + .macro insn_s, opcode, func3, rs2, simm12, rs1 + .4byte ((\opcode << INSN_S_OPCODE_SHIFT) | \ + (\func3 << INSN_S_FUNC3_SHIFT) | \ + (.L__gpr_num_\rs2 << INSN_S_RS2_SHIFT) | \ + (.L__gpr_num_\rs1 << INSN_S_RS1_SHIFT) | \ + ((\simm12 & 0x1f) << INSN_S_SIMM5_SHIFT) | \ + (((\simm12 >> 5) & 0x7f) << INSN_S_SIMM7_SHIFT)) + .endm + #endif =20 #define __INSN_R(...) insn_r __VA_ARGS__ #define __INSN_I(...) insn_i __VA_ARGS__ +#define __INSN_S(...) insn_s __VA_ARGS__ =20 #else /* ! __ASSEMBLY__ */ =20 @@ -66,6 +87,9 @@ #define __INSN_I(opcode, func3, rd, rs1, simm12) \ ".insn i " opcode ", " func3 ", " rd ", " rs1 ", " simm12 "\n" =20 +#define __INSN_S(opcode, func3, rs2, simm12, rs1) \ + ".insn s " opcode ", " func3 ", " rs2 ", " simm12 "(" rs1 ")\n" + #else =20 #include @@ -92,12 +116,26 @@ " (\\simm12 << " __stringify(INSN_I_SIMM12_SHIFT) "))\n" \ " .endm\n" =20 +#define DEFINE_INSN_S \ + __DEFINE_ASM_GPR_NUMS \ +" .macro insn_s, opcode, func3, rs2, simm12, rs1\n" \ +" .4byte ((\\opcode << " __stringify(INSN_S_OPCODE_SHIFT) ") |" \ +" (\\func3 << " __stringify(INSN_S_FUNC3_SHIFT) ") |" \ +" (.L__gpr_num_\\rs2 << " __stringify(INSN_S_RS2_SHIFT) ") |" \ +" (.L__gpr_num_\\rs1 << " __stringify(INSN_S_RS1_SHIFT) ") |" \ +" ((\\simm12 & 0x1f) << " __stringify(INSN_S_SIMM5_SHIFT) ") |" \ +" (((\\simm12 >> 5) & 0x7f) << " __stringify(INSN_S_SIMM7_SHIFT) "))\n" \ +" .endm\n" + #define UNDEFINE_INSN_R \ " .purgem insn_r\n" =20 #define UNDEFINE_INSN_I \ " .purgem insn_i\n" =20 +#define UNDEFINE_INSN_S \ +" .purgem insn_s\n" + #define __INSN_R(opcode, func3, func7, rd, rs1, rs2) \ DEFINE_INSN_R \ "insn_r " opcode ", " func3 ", " func7 ", " rd ", " rs1 ", " rs2 "\n" \ @@ -108,6 +146,11 @@ "insn_i " opcode ", " func3 ", " rd ", " rs1 ", " simm12 "\n" \ UNDEFINE_INSN_I =20 +#define __INSN_S(opcode, func3, rs2, simm12, rs1) \ + DEFINE_INSN_S \ + "insn_s " opcode ", " func3 ", " rs2 ", " simm12 ", " rs1 "\n" \ + UNDEFINE_INSN_S + #endif =20 #endif /* ! __ASSEMBLY__ */ @@ -120,6 +163,10 @@ __INSN_I(RV_##opcode, RV_##func3, RV_##rd, \ RV_##rs1, RV_##simm12) =20 +#define INSN_S(opcode, func3, rs2, simm12, rs1) \ + __INSN_S(RV_##opcode, RV_##func3, RV_##rs2, \ + RV_##simm12, RV_##rs1) + #define RV_OPCODE(v) __ASM_STR(v) #define RV_FUNC3(v) __ASM_STR(v) #define RV_FUNC7(v) __ASM_STR(v) @@ -133,6 +180,7 @@ #define RV___RS2(v) __RV_REG(v) =20 #define RV_OPCODE_MISC_MEM RV_OPCODE(15) +#define RV_OPCODE_OP_IMM RV_OPCODE(19) #define RV_OPCODE_SYSTEM RV_OPCODE(115) =20 #define HFENCE_VVMA(vaddr, asid) \ @@ -196,6 +244,18 @@ INSN_I(OPCODE_MISC_MEM, FUNC3(2), __RD(0), \ RS1(base), SIMM12(4)) =20 +#define PREFETCH_I(base, offset) \ + INSN_S(OPCODE_OP_IMM, FUNC3(6), __RS2(0), \ + SIMM12((offset) & 0xfe0), RS1(base)) + +#define PREFETCH_R(base, offset) \ + INSN_S(OPCODE_OP_IMM, FUNC3(6), __RS2(1), \ + SIMM12((offset) & 0xfe0), RS1(base)) + +#define PREFETCH_W(base, offset) \ + INSN_S(OPCODE_OP_IMM, FUNC3(6), __RS2(3), \ + SIMM12((offset) & 0xfe0), RS1(base)) + #define RISCV_PAUSE ".4byte 0x100000f" #define ZAWRS_WRS_NTO ".4byte 0x00d00073" #define ZAWRS_WRS_STO ".4byte 0x01d00073" --=20 2.39.2 From nobody Wed Dec 17 03:07:16 2025 Received: from mail-wr1-f45.google.com (mail-wr1-f45.google.com [209.85.221.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 075D718E362 for ; Mon, 21 Apr 2025 14:27:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.45 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745245651; cv=none; b=XS7ilp5ScZ7DxjF9XW6hBn+IyacPH9VqsNndI0Y1ONelwCphqxegStEjjXb5PM9fk0REMBrFlZWvr+WIOdfiIwLqzqD3BP71lSqQHO4t7Ch5BANjZV3DW0Jg51shgUhQ+rVsWH6lRyPZecgSXqFCl0E/DugiD4sVhwAfSKHS7ik= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745245651; c=relaxed/simple; bh=bkfvxlflX4M9GtG+8EbDbpAdx6dfIhOdaSWgrzBRUPg=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=OxWO34Cuj4xLATIpHmUNfKTva1hsLnzlOA1UlDWuzXeJoZ5vWkU6W8u0Z8h33TY0ZcrL/ofyiX/Y8SZMfgKDUsm6MUReIrKHBtByubR3Ll2+6GOmbOMd4fX3UbomPkONTvWeJi2lJq9mvfPziBPcIlMEy796iSK/Vz96gCnEuPs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com; spf=pass smtp.mailfrom=rivosinc.com; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b=PawZ2fOf; arc=none smtp.client-ip=209.85.221.45 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b="PawZ2fOf" Received: by mail-wr1-f45.google.com with SMTP id ffacd0b85a97d-39c266c1389so2945958f8f.1 for ; Mon, 21 Apr 2025 07:27:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1745245647; x=1745850447; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=zM2xTA3Ijm3UBkqygOS2aJTpcgU5eK9eWAc69Lekkw4=; b=PawZ2fOfrLROIIGEAWj7+IGPUChanXZ9vb2oQabSmHK0OxNN5AIwCWLWitZKKA/4ab UeTNR5haNCOW7JNVJ+SNuekvveZEpljd8AN/QTuhB0GDorPtyjEYljY9XhK+bCE1OhkF wN4pB5UyRCp34ovSplxTyJc+LEGRRq2hqNKJ0fej9b6GPUVGLXkQxaUu/P1eKmfrbmMF OnbwxUKdDRfxJsAL9gpAuON8PaJNKAc2yj5TVR+Wv6UFL/g+5Ggqz3ALgYOmnAU9jaXP clt8PPi+97AueU1inZ1NCZZrU80UsyiZv4UFyuIPBo9x0f6OJUr0QlslCwVksZ77M1wR B9xg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1745245647; x=1745850447; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=zM2xTA3Ijm3UBkqygOS2aJTpcgU5eK9eWAc69Lekkw4=; b=xLGS178NfqBnN7q2ypM+q9ieDk4/evM0QkeTr6eu6t5ML0nXWNurSUiIqEp86FN2KV /++xToOHO6YOerQG68eCoGVGz1UZlLIgobxDoigpkncf6l8+dG1hnb4w0y6zo4ZImcsZ Bl2/0FWhIS97U/jtc3xUP8zErhxP7OZssFuwqV16zPG7OiivNLS2x+Hq+ddpVo/sjrUQ mMy2/kmi2GDPply4drEgMkBOcNDcGwp9G8oXEHinLwiiulueElTLEqAD5SnBeZaqAAuE P70ltCvL9yF2yctJ+qJdRjdbzBLzMzxuPuetRkGd5hJsC5731rT1eTWlZWSK1D3kIGY4 YJ/g== X-Forwarded-Encrypted: i=1; AJvYcCWxzSZ3L8g8cf92S54bSEF8XfM8TafcuTc7dwrsvew2VtI3M9e+TKWo7HhxDwMFurORl4zvJv+eeJSwEak=@vger.kernel.org X-Gm-Message-State: AOJu0Yzl+cBoRmlMzDvwiUDfcF8QyhuviuW1xIFEPXcdwldLVKWm0U4Y tWGDhIC2ZBXEtNK3T3N2EuImLsjV6dqjUqalHSOZsRPaSKVQe/fgFu51vn4bmWM= X-Gm-Gg: ASbGncsS6co+PVAp2Tg3Ia8PwUZS2QND2HKtxAo78t+4P9MBvWrGgZmnIOYaREMZdzG GxkuzjzVADgqmb6FonEtEiijW0CXzcsyE1E3V7QEbdoMdleYy2Iqg5BOK3LeehJniD1F1UTxnag ue7pwO0eYfYTuHkR10cf4pmacuiywWJQx1pdmhgWWNyuU4urySDaFmzr5XwP/83yuLAxeiszBb/ Vl1kM+3cj2S2NZo+He7LrGndrW3fx1zKYz+0urSrsomdJ2sHrKcakLrSRcGA1ljPBcfj+YJdbdF j2csivEiDihUNQOc7GkdrafEYOgp6KW86Erg9h0Bk8Sso016vnagIEOpmB8= X-Google-Smtp-Source: AGHT+IFtp6uDvT40Wf+eeybtMtOJKhVDujKLqASqoA/CInEEp1l3aj2FR72x/bhE1vGyIsJOBUURFA== X-Received: by 2002:a05:6000:706:b0:39c:1257:dbab with SMTP id ffacd0b85a97d-39efbaf26f0mr7662645f8f.59.1745245647104; Mon, 21 Apr 2025 07:27:27 -0700 (PDT) Received: from localhost.localdomain ([2001:861:3382:ef90:cd58:9e76:e88c:5cd]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-39efa5a2300sm11840020f8f.101.2025.04.21.07.27.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 21 Apr 2025 07:27:26 -0700 (PDT) From: Alexandre Ghiti To: Paul Walmsley , Palmer Dabbelt , Alexandre Ghiti , linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org Cc: Alexandre Ghiti , Guo Ren Subject: [PATCH v3 2/4] riscv: Add support for Zicbop Date: Mon, 21 Apr 2025 16:24:39 +0200 Message-Id: <20250421142441.395849-3-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20250421142441.395849-1-alexghiti@rivosinc.com> References: <20250421142441.395849-1-alexghiti@rivosinc.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Zicbop introduces cache blocks prefetching instructions, add the necessary support for the kernel to use it in the coming commits. Co-developed-by: Guo Ren Signed-off-by: Guo Ren Signed-off-by: Alexandre Ghiti Tested-by: Andrea Parri --- arch/riscv/Kconfig | 15 +++++++++++++++ arch/riscv/include/asm/barrier.h | 5 ----- arch/riscv/include/asm/cacheflush.h | 1 + arch/riscv/include/asm/hwcap.h | 1 + arch/riscv/include/asm/insn-def.h | 6 ++++++ arch/riscv/include/asm/processor.h | 1 - arch/riscv/kernel/cpufeature.c | 21 +++++++++++++++++++++ arch/riscv/mm/cacheflush.c | 14 +++++++++++--- 8 files changed, 55 insertions(+), 9 deletions(-) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index bbec87b79309..28765ce563de 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -842,6 +842,21 @@ config RISCV_ISA_ZICBOZ =20 If you don't know what to do here, say Y. =20 +config RISCV_ISA_ZICBOP + bool "Zicbop extension support for cache block prefetch" + depends on MMU + depends on RISCV_ALTERNATIVE + default y + help + Adds support to dynamically detect the presence of the ZICBOP + extension (Cache Block Prefetch Operations) and enable its + usage. + + The Zicbop extension can be used to prefetch cache blocks for + read/write fetch. + + If you don't know what to do here, say Y. + config TOOLCHAIN_NEEDS_EXPLICIT_ZICSR_ZIFENCEI def_bool y # https://sourceware.org/git/?p=3Dbinutils-gdb.git;a=3Dcommit;h=3Daed4428= 6efa8ae8717a77d94b51ac3614e2ca6dc diff --git a/arch/riscv/include/asm/barrier.h b/arch/riscv/include/asm/barr= ier.h index e1d9bf1deca6..b8c5726d86ac 100644 --- a/arch/riscv/include/asm/barrier.h +++ b/arch/riscv/include/asm/barrier.h @@ -14,11 +14,6 @@ #include #include =20 -#define nop() __asm__ __volatile__ ("nop") -#define __nops(n) ".rept " #n "\nnop\n.endr\n" -#define nops(n) __asm__ __volatile__ (__nops(n)) - - /* These barriers need to enforce ordering on both devices or memory. */ #define __mb() RISCV_FENCE(iorw, iorw) #define __rmb() RISCV_FENCE(ir, ir) diff --git a/arch/riscv/include/asm/cacheflush.h b/arch/riscv/include/asm/c= acheflush.h index 8de73f91bfa3..effa02c2e682 100644 --- a/arch/riscv/include/asm/cacheflush.h +++ b/arch/riscv/include/asm/cacheflush.h @@ -80,6 +80,7 @@ void flush_icache_mm(struct mm_struct *mm, bool local); =20 extern unsigned int riscv_cbom_block_size; extern unsigned int riscv_cboz_block_size; +extern unsigned int riscv_cbop_block_size; void riscv_init_cbo_blocksizes(void); =20 #ifdef CONFIG_RISCV_DMA_NONCOHERENT diff --git a/arch/riscv/include/asm/hwcap.h b/arch/riscv/include/asm/hwcap.h index e3cbf203cdde..affd63e11b0a 100644 --- a/arch/riscv/include/asm/hwcap.h +++ b/arch/riscv/include/asm/hwcap.h @@ -105,6 +105,7 @@ #define RISCV_ISA_EXT_ZVFBFWMA 96 #define RISCV_ISA_EXT_ZAAMO 97 #define RISCV_ISA_EXT_ZALRSC 98 +#define RISCV_ISA_EXT_ZICBOP 99 =20 #define RISCV_ISA_EXT_XLINUXENVCFG 127 =20 diff --git a/arch/riscv/include/asm/insn-def.h b/arch/riscv/include/asm/ins= n-def.h index 02c92c1657d2..d5adbaec1d01 100644 --- a/arch/riscv/include/asm/insn-def.h +++ b/arch/riscv/include/asm/insn-def.h @@ -263,4 +263,10 @@ =20 #define RISCV_INSN_NOP4 _AC(0x00000013, U) =20 +#ifndef __ASSEMBLY__ +#define nop() __asm__ __volatile__ ("nop") +#define __nops(n) ".rept " #n "\nnop\n.endr\n" +#define nops(n) __asm__ __volatile__ (__nops(n)) +#endif + #endif /* __ASM_INSN_DEF_H */ diff --git a/arch/riscv/include/asm/processor.h b/arch/riscv/include/asm/pr= ocessor.h index 5f56eb9d114a..09d4c963399a 100644 --- a/arch/riscv/include/asm/processor.h +++ b/arch/riscv/include/asm/processor.h @@ -52,7 +52,6 @@ #endif =20 #ifndef __ASSEMBLY__ -#include =20 struct task_struct; struct pt_regs; diff --git a/arch/riscv/kernel/cpufeature.c b/arch/riscv/kernel/cpufeature.c index 2054f6c4b0ae..743d53415572 100644 --- a/arch/riscv/kernel/cpufeature.c +++ b/arch/riscv/kernel/cpufeature.c @@ -32,6 +32,7 @@ #define NUM_ALPHA_EXTS ('z' - 'a' + 1) =20 static bool any_cpu_has_zicboz; +static bool any_cpu_has_zicbop; static bool any_cpu_has_zicbom; =20 unsigned long elf_hwcap __read_mostly; @@ -119,6 +120,21 @@ static int riscv_ext_zicboz_validate(const struct risc= v_isa_ext_data *data, return 0; } =20 +static int riscv_ext_zicbop_validate(const struct riscv_isa_ext_data *data, + const unsigned long *isa_bitmap) +{ + if (!riscv_cbop_block_size) { + pr_err("Zicbop detected in ISA string, disabling as no cbop-block-size f= ound\n"); + return -EINVAL; + } + if (!is_power_of_2(riscv_cbop_block_size)) { + pr_err("Zicbop disabled as cbop-block-size present, but is not a power-o= f-2\n"); + return -EINVAL; + } + any_cpu_has_zicbop =3D true; + return 0; +} + static int riscv_ext_f_validate(const struct riscv_isa_ext_data *data, const unsigned long *isa_bitmap) { @@ -442,6 +458,7 @@ const struct riscv_isa_ext_data riscv_isa_ext[] =3D { __RISCV_ISA_EXT_SUPERSET_VALIDATE(v, RISCV_ISA_EXT_v, riscv_v_exts, riscv= _ext_vector_float_validate), __RISCV_ISA_EXT_DATA(h, RISCV_ISA_EXT_h), __RISCV_ISA_EXT_SUPERSET_VALIDATE(zicbom, RISCV_ISA_EXT_ZICBOM, riscv_xli= nuxenvcfg_exts, riscv_ext_zicbom_validate), + __RISCV_ISA_EXT_DATA_VALIDATE(zicbop, RISCV_ISA_EXT_ZICBOP, riscv_ext_zic= bop_validate), __RISCV_ISA_EXT_SUPERSET_VALIDATE(zicboz, RISCV_ISA_EXT_ZICBOZ, riscv_xli= nuxenvcfg_exts, riscv_ext_zicboz_validate), __RISCV_ISA_EXT_DATA(ziccrse, RISCV_ISA_EXT_ZICCRSE), __RISCV_ISA_EXT_DATA(zicntr, RISCV_ISA_EXT_ZICNTR), @@ -1112,6 +1129,10 @@ void __init riscv_user_isa_enable(void) current->thread.envcfg |=3D ENVCFG_CBCFE; else if (any_cpu_has_zicbom) pr_warn("Zicbom disabled as it is unavailable on some harts\n"); + + if (!riscv_has_extension_unlikely(RISCV_ISA_EXT_ZICBOP) && + any_cpu_has_zicbop) + pr_warn("Zicbop disabled as it is unavailable on some harts\n"); } =20 #ifdef CONFIG_RISCV_ALTERNATIVE diff --git a/arch/riscv/mm/cacheflush.c b/arch/riscv/mm/cacheflush.c index b81672729887..6265052ef8b6 100644 --- a/arch/riscv/mm/cacheflush.c +++ b/arch/riscv/mm/cacheflush.c @@ -101,6 +101,9 @@ EXPORT_SYMBOL_GPL(riscv_cbom_block_size); unsigned int riscv_cboz_block_size; EXPORT_SYMBOL_GPL(riscv_cboz_block_size); =20 +unsigned int riscv_cbop_block_size; +EXPORT_SYMBOL_GPL(riscv_cbop_block_size); + static void __init cbo_get_block_size(struct device_node *node, const char *name, u32 *block_size, unsigned long *first_hartid) @@ -125,8 +128,8 @@ static void __init cbo_get_block_size(struct device_nod= e *node, =20 void __init riscv_init_cbo_blocksizes(void) { - unsigned long cbom_hartid, cboz_hartid; - u32 cbom_block_size =3D 0, cboz_block_size =3D 0; + unsigned long cbom_hartid, cboz_hartid, cbop_hartid; + u32 cbom_block_size =3D 0, cboz_block_size =3D 0, cbop_block_size =3D 0; struct device_node *node; struct acpi_table_header *rhct; acpi_status status; @@ -138,13 +141,15 @@ void __init riscv_init_cbo_blocksizes(void) &cbom_block_size, &cbom_hartid); cbo_get_block_size(node, "riscv,cboz-block-size", &cboz_block_size, &cboz_hartid); + cbo_get_block_size(node, "riscv,cbop-block-size", + &cbop_block_size, &cbop_hartid); } } else { status =3D acpi_get_table(ACPI_SIG_RHCT, 0, &rhct); if (ACPI_FAILURE(status)) return; =20 - acpi_get_cbo_block_size(rhct, &cbom_block_size, &cboz_block_size, NULL); + acpi_get_cbo_block_size(rhct, &cbom_block_size, &cboz_block_size, &cbop_= block_size); acpi_put_table((struct acpi_table_header *)rhct); } =20 @@ -153,6 +158,9 @@ void __init riscv_init_cbo_blocksizes(void) =20 if (cboz_block_size) riscv_cboz_block_size =3D cboz_block_size; + + if (cbop_block_size) + riscv_cbop_block_size =3D cbop_block_size; } =20 #ifdef CONFIG_SMP --=20 2.39.2 From nobody Wed Dec 17 03:07:16 2025 Received: from mail-wm1-f43.google.com (mail-wm1-f43.google.com [209.85.128.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 83A8A13E02D for ; Mon, 21 Apr 2025 14:28:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.43 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745245712; cv=none; b=BP7lSZVi/9FtMSMT8SSFKDqtZM8pMgkd2atxH0e30YrAjzPWRggd0v1rwog8kgfLEBYhJceGhOG/U3ecLEx0QPbxArQZtBWnTrZawqvPt8YpcdttHytHz44Esd2d5lRvHfJLPii5C0r+xMpSl5JOEQWAlV/YFzJl81oTQbCLBVY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745245712; c=relaxed/simple; bh=UrUXHDgH8D7X0VsVKQ4vLuQBSFYiJsoAioszkD948cg=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=RJtXFwoPe0DVIy5ROtXMt4J+cvVrZWcD8MK2oUfEph7z0cx88bWFF1IOlMpEEZuR4bDAEPjmGBRH0BmqbeRH3p0jn4UQdTBkJ7uvzQ2qWx8SZHNJC/5UDq3VTNarjy6xv8Exi9/E4z4AZH63QKMfGV28lq0xAjLDD1lSjl8hoq8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com; spf=pass smtp.mailfrom=rivosinc.com; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b=FyJ4/9qW; arc=none smtp.client-ip=209.85.128.43 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b="FyJ4/9qW" Received: by mail-wm1-f43.google.com with SMTP id 5b1f17b1804b1-43cf0d787eeso48655585e9.3 for ; Mon, 21 Apr 2025 07:28:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1745245709; x=1745850509; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=fgXsecntwfRSGjrcDXCs8qD6E3/9J+RIf148wqrIpWA=; b=FyJ4/9qWnsl8Wgpu7VMF2QVKhEX5xTbbyHc7s7gpsZLzkKkTmxmrXhpEmsgUA+RE8e FYt6T9K6ozohuAg7aUiFnstuytdhOXKGw5dwa9f+GTJMOEXwoxiscW2IPATO+TEGv8e9 jXE5Vh59G8DZTcQ0g4UjGa9erPN9nf2fv8riJTUPCr12TyGNMoz1pxNj1ahQwoaK2cVe LeCMq3VR4uEV8BBtGIG07Z8BhVPxkkR3nOc/HvXkVreHCFM/N/BDOgTZhrL3iR0ZEFQB LeonTGE13wMqkuBpmBjPkYjJt1KdB8ecpyEAZxS7E5FqCDpcXuJ5xefjLtfOe4qOQc2o EYSg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1745245709; x=1745850509; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=fgXsecntwfRSGjrcDXCs8qD6E3/9J+RIf148wqrIpWA=; b=o8Q2UsC2k9TaGGVXnbjx7SLNA02PNBYhMqmCk6s/vVWGRQH0xzAG8sYOnWdLFwzphI 9uNf2jJxgv/Rb0LI4mekNanBvQTuuHg13G/he0XHj2nq2skP5+ZjL0lIzBPSZKtg9oWK omsVWJUifqDEszHoaeSQIiOpwKUwueYC5gwWdsyeqF3muk0LeUxLTzWj2ka/Hv9eaLIM 50Z22gyCbKM6M21XSemBSJY0KIV7q36MUpt0Qyr6O8FsBuleyXCbAwD1/7aONftBaDnb npti9b8IVVmGr28XYZweIp6GL6Vs1SIs6rfDBnbUTZUNDX1XiBX/Mvvzx1wS9p7RjZwE JM9Q== X-Forwarded-Encrypted: i=1; AJvYcCVtv/jVa8LQQmVHDh1VlCd7rw4cnVLHFVUqfGQW+S0Gw6zHZNp1u8lM8IAL88ky8vYYa9w8mEHFdiYXShs=@vger.kernel.org X-Gm-Message-State: AOJu0YwDfBs83xaQTMqGaRLNog1qTRMgouu77GigXKtFU8JA2nIvu1iT /hORzqUsoeZrR1X5SDOhz30yiDYlAj3yzSEh2yWbka1erpjLV3tlIpZWVV2nUUU= X-Gm-Gg: ASbGncvLZkwlMDDpQrjEltgPfRESna8nE0YcA/ly69ssgdZ5+Hy22FcnEMQKHv5asVp dDyAYzNv0Mm7Z7/+lRcta4u1AyKscOjf9AdhM8yxNPzKBRlNbtGNik3+nYhXLo8niJWZ/AoVCcC jEHMtYX4pp02sQDjFHpz+z1/STadEGMQ2Hd2J/4uSwFPJkJOotA+ywPSpPe9V9k7eElBXe7gEbY krspLFjLw6Lw991gvcfx1cx3amZX5/NCdCLYTVOErVt0aia1Np0ndcE6HahfJrJR6qpnDKhHpZv AxQTRPfyY69lTvUWSGkmHJL0zRrJIe+XU/xho7mw76RavVRaVG6sJZYLu+Q= X-Google-Smtp-Source: AGHT+IF004bhFEXPI3CCJKmdwXsLTAbZYWv5EdIm3QZ5L16mdpCbU2yQOcrSdZib/PGDekiXc//UtA== X-Received: by 2002:a05:600c:8411:b0:43d:83a:417d with SMTP id 5b1f17b1804b1-4406ab944edmr112676395e9.12.1745245708719; Mon, 21 Apr 2025 07:28:28 -0700 (PDT) Received: from localhost.localdomain ([2001:861:3382:ef90:cd58:9e76:e88c:5cd]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-4406d5a9e1bsm141981795e9.8.2025.04.21.07.28.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 21 Apr 2025 07:28:28 -0700 (PDT) From: Alexandre Ghiti To: Paul Walmsley , Palmer Dabbelt , Alexandre Ghiti , linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org Cc: Guo Ren , Guo Ren , Alexandre Ghiti Subject: [PATCH v3 3/4] riscv: Add ARCH_HAS_PREFETCH[W] support with Zicbop Date: Mon, 21 Apr 2025 16:24:40 +0200 Message-Id: <20250421142441.395849-4-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20250421142441.395849-1-alexghiti@rivosinc.com> References: <20250421142441.395849-1-alexghiti@rivosinc.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Guo Ren Enable Linux prefetch and prefetchw primitives using Zicbop. Signed-off-by: Guo Ren Signed-off-by: Guo Ren Link: https://lore.kernel.org/r/20231231082955.16516-3-guoren@kernel.org Signed-off-by: Alexandre Ghiti Tested-by: Andrea Parri --- arch/riscv/include/asm/processor.h | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/arch/riscv/include/asm/processor.h b/arch/riscv/include/asm/pr= ocessor.h index 09d4c963399a..39dfab495a4c 100644 --- a/arch/riscv/include/asm/processor.h +++ b/arch/riscv/include/asm/processor.h @@ -13,6 +13,9 @@ #include =20 #include +#include +#include +#include =20 #define arch_get_mmap_end(addr, len, flags) \ ({ \ @@ -135,6 +138,27 @@ static inline void arch_thread_struct_whitelist(unsign= ed long *offset, #define KSTK_EIP(tsk) (task_pt_regs(tsk)->epc) #define KSTK_ESP(tsk) (task_pt_regs(tsk)->sp) =20 +#define PREFETCH_ASM(x) \ + ALTERNATIVE(__nops(1), PREFETCH_R(x, 0), 0, \ + RISCV_ISA_EXT_ZICBOP, CONFIG_RISCV_ISA_ZICBOP) + +#define PREFETCHW_ASM(x) \ + ALTERNATIVE(__nops(1), PREFETCH_W(x, 0), 0, \ + RISCV_ISA_EXT_ZICBOP, CONFIG_RISCV_ISA_ZICBOP) + +#ifdef CONFIG_RISCV_ISA_ZICBOP +#define ARCH_HAS_PREFETCH +static inline void prefetch(const void *x) +{ + __asm__ __volatile__(PREFETCH_ASM(%0) : : "r" (x) : "memory"); +} + +#define ARCH_HAS_PREFETCHW +static inline void prefetchw(const void *x) +{ + __asm__ __volatile__(PREFETCHW_ASM(%0) : : "r" (x) : "memory"); +} +#endif /* CONFIG_RISCV_ISA_ZICBOP */ =20 /* Do necessary setup to start up a newly executed thread. */ extern void start_thread(struct pt_regs *regs, --=20 2.39.2 From nobody Wed Dec 17 03:07:16 2025 Received: from mail-wm1-f54.google.com (mail-wm1-f54.google.com [209.85.128.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 02CDB13CF9C for ; Mon, 21 Apr 2025 14:29:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.54 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745245778; cv=none; b=lVgIVyKXurRQYAGQvAJ2V4vt5GVGOy2KhHAl+hgdzulCKOKPE6QIAKOloAL84W/htaSt1PICnaWByeKxSGZgh/3OlvO58UeqEf61pfrjvmF68niSHSUDvQqkZ3D09/lW1cl4QL4aTLcXoo/yy/0OiAqKbPqmZJG/3c4SSgef0e0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745245778; c=relaxed/simple; bh=roaBEJjGaKm2arNMjpZ8l/EUIQYjq3i4Wt/e2J+fStY=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=b+YBcbuOB9YTe6iXP9Pi+YIWaffuLpUE+pfMJLLWAzyyMzl5cj3N7koq8rp68VfzVmobzyxbCs8Pi05W6HxQk0LUGtUcXpfccbh5df5PAkjJuLOtDwjVV9vWEXFAZUgDEwKt/CRvJqVpiHuCpPepUuLNLqF3XMfz5pcQmzgzehk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com; spf=pass smtp.mailfrom=rivosinc.com; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b=iZsO46rh; arc=none smtp.client-ip=209.85.128.54 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b="iZsO46rh" Received: by mail-wm1-f54.google.com with SMTP id 5b1f17b1804b1-43cfebc343dso34259665e9.2 for ; Mon, 21 Apr 2025 07:29:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1745245775; x=1745850575; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=7PqG7/ov+XLPSS/BI8rl8Yyu60TugcPqJyKJ5mLINmc=; b=iZsO46rh35HDaMu6qmJ8OdFhgkU81Mzj2oI8R3LxDV2m1HZUXrc8yrFsXTm77i0UFG OBGgrWB+seS0Hi7Sor/8hgOk+s4291QqkgbTLN2u/tY+eLz92N68K5uMFh+j3PdDHsV8 8vc3p2WcMISUkcBf5pX2QU7dBl0FKF+zvrsEvKvobcKxy+dYznE/pD76xFwa5JIdkNr9 pKmO36rRa4PYvR8g1IbBH5QA7Ljd7j+hcMMHCh/PE55UCrDASPv48CBD/MhXYI+jaBnX B3tXrIhY7kqztRKuzRjCLCiMYvJxlASBLJf0lnSnUr4RQvNNlLPR92piP1kCOe6b8hTi x/kQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1745245775; x=1745850575; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7PqG7/ov+XLPSS/BI8rl8Yyu60TugcPqJyKJ5mLINmc=; b=dzQiwW17lvtK5QtAYwIqxLO7XtmqMhiijxj29Hiu7mio++lfeF5vMK18A3+rMc4Ly4 OuDA7q7sH8vDzTggRyLFN9UXya4PJxP7RnPPeuR1DxYxmdwfvYZbGR1Ja4LLGERqJkGF Z8sZGoxfb/3/ANkdCYkEvmw1H/ePZlD9bE5AheIBBDtgWzTZ/2/OOadMNT1EyjJjBg4l wUZsDM3YQp+fOzPXGw/jUuKXPRitu0xE+4nZojdxAzXWcgONfEK0Ua5NYHah6BMQ90aE AouJzpptXyeG9CPHLyGCcNf+F5SCIUyY9/sb45qEyc8Hj7t1XBW127BfAtW3kXMVnhGD iX3Q== X-Forwarded-Encrypted: i=1; AJvYcCURb6og5V5FBO6cYLkKIQiolPo6bLtNzS90v3+GZeg+GY0A+Df5zmnIQ5YKt2jblcDSF7u2NzGrceNBprg=@vger.kernel.org X-Gm-Message-State: AOJu0Yx4x34XefoMb7pgihnMuzMG/3P4xvcweZMYeNsFx8c4m1qelSPi Zd/n0PHqUXTAgM0VMfw5LHm5S+Cc/bB1UySpKUW5K3071/E1DnoH1ZT2qWa/jNU= X-Gm-Gg: ASbGncurid9cj87bEZ663hm2RGsN4/1+qJO+CZBPI5VNVkhcW5E0QnCmLGA4AonFYSx NrZ29e9wLpF7FxcCQxVJ5DcwZ0jDecKtOcivzHrqSNw81lliRX+5IlGOmQG0Pepn30Ez2nnu3mC qHAxNhBaMjp8mvzHSYWfT0v5VdnznxLE10AyOlZXodwi3X+W+yLdLOZf4Ff5d71+Yj6auQ+z0k4 jRX7lWuFEB6DUInXuUHEvIgxNBJvExYXpWTrZPaXakAVynxhZ0WS7kJLjaQ/4u1gNeVt7gpYJH9 i+M4TOQKGATl2X4X6HBxBSj3hcE3rx1DtEAwNTzPAvOv06rNlTngMimfbXE= X-Google-Smtp-Source: AGHT+IFtaZI4HL07N6roaGJageZKu+pJvFMTwymIjfPu+o3nCKfh2t3UeftOSZHfxlA79jnfnWiZmg== X-Received: by 2002:a05:600c:3b9b:b0:43c:e478:889 with SMTP id 5b1f17b1804b1-4406aa8984fmr117511685e9.0.1745245775271; Mon, 21 Apr 2025 07:29:35 -0700 (PDT) Received: from localhost.localdomain ([2001:861:3382:ef90:cd58:9e76:e88c:5cd]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-4406d5bbc13sm135524665e9.17.2025.04.21.07.29.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 21 Apr 2025 07:29:34 -0700 (PDT) From: Alexandre Ghiti To: Paul Walmsley , Palmer Dabbelt , Alexandre Ghiti , linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org Cc: Guo Ren , Guo Ren , Alexandre Ghiti Subject: [PATCH v3 4/4] riscv: xchg: Prefetch the destination word for sc.w Date: Mon, 21 Apr 2025 16:24:41 +0200 Message-Id: <20250421142441.395849-5-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20250421142441.395849-1-alexghiti@rivosinc.com> References: <20250421142441.395849-1-alexghiti@rivosinc.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Guo Ren The cost of changing a cacheline from shared to exclusive state can be significant, especially when this is triggered by an exclusive store, since it may result in having to retry the transaction. This patch makes use of prefetch.w to prefetch cachelines for write prior to lr/sc loops when using the xchg_small atomic routine. This patch is inspired by commit 0ea366f5e1b6 ("arm64: atomics: prefetch the destination word for write prior to stxr"). Signed-off-by: Guo Ren Signed-off-by: Guo Ren Link: https://lore.kernel.org/r/20231231082955.16516-4-guoren@kernel.org Signed-off-by: Alexandre Ghiti Tested-by: Andrea Parri --- arch/riscv/include/asm/cmpxchg.h | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpx= chg.h index 2ec119eb147b..0b749e710216 100644 --- a/arch/riscv/include/asm/cmpxchg.h +++ b/arch/riscv/include/asm/cmpxchg.h @@ -13,6 +13,7 @@ #include #include #include +#include =20 #define __arch_xchg_masked(sc_sfx, swap_sfx, prepend, sc_append, \ swap_append, r, p, n) \ @@ -37,6 +38,7 @@ \ __asm__ __volatile__ ( \ prepend \ + PREFETCHW_ASM(%5) \ "0: lr.w %0, %2\n" \ " and %1, %0, %z4\n" \ " or %1, %1, %z3\n" \ @@ -44,7 +46,7 @@ " bnez %1, 0b\n" \ sc_append \ : "=3D&r" (__retx), "=3D&r" (__rc), "+A" (*(__ptr32b)) \ - : "rJ" (__newx), "rJ" (~__mask) \ + : "rJ" (__newx), "rJ" (~__mask), "rJ" (__ptr32b) \ : "memory"); \ \ r =3D (__typeof__(*(p)))((__retx & __mask) >> __s); \ --=20 2.39.2